2024-07-26T10:46:23.852 INFO:root:teuthology version: 0.0.1.dev379+g53ce146 2024-07-26T10:46:23.879 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-07-26T10:46:23.973 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818776 branch: wip-vshankar-testing-20240725.121331-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/no 6-workunit/suites/iogen}} email: vshankar@redhat.com first_in_suite: false job_id: '7818776' kernel: client: branch: testing flavor: default kdb: 1 sha1: 3e9255e907100f88049f320ca33d977e29155d12 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-vshankar-testing-20240725.121331-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug cephsqlite: 20 mgr/snap_schedule/allow_m_granularity: true mgr/snap_schedule/dump_on_update: true mgr/snap_schedule/log_level: debug mon: mon op complaint time: 120 mon warn on pool no app: false osd: osd op complaint time: 180 conf: global: ms die on skipped message: false mds: mds_bal_replicate_threshold: 1 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - slow metadata IO - SLOW_OPS - slow request mgr-modules: - snap_schedule sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-vshankar-testing-20240725.121331-debug sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 owner: scheduled_vshankar@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5532 sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 sleep_before_teardown: 0 subset: 1001/1024 suite: fs suite_branch: wip-vshankar-testing-20240725.121331-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBgC2+03pRICPNULOK/aUxVm4roQzUFrvJVH4JqrpI+oSVBxr836+CybsKhoyT48u4VDaTa6T/SpidZiKFbwdN8= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOXPek7ETb6EibahHYRIVBNDLW19VSlpcgILBoqdpHRNYGAX3nemXQtz0sKQdJ9Esmu2hhYkVmHE3EZJ5LYdvqw= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFJUwaSbCTke8MMjrFWCD9aq+zlE866h/nYNnQdlnWl2B6VSE1fSkS1y1KCnqlziHevtsac7531YLaYjYKBP/K0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/iogen.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 53ce1462e129f6eb4071986336534c740fdebd31 timestamp: 2024-07-26_04:18:59 tube: smithi user: vshankar verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525 2024-07-26T10:46:23.973 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa; will attempt to use it 2024-07-26T10:46:23.974 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/qa/tasks 2024-07-26T10:46:23.974 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-07-26T10:46:23.975 INFO:teuthology.task.internal:Checking packages... 2024-07-26T10:46:23.996 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528' 2024-07-26T10:46:23.996 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-07-26T10:46:23.996 INFO:teuthology.packaging:ref: None 2024-07-26T10:46:23.996 INFO:teuthology.packaging:tag: None 2024-07-26T10:46:23.996 INFO:teuthology.packaging:branch: wip-vshankar-testing-20240725.121331-debug 2024-07-26T10:46:23.996 INFO:teuthology.packaging:sha1: e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:46:23.996 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-vshankar-testing-20240725.121331-debug 2024-07-26T10:46:24.137 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5320.ge8f88bb1 2024-07-26T10:46:24.138 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-07-26T10:46:24.144 INFO:teuthology.task.internal:no buildpackages task found 2024-07-26T10:46:24.144 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-07-26T10:46:24.179 INFO:teuthology.task.internal:Saving configuration 2024-07-26T10:46:24.201 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-07-26T10:46:24.206 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-07-26T10:46:24.231 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818776', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 10:41:17.823425', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBgC2+03pRICPNULOK/aUxVm4roQzUFrvJVH4JqrpI+oSVBxr836+CybsKhoyT48u4VDaTa6T/SpidZiKFbwdN8='} 2024-07-26T10:46:24.259 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818776', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 10:41:17.825752', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOXPek7ETb6EibahHYRIVBNDLW19VSlpcgILBoqdpHRNYGAX3nemXQtz0sKQdJ9Esmu2hhYkVmHE3EZJ5LYdvqw='} 2024-07-26T10:46:24.281 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/vshankar-2024-07-26_04:18:59-fs-wip-vshankar-testing-20240725.121331-debug-testing-default-smithi/7818776', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-07-26 10:41:17.827498', 'locked_by': 'scheduled_vshankar@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFJUwaSbCTke8MMjrFWCD9aq+zlE866h/nYNnQdlnWl2B6VSE1fSkS1y1KCnqlziHevtsac7531YLaYjYKBP/K0='} 2024-07-26T10:46:24.281 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-07-26T10:46:24.290 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-07-26T10:46:24.290 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-07-26T10:46:24.290 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-07-26T10:46:24.291 INFO:teuthology.run_tasks:Running task console_log... 2024-07-26T10:46:24.403 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd3e04bfac0>, signals=[15]) 2024-07-26T10:46:24.403 INFO:teuthology.run_tasks:Running task internal.connect... 2024-07-26T10:46:24.408 INFO:teuthology.task.internal:Opening connections... 2024-07-26T10:46:24.408 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-07-26T10:46:24.410 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:46:24.484 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-07-26T10:46:24.484 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:46:24.558 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2024-07-26T10:46:24.559 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:46:24.633 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-07-26T10:46:24.639 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-07-26T10:46:24.657 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-07-26T10:46:24.658 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-07-26T10:46:24.715 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-07-26T10:46:24.715 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="9" 2024-07-26T10:46:24.715 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-07-26T10:46:24.716 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-07-26T10:46:24.716 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="9" 2024-07-26T10:46:24.716 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el9" 2024-07-26T10:46:24.716 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T10:46:24.716 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-07-26T10:46:24.716 INFO:teuthology.orchestra.run.smithi134.stdout:LOGO="fedora-logo-icon" 2024-07-26T10:46:24.716 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T10:46:24.716 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-07-26T10:46:24.716 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T10:46:24.716 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T10:46:24.716 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T10:46:24.717 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-07-26T10:46:24.741 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-07-26T10:46:24.759 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-07-26T10:46:24.759 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-07-26T10:46:24.816 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="CentOS Stream" 2024-07-26T10:46:24.817 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="9" 2024-07-26T10:46:24.817 INFO:teuthology.orchestra.run.smithi189.stdout:ID="centos" 2024-07-26T10:46:24.817 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="rhel fedora" 2024-07-26T10:46:24.817 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="9" 2024-07-26T10:46:24.817 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el9" 2024-07-26T10:46:24.817 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T10:46:24.817 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31" 2024-07-26T10:46:24.817 INFO:teuthology.orchestra.run.smithi189.stdout:LOGO="fedora-logo-icon" 2024-07-26T10:46:24.817 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T10:46:24.817 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://centos.org/" 2024-07-26T10:46:24.817 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T10:46:24.817 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T10:46:24.817 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T10:46:24.818 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-07-26T10:46:24.840 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2024-07-26T10:46:24.858 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2024-07-26T10:46:24.858 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2024-07-26T10:46:24.915 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Stream" 2024-07-26T10:46:24.915 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="9" 2024-07-26T10:46:24.915 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2024-07-26T10:46:24.915 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2024-07-26T10:46:24.916 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="9" 2024-07-26T10:46:24.916 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el9" 2024-07-26T10:46:24.916 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Stream 9" 2024-07-26T10:46:24.916 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2024-07-26T10:46:24.916 INFO:teuthology.orchestra.run.smithi193.stdout:LOGO="fedora-logo-icon" 2024-07-26T10:46:24.916 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-07-26T10:46:24.916 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2024-07-26T10:46:24.916 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-07-26T10:46:24.917 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-07-26T10:46:24.917 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-07-26T10:46:24.917 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2024-07-26T10:46:24.940 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-07-26T10:46:24.950 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-07-26T10:46:24.981 INFO:teuthology.task.internal:Checking for old test directory... 2024-07-26T10:46:24.981 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-07-26T10:46:24.984 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-07-26T10:46:24.986 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2024-07-26T10:46:25.002 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-07-26T10:46:25.013 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-07-26T10:46:25.013 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-07-26T10:46:25.041 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-07-26T10:46:25.043 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2024-07-26T10:46:25.157 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-07-26T10:46:25.224 INFO:teuthology.run_tasks:Running task kernel... 2024-07-26T10:46:25.237 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': 1, 'sha1': '3e9255e907100f88049f320ca33d977e29155d12'} 2024-07-26T10:46:25.237 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-07-26T10:46:25.237 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-07-26T10:46:25.237 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T10:46:25.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T10:46:25.254 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-07-26T10:46:25.435 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T10:46:25.436 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3e9255e907100f88049f320ca33d977e29155d12"... 2024-07-26T10:46:25.436 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-07-26T10:46:25.453 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-437.el9.x86_64 2024-07-26T10:46:25.453 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T10:46:25.453 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-07-26T10:46:25.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum upgrade -y linux-firmware 2024-07-26T10:46:27.892 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-07-26T10:46:29.809 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-07-26T10:46:35.120 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.5 MB 00:01 2024-07-26T10:46:36.668 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 18 kB 00:00 2024-07-26T10:46:37.534 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 30 MB/s | 22 MB 00:00 2024-07-26T10:46:42.778 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 50 kB/s | 1.7 kB 00:00 2024-07-26T10:46:44.884 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-07-26T10:46:44.892 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-07-26T10:46:44.892 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-07-26T10:46:44.892 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-07-26T10:46:44.892 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-07-26T10:46:44.892 INFO:teuthology.orchestra.run.smithi134.stdout: linux-firmware noarch 20240716-145.el9 baseos 383 M 2024-07-26T10:46:44.892 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:46:44.892 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-07-26T10:46:44.892 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-07-26T10:46:44.893 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 1 Package 2024-07-26T10:46:44.893 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:46:44.893 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 383 M 2024-07-26T10:46:44.893 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-07-26T10:47:14.843 INFO:teuthology.orchestra.run.smithi134.stdout:linux-firmware-20240716-145.el9.noarch.rpm 13 MB/s | 383 MB 00:29 2024-07-26T10:47:14.843 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-07-26T10:47:14.844 INFO:teuthology.orchestra.run.smithi134.stdout:Total 13 MB/s | 383 MB 00:29 2024-07-26T10:47:17.918 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-07-26T10:47:18.067 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-07-26T10:47:18.067 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-07-26T10:47:21.306 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-07-26T10:47:21.307 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-07-26T10:47:30.215 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-07-26T10:47:30.976 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T10:47:31.058 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T10:47:31.632 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T10:47:31.632 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : linux-firmware-20240716-145.el9.noarch 1/2 2024-07-26T10:47:32.322 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 2/2 2024-07-26T10:47:32.322 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:47:32.322 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-07-26T10:47:32.322 INFO:teuthology.orchestra.run.smithi134.stdout: linux-firmware-20240716-145.el9.noarch 2024-07-26T10:47:32.322 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:47:32.322 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-07-26T10:47:32.652 INFO:teuthology.task.kernel:Downloading kernel 3e9255e907100f88049f320ca33d977e29155d12 on client.0... 2024-07-26T10:47:32.653 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3e9255e907100f88049f320ca33d977e29155d12 2024-07-26T10:47:32.798 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ 2024-07-26T10:47:32.799 DEBUG:teuthology.orchestra.run.smithi134:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.10.0_g3e9255e90710-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/ --input-file=- 2024-07-26T10:47:32.801 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi134.front.sepia.ceph.com to complete... 2024-07-26T10:47:43.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26 10:47:43 URL:https://1.chacra.ceph.com/r/kernel/testing/3e9255e907100f88049f320ca33d977e29155d12/centos/9/flavors/default/x86_64/kernel-6.10.0_g3e9255e90710-1.x86_64.rpm [358399635/358399635] -> "/tmp/kernel.x86_64.rpm" [1] 2024-07-26T10:47:43.424 INFO:teuthology.orchestra.run.smithi134.stderr:FINISHED --2024-07-26 10:47:43-- 2024-07-26T10:47:43.424 INFO:teuthology.orchestra.run.smithi134.stderr:Total wall clock time: 11s 2024-07-26T10:47:43.424 INFO:teuthology.orchestra.run.smithi134.stderr:Downloaded: 1 files, 342M in 10s (33.5 MB/s) 2024-07-26T10:47:43.427 INFO:teuthology.task.kernel:Installing kernel 3e9255e907100f88049f320ca33d977e29155d12 on client.0... 2024-07-26T10:47:43.428 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-07-26T10:47:45.479 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying... ######################################## 2024-07-26T10:47:45.519 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing... ######################################## 2024-07-26T10:47:45.570 INFO:teuthology.orchestra.run.smithi134.stdout:Updating / installing... 2024-07-26T10:47:53.563 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-6.10.0_g3e9255e90710-1 ######################################## 2024-07-26T10:48:33.335 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-07-26T10:48:33.336 DEBUG:teuthology.orchestra.run.smithi134:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-07-26T10:48:33.416 INFO:teuthology.orchestra.run.smithi134.stdout:/boot/System.map-6.10.0-g3e9255e90710 2024-07-26T10:48:33.416 INFO:teuthology.orchestra.run.smithi134.stdout:/boot/config-6.10.0-g3e9255e90710 2024-07-26T10:48:33.416 INFO:teuthology.orchestra.run.smithi134.stdout:/boot/vmlinuz-6.10.0-g3e9255e90710 2024-07-26T10:48:33.417 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710 2024-07-26T10:48:33.417 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/System.map 2024-07-26T10:48:33.417 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/config 2024-07-26T10:48:33.417 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel 2024-07-26T10:48:33.417 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch 2024-07-26T10:48:33.417 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86 2024-07-26T10:48:33.417 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto 2024-07-26T10:48:33.417 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-07-26T10:48:33.417 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-07-26T10:48:33.417 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-07-26T10:48:33.417 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-07-26T10:48:33.417 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-07-26T10:48:33.417 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-07-26T10:48:33.418 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-07-26T10:48:33.418 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crc32c-intel.ko 2024-07-26T10:48:33.418 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-07-26T10:48:33.418 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-07-26T10:48:33.418 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-07-26T10:48:33.418 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-07-26T10:48:33.418 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-avx2.ko 2024-07-26T10:48:33.418 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-07-26T10:48:33.418 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-07-26T10:48:33.418 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-07-26T10:48:33.418 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-07-26T10:48:33.418 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-07-26T10:48:33.418 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-07-26T10:48:33.419 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-07-26T10:48:33.419 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel 2024-07-26T10:48:33.419 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu 2024-07-26T10:48:33.419 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce 2024-07-26T10:48:33.419 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-07-26T10:48:33.419 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm 2024-07-26T10:48:33.419 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-amd.ko 2024-07-26T10:48:33.419 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm-intel.ko 2024-07-26T10:48:33.419 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/arch/x86/kvm/kvm.ko 2024-07-26T10:48:33.419 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto 2024-07-26T10:48:33.419 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ansi_cprng.ko 2024-07-26T10:48:33.419 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/anubis.ko 2024-07-26T10:48:33.419 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/arc4.ko 2024-07-26T10:48:33.419 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx 2024-07-26T10:48:33.420 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_memcpy.ko 2024-07-26T10:48:33.420 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_pq.ko 2024-07-26T10:48:33.420 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_raid6_recov.ko 2024-07-26T10:48:33.420 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_tx.ko 2024-07-26T10:48:33.420 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/async_xor.ko 2024-07-26T10:48:33.420 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/async_tx/raid6test.ko 2024-07-26T10:48:33.420 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authenc.ko 2024-07-26T10:48:33.420 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/authencesn.ko 2024-07-26T10:48:33.420 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blake2b_generic.ko 2024-07-26T10:48:33.420 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_common.ko 2024-07-26T10:48:33.420 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/blowfish_generic.ko 2024-07-26T10:48:33.420 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/camellia_generic.ko 2024-07-26T10:48:33.421 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast5_generic.ko 2024-07-26T10:48:33.421 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast6_generic.ko 2024-07-26T10:48:33.421 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cast_common.ko 2024-07-26T10:48:33.421 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ccm.ko 2024-07-26T10:48:33.421 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cmac.ko 2024-07-26T10:48:33.421 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crc32_generic.ko 2024-07-26T10:48:33.421 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_engine.ko 2024-07-26T10:48:33.421 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/crypto_user.ko 2024-07-26T10:48:33.421 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/cts.ko 2024-07-26T10:48:33.421 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/des_generic.ko 2024-07-26T10:48:33.421 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/dh_generic.ko 2024-07-26T10:48:33.421 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecc.ko 2024-07-26T10:48:33.421 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/ecdh_generic.ko 2024-07-26T10:48:33.422 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/echainiv.ko 2024-07-26T10:48:33.422 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/essiv.ko 2024-07-26T10:48:33.422 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/fcrypt.ko 2024-07-26T10:48:33.422 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/khazad.ko 2024-07-26T10:48:33.422 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4.ko 2024-07-26T10:48:33.422 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/lz4hc.ko 2024-07-26T10:48:33.422 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/md4.ko 2024-07-26T10:48:33.422 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/michael_mic.ko 2024-07-26T10:48:33.422 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcbc.ko 2024-07-26T10:48:33.422 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/pcrypt.ko 2024-07-26T10:48:33.422 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/rmd160.ko 2024-07-26T10:48:33.422 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/seed.ko 2024-07-26T10:48:33.423 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/serpent_generic.ko 2024-07-26T10:48:33.423 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tcrypt.ko 2024-07-26T10:48:33.423 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/tea.ko 2024-07-26T10:48:33.423 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_common.ko 2024-07-26T10:48:33.423 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/twofish_generic.ko 2024-07-26T10:48:33.423 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/vmac.ko 2024-07-26T10:48:33.423 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/wp512.ko 2024-07-26T10:48:33.423 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xcbc.ko 2024-07-26T10:48:33.423 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xor.ko 2024-07-26T10:48:33.423 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/crypto/xxhash_generic.ko 2024-07-26T10:48:33.423 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers 2024-07-26T10:48:33.423 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi 2024-07-26T10:48:33.423 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_ipmi.ko 2024-07-26T10:48:33.424 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/acpi_pad.ko 2024-07-26T10:48:33.424 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/ec_sys.ko 2024-07-26T10:48:33.424 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/platform_profile.ko 2024-07-26T10:48:33.424 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbs.ko 2024-07-26T10:48:33.424 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/sbshc.ko 2024-07-26T10:48:33.424 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/acpi/video.ko 2024-07-26T10:48:33.424 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata 2024-07-26T10:48:33.424 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/acard-ahci.ko 2024-07-26T10:48:33.424 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ahci_platform.ko 2024-07-26T10:48:33.424 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/ata_generic.ko 2024-07-26T10:48:33.424 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/libahci_platform.ko 2024-07-26T10:48:33.424 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_acpi.ko 2024-07-26T10:48:33.425 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ali.ko 2024-07-26T10:48:33.425 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_amd.ko 2024-07-26T10:48:33.425 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_artop.ko 2024-07-26T10:48:33.425 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atiixp.ko 2024-07-26T10:48:33.425 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_atp867x.ko 2024-07-26T10:48:33.425 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd640.ko 2024-07-26T10:48:33.425 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cmd64x.ko 2024-07-26T10:48:33.425 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_cypress.ko 2024-07-26T10:48:33.425 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_efar.ko 2024-07-26T10:48:33.425 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt366.ko 2024-07-26T10:48:33.425 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt37x.ko 2024-07-26T10:48:33.425 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x2n.ko 2024-07-26T10:48:33.425 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_hpt3x3.ko 2024-07-26T10:48:33.426 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it8213.ko 2024-07-26T10:48:33.426 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_it821x.ko 2024-07-26T10:48:33.426 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_jmicron.ko 2024-07-26T10:48:33.426 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_marvell.ko 2024-07-26T10:48:33.426 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_mpiix.ko 2024-07-26T10:48:33.426 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_netcell.ko 2024-07-26T10:48:33.426 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ninja32.ko 2024-07-26T10:48:33.426 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87410.ko 2024-07-26T10:48:33.426 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_ns87415.ko 2024-07-26T10:48:33.426 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_oldpiix.ko 2024-07-26T10:48:33.426 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_opti.ko 2024-07-26T10:48:33.426 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_optidma.ko 2024-07-26T10:48:33.427 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pcmcia.ko 2024-07-26T10:48:33.427 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc2027x.ko 2024-07-26T10:48:33.427 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-07-26T10:48:33.427 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_piccolo.ko 2024-07-26T10:48:33.427 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_rdc.ko 2024-07-26T10:48:33.427 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sch.ko 2024-07-26T10:48:33.427 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_serverworks.ko 2024-07-26T10:48:33.427 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sil680.ko 2024-07-26T10:48:33.427 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sis.ko 2024-07-26T10:48:33.427 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_sl82c105.ko 2024-07-26T10:48:33.427 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_triflex.ko 2024-07-26T10:48:33.427 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pata_via.ko 2024-07-26T10:48:33.427 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/pdc_adma.ko 2024-07-26T10:48:33.428 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_inic162x.ko 2024-07-26T10:48:33.428 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_mv.ko 2024-07-26T10:48:33.428 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_nv.ko 2024-07-26T10:48:33.428 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_promise.ko 2024-07-26T10:48:33.428 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_qstor.ko 2024-07-26T10:48:33.428 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil.ko 2024-07-26T10:48:33.428 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sil24.ko 2024-07-26T10:48:33.428 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sis.ko 2024-07-26T10:48:33.428 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_svw.ko 2024-07-26T10:48:33.428 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_sx4.ko 2024-07-26T10:48:33.428 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_uli.ko 2024-07-26T10:48:33.428 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_via.ko 2024-07-26T10:48:33.428 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ata/sata_vsc.ko 2024-07-26T10:48:33.429 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay 2024-07-26T10:48:33.429 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/auxdisplay/ks0108.ko 2024-07-26T10:48:33.429 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base 2024-07-26T10:48:33.429 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap 2024-07-26T10:48:33.429 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-i2c.ko 2024-07-26T10:48:33.429 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/base/regmap/regmap-mmio.ko 2024-07-26T10:48:33.429 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma 2024-07-26T10:48:33.429 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/bcma/bcma.ko 2024-07-26T10:48:33.429 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block 2024-07-26T10:48:33.429 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe 2024-07-26T10:48:33.429 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/aoe/aoe.ko 2024-07-26T10:48:33.429 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/brd.ko 2024-07-26T10:48:33.429 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd 2024-07-26T10:48:33.430 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/drbd/drbd.ko 2024-07-26T10:48:33.430 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/floppy.ko 2024-07-26T10:48:33.430 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/loop.ko 2024-07-26T10:48:33.430 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx 2024-07-26T10:48:33.430 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-07-26T10:48:33.430 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/nbd.ko 2024-07-26T10:48:33.430 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk 2024-07-26T10:48:33.430 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/null_blk/null_blk.ko 2024-07-26T10:48:33.430 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/pktcdvd.ko 2024-07-26T10:48:33.430 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/rbd.ko 2024-07-26T10:48:33.430 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/virtio_blk.ko 2024-07-26T10:48:33.430 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback 2024-07-26T10:48:33.430 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-07-26T10:48:33.431 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/xen-blkfront.ko 2024-07-26T10:48:33.431 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram 2024-07-26T10:48:33.431 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/block/zram/zram.ko 2024-07-26T10:48:33.431 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char 2024-07-26T10:48:33.431 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hangcheck-timer.ko 2024-07-26T10:48:33.431 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random 2024-07-26T10:48:33.431 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/amd-rng.ko 2024-07-26T10:48:33.431 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/intel-rng.ko 2024-07-26T10:48:33.431 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-07-26T10:48:33.431 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/via-rng.ko 2024-07-26T10:48:33.431 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/hw_random/virtio-rng.ko 2024-07-26T10:48:33.431 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi 2024-07-26T10:48:33.431 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-07-26T10:48:33.432 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-07-26T10:48:33.432 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-07-26T10:48:33.432 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_si.ko 2024-07-26T10:48:33.432 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-07-26T10:48:33.432 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-07-26T10:48:33.432 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/lp.ko 2024-07-26T10:48:33.432 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave 2024-07-26T10:48:33.432 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/mwave/mwave.ko 2024-07-26T10:48:33.432 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/ppdev.ko 2024-07-26T10:48:33.432 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tlclk.ko 2024-07-26T10:48:33.432 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm 2024-07-26T10:48:33.432 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm.ko 2024-07-26T10:48:33.432 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_atmel.ko 2024-07-26T10:48:33.432 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_infineon.ko 2024-07-26T10:48:33.433 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_nsc.ko 2024-07-26T10:48:33.433 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis.ko 2024-07-26T10:48:33.433 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-07-26T10:48:33.433 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/uv_mmtimer.ko 2024-07-26T10:48:33.433 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/char/virtio_console.ko 2024-07-26T10:48:33.433 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq 2024-07-26T10:48:33.433 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-07-26T10:48:33.433 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-07-26T10:48:33.433 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/p4-clockmod.ko 2024-07-26T10:48:33.433 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-07-26T10:48:33.433 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/powernow-k8.ko 2024-07-26T10:48:33.433 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/cpufreq/speedstep-lib.ko 2024-07-26T10:48:33.433 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto 2024-07-26T10:48:33.433 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp 2024-07-26T10:48:33.434 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-07-26T10:48:33.434 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/ccp/ccp.ko 2024-07-26T10:48:33.434 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel 2024-07-26T10:48:33.434 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat 2024-07-26T10:48:33.434 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common 2024-07-26T10:48:33.434 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-07-26T10:48:33.434 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-07-26T10:48:33.434 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-07-26T10:48:33.434 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-aes.ko 2024-07-26T10:48:33.434 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/padlock-sha.ko 2024-07-26T10:48:33.434 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio 2024-07-26T10:48:33.434 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-07-26T10:48:33.434 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca 2024-07-26T10:48:33.435 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dca/dca.ko 2024-07-26T10:48:33.435 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq 2024-07-26T10:48:33.435 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-07-26T10:48:33.435 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma 2024-07-26T10:48:33.435 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw 2024-07-26T10:48:33.435 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/dw/dw_dmac.ko 2024-07-26T10:48:33.435 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat 2024-07-26T10:48:33.435 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/dma/ioat/ioatdma.ko 2024-07-26T10:48:33.435 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac 2024-07-26T10:48:33.435 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/amd64_edac.ko 2024-07-26T10:48:33.435 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/e752x_edac.ko 2024-07-26T10:48:33.435 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/edac_mce_amd.ko 2024-07-26T10:48:33.435 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3000_edac.ko 2024-07-26T10:48:33.436 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i3200_edac.ko 2024-07-26T10:48:33.436 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5100_edac.ko 2024-07-26T10:48:33.436 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i5400_edac.ko 2024-07-26T10:48:33.436 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7300_edac.ko 2024-07-26T10:48:33.436 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i7core_edac.ko 2024-07-26T10:48:33.436 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/i82975x_edac.ko 2024-07-26T10:48:33.436 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/ie31200_edac.ko 2024-07-26T10:48:33.436 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/sb_edac.ko 2024-07-26T10:48:33.436 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/edac/x38_edac.ko 2024-07-26T10:48:33.436 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware 2024-07-26T10:48:33.436 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/edd.ko 2024-07-26T10:48:33.436 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/firmware/iscsi_ibft.ko 2024-07-26T10:48:33.436 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio 2024-07-26T10:48:33.436 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-ich.ko 2024-07-26T10:48:33.437 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/gpio/gpio-viperboard.ko 2024-07-26T10:48:33.437 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid 2024-07-26T10:48:33.437 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-appleir.ko 2024-07-26T10:48:33.437 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-aureal.ko 2024-07-26T10:48:33.437 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-axff.ko 2024-07-26T10:48:33.437 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-dr.ko 2024-07-26T10:48:33.437 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elecom.ko 2024-07-26T10:48:33.437 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-elo.ko 2024-07-26T10:48:33.437 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-emsff.ko 2024-07-26T10:48:33.437 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gaff.ko 2024-07-26T10:48:33.437 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gt683r.ko 2024-07-26T10:48:33.437 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-gyration.ko 2024-07-26T10:48:33.437 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-kbd.ko 2024-07-26T10:48:33.438 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtek-mouse.ko 2024-07-26T10:48:33.438 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-holtekff.ko 2024-07-26T10:48:33.438 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-hyperv.ko 2024-07-26T10:48:33.438 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-icade.ko 2024-07-26T10:48:33.438 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-keytouch.ko 2024-07-26T10:48:33.438 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-kye.ko 2024-07-26T10:48:33.438 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lcpower.ko 2024-07-26T10:48:33.438 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-led.ko 2024-07-26T10:48:33.438 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-lenovo.ko 2024-07-26T10:48:33.438 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-dj.ko 2024-07-26T10:48:33.438 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-07-26T10:48:33.438 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-multitouch.ko 2024-07-26T10:48:33.438 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-ortek.ko 2024-07-26T10:48:33.439 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-penmount.ko 2024-07-26T10:48:33.439 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-petalynx.ko 2024-07-26T10:48:33.439 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-picolcd.ko 2024-07-26T10:48:33.439 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-pl.ko 2024-07-26T10:48:33.439 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-plantronics.ko 2024-07-26T10:48:33.439 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-primax.ko 2024-07-26T10:48:33.439 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-rmi.ko 2024-07-26T10:48:33.439 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-arvo.ko 2024-07-26T10:48:33.439 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-common.ko 2024-07-26T10:48:33.439 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-isku.ko 2024-07-26T10:48:33.439 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kone.ko 2024-07-26T10:48:33.439 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-07-26T10:48:33.440 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-konepure.ko 2024-07-26T10:48:33.440 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-07-26T10:48:33.440 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-lua.ko 2024-07-26T10:48:33.440 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-pyra.ko 2024-07-26T10:48:33.440 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-ryos.ko 2024-07-26T10:48:33.440 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat-savu.ko 2024-07-26T10:48:33.440 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-roccat.ko 2024-07-26T10:48:33.440 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-saitek.ko 2024-07-26T10:48:33.440 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-samsung.ko 2024-07-26T10:48:33.440 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sensor-hub.ko 2024-07-26T10:48:33.440 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sjoy.ko 2024-07-26T10:48:33.440 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sony.ko 2024-07-26T10:48:33.440 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-speedlink.ko 2024-07-26T10:48:33.440 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-steelseries.ko 2024-07-26T10:48:33.441 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-sunplus.ko 2024-07-26T10:48:33.441 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-thrustmaster.ko 2024-07-26T10:48:33.441 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tivo.ko 2024-07-26T10:48:33.441 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-tmff.ko 2024-07-26T10:48:33.441 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-topseed.ko 2024-07-26T10:48:33.441 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-twinhan.ko 2024-07-26T10:48:33.441 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-uclogic.ko 2024-07-26T10:48:33.441 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-waltop.ko 2024-07-26T10:48:33.441 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-wiimote.ko 2024-07-26T10:48:33.441 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-xinmo.ko 2024-07-26T10:48:33.441 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zpff.ko 2024-07-26T10:48:33.441 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/hid-zydacron.ko 2024-07-26T10:48:33.441 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/uhid.ko 2024-07-26T10:48:33.442 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hid/wacom.ko 2024-07-26T10:48:33.442 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv 2024-07-26T10:48:33.442 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_balloon.ko 2024-07-26T10:48:33.442 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_utils.ko 2024-07-26T10:48:33.442 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hv/hv_vmbus.ko 2024-07-26T10:48:33.442 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon 2024-07-26T10:48:33.442 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru.ko 2024-07-26T10:48:33.442 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/abituguru3.ko 2024-07-26T10:48:33.442 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/acpi_power_meter.ko 2024-07-26T10:48:33.442 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7414.ko 2024-07-26T10:48:33.442 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ad7418.ko 2024-07-26T10:48:33.442 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adc128d818.ko 2024-07-26T10:48:33.443 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1025.ko 2024-07-26T10:48:33.443 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1026.ko 2024-07-26T10:48:33.443 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1029.ko 2024-07-26T10:48:33.443 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm1031.ko 2024-07-26T10:48:33.443 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adm9240.ko 2024-07-26T10:48:33.443 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ads7828.ko 2024-07-26T10:48:33.443 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7410.ko 2024-07-26T10:48:33.443 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7411.ko 2024-07-26T10:48:33.443 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7462.ko 2024-07-26T10:48:33.443 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7470.ko 2024-07-26T10:48:33.443 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7475.ko 2024-07-26T10:48:33.443 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/adt7x10.ko 2024-07-26T10:48:33.443 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/amc6821.ko 2024-07-26T10:48:33.444 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/applesmc.ko 2024-07-26T10:48:33.444 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asb100.ko 2024-07-26T10:48:33.444 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asc7621.ko 2024-07-26T10:48:33.444 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/asus_atk0110.ko 2024-07-26T10:48:33.444 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/atxp1.ko 2024-07-26T10:48:33.444 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/coretemp.ko 2024-07-26T10:48:33.444 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-07-26T10:48:33.444 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/dme1737.ko 2024-07-26T10:48:33.444 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds1621.ko 2024-07-26T10:48:33.444 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ds620.ko 2024-07-26T10:48:33.444 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc1403.ko 2024-07-26T10:48:33.444 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/emc6w201.ko 2024-07-26T10:48:33.444 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71805f.ko 2024-07-26T10:48:33.444 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f71882fg.ko 2024-07-26T10:48:33.445 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/f75375s.ko 2024-07-26T10:48:33.445 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fam15h_power.ko 2024-07-26T10:48:33.445 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/fschmd.ko 2024-07-26T10:48:33.445 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g760a.ko 2024-07-26T10:48:33.445 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/g762.ko 2024-07-26T10:48:33.445 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl518sm.ko 2024-07-26T10:48:33.445 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/gl520sm.ko 2024-07-26T10:48:33.445 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/hwmon-vid.ko 2024-07-26T10:48:33.445 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5500_temp.ko 2024-07-26T10:48:33.445 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/i5k_amb.ko 2024-07-26T10:48:33.445 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmaem.ko 2024-07-26T10:48:33.445 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ibmpex.ko 2024-07-26T10:48:33.445 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina209.ko 2024-07-26T10:48:33.446 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ina2xx.ko 2024-07-26T10:48:33.446 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/it87.ko 2024-07-26T10:48:33.446 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k10temp.ko 2024-07-26T10:48:33.446 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/k8temp.ko 2024-07-26T10:48:33.446 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lineage-pem.ko 2024-07-26T10:48:33.446 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm63.ko 2024-07-26T10:48:33.446 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm73.ko 2024-07-26T10:48:33.446 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm75.ko 2024-07-26T10:48:33.446 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm77.ko 2024-07-26T10:48:33.446 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm78.ko 2024-07-26T10:48:33.446 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm80.ko 2024-07-26T10:48:33.446 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm83.ko 2024-07-26T10:48:33.446 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm85.ko 2024-07-26T10:48:33.447 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm87.ko 2024-07-26T10:48:33.447 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm90.ko 2024-07-26T10:48:33.447 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm92.ko 2024-07-26T10:48:33.447 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm93.ko 2024-07-26T10:48:33.447 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95234.ko 2024-07-26T10:48:33.447 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95241.ko 2024-07-26T10:48:33.447 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/lm95245.ko 2024-07-26T10:48:33.447 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc2945.ko 2024-07-26T10:48:33.447 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4151.ko 2024-07-26T10:48:33.447 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4215.ko 2024-07-26T10:48:33.447 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4222.ko 2024-07-26T10:48:33.447 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4245.ko 2024-07-26T10:48:33.447 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4260.ko 2024-07-26T10:48:33.447 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/ltc4261.ko 2024-07-26T10:48:33.448 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max16065.ko 2024-07-26T10:48:33.448 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1619.ko 2024-07-26T10:48:33.448 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max1668.ko 2024-07-26T10:48:33.448 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max197.ko 2024-07-26T10:48:33.448 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6639.ko 2024-07-26T10:48:33.448 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6650.ko 2024-07-26T10:48:33.448 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/max6697.ko 2024-07-26T10:48:33.448 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/mcp3021.ko 2024-07-26T10:48:33.448 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6683.ko 2024-07-26T10:48:33.448 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775-core.ko 2024-07-26T10:48:33.448 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct6775.ko 2024-07-26T10:48:33.448 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/nct7802.ko 2024-07-26T10:48:33.448 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87360.ko 2024-07-26T10:48:33.449 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pc87427.ko 2024-07-26T10:48:33.449 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pcf8591.ko 2024-07-26T10:48:33.449 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus 2024-07-26T10:48:33.449 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-07-26T10:48:33.449 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-07-26T10:48:33.449 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-07-26T10:48:33.449 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max16064.ko 2024-07-26T10:48:33.449 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max34440.ko 2024-07-26T10:48:33.449 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/max8688.ko 2024-07-26T10:48:33.449 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-07-26T10:48:33.449 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-07-26T10:48:33.449 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-07-26T10:48:33.449 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-07-26T10:48:33.449 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-07-26T10:48:33.450 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-07-26T10:48:33.450 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/powr1220.ko 2024-07-26T10:48:33.450 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5627.ko 2024-07-26T10:48:33.450 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch5636.ko 2024-07-26T10:48:33.450 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sch56xx-common.ko 2024-07-26T10:48:33.450 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht15.ko 2024-07-26T10:48:33.450 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sht21.ko 2024-07-26T10:48:33.450 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/shtc1.ko 2024-07-26T10:48:33.450 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/sis5595.ko 2024-07-26T10:48:33.450 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47b397.ko 2024-07-26T10:48:33.450 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m1.ko 2024-07-26T10:48:33.450 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/smsc47m192.ko 2024-07-26T10:48:33.450 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/thmc50.ko 2024-07-26T10:48:33.451 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp102.ko 2024-07-26T10:48:33.451 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp103.ko 2024-07-26T10:48:33.451 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp401.ko 2024-07-26T10:48:33.451 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/tmp421.ko 2024-07-26T10:48:33.451 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via-cputemp.ko 2024-07-26T10:48:33.451 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/via686a.ko 2024-07-26T10:48:33.451 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt1211.ko 2024-07-26T10:48:33.451 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/vt8231.ko 2024-07-26T10:48:33.451 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627ehf.ko 2024-07-26T10:48:33.451 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83627hf.ko 2024-07-26T10:48:33.451 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83781d.ko 2024-07-26T10:48:33.451 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83791d.ko 2024-07-26T10:48:33.451 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83792d.ko 2024-07-26T10:48:33.451 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83793.ko 2024-07-26T10:48:33.452 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83795.ko 2024-07-26T10:48:33.452 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l785ts.ko 2024-07-26T10:48:33.452 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/hwmon/w83l786ng.ko 2024-07-26T10:48:33.452 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c 2024-07-26T10:48:33.452 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos 2024-07-26T10:48:33.452 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-07-26T10:48:33.452 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-07-26T10:48:33.452 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses 2024-07-26T10:48:33.452 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-07-26T10:48:33.452 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-07-26T10:48:33.452 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-07-26T10:48:33.452 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-07-26T10:48:33.452 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-07-26T10:48:33.452 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-07-26T10:48:33.453 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-07-26T10:48:33.453 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-07-26T10:48:33.453 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-i801.ko 2024-07-26T10:48:33.453 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-isch.ko 2024-07-26T10:48:33.453 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-07-26T10:48:33.453 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-07-26T10:48:33.453 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-07-26T10:48:33.453 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-parport.ko 2024-07-26T10:48:33.453 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-07-26T10:48:33.453 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-07-26T10:48:33.453 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-07-26T10:48:33.453 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-07-26T10:48:33.453 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-07-26T10:48:33.453 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-07-26T10:48:33.454 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-via.ko 2024-07-26T10:48:33.454 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-07-26T10:48:33.454 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-07-26T10:48:33.454 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-dev.ko 2024-07-26T10:48:33.454 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-mux.ko 2024-07-26T10:48:33.454 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-07-26T10:48:33.454 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-smbus.ko 2024-07-26T10:48:33.454 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/i2c/i2c-stub.ko 2024-07-26T10:48:33.454 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband 2024-07-26T10:48:33.454 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core 2024-07-26T10:48:33.454 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_cm.ko 2024-07-26T10:48:33.454 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_core.ko 2024-07-26T10:48:33.455 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_umad.ko 2024-07-26T10:48:33.455 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-07-26T10:48:33.455 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/iw_cm.ko 2024-07-26T10:48:33.455 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_cm.ko 2024-07-26T10:48:33.455 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-07-26T10:48:33.455 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw 2024-07-26T10:48:33.455 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4 2024-07-26T10:48:33.455 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-07-26T10:48:33.455 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4 2024-07-26T10:48:33.455 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-07-26T10:48:33.455 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5 2024-07-26T10:48:33.455 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-07-26T10:48:33.455 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca 2024-07-26T10:48:33.455 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-07-26T10:48:33.456 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp 2024-07-26T10:48:33.456 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib 2024-07-26T10:48:33.456 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-07-26T10:48:33.456 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser 2024-07-26T10:48:33.456 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-07-26T10:48:33.456 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert 2024-07-26T10:48:33.456 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-07-26T10:48:33.456 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp 2024-07-26T10:48:33.456 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-07-26T10:48:33.456 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt 2024-07-26T10:48:33.456 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-07-26T10:48:33.456 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input 2024-07-26T10:48:33.456 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport 2024-07-26T10:48:33.457 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-07-26T10:48:33.457 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/fm801-gp.ko 2024-07-26T10:48:33.457 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/gameport/gameport.ko 2024-07-26T10:48:33.457 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/joydev.ko 2024-07-26T10:48:33.457 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard 2024-07-26T10:48:33.457 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/keyboard/gpio_keys.ko 2024-07-26T10:48:33.457 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse 2024-07-26T10:48:33.457 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/appletouch.ko 2024-07-26T10:48:33.457 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/bcm5974.ko 2024-07-26T10:48:33.457 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/cyapatp.ko 2024-07-26T10:48:33.457 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/elan_i2c.ko 2024-07-26T10:48:33.457 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/sermouse.ko 2024-07-26T10:48:33.457 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-07-26T10:48:33.457 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/synaptics_usb.ko 2024-07-26T10:48:33.458 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/mouse/vsxxxaa.ko 2024-07-26T10:48:33.458 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4 2024-07-26T10:48:33.458 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/rmi4/rmi_core.ko 2024-07-26T10:48:33.458 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio 2024-07-26T10:48:33.458 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/altera_ps2.ko 2024-07-26T10:48:33.458 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/arc_ps2.ko 2024-07-26T10:48:33.458 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-07-26T10:48:33.458 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/serio/serio_raw.ko 2024-07-26T10:48:33.458 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/input/sparse-keymap.ko 2024-07-26T10:48:33.458 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds 2024-07-26T10:48:33.458 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-blinkm.ko 2024-07-26T10:48:33.458 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lm3530.ko 2024-07-26T10:48:33.458 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-lp3944.ko 2024-07-26T10:48:33.459 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/leds-ss4200.ko 2024-07-26T10:48:33.459 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger 2024-07-26T10:48:33.459 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-07-26T10:48:33.459 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-07-26T10:48:33.459 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-07-26T10:48:33.459 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-07-26T10:48:33.459 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-07-26T10:48:33.459 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-07-26T10:48:33.459 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-07-26T10:48:33.459 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-07-26T10:48:33.459 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md 2024-07-26T10:48:33.459 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache 2024-07-26T10:48:33.459 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/bcache/bcache.ko 2024-07-26T10:48:33.459 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-bio-prison.ko 2024-07-26T10:48:33.460 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache-smq.ko 2024-07-26T10:48:33.460 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-cache.ko 2024-07-26T10:48:33.460 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-crypt.ko 2024-07-26T10:48:33.460 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-delay.ko 2024-07-26T10:48:33.460 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-flakey.ko 2024-07-26T10:48:33.460 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-log-userspace.ko 2024-07-26T10:48:33.460 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-multipath.ko 2024-07-26T10:48:33.460 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-queue-length.ko 2024-07-26T10:48:33.460 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-raid.ko 2024-07-26T10:48:33.460 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-round-robin.ko 2024-07-26T10:48:33.460 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-service-time.ko 2024-07-26T10:48:33.460 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-switch.ko 2024-07-26T10:48:33.460 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-thin-pool.ko 2024-07-26T10:48:33.460 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/dm-verity.ko 2024-07-26T10:48:33.460 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data 2024-07-26T10:48:33.461 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-07-26T10:48:33.461 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid0.ko 2024-07-26T10:48:33.461 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid1.ko 2024-07-26T10:48:33.461 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid10.ko 2024-07-26T10:48:33.461 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/md/raid456.ko 2024-07-26T10:48:33.461 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message 2024-07-26T10:48:33.461 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion 2024-07-26T10:48:33.461 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptbase.ko 2024-07-26T10:48:33.461 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptctl.ko 2024-07-26T10:48:33.461 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptfc.ko 2024-07-26T10:48:33.461 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptlan.ko 2024-07-26T10:48:33.461 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptsas.ko 2024-07-26T10:48:33.461 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptscsih.ko 2024-07-26T10:48:33.462 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/message/fusion/mptspi.ko 2024-07-26T10:48:33.462 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd 2024-07-26T10:48:33.462 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_ich.ko 2024-07-26T10:48:33.462 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/lpc_sch.ko 2024-07-26T10:48:33.462 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/mfd-core.ko 2024-07-26T10:48:33.462 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/sm501.ko 2024-07-26T10:48:33.462 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/viperboard.ko 2024-07-26T10:48:33.462 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/vx855.ko 2024-07-26T10:48:33.462 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/mfd/wl1273-core.ko 2024-07-26T10:48:33.462 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc 2024-07-26T10:48:33.462 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl 2024-07-26T10:48:33.462 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-07-26T10:48:33.462 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds9802als.ko 2024-07-26T10:48:33.463 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/apds990x.ko 2024-07-26T10:48:33.463 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/bh1770glc.ko 2024-07-26T10:48:33.463 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710 2024-07-26T10:48:33.463 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/cb710/cb710.ko 2024-07-26T10:48:33.463 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom 2024-07-26T10:48:33.463 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/at24.ko 2024-07-26T10:48:33.463 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-07-26T10:48:33.463 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/eeprom/max6875.ko 2024-07-26T10:48:33.463 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/enclosure.ko 2024-07-26T10:48:33.463 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/hpilo.ko 2024-07-26T10:48:33.464 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29003.ko 2024-07-26T10:48:33.464 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/isl29020.ko 2024-07-26T10:48:33.464 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d 2024-07-26T10:48:33.464 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-07-26T10:48:33.464 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-07-26T10:48:33.464 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei 2024-07-26T10:48:33.464 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-me.ko 2024-07-26T10:48:33.464 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei-txe.ko 2024-07-26T10:48:33.464 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/mei/mei.ko 2024-07-26T10:48:33.464 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru 2024-07-26T10:48:33.464 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-gru/gru.ko 2024-07-26T10:48:33.464 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp 2024-07-26T10:48:33.465 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xp.ko 2024-07-26T10:48:33.465 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpc.ko 2024-07-26T10:48:33.465 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-07-26T10:48:33.465 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_7xx1.ko 2024-07-26T10:48:33.465 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tifm_core.ko 2024-07-26T10:48:33.465 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/tsl2550.ko 2024-07-26T10:48:33.465 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_balloon.ko 2024-07-26T10:48:33.465 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci 2024-07-26T10:48:33.465 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-07-26T10:48:33.465 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net 2024-07-26T10:48:33.465 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding 2024-07-26T10:48:33.465 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/bonding/bonding.ko 2024-07-26T10:48:33.465 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa 2024-07-26T10:48:33.465 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53 2024-07-26T10:48:33.466 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/b53/b53_common.ko 2024-07-26T10:48:33.466 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/bcm-sf2.ko 2024-07-26T10:48:33.466 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6060.ko 2024-07-26T10:48:33.466 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx 2024-07-26T10:48:33.466 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-07-26T10:48:33.466 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/dummy.ko 2024-07-26T10:48:33.466 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/eql.ko 2024-07-26T10:48:33.466 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet 2024-07-26T10:48:33.466 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com 2024-07-26T10:48:33.466 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-07-26T10:48:33.466 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-07-26T10:48:33.466 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-07-26T10:48:33.467 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-07-26T10:48:33.467 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390 2024-07-26T10:48:33.467 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/8390.ko 2024-07-26T10:48:33.467 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-07-26T10:48:33.467 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-07-26T10:48:33.467 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-07-26T10:48:33.467 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec 2024-07-26T10:48:33.467 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-07-26T10:48:33.467 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere 2024-07-26T10:48:33.467 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/agere/et131x.ko 2024-07-26T10:48:33.467 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon 2024-07-26T10:48:33.467 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-07-26T10:48:33.467 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera 2024-07-26T10:48:33.467 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-07-26T10:48:33.468 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd 2024-07-26T10:48:33.468 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-07-26T10:48:33.468 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-07-26T10:48:33.468 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-07-26T10:48:33.468 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros 2024-07-26T10:48:33.468 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx 2024-07-26T10:48:33.468 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-07-26T10:48:33.468 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c 2024-07-26T10:48:33.468 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-07-26T10:48:33.468 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e 2024-07-26T10:48:33.468 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-07-26T10:48:33.468 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx 2024-07-26T10:48:33.468 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-07-26T10:48:33.468 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-07-26T10:48:33.468 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom 2024-07-26T10:48:33.469 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-07-26T10:48:33.469 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-07-26T10:48:33.469 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-07-26T10:48:33.469 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-07-26T10:48:33.469 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-07-26T10:48:33.469 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet 2024-07-26T10:48:33.469 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-07-26T10:48:33.469 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-07-26T10:48:33.469 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade 2024-07-26T10:48:33.469 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna 2024-07-26T10:48:33.469 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-07-26T10:48:33.469 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio 2024-07-26T10:48:33.469 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb 2024-07-26T10:48:33.469 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-07-26T10:48:33.470 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-07-26T10:48:33.470 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-07-26T10:48:33.470 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-07-26T10:48:33.470 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-07-26T10:48:33.470 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-07-26T10:48:33.470 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-07-26T10:48:33.470 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-07-26T10:48:33.470 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-07-26T10:48:33.470 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco 2024-07-26T10:48:33.470 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic 2024-07-26T10:48:33.470 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-07-26T10:48:33.470 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec 2024-07-26T10:48:33.471 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip 2024-07-26T10:48:33.471 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-07-26T10:48:33.471 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-07-26T10:48:33.471 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-07-26T10:48:33.471 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-07-26T10:48:33.471 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-07-26T10:48:33.471 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-07-26T10:48:33.471 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink 2024-07-26T10:48:33.471 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-07-26T10:48:33.471 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-07-26T10:48:33.471 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/dnet.ko 2024-07-26T10:48:33.471 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex 2024-07-26T10:48:33.471 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet 2024-07-26T10:48:33.472 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-07-26T10:48:33.472 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ethoc.ko 2024-07-26T10:48:33.472 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/fealnx.ko 2024-07-26T10:48:33.472 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel 2024-07-26T10:48:33.472 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e100.ko 2024-07-26T10:48:33.472 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000 2024-07-26T10:48:33.472 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-07-26T10:48:33.472 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e 2024-07-26T10:48:33.472 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-07-26T10:48:33.472 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k 2024-07-26T10:48:33.472 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-07-26T10:48:33.472 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e 2024-07-26T10:48:33.472 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-07-26T10:48:33.472 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf 2024-07-26T10:48:33.473 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-07-26T10:48:33.473 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb 2024-07-26T10:48:33.473 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-07-26T10:48:33.473 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf 2024-07-26T10:48:33.473 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-07-26T10:48:33.473 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe 2024-07-26T10:48:33.473 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-07-26T10:48:33.473 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf 2024-07-26T10:48:33.473 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-07-26T10:48:33.473 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth 2024-07-26T10:48:33.473 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-07-26T10:48:33.473 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie 2024-07-26T10:48:33.473 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-07-26T10:48:33.474 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/jme.ko 2024-07-26T10:48:33.474 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell 2024-07-26T10:48:33.474 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-07-26T10:48:33.474 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/skge.ko 2024-07-26T10:48:33.474 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-07-26T10:48:33.474 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox 2024-07-26T10:48:33.474 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4 2024-07-26T10:48:33.474 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-07-26T10:48:33.474 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-07-26T10:48:33.474 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5 2024-07-26T10:48:33.474 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-07-26T10:48:33.474 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-07-26T10:48:33.474 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel 2024-07-26T10:48:33.475 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-07-26T10:48:33.475 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom 2024-07-26T10:48:33.475 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge 2024-07-26T10:48:33.475 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-07-26T10:48:33.475 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi 2024-07-26T10:48:33.475 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-07-26T10:48:33.475 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-07-26T10:48:33.475 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion 2024-07-26T10:48:33.475 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-07-26T10:48:33.475 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia 2024-07-26T10:48:33.475 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-07-26T10:48:33.475 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines 2024-07-26T10:48:33.475 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-07-26T10:48:33.476 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-07-26T10:48:33.476 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic 2024-07-26T10:48:33.476 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen 2024-07-26T10:48:33.476 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-07-26T10:48:33.476 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-07-26T10:48:33.476 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-07-26T10:48:33.476 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-07-26T10:48:33.476 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc 2024-07-26T10:48:33.476 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-07-26T10:48:33.476 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek 2024-07-26T10:48:33.476 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-07-26T10:48:33.476 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-07-26T10:48:33.476 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/atp.ko 2024-07-26T10:48:33.477 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-07-26T10:48:33.477 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker 2024-07-26T10:48:33.477 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-07-26T10:48:33.477 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc 2024-07-26T10:48:33.477 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-07-26T10:48:33.477 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan 2024-07-26T10:48:33.477 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-07-26T10:48:33.477 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis 2024-07-26T10:48:33.477 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis190.ko 2024-07-26T10:48:33.477 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sis/sis900.ko 2024-07-26T10:48:33.477 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc 2024-07-26T10:48:33.477 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-07-26T10:48:33.477 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-07-26T10:48:33.477 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-07-26T10:48:33.478 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-07-26T10:48:33.478 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro 2024-07-26T10:48:33.478 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac 2024-07-26T10:48:33.478 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-07-26T10:48:33.478 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-07-26T10:48:33.478 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun 2024-07-26T10:48:33.478 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/cassini.ko 2024-07-26T10:48:33.478 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/niu.ko 2024-07-26T10:48:33.478 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sungem.ko 2024-07-26T10:48:33.478 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-07-26T10:48:33.478 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti 2024-07-26T10:48:33.478 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-07-26T10:48:33.478 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti 2024-07-26T10:48:33.479 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/ti/tlan.ko 2024-07-26T10:48:33.479 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via 2024-07-26T10:48:33.479 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-07-26T10:48:33.479 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-07-26T10:48:33.479 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet 2024-07-26T10:48:33.479 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-07-26T10:48:33.479 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-07-26T10:48:33.479 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom 2024-07-26T10:48:33.479 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-07-26T10:48:33.479 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/geneve.ko 2024-07-26T10:48:33.479 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv 2024-07-26T10:48:33.479 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-07-26T10:48:33.479 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154 2024-07-26T10:48:33.480 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ieee802154/fakelb.ko 2024-07-26T10:48:33.480 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ifb.ko 2024-07-26T10:48:33.480 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan 2024-07-26T10:48:33.480 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ipvlan/ipvlan.ko 2024-07-26T10:48:33.480 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvlan.ko 2024-07-26T10:48:33.480 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/macvtap.ko 2024-07-26T10:48:33.480 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio 2024-07-26T10:48:33.480 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio.ko 2024-07-26T10:48:33.480 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-07-26T10:48:33.480 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-07-26T10:48:33.480 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mdio/mdio-regmap.ko 2024-07-26T10:48:33.480 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/mii.ko 2024-07-26T10:48:33.480 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/net_failover.ko 2024-07-26T10:48:33.480 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/netconsole.ko 2024-07-26T10:48:33.481 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/nlmon.ko 2024-07-26T10:48:33.481 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs 2024-07-26T10:48:33.481 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs-lynx.ko 2024-07-26T10:48:33.481 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-07-26T10:48:33.481 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy 2024-07-26T10:48:33.481 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/amd.ko 2024-07-26T10:48:33.481 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-07-26T10:48:33.481 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-07-26T10:48:33.481 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm7xxx.ko 2024-07-26T10:48:33.481 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/bcm87xx.ko 2024-07-26T10:48:33.481 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/broadcom.ko 2024-07-26T10:48:33.481 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/cicada.ko 2024-07-26T10:48:33.481 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/davicom.ko 2024-07-26T10:48:33.482 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/dp83640.ko 2024-07-26T10:48:33.482 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/et1011c.ko 2024-07-26T10:48:33.482 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/icplus.ko 2024-07-26T10:48:33.482 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/lxt.ko 2024-07-26T10:48:33.482 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/marvell.ko 2024-07-26T10:48:33.482 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/micrel.ko 2024-07-26T10:48:33.482 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/national.ko 2024-07-26T10:48:33.482 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/phylink.ko 2024-07-26T10:48:33.482 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/qsemi.ko 2024-07-26T10:48:33.482 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/realtek.ko 2024-07-26T10:48:33.482 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/smsc.ko 2024-07-26T10:48:33.482 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/ste10Xp.ko 2024-07-26T10:48:33.482 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/phy/vitesse.ko 2024-07-26T10:48:33.483 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp 2024-07-26T10:48:33.483 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/bsd_comp.ko 2024-07-26T10:48:33.483 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_async.ko 2024-07-26T10:48:33.483 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_deflate.ko 2024-07-26T10:48:33.483 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_generic.ko 2024-07-26T10:48:33.483 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_mppe.ko 2024-07-26T10:48:33.483 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/ppp_synctty.ko 2024-07-26T10:48:33.483 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppoe.ko 2024-07-26T10:48:33.483 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pppox.ko 2024-07-26T10:48:33.483 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/ppp/pptp.ko 2024-07-26T10:48:33.483 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip 2024-07-26T10:48:33.483 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slhc.ko 2024-07-26T10:48:33.483 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/slip/slip.ko 2024-07-26T10:48:33.483 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/sungem_phy.ko 2024-07-26T10:48:33.484 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tap.ko 2024-07-26T10:48:33.484 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team 2024-07-26T10:48:33.484 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team.ko 2024-07-26T10:48:33.484 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_activebackup.ko 2024-07-26T10:48:33.484 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_broadcast.ko 2024-07-26T10:48:33.484 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-07-26T10:48:33.484 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_random.ko 2024-07-26T10:48:33.484 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-07-26T10:48:33.484 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/tun.ko 2024-07-26T10:48:33.484 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/veth.ko 2024-07-26T10:48:33.484 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/virtio_net.ko 2024-07-26T10:48:33.484 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3 2024-07-26T10:48:33.484 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-07-26T10:48:33.485 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan 2024-07-26T10:48:33.485 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/vxlan/vxlan.ko 2024-07-26T10:48:33.485 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback 2024-07-26T10:48:33.485 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netback/xen-netback.ko 2024-07-26T10:48:33.485 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/net/xen-netfront.ko 2024-07-26T10:48:33.485 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb 2024-07-26T10:48:33.485 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ntb/ntb.ko 2024-07-26T10:48:33.485 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme 2024-07-26T10:48:33.485 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host 2024-07-26T10:48:33.485 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme-core.ko 2024-07-26T10:48:33.485 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/nvme/host/nvme.ko 2024-07-26T10:48:33.485 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport 2024-07-26T10:48:33.485 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport.ko 2024-07-26T10:48:33.485 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_cs.ko 2024-07-26T10:48:33.486 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_pc.ko 2024-07-26T10:48:33.486 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/parport/parport_serial.ko 2024-07-26T10:48:33.486 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci 2024-07-26T10:48:33.486 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug 2024-07-26T10:48:33.486 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-07-26T10:48:33.486 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie 2024-07-26T10:48:33.486 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/pcie/aer_inject.ko 2024-07-26T10:48:33.486 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pci/xen-pcifront.ko 2024-07-26T10:48:33.486 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia 2024-07-26T10:48:33.486 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/i82092.ko 2024-07-26T10:48:33.486 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/pd6729.ko 2024-07-26T10:48:33.486 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pcmcia/yenta_socket.ko 2024-07-26T10:48:33.487 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl 2024-07-26T10:48:33.487 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel 2024-07-26T10:48:33.487 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-07-26T10:48:33.487 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform 2024-07-26T10:48:33.487 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86 2024-07-26T10:48:33.487 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acer-wmi.ko 2024-07-26T10:48:33.487 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/acerhdf.ko 2024-07-26T10:48:33.487 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/apple-gmux.ko 2024-07-26T10:48:33.487 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-laptop.ko 2024-07-26T10:48:33.487 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-07-26T10:48:33.487 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/asus-wmi.ko 2024-07-26T10:48:33.487 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/classmate-laptop.ko 2024-07-26T10:48:33.487 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-07-26T10:48:33.487 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-07-26T10:48:33.487 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-07-26T10:48:33.488 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-07-26T10:48:33.488 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/hdaps.ko 2024-07-26T10:48:33.488 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel 2024-07-26T10:48:33.488 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-07-26T10:48:33.488 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/intel_ips.ko 2024-07-26T10:48:33.488 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/msi-wmi.ko 2024-07-26T10:48:33.488 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/mxm-wmi.ko 2024-07-26T10:48:33.488 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-07-26T10:48:33.488 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-laptop.ko 2024-07-26T10:48:33.488 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/samsung-q10.ko 2024-07-26T10:48:33.488 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/topstar-laptop.ko 2024-07-26T10:48:33.488 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-07-26T10:48:33.488 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/toshiba_haps.ko 2024-07-26T10:48:33.489 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi-bmof.ko 2024-07-26T10:48:33.489 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/platform/x86/wmi.ko 2024-07-26T10:48:33.489 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap 2024-07-26T10:48:33.489 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_common.ko 2024-07-26T10:48:33.489 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/powercap/intel_rapl_msr.ko 2024-07-26T10:48:33.489 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps 2024-07-26T10:48:33.489 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients 2024-07-26T10:48:33.489 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-gpio.ko 2024-07-26T10:48:33.489 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps-ldisc.ko 2024-07-26T10:48:33.489 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/clients/pps_parport.ko 2024-07-26T10:48:33.489 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/pps/pps_core.ko 2024-07-26T10:48:33.489 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp 2024-07-26T10:48:33.489 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp.ko 2024-07-26T10:48:33.489 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ptp/ptp_kvm.ko 2024-07-26T10:48:33.489 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras 2024-07-26T10:48:33.490 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd 2024-07-26T10:48:33.490 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl 2024-07-26T10:48:33.490 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-07-26T10:48:33.490 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ras/amd/fmpm.ko 2024-07-26T10:48:33.490 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc 2024-07-26T10:48:33.490 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-bq32k.ko 2024-07-26T10:48:33.490 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1286.ko 2024-07-26T10:48:33.490 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1307.ko 2024-07-26T10:48:33.490 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1374.ko 2024-07-26T10:48:33.490 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1511.ko 2024-07-26T10:48:33.490 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1553.ko 2024-07-26T10:48:33.490 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1672.ko 2024-07-26T10:48:33.490 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds1742.ko 2024-07-26T10:48:33.491 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds2404.ko 2024-07-26T10:48:33.491 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-ds3232.ko 2024-07-26T10:48:33.491 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-em3027.ko 2024-07-26T10:48:33.491 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-fm3130.ko 2024-07-26T10:48:33.491 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl12022.ko 2024-07-26T10:48:33.491 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-isl1208.ko 2024-07-26T10:48:33.491 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m41t80.ko 2024-07-26T10:48:33.491 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t35.ko 2024-07-26T10:48:33.491 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-m48t59.ko 2024-07-26T10:48:33.491 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-max6900.ko 2024-07-26T10:48:33.491 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-msm6242.ko 2024-07-26T10:48:33.491 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf2127.ko 2024-07-26T10:48:33.491 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf85063.ko 2024-07-26T10:48:33.491 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8523.ko 2024-07-26T10:48:33.492 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8563.ko 2024-07-26T10:48:33.492 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-pcf8583.ko 2024-07-26T10:48:33.492 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rp5c01.ko 2024-07-26T10:48:33.492 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rs5c372.ko 2024-07-26T10:48:33.492 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-07-26T10:48:33.492 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8025.ko 2024-07-26T10:48:33.492 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-rx8581.ko 2024-07-26T10:48:33.492 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-07-26T10:48:33.492 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/rtc/rtc-x1205.ko 2024-07-26T10:48:33.492 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi 2024-07-26T10:48:33.492 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-9xxx.ko 2024-07-26T10:48:33.492 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-sas.ko 2024-07-26T10:48:33.492 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/3w-xxxx.ko 2024-07-26T10:48:33.492 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/BusLogic.ko 2024-07-26T10:48:33.493 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/a100u2w.ko 2024-07-26T10:48:33.493 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid 2024-07-26T10:48:33.493 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aacraid/aacraid.ko 2024-07-26T10:48:33.493 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/advansys.ko 2024-07-26T10:48:33.493 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx 2024-07-26T10:48:33.493 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-07-26T10:48:33.493 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-07-26T10:48:33.493 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/am53c974.ko 2024-07-26T10:48:33.493 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr 2024-07-26T10:48:33.493 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-07-26T10:48:33.493 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/atp870u.ko 2024-07-26T10:48:33.493 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi 2024-07-26T10:48:33.493 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-07-26T10:48:33.494 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa 2024-07-26T10:48:33.494 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bfa/bfa.ko 2024-07-26T10:48:33.494 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc 2024-07-26T10:48:33.494 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-07-26T10:48:33.494 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i 2024-07-26T10:48:33.494 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-07-26T10:48:33.494 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ch.ko 2024-07-26T10:48:33.494 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor 2024-07-26T10:48:33.494 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/csiostor/csiostor.ko 2024-07-26T10:48:33.494 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi 2024-07-26T10:48:33.494 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i 2024-07-26T10:48:33.494 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-07-26T10:48:33.494 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i 2024-07-26T10:48:33.495 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-07-26T10:48:33.495 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-07-26T10:48:33.495 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dc395x.ko 2024-07-26T10:48:33.495 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler 2024-07-26T10:48:33.495 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-07-26T10:48:33.495 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-07-26T10:48:33.495 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-07-26T10:48:33.495 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-07-26T10:48:33.495 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/dmx3191d.ko 2024-07-26T10:48:33.495 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r 2024-07-26T10:48:33.495 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esas2r/esas2r.ko 2024-07-26T10:48:33.495 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/esp_scsi.ko 2024-07-26T10:48:33.495 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe 2024-07-26T10:48:33.495 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/fcoe.ko 2024-07-26T10:48:33.496 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-07-26T10:48:33.496 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic 2024-07-26T10:48:33.496 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/fnic/fnic.ko 2024-07-26T10:48:33.496 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hpsa.ko 2024-07-26T10:48:33.496 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hptiop.ko 2024-07-26T10:48:33.496 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/hv_storvsc.ko 2024-07-26T10:48:33.496 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/initio.ko 2024-07-26T10:48:33.496 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ipr.ko 2024-07-26T10:48:33.496 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ips.ko 2024-07-26T10:48:33.496 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci 2024-07-26T10:48:33.496 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/isci/isci.ko 2024-07-26T10:48:33.496 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-07-26T10:48:33.496 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/iscsi_tcp.ko 2024-07-26T10:48:33.496 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc 2024-07-26T10:48:33.497 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libfc/libfc.ko 2024-07-26T10:48:33.497 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi.ko 2024-07-26T10:48:33.497 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libiscsi_tcp.ko 2024-07-26T10:48:33.497 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas 2024-07-26T10:48:33.497 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/libsas/libsas.ko 2024-07-26T10:48:33.497 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc 2024-07-26T10:48:33.497 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/lpfc/lpfc.ko 2024-07-26T10:48:33.497 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid 2024-07-26T10:48:33.497 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid.ko 2024-07-26T10:48:33.497 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-07-26T10:48:33.497 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-07-26T10:48:33.497 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-07-26T10:48:33.497 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas 2024-07-26T10:48:33.498 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-07-26T10:48:33.498 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas 2024-07-26T10:48:33.498 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvsas/mvsas.ko 2024-07-26T10:48:33.498 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/mvumi.ko 2024-07-26T10:48:33.498 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001 2024-07-26T10:48:33.498 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-07-26T10:48:33.498 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/pmcraid.ko 2024-07-26T10:48:33.498 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla1280.ko 2024-07-26T10:48:33.498 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx 2024-07-26T10:48:33.498 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-07-26T10:48:33.498 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-07-26T10:48:33.498 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx 2024-07-26T10:48:33.499 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-07-26T10:48:33.499 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/raid_class.ko 2024-07-26T10:48:33.499 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_debug.ko 2024-07-26T10:48:33.499 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_fc.ko 2024-07-26T10:48:33.499 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-07-26T10:48:33.499 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_sas.ko 2024-07-26T10:48:33.499 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_spi.ko 2024-07-26T10:48:33.499 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/scsi_transport_srp.ko 2024-07-26T10:48:33.499 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/ses.ko 2024-07-26T10:48:33.499 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/st.ko 2024-07-26T10:48:33.499 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/stex.ko 2024-07-26T10:48:33.499 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2 2024-07-26T10:48:33.499 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-07-26T10:48:33.499 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/virtio_scsi.ko 2024-07-26T10:48:33.500 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/vmw_pvscsi.ko 2024-07-26T10:48:33.500 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/scsi/wd719x.ko 2024-07-26T10:48:33.500 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb 2024-07-26T10:48:33.500 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ssb/ssb.ko 2024-07-26T10:48:33.500 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target 2024-07-26T10:48:33.500 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi 2024-07-26T10:48:33.500 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-07-26T10:48:33.500 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback 2024-07-26T10:48:33.500 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/loopback/tcm_loop.ko 2024-07-26T10:48:33.500 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_file.ko 2024-07-26T10:48:33.500 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_iblock.ko 2024-07-26T10:48:33.500 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_mod.ko 2024-07-26T10:48:33.501 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_pscsi.ko 2024-07-26T10:48:33.501 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/target_core_user.ko 2024-07-26T10:48:33.501 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc 2024-07-26T10:48:33.501 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-07-26T10:48:33.501 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal 2024-07-26T10:48:33.501 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel 2024-07-26T10:48:33.501 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal 2024-07-26T10:48:33.501 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-07-26T10:48:33.501 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-07-26T10:48:33.501 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-07-26T10:48:33.501 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-07-26T10:48:33.501 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-07-26T10:48:33.501 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-07-26T10:48:33.501 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-07-26T10:48:33.502 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-07-26T10:48:33.502 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-07-26T10:48:33.502 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-07-26T10:48:33.502 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-07-26T10:48:33.502 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-07-26T10:48:33.502 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-07-26T10:48:33.502 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-07-26T10:48:33.502 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-07-26T10:48:33.502 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-07-26T10:48:33.502 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-07-26T10:48:33.502 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-07-26T10:48:33.502 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty 2024-07-26T10:48:33.503 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless 2024-07-26T10:48:33.503 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-07-26T10:48:33.503 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_gsm.ko 2024-07-26T10:48:33.503 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/n_hdlc.ko 2024-07-26T10:48:33.503 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/nozomi.ko 2024-07-26T10:48:33.503 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial 2024-07-26T10:48:33.503 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250 2024-07-26T10:48:33.503 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-07-26T10:48:33.503 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/arc_uart.ko 2024-07-26T10:48:33.503 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm 2024-07-26T10:48:33.503 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/tty/serial/jsm/jsm.ko 2024-07-26T10:48:33.503 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs 2024-07-26T10:48:33.503 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core 2024-07-26T10:48:33.503 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/core/ufshcd-core.ko 2024-07-26T10:48:33.504 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host 2024-07-26T10:48:33.504 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-07-26T10:48:33.504 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio 2024-07-26T10:48:33.504 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio.ko 2024-07-26T10:48:33.504 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_aec.ko 2024-07-26T10:48:33.504 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_cif.ko 2024-07-26T10:48:33.504 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_pci_generic.ko 2024-07-26T10:48:33.504 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/uio/uio_sercos3.ko 2024-07-26T10:48:33.504 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb 2024-07-26T10:48:33.504 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class 2024-07-26T10:48:33.504 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-acm.ko 2024-07-26T10:48:33.504 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/cdc-wdm.ko 2024-07-26T10:48:33.504 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usblp.ko 2024-07-26T10:48:33.504 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/class/usbtmc.ko 2024-07-26T10:48:33.505 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host 2024-07-26T10:48:33.505 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/host/sl811-hcd.ko 2024-07-26T10:48:33.505 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image 2024-07-26T10:48:33.505 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/mdc800.ko 2024-07-26T10:48:33.505 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/image/microtek.ko 2024-07-26T10:48:33.505 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc 2024-07-26T10:48:33.505 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/adutux.ko 2024-07-26T10:48:33.505 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/appledisplay.ko 2024-07-26T10:48:33.505 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi26.ko 2024-07-26T10:48:33.505 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/emi62.ko 2024-07-26T10:48:33.505 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ezusb.ko 2024-07-26T10:48:33.505 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/idmouse.ko 2024-07-26T10:48:33.505 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/iowarrior.ko 2024-07-26T10:48:33.506 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/isight_firmware.ko 2024-07-26T10:48:33.506 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/ldusb.ko 2024-07-26T10:48:33.506 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/legousbtower.ko 2024-07-26T10:48:33.506 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga 2024-07-26T10:48:33.506 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-07-26T10:48:33.506 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/trancevibrator.ko 2024-07-26T10:48:33.506 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usb3503.ko 2024-07-26T10:48:33.506 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usblcd.ko 2024-07-26T10:48:33.506 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/usbsevseg.ko 2024-07-26T10:48:33.506 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/uss720.ko 2024-07-26T10:48:33.506 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/misc/yurex.ko 2024-07-26T10:48:33.506 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy 2024-07-26T10:48:33.507 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/phy/phy-generic.ko 2024-07-26T10:48:33.507 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial 2024-07-26T10:48:33.507 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/aircable.ko 2024-07-26T10:48:33.507 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ark3116.ko 2024-07-26T10:48:33.507 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/belkin_sa.ko 2024-07-26T10:48:33.507 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ch341.ko 2024-07-26T10:48:33.507 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cp210x.ko 2024-07-26T10:48:33.507 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cyberjack.ko 2024-07-26T10:48:33.507 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/cypress_m8.ko 2024-07-26T10:48:33.507 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/digi_acceleport.ko 2024-07-26T10:48:33.507 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/empeg.ko 2024-07-26T10:48:33.507 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ftdi_sio.ko 2024-07-26T10:48:33.507 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/garmin_gps.ko 2024-07-26T10:48:33.507 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_edgeport.ko 2024-07-26T10:48:33.508 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/io_ti.ko 2024-07-26T10:48:33.508 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipaq.ko 2024-07-26T10:48:33.508 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ipw.ko 2024-07-26T10:48:33.508 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ir-usb.ko 2024-07-26T10:48:33.508 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-07-26T10:48:33.508 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan.ko 2024-07-26T10:48:33.508 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/keyspan_pda.ko 2024-07-26T10:48:33.508 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kl5kusb105.ko 2024-07-26T10:48:33.508 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/kobil_sct.ko 2024-07-26T10:48:33.508 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mct_u232.ko 2024-07-26T10:48:33.508 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7720.ko 2024-07-26T10:48:33.508 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/mos7840.ko 2024-07-26T10:48:33.508 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/navman.ko 2024-07-26T10:48:33.508 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/omninet.ko 2024-07-26T10:48:33.509 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/opticon.ko 2024-07-26T10:48:33.509 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/option.ko 2024-07-26T10:48:33.509 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/oti6858.ko 2024-07-26T10:48:33.509 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/pl2303.ko 2024-07-26T10:48:33.509 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcaux.ko 2024-07-26T10:48:33.509 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/qcserial.ko 2024-07-26T10:48:33.509 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/quatech2.ko 2024-07-26T10:48:33.509 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/safe_serial.ko 2024-07-26T10:48:33.509 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/sierra.ko 2024-07-26T10:48:33.509 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/spcp8x5.ko 2024-07-26T10:48:33.509 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ssu100.ko 2024-07-26T10:48:33.509 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/symbolserial.ko 2024-07-26T10:48:33.509 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-07-26T10:48:33.509 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-07-26T10:48:33.510 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_debug.ko 2024-07-26T10:48:33.510 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/usb_wwan.ko 2024-07-26T10:48:33.510 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/visor.ko 2024-07-26T10:48:33.510 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/whiteheat.ko 2024-07-26T10:48:33.510 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/serial/xsens_mt.ko 2024-07-26T10:48:33.510 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage 2024-07-26T10:48:33.510 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/uas.ko 2024-07-26T10:48:33.510 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-alauda.ko 2024-07-26T10:48:33.510 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-cypress.ko 2024-07-26T10:48:33.510 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-datafab.ko 2024-07-26T10:48:33.510 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-07-26T10:48:33.510 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-freecom.ko 2024-07-26T10:48:33.510 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-isd200.ko 2024-07-26T10:48:33.511 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-07-26T10:48:33.511 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-karma.ko 2024-07-26T10:48:33.511 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-onetouch.ko 2024-07-26T10:48:33.511 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-realtek.ko 2024-07-26T10:48:33.511 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr09.ko 2024-07-26T10:48:33.511 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-sddr55.ko 2024-07-26T10:48:33.511 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/ums-usbat.ko 2024-07-26T10:48:33.511 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/storage/usb-storage.ko 2024-07-26T10:48:33.511 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip 2024-07-26T10:48:33.511 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-core.ko 2024-07-26T10:48:33.511 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/usbip-host.ko 2024-07-26T10:48:33.511 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-07-26T10:48:33.511 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio 2024-07-26T10:48:33.512 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci 2024-07-26T10:48:33.512 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-07-26T10:48:33.512 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/pci/vfio-pci.ko 2024-07-26T10:48:33.512 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio.ko 2024-07-26T10:48:33.512 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-07-26T10:48:33.512 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost 2024-07-26T10:48:33.512 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost.ko 2024-07-26T10:48:33.512 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_iotlb.ko 2024-07-26T10:48:33.512 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_net.ko 2024-07-26T10:48:33.512 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/vhost/vhost_scsi.ko 2024-07-26T10:48:33.512 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video 2024-07-26T10:48:33.512 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight 2024-07-26T10:48:33.512 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/apple_bl.ko 2024-07-26T10:48:33.512 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/lcd.ko 2024-07-26T10:48:33.512 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/video/backlight/platform_lcd.ko 2024-07-26T10:48:33.513 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio 2024-07-26T10:48:33.513 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio.ko 2024-07-26T10:48:33.513 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_balloon.ko 2024-07-26T10:48:33.513 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_mmio.ko 2024-07-26T10:48:33.513 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci.ko 2024-07-26T10:48:33.513 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-07-26T10:48:33.513 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-07-26T10:48:33.513 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/virtio/virtio_ring.ko 2024-07-26T10:48:33.513 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1 2024-07-26T10:48:33.513 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters 2024-07-26T10:48:33.513 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2482.ko 2024-07-26T10:48:33.513 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/masters/ds2490.ko 2024-07-26T10:48:33.513 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves 2024-07-26T10:48:33.514 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-07-26T10:48:33.514 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-07-26T10:48:33.514 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-07-26T10:48:33.514 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-07-26T10:48:33.514 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-07-26T10:48:33.514 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-07-26T10:48:33.514 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-07-26T10:48:33.514 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-07-26T10:48:33.514 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-07-26T10:48:33.514 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_smem.ko 2024-07-26T10:48:33.514 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/slaves/w1_therm.ko 2024-07-26T10:48:33.514 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/w1/wire.ko 2024-07-26T10:48:33.515 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog 2024-07-26T10:48:33.515 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim1535_wdt.ko 2024-07-26T10:48:33.515 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/alim7101_wdt.ko 2024-07-26T10:48:33.515 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/f71808e_wdt.ko 2024-07-26T10:48:33.515 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/hpwdt.ko 2024-07-26T10:48:33.515 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/i6300esb.ko 2024-07-26T10:48:33.515 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-07-26T10:48:33.515 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/iTCO_wdt.ko 2024-07-26T10:48:33.515 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ib700wdt.ko 2024-07-26T10:48:33.515 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ibmasr.ko 2024-07-26T10:48:33.515 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-07-26T10:48:33.515 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it8712f_wdt.ko 2024-07-26T10:48:33.515 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/it87_wdt.ko 2024-07-26T10:48:33.515 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/machzwd.ko 2024-07-26T10:48:33.516 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/nv_tco.ko 2024-07-26T10:48:33.516 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_pci.ko 2024-07-26T10:48:33.516 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/pcwd_usb.ko 2024-07-26T10:48:33.516 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-07-26T10:48:33.516 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sch311x_wdt.ko 2024-07-26T10:48:33.516 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/softdog.ko 2024-07-26T10:48:33.516 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/sp5100_tco.ko 2024-07-26T10:48:33.516 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/via_wdt.ko 2024-07-26T10:48:33.516 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-07-26T10:48:33.516 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83877f_wdt.ko 2024-07-26T10:48:33.516 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/w83977f_wdt.ko 2024-07-26T10:48:33.516 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/wdt_pci.ko 2024-07-26T10:48:33.516 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/watchdog/xen_wdt.ko 2024-07-26T10:48:33.517 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen 2024-07-26T10:48:33.517 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-acpi-processor.ko 2024-07-26T10:48:33.517 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-evtchn.ko 2024-07-26T10:48:33.517 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntalloc.ko 2024-07-26T10:48:33.517 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-gntdev.ko 2024-07-26T10:48:33.517 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback 2024-07-26T10:48:33.517 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-07-26T10:48:33.517 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xen-privcmd.ko 2024-07-26T10:48:33.517 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs 2024-07-26T10:48:33.517 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/drivers/xen/xenfs/xenfs.ko 2024-07-26T10:48:33.517 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs 2024-07-26T10:48:33.517 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p 2024-07-26T10:48:33.517 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/9p/9p.ko 2024-07-26T10:48:33.517 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs 2024-07-26T10:48:33.518 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/affs/affs.ko 2024-07-26T10:48:33.518 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs 2024-07-26T10:48:33.518 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/befs/befs.ko 2024-07-26T10:48:33.518 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/binfmt_misc.ko 2024-07-26T10:48:33.518 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs 2024-07-26T10:48:33.518 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/btrfs/btrfs.ko 2024-07-26T10:48:33.518 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles 2024-07-26T10:48:33.518 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cachefiles/cachefiles.ko 2024-07-26T10:48:33.518 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph 2024-07-26T10:48:33.518 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ceph/ceph.ko 2024-07-26T10:48:33.518 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda 2024-07-26T10:48:33.518 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/coda/coda.ko 2024-07-26T10:48:33.518 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs 2024-07-26T10:48:33.519 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/cramfs/cramfs.ko 2024-07-26T10:48:33.519 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm 2024-07-26T10:48:33.519 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/dlm/dlm.ko 2024-07-26T10:48:33.519 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs 2024-07-26T10:48:33.519 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ecryptfs/ecryptfs.ko 2024-07-26T10:48:33.519 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs 2024-07-26T10:48:33.519 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/f2fs/f2fs.ko 2024-07-26T10:48:33.519 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat 2024-07-26T10:48:33.519 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/fat.ko 2024-07-26T10:48:33.519 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/msdos.ko 2024-07-26T10:48:33.519 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fat/vfat.ko 2024-07-26T10:48:33.519 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse 2024-07-26T10:48:33.519 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/cuse.ko 2024-07-26T10:48:33.519 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/fuse/fuse.ko 2024-07-26T10:48:33.520 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2 2024-07-26T10:48:33.520 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/gfs2/gfs2.ko 2024-07-26T10:48:33.520 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs 2024-07-26T10:48:33.520 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfs/hfs.ko 2024-07-26T10:48:33.520 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus 2024-07-26T10:48:33.520 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/hfsplus/hfsplus.ko 2024-07-26T10:48:33.520 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs 2024-07-26T10:48:33.520 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/isofs/isofs.ko 2024-07-26T10:48:33.520 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs 2024-07-26T10:48:33.520 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/jfs/jfs.ko 2024-07-26T10:48:33.520 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd 2024-07-26T10:48:33.520 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/lockd/lockd.ko 2024-07-26T10:48:33.520 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix 2024-07-26T10:48:33.521 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/minix/minix.ko 2024-07-26T10:48:33.521 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs 2024-07-26T10:48:33.521 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/netfs/netfs.ko 2024-07-26T10:48:33.521 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs 2024-07-26T10:48:33.521 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout 2024-07-26T10:48:33.521 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-07-26T10:48:33.521 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout 2024-07-26T10:48:33.521 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-07-26T10:48:33.521 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout 2024-07-26T10:48:33.521 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-07-26T10:48:33.521 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfs.ko 2024-07-26T10:48:33.521 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv3.ko 2024-07-26T10:48:33.521 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs/nfsv4.ko 2024-07-26T10:48:33.521 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common 2024-07-26T10:48:33.522 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/grace.ko 2024-07-26T10:48:33.522 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfs_common/nfs_acl.ko 2024-07-26T10:48:33.522 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd 2024-07-26T10:48:33.522 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nfsd/nfsd.ko 2024-07-26T10:48:33.522 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2 2024-07-26T10:48:33.522 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nilfs2/nilfs2.ko 2024-07-26T10:48:33.522 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls 2024-07-26T10:48:33.522 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-celtic.ko 2024-07-26T10:48:33.522 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-centeuro.ko 2024-07-26T10:48:33.522 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-croatian.ko 2024-07-26T10:48:33.522 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-cyrillic.ko 2024-07-26T10:48:33.522 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-gaelic.ko 2024-07-26T10:48:33.522 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-greek.ko 2024-07-26T10:48:33.523 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-iceland.ko 2024-07-26T10:48:33.523 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-inuit.ko 2024-07-26T10:48:33.523 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-roman.ko 2024-07-26T10:48:33.523 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-romanian.ko 2024-07-26T10:48:33.523 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/mac-turkish.ko 2024-07-26T10:48:33.523 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1250.ko 2024-07-26T10:48:33.523 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1251.ko 2024-07-26T10:48:33.523 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp1255.ko 2024-07-26T10:48:33.523 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp737.ko 2024-07-26T10:48:33.523 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp775.ko 2024-07-26T10:48:33.523 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp850.ko 2024-07-26T10:48:33.523 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp852.ko 2024-07-26T10:48:33.523 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp855.ko 2024-07-26T10:48:33.523 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp857.ko 2024-07-26T10:48:33.524 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp860.ko 2024-07-26T10:48:33.524 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp861.ko 2024-07-26T10:48:33.524 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp862.ko 2024-07-26T10:48:33.524 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp863.ko 2024-07-26T10:48:33.524 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp864.ko 2024-07-26T10:48:33.524 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp865.ko 2024-07-26T10:48:33.524 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp866.ko 2024-07-26T10:48:33.524 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp869.ko 2024-07-26T10:48:33.524 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp874.ko 2024-07-26T10:48:33.524 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp932.ko 2024-07-26T10:48:33.524 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp936.ko 2024-07-26T10:48:33.524 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp949.ko 2024-07-26T10:48:33.524 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_cp950.ko 2024-07-26T10:48:33.525 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_euc-jp.ko 2024-07-26T10:48:33.525 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-1.ko 2024-07-26T10:48:33.525 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-13.ko 2024-07-26T10:48:33.525 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-14.ko 2024-07-26T10:48:33.525 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-15.ko 2024-07-26T10:48:33.525 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-2.ko 2024-07-26T10:48:33.525 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-3.ko 2024-07-26T10:48:33.525 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-4.ko 2024-07-26T10:48:33.525 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-5.ko 2024-07-26T10:48:33.525 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-6.ko 2024-07-26T10:48:33.525 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-7.ko 2024-07-26T10:48:33.525 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_iso8859-9.ko 2024-07-26T10:48:33.525 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-r.ko 2024-07-26T10:48:33.526 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-ru.ko 2024-07-26T10:48:33.526 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_koi8-u.ko 2024-07-26T10:48:33.526 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_ucs2_utils.ko 2024-07-26T10:48:33.526 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/nls/nls_utf8.ko 2024-07-26T10:48:33.526 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2 2024-07-26T10:48:33.526 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster 2024-07-26T10:48:33.526 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-07-26T10:48:33.526 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm 2024-07-26T10:48:33.526 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-07-26T10:48:33.526 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs 2024-07-26T10:48:33.526 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-07-26T10:48:33.526 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2.ko 2024-07-26T10:48:33.527 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-07-26T10:48:33.527 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-07-26T10:48:33.527 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-07-26T10:48:33.527 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs 2024-07-26T10:48:33.527 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/overlayfs/overlay.ko 2024-07-26T10:48:33.527 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore 2024-07-26T10:48:33.527 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/pstore/ramoops.ko 2024-07-26T10:48:33.527 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs 2024-07-26T10:48:33.527 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/reiserfs/reiserfs.ko 2024-07-26T10:48:33.527 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs 2024-07-26T10:48:33.527 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/romfs/romfs.ko 2024-07-26T10:48:33.527 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb 2024-07-26T10:48:33.527 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client 2024-07-26T10:48:33.527 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/client/cifs.ko 2024-07-26T10:48:33.528 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common 2024-07-26T10:48:33.528 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_arc4.ko 2024-07-26T10:48:33.528 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/smb/common/cifs_md4.ko 2024-07-26T10:48:33.528 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs 2024-07-26T10:48:33.528 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/squashfs/squashfs.ko 2024-07-26T10:48:33.528 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv 2024-07-26T10:48:33.528 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/sysv/sysv.ko 2024-07-26T10:48:33.528 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf 2024-07-26T10:48:33.528 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/udf/udf.ko 2024-07-26T10:48:33.528 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs 2024-07-26T10:48:33.528 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/ufs/ufs.ko 2024-07-26T10:48:33.528 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs 2024-07-26T10:48:33.528 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/fs/xfs/xfs.ko 2024-07-26T10:48:33.528 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel 2024-07-26T10:48:33.529 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu 2024-07-26T10:48:33.529 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/rcu/rcutorture.ko 2024-07-26T10:48:33.529 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/torture.ko 2024-07-26T10:48:33.529 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace 2024-07-26T10:48:33.529 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-07-26T10:48:33.529 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib 2024-07-26T10:48:33.529 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/asn1_encoder.ko 2024-07-26T10:48:33.529 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc-itu-t.ko 2024-07-26T10:48:33.529 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crc8.ko 2024-07-26T10:48:33.529 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto 2024-07-26T10:48:33.529 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libaescfb.ko 2024-07-26T10:48:33.529 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libarc4.ko 2024-07-26T10:48:33.529 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/crypto/libdes.ko 2024-07-26T10:48:33.529 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim 2024-07-26T10:48:33.530 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/dim/dimlib.ko 2024-07-26T10:48:33.530 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/libcrc32c.ko 2024-07-26T10:48:33.530 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lru_cache.ko 2024-07-26T10:48:33.530 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4 2024-07-26T10:48:33.530 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4_compress.ko 2024-07-26T10:48:33.530 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/lz4/lz4hc_compress.ko 2024-07-26T10:48:33.530 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math 2024-07-26T10:48:33.530 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/math/cordic.ko 2024-07-26T10:48:33.530 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6 2024-07-26T10:48:33.530 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/raid6/raid6_pq.ko 2024-07-26T10:48:33.530 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon 2024-07-26T10:48:33.530 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/reed_solomon/reed_solomon.ko 2024-07-26T10:48:33.530 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_bm.ko 2024-07-26T10:48:33.531 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_fsm.ko 2024-07-26T10:48:33.531 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/ts_kmp.ko 2024-07-26T10:48:33.531 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd 2024-07-26T10:48:33.531 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/lib/zstd/zstd_compress.ko 2024-07-26T10:48:33.531 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm 2024-07-26T10:48:33.531 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/mm/hwpoison-inject.ko 2024-07-26T10:48:33.531 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net 2024-07-26T10:48:33.531 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan 2024-07-26T10:48:33.531 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/6lowpan.ko 2024-07-26T10:48:33.531 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_dest.ko 2024-07-26T10:48:33.531 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_fragment.ko 2024-07-26T10:48:33.531 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_hop.ko 2024-07-26T10:48:33.531 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_ipv6.ko 2024-07-26T10:48:33.531 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_mobility.ko 2024-07-26T10:48:33.532 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_routing.ko 2024-07-26T10:48:33.532 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/6lowpan/nhc_udp.ko 2024-07-26T10:48:33.532 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802 2024-07-26T10:48:33.532 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/garp.ko 2024-07-26T10:48:33.532 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/mrp.ko 2024-07-26T10:48:33.532 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/p8022.ko 2024-07-26T10:48:33.532 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/psnap.ko 2024-07-26T10:48:33.532 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/802/stp.ko 2024-07-26T10:48:33.532 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q 2024-07-26T10:48:33.532 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/8021q/8021q.ko 2024-07-26T10:48:33.532 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p 2024-07-26T10:48:33.532 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet.ko 2024-07-26T10:48:33.532 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_fd.ko 2024-07-26T10:48:33.533 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_rdma.ko 2024-07-26T10:48:33.533 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/9p/9pnet_virtio.ko 2024-07-26T10:48:33.533 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv 2024-07-26T10:48:33.533 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/batman-adv/batman-adv.ko 2024-07-26T10:48:33.533 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge 2024-07-26T10:48:33.533 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/br_netfilter.ko 2024-07-26T10:48:33.533 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/bridge.ko 2024-07-26T10:48:33.533 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter 2024-07-26T10:48:33.533 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-07-26T10:48:33.533 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_among.ko 2024-07-26T10:48:33.533 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arp.ko 2024-07-26T10:48:33.533 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-07-26T10:48:33.533 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-07-26T10:48:33.533 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip.ko 2024-07-26T10:48:33.533 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-07-26T10:48:33.534 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_limit.ko 2024-07-26T10:48:33.534 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_log.ko 2024-07-26T10:48:33.534 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark.ko 2024-07-26T10:48:33.534 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-07-26T10:48:33.534 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-07-26T10:48:33.534 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-07-26T10:48:33.534 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-07-26T10:48:33.534 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_snat.ko 2024-07-26T10:48:33.534 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_stp.ko 2024-07-26T10:48:33.534 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-07-26T10:48:33.534 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-07-26T10:48:33.534 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-07-26T10:48:33.534 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-07-26T10:48:33.535 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/ebtables.ko 2024-07-26T10:48:33.535 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-07-26T10:48:33.535 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-07-26T10:48:33.535 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-07-26T10:48:33.535 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph 2024-07-26T10:48:33.535 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ceph/libceph.ko 2024-07-26T10:48:33.535 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core 2024-07-26T10:48:33.535 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/failover.ko 2024-07-26T10:48:33.535 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/core/pktgen.ko 2024-07-26T10:48:33.535 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp 2024-07-26T10:48:33.535 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp.ko 2024-07-26T10:48:33.535 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_diag.ko 2024-07-26T10:48:33.535 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv4.ko 2024-07-26T10:48:33.535 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dccp/dccp_ipv6.ko 2024-07-26T10:48:33.536 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver 2024-07-26T10:48:33.536 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dns_resolver/dns_resolver.ko 2024-07-26T10:48:33.536 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa 2024-07-26T10:48:33.536 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/dsa_core.ko 2024-07-26T10:48:33.536 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_brcm.ko 2024-07-26T10:48:33.536 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_dsa.ko 2024-07-26T10:48:33.536 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_none.ko 2024-07-26T10:48:33.536 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/dsa/tag_trailer.ko 2024-07-26T10:48:33.536 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154 2024-07-26T10:48:33.536 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan 2024-07-26T10:48:33.536 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-07-26T10:48:33.536 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154.ko 2024-07-26T10:48:33.536 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ieee802154/ieee802154_socket.ko 2024-07-26T10:48:33.536 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4 2024-07-26T10:48:33.537 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ah4.ko 2024-07-26T10:48:33.537 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/esp4.ko 2024-07-26T10:48:33.537 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/fou.ko 2024-07-26T10:48:33.537 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/gre.ko 2024-07-26T10:48:33.537 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/inet_diag.ko 2024-07-26T10:48:33.537 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_gre.ko 2024-07-26T10:48:33.537 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_tunnel.ko 2024-07-26T10:48:33.537 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ip_vti.ko 2024-07-26T10:48:33.537 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipcomp.ko 2024-07-26T10:48:33.537 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/ipip.ko 2024-07-26T10:48:33.537 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter 2024-07-26T10:48:33.537 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arp_tables.ko 2024-07-26T10:48:33.537 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-07-26T10:48:33.537 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-07-26T10:48:33.537 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-07-26T10:48:33.538 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-07-26T10:48:33.538 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-07-26T10:48:33.538 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-07-26T10:48:33.538 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-07-26T10:48:33.538 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-07-26T10:48:33.538 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-07-26T10:48:33.538 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/iptable_security.ko 2024-07-26T10:48:33.538 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-07-26T10:48:33.538 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-07-26T10:48:33.538 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-07-26T10:48:33.538 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-07-26T10:48:33.538 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-07-26T10:48:33.538 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-07-26T10:48:33.539 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-07-26T10:48:33.539 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-07-26T10:48:33.539 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-07-26T10:48:33.539 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-07-26T10:48:33.539 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_bic.ko 2024-07-26T10:48:33.539 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_dctcp.ko 2024-07-26T10:48:33.539 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_diag.ko 2024-07-26T10:48:33.539 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_highspeed.ko 2024-07-26T10:48:33.539 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_htcp.ko 2024-07-26T10:48:33.539 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_hybla.ko 2024-07-26T10:48:33.539 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_illinois.ko 2024-07-26T10:48:33.539 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_lp.ko 2024-07-26T10:48:33.539 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_scalable.ko 2024-07-26T10:48:33.539 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_vegas.ko 2024-07-26T10:48:33.540 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_veno.ko 2024-07-26T10:48:33.540 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_westwood.ko 2024-07-26T10:48:33.540 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tcp_yeah.ko 2024-07-26T10:48:33.540 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/tunnel4.ko 2024-07-26T10:48:33.540 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_diag.ko 2024-07-26T10:48:33.540 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/udp_tunnel.ko 2024-07-26T10:48:33.540 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv4/xfrm4_tunnel.ko 2024-07-26T10:48:33.540 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6 2024-07-26T10:48:33.540 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ah6.ko 2024-07-26T10:48:33.540 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/esp6.ko 2024-07-26T10:48:33.540 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/fou6.ko 2024-07-26T10:48:33.540 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_tunnel.ko 2024-07-26T10:48:33.540 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-07-26T10:48:33.540 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ip6_vti.ko 2024-07-26T10:48:33.541 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/ipcomp6.ko 2024-07-26T10:48:33.541 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter 2024-07-26T10:48:33.541 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-07-26T10:48:33.541 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-07-26T10:48:33.541 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-07-26T10:48:33.541 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-07-26T10:48:33.541 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-07-26T10:48:33.541 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-07-26T10:48:33.541 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-07-26T10:48:33.541 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-07-26T10:48:33.541 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-07-26T10:48:33.541 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-07-26T10:48:33.541 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-07-26T10:48:33.541 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-07-26T10:48:33.542 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-07-26T10:48:33.542 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-07-26T10:48:33.542 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-07-26T10:48:33.542 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-07-26T10:48:33.542 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-07-26T10:48:33.542 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-07-26T10:48:33.542 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-07-26T10:48:33.542 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-07-26T10:48:33.542 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-07-26T10:48:33.542 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-07-26T10:48:33.542 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-07-26T10:48:33.542 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-07-26T10:48:33.542 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/sit.ko 2024-07-26T10:48:33.543 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/tunnel6.ko 2024-07-26T10:48:33.543 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/ipv6/xfrm6_tunnel.ko 2024-07-26T10:48:33.543 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key 2024-07-26T10:48:33.543 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/key/af_key.ko 2024-07-26T10:48:33.543 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp 2024-07-26T10:48:33.543 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_core.ko 2024-07-26T10:48:33.543 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_debugfs.ko 2024-07-26T10:48:33.543 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_eth.ko 2024-07-26T10:48:33.543 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip.ko 2024-07-26T10:48:33.543 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ip6.ko 2024-07-26T10:48:33.543 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_netlink.ko 2024-07-26T10:48:33.543 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/l2tp/l2tp_ppp.ko 2024-07-26T10:48:33.543 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc 2024-07-26T10:48:33.543 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/llc/llc.ko 2024-07-26T10:48:33.544 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154 2024-07-26T10:48:33.544 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mac802154/mac802154.ko 2024-07-26T10:48:33.544 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls 2024-07-26T10:48:33.544 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/mpls/mpls_gso.ko 2024-07-26T10:48:33.544 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter 2024-07-26T10:48:33.544 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset 2024-07-26T10:48:33.544 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set.ko 2024-07-26T10:48:33.544 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-07-26T10:48:33.544 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-07-26T10:48:33.544 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-07-26T10:48:33.544 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-07-26T10:48:33.544 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-07-26T10:48:33.544 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-07-26T10:48:33.545 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-07-26T10:48:33.545 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-07-26T10:48:33.545 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-07-26T10:48:33.545 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-07-26T10:48:33.545 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-07-26T10:48:33.545 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-07-26T10:48:33.545 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-07-26T10:48:33.545 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-07-26T10:48:33.545 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-07-26T10:48:33.545 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs 2024-07-26T10:48:33.545 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs.ko 2024-07-26T10:48:33.545 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-07-26T10:48:33.545 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-07-26T10:48:33.546 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-07-26T10:48:33.546 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-07-26T10:48:33.546 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-07-26T10:48:33.546 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-07-26T10:48:33.546 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-07-26T10:48:33.546 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-07-26T10:48:33.546 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-07-26T10:48:33.546 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-07-26T10:48:33.546 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-07-26T10:48:33.546 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-07-26T10:48:33.546 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-07-26T10:48:33.546 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conncount.ko 2024-07-26T10:48:33.546 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack.ko 2024-07-26T10:48:33.547 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-07-26T10:48:33.547 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-07-26T10:48:33.547 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-07-26T10:48:33.547 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_h323.ko 2024-07-26T10:48:33.547 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_irc.ko 2024-07-26T10:48:33.547 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-07-26T10:48:33.547 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-07-26T10:48:33.547 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-07-26T10:48:33.547 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sane.ko 2024-07-26T10:48:33.547 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_sip.ko 2024-07-26T10:48:33.547 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-07-26T10:48:33.547 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-07-26T10:48:33.547 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_dup_netdev.ko 2024-07-26T10:48:33.548 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table.ko 2024-07-26T10:48:33.548 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_flow_table_inet.ko 2024-07-26T10:48:33.548 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_log_syslog.ko 2024-07-26T10:48:33.548 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat.ko 2024-07-26T10:48:33.548 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_amanda.ko 2024-07-26T10:48:33.548 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_ftp.ko 2024-07-26T10:48:33.548 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_irc.ko 2024-07-26T10:48:33.548 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_sip.ko 2024-07-26T10:48:33.548 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_nat_tftp.ko 2024-07-26T10:48:33.548 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_synproxy_core.ko 2024-07-26T10:48:33.548 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nf_tables.ko 2024-07-26T10:48:33.548 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink.ko 2024-07-26T10:48:33.548 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_acct.ko 2024-07-26T10:48:33.548 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_log.ko 2024-07-26T10:48:33.549 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_osf.ko 2024-07-26T10:48:33.549 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nfnetlink_queue.ko 2024-07-26T10:48:33.549 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_chain_nat.ko 2024-07-26T10:48:33.549 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_compat.ko 2024-07-26T10:48:33.549 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_ct.ko 2024-07-26T10:48:33.549 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_dup_netdev.ko 2024-07-26T10:48:33.549 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fib.ko 2024-07-26T10:48:33.549 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_fwd_netdev.ko 2024-07-26T10:48:33.549 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_hash.ko 2024-07-26T10:48:33.549 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_limit.ko 2024-07-26T10:48:33.549 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_log.ko 2024-07-26T10:48:33.549 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_masq.ko 2024-07-26T10:48:33.549 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_nat.ko 2024-07-26T10:48:33.550 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_numgen.ko 2024-07-26T10:48:33.550 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_queue.ko 2024-07-26T10:48:33.550 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_quota.ko 2024-07-26T10:48:33.550 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_redir.ko 2024-07-26T10:48:33.550 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject.ko 2024-07-26T10:48:33.550 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_reject_inet.ko 2024-07-26T10:48:33.550 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_socket.ko 2024-07-26T10:48:33.550 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/nft_tunnel.ko 2024-07-26T10:48:33.550 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_AUDIT.ko 2024-07-26T10:48:33.550 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CHECKSUM.ko 2024-07-26T10:48:33.550 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CLASSIFY.ko 2024-07-26T10:48:33.550 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-07-26T10:48:33.550 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_CT.ko 2024-07-26T10:48:33.551 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_DSCP.ko 2024-07-26T10:48:33.551 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HL.ko 2024-07-26T10:48:33.551 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_HMARK.ko 2024-07-26T10:48:33.551 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_IDLETIMER.ko 2024-07-26T10:48:33.551 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LED.ko 2024-07-26T10:48:33.551 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_LOG.ko 2024-07-26T10:48:33.551 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_MASQUERADE.ko 2024-07-26T10:48:33.551 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NETMAP.ko 2024-07-26T10:48:33.551 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFLOG.ko 2024-07-26T10:48:33.551 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_NFQUEUE.ko 2024-07-26T10:48:33.551 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_RATEEST.ko 2024-07-26T10:48:33.551 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_REDIRECT.ko 2024-07-26T10:48:33.551 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_SECMARK.ko 2024-07-26T10:48:33.552 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPMSS.ko 2024-07-26T10:48:33.552 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-07-26T10:48:33.552 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TEE.ko 2024-07-26T10:48:33.552 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TPROXY.ko 2024-07-26T10:48:33.552 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_TRACE.ko 2024-07-26T10:48:33.552 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_addrtype.ko 2024-07-26T10:48:33.552 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_bpf.ko 2024-07-26T10:48:33.552 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cgroup.ko 2024-07-26T10:48:33.552 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cluster.ko 2024-07-26T10:48:33.552 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_comment.ko 2024-07-26T10:48:33.552 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connbytes.ko 2024-07-26T10:48:33.552 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlabel.ko 2024-07-26T10:48:33.552 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connlimit.ko 2024-07-26T10:48:33.553 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_connmark.ko 2024-07-26T10:48:33.553 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_conntrack.ko 2024-07-26T10:48:33.553 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_cpu.ko 2024-07-26T10:48:33.553 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dccp.ko 2024-07-26T10:48:33.553 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_devgroup.ko 2024-07-26T10:48:33.553 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_dscp.ko 2024-07-26T10:48:33.553 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ecn.ko 2024-07-26T10:48:33.553 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_esp.ko 2024-07-26T10:48:33.553 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hashlimit.ko 2024-07-26T10:48:33.553 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_helper.ko 2024-07-26T10:48:33.553 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_hl.ko 2024-07-26T10:48:33.553 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipcomp.ko 2024-07-26T10:48:33.553 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_iprange.ko 2024-07-26T10:48:33.553 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_ipvs.ko 2024-07-26T10:48:33.554 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_l2tp.ko 2024-07-26T10:48:33.554 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_length.ko 2024-07-26T10:48:33.554 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_limit.ko 2024-07-26T10:48:33.554 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mac.ko 2024-07-26T10:48:33.554 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_mark.ko 2024-07-26T10:48:33.554 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_multiport.ko 2024-07-26T10:48:33.554 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nat.ko 2024-07-26T10:48:33.554 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_nfacct.ko 2024-07-26T10:48:33.554 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_osf.ko 2024-07-26T10:48:33.554 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_owner.ko 2024-07-26T10:48:33.554 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_physdev.ko 2024-07-26T10:48:33.554 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_pkttype.ko 2024-07-26T10:48:33.554 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_policy.ko 2024-07-26T10:48:33.555 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_quota.ko 2024-07-26T10:48:33.555 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_rateest.ko 2024-07-26T10:48:33.555 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_realm.ko 2024-07-26T10:48:33.555 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_recent.ko 2024-07-26T10:48:33.555 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_sctp.ko 2024-07-26T10:48:33.555 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_set.ko 2024-07-26T10:48:33.555 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_socket.ko 2024-07-26T10:48:33.555 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_state.ko 2024-07-26T10:48:33.555 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_statistic.ko 2024-07-26T10:48:33.555 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_string.ko 2024-07-26T10:48:33.555 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_tcpmss.ko 2024-07-26T10:48:33.555 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_time.ko 2024-07-26T10:48:33.555 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netfilter/xt_u32.ko 2024-07-26T10:48:33.555 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink 2024-07-26T10:48:33.556 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/netlink/netlink_diag.ko 2024-07-26T10:48:33.556 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh 2024-07-26T10:48:33.556 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/nsh/nsh.ko 2024-07-26T10:48:33.556 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch 2024-07-26T10:48:33.556 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/openvswitch.ko 2024-07-26T10:48:33.556 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-geneve.ko 2024-07-26T10:48:33.556 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-gre.ko 2024-07-26T10:48:33.556 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/openvswitch/vport-vxlan.ko 2024-07-26T10:48:33.556 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet 2024-07-26T10:48:33.556 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/packet/af_packet_diag.ko 2024-07-26T10:48:33.556 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds 2024-07-26T10:48:33.556 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds.ko 2024-07-26T10:48:33.556 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_rdma.ko 2024-07-26T10:48:33.557 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/rds/rds_tcp.ko 2024-07-26T10:48:33.557 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched 2024-07-26T10:48:33.557 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_csum.ko 2024-07-26T10:48:33.557 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_gact.ko 2024-07-26T10:48:33.557 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_mirred.ko 2024-07-26T10:48:33.557 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_nat.ko 2024-07-26T10:48:33.557 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_pedit.ko 2024-07-26T10:48:33.557 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_police.ko 2024-07-26T10:48:33.557 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_simple.ko 2024-07-26T10:48:33.557 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_skbedit.ko 2024-07-26T10:48:33.557 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/act_vlan.ko 2024-07-26T10:48:33.557 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_basic.ko 2024-07-26T10:48:33.557 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_bpf.ko 2024-07-26T10:48:33.557 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_flow.ko 2024-07-26T10:48:33.558 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_fw.ko 2024-07-26T10:48:33.558 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_route.ko 2024-07-26T10:48:33.558 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/cls_u32.ko 2024-07-26T10:48:33.558 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_cmp.ko 2024-07-26T10:48:33.558 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_ipset.ko 2024-07-26T10:48:33.558 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_meta.ko 2024-07-26T10:48:33.558 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_nbyte.ko 2024-07-26T10:48:33.558 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_text.ko 2024-07-26T10:48:33.558 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/em_u32.ko 2024-07-26T10:48:33.558 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_choke.ko 2024-07-26T10:48:33.558 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_codel.ko 2024-07-26T10:48:33.558 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_drr.ko 2024-07-26T10:48:33.559 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_fq.ko 2024-07-26T10:48:33.559 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_gred.ko 2024-07-26T10:48:33.559 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hfsc.ko 2024-07-26T10:48:33.559 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_hhf.ko 2024-07-26T10:48:33.559 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_htb.ko 2024-07-26T10:48:33.559 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_ingress.ko 2024-07-26T10:48:33.559 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio.ko 2024-07-26T10:48:33.559 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_mqprio_lib.ko 2024-07-26T10:48:33.559 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_multiq.ko 2024-07-26T10:48:33.559 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_netem.ko 2024-07-26T10:48:33.559 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_pie.ko 2024-07-26T10:48:33.559 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_plug.ko 2024-07-26T10:48:33.559 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_prio.ko 2024-07-26T10:48:33.559 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_qfq.ko 2024-07-26T10:48:33.559 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_red.ko 2024-07-26T10:48:33.560 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfb.ko 2024-07-26T10:48:33.560 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_sfq.ko 2024-07-26T10:48:33.560 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_tbf.ko 2024-07-26T10:48:33.560 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sched/sch_teql.ko 2024-07-26T10:48:33.560 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp 2024-07-26T10:48:33.560 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp.ko 2024-07-26T10:48:33.560 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sctp/sctp_diag.ko 2024-07-26T10:48:33.560 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc 2024-07-26T10:48:33.560 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss 2024-07-26T10:48:33.560 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-07-26T10:48:33.560 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-07-26T10:48:33.560 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/sunrpc.ko 2024-07-26T10:48:33.560 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma 2024-07-26T10:48:33.560 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-07-26T10:48:33.561 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc 2024-07-26T10:48:33.561 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc.ko 2024-07-26T10:48:33.561 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/tipc/tipc_diag.ko 2024-07-26T10:48:33.561 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix 2024-07-26T10:48:33.561 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/unix/unix_diag.ko 2024-07-26T10:48:33.561 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock 2024-07-26T10:48:33.561 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-07-26T10:48:33.561 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-07-26T10:48:33.561 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock.ko 2024-07-26T10:48:33.561 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_diag.ko 2024-07-26T10:48:33.561 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/vmw_vsock/vsock_loopback.ko 2024-07-26T10:48:33.561 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm 2024-07-26T10:48:33.561 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/net/xfrm/xfrm_ipcomp.ko 2024-07-26T10:48:33.561 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security 2024-07-26T10:48:33.562 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys 2024-07-26T10:48:33.562 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys 2024-07-26T10:48:33.562 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-07-26T10:48:33.562 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys 2024-07-26T10:48:33.562 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/kernel/security/keys/trusted-keys/trusted.ko 2024-07-26T10:48:33.562 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias 2024-07-26T10:48:33.562 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.alias.bin 2024-07-26T10:48:33.562 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin 2024-07-26T10:48:33.562 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.alias.bin 2024-07-26T10:48:33.562 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.bin 2024-07-26T10:48:33.562 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.builtin.modinfo 2024-07-26T10:48:33.562 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep 2024-07-26T10:48:33.562 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.dep.bin 2024-07-26T10:48:33.563 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.devname 2024-07-26T10:48:33.563 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.order 2024-07-26T10:48:33.563 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.softdep 2024-07-26T10:48:33.563 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols 2024-07-26T10:48:33.563 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/modules.symbols.bin 2024-07-26T10:48:33.563 INFO:teuthology.orchestra.run.smithi134.stdout:/lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T10:48:33.567 DEBUG:teuthology.task.kernel:get_image_version: 6.10.0-g3e9255e90710 2024-07-26T10:48:33.567 DEBUG:teuthology.orchestra.run.smithi134:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-07-26T10:48:33.589 INFO:teuthology.orchestra.run.smithi134.stdout:postinstall scriptlet (using /bin/sh): 2024-07-26T10:48:33.589 INFO:teuthology.orchestra.run.smithi134.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T10:48:33.590 INFO:teuthology.orchestra.run.smithi134.stdout: /usr/bin/kernel-install add 6.10.0-g3e9255e90710 /lib/modules/6.10.0-g3e9255e90710/vmlinuz 2024-07-26T10:48:33.590 INFO:teuthology.orchestra.run.smithi134.stdout:fi 2024-07-26T10:48:33.590 INFO:teuthology.orchestra.run.smithi134.stdout:for file in vmlinuz System.map config; do 2024-07-26T10:48:33.590 INFO:teuthology.orchestra.run.smithi134.stdout: if ! cmp --silent "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710"; then 2024-07-26T10:48:33.590 INFO:teuthology.orchestra.run.smithi134.stdout: cp "/lib/modules/6.10.0-g3e9255e90710/${file}" "/boot/${file}-6.10.0-g3e9255e90710" 2024-07-26T10:48:33.590 INFO:teuthology.orchestra.run.smithi134.stdout: fi 2024-07-26T10:48:33.590 INFO:teuthology.orchestra.run.smithi134.stdout:done 2024-07-26T10:48:33.590 INFO:teuthology.orchestra.run.smithi134.stdout:if [ -d "/lib/modules/6.10.0-g3e9255e90710/dtb" ] && \ 2024-07-26T10:48:33.590 INFO:teuthology.orchestra.run.smithi134.stdout: ! diff -rq "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" >/dev/null 2>&1; then 2024-07-26T10:48:33.590 INFO:teuthology.orchestra.run.smithi134.stdout: rm -rf "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T10:48:33.590 INFO:teuthology.orchestra.run.smithi134.stdout: cp -r "/lib/modules/6.10.0-g3e9255e90710/dtb" "/boot/dtb-6.10.0-g3e9255e90710" 2024-07-26T10:48:33.590 INFO:teuthology.orchestra.run.smithi134.stdout:fi 2024-07-26T10:48:33.591 INFO:teuthology.orchestra.run.smithi134.stdout:if [ ! -e "/lib/modules/6.10.0-g3e9255e90710/modules.dep" ]; then 2024-07-26T10:48:33.591 INFO:teuthology.orchestra.run.smithi134.stdout: /usr/sbin/depmod 6.10.0-g3e9255e90710 2024-07-26T10:48:33.591 INFO:teuthology.orchestra.run.smithi134.stdout:fi 2024-07-26T10:48:33.591 INFO:teuthology.orchestra.run.smithi134.stdout:preuninstall scriptlet (using /bin/sh): 2024-07-26T10:48:33.591 INFO:teuthology.orchestra.run.smithi134.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-07-26T10:48:33.591 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-install remove 6.10.0-g3e9255e90710 2024-07-26T10:48:33.591 INFO:teuthology.orchestra.run.smithi134.stdout:fi 2024-07-26T10:48:33.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -qi grub2-tools 2024-07-26T10:48:33.667 INFO:teuthology.orchestra.run.smithi134.stdout:Name : grub2-tools 2024-07-26T10:48:33.667 INFO:teuthology.orchestra.run.smithi134.stdout:Epoch : 1 2024-07-26T10:48:33.667 INFO:teuthology.orchestra.run.smithi134.stdout:Version : 2.06 2024-07-26T10:48:33.667 INFO:teuthology.orchestra.run.smithi134.stdout:Release : 68.el9 2024-07-26T10:48:33.667 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: x86_64 2024-07-26T10:48:33.667 INFO:teuthology.orchestra.run.smithi134.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-07-26T10:48:33.667 INFO:teuthology.orchestra.run.smithi134.stdout:Group : Unspecified 2024-07-26T10:48:33.667 INFO:teuthology.orchestra.run.smithi134.stdout:Size : 8264254 2024-07-26T10:48:33.667 INFO:teuthology.orchestra.run.smithi134.stdout:License : GPLv3+ 2024-07-26T10:48:33.667 INFO:teuthology.orchestra.run.smithi134.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-07-26T10:48:33.667 INFO:teuthology.orchestra.run.smithi134.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-07-26T10:48:33.667 INFO:teuthology.orchestra.run.smithi134.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-07-26T10:48:33.668 INFO:teuthology.orchestra.run.smithi134.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-07-26T10:48:33.668 INFO:teuthology.orchestra.run.smithi134.stdout:Packager : builder@centos.org 2024-07-26T10:48:33.668 INFO:teuthology.orchestra.run.smithi134.stdout:Vendor : CentOS 2024-07-26T10:48:33.668 INFO:teuthology.orchestra.run.smithi134.stdout:URL : http://www.gnu.org/software/grub/ 2024-07-26T10:48:33.668 INFO:teuthology.orchestra.run.smithi134.stdout:Summary : Support tools for GRUB. 2024-07-26T10:48:33.668 INFO:teuthology.orchestra.run.smithi134.stdout:Description : 2024-07-26T10:48:33.668 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:48:33.668 INFO:teuthology.orchestra.run.smithi134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-07-26T10:48:33.668 INFO:teuthology.orchestra.run.smithi134.stdout:customizable bootloader with modular architecture. It supports a rich 2024-07-26T10:48:33.668 INFO:teuthology.orchestra.run.smithi134.stdout:variety of kernel formats, file systems, computer architectures and 2024-07-26T10:48:33.668 INFO:teuthology.orchestra.run.smithi134.stdout:hardware devices. 2024-07-26T10:48:33.668 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:48:33.668 INFO:teuthology.orchestra.run.smithi134.stdout:This subpackage provides tools for support of all platforms. 2024-07-26T10:48:33.669 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-07-26T10:48:33.669 INFO:teuthology.task.kernel:Updating grub on smithi134 to boot 6.10.0-g3e9255e90710 2024-07-26T10:48:33.670 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-07-26T10:48:34.214 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2024-07-26T10:48:35.673 INFO:teuthology.orchestra.run.smithi134.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-07-26T10:48:35.691 INFO:teuthology.orchestra.run.smithi134.stderr:done 2024-07-26T10:48:35.693 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-07-26T10:48:35.711 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.zse9bGZghd 2024-07-26T10:48:35.711 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zse9bGZghd 2024-07-26T10:48:35.792 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.zse9bGZghd 2024-07-26T10:48:35.941 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.zse9bGZghd is 6KB 2024-07-26T10:48:35.990 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.zse9bGZghd 2024-07-26T10:48:36.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo /bin/ls /boot/loader/entries || true 2024-07-26T10:48:36.065 INFO:teuthology.orchestra.run.smithi134.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-07-26T10:48:36.065 INFO:teuthology.orchestra.run.smithi134.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-07-26T10:48:36.065 INFO:teuthology.orchestra.run.smithi134.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-07-26T10:48:36.065 INFO:teuthology.orchestra.run.smithi134.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710.conf 2024-07-26T10:48:36.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.10.0-g3e9255e90710 2024-07-26T10:48:36.231 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2024-07-26T10:49:06.234 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-07-26T10:49:06.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-07-26T10:49:06.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:49:24.706 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-07-26T10:49:33.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-07-26T10:49:33.709 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:49:36.770 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-07-26T10:49:48.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-07-26T10:49:48.772 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:50:20.672 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-07-26T10:50:21.333 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-07-26T10:50:21.333 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.10.0-g3e9255e90710"... 2024-07-26T10:50:21.334 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-07-26T10:50:21.372 INFO:teuthology.orchestra.run.smithi134.stdout:6.10.0-g3e9255e90710 2024-07-26T10:50:21.372 DEBUG:teuthology.task.kernel:current kernel version is 6.10.0-g3e9255e90710 vs 6.10.0-g3e9255e90710 2024-07-26T10:50:21.372 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-07-26T10:50:21.372 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-07-26T10:50:22.373 DEBUG:teuthology.parallel:result is None 2024-07-26T10:50:22.373 INFO:teuthology.run_tasks:Running task internal.base... 2024-07-26T10:50:22.380 INFO:teuthology.task.internal:Creating test directory... 2024-07-26T10:50:22.381 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T10:50:22.385 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T10:50:22.388 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-07-26T10:50:22.420 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-07-26T10:50:22.426 INFO:teuthology.run_tasks:Running task internal.archive... 2024-07-26T10:50:22.469 INFO:teuthology.task.internal:Creating archive directory... 2024-07-26T10:50:22.469 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T10:50:22.473 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T10:50:22.475 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-07-26T10:50:22.582 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-07-26T10:50:22.588 INFO:teuthology.task.internal:Enabling coredump saving... 2024-07-26T10:50:22.589 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T10:50:22.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T10:50:22.605 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2024-07-26T10:50:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T10:50:22.622 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T10:50:22.627 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T10:50:22.648 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-07-26T10:50:22.685 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T10:50:22.705 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T10:50:22.723 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T10:50:22.725 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T10:50:22.738 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T10:50:22.789 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-07-26T10:50:22.793 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-07-26T10:50:22.799 INFO:teuthology.task.internal:Configuring sudo... 2024-07-26T10:50:22.800 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T10:50:22.803 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T10:50:22.806 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-07-26T10:50:22.863 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-07-26T10:50:22.871 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-07-26T10:50:22.872 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T10:50:22.875 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T10:50:22.878 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-07-26T10:50:22.909 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T10:50:23.019 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T10:50:23.109 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T10:50:23.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T10:50:23.257 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T10:50:23.258 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T10:50:23.346 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T10:50:23.388 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T10:50:23.449 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T10:50:23.533 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T10:50:23.573 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-07-26T10:50:23.573 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T10:50:23.639 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T10:50:23.683 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-07-26T10:50:23.741 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T10:50:23.825 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-07-26T10:50:23.866 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-07-26T10:50:23.866 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-07-26T10:50:23.932 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-07-26T10:50:23.940 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-07-26T10:50:23.942 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2024-07-26T10:50:23.972 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T10:50:24.021 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T10:50:24.035 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-07-26T10:50:24.482 INFO:teuthology.run_tasks:Running task internal.timer... 2024-07-26T10:50:24.489 INFO:teuthology.task.internal:Starting timer... 2024-07-26T10:50:24.490 INFO:teuthology.run_tasks:Running task pcp... 2024-07-26T10:50:24.539 INFO:teuthology.run_tasks:Running task selinux... 2024-07-26T10:50:24.587 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-07-26T10:50:24.588 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-07-26T10:50:24.706 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-07-26T10:50:24.709 DEBUG:teuthology.orchestra.run.smithi189:> sudo service auditd rotate 2024-07-26T10:50:24.785 INFO:teuthology.orchestra.run.smithi189.stdout:Rotating logs: 2024-07-26T10:50:24.787 DEBUG:teuthology.orchestra.run.smithi193:> sudo service auditd rotate 2024-07-26T10:50:24.852 INFO:teuthology.orchestra.run.smithi193.stdout:Rotating logs: 2024-07-26T10:50:24.854 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-07-26T10:50:24.854 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-07-26T10:50:24.906 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-07-26T10:50:24.906 DEBUG:teuthology.orchestra.run.smithi189:> /usr/sbin/getenforce 2024-07-26T10:50:24.935 INFO:teuthology.orchestra.run.smithi189.stdout:Permissive 2024-07-26T10:50:24.935 DEBUG:teuthology.orchestra.run.smithi193:> /usr/sbin/getenforce 2024-07-26T10:50:24.958 INFO:teuthology.orchestra.run.smithi193.stdout:Permissive 2024-07-26T10:50:24.958 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi134.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi189.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi193.front.sepia.ceph.com': 'permissive'} 2024-07-26T10:50:24.959 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T10:50:25.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T10:50:25.033 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T10:50:25.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T10:50:25.062 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-07-26T10:50:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-07-26T10:50:25.088 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-07-26T10:50:25.088 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-07-26T10:50:25.169 DEBUG:teuthology.orchestra.run.smithi189:> sudo /usr/sbin/setenforce permissive 2024-07-26T10:50:25.204 DEBUG:teuthology.orchestra.run.smithi193:> sudo /usr/sbin/setenforce permissive 2024-07-26T10:50:25.228 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-07-26T10:50:25.237 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-07-26T10:50:25.240 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-07-26T10:50:25.345 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-07-26T10:50:25.356 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-07-26T10:50:25.357 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi134.front.sepia.ceph.com,smithi189.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-07-26T10:56:54.927 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi134.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2024-07-26T10:56:54.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-07-26T10:56:54.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:56:55.038 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-07-26T10:56:55.182 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-07-26T10:56:55.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-07-26T10:56:55.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:56:55.258 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-07-26T10:56:55.334 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-07-26T10:56:55.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-07-26T10:56:55.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-07-26T10:56:55.408 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-07-26T10:56:55.493 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-07-26T10:56:55.494 INFO:teuthology.run_tasks:Running task clock... 2024-07-26T10:56:55.505 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-07-26T10:56:55.505 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T10:56:55.506 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T10:56:55.513 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T10:56:55.513 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T10:56:55.516 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-07-26T10:56:55.516 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-07-26T10:56:55.546 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T10:56:55.561 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T10:56:55.561 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T10:56:55.575 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T10:56:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: ntpd: command not found 2024-07-26T10:56:55.596 INFO:teuthology.orchestra.run.smithi189.stdout:506 Cannot talk to daemon 2024-07-26T10:56:55.598 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: ntpd: command not found 2024-07-26T10:56:55.605 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-07-26T10:56:55.608 INFO:teuthology.orchestra.run.smithi193.stdout:506 Cannot talk to daemon 2024-07-26T10:56:55.610 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T10:56:55.621 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T10:56:55.624 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T10:56:55.634 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T10:56:55.648 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-07-26T10:56:55.685 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-07-26T10:56:55.689 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T10:56:55.689 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-07-26T10:56:55.689 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:56:55.689 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:56:55.689 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:56:55.690 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:56:55.693 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-07-26T10:56:55.696 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T10:56:55.696 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-07-26T10:56:55.696 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:56:55.696 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:56:55.696 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:56:55.696 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:56:55.724 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2024-07-26T10:56:55.757 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2024-07-26T10:56:55.798 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-07-26T10:56:55.839 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-07-26T10:56:55.971 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-07-26T10:56:55.978 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-07-26T10:56:55.978 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-07-26T10:56:55.978 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:56:55.979 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:56:55.979 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:56:55.979 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-07-26T10:56:55.980 INFO:teuthology.run_tasks:Running task install... 2024-07-26T10:56:55.988 DEBUG:teuthology.task.install:project ceph 2024-07-26T10:56:55.989 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'}} 2024-07-26T10:56:55.989 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'} 2024-07-26T10:56:55.989 INFO:teuthology.task.install:Using flavor: default 2024-07-26T10:56:55.994 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-07-26T10:56:55.994 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-07-26T10:56:55.995 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T10:56:55.995 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:56:55.997 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T10:56:55.998 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:56:55.998 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'tag': None, 'wait_for_package': False} 2024-07-26T10:56:55.999 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:56:56.150 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T10:56:56.150 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T10:56:56.153 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T10:56:56.153 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T10:56:56.159 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/ 2024-07-26T10:56:56.159 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5320.ge8f88bb1 2024-07-26T10:56:56.264 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T10:56:56.264 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T10:56:56.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T10:56:56.267 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T10:56:56.268 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-07-26T10:56:56.268 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T10:56:56.275 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-07-26T10:56:56.275 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-07-26T10:56:56.275 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-07-26T10:56:56.294 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T10:56:56.294 DEBUG:teuthology.orchestra.run.smithi193:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T10:56:56.302 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T10:56:56.303 DEBUG:teuthology.orchestra.run.smithi189:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T10:56:56.345 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-07-26T10:56:56.346 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-07-26T10:56:56.360 DEBUG:teuthology.orchestra.run.smithi193:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T10:56:56.369 DEBUG:teuthology.orchestra.run.smithi189:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T10:56:56.434 DEBUG:teuthology.orchestra.run.smithi193:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T10:56:56.445 DEBUG:teuthology.orchestra.run.smithi189:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T10:56:56.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-07-26T10:56:56.498 INFO:teuthology.orchestra.run.smithi193.stdout:check_obsoletes = 1 2024-07-26T10:56:56.499 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum clean all 2024-07-26T10:56:56.510 INFO:teuthology.orchestra.run.smithi189.stdout:check_obsoletes = 1 2024-07-26T10:56:56.511 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum clean all 2024-07-26T10:56:56.577 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-07-26T10:56:56.655 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2024-07-26T10:56:56.659 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2024-07-26T10:56:56.725 INFO:teuthology.orchestra.run.smithi193.stdout:45 files removed 2024-07-26T10:56:56.739 INFO:teuthology.orchestra.run.smithi189.stdout:45 files removed 2024-07-26T10:56:56.752 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T10:56:56.765 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T10:56:56.971 INFO:teuthology.orchestra.run.smithi134.stdout:45 files removed 2024-07-26T10:56:57.010 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-07-26T10:56:57.345 INFO:teuthology.orchestra.run.smithi193.stdout:ceph packages for x86_64 251 kB/s | 83 kB 00:00 2024-07-26T10:56:57.357 INFO:teuthology.orchestra.run.smithi189.stdout:ceph packages for x86_64 252 kB/s | 83 kB 00:00 2024-07-26T10:56:57.699 INFO:teuthology.orchestra.run.smithi189.stdout:ceph noarch packages 80 kB/s | 24 kB 00:00 2024-07-26T10:56:57.713 INFO:teuthology.orchestra.run.smithi193.stdout:ceph noarch packages 74 kB/s | 24 kB 00:00 2024-07-26T10:56:57.782 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 227 kB/s | 83 kB 00:00 2024-07-26T10:56:58.016 INFO:teuthology.orchestra.run.smithi189.stdout:ceph source packages 6.6 kB/s | 1.9 kB 00:00 2024-07-26T10:56:58.038 INFO:teuthology.orchestra.run.smithi193.stdout:ceph source packages 6.4 kB/s | 1.9 kB 00:00 2024-07-26T10:56:58.257 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 63 kB/s | 24 kB 00:00 2024-07-26T10:56:58.716 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-07-26T10:56:59.758 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.2 MB 00:01 2024-07-26T10:57:00.197 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.2 MB 00:02 2024-07-26T10:57:00.489 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.2 MB 00:01 2024-07-26T10:57:01.509 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-07-26T10:57:02.023 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-07-26T10:57:02.457 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-07-26T10:57:05.868 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-07-26T10:57:07.226 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.5 MB 00:01 2024-07-26T10:57:07.252 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 102 kB/s | 18 kB 00:00 2024-07-26T10:57:07.857 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.5 MB 00:01 2024-07-26T10:57:08.182 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 28 MB/s | 22 MB 00:00 2024-07-26T10:57:08.782 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 18 kB 00:00 2024-07-26T10:57:09.525 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 18 kB 00:00 2024-07-26T10:57:09.661 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 30 MB/s | 22 MB 00:00 2024-07-26T10:57:10.972 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 18 MB/s | 22 MB 00:01 2024-07-26T10:57:13.479 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-07-26T10:57:14.992 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-07-26T10:57:15.737 INFO:teuthology.orchestra.run.smithi189.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T10:57:15.738 INFO:teuthology.orchestra.run.smithi189.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T10:57:15.749 INFO:teuthology.orchestra.run.smithi189.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T10:57:15.750 INFO:teuthology.orchestra.run.smithi189.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T10:57:15.750 INFO:teuthology.orchestra.run.smithi189.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T10:57:15.751 INFO:teuthology.orchestra.run.smithi189.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T10:57:15.753 INFO:teuthology.orchestra.run.smithi189.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T10:57:15.755 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T10:57:15.756 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T10:57:15.820 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-07-26T10:57:15.838 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================= 2024-07-26T10:57:15.838 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-07-26T10:57:15.839 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================= 2024-07-26T10:57:15.839 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-07-26T10:57:15.839 INFO:teuthology.orchestra.run.smithi189.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T10:57:15.839 INFO:teuthology.orchestra.run.smithi189.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T10:57:15.839 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T10:57:15.839 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T10:57:15.839 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T10:57:15.839 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T10:57:15.839 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T10:57:15.839 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T10:57:15.840 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T10:57:15.840 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T10:57:15.840 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T10:57:15.840 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T10:57:15.840 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T10:57:15.840 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T10:57:15.840 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T10:57:15.840 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T10:57:15.840 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T10:57:15.840 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T10:57:15.840 INFO:teuthology.orchestra.run.smithi189.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T10:57:15.841 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T10:57:15.841 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T10:57:15.841 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T10:57:15.841 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T10:57:15.841 INFO:teuthology.orchestra.run.smithi189.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T10:57:15.841 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T10:57:15.841 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T10:57:15.841 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T10:57:15.841 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T10:57:15.841 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T10:57:15.842 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T10:57:15.842 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T10:57:15.842 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T10:57:15.842 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T10:57:15.842 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T10:57:15.842 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T10:57:15.842 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T10:57:15.842 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T10:57:15.842 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrading: 2024-07-26T10:57:15.842 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T10:57:15.843 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T10:57:15.843 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T10:57:15.843 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T10:57:15.843 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T10:57:15.843 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T10:57:15.843 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T10:57:15.843 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T10:57:15.843 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T10:57:15.843 INFO:teuthology.orchestra.run.smithi189.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T10:57:15.843 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T10:57:15.843 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T10:57:15.844 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-07-26T10:57:15.844 INFO:teuthology.orchestra.run.smithi189.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T10:57:15.844 INFO:teuthology.orchestra.run.smithi189.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T10:57:15.844 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T10:57:15.844 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T10:57:15.844 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T10:57:15.844 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T10:57:15.844 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T10:57:15.844 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T10:57:15.844 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T10:57:15.844 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T10:57:15.844 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T10:57:15.845 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T10:57:15.845 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T10:57:15.845 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T10:57:15.845 INFO:teuthology.orchestra.run.smithi189.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T10:57:15.845 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T10:57:15.845 INFO:teuthology.orchestra.run.smithi189.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T10:57:15.845 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T10:57:15.845 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T10:57:15.845 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T10:57:15.845 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T10:57:15.845 INFO:teuthology.orchestra.run.smithi189.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T10:57:15.846 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T10:57:15.846 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T10:57:15.846 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T10:57:15.846 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T10:57:15.846 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T10:57:15.846 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T10:57:15.846 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T10:57:15.846 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T10:57:15.846 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T10:57:15.846 INFO:teuthology.orchestra.run.smithi189.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T10:57:15.846 INFO:teuthology.orchestra.run.smithi189.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T10:57:15.847 INFO:teuthology.orchestra.run.smithi189.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T10:57:15.847 INFO:teuthology.orchestra.run.smithi189.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T10:57:15.847 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T10:57:15.847 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T10:57:15.847 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T10:57:15.847 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T10:57:15.847 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T10:57:15.847 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T10:57:15.847 INFO:teuthology.orchestra.run.smithi189.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T10:57:15.847 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T10:57:15.847 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T10:57:15.848 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T10:57:15.848 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T10:57:15.848 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T10:57:15.848 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T10:57:15.848 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T10:57:15.848 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T10:57:15.848 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T10:57:15.848 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T10:57:15.848 INFO:teuthology.orchestra.run.smithi189.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T10:57:15.848 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T10:57:15.848 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T10:57:15.849 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T10:57:15.849 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T10:57:15.849 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T10:57:15.849 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T10:57:15.849 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T10:57:15.849 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T10:57:15.849 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T10:57:15.849 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T10:57:15.849 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T10:57:15.849 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T10:57:15.849 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T10:57:15.849 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T10:57:15.850 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T10:57:15.850 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T10:57:15.850 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T10:57:15.850 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T10:57:15.850 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T10:57:15.850 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T10:57:15.850 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T10:57:15.850 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T10:57:15.850 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T10:57:15.850 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T10:57:15.850 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T10:57:15.851 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T10:57:15.851 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T10:57:15.851 INFO:teuthology.orchestra.run.smithi189.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T10:57:15.851 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T10:57:15.851 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T10:57:15.851 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T10:57:15.851 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T10:57:15.851 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T10:57:15.851 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T10:57:15.851 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T10:57:15.851 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T10:57:15.852 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T10:57:15.852 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T10:57:15.852 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T10:57:15.852 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T10:57:15.852 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T10:57:15.852 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T10:57:15.852 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T10:57:15.852 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T10:57:15.852 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T10:57:15.852 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T10:57:15.852 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T10:57:15.852 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T10:57:15.853 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T10:57:15.853 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T10:57:15.853 INFO:teuthology.orchestra.run.smithi189.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T10:57:15.853 INFO:teuthology.orchestra.run.smithi189.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T10:57:15.853 INFO:teuthology.orchestra.run.smithi189.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T10:57:15.853 INFO:teuthology.orchestra.run.smithi189.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T10:57:15.853 INFO:teuthology.orchestra.run.smithi189.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T10:57:15.853 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T10:57:15.853 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T10:57:15.853 INFO:teuthology.orchestra.run.smithi189.stdout:Installing weak dependencies: 2024-07-26T10:57:15.853 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T10:57:15.854 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-07-26T10:57:15.854 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-07-26T10:57:15.854 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================= 2024-07-26T10:57:15.854 INFO:teuthology.orchestra.run.smithi189.stdout:Install 148 Packages 2024-07-26T10:57:15.854 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrade 12 Packages 2024-07-26T10:57:15.854 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-07-26T10:57:15.854 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 233 M 2024-07-26T10:57:15.855 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-07-26T10:57:16.806 INFO:teuthology.orchestra.run.smithi189.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-07-26T10:57:16.989 INFO:teuthology.orchestra.run.smithi189.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 4.6 MB/s | 872 kB 00:00 2024-07-26T10:57:17.073 INFO:teuthology.orchestra.run.smithi189.stdout:(3/160): ceph-immutable-object-cache-19.0.0-532 1.7 MB/s | 148 kB 00:00 2024-07-26T10:57:17.156 INFO:teuthology.orchestra.run.smithi189.stdout:(4/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 10 MB/s | 5.5 MB 00:00 2024-07-26T10:57:17.218 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-07-26T10:57:17.238 INFO:teuthology.orchestra.run.smithi193.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T10:57:17.239 INFO:teuthology.orchestra.run.smithi193.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T10:57:17.239 INFO:teuthology.orchestra.run.smithi189.stdout:(5/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 18 MB/s | 1.5 MB 00:00 2024-07-26T10:57:17.250 INFO:teuthology.orchestra.run.smithi193.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T10:57:17.251 INFO:teuthology.orchestra.run.smithi193.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T10:57:17.252 INFO:teuthology.orchestra.run.smithi193.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T10:57:17.253 INFO:teuthology.orchestra.run.smithi193.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T10:57:17.254 INFO:teuthology.orchestra.run.smithi193.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T10:57:17.257 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T10:57:17.258 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T10:57:17.290 INFO:teuthology.orchestra.run.smithi189.stdout:(6/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 10 MB/s | 2.3 MB 00:00 2024-07-26T10:57:17.321 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-07-26T10:57:17.336 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================= 2024-07-26T10:57:17.336 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-07-26T10:57:17.336 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================= 2024-07-26T10:57:17.337 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-07-26T10:57:17.337 INFO:teuthology.orchestra.run.smithi193.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T10:57:17.337 INFO:teuthology.orchestra.run.smithi193.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T10:57:17.337 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T10:57:17.337 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T10:57:17.337 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T10:57:17.337 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T10:57:17.337 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T10:57:17.337 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T10:57:17.337 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T10:57:17.337 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T10:57:17.338 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T10:57:17.338 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T10:57:17.338 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T10:57:17.338 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T10:57:17.338 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T10:57:17.338 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T10:57:17.338 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T10:57:17.338 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T10:57:17.338 INFO:teuthology.orchestra.run.smithi193.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T10:57:17.338 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T10:57:17.338 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T10:57:17.338 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T10:57:17.338 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T10:57:17.339 INFO:teuthology.orchestra.run.smithi193.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T10:57:17.339 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T10:57:17.339 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T10:57:17.339 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T10:57:17.339 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T10:57:17.339 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T10:57:17.339 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T10:57:17.339 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T10:57:17.339 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T10:57:17.339 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T10:57:17.339 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T10:57:17.339 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T10:57:17.339 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T10:57:17.340 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T10:57:17.340 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrading: 2024-07-26T10:57:17.340 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T10:57:17.340 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T10:57:17.340 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T10:57:17.340 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T10:57:17.340 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T10:57:17.340 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T10:57:17.340 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T10:57:17.340 INFO:teuthology.orchestra.run.smithi193.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T10:57:17.340 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T10:57:17.340 INFO:teuthology.orchestra.run.smithi193.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T10:57:17.340 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T10:57:17.341 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T10:57:17.341 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-07-26T10:57:17.341 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T10:57:17.341 INFO:teuthology.orchestra.run.smithi193.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T10:57:17.341 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T10:57:17.341 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T10:57:17.341 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T10:57:17.341 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T10:57:17.341 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T10:57:17.341 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T10:57:17.341 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T10:57:17.341 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T10:57:17.341 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T10:57:17.341 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T10:57:17.342 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T10:57:17.342 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T10:57:17.342 INFO:teuthology.orchestra.run.smithi193.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T10:57:17.342 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T10:57:17.342 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T10:57:17.342 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T10:57:17.342 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T10:57:17.342 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T10:57:17.342 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T10:57:17.342 INFO:teuthology.orchestra.run.smithi193.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T10:57:17.342 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T10:57:17.343 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T10:57:17.343 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T10:57:17.343 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T10:57:17.343 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T10:57:17.343 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T10:57:17.343 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T10:57:17.343 INFO:teuthology.orchestra.run.smithi193.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T10:57:17.343 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T10:57:17.343 INFO:teuthology.orchestra.run.smithi193.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T10:57:17.343 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T10:57:17.343 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T10:57:17.343 INFO:teuthology.orchestra.run.smithi193.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T10:57:17.343 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T10:57:17.344 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T10:57:17.344 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T10:57:17.344 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T10:57:17.344 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T10:57:17.344 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T10:57:17.344 INFO:teuthology.orchestra.run.smithi193.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T10:57:17.344 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T10:57:17.344 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T10:57:17.344 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T10:57:17.344 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T10:57:17.345 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T10:57:17.345 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T10:57:17.345 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T10:57:17.345 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T10:57:17.345 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T10:57:17.345 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T10:57:17.345 INFO:teuthology.orchestra.run.smithi193.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T10:57:17.345 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T10:57:17.345 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T10:57:17.345 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T10:57:17.346 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T10:57:17.346 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T10:57:17.346 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T10:57:17.346 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T10:57:17.346 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T10:57:17.346 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T10:57:17.346 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T10:57:17.346 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T10:57:17.346 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T10:57:17.346 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T10:57:17.346 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T10:57:17.346 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T10:57:17.346 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T10:57:17.346 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T10:57:17.347 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T10:57:17.347 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T10:57:17.347 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T10:57:17.347 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T10:57:17.347 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T10:57:17.347 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T10:57:17.347 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T10:57:17.347 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T10:57:17.347 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T10:57:17.347 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T10:57:17.347 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T10:57:17.347 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T10:57:17.347 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T10:57:17.348 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T10:57:17.348 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T10:57:17.348 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T10:57:17.348 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T10:57:17.348 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T10:57:17.348 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T10:57:17.348 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T10:57:17.348 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T10:57:17.348 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T10:57:17.348 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T10:57:17.348 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T10:57:17.348 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T10:57:17.348 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T10:57:17.349 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T10:57:17.349 INFO:teuthology.orchestra.run.smithi193.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T10:57:17.349 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T10:57:17.349 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T10:57:17.349 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T10:57:17.349 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T10:57:17.349 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T10:57:17.349 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T10:57:17.349 INFO:teuthology.orchestra.run.smithi193.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T10:57:17.349 INFO:teuthology.orchestra.run.smithi193.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T10:57:17.349 INFO:teuthology.orchestra.run.smithi193.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T10:57:17.349 INFO:teuthology.orchestra.run.smithi193.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T10:57:17.349 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T10:57:17.350 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T10:57:17.350 INFO:teuthology.orchestra.run.smithi193.stdout:Installing weak dependencies: 2024-07-26T10:57:17.350 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T10:57:17.350 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T10:57:17.350 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-07-26T10:57:17.350 INFO:teuthology.orchestra.run.smithi193.stdout:======================================================================================= 2024-07-26T10:57:17.350 INFO:teuthology.orchestra.run.smithi193.stdout:Install 148 Packages 2024-07-26T10:57:17.350 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrade 12 Packages 2024-07-26T10:57:17.350 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T10:57:17.351 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 233 M 2024-07-26T10:57:17.351 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-07-26T10:57:17.516 INFO:teuthology.orchestra.run.smithi189.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 18 MB/s | 4.9 MB 00:00 2024-07-26T10:57:17.741 INFO:teuthology.orchestra.run.smithi189.stdout:(8/160): ceph-common-19.0.0-5320.ge8f88bb1.el9. 18 MB/s | 20 MB 00:01 2024-07-26T10:57:17.783 INFO:teuthology.orchestra.run.smithi189.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 607 kB/s | 25 kB 00:00 2024-07-26T10:57:18.085 INFO:teuthology.orchestra.run.smithi193.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-07-26T10:57:18.100 INFO:teuthology.orchestra.run.smithi189.stdout:(10/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el 18 MB/s | 10 MB 00:00 2024-07-26T10:57:18.158 INFO:teuthology.orchestra.run.smithi189.stdout:(11/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 3.8 MB/s | 225 kB 00:00 2024-07-26T10:57:18.234 INFO:teuthology.orchestra.run.smithi189.stdout:(12/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 361 kB/s | 27 kB 00:00 2024-07-26T10:57:18.301 INFO:teuthology.orchestra.run.smithi189.stdout:(13/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 477 kB/s | 32 kB 00:00 2024-07-26T10:57:18.510 INFO:teuthology.orchestra.run.smithi193.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 2.0 MB/s | 872 kB 00:00 2024-07-26T10:57:18.526 INFO:teuthology.orchestra.run.smithi189.stdout:(14/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 14 MB/s | 18 MB 00:01 2024-07-26T10:57:18.568 INFO:teuthology.orchestra.run.smithi189.stdout:(15/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 2.8 MB/s | 765 kB 00:00 2024-07-26T10:57:18.593 INFO:teuthology.orchestra.run.smithi189.stdout:(16/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 2.3 MB/s | 158 kB 00:00 2024-07-26T10:57:18.597 INFO:teuthology.orchestra.run.smithi193.stdout:(3/160): ceph-immutable-object-cache-19.0.0-532 1.7 MB/s | 148 kB 00:00 2024-07-26T10:57:18.618 INFO:teuthology.orchestra.run.smithi189.stdout:(17/160): librados-devel-19.0.0-5320.ge8f88bb1. 2.5 MB/s | 127 kB 00:00 2024-07-26T10:57:18.652 INFO:teuthology.orchestra.run.smithi189.stdout:(18/160): libradosstriper1-19.0.0-5320.ge8f88bb 8.4 MB/s | 503 kB 00:00 2024-07-26T10:57:18.711 INFO:teuthology.orchestra.run.smithi189.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 775 kB/s | 45 kB 00:00 2024-07-26T10:57:18.794 INFO:teuthology.orchestra.run.smithi189.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 1.6 MB/s | 138 kB 00:00 2024-07-26T10:57:18.895 INFO:teuthology.orchestra.run.smithi189.stdout:(21/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 1.8 MB/s | 188 kB 00:00 2024-07-26T10:57:19.028 INFO:teuthology.orchestra.run.smithi189.stdout:(22/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 13 MB/s | 5.3 MB 00:00 2024-07-26T10:57:19.054 INFO:teuthology.orchestra.run.smithi189.stdout:(23/160): python3-rados-19.0.0-5320.ge8f88bb1.e 2.3 MB/s | 372 kB 00:00 2024-07-26T10:57:19.087 INFO:teuthology.orchestra.run.smithi189.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 5.9 MB/s | 355 kB 00:00 2024-07-26T10:57:19.112 INFO:teuthology.orchestra.run.smithi189.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 1.9 MB/s | 116 kB 00:00 2024-07-26T10:57:19.119 INFO:teuthology.orchestra.run.smithi193.stdout:(4/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 4.2 MB/s | 5.5 MB 00:01 2024-07-26T10:57:19.129 INFO:teuthology.orchestra.run.smithi189.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 2.0 MB/s | 86 kB 00:00 2024-07-26T10:57:19.180 INFO:teuthology.orchestra.run.smithi189.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 3.3 MB/s | 170 kB 00:00 2024-07-26T10:57:19.244 INFO:teuthology.orchestra.run.smithi193.stdout:(5/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 3.5 MB/s | 2.3 MB 00:00 2024-07-26T10:57:19.263 INFO:teuthology.orchestra.run.smithi189.stdout:(28/160): ceph-grafana-dashboards-19.0.0-5320.g 413 kB/s | 34 kB 00:00 2024-07-26T10:57:19.339 INFO:teuthology.orchestra.run.smithi189.stdout:(29/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 14 MB/s | 3.1 MB 00:00 2024-07-26T10:57:19.369 INFO:teuthology.orchestra.run.smithi193.stdout:(6/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 6.1 MB/s | 1.5 MB 00:00 2024-07-26T10:57:19.372 INFO:teuthology.orchestra.run.smithi189.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 1.3 MB/s | 149 kB 00:00 2024-07-26T10:57:19.785 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T10:57:19.786 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-07-26T10:57:19.806 INFO:teuthology.orchestra.run.smithi134.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-07-26T10:57:19.807 INFO:teuthology.orchestra.run.smithi134.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-07-26T10:57:19.808 INFO:teuthology.orchestra.run.smithi134.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-07-26T10:57:19.810 INFO:teuthology.orchestra.run.smithi134.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-07-26T10:57:19.812 INFO:teuthology.orchestra.run.smithi134.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-07-26T10:57:19.815 INFO:teuthology.orchestra.run.smithi134.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-07-26T10:57:19.816 INFO:teuthology.orchestra.run.smithi134.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-07-26T10:57:19.890 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-07-26T10:57:19.920 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-07-26T10:57:19.920 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-07-26T10:57:19.920 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-07-26T10:57:19.920 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-07-26T10:57:19.920 INFO:teuthology.orchestra.run.smithi134.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-07-26T10:57:19.920 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 6.6 k 2024-07-26T10:57:19.920 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.5 M 2024-07-26T10:57:19.920 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 872 k 2024-07-26T10:57:19.921 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 148 k 2024-07-26T10:57:19.921 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 1.5 M 2024-07-26T10:57:19.921 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 149 k 2024-07-26T10:57:19.921 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 14 M 2024-07-26T10:57:19.921 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 7.4 M 2024-07-26T10:57:19.921 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 49 k 2024-07-26T10:57:19.921 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 10 M 2024-07-26T10:57:19.921 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 44 M 2024-07-26T10:57:19.921 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 274 k 2024-07-26T10:57:19.921 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 771 k 2024-07-26T10:57:19.921 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 225 k 2024-07-26T10:57:19.921 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-shell x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 27 k 2024-07-26T10:57:19.922 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-top noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 18 k 2024-07-26T10:57:19.922 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-07-26T10:57:19.922 INFO:teuthology.orchestra.run.smithi134.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-07-26T10:57:19.922 INFO:teuthology.orchestra.run.smithi134.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-07-26T10:57:19.922 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 32 k 2024-07-26T10:57:19.922 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 765 k 2024-07-26T10:57:19.922 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 127 k 2024-07-26T10:57:19.922 INFO:teuthology.orchestra.run.smithi134.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-07-26T10:57:19.922 INFO:teuthology.orchestra.run.smithi134.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-07-26T10:57:19.922 INFO:teuthology.orchestra.run.smithi134.stdout: openssl-devel x86_64 1:3.2.2-2.el9 appstream 4.4 M 2024-07-26T10:57:19.922 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-07-26T10:57:19.922 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-07-26T10:57:19.923 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-07-26T10:57:19.923 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 188 k 2024-07-26T10:57:19.923 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 372 k 2024-07-26T10:57:19.923 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 355 k 2024-07-26T10:57:19.923 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 116 k 2024-07-26T10:57:19.923 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 86 k 2024-07-26T10:57:19.923 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T10:57:19.923 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 170 k 2024-07-26T10:57:19.923 INFO:teuthology.orchestra.run.smithi134.stdout: xfsprogs-devel x86_64 6.4.0-3.el9 appstream 96 k 2024-07-26T10:57:19.923 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-07-26T10:57:19.923 INFO:teuthology.orchestra.run.smithi134.stdout: NetworkManager x86_64 1:1.48.2-2.el9 baseos 2.3 M 2024-07-26T10:57:19.923 INFO:teuthology.orchestra.run.smithi134.stdout: NetworkManager-libnm x86_64 1:1.48.2-2.el9 baseos 1.8 M 2024-07-26T10:57:19.923 INFO:teuthology.orchestra.run.smithi134.stdout: NetworkManager-team x86_64 1:1.48.2-2.el9 baseos 41 k 2024-07-26T10:57:19.924 INFO:teuthology.orchestra.run.smithi134.stdout: NetworkManager-tui x86_64 1:1.48.2-2.el9 baseos 247 k 2024-07-26T10:57:19.924 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-07-26T10:57:19.924 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-07-26T10:57:19.924 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-07-26T10:57:19.924 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.4 M 2024-07-26T10:57:19.924 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 3.1 M 2024-07-26T10:57:19.924 INFO:teuthology.orchestra.run.smithi134.stdout: openssl x86_64 1:3.2.2-2.el9 baseos 1.4 M 2024-07-26T10:57:19.924 INFO:teuthology.orchestra.run.smithi134.stdout: openssl-libs x86_64 1:3.2.2-2.el9 baseos 2.4 M 2024-07-26T10:57:19.924 INFO:teuthology.orchestra.run.smithi134.stdout: xfsprogs x86_64 6.4.0-3.el9 baseos 1.1 M 2024-07-26T10:57:19.924 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-07-26T10:57:19.924 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-07-26T10:57:19.924 INFO:teuthology.orchestra.run.smithi134.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-07-26T10:57:19.924 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-07-26T10:57:19.925 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 20 M 2024-07-26T10:57:19.925 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 34 k 2024-07-26T10:57:19.925 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 2.3 M 2024-07-26T10:57:19.925 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 284 k 2024-07-26T10:57:19.925 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 4.9 M 2024-07-26T10:57:19.925 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 18 M 2024-07-26T10:57:19.925 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5320.ge8f88bb1.el9 ceph-noarch 16 k 2024-07-26T10:57:19.925 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 25 k 2024-07-26T10:57:19.925 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-07-26T10:57:19.925 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-07-26T10:57:19.925 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-07-26T10:57:19.925 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-07-26T10:57:19.925 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-07-26T10:57:19.926 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-07-26T10:57:19.926 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-07-26T10:57:19.926 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-07-26T10:57:19.926 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 158 k 2024-07-26T10:57:19.926 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-07-26T10:57:19.926 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-07-26T10:57:19.926 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-07-26T10:57:19.926 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-07-26T10:57:19.926 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-07-26T10:57:19.926 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 503 k 2024-07-26T10:57:19.926 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-07-26T10:57:19.926 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 5.3 M 2024-07-26T10:57:19.927 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-07-26T10:57:19.927 INFO:teuthology.orchestra.run.smithi134.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-07-26T10:57:19.927 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-07-26T10:57:19.927 INFO:teuthology.orchestra.run.smithi134.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-07-26T10:57:19.927 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-07-26T10:57:19.927 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-07-26T10:57:19.927 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-07-26T10:57:19.927 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-07-26T10:57:19.927 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-07-26T10:57:19.927 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-07-26T10:57:19.927 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-07-26T10:57:19.927 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-07-26T10:57:19.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-07-26T10:57:19.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-07-26T10:57:19.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-07-26T10:57:19.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-07-26T10:57:19.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-07-26T10:57:19.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 45 k 2024-07-26T10:57:19.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:19.0.0-5320.ge8f88bb1.el9 ceph 138 k 2024-07-26T10:57:19.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-07-26T10:57:19.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-07-26T10:57:19.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-07-26T10:57:19.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-07-26T10:57:19.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-07-26T10:57:19.928 INFO:teuthology.orchestra.run.smithi134.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-07-26T10:57:19.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-07-26T10:57:19.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-07-26T10:57:19.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-07-26T10:57:19.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-07-26T10:57:19.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-07-26T10:57:19.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-07-26T10:57:19.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-07-26T10:57:19.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-07-26T10:57:19.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-07-26T10:57:19.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-07-26T10:57:19.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-07-26T10:57:19.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-07-26T10:57:19.929 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-07-26T10:57:19.930 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-07-26T10:57:19.930 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-07-26T10:57:19.930 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-07-26T10:57:19.930 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-07-26T10:57:19.930 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-07-26T10:57:19.930 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-07-26T10:57:19.930 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-07-26T10:57:19.930 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-07-26T10:57:19.930 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-07-26T10:57:19.930 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-07-26T10:57:19.930 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-07-26T10:57:19.930 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-07-26T10:57:19.930 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-07-26T10:57:19.931 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-07-26T10:57:19.931 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-07-26T10:57:19.931 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-07-26T10:57:19.931 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-07-26T10:57:19.931 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-07-26T10:57:19.931 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-07-26T10:57:19.931 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-07-26T10:57:19.931 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-07-26T10:57:19.931 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-07-26T10:57:19.931 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-07-26T10:57:19.931 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-07-26T10:57:19.931 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-07-26T10:57:19.931 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-07-26T10:57:19.932 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-07-26T10:57:19.932 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-07-26T10:57:19.932 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-07-26T10:57:19.932 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-07-26T10:57:19.932 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-07-26T10:57:19.932 INFO:teuthology.orchestra.run.smithi134.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-07-26T10:57:19.932 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-07-26T10:57:19.932 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-07-26T10:57:19.932 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-07-26T10:57:19.932 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-07-26T10:57:19.932 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-07-26T10:57:19.932 INFO:teuthology.orchestra.run.smithi134.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-07-26T10:57:19.932 INFO:teuthology.orchestra.run.smithi134.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-07-26T10:57:19.932 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-07-26T10:57:19.933 INFO:teuthology.orchestra.run.smithi134.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-07-26T10:57:19.933 INFO:teuthology.orchestra.run.smithi134.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-07-26T10:57:19.933 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-07-26T10:57:19.933 INFO:teuthology.orchestra.run.smithi134.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-07-26T10:57:19.933 INFO:teuthology.orchestra.run.smithi134.stdout:Installing weak dependencies: 2024-07-26T10:57:19.933 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-07-26T10:57:19.933 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:57:19.933 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-07-26T10:57:19.933 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================= 2024-07-26T10:57:19.933 INFO:teuthology.orchestra.run.smithi134.stdout:Install 148 Packages 2024-07-26T10:57:19.933 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 12 Packages 2024-07-26T10:57:19.933 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:57:19.934 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 233 M 2024-07-26T10:57:19.934 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-07-26T10:57:19.939 INFO:teuthology.orchestra.run.smithi189.stdout:(31/160): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2024-07-26T10:57:20.094 INFO:teuthology.orchestra.run.smithi193.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 5.8 MB/s | 4.9 MB 00:00 2024-07-26T10:57:20.323 INFO:teuthology.orchestra.run.smithi189.stdout:(32/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 17 MB/s | 44 MB 00:02 2024-07-26T10:57:20.348 INFO:teuthology.orchestra.run.smithi189.stdout:(33/160): ceph-mgr-modules-core-19.0.0-5320.ge8 696 kB/s | 284 kB 00:00 2024-07-26T10:57:20.390 INFO:teuthology.orchestra.run.smithi189.stdout:(34/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 739 kB/s | 49 kB 00:00 2024-07-26T10:57:20.424 INFO:teuthology.orchestra.run.smithi189.stdout:(35/160): ceph-prometheus-alerts-19.0.0-5320.ge 218 kB/s | 16 kB 00:00 2024-07-26T10:57:20.457 INFO:teuthology.orchestra.run.smithi189.stdout:(36/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 4.0 MB/s | 274 kB 00:00 2024-07-26T10:57:20.561 INFO:teuthology.orchestra.run.smithi193.stdout:(8/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x86 15 MB/s | 18 MB 00:01 2024-07-26T10:57:20.620 INFO:teuthology.orchestra.run.smithi193.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 433 kB/s | 25 kB 00:00 2024-07-26T10:57:20.641 INFO:teuthology.orchestra.run.smithi189.stdout:(37/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 11 MB/s | 14 MB 00:01 2024-07-26T10:57:20.658 INFO:teuthology.orchestra.run.smithi189.stdout:(38/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 91 kB/s | 18 kB 00:00 2024-07-26T10:57:20.692 INFO:teuthology.orchestra.run.smithi189.stdout:(39/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 2.8 MB/s | 771 kB 00:00 2024-07-26T10:57:20.800 INFO:teuthology.orchestra.run.smithi189.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 504 kB/s | 80 kB 00:00 2024-07-26T10:57:20.850 INFO:teuthology.orchestra.run.smithi189.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-07-26T10:57:20.862 INFO:teuthology.orchestra.run.smithi193.stdout:(10/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el 14 MB/s | 10 MB 00:00 2024-07-26T10:57:20.901 INFO:teuthology.orchestra.run.smithi189.stdout:(42/160): libgfortran-11.4.1-3.el9.x86_64.rpm 3.2 MB/s | 803 kB 00:00 2024-07-26T10:57:20.920 INFO:teuthology.orchestra.run.smithi193.stdout:(11/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 3.8 MB/s | 225 kB 00:00 2024-07-26T10:57:20.926 INFO:teuthology.orchestra.run.smithi189.stdout:(43/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-07-26T10:57:20.968 INFO:teuthology.orchestra.run.smithi189.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-07-26T10:57:20.993 INFO:teuthology.orchestra.run.smithi189.stdout:(45/160): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-07-26T10:57:21.019 INFO:teuthology.orchestra.run.smithi189.stdout:(46/160): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-07-26T10:57:21.052 INFO:teuthology.orchestra.run.smithi189.stdout:(47/160): python3-cryptography-36.0.1-4.el9.x86 6.1 MB/s | 1.2 MB 00:00 2024-07-26T10:57:21.069 INFO:teuthology.orchestra.run.smithi134.stdout:(1/160): ceph-19.0.0-5320.ge8f88bb1.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-07-26T10:57:21.077 INFO:teuthology.orchestra.run.smithi189.stdout:(48/160): python3-urllib3-1.26.5-6.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-07-26T10:57:21.129 INFO:teuthology.orchestra.run.smithi193.stdout:(12/160): ceph-common-19.0.0-5320.ge8f88bb1.el9 6.0 MB/s | 20 MB 00:03 2024-07-26T10:57:21.154 INFO:teuthology.orchestra.run.smithi193.stdout:(13/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 116 kB/s | 27 kB 00:00 2024-07-26T10:57:21.179 INFO:teuthology.orchestra.run.smithi193.stdout:(14/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 635 kB/s | 32 kB 00:00 2024-07-26T10:57:21.221 INFO:teuthology.orchestra.run.smithi193.stdout:(15/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 11 MB/s | 765 kB 00:00 2024-07-26T10:57:21.255 INFO:teuthology.orchestra.run.smithi193.stdout:(16/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 2.1 MB/s | 158 kB 00:00 2024-07-26T10:57:21.295 INFO:teuthology.orchestra.run.smithi134.stdout:(2/160): ceph-fuse-19.0.0-5320.ge8f88bb1.el9.x8 3.8 MB/s | 872 kB 00:00 2024-07-26T10:57:21.314 INFO:teuthology.orchestra.run.smithi193.stdout:(17/160): librados-devel-19.0.0-5320.ge8f88bb1. 1.3 MB/s | 127 kB 00:00 2024-07-26T10:57:21.356 INFO:teuthology.orchestra.run.smithi193.stdout:(18/160): libradosstriper1-19.0.0-5320.ge8f88bb 4.9 MB/s | 503 kB 00:00 2024-07-26T10:57:21.431 INFO:teuthology.orchestra.run.smithi193.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 603 kB/s | 45 kB 00:00 2024-07-26T10:57:21.461 INFO:teuthology.orchestra.run.smithi134.stdout:(3/160): ceph-base-19.0.0-5320.ge8f88bb1.el9.x8 8.3 MB/s | 5.5 MB 00:00 2024-07-26T10:57:21.489 INFO:teuthology.orchestra.run.smithi193.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 2.3 MB/s | 138 kB 00:00 2024-07-26T10:57:21.495 INFO:teuthology.orchestra.run.smithi134.stdout:(4/160): ceph-immutable-object-cache-19.0.0-532 744 kB/s | 148 kB 00:00 2024-07-26T10:57:21.582 INFO:teuthology.orchestra.run.smithi193.stdout:(21/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 2.0 MB/s | 188 kB 00:00 2024-07-26T10:57:21.654 INFO:teuthology.orchestra.run.smithi134.stdout:(5/160): ceph-mds-19.0.0-5320.ge8f88bb1.el9.x86 12 MB/s | 2.3 MB 00:00 2024-07-26T10:57:21.682 INFO:teuthology.orchestra.run.smithi193.stdout:(22/160): python3-rados-19.0.0-5320.ge8f88bb1.e 3.6 MB/s | 372 kB 00:00 2024-07-26T10:57:21.829 INFO:teuthology.orchestra.run.smithi134.stdout:(6/160): ceph-mgr-19.0.0-5320.ge8f88bb1.el9.x86 4.6 MB/s | 1.5 MB 00:00 2024-07-26T10:57:21.945 INFO:teuthology.orchestra.run.smithi189.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 123 kB/s | 107 kB 00:00 2024-07-26T10:57:21.962 INFO:teuthology.orchestra.run.smithi134.stdout:(7/160): ceph-mon-19.0.0-5320.ge8f88bb1.el9.x86 16 MB/s | 4.9 MB 00:00 2024-07-26T10:57:22.041 INFO:teuthology.orchestra.run.smithi193.stdout:(23/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 7.2 MB/s | 5.3 MB 00:00 2024-07-26T10:57:22.078 INFO:teuthology.orchestra.run.smithi189.stdout:(50/160): automake-1.16.2-8.el9.noarch.rpm 651 kB/s | 689 kB 00:01 2024-07-26T10:57:22.120 INFO:teuthology.orchestra.run.smithi189.stdout:(51/160): bison-3.7.4-5.el9.x86_64.rpm 884 kB/s | 943 kB 00:01 2024-07-26T10:57:22.157 INFO:teuthology.orchestra.run.smithi193.stdout:(24/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 1.0 MB/s | 116 kB 00:00 2024-07-26T10:57:22.538 INFO:teuthology.orchestra.run.smithi134.stdout:(8/160): ceph-radosgw-19.0.0-5320.ge8f88bb1.el9 18 MB/s | 10 MB 00:00 2024-07-26T10:57:22.541 INFO:teuthology.orchestra.run.smithi193.stdout:(25/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 414 kB/s | 355 kB 00:00 2024-07-26T10:57:22.588 INFO:teuthology.orchestra.run.smithi134.stdout:(9/160): ceph-selinux-19.0.0-5320.ge8f88bb1.el9 515 kB/s | 25 kB 00:00 2024-07-26T10:57:22.653 INFO:teuthology.orchestra.run.smithi189.stdout:(52/160): elfutils-libelf-devel-0.191-4.el9.x86 31 kB/s | 22 kB 00:00 2024-07-26T10:57:22.700 INFO:teuthology.orchestra.run.smithi193.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 159 kB/s | 86 kB 00:00 2024-07-26T10:57:22.762 INFO:teuthology.orchestra.run.smithi189.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 51 kB/s | 33 kB 00:00 2024-07-26T10:57:23.096 INFO:teuthology.orchestra.run.smithi189.stdout:(54/160): flex-2.6.4-9.el9.x86_64.rpm 308 kB/s | 313 kB 00:01 2024-07-26T10:57:23.142 INFO:teuthology.orchestra.run.smithi193.stdout:(27/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 18 MB/s | 44 MB 00:02 2024-07-26T10:57:23.167 INFO:teuthology.orchestra.run.smithi193.stdout:(28/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 364 kB/s | 170 kB 00:00 2024-07-26T10:57:23.189 INFO:teuthology.orchestra.run.smithi134.stdout:(10/160): ceph-common-19.0.0-5320.ge8f88bb1.el9 8.3 MB/s | 20 MB 00:02 2024-07-26T10:57:23.192 INFO:teuthology.orchestra.run.smithi193.stdout:(29/160): ceph-grafana-dashboards-19.0.0-5320.g 687 kB/s | 34 kB 00:00 2024-07-26T10:57:23.217 INFO:teuthology.orchestra.run.smithi193.stdout:(30/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 2.9 MB/s | 149 kB 00:00 2024-07-26T10:57:23.247 INFO:teuthology.orchestra.run.smithi134.stdout:(11/160): cephfs-mirror-19.0.0-5320.ge8f88bb1.e 3.8 MB/s | 225 kB 00:00 2024-07-26T10:57:23.284 INFO:teuthology.orchestra.run.smithi193.stdout:(31/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 4.1 MB/s | 3.1 MB 00:00 2024-07-26T10:57:23.314 INFO:teuthology.orchestra.run.smithi134.stdout:(12/160): cephfs-shell-19.0.0-5320.ge8f88bb1.el 412 kB/s | 27 kB 00:00 2024-07-26T10:57:23.385 INFO:teuthology.orchestra.run.smithi193.stdout:(32/160): ceph-mgr-modules-core-19.0.0-5320.ge8 2.8 MB/s | 284 kB 00:00 2024-07-26T10:57:23.389 INFO:teuthology.orchestra.run.smithi134.stdout:(13/160): libcephfs-devel-19.0.0-5320.ge8f88bb1 429 kB/s | 32 kB 00:00 2024-07-26T10:57:23.396 INFO:teuthology.orchestra.run.smithi189.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 29 kB/s | 18 kB 00:00 2024-07-26T10:57:23.452 INFO:teuthology.orchestra.run.smithi193.stdout:(33/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 739 kB/s | 49 kB 00:00 2024-07-26T10:57:23.540 INFO:teuthology.orchestra.run.smithi134.stdout:(14/160): libcephfs2-19.0.0-5320.ge8f88bb1.el9. 5.0 MB/s | 765 kB 00:00 2024-07-26T10:57:23.552 INFO:teuthology.orchestra.run.smithi193.stdout:(34/160): ceph-prometheus-alerts-19.0.0-5320.ge 165 kB/s | 16 kB 00:00 2024-07-26T10:57:23.598 INFO:teuthology.orchestra.run.smithi134.stdout:(15/160): libcephsqlite-19.0.0-5320.ge8f88bb1.e 2.7 MB/s | 158 kB 00:00 2024-07-26T10:57:23.638 INFO:teuthology.orchestra.run.smithi193.stdout:(35/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 3.2 MB/s | 274 kB 00:00 2024-07-26T10:57:23.638 INFO:teuthology.orchestra.run.smithi189.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 21 kB/s | 11 kB 00:00 2024-07-26T10:57:23.690 INFO:teuthology.orchestra.run.smithi134.stdout:(16/160): librados-devel-19.0.0-5320.ge8f88bb1. 1.4 MB/s | 127 kB 00:00 2024-07-26T10:57:23.761 INFO:teuthology.orchestra.run.smithi193.stdout:(36/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 6.0 MB/s | 771 kB 00:00 2024-07-26T10:57:23.819 INFO:teuthology.orchestra.run.smithi193.stdout:(37/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 312 kB/s | 18 kB 00:00 2024-07-26T10:57:23.855 INFO:teuthology.orchestra.run.smithi189.stdout:(57/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.5 MB/s | 3.0 MB 00:01 2024-07-26T10:57:23.903 INFO:teuthology.orchestra.run.smithi193.stdout:(38/160): fuse-2.9.9-16.el9.x86_64.rpm 960 kB/s | 80 kB 00:00 2024-07-26T10:57:23.949 INFO:teuthology.orchestra.run.smithi134.stdout:(17/160): ceph-osd-19.0.0-5320.ge8f88bb1.el9.x8 8.4 MB/s | 18 MB 00:02 2024-07-26T10:57:23.991 INFO:teuthology.orchestra.run.smithi134.stdout:(18/160): libradosstriper1-19.0.0-5320.ge8f88bb 1.6 MB/s | 503 kB 00:00 2024-07-26T10:57:24.041 INFO:teuthology.orchestra.run.smithi134.stdout:(19/160): python3-ceph-argparse-19.0.0-5320.ge8 922 kB/s | 45 kB 00:00 2024-07-26T10:57:24.078 INFO:teuthology.orchestra.run.smithi193.stdout:(39/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 16 MB/s | 14 MB 00:00 2024-07-26T10:57:24.100 INFO:teuthology.orchestra.run.smithi134.stdout:(20/160): python3-ceph-common-19.0.0-5320.ge8f8 2.4 MB/s | 138 kB 00:00 2024-07-26T10:57:24.137 INFO:teuthology.orchestra.run.smithi193.stdout:(40/160): libgfortran-11.4.1-3.el9.x86_64.rpm 3.4 MB/s | 803 kB 00:00 2024-07-26T10:57:24.158 INFO:teuthology.orchestra.run.smithi134.stdout:(21/160): python3-cephfs-19.0.0-5320.ge8f88bb1. 3.2 MB/s | 188 kB 00:00 2024-07-26T10:57:24.226 INFO:teuthology.orchestra.run.smithi134.stdout:(22/160): python3-rados-19.0.0-5320.ge8f88bb1.e 5.5 MB/s | 372 kB 00:00 2024-07-26T10:57:24.229 INFO:teuthology.orchestra.run.smithi193.stdout:(41/160): ceph-mgr-diskprediction-local-19.0.0- 7.3 MB/s | 7.4 MB 00:01 2024-07-26T10:57:24.246 INFO:teuthology.orchestra.run.smithi189.stdout:(58/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 188 kB/s | 160 kB 00:00 2024-07-26T10:57:24.254 INFO:teuthology.orchestra.run.smithi193.stdout:(42/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-07-26T10:57:24.288 INFO:teuthology.orchestra.run.smithi193.stdout:(43/160): libquadmath-11.4.1-3.el9.x86_64.rpm 935 kB/s | 195 kB 00:00 2024-07-26T10:57:24.313 INFO:teuthology.orchestra.run.smithi193.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-07-26T10:57:24.347 INFO:teuthology.orchestra.run.smithi193.stdout:(45/160): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-07-26T10:57:24.351 INFO:teuthology.orchestra.run.smithi134.stdout:(23/160): librgw2-19.0.0-5320.ge8f88bb1.el9.x86 13 MB/s | 5.3 MB 00:00 2024-07-26T10:57:24.372 INFO:teuthology.orchestra.run.smithi193.stdout:(46/160): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-07-26T10:57:24.384 INFO:teuthology.orchestra.run.smithi134.stdout:(24/160): python3-rbd-19.0.0-5320.ge8f88bb1.el9 2.2 MB/s | 355 kB 00:00 2024-07-26T10:57:24.405 INFO:teuthology.orchestra.run.smithi193.stdout:(47/160): python3-urllib3-1.26.5-6.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-07-26T10:57:24.418 INFO:teuthology.orchestra.run.smithi134.stdout:(25/160): python3-rgw-19.0.0-5320.ge8f88bb1.el9 1.7 MB/s | 116 kB 00:00 2024-07-26T10:57:24.422 INFO:teuthology.orchestra.run.smithi189.stdout:(59/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 60 kB/s | 45 kB 00:00 2024-07-26T10:57:24.443 INFO:teuthology.orchestra.run.smithi134.stdout:(26/160): rbd-fuse-19.0.0-5320.ge8f88bb1.el9.x8 1.5 MB/s | 86 kB 00:00 2024-07-26T10:57:24.498 INFO:teuthology.orchestra.run.smithi193.stdout:(48/160): python3-cryptography-36.0.1-4.el9.x86 4.6 MB/s | 1.2 MB 00:00 2024-07-26T10:57:24.533 INFO:teuthology.orchestra.run.smithi134.stdout:(27/160): rbd-nbd-19.0.0-5320.ge8f88bb1.el9.x86 1.9 MB/s | 170 kB 00:00 2024-07-26T10:57:24.586 INFO:teuthology.orchestra.run.smithi134.stdout:(28/160): ceph-grafana-dashboards-19.0.0-5320.g 668 kB/s | 34 kB 00:00 2024-07-26T10:57:24.636 INFO:teuthology.orchestra.run.smithi134.stdout:(29/160): ceph-mgr-cephadm-19.0.0-5320.ge8f88bb 3.0 MB/s | 149 kB 00:00 2024-07-26T10:57:24.728 INFO:teuthology.orchestra.run.smithi134.stdout:(30/160): rbd-mirror-19.0.0-5320.ge8f88bb1.el9. 10 MB/s | 3.1 MB 00:00 2024-07-26T10:57:24.905 INFO:teuthology.orchestra.run.smithi189.stdout:(60/160): librdkafka-1.6.1-102.el9.x86_64.rpm 631 kB/s | 662 kB 00:01 2024-07-26T10:57:24.923 INFO:teuthology.orchestra.run.smithi193.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 251 kB/s | 107 kB 00:00 2024-07-26T10:57:25.023 INFO:teuthology.orchestra.run.smithi193.stdout:(50/160): elfutils-libelf-devel-0.191-4.el9.x86 217 kB/s | 22 kB 00:00 2024-07-26T10:57:25.056 INFO:teuthology.orchestra.run.smithi193.stdout:(51/160): automake-1.16.2-8.el9.noarch.rpm 1.0 MB/s | 689 kB 00:00 2024-07-26T10:57:25.114 INFO:teuthology.orchestra.run.smithi189.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 37 kB/s | 25 kB 00:00 2024-07-26T10:57:25.173 INFO:teuthology.orchestra.run.smithi193.stdout:(52/160): bison-3.7.4-5.el9.x86_64.rpm 1.2 MB/s | 943 kB 00:00 2024-07-26T10:57:25.199 INFO:teuthology.orchestra.run.smithi193.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 230 kB/s | 33 kB 00:00 2024-07-26T10:57:25.232 INFO:teuthology.orchestra.run.smithi193.stdout:(54/160): flex-2.6.4-9.el9.x86_64.rpm 1.5 MB/s | 313 kB 00:00 2024-07-26T10:57:25.281 INFO:teuthology.orchestra.run.smithi189.stdout:(62/160): libtool-2.4.6-46.el9.x86_64.rpm 562 kB/s | 581 kB 00:01 2024-07-26T10:57:25.299 INFO:teuthology.orchestra.run.smithi193.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 182 kB/s | 18 kB 00:00 2024-07-26T10:57:25.333 INFO:teuthology.orchestra.run.smithi193.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-07-26T10:57:25.403 INFO:teuthology.orchestra.run.smithi134.stdout:(31/160): ceph-mgr-dashboard-19.0.0-5320.ge8f88 18 MB/s | 14 MB 00:00 2024-07-26T10:57:25.408 INFO:teuthology.orchestra.run.smithi193.stdout:(57/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-07-26T10:57:25.433 INFO:teuthology.orchestra.run.smithi193.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 451 kB/s | 45 kB 00:00 2024-07-26T10:57:25.484 INFO:teuthology.orchestra.run.smithi193.stdout:(59/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.6 MB/s | 3.0 MB 00:00 2024-07-26T10:57:25.526 INFO:teuthology.orchestra.run.smithi193.stdout:(60/160): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-07-26T10:57:25.584 INFO:teuthology.orchestra.run.smithi193.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 253 kB/s | 25 kB 00:00 2024-07-26T10:57:25.618 INFO:teuthology.orchestra.run.smithi193.stdout:(62/160): libtool-2.4.6-46.el9.x86_64.rpm 3.1 MB/s | 581 kB 00:00 2024-07-26T10:57:25.639 INFO:teuthology.orchestra.run.smithi189.stdout:(63/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 65 kB/s | 47 kB 00:00 2024-07-26T10:57:25.643 INFO:teuthology.orchestra.run.smithi193.stdout:(63/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 405 kB/s | 47 kB 00:00 2024-07-26T10:57:25.694 INFO:teuthology.orchestra.run.smithi193.stdout:(64/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-07-26T10:57:25.727 INFO:teuthology.orchestra.run.smithi193.stdout:(65/160): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-07-26T10:57:25.752 INFO:teuthology.orchestra.run.smithi193.stdout:(66/160): openblas-0.3.21-2.el9.x86_64.rpm 324 kB/s | 35 kB 00:00 2024-07-26T10:57:25.936 INFO:teuthology.orchestra.run.smithi193.stdout:(67/160): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-07-26T10:57:25.978 INFO:teuthology.orchestra.run.smithi193.stdout:(68/160): postgresql-13.14-1.el9.x86_64.rpm 7.1 MB/s | 1.6 MB 00:00 2024-07-26T10:57:26.028 INFO:teuthology.orchestra.run.smithi134.stdout:(32/160): ceph-test-19.0.0-5320.ge8f88bb1.el9.x 13 MB/s | 44 MB 00:03 2024-07-26T10:57:26.048 INFO:teuthology.orchestra.run.smithi189.stdout:(64/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 313 kB/s | 292 kB 00:00 2024-07-26T10:57:26.062 INFO:teuthology.orchestra.run.smithi134.stdout:(33/160): ceph-mgr-modules-core-19.0.0-5320.ge8 432 kB/s | 284 kB 00:00 2024-07-26T10:57:26.087 INFO:teuthology.orchestra.run.smithi134.stdout:(34/160): ceph-mgr-rook-19.0.0-5320.ge8f88bb1.e 861 kB/s | 49 kB 00:00 2024-07-26T10:57:26.095 INFO:teuthology.orchestra.run.smithi193.stdout:(69/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-07-26T10:57:26.112 INFO:teuthology.orchestra.run.smithi134.stdout:(35/160): ceph-prometheus-alerts-19.0.0-5320.ge 333 kB/s | 16 kB 00:00 2024-07-26T10:57:26.120 INFO:teuthology.orchestra.run.smithi193.stdout:(70/160): postgresql-private-libs-13.14-1.el9.x 938 kB/s | 133 kB 00:00 2024-07-26T10:57:26.154 INFO:teuthology.orchestra.run.smithi193.stdout:(71/160): postgresql-contrib-13.14-1.el9.x86_64 4.1 MB/s | 911 kB 00:00 2024-07-26T10:57:26.165 INFO:teuthology.orchestra.run.smithi189.stdout:(65/160): lua-5.4.4-4.el9.x86_64.rpm 213 kB/s | 188 kB 00:00 2024-07-26T10:57:26.254 INFO:teuthology.orchestra.run.smithi193.stdout:(72/160): protobuf-3.14.0-13.el9.x86_64.rpm 7.6 MB/s | 1.0 MB 00:00 2024-07-26T10:57:26.255 INFO:teuthology.orchestra.run.smithi134.stdout:(36/160): ceph-mgr-diskprediction-local-19.0.0- 4.9 MB/s | 7.4 MB 00:01 2024-07-26T10:57:26.279 INFO:teuthology.orchestra.run.smithi193.stdout:(73/160): python3-attrs-20.3.0-7.el9.noarch.rpm 698 kB/s | 87 kB 00:00 2024-07-26T10:57:26.288 INFO:teuthology.orchestra.run.smithi134.stdout:(37/160): ceph-volume-19.0.0-5320.ge8f88bb1.el9 1.3 MB/s | 274 kB 00:00 2024-07-26T10:57:26.299 INFO:teuthology.orchestra.run.smithi189.stdout:(66/160): openblas-0.3.21-2.el9.x86_64.rpm 53 kB/s | 35 kB 00:00 2024-07-26T10:57:26.313 INFO:teuthology.orchestra.run.smithi134.stdout:(38/160): cephfs-top-19.0.0-5320.ge8f88bb1.el9. 317 kB/s | 18 kB 00:00 2024-07-26T10:57:26.354 INFO:teuthology.orchestra.run.smithi193.stdout:(74/160): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-07-26T10:57:26.381 INFO:teuthology.orchestra.run.smithi134.stdout:(39/160): cephadm-19.0.0-5320.ge8f88bb1.el9.noa 2.8 MB/s | 771 kB 00:00 2024-07-26T10:57:26.388 INFO:teuthology.orchestra.run.smithi193.stdout:(75/160): python3-colorama-0.4.6-3.el9.noarch.r 563 kB/s | 61 kB 00:00 2024-07-26T10:57:26.463 INFO:teuthology.orchestra.run.smithi193.stdout:(76/160): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-07-26T10:57:26.480 INFO:teuthology.orchestra.run.smithi134.stdout:(40/160): fuse-2.9.9-16.el9.x86_64.rpm 419 kB/s | 80 kB 00:00 2024-07-26T10:57:26.505 INFO:teuthology.orchestra.run.smithi193.stdout:(77/160): python3-jinja2-2.11.3-6.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-07-26T10:57:26.539 INFO:teuthology.orchestra.run.smithi134.stdout:(41/160): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-07-26T10:57:26.572 INFO:teuthology.orchestra.run.smithi193.stdout:(78/160): python3-jsonpatch-1.21-16.el9.noarch. 236 kB/s | 26 kB 00:00 2024-07-26T10:57:26.573 INFO:teuthology.orchestra.run.smithi134.stdout:(42/160): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-07-26T10:57:26.606 INFO:teuthology.orchestra.run.smithi134.stdout:(43/160): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-07-26T10:57:26.631 INFO:teuthology.orchestra.run.smithi134.stdout:(44/160): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-07-26T10:57:26.648 INFO:teuthology.orchestra.run.smithi193.stdout:(79/160): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-07-26T10:57:26.657 INFO:teuthology.orchestra.run.smithi134.stdout:(45/160): python3-pycparser-2.20-6.el9.noarch.r 2.7 MB/s | 135 kB 00:00 2024-07-26T10:57:26.673 INFO:teuthology.orchestra.run.smithi193.stdout:(80/160): python3-jsonpointer-2.0-4.el9.noarch. 114 kB/s | 19 kB 00:00 2024-07-26T10:57:26.690 INFO:teuthology.orchestra.run.smithi134.stdout:(46/160): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-07-26T10:57:26.698 INFO:teuthology.orchestra.run.smithi193.stdout:(81/160): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-07-26T10:57:26.724 INFO:teuthology.orchestra.run.smithi134.stdout:(47/160): python3-urllib3-1.26.5-6.el9.noarch.r 3.2 MB/s | 215 kB 00:00 2024-07-26T10:57:26.748 INFO:teuthology.orchestra.run.smithi193.stdout:(82/160): python3-markupsafe-1.1.1-12.el9.x86_6 346 kB/s | 35 kB 00:00 2024-07-26T10:57:26.791 INFO:teuthology.orchestra.run.smithi134.stdout:(48/160): python3-cryptography-36.0.1-4.el9.x86 4.9 MB/s | 1.2 MB 00:00 2024-07-26T10:57:26.815 INFO:teuthology.orchestra.run.smithi193.stdout:(83/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.1 MB/s | 250 kB 00:00 2024-07-26T10:57:26.857 INFO:teuthology.orchestra.run.smithi193.stdout:(84/160): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-07-26T10:57:26.924 INFO:teuthology.orchestra.run.smithi193.stdout:(85/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-07-26T10:57:26.949 INFO:teuthology.orchestra.run.smithi193.stdout:(86/160): python3-packaging-20.9-5.el9.noarch.r 576 kB/s | 77 kB 00:00 2024-07-26T10:57:26.975 INFO:teuthology.orchestra.run.smithi134.stdout:(49/160): boost-program-options-1.75.0-8.el9.x8 584 kB/s | 107 kB 00:00 2024-07-26T10:57:26.975 INFO:teuthology.orchestra.run.smithi193.stdout:(87/160): python3-prettytable-0.7.2-27.el9.noar 360 kB/s | 42 kB 00:00 2024-07-26T10:57:27.016 INFO:teuthology.orchestra.run.smithi134.stdout:(50/160): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-07-26T10:57:27.033 INFO:teuthology.orchestra.run.smithi193.stdout:(88/160): python3-protobuf-3.14.0-13.el9.noarch 2.4 MB/s | 269 kB 00:00 2024-07-26T10:57:27.042 INFO:teuthology.orchestra.run.smithi134.stdout:(51/160): elfutils-libelf-devel-0.191-4.el9.x86 329 kB/s | 22 kB 00:00 2024-07-26T10:57:27.059 INFO:teuthology.orchestra.run.smithi193.stdout:(89/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-07-26T10:57:27.075 INFO:teuthology.orchestra.run.smithi134.stdout:(52/160): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 313 kB 00:00 2024-07-26T10:57:27.084 INFO:teuthology.orchestra.run.smithi193.stdout:(90/160): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-07-26T10:57:27.101 INFO:teuthology.orchestra.run.smithi134.stdout:(53/160): flexiblas-3.0.4-8.el9.x86_64.rpm 558 kB/s | 33 kB 00:00 2024-07-26T10:57:27.134 INFO:teuthology.orchestra.run.smithi193.stdout:(91/160): python3-pytz-2021.1-5.el9.noarch.rpm 506 kB/s | 51 kB 00:00 2024-07-26T10:57:27.143 INFO:teuthology.orchestra.run.smithi134.stdout:(54/160): bison-3.7.4-5.el9.x86_64.rpm 2.2 MB/s | 943 kB 00:00 2024-07-26T10:57:27.160 INFO:teuthology.orchestra.run.smithi193.stdout:(92/160): python3-requests-oauthlib-1.3.0-12.el 534 kB/s | 54 kB 00:00 2024-07-26T10:57:27.168 INFO:teuthology.orchestra.run.smithi134.stdout:(55/160): flexiblas-openblas-openmp-3.0.4-8.el9 275 kB/s | 18 kB 00:00 2024-07-26T10:57:27.210 INFO:teuthology.orchestra.run.smithi134.stdout:(56/160): libaio-devel-0.3.111-13.el9.x86_64.rp 168 kB/s | 11 kB 00:00 2024-07-26T10:57:27.235 INFO:teuthology.orchestra.run.smithi193.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 415 kB/s | 42 kB 00:00 2024-07-26T10:57:27.252 INFO:teuthology.orchestra.run.smithi134.stdout:(57/160): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-07-26T10:57:27.260 INFO:teuthology.orchestra.run.smithi193.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 441 kB/s | 44 kB 00:00 2024-07-26T10:57:27.277 INFO:teuthology.orchestra.run.smithi134.stdout:(58/160): librabbitmq-0.11.0-7.el9.x86_64.rpm 684 kB/s | 45 kB 00:00 2024-07-26T10:57:27.344 INFO:teuthology.orchestra.run.smithi193.stdout:(95/160): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-07-26T10:57:27.357 INFO:teuthology.orchestra.run.smithi189.stdout:(67/160): openblas-openmp-0.3.21-2.el9.x86_64.r 3.6 MB/s | 4.7 MB 00:01 2024-07-26T10:57:27.369 INFO:teuthology.orchestra.run.smithi193.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 330 kB/s | 36 kB 00:00 2024-07-26T10:57:27.386 INFO:teuthology.orchestra.run.smithi134.stdout:(59/160): libtool-2.4.6-46.el9.x86_64.rpm 5.3 MB/s | 581 kB 00:00 2024-07-26T10:57:27.428 INFO:teuthology.orchestra.run.smithi134.stdout:(60/160): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-07-26T10:57:27.432 INFO:teuthology.orchestra.run.smithi189.stdout:(68/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 3.5 MB/s | 4.4 MB 00:01 2024-07-26T10:57:27.452 INFO:teuthology.orchestra.run.smithi193.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 437 kB/s | 47 kB 00:00 2024-07-26T10:57:27.453 INFO:teuthology.orchestra.run.smithi134.stdout:(61/160): libuuid-devel-2.37.4-18.el9.x86_64.rp 386 kB/s | 25 kB 00:00 2024-07-26T10:57:27.478 INFO:teuthology.orchestra.run.smithi193.stdout:(98/160): socat-1.7.4.1-6.el9.x86_64.rpm 2.7 MB/s | 304 kB 00:00 2024-07-26T10:57:27.495 INFO:teuthology.orchestra.run.smithi134.stdout:(62/160): libzstd-devel-1.5.1-2.el9.x86_64.rpm 719 kB/s | 47 kB 00:00 2024-07-26T10:57:27.500 INFO:teuthology.orchestra.run.smithi189.stdout:(69/160): postgresql-13.14-1.el9.x86_64.rpm 1.4 MB/s | 1.6 MB 00:01 2024-07-26T10:57:27.553 INFO:teuthology.orchestra.run.smithi193.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 576 kB/s | 58 kB 00:00 2024-07-26T10:57:27.554 INFO:teuthology.orchestra.run.smithi134.stdout:(63/160): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-07-26T10:57:27.587 INFO:teuthology.orchestra.run.smithi193.stdout:(100/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 881 kB/s | 96 kB 00:00 2024-07-26T10:57:27.604 INFO:teuthology.orchestra.run.smithi134.stdout:(64/160): openblas-0.3.21-2.el9.x86_64.rpm 720 kB/s | 35 kB 00:00 2024-07-26T10:57:27.638 INFO:teuthology.orchestra.run.smithi134.stdout:(65/160): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 292 kB 00:00 2024-07-26T10:57:27.662 INFO:teuthology.orchestra.run.smithi193.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 586 kB/s | 64 kB 00:00 2024-07-26T10:57:27.696 INFO:teuthology.orchestra.run.smithi193.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 412 kB/s | 45 kB 00:00 2024-07-26T10:57:27.721 INFO:teuthology.orchestra.run.smithi193.stdout:(103/160): lua-devel-5.4.4-4.el9.x86_64.rpm 380 kB/s | 22 kB 00:00 2024-07-26T10:57:27.813 INFO:teuthology.orchestra.run.smithi193.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 5.9 MB/s | 548 kB 00:00 2024-07-26T10:57:27.846 INFO:teuthology.orchestra.run.smithi193.stdout:(105/160): gperftools-libs-2.9.1-2.el9.x86_64.r 9.1 MB/s | 309 kB 00:00 2024-07-26T10:57:27.863 INFO:teuthology.orchestra.run.smithi193.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-07-26T10:57:27.906 INFO:teuthology.orchestra.run.smithi193.stdout:(107/160): protobuf-compiler-3.14.0-13.el9.x86_ 4.0 MB/s | 863 kB 00:00 2024-07-26T10:57:28.098 INFO:teuthology.orchestra.run.smithi193.stdout:(108/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:01 2024-07-26T10:57:28.148 INFO:teuthology.orchestra.run.smithi193.stdout:(109/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 107 kB/s | 26 kB 00:00 2024-07-26T10:57:28.173 INFO:teuthology.orchestra.run.smithi193.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.9 MB/s | 67 kB 00:00 2024-07-26T10:57:28.198 INFO:teuthology.orchestra.run.smithi193.stdout:(111/160): liboath-2.6.7-2.el9.x86_64.rpm 498 kB/s | 49 kB 00:00 2024-07-26T10:57:28.223 INFO:teuthology.orchestra.run.smithi193.stdout:(112/160): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-07-26T10:57:28.274 INFO:teuthology.orchestra.run.smithi189.stdout:(70/160): postgresql-private-libs-13.14-1.el9.x 158 kB/s | 133 kB 00:00 2024-07-26T10:57:28.315 INFO:teuthology.orchestra.run.smithi193.stdout:(113/160): libarrow-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 4.4 MB 00:00 2024-07-26T10:57:28.341 INFO:teuthology.orchestra.run.smithi193.stdout:(114/160): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-07-26T10:57:28.354 INFO:teuthology.orchestra.run.smithi134.stdout:(66/160): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.3 MB/s | 3.0 MB 00:01 2024-07-26T10:57:28.366 INFO:teuthology.orchestra.run.smithi193.stdout:(115/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 866 kB/s | 43 kB 00:00 2024-07-26T10:57:28.391 INFO:teuthology.orchestra.run.smithi193.stdout:(116/160): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2024-07-26T10:57:28.416 INFO:teuthology.orchestra.run.smithi189.stdout:(71/160): postgresql-contrib-13.14-1.el9.x86_64 861 kB/s | 911 kB 00:01 2024-07-26T10:57:28.470 INFO:teuthology.orchestra.run.smithi193.stdout:(117/160): python3-certifi-2023.05.07-4.el9.noa 136 kB/s | 14 kB 00:00 2024-07-26T10:57:28.500 INFO:teuthology.orchestra.run.smithi193.stdout:(118/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.7 MB/s | 839 kB 00:00 2024-07-26T10:57:28.525 INFO:teuthology.orchestra.run.smithi193.stdout:(119/160): python3-cheroot-8.6.0-4.el9.noarch.r 1.2 MB/s | 172 kB 00:00 2024-07-26T10:57:28.551 INFO:teuthology.orchestra.run.smithi193.stdout:(120/160): python3-cherrypy-18.6.1-2.el9.noarch 4.3 MB/s | 358 kB 00:00 2024-07-26T10:57:28.576 INFO:teuthology.orchestra.run.smithi193.stdout:(121/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-07-26T10:57:28.601 INFO:teuthology.orchestra.run.smithi193.stdout:(122/160): python3-google-auth-2.32.0-1.el9.noa 2.8 MB/s | 216 kB 00:00 2024-07-26T10:57:28.626 INFO:teuthology.orchestra.run.smithi193.stdout:(123/160): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-07-26T10:57:28.652 INFO:teuthology.orchestra.run.smithi193.stdout:(124/160): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-07-26T10:57:28.677 INFO:teuthology.orchestra.run.smithi193.stdout:(125/160): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2024-07-26T10:57:28.702 INFO:teuthology.orchestra.run.smithi193.stdout:(126/160): python3-jaraco-collections-3.0.0-8.e 459 kB/s | 23 kB 00:00 2024-07-26T10:57:28.728 INFO:teuthology.orchestra.run.smithi193.stdout:(127/160): python3-jaraco-functools-3.5.0-2.el9 389 kB/s | 19 kB 00:00 2024-07-26T10:57:28.753 INFO:teuthology.orchestra.run.smithi193.stdout:(128/160): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-07-26T10:57:28.778 INFO:teuthology.orchestra.run.smithi193.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-07-26T10:57:28.804 INFO:teuthology.orchestra.run.smithi193.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-07-26T10:57:28.829 INFO:teuthology.orchestra.run.smithi193.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-07-26T10:57:28.850 INFO:teuthology.orchestra.run.smithi189.stdout:(72/160): postgresql-server-13.14-1.el9.x86_64. 4.3 MB/s | 5.8 MB 00:01 2024-07-26T10:57:28.862 INFO:teuthology.orchestra.run.smithi193.stdout:(132/160): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-07-26T10:57:28.888 INFO:teuthology.orchestra.run.smithi193.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 655 kB/s | 16 kB 00:00 2024-07-26T10:57:28.921 INFO:teuthology.orchestra.run.smithi193.stdout:(134/160): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-07-26T10:57:28.947 INFO:teuthology.orchestra.run.smithi193.stdout:(135/160): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-07-26T10:57:28.972 INFO:teuthology.orchestra.run.smithi193.stdout:(136/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-07-26T10:57:28.997 INFO:teuthology.orchestra.run.smithi193.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 614 kB/s | 31 kB 00:00 2024-07-26T10:57:29.022 INFO:teuthology.orchestra.run.smithi193.stdout:(138/160): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-07-26T10:57:29.048 INFO:teuthology.orchestra.run.smithi193.stdout:(139/160): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-07-26T10:57:29.073 INFO:teuthology.orchestra.run.smithi193.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 709 kB/s | 36 kB 00:00 2024-07-26T10:57:29.098 INFO:teuthology.orchestra.run.smithi193.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-07-26T10:57:29.140 INFO:teuthology.orchestra.run.smithi193.stdout:(142/160): python3-grpcio-1.46.7-10.el9.x86_64. 3.5 MB/s | 2.0 MB 00:00 2024-07-26T10:57:29.155 INFO:teuthology.orchestra.run.smithi134.stdout:(67/160): postgresql-13.14-1.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-07-26T10:57:29.166 INFO:teuthology.orchestra.run.smithi193.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-07-26T10:57:29.175 INFO:teuthology.orchestra.run.smithi189.stdout:(73/160): python3-attrs-20.3.0-7.el9.noarch.rpm 115 kB/s | 87 kB 00:00 2024-07-26T10:57:29.191 INFO:teuthology.orchestra.run.smithi193.stdout:(144/160): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-07-26T10:57:29.216 INFO:teuthology.orchestra.run.smithi193.stdout:(145/160): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-07-26T10:57:29.252 INFO:teuthology.orchestra.run.smithi193.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 3.1 MB/s | 192 kB 00:00 2024-07-26T10:57:29.277 INFO:teuthology.orchestra.run.smithi193.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-07-26T10:57:29.328 INFO:teuthology.orchestra.run.smithi193.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-07-26T10:57:29.376 INFO:teuthology.orchestra.run.smithi189.stdout:(74/160): protobuf-3.14.0-13.el9.x86_64.rpm 939 kB/s | 1.0 MB 00:01 2024-07-26T10:57:29.405 INFO:teuthology.orchestra.run.smithi134.stdout:(68/160): openssl-devel-3.2.2-2.el9.x86_64.rpm 2.5 MB/s | 4.4 MB 00:01 2024-07-26T10:57:29.530 INFO:teuthology.orchestra.run.smithi134.stdout:(69/160): postgresql-private-libs-13.14-1.el9.x 1.1 MB/s | 133 kB 00:00 2024-07-26T10:57:29.553 INFO:teuthology.orchestra.run.smithi193.stdout:(149/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 11 MB/s | 3.1 MB 00:00 2024-07-26T10:57:29.620 INFO:teuthology.orchestra.run.smithi193.stdout:(150/160): librados2-19.0.0-5320.ge8f88bb1.el9. 9.2 MB/s | 3.4 MB 00:00 2024-07-26T10:57:29.664 INFO:teuthology.orchestra.run.smithi134.stdout:(70/160): openblas-openmp-0.3.21-2.el9.x86_64.r 2.3 MB/s | 4.7 MB 00:02 2024-07-26T10:57:29.687 INFO:teuthology.orchestra.run.smithi193.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 613 kB/s | 41 kB 00:00 2024-07-26T10:57:29.731 INFO:teuthology.orchestra.run.smithi134.stdout:(71/160): postgresql-contrib-13.14-1.el9.x86_64 1.5 MB/s | 911 kB 00:00 2024-07-26T10:57:29.778 INFO:teuthology.orchestra.run.smithi193.stdout:(152/160): NetworkManager-tui-1.48.2-2.el9.x86_ 2.6 MB/s | 247 kB 00:00 2024-07-26T10:57:29.821 INFO:teuthology.orchestra.run.smithi193.stdout:(153/160): elfutils-debuginfod-client-0.191-4.e 880 kB/s | 37 kB 00:00 2024-07-26T10:57:29.843 INFO:teuthology.orchestra.run.smithi134.stdout:(72/160): python3-attrs-20.3.0-7.el9.noarch.rpm 786 kB/s | 87 kB 00:00 2024-07-26T10:57:29.862 INFO:teuthology.orchestra.run.smithi193.stdout:(154/160): elfutils-libelf-0.191-4.el9.x86_64.r 4.9 MB/s | 207 kB 00:00 2024-07-26T10:57:29.884 INFO:teuthology.orchestra.run.smithi189.stdout:(75/160): python3-colorama-0.4.6-3.el9.noarch.r 86 kB/s | 61 kB 00:00 2024-07-26T10:57:29.904 INFO:teuthology.orchestra.run.smithi193.stdout:(155/160): elfutils-libs-0.191-4.el9.x86_64.rpm 6.1 MB/s | 260 kB 00:00 2024-07-26T10:57:29.980 INFO:teuthology.orchestra.run.smithi193.stdout:(156/160): NetworkManager-1.48.2-2.el9.x86_64.r 3.5 MB/s | 2.3 MB 00:00 2024-07-26T10:57:30.055 INFO:teuthology.orchestra.run.smithi193.stdout:(157/160): NetworkManager-libnm-1.48.2-2.el9.x8 3.6 MB/s | 1.8 MB 00:00 2024-07-26T10:57:30.176 INFO:teuthology.orchestra.run.smithi189.stdout:(76/160): python3-babel-2.9.1-2.el9.noarch.rpm 4.5 MB/s | 6.0 MB 00:01 2024-07-26T10:57:30.205 INFO:teuthology.orchestra.run.smithi193.stdout:(158/160): openssl-3.2.2-2.el9.x86_64.rpm 4.5 MB/s | 1.4 MB 00:00 2024-07-26T10:57:30.231 INFO:teuthology.orchestra.run.smithi134.stdout:(73/160): protobuf-3.14.0-13.el9.x86_64.rpm 1.8 MB/s | 1.0 MB 00:00 2024-07-26T10:57:30.260 INFO:teuthology.orchestra.run.smithi189.stdout:(77/160): python3-devel-3.9.18-3.el9.x86_64.rpm 276 kB/s | 244 kB 00:00 2024-07-26T10:57:30.290 INFO:teuthology.orchestra.run.smithi134.stdout:(74/160): python3-colorama-0.4.6-3.el9.noarch.r 1.0 MB/s | 61 kB 00:00 2024-07-26T10:57:30.314 INFO:teuthology.orchestra.run.smithi193.stdout:(159/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 7.2 MB/s | 2.4 MB 00:00 2024-07-26T10:57:30.389 INFO:teuthology.orchestra.run.smithi193.stdout:(160/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 3.2 MB/s | 1.1 MB 00:00 2024-07-26T10:57:30.396 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-07-26T10:57:30.396 INFO:teuthology.orchestra.run.smithi193.stdout:Total 18 MB/s | 233 MB 00:13 2024-07-26T10:57:30.449 INFO:teuthology.orchestra.run.smithi134.stdout:(75/160): python3-devel-3.9.18-3.el9.x86_64.rpm 1.5 MB/s | 244 kB 00:00 2024-07-26T10:57:30.599 INFO:teuthology.orchestra.run.smithi134.stdout:(76/160): python3-jinja2-2.11.3-6.el9.noarch.rp 1.6 MB/s | 248 kB 00:00 2024-07-26T10:57:30.657 INFO:teuthology.orchestra.run.smithi134.stdout:(77/160): python3-jsonpatch-1.21-16.el9.noarch. 447 kB/s | 26 kB 00:00 2024-07-26T10:57:30.708 INFO:teuthology.orchestra.run.smithi134.stdout:(78/160): python3-jsonpointer-2.0-4.el9.noarch. 389 kB/s | 19 kB 00:00 2024-07-26T10:57:30.817 INFO:teuthology.orchestra.run.smithi134.stdout:(79/160): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-07-26T10:57:30.826 INFO:teuthology.orchestra.run.smithi189.stdout:(78/160): python3-jsonpatch-1.21-16.el9.noarch. 41 kB/s | 26 kB 00:00 2024-07-26T10:57:30.852 INFO:teuthology.orchestra.run.smithi189.stdout:(79/160): python3-jinja2-2.11.3-6.el9.noarch.rp 257 kB/s | 248 kB 00:00 2024-07-26T10:57:30.867 INFO:teuthology.orchestra.run.smithi134.stdout:(80/160): python3-markupsafe-1.1.1-12.el9.x86_6 707 kB/s | 35 kB 00:00 2024-07-26T10:57:30.877 INFO:teuthology.orchestra.run.smithi189.stdout:(80/160): python3-jsonpointer-2.0-4.el9.noarch. 31 kB/s | 19 kB 00:00 2024-07-26T10:57:31.502 INFO:teuthology.orchestra.run.smithi189.stdout:(81/160): python3-markupsafe-1.1.1-12.el9.x86_6 53 kB/s | 35 kB 00:00 2024-07-26T10:57:31.585 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-07-26T10:57:31.677 INFO:teuthology.orchestra.run.smithi189.stdout:(82/160): python3-mako-1.1.4-6.el9.noarch.rpm 202 kB/s | 172 kB 00:00 2024-07-26T10:57:31.779 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-07-26T10:57:31.779 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-07-26T10:57:32.238 INFO:teuthology.orchestra.run.smithi189.stdout:(83/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.8 MB/s | 5.1 MB 00:01 2024-07-26T10:57:32.395 INFO:teuthology.orchestra.run.smithi189.stdout:(84/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 281 kB/s | 250 kB 00:00 2024-07-26T10:57:32.500 INFO:teuthology.orchestra.run.smithi134.stdout:(81/160): postgresql-server-13.14-1.el9.x86_64. 1.9 MB/s | 5.8 MB 00:02 2024-07-26T10:57:32.651 INFO:teuthology.orchestra.run.smithi134.stdout:(82/160): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 250 kB 00:00 2024-07-26T10:57:32.695 INFO:teuthology.orchestra.run.smithi189.stdout:(85/160): python3-oauthlib-3.1.1-5.el9.noarch.r 218 kB/s | 222 kB 00:01 2024-07-26T10:57:32.792 INFO:teuthology.orchestra.run.smithi134.stdout:(83/160): python3-oauthlib-3.1.1-5.el9.noarch.r 1.5 MB/s | 222 kB 00:00 2024-07-26T10:57:32.943 INFO:teuthology.orchestra.run.smithi134.stdout:(84/160): python3-babel-2.9.1-2.el9.noarch.rpm 1.9 MB/s | 6.0 MB 00:03 2024-07-26T10:57:32.968 INFO:teuthology.orchestra.run.smithi134.stdout:(85/160): python3-packaging-20.9-5.el9.noarch.r 441 kB/s | 77 kB 00:00 2024-07-26T10:57:32.987 INFO:teuthology.orchestra.run.smithi189.stdout:(86/160): python3-packaging-20.9-5.el9.noarch.r 103 kB/s | 77 kB 00:00 2024-07-26T10:57:33.010 INFO:teuthology.orchestra.run.smithi134.stdout:(86/160): python3-prettytable-0.7.2-27.el9.noar 639 kB/s | 42 kB 00:00 2024-07-26T10:57:33.054 INFO:teuthology.orchestra.run.smithi189.stdout:(87/160): python3-prettytable-0.7.2-27.el9.noar 64 kB/s | 42 kB 00:00 2024-07-26T10:57:33.135 INFO:teuthology.orchestra.run.smithi134.stdout:(87/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-07-26T10:57:33.160 INFO:teuthology.orchestra.run.smithi134.stdout:(88/160): python3-protobuf-3.14.0-13.el9.noarch 1.4 MB/s | 269 kB 00:00 2024-07-26T10:57:33.219 INFO:teuthology.orchestra.run.smithi134.stdout:(89/160): python3-pytz-2021.1-5.el9.noarch.rpm 881 kB/s | 51 kB 00:00 2024-07-26T10:57:33.294 INFO:teuthology.orchestra.run.smithi134.stdout:(90/160): python3-requests-oauthlib-1.3.0-12.el 723 kB/s | 54 kB 00:00 2024-07-26T10:57:33.353 INFO:teuthology.orchestra.run.smithi134.stdout:(91/160): python3-pyasn1-modules-0.4.8-6.el9.no 1.3 MB/s | 279 kB 00:00 2024-07-26T10:57:33.403 INFO:teuthology.orchestra.run.smithi134.stdout:(92/160): python3-toml-0.10.2-6.el9.noarch.rpm 849 kB/s | 42 kB 00:00 2024-07-26T10:57:33.462 INFO:teuthology.orchestra.run.smithi134.stdout:(93/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 765 kB/s | 44 kB 00:00 2024-07-26T10:57:33.604 INFO:teuthology.orchestra.run.smithi134.stdout:(94/160): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.9 MB/s | 5.1 MB 00:02 2024-07-26T10:57:33.634 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-07-26T10:57:33.634 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-07-26T10:57:33.646 INFO:teuthology.orchestra.run.smithi189.stdout:(88/160): python3-protobuf-3.14.0-13.el9.noarch 283 kB/s | 269 kB 00:00 2024-07-26T10:57:33.662 INFO:teuthology.orchestra.run.smithi134.stdout:(95/160): qatlib-24.02.0-1.el9.x86_64.rpm 1.1 MB/s | 221 kB 00:00 2024-07-26T10:57:33.688 INFO:teuthology.orchestra.run.smithi134.stdout:(96/160): qatlib-service-24.02.0-1.el9.x86_64.r 434 kB/s | 36 kB 00:00 2024-07-26T10:57:33.738 INFO:teuthology.orchestra.run.smithi134.stdout:(97/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 972 kB/s | 47 kB 00:00 2024-07-26T10:57:33.771 INFO:teuthology.orchestra.run.smithi134.stdout:(98/160): socat-1.7.4.1-6.el9.x86_64.rpm 3.6 MB/s | 304 kB 00:00 2024-07-26T10:57:33.788 INFO:teuthology.orchestra.run.smithi134.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-07-26T10:57:33.804 INFO:teuthology.orchestra.run.smithi189.stdout:(89/160): python3-pyasn1-0.4.8-6.el9.noarch.rpm 194 kB/s | 159 kB 00:00 2024-07-26T10:57:33.822 INFO:teuthology.orchestra.run.smithi134.stdout:(100/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 1.9 MB/s | 96 kB 00:00 2024-07-26T10:57:33.874 INFO:teuthology.orchestra.run.smithi134.stdout:(101/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 767 kB/s | 64 kB 00:00 2024-07-26T10:57:33.898 INFO:teuthology.orchestra.run.smithi134.stdout:(102/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 601 kB/s | 45 kB 00:00 2024-07-26T10:57:33.955 INFO:teuthology.orchestra.run.smithi189.stdout:(90/160): python3-pyasn1-modules-0.4.8-6.el9.no 310 kB/s | 279 kB 00:00 2024-07-26T10:57:33.982 INFO:teuthology.orchestra.run.smithi134.stdout:(103/160): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2024-07-26T10:57:34.115 INFO:teuthology.orchestra.run.smithi134.stdout:(104/160): abseil-cpp-20211102.0-3.el9.x86_64.r 4.0 MB/s | 548 kB 00:00 2024-07-26T10:57:34.157 INFO:teuthology.orchestra.run.smithi134.stdout:(105/160): protobuf-compiler-3.14.0-13.el9.x86_ 3.3 MB/s | 863 kB 00:00 2024-07-26T10:57:34.249 INFO:teuthology.orchestra.run.smithi134.stdout:(106/160): gperftools-libs-2.9.1-2.el9.x86_64.r 2.3 MB/s | 309 kB 00:00 2024-07-26T10:57:34.283 INFO:teuthology.orchestra.run.smithi134.stdout:(107/160): grpc-data-1.46.7-10.el9.noarch.rpm 156 kB/s | 19 kB 00:00 2024-07-26T10:57:34.308 INFO:teuthology.orchestra.run.smithi134.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-07-26T10:57:34.333 INFO:teuthology.orchestra.run.smithi134.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 2.0 MB/s | 49 kB 00:00 2024-07-26T10:57:34.363 INFO:teuthology.orchestra.run.smithi189.stdout:(91/160): python3-pytz-2021.1-5.el9.noarch.rpm 71 kB/s | 51 kB 00:00 2024-07-26T10:57:34.367 INFO:teuthology.orchestra.run.smithi134.stdout:(110/160): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-07-26T10:57:34.401 INFO:teuthology.orchestra.run.smithi134.stdout:(111/160): luarocks-3.9.2-5.el9.noarch.rpm 4.6 MB/s | 151 kB 00:00 2024-07-26T10:57:34.484 INFO:teuthology.orchestra.run.smithi134.stdout:(112/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.9 MB/s | 839 kB 00:00 2024-07-26T10:57:34.505 INFO:teuthology.orchestra.run.smithi189.stdout:(92/160): python3-requests-oauthlib-1.3.0-12.el 77 kB/s | 54 kB 00:00 2024-07-26T10:57:34.534 INFO:teuthology.orchestra.run.smithi134.stdout:(113/160): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-07-26T10:57:34.560 INFO:teuthology.orchestra.run.smithi134.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-07-26T10:57:34.585 INFO:teuthology.orchestra.run.smithi134.stdout:(115/160): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-07-26T10:57:34.610 INFO:teuthology.orchestra.run.smithi134.stdout:(116/160): python3-certifi-2023.05.07-4.el9.noa 585 kB/s | 14 kB 00:00 2024-07-26T10:57:34.644 INFO:teuthology.orchestra.run.smithi134.stdout:(117/160): python3-cheroot-8.6.0-4.el9.noarch.r 5.2 MB/s | 172 kB 00:00 2024-07-26T10:57:34.686 INFO:teuthology.orchestra.run.smithi134.stdout:(118/160): python3-cherrypy-18.6.1-2.el9.noarch 8.6 MB/s | 358 kB 00:00 2024-07-26T10:57:34.728 INFO:teuthology.orchestra.run.smithi134.stdout:(119/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 7.3 MB/s | 303 kB 00:00 2024-07-26T10:57:34.770 INFO:teuthology.orchestra.run.smithi134.stdout:(120/160): python3-google-auth-2.32.0-1.el9.noa 5.2 MB/s | 216 kB 00:00 2024-07-26T10:57:34.895 INFO:teuthology.orchestra.run.smithi134.stdout:(121/160): libarrow-9.0.0-11.el9.x86_64.rpm 6.9 MB/s | 4.4 MB 00:00 2024-07-26T10:57:34.929 INFO:teuthology.orchestra.run.smithi134.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 4.3 MB/s | 144 kB 00:00 2024-07-26T10:57:34.954 INFO:teuthology.orchestra.run.smithi134.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 444 kB/s | 11 kB 00:00 2024-07-26T10:57:34.979 INFO:teuthology.orchestra.run.smithi134.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 728 kB/s | 18 kB 00:00 2024-07-26T10:57:35.030 INFO:teuthology.orchestra.run.smithi134.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 471 kB/s | 23 kB 00:00 2024-07-26T10:57:35.030 INFO:teuthology.orchestra.run.smithi189.stdout:(93/160): python3-toml-0.10.2-6.el9.noarch.rpm 62 kB/s | 42 kB 00:00 2024-07-26T10:57:35.055 INFO:teuthology.orchestra.run.smithi134.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 806 kB/s | 19 kB 00:00 2024-07-26T10:57:35.080 INFO:teuthology.orchestra.run.smithi134.stdout:(127/160): python3-jaraco-text-3.2.0-6.el9.noar 811 kB/s | 20 kB 00:00 2024-07-26T10:57:35.164 INFO:teuthology.orchestra.run.smithi134.stdout:(128/160): python3-grpcio-1.46.7-10.el9.x86_64. 5.2 MB/s | 2.0 MB 00:00 2024-07-26T10:57:35.172 INFO:teuthology.orchestra.run.smithi189.stdout:(94/160): python3-wcwidth-0.2.5-8.el9.noarch.rp 66 kB/s | 44 kB 00:00 2024-07-26T10:57:35.189 INFO:teuthology.orchestra.run.smithi134.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.9 MB/s | 46 kB 00:00 2024-07-26T10:57:35.223 INFO:teuthology.orchestra.run.smithi134.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 2.4 MB/s | 79 kB 00:00 2024-07-26T10:57:35.257 INFO:teuthology.orchestra.run.smithi134.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-07-26T10:57:35.298 INFO:teuthology.orchestra.run.smithi134.stdout:(132/160): python3-pecan-1.4.2-3.el9.noarch.rpm 6.5 MB/s | 272 kB 00:00 2024-07-26T10:57:35.324 INFO:teuthology.orchestra.run.smithi134.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 672 kB/s | 16 kB 00:00 2024-07-26T10:57:35.357 INFO:teuthology.orchestra.run.smithi134.stdout:(134/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.7 MB/s | 90 kB 00:00 2024-07-26T10:57:35.400 INFO:teuthology.orchestra.run.smithi134.stdout:(135/160): python3-kubernetes-26.1.0-2.el9.noar 3.2 MB/s | 1.0 MB 00:00 2024-07-26T10:57:35.416 INFO:teuthology.orchestra.run.smithi134.stdout:(136/160): python3-pyperclip-1.8.2-2.el9.noarch 489 kB/s | 28 kB 00:00 2024-07-26T10:57:35.442 INFO:teuthology.orchestra.run.smithi134.stdout:(137/160): python3-repoze-lru-0.7-16.el9.noarch 748 kB/s | 31 kB 00:00 2024-07-26T10:57:35.467 INFO:teuthology.orchestra.run.smithi134.stdout:(138/160): python3-rsa-4.9-2.el9.noarch.rpm 2.4 MB/s | 59 kB 00:00 2024-07-26T10:57:35.484 INFO:teuthology.orchestra.run.smithi134.stdout:(139/160): python3-tempora-5.0.0-2.el9.noarch.r 2.2 MB/s | 36 kB 00:00 2024-07-26T10:57:35.518 INFO:teuthology.orchestra.run.smithi134.stdout:(140/160): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-07-26T10:57:35.543 INFO:teuthology.orchestra.run.smithi134.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 882 kB/s | 51 kB 00:00 2024-07-26T10:57:35.605 INFO:teuthology.orchestra.run.smithi134.stdout:(142/160): python3-websocket-client-1.2.3-2.el9 1.4 MB/s | 90 kB 00:00 2024-07-26T10:57:35.623 INFO:teuthology.orchestra.run.smithi189.stdout:(95/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.7 MB/s | 16 MB 00:01 2024-07-26T10:57:35.644 INFO:teuthology.orchestra.run.smithi134.stdout:(143/160): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-07-26T10:57:35.669 INFO:teuthology.orchestra.run.smithi134.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 827 kB/s | 20 kB 00:00 2024-07-26T10:57:35.703 INFO:teuthology.orchestra.run.smithi134.stdout:(145/160): re2-20211101-3.el9.x86_64.rpm 5.8 MB/s | 192 kB 00:00 2024-07-26T10:57:35.736 INFO:teuthology.orchestra.run.smithi134.stdout:(146/160): python3-werkzeug-2.0.3-3.el9.1.noarc 3.2 MB/s | 427 kB 00:00 2024-07-26T10:57:35.828 INFO:teuthology.orchestra.run.smithi134.stdout:(147/160): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-07-26T10:57:35.931 INFO:teuthology.orchestra.run.smithi189.stdout:(96/160): qatlib-24.02.0-1.el9.x86_64.rpm 245 kB/s | 221 kB 00:00 2024-07-26T10:57:35.956 INFO:teuthology.orchestra.run.smithi189.stdout:(97/160): qatlib-service-24.02.0-1.el9.x86_64.r 46 kB/s | 36 kB 00:00 2024-07-26T10:57:36.053 INFO:teuthology.orchestra.run.smithi134.stdout:(148/160): librados2-19.0.0-5320.ge8f88bb1.el9. 11 MB/s | 3.4 MB 00:00 2024-07-26T10:57:36.174 INFO:teuthology.orchestra.run.smithi134.stdout:(149/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 9.1 MB/s | 3.1 MB 00:00 2024-07-26T10:57:36.307 INFO:teuthology.orchestra.run.smithi189.stdout:(98/160): qatzip-libs-1.2.0-1.el9.x86_64.rpm 69 kB/s | 47 kB 00:00 2024-07-26T10:57:36.337 INFO:teuthology.orchestra.run.smithi134.stdout:(150/160): NetworkManager-1.48.2-2.el9.x86_64.r 8.2 MB/s | 2.3 MB 00:00 2024-07-26T10:57:36.379 INFO:teuthology.orchestra.run.smithi134.stdout:(151/160): NetworkManager-team-1.48.2-2.el9.x86 1.0 MB/s | 41 kB 00:00 2024-07-26T10:57:36.429 INFO:teuthology.orchestra.run.smithi134.stdout:(152/160): NetworkManager-tui-1.48.2-2.el9.x86_ 4.9 MB/s | 247 kB 00:00 2024-07-26T10:57:36.496 INFO:teuthology.orchestra.run.smithi134.stdout:(153/160): NetworkManager-libnm-1.48.2-2.el9.x8 5.6 MB/s | 1.8 MB 00:00 2024-07-26T10:57:36.522 INFO:teuthology.orchestra.run.smithi134.stdout:(154/160): elfutils-debuginfod-client-0.191-4.e 402 kB/s | 37 kB 00:00 2024-07-26T10:57:36.555 INFO:teuthology.orchestra.run.smithi134.stdout:(155/160): elfutils-libelf-0.191-4.el9.x86_64.r 3.5 MB/s | 207 kB 00:00 2024-07-26T10:57:36.589 INFO:teuthology.orchestra.run.smithi134.stdout:(156/160): elfutils-libs-0.191-4.el9.x86_64.rpm 3.8 MB/s | 260 kB 00:00 2024-07-26T10:57:36.656 INFO:teuthology.orchestra.run.smithi134.stdout:(157/160): openssl-3.2.2-2.el9.x86_64.rpm 14 MB/s | 1.4 MB 00:00 2024-07-26T10:57:36.665 INFO:teuthology.orchestra.run.smithi189.stdout:(99/160): uuid-1.6.2-55.el9.x86_64.rpm 82 kB/s | 58 kB 00:00 2024-07-26T10:57:36.773 INFO:teuthology.orchestra.run.smithi134.stdout:(158/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 13 MB/s | 2.4 MB 00:00 2024-07-26T10:57:36.823 INFO:teuthology.orchestra.run.smithi134.stdout:(159/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 6.5 MB/s | 1.1 MB 00:00 2024-07-26T10:57:36.849 INFO:teuthology.orchestra.run.smithi189.stdout:(100/160): socat-1.7.4.1-6.el9.x86_64.rpm 332 kB/s | 304 kB 00:00 2024-07-26T10:57:37.066 INFO:teuthology.orchestra.run.smithi189.stdout:(101/160): xfsprogs-devel-6.4.0-3.el9.x86_64.rp 126 kB/s | 96 kB 00:00 2024-07-26T10:57:37.149 INFO:teuthology.orchestra.run.smithi189.stdout:(102/160): lua-devel-5.4.4-4.el9.x86_64.rpm 271 kB/s | 22 kB 00:00 2024-07-26T10:57:37.519 INFO:teuthology.orchestra.run.smithi189.stdout:(103/160): protobuf-compiler-3.14.0-13.el9.x86_ 2.3 MB/s | 863 kB 00:00 2024-07-26T10:57:37.558 INFO:teuthology.orchestra.run.smithi189.stdout:(104/160): xmlstarlet-1.6.1-20.el9.x86_64.rpm 71 kB/s | 64 kB 00:00 2024-07-26T10:57:37.650 INFO:teuthology.orchestra.run.smithi189.stdout:(105/160): abseil-cpp-20211102.0-3.el9.x86_64.r 4.1 MB/s | 548 kB 00:00 2024-07-26T10:57:37.676 INFO:teuthology.orchestra.run.smithi189.stdout:(106/160): grpc-data-1.46.7-10.el9.noarch.rpm 781 kB/s | 19 kB 00:00 2024-07-26T10:57:37.701 INFO:teuthology.orchestra.run.smithi189.stdout:(107/160): gperftools-libs-2.9.1-2.el9.x86_64.r 2.6 MB/s | 309 kB 00:00 2024-07-26T10:57:37.726 INFO:teuthology.orchestra.run.smithi189.stdout:(108/160): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-07-26T10:57:37.751 INFO:teuthology.orchestra.run.smithi189.stdout:(109/160): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-07-26T10:57:37.776 INFO:teuthology.orchestra.run.smithi189.stdout:(110/160): zlib-devel-1.2.11-41.el9.x86_64.rpm 48 kB/s | 45 kB 00:00 2024-07-26T10:57:37.802 INFO:teuthology.orchestra.run.smithi189.stdout:(111/160): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-07-26T10:57:37.869 INFO:teuthology.orchestra.run.smithi189.stdout:(112/160): parquet-libs-9.0.0-11.el9.x86_64.rpm 20 MB/s | 839 kB 00:00 2024-07-26T10:57:37.903 INFO:teuthology.orchestra.run.smithi189.stdout:(113/160): luarocks-3.9.2-5.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-07-26T10:57:37.919 INFO:teuthology.orchestra.run.smithi189.stdout:(114/160): python3-bcrypt-3.2.2-1.el9.x86_64.rp 2.6 MB/s | 43 kB 00:00 2024-07-26T10:57:37.953 INFO:teuthology.orchestra.run.smithi189.stdout:(115/160): python3-asyncssh-2.13.2-5.el9.noarch 6.4 MB/s | 548 kB 00:00 2024-07-26T10:57:37.970 INFO:teuthology.orchestra.run.smithi189.stdout:(116/160): python3-cachetools-4.2.4-1.el9.noarc 639 kB/s | 32 kB 00:00 2024-07-26T10:57:37.987 INFO:teuthology.orchestra.run.smithi189.stdout:(117/160): python3-certifi-2023.05.07-4.el9.noa 419 kB/s | 14 kB 00:00 2024-07-26T10:57:38.021 INFO:teuthology.orchestra.run.smithi189.stdout:(118/160): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-07-26T10:57:38.046 INFO:teuthology.orchestra.run.smithi189.stdout:(119/160): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-07-26T10:57:38.071 INFO:teuthology.orchestra.run.smithi189.stdout:(120/160): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-07-26T10:57:38.097 INFO:teuthology.orchestra.run.smithi189.stdout:(121/160): python3-google-auth-2.32.0-1.el9.noa 4.2 MB/s | 216 kB 00:00 2024-07-26T10:57:38.122 INFO:teuthology.orchestra.run.smithi189.stdout:(122/160): python3-grpcio-tools-1.46.7-10.el9.x 5.6 MB/s | 144 kB 00:00 2024-07-26T10:57:38.147 INFO:teuthology.orchestra.run.smithi189.stdout:(123/160): python3-jaraco-8.2.1-3.el9.noarch.rp 431 kB/s | 11 kB 00:00 2024-07-26T10:57:38.173 INFO:teuthology.orchestra.run.smithi189.stdout:(124/160): python3-jaraco-classes-3.2.1-5.el9.n 706 kB/s | 18 kB 00:00 2024-07-26T10:57:38.198 INFO:teuthology.orchestra.run.smithi189.stdout:(125/160): python3-jaraco-collections-3.0.0-8.e 923 kB/s | 23 kB 00:00 2024-07-26T10:57:38.223 INFO:teuthology.orchestra.run.smithi189.stdout:(126/160): python3-jaraco-functools-3.5.0-2.el9 784 kB/s | 19 kB 00:00 2024-07-26T10:57:38.290 INFO:teuthology.orchestra.run.smithi189.stdout:(127/160): libarrow-9.0.0-11.el9.x86_64.rpm 7.2 MB/s | 4.4 MB 00:00 2024-07-26T10:57:38.315 INFO:teuthology.orchestra.run.smithi189.stdout:(128/160): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-07-26T10:57:38.341 INFO:teuthology.orchestra.run.smithi189.stdout:(129/160): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-07-26T10:57:38.366 INFO:teuthology.orchestra.run.smithi189.stdout:(130/160): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-07-26T10:57:38.391 INFO:teuthology.orchestra.run.smithi189.stdout:(131/160): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-07-26T10:57:38.442 INFO:teuthology.orchestra.run.smithi189.stdout:(132/160): python3-grpcio-1.46.7-10.el9.x86_64. 5.5 MB/s | 2.0 MB 00:00 2024-07-26T10:57:38.467 INFO:teuthology.orchestra.run.smithi189.stdout:(133/160): python3-portend-3.1.0-2.el9.noarch.r 657 kB/s | 16 kB 00:00 2024-07-26T10:57:38.493 INFO:teuthology.orchestra.run.smithi189.stdout:(134/160): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-07-26T10:57:38.526 INFO:teuthology.orchestra.run.smithi189.stdout:(135/160): python3-kubernetes-26.1.0-2.el9.noar 4.9 MB/s | 1.0 MB 00:00 2024-07-26T10:57:38.551 INFO:teuthology.orchestra.run.smithi189.stdout:(136/160): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-07-26T10:57:38.568 INFO:teuthology.orchestra.run.smithi189.stdout:(137/160): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-07-26T10:57:38.585 INFO:teuthology.orchestra.run.smithi189.stdout:(138/160): python3-repoze-lru-0.7-16.el9.noarch 917 kB/s | 31 kB 00:00 2024-07-26T10:57:38.602 INFO:teuthology.orchestra.run.smithi189.stdout:(139/160): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-07-26T10:57:38.619 INFO:teuthology.orchestra.run.smithi189.stdout:(140/160): python3-tempora-5.0.0-2.el9.noarch.r 1.0 MB/s | 36 kB 00:00 2024-07-26T10:57:38.636 INFO:teuthology.orchestra.run.smithi189.stdout:(141/160): python3-typing-extensions-4.4.0-2.el 1.5 MB/s | 51 kB 00:00 2024-07-26T10:57:38.661 INFO:teuthology.orchestra.run.smithi189.stdout:(142/160): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-07-26T10:57:38.686 INFO:teuthology.orchestra.run.smithi189.stdout:(143/160): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-07-26T10:57:38.712 INFO:teuthology.orchestra.run.smithi189.stdout:(144/160): python3-zc-lockfile-2.0-10.el9.noarc 800 kB/s | 20 kB 00:00 2024-07-26T10:57:38.737 INFO:teuthology.orchestra.run.smithi189.stdout:(145/160): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-07-26T10:57:38.762 INFO:teuthology.orchestra.run.smithi189.stdout:(146/160): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-07-26T10:57:38.788 INFO:teuthology.orchestra.run.smithi189.stdout:(147/160): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-07-26T10:57:38.830 INFO:teuthology.orchestra.run.smithi189.stdout:(148/160): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-07-26T10:57:39.088 INFO:teuthology.orchestra.run.smithi189.stdout:(149/160): librados2-19.0.0-5320.ge8f88bb1.el9. 10 MB/s | 3.4 MB 00:00 2024-07-26T10:57:39.139 INFO:teuthology.orchestra.run.smithi189.stdout:(150/160): NetworkManager-1.48.2-2.el9.x86_64.r 7.5 MB/s | 2.3 MB 00:00 2024-07-26T10:57:39.197 INFO:teuthology.orchestra.run.smithi189.stdout:(151/160): librbd1-19.0.0-5320.ge8f88bb1.el9.x8 7.7 MB/s | 3.1 MB 00:00 2024-07-26T10:57:39.223 INFO:teuthology.orchestra.run.smithi189.stdout:(152/160): NetworkManager-team-1.48.2-2.el9.x86 488 kB/s | 41 kB 00:00 2024-07-26T10:57:39.265 INFO:teuthology.orchestra.run.smithi189.stdout:(153/160): elfutils-debuginfod-client-0.191-4.e 884 kB/s | 37 kB 00:00 2024-07-26T10:57:39.315 INFO:teuthology.orchestra.run.smithi189.stdout:(154/160): elfutils-libelf-0.191-4.el9.x86_64.r 4.1 MB/s | 207 kB 00:00 2024-07-26T10:57:39.357 INFO:teuthology.orchestra.run.smithi189.stdout:(155/160): elfutils-libs-0.191-4.el9.x86_64.rpm 6.1 MB/s | 260 kB 00:00 2024-07-26T10:57:39.382 INFO:teuthology.orchestra.run.smithi189.stdout:(156/160): NetworkManager-tui-1.48.2-2.el9.x86_ 1.3 MB/s | 247 kB 00:00 2024-07-26T10:57:39.424 INFO:teuthology.orchestra.run.smithi189.stdout:(157/160): NetworkManager-libnm-1.48.2-2.el9.x8 5.4 MB/s | 1.8 MB 00:00 2024-07-26T10:57:39.458 INFO:teuthology.orchestra.run.smithi189.stdout:(158/160): openssl-3.2.2-2.el9.x86_64.rpm 13 MB/s | 1.4 MB 00:00 2024-07-26T10:57:39.499 INFO:teuthology.orchestra.run.smithi189.stdout:(159/160): xfsprogs-6.4.0-3.el9.x86_64.rpm 14 MB/s | 1.1 MB 00:00 2024-07-26T10:57:39.575 INFO:teuthology.orchestra.run.smithi189.stdout:(160/160): openssl-libs-3.2.2-2.el9.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-07-26T10:57:39.581 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-07-26T10:57:39.581 INFO:teuthology.orchestra.run.smithi189.stdout:Total 9.8 MB/s | 233 MB 00:23 2024-07-26T10:57:40.137 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-07-26T10:57:40.226 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T10:57:40.264 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T10:57:40.318 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T10:57:40.352 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T10:57:40.378 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T10:57:40.420 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T10:57:40.752 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T10:57:40.835 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-07-26T10:57:40.870 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:57:40.997 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:57:41.021 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:57:41.030 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-07-26T10:57:41.030 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-07-26T10:57:41.116 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:57:41.142 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T10:57:41.175 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T10:57:41.372 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T10:57:41.424 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T10:57:41.452 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:57:41.556 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:57:41.586 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T10:57:41.648 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T10:57:41.670 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T10:57:41.692 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T10:57:41.853 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T10:57:41.951 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T10:57:42.127 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T10:57:42.223 INFO:teuthology.orchestra.run.smithi134.stdout:(160/160): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.8 MB/s | 16 MB 00:08 2024-07-26T10:57:42.234 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-07-26T10:57:42.234 INFO:teuthology.orchestra.run.smithi134.stdout:Total 10 MB/s | 233 MB 00:22 2024-07-26T10:57:42.304 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T10:57:42.356 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T10:57:42.460 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:57:42.549 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:57:42.756 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:57:42.881 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T10:57:42.911 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T10:57:42.915 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-07-26T10:57:42.916 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-07-26T10:57:42.946 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T10:57:43.020 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T10:57:43.073 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T10:57:43.113 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T10:57:43.182 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T10:57:43.214 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T10:57:43.242 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T10:57:43.311 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T10:57:43.364 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T10:57:43.395 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T10:57:43.439 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T10:57:43.515 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T10:57:43.549 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T10:57:43.595 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T10:57:43.621 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T10:57:43.692 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T10:57:43.745 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T10:57:43.823 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T10:57:44.183 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T10:57:44.239 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T10:57:44.762 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T10:57:44.843 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T10:57:44.872 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T10:57:44.908 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T10:57:44.949 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T10:57:44.987 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T10:57:45.028 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T10:57:45.196 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T10:57:45.239 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T10:57:45.331 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T10:57:45.384 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-07-26T10:57:45.676 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-07-26T10:57:45.676 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-07-26T10:57:45.692 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T10:57:45.764 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T10:57:45.800 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T10:57:45.851 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T10:57:45.901 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T10:57:45.936 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T10:57:45.979 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T10:57:46.021 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T10:57:46.158 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T10:57:46.206 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T10:57:46.264 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T10:57:46.362 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T10:57:46.397 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T10:57:46.437 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T10:57:46.463 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T10:57:46.488 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T10:57:46.562 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T10:57:46.689 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T10:57:46.728 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T10:57:46.756 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T10:57:46.791 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T10:57:46.830 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T10:57:46.870 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T10:57:46.900 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T10:57:46.926 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T10:57:46.959 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T10:57:47.001 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T10:57:47.035 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T10:57:47.122 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T10:57:47.165 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T10:57:47.196 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T10:57:47.299 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T10:57:47.356 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T10:57:48.091 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T10:57:48.145 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T10:57:48.178 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T10:57:48.279 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T10:57:48.342 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T10:57:48.387 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T10:57:48.475 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T10:57:48.514 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T10:57:48.589 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-07-26T10:57:48.591 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-07-26T10:57:48.692 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T10:57:48.721 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T10:57:48.799 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T10:57:49.382 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-07-26T10:57:49.446 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T10:57:49.453 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T10:57:49.500 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T10:57:49.552 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T10:57:49.553 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T10:57:49.588 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T10:57:49.613 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T10:57:49.658 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T10:57:49.989 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T10:57:50.141 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:57:50.351 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:57:50.384 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:57:50.472 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:57:50.496 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T10:57:50.529 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T10:57:50.730 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T10:57:50.786 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T10:57:50.791 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T10:57:50.815 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:57:50.891 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T10:57:50.912 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:57:50.919 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T10:57:50.941 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T10:57:50.952 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T10:57:50.995 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T10:57:51.017 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T10:57:51.037 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T10:57:51.184 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T10:57:51.284 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T10:57:51.295 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T10:57:51.340 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T10:57:51.435 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T10:57:51.453 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T10:57:51.462 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T10:57:51.497 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T10:57:51.650 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T10:57:51.706 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T10:57:51.813 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:57:51.870 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:57:52.006 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T10:57:52.036 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T10:57:52.075 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:57:52.108 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T10:57:52.142 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T10:57:52.187 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T10:57:52.233 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T10:57:52.285 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T10:57:52.367 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T10:57:52.411 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T10:57:52.452 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T10:57:52.496 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T10:57:52.528 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T10:57:52.556 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T10:57:52.625 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T10:57:52.677 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T10:57:52.708 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T10:57:52.743 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T10:57:52.820 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T10:57:52.855 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T10:57:52.909 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T10:57:52.944 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T10:57:53.013 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T10:57:53.066 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T10:57:53.140 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T10:57:53.273 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T10:57:53.328 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T10:57:53.845 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T10:57:53.924 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T10:57:53.953 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T10:57:53.989 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T10:57:54.037 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T10:57:54.076 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T10:57:54.118 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T10:57:54.278 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T10:57:54.337 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T10:57:54.363 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:57:54.397 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:57:54.427 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T10:57:54.493 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T10:57:54.567 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T10:57:54.605 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T10:57:54.642 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T10:57:54.697 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T10:57:54.733 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T10:57:54.785 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T10:57:54.835 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T10:57:54.959 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:57:54.972 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T10:57:55.001 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T10:57:55.012 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T10:57:55.053 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T10:57:55.091 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T10:57:55.091 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T10:57:55.091 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T10:57:55.092 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T10:57:55.120 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T10:57:55.150 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T10:57:55.212 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T10:57:55.293 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-07-26T10:57:55.377 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T10:57:55.395 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : openssl-libs-1:3.2.2-2.el9.x86_64 1/172 2024-07-26T10:57:55.470 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/172 2024-07-26T10:57:55.559 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T10:57:55.575 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/172 2024-07-26T10:57:55.629 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/172 2024-07-26T10:57:55.661 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T10:57:55.664 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/172 2024-07-26T10:57:55.737 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T10:57:55.762 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/172 2024-07-26T10:57:55.863 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T10:57:55.901 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T10:57:55.929 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T10:57:55.964 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T10:57:56.003 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T10:57:56.034 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T10:57:56.064 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T10:57:56.115 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T10:57:56.148 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T10:57:56.186 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/172 2024-07-26T10:57:56.190 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T10:57:56.233 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T10:57:56.289 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:57:56.321 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T10:57:56.372 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T10:57:56.403 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T10:57:56.502 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:57:56.507 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T10:57:56.540 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:57:56.562 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T10:57:56.677 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:57:56.706 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T10:57:56.771 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/172 2024-07-26T10:57:57.084 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/172 2024-07-26T10:57:57.144 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/172 2024-07-26T10:57:57.170 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:57:57.294 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T10:57:57.325 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:57:57.343 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T10:57:57.355 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T10:57:57.376 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T10:57:57.439 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 15/172 2024-07-26T10:57:57.476 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T10:57:57.490 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T10:57:57.532 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T10:57:57.555 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/172 2024-07-26T10:57:57.576 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T10:57:57.664 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T10:57:57.704 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T10:57:57.860 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T10:57:57.882 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T10:57:57.919 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T10:57:57.998 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T10:57:58.033 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/172 2024-07-26T10:57:58.421 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : xfsprogs-6.4.0-3.el9.x86_64 18/172 2024-07-26T10:57:58.598 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T10:57:58.645 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T10:57:58.687 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 19/172 2024-07-26T10:57:58.767 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T10:57:58.878 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:57:58.968 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:57:59.294 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: NetworkManager-1:1.48.2-2.el9.x86_64 20/172 2024-07-26T10:57:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/172 2024-07-26T10:57:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/172 2024-07-26T10:57:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/172 2024-07-26T10:57:59.642 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/172 2024-07-26T10:57:59.698 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/172 2024-07-26T10:57:59.744 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-packaging-20.9-5.el9.noarch 26/172 2024-07-26T10:57:59.832 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/172 2024-07-26T10:57:59.875 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/172 2024-07-26T10:57:59.914 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/172 2024-07-26T10:57:59.999 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T10:58:00.030 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/172 2024-07-26T10:58:00.082 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T10:58:00.095 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 31/172 2024-07-26T10:58:00.109 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T10:58:00.133 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/172 2024-07-26T10:58:00.142 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T10:58:00.198 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/172 2024-07-26T10:58:00.310 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/172 2024-07-26T10:58:00.364 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 35/172 2024-07-26T10:58:00.442 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 36/172 2024-07-26T10:58:00.472 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T10:58:00.478 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 37/172 2024-07-26T10:58:00.514 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T10:58:00.599 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/172 2024-07-26T10:58:00.608 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T10:58:00.636 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T10:58:00.658 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/172 2024-07-26T10:58:00.671 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T10:58:00.765 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/172 2024-07-26T10:58:00.940 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/172 2024-07-26T10:58:01.008 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/172 2024-07-26T10:58:01.174 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T10:58:01.201 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T10:58:01.273 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T10:58:01.302 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T10:58:01.684 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/172 2024-07-26T10:58:01.781 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/172 2024-07-26T10:58:01.825 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/172 2024-07-26T10:58:01.863 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/172 2024-07-26T10:58:01.919 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/172 2024-07-26T10:58:01.965 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 48/172 2024-07-26T10:58:02.035 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/172 2024-07-26T10:58:02.266 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 50/172 2024-07-26T10:58:02.336 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : openssl-1:3.2.2-2.el9.x86_64 51/172 2024-07-26T10:58:02.452 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/172 2024-07-26T10:58:02.521 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 53/172 2024-07-26T10:58:02.607 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/172 2024-07-26T10:58:02.675 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/172 2024-07-26T10:58:02.719 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/172 2024-07-26T10:58:02.785 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/172 2024-07-26T10:58:02.827 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/172 2024-07-26T10:58:02.889 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 59/172 2024-07-26T10:58:02.948 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 60/172 2024-07-26T10:58:03.143 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 61/172 2024-07-26T10:58:03.186 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 62/172 2024-07-26T10:58:03.256 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 63/172 2024-07-26T10:58:03.389 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 64/172 2024-07-26T10:58:03.428 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 65/172 2024-07-26T10:58:03.481 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 66/172 2024-07-26T10:58:03.527 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 67/172 2024-07-26T10:58:03.560 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 68/172 2024-07-26T10:58:03.573 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:58:03.610 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:58:03.743 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 69/172 2024-07-26T10:58:03.951 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 70/172 2024-07-26T10:58:04.000 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 71/172 2024-07-26T10:58:04.037 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 72/172 2024-07-26T10:58:04.074 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/172 2024-07-26T10:58:04.131 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/172 2024-07-26T10:58:04.177 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:58:04.185 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/172 2024-07-26T10:58:04.213 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T10:58:04.227 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/172 2024-07-26T10:58:04.273 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/172 2024-07-26T10:58:04.288 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T10:58:04.288 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T10:58:04.288 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T10:58:04.288 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-07-26T10:58:04.314 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T10:58:04.324 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 78/172 2024-07-26T10:58:04.399 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 79/172 2024-07-26T10:58:04.456 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 80/172 2024-07-26T10:58:04.597 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 81/172 2024-07-26T10:58:04.662 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 82/172 2024-07-26T10:58:04.705 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 83/172 2024-07-26T10:58:04.857 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/172 2024-07-26T10:58:04.926 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T10:58:05.843 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/172 2024-07-26T10:58:05.900 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/172 2024-07-26T10:58:05.934 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/172 2024-07-26T10:58:06.055 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/172 2024-07-26T10:58:06.139 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/172 2024-07-26T10:58:06.198 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/172 2024-07-26T10:58:06.309 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/172 2024-07-26T10:58:06.344 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/172 2024-07-26T10:58:06.584 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/172 2024-07-26T10:58:06.616 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/172 2024-07-26T10:58:06.716 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/172 2024-07-26T10:58:06.910 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T10:58:06.910 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /sys 2024-07-26T10:58:06.910 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /proc 2024-07-26T10:58:06.910 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /mnt 2024-07-26T10:58:06.910 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /var/tmp 2024-07-26T10:58:06.910 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /home 2024-07-26T10:58:06.910 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /root 2024-07-26T10:58:06.910 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /tmp 2024-07-26T10:58:06.910 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T10:58:07.087 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T10:58:07.103 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T10:58:07.103 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:07.103 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T10:58:07.103 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T10:58:07.529 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/172 2024-07-26T10:58:07.672 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/172 2024-07-26T10:58:08.690 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T10:58:08.715 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T10:58:08.715 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:08.716 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T10:58:08.716 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T10:58:08.716 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T10:58:08.716 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T10:58:08.949 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T10:58:08.972 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T10:58:08.972 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:08.972 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T10:58:08.972 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T10:58:08.972 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T10:58:08.972 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T10:58:09.244 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/172 2024-07-26T10:58:09.340 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/172 2024-07-26T10:58:09.375 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/172 2024-07-26T10:58:09.411 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/172 2024-07-26T10:58:09.433 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T10:58:09.455 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T10:58:09.455 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:09.456 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T10:58:09.456 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T10:58:09.456 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T10:58:09.456 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T10:58:09.572 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T10:58:09.637 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T10:58:09.692 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T10:58:09.795 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/172 2024-07-26T10:58:09.836 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T10:58:09.878 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T10:58:09.926 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T10:58:09.978 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 104/172 2024-07-26T10:58:09.995 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T10:58:10.010 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 105/172 2024-07-26T10:58:10.075 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/172 2024-07-26T10:58:10.083 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T10:58:10.238 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T10:58:10.276 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T10:58:10.316 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T10:58:10.344 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T10:58:10.636 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/172 2024-07-26T10:58:10.673 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T10:58:10.790 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 108/172 2024-07-26T10:58:10.797 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T10:58:10.816 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 109/172 2024-07-26T10:58:10.826 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T10:58:10.859 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T10:58:10.922 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T10:58:10.966 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T10:58:13.456 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:58:13.496 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:58:14.121 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T10:58:14.159 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 110/172 2024-07-26T10:58:14.189 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T10:58:14.208 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T10:58:14.367 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 111/172 2024-07-26T10:58:14.367 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-07-26T10:58:14.367 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-07-26T10:58:14.368 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:58:14.401 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T10:58:15.148 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T10:58:15.173 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T10:58:15.284 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T10:58:15.465 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T10:58:15.491 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T10:58:15.514 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T10:58:15.514 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:15.514 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T10:58:15.514 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T10:58:15.514 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T10:58:15.514 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T10:58:15.559 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T10:58:15.570 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T10:58:15.669 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T10:58:15.716 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T10:58:15.986 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T10:58:15.986 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /sys 2024-07-26T10:58:15.986 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /proc 2024-07-26T10:58:15.986 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /mnt 2024-07-26T10:58:15.986 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /var/tmp 2024-07-26T10:58:15.986 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /home 2024-07-26T10:58:15.986 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /root 2024-07-26T10:58:15.986 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /tmp 2024-07-26T10:58:15.986 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-07-26T10:58:16.161 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T10:58:16.176 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T10:58:16.176 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:16.176 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T10:58:16.176 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-07-26T10:58:16.676 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T10:58:16.703 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T10:58:16.726 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T10:58:16.726 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:16.726 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T10:58:16.726 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T10:58:16.726 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T10:58:16.726 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T10:58:16.771 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T10:58:16.792 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T10:58:16.793 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:16.793 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T10:58:16.793 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T10:58:16.838 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T10:58:16.860 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T10:58:16.860 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:16.860 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T10:58:16.860 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T10:58:16.861 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T10:58:16.861 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T10:58:17.181 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T10:58:17.203 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T10:58:17.203 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:17.203 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T10:58:17.203 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T10:58:17.203 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T10:58:17.204 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T10:58:17.747 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T10:58:17.772 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T10:58:17.772 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:17.772 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T10:58:17.772 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T10:58:17.773 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T10:58:17.773 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-07-26T10:58:17.997 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T10:58:18.020 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T10:58:18.020 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:18.020 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T10:58:18.020 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T10:58:18.020 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T10:58:18.020 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-07-26T10:58:18.481 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T10:58:18.504 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T10:58:18.504 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:18.504 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T10:58:18.504 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T10:58:18.504 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T10:58:18.504 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-07-26T10:58:18.611 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T10:58:18.668 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T10:58:18.723 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T10:58:18.917 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T10:58:18.965 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T10:58:19.043 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T10:58:19.131 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T10:58:19.269 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T10:58:19.315 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T10:58:19.347 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T10:58:19.375 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T10:58:19.794 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T10:58:19.824 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T10:58:19.855 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T10:58:19.919 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T10:58:19.946 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T10:58:21.578 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T10:58:21.627 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T10:58:21.668 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T10:58:21.700 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T10:58:21.727 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T10:58:21.913 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T10:58:21.970 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T10:58:22.003 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T10:58:22.052 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T10:58:22.090 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T10:58:22.137 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T10:58:22.759 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:58:22.803 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:58:22.823 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:58:23.093 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T10:58:23.153 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T10:58:23.648 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T10:58:23.757 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T10:58:23.798 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T10:58:23.880 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T10:58:23.911 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T10:58:23.936 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T10:58:23.961 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T10:58:23.988 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T10:58:23.988 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T10:58:24.081 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T10:58:24.111 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T10:58:24.122 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T10:58:24.122 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T10:58:24.136 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T10:58:24.207 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T10:58:24.233 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T10:58:24.238 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:58:24.238 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:58:24.249 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T10:58:24.419 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:58:24.419 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T10:58:24.432 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T10:58:24.463 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T10:58:24.486 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T10:58:24.486 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:24.487 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T10:58:24.487 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T10:58:24.487 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T10:58:24.487 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-07-26T10:58:24.503 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T10:58:24.528 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T10:58:24.530 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T10:58:24.542 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T10:58:24.554 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T10:58:24.619 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:58:24.641 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T10:58:24.703 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T10:58:25.661 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T10:58:25.691 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T10:58:25.714 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T10:58:25.715 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:25.715 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T10:58:25.715 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T10:58:25.715 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T10:58:25.715 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-07-26T10:58:25.759 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T10:58:25.781 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T10:58:25.782 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:25.782 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T10:58:25.782 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-07-26T10:58:25.835 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T10:58:25.857 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T10:58:25.858 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:25.858 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T10:58:25.858 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T10:58:25.858 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T10:58:25.858 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-07-26T10:58:26.185 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T10:58:26.208 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T10:58:26.209 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:26.209 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T10:58:26.209 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T10:58:26.209 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T10:58:26.209 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-07-26T10:58:27.048 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 112/172 2024-07-26T10:58:27.049 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2024-07-26T10:58:27.049 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2024-07-26T10:58:27.049 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2024-07-26T10:58:27.049 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2024-07-26T10:58:27.049 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2024-07-26T10:58:27.049 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /root 2024-07-26T10:58:27.049 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2024-07-26T10:58:27.049 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:58:27.303 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T10:58:27.349 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 113/172 2024-07-26T10:58:27.349 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:27.349 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-07-26T10:58:27.349 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:58:29.055 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T10:58:29.131 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 114/172 2024-07-26T10:58:29.131 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:29.131 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-07-26T10:58:29.131 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T10:58:29.131 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-07-26T10:58:29.131 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:58:29.388 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T10:58:29.457 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 115/172 2024-07-26T10:58:29.457 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:29.457 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-07-26T10:58:29.457 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T10:58:29.457 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-07-26T10:58:29.457 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:58:29.973 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T10:58:30.039 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 116/172 2024-07-26T10:58:30.039 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:30.039 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-07-26T10:58:30.039 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T10:58:30.040 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-07-26T10:58:30.040 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:58:30.172 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : automake-1.16.2-8.el9.noarch 117/172 2024-07-26T10:58:30.233 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/172 2024-07-26T10:58:30.309 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/172 2024-07-26T10:58:30.554 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/172 2024-07-26T10:58:30.565 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T10:58:30.605 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T10:58:30.645 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T10:58:30.647 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/172 2024-07-26T10:58:30.685 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T10:58:30.712 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T10:58:30.739 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/172 2024-07-26T10:58:30.853 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/172 2024-07-26T10:58:30.896 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T10:58:30.946 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T10:58:30.980 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:58:30.980 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T10:58:30.980 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T10:58:30.980 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T10:58:30.980 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T10:58:30.980 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T10:58:30.980 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T10:58:30.980 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T10:58:30.980 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:58:30.980 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:58:30.980 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T10:58:30.980 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T10:58:30.981 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T10:58:30.981 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T10:58:30.981 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:58:30.981 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T10:58:30.981 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T10:58:30.981 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T10:58:30.981 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T10:58:30.981 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T10:58:30.981 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T10:58:30.981 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T10:58:30.981 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T10:58:30.981 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T10:58:30.981 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T10:58:30.982 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T10:58:30.982 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T10:58:30.982 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T10:58:30.982 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T10:58:30.982 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T10:58:30.982 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T10:58:30.983 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T10:58:30.995 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T10:58:30.995 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T10:58:30.995 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T10:58:30.995 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T10:58:30.996 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T10:58:30.996 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T10:58:30.996 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T10:58:30.996 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T10:58:30.996 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T10:58:30.996 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T10:58:30.996 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T10:58:30.996 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T10:58:30.996 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T10:58:30.996 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T10:58:30.996 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T10:58:30.996 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T10:58:30.997 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T10:58:30.997 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T10:58:30.997 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T10:58:30.997 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T10:58:30.997 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T10:58:30.997 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T10:58:30.997 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T10:58:30.997 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T10:58:30.997 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T10:58:30.997 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T10:58:30.997 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T10:58:30.997 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T10:58:30.997 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T10:58:30.998 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T10:58:30.998 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T10:58:30.998 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T10:58:30.998 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T10:58:30.998 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T10:58:30.998 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T10:58:30.998 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T10:58:30.998 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T10:58:30.998 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T10:58:30.998 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T10:58:30.998 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T10:58:30.998 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T10:58:30.999 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T10:58:30.999 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T10:58:30.999 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T10:58:30.999 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T10:58:30.999 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T10:58:30.999 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T10:58:30.999 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T10:58:30.999 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T10:58:30.999 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T10:58:30.999 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T10:58:30.999 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T10:58:30.999 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T10:58:31.000 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T10:58:31.000 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T10:58:31.000 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T10:58:31.000 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T10:58:31.000 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T10:58:31.000 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T10:58:31.000 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T10:58:31.000 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T10:58:31.000 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T10:58:31.000 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T10:58:31.000 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T10:58:31.000 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T10:58:31.001 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T10:58:31.001 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T10:58:31.001 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T10:58:31.001 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T10:58:31.001 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T10:58:31.001 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T10:58:31.001 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T10:58:31.001 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T10:58:31.001 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T10:58:31.001 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T10:58:31.001 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T10:58:31.001 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T10:58:31.001 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T10:58:31.001 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T10:58:31.002 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T10:58:31.002 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T10:58:31.002 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T10:58:31.002 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T10:58:31.002 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T10:58:31.002 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T10:58:31.002 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T10:58:31.002 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T10:58:31.002 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T10:58:31.002 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T10:58:31.002 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T10:58:31.002 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T10:58:31.002 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T10:58:31.003 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T10:58:31.003 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T10:58:31.003 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T10:58:31.003 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T10:58:31.003 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T10:58:31.003 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T10:58:31.003 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T10:58:31.003 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T10:58:31.003 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T10:58:31.003 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T10:58:31.003 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T10:58:31.003 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T10:58:31.004 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T10:58:31.004 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T10:58:31.004 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T10:58:31.004 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T10:58:31.004 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T10:58:31.004 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T10:58:31.004 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T10:58:31.004 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T10:58:31.004 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T10:58:31.004 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T10:58:31.004 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T10:58:31.004 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T10:58:31.004 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T10:58:31.004 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T10:58:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T10:58:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T10:58:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T10:58:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T10:58:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T10:58:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T10:58:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T10:58:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T10:58:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T10:58:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T10:58:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T10:58:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T10:58:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T10:58:31.005 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T10:58:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T10:58:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T10:58:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T10:58:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T10:58:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T10:58:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T10:58:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T10:58:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T10:58:31.006 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T10:58:31.011 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T10:58:31.019 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/172 2024-07-26T10:58:31.040 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T10:58:31.067 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/172 2024-07-26T10:58:31.095 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T10:58:31.115 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/172 2024-07-26T10:58:31.149 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.9-16.el9.x86_64 127/172 2024-07-26T10:58:31.692 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:58:31.736 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:58:31.756 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:58:31.757 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T10:58:31.810 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 128/172 2024-07-26T10:58:31.868 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 129/172 2024-07-26T10:58:31.980 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 130/172 2024-07-26T10:58:32.021 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T10:58:32.602 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T10:58:32.708 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T10:58:32.749 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T10:58:32.831 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T10:58:32.862 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T10:58:32.887 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T10:58:32.912 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T10:58:32.947 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T10:58:32.947 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T10:58:33.040 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T10:58:33.073 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T10:58:33.073 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T10:58:33.157 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T10:58:33.182 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T10:58:33.187 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:58:33.187 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:58:33.369 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:58:33.370 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T10:58:33.437 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T10:58:33.462 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T10:58:33.496 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T10:58:33.576 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:58:33.689 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:58:33.689 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T10:58:33.689 INFO:teuthology.orchestra.run.smithi193.stdout:Upgraded: 2024-07-26T10:58:33.689 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T10:58:33.690 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T10:58:33.690 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T10:58:33.690 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T10:58:33.690 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T10:58:33.690 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T10:58:33.690 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T10:58:33.690 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.690 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.690 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T10:58:33.690 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T10:58:33.690 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T10:58:33.691 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-07-26T10:58:33.691 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T10:58:33.691 INFO:teuthology.orchestra.run.smithi193.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T10:58:33.691 INFO:teuthology.orchestra.run.smithi193.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T10:58:33.691 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T10:58:33.691 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.691 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.691 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.691 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.691 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:33.691 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.691 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.691 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.692 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:33.692 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:33.692 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:33.692 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:33.692 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:33.692 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.692 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.692 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:33.692 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.692 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.692 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.692 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:33.692 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:33.692 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.693 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.693 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:33.693 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T10:58:33.693 INFO:teuthology.orchestra.run.smithi193.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T10:58:33.693 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T10:58:33.693 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T10:58:33.693 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T10:58:33.693 INFO:teuthology.orchestra.run.smithi193.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T10:58:33.693 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T10:58:33.693 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T10:58:33.693 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T10:58:33.693 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T10:58:33.694 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T10:58:33.694 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.694 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.694 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.694 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T10:58:33.694 INFO:teuthology.orchestra.run.smithi193.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T10:58:33.694 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T10:58:33.694 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T10:58:33.694 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T10:58:33.694 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.694 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.694 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T10:58:33.695 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.695 INFO:teuthology.orchestra.run.smithi193.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T10:58:33.695 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T10:58:33.695 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T10:58:33.695 INFO:teuthology.orchestra.run.smithi193.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T10:58:33.695 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T10:58:33.695 INFO:teuthology.orchestra.run.smithi193.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T10:58:33.695 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T10:58:33.695 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T10:58:33.695 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T10:58:33.695 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T10:58:33.695 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T10:58:33.696 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T10:58:33.696 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T10:58:33.696 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T10:58:33.696 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T10:58:33.696 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T10:58:33.696 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T10:58:33.696 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T10:58:33.696 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T10:58:33.696 INFO:teuthology.orchestra.run.smithi193.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T10:58:33.696 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T10:58:33.696 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T10:58:33.696 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T10:58:33.696 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.696 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.697 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.697 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T10:58:33.697 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T10:58:33.697 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T10:58:33.697 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T10:58:33.697 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T10:58:33.697 INFO:teuthology.orchestra.run.smithi193.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T10:58:33.697 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T10:58:33.697 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T10:58:33.697 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T10:58:33.697 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T10:58:33.697 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T10:58:33.697 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T10:58:33.698 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T10:58:33.698 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T10:58:33.698 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T10:58:33.698 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T10:58:33.698 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T10:58:33.698 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T10:58:33.698 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T10:58:33.698 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T10:58:33.698 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T10:58:33.698 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T10:58:33.698 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T10:58:33.698 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T10:58:33.699 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T10:58:33.699 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T10:58:33.699 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T10:58:33.699 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T10:58:33.699 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T10:58:33.699 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T10:58:33.699 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T10:58:33.699 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T10:58:33.699 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T10:58:33.699 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T10:58:33.699 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T10:58:33.699 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T10:58:33.699 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T10:58:33.699 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T10:58:33.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T10:58:33.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T10:58:33.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T10:58:33.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T10:58:33.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T10:58:33.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T10:58:33.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T10:58:33.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T10:58:33.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T10:58:33.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T10:58:33.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T10:58:33.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T10:58:33.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T10:58:33.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T10:58:33.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T10:58:33.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T10:58:33.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T10:58:33.701 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T10:58:33.701 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T10:58:33.701 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T10:58:33.701 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.701 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.701 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:33.701 INFO:teuthology.orchestra.run.smithi193.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T10:58:33.702 INFO:teuthology.orchestra.run.smithi193.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T10:58:33.702 INFO:teuthology.orchestra.run.smithi193.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T10:58:33.702 INFO:teuthology.orchestra.run.smithi193.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T10:58:33.702 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T10:58:33.702 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T10:58:33.702 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T10:58:33.702 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T10:58:33.702 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-07-26T10:58:34.146 DEBUG:teuthology.parallel:result is None 2024-07-26T10:58:35.687 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 131/172 2024-07-26T10:58:35.753 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T10:58:36.850 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 132/172 2024-07-26T10:58:36.877 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T10:58:37.036 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 133/172 2024-07-26T10:58:37.226 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 134/172 2024-07-26T10:58:37.254 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T10:58:37.322 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 135/172 2024-07-26T10:58:37.322 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:37.322 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-07-26T10:58:37.322 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T10:58:37.322 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-07-26T10:58:37.323 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:58:37.380 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T10:58:37.415 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 136/172 2024-07-26T10:58:37.563 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 137/172 2024-07-26T10:58:37.644 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 138/172 2024-07-26T10:58:38.713 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libtool-2.4.6-46.el9.x86_64 139/172 2024-07-26T10:58:38.749 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T10:58:38.818 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 140/172 2024-07-26T10:58:38.819 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:38.819 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-07-26T10:58:38.819 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T10:58:38.819 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-07-26T10:58:38.819 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:58:38.874 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T10:58:38.962 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 141/172 2024-07-26T10:58:38.962 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:38.962 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-07-26T10:58:38.962 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:58:39.033 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T10:58:39.129 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 142/172 2024-07-26T10:58:39.129 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:39.129 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-07-26T10:58:39.129 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T10:58:39.129 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-07-26T10:58:39.129 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:58:39.534 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T10:58:39.606 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 143/172 2024-07-26T10:58:39.606 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-07-26T10:58:39.606 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-07-26T10:58:39.606 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T10:58:39.607 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-07-26T10:58:39.607 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:58:40.129 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:58:40.129 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T10:58:40.129 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T10:58:40.129 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T10:58:40.129 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T10:58:40.129 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T10:58:40.130 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T10:58:40.130 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T10:58:40.130 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:58:40.130 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:58:40.130 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T10:58:40.130 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T10:58:40.130 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T10:58:40.130 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T10:58:40.130 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:58:40.130 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T10:58:40.130 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T10:58:40.130 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T10:58:40.131 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T10:58:40.131 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T10:58:40.131 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T10:58:40.131 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T10:58:40.131 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T10:58:40.131 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T10:58:40.131 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T10:58:40.131 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T10:58:40.131 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T10:58:40.131 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T10:58:40.131 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T10:58:40.131 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T10:58:40.131 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T10:58:40.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T10:58:40.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T10:58:40.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T10:58:40.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T10:58:40.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T10:58:40.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T10:58:40.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T10:58:40.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T10:58:40.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T10:58:40.144 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T10:58:40.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T10:58:40.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T10:58:40.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T10:58:40.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T10:58:40.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T10:58:40.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T10:58:40.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T10:58:40.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T10:58:40.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T10:58:40.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T10:58:40.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T10:58:40.145 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T10:58:40.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T10:58:40.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T10:58:40.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T10:58:40.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T10:58:40.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T10:58:40.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T10:58:40.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T10:58:40.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T10:58:40.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T10:58:40.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T10:58:40.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T10:58:40.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T10:58:40.146 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T10:58:40.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T10:58:40.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T10:58:40.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T10:58:40.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T10:58:40.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T10:58:40.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T10:58:40.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T10:58:40.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T10:58:40.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T10:58:40.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T10:58:40.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T10:58:40.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T10:58:40.147 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T10:58:40.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T10:58:40.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T10:58:40.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T10:58:40.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T10:58:40.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T10:58:40.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T10:58:40.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T10:58:40.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T10:58:40.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T10:58:40.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T10:58:40.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T10:58:40.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T10:58:40.148 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T10:58:40.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T10:58:40.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T10:58:40.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T10:58:40.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T10:58:40.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T10:58:40.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T10:58:40.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T10:58:40.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T10:58:40.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T10:58:40.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T10:58:40.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T10:58:40.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T10:58:40.149 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T10:58:40.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T10:58:40.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T10:58:40.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T10:58:40.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T10:58:40.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T10:58:40.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T10:58:40.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T10:58:40.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T10:58:40.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T10:58:40.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T10:58:40.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T10:58:40.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T10:58:40.150 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T10:58:40.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T10:58:40.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T10:58:40.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T10:58:40.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T10:58:40.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T10:58:40.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T10:58:40.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T10:58:40.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T10:58:40.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T10:58:40.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T10:58:40.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T10:58:40.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T10:58:40.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T10:58:40.151 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T10:58:40.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T10:58:40.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T10:58:40.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T10:58:40.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T10:58:40.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T10:58:40.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T10:58:40.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T10:58:40.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T10:58:40.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T10:58:40.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T10:58:40.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T10:58:40.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T10:58:40.152 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T10:58:40.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T10:58:40.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T10:58:40.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T10:58:40.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T10:58:40.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T10:58:40.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T10:58:40.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T10:58:40.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T10:58:40.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T10:58:40.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T10:58:40.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T10:58:40.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T10:58:40.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T10:58:40.153 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T10:58:40.154 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T10:58:40.154 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T10:58:40.154 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T10:58:40.154 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T10:58:40.154 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T10:58:40.154 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T10:58:40.154 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T10:58:40.154 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T10:58:40.154 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T10:58:40.154 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T10:58:40.154 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T10:58:40.154 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T10:58:40.154 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T10:58:43.062 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:58:43.063 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-07-26T10:58:43.063 INFO:teuthology.orchestra.run.smithi189.stdout:Upgraded: 2024-07-26T10:58:43.063 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T10:58:43.063 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T10:58:43.063 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T10:58:43.063 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T10:58:43.063 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T10:58:43.063 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T10:58:43.063 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T10:58:43.063 INFO:teuthology.orchestra.run.smithi189.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.063 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.063 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T10:58:43.064 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T10:58:43.064 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T10:58:43.064 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-07-26T10:58:43.064 INFO:teuthology.orchestra.run.smithi189.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T10:58:43.064 INFO:teuthology.orchestra.run.smithi189.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T10:58:43.064 INFO:teuthology.orchestra.run.smithi189.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T10:58:43.064 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T10:58:43.064 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.064 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.064 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.064 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.064 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:43.064 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.064 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.065 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.065 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:43.065 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:43.065 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:43.065 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:43.065 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:43.065 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.065 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.065 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:43.065 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.065 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.065 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.065 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:43.066 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:43.066 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.066 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.066 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:43.066 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T10:58:43.066 INFO:teuthology.orchestra.run.smithi189.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T10:58:43.066 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T10:58:43.066 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T10:58:43.066 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T10:58:43.066 INFO:teuthology.orchestra.run.smithi189.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T10:58:43.066 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T10:58:43.066 INFO:teuthology.orchestra.run.smithi189.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T10:58:43.066 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T10:58:43.066 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T10:58:43.067 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T10:58:43.067 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.067 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.067 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.067 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T10:58:43.067 INFO:teuthology.orchestra.run.smithi189.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T10:58:43.067 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T10:58:43.067 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T10:58:43.067 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T10:58:43.067 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.067 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.067 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T10:58:43.067 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.067 INFO:teuthology.orchestra.run.smithi189.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T10:58:43.068 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T10:58:43.068 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T10:58:43.068 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T10:58:43.068 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T10:58:43.068 INFO:teuthology.orchestra.run.smithi189.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T10:58:43.068 INFO:teuthology.orchestra.run.smithi189.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T10:58:43.068 INFO:teuthology.orchestra.run.smithi189.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T10:58:43.068 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T10:58:43.068 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T10:58:43.068 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T10:58:43.068 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T10:58:43.068 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T10:58:43.068 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T10:58:43.068 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T10:58:43.069 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T10:58:43.069 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T10:58:43.069 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T10:58:43.069 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T10:58:43.069 INFO:teuthology.orchestra.run.smithi189.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T10:58:43.069 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T10:58:43.069 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T10:58:43.069 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T10:58:43.069 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.069 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.069 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.069 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T10:58:43.069 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T10:58:43.069 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T10:58:43.070 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T10:58:43.070 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T10:58:43.070 INFO:teuthology.orchestra.run.smithi189.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T10:58:43.070 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T10:58:43.070 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T10:58:43.070 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T10:58:43.070 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T10:58:43.070 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T10:58:43.070 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T10:58:43.070 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T10:58:43.070 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T10:58:43.070 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T10:58:43.070 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T10:58:43.070 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T10:58:43.071 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T10:58:43.071 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T10:58:43.071 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T10:58:43.071 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T10:58:43.071 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T10:58:43.071 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T10:58:43.071 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T10:58:43.071 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T10:58:43.071 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T10:58:43.071 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T10:58:43.071 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T10:58:43.071 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T10:58:43.071 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T10:58:43.071 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T10:58:43.071 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T10:58:43.072 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T10:58:43.072 INFO:teuthology.orchestra.run.smithi189.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T10:58:43.072 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T10:58:43.072 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T10:58:43.072 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T10:58:43.072 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T10:58:43.072 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T10:58:43.072 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T10:58:43.072 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.072 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.072 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T10:58:43.072 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T10:58:43.072 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T10:58:43.072 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.072 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T10:58:43.073 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T10:58:43.073 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T10:58:43.073 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T10:58:43.073 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T10:58:43.073 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T10:58:43.073 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T10:58:43.073 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T10:58:43.073 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T10:58:43.073 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T10:58:43.073 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T10:58:43.073 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T10:58:43.073 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T10:58:43.073 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T10:58:43.073 INFO:teuthology.orchestra.run.smithi189.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T10:58:43.073 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.073 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.074 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:43.074 INFO:teuthology.orchestra.run.smithi189.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T10:58:43.074 INFO:teuthology.orchestra.run.smithi189.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T10:58:43.074 INFO:teuthology.orchestra.run.smithi189.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T10:58:43.074 INFO:teuthology.orchestra.run.smithi189.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T10:58:43.074 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T10:58:43.074 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T10:58:43.074 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T10:58:43.074 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-07-26T10:58:43.075 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-07-26T10:58:43.530 DEBUG:teuthology.parallel:result is None 2024-07-26T10:58:44.374 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 144/172 2024-07-26T10:58:44.435 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 145/172 2024-07-26T10:58:44.505 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 146/172 2024-07-26T10:58:44.538 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xfsprogs-devel-6.4.0-3.el9.x86_64 147/172 2024-07-26T10:58:44.577 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/172 2024-07-26T10:58:44.880 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T10:58:44.927 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/172 2024-07-26T10:58:44.963 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 151/172 2024-07-26T10:58:45.045 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/172 2024-07-26T10:58:45.101 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : NetworkManager-team-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T10:58:45.151 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : NetworkManager-tui-1:1.48.2-2.el9.x86_64 154/172 2024-07-26T10:58:46.046 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:58:46.097 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:58:46.152 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/172 2024-07-26T10:58:47.523 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 156/172 2024-07-26T10:58:47.638 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openssl-devel-1:3.2.2-2.el9.x86_64 157/172 2024-07-26T10:58:47.698 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 158/172 2024-07-26T10:58:47.826 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flex-2.6.4-9.el9.x86_64 159/172 2024-07-26T10:58:47.864 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : bison-3.7.4-5.el9.x86_64 160/172 2024-07-26T10:58:47.923 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/172 2024-07-26T10:58:47.957 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T10:58:48.000 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 163/172 2024-07-26T10:58:48.000 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T10:58:48.172 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 164/172 2024-07-26T10:58:48.226 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 165/172 2024-07-26T10:58:48.226 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T10:58:48.363 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 166/172 2024-07-26T10:58:48.402 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 167/172 2024-07-26T10:58:48.442 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:58:48.443 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:58:48.760 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 168/172 2024-07-26T10:58:48.760 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T10:58:48.898 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 169/172 2024-07-26T10:58:48.942 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T10:58:48.989 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 171/172 2024-07-26T10:58:49.133 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:58:49.370 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:58:49.370 INFO:teuthology.orchestra.run.smithi134.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-07-26T10:58:49.370 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:58:55.820 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 1/172 2024-07-26T10:58:55.820 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2/172 2024-07-26T10:58:55.820 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 3/172 2024-07-26T10:58:55.821 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 4/172 2024-07-26T10:58:55.821 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5320.ge8f88 5/172 2024-07-26T10:58:55.821 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 6/172 2024-07-26T10:58:55.821 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 7/172 2024-07-26T10:58:55.821 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 8/172 2024-07-26T10:58:55.821 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 9/172 2024-07-26T10:58:55.821 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 10/172 2024-07-26T10:58:55.821 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 11/172 2024-07-26T10:58:55.821 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 12/172 2024-07-26T10:58:55.821 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 13/172 2024-07-26T10:58:55.821 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 14/172 2024-07-26T10:58:55.821 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_ 15/172 2024-07-26T10:58:55.821 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 16/172 2024-07-26T10:58:55.822 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 17/172 2024-07-26T10:58:55.822 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_6 18/172 2024-07-26T10:58:55.822 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86 19/172 2024-07-26T10:58:55.822 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 20/172 2024-07-26T10:58:55.822 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el 21/172 2024-07-26T10:58:55.822 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9. 22/172 2024-07-26T10:58:55.822 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_6 23/172 2024-07-26T10:58:55.822 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 24/172 2024-07-26T10:58:55.822 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 25/172 2024-07-26T10:58:55.822 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 26/172 2024-07-26T10:58:55.822 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 27/172 2024-07-26T10:58:55.822 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 28/172 2024-07-26T10:58:55.835 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 29/172 2024-07-26T10:58:55.835 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1. 30/172 2024-07-26T10:58:55.835 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noa 31/172 2024-07-26T10:58:55.835 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.n 32/172 2024-07-26T10:58:55.835 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f 33/172 2024-07-26T10:58:55.836 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el 34/172 2024-07-26T10:58:55.836 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 35/172 2024-07-26T10:58:55.836 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.e 36/172 2024-07-26T10:58:55.836 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 37/172 2024-07-26T10:58:55.836 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 38/172 2024-07-26T10:58:55.836 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 39/172 2024-07-26T10:58:55.836 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 40/172 2024-07-26T10:58:55.836 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/172 2024-07-26T10:58:55.836 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/172 2024-07-26T10:58:55.836 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/172 2024-07-26T10:58:55.836 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/172 2024-07-26T10:58:55.836 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/172 2024-07-26T10:58:55.836 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/172 2024-07-26T10:58:55.837 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/172 2024-07-26T10:58:55.837 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 48/172 2024-07-26T10:58:55.837 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/172 2024-07-26T10:58:55.837 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/172 2024-07-26T10:58:55.837 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/172 2024-07-26T10:58:55.837 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/172 2024-07-26T10:58:55.837 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/172 2024-07-26T10:58:55.837 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/172 2024-07-26T10:58:55.837 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/172 2024-07-26T10:58:55.837 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/172 2024-07-26T10:58:55.837 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/172 2024-07-26T10:58:55.837 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/172 2024-07-26T10:58:55.837 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/172 2024-07-26T10:58:55.838 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/172 2024-07-26T10:58:55.838 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 61/172 2024-07-26T10:58:55.838 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/172 2024-07-26T10:58:55.838 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/172 2024-07-26T10:58:55.838 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/172 2024-07-26T10:58:55.838 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/172 2024-07-26T10:58:55.838 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/172 2024-07-26T10:58:55.838 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/172 2024-07-26T10:58:55.838 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openssl-devel-1:3.2.2-2.el9.x86_64 68/172 2024-07-26T10:58:55.838 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/172 2024-07-26T10:58:55.838 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/172 2024-07-26T10:58:55.838 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/172 2024-07-26T10:58:55.839 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/172 2024-07-26T10:58:55.839 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/172 2024-07-26T10:58:55.839 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/172 2024-07-26T10:58:55.839 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/172 2024-07-26T10:58:55.839 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/172 2024-07-26T10:58:55.839 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/172 2024-07-26T10:58:55.839 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/172 2024-07-26T10:58:55.839 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/172 2024-07-26T10:58:55.839 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/172 2024-07-26T10:58:55.839 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/172 2024-07-26T10:58:55.839 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/172 2024-07-26T10:58:55.839 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/172 2024-07-26T10:58:55.839 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/172 2024-07-26T10:58:55.840 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/172 2024-07-26T10:58:55.840 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/172 2024-07-26T10:58:55.840 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/172 2024-07-26T10:58:55.840 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/172 2024-07-26T10:58:55.840 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/172 2024-07-26T10:58:55.840 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/172 2024-07-26T10:58:55.840 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/172 2024-07-26T10:58:55.840 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/172 2024-07-26T10:58:55.840 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/172 2024-07-26T10:58:55.840 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/172 2024-07-26T10:58:55.840 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/172 2024-07-26T10:58:55.840 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/172 2024-07-26T10:58:55.840 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/172 2024-07-26T10:58:55.841 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/172 2024-07-26T10:58:55.841 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 99/172 2024-07-26T10:58:55.841 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/172 2024-07-26T10:58:55.841 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xfsprogs-devel-6.4.0-3.el9.x86_64 101/172 2024-07-26T10:58:55.841 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/172 2024-07-26T10:58:55.841 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/172 2024-07-26T10:58:55.841 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/172 2024-07-26T10:58:55.841 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/172 2024-07-26T10:58:55.841 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/172 2024-07-26T10:58:55.841 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/172 2024-07-26T10:58:55.841 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/172 2024-07-26T10:58:55.841 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/172 2024-07-26T10:58:55.841 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/172 2024-07-26T10:58:55.842 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/172 2024-07-26T10:58:55.842 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/172 2024-07-26T10:58:55.842 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/172 2024-07-26T10:58:55.842 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/172 2024-07-26T10:58:55.842 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/172 2024-07-26T10:58:55.842 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/172 2024-07-26T10:58:55.842 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/172 2024-07-26T10:58:55.842 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 118/172 2024-07-26T10:58:55.842 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/172 2024-07-26T10:58:55.842 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/172 2024-07-26T10:58:55.842 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/172 2024-07-26T10:58:55.842 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 122/172 2024-07-26T10:58:55.842 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/172 2024-07-26T10:58:55.843 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/172 2024-07-26T10:58:55.843 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/172 2024-07-26T10:58:55.843 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/172 2024-07-26T10:58:55.843 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/172 2024-07-26T10:58:55.843 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/172 2024-07-26T10:58:55.843 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/172 2024-07-26T10:58:55.843 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/172 2024-07-26T10:58:55.843 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/172 2024-07-26T10:58:55.843 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/172 2024-07-26T10:58:55.843 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/172 2024-07-26T10:58:55.843 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/172 2024-07-26T10:58:55.843 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/172 2024-07-26T10:58:55.843 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/172 2024-07-26T10:58:55.844 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/172 2024-07-26T10:58:55.844 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/172 2024-07-26T10:58:55.844 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/172 2024-07-26T10:58:55.844 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/172 2024-07-26T10:58:55.844 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/172 2024-07-26T10:58:55.844 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/172 2024-07-26T10:58:55.844 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/172 2024-07-26T10:58:55.844 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/172 2024-07-26T10:58:55.844 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/172 2024-07-26T10:58:55.844 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/172 2024-07-26T10:58:55.844 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/172 2024-07-26T10:58:55.844 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/172 2024-07-26T10:58:55.844 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 149/172 2024-07-26T10:58:55.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/172 2024-07-26T10:58:55.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 151/172 2024-07-26T10:58:55.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/172 2024-07-26T10:58:55.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : NetworkManager-1:1.48.2-2.el9.x86_64 153/172 2024-07-26T10:58:55.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/172 2024-07-26T10:58:55.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : NetworkManager-libnm-1:1.48.2-2.el9.x86_64 155/172 2024-07-26T10:58:55.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/172 2024-07-26T10:58:55.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : NetworkManager-team-1:1.48.2-2.el9.x86_64 157/172 2024-07-26T10:58:55.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/172 2024-07-26T10:58:55.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : NetworkManager-tui-1:1.48.2-2.el9.x86_64 159/172 2024-07-26T10:58:55.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/172 2024-07-26T10:58:55.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/172 2024-07-26T10:58:55.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/172 2024-07-26T10:58:55.846 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/172 2024-07-26T10:58:55.846 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/172 2024-07-26T10:58:55.846 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/172 2024-07-26T10:58:55.846 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/172 2024-07-26T10:58:55.846 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openssl-1:3.2.2-2.el9.x86_64 167/172 2024-07-26T10:58:55.846 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/172 2024-07-26T10:58:55.846 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openssl-libs-1:3.2.2-2.el9.x86_64 169/172 2024-07-26T10:58:55.846 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/172 2024-07-26T10:58:55.846 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xfsprogs-6.4.0-3.el9.x86_64 171/172 2024-07-26T10:58:58.222 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 172/172 2024-07-26T10:58:58.222 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:58:58.222 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-07-26T10:58:58.223 INFO:teuthology.orchestra.run.smithi134.stdout: NetworkManager-1:1.48.2-2.el9.x86_64 2024-07-26T10:58:58.223 INFO:teuthology.orchestra.run.smithi134.stdout: NetworkManager-libnm-1:1.48.2-2.el9.x86_64 2024-07-26T10:58:58.223 INFO:teuthology.orchestra.run.smithi134.stdout: NetworkManager-team-1:1.48.2-2.el9.x86_64 2024-07-26T10:58:58.223 INFO:teuthology.orchestra.run.smithi134.stdout: NetworkManager-tui-1:1.48.2-2.el9.x86_64 2024-07-26T10:58:58.223 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-07-26T10:58:58.223 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-07-26T10:58:58.223 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-07-26T10:58:58.223 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.223 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.223 INFO:teuthology.orchestra.run.smithi134.stdout: openssl-1:3.2.2-2.el9.x86_64 2024-07-26T10:58:58.223 INFO:teuthology.orchestra.run.smithi134.stdout: openssl-libs-1:3.2.2-2.el9.x86_64 2024-07-26T10:58:58.223 INFO:teuthology.orchestra.run.smithi134.stdout: xfsprogs-6.4.0-3.el9.x86_64 2024-07-26T10:58:58.223 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-07-26T10:58:58.224 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-07-26T10:58:58.224 INFO:teuthology.orchestra.run.smithi134.stdout: automake-1.16.2-8.el9.noarch 2024-07-26T10:58:58.224 INFO:teuthology.orchestra.run.smithi134.stdout: bison-3.7.4-5.el9.x86_64 2024-07-26T10:58:58.224 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-07-26T10:58:58.224 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.224 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.224 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.224 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.224 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:58.224 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.224 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.224 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.224 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:58.225 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:58.225 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:58.225 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:58.225 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:58.225 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.225 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.225 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:58.225 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.225 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.225 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.225 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:58.225 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:58.225 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.226 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-shell-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.226 INFO:teuthology.orchestra.run.smithi134.stdout: cephfs-top-2:19.0.0-5320.ge8f88bb1.el9.noarch 2024-07-26T10:58:58.226 INFO:teuthology.orchestra.run.smithi134.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-07-26T10:58:58.226 INFO:teuthology.orchestra.run.smithi134.stdout: flex-2.6.4-9.el9.x86_64 2024-07-26T10:58:58.226 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-07-26T10:58:58.226 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-07-26T10:58:58.226 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-07-26T10:58:58.226 INFO:teuthology.orchestra.run.smithi134.stdout: fuse-2.9.9-16.el9.x86_64 2024-07-26T10:58:58.226 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-07-26T10:58:58.226 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data-1.46.7-10.el9.noarch 2024-07-26T10:58:58.226 INFO:teuthology.orchestra.run.smithi134.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-07-26T10:58:58.226 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-9.0.0-11.el9.x86_64 2024-07-26T10:58:58.226 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-07-26T10:58:58.226 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.227 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.227 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.227 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-07-26T10:58:58.227 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.7-2.el9.x86_64 2024-07-26T10:58:58.227 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-07-26T10:58:58.227 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-07-26T10:58:58.227 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-07-26T10:58:58.227 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.227 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.227 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-07-26T10:58:58.227 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.227 INFO:teuthology.orchestra.run.smithi134.stdout: libtool-2.4.6-46.el9.x86_64 2024-07-26T10:58:58.227 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.6.2-1.el9.x86_64 2024-07-26T10:58:58.228 INFO:teuthology.orchestra.run.smithi134.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-07-26T10:58:58.228 INFO:teuthology.orchestra.run.smithi134.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-07-26T10:58:58.228 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-07-26T10:58:58.228 INFO:teuthology.orchestra.run.smithi134.stdout: lua-5.4.4-4.el9.x86_64 2024-07-26T10:58:58.228 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-07-26T10:58:58.228 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks-3.9.2-5.el9.noarch 2024-07-26T10:58:58.228 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.21-2.el9.x86_64 2024-07-26T10:58:58.228 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-07-26T10:58:58.228 INFO:teuthology.orchestra.run.smithi134.stdout: openssl-devel-1:3.2.2-2.el9.x86_64 2024-07-26T10:58:58.228 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-07-26T10:58:58.228 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-13.14-1.el9.x86_64 2024-07-26T10:58:58.228 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-07-26T10:58:58.228 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-07-26T10:58:58.228 INFO:teuthology.orchestra.run.smithi134.stdout: postgresql-server-13.14-1.el9.x86_64 2024-07-26T10:58:58.229 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-3.14.0-13.el9.x86_64 2024-07-26T10:58:58.229 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-07-26T10:58:58.229 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-07-26T10:58:58.229 INFO:teuthology.orchestra.run.smithi134.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-07-26T10:58:58.229 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.9.1-2.el9.noarch 2024-07-26T10:58:58.229 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-07-26T10:58:58.229 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-07-26T10:58:58.229 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.229 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.229 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.229 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-07-26T10:58:58.229 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-07-26T10:58:58.229 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-07-26T10:58:58.229 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-07-26T10:58:58.229 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-07-26T10:58:58.230 INFO:teuthology.orchestra.run.smithi134.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-07-26T10:58:58.230 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-07-26T10:58:58.230 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-07-26T10:58:58.230 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-07-26T10:58:58.230 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-07-26T10:58:58.230 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-07-26T10:58:58.230 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-07-26T10:58:58.230 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-07-26T10:58:58.230 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-07-26T10:58:58.230 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-07-26T10:58:58.230 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-07-26T10:58:58.230 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-07-26T10:58:58.230 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-07-26T10:58:58.231 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-07-26T10:58:58.231 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-07-26T10:58:58.231 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-07-26T10:58:58.231 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.1.4-6.el9.noarch 2024-07-26T10:58:58.231 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-07-26T10:58:58.231 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-07-26T10:58:58.231 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-07-26T10:58:58.231 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-07-26T10:58:58.231 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-07-26T10:58:58.231 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-07-26T10:58:58.231 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging-20.9-5.el9.noarch 2024-07-26T10:58:58.231 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-07-26T10:58:58.231 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply-3.11-14.el9.noarch 2024-07-26T10:58:58.231 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-3.1.0-2.el9.noarch 2024-07-26T10:58:58.231 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-07-26T10:58:58.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-07-26T10:58:58.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-07-26T10:58:58.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-07-26T10:58:58.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-07-26T10:58:58.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.20-6.el9.noarch 2024-07-26T10:58:58.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-07-26T10:58:58.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2021.1-5.el9.noarch 2024-07-26T10:58:58.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-07-26T10:58:58.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-2.25.1-8.el9.noarch 2024-07-26T10:58:58.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-07-26T10:58:58.232 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.5.1-5.el9.noarch 2024-07-26T10:58:58.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el9.noarch 2024-07-26T10:58:58.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-07-26T10:58:58.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-07-26T10:58:58.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml-0.10.2-6.el9.noarch 2024-07-26T10:58:58.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-07-26T10:58:58.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-07-26T10:58:58.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-07-26T10:58:58.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.7-6.el9.noarch 2024-07-26T10:58:58.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-07-26T10:58:58.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-07-26T10:58:58.233 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-07-26T10:58:58.233 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-24.02.0-1.el9.x86_64 2024-07-26T10:58:58.233 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-07-26T10:58:58.234 INFO:teuthology.orchestra.run.smithi134.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-07-26T10:58:58.234 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.234 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.234 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:19.0.0-5320.ge8f88bb1.el9.x86_64 2024-07-26T10:58:58.234 INFO:teuthology.orchestra.run.smithi134.stdout: re2-1:20211101-3.el9.x86_64 2024-07-26T10:58:58.234 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-6.el9.x86_64 2024-07-26T10:58:58.234 INFO:teuthology.orchestra.run.smithi134.stdout: thrift-0.15.0-2.el9.x86_64 2024-07-26T10:58:58.234 INFO:teuthology.orchestra.run.smithi134.stdout: uuid-1.6.2-55.el9.x86_64 2024-07-26T10:58:58.234 INFO:teuthology.orchestra.run.smithi134.stdout: xfsprogs-devel-6.4.0-3.el9.x86_64 2024-07-26T10:58:58.234 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-07-26T10:58:58.235 INFO:teuthology.orchestra.run.smithi134.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-07-26T10:58:58.235 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T10:58:58.235 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-07-26T10:58:58.830 DEBUG:teuthology.parallel:result is None 2024-07-26T10:58:58.831 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:58:58.985 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T10:58:59.037 INFO:teuthology.orchestra.run.smithi134.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T10:58:59.037 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T10:58:59.037 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T10:58:59.039 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:58:59.185 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T10:58:59.212 INFO:teuthology.orchestra.run.smithi189.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T10:58:59.212 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T10:58:59.212 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T10:58:59.214 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:58:59.357 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-07-26T10:58:59.383 INFO:teuthology.orchestra.run.smithi193.stdout:19.0.0-5320.ge8f88bb1.el9 2024-07-26T10:58:59.383 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5320.ge8f88bb1.el9 2024-07-26T10:58:59.383 INFO:teuthology.task.install:The correct ceph version 19.0.0-5320.ge8f88bb1 is installed. 2024-07-26T10:58:59.385 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-07-26T10:58:59.385 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T10:58:59.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T10:58:59.452 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-07-26T10:58:59.452 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T10:58:59.484 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-07-26T10:58:59.484 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-07-26T10:58:59.515 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-07-26T10:58:59.515 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T10:58:59.515 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T10:58:59.587 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T10:58:59.692 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-07-26T10:58:59.692 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T10:58:59.720 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T10:58:59.788 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-07-26T10:58:59.788 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/daemon-helper 2024-07-26T10:58:59.817 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-07-26T10:58:59.883 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-07-26T10:58:59.883 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T10:58:59.883 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T10:58:59.945 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T10:59:00.047 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-07-26T10:59:00.047 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T10:59:00.071 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T10:59:00.134 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-07-26T10:59:00.134 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/adjust-ulimits 2024-07-26T10:59:00.159 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-07-26T10:59:00.223 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-07-26T10:59:00.224 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T10:59:00.224 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T10:59:00.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T10:59:00.384 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-07-26T10:59:00.385 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T10:59:00.409 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T10:59:00.473 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-07-26T10:59:00.473 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/stdin-killer 2024-07-26T10:59:00.497 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-07-26T10:59:00.560 INFO:teuthology.run_tasks:Running task cephadm... 2024-07-26T10:59:00.653 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug cephsqlite': 20, 'mgr/snap_schedule/allow_m_granularity': True, 'mgr/snap_schedule/dump_on_update': True, 'mgr/snap_schedule/log_level': 'debug'}, 'mon': {'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'osd op complaint time': 180}}, 'conf': {'global': {'ms die on skipped message': False}, 'mds': {'mds_bal_replicate_threshold': 1}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'mgr-modules': ['snap_schedule'], 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-07-26T10:59:00.653 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:59:00.654 INFO:tasks.cephadm:Cluster fsid is 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T10:59:00.654 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-07-26T10:59:00.654 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.134', 'mon.b': '172.21.15.189', 'mon.c': '172.21.15.193'} 2024-07-26T10:59:00.654 INFO:tasks.cephadm:First mon is mon.a on smithi134 2024-07-26T10:59:00.655 INFO:tasks.cephadm:First mgr is x 2024-07-26T10:59:00.655 INFO:tasks.cephadm:Normalizing hostnames... 2024-07-26T10:59:00.655 DEBUG:teuthology.orchestra.run.smithi134:> sudo hostname $(hostname -s) 2024-07-26T10:59:00.719 DEBUG:teuthology.orchestra.run.smithi189:> sudo hostname $(hostname -s) 2024-07-26T10:59:00.743 DEBUG:teuthology.orchestra.run.smithi193:> sudo hostname $(hostname -s) 2024-07-26T10:59:00.766 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-07-26T10:59:00.769 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:59:00.910 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/81524/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-5320-ge8f88bb1', 'node_name': '172.21.4.69+confusa07', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-5320.ge8f88bb1'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-07-25 16:18:49.953482', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240725.121331-debug', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'archs': ['arm64', 'x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/81524/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-5320-ge8f88bb1', 'node_name': '172.21.2.8+braggi08', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-5320.ge8f88bb1'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-07-25 15:10:42.298781', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-vshankar-testing-20240725.121331-debug', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-07-26T10:59:01.050 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-vshankar-testing-20240725.121331-debug, sha1 e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T10:59:01.051 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm 2024-07-26T10:59:01.051 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm 2024-07-26T10:59:01.051 DEBUG:teuthology.orchestra.run.smithi134:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T10:59:01.447 INFO:teuthology.orchestra.run.smithi134.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 10:59 /home/ubuntu/cephtest/cephadm 2024-07-26T10:59:01.447 DEBUG:teuthology.orchestra.run.smithi189:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T10:59:02.355 INFO:teuthology.orchestra.run.smithi189.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 10:59 /home/ubuntu/cephtest/cephadm 2024-07-26T10:59:02.355 DEBUG:teuthology.orchestra.run.smithi193:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-vshankar-testing-20240725.121331-debug/e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-07-26T10:59:02.883 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r--. 1 ubuntu ubuntu 789811 Jul 26 10:59 /home/ubuntu/cephtest/cephadm 2024-07-26T10:59:02.883 DEBUG:teuthology.orchestra.run.smithi134:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T10:59:02.943 DEBUG:teuthology.orchestra.run.smithi189:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T10:59:02.961 DEBUG:teuthology.orchestra.run.smithi193:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-07-26T10:59:02.994 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 on all hosts... 2024-07-26T10:59:02.995 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T10:59:03.002 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T10:59:03.004 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 pull 2024-07-26T10:59:03.830 INFO:teuthology.orchestra.run.smithi189.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T10:59:03.967 INFO:teuthology.orchestra.run.smithi193.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T10:59:04.579 INFO:teuthology.orchestra.run.smithi134.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T10:59:33.223 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-07-26T10:59:33.224 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T10:59:33.224 INFO:teuthology.orchestra.run.smithi189.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T10:59:33.224 INFO:teuthology.orchestra.run.smithi189.stdout: "repo_digests": [ 2024-07-26T10:59:33.224 INFO:teuthology.orchestra.run.smithi189.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T10:59:33.224 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-07-26T10:59:33.224 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-07-26T10:59:47.523 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-07-26T10:59:47.523 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T10:59:47.523 INFO:teuthology.orchestra.run.smithi193.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T10:59:47.523 INFO:teuthology.orchestra.run.smithi193.stdout: "repo_digests": [ 2024-07-26T10:59:47.524 INFO:teuthology.orchestra.run.smithi193.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T10:59:47.524 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-07-26T10:59:47.524 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-07-26T10:59:53.324 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T10:59:53.324 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T10:59:53.324 INFO:teuthology.orchestra.run.smithi134.stdout: "image_id": "ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8", 2024-07-26T10:59:53.324 INFO:teuthology.orchestra.run.smithi134.stdout: "repo_digests": [ 2024-07-26T10:59:53.324 INFO:teuthology.orchestra.run.smithi134.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2" 2024-07-26T10:59:53.325 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-07-26T10:59:53.325 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T10:59:53.392 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph 2024-07-26T10:59:53.480 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph 2024-07-26T10:59:53.512 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph 2024-07-26T10:59:53.543 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /etc/ceph 2024-07-26T10:59:53.624 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /etc/ceph 2024-07-26T10:59:53.651 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /etc/ceph 2024-07-26T10:59:53.677 INFO:tasks.cephadm:Writing seed config... 2024-07-26T10:59:53.679 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-07-26T10:59:53.679 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-07-26T10:59:53.679 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-07-26T10:59:53.679 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-07-26T10:59:53.679 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-07-26T10:59:53.679 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-07-26T10:59:53.679 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-07-26T10:59:53.679 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-07-26T10:59:53.679 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-07-26T10:59:53.679 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-07-26T10:59:53.679 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-07-26T10:59:53.680 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-07-26T10:59:53.680 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-07-26T10:59:53.680 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-07-26T10:59:53.680 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-07-26T10:59:53.680 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-07-26T10:59:53.680 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-07-26T10:59:53.680 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-07-26T10:59:53.680 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-07-26T10:59:53.680 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-07-26T10:59:53.680 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-07-26T10:59:53.680 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-07-26T10:59:53.680 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-07-26T10:59:53.680 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-07-26T10:59:53.681 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T10:59:53.681 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-07-26T10:59:53.738 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 105fbe5c-4b3e-11ef-bcb7-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [mds] mds_bal_replicate_threshold = 1 2024-07-26T10:59:53.739 DEBUG:teuthology.orchestra.run.smithi134:mon.a> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mon.a.service 2024-07-26T10:59:53.785 DEBUG:teuthology.orchestra.run.smithi134:mgr.x> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mgr.x.service 2024-07-26T10:59:53.831 INFO:tasks.cephadm:Bootstrapping... 2024-07-26T10:59:53.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 -v bootstrap --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.134 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-07-26T10:59:54.112 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-07-26T10:59:54.113 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', '-v', 'bootstrap', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.134', '--skip-admin-label'] 2024-07-26T10:59:54.113 INFO:teuthology.orchestra.run.smithi134.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-07-26T10:59:54.113 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying podman|docker is present... 2024-07-26T10:59:54.168 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stdout 5.1.2 2024-07-26T10:59:54.168 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying lvm2 is present... 2024-07-26T10:59:54.168 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying time synchronization is in place... 2024-07-26T10:59:54.185 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-07-26T10:59:54.185 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-07-26T10:59:54.199 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-07-26T10:59:54.200 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-07-26T10:59:54.215 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout enabled 2024-07-26T10:59:54.232 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout active 2024-07-26T10:59:54.232 INFO:teuthology.orchestra.run.smithi134.stdout:Unit chronyd.service is enabled and running 2024-07-26T10:59:54.233 INFO:teuthology.orchestra.run.smithi134.stdout:Repeating the final host check... 2024-07-26T10:59:54.293 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stdout 5.1.2 2024-07-26T10:59:54.293 INFO:teuthology.orchestra.run.smithi134.stdout:podman (/bin/podman) version 5.1.2 is present 2024-07-26T10:59:54.293 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl is present 2024-07-26T10:59:54.293 INFO:teuthology.orchestra.run.smithi134.stdout:lvcreate is present 2024-07-26T10:59:54.308 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-07-26T10:59:54.309 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-07-26T10:59:54.323 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-07-26T10:59:54.323 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-07-26T10:59:54.346 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout enabled 2024-07-26T10:59:54.368 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout active 2024-07-26T10:59:54.368 INFO:teuthology.orchestra.run.smithi134.stdout:Unit chronyd.service is enabled and running 2024-07-26T10:59:54.369 INFO:teuthology.orchestra.run.smithi134.stdout:Host looks OK 2024-07-26T10:59:54.369 INFO:teuthology.orchestra.run.smithi134.stdout:Cluster fsid: 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T10:59:54.369 INFO:teuthology.orchestra.run.smithi134.stdout:Acquiring lock 140641665147712 on /run/cephadm/105fbe5c-4b3e-11ef-bcb7-c7b262605968.lock 2024-07-26T10:59:54.370 INFO:teuthology.orchestra.run.smithi134.stdout:Lock 140641665147712 acquired on /run/cephadm/105fbe5c-4b3e-11ef-bcb7-c7b262605968.lock 2024-07-26T10:59:54.371 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying IP 172.21.15.134 port 3300 ... 2024-07-26T10:59:54.372 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying IP 172.21.15.134 port 6789 ... 2024-07-26T10:59:54.372 INFO:teuthology.orchestra.run.smithi134.stdout:Base mon IP(s) is [172.21.15.134:3300, 172.21.15.134:6789], mon addrv is [v2:172.21.15.134:3300,v1:172.21.15.134:6789] 2024-07-26T10:59:54.387 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.134 metric 100 2024-07-26T10:59:54.387 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.134 metric 100 2024-07-26T10:59:54.400 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-07-26T10:59:54.400 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-07-26T10:59:54.400 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium 2024-07-26T10:59:54.411 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-07-26T10:59:54.411 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-07-26T10:59:54.411 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-07-26T10:59:54.412 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-07-26T10:59:54.412 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6de9/64 scope link 2024-07-26T10:59:54.412 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-07-26T10:59:54.414 INFO:teuthology.orchestra.run.smithi134.stdout:Mon IP `172.21.15.134` is in CIDR network `172.21.0.0/20` 2024-07-26T10:59:54.414 INFO:teuthology.orchestra.run.smithi134.stdout:Mon IP `172.21.15.134` is in CIDR network `172.21.0.0/20` 2024-07-26T10:59:54.415 INFO:teuthology.orchestra.run.smithi134.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-07-26T10:59:54.417 INFO:teuthology.orchestra.run.smithi134.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-07-26T10:59:54.419 INFO:teuthology.orchestra.run.smithi134.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T10:59:55.060 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stdout ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 2024-07-26T10:59:55.061 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528... 2024-07-26T10:59:55.061 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Getting image source signatures 2024-07-26T10:59:55.061 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Copying blob sha256:27b44fafaf0d8ddf6712bf077b0e069fd1632e81c2016bcdf9194ad64e762303 2024-07-26T10:59:55.061 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Copying blob sha256:43735fbce9214f061a6894341d7822563f5aafe16fc7532189f4415cff6d2d91 2024-07-26T10:59:55.062 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Copying config sha256:ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 2024-07-26T10:59:55.062 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Writing manifest to image destination 2024-07-26T10:59:56.349 INFO:teuthology.orchestra.run.smithi134.stdout:ceph: stdout ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T10:59:56.349 INFO:teuthology.orchestra.run.smithi134.stdout:Ceph version: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T10:59:56.349 INFO:teuthology.orchestra.run.smithi134.stdout:Extracting ceph user uid/gid from container image... 2024-07-26T10:59:57.511 INFO:teuthology.orchestra.run.smithi134.stdout:stat: stdout 167 167 2024-07-26T10:59:57.511 INFO:teuthology.orchestra.run.smithi134.stdout:Creating initial keys... 2024-07-26T10:59:58.635 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph-authtool: stdout AQCugaNm0b77ARAA1HodgcLROYXactWjkhdeAw== 2024-07-26T10:59:59.877 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph-authtool: stdout AQCvgaNmDM53DxAANvDhoAKLV8er3gjTtYLrjQ== 2024-07-26T11:00:01.074 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph-authtool: stdout AQCwgaNmUnPNGBAAY85gdAFOqgsQ5XUcsnzLTQ== 2024-07-26T11:00:01.076 INFO:teuthology.orchestra.run.smithi134.stdout:Creating initial monmap... 2024-07-26T11:00:02.253 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-07-26T11:00:02.253 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-07-26T11:00:02.253 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:00:02.253 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-07-26T11:00:02.254 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool for a [v2:172.21.15.134:3300,v1:172.21.15.134:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-07-26T11:00:02.254 INFO:teuthology.orchestra.run.smithi134.stdout:setting min_mon_release = quincy 2024-07-26T11:00:02.254 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: set fsid to 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:00:02.254 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-07-26T11:00:02.254 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:00:02.254 INFO:teuthology.orchestra.run.smithi134.stdout:Creating mon... 2024-07-26T11:00:03.511 INFO:teuthology.orchestra.run.smithi134.stdout:create mon.a on 2024-07-26T11:00:03.922 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-07-26T11:00:04.162 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-07-26T11:00:04.452 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968.target → /etc/systemd/system/ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968.target. 2024-07-26T11:00:04.452 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968.target → /etc/systemd/system/ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968.target. 2024-07-26T11:00:04.837 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mon.a 2024-07-26T11:00:04.837 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Failed to reset failed state of unit ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mon.a.service: Unit ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mon.a.service not loaded. 2024-07-26T11:00:05.129 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968.target.wants/ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mon.a.service → /etc/systemd/system/ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@.service. 2024-07-26T11:00:05.804 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:05 smithi134 podman[31585]: 2024-07-26 11:00:05.421657103 +0000 UTC m=+0.032369543 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T11:00:05.804 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:05 smithi134 podman[31585]: 2024-07-26 11:00:05.80096708 +0000 UTC m=+0.411679560 container init 04cbe2a98803e64d36897c5f6be2fef38aef904407a08314dc165f51b62aac22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:00:05.940 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T11:00:05.940 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout disabled 2024-07-26T11:00:05.956 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T11:00:05.956 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-07-26T11:00:05.956 INFO:teuthology.orchestra.run.smithi134.stdout:firewalld.service is not enabled 2024-07-26T11:00:05.956 INFO:teuthology.orchestra.run.smithi134.stdout:Not possible to enable service . firewalld.service is not available 2024-07-26T11:00:05.957 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mon to start... 2024-07-26T11:00:05.957 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mon... 2024-07-26T11:00:06.105 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:05 smithi134 podman[31585]: 2024-07-26 11:00:05.812484383 +0000 UTC m=+0.423196833 container start 04cbe2a98803e64d36897c5f6be2fef38aef904407a08314dc165f51b62aac22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mon-a, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-07-26T11:00:06.105 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:05 smithi134 bash[31585]: 04cbe2a98803e64d36897c5f6be2fef38aef904407a08314dc165f51b62aac22 2024-07-26T11:00:06.105 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:05 smithi134 systemd[1]: Started Ceph mon.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:00:06.105 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:05 smithi134 ceph-mon[31604]: mkfs 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:00:06.105 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:05 smithi134 ceph-mon[31604]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T11:00:07.234 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:06 smithi134 ceph-mon[31604]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T11:00:07.234 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:06 smithi134 ceph-mon[31604]: monmap epoch 1 2024-07-26T11:00:07.234 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:06 smithi134 ceph-mon[31604]: fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:00:07.234 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:06 smithi134 ceph-mon[31604]: last_changed 2024-07-26T11:00:01.653390+0000 2024-07-26T11:00:07.234 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:06 smithi134 ceph-mon[31604]: created 2024-07-26T11:00:01.653390+0000 2024-07-26T11:00:07.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:06 smithi134 ceph-mon[31604]: min_mon_release 19 (squid) 2024-07-26T11:00:07.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:06 smithi134 ceph-mon[31604]: election_strategy: 1 2024-07-26T11:00:07.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:06 smithi134 ceph-mon[31604]: 0: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.a 2024-07-26T11:00:07.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:06 smithi134 ceph-mon[31604]: fsmap 2024-07-26T11:00:07.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:06 smithi134 ceph-mon[31604]: osdmap e1: 0 total, 0 up, 0 in 2024-07-26T11:00:07.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:06 smithi134 ceph-mon[31604]: mgrmap e1: no daemons active 2024-07-26T11:00:07.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:06 smithi134 ceph-mon[31604]: from='client.? 172.21.15.134:0/2874181308' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-07-26T11:00:07.300 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout cluster: 2024-07-26T11:00:07.301 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout id: 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:00:07.301 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-07-26T11:00:07.301 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-07-26T11:00:07.301 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout services: 2024-07-26T11:00:07.302 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.706341s) 2024-07-26T11:00:07.302 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-07-26T11:00:07.302 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-07-26T11:00:07.302 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-07-26T11:00:07.302 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout data: 2024-07-26T11:00:07.302 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-07-26T11:00:07.302 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-07-26T11:00:07.302 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-07-26T11:00:07.302 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout pgs: 2024-07-26T11:00:07.302 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-07-26T11:00:07.302 INFO:teuthology.orchestra.run.smithi134.stdout:mon is available 2024-07-26T11:00:07.302 INFO:teuthology.orchestra.run.smithi134.stdout:Assimilating anything we can from ceph.conf... 2024-07-26T11:00:08.386 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:08 smithi134 ceph-mon[31604]: from='client.? 172.21.15.134:0/4057958033' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-07-26T11:00:08.386 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:08 smithi134 ceph-mon[31604]: from='client.? 172.21.15.134:0/4057958033' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-07-26T11:00:08.692 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-07-26T11:00:08.692 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [global] 2024-07-26T11:00:08.693 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout fsid = 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:00:08.693 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-07-26T11:00:08.693 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.134:3300,v1:172.21.15.134:6789] 2024-07-26T11:00:08.693 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-07-26T11:00:08.693 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-07-26T11:00:08.693 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-07-26T11:00:08.693 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-07-26T11:00:08.693 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-07-26T11:00:08.693 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [mgr] 2024-07-26T11:00:08.694 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-07-26T11:00:08.694 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-07-26T11:00:08.694 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [osd] 2024-07-26T11:00:08.694 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-07-26T11:00:08.694 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-07-26T11:00:08.694 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-07-26T11:00:08.694 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-07-26T11:00:08.694 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-07-26T11:00:08.694 INFO:teuthology.orchestra.run.smithi134.stdout:Generating new minimal ceph.conf... 2024-07-26T11:00:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:09 smithi134 ceph-mon[31604]: from='client.? 172.21.15.134:0/196367790' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:00:10.072 INFO:teuthology.orchestra.run.smithi134.stdout:Restarting the monitor... 2024-07-26T11:00:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:10 smithi134 systemd[1]: Stopping Ceph mon.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:00:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:10 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mon-a[31600]: 2024-07-26T11:00:10.597+0000 7f45f6c00640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-07-26T11:00:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:10 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mon-a[31600]: 2024-07-26T11:00:10.597+0000 7f45f6c00640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-07-26T11:00:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:10 smithi134 podman[31847]: 2024-07-26 11:00:10.670374536 +0000 UTC m=+0.188242116 container died 04cbe2a98803e64d36897c5f6be2fef38aef904407a08314dc165f51b62aac22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-07-26T11:00:11.343 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:11 smithi134 podman[31847]: 2024-07-26 11:00:11.082318748 +0000 UTC m=+0.600186354 container cleanup 04cbe2a98803e64d36897c5f6be2fef38aef904407a08314dc165f51b62aac22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:00:11.344 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:11 smithi134 bash[31847]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mon-a 2024-07-26T11:00:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:11 smithi134 podman[31859]: 2024-07-26 11:00:11.341421036 +0000 UTC m=+0.660473073 container remove 04cbe2a98803e64d36897c5f6be2fef38aef904407a08314dc165f51b62aac22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-07-26T11:00:12.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:11 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mon.a.service: Deactivated successfully. 2024-07-26T11:00:12.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:11 smithi134 systemd[1]: Stopped Ceph mon.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:00:12.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:11 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mon.a.service: Consumed 2.063s CPU time. 2024-07-26T11:00:12.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:11 smithi134 systemd[1]: Starting Ceph mon.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:00:12.466 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 podman[31948]: 2024-07-26 11:00:12.113101798 +0000 UTC m=+0.038292571 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T11:00:12.467 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 podman[31948]: 2024-07-26 11:00:12.21745698 +0000 UTC m=+0.142647774 container create 38a9d55739352365b36157cdca60270abf0811788ef4be88c35fdc060eb270fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2) 2024-07-26T11:00:12.467 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 podman[31948]: 2024-07-26 11:00:12.464746862 +0000 UTC m=+0.389937686 container init 38a9d55739352365b36157cdca60270abf0811788ef4be88c35fdc060eb270fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724) 2024-07-26T11:00:12.556 INFO:teuthology.orchestra.run.smithi134.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-07-26T11:00:12.721 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 podman[31948]: 2024-07-26 11:00:12.475211401 +0000 UTC m=+0.400402195 container start 38a9d55739352365b36157cdca60270abf0811788ef4be88c35fdc060eb270fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2) 2024-07-26T11:00:12.721 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 bash[31948]: 38a9d55739352365b36157cdca60270abf0811788ef4be88c35fdc060eb270fd 2024-07-26T11:00:12.721 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 systemd[1]: Started Ceph mon.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:00:12.721 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: set uid:gid to 167:167 (ceph:ceph) 2024-07-26T11:00:12.721 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev), process ceph-mon, pid 2 2024-07-26T11:00:12.721 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: pidfile_write: ignore empty --pid-file 2024-07-26T11:00:12.721 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: load: jerasure load: lrc 2024-07-26T11:00:12.721 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: RocksDB version: 7.9.2 2024-07-26T11:00:12.721 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Git sha 0 2024-07-26T11:00:12.721 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Compile date 2024-07-25 13:49:38 2024-07-26T11:00:12.721 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: DB SUMMARY 2024-07-26T11:00:12.722 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: DB Session ID: QY2K7KQBLFEPR6R3GQN1 2024-07-26T11:00:12.722 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: CURRENT file: CURRENT 2024-07-26T11:00:12.722 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: IDENTITY file: IDENTITY 2024-07-26T11:00:12.722 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-07-26T11:00:12.722 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-07-26T11:00:12.722 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 94459 ; 2024-07-26T11:00:12.722 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.error_if_exists: 0 2024-07-26T11:00:12.722 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.create_if_missing: 0 2024-07-26T11:00:12.722 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.paranoid_checks: 1 2024-07-26T11:00:12.723 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.flush_verify_memtable_count: 1 2024-07-26T11:00:12.723 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-07-26T11:00:12.723 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-07-26T11:00:12.723 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.env: 0x55962e1bb280 2024-07-26T11:00:12.723 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.fs: PosixFileSystem 2024-07-26T11:00:12.723 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.info_log: 0x5596656ff060 2024-07-26T11:00:12.723 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_file_opening_threads: 16 2024-07-26T11:00:12.723 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.statistics: (nil) 2024-07-26T11:00:12.723 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.use_fsync: 0 2024-07-26T11:00:12.723 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_log_file_size: 0 2024-07-26T11:00:12.723 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-07-26T11:00:12.723 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.log_file_time_to_roll: 0 2024-07-26T11:00:12.723 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.keep_log_file_num: 1000 2024-07-26T11:00:12.724 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.recycle_log_file_num: 0 2024-07-26T11:00:12.724 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.allow_fallocate: 1 2024-07-26T11:00:12.724 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.allow_mmap_reads: 0 2024-07-26T11:00:12.724 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.allow_mmap_writes: 0 2024-07-26T11:00:12.725 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.use_direct_reads: 0 2024-07-26T11:00:12.725 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-07-26T11:00:12.725 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.create_missing_column_families: 0 2024-07-26T11:00:12.725 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.db_log_dir: 2024-07-26T11:00:12.725 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.wal_dir: 2024-07-26T11:00:12.725 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.table_cache_numshardbits: 6 2024-07-26T11:00:12.725 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.WAL_ttl_seconds: 0 2024-07-26T11:00:12.725 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.WAL_size_limit_MB: 0 2024-07-26T11:00:12.725 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-07-26T11:00:12.725 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-07-26T11:00:12.725 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.is_fd_close_on_exec: 1 2024-07-26T11:00:12.726 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.advise_random_on_open: 1 2024-07-26T11:00:12.726 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.db_write_buffer_size: 0 2024-07-26T11:00:12.726 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.write_buffer_manager: 0x5596657035e0 2024-07-26T11:00:12.726 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-07-26T11:00:12.726 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-07-26T11:00:12.726 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.use_adaptive_mutex: 0 2024-07-26T11:00:12.726 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.rate_limiter: (nil) 2024-07-26T11:00:12.726 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-07-26T11:00:12.726 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.wal_recovery_mode: 2 2024-07-26T11:00:12.726 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.enable_thread_tracking: 0 2024-07-26T11:00:12.726 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.enable_pipelined_write: 0 2024-07-26T11:00:12.726 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.unordered_write: 0 2024-07-26T11:00:12.727 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-07-26T11:00:12.727 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-07-26T11:00:12.727 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-07-26T11:00:12.727 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-07-26T11:00:12.727 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.row_cache: None 2024-07-26T11:00:12.727 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.wal_filter: None 2024-07-26T11:00:12.727 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-07-26T11:00:12.727 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.allow_ingest_behind: 0 2024-07-26T11:00:12.727 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.two_write_queues: 0 2024-07-26T11:00:12.727 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.manual_wal_flush: 0 2024-07-26T11:00:12.727 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.wal_compression: 0 2024-07-26T11:00:12.727 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.atomic_flush: 0 2024-07-26T11:00:12.728 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-07-26T11:00:12.728 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.persist_stats_to_disk: 0 2024-07-26T11:00:12.728 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.write_dbid_to_manifest: 0 2024-07-26T11:00:12.728 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.log_readahead_size: 0 2024-07-26T11:00:12.728 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-07-26T11:00:12.728 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.best_efforts_recovery: 0 2024-07-26T11:00:12.728 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-07-26T11:00:12.728 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-07-26T11:00:12.728 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.allow_data_in_errors: 0 2024-07-26T11:00:12.728 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.db_host_id: __hostname__ 2024-07-26T11:00:12.728 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.enforce_single_del_contracts: true 2024-07-26T11:00:12.728 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_background_jobs: 2 2024-07-26T11:00:12.729 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_background_compactions: -1 2024-07-26T11:00:12.729 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_subcompactions: 1 2024-07-26T11:00:12.729 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-07-26T11:00:12.729 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-07-26T11:00:12.729 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.delayed_write_rate : 16777216 2024-07-26T11:00:12.729 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_total_wal_size: 0 2024-07-26T11:00:12.729 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-07-26T11:00:12.729 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.stats_dump_period_sec: 600 2024-07-26T11:00:12.729 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.stats_persist_period_sec: 600 2024-07-26T11:00:12.729 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-07-26T11:00:12.729 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_open_files: -1 2024-07-26T11:00:12.729 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.bytes_per_sync: 0 2024-07-26T11:00:12.729 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.wal_bytes_per_sync: 0 2024-07-26T11:00:12.730 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.strict_bytes_per_sync: 0 2024-07-26T11:00:12.730 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compaction_readahead_size: 0 2024-07-26T11:00:12.730 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_background_flushes: -1 2024-07-26T11:00:12.730 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Compression algorithms supported: 2024-07-26T11:00:12.730 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: kZSTD supported: 0 2024-07-26T11:00:12.730 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: kXpressCompression supported: 0 2024-07-26T11:00:12.730 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: kBZip2Compression supported: 0 2024-07-26T11:00:12.730 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-07-26T11:00:12.730 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: kLZ4Compression supported: 1 2024-07-26T11:00:12.730 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: kZlibCompression supported: 1 2024-07-26T11:00:12.730 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: kLZ4HCCompression supported: 1 2024-07-26T11:00:12.730 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: kSnappyCompression supported: 1 2024-07-26T11:00:12.730 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Fast CRC32 supported: Supported on x86 2024-07-26T11:00:12.730 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: DMutex implementation: pthread_mutex_t 2024-07-26T11:00:12.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-07-26T11:00:12.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-07-26T11:00:12.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-07-26T11:00:12.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.merge_operator: 2024-07-26T11:00:12.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compaction_filter: None 2024-07-26T11:00:12.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compaction_filter_factory: None 2024-07-26T11:00:12.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.sst_partitioner_factory: None 2024-07-26T11:00:12.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.memtable_factory: SkipListFactory 2024-07-26T11:00:12.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.table_factory: BlockBasedTable 2024-07-26T11:00:12.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5596656fec00) 2024-07-26T11:00:12.731 INFO:journalctl@ceph.mon.a.smithi134.stdout: cache_index_and_filter_blocks: 1 2024-07-26T11:00:12.731 INFO:journalctl@ceph.mon.a.smithi134.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-07-26T11:00:12.731 INFO:journalctl@ceph.mon.a.smithi134.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-07-26T11:00:12.732 INFO:journalctl@ceph.mon.a.smithi134.stdout: pin_top_level_index_and_filter: 1 2024-07-26T11:00:12.732 INFO:journalctl@ceph.mon.a.smithi134.stdout: index_type: 0 2024-07-26T11:00:12.732 INFO:journalctl@ceph.mon.a.smithi134.stdout: data_block_index_type: 0 2024-07-26T11:00:12.732 INFO:journalctl@ceph.mon.a.smithi134.stdout: index_shortening: 1 2024-07-26T11:00:12.732 INFO:journalctl@ceph.mon.a.smithi134.stdout: data_block_hash_table_util_ratio: 0.750000 2024-07-26T11:00:12.732 INFO:journalctl@ceph.mon.a.smithi134.stdout: checksum: 4 2024-07-26T11:00:12.732 INFO:journalctl@ceph.mon.a.smithi134.stdout: no_block_cache: 0 2024-07-26T11:00:12.732 INFO:journalctl@ceph.mon.a.smithi134.stdout: block_cache: 0x5596616c3610 2024-07-26T11:00:12.732 INFO:journalctl@ceph.mon.a.smithi134.stdout: block_cache_name: BinnedLRUCache 2024-07-26T11:00:12.732 INFO:journalctl@ceph.mon.a.smithi134.stdout: block_cache_options: 2024-07-26T11:00:12.732 INFO:journalctl@ceph.mon.a.smithi134.stdout: capacity : 536870912 2024-07-26T11:00:12.732 INFO:journalctl@ceph.mon.a.smithi134.stdout: num_shard_bits : 4 2024-07-26T11:00:12.732 INFO:journalctl@ceph.mon.a.smithi134.stdout: strict_capacity_limit : 0 2024-07-26T11:00:12.732 INFO:journalctl@ceph.mon.a.smithi134.stdout: high_pri_pool_ratio: 0.000 2024-07-26T11:00:12.733 INFO:journalctl@ceph.mon.a.smithi134.stdout: block_cache_compressed: (nil) 2024-07-26T11:00:12.733 INFO:journalctl@ceph.mon.a.smithi134.stdout: persistent_cache: (nil) 2024-07-26T11:00:12.733 INFO:journalctl@ceph.mon.a.smithi134.stdout: block_size: 4096 2024-07-26T11:00:12.733 INFO:journalctl@ceph.mon.a.smithi134.stdout: block_size_deviation: 10 2024-07-26T11:00:12.733 INFO:journalctl@ceph.mon.a.smithi134.stdout: block_restart_interval: 16 2024-07-26T11:00:12.733 INFO:journalctl@ceph.mon.a.smithi134.stdout: index_block_restart_interval: 1 2024-07-26T11:00:12.733 INFO:journalctl@ceph.mon.a.smithi134.stdout: metadata_block_size: 4096 2024-07-26T11:00:12.733 INFO:journalctl@ceph.mon.a.smithi134.stdout: partition_filters: 0 2024-07-26T11:00:12.733 INFO:journalctl@ceph.mon.a.smithi134.stdout: use_delta_encoding: 1 2024-07-26T11:00:12.733 INFO:journalctl@ceph.mon.a.smithi134.stdout: filter_policy: bloomfilter 2024-07-26T11:00:12.733 INFO:journalctl@ceph.mon.a.smithi134.stdout: whole_key_filtering: 1 2024-07-26T11:00:12.733 INFO:journalctl@ceph.mon.a.smithi134.stdout: verify_compression: 0 2024-07-26T11:00:12.733 INFO:journalctl@ceph.mon.a.smithi134.stdout: read_amp_bytes_per_bit: 0 2024-07-26T11:00:12.734 INFO:journalctl@ceph.mon.a.smithi134.stdout: format_version: 5 2024-07-26T11:00:12.734 INFO:journalctl@ceph.mon.a.smithi134.stdout: enable_index_compression: 1 2024-07-26T11:00:12.734 INFO:journalctl@ceph.mon.a.smithi134.stdout: block_align: 0 2024-07-26T11:00:12.734 INFO:journalctl@ceph.mon.a.smithi134.stdout: max_auto_readahead_size: 262144 2024-07-26T11:00:12.734 INFO:journalctl@ceph.mon.a.smithi134.stdout: prepopulate_block_cache: 0 2024-07-26T11:00:12.734 INFO:journalctl@ceph.mon.a.smithi134.stdout: initial_auto_readahead_size: 8192 2024-07-26T11:00:12.734 INFO:journalctl@ceph.mon.a.smithi134.stdout: num_file_reads_for_auto_readahead: 2 2024-07-26T11:00:12.734 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.write_buffer_size: 33554432 2024-07-26T11:00:12.734 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_write_buffer_number: 2 2024-07-26T11:00:12.734 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compression: NoCompression 2024-07-26T11:00:12.734 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.bottommost_compression: Disabled 2024-07-26T11:00:12.734 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.prefix_extractor: nullptr 2024-07-26T11:00:12.734 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-07-26T11:00:12.735 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.num_levels: 7 2024-07-26T11:00:12.735 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-07-26T11:00:12.735 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-07-26T11:00:12.735 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-07-26T11:00:12.736 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-07-26T11:00:12.736 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-07-26T11:00:12.736 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-07-26T11:00:12.736 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-07-26T11:00:12.736 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-07-26T11:00:12.736 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-07-26T11:00:12.736 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-07-26T11:00:12.736 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-07-26T11:00:12.736 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-07-26T11:00:12.736 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compression_opts.window_bits: -14 2024-07-26T11:00:12.736 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compression_opts.level: 32767 2024-07-26T11:00:12.736 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compression_opts.strategy: 0 2024-07-26T11:00:12.737 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-07-26T11:00:12.737 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-07-26T11:00:12.737 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-07-26T11:00:12.737 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-07-26T11:00:12.737 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compression_opts.enabled: false 2024-07-26T11:00:12.737 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-07-26T11:00:12.737 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-07-26T11:00:12.737 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-07-26T11:00:12.737 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-07-26T11:00:12.737 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.target_file_size_base: 67108864 2024-07-26T11:00:12.737 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.target_file_size_multiplier: 1 2024-07-26T11:00:12.737 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-07-26T11:00:12.737 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-07-26T11:00:12.737 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-07-26T11:00:12.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-07-26T11:00:12.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-07-26T11:00:12.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-07-26T11:00:12.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-07-26T11:00:12.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-07-26T11:00:12.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-07-26T11:00:12.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-07-26T11:00:12.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-07-26T11:00:12.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-07-26T11:00:12.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-07-26T11:00:12.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.arena_block_size: 1048576 2024-07-26T11:00:12.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-07-26T11:00:12.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-07-26T11:00:12.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.disable_auto_compactions: 0 2024-07-26T11:00:12.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-07-26T11:00:12.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-07-26T11:00:12.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-07-26T11:00:12.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-07-26T11:00:12.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-07-26T11:00:12.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-07-26T11:00:12.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-07-26T11:00:12.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-07-26T11:00:12.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-07-26T11:00:12.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-07-26T11:00:12.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-07-26T11:00:12.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.inplace_update_support: 0 2024-07-26T11:00:12.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.inplace_update_num_locks: 10000 2024-07-26T11:00:12.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-07-26T11:00:12.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-07-26T11:00:12.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.memtable_huge_page_size: 0 2024-07-26T11:00:12.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.bloom_locality: 0 2024-07-26T11:00:12.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.max_successive_merges: 0 2024-07-26T11:00:12.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.optimize_filters_for_hits: 0 2024-07-26T11:00:12.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.paranoid_file_checks: 0 2024-07-26T11:00:12.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.force_consistency_checks: 1 2024-07-26T11:00:12.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.report_bg_io_stats: 0 2024-07-26T11:00:12.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.ttl: 2592000 2024-07-26T11:00:12.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.periodic_compaction_seconds: 0 2024-07-26T11:00:12.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-07-26T11:00:12.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-07-26T11:00:12.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.enable_blob_files: false 2024-07-26T11:00:12.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.min_blob_size: 0 2024-07-26T11:00:12.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.blob_file_size: 268435456 2024-07-26T11:00:12.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.blob_compression_type: NoCompression 2024-07-26T11:00:12.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.enable_blob_garbage_collection: false 2024-07-26T11:00:12.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-07-26T11:00:12.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-07-26T11:00:12.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-07-26T11:00:12.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.blob_file_starting_level: 0 2024-07-26T11:00:12.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-07-26T11:00:12.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-07-26T11:00:12.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-07-26T11:00:12.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2fa3bf29-366d-4838-9370-744ad93ba18a 2024-07-26T11:00:12.742 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721991612605929, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-07-26T11:00:12.742 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-07-26T11:00:12.742 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721991612608103, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 90689, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 280, "table_properties": {"data_size": 88730, "index_size": 279, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11488, "raw_average_key_size": 48, "raw_value_size": 82253, "raw_average_value_size": 344, "num_data_blocks": 12, "num_entries": 239, "num_filter_entries": 239, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1721991612, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2fa3bf29-366d-4838-9370-744ad93ba18a", "db_session_id": "QY2K7KQBLFEPR6R3GQN1", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-07-26T11:00:12.742 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721991612608306, "job": 1, "event": "recovery_finished"} 2024-07-26T11:00:12.742 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-07-26T11:00:12.742 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-07-26T11:00:12.742 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559665722a80 2024-07-26T11:00:12.742 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: DB pointer 0x5596664c2000 2024-07-26T11:00:12.742 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-07-26T11:00:12.742 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-07-26T11:00:12.742 INFO:journalctl@ceph.mon.a.smithi134.stdout: ** DB Stats ** 2024-07-26T11:00:12.742 INFO:journalctl@ceph.mon.a.smithi134.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T11:00:12.743 INFO:journalctl@ceph.mon.a.smithi134.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-07-26T11:00:12.743 INFO:journalctl@ceph.mon.a.smithi134.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T11:00:12.743 INFO:journalctl@ceph.mon.a.smithi134.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T11:00:12.743 INFO:journalctl@ceph.mon.a.smithi134.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-07-26T11:00:12.743 INFO:journalctl@ceph.mon.a.smithi134.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T11:00:12.743 INFO:journalctl@ceph.mon.a.smithi134.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T11:00:12.743 INFO:journalctl@ceph.mon.a.smithi134.stdout: 2024-07-26T11:00:12.743 INFO:journalctl@ceph.mon.a.smithi134.stdout: ** Compaction Stats [default] ** 2024-07-26T11:00:12.743 INFO:journalctl@ceph.mon.a.smithi134.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T11:00:12.743 INFO:journalctl@ceph.mon.a.smithi134.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-07-26T11:00:12.743 INFO:journalctl@ceph.mon.a.smithi134.stdout: L0 2/0 90.42 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 57.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T11:00:12.743 INFO:journalctl@ceph.mon.a.smithi134.stdout: Sum 2/0 90.42 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 57.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T11:00:12.743 INFO:journalctl@ceph.mon.a.smithi134.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 57.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T11:00:12.743 INFO:journalctl@ceph.mon.a.smithi134.stdout: 2024-07-26T11:00:12.744 INFO:journalctl@ceph.mon.a.smithi134.stdout: ** Compaction Stats [default] ** 2024-07-26T11:00:12.744 INFO:journalctl@ceph.mon.a.smithi134.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T11:00:12.744 INFO:journalctl@ceph.mon.a.smithi134.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-07-26T11:00:12.744 INFO:journalctl@ceph.mon.a.smithi134.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 57.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-07-26T11:00:12.744 INFO:journalctl@ceph.mon.a.smithi134.stdout: 2024-07-26T11:00:12.744 INFO:journalctl@ceph.mon.a.smithi134.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-07-26T11:00:12.744 INFO:journalctl@ceph.mon.a.smithi134.stdout: 2024-07-26T11:00:12.744 INFO:journalctl@ceph.mon.a.smithi134.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T11:00:12.744 INFO:journalctl@ceph.mon.a.smithi134.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-07-26T11:00:12.744 INFO:journalctl@ceph.mon.a.smithi134.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-07-26T11:00:12.744 INFO:journalctl@ceph.mon.a.smithi134.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-07-26T11:00:12.744 INFO:journalctl@ceph.mon.a.smithi134.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-07-26T11:00:12.744 INFO:journalctl@ceph.mon.a.smithi134.stdout: AddFile(Keys): cumulative 0, interval 0 2024-07-26T11:00:12.744 INFO:journalctl@ceph.mon.a.smithi134.stdout: Cumulative compaction: 0.00 GB write, 11.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T11:00:12.745 INFO:journalctl@ceph.mon.a.smithi134.stdout: Interval compaction: 0.00 GB write, 11.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T11:00:12.745 INFO:journalctl@ceph.mon.a.smithi134.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-07-26T11:00:12.745 INFO:journalctl@ceph.mon.a.smithi134.stdout: Block cache BinnedLRUCache@0x5596616c3610#2 capacity: 512.00 MB usage: 1.31 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-07-26T11:00:12.745 INFO:journalctl@ceph.mon.a.smithi134.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.84 KB,0.000160933%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-07-26T11:00:12.745 INFO:journalctl@ceph.mon.a.smithi134.stdout: 2024-07-26T11:00:12.745 INFO:journalctl@ceph.mon.a.smithi134.stdout: ** File Read Latency Histogram By Level [default] ** 2024-07-26T11:00:12.745 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: starting mon.a rank 0 at public addrs [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] at bind addrs [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:00:12.745 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: mon.a@-1(???) e1 preinit fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:00:12.745 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: mon.a@-1(???).mds e1 new map 2024-07-26T11:00:12.745 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: mon.a@-1(???).mds e1 print_map 2024-07-26T11:00:12.745 INFO:journalctl@ceph.mon.a.smithi134.stdout: e1 2024-07-26T11:00:12.745 INFO:journalctl@ceph.mon.a.smithi134.stdout: btime 2024-07-26T11:00:05:954291+0000 2024-07-26T11:00:12.745 INFO:journalctl@ceph.mon.a.smithi134.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T11:00:12.746 INFO:journalctl@ceph.mon.a.smithi134.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T11:00:12.746 INFO:journalctl@ceph.mon.a.smithi134.stdout: legacy client fscid: -1 2024-07-26T11:00:12.746 INFO:journalctl@ceph.mon.a.smithi134.stdout: 2024-07-26T11:00:12.746 INFO:journalctl@ceph.mon.a.smithi134.stdout: No filesystems configured 2024-07-26T11:00:12.746 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-26T11:00:12.746 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T11:00:12.746 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T11:00:12.746 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T11:00:12.746 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-07-26T11:00:12.746 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: mon.a is new leader, mons a in quorum (ranks 0) 2024-07-26T11:00:12.746 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: monmap epoch 1 2024-07-26T11:00:12.746 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:00:12.746 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: last_changed 2024-07-26T11:00:01.653390+0000 2024-07-26T11:00:12.747 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: created 2024-07-26T11:00:01.653390+0000 2024-07-26T11:00:12.747 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: min_mon_release 19 (squid) 2024-07-26T11:00:12.747 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: election_strategy: 1 2024-07-26T11:00:12.747 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: 0: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.a 2024-07-26T11:00:12.747 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: fsmap 2024-07-26T11:00:12.747 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: osdmap e1: 0 total, 0 up, 0 in 2024-07-26T11:00:12.747 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:12 smithi134 ceph-mon[31966]: mgrmap e1: no daemons active 2024-07-26T11:00:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:14 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/756010408' entity='client.admin' 2024-07-26T11:00:23.905 INFO:teuthology.orchestra.run.smithi134.stdout:Wrote config to /etc/ceph/ceph.conf 2024-07-26T11:00:23.935 INFO:teuthology.orchestra.run.smithi134.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-07-26T11:00:23.935 INFO:teuthology.orchestra.run.smithi134.stdout:Creating mgr... 2024-07-26T11:00:23.937 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying port 0.0.0.0:9283 ... 2024-07-26T11:00:23.938 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying port 0.0.0.0:8765 ... 2024-07-26T11:00:24.394 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mgr.x 2024-07-26T11:00:24.395 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Failed to reset failed state of unit ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mgr.x.service: Unit ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mgr.x.service not loaded. 2024-07-26T11:00:24.667 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968.target.wants/ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@.service. 2024-07-26T11:00:25.292 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:25 smithi134 podman[32179]: 2024-07-26 11:00:25.069792749 +0000 UTC m=+0.137561665 container create 55c0ab3a28af4d0417250702338e8ca489281bb304941d7f558071c8cb144edf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:00:25.292 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:25 smithi134 podman[32179]: 2024-07-26 11:00:25.289578309 +0000 UTC m=+0.357347236 container init 55c0ab3a28af4d0417250702338e8ca489281bb304941d7f558071c8cb144edf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0) 2024-07-26T11:00:25.425 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T11:00:25.426 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout disabled 2024-07-26T11:00:25.441 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T11:00:25.441 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-07-26T11:00:25.441 INFO:teuthology.orchestra.run.smithi134.stdout:firewalld.service is not enabled 2024-07-26T11:00:25.441 INFO:teuthology.orchestra.run.smithi134.stdout:Not possible to enable service . firewalld.service is not available 2024-07-26T11:00:25.459 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T11:00:25.460 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout disabled 2024-07-26T11:00:25.476 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T11:00:25.476 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-07-26T11:00:25.476 INFO:teuthology.orchestra.run.smithi134.stdout:firewalld.service is not enabled 2024-07-26T11:00:25.476 INFO:teuthology.orchestra.run.smithi134.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-07-26T11:00:25.476 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mgr to start... 2024-07-26T11:00:25.477 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mgr... 2024-07-26T11:00:25.617 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:25 smithi134 podman[32179]: 2024-07-26 11:00:25.298570261 +0000 UTC m=+0.366339181 container start 55c0ab3a28af4d0417250702338e8ca489281bb304941d7f558071c8cb144edf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-07-26T11:00:25.617 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:25 smithi134 bash[32179]: 55c0ab3a28af4d0417250702338e8ca489281bb304941d7f558071c8cb144edf 2024-07-26T11:00:25.617 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:25 smithi134 systemd[1]: Started Ceph mgr.x for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:00:26.557 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:26 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1247042197' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T11:00:26.867 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-07-26T11:00:26.868 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-07-26T11:00:26.868 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsid": "105fbe5c-4b3e-11ef-bcb7-c7b262605968", 2024-07-26T11:00:26.868 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T11:00:26.869 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T11:00:26.869 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T11:00:26.869 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T11:00:26.869 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:26.873 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T11:00:26.873 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T11:00:26.873 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 0 2024-07-26T11:00:26.873 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-07-26T11:00:26.873 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T11:00:26.873 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T11:00:26.874 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-07-26T11:00:26.874 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-07-26T11:00:26.874 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T11:00:26.874 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T11:00:26.874 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T11:00:26.874 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T11:00:26.874 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:26.874 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T11:00:26.874 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T11:00:26.874 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T11:00:26.874 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T11:00:26.874 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T11:00:26.874 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T11:00:26.875 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T11:00:26.875 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T11:00:26.875 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:26.875 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T11:00:26.875 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T11:00:26.875 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T11:00:26.875 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T11:00:26.875 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T11:00:26.875 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T11:00:26.875 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T11:00:26.875 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T11:00:26.875 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T11:00:26.875 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:26.876 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T11:00:26.876 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T11:00:26.876 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T11:00:05:954291+0000", 2024-07-26T11:00:26.876 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T11:00:26.878 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T11:00:26.878 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:26.878 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T11:00:26.878 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "available": false, 2024-07-26T11:00:26.878 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T11:00:26.878 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T11:00:26.878 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T11:00:26.878 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T11:00:26.878 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T11:00:26.879 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-07-26T11:00:26.879 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T11:00:26.879 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:26.879 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T11:00:26.879 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T11:00:26.879 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T11:00:05.955868+0000", 2024-07-26T11:00:26.879 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T11:00:26.879 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:26.879 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T11:00:26.879 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-07-26T11:00:26.879 INFO:teuthology.orchestra.run.smithi134.stdout:mgr not available, waiting (1/15)... 2024-07-26T11:00:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:29 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3981321626' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T11:00:30.238 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-07-26T11:00:30.238 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-07-26T11:00:30.238 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsid": "105fbe5c-4b3e-11ef-bcb7-c7b262605968", 2024-07-26T11:00:30.238 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T11:00:30.238 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T11:00:30.238 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T11:00:30.238 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T11:00:30.238 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:30.238 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T11:00:30.238 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T11:00:30.239 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 0 2024-07-26T11:00:30.239 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-07-26T11:00:30.239 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T11:00:30.239 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T11:00:30.239 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-07-26T11:00:30.239 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-07-26T11:00:30.239 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T11:00:30.239 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T11:00:30.239 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T11:00:30.239 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T11:00:30.239 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:30.240 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T11:00:30.240 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T11:00:30.240 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T11:00:30.240 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T11:00:30.240 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T11:00:30.243 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T11:00:30.243 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T11:00:30.243 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T11:00:30.243 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:30.244 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T11:00:30.244 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T11:00:30.244 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T11:00:30.244 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T11:00:30.244 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T11:00:30.244 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T11:00:30.244 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T11:00:30.244 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T11:00:30.244 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T11:00:30.244 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:30.245 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T11:00:30.245 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T11:00:30.245 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T11:00:05:954291+0000", 2024-07-26T11:00:30.245 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T11:00:30.245 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T11:00:30.245 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:30.245 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T11:00:30.245 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "available": false, 2024-07-26T11:00:30.245 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T11:00:30.245 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T11:00:30.246 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T11:00:30.246 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T11:00:30.246 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T11:00:30.246 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-07-26T11:00:30.246 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T11:00:30.246 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:30.246 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T11:00:30.246 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T11:00:30.246 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T11:00:05.955868+0000", 2024-07-26T11:00:30.246 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T11:00:30.246 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:30.246 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T11:00:30.246 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-07-26T11:00:30.246 INFO:teuthology.orchestra.run.smithi134.stdout:mgr not available, waiting (2/15)... 2024-07-26T11:00:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:31 smithi134 ceph-mon[31966]: Activating manager daemon x 2024-07-26T11:00:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:31 smithi134 ceph-mon[31966]: mgrmap e2: x(active, starting, since 0.0114551s) 2024-07-26T11:00:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:31 smithi134 ceph-mon[31966]: from='mgr.14100 172.21.15.134:0/3684276624' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T11:00:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:31 smithi134 ceph-mon[31966]: from='mgr.14100 172.21.15.134:0/3684276624' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T11:00:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:31 smithi134 ceph-mon[31966]: from='mgr.14100 172.21.15.134:0/3684276624' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T11:00:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:31 smithi134 ceph-mon[31966]: from='mgr.14100 172.21.15.134:0/3684276624' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:00:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:31 smithi134 ceph-mon[31966]: from='mgr.14100 172.21.15.134:0/3684276624' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T11:00:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:31 smithi134 ceph-mon[31966]: Manager daemon x is now available 2024-07-26T11:00:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:31 smithi134 ceph-mon[31966]: from='mgr.14100 172.21.15.134:0/3684276624' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T11:00:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:31 smithi134 ceph-mon[31966]: from='mgr.14100 172.21.15.134:0/3684276624' entity='mgr.x' 2024-07-26T11:00:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:31 smithi134 ceph-mon[31966]: from='mgr.14100 172.21.15.134:0/3684276624' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T11:00:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:31 smithi134 ceph-mon[31966]: from='mgr.14100 172.21.15.134:0/3684276624' entity='mgr.x' 2024-07-26T11:00:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:31 smithi134 ceph-mon[31966]: from='mgr.14100 172.21.15.134:0/3684276624' entity='mgr.x' 2024-07-26T11:00:33.827 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:33 smithi134 ceph-mon[31966]: mgrmap e3: x(active, since 1.02407s) 2024-07-26T11:00:33.827 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:33 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2083507826' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-07-26T11:00:33.837 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-07-26T11:00:33.837 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-07-26T11:00:33.838 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsid": "105fbe5c-4b3e-11ef-bcb7-c7b262605968", 2024-07-26T11:00:33.838 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "health": { 2024-07-26T11:00:33.838 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-07-26T11:00:33.838 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-07-26T11:00:33.838 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-07-26T11:00:33.838 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:33.838 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-07-26T11:00:33.838 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-07-26T11:00:33.838 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 0 2024-07-26T11:00:33.838 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-07-26T11:00:33.838 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-07-26T11:00:33.838 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "a" 2024-07-26T11:00:33.838 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-07-26T11:00:33.839 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_age": 20, 2024-07-26T11:00:33.839 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "monmap": { 2024-07-26T11:00:33.839 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T11:00:33.839 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-07-26T11:00:33.839 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-07-26T11:00:33.839 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:33.839 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-07-26T11:00:33.839 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T11:00:33.839 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-07-26T11:00:33.839 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-07-26T11:00:33.843 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-07-26T11:00:33.843 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-07-26T11:00:33.843 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-07-26T11:00:33.843 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-07-26T11:00:33.843 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:33.843 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-07-26T11:00:33.843 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-07-26T11:00:33.844 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-07-26T11:00:33.844 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-07-26T11:00:33.844 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-07-26T11:00:33.844 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-07-26T11:00:33.844 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-07-26T11:00:33.844 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-07-26T11:00:33.844 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-07-26T11:00:33.844 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:33.844 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-07-26T11:00:33.844 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T11:00:33.844 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "btime": "2024-07-26T11:00:05:954291+0000", 2024-07-26T11:00:33.844 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-07-26T11:00:33.844 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-07-26T11:00:33.845 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:33.845 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-07-26T11:00:33.845 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T11:00:33.845 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-07-26T11:00:33.845 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modules": [ 2024-07-26T11:00:33.845 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "iostat", 2024-07-26T11:00:33.845 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "nfs", 2024-07-26T11:00:33.845 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "restful" 2024-07-26T11:00:33.845 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-07-26T11:00:33.845 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T11:00:33.845 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:33.845 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-07-26T11:00:33.845 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-07-26T11:00:33.846 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modified": "2024-07-26T11:00:05.955868+0000", 2024-07-26T11:00:33.846 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-07-26T11:00:33.846 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-07-26T11:00:33.846 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-07-26T11:00:33.846 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-07-26T11:00:33.846 INFO:teuthology.orchestra.run.smithi134.stdout:mgr is available 2024-07-26T11:00:34.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:34 smithi134 ceph-mon[31966]: mgrmap e4: x(active, since 2s) 2024-07-26T11:00:35.412 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-07-26T11:00:35.414 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [global] 2024-07-26T11:00:35.414 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout fsid = 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:00:35.414 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-07-26T11:00:35.414 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.134:3300,v1:172.21.15.134:6789] 2024-07-26T11:00:35.415 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-07-26T11:00:35.415 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-07-26T11:00:35.415 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-07-26T11:00:35.415 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-07-26T11:00:35.415 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-07-26T11:00:35.415 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [mgr] 2024-07-26T11:00:35.415 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-07-26T11:00:35.415 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-07-26T11:00:35.415 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [osd] 2024-07-26T11:00:35.415 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-07-26T11:00:35.415 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-07-26T11:00:35.415 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-07-26T11:00:35.415 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-07-26T11:00:35.415 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-07-26T11:00:35.416 INFO:teuthology.orchestra.run.smithi134.stdout:Enabling cephadm module... 2024-07-26T11:00:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:35 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1826192311' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-07-26T11:00:36.828 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:36 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/873617026' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-07-26T11:00:36.828 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:36 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: ignoring --setuser ceph since I am not root 2024-07-26T11:00:36.828 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:36 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: ignoring --setgroup ceph since I am not root 2024-07-26T11:00:37.708 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:37 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/873617026' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-07-26T11:00:37.709 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:37 smithi134 ceph-mon[31966]: mgrmap e5: x(active, since 5s) 2024-07-26T11:00:38.779 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:38 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3226709025' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-07-26T11:00:38.791 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-07-26T11:00:38.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-07-26T11:00:38.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T11:00:38.793 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-07-26T11:00:38.793 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-07-26T11:00:38.793 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-07-26T11:00:38.793 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for the mgr to restart... 2024-07-26T11:00:38.793 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mgr epoch 5... 2024-07-26T11:00:43.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:42 smithi134 ceph-mon[31966]: Active manager daemon x restarted 2024-07-26T11:00:43.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:42 smithi134 ceph-mon[31966]: Activating manager daemon x 2024-07-26T11:00:43.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:42 smithi134 ceph-mon[31966]: osdmap e2: 0 total, 0 up, 0 in 2024-07-26T11:00:43.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:42 smithi134 ceph-mon[31966]: mgrmap e6: x(active, starting, since 0.00614144s) 2024-07-26T11:00:43.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:42 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:00:43.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:42 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T11:00:43.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:42 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T11:00:43.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:42 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T11:00:43.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:42 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T11:00:43.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:42 smithi134 ceph-mon[31966]: Manager daemon x is now available 2024-07-26T11:00:43.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:42 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' 2024-07-26T11:00:43.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:42 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' 2024-07-26T11:00:44.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:43 smithi134 ceph-mon[31966]: Found migration_current of "None". Setting to last migration. 2024-07-26T11:00:44.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:43 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:44.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:43 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:44.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:43 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T11:00:44.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:43 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T11:00:44.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:43 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' 2024-07-26T11:00:44.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:43 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' 2024-07-26T11:00:44.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:43 smithi134 ceph-mon[31966]: mgrmap e7: x(active, since 1.01924s) 2024-07-26T11:00:44.350 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-07-26T11:00:44.350 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-07-26T11:00:44.350 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "initialized": true 2024-07-26T11:00:44.350 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-07-26T11:00:44.350 INFO:teuthology.orchestra.run.smithi134.stdout:mgr epoch 5 is available 2024-07-26T11:00:44.350 INFO:teuthology.orchestra.run.smithi134.stdout:Setting orchestrator backend to cephadm... 2024-07-26T11:00:44.673 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:00:44.352+0000 7ff88f200640 -1 log_channel(cephadm) log [ERR] : [26/Jul/2024:11:00:44] ENGINE Error in HTTPServer.serve 2024-07-26T11:00:44.673 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: Traceback (most recent call last): 2024-07-26T11:00:44.673 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T11:00:44.673 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: self._connections.run(self.expiration_interval) 2024-07-26T11:00:44.673 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T11:00:44.673 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: self._run(expiration_interval) 2024-07-26T11:00:44.674 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T11:00:44.674 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T11:00:44.674 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T11:00:44.674 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T11:00:44.674 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T11:00:44.674 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: s = self.context.wrap_socket( 2024-07-26T11:00:44.674 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T11:00:44.674 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: return self.sslsocket_class._create( 2024-07-26T11:00:44.674 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T11:00:44.674 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: self.do_handshake() 2024-07-26T11:00:44.674 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T11:00:44.674 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: self._sslobj.do_handshake() 2024-07-26T11:00:44.674 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T11:00:44.675 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:00:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-mon[31966]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-07-26T11:00:44.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-mon[31966]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-07-26T11:00:44.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:44 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:45.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:45 smithi134 ceph-mon[31966]: [26/Jul/2024:11:00:44] ENGINE Bus STARTING 2024-07-26T11:00:45.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:45 smithi134 ceph-mon[31966]: [26/Jul/2024:11:00:44] ENGINE Serving on http://172.21.15.134:8765 2024-07-26T11:00:45.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:45 smithi134 ceph-mon[31966]: [26/Jul/2024:11:00:44] ENGINE Serving on https://172.21.15.134:7150 2024-07-26T11:00:45.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:45 smithi134 ceph-mon[31966]: [26/Jul/2024:11:00:44] ENGINE Bus STARTED 2024-07-26T11:00:45.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:45 smithi134 ceph-mon[31966]: [26/Jul/2024:11:00:44] ENGINE Error in HTTPServer.serve 2024-07-26T11:00:45.923 INFO:journalctl@ceph.mon.a.smithi134.stdout: Traceback (most recent call last): 2024-07-26T11:00:45.923 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T11:00:45.923 INFO:journalctl@ceph.mon.a.smithi134.stdout: self._connections.run(self.expiration_interval) 2024-07-26T11:00:45.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T11:00:45.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: self._run(expiration_interval) 2024-07-26T11:00:45.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T11:00:45.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T11:00:45.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T11:00:45.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T11:00:45.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T11:00:45.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: s = self.context.wrap_socket( 2024-07-26T11:00:45.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T11:00:45.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: return self.sslsocket_class._create( 2024-07-26T11:00:45.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T11:00:45.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: self.do_handshake() 2024-07-26T11:00:45.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T11:00:45.925 INFO:journalctl@ceph.mon.a.smithi134.stdout: self._sslobj.do_handshake() 2024-07-26T11:00:45.925 INFO:journalctl@ceph.mon.a.smithi134.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T11:00:45.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:45 smithi134 ceph-mon[31966]: mgrmap e8: x(active, since 2s) 2024-07-26T11:00:45.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:45 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' 2024-07-26T11:00:45.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:45 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:47.126 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:46 smithi134 ceph-mon[31966]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:00:47.490 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout value unchanged 2024-07-26T11:00:47.491 INFO:teuthology.orchestra.run.smithi134.stdout:Generating ssh key... 2024-07-26T11:00:48.050 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:47 smithi134 ceph-mon[31966]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:00:48.924 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: Generating public/private rsa key pair. 2024-07-26T11:00:48.924 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: Your identification has been saved in /tmp/tmppdhh9215/key 2024-07-26T11:00:48.925 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: Your public key has been saved in /tmp/tmppdhh9215/key.pub 2024-07-26T11:00:48.925 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: The key fingerprint is: 2024-07-26T11:00:48.925 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: SHA256:VQhE00iNA+9dxFGrLkDDf8AnKQl0bKcKlAkeqsIZPIE ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:00:48.925 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: The key's randomart image is: 2024-07-26T11:00:48.925 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: +---[RSA 3072]----+ 2024-07-26T11:00:48.925 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: |..o. ++*B* o+o. | 2024-07-26T11:00:48.925 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: |Eo..+ =*+=+. . | 2024-07-26T11:00:48.925 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: |.+.. .B+* o . | 2024-07-26T11:00:48.925 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: |o + . o.* = . | 2024-07-26T11:00:48.925 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: |oo . .S o o | 2024-07-26T11:00:48.925 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: |. . . o | 2024-07-26T11:00:48.925 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: | . . | 2024-07-26T11:00:48.926 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: | . | 2024-07-26T11:00:48.926 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: | | 2024-07-26T11:00:48.926 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:00:48 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: +----[SHA256]-----+ 2024-07-26T11:00:49.637 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:49 smithi134 ceph-mon[31966]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:00:49.637 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:49 smithi134 ceph-mon[31966]: Generating ssh key... 2024-07-26T11:00:49.637 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:49 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' 2024-07-26T11:00:49.637 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:49 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' 2024-07-26T11:00:50.894 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:50 smithi134 ceph-mon[31966]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:00:50.905 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzHD4VQQ60Awpva1659trzOYeAaCHuHopzpgBhysn58MQMLSi5aN8+pdILo1BRSyZcC+vFTxjC1uXdjVCZAvGVzNH7Tf02a9A5A9KuQNPQ8l8wZPtR+WRlX8XVWNyx1nyeG7zWGErC3MKVpvWRy83ta/AgmOvhjHz5/UqH+9yXcGAYY7SCYHVvJgyNi900f1VJWYUUrgSirpJaZKpQtI8CKrtty6SfqHEbpTtXbwmSnb747bQJid1lB28RLSyNPAlPMyvc5/HnFkndA18EsuNaATPRu4G4dUPzXpNnKeEuLtrALqdapuvA0TtDDyy+aDAkbkfJtFGCaNWhaH4wLGt/qBTx7FAoPN71WYdWrA/UY4XHTyP91etthYK3rq3EjQwAwlkA4npQOzZu5ZJfsfcEaTTwUyAIwEhAuTDSvFew18jX5svFkvUUazN8KRr6isk4lqAbhphw32dRf2DZuYDSobqJtaZWUhw0GJJFKxJ/vE/Prrez5VBfAcBpYTwm9TU= ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:00:50.906 INFO:teuthology.orchestra.run.smithi134.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-07-26T11:00:50.906 INFO:teuthology.orchestra.run.smithi134.stdout:Adding key to root@localhost authorized_keys... 2024-07-26T11:00:50.906 INFO:teuthology.orchestra.run.smithi134.stdout:Adding host smithi134... 2024-07-26T11:00:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:52 smithi134 ceph-mon[31966]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi134", "addr": "172.21.15.134", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:00:54.070 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:53 smithi134 ceph-mon[31966]: Deploying cephadm binary to smithi134 2024-07-26T11:00:57.151 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Added host 'smithi134' with addr '172.21.15.134' 2024-07-26T11:00:57.151 INFO:teuthology.orchestra.run.smithi134.stdout:Deploying unmanaged mon service... 2024-07-26T11:00:57.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:57 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' 2024-07-26T11:00:57.706 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:57 smithi134 ceph-mon[31966]: Added host smithi134 2024-07-26T11:00:57.706 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:57 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:00:58.684 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-07-26T11:00:58.685 INFO:teuthology.orchestra.run.smithi134.stdout:Deploying unmanaged mgr service... 2024-07-26T11:00:59.217 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:59 smithi134 ceph-mon[31966]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:00:59.217 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:59 smithi134 ceph-mon[31966]: Saving service mon spec with placement count:5 2024-07-26T11:00:59.217 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:00:59 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' 2024-07-26T11:01:00.941 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-07-26T11:01:01.234 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:01 smithi134 ceph-mon[31966]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:01.234 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:01 smithi134 ceph-mon[31966]: Saving service mgr spec with placement count:2 2024-07-26T11:01:01.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:01 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' 2024-07-26T11:01:01.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:01 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' 2024-07-26T11:01:01.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:01 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' 2024-07-26T11:01:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:02 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2181776766' entity='client.admin' 2024-07-26T11:01:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:02 smithi134 ceph-mon[31966]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:01:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:04 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' 2024-07-26T11:01:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:04 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3233948246' entity='client.admin' 2024-07-26T11:01:04.783 INFO:teuthology.orchestra.run.smithi134.stdout:Enabling the dashboard module... 2024-07-26T11:01:05.745 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:05 smithi134 ceph-mon[31966]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:01:05.746 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:05 smithi134 ceph-mon[31966]: from='mgr.14118 172.21.15.134:0/2699593273' entity='mgr.x' 2024-07-26T11:01:06.600 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:06 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1591558678' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-07-26T11:01:06.922 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:06 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: ignoring --setuser ceph since I am not root 2024-07-26T11:01:06.922 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:06 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: ignoring --setgroup ceph since I am not root 2024-07-26T11:01:07.659 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:07 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1591558678' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-07-26T11:01:07.660 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:07 smithi134 ceph-mon[31966]: mgrmap e9: x(active, since 23s) 2024-07-26T11:01:08.920 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:08 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/569692089' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-07-26T11:01:08.930 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-07-26T11:01:08.930 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-07-26T11:01:08.930 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "available": true, 2024-07-26T11:01:08.931 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-07-26T11:01:08.931 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-07-26T11:01:08.931 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-07-26T11:01:08.931 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for the mgr to restart... 2024-07-26T11:01:08.931 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mgr epoch 9... 2024-07-26T11:01:13.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:12 smithi134 ceph-mon[31966]: Active manager daemon x restarted 2024-07-26T11:01:13.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:12 smithi134 ceph-mon[31966]: Activating manager daemon x 2024-07-26T11:01:13.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:12 smithi134 ceph-mon[31966]: osdmap e3: 0 total, 0 up, 0 in 2024-07-26T11:01:13.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:12 smithi134 ceph-mon[31966]: mgrmap e10: x(active, starting, since 0.0100012s) 2024-07-26T11:01:13.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:12 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:01:13.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:12 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T11:01:13.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:12 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T11:01:13.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:12 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T11:01:13.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:12 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T11:01:13.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:12 smithi134 ceph-mon[31966]: Manager daemon x is now available 2024-07-26T11:01:13.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:12 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:13.960 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:13 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T11:01:13.960 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:13 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T11:01:13.961 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:13 smithi134 ceph-mon[31966]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-07-26T11:01:13.961 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:13 smithi134 ceph-mon[31966]: mgrmap e11: x(active, since 1.01054s) 2024-07-26T11:01:13.961 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:13 smithi134 ceph-mon[31966]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-07-26T11:01:14.751 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:01:14.658+0000 7f7ff8800640 -1 log_channel(cephadm) log [ERR] : [26/Jul/2024:11:01:14] ENGINE Error in HTTPServer.serve 2024-07-26T11:01:14.752 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: Traceback (most recent call last): 2024-07-26T11:01:14.752 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T11:01:14.752 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: self._connections.run(self.expiration_interval) 2024-07-26T11:01:14.752 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T11:01:14.752 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: self._run(expiration_interval) 2024-07-26T11:01:14.752 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T11:01:14.752 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T11:01:14.752 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T11:01:14.753 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T11:01:14.753 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T11:01:14.753 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: s = self.context.wrap_socket( 2024-07-26T11:01:14.753 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T11:01:14.753 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: return self.sslsocket_class._create( 2024-07-26T11:01:14.753 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T11:01:14.753 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: self.do_handshake() 2024-07-26T11:01:14.753 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T11:01:14.753 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: self._sslobj.do_handshake() 2024-07-26T11:01:14.754 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T11:01:14.754 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:01:15.163 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-mon[31966]: [26/Jul/2024:11:01:14] ENGINE Bus STARTING 2024-07-26T11:01:15.163 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-mon[31966]: [26/Jul/2024:11:01:14] ENGINE Serving on http://172.21.15.134:8765 2024-07-26T11:01:15.164 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-mon[31966]: [26/Jul/2024:11:01:14] ENGINE Serving on https://172.21.15.134:7150 2024-07-26T11:01:15.164 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-mon[31966]: [26/Jul/2024:11:01:14] ENGINE Bus STARTED 2024-07-26T11:01:15.164 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:14 smithi134 ceph-mon[31966]: [26/Jul/2024:11:01:14] ENGINE Error in HTTPServer.serve 2024-07-26T11:01:15.164 INFO:journalctl@ceph.mon.a.smithi134.stdout: Traceback (most recent call last): 2024-07-26T11:01:15.164 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T11:01:15.164 INFO:journalctl@ceph.mon.a.smithi134.stdout: self._connections.run(self.expiration_interval) 2024-07-26T11:01:15.164 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T11:01:15.164 INFO:journalctl@ceph.mon.a.smithi134.stdout: self._run(expiration_interval) 2024-07-26T11:01:15.164 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T11:01:15.164 INFO:journalctl@ceph.mon.a.smithi134.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T11:01:15.164 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T11:01:15.164 INFO:journalctl@ceph.mon.a.smithi134.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T11:01:15.164 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T11:01:15.165 INFO:journalctl@ceph.mon.a.smithi134.stdout: s = self.context.wrap_socket( 2024-07-26T11:01:15.165 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T11:01:15.165 INFO:journalctl@ceph.mon.a.smithi134.stdout: return self.sslsocket_class._create( 2024-07-26T11:01:15.165 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T11:01:15.165 INFO:journalctl@ceph.mon.a.smithi134.stdout: self.do_handshake() 2024-07-26T11:01:15.165 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T11:01:15.165 INFO:journalctl@ceph.mon.a.smithi134.stdout: self._sslobj.do_handshake() 2024-07-26T11:01:15.165 INFO:journalctl@ceph.mon.a.smithi134.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T11:01:15.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-07-26T11:01:15.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-07-26T11:01:15.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "initialized": true 2024-07-26T11:01:15.171 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-07-26T11:01:15.171 INFO:teuthology.orchestra.run.smithi134.stdout:mgr epoch 9 is available 2024-07-26T11:01:15.172 INFO:teuthology.orchestra.run.smithi134.stdout:Generating a dashboard self-signed certificate... 2024-07-26T11:01:16.114 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:15 smithi134 ceph-mon[31966]: mgrmap e12: x(active, since 2s) 2024-07-26T11:01:16.114 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:15 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:16.843 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-07-26T11:01:16.843 INFO:teuthology.orchestra.run.smithi134.stdout:Creating initial admin user... 2024-07-26T11:01:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:17 smithi134 ceph-mon[31966]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:17 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:17 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:17 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:19 smithi134 ceph-mon[31966]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:19 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:19.368 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$.oai385.Brdovz.fu.lJZ.8k5G.d2SAur7ypyyXDq9O5GQC69xrFa", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1721991678, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-07-26T11:01:19.369 INFO:teuthology.orchestra.run.smithi134.stdout:Fetching dashboard port number... 2024-07-26T11:01:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:20 smithi134 ceph-mon[31966]: mgrmap e13: x(active, since 6s) 2024-07-26T11:01:20.893 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 8443 2024-07-26T11:01:20.926 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-07-26T11:01:20.926 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout disabled 2024-07-26T11:01:20.951 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-07-26T11:01:20.951 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-07-26T11:01:20.951 INFO:teuthology.orchestra.run.smithi134.stdout:firewalld.service is not enabled 2024-07-26T11:01:20.951 INFO:teuthology.orchestra.run.smithi134.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-07-26T11:01:20.958 INFO:teuthology.orchestra.run.smithi134.stdout:Ceph Dashboard is now available at: 2024-07-26T11:01:20.958 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:01:20.958 INFO:teuthology.orchestra.run.smithi134.stdout: URL: https://smithi134.front.sepia.ceph.com:8443/ 2024-07-26T11:01:20.958 INFO:teuthology.orchestra.run.smithi134.stdout: User: admin 2024-07-26T11:01:20.958 INFO:teuthology.orchestra.run.smithi134.stdout: Password: wn9utwrtfk 2024-07-26T11:01:20.958 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:01:20.961 INFO:teuthology.orchestra.run.smithi134.stdout:Saving cluster configuration to /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config directory 2024-07-26T11:01:21.140 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:21 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3397512813' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-07-26T11:01:22.608 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-07-26T11:01:22.608 INFO:teuthology.orchestra.run.smithi134.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-07-26T11:01:22.608 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:01:22.608 INFO:teuthology.orchestra.run.smithi134.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-07-26T11:01:22.608 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:01:22.608 INFO:teuthology.orchestra.run.smithi134.stdout:Or, if you are only running a single cluster on this host: 2024-07-26T11:01:22.608 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:01:22.608 INFO:teuthology.orchestra.run.smithi134.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-07-26T11:01:22.608 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:01:22.608 INFO:teuthology.orchestra.run.smithi134.stdout:Please consider enabling telemetry to help improve Ceph: 2024-07-26T11:01:22.608 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:01:22.609 INFO:teuthology.orchestra.run.smithi134.stdout: ceph telemetry on 2024-07-26T11:01:22.609 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:01:22.609 INFO:teuthology.orchestra.run.smithi134.stdout:For more information see: 2024-07-26T11:01:22.609 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:01:22.609 INFO:teuthology.orchestra.run.smithi134.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-07-26T11:01:22.609 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:01:22.609 INFO:teuthology.orchestra.run.smithi134.stdout:Bootstrap complete. 2024-07-26T11:01:22.714 INFO:tasks.cephadm:Fetching config... 2024-07-26T11:01:22.715 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T11:01:22.715 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-07-26T11:01:22.766 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-07-26T11:01:22.766 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T11:01:22.766 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-07-26T11:01:22.807 INFO:tasks.cephadm:Fetching mon keyring... 2024-07-26T11:01:22.808 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T11:01:22.808 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/keyring of=/dev/stdout 2024-07-26T11:01:22.936 INFO:tasks.cephadm:Fetching pub ssh key... 2024-07-26T11:01:22.936 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T11:01:22.936 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-07-26T11:01:22.983 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-07-26T11:01:22.983 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzHD4VQQ60Awpva1659trzOYeAaCHuHopzpgBhysn58MQMLSi5aN8+pdILo1BRSyZcC+vFTxjC1uXdjVCZAvGVzNH7Tf02a9A5A9KuQNPQ8l8wZPtR+WRlX8XVWNyx1nyeG7zWGErC3MKVpvWRy83ta/AgmOvhjHz5/UqH+9yXcGAYY7SCYHVvJgyNi900f1VJWYUUrgSirpJaZKpQtI8CKrtty6SfqHEbpTtXbwmSnb747bQJid1lB28RLSyNPAlPMyvc5/HnFkndA18EsuNaATPRu4G4dUPzXpNnKeEuLtrALqdapuvA0TtDDyy+aDAkbkfJtFGCaNWhaH4wLGt/qBTx7FAoPN71WYdWrA/UY4XHTyP91etthYK3rq3EjQwAwlkA4npQOzZu5ZJfsfcEaTTwUyAIwEhAuTDSvFew18jX5svFkvUUazN8KRr6isk4lqAbhphw32dRf2DZuYDSobqJtaZWUhw0GJJFKxJ/vE/Prrez5VBfAcBpYTwm9TU= ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T11:01:23.128 INFO:teuthology.orchestra.run.smithi134.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzHD4VQQ60Awpva1659trzOYeAaCHuHopzpgBhysn58MQMLSi5aN8+pdILo1BRSyZcC+vFTxjC1uXdjVCZAvGVzNH7Tf02a9A5A9KuQNPQ8l8wZPtR+WRlX8XVWNyx1nyeG7zWGErC3MKVpvWRy83ta/AgmOvhjHz5/UqH+9yXcGAYY7SCYHVvJgyNi900f1VJWYUUrgSirpJaZKpQtI8CKrtty6SfqHEbpTtXbwmSnb747bQJid1lB28RLSyNPAlPMyvc5/HnFkndA18EsuNaATPRu4G4dUPzXpNnKeEuLtrALqdapuvA0TtDDyy+aDAkbkfJtFGCaNWhaH4wLGt/qBTx7FAoPN71WYdWrA/UY4XHTyP91etthYK3rq3EjQwAwlkA4npQOzZu5ZJfsfcEaTTwUyAIwEhAuTDSvFew18jX5svFkvUUazN8KRr6isk4lqAbhphw32dRf2DZuYDSobqJtaZWUhw0GJJFKxJ/vE/Prrez5VBfAcBpYTwm9TU= ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:01:23.157 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzHD4VQQ60Awpva1659trzOYeAaCHuHopzpgBhysn58MQMLSi5aN8+pdILo1BRSyZcC+vFTxjC1uXdjVCZAvGVzNH7Tf02a9A5A9KuQNPQ8l8wZPtR+WRlX8XVWNyx1nyeG7zWGErC3MKVpvWRy83ta/AgmOvhjHz5/UqH+9yXcGAYY7SCYHVvJgyNi900f1VJWYUUrgSirpJaZKpQtI8CKrtty6SfqHEbpTtXbwmSnb747bQJid1lB28RLSyNPAlPMyvc5/HnFkndA18EsuNaATPRu4G4dUPzXpNnKeEuLtrALqdapuvA0TtDDyy+aDAkbkfJtFGCaNWhaH4wLGt/qBTx7FAoPN71WYdWrA/UY4XHTyP91etthYK3rq3EjQwAwlkA4npQOzZu5ZJfsfcEaTTwUyAIwEhAuTDSvFew18jX5svFkvUUazN8KRr6isk4lqAbhphw32dRf2DZuYDSobqJtaZWUhw0GJJFKxJ/vE/Prrez5VBfAcBpYTwm9TU= ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T11:01:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:22 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3700146822' entity='client.admin' 2024-07-26T11:01:23.199 INFO:teuthology.orchestra.run.smithi189.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzHD4VQQ60Awpva1659trzOYeAaCHuHopzpgBhysn58MQMLSi5aN8+pdILo1BRSyZcC+vFTxjC1uXdjVCZAvGVzNH7Tf02a9A5A9KuQNPQ8l8wZPtR+WRlX8XVWNyx1nyeG7zWGErC3MKVpvWRy83ta/AgmOvhjHz5/UqH+9yXcGAYY7SCYHVvJgyNi900f1VJWYUUrgSirpJaZKpQtI8CKrtty6SfqHEbpTtXbwmSnb747bQJid1lB28RLSyNPAlPMyvc5/HnFkndA18EsuNaATPRu4G4dUPzXpNnKeEuLtrALqdapuvA0TtDDyy+aDAkbkfJtFGCaNWhaH4wLGt/qBTx7FAoPN71WYdWrA/UY4XHTyP91etthYK3rq3EjQwAwlkA4npQOzZu5ZJfsfcEaTTwUyAIwEhAuTDSvFew18jX5svFkvUUazN8KRr6isk4lqAbhphw32dRf2DZuYDSobqJtaZWUhw0GJJFKxJ/vE/Prrez5VBfAcBpYTwm9TU= ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:01:23.211 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzHD4VQQ60Awpva1659trzOYeAaCHuHopzpgBhysn58MQMLSi5aN8+pdILo1BRSyZcC+vFTxjC1uXdjVCZAvGVzNH7Tf02a9A5A9KuQNPQ8l8wZPtR+WRlX8XVWNyx1nyeG7zWGErC3MKVpvWRy83ta/AgmOvhjHz5/UqH+9yXcGAYY7SCYHVvJgyNi900f1VJWYUUrgSirpJaZKpQtI8CKrtty6SfqHEbpTtXbwmSnb747bQJid1lB28RLSyNPAlPMyvc5/HnFkndA18EsuNaATPRu4G4dUPzXpNnKeEuLtrALqdapuvA0TtDDyy+aDAkbkfJtFGCaNWhaH4wLGt/qBTx7FAoPN71WYdWrA/UY4XHTyP91etthYK3rq3EjQwAwlkA4npQOzZu5ZJfsfcEaTTwUyAIwEhAuTDSvFew18jX5svFkvUUazN8KRr6isk4lqAbhphw32dRf2DZuYDSobqJtaZWUhw0GJJFKxJ/vE/Prrez5VBfAcBpYTwm9TU= ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-07-26T11:01:23.254 INFO:teuthology.orchestra.run.smithi193.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzHD4VQQ60Awpva1659trzOYeAaCHuHopzpgBhysn58MQMLSi5aN8+pdILo1BRSyZcC+vFTxjC1uXdjVCZAvGVzNH7Tf02a9A5A9KuQNPQ8l8wZPtR+WRlX8XVWNyx1nyeG7zWGErC3MKVpvWRy83ta/AgmOvhjHz5/UqH+9yXcGAYY7SCYHVvJgyNi900f1VJWYUUrgSirpJaZKpQtI8CKrtty6SfqHEbpTtXbwmSnb747bQJid1lB28RLSyNPAlPMyvc5/HnFkndA18EsuNaATPRu4G4dUPzXpNnKeEuLtrALqdapuvA0TtDDyy+aDAkbkfJtFGCaNWhaH4wLGt/qBTx7FAoPN71WYdWrA/UY4XHTyP91etthYK3rq3EjQwAwlkA4npQOzZu5ZJfsfcEaTTwUyAIwEhAuTDSvFew18jX5svFkvUUazN8KRr6isk4lqAbhphw32dRf2DZuYDSobqJtaZWUhw0GJJFKxJ/vE/Prrez5VBfAcBpYTwm9TU= ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:01:23.266 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-07-26T11:01:23.706 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:26.858 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:26 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:26.859 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:26 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:26.859 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:26 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi134", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:01:26.859 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:26 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:26.859 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:26 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:26.859 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:26 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:27.610 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-07-26T11:01:27.611 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-07-26T11:01:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:27 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/4270434766' entity='client.admin' 2024-07-26T11:01:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:27 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:27 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:28.344 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:31.483 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi189 2024-07-26T11:01:31.483 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-07-26T11:01:31.483 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.conf 2024-07-26T11:01:31.502 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-07-26T11:01:31.502 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:01:31.559 INFO:tasks.cephadm:Adding host smithi189 to orchestrator... 2024-07-26T11:01:31.560 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch host add smithi189 2024-07-26T11:01:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:31 smithi134 ceph-mon[31966]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:31 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:31 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:31 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:31 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:01:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:31 smithi134 ceph-mon[31966]: Updating smithi134:/etc/ceph/ceph.conf 2024-07-26T11:01:31.970 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:32 smithi134 ceph-mon[31966]: Updating smithi134:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:01:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:32 smithi134 ceph-mon[31966]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:01:33.811 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:33 smithi134 ceph-mon[31966]: Updating smithi134:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T11:01:33.811 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:33 smithi134 ceph-mon[31966]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:01:33.811 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:33 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:33.811 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:33 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:33.811 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:33 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:34 smithi134 ceph-mon[31966]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:34 smithi134 ceph-mon[31966]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:01:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:35 smithi134 ceph-mon[31966]: Deploying cephadm binary to smithi189 2024-07-26T11:01:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:36 smithi134 ceph-mon[31966]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:01:37.739 INFO:teuthology.orchestra.run.smithi134.stdout:Added host 'smithi189' with addr '172.21.15.189' 2024-07-26T11:01:38.464 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch host ls --format=json 2024-07-26T11:01:38.908 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:39.054 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:38 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:39.054 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:38 smithi134 ceph-mon[31966]: Added host smithi189 2024-07-26T11:01:39.054 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:38 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:39.054 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:38 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:39.054 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:38 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:39.054 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:38 smithi134 ceph-mon[31966]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:01:40.343 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:40 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:41.097 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:01:41.097 INFO:teuthology.orchestra.run.smithi134.stdout:[{"addr": "172.21.15.134", "hostname": "smithi134", "labels": [], "status": ""}, {"addr": "172.21.15.189", "hostname": "smithi189", "labels": [], "status": ""}] 2024-07-26T11:01:41.400 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:41 smithi134 ceph-mon[31966]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:01:41.763 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi193 2024-07-26T11:01:41.763 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-07-26T11:01:41.763 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.conf 2024-07-26T11:01:41.781 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-07-26T11:01:41.781 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:01:41.838 INFO:tasks.cephadm:Adding host smithi193 to orchestrator... 2024-07-26T11:01:41.838 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch host add smithi193 2024-07-26T11:01:42.286 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:42.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:42 smithi134 ceph-mon[31966]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:01:42.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:42 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:42.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:42 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:42.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:42 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:42.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:42 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:42.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:42 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:01:42.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:42 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:42.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:42 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:01:43.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:43 smithi134 ceph-mon[31966]: Updating smithi189:/etc/ceph/ceph.conf 2024-07-26T11:01:43.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:43 smithi134 ceph-mon[31966]: Updating smithi189:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:01:43.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:43 smithi134 ceph-mon[31966]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:01:43.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:43 smithi134 ceph-mon[31966]: Updating smithi189:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T11:01:43.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:43 smithi134 ceph-mon[31966]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:01:43.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:43 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:43.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:43 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:43.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:43 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:45.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:44 smithi134 ceph-mon[31966]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:45.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:44 smithi134 ceph-mon[31966]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:01:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:45 smithi134 ceph-mon[31966]: Deploying cephadm binary to smithi193 2024-07-26T11:01:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:46 smithi134 ceph-mon[31966]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:01:48.087 INFO:teuthology.orchestra.run.smithi134.stdout:Added host 'smithi193' with addr '172.21.15.193' 2024-07-26T11:01:48.934 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch host ls --format=json 2024-07-26T11:01:49.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:49 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:49.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:49 smithi134 ceph-mon[31966]: Added host smithi193 2024-07-26T11:01:49.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:49 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:49.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:49 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:49.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:49 smithi134 ceph-mon[31966]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:01:49.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:49 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:49.373 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:50.535 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:50 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:51.574 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:01:51.574 INFO:teuthology.orchestra.run.smithi134.stdout:[{"addr": "172.21.15.134", "hostname": "smithi134", "labels": [], "status": ""}, {"addr": "172.21.15.189", "hostname": "smithi189", "labels": [], "status": ""}, {"addr": "172.21.15.193", "hostname": "smithi193", "labels": [], "status": ""}] 2024-07-26T11:01:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:51 smithi134 ceph-mon[31966]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:01:52.294 INFO:tasks.cephadm:Setting crush tunables to default 2024-07-26T11:01:52.294 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd crush tunables default 2024-07-26T11:01:52.729 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:01:52.877 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:52 smithi134 ceph-mon[31966]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:01:52.877 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:52 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:52.877 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:52 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:52.877 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:52 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:52.877 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:52 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:52.877 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:52 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:01:52.877 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:52 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:52.877 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:52 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:01:53.754 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:53 smithi134 ceph-mon[31966]: Updating smithi193:/etc/ceph/ceph.conf 2024-07-26T11:01:53.754 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:53 smithi134 ceph-mon[31966]: Updating smithi193:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:01:53.754 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:53 smithi134 ceph-mon[31966]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:01:53.754 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:53 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:53.754 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:53 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:53.755 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:53 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:54.588 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:54 smithi134 ceph-mon[31966]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:01:54.588 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:54 smithi134 ceph-mon[31966]: Updating smithi193:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T11:01:55.501 INFO:teuthology.orchestra.run.smithi134.stderr:adjusted tunables profile to default 2024-07-26T11:01:55.846 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:55 smithi134 ceph-mon[31966]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:01:55.846 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:55 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2352598432' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-26T11:01:56.199 INFO:tasks.cephadm:Adding mon.a on smithi134 2024-07-26T11:01:56.199 INFO:tasks.cephadm:Adding mon.b on smithi189 2024-07-26T11:01:56.200 INFO:tasks.cephadm:Adding mon.c on smithi193 2024-07-26T11:01:56.200 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch apply mon '3;smithi134:172.21.15.134=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c' 2024-07-26T11:01:56.449 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:01:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:56 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2352598432' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-26T11:01:56.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:56 smithi134 ceph-mon[31966]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T11:01:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:57 smithi134 ceph-mon[31966]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:01:58.138 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mon update... 2024-07-26T11:01:58.730 DEBUG:teuthology.orchestra.run.smithi189:mon.b> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mon.b.service 2024-07-26T11:01:58.733 DEBUG:teuthology.orchestra.run.smithi193:mon.c> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mon.c.service 2024-07-26T11:01:58.735 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T11:01:58.735 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph mon dump -f json 2024-07-26T11:01:59.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:59 smithi134 ceph-mon[31966]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi134:172.21.15.134=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:01:59.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:59 smithi134 ceph-mon[31966]: Saving service mon spec with placement smithi134:172.21.15.134=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c;count:3 2024-07-26T11:01:59.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:59 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:59.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:59 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:01:59.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:59 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:59.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:59 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:01:59.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:59 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:01:59.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:59 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T11:01:59.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:59 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:01:59.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:59 smithi134 ceph-mon[31966]: Deploying daemon mon.c on smithi193 2024-07-26T11:01:59.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:01:59 smithi134 ceph-mon[31966]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:01:59.442 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:00 smithi134 ceph-mon[31966]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:02.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:02 smithi193 systemd[1]: Starting Ceph mon.c for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:02:02.692 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T11:02:02.692 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"105fbe5c-4b3e-11ef-bcb7-c7b262605968","modified":"2024-07-26T11:00:01.653390Z","created":"2024-07-26T11:00:01.653390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-07-26T11:02:02.692 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-07-26T11:02:03.020 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:02 smithi193 podman[37437]: 2024-07-26 11:02:02.569503769 +0000 UTC m=+0.017315389 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:02:03.020 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:02 smithi193 podman[37437]: 2024-07-26 11:02:02.672859368 +0000 UTC m=+0.120670975 container create d84de133b4202e9c7063144f055a22c7b0c83b605869f3ccfce6390913dfd558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mon-c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:02:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:02 smithi134 ceph-mon[31966]: from='client.? 172.21.15.193:0/4219511000' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T11:02:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:02 smithi134 ceph-mon[31966]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:03.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 podman[37437]: 2024-07-26 11:02:03.446311981 +0000 UTC m=+0.894123605 container init d84de133b4202e9c7063144f055a22c7b0c83b605869f3ccfce6390913dfd558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mon-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , ceph=True) 2024-07-26T11:02:03.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 podman[37437]: 2024-07-26 11:02:03.449564006 +0000 UTC m=+0.897375613 container start d84de133b4202e9c7063144f055a22c7b0c83b605869f3ccfce6390913dfd558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-07-26T11:02:03.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: set uid:gid to 167:167 (ceph:ceph) 2024-07-26T11:02:03.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev), process ceph-mon, pid 2 2024-07-26T11:02:03.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: pidfile_write: ignore empty --pid-file 2024-07-26T11:02:03.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: load: jerasure load: lrc 2024-07-26T11:02:03.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: RocksDB version: 7.9.2 2024-07-26T11:02:03.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Git sha 0 2024-07-26T11:02:03.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Compile date 2024-07-25 13:49:38 2024-07-26T11:02:03.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: DB SUMMARY 2024-07-26T11:02:03.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: DB Session ID: OVAVJUVOQFDUGPR9DPK0 2024-07-26T11:02:03.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: CURRENT file: CURRENT 2024-07-26T11:02:03.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: IDENTITY file: IDENTITY 2024-07-26T11:02:03.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-07-26T11:02:03.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-07-26T11:02:03.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-07-26T11:02:03.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.error_if_exists: 0 2024-07-26T11:02:03.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.create_if_missing: 0 2024-07-26T11:02:03.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.paranoid_checks: 1 2024-07-26T11:02:03.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.flush_verify_memtable_count: 1 2024-07-26T11:02:03.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-07-26T11:02:03.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-07-26T11:02:03.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.env: 0x556c3c44b280 2024-07-26T11:02:03.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.fs: PosixFileSystem 2024-07-26T11:02:03.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.info_log: 0x556c41663080 2024-07-26T11:02:03.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_file_opening_threads: 16 2024-07-26T11:02:03.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.statistics: (nil) 2024-07-26T11:02:03.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.use_fsync: 0 2024-07-26T11:02:03.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_log_file_size: 0 2024-07-26T11:02:03.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-07-26T11:02:03.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.log_file_time_to_roll: 0 2024-07-26T11:02:03.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.keep_log_file_num: 1000 2024-07-26T11:02:03.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.recycle_log_file_num: 0 2024-07-26T11:02:03.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.allow_fallocate: 1 2024-07-26T11:02:03.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.allow_mmap_reads: 0 2024-07-26T11:02:03.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.allow_mmap_writes: 0 2024-07-26T11:02:03.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.use_direct_reads: 0 2024-07-26T11:02:03.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-07-26T11:02:03.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.create_missing_column_families: 0 2024-07-26T11:02:03.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.db_log_dir: 2024-07-26T11:02:03.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.wal_dir: 2024-07-26T11:02:03.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.table_cache_numshardbits: 6 2024-07-26T11:02:03.779 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.WAL_ttl_seconds: 0 2024-07-26T11:02:03.779 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.WAL_size_limit_MB: 0 2024-07-26T11:02:03.779 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-07-26T11:02:03.779 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-07-26T11:02:03.779 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.is_fd_close_on_exec: 1 2024-07-26T11:02:03.779 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.advise_random_on_open: 1 2024-07-26T11:02:03.779 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.db_write_buffer_size: 0 2024-07-26T11:02:03.779 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.write_buffer_manager: 0x556c416675e0 2024-07-26T11:02:03.779 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-07-26T11:02:03.779 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-07-26T11:02:03.779 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.use_adaptive_mutex: 0 2024-07-26T11:02:03.779 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.rate_limiter: (nil) 2024-07-26T11:02:03.779 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-07-26T11:02:03.780 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.wal_recovery_mode: 2 2024-07-26T11:02:03.780 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.enable_thread_tracking: 0 2024-07-26T11:02:03.780 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.enable_pipelined_write: 0 2024-07-26T11:02:03.780 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.unordered_write: 0 2024-07-26T11:02:03.780 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-07-26T11:02:03.780 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-07-26T11:02:03.780 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-07-26T11:02:03.780 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-07-26T11:02:03.780 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.row_cache: None 2024-07-26T11:02:03.780 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.wal_filter: None 2024-07-26T11:02:03.782 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-07-26T11:02:03.782 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.allow_ingest_behind: 0 2024-07-26T11:02:03.782 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.two_write_queues: 0 2024-07-26T11:02:03.782 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.manual_wal_flush: 0 2024-07-26T11:02:03.783 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.wal_compression: 0 2024-07-26T11:02:03.783 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.atomic_flush: 0 2024-07-26T11:02:03.783 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-07-26T11:02:03.783 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.persist_stats_to_disk: 0 2024-07-26T11:02:03.783 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.write_dbid_to_manifest: 0 2024-07-26T11:02:03.783 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.log_readahead_size: 0 2024-07-26T11:02:03.783 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-07-26T11:02:03.783 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.best_efforts_recovery: 0 2024-07-26T11:02:03.783 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-07-26T11:02:03.783 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-07-26T11:02:03.783 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.allow_data_in_errors: 0 2024-07-26T11:02:03.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.db_host_id: __hostname__ 2024-07-26T11:02:03.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.enforce_single_del_contracts: true 2024-07-26T11:02:03.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_background_jobs: 2 2024-07-26T11:02:03.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_background_compactions: -1 2024-07-26T11:02:03.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_subcompactions: 1 2024-07-26T11:02:03.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-07-26T11:02:03.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-07-26T11:02:03.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.delayed_write_rate : 16777216 2024-07-26T11:02:03.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_total_wal_size: 0 2024-07-26T11:02:03.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-07-26T11:02:03.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.stats_dump_period_sec: 600 2024-07-26T11:02:03.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.stats_persist_period_sec: 600 2024-07-26T11:02:03.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-07-26T11:02:03.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_open_files: -1 2024-07-26T11:02:03.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.bytes_per_sync: 0 2024-07-26T11:02:03.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.wal_bytes_per_sync: 0 2024-07-26T11:02:03.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.strict_bytes_per_sync: 0 2024-07-26T11:02:03.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compaction_readahead_size: 0 2024-07-26T11:02:03.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_background_flushes: -1 2024-07-26T11:02:03.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Compression algorithms supported: 2024-07-26T11:02:03.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: kZSTD supported: 0 2024-07-26T11:02:03.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: kXpressCompression supported: 0 2024-07-26T11:02:03.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: kBZip2Compression supported: 0 2024-07-26T11:02:03.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-07-26T11:02:03.786 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: kLZ4Compression supported: 1 2024-07-26T11:02:03.786 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: kZlibCompression supported: 1 2024-07-26T11:02:03.786 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: kLZ4HCCompression supported: 1 2024-07-26T11:02:03.786 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: kSnappyCompression supported: 1 2024-07-26T11:02:03.786 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Fast CRC32 supported: Supported on x86 2024-07-26T11:02:03.786 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: DMutex implementation: pthread_mutex_t 2024-07-26T11:02:03.786 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-07-26T11:02:03.786 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-07-26T11:02:03.786 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-07-26T11:02:03.786 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.merge_operator: 2024-07-26T11:02:03.786 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compaction_filter: None 2024-07-26T11:02:03.787 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compaction_filter_factory: None 2024-07-26T11:02:03.787 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.sst_partitioner_factory: None 2024-07-26T11:02:03.787 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.memtable_factory: SkipListFactory 2024-07-26T11:02:03.787 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.table_factory: BlockBasedTable 2024-07-26T11:02:03.787 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556c41662c80) 2024-07-26T11:02:03.787 INFO:journalctl@ceph.mon.c.smithi193.stdout: cache_index_and_filter_blocks: 1 2024-07-26T11:02:03.787 INFO:journalctl@ceph.mon.c.smithi193.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-07-26T11:02:03.787 INFO:journalctl@ceph.mon.c.smithi193.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-07-26T11:02:03.787 INFO:journalctl@ceph.mon.c.smithi193.stdout: pin_top_level_index_and_filter: 1 2024-07-26T11:02:03.787 INFO:journalctl@ceph.mon.c.smithi193.stdout: index_type: 0 2024-07-26T11:02:03.787 INFO:journalctl@ceph.mon.c.smithi193.stdout: data_block_index_type: 0 2024-07-26T11:02:03.787 INFO:journalctl@ceph.mon.c.smithi193.stdout: index_shortening: 1 2024-07-26T11:02:03.788 INFO:journalctl@ceph.mon.c.smithi193.stdout: data_block_hash_table_util_ratio: 0.750000 2024-07-26T11:02:03.788 INFO:journalctl@ceph.mon.c.smithi193.stdout: checksum: 4 2024-07-26T11:02:03.788 INFO:journalctl@ceph.mon.c.smithi193.stdout: no_block_cache: 0 2024-07-26T11:02:03.788 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_cache: 0x556c3d627610 2024-07-26T11:02:03.788 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_cache_name: BinnedLRUCache 2024-07-26T11:02:03.788 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_cache_options: 2024-07-26T11:02:03.788 INFO:journalctl@ceph.mon.c.smithi193.stdout: capacity : 536870912 2024-07-26T11:02:03.788 INFO:journalctl@ceph.mon.c.smithi193.stdout: num_shard_bits : 4 2024-07-26T11:02:03.788 INFO:journalctl@ceph.mon.c.smithi193.stdout: strict_capacity_limit : 0 2024-07-26T11:02:03.788 INFO:journalctl@ceph.mon.c.smithi193.stdout: high_pri_pool_ratio: 0.000 2024-07-26T11:02:03.788 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_cache_compressed: (nil) 2024-07-26T11:02:03.788 INFO:journalctl@ceph.mon.c.smithi193.stdout: persistent_cache: (nil) 2024-07-26T11:02:03.788 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_size: 4096 2024-07-26T11:02:03.788 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_size_deviation: 10 2024-07-26T11:02:03.789 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_restart_interval: 16 2024-07-26T11:02:03.789 INFO:journalctl@ceph.mon.c.smithi193.stdout: index_block_restart_interval: 1 2024-07-26T11:02:03.789 INFO:journalctl@ceph.mon.c.smithi193.stdout: metadata_block_size: 4096 2024-07-26T11:02:03.789 INFO:journalctl@ceph.mon.c.smithi193.stdout: partition_filters: 0 2024-07-26T11:02:03.789 INFO:journalctl@ceph.mon.c.smithi193.stdout: use_delta_encoding: 1 2024-07-26T11:02:03.789 INFO:journalctl@ceph.mon.c.smithi193.stdout: filter_policy: bloomfilter 2024-07-26T11:02:03.789 INFO:journalctl@ceph.mon.c.smithi193.stdout: whole_key_filtering: 1 2024-07-26T11:02:03.789 INFO:journalctl@ceph.mon.c.smithi193.stdout: verify_compression: 0 2024-07-26T11:02:03.789 INFO:journalctl@ceph.mon.c.smithi193.stdout: read_amp_bytes_per_bit: 0 2024-07-26T11:02:03.790 INFO:journalctl@ceph.mon.c.smithi193.stdout: format_version: 5 2024-07-26T11:02:03.790 INFO:journalctl@ceph.mon.c.smithi193.stdout: enable_index_compression: 1 2024-07-26T11:02:03.790 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_align: 0 2024-07-26T11:02:03.790 INFO:journalctl@ceph.mon.c.smithi193.stdout: max_auto_readahead_size: 262144 2024-07-26T11:02:03.790 INFO:journalctl@ceph.mon.c.smithi193.stdout: prepopulate_block_cache: 0 2024-07-26T11:02:03.790 INFO:journalctl@ceph.mon.c.smithi193.stdout: initial_auto_readahead_size: 8192 2024-07-26T11:02:03.790 INFO:journalctl@ceph.mon.c.smithi193.stdout: num_file_reads_for_auto_readahead: 2 2024-07-26T11:02:03.790 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.write_buffer_size: 33554432 2024-07-26T11:02:03.790 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_write_buffer_number: 2 2024-07-26T11:02:03.790 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compression: NoCompression 2024-07-26T11:02:03.790 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.bottommost_compression: Disabled 2024-07-26T11:02:03.790 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.prefix_extractor: nullptr 2024-07-26T11:02:03.790 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-07-26T11:02:03.791 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.num_levels: 7 2024-07-26T11:02:03.791 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-07-26T11:02:03.791 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-07-26T11:02:03.791 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-07-26T11:02:03.791 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-07-26T11:02:03.791 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-07-26T11:02:03.791 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-07-26T11:02:03.791 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-07-26T11:02:03.791 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-07-26T11:02:03.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-07-26T11:02:03.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-07-26T11:02:03.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-07-26T11:02:03.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-07-26T11:02:03.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compression_opts.window_bits: -14 2024-07-26T11:02:03.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compression_opts.level: 32767 2024-07-26T11:02:03.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compression_opts.strategy: 0 2024-07-26T11:02:03.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-07-26T11:02:03.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-07-26T11:02:03.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-07-26T11:02:03.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-07-26T11:02:03.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compression_opts.enabled: false 2024-07-26T11:02:03.792 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-07-26T11:02:03.793 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-07-26T11:02:03.793 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-07-26T11:02:03.793 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-07-26T11:02:03.793 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.target_file_size_base: 67108864 2024-07-26T11:02:03.793 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.target_file_size_multiplier: 1 2024-07-26T11:02:03.793 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-07-26T11:02:03.793 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-07-26T11:02:03.793 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-07-26T11:02:03.793 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-07-26T11:02:03.793 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-07-26T11:02:03.793 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-07-26T11:02:03.793 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-07-26T11:02:03.793 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-07-26T11:02:03.793 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-07-26T11:02:03.794 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-07-26T11:02:03.794 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-07-26T11:02:03.794 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-07-26T11:02:03.794 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-07-26T11:02:03.794 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.arena_block_size: 1048576 2024-07-26T11:02:03.794 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-07-26T11:02:03.794 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-07-26T11:02:03.794 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.disable_auto_compactions: 0 2024-07-26T11:02:03.794 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-07-26T11:02:03.794 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-07-26T11:02:03.794 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-07-26T11:02:03.794 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-07-26T11:02:03.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-07-26T11:02:03.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-07-26T11:02:03.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-07-26T11:02:03.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-07-26T11:02:03.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-07-26T11:02:03.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-07-26T11:02:03.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-07-26T11:02:03.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.inplace_update_support: 0 2024-07-26T11:02:03.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.inplace_update_num_locks: 10000 2024-07-26T11:02:03.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-07-26T11:02:03.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-07-26T11:02:03.795 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.memtable_huge_page_size: 0 2024-07-26T11:02:03.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.bloom_locality: 0 2024-07-26T11:02:03.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.max_successive_merges: 0 2024-07-26T11:02:03.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.optimize_filters_for_hits: 0 2024-07-26T11:02:03.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.paranoid_file_checks: 0 2024-07-26T11:02:03.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.force_consistency_checks: 1 2024-07-26T11:02:03.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.report_bg_io_stats: 0 2024-07-26T11:02:03.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.ttl: 2592000 2024-07-26T11:02:03.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.periodic_compaction_seconds: 0 2024-07-26T11:02:03.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-07-26T11:02:03.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-07-26T11:02:03.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.enable_blob_files: false 2024-07-26T11:02:03.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.min_blob_size: 0 2024-07-26T11:02:03.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.blob_file_size: 268435456 2024-07-26T11:02:03.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.blob_compression_type: NoCompression 2024-07-26T11:02:03.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.enable_blob_garbage_collection: false 2024-07-26T11:02:03.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-07-26T11:02:03.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-07-26T11:02:03.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-07-26T11:02:03.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.blob_file_starting_level: 0 2024-07-26T11:02:03.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-07-26T11:02:03.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-07-26T11:02:03.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-07-26T11:02:03.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a68e3e43-3051-466e-a297-71c61e1c1f66 2024-07-26T11:02:03.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721991723486671, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-07-26T11:02:03.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-07-26T11:02:03.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721991723487102, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1721991723, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a68e3e43-3051-466e-a297-71c61e1c1f66", "db_session_id": "OVAVJUVOQFDUGPR9DPK0", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-07-26T11:02:03.797 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: EVENT_LOG_v1 {"time_micros": 1721991723487216, "job": 1, "event": "recovery_finished"} 2024-07-26T11:02:03.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-07-26T11:02:03.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-07-26T11:02:03.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556c41684a80 2024-07-26T11:02:03.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: DB pointer 0x556c42424000 2024-07-26T11:02:03.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-07-26T11:02:03.798 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-07-26T11:02:03.798 INFO:journalctl@ceph.mon.c.smithi193.stdout: ** DB Stats ** 2024-07-26T11:02:03.798 INFO:journalctl@ceph.mon.c.smithi193.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T11:02:03.798 INFO:journalctl@ceph.mon.c.smithi193.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-07-26T11:02:03.798 INFO:journalctl@ceph.mon.c.smithi193.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T11:02:03.798 INFO:journalctl@ceph.mon.c.smithi193.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T11:02:03.798 INFO:journalctl@ceph.mon.c.smithi193.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-07-26T11:02:03.798 INFO:journalctl@ceph.mon.c.smithi193.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-07-26T11:02:03.799 INFO:journalctl@ceph.mon.c.smithi193.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-07-26T11:02:03.799 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-07-26T11:02:03.799 INFO:journalctl@ceph.mon.c.smithi193.stdout: ** Compaction Stats [default] ** 2024-07-26T11:02:03.799 INFO:journalctl@ceph.mon.c.smithi193.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T11:02:03.799 INFO:journalctl@ceph.mon.c.smithi193.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-07-26T11:02:03.799 INFO:journalctl@ceph.mon.c.smithi193.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T11:02:03.799 INFO:journalctl@ceph.mon.c.smithi193.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T11:02:03.799 INFO:journalctl@ceph.mon.c.smithi193.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T11:02:03.799 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-07-26T11:02:03.799 INFO:journalctl@ceph.mon.c.smithi193.stdout: ** Compaction Stats [default] ** 2024-07-26T11:02:03.799 INFO:journalctl@ceph.mon.c.smithi193.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-07-26T11:02:03.799 INFO:journalctl@ceph.mon.c.smithi193.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-07-26T11:02:03.799 INFO:journalctl@ceph.mon.c.smithi193.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-07-26T11:02:03.799 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-07-26T11:02:03.800 INFO:journalctl@ceph.mon.c.smithi193.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-07-26T11:02:03.800 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-07-26T11:02:03.800 INFO:journalctl@ceph.mon.c.smithi193.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-07-26T11:02:03.800 INFO:journalctl@ceph.mon.c.smithi193.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-07-26T11:02:03.800 INFO:journalctl@ceph.mon.c.smithi193.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-07-26T11:02:03.800 INFO:journalctl@ceph.mon.c.smithi193.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-07-26T11:02:03.800 INFO:journalctl@ceph.mon.c.smithi193.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-07-26T11:02:03.800 INFO:journalctl@ceph.mon.c.smithi193.stdout: AddFile(Keys): cumulative 0, interval 0 2024-07-26T11:02:03.800 INFO:journalctl@ceph.mon.c.smithi193.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T11:02:03.800 INFO:journalctl@ceph.mon.c.smithi193.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-07-26T11:02:03.800 INFO:journalctl@ceph.mon.c.smithi193.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-07-26T11:02:03.800 INFO:journalctl@ceph.mon.c.smithi193.stdout: Block cache BinnedLRUCache@0x556c3d627610#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-07-26T11:02:03.800 INFO:journalctl@ceph.mon.c.smithi193.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-07-26T11:02:03.800 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-07-26T11:02:03.800 INFO:journalctl@ceph.mon.c.smithi193.stdout: ** File Read Latency Histogram By Level [default] ** 2024-07-26T11:02:03.801 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-07-26T11:02:03.801 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: using public_addr v2:172.21.15.193:0/0 -> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] 2024-07-26T11:02:03.801 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: starting mon.c rank -1 at public addrs [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] at bind addrs [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:02:03.801 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: mon.c@-1(???) e0 preinit fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:02:03.801 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: mon.c@-1(synchronizing).mds e1 new map 2024-07-26T11:02:03.801 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: mon.c@-1(synchronizing).mds e1 print_map 2024-07-26T11:02:03.801 INFO:journalctl@ceph.mon.c.smithi193.stdout: e1 2024-07-26T11:02:03.801 INFO:journalctl@ceph.mon.c.smithi193.stdout: btime 2024-07-26T11:00:05:954291+0000 2024-07-26T11:02:03.801 INFO:journalctl@ceph.mon.c.smithi193.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T11:02:03.801 INFO:journalctl@ceph.mon.c.smithi193.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T11:02:03.801 INFO:journalctl@ceph.mon.c.smithi193.stdout: legacy client fscid: -1 2024-07-26T11:02:03.801 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-07-26T11:02:03.801 INFO:journalctl@ceph.mon.c.smithi193.stdout: No filesystems configured 2024-07-26T11:02:03.801 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-07-26T11:02:03.801 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-07-26T11:02:03.802 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-07-26T11:02:03.802 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-07-26T11:02:03.802 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-07-26T11:02:03.802 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-07-26T11:02:03.802 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-26T11:02:03.802 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T11:02:03.802 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T11:02:03.802 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T11:02:03.802 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:03.802 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.802 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: Added host smithi189 2024-07-26T11:02:03.802 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:03.802 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.803 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.803 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:03.803 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.803 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:03.803 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:02:03.803 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.803 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.803 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.803 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.803 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:03.803 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:03.803 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:03.803 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: Updating smithi189:/etc/ceph/ceph.conf 2024-07-26T11:02:03.803 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: Updating smithi189:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:03.803 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:02:03.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: Updating smithi189:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T11:02:03.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:03.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:02:03.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:03.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: Deploying cephadm binary to smithi193 2024-07-26T11:02:03.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:03.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: Added host smithi193 2024-07-26T11:02:03.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:03.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:03.804 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.805 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.805 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:03.805 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:02:03.805 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.805 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.805 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.805 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.805 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:03.805 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:03.805 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:03.805 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: Updating smithi193:/etc/ceph/ceph.conf 2024-07-26T11:02:03.805 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: Updating smithi193:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:03.805 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:03.805 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.806 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.806 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.806 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:02:03.806 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: Updating smithi193:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T11:02:03.806 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:03.806 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2352598432' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-26T11:02:03.806 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2352598432' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-26T11:02:03.806 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T11:02:03.806 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:03.806 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi134:172.21.15.134=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:02:03.806 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: Saving service mon spec with placement smithi134:172.21.15.134=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c;count:3 2024-07-26T11:02:03.806 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.806 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:03.806 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:03.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:03.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:03.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T11:02:03.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:03.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: Deploying daemon mon.c on smithi193 2024-07-26T11:02:03.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:03.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:03.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: from='client.? 172.21.15.193:0/4219511000' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T11:02:03.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:03.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 ceph-mon[37473]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-07-26T11:02:03.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 bash[37437]: d84de133b4202e9c7063144f055a22c7b0c83b605869f3ccfce6390913dfd558 2024-07-26T11:02:03.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:03 smithi193 systemd[1]: Started Ceph mon.c for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:02:04.324 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T11:02:04.324 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph mon dump -f json 2024-07-26T11:02:04.573 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T11:02:08.899 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:02:08.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: mon.a calling monitor election 2024-07-26T11:02:08.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:08.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:08.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:08.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:08.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: mon.c calling monitor election 2024-07-26T11:02:08.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:08.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:08.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:08.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:08.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T11:02:08.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: monmap epoch 2 2024-07-26T11:02:08.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:02:08.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: last_changed 2024-07-26T11:02:03.525290+0000 2024-07-26T11:02:08.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: created 2024-07-26T11:00:01.653390+0000 2024-07-26T11:02:08.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: min_mon_release 19 (squid) 2024-07-26T11:02:08.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: election_strategy: 1 2024-07-26T11:02:08.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: 0: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.a 2024-07-26T11:02:08.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-07-26T11:02:08.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: fsmap 2024-07-26T11:02:08.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T11:02:08.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: mgrmap e13: x(active, since 55s) 2024-07-26T11:02:08.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: overall HEALTH_OK 2024-07-26T11:02:08.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:08.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:08.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:08.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T11:02:08.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:08 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:08.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:02:08.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: mon.a calling monitor election 2024-07-26T11:02:08.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:08.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:08.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:08.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:08.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: mon.c calling monitor election 2024-07-26T11:02:08.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:08.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:08.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:08.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:08.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T11:02:08.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: monmap epoch 2 2024-07-26T11:02:08.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:02:08.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: last_changed 2024-07-26T11:02:03.525290+0000 2024-07-26T11:02:08.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: created 2024-07-26T11:00:01.653390+0000 2024-07-26T11:02:08.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: min_mon_release 19 (squid) 2024-07-26T11:02:08.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: election_strategy: 1 2024-07-26T11:02:08.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: 0: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.a 2024-07-26T11:02:08.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-07-26T11:02:08.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: fsmap 2024-07-26T11:02:08.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T11:02:08.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: mgrmap e13: x(active, since 55s) 2024-07-26T11:02:08.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: overall HEALTH_OK 2024-07-26T11:02:08.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:08.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:08.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:08.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T11:02:08.926 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:08 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:09.224 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T11:02:09.225 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":2,"fsid":"105fbe5c-4b3e-11ef-bcb7-c7b262605968","modified":"2024-07-26T11:02:03.525290Z","created":"2024-07-26T11:00:01.653390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-07-26T11:02:09.225 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 2 2024-07-26T11:02:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:09 smithi193 ceph-mon[37473]: Deploying daemon mon.b on smithi189 2024-07-26T11:02:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:09 smithi193 ceph-mon[37473]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:09 smithi193 ceph-mon[37473]: from='client.? 172.21.15.193:0/2202256986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T11:02:09.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:09 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:09 smithi134 ceph-mon[31966]: Deploying daemon mon.b on smithi189 2024-07-26T11:02:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:09 smithi134 ceph-mon[31966]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:09 smithi134 ceph-mon[31966]: from='client.? 172.21.15.193:0/2202256986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T11:02:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:09 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:10.828 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-07-26T11:02:10.829 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph mon dump -f json 2024-07-26T11:02:11.086 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T11:02:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:10 smithi134 ceph-mon[31966]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:11.201 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:10 smithi193 ceph-mon[37473]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-07-26T11:02:12.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-07-26T11:02:12.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-07-26T11:02:12.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-07-26T11:02:12.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-07-26T11:02:12.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-07-26T11:02:12.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-07-26T11:02:12.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T11:02:12.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T11:02:12.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-07-26T11:02:12.420 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.420 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.420 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: Added host smithi189 2024-07-26T11:02:12.420 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:12.420 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.420 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.420 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.420 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.420 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.420 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:02:12.420 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.420 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.420 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.420 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.421 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:12.421 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:12.421 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:12.421 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: Updating smithi189:/etc/ceph/ceph.conf 2024-07-26T11:02:12.421 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: Updating smithi189:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:12.421 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:02:12.421 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: Updating smithi189:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T11:02:12.421 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.421 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.421 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.421 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.421 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:02:12.421 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: Deploying cephadm binary to smithi193 2024-07-26T11:02:12.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: Added host smithi193 2024-07-26T11:02:12.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:12.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:02:12.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.423 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:12.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: Updating smithi193:/etc/ceph/ceph.conf 2024-07-26T11:02:12.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: Updating smithi193:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:12.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:02:12.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: Updating smithi193:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T11:02:12.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2352598432' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-07-26T11:02:12.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2352598432' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-07-26T11:02:12.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T11:02:12.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi134:172.21.15.134=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:02:12.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: Saving service mon spec with placement smithi134:172.21.15.134=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c;count:3 2024-07-26T11:02:12.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:12.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:12.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:12.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T11:02:12.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:12.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: Deploying daemon mon.c on smithi193 2024-07-26T11:02:12.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='client.? 172.21.15.193:0/4219511000' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T11:02:12.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:02:12.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: mon.a calling monitor election 2024-07-26T11:02:12.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:12.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:12.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:12.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: mon.c calling monitor election 2024-07-26T11:02:12.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:12.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:12.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:12.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T11:02:12.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: monmap epoch 2 2024-07-26T11:02:12.698 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:02:12.698 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: last_changed 2024-07-26T11:02:03.525290+0000 2024-07-26T11:02:12.698 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: created 2024-07-26T11:00:01.653390+0000 2024-07-26T11:02:12.698 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: min_mon_release 19 (squid) 2024-07-26T11:02:12.698 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: election_strategy: 1 2024-07-26T11:02:12.698 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: 0: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.a 2024-07-26T11:02:12.698 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-07-26T11:02:12.698 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: fsmap 2024-07-26T11:02:12.698 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T11:02:12.698 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: mgrmap e13: x(active, since 55s) 2024-07-26T11:02:12.698 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: overall HEALTH_OK 2024-07-26T11:02:12.698 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.698 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.699 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:12.699 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T11:02:12.699 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:12.699 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: Deploying daemon mon.b on smithi189 2024-07-26T11:02:12.699 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.699 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='client.? 172.21.15.193:0/2202256986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-07-26T11:02:12.699 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:12.699 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:12.699 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 ceph-mon[36790]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-07-26T11:02:12.699 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 bash[36772]: 8b89706869d6401e7831e27dc779e3e8e1b641bb5554d0c3c64596c7677ec273 2024-07-26T11:02:12.699 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:12 smithi189 systemd[1]: Started Ceph mon.b for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:02:17.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: mon.c calling monitor election 2024-07-26T11:02:17.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:02:17.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: mon.a calling monitor election 2024-07-26T11:02:17.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:17.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:17.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:17.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:17.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:17.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:17.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:17.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:17.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:17.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:17.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T11:02:17.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: monmap epoch 3 2024-07-26T11:02:17.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:02:17.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: last_changed 2024-07-26T11:02:12.422899+0000 2024-07-26T11:02:17.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: created 2024-07-26T11:00:01.653390+0000 2024-07-26T11:02:17.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: min_mon_release 19 (squid) 2024-07-26T11:02:17.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: election_strategy: 1 2024-07-26T11:02:17.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: 0: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.a 2024-07-26T11:02:17.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-07-26T11:02:17.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: 2: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-07-26T11:02:17.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: fsmap 2024-07-26T11:02:17.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T11:02:17.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: mgrmap e13: x(active, since 64s) 2024-07-26T11:02:17.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: overall HEALTH_OK 2024-07-26T11:02:17.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:17.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:17.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:17.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:17 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: mon.c calling monitor election 2024-07-26T11:02:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:02:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: mon.a calling monitor election 2024-07-26T11:02:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T11:02:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: monmap epoch 3 2024-07-26T11:02:17.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:02:17.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: last_changed 2024-07-26T11:02:12.422899+0000 2024-07-26T11:02:17.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: created 2024-07-26T11:00:01.653390+0000 2024-07-26T11:02:17.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: min_mon_release 19 (squid) 2024-07-26T11:02:17.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: election_strategy: 1 2024-07-26T11:02:17.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: 0: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.a 2024-07-26T11:02:17.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-07-26T11:02:17.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: 2: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-07-26T11:02:17.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: fsmap 2024-07-26T11:02:17.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T11:02:17.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: mgrmap e13: x(active, since 64s) 2024-07-26T11:02:17.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: overall HEALTH_OK 2024-07-26T11:02:17.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:17.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:17.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:17.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:17 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:18.771 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-07-26T11:02:18.771 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":3,"fsid":"105fbe5c-4b3e-11ef-bcb7-c7b262605968","modified":"2024-07-26T11:02:12.422899Z","created":"2024-07-26T11:00:01.653390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-07-26T11:02:18.771 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 3 2024-07-26T11:02:18.771 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:18 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:18.772 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:18 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:18.772 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:18 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:18 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:18 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:18.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:18 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:19.561 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-07-26T11:02:19.562 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config generate-minimal-conf 2024-07-26T11:02:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: mon.c calling monitor election 2024-07-26T11:02:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:02:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: mon.a calling monitor election 2024-07-26T11:02:19.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:19.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:19.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:19.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:19.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:19.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:19.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:19.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:19.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:19.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:19.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-07-26T11:02:19.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: monmap epoch 3 2024-07-26T11:02:19.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:02:19.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: last_changed 2024-07-26T11:02:12.422899+0000 2024-07-26T11:02:19.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: created 2024-07-26T11:00:01.653390+0000 2024-07-26T11:02:19.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: min_mon_release 19 (squid) 2024-07-26T11:02:19.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: election_strategy: 1 2024-07-26T11:02:19.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: 0: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.a 2024-07-26T11:02:19.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-07-26T11:02:19.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: 2: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-07-26T11:02:19.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: fsmap 2024-07-26T11:02:19.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T11:02:19.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: mgrmap e13: x(active, since 64s) 2024-07-26T11:02:19.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: overall HEALTH_OK 2024-07-26T11:02:19.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:19.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:19.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:19.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:19.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:19.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:19.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:19 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:19.974 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:20.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: mon.b calling monitor election 2024-07-26T11:02:20.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: Updating smithi134:/etc/ceph/ceph.conf 2024-07-26T11:02:20.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: Updating smithi189:/etc/ceph/ceph.conf 2024-07-26T11:02:20.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: Updating smithi193:/etc/ceph/ceph.conf 2024-07-26T11:02:20.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: Updating smithi189:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:20.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: Updating smithi193:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:20.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: Updating smithi134:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:20.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: mon.b calling monitor election 2024-07-26T11:02:20.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: mon.c calling monitor election 2024-07-26T11:02:20.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: mon.a calling monitor election 2024-07-26T11:02:20.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T11:02:20.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: monmap epoch 3 2024-07-26T11:02:20.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:02:20.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: last_changed 2024-07-26T11:02:12.422899+0000 2024-07-26T11:02:20.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: created 2024-07-26T11:00:01.653390+0000 2024-07-26T11:02:20.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: min_mon_release 19 (squid) 2024-07-26T11:02:20.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: election_strategy: 1 2024-07-26T11:02:20.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: 0: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.a 2024-07-26T11:02:20.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-07-26T11:02:20.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: 2: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-07-26T11:02:20.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: fsmap 2024-07-26T11:02:20.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T11:02:20.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: mgrmap e13: x(active, since 66s) 2024-07-26T11:02:20.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: overall HEALTH_OK 2024-07-26T11:02:20.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T11:02:20.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T11:02:20.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:20 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:20.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: mon.b calling monitor election 2024-07-26T11:02:20.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: Updating smithi134:/etc/ceph/ceph.conf 2024-07-26T11:02:20.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: Updating smithi189:/etc/ceph/ceph.conf 2024-07-26T11:02:20.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: Updating smithi193:/etc/ceph/ceph.conf 2024-07-26T11:02:20.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: Updating smithi189:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:20.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: Updating smithi193:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:20.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: Updating smithi134:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:20.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: mon.b calling monitor election 2024-07-26T11:02:20.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: mon.c calling monitor election 2024-07-26T11:02:20.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: mon.a calling monitor election 2024-07-26T11:02:20.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T11:02:20.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: monmap epoch 3 2024-07-26T11:02:20.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:02:20.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: last_changed 2024-07-26T11:02:12.422899+0000 2024-07-26T11:02:20.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: created 2024-07-26T11:00:01.653390+0000 2024-07-26T11:02:20.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: min_mon_release 19 (squid) 2024-07-26T11:02:20.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: election_strategy: 1 2024-07-26T11:02:20.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: 0: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.a 2024-07-26T11:02:20.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-07-26T11:02:20.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: 2: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-07-26T11:02:20.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: fsmap 2024-07-26T11:02:20.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T11:02:20.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: mgrmap e13: x(active, since 66s) 2024-07-26T11:02:20.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: overall HEALTH_OK 2024-07-26T11:02:20.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T11:02:20.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T11:02:20.697 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:20 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: mon.b calling monitor election 2024-07-26T11:02:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: Updating smithi134:/etc/ceph/ceph.conf 2024-07-26T11:02:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: Updating smithi189:/etc/ceph/ceph.conf 2024-07-26T11:02:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: Updating smithi193:/etc/ceph/ceph.conf 2024-07-26T11:02:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: Updating smithi189:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: Updating smithi193:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: Updating smithi134:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: mon.b calling monitor election 2024-07-26T11:02:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: mon.c calling monitor election 2024-07-26T11:02:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: mon.a calling monitor election 2024-07-26T11:02:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-07-26T11:02:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: monmap epoch 3 2024-07-26T11:02:20.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:02:20.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: last_changed 2024-07-26T11:02:12.422899+0000 2024-07-26T11:02:20.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: created 2024-07-26T11:00:01.653390+0000 2024-07-26T11:02:20.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: min_mon_release 19 (squid) 2024-07-26T11:02:20.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: election_strategy: 1 2024-07-26T11:02:20.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: 0: [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon.a 2024-07-26T11:02:20.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: 1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.c 2024-07-26T11:02:20.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: 2: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-07-26T11:02:20.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: fsmap 2024-07-26T11:02:20.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: osdmap e4: 0 total, 0 up, 0 in 2024-07-26T11:02:20.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: mgrmap e13: x(active, since 66s) 2024-07-26T11:02:20.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: overall HEALTH_OK 2024-07-26T11:02:20.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:20.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T11:02:20.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T11:02:20.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:20 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:21 smithi134 ceph-mon[31966]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T11:02:21.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:21 smithi134 ceph-mon[31966]: Reconfiguring daemon mon.a on smithi134 2024-07-26T11:02:21.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:21 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:21.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:21 smithi134 ceph-mon[31966]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:21 smithi189 ceph-mon[36790]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T11:02:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:21 smithi189 ceph-mon[36790]: Reconfiguring daemon mon.a on smithi134 2024-07-26T11:02:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:21 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:21 smithi189 ceph-mon[36790]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:21 smithi193 ceph-mon[37473]: Reconfiguring mon.a (unknown last config time)... 2024-07-26T11:02:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:21 smithi193 ceph-mon[37473]: Reconfiguring daemon mon.a on smithi134 2024-07-26T11:02:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:21 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:21 smithi193 ceph-mon[37473]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:22.632 INFO:teuthology.orchestra.run.smithi134.stdout:# minimal ceph.conf for 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:02:22.632 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2024-07-26T11:02:22.632 INFO:teuthology.orchestra.run.smithi134.stdout: fsid = 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:02:22.632 INFO:teuthology.orchestra.run.smithi134.stdout: mon_host = [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] 2024-07-26T11:02:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:22 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1407149018' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:22 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1407149018' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:22 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1407149018' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:23.732 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-07-26T11:02:23.732 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T11:02:23.733 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T11:02:23.810 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T11:02:23.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:02:23.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:23 smithi134 ceph-mon[31966]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:23.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:23 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:23.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:23 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:23.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:23 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T11:02:23.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:23 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T11:02:23.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:23 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:23.927 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-07-26T11:02:23.927 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T11:02:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:23 smithi189 ceph-mon[36790]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:23 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:23 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:23 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T11:02:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:23 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T11:02:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:23 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:23.954 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-07-26T11:02:23.954 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:02:24.017 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-07-26T11:02:24.017 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.conf 2024-07-26T11:02:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:23 smithi193 ceph-mon[37473]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:23 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:23 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:23 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T11:02:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:23 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T11:02:24.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:23 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:24.048 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-07-26T11:02:24.048 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:02:24.116 INFO:tasks.cephadm:enabling module snap_schedule 2024-07-26T11:02:24.116 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- sudo ceph --cluster ceph mgr module enable snap_schedule 2024-07-26T11:02:24.550 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:24 smithi189 ceph-mon[36790]: Reconfiguring mon.b (monmap changed)... 2024-07-26T11:02:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:24 smithi189 ceph-mon[36790]: Reconfiguring daemon mon.b on smithi189 2024-07-26T11:02:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:24 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:24.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:24 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:24.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:24 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T11:02:24.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:24 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T11:02:24.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:24 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:25.003 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:24 smithi134 ceph-mon[31966]: Reconfiguring mon.b (monmap changed)... 2024-07-26T11:02:25.003 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:24 smithi134 ceph-mon[31966]: Reconfiguring daemon mon.b on smithi189 2024-07-26T11:02:25.003 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:24 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:25.003 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:24 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:25.003 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:24 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T11:02:25.003 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:24 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T11:02:25.003 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:24 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:25.019 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:24 smithi193 ceph-mon[37473]: Reconfiguring mon.b (monmap changed)... 2024-07-26T11:02:25.019 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:24 smithi193 ceph-mon[37473]: Reconfiguring daemon mon.b on smithi189 2024-07-26T11:02:25.019 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:24 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:25.019 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:24 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:25.020 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:24 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-07-26T11:02:25.020 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:24 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-07-26T11:02:25.020 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:24 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:25.978 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:25 smithi134 ceph-mon[31966]: Reconfiguring mon.c (monmap changed)... 2024-07-26T11:02:25.978 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:25 smithi134 ceph-mon[31966]: Reconfiguring daemon mon.c on smithi193 2024-07-26T11:02:25.978 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:25 smithi134 ceph-mon[31966]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:26.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:25 smithi193 ceph-mon[37473]: Reconfiguring mon.c (monmap changed)... 2024-07-26T11:02:26.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:25 smithi193 ceph-mon[37473]: Reconfiguring daemon mon.c on smithi193 2024-07-26T11:02:26.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:25 smithi193 ceph-mon[37473]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:25 smithi189 ceph-mon[36790]: Reconfiguring mon.c (monmap changed)... 2024-07-26T11:02:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:25 smithi189 ceph-mon[36790]: Reconfiguring daemon mon.c on smithi193 2024-07-26T11:02:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:25 smithi189 ceph-mon[36790]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:26 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:26 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:27.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:26 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:27.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:26 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:27.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:26 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:27.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:26 smithi193 ceph-mon[37473]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:26 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:26 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:26 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:26 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:26 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:26 smithi134 ceph-mon[31966]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:26 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:26 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:26 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:26 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:26 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:26 smithi189 ceph-mon[36790]: from='mgr.14150 172.21.15.134:0/1982797652' entity='mgr.x' 2024-07-26T11:02:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:27 smithi193 ceph-mon[37473]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:27 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1527390677' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T11:02:28.041 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:27 smithi134 ceph-mon[31966]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:28.041 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:27 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1527390677' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T11:02:28.041 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:27 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: ignoring --setuser ceph since I am not root 2024-07-26T11:02:28.041 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:27 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: ignoring --setgroup ceph since I am not root 2024-07-26T11:02:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:27 smithi189 ceph-mon[36790]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:27 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1527390677' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T11:02:28.457 INFO:tasks.cephadm:Adding mgr.x on smithi134 2024-07-26T11:02:28.457 INFO:tasks.cephadm:Adding mgr.y on smithi189 2024-07-26T11:02:28.457 INFO:tasks.cephadm:Adding mgr.z on smithi193 2024-07-26T11:02:28.457 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch apply mgr '3;smithi134=x;smithi189=y;smithi193=z' 2024-07-26T11:02:28.714 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T11:02:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:28 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1527390677' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T11:02:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:28 smithi193 ceph-mon[37473]: mgrmap e14: x(active, since 75s) 2024-07-26T11:02:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:28 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1527390677' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T11:02:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:28 smithi134 ceph-mon[31966]: mgrmap e14: x(active, since 75s) 2024-07-26T11:02:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:28 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1527390677' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T11:02:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:28 smithi189 ceph-mon[36790]: mgrmap e14: x(active, since 75s) 2024-07-26T11:02:33.992 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:33 smithi134 ceph-mon[31966]: Active manager daemon x restarted 2024-07-26T11:02:33.992 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:33 smithi134 ceph-mon[31966]: Activating manager daemon x 2024-07-26T11:02:33.992 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:33 smithi134 ceph-mon[31966]: osdmap e5: 0 total, 0 up, 0 in 2024-07-26T11:02:33.992 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:33 smithi134 ceph-mon[31966]: mgrmap e15: x(active, starting, since 0.00660569s) 2024-07-26T11:02:33.992 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:02:33.992 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:33.993 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:33.993 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T11:02:33.993 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T11:02:33.993 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T11:02:33.993 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T11:02:33.993 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:33 smithi134 ceph-mon[31966]: Manager daemon x is now available 2024-07-26T11:02:33.993 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:34.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:33 smithi193 ceph-mon[37473]: Active manager daemon x restarted 2024-07-26T11:02:34.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:33 smithi193 ceph-mon[37473]: Activating manager daemon x 2024-07-26T11:02:34.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:33 smithi193 ceph-mon[37473]: osdmap e5: 0 total, 0 up, 0 in 2024-07-26T11:02:34.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:33 smithi193 ceph-mon[37473]: mgrmap e15: x(active, starting, since 0.00660569s) 2024-07-26T11:02:34.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:02:34.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:34.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:34.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T11:02:34.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T11:02:34.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T11:02:34.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T11:02:34.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:33 smithi193 ceph-mon[37473]: Manager daemon x is now available 2024-07-26T11:02:34.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:33 smithi189 ceph-mon[36790]: Active manager daemon x restarted 2024-07-26T11:02:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:33 smithi189 ceph-mon[36790]: Activating manager daemon x 2024-07-26T11:02:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:33 smithi189 ceph-mon[36790]: osdmap e5: 0 total, 0 up, 0 in 2024-07-26T11:02:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:33 smithi189 ceph-mon[36790]: mgrmap e15: x(active, starting, since 0.00660569s) 2024-07-26T11:02:34.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:02:34.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:02:34.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:02:34.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-07-26T11:02:34.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-07-26T11:02:34.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-07-26T11:02:34.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-07-26T11:02:34.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:33 smithi189 ceph-mon[36790]: Manager daemon x is now available 2024-07-26T11:02:34.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:34.736 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mgr update... 2024-07-26T11:02:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T11:02:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T11:02:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:34 smithi193 ceph-mon[37473]: mgrmap e16: x(active, since 1.01259s) 2024-07-26T11:02:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:35.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T11:02:35.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T11:02:35.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:34 smithi134 ceph-mon[31966]: mgrmap e16: x(active, since 1.01259s) 2024-07-26T11:02:35.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:35.033 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-07-26T11:02:35.033 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-07-26T11:02:35.033 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:34 smithi189 ceph-mon[36790]: mgrmap e16: x(active, since 1.01259s) 2024-07-26T11:02:35.033 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:35.406 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:02:35.023+0000 7f9007a00640 -1 log_channel(cephadm) log [ERR] : [26/Jul/2024:11:02:35] ENGINE Error in HTTPServer.serve 2024-07-26T11:02:35.406 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: Traceback (most recent call last): 2024-07-26T11:02:35.406 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T11:02:35.406 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: self._connections.run(self.expiration_interval) 2024-07-26T11:02:35.406 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T11:02:35.406 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: self._run(expiration_interval) 2024-07-26T11:02:35.406 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T11:02:35.406 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T11:02:35.406 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T11:02:35.406 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T11:02:35.407 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T11:02:35.407 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: s = self.context.wrap_socket( 2024-07-26T11:02:35.407 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T11:02:35.407 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: return self.sslsocket_class._create( 2024-07-26T11:02:35.407 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T11:02:35.407 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: self.do_handshake() 2024-07-26T11:02:35.407 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T11:02:35.407 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: self._sslobj.do_handshake() 2024-07-26T11:02:35.407 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T11:02:35.407 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:02:35.863 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:35 smithi193 ceph-mon[37473]: Saving service mgr spec with placement smithi134=x;smithi189=y;smithi193=z;count:3 2024-07-26T11:02:35.863 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:35 smithi193 ceph-mon[37473]: [26/Jul/2024:11:02:34] ENGINE Bus STARTING 2024-07-26T11:02:35.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:35 smithi193 ceph-mon[37473]: [26/Jul/2024:11:02:34] ENGINE Serving on http://172.21.15.134:8765 2024-07-26T11:02:35.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:35.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:35.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:35.864 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:35.893 DEBUG:teuthology.orchestra.run.smithi189:mgr.y> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mgr.y.service 2024-07-26T11:02:35.895 DEBUG:teuthology.orchestra.run.smithi193:mgr.z> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mgr.z.service 2024-07-26T11:02:35.897 INFO:tasks.cephadm:setting config [client] client_mount_timeout = 600 2024-07-26T11:02:35.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set client client_mount_timeout 600 2024-07-26T11:02:35.902 INFO:tasks.cephadm:setting config [client] debug_client = 20 2024-07-26T11:02:35.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set client debug_client 20 2024-07-26T11:02:35.946 INFO:tasks.cephadm:setting config [client] debug_ms = 1 2024-07-26T11:02:35.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set client debug_ms 1 2024-07-26T11:02:35.992 INFO:tasks.cephadm:setting config [client] rados_mon_op_timeout = 900 2024-07-26T11:02:35.993 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set client rados_mon_op_timeout 900 2024-07-26T11:02:36.039 INFO:tasks.cephadm:setting config [client] rados_osd_op_timeout = 900 2024-07-26T11:02:36.039 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set client rados_osd_op_timeout 900 2024-07-26T11:02:36.048 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-mon[31966]: Saving service mgr spec with placement smithi134=x;smithi189=y;smithi193=z;count:3 2024-07-26T11:02:36.048 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-mon[31966]: [26/Jul/2024:11:02:34] ENGINE Bus STARTING 2024-07-26T11:02:36.048 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-mon[31966]: [26/Jul/2024:11:02:34] ENGINE Serving on http://172.21.15.134:8765 2024-07-26T11:02:36.048 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:36.048 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:36.048 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:36.048 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:36.053 INFO:tasks.cephadm:setting config [mds] debug_mds = 20 2024-07-26T11:02:36.053 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mds debug_mds 20 2024-07-26T11:02:36.101 INFO:tasks.cephadm:setting config [mds] debug_mds_balancer = 20 2024-07-26T11:02:36.101 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mds debug_mds_balancer 20 2024-07-26T11:02:36.151 INFO:tasks.cephadm:setting config [mds] debug_ms = 2 2024-07-26T11:02:36.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mds debug_ms 2 2024-07-26T11:02:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:35 smithi189 ceph-mon[36790]: Saving service mgr spec with placement smithi134=x;smithi189=y;smithi193=z;count:3 2024-07-26T11:02:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:35 smithi189 ceph-mon[36790]: [26/Jul/2024:11:02:34] ENGINE Bus STARTING 2024-07-26T11:02:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:35 smithi189 ceph-mon[36790]: [26/Jul/2024:11:02:34] ENGINE Serving on http://172.21.15.134:8765 2024-07-26T11:02:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:36.202 INFO:tasks.cephadm:setting config [mds] mds_bal_fragment_size_max = 10000 2024-07-26T11:02:36.203 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mds mds_bal_fragment_size_max 10000 2024-07-26T11:02:36.254 INFO:tasks.cephadm:setting config [mds] mds_bal_merge_size = 5 2024-07-26T11:02:36.255 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mds mds_bal_merge_size 5 2024-07-26T11:02:36.309 INFO:tasks.cephadm:setting config [mds] mds_bal_split_bits = 3 2024-07-26T11:02:36.309 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mds mds_bal_split_bits 3 2024-07-26T11:02:36.359 INFO:tasks.cephadm:setting config [mds] mds_bal_split_size = 100 2024-07-26T11:02:36.359 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mds mds_bal_split_size 100 2024-07-26T11:02:36.407 INFO:tasks.cephadm:setting config [mds] mds_debug_frag = True 2024-07-26T11:02:36.407 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mds mds_debug_frag True 2024-07-26T11:02:36.458 INFO:tasks.cephadm:setting config [mds] mds_debug_scatterstat = True 2024-07-26T11:02:36.458 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mds mds_debug_scatterstat True 2024-07-26T11:02:36.514 INFO:tasks.cephadm:setting config [mds] mds_op_complaint_time = 180 2024-07-26T11:02:36.514 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mds mds_op_complaint_time 180 2024-07-26T11:02:36.571 INFO:tasks.cephadm:setting config [mds] mds_verify_scatter = True 2024-07-26T11:02:36.571 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mds mds_verify_scatter True 2024-07-26T11:02:36.627 INFO:tasks.cephadm:setting config [mds] osd_op_complaint_time = 180 2024-07-26T11:02:36.627 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mds osd_op_complaint_time 180 2024-07-26T11:02:36.689 INFO:tasks.cephadm:setting config [mds] rados_mon_op_timeout = 900 2024-07-26T11:02:36.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mds rados_mon_op_timeout 900 2024-07-26T11:02:36.749 INFO:tasks.cephadm:setting config [mds] rados_osd_op_timeout = 900 2024-07-26T11:02:36.749 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mds rados_osd_op_timeout 900 2024-07-26T11:02:36.807 INFO:tasks.cephadm:setting config [mgr] debug_cephsqlite = 20 2024-07-26T11:02:36.807 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mgr debug_cephsqlite 20 2024-07-26T11:02:36.808 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:36.863 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/allow_m_granularity = True 2024-07-26T11:02:36.864 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2024-07-26T11:02:36.912 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/dump_on_update = True 2024-07-26T11:02:36.912 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mgr mgr/snap_schedule/dump_on_update True 2024-07-26T11:02:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:36 smithi134 ceph-mon[31966]: [26/Jul/2024:11:02:35] ENGINE Serving on https://172.21.15.134:7150 2024-07-26T11:02:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:36 smithi134 ceph-mon[31966]: [26/Jul/2024:11:02:35] ENGINE Bus STARTED 2024-07-26T11:02:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:36 smithi134 ceph-mon[31966]: [26/Jul/2024:11:02:35] ENGINE Error in HTTPServer.serve 2024-07-26T11:02:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout: Traceback (most recent call last): 2024-07-26T11:02:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T11:02:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout: self._connections.run(self.expiration_interval) 2024-07-26T11:02:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T11:02:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout: self._run(expiration_interval) 2024-07-26T11:02:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T11:02:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T11:02:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T11:02:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T11:02:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T11:02:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: s = self.context.wrap_socket( 2024-07-26T11:02:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T11:02:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: return self.sslsocket_class._create( 2024-07-26T11:02:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T11:02:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: self.do_handshake() 2024-07-26T11:02:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T11:02:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: self._sslobj.do_handshake() 2024-07-26T11:02:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T11:02:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:36.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:36.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:36.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:36.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:36.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:36.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:36 smithi134 ceph-mon[31966]: mgrmap e17: x(active, since 2s) 2024-07-26T11:02:36.937 INFO:tasks.cephadm:setting config [mgr] mgr/snap_schedule/log_level = debug 2024-07-26T11:02:36.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mgr mgr/snap_schedule/log_level debug 2024-07-26T11:02:37.000 INFO:tasks.cephadm:setting config [mon] mon_op_complaint_time = 120 2024-07-26T11:02:37.000 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mon mon_op_complaint_time 120 2024-07-26T11:02:37.014 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:37.036 INFO:tasks.cephadm:setting config [mon] mon_warn_on_pool_no_app = False 2024-07-26T11:02:37.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set mon mon_warn_on_pool_no_app False 2024-07-26T11:02:37.064 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:37.074 INFO:tasks.cephadm:setting config [osd] osd_op_complaint_time = 180 2024-07-26T11:02:37.075 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config set osd osd_op_complaint_time 180 2024-07-26T11:02:37.131 DEBUG:tasks.cephadm:set 26 configs 2024-07-26T11:02:37.132 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi134') 2024-07-26T11:02:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:36 smithi189 ceph-mon[36790]: [26/Jul/2024:11:02:35] ENGINE Serving on https://172.21.15.134:7150 2024-07-26T11:02:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:36 smithi189 ceph-mon[36790]: [26/Jul/2024:11:02:35] ENGINE Bus STARTED 2024-07-26T11:02:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:36 smithi189 ceph-mon[36790]: [26/Jul/2024:11:02:35] ENGINE Error in HTTPServer.serve 2024-07-26T11:02:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout: Traceback (most recent call last): 2024-07-26T11:02:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T11:02:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout: self._connections.run(self.expiration_interval) 2024-07-26T11:02:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T11:02:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout: self._run(expiration_interval) 2024-07-26T11:02:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T11:02:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T11:02:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T11:02:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T11:02:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T11:02:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout: s = self.context.wrap_socket( 2024-07-26T11:02:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T11:02:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout: return self.sslsocket_class._create( 2024-07-26T11:02:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T11:02:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout: self.do_handshake() 2024-07-26T11:02:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T11:02:37.196 INFO:journalctl@ceph.mon.b.smithi189.stdout: self._sslobj.do_handshake() 2024-07-26T11:02:37.196 INFO:journalctl@ceph.mon.b.smithi189.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T11:02:37.196 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:37.196 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:37.196 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:37.196 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:37.196 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:37.196 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:37.196 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:37.196 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:37.196 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:36 smithi189 ceph-mon[36790]: mgrmap e17: x(active, since 2s) 2024-07-26T11:02:37.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:36 smithi193 ceph-mon[37473]: [26/Jul/2024:11:02:35] ENGINE Serving on https://172.21.15.134:7150 2024-07-26T11:02:37.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:36 smithi193 ceph-mon[37473]: [26/Jul/2024:11:02:35] ENGINE Bus STARTED 2024-07-26T11:02:37.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:36 smithi193 ceph-mon[37473]: [26/Jul/2024:11:02:35] ENGINE Error in HTTPServer.serve 2024-07-26T11:02:37.275 INFO:journalctl@ceph.mon.c.smithi193.stdout: Traceback (most recent call last): 2024-07-26T11:02:37.275 INFO:journalctl@ceph.mon.c.smithi193.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-07-26T11:02:37.275 INFO:journalctl@ceph.mon.c.smithi193.stdout: self._connections.run(self.expiration_interval) 2024-07-26T11:02:37.275 INFO:journalctl@ceph.mon.c.smithi193.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-07-26T11:02:37.275 INFO:journalctl@ceph.mon.c.smithi193.stdout: self._run(expiration_interval) 2024-07-26T11:02:37.275 INFO:journalctl@ceph.mon.c.smithi193.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-07-26T11:02:37.275 INFO:journalctl@ceph.mon.c.smithi193.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-07-26T11:02:37.275 INFO:journalctl@ceph.mon.c.smithi193.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-07-26T11:02:37.276 INFO:journalctl@ceph.mon.c.smithi193.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-07-26T11:02:37.276 INFO:journalctl@ceph.mon.c.smithi193.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-07-26T11:02:37.276 INFO:journalctl@ceph.mon.c.smithi193.stdout: s = self.context.wrap_socket( 2024-07-26T11:02:37.276 INFO:journalctl@ceph.mon.c.smithi193.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-07-26T11:02:37.276 INFO:journalctl@ceph.mon.c.smithi193.stdout: return self.sslsocket_class._create( 2024-07-26T11:02:37.276 INFO:journalctl@ceph.mon.c.smithi193.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-07-26T11:02:37.276 INFO:journalctl@ceph.mon.c.smithi193.stdout: self.do_handshake() 2024-07-26T11:02:37.276 INFO:journalctl@ceph.mon.c.smithi193.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-07-26T11:02:37.276 INFO:journalctl@ceph.mon.c.smithi193.stdout: self._sslobj.do_handshake() 2024-07-26T11:02:37.276 INFO:journalctl@ceph.mon.c.smithi193.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-07-26T11:02:37.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:37.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:37.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:37.277 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:37.277 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:37.277 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:37.277 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:37.277 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:37.277 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:36 smithi193 ceph-mon[37473]: mgrmap e17: x(active, since 2s) 2024-07-26T11:02:39.213 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:39.216 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:40.006 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.006 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.006 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi134", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:40.006 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:40.006 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:40.006 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:39 smithi134 ceph-mon[31966]: Updating smithi134:/etc/ceph/ceph.conf 2024-07-26T11:02:40.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:39 smithi134 ceph-mon[31966]: Updating smithi189:/etc/ceph/ceph.conf 2024-07-26T11:02:40.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:39 smithi134 ceph-mon[31966]: Updating smithi193:/etc/ceph/ceph.conf 2024-07-26T11:02:40.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.158 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.159 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.159 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi134", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:40.159 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:40.159 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:40.159 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:39 smithi189 ceph-mon[36790]: Updating smithi134:/etc/ceph/ceph.conf 2024-07-26T11:02:40.159 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:39 smithi189 ceph-mon[36790]: Updating smithi189:/etc/ceph/ceph.conf 2024-07-26T11:02:40.159 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:39 smithi189 ceph-mon[36790]: Updating smithi193:/etc/ceph/ceph.conf 2024-07-26T11:02:40.159 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.159 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.159 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.159 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi134", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:02:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:39 smithi193 ceph-mon[37473]: Updating smithi134:/etc/ceph/ceph.conf 2024-07-26T11:02:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:39 smithi193 ceph-mon[37473]: Updating smithi189:/etc/ceph/ceph.conf 2024-07-26T11:02:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:39 smithi193 ceph-mon[37473]: Updating smithi193:/etc/ceph/ceph.conf 2024-07-26T11:02:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:40.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:41.015 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:40 smithi134 ceph-mon[31966]: Updating smithi193:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:41.015 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:40 smithi134 ceph-mon[31966]: Updating smithi189:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:41.015 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:40 smithi134 ceph-mon[31966]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:02:41.015 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:40 smithi134 ceph-mon[31966]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:02:41.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:40 smithi134 ceph-mon[31966]: Updating smithi134:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:41.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:40 smithi134 ceph-mon[31966]: Updating smithi193:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T11:02:41.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:40 smithi134 ceph-mon[31966]: Updating smithi189:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T11:02:41.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:40 smithi189 ceph-mon[36790]: Updating smithi193:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:41.096 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:40 smithi189 ceph-mon[36790]: Updating smithi189:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:41.096 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:40 smithi189 ceph-mon[36790]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:02:41.096 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:40 smithi189 ceph-mon[36790]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:02:41.096 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:40 smithi189 ceph-mon[36790]: Updating smithi134:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:41.096 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:40 smithi189 ceph-mon[36790]: Updating smithi193:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T11:02:41.096 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:40 smithi189 ceph-mon[36790]: Updating smithi189:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T11:02:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:40 smithi193 ceph-mon[37473]: Updating smithi193:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:40 smithi193 ceph-mon[37473]: Updating smithi189:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:40 smithi193 ceph-mon[37473]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:02:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:40 smithi193 ceph-mon[37473]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:02:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:40 smithi193 ceph-mon[37473]: Updating smithi134:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.conf 2024-07-26T11:02:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:40 smithi193 ceph-mon[37473]: Updating smithi193:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T11:02:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:40 smithi193 ceph-mon[37473]: Updating smithi189:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T11:02:42.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:41 smithi193 ceph-mon[37473]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:02:42.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:41 smithi193 ceph-mon[37473]: Updating smithi134:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T11:02:42.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:42.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:42.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:42.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T11:02:42.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T11:02:42.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T11:02:42.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:41 smithi134 ceph-mon[31966]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:02:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:41 smithi134 ceph-mon[31966]: Updating smithi134:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T11:02:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T11:02:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T11:02:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T11:02:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:42.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:42.172+0000 7f9b17e00640 1 -- 172.21.15.134:0/3679011045 <== mon.1 v2:172.21.15.193:3300/0 8 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v10)=0 v10) ==== 109+0+0 (secure 0 0 0) 0x7f9b1002ee30 con 0x7f9b20077370 2024-07-26T11:02:42.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:42.175+0000 7f9b26600640 1 -- 172.21.15.134:0/3679011045 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b0403edb0 msgr2=0x7f9b040412c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:42.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:42.175+0000 7f9b26600640 1 --2- 172.21.15.134:0/3679011045 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b0403edb0 0x7f9b040412c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9b2006aaa0 tx=0x7f9b0c005f90 comp rx=0 tx=0).stop 2024-07-26T11:02:42.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:42.175+0000 7f9b26600640 1 -- 172.21.15.134:0/3679011045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b20077370 msgr2=0x7f9b2019f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:42.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:42.175+0000 7f9b26600640 1 --2- 172.21.15.134:0/3679011045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b20077370 0x7f9b2019f8c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9b10035040 tx=0x7f9b10033c60 comp rx=0 tx=0).stop 2024-07-26T11:02:42.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:42.175+0000 7f9b26600640 1 -- 172.21.15.134:0/3679011045 shutdown_connections 2024-07-26T11:02:42.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:42.175+0000 7f9b26600640 1 --2- 172.21.15.134:0/3679011045 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b0403edb0 0x7f9b040412c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:42.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:42.175+0000 7f9b26600640 1 --2- 172.21.15.134:0/3679011045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b20114180 0x7f9b201a5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:42.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:42.175+0000 7f9b26600640 1 --2- 172.21.15.134:0/3679011045 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b20077cc0 0x7f9b2019fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:42.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:42.175+0000 7f9b26600640 1 --2- 172.21.15.134:0/3679011045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b20077370 0x7f9b2019f8c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:42.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:42.175+0000 7f9b26600640 1 -- 172.21.15.134:0/3679011045 >> 172.21.15.134:0/3679011045 conn(0x7f9b201011f0 msgr2=0x7f9b20114750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:42.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:42.175+0000 7f9b26600640 1 -- 172.21.15.134:0/3679011045 shutdown_connections 2024-07-26T11:02:42.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:42.175+0000 7f9b26600640 1 -- 172.21.15.134:0/3679011045 wait complete. 2024-07-26T11:02:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:41 smithi189 ceph-mon[36790]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-07-26T11:02:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:41 smithi189 ceph-mon[36790]: Updating smithi134:/var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/config/ceph.client.admin.keyring 2024-07-26T11:02:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T11:02:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T11:02:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T11:02:42.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:42.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:42 smithi193 ceph-mon[37473]: Deploying daemon mgr.z on smithi193 2024-07-26T11:02:42.904 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:42 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' 2024-07-26T11:02:43.153 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:42 smithi134 ceph-mon[31966]: Deploying daemon mgr.z on smithi193 2024-07-26T11:02:43.153 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:42 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' 2024-07-26T11:02:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:42 smithi189 ceph-mon[36790]: Deploying daemon mgr.z on smithi193 2024-07-26T11:02:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:42 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' 2024-07-26T11:02:43.274 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jul 26 11:02:43 smithi193 systemd[1]: Starting Ceph mgr.z for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:02:43.544 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jul 26 11:02:43 smithi193 podman[40300]: 2024-07-26 11:02:43.182063566 +0000 UTC m=+0.017008185 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:02:43.545 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jul 26 11:02:43 smithi193 podman[40300]: 2024-07-26 11:02:43.288106309 +0000 UTC m=+0.123050917 container create fbb78e8522014a8dec9dd8d5d84566a8da15e8928ada94958370595bf2416fbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-07-26T11:02:43.545 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jul 26 11:02:43 smithi193 podman[40300]: 2024-07-26 11:02:43.50730624 +0000 UTC m=+0.342250849 container init fbb78e8522014a8dec9dd8d5d84566a8da15e8928ada94958370595bf2416fbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:02:43.545 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jul 26 11:02:43 smithi193 podman[40300]: 2024-07-26 11:02:43.50975435 +0000 UTC m=+0.344698970 container start fbb78e8522014a8dec9dd8d5d84566a8da15e8928ada94958370595bf2416fbe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-07-26T11:02:44.024 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jul 26 11:02:43 smithi193 ceph-mgr[40319]: -- 172.21.15.193:0/122996400 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5604b8fd10e0 con 0x5604bdd70000 2024-07-26T11:02:44.025 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jul 26 11:02:43 smithi193 bash[40300]: fbb78e8522014a8dec9dd8d5d84566a8da15e8928ada94958370595bf2416fbe 2024-07-26T11:02:44.025 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Jul 26 11:02:43 smithi193 systemd[1]: Started Ceph mgr.z for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:02:44.109 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:44.125 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:44.134 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:44.134 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:44.135 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:44.146 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:44.147 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:44.149 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:44.158 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:44.837 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:44.837 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:44.837 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:44.837 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T11:02:44.837 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T11:02:44.837 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T11:02:44.837 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:44.837 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:44 smithi134 ceph-mon[31966]: Deploying daemon mgr.y on smithi189 2024-07-26T11:02:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T11:02:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T11:02:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T11:02:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:44 smithi189 ceph-mon[36790]: Deploying daemon mgr.y on smithi189 2024-07-26T11:02:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:45.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:45.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T11:02:45.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-07-26T11:02:45.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T11:02:45.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:45.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:44 smithi193 ceph-mon[37473]: Deploying daemon mgr.y on smithi189 2024-07-26T11:02:45.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.041+0000 7f6922a00640 1 -- 172.21.15.134:0/3808851163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f691c108ef0 msgr2=0x7f691c109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:45.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.041+0000 7f6922a00640 1 --2- 172.21.15.134:0/3808851163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f691c108ef0 0x7f691c109300 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6904009900 tx=0x7f690402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:02:45.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.042+0000 7f6922a00640 1 -- 172.21.15.134:0/3808851163 shutdown_connections 2024-07-26T11:02:45.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.042+0000 7f6922a00640 1 --2- 172.21.15.134:0/3808851163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f691c108ef0 0x7f691c109300 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:45.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.042+0000 7f6922a00640 1 --2- 172.21.15.134:0/3808851163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f691c109840 0x7f691c110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:45.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.042+0000 7f6922a00640 1 --2- 172.21.15.134:0/3808851163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f691c1035b0 0x7f691c1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:45.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.042+0000 7f6922a00640 1 -- 172.21.15.134:0/3808851163 >> 172.21.15.134:0/3808851163 conn(0x7f691c0ff040 msgr2=0x7f691c101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:45.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.042+0000 7f6922a00640 1 -- 172.21.15.134:0/3808851163 shutdown_connections 2024-07-26T11:02:45.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.042+0000 7f6922a00640 1 -- 172.21.15.134:0/3808851163 wait complete. 2024-07-26T11:02:45.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.043+0000 7f6922a00640 1 Processor -- start 2024-07-26T11:02:45.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.043+0000 7f6922a00640 1 -- start start 2024-07-26T11:02:45.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.043+0000 7f6922a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f691c1035b0 0x7f691c1993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:45.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.043+0000 7f6922a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f691c108ef0 0x7f691c199940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:45.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.043+0000 7f6922a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f691c109840 0x7f691c19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:45.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.043+0000 7f6922a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f691c112840 con 0x7f691c109840 2024-07-26T11:02:45.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.043+0000 7f6922a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f691c1126c0 con 0x7f691c1035b0 2024-07-26T11:02:45.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.043+0000 7f6922a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f691c1129c0 con 0x7f691c108ef0 2024-07-26T11:02:45.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.044+0000 7f691be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f691c1035b0 0x7f691c1993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:45.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.044+0000 7f691be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f691c1035b0 0x7f691c1993a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56172/0 (socket says 172.21.15.134:56172) 2024-07-26T11:02:45.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.044+0000 7f691be00640 1 -- 172.21.15.134:0/2505467977 learned_addr learned my addr 172.21.15.134:0/2505467977 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:02:45.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.044+0000 7f6920a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f691c109840 0x7f691c19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:45.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.044+0000 7f691b400640 1 --2- 172.21.15.134:0/2505467977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f691c108ef0 0x7f691c199940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:45.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.044+0000 7f691be00640 1 -- 172.21.15.134:0/2505467977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f691c108ef0 msgr2=0x7f691c199940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:45.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.044+0000 7f691be00640 1 --2- 172.21.15.134:0/2505467977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f691c108ef0 0x7f691c199940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:45.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.044+0000 7f691be00640 1 -- 172.21.15.134:0/2505467977 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f691c109840 msgr2=0x7f691c19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:45.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.044+0000 7f691be00640 1 --2- 172.21.15.134:0/2505467977 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f691c109840 0x7f691c19ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:45.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.044+0000 7f691be00640 1 -- 172.21.15.134:0/2505467977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f691c19f310 con 0x7f691c1035b0 2024-07-26T11:02:45.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.044+0000 7f691b400640 1 --2- 172.21.15.134:0/2505467977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f691c108ef0 0x7f691c199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:45.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.045+0000 7f691be00640 1 --2- 172.21.15.134:0/2505467977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f691c1035b0 0x7f691c1993a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f690c00ebc0 tx=0x7f690c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:45.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.045+0000 7f6918c00640 1 -- 172.21.15.134:0/2505467977 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c010630 con 0x7f691c1035b0 2024-07-26T11:02:45.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.045+0000 7f6918c00640 1 -- 172.21.15.134:0/2505467977 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(22 keys) ==== 983+0+0 (secure 0 0 0) 0x7f690c010c10 con 0x7f691c1035b0 2024-07-26T11:02:45.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.045+0000 7f6922a00640 1 -- 172.21.15.134:0/2505467977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f691c19f600 con 0x7f691c1035b0 2024-07-26T11:02:45.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.045+0000 7f6918c00640 1 -- 172.21.15.134:0/2505467977 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c018870 con 0x7f691c1035b0 2024-07-26T11:02:45.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.045+0000 7f6922a00640 1 -- 172.21.15.134:0/2505467977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f691c1a5d60 con 0x7f691c1035b0 2024-07-26T11:02:45.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.046+0000 7f6922a00640 1 -- 172.21.15.134:0/2505467977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f691c104830 con 0x7f691c1035b0 2024-07-26T11:02:45.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.046+0000 7f6918c00640 1 -- 172.21.15.134:0/2505467977 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 50765+0+0 (secure 0 0 0) 0x7f690c0107d0 con 0x7f691c1035b0 2024-07-26T11:02:45.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.047+0000 7f6918c00640 1 --2- 172.21.15.134:0/2505467977 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f690003edb0 0x7f69000412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:45.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.047+0000 7f6918c00640 1 -- 172.21.15.134:0/2505467977 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f690c053a60 con 0x7f691c1035b0 2024-07-26T11:02:45.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.047+0000 7f691b400640 1 --2- 172.21.15.134:0/2505467977 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f690003edb0 0x7f69000412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:45.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.048+0000 7f691b400640 1 --2- 172.21.15.134:0/2505467977 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f690003edb0 0x7f69000412c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f690402a9e0 tx=0x7f6904000bc0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:45.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.052+0000 7f6918c00640 1 -- 172.21.15.134:0/2505467977 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f690c00fca0 con 0x7f691c1035b0 2024-07-26T11:02:45.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.254+0000 7f6922a00640 1 -- 172.21.15.134:0/2505467977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7f691c1039c0 con 0x7f691c1035b0 2024-07-26T11:02:45.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.263+0000 7f6918c00640 1 -- 172.21.15.134:0/2505467977 <== mon.2 v2:172.21.15.189:3300/0 7 ==== config(23 keys) ==== 1014+0+0 (secure 0 0 0) 0x7f690c00b2c0 con 0x7f691c1035b0 2024-07-26T11:02:45.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.263+0000 7f6918c00640 1 -- 172.21.15.134:0/2505467977 <== mon.2 v2:172.21.15.189:3300/0 8 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v11)=0 v11) ==== 123+0+0 (secure 0 0 0) 0x7f690c016c00 con 0x7f691c1035b0 2024-07-26T11:02:45.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.266+0000 7f6922a00640 1 -- 172.21.15.134:0/2505467977 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f690003edb0 msgr2=0x7f69000412c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:45.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.266+0000 7f6922a00640 1 --2- 172.21.15.134:0/2505467977 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f690003edb0 0x7f69000412c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f690402a9e0 tx=0x7f6904000bc0 comp rx=0 tx=0).stop 2024-07-26T11:02:45.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.266+0000 7f6922a00640 1 -- 172.21.15.134:0/2505467977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f691c1035b0 msgr2=0x7f691c1993a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:45.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.266+0000 7f6922a00640 1 --2- 172.21.15.134:0/2505467977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f691c1035b0 0x7f691c1993a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f690c00ebc0 tx=0x7f690c00c6a0 comp rx=0 tx=0).stop 2024-07-26T11:02:45.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.266+0000 7f6922a00640 1 -- 172.21.15.134:0/2505467977 shutdown_connections 2024-07-26T11:02:45.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.266+0000 7f6922a00640 1 --2- 172.21.15.134:0/2505467977 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f690003edb0 0x7f69000412c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:45.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.266+0000 7f6922a00640 1 --2- 172.21.15.134:0/2505467977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f691c108ef0 0x7f691c199940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:45.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.266+0000 7f6922a00640 1 --2- 172.21.15.134:0/2505467977 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f691c109840 0x7f691c19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:45.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.266+0000 7f6922a00640 1 --2- 172.21.15.134:0/2505467977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f691c1035b0 0x7f691c1993a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:45.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.266+0000 7f6922a00640 1 -- 172.21.15.134:0/2505467977 >> 172.21.15.134:0/2505467977 conn(0x7f691c0ff040 msgr2=0x7f691c106d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:45.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.266+0000 7f6922a00640 1 -- 172.21.15.134:0/2505467977 shutdown_connections 2024-07-26T11:02:45.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:45.266+0000 7f6922a00640 1 -- 172.21.15.134:0/2505467977 wait complete. 2024-07-26T11:02:45.694 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Jul 26 11:02:45 smithi189 podman[39148]: 2024-07-26 11:02:45.506097218 +0000 UTC m=+0.016904660 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:02:46.194 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Jul 26 11:02:45 smithi189 podman[39148]: 2024-07-26 11:02:45.823930589 +0000 UTC m=+0.334738021 container init 1f89592e7558c77794582affc0a845f00e347b44e3cc90735f1d816e9dad4268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-y, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-07-26T11:02:46.194 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Jul 26 11:02:45 smithi189 podman[39148]: 2024-07-26 11:02:45.826791904 +0000 UTC m=+0.337599333 container start 1f89592e7558c77794582affc0a845f00e347b44e3cc90735f1d816e9dad4268 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-y, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-07-26T11:02:46.194 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Jul 26 11:02:45 smithi189 ceph-mgr[39167]: -- 172.21.15.189:0/2616366217 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5635a8c050e0 con 0x5635a8c23a80 2024-07-26T11:02:46.194 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Jul 26 11:02:45 smithi189 bash[39148]: 1f89592e7558c77794582affc0a845f00e347b44e3cc90735f1d816e9dad4268 2024-07-26T11:02:46.194 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Jul 26 11:02:45 smithi189 systemd[1]: Started Ceph mgr.y for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:02:46.386 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:46 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' 2024-07-26T11:02:46.386 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:46.386 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:46.386 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:46.386 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:46.386 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:46.496 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:46.496 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:46.497 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:46.499 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:46 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' 2024-07-26T11:02:46.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:46.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:46.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:46.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:46.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:46 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' 2024-07-26T11:02:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:46.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:46.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:46.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:02:47.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.957+0000 7f75f2e00640 1 -- 172.21.15.134:0/2586486852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ec101e20 msgr2=0x7f75ec102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:47.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.957+0000 7f75f2e00640 1 --2- 172.21.15.134:0/2586486852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ec101e20 0x7f75ec102230 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f75e0009900 tx=0x7f75e002a480 comp rx=0 tx=0).stop 2024-07-26T11:02:47.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.958+0000 7f75f2e00640 1 -- 172.21.15.134:0/2586486852 shutdown_connections 2024-07-26T11:02:47.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.958+0000 7f75f2e00640 1 --2- 172.21.15.134:0/2586486852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75ec10f9d0 0x7f75ec111df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:47.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.958+0000 7f75f2e00640 1 --2- 172.21.15.134:0/2586486852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75ec102770 0x7f75ec10f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:47.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.958+0000 7f75f2e00640 1 --2- 172.21.15.134:0/2586486852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ec101e20 0x7f75ec102230 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:47.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.958+0000 7f75f2e00640 1 -- 172.21.15.134:0/2586486852 >> 172.21.15.134:0/2586486852 conn(0x7f75ec0fdd00 msgr2=0x7f75ec100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:47.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.958+0000 7f75f2e00640 1 -- 172.21.15.134:0/2586486852 shutdown_connections 2024-07-26T11:02:47.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.958+0000 7f75f2e00640 1 -- 172.21.15.134:0/2586486852 wait complete. 2024-07-26T11:02:47.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.959+0000 7f75f2e00640 1 Processor -- start 2024-07-26T11:02:47.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.960+0000 7f75f2e00640 1 -- start start 2024-07-26T11:02:47.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.960+0000 7f75f2e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75ec101e20 0x7f75ec19bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:47.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.960+0000 7f75f2e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ec102770 0x7f75ec19c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:47.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.960+0000 7f75f2e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75ec10f9d0 0x7f75ec1a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:47.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.960+0000 7f75f2e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec115130 con 0x7f75ec101e20 2024-07-26T11:02:47.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.960+0000 7f75f2e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec114fb0 con 0x7f75ec102770 2024-07-26T11:02:47.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.960+0000 7f75f2e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec1152b0 con 0x7f75ec10f9d0 2024-07-26T11:02:47.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.960+0000 7f75f1a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75ec101e20 0x7f75ec19bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:47.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.961+0000 7f75f1a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75ec101e20 0x7f75ec19bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57318/0 (socket says 172.21.15.134:57318) 2024-07-26T11:02:47.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.961+0000 7f75f1a00640 1 -- 172.21.15.134:0/1390423890 learned_addr learned my addr 172.21.15.134:0/1390423890 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:02:47.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.961+0000 7f75f1000640 1 --2- 172.21.15.134:0/1390423890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ec102770 0x7f75ec19c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:47.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.961+0000 7f75f2400640 1 --2- 172.21.15.134:0/1390423890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75ec10f9d0 0x7f75ec1a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:47.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.961+0000 7f75f1a00640 1 -- 172.21.15.134:0/1390423890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75ec10f9d0 msgr2=0x7f75ec1a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:47.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.961+0000 7f75f1a00640 1 --2- 172.21.15.134:0/1390423890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75ec10f9d0 0x7f75ec1a1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:47.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.961+0000 7f75f1a00640 1 -- 172.21.15.134:0/1390423890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ec102770 msgr2=0x7f75ec19c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:47.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.961+0000 7f75f1a00640 1 --2- 172.21.15.134:0/1390423890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ec102770 0x7f75ec19c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:47.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.961+0000 7f75f1a00640 1 -- 172.21.15.134:0/1390423890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ec1a1bb0 con 0x7f75ec101e20 2024-07-26T11:02:47.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.961+0000 7f75f1000640 1 --2- 172.21.15.134:0/1390423890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ec102770 0x7f75ec19c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:47.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.961+0000 7f75f1a00640 1 --2- 172.21.15.134:0/1390423890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75ec101e20 0x7f75ec19bc40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f75e0004770 tx=0x7f75e0032940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:47.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.962+0000 7f75ea000640 1 -- 172.21.15.134:0/1390423890 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e003f030 con 0x7f75ec101e20 2024-07-26T11:02:47.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.962+0000 7f75ea000640 1 -- 172.21.15.134:0/1390423890 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(23 keys) ==== 1014+0+0 (secure 0 0 0) 0x7f75e0032d40 con 0x7f75ec101e20 2024-07-26T11:02:47.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.962+0000 7f75f2e00640 1 -- 172.21.15.134:0/1390423890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75ec1a1e40 con 0x7f75ec101e20 2024-07-26T11:02:47.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.962+0000 7f75ea000640 1 -- 172.21.15.134:0/1390423890 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e0035070 con 0x7f75ec101e20 2024-07-26T11:02:47.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.962+0000 7f75f2e00640 1 -- 172.21.15.134:0/1390423890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75ec1a8600 con 0x7f75ec101e20 2024-07-26T11:02:47.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.963+0000 7f75f2e00640 1 -- 172.21.15.134:0/1390423890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75ec102230 con 0x7f75ec101e20 2024-07-26T11:02:47.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.964+0000 7f75ea000640 1 -- 172.21.15.134:0/1390423890 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 17) ==== 50765+0+0 (secure 0 0 0) 0x7f75e0042030 con 0x7f75ec101e20 2024-07-26T11:02:47.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.964+0000 7f75ea000640 1 --2- 172.21.15.134:0/1390423890 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75d403edb0 0x7f75d40412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:47.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.964+0000 7f75ea000640 1 -- 172.21.15.134:0/1390423890 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f75e00738b0 con 0x7f75ec101e20 2024-07-26T11:02:47.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.964+0000 7f75f1000640 1 --2- 172.21.15.134:0/1390423890 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75d403edb0 0x7f75d40412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:47.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.965+0000 7f75f1000640 1 --2- 172.21.15.134:0/1390423890 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75d403edb0 0x7f75d40412c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f75d80095f0 tx=0x7f75d8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:47.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:46.970+0000 7f75ea000640 1 -- 172.21.15.134:0/1390423890 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f75e004a260 con 0x7f75ec101e20 2024-07-26T11:02:47.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:47.172+0000 7f75f2e00640 1 -- 172.21.15.134:0/1390423890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7f75ec10ab60 con 0x7f75ec101e20 2024-07-26T11:02:47.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:47.177+0000 7f75ea000640 1 -- 172.21.15.134:0/1390423890 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v12)=0 v12) ==== 123+0+0 (secure 0 0 0) 0x7f75e0003b70 con 0x7f75ec101e20 2024-07-26T11:02:47.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:47.178+0000 7f75ea000640 1 -- 172.21.15.134:0/1390423890 <== mon.0 v2:172.21.15.134:3300/0 8 ==== config(24 keys) ==== 1045+0+0 (secure 0 0 0) 0x7f75e0079080 con 0x7f75ec101e20 2024-07-26T11:02:47.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:47.180+0000 7f75f2e00640 1 -- 172.21.15.134:0/1390423890 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75d403edb0 msgr2=0x7f75d40412c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:47.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:47.180+0000 7f75f2e00640 1 --2- 172.21.15.134:0/1390423890 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75d403edb0 0x7f75d40412c0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f75d80095f0 tx=0x7f75d8005f90 comp rx=0 tx=0).stop 2024-07-26T11:02:47.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:47.180+0000 7f75f2e00640 1 -- 172.21.15.134:0/1390423890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75ec101e20 msgr2=0x7f75ec19bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:47.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:47.180+0000 7f75f2e00640 1 --2- 172.21.15.134:0/1390423890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75ec101e20 0x7f75ec19bc40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f75e0004770 tx=0x7f75e0032940 comp rx=0 tx=0).stop 2024-07-26T11:02:47.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:47.181+0000 7f75f2e00640 1 -- 172.21.15.134:0/1390423890 shutdown_connections 2024-07-26T11:02:47.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:47.181+0000 7f75f2e00640 1 --2- 172.21.15.134:0/1390423890 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75d403edb0 0x7f75d40412c0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:47.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:47.181+0000 7f75f2e00640 1 --2- 172.21.15.134:0/1390423890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75ec10f9d0 0x7f75ec1a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:47.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:47.181+0000 7f75f2e00640 1 --2- 172.21.15.134:0/1390423890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ec102770 0x7f75ec19c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:47.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:47.181+0000 7f75f2e00640 1 --2- 172.21.15.134:0/1390423890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75ec101e20 0x7f75ec19bc40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:47.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:47.181+0000 7f75f2e00640 1 -- 172.21.15.134:0/1390423890 >> 172.21.15.134:0/1390423890 conn(0x7f75ec0fdd00 msgr2=0x7f75ec100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:47.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:47.181+0000 7f75f2e00640 1 -- 172.21.15.134:0/1390423890 shutdown_connections 2024-07-26T11:02:47.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:47.181+0000 7f75f2e00640 1 -- 172.21.15.134:0/1390423890 wait complete. 2024-07-26T11:02:48.294 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:48 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1390423890' entity='client.admin' 2024-07-26T11:02:48.294 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:48.294 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:48.294 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:48.294 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:48.294 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:48.294 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:48.294 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:48.294 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:48 smithi134 ceph-mon[31966]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T11:02:48.295 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T11:02:48.295 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T11:02:48.295 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:48.295 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:48 smithi134 ceph-mon[31966]: Reconfiguring daemon mgr.x on smithi134 2024-07-26T11:02:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:48 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1390423890' entity='client.admin' 2024-07-26T11:02:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:48.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:48.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:48 smithi189 ceph-mon[36790]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T11:02:48.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T11:02:48.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T11:02:48.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:48.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:48 smithi189 ceph-mon[36790]: Reconfiguring daemon mgr.x on smithi134 2024-07-26T11:02:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:48 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1390423890' entity='client.admin' 2024-07-26T11:02:48.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:48.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:48.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:48.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:48.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:48.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:02:48.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:48.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:48 smithi193 ceph-mon[37473]: Reconfiguring mgr.x (unknown last config time)... 2024-07-26T11:02:48.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-07-26T11:02:48.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-07-26T11:02:48.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:02:48.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:48 smithi193 ceph-mon[37473]: Reconfiguring daemon mgr.x on smithi134 2024-07-26T11:02:49.441 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:49 smithi134 ceph-mon[31966]: from='mgr.? 172.21.15.193:0/3824455366' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T11:02:49.442 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:49 smithi134 ceph-mon[31966]: from='mgr.? 172.21.15.193:0/3824455366' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T11:02:49.442 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:49 smithi134 ceph-mon[31966]: Standby manager daemon z started 2024-07-26T11:02:49.442 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:49 smithi134 ceph-mon[31966]: from='mgr.? 172.21.15.193:0/3824455366' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T11:02:49.442 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:49 smithi134 ceph-mon[31966]: from='mgr.? 172.21.15.193:0/3824455366' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T11:02:49.442 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:49 smithi189 ceph-mon[36790]: from='mgr.? 172.21.15.193:0/3824455366' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T11:02:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:49 smithi189 ceph-mon[36790]: from='mgr.? 172.21.15.193:0/3824455366' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T11:02:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:49 smithi189 ceph-mon[36790]: Standby manager daemon z started 2024-07-26T11:02:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:49 smithi189 ceph-mon[36790]: from='mgr.? 172.21.15.193:0/3824455366' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T11:02:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:49 smithi189 ceph-mon[36790]: from='mgr.? 172.21.15.193:0/3824455366' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T11:02:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:49 smithi193 ceph-mon[37473]: from='mgr.? 172.21.15.193:0/3824455366' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-07-26T11:02:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:49 smithi193 ceph-mon[37473]: from='mgr.? 172.21.15.193:0/3824455366' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T11:02:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:49 smithi193 ceph-mon[37473]: Standby manager daemon z started 2024-07-26T11:02:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:49 smithi193 ceph-mon[37473]: from='mgr.? 172.21.15.193:0/3824455366' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-07-26T11:02:49.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:49 smithi193 ceph-mon[37473]: from='mgr.? 172.21.15.193:0/3824455366' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T11:02:49.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:02:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:50 smithi189 ceph-mon[36790]: mgrmap e18: x(active, since 15s), standbys: z 2024-07-26T11:02:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T11:02:50.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:50 smithi134 ceph-mon[31966]: mgrmap e18: x(active, since 15s), standbys: z 2024-07-26T11:02:50.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T11:02:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:50 smithi193 ceph-mon[37473]: mgrmap e18: x(active, since 15s), standbys: z 2024-07-26T11:02:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-07-26T11:02:51.145 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:51.149 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:51.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.490+0000 7f8a58a00640 1 -- 172.21.15.134:0/1702483828 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a541055d0 msgr2=0x7f8a541059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:51.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.490+0000 7f8a58a00640 1 --2- 172.21.15.134:0/1702483828 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a541055d0 0x7f8a541059e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8a44009900 tx=0x7f8a4402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:02:51.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.491+0000 7f8a58a00640 1 -- 172.21.15.134:0/1702483828 shutdown_connections 2024-07-26T11:02:51.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.491+0000 7f8a58a00640 1 --2- 172.21.15.134:0/1702483828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a5410b860 0x7f8a54112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:51.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.491+0000 7f8a58a00640 1 --2- 172.21.15.134:0/1702483828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a5410af10 0x7f8a5410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:51.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.491+0000 7f8a58a00640 1 --2- 172.21.15.134:0/1702483828 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a541055d0 0x7f8a541059e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:51.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.491+0000 7f8a58a00640 1 -- 172.21.15.134:0/1702483828 >> 172.21.15.134:0/1702483828 conn(0x7f8a54101040 msgr2=0x7f8a54103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:51.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.491+0000 7f8a58a00640 1 -- 172.21.15.134:0/1702483828 shutdown_connections 2024-07-26T11:02:51.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.491+0000 7f8a58a00640 1 -- 172.21.15.134:0/1702483828 wait complete. 2024-07-26T11:02:51.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.492+0000 7f8a58a00640 1 Processor -- start 2024-07-26T11:02:51.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.492+0000 7f8a58a00640 1 -- start start 2024-07-26T11:02:51.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a58a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a541055d0 0x7f8a5419bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:51.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a58a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a5410af10 0x7f8a5419c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:51.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a58a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a5410b860 0x7f8a541a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:51.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a58a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8a54115370 con 0x7f8a5410b860 2024-07-26T11:02:51.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a58a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541151f0 con 0x7f8a5410af10 2024-07-26T11:02:51.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a58a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8a541154f0 con 0x7f8a541055d0 2024-07-26T11:02:51.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a51e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a5410af10 0x7f8a5419c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:51.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a53200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a5410b860 0x7f8a541a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:51.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a51e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a5410af10 0x7f8a5419c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:53720/0 (socket says 172.21.15.134:53720) 2024-07-26T11:02:51.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a51e00640 1 -- 172.21.15.134:0/3885440075 learned_addr learned my addr 172.21.15.134:0/3885440075 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:02:51.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a52800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a541055d0 0x7f8a5419bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:51.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a53200640 1 -- 172.21.15.134:0/3885440075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a541055d0 msgr2=0x7f8a5419bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:51.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a53200640 1 --2- 172.21.15.134:0/3885440075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a541055d0 0x7f8a5419bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:51.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a53200640 1 -- 172.21.15.134:0/3885440075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a5410af10 msgr2=0x7f8a5419c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:51.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a53200640 1 --2- 172.21.15.134:0/3885440075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a5410af10 0x7f8a5419c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:51.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a53200640 1 -- 172.21.15.134:0/3885440075 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a541a1cf0 con 0x7f8a5410b860 2024-07-26T11:02:51.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.493+0000 7f8a51e00640 1 --2- 172.21.15.134:0/3885440075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a5410af10 0x7f8a5419c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:02:51.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.494+0000 7f8a53200640 1 --2- 172.21.15.134:0/3885440075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a5410b860 0x7f8a541a15c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00e940 tx=0x7f8a4c00ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:51.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.494+0000 7f8a4b400640 1 -- 172.21.15.134:0/3885440075 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c004d20 con 0x7f8a5410b860 2024-07-26T11:02:51.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.494+0000 7f8a4b400640 1 -- 172.21.15.134:0/3885440075 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(24 keys) ==== 1045+0+0 (secure 0 0 0) 0x7f8a4c005020 con 0x7f8a5410b860 2024-07-26T11:02:51.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.494+0000 7f8a4b400640 1 -- 172.21.15.134:0/3885440075 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c0116d0 con 0x7f8a5410b860 2024-07-26T11:02:51.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.494+0000 7f8a58a00640 1 -- 172.21.15.134:0/3885440075 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a541a1fe0 con 0x7f8a5410b860 2024-07-26T11:02:51.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.494+0000 7f8a58a00640 1 -- 172.21.15.134:0/3885440075 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a541a8740 con 0x7f8a5410b860 2024-07-26T11:02:51.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.496+0000 7f8a58a00640 1 -- 172.21.15.134:0/3885440075 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a541059e0 con 0x7f8a5410b860 2024-07-26T11:02:51.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.497+0000 7f8a4b400640 1 -- 172.21.15.134:0/3885440075 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 18) ==== 100893+0+0 (secure 0 0 0) 0x7f8a4c00cb20 con 0x7f8a5410b860 2024-07-26T11:02:51.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.498+0000 7f8a4b400640 1 --2- 172.21.15.134:0/3885440075 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8a380794a0 0x7f8a3807b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:02:51.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.498+0000 7f8a4b400640 1 -- 172.21.15.134:0/3885440075 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8a4c09aa70 con 0x7f8a5410b860 2024-07-26T11:02:51.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.498+0000 7f8a52800640 1 --2- 172.21.15.134:0/3885440075 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8a380794a0 0x7f8a3807b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:02:51.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.499+0000 7f8a52800640 1 --2- 172.21.15.134:0/3885440075 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8a380794a0 0x7f8a3807b9b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8a44000c00 tx=0x7f8a44005e80 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:02:51.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.503+0000 7f8a4b400640 1 -- 172.21.15.134:0/3885440075 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8a4c016030 con 0x7f8a5410b860 2024-07-26T11:02:51.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.709+0000 7f8a58a00640 1 -- 172.21.15.134:0/3885440075 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7f8a5410d5e0 con 0x7f8a5410b860 2024-07-26T11:02:51.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.717+0000 7f8a4b400640 1 -- 172.21.15.134:0/3885440075 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v13)=0 v13) ==== 114+0+0 (secure 0 0 0) 0x7f8a4c063770 con 0x7f8a5410b860 2024-07-26T11:02:51.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.720+0000 7f8a58a00640 1 -- 172.21.15.134:0/3885440075 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8a380794a0 msgr2=0x7f8a3807b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:51.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.720+0000 7f8a58a00640 1 --2- 172.21.15.134:0/3885440075 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8a380794a0 0x7f8a3807b9b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8a44000c00 tx=0x7f8a44005e80 comp rx=0 tx=0).stop 2024-07-26T11:02:51.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.720+0000 7f8a58a00640 1 -- 172.21.15.134:0/3885440075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a5410b860 msgr2=0x7f8a541a15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:02:51.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.720+0000 7f8a4b400640 1 -- 172.21.15.134:0/3885440075 <== mon.0 v2:172.21.15.134:3300/0 8 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8a4c068e50 con 0x7f8a5410b860 2024-07-26T11:02:51.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.720+0000 7f8a58a00640 1 --2- 172.21.15.134:0/3885440075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a5410b860 0x7f8a541a15c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00e940 tx=0x7f8a4c00ee00 comp rx=0 tx=0).stop 2024-07-26T11:02:51.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.720+0000 7f8a58a00640 1 -- 172.21.15.134:0/3885440075 shutdown_connections 2024-07-26T11:02:51.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.720+0000 7f8a58a00640 1 --2- 172.21.15.134:0/3885440075 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8a380794a0 0x7f8a3807b9b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:51.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.720+0000 7f8a58a00640 1 --2- 172.21.15.134:0/3885440075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a5410b860 0x7f8a541a15c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:51.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.720+0000 7f8a58a00640 1 --2- 172.21.15.134:0/3885440075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a5410af10 0x7f8a5419c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:51.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.720+0000 7f8a58a00640 1 --2- 172.21.15.134:0/3885440075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a541055d0 0x7f8a5419bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:02:51.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.720+0000 7f8a58a00640 1 -- 172.21.15.134:0/3885440075 >> 172.21.15.134:0/3885440075 conn(0x7f8a54101040 msgr2=0x7f8a54103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:02:51.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.721+0000 7f8a58a00640 1 -- 172.21.15.134:0/3885440075 shutdown_connections 2024-07-26T11:02:51.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:50.721+0000 7f8a58a00640 1 -- 172.21.15.134:0/3885440075 wait complete. 2024-07-26T11:02:51.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:51 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3885440075' entity='client.admin' 2024-07-26T11:02:51.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:51 smithi134 ceph-mon[31966]: Standby manager daemon y started 2024-07-26T11:02:51.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:51 smithi134 ceph-mon[31966]: from='mgr.? 172.21.15.189:0/187363581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T11:02:51.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:51 smithi134 ceph-mon[31966]: from='mgr.? 172.21.15.189:0/187363581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T11:02:51.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:51 smithi134 ceph-mon[31966]: from='mgr.? 172.21.15.189:0/187363581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T11:02:51.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:51 smithi134 ceph-mon[31966]: from='mgr.? 172.21.15.189:0/187363581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T11:02:52.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:51 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3885440075' entity='client.admin' 2024-07-26T11:02:52.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:51 smithi193 ceph-mon[37473]: Standby manager daemon y started 2024-07-26T11:02:52.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:51 smithi193 ceph-mon[37473]: from='mgr.? 172.21.15.189:0/187363581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T11:02:52.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:51 smithi193 ceph-mon[37473]: from='mgr.? 172.21.15.189:0/187363581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T11:02:52.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:51 smithi193 ceph-mon[37473]: from='mgr.? 172.21.15.189:0/187363581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T11:02:52.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:51 smithi193 ceph-mon[37473]: from='mgr.? 172.21.15.189:0/187363581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T11:02:52.181 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:51 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3885440075' entity='client.admin' 2024-07-26T11:02:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:51 smithi189 ceph-mon[36790]: Standby manager daemon y started 2024-07-26T11:02:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:51 smithi189 ceph-mon[36790]: from='mgr.? 172.21.15.189:0/187363581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-07-26T11:02:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:51 smithi189 ceph-mon[36790]: from='mgr.? 172.21.15.189:0/187363581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-07-26T11:02:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:51 smithi189 ceph-mon[36790]: from='mgr.? 172.21.15.189:0/187363581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-07-26T11:02:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:51 smithi189 ceph-mon[36790]: from='mgr.? 172.21.15.189:0/187363581' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-07-26T11:02:52.872 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:52 smithi134 ceph-mon[31966]: mgrmap e19: x(active, since 18s), standbys: z, y 2024-07-26T11:02:52.873 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T11:02:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:52 smithi193 ceph-mon[37473]: mgrmap e19: x(active, since 18s), standbys: z, y 2024-07-26T11:02:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T11:02:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:52 smithi189 ceph-mon[36790]: mgrmap e19: x(active, since 18s), standbys: z, y 2024-07-26T11:02:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-07-26T11:02:54.926 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:54 smithi134 ceph-mon[31966]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:54 smithi193 ceph-mon[37473]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:54 smithi189 ceph-mon[36790]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:56 smithi193 ceph-mon[37473]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:57.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:56 smithi134 ceph-mon[31966]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:56 smithi189 ceph-mon[36790]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:58.840 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:58.840 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:02:58.890 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi134') 2024-07-26T11:02:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:58 smithi193 ceph-mon[37473]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:02:58 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1248417324' entity='client.admin' 2024-07-26T11:02:59.048 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:58 smithi134 ceph-mon[31966]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:59.048 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:02:58 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1248417324' entity='client.admin' 2024-07-26T11:02:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:58 smithi189 ceph-mon[36790]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:02:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:02:58 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1248417324' entity='client.admin' 2024-07-26T11:03:00.175 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:03:00.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.159+0000 7efe1a000640 1 -- 172.21.15.134:0/3892815411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe141057a0 msgr2=0x7efe14105bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:00.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.159+0000 7efe1a000640 1 --2- 172.21.15.134:0/3892815411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe141057a0 0x7efe14105bb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efe04009900 tx=0x7efe0402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:00.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.159+0000 7efe1a000640 1 -- 172.21.15.134:0/3892815411 shutdown_connections 2024-07-26T11:03:00.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.159+0000 7efe1a000640 1 --2- 172.21.15.134:0/3892815411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe1410ba30 0x7efe14112260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:00.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.159+0000 7efe1a000640 1 --2- 172.21.15.134:0/3892815411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe1410b0e0 0x7efe1410b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:00.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.159+0000 7efe1a000640 1 --2- 172.21.15.134:0/3892815411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe141057a0 0x7efe14105bb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:00.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.159+0000 7efe1a000640 1 -- 172.21.15.134:0/3892815411 >> 172.21.15.134:0/3892815411 conn(0x7efe14101230 msgr2=0x7efe14103680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:00.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.160+0000 7efe1a000640 1 -- 172.21.15.134:0/3892815411 shutdown_connections 2024-07-26T11:03:00.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.160+0000 7efe1a000640 1 -- 172.21.15.134:0/3892815411 wait complete. 2024-07-26T11:03:00.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.161+0000 7efe1a000640 1 Processor -- start 2024-07-26T11:03:00.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.161+0000 7efe1a000640 1 -- start start 2024-07-26T11:03:00.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.161+0000 7efe1a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe141057a0 0x7efe1419c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:00.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.161+0000 7efe1a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe1410b0e0 0x7efe1419c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:00.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.161+0000 7efe1a000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe1410ba30 0x7efe141a1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:00.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.161+0000 7efe1a000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efe14115610 con 0x7efe1410ba30 2024-07-26T11:03:00.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.161+0000 7efe1a000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efe14115490 con 0x7efe1410b0e0 2024-07-26T11:03:00.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.161+0000 7efe1a000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efe14115790 con 0x7efe141057a0 2024-07-26T11:03:00.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.162+0000 7efe13e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe1410ba30 0x7efe141a1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:00.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.162+0000 7efe13400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe141057a0 0x7efe1419c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:00.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.162+0000 7efe13e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe1410ba30 0x7efe141a1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47560/0 (socket says 172.21.15.134:47560) 2024-07-26T11:03:00.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.162+0000 7efe13e00640 1 -- 172.21.15.134:0/1045483723 learned_addr learned my addr 172.21.15.134:0/1045483723 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:00.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.162+0000 7efe12a00640 1 --2- 172.21.15.134:0/1045483723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe1410b0e0 0x7efe1419c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:00.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.162+0000 7efe13400640 1 -- 172.21.15.134:0/1045483723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe1410b0e0 msgr2=0x7efe1419c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:00.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.162+0000 7efe13400640 1 --2- 172.21.15.134:0/1045483723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe1410b0e0 0x7efe1419c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:00.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.162+0000 7efe13400640 1 -- 172.21.15.134:0/1045483723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe1410ba30 msgr2=0x7efe141a1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:00.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.162+0000 7efe13400640 1 --2- 172.21.15.134:0/1045483723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe1410ba30 0x7efe141a1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:00.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.162+0000 7efe13400640 1 -- 172.21.15.134:0/1045483723 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe141a1fb0 con 0x7efe141057a0 2024-07-26T11:03:00.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.163+0000 7efe13400640 1 --2- 172.21.15.134:0/1045483723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe141057a0 0x7efe1419c040 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7efe0402a910 tx=0x7efe04003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:00.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.163+0000 7efe0be00640 1 -- 172.21.15.134:0/1045483723 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0403f070 con 0x7efe141057a0 2024-07-26T11:03:00.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.163+0000 7efe0be00640 1 -- 172.21.15.134:0/1045483723 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7efe0403baf0 con 0x7efe141057a0 2024-07-26T11:03:00.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.163+0000 7efe0be00640 1 -- 172.21.15.134:0/1045483723 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe04035030 con 0x7efe141057a0 2024-07-26T11:03:00.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.163+0000 7efe1a000640 1 -- 172.21.15.134:0/1045483723 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe141a2240 con 0x7efe141057a0 2024-07-26T11:03:00.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.163+0000 7efe1a000640 1 -- 172.21.15.134:0/1045483723 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe141a8a00 con 0x7efe141057a0 2024-07-26T11:03:00.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.169+0000 7efe0be00640 1 -- 172.21.15.134:0/1045483723 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7efe04042070 con 0x7efe141057a0 2024-07-26T11:03:00.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.170+0000 7efe1a000640 1 -- 172.21.15.134:0/1045483723 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe14105bb0 con 0x7efe141057a0 2024-07-26T11:03:00.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.171+0000 7efe0be00640 1 --2- 172.21.15.134:0/1045483723 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7efdf80b3c10 0x7efdf80b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:00.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.171+0000 7efe0be00640 1 -- 172.21.15.134:0/1045483723 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7efe041024c0 con 0x7efe141057a0 2024-07-26T11:03:00.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.172+0000 7efe12a00640 1 --2- 172.21.15.134:0/1045483723 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7efdf80b3c10 0x7efdf80b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:00.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.173+0000 7efe12a00640 1 --2- 172.21.15.134:0/1045483723 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7efdf80b3c10 0x7efdf80b6120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efe1410ab10 tx=0x7efe00005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:00.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.176+0000 7efe0be00640 1 -- 172.21.15.134:0/1045483723 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7efe040cb0c0 con 0x7efe141057a0 2024-07-26T11:03:00.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.403+0000 7efe1a000640 1 -- 172.21.15.134:0/1045483723 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7efe1419d170 con 0x7efe141057a0 2024-07-26T11:03:00.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.410+0000 7efe0be00640 1 -- 172.21.15.134:0/1045483723 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v15)=0 v15) ==== 121+0+0 (secure 0 0 0) 0x7efe040cf610 con 0x7efe141057a0 2024-07-26T11:03:00.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.413+0000 7efe1a000640 1 -- 172.21.15.134:0/1045483723 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7efdf80b3c10 msgr2=0x7efdf80b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:00.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.413+0000 7efe1a000640 1 --2- 172.21.15.134:0/1045483723 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7efdf80b3c10 0x7efdf80b6120 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efe1410ab10 tx=0x7efe00005f90 comp rx=0 tx=0).stop 2024-07-26T11:03:00.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.413+0000 7efe1a000640 1 -- 172.21.15.134:0/1045483723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe141057a0 msgr2=0x7efe1419c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:00.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.413+0000 7efe1a000640 1 --2- 172.21.15.134:0/1045483723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe141057a0 0x7efe1419c040 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7efe0402a910 tx=0x7efe04003b90 comp rx=0 tx=0).stop 2024-07-26T11:03:00.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.413+0000 7efe1a000640 1 -- 172.21.15.134:0/1045483723 shutdown_connections 2024-07-26T11:03:00.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.413+0000 7efe1a000640 1 --2- 172.21.15.134:0/1045483723 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7efdf80b3c10 0x7efdf80b6120 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:00.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.413+0000 7efe1a000640 1 --2- 172.21.15.134:0/1045483723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe1410ba30 0x7efe141a1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:00.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.413+0000 7efe1a000640 1 --2- 172.21.15.134:0/1045483723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe1410b0e0 0x7efe1419c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:00.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.413+0000 7efe1a000640 1 --2- 172.21.15.134:0/1045483723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe141057a0 0x7efe1419c040 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:00.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.413+0000 7efe1a000640 1 -- 172.21.15.134:0/1045483723 >> 172.21.15.134:0/1045483723 conn(0x7efe14101230 msgr2=0x7efe14103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:00.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.413+0000 7efe1a000640 1 -- 172.21.15.134:0/1045483723 shutdown_connections 2024-07-26T11:03:00.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:00.413+0000 7efe1a000640 1 -- 172.21.15.134:0/1045483723 wait complete. 2024-07-26T11:03:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:00 smithi193 ceph-mon[37473]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:00 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' 2024-07-26T11:03:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:00 smithi134 ceph-mon[31966]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:00 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' 2024-07-26T11:03:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:00 smithi189 ceph-mon[36790]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:00 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' 2024-07-26T11:03:01.970 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi134') 2024-07-26T11:03:01.970 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi134') 2024-07-26T11:03:01.970 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi134') 2024-07-26T11:03:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:02 smithi193 ceph-mon[37473]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:03.106 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:02 smithi134 ceph-mon[31966]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:02 smithi189 ceph-mon[36790]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:03.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.772+0000 7f771e000640 1 -- 172.21.15.134:0/573048735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77181033e0 msgr2=0x7f77181037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:03.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.772+0000 7f771e000640 1 --2- 172.21.15.134:0/573048735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77181033e0 0x7f77181037f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f770c009900 tx=0x7f770c02a410 comp rx=0 tx=0).stop 2024-07-26T11:03:03.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.773+0000 7f771e000640 1 -- 172.21.15.134:0/573048735 shutdown_connections 2024-07-26T11:03:03.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.773+0000 7f771e000640 1 --2- 172.21.15.134:0/573048735 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7718109670 0x7f771810fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:03.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.773+0000 7f771e000640 1 --2- 172.21.15.134:0/573048735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7718108d20 0x7f7718109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:03.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.773+0000 7f771e000640 1 --2- 172.21.15.134:0/573048735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77181033e0 0x7f77181037f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:03.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.773+0000 7f771e000640 1 -- 172.21.15.134:0/573048735 >> 172.21.15.134:0/573048735 conn(0x7f77180fee30 msgr2=0x7f7718101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:03.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.773+0000 7f771e000640 1 -- 172.21.15.134:0/573048735 shutdown_connections 2024-07-26T11:03:03.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.773+0000 7f771e000640 1 -- 172.21.15.134:0/573048735 wait complete. 2024-07-26T11:03:03.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.775+0000 7f771e000640 1 Processor -- start 2024-07-26T11:03:03.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.775+0000 7f771e000640 1 -- start start 2024-07-26T11:03:03.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.776+0000 7f771e000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77181033e0 0x7f771819be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:03.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.776+0000 7f771e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7718108d20 0x7f771819c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:03.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.776+0000 7f771e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7718109670 0x7f77181a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:03.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.776+0000 7f771e000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7718113250 con 0x7f77181033e0 2024-07-26T11:03:03.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.776+0000 7f771e000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f77181130d0 con 0x7f7718109670 2024-07-26T11:03:03.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.776+0000 7f771e000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f77181133d0 con 0x7f7718108d20 2024-07-26T11:03:03.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.776+0000 7f771cc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77181033e0 0x7f771819be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:03.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.776+0000 7f771cc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77181033e0 0x7f771819be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47578/0 (socket says 172.21.15.134:47578) 2024-07-26T11:03:03.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.776+0000 7f771cc00640 1 -- 172.21.15.134:0/1094141384 learned_addr learned my addr 172.21.15.134:0/1094141384 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:03.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.777+0000 7f7717e00640 1 --2- 172.21.15.134:0/1094141384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7718108d20 0x7f771819c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:03.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.777+0000 7f771d600640 1 --2- 172.21.15.134:0/1094141384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7718109670 0x7f77181a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:03.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.777+0000 7f771cc00640 1 -- 172.21.15.134:0/1094141384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7718108d20 msgr2=0x7f771819c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:03.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.777+0000 7f771cc00640 1 --2- 172.21.15.134:0/1094141384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7718108d20 0x7f771819c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:03.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.777+0000 7f771cc00640 1 -- 172.21.15.134:0/1094141384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7718109670 msgr2=0x7f77181a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:03.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.777+0000 7f771cc00640 1 --2- 172.21.15.134:0/1094141384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7718109670 0x7f77181a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:03.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.777+0000 7f771cc00640 1 -- 172.21.15.134:0/1094141384 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77181a1db0 con 0x7f77181033e0 2024-07-26T11:03:03.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.777+0000 7f7717e00640 1 --2- 172.21.15.134:0/1094141384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7718108d20 0x7f771819c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:03.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.777+0000 7f771cc00640 1 --2- 172.21.15.134:0/1094141384 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77181033e0 0x7f771819be40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f770c02aa00 tx=0x7f770c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:03.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.778+0000 7f7715600640 1 -- 172.21.15.134:0/1094141384 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c03f070 con 0x7f77181033e0 2024-07-26T11:03:03.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.778+0000 7f7715600640 1 -- 172.21.15.134:0/1094141384 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f770c03bb00 con 0x7f77181033e0 2024-07-26T11:03:03.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.778+0000 7f771e000640 1 -- 172.21.15.134:0/1094141384 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77181a2040 con 0x7f77181033e0 2024-07-26T11:03:03.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.778+0000 7f7715600640 1 -- 172.21.15.134:0/1094141384 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c035030 con 0x7f77181033e0 2024-07-26T11:03:03.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.778+0000 7f771e000640 1 -- 172.21.15.134:0/1094141384 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77181a8800 con 0x7f77181033e0 2024-07-26T11:03:03.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.779+0000 7f771e000640 1 -- 172.21.15.134:0/1094141384 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77181037f0 con 0x7f77181033e0 2024-07-26T11:03:03.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.782+0000 7f7715600640 1 -- 172.21.15.134:0/1094141384 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f770c042070 con 0x7f77181033e0 2024-07-26T11:03:03.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.784+0000 7f7715600640 1 --2- 172.21.15.134:0/1094141384 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f77000b3c10 0x7f77000b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:03.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.785+0000 7f7715600640 1 -- 172.21.15.134:0/1094141384 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f770c102720 con 0x7f77181033e0 2024-07-26T11:03:03.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.785+0000 7f7717e00640 1 --2- 172.21.15.134:0/1094141384 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f77000b3c10 0x7f77000b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:03.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.785+0000 7f7717e00640 1 --2- 172.21.15.134:0/1094141384 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f77000b3c10 0x7f77000b6120 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7718105390 tx=0x7f7704005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:03.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.788+0000 7f7715600640 1 -- 172.21.15.134:0/1094141384 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f770c0cb280 con 0x7f77181033e0 2024-07-26T11:03:03.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.960+0000 7fcca2e00640 1 -- 172.21.15.134:0/4041869339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc9c104d20 msgr2=0x7fcc9c10d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:03.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.960+0000 7fcca2e00640 1 --2- 172.21.15.134:0/4041869339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc9c104d20 0x7fcc9c10d360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcc88009900 tx=0x7fcc8802a410 comp rx=0 tx=0).stop 2024-07-26T11:03:03.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.960+0000 7fcca2e00640 1 -- 172.21.15.134:0/4041869339 shutdown_connections 2024-07-26T11:03:03.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.960+0000 7fcca2e00640 1 --2- 172.21.15.134:0/4041869339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc9c104d20 0x7fcc9c10d360 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:03.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.960+0000 7fcca2e00640 1 --2- 172.21.15.134:0/4041869339 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc9c10d8a0 0x7fcc9c10fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:03.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.960+0000 7fcca2e00640 1 --2- 172.21.15.134:0/4041869339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc9c1042f0 0x7fcc9c104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:03.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.960+0000 7fcca2e00640 1 -- 172.21.15.134:0/4041869339 >> 172.21.15.134:0/4041869339 conn(0x7fcc9c079170 msgr2=0x7fcc9c079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:03.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.960+0000 7fcca2e00640 1 -- 172.21.15.134:0/4041869339 shutdown_connections 2024-07-26T11:03:03.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.960+0000 7fcca2e00640 1 -- 172.21.15.134:0/4041869339 wait complete. 2024-07-26T11:03:03.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.962+0000 7fcca2e00640 1 Processor -- start 2024-07-26T11:03:03.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.962+0000 7fcca2e00640 1 -- start start 2024-07-26T11:03:03.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.962+0000 7fcca2e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc9c1042f0 0x7fcc9c199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:03.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.962+0000 7fcca2e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc9c104d20 0x7fcc9c19a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:03.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.962+0000 7fcca2e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc9c10d8a0 0x7fcc9c19f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:03.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.962+0000 7fcca2e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9c113350 con 0x7fcc9c104d20 2024-07-26T11:03:03.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.962+0000 7fcca2e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9c1131d0 con 0x7fcc9c1042f0 2024-07-26T11:03:03.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.962+0000 7fcca2e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcc9c1134d0 con 0x7fcc9c10d8a0 2024-07-26T11:03:03.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.962+0000 7fcc9be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc9c1042f0 0x7fcc9c199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:03.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.962+0000 7fcc9b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc9c104d20 0x7fcc9c19a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:03.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.963+0000 7fcc9be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc9c1042f0 0x7fcc9c199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:53166/0 (socket says 172.21.15.134:53166) 2024-07-26T11:03:03.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.963+0000 7fcc9be00640 1 -- 172.21.15.134:0/258725402 learned_addr learned my addr 172.21.15.134:0/258725402 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:03.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.963+0000 7fcca0e00640 1 --2- 172.21.15.134:0/258725402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc9c10d8a0 0x7fcc9c19f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:03.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.963+0000 7fcc9be00640 1 -- 172.21.15.134:0/258725402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc9c10d8a0 msgr2=0x7fcc9c19f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:03.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.963+0000 7fcc9be00640 1 --2- 172.21.15.134:0/258725402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc9c10d8a0 0x7fcc9c19f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:03.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.963+0000 7fcc9be00640 1 -- 172.21.15.134:0/258725402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc9c104d20 msgr2=0x7fcc9c19a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:03.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.963+0000 7fcc9be00640 1 --2- 172.21.15.134:0/258725402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc9c104d20 0x7fcc9c19a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:03.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.963+0000 7fcc9be00640 1 -- 172.21.15.134:0/258725402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc9c19fdb0 con 0x7fcc9c1042f0 2024-07-26T11:03:03.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.963+0000 7fcc9be00640 1 --2- 172.21.15.134:0/258725402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc9c1042f0 0x7fcc9c199e40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00e9c0 tx=0x7fcc8c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:03.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.964+0000 7fcc98c00640 1 -- 172.21.15.134:0/258725402 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c00cd80 con 0x7fcc9c1042f0 2024-07-26T11:03:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.964+0000 7fcc98c00640 1 -- 172.21.15.134:0/258725402 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcc8c0035f0 con 0x7fcc9c1042f0 2024-07-26T11:03:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.964+0000 7fcc98c00640 1 -- 172.21.15.134:0/258725402 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c0044c0 con 0x7fcc9c1042f0 2024-07-26T11:03:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.964+0000 7fcca2e00640 1 -- 172.21.15.134:0/258725402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc9c1a00a0 con 0x7fcc9c1042f0 2024-07-26T11:03:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.964+0000 7fcca2e00640 1 -- 172.21.15.134:0/258725402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc9c1a6800 con 0x7fcc9c1042f0 2024-07-26T11:03:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.965+0000 7fcca2e00640 1 -- 172.21.15.134:0/258725402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc9c06bbb0 con 0x7fcc9c1042f0 2024-07-26T11:03:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.967+0000 7fcc98c00640 1 -- 172.21.15.134:0/258725402 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fcc8c004000 con 0x7fcc9c1042f0 2024-07-26T11:03:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.969+0000 7fcc98c00640 1 --2- 172.21.15.134:0/258725402 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc800b3c10 0x7fcc800b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.969+0000 7fcc98c00640 1 -- 172.21.15.134:0/258725402 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fcc8c0e24d0 con 0x7fcc9c1042f0 2024-07-26T11:03:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.969+0000 7fcc9b400640 1 --2- 172.21.15.134:0/258725402 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc800b3c10 0x7fcc800b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.970+0000 7fcc9b400640 1 --2- 172.21.15.134:0/258725402 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc800b3c10 0x7fcc800b6120 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcc88004770 tx=0x7fcc880058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.970+0000 7fcc98c00640 1 -- 172.21.15.134:0/258725402 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcc8c0ab680 con 0x7fcc9c1042f0 2024-07-26T11:03:03.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.992+0000 7f771e000640 1 -- 172.21.15.134:0/1094141384 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7f77181a8a00 con 0x7f77181033e0 2024-07-26T11:03:04.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:03.998+0000 7f7715600640 1 -- 172.21.15.134:0/1094141384 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v16)=0 v16) ==== 117+0+0 (secure 0 0 0) 0x7f770c0cf7d0 con 0x7f77181033e0 2024-07-26T11:03:04.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.001+0000 7f771e000640 1 -- 172.21.15.134:0/1094141384 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f77000b3c10 msgr2=0x7f77000b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:04.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.001+0000 7f771e000640 1 --2- 172.21.15.134:0/1094141384 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f77000b3c10 0x7f77000b6120 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f7718105390 tx=0x7f7704005f90 comp rx=0 tx=0).stop 2024-07-26T11:03:04.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.001+0000 7f771e000640 1 -- 172.21.15.134:0/1094141384 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77181033e0 msgr2=0x7f771819be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:04.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.002+0000 7f771e000640 1 --2- 172.21.15.134:0/1094141384 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77181033e0 0x7f771819be40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f770c02aa00 tx=0x7f770c003c10 comp rx=0 tx=0).stop 2024-07-26T11:03:04.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.002+0000 7f771e000640 1 -- 172.21.15.134:0/1094141384 shutdown_connections 2024-07-26T11:03:04.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.002+0000 7f771e000640 1 --2- 172.21.15.134:0/1094141384 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f77000b3c10 0x7f77000b6120 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:04.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.002+0000 7f771e000640 1 --2- 172.21.15.134:0/1094141384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7718108d20 0x7f771819c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:04.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.002+0000 7f771e000640 1 --2- 172.21.15.134:0/1094141384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7718109670 0x7f77181a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:04.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.002+0000 7f771e000640 1 --2- 172.21.15.134:0/1094141384 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77181033e0 0x7f771819be40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:04.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.002+0000 7f771e000640 1 -- 172.21.15.134:0/1094141384 >> 172.21.15.134:0/1094141384 conn(0x7f77180fee30 msgr2=0x7f771810e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:04.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.002+0000 7f771e000640 1 -- 172.21.15.134:0/1094141384 shutdown_connections 2024-07-26T11:03:04.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.002+0000 7f771e000640 1 -- 172.21.15.134:0/1094141384 wait complete. 2024-07-26T11:03:04.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.177+0000 7fcca2e00640 1 -- 172.21.15.134:0/258725402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7fcc9c104700 con 0x7fcc9c1042f0 2024-07-26T11:03:04.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.185+0000 7fcc98c00640 1 -- 172.21.15.134:0/258725402 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) ==== 138+0+0 (secure 0 0 0) 0x7fcc8c0aaf80 con 0x7fcc9c1042f0 2024-07-26T11:03:04.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.190+0000 7fcca2e00640 1 -- 172.21.15.134:0/258725402 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc800b3c10 msgr2=0x7fcc800b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:04.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.190+0000 7fcca2e00640 1 --2- 172.21.15.134:0/258725402 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc800b3c10 0x7fcc800b6120 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcc88004770 tx=0x7fcc880058d0 comp rx=0 tx=0).stop 2024-07-26T11:03:04.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.190+0000 7fcca2e00640 1 -- 172.21.15.134:0/258725402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc9c1042f0 msgr2=0x7fcc9c199e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:04.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.190+0000 7fcca2e00640 1 --2- 172.21.15.134:0/258725402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc9c1042f0 0x7fcc9c199e40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00e9c0 tx=0x7fcc8c00ee80 comp rx=0 tx=0).stop 2024-07-26T11:03:04.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.190+0000 7fcca2e00640 1 -- 172.21.15.134:0/258725402 shutdown_connections 2024-07-26T11:03:04.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.190+0000 7fcca2e00640 1 --2- 172.21.15.134:0/258725402 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc800b3c10 0x7fcc800b6120 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:04.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.190+0000 7fcca2e00640 1 --2- 172.21.15.134:0/258725402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc9c10d8a0 0x7fcc9c19f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:04.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.190+0000 7fcca2e00640 1 --2- 172.21.15.134:0/258725402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc9c104d20 0x7fcc9c19a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:04.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.190+0000 7fcca2e00640 1 --2- 172.21.15.134:0/258725402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc9c1042f0 0x7fcc9c199e40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:04.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.190+0000 7fcca2e00640 1 -- 172.21.15.134:0/258725402 >> 172.21.15.134:0/258725402 conn(0x7fcc9c079170 msgr2=0x7fcc9c0ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:04.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.190+0000 7fcca2e00640 1 -- 172.21.15.134:0/258725402 shutdown_connections 2024-07-26T11:03:04.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:04.190+0000 7fcca2e00640 1 -- 172.21.15.134:0/258725402 wait complete. 2024-07-26T11:03:04.894 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:04 smithi134 ceph-mon[31966]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:04.895 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:04 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1094141384' entity='client.admin' 2024-07-26T11:03:04.895 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:04 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' 2024-07-26T11:03:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:04 smithi193 ceph-mon[37473]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:04 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1094141384' entity='client.admin' 2024-07-26T11:03:05.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:04 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' 2024-07-26T11:03:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:04 smithi189 ceph-mon[36790]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:04 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1094141384' entity='client.admin' 2024-07-26T11:03:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:04 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' 2024-07-26T11:03:05.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.362+0000 7f1e7fe00640 1 -- 172.21.15.134:0/1003542925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e80104b40 msgr2=0x7f1e8010d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:05.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.362+0000 7f1e7fe00640 1 --2- 172.21.15.134:0/1003542925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e80104b40 0x7f1e8010d090 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c009900 tx=0x7f1e6c02a410 comp rx=0 tx=0).stop 2024-07-26T11:03:05.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.362+0000 7f1e7fe00640 1 -- 172.21.15.134:0/1003542925 shutdown_connections 2024-07-26T11:03:05.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.362+0000 7f1e7fe00640 1 --2- 172.21.15.134:0/1003542925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e80104b40 0x7f1e8010d090 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.362+0000 7f1e7fe00640 1 --2- 172.21.15.134:0/1003542925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e8010d650 0x7f1e8010fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.362+0000 7f1e7fe00640 1 --2- 172.21.15.134:0/1003542925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e801041f0 0x7f1e80104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.362+0000 7f1e7fe00640 1 -- 172.21.15.134:0/1003542925 >> 172.21.15.134:0/1003542925 conn(0x7f1e800fdcf0 msgr2=0x7f1e80100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:05.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.363+0000 7f1e7fe00640 1 -- 172.21.15.134:0/1003542925 shutdown_connections 2024-07-26T11:03:05.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.363+0000 7f1e7fe00640 1 -- 172.21.15.134:0/1003542925 wait complete. 2024-07-26T11:03:05.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.364+0000 7f1e7fe00640 1 Processor -- start 2024-07-26T11:03:05.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.364+0000 7f1e7fe00640 1 -- start start 2024-07-26T11:03:05.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.365+0000 7f1e7fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e801041f0 0x7f1e8019be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.365+0000 7f1e7fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e80104b40 0x7f1e8019c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.365+0000 7f1e7fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e8010d650 0x7f1e801a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.365+0000 7f1e7fe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1e80113190 con 0x7f1e801041f0 2024-07-26T11:03:05.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.365+0000 7f1e7fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1e80113010 con 0x7f1e80104b40 2024-07-26T11:03:05.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.365+0000 7f1e7fe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1e80113310 con 0x7f1e8010d650 2024-07-26T11:03:05.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.365+0000 7f1e7ea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e801041f0 0x7f1e8019be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.365+0000 7f1e7ea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e801041f0 0x7f1e8019be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47608/0 (socket says 172.21.15.134:47608) 2024-07-26T11:03:05.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.365+0000 7f1e7ea00640 1 -- 172.21.15.134:0/1342430351 learned_addr learned my addr 172.21.15.134:0/1342430351 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:05.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.365+0000 7f1e7e000640 1 --2- 172.21.15.134:0/1342430351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e80104b40 0x7f1e8019c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.366+0000 7f1e7f400640 1 --2- 172.21.15.134:0/1342430351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e8010d650 0x7f1e801a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.366+0000 7f1e7ea00640 1 -- 172.21.15.134:0/1342430351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e8010d650 msgr2=0x7f1e801a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:05.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.366+0000 7f1e7ea00640 1 --2- 172.21.15.134:0/1342430351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e8010d650 0x7f1e801a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.366+0000 7f1e7ea00640 1 -- 172.21.15.134:0/1342430351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e80104b40 msgr2=0x7f1e8019c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:05.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.366+0000 7f1e7ea00640 1 --2- 172.21.15.134:0/1342430351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e80104b40 0x7f1e8019c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.366+0000 7f1e7ea00640 1 -- 172.21.15.134:0/1342430351 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e801a1dd0 con 0x7f1e801041f0 2024-07-26T11:03:05.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.366+0000 7f1e7ea00640 1 --2- 172.21.15.134:0/1342430351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e801041f0 0x7f1e8019be60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1e7000e9f0 tx=0x7f1e7000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:05.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.366+0000 7f1e77400640 1 -- 172.21.15.134:0/1342430351 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e7000cd80 con 0x7f1e801041f0 2024-07-26T11:03:05.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.367+0000 7f1e7fe00640 1 -- 172.21.15.134:0/1342430351 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e801a20c0 con 0x7f1e801041f0 2024-07-26T11:03:05.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.367+0000 7f1e77400640 1 -- 172.21.15.134:0/1342430351 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1e700108c0 con 0x7f1e801041f0 2024-07-26T11:03:05.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.367+0000 7f1e77400640 1 -- 172.21.15.134:0/1342430351 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e700187d0 con 0x7f1e801041f0 2024-07-26T11:03:05.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.367+0000 7f1e7fe00640 1 -- 172.21.15.134:0/1342430351 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e801a8820 con 0x7f1e801041f0 2024-07-26T11:03:05.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.368+0000 7f1e7fe00640 1 -- 172.21.15.134:0/1342430351 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e80104600 con 0x7f1e801041f0 2024-07-26T11:03:05.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.370+0000 7f1e77400640 1 -- 172.21.15.134:0/1342430351 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f1e70010430 con 0x7f1e801041f0 2024-07-26T11:03:05.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.372+0000 7f1e77400640 1 --2- 172.21.15.134:0/1342430351 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1e640b3b90 0x7f1e640b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:05.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.372+0000 7f1e77400640 1 -- 172.21.15.134:0/1342430351 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f1e700e2470 con 0x7f1e801041f0 2024-07-26T11:03:05.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.372+0000 7f1e7e000640 1 --2- 172.21.15.134:0/1342430351 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1e640b3b90 0x7f1e640b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:05.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.373+0000 7f1e7e000640 1 --2- 172.21.15.134:0/1342430351 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1e640b3b90 0x7f1e640b60a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c02aa00 tx=0x7f1e6c000bc0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:05.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.376+0000 7f1e77400640 1 -- 172.21.15.134:0/1342430351 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1e700aafd0 con 0x7f1e801041f0 2024-07-26T11:03:05.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.578+0000 7f1e7fe00640 1 -- 172.21.15.134:0/1342430351 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f1e801a8b00 con 0x7f1e801041f0 2024-07-26T11:03:05.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.587+0000 7f1e77400640 1 -- 172.21.15.134:0/1342430351 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v18)=0 v18) ==== 108+0+0 (secure 0 0 0) 0x7f1e700af520 con 0x7f1e801041f0 2024-07-26T11:03:05.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.590+0000 7f1e7fe00640 1 -- 172.21.15.134:0/1342430351 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1e640b3b90 msgr2=0x7f1e640b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:05.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.590+0000 7f1e7fe00640 1 --2- 172.21.15.134:0/1342430351 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1e640b3b90 0x7f1e640b60a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c02aa00 tx=0x7f1e6c000bc0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.590+0000 7f1e7fe00640 1 -- 172.21.15.134:0/1342430351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e801041f0 msgr2=0x7f1e8019be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:05.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.590+0000 7f1e7fe00640 1 --2- 172.21.15.134:0/1342430351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e801041f0 0x7f1e8019be60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1e7000e9f0 tx=0x7f1e7000eeb0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.591+0000 7f1e7fe00640 1 -- 172.21.15.134:0/1342430351 shutdown_connections 2024-07-26T11:03:05.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.591+0000 7f1e7fe00640 1 --2- 172.21.15.134:0/1342430351 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1e640b3b90 0x7f1e640b60a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.591+0000 7f1e7fe00640 1 --2- 172.21.15.134:0/1342430351 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e8010d650 0x7f1e801a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.591+0000 7f1e7fe00640 1 --2- 172.21.15.134:0/1342430351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e80104b40 0x7f1e8019c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.591+0000 7f1e7fe00640 1 --2- 172.21.15.134:0/1342430351 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1e801041f0 0x7f1e8019be60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:05.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.591+0000 7f1e7fe00640 1 -- 172.21.15.134:0/1342430351 >> 172.21.15.134:0/1342430351 conn(0x7f1e800fdcf0 msgr2=0x7f1e80101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:05.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.591+0000 7f1e7fe00640 1 -- 172.21.15.134:0/1342430351 shutdown_connections 2024-07-26T11:03:05.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:05.591+0000 7f1e7fe00640 1 -- 172.21.15.134:0/1342430351 wait complete. 2024-07-26T11:03:06.852 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:06 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1342430351' entity='client.admin' 2024-07-26T11:03:06.852 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:06 smithi134 ceph-mon[31966]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:06.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.849+0000 7f45c0600640 1 -- 172.21.15.134:0/2646180569 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f45b8113e80 msgr2=0x7f45b8116350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:06.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.849+0000 7f45c0600640 1 --2- 172.21.15.134:0/2646180569 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f45b8113e80 0x7f45b8116350 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f45b0009860 tx=0x7f45b002a340 comp rx=0 tx=0).stop 2024-07-26T11:03:06.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.849+0000 7f45c0600640 1 -- 172.21.15.134:0/2646180569 shutdown_connections 2024-07-26T11:03:06.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.849+0000 7f45c0600640 1 --2- 172.21.15.134:0/2646180569 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f45b8113e80 0x7f45b8116350 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:06.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.849+0000 7f45c0600640 1 --2- 172.21.15.134:0/2646180569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45b8075800 0x7f45b8075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:06.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.849+0000 7f45c0600640 1 --2- 172.21.15.134:0/2646180569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b8076db0 0x7f45b80751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:06.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.849+0000 7f45c0600640 1 -- 172.21.15.134:0/2646180569 >> 172.21.15.134:0/2646180569 conn(0x7f45b80ff040 msgr2=0x7f45b8101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:06.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.850+0000 7f45c0600640 1 -- 172.21.15.134:0/2646180569 shutdown_connections 2024-07-26T11:03:06.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.850+0000 7f45c0600640 1 -- 172.21.15.134:0/2646180569 wait complete. 2024-07-26T11:03:06.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.851+0000 7f45c0600640 1 Processor -- start 2024-07-26T11:03:06.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.851+0000 7f45c0600640 1 -- start start 2024-07-26T11:03:06.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.852+0000 7f45c0600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b8075800 0x7f45b81a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:06.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.852+0000 7f45c0600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45b8076db0 0x7f45b81a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:06.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.852+0000 7f45c0600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f45b8113e80 0x7f45b81a5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:06.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.852+0000 7f45c0600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f45b81199a0 con 0x7f45b8113e80 2024-07-26T11:03:06.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.852+0000 7f45c0600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f45b8119820 con 0x7f45b8076db0 2024-07-26T11:03:06.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.852+0000 7f45c0600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f45b8119b20 con 0x7f45b8075800 2024-07-26T11:03:06.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.852+0000 7f45be600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f45b8113e80 0x7f45b81a5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:06.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.852+0000 7f45be600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f45b8113e80 0x7f45b81a5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47626/0 (socket says 172.21.15.134:47626) 2024-07-26T11:03:06.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.852+0000 7f45be600640 1 -- 172.21.15.134:0/1525833037 learned_addr learned my addr 172.21.15.134:0/1525833037 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:06.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.852+0000 7f45bd200640 1 --2- 172.21.15.134:0/1525833037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45b8076db0 0x7f45b81a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:06.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.852+0000 7f45bdc00640 1 --2- 172.21.15.134:0/1525833037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b8075800 0x7f45b81a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:06.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.853+0000 7f45be600640 1 -- 172.21.15.134:0/1525833037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b8075800 msgr2=0x7f45b81a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:06.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.853+0000 7f45be600640 1 --2- 172.21.15.134:0/1525833037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b8075800 0x7f45b81a03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:06.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.853+0000 7f45be600640 1 -- 172.21.15.134:0/1525833037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45b8076db0 msgr2=0x7f45b81a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:06.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.853+0000 7f45be600640 1 --2- 172.21.15.134:0/1525833037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45b8076db0 0x7f45b81a0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:06.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.853+0000 7f45be600640 1 -- 172.21.15.134:0/1525833037 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b81a6340 con 0x7f45b8113e80 2024-07-26T11:03:06.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.853+0000 7f45bdc00640 1 --2- 172.21.15.134:0/1525833037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b8075800 0x7f45b81a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:06.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.853+0000 7f45be600640 1 --2- 172.21.15.134:0/1525833037 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f45b8113e80 0x7f45b81a5c10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f45b002a930 tx=0x7f45b0033f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:06.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.853+0000 7f45b6000640 1 -- 172.21.15.134:0/1525833037 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b0037040 con 0x7f45b8113e80 2024-07-26T11:03:06.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.854+0000 7f45b6000640 1 -- 172.21.15.134:0/1525833037 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f45b0040070 con 0x7f45b8113e80 2024-07-26T11:03:06.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.854+0000 7f45c0600640 1 -- 172.21.15.134:0/1525833037 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45b81a65d0 con 0x7f45b8113e80 2024-07-26T11:03:06.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.854+0000 7f45b6000640 1 -- 172.21.15.134:0/1525833037 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b00035f0 con 0x7f45b8113e80 2024-07-26T11:03:06.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.854+0000 7f45c0600640 1 -- 172.21.15.134:0/1525833037 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45b81acd90 con 0x7f45b8113e80 2024-07-26T11:03:06.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.855+0000 7f45c0600640 1 -- 172.21.15.134:0/1525833037 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45b80766f0 con 0x7f45b8113e80 2024-07-26T11:03:06.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.858+0000 7f45b6000640 1 -- 172.21.15.134:0/1525833037 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f45b0004050 con 0x7f45b8113e80 2024-07-26T11:03:06.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.859+0000 7f45b6000640 1 --2- 172.21.15.134:0/1525833037 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f459c0b3b90 0x7f459c0b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:06.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.859+0000 7f45b6000640 1 -- 172.21.15.134:0/1525833037 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f45b0101f30 con 0x7f45b8113e80 2024-07-26T11:03:06.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.859+0000 7f45bdc00640 1 --2- 172.21.15.134:0/1525833037 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f459c0b3b90 0x7f459c0b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:06.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.860+0000 7f45bdc00640 1 --2- 172.21.15.134:0/1525833037 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f459c0b3b90 0x7f459c0b60a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009f40 tx=0x7f45a8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:06.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:06.861+0000 7f45b6000640 1 -- 172.21.15.134:0/1525833037 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f45b00cb180 con 0x7f45b8113e80 2024-07-26T11:03:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:06 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1342430351' entity='client.admin' 2024-07-26T11:03:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:06 smithi189 ceph-mon[36790]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:06 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1342430351' entity='client.admin' 2024-07-26T11:03:07.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:06 smithi193 ceph-mon[37473]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:07.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:07.064+0000 7f45c0600640 1 -- 172.21.15.134:0/1525833037 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7f45b8075c10 con 0x7f45b8113e80 2024-07-26T11:03:07.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:07.074+0000 7f45b6000640 1 -- 172.21.15.134:0/1525833037 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v19)=0 v19) ==== 123+0+0 (secure 0 0 0) 0x7f45b8075c10 con 0x7f45b8113e80 2024-07-26T11:03:07.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:07.075+0000 7f45b6000640 1 -- 172.21.15.134:0/1525833037 <== mon.0 v2:172.21.15.134:3300/0 8 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45b00c6200 con 0x7f45b8113e80 2024-07-26T11:03:07.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:07.077+0000 7f45c0600640 1 -- 172.21.15.134:0/1525833037 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f459c0b3b90 msgr2=0x7f459c0b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:07.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:07.077+0000 7f45c0600640 1 --2- 172.21.15.134:0/1525833037 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f459c0b3b90 0x7f459c0b60a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009f40 tx=0x7f45a8005f90 comp rx=0 tx=0).stop 2024-07-26T11:03:07.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:07.077+0000 7f45c0600640 1 -- 172.21.15.134:0/1525833037 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f45b8113e80 msgr2=0x7f45b81a5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:07.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:07.077+0000 7f45c0600640 1 --2- 172.21.15.134:0/1525833037 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f45b8113e80 0x7f45b81a5c10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f45b002a930 tx=0x7f45b0033f80 comp rx=0 tx=0).stop 2024-07-26T11:03:07.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:07.077+0000 7f45c0600640 1 -- 172.21.15.134:0/1525833037 shutdown_connections 2024-07-26T11:03:07.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:07.077+0000 7f45c0600640 1 --2- 172.21.15.134:0/1525833037 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f459c0b3b90 0x7f459c0b60a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:07.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:07.077+0000 7f45c0600640 1 --2- 172.21.15.134:0/1525833037 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f45b8113e80 0x7f45b81a5c10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:07.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:07.077+0000 7f45c0600640 1 --2- 172.21.15.134:0/1525833037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45b8076db0 0x7f45b81a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:07.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:07.077+0000 7f45c0600640 1 --2- 172.21.15.134:0/1525833037 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45b8075800 0x7f45b81a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:07.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:07.077+0000 7f45c0600640 1 -- 172.21.15.134:0/1525833037 >> 172.21.15.134:0/1525833037 conn(0x7f45b80ff040 msgr2=0x7f45b810fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:07.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:07.078+0000 7f45c0600640 1 -- 172.21.15.134:0/1525833037 shutdown_connections 2024-07-26T11:03:07.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:07.078+0000 7f45c0600640 1 -- 172.21.15.134:0/1525833037 wait complete. 2024-07-26T11:03:08.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.075+0000 7f987be00640 1 -- 172.21.15.134:0/785624538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f987c1041f0 msgr2=0x7f987c104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:08.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.075+0000 7f987be00640 1 --2- 172.21.15.134:0/785624538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f987c1041f0 0x7f987c104600 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f986c009900 tx=0x7f986c02a410 comp rx=0 tx=0).stop 2024-07-26T11:03:08.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.075+0000 7f987be00640 1 -- 172.21.15.134:0/785624538 shutdown_connections 2024-07-26T11:03:08.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.075+0000 7f987be00640 1 --2- 172.21.15.134:0/785624538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f987c10d650 0x7f987c10fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:08.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.075+0000 7f987be00640 1 --2- 172.21.15.134:0/785624538 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f987c104b40 0x7f987c10d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:08.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.075+0000 7f987be00640 1 --2- 172.21.15.134:0/785624538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f987c1041f0 0x7f987c104600 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:08.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.075+0000 7f987be00640 1 -- 172.21.15.134:0/785624538 >> 172.21.15.134:0/785624538 conn(0x7f987c0fdcf0 msgr2=0x7f987c100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:08.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.075+0000 7f987be00640 1 -- 172.21.15.134:0/785624538 shutdown_connections 2024-07-26T11:03:08.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.075+0000 7f987be00640 1 -- 172.21.15.134:0/785624538 wait complete. 2024-07-26T11:03:08.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.076+0000 7f987be00640 1 Processor -- start 2024-07-26T11:03:08.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.077+0000 7f987be00640 1 -- start start 2024-07-26T11:03:08.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.077+0000 7f987be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f987c1041f0 0x7f987c19be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:08.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.077+0000 7f987be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f987c104b40 0x7f987c19c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:08.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.077+0000 7f987be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f987c10d650 0x7f987c1a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:08.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.077+0000 7f987be00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f987c1131f0 con 0x7f987c10d650 2024-07-26T11:03:08.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.077+0000 7f987be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f987c113070 con 0x7f987c104b40 2024-07-26T11:03:08.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.077+0000 7f987be00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f987c113370 con 0x7f987c1041f0 2024-07-26T11:03:08.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.077+0000 7f987b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f987c10d650 0x7f987c1a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:08.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.077+0000 7f987b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f987c10d650 0x7f987c1a1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:46878/0 (socket says 172.21.15.134:46878) 2024-07-26T11:03:08.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.077+0000 7f987b400640 1 -- 172.21.15.134:0/1248417324 learned_addr learned my addr 172.21.15.134:0/1248417324 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:08.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.078+0000 7f987aa00640 1 --2- 172.21.15.134:0/1248417324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f987c1041f0 0x7f987c19be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:08.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.078+0000 7f9873e00640 1 --2- 172.21.15.134:0/1248417324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f987c104b40 0x7f987c19c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:08.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.078+0000 7f987b400640 1 -- 172.21.15.134:0/1248417324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f987c1041f0 msgr2=0x7f987c19be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:08.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.078+0000 7f987b400640 1 --2- 172.21.15.134:0/1248417324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f987c1041f0 0x7f987c19be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:08.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.078+0000 7f987b400640 1 -- 172.21.15.134:0/1248417324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f987c104b40 msgr2=0x7f987c19c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:08.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.078+0000 7f987b400640 1 --2- 172.21.15.134:0/1248417324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f987c104b40 0x7f987c19c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:08.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.078+0000 7f987b400640 1 -- 172.21.15.134:0/1248417324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987c1a1db0 con 0x7f987c10d650 2024-07-26T11:03:08.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.078+0000 7f987aa00640 1 --2- 172.21.15.134:0/1248417324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f987c1041f0 0x7f987c19be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:08.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.078+0000 7f987b400640 1 --2- 172.21.15.134:0/1248417324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f987c10d650 0x7f987c1a1680 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f987400eb10 tx=0x7f987400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:08.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.078+0000 7f9873400640 1 -- 172.21.15.134:0/1248417324 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9874004b20 con 0x7f987c10d650 2024-07-26T11:03:08.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.079+0000 7f9873400640 1 -- 172.21.15.134:0/1248417324 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(25 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9874004cc0 con 0x7f987c10d650 2024-07-26T11:03:08.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.079+0000 7f987be00640 1 -- 172.21.15.134:0/1248417324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f987c1a20a0 con 0x7f987c10d650 2024-07-26T11:03:08.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.079+0000 7f9873400640 1 -- 172.21.15.134:0/1248417324 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9874010690 con 0x7f987c10d650 2024-07-26T11:03:08.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.079+0000 7f987be00640 1 -- 172.21.15.134:0/1248417324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f987c1a8800 con 0x7f987c10d650 2024-07-26T11:03:08.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.080+0000 7f987be00640 1 -- 172.21.15.134:0/1248417324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f987c104600 con 0x7f987c10d650 2024-07-26T11:03:08.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.083+0000 7f9873400640 1 -- 172.21.15.134:0/1248417324 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f9874005020 con 0x7f987c10d650 2024-07-26T11:03:08.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.084+0000 7f9873400640 1 --2- 172.21.15.134:0/1248417324 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98600b3b90 0x7f98600b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:08.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.085+0000 7f9873400640 1 -- 172.21.15.134:0/1248417324 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f98740e2570 con 0x7f987c10d650 2024-07-26T11:03:08.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.085+0000 7f987aa00640 1 --2- 172.21.15.134:0/1248417324 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98600b3b90 0x7f98600b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:08.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.086+0000 7f987aa00640 1 --2- 172.21.15.134:0/1248417324 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98600b3b90 0x7f98600b60a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f986c000c00 tx=0x7f986c005e60 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:08.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.089+0000 7f9873400640 1 -- 172.21.15.134:0/1248417324 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f98740ab7c0 con 0x7f987c10d650 2024-07-26T11:03:08.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.299+0000 7f987be00640 1 -- 172.21.15.134:0/1248417324 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=mds_bal_fragment_size_max}] v 0) -- 0x7f987c19cf70 con 0x7f987c10d650 2024-07-26T11:03:08.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.306+0000 7f9873400640 1 -- 172.21.15.134:0/1248417324 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_fragment_size_max}]=0 v14)=0 v14) ==== 127+0+0 (secure 0 0 0) 0x7f98740ab0c0 con 0x7f987c10d650 2024-07-26T11:03:08.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.309+0000 7f987be00640 1 -- 172.21.15.134:0/1248417324 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98600b3b90 msgr2=0x7f98600b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:08.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.309+0000 7f987be00640 1 --2- 172.21.15.134:0/1248417324 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98600b3b90 0x7f98600b60a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f986c000c00 tx=0x7f986c005e60 comp rx=0 tx=0).stop 2024-07-26T11:03:08.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.309+0000 7f987be00640 1 -- 172.21.15.134:0/1248417324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f987c10d650 msgr2=0x7f987c1a1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:08.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.309+0000 7f987be00640 1 --2- 172.21.15.134:0/1248417324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f987c10d650 0x7f987c1a1680 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f987400eb10 tx=0x7f987400c2d0 comp rx=0 tx=0).stop 2024-07-26T11:03:08.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.310+0000 7f987be00640 1 -- 172.21.15.134:0/1248417324 shutdown_connections 2024-07-26T11:03:08.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.310+0000 7f987be00640 1 --2- 172.21.15.134:0/1248417324 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98600b3b90 0x7f98600b60a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:08.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.310+0000 7f987be00640 1 --2- 172.21.15.134:0/1248417324 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f987c10d650 0x7f987c1a1680 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:08.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.310+0000 7f987be00640 1 --2- 172.21.15.134:0/1248417324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f987c104b40 0x7f987c19c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:08.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.310+0000 7f987be00640 1 --2- 172.21.15.134:0/1248417324 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f987c1041f0 0x7f987c19be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:08.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.310+0000 7f987be00640 1 -- 172.21.15.134:0/1248417324 >> 172.21.15.134:0/1248417324 conn(0x7f987c0fdcf0 msgr2=0x7f987c101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:08.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.310+0000 7f987be00640 1 -- 172.21.15.134:0/1248417324 shutdown_connections 2024-07-26T11:03:08.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:02:58.310+0000 7f987be00640 1 -- 172.21.15.134:0/1248417324 wait complete. 2024-07-26T11:03:08.392 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:08 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1525833037' entity='client.admin' 2024-07-26T11:03:08.392 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:08 smithi134 ceph-mon[31966]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:08.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:08 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1525833037' entity='client.admin' 2024-07-26T11:03:08.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:08 smithi189 ceph-mon[36790]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:08 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1525833037' entity='client.admin' 2024-07-26T11:03:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:08 smithi193 ceph-mon[37473]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:10.548 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:10 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2337480577' entity='client.admin' 2024-07-26T11:03:10.549 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:10 smithi134 ceph-mon[31966]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:10 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2337480577' entity='client.admin' 2024-07-26T11:03:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:10 smithi189 ceph-mon[36790]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:10.712 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:03:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:10 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2337480577' entity='client.admin' 2024-07-26T11:03:10.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:10 smithi193 ceph-mon[37473]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:11.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.080+0000 7fbeeb200640 1 -- 172.21.15.134:0/2020208737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbeec109650 msgr2=0x7fbeec10fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:11.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.080+0000 7fbeeb200640 1 --2- 172.21.15.134:0/2020208737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbeec109650 0x7fbeec10fe80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbee4009900 tx=0x7fbee402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:11.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.080+0000 7fbeeb200640 1 -- 172.21.15.134:0/2020208737 shutdown_connections 2024-07-26T11:03:11.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.080+0000 7fbeeb200640 1 --2- 172.21.15.134:0/2020208737 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbeec109650 0x7fbeec10fe80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:11.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.080+0000 7fbeeb200640 1 --2- 172.21.15.134:0/2020208737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbeec108d00 0x7fbeec109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:11.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.080+0000 7fbeeb200640 1 --2- 172.21.15.134:0/2020208737 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbeec1033c0 0x7fbeec1037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:11.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.080+0000 7fbeeb200640 1 -- 172.21.15.134:0/2020208737 >> 172.21.15.134:0/2020208737 conn(0x7fbeec0fee10 msgr2=0x7fbeec101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:11.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.080+0000 7fbeeb200640 1 -- 172.21.15.134:0/2020208737 shutdown_connections 2024-07-26T11:03:11.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.081+0000 7fbeeb200640 1 -- 172.21.15.134:0/2020208737 wait complete. 2024-07-26T11:03:11.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.082+0000 7fbeeb200640 1 Processor -- start 2024-07-26T11:03:11.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.082+0000 7fbeeb200640 1 -- start start 2024-07-26T11:03:11.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.082+0000 7fbeeb200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbeec1033c0 0x7fbeec19bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:11.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.082+0000 7fbeeb200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbeec108d00 0x7fbeec19c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:11.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.082+0000 7fbeeb200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbeec109650 0x7fbeec1a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:11.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.082+0000 7fbeeb200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec113350 con 0x7fbeec1033c0 2024-07-26T11:03:11.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.082+0000 7fbeeb200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec1131d0 con 0x7fbeec108d00 2024-07-26T11:03:11.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.082+0000 7fbeeb200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec1134d0 con 0x7fbeec109650 2024-07-26T11:03:11.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.083+0000 7fbee9e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbeec1033c0 0x7fbeec19bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:11.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.083+0000 7fbee1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbeec108d00 0x7fbeec19c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:11.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.083+0000 7fbee1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbeec108d00 0x7fbeec19c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:53216/0 (socket says 172.21.15.134:53216) 2024-07-26T11:03:11.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.083+0000 7fbee9e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbeec1033c0 0x7fbeec19bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:47656/0 (socket says 172.21.15.134:47656) 2024-07-26T11:03:11.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.083+0000 7fbee1400640 1 -- 172.21.15.134:0/2337480577 learned_addr learned my addr 172.21.15.134:0/2337480577 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:11.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.083+0000 7fbeea800640 1 --2- 172.21.15.134:0/2337480577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbeec109650 0x7fbeec1a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:11.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.083+0000 7fbee9e00640 1 -- 172.21.15.134:0/2337480577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbeec109650 msgr2=0x7fbeec1a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:11.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.083+0000 7fbee9e00640 1 --2- 172.21.15.134:0/2337480577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbeec109650 0x7fbeec1a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:11.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.083+0000 7fbee9e00640 1 -- 172.21.15.134:0/2337480577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbeec108d00 msgr2=0x7fbeec19c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:11.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.083+0000 7fbee9e00640 1 --2- 172.21.15.134:0/2337480577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbeec108d00 0x7fbeec19c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:11.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.083+0000 7fbee9e00640 1 -- 172.21.15.134:0/2337480577 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeec1a1e50 con 0x7fbeec1033c0 2024-07-26T11:03:11.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.084+0000 7fbee9e00640 1 --2- 172.21.15.134:0/2337480577 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbeec1033c0 0x7fbeec19bee0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00eb00 tx=0x7fbedc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:11.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.084+0000 7fbee3400640 1 -- 172.21.15.134:0/2337480577 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc010430 con 0x7fbeec1033c0 2024-07-26T11:03:11.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.084+0000 7fbee3400640 1 -- 172.21.15.134:0/2337480577 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbedc010a80 con 0x7fbeec1033c0 2024-07-26T11:03:11.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.084+0000 7fbeeb200640 1 -- 172.21.15.134:0/2337480577 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeec1a2140 con 0x7fbeec1033c0 2024-07-26T11:03:11.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.084+0000 7fbee3400640 1 -- 172.21.15.134:0/2337480577 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc017a90 con 0x7fbeec1033c0 2024-07-26T11:03:11.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.084+0000 7fbeeb200640 1 -- 172.21.15.134:0/2337480577 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeec1a88a0 con 0x7fbeec1033c0 2024-07-26T11:03:11.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.085+0000 7fbeeb200640 1 -- 172.21.15.134:0/2337480577 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeec1037d0 con 0x7fbeec1033c0 2024-07-26T11:03:11.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.088+0000 7fbee3400640 1 -- 172.21.15.134:0/2337480577 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fbedc017c50 con 0x7fbeec1033c0 2024-07-26T11:03:11.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.090+0000 7fbee3400640 1 --2- 172.21.15.134:0/2337480577 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbed00b3c10 0x7fbed00b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:11.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.090+0000 7fbee3400640 1 -- 172.21.15.134:0/2337480577 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fbedc0e2a90 con 0x7fbeec1033c0 2024-07-26T11:03:11.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.090+0000 7fbee1400640 1 --2- 172.21.15.134:0/2337480577 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbed00b3c10 0x7fbed00b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:11.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.091+0000 7fbee1400640 1 --2- 172.21.15.134:0/2337480577 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbed00b3c10 0x7fbed00b6120 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbed40095f0 tx=0x7fbed4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:11.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.092+0000 7fbee3400640 1 -- 172.21.15.134:0/2337480577 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbedc0abd70 con 0x7fbeec1033c0 2024-07-26T11:03:11.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.294+0000 7fbeeb200640 1 -- 172.21.15.134:0/2337480577 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7fbeec1a8be0 con 0x7fbeec1033c0 2024-07-26T11:03:11.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.301+0000 7fbee3400640 1 -- 172.21.15.134:0/2337480577 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v20)=0 v20) ==== 106+0+0 (secure 0 0 0) 0x7fbedc0ab670 con 0x7fbeec1033c0 2024-07-26T11:03:11.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.304+0000 7fbeeb200640 1 -- 172.21.15.134:0/2337480577 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbed00b3c10 msgr2=0x7fbed00b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:11.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.304+0000 7fbeeb200640 1 --2- 172.21.15.134:0/2337480577 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbed00b3c10 0x7fbed00b6120 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbed40095f0 tx=0x7fbed4005f90 comp rx=0 tx=0).stop 2024-07-26T11:03:11.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.304+0000 7fbeeb200640 1 -- 172.21.15.134:0/2337480577 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbeec1033c0 msgr2=0x7fbeec19bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:11.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.304+0000 7fbeeb200640 1 --2- 172.21.15.134:0/2337480577 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbeec1033c0 0x7fbeec19bee0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00eb00 tx=0x7fbedc00c2d0 comp rx=0 tx=0).stop 2024-07-26T11:03:11.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.304+0000 7fbeeb200640 1 -- 172.21.15.134:0/2337480577 shutdown_connections 2024-07-26T11:03:11.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.304+0000 7fbeeb200640 1 --2- 172.21.15.134:0/2337480577 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbed00b3c10 0x7fbed00b6120 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:11.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.304+0000 7fbeeb200640 1 --2- 172.21.15.134:0/2337480577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbeec109650 0x7fbeec1a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:11.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.304+0000 7fbeeb200640 1 --2- 172.21.15.134:0/2337480577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbeec108d00 0x7fbeec19c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:11.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.304+0000 7fbeeb200640 1 --2- 172.21.15.134:0/2337480577 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbeec1033c0 0x7fbeec19bee0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:11.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.304+0000 7fbeeb200640 1 -- 172.21.15.134:0/2337480577 >> 172.21.15.134:0/2337480577 conn(0x7fbeec0fee10 msgr2=0x7fbeec1072e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:11.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.304+0000 7fbeeb200640 1 -- 172.21.15.134:0/2337480577 shutdown_connections 2024-07-26T11:03:11.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:09.304+0000 7fbeeb200640 1 -- 172.21.15.134:0/2337480577 wait complete. 2024-07-26T11:03:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:12 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3595947397' entity='client.admin' 2024-07-26T11:03:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:12 smithi193 ceph-mon[37473]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:12 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3595947397' entity='client.admin' 2024-07-26T11:03:12.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:12 smithi134 ceph-mon[31966]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:12 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3595947397' entity='client.admin' 2024-07-26T11:03:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:12 smithi189 ceph-mon[36790]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:14.074 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi134') 2024-07-26T11:03:15.023 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:14 smithi134 ceph-mon[31966]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:14 smithi189 ceph-mon[36790]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:14 smithi193 ceph-mon[37473]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:15.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.382+0000 7f6664e00640 1 -- 172.21.15.134:0/2987898897 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66601099f0 msgr2=0x7f6660109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:15.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.382+0000 7f6664e00640 1 --2- 172.21.15.134:0/2987898897 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66601099f0 0x7f6660109e00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6650009900 tx=0x7f665002a410 comp rx=0 tx=0).stop 2024-07-26T11:03:15.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.382+0000 7f6664e00640 1 -- 172.21.15.134:0/2987898897 shutdown_connections 2024-07-26T11:03:15.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.382+0000 7f6664e00640 1 --2- 172.21.15.134:0/2987898897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f666010fc80 0x7f66601164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:15.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.382+0000 7f6664e00640 1 --2- 172.21.15.134:0/2987898897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f666010f330 0x7f666010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:15.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.382+0000 7f6664e00640 1 --2- 172.21.15.134:0/2987898897 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66601099f0 0x7f6660109e00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:15.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.382+0000 7f6664e00640 1 -- 172.21.15.134:0/2987898897 >> 172.21.15.134:0/2987898897 conn(0x7f6660078ff0 msgr2=0x7f6660107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:15.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.382+0000 7f6664e00640 1 -- 172.21.15.134:0/2987898897 shutdown_connections 2024-07-26T11:03:15.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.383+0000 7f6664e00640 1 -- 172.21.15.134:0/2987898897 wait complete. 2024-07-26T11:03:15.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.384+0000 7f6664e00640 1 Processor -- start 2024-07-26T11:03:15.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.384+0000 7f6664e00640 1 -- start start 2024-07-26T11:03:15.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.384+0000 7f6664e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66601099f0 0x7f66601a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:15.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.384+0000 7f6664e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f666010f330 0x7f66601a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:15.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.384+0000 7f6664e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f666010fc80 0x7f66601a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:15.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.384+0000 7f6664e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6660119940 con 0x7f666010fc80 2024-07-26T11:03:15.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.384+0000 7f6664e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f66601197c0 con 0x7f66601099f0 2024-07-26T11:03:15.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.384+0000 7f6664e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6660119ac0 con 0x7f666010f330 2024-07-26T11:03:15.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.385+0000 7f665fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f666010fc80 0x7f66601a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:15.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.385+0000 7f665f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66601099f0 0x7f66601a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:15.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.385+0000 7f665fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f666010fc80 0x7f66601a5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38056/0 (socket says 172.21.15.134:38056) 2024-07-26T11:03:15.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.385+0000 7f665fe00640 1 -- 172.21.15.134:0/997789943 learned_addr learned my addr 172.21.15.134:0/997789943 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:15.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.385+0000 7f665ea00640 1 --2- 172.21.15.134:0/997789943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f666010f330 0x7f66601a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:15.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.385+0000 7f665fe00640 1 -- 172.21.15.134:0/997789943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f666010f330 msgr2=0x7f66601a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:15.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.385+0000 7f665fe00640 1 --2- 172.21.15.134:0/997789943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f666010f330 0x7f66601a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:15.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.385+0000 7f665fe00640 1 -- 172.21.15.134:0/997789943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66601099f0 msgr2=0x7f66601a0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:15.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.385+0000 7f665fe00640 1 --2- 172.21.15.134:0/997789943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66601099f0 0x7f66601a0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:15.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.385+0000 7f665fe00640 1 -- 172.21.15.134:0/997789943 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66601a62e0 con 0x7f666010fc80 2024-07-26T11:03:15.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.385+0000 7f665fe00640 1 --2- 172.21.15.134:0/997789943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f666010fc80 0x7f66601a5bb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6658009fd0 tx=0x7f665800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:15.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.386+0000 7f6657e00640 1 -- 172.21.15.134:0/997789943 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6658017070 con 0x7f666010fc80 2024-07-26T11:03:15.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.386+0000 7f6657e00640 1 -- 172.21.15.134:0/997789943 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6658003e20 con 0x7f666010fc80 2024-07-26T11:03:15.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.386+0000 7f6657e00640 1 -- 172.21.15.134:0/997789943 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6658010460 con 0x7f666010fc80 2024-07-26T11:03:15.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.386+0000 7f6664e00640 1 -- 172.21.15.134:0/997789943 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66601a65d0 con 0x7f666010fc80 2024-07-26T11:03:15.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.386+0000 7f6664e00640 1 -- 172.21.15.134:0/997789943 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66601acd30 con 0x7f666010fc80 2024-07-26T11:03:15.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.387+0000 7f6664e00640 1 -- 172.21.15.134:0/997789943 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6660109e00 con 0x7f666010fc80 2024-07-26T11:03:15.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.389+0000 7f6657e00640 1 -- 172.21.15.134:0/997789943 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f6658003970 con 0x7f666010fc80 2024-07-26T11:03:15.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.392+0000 7f6657e00640 1 --2- 172.21.15.134:0/997789943 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f66440b3c10 0x7f66440b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:15.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.392+0000 7f6657e00640 1 -- 172.21.15.134:0/997789943 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f66580e1cb0 con 0x7f666010fc80 2024-07-26T11:03:15.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.392+0000 7f665f400640 1 --2- 172.21.15.134:0/997789943 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f66440b3c10 0x7f66440b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:15.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.393+0000 7f665f400640 1 --2- 172.21.15.134:0/997789943 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f66440b3c10 0x7f66440b6120 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6650031440 tx=0x7f6650009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:15.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.396+0000 7f6657e00640 1 -- 172.21.15.134:0/997789943 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f66580aa810 con 0x7f666010fc80 2024-07-26T11:03:15.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.602+0000 7f6664e00640 1 -- 172.21.15.134:0/997789943 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=debug_cephsqlite}] v 0) -- 0x7f666002e0c0 con 0x7f666010fc80 2024-07-26T11:03:15.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.608+0000 7f6657e00640 1 -- 172.21.15.134:0/997789943 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_cephsqlite}]=0 v22)=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7f66580b3130 con 0x7f666010fc80 2024-07-26T11:03:15.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.611+0000 7f6664e00640 1 -- 172.21.15.134:0/997789943 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f66440b3c10 msgr2=0x7f66440b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:15.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.611+0000 7f6664e00640 1 --2- 172.21.15.134:0/997789943 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f66440b3c10 0x7f66440b6120 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6650031440 tx=0x7f6650009560 comp rx=0 tx=0).stop 2024-07-26T11:03:15.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.612+0000 7f6664e00640 1 -- 172.21.15.134:0/997789943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f666010fc80 msgr2=0x7f66601a5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:15.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.612+0000 7f6664e00640 1 --2- 172.21.15.134:0/997789943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f666010fc80 0x7f66601a5bb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6658009fd0 tx=0x7f665800b260 comp rx=0 tx=0).stop 2024-07-26T11:03:15.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.612+0000 7f6664e00640 1 -- 172.21.15.134:0/997789943 shutdown_connections 2024-07-26T11:03:15.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.612+0000 7f6664e00640 1 --2- 172.21.15.134:0/997789943 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f66440b3c10 0x7f66440b6120 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:15.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.612+0000 7f6664e00640 1 --2- 172.21.15.134:0/997789943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f666010f330 0x7f66601a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:15.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.612+0000 7f6664e00640 1 --2- 172.21.15.134:0/997789943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f666010fc80 0x7f66601a5bb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:15.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.612+0000 7f6664e00640 1 --2- 172.21.15.134:0/997789943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66601099f0 0x7f66601a0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:15.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.612+0000 7f6664e00640 1 -- 172.21.15.134:0/997789943 >> 172.21.15.134:0/997789943 conn(0x7f6660078ff0 msgr2=0x7f6660108010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:15.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.612+0000 7f6664e00640 1 -- 172.21.15.134:0/997789943 shutdown_connections 2024-07-26T11:03:15.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:15.612+0000 7f6664e00640 1 -- 172.21.15.134:0/997789943 wait complete. 2024-07-26T11:03:16.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.114+0000 7ff1ff200640 1 -- 172.21.15.134:0/2581945503 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff200076db0 msgr2=0x7ff2000751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:16.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.114+0000 7ff1ff200640 1 --2- 172.21.15.134:0/2581945503 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff200076db0 0x7ff2000751e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0009900 tx=0x7ff1f002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:16.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.114+0000 7ff1ff200640 1 -- 172.21.15.134:0/2581945503 shutdown_connections 2024-07-26T11:03:16.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.114+0000 7ff1ff200640 1 --2- 172.21.15.134:0/2581945503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff200113cb0 0x7ff200116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:16.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.114+0000 7ff1ff200640 1 --2- 172.21.15.134:0/2581945503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff200075800 0x7ff200075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:16.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.114+0000 7ff1ff200640 1 --2- 172.21.15.134:0/2581945503 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff200076db0 0x7ff2000751e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:16.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.114+0000 7ff1ff200640 1 -- 172.21.15.134:0/2581945503 >> 172.21.15.134:0/2581945503 conn(0x7ff2000fee70 msgr2=0x7ff2001012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:16.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.115+0000 7ff1ff200640 1 -- 172.21.15.134:0/2581945503 shutdown_connections 2024-07-26T11:03:16.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.115+0000 7ff1ff200640 1 -- 172.21.15.134:0/2581945503 wait complete. 2024-07-26T11:03:16.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.116+0000 7ff1ff200640 1 Processor -- start 2024-07-26T11:03:16.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.116+0000 7ff1ff200640 1 -- start start 2024-07-26T11:03:16.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.117+0000 7ff1ff200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff200075800 0x7ff2001a0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:16.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.117+0000 7ff1ff200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff200076db0 0x7ff2001a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:16.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.117+0000 7ff1ff200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff200113cb0 0x7ff2001a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:16.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.117+0000 7ff1ff200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff200119940 con 0x7ff200075800 2024-07-26T11:03:16.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.117+0000 7ff1ff200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff2001197c0 con 0x7ff200076db0 2024-07-26T11:03:16.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.117+0000 7ff1ff200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff200119ac0 con 0x7ff200113cb0 2024-07-26T11:03:16.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.117+0000 7ff1f5400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff200076db0 0x7ff2001a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:16.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.117+0000 7ff1f5400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff200076db0 0x7ff2001a08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:49620/0 (socket says 172.21.15.134:49620) 2024-07-26T11:03:16.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.117+0000 7ff1f5400640 1 -- 172.21.15.134:0/227327516 learned_addr learned my addr 172.21.15.134:0/227327516 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:16.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.117+0000 7ff1fe800640 1 --2- 172.21.15.134:0/227327516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff200113cb0 0x7ff2001a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:16.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.117+0000 7ff1fde00640 1 --2- 172.21.15.134:0/227327516 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff200075800 0x7ff2001a0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:16.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.118+0000 7ff1f5400640 1 -- 172.21.15.134:0/227327516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff200113cb0 msgr2=0x7ff2001a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:16.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.118+0000 7ff1f5400640 1 --2- 172.21.15.134:0/227327516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff200113cb0 0x7ff2001a5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:16.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.118+0000 7ff1f5400640 1 -- 172.21.15.134:0/227327516 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff200075800 msgr2=0x7ff2001a0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:16.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.118+0000 7ff1f5400640 1 --2- 172.21.15.134:0/227327516 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff200075800 0x7ff2001a0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:16.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.118+0000 7ff1f5400640 1 -- 172.21.15.134:0/227327516 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2001a62c0 con 0x7ff200076db0 2024-07-26T11:03:16.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.118+0000 7ff1fde00640 1 --2- 172.21.15.134:0/227327516 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff200075800 0x7ff2001a0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:16.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.118+0000 7ff1fe800640 1 --2- 172.21.15.134:0/227327516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff200113cb0 0x7ff2001a5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:16.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.118+0000 7ff1f5400640 1 --2- 172.21.15.134:0/227327516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff200076db0 0x7ff2001a08f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800e9c0 tx=0x7ff1e800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:16.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.118+0000 7ff1f7400640 1 -- 172.21.15.134:0/227327516 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e800cd80 con 0x7ff200076db0 2024-07-26T11:03:16.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.118+0000 7ff1ff200640 1 -- 172.21.15.134:0/227327516 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2001a65b0 con 0x7ff200076db0 2024-07-26T11:03:16.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.118+0000 7ff1f7400640 1 -- 172.21.15.134:0/227327516 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1e80054d0 con 0x7ff200076db0 2024-07-26T11:03:16.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.118+0000 7ff1f7400640 1 -- 172.21.15.134:0/227327516 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8010430 con 0x7ff200076db0 2024-07-26T11:03:16.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.118+0000 7ff1ff200640 1 -- 172.21.15.134:0/227327516 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2001acd10 con 0x7ff200076db0 2024-07-26T11:03:16.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.120+0000 7ff1ff200640 1 -- 172.21.15.134:0/227327516 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff200075c10 con 0x7ff200076db0 2024-07-26T11:03:16.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.123+0000 7ff1f7400640 1 -- 172.21.15.134:0/227327516 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7ff1e8004fb0 con 0x7ff200076db0 2024-07-26T11:03:16.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.125+0000 7ff1f7400640 1 --2- 172.21.15.134:0/227327516 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff1e40b3c90 0x7ff1e40b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:16.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.125+0000 7ff1f7400640 1 -- 172.21.15.134:0/227327516 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7ff1e80e26f0 con 0x7ff200076db0 2024-07-26T11:03:16.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.125+0000 7ff1fde00640 1 --2- 172.21.15.134:0/227327516 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff1e40b3c90 0x7ff1e40b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:16.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.126+0000 7ff1f7400640 1 -- 172.21.15.134:0/227327516 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff1e8055600 con 0x7ff200076db0 2024-07-26T11:03:16.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.126+0000 7ff1fde00640 1 --2- 172.21.15.134:0/227327516 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff1e40b3c90 0x7ff1e40b61a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0000c00 tx=0x7ff1f0031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:16.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.325+0000 7ff1ff200640 1 -- 172.21.15.134:0/227327516 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) -- 0x7ff2001a1480 con 0x7ff200076db0 2024-07-26T11:03:16.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.333+0000 7ff1f7400640 1 -- 172.21.15.134:0/227327516 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v23)=0 v23) ==== 118+0+0 (secure 0 0 0) 0x7ff1e80ab8a0 con 0x7ff200076db0 2024-07-26T11:03:16.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.336+0000 7ff1ff200640 1 -- 172.21.15.134:0/227327516 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff1e40b3c90 msgr2=0x7ff1e40b61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:16.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.336+0000 7ff1ff200640 1 --2- 172.21.15.134:0/227327516 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff1e40b3c90 0x7ff1e40b61a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0000c00 tx=0x7ff1f0031400 comp rx=0 tx=0).stop 2024-07-26T11:03:16.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.336+0000 7ff1ff200640 1 -- 172.21.15.134:0/227327516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff200076db0 msgr2=0x7ff2001a08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:16.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.336+0000 7ff1ff200640 1 --2- 172.21.15.134:0/227327516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff200076db0 0x7ff2001a08f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800e9c0 tx=0x7ff1e800ee80 comp rx=0 tx=0).stop 2024-07-26T11:03:16.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.337+0000 7ff1ff200640 1 -- 172.21.15.134:0/227327516 shutdown_connections 2024-07-26T11:03:16.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.337+0000 7ff1ff200640 1 --2- 172.21.15.134:0/227327516 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff1e40b3c90 0x7ff1e40b61a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:16.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.337+0000 7ff1ff200640 1 --2- 172.21.15.134:0/227327516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff200113cb0 0x7ff2001a5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:16.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.337+0000 7ff1ff200640 1 --2- 172.21.15.134:0/227327516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff200076db0 0x7ff2001a08f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:16.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.337+0000 7ff1ff200640 1 --2- 172.21.15.134:0/227327516 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff200075800 0x7ff2001a0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:16.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.337+0000 7ff1ff200640 1 -- 172.21.15.134:0/227327516 >> 172.21.15.134:0/227327516 conn(0x7ff2000fee70 msgr2=0x7ff20010fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:16.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.338+0000 7ff1ff200640 1 -- 172.21.15.134:0/227327516 shutdown_connections 2024-07-26T11:03:16.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:16.338+0000 7ff1ff200640 1 -- 172.21.15.134:0/227327516 wait complete. 2024-07-26T11:03:16.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:16 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/997789943' entity='client.admin' 2024-07-26T11:03:16.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:16 smithi134 ceph-mon[31966]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:16.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:16 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' 2024-07-26T11:03:16.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:03:16.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:03:16.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:16.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:16.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:16.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:03:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:16 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/997789943' entity='client.admin' 2024-07-26T11:03:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:16 smithi189 ceph-mon[36790]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:16 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' 2024-07-26T11:03:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:03:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:03:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:16.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:03:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:16 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/997789943' entity='client.admin' 2024-07-26T11:03:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:16 smithi193 ceph-mon[37473]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:16 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' 2024-07-26T11:03:17.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:03:17.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:03:17.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:17.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:17.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:17.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:03:17.108 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi134') 2024-07-26T11:03:17.108 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi134') 2024-07-26T11:03:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:18 smithi134 ceph-mon[31966]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:18 smithi189 ceph-mon[36790]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:18 smithi193 ceph-mon[37473]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:19.812 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_fragment_size_max', '10000'], check_status=True, hostname='smithi134') 2024-07-26T11:03:19.812 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_merge_size', '5'], check_status=True, hostname='smithi134') 2024-07-26T11:03:20.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.344+0000 7fad11c00640 1 -- 172.21.15.134:0/801256249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad0c10b120 msgr2=0x7fad0c10b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:20.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.344+0000 7fad11c00640 1 --2- 172.21.15.134:0/801256249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad0c10b120 0x7fad0c10b530 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7facf4009900 tx=0x7facf402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.344+0000 7fad11c00640 1 -- 172.21.15.134:0/801256249 shutdown_connections 2024-07-26T11:03:20.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.344+0000 7fad11c00640 1 --2- 172.21.15.134:0/801256249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad0c10ba70 0x7fad0c1122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.344+0000 7fad11c00640 1 --2- 172.21.15.134:0/801256249 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad0c10b120 0x7fad0c10b530 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.344+0000 7fad11c00640 1 --2- 172.21.15.134:0/801256249 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad0c1057e0 0x7fad0c105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.344+0000 7fad11c00640 1 -- 172.21.15.134:0/801256249 >> 172.21.15.134:0/801256249 conn(0x7fad0c1011f0 msgr2=0x7fad0c103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:20.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.344+0000 7fad11c00640 1 -- 172.21.15.134:0/801256249 shutdown_connections 2024-07-26T11:03:20.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.344+0000 7fad11c00640 1 -- 172.21.15.134:0/801256249 wait complete. 2024-07-26T11:03:20.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.345+0000 7fad11c00640 1 Processor -- start 2024-07-26T11:03:20.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.346+0000 7fad11c00640 1 -- start start 2024-07-26T11:03:20.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.346+0000 7fad11c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad0c1057e0 0x7fad0c19c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:20.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.346+0000 7fad11c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad0c10b120 0x7fad0c19c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:20.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.346+0000 7fad11c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad0c10ba70 0x7fad0c1a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:20.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.346+0000 7fad11c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c115730 con 0x7fad0c10ba70 2024-07-26T11:03:20.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.346+0000 7fad11c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c1155b0 con 0x7fad0c1057e0 2024-07-26T11:03:20.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.346+0000 7fad11c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c1158b0 con 0x7fad0c10b120 2024-07-26T11:03:20.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.346+0000 7fad0aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad0c1057e0 0x7fad0c19c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:20.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.346+0000 7fad0aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad0c1057e0 0x7fad0c19c1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:33252/0 (socket says 172.21.15.134:33252) 2024-07-26T11:03:20.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.346+0000 7fad0aa00640 1 -- 172.21.15.134:0/1375818619 learned_addr learned my addr 172.21.15.134:0/1375818619 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:20.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.347+0000 7fad03e00640 1 --2- 172.21.15.134:0/1375818619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad0c10b120 0x7fad0c19c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:20.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.347+0000 7fad0b400640 1 --2- 172.21.15.134:0/1375818619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad0c10ba70 0x7fad0c1a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:20.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.347+0000 7fad0aa00640 1 -- 172.21.15.134:0/1375818619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad0c10b120 msgr2=0x7fad0c19c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:20.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.347+0000 7fad0aa00640 1 --2- 172.21.15.134:0/1375818619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad0c10b120 0x7fad0c19c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.347+0000 7fad0aa00640 1 -- 172.21.15.134:0/1375818619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad0c10ba70 msgr2=0x7fad0c1a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:20.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.347+0000 7fad0aa00640 1 --2- 172.21.15.134:0/1375818619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad0c10ba70 0x7fad0c1a1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.347+0000 7fad0aa00640 1 -- 172.21.15.134:0/1375818619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0c1a2130 con 0x7fad0c1057e0 2024-07-26T11:03:20.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.347+0000 7fad03e00640 1 --2- 172.21.15.134:0/1375818619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad0c10b120 0x7fad0c19c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:20.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.347+0000 7fad0b400640 1 --2- 172.21.15.134:0/1375818619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad0c10ba70 0x7fad0c1a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:20.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.347+0000 7fad0aa00640 1 --2- 172.21.15.134:0/1375818619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad0c1057e0 0x7fad0c19c1c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7facfc00e9f0 tx=0x7facfc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:20.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.347+0000 7fad03400640 1 -- 172.21.15.134:0/1375818619 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc00cd80 con 0x7fad0c1057e0 2024-07-26T11:03:20.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.348+0000 7fad03400640 1 -- 172.21.15.134:0/1375818619 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7facfc0108e0 con 0x7fad0c1057e0 2024-07-26T11:03:20.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.348+0000 7fad03400640 1 -- 172.21.15.134:0/1375818619 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc018980 con 0x7fad0c1057e0 2024-07-26T11:03:20.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.348+0000 7fad11c00640 1 -- 172.21.15.134:0/1375818619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad0c1a2420 con 0x7fad0c1057e0 2024-07-26T11:03:20.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.348+0000 7fad11c00640 1 -- 172.21.15.134:0/1375818619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad0c1a8b80 con 0x7fad0c1057e0 2024-07-26T11:03:20.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.349+0000 7fad11c00640 1 -- 172.21.15.134:0/1375818619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad0c105bf0 con 0x7fad0c1057e0 2024-07-26T11:03:20.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.351+0000 7fad03400640 1 -- 172.21.15.134:0/1375818619 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7facfc010430 con 0x7fad0c1057e0 2024-07-26T11:03:20.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.353+0000 7fad03400640 1 --2- 172.21.15.134:0/1375818619 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7facf00b3b90 0x7facf00b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:20.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.353+0000 7fad03400640 1 -- 172.21.15.134:0/1375818619 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7facfc0e2790 con 0x7fad0c1057e0 2024-07-26T11:03:20.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.353+0000 7fad03e00640 1 --2- 172.21.15.134:0/1375818619 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7facf00b3b90 0x7facf00b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:20.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.354+0000 7fad03e00640 1 --2- 172.21.15.134:0/1375818619 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7facf00b3b90 0x7facf00b60a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7facf402a9e0 tx=0x7facf4031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:20.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.355+0000 7fad03400640 1 -- 172.21.15.134:0/1375818619 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7facfc0ab940 con 0x7fad0c1057e0 2024-07-26T11:03:20.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.568+0000 7fad11c00640 1 -- 172.21.15.134:0/1375818619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7fad0c02e0c0 con 0x7fad0c1057e0 2024-07-26T11:03:20.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.578+0000 7fad03400640 1 -- 172.21.15.134:0/1375818619 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v24)=0 v24) ==== 119+0+0 (secure 0 0 0) 0x7facfc0ab240 con 0x7fad0c1057e0 2024-07-26T11:03:20.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.582+0000 7fad11c00640 1 -- 172.21.15.134:0/1375818619 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7facf00b3b90 msgr2=0x7facf00b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:20.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.582+0000 7fad11c00640 1 --2- 172.21.15.134:0/1375818619 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7facf00b3b90 0x7facf00b60a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7facf402a9e0 tx=0x7facf4031400 comp rx=0 tx=0).stop 2024-07-26T11:03:20.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.582+0000 7fad11c00640 1 -- 172.21.15.134:0/1375818619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad0c1057e0 msgr2=0x7fad0c19c1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:20.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.582+0000 7fad11c00640 1 --2- 172.21.15.134:0/1375818619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad0c1057e0 0x7fad0c19c1c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7facfc00e9f0 tx=0x7facfc00eeb0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.583+0000 7fad11c00640 1 -- 172.21.15.134:0/1375818619 shutdown_connections 2024-07-26T11:03:20.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.583+0000 7fad11c00640 1 --2- 172.21.15.134:0/1375818619 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7facf00b3b90 0x7facf00b60a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.583+0000 7fad11c00640 1 --2- 172.21.15.134:0/1375818619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fad0c10b120 0x7fad0c19c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.583+0000 7fad11c00640 1 --2- 172.21.15.134:0/1375818619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fad0c10ba70 0x7fad0c1a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.583+0000 7fad11c00640 1 --2- 172.21.15.134:0/1375818619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad0c1057e0 0x7fad0c19c1c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.583+0000 7fad11c00640 1 -- 172.21.15.134:0/1375818619 >> 172.21.15.134:0/1375818619 conn(0x7fad0c1011f0 msgr2=0x7fad0c109720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:20.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.583+0000 7fad11c00640 1 -- 172.21.15.134:0/1375818619 shutdown_connections 2024-07-26T11:03:20.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.583+0000 7fad11c00640 1 -- 172.21.15.134:0/1375818619 wait complete. 2024-07-26T11:03:20.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.642+0000 7fbed7000640 1 -- 172.21.15.134:0/3160661265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbed0102490 msgr2=0x7fbed01028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:20.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.642+0000 7fbed7000640 1 --2- 172.21.15.134:0/3160661265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbed0102490 0x7fbed01028a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbec0009900 tx=0x7fbec002a410 comp rx=0 tx=0).stop 2024-07-26T11:03:20.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.642+0000 7fbed7000640 1 -- 172.21.15.134:0/3160661265 shutdown_connections 2024-07-26T11:03:20.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.642+0000 7fbed7000640 1 --2- 172.21.15.134:0/3160661265 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed0108720 0x7fbed010ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.642+0000 7fbed7000640 1 --2- 172.21.15.134:0/3160661265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed0107dd0 0x7fbed01081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.642+0000 7fbed7000640 1 --2- 172.21.15.134:0/3160661265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbed0102490 0x7fbed01028a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.642+0000 7fbed7000640 1 -- 172.21.15.134:0/3160661265 >> 172.21.15.134:0/3160661265 conn(0x7fbed0079160 msgr2=0x7fbed0079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:20.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.642+0000 7fbed7000640 1 -- 172.21.15.134:0/3160661265 shutdown_connections 2024-07-26T11:03:20.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.642+0000 7fbed7000640 1 -- 172.21.15.134:0/3160661265 wait complete. 2024-07-26T11:03:20.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.643+0000 7fbed7000640 1 Processor -- start 2024-07-26T11:03:20.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.644+0000 7fbed7000640 1 -- start start 2024-07-26T11:03:20.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.644+0000 7fbed7000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed0102490 0x7fbed019c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:20.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.644+0000 7fbed7000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbed0107dd0 0x7fbed019c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:20.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.644+0000 7fbed7000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed0108720 0x7fbed01a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:20.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.644+0000 7fbed7000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbed0112380 con 0x7fbed0108720 2024-07-26T11:03:20.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.644+0000 7fbed7000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbed0112200 con 0x7fbed0102490 2024-07-26T11:03:20.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.644+0000 7fbed7000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbed0112500 con 0x7fbed0107dd0 2024-07-26T11:03:20.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.644+0000 7fbed5000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed0108720 0x7fbed01a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:20.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.644+0000 7fbecfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed0102490 0x7fbed019c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:20.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.644+0000 7fbed5000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed0108720 0x7fbed01a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48732/0 (socket says 172.21.15.134:48732) 2024-07-26T11:03:20.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.644+0000 7fbed5000640 1 -- 172.21.15.134:0/1956812420 learned_addr learned my addr 172.21.15.134:0/1956812420 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:20.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.645+0000 7fbec7e00640 1 --2- 172.21.15.134:0/1956812420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbed0107dd0 0x7fbed019c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:20.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.645+0000 7fbed5000640 1 -- 172.21.15.134:0/1956812420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbed0107dd0 msgr2=0x7fbed019c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:20.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.645+0000 7fbed5000640 1 --2- 172.21.15.134:0/1956812420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbed0107dd0 0x7fbed019c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.645+0000 7fbed5000640 1 -- 172.21.15.134:0/1956812420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed0102490 msgr2=0x7fbed019c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:20.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.645+0000 7fbed5000640 1 --2- 172.21.15.134:0/1956812420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed0102490 0x7fbed019c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.645+0000 7fbed5000640 1 -- 172.21.15.134:0/1956812420 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed01a2060 con 0x7fbed0108720 2024-07-26T11:03:20.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.645+0000 7fbed5000640 1 --2- 172.21.15.134:0/1956812420 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed0108720 0x7fbed01a1930 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbec800eb10 tx=0x7fbec800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:20.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.646+0000 7fbecd600640 1 -- 172.21.15.134:0/1956812420 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8004b20 con 0x7fbed0108720 2024-07-26T11:03:20.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.646+0000 7fbed7000640 1 -- 172.21.15.134:0/1956812420 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed01a2350 con 0x7fbed0108720 2024-07-26T11:03:20.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.646+0000 7fbed7000640 1 -- 172.21.15.134:0/1956812420 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed01a8ab0 con 0x7fbed0108720 2024-07-26T11:03:20.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.646+0000 7fbecd600640 1 -- 172.21.15.134:0/1956812420 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbec8004cc0 con 0x7fbed0108720 2024-07-26T11:03:20.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.646+0000 7fbecd600640 1 -- 172.21.15.134:0/1956812420 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8018630 con 0x7fbed0108720 2024-07-26T11:03:20.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.647+0000 7fbed7000640 1 -- 172.21.15.134:0/1956812420 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeac0052c0 con 0x7fbed0108720 2024-07-26T11:03:20.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.650+0000 7fbecd600640 1 -- 172.21.15.134:0/1956812420 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fbec8005000 con 0x7fbed0108720 2024-07-26T11:03:20.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.651+0000 7fbecd600640 1 --2- 172.21.15.134:0/1956812420 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbeb40b3c10 0x7fbeb40b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:20.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.651+0000 7fbecd600640 1 -- 172.21.15.134:0/1956812420 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fbec80e2b60 con 0x7fbed0108720 2024-07-26T11:03:20.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.652+0000 7fbecfe00640 1 --2- 172.21.15.134:0/1956812420 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbeb40b3c10 0x7fbeb40b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:20.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.652+0000 7fbecfe00640 1 --2- 172.21.15.134:0/1956812420 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbeb40b3c10 0x7fbeb40b6120 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbec0031440 tx=0x7fbec0009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:20.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.655+0000 7fbecd600640 1 -- 172.21.15.134:0/1956812420 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbec80abd10 con 0x7fbed0108720 2024-07-26T11:03:20.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.858+0000 7fbed7000640 1 -- 172.21.15.134:0/1956812420 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7fbeac0055b0 con 0x7fbed0108720 2024-07-26T11:03:20.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.864+0000 7fbecd600640 1 -- 172.21.15.134:0/1956812420 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v25)=0 v25) ==== 129+0+0 (secure 0 0 0) 0x7fbec80ab610 con 0x7fbed0108720 2024-07-26T11:03:20.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.867+0000 7fbed7000640 1 -- 172.21.15.134:0/1956812420 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbeb40b3c10 msgr2=0x7fbeb40b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:20.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.867+0000 7fbed7000640 1 --2- 172.21.15.134:0/1956812420 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbeb40b3c10 0x7fbeb40b6120 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbec0031440 tx=0x7fbec0009560 comp rx=0 tx=0).stop 2024-07-26T11:03:20.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.867+0000 7fbed7000640 1 -- 172.21.15.134:0/1956812420 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed0108720 msgr2=0x7fbed01a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:20.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.867+0000 7fbed7000640 1 --2- 172.21.15.134:0/1956812420 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed0108720 0x7fbed01a1930 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbec800eb10 tx=0x7fbec800c2d0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.867+0000 7fbed7000640 1 -- 172.21.15.134:0/1956812420 shutdown_connections 2024-07-26T11:03:20.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.867+0000 7fbed7000640 1 --2- 172.21.15.134:0/1956812420 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbeb40b3c10 0x7fbeb40b6120 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.867+0000 7fbed7000640 1 --2- 172.21.15.134:0/1956812420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbed0107dd0 0x7fbed019c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.867+0000 7fbed7000640 1 --2- 172.21.15.134:0/1956812420 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbed0108720 0x7fbed01a1930 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.867+0000 7fbed7000640 1 --2- 172.21.15.134:0/1956812420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed0102490 0x7fbed019c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:20.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.867+0000 7fbed7000640 1 -- 172.21.15.134:0/1956812420 >> 172.21.15.134:0/1956812420 conn(0x7fbed0079160 msgr2=0x7fbed006a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:20.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.868+0000 7fbed7000640 1 -- 172.21.15.134:0/1956812420 shutdown_connections 2024-07-26T11:03:20.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:20.868+0000 7fbed7000640 1 -- 172.21.15.134:0/1956812420 wait complete. 2024-07-26T11:03:21.071 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:20 smithi134 ceph-mon[31966]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:21.072 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:20 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' 2024-07-26T11:03:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:20 smithi189 ceph-mon[36790]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:20 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' 2024-07-26T11:03:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:20 smithi193 ceph-mon[37473]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:20 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' 2024-07-26T11:03:21.964 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:21 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1956812420' entity='client.admin' 2024-07-26T11:03:21.964 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:21.964 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:21.964 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:21.964 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:03:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:21 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1956812420' entity='client.admin' 2024-07-26T11:03:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:03:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:21 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1956812420' entity='client.admin' 2024-07-26T11:03:22.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:22.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:22.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:22.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:03:22.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.622+0000 7fab5de00640 1 -- 172.21.15.134:0/4089162574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fab58076db0 msgr2=0x7fab580751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:22.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.622+0000 7fab5de00640 1 --2- 172.21.15.134:0/4089162574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fab58076db0 0x7fab580751e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fab48009900 tx=0x7fab4802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:22.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.622+0000 7fab5de00640 1 -- 172.21.15.134:0/4089162574 shutdown_connections 2024-07-26T11:03:22.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.622+0000 7fab5de00640 1 --2- 172.21.15.134:0/4089162574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab58075800 0x7fab58075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:22.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.622+0000 7fab5de00640 1 --2- 172.21.15.134:0/4089162574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab58113e70 0x7fab58116340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:22.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.622+0000 7fab5de00640 1 --2- 172.21.15.134:0/4089162574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fab58076db0 0x7fab580751e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:22.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.622+0000 7fab5de00640 1 -- 172.21.15.134:0/4089162574 >> 172.21.15.134:0/4089162574 conn(0x7fab580ff010 msgr2=0x7fab58101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:22.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.622+0000 7fab5de00640 1 -- 172.21.15.134:0/4089162574 shutdown_connections 2024-07-26T11:03:22.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.623+0000 7fab5de00640 1 -- 172.21.15.134:0/4089162574 wait complete. 2024-07-26T11:03:22.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.625+0000 7fab5de00640 1 Processor -- start 2024-07-26T11:03:22.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.625+0000 7fab5de00640 1 -- start start 2024-07-26T11:03:22.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.626+0000 7fab5de00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fab58075800 0x7fab581a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:22.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.626+0000 7fab5de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab58076db0 0x7fab581a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:22.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.626+0000 7fab5de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab58113e70 0x7fab581a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:22.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.626+0000 7fab5de00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fab58119b00 con 0x7fab58075800 2024-07-26T11:03:22.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.626+0000 7fab5de00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fab58119980 con 0x7fab58113e70 2024-07-26T11:03:22.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.626+0000 7fab5de00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fab58119c80 con 0x7fab58076db0 2024-07-26T11:03:22.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.626+0000 7fab56a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fab58075800 0x7fab581a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:22.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.626+0000 7fab56000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab58076db0 0x7fab581a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:22.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.626+0000 7fab56000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab58076db0 0x7fab581a0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:48222/0 (socket says 172.21.15.134:48222) 2024-07-26T11:03:22.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.626+0000 7fab56000640 1 -- 172.21.15.134:0/2031313696 learned_addr learned my addr 172.21.15.134:0/2031313696 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:22.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.626+0000 7fab57400640 1 --2- 172.21.15.134:0/2031313696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab58113e70 0x7fab581a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:22.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.627+0000 7fab57400640 1 -- 172.21.15.134:0/2031313696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab58076db0 msgr2=0x7fab581a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:22.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.627+0000 7fab57400640 1 --2- 172.21.15.134:0/2031313696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab58076db0 0x7fab581a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:22.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.627+0000 7fab57400640 1 -- 172.21.15.134:0/2031313696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fab58075800 msgr2=0x7fab581a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:22.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.627+0000 7fab57400640 1 --2- 172.21.15.134:0/2031313696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fab58075800 0x7fab581a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:22.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.627+0000 7fab57400640 1 -- 172.21.15.134:0/2031313696 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab581a6440 con 0x7fab58113e70 2024-07-26T11:03:22.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.627+0000 7fab56a00640 1 --2- 172.21.15.134:0/2031313696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fab58075800 0x7fab581a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:22.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.627+0000 7fab57400640 1 --2- 172.21.15.134:0/2031313696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab58113e70 0x7fab581a5d10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fab50009fd0 tx=0x7fab5000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:22.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.628+0000 7fab5ca00640 1 -- 172.21.15.134:0/2031313696 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5000ce20 con 0x7fab58113e70 2024-07-26T11:03:22.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.628+0000 7fab5ca00640 1 -- 172.21.15.134:0/2031313696 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab50005020 con 0x7fab58113e70 2024-07-26T11:03:22.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.628+0000 7fab5de00640 1 -- 172.21.15.134:0/2031313696 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab581a6730 con 0x7fab58113e70 2024-07-26T11:03:22.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.628+0000 7fab5ca00640 1 -- 172.21.15.134:0/2031313696 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab50010690 con 0x7fab58113e70 2024-07-26T11:03:22.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.628+0000 7fab5de00640 1 -- 172.21.15.134:0/2031313696 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab581ace90 con 0x7fab58113e70 2024-07-26T11:03:22.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.629+0000 7fab5de00640 1 -- 172.21.15.134:0/2031313696 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab58075c10 con 0x7fab58113e70 2024-07-26T11:03:22.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.633+0000 7fab5ca00640 1 -- 172.21.15.134:0/2031313696 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fab50004b20 con 0x7fab58113e70 2024-07-26T11:03:22.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.635+0000 7fab5ca00640 1 --2- 172.21.15.134:0/2031313696 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fab3c0b3c90 0x7fab3c0b61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:22.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.635+0000 7fab5ca00640 1 -- 172.21.15.134:0/2031313696 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fab500e26e0 con 0x7fab58113e70 2024-07-26T11:03:22.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.635+0000 7fab56a00640 1 --2- 172.21.15.134:0/2031313696 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fab3c0b3c90 0x7fab3c0b61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:22.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.636+0000 7fab56a00640 1 --2- 172.21.15.134:0/2031313696 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fab3c0b3c90 0x7fab3c0b61a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fab48005ab0 tx=0x7fab48005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:22.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.639+0000 7fab5ca00640 1 -- 172.21.15.134:0/2031313696 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fab500ab890 con 0x7fab58113e70 2024-07-26T11:03:22.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.852+0000 7fab5de00640 1 -- 172.21.15.134:0/2031313696 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7fab58112610 con 0x7fab58113e70 2024-07-26T11:03:22.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.862+0000 7fab5ca00640 1 -- 172.21.15.134:0/2031313696 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v26)=0 v26) ==== 120+0+0 (secure 0 0 0) 0x7fab500ab190 con 0x7fab58113e70 2024-07-26T11:03:22.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.866+0000 7fab5de00640 1 -- 172.21.15.134:0/2031313696 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fab3c0b3c90 msgr2=0x7fab3c0b61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:22.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.866+0000 7fab5de00640 1 --2- 172.21.15.134:0/2031313696 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fab3c0b3c90 0x7fab3c0b61a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fab48005ab0 tx=0x7fab48005a20 comp rx=0 tx=0).stop 2024-07-26T11:03:22.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.866+0000 7fab5de00640 1 -- 172.21.15.134:0/2031313696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab58113e70 msgr2=0x7fab581a5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:22.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.866+0000 7fab5de00640 1 --2- 172.21.15.134:0/2031313696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab58113e70 0x7fab581a5d10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fab50009fd0 tx=0x7fab5000ef10 comp rx=0 tx=0).stop 2024-07-26T11:03:22.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.866+0000 7fab5de00640 1 -- 172.21.15.134:0/2031313696 shutdown_connections 2024-07-26T11:03:22.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.866+0000 7fab5de00640 1 --2- 172.21.15.134:0/2031313696 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fab3c0b3c90 0x7fab3c0b61a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:22.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.866+0000 7fab5de00640 1 --2- 172.21.15.134:0/2031313696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab58076db0 0x7fab581a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:22.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.866+0000 7fab5de00640 1 --2- 172.21.15.134:0/2031313696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab58113e70 0x7fab581a5d10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:22.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.866+0000 7fab5de00640 1 --2- 172.21.15.134:0/2031313696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fab58075800 0x7fab581a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:22.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.866+0000 7fab5de00640 1 -- 172.21.15.134:0/2031313696 >> 172.21.15.134:0/2031313696 conn(0x7fab580ff010 msgr2=0x7fab5810fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:22.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.866+0000 7fab5de00640 1 -- 172.21.15.134:0/2031313696 shutdown_connections 2024-07-26T11:03:22.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:22.866+0000 7fab5de00640 1 -- 172.21.15.134:0/2031313696 wait complete. 2024-07-26T11:03:23.141 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:22 smithi134 ceph-mon[31966]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:23.141 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:22 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' 2024-07-26T11:03:23.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:22 smithi189 ceph-mon[36790]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:23.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:22 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' 2024-07-26T11:03:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:22 smithi193 ceph-mon[37473]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:22 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' 2024-07-26T11:03:23.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.864+0000 7fe6dbe00640 1 -- 172.21.15.134:0/3522447593 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6dc10fc80 msgr2=0x7fe6dc1164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:23.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.864+0000 7fe6dbe00640 1 --2- 172.21.15.134:0/3522447593 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6dc10fc80 0x7fe6dc1164b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4009ca0 tx=0x7fe6d402a8b0 comp rx=0 tx=0).stop 2024-07-26T11:03:23.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.864+0000 7fe6dbe00640 1 -- 172.21.15.134:0/3522447593 shutdown_connections 2024-07-26T11:03:23.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.864+0000 7fe6dbe00640 1 --2- 172.21.15.134:0/3522447593 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6dc10fc80 0x7fe6dc1164b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:23.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.864+0000 7fe6dbe00640 1 --2- 172.21.15.134:0/3522447593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6dc10f330 0x7fe6dc10f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:23.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.864+0000 7fe6dbe00640 1 --2- 172.21.15.134:0/3522447593 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6dc1099f0 0x7fe6dc109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:23.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.864+0000 7fe6dbe00640 1 -- 172.21.15.134:0/3522447593 >> 172.21.15.134:0/3522447593 conn(0x7fe6dc078ff0 msgr2=0x7fe6dc107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:23.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.864+0000 7fe6dbe00640 1 -- 172.21.15.134:0/3522447593 shutdown_connections 2024-07-26T11:03:23.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.864+0000 7fe6dbe00640 1 -- 172.21.15.134:0/3522447593 wait complete. 2024-07-26T11:03:23.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.866+0000 7fe6dbe00640 1 Processor -- start 2024-07-26T11:03:23.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.866+0000 7fe6dbe00640 1 -- start start 2024-07-26T11:03:23.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.867+0000 7fe6dbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6dc1099f0 0x7fe6dc1a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:23.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.867+0000 7fe6dbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6dc10f330 0x7fe6dc1a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:23.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.867+0000 7fe6dbe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6dc10fc80 0x7fe6dc1a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:23.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.867+0000 7fe6dbe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe6dc119990 con 0x7fe6dc10fc80 2024-07-26T11:03:23.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.867+0000 7fe6dbe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe6dc119810 con 0x7fe6dc1099f0 2024-07-26T11:03:23.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.867+0000 7fe6dbe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe6dc119b10 con 0x7fe6dc10f330 2024-07-26T11:03:23.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.867+0000 7fe6daa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6dc1099f0 0x7fe6dc1a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:23.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.867+0000 7fe6daa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6dc1099f0 0x7fe6dc1a0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:33350/0 (socket says 172.21.15.134:33350) 2024-07-26T11:03:23.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.867+0000 7fe6daa00640 1 -- 172.21.15.134:0/813985845 learned_addr learned my addr 172.21.15.134:0/813985845 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:23.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.867+0000 7fe6db400640 1 --2- 172.21.15.134:0/813985845 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6dc10fc80 0x7fe6dc1a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:23.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.867+0000 7fe6d3e00640 1 --2- 172.21.15.134:0/813985845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6dc10f330 0x7fe6dc1a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:23.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.867+0000 7fe6daa00640 1 -- 172.21.15.134:0/813985845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6dc10f330 msgr2=0x7fe6dc1a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:23.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.868+0000 7fe6daa00640 1 --2- 172.21.15.134:0/813985845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6dc10f330 0x7fe6dc1a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:23.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.868+0000 7fe6daa00640 1 -- 172.21.15.134:0/813985845 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6dc10fc80 msgr2=0x7fe6dc1a5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:23.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.868+0000 7fe6daa00640 1 --2- 172.21.15.134:0/813985845 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6dc10fc80 0x7fe6dc1a5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:23.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.868+0000 7fe6daa00640 1 -- 172.21.15.134:0/813985845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6dc1a61f0 con 0x7fe6dc1099f0 2024-07-26T11:03:23.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.868+0000 7fe6daa00640 1 --2- 172.21.15.134:0/813985845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6dc1099f0 0x7fe6dc1a0280 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc00d950 tx=0x7fe6cc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:23.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.868+0000 7fe6d3400640 1 -- 172.21.15.134:0/813985845 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc017030 con 0x7fe6dc1099f0 2024-07-26T11:03:23.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.868+0000 7fe6d3400640 1 -- 172.21.15.134:0/813985845 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6cc004020 con 0x7fe6dc1099f0 2024-07-26T11:03:23.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.868+0000 7fe6dbe00640 1 -- 172.21.15.134:0/813985845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6dc1a64e0 con 0x7fe6dc1099f0 2024-07-26T11:03:23.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.868+0000 7fe6d3400640 1 -- 172.21.15.134:0/813985845 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc003c50 con 0x7fe6dc1099f0 2024-07-26T11:03:23.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.868+0000 7fe6dbe00640 1 -- 172.21.15.134:0/813985845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6dc1acc40 con 0x7fe6dc1099f0 2024-07-26T11:03:23.872 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.870+0000 7fe6dbe00640 1 -- 172.21.15.134:0/813985845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6dc109e00 con 0x7fe6dc1099f0 2024-07-26T11:03:23.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.872+0000 7fe6d3400640 1 -- 172.21.15.134:0/813985845 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fe6cc01a070 con 0x7fe6dc1099f0 2024-07-26T11:03:23.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.874+0000 7fe6d3400640 1 --2- 172.21.15.134:0/813985845 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6c00b3c10 0x7fe6c00b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:23.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.874+0000 7fe6d3400640 1 -- 172.21.15.134:0/813985845 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fe6cc0e19b0 con 0x7fe6dc1099f0 2024-07-26T11:03:23.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.874+0000 7fe6d3e00640 1 --2- 172.21.15.134:0/813985845 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6c00b3c10 0x7fe6c00b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:23.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.875+0000 7fe6d3e00640 1 --2- 172.21.15.134:0/813985845 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6c00b3c10 0x7fe6c00b6120 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40095f0 tx=0x7fe6c4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:23.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.878+0000 7fe6d3400640 1 -- 172.21.15.134:0/813985845 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe6cc021070 con 0x7fe6dc1099f0 2024-07-26T11:03:24.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.139+0000 7f3fe6000640 1 -- 172.21.15.134:0/1860418812 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fe01035b0 msgr2=0x7f3fe01039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:24.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.139+0000 7f3fe6000640 1 --2- 172.21.15.134:0/1860418812 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fe01035b0 0x7f3fe01039c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009900 tx=0x7f3fd002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.139+0000 7f3fe6000640 1 -- 172.21.15.134:0/1860418812 shutdown_connections 2024-07-26T11:03:24.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.139+0000 7f3fe6000640 1 --2- 172.21.15.134:0/1860418812 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe0109840 0x7f3fe0110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.139+0000 7f3fe6000640 1 --2- 172.21.15.134:0/1860418812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe0108ef0 0x7f3fe0109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.139+0000 7f3fe6000640 1 --2- 172.21.15.134:0/1860418812 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fe01035b0 0x7f3fe01039c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.139+0000 7f3fe6000640 1 -- 172.21.15.134:0/1860418812 >> 172.21.15.134:0/1860418812 conn(0x7f3fe00fefc0 msgr2=0x7f3fe0101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:24.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.139+0000 7f3fe6000640 1 -- 172.21.15.134:0/1860418812 shutdown_connections 2024-07-26T11:03:24.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.139+0000 7f3fe6000640 1 -- 172.21.15.134:0/1860418812 wait complete. 2024-07-26T11:03:24.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.141+0000 7f3fe6000640 1 Processor -- start 2024-07-26T11:03:24.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.141+0000 7f3fe6000640 1 -- start start 2024-07-26T11:03:24.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.141+0000 7f3fe6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe01035b0 0x7f3fe01a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:24.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.142+0000 7f3fe6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe0108ef0 0x7f3fe01a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:24.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.142+0000 7f3fe6000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fe0109840 0x7f3fe01a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:24.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.142+0000 7f3fe6000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe0113560 con 0x7f3fe0109840 2024-07-26T11:03:24.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.142+0000 7f3fe6000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe01133e0 con 0x7f3fe0108ef0 2024-07-26T11:03:24.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.142+0000 7f3fe6000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3fe01136e0 con 0x7f3fe01035b0 2024-07-26T11:03:24.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.142+0000 7f3fdf400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe01035b0 0x7f3fe01a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:24.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.142+0000 7f3fdf400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe01035b0 0x7f3fe01a1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:48242/0 (socket says 172.21.15.134:48242) 2024-07-26T11:03:24.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.142+0000 7f3fdf400640 1 -- 172.21.15.134:0/3274657267 learned_addr learned my addr 172.21.15.134:0/3274657267 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:24.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.142+0000 7f3fdea00640 1 --2- 172.21.15.134:0/3274657267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe0108ef0 0x7f3fe01a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:24.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.142+0000 7f3fdfe00640 1 --2- 172.21.15.134:0/3274657267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fe0109840 0x7f3fe01a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:24.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.142+0000 7f3fdf400640 1 -- 172.21.15.134:0/3274657267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe0108ef0 msgr2=0x7f3fe01a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:24.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.142+0000 7f3fdf400640 1 --2- 172.21.15.134:0/3274657267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe0108ef0 0x7f3fe01a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.142+0000 7f3fdf400640 1 -- 172.21.15.134:0/3274657267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fe0109840 msgr2=0x7f3fe01a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:24.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.142+0000 7f3fdf400640 1 --2- 172.21.15.134:0/3274657267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fe0109840 0x7f3fe01a1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.142+0000 7f3fdf400640 1 -- 172.21.15.134:0/3274657267 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fe01ac5a0 con 0x7f3fe01035b0 2024-07-26T11:03:24.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.142+0000 7f3fdea00640 1 --2- 172.21.15.134:0/3274657267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe0108ef0 0x7f3fe01a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:24.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.143+0000 7f3fdf400640 1 --2- 172.21.15.134:0/3274657267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe01035b0 0x7f3fe01a1240 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0005ab0 tx=0x7f3fd0003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:24.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.143+0000 7f3fd7e00640 1 -- 172.21.15.134:0/3274657267 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd003f070 con 0x7f3fe01035b0 2024-07-26T11:03:24.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.143+0000 7f3fd7e00640 1 -- 172.21.15.134:0/3274657267 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fd0031b60 con 0x7f3fe01035b0 2024-07-26T11:03:24.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.143+0000 7f3fe6000640 1 -- 172.21.15.134:0/3274657267 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fe01ac830 con 0x7f3fe01035b0 2024-07-26T11:03:24.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.143+0000 7f3fd7e00640 1 -- 172.21.15.134:0/3274657267 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0035030 con 0x7f3fe01035b0 2024-07-26T11:03:24.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.143+0000 7f3fe6000640 1 -- 172.21.15.134:0/3274657267 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fe01acd60 con 0x7f3fe01035b0 2024-07-26T11:03:24.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.145+0000 7f3fe6000640 1 -- 172.21.15.134:0/3274657267 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fe01039c0 con 0x7f3fe01035b0 2024-07-26T11:03:24.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.149+0000 7f3fd7e00640 1 -- 172.21.15.134:0/3274657267 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f3fd0042070 con 0x7f3fe01035b0 2024-07-26T11:03:24.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.151+0000 7f3fd7e00640 1 --2- 172.21.15.134:0/3274657267 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3fc40b3c10 0x7f3fc40b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:24.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.152+0000 7f3fd7e00640 1 -- 172.21.15.134:0/3274657267 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f3fd0102740 con 0x7f3fe01035b0 2024-07-26T11:03:24.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.152+0000 7f3fdea00640 1 --2- 172.21.15.134:0/3274657267 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3fc40b3c10 0x7f3fc40b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:24.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.152+0000 7f3fdea00640 1 --2- 172.21.15.134:0/3274657267 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3fc40b3c10 0x7f3fc40b6120 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3fe01097c0 tx=0x7f3fcc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:24.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.155+0000 7f3fd7e00640 1 -- 172.21.15.134:0/3274657267 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3fd00cb2a0 con 0x7f3fe01035b0 2024-07-26T11:03:24.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.357+0000 7f3fe6000640 1 -- 172.21.15.134:0/3274657267 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7f3fe01ad160 con 0x7f3fe01035b0 2024-07-26T11:03:24.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.366+0000 7f3fd7e00640 1 -- 172.21.15.134:0/3274657267 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v27)=0 v27) ==== 115+0+0 (secure 0 0 0) 0x7f3fd00cf7f0 con 0x7f3fe01035b0 2024-07-26T11:03:24.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.369+0000 7f3fe6000640 1 -- 172.21.15.134:0/3274657267 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3fc40b3c10 msgr2=0x7f3fc40b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:24.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.369+0000 7f3fe6000640 1 --2- 172.21.15.134:0/3274657267 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3fc40b3c10 0x7f3fc40b6120 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3fe01097c0 tx=0x7f3fcc005f90 comp rx=0 tx=0).stop 2024-07-26T11:03:24.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.369+0000 7f3fe6000640 1 -- 172.21.15.134:0/3274657267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe01035b0 msgr2=0x7f3fe01a1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:24.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.369+0000 7f3fe6000640 1 --2- 172.21.15.134:0/3274657267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe01035b0 0x7f3fe01a1240 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0005ab0 tx=0x7f3fd0003c10 comp rx=0 tx=0).stop 2024-07-26T11:03:24.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.369+0000 7f3fe6000640 1 -- 172.21.15.134:0/3274657267 shutdown_connections 2024-07-26T11:03:24.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.369+0000 7f3fe6000640 1 --2- 172.21.15.134:0/3274657267 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3fc40b3c10 0x7f3fc40b6120 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.369+0000 7f3fe6000640 1 --2- 172.21.15.134:0/3274657267 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3fe0109840 0x7f3fe01a1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.369+0000 7f3fe6000640 1 --2- 172.21.15.134:0/3274657267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe0108ef0 0x7f3fe01a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.369+0000 7f3fe6000640 1 --2- 172.21.15.134:0/3274657267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3fe01035b0 0x7f3fe01a1240 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.369+0000 7f3fe6000640 1 -- 172.21.15.134:0/3274657267 >> 172.21.15.134:0/3274657267 conn(0x7f3fe00fefc0 msgr2=0x7f3fe01076c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:24.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.369+0000 7f3fe6000640 1 -- 172.21.15.134:0/3274657267 shutdown_connections 2024-07-26T11:03:24.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:23.370+0000 7f3fe6000640 1 -- 172.21.15.134:0/3274657267 wait complete. 2024-07-26T11:03:24.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.078+0000 7fe6dbe00640 1 -- 172.21.15.134:0/813985845 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7fe6dc1a13b0 con 0x7fe6dc1099f0 2024-07-26T11:03:24.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.086+0000 7fe6d3400640 1 -- 172.21.15.134:0/813985845 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v28)=0 v28) ==== 125+0+0 (secure 0 0 0) 0x7fe6cc0aab60 con 0x7fe6dc1099f0 2024-07-26T11:03:24.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.089+0000 7fe6dbe00640 1 -- 172.21.15.134:0/813985845 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6c00b3c10 msgr2=0x7fe6c00b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:24.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.089+0000 7fe6dbe00640 1 --2- 172.21.15.134:0/813985845 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6c00b3c10 0x7fe6c00b6120 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40095f0 tx=0x7fe6c4005f90 comp rx=0 tx=0).stop 2024-07-26T11:03:24.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.089+0000 7fe6dbe00640 1 -- 172.21.15.134:0/813985845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6dc1099f0 msgr2=0x7fe6dc1a0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:24.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.089+0000 7fe6dbe00640 1 --2- 172.21.15.134:0/813985845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6dc1099f0 0x7fe6dc1a0280 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc00d950 tx=0x7fe6cc00de10 comp rx=0 tx=0).stop 2024-07-26T11:03:24.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.089+0000 7fe6dbe00640 1 -- 172.21.15.134:0/813985845 shutdown_connections 2024-07-26T11:03:24.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.089+0000 7fe6dbe00640 1 --2- 172.21.15.134:0/813985845 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6c00b3c10 0x7fe6c00b6120 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.089+0000 7fe6dbe00640 1 --2- 172.21.15.134:0/813985845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6dc10f330 0x7fe6dc1a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.089+0000 7fe6dbe00640 1 --2- 172.21.15.134:0/813985845 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6dc10fc80 0x7fe6dc1a5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.089+0000 7fe6dbe00640 1 --2- 172.21.15.134:0/813985845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6dc1099f0 0x7fe6dc1a0280 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.089+0000 7fe6dbe00640 1 -- 172.21.15.134:0/813985845 >> 172.21.15.134:0/813985845 conn(0x7fe6dc078ff0 msgr2=0x7fe6dc079650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:24.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.089+0000 7fe6dbe00640 1 -- 172.21.15.134:0/813985845 shutdown_connections 2024-07-26T11:03:24.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.089+0000 7fe6dbe00640 1 -- 172.21.15.134:0/813985845 wait complete. 2024-07-26T11:03:24.575 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:24 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' 2024-07-26T11:03:24.575 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:24 smithi134 ceph-mon[31966]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:24.575 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:24 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' 2024-07-26T11:03:24.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.660+0000 7f58d9200640 1 -- 172.21.15.134:0/3845986142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58d410b860 msgr2=0x7f58d4112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:24.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.660+0000 7f58d9200640 1 --2- 172.21.15.134:0/3845986142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58d410b860 0x7f58d4112090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009ca0 tx=0x7f58cc02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.660+0000 7f58d9200640 1 -- 172.21.15.134:0/3845986142 shutdown_connections 2024-07-26T11:03:24.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.660+0000 7f58d9200640 1 --2- 172.21.15.134:0/3845986142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58d410b860 0x7f58d4112090 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.660+0000 7f58d9200640 1 --2- 172.21.15.134:0/3845986142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58d410af10 0x7f58d410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.660+0000 7f58d9200640 1 --2- 172.21.15.134:0/3845986142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58d41055d0 0x7f58d41059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.660+0000 7f58d9200640 1 -- 172.21.15.134:0/3845986142 >> 172.21.15.134:0/3845986142 conn(0x7f58d4101040 msgr2=0x7f58d4103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:24.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.660+0000 7f58d9200640 1 -- 172.21.15.134:0/3845986142 shutdown_connections 2024-07-26T11:03:24.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.660+0000 7f58d9200640 1 -- 172.21.15.134:0/3845986142 wait complete. 2024-07-26T11:03:24.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.661+0000 7f58d9200640 1 Processor -- start 2024-07-26T11:03:24.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.662+0000 7f58d9200640 1 -- start start 2024-07-26T11:03:24.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.662+0000 7f58d9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58d41055d0 0x7f58d419bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:24.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.662+0000 7f58d9200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58d410af10 0x7f58d419c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:24.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.662+0000 7f58d9200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58d410b860 0x7f58d41a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:24.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.662+0000 7f58d9200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f58d4115560 con 0x7f58d410b860 2024-07-26T11:03:24.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.662+0000 7f58d9200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f58d41153e0 con 0x7f58d41055d0 2024-07-26T11:03:24.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.662+0000 7f58d9200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f58d41156e0 con 0x7f58d410af10 2024-07-26T11:03:24.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.662+0000 7f58d3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58d41055d0 0x7f58d419bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:24.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.662+0000 7f58d3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58d410b860 0x7f58d41a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:24.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.662+0000 7f58d3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58d41055d0 0x7f58d419bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:33366/0 (socket says 172.21.15.134:33366) 2024-07-26T11:03:24.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.662+0000 7f58d3400640 1 -- 172.21.15.134:0/3542194782 learned_addr learned my addr 172.21.15.134:0/3542194782 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:24.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.663+0000 7f58caa00640 1 --2- 172.21.15.134:0/3542194782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58d410af10 0x7f58d419c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:24.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.663+0000 7f58d3400640 1 -- 172.21.15.134:0/3542194782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58d410af10 msgr2=0x7f58d419c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:24.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.663+0000 7f58d3400640 1 --2- 172.21.15.134:0/3542194782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58d410af10 0x7f58d419c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.663+0000 7f58d3400640 1 -- 172.21.15.134:0/3542194782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58d410b860 msgr2=0x7f58d41a17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:24.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.663+0000 7f58d3400640 1 --2- 172.21.15.134:0/3542194782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58d410b860 0x7f58d41a17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.663+0000 7f58d3400640 1 -- 172.21.15.134:0/3542194782 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d41a1f20 con 0x7f58d41055d0 2024-07-26T11:03:24.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.663+0000 7f58d3400640 1 --2- 172.21.15.134:0/3542194782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58d41055d0 0x7f58d419bfb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f58c400e9f0 tx=0x7f58c400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:24.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.663+0000 7f58d0c00640 1 -- 172.21.15.134:0/3542194782 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c400cd80 con 0x7f58d41055d0 2024-07-26T11:03:24.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.663+0000 7f58d9200640 1 -- 172.21.15.134:0/3542194782 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d41a2210 con 0x7f58d41055d0 2024-07-26T11:03:24.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.663+0000 7f58d0c00640 1 -- 172.21.15.134:0/3542194782 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58c40108e0 con 0x7f58d41055d0 2024-07-26T11:03:24.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.663+0000 7f58d0c00640 1 -- 172.21.15.134:0/3542194782 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c4018980 con 0x7f58d41055d0 2024-07-26T11:03:24.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.664+0000 7f58d9200640 1 -- 172.21.15.134:0/3542194782 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d41a8970 con 0x7f58d41055d0 2024-07-26T11:03:24.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.665+0000 7f58d9200640 1 -- 172.21.15.134:0/3542194782 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58d41059e0 con 0x7f58d41055d0 2024-07-26T11:03:24.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.667+0000 7f58d0c00640 1 -- 172.21.15.134:0/3542194782 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f58c4010430 con 0x7f58d41055d0 2024-07-26T11:03:24.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.668+0000 7f58d0c00640 1 --2- 172.21.15.134:0/3542194782 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f58b80b3b90 0x7f58b80b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:24.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.668+0000 7f58d0c00640 1 -- 172.21.15.134:0/3542194782 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f58c40e2630 con 0x7f58d41055d0 2024-07-26T11:03:24.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.668+0000 7f58caa00640 1 --2- 172.21.15.134:0/3542194782 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f58b80b3b90 0x7f58b80b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:24.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.669+0000 7f58caa00640 1 --2- 172.21.15.134:0/3542194782 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f58b80b3b90 0x7f58b80b60a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f58d4107880 tx=0x7f58bc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:24.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.670+0000 7f58d0c00640 1 -- 172.21.15.134:0/3542194782 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f58c40ab190 con 0x7f58d41055d0 2024-07-26T11:03:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:24 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' 2024-07-26T11:03:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:24 smithi189 ceph-mon[36790]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:24 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' 2024-07-26T11:03:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:24 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' 2024-07-26T11:03:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:24 smithi193 ceph-mon[37473]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:24 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' 2024-07-26T11:03:24.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.859+0000 7f58d9200640 1 -- 172.21.15.134:0/3542194782 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7f58d419d0e0 con 0x7f58d41055d0 2024-07-26T11:03:24.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.869+0000 7f58d0c00640 1 -- 172.21.15.134:0/3542194782 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v29)=0 v29) ==== 122+0+0 (secure 0 0 0) 0x7f58c40af6e0 con 0x7f58d41055d0 2024-07-26T11:03:24.873 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.871+0000 7f58d9200640 1 -- 172.21.15.134:0/3542194782 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f58b80b3b90 msgr2=0x7f58b80b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:24.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.871+0000 7f58d9200640 1 --2- 172.21.15.134:0/3542194782 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f58b80b3b90 0x7f58b80b60a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f58d4107880 tx=0x7f58bc005f90 comp rx=0 tx=0).stop 2024-07-26T11:03:24.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.871+0000 7f58d9200640 1 -- 172.21.15.134:0/3542194782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58d41055d0 msgr2=0x7f58d419bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:24.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.871+0000 7f58d9200640 1 --2- 172.21.15.134:0/3542194782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58d41055d0 0x7f58d419bfb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f58c400e9f0 tx=0x7f58c400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.872+0000 7f58d9200640 1 -- 172.21.15.134:0/3542194782 shutdown_connections 2024-07-26T11:03:24.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.872+0000 7f58d9200640 1 --2- 172.21.15.134:0/3542194782 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f58b80b3b90 0x7f58b80b60a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.872+0000 7f58d9200640 1 --2- 172.21.15.134:0/3542194782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f58d410af10 0x7f58d419c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.872+0000 7f58d9200640 1 --2- 172.21.15.134:0/3542194782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f58d410b860 0x7f58d41a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.872+0000 7f58d9200640 1 --2- 172.21.15.134:0/3542194782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58d41055d0 0x7f58d419bfb0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:24.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.872+0000 7f58d9200640 1 -- 172.21.15.134:0/3542194782 >> 172.21.15.134:0/3542194782 conn(0x7f58d4101040 msgr2=0x7f58d4103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:24.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.872+0000 7f58d9200640 1 -- 172.21.15.134:0/3542194782 shutdown_connections 2024-07-26T11:03:24.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:24.872+0000 7f58d9200640 1 -- 172.21.15.134:0/3542194782 wait complete. 2024-07-26T11:03:25.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.036+0000 7f10b1600640 1 -- 172.21.15.134:0/496717563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10ac10af10 msgr2=0x7f10ac10b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:25.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.036+0000 7f10b1600640 1 --2- 172.21.15.134:0/496717563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10ac10af10 0x7f10ac10b320 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1098009900 tx=0x7f109802a410 comp rx=0 tx=0).stop 2024-07-26T11:03:25.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.036+0000 7f10b1600640 1 -- 172.21.15.134:0/496717563 shutdown_connections 2024-07-26T11:03:25.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.036+0000 7f10b1600640 1 --2- 172.21.15.134:0/496717563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10ac10b860 0x7f10ac112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.036+0000 7f10b1600640 1 --2- 172.21.15.134:0/496717563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10ac10af10 0x7f10ac10b320 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.036+0000 7f10b1600640 1 --2- 172.21.15.134:0/496717563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10ac1055d0 0x7f10ac1059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.036+0000 7f10b1600640 1 -- 172.21.15.134:0/496717563 >> 172.21.15.134:0/496717563 conn(0x7f10ac101040 msgr2=0x7f10ac103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:25.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.037+0000 7f10b1600640 1 -- 172.21.15.134:0/496717563 shutdown_connections 2024-07-26T11:03:25.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.037+0000 7f10b1600640 1 -- 172.21.15.134:0/496717563 wait complete. 2024-07-26T11:03:25.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.038+0000 7f10b1600640 1 Processor -- start 2024-07-26T11:03:25.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.038+0000 7f10b1600640 1 -- start start 2024-07-26T11:03:25.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.038+0000 7f10b1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10ac1055d0 0x7f10ac19bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.038+0000 7f10b1600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10ac10af10 0x7f10ac19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.038+0000 7f10b1600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10ac10b860 0x7f10ac1a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.038+0000 7f10b1600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac1154f0 con 0x7f10ac10af10 2024-07-26T11:03:25.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.038+0000 7f10b1600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac115370 con 0x7f10ac1055d0 2024-07-26T11:03:25.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.038+0000 7f10b1600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac115670 con 0x7f10ac10b860 2024-07-26T11:03:25.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.038+0000 7f10ab400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10ac10af10 0x7f10ac19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.039+0000 7f10ab400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10ac10af10 0x7f10ac19c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38038/0 (socket says 172.21.15.134:38038) 2024-07-26T11:03:25.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.039+0000 7f10ab400640 1 -- 172.21.15.134:0/3595947397 learned_addr learned my addr 172.21.15.134:0/3595947397 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:25.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.039+0000 7f10abe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10ac1055d0 0x7f10ac19bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.039+0000 7f10b0c00640 1 --2- 172.21.15.134:0/3595947397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10ac10b860 0x7f10ac1a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.039+0000 7f10ab400640 1 -- 172.21.15.134:0/3595947397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10ac10b860 msgr2=0x7f10ac1a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:25.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.039+0000 7f10ab400640 1 --2- 172.21.15.134:0/3595947397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10ac10b860 0x7f10ac1a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.039+0000 7f10ab400640 1 -- 172.21.15.134:0/3595947397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10ac1055d0 msgr2=0x7f10ac19bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:25.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.039+0000 7f10ab400640 1 --2- 172.21.15.134:0/3595947397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10ac1055d0 0x7f10ac19bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.039+0000 7f10ab400640 1 -- 172.21.15.134:0/3595947397 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ac1a1ed0 con 0x7f10ac10af10 2024-07-26T11:03:25.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.039+0000 7f10b0c00640 1 --2- 172.21.15.134:0/3595947397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10ac10b860 0x7f10ac1a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:03:25.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.039+0000 7f10abe00640 1 --2- 172.21.15.134:0/3595947397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10ac1055d0 0x7f10ac19bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:25.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.039+0000 7f10ab400640 1 --2- 172.21.15.134:0/3595947397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10ac10af10 0x7f10ac19c500 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f109802a930 tx=0x7f1098031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:25.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.040+0000 7f10a8c00640 1 -- 172.21.15.134:0/3595947397 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109803f030 con 0x7f10ac10af10 2024-07-26T11:03:25.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.040+0000 7f10a8c00640 1 -- 172.21.15.134:0/3595947397 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1098004490 con 0x7f10ac10af10 2024-07-26T11:03:25.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.040+0000 7f10b1600640 1 -- 172.21.15.134:0/3595947397 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10ac1a2160 con 0x7f10ac10af10 2024-07-26T11:03:25.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.040+0000 7f10a8c00640 1 -- 172.21.15.134:0/3595947397 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1098035070 con 0x7f10ac10af10 2024-07-26T11:03:25.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.040+0000 7f10b1600640 1 -- 172.21.15.134:0/3595947397 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10ac1a8920 con 0x7f10ac10af10 2024-07-26T11:03:25.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.041+0000 7f10b1600640 1 -- 172.21.15.134:0/3595947397 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10ac1059e0 con 0x7f10ac10af10 2024-07-26T11:03:25.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.044+0000 7f10a8c00640 1 -- 172.21.15.134:0/3595947397 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f1098042030 con 0x7f10ac10af10 2024-07-26T11:03:25.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.045+0000 7f10a8c00640 1 --2- 172.21.15.134:0/3595947397 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f10900b3c10 0x7f10900b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:25.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.046+0000 7f10a8c00640 1 -- 172.21.15.134:0/3595947397 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f10981027f0 con 0x7f10ac10af10 2024-07-26T11:03:25.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.046+0000 7f10abe00640 1 --2- 172.21.15.134:0/3595947397 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f10900b3c10 0x7f10900b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:25.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.046+0000 7f10abe00640 1 --2- 172.21.15.134:0/3595947397 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f10900b3c10 0x7f10900b6120 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f109c0099a0 tx=0x7f109c009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:25.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.048+0000 7f10a8c00640 1 -- 172.21.15.134:0/3595947397 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f10980cb9a0 con 0x7f10ac10af10 2024-07-26T11:03:25.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.252+0000 7f10b1600640 1 -- 172.21.15.134:0/3595947397 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7f10ac10d6a0 con 0x7f10ac10af10 2024-07-26T11:03:25.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.258+0000 7f10a8c00640 1 -- 172.21.15.134:0/3595947397 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v21)=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f10ac10d6a0 con 0x7f10ac10af10 2024-07-26T11:03:25.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.261+0000 7f10b1600640 1 -- 172.21.15.134:0/3595947397 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f10900b3c10 msgr2=0x7f10900b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:25.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.261+0000 7f10b1600640 1 --2- 172.21.15.134:0/3595947397 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f10900b3c10 0x7f10900b6120 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f109c0099a0 tx=0x7f109c009500 comp rx=0 tx=0).stop 2024-07-26T11:03:25.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.261+0000 7f10b1600640 1 -- 172.21.15.134:0/3595947397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10ac10af10 msgr2=0x7f10ac19c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:25.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.261+0000 7f10b1600640 1 --2- 172.21.15.134:0/3595947397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10ac10af10 0x7f10ac19c500 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f109802a930 tx=0x7f1098031a20 comp rx=0 tx=0).stop 2024-07-26T11:03:25.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.261+0000 7f10b1600640 1 -- 172.21.15.134:0/3595947397 shutdown_connections 2024-07-26T11:03:25.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.261+0000 7f10b1600640 1 --2- 172.21.15.134:0/3595947397 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f10900b3c10 0x7f10900b6120 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.261+0000 7f10b1600640 1 --2- 172.21.15.134:0/3595947397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10ac10b860 0x7f10ac1a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.261+0000 7f10b1600640 1 --2- 172.21.15.134:0/3595947397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10ac10af10 0x7f10ac19c500 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.261+0000 7f10b1600640 1 --2- 172.21.15.134:0/3595947397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10ac1055d0 0x7f10ac19bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:25.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.262+0000 7f10b1600640 1 -- 172.21.15.134:0/3595947397 >> 172.21.15.134:0/3595947397 conn(0x7f10ac101040 msgr2=0x7f10ac103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:25.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.262+0000 7f10b1600640 1 -- 172.21.15.134:0/3595947397 shutdown_connections 2024-07-26T11:03:25.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:11.262+0000 7f10b1600640 1 -- 172.21.15.134:0/3595947397 wait complete. 2024-07-26T11:03:26.167 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:25 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' 2024-07-26T11:03:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:25 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' 2024-07-26T11:03:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:25 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' 2024-07-26T11:03:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:26 smithi134 ceph-mon[31966]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:26 smithi189 ceph-mon[36790]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:26 smithi193 ceph-mon[37473]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:27.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.929+0000 7f20c5c00640 1 -- 172.21.15.134:0/1857777087 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20c01055d0 msgr2=0x7f20c01059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:27.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.929+0000 7f20c5c00640 1 --2- 172.21.15.134:0/1857777087 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20c01055d0 0x7f20c01059e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f20b0009900 tx=0x7f20b002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:27.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.929+0000 7f20c5c00640 1 -- 172.21.15.134:0/1857777087 shutdown_connections 2024-07-26T11:03:27.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.929+0000 7f20c5c00640 1 --2- 172.21.15.134:0/1857777087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20c010af10 0x7f20c010b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:27.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.929+0000 7f20c5c00640 1 --2- 172.21.15.134:0/1857777087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20c010b860 0x7f20c0112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:27.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.929+0000 7f20c5c00640 1 --2- 172.21.15.134:0/1857777087 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20c01055d0 0x7f20c01059e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:27.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.929+0000 7f20c5c00640 1 -- 172.21.15.134:0/1857777087 >> 172.21.15.134:0/1857777087 conn(0x7f20c0101040 msgr2=0x7f20c0103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:27.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.929+0000 7f20c5c00640 1 -- 172.21.15.134:0/1857777087 shutdown_connections 2024-07-26T11:03:27.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.929+0000 7f20c5c00640 1 -- 172.21.15.134:0/1857777087 wait complete. 2024-07-26T11:03:27.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.930+0000 7f20c5c00640 1 Processor -- start 2024-07-26T11:03:27.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.930+0000 7f20c5c00640 1 -- start start 2024-07-26T11:03:27.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.931+0000 7f20c5c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20c01055d0 0x7f20c019bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:27.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.931+0000 7f20c5c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20c010af10 0x7f20c019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:27.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.931+0000 7f20c5c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20c010b860 0x7f20c01a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:27.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.931+0000 7f20c5c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f20c0115580 con 0x7f20c010af10 2024-07-26T11:03:27.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.931+0000 7f20c5c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f20c0115400 con 0x7f20c01055d0 2024-07-26T11:03:27.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.931+0000 7f20c5c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f20c0115700 con 0x7f20c010b860 2024-07-26T11:03:27.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.931+0000 7f20bf400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20c010af10 0x7f20c019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:27.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.931+0000 7f20bf400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20c010af10 0x7f20c019c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48818/0 (socket says 172.21.15.134:48818) 2024-07-26T11:03:27.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.931+0000 7f20bf400640 1 -- 172.21.15.134:0/1360602602 learned_addr learned my addr 172.21.15.134:0/1360602602 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:27.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.931+0000 7f20c5200640 1 --2- 172.21.15.134:0/1360602602 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20c010b860 0x7f20c01a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:27.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.931+0000 7f20bfe00640 1 --2- 172.21.15.134:0/1360602602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20c01055d0 0x7f20c019bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:27.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.931+0000 7f20bf400640 1 -- 172.21.15.134:0/1360602602 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20c010b860 msgr2=0x7f20c01a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:27.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.931+0000 7f20bf400640 1 --2- 172.21.15.134:0/1360602602 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20c010b860 0x7f20c01a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:27.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.931+0000 7f20bf400640 1 -- 172.21.15.134:0/1360602602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20c01055d0 msgr2=0x7f20c019bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:27.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.931+0000 7f20bf400640 1 --2- 172.21.15.134:0/1360602602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20c01055d0 0x7f20c019bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:27.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.931+0000 7f20bf400640 1 -- 172.21.15.134:0/1360602602 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c01a1ee0 con 0x7f20c010af10 2024-07-26T11:03:27.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.932+0000 7f20bf400640 1 --2- 172.21.15.134:0/1360602602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20c010af10 0x7f20c019c510 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f20ac00e9c0 tx=0x7f20ac00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:27.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.932+0000 7f20bcc00640 1 -- 172.21.15.134:0/1360602602 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ac00ccc0 con 0x7f20c010af10 2024-07-26T11:03:27.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.932+0000 7f20bcc00640 1 -- 172.21.15.134:0/1360602602 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20ac0054d0 con 0x7f20c010af10 2024-07-26T11:03:27.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.932+0000 7f20c5c00640 1 -- 172.21.15.134:0/1360602602 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20c01a21d0 con 0x7f20c010af10 2024-07-26T11:03:27.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.932+0000 7f20bcc00640 1 -- 172.21.15.134:0/1360602602 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ac010430 con 0x7f20c010af10 2024-07-26T11:03:27.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.932+0000 7f20c5c00640 1 -- 172.21.15.134:0/1360602602 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20c01a8930 con 0x7f20c010af10 2024-07-26T11:03:27.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.934+0000 7f20c5c00640 1 -- 172.21.15.134:0/1360602602 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20c01059e0 con 0x7f20c010af10 2024-07-26T11:03:27.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.936+0000 7f20bcc00640 1 -- 172.21.15.134:0/1360602602 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f20ac005020 con 0x7f20c010af10 2024-07-26T11:03:27.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.938+0000 7f20bcc00640 1 --2- 172.21.15.134:0/1360602602 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f20a40b3b90 0x7f20a40b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:27.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.938+0000 7f20bcc00640 1 -- 172.21.15.134:0/1360602602 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f20ac0e23d0 con 0x7f20c010af10 2024-07-26T11:03:27.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.938+0000 7f20bfe00640 1 --2- 172.21.15.134:0/1360602602 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f20a40b3b90 0x7f20a40b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:27.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.939+0000 7f20bfe00640 1 --2- 172.21.15.134:0/1360602602 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f20a40b3b90 0x7f20a40b60a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f20b0005ab0 tx=0x7f20b0005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:27.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:27.942+0000 7f20bcc00640 1 -- 172.21.15.134:0/1360602602 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f20ac0ab580 con 0x7f20c010af10 2024-07-26T11:03:27.952 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:03:28.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.146+0000 7f20c5c00640 1 -- 172.21.15.134:0/1360602602 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7f20c002e0c0 con 0x7f20c010af10 2024-07-26T11:03:28.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.152+0000 7f20bcc00640 1 -- 172.21.15.134:0/1360602602 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v30)=0 v30) ==== 121+0+0 (secure 0 0 0) 0x7f20ac0aae80 con 0x7f20c010af10 2024-07-26T11:03:28.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.157+0000 7f20c5c00640 1 -- 172.21.15.134:0/1360602602 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f20a40b3b90 msgr2=0x7f20a40b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.157+0000 7f20c5c00640 1 --2- 172.21.15.134:0/1360602602 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f20a40b3b90 0x7f20a40b60a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f20b0005ab0 tx=0x7f20b0005a20 comp rx=0 tx=0).stop 2024-07-26T11:03:28.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.157+0000 7f20c5c00640 1 -- 172.21.15.134:0/1360602602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20c010af10 msgr2=0x7f20c019c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.157+0000 7f20c5c00640 1 --2- 172.21.15.134:0/1360602602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20c010af10 0x7f20c019c510 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f20ac00e9c0 tx=0x7f20ac00ee80 comp rx=0 tx=0).stop 2024-07-26T11:03:28.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.158+0000 7f20c5c00640 1 -- 172.21.15.134:0/1360602602 shutdown_connections 2024-07-26T11:03:28.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.158+0000 7f20c5c00640 1 --2- 172.21.15.134:0/1360602602 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f20a40b3b90 0x7f20a40b60a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.158+0000 7f20c5c00640 1 --2- 172.21.15.134:0/1360602602 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f20c010b860 0x7f20c01a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.158+0000 7f20c5c00640 1 --2- 172.21.15.134:0/1360602602 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f20c010af10 0x7f20c019c510 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.158+0000 7f20c5c00640 1 --2- 172.21.15.134:0/1360602602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20c01055d0 0x7f20c019bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.158+0000 7f20c5c00640 1 -- 172.21.15.134:0/1360602602 >> 172.21.15.134:0/1360602602 conn(0x7f20c0101040 msgr2=0x7f20c0103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:28.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.158+0000 7f20c5c00640 1 -- 172.21.15.134:0/1360602602 shutdown_connections 2024-07-26T11:03:28.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.158+0000 7f20c5c00640 1 -- 172.21.15.134:0/1360602602 wait complete. 2024-07-26T11:03:28.234 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.232+0000 7f2c59400640 1 -- 172.21.15.134:0/1543145364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c54104080 msgr2=0x7f2c54104490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.234 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.232+0000 7f2c59400640 1 --2- 172.21.15.134:0/1543145364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c54104080 0x7f2c54104490 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009900 tx=0x7f2c4402a410 comp rx=0 tx=0).stop 2024-07-26T11:03:28.234 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.232+0000 7f2c59400640 1 -- 172.21.15.134:0/1543145364 shutdown_connections 2024-07-26T11:03:28.234 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.232+0000 7f2c59400640 1 --2- 172.21.15.134:0/1543145364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c5410d630 0x7f2c5410fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.234 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.232+0000 7f2c59400640 1 --2- 172.21.15.134:0/1543145364 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c54104ab0 0x7f2c5410d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.232+0000 7f2c59400640 1 --2- 172.21.15.134:0/1543145364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c54104080 0x7f2c54104490 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.232+0000 7f2c59400640 1 -- 172.21.15.134:0/1543145364 >> 172.21.15.134:0/1543145364 conn(0x7f2c540fdcf0 msgr2=0x7f2c54100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:28.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.232+0000 7f2c59400640 1 -- 172.21.15.134:0/1543145364 shutdown_connections 2024-07-26T11:03:28.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.232+0000 7f2c59400640 1 -- 172.21.15.134:0/1543145364 wait complete. 2024-07-26T11:03:28.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.233+0000 7f2c59400640 1 Processor -- start 2024-07-26T11:03:28.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.233+0000 7f2c59400640 1 -- start start 2024-07-26T11:03:28.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.234+0000 7f2c59400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c54104080 0x7f2c5419bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.234+0000 7f2c59400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c54104ab0 0x7f2c5419c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.234+0000 7f2c59400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c5410d630 0x7f2c541a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.234+0000 7f2c59400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2c54113310 con 0x7f2c54104ab0 2024-07-26T11:03:28.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.234+0000 7f2c59400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2c54113190 con 0x7f2c54104080 2024-07-26T11:03:28.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.234+0000 7f2c59400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2c54113490 con 0x7f2c5410d630 2024-07-26T11:03:28.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.234+0000 7f2c4be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c54104ab0 0x7f2c5419c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.234+0000 7f2c53e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c54104080 0x7f2c5419bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.234+0000 7f2c53e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c54104080 0x7f2c5419bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:33392/0 (socket says 172.21.15.134:33392) 2024-07-26T11:03:28.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.234+0000 7f2c53e00640 1 -- 172.21.15.134:0/3012787729 learned_addr learned my addr 172.21.15.134:0/3012787729 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:28.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.234+0000 7f2c4be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c54104ab0 0x7f2c5419c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48840/0 (socket says 172.21.15.134:48840) 2024-07-26T11:03:28.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.234+0000 7f2c58a00640 1 --2- 172.21.15.134:0/3012787729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c5410d630 0x7f2c541a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.235+0000 7f2c53e00640 1 -- 172.21.15.134:0/3012787729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c5410d630 msgr2=0x7f2c541a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.235+0000 7f2c53e00640 1 --2- 172.21.15.134:0/3012787729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c5410d630 0x7f2c541a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.235+0000 7f2c53e00640 1 -- 172.21.15.134:0/3012787729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c54104ab0 msgr2=0x7f2c5419c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.235+0000 7f2c53e00640 1 --2- 172.21.15.134:0/3012787729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c54104ab0 0x7f2c5419c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.235+0000 7f2c53e00640 1 -- 172.21.15.134:0/3012787729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c541a1e70 con 0x7f2c54104080 2024-07-26T11:03:28.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.235+0000 7f2c53e00640 1 --2- 172.21.15.134:0/3012787729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c54104080 0x7f2c5419bf00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2c44005ab0 tx=0x7f2c44003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:28.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.235+0000 7f2c51600640 1 -- 172.21.15.134:0/3012787729 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4403f070 con 0x7f2c54104080 2024-07-26T11:03:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.236+0000 7f2c51600640 1 -- 172.21.15.134:0/3012787729 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2c44031b60 con 0x7f2c54104080 2024-07-26T11:03:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.236+0000 7f2c59400640 1 -- 172.21.15.134:0/3012787729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c541a2100 con 0x7f2c54104080 2024-07-26T11:03:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.236+0000 7f2c51600640 1 -- 172.21.15.134:0/3012787729 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c44035030 con 0x7f2c54104080 2024-07-26T11:03:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.236+0000 7f2c59400640 1 -- 172.21.15.134:0/3012787729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c541a88c0 con 0x7f2c54104080 2024-07-26T11:03:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.237+0000 7f2c59400640 1 -- 172.21.15.134:0/3012787729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c54104490 con 0x7f2c54104080 2024-07-26T11:03:28.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.240+0000 7f2c51600640 1 -- 172.21.15.134:0/3012787729 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f2c44042070 con 0x7f2c54104080 2024-07-26T11:03:28.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.241+0000 7f2c51600640 1 --2- 172.21.15.134:0/3012787729 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2c380b3c10 0x7f2c380b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.241+0000 7f2c51600640 1 -- 172.21.15.134:0/3012787729 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f2c441026a0 con 0x7f2c54104080 2024-07-26T11:03:28.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.241+0000 7f2c4be00640 1 --2- 172.21.15.134:0/3012787729 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2c380b3c10 0x7f2c380b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.242+0000 7f2c4be00640 1 --2- 172.21.15.134:0/3012787729 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2c380b3c10 0x7f2c380b6120 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2c54101460 tx=0x7f2c3c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:28.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.246+0000 7f2c51600640 1 -- 172.21.15.134:0/3012787729 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2c440cb200 con 0x7f2c54104080 2024-07-26T11:03:28.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.452+0000 7f2c59400640 1 -- 172.21.15.134:0/3012787729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) -- 0x7f2c541a8af0 con 0x7f2c54104080 2024-07-26T11:03:28.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.461+0000 7f2c51600640 1 -- 172.21.15.134:0/3012787729 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v31)=0 v31) ==== 116+0+0 (secure 0 0 0) 0x7f2c440cf750 con 0x7f2c54104080 2024-07-26T11:03:28.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.461+0000 7ff01e400640 1 -- 172.21.15.134:0/3744245659 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff018077370 msgr2=0x7ff018077780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.462+0000 7ff01e400640 1 --2- 172.21.15.134:0/3744245659 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff018077370 0x7ff018077780 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff00c009900 tx=0x7ff00c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.462+0000 7ff01e400640 1 -- 172.21.15.134:0/3744245659 shutdown_connections 2024-07-26T11:03:28.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.462+0000 7ff01e400640 1 --2- 172.21.15.134:0/3744245659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff018077cc0 0x7ff018113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.462+0000 7ff01e400640 1 --2- 172.21.15.134:0/3744245659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff018113fb0 0x7ff0181163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.462+0000 7ff01e400640 1 --2- 172.21.15.134:0/3744245659 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff018077370 0x7ff018077780 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.462+0000 7ff01e400640 1 -- 172.21.15.134:0/3744245659 >> 172.21.15.134:0/3744245659 conn(0x7ff018101060 msgr2=0x7ff0181034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:28.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.462+0000 7ff01e400640 1 -- 172.21.15.134:0/3744245659 shutdown_connections 2024-07-26T11:03:28.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.462+0000 7ff01e400640 1 -- 172.21.15.134:0/3744245659 wait complete. 2024-07-26T11:03:28.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.463+0000 7f2c59400640 1 -- 172.21.15.134:0/3012787729 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2c380b3c10 msgr2=0x7f2c380b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.463+0000 7f2c59400640 1 --2- 172.21.15.134:0/3012787729 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2c380b3c10 0x7f2c380b6120 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2c54101460 tx=0x7f2c3c005f90 comp rx=0 tx=0).stop 2024-07-26T11:03:28.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.463+0000 7f2c59400640 1 -- 172.21.15.134:0/3012787729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c54104080 msgr2=0x7f2c5419bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.463+0000 7f2c59400640 1 --2- 172.21.15.134:0/3012787729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c54104080 0x7f2c5419bf00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2c44005ab0 tx=0x7f2c44003c10 comp rx=0 tx=0).stop 2024-07-26T11:03:28.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.463+0000 7f2c59400640 1 -- 172.21.15.134:0/3012787729 shutdown_connections 2024-07-26T11:03:28.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.463+0000 7ff01e400640 1 Processor -- start 2024-07-26T11:03:28.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.463+0000 7f2c59400640 1 --2- 172.21.15.134:0/3012787729 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2c380b3c10 0x7f2c380b6120 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.463+0000 7f2c59400640 1 --2- 172.21.15.134:0/3012787729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2c5410d630 0x7f2c541a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.463+0000 7f2c59400640 1 --2- 172.21.15.134:0/3012787729 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2c54104ab0 0x7f2c5419c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.463+0000 7f2c59400640 1 --2- 172.21.15.134:0/3012787729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c54104080 0x7f2c5419bf00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.463+0000 7f2c59400640 1 -- 172.21.15.134:0/3012787729 >> 172.21.15.134:0/3012787729 conn(0x7f2c540fdcf0 msgr2=0x7f2c54101e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:28.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.464+0000 7f2c59400640 1 -- 172.21.15.134:0/3012787729 shutdown_connections 2024-07-26T11:03:28.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.464+0000 7ff01e400640 1 -- start start 2024-07-26T11:03:28.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.464+0000 7f2c59400640 1 -- 172.21.15.134:0/3012787729 wait complete. 2024-07-26T11:03:28.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.464+0000 7ff01e400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff018077370 0x7ff0181a0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.464+0000 7ff01e400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff018077cc0 0x7ff0181a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.464+0000 7ff01e400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff018113fb0 0x7ff0181a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.464+0000 7ff01e400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff018119910 con 0x7ff018077370 2024-07-26T11:03:28.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.464+0000 7ff01e400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff018119790 con 0x7ff018077cc0 2024-07-26T11:03:28.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.464+0000 7ff01e400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff018119a90 con 0x7ff018113fb0 2024-07-26T11:03:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.464+0000 7ff01d000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff018077370 0x7ff0181a0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.465+0000 7ff01d000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff018077370 0x7ff0181a0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48870/0 (socket says 172.21.15.134:48870) 2024-07-26T11:03:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.465+0000 7ff01d000640 1 -- 172.21.15.134:0/2698983776 learned_addr learned my addr 172.21.15.134:0/2698983776 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.465+0000 7ff017e00640 1 --2- 172.21.15.134:0/2698983776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff018077cc0 0x7ff0181a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.465+0000 7ff01da00640 1 --2- 172.21.15.134:0/2698983776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff018113fb0 0x7ff0181a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.465+0000 7ff017e00640 1 -- 172.21.15.134:0/2698983776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff018113fb0 msgr2=0x7ff0181a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.465+0000 7ff017e00640 1 --2- 172.21.15.134:0/2698983776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff018113fb0 0x7ff0181a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.465+0000 7ff017e00640 1 -- 172.21.15.134:0/2698983776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff018077370 msgr2=0x7ff0181a0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.465+0000 7ff017e00640 1 --2- 172.21.15.134:0/2698983776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff018077370 0x7ff0181a0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.465+0000 7ff017e00640 1 -- 172.21.15.134:0/2698983776 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0181a62b0 con 0x7ff018077cc0 2024-07-26T11:03:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.465+0000 7ff01d000640 1 --2- 172.21.15.134:0/2698983776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff018077370 0x7ff0181a0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:28.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.465+0000 7ff017e00640 1 --2- 172.21.15.134:0/2698983776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff018077cc0 0x7ff0181a08e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff01806b360 tx=0x7ff00400ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:28.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.466+0000 7ff015600640 1 -- 172.21.15.134:0/2698983776 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff004009e60 con 0x7ff018077cc0 2024-07-26T11:03:28.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.466+0000 7ff015600640 1 -- 172.21.15.134:0/2698983776 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff00400ccc0 con 0x7ff018077cc0 2024-07-26T11:03:28.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.466+0000 7ff01e400640 1 -- 172.21.15.134:0/2698983776 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0181a65a0 con 0x7ff018077cc0 2024-07-26T11:03:28.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.466+0000 7ff01e400640 1 -- 172.21.15.134:0/2698983776 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0181acd00 con 0x7ff018077cc0 2024-07-26T11:03:28.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.466+0000 7ff015600640 1 -- 172.21.15.134:0/2698983776 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0040187a0 con 0x7ff018077cc0 2024-07-26T11:03:28.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.467+0000 7ff01e400640 1 -- 172.21.15.134:0/2698983776 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feff80052c0 con 0x7ff018077cc0 2024-07-26T11:03:28.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.469+0000 7ff015600640 1 -- 172.21.15.134:0/2698983776 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7ff004018940 con 0x7ff018077cc0 2024-07-26T11:03:28.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.473+0000 7ff015600640 1 --2- 172.21.15.134:0/2698983776 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff0000b3c10 0x7ff0000b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:28.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.473+0000 7ff015600640 1 -- 172.21.15.134:0/2698983776 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7ff0040e2080 con 0x7ff018077cc0 2024-07-26T11:03:28.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.473+0000 7ff01d000640 1 --2- 172.21.15.134:0/2698983776 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff0000b3c10 0x7ff0000b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:28.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.474+0000 7ff01d000640 1 --2- 172.21.15.134:0/2698983776 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff0000b3c10 0x7ff0000b6120 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff00c031440 tx=0x7ff00c0058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:28.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.475+0000 7ff015600640 1 -- 172.21.15.134:0/2698983776 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff0040ab230 con 0x7ff018077cc0 2024-07-26T11:03:28.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.674+0000 7ff01e400640 1 -- 172.21.15.134:0/2698983776 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7feff80055b0 con 0x7ff018077cc0 2024-07-26T11:03:28.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.685+0000 7ff015600640 1 -- 172.21.15.134:0/2698983776 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v32)=0 v32) ==== 120+0+0 (secure 0 0 0) 0x7ff0040aab30 con 0x7ff018077cc0 2024-07-26T11:03:28.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.688+0000 7ff01e400640 1 -- 172.21.15.134:0/2698983776 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff0000b3c10 msgr2=0x7ff0000b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.688+0000 7ff01e400640 1 --2- 172.21.15.134:0/2698983776 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff0000b3c10 0x7ff0000b6120 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff00c031440 tx=0x7ff00c0058d0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.688+0000 7ff01e400640 1 -- 172.21.15.134:0/2698983776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff018077cc0 msgr2=0x7ff0181a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:28.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.688+0000 7ff01e400640 1 --2- 172.21.15.134:0/2698983776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff018077cc0 0x7ff0181a08e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff01806b360 tx=0x7ff00400ed90 comp rx=0 tx=0).stop 2024-07-26T11:03:28.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.688+0000 7ff01e400640 1 -- 172.21.15.134:0/2698983776 shutdown_connections 2024-07-26T11:03:28.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.688+0000 7ff01e400640 1 --2- 172.21.15.134:0/2698983776 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff0000b3c10 0x7ff0000b6120 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.688+0000 7ff01e400640 1 --2- 172.21.15.134:0/2698983776 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff018113fb0 0x7ff0181a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.688+0000 7ff01e400640 1 --2- 172.21.15.134:0/2698983776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff018077cc0 0x7ff0181a08e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.688+0000 7ff01e400640 1 --2- 172.21.15.134:0/2698983776 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff018077370 0x7ff0181a0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:28.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.688+0000 7ff01e400640 1 -- 172.21.15.134:0/2698983776 >> 172.21.15.134:0/2698983776 conn(0x7ff018101060 msgr2=0x7ff0181160b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:28.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.689+0000 7ff01e400640 1 -- 172.21.15.134:0/2698983776 shutdown_connections 2024-07-26T11:03:28.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:28.689+0000 7ff01e400640 1 -- 172.21.15.134:0/2698983776 wait complete. 2024-07-26T11:03:28.932 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:28 smithi134 ceph-mon[31966]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:28.932 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:28 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1360602602' entity='client.admin' 2024-07-26T11:03:28.932 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:28 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' 2024-07-26T11:03:28.932 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:28 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' 2024-07-26T11:03:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:28 smithi189 ceph-mon[36790]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:28 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1360602602' entity='client.admin' 2024-07-26T11:03:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:28 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' 2024-07-26T11:03:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:28 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' 2024-07-26T11:03:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:28 smithi193 ceph-mon[37473]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:28 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1360602602' entity='client.admin' 2024-07-26T11:03:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:28 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' 2024-07-26T11:03:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:28 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' 2024-07-26T11:03:30.298 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_bits', '3'], check_status=True, hostname='smithi134') 2024-07-26T11:03:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:30 smithi134 ceph-mon[31966]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:31.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:30 smithi189 ceph-mon[36790]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:30 smithi193 ceph-mon[37473]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:32 smithi134 ceph-mon[31966]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:33.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:32 smithi189 ceph-mon[36790]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:32 smithi193 ceph-mon[37473]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:34.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.802+0000 7fd7fda00640 1 -- 172.21.15.134:0/1396912397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7f810af10 msgr2=0x7fd7f810b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:34.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.802+0000 7fd7fda00640 1 --2- 172.21.15.134:0/1396912397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7f810af10 0x7fd7f810b320 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0009900 tx=0x7fd7e002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:03:34.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.802+0000 7fd7fda00640 1 -- 172.21.15.134:0/1396912397 shutdown_connections 2024-07-26T11:03:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.802+0000 7fd7fda00640 1 --2- 172.21.15.134:0/1396912397 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7f810b860 0x7fd7f8112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.802+0000 7fd7fda00640 1 --2- 172.21.15.134:0/1396912397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7f810af10 0x7fd7f810b320 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.802+0000 7fd7fda00640 1 --2- 172.21.15.134:0/1396912397 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7f81055d0 0x7fd7f81059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.802+0000 7fd7fda00640 1 -- 172.21.15.134:0/1396912397 >> 172.21.15.134:0/1396912397 conn(0x7fd7f8101040 msgr2=0x7fd7f8103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.803+0000 7fd7fda00640 1 -- 172.21.15.134:0/1396912397 shutdown_connections 2024-07-26T11:03:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.803+0000 7fd7fda00640 1 -- 172.21.15.134:0/1396912397 wait complete. 2024-07-26T11:03:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.804+0000 7fd7fda00640 1 Processor -- start 2024-07-26T11:03:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.804+0000 7fd7fda00640 1 -- start start 2024-07-26T11:03:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.805+0000 7fd7fda00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7f81055d0 0x7fd7f819bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.805+0000 7fd7fda00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7f810af10 0x7fd7f819c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.805+0000 7fd7fda00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7f810b860 0x7fd7f81a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.805+0000 7fd7fda00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f8115520 con 0x7fd7f810af10 2024-07-26T11:03:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.805+0000 7fd7fda00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f81153a0 con 0x7fd7f810b860 2024-07-26T11:03:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.805+0000 7fd7fda00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f81156a0 con 0x7fd7f81055d0 2024-07-26T11:03:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.805+0000 7fd7efe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7f810af10 0x7fd7f819c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.805+0000 7fd7efe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7f810af10 0x7fd7f819c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:35272/0 (socket says 172.21.15.134:35272) 2024-07-26T11:03:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.805+0000 7fd7efe00640 1 -- 172.21.15.134:0/27859892 learned_addr learned my addr 172.21.15.134:0/27859892 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.805+0000 7fd7f7e00640 1 --2- 172.21.15.134:0/27859892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7f81055d0 0x7fd7f819bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.806+0000 7fd7fd000640 1 --2- 172.21.15.134:0/27859892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7f810b860 0x7fd7f81a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.806+0000 7fd7f7e00640 1 -- 172.21.15.134:0/27859892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7f810b860 msgr2=0x7fd7f81a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.806+0000 7fd7f7e00640 1 --2- 172.21.15.134:0/27859892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7f810b860 0x7fd7f81a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.806+0000 7fd7f7e00640 1 -- 172.21.15.134:0/27859892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7f810af10 msgr2=0x7fd7f819c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:34.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.806+0000 7fd7f7e00640 1 --2- 172.21.15.134:0/27859892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7f810af10 0x7fd7f819c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:34.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.806+0000 7fd7f7e00640 1 -- 172.21.15.134:0/27859892 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f81a1ea0 con 0x7fd7f81055d0 2024-07-26T11:03:34.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.806+0000 7fd7f7e00640 1 --2- 172.21.15.134:0/27859892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7f81055d0 0x7fd7f819bf30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd7e800e9f0 tx=0x7fd7e800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:34.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.806+0000 7fd7f5600640 1 -- 172.21.15.134:0/27859892 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e800ccc0 con 0x7fd7f81055d0 2024-07-26T11:03:34.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.807+0000 7fd7fda00640 1 -- 172.21.15.134:0/27859892 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f81a2190 con 0x7fd7f81055d0 2024-07-26T11:03:34.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.807+0000 7fd7f5600640 1 -- 172.21.15.134:0/27859892 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7e80108e0 con 0x7fd7f81055d0 2024-07-26T11:03:34.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.807+0000 7fd7f5600640 1 -- 172.21.15.134:0/27859892 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7e8018880 con 0x7fd7f81055d0 2024-07-26T11:03:34.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.807+0000 7fd7fda00640 1 -- 172.21.15.134:0/27859892 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f81a88f0 con 0x7fd7f81055d0 2024-07-26T11:03:34.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.808+0000 7fd7fda00640 1 -- 172.21.15.134:0/27859892 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7f81059e0 con 0x7fd7f81055d0 2024-07-26T11:03:34.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.810+0000 7fd7f5600640 1 -- 172.21.15.134:0/27859892 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fd7e8010430 con 0x7fd7f81055d0 2024-07-26T11:03:34.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.812+0000 7fd7f5600640 1 --2- 172.21.15.134:0/27859892 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd7dc0b3c10 0x7fd7dc0b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:34.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.812+0000 7fd7efe00640 1 --2- 172.21.15.134:0/27859892 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd7dc0b3c10 0x7fd7dc0b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:34.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.812+0000 7fd7f5600640 1 -- 172.21.15.134:0/27859892 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fd7e80e23f0 con 0x7fd7f81055d0 2024-07-26T11:03:34.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.812+0000 7fd7efe00640 1 --2- 172.21.15.134:0/27859892 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd7dc0b3c10 0x7fd7dc0b6120 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0006010 tx=0x7fd7e0005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:34.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:33.816+0000 7fd7f5600640 1 -- 172.21.15.134:0/27859892 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd7e80aaf50 con 0x7fd7f81055d0 2024-07-26T11:03:34.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:34.022+0000 7fd7fda00640 1 -- 172.21.15.134:0/27859892 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) -- 0x7fd7f802e0c0 con 0x7fd7f81055d0 2024-07-26T11:03:34.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:34.031+0000 7fd7f5600640 1 -- 172.21.15.134:0/27859892 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v33)=0 v33) ==== 116+0+0 (secure 0 0 0) 0x7fd7e80af4a0 con 0x7fd7f81055d0 2024-07-26T11:03:34.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:34.033+0000 7fd7fda00640 1 -- 172.21.15.134:0/27859892 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd7dc0b3c10 msgr2=0x7fd7dc0b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:34.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:34.033+0000 7fd7fda00640 1 --2- 172.21.15.134:0/27859892 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd7dc0b3c10 0x7fd7dc0b6120 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0006010 tx=0x7fd7e0005a20 comp rx=0 tx=0).stop 2024-07-26T11:03:34.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:34.033+0000 7fd7fda00640 1 -- 172.21.15.134:0/27859892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7f81055d0 msgr2=0x7fd7f819bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:34.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:34.033+0000 7fd7fda00640 1 --2- 172.21.15.134:0/27859892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7f81055d0 0x7fd7f819bf30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd7e800e9f0 tx=0x7fd7e800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:03:34.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:34.034+0000 7fd7fda00640 1 -- 172.21.15.134:0/27859892 shutdown_connections 2024-07-26T11:03:34.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:34.034+0000 7fd7fda00640 1 --2- 172.21.15.134:0/27859892 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd7dc0b3c10 0x7fd7dc0b6120 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:34.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:34.034+0000 7fd7fda00640 1 --2- 172.21.15.134:0/27859892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7f810b860 0x7fd7f81a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:34.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:34.034+0000 7fd7fda00640 1 --2- 172.21.15.134:0/27859892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd7f810af10 0x7fd7f819c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:34.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:34.034+0000 7fd7fda00640 1 --2- 172.21.15.134:0/27859892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd7f81055d0 0x7fd7f819bf30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:34.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:34.034+0000 7fd7fda00640 1 -- 172.21.15.134:0/27859892 >> 172.21.15.134:0/27859892 conn(0x7fd7f8101040 msgr2=0x7fd7f8103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:34.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:34.034+0000 7fd7fda00640 1 -- 172.21.15.134:0/27859892 shutdown_connections 2024-07-26T11:03:34.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:34.034+0000 7fd7fda00640 1 -- 172.21.15.134:0/27859892 wait complete. 2024-07-26T11:03:35.082 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_bal_split_size', '100'], check_status=True, hostname='smithi134') 2024-07-26T11:03:35.083 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi134') 2024-07-26T11:03:35.083 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi134') 2024-07-26T11:03:35.083 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi134') 2024-07-26T11:03:35.083 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi134') 2024-07-26T11:03:35.083 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi134') 2024-07-26T11:03:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:34 smithi134 ceph-mon[31966]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:34 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' 2024-07-26T11:03:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:34 smithi189 ceph-mon[36790]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:34 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' 2024-07-26T11:03:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:34 smithi193 ceph-mon[37473]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:34 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' 2024-07-26T11:03:35.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.394+0000 7f15b3200640 1 -- 172.21.15.134:0/2248868227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15b41055d0 msgr2=0x7f15b41059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:35.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.394+0000 7f15b3200640 1 --2- 172.21.15.134:0/2248868227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15b41055d0 0x7f15b41059e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009900 tx=0x7f15a402a410 comp rx=0 tx=0).stop 2024-07-26T11:03:35.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.394+0000 7f15b3200640 1 -- 172.21.15.134:0/2248868227 shutdown_connections 2024-07-26T11:03:35.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.394+0000 7f15b3200640 1 --2- 172.21.15.134:0/2248868227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15b410b860 0x7f15b4112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.394+0000 7f15b3200640 1 --2- 172.21.15.134:0/2248868227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15b410af10 0x7f15b410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.394+0000 7f15b3200640 1 --2- 172.21.15.134:0/2248868227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15b41055d0 0x7f15b41059e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.394+0000 7f15b3200640 1 -- 172.21.15.134:0/2248868227 >> 172.21.15.134:0/2248868227 conn(0x7f15b4101040 msgr2=0x7f15b4103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:35.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.394+0000 7f15b3200640 1 -- 172.21.15.134:0/2248868227 shutdown_connections 2024-07-26T11:03:35.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.394+0000 7f15b3200640 1 -- 172.21.15.134:0/2248868227 wait complete. 2024-07-26T11:03:35.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.396+0000 7f15b3200640 1 Processor -- start 2024-07-26T11:03:35.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.396+0000 7f15b3200640 1 -- start start 2024-07-26T11:03:35.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.396+0000 7f15b3200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15b41055d0 0x7f15b419bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:35.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.396+0000 7f15b3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15b410af10 0x7f15b419c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:35.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.397+0000 7f15b3200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15b410b860 0x7f15b41a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:35.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.397+0000 7f15b3200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f15b4115510 con 0x7f15b41055d0 2024-07-26T11:03:35.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.397+0000 7f15b3200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f15b4115390 con 0x7f15b410af10 2024-07-26T11:03:35.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.397+0000 7f15b3200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f15b4115690 con 0x7f15b410b860 2024-07-26T11:03:35.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.397+0000 7f15b1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15b410af10 0x7f15b419c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:35.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.397+0000 7f15b1e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15b41055d0 0x7f15b419bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:35.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.397+0000 7f15b1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15b410af10 0x7f15b419c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46710/0 (socket says 172.21.15.134:46710) 2024-07-26T11:03:35.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.397+0000 7f15b1400640 1 -- 172.21.15.134:0/3656898782 learned_addr learned my addr 172.21.15.134:0/3656898782 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:35.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.397+0000 7f15b2800640 1 --2- 172.21.15.134:0/3656898782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15b410b860 0x7f15b41a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:35.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.397+0000 7f15b1e00640 1 -- 172.21.15.134:0/3656898782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15b410b860 msgr2=0x7f15b41a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:35.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.397+0000 7f15b1e00640 1 --2- 172.21.15.134:0/3656898782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15b410b860 0x7f15b41a17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.397+0000 7f15b1e00640 1 -- 172.21.15.134:0/3656898782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15b410af10 msgr2=0x7f15b419c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:35.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.397+0000 7f15b1e00640 1 --2- 172.21.15.134:0/3656898782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15b410af10 0x7f15b419c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.397+0000 7f15b1e00640 1 -- 172.21.15.134:0/3656898782 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b41a1ef0 con 0x7f15b41055d0 2024-07-26T11:03:35.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.398+0000 7f15b1e00640 1 --2- 172.21.15.134:0/3656898782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15b41055d0 0x7f15b419bf80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f15a402a930 tx=0x7f15a4031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:35.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.398+0000 7f15aaa00640 1 -- 172.21.15.134:0/3656898782 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a403f030 con 0x7f15b41055d0 2024-07-26T11:03:35.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.398+0000 7f15aaa00640 1 -- 172.21.15.134:0/3656898782 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15a40043d0 con 0x7f15b41055d0 2024-07-26T11:03:35.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.398+0000 7f15b3200640 1 -- 172.21.15.134:0/3656898782 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b41a2180 con 0x7f15b41055d0 2024-07-26T11:03:35.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.398+0000 7f15aaa00640 1 -- 172.21.15.134:0/3656898782 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a4035070 con 0x7f15b41055d0 2024-07-26T11:03:35.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.398+0000 7f15b3200640 1 -- 172.21.15.134:0/3656898782 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b41a8940 con 0x7f15b41055d0 2024-07-26T11:03:35.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.400+0000 7f15b3200640 1 -- 172.21.15.134:0/3656898782 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b41059e0 con 0x7f15b41055d0 2024-07-26T11:03:35.405 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.404+0000 7f15aaa00640 1 -- 172.21.15.134:0/3656898782 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f15a4042030 con 0x7f15b41055d0 2024-07-26T11:03:35.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.405+0000 7f15aaa00640 1 --2- 172.21.15.134:0/3656898782 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f15980b3b90 0x7f15980b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:35.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.405+0000 7f15aaa00640 1 -- 172.21.15.134:0/3656898782 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f15a4102540 con 0x7f15b41055d0 2024-07-26T11:03:35.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.405+0000 7f15b1400640 1 --2- 172.21.15.134:0/3656898782 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f15980b3b90 0x7f15980b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:35.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.406+0000 7f15b1400640 1 --2- 172.21.15.134:0/3656898782 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f15980b3b90 0x7f15980b60a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f15b410aae0 tx=0x7f15a0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:35.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.408+0000 7f15aaa00640 1 -- 172.21.15.134:0/3656898782 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f15a40cb6f0 con 0x7f15b41055d0 2024-07-26T11:03:35.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.613+0000 7f15b3200640 1 -- 172.21.15.134:0/3656898782 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f15b419d0b0 con 0x7f15b41055d0 2024-07-26T11:03:35.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.623+0000 7f15aaa00640 1 -- 172.21.15.134:0/3656898782 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v34)=0 v34) ==== 133+0+0 (secure 0 0 0) 0x7f15b419d0b0 con 0x7f15b41055d0 2024-07-26T11:03:35.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.626+0000 7f15b3200640 1 -- 172.21.15.134:0/3656898782 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f15980b3b90 msgr2=0x7f15980b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:35.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.626+0000 7f15b3200640 1 --2- 172.21.15.134:0/3656898782 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f15980b3b90 0x7f15980b60a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f15b410aae0 tx=0x7f15a0005f90 comp rx=0 tx=0).stop 2024-07-26T11:03:35.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.626+0000 7f15b3200640 1 -- 172.21.15.134:0/3656898782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15b41055d0 msgr2=0x7f15b419bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:35.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.626+0000 7f15b3200640 1 --2- 172.21.15.134:0/3656898782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15b41055d0 0x7f15b419bf80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f15a402a930 tx=0x7f15a4031a20 comp rx=0 tx=0).stop 2024-07-26T11:03:35.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.626+0000 7f15b3200640 1 -- 172.21.15.134:0/3656898782 shutdown_connections 2024-07-26T11:03:35.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.626+0000 7f15b3200640 1 --2- 172.21.15.134:0/3656898782 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f15980b3b90 0x7f15980b60a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.626+0000 7f15b3200640 1 --2- 172.21.15.134:0/3656898782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f15b410b860 0x7f15b41a17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.626+0000 7f15b3200640 1 --2- 172.21.15.134:0/3656898782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f15b410af10 0x7f15b419c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.626+0000 7f15b3200640 1 --2- 172.21.15.134:0/3656898782 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f15b41055d0 0x7f15b419bf80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.626+0000 7f15b3200640 1 -- 172.21.15.134:0/3656898782 >> 172.21.15.134:0/3656898782 conn(0x7f15b4101040 msgr2=0x7f15b4103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:35.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.627+0000 7f15b3200640 1 -- 172.21.15.134:0/3656898782 shutdown_connections 2024-07-26T11:03:35.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.627+0000 7f15b3200640 1 -- 172.21.15.134:0/3656898782 wait complete. 2024-07-26T11:03:35.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.763+0000 7fa61f400640 1 -- 172.21.15.134:0/3073746246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa618077cc0 msgr2=0x7fa618113c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:35.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.764+0000 7fa61f400640 1 --2- 172.21.15.134:0/3073746246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa618077cc0 0x7fa618113c30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa604009900 tx=0x7fa60402a410 comp rx=0 tx=0).stop 2024-07-26T11:03:35.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.764+0000 7fa61f400640 1 -- 172.21.15.134:0/3073746246 shutdown_connections 2024-07-26T11:03:35.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.764+0000 7fa61f400640 1 --2- 172.21.15.134:0/3073746246 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa618114170 0x7fa618116590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.764+0000 7fa61f400640 1 --2- 172.21.15.134:0/3073746246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa618077cc0 0x7fa618113c30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.764+0000 7fa61f400640 1 --2- 172.21.15.134:0/3073746246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa618077370 0x7fa618077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.764+0000 7fa61f400640 1 -- 172.21.15.134:0/3073746246 >> 172.21.15.134:0/3073746246 conn(0x7fa618101200 msgr2=0x7fa618103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:35.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.764+0000 7fa61f400640 1 -- 172.21.15.134:0/3073746246 shutdown_connections 2024-07-26T11:03:35.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.765+0000 7fa61f400640 1 -- 172.21.15.134:0/3073746246 wait complete. 2024-07-26T11:03:35.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.766+0000 7fa61f400640 1 Processor -- start 2024-07-26T11:03:35.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.766+0000 7fa61f400640 1 -- start start 2024-07-26T11:03:35.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.767+0000 7fa61f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa618077370 0x7fa6181a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:35.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.767+0000 7fa61f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa618077cc0 0x7fa6181a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:35.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.767+0000 7fa61f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa618114170 0x7fa6181a5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:35.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.767+0000 7fa61f400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa618119ad0 con 0x7fa618077370 2024-07-26T11:03:35.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.767+0000 7fa61f400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa618119950 con 0x7fa618077cc0 2024-07-26T11:03:35.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.767+0000 7fa61f400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa618119c50 con 0x7fa618114170 2024-07-26T11:03:35.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.767+0000 7fa61ca00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa618077370 0x7fa6181a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:35.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.767+0000 7fa617e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa618077cc0 0x7fa6181a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:35.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.767+0000 7fa61ca00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa618077370 0x7fa6181a04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:35324/0 (socket says 172.21.15.134:35324) 2024-07-26T11:03:35.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.767+0000 7fa617e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa618077cc0 0x7fa6181a0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46716/0 (socket says 172.21.15.134:46716) 2024-07-26T11:03:35.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.767+0000 7fa61ca00640 1 -- 172.21.15.134:0/569870453 learned_addr learned my addr 172.21.15.134:0/569870453 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:35.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.767+0000 7fa61d400640 1 --2- 172.21.15.134:0/569870453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa618114170 0x7fa6181a5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:35.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.768+0000 7fa617e00640 1 -- 172.21.15.134:0/569870453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa618114170 msgr2=0x7fa6181a5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:35.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.768+0000 7fa617e00640 1 --2- 172.21.15.134:0/569870453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa618114170 0x7fa6181a5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.768+0000 7fa617e00640 1 -- 172.21.15.134:0/569870453 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa618077370 msgr2=0x7fa6181a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:35.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.768+0000 7fa617e00640 1 --2- 172.21.15.134:0/569870453 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa618077370 0x7fa6181a04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.768+0000 7fa617e00640 1 -- 172.21.15.134:0/569870453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6181a6450 con 0x7fa618077cc0 2024-07-26T11:03:35.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.768+0000 7fa61d400640 1 --2- 172.21.15.134:0/569870453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa618114170 0x7fa6181a5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:35.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.768+0000 7fa61ca00640 1 --2- 172.21.15.134:0/569870453 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa618077370 0x7fa6181a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:35.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.768+0000 7fa617e00640 1 --2- 172.21.15.134:0/569870453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa618077cc0 0x7fa6181a0a80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa60402a930 tx=0x7fa604031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:35.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.768+0000 7fa615600640 1 -- 172.21.15.134:0/569870453 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60403f070 con 0x7fa618077cc0 2024-07-26T11:03:35.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.768+0000 7fa615600640 1 -- 172.21.15.134:0/569870453 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa604004490 con 0x7fa618077cc0 2024-07-26T11:03:35.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.768+0000 7fa61f400640 1 -- 172.21.15.134:0/569870453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6181a66e0 con 0x7fa618077cc0 2024-07-26T11:03:35.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.768+0000 7fa615600640 1 -- 172.21.15.134:0/569870453 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa604035070 con 0x7fa618077cc0 2024-07-26T11:03:35.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.769+0000 7fa61f400640 1 -- 172.21.15.134:0/569870453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6181acea0 con 0x7fa618077cc0 2024-07-26T11:03:35.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.770+0000 7fa61f400640 1 -- 172.21.15.134:0/569870453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa618078880 con 0x7fa618077cc0 2024-07-26T11:03:35.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.772+0000 7fa615600640 1 -- 172.21.15.134:0/569870453 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fa60403b460 con 0x7fa618077cc0 2024-07-26T11:03:35.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.774+0000 7fa615600640 1 --2- 172.21.15.134:0/569870453 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6000b3c10 0x7fa6000b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:35.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.774+0000 7fa615600640 1 -- 172.21.15.134:0/569870453 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fa6041026c0 con 0x7fa618077cc0 2024-07-26T11:03:35.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.774+0000 7fa61ca00640 1 --2- 172.21.15.134:0/569870453 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6000b3c10 0x7fa6000b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:35.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.775+0000 7fa61ca00640 1 --2- 172.21.15.134:0/569870453 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6000b3c10 0x7fa6000b6120 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa60c011550 tx=0x7fa60c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:35.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.775+0000 7fa615600640 1 -- 172.21.15.134:0/569870453 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa6040cb220 con 0x7fa618077cc0 2024-07-26T11:03:35.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.975+0000 7fa61f400640 1 -- 172.21.15.134:0/569870453 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7fa618077780 con 0x7fa618077cc0 2024-07-26T11:03:35.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.986+0000 7fa615600640 1 -- 172.21.15.134:0/569870453 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v35)=0 v35) ==== 121+0+0 (secure 0 0 0) 0x7fa618077780 con 0x7fa618077cc0 2024-07-26T11:03:35.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.989+0000 7fa61f400640 1 -- 172.21.15.134:0/569870453 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6000b3c10 msgr2=0x7fa6000b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:35.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.989+0000 7fa61f400640 1 --2- 172.21.15.134:0/569870453 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6000b3c10 0x7fa6000b6120 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa60c011550 tx=0x7fa60c005f90 comp rx=0 tx=0).stop 2024-07-26T11:03:35.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.989+0000 7fa61f400640 1 -- 172.21.15.134:0/569870453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa618077cc0 msgr2=0x7fa6181a0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:35.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.989+0000 7fa61f400640 1 --2- 172.21.15.134:0/569870453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa618077cc0 0x7fa6181a0a80 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa60402a930 tx=0x7fa604031a20 comp rx=0 tx=0).stop 2024-07-26T11:03:35.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.989+0000 7fa61f400640 1 -- 172.21.15.134:0/569870453 shutdown_connections 2024-07-26T11:03:35.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.989+0000 7fa61f400640 1 --2- 172.21.15.134:0/569870453 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6000b3c10 0x7fa6000b6120 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.989+0000 7fa61f400640 1 --2- 172.21.15.134:0/569870453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa618114170 0x7fa6181a5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.989+0000 7fa61f400640 1 --2- 172.21.15.134:0/569870453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa618077cc0 0x7fa6181a0a80 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.989+0000 7fa61f400640 1 --2- 172.21.15.134:0/569870453 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa618077370 0x7fa6181a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:35.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.989+0000 7fa61f400640 1 -- 172.21.15.134:0/569870453 >> 172.21.15.134:0/569870453 conn(0x7fa618101200 msgr2=0x7fa6181162a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:35.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.990+0000 7fa61f400640 1 -- 172.21.15.134:0/569870453 shutdown_connections 2024-07-26T11:03:35.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:35.990+0000 7fa61f400640 1 -- 172.21.15.134:0/569870453 wait complete. 2024-07-26T11:03:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:36 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3656898782' entity='client.admin' 2024-07-26T11:03:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:03:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:36 smithi134 ceph-mon[31966]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:36 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' 2024-07-26T11:03:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:36 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3656898782' entity='client.admin' 2024-07-26T11:03:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:03:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:36 smithi189 ceph-mon[36790]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:36 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' 2024-07-26T11:03:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:36 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3656898782' entity='client.admin' 2024-07-26T11:03:37.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:03:37.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:37.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:03:37.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:03:37.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:36 smithi193 ceph-mon[37473]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:37.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:36 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' 2024-07-26T11:03:37.043 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi134') 2024-07-26T11:03:37.043 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi134') 2024-07-26T11:03:37.044 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'debug_cephsqlite', '20'], check_status=True, hostname='smithi134') 2024-07-26T11:03:37.044 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/allow_m_granularity', 'True'], check_status=True, hostname='smithi134') 2024-07-26T11:03:37.044 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/dump_on_update', 'True'], check_status=True, hostname='smithi134') 2024-07-26T11:03:37.044 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mgr', 'mgr/snap_schedule/log_level', 'debug'], check_status=True, hostname='smithi134') 2024-07-26T11:03:37.044 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi134') 2024-07-26T11:03:37.044 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'mon', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi134') 2024-07-26T11:03:37.044 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '105fbe5c-4b3e-11ef-bcb7-c7b262605968', '--', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi134') 2024-07-26T11:03:37.045 INFO:tasks.cephadm:Deploying OSDs... 2024-07-26T11:03:37.045 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T11:03:37.045 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T11:03:37.101 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T11:03:37.102 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 2024-07-26T11:03:37.152 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T11:03:37.152 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T11:03:37.152 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-07-26T11:03:37.153 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T11:03:37.153 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T11:03:37.153 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-07-26 11:01:27.024658968 +0000 2024-07-26T11:03:37.153 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-07-26 10:56:04.759002694 +0000 2024-07-26T11:03:37.153 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-07-26 10:56:04.759002694 +0000 2024-07-26T11:03:37.153 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-07-26 10:56:04.759002694 +0000 2024-07-26T11:03:37.153 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T11:03:37.233 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-07-26T11:03:37.233 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-07-26T11:03:37.233 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000362925 s, 1.4 MB/s 2024-07-26T11:03:37.236 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T11:03:37.287 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 2024-07-26T11:03:37.335 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T11:03:37.335 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T11:03:37.335 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-07-26T11:03:37.335 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T11:03:37.335 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T11:03:37.335 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-07-26 11:01:27.091657437 +0000 2024-07-26T11:03:37.335 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-07-26 10:56:05.336989536 +0000 2024-07-26T11:03:37.336 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-07-26 10:56:05.336989536 +0000 2024-07-26T11:03:37.336 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-07-26 10:56:05.336989536 +0000 2024-07-26T11:03:37.336 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T11:03:37.413 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-07-26T11:03:37.414 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-07-26T11:03:37.414 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000382372 s, 1.3 MB/s 2024-07-26T11:03:37.418 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T11:03:37.475 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 2024-07-26T11:03:37.523 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T11:03:37.524 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T11:03:37.524 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 6h/6d Inode: 672 Links: 1 2024-07-26T11:03:37.524 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T11:03:37.524 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T11:03:37.524 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-07-26 11:01:27.149656112 +0000 2024-07-26T11:03:37.524 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-07-26 10:56:05.907976537 +0000 2024-07-26T11:03:37.524 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-07-26 10:56:05.907976537 +0000 2024-07-26T11:03:37.524 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-07-26 10:56:05.907976537 +0000 2024-07-26T11:03:37.524 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T11:03:37.602 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-07-26T11:03:37.602 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-07-26T11:03:37.602 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.00031703 s, 1.6 MB/s 2024-07-26T11:03:37.605 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T11:03:37.660 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 2024-07-26T11:03:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T11:03:37.710 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T11:03:37.710 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-07-26T11:03:37.710 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T11:03:37.711 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T11:03:37.711 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-07-26 11:01:27.220654490 +0000 2024-07-26T11:03:37.711 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-07-26 10:56:06.483963424 +0000 2024-07-26T11:03:37.711 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-07-26 10:56:06.483963424 +0000 2024-07-26T11:03:37.711 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-07-26 10:56:06.483963424 +0000 2024-07-26T11:03:37.711 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T11:03:37.790 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-07-26T11:03:37.790 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-07-26T11:03:37.790 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.00050664 s, 1.0 MB/s 2024-07-26T11:03:37.795 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T11:03:37.852 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-07-26T11:03:37.852 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T11:03:37.871 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T11:03:37.871 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-07-26T11:03:37.930 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T11:03:37.930 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T11:03:37.930 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-07-26T11:03:37.930 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T11:03:37.930 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T11:03:37.931 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-07-26 11:02:45.333139354 +0000 2024-07-26T11:03:37.931 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-07-26 10:54:25.362231764 +0000 2024-07-26T11:03:37.931 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-07-26 10:54:25.362231764 +0000 2024-07-26T11:03:37.931 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-07-26 10:54:25.362231764 +0000 2024-07-26T11:03:37.931 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T11:03:37.999 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-07-26T11:03:38.000 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-07-26T11:03:38.000 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000259883 s, 2.0 MB/s 2024-07-26T11:03:38.001 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T11:03:38.059 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-07-26T11:03:38.117 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T11:03:38.117 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T11:03:38.117 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-07-26T11:03:38.117 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T11:03:38.117 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T11:03:38.117 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-07-26 11:02:45.333139354 +0000 2024-07-26T11:03:38.118 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-07-26 10:54:25.700224685 +0000 2024-07-26T11:03:38.118 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-07-26 10:54:25.700224685 +0000 2024-07-26T11:03:38.118 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-07-26 10:54:25.700224685 +0000 2024-07-26T11:03:38.118 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T11:03:38.184 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-07-26T11:03:38.184 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-07-26T11:03:38.184 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000228531 s, 2.2 MB/s 2024-07-26T11:03:38.185 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T11:03:38.243 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-07-26T11:03:38.300 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T11:03:38.300 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T11:03:38.300 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-07-26T11:03:38.300 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T11:03:38.300 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T11:03:38.300 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-07-26 11:02:45.333139354 +0000 2024-07-26T11:03:38.300 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-07-26 10:54:26.023217920 +0000 2024-07-26T11:03:38.300 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-07-26 10:54:26.023217920 +0000 2024-07-26T11:03:38.301 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-07-26 10:54:26.023217920 +0000 2024-07-26T11:03:38.301 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T11:03:38.364 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-07-26T11:03:38.364 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-07-26T11:03:38.364 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000212435 s, 2.4 MB/s 2024-07-26T11:03:38.365 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T11:03:38.422 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-07-26T11:03:38.478 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T11:03:38.478 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T11:03:38.479 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-07-26T11:03:38.479 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T11:03:38.479 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T11:03:38.479 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-07-26 11:02:45.333139354 +0000 2024-07-26T11:03:38.479 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-07-26 10:54:26.362210820 +0000 2024-07-26T11:03:38.479 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-07-26 10:54:26.362210820 +0000 2024-07-26T11:03:38.479 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-07-26 10:54:26.362210820 +0000 2024-07-26T11:03:38.479 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T11:03:38.542 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-07-26T11:03:38.542 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-07-26T11:03:38.542 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000195015 s, 2.6 MB/s 2024-07-26T11:03:38.543 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T11:03:38.602 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-07-26T11:03:38.602 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2024-07-26T11:03:38.621 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-07-26T11:03:38.621 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2024-07-26T11:03:38.679 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-07-26T11:03:38.679 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T11:03:38.679 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-07-26T11:03:38.679 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T11:03:38.679 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T11:03:38.679 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-07-26 11:02:43.001712112 +0000 2024-07-26T11:03:38.679 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-07-26 10:54:25.349765102 +0000 2024-07-26T11:03:38.679 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-07-26 10:54:25.349765102 +0000 2024-07-26T11:03:38.679 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-07-26 10:54:25.349765102 +0000 2024-07-26T11:03:38.680 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-07-26T11:03:38.747 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-07-26T11:03:38.747 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-07-26T11:03:38.747 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000298346 s, 1.7 MB/s 2024-07-26T11:03:38.748 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-07-26T11:03:38.807 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2024-07-26T11:03:38.863 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-07-26T11:03:38.863 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T11:03:38.863 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-07-26T11:03:38.863 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T11:03:38.863 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T11:03:38.864 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-07-26 11:02:43.002712096 +0000 2024-07-26T11:03:38.864 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-07-26 10:54:25.674759883 +0000 2024-07-26T11:03:38.864 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-07-26 10:54:25.674759883 +0000 2024-07-26T11:03:38.864 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-07-26 10:54:25.674759883 +0000 2024-07-26T11:03:38.864 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-07-26T11:03:38.928 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-07-26T11:03:38.928 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-07-26T11:03:38.928 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000233656 s, 2.2 MB/s 2024-07-26T11:03:38.929 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-07-26T11:03:38.986 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2024-07-26T11:03:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:38 smithi193 ceph-mon[37473]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:39.039 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-07-26T11:03:39.040 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T11:03:39.040 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-07-26T11:03:39.040 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T11:03:39.040 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T11:03:39.040 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-07-26 11:02:43.002712096 +0000 2024-07-26T11:03:39.040 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-07-26 10:54:26.017754374 +0000 2024-07-26T11:03:39.040 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-07-26 10:54:26.017754374 +0000 2024-07-26T11:03:39.040 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-07-26 10:54:26.017754374 +0000 2024-07-26T11:03:39.040 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-07-26T11:03:39.102 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-07-26T11:03:39.103 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-07-26T11:03:39.103 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000182999 s, 2.8 MB/s 2024-07-26T11:03:39.104 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-07-26T11:03:39.160 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2024-07-26T11:03:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:38 smithi134 ceph-mon[31966]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:38 smithi189 ceph-mon[36790]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:39.215 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-07-26T11:03:39.215 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-07-26T11:03:39.215 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-07-26T11:03:39.215 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T11:03:39.215 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-07-26T11:03:39.215 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-07-26 11:02:43.002712096 +0000 2024-07-26T11:03:39.215 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-07-26 10:54:26.368748738 +0000 2024-07-26T11:03:39.215 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-07-26 10:54:26.368748738 +0000 2024-07-26T11:03:39.216 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-07-26 10:54:26.368748738 +0000 2024-07-26T11:03:39.216 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-07-26T11:03:39.278 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-07-26T11:03:39.279 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-07-26T11:03:39.279 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000237509 s, 2.2 MB/s 2024-07-26T11:03:39.280 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-07-26T11:03:39.336 INFO:tasks.cephadm:Deploying osd.0 on smithi134 with /dev/vg_nvme/lv_4... 2024-07-26T11:03:39.336 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T11:03:39.804 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:03:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:40 smithi193 ceph-mon[37473]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:41.094 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:40 smithi134 ceph-mon[31966]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:41.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:40 smithi189 ceph-mon[36790]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:42.146 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T11:03:42.342 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T11:03:42.385 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-07-26T11:03:42.385 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-07-26T11:03:42.386 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0338908 s, 309 MB/s 2024-07-26T11:03:42.386 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-07-26T11:03:43.099 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:42 smithi134 ceph-mon[31966]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:43.154 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi134:vg_nvme/lv_4 2024-07-26T11:03:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:42 smithi189 ceph-mon[36790]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:42 smithi193 ceph-mon[37473]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:43.623 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:03:45.155 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:44 smithi134 ceph-mon[31966]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:44 smithi189 ceph-mon[36790]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:44 smithi193 ceph-mon[37473]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:45.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.657+0000 7f830e200640 1 -- 172.21.15.134:0/2339305561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f830810fc80 msgr2=0x7f83081164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:45.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.657+0000 7f830e200640 1 --2- 172.21.15.134:0/2339305561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f830810fc80 0x7f83081164b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8300009ca0 tx=0x7f830002a8c0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.657+0000 7f830e200640 1 -- 172.21.15.134:0/2339305561 shutdown_connections 2024-07-26T11:03:45.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.657+0000 7f830e200640 1 --2- 172.21.15.134:0/2339305561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f830810f330 0x7f830810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.657+0000 7f830e200640 1 --2- 172.21.15.134:0/2339305561 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f830810fc80 0x7f83081164b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.657+0000 7f830e200640 1 --2- 172.21.15.134:0/2339305561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83081099f0 0x7f8308109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.657+0000 7f830e200640 1 -- 172.21.15.134:0/2339305561 >> 172.21.15.134:0/2339305561 conn(0x7f8308078ff0 msgr2=0x7f8308107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:03:45.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.657+0000 7f830e200640 1 -- 172.21.15.134:0/2339305561 shutdown_connections 2024-07-26T11:03:45.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.657+0000 7f830e200640 1 -- 172.21.15.134:0/2339305561 wait complete. 2024-07-26T11:03:45.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.659+0000 7f830e200640 1 Processor -- start 2024-07-26T11:03:45.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.659+0000 7f830e200640 1 -- start start 2024-07-26T11:03:45.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.659+0000 7f830e200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83081099f0 0x7f83081a02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.659+0000 7f830e200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f830810f330 0x7f83081a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.659+0000 7f830e200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f830810fc80 0x7f83081a5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.659+0000 7f830e200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8308119940 con 0x7f830810fc80 2024-07-26T11:03:45.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.659+0000 7f830e200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f83081197c0 con 0x7f83081099f0 2024-07-26T11:03:45.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.659+0000 7f830e200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8308119ac0 con 0x7f830810f330 2024-07-26T11:03:45.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.660+0000 7f830ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83081099f0 0x7f83081a02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.660+0000 7f830ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83081099f0 0x7f83081a02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:57188/0 (socket says 172.21.15.134:57188) 2024-07-26T11:03:45.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.660+0000 7f830d800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f830810fc80 0x7f83081a5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.660+0000 7f830ce00640 1 -- 172.21.15.134:0/3909235028 learned_addr learned my addr 172.21.15.134:0/3909235028 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:03:45.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.660+0000 7f8307e00640 1 --2- 172.21.15.134:0/3909235028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f830810f330 0x7f83081a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.660+0000 7f830d800640 1 -- 172.21.15.134:0/3909235028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f830810f330 msgr2=0x7f83081a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:45.663 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.660+0000 7f830d800640 1 --2- 172.21.15.134:0/3909235028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f830810f330 0x7f83081a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.660+0000 7f830d800640 1 -- 172.21.15.134:0/3909235028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83081099f0 msgr2=0x7f83081a02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:03:45.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.660+0000 7f830d800640 1 --2- 172.21.15.134:0/3909235028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83081099f0 0x7f83081a02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:03:45.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.660+0000 7f830d800640 1 -- 172.21.15.134:0/3909235028 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83081a6240 con 0x7f830810fc80 2024-07-26T11:03:45.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.660+0000 7f830ce00640 1 --2- 172.21.15.134:0/3909235028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83081099f0 0x7f83081a02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:45.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.660+0000 7f8307e00640 1 --2- 172.21.15.134:0/3909235028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f830810f330 0x7f83081a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:03:45.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.661+0000 7f830d800640 1 --2- 172.21.15.134:0/3909235028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f830810fc80 0x7f83081a5b10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f830002ade0 tx=0x7f8300031660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:45.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.661+0000 7f8305600640 1 -- 172.21.15.134:0/3909235028 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8300003b70 con 0x7f830810fc80 2024-07-26T11:03:45.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.661+0000 7f830e200640 1 -- 172.21.15.134:0/3909235028 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83081a64d0 con 0x7f830810fc80 2024-07-26T11:03:45.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.661+0000 7f8305600640 1 -- 172.21.15.134:0/3909235028 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8300031910 con 0x7f830810fc80 2024-07-26T11:03:45.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.661+0000 7f8305600640 1 -- 172.21.15.134:0/3909235028 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830002f950 con 0x7f830810fc80 2024-07-26T11:03:45.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.661+0000 7f830e200640 1 -- 172.21.15.134:0/3909235028 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83081acc90 con 0x7f830810fc80 2024-07-26T11:03:45.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.662+0000 7f830e200640 1 -- 172.21.15.134:0/3909235028 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8308109e00 con 0x7f830810fc80 2024-07-26T11:03:45.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.665+0000 7f8305600640 1 -- 172.21.15.134:0/3909235028 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f8300004b40 con 0x7f830810fc80 2024-07-26T11:03:45.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.666+0000 7f8305600640 1 --2- 172.21.15.134:0/3909235028 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82f00b3b90 0x7f82f00b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:03:45.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.666+0000 7f8305600640 1 -- 172.21.15.134:0/3909235028 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8300103520 con 0x7f830810fc80 2024-07-26T11:03:45.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.666+0000 7f830ce00640 1 --2- 172.21.15.134:0/3909235028 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82f00b3b90 0x7f82f00b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:03:45.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.667+0000 7f830ce00640 1 --2- 172.21.15.134:0/3909235028 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82f00b3b90 0x7f82f00b60a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0095f0 tx=0x7f82fc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:03:45.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.668+0000 7f8305600640 1 -- 172.21.15.134:0/3909235028 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f83000cc6d0 con 0x7f830810fc80 2024-07-26T11:03:45.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:03:45.879+0000 7f830e200640 1 -- 172.21.15.134:0/3909235028 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f8308111ad0 con 0x7f82f00b3b90 2024-07-26T11:03:46.862 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:46 smithi134 ceph-mon[31966]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:46.862 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:46 smithi134 ceph-mon[31966]: from='client.14388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:03:46.862 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:03:46.862 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:03:46.862 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:47.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:46 smithi193 ceph-mon[37473]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:47.077 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:46 smithi193 ceph-mon[37473]: from='client.14388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:03:47.077 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:03:47.077 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:03:47.077 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:46 smithi189 ceph-mon[36790]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:46 smithi189 ceph-mon[36790]: from='client.14388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:03:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:03:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:03:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:48.964 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:48 smithi134 ceph-mon[31966]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:48 smithi189 ceph-mon[36790]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:48 smithi193 ceph-mon[37473]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:49 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2324538639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8abb1365-fc3e-4617-a9ac-76bece8cd7ec"}]: dispatch 2024-07-26T11:03:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:49 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8abb1365-fc3e-4617-a9ac-76bece8cd7ec"}]: dispatch 2024-07-26T11:03:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:49 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8abb1365-fc3e-4617-a9ac-76bece8cd7ec"}]': finished 2024-07-26T11:03:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:49 smithi134 ceph-mon[31966]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T11:03:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:03:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:49 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2324538639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8abb1365-fc3e-4617-a9ac-76bece8cd7ec"}]: dispatch 2024-07-26T11:03:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:49 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8abb1365-fc3e-4617-a9ac-76bece8cd7ec"}]: dispatch 2024-07-26T11:03:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:49 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8abb1365-fc3e-4617-a9ac-76bece8cd7ec"}]': finished 2024-07-26T11:03:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:49 smithi189 ceph-mon[36790]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T11:03:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:03:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:49 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2324538639' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8abb1365-fc3e-4617-a9ac-76bece8cd7ec"}]: dispatch 2024-07-26T11:03:50.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:49 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8abb1365-fc3e-4617-a9ac-76bece8cd7ec"}]: dispatch 2024-07-26T11:03:50.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:49 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8abb1365-fc3e-4617-a9ac-76bece8cd7ec"}]': finished 2024-07-26T11:03:50.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:49 smithi193 ceph-mon[37473]: osdmap e6: 1 total, 0 up, 1 in 2024-07-26T11:03:50.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:03:51.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:50 smithi134 ceph-mon[31966]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:51.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:50 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2130180735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:03:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:50 smithi189 ceph-mon[36790]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:50 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2130180735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:03:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:50 smithi193 ceph-mon[37473]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:51.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:50 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2130180735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:03:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:52 smithi134 ceph-mon[31966]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:52 smithi189 ceph-mon[36790]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:52 smithi193 ceph-mon[37473]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:54 smithi134 ceph-mon[31966]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:54 smithi189 ceph-mon[36790]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:54 smithi193 ceph-mon[37473]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:57.072 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:56 smithi134 ceph-mon[31966]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:56 smithi189 ceph-mon[36790]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:56 smithi193 ceph-mon[37473]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:59.100 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:58 smithi134 ceph-mon[31966]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:59.100 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T11:03:59.100 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:58 smithi189 ceph-mon[36790]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T11:03:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:03:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:58 smithi193 ceph-mon[37473]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:03:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-07-26T11:03:59.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:00.055 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:03:59 smithi134 ceph-mon[31966]: Deploying daemon osd.0 on smithi134 2024-07-26T11:04:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:03:59 smithi189 ceph-mon[36790]: Deploying daemon osd.0 on smithi134 2024-07-26T11:04:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:03:59 smithi193 ceph-mon[37473]: Deploying daemon osd.0 on smithi134 2024-07-26T11:04:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:00 smithi134 ceph-mon[31966]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:00 smithi189 ceph-mon[36790]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:00 smithi193 ceph-mon[37473]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:02.908 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:02 smithi134 ceph-mon[31966]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:02 smithi189 ceph-mon[36790]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:02 smithi193 ceph-mon[37473]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:04.983 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:04 smithi134 ceph-mon[31966]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:04 smithi189 ceph-mon[36790]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:04 smithi193 ceph-mon[37473]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:06 smithi134 ceph-mon[31966]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:07.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:07.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:07.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:06 smithi189 ceph-mon[36790]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:06 smithi193 ceph-mon[37473]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:08 smithi134 ceph-mon[31966]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:08 smithi189 ceph-mon[36790]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:08 smithi193 ceph-mon[37473]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:10.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:10 smithi134 ceph-mon[31966]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:10 smithi189 ceph-mon[36790]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:10.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:10 smithi193 ceph-mon[37473]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:11 smithi134 ceph-mon[31966]: from='osd.0 [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T11:04:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:11 smithi134 ceph-mon[31966]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T11:04:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:11 smithi189 ceph-mon[36790]: from='osd.0 [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T11:04:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:11 smithi189 ceph-mon[36790]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T11:04:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:11.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:11.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:11.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:11 smithi193 ceph-mon[37473]: from='osd.0 [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T11:04:11.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:11 smithi193 ceph-mon[37473]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-07-26T11:04:11.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:04:11.606+0000 7f8305600640 1 -- 172.21.15.134:0/3909235028 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8308111ad0 con 0x7f82f00b3b90 2024-07-26T11:04:11.609 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 0 on host 'smithi134' 2024-07-26T11:04:11.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:04:11.610+0000 7f830e200640 1 -- 172.21.15.134:0/3909235028 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82f00b3b90 msgr2=0x7f82f00b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:11.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:04:11.610+0000 7f830e200640 1 --2- 172.21.15.134:0/3909235028 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82f00b3b90 0x7f82f00b60a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0095f0 tx=0x7f82fc005f90 comp rx=0 tx=0).stop 2024-07-26T11:04:11.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:04:11.610+0000 7f830e200640 1 -- 172.21.15.134:0/3909235028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f830810fc80 msgr2=0x7f83081a5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:11.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:04:11.610+0000 7f830e200640 1 --2- 172.21.15.134:0/3909235028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f830810fc80 0x7f83081a5b10 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f830002ade0 tx=0x7f8300031660 comp rx=0 tx=0).stop 2024-07-26T11:04:11.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:04:11.610+0000 7f830e200640 1 -- 172.21.15.134:0/3909235028 shutdown_connections 2024-07-26T11:04:11.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:04:11.610+0000 7f830e200640 1 --2- 172.21.15.134:0/3909235028 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82f00b3b90 0x7f82f00b60a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:11.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:04:11.610+0000 7f830e200640 1 --2- 172.21.15.134:0/3909235028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f830810f330 0x7f83081a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:11.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:04:11.610+0000 7f830e200640 1 --2- 172.21.15.134:0/3909235028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f830810fc80 0x7f83081a5b10 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:11.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:04:11.610+0000 7f830e200640 1 --2- 172.21.15.134:0/3909235028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83081099f0 0x7f83081a02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:11.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:04:11.610+0000 7f830e200640 1 -- 172.21.15.134:0/3909235028 >> 172.21.15.134:0/3909235028 conn(0x7f8308078ff0 msgr2=0x7f83080796b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:11.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:04:11.610+0000 7f830e200640 1 -- 172.21.15.134:0/3909235028 shutdown_connections 2024-07-26T11:04:11.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:04:11.610+0000 7f830e200640 1 -- 172.21.15.134:0/3909235028 wait complete. 2024-07-26T11:04:12.349 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:12 smithi134 ceph-mon[31966]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T11:04:12.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:12 smithi134 ceph-mon[31966]: from='osd.0 [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:04:12.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:12 smithi134 ceph-mon[31966]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T11:04:12.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:04:12.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:12 smithi134 ceph-mon[31966]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:04:12.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:12.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:12.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:12.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:12 smithi134 ceph-mon[31966]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:12.426 DEBUG:teuthology.orchestra.run.smithi134:osd.0> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@osd.0.service 2024-07-26T11:04:12.430 INFO:tasks.cephadm:Deploying osd.1 on smithi189 with /dev/vg_nvme/lv_4... 2024-07-26T11:04:12.430 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T11:04:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:12 smithi189 ceph-mon[36790]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T11:04:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:12 smithi189 ceph-mon[36790]: from='osd.0 [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:04:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:12 smithi189 ceph-mon[36790]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T11:04:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:04:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:12 smithi189 ceph-mon[36790]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:04:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:12 smithi189 ceph-mon[36790]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:12 smithi193 ceph-mon[37473]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-07-26T11:04:12.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:12 smithi193 ceph-mon[37473]: from='osd.0 [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:04:12.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:12 smithi193 ceph-mon[37473]: osdmap e7: 1 total, 0 up, 1 in 2024-07-26T11:04:12.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:04:12.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:12 smithi193 ceph-mon[37473]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:04:12.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:12.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:12.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:12.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:12 smithi193 ceph-mon[37473]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:12.687 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T11:04:13.163 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jul 26 11:04:12 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-0[51584]: 2024-07-26T11:04:12.986+0000 7fd4d6a00640 -1 osd.0 0 waiting for initial osdmap 2024-07-26T11:04:13.163 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jul 26 11:04:13 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-0[51584]: 2024-07-26T11:04:13.011+0000 7fd4cca00640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T11:04:13.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:13 smithi134 ceph-mon[31966]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-07-26T11:04:13.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:13 smithi134 ceph-mon[31966]: osdmap e8: 1 total, 0 up, 1 in 2024-07-26T11:04:13.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:04:13.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:04:13.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:13 smithi134 ceph-mon[31966]: from='osd.0 ' entity='osd.0' 2024-07-26T11:04:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:13 smithi189 ceph-mon[36790]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-07-26T11:04:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:13 smithi189 ceph-mon[36790]: osdmap e8: 1 total, 0 up, 1 in 2024-07-26T11:04:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:04:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:04:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:13 smithi189 ceph-mon[36790]: from='osd.0 ' entity='osd.0' 2024-07-26T11:04:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:13 smithi193 ceph-mon[37473]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-07-26T11:04:13.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:13 smithi193 ceph-mon[37473]: osdmap e8: 1 total, 0 up, 1 in 2024-07-26T11:04:13.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:04:13.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:04:13.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:13 smithi193 ceph-mon[37473]: from='osd.0 ' entity='osd.0' 2024-07-26T11:04:14.263 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T11:04:14.374 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T11:04:14.387 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-07-26T11:04:14.387 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-07-26T11:04:14.388 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011339 s, 925 MB/s 2024-07-26T11:04:14.388 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-07-26T11:04:14.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:14 smithi134 ceph-mon[31966]: purged_snaps scrub starts 2024-07-26T11:04:14.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:14 smithi134 ceph-mon[31966]: purged_snaps scrub ok 2024-07-26T11:04:14.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:04:14.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:14 smithi134 ceph-mon[31966]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:14.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:14 smithi134 ceph-mon[31966]: osd.0 [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954] boot 2024-07-26T11:04:14.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:14 smithi134 ceph-mon[31966]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T11:04:14.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:04:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:14 smithi189 ceph-mon[36790]: purged_snaps scrub starts 2024-07-26T11:04:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:14 smithi189 ceph-mon[36790]: purged_snaps scrub ok 2024-07-26T11:04:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:04:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:14 smithi189 ceph-mon[36790]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:14 smithi189 ceph-mon[36790]: osd.0 [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954] boot 2024-07-26T11:04:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:14 smithi189 ceph-mon[36790]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T11:04:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:04:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:14 smithi193 ceph-mon[37473]: purged_snaps scrub starts 2024-07-26T11:04:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:14 smithi193 ceph-mon[37473]: purged_snaps scrub ok 2024-07-26T11:04:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:04:14.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:14 smithi193 ceph-mon[37473]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-07-26T11:04:14.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:14 smithi193 ceph-mon[37473]: osd.0 [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954] boot 2024-07-26T11:04:14.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:14 smithi193 ceph-mon[37473]: osdmap e9: 1 total, 1 up, 1 in 2024-07-26T11:04:14.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-07-26T11:04:15.135 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_4 2024-07-26T11:04:15.426 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T11:04:16.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:16 smithi134 ceph-mon[31966]: osdmap e10: 1 total, 1 up, 1 in 2024-07-26T11:04:16.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:16 smithi134 ceph-mon[31966]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:16.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:16 smithi189 ceph-mon[36790]: osdmap e10: 1 total, 1 up, 1 in 2024-07-26T11:04:16.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:16 smithi189 ceph-mon[36790]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:16 smithi193 ceph-mon[37473]: osdmap e10: 1 total, 1 up, 1 in 2024-07-26T11:04:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:16 smithi193 ceph-mon[37473]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:16.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.865+0000 7f4b2c71d640 1 -- 172.21.15.189:0/3179719118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b24101ab0 msgr2=0x7f4b24101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:16.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.865+0000 7f4b2c71d640 1 --2- 172.21.15.189:0/3179719118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b24101ab0 0x7f4b24101ec0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4b20009900 tx=0x7f4b2002a410 comp rx=0 tx=0).stop 2024-07-26T11:04:16.867 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.865+0000 7f4b2c71d640 1 -- 172.21.15.189:0/3179719118 shutdown_connections 2024-07-26T11:04:16.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.865+0000 7f4b2c71d640 1 --2- 172.21.15.189:0/3179719118 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b241098e0 0x7f4b24110110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:16.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.865+0000 7f4b2c71d640 1 --2- 172.21.15.189:0/3179719118 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b24108f90 0x7f4b241093a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:16.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.865+0000 7f4b2c71d640 1 --2- 172.21.15.189:0/3179719118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b24101ab0 0x7f4b24101ec0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:16.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.865+0000 7f4b2c71d640 1 -- 172.21.15.189:0/3179719118 >> 172.21.15.189:0/3179719118 conn(0x7f4b240798f0 msgr2=0x7f4b24079d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:16.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.865+0000 7f4b2c71d640 1 -- 172.21.15.189:0/3179719118 shutdown_connections 2024-07-26T11:04:16.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.865+0000 7f4b2c71d640 1 -- 172.21.15.189:0/3179719118 wait complete. 2024-07-26T11:04:16.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.865+0000 7f4b2c71d640 1 Processor -- start 2024-07-26T11:04:16.868 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.865+0000 7f4b2c71d640 1 -- start start 2024-07-26T11:04:16.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2c71d640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b24101ab0 0x7f4b2419a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:16.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2c71d640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b24108f90 0x7f4b2419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:16.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2c71d640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b241098e0 0x7f4b2419f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:16.870 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2c71d640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4b24113660 con 0x7f4b24108f90 2024-07-26T11:04:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2c71d640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4b241134e0 con 0x7f4b241098e0 2024-07-26T11:04:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2c71d640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4b241137e0 con 0x7f4b24101ab0 2024-07-26T11:04:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2acba640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b241098e0 0x7f4b2419f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2acba640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b241098e0 0x7f4b2419f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45266/0 (socket says 172.21.15.189:45266) 2024-07-26T11:04:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2acba640 1 -- 172.21.15.189:0/241516633 learned_addr learned my addr 172.21.15.189:0/241516633 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-07-26T11:04:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2a4b9640 1 --2- 172.21.15.189:0/241516633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b24101ab0 0x7f4b2419a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2acba640 1 -- 172.21.15.189:0/241516633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b24101ab0 msgr2=0x7f4b2419a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2acba640 1 --2- 172.21.15.189:0/241516633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b24101ab0 0x7f4b2419a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2acba640 1 -- 172.21.15.189:0/241516633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b24108f90 msgr2=0x7f4b2419a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b29cb8640 1 --2- 172.21.15.189:0/241516633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b24108f90 0x7f4b2419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2acba640 1 --2- 172.21.15.189:0/241516633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b24108f90 0x7f4b2419a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:16.871 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2acba640 1 -- 172.21.15.189:0/241516633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b241a0020 con 0x7f4b241098e0 2024-07-26T11:04:16.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2a4b9640 1 --2- 172.21.15.189:0/241516633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b24101ab0 0x7f4b2419a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:04:16.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.866+0000 7f4b2acba640 1 --2- 172.21.15.189:0/241516633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b241098e0 0x7f4b2419f8f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c00b970 tx=0x7f4b1c00be30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:16.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.867+0000 7f4b1b7fe640 1 -- 172.21.15.189:0/241516633 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c019070 con 0x7f4b241098e0 2024-07-26T11:04:16.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.867+0000 7f4b2c71d640 1 -- 172.21.15.189:0/241516633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b241a0310 con 0x7f4b241098e0 2024-07-26T11:04:16.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.867+0000 7f4b1b7fe640 1 -- 172.21.15.189:0/241516633 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b1c014440 con 0x7f4b241098e0 2024-07-26T11:04:16.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.867+0000 7f4b1b7fe640 1 -- 172.21.15.189:0/241516633 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c0075d0 con 0x7f4b241098e0 2024-07-26T11:04:16.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.867+0000 7f4b2c71d640 1 -- 172.21.15.189:0/241516633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b241a6a70 con 0x7f4b241098e0 2024-07-26T11:04:16.872 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.868+0000 7f4b2c71d640 1 -- 172.21.15.189:0/241516633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b24101ec0 con 0x7f4b241098e0 2024-07-26T11:04:16.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.869+0000 7f4b1b7fe640 1 -- 172.21.15.189:0/241516633 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7f4b1c01c070 con 0x7f4b241098e0 2024-07-26T11:04:16.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.870+0000 7f4b1b7fe640 1 --2- 172.21.15.189:0/241516633 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b100b3b90 0x7f4b100b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:16.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.870+0000 7f4b1b7fe640 1 -- 172.21.15.189:0/241516633 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1982+0+0 (secure 0 0 0) 0x7f4b1c0e18d0 con 0x7f4b241098e0 2024-07-26T11:04:16.873 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.870+0000 7f4b2a4b9640 1 --2- 172.21.15.189:0/241516633 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b100b3b90 0x7f4b100b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:16.874 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.871+0000 7f4b2a4b9640 1 --2- 172.21.15.189:0/241516633 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b100b3b90 0x7f4b100b60a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4b20031440 tx=0x7f4b20005d10 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:16.876 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:16.874+0000 7f4b1b7fe640 1 -- 172.21.15.189:0/241516633 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4b1c0aa8c0 con 0x7f4b241098e0 2024-07-26T11:04:17.056 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:17.054+0000 7f4b2c71d640 1 -- 172.21.15.189:0/241516633 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4b24063230 con 0x7f4b100b3b90 2024-07-26T11:04:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:04:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:04:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:04:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:17.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:04:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:04:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:04:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:17.445 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:04:17.445 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:04:17.445 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:17.445 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:17.446 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:17.446 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:04:17.446 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:17.446 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:17.446 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:17.446 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:18.464 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:18 smithi189 ceph-mon[36790]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:04:18.465 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:18 smithi189 ceph-mon[36790]: Detected new or changed devices on smithi134 2024-07-26T11:04:18.465 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:18 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 17089M 2024-07-26T11:04:18.465 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:18 smithi189 ceph-mon[36790]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:18 smithi193 ceph-mon[37473]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:04:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:18 smithi193 ceph-mon[37473]: Detected new or changed devices on smithi134 2024-07-26T11:04:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:18 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 17089M 2024-07-26T11:04:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:18 smithi193 ceph-mon[37473]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:18 smithi134 ceph-mon[31966]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:04:18.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:18 smithi134 ceph-mon[31966]: Detected new or changed devices on smithi134 2024-07-26T11:04:18.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:18 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 17089M 2024-07-26T11:04:18.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:18 smithi134 ceph-mon[31966]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:19 smithi134 ceph-mon[31966]: from='client.? 172.21.15.189:0/1494392759' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c5a7838-821e-44e1-97e4-655b9bfa9f38"}]: dispatch 2024-07-26T11:04:19.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:19 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c5a7838-821e-44e1-97e4-655b9bfa9f38"}]: dispatch 2024-07-26T11:04:19.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:19 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c5a7838-821e-44e1-97e4-655b9bfa9f38"}]': finished 2024-07-26T11:04:19.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:19 smithi134 ceph-mon[31966]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T11:04:19.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:19 smithi189 ceph-mon[36790]: from='client.? 172.21.15.189:0/1494392759' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c5a7838-821e-44e1-97e4-655b9bfa9f38"}]: dispatch 2024-07-26T11:04:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:19 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c5a7838-821e-44e1-97e4-655b9bfa9f38"}]: dispatch 2024-07-26T11:04:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:19 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c5a7838-821e-44e1-97e4-655b9bfa9f38"}]': finished 2024-07-26T11:04:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:19 smithi189 ceph-mon[36790]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T11:04:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:19 smithi193 ceph-mon[37473]: from='client.? 172.21.15.189:0/1494392759' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c5a7838-821e-44e1-97e4-655b9bfa9f38"}]: dispatch 2024-07-26T11:04:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:19 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c5a7838-821e-44e1-97e4-655b9bfa9f38"}]: dispatch 2024-07-26T11:04:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:19 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c5a7838-821e-44e1-97e4-655b9bfa9f38"}]': finished 2024-07-26T11:04:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:19 smithi193 ceph-mon[37473]: osdmap e11: 2 total, 1 up, 2 in 2024-07-26T11:04:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:20 smithi134 ceph-mon[31966]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:20 smithi134 ceph-mon[31966]: from='client.? 172.21.15.189:0/878632706' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:04:20.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:20 smithi193 ceph-mon[37473]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:20.689 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:20 smithi193 ceph-mon[37473]: from='client.? 172.21.15.189:0/878632706' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:04:20.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:20 smithi189 ceph-mon[36790]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:20.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:20 smithi189 ceph-mon[36790]: from='client.? 172.21.15.189:0/878632706' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:04:23.146 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:22 smithi189 ceph-mon[36790]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:22 smithi134 ceph-mon[31966]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:22 smithi193 ceph-mon[37473]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:24.879 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:24 smithi189 ceph-mon[36790]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:24 smithi134 ceph-mon[31966]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:24 smithi193 ceph-mon[37473]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:26 smithi134 ceph-mon[31966]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T11:04:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:26 smithi189 ceph-mon[36790]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T11:04:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:26 smithi193 ceph-mon[37473]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-07-26T11:04:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:27.960 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:27 smithi189 ceph-mon[36790]: Deploying daemon osd.1 on smithi189 2024-07-26T11:04:28.115 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:27 smithi193 ceph-mon[37473]: Deploying daemon osd.1 on smithi189 2024-07-26T11:04:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:27 smithi134 ceph-mon[31966]: Deploying daemon osd.1 on smithi189 2024-07-26T11:04:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:28 smithi193 ceph-mon[37473]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:28 smithi134 ceph-mon[31966]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:29.184 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:28 smithi189 ceph-mon[36790]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:30 smithi134 ceph-mon[31966]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:31.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:30 smithi189 ceph-mon[36790]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:30 smithi193 ceph-mon[37473]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:32.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:32 smithi189 ceph-mon[36790]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:32 smithi134 ceph-mon[31966]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:32 smithi193 ceph-mon[37473]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:33.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:33.797+0000 7f4b1b7fe640 1 -- 172.21.15.189:0/241516633 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4b24063230 con 0x7f4b100b3b90 2024-07-26T11:04:33.800 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 1 on host 'smithi189' 2024-07-26T11:04:33.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:33.800+0000 7f4b2c71d640 1 -- 172.21.15.189:0/241516633 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b100b3b90 msgr2=0x7f4b100b60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:33.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:33.800+0000 7f4b2c71d640 1 --2- 172.21.15.189:0/241516633 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b100b3b90 0x7f4b100b60a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4b20031440 tx=0x7f4b20005d10 comp rx=0 tx=0).stop 2024-07-26T11:04:33.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:33.800+0000 7f4b2c71d640 1 -- 172.21.15.189:0/241516633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b241098e0 msgr2=0x7f4b2419f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:33.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:33.800+0000 7f4b2c71d640 1 --2- 172.21.15.189:0/241516633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b241098e0 0x7f4b2419f8f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c00b970 tx=0x7f4b1c00be30 comp rx=0 tx=0).stop 2024-07-26T11:04:33.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:33.800+0000 7f4b2c71d640 1 -- 172.21.15.189:0/241516633 shutdown_connections 2024-07-26T11:04:33.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:33.800+0000 7f4b2c71d640 1 --2- 172.21.15.189:0/241516633 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b100b3b90 0x7f4b100b60a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:33.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:33.800+0000 7f4b2c71d640 1 --2- 172.21.15.189:0/241516633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b241098e0 0x7f4b2419f8f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:33.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:33.800+0000 7f4b2c71d640 1 --2- 172.21.15.189:0/241516633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b24108f90 0x7f4b2419a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:33.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:33.800+0000 7f4b2c71d640 1 --2- 172.21.15.189:0/241516633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b24101ab0 0x7f4b2419a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:33.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:33.800+0000 7f4b2c71d640 1 -- 172.21.15.189:0/241516633 >> 172.21.15.189:0/241516633 conn(0x7f4b240798f0 msgr2=0x7f4b24107090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:33.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:33.800+0000 7f4b2c71d640 1 -- 172.21.15.189:0/241516633 shutdown_connections 2024-07-26T11:04:33.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:04:33.801+0000 7f4b2c71d640 1 -- 172.21.15.189:0/241516633 wait complete. 2024-07-26T11:04:34.733 DEBUG:teuthology.orchestra.run.smithi189:osd.1> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@osd.1.service 2024-07-26T11:04:34.735 INFO:tasks.cephadm:Deploying osd.2 on smithi193 with /dev/vg_nvme/lv_4... 2024-07-26T11:04:34.736 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-07-26T11:04:34.992 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T11:04:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:34 smithi193 ceph-mon[37473]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:35.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:34 smithi134 ceph-mon[31966]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:35.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:35.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:34 smithi189 ceph-mon[36790]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:35.194 INFO:journalctl@ceph.osd.1.smithi189.stdout:Jul 26 11:04:34 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-1[42541]: 2024-07-26T11:04:34.885+0000 7f7d3387b640 -1 osd.1 0 log_to_monitors true 2024-07-26T11:04:35.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:35.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:35.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:35 smithi189 ceph-mon[36790]: from='osd.1 [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T11:04:35.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:35 smithi189 ceph-mon[36790]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T11:04:36.164 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:36.165 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:36.165 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:35 smithi193 ceph-mon[37473]: from='osd.1 [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T11:04:36.165 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:35 smithi193 ceph-mon[37473]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T11:04:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:36.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:35 smithi134 ceph-mon[31966]: from='osd.1 [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T11:04:36.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:35 smithi134 ceph-mon[31966]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-07-26T11:04:36.620 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-07-26T11:04:36.714 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-07-26T11:04:36.730 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-07-26T11:04:36.730 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-07-26T11:04:36.731 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0148352 s, 707 MB/s 2024-07-26T11:04:36.731 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-07-26T11:04:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:36 smithi193 ceph-mon[37473]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:37.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:36 smithi193 ceph-mon[37473]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T11:04:37.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:36 smithi193 ceph-mon[37473]: from='osd.1 [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:04:37.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:36 smithi193 ceph-mon[37473]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T11:04:37.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:37.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:36 smithi193 ceph-mon[37473]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:04:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:36 smithi134 ceph-mon[31966]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:36 smithi134 ceph-mon[31966]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T11:04:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:36 smithi134 ceph-mon[31966]: from='osd.1 [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:04:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:36 smithi134 ceph-mon[31966]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T11:04:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:36 smithi134 ceph-mon[31966]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:04:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:36 smithi189 ceph-mon[36790]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:36 smithi189 ceph-mon[36790]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-07-26T11:04:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:36 smithi189 ceph-mon[36790]: from='osd.1 [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:04:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:36 smithi189 ceph-mon[36790]: osdmap e12: 2 total, 1 up, 2 in 2024-07-26T11:04:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:36 smithi189 ceph-mon[36790]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:04:37.195 INFO:journalctl@ceph.osd.1.smithi189.stdout:Jul 26 11:04:36 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-1[42541]: 2024-07-26T11:04:36.899+0000 7f7d2f7fd640 -1 osd.1 0 waiting for initial osdmap 2024-07-26T11:04:37.195 INFO:journalctl@ceph.osd.1.smithi189.stdout:Jul 26 11:04:36 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-1[42541]: 2024-07-26T11:04:36.903+0000 7f7d2b5fb640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T11:04:37.277 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_4 2024-07-26T11:04:37.532 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T11:04:38.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:37 smithi134 ceph-mon[31966]: purged_snaps scrub starts 2024-07-26T11:04:38.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:37 smithi134 ceph-mon[31966]: purged_snaps scrub ok 2024-07-26T11:04:38.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:37 smithi134 ceph-mon[31966]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-07-26T11:04:38.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:37 smithi134 ceph-mon[31966]: osdmap e13: 2 total, 1 up, 2 in 2024-07-26T11:04:38.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:38.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:38.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:38.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:37 smithi189 ceph-mon[36790]: purged_snaps scrub starts 2024-07-26T11:04:38.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:37 smithi189 ceph-mon[36790]: purged_snaps scrub ok 2024-07-26T11:04:38.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:37 smithi189 ceph-mon[36790]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-07-26T11:04:38.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:37 smithi189 ceph-mon[36790]: osdmap e13: 2 total, 1 up, 2 in 2024-07-26T11:04:38.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:38.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:38.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:37 smithi193 ceph-mon[37473]: purged_snaps scrub starts 2024-07-26T11:04:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:37 smithi193 ceph-mon[37473]: purged_snaps scrub ok 2024-07-26T11:04:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:37 smithi193 ceph-mon[37473]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-07-26T11:04:38.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:37 smithi193 ceph-mon[37473]: osdmap e13: 2 total, 1 up, 2 in 2024-07-26T11:04:38.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:38.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:38.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:39.049 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.045+0000 7fb0814ba640 1 -- 172.21.15.193:0/1625561031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb07c1035b0 msgr2=0x7fb07c1039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:39.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.045+0000 7fb0814ba640 1 --2- 172.21.15.193:0/1625561031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb07c1035b0 0x7fb07c1039c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb06c009ca0 tx=0x7fb06c02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:04:39.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.045+0000 7fb0814ba640 1 -- 172.21.15.193:0/1625561031 shutdown_connections 2024-07-26T11:04:39.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.045+0000 7fb0814ba640 1 --2- 172.21.15.193:0/1625561031 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb07c109840 0x7fb07c110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:39.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.045+0000 7fb0814ba640 1 --2- 172.21.15.193:0/1625561031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb07c108ef0 0x7fb07c109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:39.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.045+0000 7fb0814ba640 1 --2- 172.21.15.193:0/1625561031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb07c1035b0 0x7fb07c1039c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:39.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.045+0000 7fb0814ba640 1 -- 172.21.15.193:0/1625561031 >> 172.21.15.193:0/1625561031 conn(0x7fb07c0ff040 msgr2=0x7fb07c101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:39.050 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.045+0000 7fb0814ba640 1 -- 172.21.15.193:0/1625561031 shutdown_connections 2024-07-26T11:04:39.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.045+0000 7fb0814ba640 1 -- 172.21.15.193:0/1625561031 wait complete. 2024-07-26T11:04:39.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.046+0000 7fb0814ba640 1 Processor -- start 2024-07-26T11:04:39.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.046+0000 7fb0814ba640 1 -- start start 2024-07-26T11:04:39.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.046+0000 7fb0814ba640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb07c1035b0 0x7fb07c19a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:39.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.046+0000 7fb0814ba640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb07c108ef0 0x7fb07c19a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:39.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.046+0000 7fb0814ba640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb07c109840 0x7fb07c19f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:39.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.046+0000 7fb0814ba640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb07c113580 con 0x7fb07c109840 2024-07-26T11:04:39.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.046+0000 7fb0814ba640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb07c113400 con 0x7fb07c108ef0 2024-07-26T11:04:39.051 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.046+0000 7fb0814ba640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb07c113700 con 0x7fb07c1035b0 2024-07-26T11:04:39.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.046+0000 7fb07affd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb07c1035b0 0x7fb07c19a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:39.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.046+0000 7fb07affd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb07c1035b0 0x7fb07c19a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:36946/0 (socket says 172.21.15.193:36946) 2024-07-26T11:04:39.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.046+0000 7fb07affd640 1 -- 172.21.15.193:0/520476085 learned_addr learned my addr 172.21.15.193:0/520476085 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-07-26T11:04:39.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.047+0000 7fb07a7fc640 1 --2- 172.21.15.193:0/520476085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb07c108ef0 0x7fb07c19a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:39.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.047+0000 7fb07affd640 1 -- 172.21.15.193:0/520476085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb07c108ef0 msgr2=0x7fb07c19a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:39.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.047+0000 7fb07affd640 1 --2- 172.21.15.193:0/520476085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb07c108ef0 0x7fb07c19a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:39.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.047+0000 7fb07affd640 1 -- 172.21.15.193:0/520476085 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb07c109840 msgr2=0x7fb07c19f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:39.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.047+0000 7fb07affd640 1 --2- 172.21.15.193:0/520476085 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb07c109840 0x7fb07c19f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:39.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.047+0000 7fb07affd640 1 -- 172.21.15.193:0/520476085 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07c19ffa0 con 0x7fb07c1035b0 2024-07-26T11:04:39.052 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.047+0000 7fb07affd640 1 --2- 172.21.15.193:0/520476085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb07c1035b0 0x7fb07c19a030 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb06c006010 tx=0x7fb06c0319f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:39.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.047+0000 7fb063fff640 1 -- 172.21.15.193:0/520476085 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06c003bf0 con 0x7fb07c1035b0 2024-07-26T11:04:39.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.047+0000 7fb063fff640 1 -- 172.21.15.193:0/520476085 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb06c003d90 con 0x7fb07c1035b0 2024-07-26T11:04:39.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.047+0000 7fb063fff640 1 -- 172.21.15.193:0/520476085 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06c02fc20 con 0x7fb07c1035b0 2024-07-26T11:04:39.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.047+0000 7fb0814ba640 1 -- 172.21.15.193:0/520476085 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb07c1a0230 con 0x7fb07c1035b0 2024-07-26T11:04:39.053 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.047+0000 7fb0814ba640 1 -- 172.21.15.193:0/520476085 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb07c1a69f0 con 0x7fb07c1035b0 2024-07-26T11:04:39.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.048+0000 7fb0814ba640 1 -- 172.21.15.193:0/520476085 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb07c1039c0 con 0x7fb07c1035b0 2024-07-26T11:04:39.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.049+0000 7fb063fff640 1 -- 172.21.15.193:0/520476085 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fb06c0046a0 con 0x7fb07c1035b0 2024-07-26T11:04:39.054 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.050+0000 7fb063fff640 1 --2- 172.21.15.193:0/520476085 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb0640b3c10 0x7fb0640b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:04:39.055 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.050+0000 7fb063fff640 1 -- 172.21.15.193:0/520476085 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2524+0+0 (secure 0 0 0) 0x7fb06c103a50 con 0x7fb07c1035b0 2024-07-26T11:04:39.055 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.051+0000 7fb07a7fc640 1 --2- 172.21.15.193:0/520476085 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb0640b3c10 0x7fb0640b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:04:39.055 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.051+0000 7fb07a7fc640 1 --2- 172.21.15.193:0/520476085 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb0640b3c10 0x7fb0640b6120 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb070005fd0 tx=0x7fb070005850 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:04:39.056 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.055+0000 7fb063fff640 1 -- 172.21.15.193:0/520476085 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb06c0cc820 con 0x7fb07c1035b0 2024-07-26T11:04:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: Detected new or changed devices on smithi189 2024-07-26T11:04:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:04:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: osd.1 [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049] boot 2024-07-26T11:04:39.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T11:04:39.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:39.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 17088M 2024-07-26T11:04:39.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:39.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:39.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:39.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:39.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:39.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:39.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:39.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: Detected new or changed devices on smithi189 2024-07-26T11:04:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:04:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: osd.1 [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049] boot 2024-07-26T11:04:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T11:04:39.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:39.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 17088M 2024-07-26T11:04:39.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:39.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:39.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:39.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:39.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:39.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:39.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:39.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:39.241 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:39.239+0000 7fb0814ba640 1 -- 172.21.15.193:0/520476085 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fb07c063230 con 0x7fb0640b3c10 2024-07-26T11:04:39.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-07-26T11:04:39.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: Detected new or changed devices on smithi189 2024-07-26T11:04:39.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:39.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:39.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:04:39.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: osd.1 [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049] boot 2024-07-26T11:04:39.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: osdmap e14: 2 total, 2 up, 2 in 2024-07-26T11:04:39.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-07-26T11:04:39.242 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 17088M 2024-07-26T11:04:39.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:39.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:39.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:39.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:39.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:39.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:39.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:39.243 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:40 smithi193 ceph-mon[37473]: osdmap e15: 2 total, 2 up, 2 in 2024-07-26T11:04:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:40 smithi193 ceph-mon[37473]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:04:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:04:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:04:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:40 smithi193 ceph-mon[37473]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:40.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:40 smithi189 ceph-mon[36790]: osdmap e15: 2 total, 2 up, 2 in 2024-07-26T11:04:40.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:40 smithi189 ceph-mon[36790]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:04:40.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:04:40.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:04:40.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:40.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:40 smithi189 ceph-mon[36790]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:40.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:40 smithi134 ceph-mon[31966]: osdmap e15: 2 total, 2 up, 2 in 2024-07-26T11:04:40.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:40 smithi134 ceph-mon[31966]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:04:40.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:04:40.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:04:40.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:40.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:40 smithi134 ceph-mon[31966]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:41.849 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:41 smithi193 ceph-mon[37473]: from='client.? 172.21.15.193:0/3951844878' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bc42f7f-39fa-4d3b-bc17-21c68f9231fa"}]: dispatch 2024-07-26T11:04:41.849 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:41 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bc42f7f-39fa-4d3b-bc17-21c68f9231fa"}]: dispatch 2024-07-26T11:04:41.849 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:41 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bc42f7f-39fa-4d3b-bc17-21c68f9231fa"}]': finished 2024-07-26T11:04:41.849 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:41 smithi193 ceph-mon[37473]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T11:04:41.849 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T11:04:41.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:41 smithi134 ceph-mon[31966]: from='client.? 172.21.15.193:0/3951844878' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bc42f7f-39fa-4d3b-bc17-21c68f9231fa"}]: dispatch 2024-07-26T11:04:41.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:41 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bc42f7f-39fa-4d3b-bc17-21c68f9231fa"}]: dispatch 2024-07-26T11:04:41.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:41 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bc42f7f-39fa-4d3b-bc17-21c68f9231fa"}]': finished 2024-07-26T11:04:41.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:41 smithi134 ceph-mon[31966]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T11:04:41.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T11:04:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:41 smithi189 ceph-mon[36790]: from='client.? 172.21.15.193:0/3951844878' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bc42f7f-39fa-4d3b-bc17-21c68f9231fa"}]: dispatch 2024-07-26T11:04:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:41 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bc42f7f-39fa-4d3b-bc17-21c68f9231fa"}]: dispatch 2024-07-26T11:04:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:41 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bc42f7f-39fa-4d3b-bc17-21c68f9231fa"}]': finished 2024-07-26T11:04:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:41 smithi189 ceph-mon[36790]: osdmap e16: 3 total, 2 up, 3 in 2024-07-26T11:04:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T11:04:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:42 smithi134 ceph-mon[31966]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:42.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:42 smithi134 ceph-mon[31966]: from='client.? 172.21.15.193:0/191105704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:04:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:42 smithi189 ceph-mon[36790]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:42 smithi189 ceph-mon[36790]: from='client.? 172.21.15.193:0/191105704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:04:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:42 smithi193 ceph-mon[37473]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:42 smithi193 ceph-mon[37473]: from='client.? 172.21.15.193:0/191105704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:04:45.069 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:44 smithi193 ceph-mon[37473]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:44 smithi134 ceph-mon[31966]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:44 smithi189 ceph-mon[36790]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:46.945 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:46 smithi193 ceph-mon[37473]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:46 smithi134 ceph-mon[31966]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:46 smithi189 ceph-mon[36790]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:49.079 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:48 smithi193 ceph-mon[37473]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:49.080 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T11:04:49.080 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:48 smithi134 ceph-mon[31966]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:49.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T11:04:49.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:48 smithi189 ceph-mon[36790]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-07-26T11:04:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:49.968 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:49 smithi193 ceph-mon[37473]: Deploying daemon osd.2 on smithi193 2024-07-26T11:04:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:49 smithi134 ceph-mon[31966]: Deploying daemon osd.2 on smithi193 2024-07-26T11:04:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:49 smithi189 ceph-mon[36790]: Deploying daemon osd.2 on smithi193 2024-07-26T11:04:51.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:50 smithi193 ceph-mon[37473]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:51.153 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:50 smithi189 ceph-mon[36790]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:50 smithi134 ceph-mon[31966]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:53.095 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:52 smithi193 ceph-mon[37473]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:52 smithi134 ceph-mon[31966]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:52 smithi189 ceph-mon[36790]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:54.003 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:54.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:54.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:54.108 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:54.109 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:54.109 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:54.971 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:54 smithi193 ceph-mon[37473]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:54 smithi134 ceph-mon[31966]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:54 smithi189 ceph-mon[36790]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:56.644 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:56.641+0000 7fb063fff640 1 -- 172.21.15.193:0/520476085 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb07c063230 con 0x7fb0640b3c10 2024-07-26T11:04:56.644 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 2 on host 'smithi193' 2024-07-26T11:04:56.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:56.644+0000 7fb0814ba640 1 -- 172.21.15.193:0/520476085 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb0640b3c10 msgr2=0x7fb0640b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:56.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:56.644+0000 7fb0814ba640 1 --2- 172.21.15.193:0/520476085 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb0640b3c10 0x7fb0640b6120 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb070005fd0 tx=0x7fb070005850 comp rx=0 tx=0).stop 2024-07-26T11:04:56.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:56.644+0000 7fb0814ba640 1 -- 172.21.15.193:0/520476085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb07c1035b0 msgr2=0x7fb07c19a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:04:56.646 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:56.644+0000 7fb0814ba640 1 --2- 172.21.15.193:0/520476085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb07c1035b0 0x7fb07c19a030 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb06c006010 tx=0x7fb06c0319f0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.647 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:56.644+0000 7fb0814ba640 1 -- 172.21.15.193:0/520476085 shutdown_connections 2024-07-26T11:04:56.647 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:56.644+0000 7fb0814ba640 1 --2- 172.21.15.193:0/520476085 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb0640b3c10 0x7fb0640b6120 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.647 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:56.644+0000 7fb0814ba640 1 --2- 172.21.15.193:0/520476085 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb07c109840 0x7fb07c19f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.647 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:56.644+0000 7fb0814ba640 1 --2- 172.21.15.193:0/520476085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb07c108ef0 0x7fb07c19a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.647 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:56.644+0000 7fb0814ba640 1 --2- 172.21.15.193:0/520476085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb07c1035b0 0x7fb07c19a030 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:04:56.647 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:56.644+0000 7fb0814ba640 1 -- 172.21.15.193:0/520476085 >> 172.21.15.193:0/520476085 conn(0x7fb07c0ff040 msgr2=0x7fb07c101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:04:56.647 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:56.644+0000 7fb0814ba640 1 -- 172.21.15.193:0/520476085 shutdown_connections 2024-07-26T11:04:56.647 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:04:56.644+0000 7fb0814ba640 1 -- 172.21.15.193:0/520476085 wait complete. 2024-07-26T11:04:56.996 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:56 smithi193 ceph-mon[37473]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:56.996 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:56.996 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:56.996 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:56.996 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:56.996 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:56.996 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:56.996 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:56.996 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:56 smithi134 ceph-mon[31966]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:56 smithi189 ceph-mon[36790]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:04:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:04:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:04:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:57.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:04:57.282 DEBUG:teuthology.orchestra.run.smithi193:osd.2> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@osd.2.service 2024-07-26T11:04:57.285 INFO:tasks.cephadm:Deploying osd.3 on smithi134 with /dev/vg_nvme/lv_3... 2024-07-26T11:04:57.285 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T11:04:57.741 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:04:58.086 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:57 smithi134 ceph-mon[31966]: from='osd.2 [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T11:04:58.086 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:57 smithi134 ceph-mon[31966]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T11:04:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:57 smithi189 ceph-mon[36790]: from='osd.2 [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T11:04:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:57 smithi189 ceph-mon[36790]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T11:04:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:57 smithi193 ceph-mon[37473]: from='osd.2 [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T11:04:58.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:57 smithi193 ceph-mon[37473]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-07-26T11:04:59.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:58 smithi134 ceph-mon[31966]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:59.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:58 smithi134 ceph-mon[31966]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T11:04:59.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:58 smithi134 ceph-mon[31966]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T11:04:59.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T11:04:59.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:58 smithi134 ceph-mon[31966]: from='osd.2 [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:04:59.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:58 smithi134 ceph-mon[31966]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:04:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:58 smithi189 ceph-mon[36790]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:58 smithi189 ceph-mon[36790]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T11:04:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:58 smithi189 ceph-mon[36790]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T11:04:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T11:04:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:58 smithi189 ceph-mon[36790]: from='osd.2 [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:04:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:58 smithi189 ceph-mon[36790]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:04:59.201 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:58 smithi193 ceph-mon[37473]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:04:59.201 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:58 smithi193 ceph-mon[37473]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-07-26T11:04:59.201 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:58 smithi193 ceph-mon[37473]: osdmap e17: 3 total, 2 up, 3 in 2024-07-26T11:04:59.201 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T11:04:59.201 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:58 smithi193 ceph-mon[37473]: from='osd.2 [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:04:59.201 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:58 smithi193 ceph-mon[37473]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:04:59.202 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jul 26 11:04:58 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-2[43828]: 2024-07-26T11:04:58.856+0000 7f8dd45c0640 -1 osd.2 0 waiting for initial osdmap 2024-07-26T11:04:59.202 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jul 26 11:04:58 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-2[43828]: 2024-07-26T11:04:58.861+0000 7f8dd03be640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T11:05:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:59 smithi134 ceph-mon[31966]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-07-26T11:05:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:59 smithi134 ceph-mon[31966]: osdmap e18: 3 total, 2 up, 3 in 2024-07-26T11:05:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T11:05:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:04:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T11:05:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:59 smithi189 ceph-mon[36790]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-07-26T11:05:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:59 smithi189 ceph-mon[36790]: osdmap e18: 3 total, 2 up, 3 in 2024-07-26T11:05:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T11:05:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:04:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T11:05:00.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:59 smithi193 ceph-mon[37473]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-07-26T11:05:00.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:59 smithi193 ceph-mon[37473]: osdmap e18: 3 total, 2 up, 3 in 2024-07-26T11:05:00.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T11:05:00.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:04:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T11:05:00.307 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T11:05:00.509 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T11:05:00.551 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-07-26T11:05:00.551 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-07-26T11:05:00.551 INFO:teuthology.orchestra.run.smithi134.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0324632 s, 323 MB/s 2024-07-26T11:05:00.552 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-07-26T11:05:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:00 smithi134 ceph-mon[31966]: purged_snaps scrub starts 2024-07-26T11:05:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:00 smithi134 ceph-mon[31966]: purged_snaps scrub ok 2024-07-26T11:05:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:00 smithi134 ceph-mon[31966]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:05:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:00 smithi134 ceph-mon[31966]: osd.2 [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976] boot 2024-07-26T11:05:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:00 smithi134 ceph-mon[31966]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T11:05:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T11:05:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:00 smithi134 ceph-mon[31966]: Detected new or changed devices on smithi193 2024-07-26T11:05:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:00 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 17088M 2024-07-26T11:05:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:00 smithi189 ceph-mon[36790]: purged_snaps scrub starts 2024-07-26T11:05:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:00 smithi189 ceph-mon[36790]: purged_snaps scrub ok 2024-07-26T11:05:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:00 smithi189 ceph-mon[36790]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:05:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:00 smithi189 ceph-mon[36790]: osd.2 [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976] boot 2024-07-26T11:05:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:00 smithi189 ceph-mon[36790]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T11:05:01.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T11:05:01.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:00 smithi189 ceph-mon[36790]: Detected new or changed devices on smithi193 2024-07-26T11:05:01.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:01.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:01.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:01.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:00 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 17088M 2024-07-26T11:05:01.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:01.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:01.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:01.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:00 smithi193 ceph-mon[37473]: purged_snaps scrub starts 2024-07-26T11:05:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:00 smithi193 ceph-mon[37473]: purged_snaps scrub ok 2024-07-26T11:05:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:00 smithi193 ceph-mon[37473]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-07-26T11:05:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:00 smithi193 ceph-mon[37473]: osd.2 [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976] boot 2024-07-26T11:05:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:00 smithi193 ceph-mon[37473]: osdmap e19: 3 total, 3 up, 3 in 2024-07-26T11:05:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-07-26T11:05:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:00 smithi193 ceph-mon[37473]: Detected new or changed devices on smithi193 2024-07-26T11:05:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:00 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 17088M 2024-07-26T11:05:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:01.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:01.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:01.532 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi134:vg_nvme/lv_3 2024-07-26T11:05:01.984 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:05:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:01 smithi189 ceph-mon[36790]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T11:05:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T11:05:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:01 smithi193 ceph-mon[37473]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T11:05:02.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T11:05:02.285 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:01 smithi134 ceph-mon[31966]: osdmap e20: 3 total, 3 up, 3 in 2024-07-26T11:05:02.285 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-07-26T11:05:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:02 smithi189 ceph-mon[36790]: pgmap v82: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T11:05:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:02 smithi189 ceph-mon[36790]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T11:05:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T11:05:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:02 smithi193 ceph-mon[37473]: pgmap v82: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T11:05:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:02 smithi193 ceph-mon[37473]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T11:05:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T11:05:03.411 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:02 smithi134 ceph-mon[31966]: pgmap v82: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:03.411 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-07-26T11:05:03.411 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:02 smithi134 ceph-mon[31966]: osdmap e21: 3 total, 3 up, 3 in 2024-07-26T11:05:03.411 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-07-26T11:05:03.411 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jul 26 11:05:03 smithi134 sudo[55747]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T11:05:03.411 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jul 26 11:05:03 smithi134 sudo[55747]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T11:05:03.412 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jul 26 11:05:03 smithi134 sudo[55747]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T11:05:03.412 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jul 26 11:05:03 smithi134 sudo[55747]: pam_unix(sudo:session): session closed for user root 2024-07-26T11:05:03.412 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jul 26 11:05:03 smithi134 sudo[55750]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T11:05:03.412 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jul 26 11:05:03 smithi134 sudo[55750]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T11:05:03.412 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jul 26 11:05:03 smithi134 sudo[55750]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T11:05:03.412 INFO:journalctl@ceph.osd.0.smithi134.stdout:Jul 26 11:05:03 smithi134 sudo[55750]: pam_unix(sudo:session): session closed for user root 2024-07-26T11:05:03.544 INFO:journalctl@ceph.osd.1.smithi189.stdout:Jul 26 11:05:03 smithi189 sudo[46363]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T11:05:03.544 INFO:journalctl@ceph.osd.1.smithi189.stdout:Jul 26 11:05:03 smithi189 sudo[46363]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T11:05:03.544 INFO:journalctl@ceph.osd.1.smithi189.stdout:Jul 26 11:05:03 smithi189 sudo[46363]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T11:05:03.545 INFO:journalctl@ceph.osd.1.smithi189.stdout:Jul 26 11:05:03 smithi189 sudo[46363]: pam_unix(sudo:session): session closed for user root 2024-07-26T11:05:03.545 INFO:journalctl@ceph.osd.1.smithi189.stdout:Jul 26 11:05:03 smithi189 sudo[46366]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T11:05:03.545 INFO:journalctl@ceph.osd.1.smithi189.stdout:Jul 26 11:05:03 smithi189 sudo[46366]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T11:05:03.545 INFO:journalctl@ceph.osd.1.smithi189.stdout:Jul 26 11:05:03 smithi189 sudo[46366]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T11:05:03.545 INFO:journalctl@ceph.osd.1.smithi189.stdout:Jul 26 11:05:03 smithi189 sudo[46366]: pam_unix(sudo:session): session closed for user root 2024-07-26T11:05:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 sudo[55753]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T11:05:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 sudo[55753]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T11:05:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 sudo[55753]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T11:05:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 sudo[55753]: pam_unix(sudo:session): session closed for user root 2024-07-26T11:05:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 sudo[55756]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T11:05:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 sudo[55756]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T11:05:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 sudo[55756]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T11:05:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 sudo[55756]: pam_unix(sudo:session): session closed for user root 2024-07-26T11:05:03.774 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jul 26 11:05:03 smithi193 sudo[47420]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T11:05:03.775 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jul 26 11:05:03 smithi193 sudo[47420]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T11:05:03.775 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jul 26 11:05:03 smithi193 sudo[47420]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T11:05:03.775 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jul 26 11:05:03 smithi193 sudo[47420]: pam_unix(sudo:session): session closed for user root 2024-07-26T11:05:03.775 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jul 26 11:05:03 smithi193 sudo[47423]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T11:05:03.775 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jul 26 11:05:03 smithi193 sudo[47423]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T11:05:03.775 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jul 26 11:05:03 smithi193 sudo[47423]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T11:05:03.775 INFO:journalctl@ceph.osd.2.smithi193.stdout:Jul 26 11:05:03 smithi193 sudo[47423]: pam_unix(sudo:session): session closed for user root 2024-07-26T11:05:03.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 sudo[47426]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T11:05:03.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 sudo[47426]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T11:05:03.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 sudo[47426]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T11:05:03.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 sudo[47426]: pam_unix(sudo:session): session closed for user root 2024-07-26T11:05:03.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 sudo[47429]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T11:05:03.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 sudo[47429]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T11:05:03.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 sudo[47429]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T11:05:03.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 sudo[47429]: pam_unix(sudo:session): session closed for user root 2024-07-26T11:05:03.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 sudo[46369]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-07-26T11:05:03.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 sudo[46369]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T11:05:03.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 sudo[46369]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T11:05:03.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 sudo[46369]: pam_unix(sudo:session): session closed for user root 2024-07-26T11:05:03.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 sudo[46372]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-07-26T11:05:03.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 sudo[46372]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-07-26T11:05:03.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 sudo[46372]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-07-26T11:05:03.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 sudo[46372]: pam_unix(sudo:session): session closed for user root 2024-07-26T11:05:04.129 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T11:05:04.129 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: osdmap e22: 3 total, 3 up, 3 in 2024-07-26T11:05:04.129 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:04.129 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:04.129 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:04.129 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T11:05:04.129 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T11:05:04.129 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:05:04.129 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:05:04.130 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:05:04.130 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T11:05:04.130 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:05:04.130 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:05:04.130 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:05:04.130 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T11:05:04.130 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T11:05:04.130 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:05:04.130 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:05:04.130 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:05:04.130 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:03 smithi134 ceph-mon[31966]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T11:05:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T11:05:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: osdmap e22: 3 total, 3 up, 3 in 2024-07-26T11:05:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T11:05:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T11:05:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:05:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:05:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:05:04.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T11:05:04.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:05:04.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:05:04.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:05:04.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T11:05:04.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T11:05:04.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:05:04.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:05:04.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:05:04.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:03 smithi193 ceph-mon[37473]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T11:05:04.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.333+0000 7fc12de00640 1 -- 172.21.15.134:0/2549617420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc12810af10 msgr2=0x7fc12810b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:04.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.333+0000 7fc12de00640 1 --2- 172.21.15.134:0/2549617420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc12810af10 0x7fc12810b320 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc114009860 tx=0x7fc11402a340 comp rx=0 tx=0).stop 2024-07-26T11:05:04.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.333+0000 7fc12de00640 1 -- 172.21.15.134:0/2549617420 shutdown_connections 2024-07-26T11:05:04.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.333+0000 7fc12de00640 1 --2- 172.21.15.134:0/2549617420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc12810af10 0x7fc12810b320 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:04.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.333+0000 7fc12de00640 1 --2- 172.21.15.134:0/2549617420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc12810b860 0x7fc128112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:04.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.333+0000 7fc12de00640 1 --2- 172.21.15.134:0/2549617420 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc1281055d0 0x7fc1281059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:04.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.333+0000 7fc12de00640 1 -- 172.21.15.134:0/2549617420 >> 172.21.15.134:0/2549617420 conn(0x7fc128101040 msgr2=0x7fc128103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:04.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.333+0000 7fc12de00640 1 -- 172.21.15.134:0/2549617420 shutdown_connections 2024-07-26T11:05:04.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.334+0000 7fc12de00640 1 -- 172.21.15.134:0/2549617420 wait complete. 2024-07-26T11:05:04.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.335+0000 7fc12de00640 1 Processor -- start 2024-07-26T11:05:04.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.335+0000 7fc12de00640 1 -- start start 2024-07-26T11:05:04.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.335+0000 7fc12de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1281055d0 0x7fc12819bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:04.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.335+0000 7fc12de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc12810af10 0x7fc12819c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:04.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.336+0000 7fc12de00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc12810b860 0x7fc1281a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:04.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.336+0000 7fc12de00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc1281154f0 con 0x7fc12810b860 2024-07-26T11:05:04.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.336+0000 7fc12de00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc128115370 con 0x7fc1281055d0 2024-07-26T11:05:04.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.336+0000 7fc12de00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc128115670 con 0x7fc12810af10 2024-07-26T11:05:04.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.336+0000 7fc127e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc12810af10 0x7fc12819c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:04.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.336+0000 7fc127e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc12810af10 0x7fc12819c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:38854/0 (socket says 172.21.15.134:38854) 2024-07-26T11:05:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.336+0000 7fc127e00640 1 -- 172.21.15.134:0/3440180143 learned_addr learned my addr 172.21.15.134:0/3440180143 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:05:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.336+0000 7fc12ca00640 1 --2- 172.21.15.134:0/3440180143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1281055d0 0x7fc12819bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.336+0000 7fc12d400640 1 --2- 172.21.15.134:0/3440180143 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc12810b860 0x7fc1281a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.336+0000 7fc127e00640 1 -- 172.21.15.134:0/3440180143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1281055d0 msgr2=0x7fc12819bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.336+0000 7fc127e00640 1 --2- 172.21.15.134:0/3440180143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1281055d0 0x7fc12819bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.336+0000 7fc127e00640 1 -- 172.21.15.134:0/3440180143 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc12810b860 msgr2=0x7fc1281a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.336+0000 7fc127e00640 1 --2- 172.21.15.134:0/3440180143 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc12810b860 0x7fc1281a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.336+0000 7fc127e00640 1 -- 172.21.15.134:0/3440180143 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1281a1e90 con 0x7fc12810af10 2024-07-26T11:05:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.336+0000 7fc12d400640 1 --2- 172.21.15.134:0/3440180143 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc12810b860 0x7fc1281a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:05:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.337+0000 7fc127e00640 1 --2- 172.21.15.134:0/3440180143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc12810af10 0x7fc12819c4c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc114035040 tx=0x7fc114032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.337+0000 7fc125600640 1 -- 172.21.15.134:0/3440180143 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc114037040 con 0x7fc12810af10 2024-07-26T11:05:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.337+0000 7fc125600640 1 -- 172.21.15.134:0/3440180143 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc114030c10 con 0x7fc12810af10 2024-07-26T11:05:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.337+0000 7fc125600640 1 -- 172.21.15.134:0/3440180143 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1140035f0 con 0x7fc12810af10 2024-07-26T11:05:04.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.337+0000 7fc12de00640 1 -- 172.21.15.134:0/3440180143 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1281a2120 con 0x7fc12810af10 2024-07-26T11:05:04.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.337+0000 7fc12de00640 1 -- 172.21.15.134:0/3440180143 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1281a88e0 con 0x7fc12810af10 2024-07-26T11:05:04.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.338+0000 7fc12de00640 1 -- 172.21.15.134:0/3440180143 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc128106850 con 0x7fc12810af10 2024-07-26T11:05:04.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.341+0000 7fc125600640 1 -- 172.21.15.134:0/3440180143 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) ==== 151021+0+0 (secure 0 0 0) 0x7fc114043020 con 0x7fc12810af10 2024-07-26T11:05:04.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.343+0000 7fc125600640 1 --2- 172.21.15.134:0/3440180143 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc1100b3c10 0x7fc1100b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:04.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.343+0000 7fc125600640 1 -- 172.21.15.134:0/3440180143 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 3452+0+0 (secure 0 0 0) 0x7fc114101b50 con 0x7fc12810af10 2024-07-26T11:05:04.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.343+0000 7fc12ca00640 1 --2- 172.21.15.134:0/3440180143 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc1100b3c10 0x7fc1100b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:04.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.344+0000 7fc12ca00640 1 --2- 172.21.15.134:0/3440180143 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc1100b3c10 0x7fc1100b6120 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc11c009530 tx=0x7fc11c009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:04.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.346+0000 7fc125600640 1 -- 172.21.15.134:0/3440180143 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc11404a060 con 0x7fc12810af10 2024-07-26T11:05:04.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.384+0000 7fc125600640 1 -- 172.21.15.134:0/3440180143 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fc1140c6070 con 0x7fc12810af10 2024-07-26T11:05:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-07-26T11:05:04.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: osdmap e22: 3 total, 3 up, 3 in 2024-07-26T11:05:04.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:04.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:04.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:04.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T11:05:04.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T11:05:04.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:05:04.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:05:04.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:05:04.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T11:05:04.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:05:04.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:05:04.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:05:04.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T11:05:04.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-07-26T11:05:04.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-07-26T11:05:04.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-07-26T11:05:04.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-07-26T11:05:04.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:03 smithi189 ceph-mon[36790]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-07-26T11:05:04.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:04.561+0000 7fc12de00640 1 -- 172.21.15.134:0/3440180143 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fc128063230 con 0x7fc1100b3c10 2024-07-26T11:05:05.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:04 smithi193 ceph-mon[37473]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:05.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:04 smithi193 ceph-mon[37473]: osdmap e23: 3 total, 3 up, 3 in 2024-07-26T11:05:05.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:04 smithi193 ceph-mon[37473]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-07-26T11:05:05.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:05:05.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:05:05.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:05.287 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:04 smithi134 ceph-mon[31966]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:05.287 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:04 smithi134 ceph-mon[31966]: osdmap e23: 3 total, 3 up, 3 in 2024-07-26T11:05:05.287 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:04 smithi134 ceph-mon[31966]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-07-26T11:05:05.287 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:05:05.287 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:05:05.287 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:04 smithi189 ceph-mon[36790]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:04 smithi189 ceph-mon[36790]: osdmap e23: 3 total, 3 up, 3 in 2024-07-26T11:05:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:04 smithi189 ceph-mon[36790]: mgrmap e20: x(active, since 2m), standbys: z, y 2024-07-26T11:05:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:05:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:05:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:05 smithi193 ceph-mon[37473]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:05:06.378 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:05 smithi134 ceph-mon[31966]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:05:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:05 smithi189 ceph-mon[36790]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:05:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:06 smithi193 ceph-mon[37473]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:07.321 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:06 smithi134 ceph-mon[31966]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:06 smithi189 ceph-mon[36790]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:08.101 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:07 smithi134 ceph-mon[31966]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:07 smithi193 ceph-mon[37473]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:08.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:07 smithi189 ceph-mon[36790]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:08 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2889394174' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0473d055-0b79-434f-9658-28269d2b1fbf"}]: dispatch 2024-07-26T11:05:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:08 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0473d055-0b79-434f-9658-28269d2b1fbf"}]: dispatch 2024-07-26T11:05:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:08 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0473d055-0b79-434f-9658-28269d2b1fbf"}]': finished 2024-07-26T11:05:09.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:08 smithi134 ceph-mon[31966]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T11:05:09.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:08 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2889394174' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0473d055-0b79-434f-9658-28269d2b1fbf"}]: dispatch 2024-07-26T11:05:09.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:08 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0473d055-0b79-434f-9658-28269d2b1fbf"}]: dispatch 2024-07-26T11:05:09.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:08 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0473d055-0b79-434f-9658-28269d2b1fbf"}]': finished 2024-07-26T11:05:09.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:08 smithi193 ceph-mon[37473]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T11:05:09.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:08 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2889394174' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0473d055-0b79-434f-9658-28269d2b1fbf"}]: dispatch 2024-07-26T11:05:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:08 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0473d055-0b79-434f-9658-28269d2b1fbf"}]: dispatch 2024-07-26T11:05:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:08 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0473d055-0b79-434f-9658-28269d2b1fbf"}]': finished 2024-07-26T11:05:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:08 smithi189 ceph-mon[36790]: osdmap e24: 4 total, 3 up, 4 in 2024-07-26T11:05:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:10.278 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:09 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3987823037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:05:10.278 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:09 smithi193 ceph-mon[37473]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:10.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:09 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3987823037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:05:10.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:09 smithi189 ceph-mon[36790]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:09 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3987823037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:05:10.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:09 smithi134 ceph-mon[31966]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:12.910 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:12 smithi134 ceph-mon[31966]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:13.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:12 smithi189 ceph-mon[36790]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:12 smithi193 ceph-mon[37473]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:14 smithi134 ceph-mon[31966]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:14 smithi189 ceph-mon[36790]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:14 smithi193 ceph-mon[37473]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:17.095 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:16 smithi193 ceph-mon[37473]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:17.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:16 smithi134 ceph-mon[31966]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:16 smithi189 ceph-mon[36790]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T11:05:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T11:05:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-07-26T11:05:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:18 smithi134 ceph-mon[31966]: Deploying daemon osd.3 on smithi134 2024-07-26T11:05:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:18 smithi134 ceph-mon[31966]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:18 smithi189 ceph-mon[36790]: Deploying daemon osd.3 on smithi134 2024-07-26T11:05:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:18 smithi189 ceph-mon[36790]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:18 smithi193 ceph-mon[37473]: Deploying daemon osd.3 on smithi134 2024-07-26T11:05:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:18 smithi193 ceph-mon[37473]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:21.123 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:20 smithi134 ceph-mon[31966]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:20 smithi189 ceph-mon[36790]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:20 smithi193 ceph-mon[37473]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:22.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:22 smithi134 ceph-mon[31966]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:23.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:22 smithi189 ceph-mon[36790]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:22 smithi193 ceph-mon[37473]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:24 smithi134 ceph-mon[31966]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:24 smithi189 ceph-mon[36790]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:24 smithi193 ceph-mon[37473]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:26 smithi134 ceph-mon[31966]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:26 smithi189 ceph-mon[36790]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:26 smithi193 ceph-mon[37473]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:28 smithi134 ceph-mon[31966]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:28 smithi189 ceph-mon[36790]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:28 smithi193 ceph-mon[37473]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:31.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:30 smithi134 ceph-mon[31966]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:31.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:31.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:31.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:30 smithi134 ceph-mon[31966]: from='osd.3 [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T11:05:31.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:30 smithi134 ceph-mon[31966]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T11:05:31.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:31.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:31.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:30 smithi193 ceph-mon[37473]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:30 smithi193 ceph-mon[37473]: from='osd.3 [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T11:05:31.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:30 smithi193 ceph-mon[37473]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T11:05:31.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:31.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:31.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:30 smithi189 ceph-mon[36790]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:30 smithi189 ceph-mon[36790]: from='osd.3 [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T11:05:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:30 smithi189 ceph-mon[36790]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-07-26T11:05:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:31.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:31.747 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 3 on host 'smithi134' 2024-07-26T11:05:31.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:31.743+0000 7fc125600640 1 -- 172.21.15.134:0/3440180143 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc128063230 con 0x7fc1100b3c10 2024-07-26T11:05:31.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:31.747+0000 7fc12de00640 1 -- 172.21.15.134:0/3440180143 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc1100b3c10 msgr2=0x7fc1100b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:31.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:31.747+0000 7fc12de00640 1 --2- 172.21.15.134:0/3440180143 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc1100b3c10 0x7fc1100b6120 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc11c009530 tx=0x7fc11c009260 comp rx=0 tx=0).stop 2024-07-26T11:05:31.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:31.747+0000 7fc12de00640 1 -- 172.21.15.134:0/3440180143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc12810af10 msgr2=0x7fc12819c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:31.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:31.747+0000 7fc12de00640 1 --2- 172.21.15.134:0/3440180143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc12810af10 0x7fc12819c4c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc114035040 tx=0x7fc114032d50 comp rx=0 tx=0).stop 2024-07-26T11:05:31.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:31.747+0000 7fc12de00640 1 -- 172.21.15.134:0/3440180143 shutdown_connections 2024-07-26T11:05:31.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:31.747+0000 7fc12de00640 1 --2- 172.21.15.134:0/3440180143 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc1100b3c10 0x7fc1100b6120 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:31.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:31.747+0000 7fc12de00640 1 --2- 172.21.15.134:0/3440180143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc12810af10 0x7fc12819c4c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:31.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:31.747+0000 7fc12de00640 1 --2- 172.21.15.134:0/3440180143 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc12810b860 0x7fc1281a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:31.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:31.747+0000 7fc12de00640 1 --2- 172.21.15.134:0/3440180143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1281055d0 0x7fc12819bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:31.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:31.747+0000 7fc12de00640 1 -- 172.21.15.134:0/3440180143 >> 172.21.15.134:0/3440180143 conn(0x7fc128101040 msgr2=0x7fc128103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:31.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:31.748+0000 7fc12de00640 1 -- 172.21.15.134:0/3440180143 shutdown_connections 2024-07-26T11:05:31.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:05:31.748+0000 7fc12de00640 1 -- 172.21.15.134:0/3440180143 wait complete. 2024-07-26T11:05:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:31 smithi193 ceph-mon[37473]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T11:05:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:31 smithi193 ceph-mon[37473]: osdmap e25: 4 total, 3 up, 4 in 2024-07-26T11:05:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:32.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:31 smithi193 ceph-mon[37473]: from='osd.3 [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:05:32.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:31 smithi193 ceph-mon[37473]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:05:32.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:32.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:32.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:32.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:31 smithi193 ceph-mon[37473]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:32.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:31 smithi134 ceph-mon[31966]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T11:05:32.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:31 smithi134 ceph-mon[31966]: osdmap e25: 4 total, 3 up, 4 in 2024-07-26T11:05:32.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:32.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:31 smithi134 ceph-mon[31966]: from='osd.3 [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:05:32.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:31 smithi134 ceph-mon[31966]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:05:32.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:32.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:32.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:32.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:31 smithi134 ceph-mon[31966]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:31 smithi189 ceph-mon[36790]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-07-26T11:05:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:31 smithi189 ceph-mon[36790]: osdmap e25: 4 total, 3 up, 4 in 2024-07-26T11:05:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:31 smithi189 ceph-mon[36790]: from='osd.3 [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:05:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:31 smithi189 ceph-mon[36790]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:05:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:32.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:31 smithi189 ceph-mon[36790]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:32.644 DEBUG:teuthology.orchestra.run.smithi134:osd.3> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@osd.3.service 2024-07-26T11:05:32.650 INFO:tasks.cephadm:Deploying osd.4 on smithi189 with /dev/vg_nvme/lv_3... 2024-07-26T11:05:32.650 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T11:05:32.911 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T11:05:33.156 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:32 smithi134 ceph-mon[31966]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-07-26T11:05:33.156 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:32 smithi134 ceph-mon[31966]: osdmap e26: 4 total, 3 up, 4 in 2024-07-26T11:05:33.156 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:33.157 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:33.157 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:32 smithi134 ceph-mon[31966]: osdmap e27: 4 total, 3 up, 4 in 2024-07-26T11:05:33.157 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:33.157 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:32 smithi134 ceph-mon[31966]: from='osd.3 ' entity='osd.3' 2024-07-26T11:05:33.157 INFO:journalctl@ceph.osd.3.smithi134.stdout:Jul 26 11:05:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-3[58046]: 2024-07-26T11:05:32.880+0000 7f8bfcc00640 -1 osd.3 0 waiting for initial osdmap 2024-07-26T11:05:33.157 INFO:journalctl@ceph.osd.3.smithi134.stdout:Jul 26 11:05:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-3[58046]: 2024-07-26T11:05:32.900+0000 7f8bf2200640 -1 osd.3 27 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T11:05:33.245 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:32 smithi189 ceph-mon[36790]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-07-26T11:05:33.245 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:32 smithi189 ceph-mon[36790]: osdmap e26: 4 total, 3 up, 4 in 2024-07-26T11:05:33.245 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:33.245 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:33.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:32 smithi189 ceph-mon[36790]: osdmap e27: 4 total, 3 up, 4 in 2024-07-26T11:05:33.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:33.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:32 smithi189 ceph-mon[36790]: from='osd.3 ' entity='osd.3' 2024-07-26T11:05:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:32 smithi193 ceph-mon[37473]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-07-26T11:05:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:32 smithi193 ceph-mon[37473]: osdmap e26: 4 total, 3 up, 4 in 2024-07-26T11:05:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:33.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:32 smithi193 ceph-mon[37473]: osdmap e27: 4 total, 3 up, 4 in 2024-07-26T11:05:33.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:33.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:32 smithi193 ceph-mon[37473]: from='osd.3 ' entity='osd.3' 2024-07-26T11:05:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:33 smithi193 ceph-mon[37473]: purged_snaps scrub starts 2024-07-26T11:05:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:33 smithi193 ceph-mon[37473]: purged_snaps scrub ok 2024-07-26T11:05:34.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:34.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:33 smithi193 ceph-mon[37473]: osd.3 [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917] boot 2024-07-26T11:05:34.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:33 smithi193 ceph-mon[37473]: osdmap e28: 4 total, 4 up, 4 in 2024-07-26T11:05:34.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:34.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:33 smithi193 ceph-mon[37473]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:34.296 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:33 smithi189 ceph-mon[36790]: purged_snaps scrub starts 2024-07-26T11:05:34.296 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:33 smithi189 ceph-mon[36790]: purged_snaps scrub ok 2024-07-26T11:05:34.296 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:34.296 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:33 smithi189 ceph-mon[36790]: osd.3 [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917] boot 2024-07-26T11:05:34.297 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:33 smithi189 ceph-mon[36790]: osdmap e28: 4 total, 4 up, 4 in 2024-07-26T11:05:34.297 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:34.297 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:33 smithi189 ceph-mon[36790]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:33 smithi134 ceph-mon[31966]: purged_snaps scrub starts 2024-07-26T11:05:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:33 smithi134 ceph-mon[31966]: purged_snaps scrub ok 2024-07-26T11:05:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:33 smithi134 ceph-mon[31966]: osd.3 [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917] boot 2024-07-26T11:05:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:33 smithi134 ceph-mon[31966]: osdmap e28: 4 total, 4 up, 4 in 2024-07-26T11:05:34.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-07-26T11:05:34.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:33 smithi134 ceph-mon[31966]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-07-26T11:05:34.665 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T11:05:34.749 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T11:05:34.763 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-07-26T11:05:34.763 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-07-26T11:05:34.763 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112156 s, 935 MB/s 2024-07-26T11:05:34.763 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-07-26T11:05:35.491 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_3 2024-07-26T11:05:35.743 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T11:05:35.999 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:35 smithi189 ceph-mon[36790]: osdmap e29: 4 total, 4 up, 4 in 2024-07-26T11:05:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:35 smithi193 ceph-mon[37473]: osdmap e29: 4 total, 4 up, 4 in 2024-07-26T11:05:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:35 smithi134 ceph-mon[31966]: osdmap e29: 4 total, 4 up, 4 in 2024-07-26T11:05:36.855 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:36 smithi189 ceph-mon[36790]: osdmap e30: 4 total, 4 up, 4 in 2024-07-26T11:05:36.855 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:36 smithi189 ceph-mon[36790]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:36 smithi193 ceph-mon[37473]: osdmap e30: 4 total, 4 up, 4 in 2024-07-26T11:05:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:36 smithi193 ceph-mon[37473]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:36 smithi134 ceph-mon[31966]: osdmap e30: 4 total, 4 up, 4 in 2024-07-26T11:05:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:36 smithi134 ceph-mon[31966]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.347+0000 7fde0b1e7640 1 -- 172.21.15.189:0/2748302788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde04108720 msgr2=0x7fde0410ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.347+0000 7fde0b1e7640 1 --2- 172.21.15.189:0/2748302788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde04108720 0x7fde0410ef50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009ca0 tx=0x7fddfc02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:05:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.347+0000 7fde0b1e7640 1 -- 172.21.15.189:0/2748302788 shutdown_connections 2024-07-26T11:05:37.352 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.347+0000 7fde0b1e7640 1 --2- 172.21.15.189:0/2748302788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde04107dd0 0x7fde041081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.348+0000 7fde0b1e7640 1 --2- 172.21.15.189:0/2748302788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde04108720 0x7fde0410ef50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.348+0000 7fde0b1e7640 1 --2- 172.21.15.189:0/2748302788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde04102490 0x7fde041028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.348+0000 7fde0b1e7640 1 -- 172.21.15.189:0/2748302788 >> 172.21.15.189:0/2748302788 conn(0x7fde04079190 msgr2=0x7fde040795a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.348+0000 7fde0b1e7640 1 -- 172.21.15.189:0/2748302788 shutdown_connections 2024-07-26T11:05:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.348+0000 7fde0b1e7640 1 -- 172.21.15.189:0/2748302788 wait complete. 2024-07-26T11:05:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.348+0000 7fde0b1e7640 1 Processor -- start 2024-07-26T11:05:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.348+0000 7fde0b1e7640 1 -- start start 2024-07-26T11:05:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.348+0000 7fde0b1e7640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde04102490 0x7fde04199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.348+0000 7fde0b1e7640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde04107dd0 0x7fde0419a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.348+0000 7fde0b1e7640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde04108720 0x7fde0419f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.348+0000 7fde0b1e7640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fde04113540 con 0x7fde04102490 2024-07-26T11:05:37.353 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.348+0000 7fde0b1e7640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fde041133c0 con 0x7fde04108720 2024-07-26T11:05:37.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.349+0000 7fde0b1e7640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fde041136c0 con 0x7fde04107dd0 2024-07-26T11:05:37.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.349+0000 7fde09784640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde04108720 0x7fde0419f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:37.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.349+0000 7fde09784640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde04108720 0x7fde0419f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:56682/0 (socket says 172.21.15.189:56682) 2024-07-26T11:05:37.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.349+0000 7fde09784640 1 -- 172.21.15.189:0/4153559843 learned_addr learned my addr 172.21.15.189:0/4153559843 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-07-26T11:05:37.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.349+0000 7fde03fff640 1 --2- 172.21.15.189:0/4153559843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde04107dd0 0x7fde0419a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:37.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.349+0000 7fde09784640 1 -- 172.21.15.189:0/4153559843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde04107dd0 msgr2=0x7fde0419a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:37.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.349+0000 7fde09784640 1 --2- 172.21.15.189:0/4153559843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde04107dd0 0x7fde0419a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:37.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.349+0000 7fde09784640 1 -- 172.21.15.189:0/4153559843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde04102490 msgr2=0x7fde04199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:37.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.349+0000 7fde09784640 1 --2- 172.21.15.189:0/4153559843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde04102490 0x7fde04199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:37.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.349+0000 7fde09784640 1 -- 172.21.15.189:0/4153559843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde0419fea0 con 0x7fde04108720 2024-07-26T11:05:37.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.349+0000 7fde03fff640 1 --2- 172.21.15.189:0/4153559843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde04107dd0 0x7fde0419a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:05:37.354 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.349+0000 7fde09784640 1 --2- 172.21.15.189:0/4153559843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde04108720 0x7fde0419f770 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fddfc005cc0 tx=0x7fddfc02f380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:37.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.349+0000 7fde01ffb640 1 -- 172.21.15.189:0/4153559843 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc003bf0 con 0x7fde04108720 2024-07-26T11:05:37.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.349+0000 7fde0b1e7640 1 -- 172.21.15.189:0/4153559843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde041a0130 con 0x7fde04108720 2024-07-26T11:05:37.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.349+0000 7fde01ffb640 1 -- 172.21.15.189:0/4153559843 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fddfc003d90 con 0x7fde04108720 2024-07-26T11:05:37.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.349+0000 7fde01ffb640 1 -- 172.21.15.189:0/4153559843 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc02fbc0 con 0x7fde04108720 2024-07-26T11:05:37.355 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.350+0000 7fde0b1e7640 1 -- 172.21.15.189:0/4153559843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde041a68f0 con 0x7fde04108720 2024-07-26T11:05:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.351+0000 7fde0b1e7640 1 -- 172.21.15.189:0/4153559843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde04103710 con 0x7fde04108720 2024-07-26T11:05:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.352+0000 7fde01ffb640 1 -- 172.21.15.189:0/4153559843 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fddfc02fd60 con 0x7fde04108720 2024-07-26T11:05:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.353+0000 7fde01ffb640 1 --2- 172.21.15.189:0/4153559843 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fddf40b3c20 0x7fddf40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:05:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.353+0000 7fde01ffb640 1 -- 172.21.15.189:0/4153559843 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 3884+0+0 (secure 0 0 0) 0x7fddfc1043e0 con 0x7fde04108720 2024-07-26T11:05:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.353+0000 7fde08f83640 1 --2- 172.21.15.189:0/4153559843 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fddf40b3c20 0x7fddf40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:05:37.356 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.354+0000 7fde08f83640 1 --2- 172.21.15.189:0/4153559843 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fddf40b3c20 0x7fddf40b6130 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009710 tx=0x7fddf8006c90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:05:37.358 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.356+0000 7fde01ffb640 1 -- 172.21.15.189:0/4153559843 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fddfc0ccb80 con 0x7fde04108720 2024-07-26T11:05:37.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:37.539+0000 7fde0b1e7640 1 -- 172.21.15.189:0/4153559843 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fde04063230 con 0x7fddf40b3c20 2024-07-26T11:05:37.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:05:37.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:05:37.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:37.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:37.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:37.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:37.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:37.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:37.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:37.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:37.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:05:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:05:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:05:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:05:38.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:38.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:38.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:38.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:38.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:05:38.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:38.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:38.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:38.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:38.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:38 smithi189 ceph-mon[36790]: from='client.24416 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:05:38.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:38 smithi189 ceph-mon[36790]: Detected new or changed devices on smithi134 2024-07-26T11:05:38.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:38 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 8544M 2024-07-26T11:05:38.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:38 smithi189 ceph-mon[36790]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:38 smithi193 ceph-mon[37473]: from='client.24416 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:05:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:38 smithi193 ceph-mon[37473]: Detected new or changed devices on smithi134 2024-07-26T11:05:39.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:38 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 8544M 2024-07-26T11:05:39.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:38 smithi193 ceph-mon[37473]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:38 smithi134 ceph-mon[31966]: from='client.24416 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:05:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:38 smithi134 ceph-mon[31966]: Detected new or changed devices on smithi134 2024-07-26T11:05:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:38 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 8544M 2024-07-26T11:05:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:38 smithi134 ceph-mon[31966]: pgmap v110: 1 pgs: 1 remapped+peering; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:40.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:40 smithi189 ceph-mon[36790]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-07-26T11:05:40.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:40 smithi189 ceph-mon[36790]: from='client.? 172.21.15.189:0/994172109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d70eaae-56cc-4cdc-8f83-b88d2121f0ad"}]: dispatch 2024-07-26T11:05:40.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:40 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d70eaae-56cc-4cdc-8f83-b88d2121f0ad"}]: dispatch 2024-07-26T11:05:40.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:40 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d70eaae-56cc-4cdc-8f83-b88d2121f0ad"}]': finished 2024-07-26T11:05:40.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:40 smithi189 ceph-mon[36790]: osdmap e31: 5 total, 4 up, 5 in 2024-07-26T11:05:40.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T11:05:40.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:40 smithi189 ceph-mon[36790]: from='client.? 172.21.15.189:0/2261604913' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:05:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:40 smithi134 ceph-mon[31966]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-07-26T11:05:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:40 smithi134 ceph-mon[31966]: from='client.? 172.21.15.189:0/994172109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d70eaae-56cc-4cdc-8f83-b88d2121f0ad"}]: dispatch 2024-07-26T11:05:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:40 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d70eaae-56cc-4cdc-8f83-b88d2121f0ad"}]: dispatch 2024-07-26T11:05:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:40 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d70eaae-56cc-4cdc-8f83-b88d2121f0ad"}]': finished 2024-07-26T11:05:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:40 smithi134 ceph-mon[31966]: osdmap e31: 5 total, 4 up, 5 in 2024-07-26T11:05:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T11:05:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:40 smithi134 ceph-mon[31966]: from='client.? 172.21.15.189:0/2261604913' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:05:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:40 smithi193 ceph-mon[37473]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 95 KiB/s, 0 objects/s recovering 2024-07-26T11:05:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:40 smithi193 ceph-mon[37473]: from='client.? 172.21.15.189:0/994172109' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d70eaae-56cc-4cdc-8f83-b88d2121f0ad"}]: dispatch 2024-07-26T11:05:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:40 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d70eaae-56cc-4cdc-8f83-b88d2121f0ad"}]: dispatch 2024-07-26T11:05:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:40 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d70eaae-56cc-4cdc-8f83-b88d2121f0ad"}]': finished 2024-07-26T11:05:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:40 smithi193 ceph-mon[37473]: osdmap e31: 5 total, 4 up, 5 in 2024-07-26T11:05:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T11:05:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:40 smithi193 ceph-mon[37473]: from='client.? 172.21.15.189:0/2261604913' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:05:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:42 smithi134 ceph-mon[31966]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-07-26T11:05:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:42 smithi189 ceph-mon[36790]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-07-26T11:05:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:42 smithi193 ceph-mon[37473]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-07-26T11:05:44.924 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:44 smithi189 ceph-mon[36790]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T11:05:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:44 smithi134 ceph-mon[31966]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T11:05:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:44 smithi193 ceph-mon[37473]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T11:05:46.873 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:46 smithi189 ceph-mon[36790]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:05:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:46 smithi134 ceph-mon[31966]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:05:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:46 smithi193 ceph-mon[37473]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:05:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T11:05:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:47 smithi189 ceph-mon[36790]: Deploying daemon osd.4 on smithi189 2024-07-26T11:05:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T11:05:48.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:48.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:47 smithi134 ceph-mon[31966]: Deploying daemon osd.4 on smithi189 2024-07-26T11:05:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-07-26T11:05:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:47 smithi193 ceph-mon[37473]: Deploying daemon osd.4 on smithi189 2024-07-26T11:05:48.880 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:48 smithi189 ceph-mon[36790]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:05:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:48 smithi134 ceph-mon[31966]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:05:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:48 smithi193 ceph-mon[37473]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:05:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:50 smithi134 ceph-mon[31966]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:50 smithi189 ceph-mon[36790]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:50 smithi193 ceph-mon[37473]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:52 smithi134 ceph-mon[31966]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:53.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:53.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:53.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:53.189 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:52 smithi189 ceph-mon[36790]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:53.189 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:53.189 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:53.189 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:52 smithi193 ceph-mon[37473]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:53.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:54 smithi134 ceph-mon[31966]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:55.189 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:54 smithi189 ceph-mon[36790]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:55.189 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:55.190 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:55.190 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:55.190 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:55.190 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:54 smithi193 ceph-mon[37473]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:05:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:05:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:55.561 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:55.559+0000 7fde01ffb640 1 -- 172.21.15.189:0/4153559843 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fde04063230 con 0x7fddf40b3c20 2024-07-26T11:05:55.561 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 4 on host 'smithi189' 2024-07-26T11:05:55.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:55.561+0000 7fde0b1e7640 1 -- 172.21.15.189:0/4153559843 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fddf40b3c20 msgr2=0x7fddf40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:55.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:55.561+0000 7fde0b1e7640 1 --2- 172.21.15.189:0/4153559843 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fddf40b3c20 0x7fddf40b6130 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fddf8009710 tx=0x7fddf8006c90 comp rx=0 tx=0).stop 2024-07-26T11:05:55.563 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:55.561+0000 7fde0b1e7640 1 -- 172.21.15.189:0/4153559843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde04108720 msgr2=0x7fde0419f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:05:55.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:55.561+0000 7fde0b1e7640 1 --2- 172.21.15.189:0/4153559843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde04108720 0x7fde0419f770 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fddfc005cc0 tx=0x7fddfc02f380 comp rx=0 tx=0).stop 2024-07-26T11:05:55.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:55.561+0000 7fde0b1e7640 1 -- 172.21.15.189:0/4153559843 shutdown_connections 2024-07-26T11:05:55.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:55.561+0000 7fde0b1e7640 1 --2- 172.21.15.189:0/4153559843 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fddf40b3c20 0x7fddf40b6130 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:55.561+0000 7fde0b1e7640 1 --2- 172.21.15.189:0/4153559843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde04107dd0 0x7fde0419a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:55.561+0000 7fde0b1e7640 1 --2- 172.21.15.189:0/4153559843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde04108720 0x7fde0419f770 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:55.561+0000 7fde0b1e7640 1 --2- 172.21.15.189:0/4153559843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fde04102490 0x7fde04199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:05:55.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:55.561+0000 7fde0b1e7640 1 -- 172.21.15.189:0/4153559843 >> 172.21.15.189:0/4153559843 conn(0x7fde04079190 msgr2=0x7fde04105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:05:55.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:55.561+0000 7fde0b1e7640 1 -- 172.21.15.189:0/4153559843 shutdown_connections 2024-07-26T11:05:55.564 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:05:55.561+0000 7fde0b1e7640 1 -- 172.21.15.189:0/4153559843 wait complete. 2024-07-26T11:05:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:56.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:56.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:56.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:56.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:56.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:56.255 DEBUG:teuthology.orchestra.run.smithi189:osd.4> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@osd.4.service 2024-07-26T11:05:56.257 INFO:tasks.cephadm:Deploying osd.5 on smithi193 with /dev/vg_nvme/lv_3... 2024-07-26T11:05:56.258 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-07-26T11:05:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:05:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:56.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:05:56.515 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T11:05:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:56 smithi134 ceph-mon[31966]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:56 smithi134 ceph-mon[31966]: from='osd.4 [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T11:05:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:56 smithi134 ceph-mon[31966]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T11:05:56.968 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:56 smithi193 ceph-mon[37473]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:56.968 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:56 smithi193 ceph-mon[37473]: from='osd.4 [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T11:05:56.968 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:56 smithi193 ceph-mon[37473]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T11:05:57.133 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:56 smithi189 ceph-mon[36790]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:57.133 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:56 smithi189 ceph-mon[36790]: from='osd.4 [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T11:05:57.133 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:56 smithi189 ceph-mon[36790]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-07-26T11:05:58.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:57 smithi134 ceph-mon[31966]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T11:05:58.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:57 smithi134 ceph-mon[31966]: osdmap e32: 5 total, 4 up, 5 in 2024-07-26T11:05:58.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T11:05:58.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:57 smithi134 ceph-mon[31966]: from='osd.4 [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:05:58.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:57 smithi134 ceph-mon[31966]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:05:58.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:57 smithi193 ceph-mon[37473]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T11:05:58.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:57 smithi193 ceph-mon[37473]: osdmap e32: 5 total, 4 up, 5 in 2024-07-26T11:05:58.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T11:05:58.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:57 smithi193 ceph-mon[37473]: from='osd.4 [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:05:58.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:57 smithi193 ceph-mon[37473]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:05:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:57 smithi189 ceph-mon[36790]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-07-26T11:05:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:57 smithi189 ceph-mon[36790]: osdmap e32: 5 total, 4 up, 5 in 2024-07-26T11:05:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T11:05:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:57 smithi189 ceph-mon[36790]: from='osd.4 [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:05:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:57 smithi189 ceph-mon[36790]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:05:58.195 INFO:journalctl@ceph.osd.4.smithi189.stdout:Jul 26 11:05:57 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-4[49157]: 2024-07-26T11:05:57.896+0000 7f0736efe640 -1 osd.4 0 waiting for initial osdmap 2024-07-26T11:05:58.195 INFO:journalctl@ceph.osd.4.smithi189.stdout:Jul 26 11:05:57 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-4[49157]: 2024-07-26T11:05:57.903+0000 7f0732cfc640 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T11:05:58.419 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-07-26T11:05:58.508 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-07-26T11:05:58.521 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-07-26T11:05:58.521 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-07-26T11:05:58.521 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115497 s, 908 MB/s 2024-07-26T11:05:58.521 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-07-26T11:05:59.060 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:58 smithi189 ceph-mon[36790]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:59.061 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:58 smithi189 ceph-mon[36790]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-07-26T11:05:59.061 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:58 smithi189 ceph-mon[36790]: osdmap e33: 5 total, 4 up, 5 in 2024-07-26T11:05:59.061 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T11:05:59.061 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T11:05:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:58 smithi193 ceph-mon[37473]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:58 smithi193 ceph-mon[37473]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-07-26T11:05:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:58 smithi193 ceph-mon[37473]: osdmap e33: 5 total, 4 up, 5 in 2024-07-26T11:05:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T11:05:59.133 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T11:05:59.134 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_3 2024-07-26T11:05:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:58 smithi134 ceph-mon[31966]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:05:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:58 smithi134 ceph-mon[31966]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-07-26T11:05:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:58 smithi134 ceph-mon[31966]: osdmap e33: 5 total, 4 up, 5 in 2024-07-26T11:05:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T11:05:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T11:05:59.434 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T11:06:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:59 smithi193 ceph-mon[37473]: purged_snaps scrub starts 2024-07-26T11:06:00.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:59 smithi193 ceph-mon[37473]: purged_snaps scrub ok 2024-07-26T11:06:00.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:59 smithi193 ceph-mon[37473]: osd.4 [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347] boot 2024-07-26T11:06:00.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:59 smithi193 ceph-mon[37473]: osdmap e34: 5 total, 5 up, 5 in 2024-07-26T11:06:00.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T11:06:00.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:00.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:00.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:00.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:00.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:00.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:00.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:00.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:05:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:59 smithi134 ceph-mon[31966]: purged_snaps scrub starts 2024-07-26T11:06:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:59 smithi134 ceph-mon[31966]: purged_snaps scrub ok 2024-07-26T11:06:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:59 smithi134 ceph-mon[31966]: osd.4 [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347] boot 2024-07-26T11:06:00.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:59 smithi134 ceph-mon[31966]: osdmap e34: 5 total, 5 up, 5 in 2024-07-26T11:06:00.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T11:06:00.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:00.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:00.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:00.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:00.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:00.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:00.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:00.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:05:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:59 smithi189 ceph-mon[36790]: purged_snaps scrub starts 2024-07-26T11:06:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:59 smithi189 ceph-mon[36790]: purged_snaps scrub ok 2024-07-26T11:06:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:59 smithi189 ceph-mon[36790]: osd.4 [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347] boot 2024-07-26T11:06:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:59 smithi189 ceph-mon[36790]: osdmap e34: 5 total, 5 up, 5 in 2024-07-26T11:06:00.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-07-26T11:06:00.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:00.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:00.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:00.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:00.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:00.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:00.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:00.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:05:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:01.117 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.115+0000 7fa6cde5f640 1 -- 172.21.15.193:0/1288622288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6c8103590 msgr2=0x7fa6c81039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:01.117 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.115+0000 7fa6cde5f640 1 --2- 172.21.15.193:0/1288622288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6c8103590 0x7fa6c81039a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8009900 tx=0x7fa6b802a410 comp rx=0 tx=0).stop 2024-07-26T11:06:01.117 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.115+0000 7fa6cde5f640 1 -- 172.21.15.193:0/1288622288 shutdown_connections 2024-07-26T11:06:01.118 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.115+0000 7fa6cde5f640 1 --2- 172.21.15.193:0/1288622288 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c8109820 0x7fa6c8110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:01.118 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.115+0000 7fa6cde5f640 1 --2- 172.21.15.193:0/1288622288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6c8108ed0 0x7fa6c81092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:01.118 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.115+0000 7fa6cde5f640 1 --2- 172.21.15.193:0/1288622288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6c8103590 0x7fa6c81039a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:01.118 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.115+0000 7fa6cde5f640 1 -- 172.21.15.193:0/1288622288 >> 172.21.15.193:0/1288622288 conn(0x7fa6c807b280 msgr2=0x7fa6c807b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:01.118 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.115+0000 7fa6cde5f640 1 -- 172.21.15.193:0/1288622288 shutdown_connections 2024-07-26T11:06:01.118 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.115+0000 7fa6cde5f640 1 -- 172.21.15.193:0/1288622288 wait complete. 2024-07-26T11:06:01.118 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.115+0000 7fa6cde5f640 1 Processor -- start 2024-07-26T11:06:01.118 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6cde5f640 1 -- start start 2024-07-26T11:06:01.120 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6cde5f640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6c8103590 0x7fa6c8199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:01.120 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6cde5f640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c8108ed0 0x7fa6c819a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:01.120 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6cde5f640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6c8109820 0x7fa6c819f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:01.120 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6cde5f640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c8113590 con 0x7fa6c8108ed0 2024-07-26T11:06:01.121 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6cde5f640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c8113410 con 0x7fa6c8109820 2024-07-26T11:06:01.121 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6cde5f640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c8113710 con 0x7fa6c8103590 2024-07-26T11:06:01.121 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6c77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6c8103590 0x7fa6c8199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:01.121 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6c77fe640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6c8103590 0x7fa6c8199f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:50276/0 (socket says 172.21.15.193:50276) 2024-07-26T11:06:01.121 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6c77fe640 1 -- 172.21.15.193:0/2912708487 learned_addr learned my addr 172.21.15.193:0/2912708487 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-07-26T11:06:01.121 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6c77fe640 1 -- 172.21.15.193:0/2912708487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6c8109820 msgr2=0x7fa6c819f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:01.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6c7fff640 1 --2- 172.21.15.193:0/2912708487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6c8109820 0x7fa6c819f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:01.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6c77fe640 1 --2- 172.21.15.193:0/2912708487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6c8109820 0x7fa6c819f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:01.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6c77fe640 1 -- 172.21.15.193:0/2912708487 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c8108ed0 msgr2=0x7fa6c819a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:01.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6c77fe640 1 --2- 172.21.15.193:0/2912708487 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c8108ed0 0x7fa6c819a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:01.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6c77fe640 1 -- 172.21.15.193:0/2912708487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6c819fe70 con 0x7fa6c8103590 2024-07-26T11:06:01.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6c7fff640 1 --2- 172.21.15.193:0/2912708487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6c8109820 0x7fa6c819f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:06:01.122 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.116+0000 7fa6c77fe640 1 --2- 172.21.15.193:0/2912708487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6c8103590 0x7fa6c8199f00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8005ab0 tx=0x7fa6b8003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:01.123 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.117+0000 7fa6c4ff9640 1 -- 172.21.15.193:0/2912708487 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b803f070 con 0x7fa6c8103590 2024-07-26T11:06:01.123 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.117+0000 7fa6c4ff9640 1 -- 172.21.15.193:0/2912708487 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6b8031d60 con 0x7fa6c8103590 2024-07-26T11:06:01.123 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.117+0000 7fa6c4ff9640 1 -- 172.21.15.193:0/2912708487 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b8035030 con 0x7fa6c8103590 2024-07-26T11:06:01.123 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.117+0000 7fa6cde5f640 1 -- 172.21.15.193:0/2912708487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6c81a0100 con 0x7fa6c8103590 2024-07-26T11:06:01.123 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.117+0000 7fa6cde5f640 1 -- 172.21.15.193:0/2912708487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6c81a68c0 con 0x7fa6c8103590 2024-07-26T11:06:01.123 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.118+0000 7fa6cde5f640 1 -- 172.21.15.193:0/2912708487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6c81039a0 con 0x7fa6c8103590 2024-07-26T11:06:01.124 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.119+0000 7fa6c4ff9640 1 -- 172.21.15.193:0/2912708487 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fa6b8042070 con 0x7fa6c8103590 2024-07-26T11:06:01.124 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.120+0000 7fa6c4ff9640 1 --2- 172.21.15.193:0/2912708487 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6b40b3c20 0x7fa6b40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:01.124 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.120+0000 7fa6c4ff9640 1 -- 172.21.15.193:0/2912708487 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4316+0+0 (secure 0 0 0) 0x7fa6b8103660 con 0x7fa6c8103590 2024-07-26T11:06:01.124 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.120+0000 7fa6c6ffd640 1 --2- 172.21.15.193:0/2912708487 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6b40b3c20 0x7fa6b40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:01.124 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:00 smithi193 ceph-mon[37473]: Detected new or changed devices on smithi189 2024-07-26T11:06:01.124 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:00 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 8544M 2024-07-26T11:06:01.124 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:00 smithi193 ceph-mon[37473]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:06:01.124 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:00 smithi193 ceph-mon[37473]: osdmap e35: 5 total, 5 up, 5 in 2024-07-26T11:06:01.125 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.121+0000 7fa6c6ffd640 1 --2- 172.21.15.193:0/2912708487 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6b40b3c20 0x7fa6b40b6130 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0005fd0 tx=0x7fa6b0005850 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:01.128 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.126+0000 7fa6c4ff9640 1 -- 172.21.15.193:0/2912708487 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa6b80cbc50 con 0x7fa6c8103590 2024-07-26T11:06:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:00 smithi189 ceph-mon[36790]: Detected new or changed devices on smithi189 2024-07-26T11:06:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:00 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 8544M 2024-07-26T11:06:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:00 smithi189 ceph-mon[36790]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:06:01.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:00 smithi189 ceph-mon[36790]: osdmap e35: 5 total, 5 up, 5 in 2024-07-26T11:06:01.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:01.306+0000 7fa6cde5f640 1 -- 172.21.15.193:0/2912708487 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fa6c8063230 con 0x7fa6b40b3c20 2024-07-26T11:06:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:00 smithi134 ceph-mon[31966]: Detected new or changed devices on smithi189 2024-07-26T11:06:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:00 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 8544M 2024-07-26T11:06:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:00 smithi134 ceph-mon[31966]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-07-26T11:06:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:00 smithi134 ceph-mon[31966]: osdmap e35: 5 total, 5 up, 5 in 2024-07-26T11:06:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:06:02.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:06:02.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:06:02.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:06:02.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:06:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:06:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:02 smithi193 ceph-mon[37473]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:06:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:02 smithi193 ceph-mon[37473]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:02 smithi134 ceph-mon[31966]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:06:03.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:02 smithi134 ceph-mon[31966]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:02 smithi189 ceph-mon[36790]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:06:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:02 smithi189 ceph-mon[36790]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:03 smithi193 ceph-mon[37473]: from='client.? 172.21.15.193:0/1167593656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9b0b4cf-1608-45ed-b64e-afcf545d97fa"}]: dispatch 2024-07-26T11:06:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:03 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9b0b4cf-1608-45ed-b64e-afcf545d97fa"}]: dispatch 2024-07-26T11:06:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:03 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9b0b4cf-1608-45ed-b64e-afcf545d97fa"}]': finished 2024-07-26T11:06:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:03 smithi193 ceph-mon[37473]: osdmap e36: 6 total, 5 up, 6 in 2024-07-26T11:06:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:03 smithi134 ceph-mon[31966]: from='client.? 172.21.15.193:0/1167593656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9b0b4cf-1608-45ed-b64e-afcf545d97fa"}]: dispatch 2024-07-26T11:06:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:03 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9b0b4cf-1608-45ed-b64e-afcf545d97fa"}]: dispatch 2024-07-26T11:06:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:03 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9b0b4cf-1608-45ed-b64e-afcf545d97fa"}]': finished 2024-07-26T11:06:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:03 smithi134 ceph-mon[31966]: osdmap e36: 6 total, 5 up, 6 in 2024-07-26T11:06:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:03 smithi189 ceph-mon[36790]: from='client.? 172.21.15.193:0/1167593656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9b0b4cf-1608-45ed-b64e-afcf545d97fa"}]: dispatch 2024-07-26T11:06:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:03 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a9b0b4cf-1608-45ed-b64e-afcf545d97fa"}]: dispatch 2024-07-26T11:06:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:03 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a9b0b4cf-1608-45ed-b64e-afcf545d97fa"}]': finished 2024-07-26T11:06:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:03 smithi189 ceph-mon[36790]: osdmap e36: 6 total, 5 up, 6 in 2024-07-26T11:06:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:04 smithi193 ceph-mon[37473]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:05.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:04 smithi193 ceph-mon[37473]: from='client.? 172.21.15.193:0/3996891073' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:06:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:04 smithi134 ceph-mon[31966]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:05.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:04 smithi134 ceph-mon[31966]: from='client.? 172.21.15.193:0/3996891073' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:06:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:04 smithi189 ceph-mon[36790]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:04 smithi189 ceph-mon[36790]: from='client.? 172.21.15.193:0/3996891073' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:06:07.247 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:06 smithi193 ceph-mon[37473]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:06 smithi134 ceph-mon[31966]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:06 smithi189 ceph-mon[36790]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:08 smithi193 ceph-mon[37473]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:08 smithi134 ceph-mon[31966]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:08 smithi189 ceph-mon[36790]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:11.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:10 smithi189 ceph-mon[36790]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:11.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T11:06:11.247 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:11.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:10 smithi193 ceph-mon[37473]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:11.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T11:06:11.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:10 smithi134 ceph-mon[31966]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-07-26T11:06:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:12.184 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:11 smithi193 ceph-mon[37473]: Deploying daemon osd.5 on smithi193 2024-07-26T11:06:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:11 smithi134 ceph-mon[31966]: Deploying daemon osd.5 on smithi193 2024-07-26T11:06:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:11 smithi189 ceph-mon[36790]: Deploying daemon osd.5 on smithi193 2024-07-26T11:06:13.266 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:12 smithi193 ceph-mon[37473]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:12 smithi134 ceph-mon[31966]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:12 smithi189 ceph-mon[36790]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:14.244 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:14 smithi193 ceph-mon[37473]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:14 smithi134 ceph-mon[31966]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:14 smithi189 ceph-mon[36790]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:16.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:16.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:16.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:16.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:17.116 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:16 smithi193 ceph-mon[37473]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:16 smithi134 ceph-mon[31966]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:16 smithi189 ceph-mon[36790]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:18.971 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:18 smithi193 ceph-mon[37473]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:18.971 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:18.971 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:18.971 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:18.971 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:18.971 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:18 smithi134 ceph-mon[31966]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:18 smithi189 ceph-mon[36790]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:19.333 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:19.331+0000 7fa6c4ff9640 1 -- 172.21.15.193:0/2912708487 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa6c8063230 con 0x7fa6b40b3c20 2024-07-26T11:06:19.334 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 5 on host 'smithi193' 2024-07-26T11:06:19.336 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:19.334+0000 7fa6cde5f640 1 -- 172.21.15.193:0/2912708487 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6b40b3c20 msgr2=0x7fa6b40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:19.336 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:19.334+0000 7fa6cde5f640 1 --2- 172.21.15.193:0/2912708487 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6b40b3c20 0x7fa6b40b6130 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0005fd0 tx=0x7fa6b0005850 comp rx=0 tx=0).stop 2024-07-26T11:06:19.337 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:19.334+0000 7fa6cde5f640 1 -- 172.21.15.193:0/2912708487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6c8103590 msgr2=0x7fa6c8199f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:19.337 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:19.334+0000 7fa6cde5f640 1 --2- 172.21.15.193:0/2912708487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6c8103590 0x7fa6c8199f00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa6b8005ab0 tx=0x7fa6b8003cd0 comp rx=0 tx=0).stop 2024-07-26T11:06:19.337 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:19.335+0000 7fa6cde5f640 1 -- 172.21.15.193:0/2912708487 shutdown_connections 2024-07-26T11:06:19.337 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:19.335+0000 7fa6cde5f640 1 --2- 172.21.15.193:0/2912708487 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6b40b3c20 0x7fa6b40b6130 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:19.338 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:19.335+0000 7fa6cde5f640 1 --2- 172.21.15.193:0/2912708487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6c8109820 0x7fa6c819f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:19.338 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:19.335+0000 7fa6cde5f640 1 --2- 172.21.15.193:0/2912708487 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6c8108ed0 0x7fa6c819a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:19.338 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:19.335+0000 7fa6cde5f640 1 --2- 172.21.15.193:0/2912708487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6c8103590 0x7fa6c8199f00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:19.338 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:19.335+0000 7fa6cde5f640 1 -- 172.21.15.193:0/2912708487 >> 172.21.15.193:0/2912708487 conn(0x7fa6c807b280 msgr2=0x7fa6c8106fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:19.338 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:19.335+0000 7fa6cde5f640 1 -- 172.21.15.193:0/2912708487 shutdown_connections 2024-07-26T11:06:19.338 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:06:19.335+0000 7fa6cde5f640 1 -- 172.21.15.193:0/2912708487 wait complete. 2024-07-26T11:06:20.017 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:20.017 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:20.017 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:20.017 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:19 smithi193 ceph-mon[37473]: from='osd.5 [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T11:06:20.017 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:19 smithi193 ceph-mon[37473]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T11:06:20.047 DEBUG:teuthology.orchestra.run.smithi193:osd.5> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@osd.5.service 2024-07-26T11:06:20.049 INFO:tasks.cephadm:Deploying osd.6 on smithi134 with /dev/vg_nvme/lv_2... 2024-07-26T11:06:20.049 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T11:06:20.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:20.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:20.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:20.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:19 smithi134 ceph-mon[31966]: from='osd.5 [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T11:06:20.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:19 smithi134 ceph-mon[31966]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T11:06:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:19 smithi189 ceph-mon[36790]: from='osd.5 [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T11:06:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:19 smithi189 ceph-mon[36790]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-07-26T11:06:20.502 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:06:20.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:20 smithi134 ceph-mon[31966]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:20.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:20 smithi134 ceph-mon[31966]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T11:06:20.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:20 smithi134 ceph-mon[31966]: from='osd.5 [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:06:20.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:20 smithi134 ceph-mon[31966]: osdmap e37: 6 total, 5 up, 6 in 2024-07-26T11:06:20.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:20.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:20 smithi134 ceph-mon[31966]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:06:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:20 smithi189 ceph-mon[36790]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:20 smithi189 ceph-mon[36790]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T11:06:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:20 smithi189 ceph-mon[36790]: from='osd.5 [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:06:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:20 smithi189 ceph-mon[36790]: osdmap e37: 6 total, 5 up, 6 in 2024-07-26T11:06:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:20 smithi189 ceph-mon[36790]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:06:21.225 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:20 smithi193 ceph-mon[37473]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:21.225 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:20 smithi193 ceph-mon[37473]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-07-26T11:06:21.225 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:20 smithi193 ceph-mon[37473]: from='osd.5 [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:06:21.225 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:20 smithi193 ceph-mon[37473]: osdmap e37: 6 total, 5 up, 6 in 2024-07-26T11:06:21.225 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:21.225 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:20 smithi193 ceph-mon[37473]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:06:22.351 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jul 26 11:06:22 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-5[50296]: 2024-07-26T11:06:22.055+0000 7f17cbb26640 -1 osd.5 0 waiting for initial osdmap 2024-07-26T11:06:22.351 INFO:journalctl@ceph.osd.5.smithi193.stdout:Jul 26 11:06:22 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-5[50296]: 2024-07-26T11:06:22.062+0000 7f17c391c640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T11:06:22.598 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:22 smithi134 ceph-mon[31966]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-07-26T11:06:22.598 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:22 smithi134 ceph-mon[31966]: osdmap e38: 6 total, 5 up, 6 in 2024-07-26T11:06:22.598 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:22.598 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:22.598 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:22 smithi134 ceph-mon[31966]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:22.598 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:22 smithi134 ceph-mon[31966]: from='osd.5 ' entity='osd.5' 2024-07-26T11:06:22.615 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:22 smithi193 ceph-mon[37473]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-07-26T11:06:22.615 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:22 smithi193 ceph-mon[37473]: osdmap e38: 6 total, 5 up, 6 in 2024-07-26T11:06:22.615 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:22.615 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:22.615 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:22 smithi193 ceph-mon[37473]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:22.615 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:22 smithi193 ceph-mon[37473]: from='osd.5 ' entity='osd.5' 2024-07-26T11:06:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:22 smithi189 ceph-mon[36790]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-07-26T11:06:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:22 smithi189 ceph-mon[36790]: osdmap e38: 6 total, 5 up, 6 in 2024-07-26T11:06:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:22 smithi189 ceph-mon[36790]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-07-26T11:06:22.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:22 smithi189 ceph-mon[36790]: from='osd.5 ' entity='osd.5' 2024-07-26T11:06:23.087 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T11:06:23.253 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T11:06:23.291 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-07-26T11:06:23.291 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-07-26T11:06:23.291 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0316199 s, 332 MB/s 2024-07-26T11:06:23.292 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-07-26T11:06:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:23 smithi134 ceph-mon[31966]: purged_snaps scrub starts 2024-07-26T11:06:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:23 smithi134 ceph-mon[31966]: purged_snaps scrub ok 2024-07-26T11:06:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:23 smithi134 ceph-mon[31966]: osd.5 [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797] boot 2024-07-26T11:06:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:23 smithi134 ceph-mon[31966]: osdmap e39: 6 total, 6 up, 6 in 2024-07-26T11:06:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:23 smithi134 ceph-mon[31966]: osdmap e40: 6 total, 6 up, 6 in 2024-07-26T11:06:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:23 smithi134 ceph-mon[31966]: Detected new or changed devices on smithi193 2024-07-26T11:06:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:23.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:23 smithi189 ceph-mon[36790]: purged_snaps scrub starts 2024-07-26T11:06:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:23 smithi189 ceph-mon[36790]: purged_snaps scrub ok 2024-07-26T11:06:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:23 smithi189 ceph-mon[36790]: osd.5 [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797] boot 2024-07-26T11:06:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:23 smithi189 ceph-mon[36790]: osdmap e39: 6 total, 6 up, 6 in 2024-07-26T11:06:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:23.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:23 smithi189 ceph-mon[36790]: osdmap e40: 6 total, 6 up, 6 in 2024-07-26T11:06:23.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:23 smithi189 ceph-mon[36790]: Detected new or changed devices on smithi193 2024-07-26T11:06:23.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:23.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:23.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:23.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:23.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:23.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:23.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:23.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:23 smithi193 ceph-mon[37473]: purged_snaps scrub starts 2024-07-26T11:06:23.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:23 smithi193 ceph-mon[37473]: purged_snaps scrub ok 2024-07-26T11:06:23.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:23.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:23 smithi193 ceph-mon[37473]: osd.5 [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797] boot 2024-07-26T11:06:23.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:23 smithi193 ceph-mon[37473]: osdmap e39: 6 total, 6 up, 6 in 2024-07-26T11:06:23.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-07-26T11:06:23.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:23 smithi193 ceph-mon[37473]: osdmap e40: 6 total, 6 up, 6 in 2024-07-26T11:06:23.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:23 smithi193 ceph-mon[37473]: Detected new or changed devices on smithi193 2024-07-26T11:06:23.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:23.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:23.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:23.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:06:23.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:23.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:23.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:23.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:24.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi134:vg_nvme/lv_2 2024-07-26T11:06:24.625 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:06:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:24 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 8544M 2024-07-26T11:06:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:24 smithi134 ceph-mon[31966]: osdmap e41: 6 total, 6 up, 6 in 2024-07-26T11:06:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:24 smithi134 ceph-mon[31966]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:24 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 8544M 2024-07-26T11:06:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:24 smithi193 ceph-mon[37473]: osdmap e41: 6 total, 6 up, 6 in 2024-07-26T11:06:25.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:24 smithi193 ceph-mon[37473]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:24 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 8544M 2024-07-26T11:06:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:24 smithi189 ceph-mon[36790]: osdmap e41: 6 total, 6 up, 6 in 2024-07-26T11:06:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:24 smithi189 ceph-mon[36790]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:27.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.033+0000 7f788f200640 1 -- 172.21.15.134:0/1738210823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78881099f0 msgr2=0x7f7888109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:27.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.033+0000 7f788f200640 1 --2- 172.21.15.134:0/1738210823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78881099f0 0x7f7888109e00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f787c009900 tx=0x7f787c02a410 comp rx=0 tx=0).stop 2024-07-26T11:06:27.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.033+0000 7f788f200640 1 -- 172.21.15.134:0/1738210823 shutdown_connections 2024-07-26T11:06:27.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.033+0000 7f788f200640 1 --2- 172.21.15.134:0/1738210823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f788810fc80 0x7f78881164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:27.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.033+0000 7f788f200640 1 --2- 172.21.15.134:0/1738210823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f788810f330 0x7f788810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:27.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.033+0000 7f788f200640 1 --2- 172.21.15.134:0/1738210823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f78881099f0 0x7f7888109e00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:27.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.033+0000 7f788f200640 1 -- 172.21.15.134:0/1738210823 >> 172.21.15.134:0/1738210823 conn(0x7f7888078ff0 msgr2=0x7f7888107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:27.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.033+0000 7f788f200640 1 -- 172.21.15.134:0/1738210823 shutdown_connections 2024-07-26T11:06:27.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.033+0000 7f788f200640 1 -- 172.21.15.134:0/1738210823 wait complete. 2024-07-26T11:06:27.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.034+0000 7f788f200640 1 Processor -- start 2024-07-26T11:06:27.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.034+0000 7f788f200640 1 -- start start 2024-07-26T11:06:27.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.035+0000 7f788f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78881099f0 0x7f78881a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:27.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.035+0000 7f788f200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f788810f330 0x7f78881a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:27.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.035+0000 7f788f200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f788810fc80 0x7f78881a5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:27.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.035+0000 7f788f200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7888119990 con 0x7f788810fc80 2024-07-26T11:06:27.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.035+0000 7f788f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7888119810 con 0x7f78881099f0 2024-07-26T11:06:27.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.035+0000 7f788f200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7888119b10 con 0x7f788810f330 2024-07-26T11:06:27.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.035+0000 7f788e800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f788810fc80 0x7f78881a5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:27.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.035+0000 7f788de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78881099f0 0x7f78881a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:27.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.035+0000 7f788de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78881099f0 0x7f78881a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:48814/0 (socket says 172.21.15.134:48814) 2024-07-26T11:06:27.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.035+0000 7f788de00640 1 -- 172.21.15.134:0/810319377 learned_addr learned my addr 172.21.15.134:0/810319377 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:06:27.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.035+0000 7f788d400640 1 --2- 172.21.15.134:0/810319377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f788810f330 0x7f78881a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:27.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.036+0000 7f788de00640 1 -- 172.21.15.134:0/810319377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f788810f330 msgr2=0x7f78881a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:27.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.036+0000 7f788de00640 1 --2- 172.21.15.134:0/810319377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f788810f330 0x7f78881a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:27.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.036+0000 7f788de00640 1 -- 172.21.15.134:0/810319377 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f788810fc80 msgr2=0x7f78881a5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:27.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.036+0000 7f788de00640 1 --2- 172.21.15.134:0/810319377 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f788810fc80 0x7f78881a5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:27.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.036+0000 7f788de00640 1 -- 172.21.15.134:0/810319377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78881a6350 con 0x7f78881099f0 2024-07-26T11:06:27.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.036+0000 7f788de00640 1 --2- 172.21.15.134:0/810319377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78881099f0 0x7f78881a03e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f787c005ab0 tx=0x7f787c003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:27.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.036+0000 7f7882a00640 1 -- 172.21.15.134:0/810319377 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c03f070 con 0x7f78881099f0 2024-07-26T11:06:27.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.036+0000 7f7882a00640 1 -- 172.21.15.134:0/810319377 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f787c031b60 con 0x7f78881099f0 2024-07-26T11:06:27.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.036+0000 7f7882a00640 1 -- 172.21.15.134:0/810319377 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c035030 con 0x7f78881099f0 2024-07-26T11:06:27.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.036+0000 7f788f200640 1 -- 172.21.15.134:0/810319377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78881a65e0 con 0x7f78881099f0 2024-07-26T11:06:27.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.037+0000 7f788f200640 1 -- 172.21.15.134:0/810319377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78881acda0 con 0x7f78881099f0 2024-07-26T11:06:27.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.038+0000 7f788f200640 1 -- 172.21.15.134:0/810319377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f788810ac70 con 0x7f78881099f0 2024-07-26T11:06:27.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.040+0000 7f7882a00640 1 -- 172.21.15.134:0/810319377 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f787c042070 con 0x7f78881099f0 2024-07-26T11:06:27.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.041+0000 7f7882a00640 1 --2- 172.21.15.134:0/810319377 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f78700b3ca0 0x7f78700b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:06:27.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.042+0000 7f7882a00640 1 -- 172.21.15.134:0/810319377 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4748+0+0 (secure 0 0 0) 0x7f787c103a00 con 0x7f78881099f0 2024-07-26T11:06:27.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.042+0000 7f788d400640 1 --2- 172.21.15.134:0/810319377 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f78700b3ca0 0x7f78700b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:06:27.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.042+0000 7f788d400640 1 --2- 172.21.15.134:0/810319377 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f78700b3ca0 0x7f78700b61b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f788810fa50 tx=0x7f7878005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:06:27.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.043+0000 7f7882a00640 1 -- 172.21.15.134:0/810319377 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f787c0cb750 con 0x7f78881099f0 2024-07-26T11:06:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:26 smithi134 ceph-mon[31966]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-07-26T11:06:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:26 smithi189 ceph-mon[36790]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-07-26T11:06:27.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:27.249+0000 7f788f200640 1 -- 172.21.15.134:0/810319377 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7888063230 con 0x7f78700b3ca0 2024-07-26T11:06:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:26 smithi193 ceph-mon[37473]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-07-26T11:06:28.142 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:06:28.142 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:06:28.142 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:06:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:06:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:06:28.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:06:28.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:28 smithi134 ceph-mon[31966]: from='client.24452 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:06:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:28 smithi134 ceph-mon[31966]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T11:06:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:28 smithi189 ceph-mon[36790]: from='client.24452 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:06:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:28 smithi189 ceph-mon[36790]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T11:06:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:28 smithi193 ceph-mon[37473]: from='client.24452 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:06:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:28 smithi193 ceph-mon[37473]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T11:06:31.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:30 smithi134 ceph-mon[31966]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-07-26T11:06:31.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:30 smithi189 ceph-mon[36790]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-07-26T11:06:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:30 smithi193 ceph-mon[37473]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-07-26T11:06:32.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:31 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3214480778' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04"}]: dispatch 2024-07-26T11:06:32.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:31 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04"}]: dispatch 2024-07-26T11:06:32.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:31 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04"}]': finished 2024-07-26T11:06:32.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:31 smithi134 ceph-mon[31966]: osdmap e42: 7 total, 6 up, 7 in 2024-07-26T11:06:32.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T11:06:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:31 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3214480778' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04"}]: dispatch 2024-07-26T11:06:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:31 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04"}]: dispatch 2024-07-26T11:06:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:31 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04"}]': finished 2024-07-26T11:06:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:31 smithi189 ceph-mon[36790]: osdmap e42: 7 total, 6 up, 7 in 2024-07-26T11:06:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T11:06:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:31 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3214480778' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04"}]: dispatch 2024-07-26T11:06:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:31 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04"}]: dispatch 2024-07-26T11:06:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:31 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04"}]': finished 2024-07-26T11:06:32.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:31 smithi193 ceph-mon[37473]: osdmap e42: 7 total, 6 up, 7 in 2024-07-26T11:06:32.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T11:06:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:32 smithi134 ceph-mon[31966]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T11:06:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:32 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/792470963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:06:33.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:32 smithi189 ceph-mon[36790]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T11:06:33.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:32 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/792470963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:06:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:32 smithi193 ceph-mon[37473]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T11:06:33.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:32 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/792470963' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:06:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:34 smithi134 ceph-mon[31966]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:06:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:34 smithi189 ceph-mon[36790]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:06:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:34 smithi193 ceph-mon[37473]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:06:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:36 smithi134 ceph-mon[31966]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:36 smithi189 ceph-mon[36790]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:36 smithi193 ceph-mon[37473]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:39.170 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:38 smithi134 ceph-mon[31966]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:38 smithi189 ceph-mon[36790]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:38 smithi193 ceph-mon[37473]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:40 smithi134 ceph-mon[31966]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T11:06:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:41.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:40 smithi189 ceph-mon[36790]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:41.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T11:06:41.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:40 smithi193 ceph-mon[37473]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-07-26T11:06:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:41 smithi134 ceph-mon[31966]: Deploying daemon osd.6 on smithi134 2024-07-26T11:06:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:41 smithi189 ceph-mon[36790]: Deploying daemon osd.6 on smithi134 2024-07-26T11:06:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:41 smithi193 ceph-mon[37473]: Deploying daemon osd.6 on smithi134 2024-07-26T11:06:43.154 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:42 smithi134 ceph-mon[31966]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:42 smithi189 ceph-mon[36790]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:42 smithi193 ceph-mon[37473]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:44.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:44 smithi134 ceph-mon[31966]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:44 smithi189 ceph-mon[36790]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:44 smithi193 ceph-mon[37473]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:46 smithi134 ceph-mon[31966]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:46 smithi189 ceph-mon[36790]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:46 smithi193 ceph-mon[37473]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:49.165 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:48 smithi134 ceph-mon[31966]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:48 smithi189 ceph-mon[36790]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:48 smithi193 ceph-mon[37473]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:50.100 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:50.101 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:50.101 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:50 smithi134 ceph-mon[31966]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:50 smithi189 ceph-mon[36790]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:50 smithi193 ceph-mon[37473]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:52 smithi189 ceph-mon[36790]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:52 smithi193 ceph-mon[37473]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:53.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:53.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:52 smithi134 ceph-mon[31966]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:54.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:54.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:54.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:54.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:53 smithi134 ceph-mon[31966]: from='osd.6 [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T11:06:54.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:53 smithi134 ceph-mon[31966]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T11:06:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:53 smithi189 ceph-mon[36790]: from='osd.6 [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T11:06:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:53 smithi189 ceph-mon[36790]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T11:06:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:06:54.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:06:54.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:54.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:53 smithi193 ceph-mon[37473]: from='osd.6 [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T11:06:54.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:53 smithi193 ceph-mon[37473]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-07-26T11:06:54.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:54.824+0000 7f7882a00640 1 -- 172.21.15.134:0/810319377 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7888063230 con 0x7f78700b3ca0 2024-07-26T11:06:54.828 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 6 on host 'smithi134' 2024-07-26T11:06:54.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:54.828+0000 7f788f200640 1 -- 172.21.15.134:0/810319377 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f78700b3ca0 msgr2=0x7f78700b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:54.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:54.828+0000 7f788f200640 1 --2- 172.21.15.134:0/810319377 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f78700b3ca0 0x7f78700b61b0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f788810fa50 tx=0x7f7878005f90 comp rx=0 tx=0).stop 2024-07-26T11:06:54.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:54.828+0000 7f788f200640 1 -- 172.21.15.134:0/810319377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78881099f0 msgr2=0x7f78881a03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:06:54.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:54.828+0000 7f788f200640 1 --2- 172.21.15.134:0/810319377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78881099f0 0x7f78881a03e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f787c005ab0 tx=0x7f787c003c10 comp rx=0 tx=0).stop 2024-07-26T11:06:54.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:54.828+0000 7f788f200640 1 -- 172.21.15.134:0/810319377 shutdown_connections 2024-07-26T11:06:54.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:54.828+0000 7f788f200640 1 --2- 172.21.15.134:0/810319377 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f78700b3ca0 0x7f78700b61b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:54.828+0000 7f788f200640 1 --2- 172.21.15.134:0/810319377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f788810f330 0x7f78881a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:54.828+0000 7f788f200640 1 --2- 172.21.15.134:0/810319377 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f788810fc80 0x7f78881a5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:54.828+0000 7f788f200640 1 --2- 172.21.15.134:0/810319377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78881099f0 0x7f78881a03e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:06:54.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:54.828+0000 7f788f200640 1 -- 172.21.15.134:0/810319377 >> 172.21.15.134:0/810319377 conn(0x7f7888078ff0 msgr2=0x7f7888107f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:06:54.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:54.829+0000 7f788f200640 1 -- 172.21.15.134:0/810319377 shutdown_connections 2024-07-26T11:06:54.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:06:54.829+0000 7f788f200640 1 -- 172.21.15.134:0/810319377 wait complete. 2024-07-26T11:06:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:54 smithi134 ceph-mon[31966]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:54 smithi134 ceph-mon[31966]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T11:06:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:54 smithi134 ceph-mon[31966]: from='osd.6 [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:06:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:54 smithi134 ceph-mon[31966]: osdmap e43: 7 total, 6 up, 7 in 2024-07-26T11:06:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T11:06:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:54 smithi134 ceph-mon[31966]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:06:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:54 smithi189 ceph-mon[36790]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:54 smithi189 ceph-mon[36790]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T11:06:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:54 smithi189 ceph-mon[36790]: from='osd.6 [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:06:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:54 smithi189 ceph-mon[36790]: osdmap e43: 7 total, 6 up, 7 in 2024-07-26T11:06:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T11:06:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:54 smithi189 ceph-mon[36790]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:06:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:54 smithi193 ceph-mon[37473]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:54 smithi193 ceph-mon[37473]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-07-26T11:06:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:54 smithi193 ceph-mon[37473]: from='osd.6 [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:06:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:54 smithi193 ceph-mon[37473]: osdmap e43: 7 total, 6 up, 7 in 2024-07-26T11:06:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T11:06:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:54 smithi193 ceph-mon[37473]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:06:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:06:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:06:55.775 DEBUG:teuthology.orchestra.run.smithi134:osd.6> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@osd.6.service 2024-07-26T11:06:55.781 INFO:tasks.cephadm:Deploying osd.7 on smithi189 with /dev/vg_nvme/lv_2... 2024-07-26T11:06:55.781 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T11:06:56.039 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T11:06:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:56 smithi193 ceph-mon[37473]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-07-26T11:06:57.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:56 smithi193 ceph-mon[37473]: osdmap e44: 7 total, 6 up, 7 in 2024-07-26T11:06:57.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T11:06:57.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T11:06:57.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:56 smithi193 ceph-mon[37473]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:57.133 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:56 smithi189 ceph-mon[36790]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-07-26T11:06:57.133 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:56 smithi189 ceph-mon[36790]: osdmap e44: 7 total, 6 up, 7 in 2024-07-26T11:06:57.133 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T11:06:57.133 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T11:06:57.133 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:56 smithi189 ceph-mon[36790]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:56 smithi134 ceph-mon[31966]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-07-26T11:06:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:56 smithi134 ceph-mon[31966]: osdmap e44: 7 total, 6 up, 7 in 2024-07-26T11:06:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T11:06:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T11:06:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:56 smithi134 ceph-mon[31966]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-07-26T11:06:57.830 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T11:06:57.912 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:57 smithi134 ceph-mon[31966]: purged_snaps scrub starts 2024-07-26T11:06:57.912 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:57 smithi134 ceph-mon[31966]: purged_snaps scrub ok 2024-07-26T11:06:57.912 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:57 smithi134 ceph-mon[31966]: osd.6 [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438] boot 2024-07-26T11:06:57.912 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:57 smithi134 ceph-mon[31966]: osdmap e45: 7 total, 7 up, 7 in 2024-07-26T11:06:57.912 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T11:06:57.917 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T11:06:57.933 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-07-26T11:06:57.933 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-07-26T11:06:57.934 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.014692 s, 714 MB/s 2024-07-26T11:06:57.934 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-07-26T11:06:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:57 smithi193 ceph-mon[37473]: purged_snaps scrub starts 2024-07-26T11:06:58.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:57 smithi193 ceph-mon[37473]: purged_snaps scrub ok 2024-07-26T11:06:58.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:57 smithi193 ceph-mon[37473]: osd.6 [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438] boot 2024-07-26T11:06:58.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:57 smithi193 ceph-mon[37473]: osdmap e45: 7 total, 7 up, 7 in 2024-07-26T11:06:58.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T11:06:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:57 smithi189 ceph-mon[36790]: purged_snaps scrub starts 2024-07-26T11:06:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:57 smithi189 ceph-mon[36790]: purged_snaps scrub ok 2024-07-26T11:06:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:57 smithi189 ceph-mon[36790]: osd.6 [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438] boot 2024-07-26T11:06:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:57 smithi189 ceph-mon[36790]: osdmap e45: 7 total, 7 up, 7 in 2024-07-26T11:06:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-07-26T11:06:58.608 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_2 2024-07-26T11:06:58.872 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T11:06:58.893 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:58 smithi189 ceph-mon[36790]: osdmap e46: 7 total, 7 up, 7 in 2024-07-26T11:06:58.893 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:06:58 smithi189 ceph-mon[36790]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:06:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:58 smithi193 ceph-mon[37473]: osdmap e46: 7 total, 7 up, 7 in 2024-07-26T11:06:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:06:58 smithi193 ceph-mon[37473]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:06:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:58 smithi134 ceph-mon[31966]: osdmap e46: 7 total, 7 up, 7 in 2024-07-26T11:06:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:06:58 smithi134 ceph-mon[31966]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:00.512 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.509+0000 7f1a51024640 1 -- 172.21.15.189:0/2282070533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a4c1087a0 msgr2=0x7f1a4c10efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:00.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.510+0000 7f1a51024640 1 --2- 172.21.15.189:0/2282070533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a4c1087a0 0x7f1a4c10efd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009ca0 tx=0x7f1a4402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:07:00.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.510+0000 7f1a51024640 1 -- 172.21.15.189:0/2282070533 shutdown_connections 2024-07-26T11:07:00.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.510+0000 7f1a51024640 1 --2- 172.21.15.189:0/2282070533 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a4c107e50 0x7f1a4c108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:00.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.510+0000 7f1a51024640 1 --2- 172.21.15.189:0/2282070533 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a4c1087a0 0x7f1a4c10efd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:00.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.510+0000 7f1a51024640 1 --2- 172.21.15.189:0/2282070533 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a4c102510 0x7f1a4c102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:00.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.510+0000 7f1a51024640 1 -- 172.21.15.189:0/2282070533 >> 172.21.15.189:0/2282070533 conn(0x7f1a4c0791a0 msgr2=0x7f1a4c0795b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:07:00.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.510+0000 7f1a51024640 1 -- 172.21.15.189:0/2282070533 shutdown_connections 2024-07-26T11:07:00.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.510+0000 7f1a51024640 1 -- 172.21.15.189:0/2282070533 wait complete. 2024-07-26T11:07:00.513 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.510+0000 7f1a51024640 1 Processor -- start 2024-07-26T11:07:00.515 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.511+0000 7f1a51024640 1 -- start start 2024-07-26T11:07:00.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.511+0000 7f1a51024640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a4c102510 0x7f1a4c074a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:00.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.511+0000 7f1a51024640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a4c107e50 0x7f1a4c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:00.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.511+0000 7f1a51024640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a4c1087a0 0x7f1a4c0703a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:00.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.511+0000 7f1a51024640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c06d110 con 0x7f1a4c102510 2024-07-26T11:07:00.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.511+0000 7f1a51024640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c06cf90 con 0x7f1a4c107e50 2024-07-26T11:07:00.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.511+0000 7f1a51024640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4c06d290 con 0x7f1a4c1087a0 2024-07-26T11:07:00.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.511+0000 7f1a4a59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a4c107e50 0x7f1a4c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:00.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.511+0000 7f1a4a59c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a4c107e50 0x7f1a4c06fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38224/0 (socket says 172.21.15.189:38224) 2024-07-26T11:07:00.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.511+0000 7f1a4a59c640 1 -- 172.21.15.189:0/150364567 learned_addr learned my addr 172.21.15.189:0/150364567 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-07-26T11:07:00.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.511+0000 7f1a4b59e640 1 --2- 172.21.15.189:0/150364567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a4c1087a0 0x7f1a4c0703a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:00.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.511+0000 7f1a4a59c640 1 -- 172.21.15.189:0/150364567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a4c1087a0 msgr2=0x7f1a4c0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:00.516 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.511+0000 7f1a4a59c640 1 --2- 172.21.15.189:0/150364567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a4c1087a0 0x7f1a4c0703a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:00.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.511+0000 7f1a4a59c640 1 -- 172.21.15.189:0/150364567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a4c102510 msgr2=0x7f1a4c074a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:00.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.511+0000 7f1a4a59c640 1 --2- 172.21.15.189:0/150364567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a4c102510 0x7f1a4c074a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:00.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.511+0000 7f1a4a59c640 1 -- 172.21.15.189:0/150364567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a4c1ae920 con 0x7f1a4c107e50 2024-07-26T11:07:00.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.512+0000 7f1a4a59c640 1 --2- 172.21.15.189:0/150364567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a4c107e50 0x7f1a4c06fcd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1a4000c8f0 tx=0x7f1a4000cdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:07:00.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.512+0000 7f1a33fff640 1 -- 172.21.15.189:0/150364567 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a40007bb0 con 0x7f1a4c107e50 2024-07-26T11:07:00.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.512+0000 7f1a51024640 1 -- 172.21.15.189:0/150364567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a4c1aeaf0 con 0x7f1a4c107e50 2024-07-26T11:07:00.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.512+0000 7f1a33fff640 1 -- 172.21.15.189:0/150364567 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a40007d50 con 0x7f1a4c107e50 2024-07-26T11:07:00.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.512+0000 7f1a33fff640 1 -- 172.21.15.189:0/150364567 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a40004dd0 con 0x7f1a4c107e50 2024-07-26T11:07:00.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.512+0000 7f1a51024640 1 -- 172.21.15.189:0/150364567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a4c1af030 con 0x7f1a4c107e50 2024-07-26T11:07:00.517 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.513+0000 7f1a51024640 1 -- 172.21.15.189:0/150364567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a4c103790 con 0x7f1a4c107e50 2024-07-26T11:07:00.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.515+0000 7f1a33fff640 1 -- 172.21.15.189:0/150364567 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f1a40004f70 con 0x7f1a4c107e50 2024-07-26T11:07:00.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.515+0000 7f1a33fff640 1 --2- 172.21.15.189:0/150364567 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1a340b3c60 0x7f1a340b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:00.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.515+0000 7f1a33fff640 1 -- 172.21.15.189:0/150364567 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5180+0+0 (secure 0 0 0) 0x7f1a400e30a0 con 0x7f1a4c107e50 2024-07-26T11:07:00.518 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.516+0000 7f1a4ad9d640 1 --2- 172.21.15.189:0/150364567 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1a340b3c60 0x7f1a340b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:00.519 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.517+0000 7f1a4ad9d640 1 --2- 172.21.15.189:0/150364567 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1a340b3c60 0x7f1a340b6170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c074750 tx=0x7f1a3c006c90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:07:00.520 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.518+0000 7f1a33fff640 1 -- 172.21.15.189:0/150364567 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1a40014030 con 0x7f1a4c107e50 2024-07-26T11:07:00.705 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:00.703+0000 7f1a51024640 1 -- 172.21.15.189:0/150364567 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f1a4c06b590 con 0x7f1a340b3c60 2024-07-26T11:07:01.151 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:00 smithi189 ceph-mon[36790]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:01.151 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:01.151 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:01.151 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:01.151 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:01.151 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:01.151 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:01.151 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:01.152 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:01.152 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:01.152 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:07:01.152 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:07:01.152 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:00 smithi134 ceph-mon[31966]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:07:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:07:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:00 smithi193 ceph-mon[37473]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:07:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:07:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:01 smithi134 ceph-mon[31966]: Detected new or changed devices on smithi134 2024-07-26T11:07:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:01 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 5696M 2024-07-26T11:07:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:01 smithi134 ceph-mon[31966]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:07:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:01 smithi189 ceph-mon[36790]: Detected new or changed devices on smithi134 2024-07-26T11:07:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:01 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 5696M 2024-07-26T11:07:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:01 smithi189 ceph-mon[36790]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:07:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:01 smithi193 ceph-mon[37473]: Detected new or changed devices on smithi134 2024-07-26T11:07:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:01 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 5696M 2024-07-26T11:07:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:01 smithi193 ceph-mon[37473]: from='client.24479 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:07:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:02 smithi134 ceph-mon[31966]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:02 smithi189 ceph-mon[36790]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:02 smithi193 ceph-mon[37473]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:04.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:03 smithi134 ceph-mon[31966]: from='client.? 172.21.15.189:0/3344103579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "394764a7-1976-4223-a3c0-5979d941a838"}]: dispatch 2024-07-26T11:07:04.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:03 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "394764a7-1976-4223-a3c0-5979d941a838"}]: dispatch 2024-07-26T11:07:04.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:03 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "394764a7-1976-4223-a3c0-5979d941a838"}]': finished 2024-07-26T11:07:04.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:03 smithi134 ceph-mon[31966]: osdmap e47: 8 total, 7 up, 8 in 2024-07-26T11:07:04.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T11:07:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:03 smithi189 ceph-mon[36790]: from='client.? 172.21.15.189:0/3344103579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "394764a7-1976-4223-a3c0-5979d941a838"}]: dispatch 2024-07-26T11:07:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:03 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "394764a7-1976-4223-a3c0-5979d941a838"}]: dispatch 2024-07-26T11:07:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:03 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "394764a7-1976-4223-a3c0-5979d941a838"}]': finished 2024-07-26T11:07:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:03 smithi189 ceph-mon[36790]: osdmap e47: 8 total, 7 up, 8 in 2024-07-26T11:07:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T11:07:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:03 smithi193 ceph-mon[37473]: from='client.? 172.21.15.189:0/3344103579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "394764a7-1976-4223-a3c0-5979d941a838"}]: dispatch 2024-07-26T11:07:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:03 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "394764a7-1976-4223-a3c0-5979d941a838"}]: dispatch 2024-07-26T11:07:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:03 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "394764a7-1976-4223-a3c0-5979d941a838"}]': finished 2024-07-26T11:07:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:03 smithi193 ceph-mon[37473]: osdmap e47: 8 total, 7 up, 8 in 2024-07-26T11:07:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T11:07:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:04 smithi134 ceph-mon[31966]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:04 smithi134 ceph-mon[31966]: from='client.? 172.21.15.189:0/4017779263' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:07:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:04 smithi189 ceph-mon[36790]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:04 smithi189 ceph-mon[36790]: from='client.? 172.21.15.189:0/4017779263' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:07:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:04 smithi193 ceph-mon[37473]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:04 smithi193 ceph-mon[37473]: from='client.? 172.21.15.189:0/4017779263' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:07:07.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:06 smithi134 ceph-mon[31966]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:06 smithi189 ceph-mon[36790]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:06 smithi193 ceph-mon[37473]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:08.994 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:08 smithi189 ceph-mon[36790]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:08 smithi134 ceph-mon[31966]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:08 smithi193 ceph-mon[37473]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:11.056 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:10 smithi189 ceph-mon[36790]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:11.056 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T11:07:11.056 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:10 smithi134 ceph-mon[31966]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:11.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T11:07:11.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:10 smithi193 ceph-mon[37473]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-07-26T11:07:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:12.119 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:11 smithi189 ceph-mon[36790]: Deploying daemon osd.7 on smithi189 2024-07-26T11:07:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:11 smithi134 ceph-mon[31966]: Deploying daemon osd.7 on smithi189 2024-07-26T11:07:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:11 smithi193 ceph-mon[37473]: Deploying daemon osd.7 on smithi189 2024-07-26T11:07:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:12 smithi134 ceph-mon[31966]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:13.175 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:12 smithi189 ceph-mon[36790]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:12 smithi193 ceph-mon[37473]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:14 smithi134 ceph-mon[31966]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:14 smithi189 ceph-mon[36790]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:14 smithi193 ceph-mon[37473]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:16 smithi189 ceph-mon[36790]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:16 smithi134 ceph-mon[31966]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:17.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:17.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:17.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:17.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:16 smithi193 ceph-mon[37473]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:17.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:17.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:17.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:18 smithi134 ceph-mon[31966]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:18 smithi189 ceph-mon[36790]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:18 smithi193 ceph-mon[37473]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:19.290 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:19.288+0000 7f1a33fff640 1 -- 172.21.15.189:0/150364567 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1a4c06b590 con 0x7f1a340b3c60 2024-07-26T11:07:19.291 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 7 on host 'smithi189' 2024-07-26T11:07:19.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:19.291+0000 7f1a51024640 1 -- 172.21.15.189:0/150364567 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1a340b3c60 msgr2=0x7f1a340b6170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:19.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:19.291+0000 7f1a51024640 1 --2- 172.21.15.189:0/150364567 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1a340b3c60 0x7f1a340b6170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c074750 tx=0x7f1a3c006c90 comp rx=0 tx=0).stop 2024-07-26T11:07:19.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:19.291+0000 7f1a51024640 1 -- 172.21.15.189:0/150364567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a4c107e50 msgr2=0x7f1a4c06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:19.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:19.291+0000 7f1a51024640 1 --2- 172.21.15.189:0/150364567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a4c107e50 0x7f1a4c06fcd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1a4000c8f0 tx=0x7f1a4000cdb0 comp rx=0 tx=0).stop 2024-07-26T11:07:19.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:19.291+0000 7f1a51024640 1 -- 172.21.15.189:0/150364567 shutdown_connections 2024-07-26T11:07:19.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:19.291+0000 7f1a51024640 1 --2- 172.21.15.189:0/150364567 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1a340b3c60 0x7f1a340b6170 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:19.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:19.291+0000 7f1a51024640 1 --2- 172.21.15.189:0/150364567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a4c1087a0 0x7f1a4c0703a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:19.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:19.291+0000 7f1a51024640 1 --2- 172.21.15.189:0/150364567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a4c107e50 0x7f1a4c06fcd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:19.293 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:19.291+0000 7f1a51024640 1 --2- 172.21.15.189:0/150364567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a4c102510 0x7f1a4c074a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:19.291+0000 7f1a51024640 1 -- 172.21.15.189:0/150364567 >> 172.21.15.189:0/150364567 conn(0x7f1a4c0791a0 msgr2=0x7f1a4c101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:07:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:19.291+0000 7f1a51024640 1 -- 172.21.15.189:0/150364567 shutdown_connections 2024-07-26T11:07:19.294 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:07:19.291+0000 7f1a51024640 1 -- 172.21.15.189:0/150364567 wait complete. 2024-07-26T11:07:19.983 DEBUG:teuthology.orchestra.run.smithi189:osd.7> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@osd.7.service 2024-07-26T11:07:19.986 INFO:tasks.cephadm:Deploying osd.8 on smithi193 with /dev/vg_nvme/lv_2... 2024-07-26T11:07:19.990 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-07-26T11:07:20.250 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T11:07:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:20.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:20.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:20 smithi193 ceph-mon[37473]: from='osd.7 [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T11:07:20.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:20 smithi193 ceph-mon[37473]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T11:07:20.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:20 smithi193 ceph-mon[37473]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:20.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:20.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:20.580 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:20.580 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:20.580 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:20 smithi189 ceph-mon[36790]: from='osd.7 [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T11:07:20.581 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:20 smithi189 ceph-mon[36790]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T11:07:20.581 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:20 smithi189 ceph-mon[36790]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:20.581 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:20.581 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:20.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:20.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:20.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:20 smithi134 ceph-mon[31966]: from='osd.7 [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T11:07:20.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:20 smithi134 ceph-mon[31966]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-07-26T11:07:20.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:20 smithi134 ceph-mon[31966]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:20.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:20.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:21.485 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:21 smithi193 ceph-mon[37473]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T11:07:21.486 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:21 smithi193 ceph-mon[37473]: from='osd.7 [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:07:21.486 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:21 smithi193 ceph-mon[37473]: osdmap e48: 8 total, 7 up, 8 in 2024-07-26T11:07:21.486 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T11:07:21.486 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:21 smithi193 ceph-mon[37473]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:07:21.608 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:21 smithi189 ceph-mon[36790]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T11:07:21.609 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:21 smithi189 ceph-mon[36790]: from='osd.7 [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:07:21.609 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:21 smithi189 ceph-mon[36790]: osdmap e48: 8 total, 7 up, 8 in 2024-07-26T11:07:21.609 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T11:07:21.609 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:21 smithi189 ceph-mon[36790]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:07:21.609 INFO:journalctl@ceph.osd.7.smithi189.stdout:Jul 26 11:07:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-7[55998]: 2024-07-26T11:07:21.302+0000 7f2f67cf2640 -1 osd.7 0 waiting for initial osdmap 2024-07-26T11:07:21.609 INFO:journalctl@ceph.osd.7.smithi189.stdout:Jul 26 11:07:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-7[55998]: 2024-07-26T11:07:21.312+0000 7f2f63af0640 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T11:07:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:21 smithi134 ceph-mon[31966]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-07-26T11:07:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:21 smithi134 ceph-mon[31966]: from='osd.7 [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:07:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:21 smithi134 ceph-mon[31966]: osdmap e48: 8 total, 7 up, 8 in 2024-07-26T11:07:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T11:07:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:21 smithi134 ceph-mon[31966]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:07:22.123 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-07-26T11:07:22.230 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-07-26T11:07:22.243 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-07-26T11:07:22.243 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-07-26T11:07:22.243 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115294 s, 909 MB/s 2024-07-26T11:07:22.243 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-07-26T11:07:22.405 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:22 smithi193 ceph-mon[37473]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-07-26T11:07:22.406 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:22 smithi193 ceph-mon[37473]: osdmap e49: 8 total, 7 up, 8 in 2024-07-26T11:07:22.406 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T11:07:22.406 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:22 smithi193 ceph-mon[37473]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:22.598 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:22 smithi189 ceph-mon[36790]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-07-26T11:07:22.598 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:22 smithi189 ceph-mon[36790]: osdmap e49: 8 total, 7 up, 8 in 2024-07-26T11:07:22.598 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T11:07:22.598 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:22 smithi189 ceph-mon[36790]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:22.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:22 smithi134 ceph-mon[31966]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-07-26T11:07:22.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:22 smithi134 ceph-mon[31966]: osdmap e49: 8 total, 7 up, 8 in 2024-07-26T11:07:22.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T11:07:22.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:22 smithi134 ceph-mon[31966]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-07-26T11:07:22.915 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_2 2024-07-26T11:07:23.168 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T11:07:23.613 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:23 smithi193 ceph-mon[37473]: purged_snaps scrub starts 2024-07-26T11:07:23.614 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:23 smithi193 ceph-mon[37473]: purged_snaps scrub ok 2024-07-26T11:07:23.614 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:23 smithi193 ceph-mon[37473]: osd.7 [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911] boot 2024-07-26T11:07:23.614 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:23 smithi193 ceph-mon[37473]: osdmap e50: 8 total, 8 up, 8 in 2024-07-26T11:07:23.614 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T11:07:23.614 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:23 smithi193 ceph-mon[37473]: osdmap e51: 8 total, 8 up, 8 in 2024-07-26T11:07:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:23 smithi134 ceph-mon[31966]: purged_snaps scrub starts 2024-07-26T11:07:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:23 smithi134 ceph-mon[31966]: purged_snaps scrub ok 2024-07-26T11:07:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:23 smithi134 ceph-mon[31966]: osd.7 [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911] boot 2024-07-26T11:07:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:23 smithi134 ceph-mon[31966]: osdmap e50: 8 total, 8 up, 8 in 2024-07-26T11:07:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T11:07:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:23 smithi134 ceph-mon[31966]: osdmap e51: 8 total, 8 up, 8 in 2024-07-26T11:07:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:23 smithi189 ceph-mon[36790]: purged_snaps scrub starts 2024-07-26T11:07:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:23 smithi189 ceph-mon[36790]: purged_snaps scrub ok 2024-07-26T11:07:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:23 smithi189 ceph-mon[36790]: osd.7 [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911] boot 2024-07-26T11:07:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:23 smithi189 ceph-mon[36790]: osdmap e50: 8 total, 8 up, 8 in 2024-07-26T11:07:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-07-26T11:07:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:23 smithi189 ceph-mon[36790]: osdmap e51: 8 total, 8 up, 8 in 2024-07-26T11:07:24.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: Detected new or changed devices on smithi189 2024-07-26T11:07:24.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:24.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:24.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:24.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:24.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:24.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 5696M 2024-07-26T11:07:24.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:24.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:24.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:24.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:24.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:24.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:24.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:24.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:24.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: osdmap e52: 8 total, 8 up, 8 in 2024-07-26T11:07:24.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:24 smithi193 ceph-mon[37473]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: Detected new or changed devices on smithi189 2024-07-26T11:07:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:24.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:24.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:24.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:24.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 5696M 2024-07-26T11:07:24.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:24.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:24.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:24.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:24.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:24.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:24.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:24.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:24.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: osdmap e52: 8 total, 8 up, 8 in 2024-07-26T11:07:24.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:24 smithi189 ceph-mon[36790]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:24.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.778+0000 7f3e5f93c640 1 -- 172.21.15.193:0/309145956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e58102490 msgr2=0x7f3e581028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:24.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.778+0000 7f3e5f93c640 1 --2- 172.21.15.193:0/309145956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e58102490 0x7f3e581028a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3e48009900 tx=0x7f3e4802a410 comp rx=0 tx=0).stop 2024-07-26T11:07:24.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.778+0000 7f3e5f93c640 1 -- 172.21.15.193:0/309145956 shutdown_connections 2024-07-26T11:07:24.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.778+0000 7f3e5f93c640 1 --2- 172.21.15.193:0/309145956 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e58108720 0x7f3e5810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:24.784 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.778+0000 7f3e5f93c640 1 --2- 172.21.15.193:0/309145956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e58107dd0 0x7f3e581081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:24.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.778+0000 7f3e5f93c640 1 --2- 172.21.15.193:0/309145956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e58102490 0x7f3e581028a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:24.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.778+0000 7f3e5f93c640 1 -- 172.21.15.193:0/309145956 >> 172.21.15.193:0/309145956 conn(0x7f3e58079160 msgr2=0x7f3e58079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:07:24.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.778+0000 7f3e5f93c640 1 -- 172.21.15.193:0/309145956 shutdown_connections 2024-07-26T11:07:24.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.778+0000 7f3e5f93c640 1 -- 172.21.15.193:0/309145956 wait complete. 2024-07-26T11:07:24.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.779+0000 7f3e5f93c640 1 Processor -- start 2024-07-26T11:07:24.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.779+0000 7f3e5f93c640 1 -- start start 2024-07-26T11:07:24.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.779+0000 7f3e5f93c640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e58102490 0x7f3e58199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:24.785 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.779+0000 7f3e5f93c640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e58107dd0 0x7f3e5819a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:24.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.779+0000 7f3e5f93c640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e58108720 0x7f3e5819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:24.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.779+0000 7f3e5f93c640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3e58112400 con 0x7f3e58107dd0 2024-07-26T11:07:24.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.779+0000 7f3e5f93c640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3e58112280 con 0x7f3e58108720 2024-07-26T11:07:24.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.779+0000 7f3e5f93c640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3e58112580 con 0x7f3e58102490 2024-07-26T11:07:24.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.779+0000 7f3e5d6d8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e58102490 0x7f3e58199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:24.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.779+0000 7f3e5d6d8640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e58102490 0x7f3e58199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:41030/0 (socket says 172.21.15.193:41030) 2024-07-26T11:07:24.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.779+0000 7f3e5d6d8640 1 -- 172.21.15.193:0/4151228154 learned_addr learned my addr 172.21.15.193:0/4151228154 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-07-26T11:07:24.786 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.779+0000 7f3e5ded9640 1 --2- 172.21.15.193:0/4151228154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e58108720 0x7f3e5819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:24.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.780+0000 7f3e5ced7640 1 --2- 172.21.15.193:0/4151228154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e58107dd0 0x7f3e5819a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:24.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.780+0000 7f3e5d6d8640 1 -- 172.21.15.193:0/4151228154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e58108720 msgr2=0x7f3e5819f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:24.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.780+0000 7f3e5d6d8640 1 --2- 172.21.15.193:0/4151228154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e58108720 0x7f3e5819f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:24.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.780+0000 7f3e5d6d8640 1 -- 172.21.15.193:0/4151228154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e58107dd0 msgr2=0x7f3e5819a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:24.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.780+0000 7f3e5d6d8640 1 --2- 172.21.15.193:0/4151228154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e58107dd0 0x7f3e5819a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:24.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.780+0000 7f3e5d6d8640 1 -- 172.21.15.193:0/4151228154 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5819fe50 con 0x7f3e58102490 2024-07-26T11:07:24.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.780+0000 7f3e5d6d8640 1 --2- 172.21.15.193:0/4151228154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e58102490 0x7f3e58199ee0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e48005940 tx=0x7f3e480042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:07:24.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.780+0000 7f3e4e7fc640 1 -- 172.21.15.193:0/4151228154 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4803f030 con 0x7f3e58102490 2024-07-26T11:07:24.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.780+0000 7f3e4e7fc640 1 -- 172.21.15.193:0/4151228154 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e48003cb0 con 0x7f3e58102490 2024-07-26T11:07:24.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.780+0000 7f3e5f93c640 1 -- 172.21.15.193:0/4151228154 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e581a00e0 con 0x7f3e58102490 2024-07-26T11:07:24.787 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.780+0000 7f3e4e7fc640 1 -- 172.21.15.193:0/4151228154 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e48035070 con 0x7f3e58102490 2024-07-26T11:07:24.788 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.780+0000 7f3e5f93c640 1 -- 172.21.15.193:0/4151228154 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e581a68a0 con 0x7f3e58102490 2024-07-26T11:07:24.788 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.781+0000 7f3e5f93c640 1 -- 172.21.15.193:0/4151228154 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e581028a0 con 0x7f3e58102490 2024-07-26T11:07:24.789 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.782+0000 7f3e4e7fc640 1 -- 172.21.15.193:0/4151228154 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f3e48042030 con 0x7f3e58102490 2024-07-26T11:07:24.789 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.783+0000 7f3e4e7fc640 1 --2- 172.21.15.193:0/4151228154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3e440b3c20 0x7f3e440b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:24.789 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.783+0000 7f3e4e7fc640 1 -- 172.21.15.193:0/4151228154 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5612+0+0 (secure 0 0 0) 0x7f3e48103cd0 con 0x7f3e58102490 2024-07-26T11:07:24.789 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.783+0000 7f3e5ced7640 1 --2- 172.21.15.193:0/4151228154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3e440b3c20 0x7f3e440b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:24.790 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.784+0000 7f3e5ced7640 1 --2- 172.21.15.193:0/4151228154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3e440b3c20 0x7f3e440b6130 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3e54005fd0 tx=0x7f3e54005c70 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:07:24.790 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.786+0000 7f3e4e7fc640 1 -- 172.21.15.193:0/4151228154 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3e480cbdb0 con 0x7f3e58102490 2024-07-26T11:07:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: Detected new or changed devices on smithi189 2024-07-26T11:07:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:24.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:24.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 5696M 2024-07-26T11:07:24.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:24.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:24.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:24.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:24.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:24.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:24.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:24.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:24.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: osdmap e52: 8 total, 8 up, 8 in 2024-07-26T11:07:24.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:24 smithi134 ceph-mon[31966]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:24.974 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:24.973+0000 7f3e5f93c640 1 -- 172.21.15.193:0/4151228154 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f3e58105d70 con 0x7f3e440b3c20 2024-07-26T11:07:26.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:25 smithi193 ceph-mon[37473]: from='client.24508 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:07:26.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:07:26.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:07:26.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:25 smithi134 ceph-mon[31966]: from='client.24508 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:07:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:07:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:07:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:25 smithi189 ceph-mon[36790]: from='client.24508 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:07:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:07:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:07:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:26 smithi193 ceph-mon[37473]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:26 smithi134 ceph-mon[31966]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:26 smithi189 ceph-mon[36790]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:27 smithi193 ceph-mon[37473]: from='client.? 172.21.15.193:0/1542589728' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c218082-0b4c-41c8-92b9-453ae8d15a41"}]: dispatch 2024-07-26T11:07:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:27 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c218082-0b4c-41c8-92b9-453ae8d15a41"}]: dispatch 2024-07-26T11:07:28.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:27 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c218082-0b4c-41c8-92b9-453ae8d15a41"}]': finished 2024-07-26T11:07:28.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:27 smithi193 ceph-mon[37473]: osdmap e53: 9 total, 8 up, 9 in 2024-07-26T11:07:28.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T11:07:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:27 smithi134 ceph-mon[31966]: from='client.? 172.21.15.193:0/1542589728' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c218082-0b4c-41c8-92b9-453ae8d15a41"}]: dispatch 2024-07-26T11:07:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:27 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c218082-0b4c-41c8-92b9-453ae8d15a41"}]: dispatch 2024-07-26T11:07:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:27 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c218082-0b4c-41c8-92b9-453ae8d15a41"}]': finished 2024-07-26T11:07:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:27 smithi134 ceph-mon[31966]: osdmap e53: 9 total, 8 up, 9 in 2024-07-26T11:07:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T11:07:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:27 smithi189 ceph-mon[36790]: from='client.? 172.21.15.193:0/1542589728' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c218082-0b4c-41c8-92b9-453ae8d15a41"}]: dispatch 2024-07-26T11:07:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:27 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1c218082-0b4c-41c8-92b9-453ae8d15a41"}]: dispatch 2024-07-26T11:07:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:27 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c218082-0b4c-41c8-92b9-453ae8d15a41"}]': finished 2024-07-26T11:07:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:27 smithi189 ceph-mon[36790]: osdmap e53: 9 total, 8 up, 9 in 2024-07-26T11:07:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T11:07:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:28 smithi134 ceph-mon[31966]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-07-26T11:07:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:28 smithi134 ceph-mon[31966]: from='client.? 172.21.15.193:0/2942628363' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:07:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:28 smithi189 ceph-mon[36790]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-07-26T11:07:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:28 smithi189 ceph-mon[36790]: from='client.? 172.21.15.193:0/2942628363' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:07:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:28 smithi193 ceph-mon[37473]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-07-26T11:07:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:28 smithi193 ceph-mon[37473]: from='client.? 172.21.15.193:0/2942628363' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:07:31.124 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:30 smithi193 ceph-mon[37473]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-07-26T11:07:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:30 smithi134 ceph-mon[31966]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-07-26T11:07:31.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:30 smithi189 ceph-mon[36790]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-07-26T11:07:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:32 smithi134 ceph-mon[31966]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T11:07:33.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:32 smithi189 ceph-mon[36790]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T11:07:33.203 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:32 smithi193 ceph-mon[37473]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T11:07:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:34 smithi134 ceph-mon[31966]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:07:35.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T11:07:35.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:35.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:34 smithi193 ceph-mon[37473]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:07:35.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T11:07:35.189 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:34 smithi189 ceph-mon[36790]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:07:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-07-26T11:07:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:35 smithi134 ceph-mon[31966]: Deploying daemon osd.8 on smithi193 2024-07-26T11:07:36.153 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:35 smithi193 ceph-mon[37473]: Deploying daemon osd.8 on smithi193 2024-07-26T11:07:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:35 smithi189 ceph-mon[36790]: Deploying daemon osd.8 on smithi193 2024-07-26T11:07:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:36 smithi193 ceph-mon[37473]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:07:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:36 smithi134 ceph-mon[31966]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:07:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:36 smithi189 ceph-mon[36790]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:07:39.081 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:38 smithi193 ceph-mon[37473]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:38 smithi134 ceph-mon[31966]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:38 smithi189 ceph-mon[36790]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:40.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:40.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:40.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:40.184 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:40.184 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:40.184 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:40 smithi134 ceph-mon[31966]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:41.184 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:40 smithi193 ceph-mon[37473]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:41.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:40 smithi189 ceph-mon[36790]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:43.116 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:43.113+0000 7f3e4e7fc640 1 -- 172.21.15.193:0/4151228154 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3e58105d70 con 0x7f3e440b3c20 2024-07-26T11:07:43.116 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 8 on host 'smithi193' 2024-07-26T11:07:43.118 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:43.116+0000 7f3e5f93c640 1 -- 172.21.15.193:0/4151228154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3e440b3c20 msgr2=0x7f3e440b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:43.118 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:43.116+0000 7f3e5f93c640 1 --2- 172.21.15.193:0/4151228154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3e440b3c20 0x7f3e440b6130 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3e54005fd0 tx=0x7f3e54005c70 comp rx=0 tx=0).stop 2024-07-26T11:07:43.118 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:43.116+0000 7f3e5f93c640 1 -- 172.21.15.193:0/4151228154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e58102490 msgr2=0x7f3e58199ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:43.119 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:43.116+0000 7f3e5f93c640 1 --2- 172.21.15.193:0/4151228154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e58102490 0x7f3e58199ee0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e48005940 tx=0x7f3e480042f0 comp rx=0 tx=0).stop 2024-07-26T11:07:43.119 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:43.116+0000 7f3e5f93c640 1 -- 172.21.15.193:0/4151228154 shutdown_connections 2024-07-26T11:07:43.119 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:43.116+0000 7f3e5f93c640 1 --2- 172.21.15.193:0/4151228154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3e440b3c20 0x7f3e440b6130 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:43.119 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:43.116+0000 7f3e5f93c640 1 --2- 172.21.15.193:0/4151228154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e58108720 0x7f3e5819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:43.119 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:43.116+0000 7f3e5f93c640 1 --2- 172.21.15.193:0/4151228154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e58107dd0 0x7f3e5819a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:43.119 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:43.116+0000 7f3e5f93c640 1 --2- 172.21.15.193:0/4151228154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e58102490 0x7f3e58199ee0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:43.119 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:43.116+0000 7f3e5f93c640 1 -- 172.21.15.193:0/4151228154 >> 172.21.15.193:0/4151228154 conn(0x7f3e58079160 msgr2=0x7f3e581013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:07:43.119 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:43.116+0000 7f3e5f93c640 1 -- 172.21.15.193:0/4151228154 shutdown_connections 2024-07-26T11:07:43.119 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:07:43.116+0000 7f3e5f93c640 1 -- 172.21.15.193:0/4151228154 wait complete. 2024-07-26T11:07:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:42 smithi134 ceph-mon[31966]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:42 smithi189 ceph-mon[36790]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:42 smithi193 ceph-mon[37473]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:43.789 DEBUG:teuthology.orchestra.run.smithi193:osd.8> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@osd.8.service 2024-07-26T11:07:43.791 INFO:tasks.cephadm:Deploying osd.9 on smithi134 with /dev/vg_nvme/lv_1... 2024-07-26T11:07:43.792 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T11:07:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:44 smithi134 ceph-mon[31966]: from='osd.8 [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T11:07:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:44 smithi134 ceph-mon[31966]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T11:07:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:44 smithi134 ceph-mon[31966]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:44.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:44.255 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:07:44.282 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:44.282 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:44.282 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:44 smithi189 ceph-mon[36790]: from='osd.8 [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T11:07:44.282 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:44 smithi189 ceph-mon[36790]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T11:07:44.282 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:44 smithi189 ceph-mon[36790]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:44.282 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:44.282 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:44.453 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:44.453 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:44.453 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:44 smithi193 ceph-mon[37473]: from='osd.8 [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T11:07:44.453 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:44 smithi193 ceph-mon[37473]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-07-26T11:07:44.453 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:44 smithi193 ceph-mon[37473]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:44.453 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:44.453 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:45.320 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:45 smithi193 ceph-mon[37473]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T11:07:45.320 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:45 smithi193 ceph-mon[37473]: osdmap e54: 9 total, 8 up, 9 in 2024-07-26T11:07:45.320 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T11:07:45.320 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:45 smithi193 ceph-mon[37473]: from='osd.8 [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:07:45.320 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:45 smithi193 ceph-mon[37473]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:07:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:45 smithi134 ceph-mon[31966]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T11:07:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:45 smithi134 ceph-mon[31966]: osdmap e54: 9 total, 8 up, 9 in 2024-07-26T11:07:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T11:07:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:45 smithi134 ceph-mon[31966]: from='osd.8 [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:07:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:45 smithi134 ceph-mon[31966]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:07:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:45 smithi189 ceph-mon[36790]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-07-26T11:07:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:45 smithi189 ceph-mon[36790]: osdmap e54: 9 total, 8 up, 9 in 2024-07-26T11:07:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T11:07:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:45 smithi189 ceph-mon[36790]: from='osd.8 [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:07:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:45 smithi189 ceph-mon[36790]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:07:45.875 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jul 26 11:07:45 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-8[57011]: 2024-07-26T11:07:45.858+0000 7f4d93e4b640 -1 osd.8 0 waiting for initial osdmap 2024-07-26T11:07:45.875 INFO:journalctl@ceph.osd.8.smithi193.stdout:Jul 26 11:07:45 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-8[57011]: 2024-07-26T11:07:45.872+0000 7f4d8bc41640 -1 osd.8 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T11:07:46.329 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:46 smithi134 ceph-mon[31966]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-07-26T11:07:46.330 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:46 smithi134 ceph-mon[31966]: osdmap e55: 9 total, 8 up, 9 in 2024-07-26T11:07:46.330 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T11:07:46.330 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:46 smithi134 ceph-mon[31966]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:46.330 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:46 smithi134 ceph-mon[31966]: from='osd.8 ' entity='osd.8' 2024-07-26T11:07:46.330 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T11:07:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:46 smithi189 ceph-mon[36790]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-07-26T11:07:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:46 smithi189 ceph-mon[36790]: osdmap e55: 9 total, 8 up, 9 in 2024-07-26T11:07:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T11:07:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:46 smithi189 ceph-mon[36790]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:46 smithi189 ceph-mon[36790]: from='osd.8 ' entity='osd.8' 2024-07-26T11:07:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T11:07:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:46 smithi193 ceph-mon[37473]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-07-26T11:07:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:46 smithi193 ceph-mon[37473]: osdmap e55: 9 total, 8 up, 9 in 2024-07-26T11:07:46.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T11:07:46.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:46 smithi193 ceph-mon[37473]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-07-26T11:07:46.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:46 smithi193 ceph-mon[37473]: from='osd.8 ' entity='osd.8' 2024-07-26T11:07:46.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T11:07:46.937 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T11:07:47.097 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T11:07:47.137 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-07-26T11:07:47.137 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-07-26T11:07:47.138 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0358616 s, 292 MB/s 2024-07-26T11:07:47.138 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-07-26T11:07:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:47 smithi193 ceph-mon[37473]: purged_snaps scrub starts 2024-07-26T11:07:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:47 smithi193 ceph-mon[37473]: purged_snaps scrub ok 2024-07-26T11:07:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:47 smithi193 ceph-mon[37473]: osd.8 [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138] boot 2024-07-26T11:07:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:47 smithi193 ceph-mon[37473]: osdmap e56: 9 total, 9 up, 9 in 2024-07-26T11:07:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T11:07:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:47 smithi193 ceph-mon[37473]: Detected new or changed devices on smithi193 2024-07-26T11:07:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:47.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:47 smithi134 ceph-mon[31966]: purged_snaps scrub starts 2024-07-26T11:07:47.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:47 smithi134 ceph-mon[31966]: purged_snaps scrub ok 2024-07-26T11:07:47.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:47 smithi134 ceph-mon[31966]: osd.8 [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138] boot 2024-07-26T11:07:47.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:47 smithi134 ceph-mon[31966]: osdmap e56: 9 total, 9 up, 9 in 2024-07-26T11:07:47.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T11:07:47.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:47 smithi134 ceph-mon[31966]: Detected new or changed devices on smithi193 2024-07-26T11:07:47.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:47.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:47.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:47.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:47.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:47.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:47.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:47.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:47.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:47 smithi189 ceph-mon[36790]: purged_snaps scrub starts 2024-07-26T11:07:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:47 smithi189 ceph-mon[36790]: purged_snaps scrub ok 2024-07-26T11:07:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:47 smithi189 ceph-mon[36790]: osd.8 [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138] boot 2024-07-26T11:07:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:47 smithi189 ceph-mon[36790]: osdmap e56: 9 total, 9 up, 9 in 2024-07-26T11:07:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-07-26T11:07:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:47 smithi189 ceph-mon[36790]: Detected new or changed devices on smithi193 2024-07-26T11:07:47.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:47.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:47.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:47.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:47.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:07:47.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:47.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:47.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:47.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:48.101 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi134:vg_nvme/lv_1 2024-07-26T11:07:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:48 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 5696M 2024-07-26T11:07:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:48 smithi134 ceph-mon[31966]: osdmap e57: 9 total, 9 up, 9 in 2024-07-26T11:07:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:48.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:48.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:48.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:48.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:48 smithi134 ceph-mon[31966]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:07:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:48 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 5696M 2024-07-26T11:07:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:48 smithi189 ceph-mon[36790]: osdmap e57: 9 total, 9 up, 9 in 2024-07-26T11:07:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:48 smithi189 ceph-mon[36790]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:07:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:48 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 5696M 2024-07-26T11:07:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:48 smithi193 ceph-mon[37473]: osdmap e57: 9 total, 9 up, 9 in 2024-07-26T11:07:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:07:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:07:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:07:48.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:48 smithi193 ceph-mon[37473]: pgmap v202: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:07:48.557 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:07:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:49 smithi134 ceph-mon[31966]: osdmap e58: 9 total, 9 up, 9 in 2024-07-26T11:07:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:49 smithi189 ceph-mon[36790]: osdmap e58: 9 total, 9 up, 9 in 2024-07-26T11:07:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:49 smithi193 ceph-mon[37473]: osdmap e58: 9 total, 9 up, 9 in 2024-07-26T11:07:50.392 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:50 smithi134 ceph-mon[31966]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:07:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:50 smithi189 ceph-mon[36790]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:07:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:50 smithi193 ceph-mon[37473]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:07:50.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.992+0000 7fa692800640 1 -- 172.21.15.134:0/1145910049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa68c10af10 msgr2=0x7fa68c10b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:50.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.992+0000 7fa692800640 1 --2- 172.21.15.134:0/1145910049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa68c10af10 0x7fa68c10b320 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa678009860 tx=0x7fa67802a340 comp rx=0 tx=0).stop 2024-07-26T11:07:50.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.992+0000 7fa692800640 1 -- 172.21.15.134:0/1145910049 shutdown_connections 2024-07-26T11:07:50.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.992+0000 7fa692800640 1 --2- 172.21.15.134:0/1145910049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa68c10b860 0x7fa68c112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:50.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.992+0000 7fa692800640 1 --2- 172.21.15.134:0/1145910049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa68c10af10 0x7fa68c10b320 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:50.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.992+0000 7fa692800640 1 --2- 172.21.15.134:0/1145910049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa68c1055d0 0x7fa68c1059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:50.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.992+0000 7fa692800640 1 -- 172.21.15.134:0/1145910049 >> 172.21.15.134:0/1145910049 conn(0x7fa68c101040 msgr2=0x7fa68c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:07:50.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.992+0000 7fa692800640 1 -- 172.21.15.134:0/1145910049 shutdown_connections 2024-07-26T11:07:50.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.992+0000 7fa692800640 1 -- 172.21.15.134:0/1145910049 wait complete. 2024-07-26T11:07:50.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.993+0000 7fa692800640 1 Processor -- start 2024-07-26T11:07:50.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.993+0000 7fa692800640 1 -- start start 2024-07-26T11:07:50.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.994+0000 7fa692800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa68c1055d0 0x7fa68c19bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:50.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.994+0000 7fa692800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa68c10af10 0x7fa68c19c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:50.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.994+0000 7fa692800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa68c10b860 0x7fa68c1a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:50.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.994+0000 7fa692800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa68c1154d0 con 0x7fa68c10b860 2024-07-26T11:07:50.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.994+0000 7fa692800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa68c115350 con 0x7fa68c1055d0 2024-07-26T11:07:50.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.994+0000 7fa692800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa68c115650 con 0x7fa68c10af10 2024-07-26T11:07:50.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.994+0000 7fa691e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa68c10b860 0x7fa68c1a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:50.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.994+0000 7fa691e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa68c10b860 0x7fa68c1a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:34292/0 (socket says 172.21.15.134:34292) 2024-07-26T11:07:50.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.994+0000 7fa691400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa68c1055d0 0x7fa68c19bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:50.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.994+0000 7fa691e00640 1 -- 172.21.15.134:0/579736127 learned_addr learned my addr 172.21.15.134:0/579736127 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:07:50.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.994+0000 7fa690a00640 1 --2- 172.21.15.134:0/579736127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa68c10af10 0x7fa68c19c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:50.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.995+0000 7fa691e00640 1 -- 172.21.15.134:0/579736127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa68c10af10 msgr2=0x7fa68c19c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:50.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.995+0000 7fa691e00640 1 --2- 172.21.15.134:0/579736127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa68c10af10 0x7fa68c19c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:50.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.995+0000 7fa691e00640 1 -- 172.21.15.134:0/579736127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa68c1055d0 msgr2=0x7fa68c19bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:07:50.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.995+0000 7fa691e00640 1 --2- 172.21.15.134:0/579736127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa68c1055d0 0x7fa68c19bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:07:50.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.995+0000 7fa691e00640 1 -- 172.21.15.134:0/579736127 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa68c1a1ed0 con 0x7fa68c10b860 2024-07-26T11:07:50.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.995+0000 7fa691400640 1 --2- 172.21.15.134:0/579736127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa68c1055d0 0x7fa68c19bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:07:50.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.995+0000 7fa691e00640 1 --2- 172.21.15.134:0/579736127 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa68c10b860 0x7fa68c1a17a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa68400eb10 tx=0x7fa68400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:07:50.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.995+0000 7fa68a000640 1 -- 172.21.15.134:0/579736127 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa684004b20 con 0x7fa68c10b860 2024-07-26T11:07:50.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.996+0000 7fa68a000640 1 -- 172.21.15.134:0/579736127 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa684004cc0 con 0x7fa68c10b860 2024-07-26T11:07:51.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.996+0000 7fa692800640 1 -- 172.21.15.134:0/579736127 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa68c1a21c0 con 0x7fa68c10b860 2024-07-26T11:07:51.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.996+0000 7fa68a000640 1 -- 172.21.15.134:0/579736127 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa684010670 con 0x7fa68c10b860 2024-07-26T11:07:51.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.996+0000 7fa692800640 1 -- 172.21.15.134:0/579736127 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa68c1a8920 con 0x7fa68c10b860 2024-07-26T11:07:51.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.997+0000 7fa692800640 1 -- 172.21.15.134:0/579736127 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa68c1059e0 con 0x7fa68c10b860 2024-07-26T11:07:51.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:50.999+0000 7fa68a000640 1 -- 172.21.15.134:0/579736127 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fa684005020 con 0x7fa68c10b860 2024-07-26T11:07:51.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:51.002+0000 7fa68a000640 1 --2- 172.21.15.134:0/579736127 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6740b3c20 0x7fa6740b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:07:51.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:51.002+0000 7fa68a000640 1 -- 172.21.15.134:0/579736127 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 6044+0+0 (secure 0 0 0) 0x7fa6840e39d0 con 0x7fa68c10b860 2024-07-26T11:07:51.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:51.002+0000 7fa691400640 1 --2- 172.21.15.134:0/579736127 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6740b3c20 0x7fa6740b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:07:51.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:51.003+0000 7fa691400640 1 --2- 172.21.15.134:0/579736127 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6740b3c20 0x7fa6740b6130 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa6800096c0 tx=0x7fa680009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:07:51.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:51.007+0000 7fa68a000640 1 -- 172.21.15.134:0/579736127 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa6840ab860 con 0x7fa68c10b860 2024-07-26T11:07:51.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:07:51.216+0000 7fa692800640 1 -- 172.21.15.134:0/579736127 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fa68c063230 con 0x7fa6740b3c20 2024-07-26T11:07:51.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:07:51.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:07:51.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:07:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:07:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:07:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:07:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:07:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:52 smithi134 ceph-mon[31966]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:07:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:52 smithi134 ceph-mon[31966]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:07:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:52 smithi189 ceph-mon[36790]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:07:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:52 smithi189 ceph-mon[36790]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:07:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:52 smithi193 ceph-mon[37473]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:07:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:52 smithi193 ceph-mon[37473]: pgmap v205: 1 pgs: 1 peering; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:07:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:54 smithi134 ceph-mon[31966]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-07-26T11:07:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:54 smithi189 ceph-mon[36790]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-07-26T11:07:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:54 smithi193 ceph-mon[37473]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-07-26T11:07:56.133 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:55 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3765425500' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48f39082-4fa9-4443-b5b7-b141c10d9e98"}]: dispatch 2024-07-26T11:07:56.133 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:55 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3765425500' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48f39082-4fa9-4443-b5b7-b141c10d9e98"}]': finished 2024-07-26T11:07:56.133 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:55 smithi134 ceph-mon[31966]: osdmap e59: 10 total, 9 up, 10 in 2024-07-26T11:07:56.134 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:07:56.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:55 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3765425500' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48f39082-4fa9-4443-b5b7-b141c10d9e98"}]: dispatch 2024-07-26T11:07:56.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:55 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3765425500' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48f39082-4fa9-4443-b5b7-b141c10d9e98"}]': finished 2024-07-26T11:07:56.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:55 smithi189 ceph-mon[36790]: osdmap e59: 10 total, 9 up, 10 in 2024-07-26T11:07:56.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:07:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:55 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3765425500' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48f39082-4fa9-4443-b5b7-b141c10d9e98"}]: dispatch 2024-07-26T11:07:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:55 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3765425500' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48f39082-4fa9-4443-b5b7-b141c10d9e98"}]': finished 2024-07-26T11:07:56.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:55 smithi193 ceph-mon[37473]: osdmap e59: 10 total, 9 up, 10 in 2024-07-26T11:07:56.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:07:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:56 smithi134 ceph-mon[31966]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T11:07:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:56 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3944286902' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:07:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:56 smithi189 ceph-mon[36790]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T11:07:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:56 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3944286902' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:07:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:56 smithi193 ceph-mon[37473]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T11:07:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:56 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3944286902' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:07:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:07:58 smithi134 ceph-mon[31966]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-07-26T11:07:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:07:58 smithi189 ceph-mon[36790]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-07-26T11:07:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:07:58 smithi193 ceph-mon[37473]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-07-26T11:08:01.145 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:00 smithi134 ceph-mon[31966]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:08:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:00 smithi189 ceph-mon[36790]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:08:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:00 smithi193 ceph-mon[37473]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:08:03.086 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:02 smithi134 ceph-mon[31966]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:08:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:02 smithi189 ceph-mon[36790]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:08:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:02 smithi193 ceph-mon[37473]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:08:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:04 smithi134 ceph-mon[31966]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T11:08:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:04 smithi189 ceph-mon[36790]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T11:08:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:04 smithi193 ceph-mon[37473]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-07-26T11:08:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:05 smithi134 ceph-mon[31966]: Deploying daemon osd.9 on smithi134 2024-07-26T11:08:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:05 smithi189 ceph-mon[36790]: Deploying daemon osd.9 on smithi134 2024-07-26T11:08:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:05 smithi193 ceph-mon[37473]: Deploying daemon osd.9 on smithi134 2024-07-26T11:08:07.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:06 smithi134 ceph-mon[31966]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:06 smithi189 ceph-mon[36790]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:06 smithi193 ceph-mon[37473]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:09.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:08 smithi134 ceph-mon[31966]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:08 smithi189 ceph-mon[36790]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:08 smithi193 ceph-mon[37473]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:11.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:10 smithi134 ceph-mon[31966]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:11.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:10 smithi189 ceph-mon[36790]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:10 smithi193 ceph-mon[37473]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:12 smithi134 ceph-mon[31966]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:13.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:12 smithi189 ceph-mon[36790]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:12 smithi193 ceph-mon[37473]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:14 smithi134 ceph-mon[31966]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:14 smithi189 ceph-mon[36790]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:14 smithi193 ceph-mon[37473]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:17.085 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:16 smithi134 ceph-mon[31966]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:16 smithi189 ceph-mon[36790]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:16 smithi193 ceph-mon[37473]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:18.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:18 smithi134 ceph-mon[31966]: from='osd.9 [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T11:08:18.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:18 smithi134 ceph-mon[31966]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T11:08:18.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:18 smithi134 ceph-mon[31966]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:18.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:18.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:18.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:18 smithi193 ceph-mon[37473]: from='osd.9 [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T11:08:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:18 smithi193 ceph-mon[37473]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T11:08:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:18 smithi193 ceph-mon[37473]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:18 smithi189 ceph-mon[36790]: from='osd.9 [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T11:08:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:18 smithi189 ceph-mon[36790]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-07-26T11:08:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:18 smithi189 ceph-mon[36790]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:19.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:08:19.299+0000 7fa68a000640 1 -- 172.21.15.134:0/579736127 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa68c063230 con 0x7fa6740b3c20 2024-07-26T11:08:19.303 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 9 on host 'smithi134' 2024-07-26T11:08:19.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:08:19.303+0000 7fa692800640 1 -- 172.21.15.134:0/579736127 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6740b3c20 msgr2=0x7fa6740b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:19.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:08:19.303+0000 7fa692800640 1 --2- 172.21.15.134:0/579736127 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6740b3c20 0x7fa6740b6130 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa6800096c0 tx=0x7fa680009260 comp rx=0 tx=0).stop 2024-07-26T11:08:19.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:08:19.303+0000 7fa692800640 1 -- 172.21.15.134:0/579736127 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa68c10b860 msgr2=0x7fa68c1a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:19.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:08:19.303+0000 7fa692800640 1 --2- 172.21.15.134:0/579736127 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa68c10b860 0x7fa68c1a17a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa68400eb10 tx=0x7fa68400c2d0 comp rx=0 tx=0).stop 2024-07-26T11:08:19.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:08:19.303+0000 7fa692800640 1 -- 172.21.15.134:0/579736127 shutdown_connections 2024-07-26T11:08:19.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:08:19.303+0000 7fa692800640 1 --2- 172.21.15.134:0/579736127 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6740b3c20 0x7fa6740b6130 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:19.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:08:19.303+0000 7fa692800640 1 --2- 172.21.15.134:0/579736127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa68c10af10 0x7fa68c19c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:19.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:08:19.303+0000 7fa692800640 1 --2- 172.21.15.134:0/579736127 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa68c10b860 0x7fa68c1a17a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:19.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:08:19.303+0000 7fa692800640 1 --2- 172.21.15.134:0/579736127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa68c1055d0 0x7fa68c19bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:19.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:08:19.303+0000 7fa692800640 1 -- 172.21.15.134:0/579736127 >> 172.21.15.134:0/579736127 conn(0x7fa68c101040 msgr2=0x7fa68c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:08:19.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:08:19.303+0000 7fa692800640 1 -- 172.21.15.134:0/579736127 shutdown_connections 2024-07-26T11:08:19.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:08:19.303+0000 7fa692800640 1 -- 172.21.15.134:0/579736127 wait complete. 2024-07-26T11:08:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:19 smithi193 ceph-mon[37473]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T11:08:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:19 smithi193 ceph-mon[37473]: from='osd.9 [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:08:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:19 smithi193 ceph-mon[37473]: osdmap e60: 10 total, 9 up, 10 in 2024-07-26T11:08:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:08:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:19 smithi193 ceph-mon[37473]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:08:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:19 smithi134 ceph-mon[31966]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T11:08:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:19 smithi134 ceph-mon[31966]: from='osd.9 [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:08:19.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:19 smithi134 ceph-mon[31966]: osdmap e60: 10 total, 9 up, 10 in 2024-07-26T11:08:19.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:08:19.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:19 smithi134 ceph-mon[31966]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:08:19.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:19.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:19.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:19 smithi189 ceph-mon[36790]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-07-26T11:08:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:19 smithi189 ceph-mon[36790]: from='osd.9 [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:08:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:19 smithi189 ceph-mon[36790]: osdmap e60: 10 total, 9 up, 10 in 2024-07-26T11:08:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:08:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:19 smithi189 ceph-mon[36790]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-07-26T11:08:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:20.464 DEBUG:teuthology.orchestra.run.smithi134:osd.9> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@osd.9.service 2024-07-26T11:08:20.468 INFO:tasks.cephadm:Deploying osd.10 on smithi189 with /dev/vg_nvme/lv_1... 2024-07-26T11:08:20.468 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T11:08:20.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:20 smithi134 ceph-mon[31966]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-07-26T11:08:20.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:20 smithi134 ceph-mon[31966]: osdmap e61: 10 total, 9 up, 10 in 2024-07-26T11:08:20.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:08:20.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:08:20.520 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:20 smithi134 ceph-mon[31966]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:20.520 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:20 smithi134 ceph-mon[31966]: from='osd.9 ' entity='osd.9' 2024-07-26T11:08:20.729 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T11:08:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:20 smithi193 ceph-mon[37473]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-07-26T11:08:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:20 smithi193 ceph-mon[37473]: osdmap e61: 10 total, 9 up, 10 in 2024-07-26T11:08:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:08:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:08:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:20 smithi193 ceph-mon[37473]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:20 smithi193 ceph-mon[37473]: from='osd.9 ' entity='osd.9' 2024-07-26T11:08:20.851 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:20 smithi189 ceph-mon[36790]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-07-26T11:08:20.851 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:20 smithi189 ceph-mon[36790]: osdmap e61: 10 total, 9 up, 10 in 2024-07-26T11:08:20.852 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:08:20.852 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:08:20.852 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:20 smithi189 ceph-mon[36790]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 804 GiB / 805 GiB avail 2024-07-26T11:08:20.852 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:20 smithi189 ceph-mon[36790]: from='osd.9 ' entity='osd.9' 2024-07-26T11:08:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:21 smithi193 ceph-mon[37473]: purged_snaps scrub starts 2024-07-26T11:08:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:21 smithi193 ceph-mon[37473]: purged_snaps scrub ok 2024-07-26T11:08:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:08:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:21 smithi193 ceph-mon[37473]: osd.9 [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025] boot 2024-07-26T11:08:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:21 smithi193 ceph-mon[37473]: osdmap e62: 10 total, 10 up, 10 in 2024-07-26T11:08:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:08:21.879 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:21 smithi189 ceph-mon[36790]: purged_snaps scrub starts 2024-07-26T11:08:21.879 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:21 smithi189 ceph-mon[36790]: purged_snaps scrub ok 2024-07-26T11:08:21.879 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:08:21.880 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:21 smithi189 ceph-mon[36790]: osd.9 [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025] boot 2024-07-26T11:08:21.880 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:21 smithi189 ceph-mon[36790]: osdmap e62: 10 total, 10 up, 10 in 2024-07-26T11:08:21.880 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:08:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:21 smithi134 ceph-mon[31966]: purged_snaps scrub starts 2024-07-26T11:08:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:21 smithi134 ceph-mon[31966]: purged_snaps scrub ok 2024-07-26T11:08:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:08:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:21 smithi134 ceph-mon[31966]: osd.9 [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025] boot 2024-07-26T11:08:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:21 smithi134 ceph-mon[31966]: osdmap e62: 10 total, 10 up, 10 in 2024-07-26T11:08:21.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-07-26T11:08:22.663 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:22 smithi134 ceph-mon[31966]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:22.684 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T11:08:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:22 smithi189 ceph-mon[36790]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:22 smithi193 ceph-mon[37473]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:22.777 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T11:08:22.791 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-07-26T11:08:22.791 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-07-26T11:08:22.791 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114289 s, 917 MB/s 2024-07-26T11:08:22.791 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-07-26T11:08:23.453 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_1 2024-07-26T11:08:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:23 smithi189 ceph-mon[36790]: osdmap e63: 10 total, 10 up, 10 in 2024-07-26T11:08:23.710 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.b/config 2024-07-26T11:08:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:23 smithi193 ceph-mon[37473]: osdmap e63: 10 total, 10 up, 10 in 2024-07-26T11:08:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:23 smithi134 ceph-mon[31966]: osdmap e63: 10 total, 10 up, 10 in 2024-07-26T11:08:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:24 smithi134 ceph-mon[31966]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:24 smithi193 ceph-mon[37473]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:24.866 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:24 smithi189 ceph-mon[36790]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:25.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.349+0000 7f17bb849640 1 -- 172.21.15.189:0/646907767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17b410fb70 msgr2=0x7f17b4111f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:25.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.349+0000 7f17bb849640 1 --2- 172.21.15.189:0/646907767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17b410fb70 0x7f17b4111f90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f17ac009ca0 tx=0x7f17ac02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:08:25.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.350+0000 7f17bb849640 1 -- 172.21.15.189:0/646907767 shutdown_connections 2024-07-26T11:08:25.361 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.350+0000 7f17bb849640 1 --2- 172.21.15.189:0/646907767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17b4075ab0 0x7f17b410f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:25.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.350+0000 7f17bb849640 1 --2- 172.21.15.189:0/646907767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17b410fb70 0x7f17b4111f90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:25.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.350+0000 7f17bb849640 1 --2- 172.21.15.189:0/646907767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17b4075160 0x7f17b4075570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:25.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.350+0000 7f17bb849640 1 -- 172.21.15.189:0/646907767 >> 172.21.15.189:0/646907767 conn(0x7f17b407b280 msgr2=0x7f17b407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:08:25.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.350+0000 7f17bb849640 1 -- 172.21.15.189:0/646907767 shutdown_connections 2024-07-26T11:08:25.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.350+0000 7f17bb849640 1 -- 172.21.15.189:0/646907767 wait complete. 2024-07-26T11:08:25.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.350+0000 7f17bb849640 1 Processor -- start 2024-07-26T11:08:25.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.350+0000 7f17bb849640 1 -- start start 2024-07-26T11:08:25.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.351+0000 7f17bb849640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17b4075160 0x7f17b419c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:25.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.351+0000 7f17bb849640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17b4075ab0 0x7f17b419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:25.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.351+0000 7f17bb849640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17b410fb70 0x7f17b41a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:25.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.351+0000 7f17bb849640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f17b4115780 con 0x7f17b4075160 2024-07-26T11:08:25.362 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.351+0000 7f17bb849640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f17b4115600 con 0x7f17b4075ab0 2024-07-26T11:08:25.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.351+0000 7f17bb849640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f17b4115900 con 0x7f17b410fb70 2024-07-26T11:08:25.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.351+0000 7f17b8de4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17b4075ab0 0x7f17b419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:25.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.351+0000 7f17b8de4640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17b4075ab0 0x7f17b419c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:59372/0 (socket says 172.21.15.189:59372) 2024-07-26T11:08:25.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.351+0000 7f17b8de4640 1 -- 172.21.15.189:0/4158554579 learned_addr learned my addr 172.21.15.189:0/4158554579 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-07-26T11:08:25.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.351+0000 7f17b8de4640 1 -- 172.21.15.189:0/4158554579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17b410fb70 msgr2=0x7f17b41a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:25.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.351+0000 7f17b8de4640 1 --2- 172.21.15.189:0/4158554579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17b410fb70 0x7f17b41a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:25.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.351+0000 7f17b8de4640 1 -- 172.21.15.189:0/4158554579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17b4075160 msgr2=0x7f17b419c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:25.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.351+0000 7f17b8de4640 1 --2- 172.21.15.189:0/4158554579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17b4075160 0x7f17b419c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:25.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.351+0000 7f17b8de4640 1 -- 172.21.15.189:0/4158554579 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b41a2080 con 0x7f17b4075ab0 2024-07-26T11:08:25.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.351+0000 7f17b8de4640 1 --2- 172.21.15.189:0/4158554579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17b4075ab0 0x7f17b419c6b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f17b000d950 tx=0x7f17b000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:08:25.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.351+0000 7f17aa7fc640 1 -- 172.21.15.189:0/4158554579 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b0017070 con 0x7f17b4075ab0 2024-07-26T11:08:25.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.352+0000 7f17aa7fc640 1 -- 172.21.15.189:0/4158554579 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17b0009c30 con 0x7f17b4075ab0 2024-07-26T11:08:25.363 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.352+0000 7f17bb849640 1 -- 172.21.15.189:0/4158554579 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17b41a2370 con 0x7f17b4075ab0 2024-07-26T11:08:25.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.352+0000 7f17aa7fc640 1 -- 172.21.15.189:0/4158554579 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b00047a0 con 0x7f17b4075ab0 2024-07-26T11:08:25.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.352+0000 7f17bb849640 1 -- 172.21.15.189:0/4158554579 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17b41a8ad0 con 0x7f17b4075ab0 2024-07-26T11:08:25.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.353+0000 7f17bb849640 1 -- 172.21.15.189:0/4158554579 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17b4075570 con 0x7f17b4075ab0 2024-07-26T11:08:25.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.354+0000 7f17aa7fc640 1 -- 172.21.15.189:0/4158554579 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f17b0004020 con 0x7f17b4075ab0 2024-07-26T11:08:25.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.355+0000 7f17aa7fc640 1 --2- 172.21.15.189:0/4158554579 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f17a00b3c20 0x7f17a00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:25.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.355+0000 7f17aa7fc640 1 -- 172.21.15.189:0/4158554579 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 6476+0+0 (secure 0 0 0) 0x7f17b00e3680 con 0x7f17b4075ab0 2024-07-26T11:08:25.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.355+0000 7f17b95e5640 1 --2- 172.21.15.189:0/4158554579 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f17a00b3c20 0x7f17a00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:25.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.356+0000 7f17b95e5640 1 --2- 172.21.15.189:0/4158554579 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f17a00b3c20 0x7f17a00b6130 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f17a4009710 tx=0x7f17a4006c90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:08:25.364 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.358+0000 7f17aa7fc640 1 -- 172.21.15.189:0/4158554579 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f17b0020020 con 0x7f17b4075ab0 2024-07-26T11:08:25.545 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:25.543+0000 7f17bb849640 1 -- 172.21.15.189:0/4158554579 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f17b4108d90 con 0x7f17a00b3c20 2024-07-26T11:08:26.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: Detected new or changed devices on smithi134 2024-07-26T11:08:26.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:26.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:26.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:26.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:26.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:26.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:26.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 4272M 2024-07-26T11:08:26.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:26.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:26.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:26.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:26.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: from='client.24560 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:08:26.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:08:26.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:08:26.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:26.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:26 smithi134 ceph-mon[31966]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: Detected new or changed devices on smithi134 2024-07-26T11:08:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:26.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:26.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:26.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:26.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:26.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 4272M 2024-07-26T11:08:26.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:26.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:26.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:26.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:26.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: from='client.24560 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:08:26.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:08:26.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:08:26.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:26.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:26 smithi189 ceph-mon[36790]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: Detected new or changed devices on smithi134 2024-07-26T11:08:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:26.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:26.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:26.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:26.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:26.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:26.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 4272M 2024-07-26T11:08:26.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:26.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:26.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:26.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:26.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: from='client.24560 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:08:26.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:08:26.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:08:26.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:26.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:26 smithi193 ceph-mon[37473]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:28 smithi134 ceph-mon[31966]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:28 smithi134 ceph-mon[31966]: from='client.? 172.21.15.189:0/2332406822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95157766-1856-42f7-aea1-9cf83b77c8ae"}]: dispatch 2024-07-26T11:08:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:28 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95157766-1856-42f7-aea1-9cf83b77c8ae"}]: dispatch 2024-07-26T11:08:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:28 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95157766-1856-42f7-aea1-9cf83b77c8ae"}]': finished 2024-07-26T11:08:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:28 smithi134 ceph-mon[31966]: osdmap e64: 11 total, 10 up, 11 in 2024-07-26T11:08:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T11:08:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:28 smithi134 ceph-mon[31966]: from='client.? 172.21.15.189:0/3376581234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:08:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:28 smithi189 ceph-mon[36790]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:28 smithi189 ceph-mon[36790]: from='client.? 172.21.15.189:0/2332406822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95157766-1856-42f7-aea1-9cf83b77c8ae"}]: dispatch 2024-07-26T11:08:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:28 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95157766-1856-42f7-aea1-9cf83b77c8ae"}]: dispatch 2024-07-26T11:08:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:28 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95157766-1856-42f7-aea1-9cf83b77c8ae"}]': finished 2024-07-26T11:08:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:28 smithi189 ceph-mon[36790]: osdmap e64: 11 total, 10 up, 11 in 2024-07-26T11:08:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T11:08:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:28 smithi189 ceph-mon[36790]: from='client.? 172.21.15.189:0/3376581234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:08:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:28 smithi193 ceph-mon[37473]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:28 smithi193 ceph-mon[37473]: from='client.? 172.21.15.189:0/2332406822' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95157766-1856-42f7-aea1-9cf83b77c8ae"}]: dispatch 2024-07-26T11:08:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:28 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95157766-1856-42f7-aea1-9cf83b77c8ae"}]: dispatch 2024-07-26T11:08:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:28 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95157766-1856-42f7-aea1-9cf83b77c8ae"}]': finished 2024-07-26T11:08:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:28 smithi193 ceph-mon[37473]: osdmap e64: 11 total, 10 up, 11 in 2024-07-26T11:08:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T11:08:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:28 smithi193 ceph-mon[37473]: from='client.? 172.21.15.189:0/3376581234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:08:31.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:30 smithi134 ceph-mon[31966]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:31.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:30 smithi189 ceph-mon[36790]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:30 smithi193 ceph-mon[37473]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:33.006 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:32 smithi189 ceph-mon[36790]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:32 smithi134 ceph-mon[31966]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:32 smithi193 ceph-mon[37473]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:35.044 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:34 smithi189 ceph-mon[36790]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:34 smithi134 ceph-mon[31966]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:34 smithi193 ceph-mon[37473]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:36.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T11:08:36.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T11:08:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-07-26T11:08:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:37.107 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:36 smithi189 ceph-mon[36790]: Deploying daemon osd.10 on smithi189 2024-07-26T11:08:37.107 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:36 smithi189 ceph-mon[36790]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:36 smithi134 ceph-mon[31966]: Deploying daemon osd.10 on smithi189 2024-07-26T11:08:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:36 smithi134 ceph-mon[31966]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:36 smithi193 ceph-mon[37473]: Deploying daemon osd.10 on smithi189 2024-07-26T11:08:37.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:36 smithi193 ceph-mon[37473]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:38 smithi189 ceph-mon[36790]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:38 smithi193 ceph-mon[37473]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:38 smithi134 ceph-mon[31966]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:41.129 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:40 smithi189 ceph-mon[36790]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:41.129 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:41.129 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:41.129 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:40 smithi193 ceph-mon[37473]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:40 smithi134 ceph-mon[31966]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:41.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:41.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:41.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:42.992 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:42 smithi189 ceph-mon[36790]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:42 smithi193 ceph-mon[37473]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:42 smithi134 ceph-mon[31966]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:44.119 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:44.117+0000 7f17aa7fc640 1 -- 172.21.15.189:0/4158554579 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f17b4108d90 con 0x7f17a00b3c20 2024-07-26T11:08:44.120 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 10 on host 'smithi189' 2024-07-26T11:08:44.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:44.120+0000 7f17bb849640 1 -- 172.21.15.189:0/4158554579 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f17a00b3c20 msgr2=0x7f17a00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:44.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:44.120+0000 7f17bb849640 1 --2- 172.21.15.189:0/4158554579 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f17a00b3c20 0x7f17a00b6130 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f17a4009710 tx=0x7f17a4006c90 comp rx=0 tx=0).stop 2024-07-26T11:08:44.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:44.120+0000 7f17bb849640 1 -- 172.21.15.189:0/4158554579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17b4075ab0 msgr2=0x7f17b419c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:44.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:44.120+0000 7f17bb849640 1 --2- 172.21.15.189:0/4158554579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17b4075ab0 0x7f17b419c6b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f17b000d950 tx=0x7f17b000de10 comp rx=0 tx=0).stop 2024-07-26T11:08:44.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:44.120+0000 7f17bb849640 1 -- 172.21.15.189:0/4158554579 shutdown_connections 2024-07-26T11:08:44.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:44.120+0000 7f17bb849640 1 --2- 172.21.15.189:0/4158554579 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f17a00b3c20 0x7f17a00b6130 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:44.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:44.120+0000 7f17bb849640 1 --2- 172.21.15.189:0/4158554579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f17b410fb70 0x7f17b41a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:44.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:44.120+0000 7f17bb849640 1 --2- 172.21.15.189:0/4158554579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17b4075ab0 0x7f17b419c6b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:44.122 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:44.120+0000 7f17bb849640 1 --2- 172.21.15.189:0/4158554579 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17b4075160 0x7f17b419c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:44.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:44.120+0000 7f17bb849640 1 -- 172.21.15.189:0/4158554579 >> 172.21.15.189:0/4158554579 conn(0x7f17b407b280 msgr2=0x7f17b4107670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:08:44.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:44.120+0000 7f17bb849640 1 -- 172.21.15.189:0/4158554579 shutdown_connections 2024-07-26T11:08:44.123 INFO:teuthology.orchestra.run.smithi189.stderr:2024-07-26T11:08:44.120+0000 7f17bb849640 1 -- 172.21.15.189:0/4158554579 wait complete. 2024-07-26T11:08:44.326 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:44.327 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:44.327 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:44.327 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:44.327 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:44.327 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:44 smithi189 ceph-mon[36790]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:44.327 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:44.327 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:44.327 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:44.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:44.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:44.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:44.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:44.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:44.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:44 smithi134 ceph-mon[31966]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:44.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:44.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:44.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:44.771 DEBUG:teuthology.orchestra.run.smithi189:osd.10> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@osd.10.service 2024-07-26T11:08:44.774 INFO:tasks.cephadm:Deploying osd.11 on smithi193 with /dev/vg_nvme/lv_1... 2024-07-26T11:08:44.774 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-07-26T11:08:44.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:44.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:44.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:44.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:44.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:44.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:44 smithi193 ceph-mon[37473]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:44.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:08:44.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:44.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:45.034 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T11:08:45.465 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:45 smithi193 ceph-mon[37473]: from='osd.10 [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T11:08:45.465 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:45 smithi193 ceph-mon[37473]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T11:08:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:45 smithi134 ceph-mon[31966]: from='osd.10 [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T11:08:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:45 smithi134 ceph-mon[31966]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T11:08:45.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:45 smithi189 ceph-mon[36790]: from='osd.10 [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T11:08:45.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:45 smithi189 ceph-mon[36790]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-07-26T11:08:46.456 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:46 smithi193 ceph-mon[37473]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T11:08:46.456 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:46 smithi193 ceph-mon[37473]: from='osd.10 [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:08:46.456 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:46 smithi193 ceph-mon[37473]: osdmap e65: 11 total, 10 up, 11 in 2024-07-26T11:08:46.457 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T11:08:46.457 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:46 smithi193 ceph-mon[37473]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:08:46.457 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:46 smithi193 ceph-mon[37473]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:46.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:46 smithi134 ceph-mon[31966]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T11:08:46.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:46 smithi134 ceph-mon[31966]: from='osd.10 [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:08:46.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:46 smithi134 ceph-mon[31966]: osdmap e65: 11 total, 10 up, 11 in 2024-07-26T11:08:46.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T11:08:46.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:46 smithi134 ceph-mon[31966]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:08:46.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:46 smithi134 ceph-mon[31966]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:46 smithi189 ceph-mon[36790]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-07-26T11:08:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:46 smithi189 ceph-mon[36790]: from='osd.10 [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:08:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:46 smithi189 ceph-mon[36790]: osdmap e65: 11 total, 10 up, 11 in 2024-07-26T11:08:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T11:08:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:46 smithi189 ceph-mon[36790]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-07-26T11:08:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:46 smithi189 ceph-mon[36790]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 292 MiB used, 894 GiB / 894 GiB avail 2024-07-26T11:08:46.695 INFO:journalctl@ceph.osd.10.smithi189.stdout:Jul 26 11:08:46 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-10[62549]: 2024-07-26T11:08:46.347+0000 7f4afa77e640 -1 osd.10 0 waiting for initial osdmap 2024-07-26T11:08:46.695 INFO:journalctl@ceph.osd.10.smithi189.stdout:Jul 26 11:08:46 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-10[62549]: 2024-07-26T11:08:46.361+0000 7f4af657c640 -1 osd.10 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T11:08:46.832 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-07-26T11:08:46.931 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-07-26T11:08:46.944 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-07-26T11:08:46.944 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-07-26T11:08:46.944 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113958 s, 920 MB/s 2024-07-26T11:08:46.944 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-07-26T11:08:47.653 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:47 smithi193 ceph-mon[37473]: purged_snaps scrub starts 2024-07-26T11:08:47.653 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:47 smithi193 ceph-mon[37473]: purged_snaps scrub ok 2024-07-26T11:08:47.653 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:47 smithi193 ceph-mon[37473]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-07-26T11:08:47.653 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:47 smithi193 ceph-mon[37473]: osdmap e66: 11 total, 10 up, 11 in 2024-07-26T11:08:47.653 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T11:08:47.654 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T11:08:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:47 smithi134 ceph-mon[31966]: purged_snaps scrub starts 2024-07-26T11:08:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:47 smithi134 ceph-mon[31966]: purged_snaps scrub ok 2024-07-26T11:08:47.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:47 smithi134 ceph-mon[31966]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-07-26T11:08:47.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:47 smithi134 ceph-mon[31966]: osdmap e66: 11 total, 10 up, 11 in 2024-07-26T11:08:47.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T11:08:47.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T11:08:47.683 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch daemon add osd smithi193:vg_nvme/lv_1 2024-07-26T11:08:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:47 smithi189 ceph-mon[36790]: purged_snaps scrub starts 2024-07-26T11:08:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:47 smithi189 ceph-mon[36790]: purged_snaps scrub ok 2024-07-26T11:08:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:47 smithi189 ceph-mon[36790]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-07-26T11:08:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:47 smithi189 ceph-mon[36790]: osdmap e66: 11 total, 10 up, 11 in 2024-07-26T11:08:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T11:08:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T11:08:47.940 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T11:08:48.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: osd.10 [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858] boot 2024-07-26T11:08:48.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: osdmap e67: 11 total, 11 up, 11 in 2024-07-26T11:08:48.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T11:08:48.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: Detected new or changed devices on smithi189 2024-07-26T11:08:48.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:48.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:48.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:48.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:48.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:48.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:48.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 4272M 2024-07-26T11:08:48.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:48.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:48.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:48.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:48.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: osdmap e68: 11 total, 11 up, 11 in 2024-07-26T11:08:48.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:48 smithi134 ceph-mon[31966]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:08:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: osd.10 [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858] boot 2024-07-26T11:08:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: osdmap e67: 11 total, 11 up, 11 in 2024-07-26T11:08:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T11:08:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: Detected new or changed devices on smithi189 2024-07-26T11:08:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:48.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:48.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:48.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 4272M 2024-07-26T11:08:48.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:48.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:48.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:48.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:48.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: osdmap e68: 11 total, 11 up, 11 in 2024-07-26T11:08:48.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:48 smithi189 ceph-mon[36790]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:08:48.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: osd.10 [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858] boot 2024-07-26T11:08:48.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: osdmap e67: 11 total, 11 up, 11 in 2024-07-26T11:08:48.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-07-26T11:08:48.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: Detected new or changed devices on smithi189 2024-07-26T11:08:48.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:48.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:48.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:48.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:48.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:48.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:08:48.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 4272M 2024-07-26T11:08:48.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:48.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:48.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:08:48.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:08:48.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: osdmap e68: 11 total, 11 up, 11 in 2024-07-26T11:08:48.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:48 smithi193 ceph-mon[37473]: pgmap v243: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:08:49.723 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.704+0000 7fe52244b640 1 -- 172.21.15.193:0/1146637301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe51c103590 msgr2=0x7fe51c1039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:49.723 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.704+0000 7fe52244b640 1 --2- 172.21.15.193:0/1146637301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe51c103590 0x7fe51c1039a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe50c009900 tx=0x7fe50c02a410 comp rx=0 tx=0).stop 2024-07-26T11:08:49.723 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.704+0000 7fe52244b640 1 -- 172.21.15.193:0/1146637301 shutdown_connections 2024-07-26T11:08:49.723 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.704+0000 7fe52244b640 1 --2- 172.21.15.193:0/1146637301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe51c109820 0x7fe51c110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:49.724 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.704+0000 7fe52244b640 1 --2- 172.21.15.193:0/1146637301 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe51c108ed0 0x7fe51c1092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:49.724 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.704+0000 7fe52244b640 1 --2- 172.21.15.193:0/1146637301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe51c103590 0x7fe51c1039a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:49.724 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.704+0000 7fe52244b640 1 -- 172.21.15.193:0/1146637301 >> 172.21.15.193:0/1146637301 conn(0x7fe51c07b280 msgr2=0x7fe51c07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:08:49.724 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.704+0000 7fe52244b640 1 -- 172.21.15.193:0/1146637301 shutdown_connections 2024-07-26T11:08:49.724 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.704+0000 7fe52244b640 1 -- 172.21.15.193:0/1146637301 wait complete. 2024-07-26T11:08:49.724 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.704+0000 7fe52244b640 1 Processor -- start 2024-07-26T11:08:49.724 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.704+0000 7fe52244b640 1 -- start start 2024-07-26T11:08:49.724 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.705+0000 7fe52244b640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe51c103590 0x7fe51c075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:49.724 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.705+0000 7fe52244b640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe51c108ed0 0x7fe51c075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:49.724 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.705+0000 7fe52244b640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe51c109820 0x7fe51c075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:49.724 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.705+0000 7fe52244b640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c113520 con 0x7fe51c108ed0 2024-07-26T11:08:49.724 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.705+0000 7fe52244b640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c1133a0 con 0x7fe51c109820 2024-07-26T11:08:49.725 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.705+0000 7fe52244b640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe51c1136a0 con 0x7fe51c103590 2024-07-26T11:08:49.725 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.705+0000 7fe51bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe51c103590 0x7fe51c075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:49.725 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.705+0000 7fe51bfff640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe51c103590 0x7fe51c075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:46656/0 (socket says 172.21.15.193:46656) 2024-07-26T11:08:49.725 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.705+0000 7fe51bfff640 1 -- 172.21.15.193:0/219003530 learned_addr learned my addr 172.21.15.193:0/219003530 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-07-26T11:08:49.725 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.705+0000 7fe51bfff640 1 -- 172.21.15.193:0/219003530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe51c109820 msgr2=0x7fe51c075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:49.725 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.705+0000 7fe51bfff640 1 --2- 172.21.15.193:0/219003530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe51c109820 0x7fe51c075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:49.725 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.705+0000 7fe51bfff640 1 -- 172.21.15.193:0/219003530 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe51c108ed0 msgr2=0x7fe51c075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:08:49.725 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.705+0000 7fe51bfff640 1 --2- 172.21.15.193:0/219003530 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe51c108ed0 0x7fe51c075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:08:49.725 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.705+0000 7fe51bfff640 1 -- 172.21.15.193:0/219003530 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51c1a81b0 con 0x7fe51c103590 2024-07-26T11:08:49.725 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.705+0000 7fe51bfff640 1 --2- 172.21.15.193:0/219003530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe51c103590 0x7fe51c075140 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe50c005940 tx=0x7fe50c0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:08:49.725 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.706+0000 7fe5197fa640 1 -- 172.21.15.193:0/219003530 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50c03f030 con 0x7fe51c103590 2024-07-26T11:08:49.725 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.706+0000 7fe5197fa640 1 -- 172.21.15.193:0/219003530 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe50c005d10 con 0x7fe51c103590 2024-07-26T11:08:49.726 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.706+0000 7fe5197fa640 1 -- 172.21.15.193:0/219003530 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50c035070 con 0x7fe51c103590 2024-07-26T11:08:49.726 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.706+0000 7fe52244b640 1 -- 172.21.15.193:0/219003530 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe51c1a8440 con 0x7fe51c103590 2024-07-26T11:08:49.726 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.706+0000 7fe52244b640 1 -- 172.21.15.193:0/219003530 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe51c1a89a0 con 0x7fe51c103590 2024-07-26T11:08:49.726 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.707+0000 7fe52244b640 1 -- 172.21.15.193:0/219003530 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe51c1039a0 con 0x7fe51c103590 2024-07-26T11:08:49.726 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.708+0000 7fe5197fa640 1 -- 172.21.15.193:0/219003530 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fe50c042030 con 0x7fe51c103590 2024-07-26T11:08:49.726 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.709+0000 7fe5197fa640 1 --2- 172.21.15.193:0/219003530 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe5040b3c20 0x7fe5040b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:08:49.726 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.709+0000 7fe5197fa640 1 -- 172.21.15.193:0/219003530 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6909+0+0 (secure 0 0 0) 0x7fe50c103f40 con 0x7fe51c103590 2024-07-26T11:08:49.726 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.709+0000 7fe51b7fe640 1 --2- 172.21.15.193:0/219003530 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe5040b3c20 0x7fe5040b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:08:49.726 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.710+0000 7fe51b7fe640 1 --2- 172.21.15.193:0/219003530 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe5040b3c20 0x7fe5040b6130 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe51c105fd0 tx=0x7fe510006c90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:08:49.726 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.712+0000 7fe5197fa640 1 -- 172.21.15.193:0/219003530 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe50c0cbb10 con 0x7fe51c103590 2024-07-26T11:08:49.896 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:08:49.894+0000 7fe52244b640 1 -- 172.21.15.193:0/219003530 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe51c063230 con 0x7fe5040b3c20 2024-07-26T11:08:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:49 smithi134 ceph-mon[31966]: osdmap e69: 11 total, 11 up, 11 in 2024-07-26T11:08:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:49 smithi189 ceph-mon[36790]: osdmap e69: 11 total, 11 up, 11 in 2024-07-26T11:08:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:49 smithi193 ceph-mon[37473]: osdmap e69: 11 total, 11 up, 11 in 2024-07-26T11:08:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:50 smithi193 ceph-mon[37473]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:08:51.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:50 smithi193 ceph-mon[37473]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:08:51.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:08:51.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:08:51.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:51.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:50 smithi134 ceph-mon[31966]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:08:51.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:50 smithi134 ceph-mon[31966]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:08:51.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:08:51.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:08:51.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:50 smithi189 ceph-mon[36790]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:08:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:50 smithi189 ceph-mon[36790]: from='client.24586 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:08:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-07-26T11:08:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-07-26T11:08:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:08:53.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:52 smithi134 ceph-mon[31966]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:08:53.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:52 smithi134 ceph-mon[31966]: from='client.? 172.21.15.193:0/1983619748' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "244fadc7-f079-4736-8e7e-c0ce5ae6f43e"}]: dispatch 2024-07-26T11:08:53.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:52 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "244fadc7-f079-4736-8e7e-c0ce5ae6f43e"}]: dispatch 2024-07-26T11:08:53.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:52 smithi134 ceph-mon[31966]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "244fadc7-f079-4736-8e7e-c0ce5ae6f43e"}]': finished 2024-07-26T11:08:53.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:52 smithi134 ceph-mon[31966]: osdmap e70: 12 total, 11 up, 12 in 2024-07-26T11:08:53.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T11:08:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:52 smithi189 ceph-mon[36790]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:08:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:52 smithi189 ceph-mon[36790]: from='client.? 172.21.15.193:0/1983619748' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "244fadc7-f079-4736-8e7e-c0ce5ae6f43e"}]: dispatch 2024-07-26T11:08:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:52 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "244fadc7-f079-4736-8e7e-c0ce5ae6f43e"}]: dispatch 2024-07-26T11:08:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:52 smithi189 ceph-mon[36790]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "244fadc7-f079-4736-8e7e-c0ce5ae6f43e"}]': finished 2024-07-26T11:08:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:52 smithi189 ceph-mon[36790]: osdmap e70: 12 total, 11 up, 12 in 2024-07-26T11:08:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T11:08:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:52 smithi193 ceph-mon[37473]: pgmap v246: 1 pgs: 1 peering; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:08:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:52 smithi193 ceph-mon[37473]: from='client.? 172.21.15.193:0/1983619748' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "244fadc7-f079-4736-8e7e-c0ce5ae6f43e"}]: dispatch 2024-07-26T11:08:53.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:52 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "244fadc7-f079-4736-8e7e-c0ce5ae6f43e"}]: dispatch 2024-07-26T11:08:53.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:52 smithi193 ceph-mon[37473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "244fadc7-f079-4736-8e7e-c0ce5ae6f43e"}]': finished 2024-07-26T11:08:53.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:52 smithi193 ceph-mon[37473]: osdmap e70: 12 total, 11 up, 12 in 2024-07-26T11:08:53.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T11:08:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:53 smithi134 ceph-mon[31966]: from='client.? 172.21.15.193:0/1043580232' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:08:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:53 smithi189 ceph-mon[36790]: from='client.? 172.21.15.193:0/1043580232' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:08:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:53 smithi193 ceph-mon[37473]: from='client.? 172.21.15.193:0/1043580232' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-07-26T11:08:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:54 smithi134 ceph-mon[31966]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T11:08:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:54 smithi189 ceph-mon[36790]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T11:08:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:54 smithi193 ceph-mon[37473]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-07-26T11:08:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:56 smithi193 ceph-mon[37473]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T11:08:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:56 smithi134 ceph-mon[31966]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T11:08:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:56 smithi189 ceph-mon[36790]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-07-26T11:08:59.017 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:58 smithi193 ceph-mon[37473]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-07-26T11:08:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:58 smithi134 ceph-mon[31966]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-07-26T11:08:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:58 smithi189 ceph-mon[36790]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-07-26T11:09:00.055 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T11:09:00.056 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:08:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T11:09:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:08:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-07-26T11:09:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:08:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:01.149 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:00 smithi193 ceph-mon[37473]: Deploying daemon osd.11 on smithi193 2024-07-26T11:09:01.149 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:00 smithi193 ceph-mon[37473]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:09:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:00 smithi189 ceph-mon[36790]: Deploying daemon osd.11 on smithi193 2024-07-26T11:09:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:00 smithi189 ceph-mon[36790]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:09:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:00 smithi134 ceph-mon[31966]: Deploying daemon osd.11 on smithi193 2024-07-26T11:09:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:00 smithi134 ceph-mon[31966]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:09:03.079 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:02 smithi193 ceph-mon[37473]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:09:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:02 smithi189 ceph-mon[36790]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:09:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:02 smithi134 ceph-mon[31966]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-07-26T11:09:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:04 smithi189 ceph-mon[36790]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:09:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:05.230 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:04 smithi193 ceph-mon[37473]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:09:05.230 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:05.230 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:05.230 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:04 smithi134 ceph-mon[31966]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:09:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:05.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:05.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:06 smithi193 ceph-mon[37473]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:09:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:06 smithi134 ceph-mon[31966]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:09:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:06 smithi189 ceph-mon[36790]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:09:08.478 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 11 on host 'smithi193' 2024-07-26T11:09:08.478 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:08.476+0000 7fe5197fa640 1 -- 172.21.15.193:0/219003530 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fe51c063230 con 0x7fe5040b3c20 2024-07-26T11:09:08.481 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:08.478+0000 7fe52244b640 1 -- 172.21.15.193:0/219003530 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe5040b3c20 msgr2=0x7fe5040b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:08.481 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:08.478+0000 7fe52244b640 1 --2- 172.21.15.193:0/219003530 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe5040b3c20 0x7fe5040b6130 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe51c105fd0 tx=0x7fe510006c90 comp rx=0 tx=0).stop 2024-07-26T11:09:08.481 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:08.478+0000 7fe52244b640 1 -- 172.21.15.193:0/219003530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe51c103590 msgr2=0x7fe51c075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:08.481 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:08.479+0000 7fe52244b640 1 --2- 172.21.15.193:0/219003530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe51c103590 0x7fe51c075140 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe50c005940 tx=0x7fe50c0042f0 comp rx=0 tx=0).stop 2024-07-26T11:09:08.481 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:08.479+0000 7fe52244b640 1 -- 172.21.15.193:0/219003530 shutdown_connections 2024-07-26T11:09:08.481 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:08.479+0000 7fe52244b640 1 --2- 172.21.15.193:0/219003530 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe5040b3c20 0x7fe5040b6130 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:08.481 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:08.479+0000 7fe52244b640 1 --2- 172.21.15.193:0/219003530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe51c109820 0x7fe51c075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:08.481 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:08.479+0000 7fe52244b640 1 --2- 172.21.15.193:0/219003530 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe51c108ed0 0x7fe51c075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:08.481 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:08.479+0000 7fe52244b640 1 --2- 172.21.15.193:0/219003530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe51c103590 0x7fe51c075140 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:08.481 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:08.479+0000 7fe52244b640 1 -- 172.21.15.193:0/219003530 >> 172.21.15.193:0/219003530 conn(0x7fe51c07b280 msgr2=0x7fe51c107020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:08.481 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:08.479+0000 7fe52244b640 1 -- 172.21.15.193:0/219003530 shutdown_connections 2024-07-26T11:09:08.482 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:08.479+0000 7fe52244b640 1 -- 172.21.15.193:0/219003530 wait complete. 2024-07-26T11:09:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:08.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:08.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:08 smithi189 ceph-mon[36790]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:09:08.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:08.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:08.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:08.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:08 smithi189 ceph-mon[36790]: from='osd.11 [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T11:09:08.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:08 smithi189 ceph-mon[36790]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T11:09:09.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:09.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:09.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:09.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:09.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:09.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:08 smithi193 ceph-mon[37473]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:09:09.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:09.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:09.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:09.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:08 smithi193 ceph-mon[37473]: from='osd.11 [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T11:09:09.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:08 smithi193 ceph-mon[37473]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T11:09:09.163 DEBUG:teuthology.orchestra.run.smithi193:osd.11> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@osd.11.service 2024-07-26T11:09:09.165 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-07-26T11:09:09.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd stat -f json 2024-07-26T11:09:09.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:09.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:09.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:09.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:09.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:09.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:08 smithi134 ceph-mon[31966]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:09:09.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:09.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:09.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:09.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:08 smithi134 ceph-mon[31966]: from='osd.11 [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T11:09:09.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:08 smithi134 ceph-mon[31966]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-07-26T11:09:09.633 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:09:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:10 smithi193 ceph-mon[37473]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T11:09:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:10 smithi193 ceph-mon[37473]: osdmap e71: 12 total, 11 up, 12 in 2024-07-26T11:09:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T11:09:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:10 smithi193 ceph-mon[37473]: from='osd.11 [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:09:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:10 smithi193 ceph-mon[37473]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:09:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:10 smithi193 ceph-mon[37473]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:09:10.711 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jul 26 11:09:10 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-11[63729]: 2024-07-26T11:09:10.480+0000 7f8d8f242640 -1 osd.11 0 waiting for initial osdmap 2024-07-26T11:09:10.711 INFO:journalctl@ceph.osd.11.smithi193.stdout:Jul 26 11:09:10 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-11[63729]: 2024-07-26T11:09:10.495+0000 7f8d8b040640 -1 osd.11 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-07-26T11:09:10.826 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:10 smithi134 ceph-mon[31966]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T11:09:10.826 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:10 smithi134 ceph-mon[31966]: osdmap e71: 12 total, 11 up, 12 in 2024-07-26T11:09:10.826 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T11:09:10.826 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:10 smithi134 ceph-mon[31966]: from='osd.11 [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:09:10.826 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:10 smithi134 ceph-mon[31966]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:09:10.826 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:10 smithi134 ceph-mon[31966]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:09:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:10 smithi189 ceph-mon[36790]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-07-26T11:09:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:10 smithi189 ceph-mon[36790]: osdmap e71: 12 total, 11 up, 12 in 2024-07-26T11:09:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T11:09:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:10 smithi189 ceph-mon[36790]: from='osd.11 [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:09:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:10 smithi189 ceph-mon[36790]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-07-26T11:09:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:10 smithi189 ceph-mon[36790]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:09:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:11 smithi193 ceph-mon[37473]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-07-26T11:09:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:11 smithi193 ceph-mon[37473]: osdmap e72: 12 total, 11 up, 12 in 2024-07-26T11:09:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T11:09:11.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T11:09:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:11 smithi134 ceph-mon[31966]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-07-26T11:09:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:11 smithi134 ceph-mon[31966]: osdmap e72: 12 total, 11 up, 12 in 2024-07-26T11:09:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T11:09:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T11:09:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:11 smithi189 ceph-mon[36790]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-07-26T11:09:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:11 smithi189 ceph-mon[36790]: osdmap e72: 12 total, 11 up, 12 in 2024-07-26T11:09:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T11:09:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T11:09:11.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.976+0000 7fcad6200640 1 -- 172.21.15.134:0/911180031 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcad0114030 msgr2=0x7fcad0116450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:11.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.976+0000 7fcad6200640 1 --2- 172.21.15.134:0/911180031 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcad0114030 0x7fcad0116450 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcac8009ca0 tx=0x7fcac802a8b0 comp rx=0 tx=0).stop 2024-07-26T11:09:11.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.977+0000 7fcad6200640 1 -- 172.21.15.134:0/911180031 shutdown_connections 2024-07-26T11:09:11.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.977+0000 7fcad6200640 1 --2- 172.21.15.134:0/911180031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcad0077cc0 0x7fcad0113af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:11.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.977+0000 7fcad6200640 1 --2- 172.21.15.134:0/911180031 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcad0114030 0x7fcad0116450 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:11.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.977+0000 7fcad6200640 1 --2- 172.21.15.134:0/911180031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad0077370 0x7fcad0077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:11.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.977+0000 7fcad6200640 1 -- 172.21.15.134:0/911180031 >> 172.21.15.134:0/911180031 conn(0x7fcad01010c0 msgr2=0x7fcad0103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:11.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.977+0000 7fcad6200640 1 -- 172.21.15.134:0/911180031 shutdown_connections 2024-07-26T11:09:11.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.977+0000 7fcad6200640 1 -- 172.21.15.134:0/911180031 wait complete. 2024-07-26T11:09:11.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.978+0000 7fcad6200640 1 Processor -- start 2024-07-26T11:09:11.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.978+0000 7fcad6200640 1 -- start start 2024-07-26T11:09:11.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.979+0000 7fcad6200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad0077370 0x7fcad01a02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:11.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.979+0000 7fcad6200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcad0077cc0 0x7fcad01a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:11.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.979+0000 7fcad6200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcad0114030 0x7fcad01a5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:11.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.979+0000 7fcad6200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcad0119960 con 0x7fcad0077cc0 2024-07-26T11:09:11.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.979+0000 7fcad6200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcad01197e0 con 0x7fcad0077370 2024-07-26T11:09:11.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.979+0000 7fcad6200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcad0119ae0 con 0x7fcad0114030 2024-07-26T11:09:11.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.979+0000 7fcacfe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcad0077cc0 0x7fcad01a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:11.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.979+0000 7fcad5800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcad0114030 0x7fcad01a5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:11.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.979+0000 7fcacfe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcad0077cc0 0x7fcad01a0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57766/0 (socket says 172.21.15.134:57766) 2024-07-26T11:09:11.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.979+0000 7fcacfe00640 1 -- 172.21.15.134:0/1512690247 learned_addr learned my addr 172.21.15.134:0/1512690247 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:09:11.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.979+0000 7fcad4e00640 1 --2- 172.21.15.134:0/1512690247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad0077370 0x7fcad01a02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:11.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.979+0000 7fcad5800640 1 -- 172.21.15.134:0/1512690247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad0077370 msgr2=0x7fcad01a02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:11.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.979+0000 7fcad5800640 1 --2- 172.21.15.134:0/1512690247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad0077370 0x7fcad01a02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:11.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.979+0000 7fcad5800640 1 -- 172.21.15.134:0/1512690247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcad0077cc0 msgr2=0x7fcad01a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:11.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.979+0000 7fcad5800640 1 --2- 172.21.15.134:0/1512690247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcad0077cc0 0x7fcad01a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:11.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.979+0000 7fcad5800640 1 -- 172.21.15.134:0/1512690247 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad01a6240 con 0x7fcad0114030 2024-07-26T11:09:11.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.980+0000 7fcad5800640 1 --2- 172.21.15.134:0/1512690247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcad0114030 0x7fcad01a5b10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcac802add0 tx=0x7fcac8030dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:11.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.980+0000 7fcacd600640 1 -- 172.21.15.134:0/1512690247 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac8003e00 con 0x7fcad0114030 2024-07-26T11:09:11.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.980+0000 7fcad6200640 1 -- 172.21.15.134:0/1512690247 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcad01a64d0 con 0x7fcad0114030 2024-07-26T11:09:11.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.980+0000 7fcacd600640 1 -- 172.21.15.134:0/1512690247 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcac8040070 con 0x7fcad0114030 2024-07-26T11:09:11.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.980+0000 7fcacd600640 1 -- 172.21.15.134:0/1512690247 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcac80459b0 con 0x7fcad0114030 2024-07-26T11:09:11.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.980+0000 7fcad6200640 1 -- 172.21.15.134:0/1512690247 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcad01acc90 con 0x7fcad0114030 2024-07-26T11:09:11.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.981+0000 7fcad6200640 1 -- 172.21.15.134:0/1512690247 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcad0077780 con 0x7fcad0114030 2024-07-26T11:09:11.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.984+0000 7fcacd600640 1 -- 172.21.15.134:0/1512690247 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fcac80046d0 con 0x7fcad0114030 2024-07-26T11:09:11.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.985+0000 7fcacd600640 1 --2- 172.21.15.134:0/1512690247 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcab80b3ca0 0x7fcab80b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:11.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.986+0000 7fcacd600640 1 -- 172.21.15.134:0/1512690247 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7342+0+0 (secure 0 0 0) 0x7fcac81049e0 con 0x7fcad0114030 2024-07-26T11:09:11.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.986+0000 7fcad4e00640 1 --2- 172.21.15.134:0/1512690247 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcab80b3ca0 0x7fcab80b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:11.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.986+0000 7fcad4e00640 1 --2- 172.21.15.134:0/1512690247 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcab80b3ca0 0x7fcab80b61b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009bd0 tx=0x7fcac4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:11.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:11.987+0000 7fcacd600640 1 -- 172.21.15.134:0/1512690247 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcac80cc360 con 0x7fcad0114030 2024-07-26T11:09:12.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:12.193+0000 7fcad6200640 1 -- 172.21.15.134:0/1512690247 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fcad010f1c0 con 0x7fcad0114030 2024-07-26T11:09:12.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:12.199+0000 7fcacd600640 1 -- 172.21.15.134:0/1512690247 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v73) ==== 74+0+133 (secure 0 0 0) 0x7fcac80cbc60 con 0x7fcad0114030 2024-07-26T11:09:12.201 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:09:12.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:12.203+0000 7fcad6200640 1 -- 172.21.15.134:0/1512690247 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcab80b3ca0 msgr2=0x7fcab80b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:12.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:12.203+0000 7fcad6200640 1 --2- 172.21.15.134:0/1512690247 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcab80b3ca0 0x7fcab80b61b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009bd0 tx=0x7fcac4005f90 comp rx=0 tx=0).stop 2024-07-26T11:09:12.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:12.203+0000 7fcad6200640 1 -- 172.21.15.134:0/1512690247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcad0114030 msgr2=0x7fcad01a5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:12.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:12.203+0000 7fcad6200640 1 --2- 172.21.15.134:0/1512690247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcad0114030 0x7fcad01a5b10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcac802add0 tx=0x7fcac8030dc0 comp rx=0 tx=0).stop 2024-07-26T11:09:12.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:12.203+0000 7fcad6200640 1 -- 172.21.15.134:0/1512690247 shutdown_connections 2024-07-26T11:09:12.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:12.203+0000 7fcad6200640 1 --2- 172.21.15.134:0/1512690247 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcab80b3ca0 0x7fcab80b61b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:12.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:12.203+0000 7fcad6200640 1 --2- 172.21.15.134:0/1512690247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcad0114030 0x7fcad01a5b10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:12.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:12.203+0000 7fcad6200640 1 --2- 172.21.15.134:0/1512690247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcad0077cc0 0x7fcad01a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:12.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:12.203+0000 7fcad6200640 1 --2- 172.21.15.134:0/1512690247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad0077370 0x7fcad01a02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:12.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:12.203+0000 7fcad6200640 1 -- 172.21.15.134:0/1512690247 >> 172.21.15.134:0/1512690247 conn(0x7fcad01010c0 msgr2=0x7fcad0116060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:12.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:12.204+0000 7fcad6200640 1 -- 172.21.15.134:0/1512690247 shutdown_connections 2024-07-26T11:09:12.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:12.204+0000 7fcad6200640 1 -- 172.21.15.134:0/1512690247 wait complete. 2024-07-26T11:09:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: purged_snaps scrub starts 2024-07-26T11:09:12.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: purged_snaps scrub ok 2024-07-26T11:09:12.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: osd.11 [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579] boot 2024-07-26T11:09:12.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: osdmap e73: 12 total, 12 up, 12 in 2024-07-26T11:09:12.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T11:09:12.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:09:12.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:12.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:12.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:09:12.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:09:12.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:09:12.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:09:12.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:12.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:12.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:12.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1512690247' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T11:09:12.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:12.846 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: purged_snaps scrub starts 2024-07-26T11:09:12.846 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: purged_snaps scrub ok 2024-07-26T11:09:12.846 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: osd.11 [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579] boot 2024-07-26T11:09:12.846 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: osdmap e73: 12 total, 12 up, 12 in 2024-07-26T11:09:12.846 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T11:09:12.846 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:09:12.846 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:12.846 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:12.846 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:09:12.846 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:09:12.846 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:09:12.847 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:09:12.847 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:12.847 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:12.847 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:12.847 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1512690247' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T11:09:12.847 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:12.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: purged_snaps scrub starts 2024-07-26T11:09:12.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: purged_snaps scrub ok 2024-07-26T11:09:12.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: osd.11 [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579] boot 2024-07-26T11:09:12.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: osdmap e73: 12 total, 12 up, 12 in 2024-07-26T11:09:12.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-07-26T11:09:12.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-07-26T11:09:12.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:12.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:12.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:09:12.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:09:12.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:09:12.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:09:12.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:12.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:12.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:12.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1512690247' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-07-26T11:09:12.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:13.142 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":73,"num_osds":12,"num_up_osds":12,"osd_up_since":1721992151,"num_in_osds":12,"osd_in_since":1721992132,"num_remapped_pgs":0} 2024-07-26T11:09:13.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T11:09:13.597 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:09:13.774 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:13 smithi134 ceph-mon[31966]: Detected new or changed devices on smithi193 2024-07-26T11:09:13.774 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:13 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 4272M 2024-07-26T11:09:13.774 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:13 smithi134 ceph-mon[31966]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T11:09:13.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:13 smithi193 ceph-mon[37473]: Detected new or changed devices on smithi193 2024-07-26T11:09:13.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:13 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 4272M 2024-07-26T11:09:13.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:13 smithi193 ceph-mon[37473]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T11:09:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:13 smithi189 ceph-mon[36790]: Detected new or changed devices on smithi193 2024-07-26T11:09:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:13 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 4272M 2024-07-26T11:09:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:13 smithi189 ceph-mon[36790]: osdmap e74: 12 total, 12 up, 12 in 2024-07-26T11:09:14.906 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:14 smithi134 ceph-mon[31966]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:14 smithi189 ceph-mon[36790]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:14 smithi193 ceph-mon[37473]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:15.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.769+0000 7f8c6e800640 1 -- 172.21.15.134:0/2923236127 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c6810af10 msgr2=0x7f8c6810b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:15.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.769+0000 7f8c6e800640 1 --2- 172.21.15.134:0/2923236127 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c6810af10 0x7f8c6810b320 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8c54009900 tx=0x7f8c5402a410 comp rx=0 tx=0).stop 2024-07-26T11:09:15.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.769+0000 7f8c6e800640 1 -- 172.21.15.134:0/2923236127 shutdown_connections 2024-07-26T11:09:15.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.769+0000 7f8c6e800640 1 --2- 172.21.15.134:0/2923236127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c6810b860 0x7f8c68112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:15.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.769+0000 7f8c6e800640 1 --2- 172.21.15.134:0/2923236127 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c6810af10 0x7f8c6810b320 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:15.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.769+0000 7f8c6e800640 1 --2- 172.21.15.134:0/2923236127 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c681055d0 0x7f8c681059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:15.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.769+0000 7f8c6e800640 1 -- 172.21.15.134:0/2923236127 >> 172.21.15.134:0/2923236127 conn(0x7f8c68101040 msgr2=0x7f8c68103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:15.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.769+0000 7f8c6e800640 1 -- 172.21.15.134:0/2923236127 shutdown_connections 2024-07-26T11:09:15.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.769+0000 7f8c6e800640 1 -- 172.21.15.134:0/2923236127 wait complete. 2024-07-26T11:09:15.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.770+0000 7f8c6e800640 1 Processor -- start 2024-07-26T11:09:15.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.771+0000 7f8c6e800640 1 -- start start 2024-07-26T11:09:15.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.771+0000 7f8c6e800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c681055d0 0x7f8c6819bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:15.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.771+0000 7f8c6e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c6810af10 0x7f8c6819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:15.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.771+0000 7f8c6e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c6810b860 0x7f8c681a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:15.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.771+0000 7f8c6e800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8c681154f0 con 0x7f8c681055d0 2024-07-26T11:09:15.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.771+0000 7f8c6e800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8c68115370 con 0x7f8c6810af10 2024-07-26T11:09:15.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.771+0000 7f8c6e800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8c68115670 con 0x7f8c6810b860 2024-07-26T11:09:15.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.771+0000 7f8c6d400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c681055d0 0x7f8c6819bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:15.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.771+0000 7f8c6d400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c681055d0 0x7f8c6819bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57784/0 (socket says 172.21.15.134:57784) 2024-07-26T11:09:15.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.771+0000 7f8c6d400640 1 -- 172.21.15.134:0/2801948227 learned_addr learned my addr 172.21.15.134:0/2801948227 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:09:15.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.772+0000 7f8c6de00640 1 --2- 172.21.15.134:0/2801948227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c6810b860 0x7f8c681a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:15.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.772+0000 7f8c6ca00640 1 --2- 172.21.15.134:0/2801948227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c6810af10 0x7f8c6819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:15.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.772+0000 7f8c6d400640 1 -- 172.21.15.134:0/2801948227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c6810b860 msgr2=0x7f8c681a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:15.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.772+0000 7f8c6d400640 1 --2- 172.21.15.134:0/2801948227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c6810b860 0x7f8c681a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:15.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.772+0000 7f8c6d400640 1 -- 172.21.15.134:0/2801948227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c6810af10 msgr2=0x7f8c6819c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:15.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.772+0000 7f8c6d400640 1 --2- 172.21.15.134:0/2801948227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c6810af10 0x7f8c6819c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:15.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.772+0000 7f8c6d400640 1 -- 172.21.15.134:0/2801948227 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c681a1ed0 con 0x7f8c681055d0 2024-07-26T11:09:15.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.772+0000 7f8c6d400640 1 --2- 172.21.15.134:0/2801948227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c681055d0 0x7f8c6819bf60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00e9f0 tx=0x7f8c5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:15.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.773+0000 7f8c66000640 1 -- 172.21.15.134:0/2801948227 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c00cd80 con 0x7f8c681055d0 2024-07-26T11:09:15.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.773+0000 7f8c66000640 1 -- 172.21.15.134:0/2801948227 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c5c0054d0 con 0x7f8c681055d0 2024-07-26T11:09:15.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.773+0000 7f8c6e800640 1 -- 172.21.15.134:0/2801948227 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c681a21c0 con 0x7f8c681055d0 2024-07-26T11:09:15.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.773+0000 7f8c66000640 1 -- 172.21.15.134:0/2801948227 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c010690 con 0x7f8c681055d0 2024-07-26T11:09:15.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.773+0000 7f8c6e800640 1 -- 172.21.15.134:0/2801948227 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c681a8920 con 0x7f8c681055d0 2024-07-26T11:09:15.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.774+0000 7f8c6e800640 1 -- 172.21.15.134:0/2801948227 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c681059e0 con 0x7f8c681055d0 2024-07-26T11:09:15.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.777+0000 7f8c66000640 1 -- 172.21.15.134:0/2801948227 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f8c5c004e30 con 0x7f8c681055d0 2024-07-26T11:09:15.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.779+0000 7f8c66000640 1 --2- 172.21.15.134:0/2801948227 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8c500b3c20 0x7f8c500b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:15.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.779+0000 7f8c66000640 1 -- 172.21.15.134:0/2801948227 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7342+0+0 (secure 0 0 0) 0x7f8c5c0e3ec0 con 0x7f8c681055d0 2024-07-26T11:09:15.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.780+0000 7f8c6ca00640 1 --2- 172.21.15.134:0/2801948227 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8c500b3c20 0x7f8c500b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:15.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.780+0000 7f8c6ca00640 1 --2- 172.21.15.134:0/2801948227 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8c500b3c20 0x7f8c500b6130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8c54009230 tx=0x7f8c54031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:15.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.781+0000 7f8c66000640 1 -- 172.21.15.134:0/2801948227 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8c5c055600 con 0x7f8c681055d0 2024-07-26T11:09:15.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.983+0000 7f8c6e800640 1 -- 172.21.15.134:0/2801948227 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8c6802e0c0 con 0x7f8c681055d0 2024-07-26T11:09:15.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.985+0000 7f8c66000640 1 -- 172.21.15.134:0/2801948227 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+18799 (secure 0 0 0) 0x7f8c5c0ab840 con 0x7f8c681055d0 2024-07-26T11:09:15.988 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:09:15.988 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":74,"fsid":"105fbe5c-4b3e-11ef-bcb7-c7b262605968","created":"2024-07-26T11:00:05.955103+0000","modified":"2024-07-26T11:09:12.501387+0000","last_up_change":"2024-07-26T11:09:11.485512+0000","last_in_change":"2024-07-26T11:08:52.341356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T11:05:01.849170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8abb1365-fc3e-4617-a9ac-76bece8cd7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6803","nonce":4183273954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6805","nonce":4183273954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6809","nonce":4183273954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6807","nonce":4183273954}]},"public_addr":"172.21.15.134:6803/4183273954","cluster_addr":"172.21.15.134:6805/4183273954","heartbeat_back_addr":"172.21.15.134:6809/4183273954","heartbeat_front_addr":"172.21.15.134:6807/4183273954","state":["exists","up"]},{"osd":1,"uuid":"6c5a7838-821e-44e1-97e4-655b9bfa9f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6801","nonce":35148049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6803","nonce":35148049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6807","nonce":35148049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6805","nonce":35148049}]},"public_addr":"172.21.15.189:6801/35148049","cluster_addr":"172.21.15.189:6803/35148049","heartbeat_back_addr":"172.21.15.189:6807/35148049","heartbeat_front_addr":"172.21.15.189:6805/35148049","state":["exists","up"]},{"osd":2,"uuid":"8bc42f7f-39fa-4d3b-bc17-21c68f9231fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6801","nonce":897516976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6803","nonce":897516976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6807","nonce":897516976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6805","nonce":897516976}]},"public_addr":"172.21.15.193:6801/897516976","cluster_addr":"172.21.15.193:6803/897516976","heartbeat_back_addr":"172.21.15.193:6807/897516976","heartbeat_front_addr":"172.21.15.193:6805/897516976","state":["exists","up"]},{"osd":3,"uuid":"0473d055-0b79-434f-9658-28269d2b1fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6811","nonce":3457190917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6813","nonce":3457190917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6817","nonce":3457190917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6815","nonce":3457190917}]},"public_addr":"172.21.15.134:6811/3457190917","cluster_addr":"172.21.15.134:6813/3457190917","heartbeat_back_addr":"172.21.15.134:6817/3457190917","heartbeat_front_addr":"172.21.15.134:6815/3457190917","state":["exists","up"]},{"osd":4,"uuid":"1d70eaae-56cc-4cdc-8f83-b88d2121f0ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6809","nonce":1786881347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6811","nonce":1786881347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6815","nonce":1786881347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6813","nonce":1786881347}]},"public_addr":"172.21.15.189:6809/1786881347","cluster_addr":"172.21.15.189:6811/1786881347","heartbeat_back_addr":"172.21.15.189:6815/1786881347","heartbeat_front_addr":"172.21.15.189:6813/1786881347","state":["exists","up"]},{"osd":5,"uuid":"a9b0b4cf-1608-45ed-b64e-afcf545d97fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6809","nonce":1092388797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6811","nonce":1092388797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6815","nonce":1092388797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6813","nonce":1092388797}]},"public_addr":"172.21.15.193:6809/1092388797","cluster_addr":"172.21.15.193:6811/1092388797","heartbeat_back_addr":"172.21.15.193:6815/1092388797","heartbeat_front_addr":"172.21.15.193:6813/1092388797","state":["exists","up"]},{"osd":6,"uuid":"0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6819","nonce":2471505438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6821","nonce":2471505438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6825","nonce":2471505438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6823","nonce":2471505438}]},"public_addr":"172.21.15.134:6819/2471505438","cluster_addr":"172.21.15.134:6821/2471505438","heartbeat_back_addr":"172.21.15.134:6825/2471505438","heartbeat_front_addr":"172.21.15.134:6823/2471505438","state":["exists","up"]},{"osd":7,"uuid":"394764a7-1976-4223-a3c0-5979d941a838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6817","nonce":1382694911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6819","nonce":1382694911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6823","nonce":1382694911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6821","nonce":1382694911}]},"public_addr":"172.21.15.189:6817/1382694911","cluster_addr":"172.21.15.189:6819/1382694911","heartbeat_back_addr":"172.21.15.189:6823/1382694911","heartbeat_front_addr":"172.21.15.189:6821/1382694911","state":["exists","up"]},{"osd":8,"uuid":"1c218082-0b4c-41c8-92b9-453ae8d15a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6817","nonce":896348138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6819","nonce":896348138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6823","nonce":896348138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6821","nonce":896348138}]},"public_addr":"172.21.15.193:6817/896348138","cluster_addr":"172.21.15.193:6819/896348138","heartbeat_back_addr":"172.21.15.193:6823/896348138","heartbeat_front_addr":"172.21.15.193:6821/896348138","state":["exists","up"]},{"osd":9,"uuid":"48f39082-4fa9-4443-b5b7-b141c10d9e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6827","nonce":511237025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6829","nonce":511237025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6833","nonce":511237025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6831","nonce":511237025}]},"public_addr":"172.21.15.134:6827/511237025","cluster_addr":"172.21.15.134:6829/511237025","heartbeat_back_addr":"172.21.15.134:6833/511237025","heartbeat_front_addr":"172.21.15.134:6831/511237025","state":["exists","up"]},{"osd":10,"uuid":"95157766-1856-42f7-aea1-9cf83b77c8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6825","nonce":431740858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6827","nonce":431740858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6831","nonce":431740858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6829","nonce":431740858}]},"public_addr":"172.21.15.189:6825/431740858","cluster_addr":"172.21.15.189:6827/431740858","heartbeat_back_addr":"172.21.15.189:6831/431740858","heartbeat_front_addr":"172.21.15.189:6829/431740858","state":["exists","up"]},{"osd":11,"uuid":"244fadc7-f079-4736-8e7e-c0ce5ae6f43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6825","nonce":1594590579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6827","nonce":1594590579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6831","nonce":1594590579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6829","nonce":1594590579}]},"public_addr":"172.21.15.193:6825/1594590579","cluster_addr":"172.21.15.193:6827/1594590579","heartbeat_back_addr":"172.21.15.193:6831/1594590579","heartbeat_front_addr":"172.21.15.193:6829/1594590579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:11.712203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:35.845013+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:58.165609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:31.929746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:56.941555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:20.688071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:54.681514+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:20.789384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:44.568246+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:18.644310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:45.365351+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:09:09.663805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3790862248":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6800/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:0/2906927791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3068153391":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3056560791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6801/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/1310359922":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/2318666906":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/2901579347":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3404218719":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3598071145":"2024-07-27T11:02:33.711793+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:09:15.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.988+0000 7f8c6e800640 1 -- 172.21.15.134:0/2801948227 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8c500b3c20 msgr2=0x7f8c500b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:15.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.988+0000 7f8c6e800640 1 --2- 172.21.15.134:0/2801948227 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8c500b3c20 0x7f8c500b6130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8c54009230 tx=0x7f8c54031400 comp rx=0 tx=0).stop 2024-07-26T11:09:15.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.988+0000 7f8c6e800640 1 -- 172.21.15.134:0/2801948227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c681055d0 msgr2=0x7f8c6819bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:15.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.988+0000 7f8c6e800640 1 --2- 172.21.15.134:0/2801948227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c681055d0 0x7f8c6819bf60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c00e9f0 tx=0x7f8c5c00edf0 comp rx=0 tx=0).stop 2024-07-26T11:09:15.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.988+0000 7f8c6e800640 1 -- 172.21.15.134:0/2801948227 shutdown_connections 2024-07-26T11:09:15.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.989+0000 7f8c6e800640 1 --2- 172.21.15.134:0/2801948227 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8c500b3c20 0x7f8c500b6130 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:15.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.989+0000 7f8c6e800640 1 --2- 172.21.15.134:0/2801948227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8c6810b860 0x7f8c681a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:15.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.989+0000 7f8c6e800640 1 --2- 172.21.15.134:0/2801948227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c6810af10 0x7f8c6819c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:15.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.989+0000 7f8c6e800640 1 --2- 172.21.15.134:0/2801948227 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8c681055d0 0x7f8c6819bf60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:15.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.989+0000 7f8c6e800640 1 -- 172.21.15.134:0/2801948227 >> 172.21.15.134:0/2801948227 conn(0x7f8c68101040 msgr2=0x7f8c68103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:15.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.989+0000 7f8c6e800640 1 -- 172.21.15.134:0/2801948227 shutdown_connections 2024-07-26T11:09:15.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:15.989+0000 7f8c6e800640 1 -- 172.21.15.134:0/2801948227 wait complete. 2024-07-26T11:09:16.960 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-26T11:05:01.849170+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-07-26T11:09:16.961 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-07-26T11:09:17.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:16 smithi134 ceph-mon[31966]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:17.220 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:16 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2801948227' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:09:17.221 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:16 smithi189 ceph-mon[36790]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:17.221 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:16 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2801948227' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:09:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:16 smithi193 ceph-mon[37473]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:16 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2801948227' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:09:17.427 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:09:19.118 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:18 smithi134 ceph-mon[31966]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:18 smithi189 ceph-mon[36790]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:18 smithi193 ceph-mon[37473]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:19.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.734+0000 7f74e1800640 1 -- 172.21.15.134:0/2540487420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74dc1055d0 msgr2=0x7f74dc1059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:19.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.734+0000 7f74e1800640 1 --2- 172.21.15.134:0/2540487420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74dc1055d0 0x7f74dc1059e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f74cc009860 tx=0x7f74cc02a340 comp rx=0 tx=0).stop 2024-07-26T11:09:19.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.734+0000 7f74e1800640 1 -- 172.21.15.134:0/2540487420 shutdown_connections 2024-07-26T11:09:19.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.734+0000 7f74e1800640 1 --2- 172.21.15.134:0/2540487420 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74dc10b860 0x7f74dc112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:19.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.734+0000 7f74e1800640 1 --2- 172.21.15.134:0/2540487420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74dc10af10 0x7f74dc10b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:19.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.734+0000 7f74e1800640 1 --2- 172.21.15.134:0/2540487420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74dc1055d0 0x7f74dc1059e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:19.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.734+0000 7f74e1800640 1 -- 172.21.15.134:0/2540487420 >> 172.21.15.134:0/2540487420 conn(0x7f74dc101040 msgr2=0x7f74dc103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:19.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.734+0000 7f74e1800640 1 -- 172.21.15.134:0/2540487420 shutdown_connections 2024-07-26T11:09:19.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.734+0000 7f74e1800640 1 -- 172.21.15.134:0/2540487420 wait complete. 2024-07-26T11:09:19.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.735+0000 7f74e1800640 1 Processor -- start 2024-07-26T11:09:19.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.735+0000 7f74e1800640 1 -- start start 2024-07-26T11:09:19.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.736+0000 7f74e1800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74dc1055d0 0x7f74dc19bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:19.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.736+0000 7f74e1800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74dc10af10 0x7f74dc19c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:19.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.736+0000 7f74e1800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74dc10b860 0x7f74dc1a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:19.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.736+0000 7f74e1800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc115510 con 0x7f74dc1055d0 2024-07-26T11:09:19.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.736+0000 7f74e1800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc115390 con 0x7f74dc10b860 2024-07-26T11:09:19.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.736+0000 7f74e1800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc115690 con 0x7f74dc10af10 2024-07-26T11:09:19.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.736+0000 7f74dbe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74dc1055d0 0x7f74dc19bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:19.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.736+0000 7f74dbe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74dc1055d0 0x7f74dc19bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57794/0 (socket says 172.21.15.134:57794) 2024-07-26T11:09:19.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.736+0000 7f74dbe00640 1 -- 172.21.15.134:0/402543391 learned_addr learned my addr 172.21.15.134:0/402543391 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:09:19.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.736+0000 7f74d3400640 1 --2- 172.21.15.134:0/402543391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74dc10af10 0x7f74dc19c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:19.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.736+0000 7f74e0e00640 1 --2- 172.21.15.134:0/402543391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74dc10b860 0x7f74dc1a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:19.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.737+0000 7f74d3400640 1 -- 172.21.15.134:0/402543391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74dc10b860 msgr2=0x7f74dc1a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:19.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.737+0000 7f74d3400640 1 --2- 172.21.15.134:0/402543391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74dc10b860 0x7f74dc1a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:19.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.737+0000 7f74d3400640 1 -- 172.21.15.134:0/402543391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74dc1055d0 msgr2=0x7f74dc19bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:19.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.737+0000 7f74d3400640 1 --2- 172.21.15.134:0/402543391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74dc1055d0 0x7f74dc19bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:19.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.737+0000 7f74d3400640 1 -- 172.21.15.134:0/402543391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74dc1a1e70 con 0x7f74dc10af10 2024-07-26T11:09:19.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.737+0000 7f74d3400640 1 --2- 172.21.15.134:0/402543391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74dc10af10 0x7f74dc19c4a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f74dc06c270 tx=0x7f74c400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:19.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.737+0000 7f74d9600640 1 -- 172.21.15.134:0/402543391 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c400cc40 con 0x7f74dc10af10 2024-07-26T11:09:19.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.738+0000 7f74e1800640 1 -- 172.21.15.134:0/402543391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74dc1a2160 con 0x7f74dc10af10 2024-07-26T11:09:19.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.738+0000 7f74d9600640 1 -- 172.21.15.134:0/402543391 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74c400cde0 con 0x7f74dc10af10 2024-07-26T11:09:19.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.738+0000 7f74d9600640 1 -- 172.21.15.134:0/402543391 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c4018760 con 0x7f74dc10af10 2024-07-26T11:09:19.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.738+0000 7f74e1800640 1 -- 172.21.15.134:0/402543391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74dc1a88c0 con 0x7f74dc10af10 2024-07-26T11:09:19.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.739+0000 7f74e1800640 1 -- 172.21.15.134:0/402543391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74dc1059e0 con 0x7f74dc10af10 2024-07-26T11:09:19.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.741+0000 7f74d9600640 1 -- 172.21.15.134:0/402543391 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f74c4010a90 con 0x7f74dc10af10 2024-07-26T11:09:19.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.743+0000 7f74d9600640 1 --2- 172.21.15.134:0/402543391 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f74c00b3c20 0x7f74c00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:19.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.743+0000 7f74d9600640 1 -- 172.21.15.134:0/402543391 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7342+0+0 (secure 0 0 0) 0x7f74c40e3bc0 con 0x7f74dc10af10 2024-07-26T11:09:19.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.743+0000 7f74dbe00640 1 --2- 172.21.15.134:0/402543391 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f74c00b3c20 0x7f74c00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:19.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.744+0000 7f74dbe00640 1 --2- 172.21.15.134:0/402543391 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f74c00b3c20 0x7f74c00b6130 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0095f0 tx=0x7f74cc030340 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:19.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.744+0000 7f74d9600640 1 -- 172.21.15.134:0/402543391 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f74c40aaef0 con 0x7f74dc10af10 2024-07-26T11:09:19.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.944+0000 7f74e1800640 1 -- 172.21.15.134:0/402543391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f74dc02e0c0 con 0x7f74dc10af10 2024-07-26T11:09:19.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.945+0000 7f74d9600640 1 -- 172.21.15.134:0/402543391 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) ==== 93+0+10 (secure 0 0 0) 0x7f74c40af440 con 0x7f74dc10af10 2024-07-26T11:09:19.947 INFO:teuthology.orchestra.run.smithi134.stdout:pg_num: 1 2024-07-26T11:09:19.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.947+0000 7f74e1800640 1 -- 172.21.15.134:0/402543391 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f74c00b3c20 msgr2=0x7f74c00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:19.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.947+0000 7f74e1800640 1 --2- 172.21.15.134:0/402543391 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f74c00b3c20 0x7f74c00b6130 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0095f0 tx=0x7f74cc030340 comp rx=0 tx=0).stop 2024-07-26T11:09:19.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.947+0000 7f74e1800640 1 -- 172.21.15.134:0/402543391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74dc10af10 msgr2=0x7f74dc19c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:19.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.947+0000 7f74e1800640 1 --2- 172.21.15.134:0/402543391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74dc10af10 0x7f74dc19c4a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f74dc06c270 tx=0x7f74c400ee50 comp rx=0 tx=0).stop 2024-07-26T11:09:19.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.948+0000 7f74e1800640 1 -- 172.21.15.134:0/402543391 shutdown_connections 2024-07-26T11:09:19.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.948+0000 7f74e1800640 1 --2- 172.21.15.134:0/402543391 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f74c00b3c20 0x7f74c00b6130 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:19.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.948+0000 7f74e1800640 1 --2- 172.21.15.134:0/402543391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74dc10af10 0x7f74dc19c4a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:19.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.948+0000 7f74e1800640 1 --2- 172.21.15.134:0/402543391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74dc10b860 0x7f74dc1a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:19.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.948+0000 7f74e1800640 1 --2- 172.21.15.134:0/402543391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f74dc1055d0 0x7f74dc19bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:19.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.948+0000 7f74e1800640 1 -- 172.21.15.134:0/402543391 >> 172.21.15.134:0/402543391 conn(0x7f74dc101040 msgr2=0x7f74dc103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:19.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.948+0000 7f74e1800640 1 -- 172.21.15.134:0/402543391 shutdown_connections 2024-07-26T11:09:19.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:19.948+0000 7f74e1800640 1 -- 172.21.15.134:0/402543391 wait complete. 2024-07-26T11:09:20.726 INFO:tasks.cephadm:Adding mds.a on smithi134 2024-07-26T11:09:20.726 INFO:tasks.cephadm:Adding mds.d on smithi134 2024-07-26T11:09:20.726 INFO:tasks.cephadm:Adding mds.g on smithi134 2024-07-26T11:09:20.726 INFO:tasks.cephadm:Adding mds.j on smithi134 2024-07-26T11:09:20.726 INFO:tasks.cephadm:Adding mds.b on smithi189 2024-07-26T11:09:20.726 INFO:tasks.cephadm:Adding mds.e on smithi189 2024-07-26T11:09:20.726 INFO:tasks.cephadm:Adding mds.h on smithi189 2024-07-26T11:09:20.727 INFO:tasks.cephadm:Adding mds.k on smithi189 2024-07-26T11:09:20.727 INFO:tasks.cephadm:Adding mds.c on smithi193 2024-07-26T11:09:20.727 INFO:tasks.cephadm:Adding mds.f on smithi193 2024-07-26T11:09:20.727 INFO:tasks.cephadm:Adding mds.i on smithi193 2024-07-26T11:09:20.727 INFO:tasks.cephadm:Adding mds.l on smithi193 2024-07-26T11:09:20.727 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph orch apply mds all '12;smithi134=a;smithi134=d;smithi134=g;smithi134=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l' 2024-07-26T11:09:20.990 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.c/config 2024-07-26T11:09:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:20 smithi193 ceph-mon[37473]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:21.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:20 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/402543391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T11:09:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:20 smithi134 ceph-mon[31966]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:21.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:20 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/402543391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T11:09:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:20 smithi189 ceph-mon[36790]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:20 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/402543391' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T11:09:22.674 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.665+0000 7fb1f5838640 1 -- 172.21.15.193:0/3270699420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1f0109820 msgr2=0x7fb1f0110050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:22.674 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.665+0000 7fb1f5838640 1 --2- 172.21.15.193:0/3270699420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1f0109820 0x7fb1f0110050 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8009900 tx=0x7fb1e802a410 comp rx=0 tx=0).stop 2024-07-26T11:09:22.674 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.665+0000 7fb1f5838640 1 -- 172.21.15.193:0/3270699420 shutdown_connections 2024-07-26T11:09:22.674 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.665+0000 7fb1f5838640 1 --2- 172.21.15.193:0/3270699420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1f0109820 0x7fb1f0110050 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:22.674 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.665+0000 7fb1f5838640 1 --2- 172.21.15.193:0/3270699420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1f0108ed0 0x7fb1f01092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:22.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.665+0000 7fb1f5838640 1 --2- 172.21.15.193:0/3270699420 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1f0103590 0x7fb1f01039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:22.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.665+0000 7fb1f5838640 1 -- 172.21.15.193:0/3270699420 >> 172.21.15.193:0/3270699420 conn(0x7fb1f007b280 msgr2=0x7fb1f007b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:22.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.665+0000 7fb1f5838640 1 -- 172.21.15.193:0/3270699420 shutdown_connections 2024-07-26T11:09:22.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.665+0000 7fb1f5838640 1 -- 172.21.15.193:0/3270699420 wait complete. 2024-07-26T11:09:22.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1f5838640 1 Processor -- start 2024-07-26T11:09:22.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1f5838640 1 -- start start 2024-07-26T11:09:22.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1f5838640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1f0103590 0x7fb1f019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:22.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1f5838640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1f0108ed0 0x7fb1f019a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:22.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1f5838640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1f0109820 0x7fb1f019f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:22.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1f5838640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f01135a0 con 0x7fb1f0108ed0 2024-07-26T11:09:22.675 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1f5838640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f0113420 con 0x7fb1f0109820 2024-07-26T11:09:22.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1f5838640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb1f0113720 con 0x7fb1f0103590 2024-07-26T11:09:22.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1eeffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1f0103590 0x7fb1f019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:22.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1eeffd640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1f0103590 0x7fb1f019a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:33282/0 (socket says 172.21.15.193:33282) 2024-07-26T11:09:22.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1eeffd640 1 -- 172.21.15.193:0/3737802151 learned_addr learned my addr 172.21.15.193:0/3737802151 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-07-26T11:09:22.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1ef7fe640 1 --2- 172.21.15.193:0/3737802151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1f0109820 0x7fb1f019f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:22.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1eeffd640 1 -- 172.21.15.193:0/3737802151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1f0109820 msgr2=0x7fb1f019f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:22.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1eeffd640 1 --2- 172.21.15.193:0/3737802151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1f0109820 0x7fb1f019f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:22.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1eeffd640 1 -- 172.21.15.193:0/3737802151 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1f0108ed0 msgr2=0x7fb1f019a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:22.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1eeffd640 1 --2- 172.21.15.193:0/3737802151 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1f0108ed0 0x7fb1f019a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:22.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.666+0000 7fb1eeffd640 1 -- 172.21.15.193:0/3737802151 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1f019ffe0 con 0x7fb1f0103590 2024-07-26T11:09:22.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.667+0000 7fb1eeffd640 1 --2- 172.21.15.193:0/3737802151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1f0103590 0x7fb1f019a070 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb1e000ba50 tx=0x7fb1e000bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:22.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.667+0000 7fb1f4836640 1 -- 172.21.15.193:0/3737802151 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0019070 con 0x7fb1f0103590 2024-07-26T11:09:22.676 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.667+0000 7fb1f5838640 1 -- 172.21.15.193:0/3737802151 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1f01a02d0 con 0x7fb1f0103590 2024-07-26T11:09:22.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.667+0000 7fb1f4836640 1 -- 172.21.15.193:0/3737802151 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1e000cbe0 con 0x7fb1f0103590 2024-07-26T11:09:22.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.667+0000 7fb1f4836640 1 -- 172.21.15.193:0/3737802151 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e00075d0 con 0x7fb1f0103590 2024-07-26T11:09:22.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.667+0000 7fb1f5838640 1 -- 172.21.15.193:0/3737802151 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1f01a6a30 con 0x7fb1f0103590 2024-07-26T11:09:22.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.668+0000 7fb1f5838640 1 -- 172.21.15.193:0/3737802151 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1f0103a20 con 0x7fb1f0103590 2024-07-26T11:09:22.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.670+0000 7fb1f4836640 1 -- 172.21.15.193:0/3737802151 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fb1e001c070 con 0x7fb1f0103590 2024-07-26T11:09:22.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.671+0000 7fb1f4836640 1 --2- 172.21.15.193:0/3737802151 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1d80b3c20 0x7fb1d80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:22.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.671+0000 7fb1f4836640 1 -- 172.21.15.193:0/3737802151 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7342+0+0 (secure 0 0 0) 0x7fb1e00e33f0 con 0x7fb1f0103590 2024-07-26T11:09:22.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.671+0000 7fb1ee7fc640 1 --2- 172.21.15.193:0/3737802151 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1d80b3c20 0x7fb1d80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:22.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.672+0000 7fb1ee7fc640 1 --2- 172.21.15.193:0/3737802151 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1d80b3c20 0x7fb1d80b6130 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0105ef0 tx=0x7fb1e4006c90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:22.677 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.674+0000 7fb1f4836640 1 -- 172.21.15.193:0/3737802151 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb1e00aae10 con 0x7fb1f0103590 2024-07-26T11:09:22.861 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.859+0000 7fb1f5838640 1 -- 172.21.15.193:0/3737802151 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi134=a;smithi134=d;smithi134=g;smithi134=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}) -- 0x7fb1f0063330 con 0x7fb1d80b3c20 2024-07-26T11:09:22.879 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.877+0000 7fb1f4836640 1 -- 172.21.15.193:0/3737802151 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fb1f0063330 con 0x7fb1d80b3c20 2024-07-26T11:09:22.880 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mds.all update... 2024-07-26T11:09:22.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.880+0000 7fb1f5838640 1 -- 172.21.15.193:0/3737802151 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1d80b3c20 msgr2=0x7fb1d80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:22.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.880+0000 7fb1f5838640 1 --2- 172.21.15.193:0/3737802151 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1d80b3c20 0x7fb1d80b6130 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0105ef0 tx=0x7fb1e4006c90 comp rx=0 tx=0).stop 2024-07-26T11:09:22.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.880+0000 7fb1f5838640 1 -- 172.21.15.193:0/3737802151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1f0103590 msgr2=0x7fb1f019a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:22.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.880+0000 7fb1f5838640 1 --2- 172.21.15.193:0/3737802151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1f0103590 0x7fb1f019a070 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb1e000ba50 tx=0x7fb1e000bf10 comp rx=0 tx=0).stop 2024-07-26T11:09:22.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.880+0000 7fb1f5838640 1 -- 172.21.15.193:0/3737802151 shutdown_connections 2024-07-26T11:09:22.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.880+0000 7fb1f5838640 1 --2- 172.21.15.193:0/3737802151 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1d80b3c20 0x7fb1d80b6130 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:22.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.880+0000 7fb1f5838640 1 --2- 172.21.15.193:0/3737802151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1f0109820 0x7fb1f019f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:22.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.880+0000 7fb1f5838640 1 --2- 172.21.15.193:0/3737802151 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1f0108ed0 0x7fb1f019a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:22.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.880+0000 7fb1f5838640 1 --2- 172.21.15.193:0/3737802151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1f0103590 0x7fb1f019a070 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:22.882 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.880+0000 7fb1f5838640 1 -- 172.21.15.193:0/3737802151 >> 172.21.15.193:0/3737802151 conn(0x7fb1f007b280 msgr2=0x7fb1f0107080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:22.883 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.880+0000 7fb1f5838640 1 -- 172.21.15.193:0/3737802151 shutdown_connections 2024-07-26T11:09:22.883 INFO:teuthology.orchestra.run.smithi193.stderr:2024-07-26T11:09:22.880+0000 7fb1f5838640 1 -- 172.21.15.193:0/3737802151 wait complete. 2024-07-26T11:09:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:22 smithi134 ceph-mon[31966]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:23.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:22 smithi189 ceph-mon[36790]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:23.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:23.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:22 smithi193 ceph-mon[37473]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:09:23.524 DEBUG:teuthology.orchestra.run.smithi134:mds.a> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service 2024-07-26T11:09:23.532 DEBUG:teuthology.orchestra.run.smithi134:mds.d> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service 2024-07-26T11:09:23.579 DEBUG:teuthology.orchestra.run.smithi134:mds.g> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service 2024-07-26T11:09:23.626 DEBUG:teuthology.orchestra.run.smithi134:mds.j> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.j.service 2024-07-26T11:09:23.674 DEBUG:teuthology.orchestra.run.smithi189:mds.b> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.b.service 2024-07-26T11:09:23.676 DEBUG:teuthology.orchestra.run.smithi189:mds.e> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service 2024-07-26T11:09:23.719 DEBUG:teuthology.orchestra.run.smithi189:mds.h> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.h.service 2024-07-26T11:09:23.761 DEBUG:teuthology.orchestra.run.smithi189:mds.k> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.k.service 2024-07-26T11:09:23.804 DEBUG:teuthology.orchestra.run.smithi193:mds.c> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.c.service 2024-07-26T11:09:23.807 DEBUG:teuthology.orchestra.run.smithi193:mds.f> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.f.service 2024-07-26T11:09:23.849 DEBUG:teuthology.orchestra.run.smithi193:mds.i> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.i.service 2024-07-26T11:09:23.892 DEBUG:teuthology.orchestra.run.smithi193:mds.l> sudo journalctl -f -n 0 -u ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.l.service 2024-07-26T11:09:23.936 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-07-26T11:09:23.936 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:09:24.120 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:23 smithi189 ceph-mon[36790]: from='client.24631 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi134=a;smithi134=d;smithi134=g;smithi134=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:09:24.120 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:23 smithi189 ceph-mon[36790]: Saving service mds.all spec with placement smithi134=a;smithi134=d;smithi134=g;smithi134=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-07-26T11:09:24.120 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:24.121 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:24.121 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:24.121 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:24.121 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:24.121 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:23 smithi134 ceph-mon[31966]: from='client.24631 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi134=a;smithi134=d;smithi134=g;smithi134=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:09:24.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:23 smithi134 ceph-mon[31966]: Saving service mds.all spec with placement smithi134=a;smithi134=d;smithi134=g;smithi134=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-07-26T11:09:24.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:24.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:24.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:24.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:24.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:24.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:24.229 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:23 smithi193 ceph-mon[37473]: from='client.24631 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi134=a;smithi134=d;smithi134=g;smithi134=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:09:24.229 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:23 smithi193 ceph-mon[37473]: Saving service mds.all spec with placement smithi134=a;smithi134=d;smithi134=g;smithi134=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-07-26T11:09:24.230 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:24.230 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:09:24.230 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:24.230 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:24.230 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:24.230 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:24.382 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:09:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:24 smithi134 ceph-mon[31966]: Deploying daemon mds.l on smithi193 2024-07-26T11:09:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:24 smithi134 ceph-mon[31966]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:24 smithi189 ceph-mon[36790]: Deploying daemon mds.l on smithi193 2024-07-26T11:09:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:24 smithi189 ceph-mon[36790]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:25.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:24 smithi193 ceph-mon[37473]: Deploying daemon mds.l on smithi193 2024-07-26T11:09:25.246 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:24 smithi193 ceph-mon[37473]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:25.926 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jul 26 11:09:25 smithi193 systemd[1]: Starting Ceph mds.l for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:09:26.204 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jul 26 11:09:25 smithi193 podman[68084]: 2024-07-26 11:09:25.826840943 +0000 UTC m=+0.017014070 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:09:26.204 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jul 26 11:09:25 smithi193 podman[68084]: 2024-07-26 11:09:25.939489865 +0000 UTC m=+0.129663016 container create b6d485e463b99d01b778adb48682207ab61914e576ecd2103962e354e820404a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-l, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-07-26T11:09:26.204 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jul 26 11:09:26 smithi193 podman[68084]: 2024-07-26 11:09:26.203088959 +0000 UTC m=+0.393262069 container init b6d485e463b99d01b778adb48682207ab61914e576ecd2103962e354e820404a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-l, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:09:26.524 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jul 26 11:09:26 smithi193 podman[68084]: 2024-07-26 11:09:26.205838159 +0000 UTC m=+0.396011280 container start b6d485e463b99d01b778adb48682207ab61914e576ecd2103962e354e820404a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-l, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:09:26.525 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jul 26 11:09:26 smithi193 ceph-mds[68103]: -- 172.21.15.193:0/4034123138 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560551f99860 con 0x56055613b180 2024-07-26T11:09:26.525 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jul 26 11:09:26 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-l[68099]: starting mds.l at 2024-07-26T11:09:26.525 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jul 26 11:09:26 smithi193 bash[68084]: b6d485e463b99d01b778adb48682207ab61914e576ecd2103962e354e820404a 2024-07-26T11:09:26.525 INFO:journalctl@ceph.mds.l.smithi193.stdout:Jul 26 11:09:26 smithi193 systemd[1]: Started Ceph mds.l for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:09:26.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.687+0000 7fe046400640 1 -- 172.21.15.134:0/2486071613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0401041f0 msgr2=0x7fe040104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:26.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.687+0000 7fe046400640 1 --2- 172.21.15.134:0/2486071613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0401041f0 0x7fe040104600 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe030009ca0 tx=0x7fe03002a8b0 comp rx=0 tx=0).stop 2024-07-26T11:09:26.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.688+0000 7fe046400640 1 -- 172.21.15.134:0/2486071613 shutdown_connections 2024-07-26T11:09:26.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.688+0000 7fe046400640 1 --2- 172.21.15.134:0/2486071613 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe04010d650 0x7fe04010fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:26.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.688+0000 7fe046400640 1 --2- 172.21.15.134:0/2486071613 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe040104b40 0x7fe04010d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:26.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.688+0000 7fe046400640 1 --2- 172.21.15.134:0/2486071613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0401041f0 0x7fe040104600 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:26.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.688+0000 7fe046400640 1 -- 172.21.15.134:0/2486071613 >> 172.21.15.134:0/2486071613 conn(0x7fe0400fdcf0 msgr2=0x7fe040100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:26.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.688+0000 7fe046400640 1 -- 172.21.15.134:0/2486071613 shutdown_connections 2024-07-26T11:09:26.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.688+0000 7fe046400640 1 -- 172.21.15.134:0/2486071613 wait complete. 2024-07-26T11:09:26.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.689+0000 7fe046400640 1 Processor -- start 2024-07-26T11:09:26.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.689+0000 7fe046400640 1 -- start start 2024-07-26T11:09:26.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.689+0000 7fe046400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0401041f0 0x7fe04019bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:26.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.690+0000 7fe046400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe040104b40 0x7fe04019c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:26.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.690+0000 7fe046400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe04010d650 0x7fe0401a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:26.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.690+0000 7fe046400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe0401132e0 con 0x7fe040104b40 2024-07-26T11:09:26.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.690+0000 7fe046400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe040113160 con 0x7fe0401041f0 2024-07-26T11:09:26.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.690+0000 7fe046400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe040113460 con 0x7fe04010d650 2024-07-26T11:09:26.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.690+0000 7fe045000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0401041f0 0x7fe04019bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:26.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.690+0000 7fe045000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0401041f0 0x7fe04019bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:50212/0 (socket says 172.21.15.134:50212) 2024-07-26T11:09:26.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.690+0000 7fe045000640 1 -- 172.21.15.134:0/1352013425 learned_addr learned my addr 172.21.15.134:0/1352013425 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:09:26.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.690+0000 7fe045a00640 1 --2- 172.21.15.134:0/1352013425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe04010d650 0x7fe0401a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:26.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.690+0000 7fe03fe00640 1 --2- 172.21.15.134:0/1352013425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe040104b40 0x7fe04019c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:26.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.690+0000 7fe045000640 1 -- 172.21.15.134:0/1352013425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe04010d650 msgr2=0x7fe0401a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:26.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.690+0000 7fe045000640 1 --2- 172.21.15.134:0/1352013425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe04010d650 0x7fe0401a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:26.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.690+0000 7fe045000640 1 -- 172.21.15.134:0/1352013425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe040104b40 msgr2=0x7fe04019c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:26.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.690+0000 7fe045000640 1 --2- 172.21.15.134:0/1352013425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe040104b40 0x7fe04019c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:26.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.690+0000 7fe045000640 1 -- 172.21.15.134:0/1352013425 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0401a1ec0 con 0x7fe0401041f0 2024-07-26T11:09:26.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.690+0000 7fe045a00640 1 --2- 172.21.15.134:0/1352013425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe04010d650 0x7fe0401a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:09:26.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.691+0000 7fe045000640 1 --2- 172.21.15.134:0/1352013425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0401041f0 0x7fe04019bf50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe030006010 tx=0x7fe030030cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:26.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.691+0000 7fe03d600640 1 -- 172.21.15.134:0/1352013425 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe030004040 con 0x7fe0401041f0 2024-07-26T11:09:26.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.691+0000 7fe03d600640 1 -- 172.21.15.134:0/1352013425 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe030040070 con 0x7fe0401041f0 2024-07-26T11:09:26.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.691+0000 7fe046400640 1 -- 172.21.15.134:0/1352013425 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0401a2150 con 0x7fe0401041f0 2024-07-26T11:09:26.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.691+0000 7fe03d600640 1 -- 172.21.15.134:0/1352013425 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe030045b50 con 0x7fe0401041f0 2024-07-26T11:09:26.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.691+0000 7fe046400640 1 -- 172.21.15.134:0/1352013425 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0401a8910 con 0x7fe0401041f0 2024-07-26T11:09:26.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.692+0000 7fe046400640 1 -- 172.21.15.134:0/1352013425 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe040104600 con 0x7fe0401041f0 2024-07-26T11:09:26.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.695+0000 7fe03d600640 1 -- 172.21.15.134:0/1352013425 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fe030045cf0 con 0x7fe0401041f0 2024-07-26T11:09:26.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.696+0000 7fe03d600640 1 --2- 172.21.15.134:0/1352013425 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe0240b3c20 0x7fe0240b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:26.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.697+0000 7fe03d600640 1 -- 172.21.15.134:0/1352013425 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7342+0+0 (secure 0 0 0) 0x7fe030104bf0 con 0x7fe0401041f0 2024-07-26T11:09:26.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.697+0000 7fe03fe00640 1 --2- 172.21.15.134:0/1352013425 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe0240b3c20 0x7fe0240b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:26.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.697+0000 7fe03fe00640 1 --2- 172.21.15.134:0/1352013425 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe0240b3c20 0x7fe0240b6130 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe0401013f0 tx=0x7fe028005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:26.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.700+0000 7fe03d600640 1 -- 172.21.15.134:0/1352013425 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe0300cc570 con 0x7fe0401041f0 2024-07-26T11:09:26.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.944+0000 7fe046400640 1 -- 172.21.15.134:0/1352013425 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0401a8bb0 con 0x7fe0401041f0 2024-07-26T11:09:26.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.945+0000 7fe03d600640 1 -- 172.21.15.134:0/1352013425 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1438 (secure 0 0 0) 0x7fe0300cbe70 con 0x7fe0401041f0 2024-07-26T11:09:26.947 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:09:26.947 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":2,"btime":"2024-07-26T11:09:26:243843+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24637,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-07-26T11:09:26.948 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 2 2024-07-26T11:09:26.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.948+0000 7fe046400640 1 -- 172.21.15.134:0/1352013425 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe0240b3c20 msgr2=0x7fe0240b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:26.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.948+0000 7fe046400640 1 --2- 172.21.15.134:0/1352013425 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe0240b3c20 0x7fe0240b6130 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe0401013f0 tx=0x7fe028005f90 comp rx=0 tx=0).stop 2024-07-26T11:09:26.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.948+0000 7fe046400640 1 -- 172.21.15.134:0/1352013425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0401041f0 msgr2=0x7fe04019bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:26.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.948+0000 7fe046400640 1 --2- 172.21.15.134:0/1352013425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0401041f0 0x7fe04019bf50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe030006010 tx=0x7fe030030cf0 comp rx=0 tx=0).stop 2024-07-26T11:09:26.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.948+0000 7fe046400640 1 -- 172.21.15.134:0/1352013425 shutdown_connections 2024-07-26T11:09:26.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.948+0000 7fe046400640 1 --2- 172.21.15.134:0/1352013425 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe0240b3c20 0x7fe0240b6130 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:26.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.948+0000 7fe046400640 1 --2- 172.21.15.134:0/1352013425 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe04010d650 0x7fe0401a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:26.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.948+0000 7fe046400640 1 --2- 172.21.15.134:0/1352013425 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe040104b40 0x7fe04019c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:26.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.948+0000 7fe046400640 1 --2- 172.21.15.134:0/1352013425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0401041f0 0x7fe04019bf50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:26.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.948+0000 7fe046400640 1 -- 172.21.15.134:0/1352013425 >> 172.21.15.134:0/1352013425 conn(0x7fe0400fdcf0 msgr2=0x7fe040101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:26.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.948+0000 7fe046400640 1 -- 172.21.15.134:0/1352013425 shutdown_connections 2024-07-26T11:09:26.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:26.948+0000 7fe046400640 1 -- 172.21.15.134:0/1352013425 wait complete. 2024-07-26T11:09:27.189 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:26 smithi189 ceph-mon[36790]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:27.189 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:26 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.193:6832/3093465661,v1:172.21.15.193:6833/3093465661] up:boot 2024-07-26T11:09:27.189 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:26 smithi189 ceph-mon[36790]: fsmap 1 up:standby 2024-07-26T11:09:27.189 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:09:27.189 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:27.189 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:27.189 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:27.189 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:27.190 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:27.190 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:26 smithi193 ceph-mon[37473]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:26 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.193:6832/3093465661,v1:172.21.15.193:6833/3093465661] up:boot 2024-07-26T11:09:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:26 smithi193 ceph-mon[37473]: fsmap 1 up:standby 2024-07-26T11:09:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:09:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:27.372 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:26 smithi134 ceph-mon[31966]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:27.372 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:26 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.193:6832/3093465661,v1:172.21.15.193:6833/3093465661] up:boot 2024-07-26T11:09:27.373 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:26 smithi134 ceph-mon[31966]: fsmap 1 up:standby 2024-07-26T11:09:27.373 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-07-26T11:09:27.373 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:27.373 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:27.373 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:27.373 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:27.373 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:27.373 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:27 smithi134 ceph-mon[31966]: Deploying daemon mds.k on smithi189 2024-07-26T11:09:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:27 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1352013425' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:09:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:27 smithi189 ceph-mon[36790]: Deploying daemon mds.k on smithi189 2024-07-26T11:09:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:27 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1352013425' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:09:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:27 smithi193 ceph-mon[37473]: Deploying daemon mds.k on smithi189 2024-07-26T11:09:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:27 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1352013425' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:09:28.694 INFO:journalctl@ceph.mds.k.smithi189.stdout:Jul 26 11:09:28 smithi189 systemd[1]: Starting Ceph mds.k for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:09:28.995 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:28 smithi189 ceph-mon[36790]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:28.995 INFO:journalctl@ceph.mds.k.smithi189.stdout:Jul 26 11:09:28 smithi189 podman[67246]: 2024-07-26 11:09:28.729359204 +0000 UTC m=+0.112937794 container create 15508beef44f04c48c35b92decaff0e5c3364027a221904320a3844243d62f6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-k, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS) 2024-07-26T11:09:28.995 INFO:journalctl@ceph.mds.k.smithi189.stdout:Jul 26 11:09:28 smithi189 podman[67246]: 2024-07-26 11:09:28.633231379 +0000 UTC m=+0.016809980 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:09:28.995 INFO:journalctl@ceph.mds.k.smithi189.stdout:Jul 26 11:09:28 smithi189 podman[67246]: 2024-07-26 11:09:28.957228015 +0000 UTC m=+0.340806601 container init 15508beef44f04c48c35b92decaff0e5c3364027a221904320a3844243d62f6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-k, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724) 2024-07-26T11:09:28.995 INFO:journalctl@ceph.mds.k.smithi189.stdout:Jul 26 11:09:28 smithi189 podman[67246]: 2024-07-26 11:09:28.959901655 +0000 UTC m=+0.343480242 container start 15508beef44f04c48c35b92decaff0e5c3364027a221904320a3844243d62f6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-k, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:09:28.995 INFO:journalctl@ceph.mds.k.smithi189.stdout:Jul 26 11:09:28 smithi189 ceph-mds[67265]: -- 172.21.15.189:0/3913152744 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56132b50b860 con 0x56132f6ada80 2024-07-26T11:09:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:28 smithi193 ceph-mon[37473]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:28 smithi134 ceph-mon[31966]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:29.444 INFO:journalctl@ceph.mds.k.smithi189.stdout:Jul 26 11:09:28 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-k[67261]: starting mds.k at 2024-07-26T11:09:29.444 INFO:journalctl@ceph.mds.k.smithi189.stdout:Jul 26 11:09:29 smithi189 bash[67246]: 15508beef44f04c48c35b92decaff0e5c3364027a221904320a3844243d62f6c 2024-07-26T11:09:29.444 INFO:journalctl@ceph.mds.k.smithi189.stdout:Jul 26 11:09:29 smithi189 systemd[1]: Started Ceph mds.k for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:09:29.829 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:09:30.150 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:30.150 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:30.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:30.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:30.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:30.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:30.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:30 smithi134 ceph-mon[31966]: Deploying daemon mds.j on smithi134 2024-07-26T11:09:30.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:30 smithi134 ceph-mon[31966]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:30.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:30 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6832/1897636383,v1:172.21.15.189:6833/1897636383] up:boot 2024-07-26T11:09:30.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:30 smithi134 ceph-mon[31966]: fsmap 2 up:standby 2024-07-26T11:09:30.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:09:30.384 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:30.384 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:30.384 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:30.384 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:30.385 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:30.385 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:30.385 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:30 smithi189 ceph-mon[36790]: Deploying daemon mds.j on smithi134 2024-07-26T11:09:30.385 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:30 smithi189 ceph-mon[36790]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:30.385 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:30 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6832/1897636383,v1:172.21.15.189:6833/1897636383] up:boot 2024-07-26T11:09:30.385 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:30 smithi189 ceph-mon[36790]: fsmap 2 up:standby 2024-07-26T11:09:30.385 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:09:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:30.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:30.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:30.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:30.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:30.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:30.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:30 smithi193 ceph-mon[37473]: Deploying daemon mds.j on smithi134 2024-07-26T11:09:30.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:30 smithi193 ceph-mon[37473]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:30.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:30 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6832/1897636383,v1:172.21.15.189:6833/1897636383] up:boot 2024-07-26T11:09:30.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:30 smithi193 ceph-mon[37473]: fsmap 2 up:standby 2024-07-26T11:09:30.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-07-26T11:09:31.136 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:09:32.363 INFO:journalctl@ceph.mds.j.smithi134.stdout:Jul 26 11:09:32 smithi134 systemd[1]: Starting Ceph mds.j for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:09:33.002 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:32 smithi134 ceph-mon[31966]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:33.003 INFO:journalctl@ceph.mds.j.smithi134.stdout:Jul 26 11:09:32 smithi134 podman[77037]: 2024-07-26 11:09:32.841288339 +0000 UTC m=+0.037063359 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:09:33.003 INFO:journalctl@ceph.mds.j.smithi134.stdout:Jul 26 11:09:33 smithi134 podman[77037]: 2024-07-26 11:09:33.00091946 +0000 UTC m=+0.196694516 container create d46577b3d6cafa11dfcc37608b1afaf10c5b287f0e7ae29909e335ecdb277c8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-j, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-07-26T11:09:33.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:32 smithi189 ceph-mon[36790]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:32 smithi193 ceph-mon[37473]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:33.534 INFO:journalctl@ceph.mds.j.smithi134.stdout:Jul 26 11:09:33 smithi134 podman[77037]: 2024-07-26 11:09:33.522999146 +0000 UTC m=+0.718774170 container init d46577b3d6cafa11dfcc37608b1afaf10c5b287f0e7ae29909e335ecdb277c8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240724) 2024-07-26T11:09:33.842 INFO:journalctl@ceph.mds.j.smithi134.stdout:Jul 26 11:09:33 smithi134 podman[77037]: 2024-07-26 11:09:33.533065944 +0000 UTC m=+0.728840985 container start d46577b3d6cafa11dfcc37608b1afaf10c5b287f0e7ae29909e335ecdb277c8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:09:33.842 INFO:journalctl@ceph.mds.j.smithi134.stdout:Jul 26 11:09:33 smithi134 bash[77037]: d46577b3d6cafa11dfcc37608b1afaf10c5b287f0e7ae29909e335ecdb277c8d 2024-07-26T11:09:33.842 INFO:journalctl@ceph.mds.j.smithi134.stdout:Jul 26 11:09:33 smithi134 systemd[1]: Started Ceph mds.j for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:09:33.843 INFO:journalctl@ceph.mds.j.smithi134.stdout:Jul 26 11:09:33 smithi134 ceph-mds[77078]: -- 172.21.15.134:0/4101949252 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5589def4dc20 con 0x5589e30ef180 2024-07-26T11:09:33.843 INFO:journalctl@ceph.mds.j.smithi134.stdout:Jul 26 11:09:33 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-j[77069]: starting mds.j at 2024-07-26T11:09:33.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.929+0000 7fd9dfe00640 1 -- 172.21.15.134:0/3929564321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd9e01033e0 msgr2=0x7fd9e01037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:33.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.929+0000 7fd9dfe00640 1 --2- 172.21.15.134:0/3929564321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd9e01033e0 0x7fd9e01037f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0009900 tx=0x7fd9d002a410 comp rx=0 tx=0).stop 2024-07-26T11:09:33.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.929+0000 7fd9dfe00640 1 -- 172.21.15.134:0/3929564321 shutdown_connections 2024-07-26T11:09:33.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.929+0000 7fd9dfe00640 1 --2- 172.21.15.134:0/3929564321 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9e0109670 0x7fd9e010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:33.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.929+0000 7fd9dfe00640 1 --2- 172.21.15.134:0/3929564321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9e0108d20 0x7fd9e0109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:33.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.929+0000 7fd9dfe00640 1 --2- 172.21.15.134:0/3929564321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd9e01033e0 0x7fd9e01037f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:33.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.929+0000 7fd9dfe00640 1 -- 172.21.15.134:0/3929564321 >> 172.21.15.134:0/3929564321 conn(0x7fd9e00fee70 msgr2=0x7fd9e01012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:33.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.930+0000 7fd9dfe00640 1 -- 172.21.15.134:0/3929564321 shutdown_connections 2024-07-26T11:09:33.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.930+0000 7fd9dfe00640 1 -- 172.21.15.134:0/3929564321 wait complete. 2024-07-26T11:09:33.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.931+0000 7fd9dfe00640 1 Processor -- start 2024-07-26T11:09:33.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.931+0000 7fd9dfe00640 1 -- start start 2024-07-26T11:09:33.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.931+0000 7fd9dfe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9e01033e0 0x7fd9e019bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:33.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.931+0000 7fd9dfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd9e0108d20 0x7fd9e019c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:33.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.931+0000 7fd9dfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9e0109670 0x7fd9e01a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:33.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.931+0000 7fd9dfe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e0113340 con 0x7fd9e01033e0 2024-07-26T11:09:33.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.931+0000 7fd9dfe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e01131c0 con 0x7fd9e0109670 2024-07-26T11:09:33.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.931+0000 7fd9dfe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e01134c0 con 0x7fd9e0108d20 2024-07-26T11:09:33.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.932+0000 7fd9dea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9e01033e0 0x7fd9e019bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:33.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.932+0000 7fd9df400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9e0109670 0x7fd9e01a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:33.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.932+0000 7fd9dea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9e01033e0 0x7fd9e019bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:52834/0 (socket says 172.21.15.134:52834) 2024-07-26T11:09:33.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.932+0000 7fd9df400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9e0109670 0x7fd9e01a17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:47340/0 (socket says 172.21.15.134:47340) 2024-07-26T11:09:33.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.932+0000 7fd9dea00640 1 -- 172.21.15.134:0/3363119857 learned_addr learned my addr 172.21.15.134:0/3363119857 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:09:33.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.932+0000 7fd9de000640 1 --2- 172.21.15.134:0/3363119857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd9e0108d20 0x7fd9e019c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:33.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.932+0000 7fd9df400640 1 -- 172.21.15.134:0/3363119857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd9e0108d20 msgr2=0x7fd9e019c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:33.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.932+0000 7fd9df400640 1 --2- 172.21.15.134:0/3363119857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd9e0108d20 0x7fd9e019c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:33.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.932+0000 7fd9df400640 1 -- 172.21.15.134:0/3363119857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9e01033e0 msgr2=0x7fd9e019bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:33.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.932+0000 7fd9df400640 1 --2- 172.21.15.134:0/3363119857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9e01033e0 0x7fd9e019bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:33.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.932+0000 7fd9df400640 1 -- 172.21.15.134:0/3363119857 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e01a1f20 con 0x7fd9e0109670 2024-07-26T11:09:33.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.932+0000 7fd9de000640 1 --2- 172.21.15.134:0/3363119857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd9e0108d20 0x7fd9e019c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:09:33.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.933+0000 7fd9df400640 1 --2- 172.21.15.134:0/3363119857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9e0109670 0x7fd9e01a17f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009fd0 tx=0x7fd9d800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:33.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.933+0000 7fd9d7400640 1 -- 172.21.15.134:0/3363119857 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d800ce20 con 0x7fd9e0109670 2024-07-26T11:09:33.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.933+0000 7fd9d7400640 1 -- 172.21.15.134:0/3363119857 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9d8005020 con 0x7fd9e0109670 2024-07-26T11:09:33.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.933+0000 7fd9dfe00640 1 -- 172.21.15.134:0/3363119857 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9e01a2210 con 0x7fd9e0109670 2024-07-26T11:09:33.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.933+0000 7fd9d7400640 1 -- 172.21.15.134:0/3363119857 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8010690 con 0x7fd9e0109670 2024-07-26T11:09:33.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.933+0000 7fd9dfe00640 1 -- 172.21.15.134:0/3363119857 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9e01a8970 con 0x7fd9e0109670 2024-07-26T11:09:33.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.935+0000 7fd9dfe00640 1 -- 172.21.15.134:0/3363119857 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9e01037f0 con 0x7fd9e0109670 2024-07-26T11:09:33.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.937+0000 7fd9d7400640 1 -- 172.21.15.134:0/3363119857 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fd9d8048080 con 0x7fd9e0109670 2024-07-26T11:09:33.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.938+0000 7fd9d7400640 1 --2- 172.21.15.134:0/3363119857 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd9c40b3ca0 0x7fd9c40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:33.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.939+0000 7fd9d7400640 1 -- 172.21.15.134:0/3363119857 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7342+0+0 (secure 0 0 0) 0x7fd9d80e41f0 con 0x7fd9e0109670 2024-07-26T11:09:33.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.939+0000 7fd9dea00640 1 --2- 172.21.15.134:0/3363119857 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd9c40b3ca0 0x7fd9c40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:33.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.940+0000 7fd9dea00640 1 --2- 172.21.15.134:0/3363119857 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd9c40b3ca0 0x7fd9c40b61b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0031440 tx=0x7fd9d00058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:33.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:33.940+0000 7fd9d7400640 1 -- 172.21.15.134:0/3363119857 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd9d80abb70 con 0x7fd9e0109670 2024-07-26T11:09:34.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:34.180+0000 7fd9dfe00640 1 -- 172.21.15.134:0/3363119857 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9e019d0e0 con 0x7fd9e0109670 2024-07-26T11:09:34.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:34.181+0000 7fd9d7400640 1 -- 172.21.15.134:0/3363119857 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3094 (secure 0 0 0) 0x7fd9e019d0e0 con 0x7fd9e0109670 2024-07-26T11:09:34.183 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:09:34.183 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":4,"btime":"2024-07-26T11:09:33:902131+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24635,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24637,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24641,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-07-26T11:09:34.183 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 4 2024-07-26T11:09:34.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:34.183+0000 7fd9dfe00640 1 -- 172.21.15.134:0/3363119857 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd9c40b3ca0 msgr2=0x7fd9c40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:34.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:34.183+0000 7fd9dfe00640 1 --2- 172.21.15.134:0/3363119857 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd9c40b3ca0 0x7fd9c40b61b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0031440 tx=0x7fd9d00058d0 comp rx=0 tx=0).stop 2024-07-26T11:09:34.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:34.183+0000 7fd9dfe00640 1 -- 172.21.15.134:0/3363119857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9e0109670 msgr2=0x7fd9e01a17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:34.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:34.183+0000 7fd9dfe00640 1 --2- 172.21.15.134:0/3363119857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9e0109670 0x7fd9e01a17f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009fd0 tx=0x7fd9d800ef10 comp rx=0 tx=0).stop 2024-07-26T11:09:34.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:34.183+0000 7fd9dfe00640 1 -- 172.21.15.134:0/3363119857 shutdown_connections 2024-07-26T11:09:34.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:34.184+0000 7fd9dfe00640 1 --2- 172.21.15.134:0/3363119857 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd9c40b3ca0 0x7fd9c40b61b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:34.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:34.184+0000 7fd9dfe00640 1 --2- 172.21.15.134:0/3363119857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd9e0108d20 0x7fd9e019c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:34.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:34.184+0000 7fd9dfe00640 1 --2- 172.21.15.134:0/3363119857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9e0109670 0x7fd9e01a17f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:34.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:34.184+0000 7fd9dfe00640 1 --2- 172.21.15.134:0/3363119857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd9e01033e0 0x7fd9e019bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:34.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:34.184+0000 7fd9dfe00640 1 -- 172.21.15.134:0/3363119857 >> 172.21.15.134:0/3363119857 conn(0x7fd9e00fee70 msgr2=0x7fd9e010e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:34.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:34.184+0000 7fd9dfe00640 1 -- 172.21.15.134:0/3363119857 shutdown_connections 2024-07-26T11:09:34.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:34.184+0000 7fd9dfe00640 1 -- 172.21.15.134:0/3363119857 wait complete. 2024-07-26T11:09:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:34 smithi193 ceph-mon[37473]: Deploying daemon mds.d on smithi134 2024-07-26T11:09:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:34 smithi193 ceph-mon[37473]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:34 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6834/4071410587,v1:172.21.15.134:6835/4071410587] up:boot 2024-07-26T11:09:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:34 smithi193 ceph-mon[37473]: fsmap 3 up:standby 2024-07-26T11:09:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:09:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:34 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3363119857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:09:35.056 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:35.056 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:35.056 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:35.056 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:35.056 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:35.056 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:35.056 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:34 smithi134 ceph-mon[31966]: Deploying daemon mds.d on smithi134 2024-07-26T11:09:35.056 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:34 smithi134 ceph-mon[31966]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:35.056 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:34 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6834/4071410587,v1:172.21.15.134:6835/4071410587] up:boot 2024-07-26T11:09:35.056 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:34 smithi134 ceph-mon[31966]: fsmap 3 up:standby 2024-07-26T11:09:35.056 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:09:35.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:34 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3363119857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:09:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:35.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:35.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:35.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:35.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:34 smithi189 ceph-mon[36790]: Deploying daemon mds.d on smithi134 2024-07-26T11:09:35.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:34 smithi189 ceph-mon[36790]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:35.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:34 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6834/4071410587,v1:172.21.15.134:6835/4071410587] up:boot 2024-07-26T11:09:35.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:34 smithi189 ceph-mon[36790]: fsmap 3 up:standby 2024-07-26T11:09:35.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-07-26T11:09:35.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:34 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3363119857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:09:37.126 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:09:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:36 smithi134 ceph-mon[31966]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:36 smithi189 ceph-mon[36790]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:36 smithi193 ceph-mon[37473]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:37.591 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:09:37 smithi134 systemd[1]: Starting Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:09:37.879 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:09:37 smithi134 podman[77425]: 2024-07-26 11:09:37.493532881 +0000 UTC m=+0.033040573 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:09:37.879 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:09:37 smithi134 podman[77425]: 2024-07-26 11:09:37.606472793 +0000 UTC m=+0.145980506 container create c8e744686ff3fd4e76f5e61c2b46eaaeff3dbecf1ce7e0840f7362504a75ce5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-07-26T11:09:37.879 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:09:37 smithi134 podman[77425]: 2024-07-26 11:09:37.876923657 +0000 UTC m=+0.416431420 container init c8e744686ff3fd4e76f5e61c2b46eaaeff3dbecf1ce7e0840f7362504a75ce5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-07-26T11:09:37.999 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:09:38.148 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:09:37 smithi134 podman[77425]: 2024-07-26 11:09:37.889208576 +0000 UTC m=+0.428716271 container start c8e744686ff3fd4e76f5e61c2b46eaaeff3dbecf1ce7e0840f7362504a75ce5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, io.buildah.version=1.36.0) 2024-07-26T11:09:38.148 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:09:37 smithi134 bash[77425]: c8e744686ff3fd4e76f5e61c2b46eaaeff3dbecf1ce7e0840f7362504a75ce5d 2024-07-26T11:09:38.148 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:09:37 smithi134 systemd[1]: Started Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:09:38.148 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:09:38 smithi134 ceph-mds[77468]: -- 172.21.15.134:0/2034005434 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5602260af860 con 0x56022a251180 2024-07-26T11:09:38.148 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:09:38 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: starting mds.d at 2024-07-26T11:09:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:38 smithi134 ceph-mon[31966]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:38 smithi189 ceph-mon[36790]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:38 smithi193 ceph-mon[37473]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:39.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:39.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:39.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:39.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:39.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:39.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:40.046 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:39 smithi134 ceph-mon[31966]: Deploying daemon mds.e on smithi189 2024-07-26T11:09:40.047 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:39 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/31742259,v1:172.21.15.134:6837/31742259] up:boot 2024-07-26T11:09:40.047 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:39 smithi134 ceph-mon[31966]: fsmap 4 up:standby 2024-07-26T11:09:40.047 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:09:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:39 smithi189 ceph-mon[36790]: Deploying daemon mds.e on smithi189 2024-07-26T11:09:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:39 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/31742259,v1:172.21.15.134:6837/31742259] up:boot 2024-07-26T11:09:40.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:39 smithi189 ceph-mon[36790]: fsmap 4 up:standby 2024-07-26T11:09:40.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:09:40.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:09:40 smithi189 systemd[1]: Starting Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:09:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:39 smithi193 ceph-mon[37473]: Deploying daemon mds.e on smithi189 2024-07-26T11:09:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:39 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/31742259,v1:172.21.15.134:6837/31742259] up:boot 2024-07-26T11:09:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:39 smithi193 ceph-mon[37473]: fsmap 4 up:standby 2024-07-26T11:09:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:09:40.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.367+0000 7ff00b600640 1 -- 172.21.15.134:0/3037747416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff00411c9b0 msgr2=0x7ff00411edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:40.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.367+0000 7ff00b600640 1 --2- 172.21.15.134:0/3037747416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff00411c9b0 0x7ff00411edd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7feffc009be0 tx=0x7feffc02a6d0 comp rx=0 tx=0).stop 2024-07-26T11:09:40.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.368+0000 7ff00b600640 1 -- 172.21.15.134:0/3037747416 shutdown_connections 2024-07-26T11:09:40.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.368+0000 7ff00b600640 1 --2- 172.21.15.134:0/3037747416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff00411c9b0 0x7ff00411edd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:40.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.368+0000 7ff00b600640 1 --2- 172.21.15.134:0/3037747416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff00406f450 0x7ff00411c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:40.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.368+0000 7ff00b600640 1 --2- 172.21.15.134:0/3037747416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff00406eb00 0x7ff00406ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:40.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.368+0000 7ff00b600640 1 -- 172.21.15.134:0/3037747416 >> 172.21.15.134:0/3037747416 conn(0x7ff004109df0 msgr2=0x7ff00410c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:40.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.368+0000 7ff00b600640 1 -- 172.21.15.134:0/3037747416 shutdown_connections 2024-07-26T11:09:40.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.368+0000 7ff00b600640 1 -- 172.21.15.134:0/3037747416 wait complete. 2024-07-26T11:09:40.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.369+0000 7ff00b600640 1 Processor -- start 2024-07-26T11:09:40.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.370+0000 7ff00b600640 1 -- start start 2024-07-26T11:09:40.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.370+0000 7ff00b600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff00406eb00 0x7ff0041a8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:40.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.370+0000 7ff00b600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff00406f450 0x7ff0041a9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:40.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.370+0000 7ff00b600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff00411c9b0 0x7ff0041ae4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:40.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.370+0000 7ff00b600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff0041222c0 con 0x7ff00406eb00 2024-07-26T11:09:40.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.370+0000 7ff00b600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff004122140 con 0x7ff00411c9b0 2024-07-26T11:09:40.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.370+0000 7ff00b600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff004122440 con 0x7ff00406f450 2024-07-26T11:09:40.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.370+0000 7ff009600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff00411c9b0 0x7ff0041ae4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:40.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.370+0000 7ff008c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff00406eb00 0x7ff0041a8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:40.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.370+0000 7ff009600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff00411c9b0 0x7ff0041ae4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38536/0 (socket says 172.21.15.134:38536) 2024-07-26T11:09:40.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.370+0000 7ff009600640 1 -- 172.21.15.134:0/3082358713 learned_addr learned my addr 172.21.15.134:0/3082358713 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:09:40.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.371+0000 7ff003e00640 1 --2- 172.21.15.134:0/3082358713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff00406f450 0x7ff0041a9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:40.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.371+0000 7ff009600640 1 -- 172.21.15.134:0/3082358713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff00406f450 msgr2=0x7ff0041a9210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:40.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.371+0000 7ff009600640 1 --2- 172.21.15.134:0/3082358713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff00406f450 0x7ff0041a9210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:40.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.371+0000 7ff009600640 1 -- 172.21.15.134:0/3082358713 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff00406eb00 msgr2=0x7ff0041a8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:40.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.371+0000 7ff009600640 1 --2- 172.21.15.134:0/3082358713 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff00406eb00 0x7ff0041a8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:40.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.371+0000 7ff009600640 1 -- 172.21.15.134:0/3082358713 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0041aebe0 con 0x7ff00411c9b0 2024-07-26T11:09:40.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.371+0000 7ff009600640 1 --2- 172.21.15.134:0/3082358713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff00411c9b0 0x7ff0041ae4b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7feffc009b30 tx=0x7feffc003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:40.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.372+0000 7ff001600640 1 -- 172.21.15.134:0/3082358713 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feffc040020 con 0x7ff00411c9b0 2024-07-26T11:09:40.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.372+0000 7ff00b600640 1 -- 172.21.15.134:0/3082358713 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0041aee70 con 0x7ff00411c9b0 2024-07-26T11:09:40.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.372+0000 7ff001600640 1 -- 172.21.15.134:0/3082358713 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feffc030970 con 0x7ff00411c9b0 2024-07-26T11:09:40.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.372+0000 7ff001600640 1 -- 172.21.15.134:0/3082358713 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feffc036040 con 0x7ff00411c9b0 2024-07-26T11:09:40.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.372+0000 7ff00b600640 1 -- 172.21.15.134:0/3082358713 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0041b5630 con 0x7ff00411c9b0 2024-07-26T11:09:40.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.373+0000 7ff00b600640 1 -- 172.21.15.134:0/3082358713 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff00406ef10 con 0x7ff00411c9b0 2024-07-26T11:09:40.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.375+0000 7ff001600640 1 -- 172.21.15.134:0/3082358713 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7feffc075020 con 0x7ff00411c9b0 2024-07-26T11:09:40.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.377+0000 7ff001600640 1 --2- 172.21.15.134:0/3082358713 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fefec0b3c20 0x7fefec0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:40.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.377+0000 7ff001600640 1 -- 172.21.15.134:0/3082358713 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7342+0+0 (secure 0 0 0) 0x7feffc104e20 con 0x7ff00411c9b0 2024-07-26T11:09:40.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.377+0000 7ff008c00640 1 --2- 172.21.15.134:0/3082358713 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fefec0b3c20 0x7fefec0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:40.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.378+0000 7ff008c00640 1 --2- 172.21.15.134:0/3082358713 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fefec0b3c20 0x7fefec0b6130 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7feff8009bd0 tx=0x7feff8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:40.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.378+0000 7ff001600640 1 -- 172.21.15.134:0/3082358713 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7feffc04a020 con 0x7ff00411c9b0 2024-07-26T11:09:40.583 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:09:40 smithi189 podman[67562]: 2024-07-26 11:09:40.230091316 +0000 UTC m=+0.016760167 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:09:40.584 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:09:40 smithi189 podman[67562]: 2024-07-26 11:09:40.346345074 +0000 UTC m=+0.133013925 container create ece0fd5f8679b7c3eb22a2ee7a1974018f0b8ae8698d6e2e6ff89d06d89aeb29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:09:40.584 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:09:40 smithi189 podman[67562]: 2024-07-26 11:09:40.582601156 +0000 UTC m=+0.369270016 container init ece0fd5f8679b7c3eb22a2ee7a1974018f0b8ae8698d6e2e6ff89d06d89aeb29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:09:40.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.614+0000 7ff00b600640 1 -- 172.21.15.134:0/3082358713 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0041b5860 con 0x7ff00411c9b0 2024-07-26T11:09:40.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.615+0000 7ff001600640 1 -- 172.21.15.134:0/3082358713 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3916 (secure 0 0 0) 0x7feffc0cc7a0 con 0x7ff00411c9b0 2024-07-26T11:09:40.617 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:09:40.617 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":5,"btime":"2024-07-26T11:09:38:915946+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24635,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24637,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24641,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-07-26T11:09:40.617 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 5 2024-07-26T11:09:40.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.617+0000 7ff00b600640 1 -- 172.21.15.134:0/3082358713 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fefec0b3c20 msgr2=0x7fefec0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:40.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.617+0000 7ff00b600640 1 --2- 172.21.15.134:0/3082358713 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fefec0b3c20 0x7fefec0b6130 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7feff8009bd0 tx=0x7feff8005f90 comp rx=0 tx=0).stop 2024-07-26T11:09:40.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.617+0000 7ff00b600640 1 -- 172.21.15.134:0/3082358713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff00411c9b0 msgr2=0x7ff0041ae4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:40.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.617+0000 7ff00b600640 1 --2- 172.21.15.134:0/3082358713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff00411c9b0 0x7ff0041ae4b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7feffc009b30 tx=0x7feffc003c10 comp rx=0 tx=0).stop 2024-07-26T11:09:40.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.618+0000 7ff00b600640 1 -- 172.21.15.134:0/3082358713 shutdown_connections 2024-07-26T11:09:40.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.618+0000 7ff00b600640 1 --2- 172.21.15.134:0/3082358713 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fefec0b3c20 0x7fefec0b6130 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:40.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.618+0000 7ff00b600640 1 --2- 172.21.15.134:0/3082358713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff00406f450 0x7ff0041a9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:40.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.618+0000 7ff00b600640 1 --2- 172.21.15.134:0/3082358713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff00411c9b0 0x7ff0041ae4b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:40.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.618+0000 7ff00b600640 1 --2- 172.21.15.134:0/3082358713 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff00406eb00 0x7ff0041a8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:40.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.618+0000 7ff00b600640 1 -- 172.21.15.134:0/3082358713 >> 172.21.15.134:0/3082358713 conn(0x7ff004109df0 msgr2=0x7ff00411d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:40.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.618+0000 7ff00b600640 1 -- 172.21.15.134:0/3082358713 shutdown_connections 2024-07-26T11:09:40.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:40.618+0000 7ff00b600640 1 -- 172.21.15.134:0/3082358713 wait complete. 2024-07-26T11:09:40.944 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:09:40 smithi189 podman[67562]: 2024-07-26 11:09:40.585533931 +0000 UTC m=+0.372202782 container start ece0fd5f8679b7c3eb22a2ee7a1974018f0b8ae8698d6e2e6ff89d06d89aeb29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:09:40.944 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:09:40 smithi189 ceph-mds[67581]: -- 172.21.15.189:0/2321701009 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e2be2bf860 con 0x55e2c2461600 2024-07-26T11:09:40.944 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:09:40 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: starting mds.e at 2024-07-26T11:09:40.944 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:09:40 smithi189 bash[67562]: ece0fd5f8679b7c3eb22a2ee7a1974018f0b8ae8698d6e2e6ff89d06d89aeb29 2024-07-26T11:09:40.945 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:09:40 smithi189 systemd[1]: Started Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:09:41.222 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:40 smithi193 ceph-mon[37473]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:41.222 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:40 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3082358713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:09:41.223 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:41.223 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:41.223 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:41.223 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:41.223 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:41.223 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:41.359 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:40 smithi134 ceph-mon[31966]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:41.359 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:40 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3082358713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:09:41.359 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:41.359 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:41.359 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:41.359 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:41.359 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:41.359 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:41.369 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:40 smithi189 ceph-mon[36790]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:41.369 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:40 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3082358713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:09:41.369 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:41.369 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:41.369 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:41.369 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:41.369 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:41.369 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:42.254 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:41 smithi193 ceph-mon[37473]: Deploying daemon mds.f on smithi193 2024-07-26T11:09:42.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:41 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/1314042896,v1:172.21.15.189:6835/1314042896] up:boot 2024-07-26T11:09:42.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:41 smithi193 ceph-mon[37473]: fsmap 5 up:standby 2024-07-26T11:09:42.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:09:42.255 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:09:42.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:41 smithi134 ceph-mon[31966]: Deploying daemon mds.f on smithi193 2024-07-26T11:09:42.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:41 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/1314042896,v1:172.21.15.189:6835/1314042896] up:boot 2024-07-26T11:09:42.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:41 smithi134 ceph-mon[31966]: fsmap 5 up:standby 2024-07-26T11:09:42.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:09:42.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:09:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:41 smithi189 ceph-mon[36790]: Deploying daemon mds.f on smithi193 2024-07-26T11:09:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:41 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/1314042896,v1:172.21.15.189:6835/1314042896] up:boot 2024-07-26T11:09:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:41 smithi189 ceph-mon[36790]: fsmap 5 up:standby 2024-07-26T11:09:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:09:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:09:42.922 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:09:42 smithi193 systemd[1]: Starting Ceph mds.f for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:09:43.108 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:42 smithi134 ceph-mon[31966]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:43.186 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:42 smithi193 ceph-mon[37473]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:43.186 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:09:42 smithi193 podman[68388]: 2024-07-26 11:09:42.82364044 +0000 UTC m=+0.016795627 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:09:43.186 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:09:42 smithi193 podman[68388]: 2024-07-26 11:09:42.948504971 +0000 UTC m=+0.141660190 container create 3091e8a1a2db831c0029256db24d24d96b50111084e0969a7ddf567ab97909ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:09:43.186 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:09:43 smithi193 podman[68388]: 2024-07-26 11:09:43.184867402 +0000 UTC m=+0.378022580 container init 3091e8a1a2db831c0029256db24d24d96b50111084e0969a7ddf567ab97909ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:09:43.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:09:43.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:42 smithi189 ceph-mon[36790]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:43.524 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:09:43 smithi193 podman[68388]: 2024-07-26 11:09:43.187564834 +0000 UTC m=+0.380720009 container start 3091e8a1a2db831c0029256db24d24d96b50111084e0969a7ddf567ab97909ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0) 2024-07-26T11:09:43.524 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:09:43 smithi193 ceph-mds[68408]: -- 172.21.15.193:0/583811454 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558c66dcd860 con 0x558c6af6f180 2024-07-26T11:09:43.525 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:09:43 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: starting mds.f at 2024-07-26T11:09:43.525 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:09:43 smithi193 bash[68388]: 3091e8a1a2db831c0029256db24d24d96b50111084e0969a7ddf567ab97909ae 2024-07-26T11:09:43.525 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:09:43 smithi193 systemd[1]: Started Ceph mds.f for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:09:43.841 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:09:44.629 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:44.629 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:44.629 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:44.629 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:44.629 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:44.630 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:44.630 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:44 smithi189 ceph-mon[36790]: Deploying daemon mds.c on smithi193 2024-07-26T11:09:44.630 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:44 smithi189 ceph-mon[36790]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:44.630 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:44 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:boot 2024-07-26T11:09:44.630 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:44 smithi189 ceph-mon[36790]: fsmap 6 up:standby 2024-07-26T11:09:44.630 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:09:44.646 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:44.646 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:44.646 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:44.646 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:44.646 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:44.646 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:44.646 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:44 smithi134 ceph-mon[31966]: Deploying daemon mds.c on smithi193 2024-07-26T11:09:44.646 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:44 smithi134 ceph-mon[31966]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:44.646 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:44 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:boot 2024-07-26T11:09:44.646 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:44 smithi134 ceph-mon[31966]: fsmap 6 up:standby 2024-07-26T11:09:44.647 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:09:44.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:44.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:44.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:44.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:44.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:44.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:44.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:44 smithi193 ceph-mon[37473]: Deploying daemon mds.c on smithi193 2024-07-26T11:09:44.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:44 smithi193 ceph-mon[37473]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:44.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:44 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:boot 2024-07-26T11:09:44.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:44 smithi193 ceph-mon[37473]: fsmap 6 up:standby 2024-07-26T11:09:44.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-07-26T11:09:45.411 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jul 26 11:09:45 smithi193 systemd[1]: Starting Ceph mds.c for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:09:45.735 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jul 26 11:09:45 smithi193 podman[68687]: 2024-07-26 11:09:45.312233096 +0000 UTC m=+0.017572071 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:09:45.735 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jul 26 11:09:45 smithi193 podman[68687]: 2024-07-26 11:09:45.475298906 +0000 UTC m=+0.180637919 container create a2f99434d2f1f9946ed397ed1d63164311b24cf320c66be76de256e7896a6d67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-c, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-07-26T11:09:45.735 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jul 26 11:09:45 smithi193 podman[68687]: 2024-07-26 11:09:45.73400218 +0000 UTC m=+0.439341157 container init a2f99434d2f1f9946ed397ed1d63164311b24cf320c66be76de256e7896a6d67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-07-26T11:09:46.024 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jul 26 11:09:45 smithi193 podman[68687]: 2024-07-26 11:09:45.736743718 +0000 UTC m=+0.442082697 container start a2f99434d2f1f9946ed397ed1d63164311b24cf320c66be76de256e7896a6d67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-c, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-07-26T11:09:46.025 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jul 26 11:09:45 smithi193 ceph-mds[68706]: -- 172.21.15.193:0/2191635610 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559850a15860 con 0x559854bb7180 2024-07-26T11:09:46.025 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jul 26 11:09:45 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-c[68702]: starting mds.c at 2024-07-26T11:09:46.025 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jul 26 11:09:45 smithi193 bash[68687]: a2f99434d2f1f9946ed397ed1d63164311b24cf320c66be76de256e7896a6d67 2024-07-26T11:09:46.025 INFO:journalctl@ceph.mds.c.smithi193.stdout:Jul 26 11:09:45 smithi193 systemd[1]: Started Ceph mds.c for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:09:46.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.214+0000 7f147ac00640 1 -- 172.21.15.134:0/4148909140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f147410b860 msgr2=0x7f1474112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:46.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.214+0000 7f147ac00640 1 --2- 172.21.15.134:0/4148909140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f147410b860 0x7f1474112090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f146c009ca0 tx=0x7f146c02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:09:46.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.215+0000 7f147ac00640 1 -- 172.21.15.134:0/4148909140 shutdown_connections 2024-07-26T11:09:46.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.215+0000 7f147ac00640 1 --2- 172.21.15.134:0/4148909140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f147410b860 0x7f1474112090 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:46.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.215+0000 7f147ac00640 1 --2- 172.21.15.134:0/4148909140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f147410af10 0x7f147410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:46.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.215+0000 7f147ac00640 1 --2- 172.21.15.134:0/4148909140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f14741055d0 0x7f14741059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:46.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.215+0000 7f147ac00640 1 -- 172.21.15.134:0/4148909140 >> 172.21.15.134:0/4148909140 conn(0x7f1474101040 msgr2=0x7f1474103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:46.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.215+0000 7f147ac00640 1 -- 172.21.15.134:0/4148909140 shutdown_connections 2024-07-26T11:09:46.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.215+0000 7f147ac00640 1 -- 172.21.15.134:0/4148909140 wait complete. 2024-07-26T11:09:46.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.216+0000 7f147ac00640 1 Processor -- start 2024-07-26T11:09:46.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.216+0000 7f147ac00640 1 -- start start 2024-07-26T11:09:46.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.217+0000 7f147ac00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14741055d0 0x7f147419beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.217+0000 7f147ac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f147410af10 0x7f147419c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.217+0000 7f147ac00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f147410b860 0x7f14741a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.217+0000 7f147ac00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1474115560 con 0x7f14741055d0 2024-07-26T11:09:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.217+0000 7f147ac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f14741153e0 con 0x7f147410af10 2024-07-26T11:09:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.217+0000 7f147ac00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f14741156e0 con 0x7f147410b860 2024-07-26T11:09:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.217+0000 7f1478e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f147410af10 0x7f147419c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.217+0000 7f1479800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14741055d0 0x7f147419beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.217+0000 7f1478e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f147410af10 0x7f147419c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38554/0 (socket says 172.21.15.134:38554) 2024-07-26T11:09:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.217+0000 7f1478e00640 1 -- 172.21.15.134:0/3002038400 learned_addr learned my addr 172.21.15.134:0/3002038400 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:09:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.217+0000 7f147a200640 1 --2- 172.21.15.134:0/3002038400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f147410b860 0x7f14741a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.217+0000 7f1478e00640 1 -- 172.21.15.134:0/3002038400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f147410b860 msgr2=0x7f14741a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:46.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.217+0000 7f1478e00640 1 --2- 172.21.15.134:0/3002038400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f147410b860 0x7f14741a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.217+0000 7f1478e00640 1 -- 172.21.15.134:0/3002038400 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14741055d0 msgr2=0x7f147419beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.217+0000 7f1478e00640 1 --2- 172.21.15.134:0/3002038400 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14741055d0 0x7f147419beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.217+0000 7f1478e00640 1 -- 172.21.15.134:0/3002038400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14741a1e20 con 0x7f147410af10 2024-07-26T11:09:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.218+0000 7f1478e00640 1 --2- 172.21.15.134:0/3002038400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f147410af10 0x7f147419c450 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f146000e9f0 tx=0x7f146000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:46.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.218+0000 7f1472000640 1 -- 172.21.15.134:0/3002038400 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146000ccc0 con 0x7f147410af10 2024-07-26T11:09:46.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.218+0000 7f1472000640 1 -- 172.21.15.134:0/3002038400 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14600108e0 con 0x7f147410af10 2024-07-26T11:09:46.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.218+0000 7f1472000640 1 -- 172.21.15.134:0/3002038400 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1460018860 con 0x7f147410af10 2024-07-26T11:09:46.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.218+0000 7f147ac00640 1 -- 172.21.15.134:0/3002038400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14741a2110 con 0x7f147410af10 2024-07-26T11:09:46.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.218+0000 7f147ac00640 1 -- 172.21.15.134:0/3002038400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14741a8870 con 0x7f147410af10 2024-07-26T11:09:46.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.219+0000 7f147ac00640 1 -- 172.21.15.134:0/3002038400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14741059e0 con 0x7f147410af10 2024-07-26T11:09:46.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.222+0000 7f1472000640 1 -- 172.21.15.134:0/3002038400 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f1460010430 con 0x7f147410af10 2024-07-26T11:09:46.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.223+0000 7f1472000640 1 --2- 172.21.15.134:0/3002038400 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f145c0b3ca0 0x7f145c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:46.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.223+0000 7f1472000640 1 -- 172.21.15.134:0/3002038400 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7342+0+0 (secure 0 0 0) 0x7f14600e3e10 con 0x7f147410af10 2024-07-26T11:09:46.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.224+0000 7f1479800640 1 --2- 172.21.15.134:0/3002038400 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f145c0b3ca0 0x7f145c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:46.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.224+0000 7f1479800640 1 --2- 172.21.15.134:0/3002038400 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f145c0b3ca0 0x7f145c0b61b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1468009bd0 tx=0x7f1468005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:46.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.225+0000 7f1472000640 1 -- 172.21.15.134:0/3002038400 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f14600ab140 con 0x7f147410af10 2024-07-26T11:09:46.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.462+0000 7f147ac00640 1 -- 172.21.15.134:0/3002038400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f147419cfe0 con 0x7f147410af10 2024-07-26T11:09:46.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.463+0000 7f1472000640 1 -- 172.21.15.134:0/3002038400 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6394 (secure 0 0 0) 0x7f147419cfe0 con 0x7f147410af10 2024-07-26T11:09:46.465 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:09:46.465 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":8,"btime":"2024-07-26T11:09:45:826041+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24635,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24637,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24641,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24665,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24682,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24688,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-07-26T11:09:46.466 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 8 2024-07-26T11:09:46.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.466+0000 7f147ac00640 1 -- 172.21.15.134:0/3002038400 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f145c0b3ca0 msgr2=0x7f145c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:46.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.466+0000 7f147ac00640 1 --2- 172.21.15.134:0/3002038400 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f145c0b3ca0 0x7f145c0b61b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1468009bd0 tx=0x7f1468005f90 comp rx=0 tx=0).stop 2024-07-26T11:09:46.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.466+0000 7f147ac00640 1 -- 172.21.15.134:0/3002038400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f147410af10 msgr2=0x7f147419c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:46.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.466+0000 7f147ac00640 1 --2- 172.21.15.134:0/3002038400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f147410af10 0x7f147419c450 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f146000e9f0 tx=0x7f146000eeb0 comp rx=0 tx=0).stop 2024-07-26T11:09:46.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.466+0000 7f147ac00640 1 -- 172.21.15.134:0/3002038400 shutdown_connections 2024-07-26T11:09:46.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.466+0000 7f147ac00640 1 --2- 172.21.15.134:0/3002038400 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f145c0b3ca0 0x7f145c0b61b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:46.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.466+0000 7f147ac00640 1 --2- 172.21.15.134:0/3002038400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f147410b860 0x7f14741a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:46.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.466+0000 7f147ac00640 1 --2- 172.21.15.134:0/3002038400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f147410af10 0x7f147419c450 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:46.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.466+0000 7f147ac00640 1 --2- 172.21.15.134:0/3002038400 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f14741055d0 0x7f147419beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:46.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.466+0000 7f147ac00640 1 -- 172.21.15.134:0/3002038400 >> 172.21.15.134:0/3002038400 conn(0x7f1474101040 msgr2=0x7f1474109510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:46.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.466+0000 7f147ac00640 1 -- 172.21.15.134:0/3002038400 shutdown_connections 2024-07-26T11:09:46.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:46.466+0000 7f147ac00640 1 -- 172.21.15.134:0/3002038400 wait complete. 2024-07-26T11:09:47.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:46 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] up:boot 2024-07-26T11:09:47.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:46 smithi134 ceph-mon[31966]: fsmap 7 up:standby 2024-07-26T11:09:47.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:09:47.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:46 smithi134 ceph-mon[31966]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:47.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:47.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:47.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:47.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:47.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:47.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:47.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:46 smithi134 ceph-mon[31966]: Deploying daemon mds.h on smithi189 2024-07-26T11:09:47.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:46 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3002038400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:09:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:46 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] up:boot 2024-07-26T11:09:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:46 smithi189 ceph-mon[36790]: fsmap 7 up:standby 2024-07-26T11:09:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:09:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:46 smithi189 ceph-mon[36790]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:47.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:47.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:47.197 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:47.197 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:47.197 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:47.197 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:46 smithi189 ceph-mon[36790]: Deploying daemon mds.h on smithi189 2024-07-26T11:09:47.197 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:46 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3002038400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:09:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:46 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] up:boot 2024-07-26T11:09:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:46 smithi193 ceph-mon[37473]: fsmap 7 up:standby 2024-07-26T11:09:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-07-26T11:09:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:46 smithi193 ceph-mon[37473]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:46 smithi193 ceph-mon[37473]: Deploying daemon mds.h on smithi189 2024-07-26T11:09:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:46 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3002038400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:09:48.116 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:09:47 smithi189 systemd[1]: Starting Ceph mds.h for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:09:48.411 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:09:48 smithi189 podman[67879]: 2024-07-26 11:09:48.017304812 +0000 UTC m=+0.017170441 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:09:48.411 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:09:48 smithi189 podman[67879]: 2024-07-26 11:09:48.138218894 +0000 UTC m=+0.138084524 container create ce38a5f212ef6631add96e73afc3d155f2f6c6adf7582bd0f2c5723a282783af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:09:48.411 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:09:48 smithi189 podman[67879]: 2024-07-26 11:09:48.40972444 +0000 UTC m=+0.409590070 container init ce38a5f212ef6631add96e73afc3d155f2f6c6adf7582bd0f2c5723a282783af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-07-26T11:09:48.694 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:09:48 smithi189 podman[67879]: 2024-07-26 11:09:48.41240308 +0000 UTC m=+0.412268710 container start ce38a5f212ef6631add96e73afc3d155f2f6c6adf7582bd0f2c5723a282783af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:09:48.694 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:09:48 smithi189 ceph-mds[67898]: -- 172.21.15.189:0/1589562987 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56433f415860 con 0x5643435b7180 2024-07-26T11:09:48.694 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:09:48 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: starting mds.h at 2024-07-26T11:09:48.694 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:09:48 smithi189 bash[67879]: ce38a5f212ef6631add96e73afc3d155f2f6c6adf7582bd0f2c5723a282783af 2024-07-26T11:09:48.694 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:09:48 smithi189 systemd[1]: Started Ceph mds.h for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:09:49.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:48 smithi134 ceph-mon[31966]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:49.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:49.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:49.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:49.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:49.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:49.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:49.182 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:48 smithi193 ceph-mon[37473]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:49.182 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:49.182 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:49.182 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:49.182 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:49.182 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:49.182 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:48 smithi189 ceph-mon[36790]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:49.288 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:09:49.753 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:09:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:49 smithi134 ceph-mon[31966]: Deploying daemon mds.i on smithi193 2024-07-26T11:09:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:49 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] up:boot 2024-07-26T11:09:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:49 smithi134 ceph-mon[31966]: fsmap 8 up:standby 2024-07-26T11:09:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T11:09:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:49 smithi189 ceph-mon[36790]: Deploying daemon mds.i on smithi193 2024-07-26T11:09:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:49 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] up:boot 2024-07-26T11:09:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:49 smithi189 ceph-mon[36790]: fsmap 8 up:standby 2024-07-26T11:09:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T11:09:50.264 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:49 smithi193 ceph-mon[37473]: Deploying daemon mds.i on smithi193 2024-07-26T11:09:50.264 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:49 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] up:boot 2024-07-26T11:09:50.264 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:49 smithi193 ceph-mon[37473]: fsmap 8 up:standby 2024-07-26T11:09:50.264 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T11:09:50.937 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:09:50 smithi193 podman[69000]: 2024-07-26 11:09:50.875002109 +0000 UTC m=+0.138120509 container create ac062bd4a9db69de49819bfc9e105117971087922adad2579bc8244d99c59584 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-07-26T11:09:51.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:50 smithi134 ceph-mon[31966]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:50 smithi189 ceph-mon[36790]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:51.230 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:50 smithi193 ceph-mon[37473]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:51.230 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:09:51 smithi193 podman[69000]: 2024-07-26 11:09:51.137036431 +0000 UTC m=+0.400154849 container init ac062bd4a9db69de49819bfc9e105117971087922adad2579bc8244d99c59584 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0) 2024-07-26T11:09:51.230 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:09:51 smithi193 podman[69000]: 2024-07-26 11:09:51.139658634 +0000 UTC m=+0.402777044 container start ac062bd4a9db69de49819bfc9e105117971087922adad2579bc8244d99c59584 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:09:51.230 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:09:51 smithi193 ceph-mds[69020]: -- 172.21.15.193:0/4072704413 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e9e4f61860 con 0x55e9e9103180 2024-07-26T11:09:51.230 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:09:51 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: starting mds.i at 2024-07-26T11:09:51.230 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:09:51 smithi193 bash[69000]: ac062bd4a9db69de49819bfc9e105117971087922adad2579bc8244d99c59584 2024-07-26T11:09:51.524 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:09:51 smithi193 systemd[1]: Started Ceph mds.i for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:09:52.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.058+0000 7f87a7000640 1 -- 172.21.15.134:0/261086941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87a0108ee0 msgr2=0x7f87a01092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:52.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.058+0000 7f87a7000640 1 --2- 172.21.15.134:0/261086941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87a0108ee0 0x7f87a01092f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f878c009900 tx=0x7f878c02a410 comp rx=0 tx=0).stop 2024-07-26T11:09:52.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.058+0000 7f87a7000640 1 -- 172.21.15.134:0/261086941 shutdown_connections 2024-07-26T11:09:52.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.058+0000 7f87a7000640 1 --2- 172.21.15.134:0/261086941 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87a0109830 0x7f87a0110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:52.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.058+0000 7f87a7000640 1 --2- 172.21.15.134:0/261086941 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87a0108ee0 0x7f87a01092f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:52.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.058+0000 7f87a7000640 1 --2- 172.21.15.134:0/261086941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87a01035a0 0x7f87a01039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:52.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.058+0000 7f87a7000640 1 -- 172.21.15.134:0/261086941 >> 172.21.15.134:0/261086941 conn(0x7f87a00ff010 msgr2=0x7f87a0101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:52.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.058+0000 7f87a7000640 1 -- 172.21.15.134:0/261086941 shutdown_connections 2024-07-26T11:09:52.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.058+0000 7f87a7000640 1 -- 172.21.15.134:0/261086941 wait complete. 2024-07-26T11:09:52.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.060+0000 7f87a7000640 1 Processor -- start 2024-07-26T11:09:52.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.060+0000 7f87a7000640 1 -- start start 2024-07-26T11:09:52.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.061+0000 7f87a7000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87a01035a0 0x7f87a0199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:52.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.061+0000 7f87a7000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87a0108ee0 0x7f87a019a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:52.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.061+0000 7f87a7000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87a0109830 0x7f87a019f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:52.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.061+0000 7f87a7000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f87a01134f0 con 0x7f87a0109830 2024-07-26T11:09:52.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.061+0000 7f87a7000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f87a0113370 con 0x7f87a0108ee0 2024-07-26T11:09:52.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.061+0000 7f87a7000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f87a0113670 con 0x7f87a01035a0 2024-07-26T11:09:52.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.061+0000 7f879fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87a01035a0 0x7f87a0199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:52.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.061+0000 7f879fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87a01035a0 0x7f87a0199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:42564/0 (socket says 172.21.15.134:42564) 2024-07-26T11:09:52.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.061+0000 7f879f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87a0108ee0 0x7f87a019a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:52.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.061+0000 7f879fe00640 1 -- 172.21.15.134:0/1329651887 learned_addr learned my addr 172.21.15.134:0/1329651887 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:09:52.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.061+0000 7f87a5000640 1 --2- 172.21.15.134:0/1329651887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87a0109830 0x7f87a019f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:52.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.061+0000 7f879fe00640 1 -- 172.21.15.134:0/1329651887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87a0108ee0 msgr2=0x7f87a019a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:52.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.061+0000 7f879fe00640 1 --2- 172.21.15.134:0/1329651887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87a0108ee0 0x7f87a019a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:52.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.062+0000 7f879fe00640 1 -- 172.21.15.134:0/1329651887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87a0109830 msgr2=0x7f87a019f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:52.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.062+0000 7f879fe00640 1 --2- 172.21.15.134:0/1329651887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87a0109830 0x7f87a019f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:52.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.062+0000 7f879fe00640 1 -- 172.21.15.134:0/1329651887 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a019fed0 con 0x7f87a01035a0 2024-07-26T11:09:52.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.062+0000 7f879fe00640 1 --2- 172.21.15.134:0/1329651887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87a01035a0 0x7f87a0199f60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f879000d950 tx=0x7f879000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:52.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.062+0000 7f879cc00640 1 -- 172.21.15.134:0/1329651887 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8790017030 con 0x7f87a01035a0 2024-07-26T11:09:52.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.062+0000 7f879cc00640 1 -- 172.21.15.134:0/1329651887 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8790004020 con 0x7f87a01035a0 2024-07-26T11:09:52.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.062+0000 7f87a7000640 1 -- 172.21.15.134:0/1329651887 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87a01a01c0 con 0x7f87a01035a0 2024-07-26T11:09:52.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.062+0000 7f879cc00640 1 -- 172.21.15.134:0/1329651887 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87900047a0 con 0x7f87a01035a0 2024-07-26T11:09:52.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.062+0000 7f87a7000640 1 -- 172.21.15.134:0/1329651887 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87a0075020 con 0x7f87a01035a0 2024-07-26T11:09:52.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.064+0000 7f87a7000640 1 -- 172.21.15.134:0/1329651887 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87a01039b0 con 0x7f87a01035a0 2024-07-26T11:09:52.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.067+0000 7f879cc00640 1 -- 172.21.15.134:0/1329651887 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f8790009a30 con 0x7f87a01035a0 2024-07-26T11:09:52.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.069+0000 7f879cc00640 1 --2- 172.21.15.134:0/1329651887 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f87840b3c20 0x7f87840b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:09:52.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.069+0000 7f879cc00640 1 -- 172.21.15.134:0/1329651887 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7342+0+0 (secure 0 0 0) 0x7f87900e32d0 con 0x7f87a01035a0 2024-07-26T11:09:52.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.069+0000 7f879f400640 1 --2- 172.21.15.134:0/1329651887 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f87840b3c20 0x7f87840b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:09:52.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.070+0000 7f879f400640 1 --2- 172.21.15.134:0/1329651887 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f87840b3c20 0x7f87840b6130 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f878c02aa00 tx=0x7f878c005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:09:52.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.073+0000 7f879cc00640 1 -- 172.21.15.134:0/1329651887 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f87900aacf0 con 0x7f87a01035a0 2024-07-26T11:09:52.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.320+0000 7f87a7000640 1 -- 172.21.15.134:0/1329651887 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87a010b670 con 0x7f87a01035a0 2024-07-26T11:09:52.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.321+0000 7f879cc00640 1 -- 172.21.15.134:0/1329651887 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8052 (secure 0 0 0) 0x7f87a010b670 con 0x7f87a01035a0 2024-07-26T11:09:52.324 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:09:52.324 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":10,"btime":"2024-07-26T11:09:51:935252+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24635,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24637,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24641,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24665,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24682,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24686,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24688,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24703,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-07-26T11:09:52.324 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 10 2024-07-26T11:09:52.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.324+0000 7f87a7000640 1 -- 172.21.15.134:0/1329651887 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f87840b3c20 msgr2=0x7f87840b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:52.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.324+0000 7f87a7000640 1 --2- 172.21.15.134:0/1329651887 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f87840b3c20 0x7f87840b6130 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f878c02aa00 tx=0x7f878c005a20 comp rx=0 tx=0).stop 2024-07-26T11:09:52.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.324+0000 7f87a7000640 1 -- 172.21.15.134:0/1329651887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87a01035a0 msgr2=0x7f87a0199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:09:52.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.325+0000 7f87a7000640 1 --2- 172.21.15.134:0/1329651887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87a01035a0 0x7f87a0199f60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f879000d950 tx=0x7f879000de10 comp rx=0 tx=0).stop 2024-07-26T11:09:52.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.325+0000 7f87a7000640 1 -- 172.21.15.134:0/1329651887 shutdown_connections 2024-07-26T11:09:52.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.325+0000 7f87a7000640 1 --2- 172.21.15.134:0/1329651887 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f87840b3c20 0x7f87840b6130 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:52.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.325+0000 7f87a7000640 1 --2- 172.21.15.134:0/1329651887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87a0109830 0x7f87a019f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:52.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.325+0000 7f87a7000640 1 --2- 172.21.15.134:0/1329651887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87a0108ee0 0x7f87a019a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:52.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.325+0000 7f87a7000640 1 --2- 172.21.15.134:0/1329651887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87a01035a0 0x7f87a0199f60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:09:52.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.325+0000 7f87a7000640 1 -- 172.21.15.134:0/1329651887 >> 172.21.15.134:0/1329651887 conn(0x7f87a00ff010 msgr2=0x7f87a0101430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:09:52.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.325+0000 7f87a7000640 1 -- 172.21.15.134:0/1329651887 shutdown_connections 2024-07-26T11:09:52.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:09:52.325+0000 7f87a7000640 1 -- 172.21.15.134:0/1329651887 wait complete. 2024-07-26T11:09:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:52.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:52.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:52.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:52.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:52 smithi134 ceph-mon[31966]: Deploying daemon mds.g on smithi134 2024-07-26T11:09:52.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:52 smithi134 ceph-mon[31966]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:52.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:52 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] up:boot 2024-07-26T11:09:52.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:52 smithi134 ceph-mon[31966]: fsmap 9 up:standby 2024-07-26T11:09:52.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:09:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:52 smithi189 ceph-mon[36790]: Deploying daemon mds.g on smithi134 2024-07-26T11:09:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:52 smithi189 ceph-mon[36790]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:52 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] up:boot 2024-07-26T11:09:52.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:52 smithi189 ceph-mon[36790]: fsmap 9 up:standby 2024-07-26T11:09:52.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:09:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:52 smithi193 ceph-mon[37473]: Deploying daemon mds.g on smithi134 2024-07-26T11:09:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:52 smithi193 ceph-mon[37473]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:52 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] up:boot 2024-07-26T11:09:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:52 smithi193 ceph-mon[37473]: fsmap 9 up:standby 2024-07-26T11:09:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:09:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:53 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1329651887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:09:53.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:53 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1329651887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:09:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:53 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1329651887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:09:54.390 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:54 smithi134 ceph-mon[31966]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:54 smithi189 ceph-mon[36790]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:54.672 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:09:54 smithi134 systemd[1]: Starting Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:09:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:54 smithi193 ceph-mon[37473]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:55.101 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:09:54 smithi134 podman[78167]: 2024-07-26 11:09:54.692965178 +0000 UTC m=+0.031096409 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:09:55.101 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:09:54 smithi134 podman[78167]: 2024-07-26 11:09:54.835953272 +0000 UTC m=+0.174084538 container create c7b6500323c2247dda218cf14db22f45ed63b84a8ec17e17989195bbaffb25bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-07-26T11:09:55.101 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:09:55 smithi134 podman[78167]: 2024-07-26 11:09:55.098111555 +0000 UTC m=+0.436242854 container init c7b6500323c2247dda218cf14db22f45ed63b84a8ec17e17989195bbaffb25bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True) 2024-07-26T11:09:55.422 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:09:55 smithi134 podman[78167]: 2024-07-26 11:09:55.107954263 +0000 UTC m=+0.446085514 container start c7b6500323c2247dda218cf14db22f45ed63b84a8ec17e17989195bbaffb25bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-07-26T11:09:55.422 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:09:55 smithi134 bash[78167]: c7b6500323c2247dda218cf14db22f45ed63b84a8ec17e17989195bbaffb25bf 2024-07-26T11:09:55.422 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:09:55 smithi134 systemd[1]: Started Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:09:55.423 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:09:55 smithi134 ceph-mds[78187]: -- 172.21.15.134:0/953310789 <== mon.0 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562500bf1860 con 0x562504d93a80 2024-07-26T11:09:55.423 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:09:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: starting mds.g at 2024-07-26T11:09:56.152 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:09:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:56 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/3307697142,v1:172.21.15.134:6839/3307697142] up:boot 2024-07-26T11:09:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:56 smithi134 ceph-mon[31966]: fsmap 10 up:standby 2024-07-26T11:09:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:09:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:56 smithi134 ceph-mon[31966]: Deploying daemon mds.a on smithi134 2024-07-26T11:09:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:56 smithi134 ceph-mon[31966]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:56 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/3307697142,v1:172.21.15.134:6839/3307697142] up:boot 2024-07-26T11:09:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:56 smithi189 ceph-mon[36790]: fsmap 10 up:standby 2024-07-26T11:09:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:09:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:56.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:56.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:56.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:56.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:56 smithi189 ceph-mon[36790]: Deploying daemon mds.a on smithi134 2024-07-26T11:09:56.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:56 smithi189 ceph-mon[36790]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:56.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:56.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:56 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/3307697142,v1:172.21.15.134:6839/3307697142] up:boot 2024-07-26T11:09:56.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:56 smithi193 ceph-mon[37473]: fsmap 10 up:standby 2024-07-26T11:09:56.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:09:56.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:09:56.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:09:56.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:09:56.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:09:56.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:56 smithi193 ceph-mon[37473]: Deploying daemon mds.a on smithi134 2024-07-26T11:09:56.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:56 smithi193 ceph-mon[37473]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:57.514 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:09:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:09:58 smithi134 ceph-mon[31966]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:09:58 smithi189 ceph-mon[36790]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:09:58 smithi193 ceph-mon[37473]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:09:59.499 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:09:59 smithi134 podman[78568]: 2024-07-26 11:09:59.334077527 +0000 UTC m=+0.032974784 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:09:59.499 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:09:59 smithi134 podman[78568]: 2024-07-26 11:09:59.496453657 +0000 UTC m=+0.195350950 container create f5e056356436200c7fe3b16c72eae01e5184a6ac07bd3d2a4fad7b3f16d8ca63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-07-26T11:10:00.012 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:09:59 smithi134 podman[78568]: 2024-07-26 11:09:59.885149872 +0000 UTC m=+0.584047171 container init f5e056356436200c7fe3b16c72eae01e5184a6ac07bd3d2a4fad7b3f16d8ca63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:10:00.013 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:09:59 smithi134 podman[78568]: 2024-07-26 11:09:59.895873388 +0000 UTC m=+0.594770684 container start f5e056356436200c7fe3b16c72eae01e5184a6ac07bd3d2a4fad7b3f16d8ca63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724) 2024-07-26T11:10:00.357 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:10:00 smithi134 ceph-mds[78605]: -- 172.21.15.134:0/1622645784 <== mon.0 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bcacb75860 con 0x55bcb0d17180 2024-07-26T11:10:00.357 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:10:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: starting mds.a at 2024-07-26T11:10:00.357 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:10:00 smithi134 bash[78568]: f5e056356436200c7fe3b16c72eae01e5184a6ac07bd3d2a4fad7b3f16d8ca63 2024-07-26T11:10:00.357 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:10:00 smithi134 systemd[1]: Started Ceph mds.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:10:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.425+0000 7ff91ac00640 1 -- 172.21.15.134:0/2635857043 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff91410f330 msgr2=0x7ff91410f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.425+0000 7ff91ac00640 1 --2- 172.21.15.134:0/2635857043 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff91410f330 0x7ff91410f740 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff900009900 tx=0x7ff90002a410 comp rx=0 tx=0).stop 2024-07-26T11:10:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.425+0000 7ff91ac00640 1 -- 172.21.15.134:0/2635857043 shutdown_connections 2024-07-26T11:10:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.425+0000 7ff91ac00640 1 --2- 172.21.15.134:0/2635857043 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff91410fc80 0x7ff9141164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.425+0000 7ff91ac00640 1 --2- 172.21.15.134:0/2635857043 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff91410f330 0x7ff91410f740 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.425+0000 7ff91ac00640 1 --2- 172.21.15.134:0/2635857043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9141099f0 0x7ff914109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.425+0000 7ff91ac00640 1 -- 172.21.15.134:0/2635857043 >> 172.21.15.134:0/2635857043 conn(0x7ff914078ff0 msgr2=0x7ff914107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:00.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.425+0000 7ff91ac00640 1 -- 172.21.15.134:0/2635857043 shutdown_connections 2024-07-26T11:10:00.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.425+0000 7ff91ac00640 1 -- 172.21.15.134:0/2635857043 wait complete. 2024-07-26T11:10:00.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.427+0000 7ff91ac00640 1 Processor -- start 2024-07-26T11:10:00.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.427+0000 7ff91ac00640 1 -- start start 2024-07-26T11:10:00.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.427+0000 7ff91ac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9141099f0 0x7ff9141a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:00.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.427+0000 7ff91ac00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff91410f330 0x7ff9141a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:00.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.427+0000 7ff91ac00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff91410fc80 0x7ff9141a5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.427+0000 7ff91ac00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff914119990 con 0x7ff91410fc80 2024-07-26T11:10:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.427+0000 7ff91ac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff914119810 con 0x7ff9141099f0 2024-07-26T11:10:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.427+0000 7ff91ac00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff914119b10 con 0x7ff91410f330 2024-07-26T11:10:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.427+0000 7ff919800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9141099f0 0x7ff9141a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.427+0000 7ff919800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9141099f0 0x7ff9141a03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:48380/0 (socket says 172.21.15.134:48380) 2024-07-26T11:10:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.427+0000 7ff919800640 1 -- 172.21.15.134:0/3704091279 learned_addr learned my addr 172.21.15.134:0/3704091279 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:10:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.428+0000 7ff918e00640 1 --2- 172.21.15.134:0/3704091279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff91410f330 0x7ff9141a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.428+0000 7ff91a200640 1 --2- 172.21.15.134:0/3704091279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff91410fc80 0x7ff9141a5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.428+0000 7ff919800640 1 -- 172.21.15.134:0/3704091279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff91410f330 msgr2=0x7ff9141a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.428+0000 7ff919800640 1 --2- 172.21.15.134:0/3704091279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff91410f330 0x7ff9141a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.428+0000 7ff919800640 1 -- 172.21.15.134:0/3704091279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff91410fc80 msgr2=0x7ff9141a5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.428+0000 7ff919800640 1 --2- 172.21.15.134:0/3704091279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff91410fc80 0x7ff9141a5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.428+0000 7ff919800640 1 -- 172.21.15.134:0/3704091279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9141a6330 con 0x7ff9141099f0 2024-07-26T11:10:00.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.428+0000 7ff919800640 1 --2- 172.21.15.134:0/3704091279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9141099f0 0x7ff9141a03c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff90800e9f0 tx=0x7ff90800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:00.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.429+0000 7ff912000640 1 -- 172.21.15.134:0/3704091279 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff90800ccc0 con 0x7ff9141099f0 2024-07-26T11:10:00.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.429+0000 7ff912000640 1 -- 172.21.15.134:0/3704091279 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9080108e0 con 0x7ff9141099f0 2024-07-26T11:10:00.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.429+0000 7ff912000640 1 -- 172.21.15.134:0/3704091279 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff908018880 con 0x7ff9141099f0 2024-07-26T11:10:00.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.429+0000 7ff91ac00640 1 -- 172.21.15.134:0/3704091279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9141a6620 con 0x7ff9141099f0 2024-07-26T11:10:00.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.429+0000 7ff91ac00640 1 -- 172.21.15.134:0/3704091279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9141acd80 con 0x7ff9141099f0 2024-07-26T11:10:00.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.430+0000 7ff91ac00640 1 -- 172.21.15.134:0/3704091279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff91410ac70 con 0x7ff9141099f0 2024-07-26T11:10:00.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.433+0000 7ff912000640 1 -- 172.21.15.134:0/3704091279 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7ff908010430 con 0x7ff9141099f0 2024-07-26T11:10:00.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.434+0000 7ff912000640 1 --2- 172.21.15.134:0/3704091279 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8fc0b3ca0 0x7ff8fc0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:00.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.434+0000 7ff912000640 1 -- 172.21.15.134:0/3704091279 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7342+0+0 (secure 0 0 0) 0x7ff9080e3c90 con 0x7ff9141099f0 2024-07-26T11:10:00.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.434+0000 7ff918e00640 1 --2- 172.21.15.134:0/3704091279 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8fc0b3ca0 0x7ff8fc0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:00.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.435+0000 7ff918e00640 1 --2- 172.21.15.134:0/3704091279 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8fc0b3ca0 0x7ff8fc0b61b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff90002aa00 tx=0x7ff900005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:00.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.436+0000 7ff912000640 1 -- 172.21.15.134:0/3704091279 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff9080ab610 con 0x7ff9141099f0 2024-07-26T11:10:00.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.671+0000 7ff91ac00640 1 -- 172.21.15.134:0/3704091279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff914109e00 con 0x7ff9141099f0 2024-07-26T11:10:00.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.672+0000 7ff912000640 1 -- 172.21.15.134:0/3704091279 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8881 (secure 0 0 0) 0x7ff914109e00 con 0x7ff9141099f0 2024-07-26T11:10:00.674 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:10:00.675 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 11 2024-07-26T11:10:00.675 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":11,"btime":"2024-07-26T11:09:55:301114+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24635,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24637,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24641,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24665,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24682,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24686,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24688,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24703,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-07-26T11:10:00.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.676+0000 7ff91ac00640 1 -- 172.21.15.134:0/3704091279 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8fc0b3ca0 msgr2=0x7ff8fc0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:00.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.676+0000 7ff91ac00640 1 --2- 172.21.15.134:0/3704091279 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8fc0b3ca0 0x7ff8fc0b61b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff90002aa00 tx=0x7ff900005a20 comp rx=0 tx=0).stop 2024-07-26T11:10:00.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.676+0000 7ff91ac00640 1 -- 172.21.15.134:0/3704091279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9141099f0 msgr2=0x7ff9141a03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:00.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.676+0000 7ff91ac00640 1 --2- 172.21.15.134:0/3704091279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9141099f0 0x7ff9141a03c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff90800e9f0 tx=0x7ff90800edf0 comp rx=0 tx=0).stop 2024-07-26T11:10:00.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.676+0000 7ff91ac00640 1 -- 172.21.15.134:0/3704091279 shutdown_connections 2024-07-26T11:10:00.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.676+0000 7ff91ac00640 1 --2- 172.21.15.134:0/3704091279 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8fc0b3ca0 0x7ff8fc0b61b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:00.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.676+0000 7ff91ac00640 1 --2- 172.21.15.134:0/3704091279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff91410f330 0x7ff9141a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:00.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.676+0000 7ff91ac00640 1 --2- 172.21.15.134:0/3704091279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff91410fc80 0x7ff9141a5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:00.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.676+0000 7ff91ac00640 1 --2- 172.21.15.134:0/3704091279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9141099f0 0x7ff9141a03c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:00.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.676+0000 7ff91ac00640 1 -- 172.21.15.134:0/3704091279 >> 172.21.15.134:0/3704091279 conn(0x7ff914078ff0 msgr2=0x7ff9140798b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:00.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.676+0000 7ff91ac00640 1 -- 172.21.15.134:0/3704091279 shutdown_connections 2024-07-26T11:10:00.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:00.676+0000 7ff91ac00640 1 -- 172.21.15.134:0/3704091279 wait complete. 2024-07-26T11:10:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:00 smithi189 ceph-mon[36790]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:00 smithi189 ceph-mon[36790]: overall HEALTH_OK 2024-07-26T11:10:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:01.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:01.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:10:01.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:10:01.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:01.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:00 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3704091279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:10:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:00 smithi193 ceph-mon[37473]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:00 smithi193 ceph-mon[37473]: overall HEALTH_OK 2024-07-26T11:10:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:10:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:10:01.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:01.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:00 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3704091279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:10:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:00 smithi134 ceph-mon[31966]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:00 smithi134 ceph-mon[31966]: overall HEALTH_OK 2024-07-26T11:10:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-07-26T11:10:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-07-26T11:10:01.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:01.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:00 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3704091279' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:10:01.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:01 smithi189 ceph-mon[36790]: Deploying daemon mds.b on smithi189 2024-07-26T11:10:01.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:01 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] up:boot 2024-07-26T11:10:01.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:01 smithi189 ceph-mon[36790]: fsmap 11 up:standby 2024-07-26T11:10:01.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:10:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:01 smithi134 ceph-mon[31966]: Deploying daemon mds.b on smithi189 2024-07-26T11:10:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:01 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] up:boot 2024-07-26T11:10:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:01 smithi134 ceph-mon[31966]: fsmap 11 up:standby 2024-07-26T11:10:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:10:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:01 smithi193 ceph-mon[37473]: Deploying daemon mds.b on smithi189 2024-07-26T11:10:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:01 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] up:boot 2024-07-26T11:10:02.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:01 smithi193 ceph-mon[37473]: fsmap 11 up:standby 2024-07-26T11:10:02.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:10:02.366 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:10:02 smithi189 systemd[1]: Starting Ceph mds.b for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:10:02.641 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:10:02 smithi189 podman[68184]: 2024-07-26 11:10:02.267379865 +0000 UTC m=+0.017148352 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:10:02.641 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:10:02 smithi189 podman[68184]: 2024-07-26 11:10:02.386922861 +0000 UTC m=+0.136691375 container create eacc34abe1c23d2c10dff5a4b08da21c581ce34b7e51da638dcb414d04f80784 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:10:02.641 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:10:02 smithi189 podman[68184]: 2024-07-26 11:10:02.639877672 +0000 UTC m=+0.389646145 container init eacc34abe1c23d2c10dff5a4b08da21c581ce34b7e51da638dcb414d04f80784 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2) 2024-07-26T11:10:02.944 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:10:02 smithi189 podman[68184]: 2024-07-26 11:10:02.642839879 +0000 UTC m=+0.392608354 container start eacc34abe1c23d2c10dff5a4b08da21c581ce34b7e51da638dcb414d04f80784 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=) 2024-07-26T11:10:02.944 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:10:02 smithi189 ceph-mds[68203]: -- 172.21.15.189:0/324616201 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b4f38bb860 con 0x55b4f7a5d180 2024-07-26T11:10:02.944 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:10:02 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: starting mds.b at 2024-07-26T11:10:02.944 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:10:02 smithi189 bash[68184]: eacc34abe1c23d2c10dff5a4b08da21c581ce34b7e51da638dcb414d04f80784 2024-07-26T11:10:02.944 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:10:02 smithi189 systemd[1]: Started Ceph mds.b for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:10:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:02 smithi193 ceph-mon[37473]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:03.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:03.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:03.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:02 smithi134 ceph-mon[31966]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:03.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:03.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:03.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:03.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:03.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:02 smithi189 ceph-mon[36790]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:03.757 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:10:04.166 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:03 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] up:boot 2024-07-26T11:10:04.166 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:03 smithi134 ceph-mon[31966]: fsmap 12 up:standby 2024-07-26T11:10:04.166 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T11:10:04.166 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:10:04.166 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:03 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] up:boot 2024-07-26T11:10:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:03 smithi193 ceph-mon[37473]: fsmap 12 up:standby 2024-07-26T11:10:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T11:10:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:10:04.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:04.423 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:03 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] up:boot 2024-07-26T11:10:04.424 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:03 smithi189 ceph-mon[36790]: fsmap 12 up:standby 2024-07-26T11:10:04.424 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T11:10:04.424 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:10:04.424 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:04.833 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:10:05.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:04 smithi134 ceph-mon[31966]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:05.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:05.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:05.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:05.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:04 smithi193 ceph-mon[37473]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:05.364 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:04 smithi189 ceph-mon[36790]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:05.364 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:05.364 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:05.364 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:05.364 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:06 smithi193 ceph-mon[37473]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:07.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.296+0000 7fe58ba00640 1 -- 172.21.15.134:0/2502824114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe584104d50 msgr2=0x7fe58410d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:07.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.296+0000 7fe58ba00640 1 --2- 172.21.15.134:0/2502824114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe584104d50 0x7fe58410d2a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe570009900 tx=0x7fe57002a410 comp rx=0 tx=0).stop 2024-07-26T11:10:07.299 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:06 smithi134 ceph-mon[31966]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:07.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.296+0000 7fe58ba00640 1 -- 172.21.15.134:0/2502824114 shutdown_connections 2024-07-26T11:10:07.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.296+0000 7fe58ba00640 1 --2- 172.21.15.134:0/2502824114 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe58410d860 0x7fe58410fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:07.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.296+0000 7fe58ba00640 1 --2- 172.21.15.134:0/2502824114 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe584104d50 0x7fe58410d2a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:07.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.296+0000 7fe58ba00640 1 --2- 172.21.15.134:0/2502824114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe584104400 0x7fe584104810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:07.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.297+0000 7fe58ba00640 1 -- 172.21.15.134:0/2502824114 >> 172.21.15.134:0/2502824114 conn(0x7fe584079170 msgr2=0x7fe584079580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:07.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.297+0000 7fe58ba00640 1 -- 172.21.15.134:0/2502824114 shutdown_connections 2024-07-26T11:10:07.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.297+0000 7fe58ba00640 1 -- 172.21.15.134:0/2502824114 wait complete. 2024-07-26T11:10:07.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.298+0000 7fe58ba00640 1 Processor -- start 2024-07-26T11:10:07.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.299+0000 7fe58ba00640 1 -- start start 2024-07-26T11:10:07.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.299+0000 7fe58ba00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe584104400 0x7fe58419c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:07.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.299+0000 7fe58ba00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe584104d50 0x7fe58419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:07.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.299+0000 7fe58ba00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe58410d860 0x7fe5841a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:07.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.299+0000 7fe58ba00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe5841134a0 con 0x7fe58410d860 2024-07-26T11:10:07.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.299+0000 7fe58ba00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe584113320 con 0x7fe584104400 2024-07-26T11:10:07.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.299+0000 7fe58ba00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe584113620 con 0x7fe584104d50 2024-07-26T11:10:07.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.299+0000 7fe589a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe58410d860 0x7fe5841a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:07.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.299+0000 7fe589000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe584104400 0x7fe58419c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:07.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.300+0000 7fe583e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe584104d50 0x7fe58419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:07.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.300+0000 7fe583e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe584104d50 0x7fe58419c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:48886/0 (socket says 172.21.15.134:48886) 2024-07-26T11:10:07.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.300+0000 7fe589000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe584104400 0x7fe58419c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:48384/0 (socket says 172.21.15.134:48384) 2024-07-26T11:10:07.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.300+0000 7fe583e00640 1 -- 172.21.15.134:0/3318416132 learned_addr learned my addr 172.21.15.134:0/3318416132 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:10:07.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.300+0000 7fe589000640 1 -- 172.21.15.134:0/3318416132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe584104d50 msgr2=0x7fe58419c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:07.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.300+0000 7fe589000640 1 --2- 172.21.15.134:0/3318416132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe584104d50 0x7fe58419c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:07.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.300+0000 7fe589000640 1 -- 172.21.15.134:0/3318416132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe58410d860 msgr2=0x7fe5841a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:07.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.300+0000 7fe589000640 1 --2- 172.21.15.134:0/3318416132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe58410d860 0x7fe5841a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:07.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.300+0000 7fe589000640 1 -- 172.21.15.134:0/3318416132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5841a2060 con 0x7fe584104400 2024-07-26T11:10:07.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.300+0000 7fe589000640 1 --2- 172.21.15.134:0/3318416132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe584104400 0x7fe58419c0f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe57800e9c0 tx=0x7fe57800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:07.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.301+0000 7fe581600640 1 -- 172.21.15.134:0/3318416132 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57800ccc0 con 0x7fe584104400 2024-07-26T11:10:07.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.301+0000 7fe581600640 1 -- 172.21.15.134:0/3318416132 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe5780035f0 con 0x7fe584104400 2024-07-26T11:10:07.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.301+0000 7fe581600640 1 -- 172.21.15.134:0/3318416132 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57800ce60 con 0x7fe584104400 2024-07-26T11:10:07.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.301+0000 7fe58ba00640 1 -- 172.21.15.134:0/3318416132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5841a2350 con 0x7fe584104400 2024-07-26T11:10:07.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.301+0000 7fe58ba00640 1 -- 172.21.15.134:0/3318416132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5841a8ab0 con 0x7fe584104400 2024-07-26T11:10:07.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.302+0000 7fe58ba00640 1 -- 172.21.15.134:0/3318416132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe584104810 con 0x7fe584104400 2024-07-26T11:10:07.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.307+0000 7fe581600640 1 -- 172.21.15.134:0/3318416132 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fe578004000 con 0x7fe584104400 2024-07-26T11:10:07.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.308+0000 7fe581600640 1 --2- 172.21.15.134:0/3318416132 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe56c0b3ca0 0x7fe56c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:07.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.308+0000 7fe581600640 1 -- 172.21.15.134:0/3318416132 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7342+0+0 (secure 0 0 0) 0x7fe5780e3cf0 con 0x7fe584104400 2024-07-26T11:10:07.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.309+0000 7fe583e00640 1 --2- 172.21.15.134:0/3318416132 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe56c0b3ca0 0x7fe56c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:07.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.309+0000 7fe581600640 1 -- 172.21.15.134:0/3318416132 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe5780e4250 con 0x7fe584104400 2024-07-26T11:10:07.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.309+0000 7fe583e00640 1 --2- 172.21.15.134:0/3318416132 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe56c0b3ca0 0x7fe56c0b61b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe58410f700 tx=0x7fe570009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:06 smithi189 ceph-mon[36790]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:07.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.558+0000 7fe58ba00640 1 -- 172.21.15.134:0/3318416132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe58419d220 con 0x7fe584104400 2024-07-26T11:10:07.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.559+0000 7fe581600640 1 -- 172.21.15.134:0/3318416132 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10536 (secure 0 0 0) 0x7fe5780ab7a0 con 0x7fe584104400 2024-07-26T11:10:07.562 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:10:07.562 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":13,"btime":"2024-07-26T11:10:02:944720+0000","default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24635,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24637,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24641,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24665,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24682,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24686,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24688,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24703,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24719,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24721,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-07-26T11:10:07.562 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 13 2024-07-26T11:10:07.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.562+0000 7fe58ba00640 1 -- 172.21.15.134:0/3318416132 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe56c0b3ca0 msgr2=0x7fe56c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:07.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.562+0000 7fe58ba00640 1 --2- 172.21.15.134:0/3318416132 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe56c0b3ca0 0x7fe56c0b61b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe58410f700 tx=0x7fe570009560 comp rx=0 tx=0).stop 2024-07-26T11:10:07.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.562+0000 7fe58ba00640 1 -- 172.21.15.134:0/3318416132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe584104400 msgr2=0x7fe58419c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:07.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.562+0000 7fe58ba00640 1 --2- 172.21.15.134:0/3318416132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe584104400 0x7fe58419c0f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe57800e9c0 tx=0x7fe57800ee80 comp rx=0 tx=0).stop 2024-07-26T11:10:07.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.562+0000 7fe58ba00640 1 -- 172.21.15.134:0/3318416132 shutdown_connections 2024-07-26T11:10:07.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.562+0000 7fe58ba00640 1 --2- 172.21.15.134:0/3318416132 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe56c0b3ca0 0x7fe56c0b61b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:07.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.562+0000 7fe58ba00640 1 --2- 172.21.15.134:0/3318416132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe584104d50 0x7fe58419c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:07.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.562+0000 7fe58ba00640 1 --2- 172.21.15.134:0/3318416132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe58410d860 0x7fe5841a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:07.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.562+0000 7fe58ba00640 1 --2- 172.21.15.134:0/3318416132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe584104400 0x7fe58419c0f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:07.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.562+0000 7fe58ba00640 1 -- 172.21.15.134:0/3318416132 >> 172.21.15.134:0/3318416132 conn(0x7fe584079170 msgr2=0x7fe5841096b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:07.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.563+0000 7fe58ba00640 1 -- 172.21.15.134:0/3318416132 shutdown_connections 2024-07-26T11:10:07.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:07.563+0000 7fe58ba00640 1 -- 172.21.15.134:0/3318416132 wait complete. 2024-07-26T11:10:08.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-07-26T11:10:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:08 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3318416132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:10:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:10:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:08 smithi193 ceph-mon[37473]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:08.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:08.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:08.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:08 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3318416132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:10:08.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:10:08.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:08 smithi134 ceph-mon[31966]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:08.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:08.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:08 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3318416132' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:10:08.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:10:08.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:08 smithi189 ceph-mon[36790]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:09.518 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:10:10.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:10.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:10.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:10.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:10.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:10 smithi134 ceph-mon[31966]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:10 smithi189 ceph-mon[36790]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:10.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:10.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:10 smithi193 ceph-mon[37473]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:11.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.779+0000 7f4e7b000640 1 -- 172.21.15.134:0/1901984666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e74116070 msgr2=0x7f4e74118580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:11.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.779+0000 7f4e7b000640 1 --2- 172.21.15.134:0/1901984666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e74116070 0x7f4e74118580 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009ca0 tx=0x7f4e6c02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:10:11.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.780+0000 7f4e7b000640 1 -- 172.21.15.134:0/1901984666 shutdown_connections 2024-07-26T11:10:11.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.780+0000 7f4e7b000640 1 --2- 172.21.15.134:0/1901984666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e7410a910 0x7f4e74115a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:11.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.780+0000 7f4e7b000640 1 --2- 172.21.15.134:0/1901984666 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e74116070 0x7f4e74118580 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:11.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.780+0000 7f4e7b000640 1 --2- 172.21.15.134:0/1901984666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e74109fc0 0x7f4e7410a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:11.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.780+0000 7f4e7b000640 1 -- 172.21.15.134:0/1901984666 >> 172.21.15.134:0/1901984666 conn(0x7f4e740708f0 msgr2=0x7f4e74070d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:11.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.780+0000 7f4e7b000640 1 -- 172.21.15.134:0/1901984666 shutdown_connections 2024-07-26T11:10:11.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.780+0000 7f4e7b000640 1 -- 172.21.15.134:0/1901984666 wait complete. 2024-07-26T11:10:11.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.781+0000 7f4e7b000640 1 Processor -- start 2024-07-26T11:10:11.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.781+0000 7f4e7b000640 1 -- start start 2024-07-26T11:10:11.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.782+0000 7f4e7b000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e74109fc0 0x7f4e7410b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:11.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.782+0000 7f4e7b000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e7410a910 0x7f4e7410bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:11.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.782+0000 7f4e7b000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e74116070 0x7f4e7410c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:11.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.782+0000 7f4e7b000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7411bd60 con 0x7f4e74109fc0 2024-07-26T11:10:11.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.782+0000 7f4e7b000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7411bbe0 con 0x7f4e7410a910 2024-07-26T11:10:11.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.782+0000 7f4e7b000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7411bee0 con 0x7f4e74116070 2024-07-26T11:10:11.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.782+0000 7f4e73e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e74109fc0 0x7f4e7410b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:11.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.782+0000 7f4e73400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e7410a910 0x7f4e7410bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:11.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.782+0000 7f4e73400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e7410a910 0x7f4e7410bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:50380/0 (socket says 172.21.15.134:50380) 2024-07-26T11:10:11.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.782+0000 7f4e73e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e74109fc0 0x7f4e7410b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50814/0 (socket says 172.21.15.134:50814) 2024-07-26T11:10:11.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.782+0000 7f4e73400640 1 -- 172.21.15.134:0/3832116183 learned_addr learned my addr 172.21.15.134:0/3832116183 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:10:11.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.782+0000 7f4e79000640 1 --2- 172.21.15.134:0/3832116183 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e74116070 0x7f4e7410c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:11.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.782+0000 7f4e73400640 1 -- 172.21.15.134:0/3832116183 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e74116070 msgr2=0x7f4e7410c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:11.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.782+0000 7f4e73400640 1 --2- 172.21.15.134:0/3832116183 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e74116070 0x7f4e7410c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:11.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.783+0000 7f4e73400640 1 -- 172.21.15.134:0/3832116183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e74109fc0 msgr2=0x7f4e7410b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:11.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.783+0000 7f4e73400640 1 --2- 172.21.15.134:0/3832116183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e74109fc0 0x7f4e7410b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:11.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.783+0000 7f4e73400640 1 -- 172.21.15.134:0/3832116183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e741b0970 con 0x7f4e7410a910 2024-07-26T11:10:11.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.783+0000 7f4e73e00640 1 --2- 172.21.15.134:0/3832116183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e74109fc0 0x7f4e7410b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:10:11.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.783+0000 7f4e73400640 1 --2- 172.21.15.134:0/3832116183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e7410a910 0x7f4e7410bdf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00e9c0 tx=0x7f4e5c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:11.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.783+0000 7f4e70c00640 1 -- 172.21.15.134:0/3832116183 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c00ccc0 con 0x7f4e7410a910 2024-07-26T11:10:11.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.783+0000 7f4e70c00640 1 -- 172.21.15.134:0/3832116183 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e5c0054d0 con 0x7f4e7410a910 2024-07-26T11:10:11.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.783+0000 7f4e70c00640 1 -- 172.21.15.134:0/3832116183 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c010430 con 0x7f4e7410a910 2024-07-26T11:10:11.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.783+0000 7f4e7b000640 1 -- 172.21.15.134:0/3832116183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e741b0c60 con 0x7f4e7410a910 2024-07-26T11:10:11.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.783+0000 7f4e7b000640 1 -- 172.21.15.134:0/3832116183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e741b1150 con 0x7f4e7410a910 2024-07-26T11:10:11.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.785+0000 7f4e7b000640 1 -- 172.21.15.134:0/3832116183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e7410a3d0 con 0x7f4e7410a910 2024-07-26T11:10:11.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.788+0000 7f4e70c00640 1 -- 172.21.15.134:0/3832116183 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f4e5c005000 con 0x7f4e7410a910 2024-07-26T11:10:11.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.789+0000 7f4e70c00640 1 --2- 172.21.15.134:0/3832116183 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4e580b3ca0 0x7f4e580b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:11.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.790+0000 7f4e70c00640 1 -- 172.21.15.134:0/3832116183 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7342+0+0 (secure 0 0 0) 0x7f4e5c0e3d60 con 0x7f4e7410a910 2024-07-26T11:10:11.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.790+0000 7f4e73e00640 1 --2- 172.21.15.134:0/3832116183 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4e580b3ca0 0x7f4e580b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:11.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.790+0000 7f4e70c00640 1 -- 172.21.15.134:0/3832116183 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4e5c055600 con 0x7f4e7410a910 2024-07-26T11:10:11.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:11.790+0000 7f4e73e00640 1 --2- 172.21.15.134:0/3832116183 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4e580b3ca0 0x7f4e580b61b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4e7410d300 tx=0x7f4e6400b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:12.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:12.001+0000 7f4e7b000640 1 -- 172.21.15.134:0/3832116183 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f4e741b15b0 con 0x7f4e7410a910 2024-07-26T11:10:12.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:12.006+0000 7f4e70c00640 1 -- 172.21.15.134:0/3832116183 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v74) ==== 84+0+65 (secure 0 0 0) 0x7f4e5c0052f0 con 0x7f4e7410a910 2024-07-26T11:10:12.009 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:10:12.009 INFO:teuthology.orchestra.run.smithi134.stdout:[ 2024-07-26T11:10:12.009 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-07-26T11:10:12.009 INFO:teuthology.orchestra.run.smithi134.stdout: "poolnum": 1, 2024-07-26T11:10:12.009 INFO:teuthology.orchestra.run.smithi134.stdout: "poolname": ".mgr" 2024-07-26T11:10:12.009 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-07-26T11:10:12.009 INFO:teuthology.orchestra.run.smithi134.stdout:] 2024-07-26T11:10:12.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:12.009+0000 7f4e7b000640 1 -- 172.21.15.134:0/3832116183 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4e580b3ca0 msgr2=0x7f4e580b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:12.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:12.009+0000 7f4e7b000640 1 --2- 172.21.15.134:0/3832116183 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4e580b3ca0 0x7f4e580b61b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4e7410d300 tx=0x7f4e6400b410 comp rx=0 tx=0).stop 2024-07-26T11:10:12.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:12.009+0000 7f4e7b000640 1 -- 172.21.15.134:0/3832116183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e7410a910 msgr2=0x7f4e7410bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:12.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:12.009+0000 7f4e7b000640 1 --2- 172.21.15.134:0/3832116183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e7410a910 0x7f4e7410bdf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00e9c0 tx=0x7f4e5c00ee80 comp rx=0 tx=0).stop 2024-07-26T11:10:12.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:12.009+0000 7f4e7b000640 1 -- 172.21.15.134:0/3832116183 shutdown_connections 2024-07-26T11:10:12.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:12.009+0000 7f4e7b000640 1 --2- 172.21.15.134:0/3832116183 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4e580b3ca0 0x7f4e580b61b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:12.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:12.009+0000 7f4e7b000640 1 --2- 172.21.15.134:0/3832116183 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e74116070 0x7f4e7410c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:12.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:12.009+0000 7f4e7b000640 1 --2- 172.21.15.134:0/3832116183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e7410a910 0x7f4e7410bdf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:12.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:12.009+0000 7f4e7b000640 1 --2- 172.21.15.134:0/3832116183 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4e74109fc0 0x7f4e7410b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:12.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:12.009+0000 7f4e7b000640 1 -- 172.21.15.134:0/3832116183 >> 172.21.15.134:0/3832116183 conn(0x7f4e740708f0 msgr2=0x7f4e74071850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:12.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:12.010+0000 7f4e7b000640 1 -- 172.21.15.134:0/3832116183 shutdown_connections 2024-07-26T11:10:12.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:12.010+0000 7f4e7b000640 1 -- 172.21.15.134:0/3832116183 wait complete. 2024-07-26T11:10:12.997 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-07-26T11:10:12.998 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-07-26T11:10:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:12 smithi193 ceph-mon[37473]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:13.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:13.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:12 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3832116183' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T11:10:13.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:13.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:13.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:13.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:12 smithi134 ceph-mon[31966]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:12 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3832116183' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T11:10:13.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:13.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:13.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:13.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:12 smithi189 ceph-mon[36790]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:12 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3832116183' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-07-26T11:10:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:10:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:10:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:10:13.452 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:10:15.254 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:14 smithi134 ceph-mon[31966]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:14 smithi193 ceph-mon[37473]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:14 smithi189 ceph-mon[36790]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:15.837 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.834+0000 7f988ce00640 1 -- 172.21.15.134:0/2314365447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f988810d840 msgr2=0x7f988810fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:15.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.834+0000 7f988ce00640 1 --2- 172.21.15.134:0/2314365447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f988810d840 0x7f988810fd30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9880009ca0 tx=0x7f988002a8c0 comp rx=0 tx=0).stop 2024-07-26T11:10:15.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.834+0000 7f988ce00640 1 -- 172.21.15.134:0/2314365447 shutdown_connections 2024-07-26T11:10:15.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.834+0000 7f988ce00640 1 --2- 172.21.15.134:0/2314365447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f988810d840 0x7f988810fd30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:15.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.834+0000 7f988ce00640 1 --2- 172.21.15.134:0/2314365447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9888104d30 0x7f988810d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:15.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.834+0000 7f988ce00640 1 --2- 172.21.15.134:0/2314365447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98881043e0 0x7f98881047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:15.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.834+0000 7f988ce00640 1 -- 172.21.15.134:0/2314365447 >> 172.21.15.134:0/2314365447 conn(0x7f9888079160 msgr2=0x7f9888079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:15.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.834+0000 7f988ce00640 1 -- 172.21.15.134:0/2314365447 shutdown_connections 2024-07-26T11:10:15.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.834+0000 7f988ce00640 1 -- 172.21.15.134:0/2314365447 wait complete. 2024-07-26T11:10:15.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.836+0000 7f988ce00640 1 Processor -- start 2024-07-26T11:10:15.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.836+0000 7f988ce00640 1 -- start start 2024-07-26T11:10:15.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.837+0000 7f988ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98881043e0 0x7f9888199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:15.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.837+0000 7f988ce00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9888104d30 0x7f988819a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:15.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.837+0000 7f988ce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f988810d840 0x7f988819f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:15.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.837+0000 7f988ce00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9888113510 con 0x7f9888104d30 2024-07-26T11:10:15.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.837+0000 7f988ce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9888113390 con 0x7f98881043e0 2024-07-26T11:10:15.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.837+0000 7f988ce00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9888113690 con 0x7f988810d840 2024-07-26T11:10:15.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.837+0000 7f9885e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98881043e0 0x7f9888199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:15.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.837+0000 7f9885400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9888104d30 0x7f988819a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:15.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.838+0000 7f9885e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98881043e0 0x7f9888199ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:50408/0 (socket says 172.21.15.134:50408) 2024-07-26T11:10:15.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.838+0000 7f9885e00640 1 -- 172.21.15.134:0/1503089382 learned_addr learned my addr 172.21.15.134:0/1503089382 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:10:15.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.838+0000 7f9886800640 1 --2- 172.21.15.134:0/1503089382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f988810d840 0x7f988819f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:15.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.838+0000 7f9885e00640 1 -- 172.21.15.134:0/1503089382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f988810d840 msgr2=0x7f988819f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:15.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.838+0000 7f9885e00640 1 --2- 172.21.15.134:0/1503089382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f988810d840 0x7f988819f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:15.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.838+0000 7f9885e00640 1 -- 172.21.15.134:0/1503089382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9888104d30 msgr2=0x7f988819a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:15.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.838+0000 7f9885e00640 1 --2- 172.21.15.134:0/1503089382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9888104d30 0x7f988819a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:15.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.838+0000 7f9885e00640 1 -- 172.21.15.134:0/1503089382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988819fe30 con 0x7f98881043e0 2024-07-26T11:10:15.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.838+0000 7f9885e00640 1 --2- 172.21.15.134:0/1503089382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98881043e0 0x7f9888199ec0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f987800e9c0 tx=0x7f987800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:15.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.839+0000 7f987ea00640 1 -- 172.21.15.134:0/1503089382 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987800cd80 con 0x7f98881043e0 2024-07-26T11:10:15.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.839+0000 7f987ea00640 1 -- 172.21.15.134:0/1503089382 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98780035f0 con 0x7f98881043e0 2024-07-26T11:10:15.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.839+0000 7f988ce00640 1 -- 172.21.15.134:0/1503089382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98881a0120 con 0x7f98881043e0 2024-07-26T11:10:15.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.839+0000 7f987ea00640 1 -- 172.21.15.134:0/1503089382 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9878010690 con 0x7f98881043e0 2024-07-26T11:10:15.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.839+0000 7f988ce00640 1 -- 172.21.15.134:0/1503089382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98881a6880 con 0x7f98881043e0 2024-07-26T11:10:15.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.840+0000 7f988ce00640 1 -- 172.21.15.134:0/1503089382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98881047f0 con 0x7f98881043e0 2024-07-26T11:10:15.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.842+0000 7f987ea00640 1 -- 172.21.15.134:0/1503089382 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f9878004000 con 0x7f98881043e0 2024-07-26T11:10:15.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.844+0000 7f987ea00640 1 --2- 172.21.15.134:0/1503089382 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f986c0b3ca0 0x7f986c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:15.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.844+0000 7f987ea00640 1 -- 172.21.15.134:0/1503089382 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7342+0+0 (secure 0 0 0) 0x7f98780e3dc0 con 0x7f98881043e0 2024-07-26T11:10:15.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.844+0000 7f9885400640 1 --2- 172.21.15.134:0/1503089382 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f986c0b3ca0 0x7f986c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:15.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.845+0000 7f9885400640 1 --2- 172.21.15.134:0/1503089382 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f986c0b3ca0 0x7f986c0b61b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f988810f4a0 tx=0x7f9874005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:15.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:15.848+0000 7f987ea00640 1 -- 172.21.15.134:0/1503089382 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f98780ab740 con 0x7f98881043e0 2024-07-26T11:10:16.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:16.036+0000 7f988ce00640 1 -- 172.21.15.134:0/1503089382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f98881a6c20 con 0x7f98881043e0 2024-07-26T11:10:17.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:17.020+0000 7f987ea00640 1 -- 172.21.15.134:0/1503089382 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v75) ==== 138+0+0 (secure 0 0 0) 0x7f98780ab040 con 0x7f98881043e0 2024-07-26T11:10:17.026 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'cephfs_metadata' created 2024-07-26T11:10:17.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:17.026+0000 7f988ce00640 1 -- 172.21.15.134:0/1503089382 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f986c0b3ca0 msgr2=0x7f986c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:17.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:17.026+0000 7f988ce00640 1 --2- 172.21.15.134:0/1503089382 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f986c0b3ca0 0x7f986c0b61b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f988810f4a0 tx=0x7f9874005f90 comp rx=0 tx=0).stop 2024-07-26T11:10:17.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:17.026+0000 7f988ce00640 1 -- 172.21.15.134:0/1503089382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98881043e0 msgr2=0x7f9888199ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:17.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:17.026+0000 7f988ce00640 1 --2- 172.21.15.134:0/1503089382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98881043e0 0x7f9888199ec0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f987800e9c0 tx=0x7f987800ee80 comp rx=0 tx=0).stop 2024-07-26T11:10:17.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:17.027+0000 7f988ce00640 1 -- 172.21.15.134:0/1503089382 shutdown_connections 2024-07-26T11:10:17.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:17.027+0000 7f988ce00640 1 --2- 172.21.15.134:0/1503089382 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f986c0b3ca0 0x7f986c0b61b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:17.027+0000 7f988ce00640 1 --2- 172.21.15.134:0/1503089382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f988810d840 0x7f988819f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:17.027+0000 7f988ce00640 1 --2- 172.21.15.134:0/1503089382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9888104d30 0x7f988819a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:17.027+0000 7f988ce00640 1 --2- 172.21.15.134:0/1503089382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98881043e0 0x7f9888199ec0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:17.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:17.027+0000 7f988ce00640 1 -- 172.21.15.134:0/1503089382 >> 172.21.15.134:0/1503089382 conn(0x7f9888079160 msgr2=0x7f9888100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:17.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:17.028+0000 7f988ce00640 1 -- 172.21.15.134:0/1503089382 shutdown_connections 2024-07-26T11:10:17.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:17.028+0000 7f988ce00640 1 -- 172.21.15.134:0/1503089382 wait complete. 2024-07-26T11:10:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:16 smithi193 ceph-mon[37473]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:16 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1503089382' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T11:10:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:16 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T11:10:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:16 smithi134 ceph-mon[31966]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:16 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1503089382' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T11:10:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:16 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T11:10:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:16 smithi189 ceph-mon[36790]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:16 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1503089382' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T11:10:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:16 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-07-26T11:10:18.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-07-26T11:10:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:18 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T11:10:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:18 smithi134 ceph-mon[31966]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T11:10:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:18 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T11:10:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:18 smithi189 ceph-mon[36790]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T11:10:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:18 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-07-26T11:10:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:18 smithi193 ceph-mon[37473]: osdmap e75: 12 total, 12 up, 12 in 2024-07-26T11:10:18.541 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:10:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:19 smithi134 ceph-mon[31966]: pgmap v295: 33 pgs: 12 creating+peering, 1 active+clean, 20 unknown; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:19.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:19 smithi134 ceph-mon[31966]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T11:10:19.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:19 smithi134 ceph-mon[31966]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T11:10:19.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:19 smithi193 ceph-mon[37473]: pgmap v295: 33 pgs: 12 creating+peering, 1 active+clean, 20 unknown; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:19.426 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:19 smithi193 ceph-mon[37473]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T11:10:19.426 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:19 smithi193 ceph-mon[37473]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T11:10:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:19 smithi189 ceph-mon[36790]: pgmap v295: 33 pgs: 12 creating+peering, 1 active+clean, 20 unknown; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:19 smithi189 ceph-mon[36790]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T11:10:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:19 smithi189 ceph-mon[36790]: osdmap e76: 12 total, 12 up, 12 in 2024-07-26T11:10:20.402 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:20 smithi134 ceph-mon[31966]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T11:10:20.402 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:20 smithi134 ceph-mon[31966]: pgmap v298: 33 pgs: 12 creating+peering, 9 active+clean, 12 unknown; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:20.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:20 smithi189 ceph-mon[36790]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T11:10:20.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:20 smithi189 ceph-mon[36790]: pgmap v298: 33 pgs: 12 creating+peering, 9 active+clean, 12 unknown; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:20 smithi193 ceph-mon[37473]: osdmap e77: 12 total, 12 up, 12 in 2024-07-26T11:10:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:20 smithi193 ceph-mon[37473]: pgmap v298: 33 pgs: 12 creating+peering, 9 active+clean, 12 unknown; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:20.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.987+0000 7f3e78a00640 1 -- 172.21.15.134:0/4069653429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e74109650 msgr2=0x7f3e7410fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:20.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.987+0000 7f3e78a00640 1 --2- 172.21.15.134:0/4069653429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e74109650 0x7f3e7410fe80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c009ca0 tx=0x7f3e6c02a8b0 comp rx=0 tx=0).stop 2024-07-26T11:10:20.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.987+0000 7f3e78a00640 1 -- 172.21.15.134:0/4069653429 shutdown_connections 2024-07-26T11:10:20.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.987+0000 7f3e78a00640 1 --2- 172.21.15.134:0/4069653429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e74109650 0x7f3e7410fe80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:20.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.987+0000 7f3e78a00640 1 --2- 172.21.15.134:0/4069653429 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e74108d00 0x7f3e74109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:20.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.987+0000 7f3e78a00640 1 --2- 172.21.15.134:0/4069653429 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e741033c0 0x7f3e741037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:20.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.987+0000 7f3e78a00640 1 -- 172.21.15.134:0/4069653429 >> 172.21.15.134:0/4069653429 conn(0x7f3e740fee10 msgr2=0x7f3e74101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:20.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.987+0000 7f3e78a00640 1 -- 172.21.15.134:0/4069653429 shutdown_connections 2024-07-26T11:10:20.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.987+0000 7f3e78a00640 1 -- 172.21.15.134:0/4069653429 wait complete. 2024-07-26T11:10:20.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.988+0000 7f3e78a00640 1 Processor -- start 2024-07-26T11:10:20.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.989+0000 7f3e78a00640 1 -- start start 2024-07-26T11:10:20.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.989+0000 7f3e78a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e741033c0 0x7f3e74199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:20.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.989+0000 7f3e78a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e74108d00 0x7f3e7419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:20.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.989+0000 7f3e78a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e74109650 0x7f3e7419f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:20.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.989+0000 7f3e78a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3e74113350 con 0x7f3e741033c0 2024-07-26T11:10:20.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.989+0000 7f3e78a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3e741131d0 con 0x7f3e74108d00 2024-07-26T11:10:20.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.989+0000 7f3e78a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3e741134d0 con 0x7f3e74109650 2024-07-26T11:10:20.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.990+0000 7f3e73e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e74109650 0x7f3e7419f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:20.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.990+0000 7f3e73400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e741033c0 0x7f3e74199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:20.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.990+0000 7f3e73e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e74109650 0x7f3e7419f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:50378/0 (socket says 172.21.15.134:50378) 2024-07-26T11:10:20.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.990+0000 7f3e73e00640 1 -- 172.21.15.134:0/4002739641 learned_addr learned my addr 172.21.15.134:0/4002739641 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:10:20.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.990+0000 7f3e72a00640 1 --2- 172.21.15.134:0/4002739641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e74108d00 0x7f3e7419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:20.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.990+0000 7f3e73e00640 1 -- 172.21.15.134:0/4002739641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e74108d00 msgr2=0x7f3e7419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:20.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.990+0000 7f3e73e00640 1 --2- 172.21.15.134:0/4002739641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e74108d00 0x7f3e7419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:20.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.990+0000 7f3e73e00640 1 -- 172.21.15.134:0/4002739641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e741033c0 msgr2=0x7f3e74199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:20.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.990+0000 7f3e73e00640 1 --2- 172.21.15.134:0/4002739641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e741033c0 0x7f3e74199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:20.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.990+0000 7f3e73e00640 1 -- 172.21.15.134:0/4002739641 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7419fc30 con 0x7f3e74109650 2024-07-26T11:10:20.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.991+0000 7f3e73e00640 1 --2- 172.21.15.134:0/4002739641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e74109650 0x7f3e7419f500 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c005e30 tx=0x7f3e6c030dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:20.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.991+0000 7f3e6be00640 1 -- 172.21.15.134:0/4002739641 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c04b070 con 0x7f3e74109650 2024-07-26T11:10:20.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.991+0000 7f3e6be00640 1 -- 172.21.15.134:0/4002739641 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e6c003c40 con 0x7f3e74109650 2024-07-26T11:10:20.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.991+0000 7f3e78a00640 1 -- 172.21.15.134:0/4002739641 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e7419fec0 con 0x7f3e74109650 2024-07-26T11:10:20.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.991+0000 7f3e6be00640 1 -- 172.21.15.134:0/4002739641 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c004460 con 0x7f3e74109650 2024-07-26T11:10:20.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.991+0000 7f3e78a00640 1 -- 172.21.15.134:0/4002739641 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e74075020 con 0x7f3e74109650 2024-07-26T11:10:20.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.992+0000 7f3e78a00640 1 -- 172.21.15.134:0/4002739641 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e741037d0 con 0x7f3e74109650 2024-07-26T11:10:20.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.994+0000 7f3e6be00640 1 -- 172.21.15.134:0/4002739641 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f3e6c04e070 con 0x7f3e74109650 2024-07-26T11:10:20.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.996+0000 7f3e6be00640 1 --2- 172.21.15.134:0/4002739641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3e580b3ca0 0x7f3e580b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:20.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.996+0000 7f3e6be00640 1 -- 172.21.15.134:0/4002739641 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 7712+0+0 (secure 0 0 0) 0x7f3e6c107d20 con 0x7f3e74109650 2024-07-26T11:10:20.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.996+0000 7f3e73400640 1 --2- 172.21.15.134:0/4002739641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3e580b3ca0 0x7f3e580b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:20.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.997+0000 7f3e73400640 1 --2- 172.21.15.134:0/4002739641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3e580b3ca0 0x7f3e580b61b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3e640095f0 tx=0x7f3e64005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:20.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:20.997+0000 7f3e6be00640 1 -- 172.21.15.134:0/4002739641 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3e6c084600 con 0x7f3e74109650 2024-07-26T11:10:21.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:21.196+0000 7f3e78a00640 1 -- 172.21.15.134:0/4002739641 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f3e7419adf0 con 0x7f3e74109650 2024-07-26T11:10:21.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:21.278+0000 7f3e6be00640 1 -- 172.21.15.134:0/4002739641 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v78) ==== 171+0+0 (secure 0 0 0) 0x7f3e7419adf0 con 0x7f3e74109650 2024-07-26T11:10:21.281 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'cephfs_data' created 2024-07-26T11:10:21.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:21.284+0000 7f3e78a00640 1 -- 172.21.15.134:0/4002739641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3e580b3ca0 msgr2=0x7f3e580b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:21.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:21.284+0000 7f3e78a00640 1 --2- 172.21.15.134:0/4002739641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3e580b3ca0 0x7f3e580b61b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3e640095f0 tx=0x7f3e64005f90 comp rx=0 tx=0).stop 2024-07-26T11:10:21.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:21.284+0000 7f3e78a00640 1 -- 172.21.15.134:0/4002739641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e74109650 msgr2=0x7f3e7419f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:21.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:21.284+0000 7f3e78a00640 1 --2- 172.21.15.134:0/4002739641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e74109650 0x7f3e7419f500 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c005e30 tx=0x7f3e6c030dc0 comp rx=0 tx=0).stop 2024-07-26T11:10:21.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:21.285+0000 7f3e78a00640 1 -- 172.21.15.134:0/4002739641 shutdown_connections 2024-07-26T11:10:21.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:21.285+0000 7f3e78a00640 1 --2- 172.21.15.134:0/4002739641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3e580b3ca0 0x7f3e580b61b0 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3e640095f0 tx=0x7f3e64005f90 comp rx=0 tx=0).stop 2024-07-26T11:10:21.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:21.285+0000 7f3e78a00640 1 --2- 172.21.15.134:0/4002739641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e74109650 0x7f3e7419f500 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:21.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:21.285+0000 7f3e78a00640 1 --2- 172.21.15.134:0/4002739641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e74108d00 0x7f3e7419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:21.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:21.285+0000 7f3e78a00640 1 --2- 172.21.15.134:0/4002739641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3e741033c0 0x7f3e74199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:21.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:21.285+0000 7f3e78a00640 1 -- 172.21.15.134:0/4002739641 >> 172.21.15.134:0/4002739641 conn(0x7f3e740fee10 msgr2=0x7f3e74107320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:21.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:21.286+0000 7f3e78a00640 1 -- 172.21.15.134:0/4002739641 shutdown_connections 2024-07-26T11:10:21.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:21.286+0000 7f3e78a00640 1 -- 172.21.15.134:0/4002739641 wait complete. 2024-07-26T11:10:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:21 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/4002739641' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T11:10:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:21 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T11:10:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:21 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/4002739641' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T11:10:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:21 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T11:10:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:21 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/4002739641' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T11:10:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:21 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-07-26T11:10:22.301 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-07-26T11:10:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:22 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-07-26T11:10:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:22 smithi134 ceph-mon[31966]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T11:10:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:22 smithi134 ceph-mon[31966]: pgmap v300: 97 pgs: 20 creating+peering, 21 active+clean, 56 unknown; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:22 smithi134 ceph-mon[31966]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T11:10:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:22 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-07-26T11:10:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:22 smithi189 ceph-mon[36790]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T11:10:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:22 smithi189 ceph-mon[36790]: pgmap v300: 97 pgs: 20 creating+peering, 21 active+clean, 56 unknown; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:22 smithi189 ceph-mon[36790]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T11:10:22.766 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:10:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:22 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-07-26T11:10:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:22 smithi193 ceph-mon[37473]: osdmap e78: 12 total, 12 up, 12 in 2024-07-26T11:10:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:22 smithi193 ceph-mon[37473]: pgmap v300: 97 pgs: 20 creating+peering, 21 active+clean, 56 unknown; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:22 smithi193 ceph-mon[37473]: osdmap e79: 12 total, 12 up, 12 in 2024-07-26T11:10:24.620 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:24 smithi134 ceph-mon[31966]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T11:10:24.620 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:24 smithi134 ceph-mon[31966]: pgmap v303: 97 pgs: 12 creating+peering, 57 active+clean, 28 unknown; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:24 smithi189 ceph-mon[36790]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T11:10:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:24 smithi189 ceph-mon[36790]: pgmap v303: 97 pgs: 12 creating+peering, 57 active+clean, 28 unknown; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:24 smithi193 ceph-mon[37473]: osdmap e80: 12 total, 12 up, 12 in 2024-07-26T11:10:24.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:24 smithi193 ceph-mon[37473]: pgmap v303: 97 pgs: 12 creating+peering, 57 active+clean, 28 unknown; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:25.248 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.246+0000 7f6517200640 1 -- 172.21.15.134:0/2008312427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f651810b860 msgr2=0x7f6518112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:25.249 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.246+0000 7f6517200640 1 --2- 172.21.15.134:0/2008312427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f651810b860 0x7f6518112090 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6510009ca0 tx=0x7f651002a8c0 comp rx=0 tx=0).stop 2024-07-26T11:10:25.249 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.246+0000 7f6517200640 1 -- 172.21.15.134:0/2008312427 shutdown_connections 2024-07-26T11:10:25.249 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.246+0000 7f6517200640 1 --2- 172.21.15.134:0/2008312427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f651810af10 0x7f651810b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:25.249 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.246+0000 7f6517200640 1 --2- 172.21.15.134:0/2008312427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f651810b860 0x7f6518112090 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:25.249 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.246+0000 7f6517200640 1 --2- 172.21.15.134:0/2008312427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65181055d0 0x7f65181059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:25.249 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.246+0000 7f6517200640 1 -- 172.21.15.134:0/2008312427 >> 172.21.15.134:0/2008312427 conn(0x7f6518101040 msgr2=0x7f6518103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:25.249 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.246+0000 7f6517200640 1 -- 172.21.15.134:0/2008312427 shutdown_connections 2024-07-26T11:10:25.249 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.246+0000 7f6517200640 1 -- 172.21.15.134:0/2008312427 wait complete. 2024-07-26T11:10:25.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.248+0000 7f6517200640 1 Processor -- start 2024-07-26T11:10:25.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.248+0000 7f6517200640 1 -- start start 2024-07-26T11:10:25.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.248+0000 7f6517200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65181055d0 0x7f651819bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:25.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.248+0000 7f6517200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f651810af10 0x7f651819c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:25.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.248+0000 7f6517200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f651810b860 0x7f65181a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:25.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.248+0000 7f6517200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6518115580 con 0x7f651810af10 2024-07-26T11:10:25.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.248+0000 7f6517200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6518115400 con 0x7f651810b860 2024-07-26T11:10:25.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.248+0000 7f6517200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6518115700 con 0x7f65181055d0 2024-07-26T11:10:25.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.248+0000 7f650fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f651810af10 0x7f651819c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:25.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.248+0000 7f650fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f651810af10 0x7f651819c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57278/0 (socket says 172.21.15.134:57278) 2024-07-26T11:10:25.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.248+0000 7f650fe00640 1 -- 172.21.15.134:0/2491007179 learned_addr learned my addr 172.21.15.134:0/2491007179 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:10:25.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.248+0000 7f6515e00640 1 --2- 172.21.15.134:0/2491007179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65181055d0 0x7f651819bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:25.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.249+0000 7f6516800640 1 --2- 172.21.15.134:0/2491007179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f651810b860 0x7f65181a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:25.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.249+0000 7f650fe00640 1 -- 172.21.15.134:0/2491007179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65181055d0 msgr2=0x7f651819bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:25.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.249+0000 7f650fe00640 1 --2- 172.21.15.134:0/2491007179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65181055d0 0x7f651819bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:25.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.249+0000 7f650fe00640 1 -- 172.21.15.134:0/2491007179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f651810b860 msgr2=0x7f65181a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:25.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.249+0000 7f650fe00640 1 --2- 172.21.15.134:0/2491007179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f651810b860 0x7f65181a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:25.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.249+0000 7f650fe00640 1 -- 172.21.15.134:0/2491007179 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65181a1ee0 con 0x7f651810af10 2024-07-26T11:10:25.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.249+0000 7f6516800640 1 --2- 172.21.15.134:0/2491007179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f651810b860 0x7f65181a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:10:25.253 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.249+0000 7f650fe00640 1 --2- 172.21.15.134:0/2491007179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f651810af10 0x7f651819c510 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f650000e9c0 tx=0x7f650000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:25.253 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.250+0000 7f650ea00640 1 -- 172.21.15.134:0/2491007179 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650000ccc0 con 0x7f651810af10 2024-07-26T11:10:25.253 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.250+0000 7f650ea00640 1 -- 172.21.15.134:0/2491007179 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65000054d0 con 0x7f651810af10 2024-07-26T11:10:25.253 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.250+0000 7f6517200640 1 -- 172.21.15.134:0/2491007179 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65181a21d0 con 0x7f651810af10 2024-07-26T11:10:25.253 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.250+0000 7f650ea00640 1 -- 172.21.15.134:0/2491007179 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6500010430 con 0x7f651810af10 2024-07-26T11:10:25.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.250+0000 7f6517200640 1 -- 172.21.15.134:0/2491007179 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65181a8930 con 0x7f651810af10 2024-07-26T11:10:25.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.251+0000 7f6517200640 1 -- 172.21.15.134:0/2491007179 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65181059e0 con 0x7f651810af10 2024-07-26T11:10:25.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.254+0000 7f650ea00640 1 -- 172.21.15.134:0/2491007179 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f6500005000 con 0x7f651810af10 2024-07-26T11:10:25.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.255+0000 7f650ea00640 1 --2- 172.21.15.134:0/2491007179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64fc0b3ca0 0x7f64fc0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:25.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.256+0000 7f650ea00640 1 -- 172.21.15.134:0/2491007179 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8094+0+0 (secure 0 0 0) 0x7f65000e4050 con 0x7f651810af10 2024-07-26T11:10:25.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.256+0000 7f6515e00640 1 --2- 172.21.15.134:0/2491007179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64fc0b3ca0 0x7f64fc0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:25.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.257+0000 7f6515e00640 1 --2- 172.21.15.134:0/2491007179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64fc0b3ca0 0x7f64fc0b61b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6508009bd0 tx=0x7f6508005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:25.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.258+0000 7f650ea00640 1 -- 172.21.15.134:0/2491007179 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6500055600 con 0x7f651810af10 2024-07-26T11:10:25.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.499+0000 7f6517200640 1 -- 172.21.15.134:0/2491007179 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f651819d0a0 con 0x7f651810af10 2024-07-26T11:10:25.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.554+0000 7f650ea00640 1 -- 172.21.15.134:0/2491007179 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f651819d0a0 con 0x7f651810af10 2024-07-26T11:10:25.557 INFO:teuthology.orchestra.run.smithi134.stderr:new fs with metadata pool 2 and data pool 3 2024-07-26T11:10:25.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.558+0000 7f6517200640 1 -- 172.21.15.134:0/2491007179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64fc0b3ca0 msgr2=0x7f64fc0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:25.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.558+0000 7f6517200640 1 --2- 172.21.15.134:0/2491007179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64fc0b3ca0 0x7f64fc0b61b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6508009bd0 tx=0x7f6508005f90 comp rx=0 tx=0).stop 2024-07-26T11:10:25.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.558+0000 7f6517200640 1 -- 172.21.15.134:0/2491007179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f651810af10 msgr2=0x7f651819c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:25.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.558+0000 7f6517200640 1 --2- 172.21.15.134:0/2491007179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f651810af10 0x7f651819c510 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f650000e9c0 tx=0x7f650000ee80 comp rx=0 tx=0).stop 2024-07-26T11:10:25.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.558+0000 7f6517200640 1 -- 172.21.15.134:0/2491007179 shutdown_connections 2024-07-26T11:10:25.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.558+0000 7f6517200640 1 --2- 172.21.15.134:0/2491007179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64fc0b3ca0 0x7f64fc0b61b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:25.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.558+0000 7f6517200640 1 --2- 172.21.15.134:0/2491007179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f651810b860 0x7f65181a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:25.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.558+0000 7f6517200640 1 --2- 172.21.15.134:0/2491007179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f651810af10 0x7f651819c510 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:25.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.558+0000 7f6517200640 1 --2- 172.21.15.134:0/2491007179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65181055d0 0x7f651819bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:25.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.558+0000 7f6517200640 1 -- 172.21.15.134:0/2491007179 >> 172.21.15.134:0/2491007179 conn(0x7f6518101040 msgr2=0x7f651810a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:25.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.559+0000 7f6517200640 1 -- 172.21.15.134:0/2491007179 shutdown_connections 2024-07-26T11:10:25.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:25.560+0000 7f6517200640 1 -- 172.21.15.134:0/2491007179 wait complete. 2024-07-26T11:10:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:25 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2491007179' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T11:10:25.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:25 smithi134 ceph-mon[31966]: daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T11:10:25.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:25 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2491007179' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T11:10:25.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:25 smithi189 ceph-mon[36790]: daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T11:10:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:25 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2491007179' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-07-26T11:10:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:25 smithi193 ceph-mon[37473]: daemon mds.a assigned to filesystem cephfs as rank 0 2024-07-26T11:10:26.467 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-07-26T11:10:26.467 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-07-26T11:10:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:26 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2491007179' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T11:10:26.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:26 smithi134 ceph-mon[31966]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T11:10:26.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:26 smithi134 ceph-mon[31966]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2024-07-26T11:10:26.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:26 smithi134 ceph-mon[31966]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T11:10:26.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:26 smithi134 ceph-mon[31966]: pgmap v305: 97 pgs: 12 creating+peering, 85 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:26.920 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:10:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:26 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2491007179' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T11:10:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:26 smithi189 ceph-mon[36790]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T11:10:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:26 smithi189 ceph-mon[36790]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2024-07-26T11:10:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:26 smithi189 ceph-mon[36790]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T11:10:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:26 smithi189 ceph-mon[36790]: pgmap v305: 97 pgs: 12 creating+peering, 85 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:26 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2491007179' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-07-26T11:10:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:26 smithi193 ceph-mon[37473]: osdmap e81: 12 total, 12 up, 12 in 2024-07-26T11:10:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:26 smithi193 ceph-mon[37473]: fsmap cephfs:1 {0=a=up:creating} 11 up:standby 2024-07-26T11:10:27.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:26 smithi193 ceph-mon[37473]: daemon mds.a is now active in filesystem cephfs as rank 0 2024-07-26T11:10:27.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:26 smithi193 ceph-mon[37473]: pgmap v305: 97 pgs: 12 creating+peering, 85 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:27.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:27 smithi134 ceph-mon[31966]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T11:10:27.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:27 smithi134 ceph-mon[31966]: Cluster is now healthy 2024-07-26T11:10:27.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:27 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] up:active 2024-07-26T11:10:27.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:27 smithi134 ceph-mon[31966]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-07-26T11:10:27.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:27 smithi134 ceph-mon[31966]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-07-26T11:10:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:27 smithi189 ceph-mon[36790]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T11:10:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:27 smithi189 ceph-mon[36790]: Cluster is now healthy 2024-07-26T11:10:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:27 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] up:active 2024-07-26T11:10:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:27 smithi189 ceph-mon[36790]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-07-26T11:10:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:27 smithi189 ceph-mon[36790]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-07-26T11:10:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:27 smithi193 ceph-mon[37473]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-07-26T11:10:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:27 smithi193 ceph-mon[37473]: Cluster is now healthy 2024-07-26T11:10:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:27 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] up:active 2024-07-26T11:10:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:27 smithi193 ceph-mon[37473]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-07-26T11:10:28.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:27 smithi193 ceph-mon[37473]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-07-26T11:10:28.805 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:28 smithi134 ceph-mon[31966]: pgmap v306: 97 pgs: 97 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s wr, 5 op/s 2024-07-26T11:10:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:28 smithi189 ceph-mon[36790]: pgmap v306: 97 pgs: 97 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s wr, 5 op/s 2024-07-26T11:10:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:28 smithi193 ceph-mon[37473]: pgmap v306: 97 pgs: 97 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 852 B/s wr, 5 op/s 2024-07-26T11:10:29.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.346+0000 7f4073200640 1 -- 172.21.15.134:0/3841468572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40741055d0 msgr2=0x7f40741059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:29.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.346+0000 7f4073200640 1 --2- 172.21.15.134:0/3841468572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40741055d0 0x7f40741059e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4064009860 tx=0x7f406402a340 comp rx=0 tx=0).stop 2024-07-26T11:10:29.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.347+0000 7f4073200640 1 -- 172.21.15.134:0/3841468572 shutdown_connections 2024-07-26T11:10:29.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.347+0000 7f4073200640 1 --2- 172.21.15.134:0/3841468572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407410b860 0x7f4074112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:29.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.347+0000 7f4073200640 1 --2- 172.21.15.134:0/3841468572 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f407410af10 0x7f407410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:29.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.347+0000 7f4073200640 1 --2- 172.21.15.134:0/3841468572 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40741055d0 0x7f40741059e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:29.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.347+0000 7f4073200640 1 -- 172.21.15.134:0/3841468572 >> 172.21.15.134:0/3841468572 conn(0x7f4074101040 msgr2=0x7f4074103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:29.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.347+0000 7f4073200640 1 -- 172.21.15.134:0/3841468572 shutdown_connections 2024-07-26T11:10:29.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.347+0000 7f4073200640 1 -- 172.21.15.134:0/3841468572 wait complete. 2024-07-26T11:10:29.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.348+0000 7f4073200640 1 Processor -- start 2024-07-26T11:10:29.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.348+0000 7f4073200640 1 -- start start 2024-07-26T11:10:29.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.348+0000 7f4073200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40741055d0 0x7f407419bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:29.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f4073200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f407410af10 0x7f407419c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:29.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f4073200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407410b860 0x7f40741a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:29.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f4073200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4074115510 con 0x7f407410af10 2024-07-26T11:10:29.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f4073200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4074115390 con 0x7f407410b860 2024-07-26T11:10:29.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f4073200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4074115690 con 0x7f40741055d0 2024-07-26T11:10:29.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f4071e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40741055d0 0x7f407419bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:29.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f4071e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40741055d0 0x7f407419bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:50424/0 (socket says 172.21.15.134:50424) 2024-07-26T11:10:29.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f4071e00640 1 -- 172.21.15.134:0/2106445950 learned_addr learned my addr 172.21.15.134:0/2106445950 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:10:29.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f406be00640 1 --2- 172.21.15.134:0/2106445950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f407410af10 0x7f407419c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:29.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f4072800640 1 --2- 172.21.15.134:0/2106445950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407410b860 0x7f40741a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:29.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f4071e00640 1 -- 172.21.15.134:0/2106445950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407410b860 msgr2=0x7f40741a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:29.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f4071e00640 1 --2- 172.21.15.134:0/2106445950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407410b860 0x7f40741a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:29.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f4071e00640 1 -- 172.21.15.134:0/2106445950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f407410af10 msgr2=0x7f407419c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:29.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f4071e00640 1 --2- 172.21.15.134:0/2106445950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f407410af10 0x7f407419c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:29.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f4071e00640 1 -- 172.21.15.134:0/2106445950 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40741a1e70 con 0x7f40741055d0 2024-07-26T11:10:29.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f406be00640 1 --2- 172.21.15.134:0/2106445950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f407410af10 0x7f407419c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:10:29.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f4072800640 1 --2- 172.21.15.134:0/2106445950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407410b860 0x7f40741a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:10:29.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.349+0000 7f4071e00640 1 --2- 172.21.15.134:0/2106445950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40741055d0 0x7f407419bf00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f40640303f0 tx=0x7f4064032da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:29.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.350+0000 7f406aa00640 1 -- 172.21.15.134:0/2106445950 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4064040020 con 0x7f40741055d0 2024-07-26T11:10:29.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.350+0000 7f406aa00640 1 -- 172.21.15.134:0/2106445950 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4064030dd0 con 0x7f40741055d0 2024-07-26T11:10:29.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.350+0000 7f406aa00640 1 -- 172.21.15.134:0/2106445950 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4064037040 con 0x7f40741055d0 2024-07-26T11:10:29.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.350+0000 7f4073200640 1 -- 172.21.15.134:0/2106445950 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40741a2100 con 0x7f40741055d0 2024-07-26T11:10:29.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.350+0000 7f4073200640 1 -- 172.21.15.134:0/2106445950 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40741a88c0 con 0x7f40741055d0 2024-07-26T11:10:29.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.351+0000 7f4073200640 1 -- 172.21.15.134:0/2106445950 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4074106850 con 0x7f40741055d0 2024-07-26T11:10:29.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.354+0000 7f406aa00640 1 -- 172.21.15.134:0/2106445950 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f4064043020 con 0x7f40741055d0 2024-07-26T11:10:29.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.355+0000 7f406aa00640 1 --2- 172.21.15.134:0/2106445950 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f40540b3c20 0x7f40540b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:29.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.355+0000 7f406aa00640 1 -- 172.21.15.134:0/2106445950 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8194+0+0 (secure 0 0 0) 0x7f40641041e0 con 0x7f40741055d0 2024-07-26T11:10:29.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.355+0000 7f406be00640 1 --2- 172.21.15.134:0/2106445950 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f40540b3c20 0x7f40540b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:29.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.356+0000 7f406be00640 1 --2- 172.21.15.134:0/2106445950 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f40540b3c20 0x7f40540b6130 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f407410aa40 tx=0x7f405c009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:29.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.357+0000 7f406aa00640 1 -- 172.21.15.134:0/2106445950 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f406404a020 con 0x7f40741055d0 2024-07-26T11:10:29.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.562+0000 7f4073200640 1 -- 172.21.15.134:0/2106445950 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f407410d560 con 0x7f40741055d0 2024-07-26T11:10:29.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.603+0000 7f406aa00640 1 -- 172.21.15.134:0/2106445950 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v82) ==== 157+0+0 (secure 0 0 0) 0x7f40640328c0 con 0x7f40741055d0 2024-07-26T11:10:29.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.606+0000 7f4073200640 1 -- 172.21.15.134:0/2106445950 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f40540b3c20 msgr2=0x7f40540b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:29.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.606+0000 7f4073200640 1 --2- 172.21.15.134:0/2106445950 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f40540b3c20 0x7f40540b6130 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f407410aa40 tx=0x7f405c009260 comp rx=0 tx=0).stop 2024-07-26T11:10:29.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.606+0000 7f4073200640 1 -- 172.21.15.134:0/2106445950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40741055d0 msgr2=0x7f407419bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:29.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.606+0000 7f4073200640 1 --2- 172.21.15.134:0/2106445950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40741055d0 0x7f407419bf00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f40640303f0 tx=0x7f4064032da0 comp rx=0 tx=0).stop 2024-07-26T11:10:29.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.606+0000 7f4073200640 1 -- 172.21.15.134:0/2106445950 shutdown_connections 2024-07-26T11:10:29.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.606+0000 7f4073200640 1 --2- 172.21.15.134:0/2106445950 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f40540b3c20 0x7f40540b6130 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:29.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.606+0000 7f4073200640 1 --2- 172.21.15.134:0/2106445950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f407410b860 0x7f40741a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:29.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.606+0000 7f4073200640 1 --2- 172.21.15.134:0/2106445950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f407410af10 0x7f407419c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:29.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.606+0000 7f4073200640 1 --2- 172.21.15.134:0/2106445950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f40741055d0 0x7f407419bf00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:29.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.606+0000 7f4073200640 1 -- 172.21.15.134:0/2106445950 >> 172.21.15.134:0/2106445950 conn(0x7f4074101040 msgr2=0x7f4074103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:29.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.606+0000 7f4073200640 1 -- 172.21.15.134:0/2106445950 shutdown_connections 2024-07-26T11:10:29.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:29.607+0000 7f4073200640 1 -- 172.21.15.134:0/2106445950 wait complete. 2024-07-26T11:10:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:29 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2106445950' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T11:10:29.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:29 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T11:10:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:29 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2106445950' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T11:10:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:29 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T11:10:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:29 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2106445950' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T11:10:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:29 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-07-26T11:10:30.557 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-07-26T11:10:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:30 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-07-26T11:10:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:30 smithi134 ceph-mon[31966]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T11:10:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:30 smithi134 ceph-mon[31966]: pgmap v308: 97 pgs: 97 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 5 op/s 2024-07-26T11:10:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:30 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-07-26T11:10:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:30 smithi189 ceph-mon[36790]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T11:10:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:30 smithi189 ceph-mon[36790]: pgmap v308: 97 pgs: 97 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 5 op/s 2024-07-26T11:10:31.016 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:10:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:30 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-07-26T11:10:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:30 smithi193 ceph-mon[37473]: osdmap e82: 12 total, 12 up, 12 in 2024-07-26T11:10:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:30 smithi193 ceph-mon[37473]: pgmap v308: 97 pgs: 97 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 5 op/s 2024-07-26T11:10:31.384 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:10:30 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 2024-07-26T11:10:30.938+0000 7fa125800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T11:10:33.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:32 smithi189 ceph-mon[36790]: pgmap v309: 97 pgs: 97 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T11:10:33.206 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:32 smithi134 ceph-mon[31966]: pgmap v309: 97 pgs: 97 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T11:10:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:32 smithi193 ceph-mon[37473]: pgmap v309: 97 pgs: 97 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T11:10:33.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.539+0000 7f6613e00640 1 -- 172.21.15.134:0/815548646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6614104220 msgr2=0x7f6614104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:33.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.539+0000 7f6613e00640 1 --2- 172.21.15.134:0/815548646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6614104220 0x7f6614104630 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6604009860 tx=0x7f660402a340 comp rx=0 tx=0).stop 2024-07-26T11:10:33.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.539+0000 7f6613e00640 1 -- 172.21.15.134:0/815548646 shutdown_connections 2024-07-26T11:10:33.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.539+0000 7f6613e00640 1 --2- 172.21.15.134:0/815548646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f661410d680 0x7f661410fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:33.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.539+0000 7f6613e00640 1 --2- 172.21.15.134:0/815548646 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6614104b70 0x7f661410d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:33.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.539+0000 7f6613e00640 1 --2- 172.21.15.134:0/815548646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6614104220 0x7f6614104630 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:33.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.539+0000 7f6613e00640 1 -- 172.21.15.134:0/815548646 >> 172.21.15.134:0/815548646 conn(0x7f66140fdd60 msgr2=0x7f66141001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:33.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.539+0000 7f6613e00640 1 -- 172.21.15.134:0/815548646 shutdown_connections 2024-07-26T11:10:33.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.539+0000 7f6613e00640 1 -- 172.21.15.134:0/815548646 wait complete. 2024-07-26T11:10:33.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.541+0000 7f6613e00640 1 Processor -- start 2024-07-26T11:10:33.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.541+0000 7f6613e00640 1 -- start start 2024-07-26T11:10:33.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.541+0000 7f6613e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6614104220 0x7f661419bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:33.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.541+0000 7f6613e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6614104b70 0x7f661419c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:33.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.541+0000 7f6613e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f661410d680 0x7f66141a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:33.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.541+0000 7f6613e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6614113310 con 0x7f661410d680 2024-07-26T11:10:33.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.541+0000 7f6613e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6614113190 con 0x7f6614104220 2024-07-26T11:10:33.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.541+0000 7f6613e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6614113490 con 0x7f6614104b70 2024-07-26T11:10:33.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.542+0000 7f6612a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6614104220 0x7f661419bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:33.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.542+0000 7f6612a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6614104220 0x7f661419bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46910/0 (socket says 172.21.15.134:46910) 2024-07-26T11:10:33.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.542+0000 7f6613400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f661410d680 0x7f66141a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:33.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.542+0000 7f6612a00640 1 -- 172.21.15.134:0/1965682961 learned_addr learned my addr 172.21.15.134:0/1965682961 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:10:33.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.542+0000 7f660a000640 1 --2- 172.21.15.134:0/1965682961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6614104b70 0x7f661419c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:33.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.542+0000 7f6612a00640 1 -- 172.21.15.134:0/1965682961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6614104b70 msgr2=0x7f661419c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:33.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.542+0000 7f6612a00640 1 --2- 172.21.15.134:0/1965682961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6614104b70 0x7f661419c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:33.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.542+0000 7f6612a00640 1 -- 172.21.15.134:0/1965682961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f661410d680 msgr2=0x7f66141a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:33.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.542+0000 7f6612a00640 1 --2- 172.21.15.134:0/1965682961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f661410d680 0x7f66141a17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:33.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.542+0000 7f6612a00640 1 -- 172.21.15.134:0/1965682961 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66141a1ef0 con 0x7f6614104220 2024-07-26T11:10:33.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.543+0000 7f6612a00640 1 --2- 172.21.15.134:0/1965682961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6614104220 0x7f661419bf80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f66040303f0 tx=0x7f6604032da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:33.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.543+0000 7f660be00640 1 -- 172.21.15.134:0/1965682961 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604040020 con 0x7f6614104220 2024-07-26T11:10:33.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.543+0000 7f660be00640 1 -- 172.21.15.134:0/1965682961 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6604030dd0 con 0x7f6614104220 2024-07-26T11:10:33.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.543+0000 7f660be00640 1 -- 172.21.15.134:0/1965682961 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6604037040 con 0x7f6614104220 2024-07-26T11:10:33.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.543+0000 7f6613e00640 1 -- 172.21.15.134:0/1965682961 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66141a2180 con 0x7f6614104220 2024-07-26T11:10:33.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.543+0000 7f6613e00640 1 -- 172.21.15.134:0/1965682961 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66141a8940 con 0x7f6614104220 2024-07-26T11:10:33.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.544+0000 7f6613e00640 1 -- 172.21.15.134:0/1965682961 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f661406af90 con 0x7f6614104220 2024-07-26T11:10:33.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.547+0000 7f660be00640 1 -- 172.21.15.134:0/1965682961 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f6604043020 con 0x7f6614104220 2024-07-26T11:10:33.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.549+0000 7f660be00640 1 --2- 172.21.15.134:0/1965682961 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65f80b3c10 0x7f65f80b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:33.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.549+0000 7f660be00640 1 -- 172.21.15.134:0/1965682961 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8492+0+0 (secure 0 0 0) 0x7f6604104190 con 0x7f6614104220 2024-07-26T11:10:33.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.549+0000 7f660a000640 1 --2- 172.21.15.134:0/1965682961 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65f80b3c10 0x7f65f80b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:33.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.550+0000 7f660a000640 1 --2- 172.21.15.134:0/1965682961 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65f80b3c10 0x7f65f80b6120 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6614101540 tx=0x7f65fc009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:33.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.550+0000 7f660be00640 1 -- 172.21.15.134:0/1965682961 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f660404a020 con 0x7f6614104220 2024-07-26T11:10:33.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:33.749+0000 7f6613e00640 1 -- 172.21.15.134:0/1965682961 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f6614104630 con 0x7f6614104220 2024-07-26T11:10:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:33 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1965682961' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T11:10:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:33 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T11:10:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-07-26T11:10:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-07-26T11:10:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-07-26T11:10:34.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-07-26T11:10:34.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-07-26T11:10:34.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-07-26T11:10:34.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-07-26T11:10:34.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2024-07-26T11:10:34.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-07-26T11:10:34.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-07-26T11:10:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:33 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1965682961' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T11:10:34.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:33 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T11:10:34.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-07-26T11:10:34.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-07-26T11:10:34.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-07-26T11:10:34.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-07-26T11:10:34.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-07-26T11:10:34.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-07-26T11:10:34.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-07-26T11:10:34.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2024-07-26T11:10:34.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-07-26T11:10:34.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-07-26T11:10:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:33 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1965682961' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T11:10:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:33 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-07-26T11:10:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-07-26T11:10:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-07-26T11:10:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-07-26T11:10:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-07-26T11:10:34.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-07-26T11:10:34.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-07-26T11:10:34.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-07-26T11:10:34.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2024-07-26T11:10:34.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-07-26T11:10:34.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-07-26T11:10:34.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:34.953+0000 7f660be00640 1 -- 172.21.15.134:0/1965682961 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v84) ==== 228+0+0 (secure 0 0 0) 0x7f6614104630 con 0x7f6614104220 2024-07-26T11:10:34.957 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'cephfs_data_ec' created 2024-07-26T11:10:34.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:34.960+0000 7f6613e00640 1 -- 172.21.15.134:0/1965682961 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65f80b3c10 msgr2=0x7f65f80b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:34.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:34.960+0000 7f6613e00640 1 --2- 172.21.15.134:0/1965682961 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65f80b3c10 0x7f65f80b6120 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6614101540 tx=0x7f65fc009260 comp rx=0 tx=0).stop 2024-07-26T11:10:34.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:34.960+0000 7f6613e00640 1 -- 172.21.15.134:0/1965682961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6614104220 msgr2=0x7f661419bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:34.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:34.960+0000 7f6613e00640 1 --2- 172.21.15.134:0/1965682961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6614104220 0x7f661419bf80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f66040303f0 tx=0x7f6604032da0 comp rx=0 tx=0).stop 2024-07-26T11:10:34.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:34.961+0000 7f6613e00640 1 -- 172.21.15.134:0/1965682961 shutdown_connections 2024-07-26T11:10:34.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:34.961+0000 7f6613e00640 1 --2- 172.21.15.134:0/1965682961 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65f80b3c10 0x7f65f80b6120 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:34.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:34.961+0000 7f6613e00640 1 --2- 172.21.15.134:0/1965682961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6614104b70 0x7f661419c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:34.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:34.961+0000 7f6613e00640 1 --2- 172.21.15.134:0/1965682961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f661410d680 0x7f66141a17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:34.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:34.961+0000 7f6613e00640 1 --2- 172.21.15.134:0/1965682961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6614104220 0x7f661419bf80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:34.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:34.961+0000 7f6613e00640 1 -- 172.21.15.134:0/1965682961 >> 172.21.15.134:0/1965682961 conn(0x7f66140fdd60 msgr2=0x7f6614101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:34.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:34.963+0000 7f6613e00640 1 -- 172.21.15.134:0/1965682961 shutdown_connections 2024-07-26T11:10:34.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:34.964+0000 7f6613e00640 1 -- 172.21.15.134:0/1965682961 wait complete. 2024-07-26T11:10:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:34 smithi193 ceph-mon[37473]: pgmap v310: 97 pgs: 97 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T11:10:35.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-07-26T11:10:35.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-07-26T11:10:35.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-07-26T11:10:35.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-07-26T11:10:35.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-07-26T11:10:35.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-07-26T11:10:35.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-07-26T11:10:35.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2024-07-26T11:10:35.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-07-26T11:10:35.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-07-26T11:10:35.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:34 smithi193 ceph-mon[37473]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T11:10:35.418 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:34 smithi134 ceph-mon[31966]: pgmap v310: 97 pgs: 97 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T11:10:35.418 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-07-26T11:10:35.418 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-07-26T11:10:35.418 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-07-26T11:10:35.418 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-07-26T11:10:35.418 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-07-26T11:10:35.418 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-07-26T11:10:35.418 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-07-26T11:10:35.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2024-07-26T11:10:35.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-07-26T11:10:35.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-07-26T11:10:35.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:34 smithi134 ceph-mon[31966]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T11:10:35.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:34 smithi189 ceph-mon[36790]: pgmap v310: 97 pgs: 97 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-07-26T11:10:35.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-07-26T11:10:35.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-07-26T11:10:35.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-07-26T11:10:35.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-07-26T11:10:35.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-07-26T11:10:35.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-07-26T11:10:35.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-07-26T11:10:35.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2024-07-26T11:10:35.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-07-26T11:10:35.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-07-26T11:10:35.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:34 smithi189 ceph-mon[36790]: osdmap e83: 12 total, 12 up, 12 in 2024-07-26T11:10:36.002 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-07-26T11:10:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:35 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T11:10:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:35 smithi134 ceph-mon[31966]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T11:10:36.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:35 smithi134 ceph-mon[31966]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T11:10:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:35 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T11:10:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:35 smithi193 ceph-mon[37473]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T11:10:36.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:35 smithi193 ceph-mon[37473]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T11:10:36.438 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:10:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:35 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-07-26T11:10:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:35 smithi189 ceph-mon[36790]: osdmap e84: 12 total, 12 up, 12 in 2024-07-26T11:10:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:35 smithi189 ceph-mon[36790]: osdmap e85: 12 total, 12 up, 12 in 2024-07-26T11:10:37.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:36 smithi134 ceph-mon[31966]: pgmap v313: 129 pgs: 32 unknown, 1 peering, 96 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-07-26T11:10:37.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:36 smithi134 ceph-mon[31966]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T11:10:37.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:36 smithi134 ceph-mon[31966]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T11:10:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:36 smithi193 ceph-mon[37473]: pgmap v313: 129 pgs: 32 unknown, 1 peering, 96 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-07-26T11:10:37.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:36 smithi193 ceph-mon[37473]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T11:10:37.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:36 smithi193 ceph-mon[37473]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T11:10:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:36 smithi189 ceph-mon[36790]: pgmap v313: 129 pgs: 32 unknown, 1 peering, 96 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2024-07-26T11:10:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:36 smithi189 ceph-mon[36790]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-07-26T11:10:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:36 smithi189 ceph-mon[36790]: osdmap e86: 12 total, 12 up, 12 in 2024-07-26T11:10:38.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.833+0000 7fcc0a000640 1 -- 172.21.15.134:0/1290559186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc0410fc80 msgr2=0x7fcc041164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:38.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.833+0000 7fcc0a000640 1 --2- 172.21.15.134:0/1290559186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc0410fc80 0x7fcc041164b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009be0 tx=0x7fcbfc02a6c0 comp rx=0 tx=0).stop 2024-07-26T11:10:38.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.833+0000 7fcc0a000640 1 -- 172.21.15.134:0/1290559186 shutdown_connections 2024-07-26T11:10:38.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.833+0000 7fcc0a000640 1 --2- 172.21.15.134:0/1290559186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc0410fc80 0x7fcc041164b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:38.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.833+0000 7fcc0a000640 1 --2- 172.21.15.134:0/1290559186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc0410f330 0x7fcc0410f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:38.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.833+0000 7fcc0a000640 1 --2- 172.21.15.134:0/1290559186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc041099f0 0x7fcc04109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:38.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.833+0000 7fcc0a000640 1 -- 172.21.15.134:0/1290559186 >> 172.21.15.134:0/1290559186 conn(0x7fcc04078ff0 msgr2=0x7fcc04107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:38.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.833+0000 7fcc0a000640 1 -- 172.21.15.134:0/1290559186 shutdown_connections 2024-07-26T11:10:38.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.833+0000 7fcc0a000640 1 -- 172.21.15.134:0/1290559186 wait complete. 2024-07-26T11:10:38.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.834+0000 7fcc0a000640 1 Processor -- start 2024-07-26T11:10:38.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.834+0000 7fcc0a000640 1 -- start start 2024-07-26T11:10:38.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.835+0000 7fcc0a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc041099f0 0x7fcc041a0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:38.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.835+0000 7fcc0a000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc0410f330 0x7fcc041a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:38.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.835+0000 7fcc0a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc0410fc80 0x7fcc041a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:38.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.835+0000 7fcc0a000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcc04119940 con 0x7fcc0410f330 2024-07-26T11:10:38.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.835+0000 7fcc0a000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcc041197c0 con 0x7fcc041099f0 2024-07-26T11:10:38.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.835+0000 7fcc0a000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcc04119ac0 con 0x7fcc0410fc80 2024-07-26T11:10:38.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.835+0000 7fcc08c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc041099f0 0x7fcc041a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:38.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.835+0000 7fcc03e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc0410f330 0x7fcc041a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:38.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.835+0000 7fcc08c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc041099f0 0x7fcc041a0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46930/0 (socket says 172.21.15.134:46930) 2024-07-26T11:10:38.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.835+0000 7fcc08c00640 1 -- 172.21.15.134:0/1704424472 learned_addr learned my addr 172.21.15.134:0/1704424472 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:10:38.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.835+0000 7fcc09600640 1 --2- 172.21.15.134:0/1704424472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc0410fc80 0x7fcc041a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:38.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.836+0000 7fcc08c00640 1 -- 172.21.15.134:0/1704424472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc0410fc80 msgr2=0x7fcc041a5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:38.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.836+0000 7fcc08c00640 1 --2- 172.21.15.134:0/1704424472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc0410fc80 0x7fcc041a5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:38.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.836+0000 7fcc08c00640 1 -- 172.21.15.134:0/1704424472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc0410f330 msgr2=0x7fcc041a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:38.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.836+0000 7fcc08c00640 1 --2- 172.21.15.134:0/1704424472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc0410f330 0x7fcc041a0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:38.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.836+0000 7fcc08c00640 1 -- 172.21.15.134:0/1704424472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc041a6200 con 0x7fcc041099f0 2024-07-26T11:10:38.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.836+0000 7fcc03e00640 1 --2- 172.21.15.134:0/1704424472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc0410f330 0x7fcc041a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:10:38.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.836+0000 7fcc08c00640 1 --2- 172.21.15.134:0/1704424472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc041099f0 0x7fcc041a0290 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400e9c0 tx=0x7fcbf400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:38.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.836+0000 7fcc01600640 1 -- 172.21.15.134:0/1704424472 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf400ccc0 con 0x7fcc041099f0 2024-07-26T11:10:38.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.836+0000 7fcc0a000640 1 -- 172.21.15.134:0/1704424472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc041a64f0 con 0x7fcc041099f0 2024-07-26T11:10:38.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.836+0000 7fcc01600640 1 -- 172.21.15.134:0/1704424472 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbf40054d0 con 0x7fcc041099f0 2024-07-26T11:10:38.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.836+0000 7fcc01600640 1 -- 172.21.15.134:0/1704424472 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4010430 con 0x7fcc041099f0 2024-07-26T11:10:38.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.836+0000 7fcc0a000640 1 -- 172.21.15.134:0/1704424472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc041acc50 con 0x7fcc041099f0 2024-07-26T11:10:38.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.838+0000 7fcc0a000640 1 -- 172.21.15.134:0/1704424472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc04109e00 con 0x7fcc041099f0 2024-07-26T11:10:38.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.840+0000 7fcc01600640 1 -- 172.21.15.134:0/1704424472 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fcbf4004f80 con 0x7fcc041099f0 2024-07-26T11:10:38.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.842+0000 7fcc01600640 1 --2- 172.21.15.134:0/1704424472 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcbec0b3c20 0x7fcbec0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:38.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.842+0000 7fcc01600640 1 -- 172.21.15.134:0/1704424472 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9275+0+0 (secure 0 0 0) 0x7fcbf40e47d0 con 0x7fcc041099f0 2024-07-26T11:10:38.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.842+0000 7fcc03e00640 1 --2- 172.21.15.134:0/1704424472 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcbec0b3c20 0x7fcbec0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:38.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.843+0000 7fcc03e00640 1 --2- 172.21.15.134:0/1704424472 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcbec0b3c20 0x7fcbec0b6130 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcbf80095f0 tx=0x7fcbf8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:38.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:38.843+0000 7fcc01600640 1 -- 172.21.15.134:0/1704424472 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcbf40ab9b0 con 0x7fcc041099f0 2024-07-26T11:10:39.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:39.045+0000 7fcc0a000640 1 -- 172.21.15.134:0/1704424472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fcc041a13c0 con 0x7fcc041099f0 2024-07-26T11:10:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:38 smithi193 ceph-mon[37473]: pgmap v316: 129 pgs: 20 unknown, 1 peering, 108 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 548 B/s, 1 objects/s recovering 2024-07-26T11:10:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:38 smithi134 ceph-mon[31966]: pgmap v316: 129 pgs: 20 unknown, 1 peering, 108 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 548 B/s, 1 objects/s recovering 2024-07-26T11:10:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:38 smithi189 ceph-mon[36790]: pgmap v316: 129 pgs: 20 unknown, 1 peering, 108 active+clean; 579 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 548 B/s, 1 objects/s recovering 2024-07-26T11:10:40.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:40.008+0000 7fcc01600640 1 -- 172.21.15.134:0/1704424472 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v87) ==== 169+0+0 (secure 0 0 0) 0x7fcbf40ab2b0 con 0x7fcc041099f0 2024-07-26T11:10:40.010 INFO:teuthology.orchestra.run.smithi134.stderr:set pool 4 allow_ec_overwrites to true 2024-07-26T11:10:40.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:40.013+0000 7fcc0a000640 1 -- 172.21.15.134:0/1704424472 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcbec0b3c20 msgr2=0x7fcbec0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:40.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:40.013+0000 7fcc0a000640 1 --2- 172.21.15.134:0/1704424472 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcbec0b3c20 0x7fcbec0b6130 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcbf80095f0 tx=0x7fcbf8005f90 comp rx=0 tx=0).stop 2024-07-26T11:10:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:40.013+0000 7fcc0a000640 1 -- 172.21.15.134:0/1704424472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc041099f0 msgr2=0x7fcc041a0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:40.013+0000 7fcc0a000640 1 --2- 172.21.15.134:0/1704424472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc041099f0 0x7fcc041a0290 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400e9c0 tx=0x7fcbf400ee80 comp rx=0 tx=0).stop 2024-07-26T11:10:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:40.013+0000 7fcc0a000640 1 -- 172.21.15.134:0/1704424472 shutdown_connections 2024-07-26T11:10:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:40.013+0000 7fcc0a000640 1 --2- 172.21.15.134:0/1704424472 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcbec0b3c20 0x7fcbec0b6130 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:40.013+0000 7fcc0a000640 1 --2- 172.21.15.134:0/1704424472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc0410fc80 0x7fcc041a5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:40.013+0000 7fcc0a000640 1 --2- 172.21.15.134:0/1704424472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc0410f330 0x7fcc041a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:40.013+0000 7fcc0a000640 1 --2- 172.21.15.134:0/1704424472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc041099f0 0x7fcc041a0290 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:40.013+0000 7fcc0a000640 1 -- 172.21.15.134:0/1704424472 >> 172.21.15.134:0/1704424472 conn(0x7fcc04078ff0 msgr2=0x7fcc040795e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:40.013+0000 7fcc0a000640 1 -- 172.21.15.134:0/1704424472 shutdown_connections 2024-07-26T11:10:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:40.013+0000 7fcc0a000640 1 -- 172.21.15.134:0/1704424472 wait complete. 2024-07-26T11:10:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:39 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1704424472' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T11:10:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:39 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T11:10:40.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:39 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1704424472' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T11:10:40.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:39 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T11:10:40.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:39 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1704424472' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T11:10:40.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:39 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-07-26T11:10:40.954 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs add_data_pool cephfs cephfs_data_ec 2024-07-26T11:10:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:41 smithi134 ceph-mon[31966]: pgmap v317: 129 pgs: 11 unknown, 118 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 370 B/s, 1 objects/s recovering 2024-07-26T11:10:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:41 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-07-26T11:10:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:41 smithi134 ceph-mon[31966]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T11:10:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:41 smithi193 ceph-mon[37473]: pgmap v317: 129 pgs: 11 unknown, 118 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 370 B/s, 1 objects/s recovering 2024-07-26T11:10:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:41 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-07-26T11:10:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:41 smithi193 ceph-mon[37473]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T11:10:41.418 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:10:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:41 smithi189 ceph-mon[36790]: pgmap v317: 129 pgs: 11 unknown, 118 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 370 B/s, 1 objects/s recovering 2024-07-26T11:10:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:41 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-07-26T11:10:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:41 smithi189 ceph-mon[36790]: osdmap e87: 12 total, 12 up, 12 in 2024-07-26T11:10:43.115 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:43 smithi134 ceph-mon[31966]: pgmap v319: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 365 B/s, 1 objects/s recovering 2024-07-26T11:10:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:43 smithi193 ceph-mon[37473]: pgmap v319: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 365 B/s, 1 objects/s recovering 2024-07-26T11:10:43.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:43 smithi189 ceph-mon[36790]: pgmap v319: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 365 B/s, 1 objects/s recovering 2024-07-26T11:10:43.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.676+0000 7fa52e400640 1 -- 172.21.15.134:0/667241290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa52810fc10 msgr2=0x7fa528112030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:43.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.676+0000 7fa52e400640 1 --2- 172.21.15.134:0/667241290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa52810fc10 0x7fa528112030 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa520009900 tx=0x7fa52002a480 comp rx=0 tx=0).stop 2024-07-26T11:10:43.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.676+0000 7fa52e400640 1 -- 172.21.15.134:0/667241290 shutdown_connections 2024-07-26T11:10:43.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.676+0000 7fa52e400640 1 --2- 172.21.15.134:0/667241290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa52810fc10 0x7fa528112030 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:43.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.676+0000 7fa52e400640 1 --2- 172.21.15.134:0/667241290 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5281029b0 0x7fa52810f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:43.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.676+0000 7fa52e400640 1 --2- 172.21.15.134:0/667241290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa528102060 0x7fa528102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:43.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.676+0000 7fa52e400640 1 -- 172.21.15.134:0/667241290 >> 172.21.15.134:0/667241290 conn(0x7fa528079160 msgr2=0x7fa528079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:43.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.676+0000 7fa52e400640 1 -- 172.21.15.134:0/667241290 shutdown_connections 2024-07-26T11:10:43.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.676+0000 7fa52e400640 1 -- 172.21.15.134:0/667241290 wait complete. 2024-07-26T11:10:43.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.678+0000 7fa52e400640 1 Processor -- start 2024-07-26T11:10:43.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.678+0000 7fa52e400640 1 -- start start 2024-07-26T11:10:43.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.678+0000 7fa52e400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa528102060 0x7fa52819bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:43.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.678+0000 7fa52e400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5281029b0 0x7fa52819c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:43.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.678+0000 7fa52e400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa52810fc10 0x7fa5281a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:43.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.678+0000 7fa52e400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa5281156b0 con 0x7fa5281029b0 2024-07-26T11:10:43.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.678+0000 7fa52e400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa528115530 con 0x7fa52810fc10 2024-07-26T11:10:43.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.679+0000 7fa52e400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa528115830 con 0x7fa528102060 2024-07-26T11:10:43.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.679+0000 7fa527400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa528102060 0x7fa52819bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:43.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.679+0000 7fa527400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa528102060 0x7fa52819bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:56400/0 (socket says 172.21.15.134:56400) 2024-07-26T11:10:43.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.679+0000 7fa527400640 1 -- 172.21.15.134:0/2387640406 learned_addr learned my addr 172.21.15.134:0/2387640406 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:10:43.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.679+0000 7fa51ea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5281029b0 0x7fa52819c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:43.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.679+0000 7fa527e00640 1 --2- 172.21.15.134:0/2387640406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa52810fc10 0x7fa5281a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:43.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.679+0000 7fa527400640 1 -- 172.21.15.134:0/2387640406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa52810fc10 msgr2=0x7fa5281a1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:43.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.679+0000 7fa527400640 1 --2- 172.21.15.134:0/2387640406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa52810fc10 0x7fa5281a1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:43.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.679+0000 7fa527400640 1 -- 172.21.15.134:0/2387640406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5281029b0 msgr2=0x7fa52819c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:43.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.679+0000 7fa527400640 1 --2- 172.21.15.134:0/2387640406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5281029b0 0x7fa52819c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:43.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.679+0000 7fa527400640 1 -- 172.21.15.134:0/2387640406 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5281a1f50 con 0x7fa528102060 2024-07-26T11:10:43.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.680+0000 7fa527400640 1 --2- 172.21.15.134:0/2387640406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa528102060 0x7fa52819bfe0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa51800eb90 tx=0x7fa51800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:43.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.680+0000 7fa524c00640 1 -- 172.21.15.134:0/2387640406 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518004d20 con 0x7fa528102060 2024-07-26T11:10:43.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.680+0000 7fa524c00640 1 -- 172.21.15.134:0/2387640406 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa51800ee90 con 0x7fa528102060 2024-07-26T11:10:43.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.680+0000 7fa52e400640 1 -- 172.21.15.134:0/2387640406 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5281a2240 con 0x7fa528102060 2024-07-26T11:10:43.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.680+0000 7fa524c00640 1 -- 172.21.15.134:0/2387640406 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518010670 con 0x7fa528102060 2024-07-26T11:10:43.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.680+0000 7fa52e400640 1 -- 172.21.15.134:0/2387640406 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5281a89a0 con 0x7fa528102060 2024-07-26T11:10:43.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.682+0000 7fa52e400640 1 -- 172.21.15.134:0/2387640406 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa528102470 con 0x7fa528102060 2024-07-26T11:10:43.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.684+0000 7fa524c00640 1 -- 172.21.15.134:0/2387640406 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fa518005020 con 0x7fa528102060 2024-07-26T11:10:43.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.685+0000 7fa524c00640 1 --2- 172.21.15.134:0/2387640406 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa50c0b3ca0 0x7fa50c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:43.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.686+0000 7fa524c00640 1 -- 172.21.15.134:0/2387640406 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9275+0+0 (secure 0 0 0) 0x7fa5180e4690 con 0x7fa528102060 2024-07-26T11:10:43.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.686+0000 7fa51ea00640 1 --2- 172.21.15.134:0/2387640406 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa50c0b3ca0 0x7fa50c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:43.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.687+0000 7fa51ea00640 1 --2- 172.21.15.134:0/2387640406 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa50c0b3ca0 0x7fa50c0b61b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa528103920 tx=0x7fa510005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:43.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.689+0000 7fa524c00640 1 -- 172.21.15.134:0/2387640406 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa5180ab870 con 0x7fa528102060 2024-07-26T11:10:43.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:43.970+0000 7fa52e400640 1 -- 172.21.15.134:0/2387640406 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fa5281a8c30 con 0x7fa528102060 2024-07-26T11:10:44.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:44.026+0000 7fa524c00640 1 -- 172.21.15.134:0/2387640406 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7fa5180ab170 con 0x7fa528102060 2024-07-26T11:10:44.029 INFO:teuthology.orchestra.run.smithi134.stderr:added data pool 4 to fsmap 2024-07-26T11:10:44.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:44.030+0000 7fa52e400640 1 -- 172.21.15.134:0/2387640406 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa50c0b3ca0 msgr2=0x7fa50c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:44.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:44.030+0000 7fa52e400640 1 --2- 172.21.15.134:0/2387640406 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa50c0b3ca0 0x7fa50c0b61b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa528103920 tx=0x7fa510005f90 comp rx=0 tx=0).stop 2024-07-26T11:10:44.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:44.030+0000 7fa52e400640 1 -- 172.21.15.134:0/2387640406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa528102060 msgr2=0x7fa52819bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:44.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:44.030+0000 7fa52e400640 1 --2- 172.21.15.134:0/2387640406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa528102060 0x7fa52819bfe0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa51800eb90 tx=0x7fa51800c6a0 comp rx=0 tx=0).stop 2024-07-26T11:10:44.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:44.030+0000 7fa52e400640 1 -- 172.21.15.134:0/2387640406 shutdown_connections 2024-07-26T11:10:44.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:44.030+0000 7fa52e400640 1 --2- 172.21.15.134:0/2387640406 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa50c0b3ca0 0x7fa50c0b61b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:44.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:44.030+0000 7fa52e400640 1 --2- 172.21.15.134:0/2387640406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa52810fc10 0x7fa5281a1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:44.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:44.030+0000 7fa52e400640 1 --2- 172.21.15.134:0/2387640406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5281029b0 0x7fa52819c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:44.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:44.030+0000 7fa52e400640 1 --2- 172.21.15.134:0/2387640406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa528102060 0x7fa52819bfe0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:44.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:44.030+0000 7fa52e400640 1 -- 172.21.15.134:0/2387640406 >> 172.21.15.134:0/2387640406 conn(0x7fa528079160 msgr2=0x7fa528110f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:44.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:44.030+0000 7fa52e400640 1 -- 172.21.15.134:0/2387640406 shutdown_connections 2024-07-26T11:10:44.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:44.031+0000 7fa52e400640 1 -- 172.21.15.134:0/2387640406 wait complete. 2024-07-26T11:10:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:44 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2387640406' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T11:10:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:44 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T11:10:44.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:44 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2387640406' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T11:10:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:44 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T11:10:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:44 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2387640406' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T11:10:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:44 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-07-26T11:10:44.853 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:10:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:45 smithi134 ceph-mon[31966]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 276 B/s, 0 objects/s recovering 2024-07-26T11:10:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:45 smithi134 ceph-mon[31966]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T11:10:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:45 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-07-26T11:10:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:45 smithi134 ceph-mon[31966]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-07-26T11:10:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:45 smithi193 ceph-mon[37473]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 276 B/s, 0 objects/s recovering 2024-07-26T11:10:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:45 smithi193 ceph-mon[37473]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T11:10:45.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:45 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-07-26T11:10:45.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:45 smithi193 ceph-mon[37473]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-07-26T11:10:45.328 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:10:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:45 smithi189 ceph-mon[36790]: pgmap v320: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 276 B/s, 0 objects/s recovering 2024-07-26T11:10:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:45 smithi189 ceph-mon[36790]: osdmap e88: 12 total, 12 up, 12 in 2024-07-26T11:10:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:45 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-07-26T11:10:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:45 smithi189 ceph-mon[36790]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-07-26T11:10:46.416 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:46 smithi134 ceph-mon[31966]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:46.416 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:46 smithi134 ceph-mon[31966]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T11:10:46.416 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:46 smithi134 ceph-mon[31966]: Cluster is now healthy 2024-07-26T11:10:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:46 smithi189 ceph-mon[36790]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:46 smithi189 ceph-mon[36790]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T11:10:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:46 smithi189 ceph-mon[36790]: Cluster is now healthy 2024-07-26T11:10:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:46 smithi193 ceph-mon[37473]: pgmap v322: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:46.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:46 smithi193 ceph-mon[37473]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-07-26T11:10:46.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:46 smithi193 ceph-mon[37473]: Cluster is now healthy 2024-07-26T11:10:47.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.673+0000 7eff9ca00640 1 -- 172.21.15.134:0/15658028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff98077cc0 msgr2=0x7eff98113c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:47.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.673+0000 7eff9ca00640 1 --2- 172.21.15.134:0/15658028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff98077cc0 0x7eff98113c60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7eff80009900 tx=0x7eff8002a410 comp rx=0 tx=0).stop 2024-07-26T11:10:47.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.673+0000 7eff9ca00640 1 -- 172.21.15.134:0/15658028 shutdown_connections 2024-07-26T11:10:47.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.673+0000 7eff9ca00640 1 --2- 172.21.15.134:0/15658028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff981141a0 0x7eff981165c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:47.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.673+0000 7eff9ca00640 1 --2- 172.21.15.134:0/15658028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff98077cc0 0x7eff98113c60 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:47.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.673+0000 7eff9ca00640 1 --2- 172.21.15.134:0/15658028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff98077370 0x7eff98077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:47.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.673+0000 7eff9ca00640 1 -- 172.21.15.134:0/15658028 >> 172.21.15.134:0/15658028 conn(0x7eff981011f0 msgr2=0x7eff98103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:47.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.674+0000 7eff9ca00640 1 -- 172.21.15.134:0/15658028 shutdown_connections 2024-07-26T11:10:47.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.674+0000 7eff9ca00640 1 -- 172.21.15.134:0/15658028 wait complete. 2024-07-26T11:10:47.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.675+0000 7eff9ca00640 1 Processor -- start 2024-07-26T11:10:47.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.675+0000 7eff9ca00640 1 -- start start 2024-07-26T11:10:47.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff9ca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff98077370 0x7eff981a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:47.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff9ca00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff98077cc0 0x7eff981a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:47.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff9ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff981141a0 0x7eff981a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:47.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff9ca00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7eff98119ae0 con 0x7eff98077cc0 2024-07-26T11:10:47.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff9ca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7eff98119960 con 0x7eff981141a0 2024-07-26T11:10:47.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff9ca00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7eff98119c60 con 0x7eff98077370 2024-07-26T11:10:47.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff95e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff98077370 0x7eff981a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:47.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff8fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff98077cc0 0x7eff981a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:47.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff95e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff98077370 0x7eff981a04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:56428/0 (socket says 172.21.15.134:56428) 2024-07-26T11:10:47.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff95e00640 1 -- 172.21.15.134:0/3250972938 learned_addr learned my addr 172.21.15.134:0/3250972938 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:10:47.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff96800640 1 --2- 172.21.15.134:0/3250972938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff981141a0 0x7eff981a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:47.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff95e00640 1 -- 172.21.15.134:0/3250972938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff981141a0 msgr2=0x7eff981a5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:47.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff95e00640 1 --2- 172.21.15.134:0/3250972938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff981141a0 0x7eff981a5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:47.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff95e00640 1 -- 172.21.15.134:0/3250972938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff98077cc0 msgr2=0x7eff981a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:47.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff95e00640 1 --2- 172.21.15.134:0/3250972938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff98077cc0 0x7eff981a0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:47.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff95e00640 1 -- 172.21.15.134:0/3250972938 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff981a6460 con 0x7eff98077370 2024-07-26T11:10:47.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.676+0000 7eff8fe00640 1 --2- 172.21.15.134:0/3250972938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff98077cc0 0x7eff981a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:10:47.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.677+0000 7eff95e00640 1 --2- 172.21.15.134:0/3250972938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff98077370 0x7eff981a04f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7eff8800e9f0 tx=0x7eff8800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:47.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.677+0000 7eff8ea00640 1 -- 172.21.15.134:0/3250972938 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8800cd80 con 0x7eff98077370 2024-07-26T11:10:47.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.677+0000 7eff9ca00640 1 -- 172.21.15.134:0/3250972938 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff981a6750 con 0x7eff98077370 2024-07-26T11:10:47.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.677+0000 7eff8ea00640 1 -- 172.21.15.134:0/3250972938 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7eff880108e0 con 0x7eff98077370 2024-07-26T11:10:47.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.677+0000 7eff8ea00640 1 -- 172.21.15.134:0/3250972938 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff88018980 con 0x7eff98077370 2024-07-26T11:10:47.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.677+0000 7eff9ca00640 1 -- 172.21.15.134:0/3250972938 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff981aceb0 con 0x7eff98077370 2024-07-26T11:10:47.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.678+0000 7eff9ca00640 1 -- 172.21.15.134:0/3250972938 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff98077780 con 0x7eff98077370 2024-07-26T11:10:47.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.681+0000 7eff8ea00640 1 -- 172.21.15.134:0/3250972938 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7eff88010430 con 0x7eff98077370 2024-07-26T11:10:47.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.682+0000 7eff8ea00640 1 --2- 172.21.15.134:0/3250972938 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7eff7c0b3c20 0x7eff7c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:47.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.683+0000 7eff8ea00640 1 -- 172.21.15.134:0/3250972938 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7eff880e4880 con 0x7eff98077370 2024-07-26T11:10:47.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.683+0000 7eff8fe00640 1 --2- 172.21.15.134:0/3250972938 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7eff7c0b3c20 0x7eff7c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:47.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.683+0000 7eff8fe00640 1 --2- 172.21.15.134:0/3250972938 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7eff7c0b3c20 0x7eff7c0b6130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7eff80009230 tx=0x7eff80031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:47.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.684+0000 7eff8ea00640 1 -- 172.21.15.134:0/3250972938 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7eff880ab3f0 con 0x7eff98077370 2024-07-26T11:10:47.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.919+0000 7eff9ca00640 1 -- 172.21.15.134:0/3250972938 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff9802e0c0 con 0x7eff98077370 2024-07-26T11:10:47.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.920+0000 7eff8ea00640 1 -- 172.21.15.134:0/3250972938 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11854 (secure 0 0 0) 0x7eff880af940 con 0x7eff98077370 2024-07-26T11:10:47.923 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:10:47.923 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 17 2024-07-26T11:10:47.923 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":17,"btime":"2024-07-26T11:10:44:009127+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24635,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24637,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24641,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24665,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24682,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24686,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24688,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24703,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24719,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:10:43.973030+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24721,"qdb_cluster":[24721]},"id":1}]} 2024-07-26T11:10:47.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.923+0000 7eff9ca00640 1 -- 172.21.15.134:0/3250972938 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7eff7c0b3c20 msgr2=0x7eff7c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:47.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.923+0000 7eff9ca00640 1 --2- 172.21.15.134:0/3250972938 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7eff7c0b3c20 0x7eff7c0b6130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7eff80009230 tx=0x7eff80031400 comp rx=0 tx=0).stop 2024-07-26T11:10:47.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.923+0000 7eff9ca00640 1 -- 172.21.15.134:0/3250972938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff98077370 msgr2=0x7eff981a04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:47.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.923+0000 7eff9ca00640 1 --2- 172.21.15.134:0/3250972938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff98077370 0x7eff981a04f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7eff8800e9f0 tx=0x7eff8800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:10:47.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.924+0000 7eff9ca00640 1 -- 172.21.15.134:0/3250972938 shutdown_connections 2024-07-26T11:10:47.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.924+0000 7eff9ca00640 1 --2- 172.21.15.134:0/3250972938 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7eff7c0b3c20 0x7eff7c0b6130 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:47.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.924+0000 7eff9ca00640 1 --2- 172.21.15.134:0/3250972938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff981141a0 0x7eff981a5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:47.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.924+0000 7eff9ca00640 1 --2- 172.21.15.134:0/3250972938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7eff98077cc0 0x7eff981a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:47.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.924+0000 7eff9ca00640 1 --2- 172.21.15.134:0/3250972938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff98077370 0x7eff981a04f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:47.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.924+0000 7eff9ca00640 1 -- 172.21.15.134:0/3250972938 >> 172.21.15.134:0/3250972938 conn(0x7eff981011f0 msgr2=0x7eff98103610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:47.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.924+0000 7eff9ca00640 1 -- 172.21.15.134:0/3250972938 shutdown_connections 2024-07-26T11:10:47.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:47.924+0000 7eff9ca00640 1 -- 172.21.15.134:0/3250972938 wait complete. 2024-07-26T11:10:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:47 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3250972938' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:10:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:47 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3250972938' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:10:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:47 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3250972938' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:10:48.786 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T11:10:49.257 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:10:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:48 smithi193 ceph-mon[37473]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:49.405 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:48 smithi134 ceph-mon[31966]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:48 smithi189 ceph-mon[36790]: pgmap v323: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:51.087 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:50 smithi134 ceph-mon[31966]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:50 smithi193 ceph-mon[37473]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:51.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:50 smithi189 ceph-mon[36790]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:51.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.697+0000 7f63fba00640 1 -- 172.21.15.134:0/230919054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63f410ba40 msgr2=0x7f63f4112270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:51.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.697+0000 7f63fba00640 1 --2- 172.21.15.134:0/230919054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63f410ba40 0x7f63f4112270 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f63ec009be0 tx=0x7f63ec02a6c0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.698+0000 7f63fba00640 1 -- 172.21.15.134:0/230919054 shutdown_connections 2024-07-26T11:10:51.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.698+0000 7f63fba00640 1 --2- 172.21.15.134:0/230919054 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63f410ba40 0x7f63f4112270 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.698+0000 7f63fba00640 1 --2- 172.21.15.134:0/230919054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f410b0f0 0x7f63f410b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.698+0000 7f63fba00640 1 --2- 172.21.15.134:0/230919054 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63f41057b0 0x7f63f4105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.698+0000 7f63fba00640 1 -- 172.21.15.134:0/230919054 >> 172.21.15.134:0/230919054 conn(0x7f63f4101240 msgr2=0x7f63f4103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:51.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.698+0000 7f63fba00640 1 -- 172.21.15.134:0/230919054 shutdown_connections 2024-07-26T11:10:51.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.698+0000 7f63fba00640 1 -- 172.21.15.134:0/230919054 wait complete. 2024-07-26T11:10:51.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.699+0000 7f63fba00640 1 Processor -- start 2024-07-26T11:10:51.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.699+0000 7f63fba00640 1 -- start start 2024-07-26T11:10:51.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.700+0000 7f63fba00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f41057b0 0x7f63f419c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.700+0000 7f63fba00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63f410b0f0 0x7f63f419c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.700+0000 7f63fba00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63f410ba40 0x7f63f41a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.700+0000 7f63fba00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f63f4115710 con 0x7f63f410ba40 2024-07-26T11:10:51.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.700+0000 7f63fba00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f63f4115590 con 0x7f63f41057b0 2024-07-26T11:10:51.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.700+0000 7f63fba00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f63f4115890 con 0x7f63f410b0f0 2024-07-26T11:10:51.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.700+0000 7f63f9a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63f410ba40 0x7f63f41a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.700+0000 7f63f9000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f41057b0 0x7f63f419c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.700+0000 7f63f9a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63f410ba40 0x7f63f41a1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:53270/0 (socket says 172.21.15.134:53270) 2024-07-26T11:10:51.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.700+0000 7f63f9a00640 1 -- 172.21.15.134:0/4150322010 learned_addr learned my addr 172.21.15.134:0/4150322010 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:10:51.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.700+0000 7f63f3e00640 1 --2- 172.21.15.134:0/4150322010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63f410b0f0 0x7f63f419c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.701+0000 7f63f9000640 1 -- 172.21.15.134:0/4150322010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63f410b0f0 msgr2=0x7f63f419c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:51.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.701+0000 7f63f9000640 1 --2- 172.21.15.134:0/4150322010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63f410b0f0 0x7f63f419c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.701+0000 7f63f9000640 1 -- 172.21.15.134:0/4150322010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63f410ba40 msgr2=0x7f63f41a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:51.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.701+0000 7f63f9000640 1 --2- 172.21.15.134:0/4150322010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63f410ba40 0x7f63f41a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.701+0000 7f63f9000640 1 -- 172.21.15.134:0/4150322010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63f41a2090 con 0x7f63f41057b0 2024-07-26T11:10:51.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.701+0000 7f63f3e00640 1 --2- 172.21.15.134:0/4150322010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63f410b0f0 0x7f63f419c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:10:51.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.701+0000 7f63f9000640 1 --2- 172.21.15.134:0/4150322010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f41057b0 0x7f63f419c120 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f63e800e9f0 tx=0x7f63e800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:51.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.701+0000 7f63f1600640 1 -- 172.21.15.134:0/4150322010 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e800ccc0 con 0x7f63f41057b0 2024-07-26T11:10:51.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.702+0000 7f63fba00640 1 -- 172.21.15.134:0/4150322010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63f41a2380 con 0x7f63f41057b0 2024-07-26T11:10:51.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.702+0000 7f63f1600640 1 -- 172.21.15.134:0/4150322010 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63e80108e0 con 0x7f63f41057b0 2024-07-26T11:10:51.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.702+0000 7f63f1600640 1 -- 172.21.15.134:0/4150322010 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e8018840 con 0x7f63f41057b0 2024-07-26T11:10:51.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.702+0000 7f63fba00640 1 -- 172.21.15.134:0/4150322010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63f41a8ae0 con 0x7f63f41057b0 2024-07-26T11:10:51.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.703+0000 7f63fba00640 1 -- 172.21.15.134:0/4150322010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63f4105bc0 con 0x7f63f41057b0 2024-07-26T11:10:51.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.706+0000 7f63f1600640 1 -- 172.21.15.134:0/4150322010 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f63e8010430 con 0x7f63f41057b0 2024-07-26T11:10:51.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.708+0000 7f63f1600640 1 --2- 172.21.15.134:0/4150322010 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63dc0b3ca0 0x7f63dc0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:51.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.708+0000 7f63f1600640 1 -- 172.21.15.134:0/4150322010 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f63e80e47a0 con 0x7f63f41057b0 2024-07-26T11:10:51.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.708+0000 7f63f3e00640 1 --2- 172.21.15.134:0/4150322010 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63dc0b3ca0 0x7f63dc0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:51.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.708+0000 7f63f1600640 1 -- 172.21.15.134:0/4150322010 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f63e80ab960 con 0x7f63f41057b0 2024-07-26T11:10:51.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.709+0000 7f63f3e00640 1 --2- 172.21.15.134:0/4150322010 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63dc0b3ca0 0x7f63dc0b61b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f63f4104940 tx=0x7f63e0009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:51.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.912+0000 7f63fba00640 1 -- 172.21.15.134:0/4150322010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f63f402e0c0 con 0x7f63f41057b0 2024-07-26T11:10:51.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.914+0000 7f63f1600640 1 -- 172.21.15.134:0/4150322010 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24999 (secure 0 0 0) 0x7f63e80ab260 con 0x7f63f41057b0 2024-07-26T11:10:51.917 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:10:51.917 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":88,"fsid":"105fbe5c-4b3e-11ef-bcb7-c7b262605968","created":"2024-07-26T11:00:05.955103+0000","modified":"2024-07-26T11:10:43.972727+0000","last_up_change":"2024-07-26T11:09:11.485512+0000","last_in_change":"2024-07-26T11:08:52.341356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T11:05:01.849170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T11:10:16.055725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T11:10:21.216611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T11:10:33.983067+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8abb1365-fc3e-4617-a9ac-76bece8cd7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6803","nonce":4183273954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6805","nonce":4183273954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6809","nonce":4183273954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6807","nonce":4183273954}]},"public_addr":"172.21.15.134:6803/4183273954","cluster_addr":"172.21.15.134:6805/4183273954","heartbeat_back_addr":"172.21.15.134:6809/4183273954","heartbeat_front_addr":"172.21.15.134:6807/4183273954","state":["exists","up"]},{"osd":1,"uuid":"6c5a7838-821e-44e1-97e4-655b9bfa9f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6801","nonce":35148049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6803","nonce":35148049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6807","nonce":35148049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6805","nonce":35148049}]},"public_addr":"172.21.15.189:6801/35148049","cluster_addr":"172.21.15.189:6803/35148049","heartbeat_back_addr":"172.21.15.189:6807/35148049","heartbeat_front_addr":"172.21.15.189:6805/35148049","state":["exists","up"]},{"osd":2,"uuid":"8bc42f7f-39fa-4d3b-bc17-21c68f9231fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6801","nonce":897516976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6803","nonce":897516976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6807","nonce":897516976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6805","nonce":897516976}]},"public_addr":"172.21.15.193:6801/897516976","cluster_addr":"172.21.15.193:6803/897516976","heartbeat_back_addr":"172.21.15.193:6807/897516976","heartbeat_front_addr":"172.21.15.193:6805/897516976","state":["exists","up"]},{"osd":3,"uuid":"0473d055-0b79-434f-9658-28269d2b1fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6811","nonce":3457190917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6813","nonce":3457190917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6817","nonce":3457190917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6815","nonce":3457190917}]},"public_addr":"172.21.15.134:6811/3457190917","cluster_addr":"172.21.15.134:6813/3457190917","heartbeat_back_addr":"172.21.15.134:6817/3457190917","heartbeat_front_addr":"172.21.15.134:6815/3457190917","state":["exists","up"]},{"osd":4,"uuid":"1d70eaae-56cc-4cdc-8f83-b88d2121f0ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6809","nonce":1786881347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6811","nonce":1786881347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6815","nonce":1786881347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6813","nonce":1786881347}]},"public_addr":"172.21.15.189:6809/1786881347","cluster_addr":"172.21.15.189:6811/1786881347","heartbeat_back_addr":"172.21.15.189:6815/1786881347","heartbeat_front_addr":"172.21.15.189:6813/1786881347","state":["exists","up"]},{"osd":5,"uuid":"a9b0b4cf-1608-45ed-b64e-afcf545d97fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6809","nonce":1092388797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6811","nonce":1092388797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6815","nonce":1092388797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6813","nonce":1092388797}]},"public_addr":"172.21.15.193:6809/1092388797","cluster_addr":"172.21.15.193:6811/1092388797","heartbeat_back_addr":"172.21.15.193:6815/1092388797","heartbeat_front_addr":"172.21.15.193:6813/1092388797","state":["exists","up"]},{"osd":6,"uuid":"0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6819","nonce":2471505438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6821","nonce":2471505438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6825","nonce":2471505438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6823","nonce":2471505438}]},"public_addr":"172.21.15.134:6819/2471505438","cluster_addr":"172.21.15.134:6821/2471505438","heartbeat_back_addr":"172.21.15.134:6825/2471505438","heartbeat_front_addr":"172.21.15.134:6823/2471505438","state":["exists","up"]},{"osd":7,"uuid":"394764a7-1976-4223-a3c0-5979d941a838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6817","nonce":1382694911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6819","nonce":1382694911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6823","nonce":1382694911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6821","nonce":1382694911}]},"public_addr":"172.21.15.189:6817/1382694911","cluster_addr":"172.21.15.189:6819/1382694911","heartbeat_back_addr":"172.21.15.189:6823/1382694911","heartbeat_front_addr":"172.21.15.189:6821/1382694911","state":["exists","up"]},{"osd":8,"uuid":"1c218082-0b4c-41c8-92b9-453ae8d15a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6817","nonce":896348138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6819","nonce":896348138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6823","nonce":896348138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6821","nonce":896348138}]},"public_addr":"172.21.15.193:6817/896348138","cluster_addr":"172.21.15.193:6819/896348138","heartbeat_back_addr":"172.21.15.193:6823/896348138","heartbeat_front_addr":"172.21.15.193:6821/896348138","state":["exists","up"]},{"osd":9,"uuid":"48f39082-4fa9-4443-b5b7-b141c10d9e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6827","nonce":511237025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6829","nonce":511237025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6833","nonce":511237025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6831","nonce":511237025}]},"public_addr":"172.21.15.134:6827/511237025","cluster_addr":"172.21.15.134:6829/511237025","heartbeat_back_addr":"172.21.15.134:6833/511237025","heartbeat_front_addr":"172.21.15.134:6831/511237025","state":["exists","up"]},{"osd":10,"uuid":"95157766-1856-42f7-aea1-9cf83b77c8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6825","nonce":431740858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6827","nonce":431740858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6831","nonce":431740858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6829","nonce":431740858}]},"public_addr":"172.21.15.189:6825/431740858","cluster_addr":"172.21.15.189:6827/431740858","heartbeat_back_addr":"172.21.15.189:6831/431740858","heartbeat_front_addr":"172.21.15.189:6829/431740858","state":["exists","up"]},{"osd":11,"uuid":"244fadc7-f079-4736-8e7e-c0ce5ae6f43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6825","nonce":1594590579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6827","nonce":1594590579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6831","nonce":1594590579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6829","nonce":1594590579}]},"public_addr":"172.21.15.193:6825/1594590579","cluster_addr":"172.21.15.193:6827/1594590579","heartbeat_back_addr":"172.21.15.193:6831/1594590579","heartbeat_front_addr":"172.21.15.193:6829/1594590579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:11.712203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:35.845013+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:58.165609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:31.929746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:56.941555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:20.688071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:54.681514+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:20.789384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:44.568246+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:18.644310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:45.365351+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:09:09.663805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3790862248":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6800/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:0/2906927791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3068153391":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3056560791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6801/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/1310359922":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/2318666906":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/2901579347":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3404218719":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3598071145":"2024-07-27T11:02:33.711793+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:10:51.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.918+0000 7f63fba00640 1 -- 172.21.15.134:0/4150322010 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63dc0b3ca0 msgr2=0x7f63dc0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:51.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.918+0000 7f63fba00640 1 --2- 172.21.15.134:0/4150322010 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63dc0b3ca0 0x7f63dc0b61b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f63f4104940 tx=0x7f63e0009260 comp rx=0 tx=0).stop 2024-07-26T11:10:51.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.918+0000 7f63fba00640 1 -- 172.21.15.134:0/4150322010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f41057b0 msgr2=0x7f63f419c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:51.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.918+0000 7f63fba00640 1 --2- 172.21.15.134:0/4150322010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f41057b0 0x7f63f419c120 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f63e800e9f0 tx=0x7f63e800edf0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.919+0000 7f63fba00640 1 -- 172.21.15.134:0/4150322010 shutdown_connections 2024-07-26T11:10:51.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.919+0000 7f63fba00640 1 --2- 172.21.15.134:0/4150322010 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63dc0b3ca0 0x7f63dc0b61b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.919+0000 7f63fba00640 1 --2- 172.21.15.134:0/4150322010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f63f410b0f0 0x7f63f419c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.919+0000 7f63fba00640 1 --2- 172.21.15.134:0/4150322010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f63f410ba40 0x7f63f41a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.919+0000 7f63fba00640 1 --2- 172.21.15.134:0/4150322010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63f41057b0 0x7f63f419c120 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:51.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.919+0000 7f63fba00640 1 -- 172.21.15.134:0/4150322010 >> 172.21.15.134:0/4150322010 conn(0x7f63f4101240 msgr2=0x7f63f410a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:51.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.919+0000 7f63fba00640 1 -- 172.21.15.134:0/4150322010 shutdown_connections 2024-07-26T11:10:51.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:51.919+0000 7f63fba00640 1 -- 172.21.15.134:0/4150322010 wait complete. 2024-07-26T11:10:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:51 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/4150322010' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:10:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:51 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/4150322010' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:10:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:51 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/4150322010' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:10:52.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T11:10:53.238 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:10:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:52 smithi193 ceph-mon[37473]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:53.389 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:52 smithi134 ceph-mon[31966]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:53.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:52 smithi189 ceph-mon[36790]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:55.259 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:54 smithi134 ceph-mon[31966]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:54 smithi193 ceph-mon[37473]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:54 smithi189 ceph-mon[36790]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:55.599 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.596+0000 7f9b8e200640 1 -- 172.21.15.134:0/2119284435 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b8810fa40 msgr2=0x7f9b88111e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:55.599 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.596+0000 7f9b8e200640 1 --2- 172.21.15.134:0/2119284435 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b8810fa40 0x7f9b88111e60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9b80009ca0 tx=0x7f9b8002a950 comp rx=0 tx=0).stop 2024-07-26T11:10:55.599 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.596+0000 7f9b8e200640 1 -- 172.21.15.134:0/2119284435 shutdown_connections 2024-07-26T11:10:55.599 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.596+0000 7f9b8e200640 1 --2- 172.21.15.134:0/2119284435 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b8810fa40 0x7f9b88111e60 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:55.599 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.596+0000 7f9b8e200640 1 --2- 172.21.15.134:0/2119284435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b881027e0 0x7f9b8810f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:55.599 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.596+0000 7f9b8e200640 1 --2- 172.21.15.134:0/2119284435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b88101e90 0x7f9b881022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:55.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.596+0000 7f9b8e200640 1 -- 172.21.15.134:0/2119284435 >> 172.21.15.134:0/2119284435 conn(0x7f9b880fdcf0 msgr2=0x7f9b88100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:55.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.596+0000 7f9b8e200640 1 -- 172.21.15.134:0/2119284435 shutdown_connections 2024-07-26T11:10:55.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.596+0000 7f9b8e200640 1 -- 172.21.15.134:0/2119284435 wait complete. 2024-07-26T11:10:55.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.598+0000 7f9b8e200640 1 Processor -- start 2024-07-26T11:10:55.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.598+0000 7f9b8e200640 1 -- start start 2024-07-26T11:10:55.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.598+0000 7f9b8e200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b88101e90 0x7f9b8819be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:55.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.598+0000 7f9b8e200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b881027e0 0x7f9b8819c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:55.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.598+0000 7f9b8e200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b8810fa40 0x7f9b881a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:55.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.598+0000 7f9b8e200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9b88115480 con 0x7f9b8810fa40 2024-07-26T11:10:55.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.598+0000 7f9b8e200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9b88115300 con 0x7f9b881027e0 2024-07-26T11:10:55.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.598+0000 7f9b8e200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9b88115600 con 0x7f9b88101e90 2024-07-26T11:10:55.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.599+0000 7f9b87e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b881027e0 0x7f9b8819c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:55.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.599+0000 7f9b8d800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b8810fa40 0x7f9b881a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:55.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.599+0000 7f9b87e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b881027e0 0x7f9b8819c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34466/0 (socket says 172.21.15.134:34466) 2024-07-26T11:10:55.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.599+0000 7f9b87e00640 1 -- 172.21.15.134:0/2285519586 learned_addr learned my addr 172.21.15.134:0/2285519586 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:10:55.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.599+0000 7f9b8ce00640 1 --2- 172.21.15.134:0/2285519586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b88101e90 0x7f9b8819be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:55.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.599+0000 7f9b87e00640 1 -- 172.21.15.134:0/2285519586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b88101e90 msgr2=0x7f9b8819be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:55.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.599+0000 7f9b87e00640 1 --2- 172.21.15.134:0/2285519586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b88101e90 0x7f9b8819be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:55.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.599+0000 7f9b87e00640 1 -- 172.21.15.134:0/2285519586 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b8810fa40 msgr2=0x7f9b881a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:55.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.599+0000 7f9b87e00640 1 --2- 172.21.15.134:0/2285519586 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b8810fa40 0x7f9b881a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:55.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.599+0000 7f9b87e00640 1 -- 172.21.15.134:0/2285519586 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b881a1da0 con 0x7f9b881027e0 2024-07-26T11:10:55.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.599+0000 7f9b87e00640 1 --2- 172.21.15.134:0/2285519586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b881027e0 0x7f9b8819c3d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9b7400e9f0 tx=0x7f9b7400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:55.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.600+0000 7f9b85600640 1 -- 172.21.15.134:0/2285519586 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7400cd80 con 0x7f9b881027e0 2024-07-26T11:10:55.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.600+0000 7f9b85600640 1 -- 172.21.15.134:0/2285519586 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b740108e0 con 0x7f9b881027e0 2024-07-26T11:10:55.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.600+0000 7f9b8e200640 1 -- 172.21.15.134:0/2285519586 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b881a2090 con 0x7f9b881027e0 2024-07-26T11:10:55.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.600+0000 7f9b85600640 1 -- 172.21.15.134:0/2285519586 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b74018980 con 0x7f9b881027e0 2024-07-26T11:10:55.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.600+0000 7f9b8e200640 1 -- 172.21.15.134:0/2285519586 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b881a87f0 con 0x7f9b881027e0 2024-07-26T11:10:55.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.601+0000 7f9b8e200640 1 -- 172.21.15.134:0/2285519586 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b881022a0 con 0x7f9b881027e0 2024-07-26T11:10:55.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.603+0000 7f9b85600640 1 -- 172.21.15.134:0/2285519586 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f9b74010430 con 0x7f9b881027e0 2024-07-26T11:10:55.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.606+0000 7f9b85600640 1 --2- 172.21.15.134:0/2285519586 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b700b3ca0 0x7f9b700b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:55.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.606+0000 7f9b85600640 1 -- 172.21.15.134:0/2285519586 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f9b740e48a0 con 0x7f9b881027e0 2024-07-26T11:10:55.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.606+0000 7f9b8ce00640 1 --2- 172.21.15.134:0/2285519586 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b700b3ca0 0x7f9b700b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:55.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.607+0000 7f9b85600640 1 -- 172.21.15.134:0/2285519586 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9b740ab410 con 0x7f9b881027e0 2024-07-26T11:10:55.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.607+0000 7f9b8ce00640 1 --2- 172.21.15.134:0/2285519586 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b700b3ca0 0x7f9b700b61b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c009bd0 tx=0x7f9b7c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:55.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.810+0000 7f9b8e200640 1 -- 172.21.15.134:0/2285519586 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9b8802e0c0 con 0x7f9b881027e0 2024-07-26T11:10:55.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.812+0000 7f9b85600640 1 -- 172.21.15.134:0/2285519586 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24999 (secure 0 0 0) 0x7f9b740af960 con 0x7f9b881027e0 2024-07-26T11:10:55.814 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:10:55.815 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":88,"fsid":"105fbe5c-4b3e-11ef-bcb7-c7b262605968","created":"2024-07-26T11:00:05.955103+0000","modified":"2024-07-26T11:10:43.972727+0000","last_up_change":"2024-07-26T11:09:11.485512+0000","last_in_change":"2024-07-26T11:08:52.341356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T11:05:01.849170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T11:10:16.055725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T11:10:21.216611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T11:10:33.983067+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8abb1365-fc3e-4617-a9ac-76bece8cd7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6803","nonce":4183273954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6805","nonce":4183273954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6809","nonce":4183273954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6807","nonce":4183273954}]},"public_addr":"172.21.15.134:6803/4183273954","cluster_addr":"172.21.15.134:6805/4183273954","heartbeat_back_addr":"172.21.15.134:6809/4183273954","heartbeat_front_addr":"172.21.15.134:6807/4183273954","state":["exists","up"]},{"osd":1,"uuid":"6c5a7838-821e-44e1-97e4-655b9bfa9f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6801","nonce":35148049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6803","nonce":35148049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6807","nonce":35148049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6805","nonce":35148049}]},"public_addr":"172.21.15.189:6801/35148049","cluster_addr":"172.21.15.189:6803/35148049","heartbeat_back_addr":"172.21.15.189:6807/35148049","heartbeat_front_addr":"172.21.15.189:6805/35148049","state":["exists","up"]},{"osd":2,"uuid":"8bc42f7f-39fa-4d3b-bc17-21c68f9231fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6801","nonce":897516976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6803","nonce":897516976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6807","nonce":897516976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6805","nonce":897516976}]},"public_addr":"172.21.15.193:6801/897516976","cluster_addr":"172.21.15.193:6803/897516976","heartbeat_back_addr":"172.21.15.193:6807/897516976","heartbeat_front_addr":"172.21.15.193:6805/897516976","state":["exists","up"]},{"osd":3,"uuid":"0473d055-0b79-434f-9658-28269d2b1fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6811","nonce":3457190917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6813","nonce":3457190917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6817","nonce":3457190917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6815","nonce":3457190917}]},"public_addr":"172.21.15.134:6811/3457190917","cluster_addr":"172.21.15.134:6813/3457190917","heartbeat_back_addr":"172.21.15.134:6817/3457190917","heartbeat_front_addr":"172.21.15.134:6815/3457190917","state":["exists","up"]},{"osd":4,"uuid":"1d70eaae-56cc-4cdc-8f83-b88d2121f0ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6809","nonce":1786881347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6811","nonce":1786881347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6815","nonce":1786881347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6813","nonce":1786881347}]},"public_addr":"172.21.15.189:6809/1786881347","cluster_addr":"172.21.15.189:6811/1786881347","heartbeat_back_addr":"172.21.15.189:6815/1786881347","heartbeat_front_addr":"172.21.15.189:6813/1786881347","state":["exists","up"]},{"osd":5,"uuid":"a9b0b4cf-1608-45ed-b64e-afcf545d97fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6809","nonce":1092388797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6811","nonce":1092388797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6815","nonce":1092388797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6813","nonce":1092388797}]},"public_addr":"172.21.15.193:6809/1092388797","cluster_addr":"172.21.15.193:6811/1092388797","heartbeat_back_addr":"172.21.15.193:6815/1092388797","heartbeat_front_addr":"172.21.15.193:6813/1092388797","state":["exists","up"]},{"osd":6,"uuid":"0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6819","nonce":2471505438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6821","nonce":2471505438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6825","nonce":2471505438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6823","nonce":2471505438}]},"public_addr":"172.21.15.134:6819/2471505438","cluster_addr":"172.21.15.134:6821/2471505438","heartbeat_back_addr":"172.21.15.134:6825/2471505438","heartbeat_front_addr":"172.21.15.134:6823/2471505438","state":["exists","up"]},{"osd":7,"uuid":"394764a7-1976-4223-a3c0-5979d941a838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6817","nonce":1382694911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6819","nonce":1382694911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6823","nonce":1382694911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6821","nonce":1382694911}]},"public_addr":"172.21.15.189:6817/1382694911","cluster_addr":"172.21.15.189:6819/1382694911","heartbeat_back_addr":"172.21.15.189:6823/1382694911","heartbeat_front_addr":"172.21.15.189:6821/1382694911","state":["exists","up"]},{"osd":8,"uuid":"1c218082-0b4c-41c8-92b9-453ae8d15a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6817","nonce":896348138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6819","nonce":896348138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6823","nonce":896348138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6821","nonce":896348138}]},"public_addr":"172.21.15.193:6817/896348138","cluster_addr":"172.21.15.193:6819/896348138","heartbeat_back_addr":"172.21.15.193:6823/896348138","heartbeat_front_addr":"172.21.15.193:6821/896348138","state":["exists","up"]},{"osd":9,"uuid":"48f39082-4fa9-4443-b5b7-b141c10d9e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6827","nonce":511237025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6829","nonce":511237025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6833","nonce":511237025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6831","nonce":511237025}]},"public_addr":"172.21.15.134:6827/511237025","cluster_addr":"172.21.15.134:6829/511237025","heartbeat_back_addr":"172.21.15.134:6833/511237025","heartbeat_front_addr":"172.21.15.134:6831/511237025","state":["exists","up"]},{"osd":10,"uuid":"95157766-1856-42f7-aea1-9cf83b77c8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6825","nonce":431740858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6827","nonce":431740858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6831","nonce":431740858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6829","nonce":431740858}]},"public_addr":"172.21.15.189:6825/431740858","cluster_addr":"172.21.15.189:6827/431740858","heartbeat_back_addr":"172.21.15.189:6831/431740858","heartbeat_front_addr":"172.21.15.189:6829/431740858","state":["exists","up"]},{"osd":11,"uuid":"244fadc7-f079-4736-8e7e-c0ce5ae6f43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6825","nonce":1594590579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6827","nonce":1594590579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6831","nonce":1594590579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6829","nonce":1594590579}]},"public_addr":"172.21.15.193:6825/1594590579","cluster_addr":"172.21.15.193:6827/1594590579","heartbeat_back_addr":"172.21.15.193:6831/1594590579","heartbeat_front_addr":"172.21.15.193:6829/1594590579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:11.712203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:35.845013+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:58.165609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:31.929746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:56.941555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:20.688071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:54.681514+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:20.789384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:44.568246+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:18.644310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:45.365351+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:09:09.663805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3790862248":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6800/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:0/2906927791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3068153391":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3056560791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6801/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/1310359922":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/2318666906":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/2901579347":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3404218719":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3598071145":"2024-07-27T11:02:33.711793+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:10:55.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.814+0000 7f9b8e200640 1 -- 172.21.15.134:0/2285519586 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b700b3ca0 msgr2=0x7f9b700b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:55.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.814+0000 7f9b8e200640 1 --2- 172.21.15.134:0/2285519586 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b700b3ca0 0x7f9b700b61b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c009bd0 tx=0x7f9b7c005f90 comp rx=0 tx=0).stop 2024-07-26T11:10:55.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.815+0000 7f9b8e200640 1 -- 172.21.15.134:0/2285519586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b881027e0 msgr2=0x7f9b8819c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:55.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.815+0000 7f9b8e200640 1 --2- 172.21.15.134:0/2285519586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b881027e0 0x7f9b8819c3d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9b7400e9f0 tx=0x7f9b7400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:10:55.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.815+0000 7f9b8e200640 1 -- 172.21.15.134:0/2285519586 shutdown_connections 2024-07-26T11:10:55.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.815+0000 7f9b8e200640 1 --2- 172.21.15.134:0/2285519586 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b700b3ca0 0x7f9b700b61b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:55.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.815+0000 7f9b8e200640 1 --2- 172.21.15.134:0/2285519586 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b8810fa40 0x7f9b881a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:55.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.815+0000 7f9b8e200640 1 --2- 172.21.15.134:0/2285519586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b881027e0 0x7f9b8819c3d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:55.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.815+0000 7f9b8e200640 1 --2- 172.21.15.134:0/2285519586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b88101e90 0x7f9b8819be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:55.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.815+0000 7f9b8e200640 1 -- 172.21.15.134:0/2285519586 >> 172.21.15.134:0/2285519586 conn(0x7f9b880fdcf0 msgr2=0x7f9b88111b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:55.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.815+0000 7f9b8e200640 1 -- 172.21.15.134:0/2285519586 shutdown_connections 2024-07-26T11:10:55.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:55.815+0000 7f9b8e200640 1 -- 172.21.15.134:0/2285519586 wait complete. 2024-07-26T11:10:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:56 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2285519586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:10:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:56 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2285519586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:10:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:56 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2285519586' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:10:56.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:10:57.125 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:10:57.273 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:57 smithi134 ceph-mon[31966]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:57 smithi193 ceph-mon[37473]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:57 smithi189 ceph-mon[36790]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:59.119 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:10:59 smithi134 ceph-mon[31966]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:10:59 smithi193 ceph-mon[37473]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:10:59 smithi189 ceph-mon[36790]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:10:59.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.474+0000 7fcbb4e00640 1 -- 172.21.15.134:0/613995631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbb01099f0 msgr2=0x7fcbb0109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:59.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.474+0000 7fcbb4e00640 1 --2- 172.21.15.134:0/613995631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbb01099f0 0x7fcbb0109e00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009900 tx=0x7fcba002a410 comp rx=0 tx=0).stop 2024-07-26T11:10:59.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.474+0000 7fcbb4e00640 1 -- 172.21.15.134:0/613995631 shutdown_connections 2024-07-26T11:10:59.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.474+0000 7fcbb4e00640 1 --2- 172.21.15.134:0/613995631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcbb010fc80 0x7fcbb01164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:59.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.474+0000 7fcbb4e00640 1 --2- 172.21.15.134:0/613995631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb010f330 0x7fcbb010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:59.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.474+0000 7fcbb4e00640 1 --2- 172.21.15.134:0/613995631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbb01099f0 0x7fcbb0109e00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:59.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.474+0000 7fcbb4e00640 1 -- 172.21.15.134:0/613995631 >> 172.21.15.134:0/613995631 conn(0x7fcbb0078ff0 msgr2=0x7fcbb0107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:59.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.474+0000 7fcbb4e00640 1 -- 172.21.15.134:0/613995631 shutdown_connections 2024-07-26T11:10:59.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.474+0000 7fcbb4e00640 1 -- 172.21.15.134:0/613995631 wait complete. 2024-07-26T11:10:59.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.475+0000 7fcbb4e00640 1 Processor -- start 2024-07-26T11:10:59.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.476+0000 7fcbb4e00640 1 -- start start 2024-07-26T11:10:59.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.476+0000 7fcbb4e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb01099f0 0x7fcbb01a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:59.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.476+0000 7fcbb4e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcbb010f330 0x7fcbb01a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:59.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.476+0000 7fcbb4e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbb010fc80 0x7fcbb01a5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:59.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.476+0000 7fcbb4e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb0119990 con 0x7fcbb010fc80 2024-07-26T11:10:59.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.476+0000 7fcbb4e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb0119810 con 0x7fcbb01099f0 2024-07-26T11:10:59.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.476+0000 7fcbb4e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb0119b10 con 0x7fcbb010f330 2024-07-26T11:10:59.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.476+0000 7fcbaea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcbb010f330 0x7fcbb01a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:59.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.476+0000 7fcbaf400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb01099f0 0x7fcbb01a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:59.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.476+0000 7fcbaf400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb01099f0 0x7fcbb01a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34470/0 (socket says 172.21.15.134:34470) 2024-07-26T11:10:59.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.476+0000 7fcbaea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcbb010f330 0x7fcbb01a0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53338/0 (socket says 172.21.15.134:53338) 2024-07-26T11:10:59.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.476+0000 7fcbaf400640 1 -- 172.21.15.134:0/20665711 learned_addr learned my addr 172.21.15.134:0/20665711 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:10:59.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.476+0000 7fcbafe00640 1 --2- 172.21.15.134:0/20665711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbb010fc80 0x7fcbb01a5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:59.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.477+0000 7fcbaf400640 1 -- 172.21.15.134:0/20665711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcbb010f330 msgr2=0x7fcbb01a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:59.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.477+0000 7fcbaf400640 1 --2- 172.21.15.134:0/20665711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcbb010f330 0x7fcbb01a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:59.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.477+0000 7fcbaf400640 1 -- 172.21.15.134:0/20665711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbb010fc80 msgr2=0x7fcbb01a5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:59.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.477+0000 7fcbaf400640 1 --2- 172.21.15.134:0/20665711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbb010fc80 0x7fcbb01a5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:59.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.477+0000 7fcbaf400640 1 -- 172.21.15.134:0/20665711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb01a6350 con 0x7fcbb01099f0 2024-07-26T11:10:59.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.477+0000 7fcbafe00640 1 --2- 172.21.15.134:0/20665711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbb010fc80 0x7fcbb01a5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:10:59.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.477+0000 7fcbaea00640 1 --2- 172.21.15.134:0/20665711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcbb010f330 0x7fcbb01a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:10:59.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.477+0000 7fcbaf400640 1 --2- 172.21.15.134:0/20665711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb01099f0 0x7fcbb01a03e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcba0005ab0 tx=0x7fcba0003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:59.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.477+0000 7fcba7e00640 1 -- 172.21.15.134:0/20665711 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba003f070 con 0x7fcbb01099f0 2024-07-26T11:10:59.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.478+0000 7fcba7e00640 1 -- 172.21.15.134:0/20665711 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcba0031b60 con 0x7fcbb01099f0 2024-07-26T11:10:59.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.478+0000 7fcbb4e00640 1 -- 172.21.15.134:0/20665711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbb01a65e0 con 0x7fcbb01099f0 2024-07-26T11:10:59.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.478+0000 7fcba7e00640 1 -- 172.21.15.134:0/20665711 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcba0035030 con 0x7fcbb01099f0 2024-07-26T11:10:59.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.478+0000 7fcbb4e00640 1 -- 172.21.15.134:0/20665711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbb01acda0 con 0x7fcbb01099f0 2024-07-26T11:10:59.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.479+0000 7fcbb4e00640 1 -- 172.21.15.134:0/20665711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbb0109e00 con 0x7fcbb01099f0 2024-07-26T11:10:59.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.481+0000 7fcba7e00640 1 -- 172.21.15.134:0/20665711 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fcba0042070 con 0x7fcbb01099f0 2024-07-26T11:10:59.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.483+0000 7fcba7e00640 1 --2- 172.21.15.134:0/20665711 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcb940b3ca0 0x7fcb940b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:10:59.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.483+0000 7fcba7e00640 1 -- 172.21.15.134:0/20665711 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7fcba0104e50 con 0x7fcbb01099f0 2024-07-26T11:10:59.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.483+0000 7fcbaea00640 1 --2- 172.21.15.134:0/20665711 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcb940b3ca0 0x7fcb940b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:10:59.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.484+0000 7fcbaea00640 1 --2- 172.21.15.134:0/20665711 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcb940b3ca0 0x7fcb940b61b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcbb007b2d0 tx=0x7fcb9c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:10:59.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.485+0000 7fcba7e00640 1 -- 172.21.15.134:0/20665711 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcba00cc010 con 0x7fcbb01099f0 2024-07-26T11:10:59.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.721+0000 7fcbb4e00640 1 -- 172.21.15.134:0/20665711 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcbb01a1510 con 0x7fcbb01099f0 2024-07-26T11:10:59.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.722+0000 7fcba7e00640 1 -- 172.21.15.134:0/20665711 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11854 (secure 0 0 0) 0x7fcba00cb910 con 0x7fcbb01099f0 2024-07-26T11:10:59.725 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:10:59.725 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":17,"btime":"2024-07-26T11:10:44:009127+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24635,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24637,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24641,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24665,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24682,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24686,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24688,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24703,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24719,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:10:43.973030+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24721,"qdb_cluster":[24721]},"id":1}]} 2024-07-26T11:10:59.726 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 17 2024-07-26T11:10:59.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.725+0000 7fcbb4e00640 1 -- 172.21.15.134:0/20665711 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcb940b3ca0 msgr2=0x7fcb940b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:59.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.725+0000 7fcbb4e00640 1 --2- 172.21.15.134:0/20665711 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcb940b3ca0 0x7fcb940b61b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcbb007b2d0 tx=0x7fcb9c005f90 comp rx=0 tx=0).stop 2024-07-26T11:10:59.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.725+0000 7fcbb4e00640 1 -- 172.21.15.134:0/20665711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb01099f0 msgr2=0x7fcbb01a03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:10:59.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.725+0000 7fcbb4e00640 1 --2- 172.21.15.134:0/20665711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb01099f0 0x7fcbb01a03e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fcba0005ab0 tx=0x7fcba0003c10 comp rx=0 tx=0).stop 2024-07-26T11:10:59.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.726+0000 7fcbb4e00640 1 -- 172.21.15.134:0/20665711 shutdown_connections 2024-07-26T11:10:59.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.726+0000 7fcbb4e00640 1 --2- 172.21.15.134:0/20665711 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcb940b3ca0 0x7fcb940b61b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:59.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.726+0000 7fcbb4e00640 1 --2- 172.21.15.134:0/20665711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcbb010f330 0x7fcbb01a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:59.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.726+0000 7fcbb4e00640 1 --2- 172.21.15.134:0/20665711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcbb010fc80 0x7fcbb01a5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:59.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.726+0000 7fcbb4e00640 1 --2- 172.21.15.134:0/20665711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbb01099f0 0x7fcbb01a03e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:10:59.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.726+0000 7fcbb4e00640 1 -- 172.21.15.134:0/20665711 >> 172.21.15.134:0/20665711 conn(0x7fcbb0078ff0 msgr2=0x7fcbb0107f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:10:59.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.726+0000 7fcbb4e00640 1 -- 172.21.15.134:0/20665711 shutdown_connections 2024-07-26T11:10:59.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:10:59.726+0000 7fcbb4e00640 1 -- 172.21.15.134:0/20665711 wait complete. 2024-07-26T11:11:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:00 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/20665711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:11:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:00 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/20665711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:11:00.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:00 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/20665711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:11:00.563 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T11:11:01.029 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:11:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:01 smithi134 ceph-mon[31966]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:01 smithi189 ceph-mon[36790]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:01 smithi193 ceph-mon[37473]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:03 smithi134 ceph-mon[31966]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:03.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.397+0000 7f2de5800640 1 -- 172.21.15.134:0/1188133808 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2de010b860 msgr2=0x7f2de0112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:03.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.397+0000 7f2de5800640 1 --2- 172.21.15.134:0/1188133808 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2de010b860 0x7f2de0112090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8009ca0 tx=0x7f2dd802a8c0 comp rx=0 tx=0).stop 2024-07-26T11:11:03.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.397+0000 7f2de5800640 1 -- 172.21.15.134:0/1188133808 shutdown_connections 2024-07-26T11:11:03.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.397+0000 7f2de5800640 1 --2- 172.21.15.134:0/1188133808 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2de010b860 0x7f2de0112090 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:03.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.397+0000 7f2de5800640 1 --2- 172.21.15.134:0/1188133808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2de010af10 0x7f2de010b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:03.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.397+0000 7f2de5800640 1 --2- 172.21.15.134:0/1188133808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2de01055d0 0x7f2de01059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:03.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.397+0000 7f2de5800640 1 -- 172.21.15.134:0/1188133808 >> 172.21.15.134:0/1188133808 conn(0x7f2de0101040 msgr2=0x7f2de0103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:03.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.397+0000 7f2de5800640 1 -- 172.21.15.134:0/1188133808 shutdown_connections 2024-07-26T11:11:03.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.397+0000 7f2de5800640 1 -- 172.21.15.134:0/1188133808 wait complete. 2024-07-26T11:11:03.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.398+0000 7f2de5800640 1 Processor -- start 2024-07-26T11:11:03.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.399+0000 7f2de5800640 1 -- start start 2024-07-26T11:11:03.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.399+0000 7f2de5800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2de01055d0 0x7f2de019bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:03.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.399+0000 7f2de5800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2de010af10 0x7f2de019c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:03.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.399+0000 7f2de5800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2de010b860 0x7f2de01a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:03.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.399+0000 7f2de5800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2de0115560 con 0x7f2de010af10 2024-07-26T11:11:03.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.399+0000 7f2de5800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2de01153e0 con 0x7f2de01055d0 2024-07-26T11:11:03.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.399+0000 7f2de5800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2de01156e0 con 0x7f2de010b860 2024-07-26T11:11:03.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.399+0000 7f2ddf400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2de010af10 0x7f2de019c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:03.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.399+0000 7f2ddfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2de01055d0 0x7f2de019bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:03.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.399+0000 7f2ddfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2de01055d0 0x7f2de019bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:48294/0 (socket says 172.21.15.134:48294) 2024-07-26T11:11:03.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.399+0000 7f2ddf400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2de010af10 0x7f2de019c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49392/0 (socket says 172.21.15.134:49392) 2024-07-26T11:11:03.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.399+0000 7f2ddfe00640 1 -- 172.21.15.134:0/3206178499 learned_addr learned my addr 172.21.15.134:0/3206178499 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:11:03.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.400+0000 7f2de4e00640 1 --2- 172.21.15.134:0/3206178499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2de010b860 0x7f2de01a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:03.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.400+0000 7f2ddfe00640 1 -- 172.21.15.134:0/3206178499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2de010b860 msgr2=0x7f2de01a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:03.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.400+0000 7f2ddfe00640 1 --2- 172.21.15.134:0/3206178499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2de010b860 0x7f2de01a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:03.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.400+0000 7f2ddfe00640 1 -- 172.21.15.134:0/3206178499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2de010af10 msgr2=0x7f2de019c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:03.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.400+0000 7f2ddfe00640 1 --2- 172.21.15.134:0/3206178499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2de010af10 0x7f2de019c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:03.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.400+0000 7f2ddfe00640 1 -- 172.21.15.134:0/3206178499 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de01a1ec0 con 0x7f2de01055d0 2024-07-26T11:11:03.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.400+0000 7f2ddf400640 1 --2- 172.21.15.134:0/3206178499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2de010af10 0x7f2de019c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:11:03.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.400+0000 7f2ddfe00640 1 --2- 172.21.15.134:0/3206178499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2de01055d0 0x7f2de019bf50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000e9f0 tx=0x7f2dd000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:03.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.400+0000 7f2ddcc00640 1 -- 172.21.15.134:0/3206178499 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd000cd80 con 0x7f2de01055d0 2024-07-26T11:11:03.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.401+0000 7f2ddcc00640 1 -- 172.21.15.134:0/3206178499 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dd00108e0 con 0x7f2de01055d0 2024-07-26T11:11:03.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.401+0000 7f2de5800640 1 -- 172.21.15.134:0/3206178499 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2de01a21b0 con 0x7f2de01055d0 2024-07-26T11:11:03.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.401+0000 7f2ddcc00640 1 -- 172.21.15.134:0/3206178499 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0018980 con 0x7f2de01055d0 2024-07-26T11:11:03.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.401+0000 7f2de5800640 1 -- 172.21.15.134:0/3206178499 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2de01a8910 con 0x7f2de01055d0 2024-07-26T11:11:03.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.402+0000 7f2de5800640 1 -- 172.21.15.134:0/3206178499 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2de01059e0 con 0x7f2de01055d0 2024-07-26T11:11:03.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.404+0000 7f2ddcc00640 1 -- 172.21.15.134:0/3206178499 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f2dd0010430 con 0x7f2de01055d0 2024-07-26T11:11:03.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.406+0000 7f2ddcc00640 1 --2- 172.21.15.134:0/3206178499 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2dc40b3c20 0x7f2dc40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:03.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.406+0000 7f2ddcc00640 1 -- 172.21.15.134:0/3206178499 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f2dd00e43f0 con 0x7f2de01055d0 2024-07-26T11:11:03.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.406+0000 7f2ddf400640 1 --2- 172.21.15.134:0/3206178499 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2dc40b3c20 0x7f2dc40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:03.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.406+0000 7f2ddf400640 1 --2- 172.21.15.134:0/3206178499 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2dc40b3c20 0x7f2dc40b6130 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2de0104740 tx=0x7f2dcc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:03.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.410+0000 7f2ddcc00640 1 -- 172.21.15.134:0/3206178499 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2dd00aaf60 con 0x7f2de01055d0 2024-07-26T11:11:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:03 smithi189 ceph-mon[36790]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:03.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:03 smithi193 ceph-mon[37473]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:03.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.606+0000 7f2de5800640 1 -- 172.21.15.134:0/3206178499 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2de010d0c0 con 0x7f2de01055d0 2024-07-26T11:11:03.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.608+0000 7f2ddcc00640 1 -- 172.21.15.134:0/3206178499 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+24999 (secure 0 0 0) 0x7f2dd00bb1c0 con 0x7f2de01055d0 2024-07-26T11:11:03.612 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:11:03.612 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":88,"fsid":"105fbe5c-4b3e-11ef-bcb7-c7b262605968","created":"2024-07-26T11:00:05.955103+0000","modified":"2024-07-26T11:10:43.972727+0000","last_up_change":"2024-07-26T11:09:11.485512+0000","last_in_change":"2024-07-26T11:08:52.341356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T11:05:01.849170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T11:10:16.055725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T11:10:21.216611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T11:10:33.983067+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8abb1365-fc3e-4617-a9ac-76bece8cd7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6803","nonce":4183273954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6805","nonce":4183273954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6809","nonce":4183273954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6807","nonce":4183273954}]},"public_addr":"172.21.15.134:6803/4183273954","cluster_addr":"172.21.15.134:6805/4183273954","heartbeat_back_addr":"172.21.15.134:6809/4183273954","heartbeat_front_addr":"172.21.15.134:6807/4183273954","state":["exists","up"]},{"osd":1,"uuid":"6c5a7838-821e-44e1-97e4-655b9bfa9f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6801","nonce":35148049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6803","nonce":35148049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6807","nonce":35148049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6805","nonce":35148049}]},"public_addr":"172.21.15.189:6801/35148049","cluster_addr":"172.21.15.189:6803/35148049","heartbeat_back_addr":"172.21.15.189:6807/35148049","heartbeat_front_addr":"172.21.15.189:6805/35148049","state":["exists","up"]},{"osd":2,"uuid":"8bc42f7f-39fa-4d3b-bc17-21c68f9231fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6801","nonce":897516976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6803","nonce":897516976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6807","nonce":897516976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6805","nonce":897516976}]},"public_addr":"172.21.15.193:6801/897516976","cluster_addr":"172.21.15.193:6803/897516976","heartbeat_back_addr":"172.21.15.193:6807/897516976","heartbeat_front_addr":"172.21.15.193:6805/897516976","state":["exists","up"]},{"osd":3,"uuid":"0473d055-0b79-434f-9658-28269d2b1fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6811","nonce":3457190917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6813","nonce":3457190917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6817","nonce":3457190917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6815","nonce":3457190917}]},"public_addr":"172.21.15.134:6811/3457190917","cluster_addr":"172.21.15.134:6813/3457190917","heartbeat_back_addr":"172.21.15.134:6817/3457190917","heartbeat_front_addr":"172.21.15.134:6815/3457190917","state":["exists","up"]},{"osd":4,"uuid":"1d70eaae-56cc-4cdc-8f83-b88d2121f0ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6809","nonce":1786881347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6811","nonce":1786881347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6815","nonce":1786881347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6813","nonce":1786881347}]},"public_addr":"172.21.15.189:6809/1786881347","cluster_addr":"172.21.15.189:6811/1786881347","heartbeat_back_addr":"172.21.15.189:6815/1786881347","heartbeat_front_addr":"172.21.15.189:6813/1786881347","state":["exists","up"]},{"osd":5,"uuid":"a9b0b4cf-1608-45ed-b64e-afcf545d97fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6809","nonce":1092388797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6811","nonce":1092388797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6815","nonce":1092388797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6813","nonce":1092388797}]},"public_addr":"172.21.15.193:6809/1092388797","cluster_addr":"172.21.15.193:6811/1092388797","heartbeat_back_addr":"172.21.15.193:6815/1092388797","heartbeat_front_addr":"172.21.15.193:6813/1092388797","state":["exists","up"]},{"osd":6,"uuid":"0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6819","nonce":2471505438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6821","nonce":2471505438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6825","nonce":2471505438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6823","nonce":2471505438}]},"public_addr":"172.21.15.134:6819/2471505438","cluster_addr":"172.21.15.134:6821/2471505438","heartbeat_back_addr":"172.21.15.134:6825/2471505438","heartbeat_front_addr":"172.21.15.134:6823/2471505438","state":["exists","up"]},{"osd":7,"uuid":"394764a7-1976-4223-a3c0-5979d941a838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6817","nonce":1382694911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6819","nonce":1382694911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6823","nonce":1382694911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6821","nonce":1382694911}]},"public_addr":"172.21.15.189:6817/1382694911","cluster_addr":"172.21.15.189:6819/1382694911","heartbeat_back_addr":"172.21.15.189:6823/1382694911","heartbeat_front_addr":"172.21.15.189:6821/1382694911","state":["exists","up"]},{"osd":8,"uuid":"1c218082-0b4c-41c8-92b9-453ae8d15a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6817","nonce":896348138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6819","nonce":896348138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6823","nonce":896348138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6821","nonce":896348138}]},"public_addr":"172.21.15.193:6817/896348138","cluster_addr":"172.21.15.193:6819/896348138","heartbeat_back_addr":"172.21.15.193:6823/896348138","heartbeat_front_addr":"172.21.15.193:6821/896348138","state":["exists","up"]},{"osd":9,"uuid":"48f39082-4fa9-4443-b5b7-b141c10d9e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6827","nonce":511237025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6829","nonce":511237025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6833","nonce":511237025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6831","nonce":511237025}]},"public_addr":"172.21.15.134:6827/511237025","cluster_addr":"172.21.15.134:6829/511237025","heartbeat_back_addr":"172.21.15.134:6833/511237025","heartbeat_front_addr":"172.21.15.134:6831/511237025","state":["exists","up"]},{"osd":10,"uuid":"95157766-1856-42f7-aea1-9cf83b77c8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6825","nonce":431740858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6827","nonce":431740858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6831","nonce":431740858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6829","nonce":431740858}]},"public_addr":"172.21.15.189:6825/431740858","cluster_addr":"172.21.15.189:6827/431740858","heartbeat_back_addr":"172.21.15.189:6831/431740858","heartbeat_front_addr":"172.21.15.189:6829/431740858","state":["exists","up"]},{"osd":11,"uuid":"244fadc7-f079-4736-8e7e-c0ce5ae6f43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6825","nonce":1594590579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6827","nonce":1594590579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6831","nonce":1594590579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6829","nonce":1594590579}]},"public_addr":"172.21.15.193:6825/1594590579","cluster_addr":"172.21.15.193:6827/1594590579","heartbeat_back_addr":"172.21.15.193:6831/1594590579","heartbeat_front_addr":"172.21.15.193:6829/1594590579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:11.712203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:35.845013+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:58.165609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:31.929746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:56.941555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:20.688071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:54.681514+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:20.789384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:44.568246+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:18.644310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:45.365351+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:09:09.663805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3790862248":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6800/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:0/2906927791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3068153391":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3056560791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6801/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/1310359922":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/2318666906":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/2901579347":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3404218719":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3598071145":"2024-07-27T11:02:33.711793+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:11:03.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.611+0000 7f2de5800640 1 -- 172.21.15.134:0/3206178499 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2dc40b3c20 msgr2=0x7f2dc40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:03.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.611+0000 7f2de5800640 1 --2- 172.21.15.134:0/3206178499 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2dc40b3c20 0x7f2dc40b6130 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2de0104740 tx=0x7f2dcc005f90 comp rx=0 tx=0).stop 2024-07-26T11:11:03.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.611+0000 7f2de5800640 1 -- 172.21.15.134:0/3206178499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2de01055d0 msgr2=0x7f2de019bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:03.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.611+0000 7f2de5800640 1 --2- 172.21.15.134:0/3206178499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2de01055d0 0x7f2de019bf50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000e9f0 tx=0x7f2dd000eeb0 comp rx=0 tx=0).stop 2024-07-26T11:11:03.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.611+0000 7f2de5800640 1 -- 172.21.15.134:0/3206178499 shutdown_connections 2024-07-26T11:11:03.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.611+0000 7f2de5800640 1 --2- 172.21.15.134:0/3206178499 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2dc40b3c20 0x7f2dc40b6130 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:03.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.611+0000 7f2de5800640 1 --2- 172.21.15.134:0/3206178499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2de010b860 0x7f2de01a1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:03.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.611+0000 7f2de5800640 1 --2- 172.21.15.134:0/3206178499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2de010af10 0x7f2de019c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:03.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.611+0000 7f2de5800640 1 --2- 172.21.15.134:0/3206178499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2de01055d0 0x7f2de019bf50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:03.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.611+0000 7f2de5800640 1 -- 172.21.15.134:0/3206178499 >> 172.21.15.134:0/3206178499 conn(0x7f2de0101040 msgr2=0x7f2de0109510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:03.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.612+0000 7f2de5800640 1 -- 172.21.15.134:0/3206178499 shutdown_connections 2024-07-26T11:11:03.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:03.612+0000 7f2de5800640 1 -- 172.21.15.134:0/3206178499 wait complete. 2024-07-26T11:11:04.392 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:04 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3206178499' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:11:04.443 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-07-26T11:10:25.500802+0000', 'modified': '2024-07-26T11:10:43.973030+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24721}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24721': {'gid': 24721, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.134:6841/4067549550', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6840', 'nonce': 4067549550}, {'type': 'v1', 'addr': '172.21.15.134:6841', 'nonce': 4067549550}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24721, 'qdb_cluster': [24721]} 2024-07-26T11:11:04.443 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-07-26T11:11:04.444 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.a status 2024-07-26T11:11:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:04 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3206178499' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:11:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:04 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3206178499' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:11:04.892 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:11:05.387 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:05 smithi134 ceph-mon[31966]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:05 smithi189 ceph-mon[36790]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:05 smithi193 ceph-mon[37473]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:06.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:06 smithi134 ceph-mon[31966]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:06 smithi189 ceph-mon[36790]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:06 smithi193 ceph-mon[37473]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:07.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.225+0000 7f4cb1c00640 1 -- 172.21.15.134:0/151498011 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cac1099f0 msgr2=0x7f4cac109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:07.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.225+0000 7f4cb1c00640 1 --2- 172.21.15.134:0/151498011 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cac1099f0 0x7f4cac109e00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c009900 tx=0x7f4c9c02a410 comp rx=0 tx=0).stop 2024-07-26T11:11:07.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.225+0000 7f4cb1c00640 1 -- 172.21.15.134:0/151498011 shutdown_connections 2024-07-26T11:11:07.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.225+0000 7f4cb1c00640 1 --2- 172.21.15.134:0/151498011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cac10fc80 0x7f4cac1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.225+0000 7f4cb1c00640 1 --2- 172.21.15.134:0/151498011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cac10f330 0x7f4cac10f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.225+0000 7f4cb1c00640 1 --2- 172.21.15.134:0/151498011 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cac1099f0 0x7f4cac109e00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.225+0000 7f4cb1c00640 1 -- 172.21.15.134:0/151498011 >> 172.21.15.134:0/151498011 conn(0x7f4cac078ff0 msgr2=0x7f4cac107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:07.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.225+0000 7f4cb1c00640 1 -- 172.21.15.134:0/151498011 shutdown_connections 2024-07-26T11:11:07.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.225+0000 7f4cb1c00640 1 -- 172.21.15.134:0/151498011 wait complete. 2024-07-26T11:11:07.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.226+0000 7f4cb1c00640 1 Processor -- start 2024-07-26T11:11:07.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.227+0000 7f4cb1c00640 1 -- start start 2024-07-26T11:11:07.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.227+0000 7f4cb1c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cac1099f0 0x7f4cac1a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.227+0000 7f4cb1c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cac10f330 0x7f4cac1a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.227+0000 7f4cb1c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cac10fc80 0x7f4cac1a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.227+0000 7f4cb1c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4cac119990 con 0x7f4cac1099f0 2024-07-26T11:11:07.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.227+0000 7f4cb1c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4cac119810 con 0x7f4cac10f330 2024-07-26T11:11:07.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.227+0000 7f4cb1c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4cac119b10 con 0x7f4cac10fc80 2024-07-26T11:11:07.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.227+0000 7f4cabe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cac1099f0 0x7f4cac1a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.227+0000 7f4cab400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cac10f330 0x7f4cac1a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.227+0000 7f4cabe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cac1099f0 0x7f4cac1a0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49418/0 (socket says 172.21.15.134:49418) 2024-07-26T11:11:07.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.227+0000 7f4cab400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cac10f330 0x7f4cac1a0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:48314/0 (socket says 172.21.15.134:48314) 2024-07-26T11:11:07.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.227+0000 7f4cabe00640 1 -- 172.21.15.134:0/2426811657 learned_addr learned my addr 172.21.15.134:0/2426811657 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:11:07.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.228+0000 7f4cb1200640 1 --2- 172.21.15.134:0/2426811657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cac10fc80 0x7f4cac1a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.228+0000 7f4cab400640 1 -- 172.21.15.134:0/2426811657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cac10fc80 msgr2=0x7f4cac1a5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:07.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.228+0000 7f4cab400640 1 --2- 172.21.15.134:0/2426811657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cac10fc80 0x7f4cac1a5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.228+0000 7f4cab400640 1 -- 172.21.15.134:0/2426811657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cac1099f0 msgr2=0x7f4cac1a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:07.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.228+0000 7f4cab400640 1 --2- 172.21.15.134:0/2426811657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cac1099f0 0x7f4cac1a0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.228+0000 7f4cab400640 1 -- 172.21.15.134:0/2426811657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cac1a62d0 con 0x7f4cac10f330 2024-07-26T11:11:07.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.228+0000 7f4cab400640 1 --2- 172.21.15.134:0/2426811657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cac10f330 0x7f4cac1a0900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4cac06c270 tx=0x7f4c9800ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:07.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.228+0000 7f4ca8c00640 1 -- 172.21.15.134:0/2426811657 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c98009e60 con 0x7f4cac10f330 2024-07-26T11:11:07.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.229+0000 7f4ca8c00640 1 -- 172.21.15.134:0/2426811657 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c9800ccc0 con 0x7f4cac10f330 2024-07-26T11:11:07.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.229+0000 7f4cb1c00640 1 -- 172.21.15.134:0/2426811657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cac1a65c0 con 0x7f4cac10f330 2024-07-26T11:11:07.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.229+0000 7f4ca8c00640 1 -- 172.21.15.134:0/2426811657 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c980187a0 con 0x7f4cac10f330 2024-07-26T11:11:07.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.229+0000 7f4cb1c00640 1 -- 172.21.15.134:0/2426811657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cac1acd20 con 0x7f4cac10f330 2024-07-26T11:11:07.234 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.232+0000 7f4cb9dc9740 1 Processor -- start 2024-07-26T11:11:07.234 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.232+0000 7f4cb9dc9740 1 -- start start 2024-07-26T11:11:07.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.233+0000 7f4ca8c00640 1 -- 172.21.15.134:0/2426811657 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f4c98010430 con 0x7f4cac10f330 2024-07-26T11:11:07.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.233+0000 7f4cb9dc9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8553af60 0x560f858e0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.233+0000 7f4cb9dc9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f855120a0 0x560f858e0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.233+0000 7f4cb9dc9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8551a280 0x560f858e1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.233+0000 7f4cb9dc9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560f85856970 con 0x560f855120a0 2024-07-26T11:11:07.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.233+0000 7f4cb9dc9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560f85845e70 con 0x560f8551a280 2024-07-26T11:11:07.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.233+0000 7f4cb9dc9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560f85849eb0 con 0x560f8553af60 2024-07-26T11:11:07.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.233+0000 7f4cb1200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8553af60 0x560f858e0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.233+0000 7f4cabe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f855120a0 0x560f858e0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.233+0000 7f4cb1200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8553af60 0x560f858e0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53806/0 (socket says 172.21.15.134:53806) 2024-07-26T11:11:07.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.233+0000 7f4cb1200640 1 -- 172.21.15.134:0/3086097938 learned_addr learned my addr 172.21.15.134:0/3086097938 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:11:07.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.233+0000 7f4cab400640 1 --2- 172.21.15.134:0/3086097938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8551a280 0x560f858e1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.234+0000 7f4cb1200640 1 -- 172.21.15.134:0/3086097938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8551a280 msgr2=0x560f858e1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:07.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.234+0000 7f4cb1200640 1 --2- 172.21.15.134:0/3086097938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8551a280 0x560f858e1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.234+0000 7f4cb1200640 1 -- 172.21.15.134:0/3086097938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f855120a0 msgr2=0x560f858e0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:07.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.234+0000 7f4cb1200640 1 --2- 172.21.15.134:0/3086097938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f855120a0 0x560f858e0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.234+0000 7f4cb1200640 1 -- 172.21.15.134:0/3086097938 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f857ed110 con 0x560f8553af60 2024-07-26T11:11:07.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.234+0000 7f4cb1200640 1 --2- 172.21.15.134:0/3086097938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8553af60 0x560f858e0780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4011440 tx=0x7f4ca4011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d346c2e24831e7e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:07.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.234+0000 7f4ca1600640 1 -- 172.21.15.134:0/3086097938 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca4003c30 con 0x560f8553af60 2024-07-26T11:11:07.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.234+0000 7f4ca1600640 1 -- 172.21.15.134:0/3086097938 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ca4003dd0 con 0x560f8553af60 2024-07-26T11:11:07.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.234+0000 7f4ca1600640 1 -- 172.21.15.134:0/3086097938 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca400fa80 con 0x560f8553af60 2024-07-26T11:11:07.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.235+0000 7f4ca8c00640 1 --2- 172.21.15.134:0/2426811657 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4c900b3c20 0x7f4c900b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.235+0000 7f4ca8c00640 1 -- 172.21.15.134:0/2426811657 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f4c980e42e0 con 0x7f4cac10f330 2024-07-26T11:11:07.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.235+0000 7f4cabe00640 1 --2- 172.21.15.134:0/2426811657 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4c900b3c20 0x7f4c900b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.235+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/3086097938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8553af60 msgr2=0x560f858e0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:07.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.235+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/3086097938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8553af60 0x560f858e0780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4011440 tx=0x7f4ca4011900 comp rx=0 tx=0).stop 2024-07-26T11:11:07.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.236+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/3086097938 shutdown_connections 2024-07-26T11:11:07.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.236+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/3086097938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8551a280 0x560f858e1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.236+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/3086097938 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f855120a0 0x560f858e0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.236+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/3086097938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8553af60 0x560f858e0780 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.236+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/3086097938 >> 172.21.15.134:0/3086097938 conn(0x560f8552f350 msgr2=0x560f858de670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:07.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.236+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/3086097938 shutdown_connections 2024-07-26T11:11:07.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.236+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/3086097938 wait complete. 2024-07-26T11:11:07.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.236+0000 7f4cabe00640 1 --2- 172.21.15.134:0/2426811657 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4c900b3c20 0x7f4c900b6130 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x560f858df400 tx=0x7f4c9c005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:07.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.236+0000 7f4cb9dc9740 1 -- start start 2024-07-26T11:11:07.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.237+0000 7f4cb9dc9740 1 Processor -- start 2024-07-26T11:11:07.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.237+0000 7f4cb9dc9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f855120a0 0x560f858ebff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.237+0000 7f4cb9dc9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8551a280 0x560f858ec5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.237+0000 7f4cb9dc9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f8553af60 0x560f858f8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.237+0000 7f4cb9dc9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560f856a95f0 con 0x560f8553af60 2024-07-26T11:11:07.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.237+0000 7f4cb9dc9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560f857e2540 con 0x560f855120a0 2024-07-26T11:11:07.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.237+0000 7f4cb9dc9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560f858eef60 con 0x560f8551a280 2024-07-26T11:11:07.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.237+0000 7f4cb9dc9740 10 client.0 _ll_register_callbacks cb 0x560f857f1f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:11:07.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.238+0000 7f4cab400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8551a280 0x560f858ec5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.238+0000 7f4cabe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f855120a0 0x560f858ebff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.238+0000 7f4cb1200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f8553af60 0x560f858f8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.238+0000 7f4cab400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8551a280 0x560f858ec5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53816/0 (socket says 172.21.15.134:53816) 2024-07-26T11:11:07.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.238+0000 7f4cabe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f855120a0 0x560f858ebff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:48342/0 (socket says 172.21.15.134:48342) 2024-07-26T11:11:07.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.238+0000 7f4cab400640 1 -- 172.21.15.134:0/3079143327 learned_addr learned my addr 172.21.15.134:0/3079143327 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:11:07.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.238+0000 7f4cab400640 1 -- 172.21.15.134:0/3079143327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f855120a0 msgr2=0x560f858ebff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:07.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.238+0000 7f4cab400640 1 --2- 172.21.15.134:0/3079143327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f855120a0 0x560f858ebff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.238+0000 7f4cab400640 1 -- 172.21.15.134:0/3079143327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f8553af60 msgr2=0x560f858f8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:07.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.238+0000 7f4cab400640 1 --2- 172.21.15.134:0/3079143327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f8553af60 0x560f858f8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.238+0000 7f4cab400640 1 -- 172.21.15.134:0/3079143327 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f855a6540 con 0x560f8551a280 2024-07-26T11:11:07.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.238+0000 7f4cab400640 1 --2- 172.21.15.134:0/3079143327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8551a280 0x560f858ec5d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4c980e7ac0 tx=0x7f4c98010c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=588f5fb9f76f3b34 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:07.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.238+0000 7f4c96000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:11:07.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.239+0000 7f4c96000640 1 -- 172.21.15.134:0/3079143327 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c980aa6f0 con 0x560f8551a280 2024-07-26T11:11:07.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.239+0000 7f4c96000640 1 -- 172.21.15.134:0/3079143327 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f4c980aa000 con 0x560f8551a280 2024-07-26T11:11:07.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.239+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/3079143327 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ca4003dd0 con 0x560f8551a280 2024-07-26T11:11:07.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.239+0000 7f4c96000640 1 -- 172.21.15.134:0/3079143327 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f4c980aa510 con 0x560f8551a280 2024-07-26T11:11:07.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.239+0000 7f4cb9dc9740 10 client.24829 fetch_fsmap learned FSMap version 17 2024-07-26T11:11:07.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.239+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/3079143327 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x560f858fae20 con 0x560f8551a280 2024-07-26T11:11:07.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.239+0000 7f4c96000640 1 -- 172.21.15.134:0/3079143327 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ca4003dd0 con 0x560f8551a280 2024-07-26T11:11:07.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.239+0000 7f4c96000640 1 -- 172.21.15.134:0/3079143327 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f4c980a9a70 con 0x560f8551a280 2024-07-26T11:11:07.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.240+0000 7f4cb9dc9740 10 client.24829 fetch_fsmap finished waiting for FSMap version 17 2024-07-26T11:11:07.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.240+0000 7f4cb9dc9740 10 client.24829 resolve_mds: resolved name 'a' to daemon mds.a 2024-07-26T11:11:07.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.240+0000 7f4cb9dc9740 20 client.24829 populate_metadata read hostname 'smithi134' 2024-07-26T11:11:07.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.240+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/3079143327 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560f854f98e0 0x560f858fe7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.240+0000 7f4cb9dc9740 4 client.24829 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:11:07.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.240+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/3079143327 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f858588c0 con 0x560f854f98e0 2024-07-26T11:11:07.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.240+0000 7f4cabe00640 1 --2- 172.21.15.134:0/3079143327 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560f854f98e0 0x560f858fe7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.241+0000 7f4c96000640 10 client.24829 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:11:07.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.241+0000 7f4cabe00640 1 --2- 172.21.15.134:0/3079143327 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560f854f98e0 0x560f858fe7f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9719dc60e941bd93 server_cookie=9814d1731d34cbc8 in_seq=0 out_seq=0 2024-07-26T11:11:07.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.241+0000 7f4c96000640 1 -- 172.21.15.134:0/3079143327 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560f858588c0 con 0x560f854f98e0 2024-07-26T11:11:07.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.241+0000 7f4c96000640 10 client.24829 handle_command_reply: tid=0 2024-07-26T11:11:07.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.242+0000 7f4cb9dc9740 1 client.24829 shutdown 2024-07-26T11:11:07.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.242+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/3079143327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8551a280 msgr2=0x560f858ec5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:07.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.242+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/3079143327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8551a280 0x560f858ec5d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4c980e7ac0 tx=0x7f4c98010c30 comp rx=0 tx=0).stop 2024-07-26T11:11:07.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.242+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/3079143327 shutdown_connections 2024-07-26T11:11:07.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.242+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/3079143327 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560f854f98e0 0x560f858fe7f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.242+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/3079143327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8551a280 0x560f858ec5d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.242+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/3079143327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f8553af60 0x560f858f8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.242+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/3079143327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f855120a0 0x560f858ebff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.242+0000 7f4c96000640 0 client.24829 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:11:07.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.242+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/3079143327 >> 172.21.15.134:0/3079143327 conn(0x560f8552f350 msgr2=0x560f858e3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:07.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.242+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/3079143327 shutdown_connections 2024-07-26T11:11:07.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.242+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/3079143327 wait complete. 2024-07-26T11:11:07.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.242+0000 7f4cb9dc9740 20 client.24829 trim_cache size 0 max 16384 2024-07-26T11:11:07.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.262+0000 7f4cb9dc9740 1 Processor -- start 2024-07-26T11:11:07.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.262+0000 7f4cb9dc9740 1 -- start start 2024-07-26T11:11:07.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.262+0000 7f4cb9dc9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f855120a0 0x560f858eb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.262+0000 7f4cb9dc9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8551a280 0x560f858f81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.262+0000 7f4cb9dc9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f8553af60 0x560f858f8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.262+0000 7f4cb9dc9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560f858fcb30 con 0x560f8553af60 2024-07-26T11:11:07.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.262+0000 7f4cb9dc9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560f858fd200 con 0x560f855120a0 2024-07-26T11:11:07.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.262+0000 7f4cb9dc9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560f858fd8d0 con 0x560f8551a280 2024-07-26T11:11:07.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.263+0000 7f4cabe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f855120a0 0x560f858eb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.263+0000 7f4cabe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f855120a0 0x560f858eb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:48354/0 (socket says 172.21.15.134:48354) 2024-07-26T11:11:07.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.263+0000 7f4cabe00640 1 -- 172.21.15.134:0/2467221684 learned_addr learned my addr 172.21.15.134:0/2467221684 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:11:07.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.263+0000 7f4cab400640 1 --2- 172.21.15.134:0/2467221684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8551a280 0x560f858f81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.263+0000 7f4cb1200640 1 --2- 172.21.15.134:0/2467221684 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f8553af60 0x560f858f8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.263+0000 7f4cabe00640 1 -- 172.21.15.134:0/2467221684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8551a280 msgr2=0x560f858f81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:07.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.263+0000 7f4cabe00640 1 --2- 172.21.15.134:0/2467221684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8551a280 0x560f858f81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.263+0000 7f4cabe00640 1 -- 172.21.15.134:0/2467221684 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f8553af60 msgr2=0x560f858f8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:07.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.263+0000 7f4cabe00640 1 --2- 172.21.15.134:0/2467221684 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f8553af60 0x560f858f8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.263+0000 7f4cabe00640 1 -- 172.21.15.134:0/2467221684 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f858d97f0 con 0x560f855120a0 2024-07-26T11:11:07.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.263+0000 7f4cabe00640 1 --2- 172.21.15.134:0/2467221684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f855120a0 0x560f858eb4e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c005ab0 tx=0x7f4c9c040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db52832d9efde6c2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:07.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.264+0000 7f4ca0c00640 1 -- 172.21.15.134:0/2467221684 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c035040 con 0x560f855120a0 2024-07-26T11:11:07.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.264+0000 7f4ca0c00640 1 -- 172.21.15.134:0/2467221684 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c9c003d10 con 0x560f855120a0 2024-07-26T11:11:07.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.264+0000 7f4ca0c00640 1 -- 172.21.15.134:0/2467221684 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c031700 con 0x560f855120a0 2024-07-26T11:11:07.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.264+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/2467221684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f855120a0 msgr2=0x560f858eb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:07.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.264+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/2467221684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f855120a0 0x560f858eb4e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c005ab0 tx=0x7f4c9c040000 comp rx=0 tx=0).stop 2024-07-26T11:11:07.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.265+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/2467221684 shutdown_connections 2024-07-26T11:11:07.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.265+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/2467221684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f8551a280 0x560f858f81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.265+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/2467221684 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f8553af60 0x560f858f8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.265+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/2467221684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f855120a0 0x560f858eb4e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.265+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/2467221684 >> 172.21.15.134:0/2467221684 conn(0x560f8552f350 msgr2=0x560f858ebf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:07.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.265+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/2467221684 shutdown_connections 2024-07-26T11:11:07.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.265+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/2467221684 wait complete. 2024-07-26T11:11:07.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.265+0000 7f4cb9dc9740 1 -- start start 2024-07-26T11:11:07.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.266+0000 7f4cb9dc9740 1 Processor -- start 2024-07-26T11:11:07.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.266+0000 7f4cb9dc9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f855120a0 0x560f858ffe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.266+0000 7f4cb9dc9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f8551a280 0x560f85900360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.266+0000 7f4cb9dc9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8553af60 0x560f85965a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.266+0000 7f4cb9dc9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560f858fe670 con 0x560f8551a280 2024-07-26T11:11:07.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.266+0000 7f4cb9dc9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560f858fdfa0 con 0x560f8553af60 2024-07-26T11:11:07.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.266+0000 7f4cb9dc9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560f858fed40 con 0x560f855120a0 2024-07-26T11:11:07.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.266+0000 7f4cb9dc9740 10 client.0 _ll_register_callbacks cb 0x7f4c980e7ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:11:07.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.266+0000 7f4cabe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f855120a0 0x560f858ffe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.266+0000 7f4cab400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f8551a280 0x560f85900360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.266+0000 7f4cb1200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8553af60 0x560f85965a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.266+0000 7f4cab400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f8551a280 0x560f85900360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49462/0 (socket says 172.21.15.134:49462) 2024-07-26T11:11:07.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.267+0000 7f4cabe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f855120a0 0x560f858ffe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53848/0 (socket says 172.21.15.134:53848) 2024-07-26T11:11:07.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.267+0000 7f4cab400640 1 -- 172.21.15.134:0/330141065 learned_addr learned my addr 172.21.15.134:0/330141065 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:11:07.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.267+0000 7f4cabe00640 1 -- 172.21.15.134:0/330141065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8553af60 msgr2=0x560f85965a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:07.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.267+0000 7f4cabe00640 1 --2- 172.21.15.134:0/330141065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8553af60 0x560f85965a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.267+0000 7f4cabe00640 1 -- 172.21.15.134:0/330141065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f8551a280 msgr2=0x560f85900360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:11:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.267+0000 7f4cabe00640 1 --2- 172.21.15.134:0/330141065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f8551a280 0x560f85900360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.267+0000 7f4cabe00640 1 -- 172.21.15.134:0/330141065 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f85965fc0 con 0x560f855120a0 2024-07-26T11:11:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.267+0000 7f4cabe00640 1 --2- 172.21.15.134:0/330141065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f855120a0 0x560f858ffe20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c005ab0 tx=0x7f4c9c02a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95fdda7a42303c4f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.267+0000 7f4c96000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:11:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.267+0000 7f4c96000640 1 -- 172.21.15.134:0/330141065 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c9c060070 con 0x560f855120a0 2024-07-26T11:11:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.268+0000 7f4c96000640 1 -- 172.21.15.134:0/330141065 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f4c9c054830 con 0x560f855120a0 2024-07-26T11:11:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.268+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/330141065 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4c9c003d10 con 0x560f855120a0 2024-07-26T11:11:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.268+0000 7f4c96000640 1 -- 172.21.15.134:0/330141065 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f4c9c06c050 con 0x560f855120a0 2024-07-26T11:11:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.268+0000 7f4cb9dc9740 10 client.24835 fetch_fsmap learned FSMap version 17 2024-07-26T11:11:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.268+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/330141065 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x560f859670b0 con 0x560f855120a0 2024-07-26T11:11:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.268+0000 7f4c96000640 1 -- 172.21.15.134:0/330141065 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c9c003d10 con 0x560f855120a0 2024-07-26T11:11:07.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.269+0000 7f4c96000640 1 -- 172.21.15.134:0/330141065 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 17) ==== 8350+0+0 (secure 0 0 0) 0x7f4c9c044030 con 0x560f855120a0 2024-07-26T11:11:07.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.269+0000 7f4cb9dc9740 10 client.24835 fetch_fsmap finished waiting for FSMap version 17 2024-07-26T11:11:07.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.269+0000 7f4cb9dc9740 10 client.24835 resolve_mds: resolved name 'a' to daemon mds.a 2024-07-26T11:11:07.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.269+0000 7f4cb9dc9740 20 client.24835 populate_metadata read hostname 'smithi134' 2024-07-26T11:11:07.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.269+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/330141065 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560f854cec90 0x560f8596a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:07.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.269+0000 7f4cb9dc9740 4 client.24835 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:11:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.269+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/330141065 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "status"}) -- 0x560f85854360 con 0x560f854cec90 2024-07-26T11:11:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.269+0000 7f4cab400640 1 --2- 172.21.15.134:0/330141065 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560f854cec90 0x560f8596a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.270+0000 7f4cab400640 1 --2- 172.21.15.134:0/330141065 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560f854cec90 0x560f8596a460 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3b6276348d264db server_cookie=bf02b4ef1961acda in_seq=0 out_seq=0 2024-07-26T11:11:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.270+0000 7f4c96000640 10 client.24835 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:11:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.270+0000 7f4c96000640 1 -- 172.21.15.134:0/330141065 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x560f85854360 con 0x560f854cec90 2024-07-26T11:11:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.270+0000 7f4c96000640 10 client.24835 handle_command_reply: tid=0 2024-07-26T11:11:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.270+0000 7f4cb9dc9740 1 client.24835 shutdown 2024-07-26T11:11:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.270+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/330141065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f855120a0 msgr2=0x560f858ffe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.270+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/330141065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f855120a0 0x560f858ffe20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4c9c005ab0 tx=0x7f4c9c02a410 comp rx=0 tx=0).stop 2024-07-26T11:11:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.270+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/330141065 shutdown_connections 2024-07-26T11:11:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.270+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/330141065 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560f854cec90 0x560f8596a460 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.270+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/330141065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f8553af60 0x560f85965a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.270+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/330141065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560f8551a280 0x560f85900360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.270+0000 7f4cb9dc9740 1 --2- 172.21.15.134:0/330141065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560f855120a0 0x560f858ffe20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.270+0000 7f4c96000640 0 client.24835 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:11:07.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.270+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/330141065 >> 172.21.15.134:0/330141065 conn(0x560f8552f350 msgr2=0x560f858ea6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:07.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.270+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/330141065 shutdown_connections 2024-07-26T11:11:07.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.271+0000 7f4cb9dc9740 1 -- 172.21.15.134:0/330141065 wait complete. 2024-07-26T11:11:07.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.271+0000 7f4cb9dc9740 20 client.24835 trim_cache size 0 max 16384 2024-07-26T11:11:07.276 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:11:07.276 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_fsid": "105fbe5c-4b3e-11ef-bcb7-c7b262605968", 2024-07-26T11:11:07.276 INFO:teuthology.orchestra.run.smithi134.stdout: "whoami": 0, 2024-07-26T11:11:07.276 INFO:teuthology.orchestra.run.smithi134.stdout: "id": 24721, 2024-07-26T11:11:07.276 INFO:teuthology.orchestra.run.smithi134.stdout: "want_state": "up:active", 2024-07-26T11:11:07.276 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "up:active", 2024-07-26T11:11:07.277 INFO:teuthology.orchestra.run.smithi134.stdout: "fs_name": "cephfs", 2024-07-26T11:11:07.277 INFO:teuthology.orchestra.run.smithi134.stdout: "rank_uptime": 41.712284965000002, 2024-07-26T11:11:07.277 INFO:teuthology.orchestra.run.smithi134.stdout: "mdsmap_epoch": 17, 2024-07-26T11:11:07.277 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch": 88, 2024-07-26T11:11:07.277 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch_barrier": 88, 2024-07-26T11:11:07.277 INFO:teuthology.orchestra.run.smithi134.stdout: "uptime": 67.253058440000004 2024-07-26T11:11:07.277 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:11:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.271+0000 7f4cb1c00640 1 -- 172.21.15.134:0/2426811657 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4c900b3c20 msgr2=0x7f4c900b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.271+0000 7f4cb1c00640 1 --2- 172.21.15.134:0/2426811657 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4c900b3c20 0x7f4c900b6130 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x560f858df400 tx=0x7f4c9c005a20 comp rx=0 tx=0).stop 2024-07-26T11:11:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.271+0000 7f4cb1c00640 1 -- 172.21.15.134:0/2426811657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cac10f330 msgr2=0x7f4cac1a0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.271+0000 7f4cb1c00640 1 --2- 172.21.15.134:0/2426811657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cac10f330 0x7f4cac1a0900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4cac06c270 tx=0x7f4c9800ed90 comp rx=0 tx=0).stop 2024-07-26T11:11:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.271+0000 7f4cb1c00640 1 -- 172.21.15.134:0/2426811657 shutdown_connections 2024-07-26T11:11:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.271+0000 7f4cb1c00640 1 --2- 172.21.15.134:0/2426811657 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4c900b3c20 0x7f4c900b6130 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.271+0000 7f4cb1c00640 1 --2- 172.21.15.134:0/2426811657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4cac10fc80 0x7f4cac1a5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.271+0000 7f4cb1c00640 1 --2- 172.21.15.134:0/2426811657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4cac10f330 0x7f4cac1a0900 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.271+0000 7f4cb1c00640 1 --2- 172.21.15.134:0/2426811657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4cac1099f0 0x7f4cac1a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.271+0000 7f4cb1c00640 1 -- 172.21.15.134:0/2426811657 >> 172.21.15.134:0/2426811657 conn(0x7f4cac078ff0 msgr2=0x7f4cac079750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.272+0000 7f4cb1c00640 1 -- 172.21.15.134:0/2426811657 shutdown_connections 2024-07-26T11:11:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:07.272+0000 7f4cb1c00640 1 -- 172.21.15.134:0/2426811657 wait complete. 2024-07-26T11:11:08.164 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-07-26T11:11:08.164 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-07-26T11:11:08.164 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs ls 2024-07-26T11:11:08.615 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:11:09.119 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:08 smithi134 ceph-mon[31966]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:08 smithi193 ceph-mon[37473]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:08 smithi189 ceph-mon[36790]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:10.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.804+0000 7fb1a6e00640 1 -- 172.21.15.134:0/1056701994 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a010f980 msgr2=0x7fb1a0111da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:10.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.804+0000 7fb1a6e00640 1 --2- 172.21.15.134:0/1056701994 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a010f980 0x7fb1a0111da0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb198009be0 tx=0x7fb19802a6c0 comp rx=0 tx=0).stop 2024-07-26T11:11:10.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.805+0000 7fb1a6e00640 1 -- 172.21.15.134:0/1056701994 shutdown_connections 2024-07-26T11:11:10.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.805+0000 7fb1a6e00640 1 --2- 172.21.15.134:0/1056701994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1a0075ad0 0x7fb1a010f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:10.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.805+0000 7fb1a6e00640 1 --2- 172.21.15.134:0/1056701994 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a010f980 0x7fb1a0111da0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:10.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.805+0000 7fb1a6e00640 1 --2- 172.21.15.134:0/1056701994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1a0075180 0x7fb1a0075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:10.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.805+0000 7fb1a6e00640 1 -- 172.21.15.134:0/1056701994 >> 172.21.15.134:0/1056701994 conn(0x7fb1a00fee10 msgr2=0x7fb1a0101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:10.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.805+0000 7fb1a6e00640 1 -- 172.21.15.134:0/1056701994 shutdown_connections 2024-07-26T11:11:10.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.805+0000 7fb1a6e00640 1 -- 172.21.15.134:0/1056701994 wait complete. 2024-07-26T11:11:10.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.806+0000 7fb1a6e00640 1 Processor -- start 2024-07-26T11:11:10.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.806+0000 7fb1a6e00640 1 -- start start 2024-07-26T11:11:10.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.806+0000 7fb1a6e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a0075180 0x7fb1a019beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:10.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.806+0000 7fb1a6e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1a0075ad0 0x7fb1a019c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:10.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.806+0000 7fb1a6e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1a010f980 0x7fb1a01a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:10.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.806+0000 7fb1a6e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a01154e0 con 0x7fb1a0075180 2024-07-26T11:11:10.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.807+0000 7fb1a6e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a0115360 con 0x7fb1a0075ad0 2024-07-26T11:11:10.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.807+0000 7fb1a6e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a0115660 con 0x7fb1a010f980 2024-07-26T11:11:10.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.807+0000 7fb1a5a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a0075180 0x7fb1a019beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:10.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.807+0000 7fb1a5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1a0075ad0 0x7fb1a019c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:10.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.807+0000 7fb1a5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1a0075ad0 0x7fb1a019c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56224/0 (socket says 172.21.15.134:56224) 2024-07-26T11:11:10.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.807+0000 7fb1a5000640 1 -- 172.21.15.134:0/2328840461 learned_addr learned my addr 172.21.15.134:0/2328840461 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:11:10.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.807+0000 7fb1a5a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a0075180 0x7fb1a019beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:41712/0 (socket says 172.21.15.134:41712) 2024-07-26T11:11:10.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.807+0000 7fb1a6400640 1 --2- 172.21.15.134:0/2328840461 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1a010f980 0x7fb1a01a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:10.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.807+0000 7fb1a5a00640 1 -- 172.21.15.134:0/2328840461 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1a010f980 msgr2=0x7fb1a01a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:10.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.807+0000 7fb1a5a00640 1 --2- 172.21.15.134:0/2328840461 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1a010f980 0x7fb1a01a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:10.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.807+0000 7fb1a5a00640 1 -- 172.21.15.134:0/2328840461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1a0075ad0 msgr2=0x7fb1a019c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:10.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.807+0000 7fb1a5a00640 1 --2- 172.21.15.134:0/2328840461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1a0075ad0 0x7fb1a019c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:10.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.807+0000 7fb1a5a00640 1 -- 172.21.15.134:0/2328840461 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a01a1e20 con 0x7fb1a0075180 2024-07-26T11:11:10.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.807+0000 7fb1a6400640 1 --2- 172.21.15.134:0/2328840461 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1a010f980 0x7fb1a01a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:11:10.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.808+0000 7fb1a5a00640 1 --2- 172.21.15.134:0/2328840461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a0075180 0x7fb1a019beb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb19400e9f0 tx=0x7fb19400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:10.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.808+0000 7fb19e000640 1 -- 172.21.15.134:0/2328840461 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19400ccc0 con 0x7fb1a0075180 2024-07-26T11:11:10.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.808+0000 7fb1a6e00640 1 -- 172.21.15.134:0/2328840461 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a01a2110 con 0x7fb1a0075180 2024-07-26T11:11:10.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.808+0000 7fb19e000640 1 -- 172.21.15.134:0/2328840461 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1940108e0 con 0x7fb1a0075180 2024-07-26T11:11:10.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.808+0000 7fb19e000640 1 -- 172.21.15.134:0/2328840461 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194018840 con 0x7fb1a0075180 2024-07-26T11:11:10.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.808+0000 7fb1a6e00640 1 -- 172.21.15.134:0/2328840461 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a01a8870 con 0x7fb1a0075180 2024-07-26T11:11:10.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.809+0000 7fb1a6e00640 1 -- 172.21.15.134:0/2328840461 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1a0075590 con 0x7fb1a0075180 2024-07-26T11:11:10.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.812+0000 7fb19e000640 1 -- 172.21.15.134:0/2328840461 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fb194010430 con 0x7fb1a0075180 2024-07-26T11:11:10.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.814+0000 7fb19e000640 1 --2- 172.21.15.134:0/2328840461 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1880b3c20 0x7fb1880b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:10.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.814+0000 7fb19e000640 1 -- 172.21.15.134:0/2328840461 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7fb1940e47d0 con 0x7fb1a0075180 2024-07-26T11:11:10.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.814+0000 7fb1a5000640 1 --2- 172.21.15.134:0/2328840461 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1880b3c20 0x7fb1880b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:10.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.815+0000 7fb1a5000640 1 --2- 172.21.15.134:0/2328840461 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1880b3c20 0x7fb1880b6130 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0111d70 tx=0x7fb18c009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:10.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:10.816+0000 7fb19e000640 1 -- 172.21.15.134:0/2328840461 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb1940ab9f0 con 0x7fb1a0075180 2024-07-26T11:11:11.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:11.052+0000 7fb1a6e00640 1 -- 172.21.15.134:0/2328840461 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb1a01a8b00 con 0x7fb1a0075180 2024-07-26T11:11:11.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:11.053+0000 7fb19e000640 1 -- 172.21.15.134:0/2328840461 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7fb1940ab2f0 con 0x7fb1a0075180 2024-07-26T11:11:11.055 INFO:teuthology.orchestra.run.smithi134.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-07-26T11:11:11.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:11.056+0000 7fb1a6e00640 1 -- 172.21.15.134:0/2328840461 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1880b3c20 msgr2=0x7fb1880b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:11.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:11.056+0000 7fb1a6e00640 1 --2- 172.21.15.134:0/2328840461 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1880b3c20 0x7fb1880b6130 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0111d70 tx=0x7fb18c009260 comp rx=0 tx=0).stop 2024-07-26T11:11:11.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:11.056+0000 7fb1a6e00640 1 -- 172.21.15.134:0/2328840461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a0075180 msgr2=0x7fb1a019beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:11.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:11.056+0000 7fb1a6e00640 1 --2- 172.21.15.134:0/2328840461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a0075180 0x7fb1a019beb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb19400e9f0 tx=0x7fb19400edf0 comp rx=0 tx=0).stop 2024-07-26T11:11:11.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:11.056+0000 7fb1a6e00640 1 -- 172.21.15.134:0/2328840461 shutdown_connections 2024-07-26T11:11:11.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:11.056+0000 7fb1a6e00640 1 --2- 172.21.15.134:0/2328840461 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1880b3c20 0x7fb1880b6130 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:11.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:11.056+0000 7fb1a6e00640 1 --2- 172.21.15.134:0/2328840461 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1a010f980 0x7fb1a01a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:11.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:11.056+0000 7fb1a6e00640 1 --2- 172.21.15.134:0/2328840461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1a0075ad0 0x7fb1a019c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:11.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:11.056+0000 7fb1a6e00640 1 --2- 172.21.15.134:0/2328840461 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a0075180 0x7fb1a019beb0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:11.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:11.056+0000 7fb1a6e00640 1 -- 172.21.15.134:0/2328840461 >> 172.21.15.134:0/2328840461 conn(0x7fb1a00fee10 msgr2=0x7fb1a0109520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:11.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:11.056+0000 7fb1a6e00640 1 -- 172.21.15.134:0/2328840461 shutdown_connections 2024-07-26T11:11:11.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:11.056+0000 7fb1a6e00640 1 -- 172.21.15.134:0/2328840461 wait complete. 2024-07-26T11:11:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:10 smithi193 ceph-mon[37473]: pgmap v334: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:10 smithi134 ceph-mon[31966]: pgmap v334: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:10 smithi189 ceph-mon[36790]: pgmap v334: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:11.941 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-07-26T11:11:11.941 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T11:11:11.941 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi134.front.sepia.ceph.com 2024-07-26T11:11:11.941 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-07-26T11:11:11.941 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-07-26T11:11:11.941 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-07-26T11:11:11.941 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-07-26T11:11:11.941 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T11:11:11.941 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-07-26T11:11:11.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:11:11.942 DEBUG:teuthology.orchestra.run.smithi134:> ip addr 2024-07-26T11:11:11.998 INFO:teuthology.orchestra.run.smithi134.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T11:11:11.998 INFO:teuthology.orchestra.run.smithi134.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T11:11:11.998 INFO:teuthology.orchestra.run.smithi134.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T11:11:11.998 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2024-07-26T11:11:11.998 INFO:teuthology.orchestra.run.smithi134.stdout: inet6 ::1/128 scope host 2024-07-26T11:11:11.999 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2024-07-26T11:11:11.999 INFO:teuthology.orchestra.run.smithi134.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T11:11:11.999 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:d9:38:e6 brd ff:ff:ff:ff:ff:ff 2024-07-26T11:11:11.999 INFO:teuthology.orchestra.run.smithi134.stdout: altname enp7s0f0 2024-07-26T11:11:11.999 INFO:teuthology.orchestra.run.smithi134.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T11:11:11.999 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:d9:38:e7 brd ff:ff:ff:ff:ff:ff 2024-07-26T11:11:11.999 INFO:teuthology.orchestra.run.smithi134.stdout: altname enp7s0f1 2024-07-26T11:11:11.999 INFO:teuthology.orchestra.run.smithi134.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T11:11:11.999 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:88:6d:e8 brd ff:ff:ff:ff:ff:ff 2024-07-26T11:11:11.999 INFO:teuthology.orchestra.run.smithi134.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T11:11:11.999 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:88:6d:e9 brd ff:ff:ff:ff:ff:ff 2024-07-26T11:11:11.999 INFO:teuthology.orchestra.run.smithi134.stdout: inet 172.21.15.134/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T11:11:11.999 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft 41942sec preferred_lft 41942sec 2024-07-26T11:11:11.999 INFO:teuthology.orchestra.run.smithi134.stdout: inet6 fe80::ec4:7aff:fe88:6de9/64 scope link 2024-07-26T11:11:12.000 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2024-07-26T11:11:12.001 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-07-26T11:11:12.001 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T11:11:12.001 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-07-26T11:11:12.001 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link add name ceph-brx type bridge 2024-07-26T11:11:12.001 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip addr flush dev ceph-brx 2024-07-26T11:11:12.001 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link set ceph-brx up 2024-07-26T11:11:12.001 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-07-26T11:11:12.001 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-07-26T11:11:12.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:11:12.131 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:11 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2328840461' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T11:11:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:11 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2328840461' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T11:11:12.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:11:12.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:11:12.429 DEBUG:teuthology.orchestra.run.smithi134:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T11:11:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:11 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2328840461' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T11:11:12.493 INFO:teuthology.orchestra.run.smithi134.stdout:1 2024-07-26T11:11:12.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:11:12.498 DEBUG:teuthology.orchestra.run.smithi134:> route 2024-07-26T11:11:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:Kernel IP routing table 2024-07-26T11:11:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T11:11:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T11:11:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T11:11:12.559 INFO:teuthology.orchestra.run.smithi134.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T11:11:12.560 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T11:11:12.560 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-07-26T11:11:12.560 DEBUG:teuthology.orchestra.run.smithi134:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T11:11:12.560 DEBUG:teuthology.orchestra.run.smithi134:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T11:11:12.560 DEBUG:teuthology.orchestra.run.smithi134:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T11:11:12.561 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-07-26T11:11:12.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:11:13.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:12 smithi189 ceph-mon[36790]: pgmap v335: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:13.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:11:13.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:13.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:12 smithi193 ceph-mon[37473]: pgmap v335: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:13.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:11:13.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:13.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:13.282 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:12 smithi134 ceph-mon[31966]: pgmap v335: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:13.283 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:11:13.283 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:13.283 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:13.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:11:13.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:11:13.296 DEBUG:teuthology.orchestra.run.smithi134:> ip netns list 2024-07-26T11:11:13.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:11:13.336 DEBUG:teuthology.orchestra.run.smithi134:> ip netns list-id 2024-07-26T11:11:13.421 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T11:11:13.421 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-07-26T11:11:13.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T11:11:13.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-07-26T11:11:13.422 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-07-26T11:11:13.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:11:13.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:11:13.625 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-07-26T11:11:13.625 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T11:11:13.625 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-07-26T11:11:13.625 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-07-26T11:11:13.625 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-07-26T11:11:13.625 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-07-26T11:11:13.626 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-07-26T11:11:13.626 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-07-26T11:11:13.626 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-07-26T11:11:13.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:11:13.902 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:13 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:11:13.910 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T11:11:13.910 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-07-26T11:11:13.910 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link set brx.0 up 2024-07-26T11:11:13.911 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link set dev brx.0 master ceph-brx 2024-07-26T11:11:13.911 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-07-26T11:11:13.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:13 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:11:14.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:11:14.091 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-07-26T11:11:14.091 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T11:11:14.092 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-07-26T11:11:14.127 INFO:teuthology.orchestra.run.smithi134.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-07-26T11:11:14.128 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T11:11:14.128 DEBUG:teuthology.orchestra.run.smithi134:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-07-26T11:11:14.209 DEBUG:teuthology.orchestra.run.smithi134:> sudo modprobe fuse 2024-07-26T11:11:14.318 DEBUG:teuthology.orchestra.run.smithi134:> cat /proc/self/mounts | awk '{print $2}' 2024-07-26T11:11:14.357 INFO:teuthology.orchestra.run.smithi134.stdout:/proc 2024-07-26T11:11:14.357 INFO:teuthology.orchestra.run.smithi134.stdout:/sys 2024-07-26T11:11:14.357 INFO:teuthology.orchestra.run.smithi134.stdout:/dev 2024-07-26T11:11:14.357 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/security 2024-07-26T11:11:14.357 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/shm 2024-07-26T11:11:14.357 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/pts 2024-07-26T11:11:14.357 INFO:teuthology.orchestra.run.smithi134.stdout:/run 2024-07-26T11:11:14.357 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/cgroup 2024-07-26T11:11:14.357 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/pstore 2024-07-26T11:11:14.357 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/bpf 2024-07-26T11:11:14.357 INFO:teuthology.orchestra.run.smithi134.stdout:/ 2024-07-26T11:11:14.358 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/selinux 2024-07-26T11:11:14.358 INFO:teuthology.orchestra.run.smithi134.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T11:11:14.358 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/hugepages 2024-07-26T11:11:14.358 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/mqueue 2024-07-26T11:11:14.358 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/debug 2024-07-26T11:11:14.358 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/tracing 2024-07-26T11:11:14.358 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/fuse/connections 2024-07-26T11:11:14.358 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/config 2024-07-26T11:11:14.358 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-sysctl.service 2024-07-26T11:11:14.358 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-07-26T11:11:14.358 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-07-26T11:11:14.358 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/nfs/rpc_pipefs 2024-07-26T11:11:14.359 INFO:teuthology.orchestra.run.smithi134.stdout:/run/user/1000 2024-07-26T11:11:14.359 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/ceph 2024-07-26T11:11:14.359 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay 2024-07-26T11:11:14.359 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/84cbd7dc563638e2813112c65f0b1f629b0a0171cd99386fa1ce340febe4c957/merged 2024-07-26T11:11:14.359 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/feb335e30d18b0f625b830f7a774571596241ba9c5ef8f111638375f2d918648/merged 2024-07-26T11:11:14.359 INFO:teuthology.orchestra.run.smithi134.stdout:/run/user/0 2024-07-26T11:11:14.359 INFO:teuthology.orchestra.run.smithi134.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T11:11:14.359 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/5928a78f2386d218993d1f3bbd39a5b5f93187b50608cdf635b5845cac7e4e07/merged 2024-07-26T11:11:14.359 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/d216bded3389482153abb5f3bc55c2084d21af5461f62b792757209bb8631e32/merged 2024-07-26T11:11:14.359 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/735e75a07d77398dd442d36feaa15b08ab60039e6246fbc7e646a75aaa5ca821/merged 2024-07-26T11:11:14.359 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/56f7d71ebb2e14817fb6b44e3589f9e8b63a2d20c0b5d5961c4448a24aa868b8/merged 2024-07-26T11:11:14.359 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/b2b11555abe75d47fa3d366cec661ba811bc7d5e337ee0a8b2873ebbfbea6167/merged 2024-07-26T11:11:14.359 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/cc746d6b6a3e39c2e5138a94336645cf56c3482e6e3a8c9e7279d4f6d7064e22/merged 2024-07-26T11:11:14.359 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/eb4f1e68a850ff004a418c3ed1dd12aeb38c61cfa7b7db9dd18c1903114773c1/merged 2024-07-26T11:11:14.360 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/03121ca86271a295f1e00418eac6c77377a9e8be1f2a68f0c54ab5fde6aba44d/merged 2024-07-26T11:11:14.360 INFO:teuthology.orchestra.run.smithi134.stdout:/run/netns 2024-07-26T11:11:14.360 INFO:teuthology.orchestra.run.smithi134.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T11:11:14.360 INFO:teuthology.orchestra.run.smithi134.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T11:11:14.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:11:14.361 DEBUG:teuthology.orchestra.run.smithi134:> ls /sys/fs/fuse/connections 2024-07-26T11:11:14.443 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-07-26T11:11:14.444 DEBUG:teuthology.orchestra.run.smithi134:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-07-26T11:11:14.488 DEBUG:teuthology.orchestra.run.smithi134:> sudo modprobe fuse 2024-07-26T11:11:14.563 DEBUG:teuthology.orchestra.run.smithi134:> cat /proc/self/mounts | awk '{print $2}' 2024-07-26T11:11:14.604 INFO:teuthology.orchestra.run.smithi134.stdout:/proc 2024-07-26T11:11:14.605 INFO:teuthology.orchestra.run.smithi134.stdout:/sys 2024-07-26T11:11:14.605 INFO:teuthology.orchestra.run.smithi134.stdout:/dev 2024-07-26T11:11:14.605 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/security 2024-07-26T11:11:14.605 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/shm 2024-07-26T11:11:14.605 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/pts 2024-07-26T11:11:14.605 INFO:teuthology.orchestra.run.smithi134.stdout:/run 2024-07-26T11:11:14.605 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/cgroup 2024-07-26T11:11:14.605 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/pstore 2024-07-26T11:11:14.605 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/bpf 2024-07-26T11:11:14.605 INFO:teuthology.orchestra.run.smithi134.stdout:/ 2024-07-26T11:11:14.605 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/selinux 2024-07-26T11:11:14.606 INFO:teuthology.orchestra.run.smithi134.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T11:11:14.606 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/hugepages 2024-07-26T11:11:14.606 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/mqueue 2024-07-26T11:11:14.606 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/debug 2024-07-26T11:11:14.606 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/tracing 2024-07-26T11:11:14.606 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/fuse/connections 2024-07-26T11:11:14.606 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/config 2024-07-26T11:11:14.606 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-sysctl.service 2024-07-26T11:11:14.606 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-07-26T11:11:14.606 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-07-26T11:11:14.606 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/nfs/rpc_pipefs 2024-07-26T11:11:14.606 INFO:teuthology.orchestra.run.smithi134.stdout:/run/user/1000 2024-07-26T11:11:14.606 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/ceph 2024-07-26T11:11:14.607 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay 2024-07-26T11:11:14.607 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/84cbd7dc563638e2813112c65f0b1f629b0a0171cd99386fa1ce340febe4c957/merged 2024-07-26T11:11:14.607 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/feb335e30d18b0f625b830f7a774571596241ba9c5ef8f111638375f2d918648/merged 2024-07-26T11:11:14.607 INFO:teuthology.orchestra.run.smithi134.stdout:/run/user/0 2024-07-26T11:11:14.607 INFO:teuthology.orchestra.run.smithi134.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T11:11:14.607 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/5928a78f2386d218993d1f3bbd39a5b5f93187b50608cdf635b5845cac7e4e07/merged 2024-07-26T11:11:14.607 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/d216bded3389482153abb5f3bc55c2084d21af5461f62b792757209bb8631e32/merged 2024-07-26T11:11:14.607 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/735e75a07d77398dd442d36feaa15b08ab60039e6246fbc7e646a75aaa5ca821/merged 2024-07-26T11:11:14.607 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/56f7d71ebb2e14817fb6b44e3589f9e8b63a2d20c0b5d5961c4448a24aa868b8/merged 2024-07-26T11:11:14.607 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/b2b11555abe75d47fa3d366cec661ba811bc7d5e337ee0a8b2873ebbfbea6167/merged 2024-07-26T11:11:14.607 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/cc746d6b6a3e39c2e5138a94336645cf56c3482e6e3a8c9e7279d4f6d7064e22/merged 2024-07-26T11:11:14.607 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/eb4f1e68a850ff004a418c3ed1dd12aeb38c61cfa7b7db9dd18c1903114773c1/merged 2024-07-26T11:11:14.607 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/03121ca86271a295f1e00418eac6c77377a9e8be1f2a68f0c54ab5fde6aba44d/merged 2024-07-26T11:11:14.607 INFO:teuthology.orchestra.run.smithi134.stdout:/run/netns 2024-07-26T11:11:14.608 INFO:teuthology.orchestra.run.smithi134.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T11:11:14.608 INFO:teuthology.orchestra.run.smithi134.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T11:11:14.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:11:14.609 DEBUG:teuthology.orchestra.run.smithi134:> ls /sys/fs/fuse/connections 2024-07-26T11:11:14.880 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi134.stderr:2024-07-26T11:11:14.878+0000 7fa9324f7480 -1 init, newargv = 0x7fa9180046a0 newargc=15 2024-07-26T11:11:14.880 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi134.stderr:ceph-fuse[82732]: starting ceph client 2024-07-26T11:11:14.892 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi134.stderr:ceph-fuse[82732]: starting fuse 2024-07-26T11:11:14.913 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:14.913 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:14.913 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:14.913 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:14.913 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:14 smithi134 ceph-mon[31966]: pgmap v336: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:14 smithi189 ceph-mon[36790]: pgmap v336: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:15.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:15.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:14 smithi193 ceph-mon[37473]: pgmap v336: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:15.791 DEBUG:teuthology.orchestra.run.smithi134:> sudo modprobe fuse 2024-07-26T11:11:15.863 DEBUG:teuthology.orchestra.run.smithi134:> cat /proc/self/mounts | awk '{print $2}' 2024-07-26T11:11:15.906 INFO:teuthology.orchestra.run.smithi134.stdout:/proc 2024-07-26T11:11:15.906 INFO:teuthology.orchestra.run.smithi134.stdout:/sys 2024-07-26T11:11:15.906 INFO:teuthology.orchestra.run.smithi134.stdout:/dev 2024-07-26T11:11:15.906 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/security 2024-07-26T11:11:15.906 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/shm 2024-07-26T11:11:15.906 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/pts 2024-07-26T11:11:15.906 INFO:teuthology.orchestra.run.smithi134.stdout:/run 2024-07-26T11:11:15.907 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/cgroup 2024-07-26T11:11:15.907 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/pstore 2024-07-26T11:11:15.907 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/bpf 2024-07-26T11:11:15.907 INFO:teuthology.orchestra.run.smithi134.stdout:/ 2024-07-26T11:11:15.907 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/selinux 2024-07-26T11:11:15.907 INFO:teuthology.orchestra.run.smithi134.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T11:11:15.907 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/hugepages 2024-07-26T11:11:15.907 INFO:teuthology.orchestra.run.smithi134.stdout:/dev/mqueue 2024-07-26T11:11:15.907 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/debug 2024-07-26T11:11:15.907 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/tracing 2024-07-26T11:11:15.907 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/fs/fuse/connections 2024-07-26T11:11:15.907 INFO:teuthology.orchestra.run.smithi134.stdout:/sys/kernel/config 2024-07-26T11:11:15.907 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-sysctl.service 2024-07-26T11:11:15.907 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-07-26T11:11:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-07-26T11:11:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/nfs/rpc_pipefs 2024-07-26T11:11:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:/run/user/1000 2024-07-26T11:11:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/ceph 2024-07-26T11:11:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay 2024-07-26T11:11:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/84cbd7dc563638e2813112c65f0b1f629b0a0171cd99386fa1ce340febe4c957/merged 2024-07-26T11:11:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/feb335e30d18b0f625b830f7a774571596241ba9c5ef8f111638375f2d918648/merged 2024-07-26T11:11:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:/run/user/0 2024-07-26T11:11:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:/proc/sys/fs/binfmt_misc 2024-07-26T11:11:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/5928a78f2386d218993d1f3bbd39a5b5f93187b50608cdf635b5845cac7e4e07/merged 2024-07-26T11:11:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/d216bded3389482153abb5f3bc55c2084d21af5461f62b792757209bb8631e32/merged 2024-07-26T11:11:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/735e75a07d77398dd442d36feaa15b08ab60039e6246fbc7e646a75aaa5ca821/merged 2024-07-26T11:11:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/56f7d71ebb2e14817fb6b44e3589f9e8b63a2d20c0b5d5961c4448a24aa868b8/merged 2024-07-26T11:11:15.908 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/b2b11555abe75d47fa3d366cec661ba811bc7d5e337ee0a8b2873ebbfbea6167/merged 2024-07-26T11:11:15.909 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/cc746d6b6a3e39c2e5138a94336645cf56c3482e6e3a8c9e7279d4f6d7064e22/merged 2024-07-26T11:11:15.909 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/eb4f1e68a850ff004a418c3ed1dd12aeb38c61cfa7b7db9dd18c1903114773c1/merged 2024-07-26T11:11:15.909 INFO:teuthology.orchestra.run.smithi134.stdout:/var/lib/containers/storage/overlay/03121ca86271a295f1e00418eac6c77377a9e8be1f2a68f0c54ab5fde6aba44d/merged 2024-07-26T11:11:15.909 INFO:teuthology.orchestra.run.smithi134.stdout:/run/netns 2024-07-26T11:11:15.909 INFO:teuthology.orchestra.run.smithi134.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T11:11:15.909 INFO:teuthology.orchestra.run.smithi134.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T11:11:15.909 INFO:teuthology.orchestra.run.smithi134.stdout:/home/ubuntu/cephtest/mnt.admin 2024-07-26T11:11:15.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:11:15.910 DEBUG:teuthology.orchestra.run.smithi134:> ls /sys/fs/fuse/connections 2024-07-26T11:11:15.994 INFO:teuthology.orchestra.run.smithi134.stdout:119 2024-07-26T11:11:15.995 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-07-26T11:11:15.995 DEBUG:teuthology.orchestra.run.smithi134:> sudo stdin-killer -- python3 -c ' 2024-07-26T11:11:15.995 DEBUG:teuthology.orchestra.run.smithi134:> import glob 2024-07-26T11:11:15.995 DEBUG:teuthology.orchestra.run.smithi134:> import re 2024-07-26T11:11:15.995 DEBUG:teuthology.orchestra.run.smithi134:> import os 2024-07-26T11:11:15.995 DEBUG:teuthology.orchestra.run.smithi134:> import subprocess 2024-07-26T11:11:15.995 DEBUG:teuthology.orchestra.run.smithi134:> 2024-07-26T11:11:15.995 DEBUG:teuthology.orchestra.run.smithi134:> def _find_admin_socket(client_name): 2024-07-26T11:11:15.995 DEBUG:teuthology.orchestra.run.smithi134:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-07-26T11:11:15.995 DEBUG:teuthology.orchestra.run.smithi134:> files = glob.glob(asok_path) 2024-07-26T11:11:15.995 DEBUG:teuthology.orchestra.run.smithi134:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-07-26T11:11:15.995 DEBUG:teuthology.orchestra.run.smithi134:> 2024-07-26T11:11:15.996 DEBUG:teuthology.orchestra.run.smithi134:> # Given a non-glob path, it better be there 2024-07-26T11:11:15.996 DEBUG:teuthology.orchestra.run.smithi134:> if "*" not in asok_path: 2024-07-26T11:11:15.996 DEBUG:teuthology.orchestra.run.smithi134:> assert(len(files) == 1) 2024-07-26T11:11:15.996 DEBUG:teuthology.orchestra.run.smithi134:> return files[0] 2024-07-26T11:11:15.996 DEBUG:teuthology.orchestra.run.smithi134:> 2024-07-26T11:11:15.996 DEBUG:teuthology.orchestra.run.smithi134:> for f in files: 2024-07-26T11:11:15.996 DEBUG:teuthology.orchestra.run.smithi134:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-07-26T11:11:15.996 DEBUG:teuthology.orchestra.run.smithi134:> if os.path.exists("/proc/{0}".format(pid)): 2024-07-26T11:11:15.996 DEBUG:teuthology.orchestra.run.smithi134:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-07-26T11:11:15.996 DEBUG:teuthology.orchestra.run.smithi134:> contents = proc_f.read() 2024-07-26T11:11:15.996 DEBUG:teuthology.orchestra.run.smithi134:> if mountpoint in contents: 2024-07-26T11:11:15.996 DEBUG:teuthology.orchestra.run.smithi134:> return f 2024-07-26T11:11:15.996 DEBUG:teuthology.orchestra.run.smithi134:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-07-26T11:11:15.996 DEBUG:teuthology.orchestra.run.smithi134:> 2024-07-26T11:11:15.996 DEBUG:teuthology.orchestra.run.smithi134:> print(_find_admin_socket("client.admin")) 2024-07-26T11:11:15.996 DEBUG:teuthology.orchestra.run.smithi134:> ' 2024-07-26T11:11:16.190 INFO:teuthology.orchestra.run.smithi134.stdout:/var/run/ceph/ceph-client.admin.82732.asok 2024-07-26T11:11:16.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:11:16.208 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.82732.asok 2024-07-26T11:11:16.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:11:16.208 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.82732.asok status 2024-07-26T11:11:16.485 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:11:16.485 INFO:teuthology.orchestra.run.smithi134.stdout: "metadata": { 2024-07-26T11:11:16.485 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_sha1": "e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528", 2024-07-26T11:11:16.485 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_version": "ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev)", 2024-07-26T11:11:16.485 INFO:teuthology.orchestra.run.smithi134.stdout: "entity_id": "admin", 2024-07-26T11:11:16.485 INFO:teuthology.orchestra.run.smithi134.stdout: "hostname": "smithi134", 2024-07-26T11:11:16.485 INFO:teuthology.orchestra.run.smithi134.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-07-26T11:11:16.485 INFO:teuthology.orchestra.run.smithi134.stdout: "pid": "82732", 2024-07-26T11:11:16.486 INFO:teuthology.orchestra.run.smithi134.stdout: "root": "/" 2024-07-26T11:11:16.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-07-26T11:11:16.486 INFO:teuthology.orchestra.run.smithi134.stdout: "dentry_count": 0, 2024-07-26T11:11:16.486 INFO:teuthology.orchestra.run.smithi134.stdout: "dentry_pinned_count": 0, 2024-07-26T11:11:16.486 INFO:teuthology.orchestra.run.smithi134.stdout: "id": 24847, 2024-07-26T11:11:16.486 INFO:teuthology.orchestra.run.smithi134.stdout: "inst": { 2024-07-26T11:11:16.486 INFO:teuthology.orchestra.run.smithi134.stdout: "name": { 2024-07-26T11:11:16.486 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "client", 2024-07-26T11:11:16.486 INFO:teuthology.orchestra.run.smithi134.stdout: "num": 24847 2024-07-26T11:11:16.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-07-26T11:11:16.486 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": { 2024-07-26T11:11:16.486 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-07-26T11:11:16.486 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:0", 2024-07-26T11:11:16.487 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 473571628 2024-07-26T11:11:16.487 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-07-26T11:11:16.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-07-26T11:11:16.487 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": { 2024-07-26T11:11:16.487 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-07-26T11:11:16.487 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:0", 2024-07-26T11:11:16.487 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 473571628 2024-07-26T11:11:16.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-07-26T11:11:16.487 INFO:teuthology.orchestra.run.smithi134.stdout: "inst_str": "client.24847 172.21.15.134:0/473571628", 2024-07-26T11:11:16.487 INFO:teuthology.orchestra.run.smithi134.stdout: "addr_str": "172.21.15.134:0/473571628", 2024-07-26T11:11:16.487 INFO:teuthology.orchestra.run.smithi134.stdout: "inode_count": 1, 2024-07-26T11:11:16.487 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_epoch": 17, 2024-07-26T11:11:16.487 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_epoch": 88, 2024-07-26T11:11:16.487 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_epoch_barrier": 0, 2024-07-26T11:11:16.487 INFO:teuthology.orchestra.run.smithi134.stdout: "blocklisted": false, 2024-07-26T11:11:16.488 INFO:teuthology.orchestra.run.smithi134.stdout: "fs_name": "cephfs" 2024-07-26T11:11:16.488 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:11:16.503 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:16.503 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:16.503 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:16 smithi134 ceph-mon[31966]: pgmap v337: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:16.503 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:11:16.503 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:11:16.503 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:16.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:11:16.506 DEBUG:teuthology.orchestra.run.smithi134:> stat --file-system '--printf=%T 2024-07-26T11:11:16.506 DEBUG:teuthology.orchestra.run.smithi134:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-07-26T11:11:16.556 INFO:teuthology.orchestra.run.smithi134.stdout:fuseblk 2024-07-26T11:11:16.556 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-07-26T11:11:16.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:11:16.557 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-07-26T11:11:16.633 DEBUG:teuthology.orchestra.run.smithi134:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-07-26T11:11:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:16 smithi189 ceph-mon[36790]: pgmap v337: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:11:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:11:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:16.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:16 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T11:11:16.766 INFO:teuthology.orchestra.run.smithi134.stdout:# file: . 2024-07-26T11:11:16.766 INFO:teuthology.orchestra.run.smithi134.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-07-26T11:11:16.766 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:11:16.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:11:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:16 smithi193 ceph-mon[37473]: pgmap v337: 129 pgs: 129 active+clean; 579 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:11:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:11:16.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:11:16.777 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T11:11:16.777 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T11:11:16.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd blocklist ls 2024-07-26T11:11:17.322 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:11:19.120 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:18 smithi134 ceph-mon[31966]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T11:11:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:18 smithi193 ceph-mon[37473]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T11:11:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:18 smithi189 ceph-mon[36790]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T11:11:19.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.730+0000 7ff127200640 1 -- 172.21.15.134:0/835501539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff12810af10 msgr2=0x7ff12810b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:19.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.730+0000 7ff127200640 1 --2- 172.21.15.134:0/835501539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff12810af10 0x7ff12810b320 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff110009860 tx=0x7ff11002a340 comp rx=0 tx=0).stop 2024-07-26T11:11:19.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.731+0000 7ff127200640 1 -- 172.21.15.134:0/835501539 shutdown_connections 2024-07-26T11:11:19.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.731+0000 7ff127200640 1 --2- 172.21.15.134:0/835501539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff12810af10 0x7ff12810b320 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:19.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.731+0000 7ff127200640 1 --2- 172.21.15.134:0/835501539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff12810b860 0x7ff128112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:19.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.731+0000 7ff127200640 1 --2- 172.21.15.134:0/835501539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff1281055d0 0x7ff1281059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:19.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.731+0000 7ff127200640 1 -- 172.21.15.134:0/835501539 >> 172.21.15.134:0/835501539 conn(0x7ff128101040 msgr2=0x7ff128103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:19.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.731+0000 7ff127200640 1 -- 172.21.15.134:0/835501539 shutdown_connections 2024-07-26T11:11:19.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.731+0000 7ff127200640 1 -- 172.21.15.134:0/835501539 wait complete. 2024-07-26T11:11:19.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.732+0000 7ff127200640 1 Processor -- start 2024-07-26T11:11:19.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.732+0000 7ff127200640 1 -- start start 2024-07-26T11:11:19.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.732+0000 7ff127200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1281055d0 0x7ff12819bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:19.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.732+0000 7ff127200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff12810af10 0x7ff12819c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:19.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.733+0000 7ff127200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff12810b860 0x7ff1281a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:19.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.733+0000 7ff127200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff1281154f0 con 0x7ff12810b860 2024-07-26T11:11:19.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.733+0000 7ff127200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff128115370 con 0x7ff1281055d0 2024-07-26T11:11:19.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.733+0000 7ff127200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff128115670 con 0x7ff12810af10 2024-07-26T11:11:19.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.733+0000 7ff126800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff12810b860 0x7ff1281a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:19.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.733+0000 7ff11fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff12810af10 0x7ff12819c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:19.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.733+0000 7ff126800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff12810b860 0x7ff1281a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:41730/0 (socket says 172.21.15.134:41730) 2024-07-26T11:11:19.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.733+0000 7ff11fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff12810af10 0x7ff12819c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34330/0 (socket says 172.21.15.134:34330) 2024-07-26T11:11:19.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.733+0000 7ff125e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1281055d0 0x7ff12819bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:19.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.733+0000 7ff126800640 1 -- 172.21.15.134:0/507796751 learned_addr learned my addr 172.21.15.134:0/507796751 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:11:19.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.733+0000 7ff11fe00640 1 -- 172.21.15.134:0/507796751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1281055d0 msgr2=0x7ff12819bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:19.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.733+0000 7ff11fe00640 1 --2- 172.21.15.134:0/507796751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1281055d0 0x7ff12819bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:19.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.733+0000 7ff11fe00640 1 -- 172.21.15.134:0/507796751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff12810b860 msgr2=0x7ff1281a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:19.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.733+0000 7ff11fe00640 1 --2- 172.21.15.134:0/507796751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff12810b860 0x7ff1281a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:19.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.733+0000 7ff11fe00640 1 -- 172.21.15.134:0/507796751 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1281a1e90 con 0x7ff12810af10 2024-07-26T11:11:19.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.734+0000 7ff11fe00640 1 --2- 172.21.15.134:0/507796751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff12810af10 0x7ff12819c4c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff11002a860 tx=0x7ff110032dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:19.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.734+0000 7ff11ea00640 1 -- 172.21.15.134:0/507796751 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff110040020 con 0x7ff12810af10 2024-07-26T11:11:19.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.734+0000 7ff11ea00640 1 -- 172.21.15.134:0/507796751 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff110030450 con 0x7ff12810af10 2024-07-26T11:11:19.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.734+0000 7ff127200640 1 -- 172.21.15.134:0/507796751 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1281a2120 con 0x7ff12810af10 2024-07-26T11:11:19.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.734+0000 7ff11ea00640 1 -- 172.21.15.134:0/507796751 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff110037040 con 0x7ff12810af10 2024-07-26T11:11:19.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.734+0000 7ff127200640 1 -- 172.21.15.134:0/507796751 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1281a88e0 con 0x7ff12810af10 2024-07-26T11:11:19.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.735+0000 7ff127200640 1 -- 172.21.15.134:0/507796751 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff128106850 con 0x7ff12810af10 2024-07-26T11:11:19.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.738+0000 7ff11ea00640 1 -- 172.21.15.134:0/507796751 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7ff110043020 con 0x7ff12810af10 2024-07-26T11:11:19.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.739+0000 7ff11ea00640 1 --2- 172.21.15.134:0/507796751 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff10c0b3c20 0x7ff10c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:19.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.739+0000 7ff11ea00640 1 -- 172.21.15.134:0/507796751 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7ff1101044c0 con 0x7ff12810af10 2024-07-26T11:11:19.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.740+0000 7ff125e00640 1 --2- 172.21.15.134:0/507796751 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff10c0b3c20 0x7ff10c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:19.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.740+0000 7ff125e00640 1 --2- 172.21.15.134:0/507796751 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff10c0b3c20 0x7ff10c0b6130 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff11800bbe0 tx=0x7ff11800b390 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:19.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.741+0000 7ff11ea00640 1 -- 172.21.15.134:0/507796751 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff11004a020 con 0x7ff12810af10 2024-07-26T11:11:19.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.949+0000 7ff127200640 1 -- 172.21.15.134:0/507796751 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff1281059e0 con 0x7ff12810af10 2024-07-26T11:11:19.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.950+0000 7ff11ea00640 1 -- 172.21.15.134:0/507796751 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v88) ==== 81+0+901 (secure 0 0 0) 0x7ff1281059e0 con 0x7ff12810af10 2024-07-26T11:11:19.952 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/3790862248 2024-07-27T11:02:33.711793+0000 2024-07-26T11:11:19.952 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:6800/2424179007 2024-07-27T11:01:12.679951+0000 2024-07-26T11:11:19.952 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/2906927791 2024-07-27T11:01:12.679951+0000 2024-07-26T11:11:19.952 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:6800/3630314144 2024-07-27T11:00:42.651939+0000 2024-07-26T11:11:19.952 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/3068153391 2024-07-27T11:00:42.651939+0000 2024-07-26T11:11:19.952 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/3056560791 2024-07-27T11:01:12.679951+0000 2024-07-26T11:11:19.952 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:6800/606980428 2024-07-27T11:02:33.711793+0000 2024-07-26T11:11:19.952 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:6801/2424179007 2024-07-27T11:01:12.679951+0000 2024-07-26T11:11:19.953 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:6801/606980428 2024-07-27T11:02:33.711793+0000 2024-07-26T11:11:19.953 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/1310359922 2024-07-27T11:02:33.711793+0000 2024-07-26T11:11:19.953 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/2318666906 2024-07-27T11:00:42.651939+0000 2024-07-26T11:11:19.953 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/2901579347 2024-07-27T11:01:12.679951+0000 2024-07-26T11:11:19.953 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:6801/3630314144 2024-07-27T11:00:42.651939+0000 2024-07-26T11:11:19.953 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/3404218719 2024-07-27T11:00:42.651939+0000 2024-07-26T11:11:19.953 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/3598071145 2024-07-27T11:02:33.711793+0000 2024-07-26T11:11:19.953 INFO:teuthology.orchestra.run.smithi134.stderr:listed 15 entries 2024-07-26T11:11:19.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.952+0000 7ff127200640 1 -- 172.21.15.134:0/507796751 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff10c0b3c20 msgr2=0x7ff10c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:19.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.953+0000 7ff127200640 1 --2- 172.21.15.134:0/507796751 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff10c0b3c20 0x7ff10c0b6130 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff11800bbe0 tx=0x7ff11800b390 comp rx=0 tx=0).stop 2024-07-26T11:11:19.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.953+0000 7ff127200640 1 -- 172.21.15.134:0/507796751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff12810af10 msgr2=0x7ff12819c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:19.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.953+0000 7ff127200640 1 --2- 172.21.15.134:0/507796751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff12810af10 0x7ff12819c4c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff11002a860 tx=0x7ff110032dc0 comp rx=0 tx=0).stop 2024-07-26T11:11:19.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.953+0000 7ff127200640 1 -- 172.21.15.134:0/507796751 shutdown_connections 2024-07-26T11:11:19.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.953+0000 7ff127200640 1 --2- 172.21.15.134:0/507796751 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff10c0b3c20 0x7ff10c0b6130 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:19.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.953+0000 7ff127200640 1 --2- 172.21.15.134:0/507796751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff12810af10 0x7ff12819c4c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:19.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.953+0000 7ff127200640 1 --2- 172.21.15.134:0/507796751 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff12810b860 0x7ff1281a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:19.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.953+0000 7ff127200640 1 --2- 172.21.15.134:0/507796751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1281055d0 0x7ff12819bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:19.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.953+0000 7ff127200640 1 -- 172.21.15.134:0/507796751 >> 172.21.15.134:0/507796751 conn(0x7ff128101040 msgr2=0x7ff128103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:19.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.953+0000 7ff127200640 1 -- 172.21.15.134:0/507796751 shutdown_connections 2024-07-26T11:11:19.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:19.953+0000 7ff127200640 1 -- 172.21.15.134:0/507796751 wait complete. 2024-07-26T11:11:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:20 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/507796751' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T11:11:20.364 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:20 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/507796751' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T11:11:20.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:20 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/507796751' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T11:11:20.826 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T11:11:20.827 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T11:11:20.886 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd blocklist ls 2024-07-26T11:11:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:21 smithi193 ceph-mon[37473]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T11:11:21.339 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:11:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:21 smithi134 ceph-mon[31966]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T11:11:21.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:21 smithi189 ceph-mon[36790]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T11:11:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:23 smithi193 ceph-mon[37473]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T11:11:23.302 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:23 smithi134 ceph-mon[31966]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T11:11:23.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:23 smithi189 ceph-mon[36790]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T11:11:23.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.874+0000 7f3db7200640 1 -- 172.21.15.134:0/1982394846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3db8077cc0 msgr2=0x7f3db8113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:23.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.874+0000 7f3db7200640 1 --2- 172.21.15.134:0/1982394846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3db8077cc0 0x7f3db8113a40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3da0009900 tx=0x7f3da002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:11:23.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.874+0000 7f3db7200640 1 -- 172.21.15.134:0/1982394846 shutdown_connections 2024-07-26T11:11:23.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.874+0000 7f3db7200640 1 --2- 172.21.15.134:0/1982394846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3db8113f80 0x7f3db81163a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:23.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.874+0000 7f3db7200640 1 --2- 172.21.15.134:0/1982394846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3db8077cc0 0x7f3db8113a40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:23.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.874+0000 7f3db7200640 1 --2- 172.21.15.134:0/1982394846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3db8077370 0x7f3db8077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:23.878 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.874+0000 7f3db7200640 1 -- 172.21.15.134:0/1982394846 >> 172.21.15.134:0/1982394846 conn(0x7f3db8101010 msgr2=0x7f3db8103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:23.878 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.874+0000 7f3db7200640 1 -- 172.21.15.134:0/1982394846 shutdown_connections 2024-07-26T11:11:23.878 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.874+0000 7f3db7200640 1 -- 172.21.15.134:0/1982394846 wait complete. 2024-07-26T11:11:23.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.875+0000 7f3db7200640 1 Processor -- start 2024-07-26T11:11:23.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.876+0000 7f3db7200640 1 -- start start 2024-07-26T11:11:23.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.876+0000 7f3db7200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3db8077370 0x7f3db81a02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:23.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.876+0000 7f3db7200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3db8077cc0 0x7f3db81a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:23.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.876+0000 7f3db7200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3db8113f80 0x7f3db81a5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:23.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.876+0000 7f3db7200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3db81198e0 con 0x7f3db8113f80 2024-07-26T11:11:23.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.876+0000 7f3db7200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3db8119760 con 0x7f3db8077370 2024-07-26T11:11:23.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.876+0000 7f3db7200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3db8119a60 con 0x7f3db8077cc0 2024-07-26T11:11:23.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.876+0000 7f3db6800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3db8113f80 0x7f3db81a5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:23.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.877+0000 7f3db5e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3db8077370 0x7f3db81a02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:23.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.876+0000 7f3db5400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3db8077cc0 0x7f3db81a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:23.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.877+0000 7f3db5e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3db8077370 0x7f3db81a02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:48998/0 (socket says 172.21.15.134:48998) 2024-07-26T11:11:23.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.877+0000 7f3db6800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3db8113f80 0x7f3db81a5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:55354/0 (socket says 172.21.15.134:55354) 2024-07-26T11:11:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.877+0000 7f3db5e00640 1 -- 172.21.15.134:0/38536024 learned_addr learned my addr 172.21.15.134:0/38536024 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:11:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.877+0000 7f3db6800640 1 -- 172.21.15.134:0/38536024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3db8077cc0 msgr2=0x7f3db81a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.877+0000 7f3db6800640 1 --2- 172.21.15.134:0/38536024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3db8077cc0 0x7f3db81a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.877+0000 7f3db6800640 1 -- 172.21.15.134:0/38536024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3db8077370 msgr2=0x7f3db81a02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.877+0000 7f3db6800640 1 --2- 172.21.15.134:0/38536024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3db8077370 0x7f3db81a02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.877+0000 7f3db6800640 1 -- 172.21.15.134:0/38536024 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db81a6240 con 0x7f3db8113f80 2024-07-26T11:11:23.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.877+0000 7f3db6800640 1 --2- 172.21.15.134:0/38536024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3db8113f80 0x7f3db81a5b10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3db0009fd0 tx=0x7f3db000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:23.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.878+0000 7f3daea00640 1 -- 172.21.15.134:0/38536024 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db000ce20 con 0x7f3db8113f80 2024-07-26T11:11:23.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.878+0000 7f3db7200640 1 -- 172.21.15.134:0/38536024 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db81a6530 con 0x7f3db8113f80 2024-07-26T11:11:23.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.878+0000 7f3daea00640 1 -- 172.21.15.134:0/38536024 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3db0005020 con 0x7f3db8113f80 2024-07-26T11:11:23.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.878+0000 7f3daea00640 1 -- 172.21.15.134:0/38536024 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db0010690 con 0x7f3db8113f80 2024-07-26T11:11:23.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.878+0000 7f3db7200640 1 -- 172.21.15.134:0/38536024 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db81acc90 con 0x7f3db8113f80 2024-07-26T11:11:23.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.879+0000 7f3db7200640 1 -- 172.21.15.134:0/38536024 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3db8077780 con 0x7f3db8113f80 2024-07-26T11:11:23.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.882+0000 7f3daea00640 1 -- 172.21.15.134:0/38536024 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f3db00054d0 con 0x7f3db8113f80 2024-07-26T11:11:23.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.883+0000 7f3daea00640 1 --2- 172.21.15.134:0/38536024 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d9c0b3ca0 0x7f3d9c0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:23.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.884+0000 7f3daea00640 1 -- 172.21.15.134:0/38536024 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 9307+0+0 (secure 0 0 0) 0x7f3db00e4660 con 0x7f3db8113f80 2024-07-26T11:11:23.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.884+0000 7f3db5e00640 1 --2- 172.21.15.134:0/38536024 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d9c0b3ca0 0x7f3d9c0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:23.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.884+0000 7f3db5e00640 1 --2- 172.21.15.134:0/38536024 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d9c0b3ca0 0x7f3d9c0b61b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009c90 tx=0x7f3da8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:23.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:23.886+0000 7f3daea00640 1 -- 172.21.15.134:0/38536024 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3db00ab880 con 0x7f3db8113f80 2024-07-26T11:11:24.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:24.105+0000 7f3db7200640 1 -- 172.21.15.134:0/38536024 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3db81a1400 con 0x7f3db8113f80 2024-07-26T11:11:24.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:24.106+0000 7f3daea00640 1 -- 172.21.15.134:0/38536024 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 15 entries v88) ==== 81+0+901 (secure 0 0 0) 0x7f3db81a1400 con 0x7f3db8113f80 2024-07-26T11:11:24.109 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/3790862248 2024-07-27T11:02:33.711793+0000 2024-07-26T11:11:24.109 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:6800/2424179007 2024-07-27T11:01:12.679951+0000 2024-07-26T11:11:24.109 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/2906927791 2024-07-27T11:01:12.679951+0000 2024-07-26T11:11:24.109 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:6800/3630314144 2024-07-27T11:00:42.651939+0000 2024-07-26T11:11:24.109 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/3068153391 2024-07-27T11:00:42.651939+0000 2024-07-26T11:11:24.109 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/3056560791 2024-07-27T11:01:12.679951+0000 2024-07-26T11:11:24.109 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:6800/606980428 2024-07-27T11:02:33.711793+0000 2024-07-26T11:11:24.109 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:6801/2424179007 2024-07-27T11:01:12.679951+0000 2024-07-26T11:11:24.109 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:6801/606980428 2024-07-27T11:02:33.711793+0000 2024-07-26T11:11:24.109 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/1310359922 2024-07-27T11:02:33.711793+0000 2024-07-26T11:11:24.109 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/2318666906 2024-07-27T11:00:42.651939+0000 2024-07-26T11:11:24.109 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/2901579347 2024-07-27T11:01:12.679951+0000 2024-07-26T11:11:24.110 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:6801/3630314144 2024-07-27T11:00:42.651939+0000 2024-07-26T11:11:24.110 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/3404218719 2024-07-27T11:00:42.651939+0000 2024-07-26T11:11:24.110 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134:0/3598071145 2024-07-27T11:02:33.711793+0000 2024-07-26T11:11:24.110 INFO:teuthology.orchestra.run.smithi134.stderr:listed 15 entries 2024-07-26T11:11:24.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:24.109+0000 7f3db7200640 1 -- 172.21.15.134:0/38536024 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d9c0b3ca0 msgr2=0x7f3d9c0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:24.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:24.109+0000 7f3db7200640 1 --2- 172.21.15.134:0/38536024 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d9c0b3ca0 0x7f3d9c0b61b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009c90 tx=0x7f3da8005f90 comp rx=0 tx=0).stop 2024-07-26T11:11:24.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:24.109+0000 7f3db7200640 1 -- 172.21.15.134:0/38536024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3db8113f80 msgr2=0x7f3db81a5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:24.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:24.109+0000 7f3db7200640 1 --2- 172.21.15.134:0/38536024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3db8113f80 0x7f3db81a5b10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3db0009fd0 tx=0x7f3db000ef10 comp rx=0 tx=0).stop 2024-07-26T11:11:24.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:24.109+0000 7f3db7200640 1 -- 172.21.15.134:0/38536024 shutdown_connections 2024-07-26T11:11:24.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:24.109+0000 7f3db7200640 1 --2- 172.21.15.134:0/38536024 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d9c0b3ca0 0x7f3d9c0b61b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:24.109+0000 7f3db7200640 1 --2- 172.21.15.134:0/38536024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3db8077cc0 0x7f3db81a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:24.109+0000 7f3db7200640 1 --2- 172.21.15.134:0/38536024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3db8113f80 0x7f3db81a5b10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:24.109+0000 7f3db7200640 1 --2- 172.21.15.134:0/38536024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3db8077370 0x7f3db81a02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:24.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:24.109+0000 7f3db7200640 1 -- 172.21.15.134:0/38536024 >> 172.21.15.134:0/38536024 conn(0x7f3db8101010 msgr2=0x7f3db8116080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:24.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:24.109+0000 7f3db7200640 1 -- 172.21.15.134:0/38536024 shutdown_connections 2024-07-26T11:11:24.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:24.109+0000 7f3db7200640 1 -- 172.21.15.134:0/38536024 wait complete. 2024-07-26T11:11:25.056 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi134.front.sepia.ceph.com... 2024-07-26T11:11:25.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:11:25.056 DEBUG:teuthology.orchestra.run.smithi134:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-07-26T11:11:25.140 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi134.stderr:ceph-fuse[82732]: fuse finished with error 0 and tester_r 0 2024-07-26T11:11:25.146 INFO:teuthology.orchestra.run:waiting for 300 2024-07-26T11:11:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:25 smithi193 ceph-mon[37473]: pgmap v341: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T11:11:25.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:25 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/38536024' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T11:11:25.378 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:25 smithi134 ceph-mon[31966]: pgmap v341: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T11:11:25.378 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:25 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/38536024' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T11:11:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:25 smithi189 ceph-mon[36790]: pgmap v341: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T11:11:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:25 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/38536024' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-07-26T11:11:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:27 smithi134 ceph-mon[31966]: pgmap v342: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T11:11:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:27 smithi189 ceph-mon[36790]: pgmap v342: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T11:11:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:27 smithi193 ceph-mon[37473]: pgmap v342: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-07-26T11:11:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:29 smithi134 ceph-mon[31966]: pgmap v343: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-07-26T11:11:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:29 smithi189 ceph-mon[36790]: pgmap v343: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-07-26T11:11:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:29 smithi193 ceph-mon[37473]: pgmap v343: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-07-26T11:11:31.248 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi134.front.sepia.ceph.com 2024-07-26T11:11:31.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:11:31.249 DEBUG:teuthology.orchestra.run.smithi134:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-07-26T11:11:31.315 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T11:11:31.389 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:31 smithi134 ceph-mon[31966]: pgmap v344: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T11:11:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:31 smithi189 ceph-mon[36790]: pgmap v344: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T11:11:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:31 smithi193 ceph-mon[37473]: pgmap v344: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T11:11:31.799 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:11:33.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:33 smithi134 ceph-mon[31966]: pgmap v345: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T11:11:33.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:33 smithi189 ceph-mon[36790]: pgmap v345: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T11:11:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:33 smithi193 ceph-mon[37473]: pgmap v345: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T11:11:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-07-26T11:11:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-07-26T11:11:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-07-26T11:11:34.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-07-26T11:11:34.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.196+0000 7f221e600640 1 -- 172.21.15.134:0/3981582244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2218108ee0 msgr2=0x7f22181092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:34.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.196+0000 7f221e600640 1 --2- 172.21.15.134:0/3981582244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2218108ee0 0x7f22181092f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f2204009900 tx=0x7f220402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:11:34.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.196+0000 7f221e600640 1 -- 172.21.15.134:0/3981582244 shutdown_connections 2024-07-26T11:11:34.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.196+0000 7f221e600640 1 --2- 172.21.15.134:0/3981582244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2218108ee0 0x7f22181092f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:34.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.196+0000 7f221e600640 1 --2- 172.21.15.134:0/3981582244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2218109830 0x7f2218110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:34.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.196+0000 7f221e600640 1 --2- 172.21.15.134:0/3981582244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22181035a0 0x7f22181039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:34.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.196+0000 7f221e600640 1 -- 172.21.15.134:0/3981582244 >> 172.21.15.134:0/3981582244 conn(0x7f22180ff010 msgr2=0x7f2218101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:34.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.196+0000 7f221e600640 1 -- 172.21.15.134:0/3981582244 shutdown_connections 2024-07-26T11:11:34.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.197+0000 7f221e600640 1 -- 172.21.15.134:0/3981582244 wait complete. 2024-07-26T11:11:34.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.198+0000 7f221e600640 1 Processor -- start 2024-07-26T11:11:34.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.198+0000 7f221e600640 1 -- start start 2024-07-26T11:11:34.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.199+0000 7f221e600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22181035a0 0x7f221819c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:34.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.199+0000 7f221e600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2218108ee0 0x7f221819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:34.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.199+0000 7f221e600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2218109830 0x7f22181a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:34.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.199+0000 7f221e600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f22181134a0 con 0x7f2218109830 2024-07-26T11:11:34.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.199+0000 7f221e600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2218113320 con 0x7f22181035a0 2024-07-26T11:11:34.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.199+0000 7f221e600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2218113620 con 0x7f2218108ee0 2024-07-26T11:11:34.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.199+0000 7f2217400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22181035a0 0x7f221819c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:34.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.199+0000 7f2217400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22181035a0 0x7f221819c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:40824/0 (socket says 172.21.15.134:40824) 2024-07-26T11:11:34.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.199+0000 7f2217400640 1 -- 172.21.15.134:0/2617858866 learned_addr learned my addr 172.21.15.134:0/2617858866 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:11:34.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.199+0000 7f2217e00640 1 --2- 172.21.15.134:0/2617858866 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2218109830 0x7f22181a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:34.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.200+0000 7f2216a00640 1 --2- 172.21.15.134:0/2617858866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2218108ee0 0x7f221819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:34.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.200+0000 7f2217e00640 1 -- 172.21.15.134:0/2617858866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2218108ee0 msgr2=0x7f221819c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:34.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.200+0000 7f2217e00640 1 --2- 172.21.15.134:0/2617858866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2218108ee0 0x7f221819c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:34.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.200+0000 7f2217e00640 1 -- 172.21.15.134:0/2617858866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22181035a0 msgr2=0x7f221819c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:34.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.200+0000 7f2217e00640 1 --2- 172.21.15.134:0/2617858866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22181035a0 0x7f221819c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:34.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.200+0000 7f2217e00640 1 -- 172.21.15.134:0/2617858866 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22181a20a0 con 0x7f2218109830 2024-07-26T11:11:34.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.201+0000 7f2217e00640 1 --2- 172.21.15.134:0/2617858866 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2218109830 0x7f22181a1970 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f221000e940 tx=0x7f221000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:34.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.201+0000 7f220fe00640 1 -- 172.21.15.134:0/2617858866 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2210004d20 con 0x7f2218109830 2024-07-26T11:11:34.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.201+0000 7f220fe00640 1 -- 172.21.15.134:0/2617858866 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2210005020 con 0x7f2218109830 2024-07-26T11:11:34.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.201+0000 7f220fe00640 1 -- 172.21.15.134:0/2617858866 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22100116d0 con 0x7f2218109830 2024-07-26T11:11:34.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.201+0000 7f221e600640 1 -- 172.21.15.134:0/2617858866 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22181a2390 con 0x7f2218109830 2024-07-26T11:11:34.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.201+0000 7f221e600640 1 -- 172.21.15.134:0/2617858866 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22181a8af0 con 0x7f2218109830 2024-07-26T11:11:34.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.204+0000 7f221e600640 1 -- 172.21.15.134:0/2617858866 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22181039b0 con 0x7f2218109830 2024-07-26T11:11:34.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.206+0000 7f220fe00640 1 -- 172.21.15.134:0/2617858866 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f2210005460 con 0x7f2218109830 2024-07-26T11:11:34.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.208+0000 7f220fe00640 1 --2- 172.21.15.134:0/2617858866 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21fc0b3ca0 0x7f21fc0b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:34.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.209+0000 7f220fe00640 1 -- 172.21.15.134:0/2617858866 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 9423+0+0 (secure 0 0 0) 0x7f22100e4140 con 0x7f2218109830 2024-07-26T11:11:34.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.209+0000 7f2217400640 1 --2- 172.21.15.134:0/2617858866 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21fc0b3ca0 0x7f21fc0b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:34.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.210+0000 7f2217400640 1 --2- 172.21.15.134:0/2617858866 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21fc0b3ca0 0x7f21fc0b61b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2208009c90 tx=0x7f2208005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:34.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.213+0000 7f220fe00640 1 -- 172.21.15.134:0/2617858866 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2210016030 con 0x7f2218109830 2024-07-26T11:11:34.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.423+0000 7f221e600640 1 -- 172.21.15.134:0/2617858866 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22181a8e20 con 0x7f2218109830 2024-07-26T11:11:34.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.428+0000 7f220fe00640 1 -- 172.21.15.134:0/2617858866 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+25185 (secure 0 0 0) 0x7f22100aac40 con 0x7f2218109830 2024-07-26T11:11:34.431 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:11:34.431 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":89,"fsid":"105fbe5c-4b3e-11ef-bcb7-c7b262605968","created":"2024-07-26T11:00:05.955103+0000","modified":"2024-07-26T11:11:34.040946+0000","last_up_change":"2024-07-26T11:09:11.485512+0000","last_in_change":"2024-07-26T11:08:52.341356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T11:05:01.849170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T11:10:16.055725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T11:10:21.216611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T11:10:33.983067+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8abb1365-fc3e-4617-a9ac-76bece8cd7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6803","nonce":4183273954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6805","nonce":4183273954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6809","nonce":4183273954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6807","nonce":4183273954}]},"public_addr":"172.21.15.134:6803/4183273954","cluster_addr":"172.21.15.134:6805/4183273954","heartbeat_back_addr":"172.21.15.134:6809/4183273954","heartbeat_front_addr":"172.21.15.134:6807/4183273954","state":["exists","up"]},{"osd":1,"uuid":"6c5a7838-821e-44e1-97e4-655b9bfa9f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6801","nonce":35148049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6803","nonce":35148049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6807","nonce":35148049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6805","nonce":35148049}]},"public_addr":"172.21.15.189:6801/35148049","cluster_addr":"172.21.15.189:6803/35148049","heartbeat_back_addr":"172.21.15.189:6807/35148049","heartbeat_front_addr":"172.21.15.189:6805/35148049","state":["exists","up"]},{"osd":2,"uuid":"8bc42f7f-39fa-4d3b-bc17-21c68f9231fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6801","nonce":897516976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6803","nonce":897516976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6807","nonce":897516976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6805","nonce":897516976}]},"public_addr":"172.21.15.193:6801/897516976","cluster_addr":"172.21.15.193:6803/897516976","heartbeat_back_addr":"172.21.15.193:6807/897516976","heartbeat_front_addr":"172.21.15.193:6805/897516976","state":["exists","up"]},{"osd":3,"uuid":"0473d055-0b79-434f-9658-28269d2b1fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6811","nonce":3457190917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6813","nonce":3457190917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6817","nonce":3457190917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6815","nonce":3457190917}]},"public_addr":"172.21.15.134:6811/3457190917","cluster_addr":"172.21.15.134:6813/3457190917","heartbeat_back_addr":"172.21.15.134:6817/3457190917","heartbeat_front_addr":"172.21.15.134:6815/3457190917","state":["exists","up"]},{"osd":4,"uuid":"1d70eaae-56cc-4cdc-8f83-b88d2121f0ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6809","nonce":1786881347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6811","nonce":1786881347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6815","nonce":1786881347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6813","nonce":1786881347}]},"public_addr":"172.21.15.189:6809/1786881347","cluster_addr":"172.21.15.189:6811/1786881347","heartbeat_back_addr":"172.21.15.189:6815/1786881347","heartbeat_front_addr":"172.21.15.189:6813/1786881347","state":["exists","up"]},{"osd":5,"uuid":"a9b0b4cf-1608-45ed-b64e-afcf545d97fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6809","nonce":1092388797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6811","nonce":1092388797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6815","nonce":1092388797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6813","nonce":1092388797}]},"public_addr":"172.21.15.193:6809/1092388797","cluster_addr":"172.21.15.193:6811/1092388797","heartbeat_back_addr":"172.21.15.193:6815/1092388797","heartbeat_front_addr":"172.21.15.193:6813/1092388797","state":["exists","up"]},{"osd":6,"uuid":"0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6819","nonce":2471505438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6821","nonce":2471505438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6825","nonce":2471505438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6823","nonce":2471505438}]},"public_addr":"172.21.15.134:6819/2471505438","cluster_addr":"172.21.15.134:6821/2471505438","heartbeat_back_addr":"172.21.15.134:6825/2471505438","heartbeat_front_addr":"172.21.15.134:6823/2471505438","state":["exists","up"]},{"osd":7,"uuid":"394764a7-1976-4223-a3c0-5979d941a838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6817","nonce":1382694911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6819","nonce":1382694911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6823","nonce":1382694911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6821","nonce":1382694911}]},"public_addr":"172.21.15.189:6817/1382694911","cluster_addr":"172.21.15.189:6819/1382694911","heartbeat_back_addr":"172.21.15.189:6823/1382694911","heartbeat_front_addr":"172.21.15.189:6821/1382694911","state":["exists","up"]},{"osd":8,"uuid":"1c218082-0b4c-41c8-92b9-453ae8d15a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6817","nonce":896348138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6819","nonce":896348138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6823","nonce":896348138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6821","nonce":896348138}]},"public_addr":"172.21.15.193:6817/896348138","cluster_addr":"172.21.15.193:6819/896348138","heartbeat_back_addr":"172.21.15.193:6823/896348138","heartbeat_front_addr":"172.21.15.193:6821/896348138","state":["exists","up"]},{"osd":9,"uuid":"48f39082-4fa9-4443-b5b7-b141c10d9e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6827","nonce":511237025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6829","nonce":511237025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6833","nonce":511237025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6831","nonce":511237025}]},"public_addr":"172.21.15.134:6827/511237025","cluster_addr":"172.21.15.134:6829/511237025","heartbeat_back_addr":"172.21.15.134:6833/511237025","heartbeat_front_addr":"172.21.15.134:6831/511237025","state":["exists","up"]},{"osd":10,"uuid":"95157766-1856-42f7-aea1-9cf83b77c8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6825","nonce":431740858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6827","nonce":431740858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6831","nonce":431740858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6829","nonce":431740858}]},"public_addr":"172.21.15.189:6825/431740858","cluster_addr":"172.21.15.189:6827/431740858","heartbeat_back_addr":"172.21.15.189:6831/431740858","heartbeat_front_addr":"172.21.15.189:6829/431740858","state":["exists","up"]},{"osd":11,"uuid":"244fadc7-f079-4736-8e7e-c0ce5ae6f43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6825","nonce":1594590579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6827","nonce":1594590579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6831","nonce":1594590579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6829","nonce":1594590579}]},"public_addr":"172.21.15.193:6825/1594590579","cluster_addr":"172.21.15.193:6827/1594590579","heartbeat_back_addr":"172.21.15.193:6831/1594590579","heartbeat_front_addr":"172.21.15.193:6829/1594590579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:11.712203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:35.845013+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:58.165609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:31.929746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:56.941555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:20.688071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:54.681514+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:20.789384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:44.568246+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:18.644310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:45.365351+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:09:09.663805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3790862248":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6800/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:0/2906927791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3068153391":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3056560791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6801/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/1310359922":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/2318666906":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/2901579347":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3404218719":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3598071145":"2024-07-27T11:02:33.711793+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:11:34.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.431+0000 7f221e600640 1 -- 172.21.15.134:0/2617858866 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21fc0b3ca0 msgr2=0x7f21fc0b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:34.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.431+0000 7f221e600640 1 --2- 172.21.15.134:0/2617858866 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21fc0b3ca0 0x7f21fc0b61b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2208009c90 tx=0x7f2208005f90 comp rx=0 tx=0).stop 2024-07-26T11:11:34.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.431+0000 7f221e600640 1 -- 172.21.15.134:0/2617858866 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2218109830 msgr2=0x7f22181a1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:34.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.431+0000 7f221e600640 1 --2- 172.21.15.134:0/2617858866 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2218109830 0x7f22181a1970 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f221000e940 tx=0x7f221000ee00 comp rx=0 tx=0).stop 2024-07-26T11:11:34.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.431+0000 7f221e600640 1 -- 172.21.15.134:0/2617858866 shutdown_connections 2024-07-26T11:11:34.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.431+0000 7f221e600640 1 --2- 172.21.15.134:0/2617858866 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21fc0b3ca0 0x7f21fc0b61b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:34.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.431+0000 7f221e600640 1 --2- 172.21.15.134:0/2617858866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2218108ee0 0x7f221819c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:34.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.431+0000 7f221e600640 1 --2- 172.21.15.134:0/2617858866 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2218109830 0x7f22181a1970 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:34.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.431+0000 7f221e600640 1 --2- 172.21.15.134:0/2617858866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22181035a0 0x7f221819c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:34.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.431+0000 7f221e600640 1 -- 172.21.15.134:0/2617858866 >> 172.21.15.134:0/2617858866 conn(0x7f22180ff010 msgr2=0x7f22181075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:34.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.431+0000 7f221e600640 1 -- 172.21.15.134:0/2617858866 shutdown_connections 2024-07-26T11:11:34.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:34.431+0000 7f221e600640 1 -- 172.21.15.134:0/2617858866 wait complete. 2024-07-26T11:11:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-07-26T11:11:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-07-26T11:11:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-07-26T11:11:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-07-26T11:11:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-07-26T11:11:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]: dispatch 2024-07-26T11:11:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2024-07-26T11:11:34.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-07-26T11:11:35.141 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:35 smithi134 ceph-mon[31966]: pgmap v346: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T11:11:35.403 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-07-26T11:11:35.449 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-07-26T11:11:35.449 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-07-26T11:11:35.449 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-07-26T11:11:35.449 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:35 smithi134 ceph-mon[31966]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T11:11:35.449 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:35 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2617858866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:11:35.450 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:35 smithi189 ceph-mon[36790]: pgmap v346: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T11:11:35.450 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-07-26T11:11:35.450 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-07-26T11:11:35.450 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-07-26T11:11:35.450 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-07-26T11:11:35.450 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:35 smithi189 ceph-mon[36790]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T11:11:35.450 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:35 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2617858866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:11:35.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T11:11:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:35 smithi193 ceph-mon[37473]: pgmap v346: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-07-26T11:11:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-07-26T11:11:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 8]}]': finished 2024-07-26T11:11:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2024-07-26T11:11:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-07-26T11:11:35.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:35 smithi193 ceph-mon[37473]: osdmap e89: 12 total, 12 up, 12 in 2024-07-26T11:11:35.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:35 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2617858866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:11:35.921 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:11:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:36 smithi134 ceph-mon[31966]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T11:11:36.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:36 smithi134 ceph-mon[31966]: pgmap v349: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:36 smithi189 ceph-mon[36790]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T11:11:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:36 smithi189 ceph-mon[36790]: pgmap v349: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:36 smithi193 ceph-mon[37473]: osdmap e90: 12 total, 12 up, 12 in 2024-07-26T11:11:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:36 smithi193 ceph-mon[37473]: pgmap v349: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:38.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.409+0000 7fc18c200640 1 -- 172.21.15.134:0/1631042013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc18410a330 msgr2=0x7fc18410a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:38.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.409+0000 7fc18c200640 1 --2- 172.21.15.134:0/1631042013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc18410a330 0x7fc18410a760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009cd0 tx=0x7fc17c02a8f0 comp rx=0 tx=0).stop 2024-07-26T11:11:38.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.410+0000 7fc18c200640 1 -- 172.21.15.134:0/1631042013 shutdown_connections 2024-07-26T11:11:38.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.410+0000 7fc18c200640 1 --2- 172.21.15.134:0/1631042013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc18410a330 0x7fc18410a760 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:38.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.410+0000 7fc18c200640 1 --2- 172.21.15.134:0/1631042013 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc18410f1a0 0x7fc184109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:38.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.410+0000 7fc18c200640 1 --2- 172.21.15.134:0/1631042013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc18406a9c0 0x7fc18410eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:38.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.410+0000 7fc18c200640 1 -- 172.21.15.134:0/1631042013 >> 172.21.15.134:0/1631042013 conn(0x7fc184070d20 msgr2=0x7fc184073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:38.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.410+0000 7fc18c200640 1 -- 172.21.15.134:0/1631042013 shutdown_connections 2024-07-26T11:11:38.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.410+0000 7fc18c200640 1 -- 172.21.15.134:0/1631042013 wait complete. 2024-07-26T11:11:38.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.411+0000 7fc18c200640 1 Processor -- start 2024-07-26T11:11:38.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.411+0000 7fc18c200640 1 -- start start 2024-07-26T11:11:38.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.411+0000 7fc18c200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc18406a9c0 0x7fc184071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:38.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.412+0000 7fc18c200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc18410a330 0x7fc184071eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:38.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.412+0000 7fc18c200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc18410f1a0 0x7fc1840723f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:38.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.412+0000 7fc18c200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc18411abe0 con 0x7fc18410a330 2024-07-26T11:11:38.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.412+0000 7fc18c200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc18411aa60 con 0x7fc18410f1a0 2024-07-26T11:11:38.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.412+0000 7fc18c200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc18411ad60 con 0x7fc18406a9c0 2024-07-26T11:11:38.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.412+0000 7fc188e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc18410a330 0x7fc184071eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:38.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.412+0000 7fc189800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc18406a9c0 0x7fc184071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:38.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.412+0000 7fc189800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc18406a9c0 0x7fc184071970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52370/0 (socket says 172.21.15.134:52370) 2024-07-26T11:11:38.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.412+0000 7fc189800640 1 -- 172.21.15.134:0/2079782012 learned_addr learned my addr 172.21.15.134:0/2079782012 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:11:38.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.412+0000 7fc18a200640 1 --2- 172.21.15.134:0/2079782012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc18410f1a0 0x7fc1840723f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:38.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.412+0000 7fc189800640 1 -- 172.21.15.134:0/2079782012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc18410f1a0 msgr2=0x7fc1840723f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:38.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.412+0000 7fc189800640 1 --2- 172.21.15.134:0/2079782012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc18410f1a0 0x7fc1840723f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:38.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.412+0000 7fc189800640 1 -- 172.21.15.134:0/2079782012 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc18410a330 msgr2=0x7fc184071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:38.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.412+0000 7fc189800640 1 --2- 172.21.15.134:0/2079782012 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc18410a330 0x7fc184071eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:38.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.412+0000 7fc189800640 1 -- 172.21.15.134:0/2079782012 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1841b0940 con 0x7fc18406a9c0 2024-07-26T11:11:38.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.413+0000 7fc189800640 1 --2- 172.21.15.134:0/2079782012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc18406a9c0 0x7fc184071970 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc17800e9f0 tx=0x7fc17800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:38.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.413+0000 7fc182000640 1 -- 172.21.15.134:0/2079782012 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17800cd80 con 0x7fc18406a9c0 2024-07-26T11:11:38.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.413+0000 7fc18c200640 1 -- 172.21.15.134:0/2079782012 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1841b0c30 con 0x7fc18406a9c0 2024-07-26T11:11:38.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.413+0000 7fc182000640 1 -- 172.21.15.134:0/2079782012 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1780108e0 con 0x7fc18406a9c0 2024-07-26T11:11:38.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.413+0000 7fc182000640 1 -- 172.21.15.134:0/2079782012 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178018980 con 0x7fc18406a9c0 2024-07-26T11:11:38.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.413+0000 7fc18c200640 1 -- 172.21.15.134:0/2079782012 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1841b1170 con 0x7fc18406a9c0 2024-07-26T11:11:38.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.415+0000 7fc18c200640 1 -- 172.21.15.134:0/2079782012 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc18410a760 con 0x7fc18406a9c0 2024-07-26T11:11:38.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.417+0000 7fc182000640 1 -- 172.21.15.134:0/2079782012 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fc178010430 con 0x7fc18406a9c0 2024-07-26T11:11:38.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.418+0000 7fc182000640 1 --2- 172.21.15.134:0/2079782012 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc16c0b3c20 0x7fc16c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:38.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.418+0000 7fc182000640 1 -- 172.21.15.134:0/2079782012 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc1780e4510 con 0x7fc18406a9c0 2024-07-26T11:11:38.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.419+0000 7fc188e00640 1 --2- 172.21.15.134:0/2079782012 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc16c0b3c20 0x7fc16c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:38.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.419+0000 7fc188e00640 1 --2- 172.21.15.134:0/2079782012 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc16c0b3c20 0x7fc16c0b6130 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc1700095f0 tx=0x7fc170005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:38.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.422+0000 7fc182000640 1 -- 172.21.15.134:0/2079782012 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc1780ab660 con 0x7fc18406a9c0 2024-07-26T11:11:38.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.622+0000 7fc18c200640 1 -- 172.21.15.134:0/2079782012 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc1841b1530 con 0x7fc18406a9c0 2024-07-26T11:11:38.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.624+0000 7fc182000640 1 -- 172.21.15.134:0/2079782012 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25185 (secure 0 0 0) 0x7fc1780aaf60 con 0x7fc18406a9c0 2024-07-26T11:11:38.627 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:11:38.627 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":90,"fsid":"105fbe5c-4b3e-11ef-bcb7-c7b262605968","created":"2024-07-26T11:00:05.955103+0000","modified":"2024-07-26T11:11:35.149490+0000","last_up_change":"2024-07-26T11:09:11.485512+0000","last_in_change":"2024-07-26T11:08:52.341356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T11:05:01.849170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T11:10:16.055725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T11:10:21.216611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T11:10:33.983067+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8abb1365-fc3e-4617-a9ac-76bece8cd7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6803","nonce":4183273954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6805","nonce":4183273954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6809","nonce":4183273954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6807","nonce":4183273954}]},"public_addr":"172.21.15.134:6803/4183273954","cluster_addr":"172.21.15.134:6805/4183273954","heartbeat_back_addr":"172.21.15.134:6809/4183273954","heartbeat_front_addr":"172.21.15.134:6807/4183273954","state":["exists","up"]},{"osd":1,"uuid":"6c5a7838-821e-44e1-97e4-655b9bfa9f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6801","nonce":35148049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6803","nonce":35148049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6807","nonce":35148049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6805","nonce":35148049}]},"public_addr":"172.21.15.189:6801/35148049","cluster_addr":"172.21.15.189:6803/35148049","heartbeat_back_addr":"172.21.15.189:6807/35148049","heartbeat_front_addr":"172.21.15.189:6805/35148049","state":["exists","up"]},{"osd":2,"uuid":"8bc42f7f-39fa-4d3b-bc17-21c68f9231fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6801","nonce":897516976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6803","nonce":897516976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6807","nonce":897516976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6805","nonce":897516976}]},"public_addr":"172.21.15.193:6801/897516976","cluster_addr":"172.21.15.193:6803/897516976","heartbeat_back_addr":"172.21.15.193:6807/897516976","heartbeat_front_addr":"172.21.15.193:6805/897516976","state":["exists","up"]},{"osd":3,"uuid":"0473d055-0b79-434f-9658-28269d2b1fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6811","nonce":3457190917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6813","nonce":3457190917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6817","nonce":3457190917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6815","nonce":3457190917}]},"public_addr":"172.21.15.134:6811/3457190917","cluster_addr":"172.21.15.134:6813/3457190917","heartbeat_back_addr":"172.21.15.134:6817/3457190917","heartbeat_front_addr":"172.21.15.134:6815/3457190917","state":["exists","up"]},{"osd":4,"uuid":"1d70eaae-56cc-4cdc-8f83-b88d2121f0ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6809","nonce":1786881347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6811","nonce":1786881347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6815","nonce":1786881347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6813","nonce":1786881347}]},"public_addr":"172.21.15.189:6809/1786881347","cluster_addr":"172.21.15.189:6811/1786881347","heartbeat_back_addr":"172.21.15.189:6815/1786881347","heartbeat_front_addr":"172.21.15.189:6813/1786881347","state":["exists","up"]},{"osd":5,"uuid":"a9b0b4cf-1608-45ed-b64e-afcf545d97fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6809","nonce":1092388797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6811","nonce":1092388797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6815","nonce":1092388797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6813","nonce":1092388797}]},"public_addr":"172.21.15.193:6809/1092388797","cluster_addr":"172.21.15.193:6811/1092388797","heartbeat_back_addr":"172.21.15.193:6815/1092388797","heartbeat_front_addr":"172.21.15.193:6813/1092388797","state":["exists","up"]},{"osd":6,"uuid":"0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6819","nonce":2471505438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6821","nonce":2471505438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6825","nonce":2471505438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6823","nonce":2471505438}]},"public_addr":"172.21.15.134:6819/2471505438","cluster_addr":"172.21.15.134:6821/2471505438","heartbeat_back_addr":"172.21.15.134:6825/2471505438","heartbeat_front_addr":"172.21.15.134:6823/2471505438","state":["exists","up"]},{"osd":7,"uuid":"394764a7-1976-4223-a3c0-5979d941a838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6817","nonce":1382694911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6819","nonce":1382694911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6823","nonce":1382694911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6821","nonce":1382694911}]},"public_addr":"172.21.15.189:6817/1382694911","cluster_addr":"172.21.15.189:6819/1382694911","heartbeat_back_addr":"172.21.15.189:6823/1382694911","heartbeat_front_addr":"172.21.15.189:6821/1382694911","state":["exists","up"]},{"osd":8,"uuid":"1c218082-0b4c-41c8-92b9-453ae8d15a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6817","nonce":896348138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6819","nonce":896348138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6823","nonce":896348138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6821","nonce":896348138}]},"public_addr":"172.21.15.193:6817/896348138","cluster_addr":"172.21.15.193:6819/896348138","heartbeat_back_addr":"172.21.15.193:6823/896348138","heartbeat_front_addr":"172.21.15.193:6821/896348138","state":["exists","up"]},{"osd":9,"uuid":"48f39082-4fa9-4443-b5b7-b141c10d9e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6827","nonce":511237025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6829","nonce":511237025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6833","nonce":511237025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6831","nonce":511237025}]},"public_addr":"172.21.15.134:6827/511237025","cluster_addr":"172.21.15.134:6829/511237025","heartbeat_back_addr":"172.21.15.134:6833/511237025","heartbeat_front_addr":"172.21.15.134:6831/511237025","state":["exists","up"]},{"osd":10,"uuid":"95157766-1856-42f7-aea1-9cf83b77c8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6825","nonce":431740858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6827","nonce":431740858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6831","nonce":431740858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6829","nonce":431740858}]},"public_addr":"172.21.15.189:6825/431740858","cluster_addr":"172.21.15.189:6827/431740858","heartbeat_back_addr":"172.21.15.189:6831/431740858","heartbeat_front_addr":"172.21.15.189:6829/431740858","state":["exists","up"]},{"osd":11,"uuid":"244fadc7-f079-4736-8e7e-c0ce5ae6f43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6825","nonce":1594590579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6827","nonce":1594590579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6831","nonce":1594590579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6829","nonce":1594590579}]},"public_addr":"172.21.15.193:6825/1594590579","cluster_addr":"172.21.15.193:6827/1594590579","heartbeat_back_addr":"172.21.15.193:6831/1594590579","heartbeat_front_addr":"172.21.15.193:6829/1594590579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:11.712203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:35.845013+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:58.165609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:31.929746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:56.941555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:20.688071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:54.681514+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:20.789384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:44.568246+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:18.644310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:45.365351+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:09:09.663805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3790862248":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6800/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:0/2906927791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3068153391":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3056560791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6801/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/1310359922":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/2318666906":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/2901579347":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3404218719":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3598071145":"2024-07-27T11:02:33.711793+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:11:38.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.627+0000 7fc18c200640 1 -- 172.21.15.134:0/2079782012 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc16c0b3c20 msgr2=0x7fc16c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:38.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.627+0000 7fc18c200640 1 --2- 172.21.15.134:0/2079782012 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc16c0b3c20 0x7fc16c0b6130 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc1700095f0 tx=0x7fc170005f90 comp rx=0 tx=0).stop 2024-07-26T11:11:38.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.627+0000 7fc18c200640 1 -- 172.21.15.134:0/2079782012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc18406a9c0 msgr2=0x7fc184071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:38.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.627+0000 7fc18c200640 1 --2- 172.21.15.134:0/2079782012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc18406a9c0 0x7fc184071970 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc17800e9f0 tx=0x7fc17800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:11:38.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.628+0000 7fc18c200640 1 -- 172.21.15.134:0/2079782012 shutdown_connections 2024-07-26T11:11:38.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.628+0000 7fc18c200640 1 --2- 172.21.15.134:0/2079782012 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc16c0b3c20 0x7fc16c0b6130 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:38.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.628+0000 7fc18c200640 1 --2- 172.21.15.134:0/2079782012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc18410f1a0 0x7fc1840723f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:38.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.628+0000 7fc18c200640 1 --2- 172.21.15.134:0/2079782012 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc18410a330 0x7fc184071eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:38.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.628+0000 7fc18c200640 1 --2- 172.21.15.134:0/2079782012 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc18406a9c0 0x7fc184071970 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:38.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.628+0000 7fc18c200640 1 -- 172.21.15.134:0/2079782012 >> 172.21.15.134:0/2079782012 conn(0x7fc184070d20 msgr2=0x7fc184073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:38.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.628+0000 7fc18c200640 1 -- 172.21.15.134:0/2079782012 shutdown_connections 2024-07-26T11:11:38.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:38.628+0000 7fc18c200640 1 -- 172.21.15.134:0/2079782012 wait complete. 2024-07-26T11:11:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:38 smithi193 ceph-mon[37473]: pgmap v350: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:38 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2079782012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:11:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:38 smithi134 ceph-mon[31966]: pgmap v350: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:38 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2079782012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:11:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:38 smithi189 ceph-mon[36790]: pgmap v350: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:38 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2079782012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:11:39.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-07-26T11:11:40.043 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:11:41.256 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:40 smithi134 ceph-mon[31966]: pgmap v351: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:40 smithi193 ceph-mon[37473]: pgmap v351: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:40 smithi189 ceph-mon[36790]: pgmap v351: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:42.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.571+0000 7f9968a00640 1 -- 172.21.15.134:0/2978663076 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99641055d0 msgr2=0x7f99641059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:42.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.571+0000 7f9968a00640 1 --2- 172.21.15.134:0/2978663076 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99641055d0 0x7f99641059e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9954009ca0 tx=0x7f995402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:11:42.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.572+0000 7f9968a00640 1 -- 172.21.15.134:0/2978663076 shutdown_connections 2024-07-26T11:11:42.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.572+0000 7f9968a00640 1 --2- 172.21.15.134:0/2978663076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f996410af10 0x7f996410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:42.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.572+0000 7f9968a00640 1 --2- 172.21.15.134:0/2978663076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f996410b860 0x7f9964112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:42.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.572+0000 7f9968a00640 1 --2- 172.21.15.134:0/2978663076 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99641055d0 0x7f99641059e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:42.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.572+0000 7f9968a00640 1 -- 172.21.15.134:0/2978663076 >> 172.21.15.134:0/2978663076 conn(0x7f9964101040 msgr2=0x7f9964103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:42.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.572+0000 7f9968a00640 1 -- 172.21.15.134:0/2978663076 shutdown_connections 2024-07-26T11:11:42.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.572+0000 7f9968a00640 1 -- 172.21.15.134:0/2978663076 wait complete. 2024-07-26T11:11:42.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.573+0000 7f9968a00640 1 Processor -- start 2024-07-26T11:11:42.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.573+0000 7f9968a00640 1 -- start start 2024-07-26T11:11:42.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.573+0000 7f9968a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99641055d0 0x7f996419bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:42.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.573+0000 7f9968a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f996410af10 0x7f996419c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:42.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.573+0000 7f9968a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f996410b860 0x7f99641a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:42.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.573+0000 7f9968a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9964115510 con 0x7f996410af10 2024-07-26T11:11:42.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.573+0000 7f9968a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9964115390 con 0x7f996410b860 2024-07-26T11:11:42.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.573+0000 7f9968a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9964115690 con 0x7f99641055d0 2024-07-26T11:11:42.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.574+0000 7f9962a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f996410af10 0x7f996419c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:42.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.574+0000 7f9963e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f996410b860 0x7f99641a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:42.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.574+0000 7f9962a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f996410af10 0x7f996419c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:45866/0 (socket says 172.21.15.134:45866) 2024-07-26T11:11:42.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.574+0000 7f9962a00640 1 -- 172.21.15.134:0/1969991789 learned_addr learned my addr 172.21.15.134:0/1969991789 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:11:42.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.574+0000 7f9963400640 1 --2- 172.21.15.134:0/1969991789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99641055d0 0x7f996419bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:42.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.574+0000 7f9963e00640 1 -- 172.21.15.134:0/1969991789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99641055d0 msgr2=0x7f996419bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:42.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.574+0000 7f9963e00640 1 --2- 172.21.15.134:0/1969991789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99641055d0 0x7f996419bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:42.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.574+0000 7f9963e00640 1 -- 172.21.15.134:0/1969991789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f996410af10 msgr2=0x7f996419c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:42.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.574+0000 7f9963e00640 1 --2- 172.21.15.134:0/1969991789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f996410af10 0x7f996419c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:42.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.574+0000 7f9963e00640 1 -- 172.21.15.134:0/1969991789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99641a1e90 con 0x7f996410b860 2024-07-26T11:11:42.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.575+0000 7f9963e00640 1 --2- 172.21.15.134:0/1969991789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f996410b860 0x7f99641a1760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f995c00e970 tx=0x7f995c00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:42.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.575+0000 7f995be00640 1 -- 172.21.15.134:0/1969991789 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c00ccc0 con 0x7f996410b860 2024-07-26T11:11:42.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.575+0000 7f9968a00640 1 -- 172.21.15.134:0/1969991789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99641a2180 con 0x7f996410b860 2024-07-26T11:11:42.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.575+0000 7f995be00640 1 -- 172.21.15.134:0/1969991789 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f995c005020 con 0x7f996410b860 2024-07-26T11:11:42.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.575+0000 7f995be00640 1 -- 172.21.15.134:0/1969991789 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c004a00 con 0x7f996410b860 2024-07-26T11:11:42.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.575+0000 7f9968a00640 1 -- 172.21.15.134:0/1969991789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99641a88e0 con 0x7f996410b860 2024-07-26T11:11:42.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.576+0000 7f9968a00640 1 -- 172.21.15.134:0/1969991789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99641059e0 con 0x7f996410b860 2024-07-26T11:11:42.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.579+0000 7f995be00640 1 -- 172.21.15.134:0/1969991789 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f995c004ba0 con 0x7f996410b860 2024-07-26T11:11:42.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.580+0000 7f995be00640 1 --2- 172.21.15.134:0/1969991789 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f99480b3c20 0x7f99480b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:42.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.580+0000 7f995be00640 1 -- 172.21.15.134:0/1969991789 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f995c0e4140 con 0x7f996410b860 2024-07-26T11:11:42.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.580+0000 7f9963400640 1 --2- 172.21.15.134:0/1969991789 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f99480b3c20 0x7f99480b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:42.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.581+0000 7f9963400640 1 --2- 172.21.15.134:0/1969991789 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f99480b3c20 0x7f99480b6130 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9954005e30 tx=0x7f9954009360 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:42.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.582+0000 7f995be00640 1 -- 172.21.15.134:0/1969991789 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f995c0ab320 con 0x7f996410b860 2024-07-26T11:11:42.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.821+0000 7f9968a00640 1 -- 172.21.15.134:0/1969991789 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f996419d050 con 0x7f996410b860 2024-07-26T11:11:42.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.978+0000 7f995be00640 1 -- 172.21.15.134:0/1969991789 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f996419d050 con 0x7f996410b860 2024-07-26T11:11:42.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.981+0000 7f9968a00640 1 -- 172.21.15.134:0/1969991789 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f99480b3c20 msgr2=0x7f99480b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:42.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.981+0000 7f9968a00640 1 --2- 172.21.15.134:0/1969991789 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f99480b3c20 0x7f99480b6130 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9954005e30 tx=0x7f9954009360 comp rx=0 tx=0).stop 2024-07-26T11:11:42.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.981+0000 7f9968a00640 1 -- 172.21.15.134:0/1969991789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f996410b860 msgr2=0x7f99641a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:42.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.981+0000 7f9968a00640 1 --2- 172.21.15.134:0/1969991789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f996410b860 0x7f99641a1760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f995c00e970 tx=0x7f995c00ee30 comp rx=0 tx=0).stop 2024-07-26T11:11:42.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.981+0000 7f9968a00640 1 -- 172.21.15.134:0/1969991789 shutdown_connections 2024-07-26T11:11:42.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.981+0000 7f9968a00640 1 --2- 172.21.15.134:0/1969991789 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f99480b3c20 0x7f99480b6130 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:42.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.981+0000 7f9968a00640 1 --2- 172.21.15.134:0/1969991789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f996410b860 0x7f99641a1760 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:42.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.981+0000 7f9968a00640 1 --2- 172.21.15.134:0/1969991789 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f996410af10 0x7f996419c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:42.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.981+0000 7f9968a00640 1 --2- 172.21.15.134:0/1969991789 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99641055d0 0x7f996419bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:42.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.981+0000 7f9968a00640 1 -- 172.21.15.134:0/1969991789 >> 172.21.15.134:0/1969991789 conn(0x7f9964101040 msgr2=0x7f9964103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:42.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.982+0000 7f9968a00640 1 -- 172.21.15.134:0/1969991789 shutdown_connections 2024-07-26T11:11:42.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:42.982+0000 7f9968a00640 1 -- 172.21.15.134:0/1969991789 wait complete. 2024-07-26T11:11:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:42 smithi134 ceph-mon[31966]: pgmap v352: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:43.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:42 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1969991789' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T11:11:43.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:42 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T11:11:43.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:42 smithi189 ceph-mon[36790]: pgmap v352: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:43.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:42 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1969991789' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T11:11:43.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:42 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T11:11:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:42 smithi193 ceph-mon[37473]: pgmap v352: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:42 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1969991789' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T11:11:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:42 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-07-26T11:11:43.925 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-07-26T11:11:43.925 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-07-26T11:11:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:43 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T11:11:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:43 smithi193 ceph-mon[37473]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-07-26T11:11:44.377 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:11:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:43 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T11:11:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:43 smithi134 ceph-mon[31966]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-07-26T11:11:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:43 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-07-26T11:11:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:43 smithi189 ceph-mon[36790]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-07-26T11:11:45.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:44 smithi134 ceph-mon[31966]: pgmap v353: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:44 smithi193 ceph-mon[37473]: pgmap v353: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:44 smithi189 ceph-mon[36790]: pgmap v353: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:46.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.655+0000 7f1af9000640 1 -- 172.21.15.134:0/708014395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af4109670 msgr2=0x7f1af410fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.655+0000 7f1af9000640 1 --2- 172.21.15.134:0/708014395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af4109670 0x7f1af410fea0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1aec009860 tx=0x7f1aec02a340 comp rx=0 tx=0).stop 2024-07-26T11:11:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.655+0000 7f1af9000640 1 -- 172.21.15.134:0/708014395 shutdown_connections 2024-07-26T11:11:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.655+0000 7f1af9000640 1 --2- 172.21.15.134:0/708014395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af4109670 0x7f1af410fea0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.655+0000 7f1af9000640 1 --2- 172.21.15.134:0/708014395 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af4108d20 0x7f1af4109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.655+0000 7f1af9000640 1 --2- 172.21.15.134:0/708014395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af41033e0 0x7f1af41037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.655+0000 7f1af9000640 1 -- 172.21.15.134:0/708014395 >> 172.21.15.134:0/708014395 conn(0x7f1af40fee70 msgr2=0x7f1af41012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.655+0000 7f1af9000640 1 -- 172.21.15.134:0/708014395 shutdown_connections 2024-07-26T11:11:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.655+0000 7f1af9000640 1 -- 172.21.15.134:0/708014395 wait complete. 2024-07-26T11:11:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.656+0000 7f1af9000640 1 Processor -- start 2024-07-26T11:11:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.657+0000 7f1af9000640 1 -- start start 2024-07-26T11:11:46.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.657+0000 7f1af9000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af41033e0 0x7f1af419bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:46.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.657+0000 7f1af9000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af4108d20 0x7f1af419c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:46.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.657+0000 7f1af9000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af4109670 0x7f1af41a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:46.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.657+0000 7f1af9000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1af4113320 con 0x7f1af41033e0 2024-07-26T11:11:46.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.657+0000 7f1af9000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1af41131a0 con 0x7f1af4108d20 2024-07-26T11:11:46.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.657+0000 7f1af9000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1af41134a0 con 0x7f1af4109670 2024-07-26T11:11:46.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.657+0000 7f1af3400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af41033e0 0x7f1af419bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:46.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.657+0000 7f1af3400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af41033e0 0x7f1af419bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:45890/0 (socket says 172.21.15.134:45890) 2024-07-26T11:11:46.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.657+0000 7f1af3400640 1 -- 172.21.15.134:0/1066533373 learned_addr learned my addr 172.21.15.134:0/1066533373 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:11:46.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.657+0000 7f1af3e00640 1 --2- 172.21.15.134:0/1066533373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af4109670 0x7f1af41a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:46.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.657+0000 7f1af2a00640 1 --2- 172.21.15.134:0/1066533373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af4108d20 0x7f1af419c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:46.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.658+0000 7f1af2a00640 1 -- 172.21.15.134:0/1066533373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af4109670 msgr2=0x7f1af41a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:46.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.658+0000 7f1af2a00640 1 --2- 172.21.15.134:0/1066533373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af4109670 0x7f1af41a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:46.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.658+0000 7f1af2a00640 1 -- 172.21.15.134:0/1066533373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af41033e0 msgr2=0x7f1af419bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:46.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.658+0000 7f1af2a00640 1 --2- 172.21.15.134:0/1066533373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af41033e0 0x7f1af419bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:46.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.658+0000 7f1af2a00640 1 -- 172.21.15.134:0/1066533373 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af41a1e60 con 0x7f1af4108d20 2024-07-26T11:11:46.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.658+0000 7f1af2a00640 1 --2- 172.21.15.134:0/1066533373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af4108d20 0x7f1af419c490 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1af406b130 tx=0x7f1adc00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:46.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.659+0000 7f1aebe00640 1 -- 172.21.15.134:0/1066533373 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1adc00ccd0 con 0x7f1af4108d20 2024-07-26T11:11:46.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.659+0000 7f1aebe00640 1 -- 172.21.15.134:0/1066533373 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1adc0035f0 con 0x7f1af4108d20 2024-07-26T11:11:46.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.659+0000 7f1af9000640 1 -- 172.21.15.134:0/1066533373 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1af41a2150 con 0x7f1af4108d20 2024-07-26T11:11:46.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.659+0000 7f1aebe00640 1 -- 172.21.15.134:0/1066533373 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1adc010690 con 0x7f1af4108d20 2024-07-26T11:11:46.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.659+0000 7f1af9000640 1 -- 172.21.15.134:0/1066533373 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1af41a88b0 con 0x7f1af4108d20 2024-07-26T11:11:46.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.660+0000 7f1af9000640 1 -- 172.21.15.134:0/1066533373 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1af41037f0 con 0x7f1af4108d20 2024-07-26T11:11:46.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.663+0000 7f1aebe00640 1 -- 172.21.15.134:0/1066533373 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f1adc003a80 con 0x7f1af4108d20 2024-07-26T11:11:46.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.665+0000 7f1aebe00640 1 --2- 172.21.15.134:0/1066533373 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1ad80b3ca0 0x7f1ad80b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:46.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.665+0000 7f1aebe00640 1 -- 172.21.15.134:0/1066533373 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f1adc0e4310 con 0x7f1af4108d20 2024-07-26T11:11:46.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.665+0000 7f1af3400640 1 --2- 172.21.15.134:0/1066533373 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1ad80b3ca0 0x7f1ad80b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:46.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.666+0000 7f1aebe00640 1 -- 172.21.15.134:0/1066533373 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1adc009510 con 0x7f1af4108d20 2024-07-26T11:11:46.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.666+0000 7f1af3400640 1 --2- 172.21.15.134:0/1066533373 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1ad80b3ca0 0x7f1ad80b61b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4009fd0 tx=0x7f1ae4009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:46.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:46.906+0000 7f1af9000640 1 -- 172.21.15.134:0/1066533373 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f1af41a8c00 con 0x7f1af4108d20 2024-07-26T11:11:47.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:47.015+0000 7f1aebe00640 1 -- 172.21.15.134:0/1066533373 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f1adc0ab4f0 con 0x7f1af4108d20 2024-07-26T11:11:47.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:47.019+0000 7f1af9000640 1 -- 172.21.15.134:0/1066533373 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1ad80b3ca0 msgr2=0x7f1ad80b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:47.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:47.019+0000 7f1af9000640 1 --2- 172.21.15.134:0/1066533373 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1ad80b3ca0 0x7f1ad80b61b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4009fd0 tx=0x7f1ae4009500 comp rx=0 tx=0).stop 2024-07-26T11:11:47.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:47.019+0000 7f1af9000640 1 -- 172.21.15.134:0/1066533373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af4108d20 msgr2=0x7f1af419c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:47.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:47.019+0000 7f1af9000640 1 --2- 172.21.15.134:0/1066533373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af4108d20 0x7f1af419c490 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1af406b130 tx=0x7f1adc00ee50 comp rx=0 tx=0).stop 2024-07-26T11:11:47.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:47.019+0000 7f1af9000640 1 -- 172.21.15.134:0/1066533373 shutdown_connections 2024-07-26T11:11:47.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:47.019+0000 7f1af9000640 1 --2- 172.21.15.134:0/1066533373 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1ad80b3ca0 0x7f1ad80b61b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:47.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:47.019+0000 7f1af9000640 1 --2- 172.21.15.134:0/1066533373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1af4109670 0x7f1af41a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:47.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:47.019+0000 7f1af9000640 1 --2- 172.21.15.134:0/1066533373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af4108d20 0x7f1af419c490 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:47.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:47.019+0000 7f1af9000640 1 --2- 172.21.15.134:0/1066533373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1af41033e0 0x7f1af419bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:47.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:47.019+0000 7f1af9000640 1 -- 172.21.15.134:0/1066533373 >> 172.21.15.134:0/1066533373 conn(0x7f1af40fee70 msgr2=0x7f1af4107370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:47.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:47.019+0000 7f1af9000640 1 -- 172.21.15.134:0/1066533373 shutdown_connections 2024-07-26T11:11:47.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:47.019+0000 7f1af9000640 1 -- 172.21.15.134:0/1066533373 wait complete. 2024-07-26T11:11:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:47 smithi134 ceph-mon[31966]: pgmap v354: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:47 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1066533373' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T11:11:47.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:47 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T11:11:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:47 smithi193 ceph-mon[37473]: pgmap v354: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:47 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1066533373' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T11:11:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:47 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T11:11:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:47 smithi189 ceph-mon[36790]: pgmap v354: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-07-26T11:11:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:47 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1066533373' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T11:11:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:47 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-07-26T11:11:47.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-07-26T11:11:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:48 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T11:11:48.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:48 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-26T11:11:48.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:48 smithi134 ceph-mon[31966]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T11:11:48.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:48 smithi134 ceph-mon[31966]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-07-26T11:11:48.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:48 smithi134 ceph-mon[31966]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2024-07-26T11:11:48.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:48 smithi134 ceph-mon[31966]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-07-26T11:11:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:48 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T11:11:48.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:48 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-26T11:11:48.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:48 smithi193 ceph-mon[37473]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T11:11:48.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:48 smithi193 ceph-mon[37473]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-07-26T11:11:48.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:48 smithi193 ceph-mon[37473]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2024-07-26T11:11:48.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:48 smithi193 ceph-mon[37473]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-07-26T11:11:48.424 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:11:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:48 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-07-26T11:11:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:48 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-07-26T11:11:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:48 smithi189 ceph-mon[36790]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-07-26T11:11:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:48 smithi189 ceph-mon[36790]: fsmap cephfs:1 {0=a=up:active} 11 up:standby 2024-07-26T11:11:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:48 smithi189 ceph-mon[36790]: fsmap cephfs:2 {0=a=up:active,1=b=up:creating} 10 up:standby 2024-07-26T11:11:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:48 smithi189 ceph-mon[36790]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-07-26T11:11:49.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:49 smithi134 ceph-mon[31966]: pgmap v355: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 1 op/s 2024-07-26T11:11:49.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:49 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] up:active 2024-07-26T11:11:49.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:49 smithi134 ceph-mon[31966]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T11:11:49.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:49 smithi134 ceph-mon[31966]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2024-07-26T11:11:49.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:49 smithi134 ceph-mon[31966]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2024-07-26T11:11:49.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:49 smithi134 ceph-mon[31966]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-07-26T11:11:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:49 smithi189 ceph-mon[36790]: pgmap v355: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 1 op/s 2024-07-26T11:11:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:49 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] up:active 2024-07-26T11:11:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:49 smithi189 ceph-mon[36790]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T11:11:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:49 smithi189 ceph-mon[36790]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2024-07-26T11:11:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:49 smithi189 ceph-mon[36790]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2024-07-26T11:11:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:49 smithi189 ceph-mon[36790]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-07-26T11:11:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:49 smithi193 ceph-mon[37473]: pgmap v355: 129 pgs: 129 active+clean; 581 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 1 op/s 2024-07-26T11:11:49.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:49 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] up:active 2024-07-26T11:11:49.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:49 smithi193 ceph-mon[37473]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-07-26T11:11:49.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:49 smithi193 ceph-mon[37473]: fsmap cephfs:2 {0=a=up:active,1=b=up:active} 10 up:standby 2024-07-26T11:11:49.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:49 smithi193 ceph-mon[37473]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2024-07-26T11:11:49.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:49 smithi193 ceph-mon[37473]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-07-26T11:11:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:50 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] up:active 2024-07-26T11:11:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:50 smithi134 ceph-mon[31966]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-26T11:11:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:50 smithi134 ceph-mon[31966]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2024-07-26T11:11:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:50 smithi134 ceph-mon[31966]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-07-26T11:11:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:50 smithi134 ceph-mon[31966]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-07-26T11:11:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:50 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] up:active 2024-07-26T11:11:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:50 smithi189 ceph-mon[36790]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-26T11:11:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:50 smithi189 ceph-mon[36790]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2024-07-26T11:11:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:50 smithi189 ceph-mon[36790]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-07-26T11:11:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:50 smithi189 ceph-mon[36790]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-07-26T11:11:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:50 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] up:active 2024-07-26T11:11:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:50 smithi193 ceph-mon[37473]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-07-26T11:11:50.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:50 smithi193 ceph-mon[37473]: fsmap cephfs:3 {0=a=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2024-07-26T11:11:50.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:50 smithi193 ceph-mon[37473]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-07-26T11:11:50.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:50 smithi193 ceph-mon[37473]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-07-26T11:11:50.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.746+0000 7f6b82800640 1 -- 172.21.15.134:0/3338036558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b7c10f330 msgr2=0x7f6b7c10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:50.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.746+0000 7f6b82800640 1 --2- 172.21.15.134:0/3338036558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b7c10f330 0x7f6b7c10f740 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6b68009860 tx=0x7f6b6802a340 comp rx=0 tx=0).stop 2024-07-26T11:11:50.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.746+0000 7f6b82800640 1 -- 172.21.15.134:0/3338036558 shutdown_connections 2024-07-26T11:11:50.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.746+0000 7f6b82800640 1 --2- 172.21.15.134:0/3338036558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b7c10f330 0x7f6b7c10f740 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:50.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.746+0000 7f6b82800640 1 --2- 172.21.15.134:0/3338036558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b7c10fc80 0x7f6b7c1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:50.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.746+0000 7f6b82800640 1 --2- 172.21.15.134:0/3338036558 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b7c1099f0 0x7f6b7c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:50.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.746+0000 7f6b82800640 1 -- 172.21.15.134:0/3338036558 >> 172.21.15.134:0/3338036558 conn(0x7f6b7c078ff0 msgr2=0x7f6b7c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:50.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.747+0000 7f6b82800640 1 -- 172.21.15.134:0/3338036558 shutdown_connections 2024-07-26T11:11:50.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.747+0000 7f6b82800640 1 -- 172.21.15.134:0/3338036558 wait complete. 2024-07-26T11:11:50.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.748+0000 7f6b82800640 1 Processor -- start 2024-07-26T11:11:50.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.748+0000 7f6b82800640 1 -- start start 2024-07-26T11:11:50.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.749+0000 7f6b82800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b7c1099f0 0x7f6b7c1a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:50.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.749+0000 7f6b82800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b7c10f330 0x7f6b7c1a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:50.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.749+0000 7f6b82800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b7c10fc80 0x7f6b7c1a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:50.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.749+0000 7f6b82800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7c119940 con 0x7f6b7c1099f0 2024-07-26T11:11:50.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.749+0000 7f6b82800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7c1197c0 con 0x7f6b7c10f330 2024-07-26T11:11:50.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.749+0000 7f6b82800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7c119ac0 con 0x7f6b7c10fc80 2024-07-26T11:11:50.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.749+0000 7f6b80a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b7c10f330 0x7f6b7c1a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:50.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.749+0000 7f6b81400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b7c1099f0 0x7f6b7c1a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:50.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.749+0000 7f6b80a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b7c10f330 0x7f6b7c1a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:36530/0 (socket says 172.21.15.134:36530) 2024-07-26T11:11:50.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.749+0000 7f6b80a00640 1 -- 172.21.15.134:0/3376509771 learned_addr learned my addr 172.21.15.134:0/3376509771 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:11:50.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.749+0000 7f6b81e00640 1 --2- 172.21.15.134:0/3376509771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b7c10fc80 0x7f6b7c1a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.749+0000 7f6b80a00640 1 -- 172.21.15.134:0/3376509771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b7c10fc80 msgr2=0x7f6b7c1a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.749+0000 7f6b80a00640 1 --2- 172.21.15.134:0/3376509771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b7c10fc80 0x7f6b7c1a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.749+0000 7f6b80a00640 1 -- 172.21.15.134:0/3376509771 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b7c1099f0 msgr2=0x7f6b7c1a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.749+0000 7f6b80a00640 1 --2- 172.21.15.134:0/3376509771 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b7c1099f0 0x7f6b7c1a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.749+0000 7f6b80a00640 1 -- 172.21.15.134:0/3376509771 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b7c1a6280 con 0x7f6b7c10f330 2024-07-26T11:11:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.750+0000 7f6b81e00640 1 --2- 172.21.15.134:0/3376509771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b7c10fc80 0x7f6b7c1a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:11:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.750+0000 7f6b80a00640 1 --2- 172.21.15.134:0/3376509771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b7c10f330 0x7f6b7c1a08b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6b68035040 tx=0x7f6b68032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.750+0000 7f6b7a000640 1 -- 172.21.15.134:0/3376509771 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b68040020 con 0x7f6b7c10f330 2024-07-26T11:11:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.750+0000 7f6b82800640 1 -- 172.21.15.134:0/3376509771 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b7c1a6510 con 0x7f6b7c10f330 2024-07-26T11:11:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.750+0000 7f6b7a000640 1 -- 172.21.15.134:0/3376509771 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b68030e10 con 0x7f6b7c10f330 2024-07-26T11:11:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.750+0000 7f6b7a000640 1 -- 172.21.15.134:0/3376509771 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b68037040 con 0x7f6b7c10f330 2024-07-26T11:11:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.750+0000 7f6b82800640 1 -- 172.21.15.134:0/3376509771 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b7c1accd0 con 0x7f6b7c10f330 2024-07-26T11:11:50.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.752+0000 7f6b82800640 1 -- 172.21.15.134:0/3376509771 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b7c10ac70 con 0x7f6b7c10f330 2024-07-26T11:11:50.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.754+0000 7f6b7a000640 1 -- 172.21.15.134:0/3376509771 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7f6b68043020 con 0x7f6b7c10f330 2024-07-26T11:11:50.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.756+0000 7f6b7a000640 1 --2- 172.21.15.134:0/3376509771 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6b640b3c20 0x7f6b640b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:50.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.756+0000 7f6b7a000640 1 -- 172.21.15.134:0/3376509771 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f6b681044f0 con 0x7f6b7c10f330 2024-07-26T11:11:50.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.756+0000 7f6b81400640 1 --2- 172.21.15.134:0/3376509771 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6b640b3c20 0x7f6b640b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:50.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.757+0000 7f6b81400640 1 --2- 172.21.15.134:0/3376509771 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6b640b3c20 0x7f6b640b6130 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009530 tx=0x7f6b70009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:50.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.757+0000 7f6b7a000640 1 -- 172.21.15.134:0/3376509771 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6b6804a030 con 0x7f6b7c10f330 2024-07-26T11:11:50.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:50.997+0000 7f6b82800640 1 -- 172.21.15.134:0/3376509771 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f6b7c109e00 con 0x7f6b7c10f330 2024-07-26T11:11:51.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:51.077+0000 7f6b7a000640 1 -- 172.21.15.134:0/3376509771 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v27) ==== 121+0+0 (secure 0 0 0) 0x7f6b7c109e00 con 0x7f6b7c10f330 2024-07-26T11:11:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:51.082+0000 7f6b82800640 1 -- 172.21.15.134:0/3376509771 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6b640b3c20 msgr2=0x7f6b640b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:51.082+0000 7f6b82800640 1 --2- 172.21.15.134:0/3376509771 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6b640b3c20 0x7f6b640b6130 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009530 tx=0x7f6b70009260 comp rx=0 tx=0).stop 2024-07-26T11:11:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:51.082+0000 7f6b82800640 1 -- 172.21.15.134:0/3376509771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b7c10f330 msgr2=0x7f6b7c1a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:51.082+0000 7f6b82800640 1 --2- 172.21.15.134:0/3376509771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b7c10f330 0x7f6b7c1a08b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6b68035040 tx=0x7f6b68032d50 comp rx=0 tx=0).stop 2024-07-26T11:11:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:51.083+0000 7f6b82800640 1 -- 172.21.15.134:0/3376509771 shutdown_connections 2024-07-26T11:11:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:51.083+0000 7f6b82800640 1 --2- 172.21.15.134:0/3376509771 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6b640b3c20 0x7f6b640b6130 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:51.083+0000 7f6b82800640 1 --2- 172.21.15.134:0/3376509771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6b7c10fc80 0x7f6b7c1a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:51.083+0000 7f6b82800640 1 --2- 172.21.15.134:0/3376509771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b7c10f330 0x7f6b7c1a08b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:51.083+0000 7f6b82800640 1 --2- 172.21.15.134:0/3376509771 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6b7c1099f0 0x7f6b7c1a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:51.083+0000 7f6b82800640 1 -- 172.21.15.134:0/3376509771 >> 172.21.15.134:0/3376509771 conn(0x7f6b7c078ff0 msgr2=0x7f6b7c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:51.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:51.083+0000 7f6b82800640 1 -- 172.21.15.134:0/3376509771 shutdown_connections 2024-07-26T11:11:51.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:51.083+0000 7f6b82800640 1 -- 172.21.15.134:0/3376509771 wait complete. 2024-07-26T11:11:51.423 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:11:50 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 2024-07-26T11:11:50.946+0000 7fa125800640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:11:51.423 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:11:50 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 2024-07-26T11:11:50.946+0000 7fa125800640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-07-26T11:11:51.423 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:11:50 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 2024-07-26T11:11:50.946+0000 7fa125800640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-07-26T11:11:51.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:51 smithi134 ceph-mon[31966]: pgmap v356: 129 pgs: 129 active+clean; 584 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 3 op/s 2024-07-26T11:11:51.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:51 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] up:active 2024-07-26T11:11:51.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:51 smithi134 ceph-mon[31966]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-26T11:11:51.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:51 smithi134 ceph-mon[31966]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T11:11:51.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:51 smithi134 ceph-mon[31966]: Cluster is now healthy 2024-07-26T11:11:51.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:51 smithi134 ceph-mon[31966]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-07-26T11:11:51.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:51 smithi134 ceph-mon[31966]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-26T11:11:51.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:51 smithi134 ceph-mon[31966]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-07-26T11:11:51.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:51 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3376509771' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T11:11:51.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:51 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T11:11:51.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:51 smithi189 ceph-mon[36790]: pgmap v356: 129 pgs: 129 active+clean; 584 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 3 op/s 2024-07-26T11:11:51.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:51 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] up:active 2024-07-26T11:11:51.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:51 smithi189 ceph-mon[36790]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-26T11:11:51.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:51 smithi189 ceph-mon[36790]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T11:11:51.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:51 smithi189 ceph-mon[36790]: Cluster is now healthy 2024-07-26T11:11:51.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:51 smithi189 ceph-mon[36790]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-07-26T11:11:51.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:51 smithi189 ceph-mon[36790]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-26T11:11:51.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:51 smithi189 ceph-mon[36790]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-07-26T11:11:51.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:51 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3376509771' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T11:11:51.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:51 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T11:11:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:51 smithi193 ceph-mon[37473]: pgmap v356: 129 pgs: 129 active+clean; 584 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 3 op/s 2024-07-26T11:11:51.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:51 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] up:active 2024-07-26T11:11:51.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:51 smithi193 ceph-mon[37473]: daemon mds.h assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-07-26T11:11:51.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:51 smithi193 ceph-mon[37473]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-07-26T11:11:51.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:51 smithi193 ceph-mon[37473]: Cluster is now healthy 2024-07-26T11:11:51.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:51 smithi193 ceph-mon[37473]: fsmap cephfs:4 {0=a=up:active,1=b=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-07-26T11:11:51.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:51 smithi193 ceph-mon[37473]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-07-26T11:11:51.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:51 smithi193 ceph-mon[37473]: daemon mds.h is now active in filesystem cephfs as rank 4 2024-07-26T11:11:51.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:51 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3376509771' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T11:11:51.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:51 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-07-26T11:11:51.961 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-07-26T11:11:52.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:52 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T11:11:52.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:52 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] up:active 2024-07-26T11:11:52.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:52 smithi134 ceph-mon[31966]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-26T11:11:52.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:52 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:11:52.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:52 smithi134 ceph-mon[31966]: pgmap v357: 129 pgs: 129 active+clean; 584 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-07-26T11:11:52.436 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:11:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:52 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T11:11:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:52 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] up:active 2024-07-26T11:11:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:52 smithi189 ceph-mon[36790]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-26T11:11:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:52 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:11:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:52 smithi189 ceph-mon[36790]: pgmap v357: 129 pgs: 129 active+clean; 584 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-07-26T11:11:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:52 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-07-26T11:11:52.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:52 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] up:active 2024-07-26T11:11:52.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:52 smithi193 ceph-mon[37473]: fsmap cephfs:5 7 up:standby 5 up:active 2024-07-26T11:11:52.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:52 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:11:52.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:52 smithi193 ceph-mon[37473]: pgmap v357: 129 pgs: 129 active+clean; 584 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-07-26T11:11:54.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.873+0000 7fd890e00640 1 -- 172.21.15.134:0/544056879 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd88c104ce0 msgr2=0x7fd88c10d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:54.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.873+0000 7fd890e00640 1 --2- 172.21.15.134:0/544056879 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd88c104ce0 0x7fd88c10d320 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd874009900 tx=0x7fd87402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:11:54.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.873+0000 7fd890e00640 1 -- 172.21.15.134:0/544056879 shutdown_connections 2024-07-26T11:11:54.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.873+0000 7fd890e00640 1 --2- 172.21.15.134:0/544056879 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd88c104ce0 0x7fd88c10d320 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:54.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.873+0000 7fd890e00640 1 --2- 172.21.15.134:0/544056879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c10d860 0x7fd88c10fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:54.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.873+0000 7fd890e00640 1 --2- 172.21.15.134:0/544056879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd88c1042b0 0x7fd88c1046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:54.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.873+0000 7fd890e00640 1 -- 172.21.15.134:0/544056879 >> 172.21.15.134:0/544056879 conn(0x7fd88c079160 msgr2=0x7fd88c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:54.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.873+0000 7fd890e00640 1 -- 172.21.15.134:0/544056879 shutdown_connections 2024-07-26T11:11:54.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.873+0000 7fd890e00640 1 -- 172.21.15.134:0/544056879 wait complete. 2024-07-26T11:11:54.878 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.874+0000 7fd890e00640 1 Processor -- start 2024-07-26T11:11:54.878 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.875+0000 7fd890e00640 1 -- start start 2024-07-26T11:11:54.878 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.875+0000 7fd890e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd88c1042b0 0x7fd88c1021c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:54.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.875+0000 7fd890e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd88c104ce0 0x7fd88c102700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:54.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.875+0000 7fd890e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c10d860 0x7fd88c102c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:54.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.875+0000 7fd890e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c1134d0 con 0x7fd88c104ce0 2024-07-26T11:11:54.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.875+0000 7fd890e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c113350 con 0x7fd88c10d860 2024-07-26T11:11:54.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.875+0000 7fd890e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c113650 con 0x7fd88c1042b0 2024-07-26T11:11:54.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.876+0000 7fd883e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd88c104ce0 0x7fd88c102700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:54.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.876+0000 7fd883e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd88c104ce0 0x7fd88c102700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:60090/0 (socket says 172.21.15.134:60090) 2024-07-26T11:11:54.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.876+0000 7fd889e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd88c1042b0 0x7fd88c1021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:54.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.876+0000 7fd883e00640 1 -- 172.21.15.134:0/3802645293 learned_addr learned my addr 172.21.15.134:0/3802645293 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:11:54.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.876+0000 7fd88a800640 1 --2- 172.21.15.134:0/3802645293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c10d860 0x7fd88c102c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:54.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.876+0000 7fd889e00640 1 -- 172.21.15.134:0/3802645293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c10d860 msgr2=0x7fd88c102c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:54.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.876+0000 7fd889e00640 1 --2- 172.21.15.134:0/3802645293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c10d860 0x7fd88c102c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:54.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.876+0000 7fd889e00640 1 -- 172.21.15.134:0/3802645293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd88c104ce0 msgr2=0x7fd88c102700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:54.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.876+0000 7fd889e00640 1 --2- 172.21.15.134:0/3802645293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd88c104ce0 0x7fd88c102700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:54.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.876+0000 7fd889e00640 1 -- 172.21.15.134:0/3802645293 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88c1a8180 con 0x7fd88c1042b0 2024-07-26T11:11:54.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.876+0000 7fd889e00640 1 --2- 172.21.15.134:0/3802645293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd88c1042b0 0x7fd88c1021c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd87c00e9c0 tx=0x7fd87c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:54.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.877+0000 7fd882a00640 1 -- 172.21.15.134:0/3802645293 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c00ccc0 con 0x7fd88c1042b0 2024-07-26T11:11:54.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.877+0000 7fd882a00640 1 -- 172.21.15.134:0/3802645293 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd87c0035f0 con 0x7fd88c1042b0 2024-07-26T11:11:54.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.877+0000 7fd882a00640 1 -- 172.21.15.134:0/3802645293 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c00ce60 con 0x7fd88c1042b0 2024-07-26T11:11:54.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.877+0000 7fd890e00640 1 -- 172.21.15.134:0/3802645293 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd88c1a8470 con 0x7fd88c1042b0 2024-07-26T11:11:54.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.877+0000 7fd890e00640 1 -- 172.21.15.134:0/3802645293 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd88c1a8930 con 0x7fd88c1042b0 2024-07-26T11:11:54.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.878+0000 7fd890e00640 1 -- 172.21.15.134:0/3802645293 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd88c1046c0 con 0x7fd88c1042b0 2024-07-26T11:11:54.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.881+0000 7fd882a00640 1 -- 172.21.15.134:0/3802645293 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fd87c004000 con 0x7fd88c1042b0 2024-07-26T11:11:54.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.882+0000 7fd882a00640 1 --2- 172.21.15.134:0/3802645293 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8700b3ca0 0x7fd8700b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:54.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.882+0000 7fd882a00640 1 -- 172.21.15.134:0/3802645293 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd87c0e4500 con 0x7fd88c1042b0 2024-07-26T11:11:54.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.883+0000 7fd883e00640 1 --2- 172.21.15.134:0/3802645293 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8700b3ca0 0x7fd8700b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:54.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.883+0000 7fd883e00640 1 --2- 172.21.15.134:0/3802645293 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8700b3ca0 0x7fd8700b61b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd874000c00 tx=0x7fd874005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:54.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:54.884+0000 7fd882a00640 1 -- 172.21.15.134:0/3802645293 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd87c0ab650 con 0x7fd88c1042b0 2024-07-26T11:11:55.127 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:55.125+0000 7fd890e00640 1 -- 172.21.15.134:0/3802645293 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fd88c1a8d50 con 0x7fd88c1042b0 2024-07-26T11:11:55.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:54 smithi134 ceph-mon[31966]: pgmap v358: 129 pgs: 129 active+clean; 587 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.9 KiB/s wr, 15 op/s 2024-07-26T11:11:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:54 smithi193 ceph-mon[37473]: pgmap v358: 129 pgs: 129 active+clean; 587 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.9 KiB/s wr, 15 op/s 2024-07-26T11:11:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:54 smithi189 ceph-mon[36790]: pgmap v358: 129 pgs: 129 active+clean; 587 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.9 KiB/s wr, 15 op/s 2024-07-26T11:11:55.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:55.989+0000 7fd882a00640 1 -- 172.21.15.134:0/3802645293 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v29) ==== 115+0+0 (secure 0 0 0) 0x7fd87c0aaf50 con 0x7fd88c1042b0 2024-07-26T11:11:55.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:55.994+0000 7fd890e00640 1 -- 172.21.15.134:0/3802645293 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8700b3ca0 msgr2=0x7fd8700b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:55.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:55.994+0000 7fd890e00640 1 --2- 172.21.15.134:0/3802645293 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8700b3ca0 0x7fd8700b61b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd874000c00 tx=0x7fd874005a20 comp rx=0 tx=0).stop 2024-07-26T11:11:55.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:55.994+0000 7fd890e00640 1 -- 172.21.15.134:0/3802645293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd88c1042b0 msgr2=0x7fd88c1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:55.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:55.994+0000 7fd890e00640 1 --2- 172.21.15.134:0/3802645293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd88c1042b0 0x7fd88c1021c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd87c00e9c0 tx=0x7fd87c00ee80 comp rx=0 tx=0).stop 2024-07-26T11:11:55.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:55.995+0000 7fd890e00640 1 -- 172.21.15.134:0/3802645293 shutdown_connections 2024-07-26T11:11:55.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:55.995+0000 7fd890e00640 1 --2- 172.21.15.134:0/3802645293 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8700b3ca0 0x7fd8700b61b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:55.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:55.995+0000 7fd890e00640 1 --2- 172.21.15.134:0/3802645293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c10d860 0x7fd88c102c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:55.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:55.995+0000 7fd890e00640 1 --2- 172.21.15.134:0/3802645293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd88c104ce0 0x7fd88c102700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:55.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:55.995+0000 7fd890e00640 1 --2- 172.21.15.134:0/3802645293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd88c1042b0 0x7fd88c1021c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:55.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:55.995+0000 7fd890e00640 1 -- 172.21.15.134:0/3802645293 >> 172.21.15.134:0/3802645293 conn(0x7fd88c079160 msgr2=0x7fd88c1062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:55.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:55.995+0000 7fd890e00640 1 -- 172.21.15.134:0/3802645293 shutdown_connections 2024-07-26T11:11:55.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:55.995+0000 7fd890e00640 1 -- 172.21.15.134:0/3802645293 wait complete. 2024-07-26T11:11:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:55 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3802645293' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T11:11:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:55 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T11:11:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:55 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3802645293' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T11:11:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:55 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T11:11:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:55 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3802645293' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T11:11:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:55 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-07-26T11:11:56.979 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-07-26T11:11:56.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-07-26T11:11:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:56 smithi134 ceph-mon[31966]: pgmap v359: 129 pgs: 129 active+clean; 587 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 4.0 KiB/s wr, 17 op/s 2024-07-26T11:11:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:56 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T11:11:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:56 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:11:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:56 smithi193 ceph-mon[37473]: pgmap v359: 129 pgs: 129 active+clean; 587 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 4.0 KiB/s wr, 17 op/s 2024-07-26T11:11:57.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:56 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T11:11:57.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:56 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:11:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:56 smithi189 ceph-mon[36790]: pgmap v359: 129 pgs: 129 active+clean; 587 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 4.0 KiB/s wr, 17 op/s 2024-07-26T11:11:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:56 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-07-26T11:11:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:56 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:11:57.445 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:11:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:11:58 smithi134 ceph-mon[31966]: pgmap v360: 129 pgs: 129 active+clean; 587 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 4.0 KiB/s wr, 20 op/s 2024-07-26T11:11:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:11:58 smithi193 ceph-mon[37473]: pgmap v360: 129 pgs: 129 active+clean; 587 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 4.0 KiB/s wr, 20 op/s 2024-07-26T11:11:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:11:58 smithi189 ceph-mon[36790]: pgmap v360: 129 pgs: 129 active+clean; 587 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 4.0 KiB/s wr, 20 op/s 2024-07-26T11:11:59.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.795+0000 7fa3a3e00640 1 -- 172.21.15.134:0/2612344923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3a41055d0 msgr2=0x7fa3a41059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:59.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.795+0000 7fa3a3e00640 1 --2- 172.21.15.134:0/2612344923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3a41055d0 0x7fa3a41059e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa394009860 tx=0x7fa39402a340 comp rx=0 tx=0).stop 2024-07-26T11:11:59.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.795+0000 7fa3a3e00640 1 -- 172.21.15.134:0/2612344923 shutdown_connections 2024-07-26T11:11:59.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.795+0000 7fa3a3e00640 1 --2- 172.21.15.134:0/2612344923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3a410b860 0x7fa3a4112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:59.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.795+0000 7fa3a3e00640 1 --2- 172.21.15.134:0/2612344923 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3a410af10 0x7fa3a410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:59.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.795+0000 7fa3a3e00640 1 --2- 172.21.15.134:0/2612344923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3a41055d0 0x7fa3a41059e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:59.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.795+0000 7fa3a3e00640 1 -- 172.21.15.134:0/2612344923 >> 172.21.15.134:0/2612344923 conn(0x7fa3a4101040 msgr2=0x7fa3a4103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:11:59.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.795+0000 7fa3a3e00640 1 -- 172.21.15.134:0/2612344923 shutdown_connections 2024-07-26T11:11:59.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.795+0000 7fa3a3e00640 1 -- 172.21.15.134:0/2612344923 wait complete. 2024-07-26T11:11:59.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.796+0000 7fa3a3e00640 1 Processor -- start 2024-07-26T11:11:59.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.797+0000 7fa3a3e00640 1 -- start start 2024-07-26T11:11:59.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.797+0000 7fa3a3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3a41055d0 0x7fa3a419bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:59.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.797+0000 7fa3a3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3a410af10 0x7fa3a419c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:59.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.797+0000 7fa3a3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3a410b860 0x7fa3a41a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:59.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.797+0000 7fa3a3e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a4115530 con 0x7fa3a410af10 2024-07-26T11:11:59.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.797+0000 7fa3a3e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a41153b0 con 0x7fa3a410b860 2024-07-26T11:11:59.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.797+0000 7fa3a3e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a41156b0 con 0x7fa3a41055d0 2024-07-26T11:11:59.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.798+0000 7fa3a2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3a41055d0 0x7fa3a419bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:59.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.798+0000 7fa39be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3a410af10 0x7fa3a419c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:59.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.798+0000 7fa3a2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3a41055d0 0x7fa3a419bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53510/0 (socket says 172.21.15.134:53510) 2024-07-26T11:11:59.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.798+0000 7fa3a2a00640 1 -- 172.21.15.134:0/4067107329 learned_addr learned my addr 172.21.15.134:0/4067107329 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:11:59.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.798+0000 7fa3a3400640 1 --2- 172.21.15.134:0/4067107329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3a410b860 0x7fa3a41a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:59.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.798+0000 7fa3a2a00640 1 -- 172.21.15.134:0/4067107329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3a410b860 msgr2=0x7fa3a41a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:59.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.798+0000 7fa3a2a00640 1 --2- 172.21.15.134:0/4067107329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3a410b860 0x7fa3a41a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:59.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.798+0000 7fa3a2a00640 1 -- 172.21.15.134:0/4067107329 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3a410af10 msgr2=0x7fa3a419c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:11:59.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.798+0000 7fa3a2a00640 1 --2- 172.21.15.134:0/4067107329 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3a410af10 0x7fa3a419c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:11:59.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.798+0000 7fa3a2a00640 1 -- 172.21.15.134:0/4067107329 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3a41a1e70 con 0x7fa3a41055d0 2024-07-26T11:11:59.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.799+0000 7fa3a2a00640 1 --2- 172.21.15.134:0/4067107329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3a41055d0 0x7fa3a419bf00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa394035040 tx=0x7fa394032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:59.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.799+0000 7fa39b400640 1 -- 172.21.15.134:0/4067107329 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa394037040 con 0x7fa3a41055d0 2024-07-26T11:11:59.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.799+0000 7fa3a3e00640 1 -- 172.21.15.134:0/4067107329 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3a41a2100 con 0x7fa3a41055d0 2024-07-26T11:11:59.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.799+0000 7fa3a3e00640 1 -- 172.21.15.134:0/4067107329 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3a41a88c0 con 0x7fa3a41055d0 2024-07-26T11:11:59.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.799+0000 7fa39b400640 1 -- 172.21.15.134:0/4067107329 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa394041070 con 0x7fa3a41055d0 2024-07-26T11:11:59.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.799+0000 7fa39b400640 1 -- 172.21.15.134:0/4067107329 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3940035f0 con 0x7fa3a41055d0 2024-07-26T11:11:59.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.800+0000 7fa3a3e00640 1 -- 172.21.15.134:0/4067107329 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3a41059e0 con 0x7fa3a41055d0 2024-07-26T11:11:59.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.803+0000 7fa39b400640 1 -- 172.21.15.134:0/4067107329 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) ==== 151077+0+0 (secure 0 0 0) 0x7fa394004050 con 0x7fa3a41055d0 2024-07-26T11:11:59.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.804+0000 7fa39b400640 1 --2- 172.21.15.134:0/4067107329 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa3840b3c20 0x7fa3840b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:11:59.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.804+0000 7fa39b400640 1 -- 172.21.15.134:0/4067107329 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa394103f00 con 0x7fa3a41055d0 2024-07-26T11:11:59.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.804+0000 7fa39be00640 1 --2- 172.21.15.134:0/4067107329 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa3840b3c20 0x7fa3840b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:11:59.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.805+0000 7fa39be00640 1 --2- 172.21.15.134:0/4067107329 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa3840b3c20 0x7fa3840b6130 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa3a410aa30 tx=0x7fa38c009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:11:59.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:11:59.806+0000 7fa39b400640 1 -- 172.21.15.134:0/4067107329 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa3940cb0e0 con 0x7fa3a41055d0 2024-07-26T11:12:00.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:00.034+0000 7fa3a3e00640 1 -- 172.21.15.134:0/4067107329 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fa3a4063230 con 0x7fa3840b3c20 2024-07-26T11:12:00.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:00.074+0000 7fa39b400640 1 -- 172.21.15.134:0/4067107329 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa3a4063230 con 0x7fa3840b3c20 2024-07-26T11:12:00.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:00.077+0000 7fa3a3e00640 1 -- 172.21.15.134:0/4067107329 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa3840b3c20 msgr2=0x7fa3840b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:00.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:00.077+0000 7fa3a3e00640 1 --2- 172.21.15.134:0/4067107329 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa3840b3c20 0x7fa3840b6130 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa3a410aa30 tx=0x7fa38c009260 comp rx=0 tx=0).stop 2024-07-26T11:12:00.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:00.077+0000 7fa3a3e00640 1 -- 172.21.15.134:0/4067107329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3a41055d0 msgr2=0x7fa3a419bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:00.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:00.077+0000 7fa3a3e00640 1 --2- 172.21.15.134:0/4067107329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3a41055d0 0x7fa3a419bf00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa394035040 tx=0x7fa394032d50 comp rx=0 tx=0).stop 2024-07-26T11:12:00.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:00.077+0000 7fa3a3e00640 1 -- 172.21.15.134:0/4067107329 shutdown_connections 2024-07-26T11:12:00.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:00.077+0000 7fa3a3e00640 1 --2- 172.21.15.134:0/4067107329 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa3840b3c20 0x7fa3840b6130 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:00.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:00.077+0000 7fa3a3e00640 1 --2- 172.21.15.134:0/4067107329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3a410b860 0x7fa3a41a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:00.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:00.077+0000 7fa3a3e00640 1 --2- 172.21.15.134:0/4067107329 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa3a410af10 0x7fa3a419c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:00.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:00.077+0000 7fa3a3e00640 1 --2- 172.21.15.134:0/4067107329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3a41055d0 0x7fa3a419bf00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:00.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:00.077+0000 7fa3a3e00640 1 -- 172.21.15.134:0/4067107329 >> 172.21.15.134:0/4067107329 conn(0x7fa3a4101040 msgr2=0x7fa3a4103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:00.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:00.077+0000 7fa3a3e00640 1 -- 172.21.15.134:0/4067107329 shutdown_connections 2024-07-26T11:12:00.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:00.077+0000 7fa3a3e00640 1 -- 172.21.15.134:0/4067107329 wait complete. 2024-07-26T11:12:00.423 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:12:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:12:00.045+0000 7f9017000640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:12:00.424 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:12:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:12:00.045+0000 7f9017000640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:12:00.424 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:12:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:12:00.045+0000 7f9017000640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:12:00.424 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:12:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:12:00.045+0000 7f9017000640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:12:00.424 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:12:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:12:00.045+0000 7f9017000640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:12:01.065 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-07-26T11:12:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:00 smithi193 ceph-mon[37473]: pgmap v361: 129 pgs: 129 active+clean; 587 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.8 KiB/s wr, 20 op/s 2024-07-26T11:12:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:00 smithi193 ceph-mon[37473]: from='client.24901 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:12:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:00 smithi134 ceph-mon[31966]: pgmap v361: 129 pgs: 129 active+clean; 587 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.8 KiB/s wr, 20 op/s 2024-07-26T11:12:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:00 smithi134 ceph-mon[31966]: from='client.24901 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:12:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:00 smithi189 ceph-mon[36790]: pgmap v361: 129 pgs: 129 active+clean; 587 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.8 KiB/s wr, 20 op/s 2024-07-26T11:12:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:00 smithi189 ceph-mon[36790]: from='client.24901 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:12:01.510 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:02 smithi134 ceph-mon[31966]: mgrmap e21: x(active, since 9m), standbys: z, y 2024-07-26T11:12:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:02 smithi189 ceph-mon[36790]: mgrmap e21: x(active, since 9m), standbys: z, y 2024-07-26T11:12:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:02 smithi193 ceph-mon[37473]: mgrmap e21: x(active, since 9m), standbys: z, y 2024-07-26T11:12:03.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:03 smithi134 ceph-mon[31966]: pgmap v362: 129 pgs: 129 active+clean; 587 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.3 KiB/s wr, 19 op/s 2024-07-26T11:12:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:03 smithi189 ceph-mon[36790]: pgmap v362: 129 pgs: 129 active+clean; 587 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.3 KiB/s wr, 19 op/s 2024-07-26T11:12:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:03 smithi193 ceph-mon[37473]: pgmap v362: 129 pgs: 129 active+clean; 587 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.3 KiB/s wr, 19 op/s 2024-07-26T11:12:03.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.772+0000 7f5670c00640 1 -- 172.21.15.134:0/4193724779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f566c1033c0 msgr2=0x7f566c1037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:03.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.772+0000 7f5670c00640 1 --2- 172.21.15.134:0/4193724779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f566c1033c0 0x7f566c1037d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f565c009900 tx=0x7f565c02a400 comp rx=0 tx=0).stop 2024-07-26T11:12:03.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.773+0000 7f5670c00640 1 -- 172.21.15.134:0/4193724779 shutdown_connections 2024-07-26T11:12:03.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.773+0000 7f5670c00640 1 --2- 172.21.15.134:0/4193724779 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f566c109650 0x7f566c10fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:03.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.773+0000 7f5670c00640 1 --2- 172.21.15.134:0/4193724779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f566c108d00 0x7f566c109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:03.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.773+0000 7f5670c00640 1 --2- 172.21.15.134:0/4193724779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f566c1033c0 0x7f566c1037d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:03.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.773+0000 7f5670c00640 1 -- 172.21.15.134:0/4193724779 >> 172.21.15.134:0/4193724779 conn(0x7f566c0fee10 msgr2=0x7f566c101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:03.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.773+0000 7f5670c00640 1 -- 172.21.15.134:0/4193724779 shutdown_connections 2024-07-26T11:12:03.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.773+0000 7f5670c00640 1 -- 172.21.15.134:0/4193724779 wait complete. 2024-07-26T11:12:03.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.774+0000 7f5670c00640 1 Processor -- start 2024-07-26T11:12:03.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.774+0000 7f5670c00640 1 -- start start 2024-07-26T11:12:03.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.774+0000 7f5670c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f566c1033c0 0x7f566c19bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:03.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.774+0000 7f5670c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f566c108d00 0x7f566c19c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:03.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.774+0000 7f5670c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f566c109650 0x7f566c1a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:03.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.775+0000 7f5670c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f566c113300 con 0x7f566c108d00 2024-07-26T11:12:03.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.775+0000 7f5670c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f566c113180 con 0x7f566c109650 2024-07-26T11:12:03.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.775+0000 7f5670c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f566c113480 con 0x7f566c1033c0 2024-07-26T11:12:03.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.775+0000 7f5663e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f566c108d00 0x7f566c19c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:03.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.775+0000 7f566b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f566c1033c0 0x7f566c19bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:03.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.775+0000 7f566be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f566c109650 0x7f566c1a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:03.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.775+0000 7f566b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f566c1033c0 0x7f566c19bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:47794/0 (socket says 172.21.15.134:47794) 2024-07-26T11:12:03.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.775+0000 7f566b400640 1 -- 172.21.15.134:0/3745006103 learned_addr learned my addr 172.21.15.134:0/3745006103 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:12:03.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.775+0000 7f5663e00640 1 -- 172.21.15.134:0/3745006103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f566c1033c0 msgr2=0x7f566c19bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:03.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.775+0000 7f5663e00640 1 --2- 172.21.15.134:0/3745006103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f566c1033c0 0x7f566c19bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:03.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.775+0000 7f5663e00640 1 -- 172.21.15.134:0/3745006103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f566c109650 msgr2=0x7f566c1a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:03.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.775+0000 7f5663e00640 1 --2- 172.21.15.134:0/3745006103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f566c109650 0x7f566c1a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:03.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.775+0000 7f5663e00640 1 -- 172.21.15.134:0/3745006103 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566c1a1ea0 con 0x7f566c108d00 2024-07-26T11:12:03.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.776+0000 7f5663e00640 1 --2- 172.21.15.134:0/3745006103 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f566c108d00 0x7f566c19c4d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f566c06c260 tx=0x7f565400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:03.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.776+0000 7f5668c00640 1 -- 172.21.15.134:0/3745006103 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565400cc40 con 0x7f566c108d00 2024-07-26T11:12:03.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.776+0000 7f5668c00640 1 -- 172.21.15.134:0/3745006103 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f565400cde0 con 0x7f566c108d00 2024-07-26T11:12:03.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.776+0000 7f5670c00640 1 -- 172.21.15.134:0/3745006103 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f566c1a2190 con 0x7f566c108d00 2024-07-26T11:12:03.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.776+0000 7f5668c00640 1 -- 172.21.15.134:0/3745006103 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5654018780 con 0x7f566c108d00 2024-07-26T11:12:03.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.776+0000 7f5670c00640 1 -- 172.21.15.134:0/3745006103 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f566c1a88f0 con 0x7f566c108d00 2024-07-26T11:12:03.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.777+0000 7f5670c00640 1 -- 172.21.15.134:0/3745006103 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f566c1037d0 con 0x7f566c108d00 2024-07-26T11:12:03.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.780+0000 7f5668c00640 1 -- 172.21.15.134:0/3745006103 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f5654010a90 con 0x7f566c108d00 2024-07-26T11:12:03.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.781+0000 7f5668c00640 1 --2- 172.21.15.134:0/3745006103 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f56500b3cb0 0x7f56500b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:03.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.782+0000 7f5668c00640 1 -- 172.21.15.134:0/3745006103 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f56540e4a00 con 0x7f566c108d00 2024-07-26T11:12:03.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.782+0000 7f566b400640 1 --2- 172.21.15.134:0/3745006103 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f56500b3cb0 0x7f56500b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:03.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.783+0000 7f566b400640 1 --2- 172.21.15.134:0/3745006103 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f56500b3cb0 0x7f56500b61c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f565c031440 tx=0x7f565c0058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:03.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:03.784+0000 7f5668c00640 1 -- 172.21.15.134:0/3745006103 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f56540ab3a0 con 0x7f566c108d00 2024-07-26T11:12:04.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:04.018+0000 7f5670c00640 1 -- 172.21.15.134:0/3745006103 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f566c10f770 con 0x7f56500b3cb0 2024-07-26T11:12:04.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:04.069+0000 7f5668c00640 1 -- 172.21.15.134:0/3745006103 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f566c10f770 con 0x7f56500b3cb0 2024-07-26T11:12:04.074 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:04.072+0000 7f5670c00640 1 -- 172.21.15.134:0/3745006103 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f56500b3cb0 msgr2=0x7f56500b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:04.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:04.072+0000 7f5670c00640 1 --2- 172.21.15.134:0/3745006103 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f56500b3cb0 0x7f56500b61c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f565c031440 tx=0x7f565c0058d0 comp rx=0 tx=0).stop 2024-07-26T11:12:04.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:04.072+0000 7f5670c00640 1 -- 172.21.15.134:0/3745006103 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f566c108d00 msgr2=0x7f566c19c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:04.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:04.072+0000 7f5670c00640 1 --2- 172.21.15.134:0/3745006103 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f566c108d00 0x7f566c19c4d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f566c06c260 tx=0x7f565400ee50 comp rx=0 tx=0).stop 2024-07-26T11:12:04.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:04.072+0000 7f5670c00640 1 -- 172.21.15.134:0/3745006103 shutdown_connections 2024-07-26T11:12:04.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:04.072+0000 7f5670c00640 1 --2- 172.21.15.134:0/3745006103 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f56500b3cb0 0x7f56500b61c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:04.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:04.072+0000 7f5670c00640 1 --2- 172.21.15.134:0/3745006103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f566c109650 0x7f566c1a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:04.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:04.072+0000 7f5670c00640 1 --2- 172.21.15.134:0/3745006103 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f566c108d00 0x7f566c19c4d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:04.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:04.072+0000 7f5670c00640 1 --2- 172.21.15.134:0/3745006103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f566c1033c0 0x7f566c19bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:04.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:04.072+0000 7f5670c00640 1 -- 172.21.15.134:0/3745006103 >> 172.21.15.134:0/3745006103 conn(0x7f566c0fee10 msgr2=0x7f566c10e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:04.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:04.072+0000 7f5670c00640 1 -- 172.21.15.134:0/3745006103 shutdown_connections 2024-07-26T11:12:04.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:04.073+0000 7f5670c00640 1 -- 172.21.15.134:0/3745006103 wait complete. 2024-07-26T11:12:04.868 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-07-26T11:12:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:05 smithi134 ceph-mon[31966]: pgmap v363: 129 pgs: 129 active+clean; 593 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.6 KiB/s wr, 18 op/s 2024-07-26T11:12:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:05 smithi134 ceph-mon[31966]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:12:05.319 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:05 smithi189 ceph-mon[36790]: pgmap v363: 129 pgs: 129 active+clean; 593 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.6 KiB/s wr, 18 op/s 2024-07-26T11:12:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:05 smithi189 ceph-mon[36790]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:12:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:05 smithi193 ceph-mon[37473]: pgmap v363: 129 pgs: 129 active+clean; 593 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 2.6 KiB/s wr, 18 op/s 2024-07-26T11:12:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:05 smithi193 ceph-mon[37473]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:12:07.325 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:07 smithi134 ceph-mon[31966]: pgmap v364: 129 pgs: 129 active+clean; 593 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1023 B/s wr, 11 op/s 2024-07-26T11:12:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:07 smithi189 ceph-mon[36790]: pgmap v364: 129 pgs: 129 active+clean; 593 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1023 B/s wr, 11 op/s 2024-07-26T11:12:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:07 smithi193 ceph-mon[37473]: pgmap v364: 129 pgs: 129 active+clean; 593 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1023 B/s wr, 11 op/s 2024-07-26T11:12:07.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.664+0000 7f31a3e00640 1 -- 172.21.15.134:0/821982910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a410f940 msgr2=0x7f31a4111d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:07.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.664+0000 7f31a3e00640 1 --2- 172.21.15.134:0/821982910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a410f940 0x7f31a4111d80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f319c009be0 tx=0x7f319c02a6c0 comp rx=0 tx=0).stop 2024-07-26T11:12:07.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.665+0000 7f31a3e00640 1 -- 172.21.15.134:0/821982910 shutdown_connections 2024-07-26T11:12:07.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.665+0000 7f31a3e00640 1 --2- 172.21.15.134:0/821982910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a410f940 0x7f31a4111d80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:07.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.665+0000 7f31a3e00640 1 --2- 172.21.15.134:0/821982910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31a4075b30 0x7f31a410f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:07.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.665+0000 7f31a3e00640 1 --2- 172.21.15.134:0/821982910 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31a40751e0 0x7f31a40755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:07.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.665+0000 7f31a3e00640 1 -- 172.21.15.134:0/821982910 >> 172.21.15.134:0/821982910 conn(0x7f31a40fee70 msgr2=0x7f31a41012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:07.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.665+0000 7f31a3e00640 1 -- 172.21.15.134:0/821982910 shutdown_connections 2024-07-26T11:12:07.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.665+0000 7f31a3e00640 1 -- 172.21.15.134:0/821982910 wait complete. 2024-07-26T11:12:07.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.666+0000 7f31a3e00640 1 Processor -- start 2024-07-26T11:12:07.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.666+0000 7f31a3e00640 1 -- start start 2024-07-26T11:12:07.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.666+0000 7f31a3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a40751e0 0x7f31a419bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:07.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.666+0000 7f31a3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31a4075b30 0x7f31a419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:07.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.666+0000 7f31a3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31a410f940 0x7f31a41a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:07.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.667+0000 7f31a3e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f31a41154d0 con 0x7f31a4075b30 2024-07-26T11:12:07.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.667+0000 7f31a3e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f31a4115350 con 0x7f31a410f940 2024-07-26T11:12:07.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.667+0000 7f31a3e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f31a4115650 con 0x7f31a40751e0 2024-07-26T11:12:07.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.667+0000 7f31a2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a40751e0 0x7f31a419bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:07.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.667+0000 7f31a2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a40751e0 0x7f31a419bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:47818/0 (socket says 172.21.15.134:47818) 2024-07-26T11:12:07.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.667+0000 7f319a000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31a4075b30 0x7f31a419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:07.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.667+0000 7f31a2a00640 1 -- 172.21.15.134:0/1608278641 learned_addr learned my addr 172.21.15.134:0/1608278641 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:12:07.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.667+0000 7f31a3400640 1 --2- 172.21.15.134:0/1608278641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31a410f940 0x7f31a41a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:07.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.667+0000 7f319a000640 1 -- 172.21.15.134:0/1608278641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a40751e0 msgr2=0x7f31a419bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:07.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.667+0000 7f319a000640 1 --2- 172.21.15.134:0/1608278641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a40751e0 0x7f31a419bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:07.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.667+0000 7f319a000640 1 -- 172.21.15.134:0/1608278641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31a410f940 msgr2=0x7f31a41a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:07.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.667+0000 7f319a000640 1 --2- 172.21.15.134:0/1608278641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31a410f940 0x7f31a41a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:07.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.667+0000 7f319a000640 1 -- 172.21.15.134:0/1608278641 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a41a1e10 con 0x7f31a4075b30 2024-07-26T11:12:07.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.667+0000 7f31a3400640 1 --2- 172.21.15.134:0/1608278641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31a410f940 0x7f31a41a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:12:07.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.668+0000 7f319a000640 1 --2- 172.21.15.134:0/1608278641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31a4075b30 0x7f31a419c440 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f318c00e9c0 tx=0x7f318c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:07.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.668+0000 7f319be00640 1 -- 172.21.15.134:0/1608278641 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c00ccc0 con 0x7f31a4075b30 2024-07-26T11:12:07.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.668+0000 7f319be00640 1 -- 172.21.15.134:0/1608278641 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f318c0035f0 con 0x7f31a4075b30 2024-07-26T11:12:07.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.668+0000 7f31a3e00640 1 -- 172.21.15.134:0/1608278641 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a41a2100 con 0x7f31a4075b30 2024-07-26T11:12:07.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.668+0000 7f319be00640 1 -- 172.21.15.134:0/1608278641 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c00ce60 con 0x7f31a4075b30 2024-07-26T11:12:07.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.668+0000 7f31a3e00640 1 -- 172.21.15.134:0/1608278641 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a41a8860 con 0x7f31a4075b30 2024-07-26T11:12:07.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.669+0000 7f31a3e00640 1 -- 172.21.15.134:0/1608278641 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31a40755f0 con 0x7f31a4075b30 2024-07-26T11:12:07.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.672+0000 7f319be00640 1 -- 172.21.15.134:0/1608278641 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f318c004000 con 0x7f31a4075b30 2024-07-26T11:12:07.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.675+0000 7f319be00640 1 --2- 172.21.15.134:0/1608278641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31880b3d30 0x7f31880b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:07.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.675+0000 7f319be00640 1 -- 172.21.15.134:0/1608278641 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f318c0e4720 con 0x7f31a4075b30 2024-07-26T11:12:07.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.675+0000 7f31a2a00640 1 --2- 172.21.15.134:0/1608278641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31880b3d30 0x7f31880b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:07.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.676+0000 7f31a2a00640 1 --2- 172.21.15.134:0/1608278641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31880b3d30 0x7f31880b6240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f31940058c0 tx=0x7f3194009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:07.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.677+0000 7f319be00640 1 -- 172.21.15.134:0/1608278641 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f318c0ab7a0 con 0x7f31a4075b30 2024-07-26T11:12:07.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.903+0000 7f31a3e00640 1 -- 172.21.15.134:0/1608278641 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f31a4109bf0 con 0x7f31880b3d30 2024-07-26T11:12:07.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.914+0000 7f319be00640 1 -- 172.21.15.134:0/1608278641 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f31a4109bf0 con 0x7f31880b3d30 2024-07-26T11:12:07.917 INFO:teuthology.orchestra.run.smithi134.stdout:/volumes/qa/sv_0/f877aa9c-6f58-40c1-af71-4e8584495c8d 2024-07-26T11:12:07.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.917+0000 7f31a3e00640 1 -- 172.21.15.134:0/1608278641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31880b3d30 msgr2=0x7f31880b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:07.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.917+0000 7f31a3e00640 1 --2- 172.21.15.134:0/1608278641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31880b3d30 0x7f31880b6240 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f31940058c0 tx=0x7f3194009260 comp rx=0 tx=0).stop 2024-07-26T11:12:07.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.917+0000 7f31a3e00640 1 -- 172.21.15.134:0/1608278641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31a4075b30 msgr2=0x7f31a419c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:07.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.917+0000 7f31a3e00640 1 --2- 172.21.15.134:0/1608278641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31a4075b30 0x7f31a419c440 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f318c00e9c0 tx=0x7f318c00ee80 comp rx=0 tx=0).stop 2024-07-26T11:12:07.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.918+0000 7f31a3e00640 1 -- 172.21.15.134:0/1608278641 shutdown_connections 2024-07-26T11:12:07.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.918+0000 7f31a3e00640 1 --2- 172.21.15.134:0/1608278641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31880b3d30 0x7f31880b6240 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:07.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.918+0000 7f31a3e00640 1 --2- 172.21.15.134:0/1608278641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31a410f940 0x7f31a41a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:07.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.918+0000 7f31a3e00640 1 --2- 172.21.15.134:0/1608278641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31a4075b30 0x7f31a419c440 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:07.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.918+0000 7f31a3e00640 1 --2- 172.21.15.134:0/1608278641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31a40751e0 0x7f31a419bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:07.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.918+0000 7f31a3e00640 1 -- 172.21.15.134:0/1608278641 >> 172.21.15.134:0/1608278641 conn(0x7f31a40fee70 msgr2=0x7f31a41094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:07.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.918+0000 7f31a3e00640 1 -- 172.21.15.134:0/1608278641 shutdown_connections 2024-07-26T11:12:07.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:07.918+0000 7f31a3e00640 1 -- 172.21.15.134:0/1608278641 wait complete. 2024-07-26T11:12:08.741 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-07-26T11:12:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:08 smithi134 ceph-mon[31966]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:12:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:08 smithi134 ceph-mon[31966]: pgmap v365: 129 pgs: 129 active+clean; 612 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-07-26T11:12:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:08 smithi189 ceph-mon[36790]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:12:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:08 smithi189 ceph-mon[36790]: pgmap v365: 129 pgs: 129 active+clean; 612 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-07-26T11:12:09.201 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:08 smithi193 ceph-mon[37473]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:12:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:08 smithi193 ceph-mon[37473]: pgmap v365: 129 pgs: 129 active+clean; 612 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 2.8 KiB/s wr, 12 op/s 2024-07-26T11:12:11.188 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:10 smithi134 ceph-mon[31966]: pgmap v366: 129 pgs: 129 active+clean; 612 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-07-26T11:12:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:10 smithi193 ceph-mon[37473]: pgmap v366: 129 pgs: 129 active+clean; 612 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-07-26T11:12:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:10 smithi189 ceph-mon[36790]: pgmap v366: 129 pgs: 129 active+clean; 612 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.8 KiB/s wr, 10 op/s 2024-07-26T11:12:11.514 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.511+0000 7fbc0a000640 1 -- 172.21.15.134:0/3227629918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc04109840 msgr2=0x7fbc04110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:11.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.511+0000 7fbc0a000640 1 --2- 172.21.15.134:0/3227629918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc04109840 0x7fbc04110070 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc009ca0 tx=0x7fbbfc02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:12:11.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.512+0000 7fbc0a000640 1 -- 172.21.15.134:0/3227629918 shutdown_connections 2024-07-26T11:12:11.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.512+0000 7fbc0a000640 1 --2- 172.21.15.134:0/3227629918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc04109840 0x7fbc04110070 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:11.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.512+0000 7fbc0a000640 1 --2- 172.21.15.134:0/3227629918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc04108ef0 0x7fbc04109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:11.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.512+0000 7fbc0a000640 1 --2- 172.21.15.134:0/3227629918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc041035b0 0x7fbc041039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:11.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.512+0000 7fbc0a000640 1 -- 172.21.15.134:0/3227629918 >> 172.21.15.134:0/3227629918 conn(0x7fbc040ff000 msgr2=0x7fbc04101450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:11.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.512+0000 7fbc0a000640 1 -- 172.21.15.134:0/3227629918 shutdown_connections 2024-07-26T11:12:11.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.512+0000 7fbc0a000640 1 -- 172.21.15.134:0/3227629918 wait complete. 2024-07-26T11:12:11.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.513+0000 7fbc0a000640 1 Processor -- start 2024-07-26T11:12:11.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.513+0000 7fbc0a000640 1 -- start start 2024-07-26T11:12:11.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.514+0000 7fbc0a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc041035b0 0x7fbc04199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:11.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.514+0000 7fbc0a000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc04108ef0 0x7fbc0419a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:11.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.514+0000 7fbc0a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc04109840 0x7fbc0419f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:11.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.514+0000 7fbc0a000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbc04113540 con 0x7fbc04108ef0 2024-07-26T11:12:11.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.514+0000 7fbc0a000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbc041133c0 con 0x7fbc041035b0 2024-07-26T11:12:11.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.514+0000 7fbc0a000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbc041136c0 con 0x7fbc04109840 2024-07-26T11:12:11.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.515+0000 7fbc02a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc04108ef0 0x7fbc0419a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:11.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.515+0000 7fbc03400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc041035b0 0x7fbc04199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:11.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.515+0000 7fbc02a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc04108ef0 0x7fbc0419a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38236/0 (socket says 172.21.15.134:38236) 2024-07-26T11:12:11.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.515+0000 7fbc02a00640 1 -- 172.21.15.134:0/4168881365 learned_addr learned my addr 172.21.15.134:0/4168881365 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:12:11.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.515+0000 7fbc03e00640 1 --2- 172.21.15.134:0/4168881365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc04109840 0x7fbc0419f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:11.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.515+0000 7fbc02a00640 1 -- 172.21.15.134:0/4168881365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc04109840 msgr2=0x7fbc0419f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:11.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.515+0000 7fbc02a00640 1 --2- 172.21.15.134:0/4168881365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc04109840 0x7fbc0419f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:11.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.515+0000 7fbc02a00640 1 -- 172.21.15.134:0/4168881365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc041035b0 msgr2=0x7fbc04199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:11.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.515+0000 7fbc02a00640 1 --2- 172.21.15.134:0/4168881365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc041035b0 0x7fbc04199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:11.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.515+0000 7fbc02a00640 1 -- 172.21.15.134:0/4168881365 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0419fe00 con 0x7fbc04108ef0 2024-07-26T11:12:11.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.516+0000 7fbc02a00640 1 --2- 172.21.15.134:0/4168881365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc04108ef0 0x7fbc0419a430 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000e9f0 tx=0x7fbbf000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:11.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.516+0000 7fbbfbe00640 1 -- 172.21.15.134:0/4168881365 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf000ccc0 con 0x7fbc04108ef0 2024-07-26T11:12:11.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.516+0000 7fbbfbe00640 1 -- 172.21.15.134:0/4168881365 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbf00108e0 con 0x7fbc04108ef0 2024-07-26T11:12:11.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.516+0000 7fbc0a000640 1 -- 172.21.15.134:0/4168881365 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc041a00f0 con 0x7fbc04108ef0 2024-07-26T11:12:11.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.516+0000 7fbbfbe00640 1 -- 172.21.15.134:0/4168881365 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0018860 con 0x7fbc04108ef0 2024-07-26T11:12:11.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.516+0000 7fbc0a000640 1 -- 172.21.15.134:0/4168881365 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc041a6850 con 0x7fbc04108ef0 2024-07-26T11:12:11.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.517+0000 7fbc0a000640 1 -- 172.21.15.134:0/4168881365 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc041039c0 con 0x7fbc04108ef0 2024-07-26T11:12:11.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.523+0000 7fbbfbe00640 1 -- 172.21.15.134:0/4168881365 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fbbf0010430 con 0x7fbc04108ef0 2024-07-26T11:12:11.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.524+0000 7fbbfbe00640 1 --2- 172.21.15.134:0/4168881365 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbbe80b3cb0 0x7fbbe80b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:11.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.525+0000 7fbbfbe00640 1 -- 172.21.15.134:0/4168881365 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fbbf00e4a70 con 0x7fbc04108ef0 2024-07-26T11:12:11.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.525+0000 7fbc03400640 1 --2- 172.21.15.134:0/4168881365 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbbe80b3cb0 0x7fbbe80b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:11.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.525+0000 7fbc03400640 1 --2- 172.21.15.134:0/4168881365 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbbe80b3cb0 0x7fbbe80b61c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbbf40095f0 tx=0x7fbbf4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:11.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.526+0000 7fbbfbe00640 1 -- 172.21.15.134:0/4168881365 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbbf00ab410 con 0x7fbc04108ef0 2024-07-26T11:12:11.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.751+0000 7fbc0a000640 1 -- 172.21.15.134:0/4168881365 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fbc04063230 con 0x7fbbe80b3cb0 2024-07-26T11:12:11.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.810+0000 7fbbfbe00640 1 -- 172.21.15.134:0/4168881365 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbc04063230 con 0x7fbbe80b3cb0 2024-07-26T11:12:11.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.813+0000 7fbc0a000640 1 -- 172.21.15.134:0/4168881365 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbbe80b3cb0 msgr2=0x7fbbe80b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:11.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.813+0000 7fbc0a000640 1 --2- 172.21.15.134:0/4168881365 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbbe80b3cb0 0x7fbbe80b61c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbbf40095f0 tx=0x7fbbf4005f90 comp rx=0 tx=0).stop 2024-07-26T11:12:11.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.813+0000 7fbc0a000640 1 -- 172.21.15.134:0/4168881365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc04108ef0 msgr2=0x7fbc0419a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:11.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.813+0000 7fbc0a000640 1 --2- 172.21.15.134:0/4168881365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc04108ef0 0x7fbc0419a430 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000e9f0 tx=0x7fbbf000eeb0 comp rx=0 tx=0).stop 2024-07-26T11:12:11.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.813+0000 7fbc0a000640 1 -- 172.21.15.134:0/4168881365 shutdown_connections 2024-07-26T11:12:11.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.813+0000 7fbc0a000640 1 --2- 172.21.15.134:0/4168881365 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbbe80b3cb0 0x7fbbe80b61c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:11.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.813+0000 7fbc0a000640 1 --2- 172.21.15.134:0/4168881365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc04109840 0x7fbc0419f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:11.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.813+0000 7fbc0a000640 1 --2- 172.21.15.134:0/4168881365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc04108ef0 0x7fbc0419a430 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:11.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.813+0000 7fbc0a000640 1 --2- 172.21.15.134:0/4168881365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc041035b0 0x7fbc04199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:11.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.813+0000 7fbc0a000640 1 -- 172.21.15.134:0/4168881365 >> 172.21.15.134:0/4168881365 conn(0x7fbc040ff000 msgr2=0x7fbc04101420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:11.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.814+0000 7fbc0a000640 1 -- 172.21.15.134:0/4168881365 shutdown_connections 2024-07-26T11:12:11.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:11.814+0000 7fbc0a000640 1 -- 172.21.15.134:0/4168881365 wait complete. 2024-07-26T11:12:12.647 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-07-26T11:12:13.112 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:13.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:12 smithi189 ceph-mon[36790]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:12:13.247 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:12 smithi189 ceph-mon[36790]: pgmap v367: 129 pgs: 129 active+clean; 612 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-07-26T11:12:13.263 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:12 smithi134 ceph-mon[31966]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:12:13.263 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:12 smithi134 ceph-mon[31966]: pgmap v367: 129 pgs: 129 active+clean; 612 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-07-26T11:12:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:12 smithi193 ceph-mon[37473]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:12:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:12 smithi193 ceph-mon[37473]: pgmap v367: 129 pgs: 129 active+clean; 612 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-07-26T11:12:15.113 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:14 smithi134 ceph-mon[31966]: pgmap v368: 129 pgs: 129 active+clean; 630 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 4.5 KiB/s wr, 13 op/s 2024-07-26T11:12:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:14 smithi193 ceph-mon[37473]: pgmap v368: 129 pgs: 129 active+clean; 630 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 4.5 KiB/s wr, 13 op/s 2024-07-26T11:12:15.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.328+0000 7f0953e00640 1 -- 172.21.15.134:0/1846551140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f095410b860 msgr2=0x7f0954112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:15.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.328+0000 7f0953e00640 1 --2- 172.21.15.134:0/1846551140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f095410b860 0x7f0954112090 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f094c009ca0 tx=0x7f094c02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:12:15.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.328+0000 7f0953e00640 1 -- 172.21.15.134:0/1846551140 shutdown_connections 2024-07-26T11:12:15.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.328+0000 7f0953e00640 1 --2- 172.21.15.134:0/1846551140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f095410af10 0x7f095410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:15.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.328+0000 7f0953e00640 1 --2- 172.21.15.134:0/1846551140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f095410b860 0x7f0954112090 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:15.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.328+0000 7f0953e00640 1 --2- 172.21.15.134:0/1846551140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09541055d0 0x7f09541059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:15.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.328+0000 7f0953e00640 1 -- 172.21.15.134:0/1846551140 >> 172.21.15.134:0/1846551140 conn(0x7f0954101040 msgr2=0x7f0954103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:15.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.328+0000 7f0953e00640 1 -- 172.21.15.134:0/1846551140 shutdown_connections 2024-07-26T11:12:15.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.328+0000 7f0953e00640 1 -- 172.21.15.134:0/1846551140 wait complete. 2024-07-26T11:12:15.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.329+0000 7f0953e00640 1 Processor -- start 2024-07-26T11:12:15.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.329+0000 7f0953e00640 1 -- start start 2024-07-26T11:12:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.330+0000 7f0953e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09541055d0 0x7f095419bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.330+0000 7f0953e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f095410af10 0x7f095419c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.330+0000 7f0953e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f095410b860 0x7f09541a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:15.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.330+0000 7f0953e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0954115580 con 0x7f095410b860 2024-07-26T11:12:15.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.330+0000 7f0953e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0954115400 con 0x7f09541055d0 2024-07-26T11:12:15.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.330+0000 7f0953e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0954115700 con 0x7f095410af10 2024-07-26T11:12:15.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.330+0000 7f0953400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f095410b860 0x7f09541a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:15.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.330+0000 7f094a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f095410af10 0x7f095419c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:15.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.330+0000 7f0953400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f095410b860 0x7f09541a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38258/0 (socket says 172.21.15.134:38258) 2024-07-26T11:12:15.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.330+0000 7f094a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f095410af10 0x7f095419c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:55760/0 (socket says 172.21.15.134:55760) 2024-07-26T11:12:15.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.330+0000 7f0953400640 1 -- 172.21.15.134:0/3913767362 learned_addr learned my addr 172.21.15.134:0/3913767362 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:12:15.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.330+0000 7f0952a00640 1 --2- 172.21.15.134:0/3913767362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09541055d0 0x7f095419bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:15.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.331+0000 7f094a000640 1 -- 172.21.15.134:0/3913767362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09541055d0 msgr2=0x7f095419bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:15.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.331+0000 7f094a000640 1 --2- 172.21.15.134:0/3913767362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09541055d0 0x7f095419bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:15.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.331+0000 7f094a000640 1 -- 172.21.15.134:0/3913767362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f095410b860 msgr2=0x7f09541a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:15.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.331+0000 7f094a000640 1 --2- 172.21.15.134:0/3913767362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f095410b860 0x7f09541a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:15.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.331+0000 7f094a000640 1 -- 172.21.15.134:0/3913767362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09541a1ee0 con 0x7f095410af10 2024-07-26T11:12:15.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.331+0000 7f094a000640 1 --2- 172.21.15.134:0/3913767362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f095410af10 0x7f095419c510 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f095406c270 tx=0x7f093c00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:15.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.331+0000 7f0958a00640 1 -- 172.21.15.134:0/3913767362 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c00cc40 con 0x7f095410af10 2024-07-26T11:12:15.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.331+0000 7f0953e00640 1 -- 172.21.15.134:0/3913767362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09541a21d0 con 0x7f095410af10 2024-07-26T11:12:15.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.331+0000 7f0958a00640 1 -- 172.21.15.134:0/3913767362 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f093c00cde0 con 0x7f095410af10 2024-07-26T11:12:15.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.332+0000 7f0958a00640 1 -- 172.21.15.134:0/3913767362 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c004a00 con 0x7f095410af10 2024-07-26T11:12:15.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.332+0000 7f0953e00640 1 -- 172.21.15.134:0/3913767362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09541a8930 con 0x7f095410af10 2024-07-26T11:12:15.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.333+0000 7f0953e00640 1 -- 172.21.15.134:0/3913767362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09541059e0 con 0x7f095410af10 2024-07-26T11:12:15.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.335+0000 7f0958a00640 1 -- 172.21.15.134:0/3913767362 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f093c004ba0 con 0x7f095410af10 2024-07-26T11:12:15.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.337+0000 7f0958a00640 1 --2- 172.21.15.134:0/3913767362 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f09380b3cb0 0x7f09380b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:15.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.337+0000 7f0958a00640 1 -- 172.21.15.134:0/3913767362 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f093c0e42d0 con 0x7f095410af10 2024-07-26T11:12:15.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.337+0000 7f0952a00640 1 --2- 172.21.15.134:0/3913767362 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f09380b3cb0 0x7f09380b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:15.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.338+0000 7f0952a00640 1 --2- 172.21.15.134:0/3913767362 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f09380b3cb0 0x7f09380b61c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0944009bd0 tx=0x7f0944005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:15.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.339+0000 7f0958a00640 1 -- 172.21.15.134:0/3913767362 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f093c0ab2c0 con 0x7f095410af10 2024-07-26T11:12:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:14 smithi189 ceph-mon[36790]: pgmap v368: 129 pgs: 129 active+clean; 630 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 4.5 KiB/s wr, 13 op/s 2024-07-26T11:12:15.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.568+0000 7f0953e00640 1 -- 172.21.15.134:0/3913767362 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f0954063230 con 0x7f09380b3cb0 2024-07-26T11:12:15.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.575+0000 7f0958a00640 1 -- 172.21.15.134:0/3913767362 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f0954063230 con 0x7f09380b3cb0 2024-07-26T11:12:15.577 INFO:teuthology.orchestra.run.smithi134.stdout:/volumes/qa/sv_1/163a060a-b704-4597-b8c8-f5cde5094a3f 2024-07-26T11:12:15.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.578+0000 7f0953e00640 1 -- 172.21.15.134:0/3913767362 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f09380b3cb0 msgr2=0x7f09380b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:15.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.578+0000 7f0953e00640 1 --2- 172.21.15.134:0/3913767362 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f09380b3cb0 0x7f09380b61c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0944009bd0 tx=0x7f0944005f90 comp rx=0 tx=0).stop 2024-07-26T11:12:15.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.578+0000 7f0953e00640 1 -- 172.21.15.134:0/3913767362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f095410af10 msgr2=0x7f095419c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:15.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.578+0000 7f0953e00640 1 --2- 172.21.15.134:0/3913767362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f095410af10 0x7f095419c510 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f095406c270 tx=0x7f093c00ee50 comp rx=0 tx=0).stop 2024-07-26T11:12:15.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.578+0000 7f0953e00640 1 -- 172.21.15.134:0/3913767362 shutdown_connections 2024-07-26T11:12:15.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.578+0000 7f0953e00640 1 --2- 172.21.15.134:0/3913767362 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f09380b3cb0 0x7f09380b61c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:15.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.578+0000 7f0953e00640 1 --2- 172.21.15.134:0/3913767362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f095410af10 0x7f095419c510 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:15.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.578+0000 7f0953e00640 1 --2- 172.21.15.134:0/3913767362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f095410b860 0x7f09541a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:15.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.578+0000 7f0953e00640 1 --2- 172.21.15.134:0/3913767362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09541055d0 0x7f095419bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:15.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.578+0000 7f0953e00640 1 -- 172.21.15.134:0/3913767362 >> 172.21.15.134:0/3913767362 conn(0x7f0954101040 msgr2=0x7f095410a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:15.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.578+0000 7f0953e00640 1 -- 172.21.15.134:0/3913767362 shutdown_connections 2024-07-26T11:12:15.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:15.578+0000 7f0953e00640 1 -- 172.21.15.134:0/3913767362 wait complete. 2024-07-26T11:12:16.459 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:12:16.911 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:17.251 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:16 smithi134 ceph-mon[31966]: from='client.24931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:12:17.251 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:16 smithi134 ceph-mon[31966]: pgmap v369: 129 pgs: 129 active+clean; 630 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 3.8 KiB/s wr, 11 op/s 2024-07-26T11:12:17.251 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:12:17.251 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:17.251 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:17.251 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:17.251 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:16 smithi193 ceph-mon[37473]: from='client.24931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:12:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:16 smithi193 ceph-mon[37473]: pgmap v369: 129 pgs: 129 active+clean; 630 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 3.8 KiB/s wr, 11 op/s 2024-07-26T11:12:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:12:17.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:17.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:17.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:17.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:17.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:16 smithi189 ceph-mon[36790]: from='client.24931 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:12:17.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:16 smithi189 ceph-mon[36790]: pgmap v369: 129 pgs: 129 active+clean; 630 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 3.8 KiB/s wr, 11 op/s 2024-07-26T11:12:17.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:12:17.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:17.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:17.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:17.419 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:19 smithi134 ceph-mon[31966]: pgmap v370: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-07-26T11:12:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:19 smithi193 ceph-mon[37473]: pgmap v370: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-07-26T11:12:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:19 smithi189 ceph-mon[36790]: pgmap v370: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-07-26T11:12:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:19.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.738+0000 7f31d2400640 1 -- 172.21.15.134:0/1742247694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31cc1043e0 msgr2=0x7f31cc1047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:19.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.738+0000 7f31d2400640 1 --2- 172.21.15.134:0/1742247694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31cc1043e0 0x7f31cc1047f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f31bc009900 tx=0x7f31bc02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:12:19.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.739+0000 7f31d2400640 1 -- 172.21.15.134:0/1742247694 shutdown_connections 2024-07-26T11:12:19.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.739+0000 7f31d2400640 1 --2- 172.21.15.134:0/1742247694 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31cc10d840 0x7f31cc10fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:19.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.739+0000 7f31d2400640 1 --2- 172.21.15.134:0/1742247694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31cc104d30 0x7f31cc10d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:19.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.739+0000 7f31d2400640 1 --2- 172.21.15.134:0/1742247694 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31cc1043e0 0x7f31cc1047f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:19.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.739+0000 7f31d2400640 1 -- 172.21.15.134:0/1742247694 >> 172.21.15.134:0/1742247694 conn(0x7f31cc079160 msgr2=0x7f31cc079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:19.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.739+0000 7f31d2400640 1 -- 172.21.15.134:0/1742247694 shutdown_connections 2024-07-26T11:12:19.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.739+0000 7f31d2400640 1 -- 172.21.15.134:0/1742247694 wait complete. 2024-07-26T11:12:19.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.740+0000 7f31d2400640 1 Processor -- start 2024-07-26T11:12:19.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.740+0000 7f31d2400640 1 -- start start 2024-07-26T11:12:19.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.741+0000 7f31d2400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31cc1043e0 0x7f31cc19c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:19.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.741+0000 7f31d2400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31cc104d30 0x7f31cc19c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:19.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.741+0000 7f31d2400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31cc10d840 0x7f31cc1a1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:19.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.741+0000 7f31d2400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f31cc113500 con 0x7f31cc10d840 2024-07-26T11:12:19.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.741+0000 7f31d2400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f31cc113380 con 0x7f31cc1043e0 2024-07-26T11:12:19.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.741+0000 7f31d2400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f31cc113680 con 0x7f31cc104d30 2024-07-26T11:12:19.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.741+0000 7f31cb400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31cc1043e0 0x7f31cc19c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:19.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.741+0000 7f31cb400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31cc1043e0 0x7f31cc19c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:60650/0 (socket says 172.21.15.134:60650) 2024-07-26T11:12:19.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.741+0000 7f31cb400640 1 -- 172.21.15.134:0/1620422179 learned_addr learned my addr 172.21.15.134:0/1620422179 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:12:19.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.741+0000 7f31c2a00640 1 --2- 172.21.15.134:0/1620422179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31cc104d30 0x7f31cc19c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:19.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.741+0000 7f31cbe00640 1 --2- 172.21.15.134:0/1620422179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31cc10d840 0x7f31cc1a1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:19.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.742+0000 7f31c2a00640 1 -- 172.21.15.134:0/1620422179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31cc1043e0 msgr2=0x7f31cc19c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:19.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.742+0000 7f31c2a00640 1 --2- 172.21.15.134:0/1620422179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31cc1043e0 0x7f31cc19c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:19.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.742+0000 7f31c2a00640 1 -- 172.21.15.134:0/1620422179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31cc10d840 msgr2=0x7f31cc1a1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:19.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.742+0000 7f31c2a00640 1 --2- 172.21.15.134:0/1620422179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31cc10d840 0x7f31cc1a1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:19.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.742+0000 7f31c2a00640 1 -- 172.21.15.134:0/1620422179 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31cc1a21e0 con 0x7f31cc104d30 2024-07-26T11:12:19.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.742+0000 7f31c2a00640 1 --2- 172.21.15.134:0/1620422179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31cc104d30 0x7f31cc19c810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f31cc06a760 tx=0x7f31b400db30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:19.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.742+0000 7f31c8c00640 1 -- 172.21.15.134:0/1620422179 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b4017070 con 0x7f31cc104d30 2024-07-26T11:12:19.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.742+0000 7f31c8c00640 1 -- 172.21.15.134:0/1620422179 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31b4009820 con 0x7f31cc104d30 2024-07-26T11:12:19.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.742+0000 7f31c8c00640 1 -- 172.21.15.134:0/1620422179 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31b4004b90 con 0x7f31cc104d30 2024-07-26T11:12:19.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.742+0000 7f31d2400640 1 -- 172.21.15.134:0/1620422179 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31cc1a24d0 con 0x7f31cc104d30 2024-07-26T11:12:19.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.743+0000 7f31d2400640 1 -- 172.21.15.134:0/1620422179 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31cc1a8c30 con 0x7f31cc104d30 2024-07-26T11:12:19.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.744+0000 7f31d2400640 1 -- 172.21.15.134:0/1620422179 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31cc1047f0 con 0x7f31cc104d30 2024-07-26T11:12:19.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.750+0000 7f31c8c00640 1 -- 172.21.15.134:0/1620422179 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f31b4004d30 con 0x7f31cc104d30 2024-07-26T11:12:19.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.752+0000 7f31c8c00640 1 --2- 172.21.15.134:0/1620422179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31b00b3d30 0x7f31b00b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:19.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.752+0000 7f31c8c00640 1 -- 172.21.15.134:0/1620422179 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f31b40e3ef0 con 0x7f31cc104d30 2024-07-26T11:12:19.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.752+0000 7f31cb400640 1 --2- 172.21.15.134:0/1620422179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31b00b3d30 0x7f31b00b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:19.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.752+0000 7f31c8c00640 1 -- 172.21.15.134:0/1620422179 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f31b40e4370 con 0x7f31cc104d30 2024-07-26T11:12:19.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:19.753+0000 7f31cb400640 1 --2- 172.21.15.134:0/1620422179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31b00b3d30 0x7f31b00b6240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f31bc000c00 tx=0x7f31bc005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:20.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:20.015+0000 7f31d2400640 1 -- 172.21.15.134:0/1620422179 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31cc19d3a0 con 0x7f31cc104d30 2024-07-26T11:12:20.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:20.016+0000 7f31c8c00640 1 -- 172.21.15.134:0/1620422179 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11990 (secure 0 0 0) 0x7f31cc19d3a0 con 0x7f31cc104d30 2024-07-26T11:12:20.019 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:12:20.019 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":29,"btime":"2024-07-26T11:11:55:972763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:11:55.128451+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24721,"mds_1":24719,"mds_2":24703,"mds_3":24688,"mds_4":24686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24635":{"gid":24635,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24641":{"gid":24641,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":32,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24719":{"gid":24719,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":28,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24721,"qdb_cluster":[24686,24688,24703,24719,24721]},"id":1}]} 2024-07-26T11:12:20.019 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 29 2024-07-26T11:12:20.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:20.019+0000 7f31d2400640 1 -- 172.21.15.134:0/1620422179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31b00b3d30 msgr2=0x7f31b00b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:20.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:20.019+0000 7f31d2400640 1 --2- 172.21.15.134:0/1620422179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31b00b3d30 0x7f31b00b6240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f31bc000c00 tx=0x7f31bc005a20 comp rx=0 tx=0).stop 2024-07-26T11:12:20.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:20.019+0000 7f31d2400640 1 -- 172.21.15.134:0/1620422179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31cc104d30 msgr2=0x7f31cc19c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:20.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:20.019+0000 7f31d2400640 1 --2- 172.21.15.134:0/1620422179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31cc104d30 0x7f31cc19c810 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f31cc06a760 tx=0x7f31b400db30 comp rx=0 tx=0).stop 2024-07-26T11:12:20.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:20.019+0000 7f31d2400640 1 -- 172.21.15.134:0/1620422179 shutdown_connections 2024-07-26T11:12:20.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:20.019+0000 7f31d2400640 1 --2- 172.21.15.134:0/1620422179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31b00b3d30 0x7f31b00b6240 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:20.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:20.019+0000 7f31d2400640 1 --2- 172.21.15.134:0/1620422179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31cc104d30 0x7f31cc19c810 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:20.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:20.019+0000 7f31d2400640 1 --2- 172.21.15.134:0/1620422179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f31cc10d840 0x7f31cc1a1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:20.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:20.019+0000 7f31d2400640 1 --2- 172.21.15.134:0/1620422179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31cc1043e0 0x7f31cc19c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:20.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:20.019+0000 7f31d2400640 1 -- 172.21.15.134:0/1620422179 >> 172.21.15.134:0/1620422179 conn(0x7f31cc079160 msgr2=0x7f31cc0fffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:20.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:20.019+0000 7f31d2400640 1 -- 172.21.15.134:0/1620422179 shutdown_connections 2024-07-26T11:12:20.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:20.019+0000 7f31d2400640 1 -- 172.21.15.134:0/1620422179 wait complete. 2024-07-26T11:12:20.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T11:12:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:21 smithi134 ceph-mon[31966]: pgmap v371: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-07-26T11:12:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:21 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1620422179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:12:21.245 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:21 smithi193 ceph-mon[37473]: pgmap v371: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-07-26T11:12:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:21 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1620422179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:12:21.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:21 smithi189 ceph-mon[36790]: pgmap v371: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.7 KiB/s wr, 10 op/s 2024-07-26T11:12:21.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:21 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1620422179' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:12:22.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:22.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:22.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:22 smithi134 ceph-mon[31966]: pgmap v372: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-07-26T11:12:22.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:12:22.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:12:22.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:22 smithi189 ceph-mon[36790]: pgmap v372: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-07-26T11:12:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:12:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:12:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:22 smithi193 ceph-mon[37473]: pgmap v372: 129 pgs: 129 active+clean; 639 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-07-26T11:12:22.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:12:22.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:12:22.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:12:23.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.551+0000 7f8971400640 1 -- 172.21.15.134:0/478649465 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f896c10f330 msgr2=0x7f896c10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:23.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.551+0000 7f8971400640 1 --2- 172.21.15.134:0/478649465 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f896c10f330 0x7f896c10f740 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8954009860 tx=0x7f895402a340 comp rx=0 tx=0).stop 2024-07-26T11:12:23.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.551+0000 7f8971400640 1 -- 172.21.15.134:0/478649465 shutdown_connections 2024-07-26T11:12:23.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.551+0000 7f8971400640 1 --2- 172.21.15.134:0/478649465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f896c10fc80 0x7f896c1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:23.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.551+0000 7f8971400640 1 --2- 172.21.15.134:0/478649465 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f896c10f330 0x7f896c10f740 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:23.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.551+0000 7f8971400640 1 --2- 172.21.15.134:0/478649465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c1099f0 0x7f896c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:23.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.551+0000 7f8971400640 1 -- 172.21.15.134:0/478649465 >> 172.21.15.134:0/478649465 conn(0x7f896c078ff0 msgr2=0x7f896c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:23.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.551+0000 7f8971400640 1 -- 172.21.15.134:0/478649465 shutdown_connections 2024-07-26T11:12:23.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.551+0000 7f8971400640 1 -- 172.21.15.134:0/478649465 wait complete. 2024-07-26T11:12:23.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.552+0000 7f8971400640 1 Processor -- start 2024-07-26T11:12:23.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.552+0000 7f8971400640 1 -- start start 2024-07-26T11:12:23.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.553+0000 7f8971400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f896c1099f0 0x7f896c1a0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:23.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.553+0000 7f8971400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c10f330 0x7f896c1a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:23.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.553+0000 7f8971400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f896c10fc80 0x7f896c1a5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:23.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.553+0000 7f8971400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f896c119940 con 0x7f896c1099f0 2024-07-26T11:12:23.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.553+0000 7f8971400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f896c1197c0 con 0x7f896c10f330 2024-07-26T11:12:23.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.553+0000 7f8971400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f896c119ac0 con 0x7f896c10fc80 2024-07-26T11:12:23.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.553+0000 7f896be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f896c1099f0 0x7f896c1a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:23.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.553+0000 7f8970a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f896c10fc80 0x7f896c1a5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:23.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.553+0000 7f896b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c10f330 0x7f896c1a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:23.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.553+0000 7f896be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f896c1099f0 0x7f896c1a0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:35026/0 (socket says 172.21.15.134:35026) 2024-07-26T11:12:23.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.553+0000 7f896be00640 1 -- 172.21.15.134:0/3669179053 learned_addr learned my addr 172.21.15.134:0/3669179053 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:12:23.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.554+0000 7f896be00640 1 -- 172.21.15.134:0/3669179053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f896c10fc80 msgr2=0x7f896c1a5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:23.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.554+0000 7f896be00640 1 --2- 172.21.15.134:0/3669179053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f896c10fc80 0x7f896c1a5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:23.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.554+0000 7f896be00640 1 -- 172.21.15.134:0/3669179053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c10f330 msgr2=0x7f896c1a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:23.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.554+0000 7f896be00640 1 --2- 172.21.15.134:0/3669179053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c10f330 0x7f896c1a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:23.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.554+0000 7f896be00640 1 -- 172.21.15.134:0/3669179053 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f896c1a6300 con 0x7f896c1099f0 2024-07-26T11:12:23.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.554+0000 7f896b400640 1 --2- 172.21.15.134:0/3669179053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c10f330 0x7f896c1a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:12:23.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.554+0000 7f896be00640 1 --2- 172.21.15.134:0/3669179053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f896c1099f0 0x7f896c1a0390 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f895c00e9c0 tx=0x7f895c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:23.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.554+0000 7f8968c00640 1 -- 172.21.15.134:0/3669179053 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c00cd80 con 0x7f896c1099f0 2024-07-26T11:12:23.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.554+0000 7f8968c00640 1 -- 172.21.15.134:0/3669179053 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f895c0035f0 con 0x7f896c1099f0 2024-07-26T11:12:23.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.554+0000 7f8968c00640 1 -- 172.21.15.134:0/3669179053 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895c010690 con 0x7f896c1099f0 2024-07-26T11:12:23.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.554+0000 7f8971400640 1 -- 172.21.15.134:0/3669179053 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f896c1a65f0 con 0x7f896c1099f0 2024-07-26T11:12:23.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.555+0000 7f8971400640 1 -- 172.21.15.134:0/3669179053 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f896c1acd50 con 0x7f896c1099f0 2024-07-26T11:12:23.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.556+0000 7f8971400640 1 -- 172.21.15.134:0/3669179053 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f896c109e00 con 0x7f896c1099f0 2024-07-26T11:12:23.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.563+0000 7f8968c00640 1 -- 172.21.15.134:0/3669179053 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f895c004000 con 0x7f896c1099f0 2024-07-26T11:12:23.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.565+0000 7f8968c00640 1 --2- 172.21.15.134:0/3669179053 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f89500b3d30 0x7f89500b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:23.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.566+0000 7f8968c00640 1 -- 172.21.15.134:0/3669179053 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f895c0e4590 con 0x7f896c1099f0 2024-07-26T11:12:23.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.566+0000 7f896b400640 1 --2- 172.21.15.134:0/3669179053 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f89500b3d30 0x7f89500b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:23.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.567+0000 7f896b400640 1 --2- 172.21.15.134:0/3669179053 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f89500b3d30 0x7f89500b6240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8954035040 tx=0x7f8954030340 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:23.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.569+0000 7f8968c00640 1 -- 172.21.15.134:0/3669179053 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f895c0ab610 con 0x7f896c1099f0 2024-07-26T11:12:23.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.770+0000 7f8971400640 1 -- 172.21.15.134:0/3669179053 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f896c1a14c0 con 0x7f896c1099f0 2024-07-26T11:12:23.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.774+0000 7f8968c00640 1 -- 172.21.15.134:0/3669179053 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25185 (secure 0 0 0) 0x7f895c0aaf10 con 0x7f896c1099f0 2024-07-26T11:12:23.777 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:12:23.777 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":90,"fsid":"105fbe5c-4b3e-11ef-bcb7-c7b262605968","created":"2024-07-26T11:00:05.955103+0000","modified":"2024-07-26T11:11:35.149490+0000","last_up_change":"2024-07-26T11:09:11.485512+0000","last_in_change":"2024-07-26T11:08:52.341356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T11:05:01.849170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T11:10:16.055725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T11:10:21.216611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T11:10:33.983067+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8abb1365-fc3e-4617-a9ac-76bece8cd7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6803","nonce":4183273954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6805","nonce":4183273954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6809","nonce":4183273954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6807","nonce":4183273954}]},"public_addr":"172.21.15.134:6803/4183273954","cluster_addr":"172.21.15.134:6805/4183273954","heartbeat_back_addr":"172.21.15.134:6809/4183273954","heartbeat_front_addr":"172.21.15.134:6807/4183273954","state":["exists","up"]},{"osd":1,"uuid":"6c5a7838-821e-44e1-97e4-655b9bfa9f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6801","nonce":35148049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6803","nonce":35148049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6807","nonce":35148049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6805","nonce":35148049}]},"public_addr":"172.21.15.189:6801/35148049","cluster_addr":"172.21.15.189:6803/35148049","heartbeat_back_addr":"172.21.15.189:6807/35148049","heartbeat_front_addr":"172.21.15.189:6805/35148049","state":["exists","up"]},{"osd":2,"uuid":"8bc42f7f-39fa-4d3b-bc17-21c68f9231fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6801","nonce":897516976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6803","nonce":897516976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6807","nonce":897516976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6805","nonce":897516976}]},"public_addr":"172.21.15.193:6801/897516976","cluster_addr":"172.21.15.193:6803/897516976","heartbeat_back_addr":"172.21.15.193:6807/897516976","heartbeat_front_addr":"172.21.15.193:6805/897516976","state":["exists","up"]},{"osd":3,"uuid":"0473d055-0b79-434f-9658-28269d2b1fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6811","nonce":3457190917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6813","nonce":3457190917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6817","nonce":3457190917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6815","nonce":3457190917}]},"public_addr":"172.21.15.134:6811/3457190917","cluster_addr":"172.21.15.134:6813/3457190917","heartbeat_back_addr":"172.21.15.134:6817/3457190917","heartbeat_front_addr":"172.21.15.134:6815/3457190917","state":["exists","up"]},{"osd":4,"uuid":"1d70eaae-56cc-4cdc-8f83-b88d2121f0ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6809","nonce":1786881347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6811","nonce":1786881347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6815","nonce":1786881347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6813","nonce":1786881347}]},"public_addr":"172.21.15.189:6809/1786881347","cluster_addr":"172.21.15.189:6811/1786881347","heartbeat_back_addr":"172.21.15.189:6815/1786881347","heartbeat_front_addr":"172.21.15.189:6813/1786881347","state":["exists","up"]},{"osd":5,"uuid":"a9b0b4cf-1608-45ed-b64e-afcf545d97fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6809","nonce":1092388797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6811","nonce":1092388797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6815","nonce":1092388797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6813","nonce":1092388797}]},"public_addr":"172.21.15.193:6809/1092388797","cluster_addr":"172.21.15.193:6811/1092388797","heartbeat_back_addr":"172.21.15.193:6815/1092388797","heartbeat_front_addr":"172.21.15.193:6813/1092388797","state":["exists","up"]},{"osd":6,"uuid":"0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6819","nonce":2471505438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6821","nonce":2471505438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6825","nonce":2471505438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6823","nonce":2471505438}]},"public_addr":"172.21.15.134:6819/2471505438","cluster_addr":"172.21.15.134:6821/2471505438","heartbeat_back_addr":"172.21.15.134:6825/2471505438","heartbeat_front_addr":"172.21.15.134:6823/2471505438","state":["exists","up"]},{"osd":7,"uuid":"394764a7-1976-4223-a3c0-5979d941a838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6817","nonce":1382694911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6819","nonce":1382694911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6823","nonce":1382694911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6821","nonce":1382694911}]},"public_addr":"172.21.15.189:6817/1382694911","cluster_addr":"172.21.15.189:6819/1382694911","heartbeat_back_addr":"172.21.15.189:6823/1382694911","heartbeat_front_addr":"172.21.15.189:6821/1382694911","state":["exists","up"]},{"osd":8,"uuid":"1c218082-0b4c-41c8-92b9-453ae8d15a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6817","nonce":896348138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6819","nonce":896348138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6823","nonce":896348138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6821","nonce":896348138}]},"public_addr":"172.21.15.193:6817/896348138","cluster_addr":"172.21.15.193:6819/896348138","heartbeat_back_addr":"172.21.15.193:6823/896348138","heartbeat_front_addr":"172.21.15.193:6821/896348138","state":["exists","up"]},{"osd":9,"uuid":"48f39082-4fa9-4443-b5b7-b141c10d9e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6827","nonce":511237025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6829","nonce":511237025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6833","nonce":511237025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6831","nonce":511237025}]},"public_addr":"172.21.15.134:6827/511237025","cluster_addr":"172.21.15.134:6829/511237025","heartbeat_back_addr":"172.21.15.134:6833/511237025","heartbeat_front_addr":"172.21.15.134:6831/511237025","state":["exists","up"]},{"osd":10,"uuid":"95157766-1856-42f7-aea1-9cf83b77c8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6825","nonce":431740858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6827","nonce":431740858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6831","nonce":431740858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6829","nonce":431740858}]},"public_addr":"172.21.15.189:6825/431740858","cluster_addr":"172.21.15.189:6827/431740858","heartbeat_back_addr":"172.21.15.189:6831/431740858","heartbeat_front_addr":"172.21.15.189:6829/431740858","state":["exists","up"]},{"osd":11,"uuid":"244fadc7-f079-4736-8e7e-c0ce5ae6f43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6825","nonce":1594590579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6827","nonce":1594590579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6831","nonce":1594590579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6829","nonce":1594590579}]},"public_addr":"172.21.15.193:6825/1594590579","cluster_addr":"172.21.15.193:6827/1594590579","heartbeat_back_addr":"172.21.15.193:6831/1594590579","heartbeat_front_addr":"172.21.15.193:6829/1594590579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:11.712203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:35.845013+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:58.165609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:31.929746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:56.941555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:20.688071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:54.681514+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:20.789384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:44.568246+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:18.644310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:45.365351+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:09:09.663805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3790862248":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6800/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:0/2906927791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3068153391":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3056560791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6801/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/1310359922":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/2318666906":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/2901579347":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3404218719":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3598071145":"2024-07-27T11:02:33.711793+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:12:23.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.777+0000 7f8971400640 1 -- 172.21.15.134:0/3669179053 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f89500b3d30 msgr2=0x7f89500b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:23.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.777+0000 7f8971400640 1 --2- 172.21.15.134:0/3669179053 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f89500b3d30 0x7f89500b6240 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8954035040 tx=0x7f8954030340 comp rx=0 tx=0).stop 2024-07-26T11:12:23.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.777+0000 7f8971400640 1 -- 172.21.15.134:0/3669179053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f896c1099f0 msgr2=0x7f896c1a0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:23.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.777+0000 7f8971400640 1 --2- 172.21.15.134:0/3669179053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f896c1099f0 0x7f896c1a0390 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f895c00e9c0 tx=0x7f895c00edc0 comp rx=0 tx=0).stop 2024-07-26T11:12:23.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.777+0000 7f8971400640 1 -- 172.21.15.134:0/3669179053 shutdown_connections 2024-07-26T11:12:23.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.777+0000 7f8971400640 1 --2- 172.21.15.134:0/3669179053 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f89500b3d30 0x7f89500b6240 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:23.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.777+0000 7f8971400640 1 --2- 172.21.15.134:0/3669179053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f896c10fc80 0x7f896c1a5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:23.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.777+0000 7f8971400640 1 --2- 172.21.15.134:0/3669179053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c10f330 0x7f896c1a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:23.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.777+0000 7f8971400640 1 --2- 172.21.15.134:0/3669179053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f896c1099f0 0x7f896c1a0390 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:23.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.777+0000 7f8971400640 1 -- 172.21.15.134:0/3669179053 >> 172.21.15.134:0/3669179053 conn(0x7f896c078ff0 msgr2=0x7f896c0798b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:23.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.777+0000 7f8971400640 1 -- 172.21.15.134:0/3669179053 shutdown_connections 2024-07-26T11:12:23.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:23.778+0000 7f8971400640 1 -- 172.21.15.134:0/3669179053 wait complete. 2024-07-26T11:12:24.166 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:23 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3669179053' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:12:24.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:23 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3669179053' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:12:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:23 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3669179053' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:12:24.628 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-26T11:12:24.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph pg dump --format=json 2024-07-26T11:12:25.104 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:24 smithi134 ceph-mon[31966]: pgmap v373: 129 pgs: 129 active+clean; 643 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.3 KiB/s wr, 12 op/s 2024-07-26T11:12:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:24 smithi189 ceph-mon[36790]: pgmap v373: 129 pgs: 129 active+clean; 643 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.3 KiB/s wr, 12 op/s 2024-07-26T11:12:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:24 smithi193 ceph-mon[37473]: pgmap v373: 129 pgs: 129 active+clean; 643 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 3.3 KiB/s wr, 12 op/s 2024-07-26T11:12:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:26 smithi193 ceph-mon[37473]: pgmap v374: 129 pgs: 129 active+clean; 643 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-07-26T11:12:27.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.326+0000 7f7b60a00640 1 -- 172.21.15.134:0/3980807158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b5c10b870 msgr2=0x7f7b5c1120a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:27.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.326+0000 7f7b60a00640 1 --2- 172.21.15.134:0/3980807158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b5c10b870 0x7f7b5c1120a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7b54009be0 tx=0x7f7b5402a6c0 comp rx=0 tx=0).stop 2024-07-26T11:12:27.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.326+0000 7f7b60a00640 1 -- 172.21.15.134:0/3980807158 shutdown_connections 2024-07-26T11:12:27.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.326+0000 7f7b60a00640 1 --2- 172.21.15.134:0/3980807158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b5c10af20 0x7f7b5c10b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:27.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.326+0000 7f7b60a00640 1 --2- 172.21.15.134:0/3980807158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b5c10b870 0x7f7b5c1120a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:27.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.326+0000 7f7b60a00640 1 --2- 172.21.15.134:0/3980807158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b5c1055e0 0x7f7b5c1059f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:27.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.326+0000 7f7b60a00640 1 -- 172.21.15.134:0/3980807158 >> 172.21.15.134:0/3980807158 conn(0x7f7b5c1010b0 msgr2=0x7f7b5c103500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:27.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.326+0000 7f7b60a00640 1 -- 172.21.15.134:0/3980807158 shutdown_connections 2024-07-26T11:12:27.330 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:26 smithi134 ceph-mon[31966]: pgmap v374: 129 pgs: 129 active+clean; 643 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-07-26T11:12:27.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.326+0000 7f7b60a00640 1 -- 172.21.15.134:0/3980807158 wait complete. 2024-07-26T11:12:27.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.328+0000 7f7b60a00640 1 Processor -- start 2024-07-26T11:12:27.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.328+0000 7f7b60a00640 1 -- start start 2024-07-26T11:12:27.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.328+0000 7f7b60a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b5c1055e0 0x7f7b5c19bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:27.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.328+0000 7f7b60a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b5c10af20 0x7f7b5c19c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:27.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.328+0000 7f7b60a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b5c10b870 0x7f7b5c1a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:27.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.328+0000 7f7b60a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5c115540 con 0x7f7b5c10af20 2024-07-26T11:12:27.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.328+0000 7f7b60a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5c1153c0 con 0x7f7b5c1055e0 2024-07-26T11:12:27.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.328+0000 7f7b60a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5c1156c0 con 0x7f7b5c10b870 2024-07-26T11:12:27.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.329+0000 7f7b5b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b5c1055e0 0x7f7b5c19bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:27.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.329+0000 7f7b5b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b5c1055e0 0x7f7b5c19bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:58650/0 (socket says 172.21.15.134:58650) 2024-07-26T11:12:27.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.329+0000 7f7b5b400640 1 -- 172.21.15.134:0/273158614 learned_addr learned my addr 172.21.15.134:0/273158614 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:12:27.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.329+0000 7f7b5aa00640 1 --2- 172.21.15.134:0/273158614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b5c10af20 0x7f7b5c19c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.329+0000 7f7b5be00640 1 --2- 172.21.15.134:0/273158614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b5c10b870 0x7f7b5c1a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.329+0000 7f7b5be00640 1 -- 172.21.15.134:0/273158614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b5c1055e0 msgr2=0x7f7b5c19bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.329+0000 7f7b5be00640 1 --2- 172.21.15.134:0/273158614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b5c1055e0 0x7f7b5c19bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.329+0000 7f7b5be00640 1 -- 172.21.15.134:0/273158614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b5c10af20 msgr2=0x7f7b5c19c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.329+0000 7f7b5be00640 1 --2- 172.21.15.134:0/273158614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b5c10af20 0x7f7b5c19c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.329+0000 7f7b5be00640 1 -- 172.21.15.134:0/273158614 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b5c1a1e60 con 0x7f7b5c10b870 2024-07-26T11:12:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.329+0000 7f7b5b400640 1 --2- 172.21.15.134:0/273158614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b5c1055e0 0x7f7b5c19bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:12:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.330+0000 7f7b5be00640 1 --2- 172.21.15.134:0/273158614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b5c10b870 0x7f7b5c1a1730 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7b5402abe0 tx=0x7f7b54003c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.330+0000 7f7b53e00640 1 -- 172.21.15.134:0/273158614 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b54041020 con 0x7f7b5c10b870 2024-07-26T11:12:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.330+0000 7f7b53e00640 1 -- 172.21.15.134:0/273158614 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b54003e10 con 0x7f7b5c10b870 2024-07-26T11:12:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.330+0000 7f7b60a00640 1 -- 172.21.15.134:0/273158614 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b5c1a20f0 con 0x7f7b5c10b870 2024-07-26T11:12:27.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.330+0000 7f7b53e00640 1 -- 172.21.15.134:0/273158614 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b54036040 con 0x7f7b5c10b870 2024-07-26T11:12:27.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.330+0000 7f7b60a00640 1 -- 172.21.15.134:0/273158614 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b5c1a88b0 con 0x7f7b5c10b870 2024-07-26T11:12:27.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.331+0000 7f7b60a00640 1 -- 172.21.15.134:0/273158614 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b5c106860 con 0x7f7b5c10b870 2024-07-26T11:12:27.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.334+0000 7f7b53e00640 1 -- 172.21.15.134:0/273158614 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f7b54044020 con 0x7f7b5c10b870 2024-07-26T11:12:27.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.335+0000 7f7b53e00640 1 --2- 172.21.15.134:0/273158614 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b400b3cb0 0x7f7b400b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:27.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.335+0000 7f7b53e00640 1 -- 172.21.15.134:0/273158614 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7b54105150 con 0x7f7b5c10b870 2024-07-26T11:12:27.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.336+0000 7f7b5b400640 1 --2- 172.21.15.134:0/273158614 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b400b3cb0 0x7f7b400b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:27.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.336+0000 7f7b5b400640 1 --2- 172.21.15.134:0/273158614 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b400b3cb0 0x7f7b400b61c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c0096c0 tx=0x7f7b4c009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:27.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.337+0000 7f7b53e00640 1 -- 172.21.15.134:0/273158614 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7b5404b020 con 0x7f7b5c10b870 2024-07-26T11:12:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:26 smithi189 ceph-mon[36790]: pgmap v374: 129 pgs: 129 active+clean; 643 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.6 KiB/s wr, 10 op/s 2024-07-26T11:12:27.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.531+0000 7f7b60a00640 1 -- 172.21.15.134:0/273158614 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7b5c063230 con 0x7f7b400b3cb0 2024-07-26T11:12:27.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.543+0000 7f7b53e00640 1 -- 172.21.15.134:0/273158614 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340416 (secure 0 0 0) 0x7f7b5c063230 con 0x7f7b400b3cb0 2024-07-26T11:12:27.548 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:12:27.553 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-07-26T11:12:27.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.551+0000 7f7b60a00640 1 -- 172.21.15.134:0/273158614 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b400b3cb0 msgr2=0x7f7b400b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:27.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.551+0000 7f7b60a00640 1 --2- 172.21.15.134:0/273158614 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b400b3cb0 0x7f7b400b61c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c0096c0 tx=0x7f7b4c009260 comp rx=0 tx=0).stop 2024-07-26T11:12:27.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.551+0000 7f7b60a00640 1 -- 172.21.15.134:0/273158614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b5c10b870 msgr2=0x7f7b5c1a1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:27.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.551+0000 7f7b60a00640 1 --2- 172.21.15.134:0/273158614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b5c10b870 0x7f7b5c1a1730 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7b5402abe0 tx=0x7f7b54003c20 comp rx=0 tx=0).stop 2024-07-26T11:12:27.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.552+0000 7f7b60a00640 1 -- 172.21.15.134:0/273158614 shutdown_connections 2024-07-26T11:12:27.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.552+0000 7f7b60a00640 1 --2- 172.21.15.134:0/273158614 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b400b3cb0 0x7f7b400b61c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:27.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.552+0000 7f7b60a00640 1 --2- 172.21.15.134:0/273158614 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b5c10b870 0x7f7b5c1a1730 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:27.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.552+0000 7f7b60a00640 1 --2- 172.21.15.134:0/273158614 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b5c10af20 0x7f7b5c19c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:27.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.552+0000 7f7b60a00640 1 --2- 172.21.15.134:0/273158614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b5c1055e0 0x7f7b5c19bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:27.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.552+0000 7f7b60a00640 1 -- 172.21.15.134:0/273158614 >> 172.21.15.134:0/273158614 conn(0x7f7b5c1010b0 msgr2=0x7f7b5c10a550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:27.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.552+0000 7f7b60a00640 1 -- 172.21.15.134:0/273158614 shutdown_connections 2024-07-26T11:12:27.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:27.552+0000 7f7b60a00640 1 -- 172.21.15.134:0/273158614 wait complete. 2024-07-26T11:12:28.392 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-07-26T11:12:25.931807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658532,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":444,"num_read_kb":322,"num_write":372,"num_write_kb":2291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":2954034,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":307,"ondisk_log_size":307,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":363380,"kb_used_data":14228,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578444,"statfs":{"total":1151940427776,"available":1151568326656,"internally_reserved":0,"allocated":14569472,"data_stored":8255162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13445,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":112,"num_read_kb":64,"num_write":14,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006558"},"pg_stats":[{"pgid":"2.1f","version":"90'13","reported_seq":99,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:20.155154+0000","last_change":"2024-07-26T11:10:34.985984+0000","last_active":"2024-07-26T11:12:20.155154+0000","last_peered":"2024-07-26T11:12:20.155154+0000","last_clean":"2024-07-26T11:12:20.155154+0000","last_became_active":"2024-07-26T11:10:34.980894+0000","last_became_peered":"2024-07-26T11:10:34.980894+0000","last_unstale":"2024-07-26T11:12:20.155154+0000","last_undegraded":"2024-07-26T11:12:20.155154+0000","last_fullsized":"2024-07-26T11:12:20.155154+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:55.718875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.148788+0000","last_change":"2024-07-26T11:10:22.320946+0000","last_active":"2024-07-26T11:11:36.148788+0000","last_peered":"2024-07-26T11:11:36.148788+0000","last_clean":"2024-07-26T11:11:36.148788+0000","last_became_active":"2024-07-26T11:10:22.316181+0000","last_became_peered":"2024-07-26T11:10:22.316181+0000","last_unstale":"2024-07-26T11:11:36.148788+0000","last_undegraded":"2024-07-26T11:11:36.148788+0000","last_fullsized":"2024-07-26T11:11:36.148788+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:13.641862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.155656+0000","last_change":"2024-07-26T11:10:35.984460+0000","last_active":"2024-07-26T11:11:35.155656+0000","last_peered":"2024-07-26T11:11:35.155656+0000","last_clean":"2024-07-26T11:11:35.155656+0000","last_became_active":"2024-07-26T11:10:35.983922+0000","last_became_peered":"2024-07-26T11:10:35.983922+0000","last_unstale":"2024-07-26T11:11:35.155656+0000","last_undegraded":"2024-07-26T11:11:35.155656+0000","last_fullsized":"2024-07-26T11:11:35.155656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:51:46.140910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:48.051375+0000","last_change":"2024-07-26T11:10:18.058553+0000","last_active":"2024-07-26T11:11:48.051375+0000","last_peered":"2024-07-26T11:11:48.051375+0000","last_clean":"2024-07-26T11:11:48.051375+0000","last_became_active":"2024-07-26T11:10:18.058034+0000","last_became_peered":"2024-07-26T11:10:18.058034+0000","last_unstale":"2024-07-26T11:11:48.051375+0000","last_undegraded":"2024-07-26T11:11:48.051375+0000","last_fullsized":"2024-07-26T11:11:48.051375+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:48:58.868988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.186702+0000","last_change":"2024-07-26T11:10:22.326760+0000","last_active":"2024-07-26T11:11:35.186702+0000","last_peered":"2024-07-26T11:11:35.186702+0000","last_clean":"2024-07-26T11:11:35.186702+0000","last_became_active":"2024-07-26T11:10:22.326275+0000","last_became_peered":"2024-07-26T11:10:22.326275+0000","last_unstale":"2024-07-26T11:11:35.186702+0000","last_undegraded":"2024-07-26T11:11:35.186702+0000","last_fullsized":"2024-07-26T11:11:35.186702+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:51:57.461453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.406545+0000","last_change":"2024-07-26T11:10:35.997585+0000","last_active":"2024-07-26T11:11:35.406545+0000","last_peered":"2024-07-26T11:11:35.406545+0000","last_clean":"2024-07-26T11:11:35.406545+0000","last_became_active":"2024-07-26T11:10:35.997274+0000","last_became_peered":"2024-07-26T11:10:35.997274+0000","last_unstale":"2024-07-26T11:11:35.406545+0000","last_undegraded":"2024-07-26T11:11:35.406545+0000","last_fullsized":"2024-07-26T11:11:35.406545+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:43.553396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.077630+0000","last_change":"2024-07-26T11:10:18.034964+0000","last_active":"2024-07-26T11:11:51.077630+0000","last_peered":"2024-07-26T11:11:51.077630+0000","last_clean":"2024-07-26T11:11:51.077630+0000","last_became_active":"2024-07-26T11:10:18.034569+0000","last_became_peered":"2024-07-26T11:10:18.034569+0000","last_unstale":"2024-07-26T11:11:51.077630+0000","last_undegraded":"2024-07-26T11:11:51.077630+0000","last_fullsized":"2024-07-26T11:11:51.077630+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:00:46.542458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.949541+0000","last_change":"2024-07-26T11:10:22.310695+0000","last_active":"2024-07-26T11:11:35.949541+0000","last_peered":"2024-07-26T11:11:35.949541+0000","last_clean":"2024-07-26T11:11:35.949541+0000","last_became_active":"2024-07-26T11:10:22.310142+0000","last_became_peered":"2024-07-26T11:10:22.310142+0000","last_unstale":"2024-07-26T11:11:35.949541+0000","last_undegraded":"2024-07-26T11:11:35.949541+0000","last_fullsized":"2024-07-26T11:11:35.949541+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:49:18.890802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.407084+0000","last_change":"2024-07-26T11:10:35.991418+0000","last_active":"2024-07-26T11:11:35.407084+0000","last_peered":"2024-07-26T11:11:35.407084+0000","last_clean":"2024-07-26T11:11:35.407084+0000","last_became_active":"2024-07-26T11:10:35.990934+0000","last_became_peered":"2024-07-26T11:10:35.990934+0000","last_unstale":"2024-07-26T11:11:35.407084+0000","last_undegraded":"2024-07-26T11:11:35.407084+0000","last_fullsized":"2024-07-26T11:11:35.407084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:56.598738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.082903+0000","last_change":"2024-07-26T11:10:34.965637+0000","last_active":"2024-07-26T11:11:50.082903+0000","last_peered":"2024-07-26T11:11:50.082903+0000","last_clean":"2024-07-26T11:11:50.082903+0000","last_became_active":"2024-07-26T11:10:34.961157+0000","last_became_peered":"2024-07-26T11:10:34.961157+0000","last_unstale":"2024-07-26T11:11:50.082903+0000","last_undegraded":"2024-07-26T11:11:50.082903+0000","last_fullsized":"2024-07-26T11:11:50.082903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:26:17.007258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158828+0000","last_change":"2024-07-26T11:10:22.303606+0000","last_active":"2024-07-26T11:11:35.158828+0000","last_peered":"2024-07-26T11:11:35.158828+0000","last_clean":"2024-07-26T11:11:35.158828+0000","last_became_active":"2024-07-26T11:10:22.303183+0000","last_became_peered":"2024-07-26T11:10:22.303183+0000","last_unstale":"2024-07-26T11:11:35.158828+0000","last_undegraded":"2024-07-26T11:11:35.158828+0000","last_fullsized":"2024-07-26T11:11:35.158828+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:31:01.153694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.406697+0000","last_change":"2024-07-26T11:10:35.981446+0000","last_active":"2024-07-26T11:11:35.406697+0000","last_peered":"2024-07-26T11:11:35.406697+0000","last_clean":"2024-07-26T11:11:35.406697+0000","last_became_active":"2024-07-26T11:10:35.980994+0000","last_became_peered":"2024-07-26T11:10:35.980994+0000","last_unstale":"2024-07-26T11:11:35.406697+0000","last_undegraded":"2024-07-26T11:11:35.406697+0000","last_fullsized":"2024-07-26T11:11:35.406697+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:38:57.405325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161463+0000","last_change":"2024-07-26T11:10:22.306758+0000","last_active":"2024-07-26T11:11:35.161463+0000","last_peered":"2024-07-26T11:11:35.161463+0000","last_clean":"2024-07-26T11:11:35.161463+0000","last_became_active":"2024-07-26T11:10:22.306370+0000","last_became_peered":"2024-07-26T11:10:22.306370+0000","last_unstale":"2024-07-26T11:11:35.161463+0000","last_undegraded":"2024-07-26T11:11:35.161463+0000","last_fullsized":"2024-07-26T11:11:35.161463+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:51:43.069470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:47.038158+0000","last_change":"2024-07-26T11:10:18.035664+0000","last_active":"2024-07-26T11:11:47.038158+0000","last_peered":"2024-07-26T11:11:47.038158+0000","last_clean":"2024-07-26T11:11:47.038158+0000","last_became_active":"2024-07-26T11:10:18.035113+0000","last_became_peered":"2024-07-26T11:10:18.035113+0000","last_unstale":"2024-07-26T11:11:47.038158+0000","last_undegraded":"2024-07-26T11:11:47.038158+0000","last_fullsized":"2024-07-26T11:11:47.038158+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:09:30.641034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.407009+0000","last_change":"2024-07-26T11:10:35.990907+0000","last_active":"2024-07-26T11:11:35.407009+0000","last_peered":"2024-07-26T11:11:35.407009+0000","last_clean":"2024-07-26T11:11:35.407009+0000","last_became_active":"2024-07-26T11:10:35.990417+0000","last_became_peered":"2024-07-26T11:10:35.990417+0000","last_unstale":"2024-07-26T11:11:35.407009+0000","last_undegraded":"2024-07-26T11:11:35.407009+0000","last_fullsized":"2024-07-26T11:11:35.407009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:24:39.319727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.147986+0000","last_change":"2024-07-26T11:10:22.333575+0000","last_active":"2024-07-26T11:11:36.147986+0000","last_peered":"2024-07-26T11:11:36.147986+0000","last_clean":"2024-07-26T11:11:36.147986+0000","last_became_active":"2024-07-26T11:10:22.332802+0000","last_became_peered":"2024-07-26T11:10:22.332802+0000","last_unstale":"2024-07-26T11:11:36.147986+0000","last_undegraded":"2024-07-26T11:11:36.147986+0000","last_fullsized":"2024-07-26T11:11:36.147986+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:14:03.238840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:49.069720+0000","last_change":"2024-07-26T11:10:18.047942+0000","last_active":"2024-07-26T11:11:49.069720+0000","last_peered":"2024-07-26T11:11:49.069720+0000","last_clean":"2024-07-26T11:11:49.069720+0000","last_became_active":"2024-07-26T11:10:18.047500+0000","last_became_peered":"2024-07-26T11:10:18.047500+0000","last_unstale":"2024-07-26T11:11:49.069720+0000","last_undegraded":"2024-07-26T11:11:49.069720+0000","last_fullsized":"2024-07-26T11:11:49.069720+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:32:30.773648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.406871+0000","last_change":"2024-07-26T11:10:36.000946+0000","last_active":"2024-07-26T11:11:35.406871+0000","last_peered":"2024-07-26T11:11:35.406871+0000","last_clean":"2024-07-26T11:11:35.406871+0000","last_became_active":"2024-07-26T11:10:36.000251+0000","last_became_peered":"2024-07-26T11:10:36.000251+0000","last_unstale":"2024-07-26T11:11:35.406871+0000","last_undegraded":"2024-07-26T11:11:35.406871+0000","last_fullsized":"2024-07-26T11:11:35.406871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:03:40.523747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162117+0000","last_change":"2024-07-26T11:10:22.326370+0000","last_active":"2024-07-26T11:11:35.162117+0000","last_peered":"2024-07-26T11:11:35.162117+0000","last_clean":"2024-07-26T11:11:35.162117+0000","last_became_active":"2024-07-26T11:10:22.325961+0000","last_became_peered":"2024-07-26T11:10:22.325961+0000","last_unstale":"2024-07-26T11:11:35.162117+0000","last_undegraded":"2024-07-26T11:11:35.162117+0000","last_fullsized":"2024-07-26T11:11:35.162117+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:16:36.946750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.088969+0000","last_change":"2024-07-26T11:10:18.051940+0000","last_active":"2024-07-26T11:11:50.088969+0000","last_peered":"2024-07-26T11:11:50.088969+0000","last_clean":"2024-07-26T11:11:50.088969+0000","last_became_active":"2024-07-26T11:10:18.051565+0000","last_became_peered":"2024-07-26T11:10:18.051565+0000","last_unstale":"2024-07-26T11:11:50.088969+0000","last_undegraded":"2024-07-26T11:11:50.088969+0000","last_fullsized":"2024-07-26T11:11:50.088969+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:17:56.418638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.383408+0000","last_change":"2024-07-26T11:10:36.000156+0000","last_active":"2024-07-26T11:11:35.383408+0000","last_peered":"2024-07-26T11:11:35.383408+0000","last_clean":"2024-07-26T11:11:35.383408+0000","last_became_active":"2024-07-26T11:10:35.999600+0000","last_became_peered":"2024-07-26T11:10:35.999600+0000","last_unstale":"2024-07-26T11:11:35.383408+0000","last_undegraded":"2024-07-26T11:11:35.383408+0000","last_fullsized":"2024-07-26T11:11:35.383408+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:35:48.504288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.157916+0000","last_change":"2024-07-26T11:10:22.298152+0000","last_active":"2024-07-26T11:11:35.157916+0000","last_peered":"2024-07-26T11:11:35.157916+0000","last_clean":"2024-07-26T11:11:35.157916+0000","last_became_active":"2024-07-26T11:10:22.297739+0000","last_became_peered":"2024-07-26T11:10:22.297739+0000","last_unstale":"2024-07-26T11:11:35.157916+0000","last_undegraded":"2024-07-26T11:11:35.157916+0000","last_fullsized":"2024-07-26T11:11:35.157916+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:48.160183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:47.040353+0000","last_change":"2024-07-26T11:10:18.057756+0000","last_active":"2024-07-26T11:11:47.040353+0000","last_peered":"2024-07-26T11:11:47.040353+0000","last_clean":"2024-07-26T11:11:47.040353+0000","last_became_active":"2024-07-26T11:10:18.057464+0000","last_became_peered":"2024-07-26T11:10:18.057464+0000","last_unstale":"2024-07-26T11:11:47.040353+0000","last_undegraded":"2024-07-26T11:11:47.040353+0000","last_fullsized":"2024-07-26T11:11:47.040353+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:39:11.523643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.406996+0000","last_change":"2024-07-26T11:10:36.000617+0000","last_active":"2024-07-26T11:11:35.406996+0000","last_peered":"2024-07-26T11:11:35.406996+0000","last_clean":"2024-07-26T11:11:35.406996+0000","last_became_active":"2024-07-26T11:10:36.000151+0000","last_became_peered":"2024-07-26T11:10:36.000151+0000","last_unstale":"2024-07-26T11:11:35.406996+0000","last_undegraded":"2024-07-26T11:11:35.406996+0000","last_fullsized":"2024-07-26T11:11:35.406996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:36.942215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.186420+0000","last_change":"2024-07-26T11:10:22.336452+0000","last_active":"2024-07-26T11:11:35.186420+0000","last_peered":"2024-07-26T11:11:35.186420+0000","last_clean":"2024-07-26T11:11:35.186420+0000","last_became_active":"2024-07-26T11:10:22.336006+0000","last_became_peered":"2024-07-26T11:10:22.336006+0000","last_unstale":"2024-07-26T11:11:35.186420+0000","last_undegraded":"2024-07-26T11:11:35.186420+0000","last_fullsized":"2024-07-26T11:11:35.186420+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:08:27.375730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.084181+0000","last_change":"2024-07-26T11:10:18.823017+0000","last_active":"2024-07-26T11:11:50.084181+0000","last_peered":"2024-07-26T11:11:50.084181+0000","last_clean":"2024-07-26T11:11:50.084181+0000","last_became_active":"2024-07-26T11:10:18.822559+0000","last_became_peered":"2024-07-26T11:10:18.822559+0000","last_unstale":"2024-07-26T11:11:50.084181+0000","last_undegraded":"2024-07-26T11:11:50.084181+0000","last_fullsized":"2024-07-26T11:11:50.084181+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:01.486263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.155695+0000","last_change":"2024-07-26T11:10:35.969698+0000","last_active":"2024-07-26T11:11:35.155695+0000","last_peered":"2024-07-26T11:11:35.155695+0000","last_clean":"2024-07-26T11:11:35.155695+0000","last_became_active":"2024-07-26T11:10:35.969240+0000","last_became_peered":"2024-07-26T11:10:35.969240+0000","last_unstale":"2024-07-26T11:11:35.155695+0000","last_undegraded":"2024-07-26T11:11:35.155695+0000","last_fullsized":"2024-07-26T11:11:35.155695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:25.210862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.148957+0000","last_change":"2024-07-26T11:10:22.314772+0000","last_active":"2024-07-26T11:11:36.148957+0000","last_peered":"2024-07-26T11:11:36.148957+0000","last_clean":"2024-07-26T11:11:36.148957+0000","last_became_active":"2024-07-26T11:10:22.313786+0000","last_became_peered":"2024-07-26T11:10:22.313786+0000","last_unstale":"2024-07-26T11:11:36.148957+0000","last_undegraded":"2024-07-26T11:11:36.148957+0000","last_fullsized":"2024-07-26T11:11:36.148957+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:19.610695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'15","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:15.586829+0000","last_change":"2024-07-26T11:10:18.049408+0000","last_active":"2024-07-26T11:12:15.586829+0000","last_peered":"2024-07-26T11:12:15.586829+0000","last_clean":"2024-07-26T11:12:15.586829+0000","last_became_active":"2024-07-26T11:10:18.048573+0000","last_became_peered":"2024-07-26T11:10:18.048573+0000","last_unstale":"2024-07-26T11:12:15.586829+0000","last_undegraded":"2024-07-26T11:12:15.586829+0000","last_fullsized":"2024-07-26T11:12:15.586829+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:26.408260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.147210+0000","last_change":"2024-07-26T11:10:35.992067+0000","last_active":"2024-07-26T11:11:36.147210+0000","last_peered":"2024-07-26T11:11:36.147210+0000","last_clean":"2024-07-26T11:11:36.147210+0000","last_became_active":"2024-07-26T11:10:35.989789+0000","last_became_peered":"2024-07-26T11:10:35.989789+0000","last_unstale":"2024-07-26T11:11:36.147210+0000","last_undegraded":"2024-07-26T11:11:36.147210+0000","last_fullsized":"2024-07-26T11:11:36.147210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:08:08.283592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.641901+0000","last_change":"2024-07-26T11:10:22.323512+0000","last_active":"2024-07-26T11:11:35.641901+0000","last_peered":"2024-07-26T11:11:35.641901+0000","last_clean":"2024-07-26T11:11:35.641901+0000","last_became_active":"2024-07-26T11:10:22.323126+0000","last_became_peered":"2024-07-26T11:10:22.323126+0000","last_unstale":"2024-07-26T11:11:35.641901+0000","last_undegraded":"2024-07-26T11:11:35.641901+0000","last_fullsized":"2024-07-26T11:11:35.641901+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:27:15.379345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.093631+0000","last_change":"2024-07-26T11:10:18.052573+0000","last_active":"2024-07-26T11:11:51.093631+0000","last_peered":"2024-07-26T11:11:51.093631+0000","last_clean":"2024-07-26T11:11:51.093631+0000","last_became_active":"2024-07-26T11:10:18.052302+0000","last_became_peered":"2024-07-26T11:10:18.052302+0000","last_unstale":"2024-07-26T11:11:51.093631+0000","last_undegraded":"2024-07-26T11:11:51.093631+0000","last_fullsized":"2024-07-26T11:11:51.093631+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:32.260966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162344+0000","last_change":"2024-07-26T11:10:35.994347+0000","last_active":"2024-07-26T11:11:35.162344+0000","last_peered":"2024-07-26T11:11:35.162344+0000","last_clean":"2024-07-26T11:11:35.162344+0000","last_became_active":"2024-07-26T11:10:35.993795+0000","last_became_peered":"2024-07-26T11:10:35.993795+0000","last_unstale":"2024-07-26T11:11:35.162344+0000","last_undegraded":"2024-07-26T11:11:35.162344+0000","last_fullsized":"2024-07-26T11:11:35.162344+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:35.399601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156114+0000","last_change":"2024-07-26T11:10:22.318109+0000","last_active":"2024-07-26T11:11:35.156114+0000","last_peered":"2024-07-26T11:11:35.156114+0000","last_clean":"2024-07-26T11:11:35.156114+0000","last_became_active":"2024-07-26T11:10:22.317674+0000","last_became_peered":"2024-07-26T11:10:22.317674+0000","last_unstale":"2024-07-26T11:11:35.156114+0000","last_undegraded":"2024-07-26T11:11:35.156114+0000","last_fullsized":"2024-07-26T11:11:35.156114+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:29.464772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.383704+0000","last_change":"2024-07-26T11:10:18.060073+0000","last_active":"2024-07-26T11:11:35.383704+0000","last_peered":"2024-07-26T11:11:35.383704+0000","last_clean":"2024-07-26T11:11:35.383704+0000","last_became_active":"2024-07-26T11:10:18.059381+0000","last_became_peered":"2024-07-26T11:10:18.059381+0000","last_unstale":"2024-07-26T11:11:35.383704+0000","last_undegraded":"2024-07-26T11:11:35.383704+0000","last_fullsized":"2024-07-26T11:11:35.383704+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:04:21.184759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.951307+0000","last_change":"2024-07-26T11:11:35.951234+0000","last_active":"2024-07-26T11:11:35.951307+0000","last_peered":"2024-07-26T11:11:35.951307+0000","last_clean":"2024-07-26T11:11:35.951307+0000","last_became_active":"2024-07-26T11:11:35.950821+0000","last_became_peered":"2024-07-26T11:11:35.950821+0000","last_unstale":"2024-07-26T11:11:35.951307+0000","last_undegraded":"2024-07-26T11:11:35.951307+0000","last_fullsized":"2024-07-26T11:11:35.951307+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:44:22.569238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156084+0000","last_change":"2024-07-26T11:10:22.326866+0000","last_active":"2024-07-26T11:11:35.156084+0000","last_peered":"2024-07-26T11:11:35.156084+0000","last_clean":"2024-07-26T11:11:35.156084+0000","last_became_active":"2024-07-26T11:10:22.326417+0000","last_became_peered":"2024-07-26T11:10:22.326417+0000","last_unstale":"2024-07-26T11:11:35.156084+0000","last_undegraded":"2024-07-26T11:11:35.156084+0000","last_fullsized":"2024-07-26T11:11:35.156084+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:49:02.871062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:22.158801+0000","last_change":"2024-07-26T11:10:18.044077+0000","last_active":"2024-07-26T11:12:22.158801+0000","last_peered":"2024-07-26T11:12:22.158801+0000","last_clean":"2024-07-26T11:12:22.158801+0000","last_became_active":"2024-07-26T11:10:18.043660+0000","last_became_peered":"2024-07-26T11:10:18.043660+0000","last_unstale":"2024-07-26T11:12:22.158801+0000","last_undegraded":"2024-07-26T11:12:22.158801+0000","last_fullsized":"2024-07-26T11:12:22.158801+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:46:54.901082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.641589+0000","last_change":"2024-07-26T11:10:35.984778+0000","last_active":"2024-07-26T11:11:35.641589+0000","last_peered":"2024-07-26T11:11:35.641589+0000","last_clean":"2024-07-26T11:11:35.641589+0000","last_became_active":"2024-07-26T11:10:35.984365+0000","last_became_peered":"2024-07-26T11:10:35.984365+0000","last_unstale":"2024-07-26T11:11:35.641589+0000","last_undegraded":"2024-07-26T11:11:35.641589+0000","last_fullsized":"2024-07-26T11:11:35.641589+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:33.174121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.381797+0000","last_change":"2024-07-26T11:10:34.976075+0000","last_active":"2024-07-26T11:11:35.381797+0000","last_peered":"2024-07-26T11:11:35.381797+0000","last_clean":"2024-07-26T11:11:35.381797+0000","last_became_active":"2024-07-26T11:10:34.973799+0000","last_became_peered":"2024-07-26T11:10:34.973799+0000","last_unstale":"2024-07-26T11:11:35.381797+0000","last_undegraded":"2024-07-26T11:11:35.381797+0000","last_fullsized":"2024-07-26T11:11:35.381797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:03:01.830655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:20.169717+0000","last_change":"2024-07-26T11:10:18.056473+0000","last_active":"2024-07-26T11:12:20.169717+0000","last_peered":"2024-07-26T11:12:20.169717+0000","last_clean":"2024-07-26T11:12:20.169717+0000","last_became_active":"2024-07-26T11:10:18.056038+0000","last_became_peered":"2024-07-26T11:10:18.056038+0000","last_unstale":"2024-07-26T11:12:20.169717+0000","last_undegraded":"2024-07-26T11:12:20.169717+0000","last_fullsized":"2024-07-26T11:12:20.169717+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:04.501018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":32,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.642321+0000","last_change":"2024-07-26T11:10:35.991741+0000","last_active":"2024-07-26T11:11:35.642321+0000","last_peered":"2024-07-26T11:11:35.642321+0000","last_clean":"2024-07-26T11:11:35.642321+0000","last_became_active":"2024-07-26T11:10:35.990995+0000","last_became_peered":"2024-07-26T11:10:35.990995+0000","last_unstale":"2024-07-26T11:11:35.642321+0000","last_undegraded":"2024-07-26T11:11:35.642321+0000","last_fullsized":"2024-07-26T11:11:35.642321+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:59:34.056646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.641111+0000","last_change":"2024-07-26T11:10:22.305053+0000","last_active":"2024-07-26T11:11:35.641111+0000","last_peered":"2024-07-26T11:11:35.641111+0000","last_clean":"2024-07-26T11:11:35.641111+0000","last_became_active":"2024-07-26T11:10:22.304035+0000","last_became_peered":"2024-07-26T11:10:22.304035+0000","last_unstale":"2024-07-26T11:11:35.641111+0000","last_undegraded":"2024-07-26T11:11:35.641111+0000","last_fullsized":"2024-07-26T11:11:35.641111+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:48.590799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.082116+0000","last_change":"2024-07-26T11:10:18.033581+0000","last_active":"2024-07-26T11:11:50.082116+0000","last_peered":"2024-07-26T11:11:50.082116+0000","last_clean":"2024-07-26T11:11:50.082116+0000","last_became_active":"2024-07-26T11:10:18.033038+0000","last_became_peered":"2024-07-26T11:10:18.033038+0000","last_unstale":"2024-07-26T11:11:50.082116+0000","last_undegraded":"2024-07-26T11:11:50.082116+0000","last_fullsized":"2024-07-26T11:11:50.082116+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:19:52.523006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.643751+0000","last_change":"2024-07-26T11:11:35.643683+0000","last_active":"2024-07-26T11:11:35.643751+0000","last_peered":"2024-07-26T11:11:35.643751+0000","last_clean":"2024-07-26T11:11:35.643751+0000","last_became_active":"2024-07-26T11:11:35.643109+0000","last_became_peered":"2024-07-26T11:11:35.643109+0000","last_unstale":"2024-07-26T11:11:35.643751+0000","last_undegraded":"2024-07-26T11:11:35.643751+0000","last_fullsized":"2024-07-26T11:11:35.643751+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:39:28.190726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.382627+0000","last_change":"2024-07-26T11:10:34.977690+0000","last_active":"2024-07-26T11:11:35.382627+0000","last_peered":"2024-07-26T11:11:35.382627+0000","last_clean":"2024-07-26T11:11:35.382627+0000","last_became_active":"2024-07-26T11:10:34.976938+0000","last_became_peered":"2024-07-26T11:10:34.976938+0000","last_unstale":"2024-07-26T11:11:35.382627+0000","last_undegraded":"2024-07-26T11:11:35.382627+0000","last_fullsized":"2024-07-26T11:11:35.382627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:26:21.510469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.094135+0000","last_change":"2024-07-26T11:10:18.045363+0000","last_active":"2024-07-26T11:11:51.094135+0000","last_peered":"2024-07-26T11:11:51.094135+0000","last_clean":"2024-07-26T11:11:51.094135+0000","last_became_active":"2024-07-26T11:10:18.044810+0000","last_became_peered":"2024-07-26T11:10:18.044810+0000","last_unstale":"2024-07-26T11:11:51.094135+0000","last_undegraded":"2024-07-26T11:11:51.094135+0000","last_fullsized":"2024-07-26T11:11:51.094135+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:34.507172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":118,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161396+0000","last_change":"2024-07-26T11:08:48.936784+0000","last_active":"2024-07-26T11:11:35.161396+0000","last_peered":"2024-07-26T11:11:35.161396+0000","last_clean":"2024-07-26T11:11:35.161396+0000","last_became_active":"2024-07-26T11:08:48.931118+0000","last_became_peered":"2024-07-26T11:08:48.931118+0000","last_unstale":"2024-07-26T11:11:35.161396+0000","last_undegraded":"2024-07-26T11:11:35.161396+0000","last_fullsized":"2024-07-26T11:11:35.161396+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:05:01.918569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:05:01.918569+0000","last_clean_scrub_stamp":"2024-07-26T11:05:01.918569+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:57.128424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.386014+0000","last_change":"2024-07-26T11:11:35.385528+0000","last_active":"2024-07-26T11:11:35.386014+0000","last_peered":"2024-07-26T11:11:35.386014+0000","last_clean":"2024-07-26T11:11:35.386014+0000","last_became_active":"2024-07-26T11:11:35.385042+0000","last_became_peered":"2024-07-26T11:11:35.385042+0000","last_unstale":"2024-07-26T11:11:35.386014+0000","last_undegraded":"2024-07-26T11:11:35.386014+0000","last_fullsized":"2024-07-26T11:11:35.386014+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:52:30.343736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161972+0000","last_change":"2024-07-26T11:10:22.302715+0000","last_active":"2024-07-26T11:11:35.161972+0000","last_peered":"2024-07-26T11:11:35.161972+0000","last_clean":"2024-07-26T11:11:35.161972+0000","last_became_active":"2024-07-26T11:10:22.302284+0000","last_became_peered":"2024-07-26T11:10:22.302284+0000","last_unstale":"2024-07-26T11:11:35.161972+0000","last_undegraded":"2024-07-26T11:11:35.161972+0000","last_fullsized":"2024-07-26T11:11:35.161972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:13:05.555439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:23.148375+0000","last_change":"2024-07-26T11:10:18.823160+0000","last_active":"2024-07-26T11:12:23.148375+0000","last_peered":"2024-07-26T11:12:23.148375+0000","last_clean":"2024-07-26T11:12:23.148375+0000","last_became_active":"2024-07-26T11:10:18.822652+0000","last_became_peered":"2024-07-26T11:10:18.822652+0000","last_unstale":"2024-07-26T11:12:23.148375+0000","last_undegraded":"2024-07-26T11:12:23.148375+0000","last_fullsized":"2024-07-26T11:12:23.148375+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:21:06.077784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":36,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.949434+0000","last_change":"2024-07-26T11:10:35.984007+0000","last_active":"2024-07-26T11:11:35.949434+0000","last_peered":"2024-07-26T11:11:35.949434+0000","last_clean":"2024-07-26T11:11:35.949434+0000","last_became_active":"2024-07-26T11:10:35.983352+0000","last_became_peered":"2024-07-26T11:10:35.983352+0000","last_unstale":"2024-07-26T11:11:35.949434+0000","last_undegraded":"2024-07-26T11:11:35.949434+0000","last_fullsized":"2024-07-26T11:11:35.949434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:05:21.199041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161837+0000","last_change":"2024-07-26T11:10:22.320559+0000","last_active":"2024-07-26T11:11:35.161837+0000","last_peered":"2024-07-26T11:11:35.161837+0000","last_clean":"2024-07-26T11:11:35.161837+0000","last_became_active":"2024-07-26T11:10:22.320192+0000","last_became_peered":"2024-07-26T11:10:22.320192+0000","last_unstale":"2024-07-26T11:11:35.161837+0000","last_undegraded":"2024-07-26T11:11:35.161837+0000","last_fullsized":"2024-07-26T11:11:35.161837+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:48:51.774418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.086952+0000","last_change":"2024-07-26T11:10:18.044108+0000","last_active":"2024-07-26T11:11:50.086952+0000","last_peered":"2024-07-26T11:11:50.086952+0000","last_clean":"2024-07-26T11:11:50.086952+0000","last_became_active":"2024-07-26T11:10:18.043712+0000","last_became_peered":"2024-07-26T11:10:18.043712+0000","last_unstale":"2024-07-26T11:11:50.086952+0000","last_undegraded":"2024-07-26T11:11:50.086952+0000","last_fullsized":"2024-07-26T11:11:50.086952+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:05:34.382520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.185761+0000","last_change":"2024-07-26T11:10:35.999071+0000","last_active":"2024-07-26T11:11:35.185761+0000","last_peered":"2024-07-26T11:11:35.185761+0000","last_clean":"2024-07-26T11:11:35.185761+0000","last_became_active":"2024-07-26T11:10:35.997179+0000","last_became_peered":"2024-07-26T11:10:35.997179+0000","last_unstale":"2024-07-26T11:11:35.185761+0000","last_undegraded":"2024-07-26T11:11:35.185761+0000","last_fullsized":"2024-07-26T11:11:35.185761+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:36:45.632086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162017+0000","last_change":"2024-07-26T11:10:22.300356+0000","last_active":"2024-07-26T11:11:35.162017+0000","last_peered":"2024-07-26T11:11:35.162017+0000","last_clean":"2024-07-26T11:11:35.162017+0000","last_became_active":"2024-07-26T11:10:22.299949+0000","last_became_peered":"2024-07-26T11:10:22.299949+0000","last_unstale":"2024-07-26T11:11:35.162017+0000","last_undegraded":"2024-07-26T11:11:35.162017+0000","last_fullsized":"2024-07-26T11:11:35.162017+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:34.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.087189+0000","last_change":"2024-07-26T11:10:18.049965+0000","last_active":"2024-07-26T11:11:50.087189+0000","last_peered":"2024-07-26T11:11:50.087189+0000","last_clean":"2024-07-26T11:11:50.087189+0000","last_became_active":"2024-07-26T11:10:18.049251+0000","last_became_peered":"2024-07-26T11:10:18.049251+0000","last_unstale":"2024-07-26T11:11:50.087189+0000","last_undegraded":"2024-07-26T11:11:50.087189+0000","last_fullsized":"2024-07-26T11:11:50.087189+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:20.996609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158580+0000","last_change":"2024-07-26T11:10:35.967661+0000","last_active":"2024-07-26T11:11:35.158580+0000","last_peered":"2024-07-26T11:11:35.158580+0000","last_clean":"2024-07-26T11:11:35.158580+0000","last_became_active":"2024-07-26T11:10:35.966867+0000","last_became_peered":"2024-07-26T11:10:35.966867+0000","last_unstale":"2024-07-26T11:11:35.158580+0000","last_undegraded":"2024-07-26T11:11:35.158580+0000","last_fullsized":"2024-07-26T11:11:35.158580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:43.951319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162847+0000","last_change":"2024-07-26T11:10:22.303733+0000","last_active":"2024-07-26T11:11:35.162847+0000","last_peered":"2024-07-26T11:11:35.162847+0000","last_clean":"2024-07-26T11:11:35.162847+0000","last_became_active":"2024-07-26T11:10:22.303322+0000","last_became_peered":"2024-07-26T11:10:22.303322+0000","last_unstale":"2024-07-26T11:11:35.162847+0000","last_undegraded":"2024-07-26T11:11:35.162847+0000","last_fullsized":"2024-07-26T11:11:35.162847+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:34:34.962492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.038971+0000","last_change":"2024-07-26T11:10:18.032539+0000","last_active":"2024-07-26T11:11:50.038971+0000","last_peered":"2024-07-26T11:11:50.038971+0000","last_clean":"2024-07-26T11:11:50.038971+0000","last_became_active":"2024-07-26T11:10:18.032059+0000","last_became_peered":"2024-07-26T11:10:18.032059+0000","last_unstale":"2024-07-26T11:11:50.038971+0000","last_undegraded":"2024-07-26T11:11:50.038971+0000","last_fullsized":"2024-07-26T11:11:50.038971+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:54.502339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158593+0000","last_change":"2024-07-26T11:10:35.977473+0000","last_active":"2024-07-26T11:11:35.158593+0000","last_peered":"2024-07-26T11:11:35.158593+0000","last_clean":"2024-07-26T11:11:35.158593+0000","last_became_active":"2024-07-26T11:10:35.976830+0000","last_became_peered":"2024-07-26T11:10:35.976830+0000","last_unstale":"2024-07-26T11:11:35.158593+0000","last_undegraded":"2024-07-26T11:11:35.158593+0000","last_fullsized":"2024-07-26T11:11:35.158593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:46:13.009246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158794+0000","last_change":"2024-07-26T11:10:22.304358+0000","last_active":"2024-07-26T11:11:35.158794+0000","last_peered":"2024-07-26T11:11:35.158794+0000","last_clean":"2024-07-26T11:11:35.158794+0000","last_became_active":"2024-07-26T11:10:22.303846+0000","last_became_peered":"2024-07-26T11:10:22.303846+0000","last_unstale":"2024-07-26T11:11:35.158794+0000","last_undegraded":"2024-07-26T11:11:35.158794+0000","last_fullsized":"2024-07-26T11:11:35.158794+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:27:28.556481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.083488+0000","last_change":"2024-07-26T11:10:18.057678+0000","last_active":"2024-07-26T11:11:50.083488+0000","last_peered":"2024-07-26T11:11:50.083488+0000","last_clean":"2024-07-26T11:11:50.083488+0000","last_became_active":"2024-07-26T11:10:18.057282+0000","last_became_peered":"2024-07-26T11:10:18.057282+0000","last_unstale":"2024-07-26T11:11:50.083488+0000","last_undegraded":"2024-07-26T11:11:50.083488+0000","last_fullsized":"2024-07-26T11:11:50.083488+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:08:35.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.642212+0000","last_change":"2024-07-26T11:10:36.000329+0000","last_active":"2024-07-26T11:11:35.642212+0000","last_peered":"2024-07-26T11:11:35.642212+0000","last_clean":"2024-07-26T11:11:35.642212+0000","last_became_active":"2024-07-26T11:10:35.999713+0000","last_became_peered":"2024-07-26T11:10:35.999713+0000","last_unstale":"2024-07-26T11:11:35.642212+0000","last_undegraded":"2024-07-26T11:11:35.642212+0000","last_fullsized":"2024-07-26T11:11:35.642212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:48.137237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161113+0000","last_change":"2024-07-26T11:10:22.301526+0000","last_active":"2024-07-26T11:11:35.161113+0000","last_peered":"2024-07-26T11:11:35.161113+0000","last_clean":"2024-07-26T11:11:35.161113+0000","last_became_active":"2024-07-26T11:10:22.301098+0000","last_became_peered":"2024-07-26T11:10:22.301098+0000","last_unstale":"2024-07-26T11:11:35.161113+0000","last_undegraded":"2024-07-26T11:11:35.161113+0000","last_fullsized":"2024-07-26T11:11:35.161113+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:01:48.118315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:23.176778+0000","last_change":"2024-07-26T11:10:18.035922+0000","last_active":"2024-07-26T11:12:23.176778+0000","last_peered":"2024-07-26T11:12:23.176778+0000","last_clean":"2024-07-26T11:12:23.176778+0000","last_became_active":"2024-07-26T11:10:18.035184+0000","last_became_peered":"2024-07-26T11:10:18.035184+0000","last_unstale":"2024-07-26T11:12:23.176778+0000","last_undegraded":"2024-07-26T11:12:23.176778+0000","last_fullsized":"2024-07-26T11:12:23.176778+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:13:10.371105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.383917+0000","last_change":"2024-07-26T11:10:35.998791+0000","last_active":"2024-07-26T11:11:35.383917+0000","last_peered":"2024-07-26T11:11:35.383917+0000","last_clean":"2024-07-26T11:11:35.383917+0000","last_became_active":"2024-07-26T11:10:35.997163+0000","last_became_peered":"2024-07-26T11:10:35.997163+0000","last_unstale":"2024-07-26T11:11:35.383917+0000","last_undegraded":"2024-07-26T11:11:35.383917+0000","last_fullsized":"2024-07-26T11:11:35.383917+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:28:50.167294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161762+0000","last_change":"2024-07-26T11:10:22.314219+0000","last_active":"2024-07-26T11:11:35.161762+0000","last_peered":"2024-07-26T11:11:35.161762+0000","last_clean":"2024-07-26T11:11:35.161762+0000","last_became_active":"2024-07-26T11:10:22.313811+0000","last_became_peered":"2024-07-26T11:10:22.313811+0000","last_unstale":"2024-07-26T11:11:35.161762+0000","last_undegraded":"2024-07-26T11:11:35.161762+0000","last_fullsized":"2024-07-26T11:11:35.161762+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:55:47.452798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.086736+0000","last_change":"2024-07-26T11:10:18.047818+0000","last_active":"2024-07-26T11:11:50.086736+0000","last_peered":"2024-07-26T11:11:50.086736+0000","last_clean":"2024-07-26T11:11:50.086736+0000","last_became_active":"2024-07-26T11:10:18.047159+0000","last_became_peered":"2024-07-26T11:10:18.047159+0000","last_unstale":"2024-07-26T11:11:50.086736+0000","last_undegraded":"2024-07-26T11:11:50.086736+0000","last_fullsized":"2024-07-26T11:11:50.086736+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:37.528512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162499+0000","last_change":"2024-07-26T11:10:35.996468+0000","last_active":"2024-07-26T11:11:35.162499+0000","last_peered":"2024-07-26T11:11:35.162499+0000","last_clean":"2024-07-26T11:11:35.162499+0000","last_became_active":"2024-07-26T11:10:35.996030+0000","last_became_peered":"2024-07-26T11:10:35.996030+0000","last_unstale":"2024-07-26T11:11:35.162499+0000","last_undegraded":"2024-07-26T11:11:35.162499+0000","last_fullsized":"2024-07-26T11:11:35.162499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:20:14.743690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.640574+0000","last_change":"2024-07-26T11:10:22.318375+0000","last_active":"2024-07-26T11:11:35.640574+0000","last_peered":"2024-07-26T11:11:35.640574+0000","last_clean":"2024-07-26T11:11:35.640574+0000","last_became_active":"2024-07-26T11:10:22.317965+0000","last_became_peered":"2024-07-26T11:10:22.317965+0000","last_unstale":"2024-07-26T11:11:35.640574+0000","last_undegraded":"2024-07-26T11:11:35.640574+0000","last_fullsized":"2024-07-26T11:11:35.640574+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:36:43.504045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.096145+0000","last_change":"2024-07-26T11:10:18.047957+0000","last_active":"2024-07-26T11:11:51.096145+0000","last_peered":"2024-07-26T11:11:51.096145+0000","last_clean":"2024-07-26T11:11:51.096145+0000","last_became_active":"2024-07-26T11:10:18.047495+0000","last_became_peered":"2024-07-26T11:10:18.047495+0000","last_unstale":"2024-07-26T11:11:51.096145+0000","last_undegraded":"2024-07-26T11:11:51.096145+0000","last_fullsized":"2024-07-26T11:11:51.096145+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:11.238295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158703+0000","last_change":"2024-07-26T11:10:35.976946+0000","last_active":"2024-07-26T11:11:35.158703+0000","last_peered":"2024-07-26T11:11:35.158703+0000","last_clean":"2024-07-26T11:11:35.158703+0000","last_became_active":"2024-07-26T11:10:35.976474+0000","last_became_peered":"2024-07-26T11:10:35.976474+0000","last_unstale":"2024-07-26T11:11:35.158703+0000","last_undegraded":"2024-07-26T11:11:35.158703+0000","last_fullsized":"2024-07-26T11:11:35.158703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:50:51.240903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158944+0000","last_change":"2024-07-26T11:10:22.302473+0000","last_active":"2024-07-26T11:11:35.158944+0000","last_peered":"2024-07-26T11:11:35.158944+0000","last_clean":"2024-07-26T11:11:35.158944+0000","last_became_active":"2024-07-26T11:10:22.302096+0000","last_became_peered":"2024-07-26T11:10:22.302096+0000","last_unstale":"2024-07-26T11:11:35.158944+0000","last_undegraded":"2024-07-26T11:11:35.158944+0000","last_fullsized":"2024-07-26T11:11:35.158944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:19.215301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'35","reported_seq":166,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:22.148816+0000","last_change":"2024-07-26T11:10:34.973502+0000","last_active":"2024-07-26T11:12:22.148816+0000","last_peered":"2024-07-26T11:12:22.148816+0000","last_clean":"2024-07-26T11:12:22.148816+0000","last_became_active":"2024-07-26T11:10:34.964155+0000","last_became_peered":"2024-07-26T11:10:34.964155+0000","last_unstale":"2024-07-26T11:12:22.148816+0000","last_undegraded":"2024-07-26T11:12:22.148816+0000","last_fullsized":"2024-07-26T11:12:22.148816+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:23:06.652746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61117,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":73,"num_read_kb":96,"num_write":37,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.382816+0000","last_change":"2024-07-26T11:10:35.981439+0000","last_active":"2024-07-26T11:11:35.382816+0000","last_peered":"2024-07-26T11:11:35.382816+0000","last_clean":"2024-07-26T11:11:35.382816+0000","last_became_active":"2024-07-26T11:10:35.980878+0000","last_became_peered":"2024-07-26T11:10:35.980878+0000","last_unstale":"2024-07-26T11:11:35.382816+0000","last_undegraded":"2024-07-26T11:11:35.382816+0000","last_fullsized":"2024-07-26T11:11:35.382816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:31.719845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.949755+0000","last_change":"2024-07-26T11:10:34.974292+0000","last_active":"2024-07-26T11:11:35.949755+0000","last_peered":"2024-07-26T11:11:35.949755+0000","last_clean":"2024-07-26T11:11:35.949755+0000","last_became_active":"2024-07-26T11:10:34.973820+0000","last_became_peered":"2024-07-26T11:10:34.973820+0000","last_unstale":"2024-07-26T11:11:35.949755+0000","last_undegraded":"2024-07-26T11:11:35.949755+0000","last_fullsized":"2024-07-26T11:11:35.949755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:35:37.629313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:49.063298+0000","last_change":"2024-07-26T11:10:18.055248+0000","last_active":"2024-07-26T11:11:49.063298+0000","last_peered":"2024-07-26T11:11:49.063298+0000","last_clean":"2024-07-26T11:11:49.063298+0000","last_became_active":"2024-07-26T11:10:18.054882+0000","last_became_peered":"2024-07-26T11:10:18.054882+0000","last_unstale":"2024-07-26T11:11:49.063298+0000","last_undegraded":"2024-07-26T11:11:49.063298+0000","last_fullsized":"2024-07-26T11:11:49.063298+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:16.390058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.407324+0000","last_change":"2024-07-26T11:10:35.985628+0000","last_active":"2024-07-26T11:11:35.407324+0000","last_peered":"2024-07-26T11:11:35.407324+0000","last_clean":"2024-07-26T11:11:35.407324+0000","last_became_active":"2024-07-26T11:10:35.985156+0000","last_became_peered":"2024-07-26T11:10:35.985156+0000","last_unstale":"2024-07-26T11:11:35.407324+0000","last_undegraded":"2024-07-26T11:11:35.407324+0000","last_fullsized":"2024-07-26T11:11:35.407324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:29:18.891825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.186805+0000","last_change":"2024-07-26T11:10:34.973261+0000","last_active":"2024-07-26T11:11:35.186805+0000","last_peered":"2024-07-26T11:11:35.186805+0000","last_clean":"2024-07-26T11:11:35.186805+0000","last_became_active":"2024-07-26T11:10:34.971592+0000","last_became_peered":"2024-07-26T11:10:34.971592+0000","last_unstale":"2024-07-26T11:11:35.186805+0000","last_undegraded":"2024-07-26T11:11:35.186805+0000","last_fullsized":"2024-07-26T11:11:35.186805+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:58:13.517884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.082517+0000","last_change":"2024-07-26T11:10:34.962248+0000","last_active":"2024-07-26T11:11:50.082517+0000","last_peered":"2024-07-26T11:11:50.082517+0000","last_clean":"2024-07-26T11:11:50.082517+0000","last_became_active":"2024-07-26T11:10:34.961683+0000","last_became_peered":"2024-07-26T11:10:34.961683+0000","last_unstale":"2024-07-26T11:11:50.082517+0000","last_undegraded":"2024-07-26T11:11:50.082517+0000","last_fullsized":"2024-07-26T11:11:50.082517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:17:08.308226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.381770+0000","last_change":"2024-07-26T11:10:35.995760+0000","last_active":"2024-07-26T11:11:35.381770+0000","last_peered":"2024-07-26T11:11:35.381770+0000","last_clean":"2024-07-26T11:11:35.381770+0000","last_became_active":"2024-07-26T11:10:35.995328+0000","last_became_peered":"2024-07-26T11:10:35.995328+0000","last_unstale":"2024-07-26T11:11:35.381770+0000","last_undegraded":"2024-07-26T11:11:35.381770+0000","last_fullsized":"2024-07-26T11:11:35.381770+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:40:33.093315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.950002+0000","last_change":"2024-07-26T11:10:22.316151+0000","last_active":"2024-07-26T11:11:35.950002+0000","last_peered":"2024-07-26T11:11:35.950002+0000","last_clean":"2024-07-26T11:11:35.950002+0000","last_became_active":"2024-07-26T11:10:22.315722+0000","last_became_peered":"2024-07-26T11:10:22.315722+0000","last_unstale":"2024-07-26T11:11:35.950002+0000","last_undegraded":"2024-07-26T11:11:35.950002+0000","last_fullsized":"2024-07-26T11:11:35.950002+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:12:58.342637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":109,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:22.145864+0000","last_change":"2024-07-26T11:10:18.046327+0000","last_active":"2024-07-26T11:12:22.145864+0000","last_peered":"2024-07-26T11:12:22.145864+0000","last_clean":"2024-07-26T11:12:22.145864+0000","last_became_active":"2024-07-26T11:10:18.045913+0000","last_became_peered":"2024-07-26T11:10:18.045913+0000","last_unstale":"2024-07-26T11:12:22.145864+0000","last_undegraded":"2024-07-26T11:12:22.145864+0000","last_fullsized":"2024-07-26T11:12:22.145864+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:58.889046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":64,"num_read_kb":64,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.147636+0000","last_change":"2024-07-26T11:10:35.983264+0000","last_active":"2024-07-26T11:11:36.147636+0000","last_peered":"2024-07-26T11:11:36.147636+0000","last_clean":"2024-07-26T11:11:36.147636+0000","last_became_active":"2024-07-26T11:10:35.982210+0000","last_became_peered":"2024-07-26T11:10:35.982210+0000","last_unstale":"2024-07-26T11:11:36.147636+0000","last_undegraded":"2024-07-26T11:11:36.147636+0000","last_fullsized":"2024-07-26T11:11:36.147636+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:37:20.272725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.185660+0000","last_change":"2024-07-26T11:10:35.985335+0000","last_active":"2024-07-26T11:11:35.185660+0000","last_peered":"2024-07-26T11:11:35.185660+0000","last_clean":"2024-07-26T11:11:35.185660+0000","last_became_active":"2024-07-26T11:10:35.983667+0000","last_became_peered":"2024-07-26T11:10:35.983667+0000","last_unstale":"2024-07-26T11:11:35.185660+0000","last_undegraded":"2024-07-26T11:11:35.185660+0000","last_fullsized":"2024-07-26T11:11:35.185660+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:00:17.744806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.382004+0000","last_change":"2024-07-26T11:10:22.333494+0000","last_active":"2024-07-26T11:11:35.382004+0000","last_peered":"2024-07-26T11:11:35.382004+0000","last_clean":"2024-07-26T11:11:35.382004+0000","last_became_active":"2024-07-26T11:10:22.332950+0000","last_became_peered":"2024-07-26T11:10:22.332950+0000","last_unstale":"2024-07-26T11:11:35.382004+0000","last_undegraded":"2024-07-26T11:11:35.382004+0000","last_fullsized":"2024-07-26T11:11:35.382004+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:58:38.340857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.082572+0000","last_change":"2024-07-26T11:10:18.034057+0000","last_active":"2024-07-26T11:11:50.082572+0000","last_peered":"2024-07-26T11:11:50.082572+0000","last_clean":"2024-07-26T11:11:50.082572+0000","last_became_active":"2024-07-26T11:10:18.033119+0000","last_became_peered":"2024-07-26T11:10:18.033119+0000","last_unstale":"2024-07-26T11:11:50.082572+0000","last_undegraded":"2024-07-26T11:11:50.082572+0000","last_fullsized":"2024-07-26T11:11:50.082572+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:21.675593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.408475+0000","last_change":"2024-07-26T11:11:35.408377+0000","last_active":"2024-07-26T11:11:35.408475+0000","last_peered":"2024-07-26T11:11:35.408475+0000","last_clean":"2024-07-26T11:11:35.408475+0000","last_became_active":"2024-07-26T11:11:35.407865+0000","last_became_peered":"2024-07-26T11:11:35.407865+0000","last_unstale":"2024-07-26T11:11:35.408475+0000","last_undegraded":"2024-07-26T11:11:35.408475+0000","last_fullsized":"2024-07-26T11:11:35.408475+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:43:02.953395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.147621+0000","last_change":"2024-07-26T11:10:22.323447+0000","last_active":"2024-07-26T11:11:36.147621+0000","last_peered":"2024-07-26T11:11:36.147621+0000","last_clean":"2024-07-26T11:11:36.147621+0000","last_became_active":"2024-07-26T11:10:22.322988+0000","last_became_peered":"2024-07-26T11:10:22.322988+0000","last_unstale":"2024-07-26T11:11:36.147621+0000","last_undegraded":"2024-07-26T11:11:36.147621+0000","last_fullsized":"2024-07-26T11:11:36.147621+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:14.880985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:48.055233+0000","last_change":"2024-07-26T11:10:18.048724+0000","last_active":"2024-07-26T11:11:48.055233+0000","last_peered":"2024-07-26T11:11:48.055233+0000","last_clean":"2024-07-26T11:11:48.055233+0000","last_became_active":"2024-07-26T11:10:18.048257+0000","last_became_peered":"2024-07-26T11:10:18.048257+0000","last_unstale":"2024-07-26T11:11:48.055233+0000","last_undegraded":"2024-07-26T11:11:48.055233+0000","last_fullsized":"2024-07-26T11:11:48.055233+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:02:13.581446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:49.071327+0000","last_change":"2024-07-26T11:10:18.044171+0000","last_active":"2024-07-26T11:11:49.071327+0000","last_peered":"2024-07-26T11:11:49.071327+0000","last_clean":"2024-07-26T11:11:49.071327+0000","last_became_active":"2024-07-26T11:10:18.043682+0000","last_became_peered":"2024-07-26T11:10:18.043682+0000","last_unstale":"2024-07-26T11:11:49.071327+0000","last_undegraded":"2024-07-26T11:11:49.071327+0000","last_fullsized":"2024-07-26T11:11:49.071327+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:46:46.864119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158875+0000","last_change":"2024-07-26T11:10:22.321081+0000","last_active":"2024-07-26T11:11:35.158875+0000","last_peered":"2024-07-26T11:11:35.158875+0000","last_clean":"2024-07-26T11:11:35.158875+0000","last_became_active":"2024-07-26T11:10:22.320656+0000","last_became_peered":"2024-07-26T11:10:22.320656+0000","last_unstale":"2024-07-26T11:11:35.158875+0000","last_undegraded":"2024-07-26T11:11:35.158875+0000","last_fullsized":"2024-07-26T11:11:35.158875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:17:20.455129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156131+0000","last_change":"2024-07-26T11:10:36.000224+0000","last_active":"2024-07-26T11:11:35.156131+0000","last_peered":"2024-07-26T11:11:35.156131+0000","last_clean":"2024-07-26T11:11:35.156131+0000","last_became_active":"2024-07-26T11:10:35.999733+0000","last_became_peered":"2024-07-26T11:10:35.999733+0000","last_unstale":"2024-07-26T11:11:35.156131+0000","last_undegraded":"2024-07-26T11:11:35.156131+0000","last_fullsized":"2024-07-26T11:11:35.156131+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:49.912401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:22.146646+0000","last_change":"2024-07-26T11:10:18.034585+0000","last_active":"2024-07-26T11:12:22.146646+0000","last_peered":"2024-07-26T11:12:22.146646+0000","last_clean":"2024-07-26T11:12:22.146646+0000","last_became_active":"2024-07-26T11:10:18.034237+0000","last_became_peered":"2024-07-26T11:10:18.034237+0000","last_unstale":"2024-07-26T11:12:22.146646+0000","last_undegraded":"2024-07-26T11:12:22.146646+0000","last_fullsized":"2024-07-26T11:12:22.146646+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:22.670277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158605+0000","last_change":"2024-07-26T11:10:22.319282+0000","last_active":"2024-07-26T11:11:35.158605+0000","last_peered":"2024-07-26T11:11:35.158605+0000","last_clean":"2024-07-26T11:11:35.158605+0000","last_became_active":"2024-07-26T11:10:22.318822+0000","last_became_peered":"2024-07-26T11:10:22.318822+0000","last_unstale":"2024-07-26T11:11:35.158605+0000","last_undegraded":"2024-07-26T11:11:35.158605+0000","last_fullsized":"2024-07-26T11:11:35.158605+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:50:57.615839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156337+0000","last_change":"2024-07-26T11:10:35.970189+0000","last_active":"2024-07-26T11:11:35.156337+0000","last_peered":"2024-07-26T11:11:35.156337+0000","last_clean":"2024-07-26T11:11:35.156337+0000","last_became_active":"2024-07-26T11:10:35.969574+0000","last_became_peered":"2024-07-26T11:10:35.969574+0000","last_unstale":"2024-07-26T11:11:35.156337+0000","last_undegraded":"2024-07-26T11:11:35.156337+0000","last_fullsized":"2024-07-26T11:11:35.156337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:07.487944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162002+0000","last_change":"2024-07-26T11:10:22.299152+0000","last_active":"2024-07-26T11:11:35.162002+0000","last_peered":"2024-07-26T11:11:35.162002+0000","last_clean":"2024-07-26T11:11:35.162002+0000","last_became_active":"2024-07-26T11:10:22.298707+0000","last_became_peered":"2024-07-26T11:10:22.298707+0000","last_unstale":"2024-07-26T11:11:35.162002+0000","last_undegraded":"2024-07-26T11:11:35.162002+0000","last_fullsized":"2024-07-26T11:11:35.162002+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:25.161634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.155991+0000","last_change":"2024-07-26T11:10:22.302476+0000","last_active":"2024-07-26T11:11:35.155991+0000","last_peered":"2024-07-26T11:11:35.155991+0000","last_clean":"2024-07-26T11:11:35.155991+0000","last_became_active":"2024-07-26T11:10:22.302091+0000","last_became_peered":"2024-07-26T11:10:22.302091+0000","last_unstale":"2024-07-26T11:11:35.155991+0000","last_undegraded":"2024-07-26T11:11:35.155991+0000","last_fullsized":"2024-07-26T11:11:35.155991+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:06:18.587588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.155754+0000","last_change":"2024-07-26T11:10:22.326583+0000","last_active":"2024-07-26T11:11:35.155754+0000","last_peered":"2024-07-26T11:11:35.155754+0000","last_clean":"2024-07-26T11:11:35.155754+0000","last_became_active":"2024-07-26T11:10:22.326185+0000","last_became_peered":"2024-07-26T11:10:22.326185+0000","last_unstale":"2024-07-26T11:11:35.155754+0000","last_undegraded":"2024-07-26T11:11:35.155754+0000","last_fullsized":"2024-07-26T11:11:35.155754+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:54:00.148964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.186134+0000","last_change":"2024-07-26T11:10:22.329247+0000","last_active":"2024-07-26T11:11:35.186134+0000","last_peered":"2024-07-26T11:11:35.186134+0000","last_clean":"2024-07-26T11:11:35.186134+0000","last_became_active":"2024-07-26T11:10:22.328585+0000","last_became_peered":"2024-07-26T11:10:22.328585+0000","last_unstale":"2024-07-26T11:11:35.186134+0000","last_undegraded":"2024-07-26T11:11:35.186134+0000","last_fullsized":"2024-07-26T11:11:35.186134+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:01:27.195399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.641804+0000","last_change":"2024-07-26T11:10:22.330717+0000","last_active":"2024-07-26T11:11:35.641804+0000","last_peered":"2024-07-26T11:11:35.641804+0000","last_clean":"2024-07-26T11:11:35.641804+0000","last_became_active":"2024-07-26T11:10:22.329576+0000","last_became_peered":"2024-07-26T11:10:22.329576+0000","last_unstale":"2024-07-26T11:11:35.641804+0000","last_undegraded":"2024-07-26T11:11:35.641804+0000","last_fullsized":"2024-07-26T11:11:35.641804+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:21.970629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162998+0000","last_change":"2024-07-26T11:10:22.307445+0000","last_active":"2024-07-26T11:11:35.162998+0000","last_peered":"2024-07-26T11:11:35.162998+0000","last_clean":"2024-07-26T11:11:35.162998+0000","last_became_active":"2024-07-26T11:10:22.305738+0000","last_became_peered":"2024-07-26T11:10:22.305738+0000","last_unstale":"2024-07-26T11:11:35.162998+0000","last_undegraded":"2024-07-26T11:11:35.162998+0000","last_fullsized":"2024-07-26T11:11:35.162998+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:09:08.125209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156570+0000","last_change":"2024-07-26T11:10:22.298392+0000","last_active":"2024-07-26T11:11:35.156570+0000","last_peered":"2024-07-26T11:11:35.156570+0000","last_clean":"2024-07-26T11:11:35.156570+0000","last_became_active":"2024-07-26T11:10:22.297979+0000","last_became_peered":"2024-07-26T11:10:22.297979+0000","last_unstale":"2024-07-26T11:11:35.156570+0000","last_undegraded":"2024-07-26T11:11:35.156570+0000","last_fullsized":"2024-07-26T11:11:35.156570+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:30:39.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.148623+0000","last_change":"2024-07-26T11:10:22.320850+0000","last_active":"2024-07-26T11:11:36.148623+0000","last_peered":"2024-07-26T11:11:36.148623+0000","last_clean":"2024-07-26T11:11:36.148623+0000","last_became_active":"2024-07-26T11:10:22.319838+0000","last_became_peered":"2024-07-26T11:10:22.319838+0000","last_unstale":"2024-07-26T11:11:36.148623+0000","last_undegraded":"2024-07-26T11:11:36.148623+0000","last_fullsized":"2024-07-26T11:11:36.148623+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:14.289366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158740+0000","last_change":"2024-07-26T11:10:22.321305+0000","last_active":"2024-07-26T11:11:35.158740+0000","last_peered":"2024-07-26T11:11:35.158740+0000","last_clean":"2024-07-26T11:11:35.158740+0000","last_became_active":"2024-07-26T11:10:22.320733+0000","last_became_peered":"2024-07-26T11:10:22.320733+0000","last_unstale":"2024-07-26T11:11:35.158740+0000","last_undegraded":"2024-07-26T11:11:35.158740+0000","last_fullsized":"2024-07-26T11:11:35.158740+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:24:33.125828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.147831+0000","last_change":"2024-07-26T11:10:22.318193+0000","last_active":"2024-07-26T11:11:36.147831+0000","last_peered":"2024-07-26T11:11:36.147831+0000","last_clean":"2024-07-26T11:11:36.147831+0000","last_became_active":"2024-07-26T11:10:22.317819+0000","last_became_peered":"2024-07-26T11:10:22.317819+0000","last_unstale":"2024-07-26T11:11:36.147831+0000","last_undegraded":"2024-07-26T11:11:36.147831+0000","last_fullsized":"2024-07-26T11:11:36.147831+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:14:06.105152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.949718+0000","last_change":"2024-07-26T11:10:22.318222+0000","last_active":"2024-07-26T11:11:35.949718+0000","last_peered":"2024-07-26T11:11:35.949718+0000","last_clean":"2024-07-26T11:11:35.949718+0000","last_became_active":"2024-07-26T11:10:22.317806+0000","last_became_peered":"2024-07-26T11:10:22.317806+0000","last_unstale":"2024-07-26T11:11:35.949718+0000","last_undegraded":"2024-07-26T11:11:35.949718+0000","last_fullsized":"2024-07-26T11:11:35.949718+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:55:27.510542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.641468+0000","last_change":"2024-07-26T11:10:22.330166+0000","last_active":"2024-07-26T11:11:35.641468+0000","last_peered":"2024-07-26T11:11:35.641468+0000","last_clean":"2024-07-26T11:11:35.641468+0000","last_became_active":"2024-07-26T11:10:22.329722+0000","last_became_peered":"2024-07-26T11:10:22.329722+0000","last_unstale":"2024-07-26T11:11:35.641468+0000","last_undegraded":"2024-07-26T11:11:35.641468+0000","last_fullsized":"2024-07-26T11:11:35.641468+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:58.816139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162551+0000","last_change":"2024-07-26T11:10:22.302774+0000","last_active":"2024-07-26T11:11:35.162551+0000","last_peered":"2024-07-26T11:11:35.162551+0000","last_clean":"2024-07-26T11:11:35.162551+0000","last_became_active":"2024-07-26T11:10:22.302382+0000","last_became_peered":"2024-07-26T11:10:22.302382+0000","last_unstale":"2024-07-26T11:11:35.162551+0000","last_undegraded":"2024-07-26T11:11:35.162551+0000","last_fullsized":"2024-07-26T11:11:35.162551+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:02:58.019598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156478+0000","last_change":"2024-07-26T11:10:22.312475+0000","last_active":"2024-07-26T11:11:35.156478+0000","last_peered":"2024-07-26T11:11:35.156478+0000","last_clean":"2024-07-26T11:11:35.156478+0000","last_became_active":"2024-07-26T11:10:22.312078+0000","last_became_peered":"2024-07-26T11:10:22.312078+0000","last_unstale":"2024-07-26T11:11:35.156478+0000","last_undegraded":"2024-07-26T11:11:35.156478+0000","last_fullsized":"2024-07-26T11:11:35.156478+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:25:37.705136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.383965+0000","last_change":"2024-07-26T11:10:22.314776+0000","last_active":"2024-07-26T11:11:35.383965+0000","last_peered":"2024-07-26T11:11:35.383965+0000","last_clean":"2024-07-26T11:11:35.383965+0000","last_became_active":"2024-07-26T11:10:22.307808+0000","last_became_peered":"2024-07-26T11:10:22.307808+0000","last_unstale":"2024-07-26T11:11:35.383965+0000","last_undegraded":"2024-07-26T11:11:35.383965+0000","last_fullsized":"2024-07-26T11:11:35.383965+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:44:06.152799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.163370+0000","last_change":"2024-07-26T11:10:22.328102+0000","last_active":"2024-07-26T11:11:35.163370+0000","last_peered":"2024-07-26T11:11:35.163370+0000","last_clean":"2024-07-26T11:11:35.163370+0000","last_became_active":"2024-07-26T11:10:22.327779+0000","last_became_peered":"2024-07-26T11:10:22.327779+0000","last_unstale":"2024-07-26T11:11:35.163370+0000","last_undegraded":"2024-07-26T11:11:35.163370+0000","last_fullsized":"2024-07-26T11:11:35.163370+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:14:34.569748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.949239+0000","last_change":"2024-07-26T11:10:22.329057+0000","last_active":"2024-07-26T11:11:35.949239+0000","last_peered":"2024-07-26T11:11:35.949239+0000","last_clean":"2024-07-26T11:11:35.949239+0000","last_became_active":"2024-07-26T11:10:22.328656+0000","last_became_peered":"2024-07-26T11:10:22.328656+0000","last_unstale":"2024-07-26T11:11:35.949239+0000","last_undegraded":"2024-07-26T11:11:35.949239+0000","last_fullsized":"2024-07-26T11:11:35.949239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:18:39.861674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.185692+0000","last_change":"2024-07-26T11:10:22.326115+0000","last_active":"2024-07-26T11:11:35.185692+0000","last_peered":"2024-07-26T11:11:35.185692+0000","last_clean":"2024-07-26T11:11:35.185692+0000","last_became_active":"2024-07-26T11:10:22.325538+0000","last_became_peered":"2024-07-26T11:10:22.325538+0000","last_unstale":"2024-07-26T11:11:35.185692+0000","last_undegraded":"2024-07-26T11:11:35.185692+0000","last_fullsized":"2024-07-26T11:11:35.185692+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:18:19.339571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.186243+0000","last_change":"2024-07-26T11:10:22.326910+0000","last_active":"2024-07-26T11:11:35.186243+0000","last_peered":"2024-07-26T11:11:35.186243+0000","last_clean":"2024-07-26T11:11:35.186243+0000","last_became_active":"2024-07-26T11:10:22.324532+0000","last_became_peered":"2024-07-26T11:10:22.324532+0000","last_unstale":"2024-07-26T11:11:35.186243+0000","last_undegraded":"2024-07-26T11:11:35.186243+0000","last_fullsized":"2024-07-26T11:11:35.186243+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:33.671780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.641379+0000","last_change":"2024-07-26T11:10:22.312926+0000","last_active":"2024-07-26T11:11:35.641379+0000","last_peered":"2024-07-26T11:11:35.641379+0000","last_clean":"2024-07-26T11:11:35.641379+0000","last_became_active":"2024-07-26T11:10:22.311642+0000","last_became_peered":"2024-07-26T11:10:22.311642+0000","last_unstale":"2024-07-26T11:11:35.641379+0000","last_undegraded":"2024-07-26T11:11:35.641379+0000","last_fullsized":"2024-07-26T11:11:35.641379+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:56:29.644965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.383396+0000","last_change":"2024-07-26T11:10:34.976934+0000","last_active":"2024-07-26T11:11:35.383396+0000","last_peered":"2024-07-26T11:11:35.383396+0000","last_clean":"2024-07-26T11:11:35.383396+0000","last_became_active":"2024-07-26T11:10:34.976551+0000","last_became_peered":"2024-07-26T11:10:34.976551+0000","last_unstale":"2024-07-26T11:11:35.383396+0000","last_undegraded":"2024-07-26T11:11:35.383396+0000","last_fullsized":"2024-07-26T11:11:35.383396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:11:43.883644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161855+0000","last_change":"2024-07-26T11:10:22.304444+0000","last_active":"2024-07-26T11:11:35.161855+0000","last_peered":"2024-07-26T11:11:35.161855+0000","last_clean":"2024-07-26T11:11:35.161855+0000","last_became_active":"2024-07-26T11:10:22.302825+0000","last_became_peered":"2024-07-26T11:10:22.302825+0000","last_unstale":"2024-07-26T11:11:35.161855+0000","last_undegraded":"2024-07-26T11:11:35.161855+0000","last_fullsized":"2024-07-26T11:11:35.161855+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:15.229057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156568+0000","last_change":"2024-07-26T11:10:22.297515+0000","last_active":"2024-07-26T11:11:35.156568+0000","last_peered":"2024-07-26T11:11:35.156568+0000","last_clean":"2024-07-26T11:11:35.156568+0000","last_became_active":"2024-07-26T11:10:22.297134+0000","last_became_peered":"2024-07-26T11:10:22.297134+0000","last_unstale":"2024-07-26T11:11:35.156568+0000","last_undegraded":"2024-07-26T11:11:35.156568+0000","last_fullsized":"2024-07-26T11:11:35.156568+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:25:08.794363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.950147+0000","last_change":"2024-07-26T11:10:22.304333+0000","last_active":"2024-07-26T11:11:35.950147+0000","last_peered":"2024-07-26T11:11:35.950147+0000","last_clean":"2024-07-26T11:11:35.950147+0000","last_became_active":"2024-07-26T11:10:22.303639+0000","last_became_peered":"2024-07-26T11:10:22.303639+0000","last_unstale":"2024-07-26T11:11:35.950147+0000","last_undegraded":"2024-07-26T11:11:35.950147+0000","last_fullsized":"2024-07-26T11:11:35.950147+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:47.053463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.147598+0000","last_change":"2024-07-26T11:10:22.320998+0000","last_active":"2024-07-26T11:11:36.147598+0000","last_peered":"2024-07-26T11:11:36.147598+0000","last_clean":"2024-07-26T11:11:36.147598+0000","last_became_active":"2024-07-26T11:10:22.320610+0000","last_became_peered":"2024-07-26T11:10:22.320610+0000","last_unstale":"2024-07-26T11:11:36.147598+0000","last_undegraded":"2024-07-26T11:11:36.147598+0000","last_fullsized":"2024-07-26T11:11:36.147598+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:52:28.802241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161283+0000","last_change":"2024-07-26T11:10:34.970486+0000","last_active":"2024-07-26T11:11:35.161283+0000","last_peered":"2024-07-26T11:11:35.161283+0000","last_clean":"2024-07-26T11:11:35.161283+0000","last_became_active":"2024-07-26T11:10:34.969965+0000","last_became_peered":"2024-07-26T11:10:34.969965+0000","last_unstale":"2024-07-26T11:11:35.161283+0000","last_undegraded":"2024-07-26T11:11:35.161283+0000","last_fullsized":"2024-07-26T11:11:35.161283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:41.593251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.155988+0000","last_change":"2024-07-26T11:10:22.321261+0000","last_active":"2024-07-26T11:11:35.155988+0000","last_peered":"2024-07-26T11:11:35.155988+0000","last_clean":"2024-07-26T11:11:35.155988+0000","last_became_active":"2024-07-26T11:10:22.320819+0000","last_became_peered":"2024-07-26T11:10:22.320819+0000","last_unstale":"2024-07-26T11:11:35.155988+0000","last_undegraded":"2024-07-26T11:11:35.155988+0000","last_fullsized":"2024-07-26T11:11:35.155988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:04:56.872600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.157906+0000","last_change":"2024-07-26T11:10:22.321206+0000","last_active":"2024-07-26T11:11:35.157906+0000","last_peered":"2024-07-26T11:11:35.157906+0000","last_clean":"2024-07-26T11:11:35.157906+0000","last_became_active":"2024-07-26T11:10:22.320700+0000","last_became_peered":"2024-07-26T11:10:22.320700+0000","last_unstale":"2024-07-26T11:11:35.157906+0000","last_undegraded":"2024-07-26T11:11:35.157906+0000","last_fullsized":"2024-07-26T11:11:35.157906+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:31:20.700808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.407201+0000","last_change":"2024-07-26T11:10:22.300061+0000","last_active":"2024-07-26T11:11:35.407201+0000","last_peered":"2024-07-26T11:11:35.407201+0000","last_clean":"2024-07-26T11:11:35.407201+0000","last_became_active":"2024-07-26T11:10:22.298737+0000","last_became_peered":"2024-07-26T11:10:22.298737+0000","last_unstale":"2024-07-26T11:11:35.407201+0000","last_undegraded":"2024-07-26T11:11:35.407201+0000","last_fullsized":"2024-07-26T11:11:35.407201+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:45:25.500816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.185495+0000","last_change":"2024-07-26T11:10:22.328534+0000","last_active":"2024-07-26T11:11:35.185495+0000","last_peered":"2024-07-26T11:11:35.185495+0000","last_clean":"2024-07-26T11:11:35.185495+0000","last_became_active":"2024-07-26T11:10:22.327929+0000","last_became_peered":"2024-07-26T11:10:22.327929+0000","last_unstale":"2024-07-26T11:11:35.185495+0000","last_undegraded":"2024-07-26T11:11:35.185495+0000","last_fullsized":"2024-07-26T11:11:35.185495+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:09:55.152881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.186076+0000","last_change":"2024-07-26T11:10:22.324753+0000","last_active":"2024-07-26T11:11:35.186076+0000","last_peered":"2024-07-26T11:11:35.186076+0000","last_clean":"2024-07-26T11:11:35.186076+0000","last_became_active":"2024-07-26T11:10:22.324307+0000","last_became_peered":"2024-07-26T11:10:22.324307+0000","last_unstale":"2024-07-26T11:11:35.186076+0000","last_undegraded":"2024-07-26T11:11:35.186076+0000","last_fullsized":"2024-07-26T11:11:35.186076+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:40.566465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.163226+0000","last_change":"2024-07-26T11:10:22.328183+0000","last_active":"2024-07-26T11:11:35.163226+0000","last_peered":"2024-07-26T11:11:35.163226+0000","last_clean":"2024-07-26T11:11:35.163226+0000","last_became_active":"2024-07-26T11:10:22.327805+0000","last_became_peered":"2024-07-26T11:10:22.327805+0000","last_unstale":"2024-07-26T11:11:35.163226+0000","last_undegraded":"2024-07-26T11:11:35.163226+0000","last_fullsized":"2024-07-26T11:11:35.163226+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:11:23.238390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67930,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":363,"num_read_kb":253,"num_write":247,"num_write_kb":139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2194,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":203790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":213,"ondisk_log_size":213,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":73,"seq":313532612648,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":527167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808876,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28700,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716452,"statfs":{"total":95995035648,"available":95965646848,"internally_reserved":0,"allocated":1724416,"data_stored":1174237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972402,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":529939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168633,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":529735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364861,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28620,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716532,"statfs":{"total":95995035648,"available":95965728768,"internally_reserved":0,"allocated":1642496,"data_stored":1122046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528387,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":521347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724619,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36952,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708200,"statfs":{"total":95995035648,"available":95957196800,"internally_reserved":0,"allocated":1654784,"data_stored":1113813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888144,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":586929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084372,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":521055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378714,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":522597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542239,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705763,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":62869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":62208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T11:12:28.398 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph pg dump --format=json 2024-07-26T11:12:28.901 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:29.270 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:28 smithi134 ceph-mon[31966]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:12:29.270 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:28 smithi134 ceph-mon[31966]: pgmap v375: 129 pgs: 129 active+clean; 647 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-07-26T11:12:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:28 smithi193 ceph-mon[37473]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:12:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:28 smithi193 ceph-mon[37473]: pgmap v375: 129 pgs: 129 active+clean; 647 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-07-26T11:12:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:28 smithi189 ceph-mon[36790]: from='client.24949 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:12:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:28 smithi189 ceph-mon[36790]: pgmap v375: 129 pgs: 129 active+clean; 647 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-07-26T11:12:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:31 smithi134 ceph-mon[31966]: pgmap v376: 129 pgs: 129 active+clean; 647 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1023 B/s wr, 10 op/s 2024-07-26T11:12:31.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.239+0000 7f9e67200640 1 -- 172.21.15.134:0/3862609850 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e6010af00 msgr2=0x7f9e6010b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:31.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.239+0000 7f9e67200640 1 --2- 172.21.15.134:0/3862609850 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e6010af00 0x7f9e6010b310 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9e50009900 tx=0x7f9e5002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:12:31.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.239+0000 7f9e67200640 1 -- 172.21.15.134:0/3862609850 shutdown_connections 2024-07-26T11:12:31.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.239+0000 7f9e67200640 1 --2- 172.21.15.134:0/3862609850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e6010b850 0x7f9e60112080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:31.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.239+0000 7f9e67200640 1 --2- 172.21.15.134:0/3862609850 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e6010af00 0x7f9e6010b310 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:31.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.239+0000 7f9e67200640 1 --2- 172.21.15.134:0/3862609850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e601055c0 0x7f9e601059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:31.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.239+0000 7f9e67200640 1 -- 172.21.15.134:0/3862609850 >> 172.21.15.134:0/3862609850 conn(0x7f9e60101010 msgr2=0x7f9e60103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:31.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.239+0000 7f9e67200640 1 -- 172.21.15.134:0/3862609850 shutdown_connections 2024-07-26T11:12:31.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.239+0000 7f9e67200640 1 -- 172.21.15.134:0/3862609850 wait complete. 2024-07-26T11:12:31.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.240+0000 7f9e67200640 1 Processor -- start 2024-07-26T11:12:31.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.240+0000 7f9e67200640 1 -- start start 2024-07-26T11:12:31.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.241+0000 7f9e67200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e601055c0 0x7f9e6019bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:31.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.241+0000 7f9e67200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e6010af00 0x7f9e6019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:31.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.241+0000 7f9e67200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e6010b850 0x7f9e601a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:31.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.241+0000 7f9e67200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9e60115510 con 0x7f9e6010b850 2024-07-26T11:12:31.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.241+0000 7f9e67200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e60115390 con 0x7f9e601055c0 2024-07-26T11:12:31.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.241+0000 7f9e67200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9e60115690 con 0x7f9e6010af00 2024-07-26T11:12:31.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.241+0000 7f9e65400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e6010af00 0x7f9e6019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:31.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.241+0000 7f9e65e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e601055c0 0x7f9e6019bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:31.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.241+0000 7f9e65400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e6010af00 0x7f9e6019c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52488/0 (socket says 172.21.15.134:52488) 2024-07-26T11:12:31.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.241+0000 7f9e65400640 1 -- 172.21.15.134:0/382502029 learned_addr learned my addr 172.21.15.134:0/382502029 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:12:31.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.241+0000 7f9e66800640 1 --2- 172.21.15.134:0/382502029 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e6010b850 0x7f9e601a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:31.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.242+0000 7f9e65e00640 1 -- 172.21.15.134:0/382502029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e6010af00 msgr2=0x7f9e6019c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:31.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.242+0000 7f9e65e00640 1 --2- 172.21.15.134:0/382502029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e6010af00 0x7f9e6019c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:31.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.242+0000 7f9e65e00640 1 -- 172.21.15.134:0/382502029 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e6010b850 msgr2=0x7f9e601a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:31.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.242+0000 7f9e65e00640 1 --2- 172.21.15.134:0/382502029 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e6010b850 0x7f9e601a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:31.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.242+0000 7f9e65e00640 1 -- 172.21.15.134:0/382502029 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e601a1ed0 con 0x7f9e601055c0 2024-07-26T11:12:31.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.242+0000 7f9e65e00640 1 --2- 172.21.15.134:0/382502029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e601055c0 0x7f9e6019bf60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9e5400e9f0 tx=0x7f9e5400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:31.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.242+0000 7f9e5aa00640 1 -- 172.21.15.134:0/382502029 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5400ccc0 con 0x7f9e601055c0 2024-07-26T11:12:31.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.243+0000 7f9e67200640 1 -- 172.21.15.134:0/382502029 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e601a21c0 con 0x7f9e601055c0 2024-07-26T11:12:31.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.243+0000 7f9e5aa00640 1 -- 172.21.15.134:0/382502029 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e540108e0 con 0x7f9e601055c0 2024-07-26T11:12:31.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.243+0000 7f9e5aa00640 1 -- 172.21.15.134:0/382502029 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e54018880 con 0x7f9e601055c0 2024-07-26T11:12:31.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.243+0000 7f9e67200640 1 -- 172.21.15.134:0/382502029 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e601a8920 con 0x7f9e601055c0 2024-07-26T11:12:31.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.244+0000 7f9e67200640 1 -- 172.21.15.134:0/382502029 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e440052c0 con 0x7f9e601055c0 2024-07-26T11:12:31.248 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.246+0000 7f9e5aa00640 1 -- 172.21.15.134:0/382502029 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f9e54010430 con 0x7f9e601055c0 2024-07-26T11:12:31.250 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.248+0000 7f9e5aa00640 1 --2- 172.21.15.134:0/382502029 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9e480b3d30 0x7f9e480b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:31.250 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.248+0000 7f9e5aa00640 1 -- 172.21.15.134:0/382502029 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f9e540e4420 con 0x7f9e601055c0 2024-07-26T11:12:31.250 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.248+0000 7f9e65400640 1 --2- 172.21.15.134:0/382502029 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9e480b3d30 0x7f9e480b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:31.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.249+0000 7f9e65400640 1 --2- 172.21.15.134:0/382502029 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9e480b3d30 0x7f9e480b6240 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9e50006010 tx=0x7f9e50005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:31.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.250+0000 7f9e5aa00640 1 -- 172.21.15.134:0/382502029 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9e540aadc0 con 0x7f9e601055c0 2024-07-26T11:12:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:31 smithi193 ceph-mon[37473]: pgmap v376: 129 pgs: 129 active+clean; 647 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1023 B/s wr, 10 op/s 2024-07-26T11:12:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:31 smithi189 ceph-mon[36790]: pgmap v376: 129 pgs: 129 active+clean; 647 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1023 B/s wr, 10 op/s 2024-07-26T11:12:31.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.443+0000 7f9e67200640 1 -- 172.21.15.134:0/382502029 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9e44000bf0 con 0x7f9e480b3d30 2024-07-26T11:12:31.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.449+0000 7f9e5aa00640 1 -- 172.21.15.134:0/382502029 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340392 (secure 0 0 0) 0x7f9e44000bf0 con 0x7f9e480b3d30 2024-07-26T11:12:31.453 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:12:31.459 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-07-26T11:12:31.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.458+0000 7f9e67200640 1 -- 172.21.15.134:0/382502029 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9e480b3d30 msgr2=0x7f9e480b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:31.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.458+0000 7f9e67200640 1 --2- 172.21.15.134:0/382502029 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9e480b3d30 0x7f9e480b6240 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9e50006010 tx=0x7f9e50005a20 comp rx=0 tx=0).stop 2024-07-26T11:12:31.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.458+0000 7f9e67200640 1 -- 172.21.15.134:0/382502029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e601055c0 msgr2=0x7f9e6019bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:31.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.458+0000 7f9e67200640 1 --2- 172.21.15.134:0/382502029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e601055c0 0x7f9e6019bf60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9e5400e9f0 tx=0x7f9e5400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:12:31.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.458+0000 7f9e67200640 1 -- 172.21.15.134:0/382502029 shutdown_connections 2024-07-26T11:12:31.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.458+0000 7f9e67200640 1 --2- 172.21.15.134:0/382502029 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9e480b3d30 0x7f9e480b6240 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:31.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.458+0000 7f9e67200640 1 --2- 172.21.15.134:0/382502029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e6010af00 0x7f9e6019c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:31.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.458+0000 7f9e67200640 1 --2- 172.21.15.134:0/382502029 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9e6010b850 0x7f9e601a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:31.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.458+0000 7f9e67200640 1 --2- 172.21.15.134:0/382502029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e601055c0 0x7f9e6019bf60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:31.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.458+0000 7f9e67200640 1 -- 172.21.15.134:0/382502029 >> 172.21.15.134:0/382502029 conn(0x7f9e60101010 msgr2=0x7f9e60103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:31.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.458+0000 7f9e67200640 1 -- 172.21.15.134:0/382502029 shutdown_connections 2024-07-26T11:12:31.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:31.458+0000 7f9e67200640 1 -- 172.21.15.134:0/382502029 wait complete. 2024-07-26T11:12:32.283 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-07-26T11:12:29.934374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":662797,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":490,"num_read_kb":352,"num_write":375,"num_write_kb":2296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":2954034,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":363388,"kb_used_data":14236,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124578436,"statfs":{"total":1151940427776,"available":1151568318464,"internally_reserved":0,"allocated":14577664,"data_stored":8263692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":357414202},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8713,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":112,"num_read_kb":60,"num_write":9,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006833"},"pg_stats":[{"pgid":"2.1f","version":"90'13","reported_seq":105,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:26.166708+0000","last_change":"2024-07-26T11:10:34.985984+0000","last_active":"2024-07-26T11:12:26.166708+0000","last_peered":"2024-07-26T11:12:26.166708+0000","last_clean":"2024-07-26T11:12:26.166708+0000","last_became_active":"2024-07-26T11:10:34.980894+0000","last_became_peered":"2024-07-26T11:10:34.980894+0000","last_unstale":"2024-07-26T11:12:26.166708+0000","last_undegraded":"2024-07-26T11:12:26.166708+0000","last_fullsized":"2024-07-26T11:12:26.166708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:55.718875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.148788+0000","last_change":"2024-07-26T11:10:22.320946+0000","last_active":"2024-07-26T11:11:36.148788+0000","last_peered":"2024-07-26T11:11:36.148788+0000","last_clean":"2024-07-26T11:11:36.148788+0000","last_became_active":"2024-07-26T11:10:22.316181+0000","last_became_peered":"2024-07-26T11:10:22.316181+0000","last_unstale":"2024-07-26T11:11:36.148788+0000","last_undegraded":"2024-07-26T11:11:36.148788+0000","last_fullsized":"2024-07-26T11:11:36.148788+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:13.641862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.155656+0000","last_change":"2024-07-26T11:10:35.984460+0000","last_active":"2024-07-26T11:11:35.155656+0000","last_peered":"2024-07-26T11:11:35.155656+0000","last_clean":"2024-07-26T11:11:35.155656+0000","last_became_active":"2024-07-26T11:10:35.983922+0000","last_became_peered":"2024-07-26T11:10:35.983922+0000","last_unstale":"2024-07-26T11:11:35.155656+0000","last_undegraded":"2024-07-26T11:11:35.155656+0000","last_fullsized":"2024-07-26T11:11:35.155656+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:51:46.140910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:48.051375+0000","last_change":"2024-07-26T11:10:18.058553+0000","last_active":"2024-07-26T11:11:48.051375+0000","last_peered":"2024-07-26T11:11:48.051375+0000","last_clean":"2024-07-26T11:11:48.051375+0000","last_became_active":"2024-07-26T11:10:18.058034+0000","last_became_peered":"2024-07-26T11:10:18.058034+0000","last_unstale":"2024-07-26T11:11:48.051375+0000","last_undegraded":"2024-07-26T11:11:48.051375+0000","last_fullsized":"2024-07-26T11:11:48.051375+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:48:58.868988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.186827+0000","last_change":"2024-07-26T11:10:22.326760+0000","last_active":"2024-07-26T11:11:35.186827+0000","last_peered":"2024-07-26T11:11:35.186827+0000","last_clean":"2024-07-26T11:11:35.186827+0000","last_became_active":"2024-07-26T11:10:22.326275+0000","last_became_peered":"2024-07-26T11:10:22.326275+0000","last_unstale":"2024-07-26T11:11:35.186827+0000","last_undegraded":"2024-07-26T11:11:35.186827+0000","last_fullsized":"2024-07-26T11:11:35.186827+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:51:57.461453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.406545+0000","last_change":"2024-07-26T11:10:35.997585+0000","last_active":"2024-07-26T11:11:35.406545+0000","last_peered":"2024-07-26T11:11:35.406545+0000","last_clean":"2024-07-26T11:11:35.406545+0000","last_became_active":"2024-07-26T11:10:35.997274+0000","last_became_peered":"2024-07-26T11:10:35.997274+0000","last_unstale":"2024-07-26T11:11:35.406545+0000","last_undegraded":"2024-07-26T11:11:35.406545+0000","last_fullsized":"2024-07-26T11:11:35.406545+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:43.553396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.077630+0000","last_change":"2024-07-26T11:10:18.034964+0000","last_active":"2024-07-26T11:11:51.077630+0000","last_peered":"2024-07-26T11:11:51.077630+0000","last_clean":"2024-07-26T11:11:51.077630+0000","last_became_active":"2024-07-26T11:10:18.034569+0000","last_became_peered":"2024-07-26T11:10:18.034569+0000","last_unstale":"2024-07-26T11:11:51.077630+0000","last_undegraded":"2024-07-26T11:11:51.077630+0000","last_fullsized":"2024-07-26T11:11:51.077630+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:00:46.542458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.949541+0000","last_change":"2024-07-26T11:10:22.310695+0000","last_active":"2024-07-26T11:11:35.949541+0000","last_peered":"2024-07-26T11:11:35.949541+0000","last_clean":"2024-07-26T11:11:35.949541+0000","last_became_active":"2024-07-26T11:10:22.310142+0000","last_became_peered":"2024-07-26T11:10:22.310142+0000","last_unstale":"2024-07-26T11:11:35.949541+0000","last_undegraded":"2024-07-26T11:11:35.949541+0000","last_fullsized":"2024-07-26T11:11:35.949541+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:49:18.890802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.407084+0000","last_change":"2024-07-26T11:10:35.991418+0000","last_active":"2024-07-26T11:11:35.407084+0000","last_peered":"2024-07-26T11:11:35.407084+0000","last_clean":"2024-07-26T11:11:35.407084+0000","last_became_active":"2024-07-26T11:10:35.990934+0000","last_became_peered":"2024-07-26T11:10:35.990934+0000","last_unstale":"2024-07-26T11:11:35.407084+0000","last_undegraded":"2024-07-26T11:11:35.407084+0000","last_fullsized":"2024-07-26T11:11:35.407084+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:56.598738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.082903+0000","last_change":"2024-07-26T11:10:34.965637+0000","last_active":"2024-07-26T11:11:50.082903+0000","last_peered":"2024-07-26T11:11:50.082903+0000","last_clean":"2024-07-26T11:11:50.082903+0000","last_became_active":"2024-07-26T11:10:34.961157+0000","last_became_peered":"2024-07-26T11:10:34.961157+0000","last_unstale":"2024-07-26T11:11:50.082903+0000","last_undegraded":"2024-07-26T11:11:50.082903+0000","last_fullsized":"2024-07-26T11:11:50.082903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:26:17.007258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158828+0000","last_change":"2024-07-26T11:10:22.303606+0000","last_active":"2024-07-26T11:11:35.158828+0000","last_peered":"2024-07-26T11:11:35.158828+0000","last_clean":"2024-07-26T11:11:35.158828+0000","last_became_active":"2024-07-26T11:10:22.303183+0000","last_became_peered":"2024-07-26T11:10:22.303183+0000","last_unstale":"2024-07-26T11:11:35.158828+0000","last_undegraded":"2024-07-26T11:11:35.158828+0000","last_fullsized":"2024-07-26T11:11:35.158828+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:31:01.153694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.406697+0000","last_change":"2024-07-26T11:10:35.981446+0000","last_active":"2024-07-26T11:11:35.406697+0000","last_peered":"2024-07-26T11:11:35.406697+0000","last_clean":"2024-07-26T11:11:35.406697+0000","last_became_active":"2024-07-26T11:10:35.980994+0000","last_became_peered":"2024-07-26T11:10:35.980994+0000","last_unstale":"2024-07-26T11:11:35.406697+0000","last_undegraded":"2024-07-26T11:11:35.406697+0000","last_fullsized":"2024-07-26T11:11:35.406697+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:38:57.405325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161463+0000","last_change":"2024-07-26T11:10:22.306758+0000","last_active":"2024-07-26T11:11:35.161463+0000","last_peered":"2024-07-26T11:11:35.161463+0000","last_clean":"2024-07-26T11:11:35.161463+0000","last_became_active":"2024-07-26T11:10:22.306370+0000","last_became_peered":"2024-07-26T11:10:22.306370+0000","last_unstale":"2024-07-26T11:11:35.161463+0000","last_undegraded":"2024-07-26T11:11:35.161463+0000","last_fullsized":"2024-07-26T11:11:35.161463+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:51:43.069470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:47.038158+0000","last_change":"2024-07-26T11:10:18.035664+0000","last_active":"2024-07-26T11:11:47.038158+0000","last_peered":"2024-07-26T11:11:47.038158+0000","last_clean":"2024-07-26T11:11:47.038158+0000","last_became_active":"2024-07-26T11:10:18.035113+0000","last_became_peered":"2024-07-26T11:10:18.035113+0000","last_unstale":"2024-07-26T11:11:47.038158+0000","last_undegraded":"2024-07-26T11:11:47.038158+0000","last_fullsized":"2024-07-26T11:11:47.038158+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:09:30.641034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.407009+0000","last_change":"2024-07-26T11:10:35.990907+0000","last_active":"2024-07-26T11:11:35.407009+0000","last_peered":"2024-07-26T11:11:35.407009+0000","last_clean":"2024-07-26T11:11:35.407009+0000","last_became_active":"2024-07-26T11:10:35.990417+0000","last_became_peered":"2024-07-26T11:10:35.990417+0000","last_unstale":"2024-07-26T11:11:35.407009+0000","last_undegraded":"2024-07-26T11:11:35.407009+0000","last_fullsized":"2024-07-26T11:11:35.407009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:24:39.319727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.147986+0000","last_change":"2024-07-26T11:10:22.333575+0000","last_active":"2024-07-26T11:11:36.147986+0000","last_peered":"2024-07-26T11:11:36.147986+0000","last_clean":"2024-07-26T11:11:36.147986+0000","last_became_active":"2024-07-26T11:10:22.332802+0000","last_became_peered":"2024-07-26T11:10:22.332802+0000","last_unstale":"2024-07-26T11:11:36.147986+0000","last_undegraded":"2024-07-26T11:11:36.147986+0000","last_fullsized":"2024-07-26T11:11:36.147986+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:14:03.238840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.059445+0000","last_change":"2024-07-26T11:10:18.047942+0000","last_active":"2024-07-26T11:11:51.059445+0000","last_peered":"2024-07-26T11:11:51.059445+0000","last_clean":"2024-07-26T11:11:51.059445+0000","last_became_active":"2024-07-26T11:10:18.047500+0000","last_became_peered":"2024-07-26T11:10:18.047500+0000","last_unstale":"2024-07-26T11:11:51.059445+0000","last_undegraded":"2024-07-26T11:11:51.059445+0000","last_fullsized":"2024-07-26T11:11:51.059445+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:32:30.773648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.406871+0000","last_change":"2024-07-26T11:10:36.000946+0000","last_active":"2024-07-26T11:11:35.406871+0000","last_peered":"2024-07-26T11:11:35.406871+0000","last_clean":"2024-07-26T11:11:35.406871+0000","last_became_active":"2024-07-26T11:10:36.000251+0000","last_became_peered":"2024-07-26T11:10:36.000251+0000","last_unstale":"2024-07-26T11:11:35.406871+0000","last_undegraded":"2024-07-26T11:11:35.406871+0000","last_fullsized":"2024-07-26T11:11:35.406871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:03:40.523747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162117+0000","last_change":"2024-07-26T11:10:22.326370+0000","last_active":"2024-07-26T11:11:35.162117+0000","last_peered":"2024-07-26T11:11:35.162117+0000","last_clean":"2024-07-26T11:11:35.162117+0000","last_became_active":"2024-07-26T11:10:22.325961+0000","last_became_peered":"2024-07-26T11:10:22.325961+0000","last_unstale":"2024-07-26T11:11:35.162117+0000","last_undegraded":"2024-07-26T11:11:35.162117+0000","last_fullsized":"2024-07-26T11:11:35.162117+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:16:36.946750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.089544+0000","last_change":"2024-07-26T11:10:18.051940+0000","last_active":"2024-07-26T11:11:50.089544+0000","last_peered":"2024-07-26T11:11:50.089544+0000","last_clean":"2024-07-26T11:11:50.089544+0000","last_became_active":"2024-07-26T11:10:18.051565+0000","last_became_peered":"2024-07-26T11:10:18.051565+0000","last_unstale":"2024-07-26T11:11:50.089544+0000","last_undegraded":"2024-07-26T11:11:50.089544+0000","last_fullsized":"2024-07-26T11:11:50.089544+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:17:56.418638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.383408+0000","last_change":"2024-07-26T11:10:36.000156+0000","last_active":"2024-07-26T11:11:35.383408+0000","last_peered":"2024-07-26T11:11:35.383408+0000","last_clean":"2024-07-26T11:11:35.383408+0000","last_became_active":"2024-07-26T11:10:35.999600+0000","last_became_peered":"2024-07-26T11:10:35.999600+0000","last_unstale":"2024-07-26T11:11:35.383408+0000","last_undegraded":"2024-07-26T11:11:35.383408+0000","last_fullsized":"2024-07-26T11:11:35.383408+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:35:48.504288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.157916+0000","last_change":"2024-07-26T11:10:22.298152+0000","last_active":"2024-07-26T11:11:35.157916+0000","last_peered":"2024-07-26T11:11:35.157916+0000","last_clean":"2024-07-26T11:11:35.157916+0000","last_became_active":"2024-07-26T11:10:22.297739+0000","last_became_peered":"2024-07-26T11:10:22.297739+0000","last_unstale":"2024-07-26T11:11:35.157916+0000","last_undegraded":"2024-07-26T11:11:35.157916+0000","last_fullsized":"2024-07-26T11:11:35.157916+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:48.160183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:47.040493+0000","last_change":"2024-07-26T11:10:18.057756+0000","last_active":"2024-07-26T11:11:47.040493+0000","last_peered":"2024-07-26T11:11:47.040493+0000","last_clean":"2024-07-26T11:11:47.040493+0000","last_became_active":"2024-07-26T11:10:18.057464+0000","last_became_peered":"2024-07-26T11:10:18.057464+0000","last_unstale":"2024-07-26T11:11:47.040493+0000","last_undegraded":"2024-07-26T11:11:47.040493+0000","last_fullsized":"2024-07-26T11:11:47.040493+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:39:11.523643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.406996+0000","last_change":"2024-07-26T11:10:36.000617+0000","last_active":"2024-07-26T11:11:35.406996+0000","last_peered":"2024-07-26T11:11:35.406996+0000","last_clean":"2024-07-26T11:11:35.406996+0000","last_became_active":"2024-07-26T11:10:36.000151+0000","last_became_peered":"2024-07-26T11:10:36.000151+0000","last_unstale":"2024-07-26T11:11:35.406996+0000","last_undegraded":"2024-07-26T11:11:35.406996+0000","last_fullsized":"2024-07-26T11:11:35.406996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:36.942215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.186420+0000","last_change":"2024-07-26T11:10:22.336452+0000","last_active":"2024-07-26T11:11:35.186420+0000","last_peered":"2024-07-26T11:11:35.186420+0000","last_clean":"2024-07-26T11:11:35.186420+0000","last_became_active":"2024-07-26T11:10:22.336006+0000","last_became_peered":"2024-07-26T11:10:22.336006+0000","last_unstale":"2024-07-26T11:11:35.186420+0000","last_undegraded":"2024-07-26T11:11:35.186420+0000","last_fullsized":"2024-07-26T11:11:35.186420+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:08:27.375730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.084181+0000","last_change":"2024-07-26T11:10:18.823017+0000","last_active":"2024-07-26T11:11:50.084181+0000","last_peered":"2024-07-26T11:11:50.084181+0000","last_clean":"2024-07-26T11:11:50.084181+0000","last_became_active":"2024-07-26T11:10:18.822559+0000","last_became_peered":"2024-07-26T11:10:18.822559+0000","last_unstale":"2024-07-26T11:11:50.084181+0000","last_undegraded":"2024-07-26T11:11:50.084181+0000","last_fullsized":"2024-07-26T11:11:50.084181+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:01.486263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.155695+0000","last_change":"2024-07-26T11:10:35.969698+0000","last_active":"2024-07-26T11:11:35.155695+0000","last_peered":"2024-07-26T11:11:35.155695+0000","last_clean":"2024-07-26T11:11:35.155695+0000","last_became_active":"2024-07-26T11:10:35.969240+0000","last_became_peered":"2024-07-26T11:10:35.969240+0000","last_unstale":"2024-07-26T11:11:35.155695+0000","last_undegraded":"2024-07-26T11:11:35.155695+0000","last_fullsized":"2024-07-26T11:11:35.155695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:25.210862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.148957+0000","last_change":"2024-07-26T11:10:22.314772+0000","last_active":"2024-07-26T11:11:36.148957+0000","last_peered":"2024-07-26T11:11:36.148957+0000","last_clean":"2024-07-26T11:11:36.148957+0000","last_became_active":"2024-07-26T11:10:22.313786+0000","last_became_peered":"2024-07-26T11:10:22.313786+0000","last_unstale":"2024-07-26T11:11:36.148957+0000","last_undegraded":"2024-07-26T11:11:36.148957+0000","last_fullsized":"2024-07-26T11:11:36.148957+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:19.610695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'16","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:17.588852+0000","last_change":"2024-07-26T11:10:18.049408+0000","last_active":"2024-07-26T11:12:17.588852+0000","last_peered":"2024-07-26T11:12:17.588852+0000","last_clean":"2024-07-26T11:12:17.588852+0000","last_became_active":"2024-07-26T11:10:18.048573+0000","last_became_peered":"2024-07-26T11:10:18.048573+0000","last_unstale":"2024-07-26T11:12:17.588852+0000","last_undegraded":"2024-07-26T11:12:17.588852+0000","last_fullsized":"2024-07-26T11:12:17.588852+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:26.408260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.147210+0000","last_change":"2024-07-26T11:10:35.992067+0000","last_active":"2024-07-26T11:11:36.147210+0000","last_peered":"2024-07-26T11:11:36.147210+0000","last_clean":"2024-07-26T11:11:36.147210+0000","last_became_active":"2024-07-26T11:10:35.989789+0000","last_became_peered":"2024-07-26T11:10:35.989789+0000","last_unstale":"2024-07-26T11:11:36.147210+0000","last_undegraded":"2024-07-26T11:11:36.147210+0000","last_fullsized":"2024-07-26T11:11:36.147210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:08:08.283592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.642060+0000","last_change":"2024-07-26T11:10:22.323512+0000","last_active":"2024-07-26T11:11:35.642060+0000","last_peered":"2024-07-26T11:11:35.642060+0000","last_clean":"2024-07-26T11:11:35.642060+0000","last_became_active":"2024-07-26T11:10:22.323126+0000","last_became_peered":"2024-07-26T11:10:22.323126+0000","last_unstale":"2024-07-26T11:11:35.642060+0000","last_undegraded":"2024-07-26T11:11:35.642060+0000","last_fullsized":"2024-07-26T11:11:35.642060+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:27:15.379345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.093631+0000","last_change":"2024-07-26T11:10:18.052573+0000","last_active":"2024-07-26T11:11:51.093631+0000","last_peered":"2024-07-26T11:11:51.093631+0000","last_clean":"2024-07-26T11:11:51.093631+0000","last_became_active":"2024-07-26T11:10:18.052302+0000","last_became_peered":"2024-07-26T11:10:18.052302+0000","last_unstale":"2024-07-26T11:11:51.093631+0000","last_undegraded":"2024-07-26T11:11:51.093631+0000","last_fullsized":"2024-07-26T11:11:51.093631+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:32.260966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162344+0000","last_change":"2024-07-26T11:10:35.994347+0000","last_active":"2024-07-26T11:11:35.162344+0000","last_peered":"2024-07-26T11:11:35.162344+0000","last_clean":"2024-07-26T11:11:35.162344+0000","last_became_active":"2024-07-26T11:10:35.993795+0000","last_became_peered":"2024-07-26T11:10:35.993795+0000","last_unstale":"2024-07-26T11:11:35.162344+0000","last_undegraded":"2024-07-26T11:11:35.162344+0000","last_fullsized":"2024-07-26T11:11:35.162344+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:35.399601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156202+0000","last_change":"2024-07-26T11:10:22.318109+0000","last_active":"2024-07-26T11:11:35.156202+0000","last_peered":"2024-07-26T11:11:35.156202+0000","last_clean":"2024-07-26T11:11:35.156202+0000","last_became_active":"2024-07-26T11:10:22.317674+0000","last_became_peered":"2024-07-26T11:10:22.317674+0000","last_unstale":"2024-07-26T11:11:35.156202+0000","last_undegraded":"2024-07-26T11:11:35.156202+0000","last_fullsized":"2024-07-26T11:11:35.156202+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:29.464772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:49.073328+0000","last_change":"2024-07-26T11:10:18.060073+0000","last_active":"2024-07-26T11:11:49.073328+0000","last_peered":"2024-07-26T11:11:49.073328+0000","last_clean":"2024-07-26T11:11:49.073328+0000","last_became_active":"2024-07-26T11:10:18.059381+0000","last_became_peered":"2024-07-26T11:10:18.059381+0000","last_unstale":"2024-07-26T11:11:49.073328+0000","last_undegraded":"2024-07-26T11:11:49.073328+0000","last_fullsized":"2024-07-26T11:11:49.073328+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:04:21.184759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.951307+0000","last_change":"2024-07-26T11:11:35.951234+0000","last_active":"2024-07-26T11:11:35.951307+0000","last_peered":"2024-07-26T11:11:35.951307+0000","last_clean":"2024-07-26T11:11:35.951307+0000","last_became_active":"2024-07-26T11:11:35.950821+0000","last_became_peered":"2024-07-26T11:11:35.950821+0000","last_unstale":"2024-07-26T11:11:35.951307+0000","last_undegraded":"2024-07-26T11:11:35.951307+0000","last_fullsized":"2024-07-26T11:11:35.951307+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:44:22.569238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156228+0000","last_change":"2024-07-26T11:10:22.326866+0000","last_active":"2024-07-26T11:11:35.156228+0000","last_peered":"2024-07-26T11:11:35.156228+0000","last_clean":"2024-07-26T11:11:35.156228+0000","last_became_active":"2024-07-26T11:10:22.326417+0000","last_became_peered":"2024-07-26T11:10:22.326417+0000","last_unstale":"2024-07-26T11:11:35.156228+0000","last_undegraded":"2024-07-26T11:11:35.156228+0000","last_fullsized":"2024-07-26T11:11:35.156228+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:49:02.871062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:26.167340+0000","last_change":"2024-07-26T11:10:18.044077+0000","last_active":"2024-07-26T11:12:26.167340+0000","last_peered":"2024-07-26T11:12:26.167340+0000","last_clean":"2024-07-26T11:12:26.167340+0000","last_became_active":"2024-07-26T11:10:18.043660+0000","last_became_peered":"2024-07-26T11:10:18.043660+0000","last_unstale":"2024-07-26T11:12:26.167340+0000","last_undegraded":"2024-07-26T11:12:26.167340+0000","last_fullsized":"2024-07-26T11:12:26.167340+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:46:54.901082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.641589+0000","last_change":"2024-07-26T11:10:35.984778+0000","last_active":"2024-07-26T11:11:35.641589+0000","last_peered":"2024-07-26T11:11:35.641589+0000","last_clean":"2024-07-26T11:11:35.641589+0000","last_became_active":"2024-07-26T11:10:35.984365+0000","last_became_peered":"2024-07-26T11:10:35.984365+0000","last_unstale":"2024-07-26T11:11:35.641589+0000","last_undegraded":"2024-07-26T11:11:35.641589+0000","last_fullsized":"2024-07-26T11:11:35.641589+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:33.174121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.381797+0000","last_change":"2024-07-26T11:10:34.976075+0000","last_active":"2024-07-26T11:11:35.381797+0000","last_peered":"2024-07-26T11:11:35.381797+0000","last_clean":"2024-07-26T11:11:35.381797+0000","last_became_active":"2024-07-26T11:10:34.973799+0000","last_became_peered":"2024-07-26T11:10:34.973799+0000","last_unstale":"2024-07-26T11:11:35.381797+0000","last_undegraded":"2024-07-26T11:11:35.381797+0000","last_fullsized":"2024-07-26T11:11:35.381797+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:03:01.830655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:26.189111+0000","last_change":"2024-07-26T11:10:18.056473+0000","last_active":"2024-07-26T11:12:26.189111+0000","last_peered":"2024-07-26T11:12:26.189111+0000","last_clean":"2024-07-26T11:12:26.189111+0000","last_became_active":"2024-07-26T11:10:18.056038+0000","last_became_peered":"2024-07-26T11:10:18.056038+0000","last_unstale":"2024-07-26T11:12:26.189111+0000","last_undegraded":"2024-07-26T11:12:26.189111+0000","last_fullsized":"2024-07-26T11:12:26.189111+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:04.501018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.642321+0000","last_change":"2024-07-26T11:10:35.991741+0000","last_active":"2024-07-26T11:11:35.642321+0000","last_peered":"2024-07-26T11:11:35.642321+0000","last_clean":"2024-07-26T11:11:35.642321+0000","last_became_active":"2024-07-26T11:10:35.990995+0000","last_became_peered":"2024-07-26T11:10:35.990995+0000","last_unstale":"2024-07-26T11:11:35.642321+0000","last_undegraded":"2024-07-26T11:11:35.642321+0000","last_fullsized":"2024-07-26T11:11:35.642321+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:59:34.056646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.641241+0000","last_change":"2024-07-26T11:10:22.305053+0000","last_active":"2024-07-26T11:11:35.641241+0000","last_peered":"2024-07-26T11:11:35.641241+0000","last_clean":"2024-07-26T11:11:35.641241+0000","last_became_active":"2024-07-26T11:10:22.304035+0000","last_became_peered":"2024-07-26T11:10:22.304035+0000","last_unstale":"2024-07-26T11:11:35.641241+0000","last_undegraded":"2024-07-26T11:11:35.641241+0000","last_fullsized":"2024-07-26T11:11:35.641241+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:48.590799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.082116+0000","last_change":"2024-07-26T11:10:18.033581+0000","last_active":"2024-07-26T11:11:50.082116+0000","last_peered":"2024-07-26T11:11:50.082116+0000","last_clean":"2024-07-26T11:11:50.082116+0000","last_became_active":"2024-07-26T11:10:18.033038+0000","last_became_peered":"2024-07-26T11:10:18.033038+0000","last_unstale":"2024-07-26T11:11:50.082116+0000","last_undegraded":"2024-07-26T11:11:50.082116+0000","last_fullsized":"2024-07-26T11:11:50.082116+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:19:52.523006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.643751+0000","last_change":"2024-07-26T11:11:35.643683+0000","last_active":"2024-07-26T11:11:35.643751+0000","last_peered":"2024-07-26T11:11:35.643751+0000","last_clean":"2024-07-26T11:11:35.643751+0000","last_became_active":"2024-07-26T11:11:35.643109+0000","last_became_peered":"2024-07-26T11:11:35.643109+0000","last_unstale":"2024-07-26T11:11:35.643751+0000","last_undegraded":"2024-07-26T11:11:35.643751+0000","last_fullsized":"2024-07-26T11:11:35.643751+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:39:28.190726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.382627+0000","last_change":"2024-07-26T11:10:34.977690+0000","last_active":"2024-07-26T11:11:35.382627+0000","last_peered":"2024-07-26T11:11:35.382627+0000","last_clean":"2024-07-26T11:11:35.382627+0000","last_became_active":"2024-07-26T11:10:34.976938+0000","last_became_peered":"2024-07-26T11:10:34.976938+0000","last_unstale":"2024-07-26T11:11:35.382627+0000","last_undegraded":"2024-07-26T11:11:35.382627+0000","last_fullsized":"2024-07-26T11:11:35.382627+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:26:21.510469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.094135+0000","last_change":"2024-07-26T11:10:18.045363+0000","last_active":"2024-07-26T11:11:51.094135+0000","last_peered":"2024-07-26T11:11:51.094135+0000","last_clean":"2024-07-26T11:11:51.094135+0000","last_became_active":"2024-07-26T11:10:18.044810+0000","last_became_peered":"2024-07-26T11:10:18.044810+0000","last_unstale":"2024-07-26T11:11:51.094135+0000","last_undegraded":"2024-07-26T11:11:51.094135+0000","last_fullsized":"2024-07-26T11:11:51.094135+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:34.507172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":118,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161396+0000","last_change":"2024-07-26T11:08:48.936784+0000","last_active":"2024-07-26T11:11:35.161396+0000","last_peered":"2024-07-26T11:11:35.161396+0000","last_clean":"2024-07-26T11:11:35.161396+0000","last_became_active":"2024-07-26T11:08:48.931118+0000","last_became_peered":"2024-07-26T11:08:48.931118+0000","last_unstale":"2024-07-26T11:11:35.161396+0000","last_undegraded":"2024-07-26T11:11:35.161396+0000","last_fullsized":"2024-07-26T11:11:35.161396+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:05:01.918569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:05:01.918569+0000","last_clean_scrub_stamp":"2024-07-26T11:05:01.918569+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:57.128424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.386014+0000","last_change":"2024-07-26T11:11:35.385528+0000","last_active":"2024-07-26T11:11:35.386014+0000","last_peered":"2024-07-26T11:11:35.386014+0000","last_clean":"2024-07-26T11:11:35.386014+0000","last_became_active":"2024-07-26T11:11:35.385042+0000","last_became_peered":"2024-07-26T11:11:35.385042+0000","last_unstale":"2024-07-26T11:11:35.386014+0000","last_undegraded":"2024-07-26T11:11:35.386014+0000","last_fullsized":"2024-07-26T11:11:35.386014+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:52:30.343736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161972+0000","last_change":"2024-07-26T11:10:22.302715+0000","last_active":"2024-07-26T11:11:35.161972+0000","last_peered":"2024-07-26T11:11:35.161972+0000","last_clean":"2024-07-26T11:11:35.161972+0000","last_became_active":"2024-07-26T11:10:22.302284+0000","last_became_peered":"2024-07-26T11:10:22.302284+0000","last_unstale":"2024-07-26T11:11:35.161972+0000","last_undegraded":"2024-07-26T11:11:35.161972+0000","last_fullsized":"2024-07-26T11:11:35.161972+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:13:05.555439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":87,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:28.156807+0000","last_change":"2024-07-26T11:10:18.823160+0000","last_active":"2024-07-26T11:12:28.156807+0000","last_peered":"2024-07-26T11:12:28.156807+0000","last_clean":"2024-07-26T11:12:28.156807+0000","last_became_active":"2024-07-26T11:10:18.822652+0000","last_became_peered":"2024-07-26T11:10:18.822652+0000","last_unstale":"2024-07-26T11:12:28.156807+0000","last_undegraded":"2024-07-26T11:12:28.156807+0000","last_fullsized":"2024-07-26T11:12:28.156807+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:21:06.077784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.949434+0000","last_change":"2024-07-26T11:10:35.984007+0000","last_active":"2024-07-26T11:11:35.949434+0000","last_peered":"2024-07-26T11:11:35.949434+0000","last_clean":"2024-07-26T11:11:35.949434+0000","last_became_active":"2024-07-26T11:10:35.983352+0000","last_became_peered":"2024-07-26T11:10:35.983352+0000","last_unstale":"2024-07-26T11:11:35.949434+0000","last_undegraded":"2024-07-26T11:11:35.949434+0000","last_fullsized":"2024-07-26T11:11:35.949434+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:05:21.199041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161837+0000","last_change":"2024-07-26T11:10:22.320559+0000","last_active":"2024-07-26T11:11:35.161837+0000","last_peered":"2024-07-26T11:11:35.161837+0000","last_clean":"2024-07-26T11:11:35.161837+0000","last_became_active":"2024-07-26T11:10:22.320192+0000","last_became_peered":"2024-07-26T11:10:22.320192+0000","last_unstale":"2024-07-26T11:11:35.161837+0000","last_undegraded":"2024-07-26T11:11:35.161837+0000","last_fullsized":"2024-07-26T11:11:35.161837+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:48:51.774418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.086952+0000","last_change":"2024-07-26T11:10:18.044108+0000","last_active":"2024-07-26T11:11:50.086952+0000","last_peered":"2024-07-26T11:11:50.086952+0000","last_clean":"2024-07-26T11:11:50.086952+0000","last_became_active":"2024-07-26T11:10:18.043712+0000","last_became_peered":"2024-07-26T11:10:18.043712+0000","last_unstale":"2024-07-26T11:11:50.086952+0000","last_undegraded":"2024-07-26T11:11:50.086952+0000","last_fullsized":"2024-07-26T11:11:50.086952+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:05:34.382520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.185761+0000","last_change":"2024-07-26T11:10:35.999071+0000","last_active":"2024-07-26T11:11:35.185761+0000","last_peered":"2024-07-26T11:11:35.185761+0000","last_clean":"2024-07-26T11:11:35.185761+0000","last_became_active":"2024-07-26T11:10:35.997179+0000","last_became_peered":"2024-07-26T11:10:35.997179+0000","last_unstale":"2024-07-26T11:11:35.185761+0000","last_undegraded":"2024-07-26T11:11:35.185761+0000","last_fullsized":"2024-07-26T11:11:35.185761+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:36:45.632086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162017+0000","last_change":"2024-07-26T11:10:22.300356+0000","last_active":"2024-07-26T11:11:35.162017+0000","last_peered":"2024-07-26T11:11:35.162017+0000","last_clean":"2024-07-26T11:11:35.162017+0000","last_became_active":"2024-07-26T11:10:22.299949+0000","last_became_peered":"2024-07-26T11:10:22.299949+0000","last_unstale":"2024-07-26T11:11:35.162017+0000","last_undegraded":"2024-07-26T11:11:35.162017+0000","last_fullsized":"2024-07-26T11:11:35.162017+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:34.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.087189+0000","last_change":"2024-07-26T11:10:18.049965+0000","last_active":"2024-07-26T11:11:50.087189+0000","last_peered":"2024-07-26T11:11:50.087189+0000","last_clean":"2024-07-26T11:11:50.087189+0000","last_became_active":"2024-07-26T11:10:18.049251+0000","last_became_peered":"2024-07-26T11:10:18.049251+0000","last_unstale":"2024-07-26T11:11:50.087189+0000","last_undegraded":"2024-07-26T11:11:50.087189+0000","last_fullsized":"2024-07-26T11:11:50.087189+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:20.996609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158580+0000","last_change":"2024-07-26T11:10:35.967661+0000","last_active":"2024-07-26T11:11:35.158580+0000","last_peered":"2024-07-26T11:11:35.158580+0000","last_clean":"2024-07-26T11:11:35.158580+0000","last_became_active":"2024-07-26T11:10:35.966867+0000","last_became_peered":"2024-07-26T11:10:35.966867+0000","last_unstale":"2024-07-26T11:11:35.158580+0000","last_undegraded":"2024-07-26T11:11:35.158580+0000","last_fullsized":"2024-07-26T11:11:35.158580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:43.951319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162847+0000","last_change":"2024-07-26T11:10:22.303733+0000","last_active":"2024-07-26T11:11:35.162847+0000","last_peered":"2024-07-26T11:11:35.162847+0000","last_clean":"2024-07-26T11:11:35.162847+0000","last_became_active":"2024-07-26T11:10:22.303322+0000","last_became_peered":"2024-07-26T11:10:22.303322+0000","last_unstale":"2024-07-26T11:11:35.162847+0000","last_undegraded":"2024-07-26T11:11:35.162847+0000","last_fullsized":"2024-07-26T11:11:35.162847+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:34:34.962492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.038971+0000","last_change":"2024-07-26T11:10:18.032539+0000","last_active":"2024-07-26T11:11:50.038971+0000","last_peered":"2024-07-26T11:11:50.038971+0000","last_clean":"2024-07-26T11:11:50.038971+0000","last_became_active":"2024-07-26T11:10:18.032059+0000","last_became_peered":"2024-07-26T11:10:18.032059+0000","last_unstale":"2024-07-26T11:11:50.038971+0000","last_undegraded":"2024-07-26T11:11:50.038971+0000","last_fullsized":"2024-07-26T11:11:50.038971+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:54.502339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158593+0000","last_change":"2024-07-26T11:10:35.977473+0000","last_active":"2024-07-26T11:11:35.158593+0000","last_peered":"2024-07-26T11:11:35.158593+0000","last_clean":"2024-07-26T11:11:35.158593+0000","last_became_active":"2024-07-26T11:10:35.976830+0000","last_became_peered":"2024-07-26T11:10:35.976830+0000","last_unstale":"2024-07-26T11:11:35.158593+0000","last_undegraded":"2024-07-26T11:11:35.158593+0000","last_fullsized":"2024-07-26T11:11:35.158593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:46:13.009246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158794+0000","last_change":"2024-07-26T11:10:22.304358+0000","last_active":"2024-07-26T11:11:35.158794+0000","last_peered":"2024-07-26T11:11:35.158794+0000","last_clean":"2024-07-26T11:11:35.158794+0000","last_became_active":"2024-07-26T11:10:22.303846+0000","last_became_peered":"2024-07-26T11:10:22.303846+0000","last_unstale":"2024-07-26T11:11:35.158794+0000","last_undegraded":"2024-07-26T11:11:35.158794+0000","last_fullsized":"2024-07-26T11:11:35.158794+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:27:28.556481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.083488+0000","last_change":"2024-07-26T11:10:18.057678+0000","last_active":"2024-07-26T11:11:50.083488+0000","last_peered":"2024-07-26T11:11:50.083488+0000","last_clean":"2024-07-26T11:11:50.083488+0000","last_became_active":"2024-07-26T11:10:18.057282+0000","last_became_peered":"2024-07-26T11:10:18.057282+0000","last_unstale":"2024-07-26T11:11:50.083488+0000","last_undegraded":"2024-07-26T11:11:50.083488+0000","last_fullsized":"2024-07-26T11:11:50.083488+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:08:35.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.642212+0000","last_change":"2024-07-26T11:10:36.000329+0000","last_active":"2024-07-26T11:11:35.642212+0000","last_peered":"2024-07-26T11:11:35.642212+0000","last_clean":"2024-07-26T11:11:35.642212+0000","last_became_active":"2024-07-26T11:10:35.999713+0000","last_became_peered":"2024-07-26T11:10:35.999713+0000","last_unstale":"2024-07-26T11:11:35.642212+0000","last_undegraded":"2024-07-26T11:11:35.642212+0000","last_fullsized":"2024-07-26T11:11:35.642212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:48.137237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161113+0000","last_change":"2024-07-26T11:10:22.301526+0000","last_active":"2024-07-26T11:11:35.161113+0000","last_peered":"2024-07-26T11:11:35.161113+0000","last_clean":"2024-07-26T11:11:35.161113+0000","last_became_active":"2024-07-26T11:10:22.301098+0000","last_became_peered":"2024-07-26T11:10:22.301098+0000","last_unstale":"2024-07-26T11:11:35.161113+0000","last_undegraded":"2024-07-26T11:11:35.161113+0000","last_fullsized":"2024-07-26T11:11:35.161113+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:01:48.118315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'6","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:23.176778+0000","last_change":"2024-07-26T11:10:18.035922+0000","last_active":"2024-07-26T11:12:23.176778+0000","last_peered":"2024-07-26T11:12:23.176778+0000","last_clean":"2024-07-26T11:12:23.176778+0000","last_became_active":"2024-07-26T11:10:18.035184+0000","last_became_peered":"2024-07-26T11:10:18.035184+0000","last_unstale":"2024-07-26T11:12:23.176778+0000","last_undegraded":"2024-07-26T11:12:23.176778+0000","last_fullsized":"2024-07-26T11:12:23.176778+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:13:10.371105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":4,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.383917+0000","last_change":"2024-07-26T11:10:35.998791+0000","last_active":"2024-07-26T11:11:35.383917+0000","last_peered":"2024-07-26T11:11:35.383917+0000","last_clean":"2024-07-26T11:11:35.383917+0000","last_became_active":"2024-07-26T11:10:35.997163+0000","last_became_peered":"2024-07-26T11:10:35.997163+0000","last_unstale":"2024-07-26T11:11:35.383917+0000","last_undegraded":"2024-07-26T11:11:35.383917+0000","last_fullsized":"2024-07-26T11:11:35.383917+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:28:50.167294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161762+0000","last_change":"2024-07-26T11:10:22.314219+0000","last_active":"2024-07-26T11:11:35.161762+0000","last_peered":"2024-07-26T11:11:35.161762+0000","last_clean":"2024-07-26T11:11:35.161762+0000","last_became_active":"2024-07-26T11:10:22.313811+0000","last_became_peered":"2024-07-26T11:10:22.313811+0000","last_unstale":"2024-07-26T11:11:35.161762+0000","last_undegraded":"2024-07-26T11:11:35.161762+0000","last_fullsized":"2024-07-26T11:11:35.161762+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:55:47.452798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.086736+0000","last_change":"2024-07-26T11:10:18.047818+0000","last_active":"2024-07-26T11:11:50.086736+0000","last_peered":"2024-07-26T11:11:50.086736+0000","last_clean":"2024-07-26T11:11:50.086736+0000","last_became_active":"2024-07-26T11:10:18.047159+0000","last_became_peered":"2024-07-26T11:10:18.047159+0000","last_unstale":"2024-07-26T11:11:50.086736+0000","last_undegraded":"2024-07-26T11:11:50.086736+0000","last_fullsized":"2024-07-26T11:11:50.086736+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:37.528512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162499+0000","last_change":"2024-07-26T11:10:35.996468+0000","last_active":"2024-07-26T11:11:35.162499+0000","last_peered":"2024-07-26T11:11:35.162499+0000","last_clean":"2024-07-26T11:11:35.162499+0000","last_became_active":"2024-07-26T11:10:35.996030+0000","last_became_peered":"2024-07-26T11:10:35.996030+0000","last_unstale":"2024-07-26T11:11:35.162499+0000","last_undegraded":"2024-07-26T11:11:35.162499+0000","last_fullsized":"2024-07-26T11:11:35.162499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:20:14.743690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.640574+0000","last_change":"2024-07-26T11:10:22.318375+0000","last_active":"2024-07-26T11:11:35.640574+0000","last_peered":"2024-07-26T11:11:35.640574+0000","last_clean":"2024-07-26T11:11:35.640574+0000","last_became_active":"2024-07-26T11:10:22.317965+0000","last_became_peered":"2024-07-26T11:10:22.317965+0000","last_unstale":"2024-07-26T11:11:35.640574+0000","last_undegraded":"2024-07-26T11:11:35.640574+0000","last_fullsized":"2024-07-26T11:11:35.640574+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:36:43.504045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.096145+0000","last_change":"2024-07-26T11:10:18.047957+0000","last_active":"2024-07-26T11:11:51.096145+0000","last_peered":"2024-07-26T11:11:51.096145+0000","last_clean":"2024-07-26T11:11:51.096145+0000","last_became_active":"2024-07-26T11:10:18.047495+0000","last_became_peered":"2024-07-26T11:10:18.047495+0000","last_unstale":"2024-07-26T11:11:51.096145+0000","last_undegraded":"2024-07-26T11:11:51.096145+0000","last_fullsized":"2024-07-26T11:11:51.096145+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:11.238295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158703+0000","last_change":"2024-07-26T11:10:35.976946+0000","last_active":"2024-07-26T11:11:35.158703+0000","last_peered":"2024-07-26T11:11:35.158703+0000","last_clean":"2024-07-26T11:11:35.158703+0000","last_became_active":"2024-07-26T11:10:35.976474+0000","last_became_peered":"2024-07-26T11:10:35.976474+0000","last_unstale":"2024-07-26T11:11:35.158703+0000","last_undegraded":"2024-07-26T11:11:35.158703+0000","last_fullsized":"2024-07-26T11:11:35.158703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:50:51.240903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158944+0000","last_change":"2024-07-26T11:10:22.302473+0000","last_active":"2024-07-26T11:11:35.158944+0000","last_peered":"2024-07-26T11:11:35.158944+0000","last_clean":"2024-07-26T11:11:35.158944+0000","last_became_active":"2024-07-26T11:10:22.302096+0000","last_became_peered":"2024-07-26T11:10:22.302096+0000","last_unstale":"2024-07-26T11:11:35.158944+0000","last_undegraded":"2024-07-26T11:11:35.158944+0000","last_fullsized":"2024-07-26T11:11:35.158944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:19.215301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'36","reported_seq":178,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:27.156090+0000","last_change":"2024-07-26T11:10:34.973502+0000","last_active":"2024-07-26T11:12:27.156090+0000","last_peered":"2024-07-26T11:12:27.156090+0000","last_clean":"2024-07-26T11:12:27.156090+0000","last_became_active":"2024-07-26T11:10:34.964155+0000","last_became_peered":"2024-07-26T11:10:34.964155+0000","last_unstale":"2024-07-26T11:12:27.156090+0000","last_undegraded":"2024-07-26T11:12:27.156090+0000","last_fullsized":"2024-07-26T11:12:27.156090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:23:06.652746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65382,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":84,"num_read_kb":106,"num_write":38,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.382816+0000","last_change":"2024-07-26T11:10:35.981439+0000","last_active":"2024-07-26T11:11:35.382816+0000","last_peered":"2024-07-26T11:11:35.382816+0000","last_clean":"2024-07-26T11:11:35.382816+0000","last_became_active":"2024-07-26T11:10:35.980878+0000","last_became_peered":"2024-07-26T11:10:35.980878+0000","last_unstale":"2024-07-26T11:11:35.382816+0000","last_undegraded":"2024-07-26T11:11:35.382816+0000","last_fullsized":"2024-07-26T11:11:35.382816+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:31.719845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.949755+0000","last_change":"2024-07-26T11:10:34.974292+0000","last_active":"2024-07-26T11:11:35.949755+0000","last_peered":"2024-07-26T11:11:35.949755+0000","last_clean":"2024-07-26T11:11:35.949755+0000","last_became_active":"2024-07-26T11:10:34.973820+0000","last_became_peered":"2024-07-26T11:10:34.973820+0000","last_unstale":"2024-07-26T11:11:35.949755+0000","last_undegraded":"2024-07-26T11:11:35.949755+0000","last_fullsized":"2024-07-26T11:11:35.949755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:35:37.629313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:49.063298+0000","last_change":"2024-07-26T11:10:18.055248+0000","last_active":"2024-07-26T11:11:49.063298+0000","last_peered":"2024-07-26T11:11:49.063298+0000","last_clean":"2024-07-26T11:11:49.063298+0000","last_became_active":"2024-07-26T11:10:18.054882+0000","last_became_peered":"2024-07-26T11:10:18.054882+0000","last_unstale":"2024-07-26T11:11:49.063298+0000","last_undegraded":"2024-07-26T11:11:49.063298+0000","last_fullsized":"2024-07-26T11:11:49.063298+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:16.390058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.407324+0000","last_change":"2024-07-26T11:10:35.985628+0000","last_active":"2024-07-26T11:11:35.407324+0000","last_peered":"2024-07-26T11:11:35.407324+0000","last_clean":"2024-07-26T11:11:35.407324+0000","last_became_active":"2024-07-26T11:10:35.985156+0000","last_became_peered":"2024-07-26T11:10:35.985156+0000","last_unstale":"2024-07-26T11:11:35.407324+0000","last_undegraded":"2024-07-26T11:11:35.407324+0000","last_fullsized":"2024-07-26T11:11:35.407324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:29:18.891825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.186805+0000","last_change":"2024-07-26T11:10:34.973261+0000","last_active":"2024-07-26T11:11:35.186805+0000","last_peered":"2024-07-26T11:11:35.186805+0000","last_clean":"2024-07-26T11:11:35.186805+0000","last_became_active":"2024-07-26T11:10:34.971592+0000","last_became_peered":"2024-07-26T11:10:34.971592+0000","last_unstale":"2024-07-26T11:11:35.186805+0000","last_undegraded":"2024-07-26T11:11:35.186805+0000","last_fullsized":"2024-07-26T11:11:35.186805+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:58:13.517884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.082517+0000","last_change":"2024-07-26T11:10:34.962248+0000","last_active":"2024-07-26T11:11:50.082517+0000","last_peered":"2024-07-26T11:11:50.082517+0000","last_clean":"2024-07-26T11:11:50.082517+0000","last_became_active":"2024-07-26T11:10:34.961683+0000","last_became_peered":"2024-07-26T11:10:34.961683+0000","last_unstale":"2024-07-26T11:11:50.082517+0000","last_undegraded":"2024-07-26T11:11:50.082517+0000","last_fullsized":"2024-07-26T11:11:50.082517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:17:08.308226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.381770+0000","last_change":"2024-07-26T11:10:35.995760+0000","last_active":"2024-07-26T11:11:35.381770+0000","last_peered":"2024-07-26T11:11:35.381770+0000","last_clean":"2024-07-26T11:11:35.381770+0000","last_became_active":"2024-07-26T11:10:35.995328+0000","last_became_peered":"2024-07-26T11:10:35.995328+0000","last_unstale":"2024-07-26T11:11:35.381770+0000","last_undegraded":"2024-07-26T11:11:35.381770+0000","last_fullsized":"2024-07-26T11:11:35.381770+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:40:33.093315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.950002+0000","last_change":"2024-07-26T11:10:22.316151+0000","last_active":"2024-07-26T11:11:35.950002+0000","last_peered":"2024-07-26T11:11:35.950002+0000","last_clean":"2024-07-26T11:11:35.950002+0000","last_became_active":"2024-07-26T11:10:22.315722+0000","last_became_peered":"2024-07-26T11:10:22.315722+0000","last_unstale":"2024-07-26T11:11:35.950002+0000","last_undegraded":"2024-07-26T11:11:35.950002+0000","last_fullsized":"2024-07-26T11:11:35.950002+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:12:58.342637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":119,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:27.154323+0000","last_change":"2024-07-26T11:10:18.046327+0000","last_active":"2024-07-26T11:12:27.154323+0000","last_peered":"2024-07-26T11:12:27.154323+0000","last_clean":"2024-07-26T11:12:27.154323+0000","last_became_active":"2024-07-26T11:10:18.045913+0000","last_became_peered":"2024-07-26T11:10:18.045913+0000","last_unstale":"2024-07-26T11:12:27.154323+0000","last_undegraded":"2024-07-26T11:12:27.154323+0000","last_fullsized":"2024-07-26T11:12:27.154323+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:58.889046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.147636+0000","last_change":"2024-07-26T11:10:35.983264+0000","last_active":"2024-07-26T11:11:36.147636+0000","last_peered":"2024-07-26T11:11:36.147636+0000","last_clean":"2024-07-26T11:11:36.147636+0000","last_became_active":"2024-07-26T11:10:35.982210+0000","last_became_peered":"2024-07-26T11:10:35.982210+0000","last_unstale":"2024-07-26T11:11:36.147636+0000","last_undegraded":"2024-07-26T11:11:36.147636+0000","last_fullsized":"2024-07-26T11:11:36.147636+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:37:20.272725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.185660+0000","last_change":"2024-07-26T11:10:35.985335+0000","last_active":"2024-07-26T11:11:35.185660+0000","last_peered":"2024-07-26T11:11:35.185660+0000","last_clean":"2024-07-26T11:11:35.185660+0000","last_became_active":"2024-07-26T11:10:35.983667+0000","last_became_peered":"2024-07-26T11:10:35.983667+0000","last_unstale":"2024-07-26T11:11:35.185660+0000","last_undegraded":"2024-07-26T11:11:35.185660+0000","last_fullsized":"2024-07-26T11:11:35.185660+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:00:17.744806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.382004+0000","last_change":"2024-07-26T11:10:22.333494+0000","last_active":"2024-07-26T11:11:35.382004+0000","last_peered":"2024-07-26T11:11:35.382004+0000","last_clean":"2024-07-26T11:11:35.382004+0000","last_became_active":"2024-07-26T11:10:22.332950+0000","last_became_peered":"2024-07-26T11:10:22.332950+0000","last_unstale":"2024-07-26T11:11:35.382004+0000","last_undegraded":"2024-07-26T11:11:35.382004+0000","last_fullsized":"2024-07-26T11:11:35.382004+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:58:38.340857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.082572+0000","last_change":"2024-07-26T11:10:18.034057+0000","last_active":"2024-07-26T11:11:50.082572+0000","last_peered":"2024-07-26T11:11:50.082572+0000","last_clean":"2024-07-26T11:11:50.082572+0000","last_became_active":"2024-07-26T11:10:18.033119+0000","last_became_peered":"2024-07-26T11:10:18.033119+0000","last_unstale":"2024-07-26T11:11:50.082572+0000","last_undegraded":"2024-07-26T11:11:50.082572+0000","last_fullsized":"2024-07-26T11:11:50.082572+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:21.675593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.408475+0000","last_change":"2024-07-26T11:11:35.408377+0000","last_active":"2024-07-26T11:11:35.408475+0000","last_peered":"2024-07-26T11:11:35.408475+0000","last_clean":"2024-07-26T11:11:35.408475+0000","last_became_active":"2024-07-26T11:11:35.407865+0000","last_became_peered":"2024-07-26T11:11:35.407865+0000","last_unstale":"2024-07-26T11:11:35.408475+0000","last_undegraded":"2024-07-26T11:11:35.408475+0000","last_fullsized":"2024-07-26T11:11:35.408475+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:43:02.953395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.147621+0000","last_change":"2024-07-26T11:10:22.323447+0000","last_active":"2024-07-26T11:11:36.147621+0000","last_peered":"2024-07-26T11:11:36.147621+0000","last_clean":"2024-07-26T11:11:36.147621+0000","last_became_active":"2024-07-26T11:10:22.322988+0000","last_became_peered":"2024-07-26T11:10:22.322988+0000","last_unstale":"2024-07-26T11:11:36.147621+0000","last_undegraded":"2024-07-26T11:11:36.147621+0000","last_fullsized":"2024-07-26T11:11:36.147621+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:14.880985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:48.055233+0000","last_change":"2024-07-26T11:10:18.048724+0000","last_active":"2024-07-26T11:11:48.055233+0000","last_peered":"2024-07-26T11:11:48.055233+0000","last_clean":"2024-07-26T11:11:48.055233+0000","last_became_active":"2024-07-26T11:10:18.048257+0000","last_became_peered":"2024-07-26T11:10:18.048257+0000","last_unstale":"2024-07-26T11:11:48.055233+0000","last_undegraded":"2024-07-26T11:11:48.055233+0000","last_fullsized":"2024-07-26T11:11:48.055233+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:02:13.581446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:49.071327+0000","last_change":"2024-07-26T11:10:18.044171+0000","last_active":"2024-07-26T11:11:49.071327+0000","last_peered":"2024-07-26T11:11:49.071327+0000","last_clean":"2024-07-26T11:11:49.071327+0000","last_became_active":"2024-07-26T11:10:18.043682+0000","last_became_peered":"2024-07-26T11:10:18.043682+0000","last_unstale":"2024-07-26T11:11:49.071327+0000","last_undegraded":"2024-07-26T11:11:49.071327+0000","last_fullsized":"2024-07-26T11:11:49.071327+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:46:46.864119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158875+0000","last_change":"2024-07-26T11:10:22.321081+0000","last_active":"2024-07-26T11:11:35.158875+0000","last_peered":"2024-07-26T11:11:35.158875+0000","last_clean":"2024-07-26T11:11:35.158875+0000","last_became_active":"2024-07-26T11:10:22.320656+0000","last_became_peered":"2024-07-26T11:10:22.320656+0000","last_unstale":"2024-07-26T11:11:35.158875+0000","last_undegraded":"2024-07-26T11:11:35.158875+0000","last_fullsized":"2024-07-26T11:11:35.158875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:17:20.455129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156131+0000","last_change":"2024-07-26T11:10:36.000224+0000","last_active":"2024-07-26T11:11:35.156131+0000","last_peered":"2024-07-26T11:11:35.156131+0000","last_clean":"2024-07-26T11:11:35.156131+0000","last_became_active":"2024-07-26T11:10:35.999733+0000","last_became_peered":"2024-07-26T11:10:35.999733+0000","last_unstale":"2024-07-26T11:11:35.156131+0000","last_undegraded":"2024-07-26T11:11:35.156131+0000","last_fullsized":"2024-07-26T11:11:35.156131+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:49.912401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:27.155234+0000","last_change":"2024-07-26T11:10:18.034585+0000","last_active":"2024-07-26T11:12:27.155234+0000","last_peered":"2024-07-26T11:12:27.155234+0000","last_clean":"2024-07-26T11:12:27.155234+0000","last_became_active":"2024-07-26T11:10:18.034237+0000","last_became_peered":"2024-07-26T11:10:18.034237+0000","last_unstale":"2024-07-26T11:12:27.155234+0000","last_undegraded":"2024-07-26T11:12:27.155234+0000","last_fullsized":"2024-07-26T11:12:27.155234+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:22.670277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158605+0000","last_change":"2024-07-26T11:10:22.319282+0000","last_active":"2024-07-26T11:11:35.158605+0000","last_peered":"2024-07-26T11:11:35.158605+0000","last_clean":"2024-07-26T11:11:35.158605+0000","last_became_active":"2024-07-26T11:10:22.318822+0000","last_became_peered":"2024-07-26T11:10:22.318822+0000","last_unstale":"2024-07-26T11:11:35.158605+0000","last_undegraded":"2024-07-26T11:11:35.158605+0000","last_fullsized":"2024-07-26T11:11:35.158605+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:50:57.615839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156337+0000","last_change":"2024-07-26T11:10:35.970189+0000","last_active":"2024-07-26T11:11:35.156337+0000","last_peered":"2024-07-26T11:11:35.156337+0000","last_clean":"2024-07-26T11:11:35.156337+0000","last_became_active":"2024-07-26T11:10:35.969574+0000","last_became_peered":"2024-07-26T11:10:35.969574+0000","last_unstale":"2024-07-26T11:11:35.156337+0000","last_undegraded":"2024-07-26T11:11:35.156337+0000","last_fullsized":"2024-07-26T11:11:35.156337+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:07.487944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162102+0000","last_change":"2024-07-26T11:10:22.299152+0000","last_active":"2024-07-26T11:11:35.162102+0000","last_peered":"2024-07-26T11:11:35.162102+0000","last_clean":"2024-07-26T11:11:35.162102+0000","last_became_active":"2024-07-26T11:10:22.298707+0000","last_became_peered":"2024-07-26T11:10:22.298707+0000","last_unstale":"2024-07-26T11:11:35.162102+0000","last_undegraded":"2024-07-26T11:11:35.162102+0000","last_fullsized":"2024-07-26T11:11:35.162102+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:25.161634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156068+0000","last_change":"2024-07-26T11:10:22.302476+0000","last_active":"2024-07-26T11:11:35.156068+0000","last_peered":"2024-07-26T11:11:35.156068+0000","last_clean":"2024-07-26T11:11:35.156068+0000","last_became_active":"2024-07-26T11:10:22.302091+0000","last_became_peered":"2024-07-26T11:10:22.302091+0000","last_unstale":"2024-07-26T11:11:35.156068+0000","last_undegraded":"2024-07-26T11:11:35.156068+0000","last_fullsized":"2024-07-26T11:11:35.156068+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:06:18.587588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.155820+0000","last_change":"2024-07-26T11:10:22.326583+0000","last_active":"2024-07-26T11:11:35.155820+0000","last_peered":"2024-07-26T11:11:35.155820+0000","last_clean":"2024-07-26T11:11:35.155820+0000","last_became_active":"2024-07-26T11:10:22.326185+0000","last_became_peered":"2024-07-26T11:10:22.326185+0000","last_unstale":"2024-07-26T11:11:35.155820+0000","last_undegraded":"2024-07-26T11:11:35.155820+0000","last_fullsized":"2024-07-26T11:11:35.155820+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:54:00.148964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.186262+0000","last_change":"2024-07-26T11:10:22.329247+0000","last_active":"2024-07-26T11:11:35.186262+0000","last_peered":"2024-07-26T11:11:35.186262+0000","last_clean":"2024-07-26T11:11:35.186262+0000","last_became_active":"2024-07-26T11:10:22.328585+0000","last_became_peered":"2024-07-26T11:10:22.328585+0000","last_unstale":"2024-07-26T11:11:35.186262+0000","last_undegraded":"2024-07-26T11:11:35.186262+0000","last_fullsized":"2024-07-26T11:11:35.186262+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:01:27.195399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.641804+0000","last_change":"2024-07-26T11:10:22.330717+0000","last_active":"2024-07-26T11:11:35.641804+0000","last_peered":"2024-07-26T11:11:35.641804+0000","last_clean":"2024-07-26T11:11:35.641804+0000","last_became_active":"2024-07-26T11:10:22.329576+0000","last_became_peered":"2024-07-26T11:10:22.329576+0000","last_unstale":"2024-07-26T11:11:35.641804+0000","last_undegraded":"2024-07-26T11:11:35.641804+0000","last_fullsized":"2024-07-26T11:11:35.641804+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:21.970629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162998+0000","last_change":"2024-07-26T11:10:22.307445+0000","last_active":"2024-07-26T11:11:35.162998+0000","last_peered":"2024-07-26T11:11:35.162998+0000","last_clean":"2024-07-26T11:11:35.162998+0000","last_became_active":"2024-07-26T11:10:22.305738+0000","last_became_peered":"2024-07-26T11:10:22.305738+0000","last_unstale":"2024-07-26T11:11:35.162998+0000","last_undegraded":"2024-07-26T11:11:35.162998+0000","last_fullsized":"2024-07-26T11:11:35.162998+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:09:08.125209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156570+0000","last_change":"2024-07-26T11:10:22.298392+0000","last_active":"2024-07-26T11:11:35.156570+0000","last_peered":"2024-07-26T11:11:35.156570+0000","last_clean":"2024-07-26T11:11:35.156570+0000","last_became_active":"2024-07-26T11:10:22.297979+0000","last_became_peered":"2024-07-26T11:10:22.297979+0000","last_unstale":"2024-07-26T11:11:35.156570+0000","last_undegraded":"2024-07-26T11:11:35.156570+0000","last_fullsized":"2024-07-26T11:11:35.156570+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:30:39.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.148623+0000","last_change":"2024-07-26T11:10:22.320850+0000","last_active":"2024-07-26T11:11:36.148623+0000","last_peered":"2024-07-26T11:11:36.148623+0000","last_clean":"2024-07-26T11:11:36.148623+0000","last_became_active":"2024-07-26T11:10:22.319838+0000","last_became_peered":"2024-07-26T11:10:22.319838+0000","last_unstale":"2024-07-26T11:11:36.148623+0000","last_undegraded":"2024-07-26T11:11:36.148623+0000","last_fullsized":"2024-07-26T11:11:36.148623+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:14.289366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158740+0000","last_change":"2024-07-26T11:10:22.321305+0000","last_active":"2024-07-26T11:11:35.158740+0000","last_peered":"2024-07-26T11:11:35.158740+0000","last_clean":"2024-07-26T11:11:35.158740+0000","last_became_active":"2024-07-26T11:10:22.320733+0000","last_became_peered":"2024-07-26T11:10:22.320733+0000","last_unstale":"2024-07-26T11:11:35.158740+0000","last_undegraded":"2024-07-26T11:11:35.158740+0000","last_fullsized":"2024-07-26T11:11:35.158740+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:24:33.125828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.147831+0000","last_change":"2024-07-26T11:10:22.318193+0000","last_active":"2024-07-26T11:11:36.147831+0000","last_peered":"2024-07-26T11:11:36.147831+0000","last_clean":"2024-07-26T11:11:36.147831+0000","last_became_active":"2024-07-26T11:10:22.317819+0000","last_became_peered":"2024-07-26T11:10:22.317819+0000","last_unstale":"2024-07-26T11:11:36.147831+0000","last_undegraded":"2024-07-26T11:11:36.147831+0000","last_fullsized":"2024-07-26T11:11:36.147831+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:14:06.105152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.949718+0000","last_change":"2024-07-26T11:10:22.318222+0000","last_active":"2024-07-26T11:11:35.949718+0000","last_peered":"2024-07-26T11:11:35.949718+0000","last_clean":"2024-07-26T11:11:35.949718+0000","last_became_active":"2024-07-26T11:10:22.317806+0000","last_became_peered":"2024-07-26T11:10:22.317806+0000","last_unstale":"2024-07-26T11:11:35.949718+0000","last_undegraded":"2024-07-26T11:11:35.949718+0000","last_fullsized":"2024-07-26T11:11:35.949718+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:55:27.510542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.641468+0000","last_change":"2024-07-26T11:10:22.330166+0000","last_active":"2024-07-26T11:11:35.641468+0000","last_peered":"2024-07-26T11:11:35.641468+0000","last_clean":"2024-07-26T11:11:35.641468+0000","last_became_active":"2024-07-26T11:10:22.329722+0000","last_became_peered":"2024-07-26T11:10:22.329722+0000","last_unstale":"2024-07-26T11:11:35.641468+0000","last_undegraded":"2024-07-26T11:11:35.641468+0000","last_fullsized":"2024-07-26T11:11:35.641468+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:58.816139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162551+0000","last_change":"2024-07-26T11:10:22.302774+0000","last_active":"2024-07-26T11:11:35.162551+0000","last_peered":"2024-07-26T11:11:35.162551+0000","last_clean":"2024-07-26T11:11:35.162551+0000","last_became_active":"2024-07-26T11:10:22.302382+0000","last_became_peered":"2024-07-26T11:10:22.302382+0000","last_unstale":"2024-07-26T11:11:35.162551+0000","last_undegraded":"2024-07-26T11:11:35.162551+0000","last_fullsized":"2024-07-26T11:11:35.162551+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:02:58.019598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156478+0000","last_change":"2024-07-26T11:10:22.312475+0000","last_active":"2024-07-26T11:11:35.156478+0000","last_peered":"2024-07-26T11:11:35.156478+0000","last_clean":"2024-07-26T11:11:35.156478+0000","last_became_active":"2024-07-26T11:10:22.312078+0000","last_became_peered":"2024-07-26T11:10:22.312078+0000","last_unstale":"2024-07-26T11:11:35.156478+0000","last_undegraded":"2024-07-26T11:11:35.156478+0000","last_fullsized":"2024-07-26T11:11:35.156478+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:25:37.705136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.383965+0000","last_change":"2024-07-26T11:10:22.314776+0000","last_active":"2024-07-26T11:11:35.383965+0000","last_peered":"2024-07-26T11:11:35.383965+0000","last_clean":"2024-07-26T11:11:35.383965+0000","last_became_active":"2024-07-26T11:10:22.307808+0000","last_became_peered":"2024-07-26T11:10:22.307808+0000","last_unstale":"2024-07-26T11:11:35.383965+0000","last_undegraded":"2024-07-26T11:11:35.383965+0000","last_fullsized":"2024-07-26T11:11:35.383965+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:44:06.152799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.163370+0000","last_change":"2024-07-26T11:10:22.328102+0000","last_active":"2024-07-26T11:11:35.163370+0000","last_peered":"2024-07-26T11:11:35.163370+0000","last_clean":"2024-07-26T11:11:35.163370+0000","last_became_active":"2024-07-26T11:10:22.327779+0000","last_became_peered":"2024-07-26T11:10:22.327779+0000","last_unstale":"2024-07-26T11:11:35.163370+0000","last_undegraded":"2024-07-26T11:11:35.163370+0000","last_fullsized":"2024-07-26T11:11:35.163370+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:14:34.569748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.949239+0000","last_change":"2024-07-26T11:10:22.329057+0000","last_active":"2024-07-26T11:11:35.949239+0000","last_peered":"2024-07-26T11:11:35.949239+0000","last_clean":"2024-07-26T11:11:35.949239+0000","last_became_active":"2024-07-26T11:10:22.328656+0000","last_became_peered":"2024-07-26T11:10:22.328656+0000","last_unstale":"2024-07-26T11:11:35.949239+0000","last_undegraded":"2024-07-26T11:11:35.949239+0000","last_fullsized":"2024-07-26T11:11:35.949239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:18:39.861674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.185692+0000","last_change":"2024-07-26T11:10:22.326115+0000","last_active":"2024-07-26T11:11:35.185692+0000","last_peered":"2024-07-26T11:11:35.185692+0000","last_clean":"2024-07-26T11:11:35.185692+0000","last_became_active":"2024-07-26T11:10:22.325538+0000","last_became_peered":"2024-07-26T11:10:22.325538+0000","last_unstale":"2024-07-26T11:11:35.185692+0000","last_undegraded":"2024-07-26T11:11:35.185692+0000","last_fullsized":"2024-07-26T11:11:35.185692+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:18:19.339571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.186243+0000","last_change":"2024-07-26T11:10:22.326910+0000","last_active":"2024-07-26T11:11:35.186243+0000","last_peered":"2024-07-26T11:11:35.186243+0000","last_clean":"2024-07-26T11:11:35.186243+0000","last_became_active":"2024-07-26T11:10:22.324532+0000","last_became_peered":"2024-07-26T11:10:22.324532+0000","last_unstale":"2024-07-26T11:11:35.186243+0000","last_undegraded":"2024-07-26T11:11:35.186243+0000","last_fullsized":"2024-07-26T11:11:35.186243+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:33.671780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.641379+0000","last_change":"2024-07-26T11:10:22.312926+0000","last_active":"2024-07-26T11:11:35.641379+0000","last_peered":"2024-07-26T11:11:35.641379+0000","last_clean":"2024-07-26T11:11:35.641379+0000","last_became_active":"2024-07-26T11:10:22.311642+0000","last_became_peered":"2024-07-26T11:10:22.311642+0000","last_unstale":"2024-07-26T11:11:35.641379+0000","last_undegraded":"2024-07-26T11:11:35.641379+0000","last_fullsized":"2024-07-26T11:11:35.641379+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:56:29.644965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.383396+0000","last_change":"2024-07-26T11:10:34.976934+0000","last_active":"2024-07-26T11:11:35.383396+0000","last_peered":"2024-07-26T11:11:35.383396+0000","last_clean":"2024-07-26T11:11:35.383396+0000","last_became_active":"2024-07-26T11:10:34.976551+0000","last_became_peered":"2024-07-26T11:10:34.976551+0000","last_unstale":"2024-07-26T11:11:35.383396+0000","last_undegraded":"2024-07-26T11:11:35.383396+0000","last_fullsized":"2024-07-26T11:11:35.383396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:11:43.883644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161855+0000","last_change":"2024-07-26T11:10:22.304444+0000","last_active":"2024-07-26T11:11:35.161855+0000","last_peered":"2024-07-26T11:11:35.161855+0000","last_clean":"2024-07-26T11:11:35.161855+0000","last_became_active":"2024-07-26T11:10:22.302825+0000","last_became_peered":"2024-07-26T11:10:22.302825+0000","last_unstale":"2024-07-26T11:11:35.161855+0000","last_undegraded":"2024-07-26T11:11:35.161855+0000","last_fullsized":"2024-07-26T11:11:35.161855+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:15.229057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156568+0000","last_change":"2024-07-26T11:10:22.297515+0000","last_active":"2024-07-26T11:11:35.156568+0000","last_peered":"2024-07-26T11:11:35.156568+0000","last_clean":"2024-07-26T11:11:35.156568+0000","last_became_active":"2024-07-26T11:10:22.297134+0000","last_became_peered":"2024-07-26T11:10:22.297134+0000","last_unstale":"2024-07-26T11:11:35.156568+0000","last_undegraded":"2024-07-26T11:11:35.156568+0000","last_fullsized":"2024-07-26T11:11:35.156568+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:25:08.794363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.950147+0000","last_change":"2024-07-26T11:10:22.304333+0000","last_active":"2024-07-26T11:11:35.950147+0000","last_peered":"2024-07-26T11:11:35.950147+0000","last_clean":"2024-07-26T11:11:35.950147+0000","last_became_active":"2024-07-26T11:10:22.303639+0000","last_became_peered":"2024-07-26T11:10:22.303639+0000","last_unstale":"2024-07-26T11:11:35.950147+0000","last_undegraded":"2024-07-26T11:11:35.950147+0000","last_fullsized":"2024-07-26T11:11:35.950147+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:47.053463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.147598+0000","last_change":"2024-07-26T11:10:22.320998+0000","last_active":"2024-07-26T11:11:36.147598+0000","last_peered":"2024-07-26T11:11:36.147598+0000","last_clean":"2024-07-26T11:11:36.147598+0000","last_became_active":"2024-07-26T11:10:22.320610+0000","last_became_peered":"2024-07-26T11:10:22.320610+0000","last_unstale":"2024-07-26T11:11:36.147598+0000","last_undegraded":"2024-07-26T11:11:36.147598+0000","last_fullsized":"2024-07-26T11:11:36.147598+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:52:28.802241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161283+0000","last_change":"2024-07-26T11:10:34.970486+0000","last_active":"2024-07-26T11:11:35.161283+0000","last_peered":"2024-07-26T11:11:35.161283+0000","last_clean":"2024-07-26T11:11:35.161283+0000","last_became_active":"2024-07-26T11:10:34.969965+0000","last_became_peered":"2024-07-26T11:10:34.969965+0000","last_unstale":"2024-07-26T11:11:35.161283+0000","last_undegraded":"2024-07-26T11:11:35.161283+0000","last_fullsized":"2024-07-26T11:11:35.161283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:41.593251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.155988+0000","last_change":"2024-07-26T11:10:22.321261+0000","last_active":"2024-07-26T11:11:35.155988+0000","last_peered":"2024-07-26T11:11:35.155988+0000","last_clean":"2024-07-26T11:11:35.155988+0000","last_became_active":"2024-07-26T11:10:22.320819+0000","last_became_peered":"2024-07-26T11:10:22.320819+0000","last_unstale":"2024-07-26T11:11:35.155988+0000","last_undegraded":"2024-07-26T11:11:35.155988+0000","last_fullsized":"2024-07-26T11:11:35.155988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:04:56.872600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.157906+0000","last_change":"2024-07-26T11:10:22.321206+0000","last_active":"2024-07-26T11:11:35.157906+0000","last_peered":"2024-07-26T11:11:35.157906+0000","last_clean":"2024-07-26T11:11:35.157906+0000","last_became_active":"2024-07-26T11:10:22.320700+0000","last_became_peered":"2024-07-26T11:10:22.320700+0000","last_unstale":"2024-07-26T11:11:35.157906+0000","last_undegraded":"2024-07-26T11:11:35.157906+0000","last_fullsized":"2024-07-26T11:11:35.157906+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:31:20.700808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.407201+0000","last_change":"2024-07-26T11:10:22.300061+0000","last_active":"2024-07-26T11:11:35.407201+0000","last_peered":"2024-07-26T11:11:35.407201+0000","last_clean":"2024-07-26T11:11:35.407201+0000","last_became_active":"2024-07-26T11:10:22.298737+0000","last_became_peered":"2024-07-26T11:10:22.298737+0000","last_unstale":"2024-07-26T11:11:35.407201+0000","last_undegraded":"2024-07-26T11:11:35.407201+0000","last_fullsized":"2024-07-26T11:11:35.407201+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:45:25.500816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.185646+0000","last_change":"2024-07-26T11:10:22.328534+0000","last_active":"2024-07-26T11:11:35.185646+0000","last_peered":"2024-07-26T11:11:35.185646+0000","last_clean":"2024-07-26T11:11:35.185646+0000","last_became_active":"2024-07-26T11:10:22.327929+0000","last_became_peered":"2024-07-26T11:10:22.327929+0000","last_unstale":"2024-07-26T11:11:35.185646+0000","last_undegraded":"2024-07-26T11:11:35.185646+0000","last_fullsized":"2024-07-26T11:11:35.185646+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:09:55.152881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.186221+0000","last_change":"2024-07-26T11:10:22.324753+0000","last_active":"2024-07-26T11:11:35.186221+0000","last_peered":"2024-07-26T11:11:35.186221+0000","last_clean":"2024-07-26T11:11:35.186221+0000","last_became_active":"2024-07-26T11:10:22.324307+0000","last_became_peered":"2024-07-26T11:10:22.324307+0000","last_unstale":"2024-07-26T11:11:35.186221+0000","last_undegraded":"2024-07-26T11:11:35.186221+0000","last_fullsized":"2024-07-26T11:11:35.186221+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:40.566465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.163290+0000","last_change":"2024-07-26T11:10:22.328183+0000","last_active":"2024-07-26T11:11:35.163290+0000","last_peered":"2024-07-26T11:11:35.163290+0000","last_clean":"2024-07-26T11:11:35.163290+0000","last_became_active":"2024-07-26T11:10:22.327805+0000","last_became_peered":"2024-07-26T11:10:22.327805+0000","last_unstale":"2024-07-26T11:11:35.163290+0000","last_undegraded":"2024-07-26T11:11:35.163290+0000","last_fullsized":"2024-07-26T11:11:35.163290+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:11:23.238390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72195,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":409,"num_read_kb":283,"num_write":250,"num_write_kb":144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2194,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":573440,"data_stored":212320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":73,"seq":313532612649,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":527167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808877,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28704,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716448,"statfs":{"total":95995035648,"available":95965642752,"internally_reserved":0,"allocated":1728512,"data_stored":1178502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972403,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":529939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168634,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":529735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364862,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28620,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716532,"statfs":{"total":95995035648,"available":95965728768,"internally_reserved":0,"allocated":1642496,"data_stored":1122046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528388,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":521347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724620,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36952,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708200,"statfs":{"total":95995035648,"available":95957196800,"internally_reserved":0,"allocated":1654784,"data_stored":1113813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888144,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":586929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084373,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1044480,"data_stored":521055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378715,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":522597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542240,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1101824,"data_stored":587759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705764,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1056768,"data_stored":522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":67134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":62208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":67509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T11:12:32.288 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-26T11:12:32.289 INFO:tasks.cephadm:Setting up client nodes... 2024-07-26T11:12:32.289 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-07-26T11:12:32.798 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:33.200 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:33 smithi134 ceph-mon[31966]: from='client.24941 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:12:33.201 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:33 smithi134 ceph-mon[31966]: pgmap v377: 129 pgs: 129 active+clean; 647 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1023 B/s wr, 10 op/s 2024-07-26T11:12:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:33 smithi193 ceph-mon[37473]: from='client.24941 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:12:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:33 smithi193 ceph-mon[37473]: pgmap v377: 129 pgs: 129 active+clean; 647 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1023 B/s wr, 10 op/s 2024-07-26T11:12:33.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:33 smithi189 ceph-mon[36790]: from='client.24941 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:12:33.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:33 smithi189 ceph-mon[36790]: pgmap v377: 129 pgs: 129 active+clean; 647 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1023 B/s wr, 10 op/s 2024-07-26T11:12:35.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.081+0000 7f3c25a00640 1 -- 172.21.15.134:0/3505603888 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c2010b870 msgr2=0x7f3c201120a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.081+0000 7f3c25a00640 1 --2- 172.21.15.134:0/3505603888 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c2010b870 0x7f3c201120a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3c18009ca0 tx=0x7f3c1802a8c0 comp rx=0 tx=0).stop 2024-07-26T11:12:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.081+0000 7f3c25a00640 1 -- 172.21.15.134:0/3505603888 shutdown_connections 2024-07-26T11:12:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.081+0000 7f3c25a00640 1 --2- 172.21.15.134:0/3505603888 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c2010b870 0x7f3c201120a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.081+0000 7f3c25a00640 1 --2- 172.21.15.134:0/3505603888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2010af20 0x7f3c2010b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.081+0000 7f3c25a00640 1 --2- 172.21.15.134:0/3505603888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c201055e0 0x7f3c201059f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.081+0000 7f3c25a00640 1 -- 172.21.15.134:0/3505603888 >> 172.21.15.134:0/3505603888 conn(0x7f3c20101010 msgr2=0x7f3c20103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.081+0000 7f3c25a00640 1 -- 172.21.15.134:0/3505603888 shutdown_connections 2024-07-26T11:12:35.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.081+0000 7f3c25a00640 1 -- 172.21.15.134:0/3505603888 wait complete. 2024-07-26T11:12:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.082+0000 7f3c25a00640 1 Processor -- start 2024-07-26T11:12:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.082+0000 7f3c25a00640 1 -- start start 2024-07-26T11:12:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.083+0000 7f3c25a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c201055e0 0x7f3c2019bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.083+0000 7f3c25a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c2010af20 0x7f3c2019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:35.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.083+0000 7f3c25a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2010b870 0x7f3c201a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.083+0000 7f3c25a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3c20115540 con 0x7f3c2010af20 2024-07-26T11:12:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.083+0000 7f3c25a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3c201153c0 con 0x7f3c2010b870 2024-07-26T11:12:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.083+0000 7f3c25a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3c201156c0 con 0x7f3c201055e0 2024-07-26T11:12:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.083+0000 7f3c1fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c201055e0 0x7f3c2019bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.083+0000 7f3c17e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c2010af20 0x7f3c2019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.083+0000 7f3c17e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c2010af20 0x7f3c2019c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57638/0 (socket says 172.21.15.134:57638) 2024-07-26T11:12:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.083+0000 7f3c17e00640 1 -- 172.21.15.134:0/2176657676 learned_addr learned my addr 172.21.15.134:0/2176657676 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:12:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.083+0000 7f3c1fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c201055e0 0x7f3c2019bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52506/0 (socket says 172.21.15.134:52506) 2024-07-26T11:12:35.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.083+0000 7f3c25000640 1 --2- 172.21.15.134:0/2176657676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2010b870 0x7f3c201a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:35.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.084+0000 7f3c1fe00640 1 -- 172.21.15.134:0/2176657676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2010b870 msgr2=0x7f3c201a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:35.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.084+0000 7f3c1fe00640 1 --2- 172.21.15.134:0/2176657676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2010b870 0x7f3c201a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:35.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.084+0000 7f3c1fe00640 1 -- 172.21.15.134:0/2176657676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c2010af20 msgr2=0x7f3c2019c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:35.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.084+0000 7f3c1fe00640 1 --2- 172.21.15.134:0/2176657676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c2010af20 0x7f3c2019c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:35.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.084+0000 7f3c1fe00640 1 -- 172.21.15.134:0/2176657676 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c201a1ee0 con 0x7f3c201055e0 2024-07-26T11:12:35.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.084+0000 7f3c1fe00640 1 --2- 172.21.15.134:0/2176657676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c201055e0 0x7f3c2019bf70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000e9f0 tx=0x7f3c1000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:35.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.084+0000 7f3c1d600640 1 -- 172.21.15.134:0/2176657676 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1000cd80 con 0x7f3c201055e0 2024-07-26T11:12:35.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.084+0000 7f3c1d600640 1 -- 172.21.15.134:0/2176657676 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c100108e0 con 0x7f3c201055e0 2024-07-26T11:12:35.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.084+0000 7f3c1d600640 1 -- 172.21.15.134:0/2176657676 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c100188a0 con 0x7f3c201055e0 2024-07-26T11:12:35.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.084+0000 7f3c25a00640 1 -- 172.21.15.134:0/2176657676 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c201a2170 con 0x7f3c201055e0 2024-07-26T11:12:35.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.084+0000 7f3c25a00640 1 -- 172.21.15.134:0/2176657676 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c201a8930 con 0x7f3c201055e0 2024-07-26T11:12:35.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.086+0000 7f3c25a00640 1 -- 172.21.15.134:0/2176657676 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c201059f0 con 0x7f3c201055e0 2024-07-26T11:12:35.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.088+0000 7f3c1d600640 1 -- 172.21.15.134:0/2176657676 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f3c10010a90 con 0x7f3c201055e0 2024-07-26T11:12:35.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.089+0000 7f3c1d600640 1 --2- 172.21.15.134:0/2176657676 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3c040b3d20 0x7f3c040b6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:35.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.090+0000 7f3c1d600640 1 -- 172.21.15.134:0/2176657676 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3c100e45f0 con 0x7f3c201055e0 2024-07-26T11:12:35.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.090+0000 7f3c17e00640 1 --2- 172.21.15.134:0/2176657676 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3c040b3d20 0x7f3c040b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:35.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.091+0000 7f3c17e00640 1 --2- 172.21.15.134:0/2176657676 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3c040b3d20 0x7f3c040b6230 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3c20104710 tx=0x7f3c08005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:35.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.093+0000 7f3c1d600640 1 -- 172.21.15.134:0/2176657676 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3c100aaf90 con 0x7f3c201055e0 2024-07-26T11:12:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:35 smithi134 ceph-mon[31966]: pgmap v378: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T11:12:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:35 smithi193 ceph-mon[37473]: pgmap v378: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T11:12:35.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.370+0000 7f3c25a00640 1 -- 172.21.15.134:0/2176657676 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f3c2002e0c0 con 0x7f3c201055e0 2024-07-26T11:12:35.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.382+0000 7f3c1d600640 1 -- 172.21.15.134:0/2176657676 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v33) ==== 170+0+59 (secure 0 0 0) 0x7f3c100af4e0 con 0x7f3c201055e0 2024-07-26T11:12:35.384 INFO:teuthology.orchestra.run.smithi134.stdout:[client.0] 2024-07-26T11:12:35.384 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQCjhKNm4TNGFhAA+rl0Q3WSQ9maLlrJpqXdpA== 2024-07-26T11:12:35.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.385+0000 7f3c25a00640 1 -- 172.21.15.134:0/2176657676 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3c040b3d20 msgr2=0x7f3c040b6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:35.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.385+0000 7f3c25a00640 1 --2- 172.21.15.134:0/2176657676 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3c040b3d20 0x7f3c040b6230 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3c20104710 tx=0x7f3c08005f90 comp rx=0 tx=0).stop 2024-07-26T11:12:35.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.385+0000 7f3c25a00640 1 -- 172.21.15.134:0/2176657676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c201055e0 msgr2=0x7f3c2019bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:35.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.385+0000 7f3c25a00640 1 --2- 172.21.15.134:0/2176657676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c201055e0 0x7f3c2019bf70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000e9f0 tx=0x7f3c1000eeb0 comp rx=0 tx=0).stop 2024-07-26T11:12:35.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.385+0000 7f3c25a00640 1 -- 172.21.15.134:0/2176657676 shutdown_connections 2024-07-26T11:12:35.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.385+0000 7f3c25a00640 1 --2- 172.21.15.134:0/2176657676 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3c040b3d20 0x7f3c040b6230 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:35.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.385+0000 7f3c25a00640 1 --2- 172.21.15.134:0/2176657676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2010b870 0x7f3c201a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:35.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.385+0000 7f3c25a00640 1 --2- 172.21.15.134:0/2176657676 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c2010af20 0x7f3c2019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:35.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.385+0000 7f3c25a00640 1 --2- 172.21.15.134:0/2176657676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c201055e0 0x7f3c2019bf70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:35.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.385+0000 7f3c25a00640 1 -- 172.21.15.134:0/2176657676 >> 172.21.15.134:0/2176657676 conn(0x7f3c20101010 msgr2=0x7f3c20108470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:35.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.385+0000 7f3c25a00640 1 -- 172.21.15.134:0/2176657676 shutdown_connections 2024-07-26T11:12:35.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:35.385+0000 7f3c25a00640 1 -- 172.21.15.134:0/2176657676 wait complete. 2024-07-26T11:12:35.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:35 smithi189 ceph-mon[36790]: pgmap v378: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T11:12:36.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:36 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2176657676' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T11:12:36.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:36 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T11:12:36.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:36 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T11:12:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:36 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2176657676' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T11:12:36.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:36 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T11:12:36.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:36 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T11:12:36.281 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T11:12:36.281 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-07-26T11:12:36.281 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-07-26T11:12:36.404 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph config log 1 --format=json 2024-07-26T11:12:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:36 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2176657676' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T11:12:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:36 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-07-26T11:12:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:36 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-07-26T11:12:36.909 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:37 smithi193 ceph-mon[37473]: pgmap v379: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 596 B/s wr, 9 op/s 2024-07-26T11:12:37.287 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:37 smithi134 ceph-mon[31966]: pgmap v379: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 596 B/s wr, 9 op/s 2024-07-26T11:12:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:37 smithi189 ceph-mon[36790]: pgmap v379: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 596 B/s wr, 9 op/s 2024-07-26T11:12:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:39 smithi193 ceph-mon[37473]: pgmap v380: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 596 B/s wr, 11 op/s 2024-07-26T11:12:39.282 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:39 smithi134 ceph-mon[31966]: pgmap v380: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 596 B/s wr, 11 op/s 2024-07-26T11:12:39.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.279+0000 7fbe0be00640 1 -- 172.21.15.134:0/4011436855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c10af10 msgr2=0x7fbe0c10b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:39.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.279+0000 7fbe0be00640 1 --2- 172.21.15.134:0/4011436855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c10af10 0x7fbe0c10b320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4009900 tx=0x7fbdf402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:12:39.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.279+0000 7fbe0be00640 1 -- 172.21.15.134:0/4011436855 shutdown_connections 2024-07-26T11:12:39.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.279+0000 7fbe0be00640 1 --2- 172.21.15.134:0/4011436855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe0c10b860 0x7fbe0c112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:39.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.279+0000 7fbe0be00640 1 --2- 172.21.15.134:0/4011436855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c10af10 0x7fbe0c10b320 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:39.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.279+0000 7fbe0be00640 1 --2- 172.21.15.134:0/4011436855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c1055d0 0x7fbe0c1059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:39.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.279+0000 7fbe0be00640 1 -- 172.21.15.134:0/4011436855 >> 172.21.15.134:0/4011436855 conn(0x7fbe0c101040 msgr2=0x7fbe0c103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:39.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.280+0000 7fbe0be00640 1 -- 172.21.15.134:0/4011436855 shutdown_connections 2024-07-26T11:12:39.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.280+0000 7fbe0be00640 1 -- 172.21.15.134:0/4011436855 wait complete. 2024-07-26T11:12:39.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.281+0000 7fbe0be00640 1 Processor -- start 2024-07-26T11:12:39.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.281+0000 7fbe0be00640 1 -- start start 2024-07-26T11:12:39.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.281+0000 7fbe0be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c1055d0 0x7fbe0c19bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:39.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.281+0000 7fbe0be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c10af10 0x7fbe0c19c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:39.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.281+0000 7fbe0be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe0c10b860 0x7fbe0c1a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:39.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.281+0000 7fbe0be00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c115520 con 0x7fbe0c1055d0 2024-07-26T11:12:39.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.281+0000 7fbe0be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c1153a0 con 0x7fbe0c10b860 2024-07-26T11:12:39.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.281+0000 7fbe0be00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c1156a0 con 0x7fbe0c10af10 2024-07-26T11:12:39.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.282+0000 7fbe0aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c1055d0 0x7fbe0c19bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:39.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.282+0000 7fbe02000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c10af10 0x7fbe0c19c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:39.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.282+0000 7fbe0aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c1055d0 0x7fbe0c19bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57658/0 (socket says 172.21.15.134:57658) 2024-07-26T11:12:39.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.282+0000 7fbe0aa00640 1 -- 172.21.15.134:0/2633192830 learned_addr learned my addr 172.21.15.134:0/2633192830 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:12:39.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.282+0000 7fbe0b400640 1 --2- 172.21.15.134:0/2633192830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe0c10b860 0x7fbe0c1a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:39.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.282+0000 7fbe02000640 1 -- 172.21.15.134:0/2633192830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe0c10b860 msgr2=0x7fbe0c1a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:39.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.282+0000 7fbe02000640 1 --2- 172.21.15.134:0/2633192830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe0c10b860 0x7fbe0c1a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:39.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.282+0000 7fbe02000640 1 -- 172.21.15.134:0/2633192830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c1055d0 msgr2=0x7fbe0c19bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:39.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.282+0000 7fbe02000640 1 --2- 172.21.15.134:0/2633192830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c1055d0 0x7fbe0c19bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:39.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.282+0000 7fbe02000640 1 -- 172.21.15.134:0/2633192830 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0c1a1ea0 con 0x7fbe0c10af10 2024-07-26T11:12:39.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.283+0000 7fbe02000640 1 --2- 172.21.15.134:0/2633192830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c10af10 0x7fbe0c19c4d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4009230 tx=0x7fbdf4003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:39.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.283+0000 7fbe10a00640 1 -- 172.21.15.134:0/2633192830 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf403f070 con 0x7fbe0c10af10 2024-07-26T11:12:39.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.283+0000 7fbe0be00640 1 -- 172.21.15.134:0/2633192830 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe0c1a2130 con 0x7fbe0c10af10 2024-07-26T11:12:39.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.283+0000 7fbe10a00640 1 -- 172.21.15.134:0/2633192830 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbdf4031b50 con 0x7fbe0c10af10 2024-07-26T11:12:39.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.283+0000 7fbe10a00640 1 -- 172.21.15.134:0/2633192830 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4035030 con 0x7fbe0c10af10 2024-07-26T11:12:39.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.283+0000 7fbe0be00640 1 -- 172.21.15.134:0/2633192830 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe0c1a88f0 con 0x7fbe0c10af10 2024-07-26T11:12:39.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.284+0000 7fbe0be00640 1 -- 172.21.15.134:0/2633192830 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe0c1059e0 con 0x7fbe0c10af10 2024-07-26T11:12:39.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.287+0000 7fbe10a00640 1 -- 172.21.15.134:0/2633192830 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fbdf4042070 con 0x7fbe0c10af10 2024-07-26T11:12:39.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.289+0000 7fbe10a00640 1 --2- 172.21.15.134:0/2633192830 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbdf00b3cb0 0x7fbdf00b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:39.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.289+0000 7fbe10a00640 1 -- 172.21.15.134:0/2633192830 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fbdf4104c50 con 0x7fbe0c10af10 2024-07-26T11:12:39.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.289+0000 7fbe0aa00640 1 --2- 172.21.15.134:0/2633192830 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbdf00b3cb0 0x7fbdf00b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:39.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.290+0000 7fbe0aa00640 1 --2- 172.21.15.134:0/2633192830 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbdf00b3cb0 0x7fbdf00b61c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc009c90 tx=0x7fbdfc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:39.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.291+0000 7fbe10a00640 1 -- 172.21.15.134:0/2633192830 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbdf40cbc40 con 0x7fbe0c10af10 2024-07-26T11:12:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:39 smithi189 ceph-mon[36790]: pgmap v380: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 596 B/s wr, 11 op/s 2024-07-26T11:12:39.495 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.493+0000 7fbe0be00640 1 -- 172.21.15.134:0/2633192830 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7fbe0c10d640 con 0x7fbe0c10af10 2024-07-26T11:12:39.496 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.494+0000 7fbe10a00640 1 -- 172.21.15.134:0/2633192830 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v53) ==== 86+0+133 (secure 0 0 0) 0x7fbdf40cb540 con 0x7fbe0c10af10 2024-07-26T11:12:39.496 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:12:39.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.497+0000 7fbe0be00640 1 -- 172.21.15.134:0/2633192830 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbdf00b3cb0 msgr2=0x7fbdf00b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:39.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.497+0000 7fbe0be00640 1 --2- 172.21.15.134:0/2633192830 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbdf00b3cb0 0x7fbdf00b61c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc009c90 tx=0x7fbdfc005f90 comp rx=0 tx=0).stop 2024-07-26T11:12:39.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.497+0000 7fbe0be00640 1 -- 172.21.15.134:0/2633192830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c10af10 msgr2=0x7fbe0c19c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:39.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.497+0000 7fbe0be00640 1 --2- 172.21.15.134:0/2633192830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c10af10 0x7fbe0c19c4d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4009230 tx=0x7fbdf4003b90 comp rx=0 tx=0).stop 2024-07-26T11:12:39.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.497+0000 7fbe0be00640 1 -- 172.21.15.134:0/2633192830 shutdown_connections 2024-07-26T11:12:39.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.497+0000 7fbe0be00640 1 --2- 172.21.15.134:0/2633192830 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbdf00b3cb0 0x7fbdf00b61c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:39.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.497+0000 7fbe0be00640 1 --2- 172.21.15.134:0/2633192830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe0c10af10 0x7fbe0c19c4d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:39.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.497+0000 7fbe0be00640 1 --2- 172.21.15.134:0/2633192830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe0c10b860 0x7fbe0c1a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:39.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.497+0000 7fbe0be00640 1 --2- 172.21.15.134:0/2633192830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbe0c1055d0 0x7fbe0c19bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:39.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.497+0000 7fbe0be00640 1 -- 172.21.15.134:0/2633192830 >> 172.21.15.134:0/2633192830 conn(0x7fbe0c101040 msgr2=0x7fbe0c103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:39.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.497+0000 7fbe0be00640 1 -- 172.21.15.134:0/2633192830 shutdown_connections 2024-07-26T11:12:39.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:39.497+0000 7fbe0be00640 1 -- 172.21.15.134:0/2633192830 wait complete. 2024-07-26T11:12:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:40 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2633192830' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T11:12:40.363 INFO:teuthology.orchestra.run.smithi134.stdout:[{"version":53,"timestamp":"2024-07-26T11:10:02.826219+0000","name":"","changes":[{"name":"mds.all/mds_join_fs","new_value":"all"}]}] 2024-07-26T11:12:40.363 INFO:tasks.ceph_manager:config epoch is 53 2024-07-26T11:12:40.363 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-07-26T11:12:40.363 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-07-26T11:12:40.364 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph mgr dump --format=json 2024-07-26T11:12:40.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:40 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2633192830' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T11:12:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:40 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2633192830' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-07-26T11:12:40.811 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:41.351 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:41 smithi134 ceph-mon[31966]: pgmap v381: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:12:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:41 smithi189 ceph-mon[36790]: pgmap v381: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:12:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:41 smithi193 ceph-mon[37473]: pgmap v381: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:12:43.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.150+0000 7f9ff8c00640 1 -- 172.21.15.134:0/699825677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ff4108d00 msgr2=0x7f9ff4109110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:43.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.150+0000 7f9ff8c00640 1 --2- 172.21.15.134:0/699825677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ff4108d00 0x7f9ff4109110 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9fe0009900 tx=0x7f9fe002a410 comp rx=0 tx=0).stop 2024-07-26T11:12:43.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.150+0000 7f9ff8c00640 1 -- 172.21.15.134:0/699825677 shutdown_connections 2024-07-26T11:12:43.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.150+0000 7f9ff8c00640 1 --2- 172.21.15.134:0/699825677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ff4109650 0x7f9ff410fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:43.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.150+0000 7f9ff8c00640 1 --2- 172.21.15.134:0/699825677 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ff4108d00 0x7f9ff4109110 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:43.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.150+0000 7f9ff8c00640 1 --2- 172.21.15.134:0/699825677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ff41017c0 0x7f9ff4101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:43.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.150+0000 7f9ff8c00640 1 -- 172.21.15.134:0/699825677 >> 172.21.15.134:0/699825677 conn(0x7f9ff40774d0 msgr2=0x7f9ff4077910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:43.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.150+0000 7f9ff8c00640 1 -- 172.21.15.134:0/699825677 shutdown_connections 2024-07-26T11:12:43.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.150+0000 7f9ff8c00640 1 -- 172.21.15.134:0/699825677 wait complete. 2024-07-26T11:12:43.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.151+0000 7f9ff8c00640 1 Processor -- start 2024-07-26T11:12:43.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.151+0000 7f9ff8c00640 1 -- start start 2024-07-26T11:12:43.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.152+0000 7f9ff8c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ff41017c0 0x7f9ff4199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:43.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.152+0000 7f9ff8c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ff4108d00 0x7f9ff419a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:43.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.152+0000 7f9ff8c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ff4109650 0x7f9ff419f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:43.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.152+0000 7f9ff8c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff4113310 con 0x7f9ff41017c0 2024-07-26T11:12:43.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.152+0000 7f9ff8c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff4113190 con 0x7f9ff4108d00 2024-07-26T11:12:43.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.152+0000 7f9ff8c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9ff4113490 con 0x7f9ff4109650 2024-07-26T11:12:43.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.152+0000 7f9ff3400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ff41017c0 0x7f9ff4199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:43.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.152+0000 7f9ff3400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ff41017c0 0x7f9ff4199d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:51610/0 (socket says 172.21.15.134:51610) 2024-07-26T11:12:43.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.152+0000 7f9ff3400640 1 -- 172.21.15.134:0/1275366992 learned_addr learned my addr 172.21.15.134:0/1275366992 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:12:43.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.152+0000 7f9ff2a00640 1 --2- 172.21.15.134:0/1275366992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ff4108d00 0x7f9ff419a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:43.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.152+0000 7f9ff3e00640 1 --2- 172.21.15.134:0/1275366992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ff4109650 0x7f9ff419f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:43.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.153+0000 7f9ff3400640 1 -- 172.21.15.134:0/1275366992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ff4109650 msgr2=0x7f9ff419f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:43.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.153+0000 7f9ff3400640 1 --2- 172.21.15.134:0/1275366992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ff4109650 0x7f9ff419f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:43.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.153+0000 7f9ff3400640 1 -- 172.21.15.134:0/1275366992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ff4108d00 msgr2=0x7f9ff419a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:43.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.153+0000 7f9ff3400640 1 --2- 172.21.15.134:0/1275366992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ff4108d00 0x7f9ff419a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:43.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.153+0000 7f9ff3400640 1 -- 172.21.15.134:0/1275366992 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ff419fcf0 con 0x7f9ff41017c0 2024-07-26T11:12:43.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.153+0000 7f9ff2a00640 1 --2- 172.21.15.134:0/1275366992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ff4108d00 0x7f9ff419a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:12:43.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.153+0000 7f9ff3e00640 1 --2- 172.21.15.134:0/1275366992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ff4109650 0x7f9ff419f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:12:43.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.153+0000 7f9ff3400640 1 --2- 172.21.15.134:0/1275366992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ff41017c0 0x7f9ff4199d80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9fe400e9f0 tx=0x7f9fe400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:43.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.153+0000 7f9febe00640 1 -- 172.21.15.134:0/1275366992 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe400cd80 con 0x7f9ff41017c0 2024-07-26T11:12:43.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.153+0000 7f9febe00640 1 -- 172.21.15.134:0/1275366992 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9fe40108e0 con 0x7f9ff41017c0 2024-07-26T11:12:43.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.154+0000 7f9ff8c00640 1 -- 172.21.15.134:0/1275366992 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ff419ffe0 con 0x7f9ff41017c0 2024-07-26T11:12:43.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.154+0000 7f9febe00640 1 -- 172.21.15.134:0/1275366992 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4018980 con 0x7f9ff41017c0 2024-07-26T11:12:43.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.154+0000 7f9ff8c00640 1 -- 172.21.15.134:0/1275366992 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ff4102640 con 0x7f9ff41017c0 2024-07-26T11:12:43.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.155+0000 7f9ff8c00640 1 -- 172.21.15.134:0/1275366992 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ff4101bd0 con 0x7f9ff41017c0 2024-07-26T11:12:43.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.157+0000 7f9febe00640 1 -- 172.21.15.134:0/1275366992 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f9fe4010430 con 0x7f9ff41017c0 2024-07-26T11:12:43.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.159+0000 7f9febe00640 1 --2- 172.21.15.134:0/1275366992 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9fd80b3cb0 0x7f9fd80b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:43.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.159+0000 7f9febe00640 1 -- 172.21.15.134:0/1275366992 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f9fe40e4ac0 con 0x7f9ff41017c0 2024-07-26T11:12:43.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.159+0000 7f9ff2a00640 1 --2- 172.21.15.134:0/1275366992 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9fd80b3cb0 0x7f9fd80b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:43.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.160+0000 7f9ff2a00640 1 --2- 172.21.15.134:0/1275366992 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9fd80b3cb0 0x7f9fd80b61c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9fe002aa00 tx=0x7f9fe0005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:43.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.161+0000 7f9febe00640 1 -- 172.21.15.134:0/1275366992 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9fe40ab460 con 0x7f9ff41017c0 2024-07-26T11:12:43.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.408+0000 7f9ff8c00640 1 -- 172.21.15.134:0/1275366992 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f9ff402e0c0 con 0x7f9ff41017c0 2024-07-26T11:12:43.410 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:43 smithi134 ceph-mon[31966]: pgmap v382: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:12:43.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.415+0000 7f9febe00640 1 -- 172.21.15.134:0/1275366992 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v21) ==== 74+0+291044 (secure 0 0 0) 0x7f9fe40af9b0 con 0x7f9ff41017c0 2024-07-26T11:12:43.424 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:12:43.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.423+0000 7f9ff8c00640 1 -- 172.21.15.134:0/1275366992 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9fd80b3cb0 msgr2=0x7f9fd80b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:43.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.423+0000 7f9ff8c00640 1 --2- 172.21.15.134:0/1275366992 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9fd80b3cb0 0x7f9fd80b61c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9fe002aa00 tx=0x7f9fe0005a20 comp rx=0 tx=0).stop 2024-07-26T11:12:43.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.423+0000 7f9ff8c00640 1 -- 172.21.15.134:0/1275366992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ff41017c0 msgr2=0x7f9ff4199d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:43.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.423+0000 7f9ff8c00640 1 --2- 172.21.15.134:0/1275366992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ff41017c0 0x7f9ff4199d80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9fe400e9f0 tx=0x7f9fe400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:12:43.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.423+0000 7f9ff8c00640 1 -- 172.21.15.134:0/1275366992 shutdown_connections 2024-07-26T11:12:43.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.423+0000 7f9ff8c00640 1 --2- 172.21.15.134:0/1275366992 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9fd80b3cb0 0x7f9fd80b61c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:43.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.423+0000 7f9ff8c00640 1 --2- 172.21.15.134:0/1275366992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ff4109650 0x7f9ff419f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:43.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.423+0000 7f9ff8c00640 1 --2- 172.21.15.134:0/1275366992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ff4108d00 0x7f9ff419a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:43.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.423+0000 7f9ff8c00640 1 --2- 172.21.15.134:0/1275366992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ff41017c0 0x7f9ff4199d80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:43.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.423+0000 7f9ff8c00640 1 -- 172.21.15.134:0/1275366992 >> 172.21.15.134:0/1275366992 conn(0x7f9ff40774d0 msgr2=0x7f9ff4106e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:43.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.423+0000 7f9ff8c00640 1 -- 172.21.15.134:0/1275366992 shutdown_connections 2024-07-26T11:12:43.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:43.423+0000 7f9ff8c00640 1 -- 172.21.15.134:0/1275366992 wait complete. 2024-07-26T11:12:43.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:43 smithi189 ceph-mon[36790]: pgmap v382: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:12:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:43 smithi193 ceph-mon[37473]: pgmap v382: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:12:44.235 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":21,"flags":0,"active_gid":14211,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":2671962751},{"type":"v1","addr":"172.21.15.134:6801","nonce":2671962751}]},"active_addr":"172.21.15.134:6801/2671962751","active_change":"2024-07-26T11:02:33.712098+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24148,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24152,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.134:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":985318679}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":1168597813}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":3459277639}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":1524383568}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":2851199413}]}]} 2024-07-26T11:12:44.241 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-07-26T11:12:44.241 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-07-26T11:12:44.241 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T11:12:44.318 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:44 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1275366992' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T11:12:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:44 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1275366992' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T11:12:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:44 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1275366992' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-07-26T11:12:44.711 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:45 smithi134 ceph-mon[31966]: pgmap v383: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:12:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:45 smithi189 ceph-mon[36790]: pgmap v383: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:12:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:45 smithi193 ceph-mon[37473]: pgmap v383: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:12:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:46.999+0000 7f2bbae00640 1 -- 172.21.15.134:0/369898245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bb4075180 msgr2=0x7f2bb4075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:46.999+0000 7f2bbae00640 1 --2- 172.21.15.134:0/369898245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bb4075180 0x7f2bb4075590 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2ba8009860 tx=0x7f2ba802a340 comp rx=0 tx=0).stop 2024-07-26T11:12:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:46.999+0000 7f2bbae00640 1 -- 172.21.15.134:0/369898245 shutdown_connections 2024-07-26T11:12:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:46.999+0000 7f2bbae00640 1 --2- 172.21.15.134:0/369898245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb410f9a0 0x7f2bb4111dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:46.999+0000 7f2bbae00640 1 --2- 172.21.15.134:0/369898245 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2bb4075ad0 0x7f2bb410f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:46.999+0000 7f2bbae00640 1 --2- 172.21.15.134:0/369898245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bb4075180 0x7f2bb4075590 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:46.999+0000 7f2bbae00640 1 -- 172.21.15.134:0/369898245 >> 172.21.15.134:0/369898245 conn(0x7f2bb40fee30 msgr2=0x7f2bb4101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:46.999+0000 7f2bbae00640 1 -- 172.21.15.134:0/369898245 shutdown_connections 2024-07-26T11:12:47.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:46.999+0000 7f2bbae00640 1 -- 172.21.15.134:0/369898245 wait complete. 2024-07-26T11:12:47.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.001+0000 7f2bbae00640 1 Processor -- start 2024-07-26T11:12:47.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.002+0000 7f2bbae00640 1 -- start start 2024-07-26T11:12:47.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.002+0000 7f2bbae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb4075180 0x7f2bb419bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:47.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.002+0000 7f2bbae00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2bb4075ad0 0x7f2bb419c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:47.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.003+0000 7f2bbae00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bb410f9a0 0x7f2bb41a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:47.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.003+0000 7f2bbae00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb4115500 con 0x7f2bb4075ad0 2024-07-26T11:12:47.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.003+0000 7f2bbae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb4115380 con 0x7f2bb4075180 2024-07-26T11:12:47.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.003+0000 7f2bbae00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb4115680 con 0x7f2bb410f9a0 2024-07-26T11:12:47.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.003+0000 7f2bb9a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb4075180 0x7f2bb419bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:47.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.003+0000 7f2bb9000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2bb4075ad0 0x7f2bb419c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:47.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.003+0000 7f2bb9a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb4075180 0x7f2bb419bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:50788/0 (socket says 172.21.15.134:50788) 2024-07-26T11:12:47.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.003+0000 7f2bb9a00640 1 -- 172.21.15.134:0/400045398 learned_addr learned my addr 172.21.15.134:0/400045398 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:12:47.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.003+0000 7f2bba400640 1 --2- 172.21.15.134:0/400045398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bb410f9a0 0x7f2bb41a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:47.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.003+0000 7f2bb9a00640 1 -- 172.21.15.134:0/400045398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bb410f9a0 msgr2=0x7f2bb41a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:47.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.003+0000 7f2bb9a00640 1 --2- 172.21.15.134:0/400045398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bb410f9a0 0x7f2bb41a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:47.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.003+0000 7f2bb9a00640 1 -- 172.21.15.134:0/400045398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2bb4075ad0 msgr2=0x7f2bb419c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:47.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.003+0000 7f2bb9a00640 1 --2- 172.21.15.134:0/400045398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2bb4075ad0 0x7f2bb419c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:47.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.003+0000 7f2bb9a00640 1 -- 172.21.15.134:0/400045398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb41a1e40 con 0x7f2bb4075180 2024-07-26T11:12:47.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.004+0000 7f2bb9a00640 1 --2- 172.21.15.134:0/400045398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb4075180 0x7f2bb419bed0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80303f0 tx=0x7f2ba8032da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:47.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.004+0000 7f2bb2000640 1 -- 172.21.15.134:0/400045398 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8040020 con 0x7f2bb4075180 2024-07-26T11:12:47.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.004+0000 7f2bbae00640 1 -- 172.21.15.134:0/400045398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb41a20d0 con 0x7f2bb4075180 2024-07-26T11:12:47.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.004+0000 7f2bb2000640 1 -- 172.21.15.134:0/400045398 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ba8030dd0 con 0x7f2bb4075180 2024-07-26T11:12:47.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.004+0000 7f2bb2000640 1 -- 172.21.15.134:0/400045398 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba8037040 con 0x7f2bb4075180 2024-07-26T11:12:47.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.004+0000 7f2bbae00640 1 -- 172.21.15.134:0/400045398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb41a8890 con 0x7f2bb4075180 2024-07-26T11:12:47.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.005+0000 7f2bbae00640 1 -- 172.21.15.134:0/400045398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bb4075590 con 0x7f2bb4075180 2024-07-26T11:12:47.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.008+0000 7f2bb2000640 1 -- 172.21.15.134:0/400045398 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f2ba8043020 con 0x7f2bb4075180 2024-07-26T11:12:47.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.009+0000 7f2bb2000640 1 --2- 172.21.15.134:0/400045398 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b9c0b3cb0 0x7f2b9c0b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:47.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.010+0000 7f2bb2000640 1 -- 172.21.15.134:0/400045398 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2ba8104610 con 0x7f2bb4075180 2024-07-26T11:12:47.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.010+0000 7f2bb9000640 1 --2- 172.21.15.134:0/400045398 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b9c0b3cb0 0x7f2b9c0b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:47.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.011+0000 7f2bb9000640 1 --2- 172.21.15.134:0/400045398 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b9c0b3cb0 0x7f2b9c0b61c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4111ef0 tx=0x7f2ba0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:47.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.011+0000 7f2bb2000640 1 -- 172.21.15.134:0/400045398 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2ba804a020 con 0x7f2bb4075180 2024-07-26T11:12:47.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.212+0000 7f2bbae00640 1 -- 172.21.15.134:0/400045398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2bb41a8a90 con 0x7f2bb4075180 2024-07-26T11:12:47.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.214+0000 7f2bb2000640 1 -- 172.21.15.134:0/400045398 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25185 (secure 0 0 0) 0x7f2ba80328c0 con 0x7f2bb4075180 2024-07-26T11:12:47.217 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:12:47.217 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":90,"fsid":"105fbe5c-4b3e-11ef-bcb7-c7b262605968","created":"2024-07-26T11:00:05.955103+0000","modified":"2024-07-26T11:11:35.149490+0000","last_up_change":"2024-07-26T11:09:11.485512+0000","last_in_change":"2024-07-26T11:08:52.341356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T11:05:01.849170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T11:10:16.055725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T11:10:21.216611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T11:10:33.983067+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8abb1365-fc3e-4617-a9ac-76bece8cd7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6803","nonce":4183273954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6805","nonce":4183273954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6809","nonce":4183273954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6807","nonce":4183273954}]},"public_addr":"172.21.15.134:6803/4183273954","cluster_addr":"172.21.15.134:6805/4183273954","heartbeat_back_addr":"172.21.15.134:6809/4183273954","heartbeat_front_addr":"172.21.15.134:6807/4183273954","state":["exists","up"]},{"osd":1,"uuid":"6c5a7838-821e-44e1-97e4-655b9bfa9f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6801","nonce":35148049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6803","nonce":35148049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6807","nonce":35148049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6805","nonce":35148049}]},"public_addr":"172.21.15.189:6801/35148049","cluster_addr":"172.21.15.189:6803/35148049","heartbeat_back_addr":"172.21.15.189:6807/35148049","heartbeat_front_addr":"172.21.15.189:6805/35148049","state":["exists","up"]},{"osd":2,"uuid":"8bc42f7f-39fa-4d3b-bc17-21c68f9231fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6801","nonce":897516976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6803","nonce":897516976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6807","nonce":897516976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6805","nonce":897516976}]},"public_addr":"172.21.15.193:6801/897516976","cluster_addr":"172.21.15.193:6803/897516976","heartbeat_back_addr":"172.21.15.193:6807/897516976","heartbeat_front_addr":"172.21.15.193:6805/897516976","state":["exists","up"]},{"osd":3,"uuid":"0473d055-0b79-434f-9658-28269d2b1fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6811","nonce":3457190917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6813","nonce":3457190917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6817","nonce":3457190917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6815","nonce":3457190917}]},"public_addr":"172.21.15.134:6811/3457190917","cluster_addr":"172.21.15.134:6813/3457190917","heartbeat_back_addr":"172.21.15.134:6817/3457190917","heartbeat_front_addr":"172.21.15.134:6815/3457190917","state":["exists","up"]},{"osd":4,"uuid":"1d70eaae-56cc-4cdc-8f83-b88d2121f0ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6809","nonce":1786881347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6811","nonce":1786881347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6815","nonce":1786881347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6813","nonce":1786881347}]},"public_addr":"172.21.15.189:6809/1786881347","cluster_addr":"172.21.15.189:6811/1786881347","heartbeat_back_addr":"172.21.15.189:6815/1786881347","heartbeat_front_addr":"172.21.15.189:6813/1786881347","state":["exists","up"]},{"osd":5,"uuid":"a9b0b4cf-1608-45ed-b64e-afcf545d97fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6809","nonce":1092388797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6811","nonce":1092388797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6815","nonce":1092388797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6813","nonce":1092388797}]},"public_addr":"172.21.15.193:6809/1092388797","cluster_addr":"172.21.15.193:6811/1092388797","heartbeat_back_addr":"172.21.15.193:6815/1092388797","heartbeat_front_addr":"172.21.15.193:6813/1092388797","state":["exists","up"]},{"osd":6,"uuid":"0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6819","nonce":2471505438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6821","nonce":2471505438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6825","nonce":2471505438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6823","nonce":2471505438}]},"public_addr":"172.21.15.134:6819/2471505438","cluster_addr":"172.21.15.134:6821/2471505438","heartbeat_back_addr":"172.21.15.134:6825/2471505438","heartbeat_front_addr":"172.21.15.134:6823/2471505438","state":["exists","up"]},{"osd":7,"uuid":"394764a7-1976-4223-a3c0-5979d941a838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6817","nonce":1382694911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6819","nonce":1382694911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6823","nonce":1382694911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6821","nonce":1382694911}]},"public_addr":"172.21.15.189:6817/1382694911","cluster_addr":"172.21.15.189:6819/1382694911","heartbeat_back_addr":"172.21.15.189:6823/1382694911","heartbeat_front_addr":"172.21.15.189:6821/1382694911","state":["exists","up"]},{"osd":8,"uuid":"1c218082-0b4c-41c8-92b9-453ae8d15a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6817","nonce":896348138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6819","nonce":896348138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6823","nonce":896348138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6821","nonce":896348138}]},"public_addr":"172.21.15.193:6817/896348138","cluster_addr":"172.21.15.193:6819/896348138","heartbeat_back_addr":"172.21.15.193:6823/896348138","heartbeat_front_addr":"172.21.15.193:6821/896348138","state":["exists","up"]},{"osd":9,"uuid":"48f39082-4fa9-4443-b5b7-b141c10d9e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6827","nonce":511237025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6829","nonce":511237025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6833","nonce":511237025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6831","nonce":511237025}]},"public_addr":"172.21.15.134:6827/511237025","cluster_addr":"172.21.15.134:6829/511237025","heartbeat_back_addr":"172.21.15.134:6833/511237025","heartbeat_front_addr":"172.21.15.134:6831/511237025","state":["exists","up"]},{"osd":10,"uuid":"95157766-1856-42f7-aea1-9cf83b77c8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6825","nonce":431740858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6827","nonce":431740858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6831","nonce":431740858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6829","nonce":431740858}]},"public_addr":"172.21.15.189:6825/431740858","cluster_addr":"172.21.15.189:6827/431740858","heartbeat_back_addr":"172.21.15.189:6831/431740858","heartbeat_front_addr":"172.21.15.189:6829/431740858","state":["exists","up"]},{"osd":11,"uuid":"244fadc7-f079-4736-8e7e-c0ce5ae6f43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6825","nonce":1594590579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6827","nonce":1594590579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6831","nonce":1594590579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6829","nonce":1594590579}]},"public_addr":"172.21.15.193:6825/1594590579","cluster_addr":"172.21.15.193:6827/1594590579","heartbeat_back_addr":"172.21.15.193:6831/1594590579","heartbeat_front_addr":"172.21.15.193:6829/1594590579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:11.712203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:35.845013+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:58.165609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:31.929746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:56.941555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:20.688071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:54.681514+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:20.789384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:44.568246+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:18.644310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:45.365351+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:09:09.663805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3790862248":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6800/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:0/2906927791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3068153391":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3056560791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6801/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/1310359922":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/2318666906":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/2901579347":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3404218719":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3598071145":"2024-07-27T11:02:33.711793+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:12:47.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.217+0000 7f2bbae00640 1 -- 172.21.15.134:0/400045398 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b9c0b3cb0 msgr2=0x7f2b9c0b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:47.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.217+0000 7f2bbae00640 1 --2- 172.21.15.134:0/400045398 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b9c0b3cb0 0x7f2b9c0b61c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4111ef0 tx=0x7f2ba0005f90 comp rx=0 tx=0).stop 2024-07-26T11:12:47.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.217+0000 7f2bbae00640 1 -- 172.21.15.134:0/400045398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb4075180 msgr2=0x7f2bb419bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:47.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.217+0000 7f2bbae00640 1 --2- 172.21.15.134:0/400045398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb4075180 0x7f2bb419bed0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80303f0 tx=0x7f2ba8032da0 comp rx=0 tx=0).stop 2024-07-26T11:12:47.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.217+0000 7f2bbae00640 1 -- 172.21.15.134:0/400045398 shutdown_connections 2024-07-26T11:12:47.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.217+0000 7f2bbae00640 1 --2- 172.21.15.134:0/400045398 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b9c0b3cb0 0x7f2b9c0b61c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:47.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.217+0000 7f2bbae00640 1 --2- 172.21.15.134:0/400045398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2bb410f9a0 0x7f2bb41a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:47.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.217+0000 7f2bbae00640 1 --2- 172.21.15.134:0/400045398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2bb4075ad0 0x7f2bb419c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:47.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.217+0000 7f2bbae00640 1 --2- 172.21.15.134:0/400045398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bb4075180 0x7f2bb419bed0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:47.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.217+0000 7f2bbae00640 1 -- 172.21.15.134:0/400045398 >> 172.21.15.134:0/400045398 conn(0x7f2bb40fee30 msgr2=0x7f2bb4109540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:47.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.218+0000 7f2bbae00640 1 -- 172.21.15.134:0/400045398 shutdown_connections 2024-07-26T11:12:47.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:47.218+0000 7f2bbae00640 1 -- 172.21.15.134:0/400045398 wait complete. 2024-07-26T11:12:47.293 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:47 smithi134 ceph-mon[31966]: pgmap v384: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:12:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:47 smithi189 ceph-mon[36790]: pgmap v384: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:12:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:47 smithi193 ceph-mon[37473]: pgmap v384: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:12:48.191 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-07-26T11:12:48.192 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T11:12:48.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:48 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/400045398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:12:48.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:48 smithi134 ceph-mon[31966]: pgmap v385: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:12:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:48 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/400045398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:12:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:48 smithi189 ceph-mon[36790]: pgmap v385: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:12:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:48 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/400045398' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:12:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:48 smithi193 ceph-mon[37473]: pgmap v385: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:12:48.653 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:51.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.081+0000 7f66b2800640 1 -- 172.21.15.134:0/2338008479 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ac1099f0 msgr2=0x7f66ac109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:51.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.081+0000 7f66b2800640 1 --2- 172.21.15.134:0/2338008479 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ac1099f0 0x7f66ac109e00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f66a0009900 tx=0x7f66a002a410 comp rx=0 tx=0).stop 2024-07-26T11:12:51.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.081+0000 7f66b2800640 1 -- 172.21.15.134:0/2338008479 shutdown_connections 2024-07-26T11:12:51.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.081+0000 7f66b2800640 1 --2- 172.21.15.134:0/2338008479 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ac10fc80 0x7f66ac1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:51.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.081+0000 7f66b2800640 1 --2- 172.21.15.134:0/2338008479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ac10f330 0x7f66ac10f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:51.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.081+0000 7f66b2800640 1 --2- 172.21.15.134:0/2338008479 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ac1099f0 0x7f66ac109e00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:51.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.081+0000 7f66b2800640 1 -- 172.21.15.134:0/2338008479 >> 172.21.15.134:0/2338008479 conn(0x7f66ac078ff0 msgr2=0x7f66ac107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:51.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.081+0000 7f66b2800640 1 -- 172.21.15.134:0/2338008479 shutdown_connections 2024-07-26T11:12:51.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.081+0000 7f66b2800640 1 -- 172.21.15.134:0/2338008479 wait complete. 2024-07-26T11:12:51.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.082+0000 7f66b2800640 1 Processor -- start 2024-07-26T11:12:51.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.082+0000 7f66b2800640 1 -- start start 2024-07-26T11:12:51.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.083+0000 7f66b2800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ac1099f0 0x7f66ac1a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:51.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.083+0000 7f66b2800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ac10f330 0x7f66ac1a0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:51.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.083+0000 7f66b2800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ac10fc80 0x7f66ac1a5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:51.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.083+0000 7f66b2800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac119940 con 0x7f66ac1099f0 2024-07-26T11:12:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.083+0000 7f66b2800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac1197c0 con 0x7f66ac10f330 2024-07-26T11:12:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.083+0000 7f66b2800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f66ac119ac0 con 0x7f66ac10fc80 2024-07-26T11:12:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.083+0000 7f66b1400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ac1099f0 0x7f66ac1a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.083+0000 7f66b1400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ac1099f0 0x7f66ac1a03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58398/0 (socket says 172.21.15.134:58398) 2024-07-26T11:12:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.083+0000 7f66b1400640 1 -- 172.21.15.134:0/3774967088 learned_addr learned my addr 172.21.15.134:0/3774967088 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:12:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.083+0000 7f66b1e00640 1 --2- 172.21.15.134:0/3774967088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ac10fc80 0x7f66ac1a5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.083+0000 7f66b0a00640 1 --2- 172.21.15.134:0/3774967088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ac10f330 0x7f66ac1a0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.084+0000 7f66b1e00640 1 -- 172.21.15.134:0/3774967088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ac10f330 msgr2=0x7f66ac1a0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.084+0000 7f66b1e00640 1 --2- 172.21.15.134:0/3774967088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ac10f330 0x7f66ac1a0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.084+0000 7f66b1e00640 1 -- 172.21.15.134:0/3774967088 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ac1099f0 msgr2=0x7f66ac1a03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.084+0000 7f66b1e00640 1 --2- 172.21.15.134:0/3774967088 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ac1099f0 0x7f66ac1a03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.084+0000 7f66b1e00640 1 -- 172.21.15.134:0/3774967088 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66ac1a6360 con 0x7f66ac10fc80 2024-07-26T11:12:51.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.084+0000 7f66b1e00640 1 --2- 172.21.15.134:0/3774967088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ac10fc80 0x7f66ac1a5c30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009fd0 tx=0x7f66a400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:51.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.084+0000 7f66aa000640 1 -- 172.21.15.134:0/3774967088 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a400ce20 con 0x7f66ac10fc80 2024-07-26T11:12:51.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.084+0000 7f66b2800640 1 -- 172.21.15.134:0/3774967088 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66ac1a6650 con 0x7f66ac10fc80 2024-07-26T11:12:51.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.084+0000 7f66aa000640 1 -- 172.21.15.134:0/3774967088 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66a4005020 con 0x7f66ac10fc80 2024-07-26T11:12:51.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.085+0000 7f66aa000640 1 -- 172.21.15.134:0/3774967088 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4010690 con 0x7f66ac10fc80 2024-07-26T11:12:51.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.085+0000 7f66b2800640 1 -- 172.21.15.134:0/3774967088 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66ac1acdb0 con 0x7f66ac10fc80 2024-07-26T11:12:51.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.086+0000 7f66b2800640 1 -- 172.21.15.134:0/3774967088 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66ac109e00 con 0x7f66ac10fc80 2024-07-26T11:12:51.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.088+0000 7f66aa000640 1 -- 172.21.15.134:0/3774967088 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f66a40054d0 con 0x7f66ac10fc80 2024-07-26T11:12:51.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.090+0000 7f66aa000640 1 --2- 172.21.15.134:0/3774967088 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f66940b3cb0 0x7f66940b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:12:51.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.090+0000 7f66aa000640 1 -- 172.21.15.134:0/3774967088 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f66a40e4ee0 con 0x7f66ac10fc80 2024-07-26T11:12:51.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.090+0000 7f66b1400640 1 --2- 172.21.15.134:0/3774967088 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f66940b3cb0 0x7f66940b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:12:51.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.091+0000 7f66b1400640 1 --2- 172.21.15.134:0/3774967088 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f66940b3cb0 0x7f66940b61c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f66a0031440 tx=0x7f66a00058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:12:51.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.092+0000 7f66aa000640 1 -- 172.21.15.134:0/3774967088 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f66a40abed0 con 0x7f66ac10fc80 2024-07-26T11:12:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:51 smithi134 ceph-mon[31966]: pgmap v386: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:12:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:51 smithi193 ceph-mon[37473]: pgmap v386: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:12:51.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.295+0000 7f66b2800640 1 -- 172.21.15.134:0/3774967088 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f66ac1a1520 con 0x7f66ac10fc80 2024-07-26T11:12:51.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.297+0000 7f66aa000640 1 -- 172.21.15.134:0/3774967088 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25185 (secure 0 0 0) 0x7f66ac1a1520 con 0x7f66ac10fc80 2024-07-26T11:12:51.300 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:12:51.300 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":90,"fsid":"105fbe5c-4b3e-11ef-bcb7-c7b262605968","created":"2024-07-26T11:00:05.955103+0000","modified":"2024-07-26T11:11:35.149490+0000","last_up_change":"2024-07-26T11:09:11.485512+0000","last_in_change":"2024-07-26T11:08:52.341356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T11:05:01.849170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T11:10:16.055725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T11:10:21.216611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T11:10:33.983067+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8abb1365-fc3e-4617-a9ac-76bece8cd7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6803","nonce":4183273954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6805","nonce":4183273954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6809","nonce":4183273954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6807","nonce":4183273954}]},"public_addr":"172.21.15.134:6803/4183273954","cluster_addr":"172.21.15.134:6805/4183273954","heartbeat_back_addr":"172.21.15.134:6809/4183273954","heartbeat_front_addr":"172.21.15.134:6807/4183273954","state":["exists","up"]},{"osd":1,"uuid":"6c5a7838-821e-44e1-97e4-655b9bfa9f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6801","nonce":35148049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6803","nonce":35148049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6807","nonce":35148049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6805","nonce":35148049}]},"public_addr":"172.21.15.189:6801/35148049","cluster_addr":"172.21.15.189:6803/35148049","heartbeat_back_addr":"172.21.15.189:6807/35148049","heartbeat_front_addr":"172.21.15.189:6805/35148049","state":["exists","up"]},{"osd":2,"uuid":"8bc42f7f-39fa-4d3b-bc17-21c68f9231fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6801","nonce":897516976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6803","nonce":897516976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6807","nonce":897516976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6805","nonce":897516976}]},"public_addr":"172.21.15.193:6801/897516976","cluster_addr":"172.21.15.193:6803/897516976","heartbeat_back_addr":"172.21.15.193:6807/897516976","heartbeat_front_addr":"172.21.15.193:6805/897516976","state":["exists","up"]},{"osd":3,"uuid":"0473d055-0b79-434f-9658-28269d2b1fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6811","nonce":3457190917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6813","nonce":3457190917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6817","nonce":3457190917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6815","nonce":3457190917}]},"public_addr":"172.21.15.134:6811/3457190917","cluster_addr":"172.21.15.134:6813/3457190917","heartbeat_back_addr":"172.21.15.134:6817/3457190917","heartbeat_front_addr":"172.21.15.134:6815/3457190917","state":["exists","up"]},{"osd":4,"uuid":"1d70eaae-56cc-4cdc-8f83-b88d2121f0ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6809","nonce":1786881347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6811","nonce":1786881347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6815","nonce":1786881347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6813","nonce":1786881347}]},"public_addr":"172.21.15.189:6809/1786881347","cluster_addr":"172.21.15.189:6811/1786881347","heartbeat_back_addr":"172.21.15.189:6815/1786881347","heartbeat_front_addr":"172.21.15.189:6813/1786881347","state":["exists","up"]},{"osd":5,"uuid":"a9b0b4cf-1608-45ed-b64e-afcf545d97fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6809","nonce":1092388797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6811","nonce":1092388797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6815","nonce":1092388797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6813","nonce":1092388797}]},"public_addr":"172.21.15.193:6809/1092388797","cluster_addr":"172.21.15.193:6811/1092388797","heartbeat_back_addr":"172.21.15.193:6815/1092388797","heartbeat_front_addr":"172.21.15.193:6813/1092388797","state":["exists","up"]},{"osd":6,"uuid":"0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6819","nonce":2471505438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6821","nonce":2471505438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6825","nonce":2471505438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6823","nonce":2471505438}]},"public_addr":"172.21.15.134:6819/2471505438","cluster_addr":"172.21.15.134:6821/2471505438","heartbeat_back_addr":"172.21.15.134:6825/2471505438","heartbeat_front_addr":"172.21.15.134:6823/2471505438","state":["exists","up"]},{"osd":7,"uuid":"394764a7-1976-4223-a3c0-5979d941a838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6817","nonce":1382694911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6819","nonce":1382694911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6823","nonce":1382694911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6821","nonce":1382694911}]},"public_addr":"172.21.15.189:6817/1382694911","cluster_addr":"172.21.15.189:6819/1382694911","heartbeat_back_addr":"172.21.15.189:6823/1382694911","heartbeat_front_addr":"172.21.15.189:6821/1382694911","state":["exists","up"]},{"osd":8,"uuid":"1c218082-0b4c-41c8-92b9-453ae8d15a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6817","nonce":896348138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6819","nonce":896348138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6823","nonce":896348138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6821","nonce":896348138}]},"public_addr":"172.21.15.193:6817/896348138","cluster_addr":"172.21.15.193:6819/896348138","heartbeat_back_addr":"172.21.15.193:6823/896348138","heartbeat_front_addr":"172.21.15.193:6821/896348138","state":["exists","up"]},{"osd":9,"uuid":"48f39082-4fa9-4443-b5b7-b141c10d9e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6827","nonce":511237025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6829","nonce":511237025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6833","nonce":511237025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6831","nonce":511237025}]},"public_addr":"172.21.15.134:6827/511237025","cluster_addr":"172.21.15.134:6829/511237025","heartbeat_back_addr":"172.21.15.134:6833/511237025","heartbeat_front_addr":"172.21.15.134:6831/511237025","state":["exists","up"]},{"osd":10,"uuid":"95157766-1856-42f7-aea1-9cf83b77c8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6825","nonce":431740858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6827","nonce":431740858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6831","nonce":431740858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6829","nonce":431740858}]},"public_addr":"172.21.15.189:6825/431740858","cluster_addr":"172.21.15.189:6827/431740858","heartbeat_back_addr":"172.21.15.189:6831/431740858","heartbeat_front_addr":"172.21.15.189:6829/431740858","state":["exists","up"]},{"osd":11,"uuid":"244fadc7-f079-4736-8e7e-c0ce5ae6f43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6825","nonce":1594590579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6827","nonce":1594590579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6831","nonce":1594590579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6829","nonce":1594590579}]},"public_addr":"172.21.15.193:6825/1594590579","cluster_addr":"172.21.15.193:6827/1594590579","heartbeat_back_addr":"172.21.15.193:6831/1594590579","heartbeat_front_addr":"172.21.15.193:6829/1594590579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:11.712203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:35.845013+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:58.165609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:31.929746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:56.941555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:20.688071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:54.681514+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:20.789384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:44.568246+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:18.644310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:45.365351+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:09:09.663805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3790862248":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6800/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:0/2906927791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3068153391":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3056560791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6801/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/1310359922":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/2318666906":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/2901579347":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3404218719":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3598071145":"2024-07-27T11:02:33.711793+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:12:51.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.300+0000 7f66b2800640 1 -- 172.21.15.134:0/3774967088 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f66940b3cb0 msgr2=0x7f66940b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:51.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.300+0000 7f66b2800640 1 --2- 172.21.15.134:0/3774967088 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f66940b3cb0 0x7f66940b61c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f66a0031440 tx=0x7f66a00058d0 comp rx=0 tx=0).stop 2024-07-26T11:12:51.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.300+0000 7f66b2800640 1 -- 172.21.15.134:0/3774967088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ac10fc80 msgr2=0x7f66ac1a5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:12:51.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.300+0000 7f66b2800640 1 --2- 172.21.15.134:0/3774967088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ac10fc80 0x7f66ac1a5c30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f66a4009fd0 tx=0x7f66a400ef10 comp rx=0 tx=0).stop 2024-07-26T11:12:51.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.300+0000 7f66b2800640 1 -- 172.21.15.134:0/3774967088 shutdown_connections 2024-07-26T11:12:51.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.300+0000 7f66b2800640 1 --2- 172.21.15.134:0/3774967088 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f66940b3cb0 0x7f66940b61c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:51.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.300+0000 7f66b2800640 1 --2- 172.21.15.134:0/3774967088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f66ac10fc80 0x7f66ac1a5c30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:51.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.300+0000 7f66b2800640 1 --2- 172.21.15.134:0/3774967088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66ac10f330 0x7f66ac1a0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:51.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.300+0000 7f66b2800640 1 --2- 172.21.15.134:0/3774967088 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f66ac1099f0 0x7f66ac1a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:12:51.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.300+0000 7f66b2800640 1 -- 172.21.15.134:0/3774967088 >> 172.21.15.134:0/3774967088 conn(0x7f66ac078ff0 msgr2=0x7f66ac107fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:12:51.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.301+0000 7f66b2800640 1 -- 172.21.15.134:0/3774967088 shutdown_connections 2024-07-26T11:12:51.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:12:51.301+0000 7f66b2800640 1 -- 172.21.15.134:0/3774967088 wait complete. 2024-07-26T11:12:51.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:51 smithi189 ceph-mon[36790]: pgmap v386: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:12:52.236 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-07-26T11:12:52.236 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-07-26T11:12:52.237 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-07-26T11:12:52.237 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-07-26T11:12:52.237 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-07-26T11:12:52.238 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-07-26T11:12:52.238 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-07-26T11:12:52.239 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-07-26T11:12:52.239 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-07-26T11:12:52.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-07-26T11:12:52.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-07-26T11:12:52.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-07-26T11:12:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:52 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3774967088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:12:52.359 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:52 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3774967088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:12:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:52 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3774967088' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:12:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:53 smithi193 ceph-mon[37473]: pgmap v387: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:53.303 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:53.316 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:53.330 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:53.330 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:53.331 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:53.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:53 smithi134 ceph-mon[31966]: pgmap v387: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:53.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:53 smithi189 ceph-mon[36790]: pgmap v387: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:12:53.445 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:53.461 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:54.714 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:54.715 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:55.288 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:55 smithi134 ceph-mon[31966]: pgmap v388: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:12:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:55 smithi189 ceph-mon[36790]: pgmap v388: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:12:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:55 smithi193 ceph-mon[37473]: pgmap v388: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:12:56.148 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:56.149 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:57.085 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:57 smithi134 ceph-mon[31966]: pgmap v389: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:12:57.395 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:12:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:57 smithi189 ceph-mon[36790]: pgmap v389: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:12:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:57 smithi193 ceph-mon[37473]: pgmap v389: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:12:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:12:59 smithi134 ceph-mon[31966]: pgmap v390: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:12:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:12:59 smithi189 ceph-mon[36790]: pgmap v390: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:12:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:12:59 smithi193 ceph-mon[37473]: pgmap v390: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:13:01.191 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:01 smithi134 ceph-mon[31966]: pgmap v391: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:01 smithi189 ceph-mon[36790]: pgmap v391: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:01 smithi193 ceph-mon[37473]: pgmap v391: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:02.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.745+0000 7fefee200640 1 -- 172.21.15.134:0/749768935 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefe8109650 msgr2=0x7fefe810fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:02.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.745+0000 7fefee200640 1 --2- 172.21.15.134:0/749768935 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefe8109650 0x7fefe810fe80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fefe0009ca0 tx=0x7fefe002a8c0 comp rx=0 tx=0).stop 2024-07-26T11:13:02.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.745+0000 7fefee200640 1 -- 172.21.15.134:0/749768935 shutdown_connections 2024-07-26T11:13:02.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.745+0000 7fefee200640 1 --2- 172.21.15.134:0/749768935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefe8108d00 0x7fefe8109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:02.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.745+0000 7fefee200640 1 --2- 172.21.15.134:0/749768935 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefe8109650 0x7fefe810fe80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:02.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.745+0000 7fefee200640 1 --2- 172.21.15.134:0/749768935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe81033c0 0x7fefe81037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:02.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.745+0000 7fefee200640 1 -- 172.21.15.134:0/749768935 >> 172.21.15.134:0/749768935 conn(0x7fefe80fee10 msgr2=0x7fefe8101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:02.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.745+0000 7fefee200640 1 -- 172.21.15.134:0/749768935 shutdown_connections 2024-07-26T11:13:02.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.745+0000 7fefee200640 1 -- 172.21.15.134:0/749768935 wait complete. 2024-07-26T11:13:02.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.746+0000 7fefee200640 1 Processor -- start 2024-07-26T11:13:02.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.747+0000 7fefee200640 1 -- start start 2024-07-26T11:13:02.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.747+0000 7fefee200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefe81033c0 0x7fefe8075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:02.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.747+0000 7fefee200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefe8108d00 0x7fefe8075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:02.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.747+0000 7fefee200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe8109650 0x7fefe8075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:02.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.747+0000 7fefee200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fefe8113300 con 0x7fefe81033c0 2024-07-26T11:13:02.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.747+0000 7fefee200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fefe8113180 con 0x7fefe8109650 2024-07-26T11:13:02.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.747+0000 7fefee200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fefe8113480 con 0x7fefe8108d00 2024-07-26T11:13:02.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.747+0000 7fefece00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefe81033c0 0x7fefe8075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:02.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.747+0000 7fefece00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefe81033c0 0x7fefe8075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:56718/0 (socket says 172.21.15.134:56718) 2024-07-26T11:13:02.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.747+0000 7fefece00640 1 -- 172.21.15.134:0/2510129196 learned_addr learned my addr 172.21.15.134:0/2510129196 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:02.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.747+0000 7fefed800640 1 --2- 172.21.15.134:0/2510129196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe8109650 0x7fefe8075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:02.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.747+0000 7fefe7e00640 1 --2- 172.21.15.134:0/2510129196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefe8108d00 0x7fefe8075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:02.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.748+0000 7fefece00640 1 -- 172.21.15.134:0/2510129196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefe8108d00 msgr2=0x7fefe8075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:02.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.748+0000 7fefece00640 1 --2- 172.21.15.134:0/2510129196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefe8108d00 0x7fefe8075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:02.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.748+0000 7fefece00640 1 -- 172.21.15.134:0/2510129196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe8109650 msgr2=0x7fefe8075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:02.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.748+0000 7fefece00640 1 --2- 172.21.15.134:0/2510129196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe8109650 0x7fefe8075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:02.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.748+0000 7fefece00640 1 -- 172.21.15.134:0/2510129196 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe81a7fb0 con 0x7fefe81033c0 2024-07-26T11:13:02.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.748+0000 7fefe7e00640 1 --2- 172.21.15.134:0/2510129196 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefe8108d00 0x7fefe8075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:02.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.748+0000 7fefed800640 1 --2- 172.21.15.134:0/2510129196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe8109650 0x7fefe8075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:13:02.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.748+0000 7fefece00640 1 --2- 172.21.15.134:0/2510129196 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefe81033c0 0x7fefe8075140 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00e9f0 tx=0x7fefdc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:02.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.749+0000 7fefe5600640 1 -- 172.21.15.134:0/2510129196 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc00cd80 con 0x7fefe81033c0 2024-07-26T11:13:02.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.749+0000 7fefee200640 1 -- 172.21.15.134:0/2510129196 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe81a82a0 con 0x7fefe81033c0 2024-07-26T11:13:02.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.749+0000 7fefe5600640 1 -- 172.21.15.134:0/2510129196 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefdc0108e0 con 0x7fefe81033c0 2024-07-26T11:13:02.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.749+0000 7fefe5600640 1 -- 172.21.15.134:0/2510129196 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc018980 con 0x7fefe81033c0 2024-07-26T11:13:02.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.749+0000 7fefee200640 1 -- 172.21.15.134:0/2510129196 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe81a8830 con 0x7fefe81033c0 2024-07-26T11:13:02.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.750+0000 7fefee200640 1 -- 172.21.15.134:0/2510129196 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fefe8063580 con 0x7fefe81033c0 2024-07-26T11:13:02.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.753+0000 7fefe5600640 1 -- 172.21.15.134:0/2510129196 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fefdc010430 con 0x7fefe81033c0 2024-07-26T11:13:02.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.754+0000 7fefe5600640 1 --2- 172.21.15.134:0/2510129196 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fefd00b3cb0 0x7fefd00b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:02.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.754+0000 7fefe5600640 1 -- 172.21.15.134:0/2510129196 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fefdc0e5250 con 0x7fefe81033c0 2024-07-26T11:13:02.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.754+0000 7fefe7e00640 1 --2- 172.21.15.134:0/2510129196 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fefd00b3cb0 0x7fefd00b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:02.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.755+0000 7fefe5600640 1 --2- 172.21.15.134:0/2510129196 >> [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438] conn(0x7fefd00bf350 0x7fefd00c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:02.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.755+0000 7fefe5600640 1 -- 172.21.15.134:0/2510129196 --> [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fefdc015e80 con 0x7fefd00bf350 2024-07-26T11:13:02.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.755+0000 7fefe5600640 1 -- 172.21.15.134:0/2510129196 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fefdc0e5600 con 0x7fefe81033c0 2024-07-26T11:13:02.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.755+0000 7fefed800640 1 --2- 172.21.15.134:0/2510129196 >> [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438] conn(0x7fefd00bf350 0x7fefd00c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:02.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.755+0000 7fefe7e00640 1 --2- 172.21.15.134:0/2510129196 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fefd00b3cb0 0x7fefd00b61c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fefe8108860 tx=0x7fefd4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:02.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.756+0000 7fefed800640 1 --2- 172.21.15.134:0/2510129196 >> [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438] conn(0x7fefd00bf350 0x7fefd00c1840 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:02.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.757+0000 7fefe5600640 1 -- 172.21.15.134:0/2510129196 <== osd.6 v2:172.21.15.134:6818/2471505438 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fefdc015e80 con 0x7fefd00bf350 2024-07-26T11:13:02.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.772+0000 7fefee200640 1 -- 172.21.15.134:0/2510129196 --> [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fefe8063980 con 0x7fefd00bf350 2024-07-26T11:13:02.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.772+0000 7fefe5600640 1 -- 172.21.15.134:0/2510129196 <== osd.6 v2:172.21.15.134:6818/2471505438 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fefe8063980 con 0x7fefd00bf350 2024-07-26T11:13:02.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.773+0000 7fefee200640 1 -- 172.21.15.134:0/2510129196 >> [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438] conn(0x7fefd00bf350 msgr2=0x7fefd00c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:02.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.773+0000 7fefee200640 1 --2- 172.21.15.134:0/2510129196 >> [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438] conn(0x7fefd00bf350 0x7fefd00c1840 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:02.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.773+0000 7fefee200640 1 -- 172.21.15.134:0/2510129196 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fefd00b3cb0 msgr2=0x7fefd00b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:02.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.773+0000 7fefee200640 1 --2- 172.21.15.134:0/2510129196 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fefd00b3cb0 0x7fefd00b61c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fefe8108860 tx=0x7fefd4005f90 comp rx=0 tx=0).stop 2024-07-26T11:13:02.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.773+0000 7fefee200640 1 -- 172.21.15.134:0/2510129196 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefe81033c0 msgr2=0x7fefe8075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:02.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.773+0000 7fefee200640 1 --2- 172.21.15.134:0/2510129196 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fefe81033c0 0x7fefe8075140 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00e9f0 tx=0x7fefdc00eeb0 comp rx=0 tx=0).stop 2024-07-26T11:13:02.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.773+0000 7fefed800640 1 -- 172.21.15.134:0/2510129196 reap_dead start 2024-07-26T11:13:02.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.773+0000 7fefee200640 1 -- 172.21.15.134:0/2510129196 shutdown_connections 2024-07-26T11:13:02.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.773+0000 7fefee200640 1 -- 172.21.15.134:0/2510129196 >> 172.21.15.134:0/2510129196 conn(0x7fefe80fee10 msgr2=0x7fefe8101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:02.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.774+0000 7fefee200640 1 -- 172.21.15.134:0/2510129196 shutdown_connections 2024-07-26T11:13:02.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:02.774+0000 7fefee200640 1 -- 172.21.15.134:0/2510129196 wait complete. 2024-07-26T11:13:03.315 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:03 smithi134 ceph-mon[31966]: pgmap v392: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:03.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.368+0000 7f2b91a00640 1 -- 172.21.15.134:0/844990022 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b8c104ce0 msgr2=0x7f2b8c10d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:03.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.368+0000 7f2b91a00640 1 --2- 172.21.15.134:0/844990022 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b8c104ce0 0x7f2b8c10d320 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009900 tx=0x7f2b7802a410 comp rx=0 tx=0).stop 2024-07-26T11:13:03.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.368+0000 7f2b91a00640 1 -- 172.21.15.134:0/844990022 shutdown_connections 2024-07-26T11:13:03.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.368+0000 7f2b91a00640 1 --2- 172.21.15.134:0/844990022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b8c10d860 0x7f2b8c10fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:03.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.368+0000 7f2b91a00640 1 --2- 172.21.15.134:0/844990022 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b8c104ce0 0x7f2b8c10d320 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:03.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.368+0000 7f2b91a00640 1 --2- 172.21.15.134:0/844990022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b8c1042b0 0x7f2b8c1046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:03.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.368+0000 7f2b91a00640 1 -- 172.21.15.134:0/844990022 >> 172.21.15.134:0/844990022 conn(0x7f2b8c079160 msgr2=0x7f2b8c079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:03.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.368+0000 7f2b91a00640 1 -- 172.21.15.134:0/844990022 shutdown_connections 2024-07-26T11:13:03.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.368+0000 7f2b91a00640 1 -- 172.21.15.134:0/844990022 wait complete. 2024-07-26T11:13:03.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.369+0000 7f2b91a00640 1 Processor -- start 2024-07-26T11:13:03.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.369+0000 7f2b91a00640 1 -- start start 2024-07-26T11:13:03.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.370+0000 7f2b91a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b8c1042b0 0x7f2b8c19c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:03.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.370+0000 7f2b91a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b8c104ce0 0x7f2b8c19c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:03.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.370+0000 7f2b91a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b8c10d860 0x7f2b8c1a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:03.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.370+0000 7f2b91a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c113480 con 0x7f2b8c1042b0 2024-07-26T11:13:03.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.370+0000 7f2b91a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c113300 con 0x7f2b8c10d860 2024-07-26T11:13:03.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.370+0000 7f2b91a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2b8c113600 con 0x7f2b8c104ce0 2024-07-26T11:13:03.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.370+0000 7f2b8aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b8c1042b0 0x7f2b8c19c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:03.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.370+0000 7f2b8aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b8c1042b0 0x7f2b8c19c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:56734/0 (socket says 172.21.15.134:56734) 2024-07-26T11:13:03.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.370+0000 7f2b8aa00640 1 -- 172.21.15.134:0/673638934 learned_addr learned my addr 172.21.15.134:0/673638934 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:03.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.370+0000 7f2b8a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b8c104ce0 0x7f2b8c19c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:03.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.370+0000 7f2b8b400640 1 --2- 172.21.15.134:0/673638934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b8c10d860 0x7f2b8c1a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:03.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.371+0000 7f2b8aa00640 1 -- 172.21.15.134:0/673638934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b8c104ce0 msgr2=0x7f2b8c19c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:03.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.371+0000 7f2b8aa00640 1 --2- 172.21.15.134:0/673638934 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b8c104ce0 0x7f2b8c19c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:03.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.371+0000 7f2b8aa00640 1 -- 172.21.15.134:0/673638934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b8c10d860 msgr2=0x7f2b8c1a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:03.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.371+0000 7f2b8aa00640 1 --2- 172.21.15.134:0/673638934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b8c10d860 0x7f2b8c1a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:03.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.371+0000 7f2b8aa00640 1 -- 172.21.15.134:0/673638934 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8c1a2060 con 0x7f2b8c1042b0 2024-07-26T11:13:03.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.371+0000 7f2b8aa00640 1 --2- 172.21.15.134:0/673638934 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b8c1042b0 0x7f2b8c19c0f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00e9c0 tx=0x7f2b7c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:03.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.372+0000 7f2b83400640 1 -- 172.21.15.134:0/673638934 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c00cde0 con 0x7f2b8c1042b0 2024-07-26T11:13:03.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.372+0000 7f2b83400640 1 -- 172.21.15.134:0/673638934 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b7c0108e0 con 0x7f2b8c1042b0 2024-07-26T11:13:03.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.372+0000 7f2b91a00640 1 -- 172.21.15.134:0/673638934 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b8c1a2350 con 0x7f2b8c1042b0 2024-07-26T11:13:03.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.372+0000 7f2b83400640 1 -- 172.21.15.134:0/673638934 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7c018960 con 0x7f2b8c1042b0 2024-07-26T11:13:03.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.372+0000 7f2b91a00640 1 -- 172.21.15.134:0/673638934 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b8c1a8ab0 con 0x7f2b8c1042b0 2024-07-26T11:13:03.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.373+0000 7f2b91a00640 1 -- 172.21.15.134:0/673638934 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2b8c100aa0 con 0x7f2b8c1042b0 2024-07-26T11:13:03.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.376+0000 7f2b83400640 1 -- 172.21.15.134:0/673638934 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f2b7c01f020 con 0x7f2b8c1042b0 2024-07-26T11:13:03.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.377+0000 7f2b83400640 1 --2- 172.21.15.134:0/673638934 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b700b3d30 0x7f2b700b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:03.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.377+0000 7f2b83400640 1 -- 172.21.15.134:0/673638934 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2b7c0e4ea0 con 0x7f2b8c1042b0 2024-07-26T11:13:03.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.377+0000 7f2b8a000640 1 --2- 172.21.15.134:0/673638934 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b700b3d30 0x7f2b700b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:03.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.378+0000 7f2b83400640 1 --2- 172.21.15.134:0/673638934 >> [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138] conn(0x7f2b700bf3d0 0x7f2b700c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:03.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.378+0000 7f2b83400640 1 -- 172.21.15.134:0/673638934 --> [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b700c1f90 con 0x7f2b700bf3d0 2024-07-26T11:13:03.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.378+0000 7f2b83400640 1 -- 172.21.15.134:0/673638934 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f2b7c0e5250 con 0x7f2b8c1042b0 2024-07-26T11:13:03.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.378+0000 7f2b8b400640 1 --2- 172.21.15.134:0/673638934 >> [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138] conn(0x7f2b700bf3d0 0x7f2b700c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:03.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.378+0000 7f2b8a000640 1 --2- 172.21.15.134:0/673638934 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b700b3d30 0x7f2b700b6240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2b78000c00 tx=0x7f2b78031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:03.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.379+0000 7f2b8b400640 1 --2- 172.21.15.134:0/673638934 >> [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138] conn(0x7f2b700bf3d0 0x7f2b700c18c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:03.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.380+0000 7f2b83400640 1 -- 172.21.15.134:0/673638934 <== osd.8 v2:172.21.15.193:6816/896348138 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2b700c1f90 con 0x7f2b700bf3d0 2024-07-26T11:13:03.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.396+0000 7f2b91a00640 1 -- 172.21.15.134:0/673638934 --> [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2b8c06aef0 con 0x7f2b700bf3d0 2024-07-26T11:13:03.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.396+0000 7f2b83400640 1 -- 172.21.15.134:0/673638934 <== osd.8 v2:172.21.15.193:6816/896348138 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2b8c06aef0 con 0x7f2b700bf3d0 2024-07-26T11:13:03.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.397+0000 7f2b91a00640 1 -- 172.21.15.134:0/673638934 >> [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138] conn(0x7f2b700bf3d0 msgr2=0x7f2b700c18c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:03.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.397+0000 7f2b91a00640 1 --2- 172.21.15.134:0/673638934 >> [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138] conn(0x7f2b700bf3d0 0x7f2b700c18c0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:03.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.397+0000 7f2b91a00640 1 -- 172.21.15.134:0/673638934 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b700b3d30 msgr2=0x7f2b700b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:03.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.397+0000 7f2b91a00640 1 --2- 172.21.15.134:0/673638934 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b700b3d30 0x7f2b700b6240 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2b78000c00 tx=0x7f2b78031400 comp rx=0 tx=0).stop 2024-07-26T11:13:03.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.397+0000 7f2b91a00640 1 -- 172.21.15.134:0/673638934 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b8c1042b0 msgr2=0x7f2b8c19c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:03.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.397+0000 7f2b91a00640 1 --2- 172.21.15.134:0/673638934 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b8c1042b0 0x7f2b8c19c0f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c00e9c0 tx=0x7f2b7c00ee80 comp rx=0 tx=0).stop 2024-07-26T11:13:03.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.397+0000 7f2b8b400640 1 -- 172.21.15.134:0/673638934 reap_dead start 2024-07-26T11:13:03.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.397+0000 7f2b91a00640 1 -- 172.21.15.134:0/673638934 shutdown_connections 2024-07-26T11:13:03.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.397+0000 7f2b91a00640 1 -- 172.21.15.134:0/673638934 >> 172.21.15.134:0/673638934 conn(0x7f2b8c079160 msgr2=0x7f2b8c100040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:03.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.397+0000 7f2b91a00640 1 -- 172.21.15.134:0/673638934 shutdown_connections 2024-07-26T11:13:03.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.397+0000 7f2b91a00640 1 -- 172.21.15.134:0/673638934 wait complete. 2024-07-26T11:13:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:03 smithi189 ceph-mon[36790]: pgmap v392: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:03 smithi193 ceph-mon[37473]: pgmap v392: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:05.387 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:05 smithi134 ceph-mon[31966]: pgmap v393: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-07-26T11:13:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:05 smithi189 ceph-mon[36790]: pgmap v393: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-07-26T11:13:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:05 smithi193 ceph-mon[37473]: pgmap v393: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-07-26T11:13:07.330 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:07 smithi134 ceph-mon[31966]: pgmap v394: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:07.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.679+0000 7f2e19000640 1 -- 172.21.15.134:0/617965754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e14104b40 msgr2=0x7f2e1410d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:07.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.679+0000 7f2e19000640 1 --2- 172.21.15.134:0/617965754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e14104b40 0x7f2e1410d090 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc009860 tx=0x7f2dfc02a340 comp rx=0 tx=0).stop 2024-07-26T11:13:07.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.679+0000 7f2e19000640 1 -- 172.21.15.134:0/617965754 shutdown_connections 2024-07-26T11:13:07.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.679+0000 7f2e19000640 1 --2- 172.21.15.134:0/617965754 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e1410d650 0x7f2e1410fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:07.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.679+0000 7f2e19000640 1 --2- 172.21.15.134:0/617965754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e14104b40 0x7f2e1410d090 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:07.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.679+0000 7f2e19000640 1 --2- 172.21.15.134:0/617965754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e141041f0 0x7f2e14104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:07.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.679+0000 7f2e19000640 1 -- 172.21.15.134:0/617965754 >> 172.21.15.134:0/617965754 conn(0x7f2e140fdcf0 msgr2=0x7f2e14100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:07.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.679+0000 7f2e19000640 1 -- 172.21.15.134:0/617965754 shutdown_connections 2024-07-26T11:13:07.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.679+0000 7f2e19000640 1 -- 172.21.15.134:0/617965754 wait complete. 2024-07-26T11:13:07.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.680+0000 7f2e19000640 1 Processor -- start 2024-07-26T11:13:07.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.680+0000 7f2e19000640 1 -- start start 2024-07-26T11:13:07.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.681+0000 7f2e19000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e141041f0 0x7f2e14101f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:07.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.681+0000 7f2e19000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e14104b40 0x7f2e14102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:07.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.681+0000 7f2e19000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1410d650 0x7f2e141029b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:07.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.681+0000 7f2e19000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2e14113280 con 0x7f2e14104b40 2024-07-26T11:13:07.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.681+0000 7f2e19000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2e14113100 con 0x7f2e1410d650 2024-07-26T11:13:07.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.681+0000 7f2e19000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2e14113400 con 0x7f2e141041f0 2024-07-26T11:13:07.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.681+0000 7f2e0be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e14104b40 0x7f2e14102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:07.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.681+0000 7f2e0be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e14104b40 0x7f2e14102470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:56764/0 (socket says 172.21.15.134:56764) 2024-07-26T11:13:07.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.681+0000 7f2e13400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e141041f0 0x7f2e14101f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:07.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.681+0000 7f2e13e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1410d650 0x7f2e141029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:07.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.681+0000 7f2e0be00640 1 -- 172.21.15.134:0/2315399984 learned_addr learned my addr 172.21.15.134:0/2315399984 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:07.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.682+0000 7f2e0be00640 1 -- 172.21.15.134:0/2315399984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e141041f0 msgr2=0x7f2e14101f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:07.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.682+0000 7f2e0be00640 1 --2- 172.21.15.134:0/2315399984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e141041f0 0x7f2e14101f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:07.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.682+0000 7f2e0be00640 1 -- 172.21.15.134:0/2315399984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1410d650 msgr2=0x7f2e141029b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:07.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.682+0000 7f2e0be00640 1 --2- 172.21.15.134:0/2315399984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1410d650 0x7f2e141029b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:07.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.682+0000 7f2e0be00640 1 -- 172.21.15.134:0/2315399984 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e141a7fb0 con 0x7f2e14104b40 2024-07-26T11:13:07.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.682+0000 7f2e13e00640 1 --2- 172.21.15.134:0/2315399984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1410d650 0x7f2e141029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:07.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.682+0000 7f2e13400640 1 --2- 172.21.15.134:0/2315399984 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2e141041f0 0x7f2e14101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:07.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.682+0000 7f2e0be00640 1 --2- 172.21.15.134:0/2315399984 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e14104b40 0x7f2e14102470 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc035040 tx=0x7f2dfc032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:07.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.683+0000 7f2e10c00640 1 -- 172.21.15.134:0/2315399984 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc037040 con 0x7f2e14104b40 2024-07-26T11:13:07.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.683+0000 7f2e19000640 1 -- 172.21.15.134:0/2315399984 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e141a8240 con 0x7f2e14104b40 2024-07-26T11:13:07.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.683+0000 7f2e10c00640 1 -- 172.21.15.134:0/2315399984 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dfc030c10 con 0x7f2e14104b40 2024-07-26T11:13:07.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.683+0000 7f2e10c00640 1 -- 172.21.15.134:0/2315399984 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc0035f0 con 0x7f2e14104b40 2024-07-26T11:13:07.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.683+0000 7f2e19000640 1 -- 172.21.15.134:0/2315399984 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e141a86c0 con 0x7f2e14104b40 2024-07-26T11:13:07.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.684+0000 7f2e19000640 1 -- 172.21.15.134:0/2315399984 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2e14107bf0 con 0x7f2e14104b40 2024-07-26T11:13:07.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.687+0000 7f2e10c00640 1 -- 172.21.15.134:0/2315399984 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f2dfc043020 con 0x7f2e14104b40 2024-07-26T11:13:07.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.689+0000 7f2e10c00640 1 --2- 172.21.15.134:0/2315399984 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2df80b3cb0 0x7f2df80b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:07.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.689+0000 7f2e10c00640 1 -- 172.21.15.134:0/2315399984 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2dfc103970 con 0x7f2e14104b40 2024-07-26T11:13:07.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.689+0000 7f2e13400640 1 --2- 172.21.15.134:0/2315399984 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2df80b3cb0 0x7f2df80b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:07.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.690+0000 7f2e10c00640 1 --2- 172.21.15.134:0/2315399984 >> [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347] conn(0x7f2df80bf350 0x7f2df80c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:07.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.690+0000 7f2e10c00640 1 -- 172.21.15.134:0/2315399984 --> [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2dfc02be50 con 0x7f2df80bf350 2024-07-26T11:13:07.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.690+0000 7f2e10c00640 1 -- 172.21.15.134:0/2315399984 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f2dfc103d20 con 0x7f2e14104b40 2024-07-26T11:13:07.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.690+0000 7f2e13e00640 1 --2- 172.21.15.134:0/2315399984 >> [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347] conn(0x7f2df80bf350 0x7f2df80c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:07.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.690+0000 7f2e13400640 1 --2- 172.21.15.134:0/2315399984 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2df80b3cb0 0x7f2df80b61c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2e14103980 tx=0x7f2e04005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:07.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.691+0000 7f2e13e00640 1 --2- 172.21.15.134:0/2315399984 >> [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347] conn(0x7f2df80bf350 0x7f2df80c1840 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:07.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.692+0000 7f2e10c00640 1 -- 172.21.15.134:0/2315399984 <== osd.4 v2:172.21.15.189:6808/1786881347 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2dfc02be50 con 0x7f2df80bf350 2024-07-26T11:13:07.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.706+0000 7f2e19000640 1 -- 172.21.15.134:0/2315399984 --> [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2e1406af90 con 0x7f2df80bf350 2024-07-26T11:13:07.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.707+0000 7f2e10c00640 1 -- 172.21.15.134:0/2315399984 <== osd.4 v2:172.21.15.189:6808/1786881347 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2e1406af90 con 0x7f2df80bf350 2024-07-26T11:13:07.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.708+0000 7f2e19000640 1 -- 172.21.15.134:0/2315399984 >> [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347] conn(0x7f2df80bf350 msgr2=0x7f2df80c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:07.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.708+0000 7f2e19000640 1 --2- 172.21.15.134:0/2315399984 >> [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347] conn(0x7f2df80bf350 0x7f2df80c1840 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:07.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.708+0000 7f2e19000640 1 -- 172.21.15.134:0/2315399984 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2df80b3cb0 msgr2=0x7f2df80b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:07.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.708+0000 7f2e19000640 1 --2- 172.21.15.134:0/2315399984 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2df80b3cb0 0x7f2df80b61c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2e14103980 tx=0x7f2e04005f90 comp rx=0 tx=0).stop 2024-07-26T11:13:07.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.708+0000 7f2e19000640 1 -- 172.21.15.134:0/2315399984 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e14104b40 msgr2=0x7f2e14102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:07.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.708+0000 7f2e19000640 1 --2- 172.21.15.134:0/2315399984 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2e14104b40 0x7f2e14102470 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc035040 tx=0x7f2dfc032d50 comp rx=0 tx=0).stop 2024-07-26T11:13:07.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.708+0000 7f2e13e00640 1 -- 172.21.15.134:0/2315399984 reap_dead start 2024-07-26T11:13:07.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.708+0000 7f2e19000640 1 -- 172.21.15.134:0/2315399984 shutdown_connections 2024-07-26T11:13:07.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.708+0000 7f2e19000640 1 -- 172.21.15.134:0/2315399984 >> 172.21.15.134:0/2315399984 conn(0x7f2e140fdcf0 msgr2=0x7f2e141071d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:07.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.708+0000 7f2e19000640 1 -- 172.21.15.134:0/2315399984 shutdown_connections 2024-07-26T11:13:07.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:03.708+0000 7f2e19000640 1 -- 172.21.15.134:0/2315399984 wait complete. 2024-07-26T11:13:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:07 smithi189 ceph-mon[36790]: pgmap v394: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:07 smithi193 ceph-mon[37473]: pgmap v394: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:08 smithi134 ceph-mon[31966]: pgmap v395: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:13:08.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:08 smithi189 ceph-mon[36790]: pgmap v395: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:13:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:08 smithi193 ceph-mon[37473]: pgmap v395: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:13:09.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.442+0000 7f77d9400640 1 -- 172.21.15.134:0/3668265866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77d41099f0 msgr2=0x7f77d4109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:09.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.442+0000 7f77d9400640 1 --2- 172.21.15.134:0/3668265866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77d41099f0 0x7f77d4109e00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f77c4009900 tx=0x7f77c402a410 comp rx=0 tx=0).stop 2024-07-26T11:13:09.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.442+0000 7f77d9400640 1 -- 172.21.15.134:0/3668265866 shutdown_connections 2024-07-26T11:13:09.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.442+0000 7f77d9400640 1 --2- 172.21.15.134:0/3668265866 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77d410fc80 0x7f77d41164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:09.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.442+0000 7f77d9400640 1 --2- 172.21.15.134:0/3668265866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77d410f330 0x7f77d410f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:09.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.442+0000 7f77d9400640 1 --2- 172.21.15.134:0/3668265866 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77d41099f0 0x7f77d4109e00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:09.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.442+0000 7f77d9400640 1 -- 172.21.15.134:0/3668265866 >> 172.21.15.134:0/3668265866 conn(0x7f77d4078ff0 msgr2=0x7f77d4107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:09.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.442+0000 7f77d9400640 1 -- 172.21.15.134:0/3668265866 shutdown_connections 2024-07-26T11:13:09.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.442+0000 7f77d9400640 1 -- 172.21.15.134:0/3668265866 wait complete. 2024-07-26T11:13:09.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.443+0000 7f77d9400640 1 Processor -- start 2024-07-26T11:13:09.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.443+0000 7f77d9400640 1 -- start start 2024-07-26T11:13:09.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.444+0000 7f77d9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77d41099f0 0x7f77d41a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.444+0000 7f77d9400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77d410f330 0x7f77d41a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.444+0000 7f77d9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77d410fc80 0x7f77d41a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.444+0000 7f77d9400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f77d4119940 con 0x7f77d410f330 2024-07-26T11:13:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.444+0000 7f77d9400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f77d41197c0 con 0x7f77d410fc80 2024-07-26T11:13:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.444+0000 7f77d9400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f77d4119ac0 con 0x7f77d41099f0 2024-07-26T11:13:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.444+0000 7f77cb400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77d410f330 0x7f77d41a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.444+0000 7f77d3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77d41099f0 0x7f77d41a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.444+0000 7f77cb400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77d410f330 0x7f77d41a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:56786/0 (socket says 172.21.15.134:56786) 2024-07-26T11:13:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.444+0000 7f77cb400640 1 -- 172.21.15.134:0/2811392434 learned_addr learned my addr 172.21.15.134:0/2811392434 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.444+0000 7f77d8a00640 1 --2- 172.21.15.134:0/2811392434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77d410fc80 0x7f77d41a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.445+0000 7f77d3e00640 1 -- 172.21.15.134:0/2811392434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77d410fc80 msgr2=0x7f77d41a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.445+0000 7f77d3e00640 1 --2- 172.21.15.134:0/2811392434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77d410fc80 0x7f77d41a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:09.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.445+0000 7f77d3e00640 1 -- 172.21.15.134:0/2811392434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77d410f330 msgr2=0x7f77d41a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:09.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.445+0000 7f77d3e00640 1 --2- 172.21.15.134:0/2811392434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77d410f330 0x7f77d41a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:09.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.445+0000 7f77d3e00640 1 -- 172.21.15.134:0/2811392434 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d41a6280 con 0x7f77d41099f0 2024-07-26T11:13:09.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.445+0000 7f77cb400640 1 --2- 172.21.15.134:0/2811392434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f77d410f330 0x7f77d41a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:09.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.445+0000 7f77d3e00640 1 --2- 172.21.15.134:0/2811392434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77d41099f0 0x7f77d41a0310 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f77c402a930 tx=0x7f77c4031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:09.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.445+0000 7f77d1600640 1 -- 172.21.15.134:0/2811392434 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c403f030 con 0x7f77d41099f0 2024-07-26T11:13:09.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.445+0000 7f77d1600640 1 -- 172.21.15.134:0/2811392434 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77c4004490 con 0x7f77d41099f0 2024-07-26T11:13:09.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.445+0000 7f77d1600640 1 -- 172.21.15.134:0/2811392434 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c4035070 con 0x7f77d41099f0 2024-07-26T11:13:09.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.445+0000 7f77d9400640 1 -- 172.21.15.134:0/2811392434 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77d41a6510 con 0x7f77d41099f0 2024-07-26T11:13:09.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.446+0000 7f77d9400640 1 -- 172.21.15.134:0/2811392434 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77d41accd0 con 0x7f77d41099f0 2024-07-26T11:13:09.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.446+0000 7f77d9400640 1 -- 172.21.15.134:0/2811392434 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f77d4063580 con 0x7f77d41099f0 2024-07-26T11:13:09.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.449+0000 7f77d1600640 1 -- 172.21.15.134:0/2811392434 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f77c4042030 con 0x7f77d41099f0 2024-07-26T11:13:09.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.451+0000 7f77d1600640 1 --2- 172.21.15.134:0/2811392434 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f77b80b3d30 0x7f77b80b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:09.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.451+0000 7f77d1600640 1 -- 172.21.15.134:0/2811392434 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f77c4104b80 con 0x7f77d41099f0 2024-07-26T11:13:09.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.451+0000 7f77cb400640 1 --2- 172.21.15.134:0/2811392434 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f77b80b3d30 0x7f77b80b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:09.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.451+0000 7f77d1600640 1 --2- 172.21.15.134:0/2811392434 >> [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049] conn(0x7f77b80bf3d0 0x7f77b80c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:09.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.451+0000 7f77d1600640 1 -- 172.21.15.134:0/2811392434 --> [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f77c403ce50 con 0x7f77b80bf3d0 2024-07-26T11:13:09.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.451+0000 7f77d1600640 1 -- 172.21.15.134:0/2811392434 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f77c4104f30 con 0x7f77d41099f0 2024-07-26T11:13:09.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.452+0000 7f77d8a00640 1 --2- 172.21.15.134:0/2811392434 >> [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049] conn(0x7f77b80bf3d0 0x7f77b80c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:09.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.452+0000 7f77cb400640 1 --2- 172.21.15.134:0/2811392434 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f77b80b3d30 0x7f77b80b6240 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f77d410fa50 tx=0x7f77bc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:09.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.452+0000 7f77d8a00640 1 --2- 172.21.15.134:0/2811392434 >> [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049] conn(0x7f77b80bf3d0 0x7f77b80c18c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:09.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.453+0000 7f77d1600640 1 -- 172.21.15.134:0/2811392434 <== osd.1 v2:172.21.15.189:6800/35148049 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f77c403ce50 con 0x7f77b80bf3d0 2024-07-26T11:13:09.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.468+0000 7f77d9400640 1 -- 172.21.15.134:0/2811392434 --> [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f77d4063980 con 0x7f77b80bf3d0 2024-07-26T11:13:09.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.469+0000 7f77d1600640 1 -- 172.21.15.134:0/2811392434 <== osd.1 v2:172.21.15.189:6800/35148049 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f77d4063980 con 0x7f77b80bf3d0 2024-07-26T11:13:09.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.469+0000 7f77d9400640 1 -- 172.21.15.134:0/2811392434 >> [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049] conn(0x7f77b80bf3d0 msgr2=0x7f77b80c18c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:09.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.469+0000 7f77d9400640 1 --2- 172.21.15.134:0/2811392434 >> [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049] conn(0x7f77b80bf3d0 0x7f77b80c18c0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:09.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.469+0000 7f77d9400640 1 -- 172.21.15.134:0/2811392434 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f77b80b3d30 msgr2=0x7f77b80b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:09.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.469+0000 7f77d9400640 1 --2- 172.21.15.134:0/2811392434 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f77b80b3d30 0x7f77b80b6240 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f77d410fa50 tx=0x7f77bc005f90 comp rx=0 tx=0).stop 2024-07-26T11:13:09.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.469+0000 7f77d9400640 1 -- 172.21.15.134:0/2811392434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77d41099f0 msgr2=0x7f77d41a0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:09.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.469+0000 7f77d9400640 1 --2- 172.21.15.134:0/2811392434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f77d41099f0 0x7f77d41a0310 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f77c402a930 tx=0x7f77c4031a20 comp rx=0 tx=0).stop 2024-07-26T11:13:09.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.469+0000 7f77d8a00640 1 -- 172.21.15.134:0/2811392434 reap_dead start 2024-07-26T11:13:09.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.470+0000 7f77d9400640 1 -- 172.21.15.134:0/2811392434 shutdown_connections 2024-07-26T11:13:09.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.470+0000 7f77d9400640 1 -- 172.21.15.134:0/2811392434 >> 172.21.15.134:0/2811392434 conn(0x7f77d4078ff0 msgr2=0x7f77d40796f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:09.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.470+0000 7f77d9400640 1 -- 172.21.15.134:0/2811392434 shutdown_connections 2024-07-26T11:13:09.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:09.470+0000 7f77d9400640 1 -- 172.21.15.134:0/2811392434 wait complete. 2024-07-26T11:13:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:11 smithi193 ceph-mon[37473]: pgmap v396: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:11.293 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:11 smithi134 ceph-mon[31966]: pgmap v396: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:11 smithi189 ceph-mon[36790]: pgmap v396: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:12.516 INFO:teuthology.orchestra.run.smithi134.stdout:193273528396 2024-07-26T11:13:12.517 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-07-26T11:13:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:13 smithi193 ceph-mon[37473]: pgmap v397: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:13 smithi134 ceph-mon[31966]: pgmap v397: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:13 smithi189 ceph-mon[36790]: pgmap v397: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:13.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.743+0000 7fd564400640 1 -- 172.21.15.134:0/898870610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55c10b120 msgr2=0x7fd55c10b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:13.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.743+0000 7fd564400640 1 --2- 172.21.15.134:0/898870610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55c10b120 0x7fd55c10b530 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd548009900 tx=0x7fd54802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:13:13.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.743+0000 7fd564400640 1 -- 172.21.15.134:0/898870610 shutdown_connections 2024-07-26T11:13:13.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.743+0000 7fd564400640 1 --2- 172.21.15.134:0/898870610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd55c10ba70 0x7fd55c1122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:13.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.743+0000 7fd564400640 1 --2- 172.21.15.134:0/898870610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55c10b120 0x7fd55c10b530 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:13.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.743+0000 7fd564400640 1 --2- 172.21.15.134:0/898870610 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd55c1057e0 0x7fd55c105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:13.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.743+0000 7fd564400640 1 -- 172.21.15.134:0/898870610 >> 172.21.15.134:0/898870610 conn(0x7fd55c1011f0 msgr2=0x7fd55c103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:13.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.743+0000 7fd564400640 1 -- 172.21.15.134:0/898870610 shutdown_connections 2024-07-26T11:13:13.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.743+0000 7fd564400640 1 -- 172.21.15.134:0/898870610 wait complete. 2024-07-26T11:13:13.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.745+0000 7fd564400640 1 Processor -- start 2024-07-26T11:13:13.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.745+0000 7fd564400640 1 -- start start 2024-07-26T11:13:13.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.745+0000 7fd564400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd55c1057e0 0x7fd55c07b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:13.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.745+0000 7fd564400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55c10b120 0x7fd55c077350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:13.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.745+0000 7fd564400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd55c10ba70 0x7fd55c077a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:13.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.745+0000 7fd564400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd55c115730 con 0x7fd55c1057e0 2024-07-26T11:13:13.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.745+0000 7fd564400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd55c1155b0 con 0x7fd55c10b120 2024-07-26T11:13:13.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.745+0000 7fd564400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd55c1158b0 con 0x7fd55c10ba70 2024-07-26T11:13:13.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.746+0000 7fd561a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd55c1057e0 0x7fd55c07b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:13.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.746+0000 7fd561000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55c10b120 0x7fd55c077350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:13.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.746+0000 7fd561000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55c10b120 0x7fd55c077350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:45808/0 (socket says 172.21.15.134:45808) 2024-07-26T11:13:13.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.746+0000 7fd561000640 1 -- 172.21.15.134:0/1270148108 learned_addr learned my addr 172.21.15.134:0/1270148108 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:13.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.746+0000 7fd562400640 1 --2- 172.21.15.134:0/1270148108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd55c10ba70 0x7fd55c077a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:13.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.746+0000 7fd562400640 1 -- 172.21.15.134:0/1270148108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55c10b120 msgr2=0x7fd55c077350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:13.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.746+0000 7fd562400640 1 --2- 172.21.15.134:0/1270148108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55c10b120 0x7fd55c077350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:13.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.746+0000 7fd562400640 1 -- 172.21.15.134:0/1270148108 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd55c1057e0 msgr2=0x7fd55c07b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:13.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.746+0000 7fd562400640 1 --2- 172.21.15.134:0/1270148108 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd55c1057e0 0x7fd55c07b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:13.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.746+0000 7fd562400640 1 -- 172.21.15.134:0/1270148108 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd55c1ac690 con 0x7fd55c10ba70 2024-07-26T11:13:13.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.746+0000 7fd561000640 1 --2- 172.21.15.134:0/1270148108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55c10b120 0x7fd55c077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:13.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.747+0000 7fd562400640 1 --2- 172.21.15.134:0/1270148108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd55c10ba70 0x7fd55c077a70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd55400e970 tx=0x7fd55400ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:13.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.747+0000 7fd55a000640 1 -- 172.21.15.134:0/1270148108 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55400ccc0 con 0x7fd55c10ba70 2024-07-26T11:13:13.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.747+0000 7fd564400640 1 -- 172.21.15.134:0/1270148108 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd55c1ac8c0 con 0x7fd55c10ba70 2024-07-26T11:13:13.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.747+0000 7fd564400640 1 -- 172.21.15.134:0/1270148108 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd55c1acdd0 con 0x7fd55c10ba70 2024-07-26T11:13:13.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.747+0000 7fd55a000640 1 -- 172.21.15.134:0/1270148108 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5540044d0 con 0x7fd55c10ba70 2024-07-26T11:13:13.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.747+0000 7fd55a000640 1 -- 172.21.15.134:0/1270148108 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55401c070 con 0x7fd55c10ba70 2024-07-26T11:13:13.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.748+0000 7fd564400640 1 -- 172.21.15.134:0/1270148108 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd55c063500 con 0x7fd55c10ba70 2024-07-26T11:13:13.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.751+0000 7fd55a000640 1 -- 172.21.15.134:0/1270148108 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fd554005020 con 0x7fd55c10ba70 2024-07-26T11:13:13.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.753+0000 7fd55a000640 1 --2- 172.21.15.134:0/1270148108 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd5440b3db0 0x7fd5440b62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:13.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.753+0000 7fd55a000640 1 -- 172.21.15.134:0/1270148108 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd5540e4570 con 0x7fd55c10ba70 2024-07-26T11:13:13.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.753+0000 7fd561a00640 1 --2- 172.21.15.134:0/1270148108 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd5440b3db0 0x7fd5440b62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:13.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.754+0000 7fd55a000640 1 --2- 172.21.15.134:0/1270148108 >> [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976] conn(0x7fd5440bf450 0x7fd5440c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:13.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.754+0000 7fd55a000640 1 -- 172.21.15.134:0/1270148108 --> [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd554023ea0 con 0x7fd5440bf450 2024-07-26T11:13:13.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.754+0000 7fd55a000640 1 -- 172.21.15.134:0/1270148108 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fd5540e4960 con 0x7fd55c10ba70 2024-07-26T11:13:13.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.754+0000 7fd561000640 1 --2- 172.21.15.134:0/1270148108 >> [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976] conn(0x7fd5440bf450 0x7fd5440c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:13.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.754+0000 7fd561a00640 1 --2- 172.21.15.134:0/1270148108 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd5440b3db0 0x7fd5440b62c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd55c07adc0 tx=0x7fd550009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:13.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.755+0000 7fd561000640 1 --2- 172.21.15.134:0/1270148108 >> [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976] conn(0x7fd5440bf450 0x7fd5440c1940 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:13.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.756+0000 7fd55a000640 1 -- 172.21.15.134:0/1270148108 <== osd.2 v2:172.21.15.193:6800/897516976 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd554023ea0 con 0x7fd5440bf450 2024-07-26T11:13:13.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.771+0000 7fd564400640 1 -- 172.21.15.134:0/1270148108 --> [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd55c0639b0 con 0x7fd5440bf450 2024-07-26T11:13:13.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.772+0000 7fd55a000640 1 -- 172.21.15.134:0/1270148108 <== osd.2 v2:172.21.15.193:6800/897516976 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd55c0639b0 con 0x7fd5440bf450 2024-07-26T11:13:13.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.772+0000 7fd564400640 1 -- 172.21.15.134:0/1270148108 >> [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976] conn(0x7fd5440bf450 msgr2=0x7fd5440c1940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:13.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.772+0000 7fd564400640 1 --2- 172.21.15.134:0/1270148108 >> [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976] conn(0x7fd5440bf450 0x7fd5440c1940 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:13.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.772+0000 7fd564400640 1 -- 172.21.15.134:0/1270148108 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd5440b3db0 msgr2=0x7fd5440b62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:13.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.772+0000 7fd564400640 1 --2- 172.21.15.134:0/1270148108 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd5440b3db0 0x7fd5440b62c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd55c07adc0 tx=0x7fd550009500 comp rx=0 tx=0).stop 2024-07-26T11:13:13.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.772+0000 7fd564400640 1 -- 172.21.15.134:0/1270148108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd55c10ba70 msgr2=0x7fd55c077a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:13.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.772+0000 7fd564400640 1 --2- 172.21.15.134:0/1270148108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd55c10ba70 0x7fd55c077a70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd55400e970 tx=0x7fd55400ee30 comp rx=0 tx=0).stop 2024-07-26T11:13:13.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.772+0000 7fd562400640 1 -- 172.21.15.134:0/1270148108 reap_dead start 2024-07-26T11:13:13.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.773+0000 7fd564400640 1 -- 172.21.15.134:0/1270148108 shutdown_connections 2024-07-26T11:13:13.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.773+0000 7fd564400640 1 -- 172.21.15.134:0/1270148108 >> 172.21.15.134:0/1270148108 conn(0x7fd55c1011f0 msgr2=0x7fd55c104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:13.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.773+0000 7fd564400640 1 -- 172.21.15.134:0/1270148108 shutdown_connections 2024-07-26T11:13:13.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:13.773+0000 7fd564400640 1 -- 172.21.15.134:0/1270148108 wait complete. 2024-07-26T11:13:14.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.369+0000 7ff0de600640 1 -- 172.21.15.134:0/2741161348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0d81099f0 msgr2=0x7ff0d8109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:14.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.369+0000 7ff0de600640 1 --2- 172.21.15.134:0/2741161348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0d81099f0 0x7ff0d8109e00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009900 tx=0x7ff0cc02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:13:14.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.369+0000 7ff0de600640 1 -- 172.21.15.134:0/2741161348 shutdown_connections 2024-07-26T11:13:14.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.369+0000 7ff0de600640 1 --2- 172.21.15.134:0/2741161348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0d810fc80 0x7ff0d81164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:14.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.369+0000 7ff0de600640 1 --2- 172.21.15.134:0/2741161348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0d810f330 0x7ff0d810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:14.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.369+0000 7ff0de600640 1 --2- 172.21.15.134:0/2741161348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0d81099f0 0x7ff0d8109e00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:14.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.369+0000 7ff0de600640 1 -- 172.21.15.134:0/2741161348 >> 172.21.15.134:0/2741161348 conn(0x7ff0d8078ff0 msgr2=0x7ff0d8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:14.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.369+0000 7ff0de600640 1 -- 172.21.15.134:0/2741161348 shutdown_connections 2024-07-26T11:13:14.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.370+0000 7ff0de600640 1 -- 172.21.15.134:0/2741161348 wait complete. 2024-07-26T11:13:14.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.371+0000 7ff0de600640 1 Processor -- start 2024-07-26T11:13:14.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.371+0000 7ff0de600640 1 -- start start 2024-07-26T11:13:14.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.371+0000 7ff0de600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0d81099f0 0x7ff0d81a0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:14.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.371+0000 7ff0de600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0d810f330 0x7ff0d81a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:14.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.371+0000 7ff0de600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0d810fc80 0x7ff0d81a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:14.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.372+0000 7ff0de600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d8119940 con 0x7ff0d81099f0 2024-07-26T11:13:14.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.372+0000 7ff0de600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d81197c0 con 0x7ff0d810f330 2024-07-26T11:13:14.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.372+0000 7ff0de600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d8119ac0 con 0x7ff0d810fc80 2024-07-26T11:13:14.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.372+0000 7ff0dd200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0d81099f0 0x7ff0d81a0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:14.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.372+0000 7ff0d7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0d810f330 0x7ff0d81a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:14.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.372+0000 7ff0dd200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0d81099f0 0x7ff0d81a0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50478/0 (socket says 172.21.15.134:50478) 2024-07-26T11:13:14.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.372+0000 7ff0dd200640 1 -- 172.21.15.134:0/2218137913 learned_addr learned my addr 172.21.15.134:0/2218137913 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:14.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.372+0000 7ff0ddc00640 1 --2- 172.21.15.134:0/2218137913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0d810fc80 0x7ff0d81a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:14.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.372+0000 7ff0d7e00640 1 -- 172.21.15.134:0/2218137913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0d810fc80 msgr2=0x7ff0d81a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:14.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.372+0000 7ff0d7e00640 1 --2- 172.21.15.134:0/2218137913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0d810fc80 0x7ff0d81a5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:14.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.372+0000 7ff0d7e00640 1 -- 172.21.15.134:0/2218137913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0d81099f0 msgr2=0x7ff0d81a0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:14.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.372+0000 7ff0d7e00640 1 --2- 172.21.15.134:0/2218137913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff0d81099f0 0x7ff0d81a0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:14.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.372+0000 7ff0d7e00640 1 -- 172.21.15.134:0/2218137913 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d81a62c0 con 0x7ff0d810f330 2024-07-26T11:13:14.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.373+0000 7ff0d7e00640 1 --2- 172.21.15.134:0/2218137913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0d810f330 0x7ff0d81a08f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff0d806c270 tx=0x7ff0c400ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:14.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.373+0000 7ff0d5600640 1 -- 172.21.15.134:0/2218137913 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4009e60 con 0x7ff0d810f330 2024-07-26T11:13:14.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.373+0000 7ff0de600640 1 -- 172.21.15.134:0/2218137913 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d81a65b0 con 0x7ff0d810f330 2024-07-26T11:13:14.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.373+0000 7ff0d5600640 1 -- 172.21.15.134:0/2218137913 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0c400ccc0 con 0x7ff0d810f330 2024-07-26T11:13:14.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.373+0000 7ff0d5600640 1 -- 172.21.15.134:0/2218137913 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c40187a0 con 0x7ff0d810f330 2024-07-26T11:13:14.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.373+0000 7ff0de600640 1 -- 172.21.15.134:0/2218137913 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d81acd10 con 0x7ff0d810f330 2024-07-26T11:13:14.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.374+0000 7ff0de600640 1 -- 172.21.15.134:0/2218137913 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff0d8063580 con 0x7ff0d810f330 2024-07-26T11:13:14.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.378+0000 7ff0d5600640 1 -- 172.21.15.134:0/2218137913 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7ff0c4010430 con 0x7ff0d810f330 2024-07-26T11:13:14.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.379+0000 7ff0d5600640 1 --2- 172.21.15.134:0/2218137913 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff0c00b3cb0 0x7ff0c00b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:14.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.379+0000 7ff0d5600640 1 -- 172.21.15.134:0/2218137913 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff0c40e4d20 con 0x7ff0d810f330 2024-07-26T11:13:14.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.379+0000 7ff0dd200640 1 --2- 172.21.15.134:0/2218137913 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff0c00b3cb0 0x7ff0c00b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:14.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.380+0000 7ff0d5600640 1 --2- 172.21.15.134:0/2218137913 >> [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858] conn(0x7ff0c00bf350 0x7ff0c00c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:14.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.380+0000 7ff0d5600640 1 -- 172.21.15.134:0/2218137913 --> [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff0c4015e80 con 0x7ff0c00bf350 2024-07-26T11:13:14.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.380+0000 7ff0d5600640 1 -- 172.21.15.134:0/2218137913 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7ff0c40e50d0 con 0x7ff0d810f330 2024-07-26T11:13:14.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.380+0000 7ff0ddc00640 1 --2- 172.21.15.134:0/2218137913 >> [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858] conn(0x7ff0c00bf350 0x7ff0c00c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:14.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.380+0000 7ff0dd200640 1 --2- 172.21.15.134:0/2218137913 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff0c00b3cb0 0x7ff0c00b61c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc031440 tx=0x7ff0cc0058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:14.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.381+0000 7ff0ddc00640 1 --2- 172.21.15.134:0/2218137913 >> [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858] conn(0x7ff0c00bf350 0x7ff0c00c1840 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:14.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.382+0000 7ff0d5600640 1 -- 172.21.15.134:0/2218137913 <== osd.10 v2:172.21.15.189:6824/431740858 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff0c4015e80 con 0x7ff0c00bf350 2024-07-26T11:13:14.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.397+0000 7ff0de600640 1 -- 172.21.15.134:0/2218137913 --> [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff0d8063a00 con 0x7ff0c00bf350 2024-07-26T11:13:14.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.398+0000 7ff0d5600640 1 -- 172.21.15.134:0/2218137913 <== osd.10 v2:172.21.15.189:6824/431740858 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff0d8063a00 con 0x7ff0c00bf350 2024-07-26T11:13:14.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.398+0000 7ff0de600640 1 -- 172.21.15.134:0/2218137913 >> [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858] conn(0x7ff0c00bf350 msgr2=0x7ff0c00c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:14.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.398+0000 7ff0de600640 1 --2- 172.21.15.134:0/2218137913 >> [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858] conn(0x7ff0c00bf350 0x7ff0c00c1840 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:14.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.399+0000 7ff0de600640 1 -- 172.21.15.134:0/2218137913 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff0c00b3cb0 msgr2=0x7ff0c00b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:14.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.399+0000 7ff0de600640 1 --2- 172.21.15.134:0/2218137913 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff0c00b3cb0 0x7ff0c00b61c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc031440 tx=0x7ff0cc0058d0 comp rx=0 tx=0).stop 2024-07-26T11:13:14.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.399+0000 7ff0de600640 1 -- 172.21.15.134:0/2218137913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0d810f330 msgr2=0x7ff0d81a08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:14.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.399+0000 7ff0de600640 1 --2- 172.21.15.134:0/2218137913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0d810f330 0x7ff0d81a08f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff0d806c270 tx=0x7ff0c400ed90 comp rx=0 tx=0).stop 2024-07-26T11:13:14.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.399+0000 7ff0ddc00640 1 -- 172.21.15.134:0/2218137913 reap_dead start 2024-07-26T11:13:14.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.399+0000 7ff0de600640 1 -- 172.21.15.134:0/2218137913 shutdown_connections 2024-07-26T11:13:14.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.399+0000 7ff0de600640 1 -- 172.21.15.134:0/2218137913 >> 172.21.15.134:0/2218137913 conn(0x7ff0d8078ff0 msgr2=0x7ff0d8107f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:14.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.399+0000 7ff0de600640 1 -- 172.21.15.134:0/2218137913 shutdown_connections 2024-07-26T11:13:14.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.399+0000 7ff0de600640 1 -- 172.21.15.134:0/2218137913 wait complete. 2024-07-26T11:13:14.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.735+0000 7f8463200640 1 -- 172.21.15.134:0/1212957293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84641022a0 msgr2=0x7f84641026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:14.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.735+0000 7f8463200640 1 --2- 172.21.15.134:0/1212957293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84641022a0 0x7f84641026b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8454009900 tx=0x7f845402a410 comp rx=0 tx=0).stop 2024-07-26T11:13:14.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.735+0000 7f8463200640 1 -- 172.21.15.134:0/1212957293 shutdown_connections 2024-07-26T11:13:14.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.735+0000 7f8463200640 1 --2- 172.21.15.134:0/1212957293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8464108530 0x7f846410ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:14.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.735+0000 7f8463200640 1 --2- 172.21.15.134:0/1212957293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8464107be0 0x7f8464107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:14.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.735+0000 7f8463200640 1 --2- 172.21.15.134:0/1212957293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f84641022a0 0x7f84641026b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:14.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.735+0000 7f8463200640 1 -- 172.21.15.134:0/1212957293 >> 172.21.15.134:0/1212957293 conn(0x7f84640fdcf0 msgr2=0x7f8464100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:14.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.735+0000 7f8463200640 1 -- 172.21.15.134:0/1212957293 shutdown_connections 2024-07-26T11:13:14.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.735+0000 7f8463200640 1 -- 172.21.15.134:0/1212957293 wait complete. 2024-07-26T11:13:14.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.737+0000 7f8463200640 1 Processor -- start 2024-07-26T11:13:14.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.737+0000 7f8463200640 1 -- start start 2024-07-26T11:13:14.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.737+0000 7f8463200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84641022a0 0x7f846419bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.737+0000 7f8463200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8464107be0 0x7f846419c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.737+0000 7f8463200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8464108530 0x7f84641a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.737+0000 7f8463200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8464112200 con 0x7f8464107be0 2024-07-26T11:13:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.737+0000 7f8463200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8464112080 con 0x7f84641022a0 2024-07-26T11:13:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.737+0000 7f8463200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8464112380 con 0x7f8464108530 2024-07-26T11:13:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.738+0000 7f8459400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8464107be0 0x7f846419c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.738+0000 7f8459400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8464107be0 0x7f846419c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50498/0 (socket says 172.21.15.134:50498) 2024-07-26T11:13:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.738+0000 7f8459400640 1 -- 172.21.15.134:0/1974105176 learned_addr learned my addr 172.21.15.134:0/1974105176 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:14.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.738+0000 7f8462800640 1 --2- 172.21.15.134:0/1974105176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8464108530 0x7f84641a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:14.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.738+0000 7f8461e00640 1 --2- 172.21.15.134:0/1974105176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84641022a0 0x7f846419bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:14.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.738+0000 7f8461e00640 1 -- 172.21.15.134:0/1974105176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8464108530 msgr2=0x7f84641a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:14.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.738+0000 7f8461e00640 1 --2- 172.21.15.134:0/1974105176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8464108530 0x7f84641a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:14.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.738+0000 7f8461e00640 1 -- 172.21.15.134:0/1974105176 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8464107be0 msgr2=0x7f846419c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:14.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.738+0000 7f8461e00640 1 --2- 172.21.15.134:0/1974105176 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8464107be0 0x7f846419c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:14.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.738+0000 7f8461e00640 1 -- 172.21.15.134:0/1974105176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84641a1ea0 con 0x7f84641022a0 2024-07-26T11:13:14.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.738+0000 7f8459400640 1 --2- 172.21.15.134:0/1974105176 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8464107be0 0x7f846419c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:14.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.739+0000 7f8461e00640 1 --2- 172.21.15.134:0/1974105176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84641022a0 0x7f846419bf30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f845402a930 tx=0x7f8454031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:14.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.739+0000 7f845b400640 1 -- 172.21.15.134:0/1974105176 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f845403f030 con 0x7f84641022a0 2024-07-26T11:13:14.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.739+0000 7f845b400640 1 -- 172.21.15.134:0/1974105176 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8454004490 con 0x7f84641022a0 2024-07-26T11:13:14.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.739+0000 7f8463200640 1 -- 172.21.15.134:0/1974105176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84641a2130 con 0x7f84641022a0 2024-07-26T11:13:14.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.739+0000 7f845b400640 1 -- 172.21.15.134:0/1974105176 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8454035070 con 0x7f84641022a0 2024-07-26T11:13:14.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.739+0000 7f8463200640 1 -- 172.21.15.134:0/1974105176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84641a88f0 con 0x7f84641022a0 2024-07-26T11:13:14.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.740+0000 7f8463200640 1 -- 172.21.15.134:0/1974105176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8464063540 con 0x7f84641022a0 2024-07-26T11:13:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.743+0000 7f845b400640 1 -- 172.21.15.134:0/1974105176 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f8454042030 con 0x7f84641022a0 2024-07-26T11:13:14.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.745+0000 7f845b400640 1 --2- 172.21.15.134:0/1974105176 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f84480b3d30 0x7f84480b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:14.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.745+0000 7f845b400640 1 -- 172.21.15.134:0/1974105176 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f8454104b80 con 0x7f84641022a0 2024-07-26T11:13:14.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.745+0000 7f8459400640 1 --2- 172.21.15.134:0/1974105176 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f84480b3d30 0x7f84480b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:14.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.745+0000 7f845b400640 1 --2- 172.21.15.134:0/1974105176 >> [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917] conn(0x7f84480bf3d0 0x7f84480c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:14.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.746+0000 7f845b400640 1 -- 172.21.15.134:0/1974105176 --> [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f84480c1f90 con 0x7f84480bf3d0 2024-07-26T11:13:14.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.746+0000 7f845b400640 1 -- 172.21.15.134:0/1974105176 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f8454104f30 con 0x7f84641022a0 2024-07-26T11:13:14.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.746+0000 7f8462800640 1 --2- 172.21.15.134:0/1974105176 >> [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917] conn(0x7f84480bf3d0 0x7f84480c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:14.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.746+0000 7f8459400640 1 --2- 172.21.15.134:0/1974105176 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f84480b3d30 0x7f84480b6240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8464106db0 tx=0x7f844c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:14.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.747+0000 7f8462800640 1 --2- 172.21.15.134:0/1974105176 >> [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917] conn(0x7f84480bf3d0 0x7f84480c18c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:14.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.748+0000 7f845b400640 1 -- 172.21.15.134:0/1974105176 <== osd.3 v2:172.21.15.134:6810/3457190917 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f84480c1f90 con 0x7f84480bf3d0 2024-07-26T11:13:14.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.763+0000 7f8463200640 1 -- 172.21.15.134:0/1974105176 --> [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8464063970 con 0x7f84480bf3d0 2024-07-26T11:13:14.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.764+0000 7f845b400640 1 -- 172.21.15.134:0/1974105176 <== osd.3 v2:172.21.15.134:6810/3457190917 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8464063970 con 0x7f84480bf3d0 2024-07-26T11:13:14.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.764+0000 7f8463200640 1 -- 172.21.15.134:0/1974105176 >> [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917] conn(0x7f84480bf3d0 msgr2=0x7f84480c18c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:14.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.764+0000 7f8463200640 1 --2- 172.21.15.134:0/1974105176 >> [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917] conn(0x7f84480bf3d0 0x7f84480c18c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:14.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.764+0000 7f8463200640 1 -- 172.21.15.134:0/1974105176 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f84480b3d30 msgr2=0x7f84480b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:14.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.764+0000 7f8463200640 1 --2- 172.21.15.134:0/1974105176 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f84480b3d30 0x7f84480b6240 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8464106db0 tx=0x7f844c005f90 comp rx=0 tx=0).stop 2024-07-26T11:13:14.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.764+0000 7f8463200640 1 -- 172.21.15.134:0/1974105176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84641022a0 msgr2=0x7f846419bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:14.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.764+0000 7f8463200640 1 --2- 172.21.15.134:0/1974105176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84641022a0 0x7f846419bf30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f845402a930 tx=0x7f8454031a20 comp rx=0 tx=0).stop 2024-07-26T11:13:14.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.764+0000 7f8462800640 1 -- 172.21.15.134:0/1974105176 reap_dead start 2024-07-26T11:13:14.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.765+0000 7f8463200640 1 -- 172.21.15.134:0/1974105176 shutdown_connections 2024-07-26T11:13:14.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.765+0000 7f8463200640 1 -- 172.21.15.134:0/1974105176 >> 172.21.15.134:0/1974105176 conn(0x7f84640fdcf0 msgr2=0x7f84641072d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:14.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.765+0000 7f8463200640 1 -- 172.21.15.134:0/1974105176 shutdown_connections 2024-07-26T11:13:14.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:14.765+0000 7f8463200640 1 -- 172.21.15.134:0/1974105176 wait complete. 2024-07-26T11:13:15.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.232+0000 7f0cf9a00640 1 -- 172.21.15.134:0/962114840 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cf4104d00 msgr2=0x7f0cf410d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:15.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.232+0000 7f0cf9a00640 1 --2- 172.21.15.134:0/962114840 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cf4104d00 0x7f0cf410d340 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0009900 tx=0x7f0ce002a400 comp rx=0 tx=0).stop 2024-07-26T11:13:15.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.233+0000 7f0cf9a00640 1 -- 172.21.15.134:0/962114840 shutdown_connections 2024-07-26T11:13:15.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.233+0000 7f0cf9a00640 1 --2- 172.21.15.134:0/962114840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cf410d880 0x7f0cf410fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:15.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.233+0000 7f0cf9a00640 1 --2- 172.21.15.134:0/962114840 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cf4104d00 0x7f0cf410d340 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:15.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.233+0000 7f0cf9a00640 1 --2- 172.21.15.134:0/962114840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0cf41042d0 0x7f0cf41046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:15.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.233+0000 7f0cf9a00640 1 -- 172.21.15.134:0/962114840 >> 172.21.15.134:0/962114840 conn(0x7f0cf4079160 msgr2=0x7f0cf4079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:15.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.233+0000 7f0cf9a00640 1 -- 172.21.15.134:0/962114840 shutdown_connections 2024-07-26T11:13:15.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.233+0000 7f0cf9a00640 1 -- 172.21.15.134:0/962114840 wait complete. 2024-07-26T11:13:15.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.234+0000 7f0cf9a00640 1 Processor -- start 2024-07-26T11:13:15.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.234+0000 7f0cf9a00640 1 -- start start 2024-07-26T11:13:15.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.235+0000 7f0cf9a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cf41042d0 0x7f0cf4199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:15.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.235+0000 7f0cf9a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cf4104d00 0x7f0cf419a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:15.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.235+0000 7f0cf9a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0cf410d880 0x7f0cf419f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:15.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.235+0000 7f0cf9a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf41134a0 con 0x7f0cf41042d0 2024-07-26T11:13:15.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.235+0000 7f0cf9a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf4113320 con 0x7f0cf4104d00 2024-07-26T11:13:15.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.235+0000 7f0cf9a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0cf4113620 con 0x7f0cf410d880 2024-07-26T11:13:15.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.235+0000 7f0cf2a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cf41042d0 0x7f0cf4199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:15.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.235+0000 7f0cf2000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cf4104d00 0x7f0cf419a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:15.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.235+0000 7f0cf2a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cf41042d0 0x7f0cf4199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50510/0 (socket says 172.21.15.134:50510) 2024-07-26T11:13:15.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.235+0000 7f0cf2a00640 1 -- 172.21.15.134:0/237875949 learned_addr learned my addr 172.21.15.134:0/237875949 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:15.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.235+0000 7f0cf3400640 1 --2- 172.21.15.134:0/237875949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0cf410d880 0x7f0cf419f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:15.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.236+0000 7f0cf2a00640 1 -- 172.21.15.134:0/237875949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0cf410d880 msgr2=0x7f0cf419f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:15.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.236+0000 7f0cf2a00640 1 --2- 172.21.15.134:0/237875949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0cf410d880 0x7f0cf419f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:15.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.236+0000 7f0cf2a00640 1 -- 172.21.15.134:0/237875949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cf4104d00 msgr2=0x7f0cf419a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:15.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.236+0000 7f0cf2a00640 1 --2- 172.21.15.134:0/237875949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cf4104d00 0x7f0cf419a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:15.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.236+0000 7f0cf2a00640 1 -- 172.21.15.134:0/237875949 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cf419fee0 con 0x7f0cf41042d0 2024-07-26T11:13:15.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.236+0000 7f0cf3400640 1 --2- 172.21.15.134:0/237875949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0cf410d880 0x7f0cf419f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:15.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.236+0000 7f0cf2a00640 1 --2- 172.21.15.134:0/237875949 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cf41042d0 0x7f0cf4199f70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0ce400e9c0 tx=0x7f0ce400edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:15.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.236+0000 7f0ceb400640 1 -- 172.21.15.134:0/237875949 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce400ccc0 con 0x7f0cf41042d0 2024-07-26T11:13:15.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.237+0000 7f0cf9a00640 1 -- 172.21.15.134:0/237875949 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cf41a01d0 con 0x7f0cf41042d0 2024-07-26T11:13:15.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.237+0000 7f0ceb400640 1 -- 172.21.15.134:0/237875949 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ce40035f0 con 0x7f0cf41042d0 2024-07-26T11:13:15.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.237+0000 7f0ceb400640 1 -- 172.21.15.134:0/237875949 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ce400ce60 con 0x7f0cf41042d0 2024-07-26T11:13:15.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.237+0000 7f0cf9a00640 1 -- 172.21.15.134:0/237875949 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cf41a6930 con 0x7f0cf41042d0 2024-07-26T11:13:15.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.238+0000 7f0cf9a00640 1 -- 172.21.15.134:0/237875949 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0cf4106d20 con 0x7f0cf41042d0 2024-07-26T11:13:15.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.240+0000 7f0ceb400640 1 -- 172.21.15.134:0/237875949 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f0ce40037a0 con 0x7f0cf41042d0 2024-07-26T11:13:15.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.242+0000 7f0ceb400640 1 --2- 172.21.15.134:0/237875949 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0cd80b3cb0 0x7f0cd80b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:15.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.242+0000 7f0ceb400640 1 -- 172.21.15.134:0/237875949 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0ce40e4960 con 0x7f0cf41042d0 2024-07-26T11:13:15.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.242+0000 7f0cf2000640 1 --2- 172.21.15.134:0/237875949 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0cd80b3cb0 0x7f0cd80b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:15.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.243+0000 7f0ceb400640 1 --2- 172.21.15.134:0/237875949 >> [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579] conn(0x7f0cd80bf350 0x7f0cd80c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:15.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.243+0000 7f0ceb400640 1 -- 172.21.15.134:0/237875949 --> [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ce4023e50 con 0x7f0cd80bf350 2024-07-26T11:13:15.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.243+0000 7f0ceb400640 1 -- 172.21.15.134:0/237875949 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f0ce40e4d10 con 0x7f0cf41042d0 2024-07-26T11:13:15.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.243+0000 7f0cf3400640 1 --2- 172.21.15.134:0/237875949 >> [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579] conn(0x7f0cd80bf350 0x7f0cd80c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:15.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.243+0000 7f0cf2000640 1 --2- 172.21.15.134:0/237875949 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0cd80b3cb0 0x7f0cd80b61c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0031440 tx=0x7f0ce0005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:15.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.244+0000 7f0cf3400640 1 --2- 172.21.15.134:0/237875949 >> [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579] conn(0x7f0cd80bf350 0x7f0cd80c1840 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:15.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.245+0000 7f0ceb400640 1 -- 172.21.15.134:0/237875949 <== osd.11 v2:172.21.15.193:6824/1594590579 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0ce4023e50 con 0x7f0cd80bf350 2024-07-26T11:13:15.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.260+0000 7f0cf9a00640 1 -- 172.21.15.134:0/237875949 --> [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0cf406aed0 con 0x7f0cd80bf350 2024-07-26T11:13:15.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.261+0000 7f0ceb400640 1 -- 172.21.15.134:0/237875949 <== osd.11 v2:172.21.15.193:6824/1594590579 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0cf406aed0 con 0x7f0cd80bf350 2024-07-26T11:13:15.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.261+0000 7f0cf9a00640 1 -- 172.21.15.134:0/237875949 >> [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579] conn(0x7f0cd80bf350 msgr2=0x7f0cd80c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:15.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.261+0000 7f0cf9a00640 1 --2- 172.21.15.134:0/237875949 >> [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579] conn(0x7f0cd80bf350 0x7f0cd80c1840 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:15.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.262+0000 7f0cf9a00640 1 -- 172.21.15.134:0/237875949 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0cd80b3cb0 msgr2=0x7f0cd80b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:15.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.262+0000 7f0cf9a00640 1 --2- 172.21.15.134:0/237875949 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0cd80b3cb0 0x7f0cd80b61c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0ce0031440 tx=0x7f0ce0005a20 comp rx=0 tx=0).stop 2024-07-26T11:13:15.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.262+0000 7f0cf9a00640 1 -- 172.21.15.134:0/237875949 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cf41042d0 msgr2=0x7f0cf4199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:15.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.262+0000 7f0cf9a00640 1 --2- 172.21.15.134:0/237875949 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0cf41042d0 0x7f0cf4199f70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0ce400e9c0 tx=0x7f0ce400edc0 comp rx=0 tx=0).stop 2024-07-26T11:13:15.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.262+0000 7f0cf3400640 1 -- 172.21.15.134:0/237875949 reap_dead start 2024-07-26T11:13:15.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.262+0000 7f0cf9a00640 1 -- 172.21.15.134:0/237875949 shutdown_connections 2024-07-26T11:13:15.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.262+0000 7f0cf9a00640 1 -- 172.21.15.134:0/237875949 >> 172.21.15.134:0/237875949 conn(0x7f0cf4079160 msgr2=0x7f0cf41062c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:15.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.262+0000 7f0cf9a00640 1 -- 172.21.15.134:0/237875949 shutdown_connections 2024-07-26T11:13:15.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:15.262+0000 7f0cf9a00640 1 -- 172.21.15.134:0/237875949 wait complete. 2024-07-26T11:13:15.287 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:15 smithi134 ceph-mon[31966]: pgmap v398: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-07-26T11:13:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:15 smithi189 ceph-mon[36790]: pgmap v398: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-07-26T11:13:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:15 smithi193 ceph-mon[37473]: pgmap v398: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-07-26T11:13:16.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.080+0000 7fe88f400640 1 -- 172.21.15.134:0/2295175413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8881057b0 msgr2=0x7fe888105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:16.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.080+0000 7fe88f400640 1 --2- 172.21.15.134:0/2295175413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8881057b0 0x7fe888105bc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009900 tx=0x7fe87c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:13:16.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.081+0000 7fe88f400640 1 -- 172.21.15.134:0/2295175413 shutdown_connections 2024-07-26T11:13:16.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.081+0000 7fe88f400640 1 --2- 172.21.15.134:0/2295175413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe88810ba40 0x7fe888112270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:16.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.081+0000 7fe88f400640 1 --2- 172.21.15.134:0/2295175413 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe88810b0f0 0x7fe88810b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:16.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.081+0000 7fe88f400640 1 --2- 172.21.15.134:0/2295175413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8881057b0 0x7fe888105bc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:16.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.081+0000 7fe88f400640 1 -- 172.21.15.134:0/2295175413 >> 172.21.15.134:0/2295175413 conn(0x7fe888101240 msgr2=0x7fe888103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:16.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.081+0000 7fe88f400640 1 -- 172.21.15.134:0/2295175413 shutdown_connections 2024-07-26T11:13:16.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.081+0000 7fe88f400640 1 -- 172.21.15.134:0/2295175413 wait complete. 2024-07-26T11:13:16.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.082+0000 7fe88f400640 1 Processor -- start 2024-07-26T11:13:16.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.082+0000 7fe88f400640 1 -- start start 2024-07-26T11:13:16.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.082+0000 7fe88f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8881057b0 0x7fe88819c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:16.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.082+0000 7fe88f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe88810b0f0 0x7fe88819c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:16.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.083+0000 7fe88f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe88810ba40 0x7fe8881a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:16.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.083+0000 7fe88f400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe888115710 con 0x7fe88810ba40 2024-07-26T11:13:16.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.083+0000 7fe88f400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe888115590 con 0x7fe8881057b0 2024-07-26T11:13:16.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.083+0000 7fe88f400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe888115890 con 0x7fe88810b0f0 2024-07-26T11:13:16.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.083+0000 7fe88ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8881057b0 0x7fe88819c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:16.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.083+0000 7fe88ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8881057b0 0x7fe88819c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:45884/0 (socket says 172.21.15.134:45884) 2024-07-26T11:13:16.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.083+0000 7fe887e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe88810b0f0 0x7fe88819c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:16.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.083+0000 7fe88ca00640 1 -- 172.21.15.134:0/3354461440 learned_addr learned my addr 172.21.15.134:0/3354461440 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:16.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.083+0000 7fe88d400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe88810ba40 0x7fe8881a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:16.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.083+0000 7fe88ca00640 1 -- 172.21.15.134:0/3354461440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe88810b0f0 msgr2=0x7fe88819c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:16.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.083+0000 7fe88ca00640 1 --2- 172.21.15.134:0/3354461440 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe88810b0f0 0x7fe88819c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:16.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.083+0000 7fe88ca00640 1 -- 172.21.15.134:0/3354461440 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe88810ba40 msgr2=0x7fe8881a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:16.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.083+0000 7fe88ca00640 1 --2- 172.21.15.134:0/3354461440 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe88810ba40 0x7fe8881a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:16.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.083+0000 7fe88ca00640 1 -- 172.21.15.134:0/3354461440 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8881a2070 con 0x7fe8881057b0 2024-07-26T11:13:16.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.084+0000 7fe88ca00640 1 --2- 172.21.15.134:0/3354461440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8881057b0 0x7fe88819c100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe87c005940 tx=0x7fe87c0040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:16.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.084+0000 7fe885600640 1 -- 172.21.15.134:0/3354461440 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c03f030 con 0x7fe8881057b0 2024-07-26T11:13:16.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.084+0000 7fe88f400640 1 -- 172.21.15.134:0/3354461440 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8881a2300 con 0x7fe8881057b0 2024-07-26T11:13:16.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.084+0000 7fe885600640 1 -- 172.21.15.134:0/3354461440 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe87c0044c0 con 0x7fe8881057b0 2024-07-26T11:13:16.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.084+0000 7fe885600640 1 -- 172.21.15.134:0/3354461440 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87c035070 con 0x7fe8881057b0 2024-07-26T11:13:16.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.084+0000 7fe88f400640 1 -- 172.21.15.134:0/3354461440 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8881a8ac0 con 0x7fe8881057b0 2024-07-26T11:13:16.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.085+0000 7fe88f400640 1 -- 172.21.15.134:0/3354461440 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe888063540 con 0x7fe8881057b0 2024-07-26T11:13:16.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.088+0000 7fe885600640 1 -- 172.21.15.134:0/3354461440 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fe87c042030 con 0x7fe8881057b0 2024-07-26T11:13:16.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.089+0000 7fe885600640 1 --2- 172.21.15.134:0/3354461440 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe8700b3cb0 0x7fe8700b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:16.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.090+0000 7fe885600640 1 -- 172.21.15.134:0/3354461440 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fe87c104cf0 con 0x7fe8881057b0 2024-07-26T11:13:16.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.090+0000 7fe887e00640 1 --2- 172.21.15.134:0/3354461440 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe8700b3cb0 0x7fe8700b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:16.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.090+0000 7fe885600640 1 --2- 172.21.15.134:0/3354461440 >> [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797] conn(0x7fe8700bf350 0x7fe8700c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:16.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.090+0000 7fe885600640 1 -- 172.21.15.134:0/3354461440 --> [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe87c03ce50 con 0x7fe8700bf350 2024-07-26T11:13:16.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.090+0000 7fe885600640 1 -- 172.21.15.134:0/3354461440 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fe87c1050a0 con 0x7fe8881057b0 2024-07-26T11:13:16.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.090+0000 7fe88d400640 1 --2- 172.21.15.134:0/3354461440 >> [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797] conn(0x7fe8700bf350 0x7fe8700c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:16.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.091+0000 7fe887e00640 1 --2- 172.21.15.134:0/3354461440 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe8700b3cb0 0x7fe8700b61c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe88810abc0 tx=0x7fe874005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:16.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.091+0000 7fe88d400640 1 --2- 172.21.15.134:0/3354461440 >> [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797] conn(0x7fe8700bf350 0x7fe8700c1840 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:16.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.092+0000 7fe885600640 1 -- 172.21.15.134:0/3354461440 <== osd.5 v2:172.21.15.193:6808/1092388797 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe87c03ce50 con 0x7fe8700bf350 2024-07-26T11:13:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.107+0000 7fe88f400640 1 -- 172.21.15.134:0/3354461440 --> [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe888063940 con 0x7fe8700bf350 2024-07-26T11:13:16.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.108+0000 7fe885600640 1 -- 172.21.15.134:0/3354461440 <== osd.5 v2:172.21.15.193:6808/1092388797 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe888063940 con 0x7fe8700bf350 2024-07-26T11:13:16.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.108+0000 7fe88f400640 1 -- 172.21.15.134:0/3354461440 >> [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797] conn(0x7fe8700bf350 msgr2=0x7fe8700c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:16.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.108+0000 7fe88f400640 1 --2- 172.21.15.134:0/3354461440 >> [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797] conn(0x7fe8700bf350 0x7fe8700c1840 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:16.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.109+0000 7fe88f400640 1 -- 172.21.15.134:0/3354461440 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe8700b3cb0 msgr2=0x7fe8700b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:16.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.109+0000 7fe88f400640 1 --2- 172.21.15.134:0/3354461440 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe8700b3cb0 0x7fe8700b61c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fe88810abc0 tx=0x7fe874005f90 comp rx=0 tx=0).stop 2024-07-26T11:13:16.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.109+0000 7fe88f400640 1 -- 172.21.15.134:0/3354461440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8881057b0 msgr2=0x7fe88819c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:16.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.109+0000 7fe88f400640 1 --2- 172.21.15.134:0/3354461440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8881057b0 0x7fe88819c100 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe87c005940 tx=0x7fe87c0040c0 comp rx=0 tx=0).stop 2024-07-26T11:13:16.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.109+0000 7fe88d400640 1 -- 172.21.15.134:0/3354461440 reap_dead start 2024-07-26T11:13:16.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.109+0000 7fe88f400640 1 -- 172.21.15.134:0/3354461440 shutdown_connections 2024-07-26T11:13:16.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.109+0000 7fe88f400640 1 -- 172.21.15.134:0/3354461440 >> 172.21.15.134:0/3354461440 conn(0x7fe888101240 msgr2=0x7fe888103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:16.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.109+0000 7fe88f400640 1 -- 172.21.15.134:0/3354461440 shutdown_connections 2024-07-26T11:13:16.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:16.109+0000 7fe88f400640 1 -- 172.21.15.134:0/3354461440 wait complete. 2024-07-26T11:13:16.914 INFO:teuthology.orchestra.run.smithi134.stdout:81604378724 2024-07-26T11:13:16.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-07-26T11:13:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:17 smithi134 ceph-mon[31966]: pgmap v399: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T11:13:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:17 smithi189 ceph-mon[36790]: pgmap v399: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T11:13:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:17 smithi193 ceph-mon[37473]: pgmap v399: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T11:13:18.237 INFO:teuthology.orchestra.run.smithi134.stdout:287762808887 2024-07-26T11:13:18.238 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-07-26T11:13:18.336 INFO:teuthology.orchestra.run.smithi134.stdout:240518168642 2024-07-26T11:13:18.336 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-07-26T11:13:18.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.913+0000 7fe861000640 1 -- 172.21.15.134:0/1847353302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe85c10f330 msgr2=0x7fe85c10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:18.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.913+0000 7fe861000640 1 --2- 172.21.15.134:0/1847353302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe85c10f330 0x7fe85c10f740 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe844009900 tx=0x7fe84402a410 comp rx=0 tx=0).stop 2024-07-26T11:13:18.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.913+0000 7fe861000640 1 -- 172.21.15.134:0/1847353302 shutdown_connections 2024-07-26T11:13:18.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.913+0000 7fe861000640 1 --2- 172.21.15.134:0/1847353302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe85c10fc80 0x7fe85c1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:18.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.913+0000 7fe861000640 1 --2- 172.21.15.134:0/1847353302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe85c10f330 0x7fe85c10f740 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:18.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.913+0000 7fe861000640 1 --2- 172.21.15.134:0/1847353302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe85c1099f0 0x7fe85c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:18.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.914+0000 7fe861000640 1 -- 172.21.15.134:0/1847353302 >> 172.21.15.134:0/1847353302 conn(0x7fe85c078ff0 msgr2=0x7fe85c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:18.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.914+0000 7fe861000640 1 -- 172.21.15.134:0/1847353302 shutdown_connections 2024-07-26T11:13:18.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.914+0000 7fe861000640 1 -- 172.21.15.134:0/1847353302 wait complete. 2024-07-26T11:13:18.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.915+0000 7fe861000640 1 Processor -- start 2024-07-26T11:13:18.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.915+0000 7fe861000640 1 -- start start 2024-07-26T11:13:18.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.915+0000 7fe861000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe85c1099f0 0x7fe85c1a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:18.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.915+0000 7fe861000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe85c10f330 0x7fe85c1a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:18.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.915+0000 7fe861000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe85c10fc80 0x7fe85c1a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:18.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.915+0000 7fe861000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe85c119940 con 0x7fe85c1099f0 2024-07-26T11:13:18.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.915+0000 7fe861000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe85c1197c0 con 0x7fe85c10fc80 2024-07-26T11:13:18.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.915+0000 7fe861000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe85c119ac0 con 0x7fe85c10f330 2024-07-26T11:13:18.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.916+0000 7fe85b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe85c1099f0 0x7fe85c1a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:18.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.916+0000 7fe852a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe85c10f330 0x7fe85c1a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:18.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.916+0000 7fe852a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe85c10f330 0x7fe85c1a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:50654/0 (socket says 172.21.15.134:50654) 2024-07-26T11:13:18.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.916+0000 7fe852a00640 1 -- 172.21.15.134:0/3648472828 learned_addr learned my addr 172.21.15.134:0/3648472828 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:18.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.916+0000 7fe85b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe85c1099f0 0x7fe85c1a0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50552/0 (socket says 172.21.15.134:50552) 2024-07-26T11:13:18.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.916+0000 7fe85be00640 1 --2- 172.21.15.134:0/3648472828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe85c10fc80 0x7fe85c1a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:18.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.916+0000 7fe852a00640 1 -- 172.21.15.134:0/3648472828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe85c10fc80 msgr2=0x7fe85c1a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:18.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.916+0000 7fe852a00640 1 --2- 172.21.15.134:0/3648472828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe85c10fc80 0x7fe85c1a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:18.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.916+0000 7fe852a00640 1 -- 172.21.15.134:0/3648472828 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe85c1099f0 msgr2=0x7fe85c1a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:18.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.916+0000 7fe852a00640 1 --2- 172.21.15.134:0/3648472828 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe85c1099f0 0x7fe85c1a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:18.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.916+0000 7fe852a00640 1 -- 172.21.15.134:0/3648472828 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe85c1a6280 con 0x7fe85c10f330 2024-07-26T11:13:18.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.916+0000 7fe85b400640 1 --2- 172.21.15.134:0/3648472828 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe85c1099f0 0x7fe85c1a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:18.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.917+0000 7fe852a00640 1 --2- 172.21.15.134:0/3648472828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe85c10f330 0x7fe85c1a08b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe84402a930 tx=0x7fe844031a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:18.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.917+0000 7fe858c00640 1 -- 172.21.15.134:0/3648472828 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84403f030 con 0x7fe85c10f330 2024-07-26T11:13:18.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.917+0000 7fe858c00640 1 -- 172.21.15.134:0/3648472828 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe844004490 con 0x7fe85c10f330 2024-07-26T11:13:18.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.917+0000 7fe861000640 1 -- 172.21.15.134:0/3648472828 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe85c1a6510 con 0x7fe85c10f330 2024-07-26T11:13:18.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.917+0000 7fe858c00640 1 -- 172.21.15.134:0/3648472828 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe844035070 con 0x7fe85c10f330 2024-07-26T11:13:18.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.917+0000 7fe861000640 1 -- 172.21.15.134:0/3648472828 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe85c1accd0 con 0x7fe85c10f330 2024-07-26T11:13:18.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.918+0000 7fe861000640 1 -- 172.21.15.134:0/3648472828 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe85c063580 con 0x7fe85c10f330 2024-07-26T11:13:18.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.921+0000 7fe858c00640 1 -- 172.21.15.134:0/3648472828 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7fe844042030 con 0x7fe85c10f330 2024-07-26T11:13:18.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.922+0000 7fe858c00640 1 --2- 172.21.15.134:0/3648472828 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe8400b3cb0 0x7fe8400b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:18.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.923+0000 7fe858c00640 1 -- 172.21.15.134:0/3648472828 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fe844104a20 con 0x7fe85c10f330 2024-07-26T11:13:18.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.923+0000 7fe85b400640 1 --2- 172.21.15.134:0/3648472828 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe8400b3cb0 0x7fe8400b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:18.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.923+0000 7fe858c00640 1 --2- 172.21.15.134:0/3648472828 >> [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954] conn(0x7fe8400bf350 0x7fe8400c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:18.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.923+0000 7fe858c00640 1 -- 172.21.15.134:0/3648472828 --> [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe8400c1f10 con 0x7fe8400bf350 2024-07-26T11:13:18.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.923+0000 7fe858c00640 1 -- 172.21.15.134:0/3648472828 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7fe844104dd0 con 0x7fe85c10f330 2024-07-26T11:13:18.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.923+0000 7fe85be00640 1 --2- 172.21.15.134:0/3648472828 >> [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954] conn(0x7fe8400bf350 0x7fe8400c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:18.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.924+0000 7fe85b400640 1 --2- 172.21.15.134:0/3648472828 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe8400b3cb0 0x7fe8400b61c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe84c005fd0 tx=0x7fe84c009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:18.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.924+0000 7fe85be00640 1 --2- 172.21.15.134:0/3648472828 >> [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954] conn(0x7fe8400bf350 0x7fe8400c1840 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:18.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.925+0000 7fe858c00640 1 -- 172.21.15.134:0/3648472828 <== osd.0 v2:172.21.15.134:6802/4183273954 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe8400c1f10 con 0x7fe8400bf350 2024-07-26T11:13:18.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.940+0000 7fe861000640 1 -- 172.21.15.134:0/3648472828 --> [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe85c063a00 con 0x7fe8400bf350 2024-07-26T11:13:18.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.941+0000 7fe858c00640 1 -- 172.21.15.134:0/3648472828 <== osd.0 v2:172.21.15.134:6802/4183273954 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe85c063a00 con 0x7fe8400bf350 2024-07-26T11:13:18.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.941+0000 7fe861000640 1 -- 172.21.15.134:0/3648472828 >> [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954] conn(0x7fe8400bf350 msgr2=0x7fe8400c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:18.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.941+0000 7fe861000640 1 --2- 172.21.15.134:0/3648472828 >> [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954] conn(0x7fe8400bf350 0x7fe8400c1840 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:18.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.941+0000 7fe861000640 1 -- 172.21.15.134:0/3648472828 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe8400b3cb0 msgr2=0x7fe8400b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:18.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.941+0000 7fe861000640 1 --2- 172.21.15.134:0/3648472828 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe8400b3cb0 0x7fe8400b61c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe84c005fd0 tx=0x7fe84c009500 comp rx=0 tx=0).stop 2024-07-26T11:13:18.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.941+0000 7fe861000640 1 -- 172.21.15.134:0/3648472828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe85c10f330 msgr2=0x7fe85c1a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:18.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.941+0000 7fe861000640 1 --2- 172.21.15.134:0/3648472828 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe85c10f330 0x7fe85c1a08b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe84402a930 tx=0x7fe844031a20 comp rx=0 tx=0).stop 2024-07-26T11:13:18.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.941+0000 7fe85be00640 1 -- 172.21.15.134:0/3648472828 reap_dead start 2024-07-26T11:13:18.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.942+0000 7fe861000640 1 -- 172.21.15.134:0/3648472828 shutdown_connections 2024-07-26T11:13:18.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.942+0000 7fe861000640 1 -- 172.21.15.134:0/3648472828 >> 172.21.15.134:0/3648472828 conn(0x7fe85c078ff0 msgr2=0x7fe85c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:18.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.942+0000 7fe861000640 1 -- 172.21.15.134:0/3648472828 shutdown_connections 2024-07-26T11:13:18.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:18.942+0000 7fe861000640 1 -- 172.21.15.134:0/3648472828 wait complete. 2024-07-26T11:13:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:19 smithi134 ceph-mon[31966]: pgmap v400: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:13:19.344 INFO:teuthology.orchestra.run.smithi134.stdout:120259084383 2024-07-26T11:13:19.344 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-07-26T11:13:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:19 smithi189 ceph-mon[36790]: pgmap v400: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:13:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:19 smithi193 ceph-mon[37473]: pgmap v400: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:13:20.355 INFO:teuthology.orchestra.run.smithi134.stdout:313532612659 2024-07-26T11:13:20.355 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-07-26T11:13:20.759 INFO:teuthology.orchestra.run.smithi134.stdout:146028888152 2024-07-26T11:13:20.759 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-07-26T11:13:20.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.372+0000 7f6f01c00640 1 -- 172.21.15.134:0/3768580233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6efc10fb50 msgr2=0x7f6efc111f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:20.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.372+0000 7f6f01c00640 1 --2- 172.21.15.134:0/3768580233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6efc10fb50 0x7f6efc111f70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009ca0 tx=0x7f6ef402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:13:20.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.372+0000 7f6f01c00640 1 -- 172.21.15.134:0/3768580233 shutdown_connections 2024-07-26T11:13:20.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.372+0000 7f6f01c00640 1 --2- 172.21.15.134:0/3768580233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6efc10fb50 0x7f6efc111f70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:20.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.372+0000 7f6f01c00640 1 --2- 172.21.15.134:0/3768580233 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6efc075b30 0x7f6efc10f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:20.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.372+0000 7f6f01c00640 1 --2- 172.21.15.134:0/3768580233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6efc0751e0 0x7f6efc0755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:20.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.372+0000 7f6f01c00640 1 -- 172.21.15.134:0/3768580233 >> 172.21.15.134:0/3768580233 conn(0x7f6efc07b340 msgr2=0x7f6efc07b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:20.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.372+0000 7f6f01c00640 1 -- 172.21.15.134:0/3768580233 shutdown_connections 2024-07-26T11:13:20.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.372+0000 7f6f01c00640 1 -- 172.21.15.134:0/3768580233 wait complete. 2024-07-26T11:13:20.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.373+0000 7f6f01c00640 1 Processor -- start 2024-07-26T11:13:20.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.374+0000 7f6f01c00640 1 -- start start 2024-07-26T11:13:20.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.374+0000 7f6f01c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6efc0751e0 0x7f6efc19c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:20.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.374+0000 7f6f01c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6efc075b30 0x7f6efc19c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:20.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.374+0000 7f6f01c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6efc10fb50 0x7f6efc1a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:20.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.374+0000 7f6f01c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc1156a0 con 0x7f6efc0751e0 2024-07-26T11:13:20.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.374+0000 7f6f01c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc115520 con 0x7f6efc075b30 2024-07-26T11:13:20.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.374+0000 7f6f01c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6efc115820 con 0x7f6efc10fb50 2024-07-26T11:13:20.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.374+0000 7f6efaa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6efc0751e0 0x7f6efc19c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:20.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.374+0000 7f6efaa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6efc0751e0 0x7f6efc19c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:43650/0 (socket says 172.21.15.134:43650) 2024-07-26T11:13:20.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.374+0000 7f6efaa00640 1 -- 172.21.15.134:0/1851431233 learned_addr learned my addr 172.21.15.134:0/1851431233 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:20.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.374+0000 7f6efb400640 1 --2- 172.21.15.134:0/1851431233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6efc10fb50 0x7f6efc1a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:20.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.375+0000 7f6efa000640 1 --2- 172.21.15.134:0/1851431233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6efc075b30 0x7f6efc19c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:20.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.375+0000 7f6efaa00640 1 -- 172.21.15.134:0/1851431233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6efc10fb50 msgr2=0x7f6efc1a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:20.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.375+0000 7f6efaa00640 1 --2- 172.21.15.134:0/1851431233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6efc10fb50 0x7f6efc1a1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:20.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.375+0000 7f6efaa00640 1 -- 172.21.15.134:0/1851431233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6efc075b30 msgr2=0x7f6efc19c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:20.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.375+0000 7f6efaa00640 1 --2- 172.21.15.134:0/1851431233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6efc075b30 0x7f6efc19c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:20.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.375+0000 7f6efaa00640 1 -- 172.21.15.134:0/1851431233 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6efc1a1fc0 con 0x7f6efc0751e0 2024-07-26T11:13:20.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.375+0000 7f6efaa00640 1 --2- 172.21.15.134:0/1851431233 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6efc0751e0 0x7f6efc19c050 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6eec00e9f0 tx=0x7f6eec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:20.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.376+0000 7f6ef3400640 1 -- 172.21.15.134:0/1851431233 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec00cd80 con 0x7f6efc0751e0 2024-07-26T11:13:20.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.376+0000 7f6f01c00640 1 -- 172.21.15.134:0/1851431233 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6efc1a22b0 con 0x7f6efc0751e0 2024-07-26T11:13:20.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.376+0000 7f6ef3400640 1 -- 172.21.15.134:0/1851431233 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6eec0054d0 con 0x7f6efc0751e0 2024-07-26T11:13:20.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.376+0000 7f6ef3400640 1 -- 172.21.15.134:0/1851431233 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec010690 con 0x7f6efc0751e0 2024-07-26T11:13:20.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.376+0000 7f6f01c00640 1 -- 172.21.15.134:0/1851431233 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6efc1a8a10 con 0x7f6efc0751e0 2024-07-26T11:13:20.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.377+0000 7f6f01c00640 1 -- 172.21.15.134:0/1851431233 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6efc10a080 con 0x7f6efc0751e0 2024-07-26T11:13:20.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.380+0000 7f6ef3400640 1 -- 172.21.15.134:0/1851431233 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f6eec004e30 con 0x7f6efc0751e0 2024-07-26T11:13:20.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.381+0000 7f6ef3400640 1 --2- 172.21.15.134:0/1851431233 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6ee00b3cb0 0x7f6ee00b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:20.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.381+0000 7f6ef3400640 1 -- 172.21.15.134:0/1851431233 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f6eec0e4b00 con 0x7f6efc0751e0 2024-07-26T11:13:20.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.382+0000 7f6efa000640 1 --2- 172.21.15.134:0/1851431233 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6ee00b3cb0 0x7f6ee00b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:20.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.382+0000 7f6ef3400640 1 --2- 172.21.15.134:0/1851431233 >> [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911] conn(0x7f6ee00bf350 0x7f6ee00c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:20.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.382+0000 7f6ef3400640 1 -- 172.21.15.134:0/1851431233 --> [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6eec015e80 con 0x7f6ee00bf350 2024-07-26T11:13:20.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.382+0000 7f6ef3400640 1 -- 172.21.15.134:0/1851431233 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f6eec0e4eb0 con 0x7f6efc0751e0 2024-07-26T11:13:20.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.382+0000 7f6efb400640 1 --2- 172.21.15.134:0/1851431233 >> [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911] conn(0x7f6ee00bf350 0x7f6ee00c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:20.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.382+0000 7f6efa000640 1 --2- 172.21.15.134:0/1851431233 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6ee00b3cb0 0x7f6ee00b61c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6efc111890 tx=0x7f6ee8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:20.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.383+0000 7f6efb400640 1 --2- 172.21.15.134:0/1851431233 >> [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911] conn(0x7f6ee00bf350 0x7f6ee00c1840 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:20.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.384+0000 7f6ef3400640 1 -- 172.21.15.134:0/1851431233 <== osd.7 v2:172.21.15.189:6816/1382694911 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6eec015e80 con 0x7f6ee00bf350 2024-07-26T11:13:20.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.399+0000 7f6f01c00640 1 -- 172.21.15.134:0/1851431233 --> [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6efc0766a0 con 0x7f6ee00bf350 2024-07-26T11:13:20.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.400+0000 7f6ef3400640 1 -- 172.21.15.134:0/1851431233 <== osd.7 v2:172.21.15.189:6816/1382694911 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6efc0766a0 con 0x7f6ee00bf350 2024-07-26T11:13:20.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.400+0000 7f6f01c00640 1 -- 172.21.15.134:0/1851431233 >> [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911] conn(0x7f6ee00bf350 msgr2=0x7f6ee00c1840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:20.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.400+0000 7f6f01c00640 1 --2- 172.21.15.134:0/1851431233 >> [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911] conn(0x7f6ee00bf350 0x7f6ee00c1840 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:20.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.401+0000 7f6f01c00640 1 -- 172.21.15.134:0/1851431233 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6ee00b3cb0 msgr2=0x7f6ee00b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:20.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.401+0000 7f6f01c00640 1 --2- 172.21.15.134:0/1851431233 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6ee00b3cb0 0x7f6ee00b61c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6efc111890 tx=0x7f6ee8005f90 comp rx=0 tx=0).stop 2024-07-26T11:13:20.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.401+0000 7f6f01c00640 1 -- 172.21.15.134:0/1851431233 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6efc0751e0 msgr2=0x7f6efc19c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:20.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.401+0000 7f6f01c00640 1 --2- 172.21.15.134:0/1851431233 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6efc0751e0 0x7f6efc19c050 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6eec00e9f0 tx=0x7f6eec00edf0 comp rx=0 tx=0).stop 2024-07-26T11:13:20.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.401+0000 7f6efb400640 1 -- 172.21.15.134:0/1851431233 reap_dead start 2024-07-26T11:13:20.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.401+0000 7f6f01c00640 1 -- 172.21.15.134:0/1851431233 shutdown_connections 2024-07-26T11:13:20.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.401+0000 7f6f01c00640 1 -- 172.21.15.134:0/1851431233 >> 172.21.15.134:0/1851431233 conn(0x7f6efc07b340 msgr2=0x7f6efc109660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:20.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.401+0000 7f6f01c00640 1 -- 172.21.15.134:0/1851431233 shutdown_connections 2024-07-26T11:13:20.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:20.401+0000 7f6f01c00640 1 -- 172.21.15.134:0/1851431233 wait complete. 2024-07-26T11:13:21.262 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:21 smithi134 ceph-mon[31966]: pgmap v401: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:21.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:21 smithi193 ceph-mon[37473]: pgmap v401: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:21.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:21 smithi189 ceph-mon[36790]: pgmap v401: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:22.042 INFO:teuthology.orchestra.run.smithi134.stdout:214748364873 2024-07-26T11:13:22.043 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-07-26T11:13:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.322+0000 7f8396000640 1 -- 172.21.15.134:0/1995416013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8390076db0 msgr2=0x7f83900751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.322+0000 7f8396000640 1 --2- 172.21.15.134:0/1995416013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8390076db0 0x7f83900751e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8380009900 tx=0x7f838002a400 comp rx=0 tx=0).stop 2024-07-26T11:13:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.322+0000 7f8396000640 1 -- 172.21.15.134:0/1995416013 shutdown_connections 2024-07-26T11:13:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.322+0000 7f8396000640 1 --2- 172.21.15.134:0/1995416013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8390113e80 0x7f8390116350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.322+0000 7f8396000640 1 --2- 172.21.15.134:0/1995416013 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8390075800 0x7f8390075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.322+0000 7f8396000640 1 --2- 172.21.15.134:0/1995416013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8390076db0 0x7f83900751e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.322+0000 7f8396000640 1 -- 172.21.15.134:0/1995416013 >> 172.21.15.134:0/1995416013 conn(0x7f83900ff040 msgr2=0x7f8390101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.323+0000 7f8396000640 1 -- 172.21.15.134:0/1995416013 shutdown_connections 2024-07-26T11:13:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.323+0000 7f8396000640 1 -- 172.21.15.134:0/1995416013 wait complete. 2024-07-26T11:13:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.325+0000 7f8396000640 1 Processor -- start 2024-07-26T11:13:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.325+0000 7f8396000640 1 -- start start 2024-07-26T11:13:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.325+0000 7f8396000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8390075800 0x7f83901a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:22.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.325+0000 7f8396000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8390076db0 0x7f83901a0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:22.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.326+0000 7f8396000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8390113e80 0x7f83901a5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:22.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.326+0000 7f8396000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8390119b10 con 0x7f8390075800 2024-07-26T11:13:22.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.326+0000 7f8396000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8390119990 con 0x7f8390113e80 2024-07-26T11:13:22.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.326+0000 7f8396000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8390119c90 con 0x7f8390076db0 2024-07-26T11:13:22.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.326+0000 7f838f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8390075800 0x7f83901a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:22.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.326+0000 7f838f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8390075800 0x7f83901a0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:43666/0 (socket says 172.21.15.134:43666) 2024-07-26T11:13:22.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.326+0000 7f838f400640 1 -- 172.21.15.134:0/1798558573 learned_addr learned my addr 172.21.15.134:0/1798558573 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:22.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.326+0000 7f838ea00640 1 --2- 172.21.15.134:0/1798558573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8390076db0 0x7f83901a0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:22.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.326+0000 7f838fe00640 1 --2- 172.21.15.134:0/1798558573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8390113e80 0x7f83901a5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.327+0000 7f838fe00640 1 -- 172.21.15.134:0/1798558573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8390076db0 msgr2=0x7f83901a0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.327+0000 7f838fe00640 1 --2- 172.21.15.134:0/1798558573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8390076db0 0x7f83901a0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.327+0000 7f838fe00640 1 -- 172.21.15.134:0/1798558573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8390075800 msgr2=0x7f83901a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.327+0000 7f838fe00640 1 --2- 172.21.15.134:0/1798558573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8390075800 0x7f83901a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.327+0000 7f838fe00640 1 -- 172.21.15.134:0/1798558573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83901a63f0 con 0x7f8390113e80 2024-07-26T11:13:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.327+0000 7f838ea00640 1 --2- 172.21.15.134:0/1798558573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8390076db0 0x7f83901a0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.327+0000 7f838f400640 1 --2- 172.21.15.134:0/1798558573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8390075800 0x7f83901a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.328+0000 7f838fe00640 1 --2- 172.21.15.134:0/1798558573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8390113e80 0x7f83901a5cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8388009fd0 tx=0x7f838800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.328+0000 7f8387e00640 1 -- 172.21.15.134:0/1798558573 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838800ce20 con 0x7f8390113e80 2024-07-26T11:13:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.328+0000 7f8387e00640 1 -- 172.21.15.134:0/1798558573 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8388005020 con 0x7f8390113e80 2024-07-26T11:13:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.328+0000 7f8387e00640 1 -- 172.21.15.134:0/1798558573 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8388010690 con 0x7f8390113e80 2024-07-26T11:13:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.328+0000 7f8396000640 1 -- 172.21.15.134:0/1798558573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83901a66e0 con 0x7f8390113e80 2024-07-26T11:13:22.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.328+0000 7f8396000640 1 -- 172.21.15.134:0/1798558573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83901ace40 con 0x7f8390113e80 2024-07-26T11:13:22.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.329+0000 7f8396000640 1 -- 172.21.15.134:0/1798558573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8390063540 con 0x7f8390113e80 2024-07-26T11:13:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.332+0000 7f8387e00640 1 -- 172.21.15.134:0/1798558573 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f83880054d0 con 0x7f8390113e80 2024-07-26T11:13:22.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.334+0000 7f8387e00640 1 --2- 172.21.15.134:0/1798558573 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f83740b3d30 0x7f83740b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:22.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.334+0000 7f8387e00640 1 -- 172.21.15.134:0/1798558573 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f83880e4910 con 0x7f8390113e80 2024-07-26T11:13:22.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.334+0000 7f8387e00640 1 --2- 172.21.15.134:0/1798558573 >> [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025] conn(0x7f83740bf3d0 0x7f83740c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:22.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.334+0000 7f8387e00640 1 -- 172.21.15.134:0/1798558573 --> [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f83740c1f90 con 0x7f83740bf3d0 2024-07-26T11:13:22.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.334+0000 7f8387e00640 1 -- 172.21.15.134:0/1798558573 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) ==== 24+0+0 (secure 0 0 0) 0x7f83880e92a0 con 0x7f8390113e80 2024-07-26T11:13:22.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.335+0000 7f838f400640 1 --2- 172.21.15.134:0/1798558573 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f83740b3d30 0x7f83740b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:22.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.335+0000 7f838ea00640 1 --2- 172.21.15.134:0/1798558573 >> [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025] conn(0x7f83740bf3d0 0x7f83740c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:22.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.336+0000 7f838f400640 1 --2- 172.21.15.134:0/1798558573 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f83740b3d30 0x7f83740b6240 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f83800096c0 tx=0x7f8380005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:22.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.336+0000 7f838ea00640 1 --2- 172.21.15.134:0/1798558573 >> [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025] conn(0x7f83740bf3d0 0x7f83740c18c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:22.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.337+0000 7f8387e00640 1 -- 172.21.15.134:0/1798558573 <== osd.9 v2:172.21.15.134:6826/511237025 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f83740c1f90 con 0x7f83740bf3d0 2024-07-26T11:13:22.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.354+0000 7f8396000640 1 -- 172.21.15.134:0/1798558573 --> [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f83900639c0 con 0x7f83740bf3d0 2024-07-26T11:13:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.355+0000 7f8387e00640 1 -- 172.21.15.134:0/1798558573 <== osd.9 v2:172.21.15.134:6826/511237025 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f83900639c0 con 0x7f83740bf3d0 2024-07-26T11:13:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.355+0000 7f8396000640 1 -- 172.21.15.134:0/1798558573 >> [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025] conn(0x7f83740bf3d0 msgr2=0x7f83740c18c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.355+0000 7f8396000640 1 --2- 172.21.15.134:0/1798558573 >> [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025] conn(0x7f83740bf3d0 0x7f83740c18c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.356+0000 7f8396000640 1 -- 172.21.15.134:0/1798558573 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f83740b3d30 msgr2=0x7f83740b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.356+0000 7f8396000640 1 --2- 172.21.15.134:0/1798558573 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f83740b3d30 0x7f83740b6240 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f83800096c0 tx=0x7f8380005a20 comp rx=0 tx=0).stop 2024-07-26T11:13:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.356+0000 7f8396000640 1 -- 172.21.15.134:0/1798558573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8390113e80 msgr2=0x7f83901a5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.356+0000 7f8396000640 1 --2- 172.21.15.134:0/1798558573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8390113e80 0x7f83901a5cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8388009fd0 tx=0x7f838800ef10 comp rx=0 tx=0).stop 2024-07-26T11:13:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.356+0000 7f838fe00640 1 -- 172.21.15.134:0/1798558573 reap_dead start 2024-07-26T11:13:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.356+0000 7f8396000640 1 -- 172.21.15.134:0/1798558573 shutdown_connections 2024-07-26T11:13:22.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.356+0000 7f8396000640 1 -- 172.21.15.134:0/1798558573 >> 172.21.15.134:0/1798558573 conn(0x7f83900ff040 msgr2=0x7f839010fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:22.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.356+0000 7f8396000640 1 -- 172.21.15.134:0/1798558573 shutdown_connections 2024-07-26T11:13:22.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:22.356+0000 7f8396000640 1 -- 172.21.15.134:0/1798558573 wait complete. 2024-07-26T11:13:23.001 INFO:teuthology.orchestra.run.smithi134.stdout:60129542249 2024-07-26T11:13:23.001 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-07-26T11:13:23.142 INFO:teuthology.orchestra.run.smithi134.stdout:167503724630 2024-07-26T11:13:23.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-07-26T11:13:23.198 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:23 smithi134 ceph-mon[31966]: pgmap v402: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:23.198 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:13:23.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:23 smithi189 ceph-mon[36790]: pgmap v402: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:23.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:13:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:23 smithi193 ceph-mon[37473]: pgmap v402: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:13:23.888 INFO:teuthology.orchestra.run.smithi134.stdout:266287972415 2024-07-26T11:13:23.889 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-07-26T11:13:24.137 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:13:24.137 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:13:24.137 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:13:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:13:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:13:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:13:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:13:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:13:24.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:13:24.718 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:13:24.796 INFO:teuthology.orchestra.run.smithi134.stdout:38654705774 2024-07-26T11:13:24.797 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-07-26T11:13:24.867 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:13:24.930 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:13:24.988 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:13:24.988 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:13:25.239 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:25 smithi134 ceph-mon[31966]: pgmap v403: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-07-26T11:13:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:25 smithi189 ceph-mon[36790]: pgmap v403: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-07-26T11:13:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:25 smithi193 ceph-mon[37473]: pgmap v403: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-07-26T11:13:25.538 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:13:25.775 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:13:25.777 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:13:27.152 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:27 smithi134 ceph-mon[31966]: pgmap v404: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:27 smithi189 ceph-mon[36790]: pgmap v404: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:27 smithi193 ceph-mon[37473]: pgmap v404: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:13:27.620 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:13:29.326 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:29 smithi134 ceph-mon[31966]: pgmap v405: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T11:13:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:29 smithi189 ceph-mon[36790]: pgmap v405: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T11:13:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:29 smithi193 ceph-mon[37473]: pgmap v405: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T11:13:30.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:30 smithi134 ceph-mon[31966]: pgmap v406: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:30 smithi189 ceph-mon[36790]: pgmap v406: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:30 smithi193 ceph-mon[37473]: pgmap v406: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:33.170 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:33 smithi134 ceph-mon[31966]: pgmap v407: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:33.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:33 smithi189 ceph-mon[36790]: pgmap v407: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:33 smithi193 ceph-mon[37473]: pgmap v407: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:33.678 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:13:33.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.531+0000 7f3c25200640 1 -- 172.21.15.134:0/4131627655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c2010fc80 msgr2=0x7f3c201164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:33.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.531+0000 7f3c25200640 1 --2- 172.21.15.134:0/4131627655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c2010fc80 0x7f3c201164b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3c18009be0 tx=0x7f3c1802a6c0 comp rx=0 tx=0).stop 2024-07-26T11:13:33.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.532+0000 7f3c25200640 1 -- 172.21.15.134:0/4131627655 shutdown_connections 2024-07-26T11:13:33.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.532+0000 7f3c25200640 1 --2- 172.21.15.134:0/4131627655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c2010fc80 0x7f3c201164b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:33.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.532+0000 7f3c25200640 1 --2- 172.21.15.134:0/4131627655 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c2010f330 0x7f3c2010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:33.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.532+0000 7f3c25200640 1 --2- 172.21.15.134:0/4131627655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c201099f0 0x7f3c20109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:33.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.532+0000 7f3c25200640 1 -- 172.21.15.134:0/4131627655 >> 172.21.15.134:0/4131627655 conn(0x7f3c20078ff0 msgr2=0x7f3c20107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:33.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.532+0000 7f3c25200640 1 -- 172.21.15.134:0/4131627655 shutdown_connections 2024-07-26T11:13:33.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.532+0000 7f3c25200640 1 -- 172.21.15.134:0/4131627655 wait complete. 2024-07-26T11:13:33.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.534+0000 7f3c25200640 1 Processor -- start 2024-07-26T11:13:33.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.534+0000 7f3c25200640 1 -- start start 2024-07-26T11:13:33.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c25200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c201099f0 0x7f3c201a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:33.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c25200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c2010f330 0x7f3c201a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:33.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c25200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2010fc80 0x7f3c201a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:33.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c25200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3c20119940 con 0x7f3c2010f330 2024-07-26T11:13:33.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c25200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3c201197c0 con 0x7f3c2010fc80 2024-07-26T11:13:33.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c25200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3c20119ac0 con 0x7f3c201099f0 2024-07-26T11:13:33.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c1ea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c2010f330 0x7f3c201a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:33.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c1f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c201099f0 0x7f3c201a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:33.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c1f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c201099f0 0x7f3c201a0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:37898/0 (socket says 172.21.15.134:37898) 2024-07-26T11:13:33.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c1ea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c2010f330 0x7f3c201a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:52486/0 (socket says 172.21.15.134:52486) 2024-07-26T11:13:33.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c1f400640 1 -- 172.21.15.134:0/2532674833 learned_addr learned my addr 172.21.15.134:0/2532674833 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:33.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c1fe00640 1 --2- 172.21.15.134:0/2532674833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2010fc80 0x7f3c201a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:33.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c1f400640 1 -- 172.21.15.134:0/2532674833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2010fc80 msgr2=0x7f3c201a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:33.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c1f400640 1 --2- 172.21.15.134:0/2532674833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2010fc80 0x7f3c201a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:33.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c1f400640 1 -- 172.21.15.134:0/2532674833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c2010f330 msgr2=0x7f3c201a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:33.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c1f400640 1 --2- 172.21.15.134:0/2532674833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c2010f330 0x7f3c201a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:33.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.535+0000 7f3c1f400640 1 -- 172.21.15.134:0/2532674833 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c201a62a0 con 0x7f3c201099f0 2024-07-26T11:13:33.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.536+0000 7f3c1f400640 1 --2- 172.21.15.134:0/2532674833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c201099f0 0x7f3c201a0330 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000e9c0 tx=0x7f3c1000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:33.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.536+0000 7f3c17e00640 1 -- 172.21.15.134:0/2532674833 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1000ccc0 con 0x7f3c201099f0 2024-07-26T11:13:33.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.536+0000 7f3c17e00640 1 -- 172.21.15.134:0/2532674833 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c100054d0 con 0x7f3c201099f0 2024-07-26T11:13:33.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.536+0000 7f3c25200640 1 -- 172.21.15.134:0/2532674833 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c201a6590 con 0x7f3c201099f0 2024-07-26T11:13:33.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.536+0000 7f3c17e00640 1 -- 172.21.15.134:0/2532674833 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c10010430 con 0x7f3c201099f0 2024-07-26T11:13:33.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.536+0000 7f3c25200640 1 -- 172.21.15.134:0/2532674833 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c201accf0 con 0x7f3c201099f0 2024-07-26T11:13:33.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.537+0000 7f3c25200640 1 -- 172.21.15.134:0/2532674833 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c20109e00 con 0x7f3c201099f0 2024-07-26T11:13:33.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.540+0000 7f3c17e00640 1 -- 172.21.15.134:0/2532674833 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f3c10004f80 con 0x7f3c201099f0 2024-07-26T11:13:33.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.542+0000 7f3c17e00640 1 --2- 172.21.15.134:0/2532674833 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3c040b3d30 0x7f3c040b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:33.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.542+0000 7f3c17e00640 1 -- 172.21.15.134:0/2532674833 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3c100e4990 con 0x7f3c201099f0 2024-07-26T11:13:33.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.542+0000 7f3c1ea00640 1 --2- 172.21.15.134:0/2532674833 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3c040b3d30 0x7f3c040b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:33.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.543+0000 7f3c1ea00640 1 --2- 172.21.15.134:0/2532674833 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3c040b3d30 0x7f3c040b6240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3c2010fac0 tx=0x7f3c0c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:33.769 INFO:teuthology.orchestra.run.smithi134.stdout:313532612662 2024-07-26T11:13:33.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.543+0000 7f3c17e00640 1 -- 172.21.15.134:0/2532674833 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3c100ab980 con 0x7f3c201099f0 2024-07-26T11:13:33.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.756+0000 7f3c25200640 1 -- 172.21.15.134:0/2532674833 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f3c201a1460 con 0x7f3c201099f0 2024-07-26T11:13:33.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.757+0000 7f3c17e00640 1 -- 172.21.15.134:0/2532674833 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f3c100ab280 con 0x7f3c201099f0 2024-07-26T11:13:33.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.760+0000 7f3c25200640 1 -- 172.21.15.134:0/2532674833 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3c040b3d30 msgr2=0x7f3c040b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:33.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.760+0000 7f3c25200640 1 --2- 172.21.15.134:0/2532674833 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3c040b3d30 0x7f3c040b6240 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3c2010fac0 tx=0x7f3c0c005f90 comp rx=0 tx=0).stop 2024-07-26T11:13:33.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.760+0000 7f3c25200640 1 -- 172.21.15.134:0/2532674833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c201099f0 msgr2=0x7f3c201a0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:33.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.760+0000 7f3c25200640 1 --2- 172.21.15.134:0/2532674833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c201099f0 0x7f3c201a0330 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000e9c0 tx=0x7f3c1000ee80 comp rx=0 tx=0).stop 2024-07-26T11:13:33.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.760+0000 7f3c25200640 1 -- 172.21.15.134:0/2532674833 shutdown_connections 2024-07-26T11:13:33.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.760+0000 7f3c25200640 1 --2- 172.21.15.134:0/2532674833 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3c040b3d30 0x7f3c040b6240 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:33.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.760+0000 7f3c25200640 1 --2- 172.21.15.134:0/2532674833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2010fc80 0x7f3c201a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:33.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.760+0000 7f3c25200640 1 --2- 172.21.15.134:0/2532674833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3c2010f330 0x7f3c201a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:33.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.760+0000 7f3c25200640 1 --2- 172.21.15.134:0/2532674833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c201099f0 0x7f3c201a0330 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:33.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.760+0000 7f3c25200640 1 -- 172.21.15.134:0/2532674833 >> 172.21.15.134:0/2532674833 conn(0x7f3c20078ff0 msgr2=0x7f3c20079710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:33.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.760+0000 7f3c25200640 1 -- 172.21.15.134:0/2532674833 shutdown_connections 2024-07-26T11:13:33.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:33.760+0000 7f3c25200640 1 -- 172.21.15.134:0/2532674833 wait complete. 2024-07-26T11:13:34.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.085+0000 7f51d6a00640 1 -- 172.21.15.134:0/3400591132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51d0075800 msgr2=0x7f51d0075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:34.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.085+0000 7f51d6a00640 1 --2- 172.21.15.134:0/3400591132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51d0075800 0x7f51d0075c10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f51bc009900 tx=0x7f51bc02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:13:34.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.085+0000 7f51d6a00640 1 -- 172.21.15.134:0/3400591132 shutdown_connections 2024-07-26T11:13:34.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.085+0000 7f51d6a00640 1 --2- 172.21.15.134:0/3400591132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51d0075800 0x7f51d0075c10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:34.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.085+0000 7f51d6a00640 1 --2- 172.21.15.134:0/3400591132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f51d0113cb0 0x7f51d0116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:34.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.085+0000 7f51d6a00640 1 --2- 172.21.15.134:0/3400591132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51d0076db0 0x7f51d00751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:34.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.085+0000 7f51d6a00640 1 -- 172.21.15.134:0/3400591132 >> 172.21.15.134:0/3400591132 conn(0x7f51d00fee70 msgr2=0x7f51d01012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:34.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.085+0000 7f51d6a00640 1 -- 172.21.15.134:0/3400591132 shutdown_connections 2024-07-26T11:13:34.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.085+0000 7f51d6a00640 1 -- 172.21.15.134:0/3400591132 wait complete. 2024-07-26T11:13:34.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.086+0000 7f51d6a00640 1 Processor -- start 2024-07-26T11:13:34.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.086+0000 7f51d6a00640 1 -- start start 2024-07-26T11:13:34.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.087+0000 7f51d6a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f51d0075800 0x7f51d01a0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:34.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.087+0000 7f51d6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51d0076db0 0x7f51d01a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:34.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.087+0000 7f51d6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51d0113cb0 0x7f51d01a5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:34.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.087+0000 7f51d6a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f51d0119940 con 0x7f51d0075800 2024-07-26T11:13:34.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.087+0000 7f51d6a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f51d01197c0 con 0x7f51d0113cb0 2024-07-26T11:13:34.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.087+0000 7f51d6a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f51d0119ac0 con 0x7f51d0076db0 2024-07-26T11:13:34.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.087+0000 7f51d5600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f51d0075800 0x7f51d01a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:34.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.087+0000 7f51d4c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51d0076db0 0x7f51d01a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:34.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.087+0000 7f51d4c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51d0076db0 0x7f51d01a0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:37912/0 (socket says 172.21.15.134:37912) 2024-07-26T11:13:34.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.087+0000 7f51d4c00640 1 -- 172.21.15.134:0/3514945893 learned_addr learned my addr 172.21.15.134:0/3514945893 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:34.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.087+0000 7f51d6000640 1 --2- 172.21.15.134:0/3514945893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51d0113cb0 0x7f51d01a5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:34.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.088+0000 7f51d4c00640 1 -- 172.21.15.134:0/3514945893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51d0113cb0 msgr2=0x7f51d01a5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:34.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.088+0000 7f51d4c00640 1 --2- 172.21.15.134:0/3514945893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51d0113cb0 0x7f51d01a5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:34.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.088+0000 7f51d4c00640 1 -- 172.21.15.134:0/3514945893 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f51d0075800 msgr2=0x7f51d01a0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:34.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.088+0000 7f51d4c00640 1 --2- 172.21.15.134:0/3514945893 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f51d0075800 0x7f51d01a0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:34.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.088+0000 7f51d4c00640 1 -- 172.21.15.134:0/3514945893 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d01a6300 con 0x7f51d0076db0 2024-07-26T11:13:34.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.088+0000 7f51d4c00640 1 --2- 172.21.15.134:0/3514945893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51d0076db0 0x7f51d01a0930 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f51bc005ab0 tx=0x7f51bc003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:34.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.088+0000 7f51ce000640 1 -- 172.21.15.134:0/3514945893 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc03f070 con 0x7f51d0076db0 2024-07-26T11:13:34.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.088+0000 7f51d6a00640 1 -- 172.21.15.134:0/3514945893 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d01a6590 con 0x7f51d0076db0 2024-07-26T11:13:34.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.088+0000 7f51ce000640 1 -- 172.21.15.134:0/3514945893 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51bc031b50 con 0x7f51d0076db0 2024-07-26T11:13:34.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.088+0000 7f51ce000640 1 -- 172.21.15.134:0/3514945893 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc035030 con 0x7f51d0076db0 2024-07-26T11:13:34.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.088+0000 7f51d6a00640 1 -- 172.21.15.134:0/3514945893 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d01acd50 con 0x7f51d0076db0 2024-07-26T11:13:34.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.090+0000 7f51d6a00640 1 -- 172.21.15.134:0/3514945893 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51d0075c10 con 0x7f51d0076db0 2024-07-26T11:13:34.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.092+0000 7f51ce000640 1 -- 172.21.15.134:0/3514945893 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 21) ==== 151128+0+0 (secure 0 0 0) 0x7f51bc042070 con 0x7f51d0076db0 2024-07-26T11:13:34.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.093+0000 7f51ce000640 1 --2- 172.21.15.134:0/3514945893 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f51b40b3d30 0x7f51b40b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:34.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.093+0000 7f51ce000640 1 -- 172.21.15.134:0/3514945893 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f51bc104800 con 0x7f51d0076db0 2024-07-26T11:13:34.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.093+0000 7f51d5600640 1 --2- 172.21.15.134:0/3514945893 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f51b40b3d30 0x7f51b40b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:34.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.094+0000 7f51d5600640 1 --2- 172.21.15.134:0/3514945893 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f51b40b3d30 0x7f51b40b6240 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009bd0 tx=0x7f51c0005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:34.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.097+0000 7f51ce000640 1 -- 172.21.15.134:0/3514945893 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f51bc0cb1a0 con 0x7f51d0076db0 2024-07-26T11:13:34.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.298+0000 7f51d6a00640 1 -- 172.21.15.134:0/3514945893 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f51d0112a10 con 0x7f51d0076db0 2024-07-26T11:13:34.300 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:34 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2532674833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T11:13:34.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.299+0000 7f51ce000640 1 -- 172.21.15.134:0/3514945893 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f51bc0cf6f0 con 0x7f51d0076db0 2024-07-26T11:13:34.300 INFO:teuthology.orchestra.run.smithi134.stdout:287762808891 2024-07-26T11:13:34.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.302+0000 7f51d6a00640 1 -- 172.21.15.134:0/3514945893 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f51b40b3d30 msgr2=0x7f51b40b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:34.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.302+0000 7f51d6a00640 1 --2- 172.21.15.134:0/3514945893 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f51b40b3d30 0x7f51b40b6240 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009bd0 tx=0x7f51c0005f90 comp rx=0 tx=0).stop 2024-07-26T11:13:34.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.302+0000 7f51d6a00640 1 -- 172.21.15.134:0/3514945893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51d0076db0 msgr2=0x7f51d01a0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:34.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.302+0000 7f51d6a00640 1 --2- 172.21.15.134:0/3514945893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51d0076db0 0x7f51d01a0930 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f51bc005ab0 tx=0x7f51bc003b90 comp rx=0 tx=0).stop 2024-07-26T11:13:34.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.303+0000 7f51d6a00640 1 -- 172.21.15.134:0/3514945893 shutdown_connections 2024-07-26T11:13:34.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.303+0000 7f51d6a00640 1 --2- 172.21.15.134:0/3514945893 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f51b40b3d30 0x7f51b40b6240 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:34.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.303+0000 7f51d6a00640 1 --2- 172.21.15.134:0/3514945893 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f51d0076db0 0x7f51d01a0930 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:34.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.303+0000 7f51d6a00640 1 --2- 172.21.15.134:0/3514945893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51d0113cb0 0x7f51d01a5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:34.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.303+0000 7f51d6a00640 1 --2- 172.21.15.134:0/3514945893 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f51d0075800 0x7f51d01a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:34.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.303+0000 7f51d6a00640 1 -- 172.21.15.134:0/3514945893 >> 172.21.15.134:0/3514945893 conn(0x7f51d00fee70 msgr2=0x7f51d010fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:34.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.303+0000 7f51d6a00640 1 -- 172.21.15.134:0/3514945893 shutdown_connections 2024-07-26T11:13:34.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:34.303+0000 7f51d6a00640 1 -- 172.21.15.134:0/3514945893 wait complete. 2024-07-26T11:13:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:34 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2532674833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T11:13:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:34 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2532674833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-07-26T11:13:35.202 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:35 smithi134 ceph-mon[31966]: pgmap v408: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:13:35.202 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:35 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3514945893' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T11:13:35.270 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808887 got 287762808891 for osd.10 2024-07-26T11:13:35.270 DEBUG:teuthology.parallel:result is None 2024-07-26T11:13:35.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:35 smithi189 ceph-mon[36790]: pgmap v408: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:13:35.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:35 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3514945893' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T11:13:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:35 smithi193 ceph-mon[37473]: pgmap v408: 129 pgs: 129 active+clean; 648 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:13:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:35 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3514945893' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-07-26T11:13:35.964 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:13:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:36 smithi134 ceph-mon[31966]: mgrmap e22: x(active, since 11m), standbys: z, y 2024-07-26T11:13:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:36 smithi189 ceph-mon[36790]: mgrmap e22: x(active, since 11m), standbys: z, y 2024-07-26T11:13:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:36 smithi193 ceph-mon[37473]: mgrmap e22: x(active, since 11m), standbys: z, y 2024-07-26T11:13:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:37 smithi134 ceph-mon[31966]: pgmap v409: 129 pgs: 129 active+clean; 649 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T11:13:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:37 smithi189 ceph-mon[36790]: pgmap v409: 129 pgs: 129 active+clean; 649 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T11:13:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:37 smithi193 ceph-mon[37473]: pgmap v409: 129 pgs: 129 active+clean; 649 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T11:13:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:38 smithi134 ceph-mon[31966]: pgmap v410: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T11:13:38.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:38 smithi189 ceph-mon[36790]: pgmap v410: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T11:13:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:38 smithi193 ceph-mon[37473]: pgmap v410: 129 pgs: 129 active+clean; 649 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T11:13:41.160 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612659 got 313532612662 for osd.11 2024-07-26T11:13:41.160 DEBUG:teuthology.parallel:result is None 2024-07-26T11:13:41.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.177+0000 7f2723400640 1 -- 172.21.15.134:0/3879918091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f271c1057e0 msgr2=0x7f271c105bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:41.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.177+0000 7f2723400640 1 --2- 172.21.15.134:0/3879918091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f271c1057e0 0x7f271c105bf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2710009900 tx=0x7f271002a410 comp rx=0 tx=0).stop 2024-07-26T11:13:41.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.177+0000 7f2723400640 1 -- 172.21.15.134:0/3879918091 shutdown_connections 2024-07-26T11:13:41.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.177+0000 7f2723400640 1 --2- 172.21.15.134:0/3879918091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f271c10ba70 0x7f271c1122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.177+0000 7f2723400640 1 --2- 172.21.15.134:0/3879918091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f271c10b120 0x7f271c10b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.177+0000 7f2723400640 1 --2- 172.21.15.134:0/3879918091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f271c1057e0 0x7f271c105bf0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.177+0000 7f2723400640 1 -- 172.21.15.134:0/3879918091 >> 172.21.15.134:0/3879918091 conn(0x7f271c1011f0 msgr2=0x7f271c103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:41.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.177+0000 7f2723400640 1 -- 172.21.15.134:0/3879918091 shutdown_connections 2024-07-26T11:13:41.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.177+0000 7f2723400640 1 -- 172.21.15.134:0/3879918091 wait complete. 2024-07-26T11:13:41.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.178+0000 7f2723400640 1 Processor -- start 2024-07-26T11:13:41.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.179+0000 7f2723400640 1 -- start start 2024-07-26T11:13:41.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.179+0000 7f2723400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f271c1057e0 0x7f271c19c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:41.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.179+0000 7f2723400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f271c10b120 0x7f271c19c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:41.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.179+0000 7f2723400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f271c10ba70 0x7f271c1a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:41.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.179+0000 7f2723400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f271c115770 con 0x7f271c10b120 2024-07-26T11:13:41.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.179+0000 7f2723400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f271c1155f0 con 0x7f271c10ba70 2024-07-26T11:13:41.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.179+0000 7f2723400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f271c1158f0 con 0x7f271c1057e0 2024-07-26T11:13:41.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.179+0000 7f271be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f271c10b120 0x7f271c19c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:41.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.179+0000 7f2720a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f271c1057e0 0x7f271c19c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:41.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.179+0000 7f271be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f271c10b120 0x7f271c19c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36188/0 (socket says 172.21.15.134:36188) 2024-07-26T11:13:41.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.179+0000 7f271be00640 1 -- 172.21.15.134:0/999502209 learned_addr learned my addr 172.21.15.134:0/999502209 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:41.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.180+0000 7f2721400640 1 --2- 172.21.15.134:0/999502209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f271c10ba70 0x7f271c1a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:41.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.180+0000 7f271be00640 1 -- 172.21.15.134:0/999502209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f271c1057e0 msgr2=0x7f271c19c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:41.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.180+0000 7f271be00640 1 --2- 172.21.15.134:0/999502209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f271c1057e0 0x7f271c19c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.180+0000 7f271be00640 1 -- 172.21.15.134:0/999502209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f271c10ba70 msgr2=0x7f271c1a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:41.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.180+0000 7f271be00640 1 --2- 172.21.15.134:0/999502209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f271c10ba70 0x7f271c1a1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.180+0000 7f271be00640 1 -- 172.21.15.134:0/999502209 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271c1a2130 con 0x7f271c10b120 2024-07-26T11:13:41.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.180+0000 7f2720a00640 1 --2- 172.21.15.134:0/999502209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f271c1057e0 0x7f271c19c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:41.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.180+0000 7f271be00640 1 --2- 172.21.15.134:0/999502209 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f271c10b120 0x7f271c19c760 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f270800e9f0 tx=0x7f270800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:41.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.181+0000 7f2719600640 1 -- 172.21.15.134:0/999502209 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270800ccc0 con 0x7f271c10b120 2024-07-26T11:13:41.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.181+0000 7f2719600640 1 -- 172.21.15.134:0/999502209 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27080108e0 con 0x7f271c10b120 2024-07-26T11:13:41.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.181+0000 7f2719600640 1 -- 172.21.15.134:0/999502209 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708018860 con 0x7f271c10b120 2024-07-26T11:13:41.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.181+0000 7f2723400640 1 -- 172.21.15.134:0/999502209 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f271c1a2420 con 0x7f271c10b120 2024-07-26T11:13:41.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.181+0000 7f2723400640 1 -- 172.21.15.134:0/999502209 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f271c1a8b80 con 0x7f271c10b120 2024-07-26T11:13:41.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.182+0000 7f2723400640 1 -- 172.21.15.134:0/999502209 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f271c105bf0 con 0x7f271c10b120 2024-07-26T11:13:41.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.186+0000 7f2719600640 1 -- 172.21.15.134:0/999502209 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f2708010430 con 0x7f271c10b120 2024-07-26T11:13:41.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.188+0000 7f2719600640 1 --2- 172.21.15.134:0/999502209 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f27040b3c20 0x7f27040b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:41.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.189+0000 7f2719600640 1 -- 172.21.15.134:0/999502209 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f27080e48e0 con 0x7f271c10b120 2024-07-26T11:13:41.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.189+0000 7f2720a00640 1 --2- 172.21.15.134:0/999502209 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f27040b3c20 0x7f27040b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:41.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.190+0000 7f2720a00640 1 --2- 172.21.15.134:0/999502209 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f27040b3c20 0x7f27040b6130 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2710005ab0 tx=0x7f2710005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:41.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.191+0000 7f2719600640 1 -- 172.21.15.134:0/999502209 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f27080ab3e0 con 0x7f271c10b120 2024-07-26T11:13:41.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.395+0000 7f2723400640 1 -- 172.21.15.134:0/999502209 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f271c1a8de0 con 0x7f271c10b120 2024-07-26T11:13:41.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.396+0000 7f2719600640 1 -- 172.21.15.134:0/999502209 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f27080af930 con 0x7f271c10b120 2024-07-26T11:13:41.201 INFO:teuthology.orchestra.run.smithi134.stdout:167503724635 2024-07-26T11:13:41.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.399+0000 7f2723400640 1 -- 172.21.15.134:0/999502209 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f27040b3c20 msgr2=0x7f27040b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:41.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.399+0000 7f2723400640 1 --2- 172.21.15.134:0/999502209 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f27040b3c20 0x7f27040b6130 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2710005ab0 tx=0x7f2710005a20 comp rx=0 tx=0).stop 2024-07-26T11:13:41.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.399+0000 7f2723400640 1 -- 172.21.15.134:0/999502209 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f271c10b120 msgr2=0x7f271c19c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:41.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.399+0000 7f2723400640 1 --2- 172.21.15.134:0/999502209 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f271c10b120 0x7f271c19c760 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f270800e9f0 tx=0x7f270800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.399+0000 7f2723400640 1 -- 172.21.15.134:0/999502209 shutdown_connections 2024-07-26T11:13:41.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.399+0000 7f2723400640 1 --2- 172.21.15.134:0/999502209 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f27040b3c20 0x7f27040b6130 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.399+0000 7f2723400640 1 --2- 172.21.15.134:0/999502209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f271c10ba70 0x7f271c1a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.399+0000 7f2723400640 1 --2- 172.21.15.134:0/999502209 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f271c10b120 0x7f271c19c760 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.399+0000 7f2723400640 1 --2- 172.21.15.134:0/999502209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f271c1057e0 0x7f271c19c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.399+0000 7f2723400640 1 -- 172.21.15.134:0/999502209 >> 172.21.15.134:0/999502209 conn(0x7f271c1011f0 msgr2=0x7f271c1098f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:41.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.399+0000 7f2723400640 1 -- 172.21.15.134:0/999502209 shutdown_connections 2024-07-26T11:13:41.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:40.399+0000 7f2723400640 1 -- 172.21.15.134:0/999502209 wait complete. 2024-07-26T11:13:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:41 smithi134 ceph-mon[31966]: pgmap v411: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T11:13:41.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:41 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/999502209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T11:13:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:41 smithi189 ceph-mon[36790]: pgmap v411: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T11:13:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:41 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/999502209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T11:13:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:41 smithi193 ceph-mon[37473]: pgmap v411: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T11:13:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:41 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/999502209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-07-26T11:13:41.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.847+0000 7f3186400640 1 -- 172.21.15.134:0/1487241206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3180104c60 msgr2=0x7f318010d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:41.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.847+0000 7f3186400640 1 --2- 172.21.15.134:0/1487241206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3180104c60 0x7f318010d380 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f316c009900 tx=0x7f316c02a410 comp rx=0 tx=0).stop 2024-07-26T11:13:41.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.847+0000 7f3186400640 1 -- 172.21.15.134:0/1487241206 shutdown_connections 2024-07-26T11:13:41.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.847+0000 7f3186400640 1 --2- 172.21.15.134:0/1487241206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f318010d8c0 0x7f318010fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.847+0000 7f3186400640 1 --2- 172.21.15.134:0/1487241206 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3180104c60 0x7f318010d380 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.847+0000 7f3186400640 1 --2- 172.21.15.134:0/1487241206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3180104310 0x7f3180104720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.847+0000 7f3186400640 1 -- 172.21.15.134:0/1487241206 >> 172.21.15.134:0/1487241206 conn(0x7f3180079150 msgr2=0x7f3180079560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:41.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.847+0000 7f3186400640 1 -- 172.21.15.134:0/1487241206 shutdown_connections 2024-07-26T11:13:41.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.847+0000 7f3186400640 1 -- 172.21.15.134:0/1487241206 wait complete. 2024-07-26T11:13:41.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.848+0000 7f3186400640 1 Processor -- start 2024-07-26T11:13:41.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.848+0000 7f3186400640 1 -- start start 2024-07-26T11:13:41.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.849+0000 7f3186400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3180104310 0x7f318019c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:41.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.849+0000 7f3186400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3180104c60 0x7f318019c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:41.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.849+0000 7f3186400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f318010d8c0 0x7f31801a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:41.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.849+0000 7f3186400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f31801134a0 con 0x7f318010d8c0 2024-07-26T11:13:41.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.849+0000 7f3186400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3180113320 con 0x7f3180104310 2024-07-26T11:13:41.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.849+0000 7f3186400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3180113620 con 0x7f3180104c60 2024-07-26T11:13:41.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.850+0000 7f317fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f318010d8c0 0x7f31801a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:41.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.850+0000 7f317ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3180104c60 0x7f318019c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:41.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.850+0000 7f317f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3180104310 0x7f318019c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:41.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.850+0000 7f317ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3180104c60 0x7f318019c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:47516/0 (socket says 172.21.15.134:47516) 2024-07-26T11:13:41.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.850+0000 7f317ea00640 1 -- 172.21.15.134:0/3920710847 learned_addr learned my addr 172.21.15.134:0/3920710847 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:41.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.850+0000 7f317f400640 1 -- 172.21.15.134:0/3920710847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3180104c60 msgr2=0x7f318019c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:41.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.850+0000 7f317f400640 1 --2- 172.21.15.134:0/3920710847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3180104c60 0x7f318019c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.850+0000 7f317f400640 1 -- 172.21.15.134:0/3920710847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f318010d8c0 msgr2=0x7f31801a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:41.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.850+0000 7f317f400640 1 --2- 172.21.15.134:0/3920710847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f318010d8c0 0x7f31801a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.850+0000 7f317f400640 1 -- 172.21.15.134:0/3920710847 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31801a2020 con 0x7f3180104310 2024-07-26T11:13:41.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.851+0000 7f317f400640 1 --2- 172.21.15.134:0/3920710847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3180104310 0x7f318019c0b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f317000e9f0 tx=0x7f317000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:41.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.851+0000 7f3177e00640 1 -- 172.21.15.134:0/3920710847 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317000cd80 con 0x7f3180104310 2024-07-26T11:13:41.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.851+0000 7f3186400640 1 -- 172.21.15.134:0/3920710847 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31801a2310 con 0x7f3180104310 2024-07-26T11:13:41.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.851+0000 7f3177e00640 1 -- 172.21.15.134:0/3920710847 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31700035f0 con 0x7f3180104310 2024-07-26T11:13:41.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.851+0000 7f3177e00640 1 -- 172.21.15.134:0/3920710847 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3170010690 con 0x7f3180104310 2024-07-26T11:13:41.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.851+0000 7f3186400640 1 -- 172.21.15.134:0/3920710847 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31801a8a70 con 0x7f3180104310 2024-07-26T11:13:41.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.852+0000 7f3186400640 1 -- 172.21.15.134:0/3920710847 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3180104720 con 0x7f3180104310 2024-07-26T11:13:41.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.855+0000 7f3177e00640 1 -- 172.21.15.134:0/3920710847 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f31700037a0 con 0x7f3180104310 2024-07-26T11:13:41.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.856+0000 7f3177e00640 1 --2- 172.21.15.134:0/3920710847 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31640b3c20 0x7f31640b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:41.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.856+0000 7f3177e00640 1 -- 172.21.15.134:0/3920710847 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f31700e44a0 con 0x7f3180104310 2024-07-26T11:13:41.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.857+0000 7f317ea00640 1 --2- 172.21.15.134:0/3920710847 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31640b3c20 0x7f31640b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:41.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.857+0000 7f317ea00640 1 --2- 172.21.15.134:0/3920710847 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31640b3c20 0x7f31640b6130 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f316c005f70 tx=0x7f316c009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:41.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.858+0000 7f3177e00640 1 -- 172.21.15.134:0/3920710847 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f31700ab5f0 con 0x7f3180104310 2024-07-26T11:13:41.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.927+0000 7fbb27e00640 1 -- 172.21.15.134:0/1072198942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb201035b0 msgr2=0x7fbb201039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:41.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.927+0000 7fbb27e00640 1 --2- 172.21.15.134:0/1072198942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb201035b0 0x7fbb201039c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbb14009860 tx=0x7fbb1402a340 comp rx=0 tx=0).stop 2024-07-26T11:13:41.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.928+0000 7fbb27e00640 1 -- 172.21.15.134:0/1072198942 shutdown_connections 2024-07-26T11:13:41.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.928+0000 7fbb27e00640 1 --2- 172.21.15.134:0/1072198942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb20109840 0x7fbb20110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.928+0000 7fbb27e00640 1 --2- 172.21.15.134:0/1072198942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb20108ef0 0x7fbb20109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.928+0000 7fbb27e00640 1 --2- 172.21.15.134:0/1072198942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb201035b0 0x7fbb201039c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.928+0000 7fbb27e00640 1 -- 172.21.15.134:0/1072198942 >> 172.21.15.134:0/1072198942 conn(0x7fbb200ff040 msgr2=0x7fbb20101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:41.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.928+0000 7fbb27e00640 1 -- 172.21.15.134:0/1072198942 shutdown_connections 2024-07-26T11:13:41.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.928+0000 7fbb27e00640 1 -- 172.21.15.134:0/1072198942 wait complete. 2024-07-26T11:13:41.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.929+0000 7fbb27e00640 1 Processor -- start 2024-07-26T11:13:41.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.929+0000 7fbb27e00640 1 -- start start 2024-07-26T11:13:41.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.930+0000 7fbb27e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb201035b0 0x7fbb2010dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:41.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.930+0000 7fbb27e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb20108ef0 0x7fbb20109f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:41.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.930+0000 7fbb27e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb20109840 0x7fbb2010a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:41.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.930+0000 7fbb27e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbb20113510 con 0x7fbb20109840 2024-07-26T11:13:41.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.930+0000 7fbb27e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbb20113390 con 0x7fbb20108ef0 2024-07-26T11:13:41.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.930+0000 7fbb27e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbb20113690 con 0x7fbb201035b0 2024-07-26T11:13:41.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.930+0000 7fbb24a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb20108ef0 0x7fbb20109f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:41.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.930+0000 7fbb24a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb20108ef0 0x7fbb20109f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46152/0 (socket says 172.21.15.134:46152) 2024-07-26T11:13:41.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.930+0000 7fbb24a00640 1 -- 172.21.15.134:0/3159013992 learned_addr learned my addr 172.21.15.134:0/3159013992 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:41.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.930+0000 7fbb25e00640 1 --2- 172.21.15.134:0/3159013992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb20109840 0x7fbb2010a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:41.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.930+0000 7fbb25400640 1 --2- 172.21.15.134:0/3159013992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb201035b0 0x7fbb2010dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:41.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.931+0000 7fbb24a00640 1 -- 172.21.15.134:0/3159013992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb201035b0 msgr2=0x7fbb2010dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:41.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.931+0000 7fbb24a00640 1 --2- 172.21.15.134:0/3159013992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb201035b0 0x7fbb2010dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.931+0000 7fbb24a00640 1 -- 172.21.15.134:0/3159013992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb20109840 msgr2=0x7fbb2010a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:41.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.931+0000 7fbb24a00640 1 --2- 172.21.15.134:0/3159013992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb20109840 0x7fbb2010a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:41.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.931+0000 7fbb24a00640 1 -- 172.21.15.134:0/3159013992 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb201ac710 con 0x7fbb20108ef0 2024-07-26T11:13:41.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.931+0000 7fbb25e00640 1 --2- 172.21.15.134:0/3159013992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb20109840 0x7fbb2010a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:13:41.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.931+0000 7fbb24a00640 1 --2- 172.21.15.134:0/3159013992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb20108ef0 0x7fbb20109f50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00e9c0 tx=0x7fbb0c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:41.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.931+0000 7fbb1e000640 1 -- 172.21.15.134:0/3159013992 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c00ccc0 con 0x7fbb20108ef0 2024-07-26T11:13:41.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.932+0000 7fbb1e000640 1 -- 172.21.15.134:0/3159013992 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb0c0054d0 con 0x7fbb20108ef0 2024-07-26T11:13:41.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.932+0000 7fbb1e000640 1 -- 172.21.15.134:0/3159013992 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c010430 con 0x7fbb20108ef0 2024-07-26T11:13:41.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.932+0000 7fbb27e00640 1 -- 172.21.15.134:0/3159013992 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb201ac940 con 0x7fbb20108ef0 2024-07-26T11:13:41.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.932+0000 7fbb27e00640 1 -- 172.21.15.134:0/3159013992 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb201ace30 con 0x7fbb20108ef0 2024-07-26T11:13:41.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.933+0000 7fbb27e00640 1 -- 172.21.15.134:0/3159013992 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb20104830 con 0x7fbb20108ef0 2024-07-26T11:13:41.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.939+0000 7fbb1e000640 1 -- 172.21.15.134:0/3159013992 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fbb0c005020 con 0x7fbb20108ef0 2024-07-26T11:13:41.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.941+0000 7fbb1e000640 1 --2- 172.21.15.134:0/3159013992 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbb080b3c20 0x7fbb080b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:41.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.941+0000 7fbb1e000640 1 -- 172.21.15.134:0/3159013992 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fbb0c0e4680 con 0x7fbb20108ef0 2024-07-26T11:13:41.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.941+0000 7fbb25400640 1 --2- 172.21.15.134:0/3159013992 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbb080b3c20 0x7fbb080b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:41.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.941+0000 7fbb1e000640 1 -- 172.21.15.134:0/3159013992 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fbb0c0e4b00 con 0x7fbb20108ef0 2024-07-26T11:13:41.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:41.942+0000 7fbb25400640 1 --2- 172.21.15.134:0/3159013992 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbb080b3c20 0x7fbb080b6130 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbb14009830 tx=0x7fbb140058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:41.961 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:13:42.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.069+0000 7f3186400640 1 -- 172.21.15.134:0/3920710847 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f318019d1e0 con 0x7f3180104310 2024-07-26T11:13:42.072 INFO:teuthology.orchestra.run.smithi134.stdout:214748364877 2024-07-26T11:13:42.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.069+0000 7f3177e00640 1 -- 172.21.15.134:0/3920710847 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f31700aaef0 con 0x7f3180104310 2024-07-26T11:13:42.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.073+0000 7f3186400640 1 -- 172.21.15.134:0/3920710847 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31640b3c20 msgr2=0x7f31640b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:42.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.073+0000 7f3186400640 1 --2- 172.21.15.134:0/3920710847 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31640b3c20 0x7f31640b6130 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f316c005f70 tx=0x7f316c009560 comp rx=0 tx=0).stop 2024-07-26T11:13:42.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.073+0000 7f3186400640 1 -- 172.21.15.134:0/3920710847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3180104310 msgr2=0x7f318019c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:42.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.073+0000 7f3186400640 1 --2- 172.21.15.134:0/3920710847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3180104310 0x7f318019c0b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f317000e9f0 tx=0x7f317000eeb0 comp rx=0 tx=0).stop 2024-07-26T11:13:42.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.073+0000 7f3186400640 1 -- 172.21.15.134:0/3920710847 shutdown_connections 2024-07-26T11:13:42.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.073+0000 7f3186400640 1 --2- 172.21.15.134:0/3920710847 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f31640b3c20 0x7f31640b6130 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:42.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.073+0000 7f3186400640 1 --2- 172.21.15.134:0/3920710847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3180104c60 0x7f318019c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:42.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.073+0000 7f3186400640 1 --2- 172.21.15.134:0/3920710847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f318010d8c0 0x7f31801a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:42.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.073+0000 7f3186400640 1 --2- 172.21.15.134:0/3920710847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3180104310 0x7f318019c0b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:42.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.073+0000 7f3186400640 1 -- 172.21.15.134:0/3920710847 >> 172.21.15.134:0/3920710847 conn(0x7f3180079150 msgr2=0x7f3180100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:42.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.074+0000 7f3186400640 1 -- 172.21.15.134:0/3920710847 shutdown_connections 2024-07-26T11:13:42.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.074+0000 7f3186400640 1 -- 172.21.15.134:0/3920710847 wait complete. 2024-07-26T11:13:42.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.185+0000 7fbb27e00640 1 -- 172.21.15.134:0/3159013992 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fbb201ad230 con 0x7fbb20108ef0 2024-07-26T11:13:42.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.185+0000 7fbb1e000640 1 -- 172.21.15.134:0/3159013992 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbb0c0ab7d0 con 0x7fbb20108ef0 2024-07-26T11:13:42.188 INFO:teuthology.orchestra.run.smithi134.stdout:81604378730 2024-07-26T11:13:42.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.188+0000 7fbb27e00640 1 -- 172.21.15.134:0/3159013992 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbb080b3c20 msgr2=0x7fbb080b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:42.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.188+0000 7fbb27e00640 1 --2- 172.21.15.134:0/3159013992 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbb080b3c20 0x7fbb080b6130 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbb14009830 tx=0x7fbb140058d0 comp rx=0 tx=0).stop 2024-07-26T11:13:42.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.188+0000 7fbb27e00640 1 -- 172.21.15.134:0/3159013992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb20108ef0 msgr2=0x7fbb20109f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:42.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.188+0000 7fbb27e00640 1 --2- 172.21.15.134:0/3159013992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb20108ef0 0x7fbb20109f50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00e9c0 tx=0x7fbb0c00ee80 comp rx=0 tx=0).stop 2024-07-26T11:13:42.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.188+0000 7fbb27e00640 1 -- 172.21.15.134:0/3159013992 shutdown_connections 2024-07-26T11:13:42.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.188+0000 7fbb27e00640 1 --2- 172.21.15.134:0/3159013992 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbb080b3c20 0x7fbb080b6130 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:42.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.188+0000 7fbb27e00640 1 --2- 172.21.15.134:0/3159013992 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbb20109840 0x7fbb2010a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:42.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.188+0000 7fbb27e00640 1 --2- 172.21.15.134:0/3159013992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb20108ef0 0x7fbb20109f50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:42.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.188+0000 7fbb27e00640 1 --2- 172.21.15.134:0/3159013992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbb201035b0 0x7fbb2010dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:42.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.188+0000 7fbb27e00640 1 -- 172.21.15.134:0/3159013992 >> 172.21.15.134:0/3159013992 conn(0x7fbb200ff040 msgr2=0x7fbb2010e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:42.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.188+0000 7fbb27e00640 1 -- 172.21.15.134:0/3159013992 shutdown_connections 2024-07-26T11:13:42.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.188+0000 7fbb27e00640 1 -- 172.21.15.134:0/3159013992 wait complete. 2024-07-26T11:13:43.059 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724630 got 167503724635 for osd.5 2024-07-26T11:13:43.059 DEBUG:teuthology.parallel:result is None 2024-07-26T11:13:43.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:43 smithi134 ceph-mon[31966]: pgmap v412: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T11:13:43.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:43 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3920710847' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T11:13:43.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:43 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3159013992' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T11:13:43.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:43 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1438528179' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T11:13:43.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:43 smithi189 ceph-mon[36790]: pgmap v412: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T11:13:43.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:43 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3920710847' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T11:13:43.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:43 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3159013992' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T11:13:43.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:43 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1438528179' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T11:13:43.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.490+0000 7f025a400640 1 -- 172.21.15.134:0/913433930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f025410ba70 msgr2=0x7f02541122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:43.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.490+0000 7f025a400640 1 --2- 172.21.15.134:0/913433930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f025410ba70 0x7f02541122a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f024c009be0 tx=0x7f024c02a6c0 comp rx=0 tx=0).stop 2024-07-26T11:13:43.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.490+0000 7f025a400640 1 -- 172.21.15.134:0/913433930 shutdown_connections 2024-07-26T11:13:43.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.490+0000 7f025a400640 1 --2- 172.21.15.134:0/913433930 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f025410ba70 0x7f02541122a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:43.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.490+0000 7f025a400640 1 --2- 172.21.15.134:0/913433930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f025410b120 0x7f025410b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:43.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.490+0000 7f025a400640 1 --2- 172.21.15.134:0/913433930 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02541057e0 0x7f0254105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:43.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.490+0000 7f025a400640 1 -- 172.21.15.134:0/913433930 >> 172.21.15.134:0/913433930 conn(0x7f02541011f0 msgr2=0x7f0254103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:43.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.490+0000 7f025a400640 1 -- 172.21.15.134:0/913433930 shutdown_connections 2024-07-26T11:13:43.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.490+0000 7f025a400640 1 -- 172.21.15.134:0/913433930 wait complete. 2024-07-26T11:13:43.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.491+0000 7f025a400640 1 Processor -- start 2024-07-26T11:13:43.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.492+0000 7f025a400640 1 -- start start 2024-07-26T11:13:43.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.492+0000 7f025a400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02541057e0 0x7f025419c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:43.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.492+0000 7f025a400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f025410b120 0x7f025419c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:43.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.492+0000 7f025a400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f025410ba70 0x7f02541a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:43.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.492+0000 7f025a400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0254115720 con 0x7f02541057e0 2024-07-26T11:13:43.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.492+0000 7f025a400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f02541155a0 con 0x7f025410ba70 2024-07-26T11:13:43.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.492+0000 7f025a400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f02541158a0 con 0x7f025410b120 2024-07-26T11:13:43.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.492+0000 7f0253400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02541057e0 0x7f025419c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:43.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.492+0000 7f0253400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02541057e0 0x7f025419c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36248/0 (socket says 172.21.15.134:36248) 2024-07-26T11:13:43.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.492+0000 7f0253400640 1 -- 172.21.15.134:0/1438528179 learned_addr learned my addr 172.21.15.134:0/1438528179 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:43.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.492+0000 7f0253e00640 1 --2- 172.21.15.134:0/1438528179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f025410ba70 0x7f02541a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:43.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.493+0000 7f0252a00640 1 --2- 172.21.15.134:0/1438528179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f025410b120 0x7f025419c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:43.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.493+0000 7f0253400640 1 -- 172.21.15.134:0/1438528179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f025410b120 msgr2=0x7f025419c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:43.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.493+0000 7f0253400640 1 --2- 172.21.15.134:0/1438528179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f025410b120 0x7f025419c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:43.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.493+0000 7f0253400640 1 -- 172.21.15.134:0/1438528179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f025410ba70 msgr2=0x7f02541a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:43.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.493+0000 7f0253400640 1 --2- 172.21.15.134:0/1438528179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f025410ba70 0x7f02541a18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:43.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.493+0000 7f0253400640 1 -- 172.21.15.134:0/1438528179 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02541a2000 con 0x7f02541057e0 2024-07-26T11:13:43.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.493+0000 7f0253e00640 1 --2- 172.21.15.134:0/1438528179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f025410ba70 0x7f02541a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:43.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.493+0000 7f0253400640 1 --2- 172.21.15.134:0/1438528179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02541057e0 0x7f025419c090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f024400e9c0 tx=0x7f024400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:43.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.494+0000 7f024be00640 1 -- 172.21.15.134:0/1438528179 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f024400ccc0 con 0x7f02541057e0 2024-07-26T11:13:43.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.494+0000 7f025a400640 1 -- 172.21.15.134:0/1438528179 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02541a22f0 con 0x7f02541057e0 2024-07-26T11:13:43.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.494+0000 7f024be00640 1 -- 172.21.15.134:0/1438528179 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02440054d0 con 0x7f02541057e0 2024-07-26T11:13:43.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.494+0000 7f024be00640 1 -- 172.21.15.134:0/1438528179 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0244010430 con 0x7f02541057e0 2024-07-26T11:13:43.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.494+0000 7f025a400640 1 -- 172.21.15.134:0/1438528179 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02541a8a50 con 0x7f02541057e0 2024-07-26T11:13:43.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.495+0000 7f025a400640 1 -- 172.21.15.134:0/1438528179 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0254105bf0 con 0x7f02541057e0 2024-07-26T11:13:43.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.499+0000 7f024be00640 1 -- 172.21.15.134:0/1438528179 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f0244004f80 con 0x7f02541057e0 2024-07-26T11:13:43.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.501+0000 7f024be00640 1 --2- 172.21.15.134:0/1438528179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f02380b3ca0 0x7f02380b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:43.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.501+0000 7f024be00640 1 -- 172.21.15.134:0/1438528179 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f02440e4890 con 0x7f02541057e0 2024-07-26T11:13:43.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.501+0000 7f0252a00640 1 --2- 172.21.15.134:0/1438528179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f02380b3ca0 0x7f02380b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:43.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.502+0000 7f0252a00640 1 --2- 172.21.15.134:0/1438528179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f02380b3ca0 0x7f02380b61b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0254103a30 tx=0x7f023c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:43.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.502+0000 7f024be00640 1 -- 172.21.15.134:0/1438528179 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f02440ab9e0 con 0x7f02541057e0 2024-07-26T11:13:43.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.706+0000 7f025a400640 1 -- 172.21.15.134:0/1438528179 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f025419d1c0 con 0x7f02541057e0 2024-07-26T11:13:43.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.707+0000 7f024be00640 1 -- 172.21.15.134:0/1438528179 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f02440ab2e0 con 0x7f02541057e0 2024-07-26T11:13:43.523 INFO:teuthology.orchestra.run.smithi134.stdout:146028888160 2024-07-26T11:13:43.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.710+0000 7f025a400640 1 -- 172.21.15.134:0/1438528179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f02380b3ca0 msgr2=0x7f02380b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:43.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.710+0000 7f025a400640 1 --2- 172.21.15.134:0/1438528179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f02380b3ca0 0x7f02380b61b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0254103a30 tx=0x7f023c005f90 comp rx=0 tx=0).stop 2024-07-26T11:13:43.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.710+0000 7f025a400640 1 -- 172.21.15.134:0/1438528179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02541057e0 msgr2=0x7f025419c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:43.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.710+0000 7f025a400640 1 --2- 172.21.15.134:0/1438528179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02541057e0 0x7f025419c090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f024400e9c0 tx=0x7f024400ee80 comp rx=0 tx=0).stop 2024-07-26T11:13:43.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.710+0000 7f025a400640 1 -- 172.21.15.134:0/1438528179 shutdown_connections 2024-07-26T11:13:43.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.710+0000 7f025a400640 1 --2- 172.21.15.134:0/1438528179 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f02380b3ca0 0x7f02380b61b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:43.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.710+0000 7f025a400640 1 --2- 172.21.15.134:0/1438528179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f025410b120 0x7f025419c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:43.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.710+0000 7f025a400640 1 --2- 172.21.15.134:0/1438528179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f025410ba70 0x7f02541a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:43.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.710+0000 7f025a400640 1 --2- 172.21.15.134:0/1438528179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f02541057e0 0x7f025419c090 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:43.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.710+0000 7f025a400640 1 -- 172.21.15.134:0/1438528179 >> 172.21.15.134:0/1438528179 conn(0x7f02541011f0 msgr2=0x7f025410a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:43.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.711+0000 7f025a400640 1 -- 172.21.15.134:0/1438528179 shutdown_connections 2024-07-26T11:13:43.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:42.711+0000 7f025a400640 1 -- 172.21.15.134:0/1438528179 wait complete. 2024-07-26T11:13:43.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:43 smithi193 ceph-mon[37473]: pgmap v412: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T11:13:43.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:43 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3920710847' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-07-26T11:13:43.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:43 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3159013992' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-07-26T11:13:43.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:43 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1438528179' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-07-26T11:13:45.261 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:45 smithi134 ceph-mon[31966]: pgmap v413: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T11:13:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:45 smithi189 ceph-mon[36790]: pgmap v413: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T11:13:45.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:45 smithi193 ceph-mon[37473]: pgmap v413: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-07-26T11:13:45.532 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888152 got 146028888160 for osd.4 2024-07-26T11:13:45.532 DEBUG:teuthology.parallel:result is None 2024-07-26T11:13:45.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.772+0000 7fd899800640 1 -- 172.21.15.134:0/1392181939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd89410b860 msgr2=0x7fd894112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:45.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.772+0000 7fd899800640 1 --2- 172.21.15.134:0/1392181939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd89410b860 0x7fd894112090 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd88c009ca0 tx=0x7fd88c02a8b0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.772+0000 7fd899800640 1 -- 172.21.15.134:0/1392181939 shutdown_connections 2024-07-26T11:13:45.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.772+0000 7fd899800640 1 --2- 172.21.15.134:0/1392181939 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd89410b860 0x7fd894112090 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.772+0000 7fd899800640 1 --2- 172.21.15.134:0/1392181939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd89410af10 0x7fd89410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.772+0000 7fd899800640 1 --2- 172.21.15.134:0/1392181939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8941055d0 0x7fd8941059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.772+0000 7fd899800640 1 -- 172.21.15.134:0/1392181939 >> 172.21.15.134:0/1392181939 conn(0x7fd894101040 msgr2=0x7fd894103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:45.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.772+0000 7fd899800640 1 -- 172.21.15.134:0/1392181939 shutdown_connections 2024-07-26T11:13:45.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.772+0000 7fd899800640 1 -- 172.21.15.134:0/1392181939 wait complete. 2024-07-26T11:13:45.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.773+0000 7fd899800640 1 Processor -- start 2024-07-26T11:13:45.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.774+0000 7fd899800640 1 -- start start 2024-07-26T11:13:45.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.774+0000 7fd899800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8941055d0 0x7fd89419be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:45.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.774+0000 7fd899800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd89410af10 0x7fd89419c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:45.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.774+0000 7fd899800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd89410b860 0x7fd8941a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:45.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.774+0000 7fd899800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd894115560 con 0x7fd8941055d0 2024-07-26T11:13:45.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.774+0000 7fd899800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd8941153e0 con 0x7fd89410b860 2024-07-26T11:13:45.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.774+0000 7fd899800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd8941156e0 con 0x7fd89410af10 2024-07-26T11:13:45.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.774+0000 7fd893e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8941055d0 0x7fd89419be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:45.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.775+0000 7fd893400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd89410af10 0x7fd89419c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:45.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.775+0000 7fd893e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8941055d0 0x7fd89419be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36300/0 (socket says 172.21.15.134:36300) 2024-07-26T11:13:45.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.775+0000 7fd893e00640 1 -- 172.21.15.134:0/900963170 learned_addr learned my addr 172.21.15.134:0/900963170 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:45.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.775+0000 7fd898e00640 1 --2- 172.21.15.134:0/900963170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd89410b860 0x7fd8941a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:45.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.775+0000 7fd893400640 1 -- 172.21.15.134:0/900963170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd89410b860 msgr2=0x7fd8941a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:45.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.775+0000 7fd893400640 1 --2- 172.21.15.134:0/900963170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd89410b860 0x7fd8941a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.775+0000 7fd893400640 1 -- 172.21.15.134:0/900963170 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8941055d0 msgr2=0x7fd89419be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:45.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.775+0000 7fd893400640 1 --2- 172.21.15.134:0/900963170 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8941055d0 0x7fd89419be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.775+0000 7fd893400640 1 -- 172.21.15.134:0/900963170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8941a1e00 con 0x7fd89410af10 2024-07-26T11:13:45.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.775+0000 7fd898e00640 1 --2- 172.21.15.134:0/900963170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd89410b860 0x7fd8941a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:13:45.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.775+0000 7fd893400640 1 --2- 172.21.15.134:0/900963170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd89410af10 0x7fd89419c430 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd89406c270 tx=0x7fd87c00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:45.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.776+0000 7fd890c00640 1 -- 172.21.15.134:0/900963170 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c00cc40 con 0x7fd89410af10 2024-07-26T11:13:45.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.776+0000 7fd890c00640 1 -- 172.21.15.134:0/900963170 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd87c00cde0 con 0x7fd89410af10 2024-07-26T11:13:45.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.776+0000 7fd890c00640 1 -- 172.21.15.134:0/900963170 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c018780 con 0x7fd89410af10 2024-07-26T11:13:45.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.776+0000 7fd899800640 1 -- 172.21.15.134:0/900963170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8941a20f0 con 0x7fd89410af10 2024-07-26T11:13:45.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.776+0000 7fd899800640 1 -- 172.21.15.134:0/900963170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8941a8850 con 0x7fd89410af10 2024-07-26T11:13:45.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.777+0000 7fd899800640 1 -- 172.21.15.134:0/900963170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8941059e0 con 0x7fd89410af10 2024-07-26T11:13:45.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.780+0000 7fd890c00640 1 -- 172.21.15.134:0/900963170 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fd87c010a90 con 0x7fd89410af10 2024-07-26T11:13:45.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.781+0000 7fd890c00640 1 --2- 172.21.15.134:0/900963170 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8780b3ca0 0x7fd8780b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:45.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.781+0000 7fd890c00640 1 -- 172.21.15.134:0/900963170 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd87c0e48a0 con 0x7fd89410af10 2024-07-26T11:13:45.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.781+0000 7fd893e00640 1 --2- 172.21.15.134:0/900963170 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8780b3ca0 0x7fd8780b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:45.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.782+0000 7fd893e00640 1 --2- 172.21.15.134:0/900963170 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8780b3ca0 0x7fd8780b61b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd8840095f0 tx=0x7fd884005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:45.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.783+0000 7fd890c00640 1 -- 172.21.15.134:0/900963170 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd87c0ab9f0 con 0x7fd89410af10 2024-07-26T11:13:45.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.104+0000 7f86d2800640 1 -- 172.21.15.134:0/3050789065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86cc10b120 msgr2=0x7f86cc10b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:45.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.104+0000 7f86d2800640 1 --2- 172.21.15.134:0/3050789065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86cc10b120 0x7f86cc10b530 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f86b4009900 tx=0x7f86b402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.104+0000 7f86d2800640 1 -- 172.21.15.134:0/3050789065 shutdown_connections 2024-07-26T11:13:45.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.104+0000 7f86d2800640 1 --2- 172.21.15.134:0/3050789065 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86cc10b120 0x7f86cc10b530 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.104+0000 7f86d2800640 1 --2- 172.21.15.134:0/3050789065 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86cc10ba70 0x7f86cc1122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.104+0000 7f86d2800640 1 --2- 172.21.15.134:0/3050789065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86cc1057e0 0x7f86cc105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.104+0000 7f86d2800640 1 -- 172.21.15.134:0/3050789065 >> 172.21.15.134:0/3050789065 conn(0x7f86cc1011f0 msgr2=0x7f86cc103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:45.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.104+0000 7f86d2800640 1 -- 172.21.15.134:0/3050789065 shutdown_connections 2024-07-26T11:13:45.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.104+0000 7f86d2800640 1 -- 172.21.15.134:0/3050789065 wait complete. 2024-07-26T11:13:45.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.105+0000 7f86d2800640 1 Processor -- start 2024-07-26T11:13:45.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.106+0000 7f86d2800640 1 -- start start 2024-07-26T11:13:45.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.106+0000 7f86d2800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86cc1057e0 0x7f86cc19c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:45.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.106+0000 7f86d2800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86cc10b120 0x7f86cc19c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:45.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.106+0000 7f86d2800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86cc10ba70 0x7f86cc1a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:45.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.106+0000 7f86d2800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f86cc115730 con 0x7f86cc10ba70 2024-07-26T11:13:45.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.106+0000 7f86d2800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f86cc1155b0 con 0x7f86cc1057e0 2024-07-26T11:13:45.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.106+0000 7f86d2800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f86cc1158b0 con 0x7f86cc10b120 2024-07-26T11:13:45.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.106+0000 7f86cbe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86cc10ba70 0x7f86cc1a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:45.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.106+0000 7f86cbe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86cc10ba70 0x7f86cc1a1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36270/0 (socket says 172.21.15.134:36270) 2024-07-26T11:13:45.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.106+0000 7f86cbe00640 1 -- 172.21.15.134:0/2403403389 learned_addr learned my addr 172.21.15.134:0/2403403389 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:45.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.106+0000 7f86cb400640 1 --2- 172.21.15.134:0/2403403389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86cc1057e0 0x7f86cc19c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:45.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.107+0000 7f86caa00640 1 --2- 172.21.15.134:0/2403403389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86cc10b120 0x7f86cc19c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:45.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.107+0000 7f86cbe00640 1 -- 172.21.15.134:0/2403403389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86cc10b120 msgr2=0x7f86cc19c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:45.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.107+0000 7f86cbe00640 1 --2- 172.21.15.134:0/2403403389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86cc10b120 0x7f86cc19c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.107+0000 7f86cbe00640 1 -- 172.21.15.134:0/2403403389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86cc1057e0 msgr2=0x7f86cc19c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:45.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.107+0000 7f86cbe00640 1 --2- 172.21.15.134:0/2403403389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86cc1057e0 0x7f86cc19c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.107+0000 7f86cbe00640 1 -- 172.21.15.134:0/2403403389 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86cc1a2130 con 0x7f86cc10ba70 2024-07-26T11:13:45.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.107+0000 7f86cbe00640 1 --2- 172.21.15.134:0/2403403389 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86cc10ba70 0x7f86cc1a1a00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f86c400eb10 tx=0x7f86c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.108+0000 7f86c3e00640 1 -- 172.21.15.134:0/2403403389 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c4004b20 con 0x7f86cc10ba70 2024-07-26T11:13:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.108+0000 7f86d2800640 1 -- 172.21.15.134:0/2403403389 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86cc1a2420 con 0x7f86cc10ba70 2024-07-26T11:13:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.108+0000 7f86c3e00640 1 -- 172.21.15.134:0/2403403389 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86c4004cc0 con 0x7f86cc10ba70 2024-07-26T11:13:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.108+0000 7f86c3e00640 1 -- 172.21.15.134:0/2403403389 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c4010670 con 0x7f86cc10ba70 2024-07-26T11:13:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.108+0000 7f86d2800640 1 -- 172.21.15.134:0/2403403389 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86cc1a8b80 con 0x7f86cc10ba70 2024-07-26T11:13:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.109+0000 7f86d2800640 1 -- 172.21.15.134:0/2403403389 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86cc105bf0 con 0x7f86cc10ba70 2024-07-26T11:13:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.111+0000 7f86c3e00640 1 -- 172.21.15.134:0/2403403389 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f86c4005000 con 0x7f86cc10ba70 2024-07-26T11:13:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.113+0000 7f86c3e00640 1 --2- 172.21.15.134:0/2403403389 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f86b00b3ca0 0x7f86b00b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.113+0000 7f86c3e00640 1 -- 172.21.15.134:0/2403403389 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f86c40e47e0 con 0x7f86cc10ba70 2024-07-26T11:13:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.113+0000 7f86cb400640 1 --2- 172.21.15.134:0/2403403389 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f86b00b3ca0 0x7f86b00b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.114+0000 7f86cb400640 1 --2- 172.21.15.134:0/2403403389 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f86b00b3ca0 0x7f86b00b61b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f86bc0095f0 tx=0x7f86bc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.115+0000 7f86c3e00640 1 -- 172.21.15.134:0/2403403389 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f86c4055600 con 0x7f86cc10ba70 2024-07-26T11:13:45.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.336+0000 7f86d2800640 1 -- 172.21.15.134:0/2403403389 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f86cc19d2f0 con 0x7f86cc10ba70 2024-07-26T11:13:45.867 INFO:teuthology.orchestra.run.smithi134.stdout:240518168650 2024-07-26T11:13:45.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.336+0000 7f86c3e00640 1 -- 172.21.15.134:0/2403403389 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f86cc19d2f0 con 0x7f86cc10ba70 2024-07-26T11:13:45.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.339+0000 7f86d2800640 1 -- 172.21.15.134:0/2403403389 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f86b00b3ca0 msgr2=0x7f86b00b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:45.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.339+0000 7f86d2800640 1 --2- 172.21.15.134:0/2403403389 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f86b00b3ca0 0x7f86b00b61b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f86bc0095f0 tx=0x7f86bc005f90 comp rx=0 tx=0).stop 2024-07-26T11:13:45.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.339+0000 7f86d2800640 1 -- 172.21.15.134:0/2403403389 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86cc10ba70 msgr2=0x7f86cc1a1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:45.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.339+0000 7f86d2800640 1 --2- 172.21.15.134:0/2403403389 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86cc10ba70 0x7f86cc1a1a00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f86c400eb10 tx=0x7f86c400c2d0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.340+0000 7f86d2800640 1 -- 172.21.15.134:0/2403403389 shutdown_connections 2024-07-26T11:13:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.340+0000 7f86d2800640 1 --2- 172.21.15.134:0/2403403389 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f86b00b3ca0 0x7f86b00b61b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.340+0000 7f86d2800640 1 --2- 172.21.15.134:0/2403403389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f86cc10b120 0x7f86cc19c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.340+0000 7f86d2800640 1 --2- 172.21.15.134:0/2403403389 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f86cc10ba70 0x7f86cc1a1a00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.340+0000 7f86d2800640 1 --2- 172.21.15.134:0/2403403389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86cc1057e0 0x7f86cc19c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.340+0000 7f86d2800640 1 -- 172.21.15.134:0/2403403389 >> 172.21.15.134:0/2403403389 conn(0x7f86cc1011f0 msgr2=0x7f86cc109700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.340+0000 7f86d2800640 1 -- 172.21.15.134:0/2403403389 shutdown_connections 2024-07-26T11:13:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.340+0000 7f86d2800640 1 -- 172.21.15.134:0/2403403389 wait complete. 2024-07-26T11:13:45.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.986+0000 7fd899800640 1 -- 172.21.15.134:0/900963170 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd89419cfc0 con 0x7fd89410af10 2024-07-26T11:13:45.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.987+0000 7fd890c00640 1 -- 172.21.15.134:0/900963170 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fd89419cfc0 con 0x7fd89410af10 2024-07-26T11:13:45.989 INFO:teuthology.orchestra.run.smithi134.stdout:120259084388 2024-07-26T11:13:45.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.990+0000 7fd899800640 1 -- 172.21.15.134:0/900963170 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8780b3ca0 msgr2=0x7fd8780b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:45.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.990+0000 7fd899800640 1 --2- 172.21.15.134:0/900963170 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8780b3ca0 0x7fd8780b61b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd8840095f0 tx=0x7fd884005f90 comp rx=0 tx=0).stop 2024-07-26T11:13:45.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.990+0000 7fd899800640 1 -- 172.21.15.134:0/900963170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd89410af10 msgr2=0x7fd89419c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:45.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.990+0000 7fd899800640 1 --2- 172.21.15.134:0/900963170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd89410af10 0x7fd89419c430 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd89406c270 tx=0x7fd87c00ee50 comp rx=0 tx=0).stop 2024-07-26T11:13:45.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.990+0000 7fd899800640 1 -- 172.21.15.134:0/900963170 shutdown_connections 2024-07-26T11:13:45.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.990+0000 7fd899800640 1 --2- 172.21.15.134:0/900963170 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8780b3ca0 0x7fd8780b61b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.990+0000 7fd899800640 1 --2- 172.21.15.134:0/900963170 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd89410af10 0x7fd89419c430 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.990+0000 7fd899800640 1 --2- 172.21.15.134:0/900963170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd89410b860 0x7fd8941a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.990+0000 7fd899800640 1 --2- 172.21.15.134:0/900963170 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8941055d0 0x7fd89419be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:45.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.990+0000 7fd899800640 1 -- 172.21.15.134:0/900963170 >> 172.21.15.134:0/900963170 conn(0x7fd894101040 msgr2=0x7fd89410a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:45.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.990+0000 7fd899800640 1 -- 172.21.15.134:0/900963170 shutdown_connections 2024-07-26T11:13:45.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:45.990+0000 7fd899800640 1 -- 172.21.15.134:0/900963170 wait complete. 2024-07-26T11:13:46.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:46 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2403403389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T11:13:46.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:46 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/900963170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T11:13:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:46 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2403403389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T11:13:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:46 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/900963170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T11:13:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:46 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2403403389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-07-26T11:13:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:46 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/900963170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-07-26T11:13:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:47 smithi134 ceph-mon[31966]: pgmap v414: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T11:13:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:47 smithi189 ceph-mon[36790]: pgmap v414: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T11:13:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:47 smithi193 ceph-mon[37473]: pgmap v414: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T11:13:48.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.706+0000 7f7b6a000640 1 -- 172.21.15.134:0/2459879983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b641022a0 msgr2=0x7f7b641026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:48.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.706+0000 7f7b6a000640 1 --2- 172.21.15.134:0/2459879983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b641022a0 0x7f7b641026b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7b58009900 tx=0x7f7b5802a410 comp rx=0 tx=0).stop 2024-07-26T11:13:48.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.706+0000 7f7b6a000640 1 -- 172.21.15.134:0/2459879983 shutdown_connections 2024-07-26T11:13:48.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.706+0000 7f7b6a000640 1 --2- 172.21.15.134:0/2459879983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b64108530 0x7f7b6410ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:48.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.706+0000 7f7b6a000640 1 --2- 172.21.15.134:0/2459879983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b64107be0 0x7f7b64107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:48.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.706+0000 7f7b6a000640 1 --2- 172.21.15.134:0/2459879983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b641022a0 0x7f7b641026b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:48.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.706+0000 7f7b6a000640 1 -- 172.21.15.134:0/2459879983 >> 172.21.15.134:0/2459879983 conn(0x7f7b640fdcf0 msgr2=0x7f7b64100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:48.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.706+0000 7f7b6a000640 1 -- 172.21.15.134:0/2459879983 shutdown_connections 2024-07-26T11:13:48.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.706+0000 7f7b6a000640 1 -- 172.21.15.134:0/2459879983 wait complete. 2024-07-26T11:13:48.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.708+0000 7f7b6a000640 1 Processor -- start 2024-07-26T11:13:48.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.708+0000 7f7b6a000640 1 -- start start 2024-07-26T11:13:48.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.708+0000 7f7b6a000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b641022a0 0x7f7b640fe220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:48.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.708+0000 7f7b6a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b64107be0 0x7f7b640fe760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:48.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.709+0000 7f7b6a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b64108530 0x7f7b640feca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:48.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.709+0000 7f7b6a000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7b64112200 con 0x7f7b641022a0 2024-07-26T11:13:48.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.709+0000 7f7b6a000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7b64112080 con 0x7f7b64107be0 2024-07-26T11:13:48.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.709+0000 7f7b6a000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7b64112380 con 0x7f7b64108530 2024-07-26T11:13:48.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.709+0000 7f7b63e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b64107be0 0x7f7b640fe760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:48.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.709+0000 7f7b68c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b641022a0 0x7f7b640fe220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:48.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.709+0000 7f7b63e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b64107be0 0x7f7b640fe760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46218/0 (socket says 172.21.15.134:46218) 2024-07-26T11:13:48.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.709+0000 7f7b63e00640 1 -- 172.21.15.134:0/1651557076 learned_addr learned my addr 172.21.15.134:0/1651557076 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:48.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.709+0000 7f7b69600640 1 --2- 172.21.15.134:0/1651557076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b64108530 0x7f7b640feca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:48.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.709+0000 7f7b63e00640 1 -- 172.21.15.134:0/1651557076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b64108530 msgr2=0x7f7b640feca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:48.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.709+0000 7f7b63e00640 1 --2- 172.21.15.134:0/1651557076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b64108530 0x7f7b640feca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:48.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.709+0000 7f7b63e00640 1 -- 172.21.15.134:0/1651557076 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b641022a0 msgr2=0x7f7b640fe220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:48.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.709+0000 7f7b63e00640 1 --2- 172.21.15.134:0/1651557076 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b641022a0 0x7f7b640fe220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:48.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.709+0000 7f7b63e00640 1 -- 172.21.15.134:0/1651557076 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b641a8040 con 0x7f7b64107be0 2024-07-26T11:13:48.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.710+0000 7f7b69600640 1 --2- 172.21.15.134:0/1651557076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b64108530 0x7f7b640feca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:13:48.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.710+0000 7f7b63e00640 1 --2- 172.21.15.134:0/1651557076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b64107be0 0x7f7b640fe760 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b5000e9c0 tx=0x7f7b5000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:48.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.710+0000 7f7b61600640 1 -- 172.21.15.134:0/1651557076 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5000ccc0 con 0x7f7b64107be0 2024-07-26T11:13:48.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.710+0000 7f7b6a000640 1 -- 172.21.15.134:0/1651557076 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b641a8330 con 0x7f7b64107be0 2024-07-26T11:13:48.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.710+0000 7f7b61600640 1 -- 172.21.15.134:0/1651557076 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b500054d0 con 0x7f7b64107be0 2024-07-26T11:13:48.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.710+0000 7f7b61600640 1 -- 172.21.15.134:0/1651557076 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b50010430 con 0x7f7b64107be0 2024-07-26T11:13:48.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.710+0000 7f7b6a000640 1 -- 172.21.15.134:0/1651557076 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b641a8820 con 0x7f7b64107be0 2024-07-26T11:13:48.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.712+0000 7f7b6a000640 1 -- 172.21.15.134:0/1651557076 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b641026b0 con 0x7f7b64107be0 2024-07-26T11:13:48.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.714+0000 7f7b61600640 1 -- 172.21.15.134:0/1651557076 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f7b500046c0 con 0x7f7b64107be0 2024-07-26T11:13:48.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.717+0000 7f7b61600640 1 --2- 172.21.15.134:0/1651557076 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b4c0b3c20 0x7f7b4c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:48.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.717+0000 7f7b61600640 1 -- 172.21.15.134:0/1651557076 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7b500e4530 con 0x7f7b64107be0 2024-07-26T11:13:48.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.717+0000 7f7b68c00640 1 --2- 172.21.15.134:0/1651557076 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b4c0b3c20 0x7f7b4c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:48.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.718+0000 7f7b68c00640 1 --2- 172.21.15.134:0/1651557076 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b4c0b3c20 0x7f7b4c0b6130 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7b58000c00 tx=0x7f7b58031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:48.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.720+0000 7f7b61600640 1 -- 172.21.15.134:0/1651557076 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f7b500ab680 con 0x7f7b64107be0 2024-07-26T11:13:48.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.929+0000 7f7b6a000640 1 -- 172.21.15.134:0/1651557076 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f7b641a8b10 con 0x7f7b64107be0 2024-07-26T11:13:48.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.930+0000 7f7b61600640 1 -- 172.21.15.134:0/1651557076 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7b500aaf80 con 0x7f7b64107be0 2024-07-26T11:13:48.933 INFO:teuthology.orchestra.run.smithi134.stdout:266287972420 2024-07-26T11:13:48.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.933+0000 7f7b6a000640 1 -- 172.21.15.134:0/1651557076 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b4c0b3c20 msgr2=0x7f7b4c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:48.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.933+0000 7f7b6a000640 1 --2- 172.21.15.134:0/1651557076 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b4c0b3c20 0x7f7b4c0b6130 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7b58000c00 tx=0x7f7b58031400 comp rx=0 tx=0).stop 2024-07-26T11:13:48.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.933+0000 7f7b6a000640 1 -- 172.21.15.134:0/1651557076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b64107be0 msgr2=0x7f7b640fe760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:48.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.933+0000 7f7b6a000640 1 --2- 172.21.15.134:0/1651557076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b64107be0 0x7f7b640fe760 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7b5000e9c0 tx=0x7f7b5000ee80 comp rx=0 tx=0).stop 2024-07-26T11:13:48.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.933+0000 7f7b6a000640 1 -- 172.21.15.134:0/1651557076 shutdown_connections 2024-07-26T11:13:48.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.933+0000 7f7b6a000640 1 --2- 172.21.15.134:0/1651557076 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b4c0b3c20 0x7f7b4c0b6130 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:48.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.934+0000 7f7b6a000640 1 --2- 172.21.15.134:0/1651557076 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b64108530 0x7f7b640feca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:48.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.934+0000 7f7b6a000640 1 --2- 172.21.15.134:0/1651557076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b64107be0 0x7f7b640fe760 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:48.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.934+0000 7f7b6a000640 1 --2- 172.21.15.134:0/1651557076 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b641022a0 0x7f7b640fe220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:48.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.934+0000 7f7b6a000640 1 -- 172.21.15.134:0/1651557076 >> 172.21.15.134:0/1651557076 conn(0x7f7b640fdcf0 msgr2=0x7f7b64100e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:48.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.934+0000 7f7b6a000640 1 -- 172.21.15.134:0/1651557076 shutdown_connections 2024-07-26T11:13:48.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.934+0000 7f7b6a000640 1 -- 172.21.15.134:0/1651557076 wait complete. 2024-07-26T11:13:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:49 smithi134 ceph-mon[31966]: pgmap v415: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-26T11:13:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:49 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2354593573' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T11:13:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:49 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1651557076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T11:13:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:49 smithi189 ceph-mon[36790]: pgmap v415: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-26T11:13:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:49 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2354593573' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T11:13:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:49 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1651557076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T11:13:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:49 smithi193 ceph-mon[37473]: pgmap v415: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-26T11:13:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:49 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2354593573' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-07-26T11:13:49.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:49 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1651557076' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-07-26T11:13:49.726 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364873 got 214748364877 for osd.7 2024-07-26T11:13:49.726 DEBUG:teuthology.parallel:result is None 2024-07-26T11:13:49.888 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168642 got 240518168650 for osd.8 2024-07-26T11:13:49.888 DEBUG:teuthology.parallel:result is None 2024-07-26T11:13:49.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.602+0000 7f0874e00640 1 -- 172.21.15.134:0/3021183711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0870108ef0 msgr2=0x7f0870109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:49.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.602+0000 7f0874e00640 1 --2- 172.21.15.134:0/3021183711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0870108ef0 0x7f0870109300 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f085c009900 tx=0x7f085c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:13:49.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.603+0000 7f0874e00640 1 -- 172.21.15.134:0/3021183711 shutdown_connections 2024-07-26T11:13:49.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.603+0000 7f0874e00640 1 --2- 172.21.15.134:0/3021183711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0870109840 0x7f0870110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:49.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.603+0000 7f0874e00640 1 --2- 172.21.15.134:0/3021183711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0870108ef0 0x7f0870109300 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:49.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.603+0000 7f0874e00640 1 --2- 172.21.15.134:0/3021183711 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08701035b0 0x7f08701039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:49.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.603+0000 7f0874e00640 1 -- 172.21.15.134:0/3021183711 >> 172.21.15.134:0/3021183711 conn(0x7f08700fefc0 msgr2=0x7f0870101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:49.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.603+0000 7f0874e00640 1 -- 172.21.15.134:0/3021183711 shutdown_connections 2024-07-26T11:13:49.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.603+0000 7f0874e00640 1 -- 172.21.15.134:0/3021183711 wait complete. 2024-07-26T11:13:49.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.605+0000 7f0874e00640 1 Processor -- start 2024-07-26T11:13:49.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.605+0000 7f0874e00640 1 -- start start 2024-07-26T11:13:49.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.606+0000 7f0874e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08701035b0 0x7f0870199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:49.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.606+0000 7f0874e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0870108ef0 0x7f087019a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:49.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.606+0000 7f0874e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0870109840 0x7f087019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:49.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.606+0000 7f0874e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0870113520 con 0x7f08701035b0 2024-07-26T11:13:49.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.606+0000 7f0874e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f08701133a0 con 0x7f0870109840 2024-07-26T11:13:49.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.606+0000 7f0874e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f08701136a0 con 0x7f0870108ef0 2024-07-26T11:13:49.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.606+0000 7f086de00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08701035b0 0x7f0870199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:49.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.606+0000 7f086d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0870108ef0 0x7f087019a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:49.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.606+0000 7f086de00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08701035b0 0x7f0870199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36330/0 (socket says 172.21.15.134:36330) 2024-07-26T11:13:49.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.606+0000 7f086de00640 1 -- 172.21.15.134:0/2354593573 learned_addr learned my addr 172.21.15.134:0/2354593573 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:49.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.606+0000 7f086d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0870108ef0 0x7f087019a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:47608/0 (socket says 172.21.15.134:47608) 2024-07-26T11:13:49.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.606+0000 7f086e800640 1 --2- 172.21.15.134:0/2354593573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0870109840 0x7f087019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:49.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.607+0000 7f086e800640 1 -- 172.21.15.134:0/2354593573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0870108ef0 msgr2=0x7f087019a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:49.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.607+0000 7f086e800640 1 --2- 172.21.15.134:0/2354593573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0870108ef0 0x7f087019a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:49.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.607+0000 7f086e800640 1 -- 172.21.15.134:0/2354593573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08701035b0 msgr2=0x7f0870199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:49.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.607+0000 7f086e800640 1 --2- 172.21.15.134:0/2354593573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08701035b0 0x7f0870199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:49.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.607+0000 7f086e800640 1 -- 172.21.15.134:0/2354593573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f087019fe60 con 0x7f0870109840 2024-07-26T11:13:49.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.607+0000 7f086d400640 1 --2- 172.21.15.134:0/2354593573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0870108ef0 0x7f087019a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:49.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.607+0000 7f086e800640 1 --2- 172.21.15.134:0/2354593573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0870109840 0x7f087019f730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0868009fd0 tx=0x7f086800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:49.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.608+0000 7f0866a00640 1 -- 172.21.15.134:0/2354593573 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086800ce20 con 0x7f0870109840 2024-07-26T11:13:49.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.608+0000 7f0874e00640 1 -- 172.21.15.134:0/2354593573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08701a0150 con 0x7f0870109840 2024-07-26T11:13:49.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.608+0000 7f0866a00640 1 -- 172.21.15.134:0/2354593573 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0868005020 con 0x7f0870109840 2024-07-26T11:13:49.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.608+0000 7f0866a00640 1 -- 172.21.15.134:0/2354593573 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0868010690 con 0x7f0870109840 2024-07-26T11:13:49.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.608+0000 7f0874e00640 1 -- 172.21.15.134:0/2354593573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0870075020 con 0x7f0870109840 2024-07-26T11:13:49.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.610+0000 7f0874e00640 1 -- 172.21.15.134:0/2354593573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08701039c0 con 0x7f0870109840 2024-07-26T11:13:49.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.612+0000 7f0866a00640 1 -- 172.21.15.134:0/2354593573 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f08680054d0 con 0x7f0870109840 2024-07-26T11:13:49.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.614+0000 7f0866a00640 1 --2- 172.21.15.134:0/2354593573 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f08540b3c20 0x7f08540b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:49.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.615+0000 7f0866a00640 1 -- 172.21.15.134:0/2354593573 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f08680e4810 con 0x7f0870109840 2024-07-26T11:13:49.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.615+0000 7f086de00640 1 --2- 172.21.15.134:0/2354593573 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f08540b3c20 0x7f08540b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:49.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.615+0000 7f086de00640 1 --2- 172.21.15.134:0/2354593573 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f08540b3c20 0x7f08540b6130 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0860010e40 tx=0x7f0860012040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:49.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.617+0000 7f0866a00640 1 -- 172.21.15.134:0/2354593573 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f08680ab960 con 0x7f0870109840 2024-07-26T11:13:49.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.839+0000 7f0874e00640 1 -- 172.21.15.134:0/2354593573 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f087010b5f0 con 0x7f0870109840 2024-07-26T11:13:49.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.839+0000 7f0866a00640 1 -- 172.21.15.134:0/2354593573 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f08680ab260 con 0x7f0870109840 2024-07-26T11:13:49.952 INFO:teuthology.orchestra.run.smithi134.stdout:60129542256 2024-07-26T11:13:49.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.842+0000 7f0874e00640 1 -- 172.21.15.134:0/2354593573 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f08540b3c20 msgr2=0x7f08540b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:49.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.842+0000 7f0874e00640 1 --2- 172.21.15.134:0/2354593573 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f08540b3c20 0x7f08540b6130 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0860010e40 tx=0x7f0860012040 comp rx=0 tx=0).stop 2024-07-26T11:13:49.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.842+0000 7f0874e00640 1 -- 172.21.15.134:0/2354593573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0870109840 msgr2=0x7f087019f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:49.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.842+0000 7f0874e00640 1 --2- 172.21.15.134:0/2354593573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0870109840 0x7f087019f730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0868009fd0 tx=0x7f086800ef10 comp rx=0 tx=0).stop 2024-07-26T11:13:49.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.842+0000 7f0874e00640 1 -- 172.21.15.134:0/2354593573 shutdown_connections 2024-07-26T11:13:49.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.842+0000 7f0874e00640 1 --2- 172.21.15.134:0/2354593573 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f08540b3c20 0x7f08540b6130 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:49.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.842+0000 7f0874e00640 1 --2- 172.21.15.134:0/2354593573 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0870108ef0 0x7f087019a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:49.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.842+0000 7f0874e00640 1 --2- 172.21.15.134:0/2354593573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0870109840 0x7f087019f730 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:49.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.842+0000 7f0874e00640 1 --2- 172.21.15.134:0/2354593573 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08701035b0 0x7f0870199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:49.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.842+0000 7f0874e00640 1 -- 172.21.15.134:0/2354593573 >> 172.21.15.134:0/2354593573 conn(0x7f08700fefc0 msgr2=0x7f0870102410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:49.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.843+0000 7f0874e00640 1 -- 172.21.15.134:0/2354593573 shutdown_connections 2024-07-26T11:13:49.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:48.843+0000 7f0874e00640 1 -- 172.21.15.134:0/2354593573 wait complete. 2024-07-26T11:13:51.038 INFO:tasks.cephadm.ceph_manager.ceph:need seq 266287972415 got 266287972420 for osd.9 2024-07-26T11:13:51.038 DEBUG:teuthology.parallel:result is None 2024-07-26T11:13:51.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.310+0000 7f6e41a00640 1 -- 172.21.15.134:0/3362713749 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e3c10ba70 msgr2=0x7f6e3c1122a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:51.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.310+0000 7f6e41a00640 1 --2- 172.21.15.134:0/3362713749 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e3c10ba70 0x7f6e3c1122a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6e34009ca0 tx=0x7f6e3402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:13:51.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.310+0000 7f6e41a00640 1 -- 172.21.15.134:0/3362713749 shutdown_connections 2024-07-26T11:13:51.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.310+0000 7f6e41a00640 1 --2- 172.21.15.134:0/3362713749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e3c10b120 0x7f6e3c10b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:51.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.310+0000 7f6e41a00640 1 --2- 172.21.15.134:0/3362713749 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e3c10ba70 0x7f6e3c1122a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:51.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.310+0000 7f6e41a00640 1 --2- 172.21.15.134:0/3362713749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e3c1057e0 0x7f6e3c105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:51.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.310+0000 7f6e41a00640 1 -- 172.21.15.134:0/3362713749 >> 172.21.15.134:0/3362713749 conn(0x7f6e3c1011f0 msgr2=0x7f6e3c103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:51.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.310+0000 7f6e41a00640 1 -- 172.21.15.134:0/3362713749 shutdown_connections 2024-07-26T11:13:51.314 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:51 smithi134 ceph-mon[31966]: pgmap v416: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:51.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.311+0000 7f6e41a00640 1 -- 172.21.15.134:0/3362713749 wait complete. 2024-07-26T11:13:51.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.312+0000 7f6e41a00640 1 Processor -- start 2024-07-26T11:13:51.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.312+0000 7f6e41a00640 1 -- start start 2024-07-26T11:13:51.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.313+0000 7f6e41a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e3c1057e0 0x7f6e3c19c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:51.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.313+0000 7f6e41a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e3c10b120 0x7f6e3c19c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:51.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.313+0000 7f6e41a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e3c10ba70 0x7f6e3c1a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:51.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.313+0000 7f6e41a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3c115740 con 0x7f6e3c10b120 2024-07-26T11:13:51.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.313+0000 7f6e41a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3c1155c0 con 0x7f6e3c1057e0 2024-07-26T11:13:51.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.313+0000 7f6e41a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3c1158c0 con 0x7f6e3c10ba70 2024-07-26T11:13:51.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.313+0000 7f6e3aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e3c1057e0 0x7f6e3c19c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:51.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.313+0000 7f6e3aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e3c1057e0 0x7f6e3c19c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:37072/0 (socket says 172.21.15.134:37072) 2024-07-26T11:13:51.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.313+0000 7f6e3aa00640 1 -- 172.21.15.134:0/2639646450 learned_addr learned my addr 172.21.15.134:0/2639646450 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:51.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.313+0000 7f6e3b400640 1 --2- 172.21.15.134:0/2639646450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e3c10ba70 0x7f6e3c1a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:51.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.313+0000 7f6e3a000640 1 --2- 172.21.15.134:0/2639646450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e3c10b120 0x7f6e3c19c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:51.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.314+0000 7f6e3aa00640 1 -- 172.21.15.134:0/2639646450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e3c10ba70 msgr2=0x7f6e3c1a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:51.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.314+0000 7f6e3aa00640 1 --2- 172.21.15.134:0/2639646450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e3c10ba70 0x7f6e3c1a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:51.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.314+0000 7f6e3aa00640 1 -- 172.21.15.134:0/2639646450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e3c10b120 msgr2=0x7f6e3c19c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:51.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.314+0000 7f6e3aa00640 1 --2- 172.21.15.134:0/2639646450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e3c10b120 0x7f6e3c19c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:51.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.314+0000 7f6e3aa00640 1 -- 172.21.15.134:0/2639646450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e3c1a2020 con 0x7f6e3c1057e0 2024-07-26T11:13:51.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.314+0000 7f6e3aa00640 1 --2- 172.21.15.134:0/2639646450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e3c1057e0 0x7f6e3c19c0b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00ea60 tx=0x7f6e2c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:51.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.314+0000 7f6e33400640 1 -- 172.21.15.134:0/2639646450 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c00cde0 con 0x7f6e3c1057e0 2024-07-26T11:13:51.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.314+0000 7f6e33400640 1 -- 172.21.15.134:0/2639646450 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6e2c0035f0 con 0x7f6e3c1057e0 2024-07-26T11:13:51.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.314+0000 7f6e41a00640 1 -- 172.21.15.134:0/2639646450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e3c1a2310 con 0x7f6e3c1057e0 2024-07-26T11:13:51.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.314+0000 7f6e33400640 1 -- 172.21.15.134:0/2639646450 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c010690 con 0x7f6e3c1057e0 2024-07-26T11:13:51.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.314+0000 7f6e41a00640 1 -- 172.21.15.134:0/2639646450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e3c1a8a70 con 0x7f6e3c1057e0 2024-07-26T11:13:51.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.316+0000 7f6e41a00640 1 -- 172.21.15.134:0/2639646450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e3c105bf0 con 0x7f6e3c1057e0 2024-07-26T11:13:51.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.318+0000 7f6e33400640 1 -- 172.21.15.134:0/2639646450 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f6e2c0037a0 con 0x7f6e3c1057e0 2024-07-26T11:13:51.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.319+0000 7f6e33400640 1 --2- 172.21.15.134:0/2639646450 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6e200b3ca0 0x7f6e200b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:51.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.320+0000 7f6e33400640 1 -- 172.21.15.134:0/2639646450 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f6e2c0e4440 con 0x7f6e3c1057e0 2024-07-26T11:13:51.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.320+0000 7f6e3a000640 1 --2- 172.21.15.134:0/2639646450 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6e200b3ca0 0x7f6e200b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:51.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.320+0000 7f6e3a000640 1 --2- 172.21.15.134:0/2639646450 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6e200b3ca0 0x7f6e200b61b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c1039c0 tx=0x7f6e28005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:51.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.324+0000 7f6e33400640 1 -- 172.21.15.134:0/2639646450 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6e2c0ab590 con 0x7f6e3c1057e0 2024-07-26T11:13:51.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:51 smithi193 ceph-mon[37473]: pgmap v416: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:51.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:51 smithi189 ceph-mon[36790]: pgmap v416: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:51.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.538+0000 7f6e41a00640 1 -- 172.21.15.134:0/2639646450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f6e3c19d1e0 con 0x7f6e3c1057e0 2024-07-26T11:13:51.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.538+0000 7f6e33400640 1 -- 172.21.15.134:0/2639646450 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6e2c0aae90 con 0x7f6e3c1057e0 2024-07-26T11:13:51.541 INFO:teuthology.orchestra.run.smithi134.stdout:193273528405 2024-07-26T11:13:51.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.541+0000 7f6e41a00640 1 -- 172.21.15.134:0/2639646450 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6e200b3ca0 msgr2=0x7f6e200b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:51.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.541+0000 7f6e41a00640 1 --2- 172.21.15.134:0/2639646450 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6e200b3ca0 0x7f6e200b61b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c1039c0 tx=0x7f6e28005f90 comp rx=0 tx=0).stop 2024-07-26T11:13:51.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.541+0000 7f6e41a00640 1 -- 172.21.15.134:0/2639646450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e3c1057e0 msgr2=0x7f6e3c19c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:51.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.541+0000 7f6e41a00640 1 --2- 172.21.15.134:0/2639646450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e3c1057e0 0x7f6e3c19c0b0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c00ea60 tx=0x7f6e2c00ef20 comp rx=0 tx=0).stop 2024-07-26T11:13:51.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.541+0000 7f6e41a00640 1 -- 172.21.15.134:0/2639646450 shutdown_connections 2024-07-26T11:13:51.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.541+0000 7f6e41a00640 1 --2- 172.21.15.134:0/2639646450 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6e200b3ca0 0x7f6e200b61b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:51.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.541+0000 7f6e41a00640 1 --2- 172.21.15.134:0/2639646450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e3c10ba70 0x7f6e3c1a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:51.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.541+0000 7f6e41a00640 1 --2- 172.21.15.134:0/2639646450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6e3c10b120 0x7f6e3c19c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:51.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.541+0000 7f6e41a00640 1 --2- 172.21.15.134:0/2639646450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e3c1057e0 0x7f6e3c19c0b0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:51.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.541+0000 7f6e41a00640 1 -- 172.21.15.134:0/2639646450 >> 172.21.15.134:0/2639646450 conn(0x7f6e3c1011f0 msgr2=0x7f6e3c10a6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:51.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.541+0000 7f6e41a00640 1 -- 172.21.15.134:0/2639646450 shutdown_connections 2024-07-26T11:13:51.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:51.541+0000 7f6e41a00640 1 -- 172.21.15.134:0/2639646450 wait complete. 2024-07-26T11:13:51.642 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542249 got 60129542256 for osd.1 2024-07-26T11:13:51.642 DEBUG:teuthology.parallel:result is None 2024-07-26T11:13:52.375 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:52 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2639646450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T11:13:52.439 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378724 got 81604378730 for osd.2 2024-07-26T11:13:52.439 DEBUG:teuthology.parallel:result is None 2024-07-26T11:13:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:52 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2639646450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T11:13:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:52 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2639646450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-07-26T11:13:52.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.703+0000 7fc67fe00640 1 -- 172.21.15.134:0/1242490483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680075800 msgr2=0x7fc680075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:52.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.703+0000 7fc67fe00640 1 --2- 172.21.15.134:0/1242490483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680075800 0x7fc680075c10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc668009900 tx=0x7fc66802a410 comp rx=0 tx=0).stop 2024-07-26T11:13:52.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.703+0000 7fc67fe00640 1 -- 172.21.15.134:0/1242490483 shutdown_connections 2024-07-26T11:13:52.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.703+0000 7fc67fe00640 1 --2- 172.21.15.134:0/1242490483 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc680113cb0 0x7fc680116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:52.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.703+0000 7fc67fe00640 1 --2- 172.21.15.134:0/1242490483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680075800 0x7fc680075c10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:52.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.703+0000 7fc67fe00640 1 --2- 172.21.15.134:0/1242490483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc680076db0 0x7fc6800751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:52.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.703+0000 7fc67fe00640 1 -- 172.21.15.134:0/1242490483 >> 172.21.15.134:0/1242490483 conn(0x7fc6800fee70 msgr2=0x7fc6801012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:52.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.703+0000 7fc67fe00640 1 -- 172.21.15.134:0/1242490483 shutdown_connections 2024-07-26T11:13:52.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.703+0000 7fc67fe00640 1 -- 172.21.15.134:0/1242490483 wait complete. 2024-07-26T11:13:52.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.704+0000 7fc67fe00640 1 Processor -- start 2024-07-26T11:13:52.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.705+0000 7fc67fe00640 1 -- start start 2024-07-26T11:13:52.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.705+0000 7fc67fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc680075800 0x7fc6801a0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:52.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.705+0000 7fc67fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680076db0 0x7fc6801a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:52.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.705+0000 7fc67fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc680113cb0 0x7fc6801a5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:52.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.705+0000 7fc67fe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc6801198f0 con 0x7fc680113cb0 2024-07-26T11:13:52.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.705+0000 7fc67fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc680119770 con 0x7fc680076db0 2024-07-26T11:13:52.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.705+0000 7fc67fe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc680119a70 con 0x7fc680075800 2024-07-26T11:13:52.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.705+0000 7fc67e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680076db0 0x7fc6801a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:52.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.705+0000 7fc67f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc680113cb0 0x7fc6801a5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:52.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.705+0000 7fc67ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc680075800 0x7fc6801a0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:52.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.706+0000 7fc67ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc680075800 0x7fc6801a0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57716/0 (socket says 172.21.15.134:57716) 2024-07-26T11:13:52.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.706+0000 7fc67ea00640 1 -- 172.21.15.134:0/424870024 learned_addr learned my addr 172.21.15.134:0/424870024 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:52.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.706+0000 7fc67f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc680113cb0 0x7fc6801a5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38392/0 (socket says 172.21.15.134:38392) 2024-07-26T11:13:52.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.706+0000 7fc67e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680076db0 0x7fc6801a0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:37080/0 (socket says 172.21.15.134:37080) 2024-07-26T11:13:52.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.706+0000 7fc67ea00640 1 -- 172.21.15.134:0/424870024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680076db0 msgr2=0x7fc6801a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:52.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.706+0000 7fc67ea00640 1 --2- 172.21.15.134:0/424870024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680076db0 0x7fc6801a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:52.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.706+0000 7fc67ea00640 1 -- 172.21.15.134:0/424870024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc680113cb0 msgr2=0x7fc6801a5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:52.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.706+0000 7fc67ea00640 1 --2- 172.21.15.134:0/424870024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc680113cb0 0x7fc6801a5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:52.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.706+0000 7fc67ea00640 1 -- 172.21.15.134:0/424870024 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6801a62f0 con 0x7fc680075800 2024-07-26T11:13:52.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.706+0000 7fc67ea00640 1 --2- 172.21.15.134:0/424870024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc680075800 0x7fc6801a0380 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc67000e9f0 tx=0x7fc67000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:52.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.707+0000 7fc677400640 1 -- 172.21.15.134:0/424870024 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67000cd80 con 0x7fc680075800 2024-07-26T11:13:52.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.707+0000 7fc677400640 1 -- 172.21.15.134:0/424870024 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6700054d0 con 0x7fc680075800 2024-07-26T11:13:52.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.707+0000 7fc67fe00640 1 -- 172.21.15.134:0/424870024 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6801a65e0 con 0x7fc680075800 2024-07-26T11:13:52.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.707+0000 7fc677400640 1 -- 172.21.15.134:0/424870024 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670010690 con 0x7fc680075800 2024-07-26T11:13:52.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.707+0000 7fc67fe00640 1 -- 172.21.15.134:0/424870024 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6801acd40 con 0x7fc680075800 2024-07-26T11:13:52.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.708+0000 7fc67fe00640 1 -- 172.21.15.134:0/424870024 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc680075c10 con 0x7fc680075800 2024-07-26T11:13:52.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.711+0000 7fc677400640 1 -- 172.21.15.134:0/424870024 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fc670004e30 con 0x7fc680075800 2024-07-26T11:13:52.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.712+0000 7fc677400640 1 --2- 172.21.15.134:0/424870024 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc6640b3c20 0x7fc6640b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:52.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.712+0000 7fc677400640 1 -- 172.21.15.134:0/424870024 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc6700e46f0 con 0x7fc680075800 2024-07-26T11:13:52.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.712+0000 7fc67e000640 1 --2- 172.21.15.134:0/424870024 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc6640b3c20 0x7fc6640b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:52.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.713+0000 7fc67e000640 1 --2- 172.21.15.134:0/424870024 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc6640b3c20 0x7fc6640b6130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc668000c00 tx=0x7fc668031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:52.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.714+0000 7fc677400640 1 -- 172.21.15.134:0/424870024 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc670055600 con 0x7fc680075800 2024-07-26T11:13:52.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.914+0000 7fc67fe00640 1 -- 172.21.15.134:0/424870024 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fc6801a14b0 con 0x7fc680075800 2024-07-26T11:13:52.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.915+0000 7fc677400640 1 -- 172.21.15.134:0/424870024 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc6801a14b0 con 0x7fc680075800 2024-07-26T11:13:52.918 INFO:teuthology.orchestra.run.smithi134.stdout:38654705781 2024-07-26T11:13:52.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.919+0000 7fc67fe00640 1 -- 172.21.15.134:0/424870024 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc6640b3c20 msgr2=0x7fc6640b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:52.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.919+0000 7fc67fe00640 1 --2- 172.21.15.134:0/424870024 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc6640b3c20 0x7fc6640b6130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc668000c00 tx=0x7fc668031400 comp rx=0 tx=0).stop 2024-07-26T11:13:52.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.919+0000 7fc67fe00640 1 -- 172.21.15.134:0/424870024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc680075800 msgr2=0x7fc6801a0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:52.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.919+0000 7fc67fe00640 1 --2- 172.21.15.134:0/424870024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc680075800 0x7fc6801a0380 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc67000e9f0 tx=0x7fc67000edf0 comp rx=0 tx=0).stop 2024-07-26T11:13:52.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.919+0000 7fc67fe00640 1 -- 172.21.15.134:0/424870024 shutdown_connections 2024-07-26T11:13:52.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.919+0000 7fc67fe00640 1 --2- 172.21.15.134:0/424870024 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc6640b3c20 0x7fc6640b6130 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:52.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.919+0000 7fc67fe00640 1 --2- 172.21.15.134:0/424870024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc680113cb0 0x7fc6801a5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:52.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.919+0000 7fc67fe00640 1 --2- 172.21.15.134:0/424870024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc680076db0 0x7fc6801a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:52.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.919+0000 7fc67fe00640 1 --2- 172.21.15.134:0/424870024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc680075800 0x7fc6801a0380 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:52.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.919+0000 7fc67fe00640 1 -- 172.21.15.134:0/424870024 >> 172.21.15.134:0/424870024 conn(0x7fc6800fee70 msgr2=0x7fc68010fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:52.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.920+0000 7fc67fe00640 1 -- 172.21.15.134:0/424870024 shutdown_connections 2024-07-26T11:13:52.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:52.920+0000 7fc67fe00640 1 -- 172.21.15.134:0/424870024 wait complete. 2024-07-26T11:13:53.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:53 smithi134 ceph-mon[31966]: pgmap v417: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:53.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:53 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/424870024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T11:13:53.396 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528396 got 193273528405 for osd.6 2024-07-26T11:13:53.396 DEBUG:teuthology.parallel:result is None 2024-07-26T11:13:53.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:53 smithi189 ceph-mon[36790]: pgmap v417: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:53.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:53 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/424870024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T11:13:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:53 smithi193 ceph-mon[37473]: pgmap v417: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:13:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:53 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/424870024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-07-26T11:13:53.792 INFO:tasks.cephadm.ceph_manager.ceph:need seq 120259084383 got 120259084388 for osd.3 2024-07-26T11:13:53.792 DEBUG:teuthology.parallel:result is None 2024-07-26T11:13:54.265 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705774 got 38654705781 for osd.0 2024-07-26T11:13:54.265 DEBUG:teuthology.parallel:result is None 2024-07-26T11:13:54.265 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-07-26T11:13:54.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph pg dump --format=json 2024-07-26T11:13:54.343 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:54 smithi134 ceph-mon[31966]: pgmap v418: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:13:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:54 smithi189 ceph-mon[36790]: pgmap v418: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:13:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:54 smithi193 ceph-mon[37473]: pgmap v418: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:13:54.713 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:13:57.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.105+0000 7f086ea00640 1 -- 172.21.15.134:0/33536399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f086810b120 msgr2=0x7f086810b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:57.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.105+0000 7f086ea00640 1 --2- 172.21.15.134:0/33536399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f086810b120 0x7f086810b530 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0854009860 tx=0x7f085402a340 comp rx=0 tx=0).stop 2024-07-26T11:13:57.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.105+0000 7f086ea00640 1 -- 172.21.15.134:0/33536399 shutdown_connections 2024-07-26T11:13:57.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.105+0000 7f086ea00640 1 --2- 172.21.15.134:0/33536399 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f086810ba70 0x7f08681122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:57.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.105+0000 7f086ea00640 1 --2- 172.21.15.134:0/33536399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f086810b120 0x7f086810b530 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:57.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.105+0000 7f086ea00640 1 --2- 172.21.15.134:0/33536399 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08681057e0 0x7f0868105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:57.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.105+0000 7f086ea00640 1 -- 172.21.15.134:0/33536399 >> 172.21.15.134:0/33536399 conn(0x7f08681011f0 msgr2=0x7f0868103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:57.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.105+0000 7f086ea00640 1 -- 172.21.15.134:0/33536399 shutdown_connections 2024-07-26T11:13:57.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.105+0000 7f086ea00640 1 -- 172.21.15.134:0/33536399 wait complete. 2024-07-26T11:13:57.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.106+0000 7f086ea00640 1 Processor -- start 2024-07-26T11:13:57.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.106+0000 7f086ea00640 1 -- start start 2024-07-26T11:13:57.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.107+0000 7f086ea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08681057e0 0x7f086819c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:57.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.107+0000 7f086ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f086810b120 0x7f086819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:57.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.107+0000 7f086ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f086810ba70 0x7f08681a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:57.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.107+0000 7f086ea00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f08681156e0 con 0x7f08681057e0 2024-07-26T11:13:57.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.107+0000 7f086ea00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0868115560 con 0x7f086810b120 2024-07-26T11:13:57.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.107+0000 7f086ea00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0868115860 con 0x7f086810ba70 2024-07-26T11:13:57.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.107+0000 7f0867e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08681057e0 0x7f086819c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:57.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.107+0000 7f0867400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f086810b120 0x7f086819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:57.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.107+0000 7f0867400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f086810b120 0x7f086819c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:37096/0 (socket says 172.21.15.134:37096) 2024-07-26T11:13:57.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.107+0000 7f0867400640 1 -- 172.21.15.134:0/1608007847 learned_addr learned my addr 172.21.15.134:0/1608007847 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:13:57.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.107+0000 7f0867e00640 1 --2- 172.21.15.134:0/1608007847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08681057e0 0x7f086819c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38416/0 (socket says 172.21.15.134:38416) 2024-07-26T11:13:57.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.107+0000 7f086ca00640 1 --2- 172.21.15.134:0/1608007847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f086810ba70 0x7f08681a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:57.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.108+0000 7f0867400640 1 -- 172.21.15.134:0/1608007847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f086810ba70 msgr2=0x7f08681a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:57.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.108+0000 7f0867400640 1 --2- 172.21.15.134:0/1608007847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f086810ba70 0x7f08681a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:57.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.108+0000 7f0867400640 1 -- 172.21.15.134:0/1608007847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08681057e0 msgr2=0x7f086819c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:57.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.108+0000 7f0867400640 1 --2- 172.21.15.134:0/1608007847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08681057e0 0x7f086819c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:57.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.108+0000 7f0867400640 1 -- 172.21.15.134:0/1608007847 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08681a20a0 con 0x7f086810b120 2024-07-26T11:13:57.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.108+0000 7f0867e00640 1 --2- 172.21.15.134:0/1608007847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08681057e0 0x7f086819c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:13:57.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.108+0000 7f0867400640 1 --2- 172.21.15.134:0/1608007847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f086810b120 0x7f086819c6d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0854035040 tx=0x7f0854032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:57.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.109+0000 7f0864c00640 1 -- 172.21.15.134:0/1608007847 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854040020 con 0x7f086810b120 2024-07-26T11:13:57.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.109+0000 7f086ea00640 1 -- 172.21.15.134:0/1608007847 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08681a2330 con 0x7f086810b120 2024-07-26T11:13:57.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.109+0000 7f0864c00640 1 -- 172.21.15.134:0/1608007847 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0854030e10 con 0x7f086810b120 2024-07-26T11:13:57.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.109+0000 7f0864c00640 1 -- 172.21.15.134:0/1608007847 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854037040 con 0x7f086810b120 2024-07-26T11:13:57.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.109+0000 7f086ea00640 1 -- 172.21.15.134:0/1608007847 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08681a8af0 con 0x7f086810b120 2024-07-26T11:13:57.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.110+0000 7f086ea00640 1 -- 172.21.15.134:0/1608007847 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0868106a60 con 0x7f086810b120 2024-07-26T11:13:57.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.112+0000 7f0864c00640 1 -- 172.21.15.134:0/1608007847 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f0854043020 con 0x7f086810b120 2024-07-26T11:13:57.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.114+0000 7f0864c00640 1 --2- 172.21.15.134:0/1608007847 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f084c0b3c20 0x7f084c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:13:57.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.114+0000 7f0864c00640 1 -- 172.21.15.134:0/1608007847 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f08541041c0 con 0x7f086810b120 2024-07-26T11:13:57.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.114+0000 7f0867e00640 1 --2- 172.21.15.134:0/1608007847 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f084c0b3c20 0x7f084c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:13:57.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.115+0000 7f0867e00640 1 --2- 172.21.15.134:0/1608007847 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f084c0b3c20 0x7f084c0b6130 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0858009fc0 tx=0x7f0858005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:13:57.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.116+0000 7f0864c00640 1 -- 172.21.15.134:0/1608007847 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f085404a020 con 0x7f086810b120 2024-07-26T11:13:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:57 smithi134 ceph-mon[31966]: pgmap v419: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:57.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.311+0000 7f086ea00640 1 -- 172.21.15.134:0/1608007847 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f086810a000 con 0x7f084c0b3c20 2024-07-26T11:13:57.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.320+0000 7f0864c00640 1 -- 172.21.15.134:0/1608007847 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340514 (secure 0 0 0) 0x7f086810a000 con 0x7f084c0b3c20 2024-07-26T11:13:57.327 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:13:57.331 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-07-26T11:13:57.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.328+0000 7f086ea00640 1 -- 172.21.15.134:0/1608007847 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f084c0b3c20 msgr2=0x7f084c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:57.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.328+0000 7f086ea00640 1 --2- 172.21.15.134:0/1608007847 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f084c0b3c20 0x7f084c0b6130 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0858009fc0 tx=0x7f0858005f90 comp rx=0 tx=0).stop 2024-07-26T11:13:57.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.328+0000 7f086ea00640 1 -- 172.21.15.134:0/1608007847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f086810b120 msgr2=0x7f086819c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:13:57.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.328+0000 7f086ea00640 1 --2- 172.21.15.134:0/1608007847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f086810b120 0x7f086819c6d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0854035040 tx=0x7f0854032d50 comp rx=0 tx=0).stop 2024-07-26T11:13:57.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.328+0000 7f086ea00640 1 -- 172.21.15.134:0/1608007847 shutdown_connections 2024-07-26T11:13:57.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.328+0000 7f086ea00640 1 --2- 172.21.15.134:0/1608007847 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f084c0b3c20 0x7f084c0b6130 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:57.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.328+0000 7f086ea00640 1 --2- 172.21.15.134:0/1608007847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f086810ba70 0x7f08681a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:57.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.328+0000 7f086ea00640 1 --2- 172.21.15.134:0/1608007847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f086810b120 0x7f086819c6d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:57.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.328+0000 7f086ea00640 1 --2- 172.21.15.134:0/1608007847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f08681057e0 0x7f086819c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:13:57.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.328+0000 7f086ea00640 1 -- 172.21.15.134:0/1608007847 >> 172.21.15.134:0/1608007847 conn(0x7f08681011f0 msgr2=0x7f08681098f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:13:57.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.328+0000 7f086ea00640 1 -- 172.21.15.134:0/1608007847 shutdown_connections 2024-07-26T11:13:57.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:13:57.328+0000 7f086ea00640 1 -- 172.21.15.134:0/1608007847 wait complete. 2024-07-26T11:13:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:57 smithi189 ceph-mon[36790]: pgmap v419: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:57 smithi193 ceph-mon[37473]: pgmap v419: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:13:58.178 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":419,"stamp":"2024-07-26T11:13:55.985821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664325,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1348,"num_read_kb":781,"num_write":383,"num_write_kb":2303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":2954034,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":816,"ondisk_log_size":816,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375884,"kb_used_data":14252,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565940,"statfs":{"total":1151940427776,"available":1151555522560,"internally_reserved":0,"allocated":14594048,"data_stored":8272541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007538"},"pg_stats":[{"pgid":"2.1f","version":"90'13","reported_seq":190,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:50.334798+0000","last_change":"2024-07-26T11:10:34.985984+0000","last_active":"2024-07-26T11:13:50.334798+0000","last_peered":"2024-07-26T11:13:50.334798+0000","last_clean":"2024-07-26T11:13:50.334798+0000","last_became_active":"2024-07-26T11:10:34.980894+0000","last_became_peered":"2024-07-26T11:10:34.980894+0000","last_unstale":"2024-07-26T11:13:50.334798+0000","last_undegraded":"2024-07-26T11:13:50.334798+0000","last_fullsized":"2024-07-26T11:13:50.334798+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:55.718875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874263+0000","last_change":"2024-07-26T11:10:22.320946+0000","last_active":"2024-07-26T11:13:33.874263+0000","last_peered":"2024-07-26T11:13:33.874263+0000","last_clean":"2024-07-26T11:13:33.874263+0000","last_became_active":"2024-07-26T11:10:22.316181+0000","last_became_peered":"2024-07-26T11:10:22.316181+0000","last_unstale":"2024-07-26T11:13:33.874263+0000","last_undegraded":"2024-07-26T11:13:33.874263+0000","last_fullsized":"2024-07-26T11:13:33.874263+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:13.641862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.155734+0000","last_change":"2024-07-26T11:10:35.984460+0000","last_active":"2024-07-26T11:11:35.155734+0000","last_peered":"2024-07-26T11:11:35.155734+0000","last_clean":"2024-07-26T11:11:35.155734+0000","last_became_active":"2024-07-26T11:10:35.983922+0000","last_became_peered":"2024-07-26T11:10:35.983922+0000","last_unstale":"2024-07-26T11:11:35.155734+0000","last_undegraded":"2024-07-26T11:11:35.155734+0000","last_fullsized":"2024-07-26T11:11:35.155734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:51:46.140910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:48.051375+0000","last_change":"2024-07-26T11:10:18.058553+0000","last_active":"2024-07-26T11:11:48.051375+0000","last_peered":"2024-07-26T11:11:48.051375+0000","last_clean":"2024-07-26T11:11:48.051375+0000","last_became_active":"2024-07-26T11:10:18.058034+0000","last_became_peered":"2024-07-26T11:10:18.058034+0000","last_unstale":"2024-07-26T11:11:48.051375+0000","last_undegraded":"2024-07-26T11:11:48.051375+0000","last_fullsized":"2024-07-26T11:11:48.051375+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:48:58.868988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.893887+0000","last_change":"2024-07-26T11:10:22.326760+0000","last_active":"2024-07-26T11:13:33.893887+0000","last_peered":"2024-07-26T11:13:33.893887+0000","last_clean":"2024-07-26T11:13:33.893887+0000","last_became_active":"2024-07-26T11:10:22.326275+0000","last_became_peered":"2024-07-26T11:10:22.326275+0000","last_unstale":"2024-07-26T11:13:33.893887+0000","last_undegraded":"2024-07-26T11:13:33.893887+0000","last_fullsized":"2024-07-26T11:13:33.893887+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:51:57.461453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.406615+0000","last_change":"2024-07-26T11:10:35.997585+0000","last_active":"2024-07-26T11:11:35.406615+0000","last_peered":"2024-07-26T11:11:35.406615+0000","last_clean":"2024-07-26T11:11:35.406615+0000","last_became_active":"2024-07-26T11:10:35.997274+0000","last_became_peered":"2024-07-26T11:10:35.997274+0000","last_unstale":"2024-07-26T11:11:35.406615+0000","last_undegraded":"2024-07-26T11:11:35.406615+0000","last_fullsized":"2024-07-26T11:11:35.406615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:43.553396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.096244+0000","last_change":"2024-07-26T11:10:18.034964+0000","last_active":"2024-07-26T11:11:51.096244+0000","last_peered":"2024-07-26T11:11:51.096244+0000","last_clean":"2024-07-26T11:11:51.096244+0000","last_became_active":"2024-07-26T11:10:18.034569+0000","last_became_peered":"2024-07-26T11:10:18.034569+0000","last_unstale":"2024-07-26T11:11:51.096244+0000","last_undegraded":"2024-07-26T11:11:51.096244+0000","last_fullsized":"2024-07-26T11:11:51.096244+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:00:46.542458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.882415+0000","last_change":"2024-07-26T11:10:22.310695+0000","last_active":"2024-07-26T11:13:33.882415+0000","last_peered":"2024-07-26T11:13:33.882415+0000","last_clean":"2024-07-26T11:13:33.882415+0000","last_became_active":"2024-07-26T11:10:22.310142+0000","last_became_peered":"2024-07-26T11:10:22.310142+0000","last_unstale":"2024-07-26T11:13:33.882415+0000","last_undegraded":"2024-07-26T11:13:33.882415+0000","last_fullsized":"2024-07-26T11:13:33.882415+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:49:18.890802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.407144+0000","last_change":"2024-07-26T11:10:35.991418+0000","last_active":"2024-07-26T11:11:35.407144+0000","last_peered":"2024-07-26T11:11:35.407144+0000","last_clean":"2024-07-26T11:11:35.407144+0000","last_became_active":"2024-07-26T11:10:35.990934+0000","last_became_peered":"2024-07-26T11:10:35.990934+0000","last_unstale":"2024-07-26T11:11:35.407144+0000","last_undegraded":"2024-07-26T11:11:35.407144+0000","last_fullsized":"2024-07-26T11:11:35.407144+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:56.598738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.098452+0000","last_change":"2024-07-26T11:10:34.965637+0000","last_active":"2024-07-26T11:11:51.098452+0000","last_peered":"2024-07-26T11:11:51.098452+0000","last_clean":"2024-07-26T11:11:51.098452+0000","last_became_active":"2024-07-26T11:10:34.961157+0000","last_became_peered":"2024-07-26T11:10:34.961157+0000","last_unstale":"2024-07-26T11:11:51.098452+0000","last_undegraded":"2024-07-26T11:11:51.098452+0000","last_fullsized":"2024-07-26T11:11:51.098452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:26:17.007258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874776+0000","last_change":"2024-07-26T11:10:22.303606+0000","last_active":"2024-07-26T11:13:33.874776+0000","last_peered":"2024-07-26T11:13:33.874776+0000","last_clean":"2024-07-26T11:13:33.874776+0000","last_became_active":"2024-07-26T11:10:22.303183+0000","last_became_peered":"2024-07-26T11:10:22.303183+0000","last_unstale":"2024-07-26T11:13:33.874776+0000","last_undegraded":"2024-07-26T11:13:33.874776+0000","last_fullsized":"2024-07-26T11:13:33.874776+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:31:01.153694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.406757+0000","last_change":"2024-07-26T11:10:35.981446+0000","last_active":"2024-07-26T11:11:35.406757+0000","last_peered":"2024-07-26T11:11:35.406757+0000","last_clean":"2024-07-26T11:11:35.406757+0000","last_became_active":"2024-07-26T11:10:35.980994+0000","last_became_peered":"2024-07-26T11:10:35.980994+0000","last_unstale":"2024-07-26T11:11:35.406757+0000","last_undegraded":"2024-07-26T11:11:35.406757+0000","last_fullsized":"2024-07-26T11:11:35.406757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:38:57.405325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.887131+0000","last_change":"2024-07-26T11:10:22.306758+0000","last_active":"2024-07-26T11:13:33.887131+0000","last_peered":"2024-07-26T11:13:33.887131+0000","last_clean":"2024-07-26T11:13:33.887131+0000","last_became_active":"2024-07-26T11:10:22.306370+0000","last_became_peered":"2024-07-26T11:10:22.306370+0000","last_unstale":"2024-07-26T11:13:33.887131+0000","last_undegraded":"2024-07-26T11:13:33.887131+0000","last_fullsized":"2024-07-26T11:13:33.887131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:51:43.069470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.095857+0000","last_change":"2024-07-26T11:10:18.035664+0000","last_active":"2024-07-26T11:11:51.095857+0000","last_peered":"2024-07-26T11:11:51.095857+0000","last_clean":"2024-07-26T11:11:51.095857+0000","last_became_active":"2024-07-26T11:10:18.035113+0000","last_became_peered":"2024-07-26T11:10:18.035113+0000","last_unstale":"2024-07-26T11:11:51.095857+0000","last_undegraded":"2024-07-26T11:11:51.095857+0000","last_fullsized":"2024-07-26T11:11:51.095857+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:09:30.641034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.407076+0000","last_change":"2024-07-26T11:10:35.990907+0000","last_active":"2024-07-26T11:11:35.407076+0000","last_peered":"2024-07-26T11:11:35.407076+0000","last_clean":"2024-07-26T11:11:35.407076+0000","last_became_active":"2024-07-26T11:10:35.990417+0000","last_became_peered":"2024-07-26T11:10:35.990417+0000","last_unstale":"2024-07-26T11:11:35.407076+0000","last_undegraded":"2024-07-26T11:11:35.407076+0000","last_fullsized":"2024-07-26T11:11:35.407076+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:24:39.319727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889003+0000","last_change":"2024-07-26T11:10:22.333575+0000","last_active":"2024-07-26T11:13:33.889003+0000","last_peered":"2024-07-26T11:13:33.889003+0000","last_clean":"2024-07-26T11:13:33.889003+0000","last_became_active":"2024-07-26T11:10:22.332802+0000","last_became_peered":"2024-07-26T11:10:22.332802+0000","last_unstale":"2024-07-26T11:13:33.889003+0000","last_undegraded":"2024-07-26T11:13:33.889003+0000","last_fullsized":"2024-07-26T11:13:33.889003+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:14:03.238840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.059445+0000","last_change":"2024-07-26T11:10:18.047942+0000","last_active":"2024-07-26T11:11:51.059445+0000","last_peered":"2024-07-26T11:11:51.059445+0000","last_clean":"2024-07-26T11:11:51.059445+0000","last_became_active":"2024-07-26T11:10:18.047500+0000","last_became_peered":"2024-07-26T11:10:18.047500+0000","last_unstale":"2024-07-26T11:11:51.059445+0000","last_undegraded":"2024-07-26T11:11:51.059445+0000","last_fullsized":"2024-07-26T11:11:51.059445+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:32:30.773648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:11.796009+0000","last_change":"2024-07-26T11:10:36.000946+0000","last_active":"2024-07-26T11:12:11.796009+0000","last_peered":"2024-07-26T11:12:11.796009+0000","last_clean":"2024-07-26T11:12:11.796009+0000","last_became_active":"2024-07-26T11:10:36.000251+0000","last_became_peered":"2024-07-26T11:10:36.000251+0000","last_unstale":"2024-07-26T11:12:11.796009+0000","last_undegraded":"2024-07-26T11:12:11.796009+0000","last_fullsized":"2024-07-26T11:12:11.796009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:03:40.523747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.890224+0000","last_change":"2024-07-26T11:10:22.326370+0000","last_active":"2024-07-26T11:13:33.890224+0000","last_peered":"2024-07-26T11:13:33.890224+0000","last_clean":"2024-07-26T11:13:33.890224+0000","last_became_active":"2024-07-26T11:10:22.325961+0000","last_became_peered":"2024-07-26T11:10:22.325961+0000","last_unstale":"2024-07-26T11:13:33.890224+0000","last_undegraded":"2024-07-26T11:13:33.890224+0000","last_fullsized":"2024-07-26T11:13:33.890224+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:16:36.946750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.089544+0000","last_change":"2024-07-26T11:10:18.051940+0000","last_active":"2024-07-26T11:11:50.089544+0000","last_peered":"2024-07-26T11:11:50.089544+0000","last_clean":"2024-07-26T11:11:50.089544+0000","last_became_active":"2024-07-26T11:10:18.051565+0000","last_became_peered":"2024-07-26T11:10:18.051565+0000","last_unstale":"2024-07-26T11:11:50.089544+0000","last_undegraded":"2024-07-26T11:11:50.089544+0000","last_fullsized":"2024-07-26T11:11:50.089544+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:17:56.418638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.383567+0000","last_change":"2024-07-26T11:10:36.000156+0000","last_active":"2024-07-26T11:11:35.383567+0000","last_peered":"2024-07-26T11:11:35.383567+0000","last_clean":"2024-07-26T11:11:35.383567+0000","last_became_active":"2024-07-26T11:10:35.999600+0000","last_became_peered":"2024-07-26T11:10:35.999600+0000","last_unstale":"2024-07-26T11:11:35.383567+0000","last_undegraded":"2024-07-26T11:11:35.383567+0000","last_fullsized":"2024-07-26T11:11:35.383567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:35:48.504288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.821261+0000","last_change":"2024-07-26T11:10:22.298152+0000","last_active":"2024-07-26T11:13:33.821261+0000","last_peered":"2024-07-26T11:13:33.821261+0000","last_clean":"2024-07-26T11:13:33.821261+0000","last_became_active":"2024-07-26T11:10:22.297739+0000","last_became_peered":"2024-07-26T11:10:22.297739+0000","last_unstale":"2024-07-26T11:13:33.821261+0000","last_undegraded":"2024-07-26T11:13:33.821261+0000","last_fullsized":"2024-07-26T11:13:33.821261+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:48.160183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:47.040493+0000","last_change":"2024-07-26T11:10:18.057756+0000","last_active":"2024-07-26T11:11:47.040493+0000","last_peered":"2024-07-26T11:11:47.040493+0000","last_clean":"2024-07-26T11:11:47.040493+0000","last_became_active":"2024-07-26T11:10:18.057464+0000","last_became_peered":"2024-07-26T11:10:18.057464+0000","last_unstale":"2024-07-26T11:11:47.040493+0000","last_undegraded":"2024-07-26T11:11:47.040493+0000","last_fullsized":"2024-07-26T11:11:47.040493+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:39:11.523643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.407148+0000","last_change":"2024-07-26T11:10:36.000617+0000","last_active":"2024-07-26T11:11:35.407148+0000","last_peered":"2024-07-26T11:11:35.407148+0000","last_clean":"2024-07-26T11:11:35.407148+0000","last_became_active":"2024-07-26T11:10:36.000151+0000","last_became_peered":"2024-07-26T11:10:36.000151+0000","last_unstale":"2024-07-26T11:11:35.407148+0000","last_undegraded":"2024-07-26T11:11:35.407148+0000","last_fullsized":"2024-07-26T11:11:35.407148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:36.942215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.892406+0000","last_change":"2024-07-26T11:10:22.336452+0000","last_active":"2024-07-26T11:13:33.892406+0000","last_peered":"2024-07-26T11:13:33.892406+0000","last_clean":"2024-07-26T11:13:33.892406+0000","last_became_active":"2024-07-26T11:10:22.336006+0000","last_became_peered":"2024-07-26T11:10:22.336006+0000","last_unstale":"2024-07-26T11:13:33.892406+0000","last_undegraded":"2024-07-26T11:13:33.892406+0000","last_fullsized":"2024-07-26T11:13:33.892406+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:08:27.375730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.096168+0000","last_change":"2024-07-26T11:10:18.823017+0000","last_active":"2024-07-26T11:11:51.096168+0000","last_peered":"2024-07-26T11:11:51.096168+0000","last_clean":"2024-07-26T11:11:51.096168+0000","last_became_active":"2024-07-26T11:10:18.822559+0000","last_became_peered":"2024-07-26T11:10:18.822559+0000","last_unstale":"2024-07-26T11:11:51.096168+0000","last_undegraded":"2024-07-26T11:11:51.096168+0000","last_fullsized":"2024-07-26T11:11:51.096168+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:01.486263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.155764+0000","last_change":"2024-07-26T11:10:35.969698+0000","last_active":"2024-07-26T11:11:35.155764+0000","last_peered":"2024-07-26T11:11:35.155764+0000","last_clean":"2024-07-26T11:11:35.155764+0000","last_became_active":"2024-07-26T11:10:35.969240+0000","last_became_peered":"2024-07-26T11:10:35.969240+0000","last_unstale":"2024-07-26T11:11:35.155764+0000","last_undegraded":"2024-07-26T11:11:35.155764+0000","last_fullsized":"2024-07-26T11:11:35.155764+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:25.210862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875247+0000","last_change":"2024-07-26T11:10:22.314772+0000","last_active":"2024-07-26T11:13:33.875247+0000","last_peered":"2024-07-26T11:13:33.875247+0000","last_clean":"2024-07-26T11:13:33.875247+0000","last_became_active":"2024-07-26T11:10:22.313786+0000","last_became_peered":"2024-07-26T11:10:22.313786+0000","last_unstale":"2024-07-26T11:13:33.875247+0000","last_undegraded":"2024-07-26T11:13:33.875247+0000","last_fullsized":"2024-07-26T11:13:33.875247+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:19.610695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'17","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:31.518589+0000","last_change":"2024-07-26T11:10:18.049408+0000","last_active":"2024-07-26T11:12:31.518589+0000","last_peered":"2024-07-26T11:12:31.518589+0000","last_clean":"2024-07-26T11:12:31.518589+0000","last_became_active":"2024-07-26T11:10:18.048573+0000","last_became_peered":"2024-07-26T11:10:18.048573+0000","last_unstale":"2024-07-26T11:12:31.518589+0000","last_undegraded":"2024-07-26T11:12:31.518589+0000","last_fullsized":"2024-07-26T11:12:31.518589+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:26.408260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.147395+0000","last_change":"2024-07-26T11:10:35.992067+0000","last_active":"2024-07-26T11:11:36.147395+0000","last_peered":"2024-07-26T11:11:36.147395+0000","last_clean":"2024-07-26T11:11:36.147395+0000","last_became_active":"2024-07-26T11:10:35.989789+0000","last_became_peered":"2024-07-26T11:10:35.989789+0000","last_unstale":"2024-07-26T11:11:36.147395+0000","last_undegraded":"2024-07-26T11:11:36.147395+0000","last_fullsized":"2024-07-26T11:11:36.147395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:08:08.283592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.886537+0000","last_change":"2024-07-26T11:10:22.323512+0000","last_active":"2024-07-26T11:13:33.886537+0000","last_peered":"2024-07-26T11:13:33.886537+0000","last_clean":"2024-07-26T11:13:33.886537+0000","last_became_active":"2024-07-26T11:10:22.323126+0000","last_became_peered":"2024-07-26T11:10:22.323126+0000","last_unstale":"2024-07-26T11:13:33.886537+0000","last_undegraded":"2024-07-26T11:13:33.886537+0000","last_fullsized":"2024-07-26T11:13:33.886537+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:27:15.379345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.093631+0000","last_change":"2024-07-26T11:10:18.052573+0000","last_active":"2024-07-26T11:11:51.093631+0000","last_peered":"2024-07-26T11:11:51.093631+0000","last_clean":"2024-07-26T11:11:51.093631+0000","last_became_active":"2024-07-26T11:10:18.052302+0000","last_became_peered":"2024-07-26T11:10:18.052302+0000","last_unstale":"2024-07-26T11:11:51.093631+0000","last_undegraded":"2024-07-26T11:11:51.093631+0000","last_fullsized":"2024-07-26T11:11:51.093631+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:32.260966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162413+0000","last_change":"2024-07-26T11:10:35.994347+0000","last_active":"2024-07-26T11:11:35.162413+0000","last_peered":"2024-07-26T11:11:35.162413+0000","last_clean":"2024-07-26T11:11:35.162413+0000","last_became_active":"2024-07-26T11:10:35.993795+0000","last_became_peered":"2024-07-26T11:10:35.993795+0000","last_unstale":"2024-07-26T11:11:35.162413+0000","last_undegraded":"2024-07-26T11:11:35.162413+0000","last_fullsized":"2024-07-26T11:11:35.162413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:35.399601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.881668+0000","last_change":"2024-07-26T11:10:22.318109+0000","last_active":"2024-07-26T11:13:33.881668+0000","last_peered":"2024-07-26T11:13:33.881668+0000","last_clean":"2024-07-26T11:13:33.881668+0000","last_became_active":"2024-07-26T11:10:22.317674+0000","last_became_peered":"2024-07-26T11:10:22.317674+0000","last_unstale":"2024-07-26T11:13:33.881668+0000","last_undegraded":"2024-07-26T11:13:33.881668+0000","last_fullsized":"2024-07-26T11:13:33.881668+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:29.464772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:49.073328+0000","last_change":"2024-07-26T11:10:18.060073+0000","last_active":"2024-07-26T11:11:49.073328+0000","last_peered":"2024-07-26T11:11:49.073328+0000","last_clean":"2024-07-26T11:11:49.073328+0000","last_became_active":"2024-07-26T11:10:18.059381+0000","last_became_peered":"2024-07-26T11:10:18.059381+0000","last_unstale":"2024-07-26T11:11:49.073328+0000","last_undegraded":"2024-07-26T11:11:49.073328+0000","last_fullsized":"2024-07-26T11:11:49.073328+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:04:21.184759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.951752+0000","last_change":"2024-07-26T11:11:35.951234+0000","last_active":"2024-07-26T11:11:35.951752+0000","last_peered":"2024-07-26T11:11:35.951752+0000","last_clean":"2024-07-26T11:11:35.951752+0000","last_became_active":"2024-07-26T11:11:35.950821+0000","last_became_peered":"2024-07-26T11:11:35.950821+0000","last_unstale":"2024-07-26T11:11:35.951752+0000","last_undegraded":"2024-07-26T11:11:35.951752+0000","last_fullsized":"2024-07-26T11:11:35.951752+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:44:22.569238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.872883+0000","last_change":"2024-07-26T11:10:22.326866+0000","last_active":"2024-07-26T11:13:33.872883+0000","last_peered":"2024-07-26T11:13:33.872883+0000","last_clean":"2024-07-26T11:13:33.872883+0000","last_became_active":"2024-07-26T11:10:22.326417+0000","last_became_peered":"2024-07-26T11:10:22.326417+0000","last_unstale":"2024-07-26T11:13:33.872883+0000","last_undegraded":"2024-07-26T11:13:33.872883+0000","last_fullsized":"2024-07-26T11:13:33.872883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:49:02.871062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":170,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:51.336874+0000","last_change":"2024-07-26T11:10:18.044077+0000","last_active":"2024-07-26T11:13:51.336874+0000","last_peered":"2024-07-26T11:13:51.336874+0000","last_clean":"2024-07-26T11:13:51.336874+0000","last_became_active":"2024-07-26T11:10:18.043660+0000","last_became_peered":"2024-07-26T11:10:18.043660+0000","last_unstale":"2024-07-26T11:13:51.336874+0000","last_undegraded":"2024-07-26T11:13:51.336874+0000","last_fullsized":"2024-07-26T11:13:51.336874+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:46:54.901082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.641747+0000","last_change":"2024-07-26T11:10:35.984778+0000","last_active":"2024-07-26T11:11:35.641747+0000","last_peered":"2024-07-26T11:11:35.641747+0000","last_clean":"2024-07-26T11:11:35.641747+0000","last_became_active":"2024-07-26T11:10:35.984365+0000","last_became_peered":"2024-07-26T11:10:35.984365+0000","last_unstale":"2024-07-26T11:11:35.641747+0000","last_undegraded":"2024-07-26T11:11:35.641747+0000","last_fullsized":"2024-07-26T11:11:35.641747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:33.174121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.859140+0000","last_change":"2024-07-26T11:10:34.976075+0000","last_active":"2024-07-26T11:13:33.859140+0000","last_peered":"2024-07-26T11:13:33.859140+0000","last_clean":"2024-07-26T11:13:33.859140+0000","last_became_active":"2024-07-26T11:10:34.973799+0000","last_became_peered":"2024-07-26T11:10:34.973799+0000","last_unstale":"2024-07-26T11:13:33.859140+0000","last_undegraded":"2024-07-26T11:13:33.859140+0000","last_fullsized":"2024-07-26T11:13:33.859140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:03:01.830655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":170,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:50.405205+0000","last_change":"2024-07-26T11:10:18.056473+0000","last_active":"2024-07-26T11:13:50.405205+0000","last_peered":"2024-07-26T11:13:50.405205+0000","last_clean":"2024-07-26T11:13:50.405205+0000","last_became_active":"2024-07-26T11:10:18.056038+0000","last_became_peered":"2024-07-26T11:10:18.056038+0000","last_unstale":"2024-07-26T11:13:50.405205+0000","last_undegraded":"2024-07-26T11:13:50.405205+0000","last_fullsized":"2024-07-26T11:13:50.405205+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:04.501018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":122,"num_read_kb":122,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.642403+0000","last_change":"2024-07-26T11:10:35.991741+0000","last_active":"2024-07-26T11:11:35.642403+0000","last_peered":"2024-07-26T11:11:35.642403+0000","last_clean":"2024-07-26T11:11:35.642403+0000","last_became_active":"2024-07-26T11:10:35.990995+0000","last_became_peered":"2024-07-26T11:10:35.990995+0000","last_unstale":"2024-07-26T11:11:35.642403+0000","last_undegraded":"2024-07-26T11:11:35.642403+0000","last_fullsized":"2024-07-26T11:11:35.642403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:59:34.056646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.884240+0000","last_change":"2024-07-26T11:10:22.305053+0000","last_active":"2024-07-26T11:13:33.884240+0000","last_peered":"2024-07-26T11:13:33.884240+0000","last_clean":"2024-07-26T11:13:33.884240+0000","last_became_active":"2024-07-26T11:10:22.304035+0000","last_became_peered":"2024-07-26T11:10:22.304035+0000","last_unstale":"2024-07-26T11:13:33.884240+0000","last_undegraded":"2024-07-26T11:13:33.884240+0000","last_fullsized":"2024-07-26T11:13:33.884240+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:48.590799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.082116+0000","last_change":"2024-07-26T11:10:18.033581+0000","last_active":"2024-07-26T11:11:50.082116+0000","last_peered":"2024-07-26T11:11:50.082116+0000","last_clean":"2024-07-26T11:11:50.082116+0000","last_became_active":"2024-07-26T11:10:18.033038+0000","last_became_peered":"2024-07-26T11:10:18.033038+0000","last_unstale":"2024-07-26T11:11:50.082116+0000","last_undegraded":"2024-07-26T11:11:50.082116+0000","last_fullsized":"2024-07-26T11:11:50.082116+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:19:52.523006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.644202+0000","last_change":"2024-07-26T11:11:35.643683+0000","last_active":"2024-07-26T11:11:35.644202+0000","last_peered":"2024-07-26T11:11:35.644202+0000","last_clean":"2024-07-26T11:11:35.644202+0000","last_became_active":"2024-07-26T11:11:35.643109+0000","last_became_peered":"2024-07-26T11:11:35.643109+0000","last_unstale":"2024-07-26T11:11:35.644202+0000","last_undegraded":"2024-07-26T11:11:35.644202+0000","last_fullsized":"2024-07-26T11:11:35.644202+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:39:28.190726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.878406+0000","last_change":"2024-07-26T11:10:34.977690+0000","last_active":"2024-07-26T11:13:33.878406+0000","last_peered":"2024-07-26T11:13:33.878406+0000","last_clean":"2024-07-26T11:13:33.878406+0000","last_became_active":"2024-07-26T11:10:34.976938+0000","last_became_peered":"2024-07-26T11:10:34.976938+0000","last_unstale":"2024-07-26T11:13:33.878406+0000","last_undegraded":"2024-07-26T11:13:33.878406+0000","last_fullsized":"2024-07-26T11:13:33.878406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:26:21.510469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.094135+0000","last_change":"2024-07-26T11:10:18.045363+0000","last_active":"2024-07-26T11:11:51.094135+0000","last_peered":"2024-07-26T11:11:51.094135+0000","last_clean":"2024-07-26T11:11:51.094135+0000","last_became_active":"2024-07-26T11:10:18.044810+0000","last_became_peered":"2024-07-26T11:10:18.044810+0000","last_unstale":"2024-07-26T11:11:51.094135+0000","last_undegraded":"2024-07-26T11:11:51.094135+0000","last_fullsized":"2024-07-26T11:11:51.094135+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:34.507172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":119,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161450+0000","last_change":"2024-07-26T11:08:48.936784+0000","last_active":"2024-07-26T11:11:35.161450+0000","last_peered":"2024-07-26T11:11:35.161450+0000","last_clean":"2024-07-26T11:11:35.161450+0000","last_became_active":"2024-07-26T11:08:48.931118+0000","last_became_peered":"2024-07-26T11:08:48.931118+0000","last_unstale":"2024-07-26T11:11:35.161450+0000","last_undegraded":"2024-07-26T11:11:35.161450+0000","last_fullsized":"2024-07-26T11:11:35.161450+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:05:01.918569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:05:01.918569+0000","last_clean_scrub_stamp":"2024-07-26T11:05:01.918569+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:57.128424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:04.061187+0000","last_change":"2024-07-26T11:11:35.385528+0000","last_active":"2024-07-26T11:12:04.061187+0000","last_peered":"2024-07-26T11:12:04.061187+0000","last_clean":"2024-07-26T11:12:04.061187+0000","last_became_active":"2024-07-26T11:11:35.385042+0000","last_became_peered":"2024-07-26T11:11:35.385042+0000","last_unstale":"2024-07-26T11:12:04.061187+0000","last_undegraded":"2024-07-26T11:12:04.061187+0000","last_fullsized":"2024-07-26T11:12:04.061187+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:52:30.343736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875091+0000","last_change":"2024-07-26T11:10:22.302715+0000","last_active":"2024-07-26T11:13:33.875091+0000","last_peered":"2024-07-26T11:13:33.875091+0000","last_clean":"2024-07-26T11:13:33.875091+0000","last_became_active":"2024-07-26T11:10:22.302284+0000","last_became_peered":"2024-07-26T11:10:22.302284+0000","last_unstale":"2024-07-26T11:13:33.875091+0000","last_undegraded":"2024-07-26T11:13:33.875091+0000","last_fullsized":"2024-07-26T11:13:33.875091+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:13:05.555439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":173,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:53.294231+0000","last_change":"2024-07-26T11:10:18.823160+0000","last_active":"2024-07-26T11:13:53.294231+0000","last_peered":"2024-07-26T11:13:53.294231+0000","last_clean":"2024-07-26T11:13:53.294231+0000","last_became_active":"2024-07-26T11:10:18.822652+0000","last_became_peered":"2024-07-26T11:10:18.822652+0000","last_unstale":"2024-07-26T11:13:53.294231+0000","last_undegraded":"2024-07-26T11:13:53.294231+0000","last_fullsized":"2024-07-26T11:13:53.294231+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:21:06.077784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":126,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.949512+0000","last_change":"2024-07-26T11:10:35.984007+0000","last_active":"2024-07-26T11:11:35.949512+0000","last_peered":"2024-07-26T11:11:35.949512+0000","last_clean":"2024-07-26T11:11:35.949512+0000","last_became_active":"2024-07-26T11:10:35.983352+0000","last_became_peered":"2024-07-26T11:10:35.983352+0000","last_unstale":"2024-07-26T11:11:35.949512+0000","last_undegraded":"2024-07-26T11:11:35.949512+0000","last_fullsized":"2024-07-26T11:11:35.949512+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:05:21.199041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.867547+0000","last_change":"2024-07-26T11:10:22.320559+0000","last_active":"2024-07-26T11:13:33.867547+0000","last_peered":"2024-07-26T11:13:33.867547+0000","last_clean":"2024-07-26T11:13:33.867547+0000","last_became_active":"2024-07-26T11:10:22.320192+0000","last_became_peered":"2024-07-26T11:10:22.320192+0000","last_unstale":"2024-07-26T11:13:33.867547+0000","last_undegraded":"2024-07-26T11:13:33.867547+0000","last_fullsized":"2024-07-26T11:13:33.867547+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:48:51.774418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.096632+0000","last_change":"2024-07-26T11:10:18.044108+0000","last_active":"2024-07-26T11:11:51.096632+0000","last_peered":"2024-07-26T11:11:51.096632+0000","last_clean":"2024-07-26T11:11:51.096632+0000","last_became_active":"2024-07-26T11:10:18.043712+0000","last_became_peered":"2024-07-26T11:10:18.043712+0000","last_unstale":"2024-07-26T11:11:51.096632+0000","last_undegraded":"2024-07-26T11:11:51.096632+0000","last_fullsized":"2024-07-26T11:11:51.096632+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:05:34.382520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.185926+0000","last_change":"2024-07-26T11:10:35.999071+0000","last_active":"2024-07-26T11:11:35.185926+0000","last_peered":"2024-07-26T11:11:35.185926+0000","last_clean":"2024-07-26T11:11:35.185926+0000","last_became_active":"2024-07-26T11:10:35.997179+0000","last_became_peered":"2024-07-26T11:10:35.997179+0000","last_unstale":"2024-07-26T11:11:35.185926+0000","last_undegraded":"2024-07-26T11:11:35.185926+0000","last_fullsized":"2024-07-26T11:11:35.185926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:36:45.632086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.871203+0000","last_change":"2024-07-26T11:10:22.300356+0000","last_active":"2024-07-26T11:13:33.871203+0000","last_peered":"2024-07-26T11:13:33.871203+0000","last_clean":"2024-07-26T11:13:33.871203+0000","last_became_active":"2024-07-26T11:10:22.299949+0000","last_became_peered":"2024-07-26T11:10:22.299949+0000","last_unstale":"2024-07-26T11:13:33.871203+0000","last_undegraded":"2024-07-26T11:13:33.871203+0000","last_fullsized":"2024-07-26T11:13:33.871203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:34.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.087457+0000","last_change":"2024-07-26T11:10:18.049965+0000","last_active":"2024-07-26T11:11:50.087457+0000","last_peered":"2024-07-26T11:11:50.087457+0000","last_clean":"2024-07-26T11:11:50.087457+0000","last_became_active":"2024-07-26T11:10:18.049251+0000","last_became_peered":"2024-07-26T11:10:18.049251+0000","last_unstale":"2024-07-26T11:11:50.087457+0000","last_undegraded":"2024-07-26T11:11:50.087457+0000","last_fullsized":"2024-07-26T11:11:50.087457+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:20.996609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158648+0000","last_change":"2024-07-26T11:10:35.967661+0000","last_active":"2024-07-26T11:11:35.158648+0000","last_peered":"2024-07-26T11:11:35.158648+0000","last_clean":"2024-07-26T11:11:35.158648+0000","last_became_active":"2024-07-26T11:10:35.966867+0000","last_became_peered":"2024-07-26T11:10:35.966867+0000","last_unstale":"2024-07-26T11:11:35.158648+0000","last_undegraded":"2024-07-26T11:11:35.158648+0000","last_fullsized":"2024-07-26T11:11:35.158648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:43.951319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.866924+0000","last_change":"2024-07-26T11:10:22.303733+0000","last_active":"2024-07-26T11:13:33.866924+0000","last_peered":"2024-07-26T11:13:33.866924+0000","last_clean":"2024-07-26T11:13:33.866924+0000","last_became_active":"2024-07-26T11:10:22.303322+0000","last_became_peered":"2024-07-26T11:10:22.303322+0000","last_unstale":"2024-07-26T11:13:33.866924+0000","last_undegraded":"2024-07-26T11:13:33.866924+0000","last_fullsized":"2024-07-26T11:13:33.866924+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:34:34.962492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.096226+0000","last_change":"2024-07-26T11:10:18.032539+0000","last_active":"2024-07-26T11:11:51.096226+0000","last_peered":"2024-07-26T11:11:51.096226+0000","last_clean":"2024-07-26T11:11:51.096226+0000","last_became_active":"2024-07-26T11:10:18.032059+0000","last_became_peered":"2024-07-26T11:10:18.032059+0000","last_unstale":"2024-07-26T11:11:51.096226+0000","last_undegraded":"2024-07-26T11:11:51.096226+0000","last_fullsized":"2024-07-26T11:11:51.096226+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:54.502339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158655+0000","last_change":"2024-07-26T11:10:35.977473+0000","last_active":"2024-07-26T11:11:35.158655+0000","last_peered":"2024-07-26T11:11:35.158655+0000","last_clean":"2024-07-26T11:11:35.158655+0000","last_became_active":"2024-07-26T11:10:35.976830+0000","last_became_peered":"2024-07-26T11:10:35.976830+0000","last_unstale":"2024-07-26T11:11:35.158655+0000","last_undegraded":"2024-07-26T11:11:35.158655+0000","last_fullsized":"2024-07-26T11:11:35.158655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:46:13.009246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.870719+0000","last_change":"2024-07-26T11:10:22.304358+0000","last_active":"2024-07-26T11:13:33.870719+0000","last_peered":"2024-07-26T11:13:33.870719+0000","last_clean":"2024-07-26T11:13:33.870719+0000","last_became_active":"2024-07-26T11:10:22.303846+0000","last_became_peered":"2024-07-26T11:10:22.303846+0000","last_unstale":"2024-07-26T11:13:33.870719+0000","last_undegraded":"2024-07-26T11:13:33.870719+0000","last_fullsized":"2024-07-26T11:13:33.870719+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:27:28.556481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.093876+0000","last_change":"2024-07-26T11:10:18.057678+0000","last_active":"2024-07-26T11:11:51.093876+0000","last_peered":"2024-07-26T11:11:51.093876+0000","last_clean":"2024-07-26T11:11:51.093876+0000","last_became_active":"2024-07-26T11:10:18.057282+0000","last_became_peered":"2024-07-26T11:10:18.057282+0000","last_unstale":"2024-07-26T11:11:51.093876+0000","last_undegraded":"2024-07-26T11:11:51.093876+0000","last_fullsized":"2024-07-26T11:11:51.093876+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:08:35.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.642357+0000","last_change":"2024-07-26T11:10:36.000329+0000","last_active":"2024-07-26T11:11:35.642357+0000","last_peered":"2024-07-26T11:11:35.642357+0000","last_clean":"2024-07-26T11:11:35.642357+0000","last_became_active":"2024-07-26T11:10:35.999713+0000","last_became_peered":"2024-07-26T11:10:35.999713+0000","last_unstale":"2024-07-26T11:11:35.642357+0000","last_undegraded":"2024-07-26T11:11:35.642357+0000","last_fullsized":"2024-07-26T11:11:35.642357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:48.137237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.866698+0000","last_change":"2024-07-26T11:10:22.301526+0000","last_active":"2024-07-26T11:13:33.866698+0000","last_peered":"2024-07-26T11:13:33.866698+0000","last_clean":"2024-07-26T11:13:33.866698+0000","last_became_active":"2024-07-26T11:10:22.301098+0000","last_became_peered":"2024-07-26T11:10:22.301098+0000","last_unstale":"2024-07-26T11:13:33.866698+0000","last_undegraded":"2024-07-26T11:13:33.866698+0000","last_fullsized":"2024-07-26T11:13:33.866698+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:01:48.118315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'7","reported_seq":177,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:53.415062+0000","last_change":"2024-07-26T11:10:18.035922+0000","last_active":"2024-07-26T11:13:53.415062+0000","last_peered":"2024-07-26T11:13:53.415062+0000","last_clean":"2024-07-26T11:13:53.415062+0000","last_became_active":"2024-07-26T11:10:18.035184+0000","last_became_peered":"2024-07-26T11:10:18.035184+0000","last_unstale":"2024-07-26T11:13:53.415062+0000","last_undegraded":"2024-07-26T11:13:53.415062+0000","last_fullsized":"2024-07-26T11:13:53.415062+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:13:10.371105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":129,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.384102+0000","last_change":"2024-07-26T11:10:35.998791+0000","last_active":"2024-07-26T11:11:35.384102+0000","last_peered":"2024-07-26T11:11:35.384102+0000","last_clean":"2024-07-26T11:11:35.384102+0000","last_became_active":"2024-07-26T11:10:35.997163+0000","last_became_peered":"2024-07-26T11:10:35.997163+0000","last_unstale":"2024-07-26T11:11:35.384102+0000","last_undegraded":"2024-07-26T11:11:35.384102+0000","last_fullsized":"2024-07-26T11:11:35.384102+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:28:50.167294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889375+0000","last_change":"2024-07-26T11:10:22.314219+0000","last_active":"2024-07-26T11:13:33.889375+0000","last_peered":"2024-07-26T11:13:33.889375+0000","last_clean":"2024-07-26T11:13:33.889375+0000","last_became_active":"2024-07-26T11:10:22.313811+0000","last_became_peered":"2024-07-26T11:10:22.313811+0000","last_unstale":"2024-07-26T11:13:33.889375+0000","last_undegraded":"2024-07-26T11:13:33.889375+0000","last_fullsized":"2024-07-26T11:13:33.889375+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:55:47.452798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.093492+0000","last_change":"2024-07-26T11:10:18.047818+0000","last_active":"2024-07-26T11:11:51.093492+0000","last_peered":"2024-07-26T11:11:51.093492+0000","last_clean":"2024-07-26T11:11:51.093492+0000","last_became_active":"2024-07-26T11:10:18.047159+0000","last_became_peered":"2024-07-26T11:10:18.047159+0000","last_unstale":"2024-07-26T11:11:51.093492+0000","last_undegraded":"2024-07-26T11:11:51.093492+0000","last_fullsized":"2024-07-26T11:11:51.093492+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:37.528512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162781+0000","last_change":"2024-07-26T11:10:35.996468+0000","last_active":"2024-07-26T11:11:35.162781+0000","last_peered":"2024-07-26T11:11:35.162781+0000","last_clean":"2024-07-26T11:11:35.162781+0000","last_became_active":"2024-07-26T11:10:35.996030+0000","last_became_peered":"2024-07-26T11:10:35.996030+0000","last_unstale":"2024-07-26T11:11:35.162781+0000","last_undegraded":"2024-07-26T11:11:35.162781+0000","last_fullsized":"2024-07-26T11:11:35.162781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:20:14.743690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.881839+0000","last_change":"2024-07-26T11:10:22.318375+0000","last_active":"2024-07-26T11:13:33.881839+0000","last_peered":"2024-07-26T11:13:33.881839+0000","last_clean":"2024-07-26T11:13:33.881839+0000","last_became_active":"2024-07-26T11:10:22.317965+0000","last_became_peered":"2024-07-26T11:10:22.317965+0000","last_unstale":"2024-07-26T11:13:33.881839+0000","last_undegraded":"2024-07-26T11:13:33.881839+0000","last_fullsized":"2024-07-26T11:13:33.881839+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:36:43.504045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.096598+0000","last_change":"2024-07-26T11:10:18.047957+0000","last_active":"2024-07-26T11:11:51.096598+0000","last_peered":"2024-07-26T11:11:51.096598+0000","last_clean":"2024-07-26T11:11:51.096598+0000","last_became_active":"2024-07-26T11:10:18.047495+0000","last_became_peered":"2024-07-26T11:10:18.047495+0000","last_unstale":"2024-07-26T11:11:51.096598+0000","last_undegraded":"2024-07-26T11:11:51.096598+0000","last_fullsized":"2024-07-26T11:11:51.096598+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:11.238295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158767+0000","last_change":"2024-07-26T11:10:35.976946+0000","last_active":"2024-07-26T11:11:35.158767+0000","last_peered":"2024-07-26T11:11:35.158767+0000","last_clean":"2024-07-26T11:11:35.158767+0000","last_became_active":"2024-07-26T11:10:35.976474+0000","last_became_peered":"2024-07-26T11:10:35.976474+0000","last_unstale":"2024-07-26T11:11:35.158767+0000","last_undegraded":"2024-07-26T11:11:35.158767+0000","last_fullsized":"2024-07-26T11:11:35.158767+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:50:51.240903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.859003+0000","last_change":"2024-07-26T11:10:22.302473+0000","last_active":"2024-07-26T11:13:33.859003+0000","last_peered":"2024-07-26T11:13:33.859003+0000","last_clean":"2024-07-26T11:13:33.859003+0000","last_became_active":"2024-07-26T11:10:22.302096+0000","last_became_peered":"2024-07-26T11:10:22.302096+0000","last_unstale":"2024-07-26T11:13:33.859003+0000","last_undegraded":"2024-07-26T11:13:33.859003+0000","last_fullsized":"2024-07-26T11:13:33.859003+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:19.215301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'40","reported_seq":355,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:52.291942+0000","last_change":"2024-07-26T11:10:34.973502+0000","last_active":"2024-07-26T11:13:52.291942+0000","last_peered":"2024-07-26T11:13:52.291942+0000","last_clean":"2024-07-26T11:13:52.291942+0000","last_became_active":"2024-07-26T11:10:34.964155+0000","last_became_peered":"2024-07-26T11:10:34.964155+0000","last_unstale":"2024-07-26T11:13:52.291942+0000","last_undegraded":"2024-07-26T11:13:52.291942+0000","last_fullsized":"2024-07-26T11:13:52.291942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:23:06.652746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66769,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":256,"num_read_kb":194,"num_write":42,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.382965+0000","last_change":"2024-07-26T11:10:35.981439+0000","last_active":"2024-07-26T11:11:35.382965+0000","last_peered":"2024-07-26T11:11:35.382965+0000","last_clean":"2024-07-26T11:11:35.382965+0000","last_became_active":"2024-07-26T11:10:35.980878+0000","last_became_peered":"2024-07-26T11:10:35.980878+0000","last_unstale":"2024-07-26T11:11:35.382965+0000","last_undegraded":"2024-07-26T11:11:35.382965+0000","last_fullsized":"2024-07-26T11:11:35.382965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:31.719845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.882512+0000","last_change":"2024-07-26T11:10:34.974292+0000","last_active":"2024-07-26T11:13:33.882512+0000","last_peered":"2024-07-26T11:13:33.882512+0000","last_clean":"2024-07-26T11:13:33.882512+0000","last_became_active":"2024-07-26T11:10:34.973820+0000","last_became_peered":"2024-07-26T11:10:34.973820+0000","last_unstale":"2024-07-26T11:13:33.882512+0000","last_undegraded":"2024-07-26T11:13:33.882512+0000","last_fullsized":"2024-07-26T11:13:33.882512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:35:37.629313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.096325+0000","last_change":"2024-07-26T11:10:18.055248+0000","last_active":"2024-07-26T11:11:51.096325+0000","last_peered":"2024-07-26T11:11:51.096325+0000","last_clean":"2024-07-26T11:11:51.096325+0000","last_became_active":"2024-07-26T11:10:18.054882+0000","last_became_peered":"2024-07-26T11:10:18.054882+0000","last_unstale":"2024-07-26T11:11:51.096325+0000","last_undegraded":"2024-07-26T11:11:51.096325+0000","last_fullsized":"2024-07-26T11:11:51.096325+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:16.390058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.407395+0000","last_change":"2024-07-26T11:10:35.985628+0000","last_active":"2024-07-26T11:11:35.407395+0000","last_peered":"2024-07-26T11:11:35.407395+0000","last_clean":"2024-07-26T11:11:35.407395+0000","last_became_active":"2024-07-26T11:10:35.985156+0000","last_became_peered":"2024-07-26T11:10:35.985156+0000","last_unstale":"2024-07-26T11:11:35.407395+0000","last_undegraded":"2024-07-26T11:11:35.407395+0000","last_fullsized":"2024-07-26T11:11:35.407395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:29:18.891825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.891553+0000","last_change":"2024-07-26T11:10:34.973261+0000","last_active":"2024-07-26T11:13:33.891553+0000","last_peered":"2024-07-26T11:13:33.891553+0000","last_clean":"2024-07-26T11:13:33.891553+0000","last_became_active":"2024-07-26T11:10:34.971592+0000","last_became_peered":"2024-07-26T11:10:34.971592+0000","last_unstale":"2024-07-26T11:13:33.891553+0000","last_undegraded":"2024-07-26T11:13:33.891553+0000","last_fullsized":"2024-07-26T11:13:33.891553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:58:13.517884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.083464+0000","last_change":"2024-07-26T11:10:34.962248+0000","last_active":"2024-07-26T11:11:50.083464+0000","last_peered":"2024-07-26T11:11:50.083464+0000","last_clean":"2024-07-26T11:11:50.083464+0000","last_became_active":"2024-07-26T11:10:34.961683+0000","last_became_peered":"2024-07-26T11:10:34.961683+0000","last_unstale":"2024-07-26T11:11:50.083464+0000","last_undegraded":"2024-07-26T11:11:50.083464+0000","last_fullsized":"2024-07-26T11:11:50.083464+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:17:08.308226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.381997+0000","last_change":"2024-07-26T11:10:35.995760+0000","last_active":"2024-07-26T11:11:35.381997+0000","last_peered":"2024-07-26T11:11:35.381997+0000","last_clean":"2024-07-26T11:11:35.381997+0000","last_became_active":"2024-07-26T11:10:35.995328+0000","last_became_peered":"2024-07-26T11:10:35.995328+0000","last_unstale":"2024-07-26T11:11:35.381997+0000","last_undegraded":"2024-07-26T11:11:35.381997+0000","last_fullsized":"2024-07-26T11:11:35.381997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:40:33.093315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.876940+0000","last_change":"2024-07-26T11:10:22.316151+0000","last_active":"2024-07-26T11:13:33.876940+0000","last_peered":"2024-07-26T11:13:33.876940+0000","last_clean":"2024-07-26T11:13:33.876940+0000","last_became_active":"2024-07-26T11:10:22.315722+0000","last_became_peered":"2024-07-26T11:10:22.315722+0000","last_unstale":"2024-07-26T11:13:33.876940+0000","last_undegraded":"2024-07-26T11:13:33.876940+0000","last_fullsized":"2024-07-26T11:13:33.876940+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:12:58.342637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":290,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:52.291880+0000","last_change":"2024-07-26T11:10:18.046327+0000","last_active":"2024-07-26T11:13:52.291880+0000","last_peered":"2024-07-26T11:13:52.291880+0000","last_clean":"2024-07-26T11:13:52.291880+0000","last_became_active":"2024-07-26T11:10:18.045913+0000","last_became_peered":"2024-07-26T11:10:18.045913+0000","last_unstale":"2024-07-26T11:13:52.291880+0000","last_undegraded":"2024-07-26T11:13:52.291880+0000","last_fullsized":"2024-07-26T11:13:52.291880+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:58.889046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":244,"num_read_kb":244,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.147807+0000","last_change":"2024-07-26T11:10:35.983264+0000","last_active":"2024-07-26T11:11:36.147807+0000","last_peered":"2024-07-26T11:11:36.147807+0000","last_clean":"2024-07-26T11:11:36.147807+0000","last_became_active":"2024-07-26T11:10:35.982210+0000","last_became_peered":"2024-07-26T11:10:35.982210+0000","last_unstale":"2024-07-26T11:11:36.147807+0000","last_undegraded":"2024-07-26T11:11:36.147807+0000","last_fullsized":"2024-07-26T11:11:36.147807+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:37:20.272725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.185837+0000","last_change":"2024-07-26T11:10:35.985335+0000","last_active":"2024-07-26T11:11:35.185837+0000","last_peered":"2024-07-26T11:11:35.185837+0000","last_clean":"2024-07-26T11:11:35.185837+0000","last_became_active":"2024-07-26T11:10:35.983667+0000","last_became_peered":"2024-07-26T11:10:35.983667+0000","last_unstale":"2024-07-26T11:11:35.185837+0000","last_undegraded":"2024-07-26T11:11:35.185837+0000","last_fullsized":"2024-07-26T11:11:35.185837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:00:17.744806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.879169+0000","last_change":"2024-07-26T11:10:22.333494+0000","last_active":"2024-07-26T11:13:33.879169+0000","last_peered":"2024-07-26T11:13:33.879169+0000","last_clean":"2024-07-26T11:13:33.879169+0000","last_became_active":"2024-07-26T11:10:22.332950+0000","last_became_peered":"2024-07-26T11:10:22.332950+0000","last_unstale":"2024-07-26T11:13:33.879169+0000","last_undegraded":"2024-07-26T11:13:33.879169+0000","last_fullsized":"2024-07-26T11:13:33.879169+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:58:38.340857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.099680+0000","last_change":"2024-07-26T11:10:18.034057+0000","last_active":"2024-07-26T11:11:51.099680+0000","last_peered":"2024-07-26T11:11:51.099680+0000","last_clean":"2024-07-26T11:11:51.099680+0000","last_became_active":"2024-07-26T11:10:18.033119+0000","last_became_peered":"2024-07-26T11:10:18.033119+0000","last_unstale":"2024-07-26T11:11:51.099680+0000","last_undegraded":"2024-07-26T11:11:51.099680+0000","last_fullsized":"2024-07-26T11:11:51.099680+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:21.675593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.408887+0000","last_change":"2024-07-26T11:11:35.408377+0000","last_active":"2024-07-26T11:11:35.408887+0000","last_peered":"2024-07-26T11:11:35.408887+0000","last_clean":"2024-07-26T11:11:35.408887+0000","last_became_active":"2024-07-26T11:11:35.407865+0000","last_became_peered":"2024-07-26T11:11:35.407865+0000","last_unstale":"2024-07-26T11:11:35.408887+0000","last_undegraded":"2024-07-26T11:11:35.408887+0000","last_fullsized":"2024-07-26T11:11:35.408887+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:43:02.953395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.886870+0000","last_change":"2024-07-26T11:10:22.323447+0000","last_active":"2024-07-26T11:13:33.886870+0000","last_peered":"2024-07-26T11:13:33.886870+0000","last_clean":"2024-07-26T11:13:33.886870+0000","last_became_active":"2024-07-26T11:10:22.322988+0000","last_became_peered":"2024-07-26T11:10:22.322988+0000","last_unstale":"2024-07-26T11:13:33.886870+0000","last_undegraded":"2024-07-26T11:13:33.886870+0000","last_fullsized":"2024-07-26T11:13:33.886870+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:14.880985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.088906+0000","last_change":"2024-07-26T11:10:18.048724+0000","last_active":"2024-07-26T11:11:50.088906+0000","last_peered":"2024-07-26T11:11:50.088906+0000","last_clean":"2024-07-26T11:11:50.088906+0000","last_became_active":"2024-07-26T11:10:18.048257+0000","last_became_peered":"2024-07-26T11:10:18.048257+0000","last_unstale":"2024-07-26T11:11:50.088906+0000","last_undegraded":"2024-07-26T11:11:50.088906+0000","last_fullsized":"2024-07-26T11:11:50.088906+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:02:13.581446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:49.071419+0000","last_change":"2024-07-26T11:10:18.044171+0000","last_active":"2024-07-26T11:11:49.071419+0000","last_peered":"2024-07-26T11:11:49.071419+0000","last_clean":"2024-07-26T11:11:49.071419+0000","last_became_active":"2024-07-26T11:10:18.043682+0000","last_became_peered":"2024-07-26T11:10:18.043682+0000","last_unstale":"2024-07-26T11:11:49.071419+0000","last_undegraded":"2024-07-26T11:11:49.071419+0000","last_fullsized":"2024-07-26T11:11:49.071419+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:46:46.864119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874838+0000","last_change":"2024-07-26T11:10:22.321081+0000","last_active":"2024-07-26T11:13:33.874838+0000","last_peered":"2024-07-26T11:13:33.874838+0000","last_clean":"2024-07-26T11:13:33.874838+0000","last_became_active":"2024-07-26T11:10:22.320656+0000","last_became_peered":"2024-07-26T11:10:22.320656+0000","last_unstale":"2024-07-26T11:13:33.874838+0000","last_undegraded":"2024-07-26T11:13:33.874838+0000","last_fullsized":"2024-07-26T11:13:33.874838+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:17:20.455129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156183+0000","last_change":"2024-07-26T11:10:36.000224+0000","last_active":"2024-07-26T11:11:35.156183+0000","last_peered":"2024-07-26T11:11:35.156183+0000","last_clean":"2024-07-26T11:11:35.156183+0000","last_became_active":"2024-07-26T11:10:35.999733+0000","last_became_peered":"2024-07-26T11:10:35.999733+0000","last_unstale":"2024-07-26T11:11:35.156183+0000","last_undegraded":"2024-07-26T11:11:35.156183+0000","last_fullsized":"2024-07-26T11:11:35.156183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:49.912401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":172,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:52.292557+0000","last_change":"2024-07-26T11:10:18.034585+0000","last_active":"2024-07-26T11:13:52.292557+0000","last_peered":"2024-07-26T11:13:52.292557+0000","last_clean":"2024-07-26T11:13:52.292557+0000","last_became_active":"2024-07-26T11:10:18.034237+0000","last_became_peered":"2024-07-26T11:10:18.034237+0000","last_unstale":"2024-07-26T11:13:52.292557+0000","last_undegraded":"2024-07-26T11:13:52.292557+0000","last_fullsized":"2024-07-26T11:13:52.292557+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:22.670277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":125,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.876424+0000","last_change":"2024-07-26T11:10:22.319282+0000","last_active":"2024-07-26T11:13:33.876424+0000","last_peered":"2024-07-26T11:13:33.876424+0000","last_clean":"2024-07-26T11:13:33.876424+0000","last_became_active":"2024-07-26T11:10:22.318822+0000","last_became_peered":"2024-07-26T11:10:22.318822+0000","last_unstale":"2024-07-26T11:13:33.876424+0000","last_undegraded":"2024-07-26T11:13:33.876424+0000","last_fullsized":"2024-07-26T11:13:33.876424+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:50:57.615839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156390+0000","last_change":"2024-07-26T11:10:35.970189+0000","last_active":"2024-07-26T11:11:35.156390+0000","last_peered":"2024-07-26T11:11:35.156390+0000","last_clean":"2024-07-26T11:11:35.156390+0000","last_became_active":"2024-07-26T11:10:35.969574+0000","last_became_peered":"2024-07-26T11:10:35.969574+0000","last_unstale":"2024-07-26T11:11:35.156390+0000","last_undegraded":"2024-07-26T11:11:35.156390+0000","last_fullsized":"2024-07-26T11:11:35.156390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:07.487944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.868112+0000","last_change":"2024-07-26T11:10:22.299152+0000","last_active":"2024-07-26T11:13:33.868112+0000","last_peered":"2024-07-26T11:13:33.868112+0000","last_clean":"2024-07-26T11:13:33.868112+0000","last_became_active":"2024-07-26T11:10:22.298707+0000","last_became_peered":"2024-07-26T11:10:22.298707+0000","last_unstale":"2024-07-26T11:13:33.868112+0000","last_undegraded":"2024-07-26T11:13:33.868112+0000","last_fullsized":"2024-07-26T11:13:33.868112+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:25.161634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.870991+0000","last_change":"2024-07-26T11:10:22.302476+0000","last_active":"2024-07-26T11:13:33.870991+0000","last_peered":"2024-07-26T11:13:33.870991+0000","last_clean":"2024-07-26T11:13:33.870991+0000","last_became_active":"2024-07-26T11:10:22.302091+0000","last_became_peered":"2024-07-26T11:10:22.302091+0000","last_unstale":"2024-07-26T11:13:33.870991+0000","last_undegraded":"2024-07-26T11:13:33.870991+0000","last_fullsized":"2024-07-26T11:13:33.870991+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:06:18.587588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.872438+0000","last_change":"2024-07-26T11:10:22.326583+0000","last_active":"2024-07-26T11:13:33.872438+0000","last_peered":"2024-07-26T11:13:33.872438+0000","last_clean":"2024-07-26T11:13:33.872438+0000","last_became_active":"2024-07-26T11:10:22.326185+0000","last_became_peered":"2024-07-26T11:10:22.326185+0000","last_unstale":"2024-07-26T11:13:33.872438+0000","last_undegraded":"2024-07-26T11:13:33.872438+0000","last_fullsized":"2024-07-26T11:13:33.872438+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:54:00.148964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.894353+0000","last_change":"2024-07-26T11:10:22.329247+0000","last_active":"2024-07-26T11:13:33.894353+0000","last_peered":"2024-07-26T11:13:33.894353+0000","last_clean":"2024-07-26T11:13:33.894353+0000","last_became_active":"2024-07-26T11:10:22.328585+0000","last_became_peered":"2024-07-26T11:10:22.328585+0000","last_unstale":"2024-07-26T11:13:33.894353+0000","last_undegraded":"2024-07-26T11:13:33.894353+0000","last_fullsized":"2024-07-26T11:13:33.894353+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:01:27.195399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.895507+0000","last_change":"2024-07-26T11:10:22.330717+0000","last_active":"2024-07-26T11:13:33.895507+0000","last_peered":"2024-07-26T11:13:33.895507+0000","last_clean":"2024-07-26T11:13:33.895507+0000","last_became_active":"2024-07-26T11:10:22.329576+0000","last_became_peered":"2024-07-26T11:10:22.329576+0000","last_unstale":"2024-07-26T11:13:33.895507+0000","last_undegraded":"2024-07-26T11:13:33.895507+0000","last_fullsized":"2024-07-26T11:13:33.895507+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:21.970629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.867561+0000","last_change":"2024-07-26T11:10:22.307445+0000","last_active":"2024-07-26T11:13:33.867561+0000","last_peered":"2024-07-26T11:13:33.867561+0000","last_clean":"2024-07-26T11:13:33.867561+0000","last_became_active":"2024-07-26T11:10:22.305738+0000","last_became_peered":"2024-07-26T11:10:22.305738+0000","last_unstale":"2024-07-26T11:13:33.867561+0000","last_undegraded":"2024-07-26T11:13:33.867561+0000","last_fullsized":"2024-07-26T11:13:33.867561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:09:08.125209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874453+0000","last_change":"2024-07-26T11:10:22.298392+0000","last_active":"2024-07-26T11:13:33.874453+0000","last_peered":"2024-07-26T11:13:33.874453+0000","last_clean":"2024-07-26T11:13:33.874453+0000","last_became_active":"2024-07-26T11:10:22.297979+0000","last_became_peered":"2024-07-26T11:10:22.297979+0000","last_unstale":"2024-07-26T11:13:33.874453+0000","last_undegraded":"2024-07-26T11:13:33.874453+0000","last_fullsized":"2024-07-26T11:13:33.874453+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:30:39.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.871413+0000","last_change":"2024-07-26T11:10:22.320850+0000","last_active":"2024-07-26T11:13:33.871413+0000","last_peered":"2024-07-26T11:13:33.871413+0000","last_clean":"2024-07-26T11:13:33.871413+0000","last_became_active":"2024-07-26T11:10:22.319838+0000","last_became_peered":"2024-07-26T11:10:22.319838+0000","last_unstale":"2024-07-26T11:13:33.871413+0000","last_undegraded":"2024-07-26T11:13:33.871413+0000","last_fullsized":"2024-07-26T11:13:33.871413+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:14.289366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.873003+0000","last_change":"2024-07-26T11:10:22.321305+0000","last_active":"2024-07-26T11:13:33.873003+0000","last_peered":"2024-07-26T11:13:33.873003+0000","last_clean":"2024-07-26T11:13:33.873003+0000","last_became_active":"2024-07-26T11:10:22.320733+0000","last_became_peered":"2024-07-26T11:10:22.320733+0000","last_unstale":"2024-07-26T11:13:33.873003+0000","last_undegraded":"2024-07-26T11:13:33.873003+0000","last_fullsized":"2024-07-26T11:13:33.873003+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:24:33.125828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874282+0000","last_change":"2024-07-26T11:10:22.318193+0000","last_active":"2024-07-26T11:13:33.874282+0000","last_peered":"2024-07-26T11:13:33.874282+0000","last_clean":"2024-07-26T11:13:33.874282+0000","last_became_active":"2024-07-26T11:10:22.317819+0000","last_became_peered":"2024-07-26T11:10:22.317819+0000","last_unstale":"2024-07-26T11:13:33.874282+0000","last_undegraded":"2024-07-26T11:13:33.874282+0000","last_fullsized":"2024-07-26T11:13:33.874282+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:14:06.105152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875131+0000","last_change":"2024-07-26T11:10:22.318222+0000","last_active":"2024-07-26T11:13:33.875131+0000","last_peered":"2024-07-26T11:13:33.875131+0000","last_clean":"2024-07-26T11:13:33.875131+0000","last_became_active":"2024-07-26T11:10:22.317806+0000","last_became_peered":"2024-07-26T11:10:22.317806+0000","last_unstale":"2024-07-26T11:13:33.875131+0000","last_undegraded":"2024-07-26T11:13:33.875131+0000","last_fullsized":"2024-07-26T11:13:33.875131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:55:27.510542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.895175+0000","last_change":"2024-07-26T11:10:22.330166+0000","last_active":"2024-07-26T11:13:33.895175+0000","last_peered":"2024-07-26T11:13:33.895175+0000","last_clean":"2024-07-26T11:13:33.895175+0000","last_became_active":"2024-07-26T11:10:22.329722+0000","last_became_peered":"2024-07-26T11:10:22.329722+0000","last_unstale":"2024-07-26T11:13:33.895175+0000","last_undegraded":"2024-07-26T11:13:33.895175+0000","last_fullsized":"2024-07-26T11:13:33.895175+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:58.816139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.866466+0000","last_change":"2024-07-26T11:10:22.302774+0000","last_active":"2024-07-26T11:13:33.866466+0000","last_peered":"2024-07-26T11:13:33.866466+0000","last_clean":"2024-07-26T11:13:33.866466+0000","last_became_active":"2024-07-26T11:10:22.302382+0000","last_became_peered":"2024-07-26T11:10:22.302382+0000","last_unstale":"2024-07-26T11:13:33.866466+0000","last_undegraded":"2024-07-26T11:13:33.866466+0000","last_fullsized":"2024-07-26T11:13:33.866466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:02:58.019598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.886235+0000","last_change":"2024-07-26T11:10:22.312475+0000","last_active":"2024-07-26T11:13:33.886235+0000","last_peered":"2024-07-26T11:13:33.886235+0000","last_clean":"2024-07-26T11:13:33.886235+0000","last_became_active":"2024-07-26T11:10:22.312078+0000","last_became_peered":"2024-07-26T11:10:22.312078+0000","last_unstale":"2024-07-26T11:13:33.886235+0000","last_undegraded":"2024-07-26T11:13:33.886235+0000","last_fullsized":"2024-07-26T11:13:33.886235+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:25:37.705136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.877966+0000","last_change":"2024-07-26T11:10:22.314776+0000","last_active":"2024-07-26T11:13:33.877966+0000","last_peered":"2024-07-26T11:13:33.877966+0000","last_clean":"2024-07-26T11:13:33.877966+0000","last_became_active":"2024-07-26T11:10:22.307808+0000","last_became_peered":"2024-07-26T11:10:22.307808+0000","last_unstale":"2024-07-26T11:13:33.877966+0000","last_undegraded":"2024-07-26T11:13:33.877966+0000","last_fullsized":"2024-07-26T11:13:33.877966+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:44:06.152799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.891049+0000","last_change":"2024-07-26T11:10:22.328102+0000","last_active":"2024-07-26T11:13:33.891049+0000","last_peered":"2024-07-26T11:13:33.891049+0000","last_clean":"2024-07-26T11:13:33.891049+0000","last_became_active":"2024-07-26T11:10:22.327779+0000","last_became_peered":"2024-07-26T11:10:22.327779+0000","last_unstale":"2024-07-26T11:13:33.891049+0000","last_undegraded":"2024-07-26T11:13:33.891049+0000","last_fullsized":"2024-07-26T11:13:33.891049+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:14:34.569748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.883870+0000","last_change":"2024-07-26T11:10:22.329057+0000","last_active":"2024-07-26T11:13:33.883870+0000","last_peered":"2024-07-26T11:13:33.883870+0000","last_clean":"2024-07-26T11:13:33.883870+0000","last_became_active":"2024-07-26T11:10:22.328656+0000","last_became_peered":"2024-07-26T11:10:22.328656+0000","last_unstale":"2024-07-26T11:13:33.883870+0000","last_undegraded":"2024-07-26T11:13:33.883870+0000","last_fullsized":"2024-07-26T11:13:33.883870+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:18:39.861674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.865904+0000","last_change":"2024-07-26T11:10:22.326115+0000","last_active":"2024-07-26T11:13:33.865904+0000","last_peered":"2024-07-26T11:13:33.865904+0000","last_clean":"2024-07-26T11:13:33.865904+0000","last_became_active":"2024-07-26T11:10:22.325538+0000","last_became_peered":"2024-07-26T11:10:22.325538+0000","last_unstale":"2024-07-26T11:13:33.865904+0000","last_undegraded":"2024-07-26T11:13:33.865904+0000","last_fullsized":"2024-07-26T11:13:33.865904+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:18:19.339571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.892395+0000","last_change":"2024-07-26T11:10:22.326910+0000","last_active":"2024-07-26T11:13:33.892395+0000","last_peered":"2024-07-26T11:13:33.892395+0000","last_clean":"2024-07-26T11:13:33.892395+0000","last_became_active":"2024-07-26T11:10:22.324532+0000","last_became_peered":"2024-07-26T11:10:22.324532+0000","last_unstale":"2024-07-26T11:13:33.892395+0000","last_undegraded":"2024-07-26T11:13:33.892395+0000","last_fullsized":"2024-07-26T11:13:33.892395+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:33.671780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.890614+0000","last_change":"2024-07-26T11:10:22.312926+0000","last_active":"2024-07-26T11:13:33.890614+0000","last_peered":"2024-07-26T11:13:33.890614+0000","last_clean":"2024-07-26T11:13:33.890614+0000","last_became_active":"2024-07-26T11:10:22.311642+0000","last_became_peered":"2024-07-26T11:10:22.311642+0000","last_unstale":"2024-07-26T11:13:33.890614+0000","last_undegraded":"2024-07-26T11:13:33.890614+0000","last_fullsized":"2024-07-26T11:13:33.890614+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:56:29.644965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.877291+0000","last_change":"2024-07-26T11:10:34.976934+0000","last_active":"2024-07-26T11:13:33.877291+0000","last_peered":"2024-07-26T11:13:33.877291+0000","last_clean":"2024-07-26T11:13:33.877291+0000","last_became_active":"2024-07-26T11:10:34.976551+0000","last_became_peered":"2024-07-26T11:10:34.976551+0000","last_unstale":"2024-07-26T11:13:33.877291+0000","last_undegraded":"2024-07-26T11:13:33.877291+0000","last_fullsized":"2024-07-26T11:13:33.877291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:11:43.883644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875097+0000","last_change":"2024-07-26T11:10:22.304444+0000","last_active":"2024-07-26T11:13:33.875097+0000","last_peered":"2024-07-26T11:13:33.875097+0000","last_clean":"2024-07-26T11:13:33.875097+0000","last_became_active":"2024-07-26T11:10:22.302825+0000","last_became_peered":"2024-07-26T11:10:22.302825+0000","last_unstale":"2024-07-26T11:13:33.875097+0000","last_undegraded":"2024-07-26T11:13:33.875097+0000","last_fullsized":"2024-07-26T11:13:33.875097+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:15.229057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.870606+0000","last_change":"2024-07-26T11:10:22.297515+0000","last_active":"2024-07-26T11:13:33.870606+0000","last_peered":"2024-07-26T11:13:33.870606+0000","last_clean":"2024-07-26T11:13:33.870606+0000","last_became_active":"2024-07-26T11:10:22.297134+0000","last_became_peered":"2024-07-26T11:10:22.297134+0000","last_unstale":"2024-07-26T11:13:33.870606+0000","last_undegraded":"2024-07-26T11:13:33.870606+0000","last_fullsized":"2024-07-26T11:13:33.870606+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:25:08.794363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875580+0000","last_change":"2024-07-26T11:10:22.304333+0000","last_active":"2024-07-26T11:13:33.875580+0000","last_peered":"2024-07-26T11:13:33.875580+0000","last_clean":"2024-07-26T11:13:33.875580+0000","last_became_active":"2024-07-26T11:10:22.303639+0000","last_became_peered":"2024-07-26T11:10:22.303639+0000","last_unstale":"2024-07-26T11:13:33.875580+0000","last_undegraded":"2024-07-26T11:13:33.875580+0000","last_fullsized":"2024-07-26T11:13:33.875580+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:47.053463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874832+0000","last_change":"2024-07-26T11:10:22.320998+0000","last_active":"2024-07-26T11:13:33.874832+0000","last_peered":"2024-07-26T11:13:33.874832+0000","last_clean":"2024-07-26T11:13:33.874832+0000","last_became_active":"2024-07-26T11:10:22.320610+0000","last_became_peered":"2024-07-26T11:10:22.320610+0000","last_unstale":"2024-07-26T11:13:33.874832+0000","last_undegraded":"2024-07-26T11:13:33.874832+0000","last_fullsized":"2024-07-26T11:13:33.874832+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:52:28.802241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.865671+0000","last_change":"2024-07-26T11:10:34.970486+0000","last_active":"2024-07-26T11:13:33.865671+0000","last_peered":"2024-07-26T11:13:33.865671+0000","last_clean":"2024-07-26T11:13:33.865671+0000","last_became_active":"2024-07-26T11:10:34.969965+0000","last_became_peered":"2024-07-26T11:10:34.969965+0000","last_unstale":"2024-07-26T11:13:33.865671+0000","last_undegraded":"2024-07-26T11:13:33.865671+0000","last_fullsized":"2024-07-26T11:13:33.865671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:41.593251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.876426+0000","last_change":"2024-07-26T11:10:22.321261+0000","last_active":"2024-07-26T11:13:33.876426+0000","last_peered":"2024-07-26T11:13:33.876426+0000","last_clean":"2024-07-26T11:13:33.876426+0000","last_became_active":"2024-07-26T11:10:22.320819+0000","last_became_peered":"2024-07-26T11:10:22.320819+0000","last_unstale":"2024-07-26T11:13:33.876426+0000","last_undegraded":"2024-07-26T11:13:33.876426+0000","last_fullsized":"2024-07-26T11:13:33.876426+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:04:56.872600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.887326+0000","last_change":"2024-07-26T11:10:22.321206+0000","last_active":"2024-07-26T11:13:33.887326+0000","last_peered":"2024-07-26T11:13:33.887326+0000","last_clean":"2024-07-26T11:13:33.887326+0000","last_became_active":"2024-07-26T11:10:22.320700+0000","last_became_peered":"2024-07-26T11:10:22.320700+0000","last_unstale":"2024-07-26T11:13:33.887326+0000","last_undegraded":"2024-07-26T11:13:33.887326+0000","last_fullsized":"2024-07-26T11:13:33.887326+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:31:20.700808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.870762+0000","last_change":"2024-07-26T11:10:22.300061+0000","last_active":"2024-07-26T11:13:33.870762+0000","last_peered":"2024-07-26T11:13:33.870762+0000","last_clean":"2024-07-26T11:13:33.870762+0000","last_became_active":"2024-07-26T11:10:22.298737+0000","last_became_peered":"2024-07-26T11:10:22.298737+0000","last_unstale":"2024-07-26T11:13:33.870762+0000","last_undegraded":"2024-07-26T11:13:33.870762+0000","last_fullsized":"2024-07-26T11:13:33.870762+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:45:25.500816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889377+0000","last_change":"2024-07-26T11:10:22.328534+0000","last_active":"2024-07-26T11:13:33.889377+0000","last_peered":"2024-07-26T11:13:33.889377+0000","last_clean":"2024-07-26T11:13:33.889377+0000","last_became_active":"2024-07-26T11:10:22.327929+0000","last_became_peered":"2024-07-26T11:10:22.327929+0000","last_unstale":"2024-07-26T11:13:33.889377+0000","last_undegraded":"2024-07-26T11:13:33.889377+0000","last_fullsized":"2024-07-26T11:13:33.889377+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:09:55.152881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.890781+0000","last_change":"2024-07-26T11:10:22.324753+0000","last_active":"2024-07-26T11:13:33.890781+0000","last_peered":"2024-07-26T11:13:33.890781+0000","last_clean":"2024-07-26T11:13:33.890781+0000","last_became_active":"2024-07-26T11:10:22.324307+0000","last_became_peered":"2024-07-26T11:10:22.324307+0000","last_unstale":"2024-07-26T11:13:33.890781+0000","last_undegraded":"2024-07-26T11:13:33.890781+0000","last_fullsized":"2024-07-26T11:13:33.890781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:40.566465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.890918+0000","last_change":"2024-07-26T11:10:22.328183+0000","last_active":"2024-07-26T11:13:33.890918+0000","last_peered":"2024-07-26T11:13:33.890918+0000","last_clean":"2024-07-26T11:13:33.890918+0000","last_became_active":"2024-07-26T11:10:22.327805+0000","last_became_peered":"2024-07-26T11:10:22.327805+0000","last_unstale":"2024-07-26T11:13:33.890918+0000","last_undegraded":"2024-07-26T11:13:33.890918+0000","last_fullsized":"2024-07-26T11:13:33.890918+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:11:23.238390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73723,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1267,"num_read_kb":712,"num_write":258,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2194,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":221169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":73,"seq":313532612667,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":527167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808895,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1732608,"data_stored":1180030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972421,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":529939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168652,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":529735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364880,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28620,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716532,"statfs":{"total":95995035648,"available":95965728768,"internally_reserved":0,"allocated":1642496,"data_stored":1122187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528406,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":521347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724638,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36952,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708200,"statfs":{"total":95995035648,"available":95957196800,"internally_reserved":0,"allocated":1654784,"data_stored":1113813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888163,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1118208,"data_stored":592722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084391,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":521055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378733,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":522597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542258,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":589146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705782,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":69037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T11:13:58.184 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph pg dump --format=json 2024-07-26T11:13:58.688 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:13:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:59 smithi134 ceph-mon[31966]: from='client.25121 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:13:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:13:59 smithi134 ceph-mon[31966]: pgmap v420: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:13:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:59 smithi189 ceph-mon[36790]: from='client.25121 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:13:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:13:59 smithi189 ceph-mon[36790]: pgmap v420: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:13:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:59 smithi193 ceph-mon[37473]: from='client.25121 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:13:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:13:59 smithi193 ceph-mon[37473]: pgmap v420: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:00.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.940+0000 7f6506200640 1 -- 172.21.15.134:0/2700351835 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6500075b30 msgr2=0x7f650010f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:00.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.940+0000 7f6506200640 1 --2- 172.21.15.134:0/2700351835 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6500075b30 0x7f650010f400 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f64e8009900 tx=0x7f64e802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:14:00.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.941+0000 7f6506200640 1 -- 172.21.15.134:0/2700351835 shutdown_connections 2024-07-26T11:14:00.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.941+0000 7f6506200640 1 --2- 172.21.15.134:0/2700351835 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f650010f940 0x7f6500111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:00.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.941+0000 7f6506200640 1 --2- 172.21.15.134:0/2700351835 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6500075b30 0x7f650010f400 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:00.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.941+0000 7f6506200640 1 --2- 172.21.15.134:0/2700351835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65000751e0 0x7f65000755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:00.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.941+0000 7f6506200640 1 -- 172.21.15.134:0/2700351835 >> 172.21.15.134:0/2700351835 conn(0x7f65000fee70 msgr2=0x7f65001012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:00.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.941+0000 7f6506200640 1 -- 172.21.15.134:0/2700351835 shutdown_connections 2024-07-26T11:14:00.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.941+0000 7f6506200640 1 -- 172.21.15.134:0/2700351835 wait complete. 2024-07-26T11:14:00.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.942+0000 7f6506200640 1 Processor -- start 2024-07-26T11:14:00.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.942+0000 7f6506200640 1 -- start start 2024-07-26T11:14:00.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.942+0000 7f6506200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65000751e0 0x7f650019bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:00.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.942+0000 7f6506200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6500075b30 0x7f650019c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:00.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.942+0000 7f6506200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f650010f940 0x7f65001a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:00.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.942+0000 7f6506200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f65001154e0 con 0x7f6500075b30 2024-07-26T11:14:00.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.942+0000 7f6506200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6500115360 con 0x7f65000751e0 2024-07-26T11:14:00.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.942+0000 7f6506200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6500115660 con 0x7f650010f940 2024-07-26T11:14:00.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.943+0000 7f6504e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65000751e0 0x7f650019bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:00.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.943+0000 7f6504e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65000751e0 0x7f650019bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:45472/0 (socket says 172.21.15.134:45472) 2024-07-26T11:14:00.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.943+0000 7f6504e00640 1 -- 172.21.15.134:0/954473045 learned_addr learned my addr 172.21.15.134:0/954473045 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:00.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.943+0000 7f64ffe00640 1 --2- 172.21.15.134:0/954473045 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6500075b30 0x7f650019c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:00.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.943+0000 7f6505800640 1 --2- 172.21.15.134:0/954473045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f650010f940 0x7f65001a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:00.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.943+0000 7f6504e00640 1 -- 172.21.15.134:0/954473045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f650010f940 msgr2=0x7f65001a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:00.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.943+0000 7f6504e00640 1 --2- 172.21.15.134:0/954473045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f650010f940 0x7f65001a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:00.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.943+0000 7f6504e00640 1 -- 172.21.15.134:0/954473045 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6500075b30 msgr2=0x7f650019c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:00.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.943+0000 7f6504e00640 1 --2- 172.21.15.134:0/954473045 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6500075b30 0x7f650019c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:00.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.943+0000 7f6504e00640 1 -- 172.21.15.134:0/954473045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65001a1e60 con 0x7f65000751e0 2024-07-26T11:14:00.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.944+0000 7f6504e00640 1 --2- 172.21.15.134:0/954473045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65000751e0 0x7f650019bef0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f64f000ea60 tx=0x7f64f000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:00.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.944+0000 7f64fd600640 1 -- 172.21.15.134:0/954473045 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f000cde0 con 0x7f65000751e0 2024-07-26T11:14:00.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.944+0000 7f64fd600640 1 -- 172.21.15.134:0/954473045 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64f00035f0 con 0x7f65000751e0 2024-07-26T11:14:00.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.944+0000 7f6506200640 1 -- 172.21.15.134:0/954473045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65001a2150 con 0x7f65000751e0 2024-07-26T11:14:00.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.944+0000 7f64fd600640 1 -- 172.21.15.134:0/954473045 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64f0010690 con 0x7f65000751e0 2024-07-26T11:14:00.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.944+0000 7f6506200640 1 -- 172.21.15.134:0/954473045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65001a88b0 con 0x7f65000751e0 2024-07-26T11:14:00.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.945+0000 7f6506200640 1 -- 172.21.15.134:0/954473045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65000755f0 con 0x7f65000751e0 2024-07-26T11:14:00.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.948+0000 7f64fd600640 1 -- 172.21.15.134:0/954473045 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f64f00037a0 con 0x7f65000751e0 2024-07-26T11:14:00.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.949+0000 7f64fd600640 1 --2- 172.21.15.134:0/954473045 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64e40b3c20 0x7f64e40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:00.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.949+0000 7f64fd600640 1 -- 172.21.15.134:0/954473045 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f64f00e4470 con 0x7f65000751e0 2024-07-26T11:14:00.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.949+0000 7f64ffe00640 1 --2- 172.21.15.134:0/954473045 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64e40b3c20 0x7f64e40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:00.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.950+0000 7f64ffe00640 1 --2- 172.21.15.134:0/954473045 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64e40b3c20 0x7f64e40b6130 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f64e8000c00 tx=0x7f64e8031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:00.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:00.953+0000 7f64fd600640 1 -- 172.21.15.134:0/954473045 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f64f00ab5c0 con 0x7f65000751e0 2024-07-26T11:14:01.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:01.150+0000 7f6506200640 1 -- 172.21.15.134:0/954473045 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6500109bf0 con 0x7f64e40b3c20 2024-07-26T11:14:01.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:01.157+0000 7f64fd600640 1 -- 172.21.15.134:0/954473045 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340514 (secure 0 0 0) 0x7f6500109bf0 con 0x7f64e40b3c20 2024-07-26T11:14:01.161 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:14:01.165 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-07-26T11:14:01.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:01.165+0000 7f6506200640 1 -- 172.21.15.134:0/954473045 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64e40b3c20 msgr2=0x7f64e40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:01.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:01.165+0000 7f6506200640 1 --2- 172.21.15.134:0/954473045 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64e40b3c20 0x7f64e40b6130 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f64e8000c00 tx=0x7f64e8031400 comp rx=0 tx=0).stop 2024-07-26T11:14:01.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:01.165+0000 7f6506200640 1 -- 172.21.15.134:0/954473045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65000751e0 msgr2=0x7f650019bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:01.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:01.165+0000 7f6506200640 1 --2- 172.21.15.134:0/954473045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65000751e0 0x7f650019bef0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f64f000ea60 tx=0x7f64f000ef20 comp rx=0 tx=0).stop 2024-07-26T11:14:01.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:01.165+0000 7f6506200640 1 -- 172.21.15.134:0/954473045 shutdown_connections 2024-07-26T11:14:01.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:01.165+0000 7f6506200640 1 --2- 172.21.15.134:0/954473045 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64e40b3c20 0x7f64e40b6130 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:01.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:01.165+0000 7f6506200640 1 --2- 172.21.15.134:0/954473045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f650010f940 0x7f65001a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:01.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:01.165+0000 7f6506200640 1 --2- 172.21.15.134:0/954473045 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6500075b30 0x7f650019c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:01.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:01.165+0000 7f6506200640 1 --2- 172.21.15.134:0/954473045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65000751e0 0x7f650019bef0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:01.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:01.165+0000 7f6506200640 1 -- 172.21.15.134:0/954473045 >> 172.21.15.134:0/954473045 conn(0x7f65000fee70 msgr2=0x7f65001094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:01.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:01.165+0000 7f6506200640 1 -- 172.21.15.134:0/954473045 shutdown_connections 2024-07-26T11:14:01.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:01.165+0000 7f6506200640 1 -- 172.21.15.134:0/954473045 wait complete. 2024-07-26T11:14:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:01 smithi134 ceph-mon[31966]: pgmap v421: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:01 smithi189 ceph-mon[36790]: pgmap v421: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:01 smithi193 ceph-mon[37473]: pgmap v421: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:01.997 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-07-26T11:13:59.988485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664325,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1398,"num_read_kb":806,"num_write":383,"num_write_kb":2303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":2954034,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":816,"ondisk_log_size":816,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375884,"kb_used_data":14252,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565940,"statfs":{"total":1151940427776,"available":1151555522560,"internally_reserved":0,"allocated":14594048,"data_stored":8272541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":115,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007684"},"pg_stats":[{"pgid":"2.1f","version":"90'13","reported_seq":195,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:55.344177+0000","last_change":"2024-07-26T11:10:34.985984+0000","last_active":"2024-07-26T11:13:55.344177+0000","last_peered":"2024-07-26T11:13:55.344177+0000","last_clean":"2024-07-26T11:13:55.344177+0000","last_became_active":"2024-07-26T11:10:34.980894+0000","last_became_peered":"2024-07-26T11:10:34.980894+0000","last_unstale":"2024-07-26T11:13:55.344177+0000","last_undegraded":"2024-07-26T11:13:55.344177+0000","last_fullsized":"2024-07-26T11:13:55.344177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:55.718875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":127,"num_read_kb":127,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874263+0000","last_change":"2024-07-26T11:10:22.320946+0000","last_active":"2024-07-26T11:13:33.874263+0000","last_peered":"2024-07-26T11:13:33.874263+0000","last_clean":"2024-07-26T11:13:33.874263+0000","last_became_active":"2024-07-26T11:10:22.316181+0000","last_became_peered":"2024-07-26T11:10:22.316181+0000","last_unstale":"2024-07-26T11:13:33.874263+0000","last_undegraded":"2024-07-26T11:13:33.874263+0000","last_fullsized":"2024-07-26T11:13:33.874263+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:13.641862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.155734+0000","last_change":"2024-07-26T11:10:35.984460+0000","last_active":"2024-07-26T11:11:35.155734+0000","last_peered":"2024-07-26T11:11:35.155734+0000","last_clean":"2024-07-26T11:11:35.155734+0000","last_became_active":"2024-07-26T11:10:35.983922+0000","last_became_peered":"2024-07-26T11:10:35.983922+0000","last_unstale":"2024-07-26T11:11:35.155734+0000","last_undegraded":"2024-07-26T11:11:35.155734+0000","last_fullsized":"2024-07-26T11:11:35.155734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:51:46.140910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:48.051375+0000","last_change":"2024-07-26T11:10:18.058553+0000","last_active":"2024-07-26T11:11:48.051375+0000","last_peered":"2024-07-26T11:11:48.051375+0000","last_clean":"2024-07-26T11:11:48.051375+0000","last_became_active":"2024-07-26T11:10:18.058034+0000","last_became_peered":"2024-07-26T11:10:18.058034+0000","last_unstale":"2024-07-26T11:11:48.051375+0000","last_undegraded":"2024-07-26T11:11:48.051375+0000","last_fullsized":"2024-07-26T11:11:48.051375+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:48:58.868988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.893887+0000","last_change":"2024-07-26T11:10:22.326760+0000","last_active":"2024-07-26T11:13:33.893887+0000","last_peered":"2024-07-26T11:13:33.893887+0000","last_clean":"2024-07-26T11:13:33.893887+0000","last_became_active":"2024-07-26T11:10:22.326275+0000","last_became_peered":"2024-07-26T11:10:22.326275+0000","last_unstale":"2024-07-26T11:13:33.893887+0000","last_undegraded":"2024-07-26T11:13:33.893887+0000","last_fullsized":"2024-07-26T11:13:33.893887+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:51:57.461453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.406615+0000","last_change":"2024-07-26T11:10:35.997585+0000","last_active":"2024-07-26T11:11:35.406615+0000","last_peered":"2024-07-26T11:11:35.406615+0000","last_clean":"2024-07-26T11:11:35.406615+0000","last_became_active":"2024-07-26T11:10:35.997274+0000","last_became_peered":"2024-07-26T11:10:35.997274+0000","last_unstale":"2024-07-26T11:11:35.406615+0000","last_undegraded":"2024-07-26T11:11:35.406615+0000","last_fullsized":"2024-07-26T11:11:35.406615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:43.553396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.096244+0000","last_change":"2024-07-26T11:10:18.034964+0000","last_active":"2024-07-26T11:11:51.096244+0000","last_peered":"2024-07-26T11:11:51.096244+0000","last_clean":"2024-07-26T11:11:51.096244+0000","last_became_active":"2024-07-26T11:10:18.034569+0000","last_became_peered":"2024-07-26T11:10:18.034569+0000","last_unstale":"2024-07-26T11:11:51.096244+0000","last_undegraded":"2024-07-26T11:11:51.096244+0000","last_fullsized":"2024-07-26T11:11:51.096244+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:00:46.542458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.882415+0000","last_change":"2024-07-26T11:10:22.310695+0000","last_active":"2024-07-26T11:13:33.882415+0000","last_peered":"2024-07-26T11:13:33.882415+0000","last_clean":"2024-07-26T11:13:33.882415+0000","last_became_active":"2024-07-26T11:10:22.310142+0000","last_became_peered":"2024-07-26T11:10:22.310142+0000","last_unstale":"2024-07-26T11:13:33.882415+0000","last_undegraded":"2024-07-26T11:13:33.882415+0000","last_fullsized":"2024-07-26T11:13:33.882415+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:49:18.890802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.407144+0000","last_change":"2024-07-26T11:10:35.991418+0000","last_active":"2024-07-26T11:11:35.407144+0000","last_peered":"2024-07-26T11:11:35.407144+0000","last_clean":"2024-07-26T11:11:35.407144+0000","last_became_active":"2024-07-26T11:10:35.990934+0000","last_became_peered":"2024-07-26T11:10:35.990934+0000","last_unstale":"2024-07-26T11:11:35.407144+0000","last_undegraded":"2024-07-26T11:11:35.407144+0000","last_fullsized":"2024-07-26T11:11:35.407144+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:56.598738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.098452+0000","last_change":"2024-07-26T11:10:34.965637+0000","last_active":"2024-07-26T11:11:51.098452+0000","last_peered":"2024-07-26T11:11:51.098452+0000","last_clean":"2024-07-26T11:11:51.098452+0000","last_became_active":"2024-07-26T11:10:34.961157+0000","last_became_peered":"2024-07-26T11:10:34.961157+0000","last_unstale":"2024-07-26T11:11:51.098452+0000","last_undegraded":"2024-07-26T11:11:51.098452+0000","last_fullsized":"2024-07-26T11:11:51.098452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:26:17.007258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874776+0000","last_change":"2024-07-26T11:10:22.303606+0000","last_active":"2024-07-26T11:13:33.874776+0000","last_peered":"2024-07-26T11:13:33.874776+0000","last_clean":"2024-07-26T11:13:33.874776+0000","last_became_active":"2024-07-26T11:10:22.303183+0000","last_became_peered":"2024-07-26T11:10:22.303183+0000","last_unstale":"2024-07-26T11:13:33.874776+0000","last_undegraded":"2024-07-26T11:13:33.874776+0000","last_fullsized":"2024-07-26T11:13:33.874776+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:31:01.153694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.406757+0000","last_change":"2024-07-26T11:10:35.981446+0000","last_active":"2024-07-26T11:11:35.406757+0000","last_peered":"2024-07-26T11:11:35.406757+0000","last_clean":"2024-07-26T11:11:35.406757+0000","last_became_active":"2024-07-26T11:10:35.980994+0000","last_became_peered":"2024-07-26T11:10:35.980994+0000","last_unstale":"2024-07-26T11:11:35.406757+0000","last_undegraded":"2024-07-26T11:11:35.406757+0000","last_fullsized":"2024-07-26T11:11:35.406757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:38:57.405325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.887131+0000","last_change":"2024-07-26T11:10:22.306758+0000","last_active":"2024-07-26T11:13:33.887131+0000","last_peered":"2024-07-26T11:13:33.887131+0000","last_clean":"2024-07-26T11:13:33.887131+0000","last_became_active":"2024-07-26T11:10:22.306370+0000","last_became_peered":"2024-07-26T11:10:22.306370+0000","last_unstale":"2024-07-26T11:13:33.887131+0000","last_undegraded":"2024-07-26T11:13:33.887131+0000","last_fullsized":"2024-07-26T11:13:33.887131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:51:43.069470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.095857+0000","last_change":"2024-07-26T11:10:18.035664+0000","last_active":"2024-07-26T11:11:51.095857+0000","last_peered":"2024-07-26T11:11:51.095857+0000","last_clean":"2024-07-26T11:11:51.095857+0000","last_became_active":"2024-07-26T11:10:18.035113+0000","last_became_peered":"2024-07-26T11:10:18.035113+0000","last_unstale":"2024-07-26T11:11:51.095857+0000","last_undegraded":"2024-07-26T11:11:51.095857+0000","last_fullsized":"2024-07-26T11:11:51.095857+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:09:30.641034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.407076+0000","last_change":"2024-07-26T11:10:35.990907+0000","last_active":"2024-07-26T11:11:35.407076+0000","last_peered":"2024-07-26T11:11:35.407076+0000","last_clean":"2024-07-26T11:11:35.407076+0000","last_became_active":"2024-07-26T11:10:35.990417+0000","last_became_peered":"2024-07-26T11:10:35.990417+0000","last_unstale":"2024-07-26T11:11:35.407076+0000","last_undegraded":"2024-07-26T11:11:35.407076+0000","last_fullsized":"2024-07-26T11:11:35.407076+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:24:39.319727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889003+0000","last_change":"2024-07-26T11:10:22.333575+0000","last_active":"2024-07-26T11:13:33.889003+0000","last_peered":"2024-07-26T11:13:33.889003+0000","last_clean":"2024-07-26T11:13:33.889003+0000","last_became_active":"2024-07-26T11:10:22.332802+0000","last_became_peered":"2024-07-26T11:10:22.332802+0000","last_unstale":"2024-07-26T11:13:33.889003+0000","last_undegraded":"2024-07-26T11:13:33.889003+0000","last_fullsized":"2024-07-26T11:13:33.889003+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:14:03.238840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.059445+0000","last_change":"2024-07-26T11:10:18.047942+0000","last_active":"2024-07-26T11:11:51.059445+0000","last_peered":"2024-07-26T11:11:51.059445+0000","last_clean":"2024-07-26T11:11:51.059445+0000","last_became_active":"2024-07-26T11:10:18.047500+0000","last_became_peered":"2024-07-26T11:10:18.047500+0000","last_unstale":"2024-07-26T11:11:51.059445+0000","last_undegraded":"2024-07-26T11:11:51.059445+0000","last_fullsized":"2024-07-26T11:11:51.059445+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:32:30.773648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:11.796009+0000","last_change":"2024-07-26T11:10:36.000946+0000","last_active":"2024-07-26T11:12:11.796009+0000","last_peered":"2024-07-26T11:12:11.796009+0000","last_clean":"2024-07-26T11:12:11.796009+0000","last_became_active":"2024-07-26T11:10:36.000251+0000","last_became_peered":"2024-07-26T11:10:36.000251+0000","last_unstale":"2024-07-26T11:12:11.796009+0000","last_undegraded":"2024-07-26T11:12:11.796009+0000","last_fullsized":"2024-07-26T11:12:11.796009+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:03:40.523747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.890224+0000","last_change":"2024-07-26T11:10:22.326370+0000","last_active":"2024-07-26T11:13:33.890224+0000","last_peered":"2024-07-26T11:13:33.890224+0000","last_clean":"2024-07-26T11:13:33.890224+0000","last_became_active":"2024-07-26T11:10:22.325961+0000","last_became_peered":"2024-07-26T11:10:22.325961+0000","last_unstale":"2024-07-26T11:13:33.890224+0000","last_undegraded":"2024-07-26T11:13:33.890224+0000","last_fullsized":"2024-07-26T11:13:33.890224+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:16:36.946750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.089544+0000","last_change":"2024-07-26T11:10:18.051940+0000","last_active":"2024-07-26T11:11:50.089544+0000","last_peered":"2024-07-26T11:11:50.089544+0000","last_clean":"2024-07-26T11:11:50.089544+0000","last_became_active":"2024-07-26T11:10:18.051565+0000","last_became_peered":"2024-07-26T11:10:18.051565+0000","last_unstale":"2024-07-26T11:11:50.089544+0000","last_undegraded":"2024-07-26T11:11:50.089544+0000","last_fullsized":"2024-07-26T11:11:50.089544+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:17:56.418638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.383567+0000","last_change":"2024-07-26T11:10:36.000156+0000","last_active":"2024-07-26T11:11:35.383567+0000","last_peered":"2024-07-26T11:11:35.383567+0000","last_clean":"2024-07-26T11:11:35.383567+0000","last_became_active":"2024-07-26T11:10:35.999600+0000","last_became_peered":"2024-07-26T11:10:35.999600+0000","last_unstale":"2024-07-26T11:11:35.383567+0000","last_undegraded":"2024-07-26T11:11:35.383567+0000","last_fullsized":"2024-07-26T11:11:35.383567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:35:48.504288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.821261+0000","last_change":"2024-07-26T11:10:22.298152+0000","last_active":"2024-07-26T11:13:33.821261+0000","last_peered":"2024-07-26T11:13:33.821261+0000","last_clean":"2024-07-26T11:13:33.821261+0000","last_became_active":"2024-07-26T11:10:22.297739+0000","last_became_peered":"2024-07-26T11:10:22.297739+0000","last_unstale":"2024-07-26T11:13:33.821261+0000","last_undegraded":"2024-07-26T11:13:33.821261+0000","last_fullsized":"2024-07-26T11:13:33.821261+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:48.160183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:47.040493+0000","last_change":"2024-07-26T11:10:18.057756+0000","last_active":"2024-07-26T11:11:47.040493+0000","last_peered":"2024-07-26T11:11:47.040493+0000","last_clean":"2024-07-26T11:11:47.040493+0000","last_became_active":"2024-07-26T11:10:18.057464+0000","last_became_peered":"2024-07-26T11:10:18.057464+0000","last_unstale":"2024-07-26T11:11:47.040493+0000","last_undegraded":"2024-07-26T11:11:47.040493+0000","last_fullsized":"2024-07-26T11:11:47.040493+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:39:11.523643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.407148+0000","last_change":"2024-07-26T11:10:36.000617+0000","last_active":"2024-07-26T11:11:35.407148+0000","last_peered":"2024-07-26T11:11:35.407148+0000","last_clean":"2024-07-26T11:11:35.407148+0000","last_became_active":"2024-07-26T11:10:36.000151+0000","last_became_peered":"2024-07-26T11:10:36.000151+0000","last_unstale":"2024-07-26T11:11:35.407148+0000","last_undegraded":"2024-07-26T11:11:35.407148+0000","last_fullsized":"2024-07-26T11:11:35.407148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:36.942215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.892406+0000","last_change":"2024-07-26T11:10:22.336452+0000","last_active":"2024-07-26T11:13:33.892406+0000","last_peered":"2024-07-26T11:13:33.892406+0000","last_clean":"2024-07-26T11:13:33.892406+0000","last_became_active":"2024-07-26T11:10:22.336006+0000","last_became_peered":"2024-07-26T11:10:22.336006+0000","last_unstale":"2024-07-26T11:13:33.892406+0000","last_undegraded":"2024-07-26T11:13:33.892406+0000","last_fullsized":"2024-07-26T11:13:33.892406+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:08:27.375730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.096168+0000","last_change":"2024-07-26T11:10:18.823017+0000","last_active":"2024-07-26T11:11:51.096168+0000","last_peered":"2024-07-26T11:11:51.096168+0000","last_clean":"2024-07-26T11:11:51.096168+0000","last_became_active":"2024-07-26T11:10:18.822559+0000","last_became_peered":"2024-07-26T11:10:18.822559+0000","last_unstale":"2024-07-26T11:11:51.096168+0000","last_undegraded":"2024-07-26T11:11:51.096168+0000","last_fullsized":"2024-07-26T11:11:51.096168+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:01.486263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.155764+0000","last_change":"2024-07-26T11:10:35.969698+0000","last_active":"2024-07-26T11:11:35.155764+0000","last_peered":"2024-07-26T11:11:35.155764+0000","last_clean":"2024-07-26T11:11:35.155764+0000","last_became_active":"2024-07-26T11:10:35.969240+0000","last_became_peered":"2024-07-26T11:10:35.969240+0000","last_unstale":"2024-07-26T11:11:35.155764+0000","last_undegraded":"2024-07-26T11:11:35.155764+0000","last_fullsized":"2024-07-26T11:11:35.155764+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:25.210862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875247+0000","last_change":"2024-07-26T11:10:22.314772+0000","last_active":"2024-07-26T11:13:33.875247+0000","last_peered":"2024-07-26T11:13:33.875247+0000","last_clean":"2024-07-26T11:13:33.875247+0000","last_became_active":"2024-07-26T11:10:22.313786+0000","last_became_peered":"2024-07-26T11:10:22.313786+0000","last_unstale":"2024-07-26T11:13:33.875247+0000","last_undegraded":"2024-07-26T11:13:33.875247+0000","last_fullsized":"2024-07-26T11:13:33.875247+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:19.610695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'17","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:31.518589+0000","last_change":"2024-07-26T11:10:18.049408+0000","last_active":"2024-07-26T11:12:31.518589+0000","last_peered":"2024-07-26T11:12:31.518589+0000","last_clean":"2024-07-26T11:12:31.518589+0000","last_became_active":"2024-07-26T11:10:18.048573+0000","last_became_peered":"2024-07-26T11:10:18.048573+0000","last_unstale":"2024-07-26T11:12:31.518589+0000","last_undegraded":"2024-07-26T11:12:31.518589+0000","last_fullsized":"2024-07-26T11:12:31.518589+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:26.408260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.147395+0000","last_change":"2024-07-26T11:10:35.992067+0000","last_active":"2024-07-26T11:11:36.147395+0000","last_peered":"2024-07-26T11:11:36.147395+0000","last_clean":"2024-07-26T11:11:36.147395+0000","last_became_active":"2024-07-26T11:10:35.989789+0000","last_became_peered":"2024-07-26T11:10:35.989789+0000","last_unstale":"2024-07-26T11:11:36.147395+0000","last_undegraded":"2024-07-26T11:11:36.147395+0000","last_fullsized":"2024-07-26T11:11:36.147395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:08:08.283592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.886537+0000","last_change":"2024-07-26T11:10:22.323512+0000","last_active":"2024-07-26T11:13:33.886537+0000","last_peered":"2024-07-26T11:13:33.886537+0000","last_clean":"2024-07-26T11:13:33.886537+0000","last_became_active":"2024-07-26T11:10:22.323126+0000","last_became_peered":"2024-07-26T11:10:22.323126+0000","last_unstale":"2024-07-26T11:13:33.886537+0000","last_undegraded":"2024-07-26T11:13:33.886537+0000","last_fullsized":"2024-07-26T11:13:33.886537+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:27:15.379345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.093631+0000","last_change":"2024-07-26T11:10:18.052573+0000","last_active":"2024-07-26T11:11:51.093631+0000","last_peered":"2024-07-26T11:11:51.093631+0000","last_clean":"2024-07-26T11:11:51.093631+0000","last_became_active":"2024-07-26T11:10:18.052302+0000","last_became_peered":"2024-07-26T11:10:18.052302+0000","last_unstale":"2024-07-26T11:11:51.093631+0000","last_undegraded":"2024-07-26T11:11:51.093631+0000","last_fullsized":"2024-07-26T11:11:51.093631+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:32.260966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162413+0000","last_change":"2024-07-26T11:10:35.994347+0000","last_active":"2024-07-26T11:11:35.162413+0000","last_peered":"2024-07-26T11:11:35.162413+0000","last_clean":"2024-07-26T11:11:35.162413+0000","last_became_active":"2024-07-26T11:10:35.993795+0000","last_became_peered":"2024-07-26T11:10:35.993795+0000","last_unstale":"2024-07-26T11:11:35.162413+0000","last_undegraded":"2024-07-26T11:11:35.162413+0000","last_fullsized":"2024-07-26T11:11:35.162413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:35.399601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.881668+0000","last_change":"2024-07-26T11:10:22.318109+0000","last_active":"2024-07-26T11:13:33.881668+0000","last_peered":"2024-07-26T11:13:33.881668+0000","last_clean":"2024-07-26T11:13:33.881668+0000","last_became_active":"2024-07-26T11:10:22.317674+0000","last_became_peered":"2024-07-26T11:10:22.317674+0000","last_unstale":"2024-07-26T11:13:33.881668+0000","last_undegraded":"2024-07-26T11:13:33.881668+0000","last_fullsized":"2024-07-26T11:13:33.881668+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:29.464772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:49.073328+0000","last_change":"2024-07-26T11:10:18.060073+0000","last_active":"2024-07-26T11:11:49.073328+0000","last_peered":"2024-07-26T11:11:49.073328+0000","last_clean":"2024-07-26T11:11:49.073328+0000","last_became_active":"2024-07-26T11:10:18.059381+0000","last_became_peered":"2024-07-26T11:10:18.059381+0000","last_unstale":"2024-07-26T11:11:49.073328+0000","last_undegraded":"2024-07-26T11:11:49.073328+0000","last_fullsized":"2024-07-26T11:11:49.073328+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:04:21.184759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.951752+0000","last_change":"2024-07-26T11:11:35.951234+0000","last_active":"2024-07-26T11:11:35.951752+0000","last_peered":"2024-07-26T11:11:35.951752+0000","last_clean":"2024-07-26T11:11:35.951752+0000","last_became_active":"2024-07-26T11:11:35.950821+0000","last_became_peered":"2024-07-26T11:11:35.950821+0000","last_unstale":"2024-07-26T11:11:35.951752+0000","last_undegraded":"2024-07-26T11:11:35.951752+0000","last_fullsized":"2024-07-26T11:11:35.951752+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:44:22.569238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.872883+0000","last_change":"2024-07-26T11:10:22.326866+0000","last_active":"2024-07-26T11:13:33.872883+0000","last_peered":"2024-07-26T11:13:33.872883+0000","last_clean":"2024-07-26T11:13:33.872883+0000","last_became_active":"2024-07-26T11:10:22.326417+0000","last_became_peered":"2024-07-26T11:10:22.326417+0000","last_unstale":"2024-07-26T11:13:33.872883+0000","last_undegraded":"2024-07-26T11:13:33.872883+0000","last_fullsized":"2024-07-26T11:13:33.872883+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:49:02.871062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:56.347025+0000","last_change":"2024-07-26T11:10:18.044077+0000","last_active":"2024-07-26T11:13:56.347025+0000","last_peered":"2024-07-26T11:13:56.347025+0000","last_clean":"2024-07-26T11:13:56.347025+0000","last_became_active":"2024-07-26T11:10:18.043660+0000","last_became_peered":"2024-07-26T11:10:18.043660+0000","last_unstale":"2024-07-26T11:13:56.347025+0000","last_undegraded":"2024-07-26T11:13:56.347025+0000","last_fullsized":"2024-07-26T11:13:56.347025+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:46:54.901082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":128,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.641747+0000","last_change":"2024-07-26T11:10:35.984778+0000","last_active":"2024-07-26T11:11:35.641747+0000","last_peered":"2024-07-26T11:11:35.641747+0000","last_clean":"2024-07-26T11:11:35.641747+0000","last_became_active":"2024-07-26T11:10:35.984365+0000","last_became_peered":"2024-07-26T11:10:35.984365+0000","last_unstale":"2024-07-26T11:11:35.641747+0000","last_undegraded":"2024-07-26T11:11:35.641747+0000","last_fullsized":"2024-07-26T11:11:35.641747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:33.174121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.859140+0000","last_change":"2024-07-26T11:10:34.976075+0000","last_active":"2024-07-26T11:13:33.859140+0000","last_peered":"2024-07-26T11:13:33.859140+0000","last_clean":"2024-07-26T11:13:33.859140+0000","last_became_active":"2024-07-26T11:10:34.973799+0000","last_became_peered":"2024-07-26T11:10:34.973799+0000","last_unstale":"2024-07-26T11:13:33.859140+0000","last_undegraded":"2024-07-26T11:13:33.859140+0000","last_fullsized":"2024-07-26T11:13:33.859140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:03:01.830655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'6","reported_seq":175,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:55.419351+0000","last_change":"2024-07-26T11:10:18.056473+0000","last_active":"2024-07-26T11:13:55.419351+0000","last_peered":"2024-07-26T11:13:55.419351+0000","last_clean":"2024-07-26T11:13:55.419351+0000","last_became_active":"2024-07-26T11:10:18.056038+0000","last_became_peered":"2024-07-26T11:10:18.056038+0000","last_unstale":"2024-07-26T11:13:55.419351+0000","last_undegraded":"2024-07-26T11:13:55.419351+0000","last_fullsized":"2024-07-26T11:13:55.419351+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:04.501018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":127,"num_read_kb":127,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.642403+0000","last_change":"2024-07-26T11:10:35.991741+0000","last_active":"2024-07-26T11:11:35.642403+0000","last_peered":"2024-07-26T11:11:35.642403+0000","last_clean":"2024-07-26T11:11:35.642403+0000","last_became_active":"2024-07-26T11:10:35.990995+0000","last_became_peered":"2024-07-26T11:10:35.990995+0000","last_unstale":"2024-07-26T11:11:35.642403+0000","last_undegraded":"2024-07-26T11:11:35.642403+0000","last_fullsized":"2024-07-26T11:11:35.642403+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:59:34.056646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.884240+0000","last_change":"2024-07-26T11:10:22.305053+0000","last_active":"2024-07-26T11:13:33.884240+0000","last_peered":"2024-07-26T11:13:33.884240+0000","last_clean":"2024-07-26T11:13:33.884240+0000","last_became_active":"2024-07-26T11:10:22.304035+0000","last_became_peered":"2024-07-26T11:10:22.304035+0000","last_unstale":"2024-07-26T11:13:33.884240+0000","last_undegraded":"2024-07-26T11:13:33.884240+0000","last_fullsized":"2024-07-26T11:13:33.884240+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:48.590799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.082116+0000","last_change":"2024-07-26T11:10:18.033581+0000","last_active":"2024-07-26T11:11:50.082116+0000","last_peered":"2024-07-26T11:11:50.082116+0000","last_clean":"2024-07-26T11:11:50.082116+0000","last_became_active":"2024-07-26T11:10:18.033038+0000","last_became_peered":"2024-07-26T11:10:18.033038+0000","last_unstale":"2024-07-26T11:11:50.082116+0000","last_undegraded":"2024-07-26T11:11:50.082116+0000","last_fullsized":"2024-07-26T11:11:50.082116+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:19:52.523006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.644202+0000","last_change":"2024-07-26T11:11:35.643683+0000","last_active":"2024-07-26T11:11:35.644202+0000","last_peered":"2024-07-26T11:11:35.644202+0000","last_clean":"2024-07-26T11:11:35.644202+0000","last_became_active":"2024-07-26T11:11:35.643109+0000","last_became_peered":"2024-07-26T11:11:35.643109+0000","last_unstale":"2024-07-26T11:11:35.644202+0000","last_undegraded":"2024-07-26T11:11:35.644202+0000","last_fullsized":"2024-07-26T11:11:35.644202+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:39:28.190726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.878406+0000","last_change":"2024-07-26T11:10:34.977690+0000","last_active":"2024-07-26T11:13:33.878406+0000","last_peered":"2024-07-26T11:13:33.878406+0000","last_clean":"2024-07-26T11:13:33.878406+0000","last_became_active":"2024-07-26T11:10:34.976938+0000","last_became_peered":"2024-07-26T11:10:34.976938+0000","last_unstale":"2024-07-26T11:13:33.878406+0000","last_undegraded":"2024-07-26T11:13:33.878406+0000","last_fullsized":"2024-07-26T11:13:33.878406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:26:21.510469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.094135+0000","last_change":"2024-07-26T11:10:18.045363+0000","last_active":"2024-07-26T11:11:51.094135+0000","last_peered":"2024-07-26T11:11:51.094135+0000","last_clean":"2024-07-26T11:11:51.094135+0000","last_became_active":"2024-07-26T11:10:18.044810+0000","last_became_peered":"2024-07-26T11:10:18.044810+0000","last_unstale":"2024-07-26T11:11:51.094135+0000","last_undegraded":"2024-07-26T11:11:51.094135+0000","last_fullsized":"2024-07-26T11:11:51.094135+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:34.507172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":119,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.161450+0000","last_change":"2024-07-26T11:08:48.936784+0000","last_active":"2024-07-26T11:11:35.161450+0000","last_peered":"2024-07-26T11:11:35.161450+0000","last_clean":"2024-07-26T11:11:35.161450+0000","last_became_active":"2024-07-26T11:08:48.931118+0000","last_became_peered":"2024-07-26T11:08:48.931118+0000","last_unstale":"2024-07-26T11:11:35.161450+0000","last_undegraded":"2024-07-26T11:11:35.161450+0000","last_fullsized":"2024-07-26T11:11:35.161450+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:05:01.918569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:05:01.918569+0000","last_clean_scrub_stamp":"2024-07-26T11:05:01.918569+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:57.128424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:04.061187+0000","last_change":"2024-07-26T11:11:35.385528+0000","last_active":"2024-07-26T11:12:04.061187+0000","last_peered":"2024-07-26T11:12:04.061187+0000","last_clean":"2024-07-26T11:12:04.061187+0000","last_became_active":"2024-07-26T11:11:35.385042+0000","last_became_peered":"2024-07-26T11:11:35.385042+0000","last_unstale":"2024-07-26T11:12:04.061187+0000","last_undegraded":"2024-07-26T11:12:04.061187+0000","last_fullsized":"2024-07-26T11:12:04.061187+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:52:30.343736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875091+0000","last_change":"2024-07-26T11:10:22.302715+0000","last_active":"2024-07-26T11:13:33.875091+0000","last_peered":"2024-07-26T11:13:33.875091+0000","last_clean":"2024-07-26T11:13:33.875091+0000","last_became_active":"2024-07-26T11:10:22.302284+0000","last_became_peered":"2024-07-26T11:10:22.302284+0000","last_unstale":"2024-07-26T11:13:33.875091+0000","last_undegraded":"2024-07-26T11:13:33.875091+0000","last_fullsized":"2024-07-26T11:13:33.875091+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:13:05.555439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":178,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:58.302664+0000","last_change":"2024-07-26T11:10:18.823160+0000","last_active":"2024-07-26T11:13:58.302664+0000","last_peered":"2024-07-26T11:13:58.302664+0000","last_clean":"2024-07-26T11:13:58.302664+0000","last_became_active":"2024-07-26T11:10:18.822652+0000","last_became_peered":"2024-07-26T11:10:18.822652+0000","last_unstale":"2024-07-26T11:13:58.302664+0000","last_undegraded":"2024-07-26T11:13:58.302664+0000","last_fullsized":"2024-07-26T11:13:58.302664+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:21:06.077784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":131,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.949512+0000","last_change":"2024-07-26T11:10:35.984007+0000","last_active":"2024-07-26T11:11:35.949512+0000","last_peered":"2024-07-26T11:11:35.949512+0000","last_clean":"2024-07-26T11:11:35.949512+0000","last_became_active":"2024-07-26T11:10:35.983352+0000","last_became_peered":"2024-07-26T11:10:35.983352+0000","last_unstale":"2024-07-26T11:11:35.949512+0000","last_undegraded":"2024-07-26T11:11:35.949512+0000","last_fullsized":"2024-07-26T11:11:35.949512+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:05:21.199041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.867547+0000","last_change":"2024-07-26T11:10:22.320559+0000","last_active":"2024-07-26T11:13:33.867547+0000","last_peered":"2024-07-26T11:13:33.867547+0000","last_clean":"2024-07-26T11:13:33.867547+0000","last_became_active":"2024-07-26T11:10:22.320192+0000","last_became_peered":"2024-07-26T11:10:22.320192+0000","last_unstale":"2024-07-26T11:13:33.867547+0000","last_undegraded":"2024-07-26T11:13:33.867547+0000","last_fullsized":"2024-07-26T11:13:33.867547+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:48:51.774418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.096632+0000","last_change":"2024-07-26T11:10:18.044108+0000","last_active":"2024-07-26T11:11:51.096632+0000","last_peered":"2024-07-26T11:11:51.096632+0000","last_clean":"2024-07-26T11:11:51.096632+0000","last_became_active":"2024-07-26T11:10:18.043712+0000","last_became_peered":"2024-07-26T11:10:18.043712+0000","last_unstale":"2024-07-26T11:11:51.096632+0000","last_undegraded":"2024-07-26T11:11:51.096632+0000","last_fullsized":"2024-07-26T11:11:51.096632+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:05:34.382520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.185926+0000","last_change":"2024-07-26T11:10:35.999071+0000","last_active":"2024-07-26T11:11:35.185926+0000","last_peered":"2024-07-26T11:11:35.185926+0000","last_clean":"2024-07-26T11:11:35.185926+0000","last_became_active":"2024-07-26T11:10:35.997179+0000","last_became_peered":"2024-07-26T11:10:35.997179+0000","last_unstale":"2024-07-26T11:11:35.185926+0000","last_undegraded":"2024-07-26T11:11:35.185926+0000","last_fullsized":"2024-07-26T11:11:35.185926+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:36:45.632086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.871203+0000","last_change":"2024-07-26T11:10:22.300356+0000","last_active":"2024-07-26T11:13:33.871203+0000","last_peered":"2024-07-26T11:13:33.871203+0000","last_clean":"2024-07-26T11:13:33.871203+0000","last_became_active":"2024-07-26T11:10:22.299949+0000","last_became_peered":"2024-07-26T11:10:22.299949+0000","last_unstale":"2024-07-26T11:13:33.871203+0000","last_undegraded":"2024-07-26T11:13:33.871203+0000","last_fullsized":"2024-07-26T11:13:33.871203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:34.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.087457+0000","last_change":"2024-07-26T11:10:18.049965+0000","last_active":"2024-07-26T11:11:50.087457+0000","last_peered":"2024-07-26T11:11:50.087457+0000","last_clean":"2024-07-26T11:11:50.087457+0000","last_became_active":"2024-07-26T11:10:18.049251+0000","last_became_peered":"2024-07-26T11:10:18.049251+0000","last_unstale":"2024-07-26T11:11:50.087457+0000","last_undegraded":"2024-07-26T11:11:50.087457+0000","last_fullsized":"2024-07-26T11:11:50.087457+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:20.996609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158648+0000","last_change":"2024-07-26T11:10:35.967661+0000","last_active":"2024-07-26T11:11:35.158648+0000","last_peered":"2024-07-26T11:11:35.158648+0000","last_clean":"2024-07-26T11:11:35.158648+0000","last_became_active":"2024-07-26T11:10:35.966867+0000","last_became_peered":"2024-07-26T11:10:35.966867+0000","last_unstale":"2024-07-26T11:11:35.158648+0000","last_undegraded":"2024-07-26T11:11:35.158648+0000","last_fullsized":"2024-07-26T11:11:35.158648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:43.951319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.866924+0000","last_change":"2024-07-26T11:10:22.303733+0000","last_active":"2024-07-26T11:13:33.866924+0000","last_peered":"2024-07-26T11:13:33.866924+0000","last_clean":"2024-07-26T11:13:33.866924+0000","last_became_active":"2024-07-26T11:10:22.303322+0000","last_became_peered":"2024-07-26T11:10:22.303322+0000","last_unstale":"2024-07-26T11:13:33.866924+0000","last_undegraded":"2024-07-26T11:13:33.866924+0000","last_fullsized":"2024-07-26T11:13:33.866924+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:34:34.962492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.096226+0000","last_change":"2024-07-26T11:10:18.032539+0000","last_active":"2024-07-26T11:11:51.096226+0000","last_peered":"2024-07-26T11:11:51.096226+0000","last_clean":"2024-07-26T11:11:51.096226+0000","last_became_active":"2024-07-26T11:10:18.032059+0000","last_became_peered":"2024-07-26T11:10:18.032059+0000","last_unstale":"2024-07-26T11:11:51.096226+0000","last_undegraded":"2024-07-26T11:11:51.096226+0000","last_fullsized":"2024-07-26T11:11:51.096226+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:54.502339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158655+0000","last_change":"2024-07-26T11:10:35.977473+0000","last_active":"2024-07-26T11:11:35.158655+0000","last_peered":"2024-07-26T11:11:35.158655+0000","last_clean":"2024-07-26T11:11:35.158655+0000","last_became_active":"2024-07-26T11:10:35.976830+0000","last_became_peered":"2024-07-26T11:10:35.976830+0000","last_unstale":"2024-07-26T11:11:35.158655+0000","last_undegraded":"2024-07-26T11:11:35.158655+0000","last_fullsized":"2024-07-26T11:11:35.158655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:46:13.009246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.870719+0000","last_change":"2024-07-26T11:10:22.304358+0000","last_active":"2024-07-26T11:13:33.870719+0000","last_peered":"2024-07-26T11:13:33.870719+0000","last_clean":"2024-07-26T11:13:33.870719+0000","last_became_active":"2024-07-26T11:10:22.303846+0000","last_became_peered":"2024-07-26T11:10:22.303846+0000","last_unstale":"2024-07-26T11:13:33.870719+0000","last_undegraded":"2024-07-26T11:13:33.870719+0000","last_fullsized":"2024-07-26T11:13:33.870719+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:27:28.556481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":60,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.093876+0000","last_change":"2024-07-26T11:10:18.057678+0000","last_active":"2024-07-26T11:11:51.093876+0000","last_peered":"2024-07-26T11:11:51.093876+0000","last_clean":"2024-07-26T11:11:51.093876+0000","last_became_active":"2024-07-26T11:10:18.057282+0000","last_became_peered":"2024-07-26T11:10:18.057282+0000","last_unstale":"2024-07-26T11:11:51.093876+0000","last_undegraded":"2024-07-26T11:11:51.093876+0000","last_fullsized":"2024-07-26T11:11:51.093876+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:08:35.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.642357+0000","last_change":"2024-07-26T11:10:36.000329+0000","last_active":"2024-07-26T11:11:35.642357+0000","last_peered":"2024-07-26T11:11:35.642357+0000","last_clean":"2024-07-26T11:11:35.642357+0000","last_became_active":"2024-07-26T11:10:35.999713+0000","last_became_peered":"2024-07-26T11:10:35.999713+0000","last_unstale":"2024-07-26T11:11:35.642357+0000","last_undegraded":"2024-07-26T11:11:35.642357+0000","last_fullsized":"2024-07-26T11:11:35.642357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:48.137237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.866698+0000","last_change":"2024-07-26T11:10:22.301526+0000","last_active":"2024-07-26T11:13:33.866698+0000","last_peered":"2024-07-26T11:13:33.866698+0000","last_clean":"2024-07-26T11:13:33.866698+0000","last_became_active":"2024-07-26T11:10:22.301098+0000","last_became_peered":"2024-07-26T11:10:22.301098+0000","last_unstale":"2024-07-26T11:13:33.866698+0000","last_undegraded":"2024-07-26T11:13:33.866698+0000","last_fullsized":"2024-07-26T11:13:33.866698+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:01:48.118315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'7","reported_seq":182,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:58.427637+0000","last_change":"2024-07-26T11:10:18.035922+0000","last_active":"2024-07-26T11:13:58.427637+0000","last_peered":"2024-07-26T11:13:58.427637+0000","last_clean":"2024-07-26T11:13:58.427637+0000","last_became_active":"2024-07-26T11:10:18.035184+0000","last_became_peered":"2024-07-26T11:10:18.035184+0000","last_unstale":"2024-07-26T11:13:58.427637+0000","last_undegraded":"2024-07-26T11:13:58.427637+0000","last_fullsized":"2024-07-26T11:13:58.427637+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:13:10.371105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":134,"num_read_kb":5,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.384102+0000","last_change":"2024-07-26T11:10:35.998791+0000","last_active":"2024-07-26T11:11:35.384102+0000","last_peered":"2024-07-26T11:11:35.384102+0000","last_clean":"2024-07-26T11:11:35.384102+0000","last_became_active":"2024-07-26T11:10:35.997163+0000","last_became_peered":"2024-07-26T11:10:35.997163+0000","last_unstale":"2024-07-26T11:11:35.384102+0000","last_undegraded":"2024-07-26T11:11:35.384102+0000","last_fullsized":"2024-07-26T11:11:35.384102+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:28:50.167294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889375+0000","last_change":"2024-07-26T11:10:22.314219+0000","last_active":"2024-07-26T11:13:33.889375+0000","last_peered":"2024-07-26T11:13:33.889375+0000","last_clean":"2024-07-26T11:13:33.889375+0000","last_became_active":"2024-07-26T11:10:22.313811+0000","last_became_peered":"2024-07-26T11:10:22.313811+0000","last_unstale":"2024-07-26T11:13:33.889375+0000","last_undegraded":"2024-07-26T11:13:33.889375+0000","last_fullsized":"2024-07-26T11:13:33.889375+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:55:47.452798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.093492+0000","last_change":"2024-07-26T11:10:18.047818+0000","last_active":"2024-07-26T11:11:51.093492+0000","last_peered":"2024-07-26T11:11:51.093492+0000","last_clean":"2024-07-26T11:11:51.093492+0000","last_became_active":"2024-07-26T11:10:18.047159+0000","last_became_peered":"2024-07-26T11:10:18.047159+0000","last_unstale":"2024-07-26T11:11:51.093492+0000","last_undegraded":"2024-07-26T11:11:51.093492+0000","last_fullsized":"2024-07-26T11:11:51.093492+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:37.528512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.162781+0000","last_change":"2024-07-26T11:10:35.996468+0000","last_active":"2024-07-26T11:11:35.162781+0000","last_peered":"2024-07-26T11:11:35.162781+0000","last_clean":"2024-07-26T11:11:35.162781+0000","last_became_active":"2024-07-26T11:10:35.996030+0000","last_became_peered":"2024-07-26T11:10:35.996030+0000","last_unstale":"2024-07-26T11:11:35.162781+0000","last_undegraded":"2024-07-26T11:11:35.162781+0000","last_fullsized":"2024-07-26T11:11:35.162781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:20:14.743690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.881839+0000","last_change":"2024-07-26T11:10:22.318375+0000","last_active":"2024-07-26T11:13:33.881839+0000","last_peered":"2024-07-26T11:13:33.881839+0000","last_clean":"2024-07-26T11:13:33.881839+0000","last_became_active":"2024-07-26T11:10:22.317965+0000","last_became_peered":"2024-07-26T11:10:22.317965+0000","last_unstale":"2024-07-26T11:13:33.881839+0000","last_undegraded":"2024-07-26T11:13:33.881839+0000","last_fullsized":"2024-07-26T11:13:33.881839+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:36:43.504045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.096598+0000","last_change":"2024-07-26T11:10:18.047957+0000","last_active":"2024-07-26T11:11:51.096598+0000","last_peered":"2024-07-26T11:11:51.096598+0000","last_clean":"2024-07-26T11:11:51.096598+0000","last_became_active":"2024-07-26T11:10:18.047495+0000","last_became_peered":"2024-07-26T11:10:18.047495+0000","last_unstale":"2024-07-26T11:11:51.096598+0000","last_undegraded":"2024-07-26T11:11:51.096598+0000","last_fullsized":"2024-07-26T11:11:51.096598+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:11.238295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.158767+0000","last_change":"2024-07-26T11:10:35.976946+0000","last_active":"2024-07-26T11:11:35.158767+0000","last_peered":"2024-07-26T11:11:35.158767+0000","last_clean":"2024-07-26T11:11:35.158767+0000","last_became_active":"2024-07-26T11:10:35.976474+0000","last_became_peered":"2024-07-26T11:10:35.976474+0000","last_unstale":"2024-07-26T11:11:35.158767+0000","last_undegraded":"2024-07-26T11:11:35.158767+0000","last_fullsized":"2024-07-26T11:11:35.158767+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:50:51.240903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.859003+0000","last_change":"2024-07-26T11:10:22.302473+0000","last_active":"2024-07-26T11:13:33.859003+0000","last_peered":"2024-07-26T11:13:33.859003+0000","last_clean":"2024-07-26T11:13:33.859003+0000","last_became_active":"2024-07-26T11:10:22.302096+0000","last_became_peered":"2024-07-26T11:10:22.302096+0000","last_unstale":"2024-07-26T11:13:33.859003+0000","last_undegraded":"2024-07-26T11:13:33.859003+0000","last_fullsized":"2024-07-26T11:13:33.859003+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:19.215301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'40","reported_seq":365,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:57.300044+0000","last_change":"2024-07-26T11:10:34.973502+0000","last_active":"2024-07-26T11:13:57.300044+0000","last_peered":"2024-07-26T11:13:57.300044+0000","last_clean":"2024-07-26T11:13:57.300044+0000","last_became_active":"2024-07-26T11:10:34.964155+0000","last_became_peered":"2024-07-26T11:10:34.964155+0000","last_unstale":"2024-07-26T11:13:57.300044+0000","last_undegraded":"2024-07-26T11:13:57.300044+0000","last_fullsized":"2024-07-26T11:13:57.300044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:23:06.652746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66769,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":266,"num_read_kb":199,"num_write":42,"num_write_kb":84,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.382965+0000","last_change":"2024-07-26T11:10:35.981439+0000","last_active":"2024-07-26T11:11:35.382965+0000","last_peered":"2024-07-26T11:11:35.382965+0000","last_clean":"2024-07-26T11:11:35.382965+0000","last_became_active":"2024-07-26T11:10:35.980878+0000","last_became_peered":"2024-07-26T11:10:35.980878+0000","last_unstale":"2024-07-26T11:11:35.382965+0000","last_undegraded":"2024-07-26T11:11:35.382965+0000","last_fullsized":"2024-07-26T11:11:35.382965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:31.719845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.882512+0000","last_change":"2024-07-26T11:10:34.974292+0000","last_active":"2024-07-26T11:13:33.882512+0000","last_peered":"2024-07-26T11:13:33.882512+0000","last_clean":"2024-07-26T11:13:33.882512+0000","last_became_active":"2024-07-26T11:10:34.973820+0000","last_became_peered":"2024-07-26T11:10:34.973820+0000","last_unstale":"2024-07-26T11:13:33.882512+0000","last_undegraded":"2024-07-26T11:13:33.882512+0000","last_fullsized":"2024-07-26T11:13:33.882512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:35:37.629313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.096325+0000","last_change":"2024-07-26T11:10:18.055248+0000","last_active":"2024-07-26T11:11:51.096325+0000","last_peered":"2024-07-26T11:11:51.096325+0000","last_clean":"2024-07-26T11:11:51.096325+0000","last_became_active":"2024-07-26T11:10:18.054882+0000","last_became_peered":"2024-07-26T11:10:18.054882+0000","last_unstale":"2024-07-26T11:11:51.096325+0000","last_undegraded":"2024-07-26T11:11:51.096325+0000","last_fullsized":"2024-07-26T11:11:51.096325+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:16.390058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.407395+0000","last_change":"2024-07-26T11:10:35.985628+0000","last_active":"2024-07-26T11:11:35.407395+0000","last_peered":"2024-07-26T11:11:35.407395+0000","last_clean":"2024-07-26T11:11:35.407395+0000","last_became_active":"2024-07-26T11:10:35.985156+0000","last_became_peered":"2024-07-26T11:10:35.985156+0000","last_unstale":"2024-07-26T11:11:35.407395+0000","last_undegraded":"2024-07-26T11:11:35.407395+0000","last_fullsized":"2024-07-26T11:11:35.407395+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:29:18.891825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.891553+0000","last_change":"2024-07-26T11:10:34.973261+0000","last_active":"2024-07-26T11:13:33.891553+0000","last_peered":"2024-07-26T11:13:33.891553+0000","last_clean":"2024-07-26T11:13:33.891553+0000","last_became_active":"2024-07-26T11:10:34.971592+0000","last_became_peered":"2024-07-26T11:10:34.971592+0000","last_unstale":"2024-07-26T11:13:33.891553+0000","last_undegraded":"2024-07-26T11:13:33.891553+0000","last_fullsized":"2024-07-26T11:13:33.891553+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:58:13.517884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.083464+0000","last_change":"2024-07-26T11:10:34.962248+0000","last_active":"2024-07-26T11:11:50.083464+0000","last_peered":"2024-07-26T11:11:50.083464+0000","last_clean":"2024-07-26T11:11:50.083464+0000","last_became_active":"2024-07-26T11:10:34.961683+0000","last_became_peered":"2024-07-26T11:10:34.961683+0000","last_unstale":"2024-07-26T11:11:50.083464+0000","last_undegraded":"2024-07-26T11:11:50.083464+0000","last_fullsized":"2024-07-26T11:11:50.083464+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:17:08.308226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.381997+0000","last_change":"2024-07-26T11:10:35.995760+0000","last_active":"2024-07-26T11:11:35.381997+0000","last_peered":"2024-07-26T11:11:35.381997+0000","last_clean":"2024-07-26T11:11:35.381997+0000","last_became_active":"2024-07-26T11:10:35.995328+0000","last_became_peered":"2024-07-26T11:10:35.995328+0000","last_unstale":"2024-07-26T11:11:35.381997+0000","last_undegraded":"2024-07-26T11:11:35.381997+0000","last_fullsized":"2024-07-26T11:11:35.381997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:40:33.093315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.876940+0000","last_change":"2024-07-26T11:10:22.316151+0000","last_active":"2024-07-26T11:13:33.876940+0000","last_peered":"2024-07-26T11:13:33.876940+0000","last_clean":"2024-07-26T11:13:33.876940+0000","last_became_active":"2024-07-26T11:10:22.315722+0000","last_became_peered":"2024-07-26T11:10:22.315722+0000","last_unstale":"2024-07-26T11:13:33.876940+0000","last_undegraded":"2024-07-26T11:13:33.876940+0000","last_fullsized":"2024-07-26T11:13:33.876940+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:12:58.342637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":300,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:57.300043+0000","last_change":"2024-07-26T11:10:18.046327+0000","last_active":"2024-07-26T11:13:57.300043+0000","last_peered":"2024-07-26T11:13:57.300043+0000","last_clean":"2024-07-26T11:13:57.300043+0000","last_became_active":"2024-07-26T11:10:18.045913+0000","last_became_peered":"2024-07-26T11:10:18.045913+0000","last_unstale":"2024-07-26T11:13:57.300043+0000","last_undegraded":"2024-07-26T11:13:57.300043+0000","last_fullsized":"2024-07-26T11:13:57.300043+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:58.889046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":254,"num_read_kb":254,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:36.147807+0000","last_change":"2024-07-26T11:10:35.983264+0000","last_active":"2024-07-26T11:11:36.147807+0000","last_peered":"2024-07-26T11:11:36.147807+0000","last_clean":"2024-07-26T11:11:36.147807+0000","last_became_active":"2024-07-26T11:10:35.982210+0000","last_became_peered":"2024-07-26T11:10:35.982210+0000","last_unstale":"2024-07-26T11:11:36.147807+0000","last_undegraded":"2024-07-26T11:11:36.147807+0000","last_fullsized":"2024-07-26T11:11:36.147807+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:37:20.272725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.185837+0000","last_change":"2024-07-26T11:10:35.985335+0000","last_active":"2024-07-26T11:11:35.185837+0000","last_peered":"2024-07-26T11:11:35.185837+0000","last_clean":"2024-07-26T11:11:35.185837+0000","last_became_active":"2024-07-26T11:10:35.983667+0000","last_became_peered":"2024-07-26T11:10:35.983667+0000","last_unstale":"2024-07-26T11:11:35.185837+0000","last_undegraded":"2024-07-26T11:11:35.185837+0000","last_fullsized":"2024-07-26T11:11:35.185837+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:00:17.744806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.879169+0000","last_change":"2024-07-26T11:10:22.333494+0000","last_active":"2024-07-26T11:13:33.879169+0000","last_peered":"2024-07-26T11:13:33.879169+0000","last_clean":"2024-07-26T11:13:33.879169+0000","last_became_active":"2024-07-26T11:10:22.332950+0000","last_became_peered":"2024-07-26T11:10:22.332950+0000","last_unstale":"2024-07-26T11:13:33.879169+0000","last_undegraded":"2024-07-26T11:13:33.879169+0000","last_fullsized":"2024-07-26T11:13:33.879169+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:58:38.340857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:51.099680+0000","last_change":"2024-07-26T11:10:18.034057+0000","last_active":"2024-07-26T11:11:51.099680+0000","last_peered":"2024-07-26T11:11:51.099680+0000","last_clean":"2024-07-26T11:11:51.099680+0000","last_became_active":"2024-07-26T11:10:18.033119+0000","last_became_peered":"2024-07-26T11:10:18.033119+0000","last_unstale":"2024-07-26T11:11:51.099680+0000","last_undegraded":"2024-07-26T11:11:51.099680+0000","last_fullsized":"2024-07-26T11:11:51.099680+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:21.675593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.408887+0000","last_change":"2024-07-26T11:11:35.408377+0000","last_active":"2024-07-26T11:11:35.408887+0000","last_peered":"2024-07-26T11:11:35.408887+0000","last_clean":"2024-07-26T11:11:35.408887+0000","last_became_active":"2024-07-26T11:11:35.407865+0000","last_became_peered":"2024-07-26T11:11:35.407865+0000","last_unstale":"2024-07-26T11:11:35.408887+0000","last_undegraded":"2024-07-26T11:11:35.408887+0000","last_fullsized":"2024-07-26T11:11:35.408887+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:43:02.953395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.886870+0000","last_change":"2024-07-26T11:10:22.323447+0000","last_active":"2024-07-26T11:13:33.886870+0000","last_peered":"2024-07-26T11:13:33.886870+0000","last_clean":"2024-07-26T11:13:33.886870+0000","last_became_active":"2024-07-26T11:10:22.322988+0000","last_became_peered":"2024-07-26T11:10:22.322988+0000","last_unstale":"2024-07-26T11:13:33.886870+0000","last_undegraded":"2024-07-26T11:13:33.886870+0000","last_fullsized":"2024-07-26T11:13:33.886870+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:14.880985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:50.088906+0000","last_change":"2024-07-26T11:10:18.048724+0000","last_active":"2024-07-26T11:11:50.088906+0000","last_peered":"2024-07-26T11:11:50.088906+0000","last_clean":"2024-07-26T11:11:50.088906+0000","last_became_active":"2024-07-26T11:10:18.048257+0000","last_became_peered":"2024-07-26T11:10:18.048257+0000","last_unstale":"2024-07-26T11:11:50.088906+0000","last_undegraded":"2024-07-26T11:11:50.088906+0000","last_fullsized":"2024-07-26T11:11:50.088906+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:02:13.581446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:49.071419+0000","last_change":"2024-07-26T11:10:18.044171+0000","last_active":"2024-07-26T11:11:49.071419+0000","last_peered":"2024-07-26T11:11:49.071419+0000","last_clean":"2024-07-26T11:11:49.071419+0000","last_became_active":"2024-07-26T11:10:18.043682+0000","last_became_peered":"2024-07-26T11:10:18.043682+0000","last_unstale":"2024-07-26T11:11:49.071419+0000","last_undegraded":"2024-07-26T11:11:49.071419+0000","last_fullsized":"2024-07-26T11:11:49.071419+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:46:46.864119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874838+0000","last_change":"2024-07-26T11:10:22.321081+0000","last_active":"2024-07-26T11:13:33.874838+0000","last_peered":"2024-07-26T11:13:33.874838+0000","last_clean":"2024-07-26T11:13:33.874838+0000","last_became_active":"2024-07-26T11:10:22.320656+0000","last_became_peered":"2024-07-26T11:10:22.320656+0000","last_unstale":"2024-07-26T11:13:33.874838+0000","last_undegraded":"2024-07-26T11:13:33.874838+0000","last_fullsized":"2024-07-26T11:13:33.874838+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:17:20.455129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156183+0000","last_change":"2024-07-26T11:10:36.000224+0000","last_active":"2024-07-26T11:11:35.156183+0000","last_peered":"2024-07-26T11:11:35.156183+0000","last_clean":"2024-07-26T11:11:35.156183+0000","last_became_active":"2024-07-26T11:10:35.999733+0000","last_became_peered":"2024-07-26T11:10:35.999733+0000","last_unstale":"2024-07-26T11:11:35.156183+0000","last_undegraded":"2024-07-26T11:11:35.156183+0000","last_fullsized":"2024-07-26T11:11:35.156183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:49.912401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":177,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:57.300691+0000","last_change":"2024-07-26T11:10:18.034585+0000","last_active":"2024-07-26T11:13:57.300691+0000","last_peered":"2024-07-26T11:13:57.300691+0000","last_clean":"2024-07-26T11:13:57.300691+0000","last_became_active":"2024-07-26T11:10:18.034237+0000","last_became_peered":"2024-07-26T11:10:18.034237+0000","last_unstale":"2024-07-26T11:13:57.300691+0000","last_undegraded":"2024-07-26T11:13:57.300691+0000","last_fullsized":"2024-07-26T11:13:57.300691+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:22.670277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":130,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.876424+0000","last_change":"2024-07-26T11:10:22.319282+0000","last_active":"2024-07-26T11:13:33.876424+0000","last_peered":"2024-07-26T11:13:33.876424+0000","last_clean":"2024-07-26T11:13:33.876424+0000","last_became_active":"2024-07-26T11:10:22.318822+0000","last_became_peered":"2024-07-26T11:10:22.318822+0000","last_unstale":"2024-07-26T11:13:33.876424+0000","last_undegraded":"2024-07-26T11:13:33.876424+0000","last_fullsized":"2024-07-26T11:13:33.876424+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:50:57.615839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:11:35.156390+0000","last_change":"2024-07-26T11:10:35.970189+0000","last_active":"2024-07-26T11:11:35.156390+0000","last_peered":"2024-07-26T11:11:35.156390+0000","last_clean":"2024-07-26T11:11:35.156390+0000","last_became_active":"2024-07-26T11:10:35.969574+0000","last_became_peered":"2024-07-26T11:10:35.969574+0000","last_unstale":"2024-07-26T11:11:35.156390+0000","last_undegraded":"2024-07-26T11:11:35.156390+0000","last_fullsized":"2024-07-26T11:11:35.156390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:07.487944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.868112+0000","last_change":"2024-07-26T11:10:22.299152+0000","last_active":"2024-07-26T11:13:33.868112+0000","last_peered":"2024-07-26T11:13:33.868112+0000","last_clean":"2024-07-26T11:13:33.868112+0000","last_became_active":"2024-07-26T11:10:22.298707+0000","last_became_peered":"2024-07-26T11:10:22.298707+0000","last_unstale":"2024-07-26T11:13:33.868112+0000","last_undegraded":"2024-07-26T11:13:33.868112+0000","last_fullsized":"2024-07-26T11:13:33.868112+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:25.161634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.870991+0000","last_change":"2024-07-26T11:10:22.302476+0000","last_active":"2024-07-26T11:13:33.870991+0000","last_peered":"2024-07-26T11:13:33.870991+0000","last_clean":"2024-07-26T11:13:33.870991+0000","last_became_active":"2024-07-26T11:10:22.302091+0000","last_became_peered":"2024-07-26T11:10:22.302091+0000","last_unstale":"2024-07-26T11:13:33.870991+0000","last_undegraded":"2024-07-26T11:13:33.870991+0000","last_fullsized":"2024-07-26T11:13:33.870991+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:06:18.587588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.872438+0000","last_change":"2024-07-26T11:10:22.326583+0000","last_active":"2024-07-26T11:13:33.872438+0000","last_peered":"2024-07-26T11:13:33.872438+0000","last_clean":"2024-07-26T11:13:33.872438+0000","last_became_active":"2024-07-26T11:10:22.326185+0000","last_became_peered":"2024-07-26T11:10:22.326185+0000","last_unstale":"2024-07-26T11:13:33.872438+0000","last_undegraded":"2024-07-26T11:13:33.872438+0000","last_fullsized":"2024-07-26T11:13:33.872438+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:54:00.148964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.894353+0000","last_change":"2024-07-26T11:10:22.329247+0000","last_active":"2024-07-26T11:13:33.894353+0000","last_peered":"2024-07-26T11:13:33.894353+0000","last_clean":"2024-07-26T11:13:33.894353+0000","last_became_active":"2024-07-26T11:10:22.328585+0000","last_became_peered":"2024-07-26T11:10:22.328585+0000","last_unstale":"2024-07-26T11:13:33.894353+0000","last_undegraded":"2024-07-26T11:13:33.894353+0000","last_fullsized":"2024-07-26T11:13:33.894353+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:01:27.195399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.895507+0000","last_change":"2024-07-26T11:10:22.330717+0000","last_active":"2024-07-26T11:13:33.895507+0000","last_peered":"2024-07-26T11:13:33.895507+0000","last_clean":"2024-07-26T11:13:33.895507+0000","last_became_active":"2024-07-26T11:10:22.329576+0000","last_became_peered":"2024-07-26T11:10:22.329576+0000","last_unstale":"2024-07-26T11:13:33.895507+0000","last_undegraded":"2024-07-26T11:13:33.895507+0000","last_fullsized":"2024-07-26T11:13:33.895507+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:21.970629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.867561+0000","last_change":"2024-07-26T11:10:22.307445+0000","last_active":"2024-07-26T11:13:33.867561+0000","last_peered":"2024-07-26T11:13:33.867561+0000","last_clean":"2024-07-26T11:13:33.867561+0000","last_became_active":"2024-07-26T11:10:22.305738+0000","last_became_peered":"2024-07-26T11:10:22.305738+0000","last_unstale":"2024-07-26T11:13:33.867561+0000","last_undegraded":"2024-07-26T11:13:33.867561+0000","last_fullsized":"2024-07-26T11:13:33.867561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:09:08.125209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874453+0000","last_change":"2024-07-26T11:10:22.298392+0000","last_active":"2024-07-26T11:13:33.874453+0000","last_peered":"2024-07-26T11:13:33.874453+0000","last_clean":"2024-07-26T11:13:33.874453+0000","last_became_active":"2024-07-26T11:10:22.297979+0000","last_became_peered":"2024-07-26T11:10:22.297979+0000","last_unstale":"2024-07-26T11:13:33.874453+0000","last_undegraded":"2024-07-26T11:13:33.874453+0000","last_fullsized":"2024-07-26T11:13:33.874453+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:30:39.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.871413+0000","last_change":"2024-07-26T11:10:22.320850+0000","last_active":"2024-07-26T11:13:33.871413+0000","last_peered":"2024-07-26T11:13:33.871413+0000","last_clean":"2024-07-26T11:13:33.871413+0000","last_became_active":"2024-07-26T11:10:22.319838+0000","last_became_peered":"2024-07-26T11:10:22.319838+0000","last_unstale":"2024-07-26T11:13:33.871413+0000","last_undegraded":"2024-07-26T11:13:33.871413+0000","last_fullsized":"2024-07-26T11:13:33.871413+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:14.289366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.873003+0000","last_change":"2024-07-26T11:10:22.321305+0000","last_active":"2024-07-26T11:13:33.873003+0000","last_peered":"2024-07-26T11:13:33.873003+0000","last_clean":"2024-07-26T11:13:33.873003+0000","last_became_active":"2024-07-26T11:10:22.320733+0000","last_became_peered":"2024-07-26T11:10:22.320733+0000","last_unstale":"2024-07-26T11:13:33.873003+0000","last_undegraded":"2024-07-26T11:13:33.873003+0000","last_fullsized":"2024-07-26T11:13:33.873003+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:24:33.125828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874282+0000","last_change":"2024-07-26T11:10:22.318193+0000","last_active":"2024-07-26T11:13:33.874282+0000","last_peered":"2024-07-26T11:13:33.874282+0000","last_clean":"2024-07-26T11:13:33.874282+0000","last_became_active":"2024-07-26T11:10:22.317819+0000","last_became_peered":"2024-07-26T11:10:22.317819+0000","last_unstale":"2024-07-26T11:13:33.874282+0000","last_undegraded":"2024-07-26T11:13:33.874282+0000","last_fullsized":"2024-07-26T11:13:33.874282+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:14:06.105152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875131+0000","last_change":"2024-07-26T11:10:22.318222+0000","last_active":"2024-07-26T11:13:33.875131+0000","last_peered":"2024-07-26T11:13:33.875131+0000","last_clean":"2024-07-26T11:13:33.875131+0000","last_became_active":"2024-07-26T11:10:22.317806+0000","last_became_peered":"2024-07-26T11:10:22.317806+0000","last_unstale":"2024-07-26T11:13:33.875131+0000","last_undegraded":"2024-07-26T11:13:33.875131+0000","last_fullsized":"2024-07-26T11:13:33.875131+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:55:27.510542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.895175+0000","last_change":"2024-07-26T11:10:22.330166+0000","last_active":"2024-07-26T11:13:33.895175+0000","last_peered":"2024-07-26T11:13:33.895175+0000","last_clean":"2024-07-26T11:13:33.895175+0000","last_became_active":"2024-07-26T11:10:22.329722+0000","last_became_peered":"2024-07-26T11:10:22.329722+0000","last_unstale":"2024-07-26T11:13:33.895175+0000","last_undegraded":"2024-07-26T11:13:33.895175+0000","last_fullsized":"2024-07-26T11:13:33.895175+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:58.816139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.866466+0000","last_change":"2024-07-26T11:10:22.302774+0000","last_active":"2024-07-26T11:13:33.866466+0000","last_peered":"2024-07-26T11:13:33.866466+0000","last_clean":"2024-07-26T11:13:33.866466+0000","last_became_active":"2024-07-26T11:10:22.302382+0000","last_became_peered":"2024-07-26T11:10:22.302382+0000","last_unstale":"2024-07-26T11:13:33.866466+0000","last_undegraded":"2024-07-26T11:13:33.866466+0000","last_fullsized":"2024-07-26T11:13:33.866466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:02:58.019598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.886235+0000","last_change":"2024-07-26T11:10:22.312475+0000","last_active":"2024-07-26T11:13:33.886235+0000","last_peered":"2024-07-26T11:13:33.886235+0000","last_clean":"2024-07-26T11:13:33.886235+0000","last_became_active":"2024-07-26T11:10:22.312078+0000","last_became_peered":"2024-07-26T11:10:22.312078+0000","last_unstale":"2024-07-26T11:13:33.886235+0000","last_undegraded":"2024-07-26T11:13:33.886235+0000","last_fullsized":"2024-07-26T11:13:33.886235+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:25:37.705136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.877966+0000","last_change":"2024-07-26T11:10:22.314776+0000","last_active":"2024-07-26T11:13:33.877966+0000","last_peered":"2024-07-26T11:13:33.877966+0000","last_clean":"2024-07-26T11:13:33.877966+0000","last_became_active":"2024-07-26T11:10:22.307808+0000","last_became_peered":"2024-07-26T11:10:22.307808+0000","last_unstale":"2024-07-26T11:13:33.877966+0000","last_undegraded":"2024-07-26T11:13:33.877966+0000","last_fullsized":"2024-07-26T11:13:33.877966+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:44:06.152799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.891049+0000","last_change":"2024-07-26T11:10:22.328102+0000","last_active":"2024-07-26T11:13:33.891049+0000","last_peered":"2024-07-26T11:13:33.891049+0000","last_clean":"2024-07-26T11:13:33.891049+0000","last_became_active":"2024-07-26T11:10:22.327779+0000","last_became_peered":"2024-07-26T11:10:22.327779+0000","last_unstale":"2024-07-26T11:13:33.891049+0000","last_undegraded":"2024-07-26T11:13:33.891049+0000","last_fullsized":"2024-07-26T11:13:33.891049+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:14:34.569748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.883870+0000","last_change":"2024-07-26T11:10:22.329057+0000","last_active":"2024-07-26T11:13:33.883870+0000","last_peered":"2024-07-26T11:13:33.883870+0000","last_clean":"2024-07-26T11:13:33.883870+0000","last_became_active":"2024-07-26T11:10:22.328656+0000","last_became_peered":"2024-07-26T11:10:22.328656+0000","last_unstale":"2024-07-26T11:13:33.883870+0000","last_undegraded":"2024-07-26T11:13:33.883870+0000","last_fullsized":"2024-07-26T11:13:33.883870+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:18:39.861674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.865904+0000","last_change":"2024-07-26T11:10:22.326115+0000","last_active":"2024-07-26T11:13:33.865904+0000","last_peered":"2024-07-26T11:13:33.865904+0000","last_clean":"2024-07-26T11:13:33.865904+0000","last_became_active":"2024-07-26T11:10:22.325538+0000","last_became_peered":"2024-07-26T11:10:22.325538+0000","last_unstale":"2024-07-26T11:13:33.865904+0000","last_undegraded":"2024-07-26T11:13:33.865904+0000","last_fullsized":"2024-07-26T11:13:33.865904+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:18:19.339571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.892395+0000","last_change":"2024-07-26T11:10:22.326910+0000","last_active":"2024-07-26T11:13:33.892395+0000","last_peered":"2024-07-26T11:13:33.892395+0000","last_clean":"2024-07-26T11:13:33.892395+0000","last_became_active":"2024-07-26T11:10:22.324532+0000","last_became_peered":"2024-07-26T11:10:22.324532+0000","last_unstale":"2024-07-26T11:13:33.892395+0000","last_undegraded":"2024-07-26T11:13:33.892395+0000","last_fullsized":"2024-07-26T11:13:33.892395+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:33.671780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.890614+0000","last_change":"2024-07-26T11:10:22.312926+0000","last_active":"2024-07-26T11:13:33.890614+0000","last_peered":"2024-07-26T11:13:33.890614+0000","last_clean":"2024-07-26T11:13:33.890614+0000","last_became_active":"2024-07-26T11:10:22.311642+0000","last_became_peered":"2024-07-26T11:10:22.311642+0000","last_unstale":"2024-07-26T11:13:33.890614+0000","last_undegraded":"2024-07-26T11:13:33.890614+0000","last_fullsized":"2024-07-26T11:13:33.890614+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:56:29.644965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.877291+0000","last_change":"2024-07-26T11:10:34.976934+0000","last_active":"2024-07-26T11:13:33.877291+0000","last_peered":"2024-07-26T11:13:33.877291+0000","last_clean":"2024-07-26T11:13:33.877291+0000","last_became_active":"2024-07-26T11:10:34.976551+0000","last_became_peered":"2024-07-26T11:10:34.976551+0000","last_unstale":"2024-07-26T11:13:33.877291+0000","last_undegraded":"2024-07-26T11:13:33.877291+0000","last_fullsized":"2024-07-26T11:13:33.877291+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:11:43.883644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875097+0000","last_change":"2024-07-26T11:10:22.304444+0000","last_active":"2024-07-26T11:13:33.875097+0000","last_peered":"2024-07-26T11:13:33.875097+0000","last_clean":"2024-07-26T11:13:33.875097+0000","last_became_active":"2024-07-26T11:10:22.302825+0000","last_became_peered":"2024-07-26T11:10:22.302825+0000","last_unstale":"2024-07-26T11:13:33.875097+0000","last_undegraded":"2024-07-26T11:13:33.875097+0000","last_fullsized":"2024-07-26T11:13:33.875097+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:15.229057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.870606+0000","last_change":"2024-07-26T11:10:22.297515+0000","last_active":"2024-07-26T11:13:33.870606+0000","last_peered":"2024-07-26T11:13:33.870606+0000","last_clean":"2024-07-26T11:13:33.870606+0000","last_became_active":"2024-07-26T11:10:22.297134+0000","last_became_peered":"2024-07-26T11:10:22.297134+0000","last_unstale":"2024-07-26T11:13:33.870606+0000","last_undegraded":"2024-07-26T11:13:33.870606+0000","last_fullsized":"2024-07-26T11:13:33.870606+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:25:08.794363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875580+0000","last_change":"2024-07-26T11:10:22.304333+0000","last_active":"2024-07-26T11:13:33.875580+0000","last_peered":"2024-07-26T11:13:33.875580+0000","last_clean":"2024-07-26T11:13:33.875580+0000","last_became_active":"2024-07-26T11:10:22.303639+0000","last_became_peered":"2024-07-26T11:10:22.303639+0000","last_unstale":"2024-07-26T11:13:33.875580+0000","last_undegraded":"2024-07-26T11:13:33.875580+0000","last_fullsized":"2024-07-26T11:13:33.875580+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:47.053463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874832+0000","last_change":"2024-07-26T11:10:22.320998+0000","last_active":"2024-07-26T11:13:33.874832+0000","last_peered":"2024-07-26T11:13:33.874832+0000","last_clean":"2024-07-26T11:13:33.874832+0000","last_became_active":"2024-07-26T11:10:22.320610+0000","last_became_peered":"2024-07-26T11:10:22.320610+0000","last_unstale":"2024-07-26T11:13:33.874832+0000","last_undegraded":"2024-07-26T11:13:33.874832+0000","last_fullsized":"2024-07-26T11:13:33.874832+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:52:28.802241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.865671+0000","last_change":"2024-07-26T11:10:34.970486+0000","last_active":"2024-07-26T11:13:33.865671+0000","last_peered":"2024-07-26T11:13:33.865671+0000","last_clean":"2024-07-26T11:13:33.865671+0000","last_became_active":"2024-07-26T11:10:34.969965+0000","last_became_peered":"2024-07-26T11:10:34.969965+0000","last_unstale":"2024-07-26T11:13:33.865671+0000","last_undegraded":"2024-07-26T11:13:33.865671+0000","last_fullsized":"2024-07-26T11:13:33.865671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:41.593251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.876426+0000","last_change":"2024-07-26T11:10:22.321261+0000","last_active":"2024-07-26T11:13:33.876426+0000","last_peered":"2024-07-26T11:13:33.876426+0000","last_clean":"2024-07-26T11:13:33.876426+0000","last_became_active":"2024-07-26T11:10:22.320819+0000","last_became_peered":"2024-07-26T11:10:22.320819+0000","last_unstale":"2024-07-26T11:13:33.876426+0000","last_undegraded":"2024-07-26T11:13:33.876426+0000","last_fullsized":"2024-07-26T11:13:33.876426+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:04:56.872600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.887326+0000","last_change":"2024-07-26T11:10:22.321206+0000","last_active":"2024-07-26T11:13:33.887326+0000","last_peered":"2024-07-26T11:13:33.887326+0000","last_clean":"2024-07-26T11:13:33.887326+0000","last_became_active":"2024-07-26T11:10:22.320700+0000","last_became_peered":"2024-07-26T11:10:22.320700+0000","last_unstale":"2024-07-26T11:13:33.887326+0000","last_undegraded":"2024-07-26T11:13:33.887326+0000","last_fullsized":"2024-07-26T11:13:33.887326+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:31:20.700808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.870762+0000","last_change":"2024-07-26T11:10:22.300061+0000","last_active":"2024-07-26T11:13:33.870762+0000","last_peered":"2024-07-26T11:13:33.870762+0000","last_clean":"2024-07-26T11:13:33.870762+0000","last_became_active":"2024-07-26T11:10:22.298737+0000","last_became_peered":"2024-07-26T11:10:22.298737+0000","last_unstale":"2024-07-26T11:13:33.870762+0000","last_undegraded":"2024-07-26T11:13:33.870762+0000","last_fullsized":"2024-07-26T11:13:33.870762+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:45:25.500816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889377+0000","last_change":"2024-07-26T11:10:22.328534+0000","last_active":"2024-07-26T11:13:33.889377+0000","last_peered":"2024-07-26T11:13:33.889377+0000","last_clean":"2024-07-26T11:13:33.889377+0000","last_became_active":"2024-07-26T11:10:22.327929+0000","last_became_peered":"2024-07-26T11:10:22.327929+0000","last_unstale":"2024-07-26T11:13:33.889377+0000","last_undegraded":"2024-07-26T11:13:33.889377+0000","last_fullsized":"2024-07-26T11:13:33.889377+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:09:55.152881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":40,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.890781+0000","last_change":"2024-07-26T11:10:22.324753+0000","last_active":"2024-07-26T11:13:33.890781+0000","last_peered":"2024-07-26T11:13:33.890781+0000","last_clean":"2024-07-26T11:13:33.890781+0000","last_became_active":"2024-07-26T11:10:22.324307+0000","last_became_peered":"2024-07-26T11:10:22.324307+0000","last_unstale":"2024-07-26T11:13:33.890781+0000","last_undegraded":"2024-07-26T11:13:33.890781+0000","last_fullsized":"2024-07-26T11:13:33.890781+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:40.566465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.890918+0000","last_change":"2024-07-26T11:10:22.328183+0000","last_active":"2024-07-26T11:13:33.890918+0000","last_peered":"2024-07-26T11:13:33.890918+0000","last_clean":"2024-07-26T11:13:33.890918+0000","last_became_active":"2024-07-26T11:10:22.327805+0000","last_became_peered":"2024-07-26T11:10:22.327805+0000","last_unstale":"2024-07-26T11:13:33.890918+0000","last_undegraded":"2024-07-26T11:13:33.890918+0000","last_fullsized":"2024-07-26T11:13:33.890918+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:11:23.238390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73723,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1317,"num_read_kb":737,"num_write":258,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2194,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":221169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":73,"seq":313532612668,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":527167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808896,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28708,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716444,"statfs":{"total":95995035648,"available":95965638656,"internally_reserved":0,"allocated":1732608,"data_stored":1180030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972422,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":529939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168653,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":529735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364881,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28620,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716532,"statfs":{"total":95995035648,"available":95965728768,"internally_reserved":0,"allocated":1642496,"data_stored":1122187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528407,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":521347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724639,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36952,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708200,"statfs":{"total":95995035648,"available":95957196800,"internally_reserved":0,"allocated":1654784,"data_stored":1113813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888164,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1118208,"data_stored":592722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084392,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":521055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378734,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":522597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542259,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":589146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705783,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":69037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T11:14:02.002 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-07-26T11:14:02.002 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-07-26T11:14:02.002 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-07-26T11:14:02.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph health --format=json 2024-07-26T11:14:02.507 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:03.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:03 smithi134 ceph-mon[31966]: from='client.25127 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:14:03.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:03 smithi134 ceph-mon[31966]: pgmap v422: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:03 smithi189 ceph-mon[36790]: from='client.25127 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:14:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:03 smithi189 ceph-mon[36790]: pgmap v422: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:03 smithi193 ceph-mon[37473]: from='client.25127 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:14:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:03 smithi193 ceph-mon[37473]: pgmap v422: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:04.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.784+0000 7fba0fc00640 1 -- 172.21.15.134:0/2705581932 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba08104d20 msgr2=0x7fba0810d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:04.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.784+0000 7fba0fc00640 1 --2- 172.21.15.134:0/2705581932 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba08104d20 0x7fba0810d360 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb9f4009900 tx=0x7fb9f402a410 comp rx=0 tx=0).stop 2024-07-26T11:14:04.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.784+0000 7fba0fc00640 1 -- 172.21.15.134:0/2705581932 shutdown_connections 2024-07-26T11:14:04.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.784+0000 7fba0fc00640 1 --2- 172.21.15.134:0/2705581932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba0810d8a0 0x7fba0810fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:04.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.784+0000 7fba0fc00640 1 --2- 172.21.15.134:0/2705581932 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba08104d20 0x7fba0810d360 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:04.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.784+0000 7fba0fc00640 1 --2- 172.21.15.134:0/2705581932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba081042f0 0x7fba08104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:04.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.784+0000 7fba0fc00640 1 -- 172.21.15.134:0/2705581932 >> 172.21.15.134:0/2705581932 conn(0x7fba080790e0 msgr2=0x7fba080794f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:04.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.784+0000 7fba0fc00640 1 -- 172.21.15.134:0/2705581932 shutdown_connections 2024-07-26T11:14:04.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.784+0000 7fba0fc00640 1 -- 172.21.15.134:0/2705581932 wait complete. 2024-07-26T11:14:04.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.786+0000 7fba0fc00640 1 Processor -- start 2024-07-26T11:14:04.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.786+0000 7fba0fc00640 1 -- start start 2024-07-26T11:14:04.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.786+0000 7fba0fc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba081042f0 0x7fba0819c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:04.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.786+0000 7fba0fc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba08104d20 0x7fba0819c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:04.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.786+0000 7fba0fc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba0810d8a0 0x7fba081a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:04.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.786+0000 7fba0fc00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fba08113510 con 0x7fba0810d8a0 2024-07-26T11:14:04.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.786+0000 7fba0fc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fba08113390 con 0x7fba08104d20 2024-07-26T11:14:04.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.786+0000 7fba0fc00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba08113690 con 0x7fba081042f0 2024-07-26T11:14:04.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.787+0000 7fba07e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba08104d20 0x7fba0819c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:04.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.787+0000 7fba0dc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba0810d8a0 0x7fba081a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:04.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.787+0000 7fba07e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba08104d20 0x7fba0819c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:45502/0 (socket says 172.21.15.134:45502) 2024-07-26T11:14:04.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.787+0000 7fba07e00640 1 -- 172.21.15.134:0/2501138189 learned_addr learned my addr 172.21.15.134:0/2501138189 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:04.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.787+0000 7fba0d200640 1 --2- 172.21.15.134:0/2501138189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba081042f0 0x7fba0819c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:04.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.787+0000 7fba07e00640 1 -- 172.21.15.134:0/2501138189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba081042f0 msgr2=0x7fba0819c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:04.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.787+0000 7fba07e00640 1 --2- 172.21.15.134:0/2501138189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba081042f0 0x7fba0819c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:04.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.787+0000 7fba07e00640 1 -- 172.21.15.134:0/2501138189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba0810d8a0 msgr2=0x7fba081a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:04.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.787+0000 7fba07e00640 1 --2- 172.21.15.134:0/2501138189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba0810d8a0 0x7fba081a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:04.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.787+0000 7fba07e00640 1 -- 172.21.15.134:0/2501138189 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba081a20f0 con 0x7fba08104d20 2024-07-26T11:14:04.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.787+0000 7fba0dc00640 1 --2- 172.21.15.134:0/2501138189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba0810d8a0 0x7fba081a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:04.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.788+0000 7fba07e00640 1 --2- 172.21.15.134:0/2501138189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba08104d20 0x7fba0819c720 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb9f402a930 tx=0x7fb9f4031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:04.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.788+0000 7fba05600640 1 -- 172.21.15.134:0/2501138189 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f403f030 con 0x7fba08104d20 2024-07-26T11:14:04.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.788+0000 7fba0fc00640 1 -- 172.21.15.134:0/2501138189 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba081a2380 con 0x7fba08104d20 2024-07-26T11:14:04.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.788+0000 7fba05600640 1 -- 172.21.15.134:0/2501138189 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb9f4004490 con 0x7fba08104d20 2024-07-26T11:14:04.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.788+0000 7fba05600640 1 -- 172.21.15.134:0/2501138189 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4035070 con 0x7fba08104d20 2024-07-26T11:14:04.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.788+0000 7fba0fc00640 1 -- 172.21.15.134:0/2501138189 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba081a8b40 con 0x7fba08104d20 2024-07-26T11:14:04.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.789+0000 7fba0fc00640 1 -- 172.21.15.134:0/2501138189 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba0806bc10 con 0x7fba08104d20 2024-07-26T11:14:04.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.792+0000 7fba05600640 1 -- 172.21.15.134:0/2501138189 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fb9f4042030 con 0x7fba08104d20 2024-07-26T11:14:04.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.793+0000 7fba05600640 1 --2- 172.21.15.134:0/2501138189 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb9f00b3c20 0x7fb9f00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:04.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.793+0000 7fba05600640 1 -- 172.21.15.134:0/2501138189 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb9f4104860 con 0x7fba08104d20 2024-07-26T11:14:04.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.793+0000 7fba0d200640 1 --2- 172.21.15.134:0/2501138189 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb9f00b3c20 0x7fb9f00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:04.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.794+0000 7fba0d200640 1 --2- 172.21.15.134:0/2501138189 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb9f00b3c20 0x7fb9f00b6130 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc009bd0 tx=0x7fb9fc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:04.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:04.795+0000 7fba05600640 1 -- 172.21.15.134:0/2501138189 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb9f40cb360 con 0x7fba08104d20 2024-07-26T11:14:05.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:05.036+0000 7fba0fc00640 1 -- 172.21.15.134:0/2501138189 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fba08104700 con 0x7fba08104d20 2024-07-26T11:14:05.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:05.036+0000 7fba05600640 1 -- 172.21.15.134:0/2501138189 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fb9f40cf8b0 con 0x7fba08104d20 2024-07-26T11:14:05.039 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:14:05.039 INFO:teuthology.orchestra.run.smithi134.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-07-26T11:14:05.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:05.039+0000 7fba0fc00640 1 -- 172.21.15.134:0/2501138189 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb9f00b3c20 msgr2=0x7fb9f00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:05.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:05.039+0000 7fba0fc00640 1 --2- 172.21.15.134:0/2501138189 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb9f00b3c20 0x7fb9f00b6130 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc009bd0 tx=0x7fb9fc005f90 comp rx=0 tx=0).stop 2024-07-26T11:14:05.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:05.039+0000 7fba0fc00640 1 -- 172.21.15.134:0/2501138189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba08104d20 msgr2=0x7fba0819c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:05.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:05.039+0000 7fba0fc00640 1 --2- 172.21.15.134:0/2501138189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba08104d20 0x7fba0819c720 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb9f402a930 tx=0x7fb9f4031a20 comp rx=0 tx=0).stop 2024-07-26T11:14:05.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:05.040+0000 7fba0fc00640 1 -- 172.21.15.134:0/2501138189 shutdown_connections 2024-07-26T11:14:05.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:05.040+0000 7fba0fc00640 1 --2- 172.21.15.134:0/2501138189 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb9f00b3c20 0x7fb9f00b6130 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:05.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:05.040+0000 7fba0fc00640 1 --2- 172.21.15.134:0/2501138189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba0810d8a0 0x7fba081a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:05.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:05.040+0000 7fba0fc00640 1 --2- 172.21.15.134:0/2501138189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba08104d20 0x7fba0819c720 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:05.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:05.040+0000 7fba0fc00640 1 --2- 172.21.15.134:0/2501138189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba081042f0 0x7fba0819c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:05.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:05.040+0000 7fba0fc00640 1 -- 172.21.15.134:0/2501138189 >> 172.21.15.134:0/2501138189 conn(0x7fba080790e0 msgr2=0x7fba081000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:05.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:05.040+0000 7fba0fc00640 1 -- 172.21.15.134:0/2501138189 shutdown_connections 2024-07-26T11:14:05.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:05.040+0000 7fba0fc00640 1 -- 172.21.15.134:0/2501138189 wait complete. 2024-07-26T11:14:05.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:05 smithi134 ceph-mon[31966]: pgmap v423: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:05.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:05 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2501138189' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T11:14:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:05 smithi189 ceph-mon[36790]: pgmap v423: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:05 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2501138189' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T11:14:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:05 smithi193 ceph-mon[37473]: pgmap v423: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:05 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2501138189' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-07-26T11:14:05.804 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-07-26T11:14:05.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:14:06.265 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:07.326 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:07 smithi134 ceph-mon[31966]: pgmap v424: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:07 smithi189 ceph-mon[36790]: pgmap v424: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:07 smithi193 ceph-mon[37473]: pgmap v424: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:08.312 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:08 smithi134 ceph-mon[31966]: pgmap v425: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:08.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:08 smithi189 ceph-mon[36790]: pgmap v425: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:08.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.513+0000 7fdcffe00640 1 -- 172.21.15.134:0/1080029954 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd00102930 msgr2=0x7fdd00106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:08.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.513+0000 7fdcffe00640 1 --2- 172.21.15.134:0/1080029954 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd00102930 0x7fdd00106e60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdce8009930 tx=0x7fdce802a420 comp rx=0 tx=0).stop 2024-07-26T11:14:08.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.513+0000 7fdcffe00640 1 -- 172.21.15.134:0/1080029954 shutdown_connections 2024-07-26T11:14:08.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.513+0000 7fdcffe00640 1 --2- 172.21.15.134:0/1080029954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd00107420 0x7fdd00107900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:08.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.513+0000 7fdcffe00640 1 --2- 172.21.15.134:0/1080029954 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd00102930 0x7fdd00106e60 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:08.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.513+0000 7fdcffe00640 1 --2- 172.21.15.134:0/1080029954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd00101fe0 0x7fdd001023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:08.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.513+0000 7fdcffe00640 1 -- 172.21.15.134:0/1080029954 >> 172.21.15.134:0/1080029954 conn(0x7fdd000fdcd0 msgr2=0x7fdd00100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:08.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.513+0000 7fdcffe00640 1 -- 172.21.15.134:0/1080029954 shutdown_connections 2024-07-26T11:14:08.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.514+0000 7fdcffe00640 1 -- 172.21.15.134:0/1080029954 wait complete. 2024-07-26T11:14:08.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.515+0000 7fdcffe00640 1 Processor -- start 2024-07-26T11:14:08.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.515+0000 7fdcffe00640 1 -- start start 2024-07-26T11:14:08.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.515+0000 7fdcffe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd00101fe0 0x7fdd001a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:08.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.515+0000 7fdcffe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd00102930 0x7fdd001a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:08.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.515+0000 7fdcffe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd00107420 0x7fdd001a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:08.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.515+0000 7fdcffe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fdd00119910 con 0x7fdd00107420 2024-07-26T11:14:08.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.515+0000 7fdcffe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdd00119790 con 0x7fdd00101fe0 2024-07-26T11:14:08.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.515+0000 7fdcffe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdd00119a90 con 0x7fdd00102930 2024-07-26T11:14:08.518 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.516+0000 7fdcfea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd00101fe0 0x7fdd001a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:08.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.516+0000 7fdcfea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd00101fe0 0x7fdd001a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:45516/0 (socket says 172.21.15.134:45516) 2024-07-26T11:14:08.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.516+0000 7fdcff400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd00107420 0x7fdd001a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:08.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.516+0000 7fdcfea00640 1 -- 172.21.15.134:0/1801362943 learned_addr learned my addr 172.21.15.134:0/1801362943 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:08.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.516+0000 7fdcf6000640 1 --2- 172.21.15.134:0/1801362943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd00102930 0x7fdd001a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:08.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.516+0000 7fdcfea00640 1 -- 172.21.15.134:0/1801362943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd00102930 msgr2=0x7fdd001a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:08.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.516+0000 7fdcfea00640 1 --2- 172.21.15.134:0/1801362943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd00102930 0x7fdd001a0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:08.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.516+0000 7fdcfea00640 1 -- 172.21.15.134:0/1801362943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd00107420 msgr2=0x7fdd001a5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:08.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.516+0000 7fdcfea00640 1 --2- 172.21.15.134:0/1801362943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd00107420 0x7fdd001a5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:08.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.516+0000 7fdcfea00640 1 -- 172.21.15.134:0/1801362943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd001a63d0 con 0x7fdd00101fe0 2024-07-26T11:14:08.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.517+0000 7fdcfea00640 1 --2- 172.21.15.134:0/1801362943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd00101fe0 0x7fdd001a0460 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdcf000ea60 tx=0x7fdcf000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:08.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.517+0000 7fdcf7e00640 1 -- 172.21.15.134:0/1801362943 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf000cde0 con 0x7fdd00101fe0 2024-07-26T11:14:08.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.517+0000 7fdcf7e00640 1 -- 172.21.15.134:0/1801362943 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcf00035f0 con 0x7fdd00101fe0 2024-07-26T11:14:08.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.517+0000 7fdcf7e00640 1 -- 172.21.15.134:0/1801362943 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0010690 con 0x7fdd00101fe0 2024-07-26T11:14:08.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.517+0000 7fdcffe00640 1 -- 172.21.15.134:0/1801362943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd001a66c0 con 0x7fdd00101fe0 2024-07-26T11:14:08.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.517+0000 7fdcffe00640 1 -- 172.21.15.134:0/1801362943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd001ace20 con 0x7fdd00101fe0 2024-07-26T11:14:08.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.518+0000 7fdcffe00640 1 -- 172.21.15.134:0/1801362943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd00106e60 con 0x7fdd00101fe0 2024-07-26T11:14:08.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.521+0000 7fdcf7e00640 1 -- 172.21.15.134:0/1801362943 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fdcf00037a0 con 0x7fdd00101fe0 2024-07-26T11:14:08.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.522+0000 7fdcf7e00640 1 --2- 172.21.15.134:0/1801362943 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdce40b3c20 0x7fdce40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:08.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.522+0000 7fdcf7e00640 1 -- 172.21.15.134:0/1801362943 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fdcf00e45c0 con 0x7fdd00101fe0 2024-07-26T11:14:08.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.522+0000 7fdcf6000640 1 --2- 172.21.15.134:0/1801362943 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdce40b3c20 0x7fdce40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:08 smithi193 ceph-mon[37473]: pgmap v425: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:08.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.523+0000 7fdcf6000640 1 --2- 172.21.15.134:0/1801362943 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdce40b3c20 0x7fdce40b6130 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdce802a940 tx=0x7fdce8005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:08.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.524+0000 7fdcf7e00640 1 -- 172.21.15.134:0/1801362943 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fdcf0055600 con 0x7fdd00101fe0 2024-07-26T11:14:08.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.759+0000 7fdcffe00640 1 -- 172.21.15.134:0/1801362943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd001a1590 con 0x7fdd00101fe0 2024-07-26T11:14:08.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.760+0000 7fdcf7e00640 1 -- 172.21.15.134:0/1801362943 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11990 (secure 0 0 0) 0x7fdd001a1590 con 0x7fdd00101fe0 2024-07-26T11:14:08.763 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:14:08.763 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":29,"btime":"2024-07-26T11:11:55:972763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:11:55.128451+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24721,"mds_1":24719,"mds_2":24703,"mds_3":24688,"mds_4":24686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24635":{"gid":24635,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24641":{"gid":24641,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":32,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24719":{"gid":24719,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":28,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24721,"qdb_cluster":[24686,24688,24703,24719,24721]},"id":1}]} 2024-07-26T11:14:08.763 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 29 2024-07-26T11:14:08.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.763+0000 7fdcffe00640 1 -- 172.21.15.134:0/1801362943 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdce40b3c20 msgr2=0x7fdce40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:08.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.763+0000 7fdcffe00640 1 --2- 172.21.15.134:0/1801362943 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdce40b3c20 0x7fdce40b6130 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fdce802a940 tx=0x7fdce8005a20 comp rx=0 tx=0).stop 2024-07-26T11:14:08.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.763+0000 7fdcffe00640 1 -- 172.21.15.134:0/1801362943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd00101fe0 msgr2=0x7fdd001a0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:08.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.763+0000 7fdcffe00640 1 --2- 172.21.15.134:0/1801362943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd00101fe0 0x7fdd001a0460 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdcf000ea60 tx=0x7fdcf000ef20 comp rx=0 tx=0).stop 2024-07-26T11:14:08.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.764+0000 7fdcffe00640 1 -- 172.21.15.134:0/1801362943 shutdown_connections 2024-07-26T11:14:08.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.764+0000 7fdcffe00640 1 --2- 172.21.15.134:0/1801362943 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdce40b3c20 0x7fdce40b6130 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:08.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.764+0000 7fdcffe00640 1 --2- 172.21.15.134:0/1801362943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd00102930 0x7fdd001a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:08.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.764+0000 7fdcffe00640 1 --2- 172.21.15.134:0/1801362943 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdd00107420 0x7fdd001a5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:08.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.764+0000 7fdcffe00640 1 --2- 172.21.15.134:0/1801362943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd00101fe0 0x7fdd001a0460 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:08.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.764+0000 7fdcffe00640 1 -- 172.21.15.134:0/1801362943 >> 172.21.15.134:0/1801362943 conn(0x7fdd000fdcd0 msgr2=0x7fdd00104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:08.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.764+0000 7fdcffe00640 1 -- 172.21.15.134:0/1801362943 shutdown_connections 2024-07-26T11:14:08.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:08.764+0000 7fdcffe00640 1 -- 172.21.15.134:0/1801362943 wait complete. 2024-07-26T11:14:09.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:09 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1801362943' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:14:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:09 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1801362943' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:14:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:09 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1801362943' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:14:09.632 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:14:10.092 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:10.405 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:10 smithi134 ceph-mon[31966]: pgmap v426: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:10 smithi189 ceph-mon[36790]: pgmap v426: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:10 smithi193 ceph-mon[37473]: pgmap v426: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:12.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.437+0000 7f44b9400640 1 -- 172.21.15.134:0/230596278 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44b410af10 msgr2=0x7f44b410b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:12.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.437+0000 7f44b9400640 1 --2- 172.21.15.134:0/230596278 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44b410af10 0x7f44b410b320 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f449c009900 tx=0x7f449c02a410 comp rx=0 tx=0).stop 2024-07-26T11:14:12.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.438+0000 7f44b9400640 1 -- 172.21.15.134:0/230596278 shutdown_connections 2024-07-26T11:14:12.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.438+0000 7f44b9400640 1 --2- 172.21.15.134:0/230596278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44b410b860 0x7f44b4112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:12.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.438+0000 7f44b9400640 1 --2- 172.21.15.134:0/230596278 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44b410af10 0x7f44b410b320 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:12.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.438+0000 7f44b9400640 1 --2- 172.21.15.134:0/230596278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44b41055d0 0x7f44b41059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:12.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.438+0000 7f44b9400640 1 -- 172.21.15.134:0/230596278 >> 172.21.15.134:0/230596278 conn(0x7f44b4101040 msgr2=0x7f44b4103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:12.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.438+0000 7f44b9400640 1 -- 172.21.15.134:0/230596278 shutdown_connections 2024-07-26T11:14:12.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.438+0000 7f44b9400640 1 -- 172.21.15.134:0/230596278 wait complete. 2024-07-26T11:14:12.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.439+0000 7f44b9400640 1 Processor -- start 2024-07-26T11:14:12.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.439+0000 7f44b9400640 1 -- start start 2024-07-26T11:14:12.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.439+0000 7f44b9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44b41055d0 0x7f44b419bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:12.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.439+0000 7f44b9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44b410af10 0x7f44b419c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:12.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.439+0000 7f44b9400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44b410b860 0x7f44b41a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:12.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.439+0000 7f44b9400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f44b41154d0 con 0x7f44b410b860 2024-07-26T11:14:12.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.439+0000 7f44b9400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f44b4115350 con 0x7f44b410af10 2024-07-26T11:14:12.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.439+0000 7f44b9400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f44b4115650 con 0x7f44b41055d0 2024-07-26T11:14:12.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.440+0000 7f44b3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44b41055d0 0x7f44b419bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:12.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.440+0000 7f44ab400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44b410af10 0x7f44b419c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.440+0000 7f44ab400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44b410af10 0x7f44b419c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:48950/0 (socket says 172.21.15.134:48950) 2024-07-26T11:14:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.440+0000 7f44b3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44b41055d0 0x7f44b419bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:54890/0 (socket says 172.21.15.134:54890) 2024-07-26T11:14:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.440+0000 7f44ab400640 1 -- 172.21.15.134:0/2484936750 learned_addr learned my addr 172.21.15.134:0/2484936750 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.440+0000 7f44b8a00640 1 --2- 172.21.15.134:0/2484936750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44b410b860 0x7f44b41a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.440+0000 7f44b3e00640 1 -- 172.21.15.134:0/2484936750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44b410af10 msgr2=0x7f44b419c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.440+0000 7f44b3e00640 1 --2- 172.21.15.134:0/2484936750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44b410af10 0x7f44b419c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.440+0000 7f44b3e00640 1 -- 172.21.15.134:0/2484936750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44b410b860 msgr2=0x7f44b41a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.440+0000 7f44b3e00640 1 --2- 172.21.15.134:0/2484936750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44b410b860 0x7f44b41a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.440+0000 7f44b3e00640 1 -- 172.21.15.134:0/2484936750 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b41a1e90 con 0x7f44b41055d0 2024-07-26T11:14:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.440+0000 7f44ab400640 1 --2- 172.21.15.134:0/2484936750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44b410af10 0x7f44b419c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.441+0000 7f44b3e00640 1 --2- 172.21.15.134:0/2484936750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44b41055d0 0x7f44b419bf20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f44a400e9f0 tx=0x7f44a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.441+0000 7f44b1600640 1 -- 172.21.15.134:0/2484936750 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a400ccc0 con 0x7f44b41055d0 2024-07-26T11:14:12.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.441+0000 7f44b1600640 1 -- 172.21.15.134:0/2484936750 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44a40108e0 con 0x7f44b41055d0 2024-07-26T11:14:12.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.441+0000 7f44b9400640 1 -- 172.21.15.134:0/2484936750 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44b41a2180 con 0x7f44b41055d0 2024-07-26T11:14:12.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.441+0000 7f44b1600640 1 -- 172.21.15.134:0/2484936750 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a4018880 con 0x7f44b41055d0 2024-07-26T11:14:12.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.441+0000 7f44b9400640 1 -- 172.21.15.134:0/2484936750 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44b41a88e0 con 0x7f44b41055d0 2024-07-26T11:14:12.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.442+0000 7f44b9400640 1 -- 172.21.15.134:0/2484936750 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44b41059e0 con 0x7f44b41055d0 2024-07-26T11:14:12.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.445+0000 7f44b1600640 1 -- 172.21.15.134:0/2484936750 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f44a4010430 con 0x7f44b41055d0 2024-07-26T11:14:12.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.446+0000 7f44b1600640 1 --2- 172.21.15.134:0/2484936750 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f44980b3ca0 0x7f44980b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:12.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.446+0000 7f44b1600640 1 -- 172.21.15.134:0/2484936750 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f44a40e43f0 con 0x7f44b41055d0 2024-07-26T11:14:12.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.446+0000 7f44ab400640 1 --2- 172.21.15.134:0/2484936750 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f44980b3ca0 0x7f44980b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:12.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.447+0000 7f44ab400640 1 --2- 172.21.15.134:0/2484936750 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f44980b3ca0 0x7f44980b61b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f44b41099c0 tx=0x7f449c032ed0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:12.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.450+0000 7f44b1600640 1 -- 172.21.15.134:0/2484936750 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f44a40aaef0 con 0x7f44b41055d0 2024-07-26T11:14:12.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.689+0000 7f44b9400640 1 -- 172.21.15.134:0/2484936750 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44b419d050 con 0x7f44b41055d0 2024-07-26T11:14:12.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.690+0000 7f44b1600640 1 -- 172.21.15.134:0/2484936750 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11990 (secure 0 0 0) 0x7f44a40af440 con 0x7f44b41055d0 2024-07-26T11:14:12.693 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:14:12.693 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":29,"btime":"2024-07-26T11:11:55:972763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:11:55.128451+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24721,"mds_1":24719,"mds_2":24703,"mds_3":24688,"mds_4":24686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24635":{"gid":24635,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24641":{"gid":24641,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":32,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24719":{"gid":24719,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":28,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24721,"qdb_cluster":[24686,24688,24703,24719,24721]},"id":1}]} 2024-07-26T11:14:12.693 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 29 2024-07-26T11:14:12.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.693+0000 7f44b9400640 1 -- 172.21.15.134:0/2484936750 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f44980b3ca0 msgr2=0x7f44980b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:12.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.693+0000 7f44b9400640 1 --2- 172.21.15.134:0/2484936750 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f44980b3ca0 0x7f44980b61b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f44b41099c0 tx=0x7f449c032ed0 comp rx=0 tx=0).stop 2024-07-26T11:14:12.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.693+0000 7f44b9400640 1 -- 172.21.15.134:0/2484936750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44b41055d0 msgr2=0x7f44b419bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:12.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.693+0000 7f44b9400640 1 --2- 172.21.15.134:0/2484936750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44b41055d0 0x7f44b419bf20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f44a400e9f0 tx=0x7f44a400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:14:12.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.693+0000 7f44b9400640 1 -- 172.21.15.134:0/2484936750 shutdown_connections 2024-07-26T11:14:12.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.693+0000 7f44b9400640 1 --2- 172.21.15.134:0/2484936750 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f44980b3ca0 0x7f44980b61b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:12.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.693+0000 7f44b9400640 1 --2- 172.21.15.134:0/2484936750 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f44b410b860 0x7f44b41a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:12.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.693+0000 7f44b9400640 1 --2- 172.21.15.134:0/2484936750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44b410af10 0x7f44b419c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:12.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.693+0000 7f44b9400640 1 --2- 172.21.15.134:0/2484936750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44b41055d0 0x7f44b419bf20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:12.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.693+0000 7f44b9400640 1 -- 172.21.15.134:0/2484936750 >> 172.21.15.134:0/2484936750 conn(0x7f44b4101040 msgr2=0x7f44b4103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:12.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.694+0000 7f44b9400640 1 -- 172.21.15.134:0/2484936750 shutdown_connections 2024-07-26T11:14:12.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:12.694+0000 7f44b9400640 1 -- 172.21.15.134:0/2484936750 wait complete. 2024-07-26T11:14:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:13 smithi134 ceph-mon[31966]: pgmap v427: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:13.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:13 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2484936750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:14:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:13 smithi189 ceph-mon[36790]: pgmap v427: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:13 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2484936750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:14:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:13 smithi193 ceph-mon[37473]: pgmap v427: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:13 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2484936750' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:14:13.620 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T11:14:14.072 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:15.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:15 smithi134 ceph-mon[31966]: pgmap v428: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:15 smithi189 ceph-mon[36790]: pgmap v428: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:15 smithi193 ceph-mon[37473]: pgmap v428: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:16.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.323+0000 7f19cb200640 1 -- 172.21.15.134:0/1466788299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19cc103400 msgr2=0x7f19cc103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:16.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.323+0000 7f19cb200640 1 --2- 172.21.15.134:0/1466788299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19cc103400 0x7f19cc103810 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f19bc009ca0 tx=0x7f19bc02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:14:16.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.323+0000 7f19cb200640 1 -- 172.21.15.134:0/1466788299 shutdown_connections 2024-07-26T11:14:16.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.323+0000 7f19cb200640 1 --2- 172.21.15.134:0/1466788299 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19cc109690 0x7f19cc10fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:16.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.323+0000 7f19cb200640 1 --2- 172.21.15.134:0/1466788299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19cc108d40 0x7f19cc109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:16.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.323+0000 7f19cb200640 1 --2- 172.21.15.134:0/1466788299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19cc103400 0x7f19cc103810 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:16.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.323+0000 7f19cb200640 1 -- 172.21.15.134:0/1466788299 >> 172.21.15.134:0/1466788299 conn(0x7f19cc0fee90 msgr2=0x7f19cc1012e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:16.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.323+0000 7f19cb200640 1 -- 172.21.15.134:0/1466788299 shutdown_connections 2024-07-26T11:14:16.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.323+0000 7f19cb200640 1 -- 172.21.15.134:0/1466788299 wait complete. 2024-07-26T11:14:16.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.324+0000 7f19cb200640 1 Processor -- start 2024-07-26T11:14:16.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.324+0000 7f19cb200640 1 -- start start 2024-07-26T11:14:16.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.325+0000 7f19cb200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19cc103400 0x7f19cc199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:16.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.325+0000 7f19cb200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19cc108d40 0x7f19cc19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:16.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.325+0000 7f19cb200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19cc109690 0x7f19cc19f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:16.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.325+0000 7f19cb200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc113390 con 0x7f19cc109690 2024-07-26T11:14:16.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.325+0000 7f19cb200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc113210 con 0x7f19cc108d40 2024-07-26T11:14:16.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.325+0000 7f19cb200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc113510 con 0x7f19cc103400 2024-07-26T11:14:16.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.325+0000 7f19c9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19cc108d40 0x7f19cc19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:16.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.325+0000 7f19c9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19cc108d40 0x7f19cc19a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:48970/0 (socket says 172.21.15.134:48970) 2024-07-26T11:14:16.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.325+0000 7f19c9400640 1 -- 172.21.15.134:0/1567923830 learned_addr learned my addr 172.21.15.134:0/1567923830 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:16.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.325+0000 7f19c9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19cc103400 0x7f19cc199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:16.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.325+0000 7f19ca800640 1 --2- 172.21.15.134:0/1567923830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19cc109690 0x7f19cc19f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:16.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.326+0000 7f19c9400640 1 -- 172.21.15.134:0/1567923830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19cc103400 msgr2=0x7f19cc199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:16.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.326+0000 7f19c9400640 1 --2- 172.21.15.134:0/1567923830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19cc103400 0x7f19cc199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:16.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.326+0000 7f19c9400640 1 -- 172.21.15.134:0/1567923830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19cc109690 msgr2=0x7f19cc19f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:16.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.326+0000 7f19c9400640 1 --2- 172.21.15.134:0/1567923830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19cc109690 0x7f19cc19f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:16.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.326+0000 7f19c9400640 1 -- 172.21.15.134:0/1567923830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19cc19fcb0 con 0x7f19cc108d40 2024-07-26T11:14:16.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.326+0000 7f19c9400640 1 --2- 172.21.15.134:0/1567923830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19cc108d40 0x7f19cc19a2e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f19b400e9f0 tx=0x7f19b400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:16.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.326+0000 7f19c2a00640 1 -- 172.21.15.134:0/1567923830 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b400ccc0 con 0x7f19cc108d40 2024-07-26T11:14:16.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.326+0000 7f19cb200640 1 -- 172.21.15.134:0/1567923830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19cc19ffa0 con 0x7f19cc108d40 2024-07-26T11:14:16.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.327+0000 7f19c2a00640 1 -- 172.21.15.134:0/1567923830 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b40108e0 con 0x7f19cc108d40 2024-07-26T11:14:16.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.327+0000 7f19c2a00640 1 -- 172.21.15.134:0/1567923830 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b4018860 con 0x7f19cc108d40 2024-07-26T11:14:16.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.327+0000 7f19cb200640 1 -- 172.21.15.134:0/1567923830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19cc1a6700 con 0x7f19cc108d40 2024-07-26T11:14:16.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.328+0000 7f19cb200640 1 -- 172.21.15.134:0/1567923830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19cc103810 con 0x7f19cc108d40 2024-07-26T11:14:16.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.330+0000 7f19c2a00640 1 -- 172.21.15.134:0/1567923830 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f19b4010430 con 0x7f19cc108d40 2024-07-26T11:14:16.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.332+0000 7f19c2a00640 1 --2- 172.21.15.134:0/1567923830 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f19b00b3c20 0x7f19b00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:16.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.332+0000 7f19c2a00640 1 -- 172.21.15.134:0/1567923830 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f19b40e4900 con 0x7f19cc108d40 2024-07-26T11:14:16.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.332+0000 7f19c9e00640 1 --2- 172.21.15.134:0/1567923830 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f19b00b3c20 0x7f19b00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:16.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.333+0000 7f19c9e00640 1 --2- 172.21.15.134:0/1567923830 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f19b00b3c20 0x7f19b00b6130 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f19bc006010 tx=0x7f19bc009820 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:16.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.334+0000 7f19c2a00640 1 -- 172.21.15.134:0/1567923830 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f19b40ab400 con 0x7f19cc108d40 2024-07-26T11:14:16.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.535+0000 7f19cb200640 1 -- 172.21.15.134:0/1567923830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19cc10b390 con 0x7f19cc108d40 2024-07-26T11:14:16.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.537+0000 7f19c2a00640 1 -- 172.21.15.134:0/1567923830 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25185 (secure 0 0 0) 0x7f19b40af950 con 0x7f19cc108d40 2024-07-26T11:14:16.540 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:14:16.540 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":90,"fsid":"105fbe5c-4b3e-11ef-bcb7-c7b262605968","created":"2024-07-26T11:00:05.955103+0000","modified":"2024-07-26T11:11:35.149490+0000","last_up_change":"2024-07-26T11:09:11.485512+0000","last_in_change":"2024-07-26T11:08:52.341356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T11:05:01.849170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T11:10:16.055725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T11:10:21.216611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T11:10:33.983067+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8abb1365-fc3e-4617-a9ac-76bece8cd7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6803","nonce":4183273954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6805","nonce":4183273954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6809","nonce":4183273954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6807","nonce":4183273954}]},"public_addr":"172.21.15.134:6803/4183273954","cluster_addr":"172.21.15.134:6805/4183273954","heartbeat_back_addr":"172.21.15.134:6809/4183273954","heartbeat_front_addr":"172.21.15.134:6807/4183273954","state":["exists","up"]},{"osd":1,"uuid":"6c5a7838-821e-44e1-97e4-655b9bfa9f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6801","nonce":35148049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6803","nonce":35148049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6807","nonce":35148049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6805","nonce":35148049}]},"public_addr":"172.21.15.189:6801/35148049","cluster_addr":"172.21.15.189:6803/35148049","heartbeat_back_addr":"172.21.15.189:6807/35148049","heartbeat_front_addr":"172.21.15.189:6805/35148049","state":["exists","up"]},{"osd":2,"uuid":"8bc42f7f-39fa-4d3b-bc17-21c68f9231fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6801","nonce":897516976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6803","nonce":897516976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6807","nonce":897516976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6805","nonce":897516976}]},"public_addr":"172.21.15.193:6801/897516976","cluster_addr":"172.21.15.193:6803/897516976","heartbeat_back_addr":"172.21.15.193:6807/897516976","heartbeat_front_addr":"172.21.15.193:6805/897516976","state":["exists","up"]},{"osd":3,"uuid":"0473d055-0b79-434f-9658-28269d2b1fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6811","nonce":3457190917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6813","nonce":3457190917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6817","nonce":3457190917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6815","nonce":3457190917}]},"public_addr":"172.21.15.134:6811/3457190917","cluster_addr":"172.21.15.134:6813/3457190917","heartbeat_back_addr":"172.21.15.134:6817/3457190917","heartbeat_front_addr":"172.21.15.134:6815/3457190917","state":["exists","up"]},{"osd":4,"uuid":"1d70eaae-56cc-4cdc-8f83-b88d2121f0ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6809","nonce":1786881347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6811","nonce":1786881347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6815","nonce":1786881347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6813","nonce":1786881347}]},"public_addr":"172.21.15.189:6809/1786881347","cluster_addr":"172.21.15.189:6811/1786881347","heartbeat_back_addr":"172.21.15.189:6815/1786881347","heartbeat_front_addr":"172.21.15.189:6813/1786881347","state":["exists","up"]},{"osd":5,"uuid":"a9b0b4cf-1608-45ed-b64e-afcf545d97fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6809","nonce":1092388797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6811","nonce":1092388797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6815","nonce":1092388797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6813","nonce":1092388797}]},"public_addr":"172.21.15.193:6809/1092388797","cluster_addr":"172.21.15.193:6811/1092388797","heartbeat_back_addr":"172.21.15.193:6815/1092388797","heartbeat_front_addr":"172.21.15.193:6813/1092388797","state":["exists","up"]},{"osd":6,"uuid":"0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6819","nonce":2471505438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6821","nonce":2471505438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6825","nonce":2471505438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6823","nonce":2471505438}]},"public_addr":"172.21.15.134:6819/2471505438","cluster_addr":"172.21.15.134:6821/2471505438","heartbeat_back_addr":"172.21.15.134:6825/2471505438","heartbeat_front_addr":"172.21.15.134:6823/2471505438","state":["exists","up"]},{"osd":7,"uuid":"394764a7-1976-4223-a3c0-5979d941a838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6817","nonce":1382694911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6819","nonce":1382694911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6823","nonce":1382694911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6821","nonce":1382694911}]},"public_addr":"172.21.15.189:6817/1382694911","cluster_addr":"172.21.15.189:6819/1382694911","heartbeat_back_addr":"172.21.15.189:6823/1382694911","heartbeat_front_addr":"172.21.15.189:6821/1382694911","state":["exists","up"]},{"osd":8,"uuid":"1c218082-0b4c-41c8-92b9-453ae8d15a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6817","nonce":896348138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6819","nonce":896348138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6823","nonce":896348138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6821","nonce":896348138}]},"public_addr":"172.21.15.193:6817/896348138","cluster_addr":"172.21.15.193:6819/896348138","heartbeat_back_addr":"172.21.15.193:6823/896348138","heartbeat_front_addr":"172.21.15.193:6821/896348138","state":["exists","up"]},{"osd":9,"uuid":"48f39082-4fa9-4443-b5b7-b141c10d9e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6827","nonce":511237025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6829","nonce":511237025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6833","nonce":511237025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6831","nonce":511237025}]},"public_addr":"172.21.15.134:6827/511237025","cluster_addr":"172.21.15.134:6829/511237025","heartbeat_back_addr":"172.21.15.134:6833/511237025","heartbeat_front_addr":"172.21.15.134:6831/511237025","state":["exists","up"]},{"osd":10,"uuid":"95157766-1856-42f7-aea1-9cf83b77c8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6825","nonce":431740858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6827","nonce":431740858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6831","nonce":431740858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6829","nonce":431740858}]},"public_addr":"172.21.15.189:6825/431740858","cluster_addr":"172.21.15.189:6827/431740858","heartbeat_back_addr":"172.21.15.189:6831/431740858","heartbeat_front_addr":"172.21.15.189:6829/431740858","state":["exists","up"]},{"osd":11,"uuid":"244fadc7-f079-4736-8e7e-c0ce5ae6f43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6825","nonce":1594590579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6827","nonce":1594590579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6831","nonce":1594590579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6829","nonce":1594590579}]},"public_addr":"172.21.15.193:6825/1594590579","cluster_addr":"172.21.15.193:6827/1594590579","heartbeat_back_addr":"172.21.15.193:6831/1594590579","heartbeat_front_addr":"172.21.15.193:6829/1594590579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:11.712203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:35.845013+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:58.165609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:31.929746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:56.941555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:20.688071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:54.681514+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:20.789384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:44.568246+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:18.644310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:45.365351+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:09:09.663805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3790862248":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6800/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:0/2906927791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3068153391":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3056560791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6801/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/1310359922":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/2318666906":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/2901579347":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3404218719":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3598071145":"2024-07-27T11:02:33.711793+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:14:16.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.541+0000 7f19cb200640 1 -- 172.21.15.134:0/1567923830 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f19b00b3c20 msgr2=0x7f19b00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:16.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.541+0000 7f19cb200640 1 --2- 172.21.15.134:0/1567923830 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f19b00b3c20 0x7f19b00b6130 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f19bc006010 tx=0x7f19bc009820 comp rx=0 tx=0).stop 2024-07-26T11:14:16.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.541+0000 7f19cb200640 1 -- 172.21.15.134:0/1567923830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19cc108d40 msgr2=0x7f19cc19a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:16.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.542+0000 7f19cb200640 1 --2- 172.21.15.134:0/1567923830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19cc108d40 0x7f19cc19a2e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f19b400e9f0 tx=0x7f19b400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:14:16.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.542+0000 7f19cb200640 1 -- 172.21.15.134:0/1567923830 shutdown_connections 2024-07-26T11:14:16.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.542+0000 7f19cb200640 1 --2- 172.21.15.134:0/1567923830 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f19b00b3c20 0x7f19b00b6130 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:16.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.542+0000 7f19cb200640 1 --2- 172.21.15.134:0/1567923830 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f19cc109690 0x7f19cc19f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:16.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.542+0000 7f19cb200640 1 --2- 172.21.15.134:0/1567923830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19cc108d40 0x7f19cc19a2e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:16.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.542+0000 7f19cb200640 1 --2- 172.21.15.134:0/1567923830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19cc103400 0x7f19cc199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:16.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.542+0000 7f19cb200640 1 -- 172.21.15.134:0/1567923830 >> 172.21.15.134:0/1567923830 conn(0x7f19cc0fee90 msgr2=0x7f19cc1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:16.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.542+0000 7f19cb200640 1 -- 172.21.15.134:0/1567923830 shutdown_connections 2024-07-26T11:14:16.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:16.542+0000 7f19cb200640 1 -- 172.21.15.134:0/1567923830 wait complete. 2024-07-26T11:14:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:17 smithi134 ceph-mon[31966]: pgmap v429: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:17 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1567923830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:14:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:17 smithi189 ceph-mon[36790]: pgmap v429: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:17 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1567923830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:14:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:17 smithi193 ceph-mon[37473]: pgmap v429: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:17 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1567923830' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:14:17.592 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:14:18.106 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:19 smithi134 ceph-mon[31966]: pgmap v430: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:19 smithi189 ceph-mon[36790]: pgmap v430: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:19 smithi193 ceph-mon[37473]: pgmap v430: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:20.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.304+0000 7f7535c00640 1 -- 172.21.15.134:0/3296487885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7530107be0 msgr2=0x7f7530107ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:20.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.304+0000 7f7535c00640 1 --2- 172.21.15.134:0/3296487885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7530107be0 0x7f7530107ff0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f751c009900 tx=0x7f751c02a410 comp rx=0 tx=0).stop 2024-07-26T11:14:20.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.304+0000 7f7535c00640 1 -- 172.21.15.134:0/3296487885 shutdown_connections 2024-07-26T11:14:20.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.304+0000 7f7535c00640 1 --2- 172.21.15.134:0/3296487885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7530107be0 0x7f7530107ff0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:20.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.304+0000 7f7535c00640 1 --2- 172.21.15.134:0/3296487885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7530108530 0x7f753010ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:20.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.304+0000 7f7535c00640 1 --2- 172.21.15.134:0/3296487885 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75301022a0 0x7f75301026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:20.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.304+0000 7f7535c00640 1 -- 172.21.15.134:0/3296487885 >> 172.21.15.134:0/3296487885 conn(0x7f75300fdcf0 msgr2=0x7f7530100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:20.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.304+0000 7f7535c00640 1 -- 172.21.15.134:0/3296487885 shutdown_connections 2024-07-26T11:14:20.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.304+0000 7f7535c00640 1 -- 172.21.15.134:0/3296487885 wait complete. 2024-07-26T11:14:20.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.306+0000 7f7535c00640 1 Processor -- start 2024-07-26T11:14:20.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.306+0000 7f7535c00640 1 -- start start 2024-07-26T11:14:20.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.306+0000 7f7535c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75301022a0 0x7f7530199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:20.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.306+0000 7f7535c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7530107be0 0x7f753019a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:20.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.306+0000 7f7535c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7530108530 0x7f753019f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:20.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.306+0000 7f7535c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f75301121c0 con 0x7f7530108530 2024-07-26T11:14:20.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.306+0000 7f7535c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7530112040 con 0x7f7530107be0 2024-07-26T11:14:20.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.306+0000 7f7535c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7530112340 con 0x7f75301022a0 2024-07-26T11:14:20.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.307+0000 7f7535200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7530108530 0x7f753019f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:20.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.307+0000 7f752f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7530107be0 0x7f753019a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:20.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.307+0000 7f752fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75301022a0 0x7f7530199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:20.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.307+0000 7f752f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7530107be0 0x7f753019a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:49848/0 (socket says 172.21.15.134:49848) 2024-07-26T11:14:20.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.307+0000 7f752f400640 1 -- 172.21.15.134:0/3650419057 learned_addr learned my addr 172.21.15.134:0/3650419057 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:20.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.307+0000 7f7535200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7530108530 0x7f753019f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:55380/0 (socket says 172.21.15.134:55380) 2024-07-26T11:14:20.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.307+0000 7f752f400640 1 -- 172.21.15.134:0/3650419057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75301022a0 msgr2=0x7f7530199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:20.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.307+0000 7f752f400640 1 --2- 172.21.15.134:0/3650419057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75301022a0 0x7f7530199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:20.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.307+0000 7f752f400640 1 -- 172.21.15.134:0/3650419057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7530108530 msgr2=0x7f753019f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:20.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.307+0000 7f752f400640 1 --2- 172.21.15.134:0/3650419057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7530108530 0x7f753019f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:20.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.307+0000 7f752f400640 1 -- 172.21.15.134:0/3650419057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753019fcf0 con 0x7f7530107be0 2024-07-26T11:14:20.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.307+0000 7f752fe00640 1 --2- 172.21.15.134:0/3650419057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75301022a0 0x7f7530199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:20.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.308+0000 7f752f400640 1 --2- 172.21.15.134:0/3650419057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7530107be0 0x7f753019a320 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f751c02a930 tx=0x7f751c031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:20.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.308+0000 7f752cc00640 1 -- 172.21.15.134:0/3650419057 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c03f030 con 0x7f7530107be0 2024-07-26T11:14:20.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.308+0000 7f752cc00640 1 -- 172.21.15.134:0/3650419057 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f751c004490 con 0x7f7530107be0 2024-07-26T11:14:20.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.308+0000 7f752cc00640 1 -- 172.21.15.134:0/3650419057 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751c035070 con 0x7f7530107be0 2024-07-26T11:14:20.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.308+0000 7f7535c00640 1 -- 172.21.15.134:0/3650419057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f753019ff80 con 0x7f7530107be0 2024-07-26T11:14:20.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.308+0000 7f7535c00640 1 -- 172.21.15.134:0/3650419057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75301a6740 con 0x7f7530107be0 2024-07-26T11:14:20.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.309+0000 7f7535c00640 1 -- 172.21.15.134:0/3650419057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7530103520 con 0x7f7530107be0 2024-07-26T11:14:20.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.312+0000 7f752cc00640 1 -- 172.21.15.134:0/3650419057 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f751c042030 con 0x7f7530107be0 2024-07-26T11:14:20.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.313+0000 7f752cc00640 1 --2- 172.21.15.134:0/3650419057 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75140b3c20 0x7f75140b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:20.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.313+0000 7f752cc00640 1 -- 172.21.15.134:0/3650419057 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f751c104420 con 0x7f7530107be0 2024-07-26T11:14:20.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.313+0000 7f752fe00640 1 --2- 172.21.15.134:0/3650419057 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75140b3c20 0x7f75140b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:20.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.314+0000 7f752fe00640 1 --2- 172.21.15.134:0/3650419057 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75140b3c20 0x7f75140b6130 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7520009fc0 tx=0x7f7520005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:20.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.317+0000 7f752cc00640 1 -- 172.21.15.134:0/3650419057 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f751c0caf20 con 0x7f7530107be0 2024-07-26T11:14:20.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.562+0000 7f7535c00640 1 -- 172.21.15.134:0/3650419057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75301a6910 con 0x7f7530107be0 2024-07-26T11:14:20.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.563+0000 7f752cc00640 1 -- 172.21.15.134:0/3650419057 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11990 (secure 0 0 0) 0x7f751c0cf470 con 0x7f7530107be0 2024-07-26T11:14:20.565 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:14:20.566 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":29,"btime":"2024-07-26T11:11:55:972763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:11:55.128451+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24721,"mds_1":24719,"mds_2":24703,"mds_3":24688,"mds_4":24686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24635":{"gid":24635,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24641":{"gid":24641,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":32,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24719":{"gid":24719,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":28,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24721,"qdb_cluster":[24686,24688,24703,24719,24721]},"id":1}]} 2024-07-26T11:14:20.566 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 29 2024-07-26T11:14:20.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.567+0000 7f7535c00640 1 -- 172.21.15.134:0/3650419057 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75140b3c20 msgr2=0x7f75140b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:20.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.567+0000 7f7535c00640 1 --2- 172.21.15.134:0/3650419057 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75140b3c20 0x7f75140b6130 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7520009fc0 tx=0x7f7520005f90 comp rx=0 tx=0).stop 2024-07-26T11:14:20.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.567+0000 7f7535c00640 1 -- 172.21.15.134:0/3650419057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7530107be0 msgr2=0x7f753019a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:20.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.567+0000 7f7535c00640 1 --2- 172.21.15.134:0/3650419057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7530107be0 0x7f753019a320 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f751c02a930 tx=0x7f751c031a20 comp rx=0 tx=0).stop 2024-07-26T11:14:20.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.567+0000 7f7535c00640 1 -- 172.21.15.134:0/3650419057 shutdown_connections 2024-07-26T11:14:20.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.567+0000 7f7535c00640 1 --2- 172.21.15.134:0/3650419057 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75140b3c20 0x7f75140b6130 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:20.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.567+0000 7f7535c00640 1 --2- 172.21.15.134:0/3650419057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7530108530 0x7f753019f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:20.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.567+0000 7f7535c00640 1 --2- 172.21.15.134:0/3650419057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7530107be0 0x7f753019a320 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:20.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.567+0000 7f7535c00640 1 --2- 172.21.15.134:0/3650419057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75301022a0 0x7f7530199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:20.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.567+0000 7f7535c00640 1 -- 172.21.15.134:0/3650419057 >> 172.21.15.134:0/3650419057 conn(0x7f75300fdcf0 msgr2=0x7f7530100d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:20.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.567+0000 7f7535c00640 1 -- 172.21.15.134:0/3650419057 shutdown_connections 2024-07-26T11:14:20.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:20.567+0000 7f7535c00640 1 -- 172.21.15.134:0/3650419057 wait complete. 2024-07-26T11:14:21.315 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:21 smithi134 ceph-mon[31966]: pgmap v431: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:21.315 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:21 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3650419057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:14:21.330 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:21 smithi193 ceph-mon[37473]: pgmap v431: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:21.330 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:21 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3650419057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:14:21.382 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-26T11:10:25.500802+0000', 'modified': '2024-07-26T11:11:55.128451+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24721, 'mds_1': 24719, 'mds_2': 24703, 'mds_3': 24688, 'mds_4': 24686}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24635': {'gid': 24635, 'name': 'k', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6833/1897636383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6832', 'nonce': 1897636383}, {'type': 'v1', 'addr': '172.21.15.189:6833', 'nonce': 1897636383}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24637': {'gid': 24637, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6833/3093465661', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6832', 'nonce': 3093465661}, {'type': 'v1', 'addr': '172.21.15.193:6833', 'nonce': 3093465661}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24641': {'gid': 24641, 'name': 'j', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.134:6835/4071410587', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6834', 'nonce': 4071410587}, {'type': 'v1', 'addr': '172.21.15.134:6835', 'nonce': 4071410587}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24665': {'gid': 24665, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6835/1314042896', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 1314042896}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 1314042896}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24682': {'gid': 24682, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6835/97099551', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 97099551}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 97099551}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24686': {'gid': 24686, 'name': 'h', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 32, 'addr': '172.21.15.189:6837/1448732093', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6836', 'nonce': 1448732093}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 1448732093}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24688': {'gid': 24688, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 32, 'addr': '172.21.15.193:6837/1968168008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 1968168008}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 1968168008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24703': {'gid': 24703, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.193:6839/3985555366', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 3985555366}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 3985555366}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24719': {'gid': 24719, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.189:6839/790931242', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6838', 'nonce': 790931242}, {'type': 'v1', 'addr': '172.21.15.189:6839', 'nonce': 790931242}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24721': {'gid': 24721, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.134:6841/4067549550', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6840', 'nonce': 4067549550}, {'type': 'v1', 'addr': '172.21.15.134:6841', 'nonce': 4067549550}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24721, 'qdb_cluster': [24686, 24688, 24703, 24719, 24721]} 2024-07-26T11:14:21.382 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-07-26T11:14:21.383 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.h status 2024-07-26T11:14:21.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:21 smithi189 ceph-mon[36790]: pgmap v431: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:21.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:21 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3650419057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:14:21.834 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:22.237 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:22 smithi134 ceph-mon[31966]: pgmap v432: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:22 smithi189 ceph-mon[36790]: pgmap v432: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:22 smithi193 ceph-mon[37473]: pgmap v432: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:23.803 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:14:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:14:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:14:24.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.149+0000 7f4bdbc00640 1 -- 172.21.15.134:0/260638080 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bd4108ed0 msgr2=0x7f4bd41092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:24.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.149+0000 7f4bdbc00640 1 --2- 172.21.15.134:0/260638080 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bd4108ed0 0x7f4bd41092e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4bc0009860 tx=0x7f4bc002a340 comp rx=0 tx=0).stop 2024-07-26T11:14:24.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.149+0000 7f4bdbc00640 1 -- 172.21.15.134:0/260638080 shutdown_connections 2024-07-26T11:14:24.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.149+0000 7f4bdbc00640 1 --2- 172.21.15.134:0/260638080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bd4109820 0x7f4bd4110050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.149+0000 7f4bdbc00640 1 --2- 172.21.15.134:0/260638080 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bd4108ed0 0x7f4bd41092e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.149+0000 7f4bdbc00640 1 --2- 172.21.15.134:0/260638080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bd4103590 0x7f4bd41039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.149+0000 7f4bdbc00640 1 -- 172.21.15.134:0/260638080 >> 172.21.15.134:0/260638080 conn(0x7f4bd407b280 msgr2=0x7f4bd407b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:24.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.149+0000 7f4bdbc00640 1 -- 172.21.15.134:0/260638080 shutdown_connections 2024-07-26T11:14:24.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.149+0000 7f4bdbc00640 1 -- 172.21.15.134:0/260638080 wait complete. 2024-07-26T11:14:24.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.151+0000 7f4bdbc00640 1 Processor -- start 2024-07-26T11:14:24.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.151+0000 7f4bdbc00640 1 -- start start 2024-07-26T11:14:24.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.152+0000 7f4bdbc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bd4103590 0x7f4bd4075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.152+0000 7f4bdbc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bd4108ed0 0x7f4bd4075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.152+0000 7f4bdbc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bd4109820 0x7f4bd4075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.152+0000 7f4bdbc00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd4113490 con 0x7f4bd4103590 2024-07-26T11:14:24.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.152+0000 7f4bdbc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd4113310 con 0x7f4bd4108ed0 2024-07-26T11:14:24.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.152+0000 7f4bdbc00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd4113610 con 0x7f4bd4109820 2024-07-26T11:14:24.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.152+0000 7f4bd9200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bd4103590 0x7f4bd4075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.152+0000 7f4bd9200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bd4103590 0x7f4bd4075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:55394/0 (socket says 172.21.15.134:55394) 2024-07-26T11:14:24.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.152+0000 7f4bd9200640 1 -- 172.21.15.134:0/1940217550 learned_addr learned my addr 172.21.15.134:0/1940217550 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:24.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.152+0000 7f4bd9c00640 1 --2- 172.21.15.134:0/1940217550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bd4109820 0x7f4bd4075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.152+0000 7f4bd3e00640 1 --2- 172.21.15.134:0/1940217550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bd4108ed0 0x7f4bd4075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.153+0000 7f4bd9200640 1 -- 172.21.15.134:0/1940217550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bd4109820 msgr2=0x7f4bd4075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:24.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.153+0000 7f4bd9200640 1 --2- 172.21.15.134:0/1940217550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bd4109820 0x7f4bd4075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.153+0000 7f4bd9200640 1 -- 172.21.15.134:0/1940217550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bd4108ed0 msgr2=0x7f4bd4075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:24.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.153+0000 7f4bd9200640 1 --2- 172.21.15.134:0/1940217550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bd4108ed0 0x7f4bd4075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.153+0000 7f4bd9200640 1 -- 172.21.15.134:0/1940217550 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd41a81e0 con 0x7f4bd4103590 2024-07-26T11:14:24.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.153+0000 7f4bd9200640 1 --2- 172.21.15.134:0/1940217550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bd4103590 0x7f4bd4075140 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4bc800e9c0 tx=0x7f4bc800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:24.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.153+0000 7f4bd1600640 1 -- 172.21.15.134:0/1940217550 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc800ccc0 con 0x7f4bd4103590 2024-07-26T11:14:24.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.154+0000 7f4bd1600640 1 -- 172.21.15.134:0/1940217550 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bc80054d0 con 0x7f4bd4103590 2024-07-26T11:14:24.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.154+0000 7f4bdbc00640 1 -- 172.21.15.134:0/1940217550 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd41a84d0 con 0x7f4bd4103590 2024-07-26T11:14:24.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.154+0000 7f4bd1600640 1 -- 172.21.15.134:0/1940217550 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc8010430 con 0x7f4bd4103590 2024-07-26T11:14:24.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.154+0000 7f4bdbc00640 1 -- 172.21.15.134:0/1940217550 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd41a89e0 con 0x7f4bd4103590 2024-07-26T11:14:24.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.158+0000 7f4be2874740 1 Processor -- start 2024-07-26T11:14:24.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.158+0000 7f4be2874740 1 -- start start 2024-07-26T11:14:24.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.159+0000 7f4be2874740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bff1ff60 0x55b3c02c5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.159+0000 7f4be2874740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfef70a0 0x55b3c02c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.159+0000 7f4be2874740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bfeff280 0x55b3c02c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.159+0000 7f4be2874740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b3c023b970 con 0x55b3bfeff280 2024-07-26T11:14:24.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.159+0000 7f4be2874740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b3c022ae70 con 0x55b3bff1ff60 2024-07-26T11:14:24.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.159+0000 7f4be2874740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b3c022eeb0 con 0x55b3bfef70a0 2024-07-26T11:14:24.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.159+0000 7f4bd1600640 1 -- 172.21.15.134:0/1940217550 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f4bc8004f20 con 0x7f4bd4103590 2024-07-26T11:14:24.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.159+0000 7f4bd3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bfeff280 0x55b3c02c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.159+0000 7f4bd3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bfeff280 0x55b3c02c5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:55402/0 (socket says 172.21.15.134:55402) 2024-07-26T11:14:24.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.159+0000 7f4bd3e00640 1 -- 172.21.15.134:0/2582369719 learned_addr learned my addr 172.21.15.134:0/2582369719 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.159+0000 7f4bd9c00640 1 --2- 172.21.15.134:0/2582369719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bff1ff60 0x55b3c02c5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.160+0000 7f4bd9200640 1 --2- 172.21.15.134:0/2582369719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfef70a0 0x55b3c02c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.160+0000 7f4bd3e00640 1 -- 172.21.15.134:0/2582369719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfef70a0 msgr2=0x55b3c02c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.160+0000 7f4bd3e00640 1 --2- 172.21.15.134:0/2582369719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfef70a0 0x55b3c02c5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.160+0000 7f4bd3e00640 1 -- 172.21.15.134:0/2582369719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bff1ff60 msgr2=0x55b3c02c5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.160+0000 7f4bd3e00640 1 --2- 172.21.15.134:0/2582369719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bff1ff60 0x55b3c02c5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.160+0000 7f4bd3e00640 1 -- 172.21.15.134:0/2582369719 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3c01d2110 con 0x55b3bfeff280 2024-07-26T11:14:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.160+0000 7f4bd9200640 1 --2- 172.21.15.134:0/2582369719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfef70a0 0x55b3c02c5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:14:24.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.160+0000 7f4bd3e00640 1 --2- 172.21.15.134:0/2582369719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bfeff280 0x55b3c02c5ff0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4bc002a930 tx=0x7f4bc0005d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6abc187e1ae94426 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:24.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.161+0000 7f4bc6000640 1 -- 172.21.15.134:0/2582369719 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0041020 con 0x55b3bfeff280 2024-07-26T11:14:24.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.161+0000 7f4bc6000640 1 -- 172.21.15.134:0/2582369719 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bc0030450 con 0x55b3bfeff280 2024-07-26T11:14:24.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.161+0000 7f4bc6000640 1 -- 172.21.15.134:0/2582369719 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0037040 con 0x55b3bfeff280 2024-07-26T11:14:24.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.161+0000 7f4bd1600640 1 --2- 172.21.15.134:0/1940217550 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4bbc0b3c20 0x7f4bbc0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.162+0000 7f4bd1600640 1 -- 172.21.15.134:0/1940217550 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4bc80e4600 con 0x7f4bd4103590 2024-07-26T11:14:24.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.162+0000 7f4bd9c00640 1 --2- 172.21.15.134:0/1940217550 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4bbc0b3c20 0x7f4bbc0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.162+0000 7f4be2874740 1 -- 172.21.15.134:0/2582369719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bfeff280 msgr2=0x55b3c02c5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:24.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.162+0000 7f4be2874740 1 --2- 172.21.15.134:0/2582369719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bfeff280 0x55b3c02c5ff0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4bc002a930 tx=0x7f4bc0005d10 comp rx=0 tx=0).stop 2024-07-26T11:14:24.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.162+0000 7f4be2874740 1 -- 172.21.15.134:0/2582369719 shutdown_connections 2024-07-26T11:14:24.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.162+0000 7f4be2874740 1 --2- 172.21.15.134:0/2582369719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfef70a0 0x55b3c02c5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.162+0000 7f4be2874740 1 --2- 172.21.15.134:0/2582369719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bfeff280 0x55b3c02c5ff0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.162+0000 7f4be2874740 1 --2- 172.21.15.134:0/2582369719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bff1ff60 0x55b3c02c5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.162+0000 7f4be2874740 1 -- 172.21.15.134:0/2582369719 >> 172.21.15.134:0/2582369719 conn(0x55b3bff14350 msgr2=0x55b3c02c3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:24.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.163+0000 7f4be2874740 1 -- 172.21.15.134:0/2582369719 shutdown_connections 2024-07-26T11:14:24.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.163+0000 7f4bd9c00640 1 --2- 172.21.15.134:0/1940217550 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4bbc0b3c20 0x7f4bbc0b6130 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc005fd0 tx=0x7f4bcc009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:24.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.163+0000 7f4be2874740 1 -- 172.21.15.134:0/2582369719 wait complete. 2024-07-26T11:14:24.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.164+0000 7f4be2874740 1 -- start start 2024-07-26T11:14:24.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.164+0000 7f4be2874740 1 Processor -- start 2024-07-26T11:14:24.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.165+0000 7f4be2874740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bfef70a0 0x55b3c02d0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.165+0000 7f4be2874740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfeff280 0x55b3c02d12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.165+0000 7f4be2874740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bff1ff60 0x55b3c02dd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.165+0000 7f4be2874740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b3c008e5f0 con 0x55b3bff1ff60 2024-07-26T11:14:24.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.165+0000 7f4be2874740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b3c01c7540 con 0x55b3bfef70a0 2024-07-26T11:14:24.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.165+0000 7f4be2874740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b3c02d3cf0 con 0x55b3bfeff280 2024-07-26T11:14:24.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.165+0000 7f4be2874740 10 client.0 _ll_register_callbacks cb 0x55b3c01d6f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:14:24.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.165+0000 7f4bd9c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bff1ff60 0x55b3c02dd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.165+0000 7f4bd9c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bff1ff60 0x55b3c02dd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:55406/0 (socket says 172.21.15.134:55406) 2024-07-26T11:14:24.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.165+0000 7f4bd9c00640 1 -- 172.21.15.134:0/2138118903 learned_addr learned my addr 172.21.15.134:0/2138118903 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:24.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.165+0000 7f4bd9200640 1 --2- 172.21.15.134:0/2138118903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bfef70a0 0x55b3c02d0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.165+0000 7f4bd3e00640 1 --2- 172.21.15.134:0/2138118903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfeff280 0x55b3c02d12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.166+0000 7f4bd9200640 1 -- 172.21.15.134:0/2138118903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfeff280 msgr2=0x55b3c02d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:24.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.166+0000 7f4bd9200640 1 --2- 172.21.15.134:0/2138118903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfeff280 0x55b3c02d12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.166+0000 7f4bd9200640 1 -- 172.21.15.134:0/2138118903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bff1ff60 msgr2=0x55b3c02dd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:24.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.166+0000 7f4bd9200640 1 --2- 172.21.15.134:0/2138118903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bff1ff60 0x55b3c02dd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.166+0000 7f4bd9200640 1 -- 172.21.15.134:0/2138118903 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3bff8b540 con 0x55b3bfef70a0 2024-07-26T11:14:24.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.166+0000 7f4bd3e00640 1 --2- 172.21.15.134:0/2138118903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfeff280 0x55b3c02d12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:24.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.166+0000 7f4bd9c00640 1 --2- 172.21.15.134:0/2138118903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bff1ff60 0x55b3c02dd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:24.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.166+0000 7f4bd9200640 1 --2- 172.21.15.134:0/2138118903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bfef70a0 0x55b3c02d0d80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x55b3c02c41f0 tx=0x7f4bc80096b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=326289607cc8ce60 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:24.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.166+0000 7f4bbaa00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:14:24.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.166+0000 7f4bbaa00640 1 -- 172.21.15.134:0/2138118903 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc80ab750 con 0x55b3bfef70a0 2024-07-26T11:14:24.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.166+0000 7f4be2874740 1 -- 172.21.15.134:0/2138118903 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bc0030450 con 0x55b3bfef70a0 2024-07-26T11:14:24.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.166+0000 7f4bbaa00640 1 -- 172.21.15.134:0/2138118903 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4bc8005210 con 0x55b3bfef70a0 2024-07-26T11:14:24.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.167+0000 7f4bbaa00640 1 -- 172.21.15.134:0/2138118903 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4bc80ab050 con 0x55b3bfef70a0 2024-07-26T11:14:24.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.167+0000 7f4be2874740 10 client.25169 fetch_fsmap learned FSMap version 29 2024-07-26T11:14:24.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.167+0000 7f4be2874740 1 -- 172.21.15.134:0/2138118903 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b3c02dfb90 con 0x55b3bfef70a0 2024-07-26T11:14:24.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.167+0000 7f4bbaa00640 1 -- 172.21.15.134:0/2138118903 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bc0030450 con 0x55b3bfef70a0 2024-07-26T11:14:24.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.167+0000 7f4bbaa00640 1 -- 172.21.15.134:0/2138118903 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4bc8070220 con 0x55b3bfef70a0 2024-07-26T11:14:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.167+0000 7f4be2874740 10 client.25169 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:14:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.167+0000 7f4be2874740 10 client.25169 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T11:14:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.167+0000 7f4be2874740 20 client.25169 populate_metadata read hostname 'smithi134' 2024-07-26T11:14:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.167+0000 7f4be2874740 1 --2- 172.21.15.134:0/2138118903 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x55b3bfede8e0 0x55b3c02e3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.167+0000 7f4be2874740 4 client.25169 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:14:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.167+0000 7f4be2874740 1 -- 172.21.15.134:0/2138118903 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3c023d8c0 con 0x55b3bfede8e0 2024-07-26T11:14:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.168+0000 7f4bd3e00640 1 --2- 172.21.15.134:0/2138118903 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x55b3bfede8e0 0x55b3c02e3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.168+0000 7f4bbaa00640 10 client.25169 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:14:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.168+0000 7f4bd3e00640 1 --2- 172.21.15.134:0/2138118903 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x55b3bfede8e0 0x55b3c02e3630 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e1e4b147d2d68f03 server_cookie=515eb9077e11acbb in_seq=0 out_seq=0 2024-07-26T11:14:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.169+0000 7f4bbaa00640 1 -- 172.21.15.134:0/2138118903 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b3c023d8c0 con 0x55b3bfede8e0 2024-07-26T11:14:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.169+0000 7f4bbaa00640 10 client.25169 handle_command_reply: tid=0 2024-07-26T11:14:24.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.169+0000 7f4be2874740 1 client.25169 shutdown 2024-07-26T11:14:24.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.170+0000 7f4be2874740 1 -- 172.21.15.134:0/2138118903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bfef70a0 msgr2=0x55b3c02d0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:24.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.170+0000 7f4be2874740 1 --2- 172.21.15.134:0/2138118903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bfef70a0 0x55b3c02d0d80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x55b3c02c41f0 tx=0x7f4bc80096b0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.170+0000 7f4be2874740 1 -- 172.21.15.134:0/2138118903 shutdown_connections 2024-07-26T11:14:24.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.170+0000 7f4be2874740 1 --2- 172.21.15.134:0/2138118903 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x55b3bfede8e0 0x55b3c02e3630 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.170+0000 7f4be2874740 1 --2- 172.21.15.134:0/2138118903 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfeff280 0x55b3c02d12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.170+0000 7f4be2874740 1 --2- 172.21.15.134:0/2138118903 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bff1ff60 0x55b3c02dd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.170+0000 7f4be2874740 1 --2- 172.21.15.134:0/2138118903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bfef70a0 0x55b3c02d0d80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.170+0000 7f4be2874740 1 -- 172.21.15.134:0/2138118903 >> 172.21.15.134:0/2138118903 conn(0x55b3bff14350 msgr2=0x55b3c02c8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:24.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.170+0000 7f4bbaa00640 0 client.25169 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:14:24.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.170+0000 7f4be2874740 1 -- 172.21.15.134:0/2138118903 shutdown_connections 2024-07-26T11:14:24.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.170+0000 7f4be2874740 1 -- 172.21.15.134:0/2138118903 wait complete. 2024-07-26T11:14:24.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.170+0000 7f4be2874740 20 client.25169 trim_cache size 0 max 16384 2024-07-26T11:14:24.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.190+0000 7f4be2874740 1 Processor -- start 2024-07-26T11:14:24.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.190+0000 7f4be2874740 1 -- start start 2024-07-26T11:14:24.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.190+0000 7f4be2874740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfef70a0 0x55b3c02e1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.190+0000 7f4be2874740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bfeff280 0x55b3c02ddbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.190+0000 7f4be2874740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bff1ff60 0x55b3c02cb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.190+0000 7f4be2874740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b3c02e1970 con 0x55b3bfeff280 2024-07-26T11:14:24.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.190+0000 7f4be2874740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b3c02e2040 con 0x55b3bff1ff60 2024-07-26T11:14:24.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.190+0000 7f4be2874740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b3c02e2710 con 0x55b3bfef70a0 2024-07-26T11:14:24.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.191+0000 7f4bd3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bfeff280 0x55b3c02ddbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.191+0000 7f4bd9200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfef70a0 0x55b3c02e1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.191+0000 7f4bd9200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfef70a0 0x55b3c02e1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:55758/0 (socket says 172.21.15.134:55758) 2024-07-26T11:14:24.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.191+0000 7f4bd9c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bff1ff60 0x55b3c02cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.194 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.191+0000 7f4bd9200640 1 -- 172.21.15.134:0/603330061 learned_addr learned my addr 172.21.15.134:0/603330061 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:24.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.191+0000 7f4bd9200640 1 -- 172.21.15.134:0/603330061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bff1ff60 msgr2=0x55b3c02cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:24.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.191+0000 7f4bd9200640 1 --2- 172.21.15.134:0/603330061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bff1ff60 0x55b3c02cb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.191+0000 7f4bd9200640 1 -- 172.21.15.134:0/603330061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bfeff280 msgr2=0x55b3c02ddbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:24.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.191+0000 7f4bd9200640 1 --2- 172.21.15.134:0/603330061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bfeff280 0x55b3c02ddbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.191+0000 7f4bd9200640 1 -- 172.21.15.134:0/603330061 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3c02cf280 con 0x55b3bfef70a0 2024-07-26T11:14:24.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.192+0000 7f4bd9200640 1 --2- 172.21.15.134:0/603330061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfef70a0 0x55b3c02e1af0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4bc80a7080 tx=0x7f4bc80e4880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc0498bdb3802ef9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:24.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.192+0000 7f4bc5600640 1 -- 172.21.15.134:0/603330061 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc80ab750 con 0x55b3bfef70a0 2024-07-26T11:14:24.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.192+0000 7f4bc5600640 1 -- 172.21.15.134:0/603330061 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bc80b4100 con 0x55b3bfef70a0 2024-07-26T11:14:24.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.192+0000 7f4bc5600640 1 -- 172.21.15.134:0/603330061 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc80a6b10 con 0x55b3bfef70a0 2024-07-26T11:14:24.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.193+0000 7f4be2874740 1 -- 172.21.15.134:0/603330061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfef70a0 msgr2=0x55b3c02e1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:24.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.193+0000 7f4be2874740 1 --2- 172.21.15.134:0/603330061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfef70a0 0x55b3c02e1af0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4bc80a7080 tx=0x7f4bc80e4880 comp rx=0 tx=0).stop 2024-07-26T11:14:24.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.193+0000 7f4be2874740 1 -- 172.21.15.134:0/603330061 shutdown_connections 2024-07-26T11:14:24.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.193+0000 7f4be2874740 1 --2- 172.21.15.134:0/603330061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bff1ff60 0x55b3c02cb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.193+0000 7f4be2874740 1 --2- 172.21.15.134:0/603330061 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bfeff280 0x55b3c02ddbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.193+0000 7f4be2874740 1 --2- 172.21.15.134:0/603330061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfef70a0 0x55b3c02e1af0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.193+0000 7f4be2874740 1 -- 172.21.15.134:0/603330061 >> 172.21.15.134:0/603330061 conn(0x55b3bff14350 msgr2=0x55b3c02d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:24.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.193+0000 7f4be2874740 1 -- 172.21.15.134:0/603330061 shutdown_connections 2024-07-26T11:14:24.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.193+0000 7f4be2874740 1 -- 172.21.15.134:0/603330061 wait complete. 2024-07-26T11:14:24.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.194+0000 7f4be2874740 1 -- start start 2024-07-26T11:14:24.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.194+0000 7f4be2874740 1 Processor -- start 2024-07-26T11:14:24.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.194+0000 7f4be2874740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bfef70a0 0x55b3c02e49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.194+0000 7f4be2874740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfeff280 0x55b3c02e4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.194+0000 7f4be2874740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bff1ff60 0x55b3c02eb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.194+0000 7f4be2874740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b3c02e34b0 con 0x55b3bff1ff60 2024-07-26T11:14:24.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.194+0000 7f4be2874740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b3c02e2de0 con 0x55b3bfef70a0 2024-07-26T11:14:24.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.194+0000 7f4be2874740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b3c02e3b80 con 0x55b3bfeff280 2024-07-26T11:14:24.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.194+0000 7f4be2874740 10 client.0 _ll_register_callbacks cb 0x7f4bc80aaf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:14:24.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.195+0000 7f4bd3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfeff280 0x55b3c02e4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.195+0000 7f4bd9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bfef70a0 0x55b3c02e49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.195+0000 7f4bd3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfeff280 0x55b3c02e4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:55768/0 (socket says 172.21.15.134:55768) 2024-07-26T11:14:24.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.195+0000 7f4bd3e00640 1 -- 172.21.15.134:0/1857655143 learned_addr learned my addr 172.21.15.134:0/1857655143 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:24.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.195+0000 7f4bd9c00640 1 --2- 172.21.15.134:0/1857655143 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bff1ff60 0x55b3c02eb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.195+0000 7f4bd3e00640 1 -- 172.21.15.134:0/1857655143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bfef70a0 msgr2=0x55b3c02e49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:24.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.195+0000 7f4bd3e00640 1 --2- 172.21.15.134:0/1857655143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bfef70a0 0x55b3c02e49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.195+0000 7f4bd3e00640 1 -- 172.21.15.134:0/1857655143 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bff1ff60 msgr2=0x55b3c02eb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:24.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.195+0000 7f4bd3e00640 1 --2- 172.21.15.134:0/1857655143 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bff1ff60 0x55b3c02eb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.195+0000 7f4bd3e00640 1 -- 172.21.15.134:0/1857655143 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3c02ebd70 con 0x55b3bfeff280 2024-07-26T11:14:24.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.195+0000 7f4bd9200640 1 --2- 172.21.15.134:0/1857655143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bfef70a0 0x55b3c02e49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:24.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.196+0000 7f4bd3e00640 1 --2- 172.21.15.134:0/1857655143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfeff280 0x55b3c02e4f10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4bc002a900 tx=0x7f4bc00042c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=243703d41edd3fba server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:24.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.196+0000 7f4bbaa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:14:24.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.196+0000 7f4bbaa00640 1 -- 172.21.15.134:0/1857655143 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc0041040 con 0x55b3bfeff280 2024-07-26T11:14:24.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.196+0000 7f4bbaa00640 1 -- 172.21.15.134:0/1857655143 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4bc0005250 con 0x55b3bfeff280 2024-07-26T11:14:24.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.196+0000 7f4be2874740 1 -- 172.21.15.134:0/1857655143 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4bc80b4100 con 0x55b3bfeff280 2024-07-26T11:14:24.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.196+0000 7f4bbaa00640 1 -- 172.21.15.134:0/1857655143 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4bc0037040 con 0x55b3bfeff280 2024-07-26T11:14:24.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.196+0000 7f4be2874740 10 client.25186 fetch_fsmap learned FSMap version 29 2024-07-26T11:14:24.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.196+0000 7f4be2874740 1 -- 172.21.15.134:0/1857655143 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b3c034b0a0 con 0x55b3bfeff280 2024-07-26T11:14:24.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.196+0000 7f4bbaa00640 1 -- 172.21.15.134:0/1857655143 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4bc80b4100 con 0x55b3bfeff280 2024-07-26T11:14:24.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.197+0000 7f4bbaa00640 1 -- 172.21.15.134:0/1857655143 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f4bc0004c90 con 0x55b3bfeff280 2024-07-26T11:14:24.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.197+0000 7f4be2874740 10 client.25186 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:14:24.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.197+0000 7f4be2874740 10 client.25186 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T11:14:24.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.197+0000 7f4be2874740 20 client.25186 populate_metadata read hostname 'smithi134' 2024-07-26T11:14:24.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.197+0000 7f4be2874740 1 --2- 172.21.15.134:0/1857655143 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x55b3bfeb3c90 0x55b3c034e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:24.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.197+0000 7f4be2874740 4 client.25186 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:14:24.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.197+0000 7f4be2874740 1 -- 172.21.15.134:0/1857655143 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "status"}) -- 0x55b3c0239360 con 0x55b3bfeb3c90 2024-07-26T11:14:24.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.197+0000 7f4bd9200640 1 --2- 172.21.15.134:0/1857655143 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x55b3bfeb3c90 0x55b3c034e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:24.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.198+0000 7f4bd9200640 1 --2- 172.21.15.134:0/1857655143 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x55b3bfeb3c90 0x55b3c034e4f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=71be805d5cc2b08 server_cookie=b0e3891d90f27da9 in_seq=0 out_seq=0 2024-07-26T11:14:24.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.198+0000 7f4bbaa00640 10 client.25186 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:14:24.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.198+0000 7f4bbaa00640 1 -- 172.21.15.134:0/1857655143 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b3c0239360 con 0x55b3bfeb3c90 2024-07-26T11:14:24.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.198+0000 7f4bbaa00640 10 client.25186 handle_command_reply: tid=0 2024-07-26T11:14:24.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.198+0000 7f4be2874740 1 client.25186 shutdown 2024-07-26T11:14:24.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.198+0000 7f4be2874740 1 -- 172.21.15.134:0/1857655143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfeff280 msgr2=0x55b3c02e4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:24.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.198+0000 7f4be2874740 1 --2- 172.21.15.134:0/1857655143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfeff280 0x55b3c02e4f10 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4bc002a900 tx=0x7f4bc00042c0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.198+0000 7f4be2874740 1 -- 172.21.15.134:0/1857655143 shutdown_connections 2024-07-26T11:14:24.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.198+0000 7f4be2874740 1 --2- 172.21.15.134:0/1857655143 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x55b3bfeb3c90 0x55b3c034e4f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.198+0000 7f4be2874740 1 --2- 172.21.15.134:0/1857655143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b3bfeff280 0x55b3c02e4f10 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.198+0000 7f4be2874740 1 --2- 172.21.15.134:0/1857655143 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b3bff1ff60 0x55b3c02eb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.198+0000 7f4be2874740 1 --2- 172.21.15.134:0/1857655143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b3bfef70a0 0x55b3c02e49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.198+0000 7f4bbaa00640 0 client.25186 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:14:24.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.198+0000 7f4be2874740 1 -- 172.21.15.134:0/1857655143 >> 172.21.15.134:0/1857655143 conn(0x55b3bff14350 msgr2=0x55b3c02d12e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:24.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.199+0000 7f4be2874740 1 -- 172.21.15.134:0/1857655143 shutdown_connections 2024-07-26T11:14:24.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.199+0000 7f4be2874740 1 -- 172.21.15.134:0/1857655143 wait complete. 2024-07-26T11:14:24.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.199+0000 7f4be2874740 20 client.25186 trim_cache size 0 max 16384 2024-07-26T11:14:24.204 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:14:24.204 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_fsid": "105fbe5c-4b3e-11ef-bcb7-c7b262605968", 2024-07-26T11:14:24.204 INFO:teuthology.orchestra.run.smithi134.stdout: "whoami": 4, 2024-07-26T11:14:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: "id": 24686, 2024-07-26T11:14:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: "want_state": "up:active", 2024-07-26T11:14:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "up:active", 2024-07-26T11:14:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: "fs_name": "cephfs", 2024-07-26T11:14:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: "rank_uptime": 154.12648386999999, 2024-07-26T11:14:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: "mdsmap_epoch": 29, 2024-07-26T11:14:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch": 90, 2024-07-26T11:14:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T11:14:24.205 INFO:teuthology.orchestra.run.smithi134.stdout: "uptime": 275.75070849600002 2024-07-26T11:14:24.205 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:14:24.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.200+0000 7f4bdbc00640 1 -- 172.21.15.134:0/1940217550 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4bbc0b3c20 msgr2=0x7f4bbc0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:24.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.200+0000 7f4bdbc00640 1 --2- 172.21.15.134:0/1940217550 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4bbc0b3c20 0x7f4bbc0b6130 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc005fd0 tx=0x7f4bcc009500 comp rx=0 tx=0).stop 2024-07-26T11:14:24.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.200+0000 7f4bdbc00640 1 -- 172.21.15.134:0/1940217550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bd4103590 msgr2=0x7f4bd4075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:24.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.200+0000 7f4bdbc00640 1 --2- 172.21.15.134:0/1940217550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bd4103590 0x7f4bd4075140 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4bc800e9c0 tx=0x7f4bc800edc0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.200+0000 7f4bdbc00640 1 -- 172.21.15.134:0/1940217550 shutdown_connections 2024-07-26T11:14:24.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.200+0000 7f4bdbc00640 1 --2- 172.21.15.134:0/1940217550 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4bbc0b3c20 0x7f4bbc0b6130 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.200+0000 7f4bdbc00640 1 --2- 172.21.15.134:0/1940217550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bd4109820 0x7f4bd4075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.200+0000 7f4bdbc00640 1 --2- 172.21.15.134:0/1940217550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4bd4108ed0 0x7f4bd4075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.200+0000 7f4bdbc00640 1 --2- 172.21.15.134:0/1940217550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4bd4103590 0x7f4bd4075140 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:24.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.200+0000 7f4bdbc00640 1 -- 172.21.15.134:0/1940217550 >> 172.21.15.134:0/1940217550 conn(0x7f4bd407b280 msgr2=0x7f4bd4107070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:24.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.200+0000 7f4bdbc00640 1 -- 172.21.15.134:0/1940217550 shutdown_connections 2024-07-26T11:14:24.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:24.200+0000 7f4bdbc00640 1 -- 172.21.15.134:0/1940217550 wait complete. 2024-07-26T11:14:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:24 smithi134 ceph-mon[31966]: pgmap v433: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:14:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:14:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:14:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:24 smithi189 ceph-mon[36790]: pgmap v433: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:14:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:14:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:14:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:24 smithi193 ceph-mon[37473]: pgmap v433: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:14:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:14:25.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:14:25.148 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.c status 2024-07-26T11:14:25.611 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:27.331 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:27 smithi134 ceph-mon[31966]: pgmap v434: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:27 smithi189 ceph-mon[36790]: pgmap v434: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:27 smithi193 ceph-mon[37473]: pgmap v434: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:27.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.877+0000 7f9bd4a00640 1 -- 172.21.15.134:0/819583187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd010f330 msgr2=0x7f9bd010f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:27.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.877+0000 7f9bd4a00640 1 --2- 172.21.15.134:0/819583187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd010f330 0x7f9bd010f740 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8009860 tx=0x7f9bb802a340 comp rx=0 tx=0).stop 2024-07-26T11:14:27.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.877+0000 7f9bd4a00640 1 -- 172.21.15.134:0/819583187 shutdown_connections 2024-07-26T11:14:27.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.877+0000 7f9bd4a00640 1 --2- 172.21.15.134:0/819583187 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bd010fc80 0x7f9bd01164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.877+0000 7f9bd4a00640 1 --2- 172.21.15.134:0/819583187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd010f330 0x7f9bd010f740 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.877+0000 7f9bd4a00640 1 --2- 172.21.15.134:0/819583187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9bd01099f0 0x7f9bd0109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.877+0000 7f9bd4a00640 1 -- 172.21.15.134:0/819583187 >> 172.21.15.134:0/819583187 conn(0x7f9bd0078ff0 msgr2=0x7f9bd0107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:27.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.877+0000 7f9bd4a00640 1 -- 172.21.15.134:0/819583187 shutdown_connections 2024-07-26T11:14:27.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.877+0000 7f9bd4a00640 1 -- 172.21.15.134:0/819583187 wait complete. 2024-07-26T11:14:27.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.879+0000 7f9bd4a00640 1 Processor -- start 2024-07-26T11:14:27.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.879+0000 7f9bd4a00640 1 -- start start 2024-07-26T11:14:27.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.879+0000 7f9bd4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd01099f0 0x7f9bd01a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.879+0000 7f9bd4a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9bd010f330 0x7f9bd01a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.879+0000 7f9bd4a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bd010fc80 0x7f9bd01a5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.879+0000 7f9bd4a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd0119940 con 0x7f9bd010fc80 2024-07-26T11:14:27.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.879+0000 7f9bd4a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd01197c0 con 0x7f9bd01099f0 2024-07-26T11:14:27.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.879+0000 7f9bd4a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd0119ac0 con 0x7f9bd010f330 2024-07-26T11:14:27.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.880+0000 7f9bcf400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd01099f0 0x7f9bd01a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.880+0000 7f9bcf400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd01099f0 0x7f9bd01a02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:49914/0 (socket says 172.21.15.134:49914) 2024-07-26T11:14:27.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.880+0000 7f9bcf400640 1 -- 172.21.15.134:0/2793319089 learned_addr learned my addr 172.21.15.134:0/2793319089 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:27.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.880+0000 7f9bc7e00640 1 --2- 172.21.15.134:0/2793319089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9bd010f330 0x7f9bd01a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.880+0000 7f9bcfe00640 1 --2- 172.21.15.134:0/2793319089 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bd010fc80 0x7f9bd01a5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.880+0000 7f9bcf400640 1 -- 172.21.15.134:0/2793319089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9bd010f330 msgr2=0x7f9bd01a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:27.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.880+0000 7f9bcf400640 1 --2- 172.21.15.134:0/2793319089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9bd010f330 0x7f9bd01a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.880+0000 7f9bcf400640 1 -- 172.21.15.134:0/2793319089 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bd010fc80 msgr2=0x7f9bd01a5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:27.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.880+0000 7f9bcf400640 1 --2- 172.21.15.134:0/2793319089 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bd010fc80 0x7f9bd01a5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.880+0000 7f9bcf400640 1 -- 172.21.15.134:0/2793319089 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd01a6260 con 0x7f9bd01099f0 2024-07-26T11:14:27.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.881+0000 7f9bcf400640 1 --2- 172.21.15.134:0/2793319089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd01099f0 0x7f9bd01a02f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000e9c0 tx=0x7f9bc000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:27.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.881+0000 7f9bccc00640 1 -- 172.21.15.134:0/2793319089 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc000ccc0 con 0x7f9bd01099f0 2024-07-26T11:14:27.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.881+0000 7f9bccc00640 1 -- 172.21.15.134:0/2793319089 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bc00054d0 con 0x7f9bd01099f0 2024-07-26T11:14:27.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.881+0000 7f9bccc00640 1 -- 172.21.15.134:0/2793319089 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0010430 con 0x7f9bd01099f0 2024-07-26T11:14:27.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.881+0000 7f9bd4a00640 1 -- 172.21.15.134:0/2793319089 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd01a6550 con 0x7f9bd01099f0 2024-07-26T11:14:27.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.881+0000 7f9bd4a00640 1 -- 172.21.15.134:0/2793319089 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd01accb0 con 0x7f9bd01099f0 2024-07-26T11:14:27.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.884+0000 7f9bdcc05740 1 Processor -- start 2024-07-26T11:14:27.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.884+0000 7f9bdcc05740 1 -- start start 2024-07-26T11:14:27.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.885+0000 7f9bdcc05740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd32bf60 0x5588bd6d1780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.885+0000 7f9bdcc05740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd3030a0 0x5588bd6d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.885+0000 7f9bdcc05740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd30b280 0x5588bd6d2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.885+0000 7f9bdcc05740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5588bd647970 con 0x5588bd32bf60 2024-07-26T11:14:27.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.885+0000 7f9bdcc05740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5588bd636e70 con 0x5588bd3030a0 2024-07-26T11:14:27.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.885+0000 7f9bdcc05740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5588bd63aeb0 con 0x5588bd30b280 2024-07-26T11:14:27.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.885+0000 7f9bccc00640 1 -- 172.21.15.134:0/2793319089 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f9bc0004f80 con 0x7f9bd01099f0 2024-07-26T11:14:27.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.885+0000 7f9bcfe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd32bf60 0x5588bd6d1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.885+0000 7f9bcfe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd32bf60 0x5588bd6d1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:55446/0 (socket says 172.21.15.134:55446) 2024-07-26T11:14:27.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.885+0000 7f9bcfe00640 1 -- 172.21.15.134:0/1851272075 learned_addr learned my addr 172.21.15.134:0/1851272075 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:27.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.885+0000 7f9bc7e00640 1 --2- 172.21.15.134:0/1851272075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd30b280 0x5588bd6d2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.885+0000 7f9bcf400640 1 --2- 172.21.15.134:0/1851272075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd3030a0 0x5588bd6d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.886+0000 7f9bc7e00640 1 -- 172.21.15.134:0/1851272075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd3030a0 msgr2=0x5588bd6d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:27.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.886+0000 7f9bc7e00640 1 --2- 172.21.15.134:0/1851272075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd3030a0 0x5588bd6d1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.886+0000 7f9bc7e00640 1 -- 172.21.15.134:0/1851272075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd32bf60 msgr2=0x5588bd6d1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:27.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.886+0000 7f9bc7e00640 1 --2- 172.21.15.134:0/1851272075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd32bf60 0x5588bd6d1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.886+0000 7f9bc7e00640 1 -- 172.21.15.134:0/1851272075 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588bd5de110 con 0x5588bd30b280 2024-07-26T11:14:27.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.886+0000 7f9bc7e00640 1 --2- 172.21.15.134:0/1851272075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd30b280 0x5588bd6d2200 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9bb802a860 tx=0x7f9bb80095a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c7a5456b4870024 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:27.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.886+0000 7f9bbfe00640 1 -- 172.21.15.134:0/1851272075 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8041070 con 0x5588bd30b280 2024-07-26T11:14:27.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.887+0000 7f9bbfe00640 1 -- 172.21.15.134:0/1851272075 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bb80044d0 con 0x5588bd30b280 2024-07-26T11:14:27.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.887+0000 7f9bbfe00640 1 -- 172.21.15.134:0/1851272075 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8037040 con 0x5588bd30b280 2024-07-26T11:14:27.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.887+0000 7f9bccc00640 1 --2- 172.21.15.134:0/2793319089 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9bb40b3ca0 0x7f9bb40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.887+0000 7f9bccc00640 1 -- 172.21.15.134:0/2793319089 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f9bc00e4580 con 0x7f9bd01099f0 2024-07-26T11:14:27.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.887+0000 7f9bcfe00640 1 --2- 172.21.15.134:0/2793319089 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9bb40b3ca0 0x7f9bb40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.888+0000 7f9bdcc05740 1 -- 172.21.15.134:0/1851272075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd30b280 msgr2=0x5588bd6d2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:27.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.888+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/1851272075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd30b280 0x5588bd6d2200 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9bb802a860 tx=0x7f9bb80095a0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.888+0000 7f9bdcc05740 1 -- 172.21.15.134:0/1851272075 shutdown_connections 2024-07-26T11:14:27.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.888+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/1851272075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd30b280 0x5588bd6d2200 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.888+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/1851272075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd3030a0 0x5588bd6d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.888+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/1851272075 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd32bf60 0x5588bd6d1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.888+0000 7f9bdcc05740 1 -- 172.21.15.134:0/1851272075 >> 172.21.15.134:0/1851272075 conn(0x5588bd320350 msgr2=0x5588bd6cf670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:27.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.888+0000 7f9bdcc05740 1 -- 172.21.15.134:0/1851272075 shutdown_connections 2024-07-26T11:14:27.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.888+0000 7f9bcfe00640 1 --2- 172.21.15.134:0/2793319089 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9bb40b3ca0 0x7f9bb40b61b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8003bc0 tx=0x7f9bc8009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:27.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.888+0000 7f9bdcc05740 1 -- 172.21.15.134:0/1851272075 wait complete. 2024-07-26T11:14:27.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.889+0000 7f9bdcc05740 1 -- start start 2024-07-26T11:14:27.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.889+0000 7f9bdcc05740 1 Processor -- start 2024-07-26T11:14:27.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.889+0000 7f9bdcc05740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd3030a0 0x5588bd6dd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.889+0000 7f9bdcc05740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd30b280 0x5588bd6dd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.889+0000 7f9bdcc05740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd32bf60 0x5588bd6e91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.889+0000 7f9bdcc05740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5588bd49a5f0 con 0x5588bd32bf60 2024-07-26T11:14:27.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.889+0000 7f9bdcc05740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5588bd5d3540 con 0x5588bd30b280 2024-07-26T11:14:27.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.890+0000 7f9bdcc05740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5588bd6dff00 con 0x5588bd3030a0 2024-07-26T11:14:27.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.890+0000 7f9bdcc05740 10 client.0 _ll_register_callbacks cb 0x5588bd5e2f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:14:27.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.890+0000 7f9bcf400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd3030a0 0x5588bd6dd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.890+0000 7f9bc7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd30b280 0x5588bd6dd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.890+0000 7f9bcfe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd32bf60 0x5588bd6e91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.890+0000 7f9bc7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd30b280 0x5588bd6dd750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:49926/0 (socket says 172.21.15.134:49926) 2024-07-26T11:14:27.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.890+0000 7f9bc7e00640 1 -- 172.21.15.134:0/2410696873 learned_addr learned my addr 172.21.15.134:0/2410696873 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:27.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.890+0000 7f9bcf400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd3030a0 0x5588bd6dd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:55800/0 (socket says 172.21.15.134:55800) 2024-07-26T11:14:27.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.890+0000 7f9bc7e00640 1 -- 172.21.15.134:0/2410696873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd3030a0 msgr2=0x5588bd6dd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:27.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.890+0000 7f9bc7e00640 1 --2- 172.21.15.134:0/2410696873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd3030a0 0x5588bd6dd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.890+0000 7f9bc7e00640 1 -- 172.21.15.134:0/2410696873 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd32bf60 msgr2=0x5588bd6e91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:27.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.890+0000 7f9bc7e00640 1 --2- 172.21.15.134:0/2410696873 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd32bf60 0x5588bd6e91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.890+0000 7f9bc7e00640 1 -- 172.21.15.134:0/2410696873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588bd397540 con 0x5588bd30b280 2024-07-26T11:14:27.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.890+0000 7f9bcf400640 1 --2- 172.21.15.134:0/2410696873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd3030a0 0x5588bd6dd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:27.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.891+0000 7f9bc7e00640 1 --2- 172.21.15.134:0/2410696873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd30b280 0x5588bd6dd750 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8035040 tx=0x7f9bb8004020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c44d296e3aa907c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:27.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.891+0000 7f9bbcc00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:14:27.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.891+0000 7f9bbcc00640 1 -- 172.21.15.134:0/2410696873 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8037030 con 0x5588bd30b280 2024-07-26T11:14:27.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.891+0000 7f9bdcc05740 1 -- 172.21.15.134:0/2410696873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bb80044d0 con 0x5588bd30b280 2024-07-26T11:14:27.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.891+0000 7f9bbcc00640 1 -- 172.21.15.134:0/2410696873 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f9bb8030650 con 0x5588bd30b280 2024-07-26T11:14:27.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.891+0000 7f9bbcc00640 1 -- 172.21.15.134:0/2410696873 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9bb8054050 con 0x5588bd30b280 2024-07-26T11:14:27.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.891+0000 7f9bdcc05740 10 client.25187 fetch_fsmap learned FSMap version 29 2024-07-26T11:14:27.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.891+0000 7f9bdcc05740 1 -- 172.21.15.134:0/2410696873 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5588bd6ebdf0 con 0x5588bd30b280 2024-07-26T11:14:27.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.892+0000 7f9bbcc00640 1 -- 172.21.15.134:0/2410696873 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bb80044d0 con 0x5588bd30b280 2024-07-26T11:14:27.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.892+0000 7f9bbcc00640 1 -- 172.21.15.134:0/2410696873 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9bb804f030 con 0x5588bd30b280 2024-07-26T11:14:27.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.892+0000 7f9bdcc05740 10 client.25187 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:14:27.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.892+0000 7f9bdcc05740 10 client.25187 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T11:14:27.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.892+0000 7f9bdcc05740 20 client.25187 populate_metadata read hostname 'smithi134' 2024-07-26T11:14:27.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.892+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/2410696873 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5588bd2ea8e0 0x5588bd6ef370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.892+0000 7f9bdcc05740 4 client.25187 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:14:27.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.892+0000 7f9bdcc05740 1 -- 172.21.15.134:0/2410696873 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5588bd6498c0 con 0x5588bd2ea8e0 2024-07-26T11:14:27.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.893+0000 7f9bcf400640 1 --2- 172.21.15.134:0/2410696873 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5588bd2ea8e0 0x5588bd6ef370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.893+0000 7f9bcf400640 1 --2- 172.21.15.134:0/2410696873 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5588bd2ea8e0 0x5588bd6ef370 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ef030f3227acc1c1 server_cookie=a580282a6b9c12bd in_seq=0 out_seq=0 2024-07-26T11:14:27.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.893+0000 7f9bbcc00640 10 client.25187 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:14:27.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.894+0000 7f9bbcc00640 1 -- 172.21.15.134:0/2410696873 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5588bd6498c0 con 0x5588bd2ea8e0 2024-07-26T11:14:27.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.894+0000 7f9bbcc00640 10 client.25187 handle_command_reply: tid=0 2024-07-26T11:14:27.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.894+0000 7f9bdcc05740 1 client.25187 shutdown 2024-07-26T11:14:27.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.894+0000 7f9bdcc05740 1 -- 172.21.15.134:0/2410696873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd30b280 msgr2=0x5588bd6dd750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:27.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.894+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/2410696873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd30b280 0x5588bd6dd750 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8035040 tx=0x7f9bb8004020 comp rx=0 tx=0).stop 2024-07-26T11:14:27.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.894+0000 7f9bdcc05740 1 -- 172.21.15.134:0/2410696873 shutdown_connections 2024-07-26T11:14:27.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.894+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/2410696873 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5588bd2ea8e0 0x5588bd6ef370 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.894+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/2410696873 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd32bf60 0x5588bd6e91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.894+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/2410696873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd30b280 0x5588bd6dd750 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.894+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/2410696873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd3030a0 0x5588bd6dd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.894+0000 7f9bbcc00640 0 client.25187 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:14:27.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.894+0000 7f9bdcc05740 1 -- 172.21.15.134:0/2410696873 >> 172.21.15.134:0/2410696873 conn(0x5588bd320350 msgr2=0x5588bd6d4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:27.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.895+0000 7f9bdcc05740 1 -- 172.21.15.134:0/2410696873 shutdown_connections 2024-07-26T11:14:27.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.895+0000 7f9bdcc05740 1 -- 172.21.15.134:0/2410696873 wait complete. 2024-07-26T11:14:27.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.895+0000 7f9bdcc05740 20 client.25187 trim_cache size 0 max 16384 2024-07-26T11:14:27.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.914+0000 7f9bdcc05740 1 Processor -- start 2024-07-26T11:14:27.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.915+0000 7f9bdcc05740 1 -- start start 2024-07-26T11:14:27.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.915+0000 7f9bdcc05740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd3030a0 0x5588bd6ef370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.915+0000 7f9bdcc05740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd30b280 0x5588bd6d78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.915+0000 7f9bdcc05740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd32bf60 0x5588bd6dc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.915+0000 7f9bdcc05740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5588bd6ed6b0 con 0x5588bd30b280 2024-07-26T11:14:27.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.915+0000 7f9bdcc05740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5588bd6edd80 con 0x5588bd32bf60 2024-07-26T11:14:27.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.915+0000 7f9bdcc05740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5588bd6ee450 con 0x5588bd3030a0 2024-07-26T11:14:27.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.915+0000 7f9bc7e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd30b280 0x5588bd6d78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.915+0000 7f9bc7e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd30b280 0x5588bd6d78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:55460/0 (socket says 172.21.15.134:55460) 2024-07-26T11:14:27.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.915+0000 7f9bcfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd32bf60 0x5588bd6dc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.915+0000 7f9bc7e00640 1 -- 172.21.15.134:0/318873077 learned_addr learned my addr 172.21.15.134:0/318873077 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:27.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.915+0000 7f9bcf400640 1 --2- 172.21.15.134:0/318873077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd3030a0 0x5588bd6ef370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.916+0000 7f9bcfe00640 1 -- 172.21.15.134:0/318873077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd3030a0 msgr2=0x5588bd6ef370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:27.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.916+0000 7f9bcfe00640 1 --2- 172.21.15.134:0/318873077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd3030a0 0x5588bd6ef370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.916+0000 7f9bcfe00640 1 -- 172.21.15.134:0/318873077 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd30b280 msgr2=0x5588bd6d78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:27.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.916+0000 7f9bcfe00640 1 --2- 172.21.15.134:0/318873077 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd30b280 0x5588bd6d78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.916+0000 7f9bcfe00640 1 -- 172.21.15.134:0/318873077 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5588bd6ca7f0 con 0x5588bd32bf60 2024-07-26T11:14:27.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.916+0000 7f9bcfe00640 1 --2- 172.21.15.134:0/318873077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd32bf60 0x5588bd6dc520 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8009fd0 tx=0x7f9bc8019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=771beae8b748e4c6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:27.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.916+0000 7f9bbf400640 1 -- 172.21.15.134:0/318873077 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8011500 con 0x5588bd32bf60 2024-07-26T11:14:27.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.917+0000 7f9bbf400640 1 -- 172.21.15.134:0/318873077 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bc8011b50 con 0x5588bd32bf60 2024-07-26T11:14:27.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.917+0000 7f9bbf400640 1 -- 172.21.15.134:0/318873077 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc801cba0 con 0x5588bd32bf60 2024-07-26T11:14:27.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.917+0000 7f9bdcc05740 1 -- 172.21.15.134:0/318873077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd32bf60 msgr2=0x5588bd6dc520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:27.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.917+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/318873077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd32bf60 0x5588bd6dc520 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8009fd0 tx=0x7f9bc8019000 comp rx=0 tx=0).stop 2024-07-26T11:14:27.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.917+0000 7f9bdcc05740 1 -- 172.21.15.134:0/318873077 shutdown_connections 2024-07-26T11:14:27.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.917+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/318873077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd32bf60 0x5588bd6dc520 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.917+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/318873077 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd30b280 0x5588bd6d78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.917+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/318873077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd3030a0 0x5588bd6ef370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.917+0000 7f9bdcc05740 1 -- 172.21.15.134:0/318873077 >> 172.21.15.134:0/318873077 conn(0x5588bd320350 msgr2=0x5588bd6cee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:27.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.918+0000 7f9bdcc05740 1 -- 172.21.15.134:0/318873077 shutdown_connections 2024-07-26T11:14:27.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.918+0000 7f9bdcc05740 1 -- 172.21.15.134:0/318873077 wait complete. 2024-07-26T11:14:27.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.918+0000 7f9bdcc05740 1 -- start start 2024-07-26T11:14:27.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.918+0000 7f9bdcc05740 1 Processor -- start 2024-07-26T11:14:27.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.919+0000 7f9bdcc05740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd3030a0 0x5588bd6f0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.919+0000 7f9bdcc05740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd30b280 0x5588bd6f0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.919+0000 7f9bdcc05740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd32bf60 0x5588bd6f5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.919+0000 7f9bdcc05740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5588bd6ef1f0 con 0x5588bd3030a0 2024-07-26T11:14:27.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.919+0000 7f9bdcc05740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5588bd6eeb20 con 0x5588bd32bf60 2024-07-26T11:14:27.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.919+0000 7f9bdcc05740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5588bd6ef8c0 con 0x5588bd30b280 2024-07-26T11:14:27.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.919+0000 7f9bdcc05740 10 client.0 _ll_register_callbacks cb 0x7f9bc0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:14:27.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.919+0000 7f9bcf400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd3030a0 0x5588bd6f0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.919+0000 7f9bcf400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd3030a0 0x5588bd6f0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:55464/0 (socket says 172.21.15.134:55464) 2024-07-26T11:14:27.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.919+0000 7f9bcfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd32bf60 0x5588bd6f5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.919+0000 7f9bcf400640 1 -- 172.21.15.134:0/2712352277 learned_addr learned my addr 172.21.15.134:0/2712352277 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:27.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.919+0000 7f9bc7e00640 1 --2- 172.21.15.134:0/2712352277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd30b280 0x5588bd6f0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.920+0000 7f9bcf400640 1 -- 172.21.15.134:0/2712352277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd30b280 msgr2=0x5588bd6f0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:27.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.920+0000 7f9bcf400640 1 --2- 172.21.15.134:0/2712352277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd30b280 0x5588bd6f0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.920+0000 7f9bcf400640 1 -- 172.21.15.134:0/2712352277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd32bf60 msgr2=0x5588bd6f5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:27.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.920+0000 7f9bcf400640 1 --2- 172.21.15.134:0/2712352277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd32bf60 0x5588bd6f5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.920+0000 7f9bcf400640 1 -- 172.21.15.134:0/2712352277 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5588bd6d45e0 con 0x5588bd3030a0 2024-07-26T11:14:27.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.920+0000 7f9bc7e00640 1 --2- 172.21.15.134:0/2712352277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd30b280 0x5588bd6f0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:27.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.920+0000 7f9bcfe00640 1 --2- 172.21.15.134:0/2712352277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd32bf60 0x5588bd6f5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:27.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.920+0000 7f9bcf400640 1 --2- 172.21.15.134:0/2712352277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd3030a0 0x5588bd6f0920 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00a7050 tx=0x7f9bc0004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c85a39e817d4dd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:27.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.920+0000 7f9bbcc00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:14:27.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.920+0000 7f9bbcc00640 1 -- 172.21.15.134:0/2712352277 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc00aaae0 con 0x5588bd3030a0 2024-07-26T11:14:27.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.921+0000 7f9bbcc00640 1 -- 172.21.15.134:0/2712352277 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f9bc0005270 con 0x5588bd3030a0 2024-07-26T11:14:27.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.921+0000 7f9bdcc05740 1 -- 172.21.15.134:0/2712352277 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9bc8011b50 con 0x5588bd3030a0 2024-07-26T11:14:27.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.921+0000 7f9bbcc00640 1 -- 172.21.15.134:0/2712352277 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9bc00ab6d0 con 0x5588bd3030a0 2024-07-26T11:14:27.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.921+0000 7f9bdcc05740 10 client.15048 fetch_fsmap learned FSMap version 29 2024-07-26T11:14:27.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.921+0000 7f9bdcc05740 1 -- 172.21.15.134:0/2712352277 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5588bd6f77e0 con 0x5588bd3030a0 2024-07-26T11:14:27.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.921+0000 7f9bbcc00640 1 -- 172.21.15.134:0/2712352277 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9bc8011b50 con 0x5588bd3030a0 2024-07-26T11:14:27.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.921+0000 7f9bbcc00640 1 -- 172.21.15.134:0/2712352277 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9bc00b3bf0 con 0x5588bd3030a0 2024-07-26T11:14:27.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.921+0000 7f9bdcc05740 10 client.15048 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:14:27.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.921+0000 7f9bdcc05740 10 client.15048 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T11:14:27.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.921+0000 7f9bdcc05740 20 client.15048 populate_metadata read hostname 'smithi134' 2024-07-26T11:14:27.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.921+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/2712352277 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5588bd2bfc90 0x5588bd75b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:27.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.922+0000 7f9bdcc05740 4 client.15048 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:14:27.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.922+0000 7f9bdcc05740 1 -- 172.21.15.134:0/2712352277 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "status"}) -- 0x5588bd645360 con 0x5588bd2bfc90 2024-07-26T11:14:27.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.922+0000 7f9bc7e00640 1 --2- 172.21.15.134:0/2712352277 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5588bd2bfc90 0x5588bd75b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:27.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.922+0000 7f9bc7e00640 1 --2- 172.21.15.134:0/2712352277 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5588bd2bfc90 0x5588bd75b270 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8d9db676a3f7be8f server_cookie=caec683e21b13c8f in_seq=0 out_seq=0 2024-07-26T11:14:27.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.922+0000 7f9bbcc00640 10 client.15048 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:14:27.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.923+0000 7f9bbcc00640 1 -- 172.21.15.134:0/2712352277 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5588bd645360 con 0x5588bd2bfc90 2024-07-26T11:14:27.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.923+0000 7f9bbcc00640 10 client.15048 handle_command_reply: tid=0 2024-07-26T11:14:27.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.923+0000 7f9bdcc05740 1 client.15048 shutdown 2024-07-26T11:14:27.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.923+0000 7f9bdcc05740 1 -- 172.21.15.134:0/2712352277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd3030a0 msgr2=0x5588bd6f0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:27.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.923+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/2712352277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd3030a0 0x5588bd6f0920 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00a7050 tx=0x7f9bc0004a20 comp rx=0 tx=0).stop 2024-07-26T11:14:27.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.923+0000 7f9bdcc05740 1 -- 172.21.15.134:0/2712352277 shutdown_connections 2024-07-26T11:14:27.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.923+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/2712352277 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5588bd2bfc90 0x5588bd75b270 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.923+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/2712352277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588bd30b280 0x5588bd6f0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.923+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/2712352277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588bd32bf60 0x5588bd6f5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.923+0000 7f9bdcc05740 1 --2- 172.21.15.134:0/2712352277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5588bd3030a0 0x5588bd6f0920 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.923+0000 7f9bdcc05740 1 -- 172.21.15.134:0/2712352277 >> 172.21.15.134:0/2712352277 conn(0x5588bd320350 msgr2=0x5588bd6cdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:27.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.923+0000 7f9bbcc00640 0 client.15048 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:14:27.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.923+0000 7f9bdcc05740 1 -- 172.21.15.134:0/2712352277 shutdown_connections 2024-07-26T11:14:27.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.923+0000 7f9bdcc05740 1 -- 172.21.15.134:0/2712352277 wait complete. 2024-07-26T11:14:27.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.924+0000 7f9bdcc05740 20 client.15048 trim_cache size 0 max 16384 2024-07-26T11:14:27.928 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:14:27.928 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_fsid": "105fbe5c-4b3e-11ef-bcb7-c7b262605968", 2024-07-26T11:14:27.929 INFO:teuthology.orchestra.run.smithi134.stdout: "whoami": 3, 2024-07-26T11:14:27.929 INFO:teuthology.orchestra.run.smithi134.stdout: "id": 24688, 2024-07-26T11:14:27.929 INFO:teuthology.orchestra.run.smithi134.stdout: "want_state": "up:active", 2024-07-26T11:14:27.929 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "up:active", 2024-07-26T11:14:27.929 INFO:teuthology.orchestra.run.smithi134.stdout: "fs_name": "cephfs", 2024-07-26T11:14:27.929 INFO:teuthology.orchestra.run.smithi134.stdout: "rank_uptime": 158.86934455799999, 2024-07-26T11:14:27.929 INFO:teuthology.orchestra.run.smithi134.stdout: "mdsmap_epoch": 29, 2024-07-26T11:14:27.929 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch": 90, 2024-07-26T11:14:27.929 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T11:14:27.929 INFO:teuthology.orchestra.run.smithi134.stdout: "uptime": 282.15103308200003 2024-07-26T11:14:27.929 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:14:27.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.925+0000 7f9bd4a00640 1 -- 172.21.15.134:0/2793319089 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9bb40b3ca0 msgr2=0x7f9bb40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:27.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.925+0000 7f9bd4a00640 1 --2- 172.21.15.134:0/2793319089 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9bb40b3ca0 0x7f9bb40b61b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8003bc0 tx=0x7f9bc8009500 comp rx=0 tx=0).stop 2024-07-26T11:14:27.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.925+0000 7f9bd4a00640 1 -- 172.21.15.134:0/2793319089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd01099f0 msgr2=0x7f9bd01a02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:27.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.925+0000 7f9bd4a00640 1 --2- 172.21.15.134:0/2793319089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd01099f0 0x7f9bd01a02f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000e9c0 tx=0x7f9bc000ee80 comp rx=0 tx=0).stop 2024-07-26T11:14:27.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.925+0000 7f9bd4a00640 1 -- 172.21.15.134:0/2793319089 shutdown_connections 2024-07-26T11:14:27.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.925+0000 7f9bd4a00640 1 --2- 172.21.15.134:0/2793319089 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9bb40b3ca0 0x7f9bb40b61b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.925+0000 7f9bd4a00640 1 --2- 172.21.15.134:0/2793319089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9bd010f330 0x7f9bd01a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.925+0000 7f9bd4a00640 1 --2- 172.21.15.134:0/2793319089 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9bd010fc80 0x7f9bd01a5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.925+0000 7f9bd4a00640 1 --2- 172.21.15.134:0/2793319089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd01099f0 0x7f9bd01a02f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:27.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.925+0000 7f9bd4a00640 1 -- 172.21.15.134:0/2793319089 >> 172.21.15.134:0/2793319089 conn(0x7f9bd0078ff0 msgr2=0x7f9bd00797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:27.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.925+0000 7f9bd4a00640 1 -- 172.21.15.134:0/2793319089 shutdown_connections 2024-07-26T11:14:27.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:27.925+0000 7f9bd4a00640 1 -- 172.21.15.134:0/2793319089 wait complete. 2024-07-26T11:14:28.813 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.i status 2024-07-26T11:14:29.284 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:29 smithi134 ceph-mon[31966]: pgmap v435: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:29 smithi189 ceph-mon[36790]: pgmap v435: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:29 smithi193 ceph-mon[37473]: pgmap v435: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:31.319 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:31 smithi134 ceph-mon[31966]: pgmap v436: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:31 smithi189 ceph-mon[36790]: pgmap v436: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:31 smithi193 ceph-mon[37473]: pgmap v436: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:31.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.937+0000 7f178f200640 1 -- 172.21.15.134:0/2644845467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f179010b860 msgr2=0x7f1790112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:31.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.937+0000 7f178f200640 1 --2- 172.21.15.134:0/2644845467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f179010b860 0x7f1790112090 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1788009be0 tx=0x7f178802a6c0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.938+0000 7f178f200640 1 -- 172.21.15.134:0/2644845467 shutdown_connections 2024-07-26T11:14:31.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.938+0000 7f178f200640 1 --2- 172.21.15.134:0/2644845467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f179010af10 0x7f179010b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.938+0000 7f178f200640 1 --2- 172.21.15.134:0/2644845467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f179010b860 0x7f1790112090 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.938+0000 7f178f200640 1 --2- 172.21.15.134:0/2644845467 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17901055d0 0x7f17901059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.938+0000 7f178f200640 1 -- 172.21.15.134:0/2644845467 >> 172.21.15.134:0/2644845467 conn(0x7f1790101040 msgr2=0x7f1790103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:31.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.938+0000 7f178f200640 1 -- 172.21.15.134:0/2644845467 shutdown_connections 2024-07-26T11:14:31.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.938+0000 7f178f200640 1 -- 172.21.15.134:0/2644845467 wait complete. 2024-07-26T11:14:31.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.939+0000 7f178f200640 1 Processor -- start 2024-07-26T11:14:31.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.939+0000 7f178f200640 1 -- start start 2024-07-26T11:14:31.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.939+0000 7f178f200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17901055d0 0x7f179019be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.939+0000 7f178f200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f179010af10 0x7f179019c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.940+0000 7f178f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f179010b860 0x7f17901a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.940+0000 7f178f200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1790115510 con 0x7f17901055d0 2024-07-26T11:14:31.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.940+0000 7f178f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1790115390 con 0x7f179010b860 2024-07-26T11:14:31.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.940+0000 7f178f200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1790115690 con 0x7f179010af10 2024-07-26T11:14:31.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.940+0000 7f178de00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17901055d0 0x7f179019be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.940+0000 7f178de00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17901055d0 0x7f179019be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57818/0 (socket says 172.21.15.134:57818) 2024-07-26T11:14:31.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.940+0000 7f178de00640 1 -- 172.21.15.134:0/509784404 learned_addr learned my addr 172.21.15.134:0/509784404 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:31.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.940+0000 7f1787e00640 1 --2- 172.21.15.134:0/509784404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f179010af10 0x7f179019c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.940+0000 7f178e800640 1 --2- 172.21.15.134:0/509784404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f179010b860 0x7f17901a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.940+0000 7f178de00640 1 -- 172.21.15.134:0/509784404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f179010af10 msgr2=0x7f179019c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:31.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.940+0000 7f178de00640 1 --2- 172.21.15.134:0/509784404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f179010af10 0x7f179019c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.940+0000 7f178de00640 1 -- 172.21.15.134:0/509784404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f179010b860 msgr2=0x7f17901a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:31.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.940+0000 7f178de00640 1 --2- 172.21.15.134:0/509784404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f179010b860 0x7f17901a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.940+0000 7f178de00640 1 -- 172.21.15.134:0/509784404 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17901a1df0 con 0x7f17901055d0 2024-07-26T11:14:31.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.940+0000 7f1787e00640 1 --2- 172.21.15.134:0/509784404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f179010af10 0x7f179019c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:31.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.941+0000 7f178de00640 1 --2- 172.21.15.134:0/509784404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17901055d0 0x7f179019be80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f178000ea60 tx=0x7f178000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:31.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.941+0000 7f1786a00640 1 -- 172.21.15.134:0/509784404 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f178000cde0 con 0x7f17901055d0 2024-07-26T11:14:31.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.941+0000 7f1786a00640 1 -- 172.21.15.134:0/509784404 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17800035f0 con 0x7f17901055d0 2024-07-26T11:14:31.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.941+0000 7f178f200640 1 -- 172.21.15.134:0/509784404 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17901a20e0 con 0x7f17901055d0 2024-07-26T11:14:31.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.941+0000 7f1786a00640 1 -- 172.21.15.134:0/509784404 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1780010690 con 0x7f17901055d0 2024-07-26T11:14:31.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.941+0000 7f178f200640 1 -- 172.21.15.134:0/509784404 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17901a8840 con 0x7f17901055d0 2024-07-26T11:14:31.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.945+0000 7f179bff0740 1 Processor -- start 2024-07-26T11:14:31.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.945+0000 7f179bff0740 1 -- start start 2024-07-26T11:14:31.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.945+0000 7f1786a00640 1 -- 172.21.15.134:0/509784404 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f17800037a0 con 0x7f17901055d0 2024-07-26T11:14:31.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.945+0000 7f179bff0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5632f60 0x5556d59d8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.946+0000 7f179bff0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d560a0a0 0x5556d59d8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.946+0000 7f179bff0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5612280 0x5556d59d91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.946+0000 7f179bff0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5556d594e950 con 0x5556d5632f60 2024-07-26T11:14:31.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.946+0000 7f179bff0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5556d593de50 con 0x5556d5612280 2024-07-26T11:14:31.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.946+0000 7f179bff0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5556d5941e90 con 0x5556d560a0a0 2024-07-26T11:14:31.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.946+0000 7f178de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d560a0a0 0x5556d59d8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.946+0000 7f178e800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5632f60 0x5556d59d8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.946+0000 7f178de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d560a0a0 0x5556d59d8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:60188/0 (socket says 172.21.15.134:60188) 2024-07-26T11:14:31.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.946+0000 7f178de00640 1 -- 172.21.15.134:0/1460159974 learned_addr learned my addr 172.21.15.134:0/1460159974 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:31.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.946+0000 7f1787e00640 1 --2- 172.21.15.134:0/1460159974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5612280 0x5556d59d91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.946+0000 7f178de00640 1 -- 172.21.15.134:0/1460159974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5612280 msgr2=0x5556d59d91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:31.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.946+0000 7f178de00640 1 --2- 172.21.15.134:0/1460159974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5612280 0x5556d59d91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.946+0000 7f178de00640 1 -- 172.21.15.134:0/1460159974 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5632f60 msgr2=0x5556d59d8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:31.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.947+0000 7f178de00640 1 --2- 172.21.15.134:0/1460159974 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5632f60 0x5556d59d8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.947+0000 7f178de00640 1 -- 172.21.15.134:0/1460159974 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556d58e50f0 con 0x5556d560a0a0 2024-07-26T11:14:31.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.947+0000 7f1787e00640 1 --2- 172.21.15.134:0/1460159974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5612280 0x5556d59d91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:31.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.947+0000 7f178de00640 1 --2- 172.21.15.134:0/1460159974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d560a0a0 0x5556d59d8ca0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f17800ec380 tx=0x7f17800ec840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=532fa5416d57a55c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:31.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.947+0000 7f177f400640 1 -- 172.21.15.134:0/1460159974 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17800e6520 con 0x5556d560a0a0 2024-07-26T11:14:31.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.947+0000 7f177f400640 1 -- 172.21.15.134:0/1460159974 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17800e4c20 con 0x5556d560a0a0 2024-07-26T11:14:31.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.947+0000 7f177f400640 1 -- 172.21.15.134:0/1460159974 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17800f3c70 con 0x5556d560a0a0 2024-07-26T11:14:31.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.948+0000 7f1786a00640 1 --2- 172.21.15.134:0/509784404 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f17740b3ca0 0x7f17740b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.948+0000 7f1786a00640 1 -- 172.21.15.134:0/509784404 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f17800e4380 con 0x7f17901055d0 2024-07-26T11:14:31.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.948+0000 7f178e800640 1 --2- 172.21.15.134:0/509784404 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f17740b3ca0 0x7f17740b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.948+0000 7f179bff0740 1 -- 172.21.15.134:0/1460159974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d560a0a0 msgr2=0x5556d59d8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:31.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.948+0000 7f179bff0740 1 --2- 172.21.15.134:0/1460159974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d560a0a0 0x5556d59d8ca0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f17800ec380 tx=0x7f17800ec840 comp rx=0 tx=0).stop 2024-07-26T11:14:31.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.949+0000 7f179bff0740 1 -- 172.21.15.134:0/1460159974 shutdown_connections 2024-07-26T11:14:31.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.949+0000 7f179bff0740 1 --2- 172.21.15.134:0/1460159974 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d560a0a0 0x5556d59d8ca0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.949+0000 7f179bff0740 1 --2- 172.21.15.134:0/1460159974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5612280 0x5556d59d91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.949+0000 7f179bff0740 1 --2- 172.21.15.134:0/1460159974 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5632f60 0x5556d59d8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.949+0000 7f179bff0740 1 -- 172.21.15.134:0/1460159974 >> 172.21.15.134:0/1460159974 conn(0x5556d5627350 msgr2=0x5556d59d6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:31.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.949+0000 7f178e800640 1 --2- 172.21.15.134:0/509784404 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f17740b3ca0 0x7f17740b61b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1788009370 tx=0x7f1788004a40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:31.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.949+0000 7f179bff0740 1 -- 172.21.15.134:0/1460159974 shutdown_connections 2024-07-26T11:14:31.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.949+0000 7f179bff0740 1 -- 172.21.15.134:0/1460159974 wait complete. 2024-07-26T11:14:31.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.949+0000 7f179bff0740 1 -- start start 2024-07-26T11:14:31.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.950+0000 7f179bff0740 1 Processor -- start 2024-07-26T11:14:31.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.950+0000 7f179bff0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d560a0a0 0x5556d59e3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.950+0000 7f179bff0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5612280 0x5556d59e4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.950+0000 7f179bff0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d5632f60 0x5556d59f0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.950+0000 7f179bff0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5556d57a15f0 con 0x5556d560a0a0 2024-07-26T11:14:31.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.950+0000 7f179bff0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5556d58da520 con 0x5556d5612280 2024-07-26T11:14:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.950+0000 7f179bff0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5556d59e6f40 con 0x5556d5632f60 2024-07-26T11:14:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.950+0000 7f179bff0740 10 client.0 _ll_register_callbacks cb 0x5556d58e9ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:14:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.951+0000 7f178de00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d560a0a0 0x5556d59e3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.951+0000 7f178de00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d560a0a0 0x5556d59e3fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57850/0 (socket says 172.21.15.134:57850) 2024-07-26T11:14:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.951+0000 7f178de00640 1 -- 172.21.15.134:0/165204648 learned_addr learned my addr 172.21.15.134:0/165204648 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.951+0000 7f1787e00640 1 --2- 172.21.15.134:0/165204648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5612280 0x5556d59e4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.951+0000 7f178e800640 1 --2- 172.21.15.134:0/165204648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d5632f60 0x5556d59f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.951+0000 7f1787e00640 1 -- 172.21.15.134:0/165204648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d5632f60 msgr2=0x5556d59f0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.951+0000 7f1787e00640 1 --2- 172.21.15.134:0/165204648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d5632f60 0x5556d59f0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.951+0000 7f1787e00640 1 -- 172.21.15.134:0/165204648 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d560a0a0 msgr2=0x5556d59e3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.951+0000 7f1787e00640 1 --2- 172.21.15.134:0/165204648 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d560a0a0 0x5556d59e3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.951+0000 7f1787e00640 1 -- 172.21.15.134:0/165204648 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556d569e540 con 0x5556d5612280 2024-07-26T11:14:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.951+0000 7f178de00640 1 --2- 172.21.15.134:0/165204648 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d560a0a0 0x5556d59e3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.952+0000 7f1787e00640 1 --2- 172.21.15.134:0/165204648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5612280 0x5556d59e4540 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1778009530 tx=0x7f177800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=437c365e91bb5ea3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.952+0000 7f1773e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:14:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.952+0000 7f1773e00640 1 -- 172.21.15.134:0/165204648 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17780120d0 con 0x5556d5612280 2024-07-26T11:14:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.952+0000 7f179bff0740 1 -- 172.21.15.134:0/165204648 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17800e4c20 con 0x5556d5612280 2024-07-26T11:14:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.952+0000 7f1773e00640 1 -- 172.21.15.134:0/165204648 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f1778010430 con 0x5556d5612280 2024-07-26T11:14:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.952+0000 7f1773e00640 1 -- 172.21.15.134:0/165204648 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1778012270 con 0x5556d5612280 2024-07-26T11:14:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.952+0000 7f179bff0740 10 client.25205 fetch_fsmap learned FSMap version 29 2024-07-26T11:14:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.952+0000 7f179bff0740 1 -- 172.21.15.134:0/165204648 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5556d59f2e80 con 0x5556d5612280 2024-07-26T11:14:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.953+0000 7f1773e00640 1 -- 172.21.15.134:0/165204648 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17800e4c20 con 0x5556d5612280 2024-07-26T11:14:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.953+0000 7f1773e00640 1 -- 172.21.15.134:0/165204648 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f177801c6f0 con 0x5556d5612280 2024-07-26T11:14:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.953+0000 7f179bff0740 10 client.25205 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:14:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.953+0000 7f179bff0740 10 client.25205 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T11:14:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.953+0000 7f179bff0740 20 client.25205 populate_metadata read hostname 'smithi134' 2024-07-26T11:14:31.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.953+0000 7f179bff0740 1 --2- 172.21.15.134:0/165204648 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x5556d55f18e0 0x5556d59f68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.953+0000 7f179bff0740 4 client.25205 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:14:31.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.953+0000 7f179bff0740 1 -- 172.21.15.134:0/165204648 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556d59508a0 con 0x5556d55f18e0 2024-07-26T11:14:31.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.954+0000 7f178de00640 1 --2- 172.21.15.134:0/165204648 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x5556d55f18e0 0x5556d59f68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.954+0000 7f178de00640 1 --2- 172.21.15.134:0/165204648 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x5556d55f18e0 0x5556d59f68c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=23c0784cd3329395 server_cookie=d52e42304b415eae in_seq=0 out_seq=0 2024-07-26T11:14:31.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.954+0000 7f1773e00640 10 client.25205 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:14:31.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.955+0000 7f1773e00640 1 -- 172.21.15.134:0/165204648 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5556d59508a0 con 0x5556d55f18e0 2024-07-26T11:14:31.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.955+0000 7f1773e00640 10 client.25205 handle_command_reply: tid=0 2024-07-26T11:14:31.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.955+0000 7f179bff0740 1 client.25205 shutdown 2024-07-26T11:14:31.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.955+0000 7f179bff0740 1 -- 172.21.15.134:0/165204648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5612280 msgr2=0x5556d59e4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:31.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.955+0000 7f179bff0740 1 --2- 172.21.15.134:0/165204648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5612280 0x5556d59e4540 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1778009530 tx=0x7f177800f790 comp rx=0 tx=0).stop 2024-07-26T11:14:31.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.956+0000 7f179bff0740 1 -- 172.21.15.134:0/165204648 shutdown_connections 2024-07-26T11:14:31.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.956+0000 7f179bff0740 1 --2- 172.21.15.134:0/165204648 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x5556d55f18e0 0x5556d59f68c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.956+0000 7f179bff0740 1 --2- 172.21.15.134:0/165204648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d5632f60 0x5556d59f0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.956+0000 7f179bff0740 1 --2- 172.21.15.134:0/165204648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5612280 0x5556d59e4540 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.956+0000 7f1773e00640 0 client.25205 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:14:31.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.956+0000 7f179bff0740 1 --2- 172.21.15.134:0/165204648 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d560a0a0 0x5556d59e3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.956+0000 7f179bff0740 1 -- 172.21.15.134:0/165204648 >> 172.21.15.134:0/165204648 conn(0x5556d5627350 msgr2=0x5556d59db940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:31.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.956+0000 7f179bff0740 1 -- 172.21.15.134:0/165204648 shutdown_connections 2024-07-26T11:14:31.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.956+0000 7f179bff0740 1 -- 172.21.15.134:0/165204648 wait complete. 2024-07-26T11:14:31.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.956+0000 7f179bff0740 20 client.25205 trim_cache size 0 max 16384 2024-07-26T11:14:31.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.975+0000 7f179bff0740 1 Processor -- start 2024-07-26T11:14:31.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.975+0000 7f179bff0740 1 -- start start 2024-07-26T11:14:31.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.976+0000 7f179bff0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d560a0a0 0x5556d59de880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.976+0000 7f179bff0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5612280 0x5556d59f01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.976+0000 7f179bff0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5632f60 0x5556d59f0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.976+0000 7f179bff0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5556d59f4c00 con 0x5556d5612280 2024-07-26T11:14:31.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.976+0000 7f179bff0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5556d59f52d0 con 0x5556d5632f60 2024-07-26T11:14:31.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.976+0000 7f179bff0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5556d59f59a0 con 0x5556d560a0a0 2024-07-26T11:14:31.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.976+0000 7f1787e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5612280 0x5556d59f01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.976+0000 7f1787e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5612280 0x5556d59f01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57864/0 (socket says 172.21.15.134:57864) 2024-07-26T11:14:31.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.976+0000 7f1787e00640 1 -- 172.21.15.134:0/1078316925 learned_addr learned my addr 172.21.15.134:0/1078316925 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:31.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.976+0000 7f178e800640 1 --2- 172.21.15.134:0/1078316925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5632f60 0x5556d59f0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.976+0000 7f178de00640 1 --2- 172.21.15.134:0/1078316925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d560a0a0 0x5556d59de880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.976+0000 7f1787e00640 1 -- 172.21.15.134:0/1078316925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d560a0a0 msgr2=0x5556d59de880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:31.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.976+0000 7f1787e00640 1 --2- 172.21.15.134:0/1078316925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d560a0a0 0x5556d59de880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.977+0000 7f1787e00640 1 -- 172.21.15.134:0/1078316925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5632f60 msgr2=0x5556d59f0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:31.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.977+0000 7f1787e00640 1 --2- 172.21.15.134:0/1078316925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5632f60 0x5556d59f0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.977+0000 7f1787e00640 1 -- 172.21.15.134:0/1078316925 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556d59d17d0 con 0x5556d5612280 2024-07-26T11:14:31.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.977+0000 7f178e800640 1 --2- 172.21.15.134:0/1078316925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5632f60 0x5556d59f0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:31.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.977+0000 7f1787e00640 1 --2- 172.21.15.134:0/1078316925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5612280 0x5556d59f01d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1778006010 tx=0x7f177801b380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d04a0e3f9ab602ff server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:31.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.977+0000 7f177ea00640 1 -- 172.21.15.134:0/1078316925 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177802f070 con 0x5556d5612280 2024-07-26T11:14:31.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.977+0000 7f177ea00640 1 -- 172.21.15.134:0/1078316925 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1778005040 con 0x5556d5612280 2024-07-26T11:14:31.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.977+0000 7f177ea00640 1 -- 172.21.15.134:0/1078316925 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17780048c0 con 0x5556d5612280 2024-07-26T11:14:31.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.978+0000 7f179bff0740 1 -- 172.21.15.134:0/1078316925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5612280 msgr2=0x5556d59f01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:31.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.978+0000 7f179bff0740 1 --2- 172.21.15.134:0/1078316925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5612280 0x5556d59f01d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1778006010 tx=0x7f177801b380 comp rx=0 tx=0).stop 2024-07-26T11:14:31.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.978+0000 7f179bff0740 1 -- 172.21.15.134:0/1078316925 shutdown_connections 2024-07-26T11:14:31.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.978+0000 7f179bff0740 1 --2- 172.21.15.134:0/1078316925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d5632f60 0x5556d59f0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.978+0000 7f179bff0740 1 --2- 172.21.15.134:0/1078316925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5612280 0x5556d59f01d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.978+0000 7f179bff0740 1 --2- 172.21.15.134:0/1078316925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d560a0a0 0x5556d59de880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.978+0000 7f179bff0740 1 -- 172.21.15.134:0/1078316925 >> 172.21.15.134:0/1078316925 conn(0x5556d5627350 msgr2=0x5556d59e2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:31.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.978+0000 7f179bff0740 1 -- 172.21.15.134:0/1078316925 shutdown_connections 2024-07-26T11:14:31.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.978+0000 7f179bff0740 1 -- 172.21.15.134:0/1078316925 wait complete. 2024-07-26T11:14:31.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.979+0000 7f179bff0740 1 -- start start 2024-07-26T11:14:31.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.979+0000 7f179bff0740 1 Processor -- start 2024-07-26T11:14:31.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.979+0000 7f179bff0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d560a0a0 0x5556d59fefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.979+0000 7f179bff0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d5612280 0x5556d59fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.979+0000 7f179bff0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5632f60 0x5556d59fd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.979+0000 7f179bff0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5556d59f6740 con 0x5556d5632f60 2024-07-26T11:14:31.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.979+0000 7f179bff0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5556d59f6070 con 0x5556d560a0a0 2024-07-26T11:14:31.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.979+0000 7f179bff0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5556d59f6e10 con 0x5556d5612280 2024-07-26T11:14:31.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.979+0000 7f179bff0740 10 client.0 _ll_register_callbacks cb 0x7f1778012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:14:31.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.980+0000 7f178de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d560a0a0 0x5556d59fefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.980+0000 7f178de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d560a0a0 0x5556d59fefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43776/0 (socket says 172.21.15.134:43776) 2024-07-26T11:14:31.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.980+0000 7f178de00640 1 -- 172.21.15.134:0/448615028 learned_addr learned my addr 172.21.15.134:0/448615028 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:31.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.980+0000 7f178e800640 1 --2- 172.21.15.134:0/448615028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5632f60 0x5556d59fd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.980+0000 7f1787e00640 1 --2- 172.21.15.134:0/448615028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d5612280 0x5556d59fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.980+0000 7f178de00640 1 -- 172.21.15.134:0/448615028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d5612280 msgr2=0x5556d59fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:31.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.980+0000 7f178de00640 1 --2- 172.21.15.134:0/448615028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d5612280 0x5556d59fcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.980+0000 7f178de00640 1 -- 172.21.15.134:0/448615028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5632f60 msgr2=0x5556d59fd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:31.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.980+0000 7f178de00640 1 --2- 172.21.15.134:0/448615028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5632f60 0x5556d59fd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.980+0000 7f178de00640 1 -- 172.21.15.134:0/448615028 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556d59fd640 con 0x5556d560a0a0 2024-07-26T11:14:31.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.981+0000 7f178de00640 1 --2- 172.21.15.134:0/448615028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d560a0a0 0x5556d59fefd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f17800ecd40 tx=0x7f1780004580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e942004633558e2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:31.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.981+0000 7f1773e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:14:31.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.981+0000 7f1773e00640 1 -- 172.21.15.134:0/448615028 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17800a6c90 con 0x5556d560a0a0 2024-07-26T11:14:31.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.981+0000 7f179bff0740 1 -- 172.21.15.134:0/448615028 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1778005040 con 0x5556d560a0a0 2024-07-26T11:14:31.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.981+0000 7f1773e00640 1 -- 172.21.15.134:0/448615028 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f17800f34e0 con 0x5556d560a0a0 2024-07-26T11:14:31.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.981+0000 7f1773e00640 1 -- 172.21.15.134:0/448615028 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f17800aa540 con 0x5556d560a0a0 2024-07-26T11:14:31.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.981+0000 7f179bff0740 10 client.25211 fetch_fsmap learned FSMap version 29 2024-07-26T11:14:31.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.981+0000 7f179bff0740 1 -- 172.21.15.134:0/448615028 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5556d5a66ff0 con 0x5556d560a0a0 2024-07-26T11:14:31.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.981+0000 7f1773e00640 1 -- 172.21.15.134:0/448615028 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1778005040 con 0x5556d560a0a0 2024-07-26T11:14:31.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.982+0000 7f1773e00640 1 -- 172.21.15.134:0/448615028 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f17800e1a00 con 0x5556d560a0a0 2024-07-26T11:14:31.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.982+0000 7f179bff0740 10 client.25211 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:14:31.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.982+0000 7f179bff0740 10 client.25211 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T11:14:31.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.982+0000 7f179bff0740 20 client.25211 populate_metadata read hostname 'smithi134' 2024-07-26T11:14:31.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.982+0000 7f179bff0740 1 --2- 172.21.15.134:0/448615028 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x5556d55c6c90 0x5556d5a6a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:31.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.982+0000 7f179bff0740 4 client.25211 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:14:31.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.982+0000 7f179bff0740 1 -- 172.21.15.134:0/448615028 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "status"}) -- 0x5556d594c340 con 0x5556d55c6c90 2024-07-26T11:14:31.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.982+0000 7f1787e00640 1 --2- 172.21.15.134:0/448615028 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x5556d55c6c90 0x5556d5a6a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:31.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.983+0000 7f1787e00640 1 --2- 172.21.15.134:0/448615028 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x5556d55c6c90 0x5556d5a6a3c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4b37d51d5032c522 server_cookie=1dd476f358495d23 in_seq=0 out_seq=0 2024-07-26T11:14:31.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.983+0000 7f1773e00640 10 client.25211 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:14:31.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.983+0000 7f1773e00640 1 -- 172.21.15.134:0/448615028 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5556d594c340 con 0x5556d55c6c90 2024-07-26T11:14:31.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.983+0000 7f1773e00640 10 client.25211 handle_command_reply: tid=0 2024-07-26T11:14:31.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.983+0000 7f179bff0740 1 client.25211 shutdown 2024-07-26T11:14:31.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.983+0000 7f179bff0740 1 -- 172.21.15.134:0/448615028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d560a0a0 msgr2=0x5556d59fefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:31.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.983+0000 7f179bff0740 1 --2- 172.21.15.134:0/448615028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d560a0a0 0x5556d59fefd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f17800ecd40 tx=0x7f1780004580 comp rx=0 tx=0).stop 2024-07-26T11:14:31.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.983+0000 7f179bff0740 1 -- 172.21.15.134:0/448615028 shutdown_connections 2024-07-26T11:14:31.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.983+0000 7f179bff0740 1 --2- 172.21.15.134:0/448615028 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x5556d55c6c90 0x5556d5a6a3c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.983+0000 7f179bff0740 1 --2- 172.21.15.134:0/448615028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5556d5612280 0x5556d59fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.983+0000 7f179bff0740 1 --2- 172.21.15.134:0/448615028 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5556d5632f60 0x5556d59fd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.983+0000 7f179bff0740 1 --2- 172.21.15.134:0/448615028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5556d560a0a0 0x5556d59fefd0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.983+0000 7f179bff0740 1 -- 172.21.15.134:0/448615028 >> 172.21.15.134:0/448615028 conn(0x5556d5627350 msgr2=0x5556d59d5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:31.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.983+0000 7f1773e00640 0 client.25211 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:14:31.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.983+0000 7f179bff0740 1 -- 172.21.15.134:0/448615028 shutdown_connections 2024-07-26T11:14:31.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.983+0000 7f179bff0740 1 -- 172.21.15.134:0/448615028 wait complete. 2024-07-26T11:14:31.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.984+0000 7f179bff0740 20 client.25211 trim_cache size 0 max 16384 2024-07-26T11:14:31.990 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:14:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_fsid": "105fbe5c-4b3e-11ef-bcb7-c7b262605968", 2024-07-26T11:14:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: "whoami": 2, 2024-07-26T11:14:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: "id": 24703, 2024-07-26T11:14:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: "want_state": "up:active", 2024-07-26T11:14:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "up:active", 2024-07-26T11:14:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: "fs_name": "cephfs", 2024-07-26T11:14:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: "rank_uptime": 163.94955921499999, 2024-07-26T11:14:31.990 INFO:teuthology.orchestra.run.smithi134.stdout: "mdsmap_epoch": 29, 2024-07-26T11:14:31.991 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch": 90, 2024-07-26T11:14:31.991 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T11:14:31.991 INFO:teuthology.orchestra.run.smithi134.stdout: "uptime": 280.81019533599999 2024-07-26T11:14:31.991 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:14:31.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.984+0000 7f178f200640 1 -- 172.21.15.134:0/509784404 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f17740b3ca0 msgr2=0x7f17740b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:31.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.984+0000 7f178f200640 1 --2- 172.21.15.134:0/509784404 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f17740b3ca0 0x7f17740b61b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1788009370 tx=0x7f1788004a40 comp rx=0 tx=0).stop 2024-07-26T11:14:31.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.984+0000 7f178f200640 1 -- 172.21.15.134:0/509784404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17901055d0 msgr2=0x7f179019be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:31.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.984+0000 7f178f200640 1 --2- 172.21.15.134:0/509784404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17901055d0 0x7f179019be80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f178000ea60 tx=0x7f178000ef20 comp rx=0 tx=0).stop 2024-07-26T11:14:31.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.984+0000 7f178f200640 1 -- 172.21.15.134:0/509784404 shutdown_connections 2024-07-26T11:14:31.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.984+0000 7f178f200640 1 --2- 172.21.15.134:0/509784404 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f17740b3ca0 0x7f17740b61b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.984+0000 7f178f200640 1 --2- 172.21.15.134:0/509784404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f179010af10 0x7f179019c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.984+0000 7f178f200640 1 --2- 172.21.15.134:0/509784404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f179010b860 0x7f17901a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.984+0000 7f178f200640 1 --2- 172.21.15.134:0/509784404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f17901055d0 0x7f179019be80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:31.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.984+0000 7f178f200640 1 -- 172.21.15.134:0/509784404 >> 172.21.15.134:0/509784404 conn(0x7f1790101040 msgr2=0x7f179010a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:31.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.985+0000 7f178f200640 1 -- 172.21.15.134:0/509784404 shutdown_connections 2024-07-26T11:14:31.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:31.985+0000 7f178f200640 1 -- 172.21.15.134:0/509784404 wait complete. 2024-07-26T11:14:32.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.b status 2024-07-26T11:14:33.368 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:33 smithi134 ceph-mon[31966]: pgmap v437: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:33.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:33 smithi189 ceph-mon[36790]: pgmap v437: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:33 smithi193 ceph-mon[37473]: pgmap v437: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:34.380 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:34 smithi134 ceph-mon[31966]: pgmap v438: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:34 smithi189 ceph-mon[36790]: pgmap v438: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:34 smithi193 ceph-mon[37473]: pgmap v438: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:35.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.701+0000 7fc98e600640 1 -- 172.21.15.134:0/497411514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9881042b0 msgr2=0x7fc9881046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:35.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.701+0000 7fc98e600640 1 --2- 172.21.15.134:0/497411514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9881042b0 0x7fc9881046c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc978009860 tx=0x7fc97802a340 comp rx=0 tx=0).stop 2024-07-26T11:14:35.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.701+0000 7fc98e600640 1 -- 172.21.15.134:0/497411514 shutdown_connections 2024-07-26T11:14:35.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.701+0000 7fc98e600640 1 --2- 172.21.15.134:0/497411514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc98810d710 0x7fc98810fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.701+0000 7fc98e600640 1 --2- 172.21.15.134:0/497411514 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc988104c00 0x7fc98810d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.701+0000 7fc98e600640 1 --2- 172.21.15.134:0/497411514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9881042b0 0x7fc9881046c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.701+0000 7fc98e600640 1 -- 172.21.15.134:0/497411514 >> 172.21.15.134:0/497411514 conn(0x7fc9880fdd90 msgr2=0x7fc9881001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:35.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.702+0000 7fc98e600640 1 -- 172.21.15.134:0/497411514 shutdown_connections 2024-07-26T11:14:35.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.702+0000 7fc98e600640 1 -- 172.21.15.134:0/497411514 wait complete. 2024-07-26T11:14:35.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.703+0000 7fc98e600640 1 Processor -- start 2024-07-26T11:14:35.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.704+0000 7fc98e600640 1 -- start start 2024-07-26T11:14:35.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.704+0000 7fc98e600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9881042b0 0x7fc988101ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.704+0000 7fc98e600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc988104c00 0x7fc988102530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.704+0000 7fc98e600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc98810d710 0x7fc988102a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.704+0000 7fc98e600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc988113380 con 0x7fc98810d710 2024-07-26T11:14:35.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.704+0000 7fc98e600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc988113200 con 0x7fc9881042b0 2024-07-26T11:14:35.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.704+0000 7fc98e600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc988113500 con 0x7fc988104c00 2024-07-26T11:14:35.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.704+0000 7fc98dc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc98810d710 0x7fc988102a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.704+0000 7fc98d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9881042b0 0x7fc988101ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.705+0000 7fc98d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9881042b0 0x7fc988101ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43800/0 (socket says 172.21.15.134:43800) 2024-07-26T11:14:35.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.705+0000 7fc98dc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc98810d710 0x7fc988102a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57884/0 (socket says 172.21.15.134:57884) 2024-07-26T11:14:35.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.705+0000 7fc98d200640 1 -- 172.21.15.134:0/3792149793 learned_addr learned my addr 172.21.15.134:0/3792149793 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:35.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.705+0000 7fc987e00640 1 --2- 172.21.15.134:0/3792149793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc988104c00 0x7fc988102530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.705+0000 7fc98d200640 1 -- 172.21.15.134:0/3792149793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc988104c00 msgr2=0x7fc988102530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:35.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.705+0000 7fc98d200640 1 --2- 172.21.15.134:0/3792149793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc988104c00 0x7fc988102530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.705+0000 7fc98d200640 1 -- 172.21.15.134:0/3792149793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc98810d710 msgr2=0x7fc988102a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:35.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.705+0000 7fc98d200640 1 --2- 172.21.15.134:0/3792149793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc98810d710 0x7fc988102a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.705+0000 7fc98d200640 1 -- 172.21.15.134:0/3792149793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9881a8090 con 0x7fc9881042b0 2024-07-26T11:14:35.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.705+0000 7fc98d200640 1 --2- 172.21.15.134:0/3792149793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9881042b0 0x7fc988101ff0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc9780303f0 tx=0x7fc978032da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:35.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.706+0000 7fc985600640 1 -- 172.21.15.134:0/3792149793 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc978040020 con 0x7fc9881042b0 2024-07-26T11:14:35.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.706+0000 7fc98e600640 1 -- 172.21.15.134:0/3792149793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9881a8320 con 0x7fc9881042b0 2024-07-26T11:14:35.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.706+0000 7fc98e600640 1 -- 172.21.15.134:0/3792149793 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9881a8850 con 0x7fc9881042b0 2024-07-26T11:14:35.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.706+0000 7fc985600640 1 -- 172.21.15.134:0/3792149793 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc978030dd0 con 0x7fc9881042b0 2024-07-26T11:14:35.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.706+0000 7fc985600640 1 -- 172.21.15.134:0/3792149793 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc978037040 con 0x7fc9881042b0 2024-07-26T11:14:35.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.709+0000 7fc9967c9740 1 Processor -- start 2024-07-26T11:14:35.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.709+0000 7fc9967c9740 1 -- start start 2024-07-26T11:14:35.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.710+0000 7fc9967c9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a430f60 0x55a11a7d6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.710+0000 7fc985600640 1 -- 172.21.15.134:0/3792149793 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fc978043020 con 0x7fc9881042b0 2024-07-26T11:14:35.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.710+0000 7fc9967c9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a4080a0 0x55a11a7d6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.710+0000 7fc9967c9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a410280 0x55a11a7d7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.710+0000 7fc9967c9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a11a74c970 con 0x55a11a4080a0 2024-07-26T11:14:35.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.710+0000 7fc9967c9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a11a73be70 con 0x55a11a410280 2024-07-26T11:14:35.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.710+0000 7fc9967c9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a11a73feb0 con 0x55a11a430f60 2024-07-26T11:14:35.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.710+0000 7fc98d200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a4080a0 0x55a11a7d6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.710+0000 7fc98dc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a430f60 0x55a11a7d6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.710+0000 7fc98d200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a4080a0 0x55a11a7d6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57898/0 (socket says 172.21.15.134:57898) 2024-07-26T11:14:35.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.710+0000 7fc98d200640 1 -- 172.21.15.134:0/2638897535 learned_addr learned my addr 172.21.15.134:0/2638897535 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:35.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.710+0000 7fc987e00640 1 --2- 172.21.15.134:0/2638897535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a410280 0x55a11a7d7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.711+0000 7fc98dc00640 1 -- 172.21.15.134:0/2638897535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a410280 msgr2=0x55a11a7d7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:35.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.711+0000 7fc98dc00640 1 --2- 172.21.15.134:0/2638897535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a410280 0x55a11a7d7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.711+0000 7fc98dc00640 1 -- 172.21.15.134:0/2638897535 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a4080a0 msgr2=0x55a11a7d6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:35.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.711+0000 7fc98dc00640 1 --2- 172.21.15.134:0/2638897535 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a4080a0 0x55a11a7d6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.711+0000 7fc98dc00640 1 -- 172.21.15.134:0/2638897535 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a11a6e3110 con 0x55a11a430f60 2024-07-26T11:14:35.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.711+0000 7fc98dc00640 1 --2- 172.21.15.134:0/2638897535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a430f60 0x55a11a7d6780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc980009b90 tx=0x7fc980011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=280761ee593fc527 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:35.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.711+0000 7fc97e000640 1 -- 172.21.15.134:0/2638897535 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc980004030 con 0x55a11a430f60 2024-07-26T11:14:35.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.711+0000 7fc97e000640 1 -- 172.21.15.134:0/2638897535 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9800035f0 con 0x55a11a430f60 2024-07-26T11:14:35.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.711+0000 7fc97e000640 1 -- 172.21.15.134:0/2638897535 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc980019620 con 0x55a11a430f60 2024-07-26T11:14:35.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.712+0000 7fc985600640 1 --2- 172.21.15.134:0/3792149793 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc9680b3c20 0x7fc9680b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.713+0000 7fc98d200640 1 --2- 172.21.15.134:0/3792149793 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc9680b3c20 0x7fc9680b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.713+0000 7fc985600640 1 -- 172.21.15.134:0/3792149793 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc978103a40 con 0x7fc9881042b0 2024-07-26T11:14:35.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.713+0000 7fc9967c9740 1 -- 172.21.15.134:0/2638897535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a430f60 msgr2=0x55a11a7d6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:35.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.713+0000 7fc9967c9740 1 --2- 172.21.15.134:0/2638897535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a430f60 0x55a11a7d6780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc980009b90 tx=0x7fc980011810 comp rx=0 tx=0).stop 2024-07-26T11:14:35.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.713+0000 7fc9967c9740 1 -- 172.21.15.134:0/2638897535 shutdown_connections 2024-07-26T11:14:35.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.713+0000 7fc9967c9740 1 --2- 172.21.15.134:0/2638897535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a410280 0x55a11a7d7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.713+0000 7fc9967c9740 1 --2- 172.21.15.134:0/2638897535 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a4080a0 0x55a11a7d6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.713+0000 7fc9967c9740 1 --2- 172.21.15.134:0/2638897535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a430f60 0x55a11a7d6780 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.713+0000 7fc9967c9740 1 -- 172.21.15.134:0/2638897535 >> 172.21.15.134:0/2638897535 conn(0x55a11a425350 msgr2=0x55a11a7d4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:35.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.713+0000 7fc9967c9740 1 -- 172.21.15.134:0/2638897535 shutdown_connections 2024-07-26T11:14:35.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.713+0000 7fc9967c9740 1 -- 172.21.15.134:0/2638897535 wait complete. 2024-07-26T11:14:35.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.713+0000 7fc98d200640 1 --2- 172.21.15.134:0/3792149793 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc9680b3c20 0x7fc9680b6130 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc978005790 tx=0x7fc978109530 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:35.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.714+0000 7fc9967c9740 1 -- start start 2024-07-26T11:14:35.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.714+0000 7fc9967c9740 1 Processor -- start 2024-07-26T11:14:35.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.714+0000 7fc9967c9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a4080a0 0x55a11a7e1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.714+0000 7fc9967c9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a410280 0x55a11a7e2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.714+0000 7fc9967c9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 0x55a11a7ee2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.714+0000 7fc9967c9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a11a59f5f0 con 0x55a11a430f60 2024-07-26T11:14:35.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.714+0000 7fc9967c9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a11a6d8540 con 0x55a11a4080a0 2024-07-26T11:14:35.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.714+0000 7fc9967c9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a11a7e4f60 con 0x55a11a410280 2024-07-26T11:14:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.715+0000 7fc9967c9740 10 client.0 _ll_register_callbacks cb 0x55a11a6e7f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:14:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.715+0000 7fc987e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a410280 0x55a11a7e2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.715+0000 7fc987e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a410280 0x55a11a7e2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:60246/0 (socket says 172.21.15.134:60246) 2024-07-26T11:14:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.715+0000 7fc987e00640 1 -- 172.21.15.134:0/1373792632 learned_addr learned my addr 172.21.15.134:0/1373792632 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.715+0000 7fc98dc00640 1 --2- 172.21.15.134:0/1373792632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a4080a0 0x55a11a7e1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.715+0000 7fc98dc00640 1 --2- 172.21.15.134:0/1373792632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 0x55a11a7ee2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.715+0000 7fc987e00640 1 -- 172.21.15.134:0/1373792632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a4080a0 msgr2=0x55a11a7e1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.715+0000 7fc987e00640 1 --2- 172.21.15.134:0/1373792632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a4080a0 0x55a11a7e1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.715+0000 7fc987e00640 1 -- 172.21.15.134:0/1373792632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 msgr2=0x55a11a7ee2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.715+0000 7fc987e00640 1 --2- 172.21.15.134:0/1373792632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 0x55a11a7ee2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.715+0000 7fc987e00640 1 -- 172.21.15.134:0/1373792632 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a11a49c540 con 0x55a11a410280 2024-07-26T11:14:35.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.715+0000 7fc98dc00640 1 --2- 172.21.15.134:0/1373792632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 0x55a11a7ee2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:14:35.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.716+0000 7fc987e00640 1 --2- 172.21.15.134:0/1373792632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a410280 0x55a11a7e2570 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc970011d10 tx=0x7fc97000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efd7586706800aed server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:35.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.716+0000 7fc976a00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:14:35.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.716+0000 7fc976a00640 1 -- 172.21.15.134:0/1373792632 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97000fe60 con 0x55a11a410280 2024-07-26T11:14:35.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.716+0000 7fc9967c9740 1 -- 172.21.15.134:0/1373792632 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9800035f0 con 0x55a11a410280 2024-07-26T11:14:35.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.716+0000 7fc976a00640 1 -- 172.21.15.134:0/1373792632 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc970004b20 con 0x55a11a410280 2024-07-26T11:14:35.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.716+0000 7fc976a00640 1 -- 172.21.15.134:0/1373792632 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc970025050 con 0x55a11a410280 2024-07-26T11:14:35.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.716+0000 7fc9967c9740 10 client.25234 fetch_fsmap learned FSMap version 29 2024-07-26T11:14:35.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.716+0000 7fc9967c9740 1 -- 172.21.15.134:0/1373792632 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a11a7f0ed0 con 0x55a11a410280 2024-07-26T11:14:35.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.717+0000 7fc976a00640 1 -- 172.21.15.134:0/1373792632 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9800035f0 con 0x55a11a410280 2024-07-26T11:14:35.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.717+0000 7fc976a00640 1 -- 172.21.15.134:0/1373792632 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc970019030 con 0x55a11a410280 2024-07-26T11:14:35.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.717+0000 7fc9967c9740 10 client.25234 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:14:35.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.717+0000 7fc9967c9740 10 client.25234 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T11:14:35.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.717+0000 7fc9967c9740 20 client.25234 populate_metadata read hostname 'smithi134' 2024-07-26T11:14:35.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.717+0000 7fc9967c9740 1 --2- 172.21.15.134:0/1373792632 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55a11a3ef8e0 0x55a11a7f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.717+0000 7fc9967c9740 4 client.25234 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:14:35.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.717+0000 7fc9967c9740 1 -- 172.21.15.134:0/1373792632 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a11a74e8c0 con 0x55a11a3ef8e0 2024-07-26T11:14:35.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.718+0000 7fc98dc00640 1 --2- 172.21.15.134:0/1373792632 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55a11a3ef8e0 0x55a11a7f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.718+0000 7fc976a00640 10 client.25234 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:14:35.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.718+0000 7fc98dc00640 1 --2- 172.21.15.134:0/1373792632 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55a11a3ef8e0 0x55a11a7f4a70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c787cae1aa1566d7 server_cookie=66cfdc0315acd392 in_seq=0 out_seq=0 2024-07-26T11:14:35.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.719+0000 7fc976a00640 1 -- 172.21.15.134:0/1373792632 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a11a74e8c0 con 0x55a11a3ef8e0 2024-07-26T11:14:35.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.719+0000 7fc976a00640 10 client.25234 handle_command_reply: tid=0 2024-07-26T11:14:35.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.719+0000 7fc9967c9740 1 client.25234 shutdown 2024-07-26T11:14:35.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.719+0000 7fc9967c9740 1 -- 172.21.15.134:0/1373792632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a410280 msgr2=0x55a11a7e2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:35.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.719+0000 7fc9967c9740 1 --2- 172.21.15.134:0/1373792632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a410280 0x55a11a7e2570 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc970011d10 tx=0x7fc97000f790 comp rx=0 tx=0).stop 2024-07-26T11:14:35.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.720+0000 7fc9967c9740 1 -- 172.21.15.134:0/1373792632 shutdown_connections 2024-07-26T11:14:35.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.720+0000 7fc9967c9740 1 --2- 172.21.15.134:0/1373792632 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55a11a3ef8e0 0x55a11a7f4a70 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.720+0000 7fc9967c9740 1 --2- 172.21.15.134:0/1373792632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a410280 0x55a11a7e2570 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.720+0000 7fc9967c9740 1 --2- 172.21.15.134:0/1373792632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 0x55a11a7ee2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.720+0000 7fc9967c9740 1 --2- 172.21.15.134:0/1373792632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a4080a0 0x55a11a7e1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.720+0000 7fc9967c9740 1 -- 172.21.15.134:0/1373792632 >> 172.21.15.134:0/1373792632 conn(0x55a11a425350 msgr2=0x55a11a7d9910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:35.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.720+0000 7fc976a00640 0 client.25234 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:14:35.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.720+0000 7fc9967c9740 1 -- 172.21.15.134:0/1373792632 shutdown_connections 2024-07-26T11:14:35.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.720+0000 7fc9967c9740 1 -- 172.21.15.134:0/1373792632 wait complete. 2024-07-26T11:14:35.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.720+0000 7fc9967c9740 20 client.25234 trim_cache size 0 max 16384 2024-07-26T11:14:35.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.739+0000 7fc9967c9740 1 Processor -- start 2024-07-26T11:14:35.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.739+0000 7fc9967c9740 1 -- start start 2024-07-26T11:14:35.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.740+0000 7fc9967c9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a4080a0 0x55a11a7dc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.740+0000 7fc9967c9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a410280 0x55a11a7d4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.740+0000 7fc9967c9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 0x55a11a7e14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.740+0000 7fc9967c9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a11a7f2db0 con 0x55a11a430f60 2024-07-26T11:14:35.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.740+0000 7fc9967c9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a11a7f3480 con 0x55a11a410280 2024-07-26T11:14:35.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.740+0000 7fc9967c9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a11a7f3b50 con 0x55a11a4080a0 2024-07-26T11:14:35.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.740+0000 7fc98dc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a4080a0 0x55a11a7dc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.740+0000 7fc987e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a410280 0x55a11a7d4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.740+0000 7fc98dc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a4080a0 0x55a11a7dc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:60252/0 (socket says 172.21.15.134:60252) 2024-07-26T11:14:35.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.740+0000 7fc98dc00640 1 -- 172.21.15.134:0/3764642358 learned_addr learned my addr 172.21.15.134:0/3764642358 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:35.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.740+0000 7fc98dc00640 1 --2- 172.21.15.134:0/3764642358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 0x55a11a7e14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.741+0000 7fc987e00640 1 -- 172.21.15.134:0/3764642358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a4080a0 msgr2=0x55a11a7dc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:35.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.741+0000 7fc987e00640 1 --2- 172.21.15.134:0/3764642358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a4080a0 0x55a11a7dc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.741+0000 7fc987e00640 1 -- 172.21.15.134:0/3764642358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 msgr2=0x55a11a7e14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:35.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.741+0000 7fc987e00640 1 --2- 172.21.15.134:0/3764642358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 0x55a11a7e14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.741+0000 7fc987e00640 1 -- 172.21.15.134:0/3764642358 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a11a7dd880 con 0x55a11a410280 2024-07-26T11:14:35.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.741+0000 7fc987e00640 1 --2- 172.21.15.134:0/3764642358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a410280 0x55a11a7d4590 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc970002230 tx=0x7fc970011820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8a5e588c46c3f32 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:35.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.741+0000 7fc97d600640 1 -- 172.21.15.134:0/3764642358 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97002d020 con 0x55a11a410280 2024-07-26T11:14:35.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.742+0000 7fc97d600640 1 -- 172.21.15.134:0/3764642358 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc970003e20 con 0x55a11a410280 2024-07-26T11:14:35.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.742+0000 7fc97d600640 1 -- 172.21.15.134:0/3764642358 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc970005310 con 0x55a11a410280 2024-07-26T11:14:35.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.742+0000 7fc9967c9740 1 -- 172.21.15.134:0/3764642358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a410280 msgr2=0x55a11a7d4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:35.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.742+0000 7fc9967c9740 1 --2- 172.21.15.134:0/3764642358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a410280 0x55a11a7d4590 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc970002230 tx=0x7fc970011820 comp rx=0 tx=0).stop 2024-07-26T11:14:35.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.742+0000 7fc9967c9740 1 -- 172.21.15.134:0/3764642358 shutdown_connections 2024-07-26T11:14:35.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.743+0000 7fc9967c9740 1 --2- 172.21.15.134:0/3764642358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 0x55a11a7e14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.743+0000 7fc9967c9740 1 --2- 172.21.15.134:0/3764642358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a410280 0x55a11a7d4590 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.743+0000 7fc9967c9740 1 --2- 172.21.15.134:0/3764642358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a4080a0 0x55a11a7dc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.743+0000 7fc9967c9740 1 -- 172.21.15.134:0/3764642358 >> 172.21.15.134:0/3764642358 conn(0x55a11a425350 msgr2=0x55a11a7d3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:35.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.743+0000 7fc9967c9740 1 -- 172.21.15.134:0/3764642358 shutdown_connections 2024-07-26T11:14:35.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.743+0000 7fc9967c9740 1 -- 172.21.15.134:0/3764642358 wait complete. 2024-07-26T11:14:35.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.743+0000 7fc9967c9740 1 -- start start 2024-07-26T11:14:35.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.744+0000 7fc9967c9740 1 Processor -- start 2024-07-26T11:14:35.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.744+0000 7fc9967c9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a4080a0 0x55a11a7fd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.744+0000 7fc9967c9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 0x55a11a7f8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.744+0000 7fc9967c9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a3c4c90 0x55a11a7f94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.744+0000 7fc9967c9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a11a7f48f0 con 0x55a11a430f60 2024-07-26T11:14:35.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.744+0000 7fc9967c9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a11a7f4220 con 0x55a11a3c4c90 2024-07-26T11:14:35.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.744+0000 7fc9967c9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a11a7f4fc0 con 0x55a11a4080a0 2024-07-26T11:14:35.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.744+0000 7fc9967c9740 10 client.0 _ll_register_callbacks cb 0x7fc970011920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:14:35.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.744+0000 7fc987e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 0x55a11a7f8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.744+0000 7fc987e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 0x55a11a7f8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57928/0 (socket says 172.21.15.134:57928) 2024-07-26T11:14:35.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.744+0000 7fc987e00640 1 -- 172.21.15.134:0/3552703870 learned_addr learned my addr 172.21.15.134:0/3552703870 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:35.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.745+0000 7fc98dc00640 1 --2- 172.21.15.134:0/3552703870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a4080a0 0x55a11a7fd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.745+0000 7fc98dc00640 1 --2- 172.21.15.134:0/3552703870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a3c4c90 0x55a11a7f94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.745+0000 7fc987e00640 1 -- 172.21.15.134:0/3552703870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a4080a0 msgr2=0x55a11a7fd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:35.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.745+0000 7fc987e00640 1 --2- 172.21.15.134:0/3552703870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a4080a0 0x55a11a7fd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.745+0000 7fc987e00640 1 -- 172.21.15.134:0/3552703870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a3c4c90 msgr2=0x55a11a7f94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:35.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.745+0000 7fc987e00640 1 --2- 172.21.15.134:0/3552703870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a3c4c90 0x55a11a7f94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.745+0000 7fc987e00640 1 -- 172.21.15.134:0/3552703870 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a11a7f9c60 con 0x55a11a430f60 2024-07-26T11:14:35.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.745+0000 7fc987e00640 1 --2- 172.21.15.134:0/3552703870 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 0x55a11a7f8f10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc970011ce0 tx=0x7fc970009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc269452a110272e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:35.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.745+0000 7fc976a00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:14:35.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.746+0000 7fc976a00640 1 -- 172.21.15.134:0/3552703870 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97002d040 con 0x55a11a430f60 2024-07-26T11:14:35.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.746+0000 7fc976a00640 1 -- 172.21.15.134:0/3552703870 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc97000ec00 con 0x55a11a430f60 2024-07-26T11:14:35.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.746+0000 7fc9967c9740 1 -- 172.21.15.134:0/3552703870 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc970003e20 con 0x55a11a430f60 2024-07-26T11:14:35.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.746+0000 7fc976a00640 1 -- 172.21.15.134:0/3552703870 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc97003f050 con 0x55a11a430f60 2024-07-26T11:14:35.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.746+0000 7fc9967c9740 10 client.15084 fetch_fsmap learned FSMap version 29 2024-07-26T11:14:35.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.746+0000 7fc9967c9740 1 -- 172.21.15.134:0/3552703870 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a11a85ccc0 con 0x55a11a430f60 2024-07-26T11:14:35.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.746+0000 7fc976a00640 1 -- 172.21.15.134:0/3552703870 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc970003e20 con 0x55a11a430f60 2024-07-26T11:14:35.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.747+0000 7fc976a00640 1 -- 172.21.15.134:0/3552703870 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7fc970019070 con 0x55a11a430f60 2024-07-26T11:14:35.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.747+0000 7fc9967c9740 10 client.15084 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:14:35.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.747+0000 7fc9967c9740 10 client.15084 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T11:14:35.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.747+0000 7fc9967c9740 20 client.15084 populate_metadata read hostname 'smithi134' 2024-07-26T11:14:35.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.747+0000 7fc9967c9740 1 --2- 172.21.15.134:0/3552703870 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55a11a85eca0 0x55a11a860110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:35.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.747+0000 7fc9967c9740 4 client.15084 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:14:35.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.747+0000 7fc9967c9740 1 -- 172.21.15.134:0/3552703870 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "status"}) -- 0x55a11a74a360 con 0x55a11a85eca0 2024-07-26T11:14:35.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.747+0000 7fc98dc00640 1 --2- 172.21.15.134:0/3552703870 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55a11a85eca0 0x55a11a860110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:35.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.748+0000 7fc98dc00640 1 --2- 172.21.15.134:0/3552703870 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55a11a85eca0 0x55a11a860110 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=57ea0785ab811f83 server_cookie=95a985218223053e in_seq=0 out_seq=0 2024-07-26T11:14:35.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.748+0000 7fc976a00640 10 client.15084 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:14:35.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.748+0000 7fc976a00640 1 -- 172.21.15.134:0/3552703870 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55a11a74a360 con 0x55a11a85eca0 2024-07-26T11:14:35.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.748+0000 7fc976a00640 10 client.15084 handle_command_reply: tid=0 2024-07-26T11:14:35.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.748+0000 7fc9967c9740 1 client.15084 shutdown 2024-07-26T11:14:35.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.748+0000 7fc9967c9740 1 -- 172.21.15.134:0/3552703870 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 msgr2=0x55a11a7f8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:35.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.748+0000 7fc9967c9740 1 --2- 172.21.15.134:0/3552703870 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 0x55a11a7f8f10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc970011ce0 tx=0x7fc970009500 comp rx=0 tx=0).stop 2024-07-26T11:14:35.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.749+0000 7fc9967c9740 1 -- 172.21.15.134:0/3552703870 shutdown_connections 2024-07-26T11:14:35.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.749+0000 7fc9967c9740 1 --2- 172.21.15.134:0/3552703870 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55a11a85eca0 0x55a11a860110 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.749+0000 7fc9967c9740 1 --2- 172.21.15.134:0/3552703870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a11a3c4c90 0x55a11a7f94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.749+0000 7fc9967c9740 1 --2- 172.21.15.134:0/3552703870 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a11a430f60 0x55a11a7f8f10 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.749+0000 7fc9967c9740 1 --2- 172.21.15.134:0/3552703870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a11a4080a0 0x55a11a7fd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.749+0000 7fc976a00640 0 client.15084 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:14:35.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.749+0000 7fc9967c9740 1 -- 172.21.15.134:0/3552703870 >> 172.21.15.134:0/3552703870 conn(0x55a11a425350 msgr2=0x55a11a7e1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:35.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.749+0000 7fc9967c9740 1 -- 172.21.15.134:0/3552703870 shutdown_connections 2024-07-26T11:14:35.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.749+0000 7fc9967c9740 1 -- 172.21.15.134:0/3552703870 wait complete. 2024-07-26T11:14:35.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.749+0000 7fc9967c9740 20 client.15084 trim_cache size 0 max 16384 2024-07-26T11:14:35.754 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:14:35.755 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_fsid": "105fbe5c-4b3e-11ef-bcb7-c7b262605968", 2024-07-26T11:14:35.755 INFO:teuthology.orchestra.run.smithi134.stdout: "whoami": 1, 2024-07-26T11:14:35.755 INFO:teuthology.orchestra.run.smithi134.stdout: "id": 24719, 2024-07-26T11:14:35.755 INFO:teuthology.orchestra.run.smithi134.stdout: "want_state": "up:active", 2024-07-26T11:14:35.755 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "up:active", 2024-07-26T11:14:35.755 INFO:teuthology.orchestra.run.smithi134.stdout: "fs_name": "cephfs", 2024-07-26T11:14:35.755 INFO:teuthology.orchestra.run.smithi134.stdout: "rank_uptime": 168.72332574000001, 2024-07-26T11:14:35.755 INFO:teuthology.orchestra.run.smithi134.stdout: "mdsmap_epoch": 29, 2024-07-26T11:14:35.755 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch": 90, 2024-07-26T11:14:35.755 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T11:14:35.755 INFO:teuthology.orchestra.run.smithi134.stdout: "uptime": 273.07121092199998 2024-07-26T11:14:35.755 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:14:35.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.750+0000 7fc98e600640 1 -- 172.21.15.134:0/3792149793 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc9680b3c20 msgr2=0x7fc9680b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:35.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.750+0000 7fc98e600640 1 --2- 172.21.15.134:0/3792149793 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc9680b3c20 0x7fc9680b6130 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc978005790 tx=0x7fc978109530 comp rx=0 tx=0).stop 2024-07-26T11:14:35.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.750+0000 7fc98e600640 1 -- 172.21.15.134:0/3792149793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9881042b0 msgr2=0x7fc988101ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:35.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.750+0000 7fc98e600640 1 --2- 172.21.15.134:0/3792149793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9881042b0 0x7fc988101ff0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc9780303f0 tx=0x7fc978032da0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.750+0000 7fc98e600640 1 -- 172.21.15.134:0/3792149793 shutdown_connections 2024-07-26T11:14:35.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.750+0000 7fc98e600640 1 --2- 172.21.15.134:0/3792149793 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc9680b3c20 0x7fc9680b6130 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.750+0000 7fc98e600640 1 --2- 172.21.15.134:0/3792149793 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc988104c00 0x7fc988102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.750+0000 7fc98e600640 1 --2- 172.21.15.134:0/3792149793 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc98810d710 0x7fc988102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.750+0000 7fc98e600640 1 --2- 172.21.15.134:0/3792149793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9881042b0 0x7fc988101ff0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:35.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.750+0000 7fc98e600640 1 -- 172.21.15.134:0/3792149793 >> 172.21.15.134:0/3792149793 conn(0x7fc9880fdd90 msgr2=0x7fc988106140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:35.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.750+0000 7fc98e600640 1 -- 172.21.15.134:0/3792149793 shutdown_connections 2024-07-26T11:14:35.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:35.750+0000 7fc98e600640 1 -- 172.21.15.134:0/3792149793 wait complete. 2024-07-26T11:14:36.715 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.a status 2024-07-26T11:14:37.182 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:37.332 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:37 smithi134 ceph-mon[31966]: pgmap v439: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:37 smithi189 ceph-mon[36790]: pgmap v439: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:37 smithi193 ceph-mon[37473]: pgmap v439: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:39.312 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:39 smithi134 ceph-mon[31966]: pgmap v440: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:39 smithi189 ceph-mon[36790]: pgmap v440: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:39.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.517+0000 7f5c2f200640 1 -- 172.21.15.134:0/1418832822 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c3010fc80 msgr2=0x7f5c301164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:39.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.517+0000 7f5c2f200640 1 --2- 172.21.15.134:0/1418832822 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c3010fc80 0x7f5c301164b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5c28009ca0 tx=0x7f5c2802a8c0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.518+0000 7f5c2f200640 1 -- 172.21.15.134:0/1418832822 shutdown_connections 2024-07-26T11:14:39.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.518+0000 7f5c2f200640 1 --2- 172.21.15.134:0/1418832822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c3010f330 0x7f5c3010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.518+0000 7f5c2f200640 1 --2- 172.21.15.134:0/1418832822 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c3010fc80 0x7f5c301164b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.518+0000 7f5c2f200640 1 --2- 172.21.15.134:0/1418832822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c301099f0 0x7f5c30109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.518+0000 7f5c2f200640 1 -- 172.21.15.134:0/1418832822 >> 172.21.15.134:0/1418832822 conn(0x7f5c30078ff0 msgr2=0x7f5c30107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:39.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.518+0000 7f5c2f200640 1 -- 172.21.15.134:0/1418832822 shutdown_connections 2024-07-26T11:14:39.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.518+0000 7f5c2f200640 1 -- 172.21.15.134:0/1418832822 wait complete. 2024-07-26T11:14:39.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.519+0000 7f5c2f200640 1 Processor -- start 2024-07-26T11:14:39.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.520+0000 7f5c2f200640 1 -- start start 2024-07-26T11:14:39.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.520+0000 7f5c2f200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c301099f0 0x7f5c301a0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.520+0000 7f5c2f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c3010f330 0x7f5c301a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.520+0000 7f5c2f200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c3010fc80 0x7f5c301a5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.520+0000 7f5c2f200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5c30119990 con 0x7f5c3010fc80 2024-07-26T11:14:39.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.520+0000 7f5c2f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5c30119810 con 0x7f5c3010f330 2024-07-26T11:14:39.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.520+0000 7f5c2f200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5c30119b10 con 0x7f5c301099f0 2024-07-26T11:14:39.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.521+0000 7f5c2d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c3010f330 0x7f5c301a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.521+0000 7f5c2de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c301099f0 0x7f5c301a0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.521+0000 7f5c2d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c3010f330 0x7f5c301a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43860/0 (socket says 172.21.15.134:43860) 2024-07-26T11:14:39.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.521+0000 7f5c2e800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c3010fc80 0x7f5c301a5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.521+0000 7f5c2d400640 1 -- 172.21.15.134:0/2880596486 learned_addr learned my addr 172.21.15.134:0/2880596486 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:39.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.521+0000 7f5c2d400640 1 -- 172.21.15.134:0/2880596486 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c301099f0 msgr2=0x7f5c301a0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:39.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.521+0000 7f5c2d400640 1 --2- 172.21.15.134:0/2880596486 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c301099f0 0x7f5c301a0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.521+0000 7f5c2d400640 1 -- 172.21.15.134:0/2880596486 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c3010fc80 msgr2=0x7f5c301a5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:39.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.521+0000 7f5c2d400640 1 --2- 172.21.15.134:0/2880596486 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c3010fc80 0x7f5c301a5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.521+0000 7f5c2d400640 1 -- 172.21.15.134:0/2880596486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c301a6270 con 0x7f5c3010f330 2024-07-26T11:14:39.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.521+0000 7f5c2de00640 1 --2- 172.21.15.134:0/2880596486 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c301099f0 0x7f5c301a0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:39.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.521+0000 7f5c2d400640 1 --2- 172.21.15.134:0/2880596486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c3010f330 0x7f5c301a08a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00e9c0 tx=0x7f5c1c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:39.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.522+0000 7f5c26a00640 1 -- 172.21.15.134:0/2880596486 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c00ccc0 con 0x7f5c3010f330 2024-07-26T11:14:39.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.522+0000 7f5c26a00640 1 -- 172.21.15.134:0/2880596486 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c1c0054d0 con 0x7f5c3010f330 2024-07-26T11:14:39.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.522+0000 7f5c2f200640 1 -- 172.21.15.134:0/2880596486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c301a6560 con 0x7f5c3010f330 2024-07-26T11:14:39.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.522+0000 7f5c26a00640 1 -- 172.21.15.134:0/2880596486 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c010430 con 0x7f5c3010f330 2024-07-26T11:14:39.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.522+0000 7f5c2f200640 1 -- 172.21.15.134:0/2880596486 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c301accc0 con 0x7f5c3010f330 2024-07-26T11:14:39.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:39 smithi193 ceph-mon[37473]: pgmap v440: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:39.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.525+0000 7f5c3b6d5740 1 Processor -- start 2024-07-26T11:14:39.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.525+0000 7f5c3b6d5740 1 -- start start 2024-07-26T11:14:39.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.526+0000 7f5c26a00640 1 -- 172.21.15.134:0/2880596486 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f5c1c005020 con 0x7f5c3010f330 2024-07-26T11:14:39.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.526+0000 7f5c3b6d5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc87ff60 0x55c6dcc25760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.526+0000 7f5c3b6d5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc8570a0 0x55c6dcc25ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.526+0000 7f5c3b6d5740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc85f280 0x55c6dcc261e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.526+0000 7f5c3b6d5740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c6dcb9b950 con 0x55c6dc85f280 2024-07-26T11:14:39.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.526+0000 7f5c3b6d5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c6dcb8ae50 con 0x55c6dc87ff60 2024-07-26T11:14:39.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.526+0000 7f5c3b6d5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c6dcb8ee90 con 0x55c6dc8570a0 2024-07-26T11:14:39.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.526+0000 7f5c2de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc8570a0 0x55c6dcc25ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.526+0000 7f5c2de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc8570a0 0x55c6dcc25ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:60292/0 (socket says 172.21.15.134:60292) 2024-07-26T11:14:39.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.526+0000 7f5c2de00640 1 -- 172.21.15.134:0/2096375890 learned_addr learned my addr 172.21.15.134:0/2096375890 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:39.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.526+0000 7f5c2e800640 1 --2- 172.21.15.134:0/2096375890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc87ff60 0x55c6dcc25760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.527+0000 7f5c2d400640 1 --2- 172.21.15.134:0/2096375890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc85f280 0x55c6dcc261e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.527+0000 7f5c2de00640 1 -- 172.21.15.134:0/2096375890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc87ff60 msgr2=0x55c6dcc25760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:39.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.527+0000 7f5c2de00640 1 --2- 172.21.15.134:0/2096375890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc87ff60 0x55c6dcc25760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.527+0000 7f5c2de00640 1 -- 172.21.15.134:0/2096375890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc85f280 msgr2=0x55c6dcc261e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:39.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.527+0000 7f5c2de00640 1 --2- 172.21.15.134:0/2096375890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc85f280 0x55c6dcc261e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.527+0000 7f5c2de00640 1 -- 172.21.15.134:0/2096375890 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6dcb320f0 con 0x55c6dc8570a0 2024-07-26T11:14:39.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.527+0000 7f5c2e800640 1 --2- 172.21.15.134:0/2096375890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc87ff60 0x55c6dcc25760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:39.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.527+0000 7f5c2de00640 1 --2- 172.21.15.134:0/2096375890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc8570a0 0x55c6dcc25ca0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5c200115f0 tx=0x7f5c20011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83f1648b2a3c8410 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:39.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.528+0000 7f5c26a00640 1 --2- 172.21.15.134:0/2880596486 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5c140b3c20 0x7f5c140b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.528+0000 7f5c1b000640 1 -- 172.21.15.134:0/2096375890 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2001e070 con 0x55c6dc8570a0 2024-07-26T11:14:39.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.528+0000 7f5c1b000640 1 -- 172.21.15.134:0/2096375890 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c2000faf0 con 0x55c6dc8570a0 2024-07-26T11:14:39.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.528+0000 7f5c1b000640 1 -- 172.21.15.134:0/2096375890 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c20019b10 con 0x55c6dc8570a0 2024-07-26T11:14:39.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.528+0000 7f5c26a00640 1 -- 172.21.15.134:0/2880596486 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5c1c0e43f0 con 0x7f5c3010f330 2024-07-26T11:14:39.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.528+0000 7f5c2e800640 1 --2- 172.21.15.134:0/2880596486 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5c140b3c20 0x7f5c140b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.529+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/2096375890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc8570a0 msgr2=0x55c6dcc25ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:39.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.529+0000 7f5c2e800640 1 --2- 172.21.15.134:0/2880596486 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5c140b3c20 0x7f5c140b6130 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5c28006010 tx=0x7f5c28005d80 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:39.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.529+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/2096375890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc8570a0 0x55c6dcc25ca0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5c200115f0 tx=0x7f5c20011ab0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.529+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/2096375890 shutdown_connections 2024-07-26T11:14:39.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.529+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/2096375890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc8570a0 0x55c6dcc25ca0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.529+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/2096375890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc85f280 0x55c6dcc261e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.529+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/2096375890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc87ff60 0x55c6dcc25760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.529+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/2096375890 >> 172.21.15.134:0/2096375890 conn(0x55c6dc874350 msgr2=0x55c6dcc23650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:39.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.529+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/2096375890 shutdown_connections 2024-07-26T11:14:39.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.529+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/2096375890 wait complete. 2024-07-26T11:14:39.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.530+0000 7f5c3b6d5740 1 -- start start 2024-07-26T11:14:39.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.530+0000 7f5c3b6d5740 1 Processor -- start 2024-07-26T11:14:39.535 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.531+0000 7f5c3b6d5740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc8570a0 0x55c6dcc30f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.531+0000 7f5c3b6d5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc314f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.531+0000 7f5c3b6d5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc87ff60 0x55c6dcc3d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.531+0000 7f5c3b6d5740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c6dc9ee5f0 con 0x55c6dc8570a0 2024-07-26T11:14:39.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.531+0000 7f5c3b6d5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c6dcb27520 con 0x55c6dc87ff60 2024-07-26T11:14:39.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.531+0000 7f5c3b6d5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c6dcc33f40 con 0x55c6dc85f280 2024-07-26T11:14:39.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.531+0000 7f5c3b6d5740 10 client.0 _ll_register_callbacks cb 0x55c6dcb36ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:14:39.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.531+0000 7f5c2d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc314f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.531+0000 7f5c2d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc314f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:60294/0 (socket says 172.21.15.134:60294) 2024-07-26T11:14:39.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.531+0000 7f5c2d400640 1 -- 172.21.15.134:0/3159527633 learned_addr learned my addr 172.21.15.134:0/3159527633 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:39.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.531+0000 7f5c2de00640 1 --2- 172.21.15.134:0/3159527633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc8570a0 0x55c6dcc30f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.531+0000 7f5c2e800640 1 --2- 172.21.15.134:0/3159527633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc87ff60 0x55c6dcc3d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.532+0000 7f5c2d400640 1 -- 172.21.15.134:0/3159527633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc87ff60 msgr2=0x55c6dcc3d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:39.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.532+0000 7f5c2d400640 1 --2- 172.21.15.134:0/3159527633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc87ff60 0x55c6dcc3d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.532+0000 7f5c2d400640 1 -- 172.21.15.134:0/3159527633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc8570a0 msgr2=0x55c6dcc30f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:39.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.532+0000 7f5c2d400640 1 --2- 172.21.15.134:0/3159527633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc8570a0 0x55c6dcc30f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.532+0000 7f5c2d400640 1 -- 172.21.15.134:0/3159527633 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6dc8eb540 con 0x55c6dc85f280 2024-07-26T11:14:39.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.532+0000 7f5c2d400640 1 --2- 172.21.15.134:0/3159527633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc314f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c0e4690 tx=0x7f5c1c0aa890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0438c42e4504ae5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:39.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.532+0000 7f5c0f400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:14:39.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.532+0000 7f5c0f400640 1 -- 172.21.15.134:0/3159527633 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c0ad070 con 0x55c6dc85f280 2024-07-26T11:14:39.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.532+0000 7f5c0f400640 1 -- 172.21.15.134:0/3159527633 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5c1c0e4170 con 0x55c6dc85f280 2024-07-26T11:14:39.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.532+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/3159527633 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c2000faf0 con 0x55c6dc85f280 2024-07-26T11:14:39.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.533+0000 7f5c0f400640 1 -- 172.21.15.134:0/3159527633 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5c1c005680 con 0x55c6dc85f280 2024-07-26T11:14:39.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.533+0000 7f5c3b6d5740 10 client.25252 fetch_fsmap learned FSMap version 29 2024-07-26T11:14:39.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.533+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/3159527633 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c6dcc3fe00 con 0x55c6dc85f280 2024-07-26T11:14:39.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.533+0000 7f5c0f400640 1 -- 172.21.15.134:0/3159527633 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c2000faf0 con 0x55c6dc85f280 2024-07-26T11:14:39.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.533+0000 7f5c0f400640 1 -- 172.21.15.134:0/3159527633 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5c1c0b3a60 con 0x55c6dc85f280 2024-07-26T11:14:39.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.533+0000 7f5c3b6d5740 10 client.25252 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:14:39.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.533+0000 7f5c3b6d5740 10 client.25252 resolve_mds: resolved name 'a' to daemon mds.a 2024-07-26T11:14:39.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.533+0000 7f5c3b6d5740 20 client.25252 populate_metadata read hostname 'smithi134' 2024-07-26T11:14:39.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.533+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/3159527633 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c6dc83e8e0 0x55c6dcc438a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.533+0000 7f5c3b6d5740 4 client.25252 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:14:39.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.533+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/3159527633 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6dcb9d8a0 con 0x55c6dc83e8e0 2024-07-26T11:14:39.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.534+0000 7f5c2de00640 1 --2- 172.21.15.134:0/3159527633 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c6dc83e8e0 0x55c6dcc438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.534+0000 7f5c2de00640 1 --2- 172.21.15.134:0/3159527633 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c6dc83e8e0 0x55c6dcc438a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f272c721d117c0a8 server_cookie=cb09cf2d35dd4ac3 in_seq=0 out_seq=0 2024-07-26T11:14:39.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.534+0000 7f5c0f400640 10 client.25252 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:14:39.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.535+0000 7f5c0f400640 1 -- 172.21.15.134:0/3159527633 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c6dcb9d8a0 con 0x55c6dc83e8e0 2024-07-26T11:14:39.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.535+0000 7f5c0f400640 10 client.25252 handle_command_reply: tid=0 2024-07-26T11:14:39.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.535+0000 7f5c3b6d5740 1 client.25252 shutdown 2024-07-26T11:14:39.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.535+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/3159527633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 msgr2=0x55c6dcc314f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:39.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.535+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/3159527633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc314f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c0e4690 tx=0x7f5c1c0aa890 comp rx=0 tx=0).stop 2024-07-26T11:14:39.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.536+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/3159527633 shutdown_connections 2024-07-26T11:14:39.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.536+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/3159527633 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c6dc83e8e0 0x55c6dcc438a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.536+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/3159527633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc314f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.536+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/3159527633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc87ff60 0x55c6dcc3d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.536+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/3159527633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc8570a0 0x55c6dcc30f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.536+0000 7f5c0f400640 0 client.25252 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:14:39.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.536+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/3159527633 >> 172.21.15.134:0/3159527633 conn(0x55c6dc874350 msgr2=0x55c6dcc28940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:39.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.536+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/3159527633 shutdown_connections 2024-07-26T11:14:39.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.536+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/3159527633 wait complete. 2024-07-26T11:14:39.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.536+0000 7f5c3b6d5740 20 client.25252 trim_cache size 0 max 16384 2024-07-26T11:14:39.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.556+0000 7f5c3b6d5740 1 Processor -- start 2024-07-26T11:14:39.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.556+0000 7f5c3b6d5740 1 -- start start 2024-07-26T11:14:39.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.556+0000 7f5c3b6d5740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc8570a0 0x55c6dcc3d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.556+0000 7f5c3b6d5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc3d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.556+0000 7f5c3b6d5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc87ff60 0x55c6dcc2f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.556+0000 7f5c3b6d5740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c6dcc41be0 con 0x55c6dc8570a0 2024-07-26T11:14:39.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.556+0000 7f5c3b6d5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c6dcc422b0 con 0x55c6dc87ff60 2024-07-26T11:14:39.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.556+0000 7f5c3b6d5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c6dcc42980 con 0x55c6dc85f280 2024-07-26T11:14:39.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.556+0000 7f5c2de00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc8570a0 0x55c6dcc3d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.556+0000 7f5c2de00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc8570a0 0x55c6dcc3d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57978/0 (socket says 172.21.15.134:57978) 2024-07-26T11:14:39.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.556+0000 7f5c2de00640 1 -- 172.21.15.134:0/1777769521 learned_addr learned my addr 172.21.15.134:0/1777769521 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:39.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.557+0000 7f5c2d400640 1 --2- 172.21.15.134:0/1777769521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc3d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.557+0000 7f5c2e800640 1 --2- 172.21.15.134:0/1777769521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc87ff60 0x55c6dcc2f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.557+0000 7f5c2de00640 1 -- 172.21.15.134:0/1777769521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 msgr2=0x55c6dcc3d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:39.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.557+0000 7f5c2de00640 1 --2- 172.21.15.134:0/1777769521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc3d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.557+0000 7f5c2de00640 1 -- 172.21.15.134:0/1777769521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc87ff60 msgr2=0x55c6dcc2f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:39.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.557+0000 7f5c2de00640 1 --2- 172.21.15.134:0/1777769521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc87ff60 0x55c6dcc2f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.557+0000 7f5c2de00640 1 -- 172.21.15.134:0/1777769521 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6dcc276a0 con 0x55c6dc8570a0 2024-07-26T11:14:39.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.557+0000 7f5c2d400640 1 --2- 172.21.15.134:0/1777769521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc3d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:39.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.557+0000 7f5c2de00640 1 --2- 172.21.15.134:0/1777769521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc8570a0 0x55c6dcc3d1a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5c20019780 tx=0x7f5c200197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19465133c9e1709c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:39.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.558+0000 7f5c1a600640 1 -- 172.21.15.134:0/1777769521 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2001e040 con 0x55c6dc8570a0 2024-07-26T11:14:39.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.558+0000 7f5c1a600640 1 -- 172.21.15.134:0/1777769521 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c20029030 con 0x55c6dc8570a0 2024-07-26T11:14:39.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.558+0000 7f5c1a600640 1 -- 172.21.15.134:0/1777769521 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c200054e0 con 0x55c6dc8570a0 2024-07-26T11:14:39.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.559+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/1777769521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc8570a0 msgr2=0x55c6dcc3d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:39.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.559+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/1777769521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc8570a0 0x55c6dcc3d1a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5c20019780 tx=0x7f5c200197b0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.559+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/1777769521 shutdown_connections 2024-07-26T11:14:39.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.559+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/1777769521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc3d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.559+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/1777769521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc87ff60 0x55c6dcc2f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.559+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/1777769521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc8570a0 0x55c6dcc3d1a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.559+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/1777769521 >> 172.21.15.134:0/1777769521 conn(0x55c6dc874350 msgr2=0x55c6dcc2c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:39.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.559+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/1777769521 shutdown_connections 2024-07-26T11:14:39.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.559+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/1777769521 wait complete. 2024-07-26T11:14:39.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.560+0000 7f5c3b6d5740 1 -- start start 2024-07-26T11:14:39.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.560+0000 7f5c3b6d5740 1 Processor -- start 2024-07-26T11:14:39.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.560+0000 7f5c3b6d5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc8570a0 0x55c6dcc4c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.560+0000 7f5c3b6d5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc49bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.560+0000 7f5c3b6d5740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc87ff60 0x55c6dcc4a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.560+0000 7f5c3b6d5740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c6dcc43720 con 0x55c6dc87ff60 2024-07-26T11:14:39.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.560+0000 7f5c3b6d5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c6dcc43050 con 0x55c6dc8570a0 2024-07-26T11:14:39.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.560+0000 7f5c3b6d5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c6dcc43df0 con 0x55c6dc85f280 2024-07-26T11:14:39.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.560+0000 7f5c3b6d5740 10 client.0 _ll_register_callbacks cb 0x7f5c1c0aac10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:14:39.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.561+0000 7f5c2d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc49bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.561+0000 7f5c2de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc8570a0 0x55c6dcc4c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.561+0000 7f5c2d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc49bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:60322/0 (socket says 172.21.15.134:60322) 2024-07-26T11:14:39.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.561+0000 7f5c2d400640 1 -- 172.21.15.134:0/2512734024 learned_addr learned my addr 172.21.15.134:0/2512734024 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:39.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.561+0000 7f5c2e800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc87ff60 0x55c6dcc4a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.561+0000 7f5c2d400640 1 -- 172.21.15.134:0/2512734024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc8570a0 msgr2=0x55c6dcc4c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:39.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.561+0000 7f5c2d400640 1 --2- 172.21.15.134:0/2512734024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc8570a0 0x55c6dcc4c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.561+0000 7f5c2d400640 1 -- 172.21.15.134:0/2512734024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc87ff60 msgr2=0x55c6dcc4a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:14:39.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.561+0000 7f5c2d400640 1 --2- 172.21.15.134:0/2512734024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc87ff60 0x55c6dcc4a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.561+0000 7f5c2d400640 1 -- 172.21.15.134:0/2512734024 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6dcc4a670 con 0x55c6dc85f280 2024-07-26T11:14:39.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.562+0000 7f5c2d400640 1 --2- 172.21.15.134:0/2512734024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc49bf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c004650 tx=0x7f5c1c0a6d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c91e641db00767d0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:39.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.562+0000 7f5c0f400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:14:39.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.562+0000 7f5c0f400640 1 -- 172.21.15.134:0/2512734024 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c0ad070 con 0x55c6dc85f280 2024-07-26T11:14:39.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.562+0000 7f5c0f400640 1 -- 172.21.15.134:0/2512734024 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5c1c0aa2b0 con 0x55c6dc85f280 2024-07-26T11:14:39.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.562+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/2512734024 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c20029030 con 0x55c6dc85f280 2024-07-26T11:14:39.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.562+0000 7f5c0f400640 1 -- 172.21.15.134:0/2512734024 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5c1c0aa950 con 0x55c6dc85f280 2024-07-26T11:14:39.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.562+0000 7f5c3b6d5740 10 client.25258 fetch_fsmap learned FSMap version 29 2024-07-26T11:14:39.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.562+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/2512734024 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c6dccab800 con 0x55c6dc85f280 2024-07-26T11:14:39.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.563+0000 7f5c0f400640 1 -- 172.21.15.134:0/2512734024 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c20029030 con 0x55c6dc85f280 2024-07-26T11:14:39.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.563+0000 7f5c0f400640 1 -- 172.21.15.134:0/2512734024 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5c1c0a9c10 con 0x55c6dc85f280 2024-07-26T11:14:39.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.563+0000 7f5c3b6d5740 10 client.25258 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:14:39.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.563+0000 7f5c3b6d5740 10 client.25258 resolve_mds: resolved name 'a' to daemon mds.a 2024-07-26T11:14:39.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.563+0000 7f5c3b6d5740 20 client.25258 populate_metadata read hostname 'smithi134' 2024-07-26T11:14:39.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.563+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/2512734024 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c6dc813c90 0x55c6dccaefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:39.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.563+0000 7f5c3b6d5740 4 client.25258 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:14:39.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.563+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/2512734024 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "status"}) -- 0x55c6dcb99340 con 0x55c6dc813c90 2024-07-26T11:14:39.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.564+0000 7f5c2de00640 1 --2- 172.21.15.134:0/2512734024 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c6dc813c90 0x55c6dccaefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:39.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.564+0000 7f5c2de00640 1 --2- 172.21.15.134:0/2512734024 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c6dc813c90 0x55c6dccaefa0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ad9e24221c5870c server_cookie=28aeeefb46dc3295 in_seq=0 out_seq=0 2024-07-26T11:14:39.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.564+0000 7f5c0f400640 10 client.25258 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:14:39.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.565+0000 7f5c0f400640 1 -- 172.21.15.134:0/2512734024 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55c6dcb99340 con 0x55c6dc813c90 2024-07-26T11:14:39.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.565+0000 7f5c0f400640 10 client.25258 handle_command_reply: tid=0 2024-07-26T11:14:39.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.565+0000 7f5c3b6d5740 1 client.25258 shutdown 2024-07-26T11:14:39.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.565+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/2512734024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 msgr2=0x55c6dcc49bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:39.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.565+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/2512734024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc49bf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c004650 tx=0x7f5c1c0a6d00 comp rx=0 tx=0).stop 2024-07-26T11:14:39.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.565+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/2512734024 shutdown_connections 2024-07-26T11:14:39.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.565+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/2512734024 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c6dc813c90 0x55c6dccaefa0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.565+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/2512734024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c6dc85f280 0x55c6dcc49bf0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.565+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/2512734024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c6dc87ff60 0x55c6dcc4a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.565+0000 7f5c3b6d5740 1 --2- 172.21.15.134:0/2512734024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c6dc8570a0 0x55c6dcc4c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.565+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/2512734024 >> 172.21.15.134:0/2512734024 conn(0x55c6dc874350 msgr2=0x55c6dcc22e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:39.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.565+0000 7f5c0f400640 0 client.25258 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:14:39.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.565+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/2512734024 shutdown_connections 2024-07-26T11:14:39.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.565+0000 7f5c3b6d5740 1 -- 172.21.15.134:0/2512734024 wait complete. 2024-07-26T11:14:39.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.566+0000 7f5c3b6d5740 20 client.25258 trim_cache size 0 max 16384 2024-07-26T11:14:39.570 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:14:39.570 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_fsid": "105fbe5c-4b3e-11ef-bcb7-c7b262605968", 2024-07-26T11:14:39.570 INFO:teuthology.orchestra.run.smithi134.stdout: "whoami": 0, 2024-07-26T11:14:39.570 INFO:teuthology.orchestra.run.smithi134.stdout: "id": 24721, 2024-07-26T11:14:39.570 INFO:teuthology.orchestra.run.smithi134.stdout: "want_state": "up:active", 2024-07-26T11:14:39.570 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "up:active", 2024-07-26T11:14:39.571 INFO:teuthology.orchestra.run.smithi134.stdout: "fs_name": "cephfs", 2024-07-26T11:14:39.571 INFO:teuthology.orchestra.run.smithi134.stdout: "rank_uptime": 254.00716016600001, 2024-07-26T11:14:39.571 INFO:teuthology.orchestra.run.smithi134.stdout: "mdsmap_epoch": 29, 2024-07-26T11:14:39.571 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch": 90, 2024-07-26T11:14:39.571 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T11:14:39.571 INFO:teuthology.orchestra.run.smithi134.stdout: "uptime": 279.54793408799998 2024-07-26T11:14:39.571 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:14:39.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.566+0000 7f5c2f200640 1 -- 172.21.15.134:0/2880596486 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5c140b3c20 msgr2=0x7f5c140b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:39.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.567+0000 7f5c2f200640 1 --2- 172.21.15.134:0/2880596486 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5c140b3c20 0x7f5c140b6130 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5c28006010 tx=0x7f5c28005d80 comp rx=0 tx=0).stop 2024-07-26T11:14:39.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.567+0000 7f5c2f200640 1 -- 172.21.15.134:0/2880596486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c3010f330 msgr2=0x7f5c301a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:39.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.567+0000 7f5c2f200640 1 --2- 172.21.15.134:0/2880596486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c3010f330 0x7f5c301a08a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00e9c0 tx=0x7f5c1c00ee80 comp rx=0 tx=0).stop 2024-07-26T11:14:39.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.567+0000 7f5c2f200640 1 -- 172.21.15.134:0/2880596486 shutdown_connections 2024-07-26T11:14:39.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.567+0000 7f5c2f200640 1 --2- 172.21.15.134:0/2880596486 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5c140b3c20 0x7f5c140b6130 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.567+0000 7f5c2f200640 1 --2- 172.21.15.134:0/2880596486 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c3010fc80 0x7f5c301a5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.567+0000 7f5c2f200640 1 --2- 172.21.15.134:0/2880596486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c3010f330 0x7f5c301a08a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.567+0000 7f5c2f200640 1 --2- 172.21.15.134:0/2880596486 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c301099f0 0x7f5c301a0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:39.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.567+0000 7f5c2f200640 1 -- 172.21.15.134:0/2880596486 >> 172.21.15.134:0/2880596486 conn(0x7f5c30078ff0 msgr2=0x7f5c30107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:39.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.567+0000 7f5c2f200640 1 -- 172.21.15.134:0/2880596486 shutdown_connections 2024-07-26T11:14:39.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:39.567+0000 7f5c2f200640 1 -- 172.21.15.134:0/2880596486 wait complete. 2024-07-26T11:14:40.467 INFO:tasks.cephadm:Setup complete, yielding 2024-07-26T11:14:40.467 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-07-26T11:14:40.477 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi134.front.sepia.ceph.com 2024-07-26T11:14:40.478 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch status' 2024-07-26T11:14:40.944 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:41.329 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:41 smithi134 ceph-mon[31966]: pgmap v441: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:41 smithi189 ceph-mon[36790]: pgmap v441: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:41 smithi193 ceph-mon[37473]: pgmap v441: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:42.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:42 smithi134 ceph-mon[31966]: pgmap v442: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:42 smithi189 ceph-mon[36790]: pgmap v442: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:42 smithi193 ceph-mon[37473]: pgmap v442: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:43.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.086+0000 7fb062400640 1 -- 172.21.15.134:0/2816962289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb05c1041f0 msgr2=0x7fb05c104600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:43.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.086+0000 7fb062400640 1 --2- 172.21.15.134:0/2816962289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb05c1041f0 0x7fb05c104600 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb050009900 tx=0x7fb05002a400 comp rx=0 tx=0).stop 2024-07-26T11:14:43.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.086+0000 7fb062400640 1 -- 172.21.15.134:0/2816962289 shutdown_connections 2024-07-26T11:14:43.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.086+0000 7fb062400640 1 --2- 172.21.15.134:0/2816962289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb05c104b40 0x7fb05c10d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:43.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.086+0000 7fb062400640 1 --2- 172.21.15.134:0/2816962289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb05c10d650 0x7fb05c10fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:43.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.086+0000 7fb062400640 1 --2- 172.21.15.134:0/2816962289 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb05c1041f0 0x7fb05c104600 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:43.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.086+0000 7fb062400640 1 -- 172.21.15.134:0/2816962289 >> 172.21.15.134:0/2816962289 conn(0x7fb05c0fdcf0 msgr2=0x7fb05c100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:43.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.086+0000 7fb062400640 1 -- 172.21.15.134:0/2816962289 shutdown_connections 2024-07-26T11:14:43.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.086+0000 7fb062400640 1 -- 172.21.15.134:0/2816962289 wait complete. 2024-07-26T11:14:43.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.088+0000 7fb062400640 1 Processor -- start 2024-07-26T11:14:43.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.088+0000 7fb062400640 1 -- start start 2024-07-26T11:14:43.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.088+0000 7fb062400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb05c1041f0 0x7fb05c19bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:43.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.088+0000 7fb062400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb05c104b40 0x7fb05c19c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:43.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.088+0000 7fb062400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb05c10d650 0x7fb05c1a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:43.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.088+0000 7fb062400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c1132e0 con 0x7fb05c104b40 2024-07-26T11:14:43.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.088+0000 7fb062400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c113160 con 0x7fb05c10d650 2024-07-26T11:14:43.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.088+0000 7fb062400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb05c113460 con 0x7fb05c1041f0 2024-07-26T11:14:43.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.089+0000 7fb05be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb05c104b40 0x7fb05c19c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:43.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.089+0000 7fb05be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb05c104b40 0x7fb05c19c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58960/0 (socket says 172.21.15.134:58960) 2024-07-26T11:14:43.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.089+0000 7fb05be00640 1 -- 172.21.15.134:0/886113592 learned_addr learned my addr 172.21.15.134:0/886113592 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:43.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.089+0000 7fb061000640 1 --2- 172.21.15.134:0/886113592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb05c1041f0 0x7fb05c19bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:43.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.089+0000 7fb061a00640 1 --2- 172.21.15.134:0/886113592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb05c10d650 0x7fb05c1a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:43.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.089+0000 7fb05be00640 1 -- 172.21.15.134:0/886113592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb05c1041f0 msgr2=0x7fb05c19bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:43.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.089+0000 7fb05be00640 1 --2- 172.21.15.134:0/886113592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb05c1041f0 0x7fb05c19bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:43.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.089+0000 7fb05be00640 1 -- 172.21.15.134:0/886113592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb05c10d650 msgr2=0x7fb05c1a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:43.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.089+0000 7fb05be00640 1 --2- 172.21.15.134:0/886113592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb05c10d650 0x7fb05c1a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:43.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.089+0000 7fb05be00640 1 -- 172.21.15.134:0/886113592 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05c1a1e60 con 0x7fb05c104b40 2024-07-26T11:14:43.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.089+0000 7fb061a00640 1 --2- 172.21.15.134:0/886113592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb05c10d650 0x7fb05c1a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:43.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.090+0000 7fb05be00640 1 --2- 172.21.15.134:0/886113592 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb05c104b40 0x7fb05c19c490 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb04800e9c0 tx=0x7fb04800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:43.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.090+0000 7fb059600640 1 -- 172.21.15.134:0/886113592 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04800ccc0 con 0x7fb05c104b40 2024-07-26T11:14:43.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.090+0000 7fb062400640 1 -- 172.21.15.134:0/886113592 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb05c1a2150 con 0x7fb05c104b40 2024-07-26T11:14:43.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.090+0000 7fb059600640 1 -- 172.21.15.134:0/886113592 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0480054d0 con 0x7fb05c104b40 2024-07-26T11:14:43.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.090+0000 7fb059600640 1 -- 172.21.15.134:0/886113592 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb048010430 con 0x7fb05c104b40 2024-07-26T11:14:43.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.090+0000 7fb062400640 1 -- 172.21.15.134:0/886113592 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb05c1a88b0 con 0x7fb05c104b40 2024-07-26T11:14:43.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.091+0000 7fb062400640 1 -- 172.21.15.134:0/886113592 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb05c104600 con 0x7fb05c104b40 2024-07-26T11:14:43.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.094+0000 7fb059600640 1 -- 172.21.15.134:0/886113592 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fb048005020 con 0x7fb05c104b40 2024-07-26T11:14:43.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.096+0000 7fb059600640 1 --2- 172.21.15.134:0/886113592 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb0440b3ca0 0x7fb0440b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:43.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.096+0000 7fb059600640 1 -- 172.21.15.134:0/886113592 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb0480e45f0 con 0x7fb05c104b40 2024-07-26T11:14:43.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.096+0000 7fb061000640 1 --2- 172.21.15.134:0/886113592 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb0440b3ca0 0x7fb0440b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:43.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.097+0000 7fb061000640 1 --2- 172.21.15.134:0/886113592 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb0440b3ca0 0x7fb0440b61b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb050031440 tx=0x7fb050009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:43.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.097+0000 7fb059600640 1 -- 172.21.15.134:0/886113592 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb048055600 con 0x7fb05c104b40 2024-07-26T11:14:43.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.315+0000 7fb062400640 1 -- 172.21.15.134:0/886113592 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7fb05c102570 con 0x7fb0440b3ca0 2024-07-26T11:14:43.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.317+0000 7fb059600640 1 -- 172.21.15.134:0/886113592 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7fb05c102570 con 0x7fb0440b3ca0 2024-07-26T11:14:43.320 INFO:teuthology.orchestra.run.smithi134.stdout:Backend: cephadm 2024-07-26T11:14:43.320 INFO:teuthology.orchestra.run.smithi134.stdout:Available: Yes 2024-07-26T11:14:43.320 INFO:teuthology.orchestra.run.smithi134.stdout:Paused: No 2024-07-26T11:14:43.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.320+0000 7fb062400640 1 -- 172.21.15.134:0/886113592 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb0440b3ca0 msgr2=0x7fb0440b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:43.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.320+0000 7fb062400640 1 --2- 172.21.15.134:0/886113592 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb0440b3ca0 0x7fb0440b61b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb050031440 tx=0x7fb050009560 comp rx=0 tx=0).stop 2024-07-26T11:14:43.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.320+0000 7fb062400640 1 -- 172.21.15.134:0/886113592 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb05c104b40 msgr2=0x7fb05c19c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:43.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.320+0000 7fb062400640 1 --2- 172.21.15.134:0/886113592 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb05c104b40 0x7fb05c19c490 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb04800e9c0 tx=0x7fb04800ee80 comp rx=0 tx=0).stop 2024-07-26T11:14:43.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.321+0000 7fb062400640 1 -- 172.21.15.134:0/886113592 shutdown_connections 2024-07-26T11:14:43.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.321+0000 7fb062400640 1 --2- 172.21.15.134:0/886113592 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb0440b3ca0 0x7fb0440b61b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:43.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.321+0000 7fb062400640 1 --2- 172.21.15.134:0/886113592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb05c10d650 0x7fb05c1a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:43.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.321+0000 7fb062400640 1 --2- 172.21.15.134:0/886113592 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb05c104b40 0x7fb05c19c490 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:43.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.321+0000 7fb062400640 1 --2- 172.21.15.134:0/886113592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb05c1041f0 0x7fb05c19bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:43.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.321+0000 7fb062400640 1 -- 172.21.15.134:0/886113592 >> 172.21.15.134:0/886113592 conn(0x7fb05c0fdcf0 msgr2=0x7fb05c101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:43.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.321+0000 7fb062400640 1 -- 172.21.15.134:0/886113592 shutdown_connections 2024-07-26T11:14:43.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:43.321+0000 7fb062400640 1 -- 172.21.15.134:0/886113592 wait complete. 2024-07-26T11:14:44.178 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch ps' 2024-07-26T11:14:44.651 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:45 smithi134 ceph-mon[31966]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:14:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:45 smithi134 ceph-mon[31966]: pgmap v443: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:45 smithi189 ceph-mon[36790]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:14:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:45 smithi189 ceph-mon[36790]: pgmap v443: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:45 smithi193 ceph-mon[37473]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:14:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:45 smithi193 ceph-mon[37473]: pgmap v443: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:46.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.996+0000 7fb16dc00640 1 -- 172.21.15.134:0/1281124521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1681055d0 msgr2=0x7fb1681059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:46.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.996+0000 7fb16dc00640 1 --2- 172.21.15.134:0/1281124521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1681055d0 0x7fb1681059e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb158009900 tx=0x7fb15802a410 comp rx=0 tx=0).stop 2024-07-26T11:14:46.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.996+0000 7fb16dc00640 1 -- 172.21.15.134:0/1281124521 shutdown_connections 2024-07-26T11:14:46.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.996+0000 7fb16dc00640 1 --2- 172.21.15.134:0/1281124521 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb16810af10 0x7fb16810b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:46.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.996+0000 7fb16dc00640 1 --2- 172.21.15.134:0/1281124521 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb16810b860 0x7fb168112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:46.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.996+0000 7fb16dc00640 1 --2- 172.21.15.134:0/1281124521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1681055d0 0x7fb1681059e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:46.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.996+0000 7fb16dc00640 1 -- 172.21.15.134:0/1281124521 >> 172.21.15.134:0/1281124521 conn(0x7fb168101040 msgr2=0x7fb168103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:46.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.996+0000 7fb16dc00640 1 -- 172.21.15.134:0/1281124521 shutdown_connections 2024-07-26T11:14:46.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.996+0000 7fb16dc00640 1 -- 172.21.15.134:0/1281124521 wait complete. 2024-07-26T11:14:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.997+0000 7fb16dc00640 1 Processor -- start 2024-07-26T11:14:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.998+0000 7fb16dc00640 1 -- start start 2024-07-26T11:14:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.998+0000 7fb16dc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1681055d0 0x7fb16819bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.998+0000 7fb16dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb16810af10 0x7fb16819c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.998+0000 7fb16dc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb16810b860 0x7fb1681a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.998+0000 7fb16dc00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb168115510 con 0x7fb1681055d0 2024-07-26T11:14:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.998+0000 7fb16dc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb168115390 con 0x7fb16810af10 2024-07-26T11:14:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.998+0000 7fb16dc00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb168115690 con 0x7fb16810b860 2024-07-26T11:14:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.998+0000 7fb167e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1681055d0 0x7fb16819bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.998+0000 7fb167e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1681055d0 0x7fb16819bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58986/0 (socket says 172.21.15.134:58986) 2024-07-26T11:14:47.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.998+0000 7fb167e00640 1 -- 172.21.15.134:0/2422539322 learned_addr learned my addr 172.21.15.134:0/2422539322 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.998+0000 7fb167400640 1 --2- 172.21.15.134:0/2422539322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb16810af10 0x7fb16819c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.999+0000 7fb16d200640 1 --2- 172.21.15.134:0/2422539322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb16810b860 0x7fb1681a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.999+0000 7fb167e00640 1 -- 172.21.15.134:0/2422539322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb16810b860 msgr2=0x7fb1681a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.999+0000 7fb167e00640 1 --2- 172.21.15.134:0/2422539322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb16810b860 0x7fb1681a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.999+0000 7fb167e00640 1 -- 172.21.15.134:0/2422539322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb16810af10 msgr2=0x7fb16819c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.999+0000 7fb167e00640 1 --2- 172.21.15.134:0/2422539322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb16810af10 0x7fb16819c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.999+0000 7fb167e00640 1 -- 172.21.15.134:0/2422539322 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1681a1e70 con 0x7fb1681055d0 2024-07-26T11:14:47.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:46.999+0000 7fb167e00640 1 --2- 172.21.15.134:0/2422539322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1681055d0 0x7fb16819bf00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb158005940 tx=0x7fb1580040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:47.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.000+0000 7fb164c00640 1 -- 172.21.15.134:0/2422539322 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb15803f030 con 0x7fb1681055d0 2024-07-26T11:14:47.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.000+0000 7fb16dc00640 1 -- 172.21.15.134:0/2422539322 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1681a2100 con 0x7fb1681055d0 2024-07-26T11:14:47.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.000+0000 7fb164c00640 1 -- 172.21.15.134:0/2422539322 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1580044c0 con 0x7fb1681055d0 2024-07-26T11:14:47.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.000+0000 7fb16dc00640 1 -- 172.21.15.134:0/2422539322 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1681a88c0 con 0x7fb1681055d0 2024-07-26T11:14:47.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.000+0000 7fb164c00640 1 -- 172.21.15.134:0/2422539322 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb158049060 con 0x7fb1681055d0 2024-07-26T11:14:47.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.001+0000 7fb16dc00640 1 -- 172.21.15.134:0/2422539322 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1681059e0 con 0x7fb1681055d0 2024-07-26T11:14:47.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.005+0000 7fb164c00640 1 -- 172.21.15.134:0/2422539322 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fb158042030 con 0x7fb1681055d0 2024-07-26T11:14:47.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.007+0000 7fb164c00640 1 --2- 172.21.15.134:0/2422539322 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb14c0b3c20 0x7fb14c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:47.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.007+0000 7fb164c00640 1 -- 172.21.15.134:0/2422539322 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb158104b20 con 0x7fb1681055d0 2024-07-26T11:14:47.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.007+0000 7fb167400640 1 --2- 172.21.15.134:0/2422539322 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb14c0b3c20 0x7fb14c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:47.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.008+0000 7fb167400640 1 --2- 172.21.15.134:0/2422539322 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb14c0b3c20 0x7fb14c0b6130 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb16810aac0 tx=0x7fb154005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:47.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.009+0000 7fb164c00640 1 -- 172.21.15.134:0/2422539322 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb1580cb620 con 0x7fb1681055d0 2024-07-26T11:14:47.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.217+0000 7fb16dc00640 1 -- 172.21.15.134:0/2422539322 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fb168063230 con 0x7fb14c0b3c20 2024-07-26T11:14:47.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.252+0000 7fb164c00640 1 -- 172.21.15.134:0/2422539322 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fb168063230 con 0x7fb14c0b3c20 2024-07-26T11:14:47.255 INFO:teuthology.orchestra.run.smithi134.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-07-26T11:14:47.255 INFO:teuthology.orchestra.run.smithi134.stdout:mds.a smithi134 running (4m) 2m ago 4m 92.7M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa f5e056356436 2024-07-26T11:14:47.255 INFO:teuthology.orchestra.run.smithi134.stdout:mds.b smithi189 running (4m) 2m ago 4m 85.2M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa eacc34abe1c2 2024-07-26T11:14:47.256 INFO:teuthology.orchestra.run.smithi134.stdout:mds.c smithi193 running (5m) 2m ago 5m 88.0M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa a2f99434d2f1 2024-07-26T11:14:47.256 INFO:teuthology.orchestra.run.smithi134.stdout:mds.d smithi134 running (5m) 2m ago 5m 80.7M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa c8e744686ff3 2024-07-26T11:14:47.256 INFO:teuthology.orchestra.run.smithi134.stdout:mds.e smithi189 running (5m) 2m ago 5m 83.7M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa ece0fd5f8679 2024-07-26T11:14:47.256 INFO:teuthology.orchestra.run.smithi134.stdout:mds.f smithi193 running (5m) 2m ago 5m 87.5M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 3091e8a1a2db 2024-07-26T11:14:47.256 INFO:teuthology.orchestra.run.smithi134.stdout:mds.g smithi134 running (4m) 2m ago 4m 80.7M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa c7b6500323c2 2024-07-26T11:14:47.256 INFO:teuthology.orchestra.run.smithi134.stdout:mds.h smithi189 running (4m) 2m ago 4m 88.1M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa ce38a5f212ef 2024-07-26T11:14:47.256 INFO:teuthology.orchestra.run.smithi134.stdout:mds.i smithi193 running (4m) 2m ago 4m 86.0M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa ac062bd4a9db 2024-07-26T11:14:47.256 INFO:teuthology.orchestra.run.smithi134.stdout:mds.j smithi134 running (5m) 2m ago 5m 82.7M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa d46577b3d6ca 2024-07-26T11:14:47.256 INFO:teuthology.orchestra.run.smithi134.stdout:mds.k smithi189 running (5m) 2m ago 5m 83.4M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 15508beef44f 2024-07-26T11:14:47.256 INFO:teuthology.orchestra.run.smithi134.stdout:mds.l smithi193 running (5m) 2m ago 5m 84.5M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa b6d485e463b9 2024-07-26T11:14:47.256 INFO:teuthology.orchestra.run.smithi134.stdout:mgr.x smithi134 *:9283,8765 running (14m) 2m ago 14m 584M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 55c0ab3a28af 2024-07-26T11:14:47.256 INFO:teuthology.orchestra.run.smithi134.stdout:mgr.y smithi189 *:8443,8765 running (12m) 2m ago 12m 526M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa 1f89592e7558 2024-07-26T11:14:47.257 INFO:teuthology.orchestra.run.smithi134.stdout:mgr.z smithi193 *:8443,8765 running (12m) 2m ago 12m 525M - 19.0.0-5320-ge8f88bb1 ae51f432d4fa fbb78e852201 2024-07-26T11:14:47.257 INFO:teuthology.orchestra.run.smithi134.stdout:mon.a smithi134 running (14m) 2m ago 14m 118M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 38a9d5573935 2024-07-26T11:14:47.257 INFO:teuthology.orchestra.run.smithi134.stdout:mon.b smithi189 running (12m) 2m ago 12m 111M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 8b89706869d6 2024-07-26T11:14:47.257 INFO:teuthology.orchestra.run.smithi134.stdout:mon.c smithi193 running (12m) 2m ago 12m 114M 2048M 19.0.0-5320-ge8f88bb1 ae51f432d4fa d84de133b420 2024-07-26T11:14:47.257 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0 smithi134 running (10m) 2m ago 10m 121M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 399f2f835cec 2024-07-26T11:14:47.257 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1 smithi189 running (10m) 2m ago 10m 116M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 1a2d88b0baf1 2024-07-26T11:14:47.257 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2 smithi193 running (9m) 2m ago 9m 114M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 572b22f14b87 2024-07-26T11:14:47.257 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3 smithi134 running (9m) 2m ago 9m 117M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 66a61e9c4f7d 2024-07-26T11:14:47.257 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4 smithi189 running (8m) 2m ago 8m 112M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 6be2f870b619 2024-07-26T11:14:47.257 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5 smithi193 running (8m) 2m ago 8m 121M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa fb97092716d5 2024-07-26T11:14:47.257 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6 smithi134 running (7m) 2m ago 7m 111M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 429f277dfa4e 2024-07-26T11:14:47.257 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7 smithi189 running (7m) 2m ago 7m 112M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 5c5704f37235 2024-07-26T11:14:47.257 INFO:teuthology.orchestra.run.smithi134.stdout:osd.8 smithi193 running (7m) 2m ago 7m 118M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 67268dab0b68 2024-07-26T11:14:47.257 INFO:teuthology.orchestra.run.smithi134.stdout:osd.9 smithi134 running (6m) 2m ago 6m 117M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 67f2e34344ef 2024-07-26T11:14:47.258 INFO:teuthology.orchestra.run.smithi134.stdout:osd.10 smithi189 running (6m) 2m ago 6m 114M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa 412a1ca028c0 2024-07-26T11:14:47.258 INFO:teuthology.orchestra.run.smithi134.stdout:osd.11 smithi193 running (5m) 2m ago 5m 110M 4272M 19.0.0-5320-ge8f88bb1 ae51f432d4fa ab89234769cc 2024-07-26T11:14:47.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.256+0000 7fb16dc00640 1 -- 172.21.15.134:0/2422539322 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb14c0b3c20 msgr2=0x7fb14c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:47.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.256+0000 7fb16dc00640 1 --2- 172.21.15.134:0/2422539322 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb14c0b3c20 0x7fb14c0b6130 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb16810aac0 tx=0x7fb154005f90 comp rx=0 tx=0).stop 2024-07-26T11:14:47.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.256+0000 7fb16dc00640 1 -- 172.21.15.134:0/2422539322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1681055d0 msgr2=0x7fb16819bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:47.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.256+0000 7fb16dc00640 1 --2- 172.21.15.134:0/2422539322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1681055d0 0x7fb16819bf00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb158005940 tx=0x7fb1580040c0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.256+0000 7fb16dc00640 1 -- 172.21.15.134:0/2422539322 shutdown_connections 2024-07-26T11:14:47.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.256+0000 7fb16dc00640 1 --2- 172.21.15.134:0/2422539322 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb14c0b3c20 0x7fb14c0b6130 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.256+0000 7fb16dc00640 1 --2- 172.21.15.134:0/2422539322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb16810b860 0x7fb1681a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.256+0000 7fb16dc00640 1 --2- 172.21.15.134:0/2422539322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb16810af10 0x7fb16819c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.256+0000 7fb16dc00640 1 --2- 172.21.15.134:0/2422539322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1681055d0 0x7fb16819bf00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:47.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.256+0000 7fb16dc00640 1 -- 172.21.15.134:0/2422539322 >> 172.21.15.134:0/2422539322 conn(0x7fb168101040 msgr2=0x7fb168103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:47.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.256+0000 7fb16dc00640 1 -- 172.21.15.134:0/2422539322 shutdown_connections 2024-07-26T11:14:47.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:47.256+0000 7fb16dc00640 1 -- 172.21.15.134:0/2422539322 wait complete. 2024-07-26T11:14:47.339 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:47 smithi134 ceph-mon[31966]: pgmap v444: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:47 smithi189 ceph-mon[36790]: pgmap v444: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:47 smithi193 ceph-mon[37473]: pgmap v444: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:48.185 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch ls' 2024-07-26T11:14:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:48 smithi134 ceph-mon[31966]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:14:48.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:48 smithi134 ceph-mon[31966]: pgmap v445: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:48 smithi189 ceph-mon[36790]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:14:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:48 smithi189 ceph-mon[36790]: pgmap v445: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:48 smithi193 ceph-mon[37473]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:14:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:48 smithi193 ceph-mon[37473]: pgmap v445: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:48.660 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:50.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.908+0000 7f9615c00640 1 -- 172.21.15.134:0/3578262922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9610104250 msgr2=0x7f9610104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:50.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.908+0000 7f9615c00640 1 --2- 172.21.15.134:0/3578262922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9610104250 0x7f9610104660 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9600009860 tx=0x7f960002a340 comp rx=0 tx=0).stop 2024-07-26T11:14:50.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.908+0000 7f9615c00640 1 -- 172.21.15.134:0/3578262922 shutdown_connections 2024-07-26T11:14:50.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.908+0000 7f9615c00640 1 --2- 172.21.15.134:0/3578262922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9610104ba0 0x7f961010d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:50.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.908+0000 7f9615c00640 1 --2- 172.21.15.134:0/3578262922 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f961010d6b0 0x7f961010fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:50.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.908+0000 7f9615c00640 1 --2- 172.21.15.134:0/3578262922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9610104250 0x7f9610104660 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:50.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.908+0000 7f9615c00640 1 -- 172.21.15.134:0/3578262922 >> 172.21.15.134:0/3578262922 conn(0x7f96100fddb0 msgr2=0x7f9610100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:50.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.909+0000 7f9615c00640 1 -- 172.21.15.134:0/3578262922 shutdown_connections 2024-07-26T11:14:50.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.909+0000 7f9615c00640 1 -- 172.21.15.134:0/3578262922 wait complete. 2024-07-26T11:14:50.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.910+0000 7f9615c00640 1 Processor -- start 2024-07-26T11:14:50.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.910+0000 7f9615c00640 1 -- start start 2024-07-26T11:14:50.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.910+0000 7f9615c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9610104250 0x7f9610101f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:50.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.910+0000 7f9615c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9610104ba0 0x7f96101024d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:50.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.910+0000 7f9615c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f961010d6b0 0x7f9610102a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:50.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.910+0000 7f9615c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9610113320 con 0x7f9610104ba0 2024-07-26T11:14:50.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.910+0000 7f9615c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f96101131a0 con 0x7f961010d6b0 2024-07-26T11:14:50.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.911+0000 7f9615c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f96101134a0 con 0x7f9610104250 2024-07-26T11:14:50.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.911+0000 7f960f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9610104ba0 0x7f96101024d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:50.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.911+0000 7f960fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9610104250 0x7f9610101f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:50.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.911+0000 7f960f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9610104ba0 0x7f96101024d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:45058/0 (socket says 172.21.15.134:45058) 2024-07-26T11:14:50.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.911+0000 7f960f400640 1 -- 172.21.15.134:0/3789648309 learned_addr learned my addr 172.21.15.134:0/3789648309 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:50.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.911+0000 7f9615200640 1 --2- 172.21.15.134:0/3789648309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f961010d6b0 0x7f9610102a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:50.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.911+0000 7f960fe00640 1 -- 172.21.15.134:0/3789648309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f961010d6b0 msgr2=0x7f9610102a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:50.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.911+0000 7f960fe00640 1 --2- 172.21.15.134:0/3789648309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f961010d6b0 0x7f9610102a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:50.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.911+0000 7f960fe00640 1 -- 172.21.15.134:0/3789648309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9610104ba0 msgr2=0x7f96101024d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:50.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.911+0000 7f960fe00640 1 --2- 172.21.15.134:0/3789648309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9610104ba0 0x7f96101024d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:50.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.911+0000 7f960fe00640 1 -- 172.21.15.134:0/3789648309 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96101a80b0 con 0x7f9610104250 2024-07-26T11:14:50.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.912+0000 7f960fe00640 1 --2- 172.21.15.134:0/3789648309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9610104250 0x7f9610101f90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9600035040 tx=0x7f9600032d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:50.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.912+0000 7f960cc00640 1 -- 172.21.15.134:0/3789648309 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9600040020 con 0x7f9610104250 2024-07-26T11:14:50.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.912+0000 7f9615c00640 1 -- 172.21.15.134:0/3789648309 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96101a8340 con 0x7f9610104250 2024-07-26T11:14:50.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.912+0000 7f960cc00640 1 -- 172.21.15.134:0/3789648309 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9600030d50 con 0x7f9610104250 2024-07-26T11:14:50.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.912+0000 7f960cc00640 1 -- 172.21.15.134:0/3789648309 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9600037040 con 0x7f9610104250 2024-07-26T11:14:50.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.912+0000 7f9615c00640 1 -- 172.21.15.134:0/3789648309 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96101a8820 con 0x7f9610104250 2024-07-26T11:14:50.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.913+0000 7f9615c00640 1 -- 172.21.15.134:0/3789648309 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9610104660 con 0x7f9610104250 2024-07-26T11:14:50.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.916+0000 7f960cc00640 1 -- 172.21.15.134:0/3789648309 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f9600043020 con 0x7f9610104250 2024-07-26T11:14:50.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.917+0000 7f960cc00640 1 --2- 172.21.15.134:0/3789648309 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95f00b3c20 0x7f95f00b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:50.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.917+0000 7f960cc00640 1 -- 172.21.15.134:0/3789648309 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f96001044a0 con 0x7f9610104250 2024-07-26T11:14:50.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.917+0000 7f960f400640 1 --2- 172.21.15.134:0/3789648309 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95f00b3c20 0x7f95f00b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:50.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.918+0000 7f960f400640 1 --2- 172.21.15.134:0/3789648309 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95f00b3c20 0x7f95f00b6130 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f961010fc60 tx=0x7f95f8009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:50.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:50.919+0000 7f960cc00640 1 -- 172.21.15.134:0/3789648309 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f960004a020 con 0x7f9610104250 2024-07-26T11:14:51.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:51.127+0000 7f9615c00640 1 -- 172.21.15.134:0/3789648309 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f9610107940 con 0x7f95f00b3c20 2024-07-26T11:14:51.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:51.143+0000 7f960cc00640 1 -- 172.21.15.134:0/3789648309 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f9610107940 con 0x7f95f00b3c20 2024-07-26T11:14:51.145 INFO:teuthology.orchestra.run.smithi134.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-07-26T11:14:51.145 INFO:teuthology.orchestra.run.smithi134.stdout:mds.all 12/12 2m ago 5m smithi134=a;smithi134=d;smithi134=g;smithi134=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-07-26T11:14:51.145 INFO:teuthology.orchestra.run.smithi134.stdout:mgr 3/3 2m ago 12m smithi134=x;smithi189=y;smithi193=z;count:3 2024-07-26T11:14:51.145 INFO:teuthology.orchestra.run.smithi134.stdout:mon 3/3 2m ago 12m smithi134:172.21.15.134=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c;count:3 2024-07-26T11:14:51.145 INFO:teuthology.orchestra.run.smithi134.stdout:osd 12 2m ago - 2024-07-26T11:14:51.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:51.146+0000 7f9615c00640 1 -- 172.21.15.134:0/3789648309 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95f00b3c20 msgr2=0x7f95f00b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:51.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:51.146+0000 7f9615c00640 1 --2- 172.21.15.134:0/3789648309 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95f00b3c20 0x7f95f00b6130 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f961010fc60 tx=0x7f95f8009500 comp rx=0 tx=0).stop 2024-07-26T11:14:51.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:51.146+0000 7f9615c00640 1 -- 172.21.15.134:0/3789648309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9610104250 msgr2=0x7f9610101f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:51.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:51.146+0000 7f9615c00640 1 --2- 172.21.15.134:0/3789648309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9610104250 0x7f9610101f90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9600035040 tx=0x7f9600032d50 comp rx=0 tx=0).stop 2024-07-26T11:14:51.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:51.146+0000 7f9615c00640 1 -- 172.21.15.134:0/3789648309 shutdown_connections 2024-07-26T11:14:51.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:51.146+0000 7f9615c00640 1 --2- 172.21.15.134:0/3789648309 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95f00b3c20 0x7f95f00b6130 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:51.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:51.146+0000 7f9615c00640 1 --2- 172.21.15.134:0/3789648309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f961010d6b0 0x7f9610102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:51.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:51.146+0000 7f9615c00640 1 --2- 172.21.15.134:0/3789648309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9610104ba0 0x7f96101024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:51.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:51.146+0000 7f9615c00640 1 --2- 172.21.15.134:0/3789648309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9610104250 0x7f9610101f90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:51.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:51.146+0000 7f9615c00640 1 -- 172.21.15.134:0/3789648309 >> 172.21.15.134:0/3789648309 conn(0x7f96100fddb0 msgr2=0x7f9610107230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:51.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:51.146+0000 7f9615c00640 1 -- 172.21.15.134:0/3789648309 shutdown_connections 2024-07-26T11:14:51.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:51.146+0000 7f9615c00640 1 -- 172.21.15.134:0/3789648309 wait complete. 2024-07-26T11:14:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:51 smithi134 ceph-mon[31966]: pgmap v446: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:51.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:51 smithi189 ceph-mon[36790]: pgmap v446: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:51 smithi193 ceph-mon[37473]: pgmap v446: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:52.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch host ls' 2024-07-26T11:14:52.565 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:53.232 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:53 smithi134 ceph-mon[31966]: from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:14:53.232 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:53 smithi134 ceph-mon[31966]: pgmap v447: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:53.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:53 smithi189 ceph-mon[36790]: from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:14:53.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:53 smithi189 ceph-mon[36790]: pgmap v447: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:53 smithi193 ceph-mon[37473]: from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:14:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:53 smithi193 ceph-mon[37473]: pgmap v447: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:14:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:54 smithi134 ceph-mon[31966]: pgmap v448: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:54 smithi189 ceph-mon[36790]: pgmap v448: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:54 smithi193 ceph-mon[37473]: pgmap v448: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:14:55.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.002+0000 7ff438c00640 1 -- 172.21.15.134:0/890021090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff434108d20 msgr2=0x7ff434109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:55.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.002+0000 7ff438c00640 1 --2- 172.21.15.134:0/890021090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff434108d20 0x7ff434109130 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff41c009900 tx=0x7ff41c02a410 comp rx=0 tx=0).stop 2024-07-26T11:14:55.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.002+0000 7ff438c00640 1 -- 172.21.15.134:0/890021090 shutdown_connections 2024-07-26T11:14:55.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.002+0000 7ff438c00640 1 --2- 172.21.15.134:0/890021090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff434109670 0x7ff43410fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:55.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.002+0000 7ff438c00640 1 --2- 172.21.15.134:0/890021090 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff434108d20 0x7ff434109130 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:55.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.002+0000 7ff438c00640 1 --2- 172.21.15.134:0/890021090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4341033e0 0x7ff4341037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:55.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.002+0000 7ff438c00640 1 -- 172.21.15.134:0/890021090 >> 172.21.15.134:0/890021090 conn(0x7ff4340fee30 msgr2=0x7ff434101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:55.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.002+0000 7ff438c00640 1 -- 172.21.15.134:0/890021090 shutdown_connections 2024-07-26T11:14:55.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.002+0000 7ff438c00640 1 -- 172.21.15.134:0/890021090 wait complete. 2024-07-26T11:14:55.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.004+0000 7ff438c00640 1 Processor -- start 2024-07-26T11:14:55.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.004+0000 7ff438c00640 1 -- start start 2024-07-26T11:14:55.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.005+0000 7ff438c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4341033e0 0x7ff434199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:55.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.005+0000 7ff438c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff434108d20 0x7ff43419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:55.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.005+0000 7ff438c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff434109670 0x7ff43419f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:55.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.005+0000 7ff438c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff434113300 con 0x7ff4341033e0 2024-07-26T11:14:55.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.005+0000 7ff438c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff434113180 con 0x7ff434109670 2024-07-26T11:14:55.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.005+0000 7ff438c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff434113480 con 0x7ff434108d20 2024-07-26T11:14:55.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.005+0000 7ff433400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4341033e0 0x7ff434199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:55.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.005+0000 7ff433400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4341033e0 0x7ff434199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:45080/0 (socket says 172.21.15.134:45080) 2024-07-26T11:14:55.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.005+0000 7ff433e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff434109670 0x7ff43419f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:55.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.005+0000 7ff42aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff434108d20 0x7ff43419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:55.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.005+0000 7ff433400640 1 -- 172.21.15.134:0/2010458597 learned_addr learned my addr 172.21.15.134:0/2010458597 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:55.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.006+0000 7ff433e00640 1 -- 172.21.15.134:0/2010458597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff434108d20 msgr2=0x7ff43419a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:55.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.006+0000 7ff433e00640 1 --2- 172.21.15.134:0/2010458597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff434108d20 0x7ff43419a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:55.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.006+0000 7ff433e00640 1 -- 172.21.15.134:0/2010458597 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4341033e0 msgr2=0x7ff434199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:55.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.006+0000 7ff433e00640 1 --2- 172.21.15.134:0/2010458597 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4341033e0 0x7ff434199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:55.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.006+0000 7ff433e00640 1 -- 172.21.15.134:0/2010458597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43419fc60 con 0x7ff434109670 2024-07-26T11:14:55.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.006+0000 7ff433e00640 1 --2- 172.21.15.134:0/2010458597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff434109670 0x7ff43419f530 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009fd0 tx=0x7ff42c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:55.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.006+0000 7ff430c00640 1 -- 172.21.15.134:0/2010458597 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42c00ce20 con 0x7ff434109670 2024-07-26T11:14:55.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.006+0000 7ff430c00640 1 -- 172.21.15.134:0/2010458597 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff42c005020 con 0x7ff434109670 2024-07-26T11:14:55.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.006+0000 7ff438c00640 1 -- 172.21.15.134:0/2010458597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff43419ff50 con 0x7ff434109670 2024-07-26T11:14:55.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.006+0000 7ff430c00640 1 -- 172.21.15.134:0/2010458597 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff42c010690 con 0x7ff434109670 2024-07-26T11:14:55.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.006+0000 7ff438c00640 1 -- 172.21.15.134:0/2010458597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4341a66b0 con 0x7ff434109670 2024-07-26T11:14:55.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.007+0000 7ff438c00640 1 -- 172.21.15.134:0/2010458597 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4341037f0 con 0x7ff434109670 2024-07-26T11:14:55.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.010+0000 7ff430c00640 1 -- 172.21.15.134:0/2010458597 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7ff42c0054d0 con 0x7ff434109670 2024-07-26T11:14:55.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.012+0000 7ff430c00640 1 --2- 172.21.15.134:0/2010458597 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff4180b3ca0 0x7ff4180b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:55.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.012+0000 7ff430c00640 1 -- 172.21.15.134:0/2010458597 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff42c0e4730 con 0x7ff434109670 2024-07-26T11:14:55.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.012+0000 7ff433400640 1 --2- 172.21.15.134:0/2010458597 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff4180b3ca0 0x7ff4180b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:55.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.013+0000 7ff433400640 1 --2- 172.21.15.134:0/2010458597 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff4180b3ca0 0x7ff4180b61b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff4240095f0 tx=0x7ff424005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:55.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.013+0000 7ff430c00640 1 -- 172.21.15.134:0/2010458597 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff42c0ab880 con 0x7ff434109670 2024-07-26T11:14:55.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.217+0000 7ff438c00640 1 -- 172.21.15.134:0/2010458597 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7ff434063230 con 0x7ff4180b3ca0 2024-07-26T11:14:55.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.220+0000 7ff430c00640 1 -- 172.21.15.134:0/2010458597 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7ff434063230 con 0x7ff4180b3ca0 2024-07-26T11:14:55.223 INFO:teuthology.orchestra.run.smithi134.stdout:HOST ADDR LABELS STATUS 2024-07-26T11:14:55.223 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134 172.21.15.134 2024-07-26T11:14:55.223 INFO:teuthology.orchestra.run.smithi134.stdout:smithi189 172.21.15.189 2024-07-26T11:14:55.223 INFO:teuthology.orchestra.run.smithi134.stdout:smithi193 172.21.15.193 2024-07-26T11:14:55.223 INFO:teuthology.orchestra.run.smithi134.stdout:3 hosts in cluster 2024-07-26T11:14:55.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.223+0000 7ff438c00640 1 -- 172.21.15.134:0/2010458597 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff4180b3ca0 msgr2=0x7ff4180b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:55.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.223+0000 7ff438c00640 1 --2- 172.21.15.134:0/2010458597 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff4180b3ca0 0x7ff4180b61b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff4240095f0 tx=0x7ff424005f90 comp rx=0 tx=0).stop 2024-07-26T11:14:55.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.223+0000 7ff438c00640 1 -- 172.21.15.134:0/2010458597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff434109670 msgr2=0x7ff43419f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:55.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.223+0000 7ff438c00640 1 --2- 172.21.15.134:0/2010458597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff434109670 0x7ff43419f530 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009fd0 tx=0x7ff42c00ef10 comp rx=0 tx=0).stop 2024-07-26T11:14:55.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.223+0000 7ff438c00640 1 -- 172.21.15.134:0/2010458597 shutdown_connections 2024-07-26T11:14:55.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.223+0000 7ff438c00640 1 --2- 172.21.15.134:0/2010458597 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff4180b3ca0 0x7ff4180b61b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:55.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.223+0000 7ff438c00640 1 --2- 172.21.15.134:0/2010458597 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff434108d20 0x7ff43419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:55.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.223+0000 7ff438c00640 1 --2- 172.21.15.134:0/2010458597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff434109670 0x7ff43419f530 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:55.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.223+0000 7ff438c00640 1 --2- 172.21.15.134:0/2010458597 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff4341033e0 0x7ff434199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:55.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.223+0000 7ff438c00640 1 -- 172.21.15.134:0/2010458597 >> 172.21.15.134:0/2010458597 conn(0x7ff4340fee30 msgr2=0x7ff434101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:55.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.223+0000 7ff438c00640 1 -- 172.21.15.134:0/2010458597 shutdown_connections 2024-07-26T11:14:55.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:55.223+0000 7ff438c00640 1 -- 172.21.15.134:0/2010458597 wait complete. 2024-07-26T11:14:56.107 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- bash -c 'ceph orch device ls' 2024-07-26T11:14:56.567 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:14:57.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:57 smithi134 ceph-mon[31966]: from='client.25271 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:14:57.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:57 smithi134 ceph-mon[31966]: pgmap v449: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:57 smithi189 ceph-mon[36790]: from='client.25271 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:14:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:57 smithi189 ceph-mon[36790]: pgmap v449: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:57 smithi193 ceph-mon[37473]: from='client.25271 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:14:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:57 smithi193 ceph-mon[37473]: pgmap v449: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:14:58.388 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:14:58 smithi134 ceph-mon[31966]: pgmap v450: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:58.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:14:58 smithi189 ceph-mon[36790]: pgmap v450: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:14:58 smithi193 ceph-mon[37473]: pgmap v450: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:14:58.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.971+0000 7fe6dca00640 1 -- 172.21.15.134:0/3002791019 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d8113c90 msgr2=0x7fe6d8116160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:58.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.971+0000 7fe6dca00640 1 --2- 172.21.15.134:0/3002791019 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d8113c90 0x7fe6d8116160 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0009ca0 tx=0x7fe6d002a8c0 comp rx=0 tx=0).stop 2024-07-26T11:14:58.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.971+0000 7fe6dca00640 1 -- 172.21.15.134:0/3002791019 shutdown_connections 2024-07-26T11:14:58.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.971+0000 7fe6dca00640 1 --2- 172.21.15.134:0/3002791019 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d8113c90 0x7fe6d8116160 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:58.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.971+0000 7fe6dca00640 1 --2- 172.21.15.134:0/3002791019 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6d8075800 0x7fe6d8075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:58.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.971+0000 7fe6dca00640 1 --2- 172.21.15.134:0/3002791019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d8076db0 0x7fe6d80751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:58.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.971+0000 7fe6dca00640 1 -- 172.21.15.134:0/3002791019 >> 172.21.15.134:0/3002791019 conn(0x7fe6d80fee10 msgr2=0x7fe6d8101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:58.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.971+0000 7fe6dca00640 1 -- 172.21.15.134:0/3002791019 shutdown_connections 2024-07-26T11:14:58.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.971+0000 7fe6dca00640 1 -- 172.21.15.134:0/3002791019 wait complete. 2024-07-26T11:14:58.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.973+0000 7fe6dca00640 1 Processor -- start 2024-07-26T11:14:58.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.973+0000 7fe6dca00640 1 -- start start 2024-07-26T11:14:58.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.973+0000 7fe6dca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d8075800 0x7fe6d81a0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:58.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.973+0000 7fe6dca00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6d8076db0 0x7fe6d81a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:58.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.973+0000 7fe6dca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d8113c90 0x7fe6d81a5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:58.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.973+0000 7fe6dca00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d8119920 con 0x7fe6d8076db0 2024-07-26T11:14:58.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.973+0000 7fe6dca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d81197a0 con 0x7fe6d8113c90 2024-07-26T11:14:58.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.973+0000 7fe6dca00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d8119aa0 con 0x7fe6d8075800 2024-07-26T11:14:58.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.974+0000 7fe6d6a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6d8076db0 0x7fe6d81a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:58.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.974+0000 7fe6d7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d8113c90 0x7fe6d81a5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:58.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.974+0000 7fe6d7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d8113c90 0x7fe6d81a5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:49530/0 (socket says 172.21.15.134:49530) 2024-07-26T11:14:58.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.974+0000 7fe6d7e00640 1 -- 172.21.15.134:0/228942996 learned_addr learned my addr 172.21.15.134:0/228942996 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:14:58.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.974+0000 7fe6d6a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6d8076db0 0x7fe6d81a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:45100/0 (socket says 172.21.15.134:45100) 2024-07-26T11:14:58.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.974+0000 7fe6d7400640 1 --2- 172.21.15.134:0/228942996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d8075800 0x7fe6d81a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:58.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.974+0000 7fe6d7e00640 1 -- 172.21.15.134:0/228942996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d8075800 msgr2=0x7fe6d81a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:58.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.974+0000 7fe6d7e00640 1 --2- 172.21.15.134:0/228942996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d8075800 0x7fe6d81a0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:58.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.974+0000 7fe6d7e00640 1 -- 172.21.15.134:0/228942996 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6d8076db0 msgr2=0x7fe6d81a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:58.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.974+0000 7fe6d7e00640 1 --2- 172.21.15.134:0/228942996 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6d8076db0 0x7fe6d81a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:58.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.974+0000 7fe6d7e00640 1 -- 172.21.15.134:0/228942996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d81a61e0 con 0x7fe6d8113c90 2024-07-26T11:14:58.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.974+0000 7fe6d6a00640 1 --2- 172.21.15.134:0/228942996 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6d8076db0 0x7fe6d81a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:14:58.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.975+0000 7fe6d7e00640 1 --2- 172.21.15.134:0/228942996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d8113c90 0x7fe6d81a5ab0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe6d002f540 tx=0x7fe6d00316f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:58.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.975+0000 7fe6cfe00640 1 -- 172.21.15.134:0/228942996 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0003b70 con 0x7fe6d8113c90 2024-07-26T11:14:58.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.975+0000 7fe6dca00640 1 -- 172.21.15.134:0/228942996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d81a6470 con 0x7fe6d8113c90 2024-07-26T11:14:58.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.975+0000 7fe6cfe00640 1 -- 172.21.15.134:0/228942996 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6d0003d10 con 0x7fe6d8113c90 2024-07-26T11:14:58.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.975+0000 7fe6cfe00640 1 -- 172.21.15.134:0/228942996 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d002f990 con 0x7fe6d8113c90 2024-07-26T11:14:58.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.975+0000 7fe6dca00640 1 -- 172.21.15.134:0/228942996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d81acc30 con 0x7fe6d8113c90 2024-07-26T11:14:58.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.976+0000 7fe6dca00640 1 -- 172.21.15.134:0/228942996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6d8075c10 con 0x7fe6d8113c90 2024-07-26T11:14:58.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.979+0000 7fe6cfe00640 1 -- 172.21.15.134:0/228942996 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fe6d0004b40 con 0x7fe6d8113c90 2024-07-26T11:14:58.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.980+0000 7fe6cfe00640 1 --2- 172.21.15.134:0/228942996 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6bc0b3c20 0x7fe6bc0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:14:58.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.981+0000 7fe6cfe00640 1 -- 172.21.15.134:0/228942996 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fe6d0105640 con 0x7fe6d8113c90 2024-07-26T11:14:58.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.981+0000 7fe6d7400640 1 --2- 172.21.15.134:0/228942996 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6bc0b3c20 0x7fe6bc0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:14:58.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.981+0000 7fe6d7400640 1 --2- 172.21.15.134:0/228942996 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6bc0b3c20 0x7fe6bc0b6130 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8009bb0 tx=0x7fe6c8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:14:58.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:58.982+0000 7fe6cfe00640 1 -- 172.21.15.134:0/228942996 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fe6d00cc790 con 0x7fe6d8113c90 2024-07-26T11:14:59.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:59.188+0000 7fe6dca00640 1 -- 172.21.15.134:0/228942996 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fe6d8063230 con 0x7fe6bc0b3c20 2024-07-26T11:14:59.194 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:59.191+0000 7fe6cfe00640 1 -- 172.21.15.134:0/228942996 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fe6d8063230 con 0x7fe6bc0b3c20 2024-07-26T11:14:59.194 INFO:teuthology.orchestra.run.smithi134.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-07-26T11:14:59.194 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT62420002400BGN 372G No 6m ago Has a FileSystem, LVM detected 2024-07-26T11:14:59.194 INFO:teuthology.orchestra.run.smithi134.stdout:smithi189 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6232004L400BGN 372G No 6m ago Has a FileSystem, LVM detected 2024-07-26T11:14:59.194 INFO:teuthology.orchestra.run.smithi134.stdout:smithi193 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500HB400BGN 372G No 5m ago Has a FileSystem, LVM detected 2024-07-26T11:14:59.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:59.194+0000 7fe6dca00640 1 -- 172.21.15.134:0/228942996 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6bc0b3c20 msgr2=0x7fe6bc0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:59.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:59.194+0000 7fe6dca00640 1 --2- 172.21.15.134:0/228942996 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6bc0b3c20 0x7fe6bc0b6130 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8009bb0 tx=0x7fe6c8005f90 comp rx=0 tx=0).stop 2024-07-26T11:14:59.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:59.194+0000 7fe6dca00640 1 -- 172.21.15.134:0/228942996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d8113c90 msgr2=0x7fe6d81a5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:14:59.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:59.194+0000 7fe6dca00640 1 --2- 172.21.15.134:0/228942996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d8113c90 0x7fe6d81a5ab0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe6d002f540 tx=0x7fe6d00316f0 comp rx=0 tx=0).stop 2024-07-26T11:14:59.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:59.194+0000 7fe6dca00640 1 -- 172.21.15.134:0/228942996 shutdown_connections 2024-07-26T11:14:59.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:59.194+0000 7fe6dca00640 1 --2- 172.21.15.134:0/228942996 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6bc0b3c20 0x7fe6bc0b6130 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:59.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:59.194+0000 7fe6dca00640 1 --2- 172.21.15.134:0/228942996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d8113c90 0x7fe6d81a5ab0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:59.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:59.194+0000 7fe6dca00640 1 --2- 172.21.15.134:0/228942996 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe6d8076db0 0x7fe6d81a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:59.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:59.194+0000 7fe6dca00640 1 --2- 172.21.15.134:0/228942996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe6d8075800 0x7fe6d81a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:14:59.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:59.194+0000 7fe6dca00640 1 -- 172.21.15.134:0/228942996 >> 172.21.15.134:0/228942996 conn(0x7fe6d80fee10 msgr2=0x7fe6d810fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:14:59.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:59.194+0000 7fe6dca00640 1 -- 172.21.15.134:0/228942996 shutdown_connections 2024-07-26T11:14:59.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:14:59.195+0000 7fe6dca00640 1 -- 172.21.15.134:0/228942996 wait complete. 2024-07-26T11:15:00.193 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-07-26T11:15:00.203 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi134.front.sepia.ceph.com 2024-07-26T11:15:00.261 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- bash -c 'ceph fs dump' 2024-07-26T11:15:00.724 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:15:01.126 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:01 smithi134 ceph-mon[31966]: from='client.25277 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:15:01.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:01 smithi134 ceph-mon[31966]: pgmap v451: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:01 smithi189 ceph-mon[36790]: from='client.25277 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:15:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:01 smithi189 ceph-mon[36790]: pgmap v451: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:01 smithi193 ceph-mon[37473]: from='client.25277 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:15:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:01 smithi193 ceph-mon[37473]: pgmap v451: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:02 smithi134 ceph-mon[31966]: pgmap v452: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:02 smithi189 ceph-mon[36790]: pgmap v452: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:02 smithi193 ceph-mon[37473]: pgmap v452: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:03.221 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.218+0000 7f3f57200640 1 -- 172.21.15.134:0/1536431504 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f58108d20 msgr2=0x7f3f58109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:03.221 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.218+0000 7f3f57200640 1 --2- 172.21.15.134:0/1536431504 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f58108d20 0x7f3f58109130 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3f40009900 tx=0x7f3f4002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:15:03.221 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.218+0000 7f3f57200640 1 -- 172.21.15.134:0/1536431504 shutdown_connections 2024-07-26T11:15:03.221 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.218+0000 7f3f57200640 1 --2- 172.21.15.134:0/1536431504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58109670 0x7f3f5810fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:03.221 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.218+0000 7f3f57200640 1 --2- 172.21.15.134:0/1536431504 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f58108d20 0x7f3f58109130 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:03.221 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.218+0000 7f3f57200640 1 --2- 172.21.15.134:0/1536431504 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f581033e0 0x7f3f581037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:03.221 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.218+0000 7f3f57200640 1 -- 172.21.15.134:0/1536431504 >> 172.21.15.134:0/1536431504 conn(0x7f3f580fee70 msgr2=0x7f3f581012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:03.221 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.218+0000 7f3f57200640 1 -- 172.21.15.134:0/1536431504 shutdown_connections 2024-07-26T11:15:03.221 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.218+0000 7f3f57200640 1 -- 172.21.15.134:0/1536431504 wait complete. 2024-07-26T11:15:03.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.220+0000 7f3f57200640 1 Processor -- start 2024-07-26T11:15:03.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.220+0000 7f3f57200640 1 -- start start 2024-07-26T11:15:03.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.220+0000 7f3f57200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f581033e0 0x7f3f5819c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:03.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.220+0000 7f3f57200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f58108d20 0x7f3f5819c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:03.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.220+0000 7f3f57200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58109670 0x7f3f581a1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:03.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.220+0000 7f3f57200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3f581132e0 con 0x7f3f58108d20 2024-07-26T11:15:03.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.220+0000 7f3f57200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f58113160 con 0x7f3f58109670 2024-07-26T11:15:03.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.220+0000 7f3f57200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3f58113460 con 0x7f3f581033e0 2024-07-26T11:15:03.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.221+0000 7f3f55e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f581033e0 0x7f3f5819c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:03.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.221+0000 7f3f55e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f581033e0 0x7f3f5819c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:36792/0 (socket says 172.21.15.134:36792) 2024-07-26T11:15:03.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.221+0000 7f3f55e00640 1 -- 172.21.15.134:0/2216299706 learned_addr learned my addr 172.21.15.134:0/2216299706 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:03.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.221+0000 7f3f55400640 1 --2- 172.21.15.134:0/2216299706 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f58108d20 0x7f3f5819c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:03.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.221+0000 7f3f56800640 1 --2- 172.21.15.134:0/2216299706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58109670 0x7f3f581a1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:03.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.221+0000 7f3f56800640 1 -- 172.21.15.134:0/2216299706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f581033e0 msgr2=0x7f3f5819c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:03.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.221+0000 7f3f56800640 1 --2- 172.21.15.134:0/2216299706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f581033e0 0x7f3f5819c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:03.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.221+0000 7f3f56800640 1 -- 172.21.15.134:0/2216299706 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f58108d20 msgr2=0x7f3f5819c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:03.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.221+0000 7f3f56800640 1 --2- 172.21.15.134:0/2216299706 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f58108d20 0x7f3f5819c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:03.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.221+0000 7f3f56800640 1 -- 172.21.15.134:0/2216299706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f581a1f80 con 0x7f3f58109670 2024-07-26T11:15:03.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.222+0000 7f3f56800640 1 --2- 172.21.15.134:0/2216299706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58109670 0x7f3f581a1850 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800da10 tx=0x7f3f4800ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:03.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.222+0000 7f3f4ea00640 1 -- 172.21.15.134:0/2216299706 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4800bd70 con 0x7f3f58109670 2024-07-26T11:15:03.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.222+0000 7f3f4ea00640 1 -- 172.21.15.134:0/2216299706 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f48003e20 con 0x7f3f58109670 2024-07-26T11:15:03.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.222+0000 7f3f57200640 1 -- 172.21.15.134:0/2216299706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f581a2270 con 0x7f3f58109670 2024-07-26T11:15:03.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.222+0000 7f3f4ea00640 1 -- 172.21.15.134:0/2216299706 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f480106c0 con 0x7f3f58109670 2024-07-26T11:15:03.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.222+0000 7f3f57200640 1 -- 172.21.15.134:0/2216299706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f581a89d0 con 0x7f3f58109670 2024-07-26T11:15:03.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.223+0000 7f3f57200640 1 -- 172.21.15.134:0/2216299706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f581037f0 con 0x7f3f58109670 2024-07-26T11:15:03.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.226+0000 7f3f4ea00640 1 -- 172.21.15.134:0/2216299706 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f3f48004270 con 0x7f3f58109670 2024-07-26T11:15:03.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.227+0000 7f3f4ea00640 1 --2- 172.21.15.134:0/2216299706 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3f3c0b3c20 0x7f3f3c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:03.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.228+0000 7f3f4ea00640 1 -- 172.21.15.134:0/2216299706 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3f480e48a0 con 0x7f3f58109670 2024-07-26T11:15:03.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.228+0000 7f3f55e00640 1 --2- 172.21.15.134:0/2216299706 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3f3c0b3c20 0x7f3f3c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:03.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.228+0000 7f3f55e00640 1 --2- 172.21.15.134:0/2216299706 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3f3c0b3c20 0x7f3f3c0b6130 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3f50009c30 tx=0x7f3f50005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:03.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.229+0000 7f3f4ea00640 1 -- 172.21.15.134:0/2216299706 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f3f480ab9f0 con 0x7f3f58109670 2024-07-26T11:15:03.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.463+0000 7f3f57200640 1 -- 172.21.15.134:0/2216299706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f3f5819d140 con 0x7f3f58109670 2024-07-26T11:15:03.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.464+0000 7f3f4ea00640 1 -- 172.21.15.134:0/2216299706 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 29 v29) ==== 76+0+3140 (secure 0 0 0) 0x7f3f480ab2f0 con 0x7f3f58109670 2024-07-26T11:15:03.467 INFO:teuthology.orchestra.run.smithi134.stdout:e29 2024-07-26T11:15:03.467 INFO:teuthology.orchestra.run.smithi134.stdout:btime 2024-07-26T11:11:55:972763+0000 2024-07-26T11:15:03.467 INFO:teuthology.orchestra.run.smithi134.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-07-26T11:15:03.467 INFO:teuthology.orchestra.run.smithi134.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T11:15:03.467 INFO:teuthology.orchestra.run.smithi134.stdout:legacy client fscid: 1 2024-07-26T11:15:03.467 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:15:03.467 INFO:teuthology.orchestra.run.smithi134.stdout:Filesystem 'cephfs' (1) 2024-07-26T11:15:03.467 INFO:teuthology.orchestra.run.smithi134.stdout:fs_name cephfs 2024-07-26T11:15:03.467 INFO:teuthology.orchestra.run.smithi134.stdout:epoch 29 2024-07-26T11:15:03.467 INFO:teuthology.orchestra.run.smithi134.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-07-26T11:15:03.468 INFO:teuthology.orchestra.run.smithi134.stdout:created 2024-07-26T11:10:25.500802+0000 2024-07-26T11:15:03.468 INFO:teuthology.orchestra.run.smithi134.stdout:modified 2024-07-26T11:11:55.128451+0000 2024-07-26T11:15:03.468 INFO:teuthology.orchestra.run.smithi134.stdout:tableserver 0 2024-07-26T11:15:03.468 INFO:teuthology.orchestra.run.smithi134.stdout:root 0 2024-07-26T11:15:03.468 INFO:teuthology.orchestra.run.smithi134.stdout:session_timeout 300 2024-07-26T11:15:03.468 INFO:teuthology.orchestra.run.smithi134.stdout:session_autoclose 300 2024-07-26T11:15:03.468 INFO:teuthology.orchestra.run.smithi134.stdout:max_file_size 1099511627776 2024-07-26T11:15:03.468 INFO:teuthology.orchestra.run.smithi134.stdout:max_xattr_size 65536 2024-07-26T11:15:03.468 INFO:teuthology.orchestra.run.smithi134.stdout:required_client_features {} 2024-07-26T11:15:03.468 INFO:teuthology.orchestra.run.smithi134.stdout:last_failure 0 2024-07-26T11:15:03.468 INFO:teuthology.orchestra.run.smithi134.stdout:last_failure_osd_epoch 0 2024-07-26T11:15:03.468 INFO:teuthology.orchestra.run.smithi134.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-07-26T11:15:03.468 INFO:teuthology.orchestra.run.smithi134.stdout:max_mds 5 2024-07-26T11:15:03.468 INFO:teuthology.orchestra.run.smithi134.stdout:in 0,1,2,3,4 2024-07-26T11:15:03.469 INFO:teuthology.orchestra.run.smithi134.stdout:up {0=24721,1=24719,2=24703,3=24688,4=24686} 2024-07-26T11:15:03.469 INFO:teuthology.orchestra.run.smithi134.stdout:failed 2024-07-26T11:15:03.469 INFO:teuthology.orchestra.run.smithi134.stdout:damaged 2024-07-26T11:15:03.469 INFO:teuthology.orchestra.run.smithi134.stdout:stopped 2024-07-26T11:15:03.469 INFO:teuthology.orchestra.run.smithi134.stdout:data_pools [3,4] 2024-07-26T11:15:03.469 INFO:teuthology.orchestra.run.smithi134.stdout:metadata_pool 2 2024-07-26T11:15:03.469 INFO:teuthology.orchestra.run.smithi134.stdout:inline_data disabled 2024-07-26T11:15:03.469 INFO:teuthology.orchestra.run.smithi134.stdout:balancer 2024-07-26T11:15:03.469 INFO:teuthology.orchestra.run.smithi134.stdout:bal_rank_mask -1 2024-07-26T11:15:03.469 INFO:teuthology.orchestra.run.smithi134.stdout:standby_count_wanted 0 2024-07-26T11:15:03.469 INFO:teuthology.orchestra.run.smithi134.stdout:qdb_cluster leader: 24721 members: 24686,24688,24703,24719,24721 2024-07-26T11:15:03.469 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.a{0:24721} state up:active seq 8 addr [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:15:03.469 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.f{0:24682} state up:standby-replay seq 1 addr [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:15:03.470 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.b{1:24719} state up:active seq 28 addr [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:15:03.470 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.e{1:24665} state up:standby-replay seq 1 addr [v2:172.21.15.189:6834/1314042896,v1:172.21.15.189:6835/1314042896] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:15:03.470 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.i{2:24703} state up:active seq 31 addr [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:15:03.470 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.j{2:24641} state up:standby-replay seq 1 addr [v2:172.21.15.134:6834/4071410587,v1:172.21.15.134:6835/4071410587] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:15:03.470 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.c{3:24688} state up:active seq 32 addr [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:15:03.470 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.l{3:24637} state up:standby-replay seq 1 addr [v2:172.21.15.193:6832/3093465661,v1:172.21.15.193:6833/3093465661] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:15:03.470 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.h{4:24686} state up:active seq 32 addr [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:15:03.470 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.k{4:24635} state up:standby-replay seq 1 addr [v2:172.21.15.189:6832/1897636383,v1:172.21.15.189:6833/1897636383] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:15:03.470 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:15:03.470 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:15:03.470 INFO:teuthology.orchestra.run.smithi134.stdout:Standby daemons: 2024-07-26T11:15:03.470 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:15:03.470 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.d{-1:14541} state up:standby seq 1 addr [v2:172.21.15.134:6836/31742259,v1:172.21.15.134:6837/31742259] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:15:03.470 INFO:teuthology.orchestra.run.smithi134.stdout:[mds.g{-1:14565} state up:standby seq 1 addr [v2:172.21.15.134:6838/3307697142,v1:172.21.15.134:6839/3307697142] compat {c=[1],r=[1],i=[1fff]}] 2024-07-26T11:15:03.471 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 29 2024-07-26T11:15:03.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.467+0000 7f3f57200640 1 -- 172.21.15.134:0/2216299706 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3f3c0b3c20 msgr2=0x7f3f3c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:03.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.467+0000 7f3f57200640 1 --2- 172.21.15.134:0/2216299706 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3f3c0b3c20 0x7f3f3c0b6130 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3f50009c30 tx=0x7f3f50005f90 comp rx=0 tx=0).stop 2024-07-26T11:15:03.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.467+0000 7f3f57200640 1 -- 172.21.15.134:0/2216299706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58109670 msgr2=0x7f3f581a1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:03.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.467+0000 7f3f57200640 1 --2- 172.21.15.134:0/2216299706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58109670 0x7f3f581a1850 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800da10 tx=0x7f3f4800ded0 comp rx=0 tx=0).stop 2024-07-26T11:15:03.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.467+0000 7f3f57200640 1 -- 172.21.15.134:0/2216299706 shutdown_connections 2024-07-26T11:15:03.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.467+0000 7f3f57200640 1 --2- 172.21.15.134:0/2216299706 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3f3c0b3c20 0x7f3f3c0b6130 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:03.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.467+0000 7f3f57200640 1 --2- 172.21.15.134:0/2216299706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58109670 0x7f3f581a1850 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:03.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.467+0000 7f3f57200640 1 --2- 172.21.15.134:0/2216299706 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3f58108d20 0x7f3f5819c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:03.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.467+0000 7f3f57200640 1 --2- 172.21.15.134:0/2216299706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f581033e0 0x7f3f5819c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:03.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.467+0000 7f3f57200640 1 -- 172.21.15.134:0/2216299706 >> 172.21.15.134:0/2216299706 conn(0x7f3f580fee70 msgr2=0x7f3f58108470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:03.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.467+0000 7f3f57200640 1 -- 172.21.15.134:0/2216299706 shutdown_connections 2024-07-26T11:15:03.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:03.467+0000 7f3f57200640 1 -- 172.21.15.134:0/2216299706 wait complete. 2024-07-26T11:15:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:03 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2216299706' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T11:15:03.836 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:03 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2216299706' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T11:15:03.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:03 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2216299706' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-07-26T11:15:04.246 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- bash -c 'ceph osd dump' 2024-07-26T11:15:04.707 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:15:04.854 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:04 smithi134 ceph-mon[31966]: pgmap v453: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:15:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:04 smithi189 ceph-mon[36790]: pgmap v453: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:15:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:04 smithi193 ceph-mon[37473]: pgmap v453: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:15:07.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.149+0000 7f69b0a00640 1 -- 172.21.15.134:0/1996451843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69ac1033e0 msgr2=0x7f69ac1037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:07.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.149+0000 7f69b0a00640 1 --2- 172.21.15.134:0/1996451843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69ac1033e0 0x7f69ac1037f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f699c009ca0 tx=0x7f699c02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:15:07.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.149+0000 7f69b0a00640 1 -- 172.21.15.134:0/1996451843 shutdown_connections 2024-07-26T11:15:07.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.149+0000 7f69b0a00640 1 --2- 172.21.15.134:0/1996451843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69ac109670 0x7f69ac10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:07.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.149+0000 7f69b0a00640 1 --2- 172.21.15.134:0/1996451843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69ac108d20 0x7f69ac109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:07.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.149+0000 7f69b0a00640 1 --2- 172.21.15.134:0/1996451843 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69ac1033e0 0x7f69ac1037f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:07.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.149+0000 7f69b0a00640 1 -- 172.21.15.134:0/1996451843 >> 172.21.15.134:0/1996451843 conn(0x7f69ac0fee70 msgr2=0x7f69ac1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:07.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.150+0000 7f69b0a00640 1 -- 172.21.15.134:0/1996451843 shutdown_connections 2024-07-26T11:15:07.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.150+0000 7f69b0a00640 1 -- 172.21.15.134:0/1996451843 wait complete. 2024-07-26T11:15:07.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.151+0000 7f69b0a00640 1 Processor -- start 2024-07-26T11:15:07.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.151+0000 7f69b0a00640 1 -- start start 2024-07-26T11:15:07.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.151+0000 7f69b0a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69ac1033e0 0x7f69ac19bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:07.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.151+0000 7f69b0a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69ac108d20 0x7f69ac19c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:07.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.151+0000 7f69b0a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69ac109670 0x7f69ac1a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:07.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.152+0000 7f69b0a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f69ac113390 con 0x7f69ac1033e0 2024-07-26T11:15:07.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.152+0000 7f69b0a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f69ac113210 con 0x7f69ac109670 2024-07-26T11:15:07.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.152+0000 7f69b0a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f69ac113510 con 0x7f69ac108d20 2024-07-26T11:15:07.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.152+0000 7f69ab400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69ac1033e0 0x7f69ac19bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:07.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.152+0000 7f69abe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69ac109670 0x7f69ac1a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:07.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.152+0000 7f69ab400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69ac1033e0 0x7f69ac19bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:43666/0 (socket says 172.21.15.134:43666) 2024-07-26T11:15:07.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.152+0000 7f69ab400640 1 -- 172.21.15.134:0/3362255559 learned_addr learned my addr 172.21.15.134:0/3362255559 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:07.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.152+0000 7f69aaa00640 1 --2- 172.21.15.134:0/3362255559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69ac108d20 0x7f69ac19c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:07.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.152+0000 7f69abe00640 1 -- 172.21.15.134:0/3362255559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69ac108d20 msgr2=0x7f69ac19c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:07.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.152+0000 7f69abe00640 1 --2- 172.21.15.134:0/3362255559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69ac108d20 0x7f69ac19c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:07.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.152+0000 7f69abe00640 1 -- 172.21.15.134:0/3362255559 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69ac1033e0 msgr2=0x7f69ac19bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:07.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.152+0000 7f69abe00640 1 --2- 172.21.15.134:0/3362255559 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69ac1033e0 0x7f69ac19bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:07.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.152+0000 7f69abe00640 1 -- 172.21.15.134:0/3362255559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69ac1a1ef0 con 0x7f69ac109670 2024-07-26T11:15:07.156 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.152+0000 7f69ab400640 1 --2- 172.21.15.134:0/3362255559 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69ac1033e0 0x7f69ac19bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:07.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.153+0000 7f69abe00640 1 --2- 172.21.15.134:0/3362255559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69ac109670 0x7f69ac1a17c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f69a400e970 tx=0x7f69a400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:07.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.153+0000 7f69a3e00640 1 -- 172.21.15.134:0/3362255559 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a400ccc0 con 0x7f69ac109670 2024-07-26T11:15:07.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.153+0000 7f69a3e00640 1 -- 172.21.15.134:0/3362255559 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69a4005020 con 0x7f69ac109670 2024-07-26T11:15:07.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.153+0000 7f69b0a00640 1 -- 172.21.15.134:0/3362255559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69ac1a21e0 con 0x7f69ac109670 2024-07-26T11:15:07.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.153+0000 7f69a3e00640 1 -- 172.21.15.134:0/3362255559 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a4004a00 con 0x7f69ac109670 2024-07-26T11:15:07.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.153+0000 7f69b0a00640 1 -- 172.21.15.134:0/3362255559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69ac1a8940 con 0x7f69ac109670 2024-07-26T11:15:07.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.154+0000 7f69b0a00640 1 -- 172.21.15.134:0/3362255559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69ac1037f0 con 0x7f69ac109670 2024-07-26T11:15:07.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.157+0000 7f69a3e00640 1 -- 172.21.15.134:0/3362255559 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f69a4004ba0 con 0x7f69ac109670 2024-07-26T11:15:07.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.158+0000 7f69a3e00640 1 --2- 172.21.15.134:0/3362255559 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f69900b3c20 0x7f69900b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:07.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.159+0000 7f69a3e00640 1 -- 172.21.15.134:0/3362255559 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f69a40e4170 con 0x7f69ac109670 2024-07-26T11:15:07.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.159+0000 7f69ab400640 1 --2- 172.21.15.134:0/3362255559 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f69900b3c20 0x7f69900b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:07.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.159+0000 7f69ab400640 1 --2- 172.21.15.134:0/3362255559 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f69900b3c20 0x7f69900b6130 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f699c000f80 tx=0x7f699c005d80 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:07.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.160+0000 7f69a3e00640 1 -- 172.21.15.134:0/3362255559 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f69a40ab2c0 con 0x7f69ac109670 2024-07-26T11:15:07.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.361+0000 7f69b0a00640 1 -- 172.21.15.134:0/3362255559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f69ac19d0b0 con 0x7f69ac109670 2024-07-26T11:15:07.364 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:07 smithi134 ceph-mon[31966]: pgmap v454: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:07.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.363+0000 7f69a3e00640 1 -- 172.21.15.134:0/3362255559 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) ==== 56+0+6110 (secure 0 0 0) 0x7f69ac19d0b0 con 0x7f69ac109670 2024-07-26T11:15:07.365 INFO:teuthology.orchestra.run.smithi134.stdout:epoch 90 2024-07-26T11:15:07.365 INFO:teuthology.orchestra.run.smithi134.stdout:fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 2024-07-26T11:15:07.365 INFO:teuthology.orchestra.run.smithi134.stdout:created 2024-07-26T11:00:05.955103+0000 2024-07-26T11:15:07.366 INFO:teuthology.orchestra.run.smithi134.stdout:modified 2024-07-26T11:11:35.149490+0000 2024-07-26T11:15:07.366 INFO:teuthology.orchestra.run.smithi134.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-07-26T11:15:07.366 INFO:teuthology.orchestra.run.smithi134.stdout:crush_version 27 2024-07-26T11:15:07.366 INFO:teuthology.orchestra.run.smithi134.stdout:full_ratio 0.95 2024-07-26T11:15:07.366 INFO:teuthology.orchestra.run.smithi134.stdout:backfillfull_ratio 0.9 2024-07-26T11:15:07.366 INFO:teuthology.orchestra.run.smithi134.stdout:nearfull_ratio 0.85 2024-07-26T11:15:07.366 INFO:teuthology.orchestra.run.smithi134.stdout:require_min_compat_client luminous 2024-07-26T11:15:07.366 INFO:teuthology.orchestra.run.smithi134.stdout:min_compat_client luminous 2024-07-26T11:15:07.366 INFO:teuthology.orchestra.run.smithi134.stdout:require_osd_release squid 2024-07-26T11:15:07.366 INFO:teuthology.orchestra.run.smithi134.stdout:stretch_mode_enabled false 2024-07-26T11:15:07.366 INFO:teuthology.orchestra.run.smithi134.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 23 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-07-26T11:15:07.367 INFO:teuthology.orchestra.run.smithi134.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 81 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-07-26T11:15:07.367 INFO:teuthology.orchestra.run.smithi134.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 81 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-07-26T11:15:07.367 INFO:teuthology.orchestra.run.smithi134.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 88 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-07-26T11:15:07.367 INFO:teuthology.orchestra.run.smithi134.stdout:max_osd 12 2024-07-26T11:15:07.367 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0 up in weight 1 up_from 9 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.134:6802/4183273954,v1:172.21.15.134:6803/4183273954] [v2:172.21.15.134:6804/4183273954,v1:172.21.15.134:6805/4183273954] exists,up 8abb1365-fc3e-4617-a9ac-76bece8cd7ec 2024-07-26T11:15:07.367 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1 up in weight 1 up_from 14 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6800/35148049,v1:172.21.15.189:6801/35148049] [v2:172.21.15.189:6802/35148049,v1:172.21.15.189:6803/35148049] exists,up 6c5a7838-821e-44e1-97e4-655b9bfa9f38 2024-07-26T11:15:07.367 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2 up in weight 1 up_from 19 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6800/897516976,v1:172.21.15.193:6801/897516976] [v2:172.21.15.193:6802/897516976,v1:172.21.15.193:6803/897516976] exists,up 8bc42f7f-39fa-4d3b-bc17-21c68f9231fa 2024-07-26T11:15:07.367 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3 up in weight 1 up_from 28 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.134:6810/3457190917,v1:172.21.15.134:6811/3457190917] [v2:172.21.15.134:6812/3457190917,v1:172.21.15.134:6813/3457190917] exists,up 0473d055-0b79-434f-9658-28269d2b1fbf 2024-07-26T11:15:07.367 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4 up in weight 1 up_from 34 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6808/1786881347,v1:172.21.15.189:6809/1786881347] [v2:172.21.15.189:6810/1786881347,v1:172.21.15.189:6811/1786881347] exists,up 1d70eaae-56cc-4cdc-8f83-b88d2121f0ad 2024-07-26T11:15:07.367 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5 up in weight 1 up_from 39 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6808/1092388797,v1:172.21.15.193:6809/1092388797] [v2:172.21.15.193:6810/1092388797,v1:172.21.15.193:6811/1092388797] exists,up a9b0b4cf-1608-45ed-b64e-afcf545d97fa 2024-07-26T11:15:07.367 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6 up in weight 1 up_from 45 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.134:6818/2471505438,v1:172.21.15.134:6819/2471505438] [v2:172.21.15.134:6820/2471505438,v1:172.21.15.134:6821/2471505438] exists,up 0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04 2024-07-26T11:15:07.367 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7 up in weight 1 up_from 50 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6816/1382694911,v1:172.21.15.189:6817/1382694911] [v2:172.21.15.189:6818/1382694911,v1:172.21.15.189:6819/1382694911] exists,up 394764a7-1976-4223-a3c0-5979d941a838 2024-07-26T11:15:07.367 INFO:teuthology.orchestra.run.smithi134.stdout:osd.8 up in weight 1 up_from 56 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6816/896348138,v1:172.21.15.193:6817/896348138] [v2:172.21.15.193:6818/896348138,v1:172.21.15.193:6819/896348138] exists,up 1c218082-0b4c-41c8-92b9-453ae8d15a41 2024-07-26T11:15:07.368 INFO:teuthology.orchestra.run.smithi134.stdout:osd.9 up in weight 1 up_from 62 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.134:6826/511237025,v1:172.21.15.134:6827/511237025] [v2:172.21.15.134:6828/511237025,v1:172.21.15.134:6829/511237025] exists,up 48f39082-4fa9-4443-b5b7-b141c10d9e98 2024-07-26T11:15:07.368 INFO:teuthology.orchestra.run.smithi134.stdout:osd.10 up in weight 1 up_from 67 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6824/431740858,v1:172.21.15.189:6825/431740858] [v2:172.21.15.189:6826/431740858,v1:172.21.15.189:6827/431740858] exists,up 95157766-1856-42f7-aea1-9cf83b77c8ae 2024-07-26T11:15:07.368 INFO:teuthology.orchestra.run.smithi134.stdout:osd.11 up in weight 1 up_from 73 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6824/1594590579,v1:172.21.15.193:6825/1594590579] [v2:172.21.15.193:6826/1594590579,v1:172.21.15.193:6827/1594590579] exists,up 244fadc7-f079-4736-8e7e-c0ce5ae6f43e 2024-07-26T11:15:07.368 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 2.14 [2,4] 2024-07-26T11:15:07.368 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 2.16 [2,4] 2024-07-26T11:15:07.368 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 2.1c [8,11] 2024-07-26T11:15:07.368 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 2.1f [7,11] 2024-07-26T11:15:07.368 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 3.0 [5,9] 2024-07-26T11:15:07.368 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 3.2 [3,9] 2024-07-26T11:15:07.368 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 3.14 [4,6] 2024-07-26T11:15:07.368 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 3.17 [4,7] 2024-07-26T11:15:07.368 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 3.34 [4,9] 2024-07-26T11:15:07.368 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 3.39 [5,8] 2024-07-26T11:15:07.368 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 4.2 [9,1] 2024-07-26T11:15:07.369 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 4.5 [5,8] 2024-07-26T11:15:07.369 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 4.6 [9,11] 2024-07-26T11:15:07.369 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 4.1f [9,8] 2024-07-26T11:15:07.369 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/3790862248 expires 2024-07-27T11:02:33.711793+0000 2024-07-26T11:15:07.369 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:6800/2424179007 expires 2024-07-27T11:01:12.679951+0000 2024-07-26T11:15:07.369 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/2906927791 expires 2024-07-27T11:01:12.679951+0000 2024-07-26T11:15:07.369 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:6800/3630314144 expires 2024-07-27T11:00:42.651939+0000 2024-07-26T11:15:07.369 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/3068153391 expires 2024-07-27T11:00:42.651939+0000 2024-07-26T11:15:07.369 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/3056560791 expires 2024-07-27T11:01:12.679951+0000 2024-07-26T11:15:07.369 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:6800/606980428 expires 2024-07-27T11:02:33.711793+0000 2024-07-26T11:15:07.369 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:6801/2424179007 expires 2024-07-27T11:01:12.679951+0000 2024-07-26T11:15:07.369 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:6801/606980428 expires 2024-07-27T11:02:33.711793+0000 2024-07-26T11:15:07.369 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/1310359922 expires 2024-07-27T11:02:33.711793+0000 2024-07-26T11:15:07.369 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/2318666906 expires 2024-07-27T11:00:42.651939+0000 2024-07-26T11:15:07.370 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/2901579347 expires 2024-07-27T11:01:12.679951+0000 2024-07-26T11:15:07.370 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:6801/3630314144 expires 2024-07-27T11:00:42.651939+0000 2024-07-26T11:15:07.370 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/3404218719 expires 2024-07-27T11:00:42.651939+0000 2024-07-26T11:15:07.370 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/3598071145 expires 2024-07-27T11:02:33.711793+0000 2024-07-26T11:15:07.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.365+0000 7f69b0a00640 1 -- 172.21.15.134:0/3362255559 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f69900b3c20 msgr2=0x7f69900b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:07.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.365+0000 7f69b0a00640 1 --2- 172.21.15.134:0/3362255559 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f69900b3c20 0x7f69900b6130 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f699c000f80 tx=0x7f699c005d80 comp rx=0 tx=0).stop 2024-07-26T11:15:07.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.365+0000 7f69b0a00640 1 -- 172.21.15.134:0/3362255559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69ac109670 msgr2=0x7f69ac1a17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:07.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.365+0000 7f69b0a00640 1 --2- 172.21.15.134:0/3362255559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69ac109670 0x7f69ac1a17c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f69a400e970 tx=0x7f69a400ee30 comp rx=0 tx=0).stop 2024-07-26T11:15:07.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.366+0000 7f69b0a00640 1 -- 172.21.15.134:0/3362255559 shutdown_connections 2024-07-26T11:15:07.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.366+0000 7f69b0a00640 1 --2- 172.21.15.134:0/3362255559 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f69900b3c20 0x7f69900b6130 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:07.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.366+0000 7f69b0a00640 1 --2- 172.21.15.134:0/3362255559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69ac108d20 0x7f69ac19c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:07.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.366+0000 7f69b0a00640 1 --2- 172.21.15.134:0/3362255559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69ac109670 0x7f69ac1a17c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:07.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.366+0000 7f69b0a00640 1 --2- 172.21.15.134:0/3362255559 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f69ac1033e0 0x7f69ac19bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:07.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.366+0000 7f69b0a00640 1 -- 172.21.15.134:0/3362255559 >> 172.21.15.134:0/3362255559 conn(0x7f69ac0fee70 msgr2=0x7f69ac10e0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:07.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.366+0000 7f69b0a00640 1 -- 172.21.15.134:0/3362255559 shutdown_connections 2024-07-26T11:15:07.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:07.366+0000 7f69b0a00640 1 -- 172.21.15.134:0/3362255559 wait complete. 2024-07-26T11:15:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:07 smithi189 ceph-mon[36790]: pgmap v454: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:07 smithi193 ceph-mon[37473]: pgmap v454: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:08.363 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:08 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3362255559' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T11:15:08.363 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:08 smithi134 ceph-mon[31966]: pgmap v455: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-07-26T11:15:08.367 INFO:teuthology.run_tasks:Running task fs.ready... 2024-07-26T11:15:08.379 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:15:08.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:08 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3362255559' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T11:15:08.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:08 smithi189 ceph-mon[36790]: pgmap v455: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-07-26T11:15:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:08 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3362255559' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-07-26T11:15:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:08 smithi193 ceph-mon[37473]: pgmap v455: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-07-26T11:15:08.838 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:15:11.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.111+0000 7f4b76600640 1 -- 172.21.15.134:0/2492429059 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b701033c0 msgr2=0x7f4b701037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:11.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.111+0000 7f4b76600640 1 --2- 172.21.15.134:0/2492429059 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b701033c0 0x7f4b701037d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009900 tx=0x7f4b6002a410 comp rx=0 tx=0).stop 2024-07-26T11:15:11.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.112+0000 7f4b76600640 1 -- 172.21.15.134:0/2492429059 shutdown_connections 2024-07-26T11:15:11.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.112+0000 7f4b76600640 1 --2- 172.21.15.134:0/2492429059 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b70109650 0x7f4b7010fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:11.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.112+0000 7f4b76600640 1 --2- 172.21.15.134:0/2492429059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b70108d00 0x7f4b70109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:11.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.112+0000 7f4b76600640 1 --2- 172.21.15.134:0/2492429059 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b701033c0 0x7f4b701037d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:11.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.112+0000 7f4b76600640 1 -- 172.21.15.134:0/2492429059 >> 172.21.15.134:0/2492429059 conn(0x7f4b700fee10 msgr2=0x7f4b70101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:11.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.112+0000 7f4b76600640 1 -- 172.21.15.134:0/2492429059 shutdown_connections 2024-07-26T11:15:11.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.112+0000 7f4b76600640 1 -- 172.21.15.134:0/2492429059 wait complete. 2024-07-26T11:15:11.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.113+0000 7f4b76600640 1 Processor -- start 2024-07-26T11:15:11.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.113+0000 7f4b76600640 1 -- start start 2024-07-26T11:15:11.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.114+0000 7f4b76600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b701033c0 0x7f4b70075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:11.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.114+0000 7f4b76600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b70108d00 0x7f4b70075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:11.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.114+0000 7f4b76600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b70109650 0x7f4b70075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:11.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.114+0000 7f4b76600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4b70113370 con 0x7f4b701033c0 2024-07-26T11:15:11.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.114+0000 7f4b76600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4b701131f0 con 0x7f4b70109650 2024-07-26T11:15:11.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.114+0000 7f4b76600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4b701134f0 con 0x7f4b70108d00 2024-07-26T11:15:11.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.114+0000 7f4b75200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b701033c0 0x7f4b70075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:11.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.114+0000 7f4b67e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b70108d00 0x7f4b70075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:11.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.114+0000 7f4b75200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b701033c0 0x7f4b70075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50296/0 (socket says 172.21.15.134:50296) 2024-07-26T11:15:11.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.114+0000 7f4b75200640 1 -- 172.21.15.134:0/1851775015 learned_addr learned my addr 172.21.15.134:0/1851775015 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:11.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.114+0000 7f4b75c00640 1 --2- 172.21.15.134:0/1851775015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b70109650 0x7f4b70075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:11.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.114+0000 7f4b67e00640 1 -- 172.21.15.134:0/1851775015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b70109650 msgr2=0x7f4b70075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:11.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.114+0000 7f4b67e00640 1 --2- 172.21.15.134:0/1851775015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b70109650 0x7f4b70075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:11.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.114+0000 7f4b67e00640 1 -- 172.21.15.134:0/1851775015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b701033c0 msgr2=0x7f4b70075140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:11.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.114+0000 7f4b67e00640 1 --2- 172.21.15.134:0/1851775015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b701033c0 0x7f4b70075140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:11.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.114+0000 7f4b67e00640 1 -- 172.21.15.134:0/1851775015 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b701a8020 con 0x7f4b70108d00 2024-07-26T11:15:11.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.115+0000 7f4b67e00640 1 --2- 172.21.15.134:0/1851775015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b70108d00 0x7f4b70075680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4b58005fd0 tx=0x7f4b5800dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:11.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.115+0000 7f4b6e000640 1 -- 172.21.15.134:0/1851775015 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58009c00 con 0x7f4b70108d00 2024-07-26T11:15:11.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.115+0000 7f4b76600640 1 -- 172.21.15.134:0/1851775015 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b701a8310 con 0x7f4b70108d00 2024-07-26T11:15:11.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.115+0000 7f4b6e000640 1 -- 172.21.15.134:0/1851775015 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b58009da0 con 0x7f4b70108d00 2024-07-26T11:15:11.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.115+0000 7f4b6e000640 1 -- 172.21.15.134:0/1851775015 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58003e10 con 0x7f4b70108d00 2024-07-26T11:15:11.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.115+0000 7f4b76600640 1 -- 172.21.15.134:0/1851775015 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b701a88a0 con 0x7f4b70108d00 2024-07-26T11:15:11.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.116+0000 7f4b76600640 1 -- 172.21.15.134:0/1851775015 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b701037d0 con 0x7f4b70108d00 2024-07-26T11:15:11.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.119+0000 7f4b6e000640 1 -- 172.21.15.134:0/1851775015 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f4b58003fb0 con 0x7f4b70108d00 2024-07-26T11:15:11.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.120+0000 7f4b6e000640 1 --2- 172.21.15.134:0/1851775015 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b500b3c20 0x7f4b500b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:11.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.120+0000 7f4b6e000640 1 -- 172.21.15.134:0/1851775015 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4b580e3c90 con 0x7f4b70108d00 2024-07-26T11:15:11.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.120+0000 7f4b75200640 1 --2- 172.21.15.134:0/1851775015 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b500b3c20 0x7f4b500b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:11.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.121+0000 7f4b75200640 1 --2- 172.21.15.134:0/1851775015 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b500b3c20 0x7f4b500b6130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4b60005ec0 tx=0x7f4b60005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:11.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.124+0000 7f4b6e000640 1 -- 172.21.15.134:0/1851775015 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4b58017020 con 0x7f4b70108d00 2024-07-26T11:15:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:11 smithi134 ceph-mon[31966]: pgmap v456: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 9 op/s 2024-07-26T11:15:11.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.367+0000 7f4b76600640 1 -- 172.21.15.134:0/1851775015 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b7002e0c0 con 0x7f4b70108d00 2024-07-26T11:15:11.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.368+0000 7f4b6e000640 1 -- 172.21.15.134:0/1851775015 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11990 (secure 0 0 0) 0x7f4b580aa790 con 0x7f4b70108d00 2024-07-26T11:15:11.371 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:15:11.372 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 29 2024-07-26T11:15:11.372 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":29,"btime":"2024-07-26T11:11:55:972763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:11:55.128451+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24721,"mds_1":24719,"mds_2":24703,"mds_3":24688,"mds_4":24686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24635":{"gid":24635,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24641":{"gid":24641,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":32,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24719":{"gid":24719,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":28,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24721,"qdb_cluster":[24686,24688,24703,24719,24721]},"id":1}]} 2024-07-26T11:15:11.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.371+0000 7f4b76600640 1 -- 172.21.15.134:0/1851775015 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b500b3c20 msgr2=0x7f4b500b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:11.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.371+0000 7f4b76600640 1 --2- 172.21.15.134:0/1851775015 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b500b3c20 0x7f4b500b6130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4b60005ec0 tx=0x7f4b60005a20 comp rx=0 tx=0).stop 2024-07-26T11:15:11.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.372+0000 7f4b76600640 1 -- 172.21.15.134:0/1851775015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b70108d00 msgr2=0x7f4b70075680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:11.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.372+0000 7f4b76600640 1 --2- 172.21.15.134:0/1851775015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b70108d00 0x7f4b70075680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4b58005fd0 tx=0x7f4b5800dde0 comp rx=0 tx=0).stop 2024-07-26T11:15:11.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.372+0000 7f4b76600640 1 -- 172.21.15.134:0/1851775015 shutdown_connections 2024-07-26T11:15:11.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.372+0000 7f4b76600640 1 --2- 172.21.15.134:0/1851775015 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b500b3c20 0x7f4b500b6130 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:11.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.372+0000 7f4b76600640 1 --2- 172.21.15.134:0/1851775015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b70108d00 0x7f4b70075680 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:11.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.372+0000 7f4b76600640 1 --2- 172.21.15.134:0/1851775015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b70109650 0x7f4b70075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:11.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.372+0000 7f4b76600640 1 --2- 172.21.15.134:0/1851775015 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b701033c0 0x7f4b70075140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:11.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.372+0000 7f4b76600640 1 -- 172.21.15.134:0/1851775015 >> 172.21.15.134:0/1851775015 conn(0x7f4b700fee10 msgr2=0x7f4b70101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:11.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.372+0000 7f4b76600640 1 -- 172.21.15.134:0/1851775015 shutdown_connections 2024-07-26T11:15:11.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:11.372+0000 7f4b76600640 1 -- 172.21.15.134:0/1851775015 wait complete. 2024-07-26T11:15:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:11 smithi189 ceph-mon[36790]: pgmap v456: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 9 op/s 2024-07-26T11:15:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:11 smithi193 ceph-mon[37473]: pgmap v456: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 9 op/s 2024-07-26T11:15:12.324 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:15:12.401 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:12 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1851775015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:15:12.401 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:12 smithi134 ceph-mon[31966]: pgmap v457: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:15:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:12 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1851775015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:15:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:12 smithi189 ceph-mon[36790]: pgmap v457: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:15:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:12 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1851775015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:15:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:12 smithi193 ceph-mon[37473]: pgmap v457: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:15:12.782 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:15:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:14 smithi134 ceph-mon[31966]: pgmap v458: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T11:15:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:14 smithi189 ceph-mon[36790]: pgmap v458: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T11:15:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:14 smithi193 ceph-mon[37473]: pgmap v458: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T11:15:15.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.128+0000 7fa4dbc00640 1 -- 172.21.15.134:0/1854945551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4d41116e0 msgr2=0x7fa4d4111af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:15.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.128+0000 7fa4dbc00640 1 --2- 172.21.15.134:0/1854945551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4d41116e0 0x7fa4d4111af0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0009900 tx=0x7fa4c002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:15:15.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.128+0000 7fa4dbc00640 1 -- 172.21.15.134:0/1854945551 shutdown_connections 2024-07-26T11:15:15.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.128+0000 7fa4dbc00640 1 --2- 172.21.15.134:0/1854945551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4d4112030 0x7fa4d4118860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:15.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.128+0000 7fa4dbc00640 1 --2- 172.21.15.134:0/1854945551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4d41116e0 0x7fa4d4111af0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:15.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.128+0000 7fa4dbc00640 1 --2- 172.21.15.134:0/1854945551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4d410bda0 0x7fa4d410c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:15.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.128+0000 7fa4dbc00640 1 -- 172.21.15.134:0/1854945551 >> 172.21.15.134:0/1854945551 conn(0x7fa4d40714b0 msgr2=0x7fa4d4109df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:15.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.128+0000 7fa4dbc00640 1 -- 172.21.15.134:0/1854945551 shutdown_connections 2024-07-26T11:15:15.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.129+0000 7fa4dbc00640 1 -- 172.21.15.134:0/1854945551 wait complete. 2024-07-26T11:15:15.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.130+0000 7fa4dbc00640 1 Processor -- start 2024-07-26T11:15:15.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.130+0000 7fa4dbc00640 1 -- start start 2024-07-26T11:15:15.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.130+0000 7fa4dbc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4d410bda0 0x7fa4d406c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:15.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.130+0000 7fa4dbc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4d41116e0 0x7fa4d406ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:15.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.130+0000 7fa4dbc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4d4112030 0x7fa4d406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:15.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.130+0000 7fa4dbc00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d411bd40 con 0x7fa4d4112030 2024-07-26T11:15:15.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.130+0000 7fa4dbc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d411bbc0 con 0x7fa4d41116e0 2024-07-26T11:15:15.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.130+0000 7fa4dbc00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d411bec0 con 0x7fa4d410bda0 2024-07-26T11:15:15.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.131+0000 7fa4d9200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4d410bda0 0x7fa4d406c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:15.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.131+0000 7fa4d9200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4d410bda0 0x7fa4d406c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:37760/0 (socket says 172.21.15.134:37760) 2024-07-26T11:15:15.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.131+0000 7fa4d9c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4d4112030 0x7fa4d406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:15.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.131+0000 7fa4d9200640 1 -- 172.21.15.134:0/2542548717 learned_addr learned my addr 172.21.15.134:0/2542548717 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:15.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.131+0000 7fa4d3e00640 1 --2- 172.21.15.134:0/2542548717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4d41116e0 0x7fa4d406ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:15.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.131+0000 7fa4d9200640 1 -- 172.21.15.134:0/2542548717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4d41116e0 msgr2=0x7fa4d406ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:15.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.131+0000 7fa4d9200640 1 --2- 172.21.15.134:0/2542548717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4d41116e0 0x7fa4d406ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:15.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.131+0000 7fa4d9200640 1 -- 172.21.15.134:0/2542548717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4d4112030 msgr2=0x7fa4d406d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:15.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.131+0000 7fa4d9200640 1 --2- 172.21.15.134:0/2542548717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4d4112030 0x7fa4d406d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:15.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.131+0000 7fa4d9200640 1 -- 172.21.15.134:0/2542548717 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d41b0a30 con 0x7fa4d410bda0 2024-07-26T11:15:15.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.132+0000 7fa4d9200640 1 --2- 172.21.15.134:0/2542548717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4d410bda0 0x7fa4d406c8d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400e9f0 tx=0x7fa4c400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:15.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.132+0000 7fa4d1600640 1 -- 172.21.15.134:0/2542548717 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c400cd80 con 0x7fa4d410bda0 2024-07-26T11:15:15.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.132+0000 7fa4dbc00640 1 -- 172.21.15.134:0/2542548717 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d41b0d20 con 0x7fa4d410bda0 2024-07-26T11:15:15.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.132+0000 7fa4d1600640 1 -- 172.21.15.134:0/2542548717 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa4c40108e0 con 0x7fa4d410bda0 2024-07-26T11:15:15.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.132+0000 7fa4d1600640 1 -- 172.21.15.134:0/2542548717 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4018980 con 0x7fa4d410bda0 2024-07-26T11:15:15.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.132+0000 7fa4dbc00640 1 -- 172.21.15.134:0/2542548717 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d41b1220 con 0x7fa4d410bda0 2024-07-26T11:15:15.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.133+0000 7fa4dbc00640 1 -- 172.21.15.134:0/2542548717 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4d410c1b0 con 0x7fa4d410bda0 2024-07-26T11:15:15.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.136+0000 7fa4d1600640 1 -- 172.21.15.134:0/2542548717 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7fa4c4010430 con 0x7fa4d410bda0 2024-07-26T11:15:15.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.137+0000 7fa4d1600640 1 --2- 172.21.15.134:0/2542548717 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa4b80b3c20 0x7fa4b80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:15.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.137+0000 7fa4d1600640 1 -- 172.21.15.134:0/2542548717 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa4c40e4930 con 0x7fa4d410bda0 2024-07-26T11:15:15.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.138+0000 7fa4d3e00640 1 --2- 172.21.15.134:0/2542548717 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa4b80b3c20 0x7fa4b80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:15.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.138+0000 7fa4d3e00640 1 --2- 172.21.15.134:0/2542548717 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa4b80b3c20 0x7fa4b80b6130 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa4c002a9e0 tx=0x7fa4c0005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:15.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.139+0000 7fa4d1600640 1 -- 172.21.15.134:0/2542548717 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa4c40ab430 con 0x7fa4d410bda0 2024-07-26T11:15:15.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.375+0000 7fa4dbc00640 1 -- 172.21.15.134:0/2542548717 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4d406dbd0 con 0x7fa4d410bda0 2024-07-26T11:15:15.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.376+0000 7fa4d1600640 1 -- 172.21.15.134:0/2542548717 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11990 (secure 0 0 0) 0x7fa4d406dbd0 con 0x7fa4d410bda0 2024-07-26T11:15:15.379 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:15:15.379 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":29,"btime":"2024-07-26T11:11:55:972763+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:11:55.128451+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24721,"mds_1":24719,"mds_2":24703,"mds_3":24688,"mds_4":24686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24635":{"gid":24635,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24641":{"gid":24641,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":32,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24719":{"gid":24719,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":28,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24721,"qdb_cluster":[24686,24688,24703,24719,24721]},"id":1}]} 2024-07-26T11:15:15.379 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 29 2024-07-26T11:15:15.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.380+0000 7fa4dbc00640 1 -- 172.21.15.134:0/2542548717 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa4b80b3c20 msgr2=0x7fa4b80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:15.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.380+0000 7fa4dbc00640 1 --2- 172.21.15.134:0/2542548717 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa4b80b3c20 0x7fa4b80b6130 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa4c002a9e0 tx=0x7fa4c0005a20 comp rx=0 tx=0).stop 2024-07-26T11:15:15.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.380+0000 7fa4dbc00640 1 -- 172.21.15.134:0/2542548717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4d410bda0 msgr2=0x7fa4d406c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:15.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.380+0000 7fa4dbc00640 1 --2- 172.21.15.134:0/2542548717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4d410bda0 0x7fa4d406c8d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400e9f0 tx=0x7fa4c400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:15:15.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.380+0000 7fa4dbc00640 1 -- 172.21.15.134:0/2542548717 shutdown_connections 2024-07-26T11:15:15.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.380+0000 7fa4dbc00640 1 --2- 172.21.15.134:0/2542548717 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa4b80b3c20 0x7fa4b80b6130 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:15.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.380+0000 7fa4dbc00640 1 --2- 172.21.15.134:0/2542548717 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa4d4112030 0x7fa4d406d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:15.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.380+0000 7fa4dbc00640 1 --2- 172.21.15.134:0/2542548717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4d41116e0 0x7fa4d406ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:15.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.380+0000 7fa4dbc00640 1 --2- 172.21.15.134:0/2542548717 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa4d410bda0 0x7fa4d406c8d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:15.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.380+0000 7fa4dbc00640 1 -- 172.21.15.134:0/2542548717 >> 172.21.15.134:0/2542548717 conn(0x7fa4d40714b0 msgr2=0x7fa4d410a120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:15.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.380+0000 7fa4dbc00640 1 -- 172.21.15.134:0/2542548717 shutdown_connections 2024-07-26T11:15:15.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:15.380+0000 7fa4dbc00640 1 -- 172.21.15.134:0/2542548717 wait complete. 2024-07-26T11:15:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:15 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2542548717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:15:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:15 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2542548717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:15:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:15 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2542548717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:15:16.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T11:15:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:16 smithi134 ceph-mon[31966]: pgmap v459: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:15:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:16 smithi189 ceph-mon[36790]: pgmap v459: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:15:16.734 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:15:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:16 smithi193 ceph-mon[37473]: pgmap v459: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T11:15:18.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:18 smithi134 ceph-mon[31966]: pgmap v460: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-07-26T11:15:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:18 smithi189 ceph-mon[36790]: pgmap v460: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-07-26T11:15:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:18 smithi193 ceph-mon[37473]: pgmap v460: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-07-26T11:15:19.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.168+0000 7f4566000640 1 -- 172.21.15.134:0/2513793574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f456010f330 msgr2=0x7f456010f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:19.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.168+0000 7f4566000640 1 --2- 172.21.15.134:0/2513793574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f456010f330 0x7f456010f740 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4548009900 tx=0x7f454802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:15:19.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.168+0000 7f4566000640 1 -- 172.21.15.134:0/2513793574 shutdown_connections 2024-07-26T11:15:19.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.168+0000 7f4566000640 1 --2- 172.21.15.134:0/2513793574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f456010f330 0x7f456010f740 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:19.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.168+0000 7f4566000640 1 --2- 172.21.15.134:0/2513793574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f456010fc80 0x7f45601164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:19.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.168+0000 7f4566000640 1 --2- 172.21.15.134:0/2513793574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f45601099f0 0x7f4560109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:19.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.168+0000 7f4566000640 1 -- 172.21.15.134:0/2513793574 >> 172.21.15.134:0/2513793574 conn(0x7f4560078ff0 msgr2=0x7f4560107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:19.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.169+0000 7f4566000640 1 -- 172.21.15.134:0/2513793574 shutdown_connections 2024-07-26T11:15:19.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.169+0000 7f4566000640 1 -- 172.21.15.134:0/2513793574 wait complete. 2024-07-26T11:15:19.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.170+0000 7f4566000640 1 Processor -- start 2024-07-26T11:15:19.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.170+0000 7f4566000640 1 -- start start 2024-07-26T11:15:19.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.171+0000 7f4566000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45601099f0 0x7f45601a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:19.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.171+0000 7f4566000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f456010f330 0x7f45601a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:19.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.171+0000 7f4566000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f456010fc80 0x7f45601a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:19.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.171+0000 7f4566000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4560119990 con 0x7f456010fc80 2024-07-26T11:15:19.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.171+0000 7f4566000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4560119810 con 0x7f456010f330 2024-07-26T11:15:19.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.171+0000 7f4566000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4560119b10 con 0x7f45601099f0 2024-07-26T11:15:19.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.171+0000 7f4565600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f456010fc80 0x7f45601a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:19.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.171+0000 7f4557e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f456010f330 0x7f45601a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:19.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.171+0000 7f4564c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45601099f0 0x7f45601a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:19.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.171+0000 7f4557e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f456010f330 0x7f45601a09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:59852/0 (socket says 172.21.15.134:59852) 2024-07-26T11:15:19.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.171+0000 7f4565600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f456010fc80 0x7f45601a5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50336/0 (socket says 172.21.15.134:50336) 2024-07-26T11:15:19.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.171+0000 7f4557e00640 1 -- 172.21.15.134:0/2940402309 learned_addr learned my addr 172.21.15.134:0/2940402309 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:19.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.171+0000 7f4564c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45601099f0 0x7f45601a0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:37780/0 (socket says 172.21.15.134:37780) 2024-07-26T11:15:19.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.172+0000 7f4564c00640 1 -- 172.21.15.134:0/2940402309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f456010f330 msgr2=0x7f45601a09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:19.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.172+0000 7f4564c00640 1 --2- 172.21.15.134:0/2940402309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f456010f330 0x7f45601a09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:19.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.172+0000 7f4564c00640 1 -- 172.21.15.134:0/2940402309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f456010fc80 msgr2=0x7f45601a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:19.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.172+0000 7f4564c00640 1 --2- 172.21.15.134:0/2940402309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f456010fc80 0x7f45601a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:19.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.172+0000 7f4564c00640 1 -- 172.21.15.134:0/2940402309 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45601a6390 con 0x7f45601099f0 2024-07-26T11:15:19.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.172+0000 7f4564c00640 1 --2- 172.21.15.134:0/2940402309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45601099f0 0x7f45601a0420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f455000e9f0 tx=0x7f455000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:19.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.172+0000 7f455e000640 1 -- 172.21.15.134:0/2940402309 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f455000ccc0 con 0x7f45601099f0 2024-07-26T11:15:19.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.172+0000 7f455e000640 1 -- 172.21.15.134:0/2940402309 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45500108e0 con 0x7f45601099f0 2024-07-26T11:15:19.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.172+0000 7f455e000640 1 -- 172.21.15.134:0/2940402309 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4550018840 con 0x7f45601099f0 2024-07-26T11:15:19.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.173+0000 7f4566000640 1 -- 172.21.15.134:0/2940402309 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45601a6680 con 0x7f45601099f0 2024-07-26T11:15:19.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.173+0000 7f4566000640 1 -- 172.21.15.134:0/2940402309 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45601acde0 con 0x7f45601099f0 2024-07-26T11:15:19.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.174+0000 7f4566000640 1 -- 172.21.15.134:0/2940402309 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4560109e00 con 0x7f45601099f0 2024-07-26T11:15:19.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.177+0000 7f455e000640 1 -- 172.21.15.134:0/2940402309 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f4550010430 con 0x7f45601099f0 2024-07-26T11:15:19.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.178+0000 7f455e000640 1 --2- 172.21.15.134:0/2940402309 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f45440b3c20 0x7f45440b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:19.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.178+0000 7f455e000640 1 -- 172.21.15.134:0/2940402309 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f45500e48a0 con 0x7f45601099f0 2024-07-26T11:15:19.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.178+0000 7f4557e00640 1 --2- 172.21.15.134:0/2940402309 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f45440b3c20 0x7f45440b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:19.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.179+0000 7f4557e00640 1 --2- 172.21.15.134:0/2940402309 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f45440b3c20 0x7f45440b6130 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4548006010 tx=0x7f4548005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:19.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.180+0000 7f455e000640 1 -- 172.21.15.134:0/2940402309 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f45500ab9f0 con 0x7f45601099f0 2024-07-26T11:15:19.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.378+0000 7f4566000640 1 -- 172.21.15.134:0/2940402309 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f45601a1550 con 0x7f45601099f0 2024-07-26T11:15:19.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.380+0000 7f455e000640 1 -- 172.21.15.134:0/2940402309 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25185 (secure 0 0 0) 0x7f45601a1550 con 0x7f45601099f0 2024-07-26T11:15:19.383 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:15:19.383 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":90,"fsid":"105fbe5c-4b3e-11ef-bcb7-c7b262605968","created":"2024-07-26T11:00:05.955103+0000","modified":"2024-07-26T11:11:35.149490+0000","last_up_change":"2024-07-26T11:09:11.485512+0000","last_in_change":"2024-07-26T11:08:52.341356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T11:05:01.849170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T11:10:16.055725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T11:10:21.216611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T11:10:33.983067+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8abb1365-fc3e-4617-a9ac-76bece8cd7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6803","nonce":4183273954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6805","nonce":4183273954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6809","nonce":4183273954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6807","nonce":4183273954}]},"public_addr":"172.21.15.134:6803/4183273954","cluster_addr":"172.21.15.134:6805/4183273954","heartbeat_back_addr":"172.21.15.134:6809/4183273954","heartbeat_front_addr":"172.21.15.134:6807/4183273954","state":["exists","up"]},{"osd":1,"uuid":"6c5a7838-821e-44e1-97e4-655b9bfa9f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6801","nonce":35148049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6803","nonce":35148049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6807","nonce":35148049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6805","nonce":35148049}]},"public_addr":"172.21.15.189:6801/35148049","cluster_addr":"172.21.15.189:6803/35148049","heartbeat_back_addr":"172.21.15.189:6807/35148049","heartbeat_front_addr":"172.21.15.189:6805/35148049","state":["exists","up"]},{"osd":2,"uuid":"8bc42f7f-39fa-4d3b-bc17-21c68f9231fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6801","nonce":897516976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6803","nonce":897516976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6807","nonce":897516976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6805","nonce":897516976}]},"public_addr":"172.21.15.193:6801/897516976","cluster_addr":"172.21.15.193:6803/897516976","heartbeat_back_addr":"172.21.15.193:6807/897516976","heartbeat_front_addr":"172.21.15.193:6805/897516976","state":["exists","up"]},{"osd":3,"uuid":"0473d055-0b79-434f-9658-28269d2b1fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6811","nonce":3457190917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6813","nonce":3457190917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6817","nonce":3457190917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6815","nonce":3457190917}]},"public_addr":"172.21.15.134:6811/3457190917","cluster_addr":"172.21.15.134:6813/3457190917","heartbeat_back_addr":"172.21.15.134:6817/3457190917","heartbeat_front_addr":"172.21.15.134:6815/3457190917","state":["exists","up"]},{"osd":4,"uuid":"1d70eaae-56cc-4cdc-8f83-b88d2121f0ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6809","nonce":1786881347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6811","nonce":1786881347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6815","nonce":1786881347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6813","nonce":1786881347}]},"public_addr":"172.21.15.189:6809/1786881347","cluster_addr":"172.21.15.189:6811/1786881347","heartbeat_back_addr":"172.21.15.189:6815/1786881347","heartbeat_front_addr":"172.21.15.189:6813/1786881347","state":["exists","up"]},{"osd":5,"uuid":"a9b0b4cf-1608-45ed-b64e-afcf545d97fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6809","nonce":1092388797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6811","nonce":1092388797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6815","nonce":1092388797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6813","nonce":1092388797}]},"public_addr":"172.21.15.193:6809/1092388797","cluster_addr":"172.21.15.193:6811/1092388797","heartbeat_back_addr":"172.21.15.193:6815/1092388797","heartbeat_front_addr":"172.21.15.193:6813/1092388797","state":["exists","up"]},{"osd":6,"uuid":"0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6819","nonce":2471505438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6821","nonce":2471505438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6825","nonce":2471505438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6823","nonce":2471505438}]},"public_addr":"172.21.15.134:6819/2471505438","cluster_addr":"172.21.15.134:6821/2471505438","heartbeat_back_addr":"172.21.15.134:6825/2471505438","heartbeat_front_addr":"172.21.15.134:6823/2471505438","state":["exists","up"]},{"osd":7,"uuid":"394764a7-1976-4223-a3c0-5979d941a838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6817","nonce":1382694911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6819","nonce":1382694911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6823","nonce":1382694911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6821","nonce":1382694911}]},"public_addr":"172.21.15.189:6817/1382694911","cluster_addr":"172.21.15.189:6819/1382694911","heartbeat_back_addr":"172.21.15.189:6823/1382694911","heartbeat_front_addr":"172.21.15.189:6821/1382694911","state":["exists","up"]},{"osd":8,"uuid":"1c218082-0b4c-41c8-92b9-453ae8d15a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6817","nonce":896348138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6819","nonce":896348138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6823","nonce":896348138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6821","nonce":896348138}]},"public_addr":"172.21.15.193:6817/896348138","cluster_addr":"172.21.15.193:6819/896348138","heartbeat_back_addr":"172.21.15.193:6823/896348138","heartbeat_front_addr":"172.21.15.193:6821/896348138","state":["exists","up"]},{"osd":9,"uuid":"48f39082-4fa9-4443-b5b7-b141c10d9e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6827","nonce":511237025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6829","nonce":511237025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6833","nonce":511237025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6831","nonce":511237025}]},"public_addr":"172.21.15.134:6827/511237025","cluster_addr":"172.21.15.134:6829/511237025","heartbeat_back_addr":"172.21.15.134:6833/511237025","heartbeat_front_addr":"172.21.15.134:6831/511237025","state":["exists","up"]},{"osd":10,"uuid":"95157766-1856-42f7-aea1-9cf83b77c8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6825","nonce":431740858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6827","nonce":431740858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6831","nonce":431740858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6829","nonce":431740858}]},"public_addr":"172.21.15.189:6825/431740858","cluster_addr":"172.21.15.189:6827/431740858","heartbeat_back_addr":"172.21.15.189:6831/431740858","heartbeat_front_addr":"172.21.15.189:6829/431740858","state":["exists","up"]},{"osd":11,"uuid":"244fadc7-f079-4736-8e7e-c0ce5ae6f43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6825","nonce":1594590579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6827","nonce":1594590579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6831","nonce":1594590579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6829","nonce":1594590579}]},"public_addr":"172.21.15.193:6825/1594590579","cluster_addr":"172.21.15.193:6827/1594590579","heartbeat_back_addr":"172.21.15.193:6831/1594590579","heartbeat_front_addr":"172.21.15.193:6829/1594590579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:11.712203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:35.845013+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:58.165609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:31.929746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:56.941555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:20.688071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:54.681514+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:20.789384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:44.568246+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:18.644310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:45.365351+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:09:09.663805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3790862248":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6800/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:0/2906927791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3068153391":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3056560791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6801/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/1310359922":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/2318666906":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/2901579347":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3404218719":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3598071145":"2024-07-27T11:02:33.711793+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:15:19.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.382+0000 7f4566000640 1 -- 172.21.15.134:0/2940402309 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f45440b3c20 msgr2=0x7f45440b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:19.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.382+0000 7f4566000640 1 --2- 172.21.15.134:0/2940402309 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f45440b3c20 0x7f45440b6130 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4548006010 tx=0x7f4548005a20 comp rx=0 tx=0).stop 2024-07-26T11:15:19.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.382+0000 7f4566000640 1 -- 172.21.15.134:0/2940402309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45601099f0 msgr2=0x7f45601a0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:19.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.382+0000 7f4566000640 1 --2- 172.21.15.134:0/2940402309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45601099f0 0x7f45601a0420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f455000e9f0 tx=0x7f455000edf0 comp rx=0 tx=0).stop 2024-07-26T11:15:19.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.383+0000 7f4566000640 1 -- 172.21.15.134:0/2940402309 shutdown_connections 2024-07-26T11:15:19.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.383+0000 7f4566000640 1 --2- 172.21.15.134:0/2940402309 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f45440b3c20 0x7f45440b6130 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:19.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.383+0000 7f4566000640 1 --2- 172.21.15.134:0/2940402309 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f456010fc80 0x7f45601a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:19.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.383+0000 7f4566000640 1 --2- 172.21.15.134:0/2940402309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f456010f330 0x7f45601a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:19.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.383+0000 7f4566000640 1 --2- 172.21.15.134:0/2940402309 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45601099f0 0x7f45601a0420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:19.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.383+0000 7f4566000640 1 -- 172.21.15.134:0/2940402309 >> 172.21.15.134:0/2940402309 conn(0x7f4560078ff0 msgr2=0x7f4560108070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:19.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.383+0000 7f4566000640 1 -- 172.21.15.134:0/2940402309 shutdown_connections 2024-07-26T11:15:19.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:19.383+0000 7f4566000640 1 -- 172.21.15.134:0/2940402309 wait complete. 2024-07-26T11:15:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:19 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2940402309' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:15:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:19 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2940402309' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:15:19.786 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:19 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2940402309' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:15:20.266 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-07-26T11:10:25.500802+0000', 'modified': '2024-07-26T11:11:55.128451+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24721, 'mds_1': 24719, 'mds_2': 24703, 'mds_3': 24688, 'mds_4': 24686}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24635': {'gid': 24635, 'name': 'k', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6833/1897636383', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6832', 'nonce': 1897636383}, {'type': 'v1', 'addr': '172.21.15.189:6833', 'nonce': 1897636383}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24637': {'gid': 24637, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6833/3093465661', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6832', 'nonce': 3093465661}, {'type': 'v1', 'addr': '172.21.15.193:6833', 'nonce': 3093465661}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24641': {'gid': 24641, 'name': 'j', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.134:6835/4071410587', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6834', 'nonce': 4071410587}, {'type': 'v1', 'addr': '172.21.15.134:6835', 'nonce': 4071410587}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24665': {'gid': 24665, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6835/1314042896', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 1314042896}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 1314042896}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24682': {'gid': 24682, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6835/97099551', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 97099551}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 97099551}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24686': {'gid': 24686, 'name': 'h', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 32, 'addr': '172.21.15.189:6837/1448732093', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6836', 'nonce': 1448732093}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 1448732093}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24688': {'gid': 24688, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 32, 'addr': '172.21.15.193:6837/1968168008', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 1968168008}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 1968168008}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24703': {'gid': 24703, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 31, 'addr': '172.21.15.193:6839/3985555366', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 3985555366}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 3985555366}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24719': {'gid': 24719, 'name': 'b', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 28, 'addr': '172.21.15.189:6839/790931242', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6838', 'nonce': 790931242}, {'type': 'v1', 'addr': '172.21.15.189:6839', 'nonce': 790931242}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24721': {'gid': 24721, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.134:6841/4067549550', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6840', 'nonce': 4067549550}, {'type': 'v1', 'addr': '172.21.15.134:6841', 'nonce': 4067549550}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24721, 'qdb_cluster': [24686, 24688, 24703, 24719, 24721]} 2024-07-26T11:15:20.267 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-07-26T11:15:20.267 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.h status 2024-07-26T11:15:20.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:20 smithi189 ceph-mon[36790]: pgmap v461: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:20.726 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:15:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:20 smithi193 ceph-mon[37473]: pgmap v461: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:20.873 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:20 smithi134 ceph-mon[31966]: pgmap v461: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:22 smithi134 ceph-mon[31966]: pgmap v462: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:22 smithi189 ceph-mon[36790]: pgmap v462: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:22 smithi193 ceph-mon[37473]: pgmap v462: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:23.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.097+0000 7f825be00640 1 -- 172.21.15.134:0/569022448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c10b880 msgr2=0x7f825c1120b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:23.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.097+0000 7f825be00640 1 --2- 172.21.15.134:0/569022448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c10b880 0x7f825c1120b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8254009ca0 tx=0x7f825402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.097+0000 7f825be00640 1 -- 172.21.15.134:0/569022448 shutdown_connections 2024-07-26T11:15:23.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.097+0000 7f825be00640 1 --2- 172.21.15.134:0/569022448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10af30 0x7f825c10b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.097+0000 7f825be00640 1 --2- 172.21.15.134:0/569022448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c10b880 0x7f825c1120b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.097+0000 7f825be00640 1 --2- 172.21.15.134:0/569022448 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f825c1055f0 0x7f825c105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.097+0000 7f825be00640 1 -- 172.21.15.134:0/569022448 >> 172.21.15.134:0/569022448 conn(0x7f825c101060 msgr2=0x7f825c1034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:23.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.098+0000 7f825be00640 1 -- 172.21.15.134:0/569022448 shutdown_connections 2024-07-26T11:15:23.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.098+0000 7f825be00640 1 -- 172.21.15.134:0/569022448 wait complete. 2024-07-26T11:15:23.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.099+0000 7f825be00640 1 Processor -- start 2024-07-26T11:15:23.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.099+0000 7f825be00640 1 -- start start 2024-07-26T11:15:23.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.099+0000 7f825be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c1055f0 0x7f825c19be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.099+0000 7f825be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f825c10af30 0x7f825c19c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.099+0000 7f825be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10b880 0x7f825c1a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.099+0000 7f825be00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f825c1155a0 con 0x7f825c10af30 2024-07-26T11:15:23.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.099+0000 7f825be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f825c115420 con 0x7f825c1055f0 2024-07-26T11:15:23.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.099+0000 7f825be00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f825c115720 con 0x7f825c10b880 2024-07-26T11:15:23.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.100+0000 7f825aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c1055f0 0x7f825c19be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.100+0000 7f825aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c1055f0 0x7f825c19be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:47082/0 (socket says 172.21.15.134:47082) 2024-07-26T11:15:23.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.100+0000 7f825aa00640 1 -- 172.21.15.134:0/4011519403 learned_addr learned my addr 172.21.15.134:0/4011519403 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:23.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.100+0000 7f825a000640 1 --2- 172.21.15.134:0/4011519403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f825c10af30 0x7f825c19c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.100+0000 7f825b400640 1 --2- 172.21.15.134:0/4011519403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10b880 0x7f825c1a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.100+0000 7f825aa00640 1 -- 172.21.15.134:0/4011519403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10b880 msgr2=0x7f825c1a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:23.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.100+0000 7f825aa00640 1 --2- 172.21.15.134:0/4011519403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10b880 0x7f825c1a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.100+0000 7f825aa00640 1 -- 172.21.15.134:0/4011519403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f825c10af30 msgr2=0x7f825c19c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:23.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.100+0000 7f825aa00640 1 --2- 172.21.15.134:0/4011519403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f825c10af30 0x7f825c19c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.100+0000 7f825aa00640 1 -- 172.21.15.134:0/4011519403 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825c1a1e00 con 0x7f825c1055f0 2024-07-26T11:15:23.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.100+0000 7f825b400640 1 --2- 172.21.15.134:0/4011519403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10b880 0x7f825c1a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:23.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.100+0000 7f825a000640 1 --2- 172.21.15.134:0/4011519403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f825c10af30 0x7f825c19c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:23.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.101+0000 7f825aa00640 1 --2- 172.21.15.134:0/4011519403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c1055f0 0x7f825c19be90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f824c00d950 tx=0x7f824c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:23.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.101+0000 7f8253400640 1 -- 172.21.15.134:0/4011519403 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c017070 con 0x7f825c1055f0 2024-07-26T11:15:23.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.101+0000 7f8253400640 1 -- 172.21.15.134:0/4011519403 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f824c009c30 con 0x7f825c1055f0 2024-07-26T11:15:23.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.101+0000 7f825be00640 1 -- 172.21.15.134:0/4011519403 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f825c1a20f0 con 0x7f825c1055f0 2024-07-26T11:15:23.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.101+0000 7f8253400640 1 -- 172.21.15.134:0/4011519403 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c00f620 con 0x7f825c1055f0 2024-07-26T11:15:23.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.101+0000 7f825be00640 1 -- 172.21.15.134:0/4011519403 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f825c1a8850 con 0x7f825c1055f0 2024-07-26T11:15:23.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.104+0000 7f82683f8740 1 Processor -- start 2024-07-26T11:15:23.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.105+0000 7f82683f8740 1 -- start start 2024-07-26T11:15:23.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.105+0000 7f82683f8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 0x5585bfd5f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.105+0000 7f82683f8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf9910a0 0x5585bfd5fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.105+0000 7f82683f8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf999280 0x5585bfd60200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.105+0000 7f82683f8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5585bfcd5970 con 0x5585bf9b9f60 2024-07-26T11:15:23.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.105+0000 7f82683f8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5585bfcc4e70 con 0x5585bf9910a0 2024-07-26T11:15:23.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.105+0000 7f82683f8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5585bfcc8eb0 con 0x5585bf999280 2024-07-26T11:15:23.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.105+0000 7f8253400640 1 -- 172.21.15.134:0/4011519403 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f824c010460 con 0x7f825c1055f0 2024-07-26T11:15:23.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.105+0000 7f825a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf999280 0x5585bfd60200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.105+0000 7f825b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 0x5585bfd5f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.106+0000 7f825b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 0x5585bfd5f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:34654/0 (socket says 172.21.15.134:34654) 2024-07-26T11:15:23.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.106+0000 7f825b400640 1 -- 172.21.15.134:0/453473248 learned_addr learned my addr 172.21.15.134:0/453473248 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:23.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.106+0000 7f825a000640 1 --2- 172.21.15.134:0/453473248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf999280 0x5585bfd60200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52034/0 (socket says 172.21.15.134:52034) 2024-07-26T11:15:23.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.106+0000 7f825aa00640 1 --2- 172.21.15.134:0/453473248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf9910a0 0x5585bfd5fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.106+0000 7f825a000640 1 -- 172.21.15.134:0/453473248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf9910a0 msgr2=0x5585bfd5fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:23.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.106+0000 7f825a000640 1 --2- 172.21.15.134:0/453473248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf9910a0 0x5585bfd5fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.106+0000 7f825a000640 1 -- 172.21.15.134:0/453473248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 msgr2=0x5585bfd5f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:23.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.106+0000 7f825a000640 1 --2- 172.21.15.134:0/453473248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 0x5585bfd5f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.106+0000 7f825a000640 1 -- 172.21.15.134:0/453473248 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585bfc6c110 con 0x5585bf999280 2024-07-26T11:15:23.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.106+0000 7f825a000640 1 --2- 172.21.15.134:0/453473248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf999280 0x5585bfd60200 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f825c06b130 tx=0x7f82480118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be7204ad655cf331 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:23.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.107+0000 7f8247400640 1 -- 172.21.15.134:0/453473248 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8248004310 con 0x5585bf999280 2024-07-26T11:15:23.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.107+0000 7f8247400640 1 -- 172.21.15.134:0/453473248 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82480044b0 con 0x5585bf999280 2024-07-26T11:15:23.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.107+0000 7f8247400640 1 -- 172.21.15.134:0/453473248 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82480035f0 con 0x5585bf999280 2024-07-26T11:15:23.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.107+0000 7f8253400640 1 --2- 172.21.15.134:0/4011519403 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82400b3ca0 0x7f82400b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.107+0000 7f8253400640 1 -- 172.21.15.134:0/4011519403 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f824c0e3c50 con 0x7f825c1055f0 2024-07-26T11:15:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.107+0000 7f825b400640 1 --2- 172.21.15.134:0/4011519403 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82400b3ca0 0x7f82400b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.108+0000 7f82683f8740 1 -- 172.21.15.134:0/453473248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf999280 msgr2=0x5585bfd60200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.108+0000 7f82683f8740 1 --2- 172.21.15.134:0/453473248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf999280 0x5585bfd60200 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f825c06b130 tx=0x7f82480118d0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.108+0000 7f82683f8740 1 -- 172.21.15.134:0/453473248 shutdown_connections 2024-07-26T11:15:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.108+0000 7f82683f8740 1 --2- 172.21.15.134:0/453473248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf999280 0x5585bfd60200 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.108+0000 7f82683f8740 1 --2- 172.21.15.134:0/453473248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf9910a0 0x5585bfd5fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.108+0000 7f82683f8740 1 --2- 172.21.15.134:0/453473248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 0x5585bfd5f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.108+0000 7f82683f8740 1 -- 172.21.15.134:0/453473248 >> 172.21.15.134:0/453473248 conn(0x5585bf9ae350 msgr2=0x5585bfd5d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:23.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.108+0000 7f82683f8740 1 -- 172.21.15.134:0/453473248 shutdown_connections 2024-07-26T11:15:23.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.108+0000 7f82683f8740 1 -- 172.21.15.134:0/453473248 wait complete. 2024-07-26T11:15:23.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.108+0000 7f825b400640 1 --2- 172.21.15.134:0/4011519403 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82400b3ca0 0x7f82400b61b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f825402aeb0 tx=0x7f8254005d80 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:23.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.109+0000 7f82683f8740 1 -- start start 2024-07-26T11:15:23.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.109+0000 7f82683f8740 1 Processor -- start 2024-07-26T11:15:23.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f82683f8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9910a0 0x5585bfd6b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f82683f8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf999280 0x5585bfd6b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f82683f8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf9b9f60 0x5585bfd77280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f82683f8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5585bfb285f0 con 0x5585bf9910a0 2024-07-26T11:15:23.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f82683f8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5585bfc61540 con 0x5585bf999280 2024-07-26T11:15:23.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f82683f8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5585bfd6df60 con 0x5585bf9b9f60 2024-07-26T11:15:23.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f82683f8740 10 client.0 _ll_register_callbacks cb 0x5585bfc70f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:15:23.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f825aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9910a0 0x5585bfd6b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f825a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf999280 0x5585bfd6b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f825aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9910a0 0x5585bfd6b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:34670/0 (socket says 172.21.15.134:34670) 2024-07-26T11:15:23.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f825aa00640 1 -- 172.21.15.134:0/179086605 learned_addr learned my addr 172.21.15.134:0/179086605 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:23.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f825b400640 1 --2- 172.21.15.134:0/179086605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf9b9f60 0x5585bfd77280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f825a000640 1 -- 172.21.15.134:0/179086605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf9b9f60 msgr2=0x5585bfd77280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:23.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f825a000640 1 --2- 172.21.15.134:0/179086605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf9b9f60 0x5585bfd77280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f825a000640 1 -- 172.21.15.134:0/179086605 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9910a0 msgr2=0x5585bfd6b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:23.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f825a000640 1 --2- 172.21.15.134:0/179086605 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9910a0 0x5585bfd6b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.110+0000 7f825a000640 1 -- 172.21.15.134:0/179086605 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585bfa25540 con 0x5585bf999280 2024-07-26T11:15:23.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.111+0000 7f825a000640 1 --2- 172.21.15.134:0/179086605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf999280 0x5585bfd6b630 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8248005890 tx=0x7f82480048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86963839c63402b7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:23.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.111+0000 7f823be00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:15:23.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.111+0000 7f823be00640 1 -- 172.21.15.134:0/179086605 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8248027030 con 0x5585bf999280 2024-07-26T11:15:23.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.111+0000 7f82683f8740 1 -- 172.21.15.134:0/179086605 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82480044b0 con 0x5585bf999280 2024-07-26T11:15:23.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.111+0000 7f823be00640 1 -- 172.21.15.134:0/179086605 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f82480035f0 con 0x5585bf999280 2024-07-26T11:15:23.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.112+0000 7f823be00640 1 -- 172.21.15.134:0/179086605 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8248034050 con 0x5585bf999280 2024-07-26T11:15:23.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.112+0000 7f82683f8740 10 client.25319 fetch_fsmap learned FSMap version 29 2024-07-26T11:15:23.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.112+0000 7f82683f8740 1 -- 172.21.15.134:0/179086605 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5585bfd79e50 con 0x5585bf999280 2024-07-26T11:15:23.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.112+0000 7f823be00640 1 -- 172.21.15.134:0/179086605 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82480044b0 con 0x5585bf999280 2024-07-26T11:15:23.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.112+0000 7f823be00640 1 -- 172.21.15.134:0/179086605 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8248005210 con 0x5585bf999280 2024-07-26T11:15:23.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.112+0000 7f82683f8740 10 client.25319 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:15:23.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.112+0000 7f82683f8740 10 client.25319 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T11:15:23.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.112+0000 7f82683f8740 20 client.25319 populate_metadata read hostname 'smithi134' 2024-07-26T11:15:23.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.112+0000 7f82683f8740 1 --2- 172.21.15.134:0/179086605 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5585bf9788e0 0x5585bfd7d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.112+0000 7f82683f8740 4 client.25319 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:15:23.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.112+0000 7f82683f8740 1 -- 172.21.15.134:0/179086605 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585bfcd78c0 con 0x5585bf9788e0 2024-07-26T11:15:23.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.113+0000 7f825aa00640 1 --2- 172.21.15.134:0/179086605 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5585bf9788e0 0x5585bfd7d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.113+0000 7f825aa00640 1 --2- 172.21.15.134:0/179086605 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5585bf9788e0 0x5585bfd7d620 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e834d0aca61057af server_cookie=ad9cba65e2b2d2c2 in_seq=0 out_seq=0 2024-07-26T11:15:23.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.113+0000 7f823be00640 10 client.25319 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:15:23.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.114+0000 7f823be00640 1 -- 172.21.15.134:0/179086605 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5585bfcd78c0 con 0x5585bf9788e0 2024-07-26T11:15:23.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.114+0000 7f823be00640 10 client.25319 handle_command_reply: tid=0 2024-07-26T11:15:23.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.114+0000 7f82683f8740 1 client.25319 shutdown 2024-07-26T11:15:23.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.114+0000 7f82683f8740 1 -- 172.21.15.134:0/179086605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf999280 msgr2=0x5585bfd6b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:23.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.114+0000 7f82683f8740 1 --2- 172.21.15.134:0/179086605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf999280 0x5585bfd6b630 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8248005890 tx=0x7f82480048f0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.115+0000 7f82683f8740 1 -- 172.21.15.134:0/179086605 shutdown_connections 2024-07-26T11:15:23.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.115+0000 7f82683f8740 1 --2- 172.21.15.134:0/179086605 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5585bf9788e0 0x5585bfd7d620 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.115+0000 7f82683f8740 1 --2- 172.21.15.134:0/179086605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf9b9f60 0x5585bfd77280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.115+0000 7f82683f8740 1 --2- 172.21.15.134:0/179086605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf999280 0x5585bfd6b630 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.115+0000 7f82683f8740 1 --2- 172.21.15.134:0/179086605 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9910a0 0x5585bfd6b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.115+0000 7f823be00640 0 client.25319 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:15:23.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.115+0000 7f82683f8740 1 -- 172.21.15.134:0/179086605 >> 172.21.15.134:0/179086605 conn(0x5585bf9ae350 msgr2=0x5585bfd62960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:23.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.115+0000 7f82683f8740 1 -- 172.21.15.134:0/179086605 shutdown_connections 2024-07-26T11:15:23.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.115+0000 7f82683f8740 1 -- 172.21.15.134:0/179086605 wait complete. 2024-07-26T11:15:23.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.115+0000 7f82683f8740 20 client.25319 trim_cache size 0 max 16384 2024-07-26T11:15:23.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.134+0000 7f82683f8740 1 Processor -- start 2024-07-26T11:15:23.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.134+0000 7f82683f8740 1 -- start start 2024-07-26T11:15:23.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.135+0000 7f82683f8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf9910a0 0x5585bfd698d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.135+0000 7f82683f8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf999280 0x5585bfd6b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.135+0000 7f82683f8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 0x5585bfd771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.135+0000 7f82683f8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5585bfd7b960 con 0x5585bf9b9f60 2024-07-26T11:15:23.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.135+0000 7f82683f8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5585bfd7c030 con 0x5585bf9910a0 2024-07-26T11:15:23.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.135+0000 7f82683f8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5585bfd7c700 con 0x5585bf999280 2024-07-26T11:15:23.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.135+0000 7f825b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 0x5585bfd771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.135+0000 7f825b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 0x5585bfd771f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:34686/0 (socket says 172.21.15.134:34686) 2024-07-26T11:15:23.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.135+0000 7f825b400640 1 -- 172.21.15.134:0/726295547 learned_addr learned my addr 172.21.15.134:0/726295547 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:23.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.135+0000 7f825aa00640 1 --2- 172.21.15.134:0/726295547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf9910a0 0x5585bfd698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.135+0000 7f825a000640 1 --2- 172.21.15.134:0/726295547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf999280 0x5585bfd6b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.136+0000 7f825b400640 1 -- 172.21.15.134:0/726295547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf999280 msgr2=0x5585bfd6b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:23.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.136+0000 7f825b400640 1 --2- 172.21.15.134:0/726295547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf999280 0x5585bfd6b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.136+0000 7f825b400640 1 -- 172.21.15.134:0/726295547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf9910a0 msgr2=0x5585bfd698d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:23.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.136+0000 7f825b400640 1 --2- 172.21.15.134:0/726295547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf9910a0 0x5585bfd698d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.136+0000 7f825b400640 1 -- 172.21.15.134:0/726295547 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585bfd69fb0 con 0x5585bf9b9f60 2024-07-26T11:15:23.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.136+0000 7f825a000640 1 --2- 172.21.15.134:0/726295547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf999280 0x5585bfd6b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:15:23.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.136+0000 7f825aa00640 1 --2- 172.21.15.134:0/726295547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf9910a0 0x5585bfd698d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:23.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.136+0000 7f825b400640 1 --2- 172.21.15.134:0/726295547 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 0x5585bfd771f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8254009ca0 tx=0x7f8254042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0cdaa57b9a0b5ed server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:23.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.137+0000 7f8246a00640 1 -- 172.21.15.134:0/726295547 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254003970 con 0x5585bf9b9f60 2024-07-26T11:15:23.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.137+0000 7f8246a00640 1 -- 172.21.15.134:0/726295547 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8254036040 con 0x5585bf9b9f60 2024-07-26T11:15:23.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.137+0000 7f8246a00640 1 -- 172.21.15.134:0/726295547 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825402f610 con 0x5585bf9b9f60 2024-07-26T11:15:23.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.138+0000 7f82683f8740 1 -- 172.21.15.134:0/726295547 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 msgr2=0x5585bfd771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:23.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.138+0000 7f82683f8740 1 --2- 172.21.15.134:0/726295547 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 0x5585bfd771f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8254009ca0 tx=0x7f8254042000 comp rx=0 tx=0).stop 2024-07-26T11:15:23.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.138+0000 7f82683f8740 1 -- 172.21.15.134:0/726295547 shutdown_connections 2024-07-26T11:15:23.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.138+0000 7f82683f8740 1 --2- 172.21.15.134:0/726295547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf999280 0x5585bfd6b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.138+0000 7f82683f8740 1 --2- 172.21.15.134:0/726295547 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 0x5585bfd771f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.138+0000 7f82683f8740 1 --2- 172.21.15.134:0/726295547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf9910a0 0x5585bfd698d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.142 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.138+0000 7f82683f8740 1 -- 172.21.15.134:0/726295547 >> 172.21.15.134:0/726295547 conn(0x5585bf9ae350 msgr2=0x5585bfd6ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:23.142 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.138+0000 7f82683f8740 1 -- 172.21.15.134:0/726295547 shutdown_connections 2024-07-26T11:15:23.142 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.138+0000 7f82683f8740 1 -- 172.21.15.134:0/726295547 wait complete. 2024-07-26T11:15:23.142 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.138+0000 7f82683f8740 1 -- start start 2024-07-26T11:15:23.142 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.139+0000 7f82683f8740 1 Processor -- start 2024-07-26T11:15:23.142 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.139+0000 7f82683f8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf9910a0 0x5585bfd80f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.142 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.139+0000 7f82683f8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf999280 0x5585bfd814e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.142 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.139+0000 7f82683f8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 0x5585bfde48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.142 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.139+0000 7f82683f8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5585bfd7d4a0 con 0x5585bf9b9f60 2024-07-26T11:15:23.142 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.139+0000 7f82683f8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5585bfd7cdd0 con 0x5585bf999280 2024-07-26T11:15:23.143 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.139+0000 7f82683f8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5585bfd7db70 con 0x5585bf9910a0 2024-07-26T11:15:23.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.139+0000 7f82683f8740 10 client.0 _ll_register_callbacks cb 0x7f8248009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:15:23.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.139+0000 7f825b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 0x5585bfde48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.140+0000 7f825aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf9910a0 0x5585bfd80f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.140+0000 7f825a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf999280 0x5585bfd814e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.140+0000 7f825b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 0x5585bfde48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:34690/0 (socket says 172.21.15.134:34690) 2024-07-26T11:15:23.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.140+0000 7f825b400640 1 -- 172.21.15.134:0/4293474794 learned_addr learned my addr 172.21.15.134:0/4293474794 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:23.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.140+0000 7f825a000640 1 -- 172.21.15.134:0/4293474794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf9910a0 msgr2=0x5585bfd80f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:23.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.140+0000 7f825a000640 1 --2- 172.21.15.134:0/4293474794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf9910a0 0x5585bfd80f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.140+0000 7f825a000640 1 -- 172.21.15.134:0/4293474794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 msgr2=0x5585bfde48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:23.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.140+0000 7f825a000640 1 --2- 172.21.15.134:0/4293474794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 0x5585bfde48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.140+0000 7f825a000640 1 -- 172.21.15.134:0/4293474794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585bfde4ea0 con 0x5585bf999280 2024-07-26T11:15:23.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.140+0000 7f825a000640 1 --2- 172.21.15.134:0/4293474794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf999280 0x5585bfd814e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8248004b60 tx=0x7f8248011ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8f4b28a2e1c4d0a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:23.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.141+0000 7f823be00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:15:23.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.141+0000 7f823be00640 1 -- 172.21.15.134:0/4293474794 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8248027040 con 0x5585bf999280 2024-07-26T11:15:23.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.141+0000 7f82683f8740 1 -- 172.21.15.134:0/4293474794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8254036040 con 0x5585bf999280 2024-07-26T11:15:23.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.141+0000 7f823be00640 1 -- 172.21.15.134:0/4293474794 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f824802e0b0 con 0x5585bf999280 2024-07-26T11:15:23.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.141+0000 7f823be00640 1 -- 172.21.15.134:0/4293474794 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f824801d040 con 0x5585bf999280 2024-07-26T11:15:23.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.141+0000 7f82683f8740 10 client.25325 fetch_fsmap learned FSMap version 29 2024-07-26T11:15:23.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.141+0000 7f82683f8740 1 -- 172.21.15.134:0/4293474794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5585bfde6120 con 0x5585bf999280 2024-07-26T11:15:23.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.141+0000 7f823be00640 1 -- 172.21.15.134:0/4293474794 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8254036040 con 0x5585bf999280 2024-07-26T11:15:23.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.141+0000 7f823be00640 1 -- 172.21.15.134:0/4293474794 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f8248003710 con 0x5585bf999280 2024-07-26T11:15:23.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.141+0000 7f82683f8740 10 client.25325 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:15:23.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.142+0000 7f82683f8740 10 client.25325 resolve_mds: resolved name 'h' to daemon mds.h 2024-07-26T11:15:23.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.142+0000 7f82683f8740 20 client.25325 populate_metadata read hostname 'smithi134' 2024-07-26T11:15:23.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.142+0000 7f82683f8740 1 --2- 172.21.15.134:0/4293474794 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5585bf94dc90 0x5585bfde9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:23.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.142+0000 7f82683f8740 4 client.25325 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:15:23.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.142+0000 7f82683f8740 1 -- 172.21.15.134:0/4293474794 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "status"}) -- 0x5585bfcd3360 con 0x5585bf94dc90 2024-07-26T11:15:23.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.142+0000 7f825aa00640 1 --2- 172.21.15.134:0/4293474794 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5585bf94dc90 0x5585bfde9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:23.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.143+0000 7f823be00640 10 client.25325 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:15:23.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.143+0000 7f825aa00640 1 --2- 172.21.15.134:0/4293474794 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5585bf94dc90 0x5585bfde9920 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b1fe9c103d78520a server_cookie=7acb649f134a3929 in_seq=0 out_seq=0 2024-07-26T11:15:23.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.143+0000 7f823be00640 1 -- 172.21.15.134:0/4293474794 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5585bfcd3360 con 0x5585bf94dc90 2024-07-26T11:15:23.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.143+0000 7f823be00640 10 client.25325 handle_command_reply: tid=0 2024-07-26T11:15:23.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.143+0000 7f82683f8740 1 client.25325 shutdown 2024-07-26T11:15:23.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.143+0000 7f82683f8740 1 -- 172.21.15.134:0/4293474794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf999280 msgr2=0x5585bfd814e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:23.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.143+0000 7f82683f8740 1 --2- 172.21.15.134:0/4293474794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf999280 0x5585bfd814e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8248004b60 tx=0x7f8248011ed0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.143+0000 7f82683f8740 1 -- 172.21.15.134:0/4293474794 shutdown_connections 2024-07-26T11:15:23.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.143+0000 7f82683f8740 1 --2- 172.21.15.134:0/4293474794 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5585bf94dc90 0x5585bfde9920 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.143+0000 7f82683f8740 1 --2- 172.21.15.134:0/4293474794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5585bf9b9f60 0x5585bfde48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.143+0000 7f82683f8740 1 --2- 172.21.15.134:0/4293474794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5585bf999280 0x5585bfd814e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.143+0000 7f82683f8740 1 --2- 172.21.15.134:0/4293474794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5585bf9910a0 0x5585bfd80f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.143+0000 7f82683f8740 1 -- 172.21.15.134:0/4293474794 >> 172.21.15.134:0/4293474794 conn(0x5585bf9ae350 msgr2=0x5585bfd607b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:23.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.143+0000 7f823be00640 0 client.25325 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:15:23.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.143+0000 7f82683f8740 1 -- 172.21.15.134:0/4293474794 shutdown_connections 2024-07-26T11:15:23.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.144+0000 7f82683f8740 1 -- 172.21.15.134:0/4293474794 wait complete. 2024-07-26T11:15:23.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.144+0000 7f82683f8740 20 client.25325 trim_cache size 0 max 16384 2024-07-26T11:15:23.152 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:15:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_fsid": "105fbe5c-4b3e-11ef-bcb7-c7b262605968", 2024-07-26T11:15:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "whoami": 4, 2024-07-26T11:15:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "id": 24686, 2024-07-26T11:15:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "want_state": "up:active", 2024-07-26T11:15:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "up:active", 2024-07-26T11:15:23.152 INFO:teuthology.orchestra.run.smithi134.stdout: "fs_name": "cephfs", 2024-07-26T11:15:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: "rank_uptime": 213.071311466, 2024-07-26T11:15:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: "mdsmap_epoch": 29, 2024-07-26T11:15:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch": 90, 2024-07-26T11:15:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T11:15:23.153 INFO:teuthology.orchestra.run.smithi134.stdout: "uptime": 334.695529072 2024-07-26T11:15:23.153 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:15:23.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.145+0000 7f825be00640 1 -- 172.21.15.134:0/4011519403 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82400b3ca0 msgr2=0x7f82400b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:23.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.145+0000 7f825be00640 1 --2- 172.21.15.134:0/4011519403 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82400b3ca0 0x7f82400b61b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f825402aeb0 tx=0x7f8254005d80 comp rx=0 tx=0).stop 2024-07-26T11:15:23.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.145+0000 7f825be00640 1 -- 172.21.15.134:0/4011519403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c1055f0 msgr2=0x7f825c19be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:23.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.145+0000 7f825be00640 1 --2- 172.21.15.134:0/4011519403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c1055f0 0x7f825c19be90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f824c00d950 tx=0x7f824c00de10 comp rx=0 tx=0).stop 2024-07-26T11:15:23.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.145+0000 7f825be00640 1 -- 172.21.15.134:0/4011519403 shutdown_connections 2024-07-26T11:15:23.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.145+0000 7f825be00640 1 --2- 172.21.15.134:0/4011519403 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82400b3ca0 0x7f82400b61b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.145+0000 7f825be00640 1 --2- 172.21.15.134:0/4011519403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10b880 0x7f825c1a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.145+0000 7f825be00640 1 --2- 172.21.15.134:0/4011519403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f825c10af30 0x7f825c19c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.145+0000 7f825be00640 1 --2- 172.21.15.134:0/4011519403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c1055f0 0x7f825c19be90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:23.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.145+0000 7f825be00640 1 -- 172.21.15.134:0/4011519403 >> 172.21.15.134:0/4011519403 conn(0x7f825c101060 msgr2=0x7f825c10a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:23.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.145+0000 7f825be00640 1 -- 172.21.15.134:0/4011519403 shutdown_connections 2024-07-26T11:15:23.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:23.145+0000 7f825be00640 1 -- 172.21.15.134:0/4011519403 wait complete. 2024-07-26T11:15:24.112 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.c status 2024-07-26T11:15:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:24 smithi134 ceph-mon[31966]: pgmap v463: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:15:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:24 smithi189 ceph-mon[36790]: pgmap v463: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:15:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:24 smithi193 ceph-mon[37473]: pgmap v463: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:15:24.568 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:15:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:15:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:15:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:15:26.327 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:15:26.327 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:15:26.327 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:15:26.327 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:26 smithi134 ceph-mon[31966]: pgmap v464: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:15:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:15:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:15:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:26 smithi189 ceph-mon[36790]: pgmap v464: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:15:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:15:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:15:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:26 smithi193 ceph-mon[37473]: pgmap v464: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:26.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.905+0000 7f10af000640 1 -- 172.21.15.134:0/2876646379 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10a81099f0 msgr2=0x7f10a8109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:26.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.905+0000 7f10af000640 1 --2- 172.21.15.134:0/2876646379 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10a81099f0 0x7f10a8109e00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f109c009900 tx=0x7f109c02a410 comp rx=0 tx=0).stop 2024-07-26T11:15:26.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.905+0000 7f10af000640 1 -- 172.21.15.134:0/2876646379 shutdown_connections 2024-07-26T11:15:26.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.905+0000 7f10af000640 1 --2- 172.21.15.134:0/2876646379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10a810fc80 0x7f10a81164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.905+0000 7f10af000640 1 --2- 172.21.15.134:0/2876646379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10a810f330 0x7f10a810f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.905+0000 7f10af000640 1 --2- 172.21.15.134:0/2876646379 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10a81099f0 0x7f10a8109e00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.905+0000 7f10af000640 1 -- 172.21.15.134:0/2876646379 >> 172.21.15.134:0/2876646379 conn(0x7f10a8078ff0 msgr2=0x7f10a8107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:26.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.905+0000 7f10af000640 1 -- 172.21.15.134:0/2876646379 shutdown_connections 2024-07-26T11:15:26.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.905+0000 7f10af000640 1 -- 172.21.15.134:0/2876646379 wait complete. 2024-07-26T11:15:26.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.907+0000 7f10af000640 1 Processor -- start 2024-07-26T11:15:26.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.907+0000 7f10af000640 1 -- start start 2024-07-26T11:15:26.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.907+0000 7f10af000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10a81099f0 0x7f10a81a04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.907+0000 7f10af000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10a810f330 0x7f10a81a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.907+0000 7f10af000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10a810fc80 0x7f10a81a5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.907+0000 7f10af000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f10a8119940 con 0x7f10a810f330 2024-07-26T11:15:26.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.907+0000 7f10af000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f10a81197c0 con 0x7f10a810fc80 2024-07-26T11:15:26.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.907+0000 7f10af000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f10a8119ac0 con 0x7f10a81099f0 2024-07-26T11:15:26.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.908+0000 7f10ad200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10a810f330 0x7f10a81a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.908+0000 7f10ae600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10a810fc80 0x7f10a81a5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.908+0000 7f10adc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10a81099f0 0x7f10a81a04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.908+0000 7f10ad200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10a810f330 0x7f10a81a0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:34704/0 (socket says 172.21.15.134:34704) 2024-07-26T11:15:26.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.908+0000 7f10ad200640 1 -- 172.21.15.134:0/1002866618 learned_addr learned my addr 172.21.15.134:0/1002866618 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:26.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.908+0000 7f10ae600640 1 -- 172.21.15.134:0/1002866618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10a81099f0 msgr2=0x7f10a81a04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:26.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.908+0000 7f10ae600640 1 --2- 172.21.15.134:0/1002866618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10a81099f0 0x7f10a81a04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.908+0000 7f10ae600640 1 -- 172.21.15.134:0/1002866618 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10a810f330 msgr2=0x7f10a81a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:26.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.908+0000 7f10ae600640 1 --2- 172.21.15.134:0/1002866618 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10a810f330 0x7f10a81a0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.908+0000 7f10ae600640 1 -- 172.21.15.134:0/1002866618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10a81a6420 con 0x7f10a810fc80 2024-07-26T11:15:26.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.908+0000 7f10ae600640 1 --2- 172.21.15.134:0/1002866618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10a810fc80 0x7f10a81a5cf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f10a0009fd0 tx=0x7f10a000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:26.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.909+0000 7f10a6000640 1 -- 172.21.15.134:0/1002866618 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a000ce20 con 0x7f10a810fc80 2024-07-26T11:15:26.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.909+0000 7f10af000640 1 -- 172.21.15.134:0/1002866618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10a81a6710 con 0x7f10a810fc80 2024-07-26T11:15:26.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.909+0000 7f10a6000640 1 -- 172.21.15.134:0/1002866618 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10a0005020 con 0x7f10a810fc80 2024-07-26T11:15:26.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.909+0000 7f10a6000640 1 -- 172.21.15.134:0/1002866618 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a0010690 con 0x7f10a810fc80 2024-07-26T11:15:26.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.909+0000 7f10af000640 1 -- 172.21.15.134:0/1002866618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10a81ace70 con 0x7f10a810fc80 2024-07-26T11:15:26.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.912+0000 7f10b71d6740 1 Processor -- start 2024-07-26T11:15:26.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.913+0000 7f10a6000640 1 -- 172.21.15.134:0/1002866618 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f10a00054d0 con 0x7f10a810fc80 2024-07-26T11:15:26.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.913+0000 7f10b71d6740 1 -- start start 2024-07-26T11:15:26.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.913+0000 7f10b71d6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562619337f60 0x5626196dd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.913+0000 7f10b71d6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56261930f0a0 0x5626196ddcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.913+0000 7f10b71d6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619317280 0x5626196de200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.913+0000 7f10b71d6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562619653970 con 0x562619337f60 2024-07-26T11:15:26.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.913+0000 7f10b71d6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562619642e70 con 0x562619317280 2024-07-26T11:15:26.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.913+0000 7f10b71d6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562619646eb0 con 0x56261930f0a0 2024-07-26T11:15:26.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.914+0000 7f10adc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56261930f0a0 0x5626196ddcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.914+0000 7f10ad200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562619337f60 0x5626196dd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.914+0000 7f10adc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56261930f0a0 0x5626196ddcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52090/0 (socket says 172.21.15.134:52090) 2024-07-26T11:15:26.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.914+0000 7f10adc00640 1 -- 172.21.15.134:0/2782796360 learned_addr learned my addr 172.21.15.134:0/2782796360 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:26.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.914+0000 7f10ad200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562619337f60 0x5626196dd780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:34710/0 (socket says 172.21.15.134:34710) 2024-07-26T11:15:26.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.914+0000 7f10ad200640 1 --2- 172.21.15.134:0/2782796360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619317280 0x5626196de200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.914+0000 7f10adc00640 1 -- 172.21.15.134:0/2782796360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619317280 msgr2=0x5626196de200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:26.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.914+0000 7f10adc00640 1 --2- 172.21.15.134:0/2782796360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619317280 0x5626196de200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.914+0000 7f10adc00640 1 -- 172.21.15.134:0/2782796360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562619337f60 msgr2=0x5626196dd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:26.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.914+0000 7f10adc00640 1 --2- 172.21.15.134:0/2782796360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562619337f60 0x5626196dd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.914+0000 7f10adc00640 1 -- 172.21.15.134:0/2782796360 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626195ea110 con 0x56261930f0a0 2024-07-26T11:15:26.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.914+0000 7f10ad200640 1 --2- 172.21.15.134:0/2782796360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619317280 0x5626196de200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:15:26.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.915+0000 7f10adc00640 1 --2- 172.21.15.134:0/2782796360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56261930f0a0 0x5626196ddcc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f109c009900 tx=0x7f109c032b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d79feb01299676a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:26.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.915+0000 7f10a6000640 1 --2- 172.21.15.134:0/1002866618 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f10900b3c20 0x7f10900b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.915+0000 7f109aa00640 1 -- 172.21.15.134:0/2782796360 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109c040070 con 0x56261930f0a0 2024-07-26T11:15:26.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.915+0000 7f109aa00640 1 -- 172.21.15.134:0/2782796360 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f109c004030 con 0x56261930f0a0 2024-07-26T11:15:26.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.915+0000 7f109aa00640 1 -- 172.21.15.134:0/2782796360 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109c035040 con 0x56261930f0a0 2024-07-26T11:15:26.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.915+0000 7f10a6000640 1 -- 172.21.15.134:0/1002866618 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f10a00e4780 con 0x7f10a810fc80 2024-07-26T11:15:26.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.915+0000 7f10ad200640 1 --2- 172.21.15.134:0/1002866618 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f10900b3c20 0x7f10900b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.916+0000 7f10b71d6740 1 -- 172.21.15.134:0/2782796360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56261930f0a0 msgr2=0x5626196ddcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:26.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.916+0000 7f10b71d6740 1 --2- 172.21.15.134:0/2782796360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56261930f0a0 0x5626196ddcc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f109c009900 tx=0x7f109c032b90 comp rx=0 tx=0).stop 2024-07-26T11:15:26.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.916+0000 7f10ad200640 1 --2- 172.21.15.134:0/1002866618 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f10900b3c20 0x7f10900b6130 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1094009790 tx=0x7f1094009800 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:26.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.916+0000 7f10b71d6740 1 -- 172.21.15.134:0/2782796360 shutdown_connections 2024-07-26T11:15:26.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.916+0000 7f10b71d6740 1 --2- 172.21.15.134:0/2782796360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56261930f0a0 0x5626196ddcc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.916+0000 7f10b71d6740 1 --2- 172.21.15.134:0/2782796360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619317280 0x5626196de200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.916+0000 7f10b71d6740 1 --2- 172.21.15.134:0/2782796360 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562619337f60 0x5626196dd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.916+0000 7f10b71d6740 1 -- 172.21.15.134:0/2782796360 >> 172.21.15.134:0/2782796360 conn(0x56261932c350 msgr2=0x5626196db670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:26.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.916+0000 7f10b71d6740 1 -- 172.21.15.134:0/2782796360 shutdown_connections 2024-07-26T11:15:26.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.916+0000 7f10b71d6740 1 -- 172.21.15.134:0/2782796360 wait complete. 2024-07-26T11:15:26.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.917+0000 7f10b71d6740 1 -- start start 2024-07-26T11:15:26.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.917+0000 7f10b71d6740 1 Processor -- start 2024-07-26T11:15:26.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.918+0000 7f10b71d6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56261930f0a0 0x5626196e9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.918+0000 7f10b71d6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562619317280 0x5626196e95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.918+0000 7f10b71d6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619337f60 0x5626196f5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.918+0000 7f10b71d6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5626194a65f0 con 0x56261930f0a0 2024-07-26T11:15:26.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.918+0000 7f10b71d6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5626195df540 con 0x562619337f60 2024-07-26T11:15:26.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.918+0000 7f10b71d6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5626196ebf00 con 0x562619317280 2024-07-26T11:15:26.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.918+0000 7f10b71d6740 10 client.0 _ll_register_callbacks cb 0x5626195eef10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:15:26.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.918+0000 7f10ad200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562619317280 0x5626196e95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.918+0000 7f10adc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56261930f0a0 0x5626196e9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.918+0000 7f10ad200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562619317280 0x5626196e95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52096/0 (socket says 172.21.15.134:52096) 2024-07-26T11:15:26.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.918+0000 7f10ad200640 1 -- 172.21.15.134:0/3484273250 learned_addr learned my addr 172.21.15.134:0/3484273250 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:26.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.918+0000 7f10ae600640 1 --2- 172.21.15.134:0/3484273250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619337f60 0x5626196f5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.918+0000 7f10ad200640 1 -- 172.21.15.134:0/3484273250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619337f60 msgr2=0x5626196f5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:26.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.918+0000 7f10ad200640 1 --2- 172.21.15.134:0/3484273250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619337f60 0x5626196f5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.918+0000 7f10ad200640 1 -- 172.21.15.134:0/3484273250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56261930f0a0 msgr2=0x5626196e9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:26.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.918+0000 7f10ad200640 1 --2- 172.21.15.134:0/3484273250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56261930f0a0 0x5626196e9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.919+0000 7f10ad200640 1 -- 172.21.15.134:0/3484273250 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626193a3540 con 0x562619317280 2024-07-26T11:15:26.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.919+0000 7f10ad200640 1 --2- 172.21.15.134:0/3484273250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562619317280 0x5626196e95e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f10940045f0 tx=0x7f1094004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7277a480cfc54df3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:26.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.919+0000 7f108f400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:15:26.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.919+0000 7f108f400640 1 -- 172.21.15.134:0/3484273250 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094020640 con 0x562619317280 2024-07-26T11:15:26.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.919+0000 7f108f400640 1 -- 172.21.15.134:0/3484273250 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f10940207e0 con 0x562619317280 2024-07-26T11:15:26.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.919+0000 7f10b71d6740 1 -- 172.21.15.134:0/3484273250 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f109c004030 con 0x562619317280 2024-07-26T11:15:26.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.920+0000 7f108f400640 1 -- 172.21.15.134:0/3484273250 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1094033050 con 0x562619317280 2024-07-26T11:15:26.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.920+0000 7f10b71d6740 10 client.25348 fetch_fsmap learned FSMap version 29 2024-07-26T11:15:26.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.920+0000 7f10b71d6740 1 -- 172.21.15.134:0/3484273250 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5626196f7da0 con 0x562619317280 2024-07-26T11:15:26.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.920+0000 7f108f400640 1 -- 172.21.15.134:0/3484273250 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f109c004030 con 0x562619317280 2024-07-26T11:15:26.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.920+0000 7f108f400640 1 -- 172.21.15.134:0/3484273250 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f109402abb0 con 0x562619317280 2024-07-26T11:15:26.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.920+0000 7f10b71d6740 10 client.25348 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:15:26.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.920+0000 7f10b71d6740 10 client.25348 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T11:15:26.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.920+0000 7f10b71d6740 20 client.25348 populate_metadata read hostname 'smithi134' 2024-07-26T11:15:26.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.920+0000 7f10b71d6740 1 --2- 172.21.15.134:0/3484273250 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5626192f68e0 0x5626196fb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.920+0000 7f10b71d6740 4 client.25348 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:15:26.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.920+0000 7f10b71d6740 1 -- 172.21.15.134:0/3484273250 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626196558c0 con 0x5626192f68e0 2024-07-26T11:15:26.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.921+0000 7f10adc00640 1 --2- 172.21.15.134:0/3484273250 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5626192f68e0 0x5626196fb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.921+0000 7f10adc00640 1 --2- 172.21.15.134:0/3484273250 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5626192f68e0 0x5626196fb8b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c05a88fbee47a0fe server_cookie=19aae3005d8f3ba0 in_seq=0 out_seq=0 2024-07-26T11:15:26.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.921+0000 7f108f400640 10 client.25348 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:15:26.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.922+0000 7f108f400640 1 -- 172.21.15.134:0/3484273250 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5626196558c0 con 0x5626192f68e0 2024-07-26T11:15:26.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.922+0000 7f108f400640 10 client.25348 handle_command_reply: tid=0 2024-07-26T11:15:26.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.922+0000 7f10b71d6740 1 client.25348 shutdown 2024-07-26T11:15:26.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.922+0000 7f10b71d6740 1 -- 172.21.15.134:0/3484273250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562619317280 msgr2=0x5626196e95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:26.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.922+0000 7f10b71d6740 1 --2- 172.21.15.134:0/3484273250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562619317280 0x5626196e95e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f10940045f0 tx=0x7f1094004730 comp rx=0 tx=0).stop 2024-07-26T11:15:26.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.922+0000 7f10b71d6740 1 -- 172.21.15.134:0/3484273250 shutdown_connections 2024-07-26T11:15:26.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.922+0000 7f10b71d6740 1 --2- 172.21.15.134:0/3484273250 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5626192f68e0 0x5626196fb8b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.922+0000 7f10b71d6740 1 --2- 172.21.15.134:0/3484273250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562619317280 0x5626196e95e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.923+0000 7f10b71d6740 1 --2- 172.21.15.134:0/3484273250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619337f60 0x5626196f5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.923+0000 7f10b71d6740 1 --2- 172.21.15.134:0/3484273250 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56261930f0a0 0x5626196e9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.923+0000 7f108f400640 0 client.25348 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:15:26.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.923+0000 7f10b71d6740 1 -- 172.21.15.134:0/3484273250 >> 172.21.15.134:0/3484273250 conn(0x56261932c350 msgr2=0x5626196e0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:26.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.923+0000 7f10b71d6740 1 -- 172.21.15.134:0/3484273250 shutdown_connections 2024-07-26T11:15:26.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.923+0000 7f10b71d6740 1 -- 172.21.15.134:0/3484273250 wait complete. 2024-07-26T11:15:26.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.923+0000 7f10b71d6740 20 client.25348 trim_cache size 0 max 16384 2024-07-26T11:15:26.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.942+0000 7f10b71d6740 1 Processor -- start 2024-07-26T11:15:26.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.943+0000 7f10b71d6740 1 -- start start 2024-07-26T11:15:26.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.943+0000 7f10b71d6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56261930f0a0 0x5626196e38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.943+0000 7f10b71d6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562619317280 0x5626196f5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.943+0000 7f10b71d6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619337f60 0x5626196f56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.943+0000 7f10b71d6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5626196f9bf0 con 0x56261930f0a0 2024-07-26T11:15:26.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.943+0000 7f10b71d6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5626196fa2c0 con 0x562619337f60 2024-07-26T11:15:26.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.943+0000 7f10b71d6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5626196fa990 con 0x562619317280 2024-07-26T11:15:26.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.943+0000 7f10adc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56261930f0a0 0x5626196e38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.943+0000 7f10ae600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619337f60 0x5626196f56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.943+0000 7f10ad200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562619317280 0x5626196f5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.943+0000 7f10adc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56261930f0a0 0x5626196e38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:34726/0 (socket says 172.21.15.134:34726) 2024-07-26T11:15:26.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.943+0000 7f10ad200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562619317280 0x5626196f5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52100/0 (socket says 172.21.15.134:52100) 2024-07-26T11:15:26.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.943+0000 7f10adc00640 1 -- 172.21.15.134:0/1367824641 learned_addr learned my addr 172.21.15.134:0/1367824641 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:26.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.944+0000 7f10ae600640 1 -- 172.21.15.134:0/1367824641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562619317280 msgr2=0x5626196f5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:26.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.944+0000 7f10ae600640 1 --2- 172.21.15.134:0/1367824641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562619317280 0x5626196f5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.944+0000 7f10ae600640 1 -- 172.21.15.134:0/1367824641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56261930f0a0 msgr2=0x5626196e38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:26.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.944+0000 7f10ae600640 1 --2- 172.21.15.134:0/1367824641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56261930f0a0 0x5626196e38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.944+0000 7f10ae600640 1 -- 172.21.15.134:0/1367824641 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626196d67f0 con 0x562619337f60 2024-07-26T11:15:26.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.944+0000 7f10ae600640 1 --2- 172.21.15.134:0/1367824641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619337f60 0x5626196f56d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f10a00aaed0 tx=0x7f10a00a7110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d33ab9cc1c0ffee server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:26.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.944+0000 7f109a000640 1 -- 172.21.15.134:0/1367824641 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a00c4070 con 0x562619337f60 2024-07-26T11:15:26.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.945+0000 7f109a000640 1 -- 172.21.15.134:0/1367824641 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10a00b4280 con 0x562619337f60 2024-07-26T11:15:26.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.945+0000 7f109a000640 1 -- 172.21.15.134:0/1367824641 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a00e42c0 con 0x562619337f60 2024-07-26T11:15:26.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.945+0000 7f10b71d6740 1 -- 172.21.15.134:0/1367824641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619337f60 msgr2=0x5626196f56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:26.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.945+0000 7f10b71d6740 1 --2- 172.21.15.134:0/1367824641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619337f60 0x5626196f56d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f10a00aaed0 tx=0x7f10a00a7110 comp rx=0 tx=0).stop 2024-07-26T11:15:26.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.945+0000 7f10b71d6740 1 -- 172.21.15.134:0/1367824641 shutdown_connections 2024-07-26T11:15:26.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.945+0000 7f10b71d6740 1 --2- 172.21.15.134:0/1367824641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562619317280 0x5626196f5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.945+0000 7f10b71d6740 1 --2- 172.21.15.134:0/1367824641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619337f60 0x5626196f56d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.945+0000 7f10b71d6740 1 --2- 172.21.15.134:0/1367824641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56261930f0a0 0x5626196e38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.945+0000 7f10b71d6740 1 -- 172.21.15.134:0/1367824641 >> 172.21.15.134:0/1367824641 conn(0x56261932c350 msgr2=0x5626196e78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:26.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.945+0000 7f10b71d6740 1 -- 172.21.15.134:0/1367824641 shutdown_connections 2024-07-26T11:15:26.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.946+0000 7f10b71d6740 1 -- 172.21.15.134:0/1367824641 wait complete. 2024-07-26T11:15:26.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.946+0000 7f10b71d6740 1 -- start start 2024-07-26T11:15:26.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.946+0000 7f10b71d6740 1 Processor -- start 2024-07-26T11:15:26.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.947+0000 7f10b71d6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56261930f0a0 0x5626196fcd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.947+0000 7f10b71d6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619317280 0x5626196fd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.947+0000 7f10b71d6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562619337f60 0x562619702320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.947+0000 7f10b71d6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5626196fb730 con 0x562619337f60 2024-07-26T11:15:26.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.947+0000 7f10b71d6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5626196fb060 con 0x562619317280 2024-07-26T11:15:26.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.947+0000 7f10b71d6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5626196fbe00 con 0x56261930f0a0 2024-07-26T11:15:26.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.947+0000 7f10b71d6740 10 client.0 _ll_register_callbacks cb 0x7f10940044b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:15:26.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.947+0000 7f10adc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56261930f0a0 0x5626196fcd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.947+0000 7f10adc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56261930f0a0 0x5626196fcd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52104/0 (socket says 172.21.15.134:52104) 2024-07-26T11:15:26.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.947+0000 7f10adc00640 1 -- 172.21.15.134:0/2175524276 learned_addr learned my addr 172.21.15.134:0/2175524276 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:26.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.947+0000 7f10ae600640 1 --2- 172.21.15.134:0/2175524276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562619337f60 0x562619702320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.947+0000 7f10ad200640 1 --2- 172.21.15.134:0/2175524276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619317280 0x5626196fd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.948+0000 7f10adc00640 1 -- 172.21.15.134:0/2175524276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619317280 msgr2=0x5626196fd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:26.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.948+0000 7f10adc00640 1 --2- 172.21.15.134:0/2175524276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619317280 0x5626196fd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.948+0000 7f10adc00640 1 -- 172.21.15.134:0/2175524276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562619337f60 msgr2=0x562619702320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:26.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.948+0000 7f10adc00640 1 --2- 172.21.15.134:0/2175524276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562619337f60 0x562619702320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.948+0000 7f10adc00640 1 -- 172.21.15.134:0/2175524276 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626196df6c0 con 0x56261930f0a0 2024-07-26T11:15:26.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.948+0000 7f10adc00640 1 --2- 172.21.15.134:0/2175524276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56261930f0a0 0x5626196fcd70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f109c006010 tx=0x7f109c02abc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55764b77bcd196f3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:26.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.948+0000 7f108f400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:15:26.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.948+0000 7f108f400640 1 -- 172.21.15.134:0/2175524276 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109c035040 con 0x56261930f0a0 2024-07-26T11:15:26.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.948+0000 7f108f400640 1 -- 172.21.15.134:0/2175524276 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f109c031b10 con 0x56261930f0a0 2024-07-26T11:15:26.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.948+0000 7f10b71d6740 1 -- 172.21.15.134:0/2175524276 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10a00b4280 con 0x56261930f0a0 2024-07-26T11:15:26.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.948+0000 7f108f400640 1 -- 172.21.15.134:0/2175524276 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f109c040040 con 0x56261930f0a0 2024-07-26T11:15:26.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.948+0000 7f10b71d6740 10 client.25354 fetch_fsmap learned FSMap version 29 2024-07-26T11:15:26.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.948+0000 7f10b71d6740 1 -- 172.21.15.134:0/2175524276 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562619703b80 con 0x56261930f0a0 2024-07-26T11:15:26.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.949+0000 7f108f400640 1 -- 172.21.15.134:0/2175524276 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10a00b4280 con 0x56261930f0a0 2024-07-26T11:15:26.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.949+0000 7f108f400640 1 -- 172.21.15.134:0/2175524276 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f109c056070 con 0x56261930f0a0 2024-07-26T11:15:26.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.949+0000 7f10b71d6740 10 client.25354 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:15:26.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.949+0000 7f10b71d6740 10 client.25354 resolve_mds: resolved name 'c' to daemon mds.c 2024-07-26T11:15:26.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.949+0000 7f10b71d6740 20 client.25354 populate_metadata read hostname 'smithi134' 2024-07-26T11:15:26.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.949+0000 7f10b71d6740 1 --2- 172.21.15.134:0/2175524276 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5626192cbc90 0x5626197670f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:26.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.949+0000 7f10b71d6740 4 client.25354 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:15:26.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.949+0000 7f10b71d6740 1 -- 172.21.15.134:0/2175524276 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "status"}) -- 0x562619651360 con 0x5626192cbc90 2024-07-26T11:15:26.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.949+0000 7f10ad200640 1 --2- 172.21.15.134:0/2175524276 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5626192cbc90 0x5626197670f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:26.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.950+0000 7f10ad200640 1 --2- 172.21.15.134:0/2175524276 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5626192cbc90 0x5626197670f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d0188ed33832a7ef server_cookie=285436ccc0632947 in_seq=0 out_seq=0 2024-07-26T11:15:26.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.950+0000 7f108f400640 10 client.25354 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:15:26.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.950+0000 7f108f400640 1 -- 172.21.15.134:0/2175524276 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x562619651360 con 0x5626192cbc90 2024-07-26T11:15:26.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.950+0000 7f108f400640 10 client.25354 handle_command_reply: tid=0 2024-07-26T11:15:26.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.950+0000 7f10b71d6740 1 client.25354 shutdown 2024-07-26T11:15:26.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.950+0000 7f10b71d6740 1 -- 172.21.15.134:0/2175524276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56261930f0a0 msgr2=0x5626196fcd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:26.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.950+0000 7f10b71d6740 1 --2- 172.21.15.134:0/2175524276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56261930f0a0 0x5626196fcd70 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f109c006010 tx=0x7f109c02abc0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.951+0000 7f10b71d6740 1 -- 172.21.15.134:0/2175524276 shutdown_connections 2024-07-26T11:15:26.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.951+0000 7f10b71d6740 1 --2- 172.21.15.134:0/2175524276 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5626192cbc90 0x5626197670f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.951+0000 7f10b71d6740 1 --2- 172.21.15.134:0/2175524276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562619337f60 0x562619702320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.951+0000 7f10b71d6740 1 --2- 172.21.15.134:0/2175524276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562619317280 0x5626196fd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.951+0000 7f10b71d6740 1 --2- 172.21.15.134:0/2175524276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56261930f0a0 0x5626196fcd70 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.951+0000 7f10b71d6740 1 -- 172.21.15.134:0/2175524276 >> 172.21.15.134:0/2175524276 conn(0x56261932c350 msgr2=0x5626196e8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:26.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.951+0000 7f108f400640 0 client.25354 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:15:26.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.951+0000 7f10b71d6740 1 -- 172.21.15.134:0/2175524276 shutdown_connections 2024-07-26T11:15:26.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.951+0000 7f10b71d6740 1 -- 172.21.15.134:0/2175524276 wait complete. 2024-07-26T11:15:26.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.951+0000 7f10b71d6740 20 client.25354 trim_cache size 0 max 16384 2024-07-26T11:15:26.957 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:15:26.957 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_fsid": "105fbe5c-4b3e-11ef-bcb7-c7b262605968", 2024-07-26T11:15:26.957 INFO:teuthology.orchestra.run.smithi134.stdout: "whoami": 3, 2024-07-26T11:15:26.957 INFO:teuthology.orchestra.run.smithi134.stdout: "id": 24688, 2024-07-26T11:15:26.957 INFO:teuthology.orchestra.run.smithi134.stdout: "want_state": "up:active", 2024-07-26T11:15:26.957 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "up:active", 2024-07-26T11:15:26.957 INFO:teuthology.orchestra.run.smithi134.stdout: "fs_name": "cephfs", 2024-07-26T11:15:26.957 INFO:teuthology.orchestra.run.smithi134.stdout: "rank_uptime": 217.89700480299999, 2024-07-26T11:15:26.957 INFO:teuthology.orchestra.run.smithi134.stdout: "mdsmap_epoch": 29, 2024-07-26T11:15:26.957 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch": 90, 2024-07-26T11:15:26.958 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T11:15:26.958 INFO:teuthology.orchestra.run.smithi134.stdout: "uptime": 341.17869400199999 2024-07-26T11:15:26.958 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:15:26.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.952+0000 7f10af000640 1 -- 172.21.15.134:0/1002866618 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f10900b3c20 msgr2=0x7f10900b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:26.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.952+0000 7f10af000640 1 --2- 172.21.15.134:0/1002866618 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f10900b3c20 0x7f10900b6130 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1094009790 tx=0x7f1094009800 comp rx=0 tx=0).stop 2024-07-26T11:15:26.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.952+0000 7f10af000640 1 -- 172.21.15.134:0/1002866618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10a810fc80 msgr2=0x7f10a81a5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:26.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.952+0000 7f10af000640 1 --2- 172.21.15.134:0/1002866618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10a810fc80 0x7f10a81a5cf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f10a0009fd0 tx=0x7f10a000ef10 comp rx=0 tx=0).stop 2024-07-26T11:15:26.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.952+0000 7f10af000640 1 -- 172.21.15.134:0/1002866618 shutdown_connections 2024-07-26T11:15:26.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.952+0000 7f10af000640 1 --2- 172.21.15.134:0/1002866618 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f10900b3c20 0x7f10900b6130 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.952+0000 7f10af000640 1 --2- 172.21.15.134:0/1002866618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10a810fc80 0x7f10a81a5cf0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.952+0000 7f10af000640 1 --2- 172.21.15.134:0/1002866618 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f10a810f330 0x7f10a81a0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.952+0000 7f10af000640 1 --2- 172.21.15.134:0/1002866618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f10a81099f0 0x7f10a81a04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:26.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.952+0000 7f10af000640 1 -- 172.21.15.134:0/1002866618 >> 172.21.15.134:0/1002866618 conn(0x7f10a8078ff0 msgr2=0x7f10a8107ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:26.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.952+0000 7f10af000640 1 -- 172.21.15.134:0/1002866618 shutdown_connections 2024-07-26T11:15:26.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:26.952+0000 7f10af000640 1 -- 172.21.15.134:0/1002866618 wait complete. 2024-07-26T11:15:27.840 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.i status 2024-07-26T11:15:28.325 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:15:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:28 smithi134 ceph-mon[31966]: pgmap v465: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:15:28.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:28 smithi189 ceph-mon[36790]: pgmap v465: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:15:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:28 smithi193 ceph-mon[37473]: pgmap v465: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:15:30.389 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:30 smithi134 ceph-mon[31966]: pgmap v466: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:30 smithi189 ceph-mon[36790]: pgmap v466: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:30 smithi193 ceph-mon[37473]: pgmap v466: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:30.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.714+0000 7f2623200640 1 -- 172.21.15.134:0/86390228 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f262410b860 msgr2=0x7f2624112090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:30.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.714+0000 7f2623200640 1 --2- 172.21.15.134:0/86390228 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f262410b860 0x7f2624112090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f261c009ca0 tx=0x7f261c02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.714+0000 7f2623200640 1 -- 172.21.15.134:0/86390228 shutdown_connections 2024-07-26T11:15:30.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.714+0000 7f2623200640 1 --2- 172.21.15.134:0/86390228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f262410af10 0x7f262410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.714+0000 7f2623200640 1 --2- 172.21.15.134:0/86390228 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f262410b860 0x7f2624112090 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.714+0000 7f2623200640 1 --2- 172.21.15.134:0/86390228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26241055d0 0x7f26241059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.714+0000 7f2623200640 1 -- 172.21.15.134:0/86390228 >> 172.21.15.134:0/86390228 conn(0x7f2624101040 msgr2=0x7f2624103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:30.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.714+0000 7f2623200640 1 -- 172.21.15.134:0/86390228 shutdown_connections 2024-07-26T11:15:30.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.714+0000 7f2623200640 1 -- 172.21.15.134:0/86390228 wait complete. 2024-07-26T11:15:30.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.716+0000 7f2623200640 1 Processor -- start 2024-07-26T11:15:30.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.716+0000 7f2623200640 1 -- start start 2024-07-26T11:15:30.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.716+0000 7f2623200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26241055d0 0x7f262419bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.716+0000 7f2623200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f262410af10 0x7f262419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.716+0000 7f2623200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f262410b860 0x7f26241a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.716+0000 7f2623200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2624115510 con 0x7f26241055d0 2024-07-26T11:15:30.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.716+0000 7f2623200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2624115390 con 0x7f262410af10 2024-07-26T11:15:30.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.716+0000 7f2623200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2624115690 con 0x7f262410b860 2024-07-26T11:15:30.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.717+0000 7f2621e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26241055d0 0x7f262419bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.717+0000 7f2621400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f262410af10 0x7f262419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.717+0000 7f2621e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26241055d0 0x7f262419bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36284/0 (socket says 172.21.15.134:36284) 2024-07-26T11:15:30.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.717+0000 7f2621e00640 1 -- 172.21.15.134:0/1241675457 learned_addr learned my addr 172.21.15.134:0/1241675457 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:30.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.717+0000 7f2622800640 1 --2- 172.21.15.134:0/1241675457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f262410b860 0x7f26241a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.717+0000 7f2621400640 1 -- 172.21.15.134:0/1241675457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f262410b860 msgr2=0x7f26241a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:30.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.717+0000 7f2621400640 1 --2- 172.21.15.134:0/1241675457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f262410b860 0x7f26241a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.717+0000 7f2621400640 1 -- 172.21.15.134:0/1241675457 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26241055d0 msgr2=0x7f262419bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:30.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.717+0000 7f2621400640 1 --2- 172.21.15.134:0/1241675457 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26241055d0 0x7f262419bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.717+0000 7f2621400640 1 -- 172.21.15.134:0/1241675457 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26241a1e10 con 0x7f262410af10 2024-07-26T11:15:30.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.717+0000 7f2622800640 1 --2- 172.21.15.134:0/1241675457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f262410b860 0x7f26241a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:15:30.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.717+0000 7f2621e00640 1 --2- 172.21.15.134:0/1241675457 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26241055d0 0x7f262419bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:30.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.718+0000 7f2621400640 1 --2- 172.21.15.134:0/1241675457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f262410af10 0x7f262419c440 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f262406c270 tx=0x7f261000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:30.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.718+0000 7f261aa00640 1 -- 172.21.15.134:0/1241675457 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261000cd00 con 0x7f262410af10 2024-07-26T11:15:30.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.718+0000 7f261aa00640 1 -- 172.21.15.134:0/1241675457 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26100035f0 con 0x7f262410af10 2024-07-26T11:15:30.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.718+0000 7f2623200640 1 -- 172.21.15.134:0/1241675457 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26241a2100 con 0x7f262410af10 2024-07-26T11:15:30.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.718+0000 7f261aa00640 1 -- 172.21.15.134:0/1241675457 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2610010690 con 0x7f262410af10 2024-07-26T11:15:30.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.718+0000 7f2623200640 1 -- 172.21.15.134:0/1241675457 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26241a8860 con 0x7f262410af10 2024-07-26T11:15:30.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.722+0000 7f262f9ec740 1 Processor -- start 2024-07-26T11:15:30.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.722+0000 7f262f9ec740 1 -- start start 2024-07-26T11:15:30.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.722+0000 7f261aa00640 1 -- 172.21.15.134:0/1241675457 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f26100037a0 con 0x7f262410af10 2024-07-26T11:15:30.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.722+0000 7f262f9ec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9315df60 0x55ac93503780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.722+0000 7f262f9ec740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 0x55ac93503cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.722+0000 7f262f9ec740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 0x55ac93504200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.722+0000 7f262f9ec740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ac93479970 con 0x55ac9313d280 2024-07-26T11:15:30.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.722+0000 7f262f9ec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ac93468e70 con 0x55ac9315df60 2024-07-26T11:15:30.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.722+0000 7f262f9ec740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ac9346ceb0 con 0x55ac931350a0 2024-07-26T11:15:30.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.723+0000 7f2621400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 0x55ac93504200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.723+0000 7f2621e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 0x55ac93503cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.723+0000 7f2621400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 0x55ac93504200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36292/0 (socket says 172.21.15.134:36292) 2024-07-26T11:15:30.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.723+0000 7f2621400640 1 -- 172.21.15.134:0/1384173472 learned_addr learned my addr 172.21.15.134:0/1384173472 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:30.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.723+0000 7f2622800640 1 --2- 172.21.15.134:0/1384173472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9315df60 0x55ac93503780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.723+0000 7f2622800640 1 -- 172.21.15.134:0/1384173472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 msgr2=0x55ac93503cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:30.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.723+0000 7f2622800640 1 --2- 172.21.15.134:0/1384173472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 0x55ac93503cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.723+0000 7f2622800640 1 -- 172.21.15.134:0/1384173472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 msgr2=0x55ac93504200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:30.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.723+0000 7f2622800640 1 --2- 172.21.15.134:0/1384173472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 0x55ac93504200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.723+0000 7f2622800640 1 -- 172.21.15.134:0/1384173472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac93410110 con 0x55ac9315df60 2024-07-26T11:15:30.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.723+0000 7f2621400640 1 --2- 172.21.15.134:0/1384173472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 0x55ac93504200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:30.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.724+0000 7f2622800640 1 --2- 172.21.15.134:0/1384173472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9315df60 0x55ac93503780 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f261c02ade0 tx=0x7f261c009440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee478ed3fec98bd5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:30.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.724+0000 7f260f000640 1 -- 172.21.15.134:0/1384173472 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c031cf0 con 0x55ac9315df60 2024-07-26T11:15:30.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.724+0000 7f260f000640 1 -- 172.21.15.134:0/1384173472 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f261c036040 con 0x55ac9315df60 2024-07-26T11:15:30.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.724+0000 7f260f000640 1 -- 172.21.15.134:0/1384173472 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c0042a0 con 0x55ac9315df60 2024-07-26T11:15:30.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.725+0000 7f261aa00640 1 --2- 172.21.15.134:0/1241675457 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f26080b3c20 0x7f26080b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.725+0000 7f262f9ec740 1 -- 172.21.15.134:0/1384173472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9315df60 msgr2=0x55ac93503780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:30.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.725+0000 7f262f9ec740 1 --2- 172.21.15.134:0/1384173472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9315df60 0x55ac93503780 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f261c02ade0 tx=0x7f261c009440 comp rx=0 tx=0).stop 2024-07-26T11:15:30.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.725+0000 7f261aa00640 1 -- 172.21.15.134:0/1241675457 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f26100e4420 con 0x7f262410af10 2024-07-26T11:15:30.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.725+0000 7f262f9ec740 1 -- 172.21.15.134:0/1384173472 shutdown_connections 2024-07-26T11:15:30.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.725+0000 7f262f9ec740 1 --2- 172.21.15.134:0/1384173472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 0x55ac93503cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.725+0000 7f262f9ec740 1 --2- 172.21.15.134:0/1384173472 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 0x55ac93504200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.725+0000 7f262f9ec740 1 --2- 172.21.15.134:0/1384173472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9315df60 0x55ac93503780 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.725+0000 7f262f9ec740 1 -- 172.21.15.134:0/1384173472 >> 172.21.15.134:0/1384173472 conn(0x55ac93152350 msgr2=0x55ac93501670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:30.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.725+0000 7f2621e00640 1 --2- 172.21.15.134:0/1241675457 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f26080b3c20 0x7f26080b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.725+0000 7f262f9ec740 1 -- 172.21.15.134:0/1384173472 shutdown_connections 2024-07-26T11:15:30.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.725+0000 7f262f9ec740 1 -- 172.21.15.134:0/1384173472 wait complete. 2024-07-26T11:15:30.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.726+0000 7f2621e00640 1 --2- 172.21.15.134:0/1241675457 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f26080b3c20 0x7f26080b6130 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x55ac93502400 tx=0x7f2614005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:30.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.726+0000 7f262f9ec740 1 -- start start 2024-07-26T11:15:30.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.726+0000 7f262f9ec740 1 Processor -- start 2024-07-26T11:15:30.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.727+0000 7f262f9ec740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 0x55ac9350efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.727+0000 7f262f9ec740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 0x55ac9350f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.727+0000 7f262f9ec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9315df60 0x55ac9351b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.727+0000 7f262f9ec740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ac932cc5f0 con 0x55ac9313d280 2024-07-26T11:15:30.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.727+0000 7f262f9ec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ac93405540 con 0x55ac9315df60 2024-07-26T11:15:30.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.727+0000 7f262f9ec740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ac93511f00 con 0x55ac931350a0 2024-07-26T11:15:30.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.727+0000 7f2621400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 0x55ac9350f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.727+0000 7f2621e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 0x55ac9350efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.727+0000 7f2621e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 0x55ac9350efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34620/0 (socket says 172.21.15.134:34620) 2024-07-26T11:15:30.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.727+0000 7f2621e00640 1 -- 172.21.15.134:0/4284531707 learned_addr learned my addr 172.21.15.134:0/4284531707 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:30.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.727+0000 7f262f9ec740 10 client.0 _ll_register_callbacks cb 0x55ac93414f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:15:30.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.727+0000 7f2622800640 1 --2- 172.21.15.134:0/4284531707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9315df60 0x55ac9351b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.728+0000 7f2622800640 1 -- 172.21.15.134:0/4284531707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 msgr2=0x55ac9350efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:30.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.728+0000 7f2622800640 1 --2- 172.21.15.134:0/4284531707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 0x55ac9350efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.728+0000 7f2622800640 1 -- 172.21.15.134:0/4284531707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 msgr2=0x55ac9350f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:30.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.728+0000 7f2622800640 1 --2- 172.21.15.134:0/4284531707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 0x55ac9350f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.728+0000 7f2622800640 1 -- 172.21.15.134:0/4284531707 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac931c9540 con 0x55ac9315df60 2024-07-26T11:15:30.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.728+0000 7f2622800640 1 --2- 172.21.15.134:0/4284531707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9315df60 0x55ac9351b1f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f261c009a60 tx=0x7f261c02af40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ed23acb9cdbd4a6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:30.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.728+0000 7f2603400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:15:30.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.728+0000 7f2603400640 1 -- 172.21.15.134:0/4284531707 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261c04a040 con 0x55ac9315df60 2024-07-26T11:15:30.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.728+0000 7f2603400640 1 -- 172.21.15.134:0/4284531707 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f261c003b70 con 0x55ac9315df60 2024-07-26T11:15:30.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.728+0000 7f262f9ec740 1 -- 172.21.15.134:0/4284531707 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f261c036040 con 0x55ac9315df60 2024-07-26T11:15:30.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.729+0000 7f2603400640 1 -- 172.21.15.134:0/4284531707 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f261c05c050 con 0x55ac9315df60 2024-07-26T11:15:30.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.729+0000 7f262f9ec740 10 client.25352 fetch_fsmap learned FSMap version 29 2024-07-26T11:15:30.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.729+0000 7f262f9ec740 1 -- 172.21.15.134:0/4284531707 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ac9351dda0 con 0x55ac9315df60 2024-07-26T11:15:30.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.729+0000 7f2603400640 1 -- 172.21.15.134:0/4284531707 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f261c036040 con 0x55ac9315df60 2024-07-26T11:15:30.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.729+0000 7f2603400640 1 -- 172.21.15.134:0/4284531707 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f261c04d070 con 0x55ac9315df60 2024-07-26T11:15:30.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.729+0000 7f262f9ec740 10 client.25352 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:15:30.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.730+0000 7f262f9ec740 10 client.25352 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T11:15:30.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.730+0000 7f262f9ec740 20 client.25352 populate_metadata read hostname 'smithi134' 2024-07-26T11:15:30.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.730+0000 7f262f9ec740 1 --2- 172.21.15.134:0/4284531707 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55ac9311c8e0 0x55ac93521350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.730+0000 7f262f9ec740 4 client.25352 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:15:30.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.730+0000 7f262f9ec740 1 -- 172.21.15.134:0/4284531707 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ac9347b8c0 con 0x55ac9311c8e0 2024-07-26T11:15:30.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.730+0000 7f2621e00640 1 --2- 172.21.15.134:0/4284531707 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55ac9311c8e0 0x55ac93521350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.731+0000 7f2621e00640 1 --2- 172.21.15.134:0/4284531707 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55ac9311c8e0 0x55ac93521350 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=84f5fdb4cff1079a server_cookie=e7faec0df5572623 in_seq=0 out_seq=0 2024-07-26T11:15:30.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.731+0000 7f2603400640 10 client.25352 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:15:30.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.731+0000 7f2603400640 1 -- 172.21.15.134:0/4284531707 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ac9347b8c0 con 0x55ac9311c8e0 2024-07-26T11:15:30.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.731+0000 7f2603400640 10 client.25352 handle_command_reply: tid=0 2024-07-26T11:15:30.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.732+0000 7f262f9ec740 1 client.25352 shutdown 2024-07-26T11:15:30.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.732+0000 7f262f9ec740 1 -- 172.21.15.134:0/4284531707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9315df60 msgr2=0x55ac9351b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:30.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.732+0000 7f262f9ec740 1 --2- 172.21.15.134:0/4284531707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9315df60 0x55ac9351b1f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f261c009a60 tx=0x7f261c02af40 comp rx=0 tx=0).stop 2024-07-26T11:15:30.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.732+0000 7f262f9ec740 1 -- 172.21.15.134:0/4284531707 shutdown_connections 2024-07-26T11:15:30.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.732+0000 7f262f9ec740 1 --2- 172.21.15.134:0/4284531707 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55ac9311c8e0 0x55ac93521350 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.732+0000 7f262f9ec740 1 --2- 172.21.15.134:0/4284531707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9315df60 0x55ac9351b1f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.732+0000 7f262f9ec740 1 --2- 172.21.15.134:0/4284531707 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 0x55ac9350f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.732+0000 7f262f9ec740 1 --2- 172.21.15.134:0/4284531707 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 0x55ac9350efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.732+0000 7f262f9ec740 1 -- 172.21.15.134:0/4284531707 >> 172.21.15.134:0/4284531707 conn(0x55ac93152350 msgr2=0x55ac93506960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:30.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.732+0000 7f2603400640 0 client.25352 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:15:30.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.732+0000 7f262f9ec740 1 -- 172.21.15.134:0/4284531707 shutdown_connections 2024-07-26T11:15:30.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.732+0000 7f262f9ec740 1 -- 172.21.15.134:0/4284531707 wait complete. 2024-07-26T11:15:30.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.732+0000 7f262f9ec740 20 client.25352 trim_cache size 0 max 16384 2024-07-26T11:15:30.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.751+0000 7f262f9ec740 1 Processor -- start 2024-07-26T11:15:30.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.752+0000 7f262f9ec740 1 -- start start 2024-07-26T11:15:30.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.752+0000 7f262f9ec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac931350a0 0x55ac93520c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.752+0000 7f262f9ec740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 0x55ac93509870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.752+0000 7f262f9ec740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac9315df60 0x55ac93500340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.752+0000 7f262f9ec740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ac9351f690 con 0x55ac9313d280 2024-07-26T11:15:30.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.752+0000 7f262f9ec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ac9351fd60 con 0x55ac931350a0 2024-07-26T11:15:30.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.752+0000 7f262f9ec740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ac93520430 con 0x55ac9315df60 2024-07-26T11:15:30.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.752+0000 7f2621400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 0x55ac93509870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.752+0000 7f2621400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 0x55ac93509870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36300/0 (socket says 172.21.15.134:36300) 2024-07-26T11:15:30.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.752+0000 7f2621400640 1 -- 172.21.15.134:0/3507556865 learned_addr learned my addr 172.21.15.134:0/3507556865 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:30.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.752+0000 7f2622800640 1 --2- 172.21.15.134:0/3507556865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac9315df60 0x55ac93500340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.752+0000 7f2621e00640 1 --2- 172.21.15.134:0/3507556865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac931350a0 0x55ac93520c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.753+0000 7f2621400640 1 -- 172.21.15.134:0/3507556865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac9315df60 msgr2=0x55ac93500340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:30.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.753+0000 7f2621400640 1 --2- 172.21.15.134:0/3507556865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac9315df60 0x55ac93500340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.753+0000 7f2621400640 1 -- 172.21.15.134:0/3507556865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac931350a0 msgr2=0x55ac93520c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:30.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.753+0000 7f2621400640 1 --2- 172.21.15.134:0/3507556865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac931350a0 0x55ac93520c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.753+0000 7f2621400640 1 -- 172.21.15.134:0/3507556865 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ac934fc7f0 con 0x55ac9313d280 2024-07-26T11:15:30.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.753+0000 7f2621e00640 1 --2- 172.21.15.134:0/3507556865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac931350a0 0x55ac93520c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:30.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.753+0000 7f2621400640 1 --2- 172.21.15.134:0/3507556865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 0x55ac93509870 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f26100a6f50 tx=0x7f26100e7f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6fb6ea30e4e2d00 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:30.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.753+0000 7f260e600640 1 -- 172.21.15.134:0/3507556865 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2610009460 con 0x55ac9313d280 2024-07-26T11:15:30.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.753+0000 7f260e600640 1 -- 172.21.15.134:0/3507556865 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26100aa9e0 con 0x55ac9313d280 2024-07-26T11:15:30.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.753+0000 7f260e600640 1 -- 172.21.15.134:0/3507556865 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26100e4160 con 0x55ac9313d280 2024-07-26T11:15:30.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.754+0000 7f262f9ec740 1 -- 172.21.15.134:0/3507556865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 msgr2=0x55ac93509870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:30.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.754+0000 7f262f9ec740 1 --2- 172.21.15.134:0/3507556865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 0x55ac93509870 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f26100a6f50 tx=0x7f26100e7f00 comp rx=0 tx=0).stop 2024-07-26T11:15:30.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.754+0000 7f262f9ec740 1 -- 172.21.15.134:0/3507556865 shutdown_connections 2024-07-26T11:15:30.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.754+0000 7f262f9ec740 1 --2- 172.21.15.134:0/3507556865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac9315df60 0x55ac93500340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.754+0000 7f262f9ec740 1 --2- 172.21.15.134:0/3507556865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9313d280 0x55ac93509870 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.754+0000 7f262f9ec740 1 --2- 172.21.15.134:0/3507556865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac931350a0 0x55ac93520c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.754+0000 7f262f9ec740 1 -- 172.21.15.134:0/3507556865 >> 172.21.15.134:0/3507556865 conn(0x55ac93152350 msgr2=0x55ac934ffc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:30.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.754+0000 7f262f9ec740 1 -- 172.21.15.134:0/3507556865 shutdown_connections 2024-07-26T11:15:30.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.755+0000 7f262f9ec740 1 -- 172.21.15.134:0/3507556865 wait complete. 2024-07-26T11:15:30.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.755+0000 7f262f9ec740 1 -- start start 2024-07-26T11:15:30.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.755+0000 7f262f9ec740 1 Processor -- start 2024-07-26T11:15:30.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.756+0000 7f262f9ec740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 0x55ac93522980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.756+0000 7f262f9ec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9313d280 0x55ac93522f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.756+0000 7f262f9ec740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9315df60 0x55ac93527f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.756+0000 7f262f9ec740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ac935211d0 con 0x55ac9315df60 2024-07-26T11:15:30.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.756+0000 7f262f9ec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ac93520b00 con 0x55ac9313d280 2024-07-26T11:15:30.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.756+0000 7f262f9ec740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ac935218a0 con 0x55ac931350a0 2024-07-26T11:15:30.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.756+0000 7f262f9ec740 10 client.0 _ll_register_callbacks cb 0x7f2610005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:15:30.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.756+0000 7f2622800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9315df60 0x55ac93527f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.756+0000 7f2622800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9315df60 0x55ac93527f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36308/0 (socket says 172.21.15.134:36308) 2024-07-26T11:15:30.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.756+0000 7f2621e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 0x55ac93522980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.756+0000 7f2622800640 1 -- 172.21.15.134:0/1780070025 learned_addr learned my addr 172.21.15.134:0/1780070025 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:30.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.756+0000 7f2621400640 1 --2- 172.21.15.134:0/1780070025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9313d280 0x55ac93522f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.757+0000 7f2621e00640 1 -- 172.21.15.134:0/1780070025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9313d280 msgr2=0x55ac93522f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:30.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.757+0000 7f2621e00640 1 --2- 172.21.15.134:0/1780070025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9313d280 0x55ac93522f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.757+0000 7f2621e00640 1 -- 172.21.15.134:0/1780070025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9315df60 msgr2=0x55ac93527f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:30.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.757+0000 7f2621e00640 1 --2- 172.21.15.134:0/1780070025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9315df60 0x55ac93527f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.757+0000 7f2621e00640 1 -- 172.21.15.134:0/1780070025 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ac935285a0 con 0x55ac931350a0 2024-07-26T11:15:30.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.757+0000 7f2621e00640 1 --2- 172.21.15.134:0/1780070025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 0x55ac93522980 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x55ac935224d0 tx=0x7f26140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5f011b070763ced server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:30.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.757+0000 7f2603400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:15:30.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.757+0000 7f2603400640 1 -- 172.21.15.134:0/1780070025 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2614003c60 con 0x55ac931350a0 2024-07-26T11:15:30.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.758+0000 7f262f9ec740 1 -- 172.21.15.134:0/1780070025 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26100aa9e0 con 0x55ac931350a0 2024-07-26T11:15:30.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.758+0000 7f2603400640 1 -- 172.21.15.134:0/1780070025 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2614003e00 con 0x55ac931350a0 2024-07-26T11:15:30.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.758+0000 7f2603400640 1 -- 172.21.15.134:0/1780070025 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2614012020 con 0x55ac931350a0 2024-07-26T11:15:30.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.758+0000 7f262f9ec740 10 client.25372 fetch_fsmap learned FSMap version 29 2024-07-26T11:15:30.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.758+0000 7f262f9ec740 1 -- 172.21.15.134:0/1780070025 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ac935299c0 con 0x55ac931350a0 2024-07-26T11:15:30.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.758+0000 7f2603400640 1 -- 172.21.15.134:0/1780070025 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26100aa9e0 con 0x55ac931350a0 2024-07-26T11:15:30.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.758+0000 7f2603400640 1 -- 172.21.15.134:0/1780070025 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f261402b410 con 0x55ac931350a0 2024-07-26T11:15:30.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.758+0000 7f262f9ec740 10 client.25372 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:15:30.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.759+0000 7f262f9ec740 10 client.25372 resolve_mds: resolved name 'i' to daemon mds.i 2024-07-26T11:15:30.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.759+0000 7f262f9ec740 20 client.25372 populate_metadata read hostname 'smithi134' 2024-07-26T11:15:30.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.759+0000 7f262f9ec740 1 --2- 172.21.15.134:0/1780070025 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55ac930f1c90 0x55ac93595b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:30.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.759+0000 7f262f9ec740 4 client.25372 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:15:30.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.759+0000 7f262f9ec740 1 -- 172.21.15.134:0/1780070025 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "status"}) -- 0x55ac93477360 con 0x55ac930f1c90 2024-07-26T11:15:30.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.759+0000 7f2621400640 1 --2- 172.21.15.134:0/1780070025 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55ac930f1c90 0x55ac93595b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:30.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.760+0000 7f2621400640 1 --2- 172.21.15.134:0/1780070025 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55ac930f1c90 0x55ac93595b60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=54a375359c3b9f61 server_cookie=fb88c98fb4b2c7c6 in_seq=0 out_seq=0 2024-07-26T11:15:30.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.760+0000 7f2603400640 10 client.25372 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:15:30.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.760+0000 7f2603400640 1 -- 172.21.15.134:0/1780070025 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ac93477360 con 0x55ac930f1c90 2024-07-26T11:15:30.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.760+0000 7f2603400640 10 client.25372 handle_command_reply: tid=0 2024-07-26T11:15:30.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.760+0000 7f262f9ec740 1 client.25372 shutdown 2024-07-26T11:15:30.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.760+0000 7f262f9ec740 1 -- 172.21.15.134:0/1780070025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 msgr2=0x55ac93522980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:30.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.760+0000 7f262f9ec740 1 --2- 172.21.15.134:0/1780070025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 0x55ac93522980 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x55ac935224d0 tx=0x7f26140046c0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.760+0000 7f262f9ec740 1 -- 172.21.15.134:0/1780070025 shutdown_connections 2024-07-26T11:15:30.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.760+0000 7f262f9ec740 1 --2- 172.21.15.134:0/1780070025 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55ac930f1c90 0x55ac93595b60 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.760+0000 7f262f9ec740 1 --2- 172.21.15.134:0/1780070025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ac9315df60 0x55ac93527f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.760+0000 7f262f9ec740 1 --2- 172.21.15.134:0/1780070025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ac9313d280 0x55ac93522f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.760+0000 7f262f9ec740 1 --2- 172.21.15.134:0/1780070025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ac931350a0 0x55ac93522980 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.760+0000 7f2603400640 0 client.25372 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:15:30.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.760+0000 7f262f9ec740 1 -- 172.21.15.134:0/1780070025 >> 172.21.15.134:0/1780070025 conn(0x55ac93152350 msgr2=0x55ac9350d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:30.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.761+0000 7f262f9ec740 1 -- 172.21.15.134:0/1780070025 shutdown_connections 2024-07-26T11:15:30.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.761+0000 7f262f9ec740 1 -- 172.21.15.134:0/1780070025 wait complete. 2024-07-26T11:15:30.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.761+0000 7f262f9ec740 20 client.25372 trim_cache size 0 max 16384 2024-07-26T11:15:30.766 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:15:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_fsid": "105fbe5c-4b3e-11ef-bcb7-c7b262605968", 2024-07-26T11:15:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: "whoami": 2, 2024-07-26T11:15:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: "id": 24703, 2024-07-26T11:15:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: "want_state": "up:active", 2024-07-26T11:15:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "up:active", 2024-07-26T11:15:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: "fs_name": "cephfs", 2024-07-26T11:15:30.766 INFO:teuthology.orchestra.run.smithi134.stdout: "rank_uptime": 222.72661588599999, 2024-07-26T11:15:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: "mdsmap_epoch": 29, 2024-07-26T11:15:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch": 90, 2024-07-26T11:15:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T11:15:30.767 INFO:teuthology.orchestra.run.smithi134.stdout: "uptime": 339.58725152400001 2024-07-26T11:15:30.767 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:15:30.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.761+0000 7f2623200640 1 -- 172.21.15.134:0/1241675457 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f26080b3c20 msgr2=0x7f26080b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:30.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.761+0000 7f2623200640 1 --2- 172.21.15.134:0/1241675457 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f26080b3c20 0x7f26080b6130 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x55ac93502400 tx=0x7f2614005f90 comp rx=0 tx=0).stop 2024-07-26T11:15:30.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.761+0000 7f2623200640 1 -- 172.21.15.134:0/1241675457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f262410af10 msgr2=0x7f262419c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:30.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.761+0000 7f2623200640 1 --2- 172.21.15.134:0/1241675457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f262410af10 0x7f262419c440 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f262406c270 tx=0x7f261000ee50 comp rx=0 tx=0).stop 2024-07-26T11:15:30.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.762+0000 7f2623200640 1 -- 172.21.15.134:0/1241675457 shutdown_connections 2024-07-26T11:15:30.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.762+0000 7f2623200640 1 --2- 172.21.15.134:0/1241675457 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f26080b3c20 0x7f26080b6130 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.762+0000 7f2623200640 1 --2- 172.21.15.134:0/1241675457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f262410b860 0x7f26241a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.762+0000 7f2623200640 1 --2- 172.21.15.134:0/1241675457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f262410af10 0x7f262419c440 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.762+0000 7f2623200640 1 --2- 172.21.15.134:0/1241675457 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f26241055d0 0x7f262419bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:30.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.762+0000 7f2623200640 1 -- 172.21.15.134:0/1241675457 >> 172.21.15.134:0/1241675457 conn(0x7f2624101040 msgr2=0x7f2624109540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:30.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.762+0000 7f2623200640 1 -- 172.21.15.134:0/1241675457 shutdown_connections 2024-07-26T11:15:30.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:30.762+0000 7f2623200640 1 -- 172.21.15.134:0/1241675457 wait complete. 2024-07-26T11:15:31.703 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.b status 2024-07-26T11:15:32.160 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:15:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:32 smithi134 ceph-mon[31966]: pgmap v467: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:32 smithi189 ceph-mon[36790]: pgmap v467: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:32 smithi193 ceph-mon[37473]: pgmap v467: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:15:34.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.224+0000 7f9f48a00640 1 -- 172.21.15.134:0/947385403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f4410af10 msgr2=0x7f9f4410b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:34.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.224+0000 7f9f48a00640 1 --2- 172.21.15.134:0/947385403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f4410af10 0x7f9f4410b320 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c009900 tx=0x7f9f2c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.224+0000 7f9f48a00640 1 -- 172.21.15.134:0/947385403 shutdown_connections 2024-07-26T11:15:34.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.224+0000 7f9f48a00640 1 --2- 172.21.15.134:0/947385403 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f4410b860 0x7f9f44112090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.224+0000 7f9f48a00640 1 --2- 172.21.15.134:0/947385403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f4410af10 0x7f9f4410b320 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.224+0000 7f9f48a00640 1 --2- 172.21.15.134:0/947385403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f441055d0 0x7f9f441059e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.224+0000 7f9f48a00640 1 -- 172.21.15.134:0/947385403 >> 172.21.15.134:0/947385403 conn(0x7f9f44101040 msgr2=0x7f9f44103490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:34.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.224+0000 7f9f48a00640 1 -- 172.21.15.134:0/947385403 shutdown_connections 2024-07-26T11:15:34.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.224+0000 7f9f48a00640 1 -- 172.21.15.134:0/947385403 wait complete. 2024-07-26T11:15:34.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.225+0000 7f9f48a00640 1 Processor -- start 2024-07-26T11:15:34.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.226+0000 7f9f48a00640 1 -- start start 2024-07-26T11:15:34.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.226+0000 7f9f48a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f441055d0 0x7f9f4419bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.226+0000 7f9f48a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f4410af10 0x7f9f4419c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.226+0000 7f9f48a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f4410b860 0x7f9f441a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.226+0000 7f9f48a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9f44115540 con 0x7f9f4410b860 2024-07-26T11:15:34.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.226+0000 7f9f48a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9f441153c0 con 0x7f9f4410af10 2024-07-26T11:15:34.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.226+0000 7f9f48a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9f441156c0 con 0x7f9f441055d0 2024-07-26T11:15:34.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.227+0000 7f9f3aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f4410af10 0x7f9f4419c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.227+0000 7f9f43400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f441055d0 0x7f9f4419bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.227+0000 7f9f43e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f4410b860 0x7f9f441a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.227+0000 7f9f3aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f4410af10 0x7f9f4419c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35860/0 (socket says 172.21.15.134:35860) 2024-07-26T11:15:34.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.227+0000 7f9f43400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f441055d0 0x7f9f4419bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34658/0 (socket says 172.21.15.134:34658) 2024-07-26T11:15:34.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.227+0000 7f9f3aa00640 1 -- 172.21.15.134:0/4077340857 learned_addr learned my addr 172.21.15.134:0/4077340857 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:34.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.227+0000 7f9f43400640 1 -- 172.21.15.134:0/4077340857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f4410af10 msgr2=0x7f9f4419c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:34.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.227+0000 7f9f43400640 1 --2- 172.21.15.134:0/4077340857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f4410af10 0x7f9f4419c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.227+0000 7f9f43400640 1 -- 172.21.15.134:0/4077340857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f4410b860 msgr2=0x7f9f441a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:34.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.227+0000 7f9f43400640 1 --2- 172.21.15.134:0/4077340857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f4410b860 0x7f9f441a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.227+0000 7f9f43400640 1 -- 172.21.15.134:0/4077340857 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f441a1e80 con 0x7f9f441055d0 2024-07-26T11:15:34.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.228+0000 7f9f43400640 1 --2- 172.21.15.134:0/4077340857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f441055d0 0x7f9f4419bf10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9f3400e9f0 tx=0x7f9f3400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:34.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.228+0000 7f9f40c00640 1 -- 172.21.15.134:0/4077340857 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3400ccc0 con 0x7f9f441055d0 2024-07-26T11:15:34.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.228+0000 7f9f40c00640 1 -- 172.21.15.134:0/4077340857 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f34005020 con 0x7f9f441055d0 2024-07-26T11:15:34.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.228+0000 7f9f48a00640 1 -- 172.21.15.134:0/4077340857 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f441a2170 con 0x7f9f441055d0 2024-07-26T11:15:34.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.228+0000 7f9f40c00640 1 -- 172.21.15.134:0/4077340857 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f34004a00 con 0x7f9f441055d0 2024-07-26T11:15:34.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.228+0000 7f9f48a00640 1 -- 172.21.15.134:0/4077340857 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f441a88d0 con 0x7f9f441055d0 2024-07-26T11:15:34.233 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.231+0000 7f9f50df5740 1 Processor -- start 2024-07-26T11:15:34.233 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.232+0000 7f9f50df5740 1 -- start start 2024-07-26T11:15:34.234 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.232+0000 7f9f40c00640 1 -- 172.21.15.134:0/4077340857 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f9f34004ba0 con 0x7f9f441055d0 2024-07-26T11:15:34.234 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.232+0000 7f9f50df5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4731f60 0x5647a4ad7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.232+0000 7f9f50df5740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a47090a0 0x5647a4ad7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.232+0000 7f9f50df5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a4711280 0x5647a4ad8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.232+0000 7f9f50df5740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5647a4a4d970 con 0x5647a47090a0 2024-07-26T11:15:34.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.232+0000 7f9f50df5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5647a4a3ce70 con 0x5647a4731f60 2024-07-26T11:15:34.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.232+0000 7f9f50df5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5647a4a40eb0 con 0x5647a4711280 2024-07-26T11:15:34.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.232+0000 7f9f43400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a47090a0 0x5647a4ad7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.232+0000 7f9f43e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4731f60 0x5647a4ad7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.232+0000 7f9f43400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a47090a0 0x5647a4ad7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36344/0 (socket says 172.21.15.134:36344) 2024-07-26T11:15:34.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.232+0000 7f9f43e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4731f60 0x5647a4ad7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35872/0 (socket says 172.21.15.134:35872) 2024-07-26T11:15:34.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.232+0000 7f9f43400640 1 -- 172.21.15.134:0/1566217361 learned_addr learned my addr 172.21.15.134:0/1566217361 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:34.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.233+0000 7f9f3aa00640 1 --2- 172.21.15.134:0/1566217361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a4711280 0x5647a4ad8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.233+0000 7f9f3aa00640 1 -- 172.21.15.134:0/1566217361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4731f60 msgr2=0x5647a4ad7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:34.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.233+0000 7f9f3aa00640 1 --2- 172.21.15.134:0/1566217361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4731f60 0x5647a4ad7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.233+0000 7f9f3aa00640 1 -- 172.21.15.134:0/1566217361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a47090a0 msgr2=0x5647a4ad7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:34.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.233+0000 7f9f3aa00640 1 --2- 172.21.15.134:0/1566217361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a47090a0 0x5647a4ad7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.233+0000 7f9f3aa00640 1 -- 172.21.15.134:0/1566217361 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647a49e4110 con 0x5647a4711280 2024-07-26T11:15:34.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.233+0000 7f9f3aa00640 1 --2- 172.21.15.134:0/1566217361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a4711280 0x5647a4ad8200 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c005ab0 tx=0x7f9f2c004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d74534f8241fe6a3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:34.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.234+0000 7f9f33e00640 1 -- 172.21.15.134:0/1566217361 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c040070 con 0x5647a4711280 2024-07-26T11:15:34.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.234+0000 7f9f33e00640 1 -- 172.21.15.134:0/1566217361 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f2c0054d0 con 0x5647a4711280 2024-07-26T11:15:34.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.234+0000 7f9f40c00640 1 --2- 172.21.15.134:0/4077340857 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9f280b3ca0 0x7f9f280b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.234+0000 7f9f33e00640 1 -- 172.21.15.134:0/1566217361 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c035040 con 0x5647a4711280 2024-07-26T11:15:34.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.234+0000 7f9f40c00640 1 -- 172.21.15.134:0/4077340857 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f9f340e4060 con 0x7f9f441055d0 2024-07-26T11:15:34.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.234+0000 7f9f43e00640 1 --2- 172.21.15.134:0/4077340857 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9f280b3ca0 0x7f9f280b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.235+0000 7f9f50df5740 1 -- 172.21.15.134:0/1566217361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a4711280 msgr2=0x5647a4ad8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:34.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.235+0000 7f9f50df5740 1 --2- 172.21.15.134:0/1566217361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a4711280 0x5647a4ad8200 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c005ab0 tx=0x7f9f2c004040 comp rx=0 tx=0).stop 2024-07-26T11:15:34.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.235+0000 7f9f43e00640 1 --2- 172.21.15.134:0/4077340857 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9f280b3ca0 0x7f9f280b61b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c005fd0 tx=0x7f9f3c009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:34.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.235+0000 7f9f50df5740 1 -- 172.21.15.134:0/1566217361 shutdown_connections 2024-07-26T11:15:34.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.235+0000 7f9f50df5740 1 --2- 172.21.15.134:0/1566217361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a4711280 0x5647a4ad8200 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.235+0000 7f9f50df5740 1 --2- 172.21.15.134:0/1566217361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a47090a0 0x5647a4ad7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.235+0000 7f9f50df5740 1 --2- 172.21.15.134:0/1566217361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4731f60 0x5647a4ad7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.235+0000 7f9f50df5740 1 -- 172.21.15.134:0/1566217361 >> 172.21.15.134:0/1566217361 conn(0x5647a4726350 msgr2=0x5647a4ad5670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:34.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.235+0000 7f9f50df5740 1 -- 172.21.15.134:0/1566217361 shutdown_connections 2024-07-26T11:15:34.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.235+0000 7f9f50df5740 1 -- 172.21.15.134:0/1566217361 wait complete. 2024-07-26T11:15:34.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.236+0000 7f9f50df5740 1 -- start start 2024-07-26T11:15:34.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.236+0000 7f9f50df5740 1 Processor -- start 2024-07-26T11:15:34.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.237+0000 7f9f50df5740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a47090a0 0x5647a4ae32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.237+0000 7f9f50df5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4711280 0x5647a4ae37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.237+0000 7f9f50df5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a4731f60 0x5647a4aef1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.237+0000 7f9f50df5740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5647a48a05f0 con 0x5647a47090a0 2024-07-26T11:15:34.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.237+0000 7f9f50df5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5647a49d9540 con 0x5647a4711280 2024-07-26T11:15:34.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.237+0000 7f9f50df5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5647a4ae5f00 con 0x5647a4731f60 2024-07-26T11:15:34.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.237+0000 7f9f50df5740 10 client.0 _ll_register_callbacks cb 0x5647a49e8f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:15:34.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.237+0000 7f9f43400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a47090a0 0x5647a4ae32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.237+0000 7f9f3aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4711280 0x5647a4ae37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.237+0000 7f9f3aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4711280 0x5647a4ae37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35876/0 (socket says 172.21.15.134:35876) 2024-07-26T11:15:34.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.237+0000 7f9f3aa00640 1 -- 172.21.15.134:0/278757222 learned_addr learned my addr 172.21.15.134:0/278757222 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:34.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.237+0000 7f9f43400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a47090a0 0x5647a4ae32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36346/0 (socket says 172.21.15.134:36346) 2024-07-26T11:15:34.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.237+0000 7f9f43e00640 1 --2- 172.21.15.134:0/278757222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a4731f60 0x5647a4aef1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.238+0000 7f9f43400640 1 -- 172.21.15.134:0/278757222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a4731f60 msgr2=0x5647a4aef1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:34.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.238+0000 7f9f43400640 1 --2- 172.21.15.134:0/278757222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a4731f60 0x5647a4aef1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.238+0000 7f9f43400640 1 -- 172.21.15.134:0/278757222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4711280 msgr2=0x5647a4ae37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:34.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.238+0000 7f9f43400640 1 --2- 172.21.15.134:0/278757222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4711280 0x5647a4ae37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.238+0000 7f9f43400640 1 -- 172.21.15.134:0/278757222 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647a479d540 con 0x5647a47090a0 2024-07-26T11:15:34.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.238+0000 7f9f43400640 1 --2- 172.21.15.134:0/278757222 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a47090a0 0x5647a4ae32a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9f340a6b30 tx=0x7f9f34009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=892f088938f8eadc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:34.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.238+0000 7f9f30c00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:15:34.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.238+0000 7f9f30c00640 1 -- 172.21.15.134:0/278757222 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3401c070 con 0x5647a47090a0 2024-07-26T11:15:34.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.238+0000 7f9f30c00640 1 -- 172.21.15.134:0/278757222 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f9f340a70a0 con 0x5647a47090a0 2024-07-26T11:15:34.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.239+0000 7f9f50df5740 1 -- 172.21.15.134:0/278757222 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f2c0054d0 con 0x5647a47090a0 2024-07-26T11:15:34.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.239+0000 7f9f30c00640 1 -- 172.21.15.134:0/278757222 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9f340aa220 con 0x5647a47090a0 2024-07-26T11:15:34.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.239+0000 7f9f50df5740 10 client.15228 fetch_fsmap learned FSMap version 29 2024-07-26T11:15:34.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.239+0000 7f9f50df5740 1 -- 172.21.15.134:0/278757222 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5647a4af1e20 con 0x5647a47090a0 2024-07-26T11:15:34.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.239+0000 7f9f30c00640 1 -- 172.21.15.134:0/278757222 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f2c0054d0 con 0x5647a47090a0 2024-07-26T11:15:34.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.239+0000 7f9f30c00640 1 -- 172.21.15.134:0/278757222 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9f340e60d0 con 0x5647a47090a0 2024-07-26T11:15:34.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.239+0000 7f9f50df5740 10 client.15228 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:15:34.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.239+0000 7f9f50df5740 10 client.15228 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T11:15:34.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.239+0000 7f9f50df5740 20 client.15228 populate_metadata read hostname 'smithi134' 2024-07-26T11:15:34.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.240+0000 7f9f50df5740 1 --2- 172.21.15.134:0/278757222 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5647a46f08e0 0x5647a4af5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.240+0000 7f9f50df5740 4 client.15228 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:15:34.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.240+0000 7f9f50df5740 1 -- 172.21.15.134:0/278757222 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5647a4a4f8c0 con 0x5647a46f08e0 2024-07-26T11:15:34.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.240+0000 7f9f3aa00640 1 --2- 172.21.15.134:0/278757222 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5647a46f08e0 0x5647a4af5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.241+0000 7f9f30c00640 10 client.15228 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:15:34.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.241+0000 7f9f3aa00640 1 --2- 172.21.15.134:0/278757222 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5647a46f08e0 0x5647a4af5830 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=276eaf3e09ac0dab server_cookie=e143ffaee983fa6 in_seq=0 out_seq=0 2024-07-26T11:15:34.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.242+0000 7f9f30c00640 1 -- 172.21.15.134:0/278757222 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5647a4a4f8c0 con 0x5647a46f08e0 2024-07-26T11:15:34.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.242+0000 7f9f30c00640 10 client.15228 handle_command_reply: tid=0 2024-07-26T11:15:34.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.242+0000 7f9f50df5740 1 client.15228 shutdown 2024-07-26T11:15:34.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.242+0000 7f9f50df5740 1 -- 172.21.15.134:0/278757222 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a47090a0 msgr2=0x5647a4ae32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:34.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.242+0000 7f9f50df5740 1 --2- 172.21.15.134:0/278757222 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a47090a0 0x5647a4ae32a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9f340a6b30 tx=0x7f9f34009f60 comp rx=0 tx=0).stop 2024-07-26T11:15:34.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.242+0000 7f9f50df5740 1 -- 172.21.15.134:0/278757222 shutdown_connections 2024-07-26T11:15:34.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.242+0000 7f9f50df5740 1 --2- 172.21.15.134:0/278757222 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5647a46f08e0 0x5647a4af5830 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.242+0000 7f9f50df5740 1 --2- 172.21.15.134:0/278757222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a4731f60 0x5647a4aef1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.242+0000 7f9f50df5740 1 --2- 172.21.15.134:0/278757222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4711280 0x5647a4ae37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.242+0000 7f9f50df5740 1 --2- 172.21.15.134:0/278757222 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a47090a0 0x5647a4ae32a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.242+0000 7f9f50df5740 1 -- 172.21.15.134:0/278757222 >> 172.21.15.134:0/278757222 conn(0x5647a4726350 msgr2=0x5647a4ada960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:34.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.242+0000 7f9f30c00640 0 client.15228 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:15:34.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.242+0000 7f9f50df5740 1 -- 172.21.15.134:0/278757222 shutdown_connections 2024-07-26T11:15:34.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.242+0000 7f9f50df5740 1 -- 172.21.15.134:0/278757222 wait complete. 2024-07-26T11:15:34.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.243+0000 7f9f50df5740 20 client.15228 trim_cache size 0 max 16384 2024-07-26T11:15:34.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.262+0000 7f9f50df5740 1 Processor -- start 2024-07-26T11:15:34.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.262+0000 7f9f50df5740 1 -- start start 2024-07-26T11:15:34.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.262+0000 7f9f50df5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a47090a0 0x5647a4add8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.262+0000 7f9f50df5740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a4711280 0x5647a4aef160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.262+0000 7f9f50df5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a4731f60 0x5647a4aef6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.262+0000 7f9f50df5740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5647a4af3b70 con 0x5647a4711280 2024-07-26T11:15:34.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.262+0000 7f9f50df5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5647a4af4240 con 0x5647a47090a0 2024-07-26T11:15:34.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.262+0000 7f9f50df5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5647a4af4910 con 0x5647a4731f60 2024-07-26T11:15:34.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.263+0000 7f9f3aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a4711280 0x5647a4aef160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.263+0000 7f9f43400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a47090a0 0x5647a4add8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.263+0000 7f9f3aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a4711280 0x5647a4aef160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36350/0 (socket says 172.21.15.134:36350) 2024-07-26T11:15:34.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.263+0000 7f9f43400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a47090a0 0x5647a4add8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35888/0 (socket says 172.21.15.134:35888) 2024-07-26T11:15:34.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.263+0000 7f9f3aa00640 1 -- 172.21.15.134:0/1691996110 learned_addr learned my addr 172.21.15.134:0/1691996110 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:34.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.263+0000 7f9f43e00640 1 --2- 172.21.15.134:0/1691996110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a4731f60 0x5647a4aef6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.263+0000 7f9f43400640 1 -- 172.21.15.134:0/1691996110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a4731f60 msgr2=0x5647a4aef6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:34.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.263+0000 7f9f43400640 1 --2- 172.21.15.134:0/1691996110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a4731f60 0x5647a4aef6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.263+0000 7f9f43400640 1 -- 172.21.15.134:0/1691996110 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a4711280 msgr2=0x5647a4aef160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:34.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.263+0000 7f9f43400640 1 --2- 172.21.15.134:0/1691996110 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a4711280 0x5647a4aef160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.263+0000 7f9f43400640 1 -- 172.21.15.134:0/1691996110 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5647a4ad07f0 con 0x5647a47090a0 2024-07-26T11:15:34.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.263+0000 7f9f43400640 1 --2- 172.21.15.134:0/1691996110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a47090a0 0x5647a4add8a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9f340aa7b0 tx=0x7f9f340e16e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=295af5507fd3c87d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:34.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.264+0000 7f9f33400640 1 -- 172.21.15.134:0/1691996110 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3401c070 con 0x5647a47090a0 2024-07-26T11:15:34.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.264+0000 7f9f33400640 1 -- 172.21.15.134:0/1691996110 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f340c53d0 con 0x5647a47090a0 2024-07-26T11:15:34.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.264+0000 7f9f33400640 1 -- 172.21.15.134:0/1691996110 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f340a6240 con 0x5647a47090a0 2024-07-26T11:15:34.267 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:34 smithi134 ceph-mon[31966]: pgmap v468: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:15:34.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.265+0000 7f9f50df5740 1 -- 172.21.15.134:0/1691996110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a47090a0 msgr2=0x5647a4add8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:34.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.265+0000 7f9f50df5740 1 --2- 172.21.15.134:0/1691996110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a47090a0 0x5647a4add8a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9f340aa7b0 tx=0x7f9f340e16e0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.265+0000 7f9f50df5740 1 -- 172.21.15.134:0/1691996110 shutdown_connections 2024-07-26T11:15:34.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.265+0000 7f9f50df5740 1 --2- 172.21.15.134:0/1691996110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a4731f60 0x5647a4aef6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.265+0000 7f9f50df5740 1 --2- 172.21.15.134:0/1691996110 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a4711280 0x5647a4aef160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.265+0000 7f9f50df5740 1 --2- 172.21.15.134:0/1691996110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a47090a0 0x5647a4add8a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.265+0000 7f9f50df5740 1 -- 172.21.15.134:0/1691996110 >> 172.21.15.134:0/1691996110 conn(0x5647a4726350 msgr2=0x5647a4ae1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:34.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.265+0000 7f9f50df5740 1 -- 172.21.15.134:0/1691996110 shutdown_connections 2024-07-26T11:15:34.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.265+0000 7f9f50df5740 1 -- 172.21.15.134:0/1691996110 wait complete. 2024-07-26T11:15:34.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.266+0000 7f9f50df5740 1 -- start start 2024-07-26T11:15:34.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.266+0000 7f9f50df5740 1 Processor -- start 2024-07-26T11:15:34.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f50df5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a47090a0 0x5647a4af71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f50df5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4711280 0x5647a4af7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f50df5740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a4731f60 0x5647a4afc770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f50df5740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5647a4af56b0 con 0x5647a4731f60 2024-07-26T11:15:34.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f50df5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5647a4af4fe0 con 0x5647a4711280 2024-07-26T11:15:34.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f50df5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5647a4af5d80 con 0x5647a47090a0 2024-07-26T11:15:34.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f50df5740 10 client.0 _ll_register_callbacks cb 0x7f9f340e15a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:15:34.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f43400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a47090a0 0x5647a4af71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f43400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a47090a0 0x5647a4af71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34702/0 (socket says 172.21.15.134:34702) 2024-07-26T11:15:34.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f43400640 1 -- 172.21.15.134:0/629727389 learned_addr learned my addr 172.21.15.134:0/629727389 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:34.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f3aa00640 1 --2- 172.21.15.134:0/629727389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4711280 0x5647a4af7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f43e00640 1 --2- 172.21.15.134:0/629727389 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a4731f60 0x5647a4afc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f43400640 1 -- 172.21.15.134:0/629727389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4711280 msgr2=0x5647a4af7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:34.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f43400640 1 --2- 172.21.15.134:0/629727389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4711280 0x5647a4af7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f43400640 1 -- 172.21.15.134:0/629727389 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a4731f60 msgr2=0x5647a4afc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f43400640 1 --2- 172.21.15.134:0/629727389 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a4731f60 0x5647a4afc770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f43400640 1 -- 172.21.15.134:0/629727389 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5647a4afcde0 con 0x5647a47090a0 2024-07-26T11:15:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.267+0000 7f9f3aa00640 1 --2- 172.21.15.134:0/629727389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4711280 0x5647a4af7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.268+0000 7f9f43400640 1 --2- 172.21.15.134:0/629727389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a47090a0 0x5647a4af71c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9f340045e0 tx=0x7f9f3400c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69d9e12217312ea6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.268+0000 7f9f30c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:15:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.268+0000 7f9f30c00640 1 -- 172.21.15.134:0/629727389 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f3401c070 con 0x5647a47090a0 2024-07-26T11:15:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.268+0000 7f9f50df5740 1 -- 172.21.15.134:0/629727389 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5647a4af39c0 con 0x5647a47090a0 2024-07-26T11:15:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.268+0000 7f9f30c00640 1 -- 172.21.15.134:0/629727389 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f9f340b36d0 con 0x5647a47090a0 2024-07-26T11:15:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.268+0000 7f9f30c00640 1 -- 172.21.15.134:0/629727389 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9f340ab1b0 con 0x5647a47090a0 2024-07-26T11:15:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.268+0000 7f9f50df5740 10 client.25390 fetch_fsmap learned FSMap version 29 2024-07-26T11:15:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.268+0000 7f9f50df5740 1 -- 172.21.15.134:0/629727389 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5647a4afe1a0 con 0x5647a47090a0 2024-07-26T11:15:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.269+0000 7f9f30c00640 1 -- 172.21.15.134:0/629727389 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5647a4af39c0 con 0x5647a47090a0 2024-07-26T11:15:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.269+0000 7f9f30c00640 1 -- 172.21.15.134:0/629727389 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f9f340a9d10 con 0x5647a47090a0 2024-07-26T11:15:34.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.269+0000 7f9f50df5740 10 client.25390 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:15:34.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.269+0000 7f9f50df5740 10 client.25390 resolve_mds: resolved name 'b' to daemon mds.b 2024-07-26T11:15:34.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.269+0000 7f9f50df5740 20 client.25390 populate_metadata read hostname 'smithi134' 2024-07-26T11:15:34.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.269+0000 7f9f50df5740 1 --2- 172.21.15.134:0/629727389 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5647a46c5c90 0x5647a4b61750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:34.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.269+0000 7f9f50df5740 4 client.25390 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:15:34.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.269+0000 7f9f50df5740 1 -- 172.21.15.134:0/629727389 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "status"}) -- 0x5647a4a4b360 con 0x5647a46c5c90 2024-07-26T11:15:34.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.270+0000 7f9f3aa00640 1 --2- 172.21.15.134:0/629727389 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5647a46c5c90 0x5647a4b61750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:34.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.270+0000 7f9f3aa00640 1 --2- 172.21.15.134:0/629727389 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5647a46c5c90 0x5647a4b61750 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1bab6eb0b93645b server_cookie=61931a1f7ae557f0 in_seq=0 out_seq=0 2024-07-26T11:15:34.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.270+0000 7f9f30c00640 10 client.25390 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:15:34.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.270+0000 7f9f30c00640 1 -- 172.21.15.134:0/629727389 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5647a4a4b360 con 0x5647a46c5c90 2024-07-26T11:15:34.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.270+0000 7f9f30c00640 10 client.25390 handle_command_reply: tid=0 2024-07-26T11:15:34.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.270+0000 7f9f50df5740 1 client.25390 shutdown 2024-07-26T11:15:34.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.271+0000 7f9f50df5740 1 -- 172.21.15.134:0/629727389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a47090a0 msgr2=0x5647a4af71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:34.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.271+0000 7f9f50df5740 1 --2- 172.21.15.134:0/629727389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a47090a0 0x5647a4af71c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9f340045e0 tx=0x7f9f3400c840 comp rx=0 tx=0).stop 2024-07-26T11:15:34.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.271+0000 7f9f50df5740 1 -- 172.21.15.134:0/629727389 shutdown_connections 2024-07-26T11:15:34.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.271+0000 7f9f50df5740 1 --2- 172.21.15.134:0/629727389 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5647a46c5c90 0x5647a4b61750 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.271+0000 7f9f50df5740 1 --2- 172.21.15.134:0/629727389 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5647a4731f60 0x5647a4afc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.271+0000 7f9f50df5740 1 --2- 172.21.15.134:0/629727389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5647a4711280 0x5647a4af7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.271+0000 7f9f50df5740 1 --2- 172.21.15.134:0/629727389 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5647a47090a0 0x5647a4af71c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.271+0000 7f9f50df5740 1 -- 172.21.15.134:0/629727389 >> 172.21.15.134:0/629727389 conn(0x5647a4726350 msgr2=0x5647a4ad86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:34.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.271+0000 7f9f30c00640 0 client.25390 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:15:34.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.271+0000 7f9f50df5740 1 -- 172.21.15.134:0/629727389 shutdown_connections 2024-07-26T11:15:34.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.271+0000 7f9f50df5740 1 -- 172.21.15.134:0/629727389 wait complete. 2024-07-26T11:15:34.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.271+0000 7f9f50df5740 20 client.25390 trim_cache size 0 max 16384 2024-07-26T11:15:34.277 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:15:34.277 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_fsid": "105fbe5c-4b3e-11ef-bcb7-c7b262605968", 2024-07-26T11:15:34.277 INFO:teuthology.orchestra.run.smithi134.stdout: "whoami": 1, 2024-07-26T11:15:34.277 INFO:teuthology.orchestra.run.smithi134.stdout: "id": 24719, 2024-07-26T11:15:34.277 INFO:teuthology.orchestra.run.smithi134.stdout: "want_state": "up:active", 2024-07-26T11:15:34.277 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "up:active", 2024-07-26T11:15:34.277 INFO:teuthology.orchestra.run.smithi134.stdout: "fs_name": "cephfs", 2024-07-26T11:15:34.277 INFO:teuthology.orchestra.run.smithi134.stdout: "rank_uptime": 227.245557808, 2024-07-26T11:15:34.277 INFO:teuthology.orchestra.run.smithi134.stdout: "mdsmap_epoch": 29, 2024-07-26T11:15:34.278 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch": 90, 2024-07-26T11:15:34.278 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T11:15:34.278 INFO:teuthology.orchestra.run.smithi134.stdout: "uptime": 331.59344295900001 2024-07-26T11:15:34.278 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:15:34.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.272+0000 7f9f48a00640 1 -- 172.21.15.134:0/4077340857 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9f280b3ca0 msgr2=0x7f9f280b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:34.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.272+0000 7f9f48a00640 1 --2- 172.21.15.134:0/4077340857 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9f280b3ca0 0x7f9f280b61b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c005fd0 tx=0x7f9f3c009500 comp rx=0 tx=0).stop 2024-07-26T11:15:34.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.272+0000 7f9f48a00640 1 -- 172.21.15.134:0/4077340857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f441055d0 msgr2=0x7f9f4419bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:34.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.272+0000 7f9f48a00640 1 --2- 172.21.15.134:0/4077340857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f441055d0 0x7f9f4419bf10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9f3400e9f0 tx=0x7f9f3400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.272+0000 7f9f48a00640 1 -- 172.21.15.134:0/4077340857 shutdown_connections 2024-07-26T11:15:34.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.272+0000 7f9f48a00640 1 --2- 172.21.15.134:0/4077340857 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9f280b3ca0 0x7f9f280b61b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.272+0000 7f9f48a00640 1 --2- 172.21.15.134:0/4077340857 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9f4410b860 0x7f9f441a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.272+0000 7f9f48a00640 1 --2- 172.21.15.134:0/4077340857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f4410af10 0x7f9f4419c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.272+0000 7f9f48a00640 1 --2- 172.21.15.134:0/4077340857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9f441055d0 0x7f9f4419bf10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:34.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.272+0000 7f9f48a00640 1 -- 172.21.15.134:0/4077340857 >> 172.21.15.134:0/4077340857 conn(0x7f9f44101040 msgr2=0x7f9f44103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:34.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.273+0000 7f9f48a00640 1 -- 172.21.15.134:0/4077340857 shutdown_connections 2024-07-26T11:15:34.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:34.273+0000 7f9f48a00640 1 -- 172.21.15.134:0/4077340857 wait complete. 2024-07-26T11:15:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:34 smithi189 ceph-mon[36790]: pgmap v468: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:15:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:34 smithi193 ceph-mon[37473]: pgmap v468: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:15:35.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.a status 2024-07-26T11:15:35.623 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:15:36.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:36 smithi134 ceph-mon[31966]: pgmap v469: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:36 smithi189 ceph-mon[36790]: pgmap v469: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:36 smithi193 ceph-mon[37473]: pgmap v469: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:15:37.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.775+0000 7f5fff200640 1 -- 172.21.15.134:0/3273395896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6000108d20 msgr2=0x7f6000109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:37.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.775+0000 7f5fff200640 1 --2- 172.21.15.134:0/3273395896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6000108d20 0x7f6000109130 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5fec009860 tx=0x7f5fec02a340 comp rx=0 tx=0).stop 2024-07-26T11:15:37.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.775+0000 7f5fff200640 1 -- 172.21.15.134:0/3273395896 shutdown_connections 2024-07-26T11:15:37.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.775+0000 7f5fff200640 1 --2- 172.21.15.134:0/3273395896 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6000109670 0x7f600010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.775+0000 7f5fff200640 1 --2- 172.21.15.134:0/3273395896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6000108d20 0x7f6000109130 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.775+0000 7f5fff200640 1 --2- 172.21.15.134:0/3273395896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60001033e0 0x7f60001037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.775+0000 7f5fff200640 1 -- 172.21.15.134:0/3273395896 >> 172.21.15.134:0/3273395896 conn(0x7f60000fee30 msgr2=0x7f6000101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:37.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.775+0000 7f5fff200640 1 -- 172.21.15.134:0/3273395896 shutdown_connections 2024-07-26T11:15:37.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.775+0000 7f5fff200640 1 -- 172.21.15.134:0/3273395896 wait complete. 2024-07-26T11:15:37.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.776+0000 7f5fff200640 1 Processor -- start 2024-07-26T11:15:37.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.777+0000 7f5fff200640 1 -- start start 2024-07-26T11:15:37.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.777+0000 7f5fff200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60001033e0 0x7f6000199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.777+0000 7f5fff200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6000108d20 0x7f600019a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.777+0000 7f5fff200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6000109670 0x7f600019f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.777+0000 7f5fff200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f60001132e0 con 0x7f6000108d20 2024-07-26T11:15:37.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.777+0000 7f5fff200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6000113160 con 0x7f6000109670 2024-07-26T11:15:37.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.777+0000 7f5fff200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6000113460 con 0x7f60001033e0 2024-07-26T11:15:37.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.777+0000 7f5ffd400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6000108d20 0x7f600019a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.778+0000 7f5ffd400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6000108d20 0x7f600019a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36380/0 (socket says 172.21.15.134:36380) 2024-07-26T11:15:37.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.778+0000 7f5ffd400640 1 -- 172.21.15.134:0/290656649 learned_addr learned my addr 172.21.15.134:0/290656649 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:37.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.778+0000 7f5ffde00640 1 --2- 172.21.15.134:0/290656649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60001033e0 0x7f6000199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.778+0000 7f5ffe800640 1 --2- 172.21.15.134:0/290656649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6000109670 0x7f600019f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.778+0000 7f5ffd400640 1 -- 172.21.15.134:0/290656649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60001033e0 msgr2=0x7f6000199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:37.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.778+0000 7f5ffd400640 1 --2- 172.21.15.134:0/290656649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60001033e0 0x7f6000199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.778+0000 7f5ffd400640 1 -- 172.21.15.134:0/290656649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6000109670 msgr2=0x7f600019f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:37.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.778+0000 7f5ffd400640 1 --2- 172.21.15.134:0/290656649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6000109670 0x7f600019f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.778+0000 7f5ffd400640 1 -- 172.21.15.134:0/290656649 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600019fd20 con 0x7f6000108d20 2024-07-26T11:15:37.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.778+0000 7f5ffde00640 1 --2- 172.21.15.134:0/290656649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60001033e0 0x7f6000199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:37.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.779+0000 7f5ffd400640 1 --2- 172.21.15.134:0/290656649 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6000108d20 0x7f600019a350 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5fec035040 tx=0x7f5fec032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:37.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.779+0000 7f5ff6a00640 1 -- 172.21.15.134:0/290656649 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fec040020 con 0x7f6000108d20 2024-07-26T11:15:37.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.779+0000 7f5fff200640 1 -- 172.21.15.134:0/290656649 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f600019ffb0 con 0x7f6000108d20 2024-07-26T11:15:37.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.779+0000 7f5ff6a00640 1 -- 172.21.15.134:0/290656649 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5fec030d50 con 0x7f6000108d20 2024-07-26T11:15:37.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.779+0000 7f5fff200640 1 -- 172.21.15.134:0/290656649 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60001a6770 con 0x7f6000108d20 2024-07-26T11:15:37.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.779+0000 7f5ff6a00640 1 -- 172.21.15.134:0/290656649 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fec037040 con 0x7f6000108d20 2024-07-26T11:15:37.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.783+0000 7f600bffa740 1 Processor -- start 2024-07-26T11:15:37.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.783+0000 7f600bffa740 1 -- start start 2024-07-26T11:15:37.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f600bffa740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba6df60 0x563c4be13780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f600bffa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba450a0 0x563c4be13cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f600bffa740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba4d280 0x563c4be14200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f600bffa740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563c4bd89970 con 0x563c4ba4d280 2024-07-26T11:15:37.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f600bffa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563c4bd78e70 con 0x563c4ba450a0 2024-07-26T11:15:37.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f600bffa740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563c4bd7ceb0 con 0x563c4ba6df60 2024-07-26T11:15:37.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f5ff6a00640 1 -- 172.21.15.134:0/290656649 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f5fec043020 con 0x7f6000108d20 2024-07-26T11:15:37.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f5ffde00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba450a0 0x563c4be13cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f5ffde00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba450a0 0x563c4be13cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35924/0 (socket says 172.21.15.134:35924) 2024-07-26T11:15:37.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f5ffde00640 1 -- 172.21.15.134:0/1401579262 learned_addr learned my addr 172.21.15.134:0/1401579262 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:37.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f5ffe800640 1 --2- 172.21.15.134:0/1401579262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba6df60 0x563c4be13780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f5ffd400640 1 --2- 172.21.15.134:0/1401579262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba4d280 0x563c4be14200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f5ffde00640 1 -- 172.21.15.134:0/1401579262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba6df60 msgr2=0x563c4be13780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:37.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f5ffde00640 1 --2- 172.21.15.134:0/1401579262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba6df60 0x563c4be13780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f5ffde00640 1 -- 172.21.15.134:0/1401579262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba4d280 msgr2=0x563c4be14200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:37.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f5ffde00640 1 --2- 172.21.15.134:0/1401579262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba4d280 0x563c4be14200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f5ffde00640 1 -- 172.21.15.134:0/1401579262 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c4bd20110 con 0x563c4ba450a0 2024-07-26T11:15:37.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.784+0000 7f5ffd400640 1 --2- 172.21.15.134:0/1401579262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba4d280 0x563c4be14200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:15:37.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.785+0000 7f5ffde00640 1 --2- 172.21.15.134:0/1401579262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba450a0 0x563c4be13cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0011860 tx=0x7f5ff0011d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab535bb8ae0aa9c2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:37.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.785+0000 7f5feb400640 1 -- 172.21.15.134:0/1401579262 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff00112f0 con 0x563c4ba450a0 2024-07-26T11:15:37.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.785+0000 7f5feb400640 1 -- 172.21.15.134:0/1401579262 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ff000f9a0 con 0x563c4ba450a0 2024-07-26T11:15:37.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.785+0000 7f5feb400640 1 -- 172.21.15.134:0/1401579262 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff00056d0 con 0x563c4ba450a0 2024-07-26T11:15:37.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.786+0000 7f5ff6a00640 1 --2- 172.21.15.134:0/290656649 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5fe40b3c20 0x7f5fe40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.786+0000 7f5ff6a00640 1 -- 172.21.15.134:0/290656649 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5fec103d50 con 0x7f6000108d20 2024-07-26T11:15:37.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.786+0000 7f5ffe800640 1 --2- 172.21.15.134:0/290656649 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5fe40b3c20 0x7f5fe40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.786+0000 7f600bffa740 1 -- 172.21.15.134:0/1401579262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba450a0 msgr2=0x563c4be13cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:37.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.786+0000 7f600bffa740 1 --2- 172.21.15.134:0/1401579262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba450a0 0x563c4be13cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0011860 tx=0x7f5ff0011d20 comp rx=0 tx=0).stop 2024-07-26T11:15:37.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.786+0000 7f600bffa740 1 -- 172.21.15.134:0/1401579262 shutdown_connections 2024-07-26T11:15:37.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.786+0000 7f600bffa740 1 --2- 172.21.15.134:0/1401579262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba4d280 0x563c4be14200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.786+0000 7f600bffa740 1 --2- 172.21.15.134:0/1401579262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba450a0 0x563c4be13cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.786+0000 7f600bffa740 1 --2- 172.21.15.134:0/1401579262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba6df60 0x563c4be13780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.786+0000 7f600bffa740 1 -- 172.21.15.134:0/1401579262 >> 172.21.15.134:0/1401579262 conn(0x563c4ba62350 msgr2=0x563c4be11670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:37.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.787+0000 7f600bffa740 1 -- 172.21.15.134:0/1401579262 shutdown_connections 2024-07-26T11:15:37.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.787+0000 7f600bffa740 1 -- 172.21.15.134:0/1401579262 wait complete. 2024-07-26T11:15:37.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.787+0000 7f5ffe800640 1 --2- 172.21.15.134:0/290656649 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5fe40b3c20 0x7f5fe40b6130 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8005fd0 tx=0x7f5ff8009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:37.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.787+0000 7f600bffa740 1 -- start start 2024-07-26T11:15:37.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.788+0000 7f600bffa740 1 Processor -- start 2024-07-26T11:15:37.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.788+0000 7f600bffa740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba450a0 0x563c4be233c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.788+0000 7f600bffa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba4d280 0x563c4be23960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.788+0000 7f600bffa740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba6df60 0x563c4be2f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.788+0000 7f600bffa740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563c4bbdc5f0 con 0x563c4ba6df60 2024-07-26T11:15:37.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.788+0000 7f600bffa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563c4bd15540 con 0x563c4ba4d280 2024-07-26T11:15:37.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.788+0000 7f600bffa740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563c4be26290 con 0x563c4ba450a0 2024-07-26T11:15:37.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.788+0000 7f600bffa740 10 client.0 _ll_register_callbacks cb 0x563c4bd24f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:15:37.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.789+0000 7f5ffde00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba450a0 0x563c4be233c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.789+0000 7f5ffd400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba4d280 0x563c4be23960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.789+0000 7f5ffde00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba450a0 0x563c4be233c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34728/0 (socket says 172.21.15.134:34728) 2024-07-26T11:15:37.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.789+0000 7f5ffde00640 1 -- 172.21.15.134:0/2930010492 learned_addr learned my addr 172.21.15.134:0/2930010492 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:37.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.789+0000 7f5ffe800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba6df60 0x563c4be2f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.789+0000 7f5ffde00640 1 -- 172.21.15.134:0/2930010492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba4d280 msgr2=0x563c4be23960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:37.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.789+0000 7f5ffde00640 1 --2- 172.21.15.134:0/2930010492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba4d280 0x563c4be23960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.789+0000 7f5ffde00640 1 -- 172.21.15.134:0/2930010492 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba6df60 msgr2=0x563c4be2f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:37.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.789+0000 7f5ffde00640 1 --2- 172.21.15.134:0/2930010492 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba6df60 0x563c4be2f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.789+0000 7f5ffde00640 1 -- 172.21.15.134:0/2930010492 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c4bad9540 con 0x563c4ba450a0 2024-07-26T11:15:37.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.789+0000 7f5ffde00640 1 --2- 172.21.15.134:0/2930010492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba450a0 0x563c4be233c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5ff000fd80 tx=0x7f5ff000fdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dbb7ac78805f46 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:37.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.789+0000 7f5fdfe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:15:37.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.790+0000 7f5fdfe00640 1 -- 172.21.15.134:0/2930010492 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0028030 con 0x563c4ba450a0 2024-07-26T11:15:37.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.790+0000 7f600bffa740 1 -- 172.21.15.134:0/2930010492 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ff000f9a0 con 0x563c4ba450a0 2024-07-26T11:15:37.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.790+0000 7f5fdfe00640 1 -- 172.21.15.134:0/2930010492 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5ff0005000 con 0x563c4ba450a0 2024-07-26T11:15:37.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.790+0000 7f5fdfe00640 1 -- 172.21.15.134:0/2930010492 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5ff0032050 con 0x563c4ba450a0 2024-07-26T11:15:37.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.790+0000 7f600bffa740 10 client.25402 fetch_fsmap learned FSMap version 29 2024-07-26T11:15:37.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.790+0000 7f600bffa740 1 -- 172.21.15.134:0/2930010492 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563c4be32150 con 0x563c4ba450a0 2024-07-26T11:15:37.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.790+0000 7f5fdfe00640 1 -- 172.21.15.134:0/2930010492 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ff000f9a0 con 0x563c4ba450a0 2024-07-26T11:15:37.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.791+0000 7f5fdfe00640 1 -- 172.21.15.134:0/2930010492 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5ff0018070 con 0x563c4ba450a0 2024-07-26T11:15:37.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.791+0000 7f600bffa740 10 client.25402 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:15:37.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.791+0000 7f600bffa740 10 client.25402 resolve_mds: resolved name 'a' to daemon mds.a 2024-07-26T11:15:37.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.791+0000 7f600bffa740 20 client.25402 populate_metadata read hostname 'smithi134' 2024-07-26T11:15:37.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.791+0000 7f600bffa740 1 --2- 172.21.15.134:0/2930010492 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563c4ba2c8e0 0x563c4be35a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.791+0000 7f600bffa740 4 client.25402 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:15:37.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.791+0000 7f600bffa740 1 -- 172.21.15.134:0/2930010492 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c4bd8b8c0 con 0x563c4ba2c8e0 2024-07-26T11:15:37.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.791+0000 7f5ffd400640 1 --2- 172.21.15.134:0/2930010492 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563c4ba2c8e0 0x563c4be35a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.792+0000 7f5ffd400640 1 --2- 172.21.15.134:0/2930010492 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563c4ba2c8e0 0x563c4be35a10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5fa9ae077b64a4 server_cookie=39943c377faa8ce7 in_seq=0 out_seq=0 2024-07-26T11:15:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.792+0000 7f5fdfe00640 10 client.25402 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:15:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.793+0000 7f5fdfe00640 1 -- 172.21.15.134:0/2930010492 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563c4bd8b8c0 con 0x563c4ba2c8e0 2024-07-26T11:15:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.793+0000 7f5fdfe00640 10 client.25402 handle_command_reply: tid=0 2024-07-26T11:15:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.793+0000 7f600bffa740 1 client.25402 shutdown 2024-07-26T11:15:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.793+0000 7f600bffa740 1 -- 172.21.15.134:0/2930010492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba450a0 msgr2=0x563c4be233c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.793+0000 7f600bffa740 1 --2- 172.21.15.134:0/2930010492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba450a0 0x563c4be233c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5ff000fd80 tx=0x7f5ff000fdb0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.793+0000 7f600bffa740 1 -- 172.21.15.134:0/2930010492 shutdown_connections 2024-07-26T11:15:37.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.793+0000 7f600bffa740 1 --2- 172.21.15.134:0/2930010492 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563c4ba2c8e0 0x563c4be35a10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.793+0000 7f600bffa740 1 --2- 172.21.15.134:0/2930010492 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba6df60 0x563c4be2f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.793+0000 7f600bffa740 1 --2- 172.21.15.134:0/2930010492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba4d280 0x563c4be23960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.793+0000 7f600bffa740 1 --2- 172.21.15.134:0/2930010492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba450a0 0x563c4be233c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.793+0000 7f600bffa740 1 -- 172.21.15.134:0/2930010492 >> 172.21.15.134:0/2930010492 conn(0x563c4ba62350 msgr2=0x563c4be168d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:37.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.793+0000 7f5fdfe00640 0 client.25402 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:15:37.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.793+0000 7f600bffa740 1 -- 172.21.15.134:0/2930010492 shutdown_connections 2024-07-26T11:15:37.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.793+0000 7f600bffa740 1 -- 172.21.15.134:0/2930010492 wait complete. 2024-07-26T11:15:37.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.794+0000 7f600bffa740 20 client.25402 trim_cache size 0 max 16384 2024-07-26T11:15:37.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.813+0000 7f600bffa740 1 Processor -- start 2024-07-26T11:15:37.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.813+0000 7f600bffa740 1 -- start start 2024-07-26T11:15:37.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.814+0000 7f600bffa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba450a0 0x563c4be35a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.814+0000 7f600bffa740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba4d280 0x563c4be33560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.814+0000 7f600bffa740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba6df60 0x563c4be2f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.814+0000 7f600bffa740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563c4be33d50 con 0x563c4ba4d280 2024-07-26T11:15:37.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.814+0000 7f600bffa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563c4be34420 con 0x563c4ba450a0 2024-07-26T11:15:37.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.814+0000 7f600bffa740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563c4be34af0 con 0x563c4ba6df60 2024-07-26T11:15:37.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.814+0000 7f5ffd400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba4d280 0x563c4be33560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.814+0000 7f5ffe800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba6df60 0x563c4be2f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.814+0000 7f5ffe800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba6df60 0x563c4be2f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34732/0 (socket says 172.21.15.134:34732) 2024-07-26T11:15:37.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.814+0000 7f5ffe800640 1 -- 172.21.15.134:0/1091797853 learned_addr learned my addr 172.21.15.134:0/1091797853 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:37.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.814+0000 7f5ffde00640 1 --2- 172.21.15.134:0/1091797853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba450a0 0x563c4be35a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.814+0000 7f5ffe800640 1 -- 172.21.15.134:0/1091797853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba450a0 msgr2=0x563c4be35a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:37.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.814+0000 7f5ffe800640 1 --2- 172.21.15.134:0/1091797853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba450a0 0x563c4be35a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.814+0000 7f5ffe800640 1 -- 172.21.15.134:0/1091797853 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba4d280 msgr2=0x563c4be33560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:37.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.814+0000 7f5ffe800640 1 --2- 172.21.15.134:0/1091797853 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba4d280 0x563c4be33560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.814+0000 7f5ffe800640 1 -- 172.21.15.134:0/1091797853 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c4be145b0 con 0x563c4ba6df60 2024-07-26T11:15:37.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.815+0000 7f5ffe800640 1 --2- 172.21.15.134:0/1091797853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba6df60 0x563c4be2f560 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8004440 tx=0x7f5ff8017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=117ba30cd22d1c58 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:37.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.815+0000 7f5feaa00640 1 -- 172.21.15.134:0/1091797853 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff8004470 con 0x563c4ba6df60 2024-07-26T11:15:37.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.815+0000 7f5feaa00640 1 -- 172.21.15.134:0/1091797853 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ff802d070 con 0x563c4ba6df60 2024-07-26T11:15:37.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.815+0000 7f5feaa00640 1 -- 172.21.15.134:0/1091797853 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff8028400 con 0x563c4ba6df60 2024-07-26T11:15:37.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.816+0000 7f600bffa740 1 -- 172.21.15.134:0/1091797853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba6df60 msgr2=0x563c4be2f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:37.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.816+0000 7f600bffa740 1 --2- 172.21.15.134:0/1091797853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba6df60 0x563c4be2f560 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8004440 tx=0x7f5ff8017000 comp rx=0 tx=0).stop 2024-07-26T11:15:37.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.816+0000 7f600bffa740 1 -- 172.21.15.134:0/1091797853 shutdown_connections 2024-07-26T11:15:37.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.816+0000 7f600bffa740 1 --2- 172.21.15.134:0/1091797853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba6df60 0x563c4be2f560 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.816+0000 7f600bffa740 1 --2- 172.21.15.134:0/1091797853 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba4d280 0x563c4be33560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.816+0000 7f600bffa740 1 --2- 172.21.15.134:0/1091797853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba450a0 0x563c4be35a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.816+0000 7f600bffa740 1 -- 172.21.15.134:0/1091797853 >> 172.21.15.134:0/1091797853 conn(0x563c4ba62350 msgr2=0x563c4be21c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:37.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.816+0000 7f600bffa740 1 -- 172.21.15.134:0/1091797853 shutdown_connections 2024-07-26T11:15:37.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.816+0000 7f600bffa740 1 -- 172.21.15.134:0/1091797853 wait complete. 2024-07-26T11:15:37.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.817+0000 7f600bffa740 1 -- start start 2024-07-26T11:15:37.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.817+0000 7f600bffa740 1 Processor -- start 2024-07-26T11:15:37.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.817+0000 7f600bffa740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba450a0 0x563c4be1d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.817+0000 7f600bffa740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba4d280 0x563c4be19ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.817+0000 7f600bffa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba6df60 0x563c4be19ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.817+0000 7f600bffa740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563c4be35890 con 0x563c4ba4d280 2024-07-26T11:15:37.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.817+0000 7f600bffa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563c4be351c0 con 0x563c4ba6df60 2024-07-26T11:15:37.821 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.817+0000 7f600bffa740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563c4be35f60 con 0x563c4ba450a0 2024-07-26T11:15:37.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.817+0000 7f600bffa740 10 client.0 _ll_register_callbacks cb 0x7f5ff00114c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:15:37.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.818+0000 7f5ffd400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba4d280 0x563c4be19ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.818+0000 7f5ffe800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba6df60 0x563c4be19ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.818+0000 7f5ffe800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba6df60 0x563c4be19ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35952/0 (socket says 172.21.15.134:35952) 2024-07-26T11:15:37.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.818+0000 7f5ffe800640 1 -- 172.21.15.134:0/2097447027 learned_addr learned my addr 172.21.15.134:0/2097447027 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:37.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.818+0000 7f5ffde00640 1 --2- 172.21.15.134:0/2097447027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba450a0 0x563c4be1d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.818+0000 7f5ffe800640 1 -- 172.21.15.134:0/2097447027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba450a0 msgr2=0x563c4be1d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:37.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.818+0000 7f5ffe800640 1 --2- 172.21.15.134:0/2097447027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba450a0 0x563c4be1d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.818+0000 7f5ffe800640 1 -- 172.21.15.134:0/2097447027 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba4d280 msgr2=0x563c4be19ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:15:37.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.818+0000 7f5ffe800640 1 --2- 172.21.15.134:0/2097447027 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba4d280 0x563c4be19ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.818+0000 7f5ffe800640 1 -- 172.21.15.134:0/2097447027 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c4be1a530 con 0x563c4ba6df60 2024-07-26T11:15:37.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.818+0000 7f5ffe800640 1 --2- 172.21.15.134:0/2097447027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba6df60 0x563c4be19ff0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8004410 tx=0x7f5ff801d7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4195dce2255ac933 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:37.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.819+0000 7f5fdfe00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:15:37.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.819+0000 7f5fdfe00640 1 -- 172.21.15.134:0/2097447027 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff801dd50 con 0x563c4ba6df60 2024-07-26T11:15:37.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.819+0000 7f600bffa740 1 -- 172.21.15.134:0/2097447027 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ff802d070 con 0x563c4ba6df60 2024-07-26T11:15:37.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.819+0000 7f5fdfe00640 1 -- 172.21.15.134:0/2097447027 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5ff8003af0 con 0x563c4ba6df60 2024-07-26T11:15:37.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.819+0000 7f5fdfe00640 1 -- 172.21.15.134:0/2097447027 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5ff8003cf0 con 0x563c4ba6df60 2024-07-26T11:15:37.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.819+0000 7f600bffa740 10 client.25394 fetch_fsmap learned FSMap version 29 2024-07-26T11:15:37.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.819+0000 7f600bffa740 1 -- 172.21.15.134:0/2097447027 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563c4be3d8e0 con 0x563c4ba6df60 2024-07-26T11:15:37.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.820+0000 7f5fdfe00640 1 -- 172.21.15.134:0/2097447027 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ff802d070 con 0x563c4ba6df60 2024-07-26T11:15:37.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.820+0000 7f5fdfe00640 1 -- 172.21.15.134:0/2097447027 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8370+0+0 (secure 0 0 0) 0x7f5ff8004a30 con 0x563c4ba6df60 2024-07-26T11:15:37.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.820+0000 7f600bffa740 10 client.25394 fetch_fsmap finished waiting for FSMap version 29 2024-07-26T11:15:37.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.820+0000 7f600bffa740 10 client.25394 resolve_mds: resolved name 'a' to daemon mds.a 2024-07-26T11:15:37.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.820+0000 7f600bffa740 20 client.25394 populate_metadata read hostname 'smithi134' 2024-07-26T11:15:37.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.820+0000 7f600bffa740 1 --2- 172.21.15.134:0/2097447027 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563c4ba01c90 0x563c4bea56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:37.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.820+0000 7f600bffa740 4 client.25394 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "status"}] 2024-07-26T11:15:37.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.820+0000 7f600bffa740 1 -- 172.21.15.134:0/2097447027 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "status"}) -- 0x563c4bd87360 con 0x563c4ba01c90 2024-07-26T11:15:37.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.820+0000 7f5ffde00640 1 --2- 172.21.15.134:0/2097447027 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563c4ba01c90 0x563c4bea56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:37.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.821+0000 7f5fdfe00640 10 client.25394 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:15:37.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.821+0000 7f5ffde00640 1 --2- 172.21.15.134:0/2097447027 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563c4ba01c90 0x563c4bea56f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d8ceab1a6d75a9b server_cookie=cf024eb784f60c5f in_seq=0 out_seq=0 2024-07-26T11:15:37.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.822+0000 7f5fdfe00640 1 -- 172.21.15.134:0/2097447027 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x563c4bd87360 con 0x563c4ba01c90 2024-07-26T11:15:37.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.822+0000 7f5fdfe00640 10 client.25394 handle_command_reply: tid=0 2024-07-26T11:15:37.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.822+0000 7f600bffa740 1 client.25394 shutdown 2024-07-26T11:15:37.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.822+0000 7f600bffa740 1 -- 172.21.15.134:0/2097447027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba6df60 msgr2=0x563c4be19ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:37.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.822+0000 7f600bffa740 1 --2- 172.21.15.134:0/2097447027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba6df60 0x563c4be19ff0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8004410 tx=0x7f5ff801d7b0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.822+0000 7f600bffa740 1 -- 172.21.15.134:0/2097447027 shutdown_connections 2024-07-26T11:15:37.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.822+0000 7f600bffa740 1 --2- 172.21.15.134:0/2097447027 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563c4ba01c90 0x563c4bea56f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.822+0000 7f600bffa740 1 --2- 172.21.15.134:0/2097447027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c4ba6df60 0x563c4be19ff0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.822+0000 7f600bffa740 1 --2- 172.21.15.134:0/2097447027 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c4ba4d280 0x563c4be19ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.822+0000 7f600bffa740 1 --2- 172.21.15.134:0/2097447027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c4ba450a0 0x563c4be1d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.822+0000 7f5fdfe00640 0 client.25394 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:15:37.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.822+0000 7f600bffa740 1 -- 172.21.15.134:0/2097447027 >> 172.21.15.134:0/2097447027 conn(0x563c4ba62350 msgr2=0x563c4be23170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:37.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.822+0000 7f600bffa740 1 -- 172.21.15.134:0/2097447027 shutdown_connections 2024-07-26T11:15:37.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.822+0000 7f600bffa740 1 -- 172.21.15.134:0/2097447027 wait complete. 2024-07-26T11:15:37.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.822+0000 7f600bffa740 20 client.25394 trim_cache size 0 max 16384 2024-07-26T11:15:37.827 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:15:37.827 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_fsid": "105fbe5c-4b3e-11ef-bcb7-c7b262605968", 2024-07-26T11:15:37.827 INFO:teuthology.orchestra.run.smithi134.stdout: "whoami": 0, 2024-07-26T11:15:37.827 INFO:teuthology.orchestra.run.smithi134.stdout: "id": 24721, 2024-07-26T11:15:37.827 INFO:teuthology.orchestra.run.smithi134.stdout: "want_state": "up:active", 2024-07-26T11:15:37.827 INFO:teuthology.orchestra.run.smithi134.stdout: "state": "up:active", 2024-07-26T11:15:37.827 INFO:teuthology.orchestra.run.smithi134.stdout: "fs_name": "cephfs", 2024-07-26T11:15:37.827 INFO:teuthology.orchestra.run.smithi134.stdout: "rank_uptime": 312.26398414699997, 2024-07-26T11:15:37.827 INFO:teuthology.orchestra.run.smithi134.stdout: "mdsmap_epoch": 29, 2024-07-26T11:15:37.827 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch": 90, 2024-07-26T11:15:37.827 INFO:teuthology.orchestra.run.smithi134.stdout: "osdmap_epoch_barrier": 90, 2024-07-26T11:15:37.828 INFO:teuthology.orchestra.run.smithi134.stdout: "uptime": 337.80475738799998 2024-07-26T11:15:37.828 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:15:37.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.823+0000 7f5fff200640 1 -- 172.21.15.134:0/290656649 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5fe40b3c20 msgr2=0x7f5fe40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:37.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.823+0000 7f5fff200640 1 --2- 172.21.15.134:0/290656649 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5fe40b3c20 0x7f5fe40b6130 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8005fd0 tx=0x7f5ff8009260 comp rx=0 tx=0).stop 2024-07-26T11:15:37.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.823+0000 7f5fff200640 1 -- 172.21.15.134:0/290656649 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6000108d20 msgr2=0x7f600019a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:37.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.823+0000 7f5fff200640 1 --2- 172.21.15.134:0/290656649 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6000108d20 0x7f600019a350 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5fec035040 tx=0x7f5fec032d50 comp rx=0 tx=0).stop 2024-07-26T11:15:37.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.824+0000 7f5fff200640 1 -- 172.21.15.134:0/290656649 shutdown_connections 2024-07-26T11:15:37.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.824+0000 7f5fff200640 1 --2- 172.21.15.134:0/290656649 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5fe40b3c20 0x7f5fe40b6130 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.824+0000 7f5fff200640 1 --2- 172.21.15.134:0/290656649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6000109670 0x7f600019f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.824+0000 7f5fff200640 1 --2- 172.21.15.134:0/290656649 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6000108d20 0x7f600019a350 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.824+0000 7f5fff200640 1 --2- 172.21.15.134:0/290656649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60001033e0 0x7f6000199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:37.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.824+0000 7f5fff200640 1 -- 172.21.15.134:0/290656649 >> 172.21.15.134:0/290656649 conn(0x7f60000fee30 msgr2=0x7f6000101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:37.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.824+0000 7f5fff200640 1 -- 172.21.15.134:0/290656649 shutdown_connections 2024-07-26T11:15:37.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:37.824+0000 7f5fff200640 1 -- 172.21.15.134:0/290656649 wait complete. 2024-07-26T11:15:38.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:38 smithi134 ceph-mon[31966]: pgmap v470: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:15:38.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:38 smithi189 ceph-mon[36790]: pgmap v470: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:15:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:38 smithi193 ceph-mon[37473]: pgmap v470: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:15:38.767 INFO:teuthology.run_tasks:Running task sequential... 2024-07-26T11:15:38.777 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-07-26T11:15:38.777 INFO:teuthology.task.sequential:In sequential, running task print... 2024-07-26T11:15:38.778 INFO:teuthology.task.print:Enabling mgr modules 2024-07-26T11:15:38.778 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-07-26T11:15:38.779 INFO:teuthology.task.exec:Executing custom commands... 2024-07-26T11:15:38.779 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi134.front.sepia.ceph.com 2024-07-26T11:15:38.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-07-26T11:15:39.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.049+0000 7f557be00640 1 -- 172.21.15.134:0/2615093465 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c109990 msgr2=0x7f557c109da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:39.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.049+0000 7f557be00640 1 --2- 172.21.15.134:0/2615093465 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c109990 0x7f557c109da0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f556c009900 tx=0x7f556c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:15:39.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.049+0000 7f557be00640 1 -- 172.21.15.134:0/2615093465 shutdown_connections 2024-07-26T11:15:39.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.049+0000 7f557be00640 1 --2- 172.21.15.134:0/2615093465 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f557c10f290 0x7f557c10f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:39.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.049+0000 7f557be00640 1 --2- 172.21.15.134:0/2615093465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f557c10fbe0 0x7f557c116410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:39.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.049+0000 7f557be00640 1 --2- 172.21.15.134:0/2615093465 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c109990 0x7f557c109da0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:39.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.049+0000 7f557be00640 1 -- 172.21.15.134:0/2615093465 >> 172.21.15.134:0/2615093465 conn(0x7f557c0790b0 msgr2=0x7f557c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:39.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.049+0000 7f557be00640 1 -- 172.21.15.134:0/2615093465 shutdown_connections 2024-07-26T11:15:39.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.049+0000 7f557be00640 1 -- 172.21.15.134:0/2615093465 wait complete. 2024-07-26T11:15:39.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.050+0000 7f557be00640 1 Processor -- start 2024-07-26T11:15:39.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.050+0000 7f557be00640 1 -- start start 2024-07-26T11:15:39.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.051+0000 7f557be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f557c109990 0x7f557c1a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:39.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.051+0000 7f557be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f557c10f290 0x7f557c1a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:39.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.051+0000 7f557be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c10fbe0 0x7f557c1a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:39.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.051+0000 7f557be00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f557c1198a0 con 0x7f557c10fbe0 2024-07-26T11:15:39.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.051+0000 7f557be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f557c119720 con 0x7f557c109990 2024-07-26T11:15:39.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.051+0000 7f557be00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f557c119a20 con 0x7f557c10f290 2024-07-26T11:15:39.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.051+0000 7f557b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c10fbe0 0x7f557c1a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:39.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.051+0000 7f557aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f557c109990 0x7f557c1a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:39.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.051+0000 7f557aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f557c109990 0x7f557c1a0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35966/0 (socket says 172.21.15.134:35966) 2024-07-26T11:15:39.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.051+0000 7f557b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c10fbe0 0x7f557c1a5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:36414/0 (socket says 172.21.15.134:36414) 2024-07-26T11:15:39.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.051+0000 7f557aa00640 1 -- 172.21.15.134:0/696569674 learned_addr learned my addr 172.21.15.134:0/696569674 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:39.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.051+0000 7f557a000640 1 --2- 172.21.15.134:0/696569674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f557c10f290 0x7f557c1a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:39.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.052+0000 7f557b400640 1 -- 172.21.15.134:0/696569674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f557c10f290 msgr2=0x7f557c1a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:39.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.052+0000 7f557b400640 1 --2- 172.21.15.134:0/696569674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f557c10f290 0x7f557c1a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:39.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.052+0000 7f557b400640 1 -- 172.21.15.134:0/696569674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f557c109990 msgr2=0x7f557c1a0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:39.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.052+0000 7f557b400640 1 --2- 172.21.15.134:0/696569674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f557c109990 0x7f557c1a0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:39.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.052+0000 7f557b400640 1 -- 172.21.15.134:0/696569674 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557c1a6190 con 0x7f557c10fbe0 2024-07-26T11:15:39.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.052+0000 7f557aa00640 1 --2- 172.21.15.134:0/696569674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f557c109990 0x7f557c1a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:39.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.052+0000 7f557a000640 1 --2- 172.21.15.134:0/696569674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f557c10f290 0x7f557c1a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:39.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.052+0000 7f557b400640 1 --2- 172.21.15.134:0/696569674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c10fbe0 0x7f557c1a5a90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5574009fd0 tx=0x7f557400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:39.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.052+0000 7f5573400640 1 -- 172.21.15.134:0/696569674 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557400ce20 con 0x7f557c10fbe0 2024-07-26T11:15:39.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.052+0000 7f5573400640 1 -- 172.21.15.134:0/696569674 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5574005020 con 0x7f557c10fbe0 2024-07-26T11:15:39.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.052+0000 7f557be00640 1 -- 172.21.15.134:0/696569674 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f557c1a6480 con 0x7f557c10fbe0 2024-07-26T11:15:39.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.053+0000 7f5573400640 1 -- 172.21.15.134:0/696569674 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5574010690 con 0x7f557c10fbe0 2024-07-26T11:15:39.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.053+0000 7f557be00640 1 -- 172.21.15.134:0/696569674 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f557c1acc10 con 0x7f557c10fbe0 2024-07-26T11:15:39.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.054+0000 7f557be00640 1 -- 172.21.15.134:0/696569674 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f557c109da0 con 0x7f557c10fbe0 2024-07-26T11:15:39.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.056+0000 7f5573400640 1 -- 172.21.15.134:0/696569674 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 22) ==== 151077+0+0 (secure 0 0 0) 0x7f55740054d0 con 0x7f557c10fbe0 2024-07-26T11:15:39.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.058+0000 7f5573400640 1 --2- 172.21.15.134:0/696569674 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f55600b3c20 0x7f55600b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:39.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.058+0000 7f5573400640 1 -- 172.21.15.134:0/696569674 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f55740e4800 con 0x7f557c10fbe0 2024-07-26T11:15:39.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.058+0000 7f557aa00640 1 --2- 172.21.15.134:0/696569674 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f55600b3c20 0x7f55600b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:39.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.059+0000 7f557aa00640 1 --2- 172.21.15.134:0/696569674 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f55600b3c20 0x7f55600b6130 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f556c031440 tx=0x7f556c0058d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:39.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.060+0000 7f5573400640 1 -- 172.21.15.134:0/696569674 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f55740ab950 con 0x7f557c10fbe0 2024-07-26T11:15:39.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.326+0000 7f557be00640 1 -- 172.21.15.134:0/696569674 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f557c02e0c0 con 0x7f557c10fbe0 2024-07-26T11:15:39.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.384+0000 7f5573400640 1 -- 172.21.15.134:0/696569674 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 module 'snap_schedule' is already enabled v23) ==== 133+0+0 (secure 0 0 0) 0x7f55740ab250 con 0x7f557c10fbe0 2024-07-26T11:15:39.387 INFO:teuthology.orchestra.run.smithi134.stderr:module 'snap_schedule' is already enabled 2024-07-26T11:15:39.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.388+0000 7f557be00640 1 -- 172.21.15.134:0/696569674 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f55600b3c20 msgr2=0x7f55600b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:39.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.388+0000 7f557be00640 1 --2- 172.21.15.134:0/696569674 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f55600b3c20 0x7f55600b6130 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f556c031440 tx=0x7f556c0058d0 comp rx=0 tx=0).stop 2024-07-26T11:15:39.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.388+0000 7f557be00640 1 -- 172.21.15.134:0/696569674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c10fbe0 msgr2=0x7f557c1a5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:39.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.388+0000 7f557be00640 1 --2- 172.21.15.134:0/696569674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c10fbe0 0x7f557c1a5a90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5574009fd0 tx=0x7f557400ef10 comp rx=0 tx=0).stop 2024-07-26T11:15:39.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.388+0000 7f557be00640 1 -- 172.21.15.134:0/696569674 shutdown_connections 2024-07-26T11:15:39.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.388+0000 7f557be00640 1 --2- 172.21.15.134:0/696569674 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f55600b3c20 0x7f55600b6130 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:39.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.388+0000 7f557be00640 1 --2- 172.21.15.134:0/696569674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f557c10f290 0x7f557c1a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:39.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.388+0000 7f557be00640 1 --2- 172.21.15.134:0/696569674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f557c10fbe0 0x7f557c1a5a90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:39.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.388+0000 7f557be00640 1 --2- 172.21.15.134:0/696569674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f557c109990 0x7f557c1a0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:39.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.388+0000 7f557be00640 1 -- 172.21.15.134:0/696569674 >> 172.21.15.134:0/696569674 conn(0x7f557c0790b0 msgr2=0x7f557c10d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:39.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.388+0000 7f557be00640 1 -- 172.21.15.134:0/696569674 shutdown_connections 2024-07-26T11:15:39.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:39.388+0000 7f557be00640 1 -- 172.21.15.134:0/696569674 wait complete. 2024-07-26T11:15:39.430 INFO:teuthology.run_tasks:Running task kclient... 2024-07-26T11:15:39.445 INFO:tasks.kclient:Mounting kernel clients... 2024-07-26T11:15:39.445 INFO:tasks.kclient:config is {'client.0': {}} 2024-07-26T11:15:39.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:15:39.445 DEBUG:teuthology.orchestra.run.smithi134:> ip netns list 2024-07-26T11:15:39.494 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-07-26T11:15:39.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:15:39.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-07-26T11:15:39.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:15:39.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link delete ceph-brx 2024-07-26T11:15:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:39 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/696569674' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T11:15:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:39 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/696569674' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T11:15:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:39 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/696569674' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-07-26T11:15:39.983 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-07-26T11:15:39.983 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-07-26T11:15:39.983 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T11:15:39.983 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs ls 2024-07-26T11:15:40.428 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:15:40.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:40 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/696569674' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T11:15:40.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:40 smithi134 ceph-mon[31966]: mgrmap e23: x(active, since 13m), standbys: z, y 2024-07-26T11:15:40.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:40 smithi134 ceph-mon[31966]: pgmap v471: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-07-26T11:15:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:40 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/696569674' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T11:15:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:40 smithi189 ceph-mon[36790]: mgrmap e23: x(active, since 13m), standbys: z, y 2024-07-26T11:15:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:40 smithi189 ceph-mon[36790]: pgmap v471: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-07-26T11:15:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:40 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/696569674' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-07-26T11:15:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:40 smithi193 ceph-mon[37473]: mgrmap e23: x(active, since 13m), standbys: z, y 2024-07-26T11:15:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:40 smithi193 ceph-mon[37473]: pgmap v471: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-07-26T11:15:42.272 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:42 smithi134 ceph-mon[31966]: pgmap v472: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:15:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:42 smithi189 ceph-mon[36790]: pgmap v472: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:15:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:42 smithi193 ceph-mon[37473]: pgmap v472: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:15:42.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.846+0000 7fc24b200640 1 -- 172.21.15.134:0/4249178658 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc24c109670 msgr2=0x7fc24c10fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:42.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.846+0000 7fc24b200640 1 --2- 172.21.15.134:0/4249178658 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc24c109670 0x7fc24c10fea0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc244009900 tx=0x7fc24402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:15:42.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.846+0000 7fc24b200640 1 -- 172.21.15.134:0/4249178658 shutdown_connections 2024-07-26T11:15:42.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.846+0000 7fc24b200640 1 --2- 172.21.15.134:0/4249178658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc24c108d20 0x7fc24c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:42.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.846+0000 7fc24b200640 1 --2- 172.21.15.134:0/4249178658 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc24c109670 0x7fc24c10fea0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:42.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.846+0000 7fc24b200640 1 --2- 172.21.15.134:0/4249178658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc24c1033e0 0x7fc24c1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:42.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.846+0000 7fc24b200640 1 -- 172.21.15.134:0/4249178658 >> 172.21.15.134:0/4249178658 conn(0x7fc24c0fee70 msgr2=0x7fc24c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:42.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.846+0000 7fc24b200640 1 -- 172.21.15.134:0/4249178658 shutdown_connections 2024-07-26T11:15:42.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.846+0000 7fc24b200640 1 -- 172.21.15.134:0/4249178658 wait complete. 2024-07-26T11:15:42.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.848+0000 7fc24b200640 1 Processor -- start 2024-07-26T11:15:42.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.848+0000 7fc24b200640 1 -- start start 2024-07-26T11:15:42.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.849+0000 7fc24b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc24c1033e0 0x7fc24c19bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:42.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.849+0000 7fc24b200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc24c108d20 0x7fc24c19c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:42.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.849+0000 7fc24b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc24c109670 0x7fc24c1a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:42.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.849+0000 7fc24b200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c113320 con 0x7fc24c108d20 2024-07-26T11:15:42.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.849+0000 7fc24b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1131a0 con 0x7fc24c109670 2024-07-26T11:15:42.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.849+0000 7fc24b200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc24c1134a0 con 0x7fc24c1033e0 2024-07-26T11:15:42.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.849+0000 7fc249400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc24c108d20 0x7fc24c19c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:42.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.849+0000 7fc249e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc24c1033e0 0x7fc24c19bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:42.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.849+0000 7fc249e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc24c1033e0 0x7fc24c19bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52642/0 (socket says 172.21.15.134:52642) 2024-07-26T11:15:42.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.849+0000 7fc249e00640 1 -- 172.21.15.134:0/1306247890 learned_addr learned my addr 172.21.15.134:0/1306247890 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:42.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.849+0000 7fc24a800640 1 --2- 172.21.15.134:0/1306247890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc24c109670 0x7fc24c1a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:42.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.850+0000 7fc249400640 1 -- 172.21.15.134:0/1306247890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc24c1033e0 msgr2=0x7fc24c19bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:42.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.850+0000 7fc249400640 1 --2- 172.21.15.134:0/1306247890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc24c1033e0 0x7fc24c19bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:42.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.850+0000 7fc249400640 1 -- 172.21.15.134:0/1306247890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc24c109670 msgr2=0x7fc24c1a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:42.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.850+0000 7fc249400640 1 --2- 172.21.15.134:0/1306247890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc24c109670 0x7fc24c1a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:42.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.850+0000 7fc249400640 1 -- 172.21.15.134:0/1306247890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc24c1a1ea0 con 0x7fc24c108d20 2024-07-26T11:15:42.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.850+0000 7fc249e00640 1 --2- 172.21.15.134:0/1306247890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc24c1033e0 0x7fc24c19bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:42.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.850+0000 7fc249400640 1 --2- 172.21.15.134:0/1306247890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc24c108d20 0x7fc24c19c4d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc24c06b3c0 tx=0x7fc23400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:42.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.850+0000 7fc242a00640 1 -- 172.21.15.134:0/1306247890 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc234010430 con 0x7fc24c108d20 2024-07-26T11:15:42.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.851+0000 7fc242a00640 1 -- 172.21.15.134:0/1306247890 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc234010a80 con 0x7fc24c108d20 2024-07-26T11:15:42.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.851+0000 7fc24b200640 1 -- 172.21.15.134:0/1306247890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc24c1a2190 con 0x7fc24c108d20 2024-07-26T11:15:42.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.851+0000 7fc242a00640 1 -- 172.21.15.134:0/1306247890 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc234017a20 con 0x7fc24c108d20 2024-07-26T11:15:42.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.851+0000 7fc24b200640 1 -- 172.21.15.134:0/1306247890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc24c1a88f0 con 0x7fc24c108d20 2024-07-26T11:15:42.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.852+0000 7fc24b200640 1 -- 172.21.15.134:0/1306247890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc24c1037f0 con 0x7fc24c108d20 2024-07-26T11:15:42.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.856+0000 7fc242a00640 1 -- 172.21.15.134:0/1306247890 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fc234017be0 con 0x7fc24c108d20 2024-07-26T11:15:42.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.858+0000 7fc242a00640 1 --2- 172.21.15.134:0/1306247890 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc22c0b3d20 0x7fc22c0b6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:42.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.858+0000 7fc242a00640 1 -- 172.21.15.134:0/1306247890 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc2340e4ad0 con 0x7fc24c108d20 2024-07-26T11:15:42.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.858+0000 7fc249e00640 1 --2- 172.21.15.134:0/1306247890 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc22c0b3d20 0x7fc22c0b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:42.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.859+0000 7fc249e00640 1 --2- 172.21.15.134:0/1306247890 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc22c0b3d20 0x7fc22c0b6230 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc23c011550 tx=0x7fc23c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:42.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:42.860+0000 7fc242a00640 1 -- 172.21.15.134:0/1306247890 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc2340abcb0 con 0x7fc24c108d20 2024-07-26T11:15:43.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:43.101+0000 7fc24b200640 1 -- 172.21.15.134:0/1306247890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc24c19d060 con 0x7fc24c108d20 2024-07-26T11:15:43.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:43.102+0000 7fc242a00640 1 -- 172.21.15.134:0/1306247890 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v29) ==== 53+0+88 (secure 0 0 0) 0x7fc24c19d060 con 0x7fc24c108d20 2024-07-26T11:15:43.105 INFO:teuthology.orchestra.run.smithi134.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-07-26T11:15:43.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:43.106+0000 7fc24b200640 1 -- 172.21.15.134:0/1306247890 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc22c0b3d20 msgr2=0x7fc22c0b6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:43.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:43.106+0000 7fc24b200640 1 --2- 172.21.15.134:0/1306247890 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc22c0b3d20 0x7fc22c0b6230 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc23c011550 tx=0x7fc23c005f90 comp rx=0 tx=0).stop 2024-07-26T11:15:43.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:43.106+0000 7fc24b200640 1 -- 172.21.15.134:0/1306247890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc24c108d20 msgr2=0x7fc24c19c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:43.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:43.106+0000 7fc24b200640 1 --2- 172.21.15.134:0/1306247890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc24c108d20 0x7fc24c19c4d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc24c06b3c0 tx=0x7fc23400ef90 comp rx=0 tx=0).stop 2024-07-26T11:15:43.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:43.106+0000 7fc24b200640 1 -- 172.21.15.134:0/1306247890 shutdown_connections 2024-07-26T11:15:43.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:43.106+0000 7fc24b200640 1 --2- 172.21.15.134:0/1306247890 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc22c0b3d20 0x7fc22c0b6230 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:43.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:43.106+0000 7fc24b200640 1 --2- 172.21.15.134:0/1306247890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc24c109670 0x7fc24c1a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:43.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:43.106+0000 7fc24b200640 1 --2- 172.21.15.134:0/1306247890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc24c108d20 0x7fc24c19c4d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:43.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:43.106+0000 7fc24b200640 1 --2- 172.21.15.134:0/1306247890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc24c1033e0 0x7fc24c19bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:43.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:43.106+0000 7fc24b200640 1 -- 172.21.15.134:0/1306247890 >> 172.21.15.134:0/1306247890 conn(0x7fc24c0fee70 msgr2=0x7fc24c108360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:43.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:43.106+0000 7fc24b200640 1 -- 172.21.15.134:0/1306247890 shutdown_connections 2024-07-26T11:15:43.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:43.106+0000 7fc24b200640 1 -- 172.21.15.134:0/1306247890 wait complete. 2024-07-26T11:15:43.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:43 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1306247890' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T11:15:43.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:43 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1306247890' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T11:15:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:43 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1306247890' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-07-26T11:15:44.069 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-07-26T11:15:44.069 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-07-26T11:15:44.070 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi134.front.sepia.ceph.com 2024-07-26T11:15:44.070 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-07-26T11:15:44.070 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-07-26T11:15:44.070 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-07-26T11:15:44.070 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-07-26T11:15:44.070 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-07-26T11:15:44.070 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-07-26T11:15:44.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:15:44.070 DEBUG:teuthology.orchestra.run.smithi134:> ip addr 2024-07-26T11:15:44.125 INFO:teuthology.orchestra.run.smithi134.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-26T11:15:44.125 INFO:teuthology.orchestra.run.smithi134.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-26T11:15:44.125 INFO:teuthology.orchestra.run.smithi134.stdout: inet 127.0.0.1/8 scope host lo 2024-07-26T11:15:44.125 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2024-07-26T11:15:44.125 INFO:teuthology.orchestra.run.smithi134.stdout: inet6 ::1/128 scope host 2024-07-26T11:15:44.126 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2024-07-26T11:15:44.126 INFO:teuthology.orchestra.run.smithi134.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T11:15:44.126 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:d9:38:e6 brd ff:ff:ff:ff:ff:ff 2024-07-26T11:15:44.126 INFO:teuthology.orchestra.run.smithi134.stdout: altname enp7s0f0 2024-07-26T11:15:44.126 INFO:teuthology.orchestra.run.smithi134.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T11:15:44.126 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:d9:38:e7 brd ff:ff:ff:ff:ff:ff 2024-07-26T11:15:44.126 INFO:teuthology.orchestra.run.smithi134.stdout: altname enp7s0f1 2024-07-26T11:15:44.126 INFO:teuthology.orchestra.run.smithi134.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-07-26T11:15:44.126 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:88:6d:e8 brd ff:ff:ff:ff:ff:ff 2024-07-26T11:15:44.126 INFO:teuthology.orchestra.run.smithi134.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-26T11:15:44.126 INFO:teuthology.orchestra.run.smithi134.stdout: link/ether 0c:c4:7a:88:6d:e9 brd ff:ff:ff:ff:ff:ff 2024-07-26T11:15:44.126 INFO:teuthology.orchestra.run.smithi134.stdout: inet 172.21.15.134/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-07-26T11:15:44.127 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft 41670sec preferred_lft 41670sec 2024-07-26T11:15:44.127 INFO:teuthology.orchestra.run.smithi134.stdout: inet6 fe80::ec4:7aff:fe88:6de9/64 scope link 2024-07-26T11:15:44.127 INFO:teuthology.orchestra.run.smithi134.stdout: valid_lft forever preferred_lft forever 2024-07-26T11:15:44.128 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-07-26T11:15:44.128 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T11:15:44.128 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-07-26T11:15:44.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link add name ceph-brx type bridge 2024-07-26T11:15:44.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip addr flush dev ceph-brx 2024-07-26T11:15:44.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link set ceph-brx up 2024-07-26T11:15:44.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-07-26T11:15:44.128 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-07-26T11:15:44.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:15:44.250 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:44 smithi134 ceph-mon[31966]: pgmap v473: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:15:44.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:15:44.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:15:44.358 DEBUG:teuthology.orchestra.run.smithi134:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-07-26T11:15:44.422 INFO:teuthology.orchestra.run.smithi134.stdout:1 2024-07-26T11:15:44.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:15:44.426 DEBUG:teuthology.orchestra.run.smithi134:> route 2024-07-26T11:15:44.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:44 smithi189 ceph-mon[36790]: pgmap v473: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:15:44.470 INFO:teuthology.orchestra.run.smithi134.stdout:Kernel IP routing table 2024-07-26T11:15:44.470 INFO:teuthology.orchestra.run.smithi134.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-07-26T11:15:44.470 INFO:teuthology.orchestra.run.smithi134.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-07-26T11:15:44.470 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-07-26T11:15:44.470 INFO:teuthology.orchestra.run.smithi134.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-07-26T11:15:44.471 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T11:15:44.471 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-07-26T11:15:44.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-07-26T11:15:44.472 DEBUG:teuthology.orchestra.run.smithi134:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-07-26T11:15:44.472 DEBUG:teuthology.orchestra.run.smithi134:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-07-26T11:15:44.472 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-07-26T11:15:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:44 smithi193 ceph-mon[37473]: pgmap v473: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:15:44.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:15:44.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:15:44.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:15:44.724 DEBUG:teuthology.orchestra.run.smithi134:> ip netns list 2024-07-26T11:15:44.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:15:44.770 DEBUG:teuthology.orchestra.run.smithi134:> ip netns list-id 2024-07-26T11:15:44.867 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T11:15:44.867 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-07-26T11:15:44.867 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-07-26T11:15:44.868 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-07-26T11:15:44.868 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-07-26T11:15:44.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:15:45.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:15:45.051 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-07-26T11:15:45.051 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T11:15:45.051 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-07-26T11:15:45.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-07-26T11:15:45.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-07-26T11:15:45.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-07-26T11:15:45.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-07-26T11:15:45.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-07-26T11:15:45.051 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-07-26T11:15:45.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:15:45.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:15:45.312 DEBUG:teuthology.orchestra.run.smithi134:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-07-26T11:15:45.312 DEBUG:teuthology.orchestra.run.smithi134:> set -e 2024-07-26T11:15:45.312 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link set brx.0 up 2024-07-26T11:15:45.312 DEBUG:teuthology.orchestra.run.smithi134:> sudo ip link set dev brx.0 master ceph-brx 2024-07-26T11:15:45.312 DEBUG:teuthology.orchestra.run.smithi134:> ') 2024-07-26T11:15:45.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:15:45.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:15:45.514 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T11:15:45.514 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-07-26T11:15:45.561 INFO:teuthology.orchestra.run.smithi134.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-07-26T11:15:45.561 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-07-26T11:15:45.561 DEBUG:teuthology.orchestra.run.smithi134:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-07-26T11:15:45.657 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-07-26T11:15:45.657 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-07-26T11:15:45.657 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/163a060a-b704-4597-b8c8-f5cde5094a3f 2024-07-26T11:15:45.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:15:45.657 DEBUG:teuthology.orchestra.run.smithi134:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/163a060a-b704-4597-b8c8-f5cde5094a3f /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-07-26T11:15:45.852 INFO:teuthology.orchestra.run.smithi134.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-07-26T11:15:45.852 INFO:teuthology.orchestra.run.smithi134.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-07-26T11:15:45.853 INFO:teuthology.orchestra.run.smithi134.stdout:invalid new device string format 2024-07-26T11:15:46.284 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:46 smithi134 ceph-mon[31966]: pgmap v474: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-07-26T11:15:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:46 smithi189 ceph-mon[36790]: pgmap v474: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-07-26T11:15:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:46 smithi193 ceph-mon[37473]: pgmap v474: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-07-26T11:15:46.600 INFO:teuthology.orchestra.run.smithi134.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-07-26T11:15:46.601 INFO:teuthology.orchestra.run.smithi134.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-07-26T11:15:46.601 INFO:teuthology.orchestra.run.smithi134.stdout:invalid new device string format 2024-07-26T11:15:46.601 INFO:teuthology.orchestra.run.smithi134.stdout:mount.ceph: resolved to: "172.21.15.134:3300,172.21.15.189:3300,172.21.15.193:3300" 2024-07-26T11:15:46.601 INFO:teuthology.orchestra.run.smithi134.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.134:3300,172.21.15.189:3300,172.21.15.193:3300:/volumes/qa/sv_1/163a060a-b704-4597-b8c8-f5cde5094a3f 2024-07-26T11:15:46.601 INFO:teuthology.orchestra.run.smithi134.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,key=0,fsid=105fbe5c-4b3e-11ef-bcb7-c7b262605968" will pass to kernel 2024-07-26T11:15:46.604 INFO:teuthology.orchestra.run.smithi134.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-07-26T11:15:46.604 INFO:teuthology.orchestra.run.smithi134.stdout: You just mounted a file system that supports labels which does not 2024-07-26T11:15:46.604 INFO:teuthology.orchestra.run.smithi134.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-07-26T11:15:46.604 INFO:teuthology.orchestra.run.smithi134.stdout: applications will generate AVC messages and not be allowed access to 2024-07-26T11:15:46.604 INFO:teuthology.orchestra.run.smithi134.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-07-26T11:15:46.609 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-07-26T11:15:46.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-07-26T11:15:46.609 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-07-26T11:15:46.695 DEBUG:teuthology.orchestra.run.smithi134:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T11:15:46.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T11:15:46.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:15:46.823 INFO:teuthology.orchestra.run.smithi134.stdout:client25409 2024-07-26T11:15:46.823 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T11:15:46.823 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/proc/self/mounts of=/dev/stdout 2024-07-26T11:15:46.873 DEBUG:teuthology.orchestra.run.smithi134:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T11:15:47.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T11:15:47.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:15:47.043 INFO:teuthology.orchestra.run.smithi134.stdout:client25409 2024-07-26T11:15:47.043 DEBUG:teuthology.orchestra.run.smithi134:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-07-26T11:15:47.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-07-26T11:15:47.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:15:47.159 INFO:teuthology.orchestra.run.smithi134.stdout:client25409 2024-07-26T11:15:47.159 DEBUG:teuthology.orchestra.run.smithi134:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968.client25409/status') 2024-07-26T11:15:47.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-07-26T11:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout:instance: client.25409 (3)172.21.15.134:0/4248055675 2024-07-26T11:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout:blocklisted: false 2024-07-26T11:15:47.291 INFO:teuthology.orchestra.run.smithi134.stderr:0+1 records in 2024-07-26T11:15:47.291 INFO:teuthology.orchestra.run.smithi134.stderr:0+1 records out 2024-07-26T11:15:47.291 INFO:teuthology.orchestra.run.smithi134.stderr:72 bytes copied, 8.4521e-05 s, 852 kB/s 2024-07-26T11:15:47.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-07-26T11:15:47.304 INFO:teuthology.run_tasks:Running task exec... 2024-07-26T11:15:47.313 INFO:teuthology.task.exec:Executing custom commands... 2024-07-26T11:15:47.313 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi134.front.sepia.ceph.com 2024-07-26T11:15:47.314 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-07-26T11:15:47.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.576+0000 7ff9d6400640 1 -- 172.21.15.134:0/247425746 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9d0075b30 msgr2=0x7ff9d010f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:47.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.576+0000 7ff9d6400640 1 --2- 172.21.15.134:0/247425746 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9d0075b30 0x7ff9d010f370 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8009900 tx=0x7ff9b802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:15:47.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.577+0000 7ff9d6400640 1 -- 172.21.15.134:0/247425746 shutdown_connections 2024-07-26T11:15:47.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.577+0000 7ff9d6400640 1 --2- 172.21.15.134:0/247425746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9d010f8b0 0x7ff9d0111d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:47.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.577+0000 7ff9d6400640 1 --2- 172.21.15.134:0/247425746 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9d0075b30 0x7ff9d010f370 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:47.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.577+0000 7ff9d6400640 1 --2- 172.21.15.134:0/247425746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9d00751e0 0x7ff9d00755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:47.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.577+0000 7ff9d6400640 1 -- 172.21.15.134:0/247425746 >> 172.21.15.134:0/247425746 conn(0x7ff9d00fedc0 msgr2=0x7ff9d0101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:47.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.577+0000 7ff9d6400640 1 -- 172.21.15.134:0/247425746 shutdown_connections 2024-07-26T11:15:47.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.577+0000 7ff9d6400640 1 -- 172.21.15.134:0/247425746 wait complete. 2024-07-26T11:15:47.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.578+0000 7ff9d6400640 1 Processor -- start 2024-07-26T11:15:47.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.578+0000 7ff9d6400640 1 -- start start 2024-07-26T11:15:47.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.578+0000 7ff9d6400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9d00751e0 0x7ff9d019bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:47.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.579+0000 7ff9d6400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9d0075b30 0x7ff9d019c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:47.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.579+0000 7ff9d6400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9d010f8b0 0x7ff9d01a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:47.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.579+0000 7ff9d6400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d01154c0 con 0x7ff9d0075b30 2024-07-26T11:15:47.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.579+0000 7ff9d6400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d0115340 con 0x7ff9d00751e0 2024-07-26T11:15:47.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.579+0000 7ff9d6400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d0115640 con 0x7ff9d010f8b0 2024-07-26T11:15:47.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.579+0000 7ff9d5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9d00751e0 0x7ff9d019bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:47.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.579+0000 7ff9d5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9d00751e0 0x7ff9d019bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35698/0 (socket says 172.21.15.134:35698) 2024-07-26T11:15:47.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.579+0000 7ff9d5000640 1 -- 172.21.15.134:0/507749322 learned_addr learned my addr 172.21.15.134:0/507749322 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:47.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.579+0000 7ff9cfe00640 1 --2- 172.21.15.134:0/507749322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9d0075b30 0x7ff9d019c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:47.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.579+0000 7ff9d5a00640 1 --2- 172.21.15.134:0/507749322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9d010f8b0 0x7ff9d01a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:47.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.579+0000 7ff9d5000640 1 -- 172.21.15.134:0/507749322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9d010f8b0 msgr2=0x7ff9d01a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:47.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.579+0000 7ff9d5000640 1 --2- 172.21.15.134:0/507749322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9d010f8b0 0x7ff9d01a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:47.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.579+0000 7ff9d5000640 1 -- 172.21.15.134:0/507749322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9d0075b30 msgr2=0x7ff9d019c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:47.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.579+0000 7ff9d5000640 1 --2- 172.21.15.134:0/507749322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9d0075b30 0x7ff9d019c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:47.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.579+0000 7ff9d5000640 1 -- 172.21.15.134:0/507749322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d01a1e10 con 0x7ff9d00751e0 2024-07-26T11:15:47.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.580+0000 7ff9d5000640 1 --2- 172.21.15.134:0/507749322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9d00751e0 0x7ff9d019bee0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000e9f0 tx=0x7ff9c000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:47.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.580+0000 7ff9cd600640 1 -- 172.21.15.134:0/507749322 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c000cd80 con 0x7ff9d00751e0 2024-07-26T11:15:47.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.580+0000 7ff9d6400640 1 -- 172.21.15.134:0/507749322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9d01a2100 con 0x7ff9d00751e0 2024-07-26T11:15:47.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.580+0000 7ff9cd600640 1 -- 172.21.15.134:0/507749322 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9c00054d0 con 0x7ff9d00751e0 2024-07-26T11:15:47.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.580+0000 7ff9cd600640 1 -- 172.21.15.134:0/507749322 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0010690 con 0x7ff9d00751e0 2024-07-26T11:15:47.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.580+0000 7ff9d6400640 1 -- 172.21.15.134:0/507749322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9d01a8890 con 0x7ff9d00751e0 2024-07-26T11:15:47.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.581+0000 7ff9d6400640 1 -- 172.21.15.134:0/507749322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9d00755f0 con 0x7ff9d00751e0 2024-07-26T11:15:47.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.584+0000 7ff9cd600640 1 -- 172.21.15.134:0/507749322 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7ff9c0004e30 con 0x7ff9d00751e0 2024-07-26T11:15:47.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.585+0000 7ff9cd600640 1 --2- 172.21.15.134:0/507749322 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff9b40b3ca0 0x7ff9b40b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:47.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.586+0000 7ff9cd600640 1 -- 172.21.15.134:0/507749322 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff9c00e46e0 con 0x7ff9d00751e0 2024-07-26T11:15:47.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.586+0000 7ff9cfe00640 1 --2- 172.21.15.134:0/507749322 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff9b40b3ca0 0x7ff9b40b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:47.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.587+0000 7ff9cfe00640 1 --2- 172.21.15.134:0/507749322 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff9b40b3ca0 0x7ff9b40b61b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff9b802a9e0 tx=0x7ff9b8005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:47.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.587+0000 7ff9cd600640 1 -- 172.21.15.134:0/507749322 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff9c0055600 con 0x7ff9d00751e0 2024-07-26T11:15:47.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.846+0000 7ff9d6400640 1 -- 172.21.15.134:0/507749322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7ff9d01a8b80 con 0x7ff9d00751e0 2024-07-26T11:15:47.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.906+0000 7ff9cd600640 1 -- 172.21.15.134:0/507749322 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v30) ==== 117+0+0 (secure 0 0 0) 0x7ff9c00ab830 con 0x7ff9d00751e0 2024-07-26T11:15:47.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.910+0000 7ff9d6400640 1 -- 172.21.15.134:0/507749322 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff9b40b3ca0 msgr2=0x7ff9b40b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:47.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.910+0000 7ff9d6400640 1 --2- 172.21.15.134:0/507749322 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff9b40b3ca0 0x7ff9b40b61b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff9b802a9e0 tx=0x7ff9b8005a20 comp rx=0 tx=0).stop 2024-07-26T11:15:47.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.910+0000 7ff9d6400640 1 -- 172.21.15.134:0/507749322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9d00751e0 msgr2=0x7ff9d019bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:47.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.910+0000 7ff9d6400640 1 --2- 172.21.15.134:0/507749322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9d00751e0 0x7ff9d019bee0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000e9f0 tx=0x7ff9c000edf0 comp rx=0 tx=0).stop 2024-07-26T11:15:47.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.910+0000 7ff9d6400640 1 -- 172.21.15.134:0/507749322 shutdown_connections 2024-07-26T11:15:47.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.910+0000 7ff9d6400640 1 --2- 172.21.15.134:0/507749322 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff9b40b3ca0 0x7ff9b40b61b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:47.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.910+0000 7ff9d6400640 1 --2- 172.21.15.134:0/507749322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff9d010f8b0 0x7ff9d01a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:47.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.910+0000 7ff9d6400640 1 --2- 172.21.15.134:0/507749322 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff9d0075b30 0x7ff9d019c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:47.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.910+0000 7ff9d6400640 1 --2- 172.21.15.134:0/507749322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9d00751e0 0x7ff9d019bee0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:47.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.910+0000 7ff9d6400640 1 -- 172.21.15.134:0/507749322 >> 172.21.15.134:0/507749322 conn(0x7ff9d00fedc0 msgr2=0x7ff9d01009a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:47.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.911+0000 7ff9d6400640 1 -- 172.21.15.134:0/507749322 shutdown_connections 2024-07-26T11:15:47.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:47.911+0000 7ff9d6400640 1 -- 172.21.15.134:0/507749322 wait complete. 2024-07-26T11:15:47.938 INFO:teuthology.run_tasks:Running task check-counter... 2024-07-26T11:15:47.949 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-07-26T11:15:47.981 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-07-26T11:15:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:47 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/507749322' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T11:15:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:47 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T11:15:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:47 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/507749322' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T11:15:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:47 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T11:15:48.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.251+0000 7f5affe00640 1 -- 172.21.15.134:0/740498946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b0010b7b0 msgr2=0x7f5b00111fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:48.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.251+0000 7f5affe00640 1 --2- 172.21.15.134:0/740498946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b0010b7b0 0x7f5b00111fe0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5af8009be0 tx=0x7f5af802a6c0 comp rx=0 tx=0).stop 2024-07-26T11:15:48.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.251+0000 7f5affe00640 1 -- 172.21.15.134:0/740498946 shutdown_connections 2024-07-26T11:15:48.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.251+0000 7f5affe00640 1 --2- 172.21.15.134:0/740498946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b0010b7b0 0x7f5b00111fe0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:48.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.251+0000 7f5affe00640 1 --2- 172.21.15.134:0/740498946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b0010ae60 0x7f5b0010b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:48.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.251+0000 7f5affe00640 1 --2- 172.21.15.134:0/740498946 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b00105530 0x7f5b00105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:48.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.251+0000 7f5affe00640 1 -- 172.21.15.134:0/740498946 >> 172.21.15.134:0/740498946 conn(0x7f5b00100f90 msgr2=0x7f5b001033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:48.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.251+0000 7f5affe00640 1 -- 172.21.15.134:0/740498946 shutdown_connections 2024-07-26T11:15:48.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.251+0000 7f5affe00640 1 -- 172.21.15.134:0/740498946 wait complete. 2024-07-26T11:15:48.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.252+0000 7f5affe00640 1 Processor -- start 2024-07-26T11:15:48.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.253+0000 7f5affe00640 1 -- start start 2024-07-26T11:15:48.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.253+0000 7f5affe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b00105530 0x7f5b0019bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:48.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.253+0000 7f5affe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b0010ae60 0x7f5b0019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:48.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.253+0000 7f5affe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b0010b7b0 0x7f5b001a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:48.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.253+0000 7f5affe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5b00115460 con 0x7f5b0010ae60 2024-07-26T11:15:48.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.253+0000 7f5affe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b001152e0 con 0x7f5b00105530 2024-07-26T11:15:48.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.253+0000 7f5affe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5b001155e0 con 0x7f5b0010b7b0 2024-07-26T11:15:48.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.253+0000 7f5afe000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b0010ae60 0x7f5b0019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:48.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.254+0000 7f5afea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b00105530 0x7f5b0019bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:48.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.254+0000 7f5aff400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b0010b7b0 0x7f5b001a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:48.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.254+0000 7f5afe000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b0010ae60 0x7f5b0019c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:40396/0 (socket says 172.21.15.134:40396) 2024-07-26T11:15:48.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.254+0000 7f5afe000640 1 -- 172.21.15.134:0/4127049519 learned_addr learned my addr 172.21.15.134:0/4127049519 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:48.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.254+0000 7f5afea00640 1 -- 172.21.15.134:0/4127049519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b0010b7b0 msgr2=0x7f5b001a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:48.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.254+0000 7f5afea00640 1 --2- 172.21.15.134:0/4127049519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b0010b7b0 0x7f5b001a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:48.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.254+0000 7f5afea00640 1 -- 172.21.15.134:0/4127049519 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b0010ae60 msgr2=0x7f5b0019c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:48.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.254+0000 7f5afea00640 1 --2- 172.21.15.134:0/4127049519 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b0010ae60 0x7f5b0019c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:48.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.254+0000 7f5afea00640 1 -- 172.21.15.134:0/4127049519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b001a1dd0 con 0x7f5b00105530 2024-07-26T11:15:48.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.254+0000 7f5afe000640 1 --2- 172.21.15.134:0/4127049519 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b0010ae60 0x7f5b0019c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:48.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.254+0000 7f5afea00640 1 --2- 172.21.15.134:0/4127049519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b00105530 0x7f5b0019bea0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5af0005fd0 tx=0x7f5af000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:48.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.255+0000 7f5af7400640 1 -- 172.21.15.134:0/4127049519 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af000cd00 con 0x7f5b00105530 2024-07-26T11:15:48.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.255+0000 7f5af7400640 1 -- 172.21.15.134:0/4127049519 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5af00035f0 con 0x7f5b00105530 2024-07-26T11:15:48.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.255+0000 7f5affe00640 1 -- 172.21.15.134:0/4127049519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b001a20c0 con 0x7f5b00105530 2024-07-26T11:15:48.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.255+0000 7f5af7400640 1 -- 172.21.15.134:0/4127049519 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af0010670 con 0x7f5b00105530 2024-07-26T11:15:48.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.255+0000 7f5affe00640 1 -- 172.21.15.134:0/4127049519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b001a8850 con 0x7f5b00105530 2024-07-26T11:15:48.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.256+0000 7f5affe00640 1 -- 172.21.15.134:0/4127049519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b00105940 con 0x7f5b00105530 2024-07-26T11:15:48.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.258+0000 7f5af7400640 1 -- 172.21.15.134:0/4127049519 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f5af00037a0 con 0x7f5b00105530 2024-07-26T11:15:48.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.260+0000 7f5af7400640 1 --2- 172.21.15.134:0/4127049519 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5ae40b3c20 0x7f5ae40b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:48.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.260+0000 7f5af7400640 1 -- 172.21.15.134:0/4127049519 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5af00e43b0 con 0x7f5b00105530 2024-07-26T11:15:48.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.260+0000 7f5afe000640 1 --2- 172.21.15.134:0/4127049519 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5ae40b3c20 0x7f5ae40b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:48.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.261+0000 7f5afe000640 1 --2- 172.21.15.134:0/4127049519 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5ae40b3c20 0x7f5ae40b6130 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5b0010a990 tx=0x7f5aec005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:48.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.262+0000 7f5af7400640 1 -- 172.21.15.134:0/4127049519 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f5af0009510 con 0x7f5b00105530 2024-07-26T11:15:48.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:47 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/507749322' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T11:15:48.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:47 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-07-26T11:15:48.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.472+0000 7f5affe00640 1 -- 172.21.15.134:0/4127049519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5b0019cfc0 con 0x7f5b00105530 2024-07-26T11:15:48.477 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:15:48.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.474+0000 7f5af7400640 1 -- 172.21.15.134:0/4127049519 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25185 (secure 0 0 0) 0x7f5af00ab590 con 0x7f5b00105530 2024-07-26T11:15:48.477 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":90,"fsid":"105fbe5c-4b3e-11ef-bcb7-c7b262605968","created":"2024-07-26T11:00:05.955103+0000","modified":"2024-07-26T11:11:35.149490+0000","last_up_change":"2024-07-26T11:09:11.485512+0000","last_in_change":"2024-07-26T11:08:52.341356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T11:05:01.849170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T11:10:16.055725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T11:10:21.216611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T11:10:33.983067+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8abb1365-fc3e-4617-a9ac-76bece8cd7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6803","nonce":4183273954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6805","nonce":4183273954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6809","nonce":4183273954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6807","nonce":4183273954}]},"public_addr":"172.21.15.134:6803/4183273954","cluster_addr":"172.21.15.134:6805/4183273954","heartbeat_back_addr":"172.21.15.134:6809/4183273954","heartbeat_front_addr":"172.21.15.134:6807/4183273954","state":["exists","up"]},{"osd":1,"uuid":"6c5a7838-821e-44e1-97e4-655b9bfa9f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6801","nonce":35148049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6803","nonce":35148049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6807","nonce":35148049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6805","nonce":35148049}]},"public_addr":"172.21.15.189:6801/35148049","cluster_addr":"172.21.15.189:6803/35148049","heartbeat_back_addr":"172.21.15.189:6807/35148049","heartbeat_front_addr":"172.21.15.189:6805/35148049","state":["exists","up"]},{"osd":2,"uuid":"8bc42f7f-39fa-4d3b-bc17-21c68f9231fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6801","nonce":897516976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6803","nonce":897516976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6807","nonce":897516976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6805","nonce":897516976}]},"public_addr":"172.21.15.193:6801/897516976","cluster_addr":"172.21.15.193:6803/897516976","heartbeat_back_addr":"172.21.15.193:6807/897516976","heartbeat_front_addr":"172.21.15.193:6805/897516976","state":["exists","up"]},{"osd":3,"uuid":"0473d055-0b79-434f-9658-28269d2b1fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6811","nonce":3457190917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6813","nonce":3457190917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6817","nonce":3457190917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6815","nonce":3457190917}]},"public_addr":"172.21.15.134:6811/3457190917","cluster_addr":"172.21.15.134:6813/3457190917","heartbeat_back_addr":"172.21.15.134:6817/3457190917","heartbeat_front_addr":"172.21.15.134:6815/3457190917","state":["exists","up"]},{"osd":4,"uuid":"1d70eaae-56cc-4cdc-8f83-b88d2121f0ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6809","nonce":1786881347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6811","nonce":1786881347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6815","nonce":1786881347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6813","nonce":1786881347}]},"public_addr":"172.21.15.189:6809/1786881347","cluster_addr":"172.21.15.189:6811/1786881347","heartbeat_back_addr":"172.21.15.189:6815/1786881347","heartbeat_front_addr":"172.21.15.189:6813/1786881347","state":["exists","up"]},{"osd":5,"uuid":"a9b0b4cf-1608-45ed-b64e-afcf545d97fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6809","nonce":1092388797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6811","nonce":1092388797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6815","nonce":1092388797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6813","nonce":1092388797}]},"public_addr":"172.21.15.193:6809/1092388797","cluster_addr":"172.21.15.193:6811/1092388797","heartbeat_back_addr":"172.21.15.193:6815/1092388797","heartbeat_front_addr":"172.21.15.193:6813/1092388797","state":["exists","up"]},{"osd":6,"uuid":"0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6819","nonce":2471505438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6821","nonce":2471505438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6825","nonce":2471505438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6823","nonce":2471505438}]},"public_addr":"172.21.15.134:6819/2471505438","cluster_addr":"172.21.15.134:6821/2471505438","heartbeat_back_addr":"172.21.15.134:6825/2471505438","heartbeat_front_addr":"172.21.15.134:6823/2471505438","state":["exists","up"]},{"osd":7,"uuid":"394764a7-1976-4223-a3c0-5979d941a838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6817","nonce":1382694911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6819","nonce":1382694911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6823","nonce":1382694911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6821","nonce":1382694911}]},"public_addr":"172.21.15.189:6817/1382694911","cluster_addr":"172.21.15.189:6819/1382694911","heartbeat_back_addr":"172.21.15.189:6823/1382694911","heartbeat_front_addr":"172.21.15.189:6821/1382694911","state":["exists","up"]},{"osd":8,"uuid":"1c218082-0b4c-41c8-92b9-453ae8d15a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6817","nonce":896348138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6819","nonce":896348138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6823","nonce":896348138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6821","nonce":896348138}]},"public_addr":"172.21.15.193:6817/896348138","cluster_addr":"172.21.15.193:6819/896348138","heartbeat_back_addr":"172.21.15.193:6823/896348138","heartbeat_front_addr":"172.21.15.193:6821/896348138","state":["exists","up"]},{"osd":9,"uuid":"48f39082-4fa9-4443-b5b7-b141c10d9e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6827","nonce":511237025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6829","nonce":511237025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6833","nonce":511237025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6831","nonce":511237025}]},"public_addr":"172.21.15.134:6827/511237025","cluster_addr":"172.21.15.134:6829/511237025","heartbeat_back_addr":"172.21.15.134:6833/511237025","heartbeat_front_addr":"172.21.15.134:6831/511237025","state":["exists","up"]},{"osd":10,"uuid":"95157766-1856-42f7-aea1-9cf83b77c8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6825","nonce":431740858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6827","nonce":431740858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6831","nonce":431740858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6829","nonce":431740858}]},"public_addr":"172.21.15.189:6825/431740858","cluster_addr":"172.21.15.189:6827/431740858","heartbeat_back_addr":"172.21.15.189:6831/431740858","heartbeat_front_addr":"172.21.15.189:6829/431740858","state":["exists","up"]},{"osd":11,"uuid":"244fadc7-f079-4736-8e7e-c0ce5ae6f43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6825","nonce":1594590579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6827","nonce":1594590579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6831","nonce":1594590579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6829","nonce":1594590579}]},"public_addr":"172.21.15.193:6825/1594590579","cluster_addr":"172.21.15.193:6827/1594590579","heartbeat_back_addr":"172.21.15.193:6831/1594590579","heartbeat_front_addr":"172.21.15.193:6829/1594590579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:11.712203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:35.845013+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:58.165609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:31.929746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:56.941555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:20.688071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:54.681514+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:20.789384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:44.568246+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:18.644310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:45.365351+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:09:09.663805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3790862248":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6800/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:0/2906927791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3068153391":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3056560791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6801/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/1310359922":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/2318666906":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/2901579347":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3404218719":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3598071145":"2024-07-27T11:02:33.711793+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:15:48.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.477+0000 7f5affe00640 1 -- 172.21.15.134:0/4127049519 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5ae40b3c20 msgr2=0x7f5ae40b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:48.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.477+0000 7f5affe00640 1 --2- 172.21.15.134:0/4127049519 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5ae40b3c20 0x7f5ae40b6130 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5b0010a990 tx=0x7f5aec005f90 comp rx=0 tx=0).stop 2024-07-26T11:15:48.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.477+0000 7f5affe00640 1 -- 172.21.15.134:0/4127049519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b00105530 msgr2=0x7f5b0019bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:48.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.477+0000 7f5affe00640 1 --2- 172.21.15.134:0/4127049519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b00105530 0x7f5b0019bea0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5af0005fd0 tx=0x7f5af000ee50 comp rx=0 tx=0).stop 2024-07-26T11:15:48.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.477+0000 7f5affe00640 1 -- 172.21.15.134:0/4127049519 shutdown_connections 2024-07-26T11:15:48.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.477+0000 7f5affe00640 1 --2- 172.21.15.134:0/4127049519 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5ae40b3c20 0x7f5ae40b6130 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:48.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.477+0000 7f5affe00640 1 --2- 172.21.15.134:0/4127049519 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b0010b7b0 0x7f5b001a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:48.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.477+0000 7f5affe00640 1 --2- 172.21.15.134:0/4127049519 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5b0010ae60 0x7f5b0019c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:48.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.477+0000 7f5affe00640 1 --2- 172.21.15.134:0/4127049519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b00105530 0x7f5b0019bea0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:48.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.477+0000 7f5affe00640 1 -- 172.21.15.134:0/4127049519 >> 172.21.15.134:0/4127049519 conn(0x7f5b00100f90 msgr2=0x7f5b001033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:48.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.478+0000 7f5affe00640 1 -- 172.21.15.134:0/4127049519 shutdown_connections 2024-07-26T11:15:48.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.478+0000 7f5affe00640 1 -- 172.21.15.134:0/4127049519 wait complete. 2024-07-26T11:15:48.507 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-07-26T11:05:01.849170+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-07-26T11:10:16.055725+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-07-26T11:10:21.216611+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-07-26T11:10:33.983067+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '88', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-07-26T11:15:48.507 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-07-26T11:15:48.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.777+0000 7fb6a3e00640 1 -- 172.21.15.134:0/1508183234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6a4105530 msgr2=0x7fb6a4105940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:48.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.777+0000 7fb6a3e00640 1 --2- 172.21.15.134:0/1508183234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6a4105530 0x7fb6a4105940 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb694009860 tx=0x7fb69402a340 comp rx=0 tx=0).stop 2024-07-26T11:15:48.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.777+0000 7fb6a3e00640 1 -- 172.21.15.134:0/1508183234 shutdown_connections 2024-07-26T11:15:48.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.777+0000 7fb6a3e00640 1 --2- 172.21.15.134:0/1508183234 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6a410b7b0 0x7fb6a4111fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:48.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.777+0000 7fb6a3e00640 1 --2- 172.21.15.134:0/1508183234 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6a410ae60 0x7fb6a410b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:48.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.777+0000 7fb6a3e00640 1 --2- 172.21.15.134:0/1508183234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6a4105530 0x7fb6a4105940 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:48.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.777+0000 7fb6a3e00640 1 -- 172.21.15.134:0/1508183234 >> 172.21.15.134:0/1508183234 conn(0x7fb6a4100f90 msgr2=0x7fb6a41033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:48.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.778+0000 7fb6a3e00640 1 -- 172.21.15.134:0/1508183234 shutdown_connections 2024-07-26T11:15:48.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.778+0000 7fb6a3e00640 1 -- 172.21.15.134:0/1508183234 wait complete. 2024-07-26T11:15:48.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.779+0000 7fb6a3e00640 1 Processor -- start 2024-07-26T11:15:48.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.779+0000 7fb6a3e00640 1 -- start start 2024-07-26T11:15:48.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.779+0000 7fb6a3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6a4105530 0x7fb6a419be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:48.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.779+0000 7fb6a3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6a410ae60 0x7fb6a419c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:48.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.779+0000 7fb6a3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6a410b7b0 0x7fb6a41a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:48.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.780+0000 7fb6a3e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a4115480 con 0x7fb6a410b7b0 2024-07-26T11:15:48.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.780+0000 7fb6a3e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a4115300 con 0x7fb6a4105530 2024-07-26T11:15:48.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.780+0000 7fb6a3e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a4115600 con 0x7fb6a410ae60 2024-07-26T11:15:48.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.780+0000 7fb69a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6a410ae60 0x7fb6a419c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:48.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.780+0000 7fb6a3400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6a410b7b0 0x7fb6a41a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:48.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.780+0000 7fb6a2a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6a4105530 0x7fb6a419be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:48.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.780+0000 7fb69a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6a410ae60 0x7fb6a419c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52692/0 (socket says 172.21.15.134:52692) 2024-07-26T11:15:48.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.780+0000 7fb6a2a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6a4105530 0x7fb6a419be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35734/0 (socket says 172.21.15.134:35734) 2024-07-26T11:15:48.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.780+0000 7fb69a000640 1 -- 172.21.15.134:0/1932013665 learned_addr learned my addr 172.21.15.134:0/1932013665 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:48.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.780+0000 7fb6a3400640 1 -- 172.21.15.134:0/1932013665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6a410ae60 msgr2=0x7fb6a419c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:48.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.780+0000 7fb6a3400640 1 --2- 172.21.15.134:0/1932013665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6a410ae60 0x7fb6a419c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:48.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.780+0000 7fb6a3400640 1 -- 172.21.15.134:0/1932013665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6a4105530 msgr2=0x7fb6a419be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:48.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.780+0000 7fb6a3400640 1 --2- 172.21.15.134:0/1932013665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6a4105530 0x7fb6a419be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:48.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.780+0000 7fb6a3400640 1 -- 172.21.15.134:0/1932013665 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a41a1db0 con 0x7fb6a410b7b0 2024-07-26T11:15:48.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.781+0000 7fb6a3400640 1 --2- 172.21.15.134:0/1932013665 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6a410b7b0 0x7fb6a41a16b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb69c009fd0 tx=0x7fb69c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:48.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.781+0000 7fb69be00640 1 -- 172.21.15.134:0/1932013665 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c00ce20 con 0x7fb6a410b7b0 2024-07-26T11:15:48.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.781+0000 7fb69be00640 1 -- 172.21.15.134:0/1932013665 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb69c005020 con 0x7fb6a410b7b0 2024-07-26T11:15:48.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.781+0000 7fb69be00640 1 -- 172.21.15.134:0/1932013665 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c010690 con 0x7fb6a410b7b0 2024-07-26T11:15:48.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.781+0000 7fb6a3e00640 1 -- 172.21.15.134:0/1932013665 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6a41a20a0 con 0x7fb6a410b7b0 2024-07-26T11:15:48.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.781+0000 7fb6a3e00640 1 -- 172.21.15.134:0/1932013665 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6a41a8830 con 0x7fb6a410b7b0 2024-07-26T11:15:48.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.782+0000 7fb6a3e00640 1 -- 172.21.15.134:0/1932013665 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6a4105940 con 0x7fb6a410b7b0 2024-07-26T11:15:48.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.785+0000 7fb69be00640 1 -- 172.21.15.134:0/1932013665 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fb69c0054d0 con 0x7fb6a410b7b0 2024-07-26T11:15:48.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.787+0000 7fb69be00640 1 --2- 172.21.15.134:0/1932013665 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb6880b3c20 0x7fb6880b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:48.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.788+0000 7fb69be00640 1 -- 172.21.15.134:0/1932013665 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb69c0e47d0 con 0x7fb6a410b7b0 2024-07-26T11:15:48.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.788+0000 7fb6a2a00640 1 --2- 172.21.15.134:0/1932013665 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb6880b3c20 0x7fb6880b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:48.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.789+0000 7fb6a2a00640 1 --2- 172.21.15.134:0/1932013665 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb6880b3c20 0x7fb6880b6130 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb694009830 tx=0x7fb694030340 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:48.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:48.790+0000 7fb69be00640 1 -- 172.21.15.134:0/1932013665 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb69c0ab920 con 0x7fb6a410b7b0 2024-07-26T11:15:49.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.001+0000 7fb6a3e00640 1 -- 172.21.15.134:0/1932013665 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb6a402e0c0 con 0x7fb6a410b7b0 2024-07-26T11:15:49.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.003+0000 7fb69be00640 1 -- 172.21.15.134:0/1932013665 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7fb69c0ab220 con 0x7fb6a410b7b0 2024-07-26T11:15:49.005 INFO:teuthology.orchestra.run.smithi134.stdout:pg_num: 1 2024-07-26T11:15:49.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.006+0000 7fb6a3e00640 1 -- 172.21.15.134:0/1932013665 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb6880b3c20 msgr2=0x7fb6880b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:49.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.006+0000 7fb6a3e00640 1 --2- 172.21.15.134:0/1932013665 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb6880b3c20 0x7fb6880b6130 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb694009830 tx=0x7fb694030340 comp rx=0 tx=0).stop 2024-07-26T11:15:49.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.006+0000 7fb6a3e00640 1 -- 172.21.15.134:0/1932013665 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6a410b7b0 msgr2=0x7fb6a41a16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:49.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.006+0000 7fb6a3e00640 1 --2- 172.21.15.134:0/1932013665 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6a410b7b0 0x7fb6a41a16b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb69c009fd0 tx=0x7fb69c00ef10 comp rx=0 tx=0).stop 2024-07-26T11:15:49.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.006+0000 7fb6a3e00640 1 -- 172.21.15.134:0/1932013665 shutdown_connections 2024-07-26T11:15:49.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.006+0000 7fb6a3e00640 1 --2- 172.21.15.134:0/1932013665 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb6880b3c20 0x7fb6880b6130 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.006+0000 7fb6a3e00640 1 --2- 172.21.15.134:0/1932013665 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6a410ae60 0x7fb6a419c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.006+0000 7fb6a3e00640 1 --2- 172.21.15.134:0/1932013665 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb6a410b7b0 0x7fb6a41a16b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.006+0000 7fb6a3e00640 1 --2- 172.21.15.134:0/1932013665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6a4105530 0x7fb6a419be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.006+0000 7fb6a3e00640 1 -- 172.21.15.134:0/1932013665 >> 172.21.15.134:0/1932013665 conn(0x7fb6a4100f90 msgr2=0x7fb6a41033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:49.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.006+0000 7fb6a3e00640 1 -- 172.21.15.134:0/1932013665 shutdown_connections 2024-07-26T11:15:49.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.006+0000 7fb6a3e00640 1 -- 172.21.15.134:0/1932013665 wait complete. 2024-07-26T11:15:49.039 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-07-26T11:15:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:48 smithi189 ceph-mon[36790]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-26T11:15:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:48 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:15:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:48 smithi189 ceph-mon[36790]: pgmap v475: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T11:15:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:48 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/4127049519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:15:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:48 smithi193 ceph-mon[37473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-26T11:15:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:48 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:15:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:48 smithi193 ceph-mon[37473]: pgmap v475: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T11:15:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:48 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/4127049519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:15:49.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.319+0000 7fcc56600640 1 -- 172.21.15.134:0/3355785710 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc5010c980 msgr2=0x7fcc501163f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:49.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.319+0000 7fcc56600640 1 --2- 172.21.15.134:0/3355785710 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc5010c980 0x7fcc501163f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcc48009ca0 tx=0x7fcc4802a8c0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.320+0000 7fcc56600640 1 -- 172.21.15.134:0/3355785710 shutdown_connections 2024-07-26T11:15:49.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.320+0000 7fcc56600640 1 --2- 172.21.15.134:0/3355785710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc5010c030 0x7fcc5010c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.320+0000 7fcc56600640 1 --2- 172.21.15.134:0/3355785710 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc5010c980 0x7fcc501163f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.320+0000 7fcc56600640 1 --2- 172.21.15.134:0/3355785710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc50109990 0x7fcc50109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.320+0000 7fcc56600640 1 -- 172.21.15.134:0/3355785710 >> 172.21.15.134:0/3355785710 conn(0x7fcc500790b0 msgr2=0x7fcc50107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:49.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.320+0000 7fcc56600640 1 -- 172.21.15.134:0/3355785710 shutdown_connections 2024-07-26T11:15:49.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.320+0000 7fcc56600640 1 -- 172.21.15.134:0/3355785710 wait complete. 2024-07-26T11:15:49.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.321+0000 7fcc56600640 1 Processor -- start 2024-07-26T11:15:49.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.321+0000 7fcc56600640 1 -- start start 2024-07-26T11:15:49.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.321+0000 7fcc56600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc50109990 0x7fcc501a0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:49.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.321+0000 7fcc56600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc5010c030 0x7fcc501a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:49.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.321+0000 7fcc56600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc5010c980 0x7fcc501a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:49.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.321+0000 7fcc56600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcc50119880 con 0x7fcc50109990 2024-07-26T11:15:49.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.321+0000 7fcc56600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcc50119700 con 0x7fcc5010c030 2024-07-26T11:15:49.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.321+0000 7fcc56600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcc50119a00 con 0x7fcc5010c980 2024-07-26T11:15:49.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.322+0000 7fcc55200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc50109990 0x7fcc501a0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:49.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.322+0000 7fcc55200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc50109990 0x7fcc501a0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:40430/0 (socket says 172.21.15.134:40430) 2024-07-26T11:15:49.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.322+0000 7fcc55200640 1 -- 172.21.15.134:0/2003314281 learned_addr learned my addr 172.21.15.134:0/2003314281 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:49.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.322+0000 7fcc55c00640 1 --2- 172.21.15.134:0/2003314281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc5010c980 0x7fcc501a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:49.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.322+0000 7fcc47e00640 1 --2- 172.21.15.134:0/2003314281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc5010c030 0x7fcc501a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:49.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.322+0000 7fcc55200640 1 -- 172.21.15.134:0/2003314281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc5010c980 msgr2=0x7fcc501a5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:49.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.322+0000 7fcc55200640 1 --2- 172.21.15.134:0/2003314281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc5010c980 0x7fcc501a5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.322+0000 7fcc55200640 1 -- 172.21.15.134:0/2003314281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc5010c030 msgr2=0x7fcc501a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:49.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.322+0000 7fcc55200640 1 --2- 172.21.15.134:0/2003314281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc5010c030 0x7fcc501a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.322+0000 7fcc55200640 1 -- 172.21.15.134:0/2003314281 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc501a6190 con 0x7fcc50109990 2024-07-26T11:15:49.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.322+0000 7fcc55c00640 1 --2- 172.21.15.134:0/2003314281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc5010c980 0x7fcc501a5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:49.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.322+0000 7fcc55200640 1 --2- 172.21.15.134:0/2003314281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc50109990 0x7fcc501a0260 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcc4000e9f0 tx=0x7fcc4000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:49.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.323+0000 7fcc4e000640 1 -- 172.21.15.134:0/2003314281 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4000cd80 con 0x7fcc50109990 2024-07-26T11:15:49.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.323+0000 7fcc4e000640 1 -- 172.21.15.134:0/2003314281 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc400108e0 con 0x7fcc50109990 2024-07-26T11:15:49.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.323+0000 7fcc4e000640 1 -- 172.21.15.134:0/2003314281 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc40018980 con 0x7fcc50109990 2024-07-26T11:15:49.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.323+0000 7fcc56600640 1 -- 172.21.15.134:0/2003314281 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc501a6480 con 0x7fcc50109990 2024-07-26T11:15:49.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.323+0000 7fcc56600640 1 -- 172.21.15.134:0/2003314281 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc501acc10 con 0x7fcc50109990 2024-07-26T11:15:49.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.324+0000 7fcc56600640 1 -- 172.21.15.134:0/2003314281 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc50109da0 con 0x7fcc50109990 2024-07-26T11:15:49.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.328+0000 7fcc4e000640 1 -- 172.21.15.134:0/2003314281 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fcc40010430 con 0x7fcc50109990 2024-07-26T11:15:49.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.329+0000 7fcc4e000640 1 --2- 172.21.15.134:0/2003314281 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc340b3ca0 0x7fcc340b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:49.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.330+0000 7fcc4e000640 1 -- 172.21.15.134:0/2003314281 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fcc400e48b0 con 0x7fcc50109990 2024-07-26T11:15:49.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.330+0000 7fcc47e00640 1 --2- 172.21.15.134:0/2003314281 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc340b3ca0 0x7fcc340b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:49.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.330+0000 7fcc47e00640 1 --2- 172.21.15.134:0/2003314281 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc340b3ca0 0x7fcc340b61b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcc50108e50 tx=0x7fcc38005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:49.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.331+0000 7fcc4e000640 1 -- 172.21.15.134:0/2003314281 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fcc400ab3b0 con 0x7fcc50109990 2024-07-26T11:15:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:48 smithi134 ceph-mon[31966]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-07-26T11:15:49.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:48 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:15:49.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:48 smithi134 ceph-mon[31966]: pgmap v475: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-07-26T11:15:49.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:48 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/4127049519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:15:49.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.542+0000 7fcc56600640 1 -- 172.21.15.134:0/2003314281 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcc5002e0c0 con 0x7fcc50109990 2024-07-26T11:15:49.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.543+0000 7fcc4e000640 1 -- 172.21.15.134:0/2003314281 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7fcc400af900 con 0x7fcc50109990 2024-07-26T11:15:49.545 INFO:teuthology.orchestra.run.smithi134.stdout:pg_num: 32 2024-07-26T11:15:49.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.546+0000 7fcc56600640 1 -- 172.21.15.134:0/2003314281 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc340b3ca0 msgr2=0x7fcc340b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:49.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.546+0000 7fcc56600640 1 --2- 172.21.15.134:0/2003314281 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc340b3ca0 0x7fcc340b61b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcc50108e50 tx=0x7fcc38005f90 comp rx=0 tx=0).stop 2024-07-26T11:15:49.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.546+0000 7fcc56600640 1 -- 172.21.15.134:0/2003314281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc50109990 msgr2=0x7fcc501a0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:49.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.546+0000 7fcc56600640 1 --2- 172.21.15.134:0/2003314281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc50109990 0x7fcc501a0260 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcc4000e9f0 tx=0x7fcc4000eeb0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.546+0000 7fcc56600640 1 -- 172.21.15.134:0/2003314281 shutdown_connections 2024-07-26T11:15:49.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.546+0000 7fcc56600640 1 --2- 172.21.15.134:0/2003314281 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc340b3ca0 0x7fcc340b61b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.546+0000 7fcc56600640 1 --2- 172.21.15.134:0/2003314281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc5010c980 0x7fcc501a5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.546+0000 7fcc56600640 1 --2- 172.21.15.134:0/2003314281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc5010c030 0x7fcc501a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.546+0000 7fcc56600640 1 --2- 172.21.15.134:0/2003314281 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc50109990 0x7fcc501a0260 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.546+0000 7fcc56600640 1 -- 172.21.15.134:0/2003314281 >> 172.21.15.134:0/2003314281 conn(0x7fcc500790b0 msgr2=0x7fcc5010d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:49.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.546+0000 7fcc56600640 1 -- 172.21.15.134:0/2003314281 shutdown_connections 2024-07-26T11:15:49.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.546+0000 7fcc56600640 1 -- 172.21.15.134:0/2003314281 wait complete. 2024-07-26T11:15:49.575 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-07-26T11:15:49.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.846+0000 7f2b97200640 1 -- 172.21.15.134:0/538908738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b9010ae60 msgr2=0x7f2b9010b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:49.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.846+0000 7f2b97200640 1 --2- 172.21.15.134:0/538908738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b9010ae60 0x7f2b9010b270 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009900 tx=0x7f2b8002a410 comp rx=0 tx=0).stop 2024-07-26T11:15:49.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.847+0000 7f2b97200640 1 -- 172.21.15.134:0/538908738 shutdown_connections 2024-07-26T11:15:49.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.847+0000 7f2b97200640 1 --2- 172.21.15.134:0/538908738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b9010ae60 0x7f2b9010b270 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.847+0000 7f2b97200640 1 --2- 172.21.15.134:0/538908738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b9010b7b0 0x7f2b90111fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.847+0000 7f2b97200640 1 --2- 172.21.15.134:0/538908738 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b90105530 0x7f2b90105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.847+0000 7f2b97200640 1 -- 172.21.15.134:0/538908738 >> 172.21.15.134:0/538908738 conn(0x7f2b90100f90 msgr2=0x7f2b901033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:49.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.847+0000 7f2b97200640 1 -- 172.21.15.134:0/538908738 shutdown_connections 2024-07-26T11:15:49.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.847+0000 7f2b97200640 1 -- 172.21.15.134:0/538908738 wait complete. 2024-07-26T11:15:49.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.848+0000 7f2b97200640 1 Processor -- start 2024-07-26T11:15:49.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.848+0000 7f2b97200640 1 -- start start 2024-07-26T11:15:49.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.849+0000 7f2b97200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b90105530 0x7f2b9019bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:49.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.849+0000 7f2b97200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b9010ae60 0x7f2b9019c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:49.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.849+0000 7f2b97200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b9010b7b0 0x7f2b901a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:49.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.849+0000 7f2b97200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2b90115420 con 0x7f2b9010ae60 2024-07-26T11:15:49.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.849+0000 7f2b97200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2b901152a0 con 0x7f2b90105530 2024-07-26T11:15:49.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.849+0000 7f2b97200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2b901155a0 con 0x7f2b9010b7b0 2024-07-26T11:15:49.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.849+0000 7f2b95400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b9010ae60 0x7f2b9019c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:49.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.849+0000 7f2b95e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b90105530 0x7f2b9019bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:49.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.849+0000 7f2b95e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b90105530 0x7f2b9019bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35780/0 (socket says 172.21.15.134:35780) 2024-07-26T11:15:49.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.849+0000 7f2b95e00640 1 -- 172.21.15.134:0/2215632409 learned_addr learned my addr 172.21.15.134:0/2215632409 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:49.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.849+0000 7f2b96800640 1 --2- 172.21.15.134:0/2215632409 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b9010b7b0 0x7f2b901a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.849+0000 7f2b95e00640 1 -- 172.21.15.134:0/2215632409 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b9010b7b0 msgr2=0x7f2b901a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.849+0000 7f2b95e00640 1 --2- 172.21.15.134:0/2215632409 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b9010b7b0 0x7f2b901a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.849+0000 7f2b95e00640 1 -- 172.21.15.134:0/2215632409 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b9010ae60 msgr2=0x7f2b9019c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.849+0000 7f2b95e00640 1 --2- 172.21.15.134:0/2215632409 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b9010ae60 0x7f2b9019c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.849+0000 7f2b95e00640 1 -- 172.21.15.134:0/2215632409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b901a1dd0 con 0x7f2b90105530 2024-07-26T11:15:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.850+0000 7f2b95e00640 1 --2- 172.21.15.134:0/2215632409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b90105530 0x7f2b9019bea0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400e9f0 tx=0x7f2b8400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.850+0000 7f2b8aa00640 1 -- 172.21.15.134:0/2215632409 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b8400ccc0 con 0x7f2b90105530 2024-07-26T11:15:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.850+0000 7f2b97200640 1 -- 172.21.15.134:0/2215632409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b901a20c0 con 0x7f2b90105530 2024-07-26T11:15:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.850+0000 7f2b8aa00640 1 -- 172.21.15.134:0/2215632409 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b840108e0 con 0x7f2b90105530 2024-07-26T11:15:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.850+0000 7f2b8aa00640 1 -- 172.21.15.134:0/2215632409 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b84018880 con 0x7f2b90105530 2024-07-26T11:15:49.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.850+0000 7f2b97200640 1 -- 172.21.15.134:0/2215632409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b901a8850 con 0x7f2b90105530 2024-07-26T11:15:49.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.851+0000 7f2b97200640 1 -- 172.21.15.134:0/2215632409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b90105940 con 0x7f2b90105530 2024-07-26T11:15:49.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.854+0000 7f2b8aa00640 1 -- 172.21.15.134:0/2215632409 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f2b84010430 con 0x7f2b90105530 2024-07-26T11:15:49.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.855+0000 7f2b8aa00640 1 --2- 172.21.15.134:0/2215632409 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b780b3c20 0x7f2b780b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:49.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.855+0000 7f2b8aa00640 1 -- 172.21.15.134:0/2215632409 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2b840e49a0 con 0x7f2b90105530 2024-07-26T11:15:49.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.855+0000 7f2b95400640 1 --2- 172.21.15.134:0/2215632409 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b780b3c20 0x7f2b780b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:49.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.856+0000 7f2b95400640 1 --2- 172.21.15.134:0/2215632409 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b780b3c20 0x7f2b780b6130 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009230 tx=0x7f2b80031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:49.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:49.857+0000 7f2b8aa00640 1 -- 172.21.15.134:0/2215632409 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f2b840abaf0 con 0x7f2b90105530 2024-07-26T11:15:50.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.065+0000 7f2b97200640 1 -- 172.21.15.134:0/2215632409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2b9002e0c0 con 0x7f2b90105530 2024-07-26T11:15:50.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.066+0000 7f2b8aa00640 1 -- 172.21.15.134:0/2215632409 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7f2b840ab3f0 con 0x7f2b90105530 2024-07-26T11:15:50.069 INFO:teuthology.orchestra.run.smithi134.stdout:pg_num: 64 2024-07-26T11:15:50.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.069+0000 7f2b97200640 1 -- 172.21.15.134:0/2215632409 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b780b3c20 msgr2=0x7f2b780b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:50.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.069+0000 7f2b97200640 1 --2- 172.21.15.134:0/2215632409 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b780b3c20 0x7f2b780b6130 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2b80009230 tx=0x7f2b80031400 comp rx=0 tx=0).stop 2024-07-26T11:15:50.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.069+0000 7f2b97200640 1 -- 172.21.15.134:0/2215632409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b90105530 msgr2=0x7f2b9019bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:50.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.069+0000 7f2b97200640 1 --2- 172.21.15.134:0/2215632409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b90105530 0x7f2b9019bea0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400e9f0 tx=0x7f2b8400edf0 comp rx=0 tx=0).stop 2024-07-26T11:15:50.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.070+0000 7f2b97200640 1 -- 172.21.15.134:0/2215632409 shutdown_connections 2024-07-26T11:15:50.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.070+0000 7f2b97200640 1 --2- 172.21.15.134:0/2215632409 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2b780b3c20 0x7f2b780b6130 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:50.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.070+0000 7f2b97200640 1 --2- 172.21.15.134:0/2215632409 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2b9010b7b0 0x7f2b901a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:50.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.070+0000 7f2b97200640 1 --2- 172.21.15.134:0/2215632409 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2b9010ae60 0x7f2b9019c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:50.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.070+0000 7f2b97200640 1 --2- 172.21.15.134:0/2215632409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b90105530 0x7f2b9019bea0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:50.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.070+0000 7f2b97200640 1 -- 172.21.15.134:0/2215632409 >> 172.21.15.134:0/2215632409 conn(0x7f2b90100f90 msgr2=0x7f2b901033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:50.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.070+0000 7f2b97200640 1 -- 172.21.15.134:0/2215632409 shutdown_connections 2024-07-26T11:15:50.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.070+0000 7f2b97200640 1 -- 172.21.15.134:0/2215632409 wait complete. 2024-07-26T11:15:50.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-07-26T11:15:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:49 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1932013665' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T11:15:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:49 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2003314281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-07-26T11:15:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:49 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1932013665' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T11:15:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:49 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2003314281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-07-26T11:15:50.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.374+0000 7efe67200640 1 -- 172.21.15.134:0/3612453925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe6810c980 msgr2=0x7efe681163f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:50.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.374+0000 7efe67200640 1 --2- 172.21.15.134:0/3612453925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe6810c980 0x7efe681163f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efe60009be0 tx=0x7efe6002a6c0 comp rx=0 tx=0).stop 2024-07-26T11:15:50.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.374+0000 7efe67200640 1 -- 172.21.15.134:0/3612453925 shutdown_connections 2024-07-26T11:15:50.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.374+0000 7efe67200640 1 --2- 172.21.15.134:0/3612453925 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe6810c980 0x7efe681163f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:50.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.374+0000 7efe67200640 1 --2- 172.21.15.134:0/3612453925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe6810c030 0x7efe6810c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:50.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.374+0000 7efe67200640 1 --2- 172.21.15.134:0/3612453925 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe68109990 0x7efe68109da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:50.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.374+0000 7efe67200640 1 -- 172.21.15.134:0/3612453925 >> 172.21.15.134:0/3612453925 conn(0x7efe680790b0 msgr2=0x7efe68107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:50.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.374+0000 7efe67200640 1 -- 172.21.15.134:0/3612453925 shutdown_connections 2024-07-26T11:15:50.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.374+0000 7efe67200640 1 -- 172.21.15.134:0/3612453925 wait complete. 2024-07-26T11:15:50.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.375+0000 7efe67200640 1 Processor -- start 2024-07-26T11:15:50.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.375+0000 7efe67200640 1 -- start start 2024-07-26T11:15:50.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.376+0000 7efe67200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe68109990 0x7efe681a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:50.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.376+0000 7efe67200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe6810c030 0x7efe681a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:50.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.376+0000 7efe67200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe6810c980 0x7efe681a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:50.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.376+0000 7efe67200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7efe68119880 con 0x7efe6810c980 2024-07-26T11:15:50.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.376+0000 7efe67200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efe68119700 con 0x7efe68109990 2024-07-26T11:15:50.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.376+0000 7efe67200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7efe68119a00 con 0x7efe6810c030 2024-07-26T11:15:50.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.376+0000 7efe5d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe6810c030 0x7efe681a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:50.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.376+0000 7efe65e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe68109990 0x7efe681a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:50.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.376+0000 7efe66800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe6810c980 0x7efe681a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:50.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.376+0000 7efe5d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe6810c030 0x7efe681a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52350/0 (socket says 172.21.15.134:52350) 2024-07-26T11:15:50.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.376+0000 7efe5d400640 1 -- 172.21.15.134:0/3519424302 learned_addr learned my addr 172.21.15.134:0/3519424302 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:50.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.377+0000 7efe65e00640 1 -- 172.21.15.134:0/3519424302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe6810c030 msgr2=0x7efe681a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:50.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.377+0000 7efe65e00640 1 --2- 172.21.15.134:0/3519424302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe6810c030 0x7efe681a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:50.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.377+0000 7efe65e00640 1 -- 172.21.15.134:0/3519424302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe6810c980 msgr2=0x7efe681a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:50.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.377+0000 7efe65e00640 1 --2- 172.21.15.134:0/3519424302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe6810c980 0x7efe681a59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:50.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.377+0000 7efe65e00640 1 -- 172.21.15.134:0/3519424302 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe681a60f0 con 0x7efe68109990 2024-07-26T11:15:50.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.377+0000 7efe5d400640 1 --2- 172.21.15.134:0/3519424302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe6810c030 0x7efe681a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:50.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.377+0000 7efe65e00640 1 --2- 172.21.15.134:0/3519424302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe68109990 0x7efe681a01c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efe5800e790 tx=0x7efe5800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:50.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.377+0000 7efe5f400640 1 -- 172.21.15.134:0/3519424302 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe5800eea0 con 0x7efe68109990 2024-07-26T11:15:50.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.377+0000 7efe67200640 1 -- 172.21.15.134:0/3519424302 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe681a63e0 con 0x7efe68109990 2024-07-26T11:15:50.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.377+0000 7efe67200640 1 -- 172.21.15.134:0/3519424302 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe681acb70 con 0x7efe68109990 2024-07-26T11:15:50.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.378+0000 7efe5f400640 1 -- 172.21.15.134:0/3519424302 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe5800ccc0 con 0x7efe68109990 2024-07-26T11:15:50.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.378+0000 7efe5f400640 1 -- 172.21.15.134:0/3519424302 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe58018680 con 0x7efe68109990 2024-07-26T11:15:50.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.379+0000 7efe67200640 1 -- 172.21.15.134:0/3519424302 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe440052c0 con 0x7efe68109990 2024-07-26T11:15:50.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.381+0000 7efe5f400640 1 -- 172.21.15.134:0/3519424302 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7efe58005070 con 0x7efe68109990 2024-07-26T11:15:50.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.383+0000 7efe5f400640 1 --2- 172.21.15.134:0/3519424302 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7efe4c0b3c20 0x7efe4c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:50.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.383+0000 7efe5f400640 1 -- 172.21.15.134:0/3519424302 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7efe580e4240 con 0x7efe68109990 2024-07-26T11:15:50.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.383+0000 7efe5d400640 1 --2- 172.21.15.134:0/3519424302 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7efe4c0b3c20 0x7efe4c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:50.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.384+0000 7efe5d400640 1 --2- 172.21.15.134:0/3519424302 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7efe4c0b3c20 0x7efe4c0b6130 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efe68109050 tx=0x7efe50005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:50.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.384+0000 7efe5f400640 1 -- 172.21.15.134:0/3519424302 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7efe580aad40 con 0x7efe68109990 2024-07-26T11:15:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:49 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1932013665' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-07-26T11:15:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:49 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2003314281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-07-26T11:15:50.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.594+0000 7efe67200640 1 -- 172.21.15.134:0/3519424302 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7efe44005880 con 0x7efe68109990 2024-07-26T11:15:50.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.596+0000 7efe5f400640 1 -- 172.21.15.134:0/3519424302 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) ==== 103+0+11 (secure 0 0 0) 0x7efe580af290 con 0x7efe68109990 2024-07-26T11:15:50.598 INFO:teuthology.orchestra.run.smithi134.stdout:pg_num: 32 2024-07-26T11:15:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.598+0000 7efe67200640 1 -- 172.21.15.134:0/3519424302 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7efe4c0b3c20 msgr2=0x7efe4c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.598+0000 7efe67200640 1 --2- 172.21.15.134:0/3519424302 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7efe4c0b3c20 0x7efe4c0b6130 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7efe68109050 tx=0x7efe50005f90 comp rx=0 tx=0).stop 2024-07-26T11:15:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.598+0000 7efe67200640 1 -- 172.21.15.134:0/3519424302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe68109990 msgr2=0x7efe681a01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.598+0000 7efe67200640 1 --2- 172.21.15.134:0/3519424302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe68109990 0x7efe681a01c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efe5800e790 tx=0x7efe5800ec50 comp rx=0 tx=0).stop 2024-07-26T11:15:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.598+0000 7efe67200640 1 -- 172.21.15.134:0/3519424302 shutdown_connections 2024-07-26T11:15:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.598+0000 7efe67200640 1 --2- 172.21.15.134:0/3519424302 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7efe4c0b3c20 0x7efe4c0b6130 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.598+0000 7efe67200640 1 --2- 172.21.15.134:0/3519424302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe6810c030 0x7efe681a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.598+0000 7efe67200640 1 --2- 172.21.15.134:0/3519424302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7efe6810c980 0x7efe681a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.598+0000 7efe67200640 1 --2- 172.21.15.134:0/3519424302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe68109990 0x7efe681a01c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.598+0000 7efe67200640 1 -- 172.21.15.134:0/3519424302 >> 172.21.15.134:0/3519424302 conn(0x7efe680790b0 msgr2=0x7efe6810d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.599+0000 7efe67200640 1 -- 172.21.15.134:0/3519424302 shutdown_connections 2024-07-26T11:15:50.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:50.599+0000 7efe67200640 1 -- 172.21.15.134:0/3519424302 wait complete. 2024-07-26T11:15:50.626 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-07-26T11:15:50.627 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:15:51.084 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:15:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:50 smithi189 ceph-mon[36790]: pgmap v476: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 9 op/s 2024-07-26T11:15:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:50 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2215632409' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-07-26T11:15:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:50 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3519424302' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-07-26T11:15:51.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:50 smithi134 ceph-mon[31966]: pgmap v476: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 9 op/s 2024-07-26T11:15:51.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:50 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2215632409' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-07-26T11:15:51.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:50 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3519424302' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-07-26T11:15:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:50 smithi193 ceph-mon[37473]: pgmap v476: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 9 op/s 2024-07-26T11:15:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:50 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2215632409' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-07-26T11:15:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:50 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3519424302' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-07-26T11:15:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:52 smithi134 ceph-mon[31966]: pgmap v477: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T11:15:52.443 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:52 smithi193 ceph-mon[37473]: pgmap v477: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T11:15:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:52 smithi189 ceph-mon[36790]: pgmap v477: 129 pgs: 129 active+clean; 649 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 9 op/s 2024-07-26T11:15:53.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.467+0000 7fed41a00640 1 -- 172.21.15.134:0/3615727151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed3c109670 msgr2=0x7fed3c10fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:53.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.467+0000 7fed41a00640 1 --2- 172.21.15.134:0/3615727151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed3c109670 0x7fed3c10fea0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fed34009ca0 tx=0x7fed3402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:15:53.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.467+0000 7fed41a00640 1 -- 172.21.15.134:0/3615727151 shutdown_connections 2024-07-26T11:15:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.467+0000 7fed41a00640 1 --2- 172.21.15.134:0/3615727151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed3c109670 0x7fed3c10fea0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.467+0000 7fed41a00640 1 --2- 172.21.15.134:0/3615727151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed3c108d20 0x7fed3c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.467+0000 7fed41a00640 1 --2- 172.21.15.134:0/3615727151 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed3c1033e0 0x7fed3c1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.467+0000 7fed41a00640 1 -- 172.21.15.134:0/3615727151 >> 172.21.15.134:0/3615727151 conn(0x7fed3c0fee70 msgr2=0x7fed3c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.467+0000 7fed41a00640 1 -- 172.21.15.134:0/3615727151 shutdown_connections 2024-07-26T11:15:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.467+0000 7fed41a00640 1 -- 172.21.15.134:0/3615727151 wait complete. 2024-07-26T11:15:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.469+0000 7fed41a00640 1 Processor -- start 2024-07-26T11:15:53.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.469+0000 7fed41a00640 1 -- start start 2024-07-26T11:15:53.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.469+0000 7fed41a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed3c1033e0 0x7fed3c199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:53.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.469+0000 7fed41a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed3c108d20 0x7fed3c19a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:53.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.469+0000 7fed41a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed3c109670 0x7fed3c19f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:53.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.469+0000 7fed41a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c113320 con 0x7fed3c108d20 2024-07-26T11:15:53.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.469+0000 7fed41a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c1131a0 con 0x7fed3c1033e0 2024-07-26T11:15:53.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.469+0000 7fed41a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fed3c1134a0 con 0x7fed3c109670 2024-07-26T11:15:53.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.470+0000 7fed41000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed3c109670 0x7fed3c19f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:53.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.470+0000 7fed41000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed3c109670 0x7fed3c19f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52374/0 (socket says 172.21.15.134:52374) 2024-07-26T11:15:53.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.470+0000 7fed41000640 1 -- 172.21.15.134:0/3985390310 learned_addr learned my addr 172.21.15.134:0/3985390310 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:53.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.470+0000 7fed3b400640 1 --2- 172.21.15.134:0/3985390310 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed3c108d20 0x7fed3c19a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:53.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.470+0000 7fed3be00640 1 --2- 172.21.15.134:0/3985390310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed3c1033e0 0x7fed3c199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:53.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.470+0000 7fed41000640 1 -- 172.21.15.134:0/3985390310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed3c1033e0 msgr2=0x7fed3c199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:53.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.470+0000 7fed41000640 1 --2- 172.21.15.134:0/3985390310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed3c1033e0 0x7fed3c199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:53.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.470+0000 7fed41000640 1 -- 172.21.15.134:0/3985390310 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed3c108d20 msgr2=0x7fed3c19a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:53.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.470+0000 7fed41000640 1 --2- 172.21.15.134:0/3985390310 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed3c108d20 0x7fed3c19a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:53.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.470+0000 7fed41000640 1 -- 172.21.15.134:0/3985390310 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed3c19fbe0 con 0x7fed3c109670 2024-07-26T11:15:53.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.471+0000 7fed41000640 1 --2- 172.21.15.134:0/3985390310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed3c109670 0x7fed3c19f4b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fed34005cc0 tx=0x7fed34031700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:53.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.471+0000 7fed38c00640 1 -- 172.21.15.134:0/3985390310 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed34003b70 con 0x7fed3c109670 2024-07-26T11:15:53.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.471+0000 7fed38c00640 1 -- 172.21.15.134:0/3985390310 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed340318f0 con 0x7fed3c109670 2024-07-26T11:15:53.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.471+0000 7fed41a00640 1 -- 172.21.15.134:0/3985390310 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed3c19fe70 con 0x7fed3c109670 2024-07-26T11:15:53.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.471+0000 7fed38c00640 1 -- 172.21.15.134:0/3985390310 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3402f950 con 0x7fed3c109670 2024-07-26T11:15:53.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.471+0000 7fed41a00640 1 -- 172.21.15.134:0/3985390310 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed3c075020 con 0x7fed3c109670 2024-07-26T11:15:53.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.472+0000 7fed41a00640 1 -- 172.21.15.134:0/3985390310 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed3c1037f0 con 0x7fed3c109670 2024-07-26T11:15:53.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.475+0000 7fed38c00640 1 -- 172.21.15.134:0/3985390310 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fed34004b40 con 0x7fed3c109670 2024-07-26T11:15:53.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.476+0000 7fed38c00640 1 --2- 172.21.15.134:0/3985390310 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fed200b3ca0 0x7fed200b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:53.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.476+0000 7fed38c00640 1 -- 172.21.15.134:0/3985390310 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fed34105700 con 0x7fed3c109670 2024-07-26T11:15:53.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.477+0000 7fed3be00640 1 --2- 172.21.15.134:0/3985390310 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fed200b3ca0 0x7fed200b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:53.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.477+0000 7fed3be00640 1 --2- 172.21.15.134:0/3985390310 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fed200b3ca0 0x7fed200b61b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fed2c010d80 tx=0x7fed2c012040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:53.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.477+0000 7fed38c00640 1 -- 172.21.15.134:0/3985390310 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fed340cc8b0 con 0x7fed3c109670 2024-07-26T11:15:53.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.718+0000 7fed41a00640 1 -- 172.21.15.134:0/3985390310 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed3c10b460 con 0x7fed3c109670 2024-07-26T11:15:53.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.719+0000 7fed38c00640 1 -- 172.21.15.134:0/3985390310 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11990 (secure 0 0 0) 0x7fed340cc1b0 con 0x7fed3c109670 2024-07-26T11:15:53.721 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:15:53.721 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":30,"btime":"2024-07-26T11:15:47:899566+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:15:47.849285+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24721,"mds_1":24719,"mds_2":24703,"mds_3":24688,"mds_4":24686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24635":{"gid":24635,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24641":{"gid":24641,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":32,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24719":{"gid":24719,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":28,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24721,"qdb_cluster":[24721,24719,24703,24688,24686]},"id":1}]} 2024-07-26T11:15:53.722 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 30 2024-07-26T11:15:53.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.722+0000 7fed41a00640 1 -- 172.21.15.134:0/3985390310 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fed200b3ca0 msgr2=0x7fed200b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:53.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.722+0000 7fed41a00640 1 --2- 172.21.15.134:0/3985390310 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fed200b3ca0 0x7fed200b61b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fed2c010d80 tx=0x7fed2c012040 comp rx=0 tx=0).stop 2024-07-26T11:15:53.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.722+0000 7fed41a00640 1 -- 172.21.15.134:0/3985390310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed3c109670 msgr2=0x7fed3c19f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:53.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.722+0000 7fed41a00640 1 --2- 172.21.15.134:0/3985390310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed3c109670 0x7fed3c19f4b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fed34005cc0 tx=0x7fed34031700 comp rx=0 tx=0).stop 2024-07-26T11:15:53.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.722+0000 7fed41a00640 1 -- 172.21.15.134:0/3985390310 shutdown_connections 2024-07-26T11:15:53.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.722+0000 7fed41a00640 1 --2- 172.21.15.134:0/3985390310 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fed200b3ca0 0x7fed200b61b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:53.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.722+0000 7fed41a00640 1 --2- 172.21.15.134:0/3985390310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fed3c109670 0x7fed3c19f4b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:53.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.722+0000 7fed41a00640 1 --2- 172.21.15.134:0/3985390310 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fed3c108d20 0x7fed3c19a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:53.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.722+0000 7fed41a00640 1 --2- 172.21.15.134:0/3985390310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed3c1033e0 0x7fed3c199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:53.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.722+0000 7fed41a00640 1 -- 172.21.15.134:0/3985390310 >> 172.21.15.134:0/3985390310 conn(0x7fed3c0fee70 msgr2=0x7fed3c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:53.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.722+0000 7fed41a00640 1 -- 172.21.15.134:0/3985390310 shutdown_connections 2024-07-26T11:15:53.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:53.722+0000 7fed41a00640 1 -- 172.21.15.134:0/3985390310 wait complete. 2024-07-26T11:15:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:53 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3985390310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:15:54.165 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:53 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3985390310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:15:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:53 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3985390310' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:15:54.654 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-07-26T11:15:54.654 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-07-26T11:15:54.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T11:15:54.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.906+0000 7f9ac9000640 1 -- 172.21.15.134:0/326473661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ac410ae60 msgr2=0x7f9ac410b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:54.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.906+0000 7f9ac9000640 1 --2- 172.21.15.134:0/326473661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ac410ae60 0x7f9ac410b270 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009860 tx=0x7f9aac02a340 comp rx=0 tx=0).stop 2024-07-26T11:15:54.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.906+0000 7f9ac9000640 1 -- 172.21.15.134:0/326473661 shutdown_connections 2024-07-26T11:15:54.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.906+0000 7f9ac9000640 1 --2- 172.21.15.134:0/326473661 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ac410b7b0 0x7f9ac4111fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:54.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.906+0000 7f9ac9000640 1 --2- 172.21.15.134:0/326473661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ac410ae60 0x7f9ac410b270 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:54.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.906+0000 7f9ac9000640 1 --2- 172.21.15.134:0/326473661 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ac4105530 0x7f9ac4105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:54.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.906+0000 7f9ac9000640 1 -- 172.21.15.134:0/326473661 >> 172.21.15.134:0/326473661 conn(0x7f9ac4100f90 msgr2=0x7f9ac41033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:54.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.906+0000 7f9ac9000640 1 -- 172.21.15.134:0/326473661 shutdown_connections 2024-07-26T11:15:54.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.906+0000 7f9ac9000640 1 -- 172.21.15.134:0/326473661 wait complete. 2024-07-26T11:15:54.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.907+0000 7f9ac9000640 1 Processor -- start 2024-07-26T11:15:54.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.907+0000 7f9ac9000640 1 -- start start 2024-07-26T11:15:54.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.908+0000 7f9ac9000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ac4105530 0x7f9ac419bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:54.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.908+0000 7f9ac9000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ac410ae60 0x7f9ac419c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:54.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.908+0000 7f9ac9000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ac410b7b0 0x7f9ac41a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:54.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.908+0000 7f9ac9000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac4115440 con 0x7f9ac410ae60 2024-07-26T11:15:54.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.908+0000 7f9ac9000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac41152c0 con 0x7f9ac4105530 2024-07-26T11:15:54.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.908+0000 7f9ac9000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac41155c0 con 0x7f9ac410b7b0 2024-07-26T11:15:54.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.908+0000 7f9ac3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ac410b7b0 0x7f9ac41a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:54.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.908+0000 7f9abbe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ac410ae60 0x7f9ac419c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:54.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.908+0000 7f9ac3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ac4105530 0x7f9ac419bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:54.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.908+0000 7f9ac3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ac410b7b0 0x7f9ac41a16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52384/0 (socket says 172.21.15.134:52384) 2024-07-26T11:15:54.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.908+0000 7f9ac3e00640 1 -- 172.21.15.134:0/4263244391 learned_addr learned my addr 172.21.15.134:0/4263244391 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:54.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.909+0000 7f9abbe00640 1 -- 172.21.15.134:0/4263244391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ac410b7b0 msgr2=0x7f9ac41a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:54.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.909+0000 7f9abbe00640 1 --2- 172.21.15.134:0/4263244391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ac410b7b0 0x7f9ac41a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:54.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.909+0000 7f9abbe00640 1 -- 172.21.15.134:0/4263244391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ac4105530 msgr2=0x7f9ac419bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:54.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.909+0000 7f9abbe00640 1 --2- 172.21.15.134:0/4263244391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ac4105530 0x7f9ac419bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:54.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.909+0000 7f9abbe00640 1 -- 172.21.15.134:0/4263244391 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac41a1dd0 con 0x7f9ac410ae60 2024-07-26T11:15:54.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.909+0000 7f9abbe00640 1 --2- 172.21.15.134:0/4263244391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ac410ae60 0x7f9ac419c440 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009780 tx=0x7f9aac032d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:54.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.909+0000 7f9ac0c00640 1 -- 172.21.15.134:0/4263244391 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac037040 con 0x7f9ac410ae60 2024-07-26T11:15:54.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.909+0000 7f9ac0c00640 1 -- 172.21.15.134:0/4263244391 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9aac030c10 con 0x7f9ac410ae60 2024-07-26T11:15:54.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.909+0000 7f9ac9000640 1 -- 172.21.15.134:0/4263244391 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac41a2060 con 0x7f9ac410ae60 2024-07-26T11:15:54.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.909+0000 7f9ac0c00640 1 -- 172.21.15.134:0/4263244391 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aac0035f0 con 0x7f9ac410ae60 2024-07-26T11:15:54.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.909+0000 7f9ac9000640 1 -- 172.21.15.134:0/4263244391 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac41a24c0 con 0x7f9ac410ae60 2024-07-26T11:15:54.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.911+0000 7f9ac9000640 1 -- 172.21.15.134:0/4263244391 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac4105940 con 0x7f9ac410ae60 2024-07-26T11:15:54.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.913+0000 7f9ac0c00640 1 -- 172.21.15.134:0/4263244391 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f9aac043020 con 0x7f9ac410ae60 2024-07-26T11:15:54.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.916+0000 7f9ac0c00640 1 --2- 172.21.15.134:0/4263244391 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9aa80b3c20 0x7f9aa80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:54.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.916+0000 7f9ac0c00640 1 -- 172.21.15.134:0/4263244391 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f9aac103870 con 0x7f9ac410ae60 2024-07-26T11:15:54.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.916+0000 7f9ac3400640 1 --2- 172.21.15.134:0/4263244391 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9aa80b3c20 0x7f9aa80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:54.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.917+0000 7f9ac3400640 1 --2- 172.21.15.134:0/4263244391 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9aa80b3c20 0x7f9aa80b6130 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4009530 tx=0x7f9ab4009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:54.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:54.918+0000 7f9ac0c00640 1 -- 172.21.15.134:0/4263244391 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f9aac0caa40 con 0x7f9ac410ae60 2024-07-26T11:15:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:54 smithi134 ceph-mon[31966]: pgmap v478: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-26T11:15:55.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.112+0000 7f9ac9000640 1 -- 172.21.15.134:0/4263244391 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9ac410d590 con 0x7f9aa80b3c20 2024-07-26T11:15:55.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.119+0000 7f9ac0c00640 1 -- 172.21.15.134:0/4263244391 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340544 (secure 0 0 0) 0x7f9ac410d590 con 0x7f9aa80b3c20 2024-07-26T11:15:55.128 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:15:55.130 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-07-26T11:15:55.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.127+0000 7f9ac9000640 1 -- 172.21.15.134:0/4263244391 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9aa80b3c20 msgr2=0x7f9aa80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:55.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.127+0000 7f9ac9000640 1 --2- 172.21.15.134:0/4263244391 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9aa80b3c20 0x7f9aa80b6130 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4009530 tx=0x7f9ab4009260 comp rx=0 tx=0).stop 2024-07-26T11:15:55.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.127+0000 7f9ac9000640 1 -- 172.21.15.134:0/4263244391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ac410ae60 msgr2=0x7f9ac419c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:55.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.127+0000 7f9ac9000640 1 --2- 172.21.15.134:0/4263244391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ac410ae60 0x7f9ac419c440 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9aac009780 tx=0x7f9aac032d50 comp rx=0 tx=0).stop 2024-07-26T11:15:55.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.127+0000 7f9ac9000640 1 -- 172.21.15.134:0/4263244391 shutdown_connections 2024-07-26T11:15:55.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.127+0000 7f9ac9000640 1 --2- 172.21.15.134:0/4263244391 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9aa80b3c20 0x7f9aa80b6130 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:55.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.127+0000 7f9ac9000640 1 --2- 172.21.15.134:0/4263244391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9ac410b7b0 0x7f9ac41a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:55.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.127+0000 7f9ac9000640 1 --2- 172.21.15.134:0/4263244391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9ac410ae60 0x7f9ac419c440 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:55.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.127+0000 7f9ac9000640 1 --2- 172.21.15.134:0/4263244391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ac4105530 0x7f9ac419bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:55.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.127+0000 7f9ac9000640 1 -- 172.21.15.134:0/4263244391 >> 172.21.15.134:0/4263244391 conn(0x7f9ac4100f90 msgr2=0x7f9ac41033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:55.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.128+0000 7f9ac9000640 1 -- 172.21.15.134:0/4263244391 shutdown_connections 2024-07-26T11:15:55.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.128+0000 7f9ac9000640 1 -- 172.21.15.134:0/4263244391 wait complete. 2024-07-26T11:15:55.157 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-07-26T11:15:54.049405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":670312,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":2534,"num_read_kb":1385,"num_write":390,"num_write_kb":2314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":2954034,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":822,"ondisk_log_size":822,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375892,"kb_used_data":14260,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565932,"statfs":{"total":1151940427776,"available":1151555514368,"internally_reserved":0,"allocated":14602240,"data_stored":8285129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5987,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":73,"num_write":7,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006132"},"pg_stats":[{"pgid":"2.1f","version":"90'13","reported_seq":310,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:49.585215+0000","last_change":"2024-07-26T11:10:34.985984+0000","last_active":"2024-07-26T11:15:49.585215+0000","last_peered":"2024-07-26T11:15:49.585215+0000","last_clean":"2024-07-26T11:15:49.585215+0000","last_became_active":"2024-07-26T11:10:34.980894+0000","last_became_peered":"2024-07-26T11:10:34.980894+0000","last_unstale":"2024-07-26T11:15:49.585215+0000","last_undegraded":"2024-07-26T11:15:49.585215+0000","last_fullsized":"2024-07-26T11:15:49.585215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:55.718875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":241,"num_read_kb":241,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874388+0000","last_change":"2024-07-26T11:10:22.320946+0000","last_active":"2024-07-26T11:13:33.874388+0000","last_peered":"2024-07-26T11:13:33.874388+0000","last_clean":"2024-07-26T11:13:33.874388+0000","last_became_active":"2024-07-26T11:10:22.316181+0000","last_became_peered":"2024-07-26T11:10:22.316181+0000","last_unstale":"2024-07-26T11:13:33.874388+0000","last_undegraded":"2024-07-26T11:13:33.874388+0000","last_fullsized":"2024-07-26T11:13:33.874388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:13.641862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:04.915763+0000","last_change":"2024-07-26T11:10:35.984460+0000","last_active":"2024-07-26T11:13:04.915763+0000","last_peered":"2024-07-26T11:13:04.915763+0000","last_clean":"2024-07-26T11:13:04.915763+0000","last_became_active":"2024-07-26T11:10:35.983922+0000","last_became_peered":"2024-07-26T11:10:35.983922+0000","last_unstale":"2024-07-26T11:13:04.915763+0000","last_undegraded":"2024-07-26T11:13:04.915763+0000","last_fullsized":"2024-07-26T11:13:04.915763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:51:46.140910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:19.061450+0000","last_change":"2024-07-26T11:10:18.058553+0000","last_active":"2024-07-26T11:12:19.061450+0000","last_peered":"2024-07-26T11:12:19.061450+0000","last_clean":"2024-07-26T11:12:19.061450+0000","last_became_active":"2024-07-26T11:10:18.058034+0000","last_became_peered":"2024-07-26T11:10:18.058034+0000","last_unstale":"2024-07-26T11:12:19.061450+0000","last_undegraded":"2024-07-26T11:12:19.061450+0000","last_fullsized":"2024-07-26T11:12:19.061450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:48:58.868988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.893963+0000","last_change":"2024-07-26T11:10:22.326760+0000","last_active":"2024-07-26T11:13:33.893963+0000","last_peered":"2024-07-26T11:13:33.893963+0000","last_clean":"2024-07-26T11:13:33.893963+0000","last_became_active":"2024-07-26T11:10:22.326275+0000","last_became_peered":"2024-07-26T11:10:22.326275+0000","last_unstale":"2024-07-26T11:13:33.893963+0000","last_undegraded":"2024-07-26T11:13:33.893963+0000","last_fullsized":"2024-07-26T11:13:33.893963+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:51:57.461453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:04.941996+0000","last_change":"2024-07-26T11:10:35.997585+0000","last_active":"2024-07-26T11:13:04.941996+0000","last_peered":"2024-07-26T11:13:04.941996+0000","last_clean":"2024-07-26T11:13:04.941996+0000","last_became_active":"2024-07-26T11:10:35.997274+0000","last_became_peered":"2024-07-26T11:10:35.997274+0000","last_unstale":"2024-07-26T11:13:04.941996+0000","last_undegraded":"2024-07-26T11:13:04.941996+0000","last_fullsized":"2024-07-26T11:13:04.941996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:43.553396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:47.162843+0000","last_change":"2024-07-26T11:10:18.034964+0000","last_active":"2024-07-26T11:12:47.162843+0000","last_peered":"2024-07-26T11:12:47.162843+0000","last_clean":"2024-07-26T11:12:47.162843+0000","last_became_active":"2024-07-26T11:10:18.034569+0000","last_became_peered":"2024-07-26T11:10:18.034569+0000","last_unstale":"2024-07-26T11:12:47.162843+0000","last_undegraded":"2024-07-26T11:12:47.162843+0000","last_fullsized":"2024-07-26T11:12:47.162843+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:00:46.542458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.882555+0000","last_change":"2024-07-26T11:10:22.310695+0000","last_active":"2024-07-26T11:13:33.882555+0000","last_peered":"2024-07-26T11:13:33.882555+0000","last_clean":"2024-07-26T11:13:33.882555+0000","last_became_active":"2024-07-26T11:10:22.310142+0000","last_became_peered":"2024-07-26T11:10:22.310142+0000","last_unstale":"2024-07-26T11:13:33.882555+0000","last_undegraded":"2024-07-26T11:13:33.882555+0000","last_fullsized":"2024-07-26T11:13:33.882555+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:49:18.890802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:09.942670+0000","last_change":"2024-07-26T11:10:35.991418+0000","last_active":"2024-07-26T11:13:09.942670+0000","last_peered":"2024-07-26T11:13:09.942670+0000","last_clean":"2024-07-26T11:13:09.942670+0000","last_became_active":"2024-07-26T11:10:35.990934+0000","last_became_peered":"2024-07-26T11:10:35.990934+0000","last_unstale":"2024-07-26T11:13:09.942670+0000","last_undegraded":"2024-07-26T11:13:09.942670+0000","last_fullsized":"2024-07-26T11:13:09.942670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:56.598738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:07.512173+0000","last_change":"2024-07-26T11:10:34.965637+0000","last_active":"2024-07-26T11:13:07.512173+0000","last_peered":"2024-07-26T11:13:07.512173+0000","last_clean":"2024-07-26T11:13:07.512173+0000","last_became_active":"2024-07-26T11:10:34.961157+0000","last_became_peered":"2024-07-26T11:10:34.961157+0000","last_unstale":"2024-07-26T11:13:07.512173+0000","last_undegraded":"2024-07-26T11:13:07.512173+0000","last_fullsized":"2024-07-26T11:13:07.512173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:26:17.007258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875528+0000","last_change":"2024-07-26T11:10:22.303606+0000","last_active":"2024-07-26T11:13:33.875528+0000","last_peered":"2024-07-26T11:13:33.875528+0000","last_clean":"2024-07-26T11:13:33.875528+0000","last_became_active":"2024-07-26T11:10:22.303183+0000","last_became_peered":"2024-07-26T11:10:22.303183+0000","last_unstale":"2024-07-26T11:13:33.875528+0000","last_undegraded":"2024-07-26T11:13:33.875528+0000","last_fullsized":"2024-07-26T11:13:33.875528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:31:01.153694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:09.942648+0000","last_change":"2024-07-26T11:10:35.981446+0000","last_active":"2024-07-26T11:13:09.942648+0000","last_peered":"2024-07-26T11:13:09.942648+0000","last_clean":"2024-07-26T11:13:09.942648+0000","last_became_active":"2024-07-26T11:10:35.980994+0000","last_became_peered":"2024-07-26T11:10:35.980994+0000","last_unstale":"2024-07-26T11:13:09.942648+0000","last_undegraded":"2024-07-26T11:13:09.942648+0000","last_fullsized":"2024-07-26T11:13:09.942648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:38:57.405325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.887229+0000","last_change":"2024-07-26T11:10:22.306758+0000","last_active":"2024-07-26T11:13:33.887229+0000","last_peered":"2024-07-26T11:13:33.887229+0000","last_clean":"2024-07-26T11:13:33.887229+0000","last_became_active":"2024-07-26T11:10:22.306370+0000","last_became_peered":"2024-07-26T11:10:22.306370+0000","last_unstale":"2024-07-26T11:13:33.887229+0000","last_undegraded":"2024-07-26T11:13:33.887229+0000","last_fullsized":"2024-07-26T11:13:33.887229+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:51:43.069470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:32.160732+0000","last_change":"2024-07-26T11:10:18.035664+0000","last_active":"2024-07-26T11:12:32.160732+0000","last_peered":"2024-07-26T11:12:32.160732+0000","last_clean":"2024-07-26T11:12:32.160732+0000","last_became_active":"2024-07-26T11:10:18.035113+0000","last_became_peered":"2024-07-26T11:10:18.035113+0000","last_unstale":"2024-07-26T11:12:32.160732+0000","last_undegraded":"2024-07-26T11:12:32.160732+0000","last_fullsized":"2024-07-26T11:12:32.160732+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:09:30.641034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:54.940745+0000","last_change":"2024-07-26T11:10:35.990907+0000","last_active":"2024-07-26T11:12:54.940745+0000","last_peered":"2024-07-26T11:12:54.940745+0000","last_clean":"2024-07-26T11:12:54.940745+0000","last_became_active":"2024-07-26T11:10:35.990417+0000","last_became_peered":"2024-07-26T11:10:35.990417+0000","last_unstale":"2024-07-26T11:12:54.940745+0000","last_undegraded":"2024-07-26T11:12:54.940745+0000","last_fullsized":"2024-07-26T11:12:54.940745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:24:39.319727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889158+0000","last_change":"2024-07-26T11:10:22.333575+0000","last_active":"2024-07-26T11:13:33.889158+0000","last_peered":"2024-07-26T11:13:33.889158+0000","last_clean":"2024-07-26T11:13:33.889158+0000","last_became_active":"2024-07-26T11:10:22.332802+0000","last_became_peered":"2024-07-26T11:10:22.332802+0000","last_unstale":"2024-07-26T11:13:33.889158+0000","last_undegraded":"2024-07-26T11:13:33.889158+0000","last_fullsized":"2024-07-26T11:13:33.889158+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:14:03.238840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:29.911881+0000","last_change":"2024-07-26T11:10:18.047942+0000","last_active":"2024-07-26T11:12:29.911881+0000","last_peered":"2024-07-26T11:12:29.911881+0000","last_clean":"2024-07-26T11:12:29.911881+0000","last_became_active":"2024-07-26T11:10:18.047500+0000","last_became_peered":"2024-07-26T11:10:18.047500+0000","last_unstale":"2024-07-26T11:12:29.911881+0000","last_undegraded":"2024-07-26T11:12:29.911881+0000","last_fullsized":"2024-07-26T11:12:29.911881+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:32:30.773648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:54.940721+0000","last_change":"2024-07-26T11:10:36.000946+0000","last_active":"2024-07-26T11:12:54.940721+0000","last_peered":"2024-07-26T11:12:54.940721+0000","last_clean":"2024-07-26T11:12:54.940721+0000","last_became_active":"2024-07-26T11:10:36.000251+0000","last_became_peered":"2024-07-26T11:10:36.000251+0000","last_unstale":"2024-07-26T11:12:54.940721+0000","last_undegraded":"2024-07-26T11:12:54.940721+0000","last_fullsized":"2024-07-26T11:12:54.940721+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:03:40.523747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.890526+0000","last_change":"2024-07-26T11:10:22.326370+0000","last_active":"2024-07-26T11:13:33.890526+0000","last_peered":"2024-07-26T11:13:33.890526+0000","last_clean":"2024-07-26T11:13:33.890526+0000","last_became_active":"2024-07-26T11:10:22.325961+0000","last_became_peered":"2024-07-26T11:10:22.325961+0000","last_unstale":"2024-07-26T11:13:33.890526+0000","last_undegraded":"2024-07-26T11:13:33.890526+0000","last_fullsized":"2024-07-26T11:13:33.890526+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:16:36.946750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:29.063642+0000","last_change":"2024-07-26T11:10:18.051940+0000","last_active":"2024-07-26T11:12:29.063642+0000","last_peered":"2024-07-26T11:12:29.063642+0000","last_clean":"2024-07-26T11:12:29.063642+0000","last_became_active":"2024-07-26T11:10:18.051565+0000","last_became_peered":"2024-07-26T11:10:18.051565+0000","last_unstale":"2024-07-26T11:12:29.063642+0000","last_undegraded":"2024-07-26T11:12:29.063642+0000","last_fullsized":"2024-07-26T11:12:29.063642+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:17:56.418638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:51.522449+0000","last_change":"2024-07-26T11:10:36.000156+0000","last_active":"2024-07-26T11:12:51.522449+0000","last_peered":"2024-07-26T11:12:51.522449+0000","last_clean":"2024-07-26T11:12:51.522449+0000","last_became_active":"2024-07-26T11:10:35.999600+0000","last_became_peered":"2024-07-26T11:10:35.999600+0000","last_unstale":"2024-07-26T11:12:51.522449+0000","last_undegraded":"2024-07-26T11:12:51.522449+0000","last_fullsized":"2024-07-26T11:12:51.522449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:35:48.504288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.858816+0000","last_change":"2024-07-26T11:10:22.298152+0000","last_active":"2024-07-26T11:13:33.858816+0000","last_peered":"2024-07-26T11:13:33.858816+0000","last_clean":"2024-07-26T11:13:33.858816+0000","last_became_active":"2024-07-26T11:10:22.297739+0000","last_became_peered":"2024-07-26T11:10:22.297739+0000","last_unstale":"2024-07-26T11:13:33.858816+0000","last_undegraded":"2024-07-26T11:13:33.858816+0000","last_fullsized":"2024-07-26T11:13:33.858816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:48.160183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:28.950352+0000","last_change":"2024-07-26T11:10:18.057756+0000","last_active":"2024-07-26T11:12:28.950352+0000","last_peered":"2024-07-26T11:12:28.950352+0000","last_clean":"2024-07-26T11:12:28.950352+0000","last_became_active":"2024-07-26T11:10:18.057464+0000","last_became_peered":"2024-07-26T11:10:18.057464+0000","last_unstale":"2024-07-26T11:12:28.950352+0000","last_undegraded":"2024-07-26T11:12:28.950352+0000","last_fullsized":"2024-07-26T11:12:28.950352+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:39:11.523643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:54.940757+0000","last_change":"2024-07-26T11:10:36.000617+0000","last_active":"2024-07-26T11:12:54.940757+0000","last_peered":"2024-07-26T11:12:54.940757+0000","last_clean":"2024-07-26T11:12:54.940757+0000","last_became_active":"2024-07-26T11:10:36.000151+0000","last_became_peered":"2024-07-26T11:10:36.000151+0000","last_unstale":"2024-07-26T11:12:54.940757+0000","last_undegraded":"2024-07-26T11:12:54.940757+0000","last_fullsized":"2024-07-26T11:12:54.940757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:36.942215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.893390+0000","last_change":"2024-07-26T11:10:22.336452+0000","last_active":"2024-07-26T11:13:33.893390+0000","last_peered":"2024-07-26T11:13:33.893390+0000","last_clean":"2024-07-26T11:13:33.893390+0000","last_became_active":"2024-07-26T11:10:22.336006+0000","last_became_peered":"2024-07-26T11:10:22.336006+0000","last_unstale":"2024-07-26T11:13:33.893390+0000","last_undegraded":"2024-07-26T11:13:33.893390+0000","last_fullsized":"2024-07-26T11:13:33.893390+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:08:27.375730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:29.937948+0000","last_change":"2024-07-26T11:10:18.823017+0000","last_active":"2024-07-26T11:12:29.937948+0000","last_peered":"2024-07-26T11:12:29.937948+0000","last_clean":"2024-07-26T11:12:29.937948+0000","last_became_active":"2024-07-26T11:10:18.822559+0000","last_became_peered":"2024-07-26T11:10:18.822559+0000","last_unstale":"2024-07-26T11:12:29.937948+0000","last_undegraded":"2024-07-26T11:12:29.937948+0000","last_fullsized":"2024-07-26T11:12:29.937948+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:01.486263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:44.913501+0000","last_change":"2024-07-26T11:10:35.969698+0000","last_active":"2024-07-26T11:12:44.913501+0000","last_peered":"2024-07-26T11:12:44.913501+0000","last_clean":"2024-07-26T11:12:44.913501+0000","last_became_active":"2024-07-26T11:10:35.969240+0000","last_became_peered":"2024-07-26T11:10:35.969240+0000","last_unstale":"2024-07-26T11:12:44.913501+0000","last_undegraded":"2024-07-26T11:12:44.913501+0000","last_fullsized":"2024-07-26T11:12:44.913501+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:25.210862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875375+0000","last_change":"2024-07-26T11:10:22.314772+0000","last_active":"2024-07-26T11:13:33.875375+0000","last_peered":"2024-07-26T11:13:33.875375+0000","last_clean":"2024-07-26T11:13:33.875375+0000","last_became_active":"2024-07-26T11:10:22.313786+0000","last_became_peered":"2024-07-26T11:10:22.313786+0000","last_unstale":"2024-07-26T11:13:33.875375+0000","last_undegraded":"2024-07-26T11:13:33.875375+0000","last_fullsized":"2024-07-26T11:13:33.875375+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:19.610695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'18","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:14:41.543437+0000","last_change":"2024-07-26T11:10:18.049408+0000","last_active":"2024-07-26T11:14:41.543437+0000","last_peered":"2024-07-26T11:14:41.543437+0000","last_clean":"2024-07-26T11:14:41.543437+0000","last_became_active":"2024-07-26T11:10:18.048573+0000","last_became_peered":"2024-07-26T11:10:18.048573+0000","last_unstale":"2024-07-26T11:14:41.543437+0000","last_undegraded":"2024-07-26T11:14:41.543437+0000","last_fullsized":"2024-07-26T11:14:41.543437+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:26.408260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:39.065841+0000","last_change":"2024-07-26T11:10:35.992067+0000","last_active":"2024-07-26T11:12:39.065841+0000","last_peered":"2024-07-26T11:12:39.065841+0000","last_clean":"2024-07-26T11:12:39.065841+0000","last_became_active":"2024-07-26T11:10:35.989789+0000","last_became_peered":"2024-07-26T11:10:35.989789+0000","last_unstale":"2024-07-26T11:12:39.065841+0000","last_undegraded":"2024-07-26T11:12:39.065841+0000","last_fullsized":"2024-07-26T11:12:39.065841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:08:08.283592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.888864+0000","last_change":"2024-07-26T11:10:22.323512+0000","last_active":"2024-07-26T11:13:33.888864+0000","last_peered":"2024-07-26T11:13:33.888864+0000","last_clean":"2024-07-26T11:13:33.888864+0000","last_became_active":"2024-07-26T11:10:22.323126+0000","last_became_peered":"2024-07-26T11:10:22.323126+0000","last_unstale":"2024-07-26T11:13:33.888864+0000","last_undegraded":"2024-07-26T11:13:33.888864+0000","last_fullsized":"2024-07-26T11:13:33.888864+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:27:15.379345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:22.159418+0000","last_change":"2024-07-26T11:10:18.052573+0000","last_active":"2024-07-26T11:12:22.159418+0000","last_peered":"2024-07-26T11:12:22.159418+0000","last_clean":"2024-07-26T11:12:22.159418+0000","last_became_active":"2024-07-26T11:10:18.052302+0000","last_became_peered":"2024-07-26T11:10:18.052302+0000","last_unstale":"2024-07-26T11:12:22.159418+0000","last_undegraded":"2024-07-26T11:12:22.159418+0000","last_fullsized":"2024-07-26T11:12:22.159418+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:32.260966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:42.508751+0000","last_change":"2024-07-26T11:10:35.994347+0000","last_active":"2024-07-26T11:12:42.508751+0000","last_peered":"2024-07-26T11:12:42.508751+0000","last_clean":"2024-07-26T11:12:42.508751+0000","last_became_active":"2024-07-26T11:10:35.993795+0000","last_became_peered":"2024-07-26T11:10:35.993795+0000","last_unstale":"2024-07-26T11:12:42.508751+0000","last_undegraded":"2024-07-26T11:12:42.508751+0000","last_fullsized":"2024-07-26T11:12:42.508751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:35.399601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.881857+0000","last_change":"2024-07-26T11:10:22.318109+0000","last_active":"2024-07-26T11:13:33.881857+0000","last_peered":"2024-07-26T11:13:33.881857+0000","last_clean":"2024-07-26T11:13:33.881857+0000","last_became_active":"2024-07-26T11:10:22.317674+0000","last_became_peered":"2024-07-26T11:10:22.317674+0000","last_unstale":"2024-07-26T11:13:33.881857+0000","last_undegraded":"2024-07-26T11:13:33.881857+0000","last_fullsized":"2024-07-26T11:13:33.881857+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:29.464772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:26.517734+0000","last_change":"2024-07-26T11:10:18.060073+0000","last_active":"2024-07-26T11:12:26.517734+0000","last_peered":"2024-07-26T11:12:26.517734+0000","last_clean":"2024-07-26T11:12:26.517734+0000","last_became_active":"2024-07-26T11:10:18.059381+0000","last_became_peered":"2024-07-26T11:10:18.059381+0000","last_unstale":"2024-07-26T11:12:26.517734+0000","last_undegraded":"2024-07-26T11:12:26.517734+0000","last_fullsized":"2024-07-26T11:12:26.517734+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:04:21.184759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:42.162020+0000","last_change":"2024-07-26T11:11:35.951234+0000","last_active":"2024-07-26T11:12:42.162020+0000","last_peered":"2024-07-26T11:12:42.162020+0000","last_clean":"2024-07-26T11:12:42.162020+0000","last_became_active":"2024-07-26T11:11:35.950821+0000","last_became_peered":"2024-07-26T11:11:35.950821+0000","last_unstale":"2024-07-26T11:12:42.162020+0000","last_undegraded":"2024-07-26T11:12:42.162020+0000","last_fullsized":"2024-07-26T11:12:42.162020+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:44:22.569238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.873134+0000","last_change":"2024-07-26T11:10:22.326866+0000","last_active":"2024-07-26T11:13:33.873134+0000","last_peered":"2024-07-26T11:13:33.873134+0000","last_clean":"2024-07-26T11:13:33.873134+0000","last_became_active":"2024-07-26T11:10:22.326417+0000","last_became_peered":"2024-07-26T11:10:22.326417+0000","last_unstale":"2024-07-26T11:13:33.873134+0000","last_undegraded":"2024-07-26T11:13:33.873134+0000","last_fullsized":"2024-07-26T11:13:33.873134+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:49:02.871062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":290,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:50.588141+0000","last_change":"2024-07-26T11:10:18.044077+0000","last_active":"2024-07-26T11:15:50.588141+0000","last_peered":"2024-07-26T11:15:50.588141+0000","last_clean":"2024-07-26T11:15:50.588141+0000","last_became_active":"2024-07-26T11:10:18.043660+0000","last_became_peered":"2024-07-26T11:10:18.043660+0000","last_unstale":"2024-07-26T11:15:50.588141+0000","last_undegraded":"2024-07-26T11:15:50.588141+0000","last_fullsized":"2024-07-26T11:15:50.588141+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:46:54.901082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:41.781727+0000","last_change":"2024-07-26T11:10:35.984778+0000","last_active":"2024-07-26T11:12:41.781727+0000","last_peered":"2024-07-26T11:12:41.781727+0000","last_clean":"2024-07-26T11:12:41.781727+0000","last_became_active":"2024-07-26T11:10:35.984365+0000","last_became_peered":"2024-07-26T11:10:35.984365+0000","last_unstale":"2024-07-26T11:12:41.781727+0000","last_undegraded":"2024-07-26T11:12:41.781727+0000","last_fullsized":"2024-07-26T11:12:41.781727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:33.174121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.867814+0000","last_change":"2024-07-26T11:10:34.976075+0000","last_active":"2024-07-26T11:13:33.867814+0000","last_peered":"2024-07-26T11:13:33.867814+0000","last_clean":"2024-07-26T11:13:33.867814+0000","last_became_active":"2024-07-26T11:10:34.973799+0000","last_became_peered":"2024-07-26T11:10:34.973799+0000","last_unstale":"2024-07-26T11:13:33.867814+0000","last_undegraded":"2024-07-26T11:13:33.867814+0000","last_fullsized":"2024-07-26T11:13:33.867814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:03:01.830655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'7","reported_seq":292,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:50.735982+0000","last_change":"2024-07-26T11:10:18.056473+0000","last_active":"2024-07-26T11:15:50.735982+0000","last_peered":"2024-07-26T11:15:50.735982+0000","last_clean":"2024-07-26T11:15:50.735982+0000","last_became_active":"2024-07-26T11:10:18.056038+0000","last_became_peered":"2024-07-26T11:10:18.056038+0000","last_unstale":"2024-07-26T11:15:50.735982+0000","last_undegraded":"2024-07-26T11:15:50.735982+0000","last_fullsized":"2024-07-26T11:15:50.735982+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:04.501018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:46.782896+0000","last_change":"2024-07-26T11:10:35.991741+0000","last_active":"2024-07-26T11:12:46.782896+0000","last_peered":"2024-07-26T11:12:46.782896+0000","last_clean":"2024-07-26T11:12:46.782896+0000","last_became_active":"2024-07-26T11:10:35.990995+0000","last_became_peered":"2024-07-26T11:10:35.990995+0000","last_unstale":"2024-07-26T11:12:46.782896+0000","last_undegraded":"2024-07-26T11:12:46.782896+0000","last_fullsized":"2024-07-26T11:12:46.782896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:59:34.056646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.884331+0000","last_change":"2024-07-26T11:10:22.305053+0000","last_active":"2024-07-26T11:13:33.884331+0000","last_peered":"2024-07-26T11:13:33.884331+0000","last_clean":"2024-07-26T11:13:33.884331+0000","last_became_active":"2024-07-26T11:10:22.304035+0000","last_became_peered":"2024-07-26T11:10:22.304035+0000","last_unstale":"2024-07-26T11:13:33.884331+0000","last_undegraded":"2024-07-26T11:13:33.884331+0000","last_fullsized":"2024-07-26T11:13:33.884331+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:48.590799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:17.349794+0000","last_change":"2024-07-26T11:10:18.033581+0000","last_active":"2024-07-26T11:12:17.349794+0000","last_peered":"2024-07-26T11:12:17.349794+0000","last_clean":"2024-07-26T11:12:17.349794+0000","last_became_active":"2024-07-26T11:10:18.033038+0000","last_became_peered":"2024-07-26T11:10:18.033038+0000","last_unstale":"2024-07-26T11:12:17.349794+0000","last_undegraded":"2024-07-26T11:12:17.349794+0000","last_fullsized":"2024-07-26T11:12:17.349794+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:19:52.523006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:48.387395+0000","last_change":"2024-07-26T11:11:35.643683+0000","last_active":"2024-07-26T11:12:48.387395+0000","last_peered":"2024-07-26T11:12:48.387395+0000","last_clean":"2024-07-26T11:12:48.387395+0000","last_became_active":"2024-07-26T11:11:35.643109+0000","last_became_peered":"2024-07-26T11:11:35.643109+0000","last_unstale":"2024-07-26T11:12:48.387395+0000","last_undegraded":"2024-07-26T11:12:48.387395+0000","last_fullsized":"2024-07-26T11:12:48.387395+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:39:28.190726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.878540+0000","last_change":"2024-07-26T11:10:34.977690+0000","last_active":"2024-07-26T11:13:33.878540+0000","last_peered":"2024-07-26T11:13:33.878540+0000","last_clean":"2024-07-26T11:13:33.878540+0000","last_became_active":"2024-07-26T11:10:34.976938+0000","last_became_peered":"2024-07-26T11:10:34.976938+0000","last_unstale":"2024-07-26T11:13:33.878540+0000","last_undegraded":"2024-07-26T11:13:33.878540+0000","last_fullsized":"2024-07-26T11:13:33.878540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:26:21.510469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:24.937213+0000","last_change":"2024-07-26T11:10:18.045363+0000","last_active":"2024-07-26T11:12:24.937213+0000","last_peered":"2024-07-26T11:12:24.937213+0000","last_clean":"2024-07-26T11:12:24.937213+0000","last_became_active":"2024-07-26T11:10:18.044810+0000","last_became_peered":"2024-07-26T11:10:18.044810+0000","last_unstale":"2024-07-26T11:12:24.937213+0000","last_undegraded":"2024-07-26T11:12:24.937213+0000","last_fullsized":"2024-07-26T11:12:24.937213+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:34.507172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":120,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:27.358372+0000","last_change":"2024-07-26T11:08:48.936784+0000","last_active":"2024-07-26T11:13:27.358372+0000","last_peered":"2024-07-26T11:13:27.358372+0000","last_clean":"2024-07-26T11:13:27.358372+0000","last_became_active":"2024-07-26T11:08:48.931118+0000","last_became_peered":"2024-07-26T11:08:48.931118+0000","last_unstale":"2024-07-26T11:13:27.358372+0000","last_undegraded":"2024-07-26T11:13:27.358372+0000","last_fullsized":"2024-07-26T11:13:27.358372+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:05:01.918569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:05:01.918569+0000","last_clean_scrub_stamp":"2024-07-26T11:05:01.918569+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:57.128424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:42.352841+0000","last_change":"2024-07-26T11:11:35.385528+0000","last_active":"2024-07-26T11:12:42.352841+0000","last_peered":"2024-07-26T11:12:42.352841+0000","last_clean":"2024-07-26T11:12:42.352841+0000","last_became_active":"2024-07-26T11:11:35.385042+0000","last_became_peered":"2024-07-26T11:11:35.385042+0000","last_unstale":"2024-07-26T11:12:42.352841+0000","last_undegraded":"2024-07-26T11:12:42.352841+0000","last_fullsized":"2024-07-26T11:12:42.352841+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:52:30.343736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875558+0000","last_change":"2024-07-26T11:10:22.302715+0000","last_active":"2024-07-26T11:13:33.875558+0000","last_peered":"2024-07-26T11:13:33.875558+0000","last_clean":"2024-07-26T11:13:33.875558+0000","last_became_active":"2024-07-26T11:10:22.302284+0000","last_became_peered":"2024-07-26T11:10:22.302284+0000","last_unstale":"2024-07-26T11:13:33.875558+0000","last_undegraded":"2024-07-26T11:13:33.875558+0000","last_fullsized":"2024-07-26T11:13:33.875558+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:13:05.555439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":289,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:48.477472+0000","last_change":"2024-07-26T11:10:18.823160+0000","last_active":"2024-07-26T11:15:48.477472+0000","last_peered":"2024-07-26T11:15:48.477472+0000","last_clean":"2024-07-26T11:15:48.477472+0000","last_became_active":"2024-07-26T11:10:18.822652+0000","last_became_peered":"2024-07-26T11:10:18.822652+0000","last_unstale":"2024-07-26T11:15:48.477472+0000","last_undegraded":"2024-07-26T11:15:48.477472+0000","last_fullsized":"2024-07-26T11:15:48.477472+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:21:06.077784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":241,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:48.952701+0000","last_change":"2024-07-26T11:10:35.984007+0000","last_active":"2024-07-26T11:12:48.952701+0000","last_peered":"2024-07-26T11:12:48.952701+0000","last_clean":"2024-07-26T11:12:48.952701+0000","last_became_active":"2024-07-26T11:10:35.983352+0000","last_became_peered":"2024-07-26T11:10:35.983352+0000","last_unstale":"2024-07-26T11:12:48.952701+0000","last_undegraded":"2024-07-26T11:12:48.952701+0000","last_fullsized":"2024-07-26T11:12:48.952701+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:05:21.199041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874961+0000","last_change":"2024-07-26T11:10:22.320559+0000","last_active":"2024-07-26T11:13:33.874961+0000","last_peered":"2024-07-26T11:13:33.874961+0000","last_clean":"2024-07-26T11:13:33.874961+0000","last_became_active":"2024-07-26T11:10:22.320192+0000","last_became_peered":"2024-07-26T11:10:22.320192+0000","last_unstale":"2024-07-26T11:13:33.874961+0000","last_undegraded":"2024-07-26T11:13:33.874961+0000","last_fullsized":"2024-07-26T11:13:33.874961+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:48:51.774418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:33.385383+0000","last_change":"2024-07-26T11:10:18.044108+0000","last_active":"2024-07-26T11:12:33.385383+0000","last_peered":"2024-07-26T11:12:33.385383+0000","last_clean":"2024-07-26T11:12:33.385383+0000","last_became_active":"2024-07-26T11:10:18.043712+0000","last_became_peered":"2024-07-26T11:10:18.043712+0000","last_unstale":"2024-07-26T11:12:33.385383+0000","last_undegraded":"2024-07-26T11:12:33.385383+0000","last_fullsized":"2024-07-26T11:12:33.385383+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:05:34.382520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:54.249813+0000","last_change":"2024-07-26T11:10:35.999071+0000","last_active":"2024-07-26T11:12:54.249813+0000","last_peered":"2024-07-26T11:12:54.249813+0000","last_clean":"2024-07-26T11:12:54.249813+0000","last_became_active":"2024-07-26T11:10:35.997179+0000","last_became_peered":"2024-07-26T11:10:35.997179+0000","last_unstale":"2024-07-26T11:12:54.249813+0000","last_undegraded":"2024-07-26T11:12:54.249813+0000","last_fullsized":"2024-07-26T11:12:54.249813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:36:45.632086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.872320+0000","last_change":"2024-07-26T11:10:22.300356+0000","last_active":"2024-07-26T11:13:33.872320+0000","last_peered":"2024-07-26T11:13:33.872320+0000","last_clean":"2024-07-26T11:13:33.872320+0000","last_became_active":"2024-07-26T11:10:22.299949+0000","last_became_peered":"2024-07-26T11:10:22.299949+0000","last_unstale":"2024-07-26T11:13:33.872320+0000","last_undegraded":"2024-07-26T11:13:33.872320+0000","last_fullsized":"2024-07-26T11:13:33.872320+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:34.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:36.780361+0000","last_change":"2024-07-26T11:10:18.049965+0000","last_active":"2024-07-26T11:12:36.780361+0000","last_peered":"2024-07-26T11:12:36.780361+0000","last_clean":"2024-07-26T11:12:36.780361+0000","last_became_active":"2024-07-26T11:10:18.049251+0000","last_became_peered":"2024-07-26T11:10:18.049251+0000","last_unstale":"2024-07-26T11:12:36.780361+0000","last_undegraded":"2024-07-26T11:12:36.780361+0000","last_fullsized":"2024-07-26T11:12:36.780361+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:20.996609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:48.387315+0000","last_change":"2024-07-26T11:10:35.967661+0000","last_active":"2024-07-26T11:12:48.387315+0000","last_peered":"2024-07-26T11:12:48.387315+0000","last_clean":"2024-07-26T11:12:48.387315+0000","last_became_active":"2024-07-26T11:10:35.966867+0000","last_became_peered":"2024-07-26T11:10:35.966867+0000","last_unstale":"2024-07-26T11:12:48.387315+0000","last_undegraded":"2024-07-26T11:12:48.387315+0000","last_fullsized":"2024-07-26T11:12:48.387315+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:43.951319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.867777+0000","last_change":"2024-07-26T11:10:22.303733+0000","last_active":"2024-07-26T11:13:33.867777+0000","last_peered":"2024-07-26T11:13:33.867777+0000","last_clean":"2024-07-26T11:13:33.867777+0000","last_became_active":"2024-07-26T11:10:22.303322+0000","last_became_peered":"2024-07-26T11:10:22.303322+0000","last_unstale":"2024-07-26T11:13:33.867777+0000","last_undegraded":"2024-07-26T11:13:33.867777+0000","last_fullsized":"2024-07-26T11:13:33.867777+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:34:34.962492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:32.160782+0000","last_change":"2024-07-26T11:10:18.032539+0000","last_active":"2024-07-26T11:12:32.160782+0000","last_peered":"2024-07-26T11:12:32.160782+0000","last_clean":"2024-07-26T11:12:32.160782+0000","last_became_active":"2024-07-26T11:10:18.032059+0000","last_became_peered":"2024-07-26T11:10:18.032059+0000","last_unstale":"2024-07-26T11:12:32.160782+0000","last_undegraded":"2024-07-26T11:12:32.160782+0000","last_fullsized":"2024-07-26T11:12:32.160782+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:54.502339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:48.387339+0000","last_change":"2024-07-26T11:10:35.977473+0000","last_active":"2024-07-26T11:12:48.387339+0000","last_peered":"2024-07-26T11:12:48.387339+0000","last_clean":"2024-07-26T11:12:48.387339+0000","last_became_active":"2024-07-26T11:10:35.976830+0000","last_became_peered":"2024-07-26T11:10:35.976830+0000","last_unstale":"2024-07-26T11:12:48.387339+0000","last_undegraded":"2024-07-26T11:12:48.387339+0000","last_fullsized":"2024-07-26T11:12:48.387339+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:46:13.009246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875320+0000","last_change":"2024-07-26T11:10:22.304358+0000","last_active":"2024-07-26T11:13:33.875320+0000","last_peered":"2024-07-26T11:13:33.875320+0000","last_clean":"2024-07-26T11:13:33.875320+0000","last_became_active":"2024-07-26T11:10:22.303846+0000","last_became_peered":"2024-07-26T11:10:22.303846+0000","last_unstale":"2024-07-26T11:13:33.875320+0000","last_undegraded":"2024-07-26T11:13:33.875320+0000","last_fullsized":"2024-07-26T11:13:33.875320+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:27:28.556481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:37.161406+0000","last_change":"2024-07-26T11:10:18.057678+0000","last_active":"2024-07-26T11:12:37.161406+0000","last_peered":"2024-07-26T11:12:37.161406+0000","last_clean":"2024-07-26T11:12:37.161406+0000","last_became_active":"2024-07-26T11:10:18.057282+0000","last_became_peered":"2024-07-26T11:10:18.057282+0000","last_unstale":"2024-07-26T11:12:37.161406+0000","last_undegraded":"2024-07-26T11:12:37.161406+0000","last_fullsized":"2024-07-26T11:12:37.161406+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:08:35.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:01.785766+0000","last_change":"2024-07-26T11:10:36.000329+0000","last_active":"2024-07-26T11:13:01.785766+0000","last_peered":"2024-07-26T11:13:01.785766+0000","last_clean":"2024-07-26T11:13:01.785766+0000","last_became_active":"2024-07-26T11:10:35.999713+0000","last_became_peered":"2024-07-26T11:10:35.999713+0000","last_unstale":"2024-07-26T11:13:01.785766+0000","last_undegraded":"2024-07-26T11:13:01.785766+0000","last_fullsized":"2024-07-26T11:13:01.785766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:48.137237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.866953+0000","last_change":"2024-07-26T11:10:22.301526+0000","last_active":"2024-07-26T11:13:33.866953+0000","last_peered":"2024-07-26T11:13:33.866953+0000","last_clean":"2024-07-26T11:13:33.866953+0000","last_became_active":"2024-07-26T11:10:22.301098+0000","last_became_peered":"2024-07-26T11:10:22.301098+0000","last_unstale":"2024-07-26T11:13:33.866953+0000","last_undegraded":"2024-07-26T11:13:33.866953+0000","last_fullsized":"2024-07-26T11:13:33.866953+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:01:48.118315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'8","reported_seq":295,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:48.731530+0000","last_change":"2024-07-26T11:10:18.035922+0000","last_active":"2024-07-26T11:15:48.731530+0000","last_peered":"2024-07-26T11:15:48.731530+0000","last_clean":"2024-07-26T11:15:48.731530+0000","last_became_active":"2024-07-26T11:10:18.035184+0000","last_became_peered":"2024-07-26T11:10:18.035184+0000","last_unstale":"2024-07-26T11:15:48.731530+0000","last_undegraded":"2024-07-26T11:15:48.731530+0000","last_fullsized":"2024-07-26T11:15:48.731530+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:13:10.371105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1427,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":245,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:01.524560+0000","last_change":"2024-07-26T11:10:35.998791+0000","last_active":"2024-07-26T11:13:01.524560+0000","last_peered":"2024-07-26T11:13:01.524560+0000","last_clean":"2024-07-26T11:13:01.524560+0000","last_became_active":"2024-07-26T11:10:35.997163+0000","last_became_peered":"2024-07-26T11:10:35.997163+0000","last_unstale":"2024-07-26T11:13:01.524560+0000","last_undegraded":"2024-07-26T11:13:01.524560+0000","last_fullsized":"2024-07-26T11:13:01.524560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:28:50.167294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889494+0000","last_change":"2024-07-26T11:10:22.314219+0000","last_active":"2024-07-26T11:13:33.889494+0000","last_peered":"2024-07-26T11:13:33.889494+0000","last_clean":"2024-07-26T11:13:33.889494+0000","last_became_active":"2024-07-26T11:10:22.313811+0000","last_became_peered":"2024-07-26T11:10:22.313811+0000","last_unstale":"2024-07-26T11:13:33.889494+0000","last_undegraded":"2024-07-26T11:13:33.889494+0000","last_fullsized":"2024-07-26T11:13:33.889494+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:55:47.452798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:38.385955+0000","last_change":"2024-07-26T11:10:18.047818+0000","last_active":"2024-07-26T11:12:38.385955+0000","last_peered":"2024-07-26T11:12:38.385955+0000","last_clean":"2024-07-26T11:12:38.385955+0000","last_became_active":"2024-07-26T11:10:18.047159+0000","last_became_peered":"2024-07-26T11:10:18.047159+0000","last_unstale":"2024-07-26T11:12:38.385955+0000","last_undegraded":"2024-07-26T11:12:38.385955+0000","last_fullsized":"2024-07-26T11:12:38.385955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:37.528512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:57.510826+0000","last_change":"2024-07-26T11:10:35.996468+0000","last_active":"2024-07-26T11:12:57.510826+0000","last_peered":"2024-07-26T11:12:57.510826+0000","last_clean":"2024-07-26T11:12:57.510826+0000","last_became_active":"2024-07-26T11:10:35.996030+0000","last_became_peered":"2024-07-26T11:10:35.996030+0000","last_unstale":"2024-07-26T11:12:57.510826+0000","last_undegraded":"2024-07-26T11:12:57.510826+0000","last_fullsized":"2024-07-26T11:12:57.510826+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:20:14.743690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.881988+0000","last_change":"2024-07-26T11:10:22.318375+0000","last_active":"2024-07-26T11:13:33.881988+0000","last_peered":"2024-07-26T11:13:33.881988+0000","last_clean":"2024-07-26T11:13:33.881988+0000","last_became_active":"2024-07-26T11:10:22.317965+0000","last_became_peered":"2024-07-26T11:10:22.317965+0000","last_unstale":"2024-07-26T11:13:33.881988+0000","last_undegraded":"2024-07-26T11:13:33.881988+0000","last_fullsized":"2024-07-26T11:13:33.881988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:36:43.504045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:39.912995+0000","last_change":"2024-07-26T11:10:18.047957+0000","last_active":"2024-07-26T11:12:39.912995+0000","last_peered":"2024-07-26T11:12:39.912995+0000","last_clean":"2024-07-26T11:12:39.912995+0000","last_became_active":"2024-07-26T11:10:18.047495+0000","last_became_peered":"2024-07-26T11:10:18.047495+0000","last_unstale":"2024-07-26T11:12:39.912995+0000","last_undegraded":"2024-07-26T11:12:39.912995+0000","last_fullsized":"2024-07-26T11:12:39.912995+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:11.238295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:03.389374+0000","last_change":"2024-07-26T11:10:35.976946+0000","last_active":"2024-07-26T11:13:03.389374+0000","last_peered":"2024-07-26T11:13:03.389374+0000","last_clean":"2024-07-26T11:13:03.389374+0000","last_became_active":"2024-07-26T11:10:35.976474+0000","last_became_peered":"2024-07-26T11:10:35.976474+0000","last_unstale":"2024-07-26T11:13:03.389374+0000","last_undegraded":"2024-07-26T11:13:03.389374+0000","last_fullsized":"2024-07-26T11:13:03.389374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:50:51.240903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.859135+0000","last_change":"2024-07-26T11:10:22.302473+0000","last_active":"2024-07-26T11:13:33.859135+0000","last_peered":"2024-07-26T11:13:33.859135+0000","last_clean":"2024-07-26T11:13:33.859135+0000","last_became_active":"2024-07-26T11:10:22.302096+0000","last_became_peered":"2024-07-26T11:10:22.302096+0000","last_unstale":"2024-07-26T11:13:33.859135+0000","last_undegraded":"2024-07-26T11:13:33.859135+0000","last_fullsized":"2024-07-26T11:13:33.859135+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:19.215301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'43","reported_seq":599,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:51.482116+0000","last_change":"2024-07-26T11:10:34.973502+0000","last_active":"2024-07-26T11:15:51.482116+0000","last_peered":"2024-07-26T11:15:51.482116+0000","last_clean":"2024-07-26T11:15:51.482116+0000","last_became_active":"2024-07-26T11:10:34.964155+0000","last_became_peered":"2024-07-26T11:10:34.964155+0000","last_unstale":"2024-07-26T11:15:51.482116+0000","last_undegraded":"2024-07-26T11:15:51.482116+0000","last_fullsized":"2024-07-26T11:15:51.482116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:23:06.652746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72449,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":496,"num_read_kb":320,"num_write":45,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:56.523464+0000","last_change":"2024-07-26T11:10:35.981439+0000","last_active":"2024-07-26T11:12:56.523464+0000","last_peered":"2024-07-26T11:12:56.523464+0000","last_clean":"2024-07-26T11:12:56.523464+0000","last_became_active":"2024-07-26T11:10:35.980878+0000","last_became_peered":"2024-07-26T11:10:35.980878+0000","last_unstale":"2024-07-26T11:12:56.523464+0000","last_undegraded":"2024-07-26T11:12:56.523464+0000","last_fullsized":"2024-07-26T11:12:56.523464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:31.719845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889855+0000","last_change":"2024-07-26T11:10:34.974292+0000","last_active":"2024-07-26T11:13:33.889855+0000","last_peered":"2024-07-26T11:13:33.889855+0000","last_clean":"2024-07-26T11:13:33.889855+0000","last_became_active":"2024-07-26T11:10:34.973820+0000","last_became_peered":"2024-07-26T11:10:34.973820+0000","last_unstale":"2024-07-26T11:13:33.889855+0000","last_undegraded":"2024-07-26T11:13:33.889855+0000","last_fullsized":"2024-07-26T11:13:33.889855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:35:37.629313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:43.386694+0000","last_change":"2024-07-26T11:10:18.055248+0000","last_active":"2024-07-26T11:12:43.386694+0000","last_peered":"2024-07-26T11:12:43.386694+0000","last_clean":"2024-07-26T11:12:43.386694+0000","last_became_active":"2024-07-26T11:10:18.054882+0000","last_became_peered":"2024-07-26T11:10:18.054882+0000","last_unstale":"2024-07-26T11:12:43.386694+0000","last_undegraded":"2024-07-26T11:12:43.386694+0000","last_fullsized":"2024-07-26T11:12:43.386694+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:16.390058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:59.941475+0000","last_change":"2024-07-26T11:10:35.985628+0000","last_active":"2024-07-26T11:12:59.941475+0000","last_peered":"2024-07-26T11:12:59.941475+0000","last_clean":"2024-07-26T11:12:59.941475+0000","last_became_active":"2024-07-26T11:10:35.985156+0000","last_became_peered":"2024-07-26T11:10:35.985156+0000","last_unstale":"2024-07-26T11:12:59.941475+0000","last_undegraded":"2024-07-26T11:12:59.941475+0000","last_fullsized":"2024-07-26T11:12:59.941475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:29:18.891825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.893557+0000","last_change":"2024-07-26T11:10:34.973261+0000","last_active":"2024-07-26T11:13:33.893557+0000","last_peered":"2024-07-26T11:13:33.893557+0000","last_clean":"2024-07-26T11:13:33.893557+0000","last_became_active":"2024-07-26T11:10:34.971592+0000","last_became_peered":"2024-07-26T11:10:34.971592+0000","last_unstale":"2024-07-26T11:13:33.893557+0000","last_undegraded":"2024-07-26T11:13:33.893557+0000","last_fullsized":"2024-07-26T11:13:33.893557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:58:13.517884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:42.352859+0000","last_change":"2024-07-26T11:10:34.962248+0000","last_active":"2024-07-26T11:12:42.352859+0000","last_peered":"2024-07-26T11:12:42.352859+0000","last_clean":"2024-07-26T11:12:42.352859+0000","last_became_active":"2024-07-26T11:10:34.961683+0000","last_became_peered":"2024-07-26T11:10:34.961683+0000","last_unstale":"2024-07-26T11:12:42.352859+0000","last_undegraded":"2024-07-26T11:12:42.352859+0000","last_fullsized":"2024-07-26T11:12:42.352859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:17:08.308226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:56.523407+0000","last_change":"2024-07-26T11:10:35.995760+0000","last_active":"2024-07-26T11:12:56.523407+0000","last_peered":"2024-07-26T11:12:56.523407+0000","last_clean":"2024-07-26T11:12:56.523407+0000","last_became_active":"2024-07-26T11:10:35.995328+0000","last_became_peered":"2024-07-26T11:10:35.995328+0000","last_unstale":"2024-07-26T11:12:56.523407+0000","last_undegraded":"2024-07-26T11:12:56.523407+0000","last_fullsized":"2024-07-26T11:12:56.523407+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:40:33.093315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.878655+0000","last_change":"2024-07-26T11:10:22.316151+0000","last_active":"2024-07-26T11:13:33.878655+0000","last_peered":"2024-07-26T11:13:33.878655+0000","last_clean":"2024-07-26T11:13:33.878655+0000","last_became_active":"2024-07-26T11:10:22.315722+0000","last_became_peered":"2024-07-26T11:10:22.315722+0000","last_unstale":"2024-07-26T11:13:33.878655+0000","last_undegraded":"2024-07-26T11:13:33.878655+0000","last_fullsized":"2024-07-26T11:13:33.878655+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:12:58.342637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":529,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:51.481269+0000","last_change":"2024-07-26T11:10:18.046327+0000","last_active":"2024-07-26T11:15:51.481269+0000","last_peered":"2024-07-26T11:15:51.481269+0000","last_clean":"2024-07-26T11:15:51.481269+0000","last_became_active":"2024-07-26T11:10:18.045913+0000","last_became_peered":"2024-07-26T11:10:18.045913+0000","last_unstale":"2024-07-26T11:15:51.481269+0000","last_undegraded":"2024-07-26T11:15:51.481269+0000","last_fullsized":"2024-07-26T11:15:51.481269+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:58.889046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":482,"num_read_kb":482,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:59.069724+0000","last_change":"2024-07-26T11:10:35.983264+0000","last_active":"2024-07-26T11:12:59.069724+0000","last_peered":"2024-07-26T11:12:59.069724+0000","last_clean":"2024-07-26T11:12:59.069724+0000","last_became_active":"2024-07-26T11:10:35.982210+0000","last_became_peered":"2024-07-26T11:10:35.982210+0000","last_unstale":"2024-07-26T11:12:59.069724+0000","last_undegraded":"2024-07-26T11:12:59.069724+0000","last_fullsized":"2024-07-26T11:12:59.069724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:37:20.272725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:39.246875+0000","last_change":"2024-07-26T11:10:35.985335+0000","last_active":"2024-07-26T11:12:39.246875+0000","last_peered":"2024-07-26T11:12:39.246875+0000","last_clean":"2024-07-26T11:12:39.246875+0000","last_became_active":"2024-07-26T11:10:35.983667+0000","last_became_peered":"2024-07-26T11:10:35.983667+0000","last_unstale":"2024-07-26T11:12:39.246875+0000","last_undegraded":"2024-07-26T11:12:39.246875+0000","last_fullsized":"2024-07-26T11:12:39.246875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:00:17.744806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.879478+0000","last_change":"2024-07-26T11:10:22.333494+0000","last_active":"2024-07-26T11:13:33.879478+0000","last_peered":"2024-07-26T11:13:33.879478+0000","last_clean":"2024-07-26T11:13:33.879478+0000","last_became_active":"2024-07-26T11:10:22.332950+0000","last_became_peered":"2024-07-26T11:10:22.332950+0000","last_unstale":"2024-07-26T11:13:33.879478+0000","last_undegraded":"2024-07-26T11:13:33.879478+0000","last_fullsized":"2024-07-26T11:13:33.879478+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:58:38.340857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:43.386643+0000","last_change":"2024-07-26T11:10:18.034057+0000","last_active":"2024-07-26T11:12:43.386643+0000","last_peered":"2024-07-26T11:12:43.386643+0000","last_clean":"2024-07-26T11:12:43.386643+0000","last_became_active":"2024-07-26T11:10:18.033119+0000","last_became_peered":"2024-07-26T11:10:18.033119+0000","last_unstale":"2024-07-26T11:12:43.386643+0000","last_undegraded":"2024-07-26T11:12:43.386643+0000","last_fullsized":"2024-07-26T11:12:43.386643+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:21.675593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:42.508793+0000","last_change":"2024-07-26T11:11:35.408377+0000","last_active":"2024-07-26T11:12:42.508793+0000","last_peered":"2024-07-26T11:12:42.508793+0000","last_clean":"2024-07-26T11:12:42.508793+0000","last_became_active":"2024-07-26T11:11:35.407865+0000","last_became_peered":"2024-07-26T11:11:35.407865+0000","last_unstale":"2024-07-26T11:12:42.508793+0000","last_undegraded":"2024-07-26T11:12:42.508793+0000","last_fullsized":"2024-07-26T11:12:42.508793+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:43:02.953395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889650+0000","last_change":"2024-07-26T11:10:22.323447+0000","last_active":"2024-07-26T11:13:33.889650+0000","last_peered":"2024-07-26T11:13:33.889650+0000","last_clean":"2024-07-26T11:13:33.889650+0000","last_became_active":"2024-07-26T11:10:22.322988+0000","last_became_peered":"2024-07-26T11:10:22.322988+0000","last_unstale":"2024-07-26T11:13:33.889650+0000","last_undegraded":"2024-07-26T11:13:33.889650+0000","last_fullsized":"2024-07-26T11:13:33.889650+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:14.880985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:46.782728+0000","last_change":"2024-07-26T11:10:18.048724+0000","last_active":"2024-07-26T11:12:46.782728+0000","last_peered":"2024-07-26T11:12:46.782728+0000","last_clean":"2024-07-26T11:12:46.782728+0000","last_became_active":"2024-07-26T11:10:18.048257+0000","last_became_peered":"2024-07-26T11:10:18.048257+0000","last_unstale":"2024-07-26T11:12:46.782728+0000","last_undegraded":"2024-07-26T11:12:46.782728+0000","last_fullsized":"2024-07-26T11:12:46.782728+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:02:13.581446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:47.389909+0000","last_change":"2024-07-26T11:10:18.044171+0000","last_active":"2024-07-26T11:12:47.389909+0000","last_peered":"2024-07-26T11:12:47.389909+0000","last_clean":"2024-07-26T11:12:47.389909+0000","last_became_active":"2024-07-26T11:10:18.043682+0000","last_became_peered":"2024-07-26T11:10:18.043682+0000","last_unstale":"2024-07-26T11:12:47.389909+0000","last_undegraded":"2024-07-26T11:12:47.389909+0000","last_fullsized":"2024-07-26T11:12:47.389909+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:46:46.864119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.876589+0000","last_change":"2024-07-26T11:10:22.321081+0000","last_active":"2024-07-26T11:13:33.876589+0000","last_peered":"2024-07-26T11:13:33.876589+0000","last_clean":"2024-07-26T11:13:33.876589+0000","last_became_active":"2024-07-26T11:10:22.320656+0000","last_became_peered":"2024-07-26T11:10:22.320656+0000","last_unstale":"2024-07-26T11:13:33.876589+0000","last_undegraded":"2024-07-26T11:13:33.876589+0000","last_fullsized":"2024-07-26T11:13:33.876589+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:17:20.455129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:09.916452+0000","last_change":"2024-07-26T11:10:36.000224+0000","last_active":"2024-07-26T11:13:09.916452+0000","last_peered":"2024-07-26T11:13:09.916452+0000","last_clean":"2024-07-26T11:13:09.916452+0000","last_became_active":"2024-07-26T11:10:35.999733+0000","last_became_peered":"2024-07-26T11:10:35.999733+0000","last_unstale":"2024-07-26T11:13:09.916452+0000","last_undegraded":"2024-07-26T11:13:09.916452+0000","last_fullsized":"2024-07-26T11:13:09.916452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:49.912401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":292,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:51.481917+0000","last_change":"2024-07-26T11:10:18.034585+0000","last_active":"2024-07-26T11:15:51.481917+0000","last_peered":"2024-07-26T11:15:51.481917+0000","last_clean":"2024-07-26T11:15:51.481917+0000","last_became_active":"2024-07-26T11:10:18.034237+0000","last_became_peered":"2024-07-26T11:10:18.034237+0000","last_unstale":"2024-07-26T11:15:51.481917+0000","last_undegraded":"2024-07-26T11:15:51.481917+0000","last_fullsized":"2024-07-26T11:15:51.481917+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:22.670277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.876583+0000","last_change":"2024-07-26T11:10:22.319282+0000","last_active":"2024-07-26T11:13:33.876583+0000","last_peered":"2024-07-26T11:13:33.876583+0000","last_clean":"2024-07-26T11:13:33.876583+0000","last_became_active":"2024-07-26T11:10:22.318822+0000","last_became_peered":"2024-07-26T11:10:22.318822+0000","last_unstale":"2024-07-26T11:13:33.876583+0000","last_undegraded":"2024-07-26T11:13:33.876583+0000","last_fullsized":"2024-07-26T11:13:33.876583+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:50:57.615839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:09.916483+0000","last_change":"2024-07-26T11:10:35.970189+0000","last_active":"2024-07-26T11:13:09.916483+0000","last_peered":"2024-07-26T11:13:09.916483+0000","last_clean":"2024-07-26T11:13:09.916483+0000","last_became_active":"2024-07-26T11:10:35.969574+0000","last_became_peered":"2024-07-26T11:10:35.969574+0000","last_unstale":"2024-07-26T11:13:09.916483+0000","last_undegraded":"2024-07-26T11:13:09.916483+0000","last_fullsized":"2024-07-26T11:13:09.916483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:07.487944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.868831+0000","last_change":"2024-07-26T11:10:22.299152+0000","last_active":"2024-07-26T11:13:33.868831+0000","last_peered":"2024-07-26T11:13:33.868831+0000","last_clean":"2024-07-26T11:13:33.868831+0000","last_became_active":"2024-07-26T11:10:22.298707+0000","last_became_peered":"2024-07-26T11:10:22.298707+0000","last_unstale":"2024-07-26T11:13:33.868831+0000","last_undegraded":"2024-07-26T11:13:33.868831+0000","last_fullsized":"2024-07-26T11:13:33.868831+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:25.161634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.872159+0000","last_change":"2024-07-26T11:10:22.302476+0000","last_active":"2024-07-26T11:13:33.872159+0000","last_peered":"2024-07-26T11:13:33.872159+0000","last_clean":"2024-07-26T11:13:33.872159+0000","last_became_active":"2024-07-26T11:10:22.302091+0000","last_became_peered":"2024-07-26T11:10:22.302091+0000","last_unstale":"2024-07-26T11:13:33.872159+0000","last_undegraded":"2024-07-26T11:13:33.872159+0000","last_fullsized":"2024-07-26T11:13:33.872159+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:06:18.587588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.892481+0000","last_change":"2024-07-26T11:10:22.326583+0000","last_active":"2024-07-26T11:13:33.892481+0000","last_peered":"2024-07-26T11:13:33.892481+0000","last_clean":"2024-07-26T11:13:33.892481+0000","last_became_active":"2024-07-26T11:10:22.326185+0000","last_became_peered":"2024-07-26T11:10:22.326185+0000","last_unstale":"2024-07-26T11:13:33.892481+0000","last_undegraded":"2024-07-26T11:13:33.892481+0000","last_fullsized":"2024-07-26T11:13:33.892481+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:54:00.148964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.894835+0000","last_change":"2024-07-26T11:10:22.329247+0000","last_active":"2024-07-26T11:13:33.894835+0000","last_peered":"2024-07-26T11:13:33.894835+0000","last_clean":"2024-07-26T11:13:33.894835+0000","last_became_active":"2024-07-26T11:10:22.328585+0000","last_became_peered":"2024-07-26T11:10:22.328585+0000","last_unstale":"2024-07-26T11:13:33.894835+0000","last_undegraded":"2024-07-26T11:13:33.894835+0000","last_fullsized":"2024-07-26T11:13:33.894835+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:01:27.195399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.895664+0000","last_change":"2024-07-26T11:10:22.330717+0000","last_active":"2024-07-26T11:13:33.895664+0000","last_peered":"2024-07-26T11:13:33.895664+0000","last_clean":"2024-07-26T11:13:33.895664+0000","last_became_active":"2024-07-26T11:10:22.329576+0000","last_became_peered":"2024-07-26T11:10:22.329576+0000","last_unstale":"2024-07-26T11:13:33.895664+0000","last_undegraded":"2024-07-26T11:13:33.895664+0000","last_fullsized":"2024-07-26T11:13:33.895664+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:21.970629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.867859+0000","last_change":"2024-07-26T11:10:22.307445+0000","last_active":"2024-07-26T11:13:33.867859+0000","last_peered":"2024-07-26T11:13:33.867859+0000","last_clean":"2024-07-26T11:13:33.867859+0000","last_became_active":"2024-07-26T11:10:22.305738+0000","last_became_peered":"2024-07-26T11:10:22.305738+0000","last_unstale":"2024-07-26T11:13:33.867859+0000","last_undegraded":"2024-07-26T11:13:33.867859+0000","last_fullsized":"2024-07-26T11:13:33.867859+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:09:08.125209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874579+0000","last_change":"2024-07-26T11:10:22.298392+0000","last_active":"2024-07-26T11:13:33.874579+0000","last_peered":"2024-07-26T11:13:33.874579+0000","last_clean":"2024-07-26T11:13:33.874579+0000","last_became_active":"2024-07-26T11:10:22.297979+0000","last_became_peered":"2024-07-26T11:10:22.297979+0000","last_unstale":"2024-07-26T11:13:33.874579+0000","last_undegraded":"2024-07-26T11:13:33.874579+0000","last_fullsized":"2024-07-26T11:13:33.874579+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:30:39.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.873814+0000","last_change":"2024-07-26T11:10:22.320850+0000","last_active":"2024-07-26T11:13:33.873814+0000","last_peered":"2024-07-26T11:13:33.873814+0000","last_clean":"2024-07-26T11:13:33.873814+0000","last_became_active":"2024-07-26T11:10:22.319838+0000","last_became_peered":"2024-07-26T11:10:22.319838+0000","last_unstale":"2024-07-26T11:13:33.873814+0000","last_undegraded":"2024-07-26T11:13:33.873814+0000","last_fullsized":"2024-07-26T11:13:33.873814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:14.289366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.873192+0000","last_change":"2024-07-26T11:10:22.321305+0000","last_active":"2024-07-26T11:13:33.873192+0000","last_peered":"2024-07-26T11:13:33.873192+0000","last_clean":"2024-07-26T11:13:33.873192+0000","last_became_active":"2024-07-26T11:10:22.320733+0000","last_became_peered":"2024-07-26T11:10:22.320733+0000","last_unstale":"2024-07-26T11:13:33.873192+0000","last_undegraded":"2024-07-26T11:13:33.873192+0000","last_fullsized":"2024-07-26T11:13:33.873192+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:24:33.125828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874816+0000","last_change":"2024-07-26T11:10:22.318193+0000","last_active":"2024-07-26T11:13:33.874816+0000","last_peered":"2024-07-26T11:13:33.874816+0000","last_clean":"2024-07-26T11:13:33.874816+0000","last_became_active":"2024-07-26T11:10:22.317819+0000","last_became_peered":"2024-07-26T11:10:22.317819+0000","last_unstale":"2024-07-26T11:13:33.874816+0000","last_undegraded":"2024-07-26T11:13:33.874816+0000","last_fullsized":"2024-07-26T11:13:33.874816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:14:06.105152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875254+0000","last_change":"2024-07-26T11:10:22.318222+0000","last_active":"2024-07-26T11:13:33.875254+0000","last_peered":"2024-07-26T11:13:33.875254+0000","last_clean":"2024-07-26T11:13:33.875254+0000","last_became_active":"2024-07-26T11:10:22.317806+0000","last_became_peered":"2024-07-26T11:10:22.317806+0000","last_unstale":"2024-07-26T11:13:33.875254+0000","last_undegraded":"2024-07-26T11:13:33.875254+0000","last_fullsized":"2024-07-26T11:13:33.875254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:55:27.510542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.895333+0000","last_change":"2024-07-26T11:10:22.330166+0000","last_active":"2024-07-26T11:13:33.895333+0000","last_peered":"2024-07-26T11:13:33.895333+0000","last_clean":"2024-07-26T11:13:33.895333+0000","last_became_active":"2024-07-26T11:10:22.329722+0000","last_became_peered":"2024-07-26T11:10:22.329722+0000","last_unstale":"2024-07-26T11:13:33.895333+0000","last_undegraded":"2024-07-26T11:13:33.895333+0000","last_fullsized":"2024-07-26T11:13:33.895333+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:58.816139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.866660+0000","last_change":"2024-07-26T11:10:22.302774+0000","last_active":"2024-07-26T11:13:33.866660+0000","last_peered":"2024-07-26T11:13:33.866660+0000","last_clean":"2024-07-26T11:13:33.866660+0000","last_became_active":"2024-07-26T11:10:22.302382+0000","last_became_peered":"2024-07-26T11:10:22.302382+0000","last_unstale":"2024-07-26T11:13:33.866660+0000","last_undegraded":"2024-07-26T11:13:33.866660+0000","last_fullsized":"2024-07-26T11:13:33.866660+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:02:58.019598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.886317+0000","last_change":"2024-07-26T11:10:22.312475+0000","last_active":"2024-07-26T11:13:33.886317+0000","last_peered":"2024-07-26T11:13:33.886317+0000","last_clean":"2024-07-26T11:13:33.886317+0000","last_became_active":"2024-07-26T11:10:22.312078+0000","last_became_peered":"2024-07-26T11:10:22.312078+0000","last_unstale":"2024-07-26T11:13:33.886317+0000","last_undegraded":"2024-07-26T11:13:33.886317+0000","last_fullsized":"2024-07-26T11:13:33.886317+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:25:37.705136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.878921+0000","last_change":"2024-07-26T11:10:22.314776+0000","last_active":"2024-07-26T11:13:33.878921+0000","last_peered":"2024-07-26T11:13:33.878921+0000","last_clean":"2024-07-26T11:13:33.878921+0000","last_became_active":"2024-07-26T11:10:22.307808+0000","last_became_peered":"2024-07-26T11:10:22.307808+0000","last_unstale":"2024-07-26T11:13:33.878921+0000","last_undegraded":"2024-07-26T11:13:33.878921+0000","last_fullsized":"2024-07-26T11:13:33.878921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:44:06.152799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.891140+0000","last_change":"2024-07-26T11:10:22.328102+0000","last_active":"2024-07-26T11:13:33.891140+0000","last_peered":"2024-07-26T11:13:33.891140+0000","last_clean":"2024-07-26T11:13:33.891140+0000","last_became_active":"2024-07-26T11:10:22.327779+0000","last_became_peered":"2024-07-26T11:10:22.327779+0000","last_unstale":"2024-07-26T11:13:33.891140+0000","last_undegraded":"2024-07-26T11:13:33.891140+0000","last_fullsized":"2024-07-26T11:13:33.891140+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:14:34.569748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.888331+0000","last_change":"2024-07-26T11:10:22.329057+0000","last_active":"2024-07-26T11:13:33.888331+0000","last_peered":"2024-07-26T11:13:33.888331+0000","last_clean":"2024-07-26T11:13:33.888331+0000","last_became_active":"2024-07-26T11:10:22.328656+0000","last_became_peered":"2024-07-26T11:10:22.328656+0000","last_unstale":"2024-07-26T11:13:33.888331+0000","last_undegraded":"2024-07-26T11:13:33.888331+0000","last_fullsized":"2024-07-26T11:13:33.888331+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:18:39.861674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.891998+0000","last_change":"2024-07-26T11:10:22.326115+0000","last_active":"2024-07-26T11:13:33.891998+0000","last_peered":"2024-07-26T11:13:33.891998+0000","last_clean":"2024-07-26T11:13:33.891998+0000","last_became_active":"2024-07-26T11:10:22.325538+0000","last_became_peered":"2024-07-26T11:10:22.325538+0000","last_unstale":"2024-07-26T11:13:33.891998+0000","last_undegraded":"2024-07-26T11:13:33.891998+0000","last_fullsized":"2024-07-26T11:13:33.891998+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:18:19.339571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.892537+0000","last_change":"2024-07-26T11:10:22.326910+0000","last_active":"2024-07-26T11:13:33.892537+0000","last_peered":"2024-07-26T11:13:33.892537+0000","last_clean":"2024-07-26T11:13:33.892537+0000","last_became_active":"2024-07-26T11:10:22.324532+0000","last_became_peered":"2024-07-26T11:10:22.324532+0000","last_unstale":"2024-07-26T11:13:33.892537+0000","last_undegraded":"2024-07-26T11:13:33.892537+0000","last_fullsized":"2024-07-26T11:13:33.892537+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:33.671780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.890787+0000","last_change":"2024-07-26T11:10:22.312926+0000","last_active":"2024-07-26T11:13:33.890787+0000","last_peered":"2024-07-26T11:13:33.890787+0000","last_clean":"2024-07-26T11:13:33.890787+0000","last_became_active":"2024-07-26T11:10:22.311642+0000","last_became_peered":"2024-07-26T11:10:22.311642+0000","last_unstale":"2024-07-26T11:13:33.890787+0000","last_undegraded":"2024-07-26T11:13:33.890787+0000","last_fullsized":"2024-07-26T11:13:33.890787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:56:29.644965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.879104+0000","last_change":"2024-07-26T11:10:34.976934+0000","last_active":"2024-07-26T11:13:33.879104+0000","last_peered":"2024-07-26T11:13:33.879104+0000","last_clean":"2024-07-26T11:13:33.879104+0000","last_became_active":"2024-07-26T11:10:34.976551+0000","last_became_peered":"2024-07-26T11:10:34.976551+0000","last_unstale":"2024-07-26T11:13:33.879104+0000","last_undegraded":"2024-07-26T11:13:33.879104+0000","last_fullsized":"2024-07-26T11:13:33.879104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:11:43.883644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875269+0000","last_change":"2024-07-26T11:10:22.304444+0000","last_active":"2024-07-26T11:13:33.875269+0000","last_peered":"2024-07-26T11:13:33.875269+0000","last_clean":"2024-07-26T11:13:33.875269+0000","last_became_active":"2024-07-26T11:10:22.302825+0000","last_became_peered":"2024-07-26T11:10:22.302825+0000","last_unstale":"2024-07-26T11:13:33.875269+0000","last_undegraded":"2024-07-26T11:13:33.875269+0000","last_fullsized":"2024-07-26T11:13:33.875269+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:15.229057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.870763+0000","last_change":"2024-07-26T11:10:22.297515+0000","last_active":"2024-07-26T11:13:33.870763+0000","last_peered":"2024-07-26T11:13:33.870763+0000","last_clean":"2024-07-26T11:13:33.870763+0000","last_became_active":"2024-07-26T11:10:22.297134+0000","last_became_peered":"2024-07-26T11:10:22.297134+0000","last_unstale":"2024-07-26T11:13:33.870763+0000","last_undegraded":"2024-07-26T11:13:33.870763+0000","last_fullsized":"2024-07-26T11:13:33.870763+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:25:08.794363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875681+0000","last_change":"2024-07-26T11:10:22.304333+0000","last_active":"2024-07-26T11:13:33.875681+0000","last_peered":"2024-07-26T11:13:33.875681+0000","last_clean":"2024-07-26T11:13:33.875681+0000","last_became_active":"2024-07-26T11:10:22.303639+0000","last_became_peered":"2024-07-26T11:10:22.303639+0000","last_unstale":"2024-07-26T11:13:33.875681+0000","last_undegraded":"2024-07-26T11:13:33.875681+0000","last_fullsized":"2024-07-26T11:13:33.875681+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:47.053463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875130+0000","last_change":"2024-07-26T11:10:22.320998+0000","last_active":"2024-07-26T11:13:33.875130+0000","last_peered":"2024-07-26T11:13:33.875130+0000","last_clean":"2024-07-26T11:13:33.875130+0000","last_became_active":"2024-07-26T11:10:22.320610+0000","last_became_peered":"2024-07-26T11:10:22.320610+0000","last_unstale":"2024-07-26T11:13:33.875130+0000","last_undegraded":"2024-07-26T11:13:33.875130+0000","last_fullsized":"2024-07-26T11:13:33.875130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:52:28.802241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.873030+0000","last_change":"2024-07-26T11:10:34.970486+0000","last_active":"2024-07-26T11:13:33.873030+0000","last_peered":"2024-07-26T11:13:33.873030+0000","last_clean":"2024-07-26T11:13:33.873030+0000","last_became_active":"2024-07-26T11:10:34.969965+0000","last_became_peered":"2024-07-26T11:10:34.969965+0000","last_unstale":"2024-07-26T11:13:33.873030+0000","last_undegraded":"2024-07-26T11:13:33.873030+0000","last_fullsized":"2024-07-26T11:13:33.873030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:41.593251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.876555+0000","last_change":"2024-07-26T11:10:22.321261+0000","last_active":"2024-07-26T11:13:33.876555+0000","last_peered":"2024-07-26T11:13:33.876555+0000","last_clean":"2024-07-26T11:13:33.876555+0000","last_became_active":"2024-07-26T11:10:22.320819+0000","last_became_peered":"2024-07-26T11:10:22.320819+0000","last_unstale":"2024-07-26T11:13:33.876555+0000","last_undegraded":"2024-07-26T11:13:33.876555+0000","last_fullsized":"2024-07-26T11:13:33.876555+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:04:56.872600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.887727+0000","last_change":"2024-07-26T11:10:22.321206+0000","last_active":"2024-07-26T11:13:33.887727+0000","last_peered":"2024-07-26T11:13:33.887727+0000","last_clean":"2024-07-26T11:13:33.887727+0000","last_became_active":"2024-07-26T11:10:22.320700+0000","last_became_peered":"2024-07-26T11:10:22.320700+0000","last_unstale":"2024-07-26T11:13:33.887727+0000","last_undegraded":"2024-07-26T11:13:33.887727+0000","last_fullsized":"2024-07-26T11:13:33.887727+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:31:20.700808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.870903+0000","last_change":"2024-07-26T11:10:22.300061+0000","last_active":"2024-07-26T11:13:33.870903+0000","last_peered":"2024-07-26T11:13:33.870903+0000","last_clean":"2024-07-26T11:13:33.870903+0000","last_became_active":"2024-07-26T11:10:22.298737+0000","last_became_peered":"2024-07-26T11:10:22.298737+0000","last_unstale":"2024-07-26T11:13:33.870903+0000","last_undegraded":"2024-07-26T11:13:33.870903+0000","last_fullsized":"2024-07-26T11:13:33.870903+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:45:25.500816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.893213+0000","last_change":"2024-07-26T11:10:22.328534+0000","last_active":"2024-07-26T11:13:33.893213+0000","last_peered":"2024-07-26T11:13:33.893213+0000","last_clean":"2024-07-26T11:13:33.893213+0000","last_became_active":"2024-07-26T11:10:22.327929+0000","last_became_peered":"2024-07-26T11:10:22.327929+0000","last_unstale":"2024-07-26T11:13:33.893213+0000","last_undegraded":"2024-07-26T11:13:33.893213+0000","last_fullsized":"2024-07-26T11:13:33.893213+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:09:55.152881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.892299+0000","last_change":"2024-07-26T11:10:22.324753+0000","last_active":"2024-07-26T11:13:33.892299+0000","last_peered":"2024-07-26T11:13:33.892299+0000","last_clean":"2024-07-26T11:13:33.892299+0000","last_became_active":"2024-07-26T11:10:22.324307+0000","last_became_peered":"2024-07-26T11:10:22.324307+0000","last_unstale":"2024-07-26T11:13:33.892299+0000","last_undegraded":"2024-07-26T11:13:33.892299+0000","last_fullsized":"2024-07-26T11:13:33.892299+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:40.566465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.894283+0000","last_change":"2024-07-26T11:10:22.328183+0000","last_active":"2024-07-26T11:13:33.894283+0000","last_peered":"2024-07-26T11:13:33.894283+0000","last_clean":"2024-07-26T11:13:33.894283+0000","last_became_active":"2024-07-26T11:10:22.327805+0000","last_became_peered":"2024-07-26T11:10:22.327805+0000","last_unstale":"2024-07-26T11:13:33.894283+0000","last_undegraded":"2024-07-26T11:13:33.894283+0000","last_fullsized":"2024-07-26T11:13:33.894283+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:11:23.238390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":79710,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":2453,"num_read_kb":1316,"num_write":265,"num_write_kb":162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2194,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":598016,"data_stored":233757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":73,"seq":313532612691,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":527167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808919,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28712,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716440,"statfs":{"total":95995035648,"available":95965634560,"internally_reserved":0,"allocated":1736704,"data_stored":1186017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972445,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":529939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168676,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":529735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364904,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28620,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716532,"statfs":{"total":95995035648,"available":95965728768,"internally_reserved":0,"allocated":1642496,"data_stored":1122494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528430,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":521347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724662,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36952,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708200,"statfs":{"total":95995035648,"available":95957196800,"internally_reserved":0,"allocated":1654784,"data_stored":1113813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888186,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1118208,"data_stored":593336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084414,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":521055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378756,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":522597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542282,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":594826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705805,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":74201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":75024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T11:15:55.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T11:15:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:54 smithi189 ceph-mon[36790]: pgmap v478: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-26T11:15:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:54 smithi193 ceph-mon[37473]: pgmap v478: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 938 B/s wr, 11 op/s 2024-07-26T11:15:55.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.466+0000 7f6564a00640 1 -- 172.21.15.134:0/2933273203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6560109990 msgr2=0x7f6560109da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:55.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.466+0000 7f6564a00640 1 --2- 172.21.15.134:0/2933273203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6560109990 0x7f6560109da0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6550009900 tx=0x7f655002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:15:55.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.466+0000 7f6564a00640 1 -- 172.21.15.134:0/2933273203 shutdown_connections 2024-07-26T11:15:55.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.466+0000 7f6564a00640 1 --2- 172.21.15.134:0/2933273203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f656010c980 0x7f65601163f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:55.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.466+0000 7f6564a00640 1 --2- 172.21.15.134:0/2933273203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656010c030 0x7f656010c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:55.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.466+0000 7f6564a00640 1 --2- 172.21.15.134:0/2933273203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6560109990 0x7f6560109da0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:55.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.466+0000 7f6564a00640 1 -- 172.21.15.134:0/2933273203 >> 172.21.15.134:0/2933273203 conn(0x7f65600790b0 msgr2=0x7f6560107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:55.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.466+0000 7f6564a00640 1 -- 172.21.15.134:0/2933273203 shutdown_connections 2024-07-26T11:15:55.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.466+0000 7f6564a00640 1 -- 172.21.15.134:0/2933273203 wait complete. 2024-07-26T11:15:55.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.467+0000 7f6564a00640 1 Processor -- start 2024-07-26T11:15:55.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.467+0000 7f6564a00640 1 -- start start 2024-07-26T11:15:55.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.467+0000 7f6564a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6560109990 0x7f65601a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:55.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.467+0000 7f6564a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f656010c030 0x7f65601a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:55.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.467+0000 7f6564a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656010c980 0x7f65601a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:55.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.467+0000 7f6564a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6560119880 con 0x7f656010c030 2024-07-26T11:15:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.467+0000 7f6564a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6560119700 con 0x7f656010c980 2024-07-26T11:15:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.467+0000 7f6564a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6560119a00 con 0x7f6560109990 2024-07-26T11:15:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.468+0000 7f6557e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f656010c030 0x7f65601a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.468+0000 7f6557e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f656010c030 0x7f65601a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:53282/0 (socket says 172.21.15.134:53282) 2024-07-26T11:15:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.468+0000 7f6557e00640 1 -- 172.21.15.134:0/750835124 learned_addr learned my addr 172.21.15.134:0/750835124 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.468+0000 7f655f400640 1 --2- 172.21.15.134:0/750835124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6560109990 0x7f65601a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.468+0000 7f655fe00640 1 --2- 172.21.15.134:0/750835124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656010c980 0x7f65601a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.468+0000 7f6557e00640 1 -- 172.21.15.134:0/750835124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6560109990 msgr2=0x7f65601a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.468+0000 7f6557e00640 1 --2- 172.21.15.134:0/750835124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6560109990 0x7f65601a0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.468+0000 7f6557e00640 1 -- 172.21.15.134:0/750835124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656010c980 msgr2=0x7f65601a5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.468+0000 7f6557e00640 1 --2- 172.21.15.134:0/750835124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656010c980 0x7f65601a5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.468+0000 7f6557e00640 1 -- 172.21.15.134:0/750835124 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65601a60e0 con 0x7f656010c030 2024-07-26T11:15:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.469+0000 7f6557e00640 1 --2- 172.21.15.134:0/750835124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f656010c030 0x7f65601a07e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f654800e9f0 tx=0x7f654800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.469+0000 7f655cc00640 1 -- 172.21.15.134:0/750835124 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f654800ccc0 con 0x7f656010c030 2024-07-26T11:15:55.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.469+0000 7f6564a00640 1 -- 172.21.15.134:0/750835124 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65601a63d0 con 0x7f656010c030 2024-07-26T11:15:55.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.469+0000 7f655cc00640 1 -- 172.21.15.134:0/750835124 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65480054d0 con 0x7f656010c030 2024-07-26T11:15:55.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.469+0000 7f655cc00640 1 -- 172.21.15.134:0/750835124 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548010430 con 0x7f656010c030 2024-07-26T11:15:55.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.469+0000 7f6564a00640 1 -- 172.21.15.134:0/750835124 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65601acbf0 con 0x7f656010c030 2024-07-26T11:15:55.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.470+0000 7f6564a00640 1 -- 172.21.15.134:0/750835124 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6560109da0 con 0x7f656010c030 2024-07-26T11:15:55.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.472+0000 7f655cc00640 1 -- 172.21.15.134:0/750835124 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f65480046c0 con 0x7f656010c030 2024-07-26T11:15:55.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.474+0000 7f655cc00640 1 --2- 172.21.15.134:0/750835124 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65440b3c20 0x7f65440b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:55.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.474+0000 7f655cc00640 1 -- 172.21.15.134:0/750835124 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f65480e4630 con 0x7f656010c030 2024-07-26T11:15:55.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.474+0000 7f655f400640 1 --2- 172.21.15.134:0/750835124 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65440b3c20 0x7f65440b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:55.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.475+0000 7f655f400640 1 --2- 172.21.15.134:0/750835124 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65440b3c20 0x7f65440b6130 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6550031440 tx=0x7f6550009560 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:55.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.476+0000 7f655cc00640 1 -- 172.21.15.134:0/750835124 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f6548055600 con 0x7f656010c030 2024-07-26T11:15:55.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.680+0000 7f6564a00640 1 -- 172.21.15.134:0/750835124 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f65601119f0 con 0x7f65440b3c20 2024-07-26T11:15:55.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.695+0000 7f655cc00640 1 -- 172.21.15.134:0/750835124 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340544 (secure 0 0 0) 0x7f65601119f0 con 0x7f65440b3c20 2024-07-26T11:15:55.704 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:15:55.705 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-07-26T11:15:55.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.704+0000 7f6564a00640 1 -- 172.21.15.134:0/750835124 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65440b3c20 msgr2=0x7f65440b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:55.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.704+0000 7f6564a00640 1 --2- 172.21.15.134:0/750835124 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65440b3c20 0x7f65440b6130 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6550031440 tx=0x7f6550009560 comp rx=0 tx=0).stop 2024-07-26T11:15:55.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.704+0000 7f6564a00640 1 -- 172.21.15.134:0/750835124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f656010c030 msgr2=0x7f65601a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:55.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.704+0000 7f6564a00640 1 --2- 172.21.15.134:0/750835124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f656010c030 0x7f65601a07e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f654800e9f0 tx=0x7f654800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:15:55.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.704+0000 7f6564a00640 1 -- 172.21.15.134:0/750835124 shutdown_connections 2024-07-26T11:15:55.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.704+0000 7f6564a00640 1 --2- 172.21.15.134:0/750835124 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65440b3c20 0x7f65440b6130 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:55.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.704+0000 7f6564a00640 1 --2- 172.21.15.134:0/750835124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f656010c980 0x7f65601a5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:55.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.704+0000 7f6564a00640 1 --2- 172.21.15.134:0/750835124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f656010c030 0x7f65601a07e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:55.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.704+0000 7f6564a00640 1 --2- 172.21.15.134:0/750835124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6560109990 0x7f65601a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:55.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.704+0000 7f6564a00640 1 -- 172.21.15.134:0/750835124 >> 172.21.15.134:0/750835124 conn(0x7f65600790b0 msgr2=0x7f656010d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:55.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.705+0000 7f6564a00640 1 -- 172.21.15.134:0/750835124 shutdown_connections 2024-07-26T11:15:55.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:55.705+0000 7f6564a00640 1 -- 172.21.15.134:0/750835124 wait complete. 2024-07-26T11:15:55.732 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-07-26T11:15:54.049405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":670312,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":2534,"num_read_kb":1385,"num_write":390,"num_write_kb":2314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":2954034,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":822,"ondisk_log_size":822,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375892,"kb_used_data":14260,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565932,"statfs":{"total":1151940427776,"available":1151555514368,"internally_reserved":0,"allocated":14602240,"data_stored":8285129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5987,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":73,"num_write":7,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006132"},"pg_stats":[{"pgid":"2.1f","version":"90'13","reported_seq":310,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:49.585215+0000","last_change":"2024-07-26T11:10:34.985984+0000","last_active":"2024-07-26T11:15:49.585215+0000","last_peered":"2024-07-26T11:15:49.585215+0000","last_clean":"2024-07-26T11:15:49.585215+0000","last_became_active":"2024-07-26T11:10:34.980894+0000","last_became_peered":"2024-07-26T11:10:34.980894+0000","last_unstale":"2024-07-26T11:15:49.585215+0000","last_undegraded":"2024-07-26T11:15:49.585215+0000","last_fullsized":"2024-07-26T11:15:49.585215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:55.718875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":241,"num_read_kb":241,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874388+0000","last_change":"2024-07-26T11:10:22.320946+0000","last_active":"2024-07-26T11:13:33.874388+0000","last_peered":"2024-07-26T11:13:33.874388+0000","last_clean":"2024-07-26T11:13:33.874388+0000","last_became_active":"2024-07-26T11:10:22.316181+0000","last_became_peered":"2024-07-26T11:10:22.316181+0000","last_unstale":"2024-07-26T11:13:33.874388+0000","last_undegraded":"2024-07-26T11:13:33.874388+0000","last_fullsized":"2024-07-26T11:13:33.874388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:13.641862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:04.915763+0000","last_change":"2024-07-26T11:10:35.984460+0000","last_active":"2024-07-26T11:13:04.915763+0000","last_peered":"2024-07-26T11:13:04.915763+0000","last_clean":"2024-07-26T11:13:04.915763+0000","last_became_active":"2024-07-26T11:10:35.983922+0000","last_became_peered":"2024-07-26T11:10:35.983922+0000","last_unstale":"2024-07-26T11:13:04.915763+0000","last_undegraded":"2024-07-26T11:13:04.915763+0000","last_fullsized":"2024-07-26T11:13:04.915763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:51:46.140910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:19.061450+0000","last_change":"2024-07-26T11:10:18.058553+0000","last_active":"2024-07-26T11:12:19.061450+0000","last_peered":"2024-07-26T11:12:19.061450+0000","last_clean":"2024-07-26T11:12:19.061450+0000","last_became_active":"2024-07-26T11:10:18.058034+0000","last_became_peered":"2024-07-26T11:10:18.058034+0000","last_unstale":"2024-07-26T11:12:19.061450+0000","last_undegraded":"2024-07-26T11:12:19.061450+0000","last_fullsized":"2024-07-26T11:12:19.061450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:48:58.868988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.893963+0000","last_change":"2024-07-26T11:10:22.326760+0000","last_active":"2024-07-26T11:13:33.893963+0000","last_peered":"2024-07-26T11:13:33.893963+0000","last_clean":"2024-07-26T11:13:33.893963+0000","last_became_active":"2024-07-26T11:10:22.326275+0000","last_became_peered":"2024-07-26T11:10:22.326275+0000","last_unstale":"2024-07-26T11:13:33.893963+0000","last_undegraded":"2024-07-26T11:13:33.893963+0000","last_fullsized":"2024-07-26T11:13:33.893963+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:51:57.461453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:04.941996+0000","last_change":"2024-07-26T11:10:35.997585+0000","last_active":"2024-07-26T11:13:04.941996+0000","last_peered":"2024-07-26T11:13:04.941996+0000","last_clean":"2024-07-26T11:13:04.941996+0000","last_became_active":"2024-07-26T11:10:35.997274+0000","last_became_peered":"2024-07-26T11:10:35.997274+0000","last_unstale":"2024-07-26T11:13:04.941996+0000","last_undegraded":"2024-07-26T11:13:04.941996+0000","last_fullsized":"2024-07-26T11:13:04.941996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:43.553396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:47.162843+0000","last_change":"2024-07-26T11:10:18.034964+0000","last_active":"2024-07-26T11:12:47.162843+0000","last_peered":"2024-07-26T11:12:47.162843+0000","last_clean":"2024-07-26T11:12:47.162843+0000","last_became_active":"2024-07-26T11:10:18.034569+0000","last_became_peered":"2024-07-26T11:10:18.034569+0000","last_unstale":"2024-07-26T11:12:47.162843+0000","last_undegraded":"2024-07-26T11:12:47.162843+0000","last_fullsized":"2024-07-26T11:12:47.162843+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:00:46.542458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.882555+0000","last_change":"2024-07-26T11:10:22.310695+0000","last_active":"2024-07-26T11:13:33.882555+0000","last_peered":"2024-07-26T11:13:33.882555+0000","last_clean":"2024-07-26T11:13:33.882555+0000","last_became_active":"2024-07-26T11:10:22.310142+0000","last_became_peered":"2024-07-26T11:10:22.310142+0000","last_unstale":"2024-07-26T11:13:33.882555+0000","last_undegraded":"2024-07-26T11:13:33.882555+0000","last_fullsized":"2024-07-26T11:13:33.882555+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:49:18.890802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:09.942670+0000","last_change":"2024-07-26T11:10:35.991418+0000","last_active":"2024-07-26T11:13:09.942670+0000","last_peered":"2024-07-26T11:13:09.942670+0000","last_clean":"2024-07-26T11:13:09.942670+0000","last_became_active":"2024-07-26T11:10:35.990934+0000","last_became_peered":"2024-07-26T11:10:35.990934+0000","last_unstale":"2024-07-26T11:13:09.942670+0000","last_undegraded":"2024-07-26T11:13:09.942670+0000","last_fullsized":"2024-07-26T11:13:09.942670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:56.598738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:07.512173+0000","last_change":"2024-07-26T11:10:34.965637+0000","last_active":"2024-07-26T11:13:07.512173+0000","last_peered":"2024-07-26T11:13:07.512173+0000","last_clean":"2024-07-26T11:13:07.512173+0000","last_became_active":"2024-07-26T11:10:34.961157+0000","last_became_peered":"2024-07-26T11:10:34.961157+0000","last_unstale":"2024-07-26T11:13:07.512173+0000","last_undegraded":"2024-07-26T11:13:07.512173+0000","last_fullsized":"2024-07-26T11:13:07.512173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:26:17.007258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875528+0000","last_change":"2024-07-26T11:10:22.303606+0000","last_active":"2024-07-26T11:13:33.875528+0000","last_peered":"2024-07-26T11:13:33.875528+0000","last_clean":"2024-07-26T11:13:33.875528+0000","last_became_active":"2024-07-26T11:10:22.303183+0000","last_became_peered":"2024-07-26T11:10:22.303183+0000","last_unstale":"2024-07-26T11:13:33.875528+0000","last_undegraded":"2024-07-26T11:13:33.875528+0000","last_fullsized":"2024-07-26T11:13:33.875528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:31:01.153694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:09.942648+0000","last_change":"2024-07-26T11:10:35.981446+0000","last_active":"2024-07-26T11:13:09.942648+0000","last_peered":"2024-07-26T11:13:09.942648+0000","last_clean":"2024-07-26T11:13:09.942648+0000","last_became_active":"2024-07-26T11:10:35.980994+0000","last_became_peered":"2024-07-26T11:10:35.980994+0000","last_unstale":"2024-07-26T11:13:09.942648+0000","last_undegraded":"2024-07-26T11:13:09.942648+0000","last_fullsized":"2024-07-26T11:13:09.942648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:38:57.405325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.887229+0000","last_change":"2024-07-26T11:10:22.306758+0000","last_active":"2024-07-26T11:13:33.887229+0000","last_peered":"2024-07-26T11:13:33.887229+0000","last_clean":"2024-07-26T11:13:33.887229+0000","last_became_active":"2024-07-26T11:10:22.306370+0000","last_became_peered":"2024-07-26T11:10:22.306370+0000","last_unstale":"2024-07-26T11:13:33.887229+0000","last_undegraded":"2024-07-26T11:13:33.887229+0000","last_fullsized":"2024-07-26T11:13:33.887229+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:51:43.069470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:32.160732+0000","last_change":"2024-07-26T11:10:18.035664+0000","last_active":"2024-07-26T11:12:32.160732+0000","last_peered":"2024-07-26T11:12:32.160732+0000","last_clean":"2024-07-26T11:12:32.160732+0000","last_became_active":"2024-07-26T11:10:18.035113+0000","last_became_peered":"2024-07-26T11:10:18.035113+0000","last_unstale":"2024-07-26T11:12:32.160732+0000","last_undegraded":"2024-07-26T11:12:32.160732+0000","last_fullsized":"2024-07-26T11:12:32.160732+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:09:30.641034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:54.940745+0000","last_change":"2024-07-26T11:10:35.990907+0000","last_active":"2024-07-26T11:12:54.940745+0000","last_peered":"2024-07-26T11:12:54.940745+0000","last_clean":"2024-07-26T11:12:54.940745+0000","last_became_active":"2024-07-26T11:10:35.990417+0000","last_became_peered":"2024-07-26T11:10:35.990417+0000","last_unstale":"2024-07-26T11:12:54.940745+0000","last_undegraded":"2024-07-26T11:12:54.940745+0000","last_fullsized":"2024-07-26T11:12:54.940745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:24:39.319727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889158+0000","last_change":"2024-07-26T11:10:22.333575+0000","last_active":"2024-07-26T11:13:33.889158+0000","last_peered":"2024-07-26T11:13:33.889158+0000","last_clean":"2024-07-26T11:13:33.889158+0000","last_became_active":"2024-07-26T11:10:22.332802+0000","last_became_peered":"2024-07-26T11:10:22.332802+0000","last_unstale":"2024-07-26T11:13:33.889158+0000","last_undegraded":"2024-07-26T11:13:33.889158+0000","last_fullsized":"2024-07-26T11:13:33.889158+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:14:03.238840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:29.911881+0000","last_change":"2024-07-26T11:10:18.047942+0000","last_active":"2024-07-26T11:12:29.911881+0000","last_peered":"2024-07-26T11:12:29.911881+0000","last_clean":"2024-07-26T11:12:29.911881+0000","last_became_active":"2024-07-26T11:10:18.047500+0000","last_became_peered":"2024-07-26T11:10:18.047500+0000","last_unstale":"2024-07-26T11:12:29.911881+0000","last_undegraded":"2024-07-26T11:12:29.911881+0000","last_fullsized":"2024-07-26T11:12:29.911881+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:32:30.773648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:54.940721+0000","last_change":"2024-07-26T11:10:36.000946+0000","last_active":"2024-07-26T11:12:54.940721+0000","last_peered":"2024-07-26T11:12:54.940721+0000","last_clean":"2024-07-26T11:12:54.940721+0000","last_became_active":"2024-07-26T11:10:36.000251+0000","last_became_peered":"2024-07-26T11:10:36.000251+0000","last_unstale":"2024-07-26T11:12:54.940721+0000","last_undegraded":"2024-07-26T11:12:54.940721+0000","last_fullsized":"2024-07-26T11:12:54.940721+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:03:40.523747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.890526+0000","last_change":"2024-07-26T11:10:22.326370+0000","last_active":"2024-07-26T11:13:33.890526+0000","last_peered":"2024-07-26T11:13:33.890526+0000","last_clean":"2024-07-26T11:13:33.890526+0000","last_became_active":"2024-07-26T11:10:22.325961+0000","last_became_peered":"2024-07-26T11:10:22.325961+0000","last_unstale":"2024-07-26T11:13:33.890526+0000","last_undegraded":"2024-07-26T11:13:33.890526+0000","last_fullsized":"2024-07-26T11:13:33.890526+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:16:36.946750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:29.063642+0000","last_change":"2024-07-26T11:10:18.051940+0000","last_active":"2024-07-26T11:12:29.063642+0000","last_peered":"2024-07-26T11:12:29.063642+0000","last_clean":"2024-07-26T11:12:29.063642+0000","last_became_active":"2024-07-26T11:10:18.051565+0000","last_became_peered":"2024-07-26T11:10:18.051565+0000","last_unstale":"2024-07-26T11:12:29.063642+0000","last_undegraded":"2024-07-26T11:12:29.063642+0000","last_fullsized":"2024-07-26T11:12:29.063642+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:17:56.418638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:51.522449+0000","last_change":"2024-07-26T11:10:36.000156+0000","last_active":"2024-07-26T11:12:51.522449+0000","last_peered":"2024-07-26T11:12:51.522449+0000","last_clean":"2024-07-26T11:12:51.522449+0000","last_became_active":"2024-07-26T11:10:35.999600+0000","last_became_peered":"2024-07-26T11:10:35.999600+0000","last_unstale":"2024-07-26T11:12:51.522449+0000","last_undegraded":"2024-07-26T11:12:51.522449+0000","last_fullsized":"2024-07-26T11:12:51.522449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:35:48.504288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.858816+0000","last_change":"2024-07-26T11:10:22.298152+0000","last_active":"2024-07-26T11:13:33.858816+0000","last_peered":"2024-07-26T11:13:33.858816+0000","last_clean":"2024-07-26T11:13:33.858816+0000","last_became_active":"2024-07-26T11:10:22.297739+0000","last_became_peered":"2024-07-26T11:10:22.297739+0000","last_unstale":"2024-07-26T11:13:33.858816+0000","last_undegraded":"2024-07-26T11:13:33.858816+0000","last_fullsized":"2024-07-26T11:13:33.858816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:48.160183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:28.950352+0000","last_change":"2024-07-26T11:10:18.057756+0000","last_active":"2024-07-26T11:12:28.950352+0000","last_peered":"2024-07-26T11:12:28.950352+0000","last_clean":"2024-07-26T11:12:28.950352+0000","last_became_active":"2024-07-26T11:10:18.057464+0000","last_became_peered":"2024-07-26T11:10:18.057464+0000","last_unstale":"2024-07-26T11:12:28.950352+0000","last_undegraded":"2024-07-26T11:12:28.950352+0000","last_fullsized":"2024-07-26T11:12:28.950352+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:39:11.523643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:54.940757+0000","last_change":"2024-07-26T11:10:36.000617+0000","last_active":"2024-07-26T11:12:54.940757+0000","last_peered":"2024-07-26T11:12:54.940757+0000","last_clean":"2024-07-26T11:12:54.940757+0000","last_became_active":"2024-07-26T11:10:36.000151+0000","last_became_peered":"2024-07-26T11:10:36.000151+0000","last_unstale":"2024-07-26T11:12:54.940757+0000","last_undegraded":"2024-07-26T11:12:54.940757+0000","last_fullsized":"2024-07-26T11:12:54.940757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:36.942215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.893390+0000","last_change":"2024-07-26T11:10:22.336452+0000","last_active":"2024-07-26T11:13:33.893390+0000","last_peered":"2024-07-26T11:13:33.893390+0000","last_clean":"2024-07-26T11:13:33.893390+0000","last_became_active":"2024-07-26T11:10:22.336006+0000","last_became_peered":"2024-07-26T11:10:22.336006+0000","last_unstale":"2024-07-26T11:13:33.893390+0000","last_undegraded":"2024-07-26T11:13:33.893390+0000","last_fullsized":"2024-07-26T11:13:33.893390+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:08:27.375730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:29.937948+0000","last_change":"2024-07-26T11:10:18.823017+0000","last_active":"2024-07-26T11:12:29.937948+0000","last_peered":"2024-07-26T11:12:29.937948+0000","last_clean":"2024-07-26T11:12:29.937948+0000","last_became_active":"2024-07-26T11:10:18.822559+0000","last_became_peered":"2024-07-26T11:10:18.822559+0000","last_unstale":"2024-07-26T11:12:29.937948+0000","last_undegraded":"2024-07-26T11:12:29.937948+0000","last_fullsized":"2024-07-26T11:12:29.937948+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:01.486263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:44.913501+0000","last_change":"2024-07-26T11:10:35.969698+0000","last_active":"2024-07-26T11:12:44.913501+0000","last_peered":"2024-07-26T11:12:44.913501+0000","last_clean":"2024-07-26T11:12:44.913501+0000","last_became_active":"2024-07-26T11:10:35.969240+0000","last_became_peered":"2024-07-26T11:10:35.969240+0000","last_unstale":"2024-07-26T11:12:44.913501+0000","last_undegraded":"2024-07-26T11:12:44.913501+0000","last_fullsized":"2024-07-26T11:12:44.913501+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:25.210862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875375+0000","last_change":"2024-07-26T11:10:22.314772+0000","last_active":"2024-07-26T11:13:33.875375+0000","last_peered":"2024-07-26T11:13:33.875375+0000","last_clean":"2024-07-26T11:13:33.875375+0000","last_became_active":"2024-07-26T11:10:22.313786+0000","last_became_peered":"2024-07-26T11:10:22.313786+0000","last_unstale":"2024-07-26T11:13:33.875375+0000","last_undegraded":"2024-07-26T11:13:33.875375+0000","last_fullsized":"2024-07-26T11:13:33.875375+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:19.610695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'18","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:14:41.543437+0000","last_change":"2024-07-26T11:10:18.049408+0000","last_active":"2024-07-26T11:14:41.543437+0000","last_peered":"2024-07-26T11:14:41.543437+0000","last_clean":"2024-07-26T11:14:41.543437+0000","last_became_active":"2024-07-26T11:10:18.048573+0000","last_became_peered":"2024-07-26T11:10:18.048573+0000","last_unstale":"2024-07-26T11:14:41.543437+0000","last_undegraded":"2024-07-26T11:14:41.543437+0000","last_fullsized":"2024-07-26T11:14:41.543437+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:26.408260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:39.065841+0000","last_change":"2024-07-26T11:10:35.992067+0000","last_active":"2024-07-26T11:12:39.065841+0000","last_peered":"2024-07-26T11:12:39.065841+0000","last_clean":"2024-07-26T11:12:39.065841+0000","last_became_active":"2024-07-26T11:10:35.989789+0000","last_became_peered":"2024-07-26T11:10:35.989789+0000","last_unstale":"2024-07-26T11:12:39.065841+0000","last_undegraded":"2024-07-26T11:12:39.065841+0000","last_fullsized":"2024-07-26T11:12:39.065841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:08:08.283592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.888864+0000","last_change":"2024-07-26T11:10:22.323512+0000","last_active":"2024-07-26T11:13:33.888864+0000","last_peered":"2024-07-26T11:13:33.888864+0000","last_clean":"2024-07-26T11:13:33.888864+0000","last_became_active":"2024-07-26T11:10:22.323126+0000","last_became_peered":"2024-07-26T11:10:22.323126+0000","last_unstale":"2024-07-26T11:13:33.888864+0000","last_undegraded":"2024-07-26T11:13:33.888864+0000","last_fullsized":"2024-07-26T11:13:33.888864+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:27:15.379345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:22.159418+0000","last_change":"2024-07-26T11:10:18.052573+0000","last_active":"2024-07-26T11:12:22.159418+0000","last_peered":"2024-07-26T11:12:22.159418+0000","last_clean":"2024-07-26T11:12:22.159418+0000","last_became_active":"2024-07-26T11:10:18.052302+0000","last_became_peered":"2024-07-26T11:10:18.052302+0000","last_unstale":"2024-07-26T11:12:22.159418+0000","last_undegraded":"2024-07-26T11:12:22.159418+0000","last_fullsized":"2024-07-26T11:12:22.159418+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:32.260966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:42.508751+0000","last_change":"2024-07-26T11:10:35.994347+0000","last_active":"2024-07-26T11:12:42.508751+0000","last_peered":"2024-07-26T11:12:42.508751+0000","last_clean":"2024-07-26T11:12:42.508751+0000","last_became_active":"2024-07-26T11:10:35.993795+0000","last_became_peered":"2024-07-26T11:10:35.993795+0000","last_unstale":"2024-07-26T11:12:42.508751+0000","last_undegraded":"2024-07-26T11:12:42.508751+0000","last_fullsized":"2024-07-26T11:12:42.508751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:35.399601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.881857+0000","last_change":"2024-07-26T11:10:22.318109+0000","last_active":"2024-07-26T11:13:33.881857+0000","last_peered":"2024-07-26T11:13:33.881857+0000","last_clean":"2024-07-26T11:13:33.881857+0000","last_became_active":"2024-07-26T11:10:22.317674+0000","last_became_peered":"2024-07-26T11:10:22.317674+0000","last_unstale":"2024-07-26T11:13:33.881857+0000","last_undegraded":"2024-07-26T11:13:33.881857+0000","last_fullsized":"2024-07-26T11:13:33.881857+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:29.464772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:26.517734+0000","last_change":"2024-07-26T11:10:18.060073+0000","last_active":"2024-07-26T11:12:26.517734+0000","last_peered":"2024-07-26T11:12:26.517734+0000","last_clean":"2024-07-26T11:12:26.517734+0000","last_became_active":"2024-07-26T11:10:18.059381+0000","last_became_peered":"2024-07-26T11:10:18.059381+0000","last_unstale":"2024-07-26T11:12:26.517734+0000","last_undegraded":"2024-07-26T11:12:26.517734+0000","last_fullsized":"2024-07-26T11:12:26.517734+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:04:21.184759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:42.162020+0000","last_change":"2024-07-26T11:11:35.951234+0000","last_active":"2024-07-26T11:12:42.162020+0000","last_peered":"2024-07-26T11:12:42.162020+0000","last_clean":"2024-07-26T11:12:42.162020+0000","last_became_active":"2024-07-26T11:11:35.950821+0000","last_became_peered":"2024-07-26T11:11:35.950821+0000","last_unstale":"2024-07-26T11:12:42.162020+0000","last_undegraded":"2024-07-26T11:12:42.162020+0000","last_fullsized":"2024-07-26T11:12:42.162020+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:44:22.569238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.873134+0000","last_change":"2024-07-26T11:10:22.326866+0000","last_active":"2024-07-26T11:13:33.873134+0000","last_peered":"2024-07-26T11:13:33.873134+0000","last_clean":"2024-07-26T11:13:33.873134+0000","last_became_active":"2024-07-26T11:10:22.326417+0000","last_became_peered":"2024-07-26T11:10:22.326417+0000","last_unstale":"2024-07-26T11:13:33.873134+0000","last_undegraded":"2024-07-26T11:13:33.873134+0000","last_fullsized":"2024-07-26T11:13:33.873134+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:49:02.871062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":290,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:50.588141+0000","last_change":"2024-07-26T11:10:18.044077+0000","last_active":"2024-07-26T11:15:50.588141+0000","last_peered":"2024-07-26T11:15:50.588141+0000","last_clean":"2024-07-26T11:15:50.588141+0000","last_became_active":"2024-07-26T11:10:18.043660+0000","last_became_peered":"2024-07-26T11:10:18.043660+0000","last_unstale":"2024-07-26T11:15:50.588141+0000","last_undegraded":"2024-07-26T11:15:50.588141+0000","last_fullsized":"2024-07-26T11:15:50.588141+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:46:54.901082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:41.781727+0000","last_change":"2024-07-26T11:10:35.984778+0000","last_active":"2024-07-26T11:12:41.781727+0000","last_peered":"2024-07-26T11:12:41.781727+0000","last_clean":"2024-07-26T11:12:41.781727+0000","last_became_active":"2024-07-26T11:10:35.984365+0000","last_became_peered":"2024-07-26T11:10:35.984365+0000","last_unstale":"2024-07-26T11:12:41.781727+0000","last_undegraded":"2024-07-26T11:12:41.781727+0000","last_fullsized":"2024-07-26T11:12:41.781727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:33.174121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.867814+0000","last_change":"2024-07-26T11:10:34.976075+0000","last_active":"2024-07-26T11:13:33.867814+0000","last_peered":"2024-07-26T11:13:33.867814+0000","last_clean":"2024-07-26T11:13:33.867814+0000","last_became_active":"2024-07-26T11:10:34.973799+0000","last_became_peered":"2024-07-26T11:10:34.973799+0000","last_unstale":"2024-07-26T11:13:33.867814+0000","last_undegraded":"2024-07-26T11:13:33.867814+0000","last_fullsized":"2024-07-26T11:13:33.867814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:03:01.830655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'7","reported_seq":292,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:50.735982+0000","last_change":"2024-07-26T11:10:18.056473+0000","last_active":"2024-07-26T11:15:50.735982+0000","last_peered":"2024-07-26T11:15:50.735982+0000","last_clean":"2024-07-26T11:15:50.735982+0000","last_became_active":"2024-07-26T11:10:18.056038+0000","last_became_peered":"2024-07-26T11:10:18.056038+0000","last_unstale":"2024-07-26T11:15:50.735982+0000","last_undegraded":"2024-07-26T11:15:50.735982+0000","last_fullsized":"2024-07-26T11:15:50.735982+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:04.501018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:46.782896+0000","last_change":"2024-07-26T11:10:35.991741+0000","last_active":"2024-07-26T11:12:46.782896+0000","last_peered":"2024-07-26T11:12:46.782896+0000","last_clean":"2024-07-26T11:12:46.782896+0000","last_became_active":"2024-07-26T11:10:35.990995+0000","last_became_peered":"2024-07-26T11:10:35.990995+0000","last_unstale":"2024-07-26T11:12:46.782896+0000","last_undegraded":"2024-07-26T11:12:46.782896+0000","last_fullsized":"2024-07-26T11:12:46.782896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:59:34.056646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.884331+0000","last_change":"2024-07-26T11:10:22.305053+0000","last_active":"2024-07-26T11:13:33.884331+0000","last_peered":"2024-07-26T11:13:33.884331+0000","last_clean":"2024-07-26T11:13:33.884331+0000","last_became_active":"2024-07-26T11:10:22.304035+0000","last_became_peered":"2024-07-26T11:10:22.304035+0000","last_unstale":"2024-07-26T11:13:33.884331+0000","last_undegraded":"2024-07-26T11:13:33.884331+0000","last_fullsized":"2024-07-26T11:13:33.884331+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:48.590799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:17.349794+0000","last_change":"2024-07-26T11:10:18.033581+0000","last_active":"2024-07-26T11:12:17.349794+0000","last_peered":"2024-07-26T11:12:17.349794+0000","last_clean":"2024-07-26T11:12:17.349794+0000","last_became_active":"2024-07-26T11:10:18.033038+0000","last_became_peered":"2024-07-26T11:10:18.033038+0000","last_unstale":"2024-07-26T11:12:17.349794+0000","last_undegraded":"2024-07-26T11:12:17.349794+0000","last_fullsized":"2024-07-26T11:12:17.349794+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:19:52.523006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:48.387395+0000","last_change":"2024-07-26T11:11:35.643683+0000","last_active":"2024-07-26T11:12:48.387395+0000","last_peered":"2024-07-26T11:12:48.387395+0000","last_clean":"2024-07-26T11:12:48.387395+0000","last_became_active":"2024-07-26T11:11:35.643109+0000","last_became_peered":"2024-07-26T11:11:35.643109+0000","last_unstale":"2024-07-26T11:12:48.387395+0000","last_undegraded":"2024-07-26T11:12:48.387395+0000","last_fullsized":"2024-07-26T11:12:48.387395+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:39:28.190726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.878540+0000","last_change":"2024-07-26T11:10:34.977690+0000","last_active":"2024-07-26T11:13:33.878540+0000","last_peered":"2024-07-26T11:13:33.878540+0000","last_clean":"2024-07-26T11:13:33.878540+0000","last_became_active":"2024-07-26T11:10:34.976938+0000","last_became_peered":"2024-07-26T11:10:34.976938+0000","last_unstale":"2024-07-26T11:13:33.878540+0000","last_undegraded":"2024-07-26T11:13:33.878540+0000","last_fullsized":"2024-07-26T11:13:33.878540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:26:21.510469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:24.937213+0000","last_change":"2024-07-26T11:10:18.045363+0000","last_active":"2024-07-26T11:12:24.937213+0000","last_peered":"2024-07-26T11:12:24.937213+0000","last_clean":"2024-07-26T11:12:24.937213+0000","last_became_active":"2024-07-26T11:10:18.044810+0000","last_became_peered":"2024-07-26T11:10:18.044810+0000","last_unstale":"2024-07-26T11:12:24.937213+0000","last_undegraded":"2024-07-26T11:12:24.937213+0000","last_fullsized":"2024-07-26T11:12:24.937213+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:34.507172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":120,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:27.358372+0000","last_change":"2024-07-26T11:08:48.936784+0000","last_active":"2024-07-26T11:13:27.358372+0000","last_peered":"2024-07-26T11:13:27.358372+0000","last_clean":"2024-07-26T11:13:27.358372+0000","last_became_active":"2024-07-26T11:08:48.931118+0000","last_became_peered":"2024-07-26T11:08:48.931118+0000","last_unstale":"2024-07-26T11:13:27.358372+0000","last_undegraded":"2024-07-26T11:13:27.358372+0000","last_fullsized":"2024-07-26T11:13:27.358372+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:05:01.918569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:05:01.918569+0000","last_clean_scrub_stamp":"2024-07-26T11:05:01.918569+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:57.128424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:42.352841+0000","last_change":"2024-07-26T11:11:35.385528+0000","last_active":"2024-07-26T11:12:42.352841+0000","last_peered":"2024-07-26T11:12:42.352841+0000","last_clean":"2024-07-26T11:12:42.352841+0000","last_became_active":"2024-07-26T11:11:35.385042+0000","last_became_peered":"2024-07-26T11:11:35.385042+0000","last_unstale":"2024-07-26T11:12:42.352841+0000","last_undegraded":"2024-07-26T11:12:42.352841+0000","last_fullsized":"2024-07-26T11:12:42.352841+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:52:30.343736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875558+0000","last_change":"2024-07-26T11:10:22.302715+0000","last_active":"2024-07-26T11:13:33.875558+0000","last_peered":"2024-07-26T11:13:33.875558+0000","last_clean":"2024-07-26T11:13:33.875558+0000","last_became_active":"2024-07-26T11:10:22.302284+0000","last_became_peered":"2024-07-26T11:10:22.302284+0000","last_unstale":"2024-07-26T11:13:33.875558+0000","last_undegraded":"2024-07-26T11:13:33.875558+0000","last_fullsized":"2024-07-26T11:13:33.875558+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:13:05.555439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":289,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:48.477472+0000","last_change":"2024-07-26T11:10:18.823160+0000","last_active":"2024-07-26T11:15:48.477472+0000","last_peered":"2024-07-26T11:15:48.477472+0000","last_clean":"2024-07-26T11:15:48.477472+0000","last_became_active":"2024-07-26T11:10:18.822652+0000","last_became_peered":"2024-07-26T11:10:18.822652+0000","last_unstale":"2024-07-26T11:15:48.477472+0000","last_undegraded":"2024-07-26T11:15:48.477472+0000","last_fullsized":"2024-07-26T11:15:48.477472+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:21:06.077784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":241,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:48.952701+0000","last_change":"2024-07-26T11:10:35.984007+0000","last_active":"2024-07-26T11:12:48.952701+0000","last_peered":"2024-07-26T11:12:48.952701+0000","last_clean":"2024-07-26T11:12:48.952701+0000","last_became_active":"2024-07-26T11:10:35.983352+0000","last_became_peered":"2024-07-26T11:10:35.983352+0000","last_unstale":"2024-07-26T11:12:48.952701+0000","last_undegraded":"2024-07-26T11:12:48.952701+0000","last_fullsized":"2024-07-26T11:12:48.952701+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:05:21.199041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874961+0000","last_change":"2024-07-26T11:10:22.320559+0000","last_active":"2024-07-26T11:13:33.874961+0000","last_peered":"2024-07-26T11:13:33.874961+0000","last_clean":"2024-07-26T11:13:33.874961+0000","last_became_active":"2024-07-26T11:10:22.320192+0000","last_became_peered":"2024-07-26T11:10:22.320192+0000","last_unstale":"2024-07-26T11:13:33.874961+0000","last_undegraded":"2024-07-26T11:13:33.874961+0000","last_fullsized":"2024-07-26T11:13:33.874961+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:48:51.774418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:33.385383+0000","last_change":"2024-07-26T11:10:18.044108+0000","last_active":"2024-07-26T11:12:33.385383+0000","last_peered":"2024-07-26T11:12:33.385383+0000","last_clean":"2024-07-26T11:12:33.385383+0000","last_became_active":"2024-07-26T11:10:18.043712+0000","last_became_peered":"2024-07-26T11:10:18.043712+0000","last_unstale":"2024-07-26T11:12:33.385383+0000","last_undegraded":"2024-07-26T11:12:33.385383+0000","last_fullsized":"2024-07-26T11:12:33.385383+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:05:34.382520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:54.249813+0000","last_change":"2024-07-26T11:10:35.999071+0000","last_active":"2024-07-26T11:12:54.249813+0000","last_peered":"2024-07-26T11:12:54.249813+0000","last_clean":"2024-07-26T11:12:54.249813+0000","last_became_active":"2024-07-26T11:10:35.997179+0000","last_became_peered":"2024-07-26T11:10:35.997179+0000","last_unstale":"2024-07-26T11:12:54.249813+0000","last_undegraded":"2024-07-26T11:12:54.249813+0000","last_fullsized":"2024-07-26T11:12:54.249813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:36:45.632086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.872320+0000","last_change":"2024-07-26T11:10:22.300356+0000","last_active":"2024-07-26T11:13:33.872320+0000","last_peered":"2024-07-26T11:13:33.872320+0000","last_clean":"2024-07-26T11:13:33.872320+0000","last_became_active":"2024-07-26T11:10:22.299949+0000","last_became_peered":"2024-07-26T11:10:22.299949+0000","last_unstale":"2024-07-26T11:13:33.872320+0000","last_undegraded":"2024-07-26T11:13:33.872320+0000","last_fullsized":"2024-07-26T11:13:33.872320+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:34.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:36.780361+0000","last_change":"2024-07-26T11:10:18.049965+0000","last_active":"2024-07-26T11:12:36.780361+0000","last_peered":"2024-07-26T11:12:36.780361+0000","last_clean":"2024-07-26T11:12:36.780361+0000","last_became_active":"2024-07-26T11:10:18.049251+0000","last_became_peered":"2024-07-26T11:10:18.049251+0000","last_unstale":"2024-07-26T11:12:36.780361+0000","last_undegraded":"2024-07-26T11:12:36.780361+0000","last_fullsized":"2024-07-26T11:12:36.780361+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:20.996609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:48.387315+0000","last_change":"2024-07-26T11:10:35.967661+0000","last_active":"2024-07-26T11:12:48.387315+0000","last_peered":"2024-07-26T11:12:48.387315+0000","last_clean":"2024-07-26T11:12:48.387315+0000","last_became_active":"2024-07-26T11:10:35.966867+0000","last_became_peered":"2024-07-26T11:10:35.966867+0000","last_unstale":"2024-07-26T11:12:48.387315+0000","last_undegraded":"2024-07-26T11:12:48.387315+0000","last_fullsized":"2024-07-26T11:12:48.387315+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:43.951319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.867777+0000","last_change":"2024-07-26T11:10:22.303733+0000","last_active":"2024-07-26T11:13:33.867777+0000","last_peered":"2024-07-26T11:13:33.867777+0000","last_clean":"2024-07-26T11:13:33.867777+0000","last_became_active":"2024-07-26T11:10:22.303322+0000","last_became_peered":"2024-07-26T11:10:22.303322+0000","last_unstale":"2024-07-26T11:13:33.867777+0000","last_undegraded":"2024-07-26T11:13:33.867777+0000","last_fullsized":"2024-07-26T11:13:33.867777+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:34:34.962492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:32.160782+0000","last_change":"2024-07-26T11:10:18.032539+0000","last_active":"2024-07-26T11:12:32.160782+0000","last_peered":"2024-07-26T11:12:32.160782+0000","last_clean":"2024-07-26T11:12:32.160782+0000","last_became_active":"2024-07-26T11:10:18.032059+0000","last_became_peered":"2024-07-26T11:10:18.032059+0000","last_unstale":"2024-07-26T11:12:32.160782+0000","last_undegraded":"2024-07-26T11:12:32.160782+0000","last_fullsized":"2024-07-26T11:12:32.160782+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:54.502339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:48.387339+0000","last_change":"2024-07-26T11:10:35.977473+0000","last_active":"2024-07-26T11:12:48.387339+0000","last_peered":"2024-07-26T11:12:48.387339+0000","last_clean":"2024-07-26T11:12:48.387339+0000","last_became_active":"2024-07-26T11:10:35.976830+0000","last_became_peered":"2024-07-26T11:10:35.976830+0000","last_unstale":"2024-07-26T11:12:48.387339+0000","last_undegraded":"2024-07-26T11:12:48.387339+0000","last_fullsized":"2024-07-26T11:12:48.387339+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:46:13.009246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875320+0000","last_change":"2024-07-26T11:10:22.304358+0000","last_active":"2024-07-26T11:13:33.875320+0000","last_peered":"2024-07-26T11:13:33.875320+0000","last_clean":"2024-07-26T11:13:33.875320+0000","last_became_active":"2024-07-26T11:10:22.303846+0000","last_became_peered":"2024-07-26T11:10:22.303846+0000","last_unstale":"2024-07-26T11:13:33.875320+0000","last_undegraded":"2024-07-26T11:13:33.875320+0000","last_fullsized":"2024-07-26T11:13:33.875320+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:27:28.556481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:37.161406+0000","last_change":"2024-07-26T11:10:18.057678+0000","last_active":"2024-07-26T11:12:37.161406+0000","last_peered":"2024-07-26T11:12:37.161406+0000","last_clean":"2024-07-26T11:12:37.161406+0000","last_became_active":"2024-07-26T11:10:18.057282+0000","last_became_peered":"2024-07-26T11:10:18.057282+0000","last_unstale":"2024-07-26T11:12:37.161406+0000","last_undegraded":"2024-07-26T11:12:37.161406+0000","last_fullsized":"2024-07-26T11:12:37.161406+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:08:35.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:01.785766+0000","last_change":"2024-07-26T11:10:36.000329+0000","last_active":"2024-07-26T11:13:01.785766+0000","last_peered":"2024-07-26T11:13:01.785766+0000","last_clean":"2024-07-26T11:13:01.785766+0000","last_became_active":"2024-07-26T11:10:35.999713+0000","last_became_peered":"2024-07-26T11:10:35.999713+0000","last_unstale":"2024-07-26T11:13:01.785766+0000","last_undegraded":"2024-07-26T11:13:01.785766+0000","last_fullsized":"2024-07-26T11:13:01.785766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:48.137237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.866953+0000","last_change":"2024-07-26T11:10:22.301526+0000","last_active":"2024-07-26T11:13:33.866953+0000","last_peered":"2024-07-26T11:13:33.866953+0000","last_clean":"2024-07-26T11:13:33.866953+0000","last_became_active":"2024-07-26T11:10:22.301098+0000","last_became_peered":"2024-07-26T11:10:22.301098+0000","last_unstale":"2024-07-26T11:13:33.866953+0000","last_undegraded":"2024-07-26T11:13:33.866953+0000","last_fullsized":"2024-07-26T11:13:33.866953+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:01:48.118315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'8","reported_seq":295,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:48.731530+0000","last_change":"2024-07-26T11:10:18.035922+0000","last_active":"2024-07-26T11:15:48.731530+0000","last_peered":"2024-07-26T11:15:48.731530+0000","last_clean":"2024-07-26T11:15:48.731530+0000","last_became_active":"2024-07-26T11:10:18.035184+0000","last_became_peered":"2024-07-26T11:10:18.035184+0000","last_unstale":"2024-07-26T11:15:48.731530+0000","last_undegraded":"2024-07-26T11:15:48.731530+0000","last_fullsized":"2024-07-26T11:15:48.731530+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:13:10.371105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1427,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":245,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:01.524560+0000","last_change":"2024-07-26T11:10:35.998791+0000","last_active":"2024-07-26T11:13:01.524560+0000","last_peered":"2024-07-26T11:13:01.524560+0000","last_clean":"2024-07-26T11:13:01.524560+0000","last_became_active":"2024-07-26T11:10:35.997163+0000","last_became_peered":"2024-07-26T11:10:35.997163+0000","last_unstale":"2024-07-26T11:13:01.524560+0000","last_undegraded":"2024-07-26T11:13:01.524560+0000","last_fullsized":"2024-07-26T11:13:01.524560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:28:50.167294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889494+0000","last_change":"2024-07-26T11:10:22.314219+0000","last_active":"2024-07-26T11:13:33.889494+0000","last_peered":"2024-07-26T11:13:33.889494+0000","last_clean":"2024-07-26T11:13:33.889494+0000","last_became_active":"2024-07-26T11:10:22.313811+0000","last_became_peered":"2024-07-26T11:10:22.313811+0000","last_unstale":"2024-07-26T11:13:33.889494+0000","last_undegraded":"2024-07-26T11:13:33.889494+0000","last_fullsized":"2024-07-26T11:13:33.889494+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:55:47.452798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:38.385955+0000","last_change":"2024-07-26T11:10:18.047818+0000","last_active":"2024-07-26T11:12:38.385955+0000","last_peered":"2024-07-26T11:12:38.385955+0000","last_clean":"2024-07-26T11:12:38.385955+0000","last_became_active":"2024-07-26T11:10:18.047159+0000","last_became_peered":"2024-07-26T11:10:18.047159+0000","last_unstale":"2024-07-26T11:12:38.385955+0000","last_undegraded":"2024-07-26T11:12:38.385955+0000","last_fullsized":"2024-07-26T11:12:38.385955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:37.528512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:57.510826+0000","last_change":"2024-07-26T11:10:35.996468+0000","last_active":"2024-07-26T11:12:57.510826+0000","last_peered":"2024-07-26T11:12:57.510826+0000","last_clean":"2024-07-26T11:12:57.510826+0000","last_became_active":"2024-07-26T11:10:35.996030+0000","last_became_peered":"2024-07-26T11:10:35.996030+0000","last_unstale":"2024-07-26T11:12:57.510826+0000","last_undegraded":"2024-07-26T11:12:57.510826+0000","last_fullsized":"2024-07-26T11:12:57.510826+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:20:14.743690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.881988+0000","last_change":"2024-07-26T11:10:22.318375+0000","last_active":"2024-07-26T11:13:33.881988+0000","last_peered":"2024-07-26T11:13:33.881988+0000","last_clean":"2024-07-26T11:13:33.881988+0000","last_became_active":"2024-07-26T11:10:22.317965+0000","last_became_peered":"2024-07-26T11:10:22.317965+0000","last_unstale":"2024-07-26T11:13:33.881988+0000","last_undegraded":"2024-07-26T11:13:33.881988+0000","last_fullsized":"2024-07-26T11:13:33.881988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:36:43.504045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:39.912995+0000","last_change":"2024-07-26T11:10:18.047957+0000","last_active":"2024-07-26T11:12:39.912995+0000","last_peered":"2024-07-26T11:12:39.912995+0000","last_clean":"2024-07-26T11:12:39.912995+0000","last_became_active":"2024-07-26T11:10:18.047495+0000","last_became_peered":"2024-07-26T11:10:18.047495+0000","last_unstale":"2024-07-26T11:12:39.912995+0000","last_undegraded":"2024-07-26T11:12:39.912995+0000","last_fullsized":"2024-07-26T11:12:39.912995+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:11.238295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:03.389374+0000","last_change":"2024-07-26T11:10:35.976946+0000","last_active":"2024-07-26T11:13:03.389374+0000","last_peered":"2024-07-26T11:13:03.389374+0000","last_clean":"2024-07-26T11:13:03.389374+0000","last_became_active":"2024-07-26T11:10:35.976474+0000","last_became_peered":"2024-07-26T11:10:35.976474+0000","last_unstale":"2024-07-26T11:13:03.389374+0000","last_undegraded":"2024-07-26T11:13:03.389374+0000","last_fullsized":"2024-07-26T11:13:03.389374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:50:51.240903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.859135+0000","last_change":"2024-07-26T11:10:22.302473+0000","last_active":"2024-07-26T11:13:33.859135+0000","last_peered":"2024-07-26T11:13:33.859135+0000","last_clean":"2024-07-26T11:13:33.859135+0000","last_became_active":"2024-07-26T11:10:22.302096+0000","last_became_peered":"2024-07-26T11:10:22.302096+0000","last_unstale":"2024-07-26T11:13:33.859135+0000","last_undegraded":"2024-07-26T11:13:33.859135+0000","last_fullsized":"2024-07-26T11:13:33.859135+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:19.215301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'43","reported_seq":599,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:51.482116+0000","last_change":"2024-07-26T11:10:34.973502+0000","last_active":"2024-07-26T11:15:51.482116+0000","last_peered":"2024-07-26T11:15:51.482116+0000","last_clean":"2024-07-26T11:15:51.482116+0000","last_became_active":"2024-07-26T11:10:34.964155+0000","last_became_peered":"2024-07-26T11:10:34.964155+0000","last_unstale":"2024-07-26T11:15:51.482116+0000","last_undegraded":"2024-07-26T11:15:51.482116+0000","last_fullsized":"2024-07-26T11:15:51.482116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:23:06.652746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72449,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":496,"num_read_kb":320,"num_write":45,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:56.523464+0000","last_change":"2024-07-26T11:10:35.981439+0000","last_active":"2024-07-26T11:12:56.523464+0000","last_peered":"2024-07-26T11:12:56.523464+0000","last_clean":"2024-07-26T11:12:56.523464+0000","last_became_active":"2024-07-26T11:10:35.980878+0000","last_became_peered":"2024-07-26T11:10:35.980878+0000","last_unstale":"2024-07-26T11:12:56.523464+0000","last_undegraded":"2024-07-26T11:12:56.523464+0000","last_fullsized":"2024-07-26T11:12:56.523464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:31.719845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889855+0000","last_change":"2024-07-26T11:10:34.974292+0000","last_active":"2024-07-26T11:13:33.889855+0000","last_peered":"2024-07-26T11:13:33.889855+0000","last_clean":"2024-07-26T11:13:33.889855+0000","last_became_active":"2024-07-26T11:10:34.973820+0000","last_became_peered":"2024-07-26T11:10:34.973820+0000","last_unstale":"2024-07-26T11:13:33.889855+0000","last_undegraded":"2024-07-26T11:13:33.889855+0000","last_fullsized":"2024-07-26T11:13:33.889855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:35:37.629313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:43.386694+0000","last_change":"2024-07-26T11:10:18.055248+0000","last_active":"2024-07-26T11:12:43.386694+0000","last_peered":"2024-07-26T11:12:43.386694+0000","last_clean":"2024-07-26T11:12:43.386694+0000","last_became_active":"2024-07-26T11:10:18.054882+0000","last_became_peered":"2024-07-26T11:10:18.054882+0000","last_unstale":"2024-07-26T11:12:43.386694+0000","last_undegraded":"2024-07-26T11:12:43.386694+0000","last_fullsized":"2024-07-26T11:12:43.386694+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:16.390058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:59.941475+0000","last_change":"2024-07-26T11:10:35.985628+0000","last_active":"2024-07-26T11:12:59.941475+0000","last_peered":"2024-07-26T11:12:59.941475+0000","last_clean":"2024-07-26T11:12:59.941475+0000","last_became_active":"2024-07-26T11:10:35.985156+0000","last_became_peered":"2024-07-26T11:10:35.985156+0000","last_unstale":"2024-07-26T11:12:59.941475+0000","last_undegraded":"2024-07-26T11:12:59.941475+0000","last_fullsized":"2024-07-26T11:12:59.941475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:29:18.891825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.893557+0000","last_change":"2024-07-26T11:10:34.973261+0000","last_active":"2024-07-26T11:13:33.893557+0000","last_peered":"2024-07-26T11:13:33.893557+0000","last_clean":"2024-07-26T11:13:33.893557+0000","last_became_active":"2024-07-26T11:10:34.971592+0000","last_became_peered":"2024-07-26T11:10:34.971592+0000","last_unstale":"2024-07-26T11:13:33.893557+0000","last_undegraded":"2024-07-26T11:13:33.893557+0000","last_fullsized":"2024-07-26T11:13:33.893557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:58:13.517884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:42.352859+0000","last_change":"2024-07-26T11:10:34.962248+0000","last_active":"2024-07-26T11:12:42.352859+0000","last_peered":"2024-07-26T11:12:42.352859+0000","last_clean":"2024-07-26T11:12:42.352859+0000","last_became_active":"2024-07-26T11:10:34.961683+0000","last_became_peered":"2024-07-26T11:10:34.961683+0000","last_unstale":"2024-07-26T11:12:42.352859+0000","last_undegraded":"2024-07-26T11:12:42.352859+0000","last_fullsized":"2024-07-26T11:12:42.352859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:17:08.308226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:56.523407+0000","last_change":"2024-07-26T11:10:35.995760+0000","last_active":"2024-07-26T11:12:56.523407+0000","last_peered":"2024-07-26T11:12:56.523407+0000","last_clean":"2024-07-26T11:12:56.523407+0000","last_became_active":"2024-07-26T11:10:35.995328+0000","last_became_peered":"2024-07-26T11:10:35.995328+0000","last_unstale":"2024-07-26T11:12:56.523407+0000","last_undegraded":"2024-07-26T11:12:56.523407+0000","last_fullsized":"2024-07-26T11:12:56.523407+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:40:33.093315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.878655+0000","last_change":"2024-07-26T11:10:22.316151+0000","last_active":"2024-07-26T11:13:33.878655+0000","last_peered":"2024-07-26T11:13:33.878655+0000","last_clean":"2024-07-26T11:13:33.878655+0000","last_became_active":"2024-07-26T11:10:22.315722+0000","last_became_peered":"2024-07-26T11:10:22.315722+0000","last_unstale":"2024-07-26T11:13:33.878655+0000","last_undegraded":"2024-07-26T11:13:33.878655+0000","last_fullsized":"2024-07-26T11:13:33.878655+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:12:58.342637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":529,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:51.481269+0000","last_change":"2024-07-26T11:10:18.046327+0000","last_active":"2024-07-26T11:15:51.481269+0000","last_peered":"2024-07-26T11:15:51.481269+0000","last_clean":"2024-07-26T11:15:51.481269+0000","last_became_active":"2024-07-26T11:10:18.045913+0000","last_became_peered":"2024-07-26T11:10:18.045913+0000","last_unstale":"2024-07-26T11:15:51.481269+0000","last_undegraded":"2024-07-26T11:15:51.481269+0000","last_fullsized":"2024-07-26T11:15:51.481269+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:58.889046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":482,"num_read_kb":482,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:59.069724+0000","last_change":"2024-07-26T11:10:35.983264+0000","last_active":"2024-07-26T11:12:59.069724+0000","last_peered":"2024-07-26T11:12:59.069724+0000","last_clean":"2024-07-26T11:12:59.069724+0000","last_became_active":"2024-07-26T11:10:35.982210+0000","last_became_peered":"2024-07-26T11:10:35.982210+0000","last_unstale":"2024-07-26T11:12:59.069724+0000","last_undegraded":"2024-07-26T11:12:59.069724+0000","last_fullsized":"2024-07-26T11:12:59.069724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:37:20.272725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:39.246875+0000","last_change":"2024-07-26T11:10:35.985335+0000","last_active":"2024-07-26T11:12:39.246875+0000","last_peered":"2024-07-26T11:12:39.246875+0000","last_clean":"2024-07-26T11:12:39.246875+0000","last_became_active":"2024-07-26T11:10:35.983667+0000","last_became_peered":"2024-07-26T11:10:35.983667+0000","last_unstale":"2024-07-26T11:12:39.246875+0000","last_undegraded":"2024-07-26T11:12:39.246875+0000","last_fullsized":"2024-07-26T11:12:39.246875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:00:17.744806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.879478+0000","last_change":"2024-07-26T11:10:22.333494+0000","last_active":"2024-07-26T11:13:33.879478+0000","last_peered":"2024-07-26T11:13:33.879478+0000","last_clean":"2024-07-26T11:13:33.879478+0000","last_became_active":"2024-07-26T11:10:22.332950+0000","last_became_peered":"2024-07-26T11:10:22.332950+0000","last_unstale":"2024-07-26T11:13:33.879478+0000","last_undegraded":"2024-07-26T11:13:33.879478+0000","last_fullsized":"2024-07-26T11:13:33.879478+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:58:38.340857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:43.386643+0000","last_change":"2024-07-26T11:10:18.034057+0000","last_active":"2024-07-26T11:12:43.386643+0000","last_peered":"2024-07-26T11:12:43.386643+0000","last_clean":"2024-07-26T11:12:43.386643+0000","last_became_active":"2024-07-26T11:10:18.033119+0000","last_became_peered":"2024-07-26T11:10:18.033119+0000","last_unstale":"2024-07-26T11:12:43.386643+0000","last_undegraded":"2024-07-26T11:12:43.386643+0000","last_fullsized":"2024-07-26T11:12:43.386643+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:21.675593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:42.508793+0000","last_change":"2024-07-26T11:11:35.408377+0000","last_active":"2024-07-26T11:12:42.508793+0000","last_peered":"2024-07-26T11:12:42.508793+0000","last_clean":"2024-07-26T11:12:42.508793+0000","last_became_active":"2024-07-26T11:11:35.407865+0000","last_became_peered":"2024-07-26T11:11:35.407865+0000","last_unstale":"2024-07-26T11:12:42.508793+0000","last_undegraded":"2024-07-26T11:12:42.508793+0000","last_fullsized":"2024-07-26T11:12:42.508793+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:43:02.953395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889650+0000","last_change":"2024-07-26T11:10:22.323447+0000","last_active":"2024-07-26T11:13:33.889650+0000","last_peered":"2024-07-26T11:13:33.889650+0000","last_clean":"2024-07-26T11:13:33.889650+0000","last_became_active":"2024-07-26T11:10:22.322988+0000","last_became_peered":"2024-07-26T11:10:22.322988+0000","last_unstale":"2024-07-26T11:13:33.889650+0000","last_undegraded":"2024-07-26T11:13:33.889650+0000","last_fullsized":"2024-07-26T11:13:33.889650+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:14.880985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:46.782728+0000","last_change":"2024-07-26T11:10:18.048724+0000","last_active":"2024-07-26T11:12:46.782728+0000","last_peered":"2024-07-26T11:12:46.782728+0000","last_clean":"2024-07-26T11:12:46.782728+0000","last_became_active":"2024-07-26T11:10:18.048257+0000","last_became_peered":"2024-07-26T11:10:18.048257+0000","last_unstale":"2024-07-26T11:12:46.782728+0000","last_undegraded":"2024-07-26T11:12:46.782728+0000","last_fullsized":"2024-07-26T11:12:46.782728+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:02:13.581446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:47.389909+0000","last_change":"2024-07-26T11:10:18.044171+0000","last_active":"2024-07-26T11:12:47.389909+0000","last_peered":"2024-07-26T11:12:47.389909+0000","last_clean":"2024-07-26T11:12:47.389909+0000","last_became_active":"2024-07-26T11:10:18.043682+0000","last_became_peered":"2024-07-26T11:10:18.043682+0000","last_unstale":"2024-07-26T11:12:47.389909+0000","last_undegraded":"2024-07-26T11:12:47.389909+0000","last_fullsized":"2024-07-26T11:12:47.389909+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:46:46.864119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.876589+0000","last_change":"2024-07-26T11:10:22.321081+0000","last_active":"2024-07-26T11:13:33.876589+0000","last_peered":"2024-07-26T11:13:33.876589+0000","last_clean":"2024-07-26T11:13:33.876589+0000","last_became_active":"2024-07-26T11:10:22.320656+0000","last_became_peered":"2024-07-26T11:10:22.320656+0000","last_unstale":"2024-07-26T11:13:33.876589+0000","last_undegraded":"2024-07-26T11:13:33.876589+0000","last_fullsized":"2024-07-26T11:13:33.876589+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:17:20.455129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:09.916452+0000","last_change":"2024-07-26T11:10:36.000224+0000","last_active":"2024-07-26T11:13:09.916452+0000","last_peered":"2024-07-26T11:13:09.916452+0000","last_clean":"2024-07-26T11:13:09.916452+0000","last_became_active":"2024-07-26T11:10:35.999733+0000","last_became_peered":"2024-07-26T11:10:35.999733+0000","last_unstale":"2024-07-26T11:13:09.916452+0000","last_undegraded":"2024-07-26T11:13:09.916452+0000","last_fullsized":"2024-07-26T11:13:09.916452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:49.912401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":292,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:51.481917+0000","last_change":"2024-07-26T11:10:18.034585+0000","last_active":"2024-07-26T11:15:51.481917+0000","last_peered":"2024-07-26T11:15:51.481917+0000","last_clean":"2024-07-26T11:15:51.481917+0000","last_became_active":"2024-07-26T11:10:18.034237+0000","last_became_peered":"2024-07-26T11:10:18.034237+0000","last_unstale":"2024-07-26T11:15:51.481917+0000","last_undegraded":"2024-07-26T11:15:51.481917+0000","last_fullsized":"2024-07-26T11:15:51.481917+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:22.670277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.876583+0000","last_change":"2024-07-26T11:10:22.319282+0000","last_active":"2024-07-26T11:13:33.876583+0000","last_peered":"2024-07-26T11:13:33.876583+0000","last_clean":"2024-07-26T11:13:33.876583+0000","last_became_active":"2024-07-26T11:10:22.318822+0000","last_became_peered":"2024-07-26T11:10:22.318822+0000","last_unstale":"2024-07-26T11:13:33.876583+0000","last_undegraded":"2024-07-26T11:13:33.876583+0000","last_fullsized":"2024-07-26T11:13:33.876583+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:50:57.615839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:09.916483+0000","last_change":"2024-07-26T11:10:35.970189+0000","last_active":"2024-07-26T11:13:09.916483+0000","last_peered":"2024-07-26T11:13:09.916483+0000","last_clean":"2024-07-26T11:13:09.916483+0000","last_became_active":"2024-07-26T11:10:35.969574+0000","last_became_peered":"2024-07-26T11:10:35.969574+0000","last_unstale":"2024-07-26T11:13:09.916483+0000","last_undegraded":"2024-07-26T11:13:09.916483+0000","last_fullsized":"2024-07-26T11:13:09.916483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:07.487944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.868831+0000","last_change":"2024-07-26T11:10:22.299152+0000","last_active":"2024-07-26T11:13:33.868831+0000","last_peered":"2024-07-26T11:13:33.868831+0000","last_clean":"2024-07-26T11:13:33.868831+0000","last_became_active":"2024-07-26T11:10:22.298707+0000","last_became_peered":"2024-07-26T11:10:22.298707+0000","last_unstale":"2024-07-26T11:13:33.868831+0000","last_undegraded":"2024-07-26T11:13:33.868831+0000","last_fullsized":"2024-07-26T11:13:33.868831+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:25.161634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.872159+0000","last_change":"2024-07-26T11:10:22.302476+0000","last_active":"2024-07-26T11:13:33.872159+0000","last_peered":"2024-07-26T11:13:33.872159+0000","last_clean":"2024-07-26T11:13:33.872159+0000","last_became_active":"2024-07-26T11:10:22.302091+0000","last_became_peered":"2024-07-26T11:10:22.302091+0000","last_unstale":"2024-07-26T11:13:33.872159+0000","last_undegraded":"2024-07-26T11:13:33.872159+0000","last_fullsized":"2024-07-26T11:13:33.872159+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:06:18.587588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.892481+0000","last_change":"2024-07-26T11:10:22.326583+0000","last_active":"2024-07-26T11:13:33.892481+0000","last_peered":"2024-07-26T11:13:33.892481+0000","last_clean":"2024-07-26T11:13:33.892481+0000","last_became_active":"2024-07-26T11:10:22.326185+0000","last_became_peered":"2024-07-26T11:10:22.326185+0000","last_unstale":"2024-07-26T11:13:33.892481+0000","last_undegraded":"2024-07-26T11:13:33.892481+0000","last_fullsized":"2024-07-26T11:13:33.892481+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:54:00.148964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.894835+0000","last_change":"2024-07-26T11:10:22.329247+0000","last_active":"2024-07-26T11:13:33.894835+0000","last_peered":"2024-07-26T11:13:33.894835+0000","last_clean":"2024-07-26T11:13:33.894835+0000","last_became_active":"2024-07-26T11:10:22.328585+0000","last_became_peered":"2024-07-26T11:10:22.328585+0000","last_unstale":"2024-07-26T11:13:33.894835+0000","last_undegraded":"2024-07-26T11:13:33.894835+0000","last_fullsized":"2024-07-26T11:13:33.894835+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:01:27.195399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.895664+0000","last_change":"2024-07-26T11:10:22.330717+0000","last_active":"2024-07-26T11:13:33.895664+0000","last_peered":"2024-07-26T11:13:33.895664+0000","last_clean":"2024-07-26T11:13:33.895664+0000","last_became_active":"2024-07-26T11:10:22.329576+0000","last_became_peered":"2024-07-26T11:10:22.329576+0000","last_unstale":"2024-07-26T11:13:33.895664+0000","last_undegraded":"2024-07-26T11:13:33.895664+0000","last_fullsized":"2024-07-26T11:13:33.895664+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:21.970629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.867859+0000","last_change":"2024-07-26T11:10:22.307445+0000","last_active":"2024-07-26T11:13:33.867859+0000","last_peered":"2024-07-26T11:13:33.867859+0000","last_clean":"2024-07-26T11:13:33.867859+0000","last_became_active":"2024-07-26T11:10:22.305738+0000","last_became_peered":"2024-07-26T11:10:22.305738+0000","last_unstale":"2024-07-26T11:13:33.867859+0000","last_undegraded":"2024-07-26T11:13:33.867859+0000","last_fullsized":"2024-07-26T11:13:33.867859+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:09:08.125209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874579+0000","last_change":"2024-07-26T11:10:22.298392+0000","last_active":"2024-07-26T11:13:33.874579+0000","last_peered":"2024-07-26T11:13:33.874579+0000","last_clean":"2024-07-26T11:13:33.874579+0000","last_became_active":"2024-07-26T11:10:22.297979+0000","last_became_peered":"2024-07-26T11:10:22.297979+0000","last_unstale":"2024-07-26T11:13:33.874579+0000","last_undegraded":"2024-07-26T11:13:33.874579+0000","last_fullsized":"2024-07-26T11:13:33.874579+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:30:39.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.873814+0000","last_change":"2024-07-26T11:10:22.320850+0000","last_active":"2024-07-26T11:13:33.873814+0000","last_peered":"2024-07-26T11:13:33.873814+0000","last_clean":"2024-07-26T11:13:33.873814+0000","last_became_active":"2024-07-26T11:10:22.319838+0000","last_became_peered":"2024-07-26T11:10:22.319838+0000","last_unstale":"2024-07-26T11:13:33.873814+0000","last_undegraded":"2024-07-26T11:13:33.873814+0000","last_fullsized":"2024-07-26T11:13:33.873814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:14.289366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.873192+0000","last_change":"2024-07-26T11:10:22.321305+0000","last_active":"2024-07-26T11:13:33.873192+0000","last_peered":"2024-07-26T11:13:33.873192+0000","last_clean":"2024-07-26T11:13:33.873192+0000","last_became_active":"2024-07-26T11:10:22.320733+0000","last_became_peered":"2024-07-26T11:10:22.320733+0000","last_unstale":"2024-07-26T11:13:33.873192+0000","last_undegraded":"2024-07-26T11:13:33.873192+0000","last_fullsized":"2024-07-26T11:13:33.873192+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:24:33.125828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874816+0000","last_change":"2024-07-26T11:10:22.318193+0000","last_active":"2024-07-26T11:13:33.874816+0000","last_peered":"2024-07-26T11:13:33.874816+0000","last_clean":"2024-07-26T11:13:33.874816+0000","last_became_active":"2024-07-26T11:10:22.317819+0000","last_became_peered":"2024-07-26T11:10:22.317819+0000","last_unstale":"2024-07-26T11:13:33.874816+0000","last_undegraded":"2024-07-26T11:13:33.874816+0000","last_fullsized":"2024-07-26T11:13:33.874816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:14:06.105152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875254+0000","last_change":"2024-07-26T11:10:22.318222+0000","last_active":"2024-07-26T11:13:33.875254+0000","last_peered":"2024-07-26T11:13:33.875254+0000","last_clean":"2024-07-26T11:13:33.875254+0000","last_became_active":"2024-07-26T11:10:22.317806+0000","last_became_peered":"2024-07-26T11:10:22.317806+0000","last_unstale":"2024-07-26T11:13:33.875254+0000","last_undegraded":"2024-07-26T11:13:33.875254+0000","last_fullsized":"2024-07-26T11:13:33.875254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:55:27.510542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.895333+0000","last_change":"2024-07-26T11:10:22.330166+0000","last_active":"2024-07-26T11:13:33.895333+0000","last_peered":"2024-07-26T11:13:33.895333+0000","last_clean":"2024-07-26T11:13:33.895333+0000","last_became_active":"2024-07-26T11:10:22.329722+0000","last_became_peered":"2024-07-26T11:10:22.329722+0000","last_unstale":"2024-07-26T11:13:33.895333+0000","last_undegraded":"2024-07-26T11:13:33.895333+0000","last_fullsized":"2024-07-26T11:13:33.895333+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:58.816139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.866660+0000","last_change":"2024-07-26T11:10:22.302774+0000","last_active":"2024-07-26T11:13:33.866660+0000","last_peered":"2024-07-26T11:13:33.866660+0000","last_clean":"2024-07-26T11:13:33.866660+0000","last_became_active":"2024-07-26T11:10:22.302382+0000","last_became_peered":"2024-07-26T11:10:22.302382+0000","last_unstale":"2024-07-26T11:13:33.866660+0000","last_undegraded":"2024-07-26T11:13:33.866660+0000","last_fullsized":"2024-07-26T11:13:33.866660+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:02:58.019598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.886317+0000","last_change":"2024-07-26T11:10:22.312475+0000","last_active":"2024-07-26T11:13:33.886317+0000","last_peered":"2024-07-26T11:13:33.886317+0000","last_clean":"2024-07-26T11:13:33.886317+0000","last_became_active":"2024-07-26T11:10:22.312078+0000","last_became_peered":"2024-07-26T11:10:22.312078+0000","last_unstale":"2024-07-26T11:13:33.886317+0000","last_undegraded":"2024-07-26T11:13:33.886317+0000","last_fullsized":"2024-07-26T11:13:33.886317+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:25:37.705136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.878921+0000","last_change":"2024-07-26T11:10:22.314776+0000","last_active":"2024-07-26T11:13:33.878921+0000","last_peered":"2024-07-26T11:13:33.878921+0000","last_clean":"2024-07-26T11:13:33.878921+0000","last_became_active":"2024-07-26T11:10:22.307808+0000","last_became_peered":"2024-07-26T11:10:22.307808+0000","last_unstale":"2024-07-26T11:13:33.878921+0000","last_undegraded":"2024-07-26T11:13:33.878921+0000","last_fullsized":"2024-07-26T11:13:33.878921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:44:06.152799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.891140+0000","last_change":"2024-07-26T11:10:22.328102+0000","last_active":"2024-07-26T11:13:33.891140+0000","last_peered":"2024-07-26T11:13:33.891140+0000","last_clean":"2024-07-26T11:13:33.891140+0000","last_became_active":"2024-07-26T11:10:22.327779+0000","last_became_peered":"2024-07-26T11:10:22.327779+0000","last_unstale":"2024-07-26T11:13:33.891140+0000","last_undegraded":"2024-07-26T11:13:33.891140+0000","last_fullsized":"2024-07-26T11:13:33.891140+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:14:34.569748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.888331+0000","last_change":"2024-07-26T11:10:22.329057+0000","last_active":"2024-07-26T11:13:33.888331+0000","last_peered":"2024-07-26T11:13:33.888331+0000","last_clean":"2024-07-26T11:13:33.888331+0000","last_became_active":"2024-07-26T11:10:22.328656+0000","last_became_peered":"2024-07-26T11:10:22.328656+0000","last_unstale":"2024-07-26T11:13:33.888331+0000","last_undegraded":"2024-07-26T11:13:33.888331+0000","last_fullsized":"2024-07-26T11:13:33.888331+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:18:39.861674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.891998+0000","last_change":"2024-07-26T11:10:22.326115+0000","last_active":"2024-07-26T11:13:33.891998+0000","last_peered":"2024-07-26T11:13:33.891998+0000","last_clean":"2024-07-26T11:13:33.891998+0000","last_became_active":"2024-07-26T11:10:22.325538+0000","last_became_peered":"2024-07-26T11:10:22.325538+0000","last_unstale":"2024-07-26T11:13:33.891998+0000","last_undegraded":"2024-07-26T11:13:33.891998+0000","last_fullsized":"2024-07-26T11:13:33.891998+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:18:19.339571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.892537+0000","last_change":"2024-07-26T11:10:22.326910+0000","last_active":"2024-07-26T11:13:33.892537+0000","last_peered":"2024-07-26T11:13:33.892537+0000","last_clean":"2024-07-26T11:13:33.892537+0000","last_became_active":"2024-07-26T11:10:22.324532+0000","last_became_peered":"2024-07-26T11:10:22.324532+0000","last_unstale":"2024-07-26T11:13:33.892537+0000","last_undegraded":"2024-07-26T11:13:33.892537+0000","last_fullsized":"2024-07-26T11:13:33.892537+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:33.671780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.890787+0000","last_change":"2024-07-26T11:10:22.312926+0000","last_active":"2024-07-26T11:13:33.890787+0000","last_peered":"2024-07-26T11:13:33.890787+0000","last_clean":"2024-07-26T11:13:33.890787+0000","last_became_active":"2024-07-26T11:10:22.311642+0000","last_became_peered":"2024-07-26T11:10:22.311642+0000","last_unstale":"2024-07-26T11:13:33.890787+0000","last_undegraded":"2024-07-26T11:13:33.890787+0000","last_fullsized":"2024-07-26T11:13:33.890787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:56:29.644965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.879104+0000","last_change":"2024-07-26T11:10:34.976934+0000","last_active":"2024-07-26T11:13:33.879104+0000","last_peered":"2024-07-26T11:13:33.879104+0000","last_clean":"2024-07-26T11:13:33.879104+0000","last_became_active":"2024-07-26T11:10:34.976551+0000","last_became_peered":"2024-07-26T11:10:34.976551+0000","last_unstale":"2024-07-26T11:13:33.879104+0000","last_undegraded":"2024-07-26T11:13:33.879104+0000","last_fullsized":"2024-07-26T11:13:33.879104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:11:43.883644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875269+0000","last_change":"2024-07-26T11:10:22.304444+0000","last_active":"2024-07-26T11:13:33.875269+0000","last_peered":"2024-07-26T11:13:33.875269+0000","last_clean":"2024-07-26T11:13:33.875269+0000","last_became_active":"2024-07-26T11:10:22.302825+0000","last_became_peered":"2024-07-26T11:10:22.302825+0000","last_unstale":"2024-07-26T11:13:33.875269+0000","last_undegraded":"2024-07-26T11:13:33.875269+0000","last_fullsized":"2024-07-26T11:13:33.875269+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:15.229057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.870763+0000","last_change":"2024-07-26T11:10:22.297515+0000","last_active":"2024-07-26T11:13:33.870763+0000","last_peered":"2024-07-26T11:13:33.870763+0000","last_clean":"2024-07-26T11:13:33.870763+0000","last_became_active":"2024-07-26T11:10:22.297134+0000","last_became_peered":"2024-07-26T11:10:22.297134+0000","last_unstale":"2024-07-26T11:13:33.870763+0000","last_undegraded":"2024-07-26T11:13:33.870763+0000","last_fullsized":"2024-07-26T11:13:33.870763+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:25:08.794363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875681+0000","last_change":"2024-07-26T11:10:22.304333+0000","last_active":"2024-07-26T11:13:33.875681+0000","last_peered":"2024-07-26T11:13:33.875681+0000","last_clean":"2024-07-26T11:13:33.875681+0000","last_became_active":"2024-07-26T11:10:22.303639+0000","last_became_peered":"2024-07-26T11:10:22.303639+0000","last_unstale":"2024-07-26T11:13:33.875681+0000","last_undegraded":"2024-07-26T11:13:33.875681+0000","last_fullsized":"2024-07-26T11:13:33.875681+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:47.053463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875130+0000","last_change":"2024-07-26T11:10:22.320998+0000","last_active":"2024-07-26T11:13:33.875130+0000","last_peered":"2024-07-26T11:13:33.875130+0000","last_clean":"2024-07-26T11:13:33.875130+0000","last_became_active":"2024-07-26T11:10:22.320610+0000","last_became_peered":"2024-07-26T11:10:22.320610+0000","last_unstale":"2024-07-26T11:13:33.875130+0000","last_undegraded":"2024-07-26T11:13:33.875130+0000","last_fullsized":"2024-07-26T11:13:33.875130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:52:28.802241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.873030+0000","last_change":"2024-07-26T11:10:34.970486+0000","last_active":"2024-07-26T11:13:33.873030+0000","last_peered":"2024-07-26T11:13:33.873030+0000","last_clean":"2024-07-26T11:13:33.873030+0000","last_became_active":"2024-07-26T11:10:34.969965+0000","last_became_peered":"2024-07-26T11:10:34.969965+0000","last_unstale":"2024-07-26T11:13:33.873030+0000","last_undegraded":"2024-07-26T11:13:33.873030+0000","last_fullsized":"2024-07-26T11:13:33.873030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:41.593251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.876555+0000","last_change":"2024-07-26T11:10:22.321261+0000","last_active":"2024-07-26T11:13:33.876555+0000","last_peered":"2024-07-26T11:13:33.876555+0000","last_clean":"2024-07-26T11:13:33.876555+0000","last_became_active":"2024-07-26T11:10:22.320819+0000","last_became_peered":"2024-07-26T11:10:22.320819+0000","last_unstale":"2024-07-26T11:13:33.876555+0000","last_undegraded":"2024-07-26T11:13:33.876555+0000","last_fullsized":"2024-07-26T11:13:33.876555+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:04:56.872600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.887727+0000","last_change":"2024-07-26T11:10:22.321206+0000","last_active":"2024-07-26T11:13:33.887727+0000","last_peered":"2024-07-26T11:13:33.887727+0000","last_clean":"2024-07-26T11:13:33.887727+0000","last_became_active":"2024-07-26T11:10:22.320700+0000","last_became_peered":"2024-07-26T11:10:22.320700+0000","last_unstale":"2024-07-26T11:13:33.887727+0000","last_undegraded":"2024-07-26T11:13:33.887727+0000","last_fullsized":"2024-07-26T11:13:33.887727+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:31:20.700808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.870903+0000","last_change":"2024-07-26T11:10:22.300061+0000","last_active":"2024-07-26T11:13:33.870903+0000","last_peered":"2024-07-26T11:13:33.870903+0000","last_clean":"2024-07-26T11:13:33.870903+0000","last_became_active":"2024-07-26T11:10:22.298737+0000","last_became_peered":"2024-07-26T11:10:22.298737+0000","last_unstale":"2024-07-26T11:13:33.870903+0000","last_undegraded":"2024-07-26T11:13:33.870903+0000","last_fullsized":"2024-07-26T11:13:33.870903+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:45:25.500816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.893213+0000","last_change":"2024-07-26T11:10:22.328534+0000","last_active":"2024-07-26T11:13:33.893213+0000","last_peered":"2024-07-26T11:13:33.893213+0000","last_clean":"2024-07-26T11:13:33.893213+0000","last_became_active":"2024-07-26T11:10:22.327929+0000","last_became_peered":"2024-07-26T11:10:22.327929+0000","last_unstale":"2024-07-26T11:13:33.893213+0000","last_undegraded":"2024-07-26T11:13:33.893213+0000","last_fullsized":"2024-07-26T11:13:33.893213+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:09:55.152881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.892299+0000","last_change":"2024-07-26T11:10:22.324753+0000","last_active":"2024-07-26T11:13:33.892299+0000","last_peered":"2024-07-26T11:13:33.892299+0000","last_clean":"2024-07-26T11:13:33.892299+0000","last_became_active":"2024-07-26T11:10:22.324307+0000","last_became_peered":"2024-07-26T11:10:22.324307+0000","last_unstale":"2024-07-26T11:13:33.892299+0000","last_undegraded":"2024-07-26T11:13:33.892299+0000","last_fullsized":"2024-07-26T11:13:33.892299+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:40.566465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.894283+0000","last_change":"2024-07-26T11:10:22.328183+0000","last_active":"2024-07-26T11:13:33.894283+0000","last_peered":"2024-07-26T11:13:33.894283+0000","last_clean":"2024-07-26T11:13:33.894283+0000","last_became_active":"2024-07-26T11:10:22.327805+0000","last_became_peered":"2024-07-26T11:10:22.327805+0000","last_unstale":"2024-07-26T11:13:33.894283+0000","last_undegraded":"2024-07-26T11:13:33.894283+0000","last_fullsized":"2024-07-26T11:13:33.894283+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:11:23.238390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":79710,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":2453,"num_read_kb":1316,"num_write":265,"num_write_kb":162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2194,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":598016,"data_stored":233757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":73,"seq":313532612691,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":527167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808919,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28712,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716440,"statfs":{"total":95995035648,"available":95965634560,"internally_reserved":0,"allocated":1736704,"data_stored":1186017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972445,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":529939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168676,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":529735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364904,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28620,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716532,"statfs":{"total":95995035648,"available":95965728768,"internally_reserved":0,"allocated":1642496,"data_stored":1122494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528430,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":521347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724662,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36952,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708200,"statfs":{"total":95995035648,"available":95957196800,"internally_reserved":0,"allocated":1654784,"data_stored":1113813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888186,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28108,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717044,"statfs":{"total":95995035648,"available":95966253056,"internally_reserved":0,"allocated":1118208,"data_stored":593336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084414,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":521055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378756,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":522597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542282,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":594826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705805,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":74201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":68615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":75024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T11:15:55.736 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-07-26T11:15:55.736 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-07-26T11:15:56.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.045+0000 7f9927200640 1 -- 172.21.15.134:0/1347601468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99281095a0 msgr2=0x7f992810fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:56.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.045+0000 7f9927200640 1 --2- 172.21.15.134:0/1347601468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99281095a0 0x7f992810fdd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9920009ca0 tx=0x7f992002a8c0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.045+0000 7f9927200640 1 -- 172.21.15.134:0/1347601468 shutdown_connections 2024-07-26T11:15:56.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.045+0000 7f9927200640 1 --2- 172.21.15.134:0/1347601468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9928108c50 0x7f9928109060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.045+0000 7f9927200640 1 --2- 172.21.15.134:0/1347601468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f99281095a0 0x7f992810fdd0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.045+0000 7f9927200640 1 --2- 172.21.15.134:0/1347601468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9928103320 0x7f9928103730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.045+0000 7f9927200640 1 -- 172.21.15.134:0/1347601468 >> 172.21.15.134:0/1347601468 conn(0x7f99280fed60 msgr2=0x7f99281011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:56.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.045+0000 7f9927200640 1 -- 172.21.15.134:0/1347601468 shutdown_connections 2024-07-26T11:15:56.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.045+0000 7f9927200640 1 -- 172.21.15.134:0/1347601468 wait complete. 2024-07-26T11:15:56.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.046+0000 7f9927200640 1 Processor -- start 2024-07-26T11:15:56.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.046+0000 7f9927200640 1 -- start start 2024-07-26T11:15:56.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f9927200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9928103320 0x7f9928199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f9927200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9928108c50 0x7f992819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f9927200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99281095a0 0x7f992819f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f9927200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9928113250 con 0x7f9928103320 2024-07-26T11:15:56.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f9927200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f99281130d0 con 0x7f9928108c50 2024-07-26T11:15:56.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f9927200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f99281133d0 con 0x7f99281095a0 2024-07-26T11:15:56.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f9925e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9928103320 0x7f9928199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f9925e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9928103320 0x7f9928199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:53290/0 (socket says 172.21.15.134:53290) 2024-07-26T11:15:56.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f9925e00640 1 -- 172.21.15.134:0/1956940410 learned_addr learned my addr 172.21.15.134:0/1956940410 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:56.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f991fe00640 1 --2- 172.21.15.134:0/1956940410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9928108c50 0x7f992819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f9926800640 1 --2- 172.21.15.134:0/1956940410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99281095a0 0x7f992819f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f9925e00640 1 -- 172.21.15.134:0/1956940410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99281095a0 msgr2=0x7f992819f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:56.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f9925e00640 1 --2- 172.21.15.134:0/1956940410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99281095a0 0x7f992819f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f9925e00640 1 -- 172.21.15.134:0/1956940410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9928108c50 msgr2=0x7f992819a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:56.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f9925e00640 1 --2- 172.21.15.134:0/1956940410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9928108c50 0x7f992819a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f9925e00640 1 -- 172.21.15.134:0/1956940410 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f992819fb80 con 0x7f9928103320 2024-07-26T11:15:56.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.047+0000 7f9926800640 1 --2- 172.21.15.134:0/1956940410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99281095a0 0x7f992819f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:15:56.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.048+0000 7f9925e00640 1 --2- 172.21.15.134:0/1956940410 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9928103320 0x7f9928199c50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f991800e9f0 tx=0x7f991800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:56.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.048+0000 7f991ea00640 1 -- 172.21.15.134:0/1956940410 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f991800cd80 con 0x7f9928103320 2024-07-26T11:15:56.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.048+0000 7f991ea00640 1 -- 172.21.15.134:0/1956940410 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99180108e0 con 0x7f9928103320 2024-07-26T11:15:56.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.048+0000 7f9927200640 1 -- 172.21.15.134:0/1956940410 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f992819fe70 con 0x7f9928103320 2024-07-26T11:15:56.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.048+0000 7f991ea00640 1 -- 172.21.15.134:0/1956940410 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9918018980 con 0x7f9928103320 2024-07-26T11:15:56.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.048+0000 7f9927200640 1 -- 172.21.15.134:0/1956940410 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99281a6600 con 0x7f9928103320 2024-07-26T11:15:56.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.049+0000 7f9927200640 1 -- 172.21.15.134:0/1956940410 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9928104590 con 0x7f9928103320 2024-07-26T11:15:56.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.051+0000 7f991ea00640 1 -- 172.21.15.134:0/1956940410 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f9918010430 con 0x7f9928103320 2024-07-26T11:15:56.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.053+0000 7f991ea00640 1 --2- 172.21.15.134:0/1956940410 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f990c0b3c20 0x7f990c0b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:56.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.053+0000 7f991ea00640 1 -- 172.21.15.134:0/1956940410 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f99180e4510 con 0x7f9928103320 2024-07-26T11:15:56.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.053+0000 7f991fe00640 1 --2- 172.21.15.134:0/1956940410 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f990c0b3c20 0x7f990c0b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:56.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.054+0000 7f991fe00640 1 --2- 172.21.15.134:0/1956940410 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f990c0b3c20 0x7f990c0b6130 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9928106620 tx=0x7f9910005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:56.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.057+0000 7f991ea00640 1 -- 172.21.15.134:0/1956940410 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f99180ab660 con 0x7f9928103320 2024-07-26T11:15:56.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.265+0000 7f9927200640 1 -- 172.21.15.134:0/1956940410 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9928063230 con 0x7f990c0b3c20 2024-07-26T11:15:56.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.281+0000 7f991ea00640 1 -- 172.21.15.134:0/1956940410 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340557 (secure 0 0 0) 0x7f9928063230 con 0x7f990c0b3c20 2024-07-26T11:15:56.289 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:15:56.291 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-07-26T11:15:56.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.290+0000 7f9927200640 1 -- 172.21.15.134:0/1956940410 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f990c0b3c20 msgr2=0x7f990c0b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:56.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.290+0000 7f9927200640 1 --2- 172.21.15.134:0/1956940410 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f990c0b3c20 0x7f990c0b6130 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9928106620 tx=0x7f9910005f90 comp rx=0 tx=0).stop 2024-07-26T11:15:56.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.290+0000 7f9927200640 1 -- 172.21.15.134:0/1956940410 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9928103320 msgr2=0x7f9928199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:56.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.290+0000 7f9927200640 1 --2- 172.21.15.134:0/1956940410 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9928103320 0x7f9928199c50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f991800e9f0 tx=0x7f991800eeb0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.290+0000 7f9927200640 1 -- 172.21.15.134:0/1956940410 shutdown_connections 2024-07-26T11:15:56.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.290+0000 7f9927200640 1 --2- 172.21.15.134:0/1956940410 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f990c0b3c20 0x7f990c0b6130 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.290+0000 7f9927200640 1 --2- 172.21.15.134:0/1956940410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f99281095a0 0x7f992819f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.290+0000 7f9927200640 1 --2- 172.21.15.134:0/1956940410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9928108c50 0x7f992819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.290+0000 7f9927200640 1 --2- 172.21.15.134:0/1956940410 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9928103320 0x7f9928199c50 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:56.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.290+0000 7f9927200640 1 -- 172.21.15.134:0/1956940410 >> 172.21.15.134:0/1956940410 conn(0x7f99280fed60 msgr2=0x7f9928101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:56.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.291+0000 7f9927200640 1 -- 172.21.15.134:0/1956940410 shutdown_connections 2024-07-26T11:15:56.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:56.292+0000 7f9927200640 1 -- 172.21.15.134:0/1956940410 wait complete. 2024-07-26T11:15:56.324 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":479,"stamp":"2024-07-26T11:15:56.050927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":670312,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":2544,"num_read_kb":1385,"num_write":390,"num_write_kb":2314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":2954034,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":822,"ondisk_log_size":822,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":375896,"kb_used_data":14264,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124565928,"statfs":{"total":1151940427776,"available":1151555510272,"internally_reserved":0,"allocated":14606336,"data_stored":8290502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":370193722},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5987,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":113,"num_read_kb":58,"num_write":7,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007110"},"pg_stats":[{"pgid":"2.1f","version":"90'13","reported_seq":310,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:49.585215+0000","last_change":"2024-07-26T11:10:34.985984+0000","last_active":"2024-07-26T11:15:49.585215+0000","last_peered":"2024-07-26T11:15:49.585215+0000","last_clean":"2024-07-26T11:15:49.585215+0000","last_became_active":"2024-07-26T11:10:34.980894+0000","last_became_peered":"2024-07-26T11:10:34.980894+0000","last_unstale":"2024-07-26T11:15:49.585215+0000","last_undegraded":"2024-07-26T11:15:49.585215+0000","last_fullsized":"2024-07-26T11:15:49.585215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:55.718875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":241,"num_read_kb":241,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874388+0000","last_change":"2024-07-26T11:10:22.320946+0000","last_active":"2024-07-26T11:13:33.874388+0000","last_peered":"2024-07-26T11:13:33.874388+0000","last_clean":"2024-07-26T11:13:33.874388+0000","last_became_active":"2024-07-26T11:10:22.316181+0000","last_became_peered":"2024-07-26T11:10:22.316181+0000","last_unstale":"2024-07-26T11:13:33.874388+0000","last_undegraded":"2024-07-26T11:13:33.874388+0000","last_fullsized":"2024-07-26T11:13:33.874388+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:50:13.641862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:04.915763+0000","last_change":"2024-07-26T11:10:35.984460+0000","last_active":"2024-07-26T11:13:04.915763+0000","last_peered":"2024-07-26T11:13:04.915763+0000","last_clean":"2024-07-26T11:13:04.915763+0000","last_became_active":"2024-07-26T11:10:35.983922+0000","last_became_peered":"2024-07-26T11:10:35.983922+0000","last_unstale":"2024-07-26T11:13:04.915763+0000","last_undegraded":"2024-07-26T11:13:04.915763+0000","last_fullsized":"2024-07-26T11:13:04.915763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:51:46.140910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:19.061450+0000","last_change":"2024-07-26T11:10:18.058553+0000","last_active":"2024-07-26T11:12:19.061450+0000","last_peered":"2024-07-26T11:12:19.061450+0000","last_clean":"2024-07-26T11:12:19.061450+0000","last_became_active":"2024-07-26T11:10:18.058034+0000","last_became_peered":"2024-07-26T11:10:18.058034+0000","last_unstale":"2024-07-26T11:12:19.061450+0000","last_undegraded":"2024-07-26T11:12:19.061450+0000","last_fullsized":"2024-07-26T11:12:19.061450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:48:58.868988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.893963+0000","last_change":"2024-07-26T11:10:22.326760+0000","last_active":"2024-07-26T11:13:33.893963+0000","last_peered":"2024-07-26T11:13:33.893963+0000","last_clean":"2024-07-26T11:13:33.893963+0000","last_became_active":"2024-07-26T11:10:22.326275+0000","last_became_peered":"2024-07-26T11:10:22.326275+0000","last_unstale":"2024-07-26T11:13:33.893963+0000","last_undegraded":"2024-07-26T11:13:33.893963+0000","last_fullsized":"2024-07-26T11:13:33.893963+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:51:57.461453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:04.941996+0000","last_change":"2024-07-26T11:10:35.997585+0000","last_active":"2024-07-26T11:13:04.941996+0000","last_peered":"2024-07-26T11:13:04.941996+0000","last_clean":"2024-07-26T11:13:04.941996+0000","last_became_active":"2024-07-26T11:10:35.997274+0000","last_became_peered":"2024-07-26T11:10:35.997274+0000","last_unstale":"2024-07-26T11:13:04.941996+0000","last_undegraded":"2024-07-26T11:13:04.941996+0000","last_fullsized":"2024-07-26T11:13:04.941996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:12:43.553396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:47.162843+0000","last_change":"2024-07-26T11:10:18.034964+0000","last_active":"2024-07-26T11:12:47.162843+0000","last_peered":"2024-07-26T11:12:47.162843+0000","last_clean":"2024-07-26T11:12:47.162843+0000","last_became_active":"2024-07-26T11:10:18.034569+0000","last_became_peered":"2024-07-26T11:10:18.034569+0000","last_unstale":"2024-07-26T11:12:47.162843+0000","last_undegraded":"2024-07-26T11:12:47.162843+0000","last_fullsized":"2024-07-26T11:12:47.162843+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:00:46.542458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.882555+0000","last_change":"2024-07-26T11:10:22.310695+0000","last_active":"2024-07-26T11:13:33.882555+0000","last_peered":"2024-07-26T11:13:33.882555+0000","last_clean":"2024-07-26T11:13:33.882555+0000","last_became_active":"2024-07-26T11:10:22.310142+0000","last_became_peered":"2024-07-26T11:10:22.310142+0000","last_unstale":"2024-07-26T11:13:33.882555+0000","last_undegraded":"2024-07-26T11:13:33.882555+0000","last_fullsized":"2024-07-26T11:13:33.882555+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:49:18.890802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:09.942670+0000","last_change":"2024-07-26T11:10:35.991418+0000","last_active":"2024-07-26T11:13:09.942670+0000","last_peered":"2024-07-26T11:13:09.942670+0000","last_clean":"2024-07-26T11:13:09.942670+0000","last_became_active":"2024-07-26T11:10:35.990934+0000","last_became_peered":"2024-07-26T11:10:35.990934+0000","last_unstale":"2024-07-26T11:13:09.942670+0000","last_undegraded":"2024-07-26T11:13:09.942670+0000","last_fullsized":"2024-07-26T11:13:09.942670+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:15:56.598738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:07.512173+0000","last_change":"2024-07-26T11:10:34.965637+0000","last_active":"2024-07-26T11:13:07.512173+0000","last_peered":"2024-07-26T11:13:07.512173+0000","last_clean":"2024-07-26T11:13:07.512173+0000","last_became_active":"2024-07-26T11:10:34.961157+0000","last_became_peered":"2024-07-26T11:10:34.961157+0000","last_unstale":"2024-07-26T11:13:07.512173+0000","last_undegraded":"2024-07-26T11:13:07.512173+0000","last_fullsized":"2024-07-26T11:13:07.512173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:26:17.007258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"90'15","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875528+0000","last_change":"2024-07-26T11:10:22.303606+0000","last_active":"2024-07-26T11:13:33.875528+0000","last_peered":"2024-07-26T11:13:33.875528+0000","last_clean":"2024-07-26T11:13:33.875528+0000","last_became_active":"2024-07-26T11:10:22.303183+0000","last_became_peered":"2024-07-26T11:10:22.303183+0000","last_unstale":"2024-07-26T11:13:33.875528+0000","last_undegraded":"2024-07-26T11:13:33.875528+0000","last_fullsized":"2024-07-26T11:13:33.875528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:31:01.153694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:09.942648+0000","last_change":"2024-07-26T11:10:35.981446+0000","last_active":"2024-07-26T11:13:09.942648+0000","last_peered":"2024-07-26T11:13:09.942648+0000","last_clean":"2024-07-26T11:13:09.942648+0000","last_became_active":"2024-07-26T11:10:35.980994+0000","last_became_peered":"2024-07-26T11:10:35.980994+0000","last_unstale":"2024-07-26T11:13:09.942648+0000","last_undegraded":"2024-07-26T11:13:09.942648+0000","last_fullsized":"2024-07-26T11:13:09.942648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:38:57.405325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.887229+0000","last_change":"2024-07-26T11:10:22.306758+0000","last_active":"2024-07-26T11:13:33.887229+0000","last_peered":"2024-07-26T11:13:33.887229+0000","last_clean":"2024-07-26T11:13:33.887229+0000","last_became_active":"2024-07-26T11:10:22.306370+0000","last_became_peered":"2024-07-26T11:10:22.306370+0000","last_unstale":"2024-07-26T11:13:33.887229+0000","last_undegraded":"2024-07-26T11:13:33.887229+0000","last_fullsized":"2024-07-26T11:13:33.887229+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:51:43.069470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:32.160732+0000","last_change":"2024-07-26T11:10:18.035664+0000","last_active":"2024-07-26T11:12:32.160732+0000","last_peered":"2024-07-26T11:12:32.160732+0000","last_clean":"2024-07-26T11:12:32.160732+0000","last_became_active":"2024-07-26T11:10:18.035113+0000","last_became_peered":"2024-07-26T11:10:18.035113+0000","last_unstale":"2024-07-26T11:12:32.160732+0000","last_undegraded":"2024-07-26T11:12:32.160732+0000","last_fullsized":"2024-07-26T11:12:32.160732+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:09:30.641034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:54.940745+0000","last_change":"2024-07-26T11:10:35.990907+0000","last_active":"2024-07-26T11:12:54.940745+0000","last_peered":"2024-07-26T11:12:54.940745+0000","last_clean":"2024-07-26T11:12:54.940745+0000","last_became_active":"2024-07-26T11:10:35.990417+0000","last_became_peered":"2024-07-26T11:10:35.990417+0000","last_unstale":"2024-07-26T11:12:54.940745+0000","last_undegraded":"2024-07-26T11:12:54.940745+0000","last_fullsized":"2024-07-26T11:12:54.940745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:24:39.319727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889158+0000","last_change":"2024-07-26T11:10:22.333575+0000","last_active":"2024-07-26T11:13:33.889158+0000","last_peered":"2024-07-26T11:13:33.889158+0000","last_clean":"2024-07-26T11:13:33.889158+0000","last_became_active":"2024-07-26T11:10:22.332802+0000","last_became_peered":"2024-07-26T11:10:22.332802+0000","last_unstale":"2024-07-26T11:13:33.889158+0000","last_undegraded":"2024-07-26T11:13:33.889158+0000","last_fullsized":"2024-07-26T11:13:33.889158+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:14:03.238840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:29.911881+0000","last_change":"2024-07-26T11:10:18.047942+0000","last_active":"2024-07-26T11:12:29.911881+0000","last_peered":"2024-07-26T11:12:29.911881+0000","last_clean":"2024-07-26T11:12:29.911881+0000","last_became_active":"2024-07-26T11:10:18.047500+0000","last_became_peered":"2024-07-26T11:10:18.047500+0000","last_unstale":"2024-07-26T11:12:29.911881+0000","last_undegraded":"2024-07-26T11:12:29.911881+0000","last_fullsized":"2024-07-26T11:12:29.911881+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:32:30.773648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:54.940721+0000","last_change":"2024-07-26T11:10:36.000946+0000","last_active":"2024-07-26T11:12:54.940721+0000","last_peered":"2024-07-26T11:12:54.940721+0000","last_clean":"2024-07-26T11:12:54.940721+0000","last_became_active":"2024-07-26T11:10:36.000251+0000","last_became_peered":"2024-07-26T11:10:36.000251+0000","last_unstale":"2024-07-26T11:12:54.940721+0000","last_undegraded":"2024-07-26T11:12:54.940721+0000","last_fullsized":"2024-07-26T11:12:54.940721+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:03:40.523747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.890526+0000","last_change":"2024-07-26T11:10:22.326370+0000","last_active":"2024-07-26T11:13:33.890526+0000","last_peered":"2024-07-26T11:13:33.890526+0000","last_clean":"2024-07-26T11:13:33.890526+0000","last_became_active":"2024-07-26T11:10:22.325961+0000","last_became_peered":"2024-07-26T11:10:22.325961+0000","last_unstale":"2024-07-26T11:13:33.890526+0000","last_undegraded":"2024-07-26T11:13:33.890526+0000","last_fullsized":"2024-07-26T11:13:33.890526+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:16:36.946750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:29.063642+0000","last_change":"2024-07-26T11:10:18.051940+0000","last_active":"2024-07-26T11:12:29.063642+0000","last_peered":"2024-07-26T11:12:29.063642+0000","last_clean":"2024-07-26T11:12:29.063642+0000","last_became_active":"2024-07-26T11:10:18.051565+0000","last_became_peered":"2024-07-26T11:10:18.051565+0000","last_unstale":"2024-07-26T11:12:29.063642+0000","last_undegraded":"2024-07-26T11:12:29.063642+0000","last_fullsized":"2024-07-26T11:12:29.063642+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:17:56.418638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:51.522449+0000","last_change":"2024-07-26T11:10:36.000156+0000","last_active":"2024-07-26T11:12:51.522449+0000","last_peered":"2024-07-26T11:12:51.522449+0000","last_clean":"2024-07-26T11:12:51.522449+0000","last_became_active":"2024-07-26T11:10:35.999600+0000","last_became_peered":"2024-07-26T11:10:35.999600+0000","last_unstale":"2024-07-26T11:12:51.522449+0000","last_undegraded":"2024-07-26T11:12:51.522449+0000","last_fullsized":"2024-07-26T11:12:51.522449+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:35:48.504288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"90'2","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.858816+0000","last_change":"2024-07-26T11:10:22.298152+0000","last_active":"2024-07-26T11:13:33.858816+0000","last_peered":"2024-07-26T11:13:33.858816+0000","last_clean":"2024-07-26T11:13:33.858816+0000","last_became_active":"2024-07-26T11:10:22.297739+0000","last_became_peered":"2024-07-26T11:10:22.297739+0000","last_unstale":"2024-07-26T11:13:33.858816+0000","last_undegraded":"2024-07-26T11:13:33.858816+0000","last_fullsized":"2024-07-26T11:13:33.858816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:04:48.160183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:28.950352+0000","last_change":"2024-07-26T11:10:18.057756+0000","last_active":"2024-07-26T11:12:28.950352+0000","last_peered":"2024-07-26T11:12:28.950352+0000","last_clean":"2024-07-26T11:12:28.950352+0000","last_became_active":"2024-07-26T11:10:18.057464+0000","last_became_peered":"2024-07-26T11:10:18.057464+0000","last_unstale":"2024-07-26T11:12:28.950352+0000","last_undegraded":"2024-07-26T11:12:28.950352+0000","last_fullsized":"2024-07-26T11:12:28.950352+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:39:11.523643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:54.940757+0000","last_change":"2024-07-26T11:10:36.000617+0000","last_active":"2024-07-26T11:12:54.940757+0000","last_peered":"2024-07-26T11:12:54.940757+0000","last_clean":"2024-07-26T11:12:54.940757+0000","last_became_active":"2024-07-26T11:10:36.000151+0000","last_became_peered":"2024-07-26T11:10:36.000151+0000","last_unstale":"2024-07-26T11:12:54.940757+0000","last_undegraded":"2024-07-26T11:12:54.940757+0000","last_fullsized":"2024-07-26T11:12:54.940757+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:10:36.942215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.893390+0000","last_change":"2024-07-26T11:10:22.336452+0000","last_active":"2024-07-26T11:13:33.893390+0000","last_peered":"2024-07-26T11:13:33.893390+0000","last_clean":"2024-07-26T11:13:33.893390+0000","last_became_active":"2024-07-26T11:10:22.336006+0000","last_became_peered":"2024-07-26T11:10:22.336006+0000","last_unstale":"2024-07-26T11:13:33.893390+0000","last_undegraded":"2024-07-26T11:13:33.893390+0000","last_fullsized":"2024-07-26T11:13:33.893390+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:08:27.375730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:29.937948+0000","last_change":"2024-07-26T11:10:18.823017+0000","last_active":"2024-07-26T11:12:29.937948+0000","last_peered":"2024-07-26T11:12:29.937948+0000","last_clean":"2024-07-26T11:12:29.937948+0000","last_became_active":"2024-07-26T11:10:18.822559+0000","last_became_peered":"2024-07-26T11:10:18.822559+0000","last_unstale":"2024-07-26T11:12:29.937948+0000","last_undegraded":"2024-07-26T11:12:29.937948+0000","last_fullsized":"2024-07-26T11:12:29.937948+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:29:01.486263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:44.913501+0000","last_change":"2024-07-26T11:10:35.969698+0000","last_active":"2024-07-26T11:12:44.913501+0000","last_peered":"2024-07-26T11:12:44.913501+0000","last_clean":"2024-07-26T11:12:44.913501+0000","last_became_active":"2024-07-26T11:10:35.969240+0000","last_became_peered":"2024-07-26T11:10:35.969240+0000","last_unstale":"2024-07-26T11:12:44.913501+0000","last_undegraded":"2024-07-26T11:12:44.913501+0000","last_fullsized":"2024-07-26T11:12:44.913501+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:44:25.210862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875375+0000","last_change":"2024-07-26T11:10:22.314772+0000","last_active":"2024-07-26T11:13:33.875375+0000","last_peered":"2024-07-26T11:13:33.875375+0000","last_clean":"2024-07-26T11:13:33.875375+0000","last_became_active":"2024-07-26T11:10:22.313786+0000","last_became_peered":"2024-07-26T11:10:22.313786+0000","last_unstale":"2024-07-26T11:13:33.875375+0000","last_undegraded":"2024-07-26T11:13:33.875375+0000","last_fullsized":"2024-07-26T11:13:33.875375+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:40:19.610695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'18","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:14:41.543437+0000","last_change":"2024-07-26T11:10:18.049408+0000","last_active":"2024-07-26T11:14:41.543437+0000","last_peered":"2024-07-26T11:14:41.543437+0000","last_clean":"2024-07-26T11:14:41.543437+0000","last_became_active":"2024-07-26T11:10:18.048573+0000","last_became_peered":"2024-07-26T11:10:18.048573+0000","last_unstale":"2024-07-26T11:14:41.543437+0000","last_undegraded":"2024-07-26T11:14:41.543437+0000","last_fullsized":"2024-07-26T11:14:41.543437+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:32:26.408260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:39.065841+0000","last_change":"2024-07-26T11:10:35.992067+0000","last_active":"2024-07-26T11:12:39.065841+0000","last_peered":"2024-07-26T11:12:39.065841+0000","last_clean":"2024-07-26T11:12:39.065841+0000","last_became_active":"2024-07-26T11:10:35.989789+0000","last_became_peered":"2024-07-26T11:10:35.989789+0000","last_unstale":"2024-07-26T11:12:39.065841+0000","last_undegraded":"2024-07-26T11:12:39.065841+0000","last_fullsized":"2024-07-26T11:12:39.065841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:08:08.283592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.888864+0000","last_change":"2024-07-26T11:10:22.323512+0000","last_active":"2024-07-26T11:13:33.888864+0000","last_peered":"2024-07-26T11:13:33.888864+0000","last_clean":"2024-07-26T11:13:33.888864+0000","last_became_active":"2024-07-26T11:10:22.323126+0000","last_became_peered":"2024-07-26T11:10:22.323126+0000","last_unstale":"2024-07-26T11:13:33.888864+0000","last_undegraded":"2024-07-26T11:13:33.888864+0000","last_fullsized":"2024-07-26T11:13:33.888864+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:27:15.379345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:22.159418+0000","last_change":"2024-07-26T11:10:18.052573+0000","last_active":"2024-07-26T11:12:22.159418+0000","last_peered":"2024-07-26T11:12:22.159418+0000","last_clean":"2024-07-26T11:12:22.159418+0000","last_became_active":"2024-07-26T11:10:18.052302+0000","last_became_peered":"2024-07-26T11:10:18.052302+0000","last_unstale":"2024-07-26T11:12:22.159418+0000","last_undegraded":"2024-07-26T11:12:22.159418+0000","last_fullsized":"2024-07-26T11:12:22.159418+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:15:32.260966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:42.508751+0000","last_change":"2024-07-26T11:10:35.994347+0000","last_active":"2024-07-26T11:12:42.508751+0000","last_peered":"2024-07-26T11:12:42.508751+0000","last_clean":"2024-07-26T11:12:42.508751+0000","last_became_active":"2024-07-26T11:10:35.993795+0000","last_became_peered":"2024-07-26T11:10:35.993795+0000","last_unstale":"2024-07-26T11:12:42.508751+0000","last_undegraded":"2024-07-26T11:12:42.508751+0000","last_fullsized":"2024-07-26T11:12:42.508751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:27:35.399601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.5","version":"90'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.881857+0000","last_change":"2024-07-26T11:10:22.318109+0000","last_active":"2024-07-26T11:13:33.881857+0000","last_peered":"2024-07-26T11:13:33.881857+0000","last_clean":"2024-07-26T11:13:33.881857+0000","last_became_active":"2024-07-26T11:10:22.317674+0000","last_became_peered":"2024-07-26T11:10:22.317674+0000","last_unstale":"2024-07-26T11:13:33.881857+0000","last_undegraded":"2024-07-26T11:13:33.881857+0000","last_fullsized":"2024-07-26T11:13:33.881857+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:29.464772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:26.517734+0000","last_change":"2024-07-26T11:10:18.060073+0000","last_active":"2024-07-26T11:12:26.517734+0000","last_peered":"2024-07-26T11:12:26.517734+0000","last_clean":"2024-07-26T11:12:26.517734+0000","last_became_active":"2024-07-26T11:10:18.059381+0000","last_became_peered":"2024-07-26T11:10:18.059381+0000","last_unstale":"2024-07-26T11:12:26.517734+0000","last_undegraded":"2024-07-26T11:12:26.517734+0000","last_fullsized":"2024-07-26T11:12:26.517734+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:04:21.184759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:42.162020+0000","last_change":"2024-07-26T11:11:35.951234+0000","last_active":"2024-07-26T11:12:42.162020+0000","last_peered":"2024-07-26T11:12:42.162020+0000","last_clean":"2024-07-26T11:12:42.162020+0000","last_became_active":"2024-07-26T11:11:35.950821+0000","last_became_peered":"2024-07-26T11:11:35.950821+0000","last_unstale":"2024-07-26T11:12:42.162020+0000","last_undegraded":"2024-07-26T11:12:42.162020+0000","last_fullsized":"2024-07-26T11:12:42.162020+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:44:22.569238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,1],"acting":[8,10,11,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.873134+0000","last_change":"2024-07-26T11:10:22.326866+0000","last_active":"2024-07-26T11:13:33.873134+0000","last_peered":"2024-07-26T11:13:33.873134+0000","last_clean":"2024-07-26T11:13:33.873134+0000","last_became_active":"2024-07-26T11:10:22.326417+0000","last_became_peered":"2024-07-26T11:10:22.326417+0000","last_unstale":"2024-07-26T11:13:33.873134+0000","last_undegraded":"2024-07-26T11:13:33.873134+0000","last_fullsized":"2024-07-26T11:13:33.873134+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:49:02.871062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":290,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:50.588141+0000","last_change":"2024-07-26T11:10:18.044077+0000","last_active":"2024-07-26T11:15:50.588141+0000","last_peered":"2024-07-26T11:15:50.588141+0000","last_clean":"2024-07-26T11:15:50.588141+0000","last_became_active":"2024-07-26T11:10:18.043660+0000","last_became_peered":"2024-07-26T11:10:18.043660+0000","last_unstale":"2024-07-26T11:15:50.588141+0000","last_undegraded":"2024-07-26T11:15:50.588141+0000","last_fullsized":"2024-07-26T11:15:50.588141+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:46:54.901082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":242,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:41.781727+0000","last_change":"2024-07-26T11:10:35.984778+0000","last_active":"2024-07-26T11:12:41.781727+0000","last_peered":"2024-07-26T11:12:41.781727+0000","last_clean":"2024-07-26T11:12:41.781727+0000","last_became_active":"2024-07-26T11:10:35.984365+0000","last_became_peered":"2024-07-26T11:10:35.984365+0000","last_unstale":"2024-07-26T11:12:41.781727+0000","last_undegraded":"2024-07-26T11:12:41.781727+0000","last_fullsized":"2024-07-26T11:12:41.781727+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:26:33.174121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"90'8","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.867814+0000","last_change":"2024-07-26T11:10:34.976075+0000","last_active":"2024-07-26T11:13:33.867814+0000","last_peered":"2024-07-26T11:13:33.867814+0000","last_clean":"2024-07-26T11:13:33.867814+0000","last_became_active":"2024-07-26T11:10:34.973799+0000","last_became_peered":"2024-07-26T11:10:34.973799+0000","last_unstale":"2024-07-26T11:13:33.867814+0000","last_undegraded":"2024-07-26T11:13:33.867814+0000","last_fullsized":"2024-07-26T11:13:33.867814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:03:01.830655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"90'7","reported_seq":292,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:50.735982+0000","last_change":"2024-07-26T11:10:18.056473+0000","last_active":"2024-07-26T11:15:50.735982+0000","last_peered":"2024-07-26T11:15:50.735982+0000","last_clean":"2024-07-26T11:15:50.735982+0000","last_became_active":"2024-07-26T11:10:18.056038+0000","last_became_peered":"2024-07-26T11:10:18.056038+0000","last_unstale":"2024-07-26T11:15:50.735982+0000","last_undegraded":"2024-07-26T11:15:50.735982+0000","last_fullsized":"2024-07-26T11:15:50.735982+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:32:04.501018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":242,"num_read_kb":242,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:46.782896+0000","last_change":"2024-07-26T11:10:35.991741+0000","last_active":"2024-07-26T11:12:46.782896+0000","last_peered":"2024-07-26T11:12:46.782896+0000","last_clean":"2024-07-26T11:12:46.782896+0000","last_became_active":"2024-07-26T11:10:35.990995+0000","last_became_peered":"2024-07-26T11:10:35.990995+0000","last_unstale":"2024-07-26T11:12:46.782896+0000","last_undegraded":"2024-07-26T11:12:46.782896+0000","last_fullsized":"2024-07-26T11:12:46.782896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:59:34.056646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.884331+0000","last_change":"2024-07-26T11:10:22.305053+0000","last_active":"2024-07-26T11:13:33.884331+0000","last_peered":"2024-07-26T11:13:33.884331+0000","last_clean":"2024-07-26T11:13:33.884331+0000","last_became_active":"2024-07-26T11:10:22.304035+0000","last_became_peered":"2024-07-26T11:10:22.304035+0000","last_unstale":"2024-07-26T11:13:33.884331+0000","last_undegraded":"2024-07-26T11:13:33.884331+0000","last_fullsized":"2024-07-26T11:13:33.884331+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:48.590799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:17.349794+0000","last_change":"2024-07-26T11:10:18.033581+0000","last_active":"2024-07-26T11:12:17.349794+0000","last_peered":"2024-07-26T11:12:17.349794+0000","last_clean":"2024-07-26T11:12:17.349794+0000","last_became_active":"2024-07-26T11:10:18.033038+0000","last_became_peered":"2024-07-26T11:10:18.033038+0000","last_unstale":"2024-07-26T11:12:17.349794+0000","last_undegraded":"2024-07-26T11:12:17.349794+0000","last_fullsized":"2024-07-26T11:12:17.349794+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:19:52.523006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:48.387395+0000","last_change":"2024-07-26T11:11:35.643683+0000","last_active":"2024-07-26T11:12:48.387395+0000","last_peered":"2024-07-26T11:12:48.387395+0000","last_clean":"2024-07-26T11:12:48.387395+0000","last_became_active":"2024-07-26T11:11:35.643109+0000","last_became_peered":"2024-07-26T11:11:35.643109+0000","last_unstale":"2024-07-26T11:12:48.387395+0000","last_undegraded":"2024-07-26T11:12:48.387395+0000","last_fullsized":"2024-07-26T11:12:48.387395+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:39:28.190726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"90'8","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.878540+0000","last_change":"2024-07-26T11:10:34.977690+0000","last_active":"2024-07-26T11:13:33.878540+0000","last_peered":"2024-07-26T11:13:33.878540+0000","last_clean":"2024-07-26T11:13:33.878540+0000","last_became_active":"2024-07-26T11:10:34.976938+0000","last_became_peered":"2024-07-26T11:10:34.976938+0000","last_unstale":"2024-07-26T11:13:33.878540+0000","last_undegraded":"2024-07-26T11:13:33.878540+0000","last_fullsized":"2024-07-26T11:13:33.878540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:26:21.510469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:24.937213+0000","last_change":"2024-07-26T11:10:18.045363+0000","last_active":"2024-07-26T11:12:24.937213+0000","last_peered":"2024-07-26T11:12:24.937213+0000","last_clean":"2024-07-26T11:12:24.937213+0000","last_became_active":"2024-07-26T11:10:18.044810+0000","last_became_peered":"2024-07-26T11:10:18.044810+0000","last_unstale":"2024-07-26T11:12:24.937213+0000","last_undegraded":"2024-07-26T11:12:24.937213+0000","last_fullsized":"2024-07-26T11:12:24.937213+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:05:34.507172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":120,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:27.358372+0000","last_change":"2024-07-26T11:08:48.936784+0000","last_active":"2024-07-26T11:13:27.358372+0000","last_peered":"2024-07-26T11:13:27.358372+0000","last_clean":"2024-07-26T11:13:27.358372+0000","last_became_active":"2024-07-26T11:08:48.931118+0000","last_became_peered":"2024-07-26T11:08:48.931118+0000","last_unstale":"2024-07-26T11:13:27.358372+0000","last_undegraded":"2024-07-26T11:13:27.358372+0000","last_fullsized":"2024-07-26T11:13:27.358372+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:05:01.918569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:05:01.918569+0000","last_clean_scrub_stamp":"2024-07-26T11:05:01.918569+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:57.128424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:42.352841+0000","last_change":"2024-07-26T11:11:35.385528+0000","last_active":"2024-07-26T11:12:42.352841+0000","last_peered":"2024-07-26T11:12:42.352841+0000","last_clean":"2024-07-26T11:12:42.352841+0000","last_became_active":"2024-07-26T11:11:35.385042+0000","last_became_peered":"2024-07-26T11:11:35.385042+0000","last_unstale":"2024-07-26T11:12:42.352841+0000","last_undegraded":"2024-07-26T11:12:42.352841+0000","last_fullsized":"2024-07-26T11:12:42.352841+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:52:30.343736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,11,9],"acting":[7,8,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875558+0000","last_change":"2024-07-26T11:10:22.302715+0000","last_active":"2024-07-26T11:13:33.875558+0000","last_peered":"2024-07-26T11:13:33.875558+0000","last_clean":"2024-07-26T11:13:33.875558+0000","last_became_active":"2024-07-26T11:10:22.302284+0000","last_became_peered":"2024-07-26T11:10:22.302284+0000","last_unstale":"2024-07-26T11:13:33.875558+0000","last_undegraded":"2024-07-26T11:13:33.875558+0000","last_fullsized":"2024-07-26T11:13:33.875558+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:13:05.555439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":294,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:53.485224+0000","last_change":"2024-07-26T11:10:18.823160+0000","last_active":"2024-07-26T11:15:53.485224+0000","last_peered":"2024-07-26T11:15:53.485224+0000","last_clean":"2024-07-26T11:15:53.485224+0000","last_became_active":"2024-07-26T11:10:18.822652+0000","last_became_peered":"2024-07-26T11:10:18.822652+0000","last_unstale":"2024-07-26T11:15:53.485224+0000","last_undegraded":"2024-07-26T11:15:53.485224+0000","last_fullsized":"2024-07-26T11:15:53.485224+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:21:06.077784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":246,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:48.952701+0000","last_change":"2024-07-26T11:10:35.984007+0000","last_active":"2024-07-26T11:12:48.952701+0000","last_peered":"2024-07-26T11:12:48.952701+0000","last_clean":"2024-07-26T11:12:48.952701+0000","last_became_active":"2024-07-26T11:10:35.983352+0000","last_became_peered":"2024-07-26T11:10:35.983352+0000","last_unstale":"2024-07-26T11:12:48.952701+0000","last_undegraded":"2024-07-26T11:12:48.952701+0000","last_fullsized":"2024-07-26T11:12:48.952701+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:05:21.199041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874961+0000","last_change":"2024-07-26T11:10:22.320559+0000","last_active":"2024-07-26T11:13:33.874961+0000","last_peered":"2024-07-26T11:13:33.874961+0000","last_clean":"2024-07-26T11:13:33.874961+0000","last_became_active":"2024-07-26T11:10:22.320192+0000","last_became_peered":"2024-07-26T11:10:22.320192+0000","last_unstale":"2024-07-26T11:13:33.874961+0000","last_undegraded":"2024-07-26T11:13:33.874961+0000","last_fullsized":"2024-07-26T11:13:33.874961+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:48:51.774418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:33.385383+0000","last_change":"2024-07-26T11:10:18.044108+0000","last_active":"2024-07-26T11:12:33.385383+0000","last_peered":"2024-07-26T11:12:33.385383+0000","last_clean":"2024-07-26T11:12:33.385383+0000","last_became_active":"2024-07-26T11:10:18.043712+0000","last_became_peered":"2024-07-26T11:10:18.043712+0000","last_unstale":"2024-07-26T11:12:33.385383+0000","last_undegraded":"2024-07-26T11:12:33.385383+0000","last_fullsized":"2024-07-26T11:12:33.385383+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:05:34.382520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:54.249813+0000","last_change":"2024-07-26T11:10:35.999071+0000","last_active":"2024-07-26T11:12:54.249813+0000","last_peered":"2024-07-26T11:12:54.249813+0000","last_clean":"2024-07-26T11:12:54.249813+0000","last_became_active":"2024-07-26T11:10:35.997179+0000","last_became_peered":"2024-07-26T11:10:35.997179+0000","last_unstale":"2024-07-26T11:12:54.249813+0000","last_undegraded":"2024-07-26T11:12:54.249813+0000","last_fullsized":"2024-07-26T11:12:54.249813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:36:45.632086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.872320+0000","last_change":"2024-07-26T11:10:22.300356+0000","last_active":"2024-07-26T11:13:33.872320+0000","last_peered":"2024-07-26T11:13:33.872320+0000","last_clean":"2024-07-26T11:13:33.872320+0000","last_became_active":"2024-07-26T11:10:22.299949+0000","last_became_peered":"2024-07-26T11:10:22.299949+0000","last_unstale":"2024-07-26T11:13:33.872320+0000","last_undegraded":"2024-07-26T11:13:33.872320+0000","last_fullsized":"2024-07-26T11:13:33.872320+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:38:34.383929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:36.780361+0000","last_change":"2024-07-26T11:10:18.049965+0000","last_active":"2024-07-26T11:12:36.780361+0000","last_peered":"2024-07-26T11:12:36.780361+0000","last_clean":"2024-07-26T11:12:36.780361+0000","last_became_active":"2024-07-26T11:10:18.049251+0000","last_became_peered":"2024-07-26T11:10:18.049251+0000","last_unstale":"2024-07-26T11:12:36.780361+0000","last_undegraded":"2024-07-26T11:12:36.780361+0000","last_fullsized":"2024-07-26T11:12:36.780361+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:10:20.996609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:48.387315+0000","last_change":"2024-07-26T11:10:35.967661+0000","last_active":"2024-07-26T11:12:48.387315+0000","last_peered":"2024-07-26T11:12:48.387315+0000","last_clean":"2024-07-26T11:12:48.387315+0000","last_became_active":"2024-07-26T11:10:35.966867+0000","last_became_peered":"2024-07-26T11:10:35.966867+0000","last_unstale":"2024-07-26T11:12:48.387315+0000","last_undegraded":"2024-07-26T11:12:48.387315+0000","last_fullsized":"2024-07-26T11:12:48.387315+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:33:43.951319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"90'5","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.867777+0000","last_change":"2024-07-26T11:10:22.303733+0000","last_active":"2024-07-26T11:13:33.867777+0000","last_peered":"2024-07-26T11:13:33.867777+0000","last_clean":"2024-07-26T11:13:33.867777+0000","last_became_active":"2024-07-26T11:10:22.303322+0000","last_became_peered":"2024-07-26T11:10:22.303322+0000","last_unstale":"2024-07-26T11:13:33.867777+0000","last_undegraded":"2024-07-26T11:13:33.867777+0000","last_fullsized":"2024-07-26T11:13:33.867777+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:34:34.962492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:32.160782+0000","last_change":"2024-07-26T11:10:18.032539+0000","last_active":"2024-07-26T11:12:32.160782+0000","last_peered":"2024-07-26T11:12:32.160782+0000","last_clean":"2024-07-26T11:12:32.160782+0000","last_became_active":"2024-07-26T11:10:18.032059+0000","last_became_peered":"2024-07-26T11:10:18.032059+0000","last_unstale":"2024-07-26T11:12:32.160782+0000","last_undegraded":"2024-07-26T11:12:32.160782+0000","last_fullsized":"2024-07-26T11:12:32.160782+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T15:02:54.502339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:48.387339+0000","last_change":"2024-07-26T11:10:35.977473+0000","last_active":"2024-07-26T11:12:48.387339+0000","last_peered":"2024-07-26T11:12:48.387339+0000","last_clean":"2024-07-26T11:12:48.387339+0000","last_became_active":"2024-07-26T11:10:35.976830+0000","last_became_peered":"2024-07-26T11:10:35.976830+0000","last_unstale":"2024-07-26T11:12:48.387339+0000","last_undegraded":"2024-07-26T11:12:48.387339+0000","last_fullsized":"2024-07-26T11:12:48.387339+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:46:13.009246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875320+0000","last_change":"2024-07-26T11:10:22.304358+0000","last_active":"2024-07-26T11:13:33.875320+0000","last_peered":"2024-07-26T11:13:33.875320+0000","last_clean":"2024-07-26T11:13:33.875320+0000","last_became_active":"2024-07-26T11:10:22.303846+0000","last_became_peered":"2024-07-26T11:10:22.303846+0000","last_unstale":"2024-07-26T11:13:33.875320+0000","last_undegraded":"2024-07-26T11:13:33.875320+0000","last_fullsized":"2024-07-26T11:13:33.875320+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:27:28.556481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:37.161406+0000","last_change":"2024-07-26T11:10:18.057678+0000","last_active":"2024-07-26T11:12:37.161406+0000","last_peered":"2024-07-26T11:12:37.161406+0000","last_clean":"2024-07-26T11:12:37.161406+0000","last_became_active":"2024-07-26T11:10:18.057282+0000","last_became_peered":"2024-07-26T11:10:18.057282+0000","last_unstale":"2024-07-26T11:12:37.161406+0000","last_undegraded":"2024-07-26T11:12:37.161406+0000","last_fullsized":"2024-07-26T11:12:37.161406+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:08:35.731665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:01.785766+0000","last_change":"2024-07-26T11:10:36.000329+0000","last_active":"2024-07-26T11:13:01.785766+0000","last_peered":"2024-07-26T11:13:01.785766+0000","last_clean":"2024-07-26T11:13:01.785766+0000","last_became_active":"2024-07-26T11:10:35.999713+0000","last_became_peered":"2024-07-26T11:10:35.999713+0000","last_unstale":"2024-07-26T11:13:01.785766+0000","last_undegraded":"2024-07-26T11:13:01.785766+0000","last_fullsized":"2024-07-26T11:13:01.785766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:45:48.137237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.866953+0000","last_change":"2024-07-26T11:10:22.301526+0000","last_active":"2024-07-26T11:13:33.866953+0000","last_peered":"2024-07-26T11:13:33.866953+0000","last_clean":"2024-07-26T11:13:33.866953+0000","last_became_active":"2024-07-26T11:10:22.301098+0000","last_became_peered":"2024-07-26T11:10:22.301098+0000","last_unstale":"2024-07-26T11:13:33.866953+0000","last_undegraded":"2024-07-26T11:13:33.866953+0000","last_fullsized":"2024-07-26T11:13:33.866953+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:01:48.118315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"90'8","reported_seq":300,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:53.745396+0000","last_change":"2024-07-26T11:10:18.035922+0000","last_active":"2024-07-26T11:15:53.745396+0000","last_peered":"2024-07-26T11:15:53.745396+0000","last_clean":"2024-07-26T11:15:53.745396+0000","last_became_active":"2024-07-26T11:10:18.035184+0000","last_became_peered":"2024-07-26T11:10:18.035184+0000","last_unstale":"2024-07-26T11:15:53.745396+0000","last_undegraded":"2024-07-26T11:15:53.745396+0000","last_fullsized":"2024-07-26T11:15:53.745396+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:13:10.371105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1427,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":250,"num_read_kb":6,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:01.524560+0000","last_change":"2024-07-26T11:10:35.998791+0000","last_active":"2024-07-26T11:13:01.524560+0000","last_peered":"2024-07-26T11:13:01.524560+0000","last_clean":"2024-07-26T11:13:01.524560+0000","last_became_active":"2024-07-26T11:10:35.997163+0000","last_became_peered":"2024-07-26T11:10:35.997163+0000","last_unstale":"2024-07-26T11:13:01.524560+0000","last_undegraded":"2024-07-26T11:13:01.524560+0000","last_fullsized":"2024-07-26T11:13:01.524560+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:28:50.167294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.13","version":"90'15","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889494+0000","last_change":"2024-07-26T11:10:22.314219+0000","last_active":"2024-07-26T11:13:33.889494+0000","last_peered":"2024-07-26T11:13:33.889494+0000","last_clean":"2024-07-26T11:13:33.889494+0000","last_became_active":"2024-07-26T11:10:22.313811+0000","last_became_peered":"2024-07-26T11:10:22.313811+0000","last_unstale":"2024-07-26T11:13:33.889494+0000","last_undegraded":"2024-07-26T11:13:33.889494+0000","last_fullsized":"2024-07-26T11:13:33.889494+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:55:47.452798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:38.385955+0000","last_change":"2024-07-26T11:10:18.047818+0000","last_active":"2024-07-26T11:12:38.385955+0000","last_peered":"2024-07-26T11:12:38.385955+0000","last_clean":"2024-07-26T11:12:38.385955+0000","last_became_active":"2024-07-26T11:10:18.047159+0000","last_became_peered":"2024-07-26T11:10:18.047159+0000","last_unstale":"2024-07-26T11:12:38.385955+0000","last_undegraded":"2024-07-26T11:12:38.385955+0000","last_fullsized":"2024-07-26T11:12:38.385955+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:08:37.528512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:57.510826+0000","last_change":"2024-07-26T11:10:35.996468+0000","last_active":"2024-07-26T11:12:57.510826+0000","last_peered":"2024-07-26T11:12:57.510826+0000","last_clean":"2024-07-26T11:12:57.510826+0000","last_became_active":"2024-07-26T11:10:35.996030+0000","last_became_peered":"2024-07-26T11:10:35.996030+0000","last_unstale":"2024-07-26T11:12:57.510826+0000","last_undegraded":"2024-07-26T11:12:57.510826+0000","last_fullsized":"2024-07-26T11:12:57.510826+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:20:14.743690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.881988+0000","last_change":"2024-07-26T11:10:22.318375+0000","last_active":"2024-07-26T11:13:33.881988+0000","last_peered":"2024-07-26T11:13:33.881988+0000","last_clean":"2024-07-26T11:13:33.881988+0000","last_became_active":"2024-07-26T11:10:22.317965+0000","last_became_peered":"2024-07-26T11:10:22.317965+0000","last_unstale":"2024-07-26T11:13:33.881988+0000","last_undegraded":"2024-07-26T11:13:33.881988+0000","last_fullsized":"2024-07-26T11:13:33.881988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:36:43.504045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:39.912995+0000","last_change":"2024-07-26T11:10:18.047957+0000","last_active":"2024-07-26T11:12:39.912995+0000","last_peered":"2024-07-26T11:12:39.912995+0000","last_clean":"2024-07-26T11:12:39.912995+0000","last_became_active":"2024-07-26T11:10:18.047495+0000","last_became_peered":"2024-07-26T11:10:18.047495+0000","last_unstale":"2024-07-26T11:12:39.912995+0000","last_undegraded":"2024-07-26T11:12:39.912995+0000","last_fullsized":"2024-07-26T11:12:39.912995+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:48:11.238295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:03.389374+0000","last_change":"2024-07-26T11:10:35.976946+0000","last_active":"2024-07-26T11:13:03.389374+0000","last_peered":"2024-07-26T11:13:03.389374+0000","last_clean":"2024-07-26T11:13:03.389374+0000","last_became_active":"2024-07-26T11:10:35.976474+0000","last_became_peered":"2024-07-26T11:10:35.976474+0000","last_unstale":"2024-07-26T11:13:03.389374+0000","last_undegraded":"2024-07-26T11:13:03.389374+0000","last_fullsized":"2024-07-26T11:13:03.389374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:50:51.240903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.859135+0000","last_change":"2024-07-26T11:10:22.302473+0000","last_active":"2024-07-26T11:13:33.859135+0000","last_peered":"2024-07-26T11:13:33.859135+0000","last_clean":"2024-07-26T11:13:33.859135+0000","last_became_active":"2024-07-26T11:10:22.302096+0000","last_became_peered":"2024-07-26T11:10:22.302096+0000","last_unstale":"2024-07-26T11:13:33.859135+0000","last_undegraded":"2024-07-26T11:13:33.859135+0000","last_fullsized":"2024-07-26T11:13:33.859135+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:21:19.215301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.14","version":"90'43","reported_seq":599,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:51.482116+0000","last_change":"2024-07-26T11:10:34.973502+0000","last_active":"2024-07-26T11:15:51.482116+0000","last_peered":"2024-07-26T11:15:51.482116+0000","last_clean":"2024-07-26T11:15:51.482116+0000","last_became_active":"2024-07-26T11:10:34.964155+0000","last_became_peered":"2024-07-26T11:10:34.964155+0000","last_unstale":"2024-07-26T11:15:51.482116+0000","last_undegraded":"2024-07-26T11:15:51.482116+0000","last_fullsized":"2024-07-26T11:15:51.482116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:23:06.652746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72449,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":496,"num_read_kb":320,"num_write":45,"num_write_kb":92,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1612,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:56.523464+0000","last_change":"2024-07-26T11:10:35.981439+0000","last_active":"2024-07-26T11:12:56.523464+0000","last_peered":"2024-07-26T11:12:56.523464+0000","last_clean":"2024-07-26T11:12:56.523464+0000","last_became_active":"2024-07-26T11:10:35.980878+0000","last_became_peered":"2024-07-26T11:10:35.980878+0000","last_unstale":"2024-07-26T11:12:56.523464+0000","last_undegraded":"2024-07-26T11:12:56.523464+0000","last_fullsized":"2024-07-26T11:12:56.523464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:31.719845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.14","version":"90'11","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889855+0000","last_change":"2024-07-26T11:10:34.974292+0000","last_active":"2024-07-26T11:13:33.889855+0000","last_peered":"2024-07-26T11:13:33.889855+0000","last_clean":"2024-07-26T11:13:33.889855+0000","last_became_active":"2024-07-26T11:10:34.973820+0000","last_became_peered":"2024-07-26T11:10:34.973820+0000","last_unstale":"2024-07-26T11:13:33.889855+0000","last_undegraded":"2024-07-26T11:13:33.889855+0000","last_fullsized":"2024-07-26T11:13:33.889855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:35:37.629313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:43.386694+0000","last_change":"2024-07-26T11:10:18.055248+0000","last_active":"2024-07-26T11:12:43.386694+0000","last_peered":"2024-07-26T11:12:43.386694+0000","last_clean":"2024-07-26T11:12:43.386694+0000","last_became_active":"2024-07-26T11:10:18.054882+0000","last_became_peered":"2024-07-26T11:10:18.054882+0000","last_unstale":"2024-07-26T11:12:43.386694+0000","last_undegraded":"2024-07-26T11:12:43.386694+0000","last_fullsized":"2024-07-26T11:12:43.386694+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:19:16.390058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:59.941475+0000","last_change":"2024-07-26T11:10:35.985628+0000","last_active":"2024-07-26T11:12:59.941475+0000","last_peered":"2024-07-26T11:12:59.941475+0000","last_clean":"2024-07-26T11:12:59.941475+0000","last_became_active":"2024-07-26T11:10:35.985156+0000","last_became_peered":"2024-07-26T11:10:35.985156+0000","last_unstale":"2024-07-26T11:12:59.941475+0000","last_undegraded":"2024-07-26T11:12:59.941475+0000","last_fullsized":"2024-07-26T11:12:59.941475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:29:18.891825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"90'9","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.893557+0000","last_change":"2024-07-26T11:10:34.973261+0000","last_active":"2024-07-26T11:13:33.893557+0000","last_peered":"2024-07-26T11:13:33.893557+0000","last_clean":"2024-07-26T11:13:33.893557+0000","last_became_active":"2024-07-26T11:10:34.971592+0000","last_became_peered":"2024-07-26T11:10:34.971592+0000","last_unstale":"2024-07-26T11:13:33.893557+0000","last_undegraded":"2024-07-26T11:13:33.893557+0000","last_fullsized":"2024-07-26T11:13:33.893557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:58:13.517884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":54,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:42.352859+0000","last_change":"2024-07-26T11:10:34.962248+0000","last_active":"2024-07-26T11:12:42.352859+0000","last_peered":"2024-07-26T11:12:42.352859+0000","last_clean":"2024-07-26T11:12:42.352859+0000","last_became_active":"2024-07-26T11:10:34.961683+0000","last_became_peered":"2024-07-26T11:10:34.961683+0000","last_unstale":"2024-07-26T11:12:42.352859+0000","last_undegraded":"2024-07-26T11:12:42.352859+0000","last_fullsized":"2024-07-26T11:12:42.352859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:17:08.308226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:56.523407+0000","last_change":"2024-07-26T11:10:35.995760+0000","last_active":"2024-07-26T11:12:56.523407+0000","last_peered":"2024-07-26T11:12:56.523407+0000","last_clean":"2024-07-26T11:12:56.523407+0000","last_became_active":"2024-07-26T11:10:35.995328+0000","last_became_peered":"2024-07-26T11:10:35.995328+0000","last_unstale":"2024-07-26T11:12:56.523407+0000","last_undegraded":"2024-07-26T11:12:56.523407+0000","last_fullsized":"2024-07-26T11:12:56.523407+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:40:33.093315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.16","version":"90'10","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.878655+0000","last_change":"2024-07-26T11:10:22.316151+0000","last_active":"2024-07-26T11:13:33.878655+0000","last_peered":"2024-07-26T11:13:33.878655+0000","last_clean":"2024-07-26T11:13:33.878655+0000","last_became_active":"2024-07-26T11:10:22.315722+0000","last_became_peered":"2024-07-26T11:10:22.315722+0000","last_unstale":"2024-07-26T11:13:33.878655+0000","last_undegraded":"2024-07-26T11:13:33.878655+0000","last_fullsized":"2024-07-26T11:13:33.878655+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:12:58.342637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":529,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:51.481269+0000","last_change":"2024-07-26T11:10:18.046327+0000","last_active":"2024-07-26T11:15:51.481269+0000","last_peered":"2024-07-26T11:15:51.481269+0000","last_clean":"2024-07-26T11:15:51.481269+0000","last_became_active":"2024-07-26T11:10:18.045913+0000","last_became_peered":"2024-07-26T11:10:18.045913+0000","last_unstale":"2024-07-26T11:15:51.481269+0000","last_undegraded":"2024-07-26T11:15:51.481269+0000","last_fullsized":"2024-07-26T11:15:51.481269+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:54:58.889046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":482,"num_read_kb":482,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:59.069724+0000","last_change":"2024-07-26T11:10:35.983264+0000","last_active":"2024-07-26T11:12:59.069724+0000","last_peered":"2024-07-26T11:12:59.069724+0000","last_clean":"2024-07-26T11:12:59.069724+0000","last_became_active":"2024-07-26T11:10:35.982210+0000","last_became_peered":"2024-07-26T11:10:35.982210+0000","last_unstale":"2024-07-26T11:12:59.069724+0000","last_undegraded":"2024-07-26T11:12:59.069724+0000","last_fullsized":"2024-07-26T11:12:59.069724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:37:20.272725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:39.246875+0000","last_change":"2024-07-26T11:10:35.985335+0000","last_active":"2024-07-26T11:12:39.246875+0000","last_peered":"2024-07-26T11:12:39.246875+0000","last_clean":"2024-07-26T11:12:39.246875+0000","last_became_active":"2024-07-26T11:10:35.983667+0000","last_became_peered":"2024-07-26T11:10:35.983667+0000","last_unstale":"2024-07-26T11:12:39.246875+0000","last_undegraded":"2024-07-26T11:12:39.246875+0000","last_fullsized":"2024-07-26T11:12:39.246875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:00:17.744806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"90'11","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.879478+0000","last_change":"2024-07-26T11:10:22.333494+0000","last_active":"2024-07-26T11:13:33.879478+0000","last_peered":"2024-07-26T11:13:33.879478+0000","last_clean":"2024-07-26T11:13:33.879478+0000","last_became_active":"2024-07-26T11:10:22.332950+0000","last_became_peered":"2024-07-26T11:10:22.332950+0000","last_unstale":"2024-07-26T11:13:33.879478+0000","last_undegraded":"2024-07-26T11:13:33.879478+0000","last_fullsized":"2024-07-26T11:13:33.879478+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:58:38.340857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:43.386643+0000","last_change":"2024-07-26T11:10:18.034057+0000","last_active":"2024-07-26T11:12:43.386643+0000","last_peered":"2024-07-26T11:12:43.386643+0000","last_clean":"2024-07-26T11:12:43.386643+0000","last_became_active":"2024-07-26T11:10:18.033119+0000","last_became_peered":"2024-07-26T11:10:18.033119+0000","last_unstale":"2024-07-26T11:12:43.386643+0000","last_undegraded":"2024-07-26T11:12:43.386643+0000","last_fullsized":"2024-07-26T11:12:43.386643+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:21.675593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:42.508793+0000","last_change":"2024-07-26T11:11:35.408377+0000","last_active":"2024-07-26T11:12:42.508793+0000","last_peered":"2024-07-26T11:12:42.508793+0000","last_clean":"2024-07-26T11:12:42.508793+0000","last_became_active":"2024-07-26T11:11:35.407865+0000","last_became_peered":"2024-07-26T11:11:35.407865+0000","last_unstale":"2024-07-26T11:12:42.508793+0000","last_undegraded":"2024-07-26T11:12:42.508793+0000","last_fullsized":"2024-07-26T11:12:42.508793+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:43:02.953395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.18","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.889650+0000","last_change":"2024-07-26T11:10:22.323447+0000","last_active":"2024-07-26T11:13:33.889650+0000","last_peered":"2024-07-26T11:13:33.889650+0000","last_clean":"2024-07-26T11:13:33.889650+0000","last_became_active":"2024-07-26T11:10:22.322988+0000","last_became_peered":"2024-07-26T11:10:22.322988+0000","last_unstale":"2024-07-26T11:13:33.889650+0000","last_undegraded":"2024-07-26T11:13:33.889650+0000","last_fullsized":"2024-07-26T11:13:33.889650+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:14.880985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:46.782728+0000","last_change":"2024-07-26T11:10:18.048724+0000","last_active":"2024-07-26T11:12:46.782728+0000","last_peered":"2024-07-26T11:12:46.782728+0000","last_clean":"2024-07-26T11:12:46.782728+0000","last_became_active":"2024-07-26T11:10:18.048257+0000","last_became_peered":"2024-07-26T11:10:18.048257+0000","last_unstale":"2024-07-26T11:12:46.782728+0000","last_undegraded":"2024-07-26T11:12:46.782728+0000","last_fullsized":"2024-07-26T11:12:46.782728+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:02:13.581446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:12:47.389909+0000","last_change":"2024-07-26T11:10:18.044171+0000","last_active":"2024-07-26T11:12:47.389909+0000","last_peered":"2024-07-26T11:12:47.389909+0000","last_clean":"2024-07-26T11:12:47.389909+0000","last_became_active":"2024-07-26T11:10:18.043682+0000","last_became_peered":"2024-07-26T11:10:18.043682+0000","last_unstale":"2024-07-26T11:12:47.389909+0000","last_undegraded":"2024-07-26T11:12:47.389909+0000","last_fullsized":"2024-07-26T11:12:47.389909+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:46:46.864119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.876589+0000","last_change":"2024-07-26T11:10:22.321081+0000","last_active":"2024-07-26T11:13:33.876589+0000","last_peered":"2024-07-26T11:13:33.876589+0000","last_clean":"2024-07-26T11:13:33.876589+0000","last_became_active":"2024-07-26T11:10:22.320656+0000","last_became_peered":"2024-07-26T11:10:22.320656+0000","last_unstale":"2024-07-26T11:13:33.876589+0000","last_undegraded":"2024-07-26T11:13:33.876589+0000","last_fullsized":"2024-07-26T11:13:33.876589+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:17:20.455129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:09.916452+0000","last_change":"2024-07-26T11:10:36.000224+0000","last_active":"2024-07-26T11:13:09.916452+0000","last_peered":"2024-07-26T11:13:09.916452+0000","last_clean":"2024-07-26T11:13:09.916452+0000","last_became_active":"2024-07-26T11:10:35.999733+0000","last_became_peered":"2024-07-26T11:10:35.999733+0000","last_unstale":"2024-07-26T11:13:09.916452+0000","last_undegraded":"2024-07-26T11:13:09.916452+0000","last_fullsized":"2024-07-26T11:13:09.916452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:40:49.912401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":292,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:15:51.481917+0000","last_change":"2024-07-26T11:10:18.034585+0000","last_active":"2024-07-26T11:15:51.481917+0000","last_peered":"2024-07-26T11:15:51.481917+0000","last_clean":"2024-07-26T11:15:51.481917+0000","last_became_active":"2024-07-26T11:10:18.034237+0000","last_became_peered":"2024-07-26T11:10:18.034237+0000","last_unstale":"2024-07-26T11:15:51.481917+0000","last_undegraded":"2024-07-26T11:15:51.481917+0000","last_fullsized":"2024-07-26T11:15:51.481917+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:16.993830+0000","last_clean_scrub_stamp":"2024-07-26T11:10:16.993830+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:48:22.670277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":244,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.876583+0000","last_change":"2024-07-26T11:10:22.319282+0000","last_active":"2024-07-26T11:13:33.876583+0000","last_peered":"2024-07-26T11:13:33.876583+0000","last_clean":"2024-07-26T11:13:33.876583+0000","last_became_active":"2024-07-26T11:10:22.318822+0000","last_became_peered":"2024-07-26T11:10:22.318822+0000","last_unstale":"2024-07-26T11:13:33.876583+0000","last_undegraded":"2024-07-26T11:13:33.876583+0000","last_fullsized":"2024-07-26T11:13:33.876583+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:50:57.615839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:09.916483+0000","last_change":"2024-07-26T11:10:35.970189+0000","last_active":"2024-07-26T11:13:09.916483+0000","last_peered":"2024-07-26T11:13:09.916483+0000","last_clean":"2024-07-26T11:13:09.916483+0000","last_became_active":"2024-07-26T11:10:35.969574+0000","last_became_peered":"2024-07-26T11:10:35.969574+0000","last_unstale":"2024-07-26T11:13:09.916483+0000","last_undegraded":"2024-07-26T11:13:09.916483+0000","last_fullsized":"2024-07-26T11:13:09.916483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:34.943929+0000","last_clean_scrub_stamp":"2024-07-26T11:10:34.943929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:30:07.487944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.868831+0000","last_change":"2024-07-26T11:10:22.299152+0000","last_active":"2024-07-26T11:13:33.868831+0000","last_peered":"2024-07-26T11:13:33.868831+0000","last_clean":"2024-07-26T11:13:33.868831+0000","last_became_active":"2024-07-26T11:10:22.298707+0000","last_became_peered":"2024-07-26T11:10:22.298707+0000","last_unstale":"2024-07-26T11:13:33.868831+0000","last_undegraded":"2024-07-26T11:13:33.868831+0000","last_fullsized":"2024-07-26T11:13:33.868831+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:49:25.161634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"90'12","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.872159+0000","last_change":"2024-07-26T11:10:22.302476+0000","last_active":"2024-07-26T11:13:33.872159+0000","last_peered":"2024-07-26T11:13:33.872159+0000","last_clean":"2024-07-26T11:13:33.872159+0000","last_became_active":"2024-07-26T11:10:22.302091+0000","last_became_peered":"2024-07-26T11:10:22.302091+0000","last_unstale":"2024-07-26T11:13:33.872159+0000","last_undegraded":"2024-07-26T11:13:33.872159+0000","last_fullsized":"2024-07-26T11:13:33.872159+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:06:18.587588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.892481+0000","last_change":"2024-07-26T11:10:22.326583+0000","last_active":"2024-07-26T11:13:33.892481+0000","last_peered":"2024-07-26T11:13:33.892481+0000","last_clean":"2024-07-26T11:13:33.892481+0000","last_became_active":"2024-07-26T11:10:22.326185+0000","last_became_peered":"2024-07-26T11:10:22.326185+0000","last_unstale":"2024-07-26T11:13:33.892481+0000","last_undegraded":"2024-07-26T11:13:33.892481+0000","last_fullsized":"2024-07-26T11:13:33.892481+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:54:00.148964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.894835+0000","last_change":"2024-07-26T11:10:22.329247+0000","last_active":"2024-07-26T11:13:33.894835+0000","last_peered":"2024-07-26T11:13:33.894835+0000","last_clean":"2024-07-26T11:13:33.894835+0000","last_became_active":"2024-07-26T11:10:22.328585+0000","last_became_peered":"2024-07-26T11:10:22.328585+0000","last_unstale":"2024-07-26T11:13:33.894835+0000","last_undegraded":"2024-07-26T11:13:33.894835+0000","last_fullsized":"2024-07-26T11:13:33.894835+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:01:27.195399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"90'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.895664+0000","last_change":"2024-07-26T11:10:22.330717+0000","last_active":"2024-07-26T11:13:33.895664+0000","last_peered":"2024-07-26T11:13:33.895664+0000","last_clean":"2024-07-26T11:13:33.895664+0000","last_became_active":"2024-07-26T11:10:22.329576+0000","last_became_peered":"2024-07-26T11:10:22.329576+0000","last_unstale":"2024-07-26T11:13:33.895664+0000","last_undegraded":"2024-07-26T11:13:33.895664+0000","last_fullsized":"2024-07-26T11:13:33.895664+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:41:21.970629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.867859+0000","last_change":"2024-07-26T11:10:22.307445+0000","last_active":"2024-07-26T11:13:33.867859+0000","last_peered":"2024-07-26T11:13:33.867859+0000","last_clean":"2024-07-26T11:13:33.867859+0000","last_became_active":"2024-07-26T11:10:22.305738+0000","last_became_peered":"2024-07-26T11:10:22.305738+0000","last_unstale":"2024-07-26T11:13:33.867859+0000","last_undegraded":"2024-07-26T11:13:33.867859+0000","last_fullsized":"2024-07-26T11:13:33.867859+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:09:08.125209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"90'15","reported_seq":52,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874579+0000","last_change":"2024-07-26T11:10:22.298392+0000","last_active":"2024-07-26T11:13:33.874579+0000","last_peered":"2024-07-26T11:13:33.874579+0000","last_clean":"2024-07-26T11:13:33.874579+0000","last_became_active":"2024-07-26T11:10:22.297979+0000","last_became_peered":"2024-07-26T11:10:22.297979+0000","last_unstale":"2024-07-26T11:13:33.874579+0000","last_undegraded":"2024-07-26T11:13:33.874579+0000","last_fullsized":"2024-07-26T11:13:33.874579+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:30:39.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"90'2","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.873814+0000","last_change":"2024-07-26T11:10:22.320850+0000","last_active":"2024-07-26T11:13:33.873814+0000","last_peered":"2024-07-26T11:13:33.873814+0000","last_clean":"2024-07-26T11:13:33.873814+0000","last_became_active":"2024-07-26T11:10:22.319838+0000","last_became_peered":"2024-07-26T11:10:22.319838+0000","last_unstale":"2024-07-26T11:13:33.873814+0000","last_undegraded":"2024-07-26T11:13:33.873814+0000","last_fullsized":"2024-07-26T11:13:33.873814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:19:14.289366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.873192+0000","last_change":"2024-07-26T11:10:22.321305+0000","last_active":"2024-07-26T11:13:33.873192+0000","last_peered":"2024-07-26T11:13:33.873192+0000","last_clean":"2024-07-26T11:13:33.873192+0000","last_became_active":"2024-07-26T11:10:22.320733+0000","last_became_peered":"2024-07-26T11:10:22.320733+0000","last_unstale":"2024-07-26T11:13:33.873192+0000","last_undegraded":"2024-07-26T11:13:33.873192+0000","last_fullsized":"2024-07-26T11:13:33.873192+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:24:33.125828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.874816+0000","last_change":"2024-07-26T11:10:22.318193+0000","last_active":"2024-07-26T11:13:33.874816+0000","last_peered":"2024-07-26T11:13:33.874816+0000","last_clean":"2024-07-26T11:13:33.874816+0000","last_became_active":"2024-07-26T11:10:22.317819+0000","last_became_peered":"2024-07-26T11:10:22.317819+0000","last_unstale":"2024-07-26T11:13:33.874816+0000","last_undegraded":"2024-07-26T11:13:33.874816+0000","last_fullsized":"2024-07-26T11:13:33.874816+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:14:06.105152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"90'10","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875254+0000","last_change":"2024-07-26T11:10:22.318222+0000","last_active":"2024-07-26T11:13:33.875254+0000","last_peered":"2024-07-26T11:13:33.875254+0000","last_clean":"2024-07-26T11:13:33.875254+0000","last_became_active":"2024-07-26T11:10:22.317806+0000","last_became_peered":"2024-07-26T11:10:22.317806+0000","last_unstale":"2024-07-26T11:13:33.875254+0000","last_undegraded":"2024-07-26T11:13:33.875254+0000","last_fullsized":"2024-07-26T11:13:33.875254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:55:27.510542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"90'11","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.895333+0000","last_change":"2024-07-26T11:10:22.330166+0000","last_active":"2024-07-26T11:13:33.895333+0000","last_peered":"2024-07-26T11:13:33.895333+0000","last_clean":"2024-07-26T11:13:33.895333+0000","last_became_active":"2024-07-26T11:10:22.329722+0000","last_became_peered":"2024-07-26T11:10:22.329722+0000","last_unstale":"2024-07-26T11:13:33.895333+0000","last_undegraded":"2024-07-26T11:13:33.895333+0000","last_fullsized":"2024-07-26T11:13:33.895333+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:02:58.816139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.866660+0000","last_change":"2024-07-26T11:10:22.302774+0000","last_active":"2024-07-26T11:13:33.866660+0000","last_peered":"2024-07-26T11:13:33.866660+0000","last_clean":"2024-07-26T11:13:33.866660+0000","last_became_active":"2024-07-26T11:10:22.302382+0000","last_became_peered":"2024-07-26T11:10:22.302382+0000","last_unstale":"2024-07-26T11:13:33.866660+0000","last_undegraded":"2024-07-26T11:13:33.866660+0000","last_fullsized":"2024-07-26T11:13:33.866660+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:02:58.019598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2d","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.886317+0000","last_change":"2024-07-26T11:10:22.312475+0000","last_active":"2024-07-26T11:13:33.886317+0000","last_peered":"2024-07-26T11:13:33.886317+0000","last_clean":"2024-07-26T11:13:33.886317+0000","last_became_active":"2024-07-26T11:10:22.312078+0000","last_became_peered":"2024-07-26T11:10:22.312078+0000","last_unstale":"2024-07-26T11:13:33.886317+0000","last_undegraded":"2024-07-26T11:13:33.886317+0000","last_fullsized":"2024-07-26T11:13:33.886317+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T18:25:37.705136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"90'10","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.878921+0000","last_change":"2024-07-26T11:10:22.314776+0000","last_active":"2024-07-26T11:13:33.878921+0000","last_peered":"2024-07-26T11:13:33.878921+0000","last_clean":"2024-07-26T11:13:33.878921+0000","last_became_active":"2024-07-26T11:10:22.307808+0000","last_became_peered":"2024-07-26T11:10:22.307808+0000","last_unstale":"2024-07-26T11:13:33.878921+0000","last_undegraded":"2024-07-26T11:13:33.878921+0000","last_fullsized":"2024-07-26T11:13:33.878921+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:44:06.152799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.891140+0000","last_change":"2024-07-26T11:10:22.328102+0000","last_active":"2024-07-26T11:13:33.891140+0000","last_peered":"2024-07-26T11:13:33.891140+0000","last_clean":"2024-07-26T11:13:33.891140+0000","last_became_active":"2024-07-26T11:10:22.327779+0000","last_became_peered":"2024-07-26T11:10:22.327779+0000","last_unstale":"2024-07-26T11:13:33.891140+0000","last_undegraded":"2024-07-26T11:13:33.891140+0000","last_fullsized":"2024-07-26T11:13:33.891140+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:14:34.569748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.30","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.888331+0000","last_change":"2024-07-26T11:10:22.329057+0000","last_active":"2024-07-26T11:13:33.888331+0000","last_peered":"2024-07-26T11:13:33.888331+0000","last_clean":"2024-07-26T11:13:33.888331+0000","last_became_active":"2024-07-26T11:10:22.328656+0000","last_became_peered":"2024-07-26T11:10:22.328656+0000","last_unstale":"2024-07-26T11:13:33.888331+0000","last_undegraded":"2024-07-26T11:13:33.888331+0000","last_fullsized":"2024-07-26T11:13:33.888331+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:18:39.861674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.891998+0000","last_change":"2024-07-26T11:10:22.326115+0000","last_active":"2024-07-26T11:13:33.891998+0000","last_peered":"2024-07-26T11:13:33.891998+0000","last_clean":"2024-07-26T11:13:33.891998+0000","last_became_active":"2024-07-26T11:10:22.325538+0000","last_became_peered":"2024-07-26T11:10:22.325538+0000","last_unstale":"2024-07-26T11:13:33.891998+0000","last_undegraded":"2024-07-26T11:13:33.891998+0000","last_fullsized":"2024-07-26T11:13:33.891998+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:18:19.339571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.32","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.892537+0000","last_change":"2024-07-26T11:10:22.326910+0000","last_active":"2024-07-26T11:13:33.892537+0000","last_peered":"2024-07-26T11:13:33.892537+0000","last_clean":"2024-07-26T11:13:33.892537+0000","last_became_active":"2024-07-26T11:10:22.324532+0000","last_became_peered":"2024-07-26T11:10:22.324532+0000","last_unstale":"2024-07-26T11:13:33.892537+0000","last_undegraded":"2024-07-26T11:13:33.892537+0000","last_fullsized":"2024-07-26T11:13:33.892537+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T11:21:33.671780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"90'10","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.890787+0000","last_change":"2024-07-26T11:10:22.312926+0000","last_active":"2024-07-26T11:13:33.890787+0000","last_peered":"2024-07-26T11:13:33.890787+0000","last_clean":"2024-07-26T11:13:33.890787+0000","last_became_active":"2024-07-26T11:10:22.311642+0000","last_became_peered":"2024-07-26T11:10:22.311642+0000","last_unstale":"2024-07-26T11:13:33.890787+0000","last_undegraded":"2024-07-26T11:13:33.890787+0000","last_fullsized":"2024-07-26T11:13:33.890787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T16:56:29.644965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"90'7","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.879104+0000","last_change":"2024-07-26T11:10:34.976934+0000","last_active":"2024-07-26T11:13:33.879104+0000","last_peered":"2024-07-26T11:13:33.879104+0000","last_clean":"2024-07-26T11:13:33.879104+0000","last_became_active":"2024-07-26T11:10:34.976551+0000","last_became_peered":"2024-07-26T11:10:34.976551+0000","last_unstale":"2024-07-26T11:13:33.879104+0000","last_undegraded":"2024-07-26T11:13:33.879104+0000","last_fullsized":"2024-07-26T11:13:33.879104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T20:11:43.883644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.35","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875269+0000","last_change":"2024-07-26T11:10:22.304444+0000","last_active":"2024-07-26T11:13:33.875269+0000","last_peered":"2024-07-26T11:13:33.875269+0000","last_clean":"2024-07-26T11:13:33.875269+0000","last_became_active":"2024-07-26T11:10:22.302825+0000","last_became_peered":"2024-07-26T11:10:22.302825+0000","last_unstale":"2024-07-26T11:13:33.875269+0000","last_undegraded":"2024-07-26T11:13:33.875269+0000","last_fullsized":"2024-07-26T11:13:33.875269+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:04:15.229057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"90'16","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.870763+0000","last_change":"2024-07-26T11:10:22.297515+0000","last_active":"2024-07-26T11:13:33.870763+0000","last_peered":"2024-07-26T11:13:33.870763+0000","last_clean":"2024-07-26T11:13:33.870763+0000","last_became_active":"2024-07-26T11:10:22.297134+0000","last_became_peered":"2024-07-26T11:10:22.297134+0000","last_unstale":"2024-07-26T11:13:33.870763+0000","last_undegraded":"2024-07-26T11:13:33.870763+0000","last_fullsized":"2024-07-26T11:13:33.870763+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:25:08.794363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.37","version":"90'11","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875681+0000","last_change":"2024-07-26T11:10:22.304333+0000","last_active":"2024-07-26T11:13:33.875681+0000","last_peered":"2024-07-26T11:13:33.875681+0000","last_clean":"2024-07-26T11:13:33.875681+0000","last_became_active":"2024-07-26T11:10:22.303639+0000","last_became_peered":"2024-07-26T11:10:22.303639+0000","last_unstale":"2024-07-26T11:13:33.875681+0000","last_undegraded":"2024-07-26T11:13:33.875681+0000","last_fullsized":"2024-07-26T11:13:33.875681+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:06:47.053463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.38","version":"90'9","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.875130+0000","last_change":"2024-07-26T11:10:22.320998+0000","last_active":"2024-07-26T11:13:33.875130+0000","last_peered":"2024-07-26T11:13:33.875130+0000","last_clean":"2024-07-26T11:13:33.875130+0000","last_became_active":"2024-07-26T11:10:22.320610+0000","last_became_peered":"2024-07-26T11:10:22.320610+0000","last_unstale":"2024-07-26T11:13:33.875130+0000","last_undegraded":"2024-07-26T11:13:33.875130+0000","last_fullsized":"2024-07-26T11:13:33.875130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T19:52:28.802241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"90'5","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.873030+0000","last_change":"2024-07-26T11:10:34.970486+0000","last_active":"2024-07-26T11:13:33.873030+0000","last_peered":"2024-07-26T11:13:33.873030+0000","last_clean":"2024-07-26T11:13:33.873030+0000","last_became_active":"2024-07-26T11:10:34.969965+0000","last_became_peered":"2024-07-26T11:10:34.969965+0000","last_unstale":"2024-07-26T11:13:33.873030+0000","last_undegraded":"2024-07-26T11:13:33.873030+0000","last_fullsized":"2024-07-26T11:13:33.873030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T13:12:41.593251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"90'6","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.876555+0000","last_change":"2024-07-26T11:10:22.321261+0000","last_active":"2024-07-26T11:13:33.876555+0000","last_peered":"2024-07-26T11:13:33.876555+0000","last_clean":"2024-07-26T11:13:33.876555+0000","last_became_active":"2024-07-26T11:10:22.320819+0000","last_became_peered":"2024-07-26T11:10:22.320819+0000","last_unstale":"2024-07-26T11:13:33.876555+0000","last_undegraded":"2024-07-26T11:13:33.876555+0000","last_fullsized":"2024-07-26T11:13:33.876555+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T23:04:56.872600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.3b","version":"90'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.887727+0000","last_change":"2024-07-26T11:10:22.321206+0000","last_active":"2024-07-26T11:13:33.887727+0000","last_peered":"2024-07-26T11:13:33.887727+0000","last_clean":"2024-07-26T11:13:33.887727+0000","last_became_active":"2024-07-26T11:10:22.320700+0000","last_became_peered":"2024-07-26T11:10:22.320700+0000","last_unstale":"2024-07-26T11:13:33.887727+0000","last_undegraded":"2024-07-26T11:13:33.887727+0000","last_fullsized":"2024-07-26T11:13:33.887727+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T14:31:20.700808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3c","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.870903+0000","last_change":"2024-07-26T11:10:22.300061+0000","last_active":"2024-07-26T11:13:33.870903+0000","last_peered":"2024-07-26T11:13:33.870903+0000","last_clean":"2024-07-26T11:13:33.870903+0000","last_became_active":"2024-07-26T11:10:22.298737+0000","last_became_peered":"2024-07-26T11:10:22.298737+0000","last_unstale":"2024-07-26T11:13:33.870903+0000","last_undegraded":"2024-07-26T11:13:33.870903+0000","last_fullsized":"2024-07-26T11:13:33.870903+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T12:45:25.500816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3d","version":"90'7","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.893213+0000","last_change":"2024-07-26T11:10:22.328534+0000","last_active":"2024-07-26T11:13:33.893213+0000","last_peered":"2024-07-26T11:13:33.893213+0000","last_clean":"2024-07-26T11:13:33.893213+0000","last_became_active":"2024-07-26T11:10:22.327929+0000","last_became_peered":"2024-07-26T11:10:22.327929+0000","last_unstale":"2024-07-26T11:13:33.893213+0000","last_undegraded":"2024-07-26T11:13:33.893213+0000","last_fullsized":"2024-07-26T11:13:33.893213+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T21:09:55.152881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"90'4","reported_seq":41,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.892299+0000","last_change":"2024-07-26T11:10:22.324753+0000","last_active":"2024-07-26T11:13:33.892299+0000","last_peered":"2024-07-26T11:13:33.892299+0000","last_clean":"2024-07-26T11:13:33.892299+0000","last_became_active":"2024-07-26T11:10:22.324307+0000","last_became_peered":"2024-07-26T11:10:22.324307+0000","last_unstale":"2024-07-26T11:13:33.892299+0000","last_undegraded":"2024-07-26T11:13:33.892299+0000","last_fullsized":"2024-07-26T11:13:33.892299+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T17:26:40.566465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3f","version":"90'8","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-07-26T11:13:33.894283+0000","last_change":"2024-07-26T11:10:22.328183+0000","last_active":"2024-07-26T11:13:33.894283+0000","last_peered":"2024-07-26T11:13:33.894283+0000","last_clean":"2024-07-26T11:13:33.894283+0000","last_became_active":"2024-07-26T11:10:22.327805+0000","last_became_peered":"2024-07-26T11:10:22.327805+0000","last_unstale":"2024-07-26T11:13:33.894283+0000","last_undegraded":"2024-07-26T11:13:33.894283+0000","last_fullsized":"2024-07-26T11:13:33.894283+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-07-26T11:10:21.266878+0000","last_clean_scrub_stamp":"2024-07-26T11:10:21.266878+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-07-27T22:11:23.238390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":79710,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":2463,"num_read_kb":1316,"num_write":265,"num_write_kb":162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2194,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":602112,"data_stored":239130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":96,"acting":96,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":69,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8}],"osd_stats":[{"osd":11,"up_from":73,"seq":313532612691,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":527167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":67,"seq":287762808919,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28712,"kb_used_data":1696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716440,"statfs":{"total":95995035648,"available":95965634560,"internally_reserved":0,"allocated":1736704,"data_stored":1186017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":62,"seq":266287972445,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":529939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":56,"seq":240518168676,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":529735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":50,"seq":214748364904,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28620,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716532,"statfs":{"total":95995035648,"available":95965728768,"internally_reserved":0,"allocated":1642496,"data_stored":1122494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528430,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":521347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724662,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36952,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708200,"statfs":{"total":95995035648,"available":95957196800,"internally_reserved":0,"allocated":1654784,"data_stored":1113813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888187,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28112,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717040,"statfs":{"total":95995035648,"available":95966248960,"internally_reserved":0,"allocated":1122304,"data_stored":598709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":28,"seq":120259084415,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1044480,"data_stored":521055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378757,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":522597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542282,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":594826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705806,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1056768,"data_stored":522803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":74201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":73988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":75024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-07-26T11:15:56.329 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:15:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:56 smithi134 ceph-mon[31966]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:15:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:56 smithi134 ceph-mon[31966]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:15:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:56 smithi134 ceph-mon[31966]: pgmap v479: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-07-26T11:15:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:56 smithi189 ceph-mon[36790]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:15:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:56 smithi189 ceph-mon[36790]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:15:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:56 smithi189 ceph-mon[36790]: pgmap v479: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-07-26T11:15:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:56 smithi193 ceph-mon[37473]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:15:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:56 smithi193 ceph-mon[37473]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:15:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:56 smithi193 ceph-mon[37473]: pgmap v479: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 938 B/s wr, 9 op/s 2024-07-26T11:15:56.807 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:15:57.328 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:57 smithi134 ceph-mon[31966]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:15:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:57 smithi189 ceph-mon[36790]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:15:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:57 smithi193 ceph-mon[37473]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-07-26T11:15:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:58 smithi134 ceph-mon[31966]: pgmap v480: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 12 op/s 2024-07-26T11:15:58.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:58 smithi189 ceph-mon[36790]: pgmap v480: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 12 op/s 2024-07-26T11:15:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:58 smithi193 ceph-mon[37473]: pgmap v480: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 938 B/s wr, 12 op/s 2024-07-26T11:15:59.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.037+0000 7f9844a00640 1 -- 172.21.15.134:0/2626943380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f984010fc80 msgr2=0x7f98401164b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:59.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.037+0000 7f9844a00640 1 --2- 172.21.15.134:0/2626943380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f984010fc80 0x7f98401164b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9838009ca0 tx=0x7f983802a8b0 comp rx=0 tx=0).stop 2024-07-26T11:15:59.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.038+0000 7f9844a00640 1 -- 172.21.15.134:0/2626943380 shutdown_connections 2024-07-26T11:15:59.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.038+0000 7f9844a00640 1 --2- 172.21.15.134:0/2626943380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f984010fc80 0x7f98401164b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:59.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.038+0000 7f9844a00640 1 --2- 172.21.15.134:0/2626943380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f984010f330 0x7f984010f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:59.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.038+0000 7f9844a00640 1 --2- 172.21.15.134:0/2626943380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98401099f0 0x7f9840109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:59.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.038+0000 7f9844a00640 1 -- 172.21.15.134:0/2626943380 >> 172.21.15.134:0/2626943380 conn(0x7f9840078ff0 msgr2=0x7f9840107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:59.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.038+0000 7f9844a00640 1 -- 172.21.15.134:0/2626943380 shutdown_connections 2024-07-26T11:15:59.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.038+0000 7f9844a00640 1 -- 172.21.15.134:0/2626943380 wait complete. 2024-07-26T11:15:59.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.039+0000 7f9844a00640 1 Processor -- start 2024-07-26T11:15:59.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.040+0000 7f9844a00640 1 -- start start 2024-07-26T11:15:59.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.040+0000 7f9844a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98401099f0 0x7f98401a0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:59.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.040+0000 7f9844a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f984010f330 0x7f98401a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:59.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.040+0000 7f9844a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f984010fc80 0x7f98401a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:59.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.040+0000 7f9844a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9840119940 con 0x7f98401099f0 2024-07-26T11:15:59.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.040+0000 7f9844a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f98401197c0 con 0x7f984010fc80 2024-07-26T11:15:59.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.040+0000 7f9844a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9840119ac0 con 0x7f984010f330 2024-07-26T11:15:59.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.041+0000 7f983e800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98401099f0 0x7f98401a0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:59.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.041+0000 7f983f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f984010fc80 0x7f98401a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:59.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.041+0000 7f983e800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98401099f0 0x7f98401a0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:53300/0 (socket says 172.21.15.134:53300) 2024-07-26T11:15:59.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.041+0000 7f983e800640 1 -- 172.21.15.134:0/3792871967 learned_addr learned my addr 172.21.15.134:0/3792871967 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:15:59.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.041+0000 7f983de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f984010f330 0x7f98401a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:59.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.041+0000 7f983f200640 1 -- 172.21.15.134:0/3792871967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f984010f330 msgr2=0x7f98401a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:59.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.041+0000 7f983f200640 1 --2- 172.21.15.134:0/3792871967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f984010f330 0x7f98401a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:59.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.041+0000 7f983f200640 1 -- 172.21.15.134:0/3792871967 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98401099f0 msgr2=0x7f98401a0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:59.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.041+0000 7f983f200640 1 --2- 172.21.15.134:0/3792871967 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98401099f0 0x7f98401a0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:59.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.041+0000 7f983f200640 1 -- 172.21.15.134:0/3792871967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98401a62c0 con 0x7f984010fc80 2024-07-26T11:15:59.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.041+0000 7f983e800640 1 --2- 172.21.15.134:0/3792871967 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98401099f0 0x7f98401a0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:59.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.041+0000 7f983de00640 1 --2- 172.21.15.134:0/3792871967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f984010f330 0x7f98401a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:15:59.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.042+0000 7f983f200640 1 --2- 172.21.15.134:0/3792871967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f984010fc80 0x7f98401a5b90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f983802add0 tx=0x7f9838030d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:59.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.042+0000 7f9837400640 1 -- 172.21.15.134:0/3792871967 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9838003e40 con 0x7f984010fc80 2024-07-26T11:15:59.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.042+0000 7f9837400640 1 -- 172.21.15.134:0/3792871967 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9838040070 con 0x7f984010fc80 2024-07-26T11:15:59.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.042+0000 7f9844a00640 1 -- 172.21.15.134:0/3792871967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98401a6550 con 0x7f984010fc80 2024-07-26T11:15:59.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.042+0000 7f9837400640 1 -- 172.21.15.134:0/3792871967 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98380459a0 con 0x7f984010fc80 2024-07-26T11:15:59.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.042+0000 7f9844a00640 1 -- 172.21.15.134:0/3792871967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98401acd10 con 0x7f984010fc80 2024-07-26T11:15:59.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.043+0000 7f9844a00640 1 -- 172.21.15.134:0/3792871967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9840109e00 con 0x7f984010fc80 2024-07-26T11:15:59.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.046+0000 7f9837400640 1 -- 172.21.15.134:0/3792871967 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7f9838004680 con 0x7f984010fc80 2024-07-26T11:15:59.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.047+0000 7f9837400640 1 --2- 172.21.15.134:0/3792871967 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98240b3ca0 0x7f98240b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:15:59.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.048+0000 7f9837400640 1 -- 172.21.15.134:0/3792871967 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f98381051e0 con 0x7f984010fc80 2024-07-26T11:15:59.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.048+0000 7f983e800640 1 --2- 172.21.15.134:0/3792871967 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98240b3ca0 0x7f98240b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:15:59.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.049+0000 7f983e800640 1 --2- 172.21.15.134:0/3792871967 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98240b3ca0 0x7f98240b61b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9830011500 tx=0x7f9830005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:15:59.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.049+0000 7f9837400640 1 -- 172.21.15.134:0/3792871967 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f98380cc450 con 0x7f984010fc80 2024-07-26T11:15:59.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.289+0000 7f9844a00640 1 -- 172.21.15.134:0/3792871967 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98401a1480 con 0x7f984010fc80 2024-07-26T11:15:59.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.290+0000 7f9837400640 1 -- 172.21.15.134:0/3792871967 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11990 (secure 0 0 0) 0x7f98401a1480 con 0x7f984010fc80 2024-07-26T11:15:59.293 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:15:59.293 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 30 2024-07-26T11:15:59.293 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":30,"btime":"2024-07-26T11:15:47:899566+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:15:47.849285+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24721,"mds_1":24719,"mds_2":24703,"mds_3":24688,"mds_4":24686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24635":{"gid":24635,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24641":{"gid":24641,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":32,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24719":{"gid":24719,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":28,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24721,"qdb_cluster":[24721,24719,24703,24688,24686]},"id":1}]} 2024-07-26T11:15:59.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.293+0000 7f9844a00640 1 -- 172.21.15.134:0/3792871967 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98240b3ca0 msgr2=0x7f98240b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:59.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.293+0000 7f9844a00640 1 --2- 172.21.15.134:0/3792871967 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98240b3ca0 0x7f98240b61b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9830011500 tx=0x7f9830005f90 comp rx=0 tx=0).stop 2024-07-26T11:15:59.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.293+0000 7f9844a00640 1 -- 172.21.15.134:0/3792871967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f984010fc80 msgr2=0x7f98401a5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:15:59.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.293+0000 7f9844a00640 1 --2- 172.21.15.134:0/3792871967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f984010fc80 0x7f98401a5b90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f983802add0 tx=0x7f9838030d80 comp rx=0 tx=0).stop 2024-07-26T11:15:59.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.293+0000 7f9844a00640 1 -- 172.21.15.134:0/3792871967 shutdown_connections 2024-07-26T11:15:59.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.293+0000 7f9844a00640 1 --2- 172.21.15.134:0/3792871967 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98240b3ca0 0x7f98240b61b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:59.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.293+0000 7f9844a00640 1 --2- 172.21.15.134:0/3792871967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f984010f330 0x7f98401a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:59.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.293+0000 7f9844a00640 1 --2- 172.21.15.134:0/3792871967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f984010fc80 0x7f98401a5b90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:59.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.293+0000 7f9844a00640 1 --2- 172.21.15.134:0/3792871967 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98401099f0 0x7f98401a0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:15:59.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.293+0000 7f9844a00640 1 -- 172.21.15.134:0/3792871967 >> 172.21.15.134:0/3792871967 conn(0x7f9840078ff0 msgr2=0x7f9840079710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:15:59.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.293+0000 7f9844a00640 1 -- 172.21.15.134:0/3792871967 shutdown_connections 2024-07-26T11:15:59.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:15:59.293+0000 7f9844a00640 1 -- 172.21.15.134:0/3792871967 wait complete. 2024-07-26T11:15:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:15:59 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3792871967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:15:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:15:59 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3792871967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:15:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:15:59 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3792871967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:16:00.191 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph osd dump --format=json 2024-07-26T11:16:00.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:00 smithi134 ceph-mon[31966]: pgmap v481: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 10 op/s 2024-07-26T11:16:00.656 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:16:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:00 smithi189 ceph-mon[36790]: pgmap v481: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 10 op/s 2024-07-26T11:16:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:00 smithi193 ceph-mon[37473]: pgmap v481: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 10 op/s 2024-07-26T11:16:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:02 smithi134 ceph-mon[31966]: pgmap v482: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 596 B/s wr, 9 op/s 2024-07-26T11:16:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:02 smithi189 ceph-mon[36790]: pgmap v482: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 596 B/s wr, 9 op/s 2024-07-26T11:16:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:02 smithi193 ceph-mon[37473]: pgmap v482: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 596 B/s wr, 9 op/s 2024-07-26T11:16:03.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.083+0000 7fc731600640 1 -- 172.21.15.134:0/797716797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72c0751e0 msgr2=0x7fc72c0755f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:03.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.083+0000 7fc731600640 1 --2- 172.21.15.134:0/797716797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72c0751e0 0x7fc72c0755f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009ca0 tx=0x7fc71c02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:16:03.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.083+0000 7fc731600640 1 -- 172.21.15.134:0/797716797 shutdown_connections 2024-07-26T11:16:03.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.083+0000 7fc731600640 1 --2- 172.21.15.134:0/797716797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc72c075b30 0x7fc72c10f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:03.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.083+0000 7fc731600640 1 --2- 172.21.15.134:0/797716797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc72c10f940 0x7fc72c111d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:03.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.083+0000 7fc731600640 1 --2- 172.21.15.134:0/797716797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72c0751e0 0x7fc72c0755f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:03.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.083+0000 7fc731600640 1 -- 172.21.15.134:0/797716797 >> 172.21.15.134:0/797716797 conn(0x7fc72c0fee70 msgr2=0x7fc72c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:03.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.083+0000 7fc731600640 1 -- 172.21.15.134:0/797716797 shutdown_connections 2024-07-26T11:16:03.086 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.083+0000 7fc731600640 1 -- 172.21.15.134:0/797716797 wait complete. 2024-07-26T11:16:03.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.084+0000 7fc731600640 1 Processor -- start 2024-07-26T11:16:03.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.084+0000 7fc731600640 1 -- start start 2024-07-26T11:16:03.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.085+0000 7fc731600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc72c0751e0 0x7fc72c19bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:03.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.085+0000 7fc731600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc72c075b30 0x7fc72c19c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:03.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.085+0000 7fc731600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72c10f940 0x7fc72c1a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:03.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.085+0000 7fc731600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c115520 con 0x7fc72c075b30 2024-07-26T11:16:03.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.085+0000 7fc731600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c1153a0 con 0x7fc72c10f940 2024-07-26T11:16:03.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.085+0000 7fc731600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fc72c1156a0 con 0x7fc72c0751e0 2024-07-26T11:16:03.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.085+0000 7fc72be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc72c0751e0 0x7fc72c19bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:03.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.085+0000 7fc72b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc72c075b30 0x7fc72c19c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:03.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.085+0000 7fc730c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72c10f940 0x7fc72c1a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:03.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.085+0000 7fc72be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc72c0751e0 0x7fc72c19bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:36324/0 (socket says 172.21.15.134:36324) 2024-07-26T11:16:03.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.085+0000 7fc730c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72c10f940 0x7fc72c1a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:36254/0 (socket says 172.21.15.134:36254) 2024-07-26T11:16:03.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.085+0000 7fc72b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc72c075b30 0x7fc72c19c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:33950/0 (socket says 172.21.15.134:33950) 2024-07-26T11:16:03.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.085+0000 7fc72be00640 1 -- 172.21.15.134:0/1819566820 learned_addr learned my addr 172.21.15.134:0/1819566820 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:03.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.086+0000 7fc730c00640 1 -- 172.21.15.134:0/1819566820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc72c0751e0 msgr2=0x7fc72c19bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:03.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.086+0000 7fc730c00640 1 --2- 172.21.15.134:0/1819566820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc72c0751e0 0x7fc72c19bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:03.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.086+0000 7fc730c00640 1 -- 172.21.15.134:0/1819566820 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc72c075b30 msgr2=0x7fc72c19c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:03.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.086+0000 7fc730c00640 1 --2- 172.21.15.134:0/1819566820 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc72c075b30 0x7fc72c19c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:03.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.086+0000 7fc730c00640 1 -- 172.21.15.134:0/1819566820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc72c1a1ee0 con 0x7fc72c10f940 2024-07-26T11:16:03.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.086+0000 7fc730c00640 1 --2- 172.21.15.134:0/1819566820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72c10f940 0x7fc72c1a17b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc72400e970 tx=0x7fc72400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:03.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.086+0000 7fc728c00640 1 -- 172.21.15.134:0/1819566820 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72400ccc0 con 0x7fc72c10f940 2024-07-26T11:16:03.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.086+0000 7fc728c00640 1 -- 172.21.15.134:0/1819566820 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc724005020 con 0x7fc72c10f940 2024-07-26T11:16:03.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.086+0000 7fc731600640 1 -- 172.21.15.134:0/1819566820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc72c1a21d0 con 0x7fc72c10f940 2024-07-26T11:16:03.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.087+0000 7fc728c00640 1 -- 172.21.15.134:0/1819566820 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc724004a00 con 0x7fc72c10f940 2024-07-26T11:16:03.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.087+0000 7fc731600640 1 -- 172.21.15.134:0/1819566820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc72c1a8930 con 0x7fc72c10f940 2024-07-26T11:16:03.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.088+0000 7fc731600640 1 -- 172.21.15.134:0/1819566820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc72c0755f0 con 0x7fc72c10f940 2024-07-26T11:16:03.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.090+0000 7fc728c00640 1 -- 172.21.15.134:0/1819566820 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fc724004ba0 con 0x7fc72c10f940 2024-07-26T11:16:03.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.092+0000 7fc728c00640 1 --2- 172.21.15.134:0/1819566820 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc7100b3c20 0x7fc7100b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:03.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.092+0000 7fc728c00640 1 -- 172.21.15.134:0/1819566820 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fc7240e40c0 con 0x7fc72c10f940 2024-07-26T11:16:03.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.092+0000 7fc72be00640 1 --2- 172.21.15.134:0/1819566820 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc7100b3c20 0x7fc7100b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:03.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.093+0000 7fc72be00640 1 --2- 172.21.15.134:0/1819566820 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc7100b3c20 0x7fc7100b6130 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009a60 tx=0x7fc71c005d80 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:03.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.094+0000 7fc728c00640 1 -- 172.21.15.134:0/1819566820 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fc7240ab2a0 con 0x7fc72c10f940 2024-07-26T11:16:03.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.294+0000 7fc731600640 1 -- 172.21.15.134:0/1819566820 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc72c19d0a0 con 0x7fc72c10f940 2024-07-26T11:16:03.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.296+0000 7fc728c00640 1 -- 172.21.15.134:0/1819566820 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+25185 (secure 0 0 0) 0x7fc7240aaba0 con 0x7fc72c10f940 2024-07-26T11:16:03.299 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:16:03.299 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":90,"fsid":"105fbe5c-4b3e-11ef-bcb7-c7b262605968","created":"2024-07-26T11:00:05.955103+0000","modified":"2024-07-26T11:11:35.149490+0000","last_up_change":"2024-07-26T11:09:11.485512+0000","last_in_change":"2024-07-26T11:08:52.341356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-07-26T11:05:01.849170+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-07-26T11:10:16.055725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-07-26T11:10:21.216611+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-07-26T11:10:33.983067+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8abb1365-fc3e-4617-a9ac-76bece8cd7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6803","nonce":4183273954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6805","nonce":4183273954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6809","nonce":4183273954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":4183273954},{"type":"v1","addr":"172.21.15.134:6807","nonce":4183273954}]},"public_addr":"172.21.15.134:6803/4183273954","cluster_addr":"172.21.15.134:6805/4183273954","heartbeat_back_addr":"172.21.15.134:6809/4183273954","heartbeat_front_addr":"172.21.15.134:6807/4183273954","state":["exists","up"]},{"osd":1,"uuid":"6c5a7838-821e-44e1-97e4-655b9bfa9f38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6801","nonce":35148049}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6803","nonce":35148049}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6807","nonce":35148049}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":35148049},{"type":"v1","addr":"172.21.15.189:6805","nonce":35148049}]},"public_addr":"172.21.15.189:6801/35148049","cluster_addr":"172.21.15.189:6803/35148049","heartbeat_back_addr":"172.21.15.189:6807/35148049","heartbeat_front_addr":"172.21.15.189:6805/35148049","state":["exists","up"]},{"osd":2,"uuid":"8bc42f7f-39fa-4d3b-bc17-21c68f9231fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6801","nonce":897516976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6803","nonce":897516976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6807","nonce":897516976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":897516976},{"type":"v1","addr":"172.21.15.193:6805","nonce":897516976}]},"public_addr":"172.21.15.193:6801/897516976","cluster_addr":"172.21.15.193:6803/897516976","heartbeat_back_addr":"172.21.15.193:6807/897516976","heartbeat_front_addr":"172.21.15.193:6805/897516976","state":["exists","up"]},{"osd":3,"uuid":"0473d055-0b79-434f-9658-28269d2b1fbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":28,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6811","nonce":3457190917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6813","nonce":3457190917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6817","nonce":3457190917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":3457190917},{"type":"v1","addr":"172.21.15.134:6815","nonce":3457190917}]},"public_addr":"172.21.15.134:6811/3457190917","cluster_addr":"172.21.15.134:6813/3457190917","heartbeat_back_addr":"172.21.15.134:6817/3457190917","heartbeat_front_addr":"172.21.15.134:6815/3457190917","state":["exists","up"]},{"osd":4,"uuid":"1d70eaae-56cc-4cdc-8f83-b88d2121f0ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6809","nonce":1786881347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6811","nonce":1786881347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6815","nonce":1786881347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1786881347},{"type":"v1","addr":"172.21.15.189:6813","nonce":1786881347}]},"public_addr":"172.21.15.189:6809/1786881347","cluster_addr":"172.21.15.189:6811/1786881347","heartbeat_back_addr":"172.21.15.189:6815/1786881347","heartbeat_front_addr":"172.21.15.189:6813/1786881347","state":["exists","up"]},{"osd":5,"uuid":"a9b0b4cf-1608-45ed-b64e-afcf545d97fa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6809","nonce":1092388797}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6811","nonce":1092388797}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6815","nonce":1092388797}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":1092388797},{"type":"v1","addr":"172.21.15.193:6813","nonce":1092388797}]},"public_addr":"172.21.15.193:6809/1092388797","cluster_addr":"172.21.15.193:6811/1092388797","heartbeat_back_addr":"172.21.15.193:6815/1092388797","heartbeat_front_addr":"172.21.15.193:6813/1092388797","state":["exists","up"]},{"osd":6,"uuid":"0064f2cf-a9d6-4fc1-8cc6-7eb1f07b0d04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6819","nonce":2471505438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6821","nonce":2471505438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6825","nonce":2471505438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":2471505438},{"type":"v1","addr":"172.21.15.134:6823","nonce":2471505438}]},"public_addr":"172.21.15.134:6819/2471505438","cluster_addr":"172.21.15.134:6821/2471505438","heartbeat_back_addr":"172.21.15.134:6825/2471505438","heartbeat_front_addr":"172.21.15.134:6823/2471505438","state":["exists","up"]},{"osd":7,"uuid":"394764a7-1976-4223-a3c0-5979d941a838","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6817","nonce":1382694911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6819","nonce":1382694911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6823","nonce":1382694911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":1382694911},{"type":"v1","addr":"172.21.15.189:6821","nonce":1382694911}]},"public_addr":"172.21.15.189:6817/1382694911","cluster_addr":"172.21.15.189:6819/1382694911","heartbeat_back_addr":"172.21.15.189:6823/1382694911","heartbeat_front_addr":"172.21.15.189:6821/1382694911","state":["exists","up"]},{"osd":8,"uuid":"1c218082-0b4c-41c8-92b9-453ae8d15a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6817","nonce":896348138}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6819","nonce":896348138}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6823","nonce":896348138}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":896348138},{"type":"v1","addr":"172.21.15.193:6821","nonce":896348138}]},"public_addr":"172.21.15.193:6817/896348138","cluster_addr":"172.21.15.193:6819/896348138","heartbeat_back_addr":"172.21.15.193:6823/896348138","heartbeat_front_addr":"172.21.15.193:6821/896348138","state":["exists","up"]},{"osd":9,"uuid":"48f39082-4fa9-4443-b5b7-b141c10d9e98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":62,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6827","nonce":511237025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6829","nonce":511237025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6833","nonce":511237025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":511237025},{"type":"v1","addr":"172.21.15.134:6831","nonce":511237025}]},"public_addr":"172.21.15.134:6827/511237025","cluster_addr":"172.21.15.134:6829/511237025","heartbeat_back_addr":"172.21.15.134:6833/511237025","heartbeat_front_addr":"172.21.15.134:6831/511237025","state":["exists","up"]},{"osd":10,"uuid":"95157766-1856-42f7-aea1-9cf83b77c8ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6825","nonce":431740858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6827","nonce":431740858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6831","nonce":431740858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":431740858},{"type":"v1","addr":"172.21.15.189:6829","nonce":431740858}]},"public_addr":"172.21.15.189:6825/431740858","cluster_addr":"172.21.15.189:6827/431740858","heartbeat_back_addr":"172.21.15.189:6831/431740858","heartbeat_front_addr":"172.21.15.189:6829/431740858","state":["exists","up"]},{"osd":11,"uuid":"244fadc7-f079-4736-8e7e-c0ce5ae6f43e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6825","nonce":1594590579}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6827","nonce":1594590579}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6831","nonce":1594590579}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":1594590579},{"type":"v1","addr":"172.21.15.193:6829","nonce":1594590579}]},"public_addr":"172.21.15.193:6825/1594590579","cluster_addr":"172.21.15.193:6827/1594590579","heartbeat_back_addr":"172.21.15.193:6831/1594590579","heartbeat_front_addr":"172.21.15.193:6829/1594590579","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:11.712203+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:35.845013+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:04:58.165609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:31.929746+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:05:56.941555+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:20.688071+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:06:54.681514+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:20.789384+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:07:44.568246+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:18.644310+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:08:45.365351+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-07-26T11:09:09.663805+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.5","mappings":[{"from":5,"to":8}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3790862248":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6800/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:0/2906927791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3068153391":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3056560791":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6800/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:6801/2424179007":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/606980428":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/1310359922":"2024-07-27T11:02:33.711793+0000","172.21.15.134:0/2318666906":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/2901579347":"2024-07-27T11:01:12.679951+0000","172.21.15.134:6801/3630314144":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3404218719":"2024-07-27T11:00:42.651939+0000","172.21.15.134:0/3598071145":"2024-07-27T11:02:33.711793+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-07-26T11:16:03.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.299+0000 7fc731600640 1 -- 172.21.15.134:0/1819566820 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc7100b3c20 msgr2=0x7fc7100b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:03.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.299+0000 7fc731600640 1 --2- 172.21.15.134:0/1819566820 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc7100b3c20 0x7fc7100b6130 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009a60 tx=0x7fc71c005d80 comp rx=0 tx=0).stop 2024-07-26T11:16:03.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.299+0000 7fc731600640 1 -- 172.21.15.134:0/1819566820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72c10f940 msgr2=0x7fc72c1a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:03.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.299+0000 7fc731600640 1 --2- 172.21.15.134:0/1819566820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72c10f940 0x7fc72c1a17b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc72400e970 tx=0x7fc72400ee30 comp rx=0 tx=0).stop 2024-07-26T11:16:03.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.299+0000 7fc731600640 1 -- 172.21.15.134:0/1819566820 shutdown_connections 2024-07-26T11:16:03.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.299+0000 7fc731600640 1 --2- 172.21.15.134:0/1819566820 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fc7100b3c20 0x7fc7100b6130 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:03.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.299+0000 7fc731600640 1 --2- 172.21.15.134:0/1819566820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72c10f940 0x7fc72c1a17b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:03.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.299+0000 7fc731600640 1 --2- 172.21.15.134:0/1819566820 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fc72c075b30 0x7fc72c19c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:03.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.299+0000 7fc731600640 1 --2- 172.21.15.134:0/1819566820 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc72c0751e0 0x7fc72c19bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:03.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.299+0000 7fc731600640 1 -- 172.21.15.134:0/1819566820 >> 172.21.15.134:0/1819566820 conn(0x7fc72c0fee70 msgr2=0x7fc72c109470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:03.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.299+0000 7fc731600640 1 -- 172.21.15.134:0/1819566820 shutdown_connections 2024-07-26T11:16:03.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:03.300+0000 7fc731600640 1 -- 172.21.15.134:0/1819566820 wait complete. 2024-07-26T11:16:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:03 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1819566820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:16:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:03 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1819566820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:16:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:03 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1819566820' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-07-26T11:16:04.306 DEBUG:tasks.fwd_scrub:Yielding 2024-07-26T11:16:04.307 INFO:teuthology.run_tasks:Running task exec... 2024-07-26T11:16:04.316 INFO:teuthology.task.exec:Executing custom commands... 2024-07-26T11:16:04.317 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi134.front.sepia.ceph.com 2024-07-26T11:16:04.317 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-07-26T11:16:04.317 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-07-26T11:16:04.317 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T11:16:04.318 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T11:16:04.384 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:04 smithi134 ceph-mon[31966]: pgmap v483: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T11:16:04.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.577+0000 7fb22a400640 1 -- 172.21.15.134:0/3538781001 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb224076dc0 msgr2=0x7fb2240751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:04.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.577+0000 7fb22a400640 1 --2- 172.21.15.134:0/3538781001 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb224076dc0 0x7fb2240751e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb214009900 tx=0x7fb21402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:16:04.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.577+0000 7fb22a400640 1 -- 172.21.15.134:0/3538781001 shutdown_connections 2024-07-26T11:16:04.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.577+0000 7fb22a400640 1 --2- 172.21.15.134:0/3538781001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb224113c00 0x7fb2241160d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:04.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.577+0000 7fb22a400640 1 --2- 172.21.15.134:0/3538781001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb224075800 0x7fb224075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:04.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.577+0000 7fb22a400640 1 --2- 172.21.15.134:0/3538781001 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb224076dc0 0x7fb2240751e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:04.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.577+0000 7fb22a400640 1 -- 172.21.15.134:0/3538781001 >> 172.21.15.134:0/3538781001 conn(0x7fb2240fed60 msgr2=0x7fb2241011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:04.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.577+0000 7fb22a400640 1 -- 172.21.15.134:0/3538781001 shutdown_connections 2024-07-26T11:16:04.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.577+0000 7fb22a400640 1 -- 172.21.15.134:0/3538781001 wait complete. 2024-07-26T11:16:04.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.578+0000 7fb22a400640 1 Processor -- start 2024-07-26T11:16:04.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.578+0000 7fb22a400640 1 -- start start 2024-07-26T11:16:04.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.578+0000 7fb22a400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb224075800 0x7fb2241a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:04.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.579+0000 7fb22a400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb224076dc0 0x7fb2241a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:04.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.579+0000 7fb22a400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb224113c00 0x7fb2241a5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:04.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.579+0000 7fb22a400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb224119890 con 0x7fb224076dc0 2024-07-26T11:16:04.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.579+0000 7fb22a400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb224119710 con 0x7fb224075800 2024-07-26T11:16:04.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.579+0000 7fb22a400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb224119a10 con 0x7fb224113c00 2024-07-26T11:16:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.579+0000 7fb229000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb224075800 0x7fb2241a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.579+0000 7fb229a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb224113c00 0x7fb2241a5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.579+0000 7fb229000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb224075800 0x7fb2241a02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:36260/0 (socket says 172.21.15.134:36260) 2024-07-26T11:16:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.579+0000 7fb21be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb224076dc0 0x7fb2241a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.579+0000 7fb229000640 1 -- 172.21.15.134:0/3040505945 learned_addr learned my addr 172.21.15.134:0/3040505945 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.579+0000 7fb229a00640 1 -- 172.21.15.134:0/3040505945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb224075800 msgr2=0x7fb2241a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.579+0000 7fb229a00640 1 --2- 172.21.15.134:0/3040505945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb224075800 0x7fb2241a02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.579+0000 7fb229a00640 1 -- 172.21.15.134:0/3040505945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb224076dc0 msgr2=0x7fb2241a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.579+0000 7fb229a00640 1 --2- 172.21.15.134:0/3040505945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb224076dc0 0x7fb2241a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.579+0000 7fb229a00640 1 -- 172.21.15.134:0/3040505945 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2241a6220 con 0x7fb224113c00 2024-07-26T11:16:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.580+0000 7fb229a00640 1 --2- 172.21.15.134:0/3040505945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb224113c00 0x7fb2241a5b20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00ec50 tx=0x7fb21c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.580+0000 7fb222000640 1 -- 172.21.15.134:0/3040505945 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c003b70 con 0x7fb224113c00 2024-07-26T11:16:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.580+0000 7fb22a400640 1 -- 172.21.15.134:0/3040505945 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2241a6510 con 0x7fb224113c00 2024-07-26T11:16:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.580+0000 7fb222000640 1 -- 172.21.15.134:0/3040505945 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb21c003d10 con 0x7fb224113c00 2024-07-26T11:16:04.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.580+0000 7fb222000640 1 -- 172.21.15.134:0/3040505945 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c010830 con 0x7fb224113c00 2024-07-26T11:16:04.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.580+0000 7fb22a400640 1 -- 172.21.15.134:0/3040505945 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2241acca0 con 0x7fb224113c00 2024-07-26T11:16:04.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.581+0000 7fb22a400640 1 -- 172.21.15.134:0/3040505945 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb224075c10 con 0x7fb224113c00 2024-07-26T11:16:04.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.584+0000 7fb222000640 1 -- 172.21.15.134:0/3040505945 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fb21c004b40 con 0x7fb224113c00 2024-07-26T11:16:04.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.585+0000 7fb222000640 1 --2- 172.21.15.134:0/3040505945 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb2080b3c20 0x7fb2080b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:04.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.585+0000 7fb222000640 1 -- 172.21.15.134:0/3040505945 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb21c0e4730 con 0x7fb224113c00 2024-07-26T11:16:04.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.585+0000 7fb229000640 1 --2- 172.21.15.134:0/3040505945 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb2080b3c20 0x7fb2080b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:04.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.586+0000 7fb229000640 1 --2- 172.21.15.134:0/3040505945 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb2080b3c20 0x7fb2080b6130 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb2140096c0 tx=0x7fb214005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:04.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.587+0000 7fb222000640 1 -- 172.21.15.134:0/3040505945 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fb21c0ab880 con 0x7fb224113c00 2024-07-26T11:16:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:04 smithi189 ceph-mon[36790]: pgmap v483: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T11:16:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:04 smithi193 ceph-mon[37473]: pgmap v483: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s 2024-07-26T11:16:04.778 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:16:04.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:04.860+0000 7fb22a400640 1 -- 172.21.15.134:0/3040505945 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fb224063230 con 0x7fb2080b3c20 2024-07-26T11:16:05.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.139+0000 7fb222000640 1 -- 172.21.15.134:0/3040505945 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb224063230 con 0x7fb2080b3c20 2024-07-26T11:16:05.141 INFO:teuthology.orchestra.run.smithi134.stdout:Schedule set for path / 2024-07-26T11:16:05.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.142+0000 7fb22a400640 1 -- 172.21.15.134:0/3040505945 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb2080b3c20 msgr2=0x7fb2080b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:05.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.142+0000 7fb22a400640 1 --2- 172.21.15.134:0/3040505945 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb2080b3c20 0x7fb2080b6130 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb2140096c0 tx=0x7fb214005a20 comp rx=0 tx=0).stop 2024-07-26T11:16:05.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.142+0000 7fb22a400640 1 -- 172.21.15.134:0/3040505945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb224113c00 msgr2=0x7fb2241a5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:05.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.142+0000 7fb22a400640 1 --2- 172.21.15.134:0/3040505945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb224113c00 0x7fb2241a5b20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00ec50 tx=0x7fb21c00c6a0 comp rx=0 tx=0).stop 2024-07-26T11:16:05.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.143+0000 7fb22a400640 1 -- 172.21.15.134:0/3040505945 shutdown_connections 2024-07-26T11:16:05.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.143+0000 7fb22a400640 1 --2- 172.21.15.134:0/3040505945 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb2080b3c20 0x7fb2080b6130 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:05.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.143+0000 7fb22a400640 1 --2- 172.21.15.134:0/3040505945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb224113c00 0x7fb2241a5b20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:05.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.143+0000 7fb22a400640 1 --2- 172.21.15.134:0/3040505945 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb224076dc0 0x7fb2241a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:05.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.143+0000 7fb22a400640 1 --2- 172.21.15.134:0/3040505945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb224075800 0x7fb2241a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:05.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.143+0000 7fb22a400640 1 -- 172.21.15.134:0/3040505945 >> 172.21.15.134:0/3040505945 conn(0x7fb2240fed60 msgr2=0x7fb22410fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:05.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.143+0000 7fb22a400640 1 -- 172.21.15.134:0/3040505945 shutdown_connections 2024-07-26T11:16:05.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.143+0000 7fb22a400640 1 -- 172.21.15.134:0/3040505945 wait complete. 2024-07-26T11:16:05.175 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-07-26T11:16:05.408 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:05 smithi134 ceph-mon[31966]: from='client.25498 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:16:05.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.481+0000 7fd36a600640 1 -- 172.21.15.134:0/3756936858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364101ac0 msgr2=0x7fd364101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:05.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.481+0000 7fd36a600640 1 --2- 172.21.15.134:0/3756936858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364101ac0 0x7fd364101ed0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd354009900 tx=0x7fd35402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:16:05.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.481+0000 7fd36a600640 1 -- 172.21.15.134:0/3756936858 shutdown_connections 2024-07-26T11:16:05.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.481+0000 7fd36a600640 1 --2- 172.21.15.134:0/3756936858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd364106580 0x7fd36410ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:05.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.481+0000 7fd36a600640 1 --2- 172.21.15.134:0/3756936858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd364105c30 0x7fd364106040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:05.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.481+0000 7fd36a600640 1 --2- 172.21.15.134:0/3756936858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364101ac0 0x7fd364101ed0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:05.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.481+0000 7fd36a600640 1 -- 172.21.15.134:0/3756936858 >> 172.21.15.134:0/3756936858 conn(0x7fd3640798f0 msgr2=0x7fd364079d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:05.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.481+0000 7fd36a600640 1 -- 172.21.15.134:0/3756936858 shutdown_connections 2024-07-26T11:16:05.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.481+0000 7fd36a600640 1 -- 172.21.15.134:0/3756936858 wait complete. 2024-07-26T11:16:05.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.482+0000 7fd36a600640 1 Processor -- start 2024-07-26T11:16:05.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.483+0000 7fd36a600640 1 -- start start 2024-07-26T11:16:05.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.483+0000 7fd36a600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd364101ac0 0x7fd36419c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:05.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.483+0000 7fd36a600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364105c30 0x7fd36419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:05.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.483+0000 7fd36a600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd364106580 0x7fd3641a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:05.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.483+0000 7fd36a600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd3641134b0 con 0x7fd364106580 2024-07-26T11:16:05.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.483+0000 7fd36a600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd364113330 con 0x7fd364101ac0 2024-07-26T11:16:05.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.483+0000 7fd36a600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd364113630 con 0x7fd364105c30 2024-07-26T11:16:05.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.483+0000 7fd363400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd364101ac0 0x7fd36419c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:05.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.483+0000 7fd363400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd364101ac0 0x7fd36419c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:36306/0 (socket says 172.21.15.134:36306) 2024-07-26T11:16:05.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.483+0000 7fd363400640 1 -- 172.21.15.134:0/3316305649 learned_addr learned my addr 172.21.15.134:0/3316305649 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:05.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.484+0000 7fd363e00640 1 --2- 172.21.15.134:0/3316305649 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd364106580 0x7fd3641a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:05.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.484+0000 7fd362a00640 1 --2- 172.21.15.134:0/3316305649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364105c30 0x7fd36419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:05.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.484+0000 7fd363400640 1 -- 172.21.15.134:0/3316305649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364105c30 msgr2=0x7fd36419c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:05.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.484+0000 7fd363400640 1 --2- 172.21.15.134:0/3316305649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364105c30 0x7fd36419c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:05.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.484+0000 7fd363400640 1 -- 172.21.15.134:0/3316305649 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd364106580 msgr2=0x7fd3641a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:05.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.484+0000 7fd363400640 1 --2- 172.21.15.134:0/3316305649 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd364106580 0x7fd3641a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:05.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.484+0000 7fd363400640 1 -- 172.21.15.134:0/3316305649 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3641a2040 con 0x7fd364101ac0 2024-07-26T11:16:05.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.484+0000 7fd363400640 1 --2- 172.21.15.134:0/3316305649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd364101ac0 0x7fd36419c110 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd354005ab0 tx=0x7fd354003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:05.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.484+0000 7fd35be00640 1 -- 172.21.15.134:0/3316305649 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd35403f070 con 0x7fd364101ac0 2024-07-26T11:16:05.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.485+0000 7fd35be00640 1 -- 172.21.15.134:0/3316305649 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd354031b60 con 0x7fd364101ac0 2024-07-26T11:16:05.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.485+0000 7fd36a600640 1 -- 172.21.15.134:0/3316305649 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3641a22d0 con 0x7fd364101ac0 2024-07-26T11:16:05.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.485+0000 7fd35be00640 1 -- 172.21.15.134:0/3316305649 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd354035030 con 0x7fd364101ac0 2024-07-26T11:16:05.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.485+0000 7fd36a600640 1 -- 172.21.15.134:0/3316305649 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3641a8ac0 con 0x7fd364101ac0 2024-07-26T11:16:05.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.486+0000 7fd36a600640 1 -- 172.21.15.134:0/3316305649 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd364101ed0 con 0x7fd364101ac0 2024-07-26T11:16:05.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.488+0000 7fd35be00640 1 -- 172.21.15.134:0/3316305649 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7fd354042070 con 0x7fd364101ac0 2024-07-26T11:16:05.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.489+0000 7fd35be00640 1 --2- 172.21.15.134:0/3316305649 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd3480b3c20 0x7fd3480b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:05.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.490+0000 7fd35be00640 1 -- 172.21.15.134:0/3316305649 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fd354104e60 con 0x7fd364101ac0 2024-07-26T11:16:05.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.490+0000 7fd362a00640 1 --2- 172.21.15.134:0/3316305649 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd3480b3c20 0x7fd3480b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:05.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.491+0000 7fd362a00640 1 --2- 172.21.15.134:0/3316305649 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd3480b3c20 0x7fd3480b6130 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd36407a260 tx=0x7fd350005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:05.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.491+0000 7fd35be00640 1 -- 172.21.15.134:0/3316305649 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fd3540cb960 con 0x7fd364101ac0 2024-07-26T11:16:05.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:05 smithi189 ceph-mon[36790]: from='client.25498 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:16:05.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.728+0000 7fd36a600640 1 -- 172.21.15.134:0/3316305649 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fd364107490 con 0x7fd3480b3c20 2024-07-26T11:16:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:05 smithi193 ceph-mon[37473]: from='client.25498 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:16:05.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.878+0000 7fd35be00640 1 -- 172.21.15.134:0/3316305649 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fd364107490 con 0x7fd3480b3c20 2024-07-26T11:16:05.881 INFO:teuthology.orchestra.run.smithi134.stdout:Retention added to path / 2024-07-26T11:16:05.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.882+0000 7fd36a600640 1 -- 172.21.15.134:0/3316305649 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd3480b3c20 msgr2=0x7fd3480b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:05.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.882+0000 7fd36a600640 1 --2- 172.21.15.134:0/3316305649 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd3480b3c20 0x7fd3480b6130 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd36407a260 tx=0x7fd350005f90 comp rx=0 tx=0).stop 2024-07-26T11:16:05.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.882+0000 7fd36a600640 1 -- 172.21.15.134:0/3316305649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd364101ac0 msgr2=0x7fd36419c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:05.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.882+0000 7fd36a600640 1 --2- 172.21.15.134:0/3316305649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd364101ac0 0x7fd36419c110 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd354005ab0 tx=0x7fd354003c10 comp rx=0 tx=0).stop 2024-07-26T11:16:05.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.882+0000 7fd36a600640 1 -- 172.21.15.134:0/3316305649 shutdown_connections 2024-07-26T11:16:05.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.882+0000 7fd36a600640 1 --2- 172.21.15.134:0/3316305649 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd3480b3c20 0x7fd3480b6130 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:05.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.883+0000 7fd36a600640 1 --2- 172.21.15.134:0/3316305649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd364105c30 0x7fd36419c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:05.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.883+0000 7fd36a600640 1 --2- 172.21.15.134:0/3316305649 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd364106580 0x7fd3641a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:05.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.883+0000 7fd36a600640 1 --2- 172.21.15.134:0/3316305649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd364101ac0 0x7fd36419c110 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:05.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.883+0000 7fd36a600640 1 -- 172.21.15.134:0/3316305649 >> 172.21.15.134:0/3316305649 conn(0x7fd3640798f0 msgr2=0x7fd364106ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:05.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.883+0000 7fd36a600640 1 -- 172.21.15.134:0/3316305649 shutdown_connections 2024-07-26T11:16:05.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:05.883+0000 7fd36a600640 1 -- 172.21.15.134:0/3316305649 wait complete. 2024-07-26T11:16:05.915 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-07-26T11:16:06.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.180+0000 7ffbd8a00640 1 -- 172.21.15.134:0/7578031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbd41095e0 msgr2=0x7ffbd410fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:06.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.180+0000 7ffbd8a00640 1 --2- 172.21.15.134:0/7578031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbd41095e0 0x7ffbd410fe10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc009be0 tx=0x7ffbcc02a6c0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.180+0000 7ffbd8a00640 1 -- 172.21.15.134:0/7578031 shutdown_connections 2024-07-26T11:16:06.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.180+0000 7ffbd8a00640 1 --2- 172.21.15.134:0/7578031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbd41095e0 0x7ffbd410fe10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.180+0000 7ffbd8a00640 1 --2- 172.21.15.134:0/7578031 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbd4108c90 0x7ffbd41090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.180+0000 7ffbd8a00640 1 --2- 172.21.15.134:0/7578031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbd4101760 0x7ffbd4101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.180+0000 7ffbd8a00640 1 -- 172.21.15.134:0/7578031 >> 172.21.15.134:0/7578031 conn(0x7ffbd4079530 msgr2=0x7ffbd4079980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:06.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.180+0000 7ffbd8a00640 1 -- 172.21.15.134:0/7578031 shutdown_connections 2024-07-26T11:16:06.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.180+0000 7ffbd8a00640 1 -- 172.21.15.134:0/7578031 wait complete. 2024-07-26T11:16:06.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.181+0000 7ffbd8a00640 1 Processor -- start 2024-07-26T11:16:06.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.181+0000 7ffbd8a00640 1 -- start start 2024-07-26T11:16:06.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.182+0000 7ffbd8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbd4101760 0x7ffbd419bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:06.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.182+0000 7ffbd8a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbd4108c90 0x7ffbd419c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:06.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.182+0000 7ffbd8a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbd41095e0 0x7ffbd41a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:06.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.182+0000 7ffbd8a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd41132b0 con 0x7ffbd41095e0 2024-07-26T11:16:06.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.182+0000 7ffbd8a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd4113130 con 0x7ffbd4101760 2024-07-26T11:16:06.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.182+0000 7ffbd8a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd4113430 con 0x7ffbd4108c90 2024-07-26T11:16:06.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.182+0000 7ffbd3200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbd41095e0 0x7ffbd41a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:06.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.182+0000 7ffbd3200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbd41095e0 0x7ffbd41a1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:34040/0 (socket says 172.21.15.134:34040) 2024-07-26T11:16:06.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.182+0000 7ffbd3200640 1 -- 172.21.15.134:0/1573009794 learned_addr learned my addr 172.21.15.134:0/1573009794 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:06.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.182+0000 7ffbd1e00640 1 --2- 172.21.15.134:0/1573009794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbd4108c90 0x7ffbd419c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:06.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.182+0000 7ffbd2800640 1 --2- 172.21.15.134:0/1573009794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbd4101760 0x7ffbd419bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:06.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.183+0000 7ffbd3200640 1 -- 172.21.15.134:0/1573009794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbd4108c90 msgr2=0x7ffbd419c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:06.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.183+0000 7ffbd3200640 1 --2- 172.21.15.134:0/1573009794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbd4108c90 0x7ffbd419c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.183+0000 7ffbd3200640 1 -- 172.21.15.134:0/1573009794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbd4101760 msgr2=0x7ffbd419bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:06.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.183+0000 7ffbd3200640 1 --2- 172.21.15.134:0/1573009794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbd4101760 0x7ffbd419bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.183+0000 7ffbd3200640 1 -- 172.21.15.134:0/1573009794 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd41a1e40 con 0x7ffbd41095e0 2024-07-26T11:16:06.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.183+0000 7ffbd2800640 1 --2- 172.21.15.134:0/1573009794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbd4101760 0x7ffbd419bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:16:06.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.183+0000 7ffbd3200640 1 --2- 172.21.15.134:0/1573009794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbd41095e0 0x7ffbd41a1740 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc009b30 tx=0x7ffbcc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:06.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.183+0000 7ffbcb400640 1 -- 172.21.15.134:0/1573009794 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc040020 con 0x7ffbd41095e0 2024-07-26T11:16:06.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.183+0000 7ffbd8a00640 1 -- 172.21.15.134:0/1573009794 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd41a20d0 con 0x7ffbd41095e0 2024-07-26T11:16:06.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.183+0000 7ffbcb400640 1 -- 172.21.15.134:0/1573009794 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffbcc0318f0 con 0x7ffbd41095e0 2024-07-26T11:16:06.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.183+0000 7ffbcb400640 1 -- 172.21.15.134:0/1573009794 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc036040 con 0x7ffbd41095e0 2024-07-26T11:16:06.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.183+0000 7ffbd8a00640 1 -- 172.21.15.134:0/1573009794 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd41a88c0 con 0x7ffbd41095e0 2024-07-26T11:16:06.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.185+0000 7ffbd8a00640 1 -- 172.21.15.134:0/1573009794 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbd4101b70 con 0x7ffbd41095e0 2024-07-26T11:16:06.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.187+0000 7ffbcb400640 1 -- 172.21.15.134:0/1573009794 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 23) ==== 151077+0+0 (secure 0 0 0) 0x7ffbcc043020 con 0x7ffbd41095e0 2024-07-26T11:16:06.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.188+0000 7ffbcb400640 1 --2- 172.21.15.134:0/1573009794 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ffbb80b3c20 0x7ffbb80b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:06.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.189+0000 7ffbcb400640 1 -- 172.21.15.134:0/1573009794 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ffbcc1051a0 con 0x7ffbd41095e0 2024-07-26T11:16:06.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.189+0000 7ffbd2800640 1 --2- 172.21.15.134:0/1573009794 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ffbb80b3c20 0x7ffbb80b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:06.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.189+0000 7ffbd2800640 1 --2- 172.21.15.134:0/1573009794 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ffbb80b3c20 0x7ffbb80b6130 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4009c20 tx=0x7ffbc4005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:06.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.190+0000 7ffbcb400640 1 -- 172.21.15.134:0/1573009794 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ffbcc04a020 con 0x7ffbd41095e0 2024-07-26T11:16:06.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.390+0000 7ffbcb400640 1 -- 172.21.15.134:0/1573009794 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7ffbcc0c7e80 con 0x7ffbd41095e0 2024-07-26T11:16:06.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.445+0000 7ffbd8a00640 1 -- 172.21.15.134:0/1573009794 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7ffbd4105130 con 0x7ffbb80b3c20 2024-07-26T11:16:06.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.456+0000 7ffbcb400640 1 -- 172.21.15.134:0/1573009794 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7ffbd4105130 con 0x7ffbb80b3c20 2024-07-26T11:16:06.458 INFO:teuthology.orchestra.run.smithi134.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-07-26T00:00:00", "created": "2024-07-26T11:16:04", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-07-26T11:16:06.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.459+0000 7ffbd8a00640 1 -- 172.21.15.134:0/1573009794 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ffbb80b3c20 msgr2=0x7ffbb80b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:06.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.459+0000 7ffbd8a00640 1 --2- 172.21.15.134:0/1573009794 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ffbb80b3c20 0x7ffbb80b6130 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ffbc4009c20 tx=0x7ffbc4005f90 comp rx=0 tx=0).stop 2024-07-26T11:16:06.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.459+0000 7ffbd8a00640 1 -- 172.21.15.134:0/1573009794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbd41095e0 msgr2=0x7ffbd41a1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:06.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.459+0000 7ffbd8a00640 1 --2- 172.21.15.134:0/1573009794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbd41095e0 0x7ffbd41a1740 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc009b30 tx=0x7ffbcc003c10 comp rx=0 tx=0).stop 2024-07-26T11:16:06.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.459+0000 7ffbd8a00640 1 -- 172.21.15.134:0/1573009794 shutdown_connections 2024-07-26T11:16:06.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.459+0000 7ffbd8a00640 1 --2- 172.21.15.134:0/1573009794 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ffbb80b3c20 0x7ffbb80b6130 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.459+0000 7ffbd8a00640 1 --2- 172.21.15.134:0/1573009794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffbd4108c90 0x7ffbd419c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.459+0000 7ffbd8a00640 1 --2- 172.21.15.134:0/1573009794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffbd41095e0 0x7ffbd41a1740 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.459+0000 7ffbd8a00640 1 --2- 172.21.15.134:0/1573009794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbd4101760 0x7ffbd419bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.459+0000 7ffbd8a00640 1 -- 172.21.15.134:0/1573009794 >> 172.21.15.134:0/1573009794 conn(0x7ffbd4079530 msgr2=0x7ffbd4104a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:06.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.459+0000 7ffbd8a00640 1 -- 172.21.15.134:0/1573009794 shutdown_connections 2024-07-26T11:16:06.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.459+0000 7ffbd8a00640 1 -- 172.21.15.134:0/1573009794 wait complete. 2024-07-26T11:16:06.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-07-26T11:16:06.543 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:06 smithi134 ceph-mon[31966]: from='client.25499 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:16:06.543 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:06 smithi134 ceph-mon[31966]: pgmap v484: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:16:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:06 smithi189 ceph-mon[36790]: from='client.25499 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:16:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:06 smithi189 ceph-mon[36790]: pgmap v484: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:16:06.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.731+0000 7f8528200640 1 -- 172.21.15.134:0/2427816705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8520108e30 msgr2=0x7f8520109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:06.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.731+0000 7f8528200640 1 --2- 172.21.15.134:0/2427816705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8520108e30 0x7f8520109240 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f850c009900 tx=0x7f850c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.731+0000 7f8528200640 1 -- 172.21.15.134:0/2427816705 shutdown_connections 2024-07-26T11:16:06.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.731+0000 7f8528200640 1 --2- 172.21.15.134:0/2427816705 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8520109780 0x7f852010ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.731+0000 7f8528200640 1 --2- 172.21.15.134:0/2427816705 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8520108e30 0x7f8520109240 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.731+0000 7f8528200640 1 --2- 172.21.15.134:0/2427816705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8520103500 0x7f8520103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.731+0000 7f8528200640 1 -- 172.21.15.134:0/2427816705 >> 172.21.15.134:0/2427816705 conn(0x7f85200fef40 msgr2=0x7f8520101390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:06.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.731+0000 7f8528200640 1 -- 172.21.15.134:0/2427816705 shutdown_connections 2024-07-26T11:16:06.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.731+0000 7f8528200640 1 -- 172.21.15.134:0/2427816705 wait complete. 2024-07-26T11:16:06.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.732+0000 7f8528200640 1 Processor -- start 2024-07-26T11:16:06.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.733+0000 7f8528200640 1 -- start start 2024-07-26T11:16:06.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.733+0000 7f8528200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8520103500 0x7f8520199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:06.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.733+0000 7f8528200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8520108e30 0x7f852019a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:06.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.733+0000 7f8528200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8520109780 0x7f852019f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:06.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.733+0000 7f8528200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f85201133f0 con 0x7f8520109780 2024-07-26T11:16:06.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.733+0000 7f8528200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8520113270 con 0x7f8520108e30 2024-07-26T11:16:06.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.733+0000 7f8528200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8520113570 con 0x7f8520103500 2024-07-26T11:16:06.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.733+0000 7f8526200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8520109780 0x7f852019f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:06.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.733+0000 7f8526200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8520109780 0x7f852019f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:34054/0 (socket says 172.21.15.134:34054) 2024-07-26T11:16:06.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.733+0000 7f8526200640 1 -- 172.21.15.134:0/3413613482 learned_addr learned my addr 172.21.15.134:0/3413613482 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:06.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.733+0000 7f8525800640 1 --2- 172.21.15.134:0/3413613482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8520103500 0x7f8520199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:06.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.733+0000 7f8524e00640 1 --2- 172.21.15.134:0/3413613482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8520108e30 0x7f852019a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:06.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.734+0000 7f8526200640 1 -- 172.21.15.134:0/3413613482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8520103500 msgr2=0x7f8520199e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:06.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.734+0000 7f8526200640 1 --2- 172.21.15.134:0/3413613482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8520103500 0x7f8520199e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.734+0000 7f8526200640 1 -- 172.21.15.134:0/3413613482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8520108e30 msgr2=0x7f852019a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:06.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.734+0000 7f8526200640 1 --2- 172.21.15.134:0/3413613482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8520108e30 0x7f852019a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.734+0000 7f8526200640 1 -- 172.21.15.134:0/3413613482 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f852019fd80 con 0x7f8520109780 2024-07-26T11:16:06.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.734+0000 7f8526200640 1 --2- 172.21.15.134:0/3413613482 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8520109780 0x7f852019f680 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f851800eb10 tx=0x7f851800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:06.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.734+0000 7f851e000640 1 -- 172.21.15.134:0/3413613482 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8518004b20 con 0x7f8520109780 2024-07-26T11:16:06.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.734+0000 7f851e000640 1 -- 172.21.15.134:0/3413613482 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8518004cc0 con 0x7f8520109780 2024-07-26T11:16:06.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.734+0000 7f8528200640 1 -- 172.21.15.134:0/3413613482 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85201a0070 con 0x7f8520109780 2024-07-26T11:16:06.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.734+0000 7f851e000640 1 -- 172.21.15.134:0/3413613482 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8518010670 con 0x7f8520109780 2024-07-26T11:16:06.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.734+0000 7f8528200640 1 -- 172.21.15.134:0/3413613482 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8520075020 con 0x7f8520109780 2024-07-26T11:16:06.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.736+0000 7f8528200640 1 -- 172.21.15.134:0/3413613482 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8520104770 con 0x7f8520109780 2024-07-26T11:16:06.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.740+0000 7f851e000640 1 -- 172.21.15.134:0/3413613482 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7f8518005000 con 0x7f8520109780 2024-07-26T11:16:06.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.741+0000 7f851e000640 1 --2- 172.21.15.134:0/3413613482 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f85080b3d30 0x7f85080b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:06.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.741+0000 7f851e000640 1 -- 172.21.15.134:0/3413613482 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f85180e4a50 con 0x7f8520109780 2024-07-26T11:16:06.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.741+0000 7f8525800640 1 --2- 172.21.15.134:0/3413613482 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f85080b3d30 0x7f85080b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:06.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.742+0000 7f8525800640 1 --2- 172.21.15.134:0/3413613482 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f85080b3d30 0x7f85080b6240 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8514009bd0 tx=0x7f8514005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:06.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.743+0000 7f851e000640 1 -- 172.21.15.134:0/3413613482 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8518055600 con 0x7f8520109780 2024-07-26T11:16:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:06 smithi193 ceph-mon[37473]: from='client.25499 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:16:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:06 smithi193 ceph-mon[37473]: pgmap v484: 129 pgs: 129 active+clean; 655 KiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T11:16:06.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.976+0000 7f8528200640 1 -- 172.21.15.134:0/3413613482 --> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f8520063230 con 0x7f85080b3d30 2024-07-26T11:16:06.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.988+0000 7f851e000640 1 -- 172.21.15.134:0/3413613482 <== mgr.14211 v2:172.21.15.134:6800/2671962751 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f8520063230 con 0x7f85080b3d30 2024-07-26T11:16:06.991 INFO:teuthology.orchestra.run.smithi134.stdout:/ 1m 6m3h 2024-07-26T11:16:06.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.991+0000 7f8528200640 1 -- 172.21.15.134:0/3413613482 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f85080b3d30 msgr2=0x7f85080b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:06.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.991+0000 7f8528200640 1 --2- 172.21.15.134:0/3413613482 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f85080b3d30 0x7f85080b6240 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8514009bd0 tx=0x7f8514005f90 comp rx=0 tx=0).stop 2024-07-26T11:16:06.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.991+0000 7f8528200640 1 -- 172.21.15.134:0/3413613482 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8520109780 msgr2=0x7f852019f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:06.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.991+0000 7f8528200640 1 --2- 172.21.15.134:0/3413613482 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8520109780 0x7f852019f680 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f851800eb10 tx=0x7f851800c2d0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.992+0000 7f8528200640 1 -- 172.21.15.134:0/3413613482 shutdown_connections 2024-07-26T11:16:06.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.992+0000 7f8528200640 1 --2- 172.21.15.134:0/3413613482 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f85080b3d30 0x7f85080b6240 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.992+0000 7f8528200640 1 --2- 172.21.15.134:0/3413613482 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8520109780 0x7f852019f680 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.992+0000 7f8528200640 1 --2- 172.21.15.134:0/3413613482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8520108e30 0x7f852019a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.992+0000 7f8528200640 1 --2- 172.21.15.134:0/3413613482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8520103500 0x7f8520199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:06.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.992+0000 7f8528200640 1 -- 172.21.15.134:0/3413613482 >> 172.21.15.134:0/3413613482 conn(0x7f85200fef40 msgr2=0x7f8520101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:06.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.992+0000 7f8528200640 1 -- 172.21.15.134:0/3413613482 shutdown_connections 2024-07-26T11:16:06.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:06.992+0000 7f8528200640 1 -- 172.21.15.134:0/3413613482 wait complete. 2024-07-26T11:16:07.019 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-07-26T11:16:07.087 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-07-26T11:16:07.098 INFO:teuthology.run_tasks:Running task background_exec... 2024-07-26T11:16:07.128 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi134.front.sepia.ceph.com 2024-07-26T11:16:07.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-07-26T11:16:07.133 INFO:teuthology.run_tasks:Running task workunit... 2024-07-26T11:16:07.172 INFO:tasks.workunit:Pulling workunits from ref e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T11:16:07.172 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-07-26T11:16:07.172 INFO:tasks.workunit:timeout=3h 2024-07-26T11:16:07.172 INFO:tasks.workunit:cleanup=True 2024-07-26T11:16:07.173 DEBUG:teuthology.orchestra.run.smithi134:> stat -- /home/ubuntu/cephtest/mnt.0 2024-07-26T11:16:07.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.153+0000 7fcc32a00640 1 -- 172.21.15.134:0/1358528993 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c109840 msgr2=0x7fcc2c110070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:07.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.153+0000 7fcc32a00640 1 --2- 172.21.15.134:0/1358528993 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c109840 0x7fcc2c110070 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcc24009ca0 tx=0x7fcc2402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.153+0000 7fcc32a00640 1 -- 172.21.15.134:0/1358528993 shutdown_connections 2024-07-26T11:16:07.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.153+0000 7fcc32a00640 1 --2- 172.21.15.134:0/1358528993 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c109840 0x7fcc2c110070 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.153+0000 7fcc32a00640 1 --2- 172.21.15.134:0/1358528993 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc2c108ef0 0x7fcc2c109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.153+0000 7fcc32a00640 1 --2- 172.21.15.134:0/1358528993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc2c1035b0 0x7fcc2c1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.153+0000 7fcc32a00640 1 -- 172.21.15.134:0/1358528993 >> 172.21.15.134:0/1358528993 conn(0x7fcc2c0fefc0 msgr2=0x7fcc2c101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:07.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.153+0000 7fcc32a00640 1 -- 172.21.15.134:0/1358528993 shutdown_connections 2024-07-26T11:16:07.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.153+0000 7fcc32a00640 1 -- 172.21.15.134:0/1358528993 wait complete. 2024-07-26T11:16:07.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.154+0000 7fcc32a00640 1 Processor -- start 2024-07-26T11:16:07.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.154+0000 7fcc32a00640 1 -- start start 2024-07-26T11:16:07.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.155+0000 7fcc32a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c1035b0 0x7fcc2c19c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.155+0000 7fcc32a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc2c108ef0 0x7fcc2c19c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.155+0000 7fcc32a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc2c109840 0x7fcc2c1a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.155+0000 7fcc32a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c113540 con 0x7fcc2c109840 2024-07-26T11:16:07.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.155+0000 7fcc32a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c1133c0 con 0x7fcc2c108ef0 2024-07-26T11:16:07.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.155+0000 7fcc32a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c1136c0 con 0x7fcc2c1035b0 2024-07-26T11:16:07.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.155+0000 7fcc23e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc2c108ef0 0x7fcc2c19c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.155+0000 7fcc2be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c1035b0 0x7fcc2c19c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.155+0000 7fcc23e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc2c108ef0 0x7fcc2c19c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:36350/0 (socket says 172.21.15.134:36350) 2024-07-26T11:16:07.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.155+0000 7fcc23e00640 1 -- 172.21.15.134:0/2332607166 learned_addr learned my addr 172.21.15.134:0/2332607166 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:07.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.155+0000 7fcc30a00640 1 --2- 172.21.15.134:0/2332607166 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc2c109840 0x7fcc2c1a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.156+0000 7fcc23e00640 1 -- 172.21.15.134:0/2332607166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c1035b0 msgr2=0x7fcc2c19c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:07.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.156+0000 7fcc23e00640 1 --2- 172.21.15.134:0/2332607166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c1035b0 0x7fcc2c19c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.156+0000 7fcc23e00640 1 -- 172.21.15.134:0/2332607166 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc2c109840 msgr2=0x7fcc2c1a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:07.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.156+0000 7fcc23e00640 1 --2- 172.21.15.134:0/2332607166 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc2c109840 0x7fcc2c1a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.156+0000 7fcc23e00640 1 -- 172.21.15.134:0/2332607166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2c1a1fe0 con 0x7fcc2c108ef0 2024-07-26T11:16:07.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.156+0000 7fcc23e00640 1 --2- 172.21.15.134:0/2332607166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc2c108ef0 0x7fcc2c19c610 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400e9f0 tx=0x7fcc1400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:07.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.156+0000 7fcc29600640 1 -- 172.21.15.134:0/2332607166 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1400ccc0 con 0x7fcc2c108ef0 2024-07-26T11:16:07.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.156+0000 7fcc29600640 1 -- 172.21.15.134:0/2332607166 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc140108e0 con 0x7fcc2c108ef0 2024-07-26T11:16:07.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.156+0000 7fcc29600640 1 -- 172.21.15.134:0/2332607166 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14018860 con 0x7fcc2c108ef0 2024-07-26T11:16:07.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.156+0000 7fcc32a00640 1 -- 172.21.15.134:0/2332607166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc2c1a22d0 con 0x7fcc2c108ef0 2024-07-26T11:16:07.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.157+0000 7fcc32a00640 1 -- 172.21.15.134:0/2332607166 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc2c1a8a30 con 0x7fcc2c108ef0 2024-07-26T11:16:07.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.160+0000 7fcc397c8740 1 Processor -- start 2024-07-26T11:16:07.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.160+0000 7fcc397c8740 1 -- start start 2024-07-26T11:16:07.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.160+0000 7fcc397c8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56519601e350 0x5651963c2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.160+0000 7fcc397c8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963c32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.160+0000 7fcc397c8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565195ffd670 0x5651963c3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.160+0000 7fcc29600640 1 -- 172.21.15.134:0/2332607166 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7fcc14010430 con 0x7fcc2c108ef0 2024-07-26T11:16:07.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.160+0000 7fcc397c8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x565196339bf0 con 0x565195ffd670 2024-07-26T11:16:07.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.160+0000 7fcc397c8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5651963290f0 con 0x565195ff5490 2024-07-26T11:16:07.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.160+0000 7fcc397c8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56519632d130 con 0x56519601e350 2024-07-26T11:16:07.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.161+0000 7fcc30a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56519601e350 0x5651963c2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.161+0000 7fcc30a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56519601e350 0x5651963c2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:36428/0 (socket says 172.21.15.134:36428) 2024-07-26T11:16:07.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.161+0000 7fcc30a00640 1 -- 172.21.15.134:0/1265501846 learned_addr learned my addr 172.21.15.134:0/1265501846 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:07.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.161+0000 7fcc2be00640 1 --2- 172.21.15.134:0/1265501846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963c32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.161+0000 7fcc23e00640 1 --2- 172.21.15.134:0/1265501846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565195ffd670 0x5651963c3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.161+0000 7fcc30a00640 1 -- 172.21.15.134:0/1265501846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 msgr2=0x5651963c32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:07.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.161+0000 7fcc30a00640 1 --2- 172.21.15.134:0/1265501846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963c32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.161+0000 7fcc30a00640 1 -- 172.21.15.134:0/1265501846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565195ffd670 msgr2=0x5651963c3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:07.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.161+0000 7fcc30a00640 1 --2- 172.21.15.134:0/1265501846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565195ffd670 0x5651963c3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.161+0000 7fcc30a00640 1 -- 172.21.15.134:0/1265501846 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565196089930 con 0x56519601e350 2024-07-26T11:16:07.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.162+0000 7fcc30a00640 1 --2- 172.21.15.134:0/1265501846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56519601e350 0x5651963c2d90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcc24009ca0 tx=0x7fcc24009430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=447a5c7be8b33b09 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:07.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.162+0000 7fcc21200640 1 -- 172.21.15.134:0/1265501846 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24031cf0 con 0x56519601e350 2024-07-26T11:16:07.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.162+0000 7fcc21200640 1 -- 172.21.15.134:0/1265501846 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc24036030 con 0x56519601e350 2024-07-26T11:16:07.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.162+0000 7fcc21200640 1 -- 172.21.15.134:0/1265501846 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc240042a0 con 0x56519601e350 2024-07-26T11:16:07.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.163+0000 7fcc29600640 1 --2- 172.21.15.134:0/2332607166 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc100b3d30 0x7fcc100b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.163+0000 7fcc29600640 1 -- 172.21.15.134:0/2332607166 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fcc140e44f0 con 0x7fcc2c108ef0 2024-07-26T11:16:07.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.163+0000 7fcc2be00640 1 --2- 172.21.15.134:0/2332607166 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc100b3d30 0x7fcc100b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.163+0000 7fcc397c8740 1 -- 172.21.15.134:0/1265501846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56519601e350 msgr2=0x5651963c2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:07.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.163+0000 7fcc397c8740 1 --2- 172.21.15.134:0/1265501846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56519601e350 0x5651963c2d90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcc24009ca0 tx=0x7fcc24009430 comp rx=0 tx=0).stop 2024-07-26T11:16:07.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.163+0000 7fcc397c8740 1 -- 172.21.15.134:0/1265501846 shutdown_connections 2024-07-26T11:16:07.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.163+0000 7fcc397c8740 1 --2- 172.21.15.134:0/1265501846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x565195ffd670 0x5651963c3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.163+0000 7fcc397c8740 1 --2- 172.21.15.134:0/1265501846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963c32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.163+0000 7fcc397c8740 1 --2- 172.21.15.134:0/1265501846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56519601e350 0x5651963c2d90 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.163+0000 7fcc397c8740 1 -- 172.21.15.134:0/1265501846 >> 172.21.15.134:0/1265501846 conn(0x565196012740 msgr2=0x5651963c0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:07.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.163+0000 7fcc397c8740 1 -- 172.21.15.134:0/1265501846 shutdown_connections 2024-07-26T11:16:07.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.164+0000 7fcc397c8740 1 -- 172.21.15.134:0/1265501846 wait complete. 2024-07-26T11:16:07.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.164+0000 7fcc2be00640 1 --2- 172.21.15.134:0/2332607166 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc100b3d30 0x7fcc100b6240 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x5651963c1a10 tx=0x7fcc1c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:07.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.164+0000 7fcc397c8740 1 -- start start 2024-07-26T11:16:07.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.164+0000 7fcc397c8740 1 Processor -- start 2024-07-26T11:16:07.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.165+0000 7fcc397c8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963ce450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.165+0000 7fcc397c8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.165+0000 7fcc397c8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 0x5651963da6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.165+0000 7fcc397c8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56519618c890 con 0x56519601e350 2024-07-26T11:16:07.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.165+0000 7fcc397c8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5651962c5330 con 0x565195ff5490 2024-07-26T11:16:07.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.165+0000 7fcc397c8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5651963d1390 con 0x565195ffd670 2024-07-26T11:16:07.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.165+0000 7fcc397c8740 10 client.0 _ll_register_callbacks cb 0x5651962f1010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:16:07.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.165+0000 7fcc30a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 0x5651963da6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.165+0000 7fcc2be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.165+0000 7fcc30a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 0x5651963da6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:34088/0 (socket says 172.21.15.134:34088) 2024-07-26T11:16:07.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.165+0000 7fcc30a00640 1 -- 172.21.15.134:0/1184567846 learned_addr learned my addr 172.21.15.134:0/1184567846 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:07.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.165+0000 7fcc23e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.166+0000 7fcc23e00640 1 -- 172.21.15.134:0/1184567846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 msgr2=0x5651963ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:07.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.166+0000 7fcc23e00640 1 --2- 172.21.15.134:0/1184567846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963ce450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.166+0000 7fcc23e00640 1 -- 172.21.15.134:0/1184567846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 msgr2=0x5651963da6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:07.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.166+0000 7fcc23e00640 1 --2- 172.21.15.134:0/1184567846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 0x5651963da6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.166+0000 7fcc23e00640 1 -- 172.21.15.134:0/1184567846 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651963cf1e0 con 0x565195ffd670 2024-07-26T11:16:07.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.166+0000 7fcc2be00640 1 --2- 172.21.15.134:0/1184567846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:07.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.166+0000 7fcc30a00640 1 --2- 172.21.15.134:0/1184567846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 0x5651963da6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:07.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.166+0000 7fcc23e00640 1 --2- 172.21.15.134:0/1184567846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963ce990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcc140e19a0 tx=0x7fcc14009740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e5663b9ba01c502 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:07.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.166+0000 7fcc19600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:16:07.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.166+0000 7fcc19600640 1 -- 172.21.15.134:0/1184567846 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14010720 con 0x565195ffd670 2024-07-26T11:16:07.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.166+0000 7fcc19600640 1 -- 172.21.15.134:0/1184567846 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fcc140e1bf0 con 0x565195ffd670 2024-07-26T11:16:07.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.166+0000 7fcc397c8740 1 -- 172.21.15.134:0/1184567846 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc24036030 con 0x565195ffd670 2024-07-26T11:16:07.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.167+0000 7fcc19600640 1 -- 172.21.15.134:0/1184567846 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcc140aa9a0 con 0x565195ffd670 2024-07-26T11:16:07.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.167+0000 7fcc397c8740 10 client.25534 fetch_fsmap learned FSMap version 30 2024-07-26T11:16:07.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.167+0000 7fcc397c8740 1 -- 172.21.15.134:0/1184567846 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5651963dd340 con 0x565195ffd670 2024-07-26T11:16:07.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.167+0000 7fcc19600640 1 -- 172.21.15.134:0/1184567846 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc24036030 con 0x565195ffd670 2024-07-26T11:16:07.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.167+0000 7fcc19600640 1 -- 172.21.15.134:0/1184567846 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fcc140a6800 con 0x565195ffd670 2024-07-26T11:16:07.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.167+0000 7fcc397c8740 10 client.25534 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:16:07.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.167+0000 7fcc397c8740 10 client.25534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:16:07.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.167+0000 7fcc397c8740 20 client.25534 populate_metadata read hostname 'smithi134' 2024-07-26T11:16:07.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.167+0000 7fcc397c8740 1 --2- 172.21.15.134:0/1184567846 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x565195fdccd0 0x5651963e0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.167+0000 7fcc397c8740 4 client.25534 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:16:07.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.167+0000 7fcc397c8740 1 -- 172.21.15.134:0/1184567846 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56519633bb40 con 0x565195fdccd0 2024-07-26T11:16:07.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.168+0000 7fcc2be00640 1 --2- 172.21.15.134:0/1184567846 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x565195fdccd0 0x5651963e0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.168+0000 7fcc2be00640 1 --2- 172.21.15.134:0/1184567846 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x565195fdccd0 0x5651963e0860 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ce70185308f3765 server_cookie=5c84958416d76b4d in_seq=0 out_seq=0 2024-07-26T11:16:07.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.168+0000 7fcc19600640 10 client.25534 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:07.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.169+0000 7fcc19600640 1 -- 172.21.15.134:0/1184567846 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56519633bb40 con 0x565195fdccd0 2024-07-26T11:16:07.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.169+0000 7fcc19600640 10 client.25534 handle_command_reply: tid=0 2024-07-26T11:16:07.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.169+0000 7fcc397c8740 1 client.25534 shutdown 2024-07-26T11:16:07.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.169+0000 7fcc397c8740 1 -- 172.21.15.134:0/1184567846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 msgr2=0x5651963ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:07.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.169+0000 7fcc397c8740 1 --2- 172.21.15.134:0/1184567846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963ce990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcc140e19a0 tx=0x7fcc14009740 comp rx=0 tx=0).stop 2024-07-26T11:16:07.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.170+0000 7fcc397c8740 1 -- 172.21.15.134:0/1184567846 shutdown_connections 2024-07-26T11:16:07.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.170+0000 7fcc397c8740 1 --2- 172.21.15.134:0/1184567846 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x565195fdccd0 0x5651963e0860 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.170+0000 7fcc397c8740 1 --2- 172.21.15.134:0/1184567846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963ce990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.170+0000 7fcc397c8740 1 --2- 172.21.15.134:0/1184567846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 0x5651963da6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.170+0000 7fcc397c8740 1 --2- 172.21.15.134:0/1184567846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.170+0000 7fcc397c8740 1 -- 172.21.15.134:0/1184567846 >> 172.21.15.134:0/1184567846 conn(0x565196012740 msgr2=0x5651963c5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:07.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.170+0000 7fcc19600640 0 client.25534 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:07.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.170+0000 7fcc397c8740 1 -- 172.21.15.134:0/1184567846 shutdown_connections 2024-07-26T11:16:07.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.170+0000 7fcc397c8740 1 -- 172.21.15.134:0/1184567846 wait complete. 2024-07-26T11:16:07.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.170+0000 7fcc397c8740 20 client.25534 trim_cache size 0 max 16384 2024-07-26T11:16:07.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.199+0000 7fcc397c8740 1 Processor -- start 2024-07-26T11:16:07.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.199+0000 7fcc397c8740 1 -- start start 2024-07-26T11:16:07.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.199+0000 7fcc397c8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.199+0000 7fcc397c8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963d6640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.199+0000 7fcc397c8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 0x5651963d6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.199+0000 7fcc397c8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c1b2a80 con 0x56519601e350 2024-07-26T11:16:07.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.199+0000 7fcc397c8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c1b45c0 con 0x565195ff5490 2024-07-26T11:16:07.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.199+0000 7fcc397c8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c1b3150 con 0x565195ffd670 2024-07-26T11:16:07.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.200+0000 7fcc23e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963d6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.200+0000 7fcc23e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963d6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:36446/0 (socket says 172.21.15.134:36446) 2024-07-26T11:16:07.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.200+0000 7fcc2be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.200+0000 7fcc23e00640 1 -- 172.21.15.134:0/393206803 learned_addr learned my addr 172.21.15.134:0/393206803 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:07.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.200+0000 7fcc30a00640 1 --2- 172.21.15.134:0/393206803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 0x5651963d6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.200+0000 7fcc23e00640 1 -- 172.21.15.134:0/393206803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 msgr2=0x5651963cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:07.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.200+0000 7fcc23e00640 1 --2- 172.21.15.134:0/393206803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963cdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.200+0000 7fcc23e00640 1 -- 172.21.15.134:0/393206803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 msgr2=0x5651963d6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:07.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.200+0000 7fcc23e00640 1 --2- 172.21.15.134:0/393206803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 0x5651963d6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.200+0000 7fcc23e00640 1 -- 172.21.15.134:0/393206803 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565195fdcf00 con 0x565195ffd670 2024-07-26T11:16:07.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.201+0000 7fcc23e00640 1 --2- 172.21.15.134:0/393206803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963d6640 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcc140aae20 tx=0x7fcc1400cef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c73647f45c9b0c9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:07.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.201+0000 7fcc1be00640 1 -- 172.21.15.134:0/393206803 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc14010720 con 0x565195ffd670 2024-07-26T11:16:07.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.201+0000 7fcc1be00640 1 -- 172.21.15.134:0/393206803 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc140b3e90 con 0x565195ffd670 2024-07-26T11:16:07.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.201+0000 7fcc1be00640 1 -- 172.21.15.134:0/393206803 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc140a9b60 con 0x565195ffd670 2024-07-26T11:16:07.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.202+0000 7fcc397c8740 1 -- 172.21.15.134:0/393206803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 msgr2=0x5651963d6640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:07.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.202+0000 7fcc397c8740 1 --2- 172.21.15.134:0/393206803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963d6640 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcc140aae20 tx=0x7fcc1400cef0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.202+0000 7fcc397c8740 1 -- 172.21.15.134:0/393206803 shutdown_connections 2024-07-26T11:16:07.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.202+0000 7fcc397c8740 1 --2- 172.21.15.134:0/393206803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963d6640 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.202+0000 7fcc397c8740 1 --2- 172.21.15.134:0/393206803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 0x5651963d6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.202+0000 7fcc397c8740 1 --2- 172.21.15.134:0/393206803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.202+0000 7fcc397c8740 1 -- 172.21.15.134:0/393206803 >> 172.21.15.134:0/393206803 conn(0x565196012740 msgr2=0x5651963bfc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:07.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.203+0000 7fcc397c8740 1 -- 172.21.15.134:0/393206803 shutdown_connections 2024-07-26T11:16:07.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.203+0000 7fcc397c8740 1 -- 172.21.15.134:0/393206803 wait complete. 2024-07-26T11:16:07.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.203+0000 7fcc397c8740 1 -- start start 2024-07-26T11:16:07.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.204+0000 7fcc397c8740 1 Processor -- start 2024-07-26T11:16:07.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.204+0000 7fcc397c8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963d64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.204+0000 7fcc397c8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963d6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.204+0000 7fcc397c8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 0x5651964415c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.204+0000 7fcc397c8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c1b3ef0 con 0x56519601e350 2024-07-26T11:16:07.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.204+0000 7fcc397c8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c1b3820 con 0x565195ff5490 2024-07-26T11:16:07.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.204+0000 7fcc397c8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2c1b0a60 con 0x565195ffd670 2024-07-26T11:16:07.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.204+0000 7fcc397c8740 10 client.0 _ll_register_callbacks cb 0x7fcc140e7c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:16:07.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.205+0000 7fcc23e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963d6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.205+0000 7fcc2be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963d64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.205+0000 7fcc23e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963d6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:36454/0 (socket says 172.21.15.134:36454) 2024-07-26T11:16:07.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.205+0000 7fcc2be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963d64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:36394/0 (socket says 172.21.15.134:36394) 2024-07-26T11:16:07.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.205+0000 7fcc23e00640 1 -- 172.21.15.134:0/3203228385 learned_addr learned my addr 172.21.15.134:0/3203228385 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:07.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.205+0000 7fcc30a00640 1 --2- 172.21.15.134:0/3203228385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 0x5651964415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.205+0000 7fcc2be00640 1 -- 172.21.15.134:0/3203228385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 msgr2=0x5651963d6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:07.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.205+0000 7fcc2be00640 1 --2- 172.21.15.134:0/3203228385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963d6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.205+0000 7fcc2be00640 1 -- 172.21.15.134:0/3203228385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 msgr2=0x5651964415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:07.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.205+0000 7fcc2be00640 1 --2- 172.21.15.134:0/3203228385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 0x5651964415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.205+0000 7fcc2be00640 1 -- 172.21.15.134:0/3203228385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565196441c30 con 0x565195ff5490 2024-07-26T11:16:07.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.205+0000 7fcc23e00640 1 --2- 172.21.15.134:0/3203228385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963d6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:07.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.206+0000 7fcc2be00640 1 --2- 172.21.15.134:0/3203228385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963d64d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x5651963e0fa0 tx=0x7fcc1c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c55e2bdc3dc3aa5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:07.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.206+0000 7fcc19600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:16:07.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.206+0000 7fcc19600640 1 -- 172.21.15.134:0/3203228385 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c030030 con 0x565195ff5490 2024-07-26T11:16:07.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.206+0000 7fcc19600640 1 -- 172.21.15.134:0/3203228385 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fcc1c018030 con 0x565195ff5490 2024-07-26T11:16:07.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.206+0000 7fcc397c8740 1 -- 172.21.15.134:0/3203228385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5651963375e0 con 0x565195ff5490 2024-07-26T11:16:07.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.206+0000 7fcc19600640 1 -- 172.21.15.134:0/3203228385 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcc1c03d050 con 0x565195ff5490 2024-07-26T11:16:07.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.207+0000 7fcc397c8740 10 client.25529 fetch_fsmap learned FSMap version 30 2024-07-26T11:16:07.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.207+0000 7fcc397c8740 1 -- 172.21.15.134:0/3203228385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x565196449110 con 0x565195ff5490 2024-07-26T11:16:07.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.207+0000 7fcc19600640 1 -- 172.21.15.134:0/3203228385 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5651963375e0 con 0x565195ff5490 2024-07-26T11:16:07.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.207+0000 7fcc19600640 1 -- 172.21.15.134:0/3203228385 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fcc1c01f430 con 0x565195ff5490 2024-07-26T11:16:07.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.207+0000 7fcc397c8740 10 client.25529 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:16:07.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.207+0000 7fcc397c8740 10 client.25529 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:16:07.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.207+0000 7fcc397c8740 20 client.25529 populate_metadata read hostname 'smithi134' 2024-07-26T11:16:07.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.207+0000 7fcc397c8740 1 --2- 172.21.15.134:0/3203228385 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x565195fb2080 0x56519644b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:07.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.207+0000 7fcc397c8740 4 client.25529 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T11:16:07.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.207+0000 7fcc397c8740 1 -- 172.21.15.134:0/3203228385 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x565196334d20 con 0x565195fb2080 2024-07-26T11:16:07.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.208+0000 7fcc23e00640 1 --2- 172.21.15.134:0/3203228385 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x565195fb2080 0x56519644b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:07.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.208+0000 7fcc23e00640 1 --2- 172.21.15.134:0/3203228385 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x565195fb2080 0x56519644b900 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1703fd9531eb26fa server_cookie=59e6cfa198f94da5 in_seq=0 out_seq=0 2024-07-26T11:16:07.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.208+0000 7fcc19600640 10 client.25529 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:07.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.210+0000 7fcc19600640 1 -- 172.21.15.134:0/3203228385 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x565196334d20 con 0x565195fb2080 2024-07-26T11:16:07.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.210+0000 7fcc19600640 10 client.25529 handle_command_reply: tid=0 2024-07-26T11:16:07.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.210+0000 7fcc397c8740 1 client.25529 shutdown 2024-07-26T11:16:07.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.211+0000 7fcc397c8740 1 -- 172.21.15.134:0/3203228385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 msgr2=0x5651963d64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:07.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.211+0000 7fcc397c8740 1 --2- 172.21.15.134:0/3203228385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963d64d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x5651963e0fa0 tx=0x7fcc1c0046c0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.211+0000 7fcc397c8740 1 -- 172.21.15.134:0/3203228385 shutdown_connections 2024-07-26T11:16:07.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.211+0000 7fcc397c8740 1 --2- 172.21.15.134:0/3203228385 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x565195fb2080 0x56519644b900 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.211+0000 7fcc397c8740 1 --2- 172.21.15.134:0/3203228385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x565195ffd670 0x5651963d6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.211+0000 7fcc397c8740 1 --2- 172.21.15.134:0/3203228385 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56519601e350 0x5651964415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.211+0000 7fcc397c8740 1 --2- 172.21.15.134:0/3203228385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565195ff5490 0x5651963d64d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.211+0000 7fcc397c8740 1 -- 172.21.15.134:0/3203228385 >> 172.21.15.134:0/3203228385 conn(0x565196012740 msgr2=0x5651963cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:07.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.211+0000 7fcc19600640 0 client.25529 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:07.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.211+0000 7fcc397c8740 1 -- 172.21.15.134:0/3203228385 shutdown_connections 2024-07-26T11:16:07.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.211+0000 7fcc397c8740 1 -- 172.21.15.134:0/3203228385 wait complete. 2024-07-26T11:16:07.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.212+0000 7fcc397c8740 20 client.25529 trim_cache size 0 max 16384 2024-07-26T11:16:07.215 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:16:07.215 INFO:teuthology.orchestra.run.smithi134.stdout: "return_code": 0, 2024-07-26T11:16:07.215 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_tag": "b163b36d-b837-4f6c-a892-2000c8eb64bc", 2024-07-26T11:16:07.215 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "asynchronous" 2024-07-26T11:16:07.215 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:16:07.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.213+0000 7fcc32a00640 1 -- 172.21.15.134:0/2332607166 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc100b3d30 msgr2=0x7fcc100b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:07.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.213+0000 7fcc32a00640 1 --2- 172.21.15.134:0/2332607166 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc100b3d30 0x7fcc100b6240 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x5651963c1a10 tx=0x7fcc1c005f90 comp rx=0 tx=0).stop 2024-07-26T11:16:07.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.213+0000 7fcc32a00640 1 -- 172.21.15.134:0/2332607166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc2c108ef0 msgr2=0x7fcc2c19c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:07.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.213+0000 7fcc32a00640 1 --2- 172.21.15.134:0/2332607166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc2c108ef0 0x7fcc2c19c610 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400e9f0 tx=0x7fcc1400eeb0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.213+0000 7fcc32a00640 1 -- 172.21.15.134:0/2332607166 shutdown_connections 2024-07-26T11:16:07.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.213+0000 7fcc32a00640 1 --2- 172.21.15.134:0/2332607166 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcc100b3d30 0x7fcc100b6240 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.213+0000 7fcc32a00640 1 --2- 172.21.15.134:0/2332607166 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcc2c109840 0x7fcc2c1a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.213+0000 7fcc32a00640 1 --2- 172.21.15.134:0/2332607166 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc2c108ef0 0x7fcc2c19c610 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.213+0000 7fcc32a00640 1 --2- 172.21.15.134:0/2332607166 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc2c1035b0 0x7fcc2c19c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:07.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.213+0000 7fcc32a00640 1 -- 172.21.15.134:0/2332607166 >> 172.21.15.134:0/2332607166 conn(0x7fcc2c0fefc0 msgr2=0x7fcc2c1063f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:07.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.214+0000 7fcc32a00640 1 -- 172.21.15.134:0/2332607166 shutdown_connections 2024-07-26T11:16:07.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:07.214+0000 7fcc32a00640 1 -- 172.21.15.134:0/2332607166 wait complete. 2024-07-26T11:16:07.234 INFO:teuthology.orchestra.run.smithi134.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-07-26T11:16:07.234 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-07-26T11:16:07.234 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-07-26T11:16:07.234 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-07-26T11:16:07.234 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-07-26T11:16:07.234 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-07-26 11:12:11.760495438 +0000 2024-07-26T11:16:07.234 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-07-26 11:12:11.760495438 +0000 2024-07-26T11:16:07.235 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-07-26 11:15:46.686961305 +0000 2024-07-26T11:16:07.235 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-07-26 11:12:11.760495438 +0000 2024-07-26T11:16:07.235 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-07-26T11:16:07.235 DEBUG:teuthology.orchestra.run.smithi134:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-07-26T11:16:07.343 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 2024-07-26T11:16:07.386 INFO:tasks.workunit.client.0.smithi134.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-07-26T11:16:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:07 smithi134 ceph-mon[31966]: mgrmap e24: x(active, since 13m), standbys: z, y 2024-07-26T11:16:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:07 smithi134 ceph-mon[31966]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:16:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:07 smithi134 ceph-mon[31966]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:16:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:07 smithi189 ceph-mon[36790]: mgrmap e24: x(active, since 13m), standbys: z, y 2024-07-26T11:16:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:07 smithi189 ceph-mon[36790]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:16:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:07 smithi189 ceph-mon[36790]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:16:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:07 smithi193 ceph-mon[37473]: mgrmap e24: x(active, since 13m), standbys: z, y 2024-07-26T11:16:07.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:07 smithi193 ceph-mon[37473]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:16:07.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:07 smithi193 ceph-mon[37473]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-07-26T11:16:08.279 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:16:08.792 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:16:09.245 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:09 smithi134 ceph-mon[31966]: scrub queued for path: / 2024-07-26T11:16:09.245 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:09 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:16:09.245 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:09 smithi134 ceph-mon[31966]: scrub summary: active paths [/] 2024-07-26T11:16:09.245 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:09 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:16:09.245 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:09 smithi134 ceph-mon[31966]: pgmap v485: 129 pgs: 129 active+clean; 1.0 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 102 KiB/s wr, 48 op/s 2024-07-26T11:16:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:09 smithi189 ceph-mon[36790]: scrub queued for path: / 2024-07-26T11:16:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:09 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:16:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:09 smithi189 ceph-mon[36790]: scrub summary: active paths [/] 2024-07-26T11:16:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:09 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:16:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:09 smithi189 ceph-mon[36790]: pgmap v485: 129 pgs: 129 active+clean; 1.0 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 102 KiB/s wr, 48 op/s 2024-07-26T11:16:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:09 smithi193 ceph-mon[37473]: scrub queued for path: / 2024-07-26T11:16:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:09 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:16:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:09 smithi193 ceph-mon[37473]: scrub summary: active paths [/] 2024-07-26T11:16:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:09 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:16:09.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:09 smithi193 ceph-mon[37473]: pgmap v485: 129 pgs: 129 active+clean; 1.0 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 102 KiB/s wr, 48 op/s 2024-07-26T11:16:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:10 smithi134 ceph-mon[31966]: pgmap v486: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 174 KiB/s wr, 55 op/s 2024-07-26T11:16:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:10 smithi189 ceph-mon[36790]: pgmap v486: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 174 KiB/s wr, 55 op/s 2024-07-26T11:16:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:10 smithi193 ceph-mon[37473]: pgmap v486: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 174 KiB/s wr, 55 op/s 2024-07-26T11:16:11.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.125+0000 7f0518400640 1 -- 172.21.15.134:0/2949574641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0510077cc0 msgr2=0x7f0510113c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:11.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.125+0000 7f0518400640 1 --2- 172.21.15.134:0/2949574641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0510077cc0 0x7f0510113c30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009900 tx=0x7f04fc02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.125+0000 7f0518400640 1 -- 172.21.15.134:0/2949574641 shutdown_connections 2024-07-26T11:16:11.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.125+0000 7f0518400640 1 --2- 172.21.15.134:0/2949574641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0510114170 0x7f0510116590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.125+0000 7f0518400640 1 --2- 172.21.15.134:0/2949574641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0510077cc0 0x7f0510113c30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.125+0000 7f0518400640 1 --2- 172.21.15.134:0/2949574641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0510077370 0x7f0510077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.125+0000 7f0518400640 1 -- 172.21.15.134:0/2949574641 >> 172.21.15.134:0/2949574641 conn(0x7f0510101240 msgr2=0x7f0510103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:11.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.125+0000 7f0518400640 1 -- 172.21.15.134:0/2949574641 shutdown_connections 2024-07-26T11:16:11.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.125+0000 7f0518400640 1 -- 172.21.15.134:0/2949574641 wait complete. 2024-07-26T11:16:11.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.126+0000 7f0518400640 1 Processor -- start 2024-07-26T11:16:11.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.127+0000 7f0518400640 1 -- start start 2024-07-26T11:16:11.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.127+0000 7f0518400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0510077370 0x7f05101a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.127+0000 7f0518400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0510077cc0 0x7f05101a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.127+0000 7f0518400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0510114170 0x7f05101a5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.127+0000 7f0518400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0510119b20 con 0x7f0510077cc0 2024-07-26T11:16:11.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.127+0000 7f0518400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f05101199a0 con 0x7f0510077370 2024-07-26T11:16:11.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.127+0000 7f0518400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0510119ca0 con 0x7f0510114170 2024-07-26T11:16:11.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.128+0000 7f0515000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0510077cc0 0x7f05101a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.128+0000 7f0515a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0510077370 0x7f05101a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.128+0000 7f0515a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0510077370 0x7f05101a05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:57952/0 (socket says 172.21.15.134:57952) 2024-07-26T11:16:11.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.128+0000 7f0515000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0510077cc0 0x7f05101a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58974/0 (socket says 172.21.15.134:58974) 2024-07-26T11:16:11.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.128+0000 7f0515a00640 1 -- 172.21.15.134:0/3616982005 learned_addr learned my addr 172.21.15.134:0/3616982005 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:11.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.128+0000 7f0516400640 1 --2- 172.21.15.134:0/3616982005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0510114170 0x7f05101a5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.128+0000 7f0515a00640 1 -- 172.21.15.134:0/3616982005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0510114170 msgr2=0x7f05101a5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:11.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.128+0000 7f0515a00640 1 --2- 172.21.15.134:0/3616982005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0510114170 0x7f05101a5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.128+0000 7f0515a00640 1 -- 172.21.15.134:0/3616982005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0510077cc0 msgr2=0x7f05101a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:11.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.128+0000 7f0515a00640 1 --2- 172.21.15.134:0/3616982005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0510077cc0 0x7f05101a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.128+0000 7f0515a00640 1 -- 172.21.15.134:0/3616982005 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05101a6520 con 0x7f0510077370 2024-07-26T11:16:11.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.129+0000 7f0515a00640 1 --2- 172.21.15.134:0/3616982005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0510077370 0x7f05101a05b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f050000e9f0 tx=0x7f050000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:11.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.129+0000 7f050e000640 1 -- 172.21.15.134:0/3616982005 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f050000cd80 con 0x7f0510077370 2024-07-26T11:16:11.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.129+0000 7f0518400640 1 -- 172.21.15.134:0/3616982005 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05101a6810 con 0x7f0510077370 2024-07-26T11:16:11.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.129+0000 7f050e000640 1 -- 172.21.15.134:0/3616982005 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f05000108e0 con 0x7f0510077370 2024-07-26T11:16:11.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.129+0000 7f050e000640 1 -- 172.21.15.134:0/3616982005 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0500018980 con 0x7f0510077370 2024-07-26T11:16:11.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.129+0000 7f0518400640 1 -- 172.21.15.134:0/3616982005 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05101acf70 con 0x7f0510077370 2024-07-26T11:16:11.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.133+0000 7f051f04f740 1 Processor -- start 2024-07-26T11:16:11.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.133+0000 7f051f04f740 1 -- start start 2024-07-26T11:16:11.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.134+0000 7f051f04f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfcf070 0x55f1ce373df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.134+0000 7f051f04f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfa61b0 0x55f1ce374330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.134+0000 7f051f04f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfae390 0x55f1ce374870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.134+0000 7f051f04f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f1ce2eab90 con 0x55f1cdfa61b0 2024-07-26T11:16:11.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.134+0000 7f051f04f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f1ce2da090 con 0x55f1cdfae390 2024-07-26T11:16:11.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.134+0000 7f051f04f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f1ce2de0d0 con 0x55f1cdfcf070 2024-07-26T11:16:11.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.134+0000 7f050e000640 1 -- 172.21.15.134:0/3616982005 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7f0500010430 con 0x7f0510077370 2024-07-26T11:16:11.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.134+0000 7f0516400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfcf070 0x55f1ce373df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.134+0000 7f0516400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfcf070 0x55f1ce373df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51906/0 (socket says 172.21.15.134:51906) 2024-07-26T11:16:11.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.134+0000 7f0516400640 1 -- 172.21.15.134:0/3978934424 learned_addr learned my addr 172.21.15.134:0/3978934424 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:11.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.134+0000 7f0515000640 1 --2- 172.21.15.134:0/3978934424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfae390 0x55f1ce374870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.135+0000 7f0515a00640 1 --2- 172.21.15.134:0/3978934424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfa61b0 0x55f1ce374330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.135+0000 7f0516400640 1 -- 172.21.15.134:0/3978934424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfae390 msgr2=0x55f1ce374870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:11.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.135+0000 7f0516400640 1 --2- 172.21.15.134:0/3978934424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfae390 0x55f1ce374870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.135+0000 7f0516400640 1 -- 172.21.15.134:0/3978934424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfa61b0 msgr2=0x55f1ce374330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:11.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.135+0000 7f0516400640 1 --2- 172.21.15.134:0/3978934424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfa61b0 0x55f1ce374330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.135+0000 7f0516400640 1 -- 172.21.15.134:0/3978934424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1ce03a650 con 0x55f1cdfcf070 2024-07-26T11:16:11.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.135+0000 7f0515a00640 1 --2- 172.21.15.134:0/3978934424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfa61b0 0x55f1ce374330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:16:11.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.135+0000 7f0516400640 1 --2- 172.21.15.134:0/3978934424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfcf070 0x55f1ce373df0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0508009f50 tx=0x7f05080118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5ac976675609262 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:11.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.136+0000 7f0506a00640 1 -- 172.21.15.134:0/3978934424 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0508004d20 con 0x55f1cdfcf070 2024-07-26T11:16:11.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.136+0000 7f0506a00640 1 -- 172.21.15.134:0/3978934424 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0508004050 con 0x55f1cdfcf070 2024-07-26T11:16:11.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.136+0000 7f0506a00640 1 -- 172.21.15.134:0/3978934424 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0508005280 con 0x55f1cdfcf070 2024-07-26T11:16:11.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.137+0000 7f051f04f740 1 -- 172.21.15.134:0/3978934424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfcf070 msgr2=0x55f1ce373df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:11.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.137+0000 7f051f04f740 1 --2- 172.21.15.134:0/3978934424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfcf070 0x55f1ce373df0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0508009f50 tx=0x7f05080118d0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.137+0000 7f051f04f740 1 -- 172.21.15.134:0/3978934424 shutdown_connections 2024-07-26T11:16:11.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.137+0000 7f051f04f740 1 --2- 172.21.15.134:0/3978934424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfae390 0x55f1ce374870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.137+0000 7f051f04f740 1 --2- 172.21.15.134:0/3978934424 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfa61b0 0x55f1ce374330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.137+0000 7f051f04f740 1 --2- 172.21.15.134:0/3978934424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfcf070 0x55f1ce373df0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.137+0000 7f051f04f740 1 -- 172.21.15.134:0/3978934424 >> 172.21.15.134:0/3978934424 conn(0x55f1cdfc3460 msgr2=0x55f1ce371ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:11.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.137+0000 7f051f04f740 1 -- 172.21.15.134:0/3978934424 shutdown_connections 2024-07-26T11:16:11.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.137+0000 7f051f04f740 1 -- 172.21.15.134:0/3978934424 wait complete. 2024-07-26T11:16:11.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.137+0000 7f050e000640 1 --2- 172.21.15.134:0/3616982005 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f04f40b3c40 0x7f04f40b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.137+0000 7f050e000640 1 -- 172.21.15.134:0/3616982005 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f05000e4520 con 0x7f0510077370 2024-07-26T11:16:11.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.137+0000 7f0515000640 1 --2- 172.21.15.134:0/3616982005 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f04f40b3c40 0x7f04f40b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.138+0000 7f051f04f740 1 -- start start 2024-07-26T11:16:11.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.138+0000 7f051f04f740 1 Processor -- start 2024-07-26T11:16:11.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.138+0000 7f0515000640 1 --2- 172.21.15.134:0/3616982005 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f04f40b3c40 0x7f04f40b6150 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f04fc02a9e0 tx=0x7f04fc005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:11.143 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.139+0000 7f051f04f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfa61b0 0x55f1ce37f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.143 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.139+0000 7f051f04f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfae390 0x55f1ce37f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.143 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.139+0000 7f051f04f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfcf070 0x55f1ce37fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.143 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.139+0000 7f051f04f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f1ce13d700 con 0x55f1cdfcf070 2024-07-26T11:16:11.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.139+0000 7f051f04f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f1ce276660 con 0x55f1cdfa61b0 2024-07-26T11:16:11.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.139+0000 7f051f04f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f1ce3825d0 con 0x55f1cdfae390 2024-07-26T11:16:11.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.139+0000 7f051f04f740 10 client.0 _ll_register_callbacks cb 0x55f1ce286110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:16:11.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.139+0000 7f0516400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfcf070 0x55f1ce37fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.139+0000 7f0515000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfae390 0x55f1ce37f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.139+0000 7f0516400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfcf070 0x55f1ce37fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58990/0 (socket says 172.21.15.134:58990) 2024-07-26T11:16:11.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.139+0000 7f0516400640 1 -- 172.21.15.134:0/2373719167 learned_addr learned my addr 172.21.15.134:0/2373719167 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:11.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.139+0000 7f0515a00640 1 --2- 172.21.15.134:0/2373719167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfa61b0 0x55f1ce37f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.139+0000 7f0515000640 1 -- 172.21.15.134:0/2373719167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfa61b0 msgr2=0x55f1ce37f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:11.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.140+0000 7f0515000640 1 --2- 172.21.15.134:0/2373719167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfa61b0 0x55f1ce37f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.140+0000 7f0515000640 1 -- 172.21.15.134:0/2373719167 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfcf070 msgr2=0x55f1ce37fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:11.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.140+0000 7f0515000640 1 --2- 172.21.15.134:0/2373719167 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfcf070 0x55f1ce37fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.140+0000 7f0515000640 1 -- 172.21.15.134:0/2373719167 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1ce28d800 con 0x55f1cdfae390 2024-07-26T11:16:11.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.140+0000 7f0515000640 1 --2- 172.21.15.134:0/2373719167 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfae390 0x55f1ce37f910 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009230 tx=0x7f04fc009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbed39e937390e8c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:11.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.140+0000 7f04fb400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:16:11.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.140+0000 7f04fb400640 1 -- 172.21.15.134:0/2373719167 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc04e070 con 0x55f1cdfae390 2024-07-26T11:16:11.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.140+0000 7f051f04f740 1 -- 172.21.15.134:0/2373719167 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0508004050 con 0x55f1cdfae390 2024-07-26T11:16:11.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.140+0000 7f04fb400640 1 -- 172.21.15.134:0/2373719167 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f04fc004b60 con 0x55f1cdfae390 2024-07-26T11:16:11.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.141+0000 7f04fb400640 1 -- 172.21.15.134:0/2373719167 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f04fc05a050 con 0x55f1cdfae390 2024-07-26T11:16:11.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.141+0000 7f051f04f740 10 client.25552 fetch_fsmap learned FSMap version 30 2024-07-26T11:16:11.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.141+0000 7f051f04f740 1 -- 172.21.15.134:0/2373719167 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f1ce38e190 con 0x55f1cdfae390 2024-07-26T11:16:11.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.141+0000 7f04fb400640 1 -- 172.21.15.134:0/2373719167 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0508004050 con 0x55f1cdfae390 2024-07-26T11:16:11.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.141+0000 7f04fb400640 1 -- 172.21.15.134:0/2373719167 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f04fc0324b0 con 0x55f1cdfae390 2024-07-26T11:16:11.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.141+0000 7f051f04f740 10 client.25552 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:16:11.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.141+0000 7f051f04f740 10 client.25552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:16:11.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.141+0000 7f051f04f740 20 client.25552 populate_metadata read hostname 'smithi134' 2024-07-26T11:16:11.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.141+0000 7f051f04f740 1 --2- 172.21.15.134:0/2373719167 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f1cdf8d9f0 0x55f1ce391600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.141+0000 7f051f04f740 4 client.25552 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:16:11.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.141+0000 7f051f04f740 1 -- 172.21.15.134:0/2373719167 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1ce2ecae0 con 0x55f1cdf8d9f0 2024-07-26T11:16:11.147 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.142+0000 7f0515a00640 1 --2- 172.21.15.134:0/2373719167 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f1cdf8d9f0 0x55f1ce391600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.147 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.142+0000 7f0515a00640 1 --2- 172.21.15.134:0/2373719167 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f1cdf8d9f0 0x55f1ce391600 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58468fde2005d2f7 server_cookie=3a524f217a49c34e in_seq=0 out_seq=0 2024-07-26T11:16:11.147 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.142+0000 7f04fb400640 10 client.25552 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:11.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.143+0000 7f04fb400640 1 -- 172.21.15.134:0/2373719167 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f1ce2ecae0 con 0x55f1cdf8d9f0 2024-07-26T11:16:11.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.143+0000 7f04fb400640 10 client.25552 handle_command_reply: tid=0 2024-07-26T11:16:11.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.143+0000 7f051f04f740 1 client.25552 shutdown 2024-07-26T11:16:11.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.143+0000 7f051f04f740 1 -- 172.21.15.134:0/2373719167 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfae390 msgr2=0x55f1ce37f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:11.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.143+0000 7f051f04f740 1 --2- 172.21.15.134:0/2373719167 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfae390 0x55f1ce37f910 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009230 tx=0x7f04fc009260 comp rx=0 tx=0).stop 2024-07-26T11:16:11.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.143+0000 7f051f04f740 1 -- 172.21.15.134:0/2373719167 shutdown_connections 2024-07-26T11:16:11.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.143+0000 7f051f04f740 1 --2- 172.21.15.134:0/2373719167 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f1cdf8d9f0 0x55f1ce391600 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.143+0000 7f051f04f740 1 --2- 172.21.15.134:0/2373719167 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfae390 0x55f1ce37f910 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.143+0000 7f051f04f740 1 --2- 172.21.15.134:0/2373719167 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfcf070 0x55f1ce37fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.143+0000 7f04fb400640 0 client.25552 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:11.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.143+0000 7f051f04f740 1 --2- 172.21.15.134:0/2373719167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfa61b0 0x55f1ce37f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.148 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.143+0000 7f051f04f740 1 -- 172.21.15.134:0/2373719167 >> 172.21.15.134:0/2373719167 conn(0x55f1cdfc3460 msgr2=0x55f1ce376f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:11.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.144+0000 7f051f04f740 1 -- 172.21.15.134:0/2373719167 shutdown_connections 2024-07-26T11:16:11.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.144+0000 7f051f04f740 1 -- 172.21.15.134:0/2373719167 wait complete. 2024-07-26T11:16:11.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.144+0000 7f051f04f740 20 client.25552 trim_cache size 0 max 16384 2024-07-26T11:16:11.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.163+0000 7f051f04f740 1 Processor -- start 2024-07-26T11:16:11.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.163+0000 7f051f04f740 1 -- start start 2024-07-26T11:16:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.163+0000 7f051f04f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfa61b0 0x55f1ce374e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.163+0000 7f051f04f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfae390 0x55f1ce396d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.163+0000 7f051f04f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfcf070 0x55f1ce3972b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.163+0000 7f051f04f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f05101b6fe0 con 0x55f1cdfa61b0 2024-07-26T11:16:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.163+0000 7f051f04f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f05101b8b20 con 0x55f1cdfcf070 2024-07-26T11:16:11.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.163+0000 7f051f04f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f05101b76b0 con 0x55f1cdfae390 2024-07-26T11:16:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.164+0000 7f0515000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfae390 0x55f1ce396d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.164+0000 7f0515a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfa61b0 0x55f1ce374e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.164+0000 7f0515000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfae390 0x55f1ce396d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51936/0 (socket says 172.21.15.134:51936) 2024-07-26T11:16:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.164+0000 7f0515000640 1 -- 172.21.15.134:0/1390454969 learned_addr learned my addr 172.21.15.134:0/1390454969 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.164+0000 7f0516400640 1 --2- 172.21.15.134:0/1390454969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfcf070 0x55f1ce3972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.164+0000 7f0515000640 1 -- 172.21.15.134:0/1390454969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfcf070 msgr2=0x55f1ce3972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.164+0000 7f0515000640 1 --2- 172.21.15.134:0/1390454969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfcf070 0x55f1ce3972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.164+0000 7f0515000640 1 -- 172.21.15.134:0/1390454969 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfa61b0 msgr2=0x55f1ce374e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.164+0000 7f0515000640 1 --2- 172.21.15.134:0/1390454969 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfa61b0 0x55f1ce374e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.164+0000 7f0515000640 1 -- 172.21.15.134:0/1390454969 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1ce391970 con 0x55f1cdfae390 2024-07-26T11:16:11.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.164+0000 7f0515000640 1 --2- 172.21.15.134:0/1390454969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfae390 0x55f1ce396d70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f04fc006010 tx=0x7f04fc005520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b6e2b18ade85925 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:11.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.165+0000 7f0506000640 1 -- 172.21.15.134:0/1390454969 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc062070 con 0x55f1cdfae390 2024-07-26T11:16:11.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.165+0000 7f0506000640 1 -- 172.21.15.134:0/1390454969 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04fc0041c0 con 0x55f1cdfae390 2024-07-26T11:16:11.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.165+0000 7f0506000640 1 -- 172.21.15.134:0/1390454969 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc04e050 con 0x55f1cdfae390 2024-07-26T11:16:11.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.166+0000 7f051f04f740 1 -- 172.21.15.134:0/1390454969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfae390 msgr2=0x55f1ce396d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:11.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.166+0000 7f051f04f740 1 --2- 172.21.15.134:0/1390454969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfae390 0x55f1ce396d70 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f04fc006010 tx=0x7f04fc005520 comp rx=0 tx=0).stop 2024-07-26T11:16:11.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.166+0000 7f051f04f740 1 -- 172.21.15.134:0/1390454969 shutdown_connections 2024-07-26T11:16:11.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.166+0000 7f051f04f740 1 --2- 172.21.15.134:0/1390454969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfae390 0x55f1ce396d70 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.166+0000 7f051f04f740 1 --2- 172.21.15.134:0/1390454969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfcf070 0x55f1ce3972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.166+0000 7f051f04f740 1 --2- 172.21.15.134:0/1390454969 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfa61b0 0x55f1ce374e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.166+0000 7f051f04f740 1 -- 172.21.15.134:0/1390454969 >> 172.21.15.134:0/1390454969 conn(0x55f1cdfc3460 msgr2=0x55f1ce37a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:11.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.166+0000 7f051f04f740 1 -- 172.21.15.134:0/1390454969 shutdown_connections 2024-07-26T11:16:11.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.166+0000 7f051f04f740 1 -- 172.21.15.134:0/1390454969 wait complete. 2024-07-26T11:16:11.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.166+0000 7f051f04f740 1 -- start start 2024-07-26T11:16:11.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.167+0000 7f051f04f740 1 Processor -- start 2024-07-26T11:16:11.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.167+0000 7f051f04f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfa61b0 0x55f1ce387bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.167+0000 7f051f04f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfae390 0x55f1ce3830a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.167+0000 7f051f04f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfcf070 0x55f1ce3835e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.167+0000 7f051f04f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f05101b8450 con 0x55f1cdfa61b0 2024-07-26T11:16:11.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.167+0000 7f051f04f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f05101b7d80 con 0x55f1cdfae390 2024-07-26T11:16:11.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.167+0000 7f051f04f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f05101b4fc0 con 0x55f1cdfcf070 2024-07-26T11:16:11.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.167+0000 7f051f04f740 10 client.0 _ll_register_callbacks cb 0x7f04fc004380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:16:11.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.167+0000 7f0516400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfcf070 0x55f1ce3835e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.168+0000 7f0516400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfcf070 0x55f1ce3835e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51950/0 (socket says 172.21.15.134:51950) 2024-07-26T11:16:11.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.168+0000 7f0516400640 1 -- 172.21.15.134:0/2347896743 learned_addr learned my addr 172.21.15.134:0/2347896743 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:11.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.168+0000 7f0515a00640 1 --2- 172.21.15.134:0/2347896743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfa61b0 0x55f1ce387bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.168+0000 7f0515000640 1 --2- 172.21.15.134:0/2347896743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfae390 0x55f1ce3830a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.168+0000 7f0516400640 1 -- 172.21.15.134:0/2347896743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfae390 msgr2=0x55f1ce3830a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:11.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.168+0000 7f0516400640 1 --2- 172.21.15.134:0/2347896743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfae390 0x55f1ce3830a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.168+0000 7f0516400640 1 -- 172.21.15.134:0/2347896743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfa61b0 msgr2=0x55f1ce387bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:11.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.168+0000 7f0516400640 1 --2- 172.21.15.134:0/2347896743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfa61b0 0x55f1ce387bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.168+0000 7f0516400640 1 -- 172.21.15.134:0/2347896743 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1ce383b20 con 0x55f1cdfcf070 2024-07-26T11:16:11.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.168+0000 7f0516400640 1 --2- 172.21.15.134:0/2347896743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfcf070 0x55f1ce3835e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0508009ef0 tx=0x7f0508003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7aedb319ab5dd9db server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:11.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.168+0000 7f04fb400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:16:11.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.169+0000 7f04fb400640 1 -- 172.21.15.134:0/2347896743 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0508004050 con 0x55f1cdfcf070 2024-07-26T11:16:11.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.169+0000 7f051f04f740 1 -- 172.21.15.134:0/2347896743 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f1ce2e8580 con 0x55f1cdfcf070 2024-07-26T11:16:11.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.169+0000 7f04fb400640 1 -- 172.21.15.134:0/2347896743 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f05080041f0 con 0x55f1cdfcf070 2024-07-26T11:16:11.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.169+0000 7f04fb400640 1 -- 172.21.15.134:0/2347896743 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0508035050 con 0x55f1cdfcf070 2024-07-26T11:16:11.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.169+0000 7f051f04f740 10 client.25558 fetch_fsmap learned FSMap version 30 2024-07-26T11:16:11.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.169+0000 7f051f04f740 1 -- 172.21.15.134:0/2347896743 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f1ce3f9950 con 0x55f1cdfcf070 2024-07-26T11:16:11.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.169+0000 7f04fb400640 1 -- 172.21.15.134:0/2347896743 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55f1ce2e8580 con 0x55f1cdfcf070 2024-07-26T11:16:11.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.170+0000 7f04fb400640 1 -- 172.21.15.134:0/2347896743 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f050802f4d0 con 0x55f1cdfcf070 2024-07-26T11:16:11.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.170+0000 7f051f04f740 10 client.25558 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:16:11.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.170+0000 7f051f04f740 10 client.25558 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:16:11.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.170+0000 7f051f04f740 20 client.25558 populate_metadata read hostname 'smithi134' 2024-07-26T11:16:11.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.170+0000 7f051f04f740 1 --2- 172.21.15.134:0/2347896743 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f1cdf62da0 0x55f1ce3fbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:11.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.170+0000 7f051f04f740 4 client.25558 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:16:11.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.170+0000 7f051f04f740 1 -- 172.21.15.134:0/2347896743 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f1ce2e5cc0 con 0x55f1cdf62da0 2024-07-26T11:16:11.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.170+0000 7f0515a00640 1 --2- 172.21.15.134:0/2347896743 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f1cdf62da0 0x55f1ce3fbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:11.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.171+0000 7f0515a00640 1 --2- 172.21.15.134:0/2347896743 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f1cdf62da0 0x55f1ce3fbed0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7f27e7362d10579 server_cookie=aaab2e01ee090ed5 in_seq=0 out_seq=0 2024-07-26T11:16:11.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.171+0000 7f04fb400640 10 client.25558 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:11.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.171+0000 7f04fb400640 1 -- 172.21.15.134:0/2347896743 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f1ce2e5cc0 con 0x55f1cdf62da0 2024-07-26T11:16:11.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.171+0000 7f04fb400640 10 client.25558 handle_command_reply: tid=0 2024-07-26T11:16:11.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.171+0000 7f051f04f740 1 client.25558 shutdown 2024-07-26T11:16:11.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.172+0000 7f051f04f740 1 -- 172.21.15.134:0/2347896743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfcf070 msgr2=0x55f1ce3835e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:11.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.172+0000 7f051f04f740 1 --2- 172.21.15.134:0/2347896743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfcf070 0x55f1ce3835e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0508009ef0 tx=0x7f0508003f10 comp rx=0 tx=0).stop 2024-07-26T11:16:11.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.172+0000 7f051f04f740 1 -- 172.21.15.134:0/2347896743 shutdown_connections 2024-07-26T11:16:11.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.172+0000 7f051f04f740 1 --2- 172.21.15.134:0/2347896743 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f1cdf62da0 0x55f1ce3fbed0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.172+0000 7f051f04f740 1 --2- 172.21.15.134:0/2347896743 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1cdfcf070 0x55f1ce3835e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.172+0000 7f051f04f740 1 --2- 172.21.15.134:0/2347896743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1cdfae390 0x55f1ce3830a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.172+0000 7f051f04f740 1 --2- 172.21.15.134:0/2347896743 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f1cdfa61b0 0x55f1ce387bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.172+0000 7f051f04f740 1 -- 172.21.15.134:0/2347896743 >> 172.21.15.134:0/2347896743 conn(0x55f1cdfc3460 msgr2=0x55f1ce37a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:11.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.172+0000 7f04fb400640 0 client.25558 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:11.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.172+0000 7f051f04f740 1 -- 172.21.15.134:0/2347896743 shutdown_connections 2024-07-26T11:16:11.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.172+0000 7f051f04f740 1 -- 172.21.15.134:0/2347896743 wait complete. 2024-07-26T11:16:11.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.172+0000 7f051f04f740 20 client.25558 trim_cache size 0 max 16384 2024-07-26T11:16:11.177 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:16:11.177 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "IDLE (waiting for more scrubs)", 2024-07-26T11:16:11.177 INFO:teuthology.orchestra.run.smithi134.stdout: "scrubs": { 2024-07-26T11:16:11.177 INFO:teuthology.orchestra.run.smithi134.stdout: "b163b36d-b837-4f6c-a892-2000c8eb64bc": { 2024-07-26T11:16:11.177 INFO:teuthology.orchestra.run.smithi134.stdout: "path": "/", 2024-07-26T11:16:11.177 INFO:teuthology.orchestra.run.smithi134.stdout: "tag": "b163b36d-b837-4f6c-a892-2000c8eb64bc", 2024-07-26T11:16:11.178 INFO:teuthology.orchestra.run.smithi134.stdout: "options": "recursive,force" 2024-07-26T11:16:11.178 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-07-26T11:16:11.178 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-07-26T11:16:11.178 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:16:11.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.173+0000 7f0518400640 1 -- 172.21.15.134:0/3616982005 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f04f40b3c40 msgr2=0x7f04f40b6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:11.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.173+0000 7f0518400640 1 --2- 172.21.15.134:0/3616982005 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f04f40b3c40 0x7f04f40b6150 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f04fc02a9e0 tx=0x7f04fc005a20 comp rx=0 tx=0).stop 2024-07-26T11:16:11.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.173+0000 7f0518400640 1 -- 172.21.15.134:0/3616982005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0510077370 msgr2=0x7f05101a05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:11.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.173+0000 7f0518400640 1 --2- 172.21.15.134:0/3616982005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0510077370 0x7f05101a05b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f050000e9f0 tx=0x7f050000eeb0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.173+0000 7f0518400640 1 -- 172.21.15.134:0/3616982005 shutdown_connections 2024-07-26T11:16:11.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.173+0000 7f0518400640 1 --2- 172.21.15.134:0/3616982005 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f04f40b3c40 0x7f04f40b6150 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.173+0000 7f0518400640 1 --2- 172.21.15.134:0/3616982005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0510114170 0x7f05101a5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.173+0000 7f0518400640 1 --2- 172.21.15.134:0/3616982005 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0510077cc0 0x7f05101a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.173+0000 7f0518400640 1 --2- 172.21.15.134:0/3616982005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0510077370 0x7f05101a05b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:11.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.173+0000 7f0518400640 1 -- 172.21.15.134:0/3616982005 >> 172.21.15.134:0/3616982005 conn(0x7f0510101240 msgr2=0x7f0510116250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:11.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.173+0000 7f0518400640 1 -- 172.21.15.134:0/3616982005 shutdown_connections 2024-07-26T11:16:11.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:11.173+0000 7f0518400640 1 -- 172.21.15.134:0/3616982005 wait complete. 2024-07-26T11:16:12.052 INFO:tasks.cephfs.filesystem:scrub status for tag:b163b36d-b837-4f6c-a892-2000c8eb64bc - {'path': '/', 'tag': 'b163b36d-b837-4f6c-a892-2000c8eb64bc', 'options': 'recursive,force'} 2024-07-26T11:16:12.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:12 smithi134 ceph-mon[31966]: pgmap v487: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-07-26T11:16:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:12 smithi189 ceph-mon[36790]: pgmap v487: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-07-26T11:16:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:12 smithi193 ceph-mon[37473]: pgmap v487: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-07-26T11:16:14.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:14 smithi134 ceph-mon[31966]: pgmap v488: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 175 KiB/s wr, 57 op/s 2024-07-26T11:16:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:14 smithi189 ceph-mon[36790]: pgmap v488: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 175 KiB/s wr, 57 op/s 2024-07-26T11:16:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:14 smithi193 ceph-mon[37473]: pgmap v488: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 175 KiB/s wr, 57 op/s 2024-07-26T11:16:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:16 smithi134 ceph-mon[31966]: pgmap v489: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-07-26T11:16:16.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:16 smithi189 ceph-mon[36790]: pgmap v489: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-07-26T11:16:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:16 smithi193 ceph-mon[37473]: pgmap v489: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 175 KiB/s wr, 56 op/s 2024-07-26T11:16:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:18 smithi134 ceph-mon[31966]: pgmap v490: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 175 KiB/s wr, 58 op/s 2024-07-26T11:16:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:18 smithi189 ceph-mon[36790]: pgmap v490: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 175 KiB/s wr, 58 op/s 2024-07-26T11:16:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:18 smithi193 ceph-mon[37473]: pgmap v490: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 175 KiB/s wr, 58 op/s 2024-07-26T11:16:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:20 smithi134 ceph-mon[31966]: scrub summary: idle 2024-07-26T11:16:20.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:20 smithi134 ceph-mon[31966]: pgmap v491: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 73 KiB/s wr, 19 op/s 2024-07-26T11:16:20.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:20 smithi189 ceph-mon[36790]: scrub summary: idle 2024-07-26T11:16:20.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:20 smithi189 ceph-mon[36790]: pgmap v491: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 73 KiB/s wr, 19 op/s 2024-07-26T11:16:20.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.442+0000 7f39b3200640 1 -- 172.21.15.134:0/2444209790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39b4108c90 msgr2=0x7f39b41090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:20.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.442+0000 7f39b3200640 1 --2- 172.21.15.134:0/2444209790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39b4108c90 0x7f39b41090a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f39a4009ca0 tx=0x7f39a402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.442+0000 7f39b3200640 1 -- 172.21.15.134:0/2444209790 shutdown_connections 2024-07-26T11:16:20.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.442+0000 7f39b3200640 1 --2- 172.21.15.134:0/2444209790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39b41095e0 0x7f39b410fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.442+0000 7f39b3200640 1 --2- 172.21.15.134:0/2444209790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39b4108c90 0x7f39b41090a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.442+0000 7f39b3200640 1 --2- 172.21.15.134:0/2444209790 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39b4103360 0x7f39b4103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.442+0000 7f39b3200640 1 -- 172.21.15.134:0/2444209790 >> 172.21.15.134:0/2444209790 conn(0x7f39b40fedc0 msgr2=0x7f39b4101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:20.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.442+0000 7f39b3200640 1 -- 172.21.15.134:0/2444209790 shutdown_connections 2024-07-26T11:16:20.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.442+0000 7f39b3200640 1 -- 172.21.15.134:0/2444209790 wait complete. 2024-07-26T11:16:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.444+0000 7f39b3200640 1 Processor -- start 2024-07-26T11:16:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.444+0000 7f39b3200640 1 -- start start 2024-07-26T11:16:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.444+0000 7f39b3200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39b4103360 0x7f39b4199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.444+0000 7f39b3200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39b4108c90 0x7f39b419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.444+0000 7f39b3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39b41095e0 0x7f39b419f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.444+0000 7f39b3200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f39b4113270 con 0x7f39b4103360 2024-07-26T11:16:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.444+0000 7f39b3200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41130f0 con 0x7f39b41095e0 2024-07-26T11:16:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.444+0000 7f39b3200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41133f0 con 0x7f39b4108c90 2024-07-26T11:16:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.445+0000 7f39b1e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39b4103360 0x7f39b4199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.445+0000 7f39b1e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39b4103360 0x7f39b4199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:54744/0 (socket says 172.21.15.134:54744) 2024-07-26T11:16:20.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.445+0000 7f39b1e00640 1 -- 172.21.15.134:0/2854330643 learned_addr learned my addr 172.21.15.134:0/2854330643 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:20.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.445+0000 7f39b1400640 1 --2- 172.21.15.134:0/2854330643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39b4108c90 0x7f39b419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.445+0000 7f39b2800640 1 --2- 172.21.15.134:0/2854330643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39b41095e0 0x7f39b419f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.445+0000 7f39b1e00640 1 -- 172.21.15.134:0/2854330643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39b4108c90 msgr2=0x7f39b419a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:20.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.445+0000 7f39b1e00640 1 --2- 172.21.15.134:0/2854330643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39b4108c90 0x7f39b419a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.445+0000 7f39b1e00640 1 -- 172.21.15.134:0/2854330643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39b41095e0 msgr2=0x7f39b419f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:20.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.445+0000 7f39b1e00640 1 --2- 172.21.15.134:0/2854330643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39b41095e0 0x7f39b419f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.445+0000 7f39b1e00640 1 -- 172.21.15.134:0/2854330643 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b419fc20 con 0x7f39b4103360 2024-07-26T11:16:20.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.445+0000 7f39b2800640 1 --2- 172.21.15.134:0/2854330643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39b41095e0 0x7f39b419f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:20.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.446+0000 7f39b1e00640 1 --2- 172.21.15.134:0/2854330643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39b4103360 0x7f39b4199cf0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f399c00e9f0 tx=0x7f399c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:20.450 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.446+0000 7f39aaa00640 1 -- 172.21.15.134:0/2854330643 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c00cd80 con 0x7f39b4103360 2024-07-26T11:16:20.450 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.446+0000 7f39aaa00640 1 -- 172.21.15.134:0/2854330643 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f399c0108e0 con 0x7f39b4103360 2024-07-26T11:16:20.450 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.446+0000 7f39aaa00640 1 -- 172.21.15.134:0/2854330643 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c018980 con 0x7f39b4103360 2024-07-26T11:16:20.450 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.446+0000 7f39b3200640 1 -- 172.21.15.134:0/2854330643 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39b419ff10 con 0x7f39b4103360 2024-07-26T11:16:20.450 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.446+0000 7f39b3200640 1 -- 172.21.15.134:0/2854330643 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39b41a66a0 con 0x7f39b4103360 2024-07-26T11:16:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.450+0000 7f39bfc2b740 1 Processor -- start 2024-07-26T11:16:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.450+0000 7f39aaa00640 1 -- 172.21.15.134:0/2854330643 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7f399c010430 con 0x7f39b4103360 2024-07-26T11:16:20.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.450+0000 7f39bfc2b740 1 -- start start 2024-07-26T11:16:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.450+0000 7f39bfc2b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946551b0 0x5584949fd030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.450+0000 7f39bfc2b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849462c2f0 0x5584949fd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.450+0000 7f39bfc2b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5584946344d0 0x5584949fdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.450+0000 7f39bfc2b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55849496d270 con 0x55849462c2f0 2024-07-26T11:16:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.450+0000 7f39bfc2b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55849495c770 con 0x5584946551b0 2024-07-26T11:16:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.450+0000 7f39bfc2b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5584949607b0 con 0x5584946344d0 2024-07-26T11:16:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.451+0000 7f39b1e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849462c2f0 0x5584949fd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.451+0000 7f39b2800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946551b0 0x5584949fd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.451+0000 7f39b1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5584946344d0 0x5584949fdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.451+0000 7f39b2800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946551b0 0x5584949fd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34102/0 (socket says 172.21.15.134:34102) 2024-07-26T11:16:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.451+0000 7f39b2800640 1 -- 172.21.15.134:0/422323160 learned_addr learned my addr 172.21.15.134:0/422323160 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.451+0000 7f39b2800640 1 -- 172.21.15.134:0/422323160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5584946344d0 msgr2=0x5584949fdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.451+0000 7f39b2800640 1 --2- 172.21.15.134:0/422323160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5584946344d0 0x5584949fdab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.451+0000 7f39b2800640 1 -- 172.21.15.134:0/422323160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849462c2f0 msgr2=0x5584949fd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.451+0000 7f39b2800640 1 --2- 172.21.15.134:0/422323160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849462c2f0 0x5584949fd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.451+0000 7f39b2800640 1 -- 172.21.15.134:0/422323160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584946c0790 con 0x5584946551b0 2024-07-26T11:16:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.451+0000 7f39b2800640 1 --2- 172.21.15.134:0/422323160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946551b0 0x5584949fd030 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f39ac011570 tx=0x7f39ac011970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31d29377a8557d72 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.452+0000 7f39aaa00640 1 --2- 172.21.15.134:0/2854330643 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f39980b3d30 0x7f39980b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.452+0000 7f39a3000640 1 -- 172.21.15.134:0/422323160 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac01e070 con 0x5584946551b0 2024-07-26T11:16:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.452+0000 7f39a3000640 1 -- 172.21.15.134:0/422323160 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39ac00f990 con 0x5584946551b0 2024-07-26T11:16:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.452+0000 7f39a3000640 1 -- 172.21.15.134:0/422323160 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac019a80 con 0x5584946551b0 2024-07-26T11:16:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.452+0000 7f39aaa00640 1 -- 172.21.15.134:0/2854330643 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f399c0e4520 con 0x7f39b4103360 2024-07-26T11:16:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.452+0000 7f39b1e00640 1 --2- 172.21.15.134:0/2854330643 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f39980b3d30 0x7f39980b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.453+0000 7f39bfc2b740 1 -- 172.21.15.134:0/422323160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946551b0 msgr2=0x5584949fd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.453+0000 7f39b1e00640 1 --2- 172.21.15.134:0/2854330643 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f39980b3d30 0x7f39980b6240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x5584949fbcb0 tx=0x7f399c0ea0e0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.453+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/422323160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946551b0 0x5584949fd030 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f39ac011570 tx=0x7f39ac011970 comp rx=0 tx=0).stop 2024-07-26T11:16:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.453+0000 7f39bfc2b740 1 -- 172.21.15.134:0/422323160 shutdown_connections 2024-07-26T11:16:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.453+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/422323160 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5584946344d0 0x5584949fdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.453+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/422323160 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849462c2f0 0x5584949fd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.453+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/422323160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946551b0 0x5584949fd030 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.453+0000 7f39bfc2b740 1 -- 172.21.15.134:0/422323160 >> 172.21.15.134:0/422323160 conn(0x558494649300 msgr2=0x5584949faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.453+0000 7f39bfc2b740 1 -- 172.21.15.134:0/422323160 shutdown_connections 2024-07-26T11:16:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.453+0000 7f39bfc2b740 1 -- 172.21.15.134:0/422323160 wait complete. 2024-07-26T11:16:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.454+0000 7f39bfc2b740 1 -- start start 2024-07-26T11:16:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.454+0000 7f39bfc2b740 1 Processor -- start 2024-07-26T11:16:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.454+0000 7f39bfc2b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849462c2f0 0x558494a08c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.454+0000 7f39bfc2b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946344d0 0x558494a09200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.454+0000 7f39bfc2b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5584946551b0 0x558494a14b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.454+0000 7f39bfc2b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x558494896ab0 con 0x5584946551b0 2024-07-26T11:16:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.454+0000 7f39bfc2b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5584948f9d50 con 0x5584946344d0 2024-07-26T11:16:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.454+0000 7f39bfc2b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558494a0b7b0 con 0x55849462c2f0 2024-07-26T11:16:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.455+0000 7f39bfc2b740 10 client.0 _ll_register_callbacks cb 0x55849466ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:16:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.455+0000 7f39b1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946344d0 0x558494a09200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.455+0000 7f39b2800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849462c2f0 0x558494a08c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.455+0000 7f39b2800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849462c2f0 0x558494a08c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:55128/0 (socket says 172.21.15.134:55128) 2024-07-26T11:16:20.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.455+0000 7f39b2800640 1 -- 172.21.15.134:0/1726040121 learned_addr learned my addr 172.21.15.134:0/1726040121 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.455+0000 7f39b1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946344d0 0x558494a09200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34104/0 (socket says 172.21.15.134:34104) 2024-07-26T11:16:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.455+0000 7f39b2800640 1 --2- 172.21.15.134:0/1726040121 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5584946551b0 0x558494a14b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.455+0000 7f39b1400640 1 -- 172.21.15.134:0/1726040121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849462c2f0 msgr2=0x558494a08c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.455+0000 7f39b1400640 1 --2- 172.21.15.134:0/1726040121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849462c2f0 0x558494a08c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.455+0000 7f39b1400640 1 -- 172.21.15.134:0/1726040121 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5584946551b0 msgr2=0x558494a14b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.455+0000 7f39b1400640 1 --2- 172.21.15.134:0/1726040121 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5584946551b0 0x558494a14b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.455+0000 7f39b1400640 1 -- 172.21.15.134:0/1726040121 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558494739620 con 0x5584946344d0 2024-07-26T11:16:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.455+0000 7f39b2800640 1 --2- 172.21.15.134:0/1726040121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849462c2f0 0x558494a08c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.456+0000 7f39b1400640 1 --2- 172.21.15.134:0/1726040121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946344d0 0x558494a09200 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f39a402ade0 tx=0x7f39a402f380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dbd5c837f1468ab server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.456+0000 7f3997400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:16:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.456+0000 7f3997400640 1 -- 172.21.15.134:0/1726040121 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a4031660 con 0x5584946344d0 2024-07-26T11:16:20.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.456+0000 7f39bfc2b740 1 -- 172.21.15.134:0/1726040121 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39ac00f990 con 0x5584946344d0 2024-07-26T11:16:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.456+0000 7f3997400640 1 -- 172.21.15.134:0/1726040121 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f39a4031800 con 0x5584946344d0 2024-07-26T11:16:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.456+0000 7f3997400640 1 -- 172.21.15.134:0/1726040121 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f39a4004b40 con 0x5584946344d0 2024-07-26T11:16:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.456+0000 7f39bfc2b740 10 client.25559 fetch_fsmap learned FSMap version 30 2024-07-26T11:16:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.456+0000 7f39bfc2b740 1 -- 172.21.15.134:0/1726040121 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558494a178f0 con 0x5584946344d0 2024-07-26T11:16:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.457+0000 7f3997400640 1 -- 172.21.15.134:0/1726040121 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39ac00f990 con 0x5584946344d0 2024-07-26T11:16:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.457+0000 7f3997400640 1 -- 172.21.15.134:0/1726040121 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f39a40044c0 con 0x5584946344d0 2024-07-26T11:16:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.457+0000 7f39bfc2b740 10 client.25559 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:16:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.457+0000 7f39bfc2b740 10 client.25559 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:16:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.457+0000 7f39bfc2b740 20 client.25559 populate_metadata read hostname 'smithi134' 2024-07-26T11:16:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.457+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/1726040121 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558494613d80 0x558494a1ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.457+0000 7f39bfc2b740 4 client.25559 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:16:20.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.457+0000 7f39bfc2b740 1 -- 172.21.15.134:0/1726040121 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55849496f1c0 con 0x558494613d80 2024-07-26T11:16:20.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.457+0000 7f39b2800640 1 --2- 172.21.15.134:0/1726040121 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558494613d80 0x558494a1ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.458+0000 7f39b2800640 1 --2- 172.21.15.134:0/1726040121 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558494613d80 0x558494a1ade0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b695b43b62a468e server_cookie=954554ca2070450c in_seq=0 out_seq=0 2024-07-26T11:16:20.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.458+0000 7f3997400640 10 client.25559 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.459+0000 7f3997400640 1 -- 172.21.15.134:0/1726040121 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55849496f1c0 con 0x558494613d80 2024-07-26T11:16:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.459+0000 7f3997400640 10 client.25559 handle_command_reply: tid=0 2024-07-26T11:16:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.459+0000 7f39bfc2b740 1 client.25559 shutdown 2024-07-26T11:16:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.459+0000 7f39bfc2b740 1 -- 172.21.15.134:0/1726040121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946344d0 msgr2=0x558494a09200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.459+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/1726040121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946344d0 0x558494a09200 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f39a402ade0 tx=0x7f39a402f380 comp rx=0 tx=0).stop 2024-07-26T11:16:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.459+0000 7f39bfc2b740 1 -- 172.21.15.134:0/1726040121 shutdown_connections 2024-07-26T11:16:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.459+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/1726040121 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558494613d80 0x558494a1ade0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.459+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/1726040121 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5584946551b0 0x558494a14b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.459+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/1726040121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946344d0 0x558494a09200 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.459+0000 7f3997400640 0 client.25559 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.459+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/1726040121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849462c2f0 0x558494a08c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.459+0000 7f39bfc2b740 1 -- 172.21.15.134:0/1726040121 >> 172.21.15.134:0/1726040121 conn(0x558494649300 msgr2=0x5584949fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.459+0000 7f39bfc2b740 1 -- 172.21.15.134:0/1726040121 shutdown_connections 2024-07-26T11:16:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.459+0000 7f39bfc2b740 1 -- 172.21.15.134:0/1726040121 wait complete. 2024-07-26T11:16:20.466 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.459+0000 7f39bfc2b740 20 client.25559 trim_cache size 0 max 16384 2024-07-26T11:16:20.478 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.476+0000 7f39bfc2b740 1 Processor -- start 2024-07-26T11:16:20.478 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.476+0000 7f39bfc2b740 1 -- start start 2024-07-26T11:16:20.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39bfc2b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849462c2f0 0x5584949faf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39bfc2b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946344d0 0x558494a03100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39bfc2b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5584946551b0 0x558494a148f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39bfc2b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41ae490 con 0x55849462c2f0 2024-07-26T11:16:20.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39bfc2b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41affd0 con 0x5584946344d0 2024-07-26T11:16:20.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39bfc2b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41aeb60 con 0x5584946551b0 2024-07-26T11:16:20.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39b2800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849462c2f0 0x5584949faf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39b2800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849462c2f0 0x5584949faf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:54776/0 (socket says 172.21.15.134:54776) 2024-07-26T11:16:20.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39b2800640 1 -- 172.21.15.134:0/3040148157 learned_addr learned my addr 172.21.15.134:0/3040148157 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:20.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39b1400640 1 --2- 172.21.15.134:0/3040148157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946344d0 0x558494a03100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39b2800640 1 --2- 172.21.15.134:0/3040148157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5584946551b0 0x558494a148f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39b2800640 1 -- 172.21.15.134:0/3040148157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5584946551b0 msgr2=0x558494a148f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:20.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39b2800640 1 --2- 172.21.15.134:0/3040148157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5584946551b0 0x558494a148f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39b2800640 1 -- 172.21.15.134:0/3040148157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946344d0 msgr2=0x558494a03100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:20.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39b2800640 1 --2- 172.21.15.134:0/3040148157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946344d0 0x558494a03100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39b2800640 1 -- 172.21.15.134:0/3040148157 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558494a06f30 con 0x55849462c2f0 2024-07-26T11:16:20.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.477+0000 7f39b1400640 1 --2- 172.21.15.134:0/3040148157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946344d0 0x558494a03100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:16:20.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.478+0000 7f39b2800640 1 --2- 172.21.15.134:0/3040148157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849462c2f0 0x5584949faf50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f39ac013010 tx=0x7f39ac005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e54e24cb45ee38 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:20.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.478+0000 7f39a2600640 1 -- 172.21.15.134:0/3040148157 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac01e040 con 0x55849462c2f0 2024-07-26T11:16:20.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.478+0000 7f39a2600640 1 -- 172.21.15.134:0/3040148157 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39ac005560 con 0x55849462c2f0 2024-07-26T11:16:20.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.478+0000 7f39a2600640 1 -- 172.21.15.134:0/3040148157 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39ac019e00 con 0x55849462c2f0 2024-07-26T11:16:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.479+0000 7f39bfc2b740 1 -- 172.21.15.134:0/3040148157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849462c2f0 msgr2=0x5584949faf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.479+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/3040148157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849462c2f0 0x5584949faf50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f39ac013010 tx=0x7f39ac005850 comp rx=0 tx=0).stop 2024-07-26T11:16:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.479+0000 7f39bfc2b740 1 -- 172.21.15.134:0/3040148157 shutdown_connections 2024-07-26T11:16:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.479+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/3040148157 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5584946551b0 0x558494a148f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.479+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/3040148157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946344d0 0x558494a03100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.479+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/3040148157 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849462c2f0 0x5584949faf50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.479+0000 7f39bfc2b740 1 -- 172.21.15.134:0/3040148157 >> 172.21.15.134:0/3040148157 conn(0x558494649300 msgr2=0x5584949fa840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.479+0000 7f39bfc2b740 1 -- 172.21.15.134:0/3040148157 shutdown_connections 2024-07-26T11:16:20.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.479+0000 7f39bfc2b740 1 -- 172.21.15.134:0/3040148157 wait complete. 2024-07-26T11:16:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.479+0000 7f39bfc2b740 1 -- start start 2024-07-26T11:16:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.480+0000 7f39bfc2b740 1 Processor -- start 2024-07-26T11:16:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.480+0000 7f39bfc2b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849462c2f0 0x558494a0c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.480+0000 7f39bfc2b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5584946344d0 0x558494a0c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.480+0000 7f39bfc2b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946551b0 0x558494a7b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.480+0000 7f39bfc2b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41af900 con 0x5584946344d0 2024-07-26T11:16:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.480+0000 7f39bfc2b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41af230 con 0x5584946551b0 2024-07-26T11:16:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.480+0000 7f39bfc2b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f39b41ac470 con 0x55849462c2f0 2024-07-26T11:16:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.480+0000 7f39bfc2b740 10 client.0 _ll_register_callbacks cb 0x7f39a4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:16:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.480+0000 7f39b1400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5584946344d0 0x558494a0c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.481+0000 7f39b1400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5584946344d0 0x558494a0c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:54778/0 (socket says 172.21.15.134:54778) 2024-07-26T11:16:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.481+0000 7f39b1400640 1 -- 172.21.15.134:0/385281794 learned_addr learned my addr 172.21.15.134:0/385281794 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.481+0000 7f39b2800640 1 --2- 172.21.15.134:0/385281794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849462c2f0 0x558494a0c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.481+0000 7f39b2800640 1 --2- 172.21.15.134:0/385281794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946551b0 0x558494a7b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.481+0000 7f39b1400640 1 -- 172.21.15.134:0/385281794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849462c2f0 msgr2=0x558494a0c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.481+0000 7f39b1400640 1 --2- 172.21.15.134:0/385281794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849462c2f0 0x558494a0c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.481+0000 7f39b1400640 1 -- 172.21.15.134:0/385281794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946551b0 msgr2=0x558494a7b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.481+0000 7f39b1400640 1 --2- 172.21.15.134:0/385281794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946551b0 0x558494a7b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.481+0000 7f39b1400640 1 -- 172.21.15.134:0/385281794 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558494a7bb50 con 0x5584946344d0 2024-07-26T11:16:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.481+0000 7f39b2800640 1 --2- 172.21.15.134:0/385281794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946551b0 0x558494a7b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:16:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.481+0000 7f39b1400640 1 --2- 172.21.15.134:0/385281794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5584946344d0 0x558494a0c820 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f39a4000f80 tx=0x7f39a4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e53b31118d41852 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.481+0000 7f3997400640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:16:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.481+0000 7f3997400640 1 -- 172.21.15.134:0/385281794 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a4058060 con 0x5584946344d0 2024-07-26T11:16:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.481+0000 7f3997400640 1 -- 172.21.15.134:0/385281794 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f39a4003d90 con 0x5584946344d0 2024-07-26T11:16:20.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.481+0000 7f39bfc2b740 1 -- 172.21.15.134:0/385281794 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39ac005560 con 0x5584946344d0 2024-07-26T11:16:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.482+0000 7f3997400640 1 -- 172.21.15.134:0/385281794 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f39a405f050 con 0x5584946344d0 2024-07-26T11:16:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.482+0000 7f39bfc2b740 10 client.15411 fetch_fsmap learned FSMap version 30 2024-07-26T11:16:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.482+0000 7f39bfc2b740 1 -- 172.21.15.134:0/385281794 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558494a82e80 con 0x5584946344d0 2024-07-26T11:16:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.482+0000 7f3997400640 1 -- 172.21.15.134:0/385281794 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39ac005560 con 0x5584946344d0 2024-07-26T11:16:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.482+0000 7f3997400640 1 -- 172.21.15.134:0/385281794 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f39a40042a0 con 0x5584946344d0 2024-07-26T11:16:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.482+0000 7f39bfc2b740 10 client.15411 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:16:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.482+0000 7f39bfc2b740 10 client.15411 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:16:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.482+0000 7f39bfc2b740 20 client.15411 populate_metadata read hostname 'smithi134' 2024-07-26T11:16:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.482+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/385281794 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5584945e91c0 0x558494a855e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.482+0000 7f39bfc2b740 4 client.15411 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:16:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.482+0000 7f39bfc2b740 1 -- 172.21.15.134:0/385281794 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55849496ac60 con 0x5584945e91c0 2024-07-26T11:16:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.483+0000 7f39b2800640 1 --2- 172.21.15.134:0/385281794 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5584945e91c0 0x558494a855e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:20.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.483+0000 7f39b2800640 1 --2- 172.21.15.134:0/385281794 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5584945e91c0 0x558494a855e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=79514b642b15fd3d server_cookie=3c6fa67c1a7896c8 in_seq=0 out_seq=0 2024-07-26T11:16:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.483+0000 7f3997400640 10 client.15411 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.498+0000 7f3997400640 1 -- 172.21.15.134:0/385281794 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55849496ac60 con 0x5584945e91c0 2024-07-26T11:16:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.498+0000 7f3997400640 10 client.15411 handle_command_reply: tid=0 2024-07-26T11:16:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.498+0000 7f39bfc2b740 1 client.15411 shutdown 2024-07-26T11:16:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.499+0000 7f39bfc2b740 1 -- 172.21.15.134:0/385281794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5584946344d0 msgr2=0x558494a0c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.499+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/385281794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5584946344d0 0x558494a0c820 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f39a4000f80 tx=0x7f39a4003ba0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.499+0000 7f39bfc2b740 1 -- 172.21.15.134:0/385281794 shutdown_connections 2024-07-26T11:16:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.499+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/385281794 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5584945e91c0 0x558494a855e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.499+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/385281794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5584946551b0 0x558494a7b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.499+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/385281794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5584946344d0 0x558494a0c820 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.499+0000 7f39bfc2b740 1 --2- 172.21.15.134:0/385281794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849462c2f0 0x558494a0c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.499+0000 7f3997400640 0 client.15411 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.499+0000 7f39bfc2b740 1 -- 172.21.15.134:0/385281794 >> 172.21.15.134:0/385281794 conn(0x558494649300 msgr2=0x5584949fe060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.499+0000 7f39bfc2b740 1 -- 172.21.15.134:0/385281794 shutdown_connections 2024-07-26T11:16:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.499+0000 7f39bfc2b740 1 -- 172.21.15.134:0/385281794 wait complete. 2024-07-26T11:16:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.499+0000 7f39bfc2b740 20 client.15411 trim_cache size 0 max 16384 2024-07-26T11:16:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:16:20.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:16:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:16:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:16:20.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.500+0000 7f39b3200640 1 -- 172.21.15.134:0/2854330643 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f39980b3d30 msgr2=0x7f39980b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.500+0000 7f39b3200640 1 --2- 172.21.15.134:0/2854330643 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f39980b3d30 0x7f39980b6240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x5584949fbcb0 tx=0x7f399c0ea0e0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.500+0000 7f39b3200640 1 -- 172.21.15.134:0/2854330643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39b4103360 msgr2=0x7f39b4199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.500+0000 7f39b3200640 1 --2- 172.21.15.134:0/2854330643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39b4103360 0x7f39b4199cf0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f399c00e9f0 tx=0x7f399c00eeb0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.500+0000 7f39b3200640 1 -- 172.21.15.134:0/2854330643 shutdown_connections 2024-07-26T11:16:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.500+0000 7f39b3200640 1 --2- 172.21.15.134:0/2854330643 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f39980b3d30 0x7f39980b6240 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.500+0000 7f39b3200640 1 --2- 172.21.15.134:0/2854330643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f39b4108c90 0x7f39b419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.500+0000 7f39b3200640 1 --2- 172.21.15.134:0/2854330643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39b41095e0 0x7f39b419f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.500+0000 7f39b3200640 1 --2- 172.21.15.134:0/2854330643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f39b4103360 0x7f39b4199cf0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.500+0000 7f39b3200640 1 -- 172.21.15.134:0/2854330643 >> 172.21.15.134:0/2854330643 conn(0x7f39b40fedc0 msgr2=0x7f39b41011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.500+0000 7f39b3200640 1 -- 172.21.15.134:0/2854330643 shutdown_connections 2024-07-26T11:16:20.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:20.500+0000 7f39b3200640 1 -- 172.21.15.134:0/2854330643 wait complete. 2024-07-26T11:16:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:20 smithi193 ceph-mon[37473]: scrub summary: idle 2024-07-26T11:16:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:20 smithi193 ceph-mon[37473]: pgmap v491: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 73 KiB/s wr, 19 op/s 2024-07-26T11:16:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:22 smithi134 ceph-mon[31966]: pgmap v492: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-26T11:16:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:22 smithi189 ceph-mon[36790]: pgmap v492: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-26T11:16:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:22 smithi193 ceph-mon[37473]: pgmap v492: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 341 B/s wr, 11 op/s 2024-07-26T11:16:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:24 smithi134 ceph-mon[31966]: pgmap v493: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-07-26T11:16:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:24 smithi189 ceph-mon[36790]: pgmap v493: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-07-26T11:16:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:24 smithi193 ceph-mon[37473]: pgmap v493: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-07-26T11:16:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:16:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:16:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:16:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:26 smithi134 ceph-mon[31966]: pgmap v494: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-07-26T11:16:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:16:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:16:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:16:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:26 smithi189 ceph-mon[36790]: pgmap v494: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-07-26T11:16:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:16:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:16:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:16:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:26 smithi193 ceph-mon[37473]: pgmap v494: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-07-26T11:16:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:16:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:16:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:16:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:28 smithi134 ceph-mon[31966]: pgmap v495: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.6 KiB/s wr, 15 op/s 2024-07-26T11:16:28.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:28 smithi189 ceph-mon[36790]: pgmap v495: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.6 KiB/s wr, 15 op/s 2024-07-26T11:16:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:28 smithi193 ceph-mon[37473]: pgmap v495: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.6 KiB/s wr, 15 op/s 2024-07-26T11:16:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:30 smithi134 ceph-mon[31966]: pgmap v496: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-07-26T11:16:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:30 smithi189 ceph-mon[36790]: pgmap v496: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-07-26T11:16:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:30 smithi193 ceph-mon[37473]: pgmap v496: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-07-26T11:16:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:32 smithi134 ceph-mon[31966]: pgmap v497: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-07-26T11:16:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:32 smithi189 ceph-mon[36790]: pgmap v497: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-07-26T11:16:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:32 smithi193 ceph-mon[37473]: pgmap v497: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-07-26T11:16:33.728 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.726+0000 7f7b30c00640 1 -- 172.21.15.134:0/4096016530 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b2c101820 msgr2=0x7f7b2c101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:33.728 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.726+0000 7f7b30c00640 1 --2- 172.21.15.134:0/4096016530 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b2c101820 0x7f7b2c101c30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c009900 tx=0x7f7b1c02a410 comp rx=0 tx=0).stop 2024-07-26T11:16:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.726+0000 7f7b30c00640 1 -- 172.21.15.134:0/4096016530 shutdown_connections 2024-07-26T11:16:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.726+0000 7f7b30c00640 1 --2- 172.21.15.134:0/4096016530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b2c1063b0 0x7f7b2c10fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.726+0000 7f7b30c00640 1 --2- 172.21.15.134:0/4096016530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b2c105a60 0x7f7b2c105e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.726+0000 7f7b30c00640 1 --2- 172.21.15.134:0/4096016530 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b2c101820 0x7f7b2c101c30 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.726+0000 7f7b30c00640 1 -- 172.21.15.134:0/4096016530 >> 172.21.15.134:0/4096016530 conn(0x7f7b2c079530 msgr2=0x7f7b2c079980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.726+0000 7f7b30c00640 1 -- 172.21.15.134:0/4096016530 shutdown_connections 2024-07-26T11:16:33.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.726+0000 7f7b30c00640 1 -- 172.21.15.134:0/4096016530 wait complete. 2024-07-26T11:16:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.727+0000 7f7b30c00640 1 Processor -- start 2024-07-26T11:16:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.727+0000 7f7b30c00640 1 -- start start 2024-07-26T11:16:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.728+0000 7f7b30c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b2c101820 0x7f7b2c19bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.728+0000 7f7b30c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b2c105a60 0x7f7b2c19c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.728+0000 7f7b30c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b2c1063b0 0x7f7b2c1a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.728+0000 7f7b30c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1132b0 con 0x7f7b2c1063b0 2024-07-26T11:16:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.728+0000 7f7b30c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c113130 con 0x7f7b2c101820 2024-07-26T11:16:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.728+0000 7f7b30c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c113430 con 0x7f7b2c105a60 2024-07-26T11:16:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.728+0000 7f7b2b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b2c101820 0x7f7b2c19bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.728+0000 7f7b2b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b2c101820 0x7f7b2c19bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:55346/0 (socket says 172.21.15.134:55346) 2024-07-26T11:16:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.728+0000 7f7b2be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b2c1063b0 0x7f7b2c1a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.728+0000 7f7b2b400640 1 -- 172.21.15.134:0/1144076539 learned_addr learned my addr 172.21.15.134:0/1144076539 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.728+0000 7f7b2aa00640 1 --2- 172.21.15.134:0/1144076539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b2c105a60 0x7f7b2c19c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.729+0000 7f7b2b400640 1 -- 172.21.15.134:0/1144076539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b2c105a60 msgr2=0x7f7b2c19c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.729+0000 7f7b2b400640 1 --2- 172.21.15.134:0/1144076539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b2c105a60 0x7f7b2c19c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.729+0000 7f7b2b400640 1 -- 172.21.15.134:0/1144076539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b2c1063b0 msgr2=0x7f7b2c1a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.729+0000 7f7b2b400640 1 --2- 172.21.15.134:0/1144076539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b2c1063b0 0x7f7b2c1a17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.729+0000 7f7b2b400640 1 -- 172.21.15.134:0/1144076539 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b2c1a1ec0 con 0x7f7b2c101820 2024-07-26T11:16:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.729+0000 7f7b2b400640 1 --2- 172.21.15.134:0/1144076539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b2c101820 0x7f7b2c19bf90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c032fd0 tx=0x7f7b1c031a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:33.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.729+0000 7f7b23e00640 1 -- 172.21.15.134:0/1144076539 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c03f030 con 0x7f7b2c101820 2024-07-26T11:16:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.729+0000 7f7b30c00640 1 -- 172.21.15.134:0/1144076539 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b2c1a2150 con 0x7f7b2c101820 2024-07-26T11:16:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.729+0000 7f7b23e00640 1 -- 172.21.15.134:0/1144076539 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b1c004490 con 0x7f7b2c101820 2024-07-26T11:16:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.729+0000 7f7b23e00640 1 -- 172.21.15.134:0/1144076539 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c035070 con 0x7f7b2c101820 2024-07-26T11:16:33.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.729+0000 7f7b30c00640 1 -- 172.21.15.134:0/1144076539 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b2c1a8940 con 0x7f7b2c101820 2024-07-26T11:16:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.733+0000 7f7b39189740 1 Processor -- start 2024-07-26T11:16:33.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.733+0000 7f7b39189740 1 -- start start 2024-07-26T11:16:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.734+0000 7f7b39189740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb971b0 0x555fbff3ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.734+0000 7f7b39189740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 0x555fbff3b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.734+0000 7f7b39189740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb764d0 0x555fbff3b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.734+0000 7f7b39189740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x555fbfeaf270 con 0x555fbfb6e2f0 2024-07-26T11:16:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.734+0000 7f7b39189740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555fbfe9e770 con 0x555fbfb764d0 2024-07-26T11:16:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.734+0000 7f7b39189740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555fbfea27b0 con 0x555fbfb971b0 2024-07-26T11:16:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.734+0000 7f7b2be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb971b0 0x555fbff3ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.734+0000 7f7b2aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb764d0 0x555fbff3b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.734+0000 7f7b2be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb971b0 0x555fbff3ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:46242/0 (socket says 172.21.15.134:46242) 2024-07-26T11:16:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.734+0000 7f7b2be00640 1 -- 172.21.15.134:0/2428417443 learned_addr learned my addr 172.21.15.134:0/2428417443 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.734+0000 7f7b23e00640 1 -- 172.21.15.134:0/1144076539 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7f7b1c042030 con 0x7f7b2c101820 2024-07-26T11:16:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.735+0000 7f7b2aa00640 1 -- 172.21.15.134:0/2428417443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb971b0 msgr2=0x555fbff3ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.735+0000 7f7b2aa00640 1 --2- 172.21.15.134:0/2428417443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb971b0 0x555fbff3ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.735+0000 7f7b2aa00640 1 -- 172.21.15.134:0/2428417443 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 msgr2=0x555fbff3b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.735+0000 7f7b2aa00640 1 --2- 172.21.15.134:0/2428417443 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 0x555fbff3b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.735+0000 7f7b2aa00640 1 -- 172.21.15.134:0/2428417443 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fbfc02790 con 0x555fbfb764d0 2024-07-26T11:16:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.735+0000 7f7b2be00640 1 --2- 172.21.15.134:0/2428417443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb971b0 0x555fbff3ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:33.738 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.735+0000 7f7b2aa00640 1 --2- 172.21.15.134:0/2428417443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb764d0 0x555fbff3b6a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c0751b0 tx=0x7f7b18011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17d7e7bddfff9108 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:33.739 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.735+0000 7f7b17e00640 1 -- 172.21.15.134:0/2428417443 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b18009d70 con 0x555fbfb764d0 2024-07-26T11:16:33.739 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.735+0000 7f7b17e00640 1 -- 172.21.15.134:0/2428417443 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b18005020 con 0x555fbfb764d0 2024-07-26T11:16:33.739 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.736+0000 7f7b17e00640 1 -- 172.21.15.134:0/2428417443 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b18004a00 con 0x555fbfb764d0 2024-07-26T11:16:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.736+0000 7f7b23e00640 1 --2- 172.21.15.134:0/1144076539 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b100b3cb0 0x7f7b100b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.736+0000 7f7b39189740 1 -- 172.21.15.134:0/2428417443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb764d0 msgr2=0x555fbff3b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.736+0000 7f7b39189740 1 --2- 172.21.15.134:0/2428417443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb764d0 0x555fbff3b6a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c0751b0 tx=0x7f7b18011810 comp rx=0 tx=0).stop 2024-07-26T11:16:33.742 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.736+0000 7f7b23e00640 1 -- 172.21.15.134:0/1144076539 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7b1c104290 con 0x7f7b2c101820 2024-07-26T11:16:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.736+0000 7f7b2be00640 1 --2- 172.21.15.134:0/1144076539 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b100b3cb0 0x7f7b100b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.737+0000 7f7b39189740 1 -- 172.21.15.134:0/2428417443 shutdown_connections 2024-07-26T11:16:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.737+0000 7f7b39189740 1 --2- 172.21.15.134:0/2428417443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb764d0 0x555fbff3b6a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.737+0000 7f7b39189740 1 --2- 172.21.15.134:0/2428417443 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 0x555fbff3b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.737+0000 7f7b39189740 1 --2- 172.21.15.134:0/2428417443 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb971b0 0x555fbff3ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.737+0000 7f7b39189740 1 -- 172.21.15.134:0/2428417443 >> 172.21.15.134:0/2428417443 conn(0x555fbfb8b300 msgr2=0x555fbff38b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.737+0000 7f7b39189740 1 -- 172.21.15.134:0/2428417443 shutdown_connections 2024-07-26T11:16:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.737+0000 7f7b39189740 1 -- 172.21.15.134:0/2428417443 wait complete. 2024-07-26T11:16:33.743 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.737+0000 7f7b2be00640 1 --2- 172.21.15.134:0/1144076539 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b100b3cb0 0x7f7b100b61c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009d60 tx=0x7f7b24009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.737+0000 7f7b39189740 1 -- start start 2024-07-26T11:16:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.738+0000 7f7b39189740 1 Processor -- start 2024-07-26T11:16:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.738+0000 7f7b39189740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb6e2f0 0x555fbff468e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.738+0000 7f7b39189740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb764d0 0x555fbff46ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.738+0000 7f7b39189740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb971b0 0x555fbff4c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.738+0000 7f7b39189740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x555fbfdd8ab0 con 0x555fbfb764d0 2024-07-26T11:16:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.738+0000 7f7b39189740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555fbfe3bd50 con 0x555fbfb971b0 2024-07-26T11:16:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.738+0000 7f7b39189740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555fbff493a0 con 0x555fbfb6e2f0 2024-07-26T11:16:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.738+0000 7f7b39189740 10 client.0 _ll_register_callbacks cb 0x555fbfbaea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:16:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.738+0000 7f7b2aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb764d0 0x555fbff46ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.738+0000 7f7b2be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb971b0 0x555fbff4c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.739+0000 7f7b2b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb6e2f0 0x555fbff468e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.739+0000 7f7b2be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb971b0 0x555fbff4c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:55356/0 (socket says 172.21.15.134:55356) 2024-07-26T11:16:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.739+0000 7f7b2be00640 1 -- 172.21.15.134:0/2564160960 learned_addr learned my addr 172.21.15.134:0/2564160960 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:33.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.739+0000 7f7b2b400640 1 -- 172.21.15.134:0/2564160960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb971b0 msgr2=0x555fbff4c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.739+0000 7f7b2b400640 1 --2- 172.21.15.134:0/2564160960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb971b0 0x555fbff4c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.739+0000 7f7b2b400640 1 -- 172.21.15.134:0/2564160960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb764d0 msgr2=0x555fbff46ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.739+0000 7f7b2b400640 1 --2- 172.21.15.134:0/2564160960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb764d0 0x555fbff46ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.739+0000 7f7b2b400640 1 -- 172.21.15.134:0/2564160960 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fbfc7b620 con 0x555fbfb6e2f0 2024-07-26T11:16:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.739+0000 7f7b2be00640 1 --2- 172.21.15.134:0/2564160960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb971b0 0x555fbff4c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.739+0000 7f7b2b400640 1 --2- 172.21.15.134:0/2564160960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb6e2f0 0x555fbff468e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c1017a0 tx=0x7f7b1c0ca400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c663a5b056ad4ae server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.739+0000 7f7b14c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:16:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.739+0000 7f7b14c00640 1 -- 172.21.15.134:0/2564160960 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c049040 con 0x555fbfb6e2f0 2024-07-26T11:16:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.739+0000 7f7b39189740 1 -- 172.21.15.134:0/2564160960 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b18005020 con 0x555fbfb6e2f0 2024-07-26T11:16:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.739+0000 7f7b14c00640 1 -- 172.21.15.134:0/2564160960 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7b1c0c6f20 con 0x555fbfb6e2f0 2024-07-26T11:16:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.740+0000 7f7b14c00640 1 -- 172.21.15.134:0/2564160960 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7b1c0cac90 con 0x555fbfb6e2f0 2024-07-26T11:16:33.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.740+0000 7f7b39189740 10 client.25588 fetch_fsmap learned FSMap version 30 2024-07-26T11:16:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.740+0000 7f7b39189740 1 -- 172.21.15.134:0/2564160960 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555fbff55510 con 0x555fbfb6e2f0 2024-07-26T11:16:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.740+0000 7f7b14c00640 1 -- 172.21.15.134:0/2564160960 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b18005020 con 0x555fbfb6e2f0 2024-07-26T11:16:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.740+0000 7f7b14c00640 1 -- 172.21.15.134:0/2564160960 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f7b1c0d3800 con 0x555fbfb6e2f0 2024-07-26T11:16:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.740+0000 7f7b39189740 10 client.25588 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:16:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.740+0000 7f7b39189740 10 client.25588 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:16:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.740+0000 7f7b39189740 20 client.25588 populate_metadata read hostname 'smithi134' 2024-07-26T11:16:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.740+0000 7f7b39189740 1 --2- 172.21.15.134:0/2564160960 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x555fbfb55d80 0x555fbff589d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.740+0000 7f7b39189740 4 client.25588 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:16:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.740+0000 7f7b39189740 1 -- 172.21.15.134:0/2564160960 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555fbfeb11c0 con 0x555fbfb55d80 2024-07-26T11:16:33.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.741+0000 7f7b2aa00640 1 --2- 172.21.15.134:0/2564160960 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x555fbfb55d80 0x555fbff589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.741+0000 7f7b14c00640 10 client.25588 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.741+0000 7f7b2aa00640 1 --2- 172.21.15.134:0/2564160960 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x555fbfb55d80 0x555fbff589d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bf5db53644f8c28 server_cookie=d4ee8c0a1ea103cb in_seq=0 out_seq=0 2024-07-26T11:16:33.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.742+0000 7f7b14c00640 1 -- 172.21.15.134:0/2564160960 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555fbfeb11c0 con 0x555fbfb55d80 2024-07-26T11:16:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.742+0000 7f7b14c00640 10 client.25588 handle_command_reply: tid=0 2024-07-26T11:16:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.742+0000 7f7b39189740 1 client.25588 shutdown 2024-07-26T11:16:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.742+0000 7f7b39189740 1 -- 172.21.15.134:0/2564160960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb6e2f0 msgr2=0x555fbff468e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.742+0000 7f7b39189740 1 --2- 172.21.15.134:0/2564160960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb6e2f0 0x555fbff468e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c1017a0 tx=0x7f7b1c0ca400 comp rx=0 tx=0).stop 2024-07-26T11:16:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.742+0000 7f7b39189740 1 -- 172.21.15.134:0/2564160960 shutdown_connections 2024-07-26T11:16:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.742+0000 7f7b39189740 1 --2- 172.21.15.134:0/2564160960 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x555fbfb55d80 0x555fbff589d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.742+0000 7f7b39189740 1 --2- 172.21.15.134:0/2564160960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb971b0 0x555fbff4c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.742+0000 7f7b39189740 1 --2- 172.21.15.134:0/2564160960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb764d0 0x555fbff46ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.742+0000 7f7b14c00640 0 client.25588 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.742+0000 7f7b39189740 1 --2- 172.21.15.134:0/2564160960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb6e2f0 0x555fbff468e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.742+0000 7f7b39189740 1 -- 172.21.15.134:0/2564160960 >> 172.21.15.134:0/2564160960 conn(0x555fbfb8b300 msgr2=0x555fbff3bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.743+0000 7f7b39189740 1 -- 172.21.15.134:0/2564160960 shutdown_connections 2024-07-26T11:16:33.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.743+0000 7f7b39189740 1 -- 172.21.15.134:0/2564160960 wait complete. 2024-07-26T11:16:33.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.743+0000 7f7b39189740 20 client.25588 trim_cache size 0 max 16384 2024-07-26T11:16:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.763+0000 7f7b39189740 1 Processor -- start 2024-07-26T11:16:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.763+0000 7f7b39189740 1 -- start start 2024-07-26T11:16:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.764+0000 7f7b39189740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 0x555fbff4c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.764+0000 7f7b39189740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb764d0 0x555fbff53480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.764+0000 7f7b39189740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb971b0 0x555fbff403f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.764+0000 7f7b39189740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1b2880 con 0x555fbfb6e2f0 2024-07-26T11:16:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.764+0000 7f7b39189740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1b43c0 con 0x555fbfb971b0 2024-07-26T11:16:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.764+0000 7f7b39189740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1b2f50 con 0x555fbfb764d0 2024-07-26T11:16:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.764+0000 7f7b2aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb764d0 0x555fbff53480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.764+0000 7f7b2b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 0x555fbff4c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.764+0000 7f7b2b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 0x555fbff4c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57472/0 (socket says 172.21.15.134:57472) 2024-07-26T11:16:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.764+0000 7f7b2b400640 1 -- 172.21.15.134:0/2230760164 learned_addr learned my addr 172.21.15.134:0/2230760164 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.764+0000 7f7b2be00640 1 --2- 172.21.15.134:0/2230760164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb971b0 0x555fbff403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.764+0000 7f7b2aa00640 1 --2- 172.21.15.134:0/2230760164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb764d0 0x555fbff53480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:46270/0 (socket says 172.21.15.134:46270) 2024-07-26T11:16:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.764+0000 7f7b2aa00640 1 -- 172.21.15.134:0/2230760164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb971b0 msgr2=0x555fbff403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.764+0000 7f7b2aa00640 1 --2- 172.21.15.134:0/2230760164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb971b0 0x555fbff403f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.764+0000 7f7b2aa00640 1 -- 172.21.15.134:0/2230760164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 msgr2=0x555fbff4c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.764+0000 7f7b2aa00640 1 --2- 172.21.15.134:0/2230760164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 0x555fbff4c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.765+0000 7f7b2aa00640 1 -- 172.21.15.134:0/2230760164 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555fbff40930 con 0x555fbfb764d0 2024-07-26T11:16:33.768 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.765+0000 7f7b2b400640 1 --2- 172.21.15.134:0/2230760164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 0x555fbff4c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.765+0000 7f7b2aa00640 1 --2- 172.21.15.134:0/2230760164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb764d0 0x555fbff53480 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7b1801b040 tx=0x7f7b18009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf0b9406261aee89 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.765+0000 7f7b17400640 1 -- 172.21.15.134:0/2230760164 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b18005430 con 0x555fbfb764d0 2024-07-26T11:16:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.765+0000 7f7b17400640 1 -- 172.21.15.134:0/2230760164 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b180055d0 con 0x555fbfb764d0 2024-07-26T11:16:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.765+0000 7f7b17400640 1 -- 172.21.15.134:0/2230760164 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b18024410 con 0x555fbfb764d0 2024-07-26T11:16:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.767+0000 7f7b39189740 1 -- 172.21.15.134:0/2230760164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb764d0 msgr2=0x555fbff53480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.767+0000 7f7b39189740 1 --2- 172.21.15.134:0/2230760164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb764d0 0x555fbff53480 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7b1801b040 tx=0x7f7b18009680 comp rx=0 tx=0).stop 2024-07-26T11:16:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.767+0000 7f7b39189740 1 -- 172.21.15.134:0/2230760164 shutdown_connections 2024-07-26T11:16:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.767+0000 7f7b39189740 1 --2- 172.21.15.134:0/2230760164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb764d0 0x555fbff53480 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.767+0000 7f7b39189740 1 --2- 172.21.15.134:0/2230760164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb971b0 0x555fbff403f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.767+0000 7f7b39189740 1 --2- 172.21.15.134:0/2230760164 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 0x555fbff4c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.767+0000 7f7b39189740 1 -- 172.21.15.134:0/2230760164 >> 172.21.15.134:0/2230760164 conn(0x555fbfb8b300 msgr2=0x555fbff3fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:33.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.767+0000 7f7b39189740 1 -- 172.21.15.134:0/2230760164 shutdown_connections 2024-07-26T11:16:33.770 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.767+0000 7f7b39189740 1 -- 172.21.15.134:0/2230760164 wait complete. 2024-07-26T11:16:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.767+0000 7f7b39189740 1 -- start start 2024-07-26T11:16:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.768+0000 7f7b39189740 1 Processor -- start 2024-07-26T11:16:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.768+0000 7f7b39189740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 0x555fbff4d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.768+0000 7f7b39189740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb764d0 0x555fbff4dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.768+0000 7f7b39189740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb971b0 0x555fbff4e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.768+0000 7f7b39189740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1b3cf0 con 0x555fbfb6e2f0 2024-07-26T11:16:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.768+0000 7f7b39189740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1b3620 con 0x555fbfb764d0 2024-07-26T11:16:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.768+0000 7f7b39189740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7b2c1b0860 con 0x555fbfb971b0 2024-07-26T11:16:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.769+0000 7f7b39189740 10 client.0 _ll_register_callbacks cb 0x7f7b1c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:16:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.769+0000 7f7b2b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 0x555fbff4d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.769+0000 7f7b2aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb764d0 0x555fbff4dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.769+0000 7f7b2b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 0x555fbff4d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57486/0 (socket says 172.21.15.134:57486) 2024-07-26T11:16:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.769+0000 7f7b2b400640 1 -- 172.21.15.134:0/268403951 learned_addr learned my addr 172.21.15.134:0/268403951 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.769+0000 7f7b2be00640 1 --2- 172.21.15.134:0/268403951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb971b0 0x555fbff4e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.769+0000 7f7b2aa00640 1 -- 172.21.15.134:0/268403951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb971b0 msgr2=0x555fbff4e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.769+0000 7f7b2aa00640 1 --2- 172.21.15.134:0/268403951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb971b0 0x555fbff4e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.769+0000 7f7b2aa00640 1 -- 172.21.15.134:0/268403951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 msgr2=0x555fbff4d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.769+0000 7f7b2aa00640 1 --2- 172.21.15.134:0/268403951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 0x555fbff4d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.778 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.769+0000 7f7b2aa00640 1 -- 172.21.15.134:0/268403951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555fbffbf150 con 0x555fbfb764d0 2024-07-26T11:16:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.769+0000 7f7b2b400640 1 --2- 172.21.15.134:0/268403951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 0x555fbff4d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.770+0000 7f7b2aa00640 1 --2- 172.21.15.134:0/268403951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb764d0 0x555fbff4dec0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7b18009730 tx=0x7f7b1802b580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bc76202aeca4c6d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.770+0000 7f7b14c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:16:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.770+0000 7f7b14c00640 1 -- 172.21.15.134:0/268403951 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b18039060 con 0x555fbfb764d0 2024-07-26T11:16:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.770+0000 7f7b39189740 1 -- 172.21.15.134:0/268403951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b180055d0 con 0x555fbfb764d0 2024-07-26T11:16:33.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.770+0000 7f7b14c00640 1 -- 172.21.15.134:0/268403951 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7b1802a040 con 0x555fbfb764d0 2024-07-26T11:16:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.770+0000 7f7b14c00640 1 -- 172.21.15.134:0/268403951 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7b18040050 con 0x555fbfb764d0 2024-07-26T11:16:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.770+0000 7f7b39189740 10 client.25580 fetch_fsmap learned FSMap version 30 2024-07-26T11:16:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.770+0000 7f7b39189740 1 -- 172.21.15.134:0/268403951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555fbffc0370 con 0x555fbfb764d0 2024-07-26T11:16:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.771+0000 7f7b14c00640 1 -- 172.21.15.134:0/268403951 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b180055d0 con 0x555fbfb764d0 2024-07-26T11:16:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.771+0000 7f7b14c00640 1 -- 172.21.15.134:0/268403951 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f7b18003aa0 con 0x555fbfb764d0 2024-07-26T11:16:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.771+0000 7f7b39189740 10 client.25580 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:16:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.771+0000 7f7b39189740 10 client.25580 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:16:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.771+0000 7f7b39189740 20 client.25580 populate_metadata read hostname 'smithi134' 2024-07-26T11:16:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.771+0000 7f7b39189740 1 --2- 172.21.15.134:0/268403951 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x555fbfb2b1c0 0x555fbffc2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.771+0000 7f7b39189740 4 client.25580 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:16:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.771+0000 7f7b39189740 1 -- 172.21.15.134:0/268403951 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555fbfeacc60 con 0x555fbfb2b1c0 2024-07-26T11:16:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.771+0000 7f7b2b400640 1 --2- 172.21.15.134:0/268403951 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x555fbfb2b1c0 0x555fbffc2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.772+0000 7f7b2b400640 1 --2- 172.21.15.134:0/268403951 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x555fbfb2b1c0 0x555fbffc2850 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43c751469635de00 server_cookie=a3ec4254483edd9d in_seq=0 out_seq=0 2024-07-26T11:16:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.772+0000 7f7b14c00640 10 client.25580 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.779+0000 7f7b14c00640 1 -- 172.21.15.134:0/268403951 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555fbfeacc60 con 0x555fbfb2b1c0 2024-07-26T11:16:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.779+0000 7f7b14c00640 10 client.25580 handle_command_reply: tid=0 2024-07-26T11:16:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.779+0000 7f7b39189740 1 client.25580 shutdown 2024-07-26T11:16:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.779+0000 7f7b39189740 1 -- 172.21.15.134:0/268403951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb764d0 msgr2=0x555fbff4dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.779+0000 7f7b39189740 1 --2- 172.21.15.134:0/268403951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb764d0 0x555fbff4dec0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7b18009730 tx=0x7f7b1802b580 comp rx=0 tx=0).stop 2024-07-26T11:16:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.779+0000 7f7b39189740 1 -- 172.21.15.134:0/268403951 shutdown_connections 2024-07-26T11:16:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.779+0000 7f7b39189740 1 --2- 172.21.15.134:0/268403951 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x555fbfb2b1c0 0x555fbffc2850 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.779+0000 7f7b39189740 1 --2- 172.21.15.134:0/268403951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555fbfb971b0 0x555fbff4e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.779+0000 7f7b14c00640 0 client.25580 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.779+0000 7f7b39189740 1 --2- 172.21.15.134:0/268403951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555fbfb764d0 0x555fbff4dec0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.779+0000 7f7b39189740 1 --2- 172.21.15.134:0/268403951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555fbfb6e2f0 0x555fbff4d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.779+0000 7f7b39189740 1 -- 172.21.15.134:0/268403951 >> 172.21.15.134:0/268403951 conn(0x555fbfb8b300 msgr2=0x555fbff45dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.779+0000 7f7b39189740 1 -- 172.21.15.134:0/268403951 shutdown_connections 2024-07-26T11:16:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.779+0000 7f7b39189740 1 -- 172.21.15.134:0/268403951 wait complete. 2024-07-26T11:16:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.780+0000 7f7b39189740 20 client.25580 trim_cache size 0 max 16384 2024-07-26T11:16:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:16:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:16:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:16:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:16:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.780+0000 7f7b30c00640 1 -- 172.21.15.134:0/1144076539 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b100b3cb0 msgr2=0x7f7b100b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.780+0000 7f7b30c00640 1 --2- 172.21.15.134:0/1144076539 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b100b3cb0 0x7f7b100b61c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7b24009d60 tx=0x7f7b24009500 comp rx=0 tx=0).stop 2024-07-26T11:16:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.780+0000 7f7b30c00640 1 -- 172.21.15.134:0/1144076539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b2c101820 msgr2=0x7f7b2c19bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.780+0000 7f7b30c00640 1 --2- 172.21.15.134:0/1144076539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b2c101820 0x7f7b2c19bf90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c032fd0 tx=0x7f7b1c031a20 comp rx=0 tx=0).stop 2024-07-26T11:16:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.780+0000 7f7b30c00640 1 -- 172.21.15.134:0/1144076539 shutdown_connections 2024-07-26T11:16:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.780+0000 7f7b30c00640 1 --2- 172.21.15.134:0/1144076539 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7b100b3cb0 0x7f7b100b61c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.780+0000 7f7b30c00640 1 --2- 172.21.15.134:0/1144076539 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b2c105a60 0x7f7b2c19c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.780+0000 7f7b30c00640 1 --2- 172.21.15.134:0/1144076539 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7b2c1063b0 0x7f7b2c1a17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.780+0000 7f7b30c00640 1 --2- 172.21.15.134:0/1144076539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b2c101820 0x7f7b2c19bf90 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.780+0000 7f7b30c00640 1 -- 172.21.15.134:0/1144076539 >> 172.21.15.134:0/1144076539 conn(0x7f7b2c079530 msgr2=0x7f7b2c106ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.781+0000 7f7b30c00640 1 -- 172.21.15.134:0/1144076539 shutdown_connections 2024-07-26T11:16:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:33.781+0000 7f7b30c00640 1 -- 172.21.15.134:0/1144076539 wait complete. 2024-07-26T11:16:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:34 smithi134 ceph-mon[31966]: pgmap v498: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 14 op/s 2024-07-26T11:16:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:34 smithi189 ceph-mon[36790]: pgmap v498: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 14 op/s 2024-07-26T11:16:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:34 smithi193 ceph-mon[37473]: pgmap v498: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.3 KiB/s wr, 14 op/s 2024-07-26T11:16:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:36 smithi134 ceph-mon[31966]: pgmap v499: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T11:16:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:36 smithi189 ceph-mon[36790]: pgmap v499: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T11:16:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:36 smithi193 ceph-mon[37473]: pgmap v499: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T11:16:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:38 smithi134 ceph-mon[31966]: pgmap v500: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 426 B/s wr, 12 op/s 2024-07-26T11:16:38.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:38 smithi189 ceph-mon[36790]: pgmap v500: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 426 B/s wr, 12 op/s 2024-07-26T11:16:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:38 smithi193 ceph-mon[37473]: pgmap v500: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 426 B/s wr, 12 op/s 2024-07-26T11:16:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:40 smithi134 ceph-mon[31966]: pgmap v501: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T11:16:40.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:40 smithi189 ceph-mon[36790]: pgmap v501: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T11:16:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:40 smithi193 ceph-mon[37473]: pgmap v501: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T11:16:42.053 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:16:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:42 smithi134 ceph-mon[31966]: pgmap v502: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T11:16:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:42 smithi189 ceph-mon[36790]: pgmap v502: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T11:16:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:42 smithi193 ceph-mon[37473]: pgmap v502: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-07-26T11:16:42.596 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:16:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:44 smithi134 ceph-mon[31966]: pgmap v503: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T11:16:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:44 smithi189 ceph-mon[36790]: pgmap v503: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T11:16:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:44 smithi193 ceph-mon[37473]: pgmap v503: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-07-26T11:16:44.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.979+0000 7fb19fe00640 1 -- 172.21.15.134:0/2735224180 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a0104ab0 msgr2=0x7fb1a010d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:44.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.979+0000 7fb19fe00640 1 --2- 172.21.15.134:0/2735224180 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a0104ab0 0x7fb1a010d0f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009900 tx=0x7fb18c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:16:44.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.980+0000 7fb19fe00640 1 -- 172.21.15.134:0/2735224180 shutdown_connections 2024-07-26T11:16:44.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.980+0000 7fb19fe00640 1 --2- 172.21.15.134:0/2735224180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1a010d630 0x7fb1a010fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:44.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.980+0000 7fb19fe00640 1 --2- 172.21.15.134:0/2735224180 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a0104ab0 0x7fb1a010d0f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:44.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.980+0000 7fb19fe00640 1 --2- 172.21.15.134:0/2735224180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1a0104080 0x7fb1a0104490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:44.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.980+0000 7fb19fe00640 1 -- 172.21.15.134:0/2735224180 >> 172.21.15.134:0/2735224180 conn(0x7fb1a00fdcf0 msgr2=0x7fb1a0100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:44.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.980+0000 7fb19fe00640 1 -- 172.21.15.134:0/2735224180 shutdown_connections 2024-07-26T11:16:44.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.980+0000 7fb19fe00640 1 -- 172.21.15.134:0/2735224180 wait complete. 2024-07-26T11:16:44.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.982+0000 7fb19fe00640 1 Processor -- start 2024-07-26T11:16:44.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.982+0000 7fb19fe00640 1 -- start start 2024-07-26T11:16:44.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.982+0000 7fb19fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1a0104080 0x7fb1a0101f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:44.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.982+0000 7fb19fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1a0104ab0 0x7fb1a01024d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:44.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.982+0000 7fb19fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a010d630 0x7fb1a0102a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:44.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.982+0000 7fb19fe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a01132d0 con 0x7fb1a010d630 2024-07-26T11:16:44.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.982+0000 7fb19fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a0113150 con 0x7fb1a0104ab0 2024-07-26T11:16:44.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.982+0000 7fb19fe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a0113450 con 0x7fb1a0104080 2024-07-26T11:16:44.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.983+0000 7fb19ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1a0104080 0x7fb1a0101f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:44.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.983+0000 7fb19e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1a0104ab0 0x7fb1a01024d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:44.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.983+0000 7fb19ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1a0104080 0x7fb1a0101f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57868/0 (socket says 172.21.15.134:57868) 2024-07-26T11:16:44.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.983+0000 7fb19ea00640 1 -- 172.21.15.134:0/581782590 learned_addr learned my addr 172.21.15.134:0/581782590 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:44.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.983+0000 7fb19f400640 1 --2- 172.21.15.134:0/581782590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a010d630 0x7fb1a0102a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:44.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.983+0000 7fb19e000640 1 -- 172.21.15.134:0/581782590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1a0104080 msgr2=0x7fb1a0101f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:44.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.983+0000 7fb19e000640 1 --2- 172.21.15.134:0/581782590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1a0104080 0x7fb1a0101f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:44.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.984+0000 7fb19e000640 1 -- 172.21.15.134:0/581782590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a010d630 msgr2=0x7fb1a0102a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:44.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.984+0000 7fb19e000640 1 --2- 172.21.15.134:0/581782590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a010d630 0x7fb1a0102a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:44.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.984+0000 7fb19e000640 1 -- 172.21.15.134:0/581782590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a01a7fc0 con 0x7fb1a0104ab0 2024-07-26T11:16:44.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.984+0000 7fb19f400640 1 --2- 172.21.15.134:0/581782590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a010d630 0x7fb1a0102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:16:44.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.984+0000 7fb19e000640 1 --2- 172.21.15.134:0/581782590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1a0104ab0 0x7fb1a01024d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009230 tx=0x7fb18c003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:44.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.984+0000 7fb197400640 1 -- 172.21.15.134:0/581782590 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18c03f070 con 0x7fb1a0104ab0 2024-07-26T11:16:44.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.985+0000 7fb19fe00640 1 -- 172.21.15.134:0/581782590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a01a8250 con 0x7fb1a0104ab0 2024-07-26T11:16:44.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.985+0000 7fb19fe00640 1 -- 172.21.15.134:0/581782590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a01a8750 con 0x7fb1a0104ab0 2024-07-26T11:16:44.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.985+0000 7fb197400640 1 -- 172.21.15.134:0/581782590 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb18c031b50 con 0x7fb1a0104ab0 2024-07-26T11:16:44.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.985+0000 7fb197400640 1 -- 172.21.15.134:0/581782590 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18c035030 con 0x7fb1a0104ab0 2024-07-26T11:16:44.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.988+0000 7fb1ac4bd740 1 Processor -- start 2024-07-26T11:16:44.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.989+0000 7fb1ac4bd740 1 -- start start 2024-07-26T11:16:44.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.989+0000 7fb1ac4bd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921f2070 0x562792597000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:44.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.989+0000 7fb1ac4bd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921c91b0 0x562792597540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:44.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.989+0000 7fb1ac4bd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921d1390 0x562792597a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:44.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.989+0000 7fb1ac4bd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56279250dbc0 con 0x5627921f2070 2024-07-26T11:16:44.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.989+0000 7fb1ac4bd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5627924fd0c0 con 0x5627921c91b0 2024-07-26T11:16:44.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.989+0000 7fb1ac4bd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562792501100 con 0x5627921d1390 2024-07-26T11:16:44.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.990+0000 7fb19ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921c91b0 0x562792597540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:44.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.990+0000 7fb19ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921c91b0 0x562792597540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42266/0 (socket says 172.21.15.134:42266) 2024-07-26T11:16:44.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.990+0000 7fb19ea00640 1 -- 172.21.15.134:0/261230864 learned_addr learned my addr 172.21.15.134:0/261230864 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:44.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.990+0000 7fb19f400640 1 --2- 172.21.15.134:0/261230864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921f2070 0x562792597000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:44.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.990+0000 7fb19ea00640 1 -- 172.21.15.134:0/261230864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921d1390 msgr2=0x562792597a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:16:44.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.990+0000 7fb19ea00640 1 --2- 172.21.15.134:0/261230864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921d1390 0x562792597a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:44.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.990+0000 7fb19ea00640 1 -- 172.21.15.134:0/261230864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921f2070 msgr2=0x562792597000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:44.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.990+0000 7fb197400640 1 -- 172.21.15.134:0/581782590 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7fb18c042070 con 0x7fb1a0104ab0 2024-07-26T11:16:44.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.991+0000 7fb19ea00640 1 --2- 172.21.15.134:0/261230864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921f2070 0x562792597000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:44.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.991+0000 7fb19ea00640 1 -- 172.21.15.134:0/261230864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56279225d650 con 0x5627921c91b0 2024-07-26T11:16:44.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.992+0000 7fb19f400640 1 --2- 172.21.15.134:0/261230864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921f2070 0x562792597000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:16:44.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.992+0000 7fb197400640 1 --2- 172.21.15.134:0/581782590 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1840b3d30 0x7fb1840b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:44.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.992+0000 7fb197400640 1 -- 172.21.15.134:0/581782590 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb18c1049e0 con 0x7fb1a0104ab0 2024-07-26T11:16:44.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.992+0000 7fb19ea00640 1 --2- 172.21.15.134:0/261230864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921c91b0 0x562792597540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb190009d70 tx=0x7fb190011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79f6438f45c3bb54 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:44.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.993+0000 7fb18be00640 1 -- 172.21.15.134:0/261230864 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19001e070 con 0x5627921c91b0 2024-07-26T11:16:44.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.993+0000 7fb18be00640 1 -- 172.21.15.134:0/261230864 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb190019ae0 con 0x5627921c91b0 2024-07-26T11:16:44.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.993+0000 7fb18be00640 1 -- 172.21.15.134:0/261230864 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19000eb40 con 0x5627921c91b0 2024-07-26T11:16:44.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.993+0000 7fb19f400640 1 --2- 172.21.15.134:0/581782590 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1840b3d30 0x7fb1840b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:44.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.994+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/261230864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921c91b0 msgr2=0x562792597540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:44.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.994+0000 7fb19f400640 1 --2- 172.21.15.134:0/581782590 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1840b3d30 0x7fb1840b6240 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb198005fd0 tx=0x7fb198009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:44.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.994+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/261230864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921c91b0 0x562792597540 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb190009d70 tx=0x7fb190011810 comp rx=0 tx=0).stop 2024-07-26T11:16:44.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.994+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/261230864 shutdown_connections 2024-07-26T11:16:44.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.994+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/261230864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921d1390 0x562792597a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:44.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.994+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/261230864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921c91b0 0x562792597540 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:44.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.994+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/261230864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921f2070 0x562792597000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:44.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.994+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/261230864 >> 172.21.15.134:0/261230864 conn(0x5627921e6460 msgr2=0x562792594ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:44.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.994+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/261230864 shutdown_connections 2024-07-26T11:16:44.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.994+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/261230864 wait complete. 2024-07-26T11:16:45.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.995+0000 7fb1ac4bd740 1 -- start start 2024-07-26T11:16:45.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.995+0000 7fb1ac4bd740 1 Processor -- start 2024-07-26T11:16:45.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.996+0000 7fb1ac4bd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921c91b0 0x56279259b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:45.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.996+0000 7fb1ac4bd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921d1390 0x56279259bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:45.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.996+0000 7fb1ac4bd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921f2070 0x5627925a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:45.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.996+0000 7fb1ac4bd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562792360700 con 0x5627921d1390 2024-07-26T11:16:45.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.996+0000 7fb1ac4bd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562792499690 con 0x5627921f2070 2024-07-26T11:16:45.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.996+0000 7fb1ac4bd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5627925a57e0 con 0x5627921c91b0 2024-07-26T11:16:45.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.996+0000 7fb1ac4bd740 10 client.0 _ll_register_callbacks cb 0x5627924a9140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:16:45.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.996+0000 7fb19ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921c91b0 0x56279259b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:45.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.996+0000 7fb19ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921c91b0 0x56279259b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57876/0 (socket says 172.21.15.134:57876) 2024-07-26T11:16:45.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.996+0000 7fb19ea00640 1 -- 172.21.15.134:0/4177218528 learned_addr learned my addr 172.21.15.134:0/4177218528 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:45.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.996+0000 7fb19f400640 1 --2- 172.21.15.134:0/4177218528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921f2070 0x5627925a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:45.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.997+0000 7fb19ea00640 1 -- 172.21.15.134:0/4177218528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921f2070 msgr2=0x5627925a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:45.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.997+0000 7fb19ea00640 1 --2- 172.21.15.134:0/4177218528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921f2070 0x5627925a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.997+0000 7fb19ea00640 1 -- 172.21.15.134:0/4177218528 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921d1390 msgr2=0x56279259bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:45.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.997+0000 7fb19ea00640 1 --2- 172.21.15.134:0/4177218528 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921d1390 0x56279259bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.997+0000 7fb19ea00640 1 -- 172.21.15.134:0/4177218528 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627924b0830 con 0x5627921c91b0 2024-07-26T11:16:45.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.997+0000 7fb19ea00640 1 --2- 172.21.15.134:0/4177218528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921c91b0 0x56279259b670 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb190009a90 tx=0x7fb190009ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e60f84d1f17798d3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:45.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.997+0000 7fb188c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:16:45.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.997+0000 7fb188c00640 1 -- 172.21.15.134:0/4177218528 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb190029020 con 0x5627921c91b0 2024-07-26T11:16:45.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.997+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/4177218528 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb190019ae0 con 0x5627921c91b0 2024-07-26T11:16:45.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.998+0000 7fb188c00640 1 -- 172.21.15.134:0/4177218528 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb190004380 con 0x5627921c91b0 2024-07-26T11:16:45.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.998+0000 7fb188c00640 1 -- 172.21.15.134:0/4177218528 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb190034050 con 0x5627921c91b0 2024-07-26T11:16:45.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.998+0000 7fb1ac4bd740 10 client.25603 fetch_fsmap learned FSMap version 30 2024-07-26T11:16:45.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.998+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/4177218528 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5627925ab310 con 0x5627921c91b0 2024-07-26T11:16:45.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.998+0000 7fb188c00640 1 -- 172.21.15.134:0/4177218528 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb190019ae0 con 0x5627921c91b0 2024-07-26T11:16:45.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.999+0000 7fb188c00640 1 -- 172.21.15.134:0/4177218528 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fb190019430 con 0x5627921c91b0 2024-07-26T11:16:45.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.999+0000 7fb1ac4bd740 10 client.25603 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:16:45.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.999+0000 7fb1ac4bd740 10 client.25603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:16:45.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.999+0000 7fb1ac4bd740 20 client.25603 populate_metadata read hostname 'smithi134' 2024-07-26T11:16:45.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.999+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/4177218528 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5627921b09f0 0x5627925ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:45.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.999+0000 7fb1ac4bd740 4 client.25603 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:16:45.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:44.999+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/4177218528 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56279250fb10 con 0x5627921b09f0 2024-07-26T11:16:45.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.000+0000 7fb19e000640 1 --2- 172.21.15.134:0/4177218528 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5627921b09f0 0x5627925ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:45.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.000+0000 7fb19e000640 1 --2- 172.21.15.134:0/4177218528 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5627921b09f0 0x5627925ae6d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b01883a9729fc99 server_cookie=989bc778541cba6a in_seq=0 out_seq=0 2024-07-26T11:16:45.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.000+0000 7fb188c00640 10 client.25603 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:45.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.001+0000 7fb188c00640 1 -- 172.21.15.134:0/4177218528 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56279250fb10 con 0x5627921b09f0 2024-07-26T11:16:45.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.001+0000 7fb188c00640 10 client.25603 handle_command_reply: tid=0 2024-07-26T11:16:45.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.001+0000 7fb1ac4bd740 1 client.25603 shutdown 2024-07-26T11:16:45.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.002+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/4177218528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921c91b0 msgr2=0x56279259b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:45.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.002+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/4177218528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921c91b0 0x56279259b670 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb190009a90 tx=0x7fb190009ac0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.002+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/4177218528 shutdown_connections 2024-07-26T11:16:45.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.002+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/4177218528 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5627921b09f0 0x5627925ae6d0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.002+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/4177218528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921f2070 0x5627925a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.002+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/4177218528 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921d1390 0x56279259bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.002+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/4177218528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921c91b0 0x56279259b670 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.002+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/4177218528 >> 172.21.15.134:0/4177218528 conn(0x5627921e6460 msgr2=0x56279259a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:45.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.002+0000 7fb188c00640 0 client.25603 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:45.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.002+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/4177218528 shutdown_connections 2024-07-26T11:16:45.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.002+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/4177218528 wait complete. 2024-07-26T11:16:45.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.002+0000 7fb1ac4bd740 20 client.25603 trim_cache size 0 max 16384 2024-07-26T11:16:45.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.021+0000 7fb1ac4bd740 1 Processor -- start 2024-07-26T11:16:45.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.022+0000 7fb1ac4bd740 1 -- start start 2024-07-26T11:16:45.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.022+0000 7fb1ac4bd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921c91b0 0x562792593ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:45.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.022+0000 7fb1ac4bd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921d1390 0x5627925a6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:45.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.022+0000 7fb1ac4bd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921f2070 0x562792598740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:45.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.022+0000 7fb1ac4bd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb18000ff10 con 0x5627921c91b0 2024-07-26T11:16:45.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.022+0000 7fb1ac4bd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb180011a50 con 0x5627921f2070 2024-07-26T11:16:45.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.022+0000 7fb1ac4bd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb1800105e0 con 0x5627921d1390 2024-07-26T11:16:45.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.023+0000 7fb19f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921f2070 0x562792598740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:45.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.023+0000 7fb19e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921d1390 0x5627925a6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:45.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.023+0000 7fb19f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921f2070 0x562792598740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42292/0 (socket says 172.21.15.134:42292) 2024-07-26T11:16:45.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.023+0000 7fb19f400640 1 -- 172.21.15.134:0/3042649070 learned_addr learned my addr 172.21.15.134:0/3042649070 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:45.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.023+0000 7fb19ea00640 1 --2- 172.21.15.134:0/3042649070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921c91b0 0x562792593ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:45.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.023+0000 7fb19f400640 1 -- 172.21.15.134:0/3042649070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921d1390 msgr2=0x5627925a6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:45.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.023+0000 7fb19f400640 1 --2- 172.21.15.134:0/3042649070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921d1390 0x5627925a6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.023+0000 7fb19f400640 1 -- 172.21.15.134:0/3042649070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921c91b0 msgr2=0x562792593ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:45.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.023+0000 7fb19f400640 1 --2- 172.21.15.134:0/3042649070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921c91b0 0x562792593ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.023+0000 7fb19f400640 1 -- 172.21.15.134:0/3042649070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562792595430 con 0x5627921f2070 2024-07-26T11:16:45.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.023+0000 7fb19ea00640 1 --2- 172.21.15.134:0/3042649070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921c91b0 0x562792593ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:16:45.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.024+0000 7fb19f400640 1 --2- 172.21.15.134:0/3042649070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921f2070 0x562792598740 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb198012af0 tx=0x7fb198014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71ce027450d01028 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:45.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.024+0000 7fb18b400640 1 -- 172.21.15.134:0/3042649070 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb198020c70 con 0x5627921f2070 2024-07-26T11:16:45.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.024+0000 7fb18b400640 1 -- 172.21.15.134:0/3042649070 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb198020e10 con 0x5627921f2070 2024-07-26T11:16:45.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.024+0000 7fb18b400640 1 -- 172.21.15.134:0/3042649070 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19801b7c0 con 0x5627921f2070 2024-07-26T11:16:45.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.025+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/3042649070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921f2070 msgr2=0x562792598740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:45.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.025+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/3042649070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921f2070 0x562792598740 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb198012af0 tx=0x7fb198014000 comp rx=0 tx=0).stop 2024-07-26T11:16:45.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.025+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/3042649070 shutdown_connections 2024-07-26T11:16:45.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.025+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/3042649070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921d1390 0x5627925a6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.025+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/3042649070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921f2070 0x562792598740 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.025+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/3042649070 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921c91b0 0x562792593ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.025+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/3042649070 >> 172.21.15.134:0/3042649070 conn(0x5627921e6460 msgr2=0x562792598030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:45.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.025+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/3042649070 shutdown_connections 2024-07-26T11:16:45.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.026+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/3042649070 wait complete. 2024-07-26T11:16:45.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.026+0000 7fb1ac4bd740 1 -- start start 2024-07-26T11:16:45.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.026+0000 7fb1ac4bd740 1 Processor -- start 2024-07-26T11:16:45.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.027+0000 7fb1ac4bd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921c91b0 0x5627925bb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:45.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.027+0000 7fb1ac4bd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921d1390 0x5627925bb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:45.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.027+0000 7fb1ac4bd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921f2070 0x5627925bbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:45.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.027+0000 7fb1ac4bd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb180011380 con 0x5627921f2070 2024-07-26T11:16:45.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.027+0000 7fb1ac4bd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb180010cb0 con 0x5627921c91b0 2024-07-26T11:16:45.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.027+0000 7fb1ac4bd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb18000def0 con 0x5627921d1390 2024-07-26T11:16:45.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.027+0000 7fb1ac4bd740 10 client.0 _ll_register_callbacks cb 0x7fb190009bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:16:45.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.027+0000 7fb19ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921c91b0 0x5627925bb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:45.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.027+0000 7fb19f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921f2070 0x5627925bbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:45.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.027+0000 7fb19ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921c91b0 0x5627925bb420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42296/0 (socket says 172.21.15.134:42296) 2024-07-26T11:16:45.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.027+0000 7fb19ea00640 1 -- 172.21.15.134:0/3429428825 learned_addr learned my addr 172.21.15.134:0/3429428825 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:45.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.028+0000 7fb19e000640 1 --2- 172.21.15.134:0/3429428825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921d1390 0x5627925bb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:45.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.028+0000 7fb19ea00640 1 -- 172.21.15.134:0/3429428825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921d1390 msgr2=0x5627925bb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:45.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.028+0000 7fb19ea00640 1 --2- 172.21.15.134:0/3429428825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921d1390 0x5627925bb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.028+0000 7fb19ea00640 1 -- 172.21.15.134:0/3429428825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921f2070 msgr2=0x5627925bbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:45.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.028+0000 7fb19ea00640 1 --2- 172.21.15.134:0/3429428825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921f2070 0x5627925bbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.028+0000 7fb19ea00640 1 -- 172.21.15.134:0/3429428825 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627925c0be0 con 0x5627921c91b0 2024-07-26T11:16:45.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.028+0000 7fb19ea00640 1 --2- 172.21.15.134:0/3429428825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921c91b0 0x5627925bb420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb190009250 tx=0x7fb190009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e504f8453e7ecfff server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:45.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.028+0000 7fb188c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:16:45.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.029+0000 7fb188c00640 1 -- 172.21.15.134:0/3429428825 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb190029040 con 0x5627921c91b0 2024-07-26T11:16:45.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.029+0000 7fb188c00640 1 -- 172.21.15.134:0/3429428825 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb190005200 con 0x5627921c91b0 2024-07-26T11:16:45.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.029+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/3429428825 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb198020e10 con 0x5627921c91b0 2024-07-26T11:16:45.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.029+0000 7fb188c00640 1 -- 172.21.15.134:0/3429428825 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fb190045050 con 0x5627921c91b0 2024-07-26T11:16:45.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.029+0000 7fb1ac4bd740 10 client.25598 fetch_fsmap learned FSMap version 30 2024-07-26T11:16:45.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.029+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/3429428825 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56279263e900 con 0x5627921c91b0 2024-07-26T11:16:45.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.030+0000 7fb188c00640 1 -- 172.21.15.134:0/3429428825 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb198020e10 con 0x5627921c91b0 2024-07-26T11:16:45.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.030+0000 7fb188c00640 1 -- 172.21.15.134:0/3429428825 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fb190040030 con 0x5627921c91b0 2024-07-26T11:16:45.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.030+0000 7fb1ac4bd740 10 client.25598 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:16:45.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.030+0000 7fb1ac4bd740 10 client.25598 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:16:45.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.030+0000 7fb1ac4bd740 20 client.25598 populate_metadata read hostname 'smithi134' 2024-07-26T11:16:45.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.030+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/3429428825 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562792185da0 0x562792641140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:45.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.030+0000 7fb1ac4bd740 4 client.25598 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:16:45.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.030+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/3429428825 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56279250b5b0 con 0x562792185da0 2024-07-26T11:16:45.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.031+0000 7fb19e000640 1 --2- 172.21.15.134:0/3429428825 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562792185da0 0x562792641140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:45.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.032+0000 7fb19e000640 1 --2- 172.21.15.134:0/3429428825 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562792185da0 0x562792641140 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=509f4ca25244a7c5 server_cookie=13073a84b3090cf8 in_seq=0 out_seq=0 2024-07-26T11:16:45.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.032+0000 7fb188c00640 10 client.25598 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:45.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.032+0000 7fb188c00640 1 -- 172.21.15.134:0/3429428825 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56279250b5b0 con 0x562792185da0 2024-07-26T11:16:45.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.032+0000 7fb188c00640 10 client.25598 handle_command_reply: tid=0 2024-07-26T11:16:45.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.032+0000 7fb1ac4bd740 1 client.25598 shutdown 2024-07-26T11:16:45.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.033+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/3429428825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921c91b0 msgr2=0x5627925bb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:45.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.033+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/3429428825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921c91b0 0x5627925bb420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb190009250 tx=0x7fb190009700 comp rx=0 tx=0).stop 2024-07-26T11:16:45.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.033+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/3429428825 shutdown_connections 2024-07-26T11:16:45.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.033+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/3429428825 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562792185da0 0x562792641140 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.033+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/3429428825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627921d1390 0x5627925bb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.033+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/3429428825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627921f2070 0x5627925bbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.033+0000 7fb1ac4bd740 1 --2- 172.21.15.134:0/3429428825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627921c91b0 0x5627925bb420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.033+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/3429428825 >> 172.21.15.134:0/3429428825 conn(0x5627921e6460 msgr2=0x562792593cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:45.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.033+0000 7fb188c00640 0 client.25598 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:45.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.034+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/3429428825 shutdown_connections 2024-07-26T11:16:45.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.034+0000 7fb1ac4bd740 1 -- 172.21.15.134:0/3429428825 wait complete. 2024-07-26T11:16:45.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.034+0000 7fb1ac4bd740 20 client.25598 trim_cache size 0 max 16384 2024-07-26T11:16:45.037 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:16:45.037 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "no active scrubs running", 2024-07-26T11:16:45.037 INFO:teuthology.orchestra.run.smithi134.stdout: "scrubs": {} 2024-07-26T11:16:45.037 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:16:45.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.035+0000 7fb19fe00640 1 -- 172.21.15.134:0/581782590 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1840b3d30 msgr2=0x7fb1840b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:45.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.035+0000 7fb19fe00640 1 --2- 172.21.15.134:0/581782590 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1840b3d30 0x7fb1840b6240 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb198005fd0 tx=0x7fb198009500 comp rx=0 tx=0).stop 2024-07-26T11:16:45.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.035+0000 7fb19fe00640 1 -- 172.21.15.134:0/581782590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1a0104ab0 msgr2=0x7fb1a01024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:45.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.035+0000 7fb19fe00640 1 --2- 172.21.15.134:0/581782590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1a0104ab0 0x7fb1a01024d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009230 tx=0x7fb18c003b90 comp rx=0 tx=0).stop 2024-07-26T11:16:45.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.035+0000 7fb19fe00640 1 -- 172.21.15.134:0/581782590 shutdown_connections 2024-07-26T11:16:45.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.035+0000 7fb19fe00640 1 --2- 172.21.15.134:0/581782590 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1840b3d30 0x7fb1840b6240 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.035+0000 7fb19fe00640 1 --2- 172.21.15.134:0/581782590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1a010d630 0x7fb1a0102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.035+0000 7fb19fe00640 1 --2- 172.21.15.134:0/581782590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1a0104ab0 0x7fb1a01024d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.035+0000 7fb19fe00640 1 --2- 172.21.15.134:0/581782590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1a0104080 0x7fb1a0101f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:45.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.035+0000 7fb19fe00640 1 -- 172.21.15.134:0/581782590 >> 172.21.15.134:0/581782590 conn(0x7fb1a00fdcf0 msgr2=0x7fb1a01071d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:45.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.036+0000 7fb19fe00640 1 -- 172.21.15.134:0/581782590 shutdown_connections 2024-07-26T11:16:45.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:45.036+0000 7fb19fe00640 1 -- 172.21.15.134:0/581782590 wait complete. 2024-07-26T11:16:46.000 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-26T11:16:46.001 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:16:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:46 smithi134 ceph-mon[31966]: pgmap v504: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-26T11:16:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:46 smithi189 ceph-mon[36790]: pgmap v504: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-26T11:16:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:46 smithi193 ceph-mon[37473]: pgmap v504: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-26T11:16:46.555 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:16:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.020+0000 7f659be00640 1 -- 172.21.15.134:0/1766106641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f659c103340 msgr2=0x7f659c103750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.020+0000 7f659be00640 1 --2- 172.21.15.134:0/1766106641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f659c103340 0x7f659c103750 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f658c009900 tx=0x7f658c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.021+0000 7f659be00640 1 -- 172.21.15.134:0/1766106641 shutdown_connections 2024-07-26T11:16:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.021+0000 7f659be00640 1 --2- 172.21.15.134:0/1766106641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f659c1095c0 0x7f659c10fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.021+0000 7f659be00640 1 --2- 172.21.15.134:0/1766106641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f659c108c70 0x7f659c109080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.021+0000 7f659be00640 1 --2- 172.21.15.134:0/1766106641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f659c103340 0x7f659c103750 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.021+0000 7f659be00640 1 -- 172.21.15.134:0/1766106641 >> 172.21.15.134:0/1766106641 conn(0x7f659c0fed60 msgr2=0x7f659c1011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.021+0000 7f659be00640 1 -- 172.21.15.134:0/1766106641 shutdown_connections 2024-07-26T11:16:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.021+0000 7f659be00640 1 -- 172.21.15.134:0/1766106641 wait complete. 2024-07-26T11:16:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.022+0000 7f659be00640 1 Processor -- start 2024-07-26T11:16:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.023+0000 7f659be00640 1 -- start start 2024-07-26T11:16:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.023+0000 7f659be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f659c103340 0x7f659c075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.023+0000 7f659be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f659c108c70 0x7f659c075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.023+0000 7f659be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f659c1095c0 0x7f659c075bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.023+0000 7f659be00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f659c1132c0 con 0x7f659c108c70 2024-07-26T11:16:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.023+0000 7f659be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f659c113140 con 0x7f659c103340 2024-07-26T11:16:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.023+0000 7f659be00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f659c113440 con 0x7f659c1095c0 2024-07-26T11:16:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.024+0000 7f659aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f659c103340 0x7f659c075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.024+0000 7f659aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f659c103340 0x7f659c075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42310/0 (socket says 172.21.15.134:42310) 2024-07-26T11:16:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.024+0000 7f659aa00640 1 -- 172.21.15.134:0/4230458921 learned_addr learned my addr 172.21.15.134:0/4230458921 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.024+0000 7f659a000640 1 --2- 172.21.15.134:0/4230458921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f659c108c70 0x7f659c075680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.024+0000 7f659b400640 1 --2- 172.21.15.134:0/4230458921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f659c1095c0 0x7f659c075bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.025+0000 7f659aa00640 1 -- 172.21.15.134:0/4230458921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f659c1095c0 msgr2=0x7f659c075bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.025+0000 7f659aa00640 1 --2- 172.21.15.134:0/4230458921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f659c1095c0 0x7f659c075bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.025+0000 7f659aa00640 1 -- 172.21.15.134:0/4230458921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f659c108c70 msgr2=0x7f659c075680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.025+0000 7f659aa00640 1 --2- 172.21.15.134:0/4230458921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f659c108c70 0x7f659c075680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.025+0000 7f659aa00640 1 -- 172.21.15.134:0/4230458921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f659c1a7fd0 con 0x7f659c103340 2024-07-26T11:16:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.025+0000 7f659aa00640 1 --2- 172.21.15.134:0/4230458921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f659c103340 0x7f659c075140 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f658c005ab0 tx=0x7f658c003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.026+0000 7f6593400640 1 -- 172.21.15.134:0/4230458921 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658c03f070 con 0x7f659c103340 2024-07-26T11:16:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.026+0000 7f6593400640 1 -- 172.21.15.134:0/4230458921 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f658c031b60 con 0x7f659c103340 2024-07-26T11:16:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.026+0000 7f6593400640 1 -- 172.21.15.134:0/4230458921 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f658c035030 con 0x7f659c103340 2024-07-26T11:16:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.026+0000 7f659be00640 1 -- 172.21.15.134:0/4230458921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f659c1a8260 con 0x7f659c103340 2024-07-26T11:16:47.029 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.026+0000 7f659be00640 1 -- 172.21.15.134:0/4230458921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f659c1a8700 con 0x7f659c103340 2024-07-26T11:16:47.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.031+0000 7f65a8dec740 1 Processor -- start 2024-07-26T11:16:47.034 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.032+0000 7f6593400640 1 -- 172.21.15.134:0/4230458921 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7f658c042070 con 0x7f659c103340 2024-07-26T11:16:47.035 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.032+0000 7f65a8dec740 1 -- start start 2024-07-26T11:16:47.035 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.032+0000 7f65a8dec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a20307a1b0 0x55a203426450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.035 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.033+0000 7f65a8dec740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030512f0 0x55a203426990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.035 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.033+0000 7f65a8dec740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a2030594d0 0x55a203426ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.035 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.033+0000 7f65a8dec740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a203392270 con 0x55a2030594d0 2024-07-26T11:16:47.035 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.033+0000 7f65a8dec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a203381770 con 0x55a20307a1b0 2024-07-26T11:16:47.035 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.033+0000 7f65a8dec740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a2033857b0 con 0x55a2030512f0 2024-07-26T11:16:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.033+0000 7f659aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030512f0 0x55a203426990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.033+0000 7f659b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a20307a1b0 0x55a203426450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.033+0000 7f659aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030512f0 0x55a203426990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57916/0 (socket says 172.21.15.134:57916) 2024-07-26T11:16:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.033+0000 7f659aa00640 1 -- 172.21.15.134:0/3152836456 learned_addr learned my addr 172.21.15.134:0/3152836456 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.034+0000 7f659b400640 1 -- 172.21.15.134:0/3152836456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030512f0 msgr2=0x55a203426990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.034+0000 7f659b400640 1 --2- 172.21.15.134:0/3152836456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030512f0 0x55a203426990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.034+0000 7f659b400640 1 -- 172.21.15.134:0/3152836456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a2030594d0 msgr2=0x55a203426ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:16:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.034+0000 7f6593400640 1 --2- 172.21.15.134:0/4230458921 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65800b3d60 0x7f65800b6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.035+0000 7f6593400640 1 -- 172.21.15.134:0/4230458921 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f658c104a50 con 0x7f659c103340 2024-07-26T11:16:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.035+0000 7f659a000640 1 --2- 172.21.15.134:0/3152836456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a2030594d0 0x55a203426ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.035+0000 7f659b400640 1 --2- 172.21.15.134:0/3152836456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a2030594d0 0x55a203426ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.037 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.035+0000 7f659b400640 1 -- 172.21.15.134:0/3152836456 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2030e5790 con 0x55a20307a1b0 2024-07-26T11:16:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.035+0000 7f659aa00640 1 --2- 172.21.15.134:0/3152836456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030512f0 0x55a203426990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.035+0000 7f659a000640 1 --2- 172.21.15.134:0/3152836456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a2030594d0 0x55a203426ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:16:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.036+0000 7f659aa00640 1 --2- 172.21.15.134:0/4230458921 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65800b3d60 0x7f65800b6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.036+0000 7f659b400640 1 --2- 172.21.15.134:0/3152836456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a20307a1b0 0x55a203426450 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6594009f30 tx=0x7f65940118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8343ed8db4346597 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:47.038 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.036+0000 7f6587e00640 1 -- 172.21.15.134:0/3152836456 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6594004d20 con 0x55a20307a1b0 2024-07-26T11:16:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.036+0000 7f6587e00640 1 -- 172.21.15.134:0/3152836456 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6594004050 con 0x55a20307a1b0 2024-07-26T11:16:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.036+0000 7f6587e00640 1 -- 172.21.15.134:0/3152836456 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6594005280 con 0x55a20307a1b0 2024-07-26T11:16:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.037+0000 7f65a8dec740 1 -- 172.21.15.134:0/3152836456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a20307a1b0 msgr2=0x55a203426450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.037+0000 7f65a8dec740 1 --2- 172.21.15.134:0/3152836456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a20307a1b0 0x55a203426450 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6594009f30 tx=0x7f65940118d0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.039 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.037+0000 7f659aa00640 1 --2- 172.21.15.134:0/4230458921 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65800b3d60 0x7f65800b6270 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f658c02a9b0 tx=0x7f658c1084a0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.038+0000 7f65a8dec740 1 -- 172.21.15.134:0/3152836456 shutdown_connections 2024-07-26T11:16:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.038+0000 7f65a8dec740 1 --2- 172.21.15.134:0/3152836456 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030512f0 0x55a203426990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.038+0000 7f65a8dec740 1 --2- 172.21.15.134:0/3152836456 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a2030594d0 0x55a203426ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.038+0000 7f65a8dec740 1 --2- 172.21.15.134:0/3152836456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a20307a1b0 0x55a203426450 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.038+0000 7f65a8dec740 1 -- 172.21.15.134:0/3152836456 >> 172.21.15.134:0/3152836456 conn(0x55a20306e300 msgr2=0x55a203424340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.038+0000 7f65a8dec740 1 -- 172.21.15.134:0/3152836456 shutdown_connections 2024-07-26T11:16:47.040 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.038+0000 7f65a8dec740 1 -- 172.21.15.134:0/3152836456 wait complete. 2024-07-26T11:16:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.039+0000 7f65a8dec740 1 -- start start 2024-07-26T11:16:47.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.039+0000 7f65a8dec740 1 Processor -- start 2024-07-26T11:16:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.040+0000 7f65a8dec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2030512f0 0x55a2034320e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.040+0000 7f65a8dec740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a2034326c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.040+0000 7f65a8dec740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a20307a1b0 0x55a20344ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.040+0000 7f65a8dec740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a2032bbab0 con 0x55a20307a1b0 2024-07-26T11:16:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.040+0000 7f65a8dec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a20331ed50 con 0x55a2030512f0 2024-07-26T11:16:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.040+0000 7f65a8dec740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a203445be0 con 0x55a2030594d0 2024-07-26T11:16:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.040+0000 7f65a8dec740 10 client.0 _ll_register_callbacks cb 0x55a203091a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:16:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.041+0000 7f659a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a2034326c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.041+0000 7f659a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a2034326c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57932/0 (socket says 172.21.15.134:57932) 2024-07-26T11:16:47.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.041+0000 7f659a000640 1 -- 172.21.15.134:0/2307345652 learned_addr learned my addr 172.21.15.134:0/2307345652 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.041+0000 7f659b400640 1 --2- 172.21.15.134:0/2307345652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a20307a1b0 0x55a20344ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.041+0000 7f659b400640 1 --2- 172.21.15.134:0/2307345652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2030512f0 0x55a2034320e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.042+0000 7f659a000640 1 -- 172.21.15.134:0/2307345652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2030512f0 msgr2=0x55a2034320e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:47.045 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.042+0000 7f659a000640 1 --2- 172.21.15.134:0/2307345652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2030512f0 0x55a2034320e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.045 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.042+0000 7f659a000640 1 -- 172.21.15.134:0/2307345652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a20307a1b0 msgr2=0x55a20344ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:47.045 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.042+0000 7f659a000640 1 --2- 172.21.15.134:0/2307345652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a20307a1b0 0x55a20344ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.045 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.043+0000 7f659a000640 1 -- 172.21.15.134:0/2307345652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a20315e620 con 0x55a2030594d0 2024-07-26T11:16:47.045 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.043+0000 7f659b400640 1 --2- 172.21.15.134:0/2307345652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a20307a1b0 0x55a20344ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:47.045 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.043+0000 7f659a000640 1 --2- 172.21.15.134:0/2307345652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a2034326c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f65880136e0 tx=0x7f6588013ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe619492f740028f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:47.045 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.043+0000 7f6584c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:16:47.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.047+0000 7f6584c00640 1 -- 172.21.15.134:0/2307345652 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65880119c0 con 0x55a2030594d0 2024-07-26T11:16:47.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.048+0000 7f6584c00640 1 -- 172.21.15.134:0/2307345652 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f6588011b60 con 0x55a2030594d0 2024-07-26T11:16:47.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.048+0000 7f65a8dec740 1 -- 172.21.15.134:0/2307345652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6594004050 con 0x55a2030594d0 2024-07-26T11:16:47.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.048+0000 7f6584c00640 1 -- 172.21.15.134:0/2307345652 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6588011d60 con 0x55a2030594d0 2024-07-26T11:16:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.048+0000 7f65a8dec740 10 client.25621 fetch_fsmap learned FSMap version 30 2024-07-26T11:16:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.048+0000 7f65a8dec740 1 -- 172.21.15.134:0/2307345652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a203451ce0 con 0x55a2030594d0 2024-07-26T11:16:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.049+0000 7f6584c00640 1 -- 172.21.15.134:0/2307345652 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6594004050 con 0x55a2030594d0 2024-07-26T11:16:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.050+0000 7f6584c00640 1 -- 172.21.15.134:0/2307345652 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f658801a070 con 0x55a2030594d0 2024-07-26T11:16:47.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.050+0000 7f65a8dec740 10 client.25621 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:16:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.050+0000 7f65a8dec740 10 client.25621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:16:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.050+0000 7f65a8dec740 20 client.25621 populate_metadata read hostname 'smithi134' 2024-07-26T11:16:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.050+0000 7f65a8dec740 1 --2- 172.21.15.134:0/2307345652 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a203038d80 0x55a203455250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.050+0000 7f65a8dec740 4 client.25621 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:16:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.050+0000 7f65a8dec740 1 -- 172.21.15.134:0/2307345652 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2033941c0 con 0x55a203038d80 2024-07-26T11:16:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.051+0000 7f659b400640 1 --2- 172.21.15.134:0/2307345652 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a203038d80 0x55a203455250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.052+0000 7f659b400640 1 --2- 172.21.15.134:0/2307345652 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a203038d80 0x55a203455250 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9dba12e74f785ac0 server_cookie=3fe5b0a681cb4a1e in_seq=0 out_seq=0 2024-07-26T11:16:47.054 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.053+0000 7f6584c00640 10 client.25621 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.053+0000 7f6584c00640 1 -- 172.21.15.134:0/2307345652 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a2033941c0 con 0x55a203038d80 2024-07-26T11:16:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.053+0000 7f6584c00640 10 client.25621 handle_command_reply: tid=0 2024-07-26T11:16:47.055 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.053+0000 7f65a8dec740 1 client.25621 shutdown 2024-07-26T11:16:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.054+0000 7f65a8dec740 1 -- 172.21.15.134:0/2307345652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 msgr2=0x55a2034326c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:47.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.054+0000 7f65a8dec740 1 --2- 172.21.15.134:0/2307345652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a2034326c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f65880136e0 tx=0x7f6588013ba0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.054+0000 7f65a8dec740 1 -- 172.21.15.134:0/2307345652 shutdown_connections 2024-07-26T11:16:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.054+0000 7f65a8dec740 1 --2- 172.21.15.134:0/2307345652 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a203038d80 0x55a203455250 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.054+0000 7f6584c00640 0 client.25621 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.054+0000 7f65a8dec740 1 --2- 172.21.15.134:0/2307345652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a2034326c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.055+0000 7f65a8dec740 1 --2- 172.21.15.134:0/2307345652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a20307a1b0 0x55a20344ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.055+0000 7f65a8dec740 1 --2- 172.21.15.134:0/2307345652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2030512f0 0x55a2034320e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.055+0000 7f65a8dec740 1 -- 172.21.15.134:0/2307345652 >> 172.21.15.134:0/2307345652 conn(0x55a20306e300 msgr2=0x55a203427480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.055+0000 7f65a8dec740 1 -- 172.21.15.134:0/2307345652 shutdown_connections 2024-07-26T11:16:47.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.055+0000 7f65a8dec740 1 -- 172.21.15.134:0/2307345652 wait complete. 2024-07-26T11:16:47.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.055+0000 7f65a8dec740 20 client.25621 trim_cache size 0 max 16384 2024-07-26T11:16:47.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.075+0000 7f65a8dec740 1 Processor -- start 2024-07-26T11:16:47.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.075+0000 7f65a8dec740 1 -- start start 2024-07-26T11:16:47.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.076+0000 7f65a8dec740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a2030512f0 0x55a203424440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.078 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.076+0000 7f65a8dec740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a20344ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.078 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.076+0000 7f65a8dec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a20307a1b0 0x55a2034323b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.078 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.076+0000 7f65a8dec740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f659c072470 con 0x55a2030512f0 2024-07-26T11:16:47.078 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.076+0000 7f65a8dec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f659c073fb0 con 0x55a20307a1b0 2024-07-26T11:16:47.078 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.076+0000 7f65a8dec740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f659c072b40 con 0x55a2030594d0 2024-07-26T11:16:47.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.076+0000 7f659a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a20344ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.076+0000 7f659a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a20344ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57944/0 (socket says 172.21.15.134:57944) 2024-07-26T11:16:47.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.076+0000 7f659a000640 1 -- 172.21.15.134:0/370910526 learned_addr learned my addr 172.21.15.134:0/370910526 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:47.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.076+0000 7f659b400640 1 --2- 172.21.15.134:0/370910526 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a2030512f0 0x55a203424440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.077+0000 7f659b400640 1 --2- 172.21.15.134:0/370910526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a20307a1b0 0x55a2034323b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.077+0000 7f659b400640 1 -- 172.21.15.134:0/370910526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 msgr2=0x55a20344ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.077+0000 7f659b400640 1 --2- 172.21.15.134:0/370910526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a20344ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.077+0000 7f659b400640 1 -- 172.21.15.134:0/370910526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a20307a1b0 msgr2=0x55a2034323b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.077+0000 7f659b400640 1 --2- 172.21.15.134:0/370910526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a20307a1b0 0x55a2034323b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.077+0000 7f659b400640 1 -- 172.21.15.134:0/370910526 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a203429260 con 0x55a2030512f0 2024-07-26T11:16:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.077+0000 7f659a000640 1 --2- 172.21.15.134:0/370910526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a20344ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:47.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.077+0000 7f659b400640 1 --2- 172.21.15.134:0/370910526 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a2030512f0 0x55a203424440 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6594009f00 tx=0x7f6594028040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41ced267db751284 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:47.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.078+0000 7f6587400640 1 -- 172.21.15.134:0/370910526 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6594034070 con 0x55a2030512f0 2024-07-26T11:16:47.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.078+0000 7f6587400640 1 -- 172.21.15.134:0/370910526 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6594003e70 con 0x55a2030512f0 2024-07-26T11:16:47.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.078+0000 7f6587400640 1 -- 172.21.15.134:0/370910526 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659400fe90 con 0x55a2030512f0 2024-07-26T11:16:47.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.079+0000 7f65a8dec740 1 -- 172.21.15.134:0/370910526 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a2030512f0 msgr2=0x55a203424440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:47.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.079+0000 7f65a8dec740 1 --2- 172.21.15.134:0/370910526 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a2030512f0 0x55a203424440 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6594009f00 tx=0x7f6594028040 comp rx=0 tx=0).stop 2024-07-26T11:16:47.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.079+0000 7f65a8dec740 1 -- 172.21.15.134:0/370910526 shutdown_connections 2024-07-26T11:16:47.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.079+0000 7f65a8dec740 1 --2- 172.21.15.134:0/370910526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a20344ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.079+0000 7f65a8dec740 1 --2- 172.21.15.134:0/370910526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a20307a1b0 0x55a2034323b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.079+0000 7f65a8dec740 1 --2- 172.21.15.134:0/370910526 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a2030512f0 0x55a203424440 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.079+0000 7f65a8dec740 1 -- 172.21.15.134:0/370910526 >> 172.21.15.134:0/370910526 conn(0x55a20306e300 msgr2=0x55a203423d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:47.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.079+0000 7f65a8dec740 1 -- 172.21.15.134:0/370910526 shutdown_connections 2024-07-26T11:16:47.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.080+0000 7f65a8dec740 1 -- 172.21.15.134:0/370910526 wait complete. 2024-07-26T11:16:47.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.080+0000 7f65a8dec740 1 -- start start 2024-07-26T11:16:47.083 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.080+0000 7f65a8dec740 1 Processor -- start 2024-07-26T11:16:47.083 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.081+0000 7f65a8dec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2030512f0 0x55a203445e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.083 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.081+0000 7f65a8dec740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a203446390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.081+0000 7f65a8dec740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a20307a1b0 0x55a2034468d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.081+0000 7f65a8dec740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f659c0738e0 con 0x55a20307a1b0 2024-07-26T11:16:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.081+0000 7f65a8dec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f659c073210 con 0x55a2030512f0 2024-07-26T11:16:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.081+0000 7f65a8dec740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f659c070450 con 0x55a2030594d0 2024-07-26T11:16:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.081+0000 7f65a8dec740 10 client.0 _ll_register_callbacks cb 0x7f65880097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:16:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.081+0000 7f659a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a203446390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.082+0000 7f659a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a203446390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57956/0 (socket says 172.21.15.134:57956) 2024-07-26T11:16:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.082+0000 7f659a000640 1 -- 172.21.15.134:0/3705982630 learned_addr learned my addr 172.21.15.134:0/3705982630 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:47.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.082+0000 7f659b400640 1 --2- 172.21.15.134:0/3705982630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2030512f0 0x55a203445e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.082+0000 7f659b400640 1 --2- 172.21.15.134:0/3705982630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a20307a1b0 0x55a2034468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.082+0000 7f659b400640 1 -- 172.21.15.134:0/3705982630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 msgr2=0x55a203446390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.082+0000 7f659b400640 1 --2- 172.21.15.134:0/3705982630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a203446390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.083+0000 7f659b400640 1 -- 172.21.15.134:0/3705982630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a20307a1b0 msgr2=0x55a2034468d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.083+0000 7f659b400640 1 --2- 172.21.15.134:0/3705982630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a20307a1b0 0x55a2034468d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.083+0000 7f659b400640 1 -- 172.21.15.134:0/3705982630 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2034bb990 con 0x55a2030512f0 2024-07-26T11:16:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.083+0000 7f659a000640 1 --2- 172.21.15.134:0/3705982630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a203446390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:47.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.083+0000 7f659b400640 1 --2- 172.21.15.134:0/3705982630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2030512f0 0x55a203445e50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6594009f00 tx=0x7f6594009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c99aec650dc3637b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.083+0000 7f6584c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:16:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.084+0000 7f6584c00640 1 -- 172.21.15.134:0/3705982630 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6594034070 con 0x55a2030512f0 2024-07-26T11:16:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.084+0000 7f6584c00640 1 -- 172.21.15.134:0/3705982630 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f659400fe40 con 0x55a2030512f0 2024-07-26T11:16:47.086 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.084+0000 7f65a8dec740 1 -- 172.21.15.134:0/3705982630 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6594003e70 con 0x55a2030512f0 2024-07-26T11:16:47.087 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.085+0000 7f6584c00640 1 -- 172.21.15.134:0/3705982630 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f659400f690 con 0x55a2030512f0 2024-07-26T11:16:47.087 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.085+0000 7f65a8dec740 10 client.25613 fetch_fsmap learned FSMap version 30 2024-07-26T11:16:47.087 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.085+0000 7f65a8dec740 1 -- 172.21.15.134:0/3705982630 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a2034bcb80 con 0x55a2030512f0 2024-07-26T11:16:47.087 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.085+0000 7f6584c00640 1 -- 172.21.15.134:0/3705982630 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6594003e70 con 0x55a2030512f0 2024-07-26T11:16:47.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.086+0000 7f6584c00640 1 -- 172.21.15.134:0/3705982630 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f6594039b60 con 0x55a2030512f0 2024-07-26T11:16:47.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.086+0000 7f65a8dec740 10 client.25613 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:16:47.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.086+0000 7f65a8dec740 10 client.25613 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:16:47.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.086+0000 7f65a8dec740 20 client.25613 populate_metadata read hostname 'smithi134' 2024-07-26T11:16:47.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.086+0000 7f65a8dec740 1 --2- 172.21.15.134:0/3705982630 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a20300e1c0 0x55a2034bf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:47.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.086+0000 7f65a8dec740 4 client.25613 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:16:47.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.086+0000 7f65a8dec740 1 -- 172.21.15.134:0/3705982630 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a20338fc60 con 0x55a20300e1c0 2024-07-26T11:16:47.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.087+0000 7f659a000640 1 --2- 172.21.15.134:0/3705982630 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a20300e1c0 0x55a2034bf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:47.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.088+0000 7f659a000640 1 --2- 172.21.15.134:0/3705982630 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a20300e1c0 0x55a2034bf0b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1154097603227189 server_cookie=b4c3938c550c52b9 in_seq=0 out_seq=0 2024-07-26T11:16:47.090 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.088+0000 7f6584c00640 10 client.25613 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:47.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.100+0000 7f6584c00640 1 -- 172.21.15.134:0/3705982630 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a20338fc60 con 0x55a20300e1c0 2024-07-26T11:16:47.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.100+0000 7f6584c00640 10 client.25613 handle_command_reply: tid=0 2024-07-26T11:16:47.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.100+0000 7f65a8dec740 1 client.25613 shutdown 2024-07-26T11:16:47.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.101+0000 7f65a8dec740 1 -- 172.21.15.134:0/3705982630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2030512f0 msgr2=0x55a203445e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:47.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.101+0000 7f65a8dec740 1 --2- 172.21.15.134:0/3705982630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2030512f0 0x55a203445e50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6594009f00 tx=0x7f6594009580 comp rx=0 tx=0).stop 2024-07-26T11:16:47.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.101+0000 7f65a8dec740 1 -- 172.21.15.134:0/3705982630 shutdown_connections 2024-07-26T11:16:47.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.101+0000 7f65a8dec740 1 --2- 172.21.15.134:0/3705982630 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a20300e1c0 0x55a2034bf0b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.101+0000 7f65a8dec740 1 --2- 172.21.15.134:0/3705982630 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a2030594d0 0x55a203446390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.101+0000 7f6584c00640 0 client.25613 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:16:47.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.101+0000 7f65a8dec740 1 --2- 172.21.15.134:0/3705982630 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a20307a1b0 0x55a2034468d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.101+0000 7f65a8dec740 1 --2- 172.21.15.134:0/3705982630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a2030512f0 0x55a203445e50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.101+0000 7f65a8dec740 1 -- 172.21.15.134:0/3705982630 >> 172.21.15.134:0/3705982630 conn(0x55a20306e300 msgr2=0x55a20341a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:47.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.101+0000 7f65a8dec740 1 -- 172.21.15.134:0/3705982630 shutdown_connections 2024-07-26T11:16:47.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.101+0000 7f65a8dec740 1 -- 172.21.15.134:0/3705982630 wait complete. 2024-07-26T11:16:47.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.101+0000 7f65a8dec740 20 client.25613 trim_cache size 0 max 16384 2024-07-26T11:16:47.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:16:47.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:16:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:16:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:16:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.102+0000 7f659be00640 1 -- 172.21.15.134:0/4230458921 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65800b3d60 msgr2=0x7f65800b6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.102+0000 7f659be00640 1 --2- 172.21.15.134:0/4230458921 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65800b3d60 0x7f65800b6270 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f658c02a9b0 tx=0x7f658c1084a0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.102+0000 7f659be00640 1 -- 172.21.15.134:0/4230458921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f659c103340 msgr2=0x7f659c075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.102+0000 7f659be00640 1 --2- 172.21.15.134:0/4230458921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f659c103340 0x7f659c075140 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f658c005ab0 tx=0x7f658c003c10 comp rx=0 tx=0).stop 2024-07-26T11:16:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.103+0000 7f659be00640 1 -- 172.21.15.134:0/4230458921 shutdown_connections 2024-07-26T11:16:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.103+0000 7f659be00640 1 --2- 172.21.15.134:0/4230458921 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65800b3d60 0x7f65800b6270 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.103+0000 7f659be00640 1 --2- 172.21.15.134:0/4230458921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f659c1095c0 0x7f659c075bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.103+0000 7f659be00640 1 --2- 172.21.15.134:0/4230458921 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f659c108c70 0x7f659c075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.103+0000 7f659be00640 1 --2- 172.21.15.134:0/4230458921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f659c103340 0x7f659c075140 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.103+0000 7f659be00640 1 -- 172.21.15.134:0/4230458921 >> 172.21.15.134:0/4230458921 conn(0x7f659c0fed60 msgr2=0x7f659c101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.103+0000 7f659be00640 1 -- 172.21.15.134:0/4230458921 shutdown_connections 2024-07-26T11:16:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:16:47.103+0000 7f659be00640 1 -- 172.21.15.134:0/4230458921 wait complete. 2024-07-26T11:16:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:48 smithi134 ceph-mon[31966]: pgmap v505: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-07-26T11:16:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:48 smithi189 ceph-mon[36790]: pgmap v505: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-07-26T11:16:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:48 smithi193 ceph-mon[37473]: pgmap v505: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-07-26T11:16:49.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.192+0000 7ff1db200640 1 -- 172.21.15.134:0/949132916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff1dc109670 msgr2=0x7ff1dc10fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:49.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.192+0000 7ff1db200640 1 --2- 172.21.15.134:0/949132916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff1dc109670 0x7ff1dc10fea0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4009be0 tx=0x7ff1d402a6c0 comp rx=0 tx=0).stop 2024-07-26T11:16:49.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.193+0000 7ff1db200640 1 -- 172.21.15.134:0/949132916 shutdown_connections 2024-07-26T11:16:49.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.193+0000 7ff1db200640 1 --2- 172.21.15.134:0/949132916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1dc108d20 0x7ff1dc109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:49.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.193+0000 7ff1db200640 1 --2- 172.21.15.134:0/949132916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff1dc109670 0x7ff1dc10fea0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:49.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.193+0000 7ff1db200640 1 --2- 172.21.15.134:0/949132916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1dc1033e0 0x7ff1dc1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:49.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.193+0000 7ff1db200640 1 -- 172.21.15.134:0/949132916 >> 172.21.15.134:0/949132916 conn(0x7ff1dc0fee70 msgr2=0x7ff1dc1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.193+0000 7ff1db200640 1 -- 172.21.15.134:0/949132916 shutdown_connections 2024-07-26T11:16:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.193+0000 7ff1db200640 1 -- 172.21.15.134:0/949132916 wait complete. 2024-07-26T11:16:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.195+0000 7ff1db200640 1 Processor -- start 2024-07-26T11:16:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.195+0000 7ff1db200640 1 -- start start 2024-07-26T11:16:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.196+0000 7ff1db200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1dc1033e0 0x7ff1dc199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:49.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.196+0000 7ff1db200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1dc108d20 0x7ff1dc19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:49.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.196+0000 7ff1db200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff1dc109670 0x7ff1dc19f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:49.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.196+0000 7ff1db200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff1dc113320 con 0x7ff1dc109670 2024-07-26T11:16:49.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.196+0000 7ff1db200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff1dc1131a0 con 0x7ff1dc108d20 2024-07-26T11:16:49.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.196+0000 7ff1db200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff1dc1134a0 con 0x7ff1dc1033e0 2024-07-26T11:16:49.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.196+0000 7ff1da800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff1dc109670 0x7ff1dc19f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:49.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.196+0000 7ff1d9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1dc108d20 0x7ff1dc19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:49.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.196+0000 7ff1da800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff1dc109670 0x7ff1dc19f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:60506/0 (socket says 172.21.15.134:60506) 2024-07-26T11:16:49.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.196+0000 7ff1da800640 1 -- 172.21.15.134:0/2671551670 learned_addr learned my addr 172.21.15.134:0/2671551670 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:49.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.196+0000 7ff1d9e00640 1 --2- 172.21.15.134:0/2671551670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1dc1033e0 0x7ff1dc199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.197+0000 7ff1da800640 1 -- 172.21.15.134:0/2671551670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1dc1033e0 msgr2=0x7ff1dc199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.197+0000 7ff1da800640 1 --2- 172.21.15.134:0/2671551670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1dc1033e0 0x7ff1dc199cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.197+0000 7ff1da800640 1 -- 172.21.15.134:0/2671551670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1dc108d20 msgr2=0x7ff1dc19a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.197+0000 7ff1da800640 1 --2- 172.21.15.134:0/2671551670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1dc108d20 0x7ff1dc19a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.197+0000 7ff1da800640 1 -- 172.21.15.134:0/2671551670 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1dc19fc40 con 0x7ff1dc109670 2024-07-26T11:16:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.198+0000 7ff1da800640 1 --2- 172.21.15.134:0/2671551670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff1dc109670 0x7ff1dc19f510 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4009b30 tx=0x7ff1d403fca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:49.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.198+0000 7ff1d2a00640 1 -- 172.21.15.134:0/2671551670 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d404c070 con 0x7ff1dc109670 2024-07-26T11:16:49.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.198+0000 7ff1d2a00640 1 -- 172.21.15.134:0/2671551670 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1d40318f0 con 0x7ff1dc109670 2024-07-26T11:16:49.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.198+0000 7ff1db200640 1 -- 172.21.15.134:0/2671551670 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1dc19fed0 con 0x7ff1dc109670 2024-07-26T11:16:49.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.198+0000 7ff1db200640 1 -- 172.21.15.134:0/2671551670 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1dc075020 con 0x7ff1dc109670 2024-07-26T11:16:49.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.198+0000 7ff1d2a00640 1 -- 172.21.15.134:0/2671551670 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4036040 con 0x7ff1dc109670 2024-07-26T11:16:49.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.201+0000 7ff1db200640 1 -- 172.21.15.134:0/2671551670 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1bc0052c0 con 0x7ff1dc109670 2024-07-26T11:16:49.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.204+0000 7ff1d2a00640 1 -- 172.21.15.134:0/2671551670 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7ff1d404f070 con 0x7ff1dc109670 2024-07-26T11:16:49.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.207+0000 7ff1d2a00640 1 --2- 172.21.15.134:0/2671551670 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff1c00b3d30 0x7ff1c00b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:49.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.207+0000 7ff1d2a00640 1 -- 172.21.15.134:0/2671551670 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff1d410c880 con 0x7ff1dc109670 2024-07-26T11:16:49.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.208+0000 7ff1d9e00640 1 --2- 172.21.15.134:0/2671551670 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff1c00b3d30 0x7ff1c00b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:49.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.209+0000 7ff1d9e00640 1 --2- 172.21.15.134:0/2671551670 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff1c00b3d30 0x7ff1c00b6240 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0095f0 tx=0x7ff1cc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:49.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.214+0000 7ff1d2a00640 1 -- 172.21.15.134:0/2671551670 <== mon.0 v2:172.21.15.134:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7ff1d4085600 con 0x7ff1dc109670 2024-07-26T11:16:49.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.665+0000 7ff1db200640 1 -- 172.21.15.134:0/2671551670 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1bc005880 con 0x7ff1dc109670 2024-07-26T11:16:49.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.666+0000 7ff1d2a00640 1 -- 172.21.15.134:0/2671551670 <== mon.0 v2:172.21.15.134:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11990 (secure 0 0 0) 0x7ff1d4044070 con 0x7ff1dc109670 2024-07-26T11:16:49.668 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:16:49.670 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":30,"btime":"2024-07-26T11:15:47:899566+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:15:47.849285+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24721,"mds_1":24719,"mds_2":24703,"mds_3":24688,"mds_4":24686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24635":{"gid":24635,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24641":{"gid":24641,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":32,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24719":{"gid":24719,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":28,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24721,"qdb_cluster":[24721,24719,24703,24688,24686]},"id":1}]} 2024-07-26T11:16:49.670 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 30 2024-07-26T11:16:49.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.670+0000 7ff1db200640 1 -- 172.21.15.134:0/2671551670 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff1c00b3d30 msgr2=0x7ff1c00b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:49.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.670+0000 7ff1db200640 1 --2- 172.21.15.134:0/2671551670 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff1c00b3d30 0x7ff1c00b6240 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0095f0 tx=0x7ff1cc005f90 comp rx=0 tx=0).stop 2024-07-26T11:16:49.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.670+0000 7ff1db200640 1 -- 172.21.15.134:0/2671551670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff1dc109670 msgr2=0x7ff1dc19f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:49.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.670+0000 7ff1db200640 1 --2- 172.21.15.134:0/2671551670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff1dc109670 0x7ff1dc19f510 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4009b30 tx=0x7ff1d403fca0 comp rx=0 tx=0).stop 2024-07-26T11:16:49.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.670+0000 7ff1db200640 1 -- 172.21.15.134:0/2671551670 shutdown_connections 2024-07-26T11:16:49.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.670+0000 7ff1db200640 1 --2- 172.21.15.134:0/2671551670 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff1c00b3d30 0x7ff1c00b6240 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:49.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.670+0000 7ff1db200640 1 --2- 172.21.15.134:0/2671551670 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff1dc109670 0x7ff1dc19f510 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:49.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.670+0000 7ff1db200640 1 --2- 172.21.15.134:0/2671551670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1dc108d20 0x7ff1dc19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:49.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.670+0000 7ff1db200640 1 --2- 172.21.15.134:0/2671551670 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1dc1033e0 0x7ff1dc199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:49.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.670+0000 7ff1db200640 1 -- 172.21.15.134:0/2671551670 >> 172.21.15.134:0/2671551670 conn(0x7ff1dc0fee70 msgr2=0x7ff1dc101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:49.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.671+0000 7ff1db200640 1 -- 172.21.15.134:0/2671551670 shutdown_connections 2024-07-26T11:16:49.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:49.671+0000 7ff1db200640 1 -- 172.21.15.134:0/2671551670 wait complete. 2024-07-26T11:16:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:49 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2671551670' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:16:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:49 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2671551670' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:16:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:49 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2671551670' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:16:50.578 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-07-26T11:16:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:50 smithi134 ceph-mon[31966]: pgmap v506: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-07-26T11:16:51.011 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:16:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:50 smithi193 ceph-mon[37473]: pgmap v506: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-07-26T11:16:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:50 smithi189 ceph-mon[36790]: pgmap v506: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-07-26T11:16:52.412 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:52 smithi134 ceph-mon[31966]: pgmap v507: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T11:16:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:52 smithi189 ceph-mon[36790]: pgmap v507: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T11:16:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:52 smithi193 ceph-mon[37473]: pgmap v507: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-07-26T11:16:53.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.445+0000 7f8735600640 1 -- 172.21.15.134:0/960050967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f873010af30 msgr2=0x7f873010b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:53.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.445+0000 7f8735600640 1 --2- 172.21.15.134:0/960050967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f873010af30 0x7f873010b340 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f871c009900 tx=0x7f871c02a410 comp rx=0 tx=0).stop 2024-07-26T11:16:53.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.445+0000 7f8735600640 1 -- 172.21.15.134:0/960050967 shutdown_connections 2024-07-26T11:16:53.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.445+0000 7f8735600640 1 --2- 172.21.15.134:0/960050967 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f873010af30 0x7f873010b340 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.445+0000 7f8735600640 1 --2- 172.21.15.134:0/960050967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f873010b880 0x7f87301120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.445+0000 7f8735600640 1 --2- 172.21.15.134:0/960050967 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f87301055f0 0x7f8730105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.445+0000 7f8735600640 1 -- 172.21.15.134:0/960050967 >> 172.21.15.134:0/960050967 conn(0x7f8730101060 msgr2=0x7f87301034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:53.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.445+0000 7f8735600640 1 -- 172.21.15.134:0/960050967 shutdown_connections 2024-07-26T11:16:53.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.445+0000 7f8735600640 1 -- 172.21.15.134:0/960050967 wait complete. 2024-07-26T11:16:53.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.447+0000 7f8735600640 1 Processor -- start 2024-07-26T11:16:53.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.447+0000 7f8735600640 1 -- start start 2024-07-26T11:16:53.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.447+0000 7f8735600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87301055f0 0x7f873019bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.448+0000 7f8735600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f873010af30 0x7f873019c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.448+0000 7f8735600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f873010b880 0x7f87301a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.448+0000 7f8735600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f87301154f0 con 0x7f873010af30 2024-07-26T11:16:53.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.448+0000 7f8735600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8730115370 con 0x7f873010b880 2024-07-26T11:16:53.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.448+0000 7f8735600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8730115670 con 0x7f87301055f0 2024-07-26T11:16:53.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.448+0000 7f872fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87301055f0 0x7f873019bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:53.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.448+0000 7f8734c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f873010b880 0x7f87301a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:53.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.448+0000 7f872f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f873010af30 0x7f873019c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:53.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.448+0000 7f8734c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f873010b880 0x7f87301a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:39316/0 (socket says 172.21.15.134:39316) 2024-07-26T11:16:53.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.448+0000 7f8734c00640 1 -- 172.21.15.134:0/1857938274 learned_addr learned my addr 172.21.15.134:0/1857938274 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:53.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.449+0000 7f8734c00640 1 -- 172.21.15.134:0/1857938274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87301055f0 msgr2=0x7f873019bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:53.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.449+0000 7f8734c00640 1 --2- 172.21.15.134:0/1857938274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87301055f0 0x7f873019bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.449+0000 7f8734c00640 1 -- 172.21.15.134:0/1857938274 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f873010af30 msgr2=0x7f873019c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:53.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.449+0000 7f8734c00640 1 --2- 172.21.15.134:0/1857938274 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f873010af30 0x7f873019c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.449+0000 7f8734c00640 1 -- 172.21.15.134:0/1857938274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87301a1e90 con 0x7f873010b880 2024-07-26T11:16:53.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.449+0000 7f872f400640 1 --2- 172.21.15.134:0/1857938274 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f873010af30 0x7f873019c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:53.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.450+0000 7f8734c00640 1 --2- 172.21.15.134:0/1857938274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f873010b880 0x7f87301a1760 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f872800eb10 tx=0x7f872800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:53.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.450+0000 7f872cc00640 1 -- 172.21.15.134:0/1857938274 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8728004b20 con 0x7f873010b880 2024-07-26T11:16:53.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.450+0000 7f8735600640 1 -- 172.21.15.134:0/1857938274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87301a2180 con 0x7f873010b880 2024-07-26T11:16:53.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.450+0000 7f872cc00640 1 -- 172.21.15.134:0/1857938274 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8728004cc0 con 0x7f873010b880 2024-07-26T11:16:53.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.450+0000 7f872cc00640 1 -- 172.21.15.134:0/1857938274 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8728010670 con 0x7f873010b880 2024-07-26T11:16:53.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.450+0000 7f8735600640 1 -- 172.21.15.134:0/1857938274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87301a88e0 con 0x7f873010b880 2024-07-26T11:16:53.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.455+0000 7f873d82a740 1 Processor -- start 2024-07-26T11:16:53.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.456+0000 7f872cc00640 1 -- 172.21.15.134:0/1857938274 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7f8728005000 con 0x7f873010b880 2024-07-26T11:16:53.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.456+0000 7f873d82a740 1 -- start start 2024-07-26T11:16:53.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.457+0000 7f873d82a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e832070 0x563c6ebdf820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.457+0000 7f873d82a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e8091b0 0x563c6ebdfd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.457+0000 7f873d82a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e811390 0x563c6ebe02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.457+0000 7f873d82a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563c6eb4dbd0 con 0x563c6e832070 2024-07-26T11:16:53.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.457+0000 7f873d82a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563c6eb3d0d0 con 0x563c6e811390 2024-07-26T11:16:53.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.457+0000 7f873d82a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563c6eb41110 con 0x563c6e8091b0 2024-07-26T11:16:53.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.457+0000 7f872fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e8091b0 0x563c6ebdfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:53.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.457+0000 7f872fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e8091b0 0x563c6ebdfd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:37562/0 (socket says 172.21.15.134:37562) 2024-07-26T11:16:53.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.457+0000 7f872fe00640 1 -- 172.21.15.134:0/1938939950 learned_addr learned my addr 172.21.15.134:0/1938939950 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:53.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.458+0000 7f872fe00640 1 -- 172.21.15.134:0/1938939950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e811390 msgr2=0x563c6ebe02a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:16:53.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.458+0000 7f872fe00640 1 --2- 172.21.15.134:0/1938939950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e811390 0x563c6ebe02a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.458+0000 7f872fe00640 1 -- 172.21.15.134:0/1938939950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e832070 msgr2=0x563c6ebdf820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:16:53.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.458+0000 7f872fe00640 1 --2- 172.21.15.134:0/1938939950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e832070 0x563c6ebdf820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.458+0000 7f872fe00640 1 -- 172.21.15.134:0/1938939950 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c6e89d650 con 0x563c6e8091b0 2024-07-26T11:16:53.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.458+0000 7f872fe00640 1 --2- 172.21.15.134:0/1938939950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e8091b0 0x563c6ebdfd60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8720009c50 tx=0x7f8720010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29229c3999f3429d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:53.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.459+0000 7f872cc00640 1 --2- 172.21.15.134:0/1857938274 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f87140b3cb0 0x7f87140b61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.459+0000 7f8725200640 1 -- 172.21.15.134:0/1938939950 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8720004020 con 0x563c6e8091b0 2024-07-26T11:16:53.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.459+0000 7f872cc00640 1 -- 172.21.15.134:0/1857938274 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f87280e4980 con 0x7f873010b880 2024-07-26T11:16:53.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.459+0000 7f8725200640 1 -- 172.21.15.134:0/1938939950 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87200041c0 con 0x563c6e8091b0 2024-07-26T11:16:53.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.460+0000 7f872f400640 1 --2- 172.21.15.134:0/1857938274 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f87140b3cb0 0x7f87140b61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:53.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.460+0000 7f8725200640 1 -- 172.21.15.134:0/1938939950 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87200126b0 con 0x563c6e8091b0 2024-07-26T11:16:53.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.460+0000 7f873d82a740 1 -- 172.21.15.134:0/1938939950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e8091b0 msgr2=0x563c6ebdfd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:53.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.460+0000 7f873d82a740 1 --2- 172.21.15.134:0/1938939950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e8091b0 0x563c6ebdfd60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8720009c50 tx=0x7f8720010740 comp rx=0 tx=0).stop 2024-07-26T11:16:53.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.461+0000 7f873d82a740 1 -- 172.21.15.134:0/1938939950 shutdown_connections 2024-07-26T11:16:53.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.461+0000 7f873d82a740 1 --2- 172.21.15.134:0/1938939950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e8091b0 0x563c6ebdfd60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.461+0000 7f873d82a740 1 --2- 172.21.15.134:0/1938939950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e811390 0x563c6ebe02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.461+0000 7f873d82a740 1 --2- 172.21.15.134:0/1938939950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e832070 0x563c6ebdf820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.461+0000 7f873d82a740 1 -- 172.21.15.134:0/1938939950 >> 172.21.15.134:0/1938939950 conn(0x563c6e826460 msgr2=0x563c6ebdd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:53.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.461+0000 7f873d82a740 1 -- 172.21.15.134:0/1938939950 shutdown_connections 2024-07-26T11:16:53.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.462+0000 7f873d82a740 1 -- 172.21.15.134:0/1938939950 wait complete. 2024-07-26T11:16:53.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.462+0000 7f872f400640 1 --2- 172.21.15.134:0/1857938274 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f87140b3cb0 0x7f87140b61c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f871c009230 tx=0x7f871c031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:53.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.462+0000 7f873d82a740 1 -- start start 2024-07-26T11:16:53.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.463+0000 7f873d82a740 1 Processor -- start 2024-07-26T11:16:53.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.463+0000 7f873d82a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e8091b0 0x563c6ebead30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.464+0000 7f873d82a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e811390 0x563c6ebeb270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.464+0000 7f873d82a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e832070 0x563c6ebeb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.464+0000 7f873d82a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563c6e9a0700 con 0x563c6e811390 2024-07-26T11:16:53.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.464+0000 7f873d82a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563c6ead96a0 con 0x563c6e832070 2024-07-26T11:16:53.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.464+0000 7f873d82a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563c6ebee050 con 0x563c6e8091b0 2024-07-26T11:16:53.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.464+0000 7f873d82a740 10 client.0 _ll_register_callbacks cb 0x563c6eae9150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:16:53.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.464+0000 7f872fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e8091b0 0x563c6ebead30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:53.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.464+0000 7f8734c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e832070 0x563c6ebeb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:53.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.464+0000 7f872fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e8091b0 0x563c6ebead30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:37566/0 (socket says 172.21.15.134:37566) 2024-07-26T11:16:53.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.464+0000 7f872fe00640 1 -- 172.21.15.134:0/2368544942 learned_addr learned my addr 172.21.15.134:0/2368544942 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:53.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.465+0000 7f872fe00640 1 -- 172.21.15.134:0/2368544942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e832070 msgr2=0x563c6ebeb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:53.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.465+0000 7f872fe00640 1 --2- 172.21.15.134:0/2368544942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e832070 0x563c6ebeb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.465+0000 7f872fe00640 1 -- 172.21.15.134:0/2368544942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e811390 msgr2=0x563c6ebeb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:53.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.465+0000 7f872f400640 1 --2- 172.21.15.134:0/2368544942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e811390 0x563c6ebeb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:53.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.465+0000 7f872fe00640 1 --2- 172.21.15.134:0/2368544942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e811390 0x563c6ebeb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.465+0000 7f872fe00640 1 -- 172.21.15.134:0/2368544942 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c6eaf0840 con 0x563c6e8091b0 2024-07-26T11:16:53.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.465+0000 7f872f400640 1 --2- 172.21.15.134:0/2368544942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e811390 0x563c6ebeb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:16:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.465+0000 7f872fe00640 1 --2- 172.21.15.134:0/2368544942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e8091b0 0x563c6ebead30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8720003d00 tx=0x7f87200058c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f474a4c7da0dc6b3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.466+0000 7f8719600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:16:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.466+0000 7f8719600640 1 -- 172.21.15.134:0/2368544942 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872002a060 con 0x563c6e8091b0 2024-07-26T11:16:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.466+0000 7f8719600640 1 -- 172.21.15.134:0/2368544942 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f8720025470 con 0x563c6e8091b0 2024-07-26T11:16:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.466+0000 7f873d82a740 1 -- 172.21.15.134:0/2368544942 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87200041c0 con 0x563c6e8091b0 2024-07-26T11:16:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.466+0000 7f8719600640 1 -- 172.21.15.134:0/2368544942 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8720004000 con 0x563c6e8091b0 2024-07-26T11:16:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.466+0000 7f873d82a740 10 client.25642 fetch_fsmap learned FSMap version 30 2024-07-26T11:16:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.466+0000 7f873d82a740 1 -- 172.21.15.134:0/2368544942 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563c6ebf9c00 con 0x563c6e8091b0 2024-07-26T11:16:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.467+0000 7f8719600640 1 -- 172.21.15.134:0/2368544942 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87200041c0 con 0x563c6e8091b0 2024-07-26T11:16:53.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.467+0000 7f8719600640 1 -- 172.21.15.134:0/2368544942 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f872000f040 con 0x563c6e8091b0 2024-07-26T11:16:53.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.468+0000 7f873d82a740 10 client.25642 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:16:53.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.468+0000 7f873d82a740 10 client.25642 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-07-26T11:16:53.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.468+0000 7f873d82a740 20 client.25642 populate_metadata read hostname 'smithi134' 2024-07-26T11:16:53.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.468+0000 7f873d82a740 1 --2- 172.21.15.134:0/2368544942 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x563c6e7f09f0 0x563c6ebfcf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.468+0000 7f873d82a740 4 client.25642 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:16:53.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.468+0000 7f873d82a740 1 -- 172.21.15.134:0/2368544942 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c6eb4fb20 con 0x563c6e7f09f0 2024-07-26T11:16:53.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.469+0000 7f872f400640 1 --2- 172.21.15.134:0/2368544942 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x563c6e7f09f0 0x563c6ebfcf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:53.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.469+0000 7f872f400640 1 --2- 172.21.15.134:0/2368544942 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x563c6e7f09f0 0x563c6ebfcf60 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2b3dbe9b98635b1a server_cookie=48039c3829fd20a7 in_seq=0 out_seq=0 2024-07-26T11:16:53.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.469+0000 7f8719600640 10 client.25642 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:16:53.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.470+0000 7f8719600640 1 -- 172.21.15.134:0/2368544942 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563c6eb4fb20 con 0x563c6e7f09f0 2024-07-26T11:16:53.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.470+0000 7f8719600640 10 client.25642 handle_command_reply: tid=0 2024-07-26T11:16:53.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.470+0000 7f873d82a740 1 client.25642 shutdown 2024-07-26T11:16:53.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.471+0000 7f873d82a740 1 -- 172.21.15.134:0/2368544942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e8091b0 msgr2=0x563c6ebead30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:53.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.471+0000 7f873d82a740 1 --2- 172.21.15.134:0/2368544942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e8091b0 0x563c6ebead30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8720003d00 tx=0x7f87200058c0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.471+0000 7f873d82a740 1 -- 172.21.15.134:0/2368544942 shutdown_connections 2024-07-26T11:16:53.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.471+0000 7f873d82a740 1 --2- 172.21.15.134:0/2368544942 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x563c6e7f09f0 0x563c6ebfcf60 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.471+0000 7f8719600640 0 client.25642 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:16:53.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.471+0000 7f873d82a740 1 --2- 172.21.15.134:0/2368544942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e832070 0x563c6ebeb7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.471+0000 7f873d82a740 1 --2- 172.21.15.134:0/2368544942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e811390 0x563c6ebeb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.471+0000 7f873d82a740 1 --2- 172.21.15.134:0/2368544942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e8091b0 0x563c6ebead30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.471+0000 7f873d82a740 1 -- 172.21.15.134:0/2368544942 >> 172.21.15.134:0/2368544942 conn(0x563c6e826460 msgr2=0x563c6ebe2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:53.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.472+0000 7f873d82a740 1 -- 172.21.15.134:0/2368544942 shutdown_connections 2024-07-26T11:16:53.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.472+0000 7f873d82a740 1 -- 172.21.15.134:0/2368544942 wait complete. 2024-07-26T11:16:53.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.472+0000 7f873d82a740 20 client.25642 trim_cache size 0 max 16384 2024-07-26T11:16:53.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.499+0000 7f873d82a740 1 Processor -- start 2024-07-26T11:16:53.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.499+0000 7f873d82a740 1 -- start start 2024-07-26T11:16:53.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.499+0000 7f873d82a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e7f09f0 0x563c6ebec250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.499+0000 7f873d82a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e8091b0 0x563c6ec02600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.499+0000 7f873d82a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e811390 0x563c6ec02b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.500+0000 7f873d82a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f87301ae530 con 0x563c6e811390 2024-07-26T11:16:53.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.500+0000 7f873d82a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f87301b0070 con 0x563c6e8091b0 2024-07-26T11:16:53.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.500+0000 7f873d82a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f87301aec00 con 0x563c6e7f09f0 2024-07-26T11:16:53.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.500+0000 7f872fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e7f09f0 0x563c6ebec250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:53.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.500+0000 7f872fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e7f09f0 0x563c6ebec250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:37574/0 (socket says 172.21.15.134:37574) 2024-07-26T11:16:53.503 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.500+0000 7f872fe00640 1 -- 172.21.15.134:0/2379914833 learned_addr learned my addr 172.21.15.134:0/2379914833 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:53.503 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.500+0000 7f872f400640 1 --2- 172.21.15.134:0/2379914833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e8091b0 0x563c6ec02600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:53.503 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.500+0000 7f8734c00640 1 --2- 172.21.15.134:0/2379914833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e811390 0x563c6ec02b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:53.503 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.500+0000 7f872f400640 1 -- 172.21.15.134:0/2379914833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e7f09f0 msgr2=0x563c6ebec250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:53.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.500+0000 7f872f400640 1 --2- 172.21.15.134:0/2379914833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e7f09f0 0x563c6ebec250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.501+0000 7f872f400640 1 -- 172.21.15.134:0/2379914833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e811390 msgr2=0x563c6ec02b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:53.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.501+0000 7f872f400640 1 --2- 172.21.15.134:0/2379914833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e811390 0x563c6ec02b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.501+0000 7f872f400640 1 -- 172.21.15.134:0/2379914833 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c6ebebaf0 con 0x563c6e8091b0 2024-07-26T11:16:53.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.501+0000 7f8734c00640 1 --2- 172.21.15.134:0/2379914833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e811390 0x563c6ec02b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:16:53.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.501+0000 7f872f400640 1 --2- 172.21.15.134:0/2379914833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e8091b0 0x563c6ec02600 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f871c02a930 tx=0x7f871c03d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=854ffbaaf6e994b8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:53.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.501+0000 7f871be00640 1 -- 172.21.15.134:0/2379914833 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c055070 con 0x563c6e8091b0 2024-07-26T11:16:53.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.501+0000 7f871be00640 1 -- 172.21.15.134:0/2379914833 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f871c0054f0 con 0x563c6e8091b0 2024-07-26T11:16:53.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.501+0000 7f871be00640 1 -- 172.21.15.134:0/2379914833 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c035040 con 0x563c6e8091b0 2024-07-26T11:16:53.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.503+0000 7f873d82a740 1 -- 172.21.15.134:0/2379914833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e8091b0 msgr2=0x563c6ec02600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:53.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.503+0000 7f873d82a740 1 --2- 172.21.15.134:0/2379914833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e8091b0 0x563c6ec02600 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f871c02a930 tx=0x7f871c03d000 comp rx=0 tx=0).stop 2024-07-26T11:16:53.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.503+0000 7f873d82a740 1 -- 172.21.15.134:0/2379914833 shutdown_connections 2024-07-26T11:16:53.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.503+0000 7f873d82a740 1 --2- 172.21.15.134:0/2379914833 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e811390 0x563c6ec02b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.503+0000 7f873d82a740 1 --2- 172.21.15.134:0/2379914833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e8091b0 0x563c6ec02600 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.503+0000 7f873d82a740 1 --2- 172.21.15.134:0/2379914833 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e7f09f0 0x563c6ebec250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.503+0000 7f873d82a740 1 -- 172.21.15.134:0/2379914833 >> 172.21.15.134:0/2379914833 conn(0x563c6e826460 msgr2=0x563c6ebe1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:53.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.503+0000 7f873d82a740 1 -- 172.21.15.134:0/2379914833 shutdown_connections 2024-07-26T11:16:53.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.503+0000 7f873d82a740 1 -- 172.21.15.134:0/2379914833 wait complete. 2024-07-26T11:16:53.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.504+0000 7f873d82a740 1 -- start start 2024-07-26T11:16:53.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.504+0000 7f873d82a740 1 Processor -- start 2024-07-26T11:16:53.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.505+0000 7f873d82a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e7f09f0 0x563c6ebf39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.505+0000 7f873d82a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e8091b0 0x563c6ebf0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.505+0000 7f873d82a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e811390 0x563c6ebf13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.505+0000 7f873d82a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f87301af9a0 con 0x563c6e8091b0 2024-07-26T11:16:53.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.505+0000 7f873d82a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f87301af2d0 con 0x563c6e811390 2024-07-26T11:16:53.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.505+0000 7f873d82a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f87301ac510 con 0x563c6e7f09f0 2024-07-26T11:16:53.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.505+0000 7f873d82a740 10 client.0 _ll_register_callbacks cb 0x7f8720003f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:16:53.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.505+0000 7f872f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e8091b0 0x563c6ebf0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:53.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.505+0000 7f872fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e7f09f0 0x563c6ebf39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:53.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.505+0000 7f872f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e8091b0 0x563c6ebf0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:49546/0 (socket says 172.21.15.134:49546) 2024-07-26T11:16:53.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.505+0000 7f872f400640 1 -- 172.21.15.134:0/3547797098 learned_addr learned my addr 172.21.15.134:0/3547797098 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:16:53.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.506+0000 7f8734c00640 1 --2- 172.21.15.134:0/3547797098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e811390 0x563c6ebf13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:53.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.506+0000 7f872fe00640 1 -- 172.21.15.134:0/3547797098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e811390 msgr2=0x563c6ebf13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:53.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.506+0000 7f872fe00640 1 --2- 172.21.15.134:0/3547797098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e811390 0x563c6ebf13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.506+0000 7f872fe00640 1 -- 172.21.15.134:0/3547797098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e8091b0 msgr2=0x563c6ebf0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:16:53.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.506+0000 7f872fe00640 1 --2- 172.21.15.134:0/3547797098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e8091b0 0x563c6ebf0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.506+0000 7f872fe00640 1 -- 172.21.15.134:0/3547797098 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c6ebf1b20 con 0x563c6e7f09f0 2024-07-26T11:16:53.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.506+0000 7f872f400640 1 --2- 172.21.15.134:0/3547797098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e8091b0 0x563c6ebf0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:16:53.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.506+0000 7f872fe00640 1 --2- 172.21.15.134:0/3547797098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e7f09f0 0x563c6ebf39d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f872001a620 tx=0x7f8720009690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbcf5adea573d36c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:16:53.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.506+0000 7f8719600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:16:53.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.507+0000 7f8719600640 1 -- 172.21.15.134:0/3547797098 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872002a040 con 0x563c6e7f09f0 2024-07-26T11:16:53.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.507+0000 7f873d82a740 1 -- 172.21.15.134:0/3547797098 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f871c0054f0 con 0x563c6e7f09f0 2024-07-26T11:16:53.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.507+0000 7f8719600640 1 -- 172.21.15.134:0/3547797098 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f8720009d40 con 0x563c6e7f09f0 2024-07-26T11:16:53.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.507+0000 7f8719600640 1 -- 172.21.15.134:0/3547797098 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f8720041050 con 0x563c6e7f09f0 2024-07-26T11:16:53.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.507+0000 7f873d82a740 10 client.25648 fetch_fsmap learned FSMap version 30 2024-07-26T11:16:53.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.507+0000 7f873d82a740 1 -- 172.21.15.134:0/3547797098 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563c6ec65440 con 0x563c6e7f09f0 2024-07-26T11:16:53.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.508+0000 7f8719600640 1 -- 172.21.15.134:0/3547797098 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f871c0054f0 con 0x563c6e7f09f0 2024-07-26T11:16:53.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.508+0000 7f8719600640 1 -- 172.21.15.134:0/3547797098 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f872002d070 con 0x563c6e7f09f0 2024-07-26T11:16:53.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.508+0000 7f873d82a740 10 client.25648 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:16:53.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.508+0000 7f873d82a740 10 client.25648 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-07-26T11:16:53.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.508+0000 7f873d82a740 20 client.25648 populate_metadata read hostname 'smithi134' 2024-07-26T11:16:53.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.508+0000 7f873d82a740 1 --2- 172.21.15.134:0/3547797098 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x563c6e832070 0x563c6ec67b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:16:53.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.508+0000 7f873d82a740 4 client.25648 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:16:53.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.508+0000 7f873d82a740 1 -- 172.21.15.134:0/3547797098 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563c6eb4b5c0 con 0x563c6e832070 2024-07-26T11:16:53.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.509+0000 7f872f400640 1 --2- 172.21.15.134:0/3547797098 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x563c6e832070 0x563c6ec67b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:16:53.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.509+0000 7f872f400640 1 --2- 172.21.15.134:0/3547797098 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x563c6e832070 0x563c6ec67b70 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b1cda2381f764fc2 server_cookie=9be670b97c4ded94 in_seq=0 out_seq=0 2024-07-26T11:16:53.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.509+0000 7f8719600640 10 client.25648 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:16:53.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.510+0000 7f8719600640 1 -- 172.21.15.134:0/3547797098 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563c6eb4b5c0 con 0x563c6e832070 2024-07-26T11:16:53.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.510+0000 7f8719600640 10 client.25648 handle_command_reply: tid=0 2024-07-26T11:16:53.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.510+0000 7f873d82a740 1 client.25648 shutdown 2024-07-26T11:16:53.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.510+0000 7f873d82a740 1 -- 172.21.15.134:0/3547797098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e7f09f0 msgr2=0x563c6ebf39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:53.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.510+0000 7f873d82a740 1 --2- 172.21.15.134:0/3547797098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e7f09f0 0x563c6ebf39d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f872001a620 tx=0x7f8720009690 comp rx=0 tx=0).stop 2024-07-26T11:16:53.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.510+0000 7f873d82a740 1 -- 172.21.15.134:0/3547797098 shutdown_connections 2024-07-26T11:16:53.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.510+0000 7f873d82a740 1 --2- 172.21.15.134:0/3547797098 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x563c6e832070 0x563c6ec67b70 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.510+0000 7f873d82a740 1 --2- 172.21.15.134:0/3547797098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c6e811390 0x563c6ebf13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.510+0000 7f873d82a740 1 --2- 172.21.15.134:0/3547797098 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c6e8091b0 0x563c6ebf0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.510+0000 7f873d82a740 1 --2- 172.21.15.134:0/3547797098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c6e7f09f0 0x563c6ebf39d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.510+0000 7f873d82a740 1 -- 172.21.15.134:0/3547797098 >> 172.21.15.134:0/3547797098 conn(0x563c6e826460 msgr2=0x563c6ebe0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:53.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.510+0000 7f8719600640 0 client.25648 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:16:53.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.510+0000 7f873d82a740 1 -- 172.21.15.134:0/3547797098 shutdown_connections 2024-07-26T11:16:53.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.511+0000 7f873d82a740 1 -- 172.21.15.134:0/3547797098 wait complete. 2024-07-26T11:16:53.514 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.511+0000 7f873d82a740 20 client.25648 trim_cache size 0 max 16384 2024-07-26T11:16:53.514 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:16:53.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.512+0000 7f8735600640 1 -- 172.21.15.134:0/1857938274 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f87140b3cb0 msgr2=0x7f87140b61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:53.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.512+0000 7f8735600640 1 --2- 172.21.15.134:0/1857938274 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f87140b3cb0 0x7f87140b61c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f871c009230 tx=0x7f871c031400 comp rx=0 tx=0).stop 2024-07-26T11:16:53.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.512+0000 7f8735600640 1 -- 172.21.15.134:0/1857938274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f873010b880 msgr2=0x7f87301a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:16:53.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.512+0000 7f8735600640 1 --2- 172.21.15.134:0/1857938274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f873010b880 0x7f87301a1760 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f872800eb10 tx=0x7f872800c2d0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.512+0000 7f8735600640 1 -- 172.21.15.134:0/1857938274 shutdown_connections 2024-07-26T11:16:53.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.512+0000 7f8735600640 1 --2- 172.21.15.134:0/1857938274 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f87140b3cb0 0x7f87140b61c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.512+0000 7f8735600640 1 --2- 172.21.15.134:0/1857938274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f873010b880 0x7f87301a1760 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.512+0000 7f8735600640 1 --2- 172.21.15.134:0/1857938274 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f873010af30 0x7f873019c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.512+0000 7f8735600640 1 --2- 172.21.15.134:0/1857938274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f87301055f0 0x7f873019bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:16:53.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.512+0000 7f8735600640 1 -- 172.21.15.134:0/1857938274 >> 172.21.15.134:0/1857938274 conn(0x7f8730101060 msgr2=0x7f8730103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:16:53.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.512+0000 7f8735600640 1 -- 172.21.15.134:0/1857938274 shutdown_connections 2024-07-26T11:16:53.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:16:53.513+0000 7f8735600640 1 -- 172.21.15.134:0/1857938274 wait complete. 2024-07-26T11:16:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:54 smithi134 ceph-mon[31966]: pgmap v508: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-07-26T11:16:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:54 smithi189 ceph-mon[36790]: pgmap v508: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-07-26T11:16:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:54 smithi193 ceph-mon[37473]: pgmap v508: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-07-26T11:16:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:56 smithi134 ceph-mon[31966]: pgmap v509: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 11 op/s 2024-07-26T11:16:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:56 smithi189 ceph-mon[36790]: pgmap v509: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 11 op/s 2024-07-26T11:16:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:56 smithi193 ceph-mon[37473]: pgmap v509: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 11 op/s 2024-07-26T11:16:57.416 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-07-26T11:16:57.848 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:16:58.325 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:16:58 smithi134 ceph-mon[31966]: pgmap v510: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 852 B/s wr, 13 op/s 2024-07-26T11:16:58.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:16:58 smithi189 ceph-mon[36790]: pgmap v510: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 852 B/s wr, 13 op/s 2024-07-26T11:16:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:16:58 smithi193 ceph-mon[37473]: pgmap v510: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 852 B/s wr, 13 op/s 2024-07-26T11:17:00.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.404+0000 7f956be00640 1 -- 172.21.15.134:0/3926289631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f956c1033e0 msgr2=0x7f956c1037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:00.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.404+0000 7f956be00640 1 --2- 172.21.15.134:0/3926289631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f956c1033e0 0x7f956c1037f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f955c009900 tx=0x7f955c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.404+0000 7f956be00640 1 -- 172.21.15.134:0/3926289631 shutdown_connections 2024-07-26T11:17:00.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.404+0000 7f956be00640 1 --2- 172.21.15.134:0/3926289631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f956c109670 0x7f956c10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.404+0000 7f956be00640 1 --2- 172.21.15.134:0/3926289631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f956c108d20 0x7f956c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.404+0000 7f956be00640 1 --2- 172.21.15.134:0/3926289631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f956c1033e0 0x7f956c1037f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.404+0000 7f956be00640 1 -- 172.21.15.134:0/3926289631 >> 172.21.15.134:0/3926289631 conn(0x7f956c0fee70 msgr2=0x7f956c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:00.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.404+0000 7f956be00640 1 -- 172.21.15.134:0/3926289631 shutdown_connections 2024-07-26T11:17:00.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.404+0000 7f956be00640 1 -- 172.21.15.134:0/3926289631 wait complete. 2024-07-26T11:17:00.407 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:00 smithi134 ceph-mon[31966]: pgmap v511: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T11:17:00.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.407+0000 7f956be00640 1 Processor -- start 2024-07-26T11:17:00.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.407+0000 7f956be00640 1 -- start start 2024-07-26T11:17:00.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.408+0000 7f956be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f956c1033e0 0x7f956c19bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.408+0000 7f956be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f956c108d20 0x7f956c19c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.408+0000 7f956be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f956c109670 0x7f956c1a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.408+0000 7f956be00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f956c113320 con 0x7f956c109670 2024-07-26T11:17:00.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.408+0000 7f956be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f956c1131a0 con 0x7f956c1033e0 2024-07-26T11:17:00.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.408+0000 7f956be00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f956c1134a0 con 0x7f956c108d20 2024-07-26T11:17:00.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.409+0000 7f956a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f956c108d20 0x7f956c19c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.409+0000 7f956a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f956c108d20 0x7f956c19c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58338/0 (socket says 172.21.15.134:58338) 2024-07-26T11:17:00.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.409+0000 7f956a000640 1 -- 172.21.15.134:0/2328513542 learned_addr learned my addr 172.21.15.134:0/2328513542 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:00.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.410+0000 7f956a000640 1 -- 172.21.15.134:0/2328513542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f956c1033e0 msgr2=0x7f956c19bf70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T11:17:00.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.410+0000 7f956a000640 1 --2- 172.21.15.134:0/2328513542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f956c1033e0 0x7f956c19bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.410+0000 7f956a000640 1 -- 172.21.15.134:0/2328513542 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f956c109670 msgr2=0x7f956c1a17b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-07-26T11:17:00.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.410+0000 7f956a000640 1 --2- 172.21.15.134:0/2328513542 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f956c109670 0x7f956c1a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.410+0000 7f956a000640 1 -- 172.21.15.134:0/2328513542 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956c1a1ee0 con 0x7f956c108d20 2024-07-26T11:17:00.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.411+0000 7f956a000640 1 --2- 172.21.15.134:0/2328513542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f956c108d20 0x7f956c19c510 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9558009ef0 tx=0x7f9558007590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:00.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.411+0000 7f9563400640 1 -- 172.21.15.134:0/2328513542 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9558023070 con 0x7f956c108d20 2024-07-26T11:17:00.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.412+0000 7f956be00640 1 -- 172.21.15.134:0/2328513542 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f956c1a21d0 con 0x7f956c108d20 2024-07-26T11:17:00.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.412+0000 7f9563400640 1 -- 172.21.15.134:0/2328513542 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9558007d60 con 0x7f956c108d20 2024-07-26T11:17:00.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.412+0000 7f9563400640 1 -- 172.21.15.134:0/2328513542 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955801c6a0 con 0x7f956c108d20 2024-07-26T11:17:00.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.412+0000 7f956be00640 1 -- 172.21.15.134:0/2328513542 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f956c109c60 con 0x7f956c108d20 2024-07-26T11:17:00.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.418+0000 7f95784a2740 1 Processor -- start 2024-07-26T11:17:00.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.418+0000 7f95784a2740 1 -- start start 2024-07-26T11:17:00.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.418+0000 7f95784a2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15eb070 0x560fc1990030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.418+0000 7f95784a2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15c21b0 0x560fc1990570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.418+0000 7f95784a2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15ca390 0x560fc1990ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.418+0000 7f95784a2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560fc1906bf0 con 0x560fc15c21b0 2024-07-26T11:17:00.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.418+0000 7f95784a2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560fc18f60f0 con 0x560fc15ca390 2024-07-26T11:17:00.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.418+0000 7f95784a2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560fc18fa130 con 0x560fc15eb070 2024-07-26T11:17:00.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.419+0000 7f956aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15c21b0 0x560fc1990570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.419+0000 7f956aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15c21b0 0x560fc1990570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:55226/0 (socket says 172.21.15.134:55226) 2024-07-26T11:17:00.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.419+0000 7f956aa00640 1 -- 172.21.15.134:0/498474890 learned_addr learned my addr 172.21.15.134:0/498474890 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:00.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.420+0000 7f956b400640 1 --2- 172.21.15.134:0/498474890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15eb070 0x560fc1990030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.420+0000 7f956aa00640 1 -- 172.21.15.134:0/498474890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15eb070 msgr2=0x560fc1990030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:00.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.420+0000 7f956aa00640 1 --2- 172.21.15.134:0/498474890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15eb070 0x560fc1990030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.420+0000 7f956aa00640 1 -- 172.21.15.134:0/498474890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15ca390 msgr2=0x560fc1990ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:17:00.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.420+0000 7f956aa00640 1 --2- 172.21.15.134:0/498474890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15ca390 0x560fc1990ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.420+0000 7f956aa00640 1 -- 172.21.15.134:0/498474890 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fc1656650 con 0x560fc15c21b0 2024-07-26T11:17:00.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.421+0000 7f956aa00640 1 --2- 172.21.15.134:0/498474890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15c21b0 0x560fc1990570 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f955c006010 tx=0x7f955c005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1aac70bbf3c777f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:00.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.422+0000 7f9557e00640 1 -- 172.21.15.134:0/498474890 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c040070 con 0x560fc15c21b0 2024-07-26T11:17:00.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.422+0000 7f9563400640 1 -- 172.21.15.134:0/2328513542 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7f9558026070 con 0x7f956c108d20 2024-07-26T11:17:00.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.422+0000 7f9557e00640 1 -- 172.21.15.134:0/498474890 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f955c032e20 con 0x560fc15c21b0 2024-07-26T11:17:00.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.422+0000 7f9557e00640 1 -- 172.21.15.134:0/498474890 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c035040 con 0x560fc15c21b0 2024-07-26T11:17:00.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.423+0000 7f95784a2740 1 -- 172.21.15.134:0/498474890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15c21b0 msgr2=0x560fc1990570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:00.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.423+0000 7f95784a2740 1 --2- 172.21.15.134:0/498474890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15c21b0 0x560fc1990570 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f955c006010 tx=0x7f955c005e60 comp rx=0 tx=0).stop 2024-07-26T11:17:00.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.423+0000 7f95784a2740 1 -- 172.21.15.134:0/498474890 shutdown_connections 2024-07-26T11:17:00.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.423+0000 7f95784a2740 1 --2- 172.21.15.134:0/498474890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15ca390 0x560fc1990ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.423+0000 7f95784a2740 1 --2- 172.21.15.134:0/498474890 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15c21b0 0x560fc1990570 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.423+0000 7f95784a2740 1 --2- 172.21.15.134:0/498474890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15eb070 0x560fc1990030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.423+0000 7f95784a2740 1 -- 172.21.15.134:0/498474890 >> 172.21.15.134:0/498474890 conn(0x560fc15df460 msgr2=0x560fc198df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:00.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.424+0000 7f95784a2740 1 -- 172.21.15.134:0/498474890 shutdown_connections 2024-07-26T11:17:00.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.424+0000 7f95784a2740 1 -- 172.21.15.134:0/498474890 wait complete. 2024-07-26T11:17:00.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.424+0000 7f95784a2740 1 -- start start 2024-07-26T11:17:00.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.425+0000 7f95784a2740 1 Processor -- start 2024-07-26T11:17:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.425+0000 7f95784a2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15c21b0 0x560fc199b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.425+0000 7f95784a2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15ca390 0x560fc199bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.426+0000 7f95784a2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15eb070 0x560fc199c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.426+0000 7f95784a2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560fc1759700 con 0x560fc15eb070 2024-07-26T11:17:00.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.426+0000 7f95784a2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560fc18926c0 con 0x560fc15ca390 2024-07-26T11:17:00.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.426+0000 7f95784a2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560fc199e7b0 con 0x560fc15c21b0 2024-07-26T11:17:00.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.426+0000 7f9563400640 1 --2- 172.21.15.134:0/2328513542 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95500b3bc0 0x7f95500b60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.426+0000 7f95784a2740 10 client.0 _ll_register_callbacks cb 0x560fc18a2170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:00.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.426+0000 7f9563400640 1 -- 172.21.15.134:0/2328513542 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f95580ebc80 con 0x7f956c108d20 2024-07-26T11:17:00.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.427+0000 7f956a000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15eb070 0x560fc199c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.427+0000 7f956b400640 1 --2- 172.21.15.134:0/2328513542 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95500b3bc0 0x7f95500b60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.427+0000 7f956a000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15eb070 0x560fc199c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:55244/0 (socket says 172.21.15.134:55244) 2024-07-26T11:17:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.427+0000 7f956a000640 1 -- 172.21.15.134:0/3289660840 learned_addr learned my addr 172.21.15.134:0/3289660840 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.427+0000 7f956aa00640 1 --2- 172.21.15.134:0/3289660840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15ca390 0x560fc199bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.428+0000 7f956a000640 1 -- 172.21.15.134:0/3289660840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15c21b0 msgr2=0x560fc199b650 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:17:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.428+0000 7f956a000640 1 --2- 172.21.15.134:0/3289660840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15c21b0 0x560fc199b650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.428+0000 7f956a000640 1 -- 172.21.15.134:0/3289660840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15ca390 msgr2=0x560fc199bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.428+0000 7f956a000640 1 --2- 172.21.15.134:0/3289660840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15ca390 0x560fc199bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.428+0000 7f956a000640 1 -- 172.21.15.134:0/3289660840 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fc18a9860 con 0x560fc15eb070 2024-07-26T11:17:00.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.428+0000 7f956aa00640 1 --2- 172.21.15.134:0/3289660840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15ca390 0x560fc199bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:17:00.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.428+0000 7f956b400640 1 --2- 172.21.15.134:0/2328513542 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95500b3bc0 0x7f95500b60d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9564009270 tx=0x7f9564009b20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:00.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.429+0000 7f956a000640 1 --2- 172.21.15.134:0/3289660840 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15eb070 0x560fc199c0d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9558020000 tx=0x7f95580b21c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=736e2290b987ef90 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:00.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.429+0000 7f9554c00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:17:00.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.429+0000 7f0ecee00640 1 -- 172.21.15.134:0/769562077 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ec8104c80 msgr2=0x7f0ec810d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:00.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.429+0000 7f0ecee00640 1 --2- 172.21.15.134:0/769562077 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ec8104c80 0x7f0ec810d1c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4009900 tx=0x7f0eb402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.429+0000 7f0ecee00640 1 -- 172.21.15.134:0/769562077 shutdown_connections 2024-07-26T11:17:00.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.429+0000 7f0ecee00640 1 --2- 172.21.15.134:0/769562077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ec810d790 0x7f0ec810fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.429+0000 7f0ecee00640 1 --2- 172.21.15.134:0/769562077 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ec8104c80 0x7f0ec810d1c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.429+0000 7f0ecee00640 1 --2- 172.21.15.134:0/769562077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ec8104330 0x7f0ec8104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.429+0000 7f0ecee00640 1 -- 172.21.15.134:0/769562077 >> 172.21.15.134:0/769562077 conn(0x7f0ec807a170 msgr2=0x7f0ec807a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:00.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.429+0000 7f9554c00640 1 -- 172.21.15.134:0/3289660840 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95580bb620 con 0x560fc15eb070 2024-07-26T11:17:00.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.430+0000 7f0ecee00640 1 -- 172.21.15.134:0/769562077 shutdown_connections 2024-07-26T11:17:00.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.430+0000 7f0ecee00640 1 -- 172.21.15.134:0/769562077 wait complete. 2024-07-26T11:17:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.429+0000 7f9554c00640 1 -- 172.21.15.134:0/3289660840 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f95580b17d0 con 0x560fc15eb070 2024-07-26T11:17:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.430+0000 7f95784a2740 1 -- 172.21.15.134:0/3289660840 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f955c032e20 con 0x560fc15eb070 2024-07-26T11:17:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.430+0000 7f9554c00640 1 -- 172.21.15.134:0/3289660840 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f95580b19d0 con 0x560fc15eb070 2024-07-26T11:17:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.430+0000 7f95784a2740 10 client.15477 fetch_fsmap learned FSMap version 30 2024-07-26T11:17:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.430+0000 7f95784a2740 1 -- 172.21.15.134:0/3289660840 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560fc19aa450 con 0x560fc15eb070 2024-07-26T11:17:00.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.431+0000 7f9554c00640 1 -- 172.21.15.134:0/3289660840 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f955c032e20 con 0x560fc15eb070 2024-07-26T11:17:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.431+0000 7f0ecee00640 1 Processor -- start 2024-07-26T11:17:00.435 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.431+0000 7f0ecee00640 1 -- start start 2024-07-26T11:17:00.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.431+0000 7f9554c00640 1 -- 172.21.15.134:0/3289660840 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f95580b7d90 con 0x560fc15eb070 2024-07-26T11:17:00.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.431+0000 7f95784a2740 10 client.15477 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:17:00.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.431+0000 7f95784a2740 10 client.15477 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T11:17:00.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.431+0000 7f95784a2740 20 client.15477 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:00.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.431+0000 7f95784a2740 1 --2- 172.21.15.134:0/3289660840 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x560fc15a99f0 0x560fc19ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.431+0000 7f95784a2740 4 client.15477 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:00.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.431+0000 7f95784a2740 1 -- 172.21.15.134:0/3289660840 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560fc1908b40 con 0x560fc15a99f0 2024-07-26T11:17:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.432+0000 7f0ecee00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ec8104330 0x7f0ec8109cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.432+0000 7f0ecee00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ec810d790 0x7f0ec810a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.432+0000 7f0ecee00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ec81a9f80 0x7f0ec81ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.432+0000 7f0ecee00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec8102d40 con 0x7f0ec810d790 2024-07-26T11:17:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.432+0000 7f0ecee00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec8102bc0 con 0x7f0ec8104330 2024-07-26T11:17:00.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.432+0000 7f0ecee00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec8102ec0 con 0x7f0ec81a9f80 2024-07-26T11:17:00.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.432+0000 7f956aa00640 1 --2- 172.21.15.134:0/3289660840 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x560fc15a99f0 0x560fc19ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.433+0000 7f0ec7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ec8104330 0x7f0ec8109cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.433+0000 7f0ecce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ec81a9f80 0x7f0ec81ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.433+0000 7f0ecce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ec81a9f80 0x7f0ec81ac3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58386/0 (socket says 172.21.15.134:58386) 2024-07-26T11:17:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.433+0000 7f0ecce00640 1 -- 172.21.15.134:0/1303591377 learned_addr learned my addr 172.21.15.134:0/1303591377 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.433+0000 7f0ec7400640 1 --2- 172.21.15.134:0/1303591377 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ec810d790 0x7f0ec810a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.433+0000 7f0ecce00640 1 -- 172.21.15.134:0/1303591377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ec8104330 msgr2=0x7f0ec8109cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.433+0000 7f0ecce00640 1 --2- 172.21.15.134:0/1303591377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ec8104330 0x7f0ec8109cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.433+0000 7f0ecce00640 1 -- 172.21.15.134:0/1303591377 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ec810d790 msgr2=0x7f0ec810a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.434+0000 7f0ecce00640 1 --2- 172.21.15.134:0/1303591377 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ec810d790 0x7f0ec810a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.434+0000 7f0ecce00640 1 -- 172.21.15.134:0/1303591377 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec81aca70 con 0x7f0ec81a9f80 2024-07-26T11:17:00.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.433+0000 7f956aa00640 1 --2- 172.21.15.134:0/3289660840 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x560fc15a99f0 0x560fc19ad8d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6b9fdda06aa0e889 server_cookie=4ec2e8b3016c3421 in_seq=0 out_seq=0 2024-07-26T11:17:00.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.433+0000 7f9554c00640 10 client.15477 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:17:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.434+0000 7f0ecce00640 1 --2- 172.21.15.134:0/1303591377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ec81a9f80 0x7f0ec81ac3a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000ead0 tx=0x7f0ec000ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.434+0000 7f0ec4c00640 1 -- 172.21.15.134:0/1303591377 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec000cde0 con 0x7f0ec81a9f80 2024-07-26T11:17:00.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.434+0000 7f9554c00640 1 -- 172.21.15.134:0/3289660840 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560fc1908b40 con 0x560fc15a99f0 2024-07-26T11:17:00.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.434+0000 7f9554c00640 10 client.15477 handle_command_reply: tid=0 2024-07-26T11:17:00.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.434+0000 7f95784a2740 1 client.15477 shutdown 2024-07-26T11:17:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.435+0000 7f0ecee00640 1 -- 172.21.15.134:0/1303591377 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ec81acd60 con 0x7f0ec81a9f80 2024-07-26T11:17:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.435+0000 7f0ec4c00640 1 -- 172.21.15.134:0/1303591377 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ec0005020 con 0x7f0ec81a9f80 2024-07-26T11:17:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.435+0000 7f0ecee00640 1 -- 172.21.15.134:0/1303591377 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ec81ad220 con 0x7f0ec81a9f80 2024-07-26T11:17:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.435+0000 7f0ec4c00640 1 -- 172.21.15.134:0/1303591377 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec00116d0 con 0x7f0ec81a9f80 2024-07-26T11:17:00.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.436+0000 7f95784a2740 1 -- 172.21.15.134:0/3289660840 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15eb070 msgr2=0x560fc199c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:00.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.436+0000 7f95784a2740 1 --2- 172.21.15.134:0/3289660840 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15eb070 0x560fc199c0d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9558020000 tx=0x7f95580b21c0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.436+0000 7f95784a2740 1 -- 172.21.15.134:0/3289660840 shutdown_connections 2024-07-26T11:17:00.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.436+0000 7f95784a2740 1 --2- 172.21.15.134:0/3289660840 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x560fc15a99f0 0x560fc19ad8d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.436+0000 7f9554c00640 0 client.15477 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:17:00.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.436+0000 7f95784a2740 1 --2- 172.21.15.134:0/3289660840 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15eb070 0x560fc199c0d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.436+0000 7f95784a2740 1 --2- 172.21.15.134:0/3289660840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15ca390 0x560fc199bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.436+0000 7f95784a2740 1 --2- 172.21.15.134:0/3289660840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15c21b0 0x560fc199b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.436+0000 7f95784a2740 1 -- 172.21.15.134:0/3289660840 >> 172.21.15.134:0/3289660840 conn(0x560fc15df460 msgr2=0x560fc19931c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:00.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.436+0000 7f95784a2740 1 -- 172.21.15.134:0/3289660840 shutdown_connections 2024-07-26T11:17:00.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.437+0000 7f95784a2740 1 -- 172.21.15.134:0/3289660840 wait complete. 2024-07-26T11:17:00.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.438+0000 7f95784a2740 20 client.15477 trim_cache size 0 max 16384 2024-07-26T11:17:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.439+0000 7f0ed5e40740 1 Processor -- start 2024-07-26T11:17:00.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:00 smithi189 ceph-mon[36790]: pgmap v511: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T11:17:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.441+0000 7f0ec4c00640 1 -- 172.21.15.134:0/1303591377 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 24) ==== 151134+0+0 (secure 0 0 0) 0x7f0ec0011870 con 0x7f0ec81a9f80 2024-07-26T11:17:00.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.441+0000 7f0ed5e40740 1 -- start start 2024-07-26T11:17:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.441+0000 7f0ed5e40740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d4da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.441+0000 7f0ed5e40740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99812f0 0x55a7a9d4dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.441+0000 7f0ed5e40740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99894d0 0x55a7a9d4e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.441+0000 7f0ed5e40740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a7a9cc2270 con 0x55a7a99894d0 2024-07-26T11:17:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.441+0000 7f0ed5e40740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a7a9cb1770 con 0x55a7a99812f0 2024-07-26T11:17:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.441+0000 7f0ed5e40740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a7a9cb57b0 con 0x55a7a99aa1b0 2024-07-26T11:17:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.442+0000 7f0ec7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d4da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.442+0000 7f0ec7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d4da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58398/0 (socket says 172.21.15.134:58398) 2024-07-26T11:17:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.442+0000 7f0ec7400640 1 -- 172.21.15.134:0/3830140358 learned_addr learned my addr 172.21.15.134:0/3830140358 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.442+0000 7f0ec7400640 1 --2- 172.21.15.134:0/3830140358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99894d0 0x55a7a9d4e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.442+0000 7f0ec7400640 1 -- 172.21.15.134:0/3830140358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99812f0 msgr2=0x55a7a9d4dfa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:17:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.442+0000 7f0ec7400640 1 --2- 172.21.15.134:0/3830140358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99812f0 0x55a7a9d4dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.443+0000 7f0ec7400640 1 -- 172.21.15.134:0/3830140358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99894d0 msgr2=0x55a7a9d4e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.443+0000 7f0ec7400640 1 --2- 172.21.15.134:0/3830140358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99894d0 0x55a7a9d4e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.443+0000 7f0ec7400640 1 -- 172.21.15.134:0/3830140358 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7a9a15790 con 0x55a7a99aa1b0 2024-07-26T11:17:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.443+0000 7f0ec4c00640 1 --2- 172.21.15.134:0/1303591377 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0eac0b3d30 0x7f0eac0b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.443+0000 7f0ec7400640 1 --2- 172.21.15.134:0/3830140358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d4da60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4009900 tx=0x7f0eb4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79b85fc6084fc655 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:00.446 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.443+0000 7f0ec4c00640 1 -- 172.21.15.134:0/1303591377 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0ec00e4420 con 0x7f0ec81a9f80 2024-07-26T11:17:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.444+0000 7f0ec7e00640 1 --2- 172.21.15.134:0/1303591377 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0eac0b3d30 0x7f0eac0b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.444+0000 7f0ebd600640 1 -- 172.21.15.134:0/3830140358 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4044020 con 0x55a7a99aa1b0 2024-07-26T11:17:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.444+0000 7f0ebd600640 1 -- 172.21.15.134:0/3830140358 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0eb4003ca0 con 0x55a7a99aa1b0 2024-07-26T11:17:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.444+0000 7f0ebd600640 1 -- 172.21.15.134:0/3830140358 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb4035070 con 0x55a7a99aa1b0 2024-07-26T11:17:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.445+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3830140358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99aa1b0 msgr2=0x55a7a9d4da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:00.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.445+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/3830140358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d4da60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0eb4009900 tx=0x7f0eb4000bc0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.445+0000 7f0ec7e00640 1 --2- 172.21.15.134:0/1303591377 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0eac0b3d30 0x7f0eac0b6240 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x55a7a9d4c6e0 tx=0x7f0eb800b390 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.446+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3830140358 shutdown_connections 2024-07-26T11:17:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.446+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/3830140358 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99894d0 0x55a7a9d4e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.446+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/3830140358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99812f0 0x55a7a9d4dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.446+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/3830140358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d4da60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.446+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3830140358 >> 172.21.15.134:0/3830140358 conn(0x55a7a999e300 msgr2=0x55a7a9d4b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.446+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3830140358 shutdown_connections 2024-07-26T11:17:00.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.446+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3830140358 wait complete. 2024-07-26T11:17:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.448+0000 7f0ed5e40740 1 -- start start 2024-07-26T11:17:00.450 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.449+0000 7f0ed5e40740 1 Processor -- start 2024-07-26T11:17:00.451 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.449+0000 7f0ed5e40740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99812f0 0x55a7a9d59650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.449+0000 7f0ed5e40740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99894d0 0x55a7a9d59bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.449+0000 7f0ed5e40740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d764b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.449+0000 7f0ed5e40740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a7a9bebab0 con 0x55a7a99812f0 2024-07-26T11:17:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.449+0000 7f0ed5e40740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a7a9c4ed50 con 0x55a7a99aa1b0 2024-07-26T11:17:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.449+0000 7f0ed5e40740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a7a9d6d190 con 0x55a7a99894d0 2024-07-26T11:17:00.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.449+0000 7f0ed5e40740 10 client.0 _ll_register_callbacks cb 0x55a7a99c1a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.450+0000 7f0ec7e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99812f0 0x55a7a9d59650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.450+0000 7f0ec7e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99812f0 0x55a7a9d59650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:55286/0 (socket says 172.21.15.134:55286) 2024-07-26T11:17:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.450+0000 7f0ec7e00640 1 -- 172.21.15.134:0/3752897900 learned_addr learned my addr 172.21.15.134:0/3752897900 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.450+0000 7f0ec7400640 1 --2- 172.21.15.134:0/3752897900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99894d0 0x55a7a9d59bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.450+0000 7f0ec7400640 1 -- 172.21.15.134:0/3752897900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99aa1b0 msgr2=0x55a7a9d764b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.450+0000 7f0ec7400640 1 --2- 172.21.15.134:0/3752897900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d764b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.450+0000 7f0ec7400640 1 -- 172.21.15.134:0/3752897900 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99812f0 msgr2=0x55a7a9d59650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.450+0000 7f0ec7400640 1 --2- 172.21.15.134:0/3752897900 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99812f0 0x55a7a9d59650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.450+0000 7f0ec7400640 1 -- 172.21.15.134:0/3752897900 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7a9a8e620 con 0x55a7a99894d0 2024-07-26T11:17:00.453 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.451+0000 7f0ec7400640 1 --2- 172.21.15.134:0/3752897900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99894d0 0x55a7a9d59bb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0ec806c3f0 tx=0x7f0eb4003f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c4943232a89f806 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.451+0000 7f0eb2000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:17:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.451+0000 7f0eb2000640 1 -- 172.21.15.134:0/3752897900 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb404e060 con 0x55a7a99894d0 2024-07-26T11:17:00.454 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.451+0000 7f0eb2000640 1 -- 172.21.15.134:0/3752897900 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0eb4003a00 con 0x55a7a99894d0 2024-07-26T11:17:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.452+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3752897900 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0eb4003ca0 con 0x55a7a99894d0 2024-07-26T11:17:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.452+0000 7f0eb2000640 1 -- 172.21.15.134:0/3752897900 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0eb4044070 con 0x55a7a99894d0 2024-07-26T11:17:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.452+0000 7f0ed5e40740 10 client.25669 fetch_fsmap learned FSMap version 30 2024-07-26T11:17:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.452+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3752897900 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a7a9d79220 con 0x55a7a99894d0 2024-07-26T11:17:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.452+0000 7f0eb2000640 1 -- 172.21.15.134:0/3752897900 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0eb4003ca0 con 0x55a7a99894d0 2024-07-26T11:17:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.453+0000 7f0eb2000640 1 -- 172.21.15.134:0/3752897900 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f0eb40046c0 con 0x55a7a99894d0 2024-07-26T11:17:00.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.453+0000 7f0ed5e40740 10 client.25669 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:17:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.453+0000 7f0ed5e40740 10 client.25669 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:17:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.453+0000 7f0ed5e40740 20 client.25669 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.453+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/3752897900 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a7a9968d80 0x55a7a9d7c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.453+0000 7f0ed5e40740 4 client.25669 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.453+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3752897900 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7a9cc41c0 con 0x55a7a9968d80 2024-07-26T11:17:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.454+0000 7f0ec7e00640 1 --2- 172.21.15.134:0/3752897900 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a7a9968d80 0x55a7a9d7c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.456 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.455+0000 7f0ec7e00640 1 --2- 172.21.15.134:0/3752897900 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a7a9968d80 0x55a7a9d7c580 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57a80fae569c3474 server_cookie=ce322d2dc7a2fd4b in_seq=0 out_seq=0 2024-07-26T11:17:00.457 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.455+0000 7f0eb2000640 10 client.25669 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:00.458 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.456+0000 7f0eb2000640 1 -- 172.21.15.134:0/3752897900 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a7a9cc41c0 con 0x55a7a9968d80 2024-07-26T11:17:00.458 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.456+0000 7f0eb2000640 10 client.25669 handle_command_reply: tid=0 2024-07-26T11:17:00.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.456+0000 7f95784a2740 1 Processor -- start 2024-07-26T11:17:00.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.456+0000 7f95784a2740 1 -- start start 2024-07-26T11:17:00.459 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.456+0000 7f0ed5e40740 1 client.25669 shutdown 2024-07-26T11:17:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.456+0000 7f95784a2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15c21b0 0x560fc199d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.456+0000 7f95784a2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15ca390 0x560fc198c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.456+0000 7f95784a2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15eb070 0x560fc1991770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.456+0000 7f95784a2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f956c1ae580 con 0x560fc15ca390 2024-07-26T11:17:00.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.456+0000 7f95784a2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f956c1b00c0 con 0x560fc15c21b0 2024-07-26T11:17:00.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.456+0000 7f95784a2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f956c1aec50 con 0x560fc15eb070 2024-07-26T11:17:00.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3752897900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99894d0 msgr2=0x55a7a9d59bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:00.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/3752897900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99894d0 0x55a7a9d59bb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0ec806c3f0 tx=0x7f0eb4003f40 comp rx=0 tx=0).stop 2024-07-26T11:17:00.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3752897900 shutdown_connections 2024-07-26T11:17:00.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/3752897900 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a7a9968d80 0x55a7a9d7c580 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/3752897900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99894d0 0x55a7a9d59bb0 secure :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0ec806c3f0 tx=0x7f0eb4003f40 comp rx=0 tx=0).stop 2024-07-26T11:17:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/3752897900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d764b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/3752897900 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99812f0 0x55a7a9d59650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3752897900 >> 172.21.15.134:0/3752897900 conn(0x55a7a999e300 msgr2=0x55a7a9d4ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:00.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f0eb2000640 0 client.25669 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:00.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f956aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15c21b0 0x560fc199d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f956aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15c21b0 0x560fc199d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42558/0 (socket says 172.21.15.134:42558) 2024-07-26T11:17:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f956aa00640 1 -- 172.21.15.134:0/1187972072 learned_addr learned my addr 172.21.15.134:0/1187972072 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f956b400640 1 --2- 172.21.15.134:0/1187972072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15eb070 0x560fc1991770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f956aa00640 1 -- 172.21.15.134:0/1187972072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15eb070 msgr2=0x560fc1991770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f956aa00640 1 --2- 172.21.15.134:0/1187972072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15eb070 0x560fc1991770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f956aa00640 1 -- 172.21.15.134:0/1187972072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15ca390 msgr2=0x560fc198c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f956a000640 1 --2- 172.21.15.134:0/1187972072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15ca390 0x560fc198c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f956aa00640 1 --2- 172.21.15.134:0/1187972072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15ca390 0x560fc198c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f956aa00640 1 -- 172.21.15.134:0/1187972072 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fc198e890 con 0x560fc15c21b0 2024-07-26T11:17:00.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.457+0000 7f956a000640 1 --2- 172.21.15.134:0/1187972072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15ca390 0x560fc198c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:17:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.458+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3752897900 shutdown_connections 2024-07-26T11:17:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.458+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3752897900 wait complete. 2024-07-26T11:17:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.458+0000 7f0ed5e40740 20 client.25669 trim_cache size 0 max 16384 2024-07-26T11:17:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.458+0000 7f956aa00640 1 --2- 172.21.15.134:0/1187972072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15c21b0 0x560fc199d530 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f955c009a20 tx=0x7f955c03c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9504b141fa12677a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.459+0000 7f9557400640 1 -- 172.21.15.134:0/1187972072 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c051070 con 0x560fc15c21b0 2024-07-26T11:17:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.459+0000 7f9557400640 1 -- 172.21.15.134:0/1187972072 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f955c004040 con 0x560fc15c21b0 2024-07-26T11:17:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.459+0000 7f9557400640 1 -- 172.21.15.134:0/1187972072 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c035040 con 0x560fc15c21b0 2024-07-26T11:17:00.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.460+0000 7f95784a2740 1 -- 172.21.15.134:0/1187972072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15c21b0 msgr2=0x560fc199d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.460+0000 7f95784a2740 1 --2- 172.21.15.134:0/1187972072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15c21b0 0x560fc199d530 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f955c009a20 tx=0x7f955c03c040 comp rx=0 tx=0).stop 2024-07-26T11:17:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.460+0000 7f95784a2740 1 -- 172.21.15.134:0/1187972072 shutdown_connections 2024-07-26T11:17:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.460+0000 7f95784a2740 1 --2- 172.21.15.134:0/1187972072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15eb070 0x560fc1991770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.460+0000 7f95784a2740 1 --2- 172.21.15.134:0/1187972072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15ca390 0x560fc198c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.460+0000 7f95784a2740 1 --2- 172.21.15.134:0/1187972072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15c21b0 0x560fc199d530 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.460+0000 7f95784a2740 1 -- 172.21.15.134:0/1187972072 >> 172.21.15.134:0/1187972072 conn(0x560fc15df460 msgr2=0x560fc1991060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.461+0000 7f95784a2740 1 -- 172.21.15.134:0/1187972072 shutdown_connections 2024-07-26T11:17:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.461+0000 7f95784a2740 1 -- 172.21.15.134:0/1187972072 wait complete. 2024-07-26T11:17:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.461+0000 7f95784a2740 1 -- start start 2024-07-26T11:17:00.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.462+0000 7f95784a2740 1 Processor -- start 2024-07-26T11:17:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.463+0000 7f95784a2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15c21b0 0x560fc19a4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.463+0000 7f95784a2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15ca390 0x560fc19a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.463+0000 7f95784a2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15eb070 0x560fc19a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.463+0000 7f95784a2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f956c1af9f0 con 0x560fc15ca390 2024-07-26T11:17:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.463+0000 7f95784a2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f956c1af320 con 0x560fc15c21b0 2024-07-26T11:17:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.463+0000 7f95784a2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f956c1ac560 con 0x560fc15eb070 2024-07-26T11:17:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.463+0000 7f95784a2740 10 client.0 _ll_register_callbacks cb 0x7f95580b1db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.463+0000 7f956aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15c21b0 0x560fc19a4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.463+0000 7f956aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15c21b0 0x560fc19a4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42562/0 (socket says 172.21.15.134:42562) 2024-07-26T11:17:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.463+0000 7f956aa00640 1 -- 172.21.15.134:0/3578154412 learned_addr learned my addr 172.21.15.134:0/3578154412 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.463+0000 7f956aa00640 1 -- 172.21.15.134:0/3578154412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15eb070 msgr2=0x560fc19a1b20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:17:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.463+0000 7f956aa00640 1 --2- 172.21.15.134:0/3578154412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15eb070 0x560fc19a1b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.464+0000 7f956aa00640 1 -- 172.21.15.134:0/3578154412 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15ca390 msgr2=0x560fc19a15e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:17:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.464+0000 7f956a000640 1 --2- 172.21.15.134:0/3578154412 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15ca390 0x560fc19a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.464+0000 7f956aa00640 1 --2- 172.21.15.134:0/3578154412 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15ca390 0x560fc19a15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.464+0000 7f956aa00640 1 -- 172.21.15.134:0/3578154412 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560fc19a2260 con 0x560fc15c21b0 2024-07-26T11:17:00.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.464+0000 7f956a000640 1 --2- 172.21.15.134:0/3578154412 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15ca390 0x560fc19a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:17:00.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.464+0000 7f956aa00640 1 --2- 172.21.15.134:0/3578154412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15c21b0 0x560fc19a4110 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f955c02a430 tx=0x7f955c032e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f7db7b74f5c6046 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:00.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.464+0000 7f9554c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:17:00.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.465+0000 7f9554c00640 1 -- 172.21.15.134:0/3578154412 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c05a070 con 0x560fc15c21b0 2024-07-26T11:17:00.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.465+0000 7f9554c00640 1 -- 172.21.15.134:0/3578154412 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f955c0317a0 con 0x560fc15c21b0 2024-07-26T11:17:00.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.465+0000 7f95784a2740 1 -- 172.21.15.134:0/3578154412 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f955c004040 con 0x560fc15c21b0 2024-07-26T11:17:00.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.465+0000 7f9554c00640 1 -- 172.21.15.134:0/3578154412 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f955c065050 con 0x560fc15c21b0 2024-07-26T11:17:00.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.466+0000 7f95784a2740 10 client.25652 fetch_fsmap learned FSMap version 30 2024-07-26T11:17:00.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.466+0000 7f95784a2740 1 -- 172.21.15.134:0/3578154412 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560fc1a15f10 con 0x560fc15c21b0 2024-07-26T11:17:00.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.466+0000 7f9554c00640 1 -- 172.21.15.134:0/3578154412 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f955c004040 con 0x560fc15c21b0 2024-07-26T11:17:00.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.466+0000 7f9554c00640 1 -- 172.21.15.134:0/3578154412 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f955c04d4d0 con 0x560fc15c21b0 2024-07-26T11:17:00.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.467+0000 7f95784a2740 10 client.25652 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:17:00.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.467+0000 7f95784a2740 10 client.25652 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T11:17:00.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.467+0000 7f95784a2740 20 client.25652 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:00.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.467+0000 7f95784a2740 1 --2- 172.21.15.134:0/3578154412 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x560fc157eda0 0x560fc1a181f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.467+0000 7f95784a2740 4 client.25652 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:17:00.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.467+0000 7f95784a2740 1 -- 172.21.15.134:0/3578154412 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560fc19045e0 con 0x560fc157eda0 2024-07-26T11:17:00.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.469+0000 7f956a000640 1 --2- 172.21.15.134:0/3578154412 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x560fc157eda0 0x560fc1a181f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.469+0000 7f956a000640 1 --2- 172.21.15.134:0/3578154412 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x560fc157eda0 0x560fc1a181f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=763f26487ed9ebfe server_cookie=ec7710f2481a80a4 in_seq=0 out_seq=0 2024-07-26T11:17:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.469+0000 7f9554c00640 10 client.25652 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:17:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.470+0000 7f9554c00640 1 -- 172.21.15.134:0/3578154412 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560fc19045e0 con 0x560fc157eda0 2024-07-26T11:17:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.470+0000 7f9554c00640 10 client.25652 handle_command_reply: tid=0 2024-07-26T11:17:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.470+0000 7f95784a2740 1 client.25652 shutdown 2024-07-26T11:17:00.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.470+0000 7f95784a2740 1 -- 172.21.15.134:0/3578154412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15c21b0 msgr2=0x560fc19a4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:00.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.470+0000 7f95784a2740 1 --2- 172.21.15.134:0/3578154412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15c21b0 0x560fc19a4110 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f955c02a430 tx=0x7f955c032e90 comp rx=0 tx=0).stop 2024-07-26T11:17:00.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.470+0000 7f95784a2740 1 -- 172.21.15.134:0/3578154412 shutdown_connections 2024-07-26T11:17:00.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.470+0000 7f95784a2740 1 --2- 172.21.15.134:0/3578154412 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x560fc157eda0 0x560fc1a181f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.471+0000 7f95784a2740 1 --2- 172.21.15.134:0/3578154412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560fc15eb070 0x560fc19a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.471+0000 7f95784a2740 1 --2- 172.21.15.134:0/3578154412 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560fc15ca390 0x560fc19a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.471+0000 7f95784a2740 1 --2- 172.21.15.134:0/3578154412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560fc15c21b0 0x560fc19a4110 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.471+0000 7f95784a2740 1 -- 172.21.15.134:0/3578154412 >> 172.21.15.134:0/3578154412 conn(0x560fc15df460 msgr2=0x560fc199bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:00.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.471+0000 7f9554c00640 0 client.25652 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:17:00.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.471+0000 7f95784a2740 1 -- 172.21.15.134:0/3578154412 shutdown_connections 2024-07-26T11:17:00.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.471+0000 7f95784a2740 1 -- 172.21.15.134:0/3578154412 wait complete. 2024-07-26T11:17:00.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.471+0000 7f95784a2740 20 client.25652 trim_cache size 0 max 16384 2024-07-26T11:17:00.474 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:17:00.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.472+0000 7f956be00640 1 -- 172.21.15.134:0/2328513542 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95500b3bc0 msgr2=0x7f95500b60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:00.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.472+0000 7f956be00640 1 --2- 172.21.15.134:0/2328513542 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95500b3bc0 0x7f95500b60d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9564009270 tx=0x7f9564009b20 comp rx=0 tx=0).stop 2024-07-26T11:17:00.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.472+0000 7f956be00640 1 -- 172.21.15.134:0/2328513542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f956c108d20 msgr2=0x7f956c19c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:00.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.472+0000 7f956be00640 1 --2- 172.21.15.134:0/2328513542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f956c108d20 0x7f956c19c510 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9558009ef0 tx=0x7f9558007590 comp rx=0 tx=0).stop 2024-07-26T11:17:00.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.473+0000 7f956be00640 1 -- 172.21.15.134:0/2328513542 shutdown_connections 2024-07-26T11:17:00.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.473+0000 7f956be00640 1 --2- 172.21.15.134:0/2328513542 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95500b3bc0 0x7f95500b60d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.473+0000 7f956be00640 1 --2- 172.21.15.134:0/2328513542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f956c108d20 0x7f956c19c510 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.473+0000 7f956be00640 1 --2- 172.21.15.134:0/2328513542 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f956c109670 0x7f956c1a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.473+0000 7f956be00640 1 --2- 172.21.15.134:0/2328513542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f956c1033e0 0x7f956c19bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.473+0000 7f956be00640 1 -- 172.21.15.134:0/2328513542 >> 172.21.15.134:0/2328513542 conn(0x7f956c0fee70 msgr2=0x7f956c10e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:00.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.474+0000 7f956be00640 1 -- 172.21.15.134:0/2328513542 shutdown_connections 2024-07-26T11:17:00.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:00.474+0000 7f956be00640 1 -- 172.21.15.134:0/2328513542 wait complete. 2024-07-26T11:17:00.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.480+0000 7f0ed5e40740 1 Processor -- start 2024-07-26T11:17:00.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.480+0000 7f0ed5e40740 1 -- start start 2024-07-26T11:17:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.480+0000 7f0ed5e40740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99812f0 0x55a7a9d4f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.480+0000 7f0ed5e40740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d81b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.480+0000 7f0ed5e40740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a993e1c0 0x55a7a9d82060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.480+0000 7f0ed5e40740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec81aeb10 con 0x55a7a99aa1b0 2024-07-26T11:17:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.480+0000 7f0ed5e40740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec81b0650 con 0x55a7a99812f0 2024-07-26T11:17:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.480+0000 7f0ed5e40740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec81af1e0 con 0x55a7a993e1c0 2024-07-26T11:17:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.481+0000 7f0ecce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a993e1c0 0x55a7a9d82060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.481+0000 7f0ecce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a993e1c0 0x55a7a9d82060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58422/0 (socket says 172.21.15.134:58422) 2024-07-26T11:17:00.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.481+0000 7f0ecce00640 1 -- 172.21.15.134:0/4020394985 learned_addr learned my addr 172.21.15.134:0/4020394985 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.481+0000 7f0ec7e00640 1 --2- 172.21.15.134:0/4020394985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99812f0 0x55a7a9d4f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.481+0000 7f0ec7400640 1 --2- 172.21.15.134:0/4020394985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d81b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.481+0000 7f0ecce00640 1 -- 172.21.15.134:0/4020394985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99812f0 msgr2=0x55a7a9d4f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.481+0000 7f0ecce00640 1 --2- 172.21.15.134:0/4020394985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99812f0 0x55a7a9d4f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.481+0000 7f0ecce00640 1 -- 172.21.15.134:0/4020394985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99aa1b0 msgr2=0x55a7a9d81b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.481+0000 7f0ecce00640 1 --2- 172.21.15.134:0/4020394985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d81b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.481+0000 7f0ecce00640 1 -- 172.21.15.134:0/4020394985 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7a9d47370 con 0x55a7a993e1c0 2024-07-26T11:17:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.481+0000 7f0ec7400640 1 --2- 172.21.15.134:0/4020394985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d81b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:17:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.482+0000 7f0ecce00640 1 --2- 172.21.15.134:0/4020394985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a993e1c0 0x55a7a9d82060 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00aaac0 tx=0x7f0ec00aa840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef73599d3e39d85 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.482+0000 7f0ebcc00640 1 -- 172.21.15.134:0/4020394985 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec001f040 con 0x55a7a993e1c0 2024-07-26T11:17:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.482+0000 7f0ebcc00640 1 -- 172.21.15.134:0/4020394985 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ec00051d0 con 0x55a7a993e1c0 2024-07-26T11:17:00.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.482+0000 7f0ebcc00640 1 -- 172.21.15.134:0/4020394985 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0090e60 con 0x55a7a993e1c0 2024-07-26T11:17:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.483+0000 7f0ed5e40740 1 -- 172.21.15.134:0/4020394985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a993e1c0 msgr2=0x55a7a9d82060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.483+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/4020394985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a993e1c0 0x55a7a9d82060 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00aaac0 tx=0x7f0ec00aa840 comp rx=0 tx=0).stop 2024-07-26T11:17:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.483+0000 7f0ed5e40740 1 -- 172.21.15.134:0/4020394985 shutdown_connections 2024-07-26T11:17:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.483+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/4020394985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a993e1c0 0x55a7a9d82060 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.483+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/4020394985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d81b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.483+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/4020394985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99812f0 0x55a7a9d4f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.483+0000 7f0ed5e40740 1 -- 172.21.15.134:0/4020394985 >> 172.21.15.134:0/4020394985 conn(0x55a7a999e300 msgr2=0x55a7a9d4ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.483+0000 7f0ed5e40740 1 -- 172.21.15.134:0/4020394985 shutdown_connections 2024-07-26T11:17:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.483+0000 7f0ed5e40740 1 -- 172.21.15.134:0/4020394985 wait complete. 2024-07-26T11:17:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.484+0000 7f0ed5e40740 1 -- start start 2024-07-26T11:17:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.484+0000 7f0ed5e40740 1 Processor -- start 2024-07-26T11:17:00.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.484+0000 7f0ed5e40740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a993e1c0 0x55a7a9d6d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.484+0000 7f0ed5e40740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99812f0 0x55a7a9d6d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.484+0000 7f0ed5e40740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d6de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.485+0000 7f0ed5e40740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec81aff80 con 0x55a7a993e1c0 2024-07-26T11:17:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.485+0000 7f0ed5e40740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec81af8b0 con 0x55a7a99812f0 2024-07-26T11:17:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.485+0000 7f0ed5e40740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec811e800 con 0x55a7a99aa1b0 2024-07-26T11:17:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.485+0000 7f0ed5e40740 10 client.0 _ll_register_callbacks cb 0x7f0eb40043b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.485+0000 7f0ec7e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a993e1c0 0x55a7a9d6d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.485+0000 7f0ec7e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a993e1c0 0x55a7a9d6d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:55318/0 (socket says 172.21.15.134:55318) 2024-07-26T11:17:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.485+0000 7f0ecce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d6de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.485+0000 7f0ec7e00640 1 -- 172.21.15.134:0/3608979927 learned_addr learned my addr 172.21.15.134:0/3608979927 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.485+0000 7f0ec7400640 1 --2- 172.21.15.134:0/3608979927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99812f0 0x55a7a9d6d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.486+0000 7f0ecce00640 1 -- 172.21.15.134:0/3608979927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99812f0 msgr2=0x55a7a9d6d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.486+0000 7f0ecce00640 1 --2- 172.21.15.134:0/3608979927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99812f0 0x55a7a9d6d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.486+0000 7f0ecce00640 1 -- 172.21.15.134:0/3608979927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a993e1c0 msgr2=0x55a7a9d6d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.486+0000 7f0ecce00640 1 --2- 172.21.15.134:0/3608979927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a993e1c0 0x55a7a9d6d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.486+0000 7f0ecce00640 1 -- 172.21.15.134:0/3608979927 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7a9de2b30 con 0x55a7a99aa1b0 2024-07-26T11:17:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.486+0000 7f0ecce00640 1 --2- 172.21.15.134:0/3608979927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d6de50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00a6e40 tx=0x7f0ec00aaf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fad5293d95e0a1d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.486+0000 7f0eb2000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:17:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.486+0000 7f0eb2000640 1 -- 172.21.15.134:0/3608979927 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec001f070 con 0x55a7a99aa1b0 2024-07-26T11:17:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.486+0000 7f0eb2000640 1 -- 172.21.15.134:0/3608979927 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0ec0011b60 con 0x55a7a99aa1b0 2024-07-26T11:17:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.486+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3608979927 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ec00051d0 con 0x55a7a99aa1b0 2024-07-26T11:17:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.487+0000 7f0eb2000640 1 -- 172.21.15.134:0/3608979927 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0ec0090e60 con 0x55a7a99aa1b0 2024-07-26T11:17:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.487+0000 7f0ed5e40740 10 client.25675 fetch_fsmap learned FSMap version 30 2024-07-26T11:17:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.487+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3608979927 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55a7a9de3e20 con 0x55a7a99aa1b0 2024-07-26T11:17:00.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.487+0000 7f0eb2000640 1 -- 172.21.15.134:0/3608979927 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ec00051d0 con 0x55a7a99aa1b0 2024-07-26T11:17:00.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.488+0000 7f0eb2000640 1 -- 172.21.15.134:0/3608979927 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f0ec000f040 con 0x55a7a99aa1b0 2024-07-26T11:17:00.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.488+0000 7f0ed5e40740 10 client.25675 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:17:00.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.488+0000 7f0ed5e40740 10 client.25675 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:17:00.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.488+0000 7f0ed5e40740 20 client.25675 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:00.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.488+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/3608979927 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a7a9de4430 0x55a7a9de6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:00.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.488+0000 7f0ed5e40740 4 client.25675 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:17:00.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.488+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3608979927 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a7a9cbfc60 con 0x55a7a9de4430 2024-07-26T11:17:00.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.489+0000 7f0ec7e00640 1 --2- 172.21.15.134:0/3608979927 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a7a9de4430 0x55a7a9de6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:00.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.489+0000 7f0eb2000640 10 client.25675 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:00.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.489+0000 7f0ec7e00640 1 --2- 172.21.15.134:0/3608979927 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a7a9de4430 0x55a7a9de6850 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d862ec292b74eef2 server_cookie=4a29184d77520192 in_seq=0 out_seq=0 2024-07-26T11:17:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.494+0000 7f0eb2000640 1 -- 172.21.15.134:0/3608979927 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a7a9cbfc60 con 0x55a7a9de4430 2024-07-26T11:17:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.494+0000 7f0eb2000640 10 client.25675 handle_command_reply: tid=0 2024-07-26T11:17:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.494+0000 7f0ed5e40740 1 client.25675 shutdown 2024-07-26T11:17:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.495+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3608979927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99aa1b0 msgr2=0x55a7a9d6de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.495+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/3608979927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d6de50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00a6e40 tx=0x7f0ec00aaf40 comp rx=0 tx=0).stop 2024-07-26T11:17:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.495+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3608979927 shutdown_connections 2024-07-26T11:17:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.495+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/3608979927 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a7a9de4430 0x55a7a9de6850 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.495+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/3608979927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a7a99aa1b0 0x55a7a9d6de50 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.495+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/3608979927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a7a99812f0 0x55a7a9d6d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.495+0000 7f0ed5e40740 1 --2- 172.21.15.134:0/3608979927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a7a993e1c0 0x55a7a9d6d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.495+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3608979927 >> 172.21.15.134:0/3608979927 conn(0x55a7a999e300 msgr2=0x55a7a9d5a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.495+0000 7f0eb2000640 0 client.25675 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.495+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3608979927 shutdown_connections 2024-07-26T11:17:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.495+0000 7f0ed5e40740 1 -- 172.21.15.134:0/3608979927 wait complete. 2024-07-26T11:17:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.495+0000 7f0ed5e40740 20 client.25675 trim_cache size 0 max 16384 2024-07-26T11:17:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:17:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:17:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:17:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:17:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.497+0000 7f0ecee00640 1 -- 172.21.15.134:0/1303591377 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0eac0b3d30 msgr2=0x7f0eac0b6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.497+0000 7f0ecee00640 1 --2- 172.21.15.134:0/1303591377 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0eac0b3d30 0x7f0eac0b6240 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x55a7a9d4c6e0 tx=0x7f0eb800b390 comp rx=0 tx=0).stop 2024-07-26T11:17:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.497+0000 7f0ecee00640 1 -- 172.21.15.134:0/1303591377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ec81a9f80 msgr2=0x7f0ec81ac3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.497+0000 7f0ecee00640 1 --2- 172.21.15.134:0/1303591377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ec81a9f80 0x7f0ec81ac3a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000ead0 tx=0x7f0ec000ef90 comp rx=0 tx=0).stop 2024-07-26T11:17:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.497+0000 7f0ecee00640 1 -- 172.21.15.134:0/1303591377 shutdown_connections 2024-07-26T11:17:00.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.497+0000 7f0ecee00640 1 --2- 172.21.15.134:0/1303591377 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0eac0b3d30 0x7f0eac0b6240 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.497+0000 7f0ecee00640 1 --2- 172.21.15.134:0/1303591377 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0ec81a9f80 0x7f0ec81ac3a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.497+0000 7f0ecee00640 1 --2- 172.21.15.134:0/1303591377 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0ec810d790 0x7f0ec810a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.497+0000 7f0ecee00640 1 --2- 172.21.15.134:0/1303591377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ec8104330 0x7f0ec8109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:00.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.497+0000 7f0ecee00640 1 -- 172.21.15.134:0/1303591377 >> 172.21.15.134:0/1303591377 conn(0x7f0ec807a170 msgr2=0x7f0ec8101600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:00.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.497+0000 7f0ecee00640 1 -- 172.21.15.134:0/1303591377 shutdown_connections 2024-07-26T11:17:00.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:00.497+0000 7f0ecee00640 1 -- 172.21.15.134:0/1303591377 wait complete. 2024-07-26T11:17:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:00 smithi193 ceph-mon[37473]: pgmap v511: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-07-26T11:17:01.173 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:17:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:17:00.895+0000 7f8ff6c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:17:01.173 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:17:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:17:00.895+0000 7f8ff6c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:17:01.173 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:17:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:17:00.895+0000 7f8ff6c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:17:01.173 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:17:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:17:00.895+0000 7f8ff6c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:17:01.174 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:17:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:17:00.895+0000 7f8ff6c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:17:01.425 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-26T11:17:01.886 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:17:02.401 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:02 smithi134 ceph-mon[31966]: pgmap v512: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 22 op/s 2024-07-26T11:17:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:02 smithi189 ceph-mon[36790]: pgmap v512: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 22 op/s 2024-07-26T11:17:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:02 smithi193 ceph-mon[37473]: pgmap v512: 129 pgs: 129 active+clean; 1.4 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 22 op/s 2024-07-26T11:17:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.337+0000 7f4b87e00640 1 -- 172.21.15.134:0/1997936286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b88109670 msgr2=0x7f4b8810fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.337+0000 7f4b87e00640 1 --2- 172.21.15.134:0/1997936286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b88109670 0x7f4b8810fea0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4b80009ca0 tx=0x7f4b8002a8c0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.337+0000 7f4b87e00640 1 -- 172.21.15.134:0/1997936286 shutdown_connections 2024-07-26T11:17:04.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.337+0000 7f4b87e00640 1 --2- 172.21.15.134:0/1997936286 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b88108d20 0x7f4b88109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.337+0000 7f4b87e00640 1 --2- 172.21.15.134:0/1997936286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b88109670 0x7f4b8810fea0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.337+0000 7f4b87e00640 1 --2- 172.21.15.134:0/1997936286 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b881033e0 0x7f4b881037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.337+0000 7f4b87e00640 1 -- 172.21.15.134:0/1997936286 >> 172.21.15.134:0/1997936286 conn(0x7f4b880fee70 msgr2=0x7f4b881012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:04.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.337+0000 7f4b87e00640 1 -- 172.21.15.134:0/1997936286 shutdown_connections 2024-07-26T11:17:04.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.337+0000 7f4b87e00640 1 -- 172.21.15.134:0/1997936286 wait complete. 2024-07-26T11:17:04.341 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:04 smithi134 ceph-mon[31966]: mgrmap e25: x(active, since 14m), standbys: z, y 2024-07-26T11:17:04.341 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:04 smithi134 ceph-mon[31966]: pgmap v513: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-07-26T11:17:04.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.339+0000 7f4b87e00640 1 Processor -- start 2024-07-26T11:17:04.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.340+0000 7f4b87e00640 1 -- start start 2024-07-26T11:17:04.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.340+0000 7f4b87e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b881033e0 0x7f4b881a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.340+0000 7f4b87e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b88108d20 0x7f4b881a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.340+0000 7f4b87e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b88109670 0x7f4b881ac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.340+0000 7f4b87e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4b88113390 con 0x7f4b881033e0 2024-07-26T11:17:04.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.340+0000 7f4b87e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4b88113210 con 0x7f4b88109670 2024-07-26T11:17:04.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.340+0000 7f4b87e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4b88113510 con 0x7f4b88108d20 2024-07-26T11:17:04.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.341+0000 7f4b87400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b88109670 0x7f4b881ac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.341+0000 7f4b87400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b88109670 0x7f4b881ac0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42628/0 (socket says 172.21.15.134:42628) 2024-07-26T11:17:04.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.341+0000 7f4b87400640 1 -- 172.21.15.134:0/3809846924 learned_addr learned my addr 172.21.15.134:0/3809846924 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:04.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.341+0000 7f4b86a00640 1 --2- 172.21.15.134:0/3809846924 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b881033e0 0x7f4b881a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.342+0000 7f4b87400640 1 -- 172.21.15.134:0/3809846924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b88108d20 msgr2=0x7f4b881a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.342+0000 7f4b86000640 1 --2- 172.21.15.134:0/3809846924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b88108d20 0x7f4b881a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.342+0000 7f4b87400640 1 --2- 172.21.15.134:0/3809846924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b88108d20 0x7f4b881a15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.342+0000 7f4b87400640 1 -- 172.21.15.134:0/3809846924 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b881033e0 msgr2=0x7f4b881a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.342+0000 7f4b87400640 1 --2- 172.21.15.134:0/3809846924 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b881033e0 0x7f4b881a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.342+0000 7f4b87400640 1 -- 172.21.15.134:0/3809846924 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b881089c0 con 0x7f4b88109670 2024-07-26T11:17:04.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.342+0000 7f4b87400640 1 --2- 172.21.15.134:0/3809846924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b88109670 0x7f4b881ac0f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4b8002f540 tx=0x7f4b800316f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:04.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.342+0000 7f4b7f400640 1 -- 172.21.15.134:0/3809846924 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b80003b70 con 0x7f4b88109670 2024-07-26T11:17:04.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.343+0000 7f4b7f400640 1 -- 172.21.15.134:0/3809846924 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b80003d10 con 0x7f4b88109670 2024-07-26T11:17:04.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.343+0000 7f4b7f400640 1 -- 172.21.15.134:0/3809846924 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b8002f990 con 0x7f4b88109670 2024-07-26T11:17:04.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.343+0000 7f4b87e00640 1 -- 172.21.15.134:0/3809846924 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b881ac630 con 0x7f4b88109670 2024-07-26T11:17:04.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.343+0000 7f4b87e00640 1 -- 172.21.15.134:0/3809846924 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b881acaf0 con 0x7f4b88109670 2024-07-26T11:17:04.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.347+0000 7f4b94264740 1 Processor -- start 2024-07-26T11:17:04.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.349+0000 7f4b7f400640 1 -- 172.21.15.134:0/3809846924 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f4b8002fb30 con 0x7f4b88109670 2024-07-26T11:17:04.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.349+0000 7f4b94264740 1 -- start start 2024-07-26T11:17:04.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.349+0000 7f4b94264740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f69070 0x55f65430e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.349+0000 7f4b94264740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 0x55f65430e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.349+0000 7f4b94264740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 0x55f65430ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.349+0000 7f4b94264740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f654284bd0 con 0x55f653f48390 2024-07-26T11:17:04.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.349+0000 7f4b94264740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f6542740d0 con 0x55f653f69070 2024-07-26T11:17:04.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.349+0000 7f4b94264740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f654278110 con 0x55f653f401b0 2024-07-26T11:17:04.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.350+0000 7f4b86000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f69070 0x55f65430e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.350+0000 7f4b86000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f69070 0x55f65430e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42632/0 (socket says 172.21.15.134:42632) 2024-07-26T11:17:04.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.350+0000 7f4b86000640 1 -- 172.21.15.134:0/1583579955 learned_addr learned my addr 172.21.15.134:0/1583579955 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:04.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.350+0000 7f4b86a00640 1 --2- 172.21.15.134:0/1583579955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 0x55f65430e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.350+0000 7f4b86000640 1 --2- 172.21.15.134:0/1583579955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 0x55f65430ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.351+0000 7f4b86a00640 1 -- 172.21.15.134:0/1583579955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f69070 msgr2=0x55f65430e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:04.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.351+0000 7f4b86a00640 1 --2- 172.21.15.134:0/1583579955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f69070 0x55f65430e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.351+0000 7f4b86a00640 1 -- 172.21.15.134:0/1583579955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 msgr2=0x55f65430ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:04.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.351+0000 7f4b86a00640 1 --2- 172.21.15.134:0/1583579955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 0x55f65430ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.351+0000 7f4b86a00640 1 -- 172.21.15.134:0/1583579955 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f653fd4650 con 0x55f653f401b0 2024-07-26T11:17:04.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.351+0000 7f4b7f400640 1 --2- 172.21.15.134:0/3809846924 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b6c0b3dc0 0x7f4b6c0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.351+0000 7f4b7f400640 1 -- 172.21.15.134:0/3809846924 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4b80105780 con 0x7f4b88109670 2024-07-26T11:17:04.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.351+0000 7f4b86a00640 1 --2- 172.21.15.134:0/1583579955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 0x55f65430e550 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4b78009b40 tx=0x7f4b780113e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee23900188fc2b57 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:04.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.352+0000 7f4b73400640 1 -- 172.21.15.134:0/1583579955 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b78011e10 con 0x55f653f401b0 2024-07-26T11:17:04.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.352+0000 7f4b86000640 1 --2- 172.21.15.134:0/3809846924 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b6c0b3dc0 0x7f4b6c0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.352+0000 7f4b73400640 1 -- 172.21.15.134:0/1583579955 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b78012900 con 0x55f653f401b0 2024-07-26T11:17:04.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.352+0000 7f4b73400640 1 -- 172.21.15.134:0/1583579955 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7800f860 con 0x55f653f401b0 2024-07-26T11:17:04.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.353+0000 7f4b94264740 1 -- 172.21.15.134:0/1583579955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 msgr2=0x55f65430e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.354+0000 7f4b94264740 1 --2- 172.21.15.134:0/1583579955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 0x55f65430e550 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4b78009b40 tx=0x7f4b780113e0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.354+0000 7f4b86000640 1 --2- 172.21.15.134:0/3809846924 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b6c0b3dc0 0x7f4b6c0b62d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4b74012640 tx=0x7f4b74005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:04.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.354+0000 7f4b94264740 1 -- 172.21.15.134:0/1583579955 shutdown_connections 2024-07-26T11:17:04.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.354+0000 7f4b94264740 1 --2- 172.21.15.134:0/1583579955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 0x55f65430e550 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.354+0000 7f4b94264740 1 --2- 172.21.15.134:0/1583579955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 0x55f65430ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.354+0000 7f4b94264740 1 --2- 172.21.15.134:0/1583579955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f69070 0x55f65430e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.354+0000 7f4b94264740 1 -- 172.21.15.134:0/1583579955 >> 172.21.15.134:0/1583579955 conn(0x55f653f5d460 msgr2=0x55f65430bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:04.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.354+0000 7f4b94264740 1 -- 172.21.15.134:0/1583579955 shutdown_connections 2024-07-26T11:17:04.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.354+0000 7f4b94264740 1 -- 172.21.15.134:0/1583579955 wait complete. 2024-07-26T11:17:04.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.355+0000 7f4b94264740 1 -- start start 2024-07-26T11:17:04.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.355+0000 7f4b94264740 1 Processor -- start 2024-07-26T11:17:04.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.356+0000 7f4b94264740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 0x55f6543195a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.356+0000 7f4b94264740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 0x55f654319ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.356+0000 7f4b94264740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f69070 0x55f65431a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.356+0000 7f4b94264740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f6540d7700 con 0x55f653f48390 2024-07-26T11:17:04.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.356+0000 7f4b94264740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f6542106a0 con 0x55f653f69070 2024-07-26T11:17:04.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.356+0000 7f4b94264740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f65431c7f0 con 0x55f653f401b0 2024-07-26T11:17:04.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.356+0000 7f4b94264740 10 client.0 _ll_register_callbacks cb 0x55f654220150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:04.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.356+0000 7f4b86a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 0x55f6543195a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.356+0000 7f4b86000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 0x55f654319ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.357+0000 7f4b86000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 0x55f654319ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:55372/0 (socket says 172.21.15.134:55372) 2024-07-26T11:17:04.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.357+0000 7f4b86000640 1 -- 172.21.15.134:0/3882033418 learned_addr learned my addr 172.21.15.134:0/3882033418 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:04.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.357+0000 7f4b86a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 0x55f6543195a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58486/0 (socket says 172.21.15.134:58486) 2024-07-26T11:17:04.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.357+0000 7f4b87400640 1 --2- 172.21.15.134:0/3882033418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f69070 0x55f65431a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.357+0000 7f4b86000640 1 -- 172.21.15.134:0/3882033418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 msgr2=0x55f6543195a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:04.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.357+0000 7f4b86000640 1 --2- 172.21.15.134:0/3882033418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 0x55f6543195a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.357+0000 7f4b86000640 1 -- 172.21.15.134:0/3882033418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f69070 msgr2=0x55f65431a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:04.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.357+0000 7f4b86000640 1 --2- 172.21.15.134:0/3882033418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f69070 0x55f65431a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.357+0000 7f4b86000640 1 -- 172.21.15.134:0/3882033418 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f654227840 con 0x55f653f48390 2024-07-26T11:17:04.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.358+0000 7f4b86000640 1 --2- 172.21.15.134:0/3882033418 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 0x55f654319ae0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4b74003870 tx=0x7f4b74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938e2af2e7152624 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:04.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.358+0000 7f4b67e00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:17:04.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.358+0000 7f4b67e00640 1 -- 172.21.15.134:0/3882033418 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74025020 con 0x55f653f48390 2024-07-26T11:17:04.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.358+0000 7f4b94264740 1 -- 172.21.15.134:0/3882033418 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b78012900 con 0x55f653f48390 2024-07-26T11:17:04.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.358+0000 7f4b67e00640 1 -- 172.21.15.134:0/3882033418 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4b74011a80 con 0x55f653f48390 2024-07-26T11:17:04.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.359+0000 7f4b67e00640 1 -- 172.21.15.134:0/3882033418 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4b74030050 con 0x55f653f48390 2024-07-26T11:17:04.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.359+0000 7f4b94264740 10 client.15510 fetch_fsmap learned FSMap version 30 2024-07-26T11:17:04.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.359+0000 7f4b94264740 1 -- 172.21.15.134:0/3882033418 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f6543283d0 con 0x55f653f48390 2024-07-26T11:17:04.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.359+0000 7f4b67e00640 1 -- 172.21.15.134:0/3882033418 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b78012900 con 0x55f653f48390 2024-07-26T11:17:04.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.361+0000 7f4b67e00640 1 -- 172.21.15.134:0/3882033418 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f4b7400f3c0 con 0x55f653f48390 2024-07-26T11:17:04.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.361+0000 7f4b94264740 10 client.15510 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:17:04.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.361+0000 7f4b94264740 10 client.15510 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T11:17:04.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.361+0000 7f4b94264740 20 client.15510 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:04.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.361+0000 7f4b94264740 1 --2- 172.21.15.134:0/3882033418 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55f653f279f0 0x55f65432b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.361+0000 7f4b94264740 4 client.15510 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:04.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.361+0000 7f4b94264740 1 -- 172.21.15.134:0/3882033418 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f654286b20 con 0x55f653f279f0 2024-07-26T11:17:04.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.361+0000 7f4b86a00640 1 --2- 172.21.15.134:0/3882033418 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55f653f279f0 0x55f65432b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.362+0000 7f4b86a00640 1 --2- 172.21.15.134:0/3882033418 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55f653f279f0 0x55f65432b9d0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d94628e7b539fa6c server_cookie=85baa3a3a9b24c7a in_seq=0 out_seq=0 2024-07-26T11:17:04.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.362+0000 7f4b67e00640 10 client.15510 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:17:04.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.363+0000 7f4b67e00640 1 -- 172.21.15.134:0/3882033418 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f654286b20 con 0x55f653f279f0 2024-07-26T11:17:04.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.363+0000 7f4b67e00640 10 client.15510 handle_command_reply: tid=0 2024-07-26T11:17:04.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.363+0000 7f4b94264740 1 client.15510 shutdown 2024-07-26T11:17:04.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.363+0000 7f4b94264740 1 -- 172.21.15.134:0/3882033418 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 msgr2=0x55f654319ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.363+0000 7f4b94264740 1 --2- 172.21.15.134:0/3882033418 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 0x55f654319ae0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4b74003870 tx=0x7f4b74004730 comp rx=0 tx=0).stop 2024-07-26T11:17:04.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.363+0000 7f4b94264740 1 -- 172.21.15.134:0/3882033418 shutdown_connections 2024-07-26T11:17:04.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.363+0000 7f4b94264740 1 --2- 172.21.15.134:0/3882033418 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55f653f279f0 0x55f65432b9d0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.363+0000 7f4b67e00640 0 client.15510 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:17:04.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.363+0000 7f4b94264740 1 --2- 172.21.15.134:0/3882033418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f69070 0x55f65431a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.363+0000 7f4b94264740 1 --2- 172.21.15.134:0/3882033418 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 0x55f654319ae0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.363+0000 7f4b94264740 1 --2- 172.21.15.134:0/3882033418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 0x55f6543195a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.363+0000 7f4b94264740 1 -- 172.21.15.134:0/3882033418 >> 172.21.15.134:0/3882033418 conn(0x55f653f5d460 msgr2=0x55f6543111f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:04.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.364+0000 7f4b94264740 1 -- 172.21.15.134:0/3882033418 shutdown_connections 2024-07-26T11:17:04.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.364+0000 7f4b94264740 1 -- 172.21.15.134:0/3882033418 wait complete. 2024-07-26T11:17:04.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.365+0000 7f4b94264740 20 client.15510 trim_cache size 0 max 16384 2024-07-26T11:17:04.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.391+0000 7f4b94264740 1 Processor -- start 2024-07-26T11:17:04.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.391+0000 7f4b94264740 1 -- start start 2024-07-26T11:17:04.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.392+0000 7f4b94264740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f279f0 0x55f654319fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.392+0000 7f4b94264740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 0x55f65431b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.392+0000 7f4b94264740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 0x55f65430f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.392+0000 7f4b94264740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4b881b6c30 con 0x55f653f48390 2024-07-26T11:17:04.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.392+0000 7f4b94264740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4b881b8770 con 0x55f653f279f0 2024-07-26T11:17:04.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.392+0000 7f4b94264740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4b881b7300 con 0x55f653f401b0 2024-07-26T11:17:04.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.392+0000 7f4b86a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f279f0 0x55f654319fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.392+0000 7f4b86a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f279f0 0x55f654319fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42650/0 (socket says 172.21.15.134:42650) 2024-07-26T11:17:04.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.392+0000 7f4b86a00640 1 -- 172.21.15.134:0/2198428145 learned_addr learned my addr 172.21.15.134:0/2198428145 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:04.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.392+0000 7f4b87400640 1 --2- 172.21.15.134:0/2198428145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 0x55f65430f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.392+0000 7f4b86000640 1 --2- 172.21.15.134:0/2198428145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 0x55f65431b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.393+0000 7f4b87400640 1 -- 172.21.15.134:0/2198428145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 msgr2=0x55f65431b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:04.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.393+0000 7f4b87400640 1 --2- 172.21.15.134:0/2198428145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 0x55f65431b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.393+0000 7f4b87400640 1 -- 172.21.15.134:0/2198428145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f279f0 msgr2=0x55f654319fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:04.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.393+0000 7f4b87400640 1 --2- 172.21.15.134:0/2198428145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f279f0 0x55f654319fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.393+0000 7f4b87400640 1 -- 172.21.15.134:0/2198428145 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f654310e70 con 0x55f653f48390 2024-07-26T11:17:04.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.393+0000 7f4b87400640 1 --2- 172.21.15.134:0/2198428145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 0x55f65430f750 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4b80102c00 tx=0x7f4b80009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4d38c1be5f21b7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:04.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.394+0000 7f4b72a00640 1 -- 172.21.15.134:0/2198428145 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b800cc7c0 con 0x55f653f48390 2024-07-26T11:17:04.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.394+0000 7f4b72a00640 1 -- 172.21.15.134:0/2198428145 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b80036070 con 0x55f653f48390 2024-07-26T11:17:04.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.394+0000 7f4b72a00640 1 -- 172.21.15.134:0/2198428145 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b800cc320 con 0x55f653f48390 2024-07-26T11:17:04.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.395+0000 7f4b94264740 1 -- 172.21.15.134:0/2198428145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 msgr2=0x55f65430f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.395+0000 7f4b94264740 1 --2- 172.21.15.134:0/2198428145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 0x55f65430f750 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4b80102c00 tx=0x7f4b80009900 comp rx=0 tx=0).stop 2024-07-26T11:17:04.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.395+0000 7f4b94264740 1 -- 172.21.15.134:0/2198428145 shutdown_connections 2024-07-26T11:17:04.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.396+0000 7f4b94264740 1 --2- 172.21.15.134:0/2198428145 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f401b0 0x55f65431b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.396+0000 7f4b94264740 1 --2- 172.21.15.134:0/2198428145 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f48390 0x55f65430f750 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.396+0000 7f4b94264740 1 --2- 172.21.15.134:0/2198428145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f279f0 0x55f654319fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.396+0000 7f4b94264740 1 -- 172.21.15.134:0/2198428145 >> 172.21.15.134:0/2198428145 conn(0x55f653f5d460 msgr2=0x55f65430f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:04.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.396+0000 7f4b94264740 1 -- 172.21.15.134:0/2198428145 shutdown_connections 2024-07-26T11:17:04.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.396+0000 7f4b94264740 1 -- 172.21.15.134:0/2198428145 wait complete. 2024-07-26T11:17:04.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.396+0000 7f4b94264740 1 -- start start 2024-07-26T11:17:04.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.397+0000 7f4b94264740 1 Processor -- start 2024-07-26T11:17:04.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.398+0000 7f4b94264740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f279f0 0x55f654322030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.398+0000 7f4b94264740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f401b0 0x55f65431f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.398+0000 7f4b94264740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f48390 0x55f65431fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.398+0000 7f4b94264740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4b881b80a0 con 0x55f653f279f0 2024-07-26T11:17:04.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.398+0000 7f4b94264740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4b881b79d0 con 0x55f653f401b0 2024-07-26T11:17:04.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.398+0000 7f4b94264740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4b881b4c10 con 0x55f653f48390 2024-07-26T11:17:04.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.398+0000 7f4b94264740 10 client.0 _ll_register_callbacks cb 0x7f4b74004400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:04.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.398+0000 7f4b86a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f279f0 0x55f654322030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.398+0000 7f4b86a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f279f0 0x55f654322030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:55388/0 (socket says 172.21.15.134:55388) 2024-07-26T11:17:04.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.398+0000 7f4b86a00640 1 -- 172.21.15.134:0/2822101862 learned_addr learned my addr 172.21.15.134:0/2822101862 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:04.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.398+0000 7f4b86000640 1 --2- 172.21.15.134:0/2822101862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f401b0 0x55f65431f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.398+0000 7f4b87400640 1 --2- 172.21.15.134:0/2822101862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f48390 0x55f65431fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.399+0000 7f4b86a00640 1 -- 172.21.15.134:0/2822101862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f48390 msgr2=0x55f65431fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:04.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.399+0000 7f4b86a00640 1 --2- 172.21.15.134:0/2822101862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f48390 0x55f65431fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.399+0000 7f4b86a00640 1 -- 172.21.15.134:0/2822101862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f401b0 msgr2=0x55f65431f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:04.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.399+0000 7f4b86a00640 1 --2- 172.21.15.134:0/2822101862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f401b0 0x55f65431f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.399+0000 7f4b86a00640 1 -- 172.21.15.134:0/2822101862 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f654315130 con 0x55f653f279f0 2024-07-26T11:17:04.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.399+0000 7f4b86a00640 1 --2- 172.21.15.134:0/2822101862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f279f0 0x55f654322030 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4b78009630 tx=0x7f4b78009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988ddc1cb0a6df69 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:04.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.399+0000 7f4b67e00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:17:04.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.400+0000 7f4b67e00640 1 -- 172.21.15.134:0/2822101862 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b78005400 con 0x55f653f279f0 2024-07-26T11:17:04.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.400+0000 7f4b67e00640 1 -- 172.21.15.134:0/2822101862 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4b7802a070 con 0x55f653f279f0 2024-07-26T11:17:04.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.400+0000 7f4b94264740 1 -- 172.21.15.134:0/2822101862 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b80036070 con 0x55f653f279f0 2024-07-26T11:17:04.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.400+0000 7f4b67e00640 1 -- 172.21.15.134:0/2822101862 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4b780055a0 con 0x55f653f279f0 2024-07-26T11:17:04.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.400+0000 7f4b94264740 10 client.15516 fetch_fsmap learned FSMap version 30 2024-07-26T11:17:04.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.400+0000 7f4b94264740 1 -- 172.21.15.134:0/2822101862 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f654393df0 con 0x55f653f279f0 2024-07-26T11:17:04.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.401+0000 7f4b67e00640 1 -- 172.21.15.134:0/2822101862 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b80036070 con 0x55f653f279f0 2024-07-26T11:17:04.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.401+0000 7f4b67e00640 1 -- 172.21.15.134:0/2822101862 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f4b7800f840 con 0x55f653f279f0 2024-07-26T11:17:04.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.401+0000 7f4b94264740 10 client.15516 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:17:04.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.401+0000 7f4b94264740 10 client.15516 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T11:17:04.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.401+0000 7f4b94264740 20 client.15516 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:04.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.401+0000 7f4b94264740 1 --2- 172.21.15.134:0/2822101862 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55f653f69070 0x55f654396370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:04.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.401+0000 7f4b94264740 4 client.15516 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:17:04.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.401+0000 7f4b94264740 1 -- 172.21.15.134:0/2822101862 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f65430b3a0 con 0x55f653f69070 2024-07-26T11:17:04.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.401+0000 7f4b86000640 1 --2- 172.21.15.134:0/2822101862 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55f653f69070 0x55f654396370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:04.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.402+0000 7f4b86000640 1 --2- 172.21.15.134:0/2822101862 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55f653f69070 0x55f654396370 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6fbd79d5421f3314 server_cookie=bece451534fcc9a0 in_seq=0 out_seq=0 2024-07-26T11:17:04.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.402+0000 7f4b67e00640 10 client.15516 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:17:04.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.402+0000 7f4b67e00640 1 -- 172.21.15.134:0/2822101862 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f65430b3a0 con 0x55f653f69070 2024-07-26T11:17:04.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.402+0000 7f4b67e00640 10 client.15516 handle_command_reply: tid=0 2024-07-26T11:17:04.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.402+0000 7f4b94264740 1 client.15516 shutdown 2024-07-26T11:17:04.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.403+0000 7f4b94264740 1 -- 172.21.15.134:0/2822101862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f279f0 msgr2=0x55f654322030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.403+0000 7f4b94264740 1 --2- 172.21.15.134:0/2822101862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f279f0 0x55f654322030 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4b78009630 tx=0x7f4b78009860 comp rx=0 tx=0).stop 2024-07-26T11:17:04.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.403+0000 7f4b94264740 1 -- 172.21.15.134:0/2822101862 shutdown_connections 2024-07-26T11:17:04.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.403+0000 7f4b94264740 1 --2- 172.21.15.134:0/2822101862 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55f653f69070 0x55f654396370 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.403+0000 7f4b94264740 1 --2- 172.21.15.134:0/2822101862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f653f48390 0x55f65431fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.403+0000 7f4b94264740 1 --2- 172.21.15.134:0/2822101862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f653f401b0 0x55f65431f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.403+0000 7f4b94264740 1 --2- 172.21.15.134:0/2822101862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f653f279f0 0x55f654322030 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.403+0000 7f4b94264740 1 -- 172.21.15.134:0/2822101862 >> 172.21.15.134:0/2822101862 conn(0x55f653f5d460 msgr2=0x55f65431b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:04.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.403+0000 7f4b67e00640 0 client.15516 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:17:04.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.403+0000 7f4b94264740 1 -- 172.21.15.134:0/2822101862 shutdown_connections 2024-07-26T11:17:04.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.403+0000 7f4b94264740 1 -- 172.21.15.134:0/2822101862 wait complete. 2024-07-26T11:17:04.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.403+0000 7f4b94264740 20 client.15516 trim_cache size 0 max 16384 2024-07-26T11:17:04.415 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:17:04.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.404+0000 7f4b87e00640 1 -- 172.21.15.134:0/3809846924 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b6c0b3dc0 msgr2=0x7f4b6c0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.404+0000 7f4b87e00640 1 --2- 172.21.15.134:0/3809846924 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b6c0b3dc0 0x7f4b6c0b62d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4b74012640 tx=0x7f4b74005f90 comp rx=0 tx=0).stop 2024-07-26T11:17:04.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.404+0000 7f4b87e00640 1 -- 172.21.15.134:0/3809846924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b88109670 msgr2=0x7f4b881ac0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:04.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.404+0000 7f4b87e00640 1 --2- 172.21.15.134:0/3809846924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b88109670 0x7f4b881ac0f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4b8002f540 tx=0x7f4b800316f0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.405+0000 7f4b87e00640 1 -- 172.21.15.134:0/3809846924 shutdown_connections 2024-07-26T11:17:04.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.405+0000 7f4b87e00640 1 --2- 172.21.15.134:0/3809846924 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4b6c0b3dc0 0x7f4b6c0b62d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.405+0000 7f4b87e00640 1 --2- 172.21.15.134:0/3809846924 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4b88108d20 0x7f4b881a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.405+0000 7f4b87e00640 1 --2- 172.21.15.134:0/3809846924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b88109670 0x7f4b881ac0f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.405+0000 7f4b87e00640 1 --2- 172.21.15.134:0/3809846924 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4b881033e0 0x7f4b881a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:04.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.405+0000 7f4b87e00640 1 -- 172.21.15.134:0/3809846924 >> 172.21.15.134:0/3809846924 conn(0x7f4b880fee70 msgr2=0x7f4b881082b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:04.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.405+0000 7f4b87e00640 1 -- 172.21.15.134:0/3809846924 shutdown_connections 2024-07-26T11:17:04.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:04.405+0000 7f4b87e00640 1 -- 172.21.15.134:0/3809846924 wait complete. 2024-07-26T11:17:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:04 smithi189 ceph-mon[36790]: mgrmap e25: x(active, since 14m), standbys: z, y 2024-07-26T11:17:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:04 smithi189 ceph-mon[36790]: pgmap v513: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-07-26T11:17:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:04 smithi193 ceph-mon[37473]: mgrmap e25: x(active, since 14m), standbys: z, y 2024-07-26T11:17:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:04 smithi193 ceph-mon[37473]: pgmap v513: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 26 op/s 2024-07-26T11:17:05.475 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-26T11:17:05.926 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:17:06.418 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:06 smithi134 ceph-mon[31966]: pgmap v514: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-07-26T11:17:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:06 smithi189 ceph-mon[36790]: pgmap v514: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-07-26T11:17:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:06 smithi193 ceph-mon[37473]: pgmap v514: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-07-26T11:17:08.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.278+0000 7f6ab3e00640 1 -- 172.21.15.134:0/3475174837 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6aac10fb50 msgr2=0x7f6aac111f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:08.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.278+0000 7f6ab3e00640 1 --2- 172.21.15.134:0/3475174837 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6aac10fb50 0x7f6aac111f70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4009ca0 tx=0x7f6aa402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.278+0000 7f6ab3e00640 1 -- 172.21.15.134:0/3475174837 shutdown_connections 2024-07-26T11:17:08.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.278+0000 7f6ab3e00640 1 --2- 172.21.15.134:0/3475174837 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6aac10fb50 0x7f6aac111f70 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.278+0000 7f6ab3e00640 1 --2- 172.21.15.134:0/3475174837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6aac075ad0 0x7f6aac10f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.278+0000 7f6ab3e00640 1 --2- 172.21.15.134:0/3475174837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6aac075180 0x7f6aac075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.278+0000 7f6ab3e00640 1 -- 172.21.15.134:0/3475174837 >> 172.21.15.134:0/3475174837 conn(0x7f6aac07b280 msgr2=0x7f6aac07b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:08.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.279+0000 7f6ab3e00640 1 -- 172.21.15.134:0/3475174837 shutdown_connections 2024-07-26T11:17:08.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.279+0000 7f6ab3e00640 1 -- 172.21.15.134:0/3475174837 wait complete. 2024-07-26T11:17:08.282 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:08 smithi134 ceph-mon[31966]: pgmap v515: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-07-26T11:17:08.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.280+0000 7f6ab3e00640 1 Processor -- start 2024-07-26T11:17:08.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.281+0000 7f6ab3e00640 1 -- start start 2024-07-26T11:17:08.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.282+0000 7f6ab3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6aac075180 0x7f6aac19c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.282+0000 7f6ab3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6aac075ad0 0x7f6aac19c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.282+0000 7f6ab3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6aac10fb50 0x7f6aac1a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.282+0000 7f6ab3e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6aac1156b0 con 0x7f6aac10fb50 2024-07-26T11:17:08.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.282+0000 7f6ab3e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6aac115530 con 0x7f6aac075ad0 2024-07-26T11:17:08.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.282+0000 7f6ab3e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6aac115830 con 0x7f6aac075180 2024-07-26T11:17:08.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.282+0000 7f6ab0a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6aac075ad0 0x7f6aac19c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:08.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.282+0000 7f6ab1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6aac075180 0x7f6aac19c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:08.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.282+0000 7f6ab0a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6aac075ad0 0x7f6aac19c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42674/0 (socket says 172.21.15.134:42674) 2024-07-26T11:17:08.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.282+0000 7f6ab0a00640 1 -- 172.21.15.134:0/2948571918 learned_addr learned my addr 172.21.15.134:0/2948571918 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:08.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.283+0000 7f6ab1e00640 1 --2- 172.21.15.134:0/2948571918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6aac10fb50 0x7f6aac1a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:08.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.283+0000 7f6ab1e00640 1 -- 172.21.15.134:0/2948571918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6aac075180 msgr2=0x7f6aac19c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:08.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.283+0000 7f6ab1e00640 1 --2- 172.21.15.134:0/2948571918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6aac075180 0x7f6aac19c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.283+0000 7f6ab1e00640 1 -- 172.21.15.134:0/2948571918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6aac075ad0 msgr2=0x7f6aac19c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:08.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.283+0000 7f6ab1e00640 1 --2- 172.21.15.134:0/2948571918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6aac075ad0 0x7f6aac19c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.283+0000 7f6ab1e00640 1 -- 172.21.15.134:0/2948571918 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aac1a20d0 con 0x7f6aac10fb50 2024-07-26T11:17:08.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.284+0000 7f6ab1e00640 1 --2- 172.21.15.134:0/2948571918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6aac10fb50 0x7f6aac1a19a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4009bf0 tx=0x7f6aa40319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:08.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.284+0000 7f6aaa000640 1 -- 172.21.15.134:0/2948571918 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4003b70 con 0x7f6aac10fb50 2024-07-26T11:17:08.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.284+0000 7f6aaa000640 1 -- 172.21.15.134:0/2948571918 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6aa4003d10 con 0x7f6aac10fb50 2024-07-26T11:17:08.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.284+0000 7f6ab3e00640 1 -- 172.21.15.134:0/2948571918 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aac1a2360 con 0x7f6aac10fb50 2024-07-26T11:17:08.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.284+0000 7f6ab3e00640 1 -- 172.21.15.134:0/2948571918 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aac1a8b20 con 0x7f6aac10fb50 2024-07-26T11:17:08.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.284+0000 7f6aaa000640 1 -- 172.21.15.134:0/2948571918 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa402f990 con 0x7f6aac10fb50 2024-07-26T11:17:08.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.289+0000 7f6ababf4740 1 Processor -- start 2024-07-26T11:17:08.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.289+0000 7f6aaa000640 1 -- 172.21.15.134:0/2948571918 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f6aa402fb30 con 0x7f6aac10fb50 2024-07-26T11:17:08.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.289+0000 7f6ababf4740 1 -- start start 2024-07-26T11:17:08.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.290+0000 7f6ababf4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ef2070 0x5613b1296e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.290+0000 7f6ababf4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ec91b0 0x5613b1297390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.290+0000 7f6ababf4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12978d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.290+0000 7f6ababf4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5613b120dbf0 con 0x5613b0ec91b0 2024-07-26T11:17:08.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.290+0000 7f6ababf4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5613b11fd0f0 con 0x5613b0ed1390 2024-07-26T11:17:08.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.290+0000 7f6ababf4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5613b1201130 con 0x5613b0ef2070 2024-07-26T11:17:08.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.291+0000 7f6ab0a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ef2070 0x5613b1296e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:08.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.291+0000 7f6ab0a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ef2070 0x5613b1296e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58540/0 (socket says 172.21.15.134:58540) 2024-07-26T11:17:08.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.291+0000 7f6ab0a00640 1 -- 172.21.15.134:0/3718189509 learned_addr learned my addr 172.21.15.134:0/3718189509 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:08.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.291+0000 7f6ab0a00640 1 --2- 172.21.15.134:0/3718189509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:08.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.291+0000 7f6ab1400640 1 --2- 172.21.15.134:0/3718189509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ec91b0 0x5613b1297390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:08.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.291+0000 7f6aaa000640 1 --2- 172.21.15.134:0/2948571918 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6a940b3e70 0x7f6a940b6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.291+0000 7f6ab0a00640 1 -- 172.21.15.134:0/3718189509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 msgr2=0x5613b12978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:08.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.291+0000 7f6ab0a00640 1 --2- 172.21.15.134:0/3718189509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.292+0000 7f6ab0a00640 1 -- 172.21.15.134:0/3718189509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ec91b0 msgr2=0x5613b1297390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:08.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.292+0000 7f6ab0a00640 1 --2- 172.21.15.134:0/3718189509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ec91b0 0x5613b1297390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.292+0000 7f6ab0a00640 1 -- 172.21.15.134:0/3718189509 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613b0f5d650 con 0x5613b0ef2070 2024-07-26T11:17:08.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.292+0000 7f6ab1400640 1 --2- 172.21.15.134:0/3718189509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ec91b0 0x5613b1297390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:17:08.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.292+0000 7f6aaa000640 1 -- 172.21.15.134:0/2948571918 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f6aa41056a0 con 0x7f6aac10fb50 2024-07-26T11:17:08.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.292+0000 7f6ab0a00640 1 --2- 172.21.15.134:0/3718189509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ef2070 0x5613b1296e50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6a980095c0 tx=0x7f6a98009530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0fe11ee8ca0ecc5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:08.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.292+0000 7f6ab1e00640 1 --2- 172.21.15.134:0/2948571918 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6a940b3e70 0x7f6a940b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:08.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.293+0000 7f6a9e600640 1 -- 172.21.15.134:0/3718189509 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a98004040 con 0x5613b0ef2070 2024-07-26T11:17:08.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.293+0000 7f6a9e600640 1 -- 172.21.15.134:0/3718189509 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a980054d0 con 0x5613b0ef2070 2024-07-26T11:17:08.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.293+0000 7f6a9e600640 1 -- 172.21.15.134:0/3718189509 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a98004ad0 con 0x5613b0ef2070 2024-07-26T11:17:08.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.294+0000 7f6ababf4740 1 -- 172.21.15.134:0/3718189509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ef2070 msgr2=0x5613b1296e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:08.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.294+0000 7f6ab1e00640 1 --2- 172.21.15.134:0/2948571918 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6a940b3e70 0x7f6a940b6380 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6aa40c9250 tx=0x7f6aa4009460 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:08.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.294+0000 7f6ababf4740 1 --2- 172.21.15.134:0/3718189509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ef2070 0x5613b1296e50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6a980095c0 tx=0x7f6a98009530 comp rx=0 tx=0).stop 2024-07-26T11:17:08.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.294+0000 7f6ababf4740 1 -- 172.21.15.134:0/3718189509 shutdown_connections 2024-07-26T11:17:08.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.294+0000 7f6ababf4740 1 --2- 172.21.15.134:0/3718189509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.295+0000 7f6ababf4740 1 --2- 172.21.15.134:0/3718189509 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ec91b0 0x5613b1297390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.295+0000 7f6ababf4740 1 --2- 172.21.15.134:0/3718189509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ef2070 0x5613b1296e50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.295+0000 7f6ababf4740 1 -- 172.21.15.134:0/3718189509 >> 172.21.15.134:0/3718189509 conn(0x5613b0ee6460 msgr2=0x5613b1294d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:08.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.295+0000 7f6ababf4740 1 -- 172.21.15.134:0/3718189509 shutdown_connections 2024-07-26T11:17:08.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.295+0000 7f6ababf4740 1 -- 172.21.15.134:0/3718189509 wait complete. 2024-07-26T11:17:08.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.296+0000 7f6ababf4740 1 -- start start 2024-07-26T11:17:08.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.296+0000 7f6ababf4740 1 Processor -- start 2024-07-26T11:17:08.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.297+0000 7f6ababf4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 0x5613b12aabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.297+0000 7f6ababf4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12ab130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.297+0000 7f6ababf4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ef2070 0x5613b12ab670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.297+0000 7f6ababf4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5613b1060700 con 0x5613b0ef2070 2024-07-26T11:17:08.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.297+0000 7f6ababf4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5613b11996c0 con 0x5613b0ed1390 2024-07-26T11:17:08.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.297+0000 7f6ababf4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5613b12addb0 con 0x5613b0ec91b0 2024-07-26T11:17:08.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.297+0000 7f6ababf4740 10 client.0 _ll_register_callbacks cb 0x5613b11a9170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:08.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.298+0000 7f6ab1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:08.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.298+0000 7f6ab1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12ab130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42704/0 (socket says 172.21.15.134:42704) 2024-07-26T11:17:08.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.298+0000 7f6ab1400640 1 -- 172.21.15.134:0/333867660 learned_addr learned my addr 172.21.15.134:0/333867660 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:08.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.298+0000 7f6ab1400640 1 -- 172.21.15.134:0/333867660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 msgr2=0x5613b12aabf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:17:08.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.298+0000 7f6ab1400640 1 --2- 172.21.15.134:0/333867660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 0x5613b12aabf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.298+0000 7f6ab1400640 1 -- 172.21.15.134:0/333867660 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ef2070 msgr2=0x5613b12ab670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:08.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.298+0000 7f6ab1400640 1 --2- 172.21.15.134:0/333867660 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ef2070 0x5613b12ab670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.298+0000 7f6ab1400640 1 -- 172.21.15.134:0/333867660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613b11b0860 con 0x5613b0ed1390 2024-07-26T11:17:08.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.299+0000 7f6ab1400640 1 --2- 172.21.15.134:0/333867660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12ab130 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6aa000b910 tx=0x7f6aa000bdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=747e59265426385e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:08.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.299+0000 7f6a92a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:17:08.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.299+0000 7f6a92a00640 1 -- 172.21.15.134:0/333867660 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0005220 con 0x5613b0ed1390 2024-07-26T11:17:08.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.299+0000 7f6a92a00640 1 -- 172.21.15.134:0/333867660 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f6aa00053c0 con 0x5613b0ed1390 2024-07-26T11:17:08.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.300+0000 7f6ababf4740 1 -- 172.21.15.134:0/333867660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a980054d0 con 0x5613b0ed1390 2024-07-26T11:17:08.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.300+0000 7f6a92a00640 1 -- 172.21.15.134:0/333867660 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6aa00055c0 con 0x5613b0ed1390 2024-07-26T11:17:08.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.300+0000 7f6ababf4740 10 client.25694 fetch_fsmap learned FSMap version 30 2024-07-26T11:17:08.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.300+0000 7f6ababf4740 1 -- 172.21.15.134:0/333867660 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5613b12b99e0 con 0x5613b0ed1390 2024-07-26T11:17:08.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.300+0000 7f6a92a00640 1 -- 172.21.15.134:0/333867660 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a980054d0 con 0x5613b0ed1390 2024-07-26T11:17:08.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.301+0000 7f6a92a00640 1 -- 172.21.15.134:0/333867660 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f6aa0012860 con 0x5613b0ed1390 2024-07-26T11:17:08.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.301+0000 7f6ababf4740 10 client.25694 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:17:08.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.301+0000 7f6ababf4740 10 client.25694 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-07-26T11:17:08.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.301+0000 7f6ababf4740 20 client.25694 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:08.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.301+0000 7f6ababf4740 1 --2- 172.21.15.134:0/333867660 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5613b0eb09f0 0x5613b12bcfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.301+0000 7f6ababf4740 4 client.25694 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:08.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.301+0000 7f6ababf4740 1 -- 172.21.15.134:0/333867660 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5613b120fb40 con 0x5613b0eb09f0 2024-07-26T11:17:08.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.302+0000 7f6ab0a00640 1 --2- 172.21.15.134:0/333867660 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5613b0eb09f0 0x5613b12bcfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:08.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.302+0000 7f6ab0a00640 1 --2- 172.21.15.134:0/333867660 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5613b0eb09f0 0x5613b12bcfc0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a96b36a455f67260 server_cookie=ca32e46a7d9c27e in_seq=0 out_seq=0 2024-07-26T11:17:08.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.302+0000 7f6a92a00640 10 client.25694 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:17:08.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.303+0000 7f6a92a00640 1 -- 172.21.15.134:0/333867660 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5613b120fb40 con 0x5613b0eb09f0 2024-07-26T11:17:08.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.303+0000 7f6a92a00640 10 client.25694 handle_command_reply: tid=0 2024-07-26T11:17:08.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.303+0000 7f6ababf4740 1 client.25694 shutdown 2024-07-26T11:17:08.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.304+0000 7f6ababf4740 1 -- 172.21.15.134:0/333867660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 msgr2=0x5613b12ab130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:08.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.304+0000 7f6ababf4740 1 --2- 172.21.15.134:0/333867660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12ab130 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6aa000b910 tx=0x7f6aa000bdd0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.304+0000 7f6ababf4740 1 -- 172.21.15.134:0/333867660 shutdown_connections 2024-07-26T11:17:08.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.304+0000 7f6ababf4740 1 --2- 172.21.15.134:0/333867660 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5613b0eb09f0 0x5613b12bcfc0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.304+0000 7f6a92a00640 0 client.25694 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:17:08.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.304+0000 7f6ababf4740 1 --2- 172.21.15.134:0/333867660 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ef2070 0x5613b12ab670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.304+0000 7f6ababf4740 1 --2- 172.21.15.134:0/333867660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12ab130 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.304+0000 7f6ababf4740 1 --2- 172.21.15.134:0/333867660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 0x5613b12aabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.304+0000 7f6ababf4740 1 -- 172.21.15.134:0/333867660 >> 172.21.15.134:0/333867660 conn(0x5613b0ee6460 msgr2=0x5613b1293400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:08.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.304+0000 7f6ababf4740 1 -- 172.21.15.134:0/333867660 shutdown_connections 2024-07-26T11:17:08.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.304+0000 7f6ababf4740 1 -- 172.21.15.134:0/333867660 wait complete. 2024-07-26T11:17:08.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.304+0000 7f6ababf4740 20 client.25694 trim_cache size 0 max 16384 2024-07-26T11:17:08.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.330+0000 7f6ababf4740 1 Processor -- start 2024-07-26T11:17:08.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.330+0000 7f6ababf4740 1 -- start start 2024-07-26T11:17:08.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.331+0000 7f6ababf4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 0x5613b12949f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.331+0000 7f6ababf4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12a60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.331+0000 7f6ababf4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ef2070 0x5613b12c2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.331+0000 7f6ababf4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c00fed0 con 0x5613b0ef2070 2024-07-26T11:17:08.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.331+0000 7f6ababf4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c011a10 con 0x5613b0ed1390 2024-07-26T11:17:08.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.331+0000 7f6ababf4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c0105a0 con 0x5613b0ec91b0 2024-07-26T11:17:08.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.331+0000 7f6ab1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12a60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:08.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.331+0000 7f6ab0a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 0x5613b12949f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:08.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.331+0000 7f6ab0a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 0x5613b12949f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58570/0 (socket says 172.21.15.134:58570) 2024-07-26T11:17:08.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.331+0000 7f6ab1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12a60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42718/0 (socket says 172.21.15.134:42718) 2024-07-26T11:17:08.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.331+0000 7f6ab1400640 1 -- 172.21.15.134:0/1155315797 learned_addr learned my addr 172.21.15.134:0/1155315797 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:08.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.332+0000 7f6ab0a00640 1 --2- 172.21.15.134:0/1155315797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ef2070 0x5613b12c2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:08.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.332+0000 7f6ab0a00640 1 -- 172.21.15.134:0/1155315797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 msgr2=0x5613b12a60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:08.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.332+0000 7f6ab0a00640 1 --2- 172.21.15.134:0/1155315797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12a60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.332+0000 7f6ab0a00640 1 -- 172.21.15.134:0/1155315797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ef2070 msgr2=0x5613b12c2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:08.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.332+0000 7f6ab0a00640 1 --2- 172.21.15.134:0/1155315797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ef2070 0x5613b12c2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.332+0000 7f6ab0a00640 1 -- 172.21.15.134:0/1155315797 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613b1299c60 con 0x5613b0ec91b0 2024-07-26T11:17:08.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.332+0000 7f6ab1400640 1 --2- 172.21.15.134:0/1155315797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12a60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:08.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.333+0000 7f6ab0a00640 1 --2- 172.21.15.134:0/1155315797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 0x5613b12949f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6a98009250 tx=0x7f6a980045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8aa60ec50d916db5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:08.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.333+0000 7f6a9dc00640 1 -- 172.21.15.134:0/1155315797 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a98004240 con 0x5613b0ec91b0 2024-07-26T11:17:08.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.333+0000 7f6a9dc00640 1 -- 172.21.15.134:0/1155315797 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a9800fc40 con 0x5613b0ec91b0 2024-07-26T11:17:08.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.333+0000 7f6a9dc00640 1 -- 172.21.15.134:0/1155315797 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a980039c0 con 0x5613b0ec91b0 2024-07-26T11:17:08.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.334+0000 7f6ababf4740 1 -- 172.21.15.134:0/1155315797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 msgr2=0x5613b12949f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:08.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.334+0000 7f6ababf4740 1 --2- 172.21.15.134:0/1155315797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 0x5613b12949f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6a98009250 tx=0x7f6a980045e0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.334+0000 7f6ababf4740 1 -- 172.21.15.134:0/1155315797 shutdown_connections 2024-07-26T11:17:08.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.334+0000 7f6ababf4740 1 --2- 172.21.15.134:0/1155315797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ef2070 0x5613b12c2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.334+0000 7f6ababf4740 1 --2- 172.21.15.134:0/1155315797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12a60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.334+0000 7f6ababf4740 1 --2- 172.21.15.134:0/1155315797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 0x5613b12949f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.335+0000 7f6ababf4740 1 -- 172.21.15.134:0/1155315797 >> 172.21.15.134:0/1155315797 conn(0x5613b0ee6460 msgr2=0x5613b12a5990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:08.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.335+0000 7f6ababf4740 1 -- 172.21.15.134:0/1155315797 shutdown_connections 2024-07-26T11:17:08.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.335+0000 7f6ababf4740 1 -- 172.21.15.134:0/1155315797 wait complete. 2024-07-26T11:17:08.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.335+0000 7f6ababf4740 1 -- start start 2024-07-26T11:17:08.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.335+0000 7f6ababf4740 1 Processor -- start 2024-07-26T11:17:08.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.336+0000 7f6ababf4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 0x5613b12ae870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.336+0000 7f6ababf4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.336+0000 7f6ababf4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ef2070 0x5613b1323740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.336+0000 7f6ababf4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c011340 con 0x5613b0ef2070 2024-07-26T11:17:08.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.336+0000 7f6ababf4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c010c70 con 0x5613b0ed1390 2024-07-26T11:17:08.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.336+0000 7f6ababf4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c00deb0 con 0x5613b0ec91b0 2024-07-26T11:17:08.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.336+0000 7f6ababf4740 10 client.0 _ll_register_callbacks cb 0x7f6aa0004530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:08.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.336+0000 7f6ab0a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 0x5613b12ae870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:08.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.336+0000 7f6ab0a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 0x5613b12ae870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58586/0 (socket says 172.21.15.134:58586) 2024-07-26T11:17:08.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.336+0000 7f6ab0a00640 1 -- 172.21.15.134:0/3259850024 learned_addr learned my addr 172.21.15.134:0/3259850024 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:08.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.336+0000 7f6ab0a00640 1 --2- 172.21.15.134:0/3259850024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ef2070 0x5613b1323740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:08.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.336+0000 7f6ab1400640 1 --2- 172.21.15.134:0/3259850024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:08.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.337+0000 7f6ab0a00640 1 -- 172.21.15.134:0/3259850024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 msgr2=0x5613b12aedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:08.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.337+0000 7f6ab0a00640 1 --2- 172.21.15.134:0/3259850024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12aedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.337+0000 7f6ab0a00640 1 -- 172.21.15.134:0/3259850024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ef2070 msgr2=0x5613b1323740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:08.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.337+0000 7f6ab0a00640 1 --2- 172.21.15.134:0/3259850024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ef2070 0x5613b1323740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.337+0000 7f6ab0a00640 1 -- 172.21.15.134:0/3259850024 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5613b1323cd0 con 0x5613b0ec91b0 2024-07-26T11:17:08.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.337+0000 7f6ab0a00640 1 --2- 172.21.15.134:0/3259850024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 0x5613b12ae870 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6a98006010 tx=0x7f6a98009ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56e7039fd5a34bfd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:08.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.337+0000 7f6a92a00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:17:08.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.337+0000 7f6a92a00640 1 -- 172.21.15.134:0/3259850024 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a98003c80 con 0x5613b0ec91b0 2024-07-26T11:17:08.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.337+0000 7f6a92a00640 1 -- 172.21.15.134:0/3259850024 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f6a98004040 con 0x5613b0ec91b0 2024-07-26T11:17:08.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.338+0000 7f6ababf4740 1 -- 172.21.15.134:0/3259850024 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a9800fc40 con 0x5613b0ec91b0 2024-07-26T11:17:08.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.339+0000 7f6a92a00640 1 -- 172.21.15.134:0/3259850024 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6a98003e20 con 0x5613b0ec91b0 2024-07-26T11:17:08.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.339+0000 7f6ababf4740 10 client.25711 fetch_fsmap learned FSMap version 30 2024-07-26T11:17:08.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.339+0000 7f6ababf4740 1 -- 172.21.15.134:0/3259850024 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5613b1324f50 con 0x5613b0ec91b0 2024-07-26T11:17:08.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.339+0000 7f6a92a00640 1 -- 172.21.15.134:0/3259850024 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a9800fc40 con 0x5613b0ec91b0 2024-07-26T11:17:08.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.339+0000 7f6a92a00640 1 -- 172.21.15.134:0/3259850024 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f6a9801f440 con 0x5613b0ec91b0 2024-07-26T11:17:08.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.340+0000 7f6ababf4740 10 client.25711 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:17:08.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.340+0000 7f6ababf4740 10 client.25711 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-07-26T11:17:08.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.340+0000 7f6ababf4740 20 client.25711 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:08.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.340+0000 7f6ababf4740 1 --2- 172.21.15.134:0/3259850024 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5613b0e85da0 0x5613b13273e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:08.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.340+0000 7f6ababf4740 4 client.25711 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:17:08.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.340+0000 7f6ababf4740 1 -- 172.21.15.134:0/3259850024 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5613b120b5e0 con 0x5613b0e85da0 2024-07-26T11:17:08.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.340+0000 7f6ab1400640 1 --2- 172.21.15.134:0/3259850024 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5613b0e85da0 0x5613b13273e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:08.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.341+0000 7f6ab1400640 1 --2- 172.21.15.134:0/3259850024 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5613b0e85da0 0x5613b13273e0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a67d137ea1241608 server_cookie=4f44da37259290ce in_seq=0 out_seq=0 2024-07-26T11:17:08.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.341+0000 7f6a92a00640 10 client.25711 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:17:08.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.341+0000 7f6a92a00640 1 -- 172.21.15.134:0/3259850024 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5613b120b5e0 con 0x5613b0e85da0 2024-07-26T11:17:08.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.341+0000 7f6a92a00640 10 client.25711 handle_command_reply: tid=0 2024-07-26T11:17:08.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.341+0000 7f6ababf4740 1 client.25711 shutdown 2024-07-26T11:17:08.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.341+0000 7f6ababf4740 1 -- 172.21.15.134:0/3259850024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 msgr2=0x5613b12ae870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:08.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.341+0000 7f6ababf4740 1 --2- 172.21.15.134:0/3259850024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 0x5613b12ae870 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6a98006010 tx=0x7f6a98009ab0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.341+0000 7f6ababf4740 1 -- 172.21.15.134:0/3259850024 shutdown_connections 2024-07-26T11:17:08.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.341+0000 7f6ababf4740 1 --2- 172.21.15.134:0/3259850024 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5613b0e85da0 0x5613b13273e0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.341+0000 7f6ababf4740 1 --2- 172.21.15.134:0/3259850024 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5613b0ef2070 0x5613b1323740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.341+0000 7f6ababf4740 1 --2- 172.21.15.134:0/3259850024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5613b0ed1390 0x5613b12aedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.341+0000 7f6ababf4740 1 --2- 172.21.15.134:0/3259850024 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5613b0ec91b0 0x5613b12ae870 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.341+0000 7f6ababf4740 1 -- 172.21.15.134:0/3259850024 >> 172.21.15.134:0/3259850024 conn(0x5613b0ee6460 msgr2=0x5613b1294760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:08.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.341+0000 7f6a92a00640 0 client.25711 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:17:08.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.342+0000 7f6ababf4740 1 -- 172.21.15.134:0/3259850024 shutdown_connections 2024-07-26T11:17:08.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.342+0000 7f6ababf4740 1 -- 172.21.15.134:0/3259850024 wait complete. 2024-07-26T11:17:08.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.342+0000 7f6ababf4740 20 client.25711 trim_cache size 0 max 16384 2024-07-26T11:17:08.346 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:17:08.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.343+0000 7f6ab3e00640 1 -- 172.21.15.134:0/2948571918 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6a940b3e70 msgr2=0x7f6a940b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:08.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.343+0000 7f6ab3e00640 1 --2- 172.21.15.134:0/2948571918 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6a940b3e70 0x7f6a940b6380 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6aa40c9250 tx=0x7f6aa4009460 comp rx=0 tx=0).stop 2024-07-26T11:17:08.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.343+0000 7f6ab3e00640 1 -- 172.21.15.134:0/2948571918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6aac10fb50 msgr2=0x7f6aac1a19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:08.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.343+0000 7f6ab3e00640 1 --2- 172.21.15.134:0/2948571918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6aac10fb50 0x7f6aac1a19a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6aa4009bf0 tx=0x7f6aa40319f0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.343+0000 7f6ab3e00640 1 -- 172.21.15.134:0/2948571918 shutdown_connections 2024-07-26T11:17:08.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.343+0000 7f6ab3e00640 1 --2- 172.21.15.134:0/2948571918 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f6a940b3e70 0x7f6a940b6380 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.343+0000 7f6ab3e00640 1 --2- 172.21.15.134:0/2948571918 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6aac10fb50 0x7f6aac1a19a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.343+0000 7f6ab3e00640 1 --2- 172.21.15.134:0/2948571918 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6aac075ad0 0x7f6aac19c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.343+0000 7f6ab3e00640 1 --2- 172.21.15.134:0/2948571918 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6aac075180 0x7f6aac19c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:08.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.343+0000 7f6ab3e00640 1 -- 172.21.15.134:0/2948571918 >> 172.21.15.134:0/2948571918 conn(0x7f6aac07b280 msgr2=0x7f6aac1011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:08.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.343+0000 7f6ab3e00640 1 -- 172.21.15.134:0/2948571918 shutdown_connections 2024-07-26T11:17:08.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:08.343+0000 7f6ab3e00640 1 -- 172.21.15.134:0/2948571918 wait complete. 2024-07-26T11:17:08.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:08 smithi189 ceph-mon[36790]: pgmap v515: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-07-26T11:17:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:08 smithi193 ceph-mon[37473]: pgmap v515: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 27 op/s 2024-07-26T11:17:09.184 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-26T11:17:09.704 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:17:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:10 smithi134 ceph-mon[31966]: pgmap v516: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-07-26T11:17:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:10 smithi189 ceph-mon[36790]: pgmap v516: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-07-26T11:17:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:10 smithi193 ceph-mon[37473]: pgmap v516: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-07-26T11:17:12.346 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:12 smithi134 ceph-mon[31966]: pgmap v517: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-07-26T11:17:12.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.414+0000 7fa850e00640 1 -- 172.21.15.134:0/2916494575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa84c107de0 msgr2=0x7fa84c1081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:12.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.414+0000 7fa850e00640 1 --2- 172.21.15.134:0/2916494575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa84c107de0 0x7fa84c1081f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa838009900 tx=0x7fa83802a410 comp rx=0 tx=0).stop 2024-07-26T11:17:12.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.414+0000 7fa850e00640 1 -- 172.21.15.134:0/2916494575 shutdown_connections 2024-07-26T11:17:12.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.414+0000 7fa850e00640 1 --2- 172.21.15.134:0/2916494575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c108730 0x7fa84c10ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.414+0000 7fa850e00640 1 --2- 172.21.15.134:0/2916494575 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa84c107de0 0x7fa84c1081f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.414+0000 7fa850e00640 1 --2- 172.21.15.134:0/2916494575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa84c1024f0 0x7fa84c102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.414+0000 7fa850e00640 1 -- 172.21.15.134:0/2916494575 >> 172.21.15.134:0/2916494575 conn(0x7fa84c079150 msgr2=0x7fa84c079560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:12.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.414+0000 7fa850e00640 1 -- 172.21.15.134:0/2916494575 shutdown_connections 2024-07-26T11:17:12.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.414+0000 7fa850e00640 1 -- 172.21.15.134:0/2916494575 wait complete. 2024-07-26T11:17:12.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.415+0000 7fa850e00640 1 Processor -- start 2024-07-26T11:17:12.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.416+0000 7fa850e00640 1 -- start start 2024-07-26T11:17:12.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.416+0000 7fa850e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c1024f0 0x7fa84c1a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.416+0000 7fa850e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa84c107de0 0x7fa84c1a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.416+0000 7fa850e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa84c108730 0x7fa84c1a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.416+0000 7fa850e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa84c1123c0 con 0x7fa84c107de0 2024-07-26T11:17:12.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.416+0000 7fa850e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa84c112240 con 0x7fa84c1024f0 2024-07-26T11:17:12.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.416+0000 7fa850e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa84c112540 con 0x7fa84c108730 2024-07-26T11:17:12.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.416+0000 7fa849400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa84c107de0 0x7fa84c1a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.417+0000 7fa849e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c1024f0 0x7fa84c1a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.417+0000 7fa849e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c1024f0 0x7fa84c1a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46446/0 (socket says 172.21.15.134:46446) 2024-07-26T11:17:12.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.417+0000 7fa849e00640 1 -- 172.21.15.134:0/1575617879 learned_addr learned my addr 172.21.15.134:0/1575617879 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:12.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.417+0000 7fa849400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa84c107de0 0x7fa84c1a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38426/0 (socket says 172.21.15.134:38426) 2024-07-26T11:17:12.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.417+0000 7fa84a800640 1 --2- 172.21.15.134:0/1575617879 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa84c108730 0x7fa84c1a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.417+0000 7fa849400640 1 -- 172.21.15.134:0/1575617879 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa84c108730 msgr2=0x7fa84c1a5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:12.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.417+0000 7fa849400640 1 --2- 172.21.15.134:0/1575617879 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa84c108730 0x7fa84c1a5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.417+0000 7fa849400640 1 -- 172.21.15.134:0/1575617879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c1024f0 msgr2=0x7fa84c1a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:12.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.417+0000 7fa849400640 1 --2- 172.21.15.134:0/1575617879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c1024f0 0x7fa84c1a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.417+0000 7fa849400640 1 -- 172.21.15.134:0/1575617879 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa84c1a6490 con 0x7fa84c107de0 2024-07-26T11:17:12.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.418+0000 7fa849400640 1 --2- 172.21.15.134:0/1575617879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa84c107de0 0x7fa84c1a0b60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa83802a930 tx=0x7fa838031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:12.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.418+0000 7fa842a00640 1 -- 172.21.15.134:0/1575617879 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83803f070 con 0x7fa84c107de0 2024-07-26T11:17:12.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.418+0000 7fa842a00640 1 -- 172.21.15.134:0/1575617879 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa838003aa0 con 0x7fa84c107de0 2024-07-26T11:17:12.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.418+0000 7fa842a00640 1 -- 172.21.15.134:0/1575617879 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa838035070 con 0x7fa84c107de0 2024-07-26T11:17:12.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.418+0000 7fa850e00640 1 -- 172.21.15.134:0/1575617879 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa84c1a6720 con 0x7fa84c107de0 2024-07-26T11:17:12.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.418+0000 7fa850e00640 1 -- 172.21.15.134:0/1575617879 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa84c1acee0 con 0x7fa84c107de0 2024-07-26T11:17:12.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.422+0000 7fa842a00640 1 -- 172.21.15.134:0/1575617879 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fa838003c50 con 0x7fa84c107de0 2024-07-26T11:17:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.423+0000 7fa857bd1740 1 Processor -- start 2024-07-26T11:17:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.423+0000 7fa842a00640 1 --2- 172.21.15.134:0/1575617879 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa8300b3d40 0x7fa8300b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.423+0000 7fa857bd1740 1 -- start start 2024-07-26T11:17:12.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.424+0000 7fa842a00640 1 -- 172.21.15.134:0/1575617879 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa8381041d0 con 0x7fa84c107de0 2024-07-26T11:17:12.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.424+0000 7fa84a800640 1 --2- 172.21.15.134:0/1575617879 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa8300b3d40 0x7fa8300b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.424+0000 7fa857bd1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eec5070 0x55819f269e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.424+0000 7fa857bd1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819ee9c1b0 0x55819f26a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.424+0000 7fa857bd1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819eea4390 0x55819f26a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.424+0000 7fa857bd1740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55819f1e0bd0 con 0x55819ee9c1b0 2024-07-26T11:17:12.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.424+0000 7fa857bd1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55819f1d00d0 con 0x55819eea4390 2024-07-26T11:17:12.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.424+0000 7fa857bd1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55819f1d4110 con 0x55819eec5070 2024-07-26T11:17:12.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.424+0000 7fa849400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819ee9c1b0 0x55819f26a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.424+0000 7fa849400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819ee9c1b0 0x55819f26a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38432/0 (socket says 172.21.15.134:38432) 2024-07-26T11:17:12.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.424+0000 7fa849400640 1 -- 172.21.15.134:0/3893612247 learned_addr learned my addr 172.21.15.134:0/3893612247 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:12.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.424+0000 7fa849e00640 1 --2- 172.21.15.134:0/3893612247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eec5070 0x55819f269e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.424+0000 7fa84a800640 1 --2- 172.21.15.134:0/3893612247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819eea4390 0x55819f26a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.425+0000 7fa849e00640 1 -- 172.21.15.134:0/3893612247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819eea4390 msgr2=0x55819f26a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:12.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.425+0000 7fa849e00640 1 --2- 172.21.15.134:0/3893612247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819eea4390 0x55819f26a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.425+0000 7fa849e00640 1 -- 172.21.15.134:0/3893612247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819ee9c1b0 msgr2=0x55819f26a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:12.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.425+0000 7fa849e00640 1 --2- 172.21.15.134:0/3893612247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819ee9c1b0 0x55819f26a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.425+0000 7fa849e00640 1 -- 172.21.15.134:0/3893612247 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55819ef30650 con 0x55819eec5070 2024-07-26T11:17:12.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.425+0000 7fa849400640 1 --2- 172.21.15.134:0/3893612247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819ee9c1b0 0x55819f26a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:12.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.425+0000 7fa84a800640 1 --2- 172.21.15.134:0/3893612247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819eea4390 0x55819f26a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:17:12.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.425+0000 7fa849e00640 1 --2- 172.21.15.134:0/3893612247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eec5070 0x55819f269e30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009cf0 tx=0x7fa83c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b08fd1e221fc13a0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:12.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.425+0000 7fa84a800640 1 --2- 172.21.15.134:0/1575617879 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa8300b3d40 0x7fa8300b6250 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa844005fd0 tx=0x7fa844011be0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:12.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.426+0000 7fa837400640 1 -- 172.21.15.134:0/3893612247 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c005220 con 0x55819eec5070 2024-07-26T11:17:12.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.426+0000 7fa837400640 1 -- 172.21.15.134:0/3893612247 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa83c004c50 con 0x55819eec5070 2024-07-26T11:17:12.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.426+0000 7fa837400640 1 -- 172.21.15.134:0/3893612247 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c018790 con 0x55819eec5070 2024-07-26T11:17:12.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.427+0000 7fa857bd1740 1 -- 172.21.15.134:0/3893612247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eec5070 msgr2=0x55819f269e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:12.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.427+0000 7fa857bd1740 1 --2- 172.21.15.134:0/3893612247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eec5070 0x55819f269e30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009cf0 tx=0x7fa83c011810 comp rx=0 tx=0).stop 2024-07-26T11:17:12.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.427+0000 7fa857bd1740 1 -- 172.21.15.134:0/3893612247 shutdown_connections 2024-07-26T11:17:12.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.427+0000 7fa857bd1740 1 --2- 172.21.15.134:0/3893612247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819eea4390 0x55819f26a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.427+0000 7fa857bd1740 1 --2- 172.21.15.134:0/3893612247 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819ee9c1b0 0x55819f26a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.427+0000 7fa857bd1740 1 --2- 172.21.15.134:0/3893612247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eec5070 0x55819f269e30 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.427+0000 7fa857bd1740 1 -- 172.21.15.134:0/3893612247 >> 172.21.15.134:0/3893612247 conn(0x55819eeb9460 msgr2=0x55819f267d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:12.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.427+0000 7fa857bd1740 1 -- 172.21.15.134:0/3893612247 shutdown_connections 2024-07-26T11:17:12.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.427+0000 7fa857bd1740 1 -- 172.21.15.134:0/3893612247 wait complete. 2024-07-26T11:17:12.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.428+0000 7fa857bd1740 1 -- start start 2024-07-26T11:17:12.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.428+0000 7fa857bd1740 1 Processor -- start 2024-07-26T11:17:12.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.429+0000 7fa857bd1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819ee9c1b0 0x55819f275430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.429+0000 7fa857bd1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eea4390 0x55819f275970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.429+0000 7fa857bd1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eec5070 0x55819f275eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.429+0000 7fa857bd1740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55819f033700 con 0x55819eec5070 2024-07-26T11:17:12.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.429+0000 7fa857bd1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55819f16c6a0 con 0x55819ee9c1b0 2024-07-26T11:17:12.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.429+0000 7fa857bd1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55819f278610 con 0x55819eea4390 2024-07-26T11:17:12.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.429+0000 7fa857bd1740 10 client.0 _ll_register_callbacks cb 0x55819f17c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:12.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.429+0000 7fa849400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eea4390 0x55819f275970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.429+0000 7fa849e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819ee9c1b0 0x55819f275430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.429+0000 7fa84a800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eec5070 0x55819f275eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.429+0000 7fa849e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819ee9c1b0 0x55819f275430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46472/0 (socket says 172.21.15.134:46472) 2024-07-26T11:17:12.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.429+0000 7fa849e00640 1 -- 172.21.15.134:0/2282522715 learned_addr learned my addr 172.21.15.134:0/2282522715 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:12.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.429+0000 7fa849400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eea4390 0x55819f275970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51358/0 (socket says 172.21.15.134:51358) 2024-07-26T11:17:12.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.430+0000 7fa849e00640 1 -- 172.21.15.134:0/2282522715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eea4390 msgr2=0x55819f275970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:12.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.430+0000 7fa849e00640 1 --2- 172.21.15.134:0/2282522715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eea4390 0x55819f275970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.430+0000 7fa849e00640 1 -- 172.21.15.134:0/2282522715 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eec5070 msgr2=0x55819f275eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:12.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.430+0000 7fa849e00640 1 --2- 172.21.15.134:0/2282522715 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eec5070 0x55819f275eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.430+0000 7fa849e00640 1 -- 172.21.15.134:0/2282522715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55819f183840 con 0x55819ee9c1b0 2024-07-26T11:17:12.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.430+0000 7fa849e00640 1 --2- 172.21.15.134:0/2282522715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819ee9c1b0 0x55819f275430 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009fa0 tx=0x7fa83c005710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3bade84fbe26d7a4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:12.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.430+0000 7fa82be00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:17:12.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.430+0000 7fa82be00640 1 -- 172.21.15.134:0/2282522715 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c027020 con 0x55819ee9c1b0 2024-07-26T11:17:12.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.431+0000 7fa857bd1740 1 -- 172.21.15.134:0/2282522715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa83c004c50 con 0x55819ee9c1b0 2024-07-26T11:17:12.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.431+0000 7fa82be00640 1 -- 172.21.15.134:0/2282522715 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa83c005000 con 0x55819ee9c1b0 2024-07-26T11:17:12.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.431+0000 7fa82be00640 1 -- 172.21.15.134:0/2282522715 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa83c032050 con 0x55819ee9c1b0 2024-07-26T11:17:12.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.431+0000 7fa857bd1740 10 client.25712 fetch_fsmap learned FSMap version 30 2024-07-26T11:17:12.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.431+0000 7fa857bd1740 1 -- 172.21.15.134:0/2282522715 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55819f284240 con 0x55819ee9c1b0 2024-07-26T11:17:12.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.431+0000 7fa82be00640 1 -- 172.21.15.134:0/2282522715 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa83c004c50 con 0x55819ee9c1b0 2024-07-26T11:17:12.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.432+0000 7fa82be00640 1 -- 172.21.15.134:0/2282522715 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fa83c019430 con 0x55819ee9c1b0 2024-07-26T11:17:12.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.432+0000 7fa857bd1740 10 client.25712 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:17:12.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.432+0000 7fa857bd1740 10 client.25712 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:17:12.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.432+0000 7fa857bd1740 20 client.25712 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:12.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.432+0000 7fa857bd1740 1 --2- 172.21.15.134:0/2282522715 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55819ee839f0 0x55819f287600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.432+0000 7fa857bd1740 4 client.25712 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:12.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.432+0000 7fa857bd1740 1 -- 172.21.15.134:0/2282522715 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55819f1e2b20 con 0x55819ee839f0 2024-07-26T11:17:12.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.432+0000 7fa849400640 1 --2- 172.21.15.134:0/2282522715 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55819ee839f0 0x55819f287600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.433+0000 7fa849400640 1 --2- 172.21.15.134:0/2282522715 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55819ee839f0 0x55819f287600 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d0062c5b4a955a4 server_cookie=54989f6a0db2b003 in_seq=0 out_seq=0 2024-07-26T11:17:12.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.433+0000 7fa82be00640 10 client.25712 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:12.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.433+0000 7fa82be00640 1 -- 172.21.15.134:0/2282522715 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55819f1e2b20 con 0x55819ee839f0 2024-07-26T11:17:12.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.434+0000 7fa82be00640 10 client.25712 handle_command_reply: tid=0 2024-07-26T11:17:12.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.434+0000 7fa857bd1740 1 client.25712 shutdown 2024-07-26T11:17:12.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.434+0000 7fa857bd1740 1 -- 172.21.15.134:0/2282522715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819ee9c1b0 msgr2=0x55819f275430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.434+0000 7fa857bd1740 1 --2- 172.21.15.134:0/2282522715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819ee9c1b0 0x55819f275430 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa83c009fa0 tx=0x7fa83c005710 comp rx=0 tx=0).stop 2024-07-26T11:17:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.434+0000 7fa857bd1740 1 -- 172.21.15.134:0/2282522715 shutdown_connections 2024-07-26T11:17:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.434+0000 7fa857bd1740 1 --2- 172.21.15.134:0/2282522715 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55819ee839f0 0x55819f287600 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.434+0000 7fa857bd1740 1 --2- 172.21.15.134:0/2282522715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eea4390 0x55819f275970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.434+0000 7fa857bd1740 1 --2- 172.21.15.134:0/2282522715 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eec5070 0x55819f275eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.434+0000 7fa857bd1740 1 --2- 172.21.15.134:0/2282522715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819ee9c1b0 0x55819f275430 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.434+0000 7fa82be00640 0 client.25712 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.434+0000 7fa857bd1740 1 -- 172.21.15.134:0/2282522715 >> 172.21.15.134:0/2282522715 conn(0x55819eeb9460 msgr2=0x55819f26d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.434+0000 7fa857bd1740 1 -- 172.21.15.134:0/2282522715 shutdown_connections 2024-07-26T11:17:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.434+0000 7fa857bd1740 1 -- 172.21.15.134:0/2282522715 wait complete. 2024-07-26T11:17:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.435+0000 7fa857bd1740 20 client.25712 trim_cache size 0 max 16384 2024-07-26T11:17:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:12 smithi189 ceph-mon[36790]: pgmap v517: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-07-26T11:17:12.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.448+0000 7fa857bd1740 1 Processor -- start 2024-07-26T11:17:12.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.448+0000 7fa857bd1740 1 -- start start 2024-07-26T11:17:12.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.449+0000 7fa857bd1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819ee9c1b0 0x55819f26ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.449+0000 7fa857bd1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eea4390 0x55819f28ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.449+0000 7fa857bd1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eec5070 0x55819f28d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.449+0000 7fa857bd1740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa84c1b6e70 con 0x55819eea4390 2024-07-26T11:17:12.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.449+0000 7fa857bd1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa84c1b89b0 con 0x55819ee9c1b0 2024-07-26T11:17:12.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.449+0000 7fa857bd1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa84c1b7540 con 0x55819eec5070 2024-07-26T11:17:12.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.449+0000 7fa849400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eea4390 0x55819f28ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.449+0000 7fa849400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eea4390 0x55819f28ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38446/0 (socket says 172.21.15.134:38446) 2024-07-26T11:17:12.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.449+0000 7fa849400640 1 -- 172.21.15.134:0/2568003364 learned_addr learned my addr 172.21.15.134:0/2568003364 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:12.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.449+0000 7fa849e00640 1 --2- 172.21.15.134:0/2568003364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819ee9c1b0 0x55819f26ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.449+0000 7fa84a800640 1 --2- 172.21.15.134:0/2568003364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eec5070 0x55819f28d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.450+0000 7fa849e00640 1 -- 172.21.15.134:0/2568003364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eec5070 msgr2=0x55819f28d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:12.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.450+0000 7fa849e00640 1 --2- 172.21.15.134:0/2568003364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eec5070 0x55819f28d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.450+0000 7fa849e00640 1 -- 172.21.15.134:0/2568003364 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eea4390 msgr2=0x55819f28ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:12.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.450+0000 7fa849e00640 1 --2- 172.21.15.134:0/2568003364 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eea4390 0x55819f28ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.450+0000 7fa849e00640 1 -- 172.21.15.134:0/2568003364 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55819f2766f0 con 0x55819ee9c1b0 2024-07-26T11:17:12.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.450+0000 7fa849400640 1 --2- 172.21.15.134:0/2568003364 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eea4390 0x55819f28ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:12.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.450+0000 7fa849e00640 1 --2- 172.21.15.134:0/2568003364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819ee9c1b0 0x55819f26ae60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa83c019800 tx=0x7fa83c009530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2affb3ed46fdff9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:12.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.450+0000 7fa836a00640 1 -- 172.21.15.134:0/2568003364 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c027040 con 0x55819ee9c1b0 2024-07-26T11:17:12.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.450+0000 7fa836a00640 1 -- 172.21.15.134:0/2568003364 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa83c019430 con 0x55819ee9c1b0 2024-07-26T11:17:12.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.450+0000 7fa836a00640 1 -- 172.21.15.134:0/2568003364 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83c01e040 con 0x55819ee9c1b0 2024-07-26T11:17:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.451+0000 7fa857bd1740 1 -- 172.21.15.134:0/2568003364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819ee9c1b0 msgr2=0x55819f26ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.451+0000 7fa857bd1740 1 --2- 172.21.15.134:0/2568003364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819ee9c1b0 0x55819f26ae60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa83c019800 tx=0x7fa83c009530 comp rx=0 tx=0).stop 2024-07-26T11:17:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.451+0000 7fa857bd1740 1 -- 172.21.15.134:0/2568003364 shutdown_connections 2024-07-26T11:17:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.451+0000 7fa857bd1740 1 --2- 172.21.15.134:0/2568003364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819eec5070 0x55819f28d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.451+0000 7fa857bd1740 1 --2- 172.21.15.134:0/2568003364 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eea4390 0x55819f28ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.451+0000 7fa857bd1740 1 --2- 172.21.15.134:0/2568003364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819ee9c1b0 0x55819f26ae60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.451+0000 7fa857bd1740 1 -- 172.21.15.134:0/2568003364 >> 172.21.15.134:0/2568003364 conn(0x55819eeb9460 msgr2=0x55819f270a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.451+0000 7fa857bd1740 1 -- 172.21.15.134:0/2568003364 shutdown_connections 2024-07-26T11:17:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.452+0000 7fa857bd1740 1 -- 172.21.15.134:0/2568003364 wait complete. 2024-07-26T11:17:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.452+0000 7fa857bd1740 1 -- start start 2024-07-26T11:17:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.452+0000 7fa857bd1740 1 Processor -- start 2024-07-26T11:17:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.453+0000 7fa857bd1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819ee9c1b0 0x55819f2789b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.453+0000 7fa857bd1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819eea4390 0x55819f278ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.453+0000 7fa857bd1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eec5070 0x55819f2edc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.453+0000 7fa857bd1740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa84c1b82e0 con 0x55819eec5070 2024-07-26T11:17:12.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.453+0000 7fa857bd1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa84c1b7c10 con 0x55819eea4390 2024-07-26T11:17:12.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.453+0000 7fa857bd1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa84c1b4e50 con 0x55819ee9c1b0 2024-07-26T11:17:12.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.453+0000 7fa857bd1740 10 client.0 _ll_register_callbacks cb 0x7fa83c004a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:12.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.453+0000 7fa84a800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eec5070 0x55819f2edc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.453+0000 7fa849e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819ee9c1b0 0x55819f2789b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.453+0000 7fa849400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819eea4390 0x55819f278ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.453+0000 7fa84a800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eec5070 0x55819f2edc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38460/0 (socket says 172.21.15.134:38460) 2024-07-26T11:17:12.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.453+0000 7fa84a800640 1 -- 172.21.15.134:0/1958351881 learned_addr learned my addr 172.21.15.134:0/1958351881 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:12.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.453+0000 7fa849400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819eea4390 0x55819f278ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46500/0 (socket says 172.21.15.134:46500) 2024-07-26T11:17:12.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.454+0000 7fa849400640 1 -- 172.21.15.134:0/1958351881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819ee9c1b0 msgr2=0x55819f2789b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:12.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.454+0000 7fa849400640 1 --2- 172.21.15.134:0/1958351881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819ee9c1b0 0x55819f2789b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.454+0000 7fa849400640 1 -- 172.21.15.134:0/1958351881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eec5070 msgr2=0x55819f2edc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:12.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.454+0000 7fa849400640 1 --2- 172.21.15.134:0/1958351881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eec5070 0x55819f2edc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.454+0000 7fa849400640 1 -- 172.21.15.134:0/1958351881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55819f279710 con 0x55819eea4390 2024-07-26T11:17:12.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.454+0000 7fa849400640 1 --2- 172.21.15.134:0/1958351881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819eea4390 0x55819f278ef0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa8380c7c70 tx=0x7fa838003f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=171febb104d60d89 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:12.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.454+0000 7fa82be00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:17:12.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.454+0000 7fa82be00640 1 -- 172.21.15.134:0/1958351881 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa838048040 con 0x55819eea4390 2024-07-26T11:17:12.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.454+0000 7fa82be00640 1 -- 172.21.15.134:0/1958351881 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa8380c6c90 con 0x55819eea4390 2024-07-26T11:17:12.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.454+0000 7fa857bd1740 1 -- 172.21.15.134:0/1958351881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa83c019430 con 0x55819eea4390 2024-07-26T11:17:12.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.455+0000 7fa82be00640 1 -- 172.21.15.134:0/1958351881 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa8380cf000 con 0x55819eea4390 2024-07-26T11:17:12.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.455+0000 7fa857bd1740 10 client.25718 fetch_fsmap learned FSMap version 30 2024-07-26T11:17:12.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.455+0000 7fa857bd1740 1 -- 172.21.15.134:0/1958351881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55819f2ef230 con 0x55819eea4390 2024-07-26T11:17:12.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.455+0000 7fa82be00640 1 -- 172.21.15.134:0/1958351881 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa83c019430 con 0x55819eea4390 2024-07-26T11:17:12.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.455+0000 7fa82be00640 1 -- 172.21.15.134:0/1958351881 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fa8380cac60 con 0x55819eea4390 2024-07-26T11:17:12.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.455+0000 7fa857bd1740 10 client.25718 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:17:12.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.455+0000 7fa857bd1740 10 client.25718 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:17:12.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.455+0000 7fa857bd1740 20 client.25718 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:12.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.455+0000 7fa857bd1740 1 --2- 172.21.15.134:0/1958351881 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55819ee58da0 0x55819f2f19a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:12.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.455+0000 7fa857bd1740 4 client.25718 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:17:12.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.455+0000 7fa857bd1740 1 -- 172.21.15.134:0/1958351881 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55819f1de5c0 con 0x55819ee58da0 2024-07-26T11:17:12.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.455+0000 7fa849e00640 1 --2- 172.21.15.134:0/1958351881 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55819ee58da0 0x55819f2f19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:12.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.456+0000 7fa849e00640 1 --2- 172.21.15.134:0/1958351881 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55819ee58da0 0x55819f2f19a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59e8b5d4125274c server_cookie=369b8e8c62e9d56d in_seq=0 out_seq=0 2024-07-26T11:17:12.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.456+0000 7fa82be00640 10 client.25718 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:12.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.456+0000 7fa82be00640 1 -- 172.21.15.134:0/1958351881 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55819f1de5c0 con 0x55819ee58da0 2024-07-26T11:17:12.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.456+0000 7fa82be00640 10 client.25718 handle_command_reply: tid=0 2024-07-26T11:17:12.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.456+0000 7fa857bd1740 1 client.25718 shutdown 2024-07-26T11:17:12.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.457+0000 7fa857bd1740 1 -- 172.21.15.134:0/1958351881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819eea4390 msgr2=0x55819f278ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:12.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.457+0000 7fa857bd1740 1 --2- 172.21.15.134:0/1958351881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819eea4390 0x55819f278ef0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa8380c7c70 tx=0x7fa838003f70 comp rx=0 tx=0).stop 2024-07-26T11:17:12.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.457+0000 7fa857bd1740 1 -- 172.21.15.134:0/1958351881 shutdown_connections 2024-07-26T11:17:12.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.457+0000 7fa857bd1740 1 --2- 172.21.15.134:0/1958351881 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55819ee58da0 0x55819f2f19a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.457+0000 7fa857bd1740 1 --2- 172.21.15.134:0/1958351881 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55819eec5070 0x55819f2edc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.457+0000 7fa857bd1740 1 --2- 172.21.15.134:0/1958351881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55819eea4390 0x55819f278ef0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.457+0000 7fa857bd1740 1 --2- 172.21.15.134:0/1958351881 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55819ee9c1b0 0x55819f2789b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.457+0000 7fa82be00640 0 client.25718 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:12.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.457+0000 7fa857bd1740 1 -- 172.21.15.134:0/1958351881 >> 172.21.15.134:0/1958351881 conn(0x55819eeb9460 msgr2=0x55819f266da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:12.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.457+0000 7fa857bd1740 1 -- 172.21.15.134:0/1958351881 shutdown_connections 2024-07-26T11:17:12.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.457+0000 7fa857bd1740 1 -- 172.21.15.134:0/1958351881 wait complete. 2024-07-26T11:17:12.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.457+0000 7fa857bd1740 20 client.25718 trim_cache size 0 max 16384 2024-07-26T11:17:12.462 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:17:12.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.458+0000 7fa850e00640 1 -- 172.21.15.134:0/1575617879 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa8300b3d40 msgr2=0x7fa8300b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:12.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.458+0000 7fa850e00640 1 --2- 172.21.15.134:0/1575617879 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa8300b3d40 0x7fa8300b6250 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa844005fd0 tx=0x7fa844011be0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.458+0000 7fa850e00640 1 -- 172.21.15.134:0/1575617879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa84c107de0 msgr2=0x7fa84c1a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:12.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.458+0000 7fa850e00640 1 --2- 172.21.15.134:0/1575617879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa84c107de0 0x7fa84c1a0b60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa83802a930 tx=0x7fa838031a20 comp rx=0 tx=0).stop 2024-07-26T11:17:12.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.458+0000 7fa850e00640 1 -- 172.21.15.134:0/1575617879 shutdown_connections 2024-07-26T11:17:12.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.458+0000 7fa850e00640 1 --2- 172.21.15.134:0/1575617879 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa8300b3d40 0x7fa8300b6250 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.458+0000 7fa850e00640 1 --2- 172.21.15.134:0/1575617879 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa84c108730 0x7fa84c1a5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.458+0000 7fa850e00640 1 --2- 172.21.15.134:0/1575617879 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa84c107de0 0x7fa84c1a0b60 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.458+0000 7fa850e00640 1 --2- 172.21.15.134:0/1575617879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84c1024f0 0x7fa84c1a05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:12.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.458+0000 7fa850e00640 1 -- 172.21.15.134:0/1575617879 >> 172.21.15.134:0/1575617879 conn(0x7fa84c079150 msgr2=0x7fa84c06b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:12.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.458+0000 7fa850e00640 1 -- 172.21.15.134:0/1575617879 shutdown_connections 2024-07-26T11:17:12.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:12.458+0000 7fa850e00640 1 -- 172.21.15.134:0/1575617879 wait complete. 2024-07-26T11:17:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:12 smithi193 ceph-mon[37473]: pgmap v517: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-07-26T11:17:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.710+0000 7fbc53200640 1 -- 172.21.15.134:0/1252689338 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc4c108c70 msgr2=0x7fbc4c109080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:13.712 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.710+0000 7fbc53200640 1 --2- 172.21.15.134:0/1252689338 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc4c108c70 0x7fbc4c109080 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c009900 tx=0x7fbc3c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.710+0000 7fbc53200640 1 -- 172.21.15.134:0/1252689338 shutdown_connections 2024-07-26T11:17:13.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.710+0000 7fbc53200640 1 --2- 172.21.15.134:0/1252689338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c1095c0 0x7fbc4c10fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.710+0000 7fbc53200640 1 --2- 172.21.15.134:0/1252689338 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc4c108c70 0x7fbc4c109080 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.710+0000 7fbc53200640 1 --2- 172.21.15.134:0/1252689338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc4c103340 0x7fbc4c103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.710+0000 7fbc53200640 1 -- 172.21.15.134:0/1252689338 >> 172.21.15.134:0/1252689338 conn(0x7fbc4c0fed60 msgr2=0x7fbc4c1011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:13.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.710+0000 7fbc53200640 1 -- 172.21.15.134:0/1252689338 shutdown_connections 2024-07-26T11:17:13.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.710+0000 7fbc53200640 1 -- 172.21.15.134:0/1252689338 wait complete. 2024-07-26T11:17:13.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.712+0000 7fbc53200640 1 Processor -- start 2024-07-26T11:17:13.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.712+0000 7fbc53200640 1 -- start start 2024-07-26T11:17:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.712+0000 7fbc53200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc4c103340 0x7fbc4c199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.712+0000 7fbc53200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc4c108c70 0x7fbc4c19a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.712+0000 7fbc53200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c1095c0 0x7fbc4c19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.712+0000 7fbc53200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c113250 con 0x7fbc4c108c70 2024-07-26T11:17:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.712+0000 7fbc53200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c1130d0 con 0x7fbc4c1095c0 2024-07-26T11:17:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.712+0000 7fbc53200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c1133d0 con 0x7fbc4c103340 2024-07-26T11:17:13.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.713+0000 7fbc51400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc4c108c70 0x7fbc4c19a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.713+0000 7fbc51e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc4c103340 0x7fbc4c199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.713+0000 7fbc52800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c1095c0 0x7fbc4c19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.713+0000 7fbc51400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc4c108c70 0x7fbc4c19a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38474/0 (socket says 172.21.15.134:38474) 2024-07-26T11:17:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.713+0000 7fbc51e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc4c103340 0x7fbc4c199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51396/0 (socket says 172.21.15.134:51396) 2024-07-26T11:17:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.713+0000 7fbc51400640 1 -- 172.21.15.134:0/1319127739 learned_addr learned my addr 172.21.15.134:0/1319127739 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.713+0000 7fbc52800640 1 -- 172.21.15.134:0/1319127739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc4c103340 msgr2=0x7fbc4c199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.713+0000 7fbc52800640 1 --2- 172.21.15.134:0/1319127739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc4c103340 0x7fbc4c199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.713+0000 7fbc52800640 1 -- 172.21.15.134:0/1319127739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc4c108c70 msgr2=0x7fbc4c19a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.713+0000 7fbc52800640 1 --2- 172.21.15.134:0/1319127739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc4c108c70 0x7fbc4c19a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.713+0000 7fbc52800640 1 -- 172.21.15.134:0/1319127739 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4c19fc20 con 0x7fbc4c1095c0 2024-07-26T11:17:13.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.713+0000 7fbc51400640 1 --2- 172.21.15.134:0/1319127739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc4c108c70 0x7fbc4c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.713+0000 7fbc52800640 1 --2- 172.21.15.134:0/1319127739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c1095c0 0x7fbc4c19f520 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbc4800ec50 tx=0x7fbc4800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.714+0000 7fbc46a00640 1 -- 172.21.15.134:0/1319127739 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc48003b70 con 0x7fbc4c1095c0 2024-07-26T11:17:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.714+0000 7fbc46a00640 1 -- 172.21.15.134:0/1319127739 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc48003d10 con 0x7fbc4c1095c0 2024-07-26T11:17:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.714+0000 7fbc53200640 1 -- 172.21.15.134:0/1319127739 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4c19ff10 con 0x7fbc4c1095c0 2024-07-26T11:17:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.714+0000 7fbc46a00640 1 -- 172.21.15.134:0/1319127739 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc48010830 con 0x7fbc4c1095c0 2024-07-26T11:17:13.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.714+0000 7fbc53200640 1 -- 172.21.15.134:0/1319127739 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc4c075020 con 0x7fbc4c1095c0 2024-07-26T11:17:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.717+0000 7fbc5b7f2740 1 Processor -- start 2024-07-26T11:17:13.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.718+0000 7fbc46a00640 1 -- 172.21.15.134:0/1319127739 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fbc48003ec0 con 0x7fbc4c1095c0 2024-07-26T11:17:13.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.718+0000 7fbc5b7f2740 1 -- start start 2024-07-26T11:17:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.718+0000 7fbc5b7f2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161871b0 0x55911652ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.718+0000 7fbc5b7f2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x55911652b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.718+0000 7fbc5b7f2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161664d0 0x55911652b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.718+0000 7fbc5b7f2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55911649f270 con 0x5591161871b0 2024-07-26T11:17:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.718+0000 7fbc5b7f2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55911648e770 con 0x5591161664d0 2024-07-26T11:17:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.718+0000 7fbc5b7f2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5591164927b0 con 0x55911615e2f0 2024-07-26T11:17:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.719+0000 7fbc51400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161871b0 0x55911652ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.719+0000 7fbc51400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161871b0 0x55911652ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38490/0 (socket says 172.21.15.134:38490) 2024-07-26T11:17:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.719+0000 7fbc51400640 1 -- 172.21.15.134:0/1754072225 learned_addr learned my addr 172.21.15.134:0/1754072225 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.719+0000 7fbc51e00640 1 --2- 172.21.15.134:0/1754072225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x55911652b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.719+0000 7fbc51400640 1 --2- 172.21.15.134:0/1754072225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161664d0 0x55911652b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.719+0000 7fbc51e00640 1 -- 172.21.15.134:0/1754072225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161664d0 msgr2=0x55911652b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.719+0000 7fbc51e00640 1 --2- 172.21.15.134:0/1754072225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161664d0 0x55911652b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.719+0000 7fbc51e00640 1 -- 172.21.15.134:0/1754072225 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161871b0 msgr2=0x55911652ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.719+0000 7fbc51e00640 1 --2- 172.21.15.134:0/1754072225 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161871b0 0x55911652ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.719+0000 7fbc51e00640 1 -- 172.21.15.134:0/1754072225 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591161f2790 con 0x55911615e2f0 2024-07-26T11:17:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.719+0000 7fbc51e00640 1 --2- 172.21.15.134:0/1754072225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x55911652b160 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbc40009c30 tx=0x7fbc40011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=417df39be2124739 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.720+0000 7fbc3b000640 1 -- 172.21.15.134:0/1754072225 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc40004230 con 0x55911615e2f0 2024-07-26T11:17:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.720+0000 7fbc3b000640 1 -- 172.21.15.134:0/1754072225 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc400043d0 con 0x55911615e2f0 2024-07-26T11:17:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.720+0000 7fbc3b000640 1 -- 172.21.15.134:0/1754072225 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc40004c60 con 0x55911615e2f0 2024-07-26T11:17:13.727 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.720+0000 7fbc46a00640 1 --2- 172.21.15.134:0/1319127739 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbc340b3d40 0x7fbc340b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.727 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.720+0000 7fbc51400640 1 --2- 172.21.15.134:0/1319127739 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbc340b3d40 0x7fbc340b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.727 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.720+0000 7fbc46a00640 1 -- 172.21.15.134:0/1319127739 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fbc480e4940 con 0x7fbc4c1095c0 2024-07-26T11:17:13.727 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.721+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/1754072225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 msgr2=0x55911652b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:13.727 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.721+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/1754072225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x55911652b160 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbc40009c30 tx=0x7fbc40011810 comp rx=0 tx=0).stop 2024-07-26T11:17:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.721+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/1754072225 shutdown_connections 2024-07-26T11:17:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.721+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/1754072225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x55911652b160 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.721+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/1754072225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161664d0 0x55911652b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.721+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/1754072225 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161871b0 0x55911652ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.721+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/1754072225 >> 172.21.15.134:0/1754072225 conn(0x55911617b300 msgr2=0x559116528b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.721+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/1754072225 shutdown_connections 2024-07-26T11:17:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.721+0000 7fbc51400640 1 --2- 172.21.15.134:0/1319127739 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbc340b3d40 0x7fbc340b6250 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0096c0 tx=0x7fbc3c032ed0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:13.728 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.721+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/1754072225 wait complete. 2024-07-26T11:17:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.722+0000 7fbc5b7f2740 1 -- start start 2024-07-26T11:17:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.722+0000 7fbc5b7f2740 1 Processor -- start 2024-07-26T11:17:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.722+0000 7fbc5b7f2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x559116536810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.722+0000 7fbc5b7f2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161664d0 0x559116536d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.722+0000 7fbc5b7f2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161871b0 0x5591165426c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.722+0000 7fbc5b7f2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5591163c8ab0 con 0x5591161664d0 2024-07-26T11:17:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.722+0000 7fbc5b7f2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55911642bd50 con 0x5591161871b0 2024-07-26T11:17:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.722+0000 7fbc5b7f2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5591165393a0 con 0x55911615e2f0 2024-07-26T11:17:13.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.723+0000 7fbc5b7f2740 10 client.0 _ll_register_callbacks cb 0x55911619ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.723+0000 7fbc51400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161664d0 0x559116536d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.723+0000 7fbc51e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x559116536810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.723+0000 7fbc51400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161664d0 0x559116536d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38504/0 (socket says 172.21.15.134:38504) 2024-07-26T11:17:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.723+0000 7fbc51400640 1 -- 172.21.15.134:0/2557334953 learned_addr learned my addr 172.21.15.134:0/2557334953 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.723+0000 7fbc52800640 1 --2- 172.21.15.134:0/2557334953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161871b0 0x5591165426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.723+0000 7fbc51400640 1 -- 172.21.15.134:0/2557334953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 msgr2=0x559116536810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.723+0000 7fbc51400640 1 --2- 172.21.15.134:0/2557334953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x559116536810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.723+0000 7fbc51400640 1 -- 172.21.15.134:0/2557334953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161871b0 msgr2=0x5591165426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.723+0000 7fbc51400640 1 --2- 172.21.15.134:0/2557334953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161871b0 0x5591165426c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.723+0000 7fbc51400640 1 -- 172.21.15.134:0/2557334953 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55911626b620 con 0x5591161664d0 2024-07-26T11:17:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.723+0000 7fbc51400640 1 --2- 172.21.15.134:0/2557334953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161664d0 0x559116536d70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0057a0 tx=0x7fbc3c03c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8daacce95a70e1e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:13.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.723+0000 7fbc2f400640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:17:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.724+0000 7fbc2f400640 1 -- 172.21.15.134:0/2557334953 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c035040 con 0x5591161664d0 2024-07-26T11:17:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.724+0000 7fbc2f400640 1 -- 172.21.15.134:0/2557334953 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fbc3c004980 con 0x5591161664d0 2024-07-26T11:17:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.724+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/2557334953 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc400043d0 con 0x5591161664d0 2024-07-26T11:17:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.724+0000 7fbc2f400640 1 -- 172.21.15.134:0/2557334953 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbc3c004b80 con 0x5591161664d0 2024-07-26T11:17:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.724+0000 7fbc5b7f2740 10 client.15558 fetch_fsmap learned FSMap version 30 2024-07-26T11:17:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.724+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/2557334953 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x559116545430 con 0x5591161664d0 2024-07-26T11:17:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.724+0000 7fbc2f400640 1 -- 172.21.15.134:0/2557334953 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc400043d0 con 0x5591161664d0 2024-07-26T11:17:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.725+0000 7fbc2f400640 1 -- 172.21.15.134:0/2557334953 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fbc3c053070 con 0x5591161664d0 2024-07-26T11:17:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.725+0000 7fbc5b7f2740 10 client.15558 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:17:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.725+0000 7fbc5b7f2740 10 client.15558 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:17:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.725+0000 7fbc5b7f2740 20 client.15558 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.725+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/2557334953 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559116145d80 0x559116548960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.725+0000 7fbc5b7f2740 4 client.15558 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.725+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/2557334953 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591164a11c0 con 0x559116145d80 2024-07-26T11:17:13.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.725+0000 7fbc51e00640 1 --2- 172.21.15.134:0/2557334953 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559116145d80 0x559116548960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.726+0000 7fbc51e00640 1 --2- 172.21.15.134:0/2557334953 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559116145d80 0x559116548960 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69e18fe287d633d3 server_cookie=ee0c6ef82c64c200 in_seq=0 out_seq=0 2024-07-26T11:17:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.726+0000 7fbc2f400640 10 client.15558 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.726+0000 7fbc2f400640 1 -- 172.21.15.134:0/2557334953 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5591164a11c0 con 0x559116145d80 2024-07-26T11:17:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.726+0000 7fbc2f400640 10 client.15558 handle_command_reply: tid=0 2024-07-26T11:17:13.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.726+0000 7fbc5b7f2740 1 client.15558 shutdown 2024-07-26T11:17:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.727+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/2557334953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161664d0 msgr2=0x559116536d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.727+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/2557334953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161664d0 0x559116536d70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0057a0 tx=0x7fbc3c03c040 comp rx=0 tx=0).stop 2024-07-26T11:17:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.727+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/2557334953 shutdown_connections 2024-07-26T11:17:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.727+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/2557334953 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559116145d80 0x559116548960 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.727+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/2557334953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161871b0 0x5591165426c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.727+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/2557334953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161664d0 0x559116536d70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.727+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/2557334953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x559116536810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.727+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/2557334953 >> 172.21.15.134:0/2557334953 conn(0x55911617b300 msgr2=0x55911652bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:13.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.727+0000 7fbc2f400640 0 client.15558 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.727+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/2557334953 shutdown_connections 2024-07-26T11:17:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.727+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/2557334953 wait complete. 2024-07-26T11:17:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.727+0000 7fbc5b7f2740 20 client.15558 trim_cache size 0 max 16384 2024-07-26T11:17:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.743+0000 7fbc5b7f2740 1 Processor -- start 2024-07-26T11:17:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.743+0000 7fbc5b7f2740 1 -- start start 2024-07-26T11:17:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.744+0000 7fbc5b7f2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x5591165424b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.744+0000 7fbc5b7f2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161664d0 0x559116535ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.744+0000 7fbc5b7f2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161871b0 0x559116536010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.744+0000 7fbc5b7f2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c1ae410 con 0x5591161871b0 2024-07-26T11:17:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.744+0000 7fbc5b7f2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c1aff50 con 0x5591161664d0 2024-07-26T11:17:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.744+0000 7fbc5b7f2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c1aeae0 con 0x55911615e2f0 2024-07-26T11:17:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.744+0000 7fbc52800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161871b0 0x559116536010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.744+0000 7fbc51400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161664d0 0x559116535ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.744+0000 7fbc51400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161664d0 0x559116535ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46552/0 (socket says 172.21.15.134:46552) 2024-07-26T11:17:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.744+0000 7fbc52800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161871b0 0x559116536010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38506/0 (socket says 172.21.15.134:38506) 2024-07-26T11:17:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.744+0000 7fbc51400640 1 -- 172.21.15.134:0/3936225651 learned_addr learned my addr 172.21.15.134:0/3936225651 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.744+0000 7fbc51e00640 1 --2- 172.21.15.134:0/3936225651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x5591165424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.745+0000 7fbc51400640 1 -- 172.21.15.134:0/3936225651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 msgr2=0x5591165424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.745+0000 7fbc51400640 1 --2- 172.21.15.134:0/3936225651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x5591165424b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.745+0000 7fbc51400640 1 -- 172.21.15.134:0/3936225651 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161871b0 msgr2=0x559116536010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:13.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.745+0000 7fbc51400640 1 --2- 172.21.15.134:0/3936225651 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161871b0 0x559116536010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.745+0000 7fbc51400640 1 -- 172.21.15.134:0/3936225651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559116529410 con 0x5591161664d0 2024-07-26T11:17:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.745+0000 7fbc51400640 1 --2- 172.21.15.134:0/3936225651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161664d0 0x559116535ad0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c055570 tx=0x7fbc3c02a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92735d9a5e417fed server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.745+0000 7fbc3a600640 1 -- 172.21.15.134:0/3936225651 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c035040 con 0x5591161664d0 2024-07-26T11:17:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.745+0000 7fbc3a600640 1 -- 172.21.15.134:0/3936225651 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc3c044040 con 0x5591161664d0 2024-07-26T11:17:13.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.745+0000 7fbc3a600640 1 -- 172.21.15.134:0/3936225651 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c055df0 con 0x5591161664d0 2024-07-26T11:17:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.746+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/3936225651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161664d0 msgr2=0x559116535ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:13.752 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.746+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/3936225651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161664d0 0x559116535ad0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c055570 tx=0x7fbc3c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.746+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/3936225651 shutdown_connections 2024-07-26T11:17:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.746+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/3936225651 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161871b0 0x559116536010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.746+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/3936225651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161664d0 0x559116535ad0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.746+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/3936225651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x5591165424b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.746+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/3936225651 >> 172.21.15.134:0/3936225651 conn(0x55911617b300 msgr2=0x559116534e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.746+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/3936225651 shutdown_connections 2024-07-26T11:17:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.746+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/3936225651 wait complete. 2024-07-26T11:17:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.747+0000 7fbc5b7f2740 1 -- start start 2024-07-26T11:17:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.747+0000 7fbc5b7f2740 1 Processor -- start 2024-07-26T11:17:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.747+0000 7fbc5b7f2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x55911653ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc5b7f2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161664d0 0x5591165395e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc5b7f2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161871b0 0x559116539c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc5b7f2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c1af880 con 0x5591161664d0 2024-07-26T11:17:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc5b7f2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c1af1b0 con 0x5591161871b0 2024-07-26T11:17:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc5b7f2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c1ac3f0 con 0x55911615e2f0 2024-07-26T11:17:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc5b7f2740 10 client.0 _ll_register_callbacks cb 0x7fbc3c031650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc52800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161871b0 0x559116539c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc52800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161871b0 0x559116539c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46564/0 (socket says 172.21.15.134:46564) 2024-07-26T11:17:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc52800640 1 -- 172.21.15.134:0/2262906034 learned_addr learned my addr 172.21.15.134:0/2262906034 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc51400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161664d0 0x5591165395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc51e00640 1 --2- 172.21.15.134:0/2262906034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x55911653ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc52800640 1 -- 172.21.15.134:0/2262906034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 msgr2=0x55911653ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc52800640 1 --2- 172.21.15.134:0/2262906034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x55911653ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc52800640 1 -- 172.21.15.134:0/2262906034 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161664d0 msgr2=0x5591165395e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc52800640 1 --2- 172.21.15.134:0/2262906034 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161664d0 0x5591165395e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc52800640 1 -- 172.21.15.134:0/2262906034 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55911653a3f0 con 0x5591161871b0 2024-07-26T11:17:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc51400640 1 --2- 172.21.15.134:0/2262906034 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161664d0 0x5591165395e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.748+0000 7fbc52800640 1 --2- 172.21.15.134:0/2262906034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161871b0 0x559116539c60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbc480a83e0 tx=0x7fbc480aa9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7bd21b204ed30d3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.749+0000 7fbc2f400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:17:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.749+0000 7fbc2f400640 1 -- 172.21.15.134:0/2262906034 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc480aac80 con 0x5591161871b0 2024-07-26T11:17:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.749+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/2262906034 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbc3c044040 con 0x5591161871b0 2024-07-26T11:17:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.749+0000 7fbc2f400640 1 -- 172.21.15.134:0/2262906034 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fbc480aa5e0 con 0x5591161871b0 2024-07-26T11:17:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.749+0000 7fbc2f400640 1 -- 172.21.15.134:0/2262906034 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbc480ab870 con 0x5591161871b0 2024-07-26T11:17:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.749+0000 7fbc5b7f2740 10 client.25736 fetch_fsmap learned FSMap version 30 2024-07-26T11:17:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.749+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/2262906034 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5591165b01c0 con 0x5591161871b0 2024-07-26T11:17:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.749+0000 7fbc2f400640 1 -- 172.21.15.134:0/2262906034 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc3c044040 con 0x5591161871b0 2024-07-26T11:17:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.750+0000 7fbc2f400640 1 -- 172.21.15.134:0/2262906034 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fbc480cc1f0 con 0x5591161871b0 2024-07-26T11:17:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.750+0000 7fbc5b7f2740 10 client.25736 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:17:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.750+0000 7fbc5b7f2740 10 client.25736 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:17:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.750+0000 7fbc5b7f2740 20 client.25736 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.750+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/2262906034 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55911611b1c0 0x5591165b2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.750+0000 7fbc5b7f2740 4 client.25736 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:17:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.750+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/2262906034 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55911649cc60 con 0x55911611b1c0 2024-07-26T11:17:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.750+0000 7fbc51e00640 1 --2- 172.21.15.134:0/2262906034 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55911611b1c0 0x5591165b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.751+0000 7fbc51e00640 1 --2- 172.21.15.134:0/2262906034 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55911611b1c0 0x5591165b2880 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebbf69e59888d0cb server_cookie=fe31d154788896cd in_seq=0 out_seq=0 2024-07-26T11:17:13.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.751+0000 7fbc2f400640 10 client.25736 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:13.760 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.758+0000 7fbc2f400640 1 -- 172.21.15.134:0/2262906034 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55911649cc60 con 0x55911611b1c0 2024-07-26T11:17:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.758+0000 7fbc2f400640 10 client.25736 handle_command_reply: tid=0 2024-07-26T11:17:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.758+0000 7fbc5b7f2740 1 client.25736 shutdown 2024-07-26T11:17:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.758+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/2262906034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161871b0 msgr2=0x559116539c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.758+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/2262906034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161871b0 0x559116539c60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbc480a83e0 tx=0x7fbc480aa9b0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.758+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/2262906034 shutdown_connections 2024-07-26T11:17:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.758+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/2262906034 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55911611b1c0 0x5591165b2880 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.758+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/2262906034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5591161871b0 0x559116539c60 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.758+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/2262906034 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5591161664d0 0x5591165395e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.758+0000 7fbc2f400640 0 client.25736 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.758+0000 7fbc5b7f2740 1 --2- 172.21.15.134:0/2262906034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55911615e2f0 0x55911653ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.758+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/2262906034 >> 172.21.15.134:0/2262906034 conn(0x55911617b300 msgr2=0x559116534a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:13.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.758+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/2262906034 shutdown_connections 2024-07-26T11:17:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.758+0000 7fbc5b7f2740 1 -- 172.21.15.134:0/2262906034 wait complete. 2024-07-26T11:17:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.758+0000 7fbc5b7f2740 20 client.25736 trim_cache size 0 max 16384 2024-07-26T11:17:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:17:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:17:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:17:13.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:17:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.759+0000 7fbc53200640 1 -- 172.21.15.134:0/1319127739 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbc340b3d40 msgr2=0x7fbc340b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.759+0000 7fbc53200640 1 --2- 172.21.15.134:0/1319127739 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbc340b3d40 0x7fbc340b6250 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0096c0 tx=0x7fbc3c032ed0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.759+0000 7fbc53200640 1 -- 172.21.15.134:0/1319127739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c1095c0 msgr2=0x7fbc4c19f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.759+0000 7fbc53200640 1 --2- 172.21.15.134:0/1319127739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c1095c0 0x7fbc4c19f520 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbc4800ec50 tx=0x7fbc4800c6a0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.759+0000 7fbc53200640 1 -- 172.21.15.134:0/1319127739 shutdown_connections 2024-07-26T11:17:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.759+0000 7fbc53200640 1 --2- 172.21.15.134:0/1319127739 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fbc340b3d40 0x7fbc340b6250 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.759+0000 7fbc53200640 1 --2- 172.21.15.134:0/1319127739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc4c1095c0 0x7fbc4c19f520 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.759+0000 7fbc53200640 1 --2- 172.21.15.134:0/1319127739 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fbc4c108c70 0x7fbc4c19a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.759+0000 7fbc53200640 1 --2- 172.21.15.134:0/1319127739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc4c103340 0x7fbc4c199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.759+0000 7fbc53200640 1 -- 172.21.15.134:0/1319127739 >> 172.21.15.134:0/1319127739 conn(0x7fbc4c0fed60 msgr2=0x7fbc4c101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.759+0000 7fbc53200640 1 -- 172.21.15.134:0/1319127739 shutdown_connections 2024-07-26T11:17:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:13.759+0000 7fbc53200640 1 -- 172.21.15.134:0/1319127739 wait complete. 2024-07-26T11:17:14.251 INFO:tasks.workunit.client.0.smithi134.stderr:Updating files: 58% (7470/12697) Updating files: 59% (7492/12697) Updating files: 60% (7619/12697) Updating files: 61% (7746/12697) Updating files: 62% (7873/12697) Updating files: 63% (8000/12697) Updating files: 64% (8127/12697) Updating files: 65% (8254/12697) Updating files: 66% (8381/12697) Updating files: 67% (8507/12697) Updating files: 68% (8634/12697) Updating files: 69% (8761/12697) Updating files: 70% (8888/12697) Updating files: 71% (9015/12697) Updating files: 72% (9142/12697) Updating files: 73% (9269/12697) Updating files: 74% (9396/12697) Updating files: 75% (9523/12697) Updating files: 76% (9650/12697) Updating files: 77% (9777/12697) Updating files: 78% (9904/12697) Updating files: 79% (10031/12697) Updating files: 80% (10158/12697) Updating files: 81% (10285/12697) Updating files: 82% (10412/12697) Updating files: 83% (10539/12697) Updating files: 84% (10666/12697) Updating files: 85% (10793/12697) Updating files: 86% (10920/12697) Updating files: 87% (11047/12697) Updating files: 88% (11174/12697) Updating files: 89% (11301/12697) Updating files: 90% (11428/12697) Updating files: 91% (11555/12697) Updating files: 92% (11682/12697) Updating files: 93% (11809/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 98% (12481/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-07-26T11:17:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:14 smithi134 ceph-mon[31966]: pgmap v518: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-07-26T11:17:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:14 smithi189 ceph-mon[36790]: pgmap v518: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-07-26T11:17:14.467 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T11:17:14.467 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T11:17:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:14 smithi193 ceph-mon[37473]: pgmap v518: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-07-26T11:17:14.689 INFO:tasks.workunit.client.0.smithi134.stderr:Note: switching to 'e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528'. 2024-07-26T11:17:14.689 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-07-26T11:17:14.690 INFO:tasks.workunit.client.0.smithi134.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-07-26T11:17:14.690 INFO:tasks.workunit.client.0.smithi134.stderr:changes and commit them, and you can discard any commits you make in this 2024-07-26T11:17:14.690 INFO:tasks.workunit.client.0.smithi134.stderr:state without impacting any branches by switching back to a branch. 2024-07-26T11:17:14.690 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-07-26T11:17:14.690 INFO:tasks.workunit.client.0.smithi134.stderr:If you want to create a new branch to retain commits you create, you may 2024-07-26T11:17:14.690 INFO:tasks.workunit.client.0.smithi134.stderr:do so (now or later) by using -c with the switch command. Example: 2024-07-26T11:17:14.690 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-07-26T11:17:14.690 INFO:tasks.workunit.client.0.smithi134.stderr: git switch -c 2024-07-26T11:17:14.690 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-07-26T11:17:14.690 INFO:tasks.workunit.client.0.smithi134.stderr:Or undo this operation with: 2024-07-26T11:17:14.690 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-07-26T11:17:14.691 INFO:tasks.workunit.client.0.smithi134.stderr: git switch - 2024-07-26T11:17:14.691 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-07-26T11:17:14.691 INFO:tasks.workunit.client.0.smithi134.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-07-26T11:17:14.691 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-07-26T11:17:14.691 INFO:tasks.workunit.client.0.smithi134.stderr:HEAD is now at e8f88bb1fe0 Merge PR #54435 into wip-vshankar-testing-20240725.121331-debug 2024-07-26T11:17:14.699 DEBUG:teuthology.orchestra.run.smithi134:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-07-26T11:17:14.750 INFO:tasks.workunit.client.0.smithi134.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-07-26T11:17:14.756 INFO:tasks.workunit.client.0.smithi134.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-26T11:17:14.756 INFO:tasks.workunit.client.0.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-07-26T11:17:14.908 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:17:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:16 smithi134 ceph-mon[31966]: pgmap v519: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 341 B/s wr, 9 op/s 2024-07-26T11:17:16.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:16 smithi189 ceph-mon[36790]: pgmap v519: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 341 B/s wr, 9 op/s 2024-07-26T11:17:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:16 smithi193 ceph-mon[37473]: pgmap v519: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 341 B/s wr, 9 op/s 2024-07-26T11:17:16.601 INFO:tasks.workunit.client.0.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-07-26T11:17:16.786 INFO:tasks.workunit.client.0.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-07-26T11:17:16.866 INFO:tasks.workunit.client.0.smithi134.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-07-26T11:17:16.871 INFO:tasks.workunit.client.0.smithi134.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-26T11:17:16.872 INFO:tasks.workunit.client.0.smithi134.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-07-26T11:17:16.951 INFO:tasks.workunit.client.0.smithi134.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-07-26T11:17:16.960 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-07-26T11:17:16.960 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-07-26T11:17:17.001 INFO:tasks.workunit:Running workunits matching suites/iogen.sh on client.0... 2024-07-26T11:17:17.002 INFO:tasks.workunit:Running workunit suites/iogen.sh... 2024-07-26T11:17:17.002 DEBUG:teuthology.orchestra.run.smithi134:workunit test suites/iogen.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-07-26T11:17:17.125 INFO:tasks.workunit.client.0.smithi134.stdout:getting iogen 2024-07-26T11:17:17.126 INFO:tasks.workunit.client.0.smithi134.stderr:+ echo 'getting iogen' 2024-07-26T11:17:17.126 INFO:tasks.workunit.client.0.smithi134.stderr:+ wget http://download.ceph.com/qa/iogen_3.1p0.tar 2024-07-26T11:17:17.258 INFO:tasks.workunit.client.0.smithi134.stderr:--2024-07-26 11:17:17-- http://download.ceph.com/qa/iogen_3.1p0.tar 2024-07-26T11:17:17.270 INFO:tasks.workunit.client.0.smithi134.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-07-26T11:17:17.292 INFO:tasks.workunit.client.0.smithi134.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-07-26T11:17:17.312 INFO:tasks.workunit.client.0.smithi134.stderr:HTTP request sent, awaiting response... 200 OK 2024-07-26T11:17:17.312 INFO:tasks.workunit.client.0.smithi134.stderr:Length: 8071 (7.9K) [application/octet-stream] 2024-07-26T11:17:17.316 INFO:tasks.workunit.client.0.smithi134.stderr:Saving to: ‘iogen_3.1p0.tar’ 2024-07-26T11:17:17.323 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-07-26T11:17:17.323 INFO:tasks.workunit.client.0.smithi134.stderr: 0K ....... 100% 275M=0s 2024-07-26T11:17:17.324 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-07-26T11:17:17.324 INFO:tasks.workunit.client.0.smithi134.stderr:2024-07-26 11:17:17 (275 MB/s) - ‘iogen_3.1p0.tar’ saved [8071/8071] 2024-07-26T11:17:17.324 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-07-26T11:17:17.324 INFO:tasks.workunit.client.0.smithi134.stderr:+ tar -xvzf iogen_3.1p0.tar 2024-07-26T11:17:17.346 INFO:tasks.workunit.client.0.smithi134.stdout:iogen_3.1p0 2024-07-26T11:17:17.348 INFO:tasks.workunit.client.0.smithi134.stdout:iogen_3.1p0/src 2024-07-26T11:17:17.349 INFO:tasks.workunit.client.0.smithi134.stdout:iogen_3.1p0/src/iogen.8 2024-07-26T11:17:17.350 INFO:tasks.workunit.client.0.smithi134.stdout:iogen_3.1p0/src/iogen.c 2024-07-26T11:17:17.352 INFO:tasks.workunit.client.0.smithi134.stdout:iogen_3.1p0/README 2024-07-26T11:17:17.354 INFO:tasks.workunit.client.0.smithi134.stdout:iogen_3.1p0/Makefile.bsd 2024-07-26T11:17:17.355 INFO:tasks.workunit.client.0.smithi134.stdout:iogen_3.1p0/Makefile 2024-07-26T11:17:17.357 INFO:tasks.workunit.client.0.smithi134.stderr:+ cd iogen_3.1p0 2024-07-26T11:17:17.357 INFO:tasks.workunit.client.0.smithi134.stderr:+ echo 'making iogen' 2024-07-26T11:17:17.358 INFO:tasks.workunit.client.0.smithi134.stderr:+ make 2024-07-26T11:17:17.358 INFO:tasks.workunit.client.0.smithi134.stdout:making iogen 2024-07-26T11:17:17.371 INFO:tasks.workunit.client.0.smithi134.stdout:gcc -D_FILE_OFFSET_BITS=64 -DVERSION=\"3.1p0\" -Dstrlcpy=strncpy -Darc4random=rand -O2 src/iogen.c -o iogen 2024-07-26T11:17:17.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.715+0000 7f5e80a00640 1 -- 172.21.15.134:0/1483074525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e7c109670 msgr2=0x7f5e7c10fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:17.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.715+0000 7f5e80a00640 1 --2- 172.21.15.134:0/1483074525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e7c109670 0x7f5e7c10fea0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009ca0 tx=0x7f5e7402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.715+0000 7f5e80a00640 1 -- 172.21.15.134:0/1483074525 shutdown_connections 2024-07-26T11:17:17.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.715+0000 7f5e80a00640 1 --2- 172.21.15.134:0/1483074525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e7c108d20 0x7f5e7c109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.715+0000 7f5e80a00640 1 --2- 172.21.15.134:0/1483074525 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e7c109670 0x7f5e7c10fea0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.715+0000 7f5e80a00640 1 --2- 172.21.15.134:0/1483074525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e7c1033e0 0x7f5e7c1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.715+0000 7f5e80a00640 1 -- 172.21.15.134:0/1483074525 >> 172.21.15.134:0/1483074525 conn(0x7f5e7c0fee70 msgr2=0x7f5e7c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:17.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.716+0000 7f5e80a00640 1 -- 172.21.15.134:0/1483074525 shutdown_connections 2024-07-26T11:17:17.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.716+0000 7f5e80a00640 1 -- 172.21.15.134:0/1483074525 wait complete. 2024-07-26T11:17:17.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.717+0000 7f5e80a00640 1 Processor -- start 2024-07-26T11:17:17.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.717+0000 7f5e80a00640 1 -- start start 2024-07-26T11:17:17.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.717+0000 7f5e80a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e7c1033e0 0x7f5e7c199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.717+0000 7f5e80a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e7c108d20 0x7f5e7c19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.717+0000 7f5e80a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e7c109670 0x7f5e7c19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.717+0000 7f5e80a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c113390 con 0x7f5e7c109670 2024-07-26T11:17:17.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.717+0000 7f5e80a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c113210 con 0x7f5e7c1033e0 2024-07-26T11:17:17.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.717+0000 7f5e80a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c113510 con 0x7f5e7c108d20 2024-07-26T11:17:17.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.718+0000 7f5e7aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e7c108d20 0x7f5e7c19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.718+0000 7f5e7b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e7c1033e0 0x7f5e7c199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.718+0000 7f5e7aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e7c108d20 0x7f5e7c19a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51452/0 (socket says 172.21.15.134:51452) 2024-07-26T11:17:17.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.718+0000 7f5e7be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e7c109670 0x7f5e7c19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.718+0000 7f5e7aa00640 1 -- 172.21.15.134:0/1553481838 learned_addr learned my addr 172.21.15.134:0/1553481838 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:17.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.718+0000 7f5e7aa00640 1 -- 172.21.15.134:0/1553481838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e7c1033e0 msgr2=0x7f5e7c199ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:17.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.718+0000 7f5e7aa00640 1 --2- 172.21.15.134:0/1553481838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e7c1033e0 0x7f5e7c199ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.718+0000 7f5e7aa00640 1 -- 172.21.15.134:0/1553481838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e7c109670 msgr2=0x7f5e7c19f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:17.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.718+0000 7f5e7aa00640 1 --2- 172.21.15.134:0/1553481838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e7c109670 0x7f5e7c19f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.718+0000 7f5e7aa00640 1 -- 172.21.15.134:0/1553481838 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e7c19fc50 con 0x7f5e7c108d20 2024-07-26T11:17:17.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.719+0000 7f5e7aa00640 1 --2- 172.21.15.134:0/1553481838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e7c108d20 0x7f5e7c19a280 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800e9c0 tx=0x7f5e6800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:17.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.719+0000 7f5e73e00640 1 -- 172.21.15.134:0/1553481838 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6800ccc0 con 0x7f5e7c108d20 2024-07-26T11:17:17.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.719+0000 7f5e80a00640 1 -- 172.21.15.134:0/1553481838 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e7c19ff40 con 0x7f5e7c108d20 2024-07-26T11:17:17.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.719+0000 7f5e73e00640 1 -- 172.21.15.134:0/1553481838 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e680054d0 con 0x7f5e7c108d20 2024-07-26T11:17:17.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.719+0000 7f5e73e00640 1 -- 172.21.15.134:0/1553481838 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e68010430 con 0x7f5e7c108d20 2024-07-26T11:17:17.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.719+0000 7f5e80a00640 1 -- 172.21.15.134:0/1553481838 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e7c1a66a0 con 0x7f5e7c108d20 2024-07-26T11:17:17.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.722+0000 7f5e88e33740 1 Processor -- start 2024-07-26T11:17:17.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.723+0000 7f5e88e33740 1 -- start start 2024-07-26T11:17:17.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.723+0000 7f5e88e33740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28c7350 0x5631a2c6bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.723+0000 7f5e88e33740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a289e490 0x5631a2c6c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.723+0000 7f5e88e33740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a28a6670 0x5631a2c6c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.723+0000 7f5e88e33740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5631a2be2be0 con 0x5631a28a6670 2024-07-26T11:17:17.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.723+0000 7f5e88e33740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5631a2bd20e0 con 0x5631a289e490 2024-07-26T11:17:17.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.723+0000 7f5e88e33740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5631a2bd6120 con 0x5631a28c7350 2024-07-26T11:17:17.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.723+0000 7f5e73e00640 1 -- 172.21.15.134:0/1553481838 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f5e68005020 con 0x7f5e7c108d20 2024-07-26T11:17:17.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.723+0000 7f5e7be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28c7350 0x5631a2c6bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.723+0000 7f5e7be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28c7350 0x5631a2c6bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51462/0 (socket says 172.21.15.134:51462) 2024-07-26T11:17:17.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.724+0000 7f5e7be00640 1 -- 172.21.15.134:0/3742310664 learned_addr learned my addr 172.21.15.134:0/3742310664 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:17.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.724+0000 7f5e7aa00640 1 --2- 172.21.15.134:0/3742310664 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a28a6670 0x5631a2c6c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.724+0000 7f5e7b400640 1 --2- 172.21.15.134:0/3742310664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a289e490 0x5631a2c6c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.724+0000 7f5e7be00640 1 -- 172.21.15.134:0/3742310664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a289e490 msgr2=0x5631a2c6c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:17.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.724+0000 7f5e7be00640 1 --2- 172.21.15.134:0/3742310664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a289e490 0x5631a2c6c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.724+0000 7f5e7be00640 1 -- 172.21.15.134:0/3742310664 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a28a6670 msgr2=0x5631a2c6c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:17.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.724+0000 7f5e7be00640 1 --2- 172.21.15.134:0/3742310664 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a28a6670 0x5631a2c6c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.724+0000 7f5e7be00640 1 -- 172.21.15.134:0/3742310664 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631a2932930 con 0x5631a28c7350 2024-07-26T11:17:17.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.724+0000 7f5e7b400640 1 --2- 172.21.15.134:0/3742310664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a289e490 0x5631a2c6c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:17:17.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.724+0000 7f5e7aa00640 1 --2- 172.21.15.134:0/3742310664 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a28a6670 0x5631a2c6c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:17.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.725+0000 7f5e7be00640 1 --2- 172.21.15.134:0/3742310664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28c7350 0x5631a2c6bf50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009ca0 tx=0x7f5e74009430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e873d9bf79edb06e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:17.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.725+0000 7f5e67e00640 1 -- 172.21.15.134:0/3742310664 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74031cf0 con 0x5631a28c7350 2024-07-26T11:17:17.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.725+0000 7f5e67e00640 1 -- 172.21.15.134:0/3742310664 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e74036030 con 0x5631a28c7350 2024-07-26T11:17:17.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.725+0000 7f5e67e00640 1 -- 172.21.15.134:0/3742310664 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e740042a0 con 0x5631a28c7350 2024-07-26T11:17:17.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.725+0000 7f5e73e00640 1 --2- 172.21.15.134:0/1553481838 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5e600b3dc0 0x7f5e600b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.726+0000 7f5e73e00640 1 -- 172.21.15.134:0/1553481838 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5e680e4680 con 0x7f5e7c108d20 2024-07-26T11:17:17.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.726+0000 7f5e7b400640 1 --2- 172.21.15.134:0/1553481838 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5e600b3dc0 0x7f5e600b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.726+0000 7f5e88e33740 1 -- 172.21.15.134:0/3742310664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28c7350 msgr2=0x5631a2c6bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:17.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.726+0000 7f5e88e33740 1 --2- 172.21.15.134:0/3742310664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28c7350 0x5631a2c6bf50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5e74009ca0 tx=0x7f5e74009430 comp rx=0 tx=0).stop 2024-07-26T11:17:17.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.726+0000 7f5e88e33740 1 -- 172.21.15.134:0/3742310664 shutdown_connections 2024-07-26T11:17:17.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.726+0000 7f5e88e33740 1 --2- 172.21.15.134:0/3742310664 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a28a6670 0x5631a2c6c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.726+0000 7f5e88e33740 1 --2- 172.21.15.134:0/3742310664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a289e490 0x5631a2c6c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.726+0000 7f5e88e33740 1 --2- 172.21.15.134:0/3742310664 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28c7350 0x5631a2c6bf50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.726+0000 7f5e88e33740 1 -- 172.21.15.134:0/3742310664 >> 172.21.15.134:0/3742310664 conn(0x5631a28bb740 msgr2=0x5631a2c69e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:17.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.727+0000 7f5e88e33740 1 -- 172.21.15.134:0/3742310664 shutdown_connections 2024-07-26T11:17:17.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.727+0000 7f5e88e33740 1 -- 172.21.15.134:0/3742310664 wait complete. 2024-07-26T11:17:17.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.727+0000 7f5e7b400640 1 --2- 172.21.15.134:0/1553481838 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5e600b3dc0 0x7f5e600b62d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c0095f0 tx=0x7f5e6c005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:17.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.727+0000 7f5e88e33740 1 -- start start 2024-07-26T11:17:17.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.728+0000 7f5e88e33740 1 Processor -- start 2024-07-26T11:17:17.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.728+0000 7f5e88e33740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a289e490 0x5631a2c7b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.728+0000 7f5e88e33740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28a6670 0x5631a2c7bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.728+0000 7f5e88e33740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28c7350 0x5631a2c7c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.728+0000 7f5e88e33740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5631a2a35890 con 0x5631a289e490 2024-07-26T11:17:17.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.728+0000 7f5e88e33740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5631a2b6e320 con 0x5631a28c7350 2024-07-26T11:17:17.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.728+0000 7f5e88e33740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5631a2c7e8b0 con 0x5631a28a6670 2024-07-26T11:17:17.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.728+0000 7f5e88e33740 10 client.0 _ll_register_callbacks cb 0x5631a2b9a000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:17.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.729+0000 7f5e7aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28a6670 0x5631a2c7bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.729+0000 7f5e7b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a289e490 0x5631a2c7b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.729+0000 7f5e7aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28a6670 0x5631a2c7bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51472/0 (socket says 172.21.15.134:51472) 2024-07-26T11:17:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.729+0000 7f5e7aa00640 1 -- 172.21.15.134:0/3004499121 learned_addr learned my addr 172.21.15.134:0/3004499121 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.729+0000 7f5e7be00640 1 --2- 172.21.15.134:0/3004499121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28c7350 0x5631a2c7c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.729+0000 7f5e7aa00640 1 -- 172.21.15.134:0/3004499121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28c7350 msgr2=0x5631a2c7c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.729+0000 7f5e7aa00640 1 --2- 172.21.15.134:0/3004499121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28c7350 0x5631a2c7c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.729+0000 7f5e7aa00640 1 -- 172.21.15.134:0/3004499121 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a289e490 msgr2=0x5631a2c7b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.729+0000 7f5e7aa00640 1 --2- 172.21.15.134:0/3004499121 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a289e490 0x5631a2c7b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.729+0000 7f5e7aa00640 1 -- 172.21.15.134:0/3004499121 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631a2c87e30 con 0x5631a28a6670 2024-07-26T11:17:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.729+0000 7f5e7b400640 1 --2- 172.21.15.134:0/3004499121 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a289e490 0x5631a2c7b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.730+0000 7f5e7aa00640 1 --2- 172.21.15.134:0/3004499121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28a6670 0x5631a2c7bf10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5e680ea2d0 tx=0x7f5e680aac80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=132b01a90c484b2a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.730+0000 7f5e64c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:17:17.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.730+0000 7f5e64c00640 1 -- 172.21.15.134:0/3004499121 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6800ce60 con 0x5631a28a6670 2024-07-26T11:17:17.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.730+0000 7f5e64c00640 1 -- 172.21.15.134:0/3004499121 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5e680e4230 con 0x5631a28a6670 2024-07-26T11:17:17.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.730+0000 7f5e88e33740 1 -- 172.21.15.134:0/3004499121 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e74036030 con 0x5631a28a6670 2024-07-26T11:17:17.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.730+0000 7f5e64c00640 1 -- 172.21.15.134:0/3004499121 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5e680e1e80 con 0x5631a28a6670 2024-07-26T11:17:17.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.730+0000 7f5e88e33740 10 client.25759 fetch_fsmap learned FSMap version 30 2024-07-26T11:17:17.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.730+0000 7f5e88e33740 1 -- 172.21.15.134:0/3004499121 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5631a2c8a7c0 con 0x5631a28a6670 2024-07-26T11:17:17.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.730+0000 7f5e64c00640 1 -- 172.21.15.134:0/3004499121 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e74036030 con 0x5631a28a6670 2024-07-26T11:17:17.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.731+0000 7f5e64c00640 1 -- 172.21.15.134:0/3004499121 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f5e680b3ad0 con 0x5631a28a6670 2024-07-26T11:17:17.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.731+0000 7f5e88e33740 10 client.25759 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:17:17.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.731+0000 7f5e88e33740 10 client.25759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:17:17.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.731+0000 7f5e88e33740 20 client.25759 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:17.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.731+0000 7f5e88e33740 1 --2- 172.21.15.134:0/3004499121 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5631a2885cd0 0x5631a2c8dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.731+0000 7f5e88e33740 4 client.25759 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:17.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.731+0000 7f5e88e33740 1 -- 172.21.15.134:0/3004499121 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5631a2be4b30 con 0x5631a2885cd0 2024-07-26T11:17:17.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.731+0000 7f5e7b400640 1 --2- 172.21.15.134:0/3004499121 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5631a2885cd0 0x5631a2c8dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.732+0000 7f5e7b400640 1 --2- 172.21.15.134:0/3004499121 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5631a2885cd0 0x5631a2c8dd00 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=747c52beca4967f1 server_cookie=584b0cc76d3d2955 in_seq=0 out_seq=0 2024-07-26T11:17:17.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.732+0000 7f5e64c00640 10 client.25759 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:17.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.733+0000 7f5e64c00640 1 -- 172.21.15.134:0/3004499121 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5631a2be4b30 con 0x5631a2885cd0 2024-07-26T11:17:17.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.733+0000 7f5e64c00640 10 client.25759 handle_command_reply: tid=0 2024-07-26T11:17:17.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.733+0000 7f5e88e33740 1 client.25759 shutdown 2024-07-26T11:17:17.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.733+0000 7f5e88e33740 1 -- 172.21.15.134:0/3004499121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28a6670 msgr2=0x5631a2c7bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:17.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.733+0000 7f5e88e33740 1 --2- 172.21.15.134:0/3004499121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28a6670 0x5631a2c7bf10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5e680ea2d0 tx=0x7f5e680aac80 comp rx=0 tx=0).stop 2024-07-26T11:17:17.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.733+0000 7f5e88e33740 1 -- 172.21.15.134:0/3004499121 shutdown_connections 2024-07-26T11:17:17.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.733+0000 7f5e88e33740 1 --2- 172.21.15.134:0/3004499121 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5631a2885cd0 0x5631a2c8dd00 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.733+0000 7f5e88e33740 1 --2- 172.21.15.134:0/3004499121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28a6670 0x5631a2c7bf10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.733+0000 7f5e88e33740 1 --2- 172.21.15.134:0/3004499121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28c7350 0x5631a2c7c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.733+0000 7f5e64c00640 0 client.25759 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:17.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.733+0000 7f5e88e33740 1 --2- 172.21.15.134:0/3004499121 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a289e490 0x5631a2c7b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.733+0000 7f5e88e33740 1 -- 172.21.15.134:0/3004499121 >> 172.21.15.134:0/3004499121 conn(0x5631a28bb740 msgr2=0x5631a2c6d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:17.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.733+0000 7f5e88e33740 1 -- 172.21.15.134:0/3004499121 shutdown_connections 2024-07-26T11:17:17.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.733+0000 7f5e88e33740 1 -- 172.21.15.134:0/3004499121 wait complete. 2024-07-26T11:17:17.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.733+0000 7f5e88e33740 20 client.25759 trim_cache size 0 max 16384 2024-07-26T11:17:17.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.753+0000 7f5e88e33740 1 Processor -- start 2024-07-26T11:17:17.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.753+0000 7f5e88e33740 1 -- start start 2024-07-26T11:17:17.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.753+0000 7f5e88e33740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a289e490 0x5631a2c7d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.753+0000 7f5e88e33740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28a6670 0x5631a2c76da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.753+0000 7f5e88e33740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28c7350 0x5631a2c7ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.753+0000 7f5e88e33740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c1ae420 con 0x5631a289e490 2024-07-26T11:17:17.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.753+0000 7f5e88e33740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c1aff60 con 0x5631a28a6670 2024-07-26T11:17:17.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.753+0000 7f5e88e33740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c1aeaf0 con 0x5631a28c7350 2024-07-26T11:17:17.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.754+0000 7f5e7aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28a6670 0x5631a2c76da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.754+0000 7f5e7aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28a6670 0x5631a2c76da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46614/0 (socket says 172.21.15.134:46614) 2024-07-26T11:17:17.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.754+0000 7f5e7b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a289e490 0x5631a2c7d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.754+0000 7f5e7aa00640 1 -- 172.21.15.134:0/2819767467 learned_addr learned my addr 172.21.15.134:0/2819767467 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:17.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.754+0000 7f5e7be00640 1 --2- 172.21.15.134:0/2819767467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28c7350 0x5631a2c7ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.754+0000 7f5e7aa00640 1 -- 172.21.15.134:0/2819767467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28c7350 msgr2=0x5631a2c7ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:17.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.754+0000 7f5e7aa00640 1 --2- 172.21.15.134:0/2819767467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28c7350 0x5631a2c7ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.754+0000 7f5e7aa00640 1 -- 172.21.15.134:0/2819767467 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a289e490 msgr2=0x5631a2c7d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:17.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.754+0000 7f5e7aa00640 1 --2- 172.21.15.134:0/2819767467 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a289e490 0x5631a2c7d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.754+0000 7f5e7aa00640 1 -- 172.21.15.134:0/2819767467 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5631a2c6e470 con 0x5631a28a6670 2024-07-26T11:17:17.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.754+0000 7f5e7b400640 1 --2- 172.21.15.134:0/2819767467 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a289e490 0x5631a2c7d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:17:17.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.755+0000 7f5e7aa00640 1 --2- 172.21.15.134:0/2819767467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28a6670 0x5631a2c76da0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009380 tx=0x7f5e68009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b12f128ace1b401e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:17.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.755+0000 7f5e67400640 1 -- 172.21.15.134:0/2819767467 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6800ce60 con 0x5631a28a6670 2024-07-26T11:17:17.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.755+0000 7f5e67400640 1 -- 172.21.15.134:0/2819767467 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e680ca020 con 0x5631a28a6670 2024-07-26T11:17:17.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.755+0000 7f5e67400640 1 -- 172.21.15.134:0/2819767467 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e680aa800 con 0x5631a28a6670 2024-07-26T11:17:17.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.756+0000 7f5e88e33740 1 -- 172.21.15.134:0/2819767467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28a6670 msgr2=0x5631a2c76da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:17.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.756+0000 7f5e88e33740 1 --2- 172.21.15.134:0/2819767467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28a6670 0x5631a2c76da0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5e68009380 tx=0x7f5e68009310 comp rx=0 tx=0).stop 2024-07-26T11:17:17.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.756+0000 7f5e88e33740 1 -- 172.21.15.134:0/2819767467 shutdown_connections 2024-07-26T11:17:17.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.756+0000 7f5e88e33740 1 --2- 172.21.15.134:0/2819767467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a28c7350 0x5631a2c7ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.756+0000 7f5e88e33740 1 --2- 172.21.15.134:0/2819767467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28a6670 0x5631a2c76da0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.756+0000 7f5e88e33740 1 --2- 172.21.15.134:0/2819767467 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a289e490 0x5631a2c7d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.756+0000 7f5e88e33740 1 -- 172.21.15.134:0/2819767467 >> 172.21.15.134:0/2819767467 conn(0x5631a28bb740 msgr2=0x5631a2c76690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:17.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.757+0000 7f5e88e33740 1 -- 172.21.15.134:0/2819767467 shutdown_connections 2024-07-26T11:17:17.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.757+0000 7f5e88e33740 1 -- 172.21.15.134:0/2819767467 wait complete. 2024-07-26T11:17:17.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.757+0000 7f5e88e33740 1 -- start start 2024-07-26T11:17:17.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.757+0000 7f5e88e33740 1 Processor -- start 2024-07-26T11:17:17.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.758+0000 7f5e88e33740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a289e490 0x5631a2c83e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.758+0000 7f5e88e33740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a28a6670 0x5631a2c843d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.758+0000 7f5e88e33740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28c7350 0x5631a2c743d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.758+0000 7f5e88e33740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c1af890 con 0x5631a28a6670 2024-07-26T11:17:17.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.758+0000 7f5e88e33740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c1af1c0 con 0x5631a28c7350 2024-07-26T11:17:17.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.758+0000 7f5e88e33740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7c1ac400 con 0x5631a289e490 2024-07-26T11:17:17.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.758+0000 7f5e88e33740 10 client.0 _ll_register_callbacks cb 0x7f5e680ea220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:17.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.758+0000 7f5e7aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a28a6670 0x5631a2c843d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.759+0000 7f5e7aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a28a6670 0x5631a2c843d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38556/0 (socket says 172.21.15.134:38556) 2024-07-26T11:17:17.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.759+0000 7f5e7aa00640 1 -- 172.21.15.134:0/227524138 learned_addr learned my addr 172.21.15.134:0/227524138 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:17.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.759+0000 7f5e7b400640 1 --2- 172.21.15.134:0/227524138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a289e490 0x5631a2c83e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.759+0000 7f5e7be00640 1 --2- 172.21.15.134:0/227524138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28c7350 0x5631a2c743d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.759+0000 7f5e7aa00640 1 -- 172.21.15.134:0/227524138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a289e490 msgr2=0x5631a2c83e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:17.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.759+0000 7f5e7aa00640 1 --2- 172.21.15.134:0/227524138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a289e490 0x5631a2c83e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.759+0000 7f5e7aa00640 1 -- 172.21.15.134:0/227524138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28c7350 msgr2=0x5631a2c743d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:17.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.759+0000 7f5e7aa00640 1 --2- 172.21.15.134:0/227524138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28c7350 0x5631a2c743d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.759+0000 7f5e7aa00640 1 -- 172.21.15.134:0/227524138 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5631a2c74b40 con 0x5631a28a6670 2024-07-26T11:17:17.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.759+0000 7f5e7b400640 1 --2- 172.21.15.134:0/227524138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a289e490 0x5631a2c83e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:17.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.759+0000 7f5e7be00640 1 --2- 172.21.15.134:0/227524138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28c7350 0x5631a2c743d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:17.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.759+0000 7f5e7aa00640 1 --2- 172.21.15.134:0/227524138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a28a6670 0x5631a2c843d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5e680aabb0 tx=0x7f5e68005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cba0ed3c35f86df server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:17.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.759+0000 7f5e64c00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:17:17.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.760+0000 7f5e64c00640 1 -- 172.21.15.134:0/227524138 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e680e4480 con 0x5631a28a6670 2024-07-26T11:17:17.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.760+0000 7f5e64c00640 1 -- 172.21.15.134:0/227524138 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5e680a74a0 con 0x5631a28a6670 2024-07-26T11:17:17.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.760+0000 7f5e88e33740 1 -- 172.21.15.134:0/227524138 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e680ca020 con 0x5631a28a6670 2024-07-26T11:17:17.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.760+0000 7f5e64c00640 1 -- 172.21.15.134:0/227524138 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5e680aa800 con 0x5631a28a6670 2024-07-26T11:17:17.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.760+0000 7f5e88e33740 10 client.15582 fetch_fsmap learned FSMap version 30 2024-07-26T11:17:17.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.760+0000 7f5e88e33740 1 -- 172.21.15.134:0/227524138 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5631a2c80900 con 0x5631a28a6670 2024-07-26T11:17:17.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.760+0000 7f5e64c00640 1 -- 172.21.15.134:0/227524138 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5e680ca020 con 0x5631a28a6670 2024-07-26T11:17:17.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.761+0000 7f5e64c00640 1 -- 172.21.15.134:0/227524138 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f5e680a6820 con 0x5631a28a6670 2024-07-26T11:17:17.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.761+0000 7f5e88e33740 10 client.15582 fetch_fsmap finished waiting for FSMap version 30 2024-07-26T11:17:17.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.761+0000 7f5e88e33740 10 client.15582 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:17:17.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.761+0000 7f5e88e33740 20 client.15582 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:17.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.761+0000 7f5e88e33740 1 --2- 172.21.15.134:0/227524138 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5631a285b080 0x5631a2cfce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:17.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.761+0000 7f5e88e33740 4 client.15582 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T11:17:17.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.761+0000 7f5e88e33740 1 -- 172.21.15.134:0/227524138 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5631a2be05d0 con 0x5631a285b080 2024-07-26T11:17:17.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.761+0000 7f5e7b400640 1 --2- 172.21.15.134:0/227524138 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5631a285b080 0x5631a2cfce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:17.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.762+0000 7f5e64c00640 10 client.15582 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:17.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.762+0000 7f5e7b400640 1 --2- 172.21.15.134:0/227524138 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5631a285b080 0x5631a2cfce40 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13a3dd33878fa12f server_cookie=a3dc32e90bb6a79f in_seq=0 out_seq=0 2024-07-26T11:17:17.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.763+0000 7f5e64c00640 1 -- 172.21.15.134:0/227524138 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5631a2be05d0 con 0x5631a285b080 2024-07-26T11:17:17.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.763+0000 7f5e64c00640 10 client.15582 handle_command_reply: tid=0 2024-07-26T11:17:17.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.763+0000 7f5e88e33740 1 client.15582 shutdown 2024-07-26T11:17:17.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.764+0000 7f5e88e33740 1 -- 172.21.15.134:0/227524138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a28a6670 msgr2=0x5631a2c843d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:17.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.764+0000 7f5e88e33740 1 --2- 172.21.15.134:0/227524138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a28a6670 0x5631a2c843d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5e680aabb0 tx=0x7f5e68005760 comp rx=0 tx=0).stop 2024-07-26T11:17:17.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.764+0000 7f5e88e33740 1 -- 172.21.15.134:0/227524138 shutdown_connections 2024-07-26T11:17:17.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.764+0000 7f5e88e33740 1 --2- 172.21.15.134:0/227524138 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5631a285b080 0x5631a2cfce40 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.764+0000 7f5e88e33740 1 --2- 172.21.15.134:0/227524138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5631a28c7350 0x5631a2c743d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.764+0000 7f5e88e33740 1 --2- 172.21.15.134:0/227524138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5631a28a6670 0x5631a2c843d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.764+0000 7f5e64c00640 0 client.15582 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:17.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.764+0000 7f5e88e33740 1 --2- 172.21.15.134:0/227524138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5631a289e490 0x5631a2c83e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.764+0000 7f5e88e33740 1 -- 172.21.15.134:0/227524138 >> 172.21.15.134:0/227524138 conn(0x5631a28bb740 msgr2=0x5631a2c7c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:17.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.764+0000 7f5e88e33740 1 -- 172.21.15.134:0/227524138 shutdown_connections 2024-07-26T11:17:17.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.764+0000 7f5e88e33740 1 -- 172.21.15.134:0/227524138 wait complete. 2024-07-26T11:17:17.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.765+0000 7f5e88e33740 20 client.15582 trim_cache size 0 max 16384 2024-07-26T11:17:17.769 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:17:17.769 INFO:teuthology.orchestra.run.smithi134.stdout: "return_code": 0, 2024-07-26T11:17:17.769 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_tag": "870660fe-b74a-4d67-ac9b-db111d2ae4ed", 2024-07-26T11:17:17.769 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "asynchronous" 2024-07-26T11:17:17.769 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:17:17.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.766+0000 7f5e80a00640 1 -- 172.21.15.134:0/1553481838 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5e600b3dc0 msgr2=0x7f5e600b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:17.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.766+0000 7f5e80a00640 1 --2- 172.21.15.134:0/1553481838 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5e600b3dc0 0x7f5e600b62d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c0095f0 tx=0x7f5e6c005f90 comp rx=0 tx=0).stop 2024-07-26T11:17:17.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.766+0000 7f5e80a00640 1 -- 172.21.15.134:0/1553481838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e7c108d20 msgr2=0x7f5e7c19a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:17.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.766+0000 7f5e80a00640 1 --2- 172.21.15.134:0/1553481838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e7c108d20 0x7f5e7c19a280 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800e9c0 tx=0x7f5e6800ee80 comp rx=0 tx=0).stop 2024-07-26T11:17:17.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.766+0000 7f5e80a00640 1 -- 172.21.15.134:0/1553481838 shutdown_connections 2024-07-26T11:17:17.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.766+0000 7f5e80a00640 1 --2- 172.21.15.134:0/1553481838 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5e600b3dc0 0x7f5e600b62d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.766+0000 7f5e80a00640 1 --2- 172.21.15.134:0/1553481838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5e7c108d20 0x7f5e7c19a280 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800e9c0 tx=0x7f5e6800ee80 comp rx=0 tx=0).stop 2024-07-26T11:17:17.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.766+0000 7f5e80a00640 1 --2- 172.21.15.134:0/1553481838 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5e7c109670 0x7f5e7c19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.766+0000 7f5e80a00640 1 --2- 172.21.15.134:0/1553481838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e7c1033e0 0x7f5e7c199ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:17.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.766+0000 7f5e80a00640 1 -- 172.21.15.134:0/1553481838 >> 172.21.15.134:0/1553481838 conn(0x7f5e7c0fee70 msgr2=0x7f5e7c108310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:17.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.766+0000 7f5e80a00640 1 -- 172.21.15.134:0/1553481838 shutdown_connections 2024-07-26T11:17:17.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:17.766+0000 7f5e80a00640 1 -- 172.21.15.134:0/1553481838 wait complete. 2024-07-26T11:17:17.969 INFO:tasks.workunit.client.0.smithi134.stdout:nroff -mandoc -Tascii src/iogen.8 > iogen.cat8 2024-07-26T11:17:18.403 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:18 smithi134 ceph-mon[31966]: scrub queued for path: / 2024-07-26T11:17:18.403 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:18 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:17:18.403 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:18 smithi134 ceph-mon[31966]: scrub summary: active paths [/] 2024-07-26T11:17:18.403 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:18 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:17:18.404 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:18 smithi134 ceph-mon[31966]: pgmap v520: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.2 KiB/s wr, 12 op/s 2024-07-26T11:17:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:18 smithi189 ceph-mon[36790]: scrub queued for path: / 2024-07-26T11:17:18.499 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:18 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:17:18.499 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:18 smithi189 ceph-mon[36790]: scrub summary: active paths [/] 2024-07-26T11:17:18.499 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:18 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:17:18.499 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:18 smithi189 ceph-mon[36790]: pgmap v520: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.2 KiB/s wr, 12 op/s 2024-07-26T11:17:18.516 INFO:tasks.workunit.client.0.smithi134.stdout:running iogen 2024-07-26T11:17:18.516 INFO:tasks.workunit.client.0.smithi134.stderr:+ echo 'running iogen' 2024-07-26T11:17:18.516 INFO:tasks.workunit.client.0.smithi134.stderr:+ ./iogen -n 5 -s 2g 2024-07-26T11:17:18.520 INFO:tasks.workunit.client.0.smithi134.stderr:child spawned 99662 2024-07-26T11:17:18.520 INFO:tasks.workunit.client.0.smithi134.stderr:child spawned 99663 2024-07-26T11:17:18.521 INFO:tasks.workunit.client.0.smithi134.stderr:child spawned 99664 2024-07-26T11:17:18.521 INFO:tasks.workunit.client.0.smithi134.stderr:child spawned 99665 2024-07-26T11:17:18.522 INFO:tasks.workunit.client.0.smithi134.stderr:child spawned 99666 2024-07-26T11:17:18.522 INFO:tasks.workunit.client.0.smithi134.stderr:+ echo 'sleep for 10 min' 2024-07-26T11:17:18.522 INFO:tasks.workunit.client.0.smithi134.stdout:sleep for 10 min 2024-07-26T11:17:18.523 INFO:tasks.workunit.client.0.smithi134.stderr:+ sleep 600 2024-07-26T11:17:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:18 smithi193 ceph-mon[37473]: scrub queued for path: / 2024-07-26T11:17:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:18 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:17:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:18 smithi193 ceph-mon[37473]: scrub summary: active paths [/] 2024-07-26T11:17:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:18 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:17:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:18 smithi193 ceph-mon[37473]: pgmap v520: 129 pgs: 129 active+clean; 1.4 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 3.2 KiB/s wr, 12 op/s 2024-07-26T11:17:18.958 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:17:19.676 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:17:20.440 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:20 smithi134 ceph-mon[31966]: scrub summary: idle 2024-07-26T11:17:20.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:20 smithi189 ceph-mon[36790]: scrub summary: idle 2024-07-26T11:17:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:20 smithi193 ceph-mon[37473]: scrub summary: idle 2024-07-26T11:17:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:21 smithi193 ceph-mon[37473]: pgmap v521: 129 pgs: 129 active+clean; 1.5 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-07-26T11:17:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:21 smithi134 ceph-mon[31966]: pgmap v521: 129 pgs: 129 active+clean; 1.5 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-07-26T11:17:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:21 smithi189 ceph-mon[36790]: pgmap v521: 129 pgs: 129 active+clean; 1.5 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 4.3 KiB/s wr, 11 op/s 2024-07-26T11:17:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:23 smithi193 ceph-mon[37473]: pgmap v522: 129 pgs: 129 active+clean; 101 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 8.3 MiB/s wr, 13 op/s 2024-07-26T11:17:23.675 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:23 smithi134 ceph-mon[31966]: pgmap v522: 129 pgs: 129 active+clean; 101 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 8.3 MiB/s wr, 13 op/s 2024-07-26T11:17:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:23 smithi189 ceph-mon[36790]: pgmap v522: 129 pgs: 129 active+clean; 101 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 8.3 MiB/s wr, 13 op/s 2024-07-26T11:17:25.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:25 smithi134 ceph-mon[31966]: pgmap v523: 129 pgs: 129 active+clean; 490 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 41 MiB/s wr, 30 op/s 2024-07-26T11:17:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:25 smithi193 ceph-mon[37473]: pgmap v523: 129 pgs: 129 active+clean; 490 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 41 MiB/s wr, 30 op/s 2024-07-26T11:17:25.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:25 smithi189 ceph-mon[36790]: pgmap v523: 129 pgs: 129 active+clean; 490 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 41 MiB/s wr, 30 op/s 2024-07-26T11:17:25.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.824+0000 7f5c18a00640 1 -- 172.21.15.134:0/1615872837 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c14109830 msgr2=0x7f5c14110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:25.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.824+0000 7f5c18a00640 1 --2- 172.21.15.134:0/1615872837 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c14109830 0x7f5c14110060 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009be0 tx=0x7f5c0c02a6c0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.824+0000 7f5c18a00640 1 -- 172.21.15.134:0/1615872837 shutdown_connections 2024-07-26T11:17:25.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.824+0000 7f5c18a00640 1 --2- 172.21.15.134:0/1615872837 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c14109830 0x7f5c14110060 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.825+0000 7f5c18a00640 1 --2- 172.21.15.134:0/1615872837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c14108ee0 0x7f5c141092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.825+0000 7f5c18a00640 1 --2- 172.21.15.134:0/1615872837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c141035a0 0x7f5c141039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.825+0000 7f5c18a00640 1 -- 172.21.15.134:0/1615872837 >> 172.21.15.134:0/1615872837 conn(0x7f5c140ff010 msgr2=0x7f5c14101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:25.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.825+0000 7f5c18a00640 1 -- 172.21.15.134:0/1615872837 shutdown_connections 2024-07-26T11:17:25.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.825+0000 7f5c18a00640 1 -- 172.21.15.134:0/1615872837 wait complete. 2024-07-26T11:17:25.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.826+0000 7f5c18a00640 1 Processor -- start 2024-07-26T11:17:25.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.827+0000 7f5c18a00640 1 -- start start 2024-07-26T11:17:25.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.827+0000 7f5c18a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c141035a0 0x7f5c14197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.827+0000 7f5c18a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c14108ee0 0x7f5c14198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.827+0000 7f5c18a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c14109830 0x7f5c1419d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.827+0000 7f5c18a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5c14111300 con 0x7f5c14108ee0 2024-07-26T11:17:25.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.827+0000 7f5c18a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5c14111180 con 0x7f5c141035a0 2024-07-26T11:17:25.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.827+0000 7f5c18a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5c14111480 con 0x7f5c14109830 2024-07-26T11:17:25.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.828+0000 7f5c11400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c14108ee0 0x7f5c14198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.828+0000 7f5c12800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c14109830 0x7f5c1419d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.828+0000 7f5c12800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c14109830 0x7f5c1419d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53788/0 (socket says 172.21.15.134:53788) 2024-07-26T11:17:25.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.828+0000 7f5c12800640 1 -- 172.21.15.134:0/153618500 learned_addr learned my addr 172.21.15.134:0/153618500 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:25.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.829+0000 7f5c11e00640 1 --2- 172.21.15.134:0/153618500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c141035a0 0x7f5c14197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.829+0000 7f5c12800640 1 -- 172.21.15.134:0/153618500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c141035a0 msgr2=0x7f5c14197cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:25.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.829+0000 7f5c12800640 1 --2- 172.21.15.134:0/153618500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c141035a0 0x7f5c14197cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.829+0000 7f5c12800640 1 -- 172.21.15.134:0/153618500 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c14108ee0 msgr2=0x7f5c14198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:25.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.829+0000 7f5c12800640 1 --2- 172.21.15.134:0/153618500 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c14108ee0 0x7f5c14198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.829+0000 7f5c12800640 1 -- 172.21.15.134:0/153618500 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c1419dc20 con 0x7f5c14109830 2024-07-26T11:17:25.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.829+0000 7f5c12800640 1 --2- 172.21.15.134:0/153618500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c14109830 0x7f5c1419d4f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c003c10 tx=0x7f5c0c003c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:25.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.830+0000 7f5c0aa00640 1 -- 172.21.15.134:0/153618500 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c040020 con 0x7f5c14109830 2024-07-26T11:17:25.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.830+0000 7f5c0aa00640 1 -- 172.21.15.134:0/153618500 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c0c003e20 con 0x7f5c14109830 2024-07-26T11:17:25.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.830+0000 7f5c0aa00640 1 -- 172.21.15.134:0/153618500 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c036040 con 0x7f5c14109830 2024-07-26T11:17:25.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.831+0000 7f5c18a00640 1 -- 172.21.15.134:0/153618500 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c1419deb0 con 0x7f5c14109830 2024-07-26T11:17:25.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.831+0000 7f5c18a00640 1 -- 172.21.15.134:0/153618500 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c141a4670 con 0x7f5c14109830 2024-07-26T11:17:25.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.834+0000 7f5c1f64e740 1 Processor -- start 2024-07-26T11:17:25.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.836+0000 7f5c0aa00640 1 -- 172.21.15.134:0/153618500 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f5c0c043020 con 0x7f5c14109830 2024-07-26T11:17:25.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.837+0000 7f5c1f64e740 1 -- start start 2024-07-26T11:17:25.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.837+0000 7f5c1f64e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56412619f070 0x564126543df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.837+0000 7f5c1f64e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641261761b0 0x564126544330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.837+0000 7f5c1f64e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412617e390 0x564126544870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.837+0000 7f5c1f64e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5641264bab90 con 0x56412617e390 2024-07-26T11:17:25.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.837+0000 7f5c1f64e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5641264aa090 con 0x56412619f070 2024-07-26T11:17:25.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.837+0000 7f5c1f64e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5641264ae0d0 con 0x5641261761b0 2024-07-26T11:17:25.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.838+0000 7f5c11400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56412619f070 0x564126543df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.838+0000 7f5c11e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641261761b0 0x564126544330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.838+0000 7f5c11400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56412619f070 0x564126543df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:52958/0 (socket says 172.21.15.134:52958) 2024-07-26T11:17:25.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.838+0000 7f5c11400640 1 -- 172.21.15.134:0/957540447 learned_addr learned my addr 172.21.15.134:0/957540447 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:25.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.838+0000 7f5c11400640 1 --2- 172.21.15.134:0/957540447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412617e390 0x564126544870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.838+0000 7f5c11e00640 1 -- 172.21.15.134:0/957540447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56412619f070 msgr2=0x564126543df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:25.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.839+0000 7f5c11e00640 1 --2- 172.21.15.134:0/957540447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56412619f070 0x564126543df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.839+0000 7f5c0aa00640 1 --2- 172.21.15.134:0/153618500 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5bf80b3df0 0x7f5bf80b6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.839+0000 7f5c11e00640 1 -- 172.21.15.134:0/957540447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412617e390 msgr2=0x564126544870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:25.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.839+0000 7f5c0aa00640 1 -- 172.21.15.134:0/153618500 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5c0c104f20 con 0x7f5c14109830 2024-07-26T11:17:25.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.839+0000 7f5c11e00640 1 --2- 172.21.15.134:0/957540447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412617e390 0x564126544870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.839+0000 7f5c11e00640 1 -- 172.21.15.134:0/957540447 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56412620a650 con 0x5641261761b0 2024-07-26T11:17:25.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.839+0000 7f5c11400640 1 --2- 172.21.15.134:0/957540447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412617e390 0x564126544870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:25.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.840+0000 7f5c11400640 1 --2- 172.21.15.134:0/153618500 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5bf80b3df0 0x7f5bf80b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.840+0000 7f5c11e00640 1 --2- 172.21.15.134:0/957540447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641261761b0 0x564126544330 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5c04005fd0 tx=0x7f5c0400d670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=878f45577fe82863 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:25.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.840+0000 7f5bff000640 1 -- 172.21.15.134:0/957540447 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0401c020 con 0x5641261761b0 2024-07-26T11:17:25.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.841+0000 7f5bff000640 1 -- 172.21.15.134:0/957540447 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c04004020 con 0x5641261761b0 2024-07-26T11:17:25.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.841+0000 7f5bff000640 1 -- 172.21.15.134:0/957540447 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c04003bc0 con 0x5641261761b0 2024-07-26T11:17:25.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.841+0000 7f5c11400640 1 --2- 172.21.15.134:0/153618500 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5bf80b3df0 0x7f5bf80b6300 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5c0001dd80 tx=0x7f5c000046c0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:25.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.842+0000 7f5c1f64e740 1 -- 172.21.15.134:0/957540447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641261761b0 msgr2=0x564126544330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:25.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.842+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/957540447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641261761b0 0x564126544330 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5c04005fd0 tx=0x7f5c0400d670 comp rx=0 tx=0).stop 2024-07-26T11:17:25.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.842+0000 7f5c1f64e740 1 -- 172.21.15.134:0/957540447 shutdown_connections 2024-07-26T11:17:25.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.842+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/957540447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641261761b0 0x564126544330 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.842+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/957540447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412617e390 0x564126544870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.842+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/957540447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56412619f070 0x564126543df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.842+0000 7f5c1f64e740 1 -- 172.21.15.134:0/957540447 >> 172.21.15.134:0/957540447 conn(0x564126193460 msgr2=0x564126541ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:25.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.842+0000 7f5c1f64e740 1 -- 172.21.15.134:0/957540447 shutdown_connections 2024-07-26T11:17:25.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.842+0000 7f5c1f64e740 1 -- 172.21.15.134:0/957540447 wait complete. 2024-07-26T11:17:25.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.843+0000 7f5c1f64e740 1 -- start start 2024-07-26T11:17:25.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.844+0000 7f5c1f64e740 1 Processor -- start 2024-07-26T11:17:25.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.845+0000 7f5c1f64e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641261761b0 0x56412654f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.845+0000 7f5c1f64e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56412617e390 0x56412654f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.845+0000 7f5c1f64e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412619f070 0x56412654fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.846+0000 7f5c1f64e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56412630d700 con 0x56412619f070 2024-07-26T11:17:25.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.846+0000 7f5c11400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56412617e390 0x56412654f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.846+0000 7f5c11400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56412617e390 0x56412654f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53806/0 (socket says 172.21.15.134:53806) 2024-07-26T11:17:25.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.846+0000 7f5c11400640 1 -- 172.21.15.134:0/2512047434 learned_addr learned my addr 172.21.15.134:0/2512047434 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:25.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.846+0000 7f5c12800640 1 --2- 172.21.15.134:0/2512047434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412619f070 0x56412654fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.846+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2512047434 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564126446660 con 0x5641261761b0 2024-07-26T11:17:25.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.846+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2512047434 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5641265525d0 con 0x56412617e390 2024-07-26T11:17:25.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.846+0000 7f5c11e00640 1 --2- 172.21.15.134:0/2512047434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641261761b0 0x56412654f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.846+0000 7f5c1f64e740 10 client.0 _ll_register_callbacks cb 0x564126456110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:25.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.847+0000 7f5c11400640 1 -- 172.21.15.134:0/2512047434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641261761b0 msgr2=0x56412654f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:25.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.847+0000 7f5c11400640 1 --2- 172.21.15.134:0/2512047434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641261761b0 0x56412654f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.847+0000 7f5c11400640 1 -- 172.21.15.134:0/2512047434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412619f070 msgr2=0x56412654fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:25.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.847+0000 7f5c11400640 1 --2- 172.21.15.134:0/2512047434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412619f070 0x56412654fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.847+0000 7f5c11400640 1 -- 172.21.15.134:0/2512047434 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56412645d800 con 0x56412617e390 2024-07-26T11:17:25.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.848+0000 7f5c11400640 1 --2- 172.21.15.134:0/2512047434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56412617e390 0x56412654f8c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5c0001ef70 tx=0x7f5c00004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac87953bc09153dc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:25.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.848+0000 7f5bf3400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:17:25.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.848+0000 7f5bf3400640 1 -- 172.21.15.134:0/2512047434 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0002f020 con 0x56412617e390 2024-07-26T11:17:25.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.848+0000 7f5bf3400640 1 -- 172.21.15.134:0/2512047434 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5c00003d70 con 0x56412617e390 2024-07-26T11:17:25.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.848+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2512047434 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c04004020 con 0x56412617e390 2024-07-26T11:17:25.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.849+0000 7f5bf3400640 1 -- 172.21.15.134:0/2512047434 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5c0002b470 con 0x56412617e390 2024-07-26T11:17:25.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.850+0000 7f5c1f64e740 10 client.25777 fetch_fsmap learned FSMap version 31 2024-07-26T11:17:25.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.850+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2512047434 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x56412655e210 con 0x56412617e390 2024-07-26T11:17:25.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.850+0000 7f5bf3400640 1 -- 172.21.15.134:0/2512047434 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c04004020 con 0x56412617e390 2024-07-26T11:17:25.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.850+0000 7f5bf3400640 1 -- 172.21.15.134:0/2512047434 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f5c00029040 con 0x56412617e390 2024-07-26T11:17:25.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.850+0000 7f5c1f64e740 10 client.25777 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:17:25.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.850+0000 7f5c1f64e740 10 client.25777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:17:25.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.851+0000 7f5c1f64e740 20 client.25777 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:25.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.851+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/2512047434 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56412615d9f0 0x564126561870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.851+0000 7f5c1f64e740 4 client.25777 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:25.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.851+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2512047434 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641264bcae0 con 0x56412615d9f0 2024-07-26T11:17:25.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.852+0000 7f5c11e00640 1 --2- 172.21.15.134:0/2512047434 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56412615d9f0 0x564126561870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.853+0000 7f5c11e00640 1 --2- 172.21.15.134:0/2512047434 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56412615d9f0 0x564126561870 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4585e8091e72fb83 server_cookie=9b19af297a114922 in_seq=0 out_seq=0 2024-07-26T11:17:25.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.853+0000 7f5bf3400640 10 client.25777 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:25.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.854+0000 7f5bf3400640 1 -- 172.21.15.134:0/2512047434 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5641264bcae0 con 0x56412615d9f0 2024-07-26T11:17:25.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.854+0000 7f5bf3400640 10 client.25777 handle_command_reply: tid=0 2024-07-26T11:17:25.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.855+0000 7f5c1f64e740 1 client.25777 shutdown 2024-07-26T11:17:25.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.855+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2512047434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56412617e390 msgr2=0x56412654f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:25.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.855+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/2512047434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56412617e390 0x56412654f8c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5c0001ef70 tx=0x7f5c00004730 comp rx=0 tx=0).stop 2024-07-26T11:17:25.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.855+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2512047434 shutdown_connections 2024-07-26T11:17:25.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.855+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/2512047434 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56412615d9f0 0x564126561870 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.856+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/2512047434 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56412617e390 0x56412654f8c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.856+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/2512047434 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412619f070 0x56412654fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.856+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/2512047434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641261761b0 0x56412654f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.856+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2512047434 >> 172.21.15.134:0/2512047434 conn(0x564126193460 msgr2=0x564126546f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:25.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.856+0000 7f5bf3400640 0 client.25777 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:25.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.856+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2512047434 shutdown_connections 2024-07-26T11:17:25.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.856+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2512047434 wait complete. 2024-07-26T11:17:25.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.856+0000 7f5c1f64e740 20 client.25777 trim_cache size 0 max 16384 2024-07-26T11:17:25.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.882+0000 7f5c1f64e740 1 Processor -- start 2024-07-26T11:17:25.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.883+0000 7f5c1f64e740 1 -- start start 2024-07-26T11:17:25.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.883+0000 7f5c1f64e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641261761b0 0x56412654fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.883+0000 7f5c1f64e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56412617e390 0x564126545530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.883+0000 7f5c1f64e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56412619f070 0x56412655b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.883+0000 7f5c1f64e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5c141ae6e0 con 0x5641261761b0 2024-07-26T11:17:25.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.883+0000 7f5c1f64e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5c141b0250 con 0x56412617e390 2024-07-26T11:17:25.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.883+0000 7f5c1f64e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5c141aedb0 con 0x56412619f070 2024-07-26T11:17:25.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.884+0000 7f5c12800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56412619f070 0x56412655b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.884+0000 7f5c12800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56412619f070 0x56412655b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53822/0 (socket says 172.21.15.134:53822) 2024-07-26T11:17:25.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.884+0000 7f5c12800640 1 -- 172.21.15.134:0/2535583864 learned_addr learned my addr 172.21.15.134:0/2535583864 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:25.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.884+0000 7f5c11400640 1 --2- 172.21.15.134:0/2535583864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56412617e390 0x564126545530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.884+0000 7f5c11e00640 1 --2- 172.21.15.134:0/2535583864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641261761b0 0x56412654fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.884+0000 7f5c11400640 1 -- 172.21.15.134:0/2535583864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56412619f070 msgr2=0x56412655b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:25.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.884+0000 7f5c11400640 1 --2- 172.21.15.134:0/2535583864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56412619f070 0x56412655b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.884+0000 7f5c11400640 1 -- 172.21.15.134:0/2535583864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641261761b0 msgr2=0x56412654fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:25.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.884+0000 7f5c11400640 1 --2- 172.21.15.134:0/2535583864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641261761b0 0x56412654fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.884+0000 7f5c11400640 1 -- 172.21.15.134:0/2535583864 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641265425d0 con 0x56412617e390 2024-07-26T11:17:25.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.885+0000 7f5c12800640 1 --2- 172.21.15.134:0/2535583864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56412619f070 0x56412655b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:25.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.885+0000 7f5c11e00640 1 --2- 172.21.15.134:0/2535583864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641261761b0 0x56412654fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:17:25.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.885+0000 7f5c11400640 1 --2- 172.21.15.134:0/2535583864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56412617e390 0x564126545530 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5c0001d420 tx=0x7f5c00011ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58e025ec643522a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:25.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.885+0000 7f5bfe600640 1 -- 172.21.15.134:0/2535583864 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0002f070 con 0x56412617e390 2024-07-26T11:17:25.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.886+0000 7f5bfe600640 1 -- 172.21.15.134:0/2535583864 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c00011ba0 con 0x56412617e390 2024-07-26T11:17:25.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.886+0000 7f5bfe600640 1 -- 172.21.15.134:0/2535583864 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0002ba80 con 0x56412617e390 2024-07-26T11:17:25.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.889+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2535583864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56412617e390 msgr2=0x564126545530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:25.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.889+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/2535583864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56412617e390 0x564126545530 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5c0001d420 tx=0x7f5c00011ab0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.889+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2535583864 shutdown_connections 2024-07-26T11:17:25.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.889+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/2535583864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56412619f070 0x56412655b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.889+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/2535583864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56412617e390 0x564126545530 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5c0001d420 tx=0x7f5c00011ab0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.889+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/2535583864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641261761b0 0x56412654fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.889+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2535583864 >> 172.21.15.134:0/2535583864 conn(0x564126193460 msgr2=0x564126544e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:25.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.891+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2535583864 shutdown_connections 2024-07-26T11:17:25.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.891+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2535583864 wait complete. 2024-07-26T11:17:25.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.892+0000 7f5c1f64e740 1 -- start start 2024-07-26T11:17:25.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.892+0000 7f5c1f64e740 1 Processor -- start 2024-07-26T11:17:25.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.893+0000 7f5c1f64e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641261761b0 0x564126557b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.893+0000 7f5c1f64e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412619f070 0x5641265580e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.893+0000 7f5c1f64e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564126132da0 0x564126553130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.893+0000 7f5c1f64e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5c141afb50 con 0x56412619f070 2024-07-26T11:17:25.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.893+0000 7f5c1f64e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5c141af480 con 0x5641261761b0 2024-07-26T11:17:25.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.893+0000 7f5c1f64e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5c141ac6c0 con 0x564126132da0 2024-07-26T11:17:25.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.893+0000 7f5c1f64e740 10 client.0 _ll_register_callbacks cb 0x7f5c0001ee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:25.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.893+0000 7f5c11e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641261761b0 0x564126557b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.893+0000 7f5c11e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641261761b0 0x564126557b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:52992/0 (socket says 172.21.15.134:52992) 2024-07-26T11:17:25.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.893+0000 7f5c11e00640 1 -- 172.21.15.134:0/2691978068 learned_addr learned my addr 172.21.15.134:0/2691978068 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:25.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.894+0000 7f5c11400640 1 --2- 172.21.15.134:0/2691978068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412619f070 0x5641265580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.894+0000 7f5c11e00640 1 -- 172.21.15.134:0/2691978068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564126132da0 msgr2=0x564126553130 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:17:25.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.894+0000 7f5c11e00640 1 --2- 172.21.15.134:0/2691978068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564126132da0 0x564126553130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.894+0000 7f5c11e00640 1 -- 172.21.15.134:0/2691978068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412619f070 msgr2=0x5641265580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:25.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.894+0000 7f5c11e00640 1 --2- 172.21.15.134:0/2691978068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412619f070 0x5641265580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.894+0000 7f5c11e00640 1 -- 172.21.15.134:0/2691978068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641265538a0 con 0x5641261761b0 2024-07-26T11:17:25.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.894+0000 7f5c11400640 1 --2- 172.21.15.134:0/2691978068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412619f070 0x5641265580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:17:25.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.894+0000 7f5c11e00640 1 --2- 172.21.15.134:0/2691978068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641261761b0 0x564126557b20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5c04008af0 tx=0x7f5c04005e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5ee8aeac80d82cd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:25.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.894+0000 7f5bf3400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:17:25.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.895+0000 7f5bf3400640 1 -- 172.21.15.134:0/2691978068 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c04030060 con 0x5641261761b0 2024-07-26T11:17:25.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.895+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2691978068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c00011ba0 con 0x5641261761b0 2024-07-26T11:17:25.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.895+0000 7f5bf3400640 1 -- 172.21.15.134:0/2691978068 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5c04005040 con 0x5641261761b0 2024-07-26T11:17:25.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.895+0000 7f5bf3400640 1 -- 172.21.15.134:0/2691978068 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5c04037050 con 0x5641261761b0 2024-07-26T11:17:25.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.895+0000 7f5c1f64e740 10 client.25772 fetch_fsmap learned FSMap version 31 2024-07-26T11:17:25.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.895+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2691978068 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5641265c9ac0 con 0x5641261761b0 2024-07-26T11:17:25.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.896+0000 7f5bf3400640 1 -- 172.21.15.134:0/2691978068 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c00011ba0 con 0x5641261761b0 2024-07-26T11:17:25.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.896+0000 7f5bf3400640 1 -- 172.21.15.134:0/2691978068 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f5c04004ab0 con 0x5641261761b0 2024-07-26T11:17:25.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.896+0000 7f5c1f64e740 10 client.25772 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:17:25.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.896+0000 7f5c1f64e740 10 client.25772 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:17:25.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.896+0000 7f5c1f64e740 20 client.25772 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:25.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.896+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/2691978068 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5641265c9f60 0x5641265cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:25.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.896+0000 7f5c1f64e740 4 client.25772 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:17:25.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.896+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2691978068 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5641264b8580 con 0x5641265c9f60 2024-07-26T11:17:25.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.896+0000 7f5c11400640 1 --2- 172.21.15.134:0/2691978068 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5641265c9f60 0x5641265cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:25.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.897+0000 7f5c11400640 1 --2- 172.21.15.134:0/2691978068 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5641265c9f60 0x5641265cc380 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=751350d239f2d4af server_cookie=c8d48cab516f88eb in_seq=0 out_seq=0 2024-07-26T11:17:25.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.897+0000 7f5bf3400640 10 client.25772 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:25.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.898+0000 7f5bf3400640 1 -- 172.21.15.134:0/2691978068 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5641264b8580 con 0x5641265c9f60 2024-07-26T11:17:25.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.898+0000 7f5bf3400640 10 client.25772 handle_command_reply: tid=0 2024-07-26T11:17:25.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.898+0000 7f5c1f64e740 1 client.25772 shutdown 2024-07-26T11:17:25.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.898+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2691978068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641261761b0 msgr2=0x564126557b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:25.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.898+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/2691978068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641261761b0 0x564126557b20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5c04008af0 tx=0x7f5c04005e60 comp rx=0 tx=0).stop 2024-07-26T11:17:25.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.898+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2691978068 shutdown_connections 2024-07-26T11:17:25.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.898+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/2691978068 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5641265c9f60 0x5641265cc380 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.898+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/2691978068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564126132da0 0x564126553130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.902 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.898+0000 7f5bf3400640 0 client.25772 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:25.902 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.898+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/2691978068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56412619f070 0x5641265580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.902 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.898+0000 7f5c1f64e740 1 --2- 172.21.15.134:0/2691978068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641261761b0 0x564126557b20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.902 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.898+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2691978068 >> 172.21.15.134:0/2691978068 conn(0x564126193460 msgr2=0x56412654fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:25.902 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.899+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2691978068 shutdown_connections 2024-07-26T11:17:25.902 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.899+0000 7f5c1f64e740 1 -- 172.21.15.134:0/2691978068 wait complete. 2024-07-26T11:17:25.902 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.899+0000 7f5c1f64e740 20 client.25772 trim_cache size 0 max 16384 2024-07-26T11:17:25.902 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:17:25.902 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "no active scrubs running", 2024-07-26T11:17:25.903 INFO:teuthology.orchestra.run.smithi134.stdout: "scrubs": {} 2024-07-26T11:17:25.903 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:17:25.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.900+0000 7f5c18a00640 1 -- 172.21.15.134:0/153618500 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5bf80b3df0 msgr2=0x7f5bf80b6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:25.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.900+0000 7f5c18a00640 1 --2- 172.21.15.134:0/153618500 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5bf80b3df0 0x7f5bf80b6300 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5c0001dd80 tx=0x7f5c000046c0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.900+0000 7f5c18a00640 1 -- 172.21.15.134:0/153618500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c14109830 msgr2=0x7f5c1419d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:25.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.900+0000 7f5c18a00640 1 --2- 172.21.15.134:0/153618500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c14109830 0x7f5c1419d4f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c003c10 tx=0x7f5c0c003c40 comp rx=0 tx=0).stop 2024-07-26T11:17:25.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.901+0000 7f5c18a00640 1 -- 172.21.15.134:0/153618500 shutdown_connections 2024-07-26T11:17:25.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.901+0000 7f5c18a00640 1 --2- 172.21.15.134:0/153618500 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5bf80b3df0 0x7f5bf80b6300 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.901+0000 7f5c18a00640 1 --2- 172.21.15.134:0/153618500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c14109830 0x7f5c1419d4f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.901+0000 7f5c18a00640 1 --2- 172.21.15.134:0/153618500 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5c14108ee0 0x7f5c14198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.901+0000 7f5c18a00640 1 --2- 172.21.15.134:0/153618500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c141035a0 0x7f5c14197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:25.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.901+0000 7f5c18a00640 1 -- 172.21.15.134:0/153618500 >> 172.21.15.134:0/153618500 conn(0x7f5c140ff010 msgr2=0x7f5c14101430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:25.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.901+0000 7f5c18a00640 1 -- 172.21.15.134:0/153618500 shutdown_connections 2024-07-26T11:17:25.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:25.901+0000 7f5c18a00640 1 -- 172.21.15.134:0/153618500 wait complete. 2024-07-26T11:17:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:26 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:17:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:26 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:17:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:26 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:17:27.178 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.176+0000 7f6561000640 1 -- 172.21.15.134:0/2335031708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f655c110000 msgr2=0x7f655c116830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:27.179 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.177+0000 7f6561000640 1 --2- 172.21.15.134:0/2335031708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f655c110000 0x7f655c116830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6548009900 tx=0x7f654802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.179 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.177+0000 7f6561000640 1 -- 172.21.15.134:0/2335031708 shutdown_connections 2024-07-26T11:17:27.179 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.177+0000 7f6561000640 1 --2- 172.21.15.134:0/2335031708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f655c110000 0x7f655c116830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.179 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.177+0000 7f6561000640 1 --2- 172.21.15.134:0/2335031708 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f655c10f6b0 0x7f655c10fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.180 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.177+0000 7f6561000640 1 --2- 172.21.15.134:0/2335031708 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f655c10a090 0x7f655c10a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.180 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.177+0000 7f6561000640 1 -- 172.21.15.134:0/2335031708 >> 172.21.15.134:0/2335031708 conn(0x7f655c070a80 msgr2=0x7f655c0735b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:27.180 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.178+0000 7f6561000640 1 -- 172.21.15.134:0/2335031708 shutdown_connections 2024-07-26T11:17:27.180 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.178+0000 7f6561000640 1 -- 172.21.15.134:0/2335031708 wait complete. 2024-07-26T11:17:27.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.180+0000 7f6561000640 1 Processor -- start 2024-07-26T11:17:27.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.180+0000 7f6561000640 1 -- start start 2024-07-26T11:17:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.181+0000 7f6561000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f655c10a090 0x7f655c1a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.181+0000 7f6561000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f655c10f6b0 0x7f655c1a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.181+0000 7f6561000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f655c110000 0x7f655c1a5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.182+0000 7f655b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f655c10a090 0x7f655c1a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.182+0000 7f655b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f655c10a090 0x7f655c1a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53842/0 (socket says 172.21.15.134:53842) 2024-07-26T11:17:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.182+0000 7f655aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f655c10f6b0 0x7f655c1a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.183+0000 7f6561000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f655c119cc0 con 0x7f655c10f6b0 2024-07-26T11:17:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.183+0000 7f655be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f655c110000 0x7f655c1a5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.183+0000 7f655be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f655c110000 0x7f655c1a5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:53004/0 (socket says 172.21.15.134:53004) 2024-07-26T11:17:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.183+0000 7f655be00640 1 -- 172.21.15.134:0/1556665622 learned_addr learned my addr 172.21.15.134:0/1556665622 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:27.189 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.184+0000 7f6561000640 1 -- 172.21.15.134:0/1556665622 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f655c119b40 con 0x7f655c110000 2024-07-26T11:17:27.189 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.184+0000 7f6561000640 1 -- 172.21.15.134:0/1556665622 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f655c119e40 con 0x7f655c10a090 2024-07-26T11:17:27.189 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.185+0000 7f655be00640 1 -- 172.21.15.134:0/1556665622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f655c10a090 msgr2=0x7f655c1a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:27.189 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.185+0000 7f655be00640 1 --2- 172.21.15.134:0/1556665622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f655c10a090 0x7f655c1a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.189 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.185+0000 7f655be00640 1 -- 172.21.15.134:0/1556665622 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f655c10f6b0 msgr2=0x7f655c1a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:27.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.185+0000 7f655be00640 1 --2- 172.21.15.134:0/1556665622 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f655c10f6b0 0x7f655c1a0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.185+0000 7f655be00640 1 -- 172.21.15.134:0/1556665622 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655c06a7e0 con 0x7f655c110000 2024-07-26T11:17:27.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.186+0000 7f655b400640 1 --2- 172.21.15.134:0/1556665622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f655c10a090 0x7f655c1a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:17:27.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.187+0000 7f655be00640 1 --2- 172.21.15.134:0/1556665622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f655c110000 0x7f655c1a5ed0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f654802a960 tx=0x7f6548004600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:27.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.188+0000 7f6553e00640 1 -- 172.21.15.134:0/1556665622 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548035030 con 0x7f655c110000 2024-07-26T11:17:27.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.188+0000 7f6561000640 1 -- 172.21.15.134:0/1556665622 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f655c1a64a0 con 0x7f655c110000 2024-07-26T11:17:27.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.188+0000 7f6561000640 1 -- 172.21.15.134:0/1556665622 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f655c1a69e0 con 0x7f655c110000 2024-07-26T11:17:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.188+0000 7f6553e00640 1 -- 172.21.15.134:0/1556665622 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6548045070 con 0x7f655c110000 2024-07-26T11:17:27.195 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.190+0000 7f6553e00640 1 -- 172.21.15.134:0/1556665622 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548035030 con 0x7f655c110000 2024-07-26T11:17:27.203 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.201+0000 7f6553e00640 1 -- 172.21.15.134:0/1556665622 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f6548031c60 con 0x7f655c110000 2024-07-26T11:17:27.216 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.214+0000 7f6569604740 1 Processor -- start 2024-07-26T11:17:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.215+0000 7f6553e00640 1 --2- 172.21.15.134:0/1556665622 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65400b3e70 0x7f65400b6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.217 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.215+0000 7f6553e00640 1 -- 172.21.15.134:0/1556665622 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f654810cc20 con 0x7f655c110000 2024-07-26T11:17:27.218 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.216+0000 7f6569604740 1 -- start start 2024-07-26T11:17:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.217+0000 7f6569604740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 0x55de2f6bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.217+0000 7f6569604740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2f32f0 0x55de2f6c0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.217+0000 7f6569604740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f2fb4d0 0x55de2f6c06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.217+0000 7f6569604740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55de2f634270 con 0x55de2f2f32f0 2024-07-26T11:17:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.217+0000 7f6569604740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55de2f623770 con 0x55de2f2fb4d0 2024-07-26T11:17:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.217+0000 7f6569604740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55de2f6277b0 con 0x55de2f31c1b0 2024-07-26T11:17:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.217+0000 7f655b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 0x55de2f6bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:27.219 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.217+0000 7f655b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 0x55de2f6bfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53854/0 (socket says 172.21.15.134:53854) 2024-07-26T11:17:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.217+0000 7f655aa00640 1 --2- 172.21.15.134:0/1556665622 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65400b3e70 0x7f65400b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.217+0000 7f655b400640 1 -- 172.21.15.134:0/3299764985 learned_addr learned my addr 172.21.15.134:0/3299764985 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.217+0000 7f655be00640 1 --2- 172.21.15.134:0/3299764985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f2fb4d0 0x55de2f6c06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.218+0000 7f655be00640 1 -- 172.21.15.134:0/3299764985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 msgr2=0x55de2f6bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.218+0000 7f655be00640 1 --2- 172.21.15.134:0/3299764985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 0x55de2f6bfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.218+0000 7f655be00640 1 -- 172.21.15.134:0/3299764985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2f32f0 msgr2=0x55de2f6c0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.218+0000 7f655be00640 1 --2- 172.21.15.134:0/3299764985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2f32f0 0x55de2f6c0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.218+0000 7f655be00640 1 -- 172.21.15.134:0/3299764985 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de2f387790 con 0x55de2f2fb4d0 2024-07-26T11:17:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.218+0000 7f655be00640 1 --2- 172.21.15.134:0/3299764985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f2fb4d0 0x55de2f6c06a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f654810a070 tx=0x7f65480d3070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=924ead4e0d6f642a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.219+0000 7f6550c00640 1 -- 172.21.15.134:0/3299764985 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65480f60d0 con 0x55de2f2fb4d0 2024-07-26T11:17:27.221 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.219+0000 7f6550c00640 1 -- 172.21.15.134:0/3299764985 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65480d7b60 con 0x55de2f2fb4d0 2024-07-26T11:17:27.222 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.219+0000 7f6550c00640 1 -- 172.21.15.134:0/3299764985 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65480dc180 con 0x55de2f2fb4d0 2024-07-26T11:17:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.220+0000 7f6569604740 1 -- 172.21.15.134:0/3299764985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f2fb4d0 msgr2=0x55de2f6c06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.220+0000 7f6569604740 1 --2- 172.21.15.134:0/3299764985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f2fb4d0 0x55de2f6c06a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f654810a070 tx=0x7f65480d3070 comp rx=0 tx=0).stop 2024-07-26T11:17:27.223 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.221+0000 7f655aa00640 1 --2- 172.21.15.134:0/1556665622 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65400b3e70 0x7f65400b6380 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x55de2f6be8a0 tx=0x7f65540094e0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:27.226 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.222+0000 7f6569604740 1 -- 172.21.15.134:0/3299764985 shutdown_connections 2024-07-26T11:17:27.226 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.222+0000 7f6569604740 1 --2- 172.21.15.134:0/3299764985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f2fb4d0 0x55de2f6c06a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.226 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.222+0000 7f6569604740 1 --2- 172.21.15.134:0/3299764985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2f32f0 0x55de2f6c0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.226 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.222+0000 7f6569604740 1 --2- 172.21.15.134:0/3299764985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 0x55de2f6bfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.226 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.222+0000 7f6569604740 1 -- 172.21.15.134:0/3299764985 >> 172.21.15.134:0/3299764985 conn(0x55de2f310300 msgr2=0x55de2f6bdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:27.226 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.224+0000 7f6569604740 1 -- 172.21.15.134:0/3299764985 shutdown_connections 2024-07-26T11:17:27.228 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.225+0000 7f6569604740 1 -- 172.21.15.134:0/3299764985 wait complete. 2024-07-26T11:17:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.227+0000 7f6569604740 1 -- start start 2024-07-26T11:17:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.230+0000 7f6569604740 1 Processor -- start 2024-07-26T11:17:27.240 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.237+0000 7f6569604740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f2f32f0 0x55de2f6cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.240 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.237+0000 7f6569604740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2fb4d0 0x55de2f6cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.241 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.237+0000 7f6569604740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 0x55de2f6d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.241 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.237+0000 7f6569604740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55de2f55dab0 con 0x55de2f2fb4d0 2024-07-26T11:17:27.241 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.237+0000 7f6569604740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55de2f5c0d50 con 0x55de2f2f32f0 2024-07-26T11:17:27.241 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.237+0000 7f6569604740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55de2f6ce160 con 0x55de2f31c1b0 2024-07-26T11:17:27.241 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.238+0000 7f6569604740 10 client.0 _ll_register_callbacks cb 0x55de2f333a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:27.242 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.239+0000 7f655b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f2f32f0 0x55de2f6cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:27.242 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.239+0000 7f655be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 0x55de2f6d1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:27.242 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.239+0000 7f655b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f2f32f0 0x55de2f6cb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:53036/0 (socket says 172.21.15.134:53036) 2024-07-26T11:17:27.242 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.239+0000 7f655b400640 1 -- 172.21.15.134:0/3514126568 learned_addr learned my addr 172.21.15.134:0/3514126568 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:27.242 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.240+0000 7f655be00640 1 -- 172.21.15.134:0/3514126568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f2f32f0 msgr2=0x55de2f6cb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:27.242 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.240+0000 7f655be00640 1 --2- 172.21.15.134:0/3514126568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f2f32f0 0x55de2f6cb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.242 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.240+0000 7f655be00640 1 -- 172.21.15.134:0/3514126568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2fb4d0 msgr2=0x55de2f6cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:27.242 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.240+0000 7f655be00640 1 --2- 172.21.15.134:0/3514126568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2fb4d0 0x55de2f6cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.242 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.240+0000 7f655be00640 1 -- 172.21.15.134:0/3514126568 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de2f400620 con 0x55de2f31c1b0 2024-07-26T11:17:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.240+0000 7f655be00640 1 --2- 172.21.15.134:0/3514126568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 0x55de2f6d1390 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f65480096c0 tx=0x7f65480d36e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e700da51b5fa8f25 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.240+0000 7f6545600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:17:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.241+0000 7f6545600640 1 -- 172.21.15.134:0/3514126568 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65480ed060 con 0x55de2f31c1b0 2024-07-26T11:17:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.241+0000 7f6545600640 1 -- 172.21.15.134:0/3514126568 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f65480dc390 con 0x55de2f31c1b0 2024-07-26T11:17:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.241+0000 7f6569604740 1 -- 172.21.15.134:0/3514126568 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65480d7b60 con 0x55de2f31c1b0 2024-07-26T11:17:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.241+0000 7f6545600640 1 -- 172.21.15.134:0/3514126568 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f65480314a0 con 0x55de2f31c1b0 2024-07-26T11:17:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.241+0000 7f6569604740 10 client.25789 fetch_fsmap learned FSMap version 31 2024-07-26T11:17:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.243+0000 7f6569604740 1 -- 172.21.15.134:0/3514126568 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55de2f6da1c0 con 0x55de2f31c1b0 2024-07-26T11:17:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.243+0000 7f6545600640 1 -- 172.21.15.134:0/3514126568 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65480d7b60 con 0x55de2f31c1b0 2024-07-26T11:17:27.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.243+0000 7f6545600640 1 -- 172.21.15.134:0/3514126568 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f65480d2230 con 0x55de2f31c1b0 2024-07-26T11:17:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.243+0000 7f6569604740 10 client.25789 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:17:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.243+0000 7f6569604740 10 client.25789 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:17:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.243+0000 7f6569604740 20 client.25789 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:27.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.243+0000 7f6569604740 1 --2- 172.21.15.134:0/3514126568 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55de2f2dad80 0x55de2f6dd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.247 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.244+0000 7f6569604740 4 client.25789 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:27.247 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.244+0000 7f6569604740 1 -- 172.21.15.134:0/3514126568 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de2f6361c0 con 0x55de2f2dad80 2024-07-26T11:17:27.247 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.244+0000 7f655b400640 1 --2- 172.21.15.134:0/3514126568 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55de2f2dad80 0x55de2f6dd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:27.247 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.245+0000 7f6545600640 10 client.25789 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.245+0000 7f655b400640 1 --2- 172.21.15.134:0/3514126568 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55de2f2dad80 0x55de2f6dd520 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e8f059e6539127f server_cookie=f44119c0eb030178 in_seq=0 out_seq=0 2024-07-26T11:17:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.248+0000 7f6545600640 1 -- 172.21.15.134:0/3514126568 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55de2f6361c0 con 0x55de2f2dad80 2024-07-26T11:17:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.248+0000 7f6545600640 10 client.25789 handle_command_reply: tid=0 2024-07-26T11:17:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.248+0000 7f6569604740 1 client.25789 shutdown 2024-07-26T11:17:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.249+0000 7f6569604740 1 -- 172.21.15.134:0/3514126568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 msgr2=0x55de2f6d1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.249+0000 7f6569604740 1 --2- 172.21.15.134:0/3514126568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 0x55de2f6d1390 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f65480096c0 tx=0x7f65480d36e0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.249+0000 7f6569604740 1 -- 172.21.15.134:0/3514126568 shutdown_connections 2024-07-26T11:17:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.249+0000 7f6569604740 1 --2- 172.21.15.134:0/3514126568 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55de2f2dad80 0x55de2f6dd520 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.249+0000 7f6569604740 1 --2- 172.21.15.134:0/3514126568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 0x55de2f6d1390 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.249+0000 7f6569604740 1 --2- 172.21.15.134:0/3514126568 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2fb4d0 0x55de2f6cbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.249+0000 7f6569604740 1 --2- 172.21.15.134:0/3514126568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f2f32f0 0x55de2f6cb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.249+0000 7f6569604740 1 -- 172.21.15.134:0/3514126568 >> 172.21.15.134:0/3514126568 conn(0x55de2f310300 msgr2=0x55de2f6c0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.249+0000 7f6545600640 0 client.25789 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.250+0000 7f6569604740 1 -- 172.21.15.134:0/3514126568 shutdown_connections 2024-07-26T11:17:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.250+0000 7f6569604740 1 -- 172.21.15.134:0/3514126568 wait complete. 2024-07-26T11:17:27.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.250+0000 7f6569604740 20 client.25789 trim_cache size 0 max 16384 2024-07-26T11:17:27.307 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.305+0000 7f6569604740 1 Processor -- start 2024-07-26T11:17:27.308 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.306+0000 7f6569604740 1 -- start start 2024-07-26T11:17:27.308 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.307+0000 7f6569604740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f2f32f0 0x55de2f6c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.309 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.307+0000 7f6569604740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2fb4d0 0x55de2f6d8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.310 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.308+0000 7f6569604740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f31c1b0 0x55de2f6d1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.310 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.307+0000 7f655b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f2f32f0 0x55de2f6c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:27.310 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.308+0000 7f655b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f2f32f0 0x55de2f6c53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53872/0 (socket says 172.21.15.134:53872) 2024-07-26T11:17:27.310 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.308+0000 7f655b400640 1 -- 172.21.15.134:0/673636648 learned_addr learned my addr 172.21.15.134:0/673636648 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:27.310 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.308+0000 7f6569604740 1 -- 172.21.15.134:0/673636648 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f653c010020 con 0x55de2f2fb4d0 2024-07-26T11:17:27.310 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.308+0000 7f6569604740 1 -- 172.21.15.134:0/673636648 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f653c011b90 con 0x55de2f31c1b0 2024-07-26T11:17:27.310 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.308+0000 7f6569604740 1 -- 172.21.15.134:0/673636648 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f653c0106f0 con 0x55de2f2f32f0 2024-07-26T11:17:27.310 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.308+0000 7f655be00640 1 --2- 172.21.15.134:0/673636648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f31c1b0 0x55de2f6d1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:27.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.308+0000 7f655aa00640 1 --2- 172.21.15.134:0/673636648 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2fb4d0 0x55de2f6d8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:27.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.309+0000 7f655be00640 1 -- 172.21.15.134:0/673636648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f2f32f0 msgr2=0x55de2f6c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:27.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.309+0000 7f655be00640 1 --2- 172.21.15.134:0/673636648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f2f32f0 0x55de2f6c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.309+0000 7f655be00640 1 -- 172.21.15.134:0/673636648 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2fb4d0 msgr2=0x55de2f6d8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:27.312 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.309+0000 7f655be00640 1 --2- 172.21.15.134:0/673636648 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2fb4d0 0x55de2f6d8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.312 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.309+0000 7f655be00640 1 -- 172.21.15.134:0/673636648 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de2f6b7de0 con 0x55de2f31c1b0 2024-07-26T11:17:27.312 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.309+0000 7f655aa00640 1 --2- 172.21.15.134:0/673636648 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2fb4d0 0x55de2f6d8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:17:27.312 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.310+0000 7f655be00640 1 --2- 172.21.15.134:0/673636648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f31c1b0 0x55de2f6d1280 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f65480dc0e0 tx=0x7f6548031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce958927bc6c404f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:27.313 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.310+0000 7f6547e00640 1 -- 172.21.15.134:0/673636648 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65480ed040 con 0x55de2f31c1b0 2024-07-26T11:17:27.313 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.310+0000 7f6547e00640 1 -- 172.21.15.134:0/673636648 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65480d7b60 con 0x55de2f31c1b0 2024-07-26T11:17:27.313 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.310+0000 7f6547e00640 1 -- 172.21.15.134:0/673636648 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6548031640 con 0x55de2f31c1b0 2024-07-26T11:17:27.315 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.313+0000 7f6569604740 1 -- 172.21.15.134:0/673636648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f31c1b0 msgr2=0x55de2f6d1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:27.315 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.313+0000 7f6569604740 1 --2- 172.21.15.134:0/673636648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f31c1b0 0x55de2f6d1280 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f65480dc0e0 tx=0x7f6548031400 comp rx=0 tx=0).stop 2024-07-26T11:17:27.315 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.313+0000 7f6569604740 1 -- 172.21.15.134:0/673636648 shutdown_connections 2024-07-26T11:17:27.315 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.313+0000 7f6569604740 1 --2- 172.21.15.134:0/673636648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f31c1b0 0x55de2f6d1280 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.315 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.313+0000 7f6569604740 1 --2- 172.21.15.134:0/673636648 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2fb4d0 0x55de2f6d8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.315 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.313+0000 7f6569604740 1 --2- 172.21.15.134:0/673636648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f2f32f0 0x55de2f6c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.315 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.313+0000 7f6569604740 1 -- 172.21.15.134:0/673636648 >> 172.21.15.134:0/673636648 conn(0x55de2f310300 msgr2=0x55de2f6c4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:27.315 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.313+0000 7f6569604740 1 -- 172.21.15.134:0/673636648 shutdown_connections 2024-07-26T11:17:27.315 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.313+0000 7f6569604740 1 -- 172.21.15.134:0/673636648 wait complete. 2024-07-26T11:17:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.314+0000 7f6569604740 1 -- start start 2024-07-26T11:17:27.316 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.314+0000 7f6569604740 1 Processor -- start 2024-07-26T11:17:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.315+0000 7f6569604740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f2f32f0 0x55de2f6d2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.315+0000 7f6569604740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2fb4d0 0x55de2f6d3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.315+0000 7f6569604740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 0x55de2f743c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.315+0000 7f6569604740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f653c011490 con 0x55de2f2fb4d0 2024-07-26T11:17:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.315+0000 7f6569604740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f653c010dc0 con 0x55de2f2f32f0 2024-07-26T11:17:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.315+0000 7f6569604740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f653c00e000 con 0x55de2f31c1b0 2024-07-26T11:17:27.317 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.315+0000 7f6569604740 10 client.0 _ll_register_callbacks cb 0x7f65480d2f40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.316+0000 7f655be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 0x55de2f743c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.316+0000 7f655be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 0x55de2f743c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53888/0 (socket says 172.21.15.134:53888) 2024-07-26T11:17:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.316+0000 7f655be00640 1 -- 172.21.15.134:0/1500697799 learned_addr learned my addr 172.21.15.134:0/1500697799 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:27.319 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.316+0000 7f655be00640 1 -- 172.21.15.134:0/1500697799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f2f32f0 msgr2=0x55de2f6d2de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:17:27.319 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.317+0000 7f655aa00640 1 --2- 172.21.15.134:0/1500697799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2fb4d0 0x55de2f6d3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:27.319 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.317+0000 7f655be00640 1 --2- 172.21.15.134:0/1500697799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f2f32f0 0x55de2f6d2de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.320 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.317+0000 7f655be00640 1 -- 172.21.15.134:0/1500697799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2fb4d0 msgr2=0x55de2f6d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:27.320 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.317+0000 7f655be00640 1 --2- 172.21.15.134:0/1500697799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2fb4d0 0x55de2f6d3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.320 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.317+0000 7f655be00640 1 -- 172.21.15.134:0/1500697799 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de2f744140 con 0x55de2f31c1b0 2024-07-26T11:17:27.320 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.318+0000 7f655aa00640 1 --2- 172.21.15.134:0/1500697799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2fb4d0 0x55de2f6d3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:17:27.320 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.318+0000 7f655be00640 1 --2- 172.21.15.134:0/1500697799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 0x55de2f743c00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6548006010 tx=0x7f65480d3070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7de9d5de0e0a7738 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.318+0000 7f6545600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:17:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.318+0000 7f6545600640 1 -- 172.21.15.134:0/1500697799 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65480b1070 con 0x55de2f31c1b0 2024-07-26T11:17:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.318+0000 7f6545600640 1 -- 172.21.15.134:0/1500697799 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f65480d8cf0 con 0x55de2f31c1b0 2024-07-26T11:17:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.318+0000 7f6569604740 1 -- 172.21.15.134:0/1500697799 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f65480d7b60 con 0x55de2f31c1b0 2024-07-26T11:17:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.318+0000 7f6545600640 1 -- 172.21.15.134:0/1500697799 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f65480cf420 con 0x55de2f31c1b0 2024-07-26T11:17:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.318+0000 7f6569604740 10 client.25795 fetch_fsmap learned FSMap version 31 2024-07-26T11:17:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.318+0000 7f6569604740 1 -- 172.21.15.134:0/1500697799 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55de2f744140 con 0x55de2f31c1b0 2024-07-26T11:17:27.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.319+0000 7f6545600640 1 -- 172.21.15.134:0/1500697799 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65480d7b60 con 0x55de2f31c1b0 2024-07-26T11:17:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.320+0000 7f6545600640 1 -- 172.21.15.134:0/1500697799 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f65480ceed0 con 0x55de2f31c1b0 2024-07-26T11:17:27.322 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.320+0000 7f6569604740 10 client.25795 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:17:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.320+0000 7f6569604740 10 client.25795 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:17:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.320+0000 7f6569604740 20 client.25795 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.320+0000 7f6569604740 1 --2- 172.21.15.134:0/1500697799 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55de2f2b01c0 0x55de2f747600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.320+0000 7f6569604740 4 client.25795 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:17:27.323 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.320+0000 7f6569604740 1 -- 172.21.15.134:0/1500697799 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55de2f631c60 con 0x55de2f2b01c0 2024-07-26T11:17:27.324 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.321+0000 7f655b400640 1 --2- 172.21.15.134:0/1500697799 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55de2f2b01c0 0x55de2f747600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:27.324 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.322+0000 7f655b400640 1 --2- 172.21.15.134:0/1500697799 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55de2f2b01c0 0x55de2f747600 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b7d9098534a857e server_cookie=1df509800cb1e107 in_seq=0 out_seq=0 2024-07-26T11:17:27.324 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.322+0000 7f6545600640 10 client.25795 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:27.397 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.395+0000 7f6545600640 1 -- 172.21.15.134:0/1500697799 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55de2f631c60 con 0x55de2f2b01c0 2024-07-26T11:17:27.397 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.395+0000 7f6545600640 10 client.25795 handle_command_reply: tid=0 2024-07-26T11:17:27.397 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.395+0000 7f6569604740 1 client.25795 shutdown 2024-07-26T11:17:27.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.406+0000 7f6569604740 1 -- 172.21.15.134:0/1500697799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 msgr2=0x55de2f743c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:27.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.406+0000 7f6569604740 1 --2- 172.21.15.134:0/1500697799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 0x55de2f743c00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6548006010 tx=0x7f65480d3070 comp rx=0 tx=0).stop 2024-07-26T11:17:27.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.410+0000 7f6569604740 1 -- 172.21.15.134:0/1500697799 shutdown_connections 2024-07-26T11:17:27.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.410+0000 7f6569604740 1 --2- 172.21.15.134:0/1500697799 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55de2f2b01c0 0x55de2f747600 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.410+0000 7f6569604740 1 --2- 172.21.15.134:0/1500697799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de2f31c1b0 0x55de2f743c00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.410+0000 7f6569604740 1 --2- 172.21.15.134:0/1500697799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55de2f2fb4d0 0x55de2f6d3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.410+0000 7f6569604740 1 --2- 172.21.15.134:0/1500697799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de2f2f32f0 0x55de2f6d2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.410+0000 7f6569604740 1 -- 172.21.15.134:0/1500697799 >> 172.21.15.134:0/1500697799 conn(0x55de2f310300 msgr2=0x55de2f6bd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:27.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.410+0000 7f6545600640 0 client.25795 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:27.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.410+0000 7f6569604740 1 -- 172.21.15.134:0/1500697799 shutdown_connections 2024-07-26T11:17:27.413 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.411+0000 7f6569604740 1 -- 172.21.15.134:0/1500697799 wait complete. 2024-07-26T11:17:27.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.414+0000 7f6569604740 20 client.25795 trim_cache size 0 max 16384 2024-07-26T11:17:27.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:17:27.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:17:27.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:17:27.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:17:27.419 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.417+0000 7f6561000640 1 -- 172.21.15.134:0/1556665622 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65400b3e70 msgr2=0x7f65400b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:27.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.417+0000 7f6561000640 1 --2- 172.21.15.134:0/1556665622 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65400b3e70 0x7f65400b6380 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x55de2f6be8a0 tx=0x7f65540094e0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.417+0000 7f6561000640 1 -- 172.21.15.134:0/1556665622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f655c110000 msgr2=0x7f655c1a5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:27.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.417+0000 7f6561000640 1 --2- 172.21.15.134:0/1556665622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f655c110000 0x7f655c1a5ed0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f654802a960 tx=0x7f6548004600 comp rx=0 tx=0).stop 2024-07-26T11:17:27.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.418+0000 7f6561000640 1 -- 172.21.15.134:0/1556665622 shutdown_connections 2024-07-26T11:17:27.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.418+0000 7f6561000640 1 --2- 172.21.15.134:0/1556665622 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f65400b3e70 0x7f65400b6380 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.418+0000 7f6561000640 1 --2- 172.21.15.134:0/1556665622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f655c110000 0x7f655c1a5ed0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.418+0000 7f6561000640 1 --2- 172.21.15.134:0/1556665622 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f655c10f6b0 0x7f655c1a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.418+0000 7f6561000640 1 --2- 172.21.15.134:0/1556665622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f655c10a090 0x7f655c1a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.418+0000 7f6561000640 1 -- 172.21.15.134:0/1556665622 >> 172.21.15.134:0/1556665622 conn(0x7f655c070a80 msgr2=0x7f655c10ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.418+0000 7f6561000640 1 -- 172.21.15.134:0/1556665622 shutdown_connections 2024-07-26T11:17:27.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:27.419+0000 7f6561000640 1 -- 172.21.15.134:0/1556665622 wait complete. 2024-07-26T11:17:27.589 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:27 smithi134 ceph-mon[31966]: pgmap v524: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 94 MiB/s wr, 42 op/s 2024-07-26T11:17:27.589 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:17:27.645 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-26T11:17:27.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:17:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:27 smithi189 ceph-mon[36790]: pgmap v524: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 94 MiB/s wr, 42 op/s 2024-07-26T11:17:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:17:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:27 smithi193 ceph-mon[37473]: pgmap v524: 129 pgs: 129 active+clean; 1.1 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 94 MiB/s wr, 42 op/s 2024-07-26T11:17:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:17:28.776 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:17:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:29 smithi134 ceph-mon[31966]: pgmap v525: 129 pgs: 129 active+clean; 1.7 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 143 MiB/s wr, 60 op/s 2024-07-26T11:17:29.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:17:29.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:17:29.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:17:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:29 smithi189 ceph-mon[36790]: pgmap v525: 129 pgs: 129 active+clean; 1.7 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 143 MiB/s wr, 60 op/s 2024-07-26T11:17:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:17:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:17:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:17:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:29 smithi193 ceph-mon[37473]: pgmap v525: 129 pgs: 129 active+clean; 1.7 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 143 MiB/s wr, 60 op/s 2024-07-26T11:17:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:17:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:17:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:17:31.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:31 smithi134 ceph-mon[31966]: pgmap v526: 129 pgs: 129 active+clean; 2.5 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 215 MiB/s wr, 80 op/s 2024-07-26T11:17:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:31 smithi189 ceph-mon[36790]: pgmap v526: 129 pgs: 129 active+clean; 2.5 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 215 MiB/s wr, 80 op/s 2024-07-26T11:17:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:31 smithi193 ceph-mon[37473]: pgmap v526: 129 pgs: 129 active+clean; 2.5 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 215 MiB/s wr, 80 op/s 2024-07-26T11:17:32.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.736+0000 7f4ff5400640 1 -- 172.21.15.134:0/3621390874 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ff010ba40 msgr2=0x7f4ff0112270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:32.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.736+0000 7f4ff5400640 1 --2- 172.21.15.134:0/3621390874 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ff010ba40 0x7f4ff0112270 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f4fe8009ca0 tx=0x7f4fe802a8b0 comp rx=0 tx=0).stop 2024-07-26T11:17:32.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.737+0000 7f4ff5400640 1 -- 172.21.15.134:0/3621390874 shutdown_connections 2024-07-26T11:17:32.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.737+0000 7f4ff5400640 1 --2- 172.21.15.134:0/3621390874 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ff010ba40 0x7f4ff0112270 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:32.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.737+0000 7f4ff5400640 1 --2- 172.21.15.134:0/3621390874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff010b0f0 0x7f4ff010b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:32.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.737+0000 7f4ff5400640 1 --2- 172.21.15.134:0/3621390874 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ff01057b0 0x7f4ff0105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:32.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.737+0000 7f4ff5400640 1 -- 172.21.15.134:0/3621390874 >> 172.21.15.134:0/3621390874 conn(0x7f4ff0101240 msgr2=0x7f4ff0103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:32.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.737+0000 7f4ff5400640 1 -- 172.21.15.134:0/3621390874 shutdown_connections 2024-07-26T11:17:32.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.737+0000 7f4ff5400640 1 -- 172.21.15.134:0/3621390874 wait complete. 2024-07-26T11:17:32.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.739+0000 7f4ff5400640 1 Processor -- start 2024-07-26T11:17:32.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.739+0000 7f4ff5400640 1 -- start start 2024-07-26T11:17:32.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.739+0000 7f4ff5400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff01057b0 0x7f4ff0111160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:32.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.740+0000 7f4ff5400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ff010b0f0 0x7f4ff0111700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:32.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.740+0000 7f4ff5400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ff010ba40 0x7f4ff01a3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:32.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.740+0000 7f4ff5400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff00782e0 con 0x7f4ff010ba40 2024-07-26T11:17:32.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.740+0000 7f4ff5400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff0078160 con 0x7f4ff01057b0 2024-07-26T11:17:32.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.740+0000 7f4ff5400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff0078460 con 0x7f4ff010b0f0 2024-07-26T11:17:32.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.740+0000 7f4fee800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff01057b0 0x7f4ff0111160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:32.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.740+0000 7f4fee800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff01057b0 0x7f4ff0111160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46878/0 (socket says 172.21.15.134:46878) 2024-07-26T11:17:32.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.740+0000 7f4fee800640 1 -- 172.21.15.134:0/1212143854 learned_addr learned my addr 172.21.15.134:0/1212143854 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:32.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.741+0000 7f4fede00640 1 --2- 172.21.15.134:0/1212143854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ff010b0f0 0x7f4ff0111700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:32.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.741+0000 7f4fef200640 1 --2- 172.21.15.134:0/1212143854 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ff010ba40 0x7f4ff01a3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:32.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.742+0000 7f4fee800640 1 -- 172.21.15.134:0/1212143854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ff010b0f0 msgr2=0x7f4ff0111700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:32.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.742+0000 7f4fee800640 1 --2- 172.21.15.134:0/1212143854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ff010b0f0 0x7f4ff0111700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:32.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.742+0000 7f4fee800640 1 -- 172.21.15.134:0/1212143854 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ff010ba40 msgr2=0x7f4ff01a3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:32.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.742+0000 7f4fee800640 1 --2- 172.21.15.134:0/1212143854 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ff010ba40 0x7f4ff01a3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:32.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.742+0000 7f4fee800640 1 -- 172.21.15.134:0/1212143854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff01a4320 con 0x7f4ff01057b0 2024-07-26T11:17:32.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.743+0000 7f4fee800640 1 --2- 172.21.15.134:0/1212143854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff01057b0 0x7f4ff0111160 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4fe000b4a0 tx=0x7f4fe000b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:32.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.743+0000 7f4fe7400640 1 -- 172.21.15.134:0/1212143854 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0014070 con 0x7f4ff01057b0 2024-07-26T11:17:32.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.743+0000 7f4ff5400640 1 -- 172.21.15.134:0/1212143854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ff01a4610 con 0x7f4ff01057b0 2024-07-26T11:17:32.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.743+0000 7f4fe7400640 1 -- 172.21.15.134:0/1212143854 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fe0010ac0 con 0x7f4ff01057b0 2024-07-26T11:17:32.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.743+0000 7f4fe7400640 1 -- 172.21.15.134:0/1212143854 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe0017c30 con 0x7f4ff01057b0 2024-07-26T11:17:32.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.743+0000 7f4ff5400640 1 -- 172.21.15.134:0/1212143854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ff01acf40 con 0x7f4ff01057b0 2024-07-26T11:17:32.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.746+0000 7f4ff5400640 1 -- 172.21.15.134:0/1212143854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fd00052c0 con 0x7f4ff01057b0 2024-07-26T11:17:32.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.747+0000 7f4fe7400640 1 -- 172.21.15.134:0/1212143854 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f4fe0017070 con 0x7f4ff01057b0 2024-07-26T11:17:32.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.750+0000 7f4fe7400640 1 --2- 172.21.15.134:0/1212143854 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4fd40b3dc0 0x7f4fd40b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:32.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.750+0000 7f4fe7400640 1 -- 172.21.15.134:0/1212143854 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4fe00e3810 con 0x7f4ff01057b0 2024-07-26T11:17:32.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.750+0000 7f4fede00640 1 --2- 172.21.15.134:0/1212143854 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4fd40b3dc0 0x7f4fd40b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:32.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.751+0000 7f4fede00640 1 --2- 172.21.15.134:0/1212143854 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4fd40b3dc0 0x7f4fd40b62d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0109ba0 tx=0x7f4fdc005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:32.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:32.753+0000 7f4fe7400640 1 -- 172.21.15.134:0/1212143854 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f4fe00aa850 con 0x7f4ff01057b0 2024-07-26T11:17:33.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:33.215+0000 7f4ff5400640 1 -- 172.21.15.134:0/1212143854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fd0005880 con 0x7f4ff01057b0 2024-07-26T11:17:33.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:33.216+0000 7f4fe7400640 1 -- 172.21.15.134:0/1212143854 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11991 (secure 0 0 0) 0x7f4fe00aa150 con 0x7f4ff01057b0 2024-07-26T11:17:33.218 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:17:33.219 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":31,"btime":"2024-07-26T11:17:25:226678+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:17:25.065552+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24721,"mds_1":24719,"mds_2":24703,"mds_3":24688,"mds_4":24686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24635":{"gid":24635,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24641":{"gid":24641,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":32,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24719":{"gid":24719,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":28,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24721,"qdb_cluster":[24686,24688,24703,24719,24721]},"id":1}]} 2024-07-26T11:17:33.219 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 31 2024-07-26T11:17:33.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:33.220+0000 7f4ff5400640 1 -- 172.21.15.134:0/1212143854 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4fd40b3dc0 msgr2=0x7f4fd40b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:33.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:33.220+0000 7f4ff5400640 1 --2- 172.21.15.134:0/1212143854 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4fd40b3dc0 0x7f4fd40b62d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0109ba0 tx=0x7f4fdc005f90 comp rx=0 tx=0).stop 2024-07-26T11:17:33.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:33.220+0000 7f4ff5400640 1 -- 172.21.15.134:0/1212143854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff01057b0 msgr2=0x7f4ff0111160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:33.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:33.220+0000 7f4ff5400640 1 --2- 172.21.15.134:0/1212143854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff01057b0 0x7f4ff0111160 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4fe000b4a0 tx=0x7f4fe000b960 comp rx=0 tx=0).stop 2024-07-26T11:17:33.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:33.220+0000 7f4ff5400640 1 -- 172.21.15.134:0/1212143854 shutdown_connections 2024-07-26T11:17:33.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:33.220+0000 7f4ff5400640 1 --2- 172.21.15.134:0/1212143854 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f4fd40b3dc0 0x7f4fd40b62d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:33.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:33.220+0000 7f4ff5400640 1 --2- 172.21.15.134:0/1212143854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ff010b0f0 0x7f4ff0111700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:33.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:33.220+0000 7f4ff5400640 1 --2- 172.21.15.134:0/1212143854 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4ff010ba40 0x7f4ff01a3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:33.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:33.220+0000 7f4ff5400640 1 --2- 172.21.15.134:0/1212143854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ff01057b0 0x7f4ff0111160 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:33.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:33.220+0000 7f4ff5400640 1 -- 172.21.15.134:0/1212143854 >> 172.21.15.134:0/1212143854 conn(0x7f4ff0101240 msgr2=0x7f4ff007aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:33.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:33.221+0000 7f4ff5400640 1 -- 172.21.15.134:0/1212143854 shutdown_connections 2024-07-26T11:17:33.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:33.221+0000 7f4ff5400640 1 -- 172.21.15.134:0/1212143854 wait complete. 2024-07-26T11:17:33.643 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:33 smithi134 ceph-mon[31966]: pgmap v527: 129 pgs: 129 active+clean; 2.9 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 251 MiB/s wr, 89 op/s 2024-07-26T11:17:33.643 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:33 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1212143854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:17:33.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:33 smithi189 ceph-mon[36790]: pgmap v527: 129 pgs: 129 active+clean; 2.9 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 251 MiB/s wr, 89 op/s 2024-07-26T11:17:33.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:33 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1212143854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:17:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:33 smithi193 ceph-mon[37473]: pgmap v527: 129 pgs: 129 active+clean; 2.9 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 251 MiB/s wr, 89 op/s 2024-07-26T11:17:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:33 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1212143854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:17:34.344 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-07-26T11:17:35.132 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:17:35.487 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:35 smithi134 ceph-mon[31966]: pgmap v528: 129 pgs: 129 active+clean; 3.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 292 MiB/s wr, 102 op/s 2024-07-26T11:17:35.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:35 smithi189 ceph-mon[36790]: pgmap v528: 129 pgs: 129 active+clean; 3.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 292 MiB/s wr, 102 op/s 2024-07-26T11:17:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:35 smithi193 ceph-mon[37473]: pgmap v528: 129 pgs: 129 active+clean; 3.5 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 292 MiB/s wr, 102 op/s 2024-07-26T11:17:37.603 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:37 smithi134 ceph-mon[31966]: pgmap v529: 129 pgs: 129 active+clean; 4.1 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 309 MiB/s wr, 97 op/s 2024-07-26T11:17:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:37 smithi189 ceph-mon[36790]: pgmap v529: 129 pgs: 129 active+clean; 4.1 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 309 MiB/s wr, 97 op/s 2024-07-26T11:17:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:37 smithi193 ceph-mon[37473]: pgmap v529: 129 pgs: 129 active+clean; 4.1 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 309 MiB/s wr, 97 op/s 2024-07-26T11:17:38.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.365+0000 7fb9a9c00640 1 -- 172.21.15.134:0/910418293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a4110070 msgr2=0x7fb9a41168c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:38.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.365+0000 7fb9a9c00640 1 --2- 172.21.15.134:0/910418293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a4110070 0x7fb9a41168c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb9940074e0 tx=0x7fb99402b060 comp rx=0 tx=0).stop 2024-07-26T11:17:38.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.365+0000 7fb9a9c00640 1 -- 172.21.15.134:0/910418293 shutdown_connections 2024-07-26T11:17:38.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.365+0000 7fb9a9c00640 1 --2- 172.21.15.134:0/910418293 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9a410f720 0x7fb9a410fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.365+0000 7fb9a9c00640 1 --2- 172.21.15.134:0/910418293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a4110070 0x7fb9a41168c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.365+0000 7fb9a9c00640 1 --2- 172.21.15.134:0/910418293 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9a410a080 0x7fb9a410a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.365+0000 7fb9a9c00640 1 -- 172.21.15.134:0/910418293 >> 172.21.15.134:0/910418293 conn(0x7fb9a406e880 msgr2=0x7fb9a406ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:38.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.366+0000 7fb9a9c00640 1 -- 172.21.15.134:0/910418293 shutdown_connections 2024-07-26T11:17:38.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.366+0000 7fb9a9c00640 1 -- 172.21.15.134:0/910418293 wait complete. 2024-07-26T11:17:38.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.367+0000 7fb9a9c00640 1 Processor -- start 2024-07-26T11:17:38.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.368+0000 7fb9a9c00640 1 -- start start 2024-07-26T11:17:38.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.370+0000 7fb9a9c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a410a080 0x7fb9a411a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.370+0000 7fb9a9c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9a410f720 0x7fb9a4118f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.370+0000 7fb9a9c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9a4110070 0x7fb9a41194e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.370+0000 7fb9a9c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a411bbe0 con 0x7fb9a410f720 2024-07-26T11:17:38.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.370+0000 7fb9a9c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a411ba60 con 0x7fb9a410a080 2024-07-26T11:17:38.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.370+0000 7fb9a9c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a411bd60 con 0x7fb9a4110070 2024-07-26T11:17:38.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.371+0000 7fb9a9200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9a4110070 0x7fb9a41194e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.372+0000 7fb9a3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a410a080 0x7fb9a411a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.372+0000 7fb9a3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a410a080 0x7fb9a411a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46896/0 (socket says 172.21.15.134:46896) 2024-07-26T11:17:38.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.372+0000 7fb9a3e00640 1 -- 172.21.15.134:0/77401022 learned_addr learned my addr 172.21.15.134:0/77401022 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:38.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.372+0000 7fb9a3400640 1 --2- 172.21.15.134:0/77401022 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9a410f720 0x7fb9a4118f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.372+0000 7fb9a9200640 1 -- 172.21.15.134:0/77401022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a410a080 msgr2=0x7fb9a411a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:38.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.373+0000 7fb9a9200640 1 --2- 172.21.15.134:0/77401022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a410a080 0x7fb9a411a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.373+0000 7fb9a9200640 1 -- 172.21.15.134:0/77401022 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9a410f720 msgr2=0x7fb9a4118f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:38.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.373+0000 7fb9a9200640 1 --2- 172.21.15.134:0/77401022 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9a410f720 0x7fb9a4118f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.373+0000 7fb9a9200640 1 -- 172.21.15.134:0/77401022 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a4119d00 con 0x7fb9a4110070 2024-07-26T11:17:38.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.373+0000 7fb9a3400640 1 --2- 172.21.15.134:0/77401022 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9a410f720 0x7fb9a4118f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:17:38.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.373+0000 7fb9a9200640 1 --2- 172.21.15.134:0/77401022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9a4110070 0x7fb9a41194e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb994006010 tx=0x7fb994005400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:38.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.374+0000 7fb9a0c00640 1 -- 172.21.15.134:0/77401022 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99402b9a0 con 0x7fb9a4110070 2024-07-26T11:17:38.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.374+0000 7fb9a0c00640 1 -- 172.21.15.134:0/77401022 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb994004920 con 0x7fb9a4110070 2024-07-26T11:17:38.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.376+0000 7fb9a9c00640 1 -- 172.21.15.134:0/77401022 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a41b4eb0 con 0x7fb9a4110070 2024-07-26T11:17:38.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.376+0000 7fb9a0c00640 1 -- 172.21.15.134:0/77401022 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99400a330 con 0x7fb9a4110070 2024-07-26T11:17:38.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.376+0000 7fb9a9c00640 1 -- 172.21.15.134:0/77401022 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a41b53e0 con 0x7fb9a4110070 2024-07-26T11:17:38.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.381+0000 7fb9b1fc8740 1 Processor -- start 2024-07-26T11:17:38.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.385+0000 7fb9a0c00640 1 -- 172.21.15.134:0/77401022 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fb994043020 con 0x7fb9a4110070 2024-07-26T11:17:38.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.387+0000 7fb9b1fc8740 1 -- start start 2024-07-26T11:17:38.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.389+0000 7fb9b1fc8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x5621537aa030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.389+0000 7fb9b1fc8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533dc1b0 0x5621537aa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.389+0000 7fb9b1fc8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 0x5621537aaab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.389+0000 7fb9b1fc8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562153720bf0 con 0x5621533dc1b0 2024-07-26T11:17:38.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.389+0000 7fb9b1fc8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5621537100f0 con 0x5621533e4390 2024-07-26T11:17:38.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.389+0000 7fb9b1fc8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562153714130 con 0x562153405070 2024-07-26T11:17:38.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.389+0000 7fb9a0c00640 1 --2- 172.21.15.134:0/77401022 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb9840b3e40 0x7fb9840b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.389+0000 7fb9a0c00640 1 -- 172.21.15.134:0/77401022 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb994103550 con 0x7fb9a4110070 2024-07-26T11:17:38.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.395+0000 7fb9a9200640 1 --2- 172.21.15.134:0/77401022 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb9840b3e40 0x7fb9840b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.395+0000 7fb9a3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 0x5621537aaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.395+0000 7fb9a3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 0x5621537aaab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46900/0 (socket says 172.21.15.134:46900) 2024-07-26T11:17:38.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.395+0000 7fb9a3400640 1 -- 172.21.15.134:0/1936635381 learned_addr learned my addr 172.21.15.134:0/1936635381 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:38.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.396+0000 7fb9a3400640 1 --2- 172.21.15.134:0/1936635381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x5621537aa030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.397+0000 7fb9a9200640 1 --2- 172.21.15.134:0/77401022 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb9840b3e40 0x7fb9840b6350 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb9940c5f10 tx=0x7fb99400ac00 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:38.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.397+0000 7fb9a3400640 1 -- 172.21.15.134:0/1936635381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 msgr2=0x5621537aa030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:38.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.397+0000 7fb9a3400640 1 --2- 172.21.15.134:0/1936635381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x5621537aa030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.397+0000 7fb9a3400640 1 -- 172.21.15.134:0/1936635381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533dc1b0 msgr2=0x5621537aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:38.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.397+0000 7fb9a3400640 1 --2- 172.21.15.134:0/1936635381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533dc1b0 0x5621537aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.397+0000 7fb9a3400640 1 -- 172.21.15.134:0/1936635381 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562153470650 con 0x5621533e4390 2024-07-26T11:17:38.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.398+0000 7fb9a3400640 1 --2- 172.21.15.134:0/1936635381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 0x5621537aaab0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb98c009fd0 tx=0x7fb98c009770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d03dac684110b5cd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:38.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.398+0000 7fb999600640 1 -- 172.21.15.134:0/1936635381 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c010710 con 0x5621533e4390 2024-07-26T11:17:38.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.399+0000 7fb999600640 1 -- 172.21.15.134:0/1936635381 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb98c010d60 con 0x5621533e4390 2024-07-26T11:17:38.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.399+0000 7fb999600640 1 -- 172.21.15.134:0/1936635381 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c01dda0 con 0x5621533e4390 2024-07-26T11:17:38.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.401+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/1936635381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 msgr2=0x5621537aaab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:38.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.402+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/1936635381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 0x5621537aaab0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb98c009fd0 tx=0x7fb98c009770 comp rx=0 tx=0).stop 2024-07-26T11:17:38.405 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.402+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/1936635381 shutdown_connections 2024-07-26T11:17:38.405 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.403+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/1936635381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 0x5621537aaab0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.403+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/1936635381 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533dc1b0 0x5621537aa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.403+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/1936635381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x5621537aa030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.403+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/1936635381 >> 172.21.15.134:0/1936635381 conn(0x5621533f9460 msgr2=0x5621537a7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:38.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.404+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/1936635381 shutdown_connections 2024-07-26T11:17:38.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.404+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/1936635381 wait complete. 2024-07-26T11:17:38.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.405+0000 7fb9b1fc8740 1 -- start start 2024-07-26T11:17:38.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.405+0000 7fb9b1fc8740 1 Processor -- start 2024-07-26T11:17:38.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.406+0000 7fb9b1fc8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533dc1b0 0x5621537b5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.406+0000 7fb9b1fc8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 0x5621537b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.406+0000 7fb9b1fc8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x5621537b6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.406+0000 7fb9b1fc8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562153573700 con 0x5621533dc1b0 2024-07-26T11:17:38.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.406+0000 7fb9b1fc8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5621536ac6c0 con 0x5621533e4390 2024-07-26T11:17:38.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.406+0000 7fb9b1fc8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5621537b8810 con 0x562153405070 2024-07-26T11:17:38.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.407+0000 7fb9b1fc8740 10 client.0 _ll_register_callbacks cb 0x5621536bc170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:38.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.407+0000 7fb9a3400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533dc1b0 0x5621537b5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.407+0000 7fb9a3400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533dc1b0 0x5621537b5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:33188/0 (socket says 172.21.15.134:33188) 2024-07-26T11:17:38.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.407+0000 7fb9a3400640 1 -- 172.21.15.134:0/2747062402 learned_addr learned my addr 172.21.15.134:0/2747062402 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:38.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.407+0000 7fb9a3e00640 1 --2- 172.21.15.134:0/2747062402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 0x5621537b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.407+0000 7fb9a3400640 1 --2- 172.21.15.134:0/2747062402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x5621537b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.408+0000 7fb9a3400640 1 -- 172.21.15.134:0/2747062402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 msgr2=0x5621537b6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:38.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.408+0000 7fb9a3400640 1 --2- 172.21.15.134:0/2747062402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x5621537b6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.408+0000 7fb9a3400640 1 -- 172.21.15.134:0/2747062402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 msgr2=0x5621537b5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:38.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.408+0000 7fb9a3400640 1 --2- 172.21.15.134:0/2747062402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 0x5621537b5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.408+0000 7fb9a3400640 1 -- 172.21.15.134:0/2747062402 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621536c3860 con 0x5621533dc1b0 2024-07-26T11:17:38.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.408+0000 7fb9a3400640 1 --2- 172.21.15.134:0/2747062402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533dc1b0 0x5621537b5660 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb98c0057c0 tx=0x7fb98c005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72d9911a884ec3b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:38.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.409+0000 7fb992000640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:17:38.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.409+0000 7fb992000640 1 -- 172.21.15.134:0/2747062402 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c031060 con 0x5621533dc1b0 2024-07-26T11:17:38.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.409+0000 7fb992000640 1 -- 172.21.15.134:0/2747062402 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb98c010a20 con 0x5621533dc1b0 2024-07-26T11:17:38.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.410+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/2747062402 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb98c010d60 con 0x5621533dc1b0 2024-07-26T11:17:38.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.410+0000 7fb992000640 1 -- 172.21.15.134:0/2747062402 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb98c023040 con 0x5621533dc1b0 2024-07-26T11:17:38.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.410+0000 7fb9b1fc8740 10 client.15609 fetch_fsmap learned FSMap version 31 2024-07-26T11:17:38.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.410+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/2747062402 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5621537c44b0 con 0x5621533dc1b0 2024-07-26T11:17:38.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.412+0000 7fb992000640 1 -- 172.21.15.134:0/2747062402 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb98c010d60 con 0x5621533dc1b0 2024-07-26T11:17:38.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.412+0000 7fb992000640 1 -- 172.21.15.134:0/2747062402 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7fb98c029070 con 0x5621533dc1b0 2024-07-26T11:17:38.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.412+0000 7fb9b1fc8740 10 client.15609 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:17:38.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.412+0000 7fb9b1fc8740 10 client.15609 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-07-26T11:17:38.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.412+0000 7fb9b1fc8740 20 client.15609 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:38.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.412+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/2747062402 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5621533c39f0 0x5621537c7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.412+0000 7fb9b1fc8740 4 client.15609 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:38.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.412+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/2747062402 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562153722b40 con 0x5621533c39f0 2024-07-26T11:17:38.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.413+0000 7fb9a3e00640 1 --2- 172.21.15.134:0/2747062402 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5621533c39f0 0x5621537c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.418+0000 7fb992000640 10 client.15609 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:17:38.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.419+0000 7fb9a3e00640 1 --2- 172.21.15.134:0/2747062402 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5621533c39f0 0x5621537c7810 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2cc8851f0e160932 server_cookie=a6fc16b9c4823698 in_seq=0 out_seq=0 2024-07-26T11:17:38.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.420+0000 7fb992000640 1 -- 172.21.15.134:0/2747062402 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562153722b40 con 0x5621533c39f0 2024-07-26T11:17:38.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.420+0000 7fb992000640 10 client.15609 handle_command_reply: tid=0 2024-07-26T11:17:38.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.420+0000 7fb9b1fc8740 1 client.15609 shutdown 2024-07-26T11:17:38.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.421+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/2747062402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533dc1b0 msgr2=0x5621537b5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:38.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.421+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/2747062402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533dc1b0 0x5621537b5660 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb98c0057c0 tx=0x7fb98c005710 comp rx=0 tx=0).stop 2024-07-26T11:17:38.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.421+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/2747062402 shutdown_connections 2024-07-26T11:17:38.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.421+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/2747062402 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5621533c39f0 0x5621537c7810 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.421+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/2747062402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x5621537b6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.421+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/2747062402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 0x5621537b5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.421+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/2747062402 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533dc1b0 0x5621537b5660 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.421+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/2747062402 >> 172.21.15.134:0/2747062402 conn(0x5621533f9460 msgr2=0x5621537ad1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:38.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.421+0000 7fb992000640 0 client.15609 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:17:38.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.422+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/2747062402 shutdown_connections 2024-07-26T11:17:38.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.422+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/2747062402 wait complete. 2024-07-26T11:17:38.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.425+0000 7fb9b1fc8740 20 client.15609 trim_cache size 0 max 16384 2024-07-26T11:17:38.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.458+0000 7fb9b1fc8740 1 Processor -- start 2024-07-26T11:17:38.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.458+0000 7fb9b1fc8740 1 -- start start 2024-07-26T11:17:38.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.459+0000 7fb9b1fc8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 0x5621537b7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.459+0000 7fb9b1fc8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x5621537ab770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.459+0000 7fb9b1fc8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562153398da0 0x5621537cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.459+0000 7fb9b1fc8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41bf590 con 0x562153398da0 2024-07-26T11:17:38.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.459+0000 7fb9b1fc8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41c1100 con 0x5621533e4390 2024-07-26T11:17:38.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.459+0000 7fb9b1fc8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41bfc60 con 0x562153405070 2024-07-26T11:17:38.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.459+0000 7fb9a3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 0x5621537b7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.459+0000 7fb9a3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 0x5621537b7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46912/0 (socket says 172.21.15.134:46912) 2024-07-26T11:17:38.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.459+0000 7fb9a3400640 1 -- 172.21.15.134:0/2188268788 learned_addr learned my addr 172.21.15.134:0/2188268788 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:38.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.459+0000 7fb9a3e00640 1 --2- 172.21.15.134:0/2188268788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x5621537ab770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.460+0000 7fb9a3400640 1 --2- 172.21.15.134:0/2188268788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562153398da0 0x5621537cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.461+0000 7fb9a3e00640 1 -- 172.21.15.134:0/2188268788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 msgr2=0x5621537b7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:38.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.461+0000 7fb9a3e00640 1 --2- 172.21.15.134:0/2188268788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 0x5621537b7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.461+0000 7fb9a3e00640 1 -- 172.21.15.134:0/2188268788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562153398da0 msgr2=0x5621537cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:38.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.461+0000 7fb9a3e00640 1 --2- 172.21.15.134:0/2188268788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562153398da0 0x5621537cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.461+0000 7fb9a3e00640 1 -- 172.21.15.134:0/2188268788 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621537ac320 con 0x562153405070 2024-07-26T11:17:38.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.461+0000 7fb9a3400640 1 --2- 172.21.15.134:0/2188268788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 0x5621537b7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:38.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.462+0000 7fb9a3e00640 1 --2- 172.21.15.134:0/2188268788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x5621537ab770 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x5621537c7db0 tx=0x7fb99c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6938409a338c5e5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:38.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.463+0000 7fb998c00640 1 -- 172.21.15.134:0/2188268788 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c0117d0 con 0x562153405070 2024-07-26T11:17:38.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.463+0000 7fb998c00640 1 -- 172.21.15.134:0/2188268788 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb99c025070 con 0x562153405070 2024-07-26T11:17:38.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.463+0000 7fb998c00640 1 -- 172.21.15.134:0/2188268788 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c020b60 con 0x562153405070 2024-07-26T11:17:38.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.465+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/2188268788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 msgr2=0x5621537ab770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:38.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.465+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/2188268788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x5621537ab770 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x5621537c7db0 tx=0x7fb99c009f90 comp rx=0 tx=0).stop 2024-07-26T11:17:38.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.468+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/2188268788 shutdown_connections 2024-07-26T11:17:38.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.469+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/2188268788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x5621537ab770 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.469+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/2188268788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562153398da0 0x5621537cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.469+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/2188268788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621533e4390 0x5621537b7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.469+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/2188268788 >> 172.21.15.134:0/2188268788 conn(0x5621533f9460 msgr2=0x5621537ab060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:38.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.469+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/2188268788 shutdown_connections 2024-07-26T11:17:38.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.469+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/2188268788 wait complete. 2024-07-26T11:17:38.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.470+0000 7fb9b1fc8740 1 -- start start 2024-07-26T11:17:38.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.471+0000 7fb9b1fc8740 1 Processor -- start 2024-07-26T11:17:38.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.472+0000 7fb9b1fc8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562153398da0 0x562153828050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.472+0000 7fb9b1fc8740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533e4390 0x562153828590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.472+0000 7fb9b1fc8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x562153828ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.472+0000 7fb9b1fc8740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41c0a00 con 0x5621533e4390 2024-07-26T11:17:38.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.472+0000 7fb9b1fc8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41c0330 con 0x562153398da0 2024-07-26T11:17:38.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.472+0000 7fb9b1fc8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a41bd570 con 0x562153405070 2024-07-26T11:17:38.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.473+0000 7fb9b1fc8740 10 client.0 _ll_register_callbacks cb 0x7fb98c01dc50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:38.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.473+0000 7fb9a3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x562153828ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.473+0000 7fb9a3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x562153828ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:32944/0 (socket says 172.21.15.134:32944) 2024-07-26T11:17:38.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.473+0000 7fb9a3400640 1 -- 172.21.15.134:0/1224846136 learned_addr learned my addr 172.21.15.134:0/1224846136 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:38.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.473+0000 7fb9a3400640 1 --2- 172.21.15.134:0/1224846136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562153398da0 0x562153828050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.473+0000 7fb9a3400640 1 -- 172.21.15.134:0/1224846136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562153398da0 msgr2=0x562153828050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:38.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.473+0000 7fb9a3400640 1 --2- 172.21.15.134:0/1224846136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562153398da0 0x562153828050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.473+0000 7fb9a3400640 1 -- 172.21.15.134:0/1224846136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533e4390 msgr2=0x562153828590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:38.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.473+0000 7fb9a3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533e4390 0x562153828590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.474+0000 7fb9a3400640 1 --2- 172.21.15.134:0/1224846136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533e4390 0x562153828590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.474+0000 7fb9a3400640 1 -- 172.21.15.134:0/1224846136 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621537bcb90 con 0x562153405070 2024-07-26T11:17:38.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.475+0000 7fb9a3e00640 1 --2- 172.21.15.134:0/1224846136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533e4390 0x562153828590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:17:38.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.475+0000 7fb9a3400640 1 --2- 172.21.15.134:0/1224846136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x562153828ad0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb98c00a010 tx=0x7fb98c02c750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a65f99e703733d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:38.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.475+0000 7fb992000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:17:38.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.475+0000 7fb992000640 1 -- 172.21.15.134:0/1224846136 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c031040 con 0x562153405070 2024-07-26T11:17:38.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.476+0000 7fb992000640 1 -- 172.21.15.134:0/1224846136 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fb98c045060 con 0x562153405070 2024-07-26T11:17:38.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.476+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/1224846136 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb99c025070 con 0x562153405070 2024-07-26T11:17:38.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.476+0000 7fb992000640 1 -- 172.21.15.134:0/1224846136 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fb98c031040 con 0x562153405070 2024-07-26T11:17:38.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.477+0000 7fb9b1fc8740 10 client.25819 fetch_fsmap learned FSMap version 31 2024-07-26T11:17:38.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.477+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/1224846136 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5621537bddf0 con 0x562153405070 2024-07-26T11:17:38.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.477+0000 7fb992000640 1 -- 172.21.15.134:0/1224846136 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb99c025070 con 0x562153405070 2024-07-26T11:17:38.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.478+0000 7fb992000640 1 -- 172.21.15.134:0/1224846136 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7fb98c029050 con 0x562153405070 2024-07-26T11:17:38.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.478+0000 7fb9b1fc8740 10 client.25819 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:17:38.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.478+0000 7fb9b1fc8740 10 client.25819 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-07-26T11:17:38.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.478+0000 7fb9b1fc8740 20 client.25819 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:38.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.478+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/1224846136 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5621537be400 0x562153842e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:38.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.479+0000 7fb9b1fc8740 4 client.25819 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:17:38.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.479+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/1224846136 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56215371e5e0 con 0x5621537be400 2024-07-26T11:17:38.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.481+0000 7fb9a3e00640 1 --2- 172.21.15.134:0/1224846136 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5621537be400 0x562153842e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:38.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.482+0000 7fb9a3e00640 1 --2- 172.21.15.134:0/1224846136 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5621537be400 0x562153842e60 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c4b7860146a2998f server_cookie=3a99c7952195f120 in_seq=0 out_seq=0 2024-07-26T11:17:38.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.482+0000 7fb992000640 10 client.25819 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:17:38.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.482+0000 7fb992000640 1 -- 172.21.15.134:0/1224846136 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56215371e5e0 con 0x5621537be400 2024-07-26T11:17:38.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.482+0000 7fb992000640 10 client.25819 handle_command_reply: tid=0 2024-07-26T11:17:38.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.482+0000 7fb9b1fc8740 1 client.25819 shutdown 2024-07-26T11:17:38.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.483+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/1224846136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 msgr2=0x562153828ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:38.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.483+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/1224846136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x562153828ad0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb98c00a010 tx=0x7fb98c02c750 comp rx=0 tx=0).stop 2024-07-26T11:17:38.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.483+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/1224846136 shutdown_connections 2024-07-26T11:17:38.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.483+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/1224846136 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5621537be400 0x562153842e60 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.483+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/1224846136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562153405070 0x562153828ad0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.483+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/1224846136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5621533e4390 0x562153828590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.483+0000 7fb9b1fc8740 1 --2- 172.21.15.134:0/1224846136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562153398da0 0x562153828050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.483+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/1224846136 >> 172.21.15.134:0/1224846136 conn(0x5621533f9460 msgr2=0x5621537ab9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:38.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.483+0000 7fb992000640 0 client.25819 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:17:38.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.484+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/1224846136 shutdown_connections 2024-07-26T11:17:38.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.484+0000 7fb9b1fc8740 1 -- 172.21.15.134:0/1224846136 wait complete. 2024-07-26T11:17:38.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.486+0000 7fb9b1fc8740 20 client.25819 trim_cache size 0 max 16384 2024-07-26T11:17:38.488 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:17:38.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.487+0000 7fb9a9c00640 1 -- 172.21.15.134:0/77401022 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb9840b3e40 msgr2=0x7fb9840b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:38.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.487+0000 7fb9a9c00640 1 --2- 172.21.15.134:0/77401022 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb9840b3e40 0x7fb9840b6350 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb9940c5f10 tx=0x7fb99400ac00 comp rx=0 tx=0).stop 2024-07-26T11:17:38.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.487+0000 7fb9a9c00640 1 -- 172.21.15.134:0/77401022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9a4110070 msgr2=0x7fb9a41194e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:38.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.487+0000 7fb9a9c00640 1 --2- 172.21.15.134:0/77401022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9a4110070 0x7fb9a41194e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb994006010 tx=0x7fb994005400 comp rx=0 tx=0).stop 2024-07-26T11:17:38.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.487+0000 7fb9a9c00640 1 -- 172.21.15.134:0/77401022 shutdown_connections 2024-07-26T11:17:38.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.487+0000 7fb9a9c00640 1 --2- 172.21.15.134:0/77401022 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb9840b3e40 0x7fb9840b6350 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.487+0000 7fb9a9c00640 1 --2- 172.21.15.134:0/77401022 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb9a4110070 0x7fb9a41194e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.487+0000 7fb9a9c00640 1 --2- 172.21.15.134:0/77401022 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb9a410f720 0x7fb9a4118f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.487+0000 7fb9a9c00640 1 --2- 172.21.15.134:0/77401022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9a410a080 0x7fb9a411a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:38.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.487+0000 7fb9a9c00640 1 -- 172.21.15.134:0/77401022 >> 172.21.15.134:0/77401022 conn(0x7fb9a406e880 msgr2=0x7fb9a410d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:38.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.487+0000 7fb9a9c00640 1 -- 172.21.15.134:0/77401022 shutdown_connections 2024-07-26T11:17:38.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:38.488+0000 7fb9a9c00640 1 -- 172.21.15.134:0/77401022 wait complete. 2024-07-26T11:17:39.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:39 smithi134 ceph-mon[31966]: pgmap v530: 129 pgs: 129 active+clean; 4.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 308 MiB/s wr, 101 op/s 2024-07-26T11:17:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:39 smithi189 ceph-mon[36790]: pgmap v530: 129 pgs: 129 active+clean; 4.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 308 MiB/s wr, 101 op/s 2024-07-26T11:17:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:39 smithi193 ceph-mon[37473]: pgmap v530: 129 pgs: 129 active+clean; 4.7 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 308 MiB/s wr, 101 op/s 2024-07-26T11:17:39.877 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-07-26T11:17:40.613 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:17:40.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.804+0000 7f1a15a00640 1 -- 172.21.15.134:0/2552002041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a10076dc0 msgr2=0x7f1a100751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:40.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.804+0000 7f1a15a00640 1 --2- 172.21.15.134:0/2552002041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a10076dc0 0x7f1a100751e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009860 tx=0x7f1a0002a340 comp rx=0 tx=0).stop 2024-07-26T11:17:40.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.804+0000 7f1a15a00640 1 -- 172.21.15.134:0/2552002041 shutdown_connections 2024-07-26T11:17:40.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.805+0000 7f1a15a00640 1 --2- 172.21.15.134:0/2552002041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a10113dd0 0x7f1a101162a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.805+0000 7f1a15a00640 1 --2- 172.21.15.134:0/2552002041 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a10075800 0x7f1a10075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.805+0000 7f1a15a00640 1 --2- 172.21.15.134:0/2552002041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a10076dc0 0x7f1a100751e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.805+0000 7f1a15a00640 1 -- 172.21.15.134:0/2552002041 >> 172.21.15.134:0/2552002041 conn(0x7f1a100fef50 msgr2=0x7f1a101013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:40.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.805+0000 7f1a15a00640 1 -- 172.21.15.134:0/2552002041 shutdown_connections 2024-07-26T11:17:40.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.805+0000 7f1a15a00640 1 -- 172.21.15.134:0/2552002041 wait complete. 2024-07-26T11:17:40.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.806+0000 7f1a15a00640 1 Processor -- start 2024-07-26T11:17:40.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.807+0000 7f1a15a00640 1 -- start start 2024-07-26T11:17:40.809 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.807+0000 7f1a15a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a10075800 0x7f1a101a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.809 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.807+0000 7f1a15a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a10076dc0 0x7f1a101a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.808+0000 7f1a15a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a10113dd0 0x7f1a101a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.808+0000 7f1a0ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a10075800 0x7f1a101a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:40.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.808+0000 7f1a0ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a10075800 0x7f1a101a04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:43504/0 (socket says 172.21.15.134:43504) 2024-07-26T11:17:40.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.808+0000 7f1a0e000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a10076dc0 0x7f1a101a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:40.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.808+0000 7f1a0e000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a10076dc0 0x7f1a101a0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57086/0 (socket says 172.21.15.134:57086) 2024-07-26T11:17:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.808+0000 7f1a0ea00640 1 -- 172.21.15.134:0/258833740 learned_addr learned my addr 172.21.15.134:0/258833740 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.808+0000 7f1a15a00640 1 -- 172.21.15.134:0/258833740 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1a10119a10 con 0x7f1a10076dc0 2024-07-26T11:17:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.808+0000 7f1a15a00640 1 -- 172.21.15.134:0/258833740 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1a10119890 con 0x7f1a10113dd0 2024-07-26T11:17:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.808+0000 7f1a15a00640 1 -- 172.21.15.134:0/258833740 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1a10119b90 con 0x7f1a10075800 2024-07-26T11:17:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.809+0000 7f1a0f400640 1 --2- 172.21.15.134:0/258833740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a10113dd0 0x7f1a101a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.809+0000 7f1a0e000640 1 -- 172.21.15.134:0/258833740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a10075800 msgr2=0x7f1a101a04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.809+0000 7f1a0e000640 1 --2- 172.21.15.134:0/258833740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a10075800 0x7f1a101a04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.809+0000 7f1a0e000640 1 -- 172.21.15.134:0/258833740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a10113dd0 msgr2=0x7f1a101a5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.809+0000 7f1a0e000640 1 --2- 172.21.15.134:0/258833740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a10113dd0 0x7f1a101a5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.809+0000 7f1a0e000640 1 -- 172.21.15.134:0/258833740 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a101a6400 con 0x7f1a10076dc0 2024-07-26T11:17:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.809+0000 7f1a0e000640 1 --2- 172.21.15.134:0/258833740 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a10076dc0 0x7f1a101a0a70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f19f8005fd0 tx=0x7f19f800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.814+0000 7f1a07400640 1 -- 172.21.15.134:0/258833740 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f800cd00 con 0x7f1a10076dc0 2024-07-26T11:17:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.814+0000 7f1a07400640 1 -- 172.21.15.134:0/258833740 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19f80035f0 con 0x7f1a10076dc0 2024-07-26T11:17:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.814+0000 7f1a07400640 1 -- 172.21.15.134:0/258833740 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f8010690 con 0x7f1a10076dc0 2024-07-26T11:17:40.816 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.814+0000 7f1a15a00640 1 -- 172.21.15.134:0/258833740 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a101a66f0 con 0x7f1a10076dc0 2024-07-26T11:17:40.817 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.814+0000 7f1a15a00640 1 -- 172.21.15.134:0/258833740 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a101ace80 con 0x7f1a10076dc0 2024-07-26T11:17:40.819 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.817+0000 7f1a1cb9c740 1 Processor -- start 2024-07-26T11:17:40.819 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.818+0000 7f1a1cb9c740 1 -- start start 2024-07-26T11:17:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.818+0000 7f1a1cb9c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a003c1b0 0x5604a03dfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.820 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.818+0000 7f1a1cb9c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5604a00132f0 0x5604a03dff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.818+0000 7f1a1cb9c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 0x5604a03e04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.818+0000 7f1a1cb9c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5604a0354270 con 0x5604a00132f0 2024-07-26T11:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.818+0000 7f1a1cb9c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5604a0343770 con 0x5604a003c1b0 2024-07-26T11:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.818+0000 7f1a1cb9c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5604a03477b0 con 0x5604a001b4d0 2024-07-26T11:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.819+0000 7f1a0f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a003c1b0 0x5604a03dfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.819+0000 7f1a0f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a003c1b0 0x5604a03dfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42164/0 (socket says 172.21.15.134:42164) 2024-07-26T11:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.819+0000 7f1a0f400640 1 -- 172.21.15.134:0/274952454 learned_addr learned my addr 172.21.15.134:0/274952454 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.819+0000 7f1a0e000640 1 --2- 172.21.15.134:0/274952454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 0x5604a03e04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:40.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.819+0000 7f1a0f400640 1 -- 172.21.15.134:0/274952454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 msgr2=0x5604a03e04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.819+0000 7f1a0f400640 1 --2- 172.21.15.134:0/274952454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 0x5604a03e04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.819+0000 7f1a0f400640 1 -- 172.21.15.134:0/274952454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5604a00132f0 msgr2=0x5604a03dff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.819+0000 7f1a0f400640 1 --2- 172.21.15.134:0/274952454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5604a00132f0 0x5604a03dff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.822 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.819+0000 7f1a0f400640 1 -- 172.21.15.134:0/274952454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604a00a7790 con 0x5604a003c1b0 2024-07-26T11:17:40.823 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.819+0000 7f1a0e000640 1 --2- 172.21.15.134:0/274952454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 0x5604a03e04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:17:40.823 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.820+0000 7f1a0f400640 1 --2- 172.21.15.134:0/274952454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a003c1b0 0x5604a03dfa40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1a08009f70 tx=0x7f1a08010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fe4fdd9a619ba73 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:40.823 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.820+0000 7f19ffe00640 1 -- 172.21.15.134:0/274952454 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a08011700 con 0x5604a003c1b0 2024-07-26T11:17:40.823 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.820+0000 7f19ffe00640 1 -- 172.21.15.134:0/274952454 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a08011d50 con 0x5604a003c1b0 2024-07-26T11:17:40.823 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.820+0000 7f19ffe00640 1 -- 172.21.15.134:0/274952454 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0801a380 con 0x5604a003c1b0 2024-07-26T11:17:40.823 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.821+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/274952454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a003c1b0 msgr2=0x5604a03dfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:40.823 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.821+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/274952454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a003c1b0 0x5604a03dfa40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1a08009f70 tx=0x7f1a08010860 comp rx=0 tx=0).stop 2024-07-26T11:17:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.823+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/274952454 shutdown_connections 2024-07-26T11:17:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.823+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/274952454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 0x5604a03e04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.823+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/274952454 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5604a00132f0 0x5604a03dff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.823+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/274952454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a003c1b0 0x5604a03dfa40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.823+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/274952454 >> 172.21.15.134:0/274952454 conn(0x5604a0030300 msgr2=0x5604a03dd930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.823+0000 7f1a07400640 1 -- 172.21.15.134:0/258833740 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f19f8003a80 con 0x7f1a10076dc0 2024-07-26T11:17:40.825 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.823+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/274952454 shutdown_connections 2024-07-26T11:17:40.827 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.825+0000 7f1a07400640 1 --2- 172.21.15.134:0/258833740 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f19f40b3e20 0x7f19f40b6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.827 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.825+0000 7f1a07400640 1 -- 172.21.15.134:0/258833740 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f19f80e4550 con 0x7f1a10076dc0 2024-07-26T11:17:40.827 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.825+0000 7f1a0ea00640 1 --2- 172.21.15.134:0/258833740 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f19f40b3e20 0x7f19f40b6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:40.827 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.825+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/274952454 wait complete. 2024-07-26T11:17:40.828 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.826+0000 7f1a1cb9c740 1 -- start start 2024-07-26T11:17:40.828 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.826+0000 7f1a1cb9c740 1 Processor -- start 2024-07-26T11:17:40.829 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.826+0000 7f1a0ea00640 1 --2- 172.21.15.134:0/258833740 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f19f40b3e20 0x7f19f40b6330 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x5604a03de6c0 tx=0x7f1a00005e30 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:40.829 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.827+0000 7f1a1cb9c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 0x5604a03eb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.829 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.827+0000 7f1a1cb9c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5604a001b4d0 0x5604a03ebb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.829 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.827+0000 7f1a1cb9c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a003c1b0 0x5604a03f74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.827+0000 7f1a1cb9c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5604a027dab0 con 0x5604a001b4d0 2024-07-26T11:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.827+0000 7f1a1cb9c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5604a02e0d50 con 0x5604a00132f0 2024-07-26T11:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.827+0000 7f1a1cb9c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5604a03ee1c0 con 0x5604a003c1b0 2024-07-26T11:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.827+0000 7f1a1cb9c740 10 client.0 _ll_register_callbacks cb 0x5604a0053a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.827+0000 7f1a0f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a003c1b0 0x5604a03f74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.827+0000 7f1a0f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a003c1b0 0x5604a03f74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:43528/0 (socket says 172.21.15.134:43528) 2024-07-26T11:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.827+0000 7f1a0f400640 1 -- 172.21.15.134:0/1701960582 learned_addr learned my addr 172.21.15.134:0/1701960582 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:40.830 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.827+0000 7f1a0ea00640 1 --2- 172.21.15.134:0/1701960582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 0x5604a03eb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.828+0000 7f1a0e000640 1 --2- 172.21.15.134:0/1701960582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5604a001b4d0 0x5604a03ebb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.828+0000 7f1a0f400640 1 -- 172.21.15.134:0/1701960582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 msgr2=0x5604a03eb620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:40.831 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.828+0000 7f1a0f400640 1 --2- 172.21.15.134:0/1701960582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 0x5604a03eb620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.828+0000 7f1a0f400640 1 -- 172.21.15.134:0/1701960582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5604a001b4d0 msgr2=0x5604a03ebb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.828+0000 7f1a0f400640 1 --2- 172.21.15.134:0/1701960582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5604a001b4d0 0x5604a03ebb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.828+0000 7f1a0f400640 1 -- 172.21.15.134:0/1701960582 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604a0120620 con 0x5604a003c1b0 2024-07-26T11:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.828+0000 7f1a0e000640 1 --2- 172.21.15.134:0/1701960582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5604a001b4d0 0x5604a03ebb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.829+0000 7f1a0f400640 1 --2- 172.21.15.134:0/1701960582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a003c1b0 0x5604a03f74b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1a0801ca60 tx=0x7f1a0801de00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58400a1f47b26cba server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.829+0000 7f19fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.829+0000 7f19fcc00640 1 -- 172.21.15.134:0/1701960582 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a08004e50 con 0x5604a003c1b0 2024-07-26T11:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.829+0000 7f19fcc00640 1 -- 172.21.15.134:0/1701960582 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f1a08011500 con 0x5604a003c1b0 2024-07-26T11:17:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.830+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/1701960582 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a08011d50 con 0x5604a003c1b0 2024-07-26T11:17:40.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.830+0000 7f19fcc00640 1 -- 172.21.15.134:0/1701960582 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f1a08011700 con 0x5604a003c1b0 2024-07-26T11:17:40.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.830+0000 7f1a1cb9c740 10 client.25831 fetch_fsmap learned FSMap version 31 2024-07-26T11:17:40.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.830+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/1701960582 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5604a040b210 con 0x5604a003c1b0 2024-07-26T11:17:40.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.831+0000 7f19fcc00640 1 -- 172.21.15.134:0/1701960582 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a08011d50 con 0x5604a003c1b0 2024-07-26T11:17:40.834 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.832+0000 7f19fcc00640 1 -- 172.21.15.134:0/1701960582 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f1a08025070 con 0x5604a003c1b0 2024-07-26T11:17:40.835 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.832+0000 7f1a1cb9c740 10 client.25831 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:17:40.835 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.832+0000 7f1a1cb9c740 10 client.25831 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:17:40.835 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.832+0000 7f1a1cb9c740 20 client.25831 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:40.835 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.833+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/1701960582 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56049fffad80 0x5604a040e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.835 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.833+0000 7f1a1cb9c740 4 client.25831 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:40.835 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.833+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/1701960582 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604a03561c0 con 0x56049fffad80 2024-07-26T11:17:40.835 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.833+0000 7f1a0ea00640 1 --2- 172.21.15.134:0/1701960582 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56049fffad80 0x5604a040e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:40.837 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.836+0000 7f19fcc00640 10 client.25831 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:40.837 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.836+0000 7f1a0ea00640 1 --2- 172.21.15.134:0/1701960582 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56049fffad80 0x5604a040e5c0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74dae9334c9e837f server_cookie=c5e0da48c2b58deb in_seq=0 out_seq=0 2024-07-26T11:17:40.849 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.846+0000 7f19fcc00640 1 -- 172.21.15.134:0/1701960582 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5604a03561c0 con 0x56049fffad80 2024-07-26T11:17:40.849 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.846+0000 7f19fcc00640 10 client.25831 handle_command_reply: tid=0 2024-07-26T11:17:40.849 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.847+0000 7f1a1cb9c740 1 client.25831 shutdown 2024-07-26T11:17:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.848+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/1701960582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a003c1b0 msgr2=0x5604a03f74b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.848+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/1701960582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a003c1b0 0x5604a03f74b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1a0801ca60 tx=0x7f1a0801de00 comp rx=0 tx=0).stop 2024-07-26T11:17:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.849+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/1701960582 shutdown_connections 2024-07-26T11:17:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.849+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/1701960582 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56049fffad80 0x5604a040e5c0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.850+0000 7f19fcc00640 0 client.25831 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.850+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/1701960582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a003c1b0 0x5604a03f74b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.850+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/1701960582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5604a001b4d0 0x5604a03ebb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.854+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/1701960582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 0x5604a03eb620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.854+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/1701960582 >> 172.21.15.134:0/1701960582 conn(0x5604a0030300 msgr2=0x5604a03dc0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.855+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/1701960582 shutdown_connections 2024-07-26T11:17:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.855+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/1701960582 wait complete. 2024-07-26T11:17:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.856+0000 7f1a1cb9c740 20 client.25831 trim_cache size 0 max 16384 2024-07-26T11:17:40.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.897+0000 7f1a1cb9c740 1 Processor -- start 2024-07-26T11:17:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.898+0000 7f1a1cb9c740 1 -- start start 2024-07-26T11:17:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.898+0000 7f1a1cb9c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56049fffad80 0x5604a03e5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.898+0000 7f1a1cb9c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 0x5604a03eb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.898+0000 7f1a1cb9c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 0x5604a03ed040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.898+0000 7f1a1cb9c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1a101ae740 con 0x56049fffad80 2024-07-26T11:17:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.898+0000 7f1a1cb9c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1a101b02b0 con 0x5604a00132f0 2024-07-26T11:17:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.898+0000 7f1a1cb9c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1a101aee10 con 0x5604a001b4d0 2024-07-26T11:17:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.898+0000 7f1a0f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 0x5604a03ed040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.898+0000 7f1a0f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 0x5604a03ed040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:43542/0 (socket says 172.21.15.134:43542) 2024-07-26T11:17:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.898+0000 7f1a0f400640 1 -- 172.21.15.134:0/1366055194 learned_addr learned my addr 172.21.15.134:0/1366055194 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.898+0000 7f1a0e000640 1 --2- 172.21.15.134:0/1366055194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 0x5604a03eb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.899+0000 7f1a0f400640 1 -- 172.21.15.134:0/1366055194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 msgr2=0x5604a03eb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.899+0000 7f1a0f400640 1 --2- 172.21.15.134:0/1366055194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 0x5604a03eb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.899+0000 7f1a0f400640 1 -- 172.21.15.134:0/1366055194 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56049fffad80 msgr2=0x5604a03e5b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:17:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.899+0000 7f1a0f400640 1 --2- 172.21.15.134:0/1366055194 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56049fffad80 0x5604a03e5b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.899+0000 7f1a0f400640 1 -- 172.21.15.134:0/1366055194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604a03ee0a0 con 0x5604a001b4d0 2024-07-26T11:17:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.899+0000 7f1a0f400640 1 --2- 172.21.15.134:0/1366055194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 0x5604a03ed040 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1a080059e0 tx=0x7f1a08009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4c42d35a3da3893 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.899+0000 7f19ff400640 1 -- 172.21.15.134:0/1366055194 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0803c070 con 0x5604a001b4d0 2024-07-26T11:17:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.899+0000 7f19ff400640 1 -- 172.21.15.134:0/1366055194 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a0801de70 con 0x5604a001b4d0 2024-07-26T11:17:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.899+0000 7f19ff400640 1 -- 172.21.15.134:0/1366055194 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a08011d00 con 0x5604a001b4d0 2024-07-26T11:17:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.901+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/1366055194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 msgr2=0x5604a03ed040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.901+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/1366055194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 0x5604a03ed040 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1a080059e0 tx=0x7f1a08009b80 comp rx=0 tx=0).stop 2024-07-26T11:17:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.901+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/1366055194 shutdown_connections 2024-07-26T11:17:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.901+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/1366055194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 0x5604a03ed040 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.901+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/1366055194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 0x5604a03eb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.901+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/1366055194 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56049fffad80 0x5604a03e5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.901+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/1366055194 >> 172.21.15.134:0/1366055194 conn(0x5604a0030300 msgr2=0x5604a03ec930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.902+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/1366055194 shutdown_connections 2024-07-26T11:17:40.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.902+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/1366055194 wait complete. 2024-07-26T11:17:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.903+0000 7f1a1cb9c740 1 -- start start 2024-07-26T11:17:40.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.903+0000 7f1a1cb9c740 1 Processor -- start 2024-07-26T11:17:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.904+0000 7f1a1cb9c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56049fffad80 0x5604a040ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.906 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.904+0000 7f1a1cb9c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 0x5604a040f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.904+0000 7f1a1cb9c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 0x5604a046ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.904+0000 7f1a1cb9c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1a101afbb0 con 0x56049fffad80 2024-07-26T11:17:40.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.904+0000 7f1a1cb9c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1a101af4e0 con 0x5604a00132f0 2024-07-26T11:17:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.904+0000 7f1a1cb9c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1a100746e0 con 0x5604a001b4d0 2024-07-26T11:17:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.904+0000 7f1a1cb9c740 10 client.0 _ll_register_callbacks cb 0x7f1a08009e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.905+0000 7f1a0e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 0x5604a040f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.905+0000 7f1a0e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 0x5604a040f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42184/0 (socket says 172.21.15.134:42184) 2024-07-26T11:17:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.905+0000 7f1a0e000640 1 -- 172.21.15.134:0/3574462468 learned_addr learned my addr 172.21.15.134:0/3574462468 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.905+0000 7f1a0e000640 1 -- 172.21.15.134:0/3574462468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 msgr2=0x5604a046ead0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:17:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.905+0000 7f1a0e000640 1 --2- 172.21.15.134:0/3574462468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 0x5604a046ead0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.905+0000 7f1a0e000640 1 -- 172.21.15.134:0/3574462468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56049fffad80 msgr2=0x5604a040ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.905+0000 7f1a0e000640 1 --2- 172.21.15.134:0/3574462468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56049fffad80 0x5604a040ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.905+0000 7f1a0e000640 1 -- 172.21.15.134:0/3574462468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604a046f170 con 0x5604a00132f0 2024-07-26T11:17:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.906+0000 7f1a0e000640 1 --2- 172.21.15.134:0/3574462468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 0x5604a040f360 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f19f80aae10 tx=0x7f19f80e1e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cf4db194ae6d83b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.906+0000 7f19fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:17:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.907+0000 7f19fcc00640 1 -- 172.21.15.134:0/3574462468 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19f80aa7d0 con 0x5604a00132f0 2024-07-26T11:17:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.907+0000 7f19fcc00640 1 -- 172.21.15.134:0/3574462468 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f19f80e42a0 con 0x5604a00132f0 2024-07-26T11:17:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.907+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/3574462468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a0801de70 con 0x5604a00132f0 2024-07-26T11:17:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.907+0000 7f19fcc00640 1 -- 172.21.15.134:0/3574462468 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f19f80aae80 con 0x5604a00132f0 2024-07-26T11:17:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.907+0000 7f1a1cb9c740 10 client.25829 fetch_fsmap learned FSMap version 31 2024-07-26T11:17:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.907+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/3574462468 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5604a0476440 con 0x5604a00132f0 2024-07-26T11:17:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.907+0000 7f19fcc00640 1 -- 172.21.15.134:0/3574462468 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a0801de70 con 0x5604a00132f0 2024-07-26T11:17:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.908+0000 7f19fcc00640 1 -- 172.21.15.134:0/3574462468 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f19f80a9b50 con 0x5604a00132f0 2024-07-26T11:17:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.908+0000 7f1a1cb9c740 10 client.25829 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:17:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.908+0000 7f1a1cb9c740 10 client.25829 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:17:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.908+0000 7f1a1cb9c740 20 client.25829 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.908+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/3574462468 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5604a003c1b0 0x5604a0478c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.908+0000 7f1a1cb9c740 4 client.25829 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:17:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.908+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/3574462468 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5604a0351c60 con 0x5604a003c1b0 2024-07-26T11:17:40.913 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.909+0000 7f1a0ea00640 1 --2- 172.21.15.134:0/3574462468 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5604a003c1b0 0x5604a0478c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.916+0000 7f1a0ea00640 1 --2- 172.21.15.134:0/3574462468 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5604a003c1b0 0x5604a0478c60 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d77b6a5a711cbce server_cookie=c13034140298a06f in_seq=0 out_seq=0 2024-07-26T11:17:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.916+0000 7f19fcc00640 10 client.25829 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:40.956 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.952+0000 7f19fcc00640 1 -- 172.21.15.134:0/3574462468 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5604a0351c60 con 0x5604a003c1b0 2024-07-26T11:17:40.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.955+0000 7f19fcc00640 10 client.25829 handle_command_reply: tid=0 2024-07-26T11:17:40.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.955+0000 7f1a1cb9c740 1 client.25829 shutdown 2024-07-26T11:17:40.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.955+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/3574462468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 msgr2=0x5604a040f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:40.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.955+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/3574462468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 0x5604a040f360 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f19f80aae10 tx=0x7f19f80e1e70 comp rx=0 tx=0).stop 2024-07-26T11:17:40.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.955+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/3574462468 shutdown_connections 2024-07-26T11:17:40.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.955+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/3574462468 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5604a003c1b0 0x5604a0478c60 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.955+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/3574462468 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5604a001b4d0 0x5604a046ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.955+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/3574462468 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5604a00132f0 0x5604a040f360 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.955+0000 7f1a1cb9c740 1 --2- 172.21.15.134:0/3574462468 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56049fffad80 0x5604a040ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.955+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/3574462468 >> 172.21.15.134:0/3574462468 conn(0x5604a0030300 msgr2=0x5604a03dd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:40.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.955+0000 7f19fcc00640 0 client.25829 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:40.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.956+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/3574462468 shutdown_connections 2024-07-26T11:17:40.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.956+0000 7f1a1cb9c740 1 -- 172.21.15.134:0/3574462468 wait complete. 2024-07-26T11:17:40.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.956+0000 7f1a1cb9c740 20 client.25829 trim_cache size 0 max 16384 2024-07-26T11:17:40.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:17:40.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:17:40.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:17:40.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:17:40.961 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.958+0000 7f1a15a00640 1 -- 172.21.15.134:0/258833740 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f19f40b3e20 msgr2=0x7f19f40b6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:40.961 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.958+0000 7f1a15a00640 1 --2- 172.21.15.134:0/258833740 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f19f40b3e20 0x7f19f40b6330 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x5604a03de6c0 tx=0x7f1a00005e30 comp rx=0 tx=0).stop 2024-07-26T11:17:40.961 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.958+0000 7f1a15a00640 1 -- 172.21.15.134:0/258833740 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a10076dc0 msgr2=0x7f1a101a0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:40.961 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.958+0000 7f1a15a00640 1 --2- 172.21.15.134:0/258833740 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a10076dc0 0x7f1a101a0a70 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f19f8005fd0 tx=0x7f19f800ee50 comp rx=0 tx=0).stop 2024-07-26T11:17:40.962 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.960+0000 7f1a15a00640 1 -- 172.21.15.134:0/258833740 shutdown_connections 2024-07-26T11:17:40.962 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.960+0000 7f1a15a00640 1 --2- 172.21.15.134:0/258833740 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f19f40b3e20 0x7f19f40b6330 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.962 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.960+0000 7f1a15a00640 1 --2- 172.21.15.134:0/258833740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a10113dd0 0x7f1a101a5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.962 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.960+0000 7f1a15a00640 1 --2- 172.21.15.134:0/258833740 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a10076dc0 0x7f1a101a0a70 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.962 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.960+0000 7f1a15a00640 1 --2- 172.21.15.134:0/258833740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a10075800 0x7f1a101a04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:40.962 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.960+0000 7f1a15a00640 1 -- 172.21.15.134:0/258833740 >> 172.21.15.134:0/258833740 conn(0x7f1a100fef50 msgr2=0x7f1a1010fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:40.964 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.961+0000 7f1a15a00640 1 -- 172.21.15.134:0/258833740 shutdown_connections 2024-07-26T11:17:40.964 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:40.962+0000 7f1a15a00640 1 -- 172.21.15.134:0/258833740 wait complete. 2024-07-26T11:17:41.595 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:41 smithi134 ceph-mon[31966]: pgmap v531: 129 pgs: 129 active+clean; 5.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 339 MiB/s wr, 103 op/s 2024-07-26T11:17:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:41 smithi189 ceph-mon[36790]: pgmap v531: 129 pgs: 129 active+clean; 5.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 339 MiB/s wr, 103 op/s 2024-07-26T11:17:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:41 smithi193 ceph-mon[37473]: pgmap v531: 129 pgs: 129 active+clean; 5.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 339 MiB/s wr, 103 op/s 2024-07-26T11:17:43.645 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:43 smithi134 ceph-mon[31966]: pgmap v532: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 296 MiB/s wr, 88 op/s 2024-07-26T11:17:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:43 smithi189 ceph-mon[36790]: pgmap v532: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 296 MiB/s wr, 88 op/s 2024-07-26T11:17:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:43 smithi193 ceph-mon[37473]: pgmap v532: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 296 MiB/s wr, 88 op/s 2024-07-26T11:17:44.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.168+0000 7f187ea00640 1 -- 172.21.15.134:0/372885823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18700ab1b0 msgr2=0x7f18700ab9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:44.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.168+0000 7f187ea00640 1 --2- 172.21.15.134:0/372885823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18700ab1b0 0x7f18700ab9d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f18780698b0 tx=0x7f187808f460 comp rx=0 tx=0).stop 2024-07-26T11:17:44.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.169+0000 7f187ea00640 1 -- 172.21.15.134:0/372885823 shutdown_connections 2024-07-26T11:17:44.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.169+0000 7f187ea00640 1 --2- 172.21.15.134:0/372885823 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18700ab1b0 0x7f18700ab9d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.169+0000 7f187ea00640 1 --2- 172.21.15.134:0/372885823 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f18700aa860 0x7f18700aac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.169+0000 7f187ea00640 1 --2- 172.21.15.134:0/372885823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18700a50f0 0x7f18700a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.169+0000 7f187ea00640 1 -- 172.21.15.134:0/372885823 >> 172.21.15.134:0/372885823 conn(0x7f187001cc70 msgr2=0x7f187001d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:44.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.169+0000 7f187ea00640 1 -- 172.21.15.134:0/372885823 shutdown_connections 2024-07-26T11:17:44.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.169+0000 7f187ea00640 1 -- 172.21.15.134:0/372885823 wait complete. 2024-07-26T11:17:44.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.171+0000 7f187ea00640 1 Processor -- start 2024-07-26T11:17:44.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.171+0000 7f187ea00640 1 -- start start 2024-07-26T11:17:44.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.172+0000 7f187ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18700a50f0 0x7f1870143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.172+0000 7f187ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18700aa860 0x7f1870144570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.172+0000 7f187ea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f18700ab1b0 0x7f1870149810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.172+0000 7f187ea00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f18700bd580 con 0x7f18700ab1b0 2024-07-26T11:17:44.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.172+0000 7f187ea00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f18700bd400 con 0x7f18700aa860 2024-07-26T11:17:44.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.172+0000 7f187ea00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f18700bd700 con 0x7f18700a50f0 2024-07-26T11:17:44.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.173+0000 7f187cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18700aa860 0x7f1870144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:44.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.173+0000 7f187cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18700aa860 0x7f1870144570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42200/0 (socket says 172.21.15.134:42200) 2024-07-26T11:17:44.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.173+0000 7f187cc00640 1 -- 172.21.15.134:0/24602132 learned_addr learned my addr 172.21.15.134:0/24602132 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:44.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.173+0000 7f187d600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18700a50f0 0x7f1870143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:44.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.173+0000 7f187e000640 1 --2- 172.21.15.134:0/24602132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f18700ab1b0 0x7f1870149810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:44.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.174+0000 7f187d600640 1 -- 172.21.15.134:0/24602132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18700aa860 msgr2=0x7f1870144570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:44.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.174+0000 7f187d600640 1 --2- 172.21.15.134:0/24602132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18700aa860 0x7f1870144570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.174+0000 7f187d600640 1 -- 172.21.15.134:0/24602132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f18700ab1b0 msgr2=0x7f1870149810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:44.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.174+0000 7f187d600640 1 --2- 172.21.15.134:0/24602132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f18700ab1b0 0x7f1870149810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.174+0000 7f187d600640 1 -- 172.21.15.134:0/24602132 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1870149f10 con 0x7f18700a50f0 2024-07-26T11:17:44.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.174+0000 7f187cc00640 1 --2- 172.21.15.134:0/24602132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18700aa860 0x7f1870144570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:44.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.174+0000 7f187e000640 1 --2- 172.21.15.134:0/24602132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f18700ab1b0 0x7f1870149810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:17:44.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.174+0000 7f187d600640 1 --2- 172.21.15.134:0/24602132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18700a50f0 0x7f1870143fd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f186800d700 tx=0x7f186800dbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:44.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.175+0000 7f1876000640 1 -- 172.21.15.134:0/24602132 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1868017070 con 0x7f18700a50f0 2024-07-26T11:17:44.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.175+0000 7f1876000640 1 -- 172.21.15.134:0/24602132 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1868010460 con 0x7f18700a50f0 2024-07-26T11:17:44.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.175+0000 7f1876000640 1 -- 172.21.15.134:0/24602132 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186800f600 con 0x7f18700a50f0 2024-07-26T11:17:44.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.175+0000 7f187ea00640 1 -- 172.21.15.134:0/24602132 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f187014a200 con 0x7f18700a50f0 2024-07-26T11:17:44.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.175+0000 7f187ea00640 1 -- 172.21.15.134:0/24602132 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18700ace20 con 0x7f18700a50f0 2024-07-26T11:17:44.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.182+0000 7f1886922740 1 Processor -- start 2024-07-26T11:17:44.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.182+0000 7f1886922740 1 -- start start 2024-07-26T11:17:44.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.183+0000 7f1876000640 1 -- 172.21.15.134:0/24602132 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f186801a070 con 0x7f18700a50f0 2024-07-26T11:17:44.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.183+0000 7f1886922740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9291aa070 0x55d92954f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.183+0000 7f1886922740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291811b0 0x55d92954f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.183+0000 7f1886922740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d929189390 0x55d92954fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.183+0000 7f187e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9291aa070 0x55d92954f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:44.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.184+0000 7f1886922740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d9294c5bd0 con 0x55d929189390 2024-07-26T11:17:44.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.184+0000 7f1886922740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d9294b50d0 con 0x55d9291aa070 2024-07-26T11:17:44.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.184+0000 7f187e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9291aa070 0x55d92954f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42204/0 (socket says 172.21.15.134:42204) 2024-07-26T11:17:44.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.184+0000 7f187e000640 1 -- 172.21.15.134:0/2419160892 learned_addr learned my addr 172.21.15.134:0/2419160892 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:44.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.184+0000 7f1886922740 1 -- 172.21.15.134:0/2419160892 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d9294b9110 con 0x55d9291811b0 2024-07-26T11:17:44.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.184+0000 7f187cc00640 1 --2- 172.21.15.134:0/2419160892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d929189390 0x55d92954fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:44.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.184+0000 7f187e000640 1 -- 172.21.15.134:0/2419160892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291811b0 msgr2=0x55d92954f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:44.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.184+0000 7f187e000640 1 --2- 172.21.15.134:0/2419160892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291811b0 0x55d92954f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.184+0000 7f187e000640 1 -- 172.21.15.134:0/2419160892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d929189390 msgr2=0x55d92954fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:44.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.185+0000 7f187e000640 1 --2- 172.21.15.134:0/2419160892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d929189390 0x55d92954fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.185+0000 7f187e000640 1 -- 172.21.15.134:0/2419160892 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d929215650 con 0x55d9291aa070 2024-07-26T11:17:44.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.185+0000 7f187cc00640 1 --2- 172.21.15.134:0/2419160892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d929189390 0x55d92954fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:17:44.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.185+0000 7f187e000640 1 --2- 172.21.15.134:0/2419160892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9291aa070 0x55d92954f010 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1878050c60 tx=0x7f1878069240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b9a0b558e54c7a6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:44.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.186+0000 7f1876000640 1 --2- 172.21.15.134:0/24602132 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f18600b3ed0 0x7f18600b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.186+0000 7f1866600640 1 -- 172.21.15.134:0/2419160892 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18780a7020 con 0x55d9291aa070 2024-07-26T11:17:44.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.186+0000 7f1866600640 1 -- 172.21.15.134:0/2419160892 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f187809d040 con 0x55d9291aa070 2024-07-26T11:17:44.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.186+0000 7f1866600640 1 -- 172.21.15.134:0/2419160892 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18780a7020 con 0x55d9291aa070 2024-07-26T11:17:44.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.186+0000 7f1876000640 1 -- 172.21.15.134:0/24602132 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f18680e3c00 con 0x7f18700a50f0 2024-07-26T11:17:44.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.186+0000 7f187d600640 1 --2- 172.21.15.134:0/24602132 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f18600b3ed0 0x7f18600b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:44.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.187+0000 7f1886922740 1 -- 172.21.15.134:0/2419160892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9291aa070 msgr2=0x55d92954f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:44.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.187+0000 7f1886922740 1 --2- 172.21.15.134:0/2419160892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9291aa070 0x55d92954f010 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1878050c60 tx=0x7f1878069240 comp rx=0 tx=0).stop 2024-07-26T11:17:44.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.187+0000 7f1886922740 1 -- 172.21.15.134:0/2419160892 shutdown_connections 2024-07-26T11:17:44.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.187+0000 7f1886922740 1 --2- 172.21.15.134:0/2419160892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291811b0 0x55d92954f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.187+0000 7f1886922740 1 --2- 172.21.15.134:0/2419160892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d929189390 0x55d92954fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.187+0000 7f1886922740 1 --2- 172.21.15.134:0/2419160892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9291aa070 0x55d92954f010 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.187+0000 7f1886922740 1 -- 172.21.15.134:0/2419160892 >> 172.21.15.134:0/2419160892 conn(0x55d92919e460 msgr2=0x55d92954cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:44.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.188+0000 7f1886922740 1 -- 172.21.15.134:0/2419160892 shutdown_connections 2024-07-26T11:17:44.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.188+0000 7f187d600640 1 --2- 172.21.15.134:0/24602132 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f18600b3ed0 0x7f18600b63e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55d92954dc90 tx=0x7f1868000bc0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:44.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.188+0000 7f1886922740 1 -- 172.21.15.134:0/2419160892 wait complete. 2024-07-26T11:17:44.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.189+0000 7f1886922740 1 -- start start 2024-07-26T11:17:44.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.189+0000 7f1886922740 1 Processor -- start 2024-07-26T11:17:44.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.190+0000 7f1886922740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d9291811b0 0x55d92955a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.190+0000 7f1886922740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d929189390 0x55d92955aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.190+0000 7f1886922740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291aa070 0x55d92955afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.190+0000 7f1886922740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d929318700 con 0x55d9291811b0 2024-07-26T11:17:44.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.190+0000 7f1886922740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d9294516a0 con 0x55d929189390 2024-07-26T11:17:44.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.190+0000 7f1886922740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d92955d790 con 0x55d9291aa070 2024-07-26T11:17:44.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.190+0000 7f1886922740 10 client.0 _ll_register_callbacks cb 0x55d929461150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:44.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.190+0000 7f187e000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d9291811b0 0x55d92955a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:44.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.190+0000 7f187cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d929189390 0x55d92955aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:44.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.190+0000 7f187cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d929189390 0x55d92955aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42206/0 (socket says 172.21.15.134:42206) 2024-07-26T11:17:44.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.190+0000 7f187e000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d9291811b0 0x55d92955a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57146/0 (socket says 172.21.15.134:57146) 2024-07-26T11:17:44.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.190+0000 7f187cc00640 1 -- 172.21.15.134:0/2044504786 learned_addr learned my addr 172.21.15.134:0/2044504786 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:44.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.191+0000 7f187e000640 1 --2- 172.21.15.134:0/2044504786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291aa070 0x55d92955afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:44.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.191+0000 7f187cc00640 1 -- 172.21.15.134:0/2044504786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291aa070 msgr2=0x55d92955afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:44.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.191+0000 7f187cc00640 1 --2- 172.21.15.134:0/2044504786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291aa070 0x55d92955afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.191+0000 7f187cc00640 1 -- 172.21.15.134:0/2044504786 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d9291811b0 msgr2=0x55d92955a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:44.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.191+0000 7f187cc00640 1 --2- 172.21.15.134:0/2044504786 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d9291811b0 0x55d92955a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.191+0000 7f187cc00640 1 -- 172.21.15.134:0/2044504786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d929468840 con 0x55d929189390 2024-07-26T11:17:44.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.192+0000 7f187cc00640 1 --2- 172.21.15.134:0/2044504786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d929189390 0x55d92955aa90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f186c0107e0 tx=0x7f186c010ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2e0671b29a3f730 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:44.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.192+0000 7f185aa00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:17:44.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.192+0000 7f185aa00640 1 -- 172.21.15.134:0/2044504786 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c00f040 con 0x55d929189390 2024-07-26T11:17:44.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.192+0000 7f185aa00640 1 -- 172.21.15.134:0/2044504786 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f186c00eb00 con 0x55d929189390 2024-07-26T11:17:44.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.192+0000 7f1886922740 1 -- 172.21.15.134:0/2044504786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f187809d040 con 0x55d929189390 2024-07-26T11:17:44.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.192+0000 7f185aa00640 1 -- 172.21.15.134:0/2044504786 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f186c00ed00 con 0x55d929189390 2024-07-26T11:17:44.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.192+0000 7f1886922740 10 client.25841 fetch_fsmap learned FSMap version 31 2024-07-26T11:17:44.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.192+0000 7f1886922740 1 -- 172.21.15.134:0/2044504786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d929569370 con 0x55d929189390 2024-07-26T11:17:44.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.193+0000 7f185aa00640 1 -- 172.21.15.134:0/2044504786 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f187809d040 con 0x55d929189390 2024-07-26T11:17:44.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.193+0000 7f185aa00640 1 -- 172.21.15.134:0/2044504786 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f186c01c400 con 0x55d929189390 2024-07-26T11:17:44.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.193+0000 7f1886922740 10 client.25841 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:17:44.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.193+0000 7f1886922740 10 client.25841 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T11:17:44.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.193+0000 7f1886922740 20 client.25841 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:44.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.193+0000 7f1886922740 1 --2- 172.21.15.134:0/2044504786 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55d9291689f0 0x55d92956c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.193+0000 7f1886922740 4 client.25841 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:44.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.193+0000 7f1886922740 1 -- 172.21.15.134:0/2044504786 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9294c7b20 con 0x55d9291689f0 2024-07-26T11:17:44.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.193+0000 7f187e000640 1 --2- 172.21.15.134:0/2044504786 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55d9291689f0 0x55d92956c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:44.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.194+0000 7f187e000640 1 --2- 172.21.15.134:0/2044504786 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55d9291689f0 0x55d92956c8a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dbfe152e1c4c5d17 server_cookie=a3afd8024c602bab in_seq=0 out_seq=0 2024-07-26T11:17:44.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.194+0000 7f185aa00640 10 client.25841 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:17:44.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.194+0000 7f185aa00640 1 -- 172.21.15.134:0/2044504786 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d9294c7b20 con 0x55d9291689f0 2024-07-26T11:17:44.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.194+0000 7f185aa00640 10 client.25841 handle_command_reply: tid=0 2024-07-26T11:17:44.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.194+0000 7f1886922740 1 client.25841 shutdown 2024-07-26T11:17:44.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.195+0000 7f1886922740 1 -- 172.21.15.134:0/2044504786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d929189390 msgr2=0x55d92955aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:44.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.195+0000 7f1886922740 1 --2- 172.21.15.134:0/2044504786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d929189390 0x55d92955aa90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f186c0107e0 tx=0x7f186c010ca0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.195+0000 7f1886922740 1 -- 172.21.15.134:0/2044504786 shutdown_connections 2024-07-26T11:17:44.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.195+0000 7f1886922740 1 --2- 172.21.15.134:0/2044504786 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55d9291689f0 0x55d92956c8a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.195+0000 7f1886922740 1 --2- 172.21.15.134:0/2044504786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291aa070 0x55d92955afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.195+0000 7f1886922740 1 --2- 172.21.15.134:0/2044504786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d929189390 0x55d92955aa90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.195+0000 7f1886922740 1 --2- 172.21.15.134:0/2044504786 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d9291811b0 0x55d92955a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.195+0000 7f185aa00640 0 client.25841 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:17:44.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.195+0000 7f1886922740 1 -- 172.21.15.134:0/2044504786 >> 172.21.15.134:0/2044504786 conn(0x55d92919e460 msgr2=0x55d9295521a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:44.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.195+0000 7f1886922740 1 -- 172.21.15.134:0/2044504786 shutdown_connections 2024-07-26T11:17:44.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.195+0000 7f1886922740 1 -- 172.21.15.134:0/2044504786 wait complete. 2024-07-26T11:17:44.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.195+0000 7f1886922740 20 client.25841 trim_cache size 0 max 16384 2024-07-26T11:17:44.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.209+0000 7f1886922740 1 Processor -- start 2024-07-26T11:17:44.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.209+0000 7f1886922740 1 -- start start 2024-07-26T11:17:44.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.209+0000 7f1886922740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291811b0 0x55d929550a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.210+0000 7f1886922740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d929189390 0x55d92954b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.210+0000 7f1886922740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d9291aa070 0x55d92955c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.210+0000 7f1886922740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f18701653f0 con 0x55d9291aa070 2024-07-26T11:17:44.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.210+0000 7f1886922740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f18701631b0 con 0x55d929189390 2024-07-26T11:17:44.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.210+0000 7f1886922740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1870164d20 con 0x55d9291811b0 2024-07-26T11:17:44.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.210+0000 7f187e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291811b0 0x55d929550a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:44.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.210+0000 7f187cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d929189390 0x55d92954b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:44.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.210+0000 7f187cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d929189390 0x55d92954b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42216/0 (socket says 172.21.15.134:42216) 2024-07-26T11:17:44.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.210+0000 7f187cc00640 1 -- 172.21.15.134:0/546399101 learned_addr learned my addr 172.21.15.134:0/546399101 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:44.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.210+0000 7f187e000640 1 --2- 172.21.15.134:0/546399101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d9291aa070 0x55d92955c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:44.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.210+0000 7f187cc00640 1 -- 172.21.15.134:0/546399101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291811b0 msgr2=0x55d929550a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:44.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.210+0000 7f187cc00640 1 --2- 172.21.15.134:0/546399101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291811b0 0x55d929550a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.210+0000 7f187cc00640 1 -- 172.21.15.134:0/546399101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d9291aa070 msgr2=0x55d92955c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:44.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.210+0000 7f187cc00640 1 --2- 172.21.15.134:0/546399101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d9291aa070 0x55d92955c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.210+0000 7f187cc00640 1 -- 172.21.15.134:0/546399101 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d929551740 con 0x55d929189390 2024-07-26T11:17:44.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.211+0000 7f187cc00640 1 --2- 172.21.15.134:0/546399101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d929189390 0x55d92954b540 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f186c01c820 tx=0x7f186c01de50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c8d5729b4761f7e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:44.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.211+0000 7f1865c00640 1 -- 172.21.15.134:0/546399101 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c01c400 con 0x55d929189390 2024-07-26T11:17:44.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.211+0000 7f1865c00640 1 -- 172.21.15.134:0/546399101 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f186c01c5a0 con 0x55d929189390 2024-07-26T11:17:44.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.211+0000 7f1865c00640 1 -- 172.21.15.134:0/546399101 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186c0062a0 con 0x55d929189390 2024-07-26T11:17:44.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.213+0000 7f1886922740 1 -- 172.21.15.134:0/546399101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d929189390 msgr2=0x55d92954b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:44.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.213+0000 7f1886922740 1 --2- 172.21.15.134:0/546399101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d929189390 0x55d92954b540 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f186c01c820 tx=0x7f186c01de50 comp rx=0 tx=0).stop 2024-07-26T11:17:44.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.213+0000 7f1886922740 1 -- 172.21.15.134:0/546399101 shutdown_connections 2024-07-26T11:17:44.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.213+0000 7f1886922740 1 --2- 172.21.15.134:0/546399101 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d9291aa070 0x55d92955c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.213+0000 7f1886922740 1 --2- 172.21.15.134:0/546399101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d929189390 0x55d92954b540 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.213+0000 7f1886922740 1 --2- 172.21.15.134:0/546399101 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291811b0 0x55d929550a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.213+0000 7f1886922740 1 -- 172.21.15.134:0/546399101 >> 172.21.15.134:0/546399101 conn(0x55d92919e460 msgr2=0x55d92955ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:44.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.213+0000 7f1886922740 1 -- 172.21.15.134:0/546399101 shutdown_connections 2024-07-26T11:17:44.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.213+0000 7f1886922740 1 -- 172.21.15.134:0/546399101 wait complete. 2024-07-26T11:17:44.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.214+0000 7f1886922740 1 -- start start 2024-07-26T11:17:44.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.214+0000 7f1886922740 1 Processor -- start 2024-07-26T11:17:44.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.215+0000 7f1886922740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291811b0 0x55d929563250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.215+0000 7f1886922740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d929189390 0x55d929560770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.215+0000 7f1886922740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9291aa070 0x55d929560e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.215+0000 7f1886922740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1870163f50 con 0x55d929189390 2024-07-26T11:17:44.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.215+0000 7f1886922740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1870163880 con 0x55d9291aa070 2024-07-26T11:17:44.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.215+0000 7f1886922740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1870164620 con 0x55d9291811b0 2024-07-26T11:17:44.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.215+0000 7f1886922740 10 client.0 _ll_register_callbacks cb 0x7f186c010420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:44.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.216+0000 7f187e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291811b0 0x55d929563250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:44.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.217+0000 7f187e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291811b0 0x55d929563250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:43596/0 (socket says 172.21.15.134:43596) 2024-07-26T11:17:44.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.217+0000 7f187e000640 1 -- 172.21.15.134:0/2894508633 learned_addr learned my addr 172.21.15.134:0/2894508633 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:44.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.217+0000 7f187e000640 1 --2- 172.21.15.134:0/2894508633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9291aa070 0x55d929560e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:44.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.218+0000 7f187e000640 1 -- 172.21.15.134:0/2894508633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9291aa070 msgr2=0x55d929560e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:44.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.218+0000 7f187e000640 1 --2- 172.21.15.134:0/2894508633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9291aa070 0x55d929560e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.218+0000 7f187e000640 1 -- 172.21.15.134:0/2894508633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d929189390 msgr2=0x55d929560770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:44.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.218+0000 7f187e000640 1 --2- 172.21.15.134:0/2894508633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d929189390 0x55d929560770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.218+0000 7f187e000640 1 -- 172.21.15.134:0/2894508633 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9295615a0 con 0x55d9291811b0 2024-07-26T11:17:44.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.220+0000 7f187e000640 1 --2- 172.21.15.134:0/2894508633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291811b0 0x55d929563250 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1878068cd0 tx=0x7f1878069240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a5e37d995d1308a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:44.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.221+0000 7f185aa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:17:44.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.221+0000 7f185aa00640 1 -- 172.21.15.134:0/2894508633 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18780a7020 con 0x55d9291811b0 2024-07-26T11:17:44.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.221+0000 7f185aa00640 1 -- 172.21.15.134:0/2894508633 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f18780b9040 con 0x55d9291811b0 2024-07-26T11:17:44.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.222+0000 7f1886922740 1 -- 172.21.15.134:0/2894508633 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f186c01c5a0 con 0x55d9291811b0 2024-07-26T11:17:44.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.223+0000 7f185aa00640 1 -- 172.21.15.134:0/2894508633 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f18780a7020 con 0x55d9291811b0 2024-07-26T11:17:44.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.223+0000 7f1886922740 10 client.25849 fetch_fsmap learned FSMap version 31 2024-07-26T11:17:44.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.224+0000 7f1886922740 1 -- 172.21.15.134:0/2894508633 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d9295d50f0 con 0x55d9291811b0 2024-07-26T11:17:44.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.224+0000 7f185aa00640 1 -- 172.21.15.134:0/2894508633 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f186c01c5a0 con 0x55d9291811b0 2024-07-26T11:17:44.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.224+0000 7f185aa00640 1 -- 172.21.15.134:0/2894508633 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f18780a2040 con 0x55d9291811b0 2024-07-26T11:17:44.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.225+0000 7f1886922740 10 client.25849 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:17:44.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.225+0000 7f1886922740 10 client.25849 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T11:17:44.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.225+0000 7f1886922740 20 client.25849 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:44.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.225+0000 7f1886922740 1 --2- 172.21.15.134:0/2894508633 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55d92913dda0 0x55d9295d7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:44.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.225+0000 7f1886922740 4 client.25849 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:17:44.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.225+0000 7f1886922740 1 -- 172.21.15.134:0/2894508633 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d9294c35c0 con 0x55d92913dda0 2024-07-26T11:17:44.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.225+0000 7f187cc00640 1 --2- 172.21.15.134:0/2894508633 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55d92913dda0 0x55d9295d7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:44.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.226+0000 7f187cc00640 1 --2- 172.21.15.134:0/2894508633 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55d92913dda0 0x55d9295d7390 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1b6ae0a19dd8329b server_cookie=2091c4fa01ef82f9 in_seq=0 out_seq=0 2024-07-26T11:17:44.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.226+0000 7f185aa00640 10 client.25849 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:17:44.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.226+0000 7f185aa00640 1 -- 172.21.15.134:0/2894508633 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d9294c35c0 con 0x55d92913dda0 2024-07-26T11:17:44.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.226+0000 7f185aa00640 10 client.25849 handle_command_reply: tid=0 2024-07-26T11:17:44.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.226+0000 7f1886922740 1 client.25849 shutdown 2024-07-26T11:17:44.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.228+0000 7f1886922740 1 -- 172.21.15.134:0/2894508633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291811b0 msgr2=0x55d929563250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:44.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.228+0000 7f1886922740 1 --2- 172.21.15.134:0/2894508633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291811b0 0x55d929563250 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1878068cd0 tx=0x7f1878069240 comp rx=0 tx=0).stop 2024-07-26T11:17:44.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.228+0000 7f1886922740 1 -- 172.21.15.134:0/2894508633 shutdown_connections 2024-07-26T11:17:44.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.228+0000 7f1886922740 1 --2- 172.21.15.134:0/2894508633 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55d92913dda0 0x55d9295d7390 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.228+0000 7f1886922740 1 --2- 172.21.15.134:0/2894508633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d9291aa070 0x55d929560e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.229+0000 7f185aa00640 0 client.25849 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:17:44.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.229+0000 7f1886922740 1 --2- 172.21.15.134:0/2894508633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d929189390 0x55d929560770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.229+0000 7f1886922740 1 --2- 172.21.15.134:0/2894508633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d9291811b0 0x55d929563250 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.229+0000 7f1886922740 1 -- 172.21.15.134:0/2894508633 >> 172.21.15.134:0/2894508633 conn(0x55d92919e460 msgr2=0x55d929566250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:44.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.229+0000 7f1886922740 1 -- 172.21.15.134:0/2894508633 shutdown_connections 2024-07-26T11:17:44.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.229+0000 7f1886922740 1 -- 172.21.15.134:0/2894508633 wait complete. 2024-07-26T11:17:44.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.230+0000 7f1886922740 20 client.25849 trim_cache size 0 max 16384 2024-07-26T11:17:44.233 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:17:44.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.232+0000 7f187ea00640 1 -- 172.21.15.134:0/24602132 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f18600b3ed0 msgr2=0x7f18600b63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:44.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.232+0000 7f187ea00640 1 --2- 172.21.15.134:0/24602132 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f18600b3ed0 0x7f18600b63e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55d92954dc90 tx=0x7f1868000bc0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.232+0000 7f187ea00640 1 -- 172.21.15.134:0/24602132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18700a50f0 msgr2=0x7f1870143fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:44.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.232+0000 7f187ea00640 1 --2- 172.21.15.134:0/24602132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18700a50f0 0x7f1870143fd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f186800d700 tx=0x7f186800dbc0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.233+0000 7f187ea00640 1 -- 172.21.15.134:0/24602132 shutdown_connections 2024-07-26T11:17:44.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.233+0000 7f187ea00640 1 --2- 172.21.15.134:0/24602132 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f18600b3ed0 0x7f18600b63e0 secure :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55d92954dc90 tx=0x7f1868000bc0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.233+0000 7f187ea00640 1 --2- 172.21.15.134:0/24602132 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f18700ab1b0 0x7f1870149810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.233+0000 7f187ea00640 1 --2- 172.21.15.134:0/24602132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18700aa860 0x7f1870144570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.233+0000 7f187ea00640 1 --2- 172.21.15.134:0/24602132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f18700a50f0 0x7f1870143fd0 secure :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f186800d700 tx=0x7f186800dbc0 comp rx=0 tx=0).stop 2024-07-26T11:17:44.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.233+0000 7f187ea00640 1 -- 172.21.15.134:0/24602132 >> 172.21.15.134:0/24602132 conn(0x7f187001cc70 msgr2=0x7f18700a39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:44.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.237+0000 7f187ea00640 1 -- 172.21.15.134:0/24602132 shutdown_connections 2024-07-26T11:17:44.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:44.237+0000 7f187ea00640 1 -- 172.21.15.134:0/24602132 wait complete. 2024-07-26T11:17:45.447 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:45 smithi134 ceph-mon[31966]: pgmap v533: 129 pgs: 129 active+clean; 6.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 300 MiB/s wr, 92 op/s 2024-07-26T11:17:45.606 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-26T11:17:45.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:45 smithi189 ceph-mon[36790]: pgmap v533: 129 pgs: 129 active+clean; 6.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 300 MiB/s wr, 92 op/s 2024-07-26T11:17:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:45 smithi193 ceph-mon[37473]: pgmap v533: 129 pgs: 129 active+clean; 6.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 300 MiB/s wr, 92 op/s 2024-07-26T11:17:46.360 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:17:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:47 smithi134 ceph-mon[31966]: pgmap v534: 129 pgs: 129 active+clean; 7.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 307 MiB/s wr, 91 op/s 2024-07-26T11:17:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:47 smithi189 ceph-mon[36790]: pgmap v534: 129 pgs: 129 active+clean; 7.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 307 MiB/s wr, 91 op/s 2024-07-26T11:17:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:47 smithi193 ceph-mon[37473]: pgmap v534: 129 pgs: 129 active+clean; 7.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 307 MiB/s wr, 91 op/s 2024-07-26T11:17:49.545 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:49 smithi134 ceph-mon[31966]: pgmap v535: 129 pgs: 129 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 308 MiB/s wr, 93 op/s 2024-07-26T11:17:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:49 smithi189 ceph-mon[36790]: pgmap v535: 129 pgs: 129 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 308 MiB/s wr, 93 op/s 2024-07-26T11:17:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:49 smithi193 ceph-mon[37473]: pgmap v535: 129 pgs: 129 active+clean; 7.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 308 MiB/s wr, 93 op/s 2024-07-26T11:17:49.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.842+0000 7fe007200640 1 -- 172.21.15.134:0/2316448382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe00810fc60 msgr2=0x7fe008116490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:49.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.842+0000 7fe007200640 1 --2- 172.21.15.134:0/2316448382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe00810fc60 0x7fe008116490 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe000009ca0 tx=0x7fe00002a8c0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.842+0000 7fe007200640 1 -- 172.21.15.134:0/2316448382 shutdown_connections 2024-07-26T11:17:49.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.842+0000 7fe007200640 1 --2- 172.21.15.134:0/2316448382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe00810fc60 0x7fe008116490 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.842+0000 7fe007200640 1 --2- 172.21.15.134:0/2316448382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe00810f310 0x7fe00810f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.842+0000 7fe007200640 1 --2- 172.21.15.134:0/2316448382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe0081099d0 0x7fe008109de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.842+0000 7fe007200640 1 -- 172.21.15.134:0/2316448382 >> 172.21.15.134:0/2316448382 conn(0x7fe008078fb0 msgr2=0x7fe008107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:49.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.842+0000 7fe007200640 1 -- 172.21.15.134:0/2316448382 shutdown_connections 2024-07-26T11:17:49.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.842+0000 7fe007200640 1 -- 172.21.15.134:0/2316448382 wait complete. 2024-07-26T11:17:49.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.844+0000 7fe007200640 1 Processor -- start 2024-07-26T11:17:49.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.844+0000 7fe007200640 1 -- start start 2024-07-26T11:17:49.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.844+0000 7fe007200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0081099d0 0x7fe0081a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.844+0000 7fe007200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe00810f310 0x7fe0081a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.844+0000 7fe007200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe00810fc60 0x7fe0081a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.844+0000 7fe007200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe008119970 con 0x7fe00810fc60 2024-07-26T11:17:49.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.844+0000 7fe007200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe0081197f0 con 0x7fe0081099d0 2024-07-26T11:17:49.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.844+0000 7fe007200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe008119af0 con 0x7fe00810f310 2024-07-26T11:17:49.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.845+0000 7fe005400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe00810f310 0x7fe0081a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:49.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.845+0000 7fe005400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe00810f310 0x7fe0081a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:43608/0 (socket says 172.21.15.134:43608) 2024-07-26T11:17:49.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.845+0000 7fe005400640 1 -- 172.21.15.134:0/3736699193 learned_addr learned my addr 172.21.15.134:0/3736699193 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:49.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.845+0000 7fe005e00640 1 --2- 172.21.15.134:0/3736699193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0081099d0 0x7fe0081a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:49.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.845+0000 7fe006800640 1 --2- 172.21.15.134:0/3736699193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe00810fc60 0x7fe0081a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:49.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.845+0000 7fe005400640 1 -- 172.21.15.134:0/3736699193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0081099d0 msgr2=0x7fe0081a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:49.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.845+0000 7fe005400640 1 --2- 172.21.15.134:0/3736699193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0081099d0 0x7fe0081a0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.845+0000 7fe005400640 1 -- 172.21.15.134:0/3736699193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe00810fc60 msgr2=0x7fe0081a5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:49.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.845+0000 7fe005400640 1 --2- 172.21.15.134:0/3736699193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe00810fc60 0x7fe0081a5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.845+0000 7fe005400640 1 -- 172.21.15.134:0/3736699193 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0081a61f0 con 0x7fe00810f310 2024-07-26T11:17:49.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.845+0000 7fe005400640 1 --2- 172.21.15.134:0/3736699193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe00810f310 0x7fe0081a0820 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdff400d750 tx=0x7fdff400dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:49.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.846+0000 7fdffea00640 1 -- 172.21.15.134:0/3736699193 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4017070 con 0x7fe00810f310 2024-07-26T11:17:49.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.846+0000 7fdffea00640 1 -- 172.21.15.134:0/3736699193 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdff40054d0 con 0x7fe00810f310 2024-07-26T11:17:49.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.846+0000 7fe007200640 1 -- 172.21.15.134:0/3736699193 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0081a64e0 con 0x7fe00810f310 2024-07-26T11:17:49.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.846+0000 7fdffea00640 1 -- 172.21.15.134:0/3736699193 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4004c30 con 0x7fe00810f310 2024-07-26T11:17:49.849 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.846+0000 7fe007200640 1 -- 172.21.15.134:0/3736699193 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0081acc40 con 0x7fe00810f310 2024-07-26T11:17:49.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.849+0000 7fdffea00640 1 -- 172.21.15.134:0/3736699193 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fdff4004dd0 con 0x7fe00810f310 2024-07-26T11:17:49.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.850+0000 7fe013a5d740 1 Processor -- start 2024-07-26T11:17:49.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.850+0000 7fe013a5d740 1 -- start start 2024-07-26T11:17:49.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.850+0000 7fe013a5d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064cd070 0x559206872030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.850+0000 7fe013a5d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x559206872570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.850+0000 7fe013a5d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064ac390 0x559206872ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.850+0000 7fe013a5d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5592067e8bf0 con 0x5592064a41b0 2024-07-26T11:17:49.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.850+0000 7fe013a5d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5592067d80f0 con 0x5592064cd070 2024-07-26T11:17:49.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.850+0000 7fe013a5d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5592067dc130 con 0x5592064ac390 2024-07-26T11:17:49.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.851+0000 7fe006800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064cd070 0x559206872030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:49.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.851+0000 7fe006800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064cd070 0x559206872030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42248/0 (socket says 172.21.15.134:42248) 2024-07-26T11:17:49.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.851+0000 7fe006800640 1 -- 172.21.15.134:0/4254306805 learned_addr learned my addr 172.21.15.134:0/4254306805 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:49.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.851+0000 7fe005e00640 1 --2- 172.21.15.134:0/4254306805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x559206872570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.851+0000 7fe005400640 1 --2- 172.21.15.134:0/4254306805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064ac390 0x559206872ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.851+0000 7fe006800640 1 -- 172.21.15.134:0/4254306805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064ac390 msgr2=0x559206872ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.851+0000 7fe006800640 1 --2- 172.21.15.134:0/4254306805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064ac390 0x559206872ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.851+0000 7fe006800640 1 -- 172.21.15.134:0/4254306805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 msgr2=0x559206872570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.851+0000 7fe006800640 1 --2- 172.21.15.134:0/4254306805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x559206872570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.851+0000 7fe006800640 1 -- 172.21.15.134:0/4254306805 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559206538650 con 0x5592064cd070 2024-07-26T11:17:49.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.851+0000 7fdffea00640 1 --2- 172.21.15.134:0/3736699193 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdfec0b3d40 0x7fdfec0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.852+0000 7fe005e00640 1 --2- 172.21.15.134:0/3736699193 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdfec0b3d40 0x7fdfec0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:49.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.852+0000 7fe006800640 1 --2- 172.21.15.134:0/4254306805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064cd070 0x559206872030 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe000005e30 tx=0x7fe000009800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d614dca88ef56e4a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:49.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.853+0000 7fdffea00640 1 -- 172.21.15.134:0/3736699193 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fdff40e3d90 con 0x7fe00810f310 2024-07-26T11:17:49.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.853+0000 7fdff3400640 1 -- 172.21.15.134:0/4254306805 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe000031cf0 con 0x5592064cd070 2024-07-26T11:17:49.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.853+0000 7fdff3400640 1 -- 172.21.15.134:0/4254306805 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe000036040 con 0x5592064cd070 2024-07-26T11:17:49.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.853+0000 7fdff3400640 1 -- 172.21.15.134:0/4254306805 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0000042a0 con 0x5592064cd070 2024-07-26T11:17:49.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.854+0000 7fe013a5d740 1 -- 172.21.15.134:0/4254306805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064cd070 msgr2=0x559206872030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:49.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.854+0000 7fe013a5d740 1 --2- 172.21.15.134:0/4254306805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064cd070 0x559206872030 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe000005e30 tx=0x7fe000009800 comp rx=0 tx=0).stop 2024-07-26T11:17:49.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.854+0000 7fe005e00640 1 --2- 172.21.15.134:0/3736699193 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdfec0b3d40 0x7fdfec0b6250 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x559206870cb0 tx=0x7fdff800b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:49.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.854+0000 7fe013a5d740 1 -- 172.21.15.134:0/4254306805 shutdown_connections 2024-07-26T11:17:49.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.854+0000 7fe013a5d740 1 --2- 172.21.15.134:0/4254306805 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064ac390 0x559206872ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.854+0000 7fe013a5d740 1 --2- 172.21.15.134:0/4254306805 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x559206872570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.854+0000 7fe013a5d740 1 --2- 172.21.15.134:0/4254306805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064cd070 0x559206872030 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.854+0000 7fe013a5d740 1 -- 172.21.15.134:0/4254306805 >> 172.21.15.134:0/4254306805 conn(0x5592064c1460 msgr2=0x55920686ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:49.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.855+0000 7fe013a5d740 1 -- 172.21.15.134:0/4254306805 shutdown_connections 2024-07-26T11:17:49.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.855+0000 7fe013a5d740 1 -- 172.21.15.134:0/4254306805 wait complete. 2024-07-26T11:17:49.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.855+0000 7fe013a5d740 1 -- start start 2024-07-26T11:17:49.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.855+0000 7fe013a5d740 1 Processor -- start 2024-07-26T11:17:49.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.856+0000 7fe013a5d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x55920687d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.856+0000 7fe013a5d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064ac390 0x55920687db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.856+0000 7fe013a5d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064cd070 0x55920687e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.856+0000 7fe013a5d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55920663b700 con 0x5592064a41b0 2024-07-26T11:17:49.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.856+0000 7fe013a5d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5592067746c0 con 0x5592064ac390 2024-07-26T11:17:49.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.856+0000 7fe013a5d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5592068807b0 con 0x5592064cd070 2024-07-26T11:17:49.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.856+0000 7fe013a5d740 10 client.0 _ll_register_callbacks cb 0x559206784170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:49.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.856+0000 7fe006800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064cd070 0x55920687e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:49.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.856+0000 7fe005400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064ac390 0x55920687db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:49.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.856+0000 7fe006800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064cd070 0x55920687e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:36674/0 (socket says 172.21.15.134:36674) 2024-07-26T11:17:49.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.856+0000 7fe006800640 1 -- 172.21.15.134:0/2236139365 learned_addr learned my addr 172.21.15.134:0/2236139365 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:49.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.856+0000 7fe005e00640 1 --2- 172.21.15.134:0/2236139365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x55920687d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:49.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.857+0000 7fe005400640 1 -- 172.21.15.134:0/2236139365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064cd070 msgr2=0x55920687e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:49.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.857+0000 7fe005400640 1 --2- 172.21.15.134:0/2236139365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064cd070 0x55920687e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.857+0000 7fe005400640 1 -- 172.21.15.134:0/2236139365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 msgr2=0x55920687d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:49.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.857+0000 7fe005400640 1 --2- 172.21.15.134:0/2236139365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x55920687d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.857+0000 7fe005400640 1 -- 172.21.15.134:0/2236139365 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55920678b860 con 0x5592064ac390 2024-07-26T11:17:49.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.857+0000 7fe005e00640 1 --2- 172.21.15.134:0/2236139365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x55920687d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:17:49.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.857+0000 7fe005400640 1 --2- 172.21.15.134:0/2236139365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064ac390 0x55920687db50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdff40e1100 tx=0x7fdff40a9f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d964a5f9581102a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:49.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.857+0000 7fdfe7e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:17:49.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.857+0000 7fdfe7e00640 1 -- 172.21.15.134:0/2236139365 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4020070 con 0x5592064ac390 2024-07-26T11:17:49.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.857+0000 7fe013a5d740 1 -- 172.21.15.134:0/2236139365 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe000036040 con 0x5592064ac390 2024-07-26T11:17:49.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.857+0000 7fdfe7e00640 1 -- 172.21.15.134:0/2236139365 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fdff40a6850 con 0x5592064ac390 2024-07-26T11:17:49.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.858+0000 7fdfe7e00640 1 -- 172.21.15.134:0/2236139365 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdff40aacc0 con 0x5592064ac390 2024-07-26T11:17:49.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.858+0000 7fe013a5d740 10 client.25859 fetch_fsmap learned FSMap version 31 2024-07-26T11:17:49.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.858+0000 7fe013a5d740 1 -- 172.21.15.134:0/2236139365 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55920688c3b0 con 0x5592064ac390 2024-07-26T11:17:49.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.859+0000 7fdfe7e00640 1 -- 172.21.15.134:0/2236139365 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe000036040 con 0x5592064ac390 2024-07-26T11:17:49.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.859+0000 7fdfe7e00640 1 -- 172.21.15.134:0/2236139365 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7fdff40a4ce0 con 0x5592064ac390 2024-07-26T11:17:49.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.861+0000 7fe013a5d740 10 client.25859 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:17:49.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.861+0000 7fe013a5d740 10 client.25859 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T11:17:49.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.861+0000 7fe013a5d740 20 client.25859 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:49.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.861+0000 7fe013a5d740 1 --2- 172.21.15.134:0/2236139365 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55920648b9f0 0x55920688f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.861+0000 7fe013a5d740 4 client.25859 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:49.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.861+0000 7fe013a5d740 1 -- 172.21.15.134:0/2236139365 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592067eab40 con 0x55920648b9f0 2024-07-26T11:17:49.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.861+0000 7fe005e00640 1 --2- 172.21.15.134:0/2236139365 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55920648b9f0 0x55920688f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:49.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.863+0000 7fe005e00640 1 --2- 172.21.15.134:0/2236139365 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55920648b9f0 0x55920688f880 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f3e4b53d08de66f1 server_cookie=58acb4fe828bee9f in_seq=0 out_seq=0 2024-07-26T11:17:49.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.863+0000 7fdfe7e00640 10 client.25859 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:17:49.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.864+0000 7fdfe7e00640 1 -- 172.21.15.134:0/2236139365 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5592067eab40 con 0x55920648b9f0 2024-07-26T11:17:49.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.864+0000 7fdfe7e00640 10 client.25859 handle_command_reply: tid=0 2024-07-26T11:17:49.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.864+0000 7fe013a5d740 1 client.25859 shutdown 2024-07-26T11:17:49.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.864+0000 7fe013a5d740 1 -- 172.21.15.134:0/2236139365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064ac390 msgr2=0x55920687db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:49.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.864+0000 7fe013a5d740 1 --2- 172.21.15.134:0/2236139365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064ac390 0x55920687db50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdff40e1100 tx=0x7fdff40a9f10 comp rx=0 tx=0).stop 2024-07-26T11:17:49.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.864+0000 7fe013a5d740 1 -- 172.21.15.134:0/2236139365 shutdown_connections 2024-07-26T11:17:49.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.864+0000 7fe013a5d740 1 --2- 172.21.15.134:0/2236139365 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55920648b9f0 0x55920688f880 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.864+0000 7fe013a5d740 1 --2- 172.21.15.134:0/2236139365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064cd070 0x55920687e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.864+0000 7fe013a5d740 1 --2- 172.21.15.134:0/2236139365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064ac390 0x55920687db50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.864+0000 7fe013a5d740 1 --2- 172.21.15.134:0/2236139365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x55920687d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.864+0000 7fdfe7e00640 0 client.25859 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:17:49.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.864+0000 7fe013a5d740 1 -- 172.21.15.134:0/2236139365 >> 172.21.15.134:0/2236139365 conn(0x5592064c1460 msgr2=0x5592068751c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:49.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.864+0000 7fe013a5d740 1 -- 172.21.15.134:0/2236139365 shutdown_connections 2024-07-26T11:17:49.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.865+0000 7fe013a5d740 1 -- 172.21.15.134:0/2236139365 wait complete. 2024-07-26T11:17:49.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.865+0000 7fe013a5d740 20 client.25859 trim_cache size 0 max 16384 2024-07-26T11:17:49.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.889+0000 7fe013a5d740 1 Processor -- start 2024-07-26T11:17:49.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.889+0000 7fe013a5d740 1 -- start start 2024-07-26T11:17:49.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.890+0000 7fe013a5d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x55920687f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.890+0000 7fe013a5d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064ac390 0x55920686e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.890+0000 7fe013a5d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064cd070 0x559206873770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.890+0000 7fe013a5d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe0081b6cf0 con 0x5592064a41b0 2024-07-26T11:17:49.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.890+0000 7fe013a5d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe0081b8860 con 0x5592064cd070 2024-07-26T11:17:49.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.890+0000 7fe013a5d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe0081b73c0 con 0x5592064ac390 2024-07-26T11:17:49.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.891+0000 7fe005e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x55920687f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:49.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.891+0000 7fe005400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064ac390 0x55920686e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:49.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.891+0000 7fe005400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064ac390 0x55920686e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:36678/0 (socket says 172.21.15.134:36678) 2024-07-26T11:17:49.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.891+0000 7fe005400640 1 -- 172.21.15.134:0/3223181844 learned_addr learned my addr 172.21.15.134:0/3223181844 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:49.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.891+0000 7fe006800640 1 --2- 172.21.15.134:0/3223181844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064cd070 0x559206873770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:49.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.891+0000 7fe005e00640 1 -- 172.21.15.134:0/3223181844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064ac390 msgr2=0x55920686e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:49.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.891+0000 7fe005e00640 1 --2- 172.21.15.134:0/3223181844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064ac390 0x55920686e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.891+0000 7fe005e00640 1 -- 172.21.15.134:0/3223181844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064cd070 msgr2=0x559206873770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:49.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.891+0000 7fe005e00640 1 --2- 172.21.15.134:0/3223181844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064cd070 0x559206873770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.891+0000 7fe005e00640 1 -- 172.21.15.134:0/3223181844 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55920687bb10 con 0x5592064a41b0 2024-07-26T11:17:49.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.892+0000 7fe005e00640 1 --2- 172.21.15.134:0/3223181844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x55920687f4f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55920688ffb0 tx=0x7fdff8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2859c27a187751f4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:49.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.892+0000 7fdff2a00640 1 -- 172.21.15.134:0/3223181844 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff80285f0 con 0x5592064a41b0 2024-07-26T11:17:49.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.892+0000 7fdff2a00640 1 -- 172.21.15.134:0/3223181844 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdff8028c40 con 0x5592064a41b0 2024-07-26T11:17:49.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.892+0000 7fdff2a00640 1 -- 172.21.15.134:0/3223181844 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff801fd70 con 0x5592064a41b0 2024-07-26T11:17:49.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.894+0000 7fe013a5d740 1 -- 172.21.15.134:0/3223181844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 msgr2=0x55920687f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:49.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.894+0000 7fe013a5d740 1 --2- 172.21.15.134:0/3223181844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x55920687f4f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55920688ffb0 tx=0x7fdff8015000 comp rx=0 tx=0).stop 2024-07-26T11:17:49.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.900+0000 7fe013a5d740 1 -- 172.21.15.134:0/3223181844 shutdown_connections 2024-07-26T11:17:49.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.900+0000 7fe013a5d740 1 --2- 172.21.15.134:0/3223181844 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064ac390 0x55920686e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.900+0000 7fe013a5d740 1 --2- 172.21.15.134:0/3223181844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064cd070 0x559206873770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.900+0000 7fe013a5d740 1 --2- 172.21.15.134:0/3223181844 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x55920687f4f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.900+0000 7fe013a5d740 1 -- 172.21.15.134:0/3223181844 >> 172.21.15.134:0/3223181844 conn(0x5592064c1460 msgr2=0x559206873060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:49.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.902+0000 7fe013a5d740 1 -- 172.21.15.134:0/3223181844 shutdown_connections 2024-07-26T11:17:49.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.902+0000 7fe013a5d740 1 -- 172.21.15.134:0/3223181844 wait complete. 2024-07-26T11:17:49.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.903+0000 7fe013a5d740 1 -- start start 2024-07-26T11:17:49.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.904+0000 7fe013a5d740 1 Processor -- start 2024-07-26T11:17:49.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.908+0000 7fe013a5d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x5592068819c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.908+0000 7fe013a5d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064ac390 0x559206881f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.908+0000 7fe013a5d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064cd070 0x559206882440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.908+0000 7fe013a5d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe0081b8160 con 0x5592064a41b0 2024-07-26T11:17:49.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.908+0000 7fe013a5d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe0081b7a90 con 0x5592064ac390 2024-07-26T11:17:49.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.908+0000 7fe013a5d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe0081b4cd0 con 0x5592064cd070 2024-07-26T11:17:49.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.908+0000 7fe013a5d740 10 client.0 _ll_register_callbacks cb 0x7fdff40a6550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:49.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.908+0000 7fe005400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064ac390 0x559206881f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:49.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.908+0000 7fe005400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064ac390 0x559206881f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38016/0 (socket says 172.21.15.134:38016) 2024-07-26T11:17:49.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.908+0000 7fe005400640 1 -- 172.21.15.134:0/847657959 learned_addr learned my addr 172.21.15.134:0/847657959 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:49.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.909+0000 7fe005400640 1 -- 172.21.15.134:0/847657959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064cd070 msgr2=0x559206882440 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:17:49.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.909+0000 7fe005400640 1 --2- 172.21.15.134:0/847657959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064cd070 0x559206882440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.909+0000 7fe005400640 1 -- 172.21.15.134:0/847657959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 msgr2=0x5592068819c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:17:49.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.910+0000 7fe005400640 1 --2- 172.21.15.134:0/847657959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x5592068819c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.910+0000 7fe005400640 1 -- 172.21.15.134:0/847657959 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592068f70e0 con 0x5592064ac390 2024-07-26T11:17:49.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.910+0000 7fe005400640 1 --2- 172.21.15.134:0/847657959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064ac390 0x559206881f00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdff40b3140 tx=0x7fdff40a6a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=698e44ddd05ef50b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:49.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.910+0000 7fdfe7e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:17:49.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.911+0000 7fdfe7e00640 1 -- 172.21.15.134:0/847657959 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff40c3060 con 0x5592064ac390 2024-07-26T11:17:49.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.912+0000 7fdfe7e00640 1 -- 172.21.15.134:0/847657959 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fdff40a9a30 con 0x5592064ac390 2024-07-26T11:17:49.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.912+0000 7fe013a5d740 1 -- 172.21.15.134:0/847657959 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdff8028c40 con 0x5592064ac390 2024-07-26T11:17:49.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.912+0000 7fdfe7e00640 1 -- 172.21.15.134:0/847657959 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fdff4014070 con 0x5592064ac390 2024-07-26T11:17:49.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.913+0000 7fe013a5d740 10 client.25865 fetch_fsmap learned FSMap version 31 2024-07-26T11:17:49.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.913+0000 7fe013a5d740 1 -- 172.21.15.134:0/847657959 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5592068f8390 con 0x5592064ac390 2024-07-26T11:17:49.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.914+0000 7fdfe7e00640 1 -- 172.21.15.134:0/847657959 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdff8028c40 con 0x5592064ac390 2024-07-26T11:17:49.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.914+0000 7fdfe7e00640 1 -- 172.21.15.134:0/847657959 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7fdff40a4ce0 con 0x5592064ac390 2024-07-26T11:17:49.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.914+0000 7fe013a5d740 10 client.25865 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:17:49.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.914+0000 7fe013a5d740 10 client.25865 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T11:17:49.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.914+0000 7fe013a5d740 20 client.25865 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:49.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.914+0000 7fe013a5d740 1 --2- 172.21.15.134:0/847657959 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x559206460da0 0x5592068faba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:49.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.914+0000 7fe013a5d740 4 client.25865 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:17:49.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.914+0000 7fe013a5d740 1 -- 172.21.15.134:0/847657959 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5592067e65e0 con 0x559206460da0 2024-07-26T11:17:49.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.917+0000 7fe005e00640 1 --2- 172.21.15.134:0/847657959 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x559206460da0 0x5592068faba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:49.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.919+0000 7fe005e00640 1 --2- 172.21.15.134:0/847657959 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x559206460da0 0x5592068faba0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=872ab6ebedf5fa69 server_cookie=b36df722a342dcb6 in_seq=0 out_seq=0 2024-07-26T11:17:49.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.924+0000 7fdfe7e00640 10 client.25865 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:17:49.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.929+0000 7fdfe7e00640 1 -- 172.21.15.134:0/847657959 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5592067e65e0 con 0x559206460da0 2024-07-26T11:17:49.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.929+0000 7fdfe7e00640 10 client.25865 handle_command_reply: tid=0 2024-07-26T11:17:49.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.929+0000 7fe013a5d740 1 client.25865 shutdown 2024-07-26T11:17:49.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.930+0000 7fe013a5d740 1 -- 172.21.15.134:0/847657959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064ac390 msgr2=0x559206881f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:49.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.930+0000 7fe013a5d740 1 --2- 172.21.15.134:0/847657959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064ac390 0x559206881f00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdff40b3140 tx=0x7fdff40a6a80 comp rx=0 tx=0).stop 2024-07-26T11:17:49.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.930+0000 7fe013a5d740 1 -- 172.21.15.134:0/847657959 shutdown_connections 2024-07-26T11:17:49.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.930+0000 7fe013a5d740 1 --2- 172.21.15.134:0/847657959 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x559206460da0 0x5592068faba0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.930+0000 7fe013a5d740 1 --2- 172.21.15.134:0/847657959 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5592064cd070 0x559206882440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.930+0000 7fe013a5d740 1 --2- 172.21.15.134:0/847657959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5592064ac390 0x559206881f00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.930+0000 7fe013a5d740 1 --2- 172.21.15.134:0/847657959 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5592064a41b0 0x5592068819c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.930+0000 7fe013a5d740 1 -- 172.21.15.134:0/847657959 >> 172.21.15.134:0/847657959 conn(0x5592064c1460 msgr2=0x55920687f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:49.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.930+0000 7fdfe7e00640 0 client.25865 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:17:49.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.931+0000 7fe013a5d740 1 -- 172.21.15.134:0/847657959 shutdown_connections 2024-07-26T11:17:49.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.932+0000 7fe013a5d740 1 -- 172.21.15.134:0/847657959 wait complete. 2024-07-26T11:17:49.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.933+0000 7fe013a5d740 20 client.25865 trim_cache size 0 max 16384 2024-07-26T11:17:49.936 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:17:49.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.934+0000 7fe007200640 1 -- 172.21.15.134:0/3736699193 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdfec0b3d40 msgr2=0x7fdfec0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:49.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.934+0000 7fe007200640 1 --2- 172.21.15.134:0/3736699193 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdfec0b3d40 0x7fdfec0b6250 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x559206870cb0 tx=0x7fdff800b410 comp rx=0 tx=0).stop 2024-07-26T11:17:49.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.934+0000 7fe007200640 1 -- 172.21.15.134:0/3736699193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe00810f310 msgr2=0x7fe0081a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:49.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.934+0000 7fe007200640 1 --2- 172.21.15.134:0/3736699193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe00810f310 0x7fe0081a0820 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fdff400d750 tx=0x7fdff400dc10 comp rx=0 tx=0).stop 2024-07-26T11:17:49.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.935+0000 7fe007200640 1 -- 172.21.15.134:0/3736699193 shutdown_connections 2024-07-26T11:17:49.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.935+0000 7fe007200640 1 --2- 172.21.15.134:0/3736699193 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdfec0b3d40 0x7fdfec0b6250 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.935+0000 7fe007200640 1 --2- 172.21.15.134:0/3736699193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe00810f310 0x7fe0081a0820 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.935+0000 7fe007200640 1 --2- 172.21.15.134:0/3736699193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe00810fc60 0x7fe0081a5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.935+0000 7fe007200640 1 --2- 172.21.15.134:0/3736699193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0081099d0 0x7fe0081a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:49.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.935+0000 7fe007200640 1 -- 172.21.15.134:0/3736699193 >> 172.21.15.134:0/3736699193 conn(0x7fe008078fb0 msgr2=0x7fe008079560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:49.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.937+0000 7fe007200640 1 -- 172.21.15.134:0/3736699193 shutdown_connections 2024-07-26T11:17:49.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:49.937+0000 7fe007200640 1 -- 172.21.15.134:0/3736699193 wait complete. 2024-07-26T11:17:51.109 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-26T11:17:51.675 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:51 smithi134 ceph-mon[31966]: pgmap v536: 129 pgs: 129 active+clean; 8.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 333 MiB/s wr, 96 op/s 2024-07-26T11:17:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:51 smithi189 ceph-mon[36790]: pgmap v536: 129 pgs: 129 active+clean; 8.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 333 MiB/s wr, 96 op/s 2024-07-26T11:17:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:51 smithi193 ceph-mon[37473]: pgmap v536: 129 pgs: 129 active+clean; 8.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 333 MiB/s wr, 96 op/s 2024-07-26T11:17:51.948 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:17:53.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:53 smithi189 ceph-mon[36790]: pgmap v537: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 281 MiB/s wr, 83 op/s 2024-07-26T11:17:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:53 smithi193 ceph-mon[37473]: pgmap v537: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 281 MiB/s wr, 83 op/s 2024-07-26T11:17:53.822 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:53 smithi134 ceph-mon[31966]: pgmap v537: 129 pgs: 129 active+clean; 9.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 281 MiB/s wr, 83 op/s 2024-07-26T11:17:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.299+0000 7f70a9200640 1 -- 172.21.15.134:0/4112396920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70a4104cb0 msgr2=0x7f70a410d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.299+0000 7f70a9200640 1 --2- 172.21.15.134:0/4112396920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70a4104cb0 0x7f70a410d1f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7090009900 tx=0x7f709002a420 comp rx=0 tx=0).stop 2024-07-26T11:17:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.300+0000 7f70a9200640 1 -- 172.21.15.134:0/4112396920 shutdown_connections 2024-07-26T11:17:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.300+0000 7f70a9200640 1 --2- 172.21.15.134:0/4112396920 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f70a410d7c0 0x7f70a410fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.302 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.300+0000 7f70a9200640 1 --2- 172.21.15.134:0/4112396920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70a4104cb0 0x7f70a410d1f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.300+0000 7f70a9200640 1 --2- 172.21.15.134:0/4112396920 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70a4104360 0x7f70a4104770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.300+0000 7f70a9200640 1 -- 172.21.15.134:0/4112396920 >> 172.21.15.134:0/4112396920 conn(0x7f70a40fde10 msgr2=0x7f70a4100260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.300+0000 7f70a9200640 1 -- 172.21.15.134:0/4112396920 shutdown_connections 2024-07-26T11:17:54.303 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.301+0000 7f70a9200640 1 -- 172.21.15.134:0/4112396920 wait complete. 2024-07-26T11:17:54.304 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.302+0000 7f70a9200640 1 Processor -- start 2024-07-26T11:17:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.303+0000 7f70a9200640 1 -- start start 2024-07-26T11:17:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.303+0000 7f70a9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70a4104360 0x7f70a410a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.303+0000 7f70a9200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f70a410d7c0 0x7f70a410a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.303+0000 7f70a9200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70a410af30 0x7f70a41ac6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.303+0000 7f70a9200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4103680 con 0x7f70a410d7c0 2024-07-26T11:17:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.304+0000 7f70a9200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4103500 con 0x7f70a4104360 2024-07-26T11:17:54.306 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.304+0000 7f70a9200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4103800 con 0x7f70a410af30 2024-07-26T11:17:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.304+0000 7f70a1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70a4104360 0x7f70a410a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.304+0000 7f70a1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70a4104360 0x7f70a410a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38026/0 (socket says 172.21.15.134:38026) 2024-07-26T11:17:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.304+0000 7f70a1e00640 1 -- 172.21.15.134:0/1520506471 learned_addr learned my addr 172.21.15.134:0/1520506471 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:54.307 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.304+0000 7f70a2800640 1 --2- 172.21.15.134:0/1520506471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70a410af30 0x7f70a41ac6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.305+0000 7f70a1400640 1 --2- 172.21.15.134:0/1520506471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f70a410d7c0 0x7f70a410a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.305+0000 7f70a1e00640 1 -- 172.21.15.134:0/1520506471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70a410af30 msgr2=0x7f70a41ac6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.305+0000 7f70a1e00640 1 --2- 172.21.15.134:0/1520506471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70a410af30 0x7f70a41ac6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.305+0000 7f70a1e00640 1 -- 172.21.15.134:0/1520506471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f70a410d7c0 msgr2=0x7f70a410a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.305+0000 7f70a1e00640 1 --2- 172.21.15.134:0/1520506471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f70a410d7c0 0x7f70a410a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.308 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.305+0000 7f70a1e00640 1 -- 172.21.15.134:0/1520506471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a406c810 con 0x7f70a4104360 2024-07-26T11:17:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.306+0000 7f70a1e00640 1 --2- 172.21.15.134:0/1520506471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70a4104360 0x7f70a410a4b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f709400ea20 tx=0x7f709400eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.306+0000 7f709aa00640 1 -- 172.21.15.134:0/1520506471 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709400cd80 con 0x7f70a4104360 2024-07-26T11:17:54.309 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.307+0000 7f709aa00640 1 -- 172.21.15.134:0/1520506471 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f70940108e0 con 0x7f70a4104360 2024-07-26T11:17:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.307+0000 7f70a9200640 1 -- 172.21.15.134:0/1520506471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a406cb00 con 0x7f70a4104360 2024-07-26T11:17:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.308+0000 7f70a9200640 1 -- 172.21.15.134:0/1520506471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a406d070 con 0x7f70a4104360 2024-07-26T11:17:54.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.309+0000 7f709aa00640 1 -- 172.21.15.134:0/1520506471 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70940188c0 con 0x7f70a4104360 2024-07-26T11:17:54.323 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.321+0000 7f70b0249740 1 Processor -- start 2024-07-26T11:17:54.326 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.324+0000 7f70b0249740 1 -- start start 2024-07-26T11:17:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.324+0000 7f70b0249740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7e31b0 0x55f03bb86a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.325+0000 7f70b0249740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7ba2f0 0x55f03bb86f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.325+0000 7f70b0249740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7c24d0 0x55f03bb874c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.325+0000 7f70b0249740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f03bafb270 con 0x55f03b7c24d0 2024-07-26T11:17:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.325+0000 7f70b0249740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f03baea770 con 0x55f03b7ba2f0 2024-07-26T11:17:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.325+0000 7f70b0249740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f03baee7b0 con 0x55f03b7e31b0 2024-07-26T11:17:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.326+0000 7f70a2800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7e31b0 0x55f03bb86a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.326+0000 7f70a2800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7e31b0 0x55f03bb86a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:36718/0 (socket says 172.21.15.134:36718) 2024-07-26T11:17:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.326+0000 7f70a2800640 1 -- 172.21.15.134:0/3685699 learned_addr learned my addr 172.21.15.134:0/3685699 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.327+0000 7f70a1e00640 1 --2- 172.21.15.134:0/3685699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7ba2f0 0x55f03bb86f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.327+0000 7f70a2800640 1 -- 172.21.15.134:0/3685699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7ba2f0 msgr2=0x55f03bb86f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.327+0000 7f70a2800640 1 --2- 172.21.15.134:0/3685699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7ba2f0 0x55f03bb86f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.327+0000 7f70a1400640 1 --2- 172.21.15.134:0/3685699 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7c24d0 0x55f03bb874c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.327+0000 7f70a2800640 1 -- 172.21.15.134:0/3685699 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7c24d0 msgr2=0x55f03bb874c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.327+0000 7f70a2800640 1 --2- 172.21.15.134:0/3685699 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7c24d0 0x55f03bb874c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.327+0000 7f70a2800640 1 -- 172.21.15.134:0/3685699 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f03b84e790 con 0x55f03b7e31b0 2024-07-26T11:17:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.327+0000 7f709aa00640 1 -- 172.21.15.134:0/1520506471 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f7094010430 con 0x7f70a4104360 2024-07-26T11:17:54.334 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.327+0000 7f70a1400640 1 --2- 172.21.15.134:0/3685699 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7c24d0 0x55f03bb874c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:17:54.335 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.328+0000 7f70a2800640 1 --2- 172.21.15.134:0/3685699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7e31b0 0x55f03bb86a40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f709c007e00 tx=0x7f709c00e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5beec53fc1f62966 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:54.335 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.328+0000 7f708f400640 1 -- 172.21.15.134:0/3685699 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c014070 con 0x55f03b7e31b0 2024-07-26T11:17:54.335 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.329+0000 7f708f400640 1 -- 172.21.15.134:0/3685699 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f709c004d20 con 0x55f03b7e31b0 2024-07-26T11:17:54.335 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.329+0000 7f708f400640 1 -- 172.21.15.134:0/3685699 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c014070 con 0x55f03b7e31b0 2024-07-26T11:17:54.335 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.330+0000 7f70b0249740 1 -- 172.21.15.134:0/3685699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7e31b0 msgr2=0x55f03bb86a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:54.335 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.330+0000 7f70b0249740 1 --2- 172.21.15.134:0/3685699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7e31b0 0x55f03bb86a40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f709c007e00 tx=0x7f709c00e720 comp rx=0 tx=0).stop 2024-07-26T11:17:54.335 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.332+0000 7f709aa00640 1 --2- 172.21.15.134:0/1520506471 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f70880b3e40 0x7f70880b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.335 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.332+0000 7f709aa00640 1 -- 172.21.15.134:0/1520506471 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f70940e4560 con 0x7f70a4104360 2024-07-26T11:17:54.336 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.334+0000 7f70a2800640 1 --2- 172.21.15.134:0/1520506471 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f70880b3e40 0x7f70880b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.336 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.335+0000 7f70b0249740 1 -- 172.21.15.134:0/3685699 shutdown_connections 2024-07-26T11:17:54.336 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.335+0000 7f70b0249740 1 --2- 172.21.15.134:0/3685699 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7c24d0 0x55f03bb874c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.336 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.335+0000 7f70b0249740 1 --2- 172.21.15.134:0/3685699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7ba2f0 0x55f03bb86f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.337 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.335+0000 7f70b0249740 1 --2- 172.21.15.134:0/3685699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7e31b0 0x55f03bb86a40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.337 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.335+0000 7f70b0249740 1 -- 172.21.15.134:0/3685699 >> 172.21.15.134:0/3685699 conn(0x55f03b7d7300 msgr2=0x55f03bb84930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:54.338 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.337+0000 7f70a2800640 1 --2- 172.21.15.134:0/1520506471 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f70880b3e40 0x7f70880b6350 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f709c00f010 tx=0x7f709c010f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.339+0000 7f70b0249740 1 -- 172.21.15.134:0/3685699 shutdown_connections 2024-07-26T11:17:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.339+0000 7f70b0249740 1 -- 172.21.15.134:0/3685699 wait complete. 2024-07-26T11:17:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.340+0000 7f70b0249740 1 -- start start 2024-07-26T11:17:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.340+0000 7f70b0249740 1 Processor -- start 2024-07-26T11:17:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.341+0000 7f70b0249740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7ba2f0 0x55f03bb925f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.341+0000 7f70b0249740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7c24d0 0x55f03bb92b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.341+0000 7f70b0249740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7e31b0 0x55f03bbc0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.341+0000 7f70b0249740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f03ba24ab0 con 0x55f03b7e31b0 2024-07-26T11:17:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.341+0000 7f70b0249740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f03ba87d50 con 0x55f03b7c24d0 2024-07-26T11:17:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.341+0000 7f70b0249740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f03bb951c0 con 0x55f03b7ba2f0 2024-07-26T11:17:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.341+0000 7f70b0249740 10 client.0 _ll_register_callbacks cb 0x55f03b7faa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.342+0000 7f70a1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7c24d0 0x55f03bb92b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.343+0000 7f70a1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7c24d0 0x55f03bb92b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38036/0 (socket says 172.21.15.134:38036) 2024-07-26T11:17:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.343+0000 7f70a1400640 1 -- 172.21.15.134:0/2056300908 learned_addr learned my addr 172.21.15.134:0/2056300908 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.343+0000 7f70a1e00640 1 --2- 172.21.15.134:0/2056300908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7ba2f0 0x55f03bb925f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.347 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.343+0000 7f70a2800640 1 --2- 172.21.15.134:0/2056300908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7e31b0 0x55f03bbc0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.348 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.345+0000 7f70a1400640 1 -- 172.21.15.134:0/2056300908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7ba2f0 msgr2=0x55f03bb925f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:54.348 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.345+0000 7f70a1400640 1 --2- 172.21.15.134:0/2056300908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7ba2f0 0x55f03bb925f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.348 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.345+0000 7f70a1400640 1 -- 172.21.15.134:0/2056300908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7e31b0 msgr2=0x55f03bbc0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:54.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.345+0000 7f70a1400640 1 --2- 172.21.15.134:0/2056300908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7e31b0 0x55f03bbc0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.345+0000 7f70a1400640 1 -- 172.21.15.134:0/2056300908 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f03b8c7620 con 0x55f03b7c24d0 2024-07-26T11:17:54.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.346+0000 7f70a1400640 1 --2- 172.21.15.134:0/2056300908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7c24d0 0x55f03bb92b30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7090033040 tx=0x7f7090005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1581e1a5ae30643 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:54.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.346+0000 7f7083e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:17:54.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.346+0000 7f7083e00640 1 -- 172.21.15.134:0/2056300908 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7090031740 con 0x55f03b7c24d0 2024-07-26T11:17:54.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.346+0000 7f7083e00640 1 -- 172.21.15.134:0/2056300908 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f70900318e0 con 0x55f03b7c24d0 2024-07-26T11:17:54.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.346+0000 7f70b0249740 1 -- 172.21.15.134:0/2056300908 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f709c004d20 con 0x55f03b7c24d0 2024-07-26T11:17:54.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.346+0000 7f7083e00640 1 -- 172.21.15.134:0/2056300908 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7090031ae0 con 0x55f03b7c24d0 2024-07-26T11:17:54.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.347+0000 7f70b0249740 10 client.25877 fetch_fsmap learned FSMap version 31 2024-07-26T11:17:54.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.347+0000 7f70b0249740 1 -- 172.21.15.134:0/2056300908 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f03bbc31e0 con 0x55f03b7c24d0 2024-07-26T11:17:54.350 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.347+0000 7f7083e00640 1 -- 172.21.15.134:0/2056300908 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f709c004d20 con 0x55f03b7c24d0 2024-07-26T11:17:54.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.349+0000 7f7083e00640 1 -- 172.21.15.134:0/2056300908 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f7090045430 con 0x55f03b7c24d0 2024-07-26T11:17:54.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.349+0000 7f70b0249740 10 client.25877 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:17:54.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.349+0000 7f70b0249740 10 client.25877 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:17:54.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.349+0000 7f70b0249740 20 client.25877 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:54.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.349+0000 7f70b0249740 1 --2- 172.21.15.134:0/2056300908 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f03b7a1d80 0x55f03bbc6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.349+0000 7f70b0249740 4 client.25877 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:54.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.349+0000 7f70b0249740 1 -- 172.21.15.134:0/2056300908 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f03bafd1c0 con 0x55f03b7a1d80 2024-07-26T11:17:54.354 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.353+0000 7f70a1e00640 1 --2- 172.21.15.134:0/2056300908 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f03b7a1d80 0x55f03bbc6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.355+0000 7f70a1e00640 1 --2- 172.21.15.134:0/2056300908 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f03b7a1d80 0x55f03bbc6850 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60e070005a320b9d server_cookie=14b5540ec66e503b in_seq=0 out_seq=0 2024-07-26T11:17:54.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.356+0000 7f7083e00640 10 client.25877 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:54.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.356+0000 7f7083e00640 1 -- 172.21.15.134:0/2056300908 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f03bafd1c0 con 0x55f03b7a1d80 2024-07-26T11:17:54.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.356+0000 7f7083e00640 10 client.25877 handle_command_reply: tid=0 2024-07-26T11:17:54.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.356+0000 7f70b0249740 1 client.25877 shutdown 2024-07-26T11:17:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.357+0000 7f70b0249740 1 -- 172.21.15.134:0/2056300908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7c24d0 msgr2=0x55f03bb92b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.357+0000 7f70b0249740 1 --2- 172.21.15.134:0/2056300908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7c24d0 0x55f03bb92b30 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7090033040 tx=0x7f7090005ef0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.358+0000 7f70b0249740 1 -- 172.21.15.134:0/2056300908 shutdown_connections 2024-07-26T11:17:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.358+0000 7f70b0249740 1 --2- 172.21.15.134:0/2056300908 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f03b7a1d80 0x55f03bbc6850 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.358+0000 7f70b0249740 1 --2- 172.21.15.134:0/2056300908 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7e31b0 0x55f03bbc0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.358+0000 7f70b0249740 1 --2- 172.21.15.134:0/2056300908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7c24d0 0x55f03bb92b30 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.358+0000 7f70b0249740 1 --2- 172.21.15.134:0/2056300908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7ba2f0 0x55f03bb925f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.358+0000 7f70b0249740 1 -- 172.21.15.134:0/2056300908 >> 172.21.15.134:0/2056300908 conn(0x55f03b7d7300 msgr2=0x55f03bb87a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.358+0000 7f7083e00640 0 client.25877 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.359+0000 7f70b0249740 1 -- 172.21.15.134:0/2056300908 shutdown_connections 2024-07-26T11:17:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.359+0000 7f70b0249740 1 -- 172.21.15.134:0/2056300908 wait complete. 2024-07-26T11:17:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.359+0000 7f70b0249740 20 client.25877 trim_cache size 0 max 16384 2024-07-26T11:17:54.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.391+0000 7f70b0249740 1 Processor -- start 2024-07-26T11:17:54.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.391+0000 7f70b0249740 1 -- start start 2024-07-26T11:17:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.392+0000 7f70b0249740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7ba2f0 0x55f03bb849a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.392+0000 7f70b0249740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7c24d0 0x55f03bb8cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.392+0000 7f70b0249740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7e31b0 0x55f03bb918b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.392+0000 7f70b0249740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f708400ff50 con 0x55f03b7ba2f0 2024-07-26T11:17:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.392+0000 7f70b0249740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7084011ac0 con 0x55f03b7c24d0 2024-07-26T11:17:54.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.392+0000 7f70b0249740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7084010620 con 0x55f03b7e31b0 2024-07-26T11:17:54.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.393+0000 7f70a2800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7e31b0 0x55f03bb918b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.393+0000 7f70a2800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7e31b0 0x55f03bb918b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:36728/0 (socket says 172.21.15.134:36728) 2024-07-26T11:17:54.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.393+0000 7f70a2800640 1 -- 172.21.15.134:0/3881836185 learned_addr learned my addr 172.21.15.134:0/3881836185 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:54.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.393+0000 7f70a1400640 1 --2- 172.21.15.134:0/3881836185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7c24d0 0x55f03bb8cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.394+0000 7f70a2800640 1 -- 172.21.15.134:0/3881836185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7c24d0 msgr2=0x55f03bb8cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:54.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.394+0000 7f70a2800640 1 --2- 172.21.15.134:0/3881836185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7c24d0 0x55f03bb8cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.394+0000 7f70a2800640 1 -- 172.21.15.134:0/3881836185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7ba2f0 msgr2=0x55f03bb849a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:54.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.394+0000 7f70a1e00640 1 --2- 172.21.15.134:0/3881836185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7ba2f0 0x55f03bb849a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.399 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.395+0000 7f70a2800640 1 --2- 172.21.15.134:0/3881836185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7ba2f0 0x55f03bb849a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.399 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.396+0000 7f70a2800640 1 -- 172.21.15.134:0/3881836185 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f03bb936a0 con 0x55f03b7e31b0 2024-07-26T11:17:54.399 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.397+0000 7f70a1400640 1 --2- 172.21.15.134:0/3881836185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7c24d0 0x55f03bb8cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:17:54.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.397+0000 7f70a1e00640 1 --2- 172.21.15.134:0/3881836185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7ba2f0 0x55f03bb849a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:17:54.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.398+0000 7f70a2800640 1 --2- 172.21.15.134:0/3881836185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7e31b0 0x55f03bb918b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f709c000f60 tx=0x7f709c020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa49f6573273ceaa server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:54.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.400+0000 7f708ea00640 1 -- 172.21.15.134:0/3881836185 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c035070 con 0x55f03b7e31b0 2024-07-26T11:17:54.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.400+0000 7f708ea00640 1 -- 172.21.15.134:0/3881836185 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f709c005570 con 0x55f03b7e31b0 2024-07-26T11:17:54.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.400+0000 7f708ea00640 1 -- 172.21.15.134:0/3881836185 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c035070 con 0x55f03b7e31b0 2024-07-26T11:17:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.403+0000 7f70b0249740 1 -- 172.21.15.134:0/3881836185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7e31b0 msgr2=0x55f03bb918b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.403+0000 7f70b0249740 1 --2- 172.21.15.134:0/3881836185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7e31b0 0x55f03bb918b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f709c000f60 tx=0x7f709c020000 comp rx=0 tx=0).stop 2024-07-26T11:17:54.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.404+0000 7f70b0249740 1 -- 172.21.15.134:0/3881836185 shutdown_connections 2024-07-26T11:17:54.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.404+0000 7f70b0249740 1 --2- 172.21.15.134:0/3881836185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7e31b0 0x55f03bb918b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.404+0000 7f70b0249740 1 --2- 172.21.15.134:0/3881836185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7c24d0 0x55f03bb8cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.404+0000 7f70b0249740 1 --2- 172.21.15.134:0/3881836185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7ba2f0 0x55f03bb849a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.404+0000 7f70b0249740 1 -- 172.21.15.134:0/3881836185 >> 172.21.15.134:0/3881836185 conn(0x55f03b7d7300 msgr2=0x55f03bb84290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:54.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.406+0000 7f70b0249740 1 -- 172.21.15.134:0/3881836185 shutdown_connections 2024-07-26T11:17:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.406+0000 7f70b0249740 1 -- 172.21.15.134:0/3881836185 wait complete. 2024-07-26T11:17:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.407+0000 7f70b0249740 1 -- start start 2024-07-26T11:17:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.408+0000 7f70b0249740 1 Processor -- start 2024-07-26T11:17:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.409+0000 7f70b0249740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7ba2f0 0x55f03bbb7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.409+0000 7f70b0249740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7c24d0 0x55f03bbb7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.410+0000 7f70b0249740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7e31b0 0x55f03bc2cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.410+0000 7f70b0249740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f70840113c0 con 0x55f03b7e31b0 2024-07-26T11:17:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.410+0000 7f70b0249740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7084010cf0 con 0x55f03b7ba2f0 2024-07-26T11:17:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.410+0000 7f70b0249740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f708400df30 con 0x55f03b7c24d0 2024-07-26T11:17:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.410+0000 7f70b0249740 10 client.0 _ll_register_callbacks cb 0x7f7094005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:54.413 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.411+0000 7f70a1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7c24d0 0x55f03bbb7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.413 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.411+0000 7f70a1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7c24d0 0x55f03bbb7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:36744/0 (socket says 172.21.15.134:36744) 2024-07-26T11:17:54.413 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.411+0000 7f70a1400640 1 -- 172.21.15.134:0/542974756 learned_addr learned my addr 172.21.15.134:0/542974756 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:54.413 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.411+0000 7f70a1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7ba2f0 0x55f03bbb7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.413 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.411+0000 7f70a2800640 1 --2- 172.21.15.134:0/542974756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7e31b0 0x55f03bc2cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.411+0000 7f70a1400640 1 -- 172.21.15.134:0/542974756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7ba2f0 msgr2=0x55f03bbb7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.411+0000 7f70a1400640 1 --2- 172.21.15.134:0/542974756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7ba2f0 0x55f03bbb7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.411+0000 7f70a1400640 1 -- 172.21.15.134:0/542974756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7e31b0 msgr2=0x55f03bc2cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.412+0000 7f70a1400640 1 --2- 172.21.15.134:0/542974756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7e31b0 0x55f03bc2cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.412+0000 7f70a1400640 1 -- 172.21.15.134:0/542974756 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f03bc2d510 con 0x55f03b7c24d0 2024-07-26T11:17:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.412+0000 7f70a2800640 1 --2- 172.21.15.134:0/542974756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7e31b0 0x55f03bc2cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.412+0000 7f70a1400640 1 --2- 172.21.15.134:0/542974756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7c24d0 0x55f03bbb7eb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f70900096c0 tx=0x7f7090045660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80946ab11d83853 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.412+0000 7f7083e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:17:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.412+0000 7f7083e00640 1 -- 172.21.15.134:0/542974756 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7090045c70 con 0x55f03b7c24d0 2024-07-26T11:17:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.412+0000 7f70b0249740 1 -- 172.21.15.134:0/542974756 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f709c005570 con 0x55f03b7c24d0 2024-07-26T11:17:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.412+0000 7f7083e00640 1 -- 172.21.15.134:0/542974756 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7090031440 con 0x55f03b7c24d0 2024-07-26T11:17:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.413+0000 7f7083e00640 1 -- 172.21.15.134:0/542974756 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7090045e10 con 0x55f03b7c24d0 2024-07-26T11:17:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.415+0000 7f70b0249740 10 client.25882 fetch_fsmap learned FSMap version 31 2024-07-26T11:17:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.415+0000 7f70b0249740 1 -- 172.21.15.134:0/542974756 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f03bc2e550 con 0x55f03b7c24d0 2024-07-26T11:17:54.416 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.415+0000 7f7083e00640 1 -- 172.21.15.134:0/542974756 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f709c005570 con 0x55f03b7c24d0 2024-07-26T11:17:54.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.415+0000 7f7083e00640 1 -- 172.21.15.134:0/542974756 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f7090054cc0 con 0x55f03b7c24d0 2024-07-26T11:17:54.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.415+0000 7f70b0249740 10 client.25882 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:17:54.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.415+0000 7f70b0249740 10 client.25882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:17:54.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.415+0000 7f70b0249740 20 client.25882 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:54.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.415+0000 7f70b0249740 1 --2- 172.21.15.134:0/542974756 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f03b7771c0 0x55f03bc30d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:54.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.415+0000 7f70b0249740 4 client.25882 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:17:54.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.415+0000 7f70b0249740 1 -- 172.21.15.134:0/542974756 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f03baf8c60 con 0x55f03b7771c0 2024-07-26T11:17:54.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.417+0000 7f70a1e00640 1 --2- 172.21.15.134:0/542974756 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f03b7771c0 0x55f03bc30d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.419+0000 7f7083e00640 10 client.25882 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.423+0000 7f70a1e00640 1 --2- 172.21.15.134:0/542974756 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f03b7771c0 0x55f03bc30d60 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9feeb89563a9049 server_cookie=58e8e1615ea76c7b in_seq=0 out_seq=0 2024-07-26T11:17:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.488+0000 7f7083e00640 1 -- 172.21.15.134:0/542974756 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f03baf8c60 con 0x55f03b7771c0 2024-07-26T11:17:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.488+0000 7f7083e00640 10 client.25882 handle_command_reply: tid=0 2024-07-26T11:17:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.488+0000 7f70b0249740 1 client.25882 shutdown 2024-07-26T11:17:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.491+0000 7f70b0249740 1 -- 172.21.15.134:0/542974756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7c24d0 msgr2=0x55f03bbb7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.491+0000 7f70b0249740 1 --2- 172.21.15.134:0/542974756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7c24d0 0x55f03bbb7eb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f70900096c0 tx=0x7f7090045660 comp rx=0 tx=0).stop 2024-07-26T11:17:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.491+0000 7f70b0249740 1 -- 172.21.15.134:0/542974756 shutdown_connections 2024-07-26T11:17:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.491+0000 7f70b0249740 1 --2- 172.21.15.134:0/542974756 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f03b7771c0 0x55f03bc30d60 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.491+0000 7f70b0249740 1 --2- 172.21.15.134:0/542974756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f03b7c24d0 0x55f03bbb7eb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.491+0000 7f70b0249740 1 --2- 172.21.15.134:0/542974756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f03b7e31b0 0x55f03bc2cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.491+0000 7f70b0249740 1 --2- 172.21.15.134:0/542974756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f03b7ba2f0 0x55f03bbb7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.491+0000 7f70b0249740 1 -- 172.21.15.134:0/542974756 >> 172.21.15.134:0/542974756 conn(0x55f03b7d7300 msgr2=0x55f03bb918b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.491+0000 7f7083e00640 0 client.25882 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:17:54.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.494+0000 7f70b0249740 1 -- 172.21.15.134:0/542974756 shutdown_connections 2024-07-26T11:17:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.496+0000 7f70b0249740 1 -- 172.21.15.134:0/542974756 wait complete. 2024-07-26T11:17:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.496+0000 7f70b0249740 20 client.25882 trim_cache size 0 max 16384 2024-07-26T11:17:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:17:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:17:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:17:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:17:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.501+0000 7f70a9200640 1 -- 172.21.15.134:0/1520506471 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f70880b3e40 msgr2=0x7f70880b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.501+0000 7f70a9200640 1 --2- 172.21.15.134:0/1520506471 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f70880b3e40 0x7f70880b6350 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f709c00f010 tx=0x7f709c010f90 comp rx=0 tx=0).stop 2024-07-26T11:17:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.501+0000 7f70a9200640 1 -- 172.21.15.134:0/1520506471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70a4104360 msgr2=0x7f70a410a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:54.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.501+0000 7f70a9200640 1 --2- 172.21.15.134:0/1520506471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70a4104360 0x7f70a410a4b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f709400ea20 tx=0x7f709400eee0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.505 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.503+0000 7f70a9200640 1 -- 172.21.15.134:0/1520506471 shutdown_connections 2024-07-26T11:17:54.505 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.503+0000 7f70a9200640 1 --2- 172.21.15.134:0/1520506471 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f70880b3e40 0x7f70880b6350 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.505 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.503+0000 7f70a9200640 1 --2- 172.21.15.134:0/1520506471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70a410af30 0x7f70a41ac6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.505 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.503+0000 7f70a9200640 1 --2- 172.21.15.134:0/1520506471 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f70a410d7c0 0x7f70a410a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.505 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.503+0000 7f70a9200640 1 --2- 172.21.15.134:0/1520506471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70a4104360 0x7f70a410a4b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:54.505 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.503+0000 7f70a9200640 1 -- 172.21.15.134:0/1520506471 >> 172.21.15.134:0/1520506471 conn(0x7f70a40fde10 msgr2=0x7f70a4105190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:54.506 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.503+0000 7f70a9200640 1 -- 172.21.15.134:0/1520506471 shutdown_connections 2024-07-26T11:17:54.508 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:17:54.506+0000 7f70a9200640 1 -- 172.21.15.134:0/1520506471 wait complete. 2024-07-26T11:17:55.702 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:55 smithi134 ceph-mon[31966]: pgmap v538: 129 pgs: 129 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 297 MiB/s wr, 110 op/s 2024-07-26T11:17:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:55 smithi193 ceph-mon[37473]: pgmap v538: 129 pgs: 129 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 297 MiB/s wr, 110 op/s 2024-07-26T11:17:55.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:55 smithi189 ceph-mon[36790]: pgmap v538: 129 pgs: 129 active+clean; 9.4 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 297 MiB/s wr, 110 op/s 2024-07-26T11:17:56.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.782+0000 7f031fe00640 1 -- 172.21.15.134:0/3323501546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0320077cc0 msgr2=0x7f0320113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:56.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.782+0000 7f031fe00640 1 --2- 172.21.15.134:0/3323501546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0320077cc0 0x7f0320113a70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f030c009900 tx=0x7f030c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.783+0000 7f031fe00640 1 -- 172.21.15.134:0/3323501546 shutdown_connections 2024-07-26T11:17:56.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.783+0000 7f031fe00640 1 --2- 172.21.15.134:0/3323501546 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0320113fb0 0x7f03201163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.783+0000 7f031fe00640 1 --2- 172.21.15.134:0/3323501546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0320077cc0 0x7f0320113a70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.783+0000 7f031fe00640 1 --2- 172.21.15.134:0/3323501546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0320077370 0x7f0320077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.783+0000 7f031fe00640 1 -- 172.21.15.134:0/3323501546 >> 172.21.15.134:0/3323501546 conn(0x7f0320101060 msgr2=0x7f03201034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:56.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.785+0000 7f031fe00640 1 -- 172.21.15.134:0/3323501546 shutdown_connections 2024-07-26T11:17:56.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.785+0000 7f031fe00640 1 -- 172.21.15.134:0/3323501546 wait complete. 2024-07-26T11:17:56.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.787+0000 7f031fe00640 1 Processor -- start 2024-07-26T11:17:56.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.788+0000 7f031fe00640 1 -- start start 2024-07-26T11:17:56.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.789+0000 7f031fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0320077370 0x7f03201a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.789+0000 7f031fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0320077cc0 0x7f03201a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.789+0000 7f031fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0320113fb0 0x7f03201a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.789+0000 7f031fe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0320119960 con 0x7f0320077cc0 2024-07-26T11:17:56.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.789+0000 7f031fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f03201197e0 con 0x7f0320113fb0 2024-07-26T11:17:56.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.789+0000 7f031fe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0320119ae0 con 0x7f0320077370 2024-07-26T11:17:56.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.790+0000 7f031ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0320077370 0x7f03201a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:56.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.790+0000 7f031ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0320077370 0x7f03201a0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:36754/0 (socket says 172.21.15.134:36754) 2024-07-26T11:17:56.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.790+0000 7f031f400640 1 --2- 172.21.15.134:0/4230135636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0320113fb0 0x7f03201a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:56.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.790+0000 7f031ea00640 1 -- 172.21.15.134:0/4230135636 learned_addr learned my addr 172.21.15.134:0/4230135636 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:56.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.791+0000 7f031ea00640 1 -- 172.21.15.134:0/4230135636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0320113fb0 msgr2=0x7f03201a5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:56.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.791+0000 7f031ea00640 1 --2- 172.21.15.134:0/4230135636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0320113fb0 0x7f03201a5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.791+0000 7f031ea00640 1 -- 172.21.15.134:0/4230135636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0320077cc0 msgr2=0x7f03201a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:56.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.791+0000 7f031ea00640 1 --2- 172.21.15.134:0/4230135636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0320077cc0 0x7f03201a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.791+0000 7f031ea00640 1 -- 172.21.15.134:0/4230135636 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03201a61c0 con 0x7f0320077370 2024-07-26T11:17:56.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.791+0000 7f031ea00640 1 --2- 172.21.15.134:0/4230135636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0320077370 0x7f03201a0310 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f031000c920 tx=0x7f031000cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:56.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.791+0000 7f0324a00640 1 -- 172.21.15.134:0/4230135636 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310007c70 con 0x7f0320077370 2024-07-26T11:17:56.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.792+0000 7f0324a00640 1 -- 172.21.15.134:0/4230135636 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0310007e10 con 0x7f0320077370 2024-07-26T11:17:56.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.792+0000 7f0324a00640 1 -- 172.21.15.134:0/4230135636 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310018820 con 0x7f0320077370 2024-07-26T11:17:56.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.793+0000 7f031fe00640 1 -- 172.21.15.134:0/4230135636 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03201a64b0 con 0x7f0320077370 2024-07-26T11:17:56.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.793+0000 7f031fe00640 1 -- 172.21.15.134:0/4230135636 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03201a6970 con 0x7f0320077370 2024-07-26T11:17:56.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.806+0000 7f032cab2740 1 Processor -- start 2024-07-26T11:17:56.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.808+0000 7f0324a00640 1 -- 172.21.15.134:0/4230135636 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f031000f450 con 0x7f0320077370 2024-07-26T11:17:56.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.811+0000 7f0324a00640 1 --2- 172.21.15.134:0/4230135636 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f03040b3e70 0x7f03040b6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.811+0000 7f0324a00640 1 -- 172.21.15.134:0/4230135636 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f03100e4340 con 0x7f0320077370 2024-07-26T11:17:56.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.815+0000 7f032cab2740 1 -- start start 2024-07-26T11:17:56.821 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.818+0000 7f031f400640 1 --2- 172.21.15.134:0/4230135636 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f03040b3e70 0x7f03040b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:56.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.819+0000 7f032cab2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0864c070 0x55ec089f1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.819+0000 7f032cab2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec089f1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.819+0000 7f032cab2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0862b390 0x55ec089f1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.820+0000 7f032cab2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ec08967bf0 con 0x55ec086231b0 2024-07-26T11:17:56.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.820+0000 7f032cab2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ec089570f0 con 0x55ec0864c070 2024-07-26T11:17:56.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.820+0000 7f032cab2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ec0895b130 con 0x55ec0862b390 2024-07-26T11:17:56.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.821+0000 7f031e000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec089f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:56.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.821+0000 7f031e000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec089f1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:35168/0 (socket says 172.21.15.134:35168) 2024-07-26T11:17:56.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.821+0000 7f031e000640 1 -- 172.21.15.134:0/4084685139 learned_addr learned my addr 172.21.15.134:0/4084685139 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:56.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.821+0000 7f031ea00640 1 --2- 172.21.15.134:0/4084685139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0864c070 0x55ec089f1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:56.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.822+0000 7f031e000640 1 -- 172.21.15.134:0/4084685139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0862b390 msgr2=0x55ec089f1ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:17:56.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.822+0000 7f031e000640 1 --2- 172.21.15.134:0/4084685139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0862b390 0x55ec089f1ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.822+0000 7f031e000640 1 -- 172.21.15.134:0/4084685139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0864c070 msgr2=0x55ec089f1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:56.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.822+0000 7f031e000640 1 --2- 172.21.15.134:0/4084685139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0864c070 0x55ec089f1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.822+0000 7f031e000640 1 -- 172.21.15.134:0/4084685139 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec086b7650 con 0x55ec086231b0 2024-07-26T11:17:56.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.822+0000 7f031e000640 1 --2- 172.21.15.134:0/4084685139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec089f1760 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f030c02a910 tx=0x7f030c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be865a5453122a8c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:56.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.823+0000 7f030be00640 1 -- 172.21.15.134:0/4084685139 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030c03e030 con 0x55ec086231b0 2024-07-26T11:17:56.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.823+0000 7f030be00640 1 -- 172.21.15.134:0/4084685139 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f030c0054d0 con 0x55ec086231b0 2024-07-26T11:17:56.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.823+0000 7f030be00640 1 -- 172.21.15.134:0/4084685139 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030c035040 con 0x55ec086231b0 2024-07-26T11:17:56.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.824+0000 7f032cab2740 1 -- 172.21.15.134:0/4084685139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 msgr2=0x55ec089f1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:56.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.824+0000 7f032cab2740 1 --2- 172.21.15.134:0/4084685139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec089f1760 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f030c02a910 tx=0x7f030c000bc0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.825+0000 7f032cab2740 1 -- 172.21.15.134:0/4084685139 shutdown_connections 2024-07-26T11:17:56.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.825+0000 7f032cab2740 1 --2- 172.21.15.134:0/4084685139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0862b390 0x55ec089f1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.825+0000 7f032cab2740 1 --2- 172.21.15.134:0/4084685139 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec089f1760 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.825+0000 7f032cab2740 1 --2- 172.21.15.134:0/4084685139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0864c070 0x55ec089f1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.825+0000 7f032cab2740 1 -- 172.21.15.134:0/4084685139 >> 172.21.15.134:0/4084685139 conn(0x55ec08640460 msgr2=0x55ec089ef110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:56.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.825+0000 7f032cab2740 1 -- 172.21.15.134:0/4084685139 shutdown_connections 2024-07-26T11:17:56.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.825+0000 7f032cab2740 1 -- 172.21.15.134:0/4084685139 wait complete. 2024-07-26T11:17:56.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.826+0000 7f032cab2740 1 -- start start 2024-07-26T11:17:56.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.826+0000 7f031f400640 1 --2- 172.21.15.134:0/4230135636 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f03040b3e70 0x7f03040b6380 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0318005fd0 tx=0x7f0318009600 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:56.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.827+0000 7f032cab2740 1 Processor -- start 2024-07-26T11:17:56.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.828+0000 7f032cab2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec089fc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.828+0000 7f032cab2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec089fccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.828+0000 7f032cab2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0864c070 0x55ec089fd200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.828+0000 7f032cab2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ec087ba700 con 0x55ec086231b0 2024-07-26T11:17:56.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.829+0000 7f031f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0864c070 0x55ec089fd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:56.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.829+0000 7f031f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0864c070 0x55ec089fd200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:36764/0 (socket says 172.21.15.134:36764) 2024-07-26T11:17:56.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.829+0000 7f031f400640 1 -- 172.21.15.134:0/2888178647 learned_addr learned my addr 172.21.15.134:0/2888178647 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:56.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.829+0000 7f031e000640 1 --2- 172.21.15.134:0/2888178647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec089fccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:56.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.829+0000 7f032cab2740 1 -- 172.21.15.134:0/2888178647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ec088f36c0 con 0x55ec0862b390 2024-07-26T11:17:56.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.829+0000 7f032cab2740 1 -- 172.21.15.134:0/2888178647 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ec089ff9a0 con 0x55ec0864c070 2024-07-26T11:17:56.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.829+0000 7f031ea00640 1 --2- 172.21.15.134:0/2888178647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec089fc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:56.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.830+0000 7f032cab2740 10 client.0 _ll_register_callbacks cb 0x55ec08903170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:56.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.830+0000 7f031f400640 1 -- 172.21.15.134:0/2888178647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 msgr2=0x55ec089fccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:56.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.830+0000 7f031f400640 1 --2- 172.21.15.134:0/2888178647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec089fccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.830+0000 7f031f400640 1 -- 172.21.15.134:0/2888178647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 msgr2=0x55ec089fc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:56.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.831+0000 7f031f400640 1 --2- 172.21.15.134:0/2888178647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec089fc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.831+0000 7f031f400640 1 -- 172.21.15.134:0/2888178647 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec0890a860 con 0x55ec0864c070 2024-07-26T11:17:56.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.831+0000 7f031ea00640 1 --2- 172.21.15.134:0/2888178647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec089fc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:56.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.831+0000 7f031e000640 1 --2- 172.21.15.134:0/2888178647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec089fccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:17:56.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.831+0000 7f031f400640 1 --2- 172.21.15.134:0/2888178647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0864c070 0x55ec089fd200 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0318012c40 tx=0x7f0318004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49301f1047cf7c4b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:56.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.831+0000 7f0308c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:17:56.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.832+0000 7f0308c00640 1 -- 172.21.15.134:0/2888178647 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0318023070 con 0x55ec0864c070 2024-07-26T11:17:56.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.832+0000 7f0308c00640 1 -- 172.21.15.134:0/2888178647 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f031801f650 con 0x55ec0864c070 2024-07-26T11:17:56.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.832+0000 7f032cab2740 1 -- 172.21.15.134:0/2888178647 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f030c0054d0 con 0x55ec0864c070 2024-07-26T11:17:56.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.833+0000 7f0308c00640 1 -- 172.21.15.134:0/2888178647 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f031802f050 con 0x55ec0864c070 2024-07-26T11:17:56.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.833+0000 7f032cab2740 10 client.25888 fetch_fsmap learned FSMap version 31 2024-07-26T11:17:56.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.833+0000 7f032cab2740 1 -- 172.21.15.134:0/2888178647 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ec08a0b580 con 0x55ec0864c070 2024-07-26T11:17:56.837 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.834+0000 7f0308c00640 1 -- 172.21.15.134:0/2888178647 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f030c0054d0 con 0x55ec0864c070 2024-07-26T11:17:56.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.836+0000 7f0308c00640 1 -- 172.21.15.134:0/2888178647 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f031800f040 con 0x55ec0864c070 2024-07-26T11:17:56.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.837+0000 7f032cab2740 10 client.25888 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:17:56.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.837+0000 7f032cab2740 10 client.25888 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-07-26T11:17:56.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.837+0000 7f032cab2740 20 client.25888 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:56.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.837+0000 7f032cab2740 1 --2- 172.21.15.134:0/2888178647 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55ec0860a9f0 0x55ec08a0ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.837+0000 7f032cab2740 4 client.25888 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:17:56.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.837+0000 7f032cab2740 1 -- 172.21.15.134:0/2888178647 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec08969b40 con 0x55ec0860a9f0 2024-07-26T11:17:56.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.839+0000 7f031ea00640 1 --2- 172.21.15.134:0/2888178647 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55ec0860a9f0 0x55ec08a0ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:56.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.840+0000 7f031ea00640 1 --2- 172.21.15.134:0/2888178647 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55ec0860a9f0 0x55ec08a0ea60 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cb0b8b9699c7bdc4 server_cookie=ecd3ff1d70d79c17 in_seq=0 out_seq=0 2024-07-26T11:17:56.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.840+0000 7f0308c00640 10 client.25888 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:17:56.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.841+0000 7f0308c00640 1 -- 172.21.15.134:0/2888178647 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ec08969b40 con 0x55ec0860a9f0 2024-07-26T11:17:56.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.841+0000 7f0308c00640 10 client.25888 handle_command_reply: tid=0 2024-07-26T11:17:56.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.841+0000 7f032cab2740 1 client.25888 shutdown 2024-07-26T11:17:56.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.844+0000 7f032cab2740 1 -- 172.21.15.134:0/2888178647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0864c070 msgr2=0x55ec089fd200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:56.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.844+0000 7f032cab2740 1 --2- 172.21.15.134:0/2888178647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0864c070 0x55ec089fd200 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0318012c40 tx=0x7f0318004a40 comp rx=0 tx=0).stop 2024-07-26T11:17:56.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.845+0000 7f032cab2740 1 -- 172.21.15.134:0/2888178647 shutdown_connections 2024-07-26T11:17:56.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.845+0000 7f032cab2740 1 --2- 172.21.15.134:0/2888178647 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55ec0860a9f0 0x55ec08a0ea60 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.845+0000 7f032cab2740 1 --2- 172.21.15.134:0/2888178647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0864c070 0x55ec089fd200 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.845+0000 7f0308c00640 0 client.25888 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:17:56.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.845+0000 7f032cab2740 1 --2- 172.21.15.134:0/2888178647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec089fccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.845+0000 7f032cab2740 1 --2- 172.21.15.134:0/2888178647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec089fc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.845+0000 7f032cab2740 1 -- 172.21.15.134:0/2888178647 >> 172.21.15.134:0/2888178647 conn(0x55ec08640460 msgr2=0x55ec089f4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:56.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.846+0000 7f032cab2740 1 -- 172.21.15.134:0/2888178647 shutdown_connections 2024-07-26T11:17:56.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.846+0000 7f032cab2740 1 -- 172.21.15.134:0/2888178647 wait complete. 2024-07-26T11:17:56.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.846+0000 7f032cab2740 20 client.25888 trim_cache size 0 max 16384 2024-07-26T11:17:56.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.889+0000 7f032cab2740 1 Processor -- start 2024-07-26T11:17:56.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.890+0000 7f032cab2740 1 -- start start 2024-07-26T11:17:56.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.890+0000 7f032cab2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0860a9f0 0x55ec08a08aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.890+0000 7f032cab2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec08a14220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.890+0000 7f032cab2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec08a14760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.890+0000 7f032cab2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f03201b6c20 con 0x55ec086231b0 2024-07-26T11:17:56.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.890+0000 7f032cab2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f03201b8790 con 0x55ec0862b390 2024-07-26T11:17:56.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.890+0000 7f032cab2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f03201b72f0 con 0x55ec0860a9f0 2024-07-26T11:17:56.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.893+0000 7f031f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec08a14760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:56.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.893+0000 7f031f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec08a14760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38116/0 (socket says 172.21.15.134:38116) 2024-07-26T11:17:56.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.893+0000 7f031f400640 1 -- 172.21.15.134:0/2364011951 learned_addr learned my addr 172.21.15.134:0/2364011951 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:56.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.893+0000 7f031ea00640 1 --2- 172.21.15.134:0/2364011951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0860a9f0 0x55ec08a08aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:56.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.894+0000 7f031e000640 1 --2- 172.21.15.134:0/2364011951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec08a14220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:56.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.894+0000 7f031f400640 1 -- 172.21.15.134:0/2364011951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0860a9f0 msgr2=0x55ec08a08aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:56.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.894+0000 7f031f400640 1 --2- 172.21.15.134:0/2364011951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0860a9f0 0x55ec08a08aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.894+0000 7f031f400640 1 -- 172.21.15.134:0/2364011951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 msgr2=0x55ec08a14220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:56.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.894+0000 7f031f400640 1 --2- 172.21.15.134:0/2364011951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec08a14220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.894+0000 7f031f400640 1 -- 172.21.15.134:0/2364011951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec089f3510 con 0x55ec0862b390 2024-07-26T11:17:56.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.895+0000 7f031f400640 1 --2- 172.21.15.134:0/2364011951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec08a14760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0318011fd0 tx=0x7f0318009730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea800c2ecf43139d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:56.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.895+0000 7f030b400640 1 -- 172.21.15.134:0/2364011951 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0318023070 con 0x55ec0862b390 2024-07-26T11:17:56.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.896+0000 7f030b400640 1 -- 172.21.15.134:0/2364011951 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0318038070 con 0x55ec0862b390 2024-07-26T11:17:56.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.896+0000 7f030b400640 1 -- 172.21.15.134:0/2364011951 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0318028710 con 0x55ec0862b390 2024-07-26T11:17:56.902 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.897+0000 7f032cab2740 1 -- 172.21.15.134:0/2364011951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 msgr2=0x55ec08a14760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:56.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.901+0000 7f032cab2740 1 --2- 172.21.15.134:0/2364011951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec08a14760 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0318011fd0 tx=0x7f0318009730 comp rx=0 tx=0).stop 2024-07-26T11:17:56.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.901+0000 7f032cab2740 1 -- 172.21.15.134:0/2364011951 shutdown_connections 2024-07-26T11:17:56.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.901+0000 7f032cab2740 1 --2- 172.21.15.134:0/2364011951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec08a14760 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.901+0000 7f032cab2740 1 --2- 172.21.15.134:0/2364011951 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec08a14220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.901+0000 7f032cab2740 1 --2- 172.21.15.134:0/2364011951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0860a9f0 0x55ec08a08aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.901+0000 7f032cab2740 1 -- 172.21.15.134:0/2364011951 >> 172.21.15.134:0/2364011951 conn(0x55ec08640460 msgr2=0x55ec089f4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:56.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.902+0000 7f032cab2740 1 -- 172.21.15.134:0/2364011951 shutdown_connections 2024-07-26T11:17:56.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.902+0000 7f032cab2740 1 -- 172.21.15.134:0/2364011951 wait complete. 2024-07-26T11:17:56.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.903+0000 7f032cab2740 1 -- start start 2024-07-26T11:17:56.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.903+0000 7f032cab2740 1 Processor -- start 2024-07-26T11:17:56.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.905+0000 7f032cab2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0860a9f0 0x55ec08a02010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.905+0000 7f032cab2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec08a02550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.905+0000 7f032cab2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec08a751f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.908+0000 7f031f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec08a751f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:56.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.908+0000 7f031f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec08a751f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38126/0 (socket says 172.21.15.134:38126) 2024-07-26T11:17:56.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.908+0000 7f031f400640 1 -- 172.21.15.134:0/1377499382 learned_addr learned my addr 172.21.15.134:0/1377499382 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:17:56.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.909+0000 7f032cab2740 1 -- 172.21.15.134:0/1377499382 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f03201b8090 con 0x55ec086231b0 2024-07-26T11:17:56.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.909+0000 7f032cab2740 1 -- 172.21.15.134:0/1377499382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f03201b79c0 con 0x55ec0862b390 2024-07-26T11:17:56.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.909+0000 7f032cab2740 1 -- 172.21.15.134:0/1377499382 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f03201b4c00 con 0x55ec0860a9f0 2024-07-26T11:17:56.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.910+0000 7f032cab2740 10 client.0 _ll_register_callbacks cb 0x7f03180128c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:17:56.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.910+0000 7f031f400640 1 -- 172.21.15.134:0/1377499382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0860a9f0 msgr2=0x55ec08a02010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:56.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.910+0000 7f031ea00640 1 --2- 172.21.15.134:0/1377499382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0860a9f0 0x55ec08a02010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:56.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.911+0000 7f031f400640 1 --2- 172.21.15.134:0/1377499382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0860a9f0 0x55ec08a02010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.911+0000 7f031f400640 1 -- 172.21.15.134:0/1377499382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 msgr2=0x55ec08a02550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:17:56.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.912+0000 7f031e000640 1 --2- 172.21.15.134:0/1377499382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec08a02550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:56.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.913+0000 7f031f400640 1 --2- 172.21.15.134:0/1377499382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec08a02550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.913+0000 7f031f400640 1 -- 172.21.15.134:0/1377499382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec08a75730 con 0x55ec0862b390 2024-07-26T11:17:56.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.914+0000 7f031ea00640 1 --2- 172.21.15.134:0/1377499382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0860a9f0 0x55ec08a02010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:17:56.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.914+0000 7f0308c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:17:56.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.914+0000 7f031f400640 1 --2- 172.21.15.134:0/1377499382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec08a751f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0318010bd0 tx=0x7f0318010cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2afe621b99bd1dd9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:17:56.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.916+0000 7f0308c00640 1 -- 172.21.15.134:0/1377499382 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0318040070 con 0x55ec0862b390 2024-07-26T11:17:56.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.916+0000 7f0308c00640 1 -- 172.21.15.134:0/1377499382 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f031801f650 con 0x55ec0862b390 2024-07-26T11:17:56.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.916+0000 7f032cab2740 1 -- 172.21.15.134:0/1377499382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0318038070 con 0x55ec0862b390 2024-07-26T11:17:56.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.917+0000 7f0308c00640 1 -- 172.21.15.134:0/1377499382 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0318049050 con 0x55ec0862b390 2024-07-26T11:17:56.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.918+0000 7f032cab2740 10 client.25898 fetch_fsmap learned FSMap version 31 2024-07-26T11:17:56.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.918+0000 7f032cab2740 1 -- 172.21.15.134:0/1377499382 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55ec08a767c0 con 0x55ec0862b390 2024-07-26T11:17:56.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.919+0000 7f0308c00640 1 -- 172.21.15.134:0/1377499382 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0318038070 con 0x55ec0862b390 2024-07-26T11:17:56.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.920+0000 7f0308c00640 1 -- 172.21.15.134:0/1377499382 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f031800f040 con 0x55ec0862b390 2024-07-26T11:17:56.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.920+0000 7f032cab2740 10 client.25898 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:17:56.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.921+0000 7f032cab2740 10 client.25898 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-07-26T11:17:56.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.921+0000 7f032cab2740 20 client.25898 populate_metadata read hostname 'smithi134' 2024-07-26T11:17:56.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.921+0000 7f032cab2740 1 --2- 172.21.15.134:0/1377499382 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55ec0864c070 0x55ec08a78dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:17:56.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.921+0000 7f032cab2740 4 client.25898 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:17:56.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.921+0000 7f032cab2740 1 -- 172.21.15.134:0/1377499382 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ec089655e0 con 0x55ec0864c070 2024-07-26T11:17:56.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.928+0000 7f031ea00640 1 --2- 172.21.15.134:0/1377499382 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55ec0864c070 0x55ec08a78dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:17:56.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.931+0000 7f031ea00640 1 --2- 172.21.15.134:0/1377499382 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55ec0864c070 0x55ec08a78dd0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=142a5635fe5ae3f2 server_cookie=64169ee881dc979e in_seq=0 out_seq=0 2024-07-26T11:17:56.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.931+0000 7f0308c00640 10 client.25898 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:17:56.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.931+0000 7f0308c00640 1 -- 172.21.15.134:0/1377499382 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ec089655e0 con 0x55ec0864c070 2024-07-26T11:17:56.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.931+0000 7f0308c00640 10 client.25898 handle_command_reply: tid=0 2024-07-26T11:17:56.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.931+0000 7f032cab2740 1 client.25898 shutdown 2024-07-26T11:17:56.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.932+0000 7f032cab2740 1 -- 172.21.15.134:0/1377499382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 msgr2=0x55ec08a751f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:56.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.932+0000 7f032cab2740 1 --2- 172.21.15.134:0/1377499382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec08a751f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0318010bd0 tx=0x7f0318010cc0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.932+0000 7f032cab2740 1 -- 172.21.15.134:0/1377499382 shutdown_connections 2024-07-26T11:17:56.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.932+0000 7f032cab2740 1 --2- 172.21.15.134:0/1377499382 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55ec0864c070 0x55ec08a78dd0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.932+0000 7f0308c00640 0 client.25898 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:17:56.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.932+0000 7f032cab2740 1 --2- 172.21.15.134:0/1377499382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ec0862b390 0x55ec08a751f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.932+0000 7f032cab2740 1 --2- 172.21.15.134:0/1377499382 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ec086231b0 0x55ec08a02550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.932+0000 7f032cab2740 1 --2- 172.21.15.134:0/1377499382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ec0860a9f0 0x55ec08a02010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.932+0000 7f032cab2740 1 -- 172.21.15.134:0/1377499382 >> 172.21.15.134:0/1377499382 conn(0x55ec08640460 msgr2=0x55ec089f4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:56.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.933+0000 7f032cab2740 1 -- 172.21.15.134:0/1377499382 shutdown_connections 2024-07-26T11:17:56.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.933+0000 7f032cab2740 1 -- 172.21.15.134:0/1377499382 wait complete. 2024-07-26T11:17:56.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.933+0000 7f032cab2740 20 client.25898 trim_cache size 0 max 16384 2024-07-26T11:17:56.936 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:17:56.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.934+0000 7f031fe00640 1 -- 172.21.15.134:0/4230135636 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f03040b3e70 msgr2=0x7f03040b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:56.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.934+0000 7f031fe00640 1 --2- 172.21.15.134:0/4230135636 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f03040b3e70 0x7f03040b6380 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0318005fd0 tx=0x7f0318009600 comp rx=0 tx=0).stop 2024-07-26T11:17:56.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.934+0000 7f031fe00640 1 -- 172.21.15.134:0/4230135636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0320077370 msgr2=0x7f03201a0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:17:56.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.934+0000 7f031fe00640 1 --2- 172.21.15.134:0/4230135636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0320077370 0x7f03201a0310 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f031000c920 tx=0x7f031000cde0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.935+0000 7f031fe00640 1 -- 172.21.15.134:0/4230135636 shutdown_connections 2024-07-26T11:17:56.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.935+0000 7f031fe00640 1 --2- 172.21.15.134:0/4230135636 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f03040b3e70 0x7f03040b6380 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.935+0000 7f031fe00640 1 --2- 172.21.15.134:0/4230135636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0320113fb0 0x7f03201a5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.935+0000 7f031fe00640 1 --2- 172.21.15.134:0/4230135636 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0320077cc0 0x7f03201a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.935+0000 7f031fe00640 1 --2- 172.21.15.134:0/4230135636 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0320077370 0x7f03201a0310 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:17:56.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.935+0000 7f031fe00640 1 -- 172.21.15.134:0/4230135636 >> 172.21.15.134:0/4230135636 conn(0x7f0320101060 msgr2=0x7f0320116020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:17:56.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.935+0000 7f031fe00640 1 -- 172.21.15.134:0/4230135636 shutdown_connections 2024-07-26T11:17:56.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:17:56.935+0000 7f031fe00640 1 -- 172.21.15.134:0/4230135636 wait complete. 2024-07-26T11:17:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:57 smithi193 ceph-mon[37473]: pgmap v539: 129 pgs: 129 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 308 MiB/s wr, 203 op/s 2024-07-26T11:17:57.878 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:57 smithi134 ceph-mon[31966]: pgmap v539: 129 pgs: 129 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 308 MiB/s wr, 203 op/s 2024-07-26T11:17:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:57 smithi189 ceph-mon[36790]: pgmap v539: 129 pgs: 129 active+clean; 9.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 308 MiB/s wr, 203 op/s 2024-07-26T11:17:59.593 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-26T11:17:59.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:17:59 smithi134 ceph-mon[31966]: pgmap v540: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 295 MiB/s wr, 355 op/s 2024-07-26T11:17:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:17:59 smithi193 ceph-mon[37473]: pgmap v540: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 295 MiB/s wr, 355 op/s 2024-07-26T11:17:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:17:59 smithi189 ceph-mon[36790]: pgmap v540: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 295 MiB/s wr, 355 op/s 2024-07-26T11:18:00.483 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:18:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:01 smithi193 ceph-mon[37473]: pgmap v541: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 313 MiB/s wr, 668 op/s 2024-07-26T11:18:01.905 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:01 smithi134 ceph-mon[31966]: pgmap v541: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 313 MiB/s wr, 668 op/s 2024-07-26T11:18:01.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:01 smithi189 ceph-mon[36790]: pgmap v541: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 313 MiB/s wr, 668 op/s 2024-07-26T11:18:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:03 smithi193 ceph-mon[37473]: pgmap v542: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 264 MiB/s wr, 801 op/s 2024-07-26T11:18:03.927 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:03 smithi134 ceph-mon[31966]: pgmap v542: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 264 MiB/s wr, 801 op/s 2024-07-26T11:18:03.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:03 smithi189 ceph-mon[36790]: pgmap v542: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 264 MiB/s wr, 801 op/s 2024-07-26T11:18:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:05 smithi193 ceph-mon[37473]: pgmap v543: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 282 MiB/s wr, 1.02k op/s 2024-07-26T11:18:05.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:05 smithi134 ceph-mon[31966]: pgmap v543: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 282 MiB/s wr, 1.02k op/s 2024-07-26T11:18:05.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:05 smithi189 ceph-mon[36790]: pgmap v543: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 282 MiB/s wr, 1.02k op/s 2024-07-26T11:18:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.912+0000 7fa037200640 1 -- 172.21.15.134:0/1071719550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa038103340 msgr2=0x7fa038103750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.912+0000 7fa037200640 1 --2- 172.21.15.134:0/1071719550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa038103340 0x7fa038103750 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa028009900 tx=0x7fa02802a420 comp rx=0 tx=0).stop 2024-07-26T11:18:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.913+0000 7fa037200640 1 -- 172.21.15.134:0/1071719550 shutdown_connections 2024-07-26T11:18:07.915 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.913+0000 7fa037200640 1 --2- 172.21.15.134:0/1071719550 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa038108c70 0x7fa038109080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.913+0000 7fa037200640 1 --2- 172.21.15.134:0/1071719550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0381095c0 0x7fa03810fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.913+0000 7fa037200640 1 --2- 172.21.15.134:0/1071719550 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa038103340 0x7fa038103750 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.913+0000 7fa037200640 1 -- 172.21.15.134:0/1071719550 >> 172.21.15.134:0/1071719550 conn(0x7fa0380fed60 msgr2=0x7fa0381011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.913+0000 7fa037200640 1 -- 172.21.15.134:0/1071719550 shutdown_connections 2024-07-26T11:18:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.913+0000 7fa037200640 1 -- 172.21.15.134:0/1071719550 wait complete. 2024-07-26T11:18:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.914+0000 7fa037200640 1 Processor -- start 2024-07-26T11:18:07.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.914+0000 7fa037200640 1 -- start start 2024-07-26T11:18:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.915+0000 7fa037200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa038103340 0x7fa038197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.915+0000 7fa037200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa038108c70 0x7fa038198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.915+0000 7fa037200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0381095c0 0x7fa03819d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.915+0000 7fa037200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa0381110c0 con 0x7fa0381095c0 2024-07-26T11:18:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.915+0000 7fa037200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa038110f40 con 0x7fa038103340 2024-07-26T11:18:07.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.915+0000 7fa037200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa038111240 con 0x7fa038108c70 2024-07-26T11:18:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.915+0000 7fa035400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa038108c70 0x7fa038198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.915+0000 7fa035400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa038108c70 0x7fa038198080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:47484/0 (socket says 172.21.15.134:47484) 2024-07-26T11:18:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.915+0000 7fa035400640 1 -- 172.21.15.134:0/1173355416 learned_addr learned my addr 172.21.15.134:0/1173355416 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.915+0000 7fa035e00640 1 --2- 172.21.15.134:0/1173355416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa038103340 0x7fa038197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.915+0000 7fa036800640 1 --2- 172.21.15.134:0/1173355416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0381095c0 0x7fa03819d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:07.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.916+0000 7fa035400640 1 -- 172.21.15.134:0/1173355416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa038103340 msgr2=0x7fa038197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.916+0000 7fa035400640 1 --2- 172.21.15.134:0/1173355416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa038103340 0x7fa038197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.916+0000 7fa035400640 1 -- 172.21.15.134:0/1173355416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0381095c0 msgr2=0x7fa03819d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.916+0000 7fa035400640 1 --2- 172.21.15.134:0/1173355416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0381095c0 0x7fa03819d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.916+0000 7fa035400640 1 -- 172.21.15.134:0/1173355416 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa03819da10 con 0x7fa038108c70 2024-07-26T11:18:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.916+0000 7fa036800640 1 --2- 172.21.15.134:0/1173355416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0381095c0 0x7fa03819d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:18:07.919 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.916+0000 7fa035400640 1 --2- 172.21.15.134:0/1173355416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa038108c70 0x7fa038198080 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa02400d750 tx=0x7fa02400dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.916+0000 7fa02ea00640 1 -- 172.21.15.134:0/1173355416 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa024017070 con 0x7fa038108c70 2024-07-26T11:18:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.916+0000 7fa02ea00640 1 -- 172.21.15.134:0/1173355416 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0240054d0 con 0x7fa038108c70 2024-07-26T11:18:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.916+0000 7fa037200640 1 -- 172.21.15.134:0/1173355416 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa03819dd00 con 0x7fa038108c70 2024-07-26T11:18:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.917+0000 7fa02ea00640 1 -- 172.21.15.134:0/1173355416 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa024004c30 con 0x7fa038108c70 2024-07-26T11:18:07.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.917+0000 7fa037200640 1 -- 172.21.15.134:0/1173355416 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0381acca0 con 0x7fa038108c70 2024-07-26T11:18:07.923 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.921+0000 7fa0443e7740 1 Processor -- start 2024-07-26T11:18:07.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:07 smithi134 ceph-mon[31966]: pgmap v544: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 277 MiB/s wr, 1.18k op/s 2024-07-26T11:18:07.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.923+0000 7fa0443e7740 1 -- start start 2024-07-26T11:18:07.925 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.923+0000 7fa0443e7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0811b0 0x55849b424c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.926 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.923+0000 7fa02ea00640 1 -- 172.21.15.134:0/1173355416 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fa024010460 con 0x7fa038108c70 2024-07-26T11:18:07.926 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.924+0000 7fa0443e7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0582f0 0x55849b425180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.926 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.924+0000 7fa0443e7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0604d0 0x55849b4256c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.926 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.924+0000 7fa0443e7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55849b399270 con 0x55849b0582f0 2024-07-26T11:18:07.926 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.924+0000 7fa0443e7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55849b388770 con 0x55849b0811b0 2024-07-26T11:18:07.926 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.924+0000 7fa0443e7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55849b38c7b0 con 0x55849b0604d0 2024-07-26T11:18:07.926 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.924+0000 7fa036800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0811b0 0x55849b424c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.924+0000 7fa036800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0811b0 0x55849b424c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43636/0 (socket says 172.21.15.134:43636) 2024-07-26T11:18:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.924+0000 7fa036800640 1 -- 172.21.15.134:0/2413087946 learned_addr learned my addr 172.21.15.134:0/2413087946 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.924+0000 7fa036800640 1 -- 172.21.15.134:0/2413087946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0604d0 msgr2=0x55849b4256c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:18:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.924+0000 7fa036800640 1 --2- 172.21.15.134:0/2413087946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0604d0 0x55849b4256c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.924+0000 7fa035e00640 1 --2- 172.21.15.134:0/2413087946 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0582f0 0x55849b425180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:07.927 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.924+0000 7fa036800640 1 -- 172.21.15.134:0/2413087946 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0582f0 msgr2=0x55849b425180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.924+0000 7fa036800640 1 --2- 172.21.15.134:0/2413087946 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0582f0 0x55849b425180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.924+0000 7fa036800640 1 -- 172.21.15.134:0/2413087946 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55849b0ec790 con 0x55849b0811b0 2024-07-26T11:18:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.925+0000 7fa035e00640 1 --2- 172.21.15.134:0/2413087946 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0582f0 0x55849b425180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:18:07.928 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.925+0000 7fa036800640 1 --2- 172.21.15.134:0/2413087946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0811b0 0x55849b424c40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa03000b340 tx=0x7fa03000b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=115fbe59841525d8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.926+0000 7fa02ea00640 1 --2- 172.21.15.134:0/1173355416 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa01c0b3e40 0x7fa01c0b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.926+0000 7fa023400640 1 -- 172.21.15.134:0/2413087946 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03001c070 con 0x55849b0811b0 2024-07-26T11:18:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.926+0000 7fa023400640 1 -- 172.21.15.134:0/2413087946 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa03000aa30 con 0x55849b0811b0 2024-07-26T11:18:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.926+0000 7fa023400640 1 -- 172.21.15.134:0/2413087946 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa030017b90 con 0x55849b0811b0 2024-07-26T11:18:07.929 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.927+0000 7fa035e00640 1 --2- 172.21.15.134:0/1173355416 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa01c0b3e40 0x7fa01c0b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.928+0000 7fa02ea00640 1 -- 172.21.15.134:0/1173355416 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa0240e3db0 con 0x7fa038108c70 2024-07-26T11:18:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.928+0000 7fa0443e7740 1 -- 172.21.15.134:0/2413087946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0811b0 msgr2=0x55849b424c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.928+0000 7fa0443e7740 1 --2- 172.21.15.134:0/2413087946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0811b0 0x55849b424c40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa03000b340 tx=0x7fa03000b800 comp rx=0 tx=0).stop 2024-07-26T11:18:07.930 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.928+0000 7fa035e00640 1 --2- 172.21.15.134:0/1173355416 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa01c0b3e40 0x7fa01c0b6350 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa028000c00 tx=0x7fa028005e30 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.930+0000 7fa0443e7740 1 -- 172.21.15.134:0/2413087946 shutdown_connections 2024-07-26T11:18:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.930+0000 7fa0443e7740 1 --2- 172.21.15.134:0/2413087946 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0604d0 0x55849b4256c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.930+0000 7fa0443e7740 1 --2- 172.21.15.134:0/2413087946 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0582f0 0x55849b425180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.930+0000 7fa0443e7740 1 --2- 172.21.15.134:0/2413087946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0811b0 0x55849b424c40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.930+0000 7fa0443e7740 1 -- 172.21.15.134:0/2413087946 >> 172.21.15.134:0/2413087946 conn(0x55849b075300 msgr2=0x55849b422b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.930+0000 7fa0443e7740 1 -- 172.21.15.134:0/2413087946 shutdown_connections 2024-07-26T11:18:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.931+0000 7fa0443e7740 1 -- 172.21.15.134:0/2413087946 wait complete. 2024-07-26T11:18:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.933+0000 7fa0443e7740 1 -- start start 2024-07-26T11:18:07.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.934+0000 7fa0443e7740 1 Processor -- start 2024-07-26T11:18:07.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.934+0000 7fa0443e7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0582f0 0x55849b4309c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.934+0000 7fa0443e7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0604d0 0x55849b430fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.934+0000 7fa0443e7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0811b0 0x55849b4366f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.935+0000 7fa0443e7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55849b2c2ab0 con 0x55849b0582f0 2024-07-26T11:18:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.935+0000 7fa0443e7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55849b325d50 con 0x55849b0811b0 2024-07-26T11:18:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.935+0000 7fa0443e7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55849b4333c0 con 0x55849b0604d0 2024-07-26T11:18:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.935+0000 7fa0443e7740 10 client.0 _ll_register_callbacks cb 0x55849b098a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.935+0000 7fa036800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0811b0 0x55849b4366f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.935+0000 7fa036800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0811b0 0x55849b4366f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43640/0 (socket says 172.21.15.134:43640) 2024-07-26T11:18:07.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.935+0000 7fa036800640 1 -- 172.21.15.134:0/4066593411 learned_addr learned my addr 172.21.15.134:0/4066593411 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:07.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.935+0000 7fa035e00640 1 --2- 172.21.15.134:0/4066593411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0582f0 0x55849b4309c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:07.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.936+0000 7fa035400640 1 --2- 172.21.15.134:0/4066593411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0604d0 0x55849b430fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:07.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.936+0000 7fa036800640 1 -- 172.21.15.134:0/4066593411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0604d0 msgr2=0x55849b430fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:07.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.936+0000 7fa036800640 1 --2- 172.21.15.134:0/4066593411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0604d0 0x55849b430fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.936+0000 7fa036800640 1 -- 172.21.15.134:0/4066593411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0582f0 msgr2=0x55849b4309c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.936+0000 7fa036800640 1 --2- 172.21.15.134:0/4066593411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0582f0 0x55849b4309c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.936+0000 7fa036800640 1 -- 172.21.15.134:0/4066593411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55849b165620 con 0x55849b0811b0 2024-07-26T11:18:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.936+0000 7fa035400640 1 --2- 172.21.15.134:0/4066593411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0604d0 0x55849b430fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:18:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.936+0000 7fa017e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:18:07.939 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.936+0000 7fa036800640 1 --2- 172.21.15.134:0/4066593411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0811b0 0x55849b4366f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa03000de60 tx=0x7fa030017690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=accf3a1d7ea6a37a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.937+0000 7fa017e00640 1 -- 172.21.15.134:0/4066593411 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa030025060 con 0x55849b0811b0 2024-07-26T11:18:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.937+0000 7fa0443e7740 1 -- 172.21.15.134:0/4066593411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa03000aa30 con 0x55849b0811b0 2024-07-26T11:18:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.937+0000 7fa017e00640 1 -- 172.21.15.134:0/4066593411 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa030013070 con 0x55849b0811b0 2024-07-26T11:18:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.938+0000 7fa017e00640 1 -- 172.21.15.134:0/4066593411 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa030025060 con 0x55849b0811b0 2024-07-26T11:18:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.938+0000 7fa0443e7740 10 client.25910 fetch_fsmap learned FSMap version 31 2024-07-26T11:18:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.938+0000 7fa0443e7740 1 -- 172.21.15.134:0/4066593411 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55849b43f500 con 0x55849b0811b0 2024-07-26T11:18:07.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.938+0000 7fa017e00640 1 -- 172.21.15.134:0/4066593411 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa03000aa30 con 0x55849b0811b0 2024-07-26T11:18:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.939+0000 7fa017e00640 1 -- 172.21.15.134:0/4066593411 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7fa030004a40 con 0x55849b0811b0 2024-07-26T11:18:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.939+0000 7fa0443e7740 10 client.25910 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:18:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.939+0000 7fa0443e7740 10 client.25910 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:18:07.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.939+0000 7fa0443e7740 20 client.25910 populate_metadata read hostname 'smithi134' 2024-07-26T11:18:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.939+0000 7fa0443e7740 1 --2- 172.21.15.134:0/4066593411 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55849b03fd80 0x55849b442920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.939+0000 7fa0443e7740 4 client.25910 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:18:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.939+0000 7fa0443e7740 1 -- 172.21.15.134:0/4066593411 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55849b39b1c0 con 0x55849b03fd80 2024-07-26T11:18:07.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.940+0000 7fa035e00640 1 --2- 172.21.15.134:0/4066593411 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55849b03fd80 0x55849b442920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.941+0000 7fa017e00640 10 client.25910 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:07.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.941+0000 7fa035e00640 1 --2- 172.21.15.134:0/4066593411 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55849b03fd80 0x55849b442920 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b897709e5a6a73da server_cookie=8624a560073a42e4 in_seq=0 out_seq=0 2024-07-26T11:18:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:07 smithi189 ceph-mon[36790]: pgmap v544: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 277 MiB/s wr, 1.18k op/s 2024-07-26T11:18:07.944 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.942+0000 7fa017e00640 1 -- 172.21.15.134:0/4066593411 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55849b39b1c0 con 0x55849b03fd80 2024-07-26T11:18:07.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.942+0000 7fa017e00640 10 client.25910 handle_command_reply: tid=0 2024-07-26T11:18:07.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.942+0000 7fa0443e7740 1 client.25910 shutdown 2024-07-26T11:18:07.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.942+0000 7fa0443e7740 1 -- 172.21.15.134:0/4066593411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0811b0 msgr2=0x55849b4366f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:07.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.942+0000 7fa0443e7740 1 --2- 172.21.15.134:0/4066593411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0811b0 0x55849b4366f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa03000de60 tx=0x7fa030017690 comp rx=0 tx=0).stop 2024-07-26T11:18:07.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.943+0000 7fa0443e7740 1 -- 172.21.15.134:0/4066593411 shutdown_connections 2024-07-26T11:18:07.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.943+0000 7fa0443e7740 1 --2- 172.21.15.134:0/4066593411 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55849b03fd80 0x55849b442920 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.943+0000 7fa0443e7740 1 --2- 172.21.15.134:0/4066593411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0604d0 0x55849b430fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.943+0000 7fa0443e7740 1 --2- 172.21.15.134:0/4066593411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0811b0 0x55849b4366f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.943+0000 7fa0443e7740 1 --2- 172.21.15.134:0/4066593411 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0582f0 0x55849b4309c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.943+0000 7fa0443e7740 1 -- 172.21.15.134:0/4066593411 >> 172.21.15.134:0/4066593411 conn(0x55849b075300 msgr2=0x55849b425c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:07.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.943+0000 7fa017e00640 0 client.25910 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:07.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.944+0000 7fa0443e7740 1 -- 172.21.15.134:0/4066593411 shutdown_connections 2024-07-26T11:18:07.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.944+0000 7fa0443e7740 1 -- 172.21.15.134:0/4066593411 wait complete. 2024-07-26T11:18:07.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.944+0000 7fa0443e7740 20 client.25910 trim_cache size 0 max 16384 2024-07-26T11:18:07.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.977+0000 7fa0443e7740 1 Processor -- start 2024-07-26T11:18:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.979+0000 7fa0443e7740 1 -- start start 2024-07-26T11:18:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.979+0000 7fa0443e7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0582f0 0x55849b430390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.979+0000 7fa0443e7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0604d0 0x55849b429d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.979+0000 7fa0443e7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0811b0 0x55849b42a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.979+0000 7fa0443e7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa0381b6f80 con 0x55849b0582f0 2024-07-26T11:18:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.979+0000 7fa0443e7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa0381b8af0 con 0x55849b0604d0 2024-07-26T11:18:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.980+0000 7fa0443e7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa0381b7650 con 0x55849b0811b0 2024-07-26T11:18:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.980+0000 7fa036800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0811b0 0x55849b42a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.980+0000 7fa036800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0811b0 0x55849b42a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:47504/0 (socket says 172.21.15.134:47504) 2024-07-26T11:18:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.980+0000 7fa036800640 1 -- 172.21.15.134:0/1854981771 learned_addr learned my addr 172.21.15.134:0/1854981771 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.980+0000 7fa035400640 1 --2- 172.21.15.134:0/1854981771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0604d0 0x55849b429d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:07.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.981+0000 7fa036800640 1 -- 172.21.15.134:0/1854981771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0604d0 msgr2=0x55849b429d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.981+0000 7fa036800640 1 --2- 172.21.15.134:0/1854981771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0604d0 0x55849b429d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.981+0000 7fa036800640 1 -- 172.21.15.134:0/1854981771 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0582f0 msgr2=0x55849b430390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.981+0000 7fa036800640 1 --2- 172.21.15.134:0/1854981771 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0582f0 0x55849b430390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.981+0000 7fa036800640 1 -- 172.21.15.134:0/1854981771 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55849b42f680 con 0x55849b0811b0 2024-07-26T11:18:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.981+0000 7fa036800640 1 --2- 172.21.15.134:0/1854981771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0811b0 0x55849b42a240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa03000d5e0 tx=0x7fa030017b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=260586f851aa95fc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.981+0000 7fa022a00640 1 -- 172.21.15.134:0/1854981771 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa030025060 con 0x55849b0811b0 2024-07-26T11:18:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.982+0000 7fa022a00640 1 -- 172.21.15.134:0/1854981771 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa03000aa30 con 0x55849b0811b0 2024-07-26T11:18:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.982+0000 7fa022a00640 1 -- 172.21.15.134:0/1854981771 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa030025060 con 0x55849b0811b0 2024-07-26T11:18:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.983+0000 7fa0443e7740 1 -- 172.21.15.134:0/1854981771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0811b0 msgr2=0x55849b42a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.983+0000 7fa0443e7740 1 --2- 172.21.15.134:0/1854981771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0811b0 0x55849b42a240 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa03000d5e0 tx=0x7fa030017b60 comp rx=0 tx=0).stop 2024-07-26T11:18:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.983+0000 7fa0443e7740 1 -- 172.21.15.134:0/1854981771 shutdown_connections 2024-07-26T11:18:07.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.983+0000 7fa0443e7740 1 --2- 172.21.15.134:0/1854981771 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0811b0 0x55849b42a240 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.983+0000 7fa0443e7740 1 --2- 172.21.15.134:0/1854981771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0604d0 0x55849b429d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.983+0000 7fa0443e7740 1 --2- 172.21.15.134:0/1854981771 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0582f0 0x55849b430390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.983+0000 7fa0443e7740 1 -- 172.21.15.134:0/1854981771 >> 172.21.15.134:0/1854981771 conn(0x55849b075300 msgr2=0x55849b42fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.985+0000 7fa0443e7740 1 -- 172.21.15.134:0/1854981771 shutdown_connections 2024-07-26T11:18:07.988 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.986+0000 7fa0443e7740 1 -- 172.21.15.134:0/1854981771 wait complete. 2024-07-26T11:18:07.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.987+0000 7fa0443e7740 1 -- start start 2024-07-26T11:18:07.990 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.988+0000 7fa0443e7740 1 Processor -- start 2024-07-26T11:18:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.989+0000 7fa0443e7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0604d0 0x55849b4a3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.989+0000 7fa0443e7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0811b0 0x55849b4a3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.989+0000 7fa0443e7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0151c0 0x55849b4a4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.993 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.989+0000 7fa0443e7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa0381b83f0 con 0x55849b0811b0 2024-07-26T11:18:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.989+0000 7fa0443e7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa0381b7d20 con 0x55849b0604d0 2024-07-26T11:18:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.989+0000 7fa0443e7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa0381b4f00 con 0x55849b0151c0 2024-07-26T11:18:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.989+0000 7fa0443e7740 10 client.0 _ll_register_callbacks cb 0x7fa03001a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.990+0000 7fa035e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0604d0 0x55849b4a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.990+0000 7fa035e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0604d0 0x55849b4a3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43658/0 (socket says 172.21.15.134:43658) 2024-07-26T11:18:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.990+0000 7fa035e00640 1 -- 172.21.15.134:0/924942263 learned_addr learned my addr 172.21.15.134:0/924942263 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.990+0000 7fa035e00640 1 -- 172.21.15.134:0/924942263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0151c0 msgr2=0x55849b4a4100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:18:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.990+0000 7fa035e00640 1 --2- 172.21.15.134:0/924942263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0151c0 0x55849b4a4100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.990+0000 7fa035e00640 1 -- 172.21.15.134:0/924942263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0811b0 msgr2=0x55849b4a3bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:18:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.990+0000 7fa035e00640 1 --2- 172.21.15.134:0/924942263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0811b0 0x55849b4a3bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.990+0000 7fa035e00640 1 -- 172.21.15.134:0/924942263 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55849b4b9ee0 con 0x55849b0604d0 2024-07-26T11:18:07.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.991+0000 7fa035e00640 1 --2- 172.21.15.134:0/924942263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0604d0 0x55849b4a3680 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa028009230 tx=0x7fa028009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee404c0fc5873b7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.991+0000 7fa017e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:18:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.991+0000 7fa017e00640 1 -- 172.21.15.134:0/924942263 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa028035050 con 0x55849b0604d0 2024-07-26T11:18:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.991+0000 7fa017e00640 1 -- 172.21.15.134:0/924942263 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa028003800 con 0x55849b0604d0 2024-07-26T11:18:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.991+0000 7fa0443e7740 1 -- 172.21.15.134:0/924942263 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa03000aa30 con 0x55849b0604d0 2024-07-26T11:18:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.992+0000 7fa017e00640 1 -- 172.21.15.134:0/924942263 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa02806f050 con 0x55849b0604d0 2024-07-26T11:18:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.992+0000 7fa0443e7740 10 client.25916 fetch_fsmap learned FSMap version 31 2024-07-26T11:18:07.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.992+0000 7fa0443e7740 1 -- 172.21.15.134:0/924942263 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55849b4bb200 con 0x55849b0604d0 2024-07-26T11:18:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.993+0000 7fa017e00640 1 -- 172.21.15.134:0/924942263 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa03000aa30 con 0x55849b0604d0 2024-07-26T11:18:07.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.995+0000 7fa017e00640 1 -- 172.21.15.134:0/924942263 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7fa02803c440 con 0x55849b0604d0 2024-07-26T11:18:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.996+0000 7fa0443e7740 10 client.25916 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:18:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.996+0000 7fa0443e7740 10 client.25916 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:18:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.996+0000 7fa0443e7740 20 client.25916 populate_metadata read hostname 'smithi134' 2024-07-26T11:18:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.996+0000 7fa0443e7740 1 --2- 172.21.15.134:0/924942263 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55849b4bba10 0x55849b4bde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.996+0000 7fa0443e7740 4 client.25916 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:18:07.998 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.996+0000 7fa0443e7740 1 -- 172.21.15.134:0/924942263 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55849b396c60 con 0x55849b4bba10 2024-07-26T11:18:08.001 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:07.999+0000 7fa035400640 1 --2- 172.21.15.134:0/924942263 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55849b4bba10 0x55849b4bde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.001+0000 7fa017e00640 10 client.25916 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:08.005 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.001+0000 7fa035400640 1 --2- 172.21.15.134:0/924942263 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55849b4bba10 0x55849b4bde60 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a390d6ea65ec02c server_cookie=83ab84e7d4993405 in_seq=0 out_seq=0 2024-07-26T11:18:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:07 smithi193 ceph-mon[37473]: pgmap v544: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 277 MiB/s wr, 1.18k op/s 2024-07-26T11:18:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.086+0000 7fa017e00640 1 -- 172.21.15.134:0/924942263 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55849b396c60 con 0x55849b4bba10 2024-07-26T11:18:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.086+0000 7fa017e00640 10 client.25916 handle_command_reply: tid=0 2024-07-26T11:18:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.086+0000 7fa0443e7740 1 client.25916 shutdown 2024-07-26T11:18:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.095+0000 7fa0443e7740 1 -- 172.21.15.134:0/924942263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0604d0 msgr2=0x55849b4a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.095+0000 7fa0443e7740 1 --2- 172.21.15.134:0/924942263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0604d0 0x55849b4a3680 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa028009230 tx=0x7fa028009260 comp rx=0 tx=0).stop 2024-07-26T11:18:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.096+0000 7fa0443e7740 1 -- 172.21.15.134:0/924942263 shutdown_connections 2024-07-26T11:18:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.096+0000 7fa0443e7740 1 --2- 172.21.15.134:0/924942263 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55849b4bba10 0x55849b4bde60 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.096+0000 7fa017e00640 0 client.25916 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.096+0000 7fa0443e7740 1 --2- 172.21.15.134:0/924942263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55849b0151c0 0x55849b4a4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.096+0000 7fa0443e7740 1 --2- 172.21.15.134:0/924942263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55849b0811b0 0x55849b4a3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.096+0000 7fa0443e7740 1 --2- 172.21.15.134:0/924942263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55849b0604d0 0x55849b4a3680 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.096+0000 7fa0443e7740 1 -- 172.21.15.134:0/924942263 >> 172.21.15.134:0/924942263 conn(0x55849b075300 msgr2=0x55849b429d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.096+0000 7fa0443e7740 1 -- 172.21.15.134:0/924942263 shutdown_connections 2024-07-26T11:18:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.096+0000 7fa0443e7740 1 -- 172.21.15.134:0/924942263 wait complete. 2024-07-26T11:18:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.098+0000 7fa0443e7740 20 client.25916 trim_cache size 0 max 16384 2024-07-26T11:18:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:18:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:18:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:18:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:18:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.103+0000 7fa037200640 1 -- 172.21.15.134:0/1173355416 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa01c0b3e40 msgr2=0x7fa01c0b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.103+0000 7fa037200640 1 --2- 172.21.15.134:0/1173355416 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa01c0b3e40 0x7fa01c0b6350 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa028000c00 tx=0x7fa028005e30 comp rx=0 tx=0).stop 2024-07-26T11:18:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.103+0000 7fa037200640 1 -- 172.21.15.134:0/1173355416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa038108c70 msgr2=0x7fa038198080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.103+0000 7fa037200640 1 --2- 172.21.15.134:0/1173355416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa038108c70 0x7fa038198080 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa02400d750 tx=0x7fa02400dc10 comp rx=0 tx=0).stop 2024-07-26T11:18:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.103+0000 7fa037200640 1 -- 172.21.15.134:0/1173355416 shutdown_connections 2024-07-26T11:18:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.103+0000 7fa037200640 1 --2- 172.21.15.134:0/1173355416 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa01c0b3e40 0x7fa01c0b6350 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.103+0000 7fa037200640 1 --2- 172.21.15.134:0/1173355416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa038108c70 0x7fa038198080 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.103+0000 7fa037200640 1 --2- 172.21.15.134:0/1173355416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa0381095c0 0x7fa03819d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.103+0000 7fa037200640 1 --2- 172.21.15.134:0/1173355416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa038103340 0x7fa038197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.103+0000 7fa037200640 1 -- 172.21.15.134:0/1173355416 >> 172.21.15.134:0/1173355416 conn(0x7fa0380fed60 msgr2=0x7fa038101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.104+0000 7fa037200640 1 -- 172.21.15.134:0/1173355416 shutdown_connections 2024-07-26T11:18:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:08.104+0000 7fa037200640 1 -- 172.21.15.134:0/1173355416 wait complete. 2024-07-26T11:18:09.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.032+0000 7f6423400640 1 -- 172.21.15.134:0/1208818241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f641c077cc0 msgr2=0x7f641c113c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:09.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.032+0000 7f6423400640 1 --2- 172.21.15.134:0/1208818241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f641c077cc0 0x7f641c113c60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f640c01a910 tx=0x7f640c03b470 comp rx=0 tx=0).stop 2024-07-26T11:18:09.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.034+0000 7f6423400640 1 -- 172.21.15.134:0/1208818241 shutdown_connections 2024-07-26T11:18:09.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.034+0000 7f6423400640 1 --2- 172.21.15.134:0/1208818241 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f641c077cc0 0x7f641c113c60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.034+0000 7f6423400640 1 --2- 172.21.15.134:0/1208818241 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f641c1141a0 0x7f641c1165c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.034+0000 7f6423400640 1 --2- 172.21.15.134:0/1208818241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f641c077370 0x7f641c077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.034+0000 7f6423400640 1 -- 172.21.15.134:0/1208818241 >> 172.21.15.134:0/1208818241 conn(0x7f641c101210 msgr2=0x7f641c103660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:09.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.038+0000 7f6423400640 1 -- 172.21.15.134:0/1208818241 shutdown_connections 2024-07-26T11:18:09.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.039+0000 7f6423400640 1 -- 172.21.15.134:0/1208818241 wait complete. 2024-07-26T11:18:09.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.040+0000 7f6423400640 1 Processor -- start 2024-07-26T11:18:09.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.043+0000 7f6423400640 1 -- start start 2024-07-26T11:18:09.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.043+0000 7f6423400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f641c077370 0x7f641c113c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.043+0000 7f6423400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f641c077cc0 0x7f641c10ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.044+0000 7f6423400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f641c1141a0 0x7f641c1106f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.044+0000 7f6423400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f641c107da0 con 0x7f641c077cc0 2024-07-26T11:18:09.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.044+0000 7f6423400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f641c107c20 con 0x7f641c077370 2024-07-26T11:18:09.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.044+0000 7f6423400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f641c107f20 con 0x7f641c1141a0 2024-07-26T11:18:09.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.045+0000 7f6420a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f641c077370 0x7f641c113c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:09.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.045+0000 7f6420a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f641c077370 0x7f641c113c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43674/0 (socket says 172.21.15.134:43674) 2024-07-26T11:18:09.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.045+0000 7f6420a00640 1 -- 172.21.15.134:0/1954178137 learned_addr learned my addr 172.21.15.134:0/1954178137 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:09.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.045+0000 7f6420a00640 1 -- 172.21.15.134:0/1954178137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f641c1141a0 msgr2=0x7f641c1106f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-07-26T11:18:09.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.045+0000 7f6420a00640 1 --2- 172.21.15.134:0/1954178137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f641c1141a0 0x7f641c1106f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.045+0000 7f6420a00640 1 -- 172.21.15.134:0/1954178137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f641c077cc0 msgr2=0x7f641c10ff80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-07-26T11:18:09.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.045+0000 7f6420a00640 1 --2- 172.21.15.134:0/1954178137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f641c077cc0 0x7f641c10ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.045+0000 7f6420a00640 1 -- 172.21.15.134:0/1954178137 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f641c1a4020 con 0x7f641c077370 2024-07-26T11:18:09.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.046+0000 7f6420a00640 1 --2- 172.21.15.134:0/1954178137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f641c077370 0x7f641c113c60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f640400b450 tx=0x7f640400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:09.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.046+0000 7f641a000640 1 -- 172.21.15.134:0/1954178137 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6404015020 con 0x7f641c077370 2024-07-26T11:18:09.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.046+0000 7f641a000640 1 -- 172.21.15.134:0/1954178137 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6404011d40 con 0x7f641c077370 2024-07-26T11:18:09.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.046+0000 7f6423400640 1 -- 172.21.15.134:0/1954178137 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f641c1a4310 con 0x7f641c077370 2024-07-26T11:18:09.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.046+0000 7f641a000640 1 -- 172.21.15.134:0/1954178137 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640400f340 con 0x7f641c077370 2024-07-26T11:18:09.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.046+0000 7f6423400640 1 -- 172.21.15.134:0/1954178137 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f641c1a4870 con 0x7f641c077370 2024-07-26T11:18:09.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.051+0000 7f642a02c740 1 Processor -- start 2024-07-26T11:18:09.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.051+0000 7f642a02c740 1 -- start start 2024-07-26T11:18:09.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.052+0000 7f642a02c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d191f7070 0x558d1959be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.052+0000 7f642a02c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191ce1b0 0x558d1959c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.052+0000 7f642a02c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191d6390 0x558d1959c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.052+0000 7f642a02c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x558d19512bd0 con 0x558d191f7070 2024-07-26T11:18:09.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.052+0000 7f642a02c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558d195020d0 con 0x558d191ce1b0 2024-07-26T11:18:09.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.052+0000 7f642a02c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558d19506110 con 0x558d191d6390 2024-07-26T11:18:09.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.052+0000 7f6413e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191d6390 0x558d1959c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:09.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.052+0000 7f6413e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191d6390 0x558d1959c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:47520/0 (socket says 172.21.15.134:47520) 2024-07-26T11:18:09.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.052+0000 7f6413e00640 1 -- 172.21.15.134:0/826048868 learned_addr learned my addr 172.21.15.134:0/826048868 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:09.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.053+0000 7f6413e00640 1 -- 172.21.15.134:0/826048868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191ce1b0 msgr2=0x558d1959c370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:18:09.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.053+0000 7f6413e00640 1 --2- 172.21.15.134:0/826048868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191ce1b0 0x558d1959c370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.053+0000 7f6413e00640 1 -- 172.21.15.134:0/826048868 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d191f7070 msgr2=0x558d1959be30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:18:09.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.053+0000 7f6413e00640 1 --2- 172.21.15.134:0/826048868 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d191f7070 0x558d1959be30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.053+0000 7f6413e00640 1 -- 172.21.15.134:0/826048868 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d19262650 con 0x558d191d6390 2024-07-26T11:18:09.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.053+0000 7f6413e00640 1 --2- 172.21.15.134:0/826048868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191d6390 0x558d1959c8b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f640c006010 tx=0x7f640c004990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5165fd853837b189 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:09.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.053+0000 7f6411c00640 1 -- 172.21.15.134:0/826048868 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640c04f020 con 0x558d191d6390 2024-07-26T11:18:09.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.054+0000 7f6411c00640 1 -- 172.21.15.134:0/826048868 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f640c004b00 con 0x558d191d6390 2024-07-26T11:18:09.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.054+0000 7f6411c00640 1 -- 172.21.15.134:0/826048868 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640c046070 con 0x558d191d6390 2024-07-26T11:18:09.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.054+0000 7f641a000640 1 -- 172.21.15.134:0/1954178137 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f6404018020 con 0x7f641c077370 2024-07-26T11:18:09.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.055+0000 7f642a02c740 1 -- 172.21.15.134:0/826048868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191d6390 msgr2=0x558d1959c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:09.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.055+0000 7f642a02c740 1 --2- 172.21.15.134:0/826048868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191d6390 0x558d1959c8b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f640c006010 tx=0x7f640c004990 comp rx=0 tx=0).stop 2024-07-26T11:18:09.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.059+0000 7f642a02c740 1 -- 172.21.15.134:0/826048868 shutdown_connections 2024-07-26T11:18:09.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.059+0000 7f642a02c740 1 --2- 172.21.15.134:0/826048868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191d6390 0x558d1959c8b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.059+0000 7f642a02c740 1 --2- 172.21.15.134:0/826048868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191ce1b0 0x558d1959c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.059+0000 7f642a02c740 1 --2- 172.21.15.134:0/826048868 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d191f7070 0x558d1959be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.059+0000 7f642a02c740 1 -- 172.21.15.134:0/826048868 >> 172.21.15.134:0/826048868 conn(0x558d191eb460 msgr2=0x558d19599d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:09.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.059+0000 7f642a02c740 1 -- 172.21.15.134:0/826048868 shutdown_connections 2024-07-26T11:18:09.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.059+0000 7f642a02c740 1 -- 172.21.15.134:0/826048868 wait complete. 2024-07-26T11:18:09.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.060+0000 7f642a02c740 1 -- start start 2024-07-26T11:18:09.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.061+0000 7f642a02c740 1 Processor -- start 2024-07-26T11:18:09.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.061+0000 7f641a000640 1 --2- 172.21.15.134:0/1954178137 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64000b3d80 0x7f64000b6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.061+0000 7f641a000640 1 -- 172.21.15.134:0/1954178137 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f64040e3e10 con 0x7f641c077370 2024-07-26T11:18:09.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.062+0000 7f642a02c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191ce1b0 0x558d195a7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.062+0000 7f642a02c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d191d6390 0x558d195a7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.062+0000 7f6420a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191ce1b0 0x558d195a7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:09.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.062+0000 7f6420a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191ce1b0 0x558d195a7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:47524/0 (socket says 172.21.15.134:47524) 2024-07-26T11:18:09.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.063+0000 7f6420a00640 1 -- 172.21.15.134:0/2691945003 learned_addr learned my addr 172.21.15.134:0/2691945003 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:09.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.063+0000 7f642a02c740 1 --2- 172.21.15.134:0/2691945003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191f7070 0x558d195a7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.063+0000 7f6421400640 1 --2- 172.21.15.134:0/1954178137 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64000b3d80 0x7f64000b6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:09.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.063+0000 7f642a02c740 1 -- 172.21.15.134:0/2691945003 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x558d19365700 con 0x558d191d6390 2024-07-26T11:18:09.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.063+0000 7f642a02c740 1 -- 172.21.15.134:0/2691945003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558d1949e6a0 con 0x558d191f7070 2024-07-26T11:18:09.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.063+0000 7f642a02c740 1 -- 172.21.15.134:0/2691945003 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558d195aa600 con 0x558d191ce1b0 2024-07-26T11:18:09.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.063+0000 7f642a02c740 10 client.0 _ll_register_callbacks cb 0x558d194ae150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:09.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.063+0000 7f6421400640 1 --2- 172.21.15.134:0/2691945003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191f7070 0x558d195a7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:09.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.063+0000 7f6420a00640 1 -- 172.21.15.134:0/2691945003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191f7070 msgr2=0x558d195a7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:09.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.064+0000 7f6420a00640 1 --2- 172.21.15.134:0/2691945003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191f7070 0x558d195a7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.064+0000 7f6420a00640 1 -- 172.21.15.134:0/2691945003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d191d6390 msgr2=0x558d195a7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:09.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.064+0000 7f6420a00640 1 --2- 172.21.15.134:0/2691945003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d191d6390 0x558d195a7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.064+0000 7f6420a00640 1 -- 172.21.15.134:0/2691945003 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d194b5840 con 0x558d191ce1b0 2024-07-26T11:18:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.064+0000 7f6421400640 1 --2- 172.21.15.134:0/2691945003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191f7070 0x558d195a7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:18:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.064+0000 7f6420a00640 1 --2- 172.21.15.134:0/2691945003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191ce1b0 0x558d195a7430 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6404009fd0 tx=0x7f64040e1700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29befa10cb2ebde server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.065+0000 7f640a000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:18:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.065+0000 7f640a000640 1 -- 172.21.15.134:0/2691945003 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64040cd2f0 con 0x558d191ce1b0 2024-07-26T11:18:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.065+0000 7f640a000640 1 -- 172.21.15.134:0/2691945003 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f6404011890 con 0x558d191ce1b0 2024-07-26T11:18:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.065+0000 7f642a02c740 1 -- 172.21.15.134:0/2691945003 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f640c004b00 con 0x558d191ce1b0 2024-07-26T11:18:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.065+0000 7f640a000640 1 -- 172.21.15.134:0/2691945003 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f64040c5080 con 0x558d191ce1b0 2024-07-26T11:18:09.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.065+0000 7f642a02c740 10 client.25909 fetch_fsmap learned FSMap version 31 2024-07-26T11:18:09.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.065+0000 7f642a02c740 1 -- 172.21.15.134:0/2691945003 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558d195b6290 con 0x558d191ce1b0 2024-07-26T11:18:09.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.066+0000 7f640a000640 1 -- 172.21.15.134:0/2691945003 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f640c004b00 con 0x558d191ce1b0 2024-07-26T11:18:09.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.066+0000 7f640a000640 1 -- 172.21.15.134:0/2691945003 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f64040a60c0 con 0x558d191ce1b0 2024-07-26T11:18:09.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.066+0000 7f642a02c740 10 client.25909 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:18:09.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.066+0000 7f642a02c740 10 client.25909 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:18:09.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.066+0000 7f642a02c740 20 client.25909 populate_metadata read hostname 'smithi134' 2024-07-26T11:18:09.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.066+0000 7f642a02c740 1 --2- 172.21.15.134:0/2691945003 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558d191b59f0 0x558d195b9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.067+0000 7f6413e00640 1 --2- 172.21.15.134:0/2691945003 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558d191b59f0 0x558d195b9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:09.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.067+0000 7f6421400640 1 --2- 172.21.15.134:0/1954178137 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64000b3d80 0x7f64000b6290 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f641400aca0 tx=0x7f641400a490 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:09.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.067+0000 7f642a02c740 4 client.25909 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:18:09.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.067+0000 7f642a02c740 1 -- 172.21.15.134:0/2691945003 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558d19514b20 con 0x558d191b59f0 2024-07-26T11:18:09.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.074+0000 7f6413e00640 1 --2- 172.21.15.134:0/2691945003 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558d191b59f0 0x558d195b9760 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ecce74b4198ecc2 server_cookie=e12c23e2ad230014 in_seq=0 out_seq=0 2024-07-26T11:18:09.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.074+0000 7f640a000640 10 client.25909 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:09.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.076+0000 7f640a000640 1 -- 172.21.15.134:0/2691945003 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558d19514b20 con 0x558d191b59f0 2024-07-26T11:18:09.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.076+0000 7f640a000640 10 client.25909 handle_command_reply: tid=0 2024-07-26T11:18:09.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.077+0000 7f642a02c740 1 client.25909 shutdown 2024-07-26T11:18:09.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.077+0000 7f642a02c740 1 -- 172.21.15.134:0/2691945003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191ce1b0 msgr2=0x558d195a7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:09.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.077+0000 7f642a02c740 1 --2- 172.21.15.134:0/2691945003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191ce1b0 0x558d195a7430 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6404009fd0 tx=0x7f64040e1700 comp rx=0 tx=0).stop 2024-07-26T11:18:09.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.077+0000 7f642a02c740 1 -- 172.21.15.134:0/2691945003 shutdown_connections 2024-07-26T11:18:09.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.077+0000 7f642a02c740 1 --2- 172.21.15.134:0/2691945003 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558d191b59f0 0x558d195b9760 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.077+0000 7f642a02c740 1 --2- 172.21.15.134:0/2691945003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191f7070 0x558d195a7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.077+0000 7f642a02c740 1 --2- 172.21.15.134:0/2691945003 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d191d6390 0x558d195a7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.077+0000 7f642a02c740 1 --2- 172.21.15.134:0/2691945003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191ce1b0 0x558d195a7430 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.078+0000 7f642a02c740 1 -- 172.21.15.134:0/2691945003 >> 172.21.15.134:0/2691945003 conn(0x558d191eb460 msgr2=0x558d1959ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:09.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.078+0000 7f640a000640 0 client.25909 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:09.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.079+0000 7f642a02c740 1 -- 172.21.15.134:0/2691945003 shutdown_connections 2024-07-26T11:18:09.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.079+0000 7f642a02c740 1 -- 172.21.15.134:0/2691945003 wait complete. 2024-07-26T11:18:09.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.079+0000 7f642a02c740 20 client.25909 trim_cache size 0 max 16384 2024-07-26T11:18:09.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.108+0000 7f642a02c740 1 Processor -- start 2024-07-26T11:18:09.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.108+0000 7f642a02c740 1 -- start start 2024-07-26T11:18:09.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.109+0000 7f642a02c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191ce1b0 0x558d195a9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.109+0000 7f642a02c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d191d6390 0x558d1959d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.109+0000 7f642a02c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191f7070 0x558d195bef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.111+0000 7f642a02c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f641c1aea90 con 0x558d191d6390 2024-07-26T11:18:09.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.111+0000 7f642a02c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f641c1b0600 con 0x558d191f7070 2024-07-26T11:18:09.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.111+0000 7f642a02c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f641c1af160 con 0x558d191ce1b0 2024-07-26T11:18:09.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.111+0000 7f6421400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191f7070 0x558d195bef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:09.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.111+0000 7f6420a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191ce1b0 0x558d195a9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:09.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.111+0000 7f6420a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191ce1b0 0x558d195a9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:47532/0 (socket says 172.21.15.134:47532) 2024-07-26T11:18:09.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.111+0000 7f6420a00640 1 -- 172.21.15.134:0/2220256367 learned_addr learned my addr 172.21.15.134:0/2220256367 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:09.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.112+0000 7f6420a00640 1 -- 172.21.15.134:0/2220256367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191f7070 msgr2=0x558d195bef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:09.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.112+0000 7f6420a00640 1 --2- 172.21.15.134:0/2220256367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191f7070 0x558d195bef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.112+0000 7f6420a00640 1 -- 172.21.15.134:0/2220256367 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d191d6390 msgr2=0x558d1959d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:09.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.113+0000 7f6420a00640 1 --2- 172.21.15.134:0/2220256367 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d191d6390 0x558d1959d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.113+0000 7f6420a00640 1 -- 172.21.15.134:0/2220256367 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558d191b5c20 con 0x558d191ce1b0 2024-07-26T11:18:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.113+0000 7f6421400640 1 --2- 172.21.15.134:0/2220256367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191f7070 0x558d195bef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:18:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.113+0000 7f6420a00640 1 --2- 172.21.15.134:0/2220256367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191ce1b0 0x558d195a9310 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f64040a79c0 tx=0x7f6404000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c3dfd5962916d96 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.113+0000 7f6411200640 1 -- 172.21.15.134:0/2220256367 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64040cd2f0 con 0x558d191ce1b0 2024-07-26T11:18:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.114+0000 7f6411200640 1 -- 172.21.15.134:0/2220256367 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64040aa260 con 0x558d191ce1b0 2024-07-26T11:18:09.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.114+0000 7f6411200640 1 -- 172.21.15.134:0/2220256367 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6404011890 con 0x558d191ce1b0 2024-07-26T11:18:09.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.115+0000 7f642a02c740 1 -- 172.21.15.134:0/2220256367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191ce1b0 msgr2=0x558d195a9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:09.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.115+0000 7f642a02c740 1 --2- 172.21.15.134:0/2220256367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191ce1b0 0x558d195a9310 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f64040a79c0 tx=0x7f6404000bc0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.115+0000 7f642a02c740 1 -- 172.21.15.134:0/2220256367 shutdown_connections 2024-07-26T11:18:09.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.115+0000 7f642a02c740 1 --2- 172.21.15.134:0/2220256367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191f7070 0x558d195bef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.115+0000 7f642a02c740 1 --2- 172.21.15.134:0/2220256367 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d191d6390 0x558d1959d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.115+0000 7f642a02c740 1 --2- 172.21.15.134:0/2220256367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191ce1b0 0x558d195a9310 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.115+0000 7f642a02c740 1 -- 172.21.15.134:0/2220256367 >> 172.21.15.134:0/2220256367 conn(0x558d191eb460 msgr2=0x558d1959ca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:09.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.116+0000 7f642a02c740 1 -- 172.21.15.134:0/2220256367 shutdown_connections 2024-07-26T11:18:09.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.116+0000 7f642a02c740 1 -- 172.21.15.134:0/2220256367 wait complete. 2024-07-26T11:18:09.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.117+0000 7f642a02c740 1 -- start start 2024-07-26T11:18:09.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.117+0000 7f642a02c740 1 Processor -- start 2024-07-26T11:18:09.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.118+0000 7f642a02c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191ce1b0 0x558d195ac9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.118+0000 7f642a02c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191d6390 0x558d195acee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.118+0000 7f642a02c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d191f7070 0x558d195ad420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.118+0000 7f642a02c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f641c1aff00 con 0x558d191f7070 2024-07-26T11:18:09.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.118+0000 7f642a02c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f641c1af830 con 0x558d191ce1b0 2024-07-26T11:18:09.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.118+0000 7f642a02c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f641c1acab0 con 0x558d191d6390 2024-07-26T11:18:09.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.118+0000 7f642a02c740 10 client.0 _ll_register_callbacks cb 0x7f64040107a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:09.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.118+0000 7f6413e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191d6390 0x558d195acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:09.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.119+0000 7f6420a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191ce1b0 0x558d195ac9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:09.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.119+0000 7f6413e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191d6390 0x558d195acee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:47542/0 (socket says 172.21.15.134:47542) 2024-07-26T11:18:09.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.119+0000 7f6413e00640 1 -- 172.21.15.134:0/4217946571 learned_addr learned my addr 172.21.15.134:0/4217946571 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:09.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.119+0000 7f6420a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191ce1b0 0x558d195ac9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43694/0 (socket says 172.21.15.134:43694) 2024-07-26T11:18:09.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.119+0000 7f6413e00640 1 -- 172.21.15.134:0/4217946571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191ce1b0 msgr2=0x558d195ac9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:09.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.119+0000 7f6413e00640 1 --2- 172.21.15.134:0/4217946571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191ce1b0 0x558d195ac9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.119+0000 7f6413e00640 1 -- 172.21.15.134:0/4217946571 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d191f7070 msgr2=0x558d195ad420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:09.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.119+0000 7f6413e00640 1 --2- 172.21.15.134:0/4217946571 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d191f7070 0x558d195ad420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.119+0000 7f6413e00640 1 -- 172.21.15.134:0/4217946571 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558d19620120 con 0x558d191d6390 2024-07-26T11:18:09.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.120+0000 7f6413e00640 1 --2- 172.21.15.134:0/4217946571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191d6390 0x558d195acee0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f640c03ba60 tx=0x7f640c03bbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22e819c65c486c56 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:09.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.120+0000 7f640a000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:18:09.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.120+0000 7f640a000640 1 -- 172.21.15.134:0/4217946571 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640c062060 con 0x558d191d6390 2024-07-26T11:18:09.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.120+0000 7f640a000640 1 -- 172.21.15.134:0/4217946571 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f640c004f70 con 0x558d191d6390 2024-07-26T11:18:09.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.120+0000 7f642a02c740 1 -- 172.21.15.134:0/4217946571 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64040aa260 con 0x558d191d6390 2024-07-26T11:18:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.121+0000 7f640a000640 1 -- 172.21.15.134:0/4217946571 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f640c069050 con 0x558d191d6390 2024-07-26T11:18:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.121+0000 7f642a02c740 10 client.25915 fetch_fsmap learned FSMap version 31 2024-07-26T11:18:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.121+0000 7f642a02c740 1 -- 172.21.15.134:0/4217946571 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558d19621370 con 0x558d191d6390 2024-07-26T11:18:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.121+0000 7f640a000640 1 -- 172.21.15.134:0/4217946571 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f640c004b00 con 0x558d191d6390 2024-07-26T11:18:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.121+0000 7f640a000640 1 -- 172.21.15.134:0/4217946571 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f640c05d980 con 0x558d191d6390 2024-07-26T11:18:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.121+0000 7f642a02c740 10 client.25915 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:18:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.121+0000 7f642a02c740 10 client.25915 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:18:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.121+0000 7f642a02c740 20 client.25915 populate_metadata read hostname 'smithi134' 2024-07-26T11:18:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.121+0000 7f642a02c740 1 --2- 172.21.15.134:0/4217946571 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558d1918ada0 0x558d19623740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.121+0000 7f642a02c740 4 client.25915 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:18:09.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.121+0000 7f642a02c740 1 -- 172.21.15.134:0/4217946571 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f64040aa260 con 0x558d1918ada0 2024-07-26T11:18:09.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.123+0000 7f6420a00640 1 --2- 172.21.15.134:0/4217946571 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558d1918ada0 0x558d19623740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:09.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.124+0000 7f6420a00640 1 --2- 172.21.15.134:0/4217946571 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558d1918ada0 0x558d19623740 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0855a74a3d87505 server_cookie=9b8687becc5c6549 in_seq=0 out_seq=0 2024-07-26T11:18:09.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.124+0000 7f640a000640 10 client.25915 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:09.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.128+0000 7f640a000640 1 -- 172.21.15.134:0/4217946571 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f64040aa260 con 0x558d1918ada0 2024-07-26T11:18:09.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.128+0000 7f640a000640 10 client.25915 handle_command_reply: tid=0 2024-07-26T11:18:09.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.129+0000 7f642a02c740 1 client.25915 shutdown 2024-07-26T11:18:09.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.129+0000 7f642a02c740 1 -- 172.21.15.134:0/4217946571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191d6390 msgr2=0x558d195acee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:09.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.129+0000 7f642a02c740 1 --2- 172.21.15.134:0/4217946571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191d6390 0x558d195acee0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f640c03ba60 tx=0x7f640c03bbf0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.129+0000 7f642a02c740 1 -- 172.21.15.134:0/4217946571 shutdown_connections 2024-07-26T11:18:09.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.130+0000 7f642a02c740 1 --2- 172.21.15.134:0/4217946571 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558d1918ada0 0x558d19623740 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.130+0000 7f642a02c740 1 --2- 172.21.15.134:0/4217946571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558d191d6390 0x558d195acee0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.130+0000 7f642a02c740 1 --2- 172.21.15.134:0/4217946571 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558d191f7070 0x558d195ad420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.130+0000 7f642a02c740 1 --2- 172.21.15.134:0/4217946571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558d191ce1b0 0x558d195ac9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.130+0000 7f642a02c740 1 -- 172.21.15.134:0/4217946571 >> 172.21.15.134:0/4217946571 conn(0x558d191eb460 msgr2=0x558d195a90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:09.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.130+0000 7f640a000640 0 client.25915 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:09.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.130+0000 7f642a02c740 1 -- 172.21.15.134:0/4217946571 shutdown_connections 2024-07-26T11:18:09.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.130+0000 7f642a02c740 1 -- 172.21.15.134:0/4217946571 wait complete. 2024-07-26T11:18:09.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.131+0000 7f642a02c740 20 client.25915 trim_cache size 0 max 16384 2024-07-26T11:18:09.135 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:18:09.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.134+0000 7f6423400640 1 -- 172.21.15.134:0/1954178137 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64000b3d80 msgr2=0x7f64000b6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:09.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.134+0000 7f6423400640 1 --2- 172.21.15.134:0/1954178137 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64000b3d80 0x7f64000b6290 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f641400aca0 tx=0x7f641400a490 comp rx=0 tx=0).stop 2024-07-26T11:18:09.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.134+0000 7f6423400640 1 -- 172.21.15.134:0/1954178137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f641c077370 msgr2=0x7f641c113c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:09.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.134+0000 7f6423400640 1 --2- 172.21.15.134:0/1954178137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f641c077370 0x7f641c113c60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f640400b450 tx=0x7f640400b910 comp rx=0 tx=0).stop 2024-07-26T11:18:09.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.134+0000 7f6423400640 1 -- 172.21.15.134:0/1954178137 shutdown_connections 2024-07-26T11:18:09.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.134+0000 7f6423400640 1 --2- 172.21.15.134:0/1954178137 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f64000b3d80 0x7f64000b6290 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.134+0000 7f6423400640 1 --2- 172.21.15.134:0/1954178137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f641c1141a0 0x7f641c1106f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.134+0000 7f6423400640 1 --2- 172.21.15.134:0/1954178137 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f641c077cc0 0x7f641c10ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.134+0000 7f6423400640 1 --2- 172.21.15.134:0/1954178137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f641c077370 0x7f641c113c60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:09.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.134+0000 7f6423400640 1 -- 172.21.15.134:0/1954178137 >> 172.21.15.134:0/1954178137 conn(0x7f641c101210 msgr2=0x7f641c116250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:09.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.135+0000 7f6423400640 1 -- 172.21.15.134:0/1954178137 shutdown_connections 2024-07-26T11:18:09.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:09.135+0000 7f6423400640 1 -- 172.21.15.134:0/1954178137 wait complete. 2024-07-26T11:18:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:09 smithi134 ceph-mon[31966]: pgmap v545: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 272 MiB/s wr, 1.30k op/s 2024-07-26T11:18:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:09 smithi189 ceph-mon[36790]: pgmap v545: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 272 MiB/s wr, 1.30k op/s 2024-07-26T11:18:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:09 smithi193 ceph-mon[37473]: pgmap v545: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 272 MiB/s wr, 1.30k op/s 2024-07-26T11:18:11.285 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T11:18:11.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T11:18:11.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:11 smithi134 ceph-mon[31966]: pgmap v546: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 297 MiB/s wr, 1.48k op/s 2024-07-26T11:18:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:11 smithi189 ceph-mon[36790]: pgmap v546: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 297 MiB/s wr, 1.48k op/s 2024-07-26T11:18:11.948 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:18:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:11 smithi193 ceph-mon[37473]: pgmap v546: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 297 MiB/s wr, 1.48k op/s 2024-07-26T11:18:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:13 smithi134 ceph-mon[31966]: pgmap v547: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 247 MiB/s wr, 1.27k op/s 2024-07-26T11:18:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:13 smithi189 ceph-mon[36790]: pgmap v547: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 247 MiB/s wr, 1.27k op/s 2024-07-26T11:18:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:13 smithi193 ceph-mon[37473]: pgmap v547: 129 pgs: 129 active+clean; 10 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 247 MiB/s wr, 1.27k op/s 2024-07-26T11:18:14.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.663+0000 7f3d8c000640 1 -- 172.21.15.134:0/1373777950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d8410b120 msgr2=0x7f3d8410b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:14.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.663+0000 7f3d8c000640 1 --2- 172.21.15.134:0/1373777950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d8410b120 0x7f3d8410b530 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009900 tx=0x7f3d7002a400 comp rx=0 tx=0).stop 2024-07-26T11:18:14.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.665+0000 7f3d8c000640 1 -- 172.21.15.134:0/1373777950 shutdown_connections 2024-07-26T11:18:14.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.665+0000 7f3d8c000640 1 --2- 172.21.15.134:0/1373777950 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d8410b120 0x7f3d8410b530 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.665+0000 7f3d8c000640 1 --2- 172.21.15.134:0/1373777950 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d8410ba70 0x7f3d841122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.665+0000 7f3d8c000640 1 --2- 172.21.15.134:0/1373777950 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d841057e0 0x7f3d84105bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.665+0000 7f3d8c000640 1 -- 172.21.15.134:0/1373777950 >> 172.21.15.134:0/1373777950 conn(0x7f3d841011f0 msgr2=0x7f3d84103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:14.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.665+0000 7f3d8c000640 1 -- 172.21.15.134:0/1373777950 shutdown_connections 2024-07-26T11:18:14.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.666+0000 7f3d8c000640 1 -- 172.21.15.134:0/1373777950 wait complete. 2024-07-26T11:18:14.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.667+0000 7f3d8c000640 1 Processor -- start 2024-07-26T11:18:14.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.667+0000 7f3d8c000640 1 -- start start 2024-07-26T11:18:14.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.668+0000 7f3d8c000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d841057e0 0x7f3d840748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.668+0000 7f3d8c000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d8410b120 0x7f3d8406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.668+0000 7f3d8c000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d8410ba70 0x7f3d8406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.668+0000 7f3d8c000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3d84115730 con 0x7f3d8410ba70 2024-07-26T11:18:14.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.668+0000 7f3d8c000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3d841155b0 con 0x7f3d8410b120 2024-07-26T11:18:14.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.668+0000 7f3d8c000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3d841158b0 con 0x7f3d841057e0 2024-07-26T11:18:14.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.668+0000 7f3d89600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d841057e0 0x7f3d840748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.668+0000 7f3d88c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d8410b120 0x7f3d8406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.668+0000 7f3d89600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d841057e0 0x7f3d840748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59342/0 (socket says 172.21.15.134:59342) 2024-07-26T11:18:14.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.668+0000 7f3d89600640 1 -- 172.21.15.134:0/429716671 learned_addr learned my addr 172.21.15.134:0/429716671 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:14.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.669+0000 7f3d89600640 1 -- 172.21.15.134:0/429716671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d8410b120 msgr2=0x7f3d8406db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:14.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.669+0000 7f3d89600640 1 --2- 172.21.15.134:0/429716671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d8410b120 0x7f3d8406db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.669+0000 7f3d89600640 1 -- 172.21.15.134:0/429716671 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d8410ba70 msgr2=0x7f3d8406e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:14.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.669+0000 7f3d89600640 1 --2- 172.21.15.134:0/429716671 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d8410ba70 0x7f3d8406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.669+0000 7f3d89600640 1 -- 172.21.15.134:0/429716671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d8406e8a0 con 0x7f3d841057e0 2024-07-26T11:18:14.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.669+0000 7f3d89600640 1 --2- 172.21.15.134:0/429716671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d841057e0 0x7f3d840748a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800c920 tx=0x7f3d7800cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:14.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.670+0000 7f3d82000640 1 -- 172.21.15.134:0/429716671 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78007c70 con 0x7f3d841057e0 2024-07-26T11:18:14.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.670+0000 7f3d82000640 1 -- 172.21.15.134:0/429716671 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d78007e10 con 0x7f3d841057e0 2024-07-26T11:18:14.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.670+0000 7f3d82000640 1 -- 172.21.15.134:0/429716671 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7800e7f0 con 0x7f3d841057e0 2024-07-26T11:18:14.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.671+0000 7f3d8c000640 1 -- 172.21.15.134:0/429716671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d84073570 con 0x7f3d841057e0 2024-07-26T11:18:14.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.671+0000 7f3d8c000640 1 -- 172.21.15.134:0/429716671 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d84073a30 con 0x7f3d841057e0 2024-07-26T11:18:14.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.675+0000 7f3d92c9a740 1 Processor -- start 2024-07-26T11:18:14.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.675+0000 7f3d92c9a740 1 -- start start 2024-07-26T11:18:14.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.676+0000 7f3d92c9a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51324350 0x562b516c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.676+0000 7f3d92c9a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b512fb490 0x562b516c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.676+0000 7f3d92c9a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51303670 0x562b516c9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.676+0000 7f3d92c9a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562b5163fbf0 con 0x562b51303670 2024-07-26T11:18:14.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.676+0000 7f3d92c9a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562b5162f0f0 con 0x562b51324350 2024-07-26T11:18:14.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.676+0000 7f3d92c9a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562b51633130 con 0x562b512fb490 2024-07-26T11:18:14.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.676+0000 7f3d8a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51324350 0x562b516c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.676+0000 7f3d88c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51303670 0x562b516c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.676+0000 7f3d88c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51303670 0x562b516c9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:53692/0 (socket says 172.21.15.134:53692) 2024-07-26T11:18:14.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.676+0000 7f3d88c00640 1 -- 172.21.15.134:0/1276286189 learned_addr learned my addr 172.21.15.134:0/1276286189 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:14.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.677+0000 7f3d82000640 1 -- 172.21.15.134:0/429716671 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f3d7800fb00 con 0x7f3d841057e0 2024-07-26T11:18:14.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.677+0000 7f3d8a000640 1 -- 172.21.15.134:0/1276286189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b512fb490 msgr2=0x562b516c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:14.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.677+0000 7f3d89600640 1 --2- 172.21.15.134:0/1276286189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b512fb490 0x562b516c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.677+0000 7f3d8a000640 1 --2- 172.21.15.134:0/1276286189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b512fb490 0x562b516c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.677+0000 7f3d8a000640 1 -- 172.21.15.134:0/1276286189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51303670 msgr2=0x562b516c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:14.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.677+0000 7f3d8a000640 1 --2- 172.21.15.134:0/1276286189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51303670 0x562b516c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.678+0000 7f3d8a000640 1 -- 172.21.15.134:0/1276286189 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b5138f930 con 0x562b51324350 2024-07-26T11:18:14.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.678+0000 7f3d89600640 1 --2- 172.21.15.134:0/1276286189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b512fb490 0x562b516c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:18:14.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.678+0000 7f3d8a000640 1 --2- 172.21.15.134:0/1276286189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51324350 0x562b516c8d90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c005fd0 tx=0x7f3d7c00e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d05b331e6588c887 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:14.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.679+0000 7f3d76600640 1 -- 172.21.15.134:0/1276286189 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c016070 con 0x562b51324350 2024-07-26T11:18:14.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.679+0000 7f3d88c00640 1 --2- 172.21.15.134:0/1276286189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51303670 0x562b516c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:18:14.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.679+0000 7f3d76600640 1 -- 172.21.15.134:0/1276286189 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d7c005020 con 0x562b51324350 2024-07-26T11:18:14.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.679+0000 7f3d76600640 1 -- 172.21.15.134:0/1276286189 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c0116d0 con 0x562b51324350 2024-07-26T11:18:14.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.680+0000 7f3d82000640 1 --2- 172.21.15.134:0/429716671 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d6c0b3dc0 0x7f3d6c0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.680+0000 7f3d82000640 1 -- 172.21.15.134:0/429716671 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3d780e4370 con 0x7f3d841057e0 2024-07-26T11:18:14.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.680+0000 7f3d92c9a740 1 -- 172.21.15.134:0/1276286189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51324350 msgr2=0x562b516c8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:14.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.680+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/1276286189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51324350 0x562b516c8d90 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c005fd0 tx=0x7f3d7c00e6c0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.681+0000 7f3d92c9a740 1 -- 172.21.15.134:0/1276286189 shutdown_connections 2024-07-26T11:18:14.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.681+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/1276286189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b512fb490 0x562b516c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.681+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/1276286189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51303670 0x562b516c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.681+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/1276286189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51324350 0x562b516c8d90 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.681+0000 7f3d92c9a740 1 -- 172.21.15.134:0/1276286189 >> 172.21.15.134:0/1276286189 conn(0x562b51318740 msgr2=0x562b516c6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:14.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.681+0000 7f3d89600640 1 --2- 172.21.15.134:0/429716671 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d6c0b3dc0 0x7f3d6c0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.682+0000 7f3d89600640 1 --2- 172.21.15.134:0/429716671 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d6c0b3dc0 0x7f3d6c0b62d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3d780e17f0 tx=0x7f3d78000bc0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:14.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.682+0000 7f3d92c9a740 1 -- 172.21.15.134:0/1276286189 shutdown_connections 2024-07-26T11:18:14.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.682+0000 7f3d92c9a740 1 -- 172.21.15.134:0/1276286189 wait complete. 2024-07-26T11:18:14.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.683+0000 7f3d92c9a740 1 -- start start 2024-07-26T11:18:14.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.683+0000 7f3d92c9a740 1 Processor -- start 2024-07-26T11:18:14.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.683+0000 7f3d92c9a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b512fb490 0x562b516dd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.683+0000 7f3d92c9a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b51303670 0x562b516ddb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.684+0000 7f3d92c9a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51324350 0x562b516e2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.684+0000 7f3d92c9a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562b51492890 con 0x562b51324350 2024-07-26T11:18:14.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.684+0000 7f3d92c9a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562b515cb330 con 0x562b512fb490 2024-07-26T11:18:14.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.684+0000 7f3d92c9a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562b516d7570 con 0x562b51303670 2024-07-26T11:18:14.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.684+0000 7f3d92c9a740 10 client.0 _ll_register_callbacks cb 0x562b515f7010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:14.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.684+0000 7f3d8a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b512fb490 0x562b516dd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.684+0000 7f3d8a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b512fb490 0x562b516dd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35268/0 (socket says 172.21.15.134:35268) 2024-07-26T11:18:14.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.684+0000 7f3d8a000640 1 -- 172.21.15.134:0/1542615329 learned_addr learned my addr 172.21.15.134:0/1542615329 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:14.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.684+0000 7f3d8a000640 1 --2- 172.21.15.134:0/1542615329 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51324350 0x562b516e2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.684+0000 7f3d88c00640 1 --2- 172.21.15.134:0/1542615329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b51303670 0x562b516ddb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.684+0000 7f3d8a000640 1 -- 172.21.15.134:0/1542615329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b51303670 msgr2=0x562b516ddb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:14.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.684+0000 7f3d8a000640 1 --2- 172.21.15.134:0/1542615329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b51303670 0x562b516ddb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.684+0000 7f3d8a000640 1 -- 172.21.15.134:0/1542615329 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51324350 msgr2=0x562b516e2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:14.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.684+0000 7f3d8a000640 1 --2- 172.21.15.134:0/1542615329 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51324350 0x562b516e2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.684+0000 7f3d8a000640 1 -- 172.21.15.134:0/1542615329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b516e31d0 con 0x562b512fb490 2024-07-26T11:18:14.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.684+0000 7f3d88c00640 1 --2- 172.21.15.134:0/1542615329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b51303670 0x562b516ddb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:18:14.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.685+0000 7f3d8a000640 1 --2- 172.21.15.134:0/1542615329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b512fb490 0x562b516dd560 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c004d80 tx=0x7f3d7c0059c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3404a4bb69e299f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:14.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.685+0000 7f3d6aa00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:18:14.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.685+0000 7f3d6aa00640 1 -- 172.21.15.134:0/1542615329 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c016030 con 0x562b512fb490 2024-07-26T11:18:14.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.686+0000 7f3d6aa00640 1 -- 172.21.15.134:0/1542615329 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3d7c011870 con 0x562b512fb490 2024-07-26T11:18:14.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.686+0000 7f3d92c9a740 1 -- 172.21.15.134:0/1542615329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d7c005020 con 0x562b512fb490 2024-07-26T11:18:14.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.686+0000 7f3d6aa00640 1 -- 172.21.15.134:0/1542615329 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3d7c011a70 con 0x562b512fb490 2024-07-26T11:18:14.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.686+0000 7f3d92c9a740 10 client.25931 fetch_fsmap learned FSMap version 31 2024-07-26T11:18:14.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.686+0000 7f3d92c9a740 1 -- 172.21.15.134:0/1542615329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562b516ebbe0 con 0x562b512fb490 2024-07-26T11:18:14.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.687+0000 7f3d6aa00640 1 -- 172.21.15.134:0/1542615329 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d7c005020 con 0x562b512fb490 2024-07-26T11:18:14.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.687+0000 7f3d6aa00640 1 -- 172.21.15.134:0/1542615329 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f3d7c00f040 con 0x562b512fb490 2024-07-26T11:18:14.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.687+0000 7f3d92c9a740 10 client.25931 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:18:14.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.687+0000 7f3d92c9a740 10 client.25931 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:18:14.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.687+0000 7f3d92c9a740 20 client.25931 populate_metadata read hostname 'smithi134' 2024-07-26T11:18:14.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.688+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/1542615329 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562b512e2cd0 0x562b516eefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.688+0000 7f3d92c9a740 4 client.25931 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:18:14.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.688+0000 7f3d92c9a740 1 -- 172.21.15.134:0/1542615329 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b51641b40 con 0x562b512e2cd0 2024-07-26T11:18:14.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.688+0000 7f3d88c00640 1 --2- 172.21.15.134:0/1542615329 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562b512e2cd0 0x562b516eefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.690+0000 7f3d88c00640 1 --2- 172.21.15.134:0/1542615329 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562b512e2cd0 0x562b516eefa0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=796d1904ed588f3e server_cookie=a9cd5406a75c58be in_seq=0 out_seq=0 2024-07-26T11:18:14.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.690+0000 7f3d6aa00640 10 client.25931 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:14.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.691+0000 7f3d6aa00640 1 -- 172.21.15.134:0/1542615329 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562b51641b40 con 0x562b512e2cd0 2024-07-26T11:18:14.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.691+0000 7f3d6aa00640 10 client.25931 handle_command_reply: tid=0 2024-07-26T11:18:14.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.691+0000 7f3d92c9a740 1 client.25931 shutdown 2024-07-26T11:18:14.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.692+0000 7f3d92c9a740 1 -- 172.21.15.134:0/1542615329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b512fb490 msgr2=0x562b516dd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:14.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.692+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/1542615329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b512fb490 0x562b516dd560 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c004d80 tx=0x7f3d7c0059c0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.692+0000 7f3d92c9a740 1 -- 172.21.15.134:0/1542615329 shutdown_connections 2024-07-26T11:18:14.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.692+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/1542615329 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562b512e2cd0 0x562b516eefa0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.692+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/1542615329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b51303670 0x562b516ddb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.692+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/1542615329 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51324350 0x562b516e2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.692+0000 7f3d6aa00640 0 client.25931 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:14.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.692+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/1542615329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b512fb490 0x562b516dd560 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.692+0000 7f3d92c9a740 1 -- 172.21.15.134:0/1542615329 >> 172.21.15.134:0/1542615329 conn(0x562b51318740 msgr2=0x562b516cbf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:14.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.692+0000 7f3d92c9a740 1 -- 172.21.15.134:0/1542615329 shutdown_connections 2024-07-26T11:18:14.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.692+0000 7f3d92c9a740 1 -- 172.21.15.134:0/1542615329 wait complete. 2024-07-26T11:18:14.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.693+0000 7f3d92c9a740 20 client.25931 trim_cache size 0 max 16384 2024-07-26T11:18:14.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.721+0000 7f3d92c9a740 1 Processor -- start 2024-07-26T11:18:14.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.722+0000 7f3d92c9a740 1 -- start start 2024-07-26T11:18:14.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.722+0000 7f3d92c9a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b512fb490 0x562b516d6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.723+0000 7f3d8a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b512fb490 0x562b516d6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.723+0000 7f3d8a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b512fb490 0x562b516d6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59378/0 (socket says 172.21.15.134:59378) 2024-07-26T11:18:14.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.723+0000 7f3d92c9a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51303670 0x562b516ca780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.723+0000 7f3d92c9a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51324350 0x562b516d87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.723+0000 7f3d92c9a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3d841b6fc0 con 0x562b51303670 2024-07-26T11:18:14.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.723+0000 7f3d92c9a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3d841b8b30 con 0x562b51324350 2024-07-26T11:18:14.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.723+0000 7f3d92c9a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3d841b7690 con 0x562b512fb490 2024-07-26T11:18:14.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.723+0000 7f3d8a000640 1 -- 172.21.15.134:0/2535816428 learned_addr learned my addr 172.21.15.134:0/2535816428 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:14.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.724+0000 7f3d88c00640 1 --2- 172.21.15.134:0/2535816428 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51303670 0x562b516ca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.724+0000 7f3d8a000640 1 --2- 172.21.15.134:0/2535816428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51324350 0x562b516d87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.724+0000 7f3d8a000640 1 -- 172.21.15.134:0/2535816428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51324350 msgr2=0x562b516d87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:14.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.724+0000 7f3d8a000640 1 --2- 172.21.15.134:0/2535816428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51324350 0x562b516d87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.724+0000 7f3d8a000640 1 -- 172.21.15.134:0/2535816428 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51303670 msgr2=0x562b516ca780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:14.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.724+0000 7f3d8a000640 1 --2- 172.21.15.134:0/2535816428 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51303670 0x562b516ca780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.724+0000 7f3d8a000640 1 -- 172.21.15.134:0/2535816428 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b516cbbf0 con 0x562b512fb490 2024-07-26T11:18:14.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.725+0000 7f3d88c00640 1 --2- 172.21.15.134:0/2535816428 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51303670 0x562b516ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:18:14.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.725+0000 7f3d8a000640 1 --2- 172.21.15.134:0/2535816428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b512fb490 0x562b516d6890 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c000f80 tx=0x7f3d7c00e6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6af8db20443c39f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:14.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.725+0000 7f3d75c00640 1 -- 172.21.15.134:0/2535816428 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c016040 con 0x562b512fb490 2024-07-26T11:18:14.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.725+0000 7f3d75c00640 1 -- 172.21.15.134:0/2535816428 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d7c011850 con 0x562b512fb490 2024-07-26T11:18:14.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.725+0000 7f3d75c00640 1 -- 172.21.15.134:0/2535816428 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c003bd0 con 0x562b512fb490 2024-07-26T11:18:14.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.727+0000 7f3d92c9a740 1 -- 172.21.15.134:0/2535816428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b512fb490 msgr2=0x562b516d6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:14.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.727+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/2535816428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b512fb490 0x562b516d6890 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c000f80 tx=0x7f3d7c00e6c0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.727+0000 7f3d92c9a740 1 -- 172.21.15.134:0/2535816428 shutdown_connections 2024-07-26T11:18:14.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.727+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/2535816428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51324350 0x562b516d87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.727+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/2535816428 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b51303670 0x562b516ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.727+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/2535816428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b512fb490 0x562b516d6890 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.727+0000 7f3d92c9a740 1 -- 172.21.15.134:0/2535816428 >> 172.21.15.134:0/2535816428 conn(0x562b51318740 msgr2=0x562b516e1930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:14.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.728+0000 7f3d92c9a740 1 -- 172.21.15.134:0/2535816428 shutdown_connections 2024-07-26T11:18:14.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.728+0000 7f3d92c9a740 1 -- 172.21.15.134:0/2535816428 wait complete. 2024-07-26T11:18:14.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.729+0000 7f3d92c9a740 1 -- start start 2024-07-26T11:18:14.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.729+0000 7f3d92c9a740 1 Processor -- start 2024-07-26T11:18:14.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.730+0000 7f3d92c9a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b512fb490 0x562b516e9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.731+0000 7f3d92c9a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51303670 0x562b516e6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.731+0000 7f3d8a000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b512fb490 0x562b516e9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.731+0000 7f3d8a000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b512fb490 0x562b516e9150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:53722/0 (socket says 172.21.15.134:53722) 2024-07-26T11:18:14.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.731+0000 7f3d8a000640 1 -- 172.21.15.134:0/3902323594 learned_addr learned my addr 172.21.15.134:0/3902323594 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:14.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.732+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/3902323594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b51324350 0x562b516e6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.732+0000 7f3d92c9a740 1 -- 172.21.15.134:0/3902323594 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3d841b8430 con 0x562b512fb490 2024-07-26T11:18:14.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.732+0000 7f3d92c9a740 1 -- 172.21.15.134:0/3902323594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3d841b7d60 con 0x562b51303670 2024-07-26T11:18:14.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.732+0000 7f3d92c9a740 1 -- 172.21.15.134:0/3902323594 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3d841b4fe0 con 0x562b51324350 2024-07-26T11:18:14.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.733+0000 7f3d8a000640 1 --2- 172.21.15.134:0/3902323594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b51324350 0x562b516e6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.733+0000 7f3d88c00640 1 --2- 172.21.15.134:0/3902323594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51303670 0x562b516e6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.733+0000 7f3d92c9a740 10 client.0 _ll_register_callbacks cb 0x7f3d70009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:14.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.734+0000 7f3d8a000640 1 -- 172.21.15.134:0/3902323594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51303670 msgr2=0x562b516e6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:14.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.734+0000 7f3d8a000640 1 --2- 172.21.15.134:0/3902323594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51303670 0x562b516e6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.734+0000 7f3d8a000640 1 -- 172.21.15.134:0/3902323594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b512fb490 msgr2=0x562b516e9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:14.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.735+0000 7f3d8a000640 1 --2- 172.21.15.134:0/3902323594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b512fb490 0x562b516e9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.735+0000 7f3d8a000640 1 -- 172.21.15.134:0/3902323594 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b516e7250 con 0x562b51324350 2024-07-26T11:18:14.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.735+0000 7f3d88c00640 1 --2- 172.21.15.134:0/3902323594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51303670 0x562b516e6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:18:14.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.736+0000 7f3d8a000640 1 --2- 172.21.15.134:0/3902323594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b51324350 0x562b516e6b60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009640 tx=0x7f3d7c009540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7f0d52d68b6b185 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:14.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.736+0000 7f3d6aa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:18:14.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.736+0000 7f3d6aa00640 1 -- 172.21.15.134:0/3902323594 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c016070 con 0x562b51324350 2024-07-26T11:18:14.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.736+0000 7f3d6aa00640 1 -- 172.21.15.134:0/3902323594 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3d7c03a040 con 0x562b51324350 2024-07-26T11:18:14.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.736+0000 7f3d92c9a740 1 -- 172.21.15.134:0/3902323594 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d7c011850 con 0x562b51324350 2024-07-26T11:18:14.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.737+0000 7f3d6aa00640 1 -- 172.21.15.134:0/3902323594 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3d7c0348e0 con 0x562b51324350 2024-07-26T11:18:14.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.737+0000 7f3d92c9a740 10 client.25930 fetch_fsmap learned FSMap version 31 2024-07-26T11:18:14.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.737+0000 7f3d92c9a740 1 -- 172.21.15.134:0/3902323594 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562b517572f0 con 0x562b51324350 2024-07-26T11:18:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.737+0000 7f3d6aa00640 1 -- 172.21.15.134:0/3902323594 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d7c011850 con 0x562b51324350 2024-07-26T11:18:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.737+0000 7f3d6aa00640 1 -- 172.21.15.134:0/3902323594 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f3d7c025050 con 0x562b51324350 2024-07-26T11:18:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.738+0000 7f3d92c9a740 10 client.25930 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:18:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.738+0000 7f3d92c9a740 10 client.25930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:18:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.738+0000 7f3d92c9a740 20 client.25930 populate_metadata read hostname 'smithi134' 2024-07-26T11:18:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.738+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/3902323594 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562b512b8080 0x562b51759a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:14.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.738+0000 7f3d92c9a740 4 client.25930 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T11:18:14.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.738+0000 7f3d92c9a740 1 -- 172.21.15.134:0/3902323594 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x562b5163d5e0 con 0x562b512b8080 2024-07-26T11:18:14.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.738+0000 7f3d88c00640 1 --2- 172.21.15.134:0/3902323594 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562b512b8080 0x562b51759a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:14.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.739+0000 7f3d88c00640 1 --2- 172.21.15.134:0/3902323594 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562b512b8080 0x562b51759a10 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb442c65a47026d4 server_cookie=a75272c40e92317a in_seq=0 out_seq=0 2024-07-26T11:18:14.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.739+0000 7f3d6aa00640 10 client.25930 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.742+0000 7f3d6aa00640 1 -- 172.21.15.134:0/3902323594 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x562b5163d5e0 con 0x562b512b8080 2024-07-26T11:18:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.742+0000 7f3d6aa00640 10 client.25930 handle_command_reply: tid=0 2024-07-26T11:18:14.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.742+0000 7f3d92c9a740 1 client.25930 shutdown 2024-07-26T11:18:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.742+0000 7f3d92c9a740 1 -- 172.21.15.134:0/3902323594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b51324350 msgr2=0x562b516e6b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.742+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/3902323594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b51324350 0x562b516e6b60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009640 tx=0x7f3d7c009540 comp rx=0 tx=0).stop 2024-07-26T11:18:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.742+0000 7f3d92c9a740 1 -- 172.21.15.134:0/3902323594 shutdown_connections 2024-07-26T11:18:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.742+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/3902323594 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562b512b8080 0x562b51759a10 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.742+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/3902323594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562b51324350 0x562b516e6b60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.742+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/3902323594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562b51303670 0x562b516e6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.742+0000 7f3d92c9a740 1 --2- 172.21.15.134:0/3902323594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562b512fb490 0x562b516e9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.742+0000 7f3d92c9a740 1 -- 172.21.15.134:0/3902323594 >> 172.21.15.134:0/3902323594 conn(0x562b51318740 msgr2=0x562b516e22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.743+0000 7f3d6aa00640 0 client.25930 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.743+0000 7f3d92c9a740 1 -- 172.21.15.134:0/3902323594 shutdown_connections 2024-07-26T11:18:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.743+0000 7f3d92c9a740 1 -- 172.21.15.134:0/3902323594 wait complete. 2024-07-26T11:18:14.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.743+0000 7f3d92c9a740 20 client.25930 trim_cache size 0 max 16384 2024-07-26T11:18:14.746 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:18:14.746 INFO:teuthology.orchestra.run.smithi134.stdout: "return_code": 0, 2024-07-26T11:18:14.746 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_tag": "12d59c8d-a2aa-4edb-84ca-3f7d476e7766", 2024-07-26T11:18:14.747 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "asynchronous" 2024-07-26T11:18:14.747 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:18:14.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.745+0000 7f3d8c000640 1 -- 172.21.15.134:0/429716671 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d6c0b3dc0 msgr2=0x7f3d6c0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:14.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.745+0000 7f3d8c000640 1 --2- 172.21.15.134:0/429716671 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d6c0b3dc0 0x7f3d6c0b62d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3d780e17f0 tx=0x7f3d78000bc0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.745+0000 7f3d8c000640 1 -- 172.21.15.134:0/429716671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d841057e0 msgr2=0x7f3d840748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:14.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.745+0000 7f3d8c000640 1 --2- 172.21.15.134:0/429716671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d841057e0 0x7f3d840748a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800c920 tx=0x7f3d7800cde0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.745+0000 7f3d8c000640 1 -- 172.21.15.134:0/429716671 shutdown_connections 2024-07-26T11:18:14.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.745+0000 7f3d8c000640 1 --2- 172.21.15.134:0/429716671 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d6c0b3dc0 0x7f3d6c0b62d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.745+0000 7f3d8c000640 1 --2- 172.21.15.134:0/429716671 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d8410ba70 0x7f3d8406e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.745+0000 7f3d8c000640 1 --2- 172.21.15.134:0/429716671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d8410b120 0x7f3d8406db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.745+0000 7f3d8c000640 1 --2- 172.21.15.134:0/429716671 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d841057e0 0x7f3d840748a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:14.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.745+0000 7f3d8c000640 1 -- 172.21.15.134:0/429716671 >> 172.21.15.134:0/429716671 conn(0x7f3d841011f0 msgr2=0x7f3d841046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:14.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.746+0000 7f3d8c000640 1 -- 172.21.15.134:0/429716671 shutdown_connections 2024-07-26T11:18:14.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:14.746+0000 7f3d8c000640 1 -- 172.21.15.134:0/429716671 wait complete. 2024-07-26T11:18:15.861 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:15 smithi134 ceph-mon[31966]: pgmap v548: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 234 MiB/s wr, 1.31k op/s 2024-07-26T11:18:15.861 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:15 smithi134 ceph-mon[31966]: scrub queued for path: / 2024-07-26T11:18:15.861 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:15 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:18:15.861 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:15 smithi134 ceph-mon[31966]: scrub summary: active paths [/] 2024-07-26T11:18:15.861 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:15 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:18:15.867 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:18:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:15 smithi189 ceph-mon[36790]: pgmap v548: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 234 MiB/s wr, 1.31k op/s 2024-07-26T11:18:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:15 smithi189 ceph-mon[36790]: scrub queued for path: / 2024-07-26T11:18:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:15 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:18:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:15 smithi189 ceph-mon[36790]: scrub summary: active paths [/] 2024-07-26T11:18:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:15 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:18:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:15 smithi193 ceph-mon[37473]: pgmap v548: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 234 MiB/s wr, 1.31k op/s 2024-07-26T11:18:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:15 smithi193 ceph-mon[37473]: scrub queued for path: / 2024-07-26T11:18:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:15 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:18:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:15 smithi193 ceph-mon[37473]: scrub summary: active paths [/] 2024-07-26T11:18:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:15 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:18:16.771 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:18:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:16 smithi134 ceph-mon[31966]: scrub summary: active paths [/] 2024-07-26T11:18:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:16 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:18:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:16 smithi189 ceph-mon[36790]: scrub summary: active paths [/] 2024-07-26T11:18:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:16 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:18:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:16 smithi193 ceph-mon[37473]: scrub summary: active paths [/] 2024-07-26T11:18:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:16 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:18:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:17 smithi134 ceph-mon[31966]: pgmap v549: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 194 MiB/s wr, 1.25k op/s 2024-07-26T11:18:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:17 smithi189 ceph-mon[36790]: pgmap v549: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 194 MiB/s wr, 1.25k op/s 2024-07-26T11:18:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:17 smithi193 ceph-mon[37473]: pgmap v549: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 194 MiB/s wr, 1.25k op/s 2024-07-26T11:18:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:19 smithi134 ceph-mon[31966]: pgmap v550: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 170 MiB/s wr, 1.31k op/s 2024-07-26T11:18:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:19 smithi189 ceph-mon[36790]: pgmap v550: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 170 MiB/s wr, 1.31k op/s 2024-07-26T11:18:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:19 smithi193 ceph-mon[37473]: pgmap v550: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 170 MiB/s wr, 1.31k op/s 2024-07-26T11:18:20.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:20 smithi134 ceph-mon[31966]: scrub summary: idle 2024-07-26T11:18:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:20 smithi189 ceph-mon[36790]: scrub summary: idle 2024-07-26T11:18:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:20 smithi193 ceph-mon[37473]: scrub summary: idle 2024-07-26T11:18:21.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.349+0000 7f0647e00640 1 -- 172.21.15.134:0/340786497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f064810af30 msgr2=0x7f064810b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:21.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.349+0000 7f0647e00640 1 --2- 172.21.15.134:0/340786497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f064810af30 0x7f064810b340 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0634009900 tx=0x7f063402a410 comp rx=0 tx=0).stop 2024-07-26T11:18:21.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.350+0000 7f0647e00640 1 -- 172.21.15.134:0/340786497 shutdown_connections 2024-07-26T11:18:21.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.350+0000 7f0647e00640 1 --2- 172.21.15.134:0/340786497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f064810af30 0x7f064810b340 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.350+0000 7f0647e00640 1 --2- 172.21.15.134:0/340786497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f064810b880 0x7f06481120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.350+0000 7f0647e00640 1 --2- 172.21.15.134:0/340786497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f06481055f0 0x7f0648105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.350+0000 7f0647e00640 1 -- 172.21.15.134:0/340786497 >> 172.21.15.134:0/340786497 conn(0x7f0648101060 msgr2=0x7f06481034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:21.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.350+0000 7f0647e00640 1 -- 172.21.15.134:0/340786497 shutdown_connections 2024-07-26T11:18:21.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.350+0000 7f0647e00640 1 -- 172.21.15.134:0/340786497 wait complete. 2024-07-26T11:18:21.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.352+0000 7f0647e00640 1 Processor -- start 2024-07-26T11:18:21.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.353+0000 7f0647e00640 1 -- start start 2024-07-26T11:18:21.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.354+0000 7f0647e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06481055f0 0x7f064819c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.355+0000 7f0647e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f064810af30 0x7f064819c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.355+0000 7f0646a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06481055f0 0x7f064819c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.355+0000 7f0646a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06481055f0 0x7f064819c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43428/0 (socket says 172.21.15.134:43428) 2024-07-26T11:18:21.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.355+0000 7f0646a00640 1 -- 172.21.15.134:0/187636641 learned_addr learned my addr 172.21.15.134:0/187636641 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:21.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.356+0000 7f0646000640 1 --2- 172.21.15.134:0/187636641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f064810af30 0x7f064819c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.356+0000 7f0647e00640 1 --2- 172.21.15.134:0/187636641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f064810b880 0x7f06481a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.356+0000 7f0647e00640 1 -- 172.21.15.134:0/187636641 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0648115630 con 0x7f064810af30 2024-07-26T11:18:21.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.356+0000 7f0647e00640 1 -- 172.21.15.134:0/187636641 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f06481154b0 con 0x7f06481055f0 2024-07-26T11:18:21.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.356+0000 7f0647e00640 1 -- 172.21.15.134:0/187636641 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f06481157b0 con 0x7f064810b880 2024-07-26T11:18:21.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.357+0000 7f0647400640 1 --2- 172.21.15.134:0/187636641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f064810b880 0x7f06481a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.358+0000 7f0646000640 1 -- 172.21.15.134:0/187636641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f064810b880 msgr2=0x7f06481a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:21.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.358+0000 7f0646000640 1 --2- 172.21.15.134:0/187636641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f064810b880 0x7f06481a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.358+0000 7f0646000640 1 -- 172.21.15.134:0/187636641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06481055f0 msgr2=0x7f064819c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:21.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.358+0000 7f0646000640 1 --2- 172.21.15.134:0/187636641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06481055f0 0x7f064819c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.358+0000 7f0646000640 1 -- 172.21.15.134:0/187636641 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06481a1fd0 con 0x7f064810af30 2024-07-26T11:18:21.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.358+0000 7f0647400640 1 --2- 172.21.15.134:0/187636641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f064810b880 0x7f06481a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:18:21.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.358+0000 7f0646000640 1 --2- 172.21.15.134:0/187636641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f064810af30 0x7f064819c600 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0634009230 tx=0x7f0634003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:21.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.362+0000 7f063f400640 1 -- 172.21.15.134:0/187636641 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063403f070 con 0x7f064810af30 2024-07-26T11:18:21.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.362+0000 7f063f400640 1 -- 172.21.15.134:0/187636641 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0634031d60 con 0x7f064810af30 2024-07-26T11:18:21.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.362+0000 7f063f400640 1 -- 172.21.15.134:0/187636641 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0634035030 con 0x7f064810af30 2024-07-26T11:18:21.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.365+0000 7f0647e00640 1 -- 172.21.15.134:0/187636641 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06481a2260 con 0x7f064810af30 2024-07-26T11:18:21.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.365+0000 7f0647e00640 1 -- 172.21.15.134:0/187636641 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06481a8a20 con 0x7f064810af30 2024-07-26T11:18:21.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.371+0000 7f0654a32740 1 Processor -- start 2024-07-26T11:18:21.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.371+0000 7f0654a32740 1 -- start start 2024-07-26T11:18:21.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.373+0000 7f0654a32740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4d004070 0x559f4d3a9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.375+0000 7f0654a32740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfdb1b0 0x559f4d3a9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.375+0000 7f0654a32740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfe3390 0x559f4d3a9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.375+0000 7f0654a32740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559f4d31fbc0 con 0x559f4d004070 2024-07-26T11:18:21.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.375+0000 7f0654a32740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559f4d30f0c0 con 0x559f4cfdb1b0 2024-07-26T11:18:21.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.375+0000 7f0654a32740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559f4d313100 con 0x559f4cfe3390 2024-07-26T11:18:21.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.375+0000 7f0646a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfdb1b0 0x559f4d3a9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.375+0000 7f0646a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfdb1b0 0x559f4d3a9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43436/0 (socket says 172.21.15.134:43436) 2024-07-26T11:18:21.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.375+0000 7f0646a00640 1 -- 172.21.15.134:0/639282283 learned_addr learned my addr 172.21.15.134:0/639282283 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:21.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.375+0000 7f0646a00640 1 -- 172.21.15.134:0/639282283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfe3390 msgr2=0x559f4d3a9a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:18:21.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.375+0000 7f0646a00640 1 --2- 172.21.15.134:0/639282283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfe3390 0x559f4d3a9a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.375+0000 7f0646a00640 1 -- 172.21.15.134:0/639282283 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4d004070 msgr2=0x559f4d3a9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:21.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.375+0000 7f0646a00640 1 --2- 172.21.15.134:0/639282283 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4d004070 0x559f4d3a9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.376+0000 7f0646a00640 1 -- 172.21.15.134:0/639282283 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f4d06f650 con 0x559f4cfdb1b0 2024-07-26T11:18:21.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.376+0000 7f0646a00640 1 --2- 172.21.15.134:0/639282283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfdb1b0 0x559f4d3a9540 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0638009b20 tx=0x7f063800e1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b0df79bf7c0d1af server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:21.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.376+0000 7f0633e00640 1 -- 172.21.15.134:0/639282283 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0638017070 con 0x559f4cfdb1b0 2024-07-26T11:18:21.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.376+0000 7f0633e00640 1 -- 172.21.15.134:0/639282283 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f063800ec10 con 0x559f4cfdb1b0 2024-07-26T11:18:21.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.378+0000 7f0654a32740 1 -- 172.21.15.134:0/639282283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfdb1b0 msgr2=0x559f4d3a9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:21.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.378+0000 7f0654a32740 1 --2- 172.21.15.134:0/639282283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfdb1b0 0x559f4d3a9540 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0638009b20 tx=0x7f063800e1d0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.379+0000 7f063f400640 1 -- 172.21.15.134:0/187636641 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f0634042070 con 0x7f064810af30 2024-07-26T11:18:21.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.380+0000 7f0654a32740 1 -- 172.21.15.134:0/639282283 shutdown_connections 2024-07-26T11:18:21.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.380+0000 7f0654a32740 1 --2- 172.21.15.134:0/639282283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfe3390 0x559f4d3a9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.380+0000 7f0654a32740 1 --2- 172.21.15.134:0/639282283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfdb1b0 0x559f4d3a9540 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.380+0000 7f0654a32740 1 --2- 172.21.15.134:0/639282283 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4d004070 0x559f4d3a9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.380+0000 7f0654a32740 1 -- 172.21.15.134:0/639282283 >> 172.21.15.134:0/639282283 conn(0x559f4cff8460 msgr2=0x559f4d3a6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:21.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.384+0000 7f063f400640 1 --2- 172.21.15.134:0/187636641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f062c0b3dd0 0x7f062c0b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.385+0000 7f0646a00640 1 --2- 172.21.15.134:0/187636641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f062c0b3dd0 0x7f062c0b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.389 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.387+0000 7f0646a00640 1 --2- 172.21.15.134:0/187636641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f062c0b3dd0 0x7f062c0b62e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f063800efe0 tx=0x7f06380047d0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:21.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.388+0000 7f063f400640 1 -- 172.21.15.134:0/187636641 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0634104df0 con 0x7f064810af30 2024-07-26T11:18:21.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.393+0000 7f0654a32740 1 -- 172.21.15.134:0/639282283 shutdown_connections 2024-07-26T11:18:21.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.393+0000 7f0654a32740 1 -- 172.21.15.134:0/639282283 wait complete. 2024-07-26T11:18:21.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.394+0000 7f0654a32740 1 -- start start 2024-07-26T11:18:21.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.395+0000 7f0654a32740 1 Processor -- start 2024-07-26T11:18:21.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.397+0000 7f0654a32740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfdb1b0 0x559f4d3c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.398+0000 7f0654a32740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfe3390 0x559f4d3c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.398+0000 7f0654a32740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4d004070 0x559f4d3c5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.398+0000 7f0654a32740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559f4d172700 con 0x559f4d004070 2024-07-26T11:18:21.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.398+0000 7f0646a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfdb1b0 0x559f4d3c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.398+0000 7f0654a32740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559f4d2ab690 con 0x559f4cfe3390 2024-07-26T11:18:21.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.398+0000 7f0654a32740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559f4d3c8780 con 0x559f4cfdb1b0 2024-07-26T11:18:21.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.398+0000 7f0654a32740 10 client.0 _ll_register_callbacks cb 0x559f4d2bb140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:21.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.399+0000 7f0646a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfdb1b0 0x559f4d3c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:41956/0 (socket says 172.21.15.134:41956) 2024-07-26T11:18:21.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.399+0000 7f0646a00640 1 -- 172.21.15.134:0/2427846576 learned_addr learned my addr 172.21.15.134:0/2427846576 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:21.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.398+0000 7f0646000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfe3390 0x559f4d3c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.400+0000 7f0646a00640 1 -- 172.21.15.134:0/2427846576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfe3390 msgr2=0x559f4d3c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:21.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.400+0000 7f0646a00640 1 --2- 172.21.15.134:0/2427846576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfe3390 0x559f4d3c5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.400+0000 7f0646a00640 1 -- 172.21.15.134:0/2427846576 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4d004070 msgr2=0x559f4d3c5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:21.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.400+0000 7f0646a00640 1 --2- 172.21.15.134:0/2427846576 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4d004070 0x559f4d3c5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.400+0000 7f0646a00640 1 -- 172.21.15.134:0/2427846576 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f4d2c2830 con 0x559f4cfdb1b0 2024-07-26T11:18:21.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.402+0000 7f0630c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:18:21.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.402+0000 7f0646a00640 1 --2- 172.21.15.134:0/2427846576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfdb1b0 0x559f4d3c5520 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0638009b20 tx=0x7f0638013c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6977a371b53a07bc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:21.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.402+0000 7f0630c00640 1 -- 172.21.15.134:0/2427846576 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0638017040 con 0x559f4cfdb1b0 2024-07-26T11:18:21.405 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.402+0000 7f0630c00640 1 -- 172.21.15.134:0/2427846576 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0638012d60 con 0x559f4cfdb1b0 2024-07-26T11:18:21.405 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.403+0000 7f0654a32740 1 -- 172.21.15.134:0/2427846576 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f063800ec10 con 0x559f4cfdb1b0 2024-07-26T11:18:21.405 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.403+0000 7f0630c00640 1 -- 172.21.15.134:0/2427846576 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f063801c450 con 0x559f4cfdb1b0 2024-07-26T11:18:21.405 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.403+0000 7f0654a32740 10 client.25936 fetch_fsmap learned FSMap version 31 2024-07-26T11:18:21.405 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.403+0000 7f0654a32740 1 -- 172.21.15.134:0/2427846576 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559f4d3d4340 con 0x559f4cfdb1b0 2024-07-26T11:18:21.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.405+0000 7f0630c00640 1 -- 172.21.15.134:0/2427846576 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f063800ec10 con 0x559f4cfdb1b0 2024-07-26T11:18:21.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.405+0000 7f0630c00640 1 -- 172.21.15.134:0/2427846576 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f06380375c0 con 0x559f4cfdb1b0 2024-07-26T11:18:21.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.406+0000 7f0654a32740 10 client.25936 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:18:21.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.406+0000 7f0654a32740 10 client.25936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:18:21.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.406+0000 7f0654a32740 20 client.25936 populate_metadata read hostname 'smithi134' 2024-07-26T11:18:21.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.406+0000 7f0654a32740 1 --2- 172.21.15.134:0/2427846576 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559f4cfc29f0 0x559f4d3d76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.408+0000 7f0654a32740 4 client.25936 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:18:21.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.410+0000 7f0654a32740 1 -- 172.21.15.134:0/2427846576 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f4d321b10 con 0x559f4cfc29f0 2024-07-26T11:18:21.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.410+0000 7f0646000640 1 --2- 172.21.15.134:0/2427846576 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559f4cfc29f0 0x559f4d3d76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.411+0000 7f0646000640 1 --2- 172.21.15.134:0/2427846576 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559f4cfc29f0 0x559f4d3d76a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6ce102a29d8e26e server_cookie=59580bf9e195c682 in_seq=0 out_seq=0 2024-07-26T11:18:21.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.411+0000 7f0630c00640 10 client.25936 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:21.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.413+0000 7f0630c00640 1 -- 172.21.15.134:0/2427846576 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559f4d321b10 con 0x559f4cfc29f0 2024-07-26T11:18:21.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.413+0000 7f0630c00640 10 client.25936 handle_command_reply: tid=0 2024-07-26T11:18:21.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.413+0000 7f0654a32740 1 client.25936 shutdown 2024-07-26T11:18:21.415 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.413+0000 7f0654a32740 1 -- 172.21.15.134:0/2427846576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfdb1b0 msgr2=0x559f4d3c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:21.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.413+0000 7f0654a32740 1 --2- 172.21.15.134:0/2427846576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfdb1b0 0x559f4d3c5520 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0638009b20 tx=0x7f0638013c60 comp rx=0 tx=0).stop 2024-07-26T11:18:21.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.414+0000 7f0654a32740 1 -- 172.21.15.134:0/2427846576 shutdown_connections 2024-07-26T11:18:21.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.414+0000 7f0654a32740 1 --2- 172.21.15.134:0/2427846576 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559f4cfc29f0 0x559f4d3d76a0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.414+0000 7f0630c00640 0 client.25936 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:21.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.414+0000 7f0654a32740 1 --2- 172.21.15.134:0/2427846576 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4d004070 0x559f4d3c5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.414+0000 7f0654a32740 1 --2- 172.21.15.134:0/2427846576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfe3390 0x559f4d3c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.414+0000 7f0654a32740 1 --2- 172.21.15.134:0/2427846576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfdb1b0 0x559f4d3c5520 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.414+0000 7f0654a32740 1 -- 172.21.15.134:0/2427846576 >> 172.21.15.134:0/2427846576 conn(0x559f4cff8460 msgr2=0x559f4d3aa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:21.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.415+0000 7f0654a32740 1 -- 172.21.15.134:0/2427846576 shutdown_connections 2024-07-26T11:18:21.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.415+0000 7f0654a32740 1 -- 172.21.15.134:0/2427846576 wait complete. 2024-07-26T11:18:21.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.415+0000 7f0654a32740 20 client.25936 trim_cache size 0 max 16384 2024-07-26T11:18:21.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.460+0000 7f0654a32740 1 Processor -- start 2024-07-26T11:18:21.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.460+0000 7f0654a32740 1 -- start start 2024-07-26T11:18:21.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.460+0000 7f0654a32740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cff8460 0x559f4d3c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.461+0000 7f0654a32740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfc29f0 0x559f4d3aa740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.461+0000 7f0654a32740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4cfdb1b0 0x559f4d3dcda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.461+0000 7f0654a32740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0648072310 con 0x559f4cfdb1b0 2024-07-26T11:18:21.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.461+0000 7f0654a32740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0648073e80 con 0x559f4cfc29f0 2024-07-26T11:18:21.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.461+0000 7f0654a32740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f06480729e0 con 0x559f4cff8460 2024-07-26T11:18:21.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.463+0000 7f0646a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cff8460 0x559f4d3c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.463+0000 7f0646000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfc29f0 0x559f4d3aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.463+0000 7f0646000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfc29f0 0x559f4d3aa740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43444/0 (socket says 172.21.15.134:43444) 2024-07-26T11:18:21.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.463+0000 7f0646000640 1 -- 172.21.15.134:0/634956235 learned_addr learned my addr 172.21.15.134:0/634956235 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:21.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.464+0000 7f0647400640 1 --2- 172.21.15.134:0/634956235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4cfdb1b0 0x559f4d3dcda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.466+0000 7f0646000640 1 -- 172.21.15.134:0/634956235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cff8460 msgr2=0x559f4d3c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:21.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.466+0000 7f0646000640 1 --2- 172.21.15.134:0/634956235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cff8460 0x559f4d3c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.467+0000 7f0646000640 1 -- 172.21.15.134:0/634956235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4cfdb1b0 msgr2=0x559f4d3dcda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:21.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.467+0000 7f0646000640 1 --2- 172.21.15.134:0/634956235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4cfdb1b0 0x559f4d3dcda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.467+0000 7f0646000640 1 -- 172.21.15.134:0/634956235 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f4d3c8620 con 0x559f4cfc29f0 2024-07-26T11:18:21.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.467+0000 7f0647400640 1 --2- 172.21.15.134:0/634956235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4cfdb1b0 0x559f4d3dcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:18:21.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.467+0000 7f0646000640 1 --2- 172.21.15.134:0/634956235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfc29f0 0x559f4d3aa740 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f063402a930 tx=0x7f0634005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77224dd9fdc2d176 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:21.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.467+0000 7f0633400640 1 -- 172.21.15.134:0/634956235 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06340cb620 con 0x559f4cfc29f0 2024-07-26T11:18:21.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.467+0000 7f0633400640 1 -- 172.21.15.134:0/634956235 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06340cfc20 con 0x559f4cfc29f0 2024-07-26T11:18:21.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.467+0000 7f0633400640 1 -- 172.21.15.134:0/634956235 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06341023e0 con 0x559f4cfc29f0 2024-07-26T11:18:21.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.469+0000 7f0654a32740 1 -- 172.21.15.134:0/634956235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfc29f0 msgr2=0x559f4d3aa740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:21.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.469+0000 7f0654a32740 1 --2- 172.21.15.134:0/634956235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfc29f0 0x559f4d3aa740 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f063402a930 tx=0x7f0634005fd0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.469+0000 7f0654a32740 1 -- 172.21.15.134:0/634956235 shutdown_connections 2024-07-26T11:18:21.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.469+0000 7f0654a32740 1 --2- 172.21.15.134:0/634956235 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4cfdb1b0 0x559f4d3dcda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.469+0000 7f0654a32740 1 --2- 172.21.15.134:0/634956235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cfc29f0 0x559f4d3aa740 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.469+0000 7f0654a32740 1 --2- 172.21.15.134:0/634956235 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cff8460 0x559f4d3c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.469+0000 7f0654a32740 1 -- 172.21.15.134:0/634956235 >> 172.21.15.134:0/634956235 conn(0x559f4d3aee90 msgr2=0x559f4d3aa030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:21.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.469+0000 7f0654a32740 1 -- 172.21.15.134:0/634956235 shutdown_connections 2024-07-26T11:18:21.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.469+0000 7f0654a32740 1 -- 172.21.15.134:0/634956235 wait complete. 2024-07-26T11:18:21.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.471+0000 7f0654a32740 1 -- start start 2024-07-26T11:18:21.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.472+0000 7f0654a32740 1 Processor -- start 2024-07-26T11:18:21.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.472+0000 7f0654a32740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfc29f0 0x559f4d3cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.472+0000 7f0654a32740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4cfdb1b0 0x559f4d3c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.474+0000 7f0654a32740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cff8460 0x559f4d3c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.482+0000 7f0646a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfc29f0 0x559f4d3cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.482+0000 7f0646a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfc29f0 0x559f4d3cdeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:41974/0 (socket says 172.21.15.134:41974) 2024-07-26T11:18:21.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.482+0000 7f0646a00640 1 -- 172.21.15.134:0/2036771682 learned_addr learned my addr 172.21.15.134:0/2036771682 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:21.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.482+0000 7f0647400640 1 --2- 172.21.15.134:0/2036771682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cff8460 0x559f4d3c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.482+0000 7f0654a32740 1 -- 172.21.15.134:0/2036771682 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0648073780 con 0x559f4cfdb1b0 2024-07-26T11:18:21.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.482+0000 7f0654a32740 1 -- 172.21.15.134:0/2036771682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f06480730b0 con 0x559f4cff8460 2024-07-26T11:18:21.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.482+0000 7f0654a32740 1 -- 172.21.15.134:0/2036771682 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f06480702f0 con 0x559f4cfc29f0 2024-07-26T11:18:21.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.483+0000 7f0654a32740 10 client.0 _ll_register_callbacks cb 0x7f063802cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:21.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.483+0000 7f0646000640 1 --2- 172.21.15.134:0/2036771682 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4cfdb1b0 0x559f4d3c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.483+0000 7f0646a00640 1 -- 172.21.15.134:0/2036771682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cff8460 msgr2=0x559f4d3c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:21.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.483+0000 7f0646a00640 1 --2- 172.21.15.134:0/2036771682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cff8460 0x559f4d3c98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.483+0000 7f0646a00640 1 -- 172.21.15.134:0/2036771682 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4cfdb1b0 msgr2=0x559f4d3c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:21.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.484+0000 7f0646a00640 1 --2- 172.21.15.134:0/2036771682 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4cfdb1b0 0x559f4d3c9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.484+0000 7f0646a00640 1 -- 172.21.15.134:0/2036771682 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f4d3c9de0 con 0x559f4cfc29f0 2024-07-26T11:18:21.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.485+0000 7f0654a32740 1 -- 172.21.15.134:0/2036771682 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06340cfc20 con 0x559f4cfc29f0 2024-07-26T11:18:21.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.485+0000 7f0646a00640 1 --2- 172.21.15.134:0/2036771682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfc29f0 0x559f4d3cdeb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0638037a60 tx=0x7f0638012ff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7812d53c58915c66 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:21.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.485+0000 7f0630c00640 10 client.25942 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:18:21.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.486+0000 7f0630c00640 1 -- 172.21.15.134:0/2036771682 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0638009230 con 0x559f4cfc29f0 2024-07-26T11:18:21.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.486+0000 7f0630c00640 1 -- 172.21.15.134:0/2036771682 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f063800e040 con 0x559f4cfc29f0 2024-07-26T11:18:21.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.486+0000 7f0630c00640 1 -- 172.21.15.134:0/2036771682 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0638037630 con 0x559f4cfc29f0 2024-07-26T11:18:21.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.487+0000 7f0654a32740 10 client.25942 fetch_fsmap learned FSMap version 31 2024-07-26T11:18:21.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.487+0000 7f0654a32740 1 -- 172.21.15.134:0/2036771682 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=31}) -- 0x559f4d43f880 con 0x559f4cfc29f0 2024-07-26T11:18:21.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.487+0000 7f0630c00640 1 -- 172.21.15.134:0/2036771682 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f0638048030 con 0x559f4cfc29f0 2024-07-26T11:18:21.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.488+0000 7f0654a32740 10 client.25942 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:18:21.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.488+0000 7f0654a32740 10 client.25942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:18:21.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.488+0000 7f0654a32740 20 client.25942 populate_metadata read hostname 'smithi134' 2024-07-26T11:18:21.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.488+0000 7f0654a32740 1 --2- 172.21.15.134:0/2036771682 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559f4cfe3390 0x559f4d441d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.488+0000 7f0654a32740 4 client.25942 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:18:21.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.488+0000 7f0654a32740 1 -- 172.21.15.134:0/2036771682 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559f4d31d5b0 con 0x559f4cfe3390 2024-07-26T11:18:21.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.490+0000 7f0646000640 1 --2- 172.21.15.134:0/2036771682 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559f4cfe3390 0x559f4d441d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.506+0000 7f0646000640 1 --2- 172.21.15.134:0/2036771682 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559f4cfe3390 0x559f4d441d60 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb085434538201b7 server_cookie=b855b61aa4f93f14 in_seq=0 out_seq=0 2024-07-26T11:18:21.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.506+0000 7f0630c00640 10 client.25942 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:21.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.507+0000 7f0630c00640 1 -- 172.21.15.134:0/2036771682 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x559f4d31d5b0 con 0x559f4cfe3390 2024-07-26T11:18:21.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.507+0000 7f0630c00640 10 client.25942 handle_command_reply: tid=0 2024-07-26T11:18:21.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.507+0000 7f0654a32740 1 client.25942 shutdown 2024-07-26T11:18:21.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.507+0000 7f0654a32740 1 -- 172.21.15.134:0/2036771682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfc29f0 msgr2=0x559f4d3cdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:21.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.507+0000 7f0654a32740 1 --2- 172.21.15.134:0/2036771682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfc29f0 0x559f4d3cdeb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0638037a60 tx=0x7f0638012ff0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.508+0000 7f0654a32740 1 -- 172.21.15.134:0/2036771682 shutdown_connections 2024-07-26T11:18:21.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.508+0000 7f0654a32740 1 --2- 172.21.15.134:0/2036771682 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559f4cfe3390 0x559f4d441d60 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.508+0000 7f0654a32740 1 --2- 172.21.15.134:0/2036771682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f4cff8460 0x559f4d3c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.508+0000 7f0654a32740 1 --2- 172.21.15.134:0/2036771682 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f4cfdb1b0 0x559f4d3c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.508+0000 7f0654a32740 1 --2- 172.21.15.134:0/2036771682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f4cfc29f0 0x559f4d3cdeb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.508+0000 7f0654a32740 1 -- 172.21.15.134:0/2036771682 >> 172.21.15.134:0/2036771682 conn(0x559f4d3aee90 msgr2=0x559f4d3c0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:21.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.508+0000 7f0630c00640 0 client.25942 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:21.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.511+0000 7f0654a32740 1 -- 172.21.15.134:0/2036771682 shutdown_connections 2024-07-26T11:18:21.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.511+0000 7f0654a32740 1 -- 172.21.15.134:0/2036771682 wait complete. 2024-07-26T11:18:21.514 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.512+0000 7f0654a32740 20 client.25942 trim_cache size 0 max 16384 2024-07-26T11:18:21.516 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:18:21.516 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "no active scrubs running", 2024-07-26T11:18:21.516 INFO:teuthology.orchestra.run.smithi134.stdout: "scrubs": {} 2024-07-26T11:18:21.516 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:18:21.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.516+0000 7f0647e00640 1 -- 172.21.15.134:0/187636641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f062c0b3dd0 msgr2=0x7f062c0b62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:21.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.516+0000 7f0647e00640 1 --2- 172.21.15.134:0/187636641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f062c0b3dd0 0x7f062c0b62e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f063800efe0 tx=0x7f06380047d0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.518+0000 7f0647e00640 1 -- 172.21.15.134:0/187636641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f064810af30 msgr2=0x7f064819c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:21.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.518+0000 7f0647e00640 1 --2- 172.21.15.134:0/187636641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f064810af30 0x7f064819c600 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0634009230 tx=0x7f0634003d50 comp rx=0 tx=0).stop 2024-07-26T11:18:21.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.520+0000 7f0647e00640 1 -- 172.21.15.134:0/187636641 shutdown_connections 2024-07-26T11:18:21.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.520+0000 7f0647e00640 1 --2- 172.21.15.134:0/187636641 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f062c0b3dd0 0x7f062c0b62e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.520+0000 7f0647e00640 1 --2- 172.21.15.134:0/187636641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f064810b880 0x7f06481a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.520+0000 7f0647e00640 1 --2- 172.21.15.134:0/187636641 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f064810af30 0x7f064819c600 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.520+0000 7f0647e00640 1 --2- 172.21.15.134:0/187636641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06481055f0 0x7f064819c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.520+0000 7f0647e00640 1 -- 172.21.15.134:0/187636641 >> 172.21.15.134:0/187636641 conn(0x7f0648101060 msgr2=0x7f0648103480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:21.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.520+0000 7f0647e00640 1 -- 172.21.15.134:0/187636641 shutdown_connections 2024-07-26T11:18:21.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:21.521+0000 7f0647e00640 1 -- 172.21.15.134:0/187636641 wait complete. 2024-07-26T11:18:21.535 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.533+0000 7f959a200640 1 -- 172.21.15.134:0/2944453179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9594108e40 msgr2=0x7f9594109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:21.535 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.533+0000 7f959a200640 1 --2- 172.21.15.134:0/2944453179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9594108e40 0x7f9594109250 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9584009900 tx=0x7f958402a420 comp rx=0 tx=0).stop 2024-07-26T11:18:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.534+0000 7f959a200640 1 -- 172.21.15.134:0/2944453179 shutdown_connections 2024-07-26T11:18:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.534+0000 7f959a200640 1 --2- 172.21.15.134:0/2944453179 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9594109790 0x7f959410ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.534+0000 7f959a200640 1 --2- 172.21.15.134:0/2944453179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9594108e40 0x7f9594109250 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.534+0000 7f959a200640 1 --2- 172.21.15.134:0/2944453179 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9594103510 0x7f9594103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.534+0000 7f959a200640 1 -- 172.21.15.134:0/2944453179 >> 172.21.15.134:0/2944453179 conn(0x7f95940fef50 msgr2=0x7f95941013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.535+0000 7f959a200640 1 -- 172.21.15.134:0/2944453179 shutdown_connections 2024-07-26T11:18:21.537 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.535+0000 7f959a200640 1 -- 172.21.15.134:0/2944453179 wait complete. 2024-07-26T11:18:21.539 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.537+0000 7f959a200640 1 Processor -- start 2024-07-26T11:18:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.538+0000 7f959a200640 1 -- start start 2024-07-26T11:18:21.541 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.539+0000 7f959a200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9594103510 0x7f9594076b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.540+0000 7f959a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9594108e40 0x7f9594075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.540+0000 7f959a200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9594109790 0x7f9594075680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.540+0000 7f959a200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9594113450 con 0x7f9594109790 2024-07-26T11:18:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.540+0000 7f959a200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f95941132d0 con 0x7f9594108e40 2024-07-26T11:18:21.542 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.540+0000 7f959a200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f95941135d0 con 0x7f9594103510 2024-07-26T11:18:21.543 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.541+0000 7f9592a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9594108e40 0x7f9594075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.543 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.541+0000 7f9592a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9594108e40 0x7f9594075140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43460/0 (socket says 172.21.15.134:43460) 2024-07-26T11:18:21.543 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.541+0000 7f9592a00640 1 -- 172.21.15.134:0/3162927544 learned_addr learned my addr 172.21.15.134:0/3162927544 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:21.544 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.542+0000 7f9592a00640 1 -- 172.21.15.134:0/3162927544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9594103510 msgr2=0x7f9594076b60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T11:18:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.542+0000 7f9592a00640 1 --2- 172.21.15.134:0/3162927544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9594103510 0x7f9594076b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.542+0000 7f9592a00640 1 -- 172.21.15.134:0/3162927544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9594109790 msgr2=0x7f9594075680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-07-26T11:18:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.542+0000 7f9592a00640 1 --2- 172.21.15.134:0/3162927544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9594109790 0x7f9594075680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.542+0000 7f9592a00640 1 -- 172.21.15.134:0/3162927544 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9594075ea0 con 0x7f9594108e40 2024-07-26T11:18:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.543+0000 7f9592a00640 1 --2- 172.21.15.134:0/3162927544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9594108e40 0x7f9594075140 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9584031440 tx=0x7f9584003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.543+0000 7f958be00640 1 -- 172.21.15.134:0/3162927544 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958403f070 con 0x7f9594108e40 2024-07-26T11:18:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.544+0000 7f958be00640 1 -- 172.21.15.134:0/3162927544 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9584031b50 con 0x7f9594108e40 2024-07-26T11:18:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.544+0000 7f958be00640 1 -- 172.21.15.134:0/3162927544 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958403f070 con 0x7f9594108e40 2024-07-26T11:18:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.544+0000 7f959a200640 1 -- 172.21.15.134:0/3162927544 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95941ac7a0 con 0x7f9594108e40 2024-07-26T11:18:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.544+0000 7f959a200640 1 -- 172.21.15.134:0/3162927544 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95941acb80 con 0x7f9594108e40 2024-07-26T11:18:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.550+0000 7f95a11db740 1 Processor -- start 2024-07-26T11:18:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.553+0000 7f958be00640 1 -- 172.21.15.134:0/3162927544 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f9584042070 con 0x7f9594108e40 2024-07-26T11:18:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.556+0000 7f958be00640 1 --2- 172.21.15.134:0/3162927544 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95780b4070 0x7f95780b6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.557+0000 7f95a11db740 1 -- start start 2024-07-26T11:18:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.557+0000 7f95a11db740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14d0071b0 0x55e14d3aaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.557+0000 7f95a11db740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 0x55e14d3aafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.557+0000 7f95a11db740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cfe64d0 0x55e14d3ab4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.557+0000 7f95a11db740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e14d31f270 con 0x55e14d0071b0 2024-07-26T11:18:21.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.557+0000 7f95a11db740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e14d30e770 con 0x55e14cfde2f0 2024-07-26T11:18:21.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.557+0000 7f95a11db740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e14d3127b0 con 0x55e14cfe64d0 2024-07-26T11:18:21.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.559+0000 7f9593400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14d0071b0 0x55e14d3aaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.559+0000 7f9593e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cfe64d0 0x55e14d3ab4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.559+0000 7f9593e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cfe64d0 0x55e14d3ab4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:42008/0 (socket says 172.21.15.134:42008) 2024-07-26T11:18:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.559+0000 7f9593e00640 1 -- 172.21.15.134:0/2644486497 learned_addr learned my addr 172.21.15.134:0/2644486497 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.559+0000 7f9593e00640 1 --2- 172.21.15.134:0/3162927544 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95780b4070 0x7f95780b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.559+0000 7f9592a00640 1 --2- 172.21.15.134:0/2644486497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 0x55e14d3aafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.559+0000 7f9593400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14d0071b0 0x55e14d3aaa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50602/0 (socket says 172.21.15.134:50602) 2024-07-26T11:18:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.560+0000 7f958be00640 1 -- 172.21.15.134:0/3162927544 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f95841043c0 con 0x7f9594108e40 2024-07-26T11:18:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.561+0000 7f9593e00640 1 -- 172.21.15.134:0/2644486497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 msgr2=0x55e14d3aafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.561+0000 7f9593e00640 1 --2- 172.21.15.134:0/2644486497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 0x55e14d3aafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.561+0000 7f9593e00640 1 -- 172.21.15.134:0/2644486497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14d0071b0 msgr2=0x55e14d3aaa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:21.564 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.561+0000 7f9593e00640 1 --2- 172.21.15.134:0/2644486497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14d0071b0 0x55e14d3aaa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.564 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.561+0000 7f9593e00640 1 -- 172.21.15.134:0/2644486497 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e14d072790 con 0x55e14cfe64d0 2024-07-26T11:18:21.564 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.562+0000 7f9593e00640 1 --2- 172.21.15.134:0/2644486497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cfe64d0 0x55e14d3ab4e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f958000dda0 tx=0x7f95800117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd7623c3f877164f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:21.564 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.562+0000 7f9593400640 1 --2- 172.21.15.134:0/2644486497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14d0071b0 0x55e14d3aaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:18:21.566 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.563+0000 7f957fe00640 1 -- 172.21.15.134:0/2644486497 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580022070 con 0x55e14cfe64d0 2024-07-26T11:18:21.566 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.563+0000 7f957fe00640 1 -- 172.21.15.134:0/2644486497 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f958001cac0 con 0x55e14cfe64d0 2024-07-26T11:18:21.566 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.563+0000 7f957fe00640 1 -- 172.21.15.134:0/2644486497 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958000f7f0 con 0x55e14cfe64d0 2024-07-26T11:18:21.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.565+0000 7f95a11db740 1 -- 172.21.15.134:0/2644486497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cfe64d0 msgr2=0x55e14d3ab4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:21.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.565+0000 7f95a11db740 1 --2- 172.21.15.134:0/2644486497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cfe64d0 0x55e14d3ab4e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f958000dda0 tx=0x7f95800117b0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.565+0000 7f9593e00640 1 --2- 172.21.15.134:0/3162927544 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95780b4070 0x7f95780b6580 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f958001f000 tx=0x7f958001c000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:21.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.565+0000 7f95a11db740 1 -- 172.21.15.134:0/2644486497 shutdown_connections 2024-07-26T11:18:21.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.565+0000 7f95a11db740 1 --2- 172.21.15.134:0/2644486497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cfe64d0 0x55e14d3ab4e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.565+0000 7f95a11db740 1 --2- 172.21.15.134:0/2644486497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 0x55e14d3aafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.565+0000 7f95a11db740 1 --2- 172.21.15.134:0/2644486497 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14d0071b0 0x55e14d3aaa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.565+0000 7f95a11db740 1 -- 172.21.15.134:0/2644486497 >> 172.21.15.134:0/2644486497 conn(0x55e14cffb300 msgr2=0x55e14d3a8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.569+0000 7f95a11db740 1 -- 172.21.15.134:0/2644486497 shutdown_connections 2024-07-26T11:18:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.572+0000 7f95a11db740 1 -- 172.21.15.134:0/2644486497 wait complete. 2024-07-26T11:18:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.573+0000 7f95a11db740 1 -- start start 2024-07-26T11:18:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.573+0000 7f95a11db740 1 Processor -- start 2024-07-26T11:18:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.574+0000 7f95a11db740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 0x55e14d3b66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.575+0000 7f95a11db740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cfe64d0 0x55e14d3b6c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.575+0000 7f9592a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cfe64d0 0x55e14d3b6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.575+0000 7f9592a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cfe64d0 0x55e14d3b6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:42022/0 (socket says 172.21.15.134:42022) 2024-07-26T11:18:21.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.575+0000 7f95a11db740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14d0071b0 0x55e14d3d34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.577+0000 7f9593400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 0x55e14d3b66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.577+0000 7f9593400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 0x55e14d3b66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43474/0 (socket says 172.21.15.134:43474) 2024-07-26T11:18:21.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.577+0000 7f95a11db740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e14d248ab0 con 0x55e14d0071b0 2024-07-26T11:18:21.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.577+0000 7f95a11db740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e14d2abd50 con 0x55e14cfde2f0 2024-07-26T11:18:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.577+0000 7f95a11db740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e14d3ca1f0 con 0x55e14cfe64d0 2024-07-26T11:18:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.577+0000 7f95a11db740 10 client.0 _ll_register_callbacks cb 0x55e14d01ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.577+0000 7f9592a00640 1 -- 172.21.15.134:0/2731642017 learned_addr learned my addr 172.21.15.134:0/2731642017 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.578+0000 7f9592a00640 1 -- 172.21.15.134:0/2731642017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 msgr2=0x55e14d3b66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.578+0000 7f9592a00640 1 --2- 172.21.15.134:0/2731642017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 0x55e14d3b66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.578+0000 7f9592a00640 1 -- 172.21.15.134:0/2731642017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14d0071b0 msgr2=0x55e14d3d34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.578+0000 7f9592a00640 1 --2- 172.21.15.134:0/2731642017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14d0071b0 0x55e14d3d34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.578+0000 7f9592a00640 1 -- 172.21.15.134:0/2731642017 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e14d0eb620 con 0x55e14cfe64d0 2024-07-26T11:18:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.579+0000 7f957cc00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:18:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.579+0000 7f9592a00640 1 --2- 172.21.15.134:0/2731642017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cfe64d0 0x55e14d3b6c80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f95840c7e60 tx=0x7f958402a420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4813b3ae6aa58c79 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.580+0000 7f957cc00640 1 -- 172.21.15.134:0/2731642017 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9584049030 con 0x55e14cfe64d0 2024-07-26T11:18:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.580+0000 7f957cc00640 1 -- 172.21.15.134:0/2731642017 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f95840c6e80 con 0x55e14cfe64d0 2024-07-26T11:18:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.580+0000 7f95a11db740 1 -- 172.21.15.134:0/2731642017 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f958001cac0 con 0x55e14cfe64d0 2024-07-26T11:18:21.585 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.581+0000 7f957cc00640 1 -- 172.21.15.134:0/2731642017 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f95840ca700 con 0x55e14cfe64d0 2024-07-26T11:18:21.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.582+0000 7f95a11db740 10 client.25951 fetch_fsmap learned FSMap version 31 2024-07-26T11:18:21.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.582+0000 7f95a11db740 1 -- 172.21.15.134:0/2731642017 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e14d3d6220 con 0x55e14cfe64d0 2024-07-26T11:18:21.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.583+0000 7f957cc00640 1 -- 172.21.15.134:0/2731642017 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f958001cac0 con 0x55e14cfe64d0 2024-07-26T11:18:21.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.583+0000 7f957cc00640 1 -- 172.21.15.134:0/2731642017 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f958404c060 con 0x55e14cfe64d0 2024-07-26T11:18:21.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.585+0000 7f95a11db740 10 client.25951 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:18:21.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.585+0000 7f95a11db740 10 client.25951 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:18:21.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.586+0000 7f95a11db740 20 client.25951 populate_metadata read hostname 'smithi134' 2024-07-26T11:18:21.588 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.586+0000 7f95a11db740 1 --2- 172.21.15.134:0/2731642017 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e14cfc5d80 0x55e14d3d9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.589 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.587+0000 7f95a11db740 4 client.25951 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:18:21.589 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.587+0000 7f95a11db740 1 -- 172.21.15.134:0/2731642017 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e14d3211c0 con 0x55e14cfc5d80 2024-07-26T11:18:21.589 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.587+0000 7f9593400640 1 --2- 172.21.15.134:0/2731642017 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e14cfc5d80 0x55e14d3d9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.592+0000 7f9593400640 1 --2- 172.21.15.134:0/2731642017 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e14cfc5d80 0x55e14d3d9720 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c64c896e7fd8b server_cookie=9adac68fb6db8003 in_seq=0 out_seq=0 2024-07-26T11:18:21.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.593+0000 7f957cc00640 10 client.25951 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:21.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.594+0000 7f957cc00640 1 -- 172.21.15.134:0/2731642017 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e14d3211c0 con 0x55e14cfc5d80 2024-07-26T11:18:21.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.595+0000 7f957cc00640 10 client.25951 handle_command_reply: tid=0 2024-07-26T11:18:21.599 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.595+0000 7f95a11db740 1 client.25951 shutdown 2024-07-26T11:18:21.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.598+0000 7f95a11db740 1 -- 172.21.15.134:0/2731642017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cfe64d0 msgr2=0x55e14d3b6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:21.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.598+0000 7f95a11db740 1 --2- 172.21.15.134:0/2731642017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cfe64d0 0x55e14d3b6c80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f95840c7e60 tx=0x7f958402a420 comp rx=0 tx=0).stop 2024-07-26T11:18:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.599+0000 7f95a11db740 1 -- 172.21.15.134:0/2731642017 shutdown_connections 2024-07-26T11:18:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.599+0000 7f95a11db740 1 --2- 172.21.15.134:0/2731642017 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e14cfc5d80 0x55e14d3d9720 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.599+0000 7f95a11db740 1 --2- 172.21.15.134:0/2731642017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cfe64d0 0x55e14d3b6c80 secure :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f95840c7e60 tx=0x7f958402a420 comp rx=0 tx=0).stop 2024-07-26T11:18:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.599+0000 7f95a11db740 1 --2- 172.21.15.134:0/2731642017 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14d0071b0 0x55e14d3d34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.599+0000 7f95a11db740 1 --2- 172.21.15.134:0/2731642017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 0x55e14d3b66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.599+0000 7f95a11db740 1 -- 172.21.15.134:0/2731642017 >> 172.21.15.134:0/2731642017 conn(0x55e14cffb300 msgr2=0x55e14d3aba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:21.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.600+0000 7f957cc00640 0 client.25951 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:21.603 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.601+0000 7f95a11db740 1 -- 172.21.15.134:0/2731642017 shutdown_connections 2024-07-26T11:18:21.605 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.602+0000 7f95a11db740 1 -- 172.21.15.134:0/2731642017 wait complete. 2024-07-26T11:18:21.605 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.603+0000 7f95a11db740 20 client.25951 trim_cache size 0 max 16384 2024-07-26T11:18:21.675 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:21 smithi134 ceph-mon[31966]: scrub summary: idle 2024-07-26T11:18:21.675 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:21 smithi134 ceph-mon[31966]: pgmap v551: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 193 MiB/s wr, 1.71k op/s 2024-07-26T11:18:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.686+0000 7f95a11db740 1 Processor -- start 2024-07-26T11:18:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.688+0000 7f95a11db740 1 -- start start 2024-07-26T11:18:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.690+0000 7f95a11db740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14cfde2f0 0x55e14d3b0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.690+0000 7f95a11db740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d0071b0 0x55e14d3deee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.690+0000 7f95a11db740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cf9b1c0 0x55e14d3df420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.690+0000 7f95a11db740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f95941ae780 con 0x55e14cfde2f0 2024-07-26T11:18:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.690+0000 7f95a11db740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f95941b02f0 con 0x55e14d0071b0 2024-07-26T11:18:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.690+0000 7f95a11db740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f95941aee50 con 0x55e14cf9b1c0 2024-07-26T11:18:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.691+0000 7f9593e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cf9b1c0 0x55e14d3df420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.691+0000 7f9593e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cf9b1c0 0x55e14d3df420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:42038/0 (socket says 172.21.15.134:42038) 2024-07-26T11:18:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.691+0000 7f9593e00640 1 -- 172.21.15.134:0/2908453378 learned_addr learned my addr 172.21.15.134:0/2908453378 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.691+0000 7f9593e00640 1 -- 172.21.15.134:0/2908453378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d0071b0 msgr2=0x55e14d3deee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:18:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.691+0000 7f9593e00640 1 --2- 172.21.15.134:0/2908453378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d0071b0 0x55e14d3deee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.691+0000 7f9593e00640 1 -- 172.21.15.134:0/2908453378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14cfde2f0 msgr2=0x55e14d3b0230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:18:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.691+0000 7f9593e00640 1 --2- 172.21.15.134:0/2908453378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14cfde2f0 0x55e14d3b0230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.691+0000 7f9593e00640 1 -- 172.21.15.134:0/2908453378 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e14d3b7770 con 0x55e14cf9b1c0 2024-07-26T11:18:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.696+0000 7f9593e00640 1 --2- 172.21.15.134:0/2908453378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cf9b1c0 0x55e14d3df420 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f958000d8d0 tx=0x7f958000d900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1f8d2e825dc6265 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.696+0000 7f957f400640 1 -- 172.21.15.134:0/2908453378 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958002d070 con 0x55e14cf9b1c0 2024-07-26T11:18:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.697+0000 7f957f400640 1 -- 172.21.15.134:0/2908453378 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f958000f3d0 con 0x55e14cf9b1c0 2024-07-26T11:18:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.697+0000 7f957f400640 1 -- 172.21.15.134:0/2908453378 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9580022030 con 0x55e14cf9b1c0 2024-07-26T11:18:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.700+0000 7f95a11db740 1 -- 172.21.15.134:0/2908453378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cf9b1c0 msgr2=0x55e14d3df420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.700+0000 7f95a11db740 1 --2- 172.21.15.134:0/2908453378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cf9b1c0 0x55e14d3df420 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f958000d8d0 tx=0x7f958000d900 comp rx=0 tx=0).stop 2024-07-26T11:18:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.700+0000 7f95a11db740 1 -- 172.21.15.134:0/2908453378 shutdown_connections 2024-07-26T11:18:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.701+0000 7f95a11db740 1 --2- 172.21.15.134:0/2908453378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14cf9b1c0 0x55e14d3df420 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.701+0000 7f95a11db740 1 --2- 172.21.15.134:0/2908453378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d0071b0 0x55e14d3deee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.701+0000 7f95a11db740 1 --2- 172.21.15.134:0/2908453378 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14cfde2f0 0x55e14d3b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.701+0000 7f95a11db740 1 -- 172.21.15.134:0/2908453378 >> 172.21.15.134:0/2908453378 conn(0x55e14cffb300 msgr2=0x55e14d3afb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.701+0000 7f95a11db740 1 -- 172.21.15.134:0/2908453378 shutdown_connections 2024-07-26T11:18:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.701+0000 7f95a11db740 1 -- 172.21.15.134:0/2908453378 wait complete. 2024-07-26T11:18:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.702+0000 7f95a11db740 1 -- start start 2024-07-26T11:18:21.704 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.703+0000 7f95a11db740 1 Processor -- start 2024-07-26T11:18:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.704+0000 7f95a11db740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14cf9b1c0 0x55e14d3cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.704+0000 7f95a11db740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 0x55e14d3cb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.705+0000 7f9593400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14cf9b1c0 0x55e14d3cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.705+0000 7f95a11db740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d0071b0 0x55e14d3cb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.705+0000 7f9593400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14cf9b1c0 0x55e14d3cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50620/0 (socket says 172.21.15.134:50620) 2024-07-26T11:18:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.705+0000 7f95a11db740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f95941afbf0 con 0x55e14cf9b1c0 2024-07-26T11:18:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.705+0000 7f95a11db740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f95941af520 con 0x55e14cfde2f0 2024-07-26T11:18:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.705+0000 7f95a11db740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f95941a6030 con 0x55e14d0071b0 2024-07-26T11:18:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.705+0000 7f9592a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 0x55e14d3cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.705+0000 7f9592a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 0x55e14d3cb210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43488/0 (socket says 172.21.15.134:43488) 2024-07-26T11:18:21.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.705+0000 7f9592a00640 1 -- 172.21.15.134:0/1086856086 learned_addr learned my addr 172.21.15.134:0/1086856086 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.705+0000 7f95a11db740 10 client.0 _ll_register_callbacks cb 0x7f95840ca9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.706+0000 7f9593e00640 1 --2- 172.21.15.134:0/1086856086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d0071b0 0x55e14d3cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.707+0000 7f9592a00640 1 -- 172.21.15.134:0/1086856086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d0071b0 msgr2=0x55e14d3cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.707+0000 7f9592a00640 1 --2- 172.21.15.134:0/1086856086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d0071b0 0x55e14d3cb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.707+0000 7f9592a00640 1 -- 172.21.15.134:0/1086856086 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14cf9b1c0 msgr2=0x55e14d3cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:21.709 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.707+0000 7f9592a00640 1 --2- 172.21.15.134:0/1086856086 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14cf9b1c0 0x55e14d3cfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.710 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.707+0000 7f9592a00640 1 -- 172.21.15.134:0/1086856086 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e14d440eb0 con 0x55e14cfde2f0 2024-07-26T11:18:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.709+0000 7f9592a00640 1 --2- 172.21.15.134:0/1086856086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 0x55e14d3cb210 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9584036040 tx=0x7f95840c70b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa53da4c24f4b9df server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.709+0000 7f957cc00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:18:21.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.711+0000 7f957cc00640 1 -- 172.21.15.134:0/1086856086 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95840ca360 con 0x55e14cfde2f0 2024-07-26T11:18:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.712+0000 7f957cc00640 1 -- 172.21.15.134:0/1086856086 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f95840e50d0 con 0x55e14cfde2f0 2024-07-26T11:18:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.713+0000 7f95a11db740 1 -- 172.21.15.134:0/1086856086 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f958000f3d0 con 0x55e14cfde2f0 2024-07-26T11:18:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.713+0000 7f957cc00640 1 -- 172.21.15.134:0/1086856086 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f95840b2320 con 0x55e14cfde2f0 2024-07-26T11:18:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.713+0000 7f95a11db740 10 client.25967 fetch_fsmap learned FSMap version 31 2024-07-26T11:18:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.714+0000 7f95a11db740 1 -- 172.21.15.134:0/1086856086 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e14d441ff0 con 0x55e14cfde2f0 2024-07-26T11:18:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.714+0000 7f957cc00640 1 -- 172.21.15.134:0/1086856086 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f958000f3d0 con 0x55e14cfde2f0 2024-07-26T11:18:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.715+0000 7f957cc00640 1 -- 172.21.15.134:0/1086856086 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f95840c9e50 con 0x55e14cfde2f0 2024-07-26T11:18:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.715+0000 7f95a11db740 10 client.25967 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:18:21.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.716+0000 7f95a11db740 10 client.25967 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:18:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.716+0000 7f95a11db740 20 client.25967 populate_metadata read hostname 'smithi134' 2024-07-26T11:18:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.716+0000 7f95a11db740 1 --2- 172.21.15.134:0/1086856086 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e14d442500 0x55e14d444920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.716+0000 7f95a11db740 4 client.25967 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:18:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.716+0000 7f95a11db740 1 -- 172.21.15.134:0/1086856086 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e14d31cc60 con 0x55e14d442500 2024-07-26T11:18:21.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.717+0000 7f9593400640 1 --2- 172.21.15.134:0/1086856086 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e14d442500 0x55e14d444920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:21.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.720+0000 7f9593400640 1 --2- 172.21.15.134:0/1086856086 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e14d442500 0x55e14d444920 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb4fa2dc655eb690 server_cookie=b95037d59dc57123 in_seq=0 out_seq=0 2024-07-26T11:18:21.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.720+0000 7f957cc00640 10 client.25967 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:21.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.918+0000 7f957cc00640 1 -- 172.21.15.134:0/1086856086 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e14d31cc60 con 0x55e14d442500 2024-07-26T11:18:21.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.918+0000 7f957cc00640 10 client.25967 handle_command_reply: tid=0 2024-07-26T11:18:21.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.918+0000 7f95a11db740 1 client.25967 shutdown 2024-07-26T11:18:21.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.919+0000 7f95a11db740 1 -- 172.21.15.134:0/1086856086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 msgr2=0x55e14d3cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:21.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.919+0000 7f95a11db740 1 --2- 172.21.15.134:0/1086856086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 0x55e14d3cb210 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9584036040 tx=0x7f95840c70b0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.919+0000 7f95a11db740 1 -- 172.21.15.134:0/1086856086 shutdown_connections 2024-07-26T11:18:21.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.919+0000 7f95a11db740 1 --2- 172.21.15.134:0/1086856086 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e14d442500 0x55e14d444920 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.919+0000 7f95a11db740 1 --2- 172.21.15.134:0/1086856086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d0071b0 0x55e14d3cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.919+0000 7f95a11db740 1 --2- 172.21.15.134:0/1086856086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14cfde2f0 0x55e14d3cb210 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.919+0000 7f95a11db740 1 --2- 172.21.15.134:0/1086856086 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e14cf9b1c0 0x55e14d3cfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.919+0000 7f95a11db740 1 -- 172.21.15.134:0/1086856086 >> 172.21.15.134:0/1086856086 conn(0x55e14cffb300 msgr2=0x55e14d3b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.920+0000 7f957cc00640 0 client.25967 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.920+0000 7f95a11db740 1 -- 172.21.15.134:0/1086856086 shutdown_connections 2024-07-26T11:18:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.920+0000 7f95a11db740 1 -- 172.21.15.134:0/1086856086 wait complete. 2024-07-26T11:18:21.923 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.920+0000 7f95a11db740 20 client.25967 trim_cache size 0 max 16384 2024-07-26T11:18:21.923 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:18:21.923 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:18:21.923 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:18:21.923 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:18:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.921+0000 7f959a200640 1 -- 172.21.15.134:0/3162927544 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95780b4070 msgr2=0x7f95780b6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.921+0000 7f959a200640 1 --2- 172.21.15.134:0/3162927544 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95780b4070 0x7f95780b6580 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f958001f000 tx=0x7f958001c000 comp rx=0 tx=0).stop 2024-07-26T11:18:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.921+0000 7f959a200640 1 -- 172.21.15.134:0/3162927544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9594108e40 msgr2=0x7f9594075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.921+0000 7f959a200640 1 --2- 172.21.15.134:0/3162927544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9594108e40 0x7f9594075140 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9584031440 tx=0x7f9584003b20 comp rx=0 tx=0).stop 2024-07-26T11:18:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.922+0000 7f959a200640 1 -- 172.21.15.134:0/3162927544 shutdown_connections 2024-07-26T11:18:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.922+0000 7f959a200640 1 --2- 172.21.15.134:0/3162927544 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f95780b4070 0x7f95780b6580 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.922+0000 7f959a200640 1 --2- 172.21.15.134:0/3162927544 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9594109790 0x7f9594075680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.922+0000 7f959a200640 1 --2- 172.21.15.134:0/3162927544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9594108e40 0x7f9594075140 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.925 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.922+0000 7f959a200640 1 --2- 172.21.15.134:0/3162927544 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9594103510 0x7f9594076b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:21.925 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.922+0000 7f959a200640 1 -- 172.21.15.134:0/3162927544 >> 172.21.15.134:0/3162927544 conn(0x7f95940fef50 msgr2=0x7f9594101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:21.925 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.922+0000 7f959a200640 1 -- 172.21.15.134:0/3162927544 shutdown_connections 2024-07-26T11:18:21.925 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:21.922+0000 7f959a200640 1 -- 172.21.15.134:0/3162927544 wait complete. 2024-07-26T11:18:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:21 smithi189 ceph-mon[36790]: scrub summary: idle 2024-07-26T11:18:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:21 smithi189 ceph-mon[36790]: pgmap v551: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 193 MiB/s wr, 1.71k op/s 2024-07-26T11:18:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:21 smithi193 ceph-mon[37473]: scrub summary: idle 2024-07-26T11:18:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:21 smithi193 ceph-mon[37473]: pgmap v551: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 193 MiB/s wr, 1.71k op/s 2024-07-26T11:18:22.612 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-26T11:18:22.612 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:18:23.386 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:18:23.914 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:23 smithi134 ceph-mon[31966]: pgmap v552: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 125 MiB/s wr, 1.36k op/s 2024-07-26T11:18:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:23 smithi189 ceph-mon[36790]: pgmap v552: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 125 MiB/s wr, 1.36k op/s 2024-07-26T11:18:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:23 smithi193 ceph-mon[37473]: pgmap v552: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 125 MiB/s wr, 1.36k op/s 2024-07-26T11:18:25.979 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:25 smithi134 ceph-mon[31966]: pgmap v553: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 178 MiB/s wr, 1.84k op/s 2024-07-26T11:18:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:25 smithi193 ceph-mon[37473]: pgmap v553: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 178 MiB/s wr, 1.84k op/s 2024-07-26T11:18:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:25 smithi189 ceph-mon[36790]: pgmap v553: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 178 MiB/s wr, 1.84k op/s 2024-07-26T11:18:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:27 smithi193 ceph-mon[37473]: pgmap v554: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 196 MiB/s wr, 1.99k op/s 2024-07-26T11:18:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:27 smithi134 ceph-mon[31966]: pgmap v554: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 196 MiB/s wr, 1.99k op/s 2024-07-26T11:18:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:27 smithi189 ceph-mon[36790]: pgmap v554: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 196 MiB/s wr, 1.99k op/s 2024-07-26T11:18:30.176 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:29 smithi134 ceph-mon[31966]: pgmap v555: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 203 MiB/s wr, 2.00k op/s 2024-07-26T11:18:30.176 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:18:30.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:29 smithi189 ceph-mon[36790]: pgmap v555: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 203 MiB/s wr, 2.00k op/s 2024-07-26T11:18:30.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:18:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:29 smithi193 ceph-mon[37473]: pgmap v555: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 203 MiB/s wr, 2.00k op/s 2024-07-26T11:18:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:18:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:30 smithi134 ceph-mon[31966]: pgmap v556: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 244 MiB/s wr, 2.35k op/s 2024-07-26T11:18:31.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:30 smithi189 ceph-mon[36790]: pgmap v556: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 244 MiB/s wr, 2.35k op/s 2024-07-26T11:18:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:30 smithi193 ceph-mon[37473]: pgmap v556: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 244 MiB/s wr, 2.35k op/s 2024-07-26T11:18:32.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:32 smithi134 ceph-mon[31966]: pgmap v557: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 175 MiB/s wr, 1.72k op/s 2024-07-26T11:18:32.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:32 smithi189 ceph-mon[36790]: pgmap v557: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 175 MiB/s wr, 1.72k op/s 2024-07-26T11:18:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:32 smithi193 ceph-mon[37473]: pgmap v557: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 175 MiB/s wr, 1.72k op/s 2024-07-26T11:18:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:34 smithi193 ceph-mon[37473]: pgmap v558: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 232 MiB/s wr, 2.26k op/s 2024-07-26T11:18:34.677 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:34 smithi134 ceph-mon[31966]: pgmap v558: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 232 MiB/s wr, 2.26k op/s 2024-07-26T11:18:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:34 smithi189 ceph-mon[36790]: pgmap v558: 129 pgs: 129 active+clean; 12 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 232 MiB/s wr, 2.26k op/s 2024-07-26T11:18:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.337+0000 7f0dc7e00640 1 -- 172.21.15.134:0/11429002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc8104170 msgr2=0x7f0dc8104580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.337+0000 7f0dc7e00640 1 --2- 172.21.15.134:0/11429002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc8104170 0x7f0dc8104580 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0db8009900 tx=0x7f0db802a440 comp rx=0 tx=0).stop 2024-07-26T11:18:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.343+0000 7f0dc7e00640 1 -- 172.21.15.134:0/11429002 shutdown_connections 2024-07-26T11:18:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.343+0000 7f0dc7e00640 1 --2- 172.21.15.134:0/11429002 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8104ac0 0x7f0dc810d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.343+0000 7f0dc7e00640 1 --2- 172.21.15.134:0/11429002 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0dc810d5d0 0x7f0dc810fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.343+0000 7f0dc7e00640 1 --2- 172.21.15.134:0/11429002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc8104170 0x7f0dc8104580 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.344+0000 7f0dc7e00640 1 -- 172.21.15.134:0/11429002 >> 172.21.15.134:0/11429002 conn(0x7f0dc80fdc60 msgr2=0x7f0dc81000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.346+0000 7f0dc7e00640 1 -- 172.21.15.134:0/11429002 shutdown_connections 2024-07-26T11:18:35.348 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.346+0000 7f0dc7e00640 1 -- 172.21.15.134:0/11429002 wait complete. 2024-07-26T11:18:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.348+0000 7f0dc7e00640 1 Processor -- start 2024-07-26T11:18:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.348+0000 7f0dc7e00640 1 -- start start 2024-07-26T11:18:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.349+0000 7f0dc7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8104170 0x7f0dc810cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.349+0000 7f0dc7e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0dc8104ac0 0x7f0dc8107080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.349+0000 7f0dc7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc810d5d0 0x7f0dc8107600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.349+0000 7f0dc7e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc8111150 con 0x7f0dc8104ac0 2024-07-26T11:18:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.349+0000 7f0dc7e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc8110fd0 con 0x7f0dc810d5d0 2024-07-26T11:18:35.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.349+0000 7f0dc7e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc81112d0 con 0x7f0dc8104170 2024-07-26T11:18:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.349+0000 7f0dc6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8104170 0x7f0dc810cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.349+0000 7f0dc6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8104170 0x7f0dc810cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:41754/0 (socket says 172.21.15.134:41754) 2024-07-26T11:18:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.349+0000 7f0dc6a00640 1 -- 172.21.15.134:0/343818949 learned_addr learned my addr 172.21.15.134:0/343818949 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:35.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.350+0000 7f0dc6a00640 1 -- 172.21.15.134:0/343818949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc810d5d0 msgr2=0x7f0dc8107600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-07-26T11:18:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.350+0000 7f0dc6a00640 1 --2- 172.21.15.134:0/343818949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc810d5d0 0x7f0dc8107600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.350+0000 7f0dc6a00640 1 -- 172.21.15.134:0/343818949 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0dc8104ac0 msgr2=0x7f0dc8107080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-07-26T11:18:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.350+0000 7f0dc6a00640 1 --2- 172.21.15.134:0/343818949 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0dc8104ac0 0x7f0dc8107080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.350+0000 7f0dc6a00640 1 -- 172.21.15.134:0/343818949 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc8107e20 con 0x7f0dc8104170 2024-07-26T11:18:35.355 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.353+0000 7f0dc6a00640 1 --2- 172.21.15.134:0/343818949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8104170 0x7f0dc810cdd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0db802a9b0 tx=0x7f0db8003e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:35.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.357+0000 7f0dbf400640 1 -- 172.21.15.134:0/343818949 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db803f030 con 0x7f0dc8104170 2024-07-26T11:18:35.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.358+0000 7f0dbf400640 1 -- 172.21.15.134:0/343818949 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0db8032bf0 con 0x7f0dc8104170 2024-07-26T11:18:35.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.358+0000 7f0dbf400640 1 -- 172.21.15.134:0/343818949 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db8035070 con 0x7f0dc8104170 2024-07-26T11:18:35.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.358+0000 7f0dc7e00640 1 -- 172.21.15.134:0/343818949 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dc811f290 con 0x7f0dc8104170 2024-07-26T11:18:35.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.358+0000 7f0dc7e00640 1 -- 172.21.15.134:0/343818949 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dc811f7b0 con 0x7f0dc8104170 2024-07-26T11:18:35.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.375+0000 7f0dd4c39740 1 Processor -- start 2024-07-26T11:18:35.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.375+0000 7f0dbf400640 1 -- 172.21.15.134:0/343818949 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f0db8042030 con 0x7f0dc8104170 2024-07-26T11:18:35.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.378+0000 7f0dd4c39740 1 -- start start 2024-07-26T11:18:35.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.379+0000 7f0dd4c39740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21db0c1b0 0x55e21deafc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.379+0000 7f0dd4c39740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21dae32f0 0x55e21deb0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.379+0000 7f0dd4c39740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21daeb4d0 0x55e21deb06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.380+0000 7f0dd4c39740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e21de24270 con 0x55e21daeb4d0 2024-07-26T11:18:35.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.380+0000 7f0dd4c39740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e21de13770 con 0x55e21dae32f0 2024-07-26T11:18:35.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.380+0000 7f0dd4c39740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e21de177b0 con 0x55e21db0c1b0 2024-07-26T11:18:35.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.382+0000 7f0dc6000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21daeb4d0 0x55e21deb06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:35.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.382+0000 7f0dc6000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21daeb4d0 0x55e21deb06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:40032/0 (socket says 172.21.15.134:40032) 2024-07-26T11:18:35.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.382+0000 7f0dc6000640 1 -- 172.21.15.134:0/828805258 learned_addr learned my addr 172.21.15.134:0/828805258 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:35.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.384+0000 7f0dc6000640 1 -- 172.21.15.134:0/828805258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21db0c1b0 msgr2=0x55e21deafc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:35.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.384+0000 7f0dc6000640 1 --2- 172.21.15.134:0/828805258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21db0c1b0 0x55e21deafc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.384+0000 7f0dc6000640 1 -- 172.21.15.134:0/828805258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21dae32f0 msgr2=0x55e21deb0160 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:18:35.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.384+0000 7f0dc6000640 1 --2- 172.21.15.134:0/828805258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21dae32f0 0x55e21deb0160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.384+0000 7f0dc6000640 1 -- 172.21.15.134:0/828805258 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e21db77790 con 0x55e21daeb4d0 2024-07-26T11:18:35.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.385+0000 7f0dbf400640 1 --2- 172.21.15.134:0/343818949 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0dac0b3f50 0x7f0dac0b6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.386+0000 7f0dbf400640 1 -- 172.21.15.134:0/343818949 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0db81046e0 con 0x7f0dc8104170 2024-07-26T11:18:35.389 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.387+0000 7f0dc7400640 1 --2- 172.21.15.134:0/343818949 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0dac0b3f50 0x7f0dac0b6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:35.391 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.389+0000 7f0dc6000640 1 --2- 172.21.15.134:0/828805258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21daeb4d0 0x55e21deb06a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0db40096e0 tx=0x7f0db4009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbc0745bc12f845f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:35.391 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.389+0000 7f0dc7400640 1 --2- 172.21.15.134:0/343818949 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0dac0b3f50 0x7f0dac0b6460 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000aa90 tx=0x7f0dc0009250 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:35.392 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.390+0000 7f0db3e00640 1 -- 172.21.15.134:0/828805258 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db4015020 con 0x55e21daeb4d0 2024-07-26T11:18:35.392 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.390+0000 7f0db3e00640 1 -- 172.21.15.134:0/828805258 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0db400bd70 con 0x55e21daeb4d0 2024-07-26T11:18:35.392 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.390+0000 7f0db3e00640 1 -- 172.21.15.134:0/828805258 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db4010380 con 0x55e21daeb4d0 2024-07-26T11:18:35.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.392+0000 7f0dd4c39740 1 -- 172.21.15.134:0/828805258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21daeb4d0 msgr2=0x55e21deb06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:35.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.392+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/828805258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21daeb4d0 0x55e21deb06a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0db40096e0 tx=0x7f0db4009ba0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.392+0000 7f0dd4c39740 1 -- 172.21.15.134:0/828805258 shutdown_connections 2024-07-26T11:18:35.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.393+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/828805258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21daeb4d0 0x55e21deb06a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.393+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/828805258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21dae32f0 0x55e21deb0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.393+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/828805258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21db0c1b0 0x55e21deafc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.393+0000 7f0dd4c39740 1 -- 172.21.15.134:0/828805258 >> 172.21.15.134:0/828805258 conn(0x55e21db00300 msgr2=0x55e21deadb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:35.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.394+0000 7f0dd4c39740 1 -- 172.21.15.134:0/828805258 shutdown_connections 2024-07-26T11:18:35.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.394+0000 7f0dd4c39740 1 -- 172.21.15.134:0/828805258 wait complete. 2024-07-26T11:18:35.397 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.395+0000 7f0dd4c39740 1 -- start start 2024-07-26T11:18:35.397 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.396+0000 7f0dd4c39740 1 Processor -- start 2024-07-26T11:18:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.396+0000 7f0dd4c39740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dae32f0 0x55e21debb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.396+0000 7f0dd4c39740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21daeb4d0 0x55e21debbf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.396+0000 7f0dd4c39740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21db0c1b0 0x55e21dec1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.396+0000 7f0dd4c39740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e21dd4dab0 con 0x55e21daeb4d0 2024-07-26T11:18:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.396+0000 7f0dd4c39740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e21ddb0d50 con 0x55e21db0c1b0 2024-07-26T11:18:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.396+0000 7f0dd4c39740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e21debe3a0 con 0x55e21dae32f0 2024-07-26T11:18:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.396+0000 7f0dd4c39740 10 client.0 _ll_register_callbacks cb 0x55e21db23a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.397+0000 7f0dc6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dae32f0 0x55e21debb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.397+0000 7f0dc6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dae32f0 0x55e21debb940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:41770/0 (socket says 172.21.15.134:41770) 2024-07-26T11:18:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.397+0000 7f0dc6a00640 1 -- 172.21.15.134:0/1270893044 learned_addr learned my addr 172.21.15.134:0/1270893044 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.397+0000 7f0dc7400640 1 --2- 172.21.15.134:0/1270893044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21db0c1b0 0x55e21dec1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:35.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.398+0000 7f0dc6000640 1 --2- 172.21.15.134:0/1270893044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21daeb4d0 0x55e21debbf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:35.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.398+0000 7f0dc6a00640 1 -- 172.21.15.134:0/1270893044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21db0c1b0 msgr2=0x55e21dec1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:35.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.398+0000 7f0dc6a00640 1 --2- 172.21.15.134:0/1270893044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21db0c1b0 0x55e21dec1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.398+0000 7f0dc6a00640 1 -- 172.21.15.134:0/1270893044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21daeb4d0 msgr2=0x55e21debbf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:35.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.398+0000 7f0dc6a00640 1 --2- 172.21.15.134:0/1270893044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21daeb4d0 0x55e21debbf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.398+0000 7f0dc6a00640 1 -- 172.21.15.134:0/1270893044 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e21dbf0620 con 0x55e21dae32f0 2024-07-26T11:18:35.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.398+0000 7f0dc6000640 1 --2- 172.21.15.134:0/1270893044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21daeb4d0 0x55e21debbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:18:35.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.399+0000 7f0dc6a00640 1 --2- 172.21.15.134:0/1270893044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dae32f0 0x55e21debb940 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0db802aa30 tx=0x7f0db80caae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5567c994cc5aa22 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:35.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.399+0000 7f0db0c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:18:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.402+0000 7f0db0c00640 1 -- 172.21.15.134:0/1270893044 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db80e4020 con 0x55e21dae32f0 2024-07-26T11:18:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.402+0000 7f0db0c00640 1 -- 172.21.15.134:0/1270893044 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0db803bc70 con 0x55e21dae32f0 2024-07-26T11:18:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.402+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1270893044 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0db400bd70 con 0x55e21dae32f0 2024-07-26T11:18:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.403+0000 7f0db0c00640 1 -- 172.21.15.134:0/1270893044 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0db80cf2f0 con 0x55e21dae32f0 2024-07-26T11:18:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.403+0000 7f0dd4c39740 10 client.25963 fetch_fsmap learned FSMap version 31 2024-07-26T11:18:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.403+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1270893044 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55e21deca500 con 0x55e21dae32f0 2024-07-26T11:18:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.404+0000 7f0db0c00640 1 -- 172.21.15.134:0/1270893044 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0db80caf10 con 0x55e21dae32f0 2024-07-26T11:18:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.404+0000 7f0db0c00640 1 -- 172.21.15.134:0/1270893044 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f0db8101cd0 con 0x55e21dae32f0 2024-07-26T11:18:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.404+0000 7f0dd4c39740 10 client.25963 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:18:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.404+0000 7f0dd4c39740 10 client.25963 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:18:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.404+0000 7f0dd4c39740 20 client.25963 populate_metadata read hostname 'smithi134' 2024-07-26T11:18:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.405+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/1270893044 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e21dacad80 0x55e21decd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.405+0000 7f0dd4c39740 4 client.25963 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:18:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.405+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1270893044 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0db400bd70 con 0x55e21dacad80 2024-07-26T11:18:35.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.405+0000 7f0dc6000640 1 --2- 172.21.15.134:0/1270893044 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e21dacad80 0x55e21decd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:35.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.407+0000 7f0dc6000640 1 --2- 172.21.15.134:0/1270893044 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e21dacad80 0x55e21decd970 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d552e36594561725 server_cookie=3361e49af2481dcf in_seq=0 out_seq=0 2024-07-26T11:18:35.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.407+0000 7f0db0c00640 10 client.25963 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.410+0000 7f0db0c00640 1 -- 172.21.15.134:0/1270893044 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f0db400bd70 con 0x55e21dacad80 2024-07-26T11:18:35.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.410+0000 7f0db0c00640 10 client.25963 handle_command_reply: tid=0 2024-07-26T11:18:35.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.410+0000 7f0dd4c39740 1 client.25963 shutdown 2024-07-26T11:18:35.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.411+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1270893044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dae32f0 msgr2=0x55e21debb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:35.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.411+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/1270893044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dae32f0 0x55e21debb940 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0db802aa30 tx=0x7f0db80caae0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.413 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.411+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1270893044 shutdown_connections 2024-07-26T11:18:35.413 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.411+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/1270893044 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e21dacad80 0x55e21decd970 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.413 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.411+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/1270893044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21db0c1b0 0x55e21dec1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.413 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.411+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/1270893044 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21daeb4d0 0x55e21debbf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.411+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/1270893044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21dae32f0 0x55e21debb940 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.411+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1270893044 >> 172.21.15.134:0/1270893044 conn(0x55e21db00300 msgr2=0x55e21deb0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:35.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.412+0000 7f0db0c00640 0 client.25963 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:35.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.413+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1270893044 shutdown_connections 2024-07-26T11:18:35.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.418+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1270893044 wait complete. 2024-07-26T11:18:35.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.419+0000 7f0dd4c39740 20 client.25963 trim_cache size 0 max 16384 2024-07-26T11:18:35.484 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.482+0000 7f0dd4c39740 1 Processor -- start 2024-07-26T11:18:35.484 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.482+0000 7f0dd4c39740 1 -- start start 2024-07-26T11:18:35.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.483+0000 7f0dd4c39740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21dae32f0 0x55e21dec15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.483+0000 7f0dd4c39740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21daeb4d0 0x55e21dec8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.483+0000 7f0dd4c39740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21db0c1b0 0x55e21deada10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.483+0000 7f0dd4c39740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc81d28a0 con 0x55e21dae32f0 2024-07-26T11:18:35.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.483+0000 7f0dd4c39740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc8129940 con 0x55e21db0c1b0 2024-07-26T11:18:35.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.483+0000 7f0dd4c39740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc81d21d0 con 0x55e21daeb4d0 2024-07-26T11:18:35.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.483+0000 7f0dc6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21daeb4d0 0x55e21dec8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:35.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.483+0000 7f0dc6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21daeb4d0 0x55e21dec8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:41786/0 (socket says 172.21.15.134:41786) 2024-07-26T11:18:35.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.483+0000 7f0dc6000640 1 -- 172.21.15.134:0/3121184107 learned_addr learned my addr 172.21.15.134:0/3121184107 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.484+0000 7f0dc7400640 1 --2- 172.21.15.134:0/3121184107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21db0c1b0 0x55e21deada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.484+0000 7f0dc6a00640 1 --2- 172.21.15.134:0/3121184107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21dae32f0 0x55e21dec15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.484+0000 7f0dc6000640 1 -- 172.21.15.134:0/3121184107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21db0c1b0 msgr2=0x55e21deada10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.484+0000 7f0dc6000640 1 --2- 172.21.15.134:0/3121184107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21db0c1b0 0x55e21deada10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.484+0000 7f0dc6000640 1 -- 172.21.15.134:0/3121184107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21dae32f0 msgr2=0x55e21dec15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.484+0000 7f0dc6000640 1 --2- 172.21.15.134:0/3121184107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21dae32f0 0x55e21dec15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.484+0000 7f0dc6000640 1 -- 172.21.15.134:0/3121184107 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e21dec2160 con 0x55e21daeb4d0 2024-07-26T11:18:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.484+0000 7f0dc6a00640 1 --2- 172.21.15.134:0/3121184107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21dae32f0 0x55e21dec15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:18:35.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.485+0000 7f0dc6000640 1 --2- 172.21.15.134:0/3121184107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21daeb4d0 0x55e21dec8460 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0db4011bf0 tx=0x7f0db4009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d85249e2a39bf300 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.488+0000 7f0db3400640 1 -- 172.21.15.134:0/3121184107 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db4027020 con 0x55e21daeb4d0 2024-07-26T11:18:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.488+0000 7f0db3400640 1 -- 172.21.15.134:0/3121184107 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0db40044f0 con 0x55e21daeb4d0 2024-07-26T11:18:35.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.488+0000 7f0db3400640 1 -- 172.21.15.134:0/3121184107 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db4015040 con 0x55e21daeb4d0 2024-07-26T11:18:35.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.494+0000 7f0dd4c39740 1 -- 172.21.15.134:0/3121184107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21daeb4d0 msgr2=0x55e21dec8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:35.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.494+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/3121184107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21daeb4d0 0x55e21dec8460 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0db4011bf0 tx=0x7f0db4009060 comp rx=0 tx=0).stop 2024-07-26T11:18:35.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.497+0000 7f0dd4c39740 1 -- 172.21.15.134:0/3121184107 shutdown_connections 2024-07-26T11:18:35.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.497+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/3121184107 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21daeb4d0 0x55e21dec8460 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.497+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/3121184107 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21db0c1b0 0x55e21deada10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.497+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/3121184107 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21dae32f0 0x55e21dec15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.497+0000 7f0dd4c39740 1 -- 172.21.15.134:0/3121184107 >> 172.21.15.134:0/3121184107 conn(0x55e21db00300 msgr2=0x55e21dead300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:35.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.498+0000 7f0dd4c39740 1 -- 172.21.15.134:0/3121184107 shutdown_connections 2024-07-26T11:18:35.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.498+0000 7f0dd4c39740 1 -- 172.21.15.134:0/3121184107 wait complete. 2024-07-26T11:18:35.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.498+0000 7f0dd4c39740 1 -- start start 2024-07-26T11:18:35.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.500+0000 7f0dd4c39740 1 Processor -- start 2024-07-26T11:18:35.508 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.505+0000 7f0dd4c39740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21dae32f0 0x55e21dec2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.508 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.505+0000 7f0dd4c39740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21daeb4d0 0x55e21dec3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.508 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.505+0000 7f0dd4c39740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21db0c1b0 0x55e21dec35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.508 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.505+0000 7f0dd4c39740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc81d1400 con 0x55e21dae32f0 2024-07-26T11:18:35.509 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.506+0000 7f0dd4c39740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc812a010 con 0x55e21daeb4d0 2024-07-26T11:18:35.509 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.506+0000 7f0dc7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21db0c1b0 0x55e21dec35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:35.509 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.506+0000 7f0dc7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21db0c1b0 0x55e21dec35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:41790/0 (socket says 172.21.15.134:41790) 2024-07-26T11:18:35.509 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.506+0000 7f0dc7400640 1 -- 172.21.15.134:0/1096322212 learned_addr learned my addr 172.21.15.134:0/1096322212 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:35.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.506+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1096322212 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0dc81d1ad0 con 0x55e21db0c1b0 2024-07-26T11:18:35.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.506+0000 7f0dc7400640 1 -- 172.21.15.134:0/1096322212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21daeb4d0 msgr2=0x55e21dec3080 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:18:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.506+0000 7f0dc7400640 1 --2- 172.21.15.134:0/1096322212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21daeb4d0 0x55e21dec3080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.506+0000 7f0dc7400640 1 -- 172.21.15.134:0/1096322212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21dae32f0 msgr2=0x55e21dec2b40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:18:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.507+0000 7f0dd4c39740 10 client.0 _ll_register_callbacks cb 0x7f0db8009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.507+0000 7f0dc6a00640 1 --2- 172.21.15.134:0/1096322212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21dae32f0 0x55e21dec2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.507+0000 7f0dc7400640 1 --2- 172.21.15.134:0/1096322212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21dae32f0 0x55e21dec2b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.508+0000 7f0dc7400640 1 -- 172.21.15.134:0/1096322212 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e21df34270 con 0x55e21db0c1b0 2024-07-26T11:18:35.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.508+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1096322212 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0db40044f0 con 0x55e21db0c1b0 2024-07-26T11:18:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.509+0000 7f0dc7400640 1 --2- 172.21.15.134:0/1096322212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21db0c1b0 0x55e21dec35c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0dc001be30 tx=0x7f0dc0004bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67114390a0922b3c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:35.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.509+0000 7f0db0c00640 10 client.25969 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:18:35.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.510+0000 7f0db0c00640 1 -- 172.21.15.134:0/1096322212 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc0029060 con 0x55e21db0c1b0 2024-07-26T11:18:35.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.510+0000 7f0db0c00640 1 -- 172.21.15.134:0/1096322212 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f0dc0024410 con 0x55e21db0c1b0 2024-07-26T11:18:35.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.511+0000 7f0db0c00640 1 -- 172.21.15.134:0/1096322212 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f0dc0024670 con 0x55e21db0c1b0 2024-07-26T11:18:35.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.511+0000 7f0dd4c39740 10 client.25969 fetch_fsmap learned FSMap version 31 2024-07-26T11:18:35.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.511+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1096322212 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55e21df35480 con 0x55e21db0c1b0 2024-07-26T11:18:35.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.512+0000 7f0db0c00640 1 -- 172.21.15.134:0/1096322212 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f0dc002c050 con 0x55e21db0c1b0 2024-07-26T11:18:35.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.512+0000 7f0dd4c39740 10 client.25969 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:18:35.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.512+0000 7f0dd4c39740 10 client.25969 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:18:35.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.512+0000 7f0dd4c39740 20 client.25969 populate_metadata read hostname 'smithi134' 2024-07-26T11:18:35.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.512+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/1096322212 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e21daa01c0 0x55e21df37be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:35.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.512+0000 7f0dd4c39740 4 client.25969 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:18:35.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.512+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1096322212 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e21de261c0 con 0x55e21daa01c0 2024-07-26T11:18:35.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.513+0000 7f0dc6a00640 1 --2- 172.21.15.134:0/1096322212 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e21daa01c0 0x55e21df37be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:35.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.514+0000 7f0dc6a00640 1 --2- 172.21.15.134:0/1096322212 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e21daa01c0 0x55e21df37be0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c17d386302104ca server_cookie=e5eabf01959cb8e8 in_seq=0 out_seq=0 2024-07-26T11:18:35.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:35.514+0000 7f0db0c00640 10 client.25969 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.024+0000 7f0db0c00640 1 -- 172.21.15.134:0/1096322212 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e21de261c0 con 0x55e21daa01c0 2024-07-26T11:18:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.024+0000 7f0db0c00640 10 client.25969 handle_command_reply: tid=0 2024-07-26T11:18:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.024+0000 7f0dd4c39740 1 client.25969 shutdown 2024-07-26T11:18:36.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.025+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1096322212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21db0c1b0 msgr2=0x55e21dec35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:36.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.025+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/1096322212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21db0c1b0 0x55e21dec35c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0dc001be30 tx=0x7f0dc0004bc0 comp rx=0 tx=0).stop 2024-07-26T11:18:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.030+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1096322212 shutdown_connections 2024-07-26T11:18:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.030+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/1096322212 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e21daa01c0 0x55e21df37be0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.030+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/1096322212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e21db0c1b0 0x55e21dec35c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.030+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/1096322212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e21daeb4d0 0x55e21dec3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:36.032 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.030+0000 7f0dd4c39740 1 --2- 172.21.15.134:0/1096322212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e21dae32f0 0x55e21dec2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:36.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.030+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1096322212 >> 172.21.15.134:0/1096322212 conn(0x55e21db00300 msgr2=0x55e21deb0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:36.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.030+0000 7f0db0c00640 0 client.25969 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:36.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.031+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1096322212 shutdown_connections 2024-07-26T11:18:36.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.031+0000 7f0dd4c39740 1 -- 172.21.15.134:0/1096322212 wait complete. 2024-07-26T11:18:36.034 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.031+0000 7f0dd4c39740 20 client.25969 trim_cache size 0 max 16384 2024-07-26T11:18:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:18:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:18:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:18:36.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:18:36.045 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.042+0000 7f0dc7e00640 1 -- 172.21.15.134:0/343818949 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0dac0b3f50 msgr2=0x7f0dac0b6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:36.045 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.042+0000 7f0dc7e00640 1 --2- 172.21.15.134:0/343818949 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0dac0b3f50 0x7f0dac0b6460 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0dc000aa90 tx=0x7f0dc0009250 comp rx=0 tx=0).stop 2024-07-26T11:18:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.043+0000 7f0dc7e00640 1 -- 172.21.15.134:0/343818949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8104170 msgr2=0x7f0dc810cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.043+0000 7f0dc7e00640 1 --2- 172.21.15.134:0/343818949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8104170 0x7f0dc810cdd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0db802a9b0 tx=0x7f0db8003e30 comp rx=0 tx=0).stop 2024-07-26T11:18:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.044+0000 7f0dc7e00640 1 -- 172.21.15.134:0/343818949 shutdown_connections 2024-07-26T11:18:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.044+0000 7f0dc7e00640 1 --2- 172.21.15.134:0/343818949 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f0dac0b3f50 0x7f0dac0b6460 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.044+0000 7f0dc7e00640 1 --2- 172.21.15.134:0/343818949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc810d5d0 0x7f0dc8107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.044+0000 7f0dc7e00640 1 --2- 172.21.15.134:0/343818949 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0dc8104ac0 0x7f0dc8107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.044+0000 7f0dc7e00640 1 --2- 172.21.15.134:0/343818949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0dc8104170 0x7f0dc810cdd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:36.046 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.044+0000 7f0dc7e00640 1 -- 172.21.15.134:0/343818949 >> 172.21.15.134:0/343818949 conn(0x7f0dc80fdc60 msgr2=0x7f0dc8112940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.045+0000 7f0dc7e00640 1 -- 172.21.15.134:0/343818949 shutdown_connections 2024-07-26T11:18:36.047 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:36.045+0000 7f0dc7e00640 1 -- 172.21.15.134:0/343818949 wait complete. 2024-07-26T11:18:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:36 smithi193 ceph-mon[37473]: pgmap v559: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 204 MiB/s wr, 2.01k op/s 2024-07-26T11:18:36.676 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:36 smithi134 ceph-mon[31966]: pgmap v559: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 204 MiB/s wr, 2.01k op/s 2024-07-26T11:18:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:36 smithi189 ceph-mon[36790]: pgmap v559: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 204 MiB/s wr, 2.01k op/s 2024-07-26T11:18:37.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:18:37.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:18:37.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:18:37.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:18:37.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:37 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T11:18:37.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:37 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T11:18:37.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:18:37.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:18:37.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:18:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:18:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:18:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:18:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:18:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:37 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T11:18:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:37 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T11:18:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:18:37.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:18:37.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:18:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:18:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:18:37.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:18:37.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:18:37.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:37 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T11:18:37.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:37 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T11:18:37.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:18:37.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:18:37.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:18:38.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:38 smithi134 ceph-mon[31966]: pgmap v560: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 185 MiB/s wr, 1.77k op/s 2024-07-26T11:18:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:38 smithi189 ceph-mon[36790]: pgmap v560: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 185 MiB/s wr, 1.77k op/s 2024-07-26T11:18:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:38 smithi193 ceph-mon[37473]: pgmap v560: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 185 MiB/s wr, 1.77k op/s 2024-07-26T11:18:40.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:40 smithi134 ceph-mon[31966]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 263 MiB/s wr, 2.25k op/s 2024-07-26T11:18:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:40 smithi189 ceph-mon[36790]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 263 MiB/s wr, 2.25k op/s 2024-07-26T11:18:40.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:40 smithi193 ceph-mon[37473]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 263 MiB/s wr, 2.25k op/s 2024-07-26T11:18:41.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.929+0000 7fa6f4000640 1 -- 172.21.15.134:0/880693686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6ec077cc0 msgr2=0x7fa6ec113c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:41.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.929+0000 7fa6f4000640 1 --2- 172.21.15.134:0/880693686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6ec077cc0 0x7fa6ec113c30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8009860 tx=0x7fa6d802a340 comp rx=0 tx=0).stop 2024-07-26T11:18:41.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.930+0000 7fa6f4000640 1 -- 172.21.15.134:0/880693686 shutdown_connections 2024-07-26T11:18:41.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.930+0000 7fa6f4000640 1 --2- 172.21.15.134:0/880693686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6ec114170 0x7fa6ec116590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:41.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.930+0000 7fa6f4000640 1 --2- 172.21.15.134:0/880693686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6ec077cc0 0x7fa6ec113c30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:41.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.930+0000 7fa6f4000640 1 --2- 172.21.15.134:0/880693686 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6ec077370 0x7fa6ec077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:41.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.930+0000 7fa6f4000640 1 -- 172.21.15.134:0/880693686 >> 172.21.15.134:0/880693686 conn(0x7fa6ec101240 msgr2=0x7fa6ec103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:41.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.932+0000 7fa6f4000640 1 -- 172.21.15.134:0/880693686 shutdown_connections 2024-07-26T11:18:41.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.932+0000 7fa6f4000640 1 -- 172.21.15.134:0/880693686 wait complete. 2024-07-26T11:18:41.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.935+0000 7fa6f4000640 1 Processor -- start 2024-07-26T11:18:41.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.935+0000 7fa6f4000640 1 -- start start 2024-07-26T11:18:41.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.936+0000 7fa6f4000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6ec077370 0x7fa6ec113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:41.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.936+0000 7fa6f4000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6ec077cc0 0x7fa6ec10dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:41.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.936+0000 7fa6f4000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6ec114170 0x7fa6ec10e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:41.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.936+0000 7fa6f4000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec105ca0 con 0x7fa6ec077370 2024-07-26T11:18:41.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.936+0000 7fa6f4000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec105b20 con 0x7fa6ec114170 2024-07-26T11:18:41.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.936+0000 7fa6f4000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ec105e20 con 0x7fa6ec077cc0 2024-07-26T11:18:41.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.936+0000 7fa6f0c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6ec077cc0 0x7fa6ec10dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:41.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.936+0000 7fa6f0c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6ec077cc0 0x7fa6ec10dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:37838/0 (socket says 172.21.15.134:37838) 2024-07-26T11:18:41.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.936+0000 7fa6f0c00640 1 -- 172.21.15.134:0/3408346068 learned_addr learned my addr 172.21.15.134:0/3408346068 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:41.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.937+0000 7fa6f2000640 1 --2- 172.21.15.134:0/3408346068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6ec114170 0x7fa6ec10e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:41.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.937+0000 7fa6f0c00640 1 -- 172.21.15.134:0/3408346068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6ec114170 msgr2=0x7fa6ec10e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:41.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.937+0000 7fa6f0c00640 1 --2- 172.21.15.134:0/3408346068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6ec114170 0x7fa6ec10e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:41.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.937+0000 7fa6f0c00640 1 -- 172.21.15.134:0/3408346068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6ec077370 msgr2=0x7fa6ec113ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-07-26T11:18:41.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.937+0000 7fa6f0c00640 1 --2- 172.21.15.134:0/3408346068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6ec077370 0x7fa6ec113ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:41.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.937+0000 7fa6f0c00640 1 -- 172.21.15.134:0/3408346068 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6ec1b4f60 con 0x7fa6ec077cc0 2024-07-26T11:18:41.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.937+0000 7fa6f0c00640 1 --2- 172.21.15.134:0/3408346068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6ec077cc0 0x7fa6ec10dd90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8035040 tx=0x7fa6d8032dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:41.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.938+0000 7fa6ea000640 1 -- 172.21.15.134:0/3408346068 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8037040 con 0x7fa6ec077cc0 2024-07-26T11:18:41.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.938+0000 7fa6ea000640 1 -- 172.21.15.134:0/3408346068 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6d8030c90 con 0x7fa6ec077cc0 2024-07-26T11:18:41.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.938+0000 7fa6ea000640 1 -- 172.21.15.134:0/3408346068 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8003850 con 0x7fa6ec077cc0 2024-07-26T11:18:41.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.938+0000 7fa6f4000640 1 -- 172.21.15.134:0/3408346068 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6ec1b5130 con 0x7fa6ec077cc0 2024-07-26T11:18:41.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.939+0000 7fa6f4000640 1 -- 172.21.15.134:0/3408346068 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6ec1b54d0 con 0x7fa6ec077cc0 2024-07-26T11:18:41.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.940+0000 7fa6f4000640 1 -- 172.21.15.134:0/3408346068 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6ec077780 con 0x7fa6ec077cc0 2024-07-26T11:18:41.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.945+0000 7fa6ea000640 1 -- 172.21.15.134:0/3408346068 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fa6d8043030 con 0x7fa6ec077cc0 2024-07-26T11:18:41.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.946+0000 7fa6ea000640 1 --2- 172.21.15.134:0/3408346068 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6d40b3e40 0x7fa6d40b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:41.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.947+0000 7fa6ea000640 1 -- 172.21.15.134:0/3408346068 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa6d8103f90 con 0x7fa6ec077cc0 2024-07-26T11:18:41.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.947+0000 7fa6f1600640 1 --2- 172.21.15.134:0/3408346068 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6d40b3e40 0x7fa6d40b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:41.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.949+0000 7fa6f1600640 1 --2- 172.21.15.134:0/3408346068 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6d40b3e40 0x7fa6d40b6350 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec113820 tx=0x7fa6e0009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:41.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:41.949+0000 7fa6ea000640 1 -- 172.21.15.134:0/3408346068 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fa6d804a020 con 0x7fa6ec077cc0 2024-07-26T11:18:42.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:42.404+0000 7fa6f4000640 1 -- 172.21.15.134:0/3408346068 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6ec10f090 con 0x7fa6ec077cc0 2024-07-26T11:18:42.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:42.405+0000 7fa6ea000640 1 -- 172.21.15.134:0/3408346068 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11991 (secure 0 0 0) 0x7fa6d80cb060 con 0x7fa6ec077cc0 2024-07-26T11:18:42.417 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:18:42.417 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":31,"btime":"2024-07-26T11:17:25:226678+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:17:25.065552+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24721,"mds_1":24719,"mds_2":24703,"mds_3":24688,"mds_4":24686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24635":{"gid":24635,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24641":{"gid":24641,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":32,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24719":{"gid":24719,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":28,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24721,"qdb_cluster":[24686,24688,24703,24719,24721]},"id":1}]} 2024-07-26T11:18:42.417 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 31 2024-07-26T11:18:42.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:42.411+0000 7fa6f4000640 1 -- 172.21.15.134:0/3408346068 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6d40b3e40 msgr2=0x7fa6d40b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:42.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:42.411+0000 7fa6f4000640 1 --2- 172.21.15.134:0/3408346068 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6d40b3e40 0x7fa6d40b6350 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec113820 tx=0x7fa6e0009500 comp rx=0 tx=0).stop 2024-07-26T11:18:42.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:42.411+0000 7fa6f4000640 1 -- 172.21.15.134:0/3408346068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6ec077cc0 msgr2=0x7fa6ec10dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:42.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:42.411+0000 7fa6f4000640 1 --2- 172.21.15.134:0/3408346068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6ec077cc0 0x7fa6ec10dd90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa6d8035040 tx=0x7fa6d8032dc0 comp rx=0 tx=0).stop 2024-07-26T11:18:42.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:42.412+0000 7fa6f4000640 1 -- 172.21.15.134:0/3408346068 shutdown_connections 2024-07-26T11:18:42.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:42.412+0000 7fa6f4000640 1 --2- 172.21.15.134:0/3408346068 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6d40b3e40 0x7fa6d40b6350 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:42.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:42.412+0000 7fa6f4000640 1 --2- 172.21.15.134:0/3408346068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6ec077cc0 0x7fa6ec10dd90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:42.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:42.412+0000 7fa6f4000640 1 --2- 172.21.15.134:0/3408346068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6ec114170 0x7fa6ec10e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:42.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:42.412+0000 7fa6f4000640 1 --2- 172.21.15.134:0/3408346068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6ec077370 0x7fa6ec113ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:42.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:42.412+0000 7fa6f4000640 1 -- 172.21.15.134:0/3408346068 >> 172.21.15.134:0/3408346068 conn(0x7fa6ec101240 msgr2=0x7fa6ec116210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:42.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:42.412+0000 7fa6f4000640 1 -- 172.21.15.134:0/3408346068 shutdown_connections 2024-07-26T11:18:42.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:42.413+0000 7fa6f4000640 1 -- 172.21.15.134:0/3408346068 wait complete. 2024-07-26T11:18:42.546 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:42 smithi134 ceph-mon[31966]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 207 MiB/s wr, 1.65k op/s 2024-07-26T11:18:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:42 smithi189 ceph-mon[36790]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 207 MiB/s wr, 1.65k op/s 2024-07-26T11:18:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:42 smithi193 ceph-mon[37473]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 207 MiB/s wr, 1.65k op/s 2024-07-26T11:18:43.618 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:43 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/3408346068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:18:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:43 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/3408346068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:18:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:43 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/3408346068' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:18:44.572 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:44 smithi134 ceph-mon[31966]: pgmap v563: 129 pgs: 129 active+clean; 14 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 263 MiB/s wr, 1.99k op/s 2024-07-26T11:18:44.578 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-07-26T11:18:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:44 smithi189 ceph-mon[36790]: pgmap v563: 129 pgs: 129 active+clean; 14 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 263 MiB/s wr, 1.99k op/s 2024-07-26T11:18:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:44 smithi193 ceph-mon[37473]: pgmap v563: 129 pgs: 129 active+clean; 14 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 263 MiB/s wr, 1.99k op/s 2024-07-26T11:18:45.397 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:18:46.274 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:46 smithi134 ceph-mon[31966]: pgmap v564: 129 pgs: 129 active+clean; 14 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 253 MiB/s wr, 1.73k op/s 2024-07-26T11:18:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:46 smithi193 ceph-mon[37473]: pgmap v564: 129 pgs: 129 active+clean; 14 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 253 MiB/s wr, 1.73k op/s 2024-07-26T11:18:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:46 smithi189 ceph-mon[36790]: pgmap v564: 129 pgs: 129 active+clean; 14 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 253 MiB/s wr, 1.73k op/s 2024-07-26T11:18:48.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:48 smithi134 ceph-mon[31966]: pgmap v565: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 228 MiB/s wr, 1.48k op/s 2024-07-26T11:18:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:48 smithi189 ceph-mon[36790]: pgmap v565: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 228 MiB/s wr, 1.48k op/s 2024-07-26T11:18:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:48 smithi193 ceph-mon[37473]: pgmap v565: 129 pgs: 129 active+clean; 14 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 228 MiB/s wr, 1.48k op/s 2024-07-26T11:18:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.392+0000 7fa559a00640 1 -- 172.21.15.134:0/3997655940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa554113c00 msgr2=0x7fa5541160d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:49.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.392+0000 7fa559a00640 1 --2- 172.21.15.134:0/3997655940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa554113c00 0x7fa5541160d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa54c009900 tx=0x7fa54c02a410 comp rx=0 tx=0).stop 2024-07-26T11:18:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.393+0000 7fa559a00640 1 -- 172.21.15.134:0/3997655940 shutdown_connections 2024-07-26T11:18:49.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.393+0000 7fa559a00640 1 --2- 172.21.15.134:0/3997655940 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa554075800 0x7fa554075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.393+0000 7fa559a00640 1 --2- 172.21.15.134:0/3997655940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa554113c00 0x7fa5541160d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.393+0000 7fa559a00640 1 --2- 172.21.15.134:0/3997655940 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa554076dc0 0x7fa5540751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.393+0000 7fa559a00640 1 -- 172.21.15.134:0/3997655940 >> 172.21.15.134:0/3997655940 conn(0x7fa5540fed60 msgr2=0x7fa5541011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.396+0000 7fa559a00640 1 -- 172.21.15.134:0/3997655940 shutdown_connections 2024-07-26T11:18:49.398 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.396+0000 7fa559a00640 1 -- 172.21.15.134:0/3997655940 wait complete. 2024-07-26T11:18:49.399 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.398+0000 7fa559a00640 1 Processor -- start 2024-07-26T11:18:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.398+0000 7fa559a00640 1 -- start start 2024-07-26T11:18:49.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.399+0000 7fa559a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa554075800 0x7fa5541a0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.399+0000 7fa559a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa554076dc0 0x7fa5541a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.399+0000 7fa553e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa554075800 0x7fa5541a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:49.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.399+0000 7fa553e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa554075800 0x7fa5541a0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:39800/0 (socket says 172.21.15.134:39800) 2024-07-26T11:18:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.399+0000 7fa559a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa554113c00 0x7fa5541a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.400+0000 7fa559a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa5541198f0 con 0x7fa554113c00 2024-07-26T11:18:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.400+0000 7fa559a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa554119770 con 0x7fa554075800 2024-07-26T11:18:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.400+0000 7fa559a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa554119a70 con 0x7fa554076dc0 2024-07-26T11:18:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.400+0000 7fa553400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa554076dc0 0x7fa5541a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:49.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.400+0000 7fa553400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa554076dc0 0x7fa5541a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:37854/0 (socket says 172.21.15.134:37854) 2024-07-26T11:18:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.400+0000 7fa553400640 1 -- 172.21.15.134:0/1618929340 learned_addr learned my addr 172.21.15.134:0/1618929340 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.401+0000 7fa553400640 1 -- 172.21.15.134:0/1618929340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa554075800 msgr2=0x7fa5541a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:49.403 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.401+0000 7fa553400640 1 --2- 172.21.15.134:0/1618929340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa554075800 0x7fa5541a0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.401+0000 7fa553400640 1 -- 172.21.15.134:0/1618929340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa554113c00 msgr2=0x7fa5541a5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.401+0000 7fa553400640 1 --2- 172.21.15.134:0/1618929340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa554113c00 0x7fa5541a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.401+0000 7fa553400640 1 -- 172.21.15.134:0/1618929340 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5541a61a0 con 0x7fa554076dc0 2024-07-26T11:18:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.401+0000 7fa553400640 1 --2- 172.21.15.134:0/1618929340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa554076dc0 0x7fa5541a0810 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa54000e9f0 tx=0x7fa54000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.402+0000 7fa550c00640 1 -- 172.21.15.134:0/1618929340 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54000cd80 con 0x7fa554076dc0 2024-07-26T11:18:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.402+0000 7fa550c00640 1 -- 172.21.15.134:0/1618929340 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5400054d0 con 0x7fa554076dc0 2024-07-26T11:18:49.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.402+0000 7fa559a00640 1 -- 172.21.15.134:0/1618929340 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5541a6490 con 0x7fa554076dc0 2024-07-26T11:18:49.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.402+0000 7fa559a00640 1 -- 172.21.15.134:0/1618929340 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5541acc20 con 0x7fa554076dc0 2024-07-26T11:18:49.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.407+0000 7fa550c00640 1 -- 172.21.15.134:0/1618929340 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540010690 con 0x7fa554076dc0 2024-07-26T11:18:49.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.410+0000 7fa550c00640 1 -- 172.21.15.134:0/1618929340 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fa5400108f0 con 0x7fa554076dc0 2024-07-26T11:18:49.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.413+0000 7fa550c00640 1 --2- 172.21.15.134:0/1618929340 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5380b3ca0 0x7fa5380b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.413+0000 7fa56201c740 1 Processor -- start 2024-07-26T11:18:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.413+0000 7fa56201c740 1 -- start start 2024-07-26T11:18:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.414+0000 7fa56201c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a011231b0 0x562a014c6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.414+0000 7fa56201c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a010fa2f0 0x562a014c7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.416 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.414+0000 7fa56201c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011024d0 0x562a014c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.414+0000 7fa550c00640 1 -- 172.21.15.134:0/1618929340 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa5400e47e0 con 0x7fa554076dc0 2024-07-26T11:18:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.415+0000 7fa56201c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562a0143b270 con 0x562a011024d0 2024-07-26T11:18:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.415+0000 7fa56201c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562a0142a770 con 0x562a011231b0 2024-07-26T11:18:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.415+0000 7fa56201c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562a0142e7b0 con 0x562a010fa2f0 2024-07-26T11:18:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.415+0000 7fa553e00640 1 --2- 172.21.15.134:0/1618929340 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5380b3ca0 0x7fa5380b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:49.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.415+0000 7fa559000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011024d0 0x562a014c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.415+0000 7fa553400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a010fa2f0 0x562a014c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.416+0000 7fa553400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a010fa2f0 0x562a014c7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:37860/0 (socket says 172.21.15.134:37860) 2024-07-26T11:18:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.416+0000 7fa553400640 1 -- 172.21.15.134:0/2189126001 learned_addr learned my addr 172.21.15.134:0/2189126001 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:49.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.416+0000 7fa559000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011024d0 0x562a014c76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:52798/0 (socket says 172.21.15.134:52798) 2024-07-26T11:18:49.422 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.416+0000 7fa553400640 1 -- 172.21.15.134:0/2189126001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a011231b0 msgr2=0x562a014c6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.416+0000 7fa553400640 1 --2- 172.21.15.134:0/2189126001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a011231b0 0x562a014c6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.416+0000 7fa553400640 1 -- 172.21.15.134:0/2189126001 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011024d0 msgr2=0x562a014c76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.416+0000 7fa553400640 1 --2- 172.21.15.134:0/2189126001 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011024d0 0x562a014c76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.416+0000 7fa553400640 1 -- 172.21.15.134:0/2189126001 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a0118e790 con 0x562a010fa2f0 2024-07-26T11:18:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.417+0000 7fa553e00640 1 --2- 172.21.15.134:0/1618929340 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5380b3ca0 0x7fa5380b61b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa54400a450 tx=0x7fa54400d040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.417+0000 7fa553400640 1 --2- 172.21.15.134:0/2189126001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a010fa2f0 0x562a014c7160 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa5400a8280 tx=0x7fa5400aa840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=718c256009a4aa77 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.417+0000 7fa549600640 1 -- 172.21.15.134:0/2189126001 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5400c5070 con 0x562a010fa2f0 2024-07-26T11:18:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.417+0000 7fa549600640 1 -- 172.21.15.134:0/2189126001 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5400ab010 con 0x562a010fa2f0 2024-07-26T11:18:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.417+0000 7fa549600640 1 -- 172.21.15.134:0/2189126001 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5400c5070 con 0x562a010fa2f0 2024-07-26T11:18:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.419+0000 7fa56201c740 1 -- 172.21.15.134:0/2189126001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a010fa2f0 msgr2=0x562a014c7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:49.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.419+0000 7fa56201c740 1 --2- 172.21.15.134:0/2189126001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a010fa2f0 0x562a014c7160 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa5400a8280 tx=0x7fa5400aa840 comp rx=0 tx=0).stop 2024-07-26T11:18:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.419+0000 7fa56201c740 1 -- 172.21.15.134:0/2189126001 shutdown_connections 2024-07-26T11:18:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.420+0000 7fa56201c740 1 --2- 172.21.15.134:0/2189126001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a010fa2f0 0x562a014c7160 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.420+0000 7fa56201c740 1 --2- 172.21.15.134:0/2189126001 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011024d0 0x562a014c76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.420+0000 7fa56201c740 1 --2- 172.21.15.134:0/2189126001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a011231b0 0x562a014c6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.420+0000 7fa56201c740 1 -- 172.21.15.134:0/2189126001 >> 172.21.15.134:0/2189126001 conn(0x562a01117300 msgr2=0x562a014c4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.420+0000 7fa56201c740 1 -- 172.21.15.134:0/2189126001 shutdown_connections 2024-07-26T11:18:49.424 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.421+0000 7fa56201c740 1 -- 172.21.15.134:0/2189126001 wait complete. 2024-07-26T11:18:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.422+0000 7fa56201c740 1 -- start start 2024-07-26T11:18:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.422+0000 7fa56201c740 1 Processor -- start 2024-07-26T11:18:49.425 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.424+0000 7fa56201c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a010fa2f0 0x562a014d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.426 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.424+0000 7fa56201c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011024d0 0x562a014d2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.425+0000 7fa56201c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a011231b0 0x562a014de6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.425+0000 7fa56201c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562a01364ab0 con 0x562a011024d0 2024-07-26T11:18:49.427 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.425+0000 7fa56201c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562a013c7d50 con 0x562a010fa2f0 2024-07-26T11:18:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.425+0000 7fa56201c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562a014d53a0 con 0x562a011231b0 2024-07-26T11:18:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.425+0000 7fa559000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a011231b0 0x562a014de6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.425+0000 7fa559000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a011231b0 0x562a014de6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:37874/0 (socket says 172.21.15.134:37874) 2024-07-26T11:18:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.425+0000 7fa559000640 1 -- 172.21.15.134:0/3865296592 learned_addr learned my addr 172.21.15.134:0/3865296592 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:49.428 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.425+0000 7fa553e00640 1 --2- 172.21.15.134:0/3865296592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a010fa2f0 0x562a014d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.426+0000 7fa56201c740 10 client.0 _ll_register_callbacks cb 0x562a0113aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.426+0000 7fa553400640 1 --2- 172.21.15.134:0/3865296592 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011024d0 0x562a014d2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.426+0000 7fa559000640 1 -- 172.21.15.134:0/3865296592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a010fa2f0 msgr2=0x562a014d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.426+0000 7fa559000640 1 --2- 172.21.15.134:0/3865296592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a010fa2f0 0x562a014d2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.426+0000 7fa559000640 1 -- 172.21.15.134:0/3865296592 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011024d0 msgr2=0x562a014d2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.426+0000 7fa559000640 1 --2- 172.21.15.134:0/3865296592 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011024d0 0x562a014d2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.427+0000 7fa559000640 1 -- 172.21.15.134:0/3865296592 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a01207620 con 0x562a011231b0 2024-07-26T11:18:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.427+0000 7fa53e000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:18:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.427+0000 7fa559000640 1 --2- 172.21.15.134:0/3865296592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a011231b0 0x562a014de6f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa54c003820 tx=0x7fa54c003850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ab689f4015e165c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:49.430 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.428+0000 7fa53e000640 1 -- 172.21.15.134:0/3865296592 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54c040030 con 0x562a011231b0 2024-07-26T11:18:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.428+0000 7fa53e000640 1 -- 172.21.15.134:0/3865296592 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa54c041070 con 0x562a011231b0 2024-07-26T11:18:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.429+0000 7fa56201c740 1 -- 172.21.15.134:0/3865296592 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5400ab010 con 0x562a011231b0 2024-07-26T11:18:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.429+0000 7fa53e000640 1 -- 172.21.15.134:0/3865296592 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa54c04a050 con 0x562a011231b0 2024-07-26T11:18:49.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.430+0000 7fa56201c740 10 client.25987 fetch_fsmap learned FSMap version 31 2024-07-26T11:18:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.430+0000 7fa56201c740 1 -- 172.21.15.134:0/3865296592 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562a014e1370 con 0x562a011231b0 2024-07-26T11:18:49.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.430+0000 7fa53e000640 1 -- 172.21.15.134:0/3865296592 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5400ab010 con 0x562a011231b0 2024-07-26T11:18:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.431+0000 7fa53e000640 1 -- 172.21.15.134:0/3865296592 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7fa54c03a040 con 0x562a011231b0 2024-07-26T11:18:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.431+0000 7fa56201c740 10 client.25987 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:18:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.431+0000 7fa56201c740 10 client.25987 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:18:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.431+0000 7fa56201c740 20 client.25987 populate_metadata read hostname 'smithi134' 2024-07-26T11:18:49.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.431+0000 7fa56201c740 1 --2- 172.21.15.134:0/3865296592 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562a010e1d80 0x562a014e4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.434 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.431+0000 7fa56201c740 4 client.25987 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:18:49.434 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.432+0000 7fa56201c740 1 -- 172.21.15.134:0/3865296592 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a0143d1c0 con 0x562a010e1d80 2024-07-26T11:18:49.434 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.432+0000 7fa553e00640 1 --2- 172.21.15.134:0/3865296592 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562a010e1d80 0x562a014e4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:49.435 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.434+0000 7fa553e00640 1 --2- 172.21.15.134:0/3865296592 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562a010e1d80 0x562a014e4a20 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a625854d3a1123a server_cookie=b80f9c46317e09df in_seq=0 out_seq=0 2024-07-26T11:18:49.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.434+0000 7fa53e000640 10 client.25987 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.435+0000 7fa53e000640 1 -- 172.21.15.134:0/3865296592 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562a0143d1c0 con 0x562a010e1d80 2024-07-26T11:18:49.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.435+0000 7fa53e000640 10 client.25987 handle_command_reply: tid=0 2024-07-26T11:18:49.438 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.435+0000 7fa56201c740 1 client.25987 shutdown 2024-07-26T11:18:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.446+0000 7fa56201c740 1 -- 172.21.15.134:0/3865296592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a011231b0 msgr2=0x562a014de6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:49.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.446+0000 7fa56201c740 1 --2- 172.21.15.134:0/3865296592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a011231b0 0x562a014de6f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa54c003820 tx=0x7fa54c003850 comp rx=0 tx=0).stop 2024-07-26T11:18:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.447+0000 7fa56201c740 1 -- 172.21.15.134:0/3865296592 shutdown_connections 2024-07-26T11:18:49.451 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.447+0000 7fa56201c740 1 --2- 172.21.15.134:0/3865296592 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562a010e1d80 0x562a014e4a20 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.447+0000 7fa56201c740 1 --2- 172.21.15.134:0/3865296592 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a011231b0 0x562a014de6f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.447+0000 7fa56201c740 1 --2- 172.21.15.134:0/3865296592 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011024d0 0x562a014d2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.447+0000 7fa56201c740 1 --2- 172.21.15.134:0/3865296592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a010fa2f0 0x562a014d2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.447+0000 7fa56201c740 1 -- 172.21.15.134:0/3865296592 >> 172.21.15.134:0/3865296592 conn(0x562a01117300 msgr2=0x562a014c7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.447+0000 7fa53e000640 0 client.25987 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.449+0000 7fa56201c740 1 -- 172.21.15.134:0/3865296592 shutdown_connections 2024-07-26T11:18:49.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.450+0000 7fa56201c740 1 -- 172.21.15.134:0/3865296592 wait complete. 2024-07-26T11:18:49.454 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.450+0000 7fa56201c740 20 client.25987 trim_cache size 0 max 16384 2024-07-26T11:18:49.506 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.504+0000 7fa56201c740 1 Processor -- start 2024-07-26T11:18:49.509 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.507+0000 7fa56201c740 1 -- start start 2024-07-26T11:18:49.509 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.507+0000 7fa56201c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a010fa2f0 0x562a014c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.509 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.507+0000 7fa56201c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a011024d0 0x562a014d4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.508+0000 7fa553400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a011024d0 0x562a014d4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:49.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.508+0000 7fa553400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a011024d0 0x562a014d4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:37878/0 (socket says 172.21.15.134:37878) 2024-07-26T11:18:49.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.508+0000 7fa56201c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011231b0 0x562a014d1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.508+0000 7fa553400640 1 -- 172.21.15.134:0/3244220577 learned_addr learned my addr 172.21.15.134:0/3244220577 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.508+0000 7fa553e00640 1 --2- 172.21.15.134:0/3244220577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a010fa2f0 0x562a014c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.509+0000 7fa559000640 1 --2- 172.21.15.134:0/3244220577 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011231b0 0x562a014d1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.509+0000 7fa56201c740 1 -- 172.21.15.134:0/3244220577 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa53400ffb0 con 0x562a011231b0 2024-07-26T11:18:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.510+0000 7fa56201c740 1 -- 172.21.15.134:0/3244220577 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa534011b20 con 0x562a010fa2f0 2024-07-26T11:18:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.510+0000 7fa56201c740 1 -- 172.21.15.134:0/3244220577 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa534010680 con 0x562a011024d0 2024-07-26T11:18:49.514 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.512+0000 7fa553e00640 1 -- 172.21.15.134:0/3244220577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a011024d0 msgr2=0x562a014d4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.513+0000 7fa553e00640 1 --2- 172.21.15.134:0/3244220577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a011024d0 0x562a014d4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.513+0000 7fa553e00640 1 -- 172.21.15.134:0/3244220577 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011231b0 msgr2=0x562a014d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.513+0000 7fa553e00640 1 --2- 172.21.15.134:0/3244220577 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011231b0 0x562a014d1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.513+0000 7fa553e00640 1 -- 172.21.15.134:0/3244220577 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a014d09f0 con 0x562a010fa2f0 2024-07-26T11:18:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.514+0000 7fa559000640 1 --2- 172.21.15.134:0/3244220577 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011231b0 0x562a014d1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:18:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.514+0000 7fa553400640 1 --2- 172.21.15.134:0/3244220577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a011024d0 0x562a014d4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:18:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.515+0000 7fa553e00640 1 --2- 172.21.15.134:0/3244220577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a010fa2f0 0x562a014c3a70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x562a014e5180 tx=0x7fa544017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bc4496ff851e882 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:49.520 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.516+0000 7fa548c00640 1 -- 172.21.15.134:0/3244220577 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa544018b30 con 0x562a010fa2f0 2024-07-26T11:18:49.520 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.516+0000 7fa548c00640 1 -- 172.21.15.134:0/3244220577 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa544018cd0 con 0x562a010fa2f0 2024-07-26T11:18:49.520 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.516+0000 7fa548c00640 1 -- 172.21.15.134:0/3244220577 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5440236c0 con 0x562a010fa2f0 2024-07-26T11:18:49.521 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.518+0000 7fa56201c740 1 -- 172.21.15.134:0/3244220577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a010fa2f0 msgr2=0x562a014c3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:49.521 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.518+0000 7fa56201c740 1 --2- 172.21.15.134:0/3244220577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a010fa2f0 0x562a014c3a70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x562a014e5180 tx=0x7fa544017000 comp rx=0 tx=0).stop 2024-07-26T11:18:49.521 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.519+0000 7fa56201c740 1 -- 172.21.15.134:0/3244220577 shutdown_connections 2024-07-26T11:18:49.521 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.519+0000 7fa56201c740 1 --2- 172.21.15.134:0/3244220577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a011024d0 0x562a014d4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.524 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.519+0000 7fa56201c740 1 --2- 172.21.15.134:0/3244220577 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011231b0 0x562a014d1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.524 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.519+0000 7fa56201c740 1 --2- 172.21.15.134:0/3244220577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a010fa2f0 0x562a014c3a70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.524 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.519+0000 7fa56201c740 1 -- 172.21.15.134:0/3244220577 >> 172.21.15.134:0/3244220577 conn(0x562a01117300 msgr2=0x562a014d3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.523+0000 7fa56201c740 1 -- 172.21.15.134:0/3244220577 shutdown_connections 2024-07-26T11:18:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.523+0000 7fa56201c740 1 -- 172.21.15.134:0/3244220577 wait complete. 2024-07-26T11:18:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.524+0000 7fa56201c740 1 -- start start 2024-07-26T11:18:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.524+0000 7fa56201c740 1 Processor -- start 2024-07-26T11:18:49.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.526+0000 7fa56201c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a010fa2f0 0x562a014d5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.529+0000 7fa56201c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a011024d0 0x562a014d61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.529+0000 7fa56201c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011231b0 0x562a0154b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.529+0000 7fa553400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a011024d0 0x562a014d61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.529+0000 7fa553400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a011024d0 0x562a014d61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:39838/0 (socket says 172.21.15.134:39838) 2024-07-26T11:18:49.534 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.532+0000 7fa553e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a010fa2f0 0x562a014d5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.532+0000 7fa553400640 1 -- 172.21.15.134:0/2242366049 learned_addr learned my addr 172.21.15.134:0/2242366049 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.532+0000 7fa56201c740 1 -- 172.21.15.134:0/2242366049 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa534011420 con 0x562a011231b0 2024-07-26T11:18:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.532+0000 7fa56201c740 1 -- 172.21.15.134:0/2242366049 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa534010d50 con 0x562a011024d0 2024-07-26T11:18:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.532+0000 7fa56201c740 1 -- 172.21.15.134:0/2242366049 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa53400df90 con 0x562a010fa2f0 2024-07-26T11:18:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.533+0000 7fa553400640 1 -- 172.21.15.134:0/2242366049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a010fa2f0 msgr2=0x562a014d5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.533+0000 7fa553400640 1 --2- 172.21.15.134:0/2242366049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a010fa2f0 0x562a014d5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.533+0000 7fa553400640 1 -- 172.21.15.134:0/2242366049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011231b0 msgr2=0x562a0154b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.533+0000 7fa553400640 1 --2- 172.21.15.134:0/2242366049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011231b0 0x562a0154b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:49.536 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.533+0000 7fa553400640 1 -- 172.21.15.134:0/2242366049 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a0154b6e0 con 0x562a011024d0 2024-07-26T11:18:49.536 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.533+0000 7fa553400640 1 --2- 172.21.15.134:0/2242366049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a011024d0 0x562a014d61a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa5400a6f40 tx=0x7fa5400e45e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d32c415bb040200c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:49.536 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.533+0000 7fa53e000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:18:49.536 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.533+0000 7fa56201c740 10 client.0 _ll_register_callbacks cb 0x7fa540005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:49.537 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.533+0000 7fa56201c740 1 -- 172.21.15.134:0/2242366049 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa544018cd0 con 0x562a011024d0 2024-07-26T11:18:49.537 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.534+0000 7fa53e000640 1 -- 172.21.15.134:0/2242366049 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5400cc030 con 0x562a011024d0 2024-07-26T11:18:49.537 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.534+0000 7fa53e000640 1 -- 172.21.15.134:0/2242366049 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa540010df0 con 0x562a011024d0 2024-07-26T11:18:49.537 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.534+0000 7fa53e000640 1 -- 172.21.15.134:0/2242366049 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa5400c5070 con 0x562a011024d0 2024-07-26T11:18:49.537 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.534+0000 7fa56201c740 10 client.25997 fetch_fsmap learned FSMap version 31 2024-07-26T11:18:49.537 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.534+0000 7fa56201c740 1 -- 172.21.15.134:0/2242366049 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=31}) -- 0x562a0154c830 con 0x562a011024d0 2024-07-26T11:18:49.538 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.536+0000 7fa53e000640 1 -- 172.21.15.134:0/2242366049 <== mon.2 v2:172.21.15.189:3300/0 4 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7fa5400c0200 con 0x562a011024d0 2024-07-26T11:18:49.538 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.536+0000 7fa56201c740 10 client.25997 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:18:49.538 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.536+0000 7fa56201c740 10 client.25997 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:18:49.538 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.536+0000 7fa56201c740 20 client.25997 populate_metadata read hostname 'smithi134' 2024-07-26T11:18:49.538 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.536+0000 7fa56201c740 1 --2- 172.21.15.134:0/2242366049 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562a010b71c0 0x562a0154eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:49.539 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.536+0000 7fa56201c740 4 client.25997 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:18:49.539 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.536+0000 7fa56201c740 1 -- 172.21.15.134:0/2242366049 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562a01438c60 con 0x562a010b71c0 2024-07-26T11:18:49.539 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.538+0000 7fa553e00640 1 --2- 172.21.15.134:0/2242366049 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562a010b71c0 0x562a0154eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:49.541 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.539+0000 7fa553e00640 1 --2- 172.21.15.134:0/2242366049 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562a010b71c0 0x562a0154eb20 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f20658fb3567d9e server_cookie=b6f8e832c2c8009e in_seq=0 out_seq=0 2024-07-26T11:18:49.541 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:49.539+0000 7fa53e000640 10 client.25997 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:50.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.160+0000 7fa53e000640 1 -- 172.21.15.134:0/2242366049 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562a01438c60 con 0x562a010b71c0 2024-07-26T11:18:50.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.161+0000 7fa53e000640 10 client.25997 handle_command_reply: tid=0 2024-07-26T11:18:50.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.166+0000 7fa56201c740 1 client.25997 shutdown 2024-07-26T11:18:50.174 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.171+0000 7fa56201c740 1 -- 172.21.15.134:0/2242366049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a011024d0 msgr2=0x562a014d61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:50.174 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.171+0000 7fa56201c740 1 --2- 172.21.15.134:0/2242366049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a011024d0 0x562a014d61a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa5400a6f40 tx=0x7fa5400e45e0 comp rx=0 tx=0).stop 2024-07-26T11:18:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.172+0000 7fa56201c740 1 -- 172.21.15.134:0/2242366049 shutdown_connections 2024-07-26T11:18:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.172+0000 7fa56201c740 1 --2- 172.21.15.134:0/2242366049 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562a010b71c0 0x562a0154eb20 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.172+0000 7fa56201c740 1 --2- 172.21.15.134:0/2242366049 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562a011231b0 0x562a0154b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.172+0000 7fa56201c740 1 --2- 172.21.15.134:0/2242366049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a011024d0 0x562a014d61a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.172+0000 7fa56201c740 1 --2- 172.21.15.134:0/2242366049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562a010fa2f0 0x562a014d5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.172+0000 7fa56201c740 1 -- 172.21.15.134:0/2242366049 >> 172.21.15.134:0/2242366049 conn(0x562a01117300 msgr2=0x562a014c3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:50.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.174+0000 7fa53e000640 0 client.25997 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:18:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.174+0000 7fa56201c740 1 -- 172.21.15.134:0/2242366049 shutdown_connections 2024-07-26T11:18:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.174+0000 7fa56201c740 1 -- 172.21.15.134:0/2242366049 wait complete. 2024-07-26T11:18:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.175+0000 7fa56201c740 20 client.25997 trim_cache size 0 max 16384 2024-07-26T11:18:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:18:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:18:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:18:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:18:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.188+0000 7fa559a00640 1 -- 172.21.15.134:0/1618929340 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5380b3ca0 msgr2=0x7fa5380b61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.188+0000 7fa559a00640 1 --2- 172.21.15.134:0/1618929340 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5380b3ca0 0x7fa5380b61b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa54400a450 tx=0x7fa54400d040 comp rx=0 tx=0).stop 2024-07-26T11:18:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.190+0000 7fa559a00640 1 -- 172.21.15.134:0/1618929340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa554076dc0 msgr2=0x7fa5541a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.190+0000 7fa559a00640 1 --2- 172.21.15.134:0/1618929340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa554076dc0 0x7fa5541a0810 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa54000e9f0 tx=0x7fa54000edf0 comp rx=0 tx=0).stop 2024-07-26T11:18:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.194+0000 7fa559a00640 1 -- 172.21.15.134:0/1618929340 shutdown_connections 2024-07-26T11:18:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.194+0000 7fa559a00640 1 --2- 172.21.15.134:0/1618929340 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5380b3ca0 0x7fa5380b61b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.195+0000 7fa559a00640 1 --2- 172.21.15.134:0/1618929340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa554076dc0 0x7fa5541a0810 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.195+0000 7fa559a00640 1 --2- 172.21.15.134:0/1618929340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa554113c00 0x7fa5541a5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.195+0000 7fa559a00640 1 --2- 172.21.15.134:0/1618929340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa554075800 0x7fa5541a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.195+0000 7fa559a00640 1 -- 172.21.15.134:0/1618929340 >> 172.21.15.134:0/1618929340 conn(0x7fa5540fed60 msgr2=0x7fa55410fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.195+0000 7fa559a00640 1 -- 172.21.15.134:0/1618929340 shutdown_connections 2024-07-26T11:18:50.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:18:50.195+0000 7fa559a00640 1 -- 172.21.15.134:0/1618929340 wait complete. 2024-07-26T11:18:50.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:50 smithi134 ceph-mon[31966]: pgmap v566: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 304 MiB/s wr, 1.86k op/s 2024-07-26T11:18:50.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:50 smithi189 ceph-mon[36790]: pgmap v566: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 304 MiB/s wr, 1.86k op/s 2024-07-26T11:18:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:50 smithi193 ceph-mon[37473]: pgmap v566: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 304 MiB/s wr, 1.86k op/s 2024-07-26T11:18:51.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.243+0000 7f538b200640 1 -- 172.21.15.134:0/1836801189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f538c108d20 msgr2=0x7f538c109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:51.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.243+0000 7f538b200640 1 --2- 172.21.15.134:0/1836801189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f538c108d20 0x7f538c109130 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5378009900 tx=0x7f537802a410 comp rx=0 tx=0).stop 2024-07-26T11:18:51.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.243+0000 7f538b200640 1 -- 172.21.15.134:0/1836801189 shutdown_connections 2024-07-26T11:18:51.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.243+0000 7f538b200640 1 --2- 172.21.15.134:0/1836801189 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f538c108d20 0x7f538c109130 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.243+0000 7f538b200640 1 --2- 172.21.15.134:0/1836801189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f538c109670 0x7f538c10fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.243+0000 7f538b200640 1 --2- 172.21.15.134:0/1836801189 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f538c1033e0 0x7f538c1037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.243+0000 7f538b200640 1 -- 172.21.15.134:0/1836801189 >> 172.21.15.134:0/1836801189 conn(0x7f538c0fee70 msgr2=0x7f538c1012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:51.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.243+0000 7f538b200640 1 -- 172.21.15.134:0/1836801189 shutdown_connections 2024-07-26T11:18:51.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.243+0000 7f538b200640 1 -- 172.21.15.134:0/1836801189 wait complete. 2024-07-26T11:18:51.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.245+0000 7f538b200640 1 Processor -- start 2024-07-26T11:18:51.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.245+0000 7f538b200640 1 -- start start 2024-07-26T11:18:51.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.245+0000 7f538b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f538c1033e0 0x7f538c1a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.248 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.245+0000 7f538b200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f538c108d20 0x7f538c1a1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.248 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.245+0000 7f538b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f538c109670 0x7f538c1ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.248 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.246+0000 7f538b200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f538c113350 con 0x7f538c108d20 2024-07-26T11:18:51.248 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.246+0000 7f538b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f538c1131d0 con 0x7f538c109670 2024-07-26T11:18:51.248 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.246+0000 7f538b200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f538c1134d0 con 0x7f538c1033e0 2024-07-26T11:18:51.248 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.246+0000 7f5389e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f538c1033e0 0x7f538c1a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.248 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.246+0000 7f5389400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f538c108d20 0x7f538c1a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.249 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.246+0000 7f5389e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f538c1033e0 0x7f538c1a1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:48578/0 (socket says 172.21.15.134:48578) 2024-07-26T11:18:51.249 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.246+0000 7f5389400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f538c108d20 0x7f538c1a1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:37348/0 (socket says 172.21.15.134:37348) 2024-07-26T11:18:51.249 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.246+0000 7f5389e00640 1 -- 172.21.15.134:0/4055491287 learned_addr learned my addr 172.21.15.134:0/4055491287 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:51.249 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.246+0000 7f538a800640 1 --2- 172.21.15.134:0/4055491287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f538c109670 0x7f538c1ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.250 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.247+0000 7f5389e00640 1 -- 172.21.15.134:0/4055491287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f538c109670 msgr2=0x7f538c1ac0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:51.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.247+0000 7f5389e00640 1 --2- 172.21.15.134:0/4055491287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f538c109670 0x7f538c1ac0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.247+0000 7f5389e00640 1 -- 172.21.15.134:0/4055491287 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f538c108d20 msgr2=0x7f538c1a1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:51.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.247+0000 7f5389e00640 1 --2- 172.21.15.134:0/4055491287 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f538c108d20 0x7f538c1a1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.247+0000 7f5389e00640 1 -- 172.21.15.134:0/4055491287 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f538c1a1e40 con 0x7f538c1033e0 2024-07-26T11:18:51.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.247+0000 7f5389e00640 1 --2- 172.21.15.134:0/4055491287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f538c1033e0 0x7f538c1a1030 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f537c00e9f0 tx=0x7f537c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:51.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.248+0000 7f5382a00640 1 -- 172.21.15.134:0/4055491287 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c00cd80 con 0x7f538c1033e0 2024-07-26T11:18:51.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.248+0000 7f5382a00640 1 -- 172.21.15.134:0/4055491287 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f537c0108e0 con 0x7f538c1033e0 2024-07-26T11:18:51.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.248+0000 7f5382a00640 1 -- 172.21.15.134:0/4055491287 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c018980 con 0x7f538c1033e0 2024-07-26T11:18:51.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.248+0000 7f538b200640 1 -- 172.21.15.134:0/4055491287 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f538c1ac710 con 0x7f538c1033e0 2024-07-26T11:18:51.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.248+0000 7f538b200640 1 -- 172.21.15.134:0/4055491287 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f538c1acc70 con 0x7f538c1033e0 2024-07-26T11:18:51.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.257+0000 7f5382a00640 1 -- 172.21.15.134:0/4055491287 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f537c010430 con 0x7f538c1033e0 2024-07-26T11:18:51.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.257+0000 7f53977fc740 1 Processor -- start 2024-07-26T11:18:51.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.258+0000 7f53977fc740 1 -- start start 2024-07-26T11:18:51.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.258+0000 7f53977fc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556510353070 0x5565106f8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.258+0000 7f53977fc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55651032a1b0 0x5565106f8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.258+0000 7f53977fc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 0x5565106f8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.259+0000 7f53977fc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55651066ebd0 con 0x556510332390 2024-07-26T11:18:51.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.259+0000 7f53977fc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55651065e0d0 con 0x55651032a1b0 2024-07-26T11:18:51.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.259+0000 7f53977fc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556510662110 con 0x556510353070 2024-07-26T11:18:51.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.259+0000 7f5389e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55651032a1b0 0x5565106f8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.259+0000 7f5389400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 0x5565106f8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.259+0000 7f5389400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 0x5565106f8a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:37362/0 (socket says 172.21.15.134:37362) 2024-07-26T11:18:51.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.259+0000 7f5389400640 1 -- 172.21.15.134:0/4226861567 learned_addr learned my addr 172.21.15.134:0/4226861567 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:51.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.260+0000 7f5389400640 1 -- 172.21.15.134:0/4226861567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556510353070 msgr2=0x5565106f8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:51.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.260+0000 7f538a800640 1 --2- 172.21.15.134:0/4226861567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556510353070 0x5565106f8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.260+0000 7f5389400640 1 --2- 172.21.15.134:0/4226861567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556510353070 0x5565106f8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.260+0000 7f5389400640 1 -- 172.21.15.134:0/4226861567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55651032a1b0 msgr2=0x5565106f8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:51.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.261+0000 7f5382a00640 1 --2- 172.21.15.134:0/4055491287 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f53700b3e40 0x7f53700b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.261+0000 7f5389400640 1 --2- 172.21.15.134:0/4226861567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55651032a1b0 0x5565106f8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.261+0000 7f5389400640 1 -- 172.21.15.134:0/4226861567 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565103be650 con 0x556510332390 2024-07-26T11:18:51.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.262+0000 7f5382a00640 1 -- 172.21.15.134:0/4055491287 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f537c0e4670 con 0x7f538c1033e0 2024-07-26T11:18:51.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.262+0000 7f5389e00640 1 --2- 172.21.15.134:0/4226861567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55651032a1b0 0x5565106f8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:18:51.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.262+0000 7f538a800640 1 --2- 172.21.15.134:0/4226861567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556510353070 0x5565106f8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:18:51.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.263+0000 7f5389400640 1 --2- 172.21.15.134:0/4226861567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 0x5565106f8a90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5378005ab0 tx=0x7f5378004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eae658c3c60b1a6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:51.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.263+0000 7f538a800640 1 --2- 172.21.15.134:0/4055491287 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f53700b3e40 0x7f53700b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.263+0000 7f5377000640 1 -- 172.21.15.134:0/4226861567 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5378035040 con 0x556510332390 2024-07-26T11:18:51.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.264+0000 7f538a800640 1 --2- 172.21.15.134:0/4055491287 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f53700b3e40 0x7f53700b6350 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5384005fd0 tx=0x7f5384009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:51.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.264+0000 7f5377000640 1 -- 172.21.15.134:0/4226861567 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53780054f0 con 0x556510332390 2024-07-26T11:18:51.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.264+0000 7f5377000640 1 -- 172.21.15.134:0/4226861567 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53780035f0 con 0x556510332390 2024-07-26T11:18:51.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.266+0000 7f53977fc740 1 -- 172.21.15.134:0/4226861567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 msgr2=0x5565106f8a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:51.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.266+0000 7f53977fc740 1 --2- 172.21.15.134:0/4226861567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 0x5565106f8a90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5378005ab0 tx=0x7f5378004040 comp rx=0 tx=0).stop 2024-07-26T11:18:51.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.266+0000 7f53977fc740 1 -- 172.21.15.134:0/4226861567 shutdown_connections 2024-07-26T11:18:51.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.266+0000 7f53977fc740 1 --2- 172.21.15.134:0/4226861567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 0x5565106f8a90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.266+0000 7f53977fc740 1 --2- 172.21.15.134:0/4226861567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55651032a1b0 0x5565106f8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.266+0000 7f53977fc740 1 --2- 172.21.15.134:0/4226861567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556510353070 0x5565106f8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.266+0000 7f53977fc740 1 -- 172.21.15.134:0/4226861567 >> 172.21.15.134:0/4226861567 conn(0x556510347460 msgr2=0x5565106f5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:51.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.268+0000 7f53977fc740 1 -- 172.21.15.134:0/4226861567 shutdown_connections 2024-07-26T11:18:51.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.268+0000 7f53977fc740 1 -- 172.21.15.134:0/4226861567 wait complete. 2024-07-26T11:18:51.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.269+0000 7f53977fc740 1 -- start start 2024-07-26T11:18:51.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.269+0000 7f53977fc740 1 Processor -- start 2024-07-26T11:18:51.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.270+0000 7f53977fc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55651032a1b0 0x556510703660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.270+0000 7f53977fc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 0x556510703ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.270+0000 7f53977fc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 0x55651070fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.271+0000 7f53977fc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5565104c1700 con 0x556510332390 2024-07-26T11:18:51.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.271+0000 7f53977fc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5565105fa6a0 con 0x556510353070 2024-07-26T11:18:51.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.271+0000 7f53977fc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556510706790 con 0x55651032a1b0 2024-07-26T11:18:51.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.271+0000 7f5389e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55651032a1b0 0x556510703660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.271+0000 7f5389e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55651032a1b0 0x556510703660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:48600/0 (socket says 172.21.15.134:48600) 2024-07-26T11:18:51.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.271+0000 7f5389e00640 1 -- 172.21.15.134:0/396031931 learned_addr learned my addr 172.21.15.134:0/396031931 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:51.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.271+0000 7f538a800640 1 --2- 172.21.15.134:0/396031931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 0x55651070fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.271+0000 7f5389400640 1 --2- 172.21.15.134:0/396031931 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 0x556510703ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.272+0000 7f5389e00640 1 -- 172.21.15.134:0/396031931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 msgr2=0x55651070fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:51.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.272+0000 7f5389e00640 1 --2- 172.21.15.134:0/396031931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 0x55651070fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.272+0000 7f5389e00640 1 -- 172.21.15.134:0/396031931 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 msgr2=0x556510703ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:51.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.272+0000 7f5389e00640 1 --2- 172.21.15.134:0/396031931 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 0x556510703ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.272+0000 7f5389e00640 1 -- 172.21.15.134:0/396031931 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556510611840 con 0x55651032a1b0 2024-07-26T11:18:51.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.272+0000 7f5389400640 1 --2- 172.21.15.134:0/396031931 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 0x556510703ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:18:51.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.273+0000 7f5389e00640 1 --2- 172.21.15.134:0/396031931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55651032a1b0 0x556510703660 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f537c00e9c0 tx=0x7f537c011fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41de9ae7f39e8794 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:51.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.273+0000 7f536b400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:18:51.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.273+0000 7f53977fc740 10 client.0 _ll_register_callbacks cb 0x55651060a150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:51.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.273+0000 7f536b400640 1 -- 172.21.15.134:0/396031931 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c0af4a0 con 0x55651032a1b0 2024-07-26T11:18:51.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.273+0000 7f536b400640 1 -- 172.21.15.134:0/396031931 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f537c0a7540 con 0x55651032a1b0 2024-07-26T11:18:51.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.273+0000 7f53977fc740 1 -- 172.21.15.134:0/396031931 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53780054f0 con 0x55651032a1b0 2024-07-26T11:18:51.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.273+0000 7f536b400640 1 -- 172.21.15.134:0/396031931 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f537c0b3f50 con 0x55651032a1b0 2024-07-26T11:18:51.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.277+0000 7f53977fc740 10 client.25999 fetch_fsmap learned FSMap version 31 2024-07-26T11:18:51.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.277+0000 7f53977fc740 1 -- 172.21.15.134:0/396031931 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=31}) -- 0x5565107121c0 con 0x55651032a1b0 2024-07-26T11:18:51.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.277+0000 7f536b400640 1 -- 172.21.15.134:0/396031931 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f537c0e4230 con 0x55651032a1b0 2024-07-26T11:18:51.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.277+0000 7f53977fc740 10 client.25999 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:18:51.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.278+0000 7f53977fc740 10 client.25999 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-07-26T11:18:51.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.278+0000 7f53977fc740 20 client.25999 populate_metadata read hostname 'smithi134' 2024-07-26T11:18:51.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.278+0000 7f53977fc740 1 --2- 172.21.15.134:0/396031931 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5565103119f0 0x556510715680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.278+0000 7f53977fc740 4 client.25999 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:18:51.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.278+0000 7f53977fc740 1 -- 172.21.15.134:0/396031931 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f53780054f0 con 0x5565103119f0 2024-07-26T11:18:51.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.278+0000 7f5389400640 1 --2- 172.21.15.134:0/396031931 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5565103119f0 0x556510715680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.279+0000 7f5389400640 1 --2- 172.21.15.134:0/396031931 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5565103119f0 0x556510715680 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d91f9906ee74403a server_cookie=7503940aa7cd74c in_seq=0 out_seq=0 2024-07-26T11:18:51.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.279+0000 7f536b400640 10 client.25999 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:18:51.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.280+0000 7f536b400640 1 -- 172.21.15.134:0/396031931 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f53780054f0 con 0x5565103119f0 2024-07-26T11:18:51.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.280+0000 7f536b400640 10 client.25999 handle_command_reply: tid=0 2024-07-26T11:18:51.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.280+0000 7f53977fc740 1 client.25999 shutdown 2024-07-26T11:18:51.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.281+0000 7f53977fc740 1 -- 172.21.15.134:0/396031931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55651032a1b0 msgr2=0x556510703660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:51.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.281+0000 7f53977fc740 1 --2- 172.21.15.134:0/396031931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55651032a1b0 0x556510703660 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f537c00e9c0 tx=0x7f537c011fe0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.281+0000 7f53977fc740 1 -- 172.21.15.134:0/396031931 shutdown_connections 2024-07-26T11:18:51.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.281+0000 7f53977fc740 1 --2- 172.21.15.134:0/396031931 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5565103119f0 0x556510715680 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.281+0000 7f53977fc740 1 --2- 172.21.15.134:0/396031931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 0x55651070fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.281+0000 7f53977fc740 1 --2- 172.21.15.134:0/396031931 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 0x556510703ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.281+0000 7f53977fc740 1 --2- 172.21.15.134:0/396031931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55651032a1b0 0x556510703660 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.281+0000 7f53977fc740 1 -- 172.21.15.134:0/396031931 >> 172.21.15.134:0/396031931 conn(0x556510347460 msgr2=0x5565106fb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:51.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.281+0000 7f536b400640 0 client.25999 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:18:51.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.285+0000 7f53977fc740 1 -- 172.21.15.134:0/396031931 shutdown_connections 2024-07-26T11:18:51.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.286+0000 7f53977fc740 1 -- 172.21.15.134:0/396031931 wait complete. 2024-07-26T11:18:51.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.287+0000 7f53977fc740 20 client.25999 trim_cache size 0 max 16384 2024-07-26T11:18:51.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.339+0000 7f53977fc740 1 Processor -- start 2024-07-26T11:18:51.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.340+0000 7f53977fc740 1 -- start start 2024-07-26T11:18:51.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.340+0000 7f53977fc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55651032a1b0 0x556510705290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.340+0000 7f53977fc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556510332390 0x55651070f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.340+0000 7f53977fc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 0x55651071ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.340+0000 7f53977fc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f538c1b6cc0 con 0x55651032a1b0 2024-07-26T11:18:51.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.340+0000 7f53977fc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f538c1b8830 con 0x556510353070 2024-07-26T11:18:51.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.340+0000 7f53977fc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f538c1b7390 con 0x556510332390 2024-07-26T11:18:51.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.341+0000 7f538a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 0x55651071ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.341+0000 7f538a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 0x55651071ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:55838/0 (socket says 172.21.15.134:55838) 2024-07-26T11:18:51.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.341+0000 7f538a800640 1 -- 172.21.15.134:0/2593247807 learned_addr learned my addr 172.21.15.134:0/2593247807 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:51.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.341+0000 7f5389400640 1 --2- 172.21.15.134:0/2593247807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556510332390 0x55651070f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.341+0000 7f5389e00640 1 --2- 172.21.15.134:0/2593247807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55651032a1b0 0x556510705290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.341+0000 7f538a800640 1 -- 172.21.15.134:0/2593247807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556510332390 msgr2=0x55651070f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:51.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.341+0000 7f538a800640 1 --2- 172.21.15.134:0/2593247807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556510332390 0x55651070f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.341+0000 7f538a800640 1 -- 172.21.15.134:0/2593247807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55651032a1b0 msgr2=0x556510705290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:51.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.341+0000 7f538a800640 1 --2- 172.21.15.134:0/2593247807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55651032a1b0 0x556510705290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.341+0000 7f538a800640 1 -- 172.21.15.134:0/2593247807 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565106fa300 con 0x556510353070 2024-07-26T11:18:51.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.341+0000 7f5389e00640 1 --2- 172.21.15.134:0/2593247807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55651032a1b0 0x556510705290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:18:51.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.342+0000 7f538a800640 1 --2- 172.21.15.134:0/2593247807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 0x55651071ae40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5384003cf0 tx=0x7f5384016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce338b107b130845 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:51.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.342+0000 7f5376600640 1 -- 172.21.15.134:0/2593247807 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5384003e60 con 0x556510353070 2024-07-26T11:18:51.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.342+0000 7f5376600640 1 -- 172.21.15.134:0/2593247807 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5384011b50 con 0x556510353070 2024-07-26T11:18:51.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.342+0000 7f5376600640 1 -- 172.21.15.134:0/2593247807 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538402abe0 con 0x556510353070 2024-07-26T11:18:51.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.344+0000 7f53977fc740 1 -- 172.21.15.134:0/2593247807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 msgr2=0x55651071ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:51.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.344+0000 7f53977fc740 1 --2- 172.21.15.134:0/2593247807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 0x55651071ae40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5384003cf0 tx=0x7f5384016000 comp rx=0 tx=0).stop 2024-07-26T11:18:51.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.344+0000 7f53977fc740 1 -- 172.21.15.134:0/2593247807 shutdown_connections 2024-07-26T11:18:51.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.344+0000 7f53977fc740 1 --2- 172.21.15.134:0/2593247807 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556510332390 0x55651070f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.344+0000 7f53977fc740 1 --2- 172.21.15.134:0/2593247807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 0x55651071ae40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.344+0000 7f53977fc740 1 --2- 172.21.15.134:0/2593247807 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55651032a1b0 0x556510705290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.344+0000 7f53977fc740 1 -- 172.21.15.134:0/2593247807 >> 172.21.15.134:0/2593247807 conn(0x556510347460 msgr2=0x556510704b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:51.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.345+0000 7f53977fc740 1 -- 172.21.15.134:0/2593247807 shutdown_connections 2024-07-26T11:18:51.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.345+0000 7f53977fc740 1 -- 172.21.15.134:0/2593247807 wait complete. 2024-07-26T11:18:51.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.346+0000 7f53977fc740 1 -- start start 2024-07-26T11:18:51.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.346+0000 7f53977fc740 1 Processor -- start 2024-07-26T11:18:51.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.348+0000 7f53977fc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55651032a1b0 0x556510776280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.348+0000 7f53977fc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 0x5565107767c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.348+0000 7f53977fc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 0x55651070a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.348+0000 7f53977fc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f538c1b8130 con 0x556510332390 2024-07-26T11:18:51.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.348+0000 7f53977fc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f538c1b7a60 con 0x556510353070 2024-07-26T11:18:51.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.348+0000 7f53977fc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f538c1b4ca0 con 0x55651032a1b0 2024-07-26T11:18:51.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.348+0000 7f53977fc740 10 client.0 _ll_register_callbacks cb 0x7f537c0aa970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:18:51.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.349+0000 7f538a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 0x55651070a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.349+0000 7f5389400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 0x5565107767c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.349+0000 7f538a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 0x55651070a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:55854/0 (socket says 172.21.15.134:55854) 2024-07-26T11:18:51.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.349+0000 7f538a800640 1 -- 172.21.15.134:0/1444256792 learned_addr learned my addr 172.21.15.134:0/1444256792 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:18:51.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.349+0000 7f5389e00640 1 --2- 172.21.15.134:0/1444256792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55651032a1b0 0x556510776280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.350+0000 7f5389400640 1 -- 172.21.15.134:0/1444256792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55651032a1b0 msgr2=0x556510776280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:51.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.350+0000 7f5389400640 1 --2- 172.21.15.134:0/1444256792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55651032a1b0 0x556510776280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.350+0000 7f5389400640 1 -- 172.21.15.134:0/1444256792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 msgr2=0x55651070a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:18:51.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.350+0000 7f5389400640 1 --2- 172.21.15.134:0/1444256792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 0x55651070a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.350+0000 7f5389400640 1 -- 172.21.15.134:0/1444256792 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55651070af20 con 0x556510332390 2024-07-26T11:18:51.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.350+0000 7f5389e00640 1 --2- 172.21.15.134:0/1444256792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55651032a1b0 0x556510776280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:18:51.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.351+0000 7f5389400640 1 --2- 172.21.15.134:0/1444256792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 0x5565107767c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f537802a930 tx=0x7f5378004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=423fff1ee60da4de server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:18:51.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.351+0000 7f536b400640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:18:51.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.351+0000 7f536b400640 1 -- 172.21.15.134:0/1444256792 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537805d060 con 0x556510332390 2024-07-26T11:18:51.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.351+0000 7f536b400640 1 -- 172.21.15.134:0/1444256792 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5378003a00 con 0x556510332390 2024-07-26T11:18:51.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.351+0000 7f53977fc740 1 -- 172.21.15.134:0/1444256792 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5384011b50 con 0x556510332390 2024-07-26T11:18:51.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.353+0000 7f536b400640 1 -- 172.21.15.134:0/1444256792 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f5378064050 con 0x556510332390 2024-07-26T11:18:51.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.353+0000 7f53977fc740 10 client.15717 fetch_fsmap learned FSMap version 31 2024-07-26T11:18:51.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.354+0000 7f53977fc740 1 -- 172.21.15.134:0/1444256792 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55651070c110 con 0x556510332390 2024-07-26T11:18:51.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.354+0000 7f536b400640 1 -- 172.21.15.134:0/1444256792 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5384011b50 con 0x556510332390 2024-07-26T11:18:51.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.354+0000 7f536b400640 1 -- 172.21.15.134:0/1444256792 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f5378031cd0 con 0x556510332390 2024-07-26T11:18:51.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.354+0000 7f53977fc740 10 client.15717 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:18:51.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.355+0000 7f53977fc740 10 client.15717 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-07-26T11:18:51.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.355+0000 7f53977fc740 20 client.15717 populate_metadata read hostname 'smithi134' 2024-07-26T11:18:51.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.355+0000 7f53977fc740 1 --2- 172.21.15.134:0/1444256792 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5565102e6da0 0x556510790e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:18:51.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.355+0000 7f53977fc740 4 client.15717 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:18:51.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.355+0000 7f53977fc740 1 -- 172.21.15.134:0/1444256792 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556510670b20 con 0x5565102e6da0 2024-07-26T11:18:51.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.356+0000 7f5389e00640 1 --2- 172.21.15.134:0/1444256792 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5565102e6da0 0x556510790e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:18:51.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.356+0000 7f5389e00640 1 --2- 172.21.15.134:0/1444256792 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5565102e6da0 0x556510790e30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=64762b4e02531dd7 server_cookie=cd00ff9845180cb5 in_seq=0 out_seq=0 2024-07-26T11:18:51.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.356+0000 7f536b400640 10 client.15717 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:18:51.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.357+0000 7f536b400640 1 -- 172.21.15.134:0/1444256792 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556510670b20 con 0x5565102e6da0 2024-07-26T11:18:51.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.357+0000 7f536b400640 10 client.15717 handle_command_reply: tid=0 2024-07-26T11:18:51.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.357+0000 7f53977fc740 1 client.15717 shutdown 2024-07-26T11:18:51.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.357+0000 7f53977fc740 1 -- 172.21.15.134:0/1444256792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 msgr2=0x5565107767c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:51.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.357+0000 7f53977fc740 1 --2- 172.21.15.134:0/1444256792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 0x5565107767c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f537802a930 tx=0x7f5378004a90 comp rx=0 tx=0).stop 2024-07-26T11:18:51.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.357+0000 7f53977fc740 1 -- 172.21.15.134:0/1444256792 shutdown_connections 2024-07-26T11:18:51.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.357+0000 7f53977fc740 1 --2- 172.21.15.134:0/1444256792 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5565102e6da0 0x556510790e30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.357+0000 7f536b400640 0 client.15717 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:18:51.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.357+0000 7f53977fc740 1 --2- 172.21.15.134:0/1444256792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556510353070 0x55651070a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.357+0000 7f53977fc740 1 --2- 172.21.15.134:0/1444256792 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556510332390 0x5565107767c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.357+0000 7f53977fc740 1 --2- 172.21.15.134:0/1444256792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55651032a1b0 0x556510776280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.357+0000 7f53977fc740 1 -- 172.21.15.134:0/1444256792 >> 172.21.15.134:0/1444256792 conn(0x556510347460 msgr2=0x556510704ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:51.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.360+0000 7f53977fc740 1 -- 172.21.15.134:0/1444256792 shutdown_connections 2024-07-26T11:18:51.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.360+0000 7f53977fc740 1 -- 172.21.15.134:0/1444256792 wait complete. 2024-07-26T11:18:51.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.360+0000 7f53977fc740 20 client.15717 trim_cache size 0 max 16384 2024-07-26T11:18:51.362 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:18:51.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.361+0000 7f538b200640 1 -- 172.21.15.134:0/4055491287 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f53700b3e40 msgr2=0x7f53700b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:51.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.361+0000 7f538b200640 1 --2- 172.21.15.134:0/4055491287 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f53700b3e40 0x7f53700b6350 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5384005fd0 tx=0x7f5384009500 comp rx=0 tx=0).stop 2024-07-26T11:18:51.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.361+0000 7f538b200640 1 -- 172.21.15.134:0/4055491287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f538c1033e0 msgr2=0x7f538c1a1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:18:51.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.361+0000 7f538b200640 1 --2- 172.21.15.134:0/4055491287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f538c1033e0 0x7f538c1a1030 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f537c00e9f0 tx=0x7f537c00eeb0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.361+0000 7f538b200640 1 -- 172.21.15.134:0/4055491287 shutdown_connections 2024-07-26T11:18:51.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.361+0000 7f538b200640 1 --2- 172.21.15.134:0/4055491287 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f53700b3e40 0x7f53700b6350 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.361+0000 7f538b200640 1 --2- 172.21.15.134:0/4055491287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f538c109670 0x7f538c1ac0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.361+0000 7f538b200640 1 --2- 172.21.15.134:0/4055491287 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f538c108d20 0x7f538c1a1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.361+0000 7f538b200640 1 --2- 172.21.15.134:0/4055491287 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f538c1033e0 0x7f538c1a1030 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:18:51.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.361+0000 7f538b200640 1 -- 172.21.15.134:0/4055491287 >> 172.21.15.134:0/4055491287 conn(0x7f538c0fee70 msgr2=0x7f538c108380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:18:51.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.362+0000 7f538b200640 1 -- 172.21.15.134:0/4055491287 shutdown_connections 2024-07-26T11:18:51.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:18:51.362+0000 7f538b200640 1 -- 172.21.15.134:0/4055491287 wait complete. 2024-07-26T11:18:52.678 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:52 smithi134 ceph-mon[31966]: pgmap v567: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 213 MiB/s wr, 1.22k op/s 2024-07-26T11:18:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:52 smithi189 ceph-mon[36790]: pgmap v567: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 213 MiB/s wr, 1.22k op/s 2024-07-26T11:18:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:52 smithi193 ceph-mon[37473]: pgmap v567: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 213 MiB/s wr, 1.22k op/s 2024-07-26T11:18:53.909 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-07-26T11:18:54.684 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:54 smithi134 ceph-mon[31966]: pgmap v568: 129 pgs: 129 active+clean; 16 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 268 MiB/s wr, 1.50k op/s 2024-07-26T11:18:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:54 smithi189 ceph-mon[36790]: pgmap v568: 129 pgs: 129 active+clean; 16 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 268 MiB/s wr, 1.50k op/s 2024-07-26T11:18:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:54 smithi193 ceph-mon[37473]: pgmap v568: 129 pgs: 129 active+clean; 16 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 268 MiB/s wr, 1.50k op/s 2024-07-26T11:18:54.941 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:18:56.427 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:56 smithi134 ceph-mon[31966]: pgmap v569: 129 pgs: 129 active+clean; 16 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 261 MiB/s wr, 1.38k op/s 2024-07-26T11:18:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:56 smithi189 ceph-mon[36790]: pgmap v569: 129 pgs: 129 active+clean; 16 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 261 MiB/s wr, 1.38k op/s 2024-07-26T11:18:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:56 smithi193 ceph-mon[37473]: pgmap v569: 129 pgs: 129 active+clean; 16 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 261 MiB/s wr, 1.38k op/s 2024-07-26T11:18:58.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:18:58 smithi134 ceph-mon[31966]: pgmap v570: 129 pgs: 129 active+clean; 16 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 230 MiB/s wr, 1.19k op/s 2024-07-26T11:18:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:18:58 smithi189 ceph-mon[36790]: pgmap v570: 129 pgs: 129 active+clean; 16 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 230 MiB/s wr, 1.19k op/s 2024-07-26T11:18:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:18:58 smithi193 ceph-mon[37473]: pgmap v570: 129 pgs: 129 active+clean; 16 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 230 MiB/s wr, 1.19k op/s 2024-07-26T11:19:00.308 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:00 smithi134 ceph-mon[31966]: pgmap v571: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 MiB/s wr, 1.55k op/s 2024-07-26T11:19:00.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.360+0000 7fea9ac00640 1 -- 172.21.15.134:0/3443438224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea94109670 msgr2=0x7fea9410fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:00.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.360+0000 7fea9ac00640 1 --2- 172.21.15.134:0/3443438224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea94109670 0x7fea9410fea0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fea8c009be0 tx=0x7fea8c02a710 comp rx=0 tx=0).stop 2024-07-26T11:19:00.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.360+0000 7fea9ac00640 1 -- 172.21.15.134:0/3443438224 shutdown_connections 2024-07-26T11:19:00.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.360+0000 7fea9ac00640 1 --2- 172.21.15.134:0/3443438224 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea94108d20 0x7fea94109130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.360+0000 7fea9ac00640 1 --2- 172.21.15.134:0/3443438224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea94109670 0x7fea9410fea0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.360+0000 7fea9ac00640 1 --2- 172.21.15.134:0/3443438224 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea941033e0 0x7fea941037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.360+0000 7fea9ac00640 1 -- 172.21.15.134:0/3443438224 >> 172.21.15.134:0/3443438224 conn(0x7fea940fee70 msgr2=0x7fea941012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:00.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.361+0000 7fea9ac00640 1 -- 172.21.15.134:0/3443438224 shutdown_connections 2024-07-26T11:19:00.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.361+0000 7fea9ac00640 1 -- 172.21.15.134:0/3443438224 wait complete. 2024-07-26T11:19:00.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.363+0000 7fea9ac00640 1 Processor -- start 2024-07-26T11:19:00.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.363+0000 7fea9ac00640 1 -- start start 2024-07-26T11:19:00.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.363+0000 7fea9ac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea941033e0 0x7fea941a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.363+0000 7fea9ac00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea94108d20 0x7fea941a1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.363+0000 7fea9ac00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea94109670 0x7fea941a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.363+0000 7fea9ac00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fea94113320 con 0x7fea94108d20 2024-07-26T11:19:00.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.363+0000 7fea9ac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fea941131a0 con 0x7fea941033e0 2024-07-26T11:19:00.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.363+0000 7fea9ac00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea941134a0 con 0x7fea94109670 2024-07-26T11:19:00.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.364+0000 7fea99800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea941033e0 0x7fea941a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:00.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.364+0000 7fea99800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea941033e0 0x7fea941a1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42062/0 (socket says 172.21.15.134:42062) 2024-07-26T11:19:00.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.364+0000 7fea99800640 1 -- 172.21.15.134:0/3467172142 learned_addr learned my addr 172.21.15.134:0/3467172142 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:00.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.364+0000 7fea9a200640 1 --2- 172.21.15.134:0/3467172142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea94109670 0x7fea941a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:00.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.365+0000 7fea99800640 1 -- 172.21.15.134:0/3467172142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea94109670 msgr2=0x7fea941a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:00.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.365+0000 7fea99800640 1 --2- 172.21.15.134:0/3467172142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea94109670 0x7fea941a1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.365+0000 7fea99800640 1 -- 172.21.15.134:0/3467172142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea94108d20 msgr2=0x7fea941a1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:00.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.365+0000 7fea99800640 1 --2- 172.21.15.134:0/3467172142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea94108d20 0x7fea941a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.365+0000 7fea99800640 1 -- 172.21.15.134:0/3467172142 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea941ac280 con 0x7fea941033e0 2024-07-26T11:19:00.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.365+0000 7fea99800640 1 --2- 172.21.15.134:0/3467172142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea941033e0 0x7fea941a1000 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fea8800c920 tx=0x7fea8800cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:00.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.365+0000 7fea92000640 1 -- 172.21.15.134:0/3467172142 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea88007c70 con 0x7fea941033e0 2024-07-26T11:19:00.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.366+0000 7fea92000640 1 -- 172.21.15.134:0/3467172142 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea88007e10 con 0x7fea941033e0 2024-07-26T11:19:00.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.366+0000 7fea92000640 1 -- 172.21.15.134:0/3467172142 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea88018820 con 0x7fea941033e0 2024-07-26T11:19:00.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.367+0000 7fea9ac00640 1 -- 172.21.15.134:0/3467172142 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea941ac570 con 0x7fea941033e0 2024-07-26T11:19:00.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.367+0000 7fea9ac00640 1 -- 172.21.15.134:0/3467172142 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea941aca60 con 0x7fea941033e0 2024-07-26T11:19:00.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.373+0000 7fea92000640 1 -- 172.21.15.134:0/3467172142 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fea880189c0 con 0x7fea941033e0 2024-07-26T11:19:00.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.374+0000 7feaa2ed4740 1 Processor -- start 2024-07-26T11:19:00.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.375+0000 7feaa2ed4740 1 -- start start 2024-07-26T11:19:00.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.375+0000 7feaa2ed4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c143070 0x555f0c4e8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.375+0000 7feaa2ed4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c11a1b0 0x555f0c4e8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.375+0000 7fea92000640 1 --2- 172.21.15.134:0/3467172142 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fea7c0b3e90 0x7fea7c0b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.376+0000 7feaa2ed4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c122390 0x555f0c4e8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.376+0000 7feaa2ed4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x555f0c45ebf0 con 0x555f0c11a1b0 2024-07-26T11:19:00.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.376+0000 7feaa2ed4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555f0c44e0f0 con 0x555f0c122390 2024-07-26T11:19:00.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.376+0000 7feaa2ed4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555f0c452130 con 0x555f0c143070 2024-07-26T11:19:00.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.376+0000 7fea92000640 1 -- 172.21.15.134:0/3467172142 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fea880e4390 con 0x7fea941033e0 2024-07-26T11:19:00.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.376+0000 7fea98e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c11a1b0 0x555f0c4e8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:00.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.376+0000 7fea98e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c11a1b0 0x555f0c4e8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:40204/0 (socket says 172.21.15.134:40204) 2024-07-26T11:19:00.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.376+0000 7fea98e00640 1 -- 172.21.15.134:0/3314637726 learned_addr learned my addr 172.21.15.134:0/3314637726 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:00.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.377+0000 7fea99800640 1 --2- 172.21.15.134:0/3467172142 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fea7c0b3e90 0x7fea7c0b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:00.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.377+0000 7fea9a200640 1 --2- 172.21.15.134:0/3314637726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c143070 0x555f0c4e8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:00.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.377+0000 7fea9a200640 1 --2- 172.21.15.134:0/3314637726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c122390 0x555f0c4e8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:00.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.378+0000 7fea99800640 1 --2- 172.21.15.134:0/3467172142 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fea7c0b3e90 0x7fea7c0b63a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fea880e1810 tx=0x7fea88000bc0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:00.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.378+0000 7fea9a200640 1 -- 172.21.15.134:0/3314637726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c122390 msgr2=0x555f0c4e8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:00.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.378+0000 7fea9a200640 1 --2- 172.21.15.134:0/3314637726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c122390 0x555f0c4e8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.379+0000 7fea9a200640 1 -- 172.21.15.134:0/3314637726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c11a1b0 msgr2=0x555f0c4e8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:00.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.379+0000 7fea9a200640 1 --2- 172.21.15.134:0/3314637726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c11a1b0 0x555f0c4e8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.379+0000 7fea9a200640 1 -- 172.21.15.134:0/3314637726 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f0c1ae650 con 0x555f0c143070 2024-07-26T11:19:00.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.380+0000 7fea9a200640 1 --2- 172.21.15.134:0/3314637726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c143070 0x555f0c4e8030 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fea8c02ad00 tx=0x7fea8c009280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=697ea3f24c81ee68 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:00.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.380+0000 7fea86a00640 1 -- 172.21.15.134:0/3314637726 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c046070 con 0x555f0c143070 2024-07-26T11:19:00.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.382+0000 7fea86a00640 1 -- 172.21.15.134:0/3314637726 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea8c003e20 con 0x555f0c143070 2024-07-26T11:19:00.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.382+0000 7fea86a00640 1 -- 172.21.15.134:0/3314637726 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c036040 con 0x555f0c143070 2024-07-26T11:19:00.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.384+0000 7feaa2ed4740 1 -- 172.21.15.134:0/3314637726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c143070 msgr2=0x555f0c4e8030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:00.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.384+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/3314637726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c143070 0x555f0c4e8030 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fea8c02ad00 tx=0x7fea8c009280 comp rx=0 tx=0).stop 2024-07-26T11:19:00.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.385+0000 7feaa2ed4740 1 -- 172.21.15.134:0/3314637726 shutdown_connections 2024-07-26T11:19:00.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.385+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/3314637726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c122390 0x555f0c4e8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.385+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/3314637726 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c11a1b0 0x555f0c4e8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.385+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/3314637726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c143070 0x555f0c4e8030 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.385+0000 7feaa2ed4740 1 -- 172.21.15.134:0/3314637726 >> 172.21.15.134:0/3314637726 conn(0x555f0c137460 msgr2=0x555f0c4e5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:00.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.387+0000 7feaa2ed4740 1 -- 172.21.15.134:0/3314637726 shutdown_connections 2024-07-26T11:19:00.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.389+0000 7feaa2ed4740 1 -- 172.21.15.134:0/3314637726 wait complete. 2024-07-26T11:19:00.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.390+0000 7feaa2ed4740 1 -- start start 2024-07-26T11:19:00.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.390+0000 7feaa2ed4740 1 Processor -- start 2024-07-26T11:19:00.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.392+0000 7feaa2ed4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c11a1b0 0x555f0c504680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.394+0000 7fea9a200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c11a1b0 0x555f0c504680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:00.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.394+0000 7fea9a200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c11a1b0 0x555f0c504680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:40214/0 (socket says 172.21.15.134:40214) 2024-07-26T11:19:00.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.394+0000 7fea9a200640 1 -- 172.21.15.134:0/2964408628 learned_addr learned my addr 172.21.15.134:0/2964408628 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:00.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.395+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/2964408628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c504bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.395+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/2964408628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c143070 0x555f0c505120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.395+0000 7feaa2ed4740 1 -- 172.21.15.134:0/2964408628 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x555f0c2b1700 con 0x555f0c11a1b0 2024-07-26T11:19:00.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.395+0000 7feaa2ed4740 1 -- 172.21.15.134:0/2964408628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555f0c3ea6c0 con 0x555f0c143070 2024-07-26T11:19:00.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.395+0000 7feaa2ed4740 1 -- 172.21.15.134:0/2964408628 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555f0c507710 con 0x555f0c122390 2024-07-26T11:19:00.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.395+0000 7fea98e00640 1 --2- 172.21.15.134:0/2964408628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c504bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:00.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.395+0000 7feaa2ed4740 10 client.0 _ll_register_callbacks cb 0x555f0c3fa170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:00.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.396+0000 7fea9a200640 1 --2- 172.21.15.134:0/2964408628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c143070 0x555f0c505120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:00.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.396+0000 7fea98e00640 1 -- 172.21.15.134:0/2964408628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c143070 msgr2=0x555f0c505120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:00.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.396+0000 7fea98e00640 1 --2- 172.21.15.134:0/2964408628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c143070 0x555f0c505120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.396+0000 7fea98e00640 1 -- 172.21.15.134:0/2964408628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c11a1b0 msgr2=0x555f0c504680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:00.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.396+0000 7fea98e00640 1 --2- 172.21.15.134:0/2964408628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c11a1b0 0x555f0c504680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.396+0000 7fea98e00640 1 -- 172.21.15.134:0/2964408628 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f0c401860 con 0x555f0c122390 2024-07-26T11:19:00.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.396+0000 7fea98e00640 1 --2- 172.21.15.134:0/2964408628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c504bc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x555f0c4e6cb0 tx=0x7fea80011b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2b48d298c70d596 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:00.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.397+0000 7fea7b400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:19:00.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.397+0000 7fea7b400640 1 -- 172.21.15.134:0/2964408628 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8000fdf0 con 0x555f0c122390 2024-07-26T11:19:00.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.397+0000 7fea7b400640 1 -- 172.21.15.134:0/2964408628 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fea80004030 con 0x555f0c122390 2024-07-26T11:19:00.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.397+0000 7feaa2ed4740 1 -- 172.21.15.134:0/2964408628 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea8c003e20 con 0x555f0c122390 2024-07-26T11:19:00.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.398+0000 7fea7b400640 1 -- 172.21.15.134:0/2964408628 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fea80004270 con 0x555f0c122390 2024-07-26T11:19:00.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.398+0000 7feaa2ed4740 10 client.26017 fetch_fsmap learned FSMap version 31 2024-07-26T11:19:00.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.399+0000 7feaa2ed4740 1 -- 172.21.15.134:0/2964408628 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555f0c513480 con 0x555f0c122390 2024-07-26T11:19:00.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.400+0000 7fea7b400640 1 -- 172.21.15.134:0/2964408628 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea8c003e20 con 0x555f0c122390 2024-07-26T11:19:00.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.400+0000 7fea7b400640 1 -- 172.21.15.134:0/2964408628 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7fea80019420 con 0x555f0c122390 2024-07-26T11:19:00.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.401+0000 7feaa2ed4740 10 client.26017 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:19:00.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.401+0000 7feaa2ed4740 10 client.26017 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T11:19:00.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.401+0000 7feaa2ed4740 20 client.26017 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:00.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.402+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/2964408628 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x555f0c1019f0 0x555f0c516990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.402+0000 7feaa2ed4740 4 client.26017 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:19:00.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.402+0000 7feaa2ed4740 1 -- 172.21.15.134:0/2964408628 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f0c460b40 con 0x555f0c1019f0 2024-07-26T11:19:00.405 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.402+0000 7fea9a200640 1 --2- 172.21.15.134:0/2964408628 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x555f0c1019f0 0x555f0c516990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:00.405 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.403+0000 7fea9a200640 1 --2- 172.21.15.134:0/2964408628 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x555f0c1019f0 0x555f0c516990 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=af62a61cad875ad4 server_cookie=53f1acdf6f53b3cf in_seq=0 out_seq=0 2024-07-26T11:19:00.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.403+0000 7fea7b400640 10 client.26017 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:19:00.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.405+0000 7fea7b400640 1 -- 172.21.15.134:0/2964408628 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555f0c460b40 con 0x555f0c1019f0 2024-07-26T11:19:00.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.405+0000 7fea7b400640 10 client.26017 handle_command_reply: tid=0 2024-07-26T11:19:00.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.406+0000 7feaa2ed4740 1 client.26017 shutdown 2024-07-26T11:19:00.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.407+0000 7feaa2ed4740 1 -- 172.21.15.134:0/2964408628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 msgr2=0x555f0c504bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:00.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.407+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/2964408628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c504bc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x555f0c4e6cb0 tx=0x7fea80011b50 comp rx=0 tx=0).stop 2024-07-26T11:19:00.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.407+0000 7feaa2ed4740 1 -- 172.21.15.134:0/2964408628 shutdown_connections 2024-07-26T11:19:00.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.408+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/2964408628 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x555f0c1019f0 0x555f0c516990 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.408+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/2964408628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c504bc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.408+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/2964408628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c143070 0x555f0c505120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.408+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/2964408628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c11a1b0 0x555f0c504680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.408+0000 7feaa2ed4740 1 -- 172.21.15.134:0/2964408628 >> 172.21.15.134:0/2964408628 conn(0x555f0c137460 msgr2=0x555f0c4e9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:00.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.408+0000 7fea7b400640 0 client.26017 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:19:00.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.410+0000 7feaa2ed4740 1 -- 172.21.15.134:0/2964408628 shutdown_connections 2024-07-26T11:19:00.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.410+0000 7feaa2ed4740 1 -- 172.21.15.134:0/2964408628 wait complete. 2024-07-26T11:19:00.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.412+0000 7feaa2ed4740 20 client.26017 trim_cache size 0 max 16384 2024-07-26T11:19:00.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.442+0000 7feaa2ed4740 1 Processor -- start 2024-07-26T11:19:00.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.442+0000 7feaa2ed4740 1 -- start start 2024-07-26T11:19:00.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.442+0000 7feaa2ed4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c11a1b0 0x555f0c5062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.442+0000 7feaa2ed4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c51c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.442+0000 7feaa2ed4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c143070 0x555f0c51c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.442+0000 7feaa2ed4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fea941ae450 con 0x555f0c143070 2024-07-26T11:19:00.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.442+0000 7feaa2ed4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fea941affc0 con 0x555f0c11a1b0 2024-07-26T11:19:00.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.442+0000 7feaa2ed4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea941aeb20 con 0x555f0c122390 2024-07-26T11:19:00.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.443+0000 7fea98e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c51c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:00.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.443+0000 7fea98e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c51c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:40112/0 (socket says 172.21.15.134:40112) 2024-07-26T11:19:00.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.443+0000 7fea98e00640 1 -- 172.21.15.134:0/2810734551 learned_addr learned my addr 172.21.15.134:0/2810734551 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:00.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.443+0000 7fea9a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c11a1b0 0x555f0c5062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:00.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.444+0000 7fea98e00640 1 -- 172.21.15.134:0/2810734551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c11a1b0 msgr2=0x555f0c5062b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:00.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.444+0000 7fea98e00640 1 --2- 172.21.15.134:0/2810734551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c11a1b0 0x555f0c5062b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.444+0000 7fea98e00640 1 -- 172.21.15.134:0/2810734551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c143070 msgr2=0x555f0c51c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:00.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.444+0000 7fea98e00640 1 --2- 172.21.15.134:0/2810734551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c143070 0x555f0c51c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.444+0000 7fea98e00640 1 -- 172.21.15.134:0/2810734551 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f0c507650 con 0x555f0c122390 2024-07-26T11:19:00.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.445+0000 7fea98e00640 1 --2- 172.21.15.134:0/2810734551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c51c150 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fea80006010 tx=0x7fea8001b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61e95472b3aaad46 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:00.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.446+0000 7fea86000640 1 -- 172.21.15.134:0/2810734551 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea80004410 con 0x555f0c122390 2024-07-26T11:19:00.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.446+0000 7fea86000640 1 -- 172.21.15.134:0/2810734551 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea80031070 con 0x555f0c122390 2024-07-26T11:19:00.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.446+0000 7fea86000640 1 -- 172.21.15.134:0/2810734551 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea80023420 con 0x555f0c122390 2024-07-26T11:19:00.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.447+0000 7feaa2ed4740 1 -- 172.21.15.134:0/2810734551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 msgr2=0x555f0c51c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:00.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.448+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/2810734551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c51c150 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fea80006010 tx=0x7fea8001b000 comp rx=0 tx=0).stop 2024-07-26T11:19:00.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.448+0000 7feaa2ed4740 1 -- 172.21.15.134:0/2810734551 shutdown_connections 2024-07-26T11:19:00.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.448+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/2810734551 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c51c150 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.448+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/2810734551 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c143070 0x555f0c51c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.448+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/2810734551 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c11a1b0 0x555f0c5062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.448+0000 7feaa2ed4740 1 -- 172.21.15.134:0/2810734551 >> 172.21.15.134:0/2810734551 conn(0x555f0c137460 msgr2=0x555f0c505ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:00.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.449+0000 7feaa2ed4740 1 -- 172.21.15.134:0/2810734551 shutdown_connections 2024-07-26T11:19:00.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.449+0000 7feaa2ed4740 1 -- 172.21.15.134:0/2810734551 wait complete. 2024-07-26T11:19:00.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.449+0000 7feaa2ed4740 1 -- start start 2024-07-26T11:19:00.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.449+0000 7feaa2ed4740 1 Processor -- start 2024-07-26T11:19:00.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.450+0000 7feaa2ed4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c11a1b0 0x555f0c508170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.450+0000 7feaa2ed4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c5086b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.450+0000 7feaa2ed4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c143070 0x555f0c57d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.450+0000 7feaa2ed4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fea941af8c0 con 0x555f0c143070 2024-07-26T11:19:00.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.450+0000 7feaa2ed4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fea941af1f0 con 0x555f0c11a1b0 2024-07-26T11:19:00.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.450+0000 7feaa2ed4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea94074760 con 0x555f0c122390 2024-07-26T11:19:00.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.450+0000 7feaa2ed4740 10 client.0 _ll_register_callbacks cb 0x7fea80009d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:00.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.451+0000 7fea9a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c11a1b0 0x555f0c508170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:00.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.451+0000 7fea98e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c5086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:00.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.451+0000 7fea9a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c11a1b0 0x555f0c508170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42088/0 (socket says 172.21.15.134:42088) 2024-07-26T11:19:00.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.451+0000 7fea98e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c5086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:40114/0 (socket says 172.21.15.134:40114) 2024-07-26T11:19:00.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.451+0000 7fea9a200640 1 -- 172.21.15.134:0/3750045653 learned_addr learned my addr 172.21.15.134:0/3750045653 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:00.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.451+0000 7fea98e00640 1 -- 172.21.15.134:0/3750045653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c11a1b0 msgr2=0x555f0c508170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:00.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.451+0000 7fea98e00640 1 --2- 172.21.15.134:0/3750045653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c11a1b0 0x555f0c508170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.451+0000 7fea98e00640 1 -- 172.21.15.134:0/3750045653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c143070 msgr2=0x555f0c57d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.451+0000 7fea98e00640 1 --2- 172.21.15.134:0/3750045653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c143070 0x555f0c57d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.451+0000 7fea98e00640 1 -- 172.21.15.134:0/3750045653 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f0c57d6d0 con 0x555f0c122390 2024-07-26T11:19:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.452+0000 7fea98e00640 1 --2- 172.21.15.134:0/3750045653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c5086b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fea800198c0 tx=0x7fea80004660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f69ae0e389dd3b5a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.452+0000 7fea7b400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:19:00.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.452+0000 7fea7b400640 1 -- 172.21.15.134:0/3750045653 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea80019a30 con 0x555f0c122390 2024-07-26T11:19:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.452+0000 7fea7b400640 1 -- 172.21.15.134:0/3750045653 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fea80019bd0 con 0x555f0c122390 2024-07-26T11:19:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.453+0000 7feaa2ed4740 1 -- 172.21.15.134:0/3750045653 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea80031070 con 0x555f0c122390 2024-07-26T11:19:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.453+0000 7fea7b400640 1 -- 172.21.15.134:0/3750045653 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fea80019dd0 con 0x555f0c122390 2024-07-26T11:19:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.453+0000 7feaa2ed4740 10 client.26023 fetch_fsmap learned FSMap version 31 2024-07-26T11:19:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.453+0000 7feaa2ed4740 1 -- 172.21.15.134:0/3750045653 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x555f0c57e950 con 0x555f0c122390 2024-07-26T11:19:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.454+0000 7fea7b400640 1 -- 172.21.15.134:0/3750045653 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea80031070 con 0x555f0c122390 2024-07-26T11:19:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.454+0000 7fea7b400640 1 -- 172.21.15.134:0/3750045653 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7fea80023420 con 0x555f0c122390 2024-07-26T11:19:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.454+0000 7feaa2ed4740 10 client.26023 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:19:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.454+0000 7feaa2ed4740 10 client.26023 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T11:19:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.454+0000 7feaa2ed4740 20 client.26023 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.455+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/3750045653 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x555f0c0d6da0 0x555f0c581000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.455+0000 7feaa2ed4740 4 client.26023 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:19:00.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.455+0000 7feaa2ed4740 1 -- 172.21.15.134:0/3750045653 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555f0c45c5e0 con 0x555f0c0d6da0 2024-07-26T11:19:00.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.456+0000 7fea9a200640 1 --2- 172.21.15.134:0/3750045653 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x555f0c0d6da0 0x555f0c581000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.457+0000 7fea7b400640 10 client.26023 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:19:00.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.461+0000 7fea9a200640 1 --2- 172.21.15.134:0/3750045653 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x555f0c0d6da0 0x555f0c581000 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=20829f76bd5c6066 server_cookie=eca343a7f2d44b6e in_seq=0 out_seq=0 2024-07-26T11:19:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.464+0000 7fea7b400640 1 -- 172.21.15.134:0/3750045653 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555f0c45c5e0 con 0x555f0c0d6da0 2024-07-26T11:19:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.464+0000 7fea7b400640 10 client.26023 handle_command_reply: tid=0 2024-07-26T11:19:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.464+0000 7feaa2ed4740 1 client.26023 shutdown 2024-07-26T11:19:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.464+0000 7feaa2ed4740 1 -- 172.21.15.134:0/3750045653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 msgr2=0x555f0c5086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:00.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.464+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/3750045653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c5086b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fea800198c0 tx=0x7fea80004660 comp rx=0 tx=0).stop 2024-07-26T11:19:00.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.466+0000 7feaa2ed4740 1 -- 172.21.15.134:0/3750045653 shutdown_connections 2024-07-26T11:19:00.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.466+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/3750045653 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x555f0c0d6da0 0x555f0c581000 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.466+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/3750045653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f0c122390 0x555f0c5086b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.466+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/3750045653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555f0c143070 0x555f0c57d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.466+0000 7feaa2ed4740 1 --2- 172.21.15.134:0/3750045653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f0c11a1b0 0x555f0c508170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.466+0000 7feaa2ed4740 1 -- 172.21.15.134:0/3750045653 >> 172.21.15.134:0/3750045653 conn(0x555f0c137460 msgr2=0x555f0c4ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:00.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.466+0000 7fea7b400640 0 client.26023 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:19:00.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.466+0000 7feaa2ed4740 1 -- 172.21.15.134:0/3750045653 shutdown_connections 2024-07-26T11:19:00.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.467+0000 7feaa2ed4740 1 -- 172.21.15.134:0/3750045653 wait complete. 2024-07-26T11:19:00.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.467+0000 7feaa2ed4740 20 client.26023 trim_cache size 0 max 16384 2024-07-26T11:19:00.471 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:19:00.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.468+0000 7fea9ac00640 1 -- 172.21.15.134:0/3467172142 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fea7c0b3e90 msgr2=0x7fea7c0b63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:00.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.468+0000 7fea9ac00640 1 --2- 172.21.15.134:0/3467172142 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fea7c0b3e90 0x7fea7c0b63a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fea880e1810 tx=0x7fea88000bc0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.468+0000 7fea9ac00640 1 -- 172.21.15.134:0/3467172142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea941033e0 msgr2=0x7fea941a1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:00.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.468+0000 7fea9ac00640 1 --2- 172.21.15.134:0/3467172142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea941033e0 0x7fea941a1000 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fea8800c920 tx=0x7fea8800cde0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.468+0000 7fea9ac00640 1 -- 172.21.15.134:0/3467172142 shutdown_connections 2024-07-26T11:19:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.468+0000 7fea9ac00640 1 --2- 172.21.15.134:0/3467172142 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fea7c0b3e90 0x7fea7c0b63a0 secure :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fea880e1810 tx=0x7fea88000bc0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.468+0000 7fea9ac00640 1 --2- 172.21.15.134:0/3467172142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea94109670 0x7fea941a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.468+0000 7fea9ac00640 1 --2- 172.21.15.134:0/3467172142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea94108d20 0x7fea941a1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.468+0000 7fea9ac00640 1 --2- 172.21.15.134:0/3467172142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea941033e0 0x7fea941a1000 secure :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fea8800c920 tx=0x7fea8800cde0 comp rx=0 tx=0).stop 2024-07-26T11:19:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.468+0000 7fea9ac00640 1 -- 172.21.15.134:0/3467172142 >> 172.21.15.134:0/3467172142 conn(0x7fea940fee70 msgr2=0x7fea940ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.469+0000 7fea9ac00640 1 -- 172.21.15.134:0/3467172142 shutdown_connections 2024-07-26T11:19:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:00.469+0000 7fea9ac00640 1 -- 172.21.15.134:0/3467172142 wait complete. 2024-07-26T11:19:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:00 smithi189 ceph-mon[36790]: pgmap v571: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 MiB/s wr, 1.55k op/s 2024-07-26T11:19:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:00 smithi193 ceph-mon[37473]: pgmap v571: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 MiB/s wr, 1.55k op/s 2024-07-26T11:19:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:02 smithi134 ceph-mon[31966]: pgmap v572: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 219 MiB/s wr, 1.08k op/s 2024-07-26T11:19:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:02 smithi189 ceph-mon[36790]: pgmap v572: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 219 MiB/s wr, 1.08k op/s 2024-07-26T11:19:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:02 smithi193 ceph-mon[37473]: pgmap v572: 129 pgs: 129 active+clean; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 219 MiB/s wr, 1.08k op/s 2024-07-26T11:19:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.576+0000 7f2549a00640 1 -- 172.21.15.134:0/778484963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2544075800 msgr2=0x7f2544075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.576+0000 7f2549a00640 1 --2- 172.21.15.134:0/778484963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2544075800 0x7f2544075c10 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2530009900 tx=0x7f253002a400 comp rx=0 tx=0).stop 2024-07-26T11:19:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.577+0000 7f2549a00640 1 -- 172.21.15.134:0/778484963 shutdown_connections 2024-07-26T11:19:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.577+0000 7f2549a00640 1 --2- 172.21.15.134:0/778484963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2544075800 0x7f2544075c10 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.577+0000 7f2549a00640 1 --2- 172.21.15.134:0/778484963 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2544113c00 0x7f25441160d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.577+0000 7f2549a00640 1 --2- 172.21.15.134:0/778484963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2544076dc0 0x7f25440751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.577+0000 7f2549a00640 1 -- 172.21.15.134:0/778484963 >> 172.21.15.134:0/778484963 conn(0x7f25440fed60 msgr2=0x7f25441011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:03.580 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.577+0000 7f2549a00640 1 -- 172.21.15.134:0/778484963 shutdown_connections 2024-07-26T11:19:03.580 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.577+0000 7f2549a00640 1 -- 172.21.15.134:0/778484963 wait complete. 2024-07-26T11:19:03.580 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.578+0000 7f2549a00640 1 Processor -- start 2024-07-26T11:19:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.579+0000 7f2549a00640 1 -- start start 2024-07-26T11:19:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.579+0000 7f2549a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2544075800 0x7f25441a02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.579+0000 7f2549a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2544076dc0 0x7f25441a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.579+0000 7f2549a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2544113c00 0x7f25441a5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.579+0000 7f2549a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2544119890 con 0x7f2544075800 2024-07-26T11:19:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.579+0000 7f2549a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2544119710 con 0x7f2544076dc0 2024-07-26T11:19:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.579+0000 7f2549a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2544119a10 con 0x7f2544113c00 2024-07-26T11:19:03.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.579+0000 7f2543400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2544076dc0 0x7f25441a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.579+0000 7f2543400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2544076dc0 0x7f25441a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42102/0 (socket says 172.21.15.134:42102) 2024-07-26T11:19:03.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.579+0000 7f2543400640 1 -- 172.21.15.134:0/1675880947 learned_addr learned my addr 172.21.15.134:0/1675880947 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:03.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.580+0000 7f2543400640 1 -- 172.21.15.134:0/1675880947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2544113c00 msgr2=0x7f25441a5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:03.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.580+0000 7f2549000640 1 --2- 172.21.15.134:0/1675880947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2544113c00 0x7f25441a5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.580+0000 7f2543e00640 1 --2- 172.21.15.134:0/1675880947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2544075800 0x7f25441a02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.580+0000 7f2543400640 1 --2- 172.21.15.134:0/1675880947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2544113c00 0x7f25441a5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.580+0000 7f2543400640 1 -- 172.21.15.134:0/1675880947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2544075800 msgr2=0x7f25441a02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:03.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.580+0000 7f2543400640 1 --2- 172.21.15.134:0/1675880947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2544075800 0x7f25441a02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.580+0000 7f2543400640 1 -- 172.21.15.134:0/1675880947 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25441a6220 con 0x7f2544076dc0 2024-07-26T11:19:03.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.580+0000 7f2543e00640 1 --2- 172.21.15.134:0/1675880947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2544075800 0x7f25441a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:19:03.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.581+0000 7f2543400640 1 --2- 172.21.15.134:0/1675880947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2544076dc0 0x7f25441a0890 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2530005ab0 tx=0x7f2530003c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:03.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.581+0000 7f2540c00640 1 -- 172.21.15.134:0/1675880947 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253003f070 con 0x7f2544076dc0 2024-07-26T11:19:03.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.581+0000 7f2540c00640 1 -- 172.21.15.134:0/1675880947 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2530031d60 con 0x7f2544076dc0 2024-07-26T11:19:03.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.581+0000 7f2540c00640 1 -- 172.21.15.134:0/1675880947 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2530035030 con 0x7f2544076dc0 2024-07-26T11:19:03.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.581+0000 7f2549a00640 1 -- 172.21.15.134:0/1675880947 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25441a64b0 con 0x7f2544076dc0 2024-07-26T11:19:03.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.582+0000 7f2549a00640 1 -- 172.21.15.134:0/1675880947 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25441a6950 con 0x7f2544076dc0 2024-07-26T11:19:03.590 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.588+0000 7f2540c00640 1 -- 172.21.15.134:0/1675880947 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f2530042070 con 0x7f2544076dc0 2024-07-26T11:19:03.591 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.589+0000 7f2552080740 1 Processor -- start 2024-07-26T11:19:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.590+0000 7f2552080740 1 -- start start 2024-07-26T11:19:03.593 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.590+0000 7f2540c00640 1 --2- 172.21.15.134:0/1675880947 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f25280b3dc0 0x7f25280b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.590+0000 7f2540c00640 1 -- 172.21.15.134:0/1675880947 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2530104be0 con 0x7f2544076dc0 2024-07-26T11:19:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.591+0000 7f2552080740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff931b0 0x5612a0336c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.591+0000 7f2552080740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff6a2f0 0x5612a0337160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.591+0000 7f2552080740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff724d0 0x5612a03376a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.591+0000 7f2552080740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5612a02ab270 con 0x56129ff724d0 2024-07-26T11:19:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.591+0000 7f2552080740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5612a029a770 con 0x56129ff6a2f0 2024-07-26T11:19:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.591+0000 7f2552080740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5612a029e7b0 con 0x56129ff931b0 2024-07-26T11:19:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.591+0000 7f2549000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff931b0 0x5612a0336c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.592+0000 7f2549000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff931b0 0x5612a0336c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:40146/0 (socket says 172.21.15.134:40146) 2024-07-26T11:19:03.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.592+0000 7f2549000640 1 -- 172.21.15.134:0/4024580598 learned_addr learned my addr 172.21.15.134:0/4024580598 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.592+0000 7f2549000640 1 --2- 172.21.15.134:0/4024580598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff724d0 0x5612a03376a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.592+0000 7f2543400640 1 --2- 172.21.15.134:0/4024580598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff6a2f0 0x5612a0337160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.592+0000 7f2549000640 1 -- 172.21.15.134:0/4024580598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff6a2f0 msgr2=0x5612a0337160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.592+0000 7f2549000640 1 --2- 172.21.15.134:0/4024580598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff6a2f0 0x5612a0337160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.592+0000 7f2549000640 1 -- 172.21.15.134:0/4024580598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff724d0 msgr2=0x5612a03376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:03.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.592+0000 7f2549000640 1 --2- 172.21.15.134:0/4024580598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff724d0 0x5612a03376a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.592+0000 7f2549000640 1 -- 172.21.15.134:0/4024580598 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56129fffe790 con 0x56129ff931b0 2024-07-26T11:19:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.592+0000 7f2543400640 1 --2- 172.21.15.134:0/4024580598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff6a2f0 0x5612a0337160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:19:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.593+0000 7f2543e00640 1 --2- 172.21.15.134:0/1675880947 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f25280b3dc0 0x7f25280b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.593+0000 7f2549000640 1 --2- 172.21.15.134:0/4024580598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff931b0 0x5612a0336c20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f253c013670 tx=0x7f253c009760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c33a4deec4beeb2f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.593+0000 7f2539600640 1 -- 172.21.15.134:0/4024580598 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c019030 con 0x56129ff931b0 2024-07-26T11:19:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.593+0000 7f2539600640 1 -- 172.21.15.134:0/4024580598 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f253c011570 con 0x56129ff931b0 2024-07-26T11:19:03.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.593+0000 7f2539600640 1 -- 172.21.15.134:0/4024580598 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c005220 con 0x56129ff931b0 2024-07-26T11:19:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.594+0000 7f2552080740 1 -- 172.21.15.134:0/4024580598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff931b0 msgr2=0x5612a0336c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.594+0000 7f2552080740 1 --2- 172.21.15.134:0/4024580598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff931b0 0x5612a0336c20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f253c013670 tx=0x7f253c009760 comp rx=0 tx=0).stop 2024-07-26T11:19:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.594+0000 7f2552080740 1 -- 172.21.15.134:0/4024580598 shutdown_connections 2024-07-26T11:19:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.594+0000 7f2552080740 1 --2- 172.21.15.134:0/4024580598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff724d0 0x5612a03376a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.594+0000 7f2552080740 1 --2- 172.21.15.134:0/4024580598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff6a2f0 0x5612a0337160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.594+0000 7f2552080740 1 --2- 172.21.15.134:0/4024580598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff931b0 0x5612a0336c20 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.595+0000 7f2552080740 1 -- 172.21.15.134:0/4024580598 >> 172.21.15.134:0/4024580598 conn(0x56129ff87300 msgr2=0x5612a0334b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.597+0000 7f2543e00640 1 --2- 172.21.15.134:0/1675880947 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f25280b3dc0 0x7f25280b62d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2534009b50 tx=0x7f2534005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.597+0000 7f2552080740 1 -- 172.21.15.134:0/4024580598 shutdown_connections 2024-07-26T11:19:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.597+0000 7f2552080740 1 -- 172.21.15.134:0/4024580598 wait complete. 2024-07-26T11:19:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.597+0000 7f2552080740 1 -- start start 2024-07-26T11:19:03.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.598+0000 7f2552080740 1 Processor -- start 2024-07-26T11:19:03.601 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.599+0000 7f2552080740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff6a2f0 0x5612a03427c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.603 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.600+0000 7f2552080740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff724d0 0x5612a0342d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.603 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.600+0000 7f2552080740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff931b0 0x5612a034e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.603 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.600+0000 7f2552080740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5612a01d4ab0 con 0x56129ff931b0 2024-07-26T11:19:03.603 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.600+0000 7f2552080740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5612a0237d50 con 0x56129ff724d0 2024-07-26T11:19:03.603 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.600+0000 7f2552080740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5612a03453a0 con 0x56129ff6a2f0 2024-07-26T11:19:03.603 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.600+0000 7f2543e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff6a2f0 0x5612a03427c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.603 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.601+0000 7f2543e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff6a2f0 0x5612a03427c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:40156/0 (socket says 172.21.15.134:40156) 2024-07-26T11:19:03.603 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.601+0000 7f2543e00640 1 -- 172.21.15.134:0/252521884 learned_addr learned my addr 172.21.15.134:0/252521884 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.601+0000 7f2552080740 10 client.0 _ll_register_callbacks cb 0x56129ffaaa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.601+0000 7f2549000640 1 --2- 172.21.15.134:0/252521884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff931b0 0x5612a034e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.601+0000 7f2543e00640 1 -- 172.21.15.134:0/252521884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff724d0 msgr2=0x5612a0342d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.601+0000 7f2543e00640 1 --2- 172.21.15.134:0/252521884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff724d0 0x5612a0342d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.601+0000 7f2543e00640 1 -- 172.21.15.134:0/252521884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff931b0 msgr2=0x5612a034e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:03.604 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.601+0000 7f2543e00640 1 --2- 172.21.15.134:0/252521884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff931b0 0x5612a034e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.601+0000 7f2543e00640 1 -- 172.21.15.134:0/252521884 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612a0077620 con 0x56129ff6a2f0 2024-07-26T11:19:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.602+0000 7f2543e00640 1 --2- 172.21.15.134:0/252521884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff6a2f0 0x5612a03427c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2534010fd0 tx=0x7f25340046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4167d3078901205b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.602+0000 7f252e000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:19:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.603+0000 7f252e000640 1 -- 172.21.15.134:0/252521884 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534023070 con 0x56129ff6a2f0 2024-07-26T11:19:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.603+0000 7f2552080740 1 -- 172.21.15.134:0/252521884 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f253c011570 con 0x56129ff6a2f0 2024-07-26T11:19:03.605 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.603+0000 7f252e000640 1 -- 172.21.15.134:0/252521884 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f253401f860 con 0x56129ff6a2f0 2024-07-26T11:19:03.607 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.604+0000 7f252e000640 1 -- 172.21.15.134:0/252521884 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f253402f050 con 0x56129ff6a2f0 2024-07-26T11:19:03.607 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.605+0000 7f2552080740 10 client.26032 fetch_fsmap learned FSMap version 31 2024-07-26T11:19:03.607 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.605+0000 7f2552080740 1 -- 172.21.15.134:0/252521884 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5612a03513d0 con 0x56129ff6a2f0 2024-07-26T11:19:03.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.606+0000 7f252e000640 1 -- 172.21.15.134:0/252521884 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f253c011570 con 0x56129ff6a2f0 2024-07-26T11:19:03.611 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.609+0000 7f252e000640 1 -- 172.21.15.134:0/252521884 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f253400f040 con 0x56129ff6a2f0 2024-07-26T11:19:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.610+0000 7f2552080740 10 client.26032 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:19:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.610+0000 7f2552080740 10 client.26032 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:19:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.610+0000 7f2552080740 20 client.26032 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.610+0000 7f2552080740 1 --2- 172.21.15.134:0/252521884 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56129ff51d80 0x5612a03549f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.610+0000 7f2552080740 4 client.26032 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:19:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.610+0000 7f2552080740 1 -- 172.21.15.134:0/252521884 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612a02ad1c0 con 0x56129ff51d80 2024-07-26T11:19:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.611+0000 7f2543400640 1 --2- 172.21.15.134:0/252521884 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56129ff51d80 0x5612a03549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.614 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.612+0000 7f252e000640 10 client.26032 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:03.615 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.612+0000 7f2543400640 1 --2- 172.21.15.134:0/252521884 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56129ff51d80 0x5612a03549f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa4205e8201f818c server_cookie=3e9407b856be6852 in_seq=0 out_seq=0 2024-07-26T11:19:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.614+0000 7f252e000640 1 -- 172.21.15.134:0/252521884 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5612a02ad1c0 con 0x56129ff51d80 2024-07-26T11:19:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.614+0000 7f252e000640 10 client.26032 handle_command_reply: tid=0 2024-07-26T11:19:03.616 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.614+0000 7f2552080740 1 client.26032 shutdown 2024-07-26T11:19:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.616+0000 7f2552080740 1 -- 172.21.15.134:0/252521884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff6a2f0 msgr2=0x5612a03427c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:03.618 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.616+0000 7f2552080740 1 --2- 172.21.15.134:0/252521884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff6a2f0 0x5612a03427c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2534010fd0 tx=0x7f25340046c0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.619 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.617+0000 7f2552080740 1 -- 172.21.15.134:0/252521884 shutdown_connections 2024-07-26T11:19:03.619 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.617+0000 7f2552080740 1 --2- 172.21.15.134:0/252521884 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56129ff51d80 0x5612a03549f0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.619 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.617+0000 7f2552080740 1 --2- 172.21.15.134:0/252521884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff931b0 0x5612a034e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.619 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.617+0000 7f2552080740 1 --2- 172.21.15.134:0/252521884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff724d0 0x5612a0342d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.619 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.617+0000 7f2552080740 1 --2- 172.21.15.134:0/252521884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff6a2f0 0x5612a03427c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.619 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.617+0000 7f2552080740 1 -- 172.21.15.134:0/252521884 >> 172.21.15.134:0/252521884 conn(0x56129ff87300 msgr2=0x5612a0337c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:03.619 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.617+0000 7f252e000640 0 client.26032 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:03.620 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.618+0000 7f2552080740 1 -- 172.21.15.134:0/252521884 shutdown_connections 2024-07-26T11:19:03.620 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.619+0000 7f2552080740 1 -- 172.21.15.134:0/252521884 wait complete. 2024-07-26T11:19:03.621 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.619+0000 7f2552080740 20 client.26032 trim_cache size 0 max 16384 2024-07-26T11:19:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.648+0000 7f2552080740 1 Processor -- start 2024-07-26T11:19:03.650 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.648+0000 7f2552080740 1 -- start start 2024-07-26T11:19:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.648+0000 7f2552080740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff6a2f0 0x5612a0334cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.648+0000 7f2552080740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff724d0 0x5612a0342a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.648+0000 7f2552080740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff931b0 0x5612a03441e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.648+0000 7f2552080740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f25441b6b80 con 0x56129ff6a2f0 2024-07-26T11:19:03.651 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.648+0000 7f2552080740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f25441b86f0 con 0x56129ff931b0 2024-07-26T11:19:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.648+0000 7f2552080740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f25441b7250 con 0x56129ff724d0 2024-07-26T11:19:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.649+0000 7f2543400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff724d0 0x5612a0342a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.649+0000 7f2543400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff724d0 0x5612a0342a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:40158/0 (socket says 172.21.15.134:40158) 2024-07-26T11:19:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.649+0000 7f2543400640 1 -- 172.21.15.134:0/1643923407 learned_addr learned my addr 172.21.15.134:0/1643923407 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.649+0000 7f2549000640 1 --2- 172.21.15.134:0/1643923407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff931b0 0x5612a03441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.649+0000 7f2543400640 1 -- 172.21.15.134:0/1643923407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff931b0 msgr2=0x5612a03441e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.649+0000 7f2543400640 1 --2- 172.21.15.134:0/1643923407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff931b0 0x5612a03441e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.649+0000 7f2543400640 1 -- 172.21.15.134:0/1643923407 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff6a2f0 msgr2=0x5612a0334cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:03.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.649+0000 7f2543e00640 1 --2- 172.21.15.134:0/1643923407 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff6a2f0 0x5612a0334cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.650+0000 7f2543400640 1 --2- 172.21.15.134:0/1643923407 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff6a2f0 0x5612a0334cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.650+0000 7f2543400640 1 -- 172.21.15.134:0/1643923407 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612a033c820 con 0x56129ff724d0 2024-07-26T11:19:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.650+0000 7f2543e00640 1 --2- 172.21.15.134:0/1643923407 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff6a2f0 0x5612a0334cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:19:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.650+0000 7f2543400640 1 --2- 172.21.15.134:0/1643923407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff724d0 0x5612a0342a90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2530005f70 tx=0x7f2530031f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df01260d0cd15da4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:03.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.651+0000 7f2538c00640 1 -- 172.21.15.134:0/1643923407 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25300cb470 con 0x56129ff724d0 2024-07-26T11:19:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.651+0000 7f2538c00640 1 -- 172.21.15.134:0/1643923407 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25300cfa70 con 0x56129ff724d0 2024-07-26T11:19:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.651+0000 7f2538c00640 1 -- 172.21.15.134:0/1643923407 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25300cb470 con 0x56129ff724d0 2024-07-26T11:19:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.653+0000 7f2552080740 1 -- 172.21.15.134:0/1643923407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff724d0 msgr2=0x5612a0342a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:03.654 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.653+0000 7f2552080740 1 --2- 172.21.15.134:0/1643923407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff724d0 0x5612a0342a90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2530005f70 tx=0x7f2530031f10 comp rx=0 tx=0).stop 2024-07-26T11:19:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.653+0000 7f2552080740 1 -- 172.21.15.134:0/1643923407 shutdown_connections 2024-07-26T11:19:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.653+0000 7f2552080740 1 --2- 172.21.15.134:0/1643923407 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff724d0 0x5612a0342a90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.653+0000 7f2552080740 1 --2- 172.21.15.134:0/1643923407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff931b0 0x5612a03441e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.653+0000 7f2552080740 1 --2- 172.21.15.134:0/1643923407 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff6a2f0 0x5612a0334cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.653+0000 7f2552080740 1 -- 172.21.15.134:0/1643923407 >> 172.21.15.134:0/1643923407 conn(0x56129ff87300 msgr2=0x5612a0343ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.653+0000 7f2552080740 1 -- 172.21.15.134:0/1643923407 shutdown_connections 2024-07-26T11:19:03.655 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.653+0000 7f2552080740 1 -- 172.21.15.134:0/1643923407 wait complete. 2024-07-26T11:19:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.654+0000 7f2552080740 1 -- start start 2024-07-26T11:19:03.656 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.654+0000 7f2552080740 1 Processor -- start 2024-07-26T11:19:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.655+0000 7f2552080740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff6a2f0 0x5612a0345c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.655+0000 7f2552080740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff724d0 0x5612a0346140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.657 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.655+0000 7f2552080740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff931b0 0x5612a03bb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.655+0000 7f2552080740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f25441b7ff0 con 0x56129ff724d0 2024-07-26T11:19:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.655+0000 7f2552080740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f25441b7920 con 0x56129ff6a2f0 2024-07-26T11:19:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.655+0000 7f2552080740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f25441b4c10 con 0x56129ff931b0 2024-07-26T11:19:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.655+0000 7f2552080740 10 client.0 _ll_register_callbacks cb 0x7f2534016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.655+0000 7f2543e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff6a2f0 0x5612a0345c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.655+0000 7f2543e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff6a2f0 0x5612a0345c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:42136/0 (socket says 172.21.15.134:42136) 2024-07-26T11:19:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.655+0000 7f2543e00640 1 -- 172.21.15.134:0/68112243 learned_addr learned my addr 172.21.15.134:0/68112243 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:03.658 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.656+0000 7f2549000640 1 --2- 172.21.15.134:0/68112243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff931b0 0x5612a03bb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.659 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.656+0000 7f2543400640 1 --2- 172.21.15.134:0/68112243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff724d0 0x5612a0346140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.659 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.656+0000 7f2543e00640 1 -- 172.21.15.134:0/68112243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff931b0 msgr2=0x5612a03bb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.656+0000 7f2543e00640 1 --2- 172.21.15.134:0/68112243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff931b0 0x5612a03bb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.656+0000 7f2543e00640 1 -- 172.21.15.134:0/68112243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff724d0 msgr2=0x5612a0346140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.656+0000 7f2543e00640 1 --2- 172.21.15.134:0/68112243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff724d0 0x5612a0346140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.656+0000 7f2543e00640 1 -- 172.21.15.134:0/68112243 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612a03bb6b0 con 0x56129ff6a2f0 2024-07-26T11:19:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.657+0000 7f2543e00640 1 --2- 172.21.15.134:0/68112243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff6a2f0 0x5612a0345c00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2534011b60 tx=0x7f2534005370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b055038cf966efd2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.657+0000 7f252e000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:19:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.657+0000 7f252e000640 1 -- 172.21.15.134:0/68112243 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534023040 con 0x56129ff6a2f0 2024-07-26T11:19:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.657+0000 7f252e000640 1 -- 172.21.15.134:0/68112243 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2534011dd0 con 0x56129ff6a2f0 2024-07-26T11:19:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.657+0000 7f2552080740 1 -- 172.21.15.134:0/68112243 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25300cfa70 con 0x56129ff6a2f0 2024-07-26T11:19:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.658+0000 7f252e000640 1 -- 172.21.15.134:0/68112243 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f2534005bd0 con 0x56129ff6a2f0 2024-07-26T11:19:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.658+0000 7f2552080740 10 client.26045 fetch_fsmap learned FSMap version 31 2024-07-26T11:19:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.658+0000 7f2552080740 1 -- 172.21.15.134:0/68112243 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5612a03bc810 con 0x56129ff6a2f0 2024-07-26T11:19:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.658+0000 7f252e000640 1 -- 172.21.15.134:0/68112243 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2534004df0 con 0x56129ff6a2f0 2024-07-26T11:19:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.659+0000 7f252e000640 1 -- 172.21.15.134:0/68112243 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f253401fa30 con 0x56129ff6a2f0 2024-07-26T11:19:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.659+0000 7f2552080740 10 client.26045 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:19:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.659+0000 7f2552080740 10 client.26045 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:19:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.659+0000 7f2552080740 20 client.26045 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.659+0000 7f2552080740 1 --2- 172.21.15.134:0/68112243 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56129ff271c0 0x5612a03beff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.659+0000 7f2552080740 4 client.26045 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:19:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.659+0000 7f2552080740 1 -- 172.21.15.134:0/68112243 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f25300cfa70 con 0x56129ff271c0 2024-07-26T11:19:03.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.660+0000 7f2543400640 1 --2- 172.21.15.134:0/68112243 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56129ff271c0 0x5612a03beff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.661+0000 7f2543400640 1 --2- 172.21.15.134:0/68112243 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56129ff271c0 0x5612a03beff0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d481104d78ada22 server_cookie=58e802c86c819b7 in_seq=0 out_seq=0 2024-07-26T11:19:03.663 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:03.661+0000 7f252e000640 10 client.26045 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:04.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-26T11:19:04.580 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:04 smithi134 ceph-mon[31966]: pgmap v573: 129 pgs: 129 active+clean; 18 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 279 MiB/s wr, 1.41k op/s 2024-07-26T11:19:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:04 smithi189 ceph-mon[36790]: pgmap v573: 129 pgs: 129 active+clean; 18 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 279 MiB/s wr, 1.41k op/s 2024-07-26T11:19:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:04 smithi193 ceph-mon[37473]: pgmap v573: 129 pgs: 129 active+clean; 18 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 279 MiB/s wr, 1.41k op/s 2024-07-26T11:19:05.149 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.147+0000 7f252e000640 1 -- 172.21.15.134:0/68112243 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f25300cfa70 con 0x56129ff271c0 2024-07-26T11:19:05.149 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.147+0000 7f252e000640 10 client.26045 handle_command_reply: tid=0 2024-07-26T11:19:05.149 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.148+0000 7f2552080740 1 client.26045 shutdown 2024-07-26T11:19:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.148+0000 7f2552080740 1 -- 172.21.15.134:0/68112243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff6a2f0 msgr2=0x5612a0345c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:05.151 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.148+0000 7f2552080740 1 --2- 172.21.15.134:0/68112243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff6a2f0 0x5612a0345c00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2534011b60 tx=0x7f2534005370 comp rx=0 tx=0).stop 2024-07-26T11:19:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.151+0000 7f2552080740 1 -- 172.21.15.134:0/68112243 shutdown_connections 2024-07-26T11:19:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.151+0000 7f2552080740 1 --2- 172.21.15.134:0/68112243 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56129ff271c0 0x5612a03beff0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.151+0000 7f2552080740 1 --2- 172.21.15.134:0/68112243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56129ff931b0 0x5612a03bb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.151+0000 7f2552080740 1 --2- 172.21.15.134:0/68112243 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56129ff724d0 0x5612a0346140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.151+0000 7f2552080740 1 --2- 172.21.15.134:0/68112243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56129ff6a2f0 0x5612a0345c00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.151+0000 7f2552080740 1 -- 172.21.15.134:0/68112243 >> 172.21.15.134:0/68112243 conn(0x56129ff87300 msgr2=0x5612a0334a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.151+0000 7f252e000640 0 client.26045 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.152+0000 7f2552080740 1 -- 172.21.15.134:0/68112243 shutdown_connections 2024-07-26T11:19:05.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.152+0000 7f2552080740 1 -- 172.21.15.134:0/68112243 wait complete. 2024-07-26T11:19:05.156 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.154+0000 7f2552080740 20 client.26045 trim_cache size 0 max 16384 2024-07-26T11:19:05.159 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:19:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:19:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:19:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:19:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.158+0000 7f2549a00640 1 -- 172.21.15.134:0/1675880947 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f25280b3dc0 msgr2=0x7f25280b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.158+0000 7f2549a00640 1 --2- 172.21.15.134:0/1675880947 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f25280b3dc0 0x7f25280b62d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2534009b50 tx=0x7f2534005f90 comp rx=0 tx=0).stop 2024-07-26T11:19:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.158+0000 7f2549a00640 1 -- 172.21.15.134:0/1675880947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2544076dc0 msgr2=0x7f25441a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.158+0000 7f2549a00640 1 --2- 172.21.15.134:0/1675880947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2544076dc0 0x7f25441a0890 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2530005ab0 tx=0x7f2530003c80 comp rx=0 tx=0).stop 2024-07-26T11:19:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.161+0000 7f2549a00640 1 -- 172.21.15.134:0/1675880947 shutdown_connections 2024-07-26T11:19:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.161+0000 7f2549a00640 1 --2- 172.21.15.134:0/1675880947 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f25280b3dc0 0x7f25280b62d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.161+0000 7f2549a00640 1 --2- 172.21.15.134:0/1675880947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2544113c00 0x7f25441a5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.161+0000 7f2549a00640 1 --2- 172.21.15.134:0/1675880947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2544076dc0 0x7f25441a0890 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.161+0000 7f2549a00640 1 --2- 172.21.15.134:0/1675880947 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2544075800 0x7f25441a02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:05.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.161+0000 7f2549a00640 1 -- 172.21.15.134:0/1675880947 >> 172.21.15.134:0/1675880947 conn(0x7f25440fed60 msgr2=0x7f254410fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:05.164 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.162+0000 7f2549a00640 1 -- 172.21.15.134:0/1675880947 shutdown_connections 2024-07-26T11:19:05.164 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:05.162+0000 7f2549a00640 1 -- 172.21.15.134:0/1675880947 wait complete. 2024-07-26T11:19:05.383 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:19:06.312 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:06 smithi134 ceph-mon[31966]: pgmap v574: 129 pgs: 129 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 267 MiB/s wr, 1.36k op/s 2024-07-26T11:19:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:06 smithi189 ceph-mon[36790]: pgmap v574: 129 pgs: 129 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 267 MiB/s wr, 1.36k op/s 2024-07-26T11:19:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:06 smithi193 ceph-mon[37473]: pgmap v574: 129 pgs: 129 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 267 MiB/s wr, 1.36k op/s 2024-07-26T11:19:08.630 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:08 smithi134 ceph-mon[31966]: pgmap v575: 129 pgs: 129 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 239 MiB/s wr, 1.22k op/s 2024-07-26T11:19:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:08 smithi189 ceph-mon[36790]: pgmap v575: 129 pgs: 129 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 239 MiB/s wr, 1.22k op/s 2024-07-26T11:19:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:08 smithi193 ceph-mon[37473]: pgmap v575: 129 pgs: 129 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 239 MiB/s wr, 1.22k op/s 2024-07-26T11:19:10.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.418+0000 7ff797800640 1 -- 172.21.15.134:0/3008932786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff790109ec0 msgr2=0x7ff79010a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:10.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.419+0000 7ff797800640 1 --2- 172.21.15.134:0/3008932786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff790109ec0 0x7ff79010a2d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff788009cd0 tx=0x7ff78802a8e0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.419+0000 7ff797800640 1 -- 172.21.15.134:0/3008932786 shutdown_connections 2024-07-26T11:19:10.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.420+0000 7ff797800640 1 --2- 172.21.15.134:0/3008932786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff79010c2f0 0x7ff79010c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.420+0000 7ff797800640 1 --2- 172.21.15.134:0/3008932786 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff79010cc40 0x7ff7901166d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.420+0000 7ff797800640 1 --2- 172.21.15.134:0/3008932786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff790109ec0 0x7ff79010a2d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.420+0000 7ff797800640 1 -- 172.21.15.134:0/3008932786 >> 172.21.15.134:0/3008932786 conn(0x7ff790079100 msgr2=0x7ff79007b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:10.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.425+0000 7ff797800640 1 -- 172.21.15.134:0/3008932786 shutdown_connections 2024-07-26T11:19:10.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.426+0000 7ff797800640 1 -- 172.21.15.134:0/3008932786 wait complete. 2024-07-26T11:19:10.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.428+0000 7ff797800640 1 Processor -- start 2024-07-26T11:19:10.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.428+0000 7ff797800640 1 -- start start 2024-07-26T11:19:10.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.430+0000 7ff797800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff790109ec0 0x7ff7900856e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.430+0000 7ff797800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff79010c2f0 0x7ff79007e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.430+0000 7ff797800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79010cc40 0x7ff79007ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.430+0000 7ff794e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff790109ec0 0x7ff7900856e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:10.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.430+0000 7ff794e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff790109ec0 0x7ff7900856e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59424/0 (socket says 172.21.15.134:59424) 2024-07-26T11:19:10.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.431+0000 7ff794e00640 1 -- 172.21.15.134:0/298594306 learned_addr learned my addr 172.21.15.134:0/298594306 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:10.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.431+0000 7ff797800640 1 -- 172.21.15.134:0/298594306 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff790119be0 con 0x7ff79010c2f0 2024-07-26T11:19:10.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.431+0000 7ff797800640 1 -- 172.21.15.134:0/298594306 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff790119a60 con 0x7ff79010cc40 2024-07-26T11:19:10.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.431+0000 7ff797800640 1 -- 172.21.15.134:0/298594306 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff790119d60 con 0x7ff790109ec0 2024-07-26T11:19:10.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.431+0000 7ff78fe00640 1 --2- 172.21.15.134:0/298594306 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff79010c2f0 0x7ff79007e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:10.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.431+0000 7ff795800640 1 --2- 172.21.15.134:0/298594306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79010cc40 0x7ff79007ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:10.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.432+0000 7ff795800640 1 -- 172.21.15.134:0/298594306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff790109ec0 msgr2=0x7ff7900856e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:10.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.432+0000 7ff795800640 1 --2- 172.21.15.134:0/298594306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff790109ec0 0x7ff7900856e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.432+0000 7ff795800640 1 -- 172.21.15.134:0/298594306 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff79010c2f0 msgr2=0x7ff79007e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:10.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.432+0000 7ff795800640 1 --2- 172.21.15.134:0/298594306 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff79010c2f0 0x7ff79007e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.432+0000 7ff795800640 1 -- 172.21.15.134:0/298594306 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff79007f750 con 0x7ff79010cc40 2024-07-26T11:19:10.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.432+0000 7ff795800640 1 --2- 172.21.15.134:0/298594306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79010cc40 0x7ff79007ef30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff78000b410 tx=0x7ff78000b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:10.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.433+0000 7ff78d600640 1 -- 172.21.15.134:0/298594306 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff780015020 con 0x7ff79010cc40 2024-07-26T11:19:10.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.433+0000 7ff78d600640 1 -- 172.21.15.134:0/298594306 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff780011ce0 con 0x7ff79010cc40 2024-07-26T11:19:10.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.433+0000 7ff78d600640 1 -- 172.21.15.134:0/298594306 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff780010dc0 con 0x7ff79010cc40 2024-07-26T11:19:10.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.433+0000 7ff797800640 1 -- 172.21.15.134:0/298594306 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff790084410 con 0x7ff79010cc40 2024-07-26T11:19:10.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.434+0000 7ff797800640 1 -- 172.21.15.134:0/298594306 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff790084980 con 0x7ff79010cc40 2024-07-26T11:19:10.440 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:10 smithi134 ceph-mon[31966]: pgmap v576: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 320 MiB/s wr, 1.67k op/s 2024-07-26T11:19:10.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.441+0000 7ff78d600640 1 -- 172.21.15.134:0/298594306 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7ff780018020 con 0x7ff79010cc40 2024-07-26T11:19:10.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.444+0000 7ff78d600640 1 --2- 172.21.15.134:0/298594306 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff7740b3e40 0x7ff7740b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.445+0000 7ff78d600640 1 -- 172.21.15.134:0/298594306 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff7800e38f0 con 0x7ff79010cc40 2024-07-26T11:19:10.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.445+0000 7ff794e00640 1 --2- 172.21.15.134:0/298594306 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff7740b3e40 0x7ff7740b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:10.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.450+0000 7ff794e00640 1 --2- 172.21.15.134:0/298594306 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff7740b3e40 0x7ff7740b6350 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff7880094e0 tx=0x7ff788004a80 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:10.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.451+0000 7ff79e429740 1 Processor -- start 2024-07-26T11:19:10.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.451+0000 7ff79e429740 1 -- start start 2024-07-26T11:19:10.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.451+0000 7ff79e429740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d4a070 0x5639410eee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.451+0000 7ff79e429740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940d211b0 0x5639410ef390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.451+0000 7ff79e429740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d29390 0x5639410ef8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.451+0000 7ff79e429740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563941065bf0 con 0x563940d211b0 2024-07-26T11:19:10.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.451+0000 7ff79e429740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5639410550f0 con 0x563940d29390 2024-07-26T11:19:10.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.451+0000 7ff79e429740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563941059130 con 0x563940d4a070 2024-07-26T11:19:10.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.452+0000 7ff795800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d29390 0x5639410ef8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:10.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.452+0000 7ff795800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d29390 0x5639410ef8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56362/0 (socket says 172.21.15.134:56362) 2024-07-26T11:19:10.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.452+0000 7ff795800640 1 -- 172.21.15.134:0/2444411679 learned_addr learned my addr 172.21.15.134:0/2444411679 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:10.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.452+0000 7ff795800640 1 -- 172.21.15.134:0/2444411679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d4a070 msgr2=0x5639410eee50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:19:10.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.452+0000 7ff795800640 1 --2- 172.21.15.134:0/2444411679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d4a070 0x5639410eee50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.452+0000 7ff795800640 1 -- 172.21.15.134:0/2444411679 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940d211b0 msgr2=0x5639410ef390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:10.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.453+0000 7ff795800640 1 --2- 172.21.15.134:0/2444411679 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940d211b0 0x5639410ef390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.453+0000 7ff795800640 1 -- 172.21.15.134:0/2444411679 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563940db5650 con 0x563940d29390 2024-07-26T11:19:10.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.453+0000 7ff795800640 1 --2- 172.21.15.134:0/2444411679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d29390 0x5639410ef8d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff7800aa1c0 tx=0x7ff7800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d98f7ced507f571 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:10.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.453+0000 7ff786000640 1 -- 172.21.15.134:0/2444411679 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7800c4070 con 0x563940d29390 2024-07-26T11:19:10.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.453+0000 7ff786000640 1 -- 172.21.15.134:0/2444411679 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7800aa230 con 0x563940d29390 2024-07-26T11:19:10.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.453+0000 7ff786000640 1 -- 172.21.15.134:0/2444411679 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7800b1bb0 con 0x563940d29390 2024-07-26T11:19:10.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.456+0000 7ff79e429740 1 -- 172.21.15.134:0/2444411679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d29390 msgr2=0x5639410ef8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:10.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.456+0000 7ff79e429740 1 --2- 172.21.15.134:0/2444411679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d29390 0x5639410ef8d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff7800aa1c0 tx=0x7ff7800046c0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.458+0000 7ff79e429740 1 -- 172.21.15.134:0/2444411679 shutdown_connections 2024-07-26T11:19:10.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.458+0000 7ff79e429740 1 --2- 172.21.15.134:0/2444411679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d29390 0x5639410ef8d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.458+0000 7ff79e429740 1 --2- 172.21.15.134:0/2444411679 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940d211b0 0x5639410ef390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.458+0000 7ff79e429740 1 --2- 172.21.15.134:0/2444411679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d4a070 0x5639410eee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.458+0000 7ff79e429740 1 -- 172.21.15.134:0/2444411679 >> 172.21.15.134:0/2444411679 conn(0x563940d3e460 msgr2=0x5639410ecd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:10.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.459+0000 7ff79e429740 1 -- 172.21.15.134:0/2444411679 shutdown_connections 2024-07-26T11:19:10.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.461+0000 7ff79e429740 1 -- 172.21.15.134:0/2444411679 wait complete. 2024-07-26T11:19:10.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.462+0000 7ff79e429740 1 -- start start 2024-07-26T11:19:10.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.463+0000 7ff79e429740 1 Processor -- start 2024-07-26T11:19:10.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.468+0000 7ff79e429740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940d211b0 0x5639410fa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.471+0000 7ff79e429740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d29390 0x5639410fa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.471+0000 7ff79e429740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d4a070 0x5639410faf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.471+0000 7ff79e429740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563940eb8700 con 0x563940d211b0 2024-07-26T11:19:10.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.471+0000 7ff79e429740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563940ff16c0 con 0x563940d4a070 2024-07-26T11:19:10.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.471+0000 7ff79e429740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5639410fd4b0 con 0x563940d29390 2024-07-26T11:19:10.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.471+0000 7ff79e429740 10 client.0 _ll_register_callbacks cb 0x563941001170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:10.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.471+0000 7ff78fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d29390 0x5639410fa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:10.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.472+0000 7ff78fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d29390 0x5639410fa9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59432/0 (socket says 172.21.15.134:59432) 2024-07-26T11:19:10.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.472+0000 7ff78fe00640 1 -- 172.21.15.134:0/1523902995 learned_addr learned my addr 172.21.15.134:0/1523902995 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:10.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.472+0000 7ff794e00640 1 --2- 172.21.15.134:0/1523902995 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940d211b0 0x5639410fa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:10.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.472+0000 7ff795800640 1 --2- 172.21.15.134:0/1523902995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d4a070 0x5639410faf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:10.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.473+0000 7ff78fe00640 1 -- 172.21.15.134:0/1523902995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d4a070 msgr2=0x5639410faf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:10.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.473+0000 7ff78fe00640 1 --2- 172.21.15.134:0/1523902995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d4a070 0x5639410faf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.473+0000 7ff78fe00640 1 -- 172.21.15.134:0/1523902995 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940d211b0 msgr2=0x5639410fa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:10.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.473+0000 7ff78fe00640 1 --2- 172.21.15.134:0/1523902995 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940d211b0 0x5639410fa4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.473+0000 7ff78fe00640 1 -- 172.21.15.134:0/1523902995 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563941008860 con 0x563940d29390 2024-07-26T11:19:10.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.474+0000 7ff795800640 1 --2- 172.21.15.134:0/1523902995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d4a070 0x5639410faf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:19:10.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.474+0000 7ff794e00640 1 --2- 172.21.15.134:0/1523902995 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940d211b0 0x5639410fa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:19:10.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.475+0000 7ff78fe00640 1 --2- 172.21.15.134:0/1523902995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d29390 0x5639410fa9f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff77c00bd10 tx=0x7ff77c00f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9809750849a19574 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:10.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.475+0000 7ff77aa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:19:10.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.475+0000 7ff77aa00640 1 -- 172.21.15.134:0/1523902995 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c015070 con 0x563940d29390 2024-07-26T11:19:10.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.476+0000 7ff79e429740 1 -- 172.21.15.134:0/1523902995 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7800aa230 con 0x563940d29390 2024-07-26T11:19:10.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.476+0000 7ff77aa00640 1 -- 172.21.15.134:0/1523902995 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff77c011650 con 0x563940d29390 2024-07-26T11:19:10.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.477+0000 7ff77aa00640 1 -- 172.21.15.134:0/1523902995 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff77c011850 con 0x563940d29390 2024-07-26T11:19:10.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.477+0000 7ff79e429740 10 client.26047 fetch_fsmap learned FSMap version 31 2024-07-26T11:19:10.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.477+0000 7ff79e429740 1 -- 172.21.15.134:0/1523902995 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563941109190 con 0x563940d29390 2024-07-26T11:19:10.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.478+0000 7ff77aa00640 1 -- 172.21.15.134:0/1523902995 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7800aa230 con 0x563940d29390 2024-07-26T11:19:10.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.478+0000 7ff77aa00640 1 -- 172.21.15.134:0/1523902995 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7ff77c01ab60 con 0x563940d29390 2024-07-26T11:19:10.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.478+0000 7ff79e429740 10 client.26047 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:19:10.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.478+0000 7ff79e429740 10 client.26047 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T11:19:10.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.478+0000 7ff79e429740 20 client.26047 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:10.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.478+0000 7ff79e429740 1 --2- 172.21.15.134:0/1523902995 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x563940d089f0 0x56394110c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.478+0000 7ff79e429740 4 client.26047 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:19:10.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.478+0000 7ff79e429740 1 -- 172.21.15.134:0/1523902995 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563941067b40 con 0x563940d089f0 2024-07-26T11:19:10.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.485+0000 7ff794e00640 1 --2- 172.21.15.134:0/1523902995 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x563940d089f0 0x56394110c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:10.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.487+0000 7ff794e00640 1 --2- 172.21.15.134:0/1523902995 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x563940d089f0 0x56394110c5b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4b10065f65e609b3 server_cookie=de5dd07cc8d1afb7 in_seq=0 out_seq=0 2024-07-26T11:19:10.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.487+0000 7ff77aa00640 10 client.26047 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:19:10.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.490+0000 7ff77aa00640 1 -- 172.21.15.134:0/1523902995 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563941067b40 con 0x563940d089f0 2024-07-26T11:19:10.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.490+0000 7ff77aa00640 10 client.26047 handle_command_reply: tid=0 2024-07-26T11:19:10.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.490+0000 7ff79e429740 1 client.26047 shutdown 2024-07-26T11:19:10.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.491+0000 7ff79e429740 1 -- 172.21.15.134:0/1523902995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d29390 msgr2=0x5639410fa9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:10.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.491+0000 7ff79e429740 1 --2- 172.21.15.134:0/1523902995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d29390 0x5639410fa9f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff77c00bd10 tx=0x7ff77c00f730 comp rx=0 tx=0).stop 2024-07-26T11:19:10.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.491+0000 7ff79e429740 1 -- 172.21.15.134:0/1523902995 shutdown_connections 2024-07-26T11:19:10.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.491+0000 7ff79e429740 1 --2- 172.21.15.134:0/1523902995 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x563940d089f0 0x56394110c5b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.491+0000 7ff79e429740 1 --2- 172.21.15.134:0/1523902995 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d29390 0x5639410fa9f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.491+0000 7ff79e429740 1 --2- 172.21.15.134:0/1523902995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d4a070 0x5639410faf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.491+0000 7ff79e429740 1 --2- 172.21.15.134:0/1523902995 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940d211b0 0x5639410fa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.491+0000 7ff79e429740 1 -- 172.21.15.134:0/1523902995 >> 172.21.15.134:0/1523902995 conn(0x563940d3e460 msgr2=0x5639410f1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:10.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.491+0000 7ff77aa00640 0 client.26047 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:19:10.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.492+0000 7ff79e429740 1 -- 172.21.15.134:0/1523902995 shutdown_connections 2024-07-26T11:19:10.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.492+0000 7ff79e429740 1 -- 172.21.15.134:0/1523902995 wait complete. 2024-07-26T11:19:10.495 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.492+0000 7ff79e429740 20 client.26047 trim_cache size 0 max 16384 2024-07-26T11:19:10.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.520+0000 7ff79e429740 1 Processor -- start 2024-07-26T11:19:10.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.520+0000 7ff79e429740 1 -- start start 2024-07-26T11:19:10.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.522+0000 7ff79e429740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d211b0 0x5639410fae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.522+0000 7ff79e429740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d4a070 0x5639410f0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.522+0000 7ff79e429740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940cddda0 0x563941111d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.522+0000 7ff79e429740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff7901bf980 con 0x563940cddda0 2024-07-26T11:19:10.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.522+0000 7ff79e429740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff7901c14f0 con 0x563940d4a070 2024-07-26T11:19:10.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.522+0000 7ff79e429740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff7901c0050 con 0x563940d211b0 2024-07-26T11:19:10.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.522+0000 7ff794e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d211b0 0x5639410fae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:10.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.522+0000 7ff794e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d211b0 0x5639410fae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59434/0 (socket says 172.21.15.134:59434) 2024-07-26T11:19:10.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.522+0000 7ff794e00640 1 -- 172.21.15.134:0/4015453220 learned_addr learned my addr 172.21.15.134:0/4015453220 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:10.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.523+0000 7ff78fe00640 1 --2- 172.21.15.134:0/4015453220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d4a070 0x5639410f0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:10.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.523+0000 7ff794e00640 1 -- 172.21.15.134:0/4015453220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d4a070 msgr2=0x5639410f0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:10.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.523+0000 7ff794e00640 1 --2- 172.21.15.134:0/4015453220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d4a070 0x5639410f0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.523+0000 7ff794e00640 1 -- 172.21.15.134:0/4015453220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940cddda0 msgr2=0x563941111d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:10.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.523+0000 7ff794e00640 1 --2- 172.21.15.134:0/4015453220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940cddda0 0x563941111d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.523+0000 7ff794e00640 1 -- 172.21.15.134:0/4015453220 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639410efc60 con 0x563940d211b0 2024-07-26T11:19:10.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.524+0000 7ff794e00640 1 --2- 172.21.15.134:0/4015453220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d211b0 0x5639410fae70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff788000f80 tx=0x7ff788041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6aac899b204bcd78 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:10.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.524+0000 7ff785600640 1 -- 172.21.15.134:0/4015453220 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff788030b30 con 0x563940d211b0 2024-07-26T11:19:10.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.524+0000 7ff785600640 1 -- 172.21.15.134:0/4015453220 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff788055070 con 0x563940d211b0 2024-07-26T11:19:10.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.524+0000 7ff785600640 1 -- 172.21.15.134:0/4015453220 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff788047410 con 0x563940d211b0 2024-07-26T11:19:10.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.526+0000 7ff79e429740 1 -- 172.21.15.134:0/4015453220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d211b0 msgr2=0x5639410fae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:10.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.526+0000 7ff79e429740 1 --2- 172.21.15.134:0/4015453220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d211b0 0x5639410fae70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff788000f80 tx=0x7ff788041000 comp rx=0 tx=0).stop 2024-07-26T11:19:10.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.526+0000 7ff79e429740 1 -- 172.21.15.134:0/4015453220 shutdown_connections 2024-07-26T11:19:10.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.526+0000 7ff79e429740 1 --2- 172.21.15.134:0/4015453220 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940cddda0 0x563941111d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.528 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.526+0000 7ff79e429740 1 --2- 172.21.15.134:0/4015453220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940d4a070 0x5639410f0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.526+0000 7ff79e429740 1 --2- 172.21.15.134:0/4015453220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563940d211b0 0x5639410fae70 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.526+0000 7ff79e429740 1 -- 172.21.15.134:0/4015453220 >> 172.21.15.134:0/4015453220 conn(0x563940d3e460 msgr2=0x5639410efe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:10.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.526+0000 7ff79e429740 1 -- 172.21.15.134:0/4015453220 shutdown_connections 2024-07-26T11:19:10.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.527+0000 7ff79e429740 1 -- 172.21.15.134:0/4015453220 wait complete. 2024-07-26T11:19:10.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.528+0000 7ff79e429740 1 -- start start 2024-07-26T11:19:10.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.528+0000 7ff79e429740 1 Processor -- start 2024-07-26T11:19:10.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.529+0000 7ff79e429740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940cddda0 0x563941102e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.529+0000 7ff79e429740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940d4a070 0x5639410fe310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.529+0000 7ff79e429740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639410fe850 0x5639410fecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.529+0000 7ff794e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940cddda0 0x563941102e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:10.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.529+0000 7ff794e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940cddda0 0x563941102e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56374/0 (socket says 172.21.15.134:56374) 2024-07-26T11:19:10.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.531+0000 7ff79e429740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff7901c0df0 con 0x563940d4a070 2024-07-26T11:19:10.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.531+0000 7ff79e429740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff7901c0720 con 0x563940cddda0 2024-07-26T11:19:10.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.531+0000 7ff79e429740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff7901bd960 con 0x5639410fe850 2024-07-26T11:19:10.534 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.531+0000 7ff79e429740 10 client.0 _ll_register_callbacks cb 0x7ff77c00b8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:10.535 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.532+0000 7ff795800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639410fe850 0x5639410fecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:10.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.532+0000 7ff795800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639410fe850 0x5639410fecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59444/0 (socket says 172.21.15.134:59444) 2024-07-26T11:19:10.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.532+0000 7ff795800640 1 -- 172.21.15.134:0/1649152317 learned_addr learned my addr 172.21.15.134:0/1649152317 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:10.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.532+0000 7ff78fe00640 1 --2- 172.21.15.134:0/1649152317 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940d4a070 0x5639410fe310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:10.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.533+0000 7ff78fe00640 1 -- 172.21.15.134:0/1649152317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639410fe850 msgr2=0x5639410fecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:10.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.533+0000 7ff78fe00640 1 --2- 172.21.15.134:0/1649152317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639410fe850 0x5639410fecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.533+0000 7ff78fe00640 1 -- 172.21.15.134:0/1649152317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940cddda0 msgr2=0x563941102e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:10.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.533+0000 7ff78fe00640 1 --2- 172.21.15.134:0/1649152317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940cddda0 0x563941102e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.533+0000 7ff78fe00640 1 -- 172.21.15.134:0/1649152317 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563941173e20 con 0x563940d4a070 2024-07-26T11:19:10.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.535+0000 7ff77aa00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:19:10.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.535+0000 7ff78fe00640 1 --2- 172.21.15.134:0/1649152317 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940d4a070 0x5639410fe310 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff77c024510 tx=0x7ff77c00b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a1df45ac0d6e2e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:10.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.535+0000 7ff794e00640 1 --2- 172.21.15.134:0/1649152317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940cddda0 0x563941102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-07-26T11:19:10.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.536+0000 7ff77aa00640 1 -- 172.21.15.134:0/1649152317 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c032020 con 0x563940d4a070 2024-07-26T11:19:10.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.536+0000 7ff77aa00640 1 -- 172.21.15.134:0/1649152317 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff77c024a10 con 0x563940d4a070 2024-07-26T11:19:10.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.536+0000 7ff79e429740 1 -- 172.21.15.134:0/1649152317 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff788055070 con 0x563940d4a070 2024-07-26T11:19:10.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.537+0000 7ff77aa00640 1 -- 172.21.15.134:0/1649152317 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff77c02a040 con 0x563940d4a070 2024-07-26T11:19:10.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.537+0000 7ff79e429740 10 client.15744 fetch_fsmap learned FSMap version 31 2024-07-26T11:19:10.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.537+0000 7ff79e429740 1 -- 172.21.15.134:0/1649152317 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563941175100 con 0x563940d4a070 2024-07-26T11:19:10.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.538+0000 7ff77aa00640 1 -- 172.21.15.134:0/1649152317 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff788055070 con 0x563940d4a070 2024-07-26T11:19:10.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.539+0000 7ff77aa00640 1 -- 172.21.15.134:0/1649152317 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7ff77c035020 con 0x563940d4a070 2024-07-26T11:19:10.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.540+0000 7ff79e429740 10 client.15744 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:19:10.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.540+0000 7ff79e429740 10 client.15744 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T11:19:10.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.540+0000 7ff79e429740 20 client.15744 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:10.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.540+0000 7ff79e429740 1 --2- 172.21.15.134:0/1649152317 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x563941175870 0x563941177c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:10.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.540+0000 7ff79e429740 4 client.15744 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:19:10.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.540+0000 7ff79e429740 1 -- 172.21.15.134:0/1649152317 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5639410635e0 con 0x563941175870 2024-07-26T11:19:10.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.540+0000 7ff794e00640 1 --2- 172.21.15.134:0/1649152317 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x563941175870 0x563941177c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:10.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.541+0000 7ff77aa00640 10 client.15744 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:19:10.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.541+0000 7ff794e00640 1 --2- 172.21.15.134:0/1649152317 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x563941175870 0x563941177c90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=27f810917e2feadb server_cookie=ba9088f663e08750 in_seq=0 out_seq=0 2024-07-26T11:19:10.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.541+0000 7ff77aa00640 1 -- 172.21.15.134:0/1649152317 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5639410635e0 con 0x563941175870 2024-07-26T11:19:10.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.541+0000 7ff77aa00640 10 client.15744 handle_command_reply: tid=0 2024-07-26T11:19:10.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.541+0000 7ff79e429740 1 client.15744 shutdown 2024-07-26T11:19:10.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.542+0000 7ff79e429740 1 -- 172.21.15.134:0/1649152317 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940d4a070 msgr2=0x5639410fe310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:10.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.542+0000 7ff79e429740 1 --2- 172.21.15.134:0/1649152317 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940d4a070 0x5639410fe310 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff77c024510 tx=0x7ff77c00b760 comp rx=0 tx=0).stop 2024-07-26T11:19:10.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.542+0000 7ff79e429740 1 -- 172.21.15.134:0/1649152317 shutdown_connections 2024-07-26T11:19:10.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.542+0000 7ff79e429740 1 --2- 172.21.15.134:0/1649152317 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x563941175870 0x563941177c90 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.542+0000 7ff79e429740 1 --2- 172.21.15.134:0/1649152317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5639410fe850 0x5639410fecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.542+0000 7ff79e429740 1 --2- 172.21.15.134:0/1649152317 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563940d4a070 0x5639410fe310 secure :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff77c024510 tx=0x7ff77c00b760 comp rx=0 tx=0).stop 2024-07-26T11:19:10.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.542+0000 7ff79e429740 1 --2- 172.21.15.134:0/1649152317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563940cddda0 0x563941102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.542+0000 7ff79e429740 1 -- 172.21.15.134:0/1649152317 >> 172.21.15.134:0/1649152317 conn(0x563940d3e460 msgr2=0x5639410fac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:10.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.543+0000 7ff77aa00640 0 client.15744 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:19:10.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.543+0000 7ff79e429740 1 -- 172.21.15.134:0/1649152317 shutdown_connections 2024-07-26T11:19:10.546 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.544+0000 7ff79e429740 1 -- 172.21.15.134:0/1649152317 wait complete. 2024-07-26T11:19:10.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.544+0000 7ff79e429740 20 client.15744 trim_cache size 0 max 16384 2024-07-26T11:19:10.550 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:19:10.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.548+0000 7ff797800640 1 -- 172.21.15.134:0/298594306 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff7740b3e40 msgr2=0x7ff7740b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:10.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.548+0000 7ff797800640 1 --2- 172.21.15.134:0/298594306 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff7740b3e40 0x7ff7740b6350 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff7880094e0 tx=0x7ff788004a80 comp rx=0 tx=0).stop 2024-07-26T11:19:10.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.548+0000 7ff797800640 1 -- 172.21.15.134:0/298594306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79010cc40 msgr2=0x7ff79007ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:10.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.548+0000 7ff797800640 1 --2- 172.21.15.134:0/298594306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79010cc40 0x7ff79007ef30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff78000b410 tx=0x7ff78000b8d0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.548+0000 7ff797800640 1 -- 172.21.15.134:0/298594306 shutdown_connections 2024-07-26T11:19:10.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.548+0000 7ff797800640 1 --2- 172.21.15.134:0/298594306 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff7740b3e40 0x7ff7740b6350 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.548+0000 7ff797800640 1 --2- 172.21.15.134:0/298594306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff79010cc40 0x7ff79007ef30 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.548+0000 7ff797800640 1 --2- 172.21.15.134:0/298594306 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff79010c2f0 0x7ff79007e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.548+0000 7ff797800640 1 --2- 172.21.15.134:0/298594306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff790109ec0 0x7ff7900856e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:10.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.548+0000 7ff797800640 1 -- 172.21.15.134:0/298594306 >> 172.21.15.134:0/298594306 conn(0x7ff790079100 msgr2=0x7ff79010eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:10.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.549+0000 7ff797800640 1 -- 172.21.15.134:0/298594306 shutdown_connections 2024-07-26T11:19:10.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:10.549+0000 7ff797800640 1 -- 172.21.15.134:0/298594306 wait complete. 2024-07-26T11:19:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:10 smithi189 ceph-mon[36790]: pgmap v576: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 320 MiB/s wr, 1.67k op/s 2024-07-26T11:19:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:10 smithi193 ceph-mon[37473]: pgmap v576: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 320 MiB/s wr, 1.67k op/s 2024-07-26T11:19:11.603 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-26T11:19:12.461 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:19:12.656 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:12 smithi134 ceph-mon[31966]: pgmap v577: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 228 MiB/s wr, 1.23k op/s 2024-07-26T11:19:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:12 smithi189 ceph-mon[36790]: pgmap v577: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 228 MiB/s wr, 1.23k op/s 2024-07-26T11:19:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:12 smithi193 ceph-mon[37473]: pgmap v577: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 228 MiB/s wr, 1.23k op/s 2024-07-26T11:19:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:14 smithi134 ceph-mon[31966]: pgmap v578: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 277 MiB/s wr, 1.62k op/s 2024-07-26T11:19:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:14 smithi189 ceph-mon[36790]: pgmap v578: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 277 MiB/s wr, 1.62k op/s 2024-07-26T11:19:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:14 smithi193 ceph-mon[37473]: pgmap v578: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 277 MiB/s wr, 1.62k op/s 2024-07-26T11:19:16.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:16 smithi134 ceph-mon[31966]: pgmap v579: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 249 MiB/s wr, 1.59k op/s 2024-07-26T11:19:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:16 smithi189 ceph-mon[36790]: pgmap v579: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 249 MiB/s wr, 1.59k op/s 2024-07-26T11:19:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:16 smithi193 ceph-mon[37473]: pgmap v579: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 249 MiB/s wr, 1.59k op/s 2024-07-26T11:19:16.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.957+0000 7fcde0e00640 1 -- 172.21.15.134:0/2312274638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcddc109df0 msgr2=0x7fcddc10a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:16.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.957+0000 7fcde0e00640 1 --2- 172.21.15.134:0/2312274638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcddc109df0 0x7fcddc10a200 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc009860 tx=0x7fcdcc02a360 comp rx=0 tx=0).stop 2024-07-26T11:19:16.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.958+0000 7fcde0e00640 1 -- 172.21.15.134:0/2312274638 shutdown_connections 2024-07-26T11:19:16.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.958+0000 7fcde0e00640 1 --2- 172.21.15.134:0/2312274638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcddc109df0 0x7fcddc10a200 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:16.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.958+0000 7fcde0e00640 1 --2- 172.21.15.134:0/2312274638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcddc10a740 0x7fcddc1169f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:16.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.958+0000 7fcde0e00640 1 --2- 172.21.15.134:0/2312274638 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcddc10eb80 0x7fcddc10ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:16.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.958+0000 7fcde0e00640 1 -- 172.21.15.134:0/2312274638 >> 172.21.15.134:0/2312274638 conn(0x7fcddc070d20 msgr2=0x7fcddc073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:16.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.963+0000 7fcde0e00640 1 -- 172.21.15.134:0/2312274638 shutdown_connections 2024-07-26T11:19:16.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.963+0000 7fcde0e00640 1 -- 172.21.15.134:0/2312274638 wait complete. 2024-07-26T11:19:16.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.966+0000 7fcde0e00640 1 Processor -- start 2024-07-26T11:19:16.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.967+0000 7fcde0e00640 1 -- start start 2024-07-26T11:19:16.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.970+0000 7fcde0e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcddc109df0 0x7fcddc114750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:16.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.970+0000 7fcde0e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcddc10a740 0x7fcddc110a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:16.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.970+0000 7fcde0e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcddc111060 0x7fcddc1b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:16.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.971+0000 7fcdd9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcddc109df0 0x7fcddc114750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:16.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.971+0000 7fcdd9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcddc109df0 0x7fcddc114750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59472/0 (socket says 172.21.15.134:59472) 2024-07-26T11:19:16.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.971+0000 7fcdd3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcddc10a740 0x7fcddc110a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:16.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.972+0000 7fcdd3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcddc10a740 0x7fcddc110a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39250/0 (socket says 172.21.15.134:39250) 2024-07-26T11:19:16.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.975+0000 7fcde0e00640 1 -- 172.21.15.134:0/1251076421 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc11a000 con 0x7fcddc10a740 2024-07-26T11:19:16.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.975+0000 7fcde0e00640 1 -- 172.21.15.134:0/1251076421 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc119e80 con 0x7fcddc111060 2024-07-26T11:19:16.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.975+0000 7fcde0e00640 1 -- 172.21.15.134:0/1251076421 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc11a180 con 0x7fcddc109df0 2024-07-26T11:19:16.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.972+0000 7fcdd9e00640 1 -- 172.21.15.134:0/1251076421 learned_addr learned my addr 172.21.15.134:0/1251076421 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:16.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.972+0000 7fcdda800640 1 --2- 172.21.15.134:0/1251076421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcddc111060 0x7fcddc1b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:16.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.976+0000 7fcdd9e00640 1 -- 172.21.15.134:0/1251076421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcddc111060 msgr2=0x7fcddc1b4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:16.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.976+0000 7fcdd9e00640 1 --2- 172.21.15.134:0/1251076421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcddc111060 0x7fcddc1b4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:16.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.976+0000 7fcdd9e00640 1 -- 172.21.15.134:0/1251076421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcddc10a740 msgr2=0x7fcddc110a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:16.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.976+0000 7fcdd9e00640 1 --2- 172.21.15.134:0/1251076421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcddc10a740 0x7fcddc110a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:16.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.977+0000 7fcdd9e00640 1 -- 172.21.15.134:0/1251076421 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcddc111840 con 0x7fcddc109df0 2024-07-26T11:19:16.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.979+0000 7fcdd9e00640 1 --2- 172.21.15.134:0/1251076421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcddc109df0 0x7fcddc114750 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400b3e0 tx=0x7fcdc400b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:16.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.979+0000 7fcdd2a00640 1 -- 172.21.15.134:0/1251076421 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4015020 con 0x7fcddc109df0 2024-07-26T11:19:16.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.979+0000 7fcdd2a00640 1 -- 172.21.15.134:0/1251076421 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdc4011ce0 con 0x7fcddc109df0 2024-07-26T11:19:16.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.979+0000 7fcdd2a00640 1 -- 172.21.15.134:0/1251076421 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4010de0 con 0x7fcddc109df0 2024-07-26T11:19:16.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.980+0000 7fcde0e00640 1 -- 172.21.15.134:0/1251076421 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcddc1b5650 con 0x7fcddc109df0 2024-07-26T11:19:16.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.980+0000 7fcde0e00640 1 -- 172.21.15.134:0/1251076421 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcddc1b5b10 con 0x7fcddc109df0 2024-07-26T11:19:16.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.988+0000 7fcde7a3a740 1 Processor -- start 2024-07-26T11:19:16.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.988+0000 7fcde7a3a740 1 -- start start 2024-07-26T11:19:16.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.989+0000 7fcde7a3a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56934070 0x559f56cd8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:16.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.990+0000 7fcde7a3a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f5690b1b0 0x559f56cd9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:16.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.990+0000 7fcdda800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56934070 0x559f56cd8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:16.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.990+0000 7fcdda800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56934070 0x559f56cd8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59476/0 (socket says 172.21.15.134:59476) 2024-07-26T11:19:16.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.990+0000 7fcdda800640 1 -- 172.21.15.134:0/1169879032 learned_addr learned my addr 172.21.15.134:0/1169879032 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:16.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.990+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/1169879032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f56913390 0x559f56cd98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:16.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.990+0000 7fcde7a3a740 1 -- 172.21.15.134:0/1169879032 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559f56c4fbd0 con 0x559f56913390 2024-07-26T11:19:16.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.990+0000 7fcde7a3a740 1 -- 172.21.15.134:0/1169879032 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559f56c3f0d0 con 0x559f5690b1b0 2024-07-26T11:19:16.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.990+0000 7fcde7a3a740 1 -- 172.21.15.134:0/1169879032 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559f56c43110 con 0x559f56934070 2024-07-26T11:19:16.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.991+0000 7fcdd3e00640 1 --2- 172.21.15.134:0/1169879032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f56913390 0x559f56cd98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:16.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.992+0000 7fcdda800640 1 -- 172.21.15.134:0/1169879032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f5690b1b0 msgr2=0x559f56cd9370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:19:16.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.992+0000 7fcdda800640 1 --2- 172.21.15.134:0/1169879032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f5690b1b0 0x559f56cd9370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:16.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.992+0000 7fcdda800640 1 -- 172.21.15.134:0/1169879032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f56913390 msgr2=0x559f56cd98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:16.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.992+0000 7fcdd2a00640 1 -- 172.21.15.134:0/1251076421 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fcdc4018020 con 0x7fcddc109df0 2024-07-26T11:19:16.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.992+0000 7fcdda800640 1 --2- 172.21.15.134:0/1169879032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f56913390 0x559f56cd98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:16.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.992+0000 7fcdda800640 1 -- 172.21.15.134:0/1169879032 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f5699f650 con 0x559f56934070 2024-07-26T11:19:16.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.994+0000 7fcdda800640 1 --2- 172.21.15.134:0/1169879032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56934070 0x559f56cd8e30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400e580 tx=0x7fcdd40095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=765bd4bb3469325a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:16.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.994+0000 7fcdd2a00640 1 --2- 172.21.15.134:0/1251076421 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcdc00b3f50 0x7fcdc00b6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:16.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.995+0000 7fcdcb400640 1 -- 172.21.15.134:0/1169879032 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4003970 con 0x559f56934070 2024-07-26T11:19:16.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.995+0000 7fcdd2a00640 1 -- 172.21.15.134:0/1251076421 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fcdc40e3920 con 0x7fcddc109df0 2024-07-26T11:19:16.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.995+0000 7fcdcb400640 1 -- 172.21.15.134:0/1169879032 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdd400d830 con 0x559f56934070 2024-07-26T11:19:16.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.996+0000 7fcdcb400640 1 -- 172.21.15.134:0/1169879032 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4003970 con 0x559f56934070 2024-07-26T11:19:16.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.997+0000 7fcdd9e00640 1 --2- 172.21.15.134:0/1251076421 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcdc00b3f50 0x7fcdc00b6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:16.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.997+0000 7fcde7a3a740 1 -- 172.21.15.134:0/1169879032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56934070 msgr2=0x559f56cd8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:17.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.997+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/1169879032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56934070 0x559f56cd8e30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400e580 tx=0x7fcdd40095c0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.998+0000 7fcdd9e00640 1 --2- 172.21.15.134:0/1251076421 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcdc00b3f50 0x7fcdc00b6460 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x559f56cd7ab0 tx=0x7fcdc400bf50 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:17.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.998+0000 7fcde7a3a740 1 -- 172.21.15.134:0/1169879032 shutdown_connections 2024-07-26T11:19:17.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.998+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/1169879032 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f56913390 0x559f56cd98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.998+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/1169879032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f5690b1b0 0x559f56cd9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.998+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/1169879032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56934070 0x559f56cd8e30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.998+0000 7fcde7a3a740 1 -- 172.21.15.134:0/1169879032 >> 172.21.15.134:0/1169879032 conn(0x559f56928460 msgr2=0x559f56cd6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:17.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.998+0000 7fcde7a3a740 1 -- 172.21.15.134:0/1169879032 shutdown_connections 2024-07-26T11:19:17.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.999+0000 7fcde7a3a740 1 -- 172.21.15.134:0/1169879032 wait complete. 2024-07-26T11:19:17.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:16.999+0000 7fcde7a3a740 1 -- start start 2024-07-26T11:19:17.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.000+0000 7fcde7a3a740 1 Processor -- start 2024-07-26T11:19:17.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.000+0000 7fcde7a3a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f5690b1b0 0x559f56ce4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:17.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.001+0000 7fcde7a3a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56913390 0x559f56ce4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:17.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.001+0000 7fcde7a3a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f56934070 0x559f56ce4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:17.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.001+0000 7fcde7a3a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559f56aa2700 con 0x559f56934070 2024-07-26T11:19:17.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.001+0000 7fcde7a3a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559f56bdb6a0 con 0x559f5690b1b0 2024-07-26T11:19:17.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.001+0000 7fcde7a3a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559f56ce7660 con 0x559f56913390 2024-07-26T11:19:17.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.001+0000 7fcde7a3a740 10 client.0 _ll_register_callbacks cb 0x559f56beb150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:17.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.001+0000 7fcdd3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56913390 0x559f56ce4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:17.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.001+0000 7fcdd3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56913390 0x559f56ce4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59488/0 (socket says 172.21.15.134:59488) 2024-07-26T11:19:17.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.001+0000 7fcdd3e00640 1 -- 172.21.15.134:0/549094499 learned_addr learned my addr 172.21.15.134:0/549094499 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:17.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.001+0000 7fcdd3e00640 1 -- 172.21.15.134:0/549094499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f5690b1b0 msgr2=0x559f56ce4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:17.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.002+0000 7fcdd3e00640 1 --2- 172.21.15.134:0/549094499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f5690b1b0 0x559f56ce4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.002+0000 7fcdd3e00640 1 -- 172.21.15.134:0/549094499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f56934070 msgr2=0x559f56ce4dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:19:17.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.002+0000 7fcdda800640 1 --2- 172.21.15.134:0/549094499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f56934070 0x559f56ce4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:17.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.002+0000 7fcdd3e00640 1 --2- 172.21.15.134:0/549094499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f56934070 0x559f56ce4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.003+0000 7fcdd3e00640 1 -- 172.21.15.134:0/549094499 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f56bf2840 con 0x559f56913390 2024-07-26T11:19:17.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.003+0000 7fcdda800640 1 --2- 172.21.15.134:0/549094499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f56934070 0x559f56ce4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:19:17.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.004+0000 7fcdd3e00640 1 --2- 172.21.15.134:0/549094499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56913390 0x559f56ce4880 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc02a950 tx=0x7fcdcc004250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e2a6205486b8376 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:17.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.004+0000 7fcdbfe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:19:17.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.005+0000 7fcdbfe00640 1 -- 172.21.15.134:0/549094499 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc037040 con 0x559f56913390 2024-07-26T11:19:17.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.005+0000 7fcdbfe00640 1 -- 172.21.15.134:0/549094499 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fcdcc0306c0 con 0x559f56913390 2024-07-26T11:19:17.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.005+0000 7fcde7a3a740 1 -- 172.21.15.134:0/549094499 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdd400d830 con 0x559f56913390 2024-07-26T11:19:17.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.006+0000 7fcdbfe00640 1 -- 172.21.15.134:0/549094499 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcdcc0308c0 con 0x559f56913390 2024-07-26T11:19:17.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.006+0000 7fcde7a3a740 10 client.26065 fetch_fsmap learned FSMap version 31 2024-07-26T11:19:17.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.006+0000 7fcde7a3a740 1 -- 172.21.15.134:0/549094499 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559f56cf31c0 con 0x559f56913390 2024-07-26T11:19:17.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.006+0000 7fcdbfe00640 1 -- 172.21.15.134:0/549094499 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdd400d830 con 0x559f56913390 2024-07-26T11:19:17.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.007+0000 7fcdbfe00640 1 -- 172.21.15.134:0/549094499 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7fcdcc004c20 con 0x559f56913390 2024-07-26T11:19:17.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.007+0000 7fcde7a3a740 10 client.26065 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:19:17.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.007+0000 7fcde7a3a740 10 client.26065 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-07-26T11:19:17.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.007+0000 7fcde7a3a740 20 client.26065 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:17.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.007+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/549094499 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x559f568f29f0 0x559f56cf67c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:17.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.007+0000 7fcde7a3a740 4 client.26065 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:19:17.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.007+0000 7fcde7a3a740 1 -- 172.21.15.134:0/549094499 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f56c51b20 con 0x559f568f29f0 2024-07-26T11:19:17.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.008+0000 7fcdda800640 1 --2- 172.21.15.134:0/549094499 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x559f568f29f0 0x559f56cf67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:17.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.010+0000 7fcdda800640 1 --2- 172.21.15.134:0/549094499 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x559f568f29f0 0x559f56cf67c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3407b3b9fb32da6f server_cookie=66d481457795a050 in_seq=0 out_seq=0 2024-07-26T11:19:17.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.011+0000 7fcdbfe00640 10 client.26065 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:19:17.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.011+0000 7fcdbfe00640 1 -- 172.21.15.134:0/549094499 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559f56c51b20 con 0x559f568f29f0 2024-07-26T11:19:17.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.011+0000 7fcdbfe00640 10 client.26065 handle_command_reply: tid=0 2024-07-26T11:19:17.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.011+0000 7fcde7a3a740 1 client.26065 shutdown 2024-07-26T11:19:17.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.012+0000 7fcde7a3a740 1 -- 172.21.15.134:0/549094499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56913390 msgr2=0x559f56ce4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:17.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.012+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/549094499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56913390 0x559f56ce4880 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc02a950 tx=0x7fcdcc004250 comp rx=0 tx=0).stop 2024-07-26T11:19:17.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.012+0000 7fcde7a3a740 1 -- 172.21.15.134:0/549094499 shutdown_connections 2024-07-26T11:19:17.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.012+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/549094499 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x559f568f29f0 0x559f56cf67c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.012+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/549094499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56913390 0x559f56ce4880 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.012+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/549094499 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f56934070 0x559f56ce4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.012+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/549094499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f5690b1b0 0x559f56ce4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.012+0000 7fcde7a3a740 1 -- 172.21.15.134:0/549094499 >> 172.21.15.134:0/549094499 conn(0x559f56928460 msgr2=0x559f56cdbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:17.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.012+0000 7fcdbfe00640 0 client.26065 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:19:17.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.013+0000 7fcde7a3a740 1 -- 172.21.15.134:0/549094499 shutdown_connections 2024-07-26T11:19:17.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.013+0000 7fcde7a3a740 1 -- 172.21.15.134:0/549094499 wait complete. 2024-07-26T11:19:17.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.013+0000 7fcde7a3a740 20 client.26065 trim_cache size 0 max 16384 2024-07-26T11:19:17.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.043+0000 7fcde7a3a740 1 Processor -- start 2024-07-26T11:19:17.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.043+0000 7fcde7a3a740 1 -- start start 2024-07-26T11:19:17.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.044+0000 7fcde7a3a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f5690b1b0 0x559f56cda930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:17.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.044+0000 7fcde7a3a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56934070 0x559f56ce4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:17.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.044+0000 7fcde7a3a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f568c7da0 0x559f56cdf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:17.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.044+0000 7fcde7a3a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc1bfcc0 con 0x559f5690b1b0 2024-07-26T11:19:17.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.044+0000 7fcde7a3a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc1c1830 con 0x559f568c7da0 2024-07-26T11:19:17.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.044+0000 7fcde7a3a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc1c0390 con 0x559f56934070 2024-07-26T11:19:17.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.044+0000 7fcdda800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f568c7da0 0x559f56cdf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:17.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.044+0000 7fcdda800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f568c7da0 0x559f56cdf5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56394/0 (socket says 172.21.15.134:56394) 2024-07-26T11:19:17.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.045+0000 7fcdda800640 1 -- 172.21.15.134:0/2312431932 learned_addr learned my addr 172.21.15.134:0/2312431932 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:17.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.045+0000 7fcdd3e00640 1 --2- 172.21.15.134:0/2312431932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56934070 0x559f56ce4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:17.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.045+0000 7fcdda800640 1 --2- 172.21.15.134:0/2312431932 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f5690b1b0 0x559f56cda930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:17.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.046+0000 7fcdda800640 1 -- 172.21.15.134:0/2312431932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56934070 msgr2=0x559f56ce4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:17.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.046+0000 7fcdda800640 1 --2- 172.21.15.134:0/2312431932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56934070 0x559f56ce4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.046+0000 7fcdda800640 1 -- 172.21.15.134:0/2312431932 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f5690b1b0 msgr2=0x559f56cda930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:17.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.046+0000 7fcdda800640 1 --2- 172.21.15.134:0/2312431932 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f5690b1b0 0x559f56cda930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.046+0000 7fcdda800640 1 -- 172.21.15.134:0/2312431932 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f56cdb350 con 0x559f568c7da0 2024-07-26T11:19:17.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.046+0000 7fcdda800640 1 --2- 172.21.15.134:0/2312431932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f568c7da0 0x559f56cdf5d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400e520 tx=0x7fcdd4009cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aad820ae7b357a60 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:17.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.047+0000 7fcdcaa00640 1 -- 172.21.15.134:0/2312431932 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4032070 con 0x559f568c7da0 2024-07-26T11:19:17.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.047+0000 7fcdcaa00640 1 -- 172.21.15.134:0/2312431932 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdd4003e20 con 0x559f568c7da0 2024-07-26T11:19:17.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.047+0000 7fcdcaa00640 1 -- 172.21.15.134:0/2312431932 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4005690 con 0x559f568c7da0 2024-07-26T11:19:17.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.049+0000 7fcde7a3a740 1 -- 172.21.15.134:0/2312431932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f568c7da0 msgr2=0x559f56cdf5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:17.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.049+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/2312431932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f568c7da0 0x559f56cdf5d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcdd400e520 tx=0x7fcdd4009cc0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.050+0000 7fcde7a3a740 1 -- 172.21.15.134:0/2312431932 shutdown_connections 2024-07-26T11:19:17.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.050+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/2312431932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f56934070 0x559f56ce4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.050+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/2312431932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f568c7da0 0x559f56cdf5d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.050+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/2312431932 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f5690b1b0 0x559f56cda930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.050+0000 7fcde7a3a740 1 -- 172.21.15.134:0/2312431932 >> 172.21.15.134:0/2312431932 conn(0x559f56928460 msgr2=0x559f56cdeec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:17.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.050+0000 7fcde7a3a740 1 -- 172.21.15.134:0/2312431932 shutdown_connections 2024-07-26T11:19:17.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.050+0000 7fcde7a3a740 1 -- 172.21.15.134:0/2312431932 wait complete. 2024-07-26T11:19:17.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.051+0000 7fcde7a3a740 1 -- start start 2024-07-26T11:19:17.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.052+0000 7fcde7a3a740 1 Processor -- start 2024-07-26T11:19:17.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.053+0000 7fcde7a3a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f568c7da0 0x559f56ce8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:17.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.053+0000 7fcde7a3a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f5690b1b0 0x559f56ce8600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:17.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.053+0000 7fcde7a3a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f56934070 0x559f56d5cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:17.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.053+0000 7fcde7a3a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc1c1130 con 0x559f56934070 2024-07-26T11:19:17.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.053+0000 7fcde7a3a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc1c0a60 con 0x559f5690b1b0 2024-07-26T11:19:17.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.053+0000 7fcde7a3a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc1bdca0 con 0x559f568c7da0 2024-07-26T11:19:17.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.053+0000 7fcde7a3a740 10 client.0 _ll_register_callbacks cb 0x7fcdcc0096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:17.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.053+0000 7fcdda800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f568c7da0 0x559f56ce8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:17.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.053+0000 7fcdda800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f568c7da0 0x559f56ce8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59502/0 (socket says 172.21.15.134:59502) 2024-07-26T11:19:17.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.053+0000 7fcdda800640 1 -- 172.21.15.134:0/4120423543 learned_addr learned my addr 172.21.15.134:0/4120423543 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:17.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.054+0000 7fcdda800640 1 --2- 172.21.15.134:0/4120423543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f56934070 0x559f56d5cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:17.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.054+0000 7fcdd3e00640 1 --2- 172.21.15.134:0/4120423543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f5690b1b0 0x559f56ce8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:17.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.054+0000 7fcdda800640 1 -- 172.21.15.134:0/4120423543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f5690b1b0 msgr2=0x559f56ce8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:17.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.054+0000 7fcdda800640 1 --2- 172.21.15.134:0/4120423543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f5690b1b0 0x559f56ce8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.054+0000 7fcdda800640 1 -- 172.21.15.134:0/4120423543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f56934070 msgr2=0x559f56d5cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:17.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.054+0000 7fcdda800640 1 --2- 172.21.15.134:0/4120423543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f56934070 0x559f56d5cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.054+0000 7fcdda800640 1 -- 172.21.15.134:0/4120423543 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f56d5d550 con 0x559f568c7da0 2024-07-26T11:19:17.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.054+0000 7fcdd3e00640 1 --2- 172.21.15.134:0/4120423543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f5690b1b0 0x559f56ce8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:19:17.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.055+0000 7fcdda800640 1 --2- 172.21.15.134:0/4120423543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f568c7da0 0x559f56ce8080 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4002230 tx=0x7fcdd4019040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3448f9be74e5657f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:17.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.055+0000 7fcdbfe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:19:17.060 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.055+0000 7fcdbfe00640 1 -- 172.21.15.134:0/4120423543 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4032040 con 0x559f568c7da0 2024-07-26T11:19:17.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.055+0000 7fcdbfe00640 1 -- 172.21.15.134:0/4120423543 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fcdd4003b70 con 0x559f568c7da0 2024-07-26T11:19:17.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.055+0000 7fcde7a3a740 1 -- 172.21.15.134:0/4120423543 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdd4003e20 con 0x559f568c7da0 2024-07-26T11:19:17.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.055+0000 7fcdbfe00640 1 -- 172.21.15.134:0/4120423543 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fcdd403f050 con 0x559f568c7da0 2024-07-26T11:19:17.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.055+0000 7fcde7a3a740 10 client.26071 fetch_fsmap learned FSMap version 31 2024-07-26T11:19:17.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.055+0000 7fcde7a3a740 1 -- 172.21.15.134:0/4120423543 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x559f56d5e7d0 con 0x559f568c7da0 2024-07-26T11:19:17.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.056+0000 7fcdbfe00640 1 -- 172.21.15.134:0/4120423543 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdd4003e20 con 0x559f568c7da0 2024-07-26T11:19:17.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.056+0000 7fcdbfe00640 1 -- 172.21.15.134:0/4120423543 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7fcdd400b040 con 0x559f568c7da0 2024-07-26T11:19:17.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.056+0000 7fcde7a3a740 10 client.26071 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:19:17.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.056+0000 7fcde7a3a740 10 client.26071 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-07-26T11:19:17.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.056+0000 7fcde7a3a740 20 client.26071 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:17.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.056+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/4120423543 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x559f56d5ed60 0x559f56d61180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:17.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.056+0000 7fcde7a3a740 4 client.26071 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:19:17.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.056+0000 7fcde7a3a740 1 -- 172.21.15.134:0/4120423543 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559f56c4d5c0 con 0x559f56d5ed60 2024-07-26T11:19:17.062 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.058+0000 7fcdd3e00640 1 --2- 172.21.15.134:0/4120423543 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x559f56d5ed60 0x559f56d61180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:17.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.059+0000 7fcdd3e00640 1 --2- 172.21.15.134:0/4120423543 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x559f56d5ed60 0x559f56d61180 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1ba5efb9e7a3d7c7 server_cookie=6928e5a853590eac in_seq=0 out_seq=0 2024-07-26T11:19:17.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.059+0000 7fcdbfe00640 10 client.26071 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:19:17.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.060+0000 7fcdbfe00640 1 -- 172.21.15.134:0/4120423543 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559f56c4d5c0 con 0x559f56d5ed60 2024-07-26T11:19:17.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.060+0000 7fcdbfe00640 10 client.26071 handle_command_reply: tid=0 2024-07-26T11:19:17.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.060+0000 7fcde7a3a740 1 client.26071 shutdown 2024-07-26T11:19:17.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.060+0000 7fcde7a3a740 1 -- 172.21.15.134:0/4120423543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f568c7da0 msgr2=0x559f56ce8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:17.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.060+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/4120423543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f568c7da0 0x559f56ce8080 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4002230 tx=0x7fcdd4019040 comp rx=0 tx=0).stop 2024-07-26T11:19:17.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.061+0000 7fcde7a3a740 1 -- 172.21.15.134:0/4120423543 shutdown_connections 2024-07-26T11:19:17.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.061+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/4120423543 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x559f56d5ed60 0x559f56d61180 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.061+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/4120423543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559f56934070 0x559f56d5cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.061+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/4120423543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559f5690b1b0 0x559f56ce8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.061+0000 7fcde7a3a740 1 --2- 172.21.15.134:0/4120423543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559f568c7da0 0x559f56ce8080 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.061+0000 7fcde7a3a740 1 -- 172.21.15.134:0/4120423543 >> 172.21.15.134:0/4120423543 conn(0x559f56928460 msgr2=0x559f56ce4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:17.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.061+0000 7fcdbfe00640 0 client.26071 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:19:17.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.063+0000 7fcde7a3a740 1 -- 172.21.15.134:0/4120423543 shutdown_connections 2024-07-26T11:19:17.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.064+0000 7fcde7a3a740 1 -- 172.21.15.134:0/4120423543 wait complete. 2024-07-26T11:19:17.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.064+0000 7fcde7a3a740 20 client.26071 trim_cache size 0 max 16384 2024-07-26T11:19:17.066 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:19:17.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.065+0000 7fcde0e00640 1 -- 172.21.15.134:0/1251076421 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcdc00b3f50 msgr2=0x7fcdc00b6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:17.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.065+0000 7fcde0e00640 1 --2- 172.21.15.134:0/1251076421 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcdc00b3f50 0x7fcdc00b6460 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x559f56cd7ab0 tx=0x7fcdc400bf50 comp rx=0 tx=0).stop 2024-07-26T11:19:17.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.065+0000 7fcde0e00640 1 -- 172.21.15.134:0/1251076421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcddc109df0 msgr2=0x7fcddc114750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:17.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.065+0000 7fcde0e00640 1 --2- 172.21.15.134:0/1251076421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcddc109df0 0x7fcddc114750 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcdc400b3e0 tx=0x7fcdc400b8a0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.065+0000 7fcde0e00640 1 -- 172.21.15.134:0/1251076421 shutdown_connections 2024-07-26T11:19:17.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.065+0000 7fcde0e00640 1 --2- 172.21.15.134:0/1251076421 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcdc00b3f50 0x7fcdc00b6460 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.065+0000 7fcde0e00640 1 --2- 172.21.15.134:0/1251076421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcddc111060 0x7fcddc1b4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.065+0000 7fcde0e00640 1 --2- 172.21.15.134:0/1251076421 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fcddc10a740 0x7fcddc110a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.065+0000 7fcde0e00640 1 --2- 172.21.15.134:0/1251076421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcddc109df0 0x7fcddc114750 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:17.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.065+0000 7fcde0e00640 1 -- 172.21.15.134:0/1251076421 >> 172.21.15.134:0/1251076421 conn(0x7fcddc070d20 msgr2=0x7fcddc10b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:17.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.065+0000 7fcde0e00640 1 -- 172.21.15.134:0/1251076421 shutdown_connections 2024-07-26T11:19:17.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:17.066+0000 7fcde0e00640 1 -- 172.21.15.134:0/1251076421 wait complete. 2024-07-26T11:19:18.549 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-26T11:19:18.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.593+0000 7f3a7be00640 1 -- 172.21.15.134:0/2509166225 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a7c109df0 msgr2=0x7f3a7c10a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:18.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.593+0000 7f3a7be00640 1 --2- 172.21.15.134:0/2509166225 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a7c109df0 0x7f3a7c10a200 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3a64009930 tx=0x7f3a6402a480 comp rx=0 tx=0).stop 2024-07-26T11:19:18.599 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.596+0000 7f3a7be00640 1 -- 172.21.15.134:0/2509166225 shutdown_connections 2024-07-26T11:19:18.599 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.596+0000 7f3a7be00640 1 --2- 172.21.15.134:0/2509166225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a7c10a740 0x7f3a7c117df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.599 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.596+0000 7f3a7be00640 1 --2- 172.21.15.134:0/2509166225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a7c118330 0x7f3a7c11a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.599 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.596+0000 7f3a7be00640 1 --2- 172.21.15.134:0/2509166225 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a7c109df0 0x7f3a7c10a200 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.599 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.596+0000 7f3a7be00640 1 -- 172.21.15.134:0/2509166225 >> 172.21.15.134:0/2509166225 conn(0x7f3a7c070630 msgr2=0x7f3a7c070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:18.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.596+0000 7f3a7be00640 1 -- 172.21.15.134:0/2509166225 shutdown_connections 2024-07-26T11:19:18.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.597+0000 7f3a7be00640 1 -- 172.21.15.134:0/2509166225 wait complete. 2024-07-26T11:19:18.601 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.600+0000 7f3a7be00640 1 Processor -- start 2024-07-26T11:19:18.604 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.602+0000 7f3a7be00640 1 -- start start 2024-07-26T11:19:18.607 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.603+0000 7f3a7be00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a7c109df0 0x7f3a7c1a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.603+0000 7f3a7be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a7c10a740 0x7f3a7c1a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.603+0000 7f3a7be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a7c118330 0x7f3a7c1a9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.603+0000 7f3a7be00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c11dd00 con 0x7f3a7c109df0 2024-07-26T11:19:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.603+0000 7f3a7be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c11db80 con 0x7f3a7c118330 2024-07-26T11:19:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.603+0000 7f3a7be00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c11de80 con 0x7f3a7c10a740 2024-07-26T11:19:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.603+0000 7f3a7b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a7c118330 0x7f3a7c1a9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.603+0000 7f3a7b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a7c118330 0x7f3a7c1a9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56408/0 (socket says 172.21.15.134:56408) 2024-07-26T11:19:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.603+0000 7f3a7b400640 1 -- 172.21.15.134:0/3468624763 learned_addr learned my addr 172.21.15.134:0/3468624763 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.603+0000 7f3a7aa00640 1 --2- 172.21.15.134:0/3468624763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a7c109df0 0x7f3a7c1a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.603+0000 7f3a72000640 1 --2- 172.21.15.134:0/3468624763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a7c10a740 0x7f3a7c1a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.604+0000 7f3a72000640 1 -- 172.21.15.134:0/3468624763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a7c118330 msgr2=0x7f3a7c1a9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:18.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.604+0000 7f3a72000640 1 --2- 172.21.15.134:0/3468624763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a7c118330 0x7f3a7c1a9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.604+0000 7f3a72000640 1 -- 172.21.15.134:0/3468624763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a7c109df0 msgr2=0x7f3a7c1a4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:18.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.604+0000 7f3a72000640 1 --2- 172.21.15.134:0/3468624763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a7c109df0 0x7f3a7c1a4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.604+0000 7f3a72000640 1 -- 172.21.15.134:0/3468624763 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a7c1aa600 con 0x7f3a7c10a740 2024-07-26T11:19:18.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.605+0000 7f3a72000640 1 --2- 172.21.15.134:0/3468624763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a7c10a740 0x7f3a7c1a4d00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c06c0f0 tx=0x7f3a6c00cdd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:18.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.605+0000 7f3a73e00640 1 -- 172.21.15.134:0/3468624763 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c007bb0 con 0x7f3a7c10a740 2024-07-26T11:19:18.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.605+0000 7f3a73e00640 1 -- 172.21.15.134:0/3468624763 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a6c007d50 con 0x7f3a7c10a740 2024-07-26T11:19:18.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.605+0000 7f3a73e00640 1 -- 172.21.15.134:0/3468624763 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c0186f0 con 0x7f3a7c10a740 2024-07-26T11:19:18.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.605+0000 7f3a7be00640 1 -- 172.21.15.134:0/3468624763 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a7c1aa8f0 con 0x7f3a7c10a740 2024-07-26T11:19:18.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.605+0000 7f3a7be00640 1 -- 172.21.15.134:0/3468624763 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a7c1b1110 con 0x7f3a7c10a740 2024-07-26T11:19:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.619+0000 7f3a73e00640 1 -- 172.21.15.134:0/3468624763 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f3a6c00f450 con 0x7f3a7c10a740 2024-07-26T11:19:18.626 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.624+0000 7f3a88c33740 1 Processor -- start 2024-07-26T11:19:18.627 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.625+0000 7f3a73e00640 1 --2- 172.21.15.134:0/3468624763 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3a600b3dc0 0x7f3a600b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.627 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.626+0000 7f3a73e00640 1 -- 172.21.15.134:0/3468624763 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3a6c0e42c0 con 0x7f3a7c10a740 2024-07-26T11:19:18.629 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.627+0000 7f3a88c33740 1 -- start start 2024-07-26T11:19:18.630 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.628+0000 7f3a88c33740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5edc81b0 0x560c5f174450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.630 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.628+0000 7f3a88c33740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5ed9f2f0 0x560c5f174990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.630 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.628+0000 7f3a88c33740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5eda74d0 0x560c5f174ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.630 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.628+0000 7f3a88c33740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560c5f0e0270 con 0x560c5edc81b0 2024-07-26T11:19:18.630 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.628+0000 7f3a88c33740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560c5f0cf770 con 0x560c5eda74d0 2024-07-26T11:19:18.630 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.628+0000 7f3a88c33740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560c5f0d37b0 con 0x560c5ed9f2f0 2024-07-26T11:19:18.641 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.628+0000 7f3a72000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5ed9f2f0 0x560c5f174990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.641 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.628+0000 7f3a72000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5ed9f2f0 0x560c5f174990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59528/0 (socket says 172.21.15.134:59528) 2024-07-26T11:19:18.641 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.628+0000 7f3a72000640 1 -- 172.21.15.134:0/738181960 learned_addr learned my addr 172.21.15.134:0/738181960 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:18.641 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.629+0000 7f3a7aa00640 1 --2- 172.21.15.134:0/738181960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5edc81b0 0x560c5f174450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.630+0000 7f3a7b400640 1 --2- 172.21.15.134:0/3468624763 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3a600b3dc0 0x7f3a600b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.630+0000 7f3a7b400640 1 --2- 172.21.15.134:0/738181960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5eda74d0 0x560c5f174ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.630+0000 7f3a72000640 1 -- 172.21.15.134:0/738181960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5eda74d0 msgr2=0x560c5f174ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.630+0000 7f3a72000640 1 --2- 172.21.15.134:0/738181960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5eda74d0 0x560c5f174ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.630+0000 7f3a72000640 1 -- 172.21.15.134:0/738181960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5edc81b0 msgr2=0x560c5f174450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.630+0000 7f3a72000640 1 --2- 172.21.15.134:0/738181960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5edc81b0 0x560c5f174450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.630+0000 7f3a72000640 1 -- 172.21.15.134:0/738181960 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c5ee33790 con 0x560c5ed9f2f0 2024-07-26T11:19:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.632+0000 7f3a7b400640 1 --2- 172.21.15.134:0/3468624763 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3a600b3dc0 0x7f3a600b62d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3a74013b70 tx=0x7f3a74009710 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.633+0000 7f3a72000640 1 --2- 172.21.15.134:0/738181960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5ed9f2f0 0x560c5f174990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c000c00 tx=0x7f3a6c0aa720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38bdbf047f99424d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:18.642 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.633+0000 7f3a6be00640 1 -- 172.21.15.134:0/738181960 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c0af150 con 0x560c5ed9f2f0 2024-07-26T11:19:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.633+0000 7f3a6be00640 1 -- 172.21.15.134:0/738181960 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a6c0b3c00 con 0x560c5ed9f2f0 2024-07-26T11:19:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.633+0000 7f3a6be00640 1 -- 172.21.15.134:0/738181960 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c0920d0 con 0x560c5ed9f2f0 2024-07-26T11:19:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.635+0000 7f3a88c33740 1 -- 172.21.15.134:0/738181960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5ed9f2f0 msgr2=0x560c5f174990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.635+0000 7f3a88c33740 1 --2- 172.21.15.134:0/738181960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5ed9f2f0 0x560c5f174990 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c000c00 tx=0x7f3a6c0aa720 comp rx=0 tx=0).stop 2024-07-26T11:19:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.636+0000 7f3a88c33740 1 -- 172.21.15.134:0/738181960 shutdown_connections 2024-07-26T11:19:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.636+0000 7f3a88c33740 1 --2- 172.21.15.134:0/738181960 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5ed9f2f0 0x560c5f174990 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.636+0000 7f3a88c33740 1 --2- 172.21.15.134:0/738181960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5eda74d0 0x560c5f174ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.636+0000 7f3a88c33740 1 --2- 172.21.15.134:0/738181960 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5edc81b0 0x560c5f174450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.636+0000 7f3a88c33740 1 -- 172.21.15.134:0/738181960 >> 172.21.15.134:0/738181960 conn(0x560c5edbc300 msgr2=0x560c5f172340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.639+0000 7f3a88c33740 1 -- 172.21.15.134:0/738181960 shutdown_connections 2024-07-26T11:19:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.639+0000 7f3a88c33740 1 -- 172.21.15.134:0/738181960 wait complete. 2024-07-26T11:19:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.640+0000 7f3a88c33740 1 -- start start 2024-07-26T11:19:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.641+0000 7f3a88c33740 1 Processor -- start 2024-07-26T11:19:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.643+0000 7f3a88c33740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5ed9f2f0 0x560c5f178730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.644+0000 7f3a88c33740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5eda74d0 0x560c5f178c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.644+0000 7f3a88c33740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5edc81b0 0x560c5f1817a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.644+0000 7f3a88c33740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560c5f009ab0 con 0x560c5eda74d0 2024-07-26T11:19:18.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.644+0000 7f3a88c33740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560c5f06cd50 con 0x560c5edc81b0 2024-07-26T11:19:18.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.644+0000 7f3a88c33740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560c5f1695a0 con 0x560c5ed9f2f0 2024-07-26T11:19:18.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.644+0000 7f3a88c33740 10 client.0 _ll_register_callbacks cb 0x560c5eddfa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:18.648 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.646+0000 7f3a7aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5ed9f2f0 0x560c5f178730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.648 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.646+0000 7f3a7b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5edc81b0 0x560c5f1817a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.648 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.646+0000 7f3a7b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5edc81b0 0x560c5f1817a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56434/0 (socket says 172.21.15.134:56434) 2024-07-26T11:19:18.648 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.646+0000 7f3a7b400640 1 -- 172.21.15.134:0/4179721558 learned_addr learned my addr 172.21.15.134:0/4179721558 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:18.649 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.647+0000 7f3a7b400640 1 -- 172.21.15.134:0/4179721558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5ed9f2f0 msgr2=0x560c5f178730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:18.649 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.647+0000 7f3a7b400640 1 --2- 172.21.15.134:0/4179721558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5ed9f2f0 0x560c5f178730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.647+0000 7f3a7b400640 1 -- 172.21.15.134:0/4179721558 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5eda74d0 msgr2=0x560c5f178c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:18.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.647+0000 7f3a7b400640 1 --2- 172.21.15.134:0/4179721558 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5eda74d0 0x560c5f178c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.647+0000 7f3a7b400640 1 -- 172.21.15.134:0/4179721558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c5eeac620 con 0x560c5edc81b0 2024-07-26T11:19:18.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.648+0000 7f3a7b400640 1 --2- 172.21.15.134:0/4179721558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5edc81b0 0x560c5f1817a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3a74019680 tx=0x7f3a74004ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=361a1266faf4c191 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:18.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.648+0000 7f3a68c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:19:18.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.650+0000 7f3a68c00640 1 -- 172.21.15.134:0/4179721558 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74029060 con 0x560c5edc81b0 2024-07-26T11:19:18.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.651+0000 7f3a68c00640 1 -- 172.21.15.134:0/4179721558 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3a74018570 con 0x560c5edc81b0 2024-07-26T11:19:18.655 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.653+0000 7f3a88c33740 1 -- 172.21.15.134:0/4179721558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a6c0b3c00 con 0x560c5edc81b0 2024-07-26T11:19:18.659 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.657+0000 7f3a68c00640 1 -- 172.21.15.134:0/4179721558 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3a74018770 con 0x560c5edc81b0 2024-07-26T11:19:18.659 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.657+0000 7f3a88c33740 10 client.26081 fetch_fsmap learned FSMap version 31 2024-07-26T11:19:18.659 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.657+0000 7f3a88c33740 1 -- 172.21.15.134:0/4179721558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560c5f186450 con 0x560c5edc81b0 2024-07-26T11:19:18.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.658+0000 7f3a68c00640 1 -- 172.21.15.134:0/4179721558 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a6c0b3c00 con 0x560c5edc81b0 2024-07-26T11:19:18.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.658+0000 7f3a68c00640 1 -- 172.21.15.134:0/4179721558 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f3a74023070 con 0x560c5edc81b0 2024-07-26T11:19:18.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.659+0000 7f3a88c33740 10 client.26081 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:19:18.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.659+0000 7f3a88c33740 10 client.26081 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:19:18.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.659+0000 7f3a88c33740 20 client.26081 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:18.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.659+0000 7f3a88c33740 1 --2- 172.21.15.134:0/4179721558 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560c5ed86d80 0x560c5f189960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.659+0000 7f3a88c33740 4 client.26081 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:19:18.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.659+0000 7f3a88c33740 1 -- 172.21.15.134:0/4179721558 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c5f0e21c0 con 0x560c5ed86d80 2024-07-26T11:19:18.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.660+0000 7f3a7aa00640 1 --2- 172.21.15.134:0/4179721558 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560c5ed86d80 0x560c5f189960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.666 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.664+0000 7f3a68c00640 10 client.26081 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:18.671 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.669+0000 7f3a7aa00640 1 --2- 172.21.15.134:0/4179721558 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560c5ed86d80 0x560c5f189960 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e58deb8cfa92fb5c server_cookie=ef901e6eff3f60f4 in_seq=0 out_seq=0 2024-07-26T11:19:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.670+0000 7f3a68c00640 1 -- 172.21.15.134:0/4179721558 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560c5f0e21c0 con 0x560c5ed86d80 2024-07-26T11:19:18.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.670+0000 7f3a68c00640 10 client.26081 handle_command_reply: tid=0 2024-07-26T11:19:18.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.671+0000 7f3a88c33740 1 client.26081 shutdown 2024-07-26T11:19:18.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.671+0000 7f3a88c33740 1 -- 172.21.15.134:0/4179721558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5edc81b0 msgr2=0x560c5f1817a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:18.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.671+0000 7f3a88c33740 1 --2- 172.21.15.134:0/4179721558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5edc81b0 0x560c5f1817a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3a74019680 tx=0x7f3a74004ad0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.672+0000 7f3a88c33740 1 -- 172.21.15.134:0/4179721558 shutdown_connections 2024-07-26T11:19:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.673+0000 7f3a88c33740 1 --2- 172.21.15.134:0/4179721558 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560c5ed86d80 0x560c5f189960 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.673+0000 7f3a88c33740 1 --2- 172.21.15.134:0/4179721558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5edc81b0 0x560c5f1817a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.673+0000 7f3a88c33740 1 --2- 172.21.15.134:0/4179721558 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5eda74d0 0x560c5f178c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.673+0000 7f3a88c33740 1 --2- 172.21.15.134:0/4179721558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5ed9f2f0 0x560c5f178730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.673+0000 7f3a88c33740 1 -- 172.21.15.134:0/4179721558 >> 172.21.15.134:0/4179721558 conn(0x560c5edbc300 msgr2=0x560c5f175480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:18.676 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:18 smithi134 ceph-mon[31966]: pgmap v580: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 215 MiB/s wr, 1.46k op/s 2024-07-26T11:19:18.676 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.674+0000 7f3a68c00640 0 client.26081 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:18.676 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.675+0000 7f3a88c33740 1 -- 172.21.15.134:0/4179721558 shutdown_connections 2024-07-26T11:19:18.678 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.676+0000 7f3a88c33740 1 -- 172.21.15.134:0/4179721558 wait complete. 2024-07-26T11:19:18.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.677+0000 7f3a88c33740 20 client.26081 trim_cache size 0 max 16384 2024-07-26T11:19:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:18 smithi189 ceph-mon[36790]: pgmap v580: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 215 MiB/s wr, 1.46k op/s 2024-07-26T11:19:18.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.714+0000 7f3a88c33740 1 Processor -- start 2024-07-26T11:19:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.715+0000 7f3a88c33740 1 -- start start 2024-07-26T11:19:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.715+0000 7f3a88c33740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5ed9f2f0 0x560c5f175480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.715+0000 7f3a88c33740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5eda74d0 0x560c5f1759c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.715+0000 7f3a88c33740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5edc81b0 0x560c5f178510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.716+0000 7f3a7aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5ed9f2f0 0x560c5f175480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.716+0000 7f3a7aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5ed9f2f0 0x560c5f175480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56446/0 (socket says 172.21.15.134:56446) 2024-07-26T11:19:18.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.716+0000 7f3a7aa00640 1 -- 172.21.15.134:0/2638997577 learned_addr learned my addr 172.21.15.134:0/2638997577 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:18.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.716+0000 7f3a7b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5edc81b0 0x560c5f178510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.716+0000 7f3a88c33740 1 -- 172.21.15.134:0/2638997577 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b6e50 con 0x560c5eda74d0 2024-07-26T11:19:18.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.717+0000 7f3a88c33740 1 -- 172.21.15.134:0/2638997577 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b89c0 con 0x560c5ed9f2f0 2024-07-26T11:19:18.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.717+0000 7f3a88c33740 1 -- 172.21.15.134:0/2638997577 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b7520 con 0x560c5edc81b0 2024-07-26T11:19:18.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.718+0000 7f3a7aa00640 1 -- 172.21.15.134:0/2638997577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5edc81b0 msgr2=0x560c5f178510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:18.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.718+0000 7f3a7aa00640 1 --2- 172.21.15.134:0/2638997577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5edc81b0 0x560c5f178510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.718+0000 7f3a7aa00640 1 -- 172.21.15.134:0/2638997577 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5eda74d0 msgr2=0x560c5f1759c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:18.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.718+0000 7f3a7aa00640 1 --2- 172.21.15.134:0/2638997577 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5eda74d0 0x560c5f1759c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.718+0000 7f3a7aa00640 1 -- 172.21.15.134:0/2638997577 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c5f171290 con 0x560c5ed9f2f0 2024-07-26T11:19:18.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.719+0000 7f3a7aa00640 1 --2- 172.21.15.134:0/2638997577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5ed9f2f0 0x560c5f175480 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3a64009930 tx=0x7f3a64000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d483b9ded647bf4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:18.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.720+0000 7f3a6b400640 1 -- 172.21.15.134:0/2638997577 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a64004230 con 0x560c5ed9f2f0 2024-07-26T11:19:18.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.720+0000 7f3a6b400640 1 -- 172.21.15.134:0/2638997577 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a64005c90 con 0x560c5ed9f2f0 2024-07-26T11:19:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.720+0000 7f3a6b400640 1 -- 172.21.15.134:0/2638997577 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a64004230 con 0x560c5ed9f2f0 2024-07-26T11:19:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.725+0000 7f3a88c33740 1 -- 172.21.15.134:0/2638997577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5ed9f2f0 msgr2=0x560c5f175480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.725+0000 7f3a88c33740 1 --2- 172.21.15.134:0/2638997577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5ed9f2f0 0x560c5f175480 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3a64009930 tx=0x7f3a64000bc0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.725+0000 7f3a88c33740 1 -- 172.21.15.134:0/2638997577 shutdown_connections 2024-07-26T11:19:18.727 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.725+0000 7f3a88c33740 1 --2- 172.21.15.134:0/2638997577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5edc81b0 0x560c5f178510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.725+0000 7f3a88c33740 1 --2- 172.21.15.134:0/2638997577 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5eda74d0 0x560c5f1759c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.726+0000 7f3a88c33740 1 --2- 172.21.15.134:0/2638997577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5ed9f2f0 0x560c5f175480 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.728 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.726+0000 7f3a88c33740 1 -- 172.21.15.134:0/2638997577 >> 172.21.15.134:0/2638997577 conn(0x560c5edbc300 msgr2=0x560c5f16f840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:18.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.726+0000 7f3a88c33740 1 -- 172.21.15.134:0/2638997577 shutdown_connections 2024-07-26T11:19:18.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.727+0000 7f3a88c33740 1 -- 172.21.15.134:0/2638997577 wait complete. 2024-07-26T11:19:18.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.728+0000 7f3a88c33740 1 -- start start 2024-07-26T11:19:18.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.728+0000 7f3a88c33740 1 Processor -- start 2024-07-26T11:19:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.730+0000 7f3a88c33740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5ed9f2f0 0x560c5f179ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.730+0000 7f3a88c33740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5eda74d0 0x560c5f17a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.730+0000 7f3a88c33740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5edc81b0 0x560c5f1f0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.730+0000 7f3a88c33740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b82c0 con 0x560c5edc81b0 2024-07-26T11:19:18.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.730+0000 7f3a88c33740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b7bf0 con 0x560c5ed9f2f0 2024-07-26T11:19:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.730+0000 7f3a88c33740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c1b4e50 con 0x560c5eda74d0 2024-07-26T11:19:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.730+0000 7f3a88c33740 10 client.0 _ll_register_callbacks cb 0x7f3a74017f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.731+0000 7f3a7b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5edc81b0 0x560c5f1f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.731+0000 7f3a7b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5edc81b0 0x560c5f1f0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39328/0 (socket says 172.21.15.134:39328) 2024-07-26T11:19:18.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.731+0000 7f3a7b400640 1 -- 172.21.15.134:0/4047885964 learned_addr learned my addr 172.21.15.134:0/4047885964 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.731+0000 7f3a72000640 1 --2- 172.21.15.134:0/4047885964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5eda74d0 0x560c5f17a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.731+0000 7f3a7aa00640 1 --2- 172.21.15.134:0/4047885964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5ed9f2f0 0x560c5f179ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.732+0000 7f3a7b400640 1 -- 172.21.15.134:0/4047885964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5eda74d0 msgr2=0x560c5f17a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.732+0000 7f3a7b400640 1 --2- 172.21.15.134:0/4047885964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5eda74d0 0x560c5f17a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.732+0000 7f3a7b400640 1 -- 172.21.15.134:0/4047885964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5ed9f2f0 msgr2=0x560c5f179ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.732+0000 7f3a7b400640 1 --2- 172.21.15.134:0/4047885964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5ed9f2f0 0x560c5f179ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.732+0000 7f3a7b400640 1 -- 172.21.15.134:0/4047885964 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c5f1f0690 con 0x560c5edc81b0 2024-07-26T11:19:18.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.732+0000 7f3a72000640 1 --2- 172.21.15.134:0/4047885964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5eda74d0 0x560c5f17a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:19:18.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.732+0000 7f3a7b400640 1 --2- 172.21.15.134:0/4047885964 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5edc81b0 0x560c5f1f0120 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3a74024b40 tx=0x7f3a740246a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92538af40c43682b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:18.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.733+0000 7f3a68c00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:19:18.736 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.733+0000 7f3a68c00640 1 -- 172.21.15.134:0/4047885964 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7403c060 con 0x560c5edc81b0 2024-07-26T11:19:18.736 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.734+0000 7f3a88c33740 1 -- 172.21.15.134:0/4047885964 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560c5f0ddc60 con 0x560c5edc81b0 2024-07-26T11:19:18.736 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.734+0000 7f3a68c00640 1 -- 172.21.15.134:0/4047885964 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3a74012040 con 0x560c5edc81b0 2024-07-26T11:19:18.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.735+0000 7f3a68c00640 1 -- 172.21.15.134:0/4047885964 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3a74043050 con 0x560c5edc81b0 2024-07-26T11:19:18.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.735+0000 7f3a88c33740 10 client.15768 fetch_fsmap learned FSMap version 31 2024-07-26T11:19:18.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.735+0000 7f3a88c33740 1 -- 172.21.15.134:0/4047885964 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560c5f1f1920 con 0x560c5edc81b0 2024-07-26T11:19:18.740 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.738+0000 7f3a68c00640 1 -- 172.21.15.134:0/4047885964 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560c5f0ddc60 con 0x560c5edc81b0 2024-07-26T11:19:18.740 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.738+0000 7f3a68c00640 1 -- 172.21.15.134:0/4047885964 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f3a74023040 con 0x560c5edc81b0 2024-07-26T11:19:18.740 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.738+0000 7f3a88c33740 10 client.15768 fetch_fsmap finished waiting for FSMap version 31 2024-07-26T11:19:18.740 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.738+0000 7f3a88c33740 10 client.15768 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:19:18.741 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.738+0000 7f3a88c33740 20 client.15768 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:18.741 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.738+0000 7f3a88c33740 1 --2- 172.21.15.134:0/4047885964 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560c5ed5c1c0 0x560c5f1f40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:18.741 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.738+0000 7f3a88c33740 4 client.15768 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:19:18.741 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.738+0000 7f3a88c33740 1 -- 172.21.15.134:0/4047885964 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560c5f0db3a0 con 0x560c5ed5c1c0 2024-07-26T11:19:18.742 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.740+0000 7f3a7aa00640 1 --2- 172.21.15.134:0/4047885964 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560c5ed5c1c0 0x560c5f1f40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:18.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.742+0000 7f3a7aa00640 1 --2- 172.21.15.134:0/4047885964 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560c5ed5c1c0 0x560c5f1f40b0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d61a0d9f38a79fe server_cookie=feb712414f47263 in_seq=0 out_seq=0 2024-07-26T11:19:18.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:18.743+0000 7f3a68c00640 10 client.15768 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:18 smithi193 ceph-mon[37473]: pgmap v580: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 215 MiB/s wr, 1.46k op/s 2024-07-26T11:19:19.587 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:19:20.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.488+0000 7f3a68c00640 1 -- 172.21.15.134:0/4047885964 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560c5f0db3a0 con 0x560c5ed5c1c0 2024-07-26T11:19:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.488+0000 7f3a68c00640 10 client.15768 handle_command_reply: tid=0 2024-07-26T11:19:20.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.488+0000 7f3a88c33740 1 client.15768 shutdown 2024-07-26T11:19:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.490+0000 7f3a88c33740 1 -- 172.21.15.134:0/4047885964 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5edc81b0 msgr2=0x560c5f1f0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:20.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.490+0000 7f3a88c33740 1 --2- 172.21.15.134:0/4047885964 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5edc81b0 0x560c5f1f0120 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3a74024b40 tx=0x7f3a740246a0 comp rx=0 tx=0).stop 2024-07-26T11:19:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.490+0000 7f3a88c33740 1 -- 172.21.15.134:0/4047885964 shutdown_connections 2024-07-26T11:19:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.490+0000 7f3a88c33740 1 --2- 172.21.15.134:0/4047885964 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560c5ed5c1c0 0x560c5f1f40b0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.490+0000 7f3a88c33740 1 --2- 172.21.15.134:0/4047885964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c5eda74d0 0x560c5f17a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.490+0000 7f3a88c33740 1 --2- 172.21.15.134:0/4047885964 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560c5edc81b0 0x560c5f1f0120 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.490+0000 7f3a88c33740 1 --2- 172.21.15.134:0/4047885964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c5ed9f2f0 0x560c5f179ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.490+0000 7f3a88c33740 1 -- 172.21.15.134:0/4047885964 >> 172.21.15.134:0/4047885964 conn(0x560c5edbc300 msgr2=0x560c5f175480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.490+0000 7f3a68c00640 0 client.15768 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.491+0000 7f3a88c33740 1 -- 172.21.15.134:0/4047885964 shutdown_connections 2024-07-26T11:19:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.492+0000 7f3a88c33740 1 -- 172.21.15.134:0/4047885964 wait complete. 2024-07-26T11:19:20.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.492+0000 7f3a88c33740 20 client.15768 trim_cache size 0 max 16384 2024-07-26T11:19:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:19:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:19:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:19:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:19:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.494+0000 7f3a7be00640 1 -- 172.21.15.134:0/3468624763 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3a600b3dc0 msgr2=0x7f3a600b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.494+0000 7f3a7be00640 1 --2- 172.21.15.134:0/3468624763 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3a600b3dc0 0x7f3a600b62d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3a74013b70 tx=0x7f3a74009710 comp rx=0 tx=0).stop 2024-07-26T11:19:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.494+0000 7f3a7be00640 1 -- 172.21.15.134:0/3468624763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a7c10a740 msgr2=0x7f3a7c1a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.494+0000 7f3a7be00640 1 --2- 172.21.15.134:0/3468624763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a7c10a740 0x7f3a7c1a4d00 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c06c0f0 tx=0x7f3a6c00cdd0 comp rx=0 tx=0).stop 2024-07-26T11:19:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.495+0000 7f3a7be00640 1 -- 172.21.15.134:0/3468624763 shutdown_connections 2024-07-26T11:19:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.495+0000 7f3a7be00640 1 --2- 172.21.15.134:0/3468624763 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3a600b3dc0 0x7f3a600b62d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.495+0000 7f3a7be00640 1 --2- 172.21.15.134:0/3468624763 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a7c10a740 0x7f3a7c1a4d00 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.495+0000 7f3a7be00640 1 --2- 172.21.15.134:0/3468624763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a7c118330 0x7f3a7c1a9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.495+0000 7f3a7be00640 1 --2- 172.21.15.134:0/3468624763 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3a7c109df0 0x7f3a7c1a4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.495+0000 7f3a7be00640 1 -- 172.21.15.134:0/3468624763 >> 172.21.15.134:0/3468624763 conn(0x7f3a7c070630 msgr2=0x7f3a7c118920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.495+0000 7f3a7be00640 1 -- 172.21.15.134:0/3468624763 shutdown_connections 2024-07-26T11:19:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:20.495+0000 7f3a7be00640 1 -- 172.21.15.134:0/3468624763 wait complete. 2024-07-26T11:19:20.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:20 smithi134 ceph-mon[31966]: pgmap v581: 129 pgs: 129 active+clean; 19 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 259 MiB/s wr, 1.93k op/s 2024-07-26T11:19:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:20 smithi189 ceph-mon[36790]: pgmap v581: 129 pgs: 129 active+clean; 19 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 259 MiB/s wr, 1.93k op/s 2024-07-26T11:19:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:20 smithi193 ceph-mon[37473]: pgmap v581: 129 pgs: 129 active+clean; 19 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 259 MiB/s wr, 1.93k op/s 2024-07-26T11:19:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:22 smithi134 ceph-mon[31966]: pgmap v582: 129 pgs: 129 active+clean; 19 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 182 MiB/s wr, 1.59k op/s 2024-07-26T11:19:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:22 smithi189 ceph-mon[36790]: pgmap v582: 129 pgs: 129 active+clean; 19 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 182 MiB/s wr, 1.59k op/s 2024-07-26T11:19:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:22 smithi193 ceph-mon[37473]: pgmap v582: 129 pgs: 129 active+clean; 19 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 182 MiB/s wr, 1.59k op/s 2024-07-26T11:19:24.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:24 smithi134 ceph-mon[31966]: pgmap v583: 129 pgs: 129 active+clean; 19 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 203 MiB/s wr, 1.76k op/s 2024-07-26T11:19:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:24 smithi189 ceph-mon[36790]: pgmap v583: 129 pgs: 129 active+clean; 19 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 203 MiB/s wr, 1.76k op/s 2024-07-26T11:19:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:24 smithi193 ceph-mon[37473]: pgmap v583: 129 pgs: 129 active+clean; 19 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 203 MiB/s wr, 1.76k op/s 2024-07-26T11:19:26.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:26 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:19:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:26 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:19:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:26 smithi189 ceph-mon[36790]: pgmap v584: 129 pgs: 129 active+clean; 19 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 202 MiB/s wr, 1.74k op/s 2024-07-26T11:19:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:26 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:19:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:26 smithi193 ceph-mon[37473]: pgmap v584: 129 pgs: 129 active+clean; 19 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 202 MiB/s wr, 1.74k op/s 2024-07-26T11:19:27.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:27 smithi134 ceph-mon[31966]: pgmap v584: 129 pgs: 129 active+clean; 19 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 202 MiB/s wr, 1.74k op/s 2024-07-26T11:19:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:28 smithi134 ceph-mon[31966]: pgmap v585: 129 pgs: 129 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 183 MiB/s wr, 1.52k op/s 2024-07-26T11:19:28.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:28 smithi189 ceph-mon[36790]: pgmap v585: 129 pgs: 129 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 183 MiB/s wr, 1.52k op/s 2024-07-26T11:19:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:28 smithi193 ceph-mon[37473]: pgmap v585: 129 pgs: 129 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 183 MiB/s wr, 1.52k op/s 2024-07-26T11:19:30.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:30 smithi134 ceph-mon[31966]: pgmap v586: 129 pgs: 129 active+clean; 20 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 231 MiB/s wr, 1.77k op/s 2024-07-26T11:19:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:30 smithi189 ceph-mon[36790]: pgmap v586: 129 pgs: 129 active+clean; 20 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 231 MiB/s wr, 1.77k op/s 2024-07-26T11:19:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:30 smithi193 ceph-mon[37473]: pgmap v586: 129 pgs: 129 active+clean; 20 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 231 MiB/s wr, 1.77k op/s 2024-07-26T11:19:32.437 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:32 smithi134 ceph-mon[31966]: pgmap v587: 129 pgs: 129 active+clean; 21 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 202 MiB/s wr, 1.44k op/s 2024-07-26T11:19:32.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:32 smithi189 ceph-mon[36790]: pgmap v587: 129 pgs: 129 active+clean; 21 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 202 MiB/s wr, 1.44k op/s 2024-07-26T11:19:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:32 smithi193 ceph-mon[37473]: pgmap v587: 129 pgs: 129 active+clean; 21 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 202 MiB/s wr, 1.44k op/s 2024-07-26T11:19:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.877+0000 7f50e3800640 1 -- 172.21.15.134:0/74731138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50dc10a090 msgr2=0x7f50dc10a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.877+0000 7f50e3800640 1 --2- 172.21.15.134:0/74731138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50dc10a090 0x7f50dc10a4a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009be0 tx=0x7f50d402a6c0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.879+0000 7f50e3800640 1 -- 172.21.15.134:0/74731138 shutdown_connections 2024-07-26T11:19:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.879+0000 7f50e3800640 1 --2- 172.21.15.134:0/74731138 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50dc10f430 0x7f50dc10f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.882+0000 7f50e3800640 1 --2- 172.21.15.134:0/74731138 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f50dc10fd80 0x7f50dc1105a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.882+0000 7f50e3800640 1 --2- 172.21.15.134:0/74731138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50dc10a090 0x7f50dc10a4a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.882+0000 7f50e3800640 1 -- 172.21.15.134:0/74731138 >> 172.21.15.134:0/74731138 conn(0x7f50dc070800 msgr2=0x7f50dc070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.884+0000 7f50e3800640 1 -- 172.21.15.134:0/74731138 shutdown_connections 2024-07-26T11:19:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.884+0000 7f50e3800640 1 -- 172.21.15.134:0/74731138 wait complete. 2024-07-26T11:19:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.886+0000 7f50e3800640 1 Processor -- start 2024-07-26T11:19:33.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.887+0000 7f50e3800640 1 -- start start 2024-07-26T11:19:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.889+0000 7f50e3800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f50dc10a090 0x7f50dc11fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.889+0000 7f50e3800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50dc10f430 0x7f50dc119e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.889+0000 7f50e3800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50dc10fd80 0x7f50dc11a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.889+0000 7f50e3800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc112260 con 0x7f50dc10a090 2024-07-26T11:19:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.889+0000 7f50e3800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc1120e0 con 0x7f50dc10fd80 2024-07-26T11:19:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.889+0000 7f50e3800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc1123e0 con 0x7f50dc10f430 2024-07-26T11:19:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.889+0000 7f50e0e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f50dc10a090 0x7f50dc11fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.889+0000 7f50e0e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f50dc10a090 0x7f50dc11fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:46898/0 (socket says 172.21.15.134:46898) 2024-07-26T11:19:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.889+0000 7f50e0e00640 1 -- 172.21.15.134:0/156464719 learned_addr learned my addr 172.21.15.134:0/156464719 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.890+0000 7f50e1800640 1 --2- 172.21.15.134:0/156464719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50dc10fd80 0x7f50dc11a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.890+0000 7f50dbe00640 1 --2- 172.21.15.134:0/156464719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50dc10f430 0x7f50dc119e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.891+0000 7f50e0e00640 1 -- 172.21.15.134:0/156464719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50dc10f430 msgr2=0x7f50dc119e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.891+0000 7f50e0e00640 1 --2- 172.21.15.134:0/156464719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50dc10f430 0x7f50dc119e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.892+0000 7f50e0e00640 1 -- 172.21.15.134:0/156464719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50dc10fd80 msgr2=0x7f50dc11a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.892+0000 7f50e0e00640 1 --2- 172.21.15.134:0/156464719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50dc10fd80 0x7f50dc11a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.892+0000 7f50e0e00640 1 -- 172.21.15.134:0/156464719 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50dc11a8c0 con 0x7f50dc10a090 2024-07-26T11:19:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.892+0000 7f50dbe00640 1 --2- 172.21.15.134:0/156464719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50dc10f430 0x7f50dc119e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:19:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.893+0000 7f50e0e00640 1 --2- 172.21.15.134:0/156464719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f50dc10a090 0x7f50dc11fb90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f50d4034040 tx=0x7f50d4003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.894+0000 7f50d9600640 1 -- 172.21.15.134:0/156464719 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d4040020 con 0x7f50dc10a090 2024-07-26T11:19:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.894+0000 7f50d9600640 1 -- 172.21.15.134:0/156464719 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50d4030e20 con 0x7f50dc10a090 2024-07-26T11:19:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.895+0000 7f50d9600640 1 -- 172.21.15.134:0/156464719 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d4036040 con 0x7f50dc10a090 2024-07-26T11:19:33.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.896+0000 7f50e3800640 1 -- 172.21.15.134:0/156464719 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50dc11aaf0 con 0x7f50dc10a090 2024-07-26T11:19:33.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.897+0000 7f50e3800640 1 -- 172.21.15.134:0/156464719 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50dc1bdca0 con 0x7f50dc10a090 2024-07-26T11:19:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.906+0000 7f50d9600640 1 -- 172.21.15.134:0/156464719 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f50d4043020 con 0x7f50dc10a090 2024-07-26T11:19:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.910+0000 7f50d9600640 1 --2- 172.21.15.134:0/156464719 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f50c00b3d70 0x7f50c00b6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.914 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.910+0000 7f50d9600640 1 -- 172.21.15.134:0/156464719 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f50d4104af0 con 0x7f50dc10a090 2024-07-26T11:19:33.915 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.913+0000 7f50dbe00640 1 --2- 172.21.15.134:0/156464719 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f50c00b3d70 0x7f50c00b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:33.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.913+0000 7f50ea9bc740 1 Processor -- start 2024-07-26T11:19:33.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.914+0000 7f50ea9bc740 1 -- start start 2024-07-26T11:19:33.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.915+0000 7f50ea9bc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a49a1f1b0 0x556a49dc2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.915+0000 7f50ea9bc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dc2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.915+0000 7f50ea9bc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 0x556a49dc34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.915+0000 7f50ea9bc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556a49d37270 con 0x556a49a1f1b0 2024-07-26T11:19:33.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.915+0000 7f50ea9bc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556a49d26770 con 0x556a499f62f0 2024-07-26T11:19:33.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.915+0000 7f50ea9bc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556a49d2a7b0 con 0x556a499fe4d0 2024-07-26T11:19:33.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.915+0000 7f50dbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dc2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.915+0000 7f50dbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dc2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:45738/0 (socket says 172.21.15.134:45738) 2024-07-26T11:19:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.915+0000 7f50dbe00640 1 -- 172.21.15.134:0/1413122803 learned_addr learned my addr 172.21.15.134:0/1413122803 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.916+0000 7f50dbe00640 1 --2- 172.21.15.134:0/156464719 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f50c00b3d70 0x7f50c00b6280 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f50dc070f00 tx=0x7f50cc00a300 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:33.919 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.916+0000 7f50dbe00640 1 -- 172.21.15.134:0/1413122803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 msgr2=0x556a49dc34c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:19:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.916+0000 7f50e1800640 1 --2- 172.21.15.134:0/1413122803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 0x556a49dc34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.917+0000 7f50dbe00640 1 --2- 172.21.15.134:0/1413122803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 0x556a49dc34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.917+0000 7f50dbe00640 1 -- 172.21.15.134:0/1413122803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a49a1f1b0 msgr2=0x556a49dc2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.917+0000 7f50dbe00640 1 --2- 172.21.15.134:0/1413122803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a49a1f1b0 0x556a49dc2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.917+0000 7f50dbe00640 1 -- 172.21.15.134:0/1413122803 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a49a8a790 con 0x556a499f62f0 2024-07-26T11:19:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.917+0000 7f50e1800640 1 --2- 172.21.15.134:0/1413122803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 0x556a49dc34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:19:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.918+0000 7f50dbe00640 1 --2- 172.21.15.134:0/1413122803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dc2f80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0107f0 tx=0x7f50cc01a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=468e847852fd01a8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.919+0000 7f50d2000640 1 -- 172.21.15.134:0/1413122803 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc016040 con 0x556a499f62f0 2024-07-26T11:19:33.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.919+0000 7f50d2000640 1 -- 172.21.15.134:0/1413122803 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50cc01bd80 con 0x556a499f62f0 2024-07-26T11:19:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.919+0000 7f50d2000640 1 -- 172.21.15.134:0/1413122803 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc016040 con 0x556a499f62f0 2024-07-26T11:19:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.921+0000 7f50ea9bc740 1 -- 172.21.15.134:0/1413122803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 msgr2=0x556a49dc2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.921+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/1413122803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dc2f80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0107f0 tx=0x7f50cc01a000 comp rx=0 tx=0).stop 2024-07-26T11:19:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.921+0000 7f50ea9bc740 1 -- 172.21.15.134:0/1413122803 shutdown_connections 2024-07-26T11:19:33.923 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.921+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/1413122803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 0x556a49dc34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.921+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/1413122803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dc2f80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.921+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/1413122803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a49a1f1b0 0x556a49dc2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.921+0000 7f50ea9bc740 1 -- 172.21.15.134:0/1413122803 >> 172.21.15.134:0/1413122803 conn(0x556a49a13300 msgr2=0x556a49dc0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:33.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.922+0000 7f50ea9bc740 1 -- 172.21.15.134:0/1413122803 shutdown_connections 2024-07-26T11:19:33.928 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.925+0000 7f50ea9bc740 1 -- 172.21.15.134:0/1413122803 wait complete. 2024-07-26T11:19:33.928 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.925+0000 7f50ea9bc740 1 -- start start 2024-07-26T11:19:33.928 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.926+0000 7f50ea9bc740 1 Processor -- start 2024-07-26T11:19:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.928+0000 7f50ea9bc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dce660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.928+0000 7f50ea9bc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 0x556a49dcebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.929+0000 7f50e0e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dce660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.929+0000 7f50dbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 0x556a49dcebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.929+0000 7f50dbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 0x556a49dcebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:44862/0 (socket says 172.21.15.134:44862) 2024-07-26T11:19:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.929+0000 7f50e0e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dce660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:45744/0 (socket says 172.21.15.134:45744) 2024-07-26T11:19:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.929+0000 7f50dbe00640 1 -- 172.21.15.134:0/608787898 learned_addr learned my addr 172.21.15.134:0/608787898 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.929+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/608787898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a49a1f1b0 0x556a49dda580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.930+0000 7f50ea9bc740 1 -- 172.21.15.134:0/608787898 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556a49c60ab0 con 0x556a49a1f1b0 2024-07-26T11:19:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.930+0000 7f50ea9bc740 1 -- 172.21.15.134:0/608787898 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556a49cc3d50 con 0x556a499f62f0 2024-07-26T11:19:33.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.930+0000 7f50e1800640 1 --2- 172.21.15.134:0/608787898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a49a1f1b0 0x556a49dda580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:33.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.931+0000 7f50ea9bc740 1 -- 172.21.15.134:0/608787898 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556a49dd1210 con 0x556a499fe4d0 2024-07-26T11:19:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.935+0000 7f50e0e00640 1 -- 172.21.15.134:0/608787898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 msgr2=0x556a49dcebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.935+0000 7f50e0e00640 1 --2- 172.21.15.134:0/608787898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 0x556a49dcebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.935+0000 7f50e0e00640 1 -- 172.21.15.134:0/608787898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a49a1f1b0 msgr2=0x556a49dda580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.935+0000 7f50e0e00640 1 --2- 172.21.15.134:0/608787898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a49a1f1b0 0x556a49dda580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.935+0000 7f50e0e00640 1 -- 172.21.15.134:0/608787898 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a49b03620 con 0x556a499f62f0 2024-07-26T11:19:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.936+0000 7f50ea9bc740 10 client.0 _ll_register_callbacks cb 0x556a49a36a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:33.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.936+0000 7f50ea9bc740 1 -- 172.21.15.134:0/608787898 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50cc01bd80 con 0x556a499f62f0 2024-07-26T11:19:33.939 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.936+0000 7f50e0e00640 1 --2- 172.21.15.134:0/608787898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dce660 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f50d4101ec0 tx=0x7f50d40cabb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1555c8f510da79b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:33.939 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.936+0000 7f50c6a00640 10 client.26099 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:19:33.939 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.937+0000 7f50c6a00640 1 -- 172.21.15.134:0/608787898 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d40e5060 con 0x556a499f62f0 2024-07-26T11:19:33.939 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.937+0000 7f50c6a00640 1 -- 172.21.15.134:0/608787898 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f50d40c7610 con 0x556a499f62f0 2024-07-26T11:19:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.937+0000 7f50c6a00640 1 -- 172.21.15.134:0/608787898 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f50d40cf980 con 0x556a499f62f0 2024-07-26T11:19:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.937+0000 7f50ea9bc740 10 client.26099 fetch_fsmap learned FSMap version 32 2024-07-26T11:19:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.937+0000 7f50ea9bc740 1 -- 172.21.15.134:0/608787898 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=32}) -- 0x556a49ddcd10 con 0x556a499f62f0 2024-07-26T11:19:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.938+0000 7f50c6a00640 1 -- 172.21.15.134:0/608787898 <== mon.2 v2:172.21.15.189:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f50d40d3fa0 con 0x556a499f62f0 2024-07-26T11:19:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.938+0000 7f50ea9bc740 10 client.26099 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:19:33.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.938+0000 7f50ea9bc740 10 client.26099 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:19:33.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.938+0000 7f50ea9bc740 20 client.26099 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:33.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.938+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/608787898 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556a499ddd80 0x556a49de04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.939+0000 7f50ea9bc740 4 client.26099 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:19:33.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.939+0000 7f50ea9bc740 1 -- 172.21.15.134:0/608787898 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a49d391c0 con 0x556a499ddd80 2024-07-26T11:19:33.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.939+0000 7f50dbe00640 1 --2- 172.21.15.134:0/608787898 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556a499ddd80 0x556a49de04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:33.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.941+0000 7f50dbe00640 1 --2- 172.21.15.134:0/608787898 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556a499ddd80 0x556a49de04c0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be76578c1d0504ac server_cookie=cbb65ae0b33a87d7 in_seq=0 out_seq=0 2024-07-26T11:19:33.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.941+0000 7f50c6a00640 10 client.26099 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:33.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.943+0000 7f50c6a00640 1 -- 172.21.15.134:0/608787898 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556a49d391c0 con 0x556a499ddd80 2024-07-26T11:19:33.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.943+0000 7f50c6a00640 10 client.26099 handle_command_reply: tid=0 2024-07-26T11:19:33.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.944+0000 7f50ea9bc740 1 client.26099 shutdown 2024-07-26T11:19:33.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.944+0000 7f50ea9bc740 1 -- 172.21.15.134:0/608787898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 msgr2=0x556a49dce660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:33.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.944+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/608787898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dce660 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f50d4101ec0 tx=0x7f50d40cabb0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.945+0000 7f50ea9bc740 1 -- 172.21.15.134:0/608787898 shutdown_connections 2024-07-26T11:19:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.945+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/608787898 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556a499ddd80 0x556a49de04c0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.945+0000 7f50c6a00640 0 client.26099 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.945+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/608787898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 0x556a49dcebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.945+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/608787898 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a49a1f1b0 0x556a49dda580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.945+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/608787898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dce660 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.945+0000 7f50ea9bc740 1 -- 172.21.15.134:0/608787898 >> 172.21.15.134:0/608787898 conn(0x556a49a13300 msgr2=0x556a49dbf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:33.948 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.945+0000 7f50ea9bc740 1 -- 172.21.15.134:0/608787898 shutdown_connections 2024-07-26T11:19:33.948 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.946+0000 7f50ea9bc740 1 -- 172.21.15.134:0/608787898 wait complete. 2024-07-26T11:19:33.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.946+0000 7f50ea9bc740 20 client.26099 trim_cache size 0 max 16384 2024-07-26T11:19:33.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.978+0000 7f50ea9bc740 1 Processor -- start 2024-07-26T11:19:33.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.979+0000 7f50ea9bc740 1 -- start start 2024-07-26T11:19:33.982 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.981+0000 7f50ea9bc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a499ddd80 0x556a49dc0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.981+0000 7f50ea9bc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499f62f0 0x556a49dc8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.983 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.982+0000 7f50ea9bc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499fe4d0 0x556a49dda370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.982+0000 7f50dbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499f62f0 0x556a49dc8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:33.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.982+0000 7f50ea9bc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc1c7990 con 0x556a499ddd80 2024-07-26T11:19:33.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.982+0000 7f50ea9bc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc1c94d0 con 0x556a499fe4d0 2024-07-26T11:19:33.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.982+0000 7f50ea9bc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc1c8060 con 0x556a499f62f0 2024-07-26T11:19:33.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.982+0000 7f50e0e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a499ddd80 0x556a49dc0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:33.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.982+0000 7f50e1800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499fe4d0 0x556a49dda370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:33.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.983+0000 7f50e1800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499fe4d0 0x556a49dda370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:45746/0 (socket says 172.21.15.134:45746) 2024-07-26T11:19:33.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.983+0000 7f50e1800640 1 -- 172.21.15.134:0/2278741518 learned_addr learned my addr 172.21.15.134:0/2278741518 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:33.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.984+0000 7f50e1800640 1 -- 172.21.15.134:0/2278741518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499f62f0 msgr2=0x556a49dc8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:33.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.984+0000 7f50e1800640 1 --2- 172.21.15.134:0/2278741518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499f62f0 0x556a49dc8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.984+0000 7f50e1800640 1 -- 172.21.15.134:0/2278741518 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a499ddd80 msgr2=0x556a49dc0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:33.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.984+0000 7f50e1800640 1 --2- 172.21.15.134:0/2278741518 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a499ddd80 0x556a49dc0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.984+0000 7f50e1800640 1 -- 172.21.15.134:0/2278741518 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a49dc4f70 con 0x556a499fe4d0 2024-07-26T11:19:33.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.985+0000 7f50e1800640 1 --2- 172.21.15.134:0/2278741518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499fe4d0 0x556a49dda370 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f50c800abf0 tx=0x7f50c80117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aed1df36ede66b69 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.989+0000 7f50d1600640 1 -- 172.21.15.134:0/2278741518 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c8004020 con 0x556a499fe4d0 2024-07-26T11:19:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.989+0000 7f50d1600640 1 -- 172.21.15.134:0/2278741518 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50c80041c0 con 0x556a499fe4d0 2024-07-26T11:19:33.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.989+0000 7f50d1600640 1 -- 172.21.15.134:0/2278741518 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c801b800 con 0x556a499fe4d0 2024-07-26T11:19:33.993 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.990+0000 7f50ea9bc740 1 -- 172.21.15.134:0/2278741518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499fe4d0 msgr2=0x556a49dda370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.990+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/2278741518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499fe4d0 0x556a49dda370 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f50c800abf0 tx=0x7f50c80117b0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.991+0000 7f50ea9bc740 1 -- 172.21.15.134:0/2278741518 shutdown_connections 2024-07-26T11:19:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.991+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/2278741518 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499f62f0 0x556a49dc8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.991+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/2278741518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499fe4d0 0x556a49dda370 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.991+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/2278741518 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a499ddd80 0x556a49dc0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.991+0000 7f50ea9bc740 1 -- 172.21.15.134:0/2278741518 >> 172.21.15.134:0/2278741518 conn(0x556a49a13300 msgr2=0x556a49dcf970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.992+0000 7f50ea9bc740 1 -- 172.21.15.134:0/2278741518 shutdown_connections 2024-07-26T11:19:33.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.992+0000 7f50ea9bc740 1 -- 172.21.15.134:0/2278741518 wait complete. 2024-07-26T11:19:33.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.992+0000 7f50ea9bc740 1 -- start start 2024-07-26T11:19:33.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.993+0000 7f50ea9bc740 1 Processor -- start 2024-07-26T11:19:33.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.994+0000 7f50ea9bc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a499ddd80 0x556a49dd6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.994+0000 7f50ea9bc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dd1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.994+0000 7f50ea9bc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 0x556a49dd2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:33.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.994+0000 7f50ea9bc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc1c8e00 con 0x556a499ddd80 2024-07-26T11:19:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.994+0000 7f50ea9bc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc1c8730 con 0x556a499f62f0 2024-07-26T11:19:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.994+0000 7f50ea9bc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc1c5970 con 0x556a499fe4d0 2024-07-26T11:19:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.994+0000 7f50ea9bc740 10 client.0 _ll_register_callbacks cb 0x7f50d4009480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.995+0000 7f50dbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dd1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.995+0000 7f50dbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dd1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:45756/0 (socket says 172.21.15.134:45756) 2024-07-26T11:19:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.995+0000 7f50dbe00640 1 -- 172.21.15.134:0/1231601799 learned_addr learned my addr 172.21.15.134:0/1231601799 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.995+0000 7f50e1800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 0x556a49dd2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:33.998 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.995+0000 7f50dbe00640 1 -- 172.21.15.134:0/1231601799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 msgr2=0x556a49dd2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:33.998 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.995+0000 7f50dbe00640 1 --2- 172.21.15.134:0/1231601799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 0x556a49dd2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.998 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.995+0000 7f50dbe00640 1 -- 172.21.15.134:0/1231601799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a499ddd80 msgr2=0x556a49dd6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:33.998 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.996+0000 7f50e0e00640 1 --2- 172.21.15.134:0/1231601799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a499ddd80 0x556a49dd6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:33.998 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.996+0000 7f50dbe00640 1 --2- 172.21.15.134:0/1231601799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a499ddd80 0x556a49dd6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:33.998 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.996+0000 7f50dbe00640 1 -- 172.21.15.134:0/1231601799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a49dd2780 con 0x556a499f62f0 2024-07-26T11:19:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.996+0000 7f50e0e00640 1 --2- 172.21.15.134:0/1231601799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a499ddd80 0x556a49dd6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:19:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.997+0000 7f50c6a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:19:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.997+0000 7f50dbe00640 1 --2- 172.21.15.134:0/1231601799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dd1d00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f50cc01c010 tx=0x7f50cc01ba60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=981c8c1e93f683e3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.997+0000 7f50c6a00640 1 -- 172.21.15.134:0/1231601799 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50cc03e060 con 0x556a499f62f0 2024-07-26T11:19:34.000 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.998+0000 7f50c6a00640 1 -- 172.21.15.134:0/1231601799 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f50cc01c650 con 0x556a499f62f0 2024-07-26T11:19:34.000 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.998+0000 7f50ea9bc740 1 -- 172.21.15.134:0/1231601799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50c80041c0 con 0x556a499f62f0 2024-07-26T11:19:34.000 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.998+0000 7f50c6a00640 1 -- 172.21.15.134:0/1231601799 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f50cc016040 con 0x556a499f62f0 2024-07-26T11:19:34.001 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.998+0000 7f50ea9bc740 10 client.26105 fetch_fsmap learned FSMap version 32 2024-07-26T11:19:34.001 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.999+0000 7f50ea9bc740 1 -- 172.21.15.134:0/1231601799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556a49e48880 con 0x556a499f62f0 2024-07-26T11:19:34.001 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.999+0000 7f50c6a00640 1 -- 172.21.15.134:0/1231601799 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50c80041c0 con 0x556a499f62f0 2024-07-26T11:19:34.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.999+0000 7f50c6a00640 1 -- 172.21.15.134:0/1231601799 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f50cc0307d0 con 0x556a499f62f0 2024-07-26T11:19:34.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:33.999+0000 7f50ea9bc740 10 client.26105 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:19:34.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.000+0000 7f50ea9bc740 10 client.26105 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:19:34.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.000+0000 7f50ea9bc740 20 client.26105 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:34.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.000+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/1231601799 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556a49a1f1b0 0x556a49e4ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.000+0000 7f50ea9bc740 4 client.26105 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:19:34.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.000+0000 7f50ea9bc740 1 -- 172.21.15.134:0/1231601799 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556a49d34c60 con 0x556a49a1f1b0 2024-07-26T11:19:34.003 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.000+0000 7f50e0e00640 1 --2- 172.21.15.134:0/1231601799 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556a49a1f1b0 0x556a49e4ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.003 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.001+0000 7f50c6a00640 10 client.26105 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:34.005 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.003+0000 7f50e0e00640 1 --2- 172.21.15.134:0/1231601799 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556a49a1f1b0 0x556a49e4ac30 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23fa1da36377a251 server_cookie=16a0d482573c8e16 in_seq=0 out_seq=0 2024-07-26T11:19:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.055+0000 7f50c6a00640 1 -- 172.21.15.134:0/1231601799 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556a49d34c60 con 0x556a49a1f1b0 2024-07-26T11:19:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.055+0000 7f50c6a00640 10 client.26105 handle_command_reply: tid=0 2024-07-26T11:19:34.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.055+0000 7f50ea9bc740 1 client.26105 shutdown 2024-07-26T11:19:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.055+0000 7f50ea9bc740 1 -- 172.21.15.134:0/1231601799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 msgr2=0x556a49dd1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.056+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/1231601799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dd1d00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f50cc01c010 tx=0x7f50cc01ba60 comp rx=0 tx=0).stop 2024-07-26T11:19:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.056+0000 7f50ea9bc740 1 -- 172.21.15.134:0/1231601799 shutdown_connections 2024-07-26T11:19:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.056+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/1231601799 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556a49a1f1b0 0x556a49e4ac30 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.056+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/1231601799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556a499fe4d0 0x556a49dd2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.056+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/1231601799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556a499f62f0 0x556a49dd1d00 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.056+0000 7f50ea9bc740 1 --2- 172.21.15.134:0/1231601799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556a499ddd80 0x556a49dd6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.056+0000 7f50ea9bc740 1 -- 172.21.15.134:0/1231601799 >> 172.21.15.134:0/1231601799 conn(0x556a49a13300 msgr2=0x556a49dc0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.056+0000 7f50c6a00640 0 client.26105 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.056+0000 7f50ea9bc740 1 -- 172.21.15.134:0/1231601799 shutdown_connections 2024-07-26T11:19:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.056+0000 7f50ea9bc740 1 -- 172.21.15.134:0/1231601799 wait complete. 2024-07-26T11:19:34.059 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.057+0000 7f50ea9bc740 20 client.26105 trim_cache size 0 max 16384 2024-07-26T11:19:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:19:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:19:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:19:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:19:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.059+0000 7f50e3800640 1 -- 172.21.15.134:0/156464719 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f50c00b3d70 msgr2=0x7f50c00b6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.059+0000 7f50e3800640 1 --2- 172.21.15.134:0/156464719 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f50c00b3d70 0x7f50c00b6280 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f50dc070f00 tx=0x7f50cc00a300 comp rx=0 tx=0).stop 2024-07-26T11:19:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.059+0000 7f50e3800640 1 -- 172.21.15.134:0/156464719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f50dc10a090 msgr2=0x7f50dc11fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.059+0000 7f50e3800640 1 --2- 172.21.15.134:0/156464719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f50dc10a090 0x7f50dc11fb90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f50d4034040 tx=0x7f50d4003f10 comp rx=0 tx=0).stop 2024-07-26T11:19:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.060+0000 7f50e3800640 1 -- 172.21.15.134:0/156464719 shutdown_connections 2024-07-26T11:19:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.060+0000 7f50e3800640 1 --2- 172.21.15.134:0/156464719 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f50c00b3d70 0x7f50c00b6280 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.060+0000 7f50e3800640 1 --2- 172.21.15.134:0/156464719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50dc10f430 0x7f50dc119e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.060+0000 7f50e3800640 1 --2- 172.21.15.134:0/156464719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50dc10fd80 0x7f50dc11a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.060+0000 7f50e3800640 1 --2- 172.21.15.134:0/156464719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f50dc10a090 0x7f50dc11fb90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.060+0000 7f50e3800640 1 -- 172.21.15.134:0/156464719 >> 172.21.15.134:0/156464719 conn(0x7f50dc070800 msgr2=0x7f50dc073760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.061+0000 7f50e3800640 1 -- 172.21.15.134:0/156464719 shutdown_connections 2024-07-26T11:19:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:34.061+0000 7f50e3800640 1 -- 172.21.15.134:0/156464719 wait complete. 2024-07-26T11:19:34.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.172+0000 7f8a8b000640 1 -- 172.21.15.134:0/2443784686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a84113e70 msgr2=0x7f8a84116340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.172+0000 7f8a8b000640 1 --2- 172.21.15.134:0/2443784686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a84113e70 0x7f8a84116340 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c009ca0 tx=0x7f8a7c02a8f0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.173+0000 7f8a8b000640 1 -- 172.21.15.134:0/2443784686 shutdown_connections 2024-07-26T11:19:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.173+0000 7f8a8b000640 1 --2- 172.21.15.134:0/2443784686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a84113e70 0x7f8a84116340 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.173+0000 7f8a8b000640 1 --2- 172.21.15.134:0/2443784686 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a84075800 0x7f8a84075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.173+0000 7f8a8b000640 1 --2- 172.21.15.134:0/2443784686 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a84076db0 0x7f8a840751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.173+0000 7f8a8b000640 1 -- 172.21.15.134:0/2443784686 >> 172.21.15.134:0/2443784686 conn(0x7f8a840ff010 msgr2=0x7f8a84101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:34.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.173+0000 7f8a8b000640 1 -- 172.21.15.134:0/2443784686 shutdown_connections 2024-07-26T11:19:34.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.175+0000 7f8a8b000640 1 -- 172.21.15.134:0/2443784686 wait complete. 2024-07-26T11:19:34.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.177+0000 7f8a8b000640 1 Processor -- start 2024-07-26T11:19:34.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.177+0000 7f8a8b000640 1 -- start start 2024-07-26T11:19:34.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.178+0000 7f8a8b000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a84075800 0x7f8a841a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.178+0000 7f8a8b000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a84076db0 0x7f8a841a0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.178+0000 7f8a8b000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a84113e70 0x7f8a841a5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.178+0000 7f8a8b000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8a84119b10 con 0x7f8a84113e70 2024-07-26T11:19:34.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.178+0000 7f8a8b000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8a84119990 con 0x7f8a84075800 2024-07-26T11:19:34.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.178+0000 7f8a8b000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8a84119c90 con 0x7f8a84076db0 2024-07-26T11:19:34.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.179+0000 7f8a89000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a84113e70 0x7f8a841a5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.179+0000 7f8a83e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a84075800 0x7f8a841a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.179+0000 7f8a83e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a84075800 0x7f8a841a04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:45782/0 (socket says 172.21.15.134:45782) 2024-07-26T11:19:34.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.179+0000 7f8a83e00640 1 -- 172.21.15.134:0/4237101718 learned_addr learned my addr 172.21.15.134:0/4237101718 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:34.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.179+0000 7f8a83400640 1 --2- 172.21.15.134:0/4237101718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a84076db0 0x7f8a841a0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.179+0000 7f8a83400640 1 -- 172.21.15.134:0/4237101718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a84075800 msgr2=0x7f8a841a04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:34.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.179+0000 7f8a83400640 1 --2- 172.21.15.134:0/4237101718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a84075800 0x7f8a841a04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.179+0000 7f8a83400640 1 -- 172.21.15.134:0/4237101718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a84113e70 msgr2=0x7f8a841a5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:34.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.180+0000 7f8a83400640 1 --2- 172.21.15.134:0/4237101718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a84113e70 0x7f8a841a5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.180+0000 7f8a83400640 1 -- 172.21.15.134:0/4237101718 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a841a6410 con 0x7f8a84076db0 2024-07-26T11:19:34.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.180+0000 7f8a89000640 1 --2- 172.21.15.134:0/4237101718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a84113e70 0x7f8a841a5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:19:34.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.181+0000 7f8a83400640 1 --2- 172.21.15.134:0/4237101718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a84076db0 0x7f8a841a0a40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000b3c0 tx=0x7f8a7000b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:34.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.183+0000 7f8a80c00640 1 -- 172.21.15.134:0/4237101718 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a70022640 con 0x7f8a84076db0 2024-07-26T11:19:34.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.183+0000 7f8a80c00640 1 -- 172.21.15.134:0/4237101718 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a70027070 con 0x7f8a84076db0 2024-07-26T11:19:34.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.183+0000 7f8a80c00640 1 -- 172.21.15.134:0/4237101718 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a70021b60 con 0x7f8a84076db0 2024-07-26T11:19:34.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.183+0000 7f8a8b000640 1 -- 172.21.15.134:0/4237101718 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a841a6700 con 0x7f8a84076db0 2024-07-26T11:19:34.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.183+0000 7f8a8b000640 1 -- 172.21.15.134:0/4237101718 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a8406c810 con 0x7f8a84076db0 2024-07-26T11:19:34.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.189+0000 7f8a91e37740 1 Processor -- start 2024-07-26T11:19:34.194 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.191+0000 7f8a91e37740 1 -- start start 2024-07-26T11:19:34.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.191+0000 7f8a91e37740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e28614e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.191+0000 7f8a91e37740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e28615370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.191+0000 7f8a91e37740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e2824f390 0x561e286158b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.191+0000 7f8a91e37740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x561e2858bbd0 con 0x561e28270070 2024-07-26T11:19:34.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.191+0000 7f8a91e37740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561e2857b0d0 con 0x561e282471b0 2024-07-26T11:19:34.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.191+0000 7f8a91e37740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561e2857f110 con 0x561e2824f390 2024-07-26T11:19:34.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.192+0000 7f8a89000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e28614e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.192+0000 7f8a89000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e28614e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:46936/0 (socket says 172.21.15.134:46936) 2024-07-26T11:19:34.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.192+0000 7f8a89000640 1 -- 172.21.15.134:0/4005912056 learned_addr learned my addr 172.21.15.134:0/4005912056 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:34.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.193+0000 7f8a83e00640 1 --2- 172.21.15.134:0/4005912056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e28615370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.193+0000 7f8a89000640 1 -- 172.21.15.134:0/4005912056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e2824f390 msgr2=0x561e286158b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:19:34.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.193+0000 7f8a89000640 1 --2- 172.21.15.134:0/4005912056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e2824f390 0x561e286158b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.193+0000 7f8a89000640 1 -- 172.21.15.134:0/4005912056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 msgr2=0x561e28615370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:34.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.193+0000 7f8a89000640 1 --2- 172.21.15.134:0/4005912056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e28615370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.193+0000 7f8a89000640 1 -- 172.21.15.134:0/4005912056 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e282db650 con 0x561e28270070 2024-07-26T11:19:34.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.193+0000 7f8a83e00640 1 --2- 172.21.15.134:0/4005912056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e28615370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:19:34.198 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.194+0000 7f8a80c00640 1 -- 172.21.15.134:0/4237101718 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f8a7002c460 con 0x7f8a84076db0 2024-07-26T11:19:34.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.197+0000 7f8a80c00640 1 --2- 172.21.15.134:0/4237101718 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8a680b3e70 0x7f8a680b6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.197+0000 7f8a80c00640 1 -- 172.21.15.134:0/4237101718 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f8a700f4680 con 0x7f8a84076db0 2024-07-26T11:19:34.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.197+0000 7f8a89000640 1 --2- 172.21.15.134:0/4005912056 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e28614e30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c009ca0 tx=0x7f8a7c009930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ca1e709c021c02 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:34.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.197+0000 7f8a83e00640 1 --2- 172.21.15.134:0/4237101718 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8a680b3e70 0x7f8a680b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.199+0000 7f8a79600640 1 -- 172.21.15.134:0/4005912056 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c031d20 con 0x561e28270070 2024-07-26T11:19:34.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.199+0000 7f8a79600640 1 -- 172.21.15.134:0/4005912056 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a7c036030 con 0x561e28270070 2024-07-26T11:19:34.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.199+0000 7f8a79600640 1 -- 172.21.15.134:0/4005912056 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c0305f0 con 0x561e28270070 2024-07-26T11:19:34.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.200+0000 7f8a91e37740 1 -- 172.21.15.134:0/4005912056 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 msgr2=0x561e28614e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:34.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.200+0000 7f8a83e00640 1 --2- 172.21.15.134:0/4237101718 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8a680b3e70 0x7f8a680b6380 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x561e28613ab0 tx=0x7f8a74005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:34.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.200+0000 7f8a91e37740 1 --2- 172.21.15.134:0/4005912056 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e28614e30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c009ca0 tx=0x7f8a7c009930 comp rx=0 tx=0).stop 2024-07-26T11:19:34.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.201+0000 7f8a91e37740 1 -- 172.21.15.134:0/4005912056 shutdown_connections 2024-07-26T11:19:34.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.201+0000 7f8a91e37740 1 --2- 172.21.15.134:0/4005912056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e2824f390 0x561e286158b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.201+0000 7f8a91e37740 1 --2- 172.21.15.134:0/4005912056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e28615370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.201+0000 7f8a91e37740 1 --2- 172.21.15.134:0/4005912056 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e28614e30 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.202+0000 7f8a91e37740 1 -- 172.21.15.134:0/4005912056 >> 172.21.15.134:0/4005912056 conn(0x561e28264460 msgr2=0x561e28612d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:34.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.202+0000 7f8a91e37740 1 -- 172.21.15.134:0/4005912056 shutdown_connections 2024-07-26T11:19:34.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.202+0000 7f8a91e37740 1 -- 172.21.15.134:0/4005912056 wait complete. 2024-07-26T11:19:34.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.203+0000 7f8a91e37740 1 -- start start 2024-07-26T11:19:34.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.203+0000 7f8a91e37740 1 Processor -- start 2024-07-26T11:19:34.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.204+0000 7f8a91e37740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e28620550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.204+0000 7f8a91e37740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e2824f390 0x561e28620a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.204+0000 7f8a91e37740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e2862c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.204+0000 7f8a91e37740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x561e283de700 con 0x561e28270070 2024-07-26T11:19:34.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.204+0000 7f8a91e37740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561e285176a0 con 0x561e282471b0 2024-07-26T11:19:34.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.204+0000 7f8a91e37740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561e286235b0 con 0x561e2824f390 2024-07-26T11:19:34.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.204+0000 7f8a91e37740 10 client.0 _ll_register_callbacks cb 0x561e28527150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:34.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.205+0000 7f8a83400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e2824f390 0x561e28620a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.205+0000 7f8a83400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e2824f390 0x561e28620a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:44916/0 (socket says 172.21.15.134:44916) 2024-07-26T11:19:34.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.205+0000 7f8a83400640 1 -- 172.21.15.134:0/912182669 learned_addr learned my addr 172.21.15.134:0/912182669 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:34.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.205+0000 7f8a83e00640 1 --2- 172.21.15.134:0/912182669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e28620550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.205+0000 7f8a83e00640 1 -- 172.21.15.134:0/912182669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e2824f390 msgr2=0x561e28620a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:34.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.205+0000 7f8a83e00640 1 --2- 172.21.15.134:0/912182669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e2824f390 0x561e28620a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.205+0000 7f8a89000640 1 --2- 172.21.15.134:0/912182669 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e2862c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.205+0000 7f8a83e00640 1 -- 172.21.15.134:0/912182669 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 msgr2=0x561e2862c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:34.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.205+0000 7f8a83e00640 1 --2- 172.21.15.134:0/912182669 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e2862c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.205+0000 7f8a83e00640 1 -- 172.21.15.134:0/912182669 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e2852e840 con 0x561e282471b0 2024-07-26T11:19:34.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.205+0000 7f8a89000640 1 --2- 172.21.15.134:0/912182669 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e2862c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:19:34.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.206+0000 7f8a83e00640 1 --2- 172.21.15.134:0/912182669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e28620550 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8a74013cc0 tx=0x7f8a740046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a14da99bde404c18 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:34.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.206+0000 7f8a6e000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:19:34.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.206+0000 7f8a6e000640 1 -- 172.21.15.134:0/912182669 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a740108d0 con 0x561e282471b0 2024-07-26T11:19:34.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.206+0000 7f8a91e37740 1 -- 172.21.15.134:0/912182669 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a7c036030 con 0x561e282471b0 2024-07-26T11:19:34.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.206+0000 7f8a6e000640 1 -- 172.21.15.134:0/912182669 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f8a74010a70 con 0x561e282471b0 2024-07-26T11:19:34.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.206+0000 7f8a6e000640 1 -- 172.21.15.134:0/912182669 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8a74010c70 con 0x561e282471b0 2024-07-26T11:19:34.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.206+0000 7f8a91e37740 10 client.26114 fetch_fsmap learned FSMap version 32 2024-07-26T11:19:34.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.207+0000 7f8a91e37740 1 -- 172.21.15.134:0/912182669 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561e2862f290 con 0x561e282471b0 2024-07-26T11:19:34.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.207+0000 7f8a6e000640 1 -- 172.21.15.134:0/912182669 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a7c036030 con 0x561e282471b0 2024-07-26T11:19:34.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.207+0000 7f8a6e000640 1 -- 172.21.15.134:0/912182669 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f8a7401e410 con 0x561e282471b0 2024-07-26T11:19:34.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.207+0000 7f8a91e37740 10 client.26114 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:19:34.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.207+0000 7f8a91e37740 10 client.26114 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:19:34.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.207+0000 7f8a91e37740 20 client.26114 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:34.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.208+0000 7f8a91e37740 1 --2- 172.21.15.134:0/912182669 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x561e2822e9f0 0x561e286328d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.208+0000 7f8a91e37740 4 client.26114 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:19:34.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.208+0000 7f8a91e37740 1 -- 172.21.15.134:0/912182669 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e2858db20 con 0x561e2822e9f0 2024-07-26T11:19:34.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.209+0000 7f8a83400640 1 --2- 172.21.15.134:0/912182669 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x561e2822e9f0 0x561e286328d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.219+0000 7f8a6e000640 10 client.26114 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:34.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.219+0000 7f8a83400640 1 --2- 172.21.15.134:0/912182669 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x561e2822e9f0 0x561e286328d0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c128cb5bb5f4410 server_cookie=57da3ee91348cb10 in_seq=0 out_seq=0 2024-07-26T11:19:34.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.222+0000 7f8a6e000640 1 -- 172.21.15.134:0/912182669 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561e2858db20 con 0x561e2822e9f0 2024-07-26T11:19:34.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.222+0000 7f8a6e000640 10 client.26114 handle_command_reply: tid=0 2024-07-26T11:19:34.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.222+0000 7f8a91e37740 1 client.26114 shutdown 2024-07-26T11:19:34.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.225+0000 7f8a91e37740 1 -- 172.21.15.134:0/912182669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 msgr2=0x561e28620550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:34.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.225+0000 7f8a91e37740 1 --2- 172.21.15.134:0/912182669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e28620550 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8a74013cc0 tx=0x7f8a740046c0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.225+0000 7f8a91e37740 1 -- 172.21.15.134:0/912182669 shutdown_connections 2024-07-26T11:19:34.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.225+0000 7f8a91e37740 1 --2- 172.21.15.134:0/912182669 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x561e2822e9f0 0x561e286328d0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.225+0000 7f8a91e37740 1 --2- 172.21.15.134:0/912182669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e2824f390 0x561e28620a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.225+0000 7f8a91e37740 1 --2- 172.21.15.134:0/912182669 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e2862c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.225+0000 7f8a91e37740 1 --2- 172.21.15.134:0/912182669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e28620550 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.225+0000 7f8a91e37740 1 -- 172.21.15.134:0/912182669 >> 172.21.15.134:0/912182669 conn(0x561e28264460 msgr2=0x561e28618010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:34.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.225+0000 7f8a6e000640 0 client.26114 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:34.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.226+0000 7f8a91e37740 1 -- 172.21.15.134:0/912182669 shutdown_connections 2024-07-26T11:19:34.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.227+0000 7f8a91e37740 1 -- 172.21.15.134:0/912182669 wait complete. 2024-07-26T11:19:34.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.227+0000 7f8a91e37740 20 client.26114 trim_cache size 0 max 16384 2024-07-26T11:19:34.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.268+0000 7f8a91e37740 1 Processor -- start 2024-07-26T11:19:34.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.268+0000 7f8a91e37740 1 -- start start 2024-07-26T11:19:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.269+0000 7f8a91e37740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e2861bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.269+0000 7f8a91e37740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e2824f390 0x561e28622180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.269+0000 7f8a91e37740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e28270070 0x561e2862c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.269+0000 7f8a91e37740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8a841ae790 con 0x561e2824f390 2024-07-26T11:19:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.269+0000 7f8a91e37740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8a841b02d0 con 0x561e282471b0 2024-07-26T11:19:34.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.269+0000 7f8a91e37740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8a841aee60 con 0x561e28270070 2024-07-26T11:19:34.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.270+0000 7f8a83e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e2861bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.270+0000 7f8a83e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e2861bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:45802/0 (socket says 172.21.15.134:45802) 2024-07-26T11:19:34.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.270+0000 7f8a83e00640 1 -- 172.21.15.134:0/4293377522 learned_addr learned my addr 172.21.15.134:0/4293377522 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:34.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.270+0000 7f8a89000640 1 --2- 172.21.15.134:0/4293377522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e28270070 0x561e2862c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.273+0000 7f8a89000640 1 -- 172.21.15.134:0/4293377522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 msgr2=0x561e2861bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:34.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.273+0000 7f8a89000640 1 --2- 172.21.15.134:0/4293377522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e2861bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.273+0000 7f8a89000640 1 -- 172.21.15.134:0/4293377522 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e2824f390 msgr2=0x561e28622180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:34.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.273+0000 7f8a89000640 1 --2- 172.21.15.134:0/4293377522 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e2824f390 0x561e28622180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.273+0000 7f8a89000640 1 -- 172.21.15.134:0/4293377522 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e28613690 con 0x561e28270070 2024-07-26T11:19:34.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.275+0000 7f8a89000640 1 --2- 172.21.15.134:0/4293377522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e28270070 0x561e2862c690 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c02f3d0 tx=0x7f8a7c000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afaa1428cf7d903b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:34.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.276+0000 7f8a78c00640 1 -- 172.21.15.134:0/4293377522 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c054070 con 0x561e28270070 2024-07-26T11:19:34.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.276+0000 7f8a78c00640 1 -- 172.21.15.134:0/4293377522 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a7c036050 con 0x561e28270070 2024-07-26T11:19:34.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.277+0000 7f8a91e37740 1 -- 172.21.15.134:0/4293377522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e28270070 msgr2=0x561e2862c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:34.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.277+0000 7f8a91e37740 1 --2- 172.21.15.134:0/4293377522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e28270070 0x561e2862c690 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c02f3d0 tx=0x7f8a7c000f40 comp rx=0 tx=0).stop 2024-07-26T11:19:34.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.278+0000 7f8a91e37740 1 -- 172.21.15.134:0/4293377522 shutdown_connections 2024-07-26T11:19:34.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.278+0000 7f8a91e37740 1 --2- 172.21.15.134:0/4293377522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e28270070 0x561e2862c690 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.278+0000 7f8a91e37740 1 --2- 172.21.15.134:0/4293377522 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e2824f390 0x561e28622180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.278+0000 7f8a91e37740 1 --2- 172.21.15.134:0/4293377522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e2861bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.278+0000 7f8a91e37740 1 -- 172.21.15.134:0/4293377522 >> 172.21.15.134:0/4293377522 conn(0x561e28264460 msgr2=0x561e28621a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:34.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.278+0000 7f8a91e37740 1 -- 172.21.15.134:0/4293377522 shutdown_connections 2024-07-26T11:19:34.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.278+0000 7f8a91e37740 1 -- 172.21.15.134:0/4293377522 wait complete. 2024-07-26T11:19:34.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.279+0000 7f8a91e37740 1 -- start start 2024-07-26T11:19:34.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.279+0000 7f8a91e37740 1 Processor -- start 2024-07-26T11:19:34.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.281+0000 7f8a91e37740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e28693810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.281+0000 7f8a91e37740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e2824f390 0x561e28693d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.282+0000 7f8a91e37740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e286277f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.282+0000 7f8a83400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e2824f390 0x561e28693d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.282+0000 7f8a83400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e2824f390 0x561e28693d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:44928/0 (socket says 172.21.15.134:44928) 2024-07-26T11:19:34.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.282+0000 7f8a89000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e286277f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.282+0000 7f8a91e37740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8a841afc00 con 0x561e28270070 2024-07-26T11:19:34.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.283+0000 7f8a89000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e286277f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:46962/0 (socket says 172.21.15.134:46962) 2024-07-26T11:19:34.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.285+0000 7f8a91e37740 1 -- 172.21.15.134:0/1178769652 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8a841af530 con 0x561e282471b0 2024-07-26T11:19:34.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.285+0000 7f8a91e37740 1 -- 172.21.15.134:0/1178769652 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8a84074920 con 0x561e2824f390 2024-07-26T11:19:34.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.285+0000 7f8a91e37740 10 client.0 _ll_register_callbacks cb 0x7f8a740118e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:34.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.283+0000 7f8a83400640 1 -- 172.21.15.134:0/1178769652 learned_addr learned my addr 172.21.15.134:0/1178769652 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:34.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.283+0000 7f8a83e00640 1 --2- 172.21.15.134:0/1178769652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e28693810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.286+0000 7f8a89000640 1 -- 172.21.15.134:0/1178769652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e2824f390 msgr2=0x561e28693d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:34.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.286+0000 7f8a89000640 1 --2- 172.21.15.134:0/1178769652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e2824f390 0x561e28693d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.286+0000 7f8a89000640 1 -- 172.21.15.134:0/1178769652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 msgr2=0x561e28693810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:34.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.286+0000 7f8a89000640 1 --2- 172.21.15.134:0/1178769652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e28693810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.286+0000 7f8a89000640 1 -- 172.21.15.134:0/1178769652 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e28627d60 con 0x561e28270070 2024-07-26T11:19:34.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.287+0000 7f8a89000640 1 --2- 172.21.15.134:0/1178769652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e286277f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c052040 tx=0x7f8a7c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26f73332c2a84f5d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:34.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.288+0000 7f8a6e000640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:19:34.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.291+0000 7f8a6e000640 1 -- 172.21.15.134:0/1178769652 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c05d060 con 0x561e28270070 2024-07-26T11:19:34.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.291+0000 7f8a6e000640 1 -- 172.21.15.134:0/1178769652 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f8a7c031b20 con 0x561e28270070 2024-07-26T11:19:34.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.291+0000 7f8a91e37740 1 -- 172.21.15.134:0/1178769652 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a7c036050 con 0x561e28270070 2024-07-26T11:19:34.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.292+0000 7f8a6e000640 1 -- 172.21.15.134:0/1178769652 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8a7c054040 con 0x561e28270070 2024-07-26T11:19:34.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.292+0000 7f8a91e37740 10 client.15786 fetch_fsmap learned FSMap version 32 2024-07-26T11:19:34.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.292+0000 7f8a91e37740 1 -- 172.21.15.134:0/1178769652 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561e28628fe0 con 0x561e28270070 2024-07-26T11:19:34.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.292+0000 7f8a6e000640 1 -- 172.21.15.134:0/1178769652 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a7c036050 con 0x561e28270070 2024-07-26T11:19:34.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.293+0000 7f8a6e000640 1 -- 172.21.15.134:0/1178769652 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f8a7c04e070 con 0x561e28270070 2024-07-26T11:19:34.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.293+0000 7f8a91e37740 10 client.15786 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:19:34.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.293+0000 7f8a91e37740 10 client.15786 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:19:34.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.293+0000 7f8a91e37740 20 client.15786 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:34.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.293+0000 7f8a91e37740 1 --2- 172.21.15.134:0/1178769652 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x561e28203da0 0x561e286ae080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:34.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.293+0000 7f8a91e37740 4 client.15786 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:19:34.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.293+0000 7f8a91e37740 1 -- 172.21.15.134:0/1178769652 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561e285895c0 con 0x561e28203da0 2024-07-26T11:19:34.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.293+0000 7f8a83e00640 1 --2- 172.21.15.134:0/1178769652 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x561e28203da0 0x561e286ae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:34.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.296+0000 7f8a83e00640 1 --2- 172.21.15.134:0/1178769652 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x561e28203da0 0x561e286ae080 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=762cf80babef98ef server_cookie=5970142634b6a76 in_seq=0 out_seq=0 2024-07-26T11:19:34.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.296+0000 7f8a6e000640 10 client.15786 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:34.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.297+0000 7f8a6e000640 1 -- 172.21.15.134:0/1178769652 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561e285895c0 con 0x561e28203da0 2024-07-26T11:19:34.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.297+0000 7f8a6e000640 10 client.15786 handle_command_reply: tid=0 2024-07-26T11:19:34.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.297+0000 7f8a91e37740 1 client.15786 shutdown 2024-07-26T11:19:34.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.297+0000 7f8a91e37740 1 -- 172.21.15.134:0/1178769652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 msgr2=0x561e286277f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:34.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.297+0000 7f8a91e37740 1 --2- 172.21.15.134:0/1178769652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e286277f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c052040 tx=0x7f8a7c003ba0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.297+0000 7f8a91e37740 1 -- 172.21.15.134:0/1178769652 shutdown_connections 2024-07-26T11:19:34.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.297+0000 7f8a91e37740 1 --2- 172.21.15.134:0/1178769652 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x561e28203da0 0x561e286ae080 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.297+0000 7f8a91e37740 1 --2- 172.21.15.134:0/1178769652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561e2824f390 0x561e28693d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.297+0000 7f8a91e37740 1 --2- 172.21.15.134:0/1178769652 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561e28270070 0x561e286277f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.297+0000 7f8a91e37740 1 --2- 172.21.15.134:0/1178769652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e282471b0 0x561e28693810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.297+0000 7f8a91e37740 1 -- 172.21.15.134:0/1178769652 >> 172.21.15.134:0/1178769652 conn(0x561e28264460 msgr2=0x561e286165f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:34.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.297+0000 7f8a6e000640 0 client.15786 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:34.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.298+0000 7f8a91e37740 1 -- 172.21.15.134:0/1178769652 shutdown_connections 2024-07-26T11:19:34.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.298+0000 7f8a91e37740 1 -- 172.21.15.134:0/1178769652 wait complete. 2024-07-26T11:19:34.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.298+0000 7f8a91e37740 20 client.15786 trim_cache size 0 max 16384 2024-07-26T11:19:34.304 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:19:34.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.299+0000 7f8a8b000640 1 -- 172.21.15.134:0/4237101718 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8a680b3e70 msgr2=0x7f8a680b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:34.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.299+0000 7f8a8b000640 1 --2- 172.21.15.134:0/4237101718 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8a680b3e70 0x7f8a680b6380 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x561e28613ab0 tx=0x7f8a74005f90 comp rx=0 tx=0).stop 2024-07-26T11:19:34.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.299+0000 7f8a8b000640 1 -- 172.21.15.134:0/4237101718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a84076db0 msgr2=0x7f8a841a0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:34.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.299+0000 7f8a8b000640 1 --2- 172.21.15.134:0/4237101718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a84076db0 0x7f8a841a0a40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000b3c0 tx=0x7f8a7000b880 comp rx=0 tx=0).stop 2024-07-26T11:19:34.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.299+0000 7f8a8b000640 1 -- 172.21.15.134:0/4237101718 shutdown_connections 2024-07-26T11:19:34.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.299+0000 7f8a8b000640 1 --2- 172.21.15.134:0/4237101718 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8a680b3e70 0x7f8a680b6380 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.300+0000 7f8a8b000640 1 --2- 172.21.15.134:0/4237101718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8a84076db0 0x7f8a841a0a40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.300+0000 7f8a8b000640 1 --2- 172.21.15.134:0/4237101718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8a84113e70 0x7f8a841a5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.300+0000 7f8a8b000640 1 --2- 172.21.15.134:0/4237101718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a84075800 0x7f8a841a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:34.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.300+0000 7f8a8b000640 1 -- 172.21.15.134:0/4237101718 >> 172.21.15.134:0/4237101718 conn(0x7f8a840ff010 msgr2=0x7f8a8410fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:34.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.300+0000 7f8a8b000640 1 -- 172.21.15.134:0/4237101718 shutdown_connections 2024-07-26T11:19:34.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:34.300+0000 7f8a8b000640 1 -- 172.21.15.134:0/4237101718 wait complete. 2024-07-26T11:19:34.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:34 smithi134 ceph-mon[31966]: pgmap v588: 129 pgs: 129 active+clean; 21 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 214 MiB/s wr, 1.48k op/s 2024-07-26T11:19:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:34 smithi189 ceph-mon[36790]: pgmap v588: 129 pgs: 129 active+clean; 21 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 214 MiB/s wr, 1.48k op/s 2024-07-26T11:19:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:34 smithi193 ceph-mon[37473]: pgmap v588: 129 pgs: 129 active+clean; 21 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 214 MiB/s wr, 1.48k op/s 2024-07-26T11:19:36.628 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:36 smithi134 ceph-mon[31966]: pgmap v589: 129 pgs: 129 active+clean; 22 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 260 MiB/s wr, 1.76k op/s 2024-07-26T11:19:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:36 smithi189 ceph-mon[36790]: pgmap v589: 129 pgs: 129 active+clean; 22 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 260 MiB/s wr, 1.76k op/s 2024-07-26T11:19:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:36 smithi193 ceph-mon[37473]: pgmap v589: 129 pgs: 129 active+clean; 22 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 260 MiB/s wr, 1.76k op/s 2024-07-26T11:19:37.677 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:19:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:19:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:19:38.120 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T11:19:38.121 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T11:19:38.679 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.680 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.680 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.680 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.680 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.680 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.680 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.680 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.680 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:38 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T11:19:38.680 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:38 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T11:19:38.680 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:38 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T11:19:38.680 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:38 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T11:19:38.681 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:38 smithi134 ceph-mon[31966]: pgmap v590: 129 pgs: 129 active+clean; 22 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 224 MiB/s wr, 1.47k op/s 2024-07-26T11:19:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:38 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T11:19:38.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:38 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T11:19:38.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:38 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T11:19:38.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:38 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T11:19:38.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:38 smithi189 ceph-mon[36790]: pgmap v590: 129 pgs: 129 active+clean; 22 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 224 MiB/s wr, 1.47k op/s 2024-07-26T11:19:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:19:38.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:38 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T11:19:38.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:38 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T11:19:38.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:38 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T11:19:38.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:38 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T11:19:38.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:38 smithi193 ceph-mon[37473]: pgmap v590: 129 pgs: 129 active+clean; 22 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 224 MiB/s wr, 1.47k op/s 2024-07-26T11:19:39.149 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:19:39.676 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:19:39.676 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:19:39.676 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:19:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:19:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:19:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:19:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:19:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:19:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:19:40.652 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:40 smithi134 ceph-mon[31966]: pgmap v591: 129 pgs: 129 active+clean; 22 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 264 MiB/s wr, 1.72k op/s 2024-07-26T11:19:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:40 smithi189 ceph-mon[36790]: pgmap v591: 129 pgs: 129 active+clean; 22 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 264 MiB/s wr, 1.72k op/s 2024-07-26T11:19:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:40 smithi193 ceph-mon[37473]: pgmap v591: 129 pgs: 129 active+clean; 22 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 264 MiB/s wr, 1.72k op/s 2024-07-26T11:19:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:42 smithi134 ceph-mon[31966]: pgmap v592: 129 pgs: 129 active+clean; 23 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 245 MiB/s wr, 1.57k op/s 2024-07-26T11:19:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:42 smithi189 ceph-mon[36790]: pgmap v592: 129 pgs: 129 active+clean; 23 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 245 MiB/s wr, 1.57k op/s 2024-07-26T11:19:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:42 smithi193 ceph-mon[37473]: pgmap v592: 129 pgs: 129 active+clean; 23 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 245 MiB/s wr, 1.57k op/s 2024-07-26T11:19:44.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:44 smithi134 ceph-mon[31966]: pgmap v593: 129 pgs: 129 active+clean; 23 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 243 MiB/s wr, 1.50k op/s 2024-07-26T11:19:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:44 smithi189 ceph-mon[36790]: pgmap v593: 129 pgs: 129 active+clean; 23 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 243 MiB/s wr, 1.50k op/s 2024-07-26T11:19:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:44 smithi193 ceph-mon[37473]: pgmap v593: 129 pgs: 129 active+clean; 23 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 243 MiB/s wr, 1.50k op/s 2024-07-26T11:19:45.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.707+0000 7f11eb200640 1 -- 172.21.15.134:0/1764448154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11ec102930 msgr2=0x7f11ec106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:45.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.707+0000 7f11eb200640 1 --2- 172.21.15.134:0/1764448154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11ec102930 0x7f11ec106e60 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009900 tx=0x7f11d402a440 comp rx=0 tx=0).stop 2024-07-26T11:19:45.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.708+0000 7f11eb200640 1 -- 172.21.15.134:0/1764448154 shutdown_connections 2024-07-26T11:19:45.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.708+0000 7f11eb200640 1 --2- 172.21.15.134:0/1764448154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11ec102930 0x7f11ec106e60 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.708+0000 7f11eb200640 1 --2- 172.21.15.134:0/1764448154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11ec107420 0x7f11ec107900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.708+0000 7f11eb200640 1 --2- 172.21.15.134:0/1764448154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f11ec101fe0 0x7f11ec1023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.708+0000 7f11eb200640 1 -- 172.21.15.134:0/1764448154 >> 172.21.15.134:0/1764448154 conn(0x7f11ec0fdcf0 msgr2=0x7f11ec100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:45.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.708+0000 7f11eb200640 1 -- 172.21.15.134:0/1764448154 shutdown_connections 2024-07-26T11:19:45.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.709+0000 7f11eb200640 1 -- 172.21.15.134:0/1764448154 wait complete. 2024-07-26T11:19:45.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.711+0000 7f11eb200640 1 Processor -- start 2024-07-26T11:19:45.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.715+0000 7f11eb200640 1 -- start start 2024-07-26T11:19:45.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.715+0000 7f11eb200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f11ec101fe0 0x7f11ec1a02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.715+0000 7f11eb200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11ec102930 0x7f11ec1a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.715+0000 7f11eb200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11ec107420 0x7f11ec1a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.716+0000 7f11e1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11ec102930 0x7f11ec1a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.716+0000 7f11e1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11ec102930 0x7f11ec1a0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:45572/0 (socket says 172.21.15.134:45572) 2024-07-26T11:19:45.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.717+0000 7f11ea800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11ec107420 0x7f11ec1a5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.717+0000 7f11ea800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11ec107420 0x7f11ec1a5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:32862/0 (socket says 172.21.15.134:32862) 2024-07-26T11:19:45.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.717+0000 7f11e9e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f11ec101fe0 0x7f11ec1a02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.717+0000 7f11eb200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec119910 con 0x7f11ec101fe0 2024-07-26T11:19:45.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.717+0000 7f11eb200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec119790 con 0x7f11ec102930 2024-07-26T11:19:45.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.717+0000 7f11eb200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec119a90 con 0x7f11ec107420 2024-07-26T11:19:45.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.717+0000 7f11e9e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f11ec101fe0 0x7f11ec1a02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50574/0 (socket says 172.21.15.134:50574) 2024-07-26T11:19:45.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.717+0000 7f11e9e00640 1 -- 172.21.15.134:0/2885248185 learned_addr learned my addr 172.21.15.134:0/2885248185 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:45.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.718+0000 7f11e1400640 1 -- 172.21.15.134:0/2885248185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11ec107420 msgr2=0x7f11ec1a5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:45.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.718+0000 7f11e1400640 1 --2- 172.21.15.134:0/2885248185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11ec107420 0x7f11ec1a5b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.718+0000 7f11e1400640 1 -- 172.21.15.134:0/2885248185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f11ec101fe0 msgr2=0x7f11ec1a02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:45.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.718+0000 7f11e1400640 1 --2- 172.21.15.134:0/2885248185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f11ec101fe0 0x7f11ec1a02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.718+0000 7f11e1400640 1 -- 172.21.15.134:0/2885248185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ec1a6230 con 0x7f11ec102930 2024-07-26T11:19:45.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.719+0000 7f11ea800640 1 --2- 172.21.15.134:0/2885248185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11ec107420 0x7f11ec1a5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-07-26T11:19:45.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.719+0000 7f11e1400640 1 --2- 172.21.15.134:0/2885248185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11ec102930 0x7f11ec1a0860 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009230 tx=0x7f11d4003f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:45.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.719+0000 7f11e3400640 1 -- 172.21.15.134:0/2885248185 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d4040030 con 0x7f11ec102930 2024-07-26T11:19:45.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.720+0000 7f11e3400640 1 -- 172.21.15.134:0/2885248185 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11d403c8f0 con 0x7f11ec102930 2024-07-26T11:19:45.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.720+0000 7f11eb200640 1 -- 172.21.15.134:0/2885248185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11ec1a64c0 con 0x7f11ec102930 2024-07-26T11:19:45.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.721+0000 7f11e3400640 1 -- 172.21.15.134:0/2885248185 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d4031830 con 0x7f11ec102930 2024-07-26T11:19:45.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.721+0000 7f11eb200640 1 -- 172.21.15.134:0/2885248185 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11ec1acc80 con 0x7f11ec102930 2024-07-26T11:19:45.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.735+0000 7f11f765e740 1 Processor -- start 2024-07-26T11:19:45.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.735+0000 7f11f765e740 1 -- start start 2024-07-26T11:19:45.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.736+0000 7f11f765e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94954350 0x555b94cf8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.736+0000 7f11f765e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b9492b490 0x555b94cf94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.736+0000 7f11f765e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b94933670 0x555b94cf99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.736+0000 7f11e3400640 1 -- 172.21.15.134:0/2885248185 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f11d4043030 con 0x7f11ec102930 2024-07-26T11:19:45.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.736+0000 7f11f765e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x555b94c6fbf0 con 0x555b94954350 2024-07-26T11:19:45.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.736+0000 7f11f765e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555b94c5f0f0 con 0x555b94933670 2024-07-26T11:19:45.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.736+0000 7f11f765e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555b94c63130 con 0x555b9492b490 2024-07-26T11:19:45.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.741+0000 7f11ea800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94954350 0x555b94cf8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.742+0000 7f11e9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b9492b490 0x555b94cf94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.742+0000 7f11e9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b9492b490 0x555b94cf94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:32874/0 (socket says 172.21.15.134:32874) 2024-07-26T11:19:45.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.742+0000 7f11e9e00640 1 -- 172.21.15.134:0/94010892 learned_addr learned my addr 172.21.15.134:0/94010892 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:45.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.742+0000 7f11e1400640 1 --2- 172.21.15.134:0/94010892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b94933670 0x555b94cf99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.742+0000 7f11e9e00640 1 -- 172.21.15.134:0/94010892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b94933670 msgr2=0x555b94cf99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:45.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.742+0000 7f11e9e00640 1 --2- 172.21.15.134:0/94010892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b94933670 0x555b94cf99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.742+0000 7f11e9e00640 1 -- 172.21.15.134:0/94010892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94954350 msgr2=0x555b94cf8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:45.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.742+0000 7f11e9e00640 1 --2- 172.21.15.134:0/94010892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94954350 0x555b94cf8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.742+0000 7f11e9e00640 1 -- 172.21.15.134:0/94010892 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b949bf930 con 0x555b9492b490 2024-07-26T11:19:45.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.746+0000 7f11e3400640 1 --2- 172.21.15.134:0/2885248185 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f11d00b3e40 0x7f11d00b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.746+0000 7f11e9e00640 1 --2- 172.21.15.134:0/94010892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b9492b490 0x555b94cf94a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f11dc0114e0 tx=0x7f11dc0119a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56e85fe3883e8201 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:45.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.747+0000 7f11e3400640 1 -- 172.21.15.134:0/2885248185 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f11d4104710 con 0x7f11ec102930 2024-07-26T11:19:45.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.748+0000 7f11db000640 1 -- 172.21.15.134:0/94010892 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc01f070 con 0x555b9492b490 2024-07-26T11:19:45.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.748+0000 7f11db000640 1 -- 172.21.15.134:0/94010892 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11dc012c20 con 0x555b9492b490 2024-07-26T11:19:45.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.748+0000 7f11db000640 1 -- 172.21.15.134:0/94010892 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc01ad00 con 0x555b9492b490 2024-07-26T11:19:45.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.750+0000 7f11ea800640 1 --2- 172.21.15.134:0/2885248185 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f11d00b3e40 0x7f11d00b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.752+0000 7f11ea800640 1 --2- 172.21.15.134:0/2885248185 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f11d00b3e40 0x7f11d00b6350 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f11e4009ae0 tx=0x7f11e4005e70 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:45.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.753+0000 7f11f765e740 1 -- 172.21.15.134:0/94010892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b9492b490 msgr2=0x555b94cf94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:45.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.753+0000 7f11f765e740 1 --2- 172.21.15.134:0/94010892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b9492b490 0x555b94cf94a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f11dc0114e0 tx=0x7f11dc0119a0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.753+0000 7f11f765e740 1 -- 172.21.15.134:0/94010892 shutdown_connections 2024-07-26T11:19:45.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.754+0000 7f11f765e740 1 --2- 172.21.15.134:0/94010892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b9492b490 0x555b94cf94a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.754+0000 7f11f765e740 1 --2- 172.21.15.134:0/94010892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b94933670 0x555b94cf99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.754+0000 7f11f765e740 1 --2- 172.21.15.134:0/94010892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94954350 0x555b94cf8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.754+0000 7f11f765e740 1 -- 172.21.15.134:0/94010892 >> 172.21.15.134:0/94010892 conn(0x555b94948740 msgr2=0x555b94cf6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:45.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.755+0000 7f11f765e740 1 -- 172.21.15.134:0/94010892 shutdown_connections 2024-07-26T11:19:45.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.758+0000 7f11f765e740 1 -- 172.21.15.134:0/94010892 wait complete. 2024-07-26T11:19:45.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.765+0000 7f11f765e740 1 -- start start 2024-07-26T11:19:45.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.765+0000 7f11f765e740 1 Processor -- start 2024-07-26T11:19:45.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.766+0000 7f11f765e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b9492b490 0x555b94d04670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.766+0000 7f11f765e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94933670 0x555b94d04bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.766+0000 7f11f765e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b94954350 0x555b94d21a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.766+0000 7f11f765e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x555b94ac2890 con 0x555b94933670 2024-07-26T11:19:45.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.766+0000 7f11f765e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555b94bfb330 con 0x555b9492b490 2024-07-26T11:19:45.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.766+0000 7f11f765e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555b94d18750 con 0x555b94954350 2024-07-26T11:19:45.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.766+0000 7f11f765e740 10 client.0 _ll_register_callbacks cb 0x555b94c27010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:45.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.767+0000 7f11e1400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94933670 0x555b94d04bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.768+0000 7f11ea800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b94954350 0x555b94d21a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.769+0000 7f11ea800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b94954350 0x555b94d21a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:32890/0 (socket says 172.21.15.134:32890) 2024-07-26T11:19:45.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.769+0000 7f11ea800640 1 -- 172.21.15.134:0/2255427391 learned_addr learned my addr 172.21.15.134:0/2255427391 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:45.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.769+0000 7f11e9e00640 1 --2- 172.21.15.134:0/2255427391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b9492b490 0x555b94d04670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.769+0000 7f11ea800640 1 -- 172.21.15.134:0/2255427391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b9492b490 msgr2=0x555b94d04670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:45.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.769+0000 7f11ea800640 1 --2- 172.21.15.134:0/2255427391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b9492b490 0x555b94d04670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.769+0000 7f11ea800640 1 -- 172.21.15.134:0/2255427391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94933670 msgr2=0x555b94d04bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:45.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.769+0000 7f11ea800640 1 --2- 172.21.15.134:0/2255427391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94933670 0x555b94d04bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.769+0000 7f11ea800640 1 -- 172.21.15.134:0/2255427391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b94d05450 con 0x555b94954350 2024-07-26T11:19:45.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.770+0000 7f11ea800640 1 --2- 172.21.15.134:0/2255427391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b94954350 0x555b94d21a40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f11e4011440 tx=0x7f11e4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac24ae8207e9b558 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:45.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.771+0000 7f11cf400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:19:45.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.772+0000 7f11cf400640 1 -- 172.21.15.134:0/2255427391 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4022070 con 0x555b94954350 2024-07-26T11:19:45.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.773+0000 7f11cf400640 1 -- 172.21.15.134:0/2255427391 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f11e401e860 con 0x555b94954350 2024-07-26T11:19:45.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.775+0000 7f11f765e740 1 -- 172.21.15.134:0/2255427391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11dc012c20 con 0x555b94954350 2024-07-26T11:19:45.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.775+0000 7f11cf400640 1 -- 172.21.15.134:0/2255427391 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f11e402c050 con 0x555b94954350 2024-07-26T11:19:45.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.775+0000 7f11f765e740 10 client.26119 fetch_fsmap learned FSMap version 32 2024-07-26T11:19:45.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.776+0000 7f11f765e740 1 -- 172.21.15.134:0/2255427391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555b94d246e0 con 0x555b94954350 2024-07-26T11:19:45.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.777+0000 7f11cf400640 1 -- 172.21.15.134:0/2255427391 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11dc012c20 con 0x555b94954350 2024-07-26T11:19:45.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.780+0000 7f11cf400640 1 -- 172.21.15.134:0/2255427391 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f11e401ea60 con 0x555b94954350 2024-07-26T11:19:45.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.780+0000 7f11f765e740 10 client.26119 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:19:45.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.780+0000 7f11f765e740 10 client.26119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:19:45.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.780+0000 7f11f765e740 20 client.26119 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:45.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.780+0000 7f11f765e740 1 --2- 172.21.15.134:0/2255427391 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x555b94912cd0 0x555b94d28070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.780+0000 7f11f765e740 4 client.26119 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:19:45.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.780+0000 7f11f765e740 1 -- 172.21.15.134:0/2255427391 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b94c71b40 con 0x555b94912cd0 2024-07-26T11:19:45.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.781+0000 7f11e9e00640 1 --2- 172.21.15.134:0/2255427391 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x555b94912cd0 0x555b94d28070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.789+0000 7f11cf400640 10 client.26119 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:45.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.789+0000 7f11e9e00640 1 --2- 172.21.15.134:0/2255427391 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x555b94912cd0 0x555b94d28070 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a20f9098cf887c2 server_cookie=cc9b85ba3a663821 in_seq=0 out_seq=0 2024-07-26T11:19:45.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.790+0000 7f11cf400640 1 -- 172.21.15.134:0/2255427391 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x555b94c71b40 con 0x555b94912cd0 2024-07-26T11:19:45.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.790+0000 7f11cf400640 10 client.26119 handle_command_reply: tid=0 2024-07-26T11:19:45.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.791+0000 7f11f765e740 1 client.26119 shutdown 2024-07-26T11:19:45.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.795+0000 7f11f765e740 1 -- 172.21.15.134:0/2255427391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b94954350 msgr2=0x555b94d21a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:45.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.795+0000 7f11f765e740 1 --2- 172.21.15.134:0/2255427391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b94954350 0x555b94d21a40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f11e4011440 tx=0x7f11e4004730 comp rx=0 tx=0).stop 2024-07-26T11:19:45.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.796+0000 7f11f765e740 1 -- 172.21.15.134:0/2255427391 shutdown_connections 2024-07-26T11:19:45.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.796+0000 7f11f765e740 1 --2- 172.21.15.134:0/2255427391 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x555b94912cd0 0x555b94d28070 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.796+0000 7f11f765e740 1 --2- 172.21.15.134:0/2255427391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b94954350 0x555b94d21a40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.796+0000 7f11f765e740 1 --2- 172.21.15.134:0/2255427391 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94933670 0x555b94d04bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.796+0000 7f11f765e740 1 --2- 172.21.15.134:0/2255427391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b9492b490 0x555b94d04670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.796+0000 7f11f765e740 1 -- 172.21.15.134:0/2255427391 >> 172.21.15.134:0/2255427391 conn(0x555b94948740 msgr2=0x555b94cfc140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:45.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.796+0000 7f11cf400640 0 client.26119 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:45.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.797+0000 7f11f765e740 1 -- 172.21.15.134:0/2255427391 shutdown_connections 2024-07-26T11:19:45.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.797+0000 7f11f765e740 1 -- 172.21.15.134:0/2255427391 wait complete. 2024-07-26T11:19:45.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.798+0000 7f11f765e740 20 client.26119 trim_cache size 0 max 16384 2024-07-26T11:19:45.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.847+0000 7f11f765e740 1 Processor -- start 2024-07-26T11:19:45.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.850+0000 7f11f765e740 1 -- start start 2024-07-26T11:19:45.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.856+0000 7f11f765e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b9492b490 0x555b94cfaa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.857+0000 7f11f765e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b94933670 0x555b94d04b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.859+0000 7f11e9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b9492b490 0x555b94cfaa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.859+0000 7f11e9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b9492b490 0x555b94cfaa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:45600/0 (socket says 172.21.15.134:45600) 2024-07-26T11:19:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.862+0000 7f11f765e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94954350 0x555b94cf64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.863+0000 7f11e1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b94933670 0x555b94d04b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.863+0000 7f11f765e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1b08a0 con 0x555b94954350 2024-07-26T11:19:45.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.863+0000 7f11f765e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1ae690 con 0x555b9492b490 2024-07-26T11:19:45.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.863+0000 7f11f765e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1b01d0 con 0x555b94933670 2024-07-26T11:19:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.863+0000 7f11e1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b94933670 0x555b94d04b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:32900/0 (socket says 172.21.15.134:32900) 2024-07-26T11:19:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.864+0000 7f11e1400640 1 -- 172.21.15.134:0/4049260121 learned_addr learned my addr 172.21.15.134:0/4049260121 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.865+0000 7f11e9e00640 1 -- 172.21.15.134:0/4049260121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b94933670 msgr2=0x555b94d04b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.865+0000 7f11e9e00640 1 --2- 172.21.15.134:0/4049260121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b94933670 0x555b94d04b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.865+0000 7f11e9e00640 1 -- 172.21.15.134:0/4049260121 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94954350 msgr2=0x555b94cf64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.865+0000 7f11e9e00640 1 --2- 172.21.15.134:0/4049260121 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94954350 0x555b94cf64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.865+0000 7f11e9e00640 1 -- 172.21.15.134:0/4049260121 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b94cfbdc0 con 0x555b9492b490 2024-07-26T11:19:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.865+0000 7f11e9e00640 1 --2- 172.21.15.134:0/4049260121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b9492b490 0x555b94cfaa60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f11dc000f80 tx=0x7f11dc01a980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8369b555945b7d8e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:45.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.865+0000 7f11da600640 1 -- 172.21.15.134:0/4049260121 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc01f040 con 0x555b9492b490 2024-07-26T11:19:45.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.866+0000 7f11da600640 1 -- 172.21.15.134:0/4049260121 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11dc02a040 con 0x555b9492b490 2024-07-26T11:19:45.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.867+0000 7f11f765e740 1 -- 172.21.15.134:0/4049260121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b9492b490 msgr2=0x555b94cfaa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:45.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.867+0000 7f11f765e740 1 --2- 172.21.15.134:0/4049260121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b9492b490 0x555b94cfaa60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f11dc000f80 tx=0x7f11dc01a980 comp rx=0 tx=0).stop 2024-07-26T11:19:45.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.867+0000 7f11f765e740 1 -- 172.21.15.134:0/4049260121 shutdown_connections 2024-07-26T11:19:45.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.867+0000 7f11f765e740 1 --2- 172.21.15.134:0/4049260121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b94933670 0x555b94d04b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.867+0000 7f11f765e740 1 --2- 172.21.15.134:0/4049260121 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94954350 0x555b94cf64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.867+0000 7f11f765e740 1 --2- 172.21.15.134:0/4049260121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b9492b490 0x555b94cfaa60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.867+0000 7f11f765e740 1 -- 172.21.15.134:0/4049260121 >> 172.21.15.134:0/4049260121 conn(0x555b94948740 msgr2=0x555b94cff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:45.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.868+0000 7f11f765e740 1 -- 172.21.15.134:0/4049260121 shutdown_connections 2024-07-26T11:19:45.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.868+0000 7f11f765e740 1 -- 172.21.15.134:0/4049260121 wait complete. 2024-07-26T11:19:45.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.871+0000 7f11f765e740 1 -- start start 2024-07-26T11:19:45.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.871+0000 7f11f765e740 1 Processor -- start 2024-07-26T11:19:45.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.872+0000 7f11f765e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b9492b490 0x555b94d2dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.873+0000 7f11e9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b9492b490 0x555b94d2dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.873+0000 7f11e9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b9492b490 0x555b94d2dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:32908/0 (socket says 172.21.15.134:32908) 2024-07-26T11:19:45.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.874+0000 7f11e9e00640 1 -- 172.21.15.134:0/2770294884 learned_addr learned my addr 172.21.15.134:0/2770294884 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:45.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.874+0000 7f11f765e740 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94933670 0x555b94d2af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.874+0000 7f11f765e740 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b94954350 0x555b94d2b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.878 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.876+0000 7f11f765e740 1 -- 172.21.15.134:0/2770294884 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1af430 con 0x555b94933670 2024-07-26T11:19:45.878 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.876+0000 7f11f765e740 1 -- 172.21.15.134:0/2770294884 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1aed60 con 0x555b94954350 2024-07-26T11:19:45.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.876+0000 7f11ea800640 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b94954350 0x555b94d2b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.876+0000 7f11f765e740 1 -- 172.21.15.134:0/2770294884 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec1afb00 con 0x555b9492b490 2024-07-26T11:19:45.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.877+0000 7f11e9e00640 1 -- 172.21.15.134:0/2770294884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b94954350 msgr2=0x555b94d2b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:45.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.877+0000 7f11e9e00640 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b94954350 0x555b94d2b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.877+0000 7f11e9e00640 1 -- 172.21.15.134:0/2770294884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94933670 msgr2=0x555b94d2af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:45.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.877+0000 7f11e1400640 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94933670 0x555b94d2af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.879+0000 7f11e9e00640 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94933670 0x555b94d2af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.879+0000 7f11f765e740 10 client.0 _ll_register_callbacks cb 0x7f11e4015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:45.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.879+0000 7f11e9e00640 1 -- 172.21.15.134:0/2770294884 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b94d2ba10 con 0x555b9492b490 2024-07-26T11:19:45.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.880+0000 7f11ea800640 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b94954350 0x555b94d2b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:19:45.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.880+0000 7f11e1400640 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94933670 0x555b94d2af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:19:45.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.880+0000 7f11f765e740 1 -- 172.21.15.134:0/2770294884 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11dc02a040 con 0x555b9492b490 2024-07-26T11:19:45.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.880+0000 7f11e9e00640 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b9492b490 0x555b94d2dac0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f11dc009dc0 tx=0x7f11dc009df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7dfa0bf35e991b8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:45.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.883+0000 7f11cf400640 10 client.26125 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:19:45.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.883+0000 7f11cf400640 1 -- 172.21.15.134:0/2770294884 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc033070 con 0x555b9492b490 2024-07-26T11:19:45.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.883+0000 7f11cf400640 1 -- 172.21.15.134:0/2770294884 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f11dc012790 con 0x555b9492b490 2024-07-26T11:19:45.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.883+0000 7f11cf400640 1 -- 172.21.15.134:0/2770294884 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f11dc01f040 con 0x555b9492b490 2024-07-26T11:19:45.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.883+0000 7f11f765e740 10 client.26125 fetch_fsmap learned FSMap version 32 2024-07-26T11:19:45.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.883+0000 7f11f765e740 1 -- 172.21.15.134:0/2770294884 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=32}) -- 0x555b94d90290 con 0x555b9492b490 2024-07-26T11:19:45.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.886+0000 7f11cf400640 1 -- 172.21.15.134:0/2770294884 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f11dc02d070 con 0x555b9492b490 2024-07-26T11:19:45.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.886+0000 7f11f765e740 10 client.26125 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:19:45.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.886+0000 7f11f765e740 10 client.26125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:19:45.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.886+0000 7f11f765e740 20 client.26125 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.886+0000 7f11f765e740 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x555b948e8080 0x555b94d92680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.886+0000 7f11f765e740 4 client.26125 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T11:19:45.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.887+0000 7f11f765e740 1 -- 172.21.15.134:0/2770294884 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555b94c6d5e0 con 0x555b948e8080 2024-07-26T11:19:45.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.890+0000 7f11e1400640 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x555b948e8080 0x555b94d92680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:45.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.892+0000 7f11e1400640 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x555b948e8080 0x555b94d92680 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f78a05532ea2917 server_cookie=56268be920be0b51 in_seq=0 out_seq=0 2024-07-26T11:19:45.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.892+0000 7f11cf400640 10 client.26125 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:45.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.898+0000 7f11cf400640 1 -- 172.21.15.134:0/2770294884 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555b94c6d5e0 con 0x555b948e8080 2024-07-26T11:19:45.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.898+0000 7f11cf400640 10 client.26125 handle_command_reply: tid=0 2024-07-26T11:19:45.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.898+0000 7f11f765e740 1 client.26125 shutdown 2024-07-26T11:19:45.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.899+0000 7f11f765e740 1 -- 172.21.15.134:0/2770294884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b9492b490 msgr2=0x555b94d2dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:45.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.899+0000 7f11f765e740 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b9492b490 0x555b94d2dac0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f11dc009dc0 tx=0x7f11dc009df0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.900+0000 7f11f765e740 1 -- 172.21.15.134:0/2770294884 shutdown_connections 2024-07-26T11:19:45.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.900+0000 7f11f765e740 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x555b948e8080 0x555b94d92680 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.900+0000 7f11f765e740 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b94954350 0x555b94d2b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.900+0000 7f11f765e740 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x555b94933670 0x555b94d2af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.900+0000 7f11f765e740 1 --2- 172.21.15.134:0/2770294884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b9492b490 0x555b94d2dac0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.900+0000 7f11f765e740 1 -- 172.21.15.134:0/2770294884 >> 172.21.15.134:0/2770294884 conn(0x555b94948740 msgr2=0x555b94cfeb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:45.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.900+0000 7f11cf400640 0 client.26125 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:45.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.900+0000 7f11f765e740 1 -- 172.21.15.134:0/2770294884 shutdown_connections 2024-07-26T11:19:45.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.900+0000 7f11f765e740 1 -- 172.21.15.134:0/2770294884 wait complete. 2024-07-26T11:19:45.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.900+0000 7f11f765e740 20 client.26125 trim_cache size 0 max 16384 2024-07-26T11:19:45.904 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:19:45.904 INFO:teuthology.orchestra.run.smithi134.stdout: "return_code": 0, 2024-07-26T11:19:45.904 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_tag": "7b86a24f-cf81-465e-a160-25cafe480e4b", 2024-07-26T11:19:45.904 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "asynchronous" 2024-07-26T11:19:45.904 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:19:45.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.902+0000 7f11eb200640 1 -- 172.21.15.134:0/2885248185 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f11d00b3e40 msgr2=0x7f11d00b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:45.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.902+0000 7f11eb200640 1 --2- 172.21.15.134:0/2885248185 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f11d00b3e40 0x7f11d00b6350 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f11e4009ae0 tx=0x7f11e4005e70 comp rx=0 tx=0).stop 2024-07-26T11:19:45.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.902+0000 7f11eb200640 1 -- 172.21.15.134:0/2885248185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11ec102930 msgr2=0x7f11ec1a0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:45.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.902+0000 7f11eb200640 1 --2- 172.21.15.134:0/2885248185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11ec102930 0x7f11ec1a0860 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009230 tx=0x7f11d4003f10 comp rx=0 tx=0).stop 2024-07-26T11:19:45.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.903+0000 7f11eb200640 1 -- 172.21.15.134:0/2885248185 shutdown_connections 2024-07-26T11:19:45.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.903+0000 7f11eb200640 1 --2- 172.21.15.134:0/2885248185 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f11d00b3e40 0x7f11d00b6350 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.903+0000 7f11eb200640 1 --2- 172.21.15.134:0/2885248185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11ec107420 0x7f11ec1a5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.903+0000 7f11eb200640 1 --2- 172.21.15.134:0/2885248185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11ec102930 0x7f11ec1a0860 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.903+0000 7f11eb200640 1 --2- 172.21.15.134:0/2885248185 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f11ec101fe0 0x7f11ec1a02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:45.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.904+0000 7f11eb200640 1 -- 172.21.15.134:0/2885248185 >> 172.21.15.134:0/2885248185 conn(0x7f11ec0fdcf0 msgr2=0x7f11ec104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:45.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.904+0000 7f11eb200640 1 -- 172.21.15.134:0/2885248185 shutdown_connections 2024-07-26T11:19:45.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:45.904+0000 7f11eb200640 1 -- 172.21.15.134:0/2885248185 wait complete. 2024-07-26T11:19:46.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:46 smithi134 ceph-mon[31966]: pgmap v594: 129 pgs: 129 active+clean; 24 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 26 KiB/s rd, 292 MiB/s wr, 1.65k op/s 2024-07-26T11:19:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:46 smithi189 ceph-mon[36790]: pgmap v594: 129 pgs: 129 active+clean; 24 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 26 KiB/s rd, 292 MiB/s wr, 1.65k op/s 2024-07-26T11:19:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:46 smithi193 ceph-mon[37473]: pgmap v594: 129 pgs: 129 active+clean; 24 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 26 KiB/s rd, 292 MiB/s wr, 1.65k op/s 2024-07-26T11:19:47.467 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.465+0000 7f47b5c00640 1 -- 172.21.15.134:0/315623039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47b0115fd0 msgr2=0x7f47b01184b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:47.467 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.465+0000 7f47b5c00640 1 --2- 172.21.15.134:0/315623039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47b0115fd0 0x7f47b01184b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009ca0 tx=0x7f47a802a8c0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.471 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.469+0000 7f47b5c00640 1 -- 172.21.15.134:0/315623039 shutdown_connections 2024-07-26T11:19:47.471 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.469+0000 7f47b5c00640 1 --2- 172.21.15.134:0/315623039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47b0115fd0 0x7f47b01184b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.472 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.469+0000 7f47b5c00640 1 --2- 172.21.15.134:0/315623039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47b010a920 0x7f47b0115a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.472 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.469+0000 7f47b5c00640 1 --2- 172.21.15.134:0/315623039 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47b0109fd0 0x7f47b010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.472 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.469+0000 7f47b5c00640 1 -- 172.21.15.134:0/315623039 >> 172.21.15.134:0/315623039 conn(0x7f47b0070800 msgr2=0x7f47b0070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:47.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.471+0000 7f47b5c00640 1 -- 172.21.15.134:0/315623039 shutdown_connections 2024-07-26T11:19:47.477 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.475+0000 7f47b5c00640 1 -- 172.21.15.134:0/315623039 wait complete. 2024-07-26T11:19:47.479 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.477+0000 7f47b5c00640 1 Processor -- start 2024-07-26T11:19:47.479 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.477+0000 7f47b5c00640 1 -- start start 2024-07-26T11:19:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.480+0000 7f47b5c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47b0109fd0 0x7f47b01a24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.480+0000 7f47b5c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47b010a920 0x7f47b01a2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.480+0000 7f47b5c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47b0115fd0 0x7f47b01a7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.481+0000 7f47b5c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f47b011bcf0 con 0x7f47b0109fd0 2024-07-26T11:19:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.481+0000 7f47b5c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f47b011bb70 con 0x7f47b010a920 2024-07-26T11:19:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.481+0000 7f47b5c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f47b011be70 con 0x7f47b0115fd0 2024-07-26T11:19:47.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.481+0000 7f47aea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47b0109fd0 0x7f47b01a24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.481+0000 7f47af400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47b0115fd0 0x7f47b01a7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.481+0000 7f47aea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47b0109fd0 0x7f47b01a24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50618/0 (socket says 172.21.15.134:50618) 2024-07-26T11:19:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.481+0000 7f47aea00640 1 -- 172.21.15.134:0/591250873 learned_addr learned my addr 172.21.15.134:0/591250873 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.482+0000 7f47aea00640 1 -- 172.21.15.134:0/591250873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47b0115fd0 msgr2=0x7f47b01a7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.482+0000 7f47aea00640 1 --2- 172.21.15.134:0/591250873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47b0115fd0 0x7f47b01a7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.482+0000 7f47aea00640 1 -- 172.21.15.134:0/591250873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47b010a920 msgr2=0x7f47b01a2a40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T11:19:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.482+0000 7f47aea00640 1 --2- 172.21.15.134:0/591250873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47b010a920 0x7f47b01a2a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.482+0000 7f47aea00640 1 -- 172.21.15.134:0/591250873 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47b01a83d0 con 0x7f47b0109fd0 2024-07-26T11:19:47.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.487+0000 7f47aea00640 1 --2- 172.21.15.134:0/591250873 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47b0109fd0 0x7f47b01a24a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f479c00b4a0 tx=0x7f479c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:47.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.488+0000 7f47a7400640 1 -- 172.21.15.134:0/591250873 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c014070 con 0x7f47b0109fd0 2024-07-26T11:19:47.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.488+0000 7f47b5c00640 1 -- 172.21.15.134:0/591250873 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47b01a86c0 con 0x7f47b0109fd0 2024-07-26T11:19:47.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.488+0000 7f47b5c00640 1 -- 172.21.15.134:0/591250873 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47b01aee50 con 0x7f47b0109fd0 2024-07-26T11:19:47.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.494+0000 7f47a7400640 1 -- 172.21.15.134:0/591250873 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f479c010ac0 con 0x7f47b0109fd0 2024-07-26T11:19:47.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.494+0000 7f47a7400640 1 -- 172.21.15.134:0/591250873 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c014070 con 0x7f47b0109fd0 2024-07-26T11:19:47.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.507+0000 7f47a7400640 1 -- 172.21.15.134:0/591250873 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f479c017070 con 0x7f47b0109fd0 2024-07-26T11:19:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.511+0000 7f47a7400640 1 --2- 172.21.15.134:0/591250873 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f47940b4040 0x7f47940b6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.511+0000 7f47a7400640 1 -- 172.21.15.134:0/591250873 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f479c0e3420 con 0x7f47b0109fd0 2024-07-26T11:19:47.517 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.513+0000 7f47bcb78740 1 Processor -- start 2024-07-26T11:19:47.517 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.515+0000 7f47bcb78740 1 -- start start 2024-07-26T11:19:47.526 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.520+0000 7f47bcb78740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 0x5627d6353a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.526 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.520+0000 7f47bcb78740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5f872f0 0x5627d6353f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.520+0000 7f47bcb78740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f8f4d0 0x5627d63544c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.520+0000 7f47bcb78740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5627d62c8270 con 0x5627d5f8f4d0 2024-07-26T11:19:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.520+0000 7f47bcb78740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5627d62b7770 con 0x5627d5f872f0 2024-07-26T11:19:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.520+0000 7f47bcb78740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5627d62bb7b0 con 0x5627d5fb01b0 2024-07-26T11:19:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.521+0000 7f47aea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 0x5627d6353a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.521+0000 7f47aea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 0x5627d6353a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:32938/0 (socket says 172.21.15.134:32938) 2024-07-26T11:19:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.521+0000 7f47aea00640 1 -- 172.21.15.134:0/1296734631 learned_addr learned my addr 172.21.15.134:0/1296734631 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.522+0000 7f47aea00640 1 -- 172.21.15.134:0/1296734631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5f872f0 msgr2=0x5627d6353f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:19:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.522+0000 7f47aea00640 1 --2- 172.21.15.134:0/1296734631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5f872f0 0x5627d6353f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.522+0000 7f47aea00640 1 -- 172.21.15.134:0/1296734631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f8f4d0 msgr2=0x5627d63544c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.522+0000 7f47aea00640 1 --2- 172.21.15.134:0/1296734631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f8f4d0 0x5627d63544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.522+0000 7f47aea00640 1 -- 172.21.15.134:0/1296734631 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627d601b790 con 0x5627d5fb01b0 2024-07-26T11:19:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.523+0000 7f47aea00640 1 --2- 172.21.15.134:0/1296734631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 0x5627d6353a40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f479c00b470 tx=0x7f479c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87a112a4211ba612 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.524+0000 7f479be00640 1 -- 172.21.15.134:0/1296734631 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c01e030 con 0x5627d5fb01b0 2024-07-26T11:19:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.524+0000 7f479be00640 1 -- 172.21.15.134:0/1296734631 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f479c0ae120 con 0x5627d5fb01b0 2024-07-26T11:19:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.524+0000 7f479be00640 1 -- 172.21.15.134:0/1296734631 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c0e0a10 con 0x5627d5fb01b0 2024-07-26T11:19:47.528 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.518+0000 7f47ae000640 1 --2- 172.21.15.134:0/591250873 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f47940b4040 0x7f47940b6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.525+0000 7f47bcb78740 1 -- 172.21.15.134:0/1296734631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 msgr2=0x5627d6353a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.525+0000 7f47bcb78740 1 --2- 172.21.15.134:0/1296734631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 0x5627d6353a40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f479c00b470 tx=0x7f479c0046c0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.526+0000 7f47bcb78740 1 -- 172.21.15.134:0/1296734631 shutdown_connections 2024-07-26T11:19:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.526+0000 7f47bcb78740 1 --2- 172.21.15.134:0/1296734631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f8f4d0 0x5627d63544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.526+0000 7f47bcb78740 1 --2- 172.21.15.134:0/1296734631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5f872f0 0x5627d6353f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.526+0000 7f47bcb78740 1 --2- 172.21.15.134:0/1296734631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 0x5627d6353a40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.526+0000 7f47bcb78740 1 -- 172.21.15.134:0/1296734631 >> 172.21.15.134:0/1296734631 conn(0x5627d5fa4300 msgr2=0x5627d6351930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.527+0000 7f47ae000640 1 --2- 172.21.15.134:0/591250873 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f47940b4040 0x7f47940b6550 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x5627d63526c0 tx=0x7f47a000b410 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.528+0000 7f47bcb78740 1 -- 172.21.15.134:0/1296734631 shutdown_connections 2024-07-26T11:19:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.528+0000 7f47bcb78740 1 -- 172.21.15.134:0/1296734631 wait complete. 2024-07-26T11:19:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.530+0000 7f47bcb78740 1 -- start start 2024-07-26T11:19:47.533 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.531+0000 7f47bcb78740 1 Processor -- start 2024-07-26T11:19:47.537 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.535+0000 7f47bcb78740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5f872f0 0x5627d6361150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.537 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.535+0000 7f47bcb78740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f8f4d0 0x5627d6361710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.537 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.535+0000 7f47bcb78740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 0x5627d635b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.539 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.536+0000 7f47bcb78740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5627d61f1ab0 con 0x5627d5f8f4d0 2024-07-26T11:19:47.544 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.536+0000 7f47bcb78740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5627d6254d50 con 0x5627d5f872f0 2024-07-26T11:19:47.546 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.536+0000 7f47bcb78740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5627d6362000 con 0x5627d5fb01b0 2024-07-26T11:19:47.546 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.536+0000 7f47af400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 0x5627d635b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:47.546 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.536+0000 7f47af400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 0x5627d635b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:32948/0 (socket says 172.21.15.134:32948) 2024-07-26T11:19:47.546 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.536+0000 7f47af400640 1 -- 172.21.15.134:0/2366869084 learned_addr learned my addr 172.21.15.134:0/2366869084 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:47.546 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.538+0000 7f47bcb78740 10 client.0 _ll_register_callbacks cb 0x5627d5fc7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:47.546 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.542+0000 7f47aea00640 1 --2- 172.21.15.134:0/2366869084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5f872f0 0x5627d6361150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:47.549 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.544+0000 7f47ae000640 1 --2- 172.21.15.134:0/2366869084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f8f4d0 0x5627d6361710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:47.549 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.544+0000 7f47af400640 1 -- 172.21.15.134:0/2366869084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5f872f0 msgr2=0x5627d6361150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:47.549 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.544+0000 7f47af400640 1 --2- 172.21.15.134:0/2366869084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5f872f0 0x5627d6361150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.549 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.544+0000 7f47af400640 1 -- 172.21.15.134:0/2366869084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f8f4d0 msgr2=0x5627d6361710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:47.549 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.544+0000 7f47af400640 1 --2- 172.21.15.134:0/2366869084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f8f4d0 0x5627d6361710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.549 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.544+0000 7f47af400640 1 -- 172.21.15.134:0/2366869084 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627d6094620 con 0x5627d5fb01b0 2024-07-26T11:19:47.549 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.547+0000 7f47aea00640 1 --2- 172.21.15.134:0/2366869084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5f872f0 0x5627d6361150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:19:47.550 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.547+0000 7f47ae000640 1 --2- 172.21.15.134:0/2366869084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f8f4d0 0x5627d6361710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:19:47.550 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.547+0000 7f47af400640 1 --2- 172.21.15.134:0/2366869084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 0x5627d635b640 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f47a8004af0 tx=0x7f47a8009410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a2eca2dc2fcba47 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:47.550 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.547+0000 7f4798c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:19:47.551 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.548+0000 7f4798c00640 1 -- 172.21.15.134:0/2366869084 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8003970 con 0x5627d5fb01b0 2024-07-26T11:19:47.551 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.548+0000 7f47bcb78740 1 -- 172.21.15.134:0/2366869084 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f479c0ae120 con 0x5627d5fb01b0 2024-07-26T11:19:47.551 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.548+0000 7f4798c00640 1 -- 172.21.15.134:0/2366869084 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f47a8003b10 con 0x5627d5fb01b0 2024-07-26T11:19:47.551 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.549+0000 7f4798c00640 1 -- 172.21.15.134:0/2366869084 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f47a8003d10 con 0x5627d5fb01b0 2024-07-26T11:19:47.551 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.549+0000 7f47bcb78740 10 client.26134 fetch_fsmap learned FSMap version 32 2024-07-26T11:19:47.551 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.549+0000 7f47bcb78740 1 -- 172.21.15.134:0/2366869084 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5627d636e2f0 con 0x5627d5fb01b0 2024-07-26T11:19:47.554 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.549+0000 7f4798c00640 1 -- 172.21.15.134:0/2366869084 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f479c0ae120 con 0x5627d5fb01b0 2024-07-26T11:19:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.554+0000 7f4798c00640 1 -- 172.21.15.134:0/2366869084 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f47a8030880 con 0x5627d5fb01b0 2024-07-26T11:19:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.554+0000 7f47bcb78740 10 client.26134 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:19:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.554+0000 7f47bcb78740 10 client.26134 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:19:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.554+0000 7f47bcb78740 20 client.26134 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.554+0000 7f47bcb78740 1 --2- 172.21.15.134:0/2366869084 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5627d5f6ed80 0x5627d63717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.557 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.555+0000 7f47bcb78740 4 client.26134 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:19:47.557 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.555+0000 7f47bcb78740 1 -- 172.21.15.134:0/2366869084 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627d62ca1c0 con 0x5627d5f6ed80 2024-07-26T11:19:47.557 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.555+0000 7f47aea00640 1 --2- 172.21.15.134:0/2366869084 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5627d5f6ed80 0x5627d63717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.557+0000 7f47aea00640 1 --2- 172.21.15.134:0/2366869084 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5627d5f6ed80 0x5627d63717e0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c4947656cb95b92 server_cookie=12c40694bc087af9 in_seq=0 out_seq=0 2024-07-26T11:19:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.557+0000 7f4798c00640 10 client.26134 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.558+0000 7f4798c00640 1 -- 172.21.15.134:0/2366869084 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5627d62ca1c0 con 0x5627d5f6ed80 2024-07-26T11:19:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.559+0000 7f4798c00640 10 client.26134 handle_command_reply: tid=0 2024-07-26T11:19:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.559+0000 7f47bcb78740 1 client.26134 shutdown 2024-07-26T11:19:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.559+0000 7f47bcb78740 1 -- 172.21.15.134:0/2366869084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 msgr2=0x5627d635b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.559+0000 7f47bcb78740 1 --2- 172.21.15.134:0/2366869084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 0x5627d635b640 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f47a8004af0 tx=0x7f47a8009410 comp rx=0 tx=0).stop 2024-07-26T11:19:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.560+0000 7f47bcb78740 1 -- 172.21.15.134:0/2366869084 shutdown_connections 2024-07-26T11:19:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.560+0000 7f47bcb78740 1 --2- 172.21.15.134:0/2366869084 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5627d5f6ed80 0x5627d63717e0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.560+0000 7f47bcb78740 1 --2- 172.21.15.134:0/2366869084 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 0x5627d635b640 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.560+0000 7f47bcb78740 1 --2- 172.21.15.134:0/2366869084 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f8f4d0 0x5627d6361710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.560+0000 7f47bcb78740 1 --2- 172.21.15.134:0/2366869084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5f872f0 0x5627d6361150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.560+0000 7f47bcb78740 1 -- 172.21.15.134:0/2366869084 >> 172.21.15.134:0/2366869084 conn(0x5627d5fa4300 msgr2=0x5627d63500b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.560+0000 7f4798c00640 0 client.26134 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.560+0000 7f47bcb78740 1 -- 172.21.15.134:0/2366869084 shutdown_connections 2024-07-26T11:19:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.560+0000 7f47bcb78740 1 -- 172.21.15.134:0/2366869084 wait complete. 2024-07-26T11:19:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.561+0000 7f47bcb78740 20 client.26134 trim_cache size 0 max 16384 2024-07-26T11:19:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.593+0000 7f47bcb78740 1 Processor -- start 2024-07-26T11:19:47.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.593+0000 7f47bcb78740 1 -- start start 2024-07-26T11:19:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.594+0000 7f47bcb78740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5f872f0 0x5627d6350a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.594+0000 7f47bcb78740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f8f4d0 0x5627d6356af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.594+0000 7f47bcb78740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5fb01b0 0x5627d635b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.594+0000 7f47bcb78740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4790010010 con 0x5627d5f8f4d0 2024-07-26T11:19:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.594+0000 7f47bcb78740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4790011b50 con 0x5627d5fb01b0 2024-07-26T11:19:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.594+0000 7f47bcb78740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f47900106e0 con 0x5627d5f872f0 2024-07-26T11:19:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.595+0000 7f47af400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5fb01b0 0x5627d635b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:47.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.595+0000 7f47af400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5fb01b0 0x5627d635b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:45626/0 (socket says 172.21.15.134:45626) 2024-07-26T11:19:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.595+0000 7f47af400640 1 -- 172.21.15.134:0/3216124392 learned_addr learned my addr 172.21.15.134:0/3216124392 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.595+0000 7f47af400640 1 -- 172.21.15.134:0/3216124392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5f872f0 msgr2=0x5627d6350a70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:19:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.595+0000 7f47af400640 1 --2- 172.21.15.134:0/3216124392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5f872f0 0x5627d6350a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.595+0000 7f47af400640 1 -- 172.21.15.134:0/3216124392 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f8f4d0 msgr2=0x5627d6356af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.595+0000 7f47af400640 1 --2- 172.21.15.134:0/3216124392 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f8f4d0 0x5627d6356af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.595+0000 7f47af400640 1 -- 172.21.15.134:0/3216124392 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627d635a520 con 0x5627d5fb01b0 2024-07-26T11:19:47.598 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.597+0000 7f47af400640 1 --2- 172.21.15.134:0/3216124392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5fb01b0 0x5627d635b1c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009bf0 tx=0x7f47a802a8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a915494cc4b7172 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:47.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.598+0000 7f479b400640 1 -- 172.21.15.134:0/3216124392 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8057070 con 0x5627d5fb01b0 2024-07-26T11:19:47.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.598+0000 7f479b400640 1 -- 172.21.15.134:0/3216124392 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47a8036070 con 0x5627d5fb01b0 2024-07-26T11:19:47.601 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.599+0000 7f479b400640 1 -- 172.21.15.134:0/3216124392 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8044d80 con 0x5627d5fb01b0 2024-07-26T11:19:47.601 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.599+0000 7f47bcb78740 1 -- 172.21.15.134:0/3216124392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5fb01b0 msgr2=0x5627d635b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:47.601 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.599+0000 7f47bcb78740 1 --2- 172.21.15.134:0/3216124392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5fb01b0 0x5627d635b1c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009bf0 tx=0x7f47a802a8c0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.600+0000 7f47bcb78740 1 -- 172.21.15.134:0/3216124392 shutdown_connections 2024-07-26T11:19:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.600+0000 7f47bcb78740 1 --2- 172.21.15.134:0/3216124392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5fb01b0 0x5627d635b1c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.600+0000 7f47bcb78740 1 --2- 172.21.15.134:0/3216124392 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f8f4d0 0x5627d6356af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.600+0000 7f47bcb78740 1 --2- 172.21.15.134:0/3216124392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5f872f0 0x5627d6350a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.600+0000 7f47bcb78740 1 -- 172.21.15.134:0/3216124392 >> 172.21.15.134:0/3216124392 conn(0x5627d5fa4300 msgr2=0x5627d635aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:47.606 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.603+0000 7f47bcb78740 1 -- 172.21.15.134:0/3216124392 shutdown_connections 2024-07-26T11:19:47.607 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.604+0000 7f47bcb78740 1 -- 172.21.15.134:0/3216124392 wait complete. 2024-07-26T11:19:47.607 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.605+0000 7f47bcb78740 1 -- start start 2024-07-26T11:19:47.607 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.606+0000 7f47bcb78740 1 Processor -- start 2024-07-26T11:19:47.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.607+0000 7f47bcb78740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f872f0 0x5627d636c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.607+0000 7f47bcb78740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5f8f4d0 0x5627d6369540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.607+0000 7f47bcb78740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 0x5627d6369a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.607+0000 7f47bcb78740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4790011480 con 0x5627d5f872f0 2024-07-26T11:19:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.607+0000 7f47bcb78740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4790010db0 con 0x5627d5f8f4d0 2024-07-26T11:19:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.607+0000 7f47bcb78740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f479000dff0 con 0x5627d5fb01b0 2024-07-26T11:19:47.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.607+0000 7f47bcb78740 10 client.0 _ll_register_callbacks cb 0x7f47a8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.609+0000 7f47aea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f872f0 0x5627d636c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.610+0000 7f47aea00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f872f0 0x5627d636c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50660/0 (socket says 172.21.15.134:50660) 2024-07-26T11:19:47.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.610+0000 7f47aea00640 1 -- 172.21.15.134:0/2940337784 learned_addr learned my addr 172.21.15.134:0/2940337784 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.611+0000 7f47ae000640 1 --2- 172.21.15.134:0/2940337784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5f8f4d0 0x5627d6369540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.612+0000 7f47aea00640 1 -- 172.21.15.134:0/2940337784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 msgr2=0x5627d6369a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.612+0000 7f47af400640 1 --2- 172.21.15.134:0/2940337784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 0x5627d6369a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.612+0000 7f47aea00640 1 --2- 172.21.15.134:0/2940337784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 0x5627d6369a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.612+0000 7f47aea00640 1 -- 172.21.15.134:0/2940337784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5f8f4d0 msgr2=0x5627d6369540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.612+0000 7f47aea00640 1 --2- 172.21.15.134:0/2940337784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5f8f4d0 0x5627d6369540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.612+0000 7f47aea00640 1 -- 172.21.15.134:0/2940337784 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627d636a1f0 con 0x5627d5f872f0 2024-07-26T11:19:47.618 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.616+0000 7f47ae000640 1 --2- 172.21.15.134:0/2940337784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5f8f4d0 0x5627d6369540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:19:47.620 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.618+0000 7f47aea00640 1 --2- 172.21.15.134:0/2940337784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f872f0 0x5627d636c080 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f479c0175a0 tx=0x7f479c0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b508bc1dcd665f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:47.620 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.618+0000 7f4798c00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:19:47.625 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.623+0000 7f4798c00640 1 -- 172.21.15.134:0/2940337784 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c01e040 con 0x5627d5f872f0 2024-07-26T11:19:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.623+0000 7f4798c00640 1 -- 172.21.15.134:0/2940337784 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f479c0b2980 con 0x5627d5f872f0 2024-07-26T11:19:47.627 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.625+0000 7f47bcb78740 1 -- 172.21.15.134:0/2940337784 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47a8036070 con 0x5627d5f872f0 2024-07-26T11:19:47.628 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.626+0000 7f4798c00640 1 -- 172.21.15.134:0/2940337784 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f479c0ae120 con 0x5627d5f872f0 2024-07-26T11:19:47.628 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.626+0000 7f47bcb78740 10 client.15801 fetch_fsmap learned FSMap version 32 2024-07-26T11:19:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.626+0000 7f47bcb78740 1 -- 172.21.15.134:0/2940337784 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5627d63d9f10 con 0x5627d5f872f0 2024-07-26T11:19:47.629 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.627+0000 7f4798c00640 1 -- 172.21.15.134:0/2940337784 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47a8036070 con 0x5627d5f872f0 2024-07-26T11:19:47.630 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.627+0000 7f4798c00640 1 -- 172.21.15.134:0/2940337784 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f479c0e0930 con 0x5627d5f872f0 2024-07-26T11:19:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.630+0000 7f47bcb78740 10 client.15801 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:19:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.630+0000 7f47bcb78740 10 client.15801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:19:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.630+0000 7f47bcb78740 20 client.15801 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.630+0000 7f47bcb78740 1 --2- 172.21.15.134:0/2940337784 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5627d5f441c0 0x5627d63dc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.630+0000 7f47bcb78740 4 client.15801 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:19:47.634 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.630+0000 7f47bcb78740 1 -- 172.21.15.134:0/2940337784 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5627d62c5c60 con 0x5627d5f441c0 2024-07-26T11:19:47.635 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.633+0000 7f47ae000640 1 --2- 172.21.15.134:0/2940337784 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5627d5f441c0 0x5627d63dc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.636+0000 7f47ae000640 1 --2- 172.21.15.134:0/2940337784 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5627d5f441c0 0x5627d63dc1b0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66b061b670fe53d5 server_cookie=efa2793ae90403ff in_seq=0 out_seq=0 2024-07-26T11:19:47.638 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.636+0000 7f4798c00640 10 client.15801 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:47.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:47 smithi134 ceph-mon[31966]: scrub queued for path: / 2024-07-26T11:19:47.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:47 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:19:47.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:47 smithi134 ceph-mon[31966]: scrub summary: active paths [/] 2024-07-26T11:19:47.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:47 smithi134 ceph-mon[31966]: scrub summary: active paths [/] 2024-07-26T11:19:47.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:47 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:19:47.675 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:47 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:19:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:47 smithi189 ceph-mon[36790]: scrub queued for path: / 2024-07-26T11:19:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:47 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:19:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:47 smithi189 ceph-mon[36790]: scrub summary: active paths [/] 2024-07-26T11:19:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:47 smithi189 ceph-mon[36790]: scrub summary: active paths [/] 2024-07-26T11:19:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:47 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:19:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:47 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:19:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.717+0000 7f4798c00640 1 -- 172.21.15.134:0/2940337784 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5627d62c5c60 con 0x5627d5f441c0 2024-07-26T11:19:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.717+0000 7f4798c00640 10 client.15801 handle_command_reply: tid=0 2024-07-26T11:19:47.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.719+0000 7f47bcb78740 1 client.15801 shutdown 2024-07-26T11:19:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.719+0000 7f47bcb78740 1 -- 172.21.15.134:0/2940337784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f872f0 msgr2=0x5627d636c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:47.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.719+0000 7f47bcb78740 1 --2- 172.21.15.134:0/2940337784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f872f0 0x5627d636c080 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f479c0175a0 tx=0x7f479c0047d0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.719+0000 7f47bcb78740 1 -- 172.21.15.134:0/2940337784 shutdown_connections 2024-07-26T11:19:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.720+0000 7f47bcb78740 1 --2- 172.21.15.134:0/2940337784 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5627d5f441c0 0x5627d63dc1b0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.720+0000 7f47bcb78740 1 --2- 172.21.15.134:0/2940337784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d5fb01b0 0x5627d6369a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.720+0000 7f47bcb78740 1 --2- 172.21.15.134:0/2940337784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d5f8f4d0 0x5627d6369540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.720+0000 7f47bcb78740 1 --2- 172.21.15.134:0/2940337784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5627d5f872f0 0x5627d636c080 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.720+0000 7f47bcb78740 1 -- 172.21.15.134:0/2940337784 >> 172.21.15.134:0/2940337784 conn(0x5627d5fa4300 msgr2=0x5627d6350840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.720+0000 7f4798c00640 0 client.15801 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:47.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.721+0000 7f47bcb78740 1 -- 172.21.15.134:0/2940337784 shutdown_connections 2024-07-26T11:19:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.721+0000 7f47bcb78740 1 -- 172.21.15.134:0/2940337784 wait complete. 2024-07-26T11:19:47.724 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.722+0000 7f47bcb78740 20 client.15801 trim_cache size 0 max 16384 2024-07-26T11:19:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:19:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:19:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:19:47.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:19:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.727+0000 7f47b5c00640 1 -- 172.21.15.134:0/591250873 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f47940b4040 msgr2=0x7f47940b6550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.727+0000 7f47b5c00640 1 --2- 172.21.15.134:0/591250873 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f47940b4040 0x7f47940b6550 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x5627d63526c0 tx=0x7f47a000b410 comp rx=0 tx=0).stop 2024-07-26T11:19:47.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.727+0000 7f47b5c00640 1 -- 172.21.15.134:0/591250873 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47b0109fd0 msgr2=0x7f47b01a24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:47.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.727+0000 7f47b5c00640 1 --2- 172.21.15.134:0/591250873 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47b0109fd0 0x7f47b01a24a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f479c00b4a0 tx=0x7f479c00b960 comp rx=0 tx=0).stop 2024-07-26T11:19:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.730+0000 7f47b5c00640 1 -- 172.21.15.134:0/591250873 shutdown_connections 2024-07-26T11:19:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.730+0000 7f47b5c00640 1 --2- 172.21.15.134:0/591250873 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f47940b4040 0x7f47940b6550 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.730+0000 7f47b5c00640 1 --2- 172.21.15.134:0/591250873 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f47b0115fd0 0x7f47b01a7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.730+0000 7f47b5c00640 1 --2- 172.21.15.134:0/591250873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47b010a920 0x7f47b01a2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.730+0000 7f47b5c00640 1 --2- 172.21.15.134:0/591250873 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f47b0109fd0 0x7f47b01a24a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:47.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.730+0000 7f47b5c00640 1 -- 172.21.15.134:0/591250873 >> 172.21.15.134:0/591250873 conn(0x7f47b0070800 msgr2=0x7f47b010d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.731+0000 7f47b5c00640 1 -- 172.21.15.134:0/591250873 shutdown_connections 2024-07-26T11:19:47.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:19:47.731+0000 7f47b5c00640 1 -- 172.21.15.134:0/591250873 wait complete. 2024-07-26T11:19:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:47 smithi193 ceph-mon[37473]: scrub queued for path: / 2024-07-26T11:19:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:47 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:19:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:47 smithi193 ceph-mon[37473]: scrub summary: active paths [/] 2024-07-26T11:19:47.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:47 smithi193 ceph-mon[37473]: scrub summary: active paths [/] 2024-07-26T11:19:47.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:47 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:19:47.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:47 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:19:48.575 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:19:48.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:48 smithi134 ceph-mon[31966]: pgmap v595: 129 pgs: 129 active+clean; 24 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 27 KiB/s rd, 237 MiB/s wr, 1.26k op/s 2024-07-26T11:19:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:48 smithi189 ceph-mon[36790]: pgmap v595: 129 pgs: 129 active+clean; 24 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 27 KiB/s rd, 237 MiB/s wr, 1.26k op/s 2024-07-26T11:19:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:48 smithi193 ceph-mon[37473]: pgmap v595: 129 pgs: 129 active+clean; 24 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 27 KiB/s rd, 237 MiB/s wr, 1.26k op/s 2024-07-26T11:19:49.552 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:19:50.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:50 smithi134 ceph-mon[31966]: pgmap v596: 129 pgs: 129 active+clean; 25 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 42 KiB/s rd, 275 MiB/s wr, 1.40k op/s 2024-07-26T11:19:50.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:50 smithi189 ceph-mon[36790]: pgmap v596: 129 pgs: 129 active+clean; 25 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 42 KiB/s rd, 275 MiB/s wr, 1.40k op/s 2024-07-26T11:19:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:50 smithi193 ceph-mon[37473]: pgmap v596: 129 pgs: 129 active+clean; 25 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 42 KiB/s rd, 275 MiB/s wr, 1.40k op/s 2024-07-26T11:19:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:52 smithi134 ceph-mon[31966]: pgmap v597: 129 pgs: 129 active+clean; 25 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 32 KiB/s rd, 267 MiB/s wr, 1.32k op/s 2024-07-26T11:19:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:52 smithi189 ceph-mon[36790]: pgmap v597: 129 pgs: 129 active+clean; 25 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 32 KiB/s rd, 267 MiB/s wr, 1.32k op/s 2024-07-26T11:19:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:52 smithi193 ceph-mon[37473]: pgmap v597: 129 pgs: 129 active+clean; 25 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 32 KiB/s rd, 267 MiB/s wr, 1.32k op/s 2024-07-26T11:19:54.364 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:54 smithi134 ceph-mon[31966]: pgmap v598: 129 pgs: 129 active+clean; 25 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 35 KiB/s rd, 261 MiB/s wr, 1.31k op/s 2024-07-26T11:19:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:54 smithi189 ceph-mon[36790]: pgmap v598: 129 pgs: 129 active+clean; 25 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 35 KiB/s rd, 261 MiB/s wr, 1.31k op/s 2024-07-26T11:19:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:54 smithi193 ceph-mon[37473]: pgmap v598: 129 pgs: 129 active+clean; 25 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 35 KiB/s rd, 261 MiB/s wr, 1.31k op/s 2024-07-26T11:19:55.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.319+0000 7f5eeba00640 1 -- 172.21.15.134:0/3607644298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ee410fe50 msgr2=0x7f5ee4116680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:55.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.319+0000 7f5eeba00640 1 --2- 172.21.15.134:0/3607644298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ee410fe50 0x7f5ee4116680 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5edc009ca0 tx=0x7f5edc02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.319+0000 7f5eeba00640 1 -- 172.21.15.134:0/3607644298 shutdown_connections 2024-07-26T11:19:55.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.319+0000 7f5eeba00640 1 --2- 172.21.15.134:0/3607644298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ee410fe50 0x7f5ee4116680 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.319+0000 7f5eeba00640 1 --2- 172.21.15.134:0/3607644298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ee410f500 0x7f5ee410f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.319+0000 7f5eeba00640 1 --2- 172.21.15.134:0/3607644298 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ee4109bc0 0x7f5ee4109fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.319+0000 7f5eeba00640 1 -- 172.21.15.134:0/3607644298 >> 172.21.15.134:0/3607644298 conn(0x7f5ee4077b20 msgr2=0x7f5ee4107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:55.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.320+0000 7f5eeba00640 1 -- 172.21.15.134:0/3607644298 shutdown_connections 2024-07-26T11:19:55.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.320+0000 7f5eeba00640 1 -- 172.21.15.134:0/3607644298 wait complete. 2024-07-26T11:19:55.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.323+0000 7f5eeba00640 1 Processor -- start 2024-07-26T11:19:55.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.323+0000 7f5eeba00640 1 -- start start 2024-07-26T11:19:55.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.324+0000 7f5eeba00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ee4109bc0 0x7f5ee406cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.324+0000 7f5eeba00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ee410f500 0x7f5ee406d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.324+0000 7f5eeba00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ee410fe50 0x7f5ee4072340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.324+0000 7f5eeba00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee4119b60 con 0x7f5ee4109bc0 2024-07-26T11:19:55.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.324+0000 7f5eeba00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee41199e0 con 0x7f5ee410fe50 2024-07-26T11:19:55.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.324+0000 7f5eeba00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee4119ce0 con 0x7f5ee410f500 2024-07-26T11:19:55.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.324+0000 7f5ee9a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ee410fe50 0x7f5ee4072340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.325+0000 7f5ee3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ee410f500 0x7f5ee406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.325+0000 7f5ee3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ee410f500 0x7f5ee406d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53126/0 (socket says 172.21.15.134:53126) 2024-07-26T11:19:55.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.325+0000 7f5ee3e00640 1 -- 172.21.15.134:0/3959328320 learned_addr learned my addr 172.21.15.134:0/3959328320 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:55.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.325+0000 7f5ee9000640 1 --2- 172.21.15.134:0/3959328320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ee4109bc0 0x7f5ee406cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.326+0000 7f5ee9a00640 1 -- 172.21.15.134:0/3959328320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ee410f500 msgr2=0x7f5ee406d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:55.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.326+0000 7f5ee9a00640 1 --2- 172.21.15.134:0/3959328320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ee410f500 0x7f5ee406d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.326+0000 7f5ee9a00640 1 -- 172.21.15.134:0/3959328320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ee4109bc0 msgr2=0x7f5ee406cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:55.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.326+0000 7f5ee9a00640 1 --2- 172.21.15.134:0/3959328320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ee4109bc0 0x7f5ee406cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.326+0000 7f5ee9a00640 1 -- 172.21.15.134:0/3959328320 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee4072a70 con 0x7f5ee410fe50 2024-07-26T11:19:55.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.326+0000 7f5ee9a00640 1 --2- 172.21.15.134:0/3959328320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ee410fe50 0x7f5ee4072340 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5edc005e30 tx=0x7f5edc0317c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:55.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.327+0000 7f5ee1600640 1 -- 172.21.15.134:0/3959328320 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc003b70 con 0x7f5ee410fe50 2024-07-26T11:19:55.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.327+0000 7f5ee1600640 1 -- 172.21.15.134:0/3959328320 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5edc003d10 con 0x7f5ee410fe50 2024-07-26T11:19:55.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.327+0000 7f5ee1600640 1 -- 172.21.15.134:0/3959328320 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5edc02f970 con 0x7f5ee410fe50 2024-07-26T11:19:55.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.327+0000 7f5eeba00640 1 -- 172.21.15.134:0/3959328320 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ee4072ca0 con 0x7f5ee410fe50 2024-07-26T11:19:55.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.327+0000 7f5eeba00640 1 -- 172.21.15.134:0/3959328320 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ee40734b0 con 0x7f5ee410fe50 2024-07-26T11:19:55.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.335+0000 7f5ee1600640 1 -- 172.21.15.134:0/3959328320 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f5edc02fb10 con 0x7f5ee410fe50 2024-07-26T11:19:55.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.335+0000 7f5ef2671740 1 Processor -- start 2024-07-26T11:19:55.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.337+0000 7f5ef2671740 1 -- start start 2024-07-26T11:19:55.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.337+0000 7f5ef2671740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564129819070 0x564129bbdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.338+0000 7f5ef2671740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641297f01b0 0x564129bbe520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.338+0000 7f5ef2671740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641297f8390 0x564129bbea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.338+0000 7f5ef2671740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564129b34b90 con 0x5641297f01b0 2024-07-26T11:19:55.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.338+0000 7f5ef2671740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564129b24090 con 0x5641297f8390 2024-07-26T11:19:55.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.338+0000 7f5ef2671740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564129b280d0 con 0x564129819070 2024-07-26T11:19:55.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.341+0000 7f5ee1600640 1 --2- 172.21.15.134:0/3959328320 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5ecc0b3e40 0x7f5ecc0b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.342+0000 7f5ee3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564129819070 0x564129bbdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.342+0000 7f5ee3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564129819070 0x564129bbdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53134/0 (socket says 172.21.15.134:53134) 2024-07-26T11:19:55.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.342+0000 7f5ee3e00640 1 -- 172.21.15.134:0/3438859091 learned_addr learned my addr 172.21.15.134:0/3438859091 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:55.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.342+0000 7f5ee3e00640 1 --2- 172.21.15.134:0/3438859091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641297f8390 0x564129bbea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.342+0000 7f5ee1600640 1 -- 172.21.15.134:0/3959328320 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5edc105660 con 0x7f5ee410fe50 2024-07-26T11:19:55.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.343+0000 7f5ee9000640 1 --2- 172.21.15.134:0/3438859091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641297f01b0 0x564129bbe520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.344+0000 7f5ee9000640 1 -- 172.21.15.134:0/3438859091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564129819070 msgr2=0x564129bbdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:55.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.344+0000 7f5ee9000640 1 --2- 172.21.15.134:0/3438859091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564129819070 0x564129bbdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.344+0000 7f5ee9000640 1 -- 172.21.15.134:0/3438859091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641297f8390 msgr2=0x564129bbea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:55.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.344+0000 7f5ee9000640 1 --2- 172.21.15.134:0/3438859091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641297f8390 0x564129bbea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.344+0000 7f5ee9000640 1 -- 172.21.15.134:0/3438859091 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564129884650 con 0x5641297f01b0 2024-07-26T11:19:55.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.344+0000 7f5ee9a00640 1 --2- 172.21.15.134:0/3959328320 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5ecc0b3e40 0x7f5ecc0b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.346+0000 7f5ee9000640 1 --2- 172.21.15.134:0/3438859091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641297f01b0 0x564129bbe520 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8009bf0 tx=0x7f5ed8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da8d29b2e9281769 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:55.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.347+0000 7f5ed6000640 1 -- 172.21.15.134:0/3438859091 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed801e070 con 0x5641297f01b0 2024-07-26T11:19:55.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.347+0000 7f5ed6000640 1 -- 172.21.15.134:0/3438859091 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ed8009da0 con 0x5641297f01b0 2024-07-26T11:19:55.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.347+0000 7f5ed6000640 1 -- 172.21.15.134:0/3438859091 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed801e070 con 0x5641297f01b0 2024-07-26T11:19:55.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.348+0000 7f5ee9a00640 1 --2- 172.21.15.134:0/3959328320 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5ecc0b3e40 0x7f5ecc0b6350 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5edc004af0 tx=0x7f5edc004890 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:55.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.349+0000 7f5ef2671740 1 -- 172.21.15.134:0/3438859091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641297f01b0 msgr2=0x564129bbe520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:55.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.349+0000 7f5ef2671740 1 --2- 172.21.15.134:0/3438859091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641297f01b0 0x564129bbe520 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8009bf0 tx=0x7f5ed8011810 comp rx=0 tx=0).stop 2024-07-26T11:19:55.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.349+0000 7f5ef2671740 1 -- 172.21.15.134:0/3438859091 shutdown_connections 2024-07-26T11:19:55.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.349+0000 7f5ef2671740 1 --2- 172.21.15.134:0/3438859091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641297f8390 0x564129bbea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.349+0000 7f5ef2671740 1 --2- 172.21.15.134:0/3438859091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641297f01b0 0x564129bbe520 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.349+0000 7f5ef2671740 1 --2- 172.21.15.134:0/3438859091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564129819070 0x564129bbdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.349+0000 7f5ef2671740 1 -- 172.21.15.134:0/3438859091 >> 172.21.15.134:0/3438859091 conn(0x56412980d460 msgr2=0x564129bbbed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:55.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.350+0000 7f5ef2671740 1 -- 172.21.15.134:0/3438859091 shutdown_connections 2024-07-26T11:19:55.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.350+0000 7f5ef2671740 1 -- 172.21.15.134:0/3438859091 wait complete. 2024-07-26T11:19:55.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.351+0000 7f5ef2671740 1 -- start start 2024-07-26T11:19:55.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.351+0000 7f5ef2671740 1 Processor -- start 2024-07-26T11:19:55.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.352+0000 7f5ef2671740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641297f01b0 0x564129bc9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.352+0000 7f5ef2671740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f8390 0x564129bc9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.352+0000 7f5ef2671740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564129819070 0x564129bca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.352+0000 7f5ef2671740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564129987700 con 0x564129819070 2024-07-26T11:19:55.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.352+0000 7f5ef2671740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564129ac0660 con 0x5641297f01b0 2024-07-26T11:19:55.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.352+0000 7f5ef2671740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564129bcc7c0 con 0x5641297f8390 2024-07-26T11:19:55.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.352+0000 7f5ef2671740 10 client.0 _ll_register_callbacks cb 0x564129ad0110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:55.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.353+0000 7f5ee9000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f8390 0x564129bc9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.353+0000 7f5ee9000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f8390 0x564129bc9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53148/0 (socket says 172.21.15.134:53148) 2024-07-26T11:19:55.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.353+0000 7f5ee9000640 1 -- 172.21.15.134:0/2515901916 learned_addr learned my addr 172.21.15.134:0/2515901916 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:55.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.353+0000 7f5ee3e00640 1 --2- 172.21.15.134:0/2515901916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641297f01b0 0x564129bc9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.353+0000 7f5ee3e00640 1 --2- 172.21.15.134:0/2515901916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564129819070 0x564129bca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.354+0000 7f5ee9000640 1 -- 172.21.15.134:0/2515901916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641297f01b0 msgr2=0x564129bc9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:55.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.354+0000 7f5ee9000640 1 --2- 172.21.15.134:0/2515901916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641297f01b0 0x564129bc9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.354+0000 7f5ee9000640 1 -- 172.21.15.134:0/2515901916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564129819070 msgr2=0x564129bca0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:55.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.354+0000 7f5ee9000640 1 --2- 172.21.15.134:0/2515901916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564129819070 0x564129bca0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.354+0000 7f5ee9000640 1 -- 172.21.15.134:0/2515901916 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564129ad7800 con 0x5641297f8390 2024-07-26T11:19:55.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.354+0000 7f5ee3e00640 1 --2- 172.21.15.134:0/2515901916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641297f01b0 0x564129bc9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:19:55.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.355+0000 7f5ee9000640 1 --2- 172.21.15.134:0/2515901916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f8390 0x564129bc9b80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8005540 tx=0x7f5ed8005570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=847cc630a57e6ef6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:55.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.355+0000 7f5ecaa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:19:55.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.356+0000 7f5ecaa00640 1 -- 172.21.15.134:0/2515901916 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed801e070 con 0x5641297f8390 2024-07-26T11:19:55.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.356+0000 7f5ef2671740 1 -- 172.21.15.134:0/2515901916 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ed8009da0 con 0x5641297f8390 2024-07-26T11:19:55.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.356+0000 7f5ecaa00640 1 -- 172.21.15.134:0/2515901916 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5ed80048d0 con 0x5641297f8390 2024-07-26T11:19:55.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.357+0000 7f5ecaa00640 1 -- 172.21.15.134:0/2515901916 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5ed8031050 con 0x5641297f8390 2024-07-26T11:19:55.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.358+0000 7f5ef2671740 10 client.26146 fetch_fsmap learned FSMap version 32 2024-07-26T11:19:55.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.358+0000 7f5ef2671740 1 -- 172.21.15.134:0/2515901916 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564129bd8440 con 0x5641297f8390 2024-07-26T11:19:55.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.362+0000 7f5ecaa00640 1 -- 172.21.15.134:0/2515901916 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ed8009da0 con 0x5641297f8390 2024-07-26T11:19:55.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.362+0000 7f5ecaa00640 1 -- 172.21.15.134:0/2515901916 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f5ed8004040 con 0x5641297f8390 2024-07-26T11:19:55.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.362+0000 7f5ef2671740 10 client.26146 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:19:55.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.362+0000 7f5ef2671740 10 client.26146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:19:55.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.362+0000 7f5ef2671740 20 client.26146 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:55.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.362+0000 7f5ef2671740 1 --2- 172.21.15.134:0/2515901916 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5641297d79f0 0x564129bdb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.362+0000 7f5ef2671740 4 client.26146 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:19:55.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.362+0000 7f5ef2671740 1 -- 172.21.15.134:0/2515901916 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564129b36ae0 con 0x5641297d79f0 2024-07-26T11:19:55.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.369+0000 7f5ee3e00640 1 --2- 172.21.15.134:0/2515901916 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5641297d79f0 0x564129bdb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.376+0000 7f5ee3e00640 1 --2- 172.21.15.134:0/2515901916 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5641297d79f0 0x564129bdb7a0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9a9eac8be45d6cd server_cookie=dc5d5679763e9e26 in_seq=0 out_seq=0 2024-07-26T11:19:55.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.376+0000 7f5ecaa00640 10 client.26146 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:55.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.382+0000 7f5ecaa00640 1 -- 172.21.15.134:0/2515901916 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564129b36ae0 con 0x5641297d79f0 2024-07-26T11:19:55.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.382+0000 7f5ecaa00640 10 client.26146 handle_command_reply: tid=0 2024-07-26T11:19:55.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.383+0000 7f5ef2671740 1 client.26146 shutdown 2024-07-26T11:19:55.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.386+0000 7f5ef2671740 1 -- 172.21.15.134:0/2515901916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f8390 msgr2=0x564129bc9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:55.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.386+0000 7f5ef2671740 1 --2- 172.21.15.134:0/2515901916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f8390 0x564129bc9b80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8005540 tx=0x7f5ed8005570 comp rx=0 tx=0).stop 2024-07-26T11:19:55.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.392+0000 7f5ef2671740 1 -- 172.21.15.134:0/2515901916 shutdown_connections 2024-07-26T11:19:55.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.392+0000 7f5ef2671740 1 --2- 172.21.15.134:0/2515901916 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5641297d79f0 0x564129bdb7a0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.394+0000 7f5ef2671740 1 --2- 172.21.15.134:0/2515901916 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f8390 0x564129bc9b80 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.394+0000 7f5ef2671740 1 --2- 172.21.15.134:0/2515901916 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564129819070 0x564129bca0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.394+0000 7f5ef2671740 1 --2- 172.21.15.134:0/2515901916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641297f01b0 0x564129bc9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.394+0000 7f5ef2671740 1 -- 172.21.15.134:0/2515901916 >> 172.21.15.134:0/2515901916 conn(0x56412980d460 msgr2=0x564129bc1170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:55.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.394+0000 7f5ecaa00640 0 client.26146 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:55.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.394+0000 7f5ef2671740 1 -- 172.21.15.134:0/2515901916 shutdown_connections 2024-07-26T11:19:55.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.394+0000 7f5ef2671740 1 -- 172.21.15.134:0/2515901916 wait complete. 2024-07-26T11:19:55.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.398+0000 7f5ef2671740 20 client.26146 trim_cache size 0 max 16384 2024-07-26T11:19:55.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.460+0000 7f5ef2671740 1 Processor -- start 2024-07-26T11:19:55.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.460+0000 7f5ef2671740 1 -- start start 2024-07-26T11:19:55.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.461+0000 7f5ef2671740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f01b0 0x564129bc4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.461+0000 7f5ef2671740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641297f8390 0x564129bbfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.461+0000 7f5ef2671740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564129819070 0x564129bc3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.461+0000 7f5ef2671740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee41bf6e0 con 0x5641297f8390 2024-07-26T11:19:55.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.461+0000 7f5ef2671740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee41c1220 con 0x564129819070 2024-07-26T11:19:55.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.461+0000 7f5ef2671740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee41bfdb0 con 0x5641297f01b0 2024-07-26T11:19:55.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.463+0000 7f5ee3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f01b0 0x564129bc4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.463+0000 7f5ee3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f01b0 0x564129bc4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53158/0 (socket says 172.21.15.134:53158) 2024-07-26T11:19:55.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.464+0000 7f5ee3e00640 1 -- 172.21.15.134:0/1729666756 learned_addr learned my addr 172.21.15.134:0/1729666756 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:55.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.464+0000 7f5ee9000640 1 --2- 172.21.15.134:0/1729666756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641297f8390 0x564129bbfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.464+0000 7f5ee3e00640 1 --2- 172.21.15.134:0/1729666756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564129819070 0x564129bc3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.464+0000 7f5ee9000640 1 -- 172.21.15.134:0/1729666756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f01b0 msgr2=0x564129bc4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:55.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.465+0000 7f5ee9000640 1 --2- 172.21.15.134:0/1729666756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f01b0 0x564129bc4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.465+0000 7f5ee9000640 1 -- 172.21.15.134:0/1729666756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564129819070 msgr2=0x564129bc3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:55.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.465+0000 7f5ee9000640 1 --2- 172.21.15.134:0/1729666756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564129819070 0x564129bc3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.465+0000 7f5ee9000640 1 -- 172.21.15.134:0/1729666756 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564129bca900 con 0x5641297f8390 2024-07-26T11:19:55.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.465+0000 7f5ee3e00640 1 --2- 172.21.15.134:0/1729666756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564129819070 0x564129bc3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:19:55.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.466+0000 7f5ee9000640 1 --2- 172.21.15.134:0/1729666756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641297f8390 0x564129bbfae0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8000c00 tx=0x7f5ed8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63a08f13f5da6f4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:55.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.467+0000 7f5ed5600640 1 -- 172.21.15.134:0/1729666756 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed801e040 con 0x5641297f8390 2024-07-26T11:19:55.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.467+0000 7f5ed5600640 1 -- 172.21.15.134:0/1729666756 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ed8028070 con 0x5641297f8390 2024-07-26T11:19:55.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.467+0000 7f5ed5600640 1 -- 172.21.15.134:0/1729666756 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed80048d0 con 0x5641297f8390 2024-07-26T11:19:55.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.468+0000 7f5ef2671740 1 -- 172.21.15.134:0/1729666756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641297f8390 msgr2=0x564129bbfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:55.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.468+0000 7f5ef2671740 1 --2- 172.21.15.134:0/1729666756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641297f8390 0x564129bbfae0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8000c00 tx=0x7f5ed8004600 comp rx=0 tx=0).stop 2024-07-26T11:19:55.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.468+0000 7f5ef2671740 1 -- 172.21.15.134:0/1729666756 shutdown_connections 2024-07-26T11:19:55.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.468+0000 7f5ef2671740 1 --2- 172.21.15.134:0/1729666756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564129819070 0x564129bc3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.468+0000 7f5ef2671740 1 --2- 172.21.15.134:0/1729666756 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5641297f8390 0x564129bbfae0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.468+0000 7f5ef2671740 1 --2- 172.21.15.134:0/1729666756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f01b0 0x564129bc4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.468+0000 7f5ef2671740 1 -- 172.21.15.134:0/1729666756 >> 172.21.15.134:0/1729666756 conn(0x56412980d460 msgr2=0x564129bc9f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:55.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.468+0000 7f5ef2671740 1 -- 172.21.15.134:0/1729666756 shutdown_connections 2024-07-26T11:19:55.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.468+0000 7f5ef2671740 1 -- 172.21.15.134:0/1729666756 wait complete. 2024-07-26T11:19:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.469+0000 7f5ef2671740 1 -- start start 2024-07-26T11:19:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.469+0000 7f5ef2671740 1 Processor -- start 2024-07-26T11:19:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.470+0000 7f5ef2671740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641297f01b0 0x564129bcce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.470+0000 7f5ef2671740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f8390 0x564129bcd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.470+0000 7f5ef2671740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564129819070 0x564129c41dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.470+0000 7f5ef2671740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee41c0b50 con 0x564129819070 2024-07-26T11:19:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.470+0000 7f5ef2671740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee41c0480 con 0x5641297f01b0 2024-07-26T11:19:55.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.470+0000 7f5ef2671740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f5ee41bd6c0 con 0x5641297f8390 2024-07-26T11:19:55.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.470+0000 7f5ef2671740 10 client.0 _ll_register_callbacks cb 0x7f5ed0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:19:55.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.470+0000 7f5ee9000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f8390 0x564129bcd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.470+0000 7f5ee9000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f8390 0x564129bcd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53160/0 (socket says 172.21.15.134:53160) 2024-07-26T11:19:55.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.470+0000 7f5ee9000640 1 -- 172.21.15.134:0/1410832392 learned_addr learned my addr 172.21.15.134:0/1410832392 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:19:55.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.470+0000 7f5ee3e00640 1 --2- 172.21.15.134:0/1410832392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641297f01b0 0x564129bcce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.471+0000 7f5ee9000640 1 -- 172.21.15.134:0/1410832392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641297f01b0 msgr2=0x564129bcce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:55.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.471+0000 7f5ee9000640 1 --2- 172.21.15.134:0/1410832392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641297f01b0 0x564129bcce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.471+0000 7f5ee9000640 1 -- 172.21.15.134:0/1410832392 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564129819070 msgr2=0x564129c41dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:19:55.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.471+0000 7f5ee9000640 1 --2- 172.21.15.134:0/1410832392 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564129819070 0x564129c41dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.471+0000 7f5ee9000640 1 -- 172.21.15.134:0/1410832392 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564129c42300 con 0x5641297f8390 2024-07-26T11:19:55.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.472+0000 7f5ee9000640 1 --2- 172.21.15.134:0/1410832392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f8390 0x564129bcd370 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8005510 tx=0x7f5ed8009b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6728efc244874623 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:19:55.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.472+0000 7f5ecaa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:19:55.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.472+0000 7f5ecaa00640 1 -- 172.21.15.134:0/1410832392 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8045060 con 0x5641297f8390 2024-07-26T11:19:55.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.472+0000 7f5ecaa00640 1 -- 172.21.15.134:0/1410832392 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f5ed80097a0 con 0x5641297f8390 2024-07-26T11:19:55.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.473+0000 7f5ef2671740 1 -- 172.21.15.134:0/1410832392 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ed8028070 con 0x5641297f8390 2024-07-26T11:19:55.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.474+0000 7f5ecaa00640 1 -- 172.21.15.134:0/1410832392 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f5ed8023910 con 0x5641297f8390 2024-07-26T11:19:55.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.474+0000 7f5ef2671740 10 client.26152 fetch_fsmap learned FSMap version 32 2024-07-26T11:19:55.476 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.474+0000 7f5ef2671740 1 -- 172.21.15.134:0/1410832392 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564129c433e0 con 0x5641297f8390 2024-07-26T11:19:55.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.474+0000 7f5ecaa00640 1 -- 172.21.15.134:0/1410832392 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ed8028070 con 0x5641297f8390 2024-07-26T11:19:55.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.475+0000 7f5ecaa00640 1 -- 172.21.15.134:0/1410832392 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f5ed8003cf0 con 0x5641297f8390 2024-07-26T11:19:55.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.478+0000 7f5ef2671740 10 client.26152 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:19:55.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.478+0000 7f5ef2671740 10 client.26152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:19:55.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.478+0000 7f5ef2671740 20 client.26152 populate_metadata read hostname 'smithi134' 2024-07-26T11:19:55.480 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.478+0000 7f5ef2671740 1 --2- 172.21.15.134:0/1410832392 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5641297acda0 0x564129c45950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:19:55.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.481+0000 7f5ef2671740 4 client.26152 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:19:55.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.481+0000 7f5ef2671740 1 -- 172.21.15.134:0/1410832392 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564129b32580 con 0x5641297acda0 2024-07-26T11:19:55.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.481+0000 7f5ee3e00640 1 --2- 172.21.15.134:0/1410832392 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5641297acda0 0x564129c45950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:19:55.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.483+0000 7f5ecaa00640 10 client.26152 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:55.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.483+0000 7f5ee3e00640 1 --2- 172.21.15.134:0/1410832392 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5641297acda0 0x564129c45950 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7bc78efee618537 server_cookie=ec2b071849151952 in_seq=0 out_seq=0 2024-07-26T11:19:55.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.484+0000 7f5ecaa00640 1 -- 172.21.15.134:0/1410832392 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x564129b32580 con 0x5641297acda0 2024-07-26T11:19:55.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.484+0000 7f5ecaa00640 10 client.26152 handle_command_reply: tid=0 2024-07-26T11:19:55.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.484+0000 7f5ef2671740 1 client.26152 shutdown 2024-07-26T11:19:55.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.484+0000 7f5ef2671740 1 -- 172.21.15.134:0/1410832392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f8390 msgr2=0x564129bcd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:55.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.484+0000 7f5ef2671740 1 --2- 172.21.15.134:0/1410832392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f8390 0x564129bcd370 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f5ed8005510 tx=0x7f5ed8009b40 comp rx=0 tx=0).stop 2024-07-26T11:19:55.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.484+0000 7f5ef2671740 1 -- 172.21.15.134:0/1410832392 shutdown_connections 2024-07-26T11:19:55.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.484+0000 7f5ef2671740 1 --2- 172.21.15.134:0/1410832392 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5641297acda0 0x564129c45950 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.484+0000 7f5ef2671740 1 --2- 172.21.15.134:0/1410832392 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641297f8390 0x564129bcd370 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.484+0000 7f5ef2671740 1 --2- 172.21.15.134:0/1410832392 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564129819070 0x564129c41dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.484+0000 7f5ef2671740 1 --2- 172.21.15.134:0/1410832392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641297f01b0 0x564129bcce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.484+0000 7f5ef2671740 1 -- 172.21.15.134:0/1410832392 >> 172.21.15.134:0/1410832392 conn(0x56412980d460 msgr2=0x564129bc9590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:55.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.484+0000 7f5ecaa00640 0 client.26152 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:19:55.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.485+0000 7f5ef2671740 1 -- 172.21.15.134:0/1410832392 shutdown_connections 2024-07-26T11:19:55.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.485+0000 7f5ef2671740 1 -- 172.21.15.134:0/1410832392 wait complete. 2024-07-26T11:19:55.488 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:19:55.488 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "no active scrubs running", 2024-07-26T11:19:55.488 INFO:teuthology.orchestra.run.smithi134.stdout: "scrubs": {} 2024-07-26T11:19:55.488 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:19:55.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.485+0000 7f5ef2671740 20 client.26152 trim_cache size 0 max 16384 2024-07-26T11:19:55.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.486+0000 7f5eeba00640 1 -- 172.21.15.134:0/3959328320 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5ecc0b3e40 msgr2=0x7f5ecc0b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:55.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.486+0000 7f5eeba00640 1 --2- 172.21.15.134:0/3959328320 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5ecc0b3e40 0x7f5ecc0b6350 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5edc004af0 tx=0x7f5edc004890 comp rx=0 tx=0).stop 2024-07-26T11:19:55.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.486+0000 7f5eeba00640 1 -- 172.21.15.134:0/3959328320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ee410fe50 msgr2=0x7f5ee4072340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:19:55.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.486+0000 7f5eeba00640 1 --2- 172.21.15.134:0/3959328320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ee410fe50 0x7f5ee4072340 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5edc005e30 tx=0x7f5edc0317c0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.486+0000 7f5eeba00640 1 -- 172.21.15.134:0/3959328320 shutdown_connections 2024-07-26T11:19:55.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.486+0000 7f5eeba00640 1 --2- 172.21.15.134:0/3959328320 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f5ecc0b3e40 0x7f5ecc0b6350 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.486+0000 7f5eeba00640 1 --2- 172.21.15.134:0/3959328320 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ee410f500 0x7f5ee406d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.486+0000 7f5eeba00640 1 --2- 172.21.15.134:0/3959328320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ee410fe50 0x7f5ee4072340 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.486+0000 7f5eeba00640 1 --2- 172.21.15.134:0/3959328320 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5ee4109bc0 0x7f5ee406cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:19:55.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.486+0000 7f5eeba00640 1 -- 172.21.15.134:0/3959328320 >> 172.21.15.134:0/3959328320 conn(0x7f5ee4077b20 msgr2=0x7f5ee410d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:19:55.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.487+0000 7f5eeba00640 1 -- 172.21.15.134:0/3959328320 shutdown_connections 2024-07-26T11:19:55.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:19:55.487+0000 7f5eeba00640 1 -- 172.21.15.134:0/3959328320 wait complete. 2024-07-26T11:19:55.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:55 smithi134 ceph-mon[31966]: scrub summary: idle 2024-07-26T11:19:55.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:55 smithi189 ceph-mon[36790]: scrub summary: idle 2024-07-26T11:19:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:55 smithi193 ceph-mon[37473]: scrub summary: idle 2024-07-26T11:19:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:56 smithi193 ceph-mon[37473]: scrub summary: idle 2024-07-26T11:19:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:56 smithi193 ceph-mon[37473]: pgmap v599: 129 pgs: 129 active+clean; 26 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 25 KiB/s rd, 309 MiB/s wr, 1.58k op/s 2024-07-26T11:19:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:56 smithi134 ceph-mon[31966]: scrub summary: idle 2024-07-26T11:19:57.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:56 smithi134 ceph-mon[31966]: pgmap v599: 129 pgs: 129 active+clean; 26 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 25 KiB/s rd, 309 MiB/s wr, 1.58k op/s 2024-07-26T11:19:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:56 smithi189 ceph-mon[36790]: scrub summary: idle 2024-07-26T11:19:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:56 smithi189 ceph-mon[36790]: pgmap v599: 129 pgs: 129 active+clean; 26 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 25 KiB/s rd, 309 MiB/s wr, 1.58k op/s 2024-07-26T11:19:57.331 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-26T11:19:57.332 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:19:58.259 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:19:58.653 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:19:58 smithi134 ceph-mon[31966]: pgmap v600: 129 pgs: 129 active+clean; 26 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 26 KiB/s rd, 243 MiB/s wr, 1.26k op/s 2024-07-26T11:19:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:19:58 smithi189 ceph-mon[36790]: pgmap v600: 129 pgs: 129 active+clean; 26 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 26 KiB/s rd, 243 MiB/s wr, 1.26k op/s 2024-07-26T11:19:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:19:58 smithi193 ceph-mon[37473]: pgmap v600: 129 pgs: 129 active+clean; 26 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 26 KiB/s rd, 243 MiB/s wr, 1.26k op/s 2024-07-26T11:20:00.064 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:00 smithi134 ceph-mon[31966]: overall HEALTH_OK 2024-07-26T11:20:00.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:00 smithi189 ceph-mon[36790]: overall HEALTH_OK 2024-07-26T11:20:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:00 smithi193 ceph-mon[37473]: overall HEALTH_OK 2024-07-26T11:20:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.056+0000 7f98ef800640 1 -- 172.21.15.134:0/4191401826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e8109fd0 msgr2=0x7f98e810a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:01.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.056+0000 7f98ef800640 1 --2- 172.21.15.134:0/4191401826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e8109fd0 0x7f98e810a3e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f98dc009860 tx=0x7f98dc02a380 comp rx=0 tx=0).stop 2024-07-26T11:20:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.057+0000 7f98ef800640 1 -- 172.21.15.134:0/4191401826 shutdown_connections 2024-07-26T11:20:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.057+0000 7f98ef800640 1 --2- 172.21.15.134:0/4191401826 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98e810a920 0x7f98e8115a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.062 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.057+0000 7f98ef800640 1 --2- 172.21.15.134:0/4191401826 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98e8115fd0 0x7f98e81184b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.057+0000 7f98ef800640 1 --2- 172.21.15.134:0/4191401826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e8109fd0 0x7f98e810a3e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.057+0000 7f98ef800640 1 -- 172.21.15.134:0/4191401826 >> 172.21.15.134:0/4191401826 conn(0x7f98e8070800 msgr2=0x7f98e8070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.057+0000 7f98ef800640 1 -- 172.21.15.134:0/4191401826 shutdown_connections 2024-07-26T11:20:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.057+0000 7f98ef800640 1 -- 172.21.15.134:0/4191401826 wait complete. 2024-07-26T11:20:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.059+0000 7f98ef800640 1 Processor -- start 2024-07-26T11:20:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.059+0000 7f98ef800640 1 -- start start 2024-07-26T11:20:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.060+0000 7f98ef800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98e8109fd0 0x7f98e81a48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.060+0000 7f98ef800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e810a920 0x7f98e81a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.061+0000 7f98e7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e810a920 0x7f98e81a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.061+0000 7f98e7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e810a920 0x7f98e81a4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:54204/0 (socket says 172.21.15.134:54204) 2024-07-26T11:20:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.061+0000 7f98ef800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98e8115fd0 0x7f98e81aa100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.061+0000 7f98e7e00640 1 -- 172.21.15.134:0/4116220706 learned_addr learned my addr 172.21.15.134:0/4116220706 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.061+0000 7f98ed800640 1 --2- 172.21.15.134:0/4116220706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98e8115fd0 0x7f98e81aa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.061+0000 7f98ef800640 1 -- 172.21.15.134:0/4116220706 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f98e811bc70 con 0x7f98e8109fd0 2024-07-26T11:20:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.061+0000 7f98ef800640 1 -- 172.21.15.134:0/4116220706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f98e811baf0 con 0x7f98e810a920 2024-07-26T11:20:01.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.061+0000 7f98ef800640 1 -- 172.21.15.134:0/4116220706 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f98e811bdf0 con 0x7f98e8115fd0 2024-07-26T11:20:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.062+0000 7f98ed800640 1 -- 172.21.15.134:0/4116220706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e810a920 msgr2=0x7f98e81a4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.062+0000 7f98ed800640 1 --2- 172.21.15.134:0/4116220706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e810a920 0x7f98e81a4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.062+0000 7f98ed800640 1 -- 172.21.15.134:0/4116220706 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98e8109fd0 msgr2=0x7f98e81a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.062+0000 7f98ed800640 1 --2- 172.21.15.134:0/4116220706 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98e8109fd0 0x7f98e81a48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.062+0000 7f98ed800640 1 -- 172.21.15.134:0/4116220706 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e81aa800 con 0x7f98e8115fd0 2024-07-26T11:20:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.062+0000 7f98e7e00640 1 --2- 172.21.15.134:0/4116220706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e810a920 0x7f98e81a4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:20:01.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.063+0000 7f98ed800640 1 --2- 172.21.15.134:0/4116220706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98e8115fd0 0x7f98e81aa100 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f98e0009fd0 tx=0x7f98e000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:01.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.066+0000 7f98e5600640 1 -- 172.21.15.134:0/4116220706 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e000cd60 con 0x7f98e8115fd0 2024-07-26T11:20:01.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.066+0000 7f98ef800640 1 -- 172.21.15.134:0/4116220706 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e81aaaf0 con 0x7f98e8115fd0 2024-07-26T11:20:01.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.066+0000 7f98e5600640 1 -- 172.21.15.134:0/4116220706 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98e0005020 con 0x7f98e8115fd0 2024-07-26T11:20:01.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.066+0000 7f98ef800640 1 -- 172.21.15.134:0/4116220706 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e81b1280 con 0x7f98e8115fd0 2024-07-26T11:20:01.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.066+0000 7f98e5600640 1 -- 172.21.15.134:0/4116220706 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e0010670 con 0x7f98e8115fd0 2024-07-26T11:20:01.076 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.074+0000 7f98e5600640 1 -- 172.21.15.134:0/4116220706 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f98e0010810 con 0x7f98e8115fd0 2024-07-26T11:20:01.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.077+0000 7f98e5600640 1 --2- 172.21.15.134:0/4116220706 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98d00b3e70 0x7f98d00b6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.077+0000 7f98e5600640 1 -- 172.21.15.134:0/4116220706 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f98e00e58a0 con 0x7f98e8115fd0 2024-07-26T11:20:01.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.079+0000 7f98ece00640 1 --2- 172.21.15.134:0/4116220706 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98d00b3e70 0x7f98d00b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:01.083 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.082+0000 7f98f696b740 1 Processor -- start 2024-07-26T11:20:01.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.082+0000 7f98f696b740 1 -- start start 2024-07-26T11:20:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.082+0000 7f98f696b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541de1b0 0x555854581a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.084+0000 7f98ece00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541de1b0 0x555854581a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.084+0000 7f98ece00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541de1b0 0x555854581a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52558/0 (socket says 172.21.15.134:52558) 2024-07-26T11:20:01.086 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.084+0000 7f98ece00640 1 -- 172.21.15.134:0/2987189412 learned_addr learned my addr 172.21.15.134:0/2987189412 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.084+0000 7f98f696b740 1 --2- 172.21.15.134:0/2987189412 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 0x555854581fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.084+0000 7f98f696b740 1 --2- 172.21.15.134:0/2987189412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541bd4d0 0x5558545824e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.084+0000 7f98f696b740 1 -- 172.21.15.134:0/2987189412 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5558544f6270 con 0x5558541b52f0 2024-07-26T11:20:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.084+0000 7f98f696b740 1 -- 172.21.15.134:0/2987189412 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5558544e5770 con 0x5558541bd4d0 2024-07-26T11:20:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.084+0000 7f98f696b740 1 -- 172.21.15.134:0/2987189412 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5558544e97b0 con 0x5558541de1b0 2024-07-26T11:20:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.085+0000 7f98ece00640 1 --2- 172.21.15.134:0/4116220706 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98d00b3e70 0x7f98d00b6380 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f98dc005eb0 tx=0x7f98dc005e40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.085+0000 7f98ece00640 1 -- 172.21.15.134:0/2987189412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541bd4d0 msgr2=0x5558545824e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:20:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.085+0000 7f98ece00640 1 --2- 172.21.15.134:0/2987189412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541bd4d0 0x5558545824e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.085+0000 7f98ece00640 1 -- 172.21.15.134:0/2987189412 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 msgr2=0x555854581fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.085+0000 7f98ece00640 1 --2- 172.21.15.134:0/2987189412 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 0x555854581fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.085+0000 7f98ece00640 1 -- 172.21.15.134:0/2987189412 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555854249790 con 0x5558541de1b0 2024-07-26T11:20:01.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.086+0000 7f98ece00640 1 --2- 172.21.15.134:0/2987189412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541de1b0 0x555854581a60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f98dc030c10 tx=0x7f98dc039000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdc26ecbd8e93ace server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:01.090 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.086+0000 7f98da000640 1 -- 172.21.15.134:0/2987189412 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc04d070 con 0x5558541de1b0 2024-07-26T11:20:01.090 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.086+0000 7f98da000640 1 -- 172.21.15.134:0/2987189412 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98dc030680 con 0x5558541de1b0 2024-07-26T11:20:01.090 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.086+0000 7f98da000640 1 -- 172.21.15.134:0/2987189412 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc037040 con 0x5558541de1b0 2024-07-26T11:20:01.090 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.088+0000 7f98f696b740 1 -- 172.21.15.134:0/2987189412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541de1b0 msgr2=0x555854581a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:01.090 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.088+0000 7f98f696b740 1 --2- 172.21.15.134:0/2987189412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541de1b0 0x555854581a60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f98dc030c10 tx=0x7f98dc039000 comp rx=0 tx=0).stop 2024-07-26T11:20:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.088+0000 7f98f696b740 1 -- 172.21.15.134:0/2987189412 shutdown_connections 2024-07-26T11:20:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.088+0000 7f98f696b740 1 --2- 172.21.15.134:0/2987189412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541bd4d0 0x5558545824e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.088+0000 7f98f696b740 1 --2- 172.21.15.134:0/2987189412 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 0x555854581fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.088+0000 7f98f696b740 1 --2- 172.21.15.134:0/2987189412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541de1b0 0x555854581a60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.088+0000 7f98f696b740 1 -- 172.21.15.134:0/2987189412 >> 172.21.15.134:0/2987189412 conn(0x5558541d2300 msgr2=0x55585457f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.089+0000 7f98f696b740 1 -- 172.21.15.134:0/2987189412 shutdown_connections 2024-07-26T11:20:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.089+0000 7f98f696b740 1 -- 172.21.15.134:0/2987189412 wait complete. 2024-07-26T11:20:01.091 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.090+0000 7f98f696b740 1 -- start start 2024-07-26T11:20:01.092 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.090+0000 7f98f696b740 1 Processor -- start 2024-07-26T11:20:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.091+0000 7f98f696b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541b52f0 0x55585458d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.091+0000 7f98f696b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541bd4d0 0x55585458dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.091+0000 7f98f696b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541de1b0 0x555854593540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.091+0000 7f98f696b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55585441fab0 con 0x5558541de1b0 2024-07-26T11:20:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.091+0000 7f98f696b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555854482d50 con 0x5558541b52f0 2024-07-26T11:20:01.094 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.091+0000 7f98f696b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x555854590180 con 0x5558541bd4d0 2024-07-26T11:20:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.091+0000 7f98f696b740 10 client.0 _ll_register_callbacks cb 0x5558541f5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.091+0000 7f98ece00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541b52f0 0x55585458d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.092+0000 7f98ece00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541b52f0 0x55585458d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:54228/0 (socket says 172.21.15.134:54228) 2024-07-26T11:20:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.092+0000 7f98ece00640 1 -- 172.21.15.134:0/1193568481 learned_addr learned my addr 172.21.15.134:0/1193568481 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.092+0000 7f98e7e00640 1 --2- 172.21.15.134:0/1193568481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541bd4d0 0x55585458dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.092+0000 7f98ed800640 1 --2- 172.21.15.134:0/1193568481 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541de1b0 0x555854593540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.092+0000 7f98ece00640 1 -- 172.21.15.134:0/1193568481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541bd4d0 msgr2=0x55585458dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.092+0000 7f98ece00640 1 --2- 172.21.15.134:0/1193568481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541bd4d0 0x55585458dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.092+0000 7f98ece00640 1 -- 172.21.15.134:0/1193568481 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541de1b0 msgr2=0x555854593540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.092+0000 7f98ece00640 1 --2- 172.21.15.134:0/1193568481 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541de1b0 0x555854593540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.092+0000 7f98ece00640 1 -- 172.21.15.134:0/1193568481 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558542c2620 con 0x5558541b52f0 2024-07-26T11:20:01.095 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.092+0000 7f98ed800640 1 --2- 172.21.15.134:0/1193568481 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541de1b0 0x555854593540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:20:01.096 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.093+0000 7f98ece00640 1 --2- 172.21.15.134:0/1193568481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541b52f0 0x55585458d7c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f98dc006010 tx=0x7f98dc003cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1db1717e625eea2d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:01.096 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.093+0000 7f98cea00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:20:01.096 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.094+0000 7f98cea00640 1 -- 172.21.15.134:0/1193568481 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98dc067060 con 0x5558541b52f0 2024-07-26T11:20:01.096 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.094+0000 7f98cea00640 1 -- 172.21.15.134:0/1193568481 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f98dc004200 con 0x5558541b52f0 2024-07-26T11:20:01.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.094+0000 7f98f696b740 1 -- 172.21.15.134:0/1193568481 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98dc030680 con 0x5558541b52f0 2024-07-26T11:20:01.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.095+0000 7f98cea00640 1 -- 172.21.15.134:0/1193568481 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f98dc06e050 con 0x5558541b52f0 2024-07-26T11:20:01.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.095+0000 7f98f696b740 10 client.26171 fetch_fsmap learned FSMap version 32 2024-07-26T11:20:01.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.095+0000 7f98f696b740 1 -- 172.21.15.134:0/1193568481 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55585459c260 con 0x5558541b52f0 2024-07-26T11:20:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.096+0000 7f98cea00640 1 -- 172.21.15.134:0/1193568481 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98dc030680 con 0x5558541b52f0 2024-07-26T11:20:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.096+0000 7f98cea00640 1 -- 172.21.15.134:0/1193568481 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f98dc03a030 con 0x5558541b52f0 2024-07-26T11:20:01.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.096+0000 7f98f696b740 10 client.26171 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:20:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.096+0000 7f98f696b740 10 client.26171 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:20:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.096+0000 7f98f696b740 20 client.26171 populate_metadata read hostname 'smithi134' 2024-07-26T11:20:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.096+0000 7f98f696b740 1 --2- 172.21.15.134:0/1193568481 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55585419cd80 0x55585459f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.097+0000 7f98f696b740 4 client.26171 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:20:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.097+0000 7f98f696b740 1 -- 172.21.15.134:0/1193568481 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558544f81c0 con 0x55585419cd80 2024-07-26T11:20:01.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.098+0000 7f98e7e00640 1 --2- 172.21.15.134:0/1193568481 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55585419cd80 0x55585459f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:01.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.099+0000 7f98e7e00640 1 --2- 172.21.15.134:0/1193568481 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55585419cd80 0x55585459f680 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=742182d337a3cb01 server_cookie=e067ca4afa88515a in_seq=0 out_seq=0 2024-07-26T11:20:01.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.099+0000 7f98cea00640 10 client.26171 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:20:01.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.100+0000 7f98cea00640 1 -- 172.21.15.134:0/1193568481 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5558544f81c0 con 0x55585419cd80 2024-07-26T11:20:01.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.100+0000 7f98cea00640 10 client.26171 handle_command_reply: tid=0 2024-07-26T11:20:01.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.100+0000 7f98f696b740 1 client.26171 shutdown 2024-07-26T11:20:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.101+0000 7f98f696b740 1 -- 172.21.15.134:0/1193568481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541b52f0 msgr2=0x55585458d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.101+0000 7f98f696b740 1 --2- 172.21.15.134:0/1193568481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541b52f0 0x55585458d7c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f98dc006010 tx=0x7f98dc003cc0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.101+0000 7f98f696b740 1 -- 172.21.15.134:0/1193568481 shutdown_connections 2024-07-26T11:20:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.101+0000 7f98f696b740 1 --2- 172.21.15.134:0/1193568481 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55585419cd80 0x55585459f680 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.101+0000 7f98f696b740 1 --2- 172.21.15.134:0/1193568481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541bd4d0 0x55585458dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.101+0000 7f98f696b740 1 --2- 172.21.15.134:0/1193568481 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541de1b0 0x555854593540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.101+0000 7f98f696b740 1 --2- 172.21.15.134:0/1193568481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541b52f0 0x55585458d7c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.101+0000 7f98f696b740 1 -- 172.21.15.134:0/1193568481 >> 172.21.15.134:0/1193568481 conn(0x5558541d2300 msgr2=0x55585457e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.101+0000 7f98cea00640 0 client.26171 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:20:01.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.102+0000 7f98f696b740 1 -- 172.21.15.134:0/1193568481 shutdown_connections 2024-07-26T11:20:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.103+0000 7f98f696b740 1 -- 172.21.15.134:0/1193568481 wait complete. 2024-07-26T11:20:01.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.104+0000 7f98f696b740 20 client.26171 trim_cache size 0 max 16384 2024-07-26T11:20:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.136+0000 7f98f696b740 1 Processor -- start 2024-07-26T11:20:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.136+0000 7f98f696b740 1 -- start start 2024-07-26T11:20:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.136+0000 7f98f696b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 0x555854593440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.136+0000 7f98f696b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541bd4d0 0x5558545872a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.136+0000 7f98f696b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541de1b0 0x55585458a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.138 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.136+0000 7f98f696b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f98c800ff50 con 0x5558541b52f0 2024-07-26T11:20:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.136+0000 7f98f696b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f98c8011a90 con 0x5558541bd4d0 2024-07-26T11:20:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.136+0000 7f98f696b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f98c8010620 con 0x5558541de1b0 2024-07-26T11:20:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.137+0000 7f98ed800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541de1b0 0x55585458a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.137+0000 7f98e7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541bd4d0 0x5558545872a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:01.139 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.137+0000 7f98e7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541bd4d0 0x5558545872a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:54236/0 (socket says 172.21.15.134:54236) 2024-07-26T11:20:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.137+0000 7f98ece00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 0x555854593440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.137+0000 7f98e7e00640 1 -- 172.21.15.134:0/4015396674 learned_addr learned my addr 172.21.15.134:0/4015396674 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.137+0000 7f98e7e00640 1 -- 172.21.15.134:0/4015396674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541de1b0 msgr2=0x55585458a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.137+0000 7f98e7e00640 1 --2- 172.21.15.134:0/4015396674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541de1b0 0x55585458a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.137+0000 7f98e7e00640 1 -- 172.21.15.134:0/4015396674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 msgr2=0x555854593440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.137+0000 7f98e7e00640 1 --2- 172.21.15.134:0/4015396674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 0x555854593440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.137+0000 7f98e7e00640 1 -- 172.21.15.134:0/4015396674 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558545877e0 con 0x5558541bd4d0 2024-07-26T11:20:01.141 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.138+0000 7f98ece00640 1 --2- 172.21.15.134:0/4015396674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 0x555854593440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:20:01.141 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.138+0000 7f98e7e00640 1 --2- 172.21.15.134:0/4015396674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541bd4d0 0x5558545872a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f98d4000c00 tx=0x7f98d4004610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e31da73a5328bfdd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:01.141 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.139+0000 7f98d9600640 1 -- 172.21.15.134:0/4015396674 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d401e420 con 0x5558541bd4d0 2024-07-26T11:20:01.141 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.139+0000 7f98d9600640 1 -- 172.21.15.134:0/4015396674 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98d401ea70 con 0x5558541bd4d0 2024-07-26T11:20:01.141 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.139+0000 7f98d9600640 1 -- 172.21.15.134:0/4015396674 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d4012c50 con 0x5558541bd4d0 2024-07-26T11:20:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.141+0000 7f98f696b740 1 -- 172.21.15.134:0/4015396674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541bd4d0 msgr2=0x5558545872a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.141+0000 7f98f696b740 1 --2- 172.21.15.134:0/4015396674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541bd4d0 0x5558545872a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f98d4000c00 tx=0x7f98d4004610 comp rx=0 tx=0).stop 2024-07-26T11:20:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.141+0000 7f98f696b740 1 -- 172.21.15.134:0/4015396674 shutdown_connections 2024-07-26T11:20:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.141+0000 7f98f696b740 1 --2- 172.21.15.134:0/4015396674 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541de1b0 0x55585458a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.141+0000 7f98f696b740 1 --2- 172.21.15.134:0/4015396674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541bd4d0 0x5558545872a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.141+0000 7f98f696b740 1 --2- 172.21.15.134:0/4015396674 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 0x555854593440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.141+0000 7f98f696b740 1 -- 172.21.15.134:0/4015396674 >> 172.21.15.134:0/4015396674 conn(0x5558541d2300 msgr2=0x555854586b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:01.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.141+0000 7f98f696b740 1 -- 172.21.15.134:0/4015396674 shutdown_connections 2024-07-26T11:20:01.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.142+0000 7f98f696b740 1 -- 172.21.15.134:0/4015396674 wait complete. 2024-07-26T11:20:01.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.142+0000 7f98f696b740 1 -- start start 2024-07-26T11:20:01.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.143+0000 7f98f696b740 1 Processor -- start 2024-07-26T11:20:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.143+0000 7f98f696b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 0x5558545a6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.143+0000 7f98f696b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541bd4d0 0x5558545a69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.143+0000 7f98f696b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541de1b0 0x5558545ffbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.143+0000 7f98f696b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f98c80113c0 con 0x5558541b52f0 2024-07-26T11:20:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.143+0000 7f98f696b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f98c8010cf0 con 0x5558541de1b0 2024-07-26T11:20:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.143+0000 7f98f696b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f98c800df30 con 0x5558541bd4d0 2024-07-26T11:20:01.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.143+0000 7f98f696b740 10 client.0 _ll_register_callbacks cb 0x7f98dc04b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.144+0000 7f98e7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541bd4d0 0x5558545a69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.144+0000 7f98ece00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 0x5558545a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.144+0000 7f98ece00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 0x5558545a6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:42354/0 (socket says 172.21.15.134:42354) 2024-07-26T11:20:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.144+0000 7f98ece00640 1 -- 172.21.15.134:0/2860059171 learned_addr learned my addr 172.21.15.134:0/2860059171 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:01.147 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.145+0000 7f98ed800640 1 --2- 172.21.15.134:0/2860059171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541de1b0 0x5558545ffbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.145+0000 7f98ed800640 1 -- 172.21.15.134:0/2860059171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541bd4d0 msgr2=0x5558545a69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.145+0000 7f98ed800640 1 --2- 172.21.15.134:0/2860059171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541bd4d0 0x5558545a69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.145+0000 7f98ed800640 1 -- 172.21.15.134:0/2860059171 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 msgr2=0x5558545a6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:01.148 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.146+0000 7f98ed800640 1 --2- 172.21.15.134:0/2860059171 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 0x5558545a6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.146+0000 7f98ed800640 1 -- 172.21.15.134:0/2860059171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555854600250 con 0x5558541de1b0 2024-07-26T11:20:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.146+0000 7f98ece00640 1 --2- 172.21.15.134:0/2860059171 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 0x5558545a6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:20:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.147+0000 7f98ed800640 1 --2- 172.21.15.134:0/2860059171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541de1b0 0x5558545ffbe0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f98e00e2c10 tx=0x7f98e00ac6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8821fede3f419d5a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:01.149 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.147+0000 7f98cea00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:20:01.150 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.147+0000 7f98cea00640 1 -- 172.21.15.134:0/2860059171 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e00c5070 con 0x5558541de1b0 2024-07-26T11:20:01.150 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.148+0000 7f98cea00640 1 -- 172.21.15.134:0/2860059171 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f98e00e55f0 con 0x5558541de1b0 2024-07-26T11:20:01.150 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.148+0000 7f98f696b740 1 -- 172.21.15.134:0/2860059171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98d401ea70 con 0x5558541de1b0 2024-07-26T11:20:01.151 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.148+0000 7f98cea00640 1 -- 172.21.15.134:0/2860059171 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f98e00cc050 con 0x5558541de1b0 2024-07-26T11:20:01.151 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.148+0000 7f98f696b740 10 client.26177 fetch_fsmap learned FSMap version 32 2024-07-26T11:20:01.151 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.148+0000 7f98f696b740 1 -- 172.21.15.134:0/2860059171 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x555854607600 con 0x5558541de1b0 2024-07-26T11:20:01.151 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.149+0000 7f98cea00640 1 -- 172.21.15.134:0/2860059171 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98d401ea70 con 0x5558541de1b0 2024-07-26T11:20:01.151 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.149+0000 7f98cea00640 1 -- 172.21.15.134:0/2860059171 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f98e0010b00 con 0x5558541de1b0 2024-07-26T11:20:01.151 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.149+0000 7f98f696b740 10 client.26177 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:20:01.151 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.149+0000 7f98f696b740 10 client.26177 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:20:01.151 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.149+0000 7f98f696b740 20 client.26177 populate_metadata read hostname 'smithi134' 2024-07-26T11:20:01.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.149+0000 7f98f696b740 1 --2- 172.21.15.134:0/2860059171 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5558541721c0 0x555854609d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:01.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.149+0000 7f98f696b740 4 client.26177 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:20:01.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.149+0000 7f98f696b740 1 -- 172.21.15.134:0/2860059171 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5558544f3c60 con 0x5558541721c0 2024-07-26T11:20:01.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.149+0000 7f98ece00640 1 --2- 172.21.15.134:0/2860059171 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5558541721c0 0x555854609d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:01.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.150+0000 7f98cea00640 10 client.26177 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:20:01.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:01.150+0000 7f98ece00640 1 --2- 172.21.15.134:0/2860059171 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5558541721c0 0x555854609d40 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=580bf6bd92b9ac30 server_cookie=3ca818f70d5e5dea in_seq=0 out_seq=0 2024-07-26T11:20:01.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:01 smithi134 ceph-mon[31966]: pgmap v601: 129 pgs: 129 active+clean; 26 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 35 KiB/s rd, 285 MiB/s wr, 1.41k op/s 2024-07-26T11:20:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:01 smithi189 ceph-mon[36790]: pgmap v601: 129 pgs: 129 active+clean; 26 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 35 KiB/s rd, 285 MiB/s wr, 1.41k op/s 2024-07-26T11:20:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:01 smithi193 ceph-mon[37473]: pgmap v601: 129 pgs: 129 active+clean; 26 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 35 KiB/s rd, 285 MiB/s wr, 1.41k op/s 2024-07-26T11:20:02.668 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:02 smithi134 ceph-mon[31966]: pgmap v602: 129 pgs: 129 active+clean; 26 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 27 KiB/s rd, 271 MiB/s wr, 1.35k op/s 2024-07-26T11:20:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:02 smithi189 ceph-mon[36790]: pgmap v602: 129 pgs: 129 active+clean; 26 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 27 KiB/s rd, 271 MiB/s wr, 1.35k op/s 2024-07-26T11:20:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:02 smithi193 ceph-mon[37473]: pgmap v602: 129 pgs: 129 active+clean; 26 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 27 KiB/s rd, 271 MiB/s wr, 1.35k op/s 2024-07-26T11:20:04.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:04 smithi134 ceph-mon[31966]: pgmap v603: 129 pgs: 129 active+clean; 26 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 39 KiB/s rd, 255 MiB/s wr, 1.27k op/s 2024-07-26T11:20:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:04 smithi189 ceph-mon[36790]: pgmap v603: 129 pgs: 129 active+clean; 26 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 39 KiB/s rd, 255 MiB/s wr, 1.27k op/s 2024-07-26T11:20:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:04 smithi193 ceph-mon[37473]: pgmap v603: 129 pgs: 129 active+clean; 26 GiB data, 55 GiB used, 1018 GiB / 1.0 TiB avail; 39 KiB/s rd, 255 MiB/s wr, 1.27k op/s 2024-07-26T11:20:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.176+0000 7f98cea00640 1 -- 172.21.15.134:0/2860059171 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5558544f3c60 con 0x5558541721c0 2024-07-26T11:20:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.176+0000 7f98cea00640 10 client.26177 handle_command_reply: tid=0 2024-07-26T11:20:05.178 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.176+0000 7f98f696b740 1 client.26177 shutdown 2024-07-26T11:20:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.179+0000 7f98f696b740 1 -- 172.21.15.134:0/2860059171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541de1b0 msgr2=0x5558545ffbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.179+0000 7f98f696b740 1 --2- 172.21.15.134:0/2860059171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541de1b0 0x5558545ffbe0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f98e00e2c10 tx=0x7f98e00ac6a0 comp rx=0 tx=0).stop 2024-07-26T11:20:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.181+0000 7f98f696b740 1 -- 172.21.15.134:0/2860059171 shutdown_connections 2024-07-26T11:20:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.181+0000 7f98f696b740 1 --2- 172.21.15.134:0/2860059171 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5558541721c0 0x555854609d40 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.181+0000 7f98f696b740 1 --2- 172.21.15.134:0/2860059171 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5558541bd4d0 0x5558545a69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.181+0000 7f98f696b740 1 --2- 172.21.15.134:0/2860059171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558541de1b0 0x5558545ffbe0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.181+0000 7f98f696b740 1 --2- 172.21.15.134:0/2860059171 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5558541b52f0 0x5558545a6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.181+0000 7f98cea00640 0 client.26177 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:20:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.181+0000 7f98f696b740 1 -- 172.21.15.134:0/2860059171 >> 172.21.15.134:0/2860059171 conn(0x5558541d2300 msgr2=0x55585458ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.183+0000 7f98f696b740 1 -- 172.21.15.134:0/2860059171 shutdown_connections 2024-07-26T11:20:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.184+0000 7f98f696b740 1 -- 172.21.15.134:0/2860059171 wait complete. 2024-07-26T11:20:05.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.184+0000 7f98f696b740 20 client.26177 trim_cache size 0 max 16384 2024-07-26T11:20:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:20:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:20:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:20:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:20:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.186+0000 7f98ef800640 1 -- 172.21.15.134:0/4116220706 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98d00b3e70 msgr2=0x7f98d00b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.186+0000 7f98ef800640 1 --2- 172.21.15.134:0/4116220706 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98d00b3e70 0x7f98d00b6380 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f98dc005eb0 tx=0x7f98dc005e40 comp rx=0 tx=0).stop 2024-07-26T11:20:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.186+0000 7f98ef800640 1 -- 172.21.15.134:0/4116220706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98e8115fd0 msgr2=0x7f98e81aa100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:05.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.186+0000 7f98ef800640 1 --2- 172.21.15.134:0/4116220706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98e8115fd0 0x7f98e81aa100 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f98e0009fd0 tx=0x7f98e000ee50 comp rx=0 tx=0).stop 2024-07-26T11:20:05.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.192+0000 7f98ef800640 1 -- 172.21.15.134:0/4116220706 shutdown_connections 2024-07-26T11:20:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.192+0000 7f98ef800640 1 --2- 172.21.15.134:0/4116220706 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98d00b3e70 0x7f98d00b6380 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.192+0000 7f98ef800640 1 --2- 172.21.15.134:0/4116220706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f98e8115fd0 0x7f98e81aa100 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:05.195 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.192+0000 7f98ef800640 1 --2- 172.21.15.134:0/4116220706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e810a920 0x7f98e81a4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.192+0000 7f98ef800640 1 --2- 172.21.15.134:0/4116220706 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f98e8109fd0 0x7f98e81a48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.192+0000 7f98ef800640 1 -- 172.21.15.134:0/4116220706 >> 172.21.15.134:0/4116220706 conn(0x7f98e8070800 msgr2=0x7f98e8115020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.192+0000 7f98ef800640 1 -- 172.21.15.134:0/4116220706 shutdown_connections 2024-07-26T11:20:05.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:05.193+0000 7f98ef800640 1 -- 172.21.15.134:0/4116220706 wait complete. 2024-07-26T11:20:06.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:06 smithi134 ceph-mon[31966]: pgmap v604: 129 pgs: 129 active+clean; 27 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 36 KiB/s rd, 298 MiB/s wr, 1.53k op/s 2024-07-26T11:20:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:06 smithi189 ceph-mon[36790]: pgmap v604: 129 pgs: 129 active+clean; 27 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 36 KiB/s rd, 298 MiB/s wr, 1.53k op/s 2024-07-26T11:20:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:06 smithi193 ceph-mon[37473]: pgmap v604: 129 pgs: 129 active+clean; 27 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 36 KiB/s rd, 298 MiB/s wr, 1.53k op/s 2024-07-26T11:20:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.738+0000 7f8b2f000640 1 -- 172.21.15.134:0/2027823899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b28075800 msgr2=0x7f8b28075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.739+0000 7f8b2f000640 1 --2- 172.21.15.134:0/2027823899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b28075800 0x7f8b28075c10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009860 tx=0x7f8b1402a340 comp rx=0 tx=0).stop 2024-07-26T11:20:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.739+0000 7f8b2f000640 1 -- 172.21.15.134:0/2027823899 shutdown_connections 2024-07-26T11:20:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.739+0000 7f8b2f000640 1 --2- 172.21.15.134:0/2027823899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b28113eb0 0x7f8b28116380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.739+0000 7f8b2f000640 1 --2- 172.21.15.134:0/2027823899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b28075800 0x7f8b28075c10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:07.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.739+0000 7f8b2f000640 1 --2- 172.21.15.134:0/2027823899 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b28076db0 0x7f8b280751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:07.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.739+0000 7f8b2f000640 1 -- 172.21.15.134:0/2027823899 >> 172.21.15.134:0/2027823899 conn(0x7f8b280ff030 msgr2=0x7f8b28101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:07.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.740+0000 7f8b2f000640 1 -- 172.21.15.134:0/2027823899 shutdown_connections 2024-07-26T11:20:07.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.744+0000 7f8b2f000640 1 -- 172.21.15.134:0/2027823899 wait complete. 2024-07-26T11:20:07.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.745+0000 7f8b2f000640 1 Processor -- start 2024-07-26T11:20:07.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.746+0000 7f8b2f000640 1 -- start start 2024-07-26T11:20:07.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.746+0000 7f8b2f000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b28075800 0x7f8b281a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:07.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.746+0000 7f8b2f000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b28076db0 0x7f8b281a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:07.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.746+0000 7f8b2f000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b28113eb0 0x7f8b281a5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:07.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.746+0000 7f8b2f000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8b28119b80 con 0x7f8b28076db0 2024-07-26T11:20:07.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.746+0000 7f8b2f000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8b28119a00 con 0x7f8b28113eb0 2024-07-26T11:20:07.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.746+0000 7f8b2f000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8b28119d00 con 0x7f8b28075800 2024-07-26T11:20:07.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.747+0000 7f8b2d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b28113eb0 0x7f8b281a5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:07.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.747+0000 7f8b2d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b28113eb0 0x7f8b281a5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:54266/0 (socket says 172.21.15.134:54266) 2024-07-26T11:20:07.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.747+0000 7f8b2d000640 1 -- 172.21.15.134:0/1209181871 learned_addr learned my addr 172.21.15.134:0/1209181871 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:07.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.747+0000 7f8b27e00640 1 --2- 172.21.15.134:0/1209181871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b28075800 0x7f8b281a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:07.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.748+0000 7f8b27400640 1 --2- 172.21.15.134:0/1209181871 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b28076db0 0x7f8b281a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:07.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.748+0000 7f8b27e00640 1 -- 172.21.15.134:0/1209181871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b28113eb0 msgr2=0x7f8b281a5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:07.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.748+0000 7f8b27e00640 1 --2- 172.21.15.134:0/1209181871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b28113eb0 0x7f8b281a5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:07.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.748+0000 7f8b27e00640 1 -- 172.21.15.134:0/1209181871 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b28076db0 msgr2=0x7f8b281a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.748+0000 7f8b27e00640 1 --2- 172.21.15.134:0/1209181871 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b28076db0 0x7f8b281a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.748+0000 7f8b27e00640 1 -- 172.21.15.134:0/1209181871 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b281a6560 con 0x7f8b28075800 2024-07-26T11:20:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.748+0000 7f8b27e00640 1 --2- 172.21.15.134:0/1209181871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b28075800 0x7f8b281a05f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8b1800b3f0 tx=0x7f8b1800b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.749+0000 7f8b24c00640 1 -- 172.21.15.134:0/1209181871 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b18015020 con 0x7f8b28075800 2024-07-26T11:20:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.749+0000 7f8b2f000640 1 -- 172.21.15.134:0/1209181871 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b281a6850 con 0x7f8b28075800 2024-07-26T11:20:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.749+0000 7f8b2f000640 1 -- 172.21.15.134:0/1209181871 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b281acfb0 con 0x7f8b28075800 2024-07-26T11:20:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.749+0000 7f8b24c00640 1 -- 172.21.15.134:0/1209181871 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b18011ce0 con 0x7f8b28075800 2024-07-26T11:20:07.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.749+0000 7f8b24c00640 1 -- 172.21.15.134:0/1209181871 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b18015020 con 0x7f8b28075800 2024-07-26T11:20:07.276 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:20:07.276 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":32,"btime":"2024-07-26T11:19:25:331870+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:19:25.131276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24721,"mds_1":24719,"mds_2":24703,"mds_3":24688,"mds_4":24686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24635":{"gid":24635,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24641":{"gid":24641,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":32,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24719":{"gid":24719,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":28,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24721,"qdb_cluster":[24721,24719,24703,24688,24686]},"id":1}]} 2024-07-26T11:20:07.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.755+0000 7f8b2f000640 1 -- 172.21.15.134:0/1209181871 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b08005290 con 0x7f8b28075800 2024-07-26T11:20:07.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.759+0000 7f8b24c00640 1 -- 172.21.15.134:0/1209181871 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f8b18018020 con 0x7f8b28075800 2024-07-26T11:20:07.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.761+0000 7f8b24c00640 1 --2- 172.21.15.134:0/1209181871 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8b0c0b4070 0x7f8b0c0b6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:07.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.762+0000 7f8b24c00640 1 -- 172.21.15.134:0/1209181871 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f8b180e32d0 con 0x7f8b28075800 2024-07-26T11:20:07.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.762+0000 7f8b27400640 1 --2- 172.21.15.134:0/1209181871 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8b0c0b4070 0x7f8b0c0b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:07.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.763+0000 7f8b27400640 1 --2- 172.21.15.134:0/1209181871 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8b0c0b4070 0x7f8b0c0b6580 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8b1402a930 tx=0x7f8b14009490 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:07.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:06.765+0000 7f8b24c00640 1 -- 172.21.15.134:0/1209181871 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f8b180aa310 con 0x7f8b28075800 2024-07-26T11:20:07.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:07.190+0000 7f8b2f000640 1 -- 172.21.15.134:0/1209181871 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b08005580 con 0x7f8b28075800 2024-07-26T11:20:07.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:07.191+0000 7f8b24c00640 1 -- 172.21.15.134:0/1209181871 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11990 (secure 0 0 0) 0x7f8b180aa310 con 0x7f8b28075800 2024-07-26T11:20:07.277 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 32 2024-07-26T11:20:07.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:07.197+0000 7f8b2f000640 1 -- 172.21.15.134:0/1209181871 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8b0c0b4070 msgr2=0x7f8b0c0b6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:07.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:07.197+0000 7f8b2f000640 1 --2- 172.21.15.134:0/1209181871 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8b0c0b4070 0x7f8b0c0b6580 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8b1402a930 tx=0x7f8b14009490 comp rx=0 tx=0).stop 2024-07-26T11:20:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:07.197+0000 7f8b2f000640 1 -- 172.21.15.134:0/1209181871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b28075800 msgr2=0x7f8b281a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:07.198+0000 7f8b2f000640 1 --2- 172.21.15.134:0/1209181871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b28075800 0x7f8b281a05f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8b1800b3f0 tx=0x7f8b1800b8b0 comp rx=0 tx=0).stop 2024-07-26T11:20:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:07.201+0000 7f8b2f000640 1 -- 172.21.15.134:0/1209181871 shutdown_connections 2024-07-26T11:20:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:07.201+0000 7f8b2f000640 1 --2- 172.21.15.134:0/1209181871 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8b0c0b4070 0x7f8b0c0b6580 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:07.201+0000 7f8b2f000640 1 --2- 172.21.15.134:0/1209181871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b28113eb0 0x7f8b281a5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:07.201+0000 7f8b2f000640 1 --2- 172.21.15.134:0/1209181871 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8b28076db0 0x7f8b281a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:07.201+0000 7f8b2f000640 1 --2- 172.21.15.134:0/1209181871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8b28075800 0x7f8b281a05f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:07.201+0000 7f8b2f000640 1 -- 172.21.15.134:0/1209181871 >> 172.21.15.134:0/1209181871 conn(0x7f8b280ff030 msgr2=0x7f8b2810fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:07.202+0000 7f8b2f000640 1 -- 172.21.15.134:0/1209181871 shutdown_connections 2024-07-26T11:20:07.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:07.203+0000 7f8b2f000640 1 -- 172.21.15.134:0/1209181871 wait complete. 2024-07-26T11:20:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:07 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/1209181871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:20:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:07 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/1209181871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:20:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:07 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/1209181871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:20:08.780 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-07-26T11:20:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:09 smithi189 ceph-mon[36790]: pgmap v605: 129 pgs: 129 active+clean; 27 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 36 KiB/s rd, 235 MiB/s wr, 1.22k op/s 2024-07-26T11:20:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:09 smithi193 ceph-mon[37473]: pgmap v605: 129 pgs: 129 active+clean; 27 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 36 KiB/s rd, 235 MiB/s wr, 1.22k op/s 2024-07-26T11:20:09.623 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:20:09.675 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:09 smithi134 ceph-mon[31966]: pgmap v605: 129 pgs: 129 active+clean; 27 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 36 KiB/s rd, 235 MiB/s wr, 1.22k op/s 2024-07-26T11:20:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:10 smithi134 ceph-mon[31966]: pgmap v606: 129 pgs: 129 active+clean; 27 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 45 KiB/s rd, 276 MiB/s wr, 1.43k op/s 2024-07-26T11:20:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:10 smithi189 ceph-mon[36790]: pgmap v606: 129 pgs: 129 active+clean; 27 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 45 KiB/s rd, 276 MiB/s wr, 1.43k op/s 2024-07-26T11:20:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:10 smithi193 ceph-mon[37473]: pgmap v606: 129 pgs: 129 active+clean; 27 GiB data, 56 GiB used, 1017 GiB / 1.0 TiB avail; 45 KiB/s rd, 276 MiB/s wr, 1.43k op/s 2024-07-26T11:20:12.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.371+0000 7f2378200640 1 -- 172.21.15.134:0/1857428862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2370109830 msgr2=0x7f2370110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:12.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.371+0000 7f2378200640 1 --2- 172.21.15.134:0/1857428862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2370109830 0x7f2370110060 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2368009860 tx=0x7f236802a340 comp rx=0 tx=0).stop 2024-07-26T11:20:12.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.371+0000 7f2378200640 1 -- 172.21.15.134:0/1857428862 shutdown_connections 2024-07-26T11:20:12.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.371+0000 7f2378200640 1 --2- 172.21.15.134:0/1857428862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2370108ee0 0x7f23701092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.371+0000 7f2378200640 1 --2- 172.21.15.134:0/1857428862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2370109830 0x7f2370110060 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.371+0000 7f2378200640 1 --2- 172.21.15.134:0/1857428862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23701035a0 0x7f23701039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.374 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.371+0000 7f2378200640 1 -- 172.21.15.134:0/1857428862 >> 172.21.15.134:0/1857428862 conn(0x7f23700ff010 msgr2=0x7f2370101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:12.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.372+0000 7f2378200640 1 -- 172.21.15.134:0/1857428862 shutdown_connections 2024-07-26T11:20:12.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.374+0000 7f2378200640 1 -- 172.21.15.134:0/1857428862 wait complete. 2024-07-26T11:20:12.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.375+0000 7f2378200640 1 Processor -- start 2024-07-26T11:20:12.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.375+0000 7f2378200640 1 -- start start 2024-07-26T11:20:12.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.376+0000 7f2378200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23701035a0 0x7f2370109f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.376+0000 7f2378200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2370108ee0 0x7f237010a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.376+0000 7f2378200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2370109830 0x7f2370112f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.376+0000 7f2378200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f23701156f0 con 0x7f23701035a0 2024-07-26T11:20:12.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.376+0000 7f2378200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2370115570 con 0x7f2370108ee0 2024-07-26T11:20:12.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.376+0000 7f2378200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2370115870 con 0x7f2370109830 2024-07-26T11:20:12.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.377+0000 7f2375800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23701035a0 0x7f2370109f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.377+0000 7f2375800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23701035a0 0x7f2370109f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:42168/0 (socket says 172.21.15.134:42168) 2024-07-26T11:20:12.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.377+0000 7f2375800640 1 -- 172.21.15.134:0/2793373459 learned_addr learned my addr 172.21.15.134:0/2793373459 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:12.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.377+0000 7f2375800640 1 -- 172.21.15.134:0/2793373459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2370109830 msgr2=0x7f2370112f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-07-26T11:20:12.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.377+0000 7f2374e00640 1 --2- 172.21.15.134:0/2793373459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2370108ee0 0x7f237010a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.378+0000 7f2375800640 1 --2- 172.21.15.134:0/2793373459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2370109830 0x7f2370112f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.378+0000 7f2375800640 1 -- 172.21.15.134:0/2793373459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2370108ee0 msgr2=0x7f237010a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:12.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.378+0000 7f2375800640 1 --2- 172.21.15.134:0/2793373459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2370108ee0 0x7f237010a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.378+0000 7f2375800640 1 -- 172.21.15.134:0/2793373459 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23701134c0 con 0x7f23701035a0 2024-07-26T11:20:12.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.378+0000 7f2375800640 1 --2- 172.21.15.134:0/2793373459 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23701035a0 0x7f2370109f10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f236400eeb0 tx=0x7f236400c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:12.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.379+0000 7f236e000640 1 -- 172.21.15.134:0/2793373459 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2364003b70 con 0x7f23701035a0 2024-07-26T11:20:12.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.379+0000 7f236e000640 1 -- 172.21.15.134:0/2793373459 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f236400ce50 con 0x7f23701035a0 2024-07-26T11:20:12.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.380+0000 7f236e000640 1 -- 172.21.15.134:0/2793373459 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236400f9d0 con 0x7f23701035a0 2024-07-26T11:20:12.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.380+0000 7f2378200640 1 -- 172.21.15.134:0/2793373459 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2370113750 con 0x7f23701035a0 2024-07-26T11:20:12.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.380+0000 7f2378200640 1 -- 172.21.15.134:0/2793373459 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2370113c40 con 0x7f23701035a0 2024-07-26T11:20:12.389 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.386+0000 7f237f035740 1 Processor -- start 2024-07-26T11:20:12.389 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.387+0000 7f237f035740 1 -- start start 2024-07-26T11:20:12.389 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.387+0000 7f237f035740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 0x5651e409be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.387+0000 7f237f035740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 0x5651e409c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.387+0000 7f237f035740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cd6390 0x5651e409c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.387+0000 7f237f035740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5651e4012bd0 con 0x5651e3cd6390 2024-07-26T11:20:12.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.387+0000 7f237f035740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5651e40020d0 con 0x5651e3cce1b0 2024-07-26T11:20:12.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.387+0000 7f237f035740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5651e4006110 con 0x5651e3cf7070 2024-07-26T11:20:12.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.388+0000 7f2376200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 0x5651e409be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.388+0000 7f2376200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 0x5651e409be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57088/0 (socket says 172.21.15.134:57088) 2024-07-26T11:20:12.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.388+0000 7f2376200640 1 -- 172.21.15.134:0/1986629035 learned_addr learned my addr 172.21.15.134:0/1986629035 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:12.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.388+0000 7f2374e00640 1 --2- 172.21.15.134:0/1986629035 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cd6390 0x5651e409c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.389+0000 7f2374e00640 1 -- 172.21.15.134:0/1986629035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 msgr2=0x5651e409be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:12.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.389+0000 7f2374e00640 1 --2- 172.21.15.134:0/1986629035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 0x5651e409be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.389+0000 7f2374e00640 1 -- 172.21.15.134:0/1986629035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 msgr2=0x5651e409c370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:20:12.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.389+0000 7f2374e00640 1 --2- 172.21.15.134:0/1986629035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 0x5651e409c370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.389+0000 7f2374e00640 1 -- 172.21.15.134:0/1986629035 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651e3d62650 con 0x5651e3cd6390 2024-07-26T11:20:12.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.389+0000 7f2374e00640 1 --2- 172.21.15.134:0/1986629035 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cd6390 0x5651e409c8b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f237006b4e0 tx=0x7f235c009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e611dfdda6cfc6b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:12.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.390+0000 7f236e000640 1 -- 172.21.15.134:0/2793373459 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f236400fb70 con 0x7f23701035a0 2024-07-26T11:20:12.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.390+0000 7f2362a00640 1 -- 172.21.15.134:0/1986629035 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235c016070 con 0x5651e3cd6390 2024-07-26T11:20:12.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.391+0000 7f2362a00640 1 -- 172.21.15.134:0/1986629035 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f235c0054d0 con 0x5651e3cd6390 2024-07-26T11:20:12.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.391+0000 7f2362a00640 1 -- 172.21.15.134:0/1986629035 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235c004d00 con 0x5651e3cd6390 2024-07-26T11:20:12.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.392+0000 7f236e000640 1 --2- 172.21.15.134:0/2793373459 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f23580b3e90 0x7f23580b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.392+0000 7f237f035740 1 -- 172.21.15.134:0/1986629035 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cd6390 msgr2=0x5651e409c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:12.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.392+0000 7f237f035740 1 --2- 172.21.15.134:0/1986629035 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cd6390 0x5651e409c8b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f237006b4e0 tx=0x7f235c009ea0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.392+0000 7f236e000640 1 -- 172.21.15.134:0/2793373459 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f23640e4580 con 0x7f23701035a0 2024-07-26T11:20:12.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.393+0000 7f237f035740 1 -- 172.21.15.134:0/1986629035 shutdown_connections 2024-07-26T11:20:12.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.393+0000 7f237f035740 1 --2- 172.21.15.134:0/1986629035 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cd6390 0x5651e409c8b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.393+0000 7f237f035740 1 --2- 172.21.15.134:0/1986629035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 0x5651e409c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.393+0000 7f237f035740 1 --2- 172.21.15.134:0/1986629035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 0x5651e409be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.393+0000 7f237f035740 1 -- 172.21.15.134:0/1986629035 >> 172.21.15.134:0/1986629035 conn(0x5651e3ceb460 msgr2=0x5651e4099d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:12.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.393+0000 7f237f035740 1 -- 172.21.15.134:0/1986629035 shutdown_connections 2024-07-26T11:20:12.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.393+0000 7f2376200640 1 --2- 172.21.15.134:0/2793373459 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f23580b3e90 0x7f23580b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.394+0000 7f237f035740 1 -- 172.21.15.134:0/1986629035 wait complete. 2024-07-26T11:20:12.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.394+0000 7f2376200640 1 --2- 172.21.15.134:0/2793373459 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f23580b3e90 0x7f23580b63a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2368005eb0 tx=0x7f2368030340 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:12.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.395+0000 7f237f035740 1 -- start start 2024-07-26T11:20:12.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.395+0000 7f237f035740 1 Processor -- start 2024-07-26T11:20:12.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.396+0000 7f237f035740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cce1b0 0x5651e40a73d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.396+0000 7f237f035740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cd6390 0x5651e40a7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.396+0000 7f237f035740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 0x5651e40a7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.396+0000 7f237f035740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5651e3e65700 con 0x5651e3cce1b0 2024-07-26T11:20:12.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.396+0000 7f237f035740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5651e3f9e6a0 con 0x5651e3cd6390 2024-07-26T11:20:12.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.396+0000 7f237f035740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5651e40aa610 con 0x5651e3cf7070 2024-07-26T11:20:12.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.396+0000 7f237f035740 10 client.0 _ll_register_callbacks cb 0x5651e3fae150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:12.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.397+0000 7f2375800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cce1b0 0x5651e40a73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.397+0000 7f2374e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cd6390 0x5651e40a7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.397+0000 7f2374e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cd6390 0x5651e40a7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:57788/0 (socket says 172.21.15.134:57788) 2024-07-26T11:20:12.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.397+0000 7f2374e00640 1 -- 172.21.15.134:0/1683510557 learned_addr learned my addr 172.21.15.134:0/1683510557 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:12.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.398+0000 7f2376200640 1 --2- 172.21.15.134:0/1683510557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 0x5651e40a7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.398+0000 7f2374e00640 1 -- 172.21.15.134:0/1683510557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 msgr2=0x5651e40a7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:12.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.398+0000 7f2374e00640 1 --2- 172.21.15.134:0/1683510557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 0x5651e40a7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.398+0000 7f2374e00640 1 -- 172.21.15.134:0/1683510557 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cce1b0 msgr2=0x5651e40a73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:12.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.398+0000 7f2374e00640 1 --2- 172.21.15.134:0/1683510557 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cce1b0 0x5651e40a73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.398+0000 7f2374e00640 1 -- 172.21.15.134:0/1683510557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651e3fb5840 con 0x5651e3cd6390 2024-07-26T11:20:12.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.399+0000 7f2375800640 1 --2- 172.21.15.134:0/1683510557 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cce1b0 0x5651e40a73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:20:12.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.399+0000 7f2374e00640 1 --2- 172.21.15.134:0/1683510557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cd6390 0x5651e40a7910 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f235c000c00 tx=0x7f235c004cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b494e10fe237050c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:12.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.399+0000 7f2357400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:20:12.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.399+0000 7f2357400640 1 -- 172.21.15.134:0/1683510557 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235c016040 con 0x5651e3cd6390 2024-07-26T11:20:12.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.400+0000 7f2357400640 1 -- 172.21.15.134:0/1683510557 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f235c004fe0 con 0x5651e3cd6390 2024-07-26T11:20:12.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.400+0000 7f237f035740 1 -- 172.21.15.134:0/1683510557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f235c0054d0 con 0x5651e3cd6390 2024-07-26T11:20:12.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.400+0000 7f2357400640 1 -- 172.21.15.134:0/1683510557 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f235c00eea0 con 0x5651e3cd6390 2024-07-26T11:20:12.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.400+0000 7f237f035740 10 client.26189 fetch_fsmap learned FSMap version 32 2024-07-26T11:20:12.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.400+0000 7f237f035740 1 -- 172.21.15.134:0/1683510557 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5651e40b6240 con 0x5651e3cd6390 2024-07-26T11:20:12.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.400+0000 7f2357400640 1 -- 172.21.15.134:0/1683510557 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f235c0054d0 con 0x5651e3cd6390 2024-07-26T11:20:12.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.401+0000 7f2357400640 1 -- 172.21.15.134:0/1683510557 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f235c00f040 con 0x5651e3cd6390 2024-07-26T11:20:12.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.401+0000 7f237f035740 10 client.26189 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:20:12.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.401+0000 7f237f035740 10 client.26189 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-07-26T11:20:12.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.401+0000 7f237f035740 20 client.26189 populate_metadata read hostname 'smithi134' 2024-07-26T11:20:12.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.402+0000 7f237f035740 1 --2- 172.21.15.134:0/1683510557 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5651e3cb59f0 0x5651e40b9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.402+0000 7f237f035740 4 client.26189 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:20:12.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.402+0000 7f237f035740 1 -- 172.21.15.134:0/1683510557 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651e4014b20 con 0x5651e3cb59f0 2024-07-26T11:20:12.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.402+0000 7f2375800640 1 --2- 172.21.15.134:0/1683510557 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5651e3cb59f0 0x5651e40b9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.405 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.403+0000 7f2375800640 1 --2- 172.21.15.134:0/1683510557 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5651e3cb59f0 0x5651e40b9650 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c155f60f5fc62423 server_cookie=65b8140b19791cc3 in_seq=0 out_seq=0 2024-07-26T11:20:12.405 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.403+0000 7f2357400640 10 client.26189 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:20:12.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.404+0000 7f2357400640 1 -- 172.21.15.134:0/1683510557 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5651e4014b20 con 0x5651e3cb59f0 2024-07-26T11:20:12.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.404+0000 7f2357400640 10 client.26189 handle_command_reply: tid=0 2024-07-26T11:20:12.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.404+0000 7f237f035740 1 client.26189 shutdown 2024-07-26T11:20:12.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.404+0000 7f237f035740 1 -- 172.21.15.134:0/1683510557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cd6390 msgr2=0x5651e40a7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:12.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.404+0000 7f237f035740 1 --2- 172.21.15.134:0/1683510557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cd6390 0x5651e40a7910 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f235c000c00 tx=0x7f235c004cb0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.405+0000 7f237f035740 1 -- 172.21.15.134:0/1683510557 shutdown_connections 2024-07-26T11:20:12.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.405+0000 7f237f035740 1 --2- 172.21.15.134:0/1683510557 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5651e3cb59f0 0x5651e40b9650 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.405+0000 7f2357400640 0 client.26189 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:20:12.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.405+0000 7f237f035740 1 --2- 172.21.15.134:0/1683510557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 0x5651e40a7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.405+0000 7f237f035740 1 --2- 172.21.15.134:0/1683510557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cd6390 0x5651e40a7910 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.405+0000 7f237f035740 1 --2- 172.21.15.134:0/1683510557 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cce1b0 0x5651e40a73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.405+0000 7f237f035740 1 -- 172.21.15.134:0/1683510557 >> 172.21.15.134:0/1683510557 conn(0x5651e3ceb460 msgr2=0x5651e409efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:12.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.405+0000 7f237f035740 1 -- 172.21.15.134:0/1683510557 shutdown_connections 2024-07-26T11:20:12.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.405+0000 7f237f035740 1 -- 172.21.15.134:0/1683510557 wait complete. 2024-07-26T11:20:12.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.405+0000 7f237f035740 20 client.26189 trim_cache size 0 max 16384 2024-07-26T11:20:12.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.429+0000 7f237f035740 1 Processor -- start 2024-07-26T11:20:12.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.430+0000 7f237f035740 1 -- start start 2024-07-26T11:20:12.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.430+0000 7f237f035740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 0x5651e40a92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.430+0000 7f237f035740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cd6390 0x5651e409d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.430+0000 7f237f035740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cf7070 0x5651e40bec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.430+0000 7f237f035740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f23701bf710 con 0x5651e3cf7070 2024-07-26T11:20:12.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.430+0000 7f237f035740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f23701c1250 con 0x5651e3cce1b0 2024-07-26T11:20:12.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.430+0000 7f237f035740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f23701bfde0 con 0x5651e3cd6390 2024-07-26T11:20:12.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.431+0000 7f2374e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cd6390 0x5651e409d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.431+0000 7f2374e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cd6390 0x5651e409d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57110/0 (socket says 172.21.15.134:57110) 2024-07-26T11:20:12.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.431+0000 7f2374e00640 1 -- 172.21.15.134:0/1088011892 learned_addr learned my addr 172.21.15.134:0/1088011892 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:12.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.431+0000 7f2375800640 1 --2- 172.21.15.134:0/1088011892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 0x5651e40a92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.431+0000 7f2376200640 1 --2- 172.21.15.134:0/1088011892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cf7070 0x5651e40bec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.432+0000 7f2375800640 1 -- 172.21.15.134:0/1088011892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cd6390 msgr2=0x5651e409d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:12.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.432+0000 7f2375800640 1 --2- 172.21.15.134:0/1088011892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cd6390 0x5651e409d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.432+0000 7f2375800640 1 -- 172.21.15.134:0/1088011892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cf7070 msgr2=0x5651e40bec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:12.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.432+0000 7f2375800640 1 --2- 172.21.15.134:0/1088011892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cf7070 0x5651e40bec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.432+0000 7f2375800640 1 -- 172.21.15.134:0/1088011892 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651e40a8690 con 0x5651e3cce1b0 2024-07-26T11:20:12.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.432+0000 7f2375800640 1 --2- 172.21.15.134:0/1088011892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 0x5651e40a92b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f23640e1a60 tx=0x7f2364011040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd826fe9c6fc8b58 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:12.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.432+0000 7f2374e00640 1 --2- 172.21.15.134:0/1088011892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cd6390 0x5651e409d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:20:12.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.433+0000 7f2362000640 1 -- 172.21.15.134:0/1088011892 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23640bd070 con 0x5651e3cce1b0 2024-07-26T11:20:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.433+0000 7f2362000640 1 -- 172.21.15.134:0/1088011892 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23640af520 con 0x5651e3cce1b0 2024-07-26T11:20:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.433+0000 7f2362000640 1 -- 172.21.15.134:0/1088011892 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23640b3b40 con 0x5651e3cce1b0 2024-07-26T11:20:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.434+0000 7f237f035740 1 -- 172.21.15.134:0/1088011892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 msgr2=0x5651e40a92b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.434+0000 7f237f035740 1 --2- 172.21.15.134:0/1088011892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 0x5651e40a92b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f23640e1a60 tx=0x7f2364011040 comp rx=0 tx=0).stop 2024-07-26T11:20:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.435+0000 7f237f035740 1 -- 172.21.15.134:0/1088011892 shutdown_connections 2024-07-26T11:20:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.435+0000 7f237f035740 1 --2- 172.21.15.134:0/1088011892 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cd6390 0x5651e409d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.435+0000 7f237f035740 1 --2- 172.21.15.134:0/1088011892 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cf7070 0x5651e40bec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.435+0000 7f237f035740 1 --2- 172.21.15.134:0/1088011892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 0x5651e40a92b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.435+0000 7f237f035740 1 -- 172.21.15.134:0/1088011892 >> 172.21.15.134:0/1088011892 conn(0x5651e3ceb460 msgr2=0x5651e409ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:12.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.436+0000 7f237f035740 1 -- 172.21.15.134:0/1088011892 shutdown_connections 2024-07-26T11:20:12.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.436+0000 7f237f035740 1 -- 172.21.15.134:0/1088011892 wait complete. 2024-07-26T11:20:12.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.437+0000 7f237f035740 1 -- start start 2024-07-26T11:20:12.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.437+0000 7f237f035740 1 Processor -- start 2024-07-26T11:20:12.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.438+0000 7f237f035740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 0x5651e40abe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.438+0000 7f237f035740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cd6390 0x5651e40ac370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.438+0000 7f237f035740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 0x5651e40ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.438+0000 7f2375800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 0x5651e40abe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.438+0000 7f2375800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 0x5651e40abe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:57808/0 (socket says 172.21.15.134:57808) 2024-07-26T11:20:12.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.439+0000 7f2376200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 0x5651e40ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.439+0000 7f2376200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 0x5651e40ac8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57112/0 (socket says 172.21.15.134:57112) 2024-07-26T11:20:12.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.439+0000 7f2374e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cd6390 0x5651e40ac370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.439+0000 7f2374e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cd6390 0x5651e40ac370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:42200/0 (socket says 172.21.15.134:42200) 2024-07-26T11:20:12.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.439+0000 7f2375800640 1 -- 172.21.15.134:0/3116175679 learned_addr learned my addr 172.21.15.134:0/3116175679 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:12.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.439+0000 7f237f035740 1 -- 172.21.15.134:0/3116175679 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f23701c0b80 con 0x5651e3cd6390 2024-07-26T11:20:12.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.439+0000 7f237f035740 1 -- 172.21.15.134:0/3116175679 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f23701c04b0 con 0x5651e3cce1b0 2024-07-26T11:20:12.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.439+0000 7f237f035740 1 -- 172.21.15.134:0/3116175679 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f23701bd6f0 con 0x5651e3cf7070 2024-07-26T11:20:12.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.439+0000 7f237f035740 10 client.0 _ll_register_callbacks cb 0x7f235c00ed60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:12.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.440+0000 7f2376200640 1 -- 172.21.15.134:0/3116175679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 msgr2=0x5651e40abe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:12.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.440+0000 7f2376200640 1 --2- 172.21.15.134:0/3116175679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 0x5651e40abe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.440+0000 7f2376200640 1 -- 172.21.15.134:0/3116175679 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cd6390 msgr2=0x5651e40ac370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.440+0000 7f2376200640 1 --2- 172.21.15.134:0/3116175679 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cd6390 0x5651e40ac370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.440+0000 7f2376200640 1 -- 172.21.15.134:0/3116175679 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651e411fda0 con 0x5651e3cf7070 2024-07-26T11:20:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.441+0000 7f2375800640 1 --2- 172.21.15.134:0/3116175679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 0x5651e40abe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:20:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.441+0000 7f2376200640 1 --2- 172.21.15.134:0/3116175679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 0x5651e40ac8b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f23680057c0 tx=0x7f23680056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d15cc10d7b9d1da5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.441+0000 7f2357400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:20:12.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.442+0000 7f2357400640 1 -- 172.21.15.134:0/3116175679 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236804e070 con 0x5651e3cf7070 2024-07-26T11:20:12.444 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:12 smithi134 ceph-mon[31966]: pgmap v607: 129 pgs: 129 active+clean; 27 GiB data, 56 GiB used, 1016 GiB / 1.0 TiB avail; 35 KiB/s rd, 259 MiB/s wr, 1.39k op/s 2024-07-26T11:20:12.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.442+0000 7f2357400640 1 -- 172.21.15.134:0/3116175679 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2368004050 con 0x5651e3cf7070 2024-07-26T11:20:12.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.442+0000 7f237f035740 1 -- 172.21.15.134:0/3116175679 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23640af520 con 0x5651e3cf7070 2024-07-26T11:20:12.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.443+0000 7f2357400640 1 -- 172.21.15.134:0/3116175679 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2368058050 con 0x5651e3cf7070 2024-07-26T11:20:12.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.443+0000 7f237f035740 10 client.26191 fetch_fsmap learned FSMap version 32 2024-07-26T11:20:12.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.443+0000 7f237f035740 1 -- 172.21.15.134:0/3116175679 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5651e4121020 con 0x5651e3cf7070 2024-07-26T11:20:12.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.443+0000 7f2357400640 1 -- 172.21.15.134:0/3116175679 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23640af520 con 0x5651e3cf7070 2024-07-26T11:20:12.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.445+0000 7f2357400640 1 -- 172.21.15.134:0/3116175679 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f2368053440 con 0x5651e3cf7070 2024-07-26T11:20:12.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.445+0000 7f237f035740 10 client.26191 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:20:12.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.445+0000 7f237f035740 10 client.26191 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-07-26T11:20:12.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.445+0000 7f237f035740 20 client.26191 populate_metadata read hostname 'smithi134' 2024-07-26T11:20:12.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.445+0000 7f237f035740 1 --2- 172.21.15.134:0/3116175679 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5651e3c8ada0 0x5651e4123800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:12.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.445+0000 7f237f035740 4 client.26191 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:20:12.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.445+0000 7f237f035740 1 -- 172.21.15.134:0/3116175679 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5651e40105c0 con 0x5651e3c8ada0 2024-07-26T11:20:12.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.445+0000 7f2375800640 1 --2- 172.21.15.134:0/3116175679 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5651e3c8ada0 0x5651e4123800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:12.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.447+0000 7f2375800640 1 --2- 172.21.15.134:0/3116175679 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5651e3c8ada0 0x5651e4123800 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=99030551469aa68e server_cookie=99049700149b0e80 in_seq=0 out_seq=0 2024-07-26T11:20:12.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.447+0000 7f2357400640 10 client.26191 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:20:12.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.447+0000 7f2357400640 1 -- 172.21.15.134:0/3116175679 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5651e40105c0 con 0x5651e3c8ada0 2024-07-26T11:20:12.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.447+0000 7f2357400640 10 client.26191 handle_command_reply: tid=0 2024-07-26T11:20:12.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.447+0000 7f237f035740 1 client.26191 shutdown 2024-07-26T11:20:12.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.447+0000 7f237f035740 1 -- 172.21.15.134:0/3116175679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 msgr2=0x5651e40ac8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:12.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.447+0000 7f237f035740 1 --2- 172.21.15.134:0/3116175679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 0x5651e40ac8b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f23680057c0 tx=0x7f23680056d0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.447+0000 7f237f035740 1 -- 172.21.15.134:0/3116175679 shutdown_connections 2024-07-26T11:20:12.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.448+0000 7f237f035740 1 --2- 172.21.15.134:0/3116175679 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5651e3c8ada0 0x5651e4123800 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.448+0000 7f237f035740 1 --2- 172.21.15.134:0/3116175679 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5651e3cf7070 0x5651e40ac8b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.448+0000 7f237f035740 1 --2- 172.21.15.134:0/3116175679 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5651e3cd6390 0x5651e40ac370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.448+0000 7f237f035740 1 --2- 172.21.15.134:0/3116175679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5651e3cce1b0 0x5651e40abe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.448+0000 7f237f035740 1 -- 172.21.15.134:0/3116175679 >> 172.21.15.134:0/3116175679 conn(0x5651e3ceb460 msgr2=0x5651e40a9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:12.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.448+0000 7f2357400640 0 client.26191 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:20:12.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.449+0000 7f237f035740 1 -- 172.21.15.134:0/3116175679 shutdown_connections 2024-07-26T11:20:12.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.450+0000 7f237f035740 1 -- 172.21.15.134:0/3116175679 wait complete. 2024-07-26T11:20:12.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.450+0000 7f237f035740 20 client.26191 trim_cache size 0 max 16384 2024-07-26T11:20:12.452 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:20:12.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.451+0000 7f2378200640 1 -- 172.21.15.134:0/2793373459 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f23580b3e90 msgr2=0x7f23580b63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:12.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.451+0000 7f2378200640 1 --2- 172.21.15.134:0/2793373459 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f23580b3e90 0x7f23580b63a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2368005eb0 tx=0x7f2368030340 comp rx=0 tx=0).stop 2024-07-26T11:20:12.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.451+0000 7f2378200640 1 -- 172.21.15.134:0/2793373459 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23701035a0 msgr2=0x7f2370109f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:12.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.451+0000 7f2378200640 1 --2- 172.21.15.134:0/2793373459 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23701035a0 0x7f2370109f10 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f236400eeb0 tx=0x7f236400c6e0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.451+0000 7f2378200640 1 -- 172.21.15.134:0/2793373459 shutdown_connections 2024-07-26T11:20:12.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.451+0000 7f2378200640 1 --2- 172.21.15.134:0/2793373459 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f23580b3e90 0x7f23580b63a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.451+0000 7f2378200640 1 --2- 172.21.15.134:0/2793373459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2370109830 0x7f2370112f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.451+0000 7f2378200640 1 --2- 172.21.15.134:0/2793373459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2370108ee0 0x7f237010a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.451+0000 7f2378200640 1 --2- 172.21.15.134:0/2793373459 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f23701035a0 0x7f2370109f10 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:12.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.451+0000 7f2378200640 1 -- 172.21.15.134:0/2793373459 >> 172.21.15.134:0/2793373459 conn(0x7f23700ff010 msgr2=0x7f2370101430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:12.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.452+0000 7f2378200640 1 -- 172.21.15.134:0/2793373459 shutdown_connections 2024-07-26T11:20:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:12.452+0000 7f2378200640 1 -- 172.21.15.134:0/2793373459 wait complete. 2024-07-26T11:20:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:12 smithi189 ceph-mon[36790]: pgmap v607: 129 pgs: 129 active+clean; 27 GiB data, 56 GiB used, 1016 GiB / 1.0 TiB avail; 35 KiB/s rd, 259 MiB/s wr, 1.39k op/s 2024-07-26T11:20:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:12 smithi193 ceph-mon[37473]: pgmap v607: 129 pgs: 129 active+clean; 27 GiB data, 56 GiB used, 1016 GiB / 1.0 TiB avail; 35 KiB/s rd, 259 MiB/s wr, 1.39k op/s 2024-07-26T11:20:13.946 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-07-26T11:20:14.582 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:20:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:14 smithi134 ceph-mon[31966]: pgmap v608: 129 pgs: 129 active+clean; 27 GiB data, 56 GiB used, 1016 GiB / 1.0 TiB avail; 34 KiB/s rd, 237 MiB/s wr, 1.30k op/s 2024-07-26T11:20:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:14 smithi189 ceph-mon[36790]: pgmap v608: 129 pgs: 129 active+clean; 27 GiB data, 56 GiB used, 1016 GiB / 1.0 TiB avail; 34 KiB/s rd, 237 MiB/s wr, 1.30k op/s 2024-07-26T11:20:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:14 smithi193 ceph-mon[37473]: pgmap v608: 129 pgs: 129 active+clean; 27 GiB data, 56 GiB used, 1016 GiB / 1.0 TiB avail; 34 KiB/s rd, 237 MiB/s wr, 1.30k op/s 2024-07-26T11:20:16.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:16 smithi134 ceph-mon[31966]: pgmap v609: 129 pgs: 129 active+clean; 27 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 21 KiB/s rd, 216 MiB/s wr, 1.37k op/s 2024-07-26T11:20:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:16 smithi189 ceph-mon[36790]: pgmap v609: 129 pgs: 129 active+clean; 27 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 21 KiB/s rd, 216 MiB/s wr, 1.37k op/s 2024-07-26T11:20:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:16 smithi193 ceph-mon[37473]: pgmap v609: 129 pgs: 129 active+clean; 27 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 21 KiB/s rd, 216 MiB/s wr, 1.37k op/s 2024-07-26T11:20:17.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.509+0000 7fa439600640 1 -- 172.21.15.134:0/211581991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa434111970 msgr2=0x7fa434109df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:17.512 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.509+0000 7fa439600640 1 --2- 172.21.15.134:0/211581991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa434111970 0x7fa434109df0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa424009900 tx=0x7fa42402a440 comp rx=0 tx=0).stop 2024-07-26T11:20:17.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.510+0000 7fa439600640 1 -- 172.21.15.134:0/211581991 shutdown_connections 2024-07-26T11:20:17.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.510+0000 7fa439600640 1 --2- 172.21.15.134:0/211581991 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa43411c660 0x7fa43411eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.510+0000 7fa439600640 1 --2- 172.21.15.134:0/211581991 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa43410a410 0x7fa43410a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.510+0000 7fa439600640 1 --2- 172.21.15.134:0/211581991 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa434111970 0x7fa434109df0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.510+0000 7fa439600640 1 -- 172.21.15.134:0/211581991 >> 172.21.15.134:0/211581991 conn(0x7fa4340708f0 msgr2=0x7fa434070d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:17.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.511+0000 7fa439600640 1 -- 172.21.15.134:0/211581991 shutdown_connections 2024-07-26T11:20:17.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.511+0000 7fa439600640 1 -- 172.21.15.134:0/211581991 wait complete. 2024-07-26T11:20:17.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.513+0000 7fa439600640 1 Processor -- start 2024-07-26T11:20:17.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.513+0000 7fa439600640 1 -- start start 2024-07-26T11:20:17.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.514+0000 7fa439600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa43410a410 0x7fa4341a8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.514+0000 7fa439600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa434111970 0x7fa4341a9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.515+0000 7fa439600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa43411c660 0x7fa4341ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.515+0000 7fa432a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa43410a410 0x7fa4341a8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:17.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.517+0000 7fa432a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa43410a410 0x7fa4341a8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:57836/0 (socket says 172.21.15.134:57836) 2024-07-26T11:20:17.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.517+0000 7fa432a00640 1 -- 172.21.15.134:0/4228418887 learned_addr learned my addr 172.21.15.134:0/4228418887 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:17.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.516+0000 7fa432000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa434111970 0x7fa4341a9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:17.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.517+0000 7fa439600640 1 -- 172.21.15.134:0/4228418887 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa434122410 con 0x7fa43411c660 2024-07-26T11:20:17.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.517+0000 7fa439600640 1 -- 172.21.15.134:0/4228418887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa434122290 con 0x7fa43410a410 2024-07-26T11:20:17.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.517+0000 7fa439600640 1 -- 172.21.15.134:0/4228418887 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa434122590 con 0x7fa434111970 2024-07-26T11:20:17.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.518+0000 7fa432000640 1 -- 172.21.15.134:0/4228418887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa43410a410 msgr2=0x7fa4341a8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:17.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.519+0000 7fa432000640 1 --2- 172.21.15.134:0/4228418887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa43410a410 0x7fa4341a8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.519+0000 7fa432000640 1 -- 172.21.15.134:0/4228418887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa43411c660 msgr2=0x7fa4341ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:17.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.519+0000 7fa432000640 1 --2- 172.21.15.134:0/4228418887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa43411c660 0x7fa4341ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.519+0000 7fa432000640 1 -- 172.21.15.134:0/4228418887 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4341aec40 con 0x7fa434111970 2024-07-26T11:20:17.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.520+0000 7fa432000640 1 --2- 172.21.15.134:0/4228418887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa434111970 0x7fa4341a9270 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa43406c110 tx=0x7fa42000b850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:17.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.524+0000 7fa42b400640 1 -- 172.21.15.134:0/4228418887 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa420015020 con 0x7fa434111970 2024-07-26T11:20:17.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.527+0000 7fa42b400640 1 -- 172.21.15.134:0/4228418887 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa420011ce0 con 0x7fa434111970 2024-07-26T11:20:17.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.527+0000 7fa439600640 1 -- 172.21.15.134:0/4228418887 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4341aef30 con 0x7fa434111970 2024-07-26T11:20:17.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.527+0000 7fa42b400640 1 -- 172.21.15.134:0/4228418887 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa420010e50 con 0x7fa434111970 2024-07-26T11:20:17.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.530+0000 7fa439600640 1 -- 172.21.15.134:0/4228418887 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4341b5690 con 0x7fa434111970 2024-07-26T11:20:17.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.542+0000 7fa42b400640 1 -- 172.21.15.134:0/4228418887 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fa420018020 con 0x7fa434111970 2024-07-26T11:20:17.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.543+0000 7fa44028d740 1 Processor -- start 2024-07-26T11:20:17.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.546+0000 7fa44028d740 1 -- start start 2024-07-26T11:20:17.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.547+0000 7fa42b400640 1 --2- 172.21.15.134:0/4228418887 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa4180b40b0 0x7fa4180b65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.548+0000 7fa44028d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4df17070 0x55dc4e2bbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.548+0000 7fa44028d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4deee1b0 0x55dc4e2bc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.548+0000 7fa42b400640 1 -- 172.21.15.134:0/4228418887 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa4200e3360 con 0x7fa434111970 2024-07-26T11:20:17.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.548+0000 7fa44028d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4def6390 0x55dc4e2bc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.548+0000 7fa44028d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55dc4e232bf0 con 0x55dc4df17070 2024-07-26T11:20:17.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.548+0000 7fa44028d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dc4e2220f0 con 0x55dc4def6390 2024-07-26T11:20:17.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.548+0000 7fa44028d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55dc4e226130 con 0x55dc4deee1b0 2024-07-26T11:20:17.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.548+0000 7fa432a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4df17070 0x55dc4e2bbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:17.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.549+0000 7fa432a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4df17070 0x55dc4e2bbe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:42226/0 (socket says 172.21.15.134:42226) 2024-07-26T11:20:17.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.549+0000 7fa432a00640 1 -- 172.21.15.134:0/1055671048 learned_addr learned my addr 172.21.15.134:0/1055671048 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:17.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.549+0000 7fa433400640 1 --2- 172.21.15.134:0/1055671048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4def6390 0x55dc4e2bc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:17.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.550+0000 7fa432000640 1 --2- 172.21.15.134:0/1055671048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4deee1b0 0x55dc4e2bc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:17.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.551+0000 7fa433400640 1 -- 172.21.15.134:0/1055671048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4deee1b0 msgr2=0x55dc4e2bc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:17.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.551+0000 7fa433400640 1 --2- 172.21.15.134:0/1055671048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4deee1b0 0x55dc4e2bc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.551+0000 7fa433400640 1 -- 172.21.15.134:0/1055671048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4df17070 msgr2=0x55dc4e2bbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:17.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.551+0000 7fa433400640 1 --2- 172.21.15.134:0/1055671048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4df17070 0x55dc4e2bbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.551+0000 7fa433400640 1 -- 172.21.15.134:0/1055671048 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc4df82650 con 0x55dc4def6390 2024-07-26T11:20:17.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.551+0000 7fa433400640 1 --2- 172.21.15.134:0/4228418887 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa4180b40b0 0x7fa4180b65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:17.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.553+0000 7fa432000640 1 --2- 172.21.15.134:0/1055671048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4deee1b0 0x55dc4e2bc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:20:17.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.555+0000 7fa433400640 1 --2- 172.21.15.134:0/1055671048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4def6390 0x55dc4e2bc8d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa42c00bc90 tx=0x7fa42c00d7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ae244a62e897945 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:17.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.555+0000 7fa41f400640 1 -- 172.21.15.134:0/1055671048 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42c006550 con 0x55dc4def6390 2024-07-26T11:20:17.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.556+0000 7fa41f400640 1 -- 172.21.15.134:0/1055671048 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa42c016380 con 0x55dc4def6390 2024-07-26T11:20:17.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.557+0000 7fa44028d740 1 -- 172.21.15.134:0/1055671048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4def6390 msgr2=0x55dc4e2bc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:17.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.559+0000 7fa41f400640 1 -- 172.21.15.134:0/1055671048 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42c006550 con 0x55dc4def6390 2024-07-26T11:20:17.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.559+0000 7fa44028d740 1 --2- 172.21.15.134:0/1055671048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4def6390 0x55dc4e2bc8d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa42c00bc90 tx=0x7fa42c00d7e0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.560+0000 7fa433400640 1 --2- 172.21.15.134:0/4228418887 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa4180b40b0 0x7fa4180b65c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa42c00bc90 tx=0x7fa42c00d7e0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:17.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.561+0000 7fa44028d740 1 -- 172.21.15.134:0/1055671048 shutdown_connections 2024-07-26T11:20:17.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.561+0000 7fa44028d740 1 --2- 172.21.15.134:0/1055671048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4deee1b0 0x55dc4e2bc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.561+0000 7fa44028d740 1 --2- 172.21.15.134:0/1055671048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4def6390 0x55dc4e2bc8d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.561+0000 7fa44028d740 1 --2- 172.21.15.134:0/1055671048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4df17070 0x55dc4e2bbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.561+0000 7fa44028d740 1 -- 172.21.15.134:0/1055671048 >> 172.21.15.134:0/1055671048 conn(0x55dc4df0b460 msgr2=0x55dc4e2b9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:17.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.566+0000 7fa44028d740 1 -- 172.21.15.134:0/1055671048 shutdown_connections 2024-07-26T11:20:17.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.566+0000 7fa44028d740 1 -- 172.21.15.134:0/1055671048 wait complete. 2024-07-26T11:20:17.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.567+0000 7fa44028d740 1 -- start start 2024-07-26T11:20:17.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.567+0000 7fa44028d740 1 Processor -- start 2024-07-26T11:20:17.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.568+0000 7fa44028d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4deee1b0 0x55dc4e2c7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.568+0000 7fa44028d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4def6390 0x55dc4e2c7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.568+0000 7fa44028d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4df17070 0x55dc4e2c7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.568+0000 7fa44028d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55dc4e085700 con 0x55dc4def6390 2024-07-26T11:20:17.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.568+0000 7fa44028d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dc4e1be6c0 con 0x55dc4deee1b0 2024-07-26T11:20:17.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.568+0000 7fa44028d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55dc4e2ca680 con 0x55dc4df17070 2024-07-26T11:20:17.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.568+0000 7fa44028d740 10 client.0 _ll_register_callbacks cb 0x55dc4e1ce170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:17.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.570+0000 7fa432000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4def6390 0x55dc4e2c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:17.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.570+0000 7fa433400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4df17070 0x55dc4e2c7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:17.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.570+0000 7fa432000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4def6390 0x55dc4e2c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:42228/0 (socket says 172.21.15.134:42228) 2024-07-26T11:20:17.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.571+0000 7fa432000640 1 -- 172.21.15.134:0/2003592373 learned_addr learned my addr 172.21.15.134:0/2003592373 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:17.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.570+0000 7fa432a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4deee1b0 0x55dc4e2c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:17.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.571+0000 7fa432000640 1 -- 172.21.15.134:0/2003592373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4df17070 msgr2=0x55dc4e2c7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:17.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.571+0000 7fa432000640 1 --2- 172.21.15.134:0/2003592373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4df17070 0x55dc4e2c7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.571+0000 7fa432000640 1 -- 172.21.15.134:0/2003592373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4deee1b0 msgr2=0x55dc4e2c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:17.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.571+0000 7fa432000640 1 --2- 172.21.15.134:0/2003592373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4deee1b0 0x55dc4e2c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.571+0000 7fa432000640 1 -- 172.21.15.134:0/2003592373 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc4e1d5860 con 0x55dc4def6390 2024-07-26T11:20:17.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.571+0000 7fa433400640 1 --2- 172.21.15.134:0/2003592373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4df17070 0x55dc4e2c7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:20:17.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.572+0000 7fa432a00640 1 --2- 172.21.15.134:0/2003592373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4deee1b0 0x55dc4e2c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:20:17.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.572+0000 7fa432000640 1 --2- 172.21.15.134:0/2003592373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4def6390 0x55dc4e2c7870 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa4200e0840 tx=0x7fa4200106d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1134b170f033c79 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:17.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.572+0000 7fa413e00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:20:17.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.573+0000 7fa413e00640 1 -- 172.21.15.134:0/2003592373 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4200e0b60 con 0x55dc4def6390 2024-07-26T11:20:17.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.573+0000 7fa413e00640 1 -- 172.21.15.134:0/2003592373 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa420011830 con 0x55dc4def6390 2024-07-26T11:20:17.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.573+0000 7fa44028d740 1 -- 172.21.15.134:0/2003592373 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa42c016380 con 0x55dc4def6390 2024-07-26T11:20:17.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.574+0000 7fa413e00640 1 -- 172.21.15.134:0/2003592373 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa4200a9d30 con 0x55dc4def6390 2024-07-26T11:20:17.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.574+0000 7fa44028d740 10 client.15855 fetch_fsmap learned FSMap version 32 2024-07-26T11:20:17.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.574+0000 7fa44028d740 1 -- 172.21.15.134:0/2003592373 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dc4e2d6260 con 0x55dc4def6390 2024-07-26T11:20:17.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.575+0000 7fa413e00640 1 -- 172.21.15.134:0/2003592373 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa42c016380 con 0x55dc4def6390 2024-07-26T11:20:17.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.575+0000 7fa413e00640 1 -- 172.21.15.134:0/2003592373 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fa4200a56a0 con 0x55dc4def6390 2024-07-26T11:20:17.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.575+0000 7fa44028d740 10 client.15855 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:20:17.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.575+0000 7fa44028d740 10 client.15855 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T11:20:17.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.575+0000 7fa44028d740 20 client.15855 populate_metadata read hostname 'smithi134' 2024-07-26T11:20:17.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.575+0000 7fa44028d740 1 --2- 172.21.15.134:0/2003592373 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55dc4ded59f0 0x55dc4e2d9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.575+0000 7fa44028d740 4 client.15855 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:20:17.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.575+0000 7fa44028d740 1 -- 172.21.15.134:0/2003592373 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dc4e234b40 con 0x55dc4ded59f0 2024-07-26T11:20:17.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.578+0000 7fa432a00640 1 --2- 172.21.15.134:0/2003592373 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55dc4ded59f0 0x55dc4e2d9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:17.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.579+0000 7fa432a00640 1 --2- 172.21.15.134:0/2003592373 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55dc4ded59f0 0x55dc4e2d9670 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d2514e8566b26b50 server_cookie=6bb173c8ef8d09d0 in_seq=0 out_seq=0 2024-07-26T11:20:17.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.579+0000 7fa413e00640 10 client.15855 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:20:17.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.580+0000 7fa413e00640 1 -- 172.21.15.134:0/2003592373 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dc4e234b40 con 0x55dc4ded59f0 2024-07-26T11:20:17.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.580+0000 7fa413e00640 10 client.15855 handle_command_reply: tid=0 2024-07-26T11:20:17.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.580+0000 7fa44028d740 1 client.15855 shutdown 2024-07-26T11:20:17.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.581+0000 7fa44028d740 1 -- 172.21.15.134:0/2003592373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4def6390 msgr2=0x55dc4e2c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:17.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.581+0000 7fa44028d740 1 --2- 172.21.15.134:0/2003592373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4def6390 0x55dc4e2c7870 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa4200e0840 tx=0x7fa4200106d0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.582+0000 7fa44028d740 1 -- 172.21.15.134:0/2003592373 shutdown_connections 2024-07-26T11:20:17.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.582+0000 7fa44028d740 1 --2- 172.21.15.134:0/2003592373 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55dc4ded59f0 0x55dc4e2d9670 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.582+0000 7fa44028d740 1 --2- 172.21.15.134:0/2003592373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4df17070 0x55dc4e2c7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.582+0000 7fa44028d740 1 --2- 172.21.15.134:0/2003592373 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4def6390 0x55dc4e2c7870 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.582+0000 7fa44028d740 1 --2- 172.21.15.134:0/2003592373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4deee1b0 0x55dc4e2c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.582+0000 7fa44028d740 1 -- 172.21.15.134:0/2003592373 >> 172.21.15.134:0/2003592373 conn(0x55dc4df0b460 msgr2=0x55dc4e2bf030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:17.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.582+0000 7fa413e00640 0 client.15855 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:20:17.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.583+0000 7fa44028d740 1 -- 172.21.15.134:0/2003592373 shutdown_connections 2024-07-26T11:20:17.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.583+0000 7fa44028d740 1 -- 172.21.15.134:0/2003592373 wait complete. 2024-07-26T11:20:17.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.584+0000 7fa44028d740 20 client.15855 trim_cache size 0 max 16384 2024-07-26T11:20:17.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.644+0000 7fa44028d740 1 Processor -- start 2024-07-26T11:20:17.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.644+0000 7fa44028d740 1 -- start start 2024-07-26T11:20:17.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.661+0000 7fa44028d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4deee1b0 0x55dc4e2c9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.661+0000 7fa44028d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4def6390 0x55dc4e2b9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.661+0000 7fa44028d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4df17070 0x55dc4e2b9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.662+0000 7fa432a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4deee1b0 0x55dc4e2c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:17.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.662+0000 7fa432a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4deee1b0 0x55dc4e2c9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:57862/0 (socket says 172.21.15.134:57862) 2024-07-26T11:20:17.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.667+0000 7fa432000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4def6390 0x55dc4e2b9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:17.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.668+0000 7fa432000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4def6390 0x55dc4e2b9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57154/0 (socket says 172.21.15.134:57154) 2024-07-26T11:20:17.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.668+0000 7fa44028d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa4341bf8e0 con 0x55dc4df17070 2024-07-26T11:20:17.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.668+0000 7fa44028d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa4341c1420 con 0x55dc4deee1b0 2024-07-26T11:20:17.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.668+0000 7fa44028d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa4341bffb0 con 0x55dc4def6390 2024-07-26T11:20:17.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.669+0000 7fa432a00640 1 -- 172.21.15.134:0/604216485 learned_addr learned my addr 172.21.15.134:0/604216485 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:17.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.672+0000 7fa432a00640 1 -- 172.21.15.134:0/604216485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4def6390 msgr2=0x55dc4e2b9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:17.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.672+0000 7fa432a00640 1 --2- 172.21.15.134:0/604216485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4def6390 0x55dc4e2b9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.672+0000 7fa432a00640 1 -- 172.21.15.134:0/604216485 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4df17070 msgr2=0x55dc4e2b9950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:20:17.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.672+0000 7fa432a00640 1 --2- 172.21.15.134:0/604216485 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4df17070 0x55dc4e2b9950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.674+0000 7fa432a00640 1 -- 172.21.15.134:0/604216485 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dc4e2ba260 con 0x55dc4deee1b0 2024-07-26T11:20:17.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.674+0000 7fa432000640 1 --2- 172.21.15.134:0/604216485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4def6390 0x55dc4e2b9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:20:17.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.675+0000 7fa432a00640 1 --2- 172.21.15.134:0/604216485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4deee1b0 0x55dc4e2c9320 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa42400b230 tx=0x7fa424005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7b359a5e0bbac33 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:17.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.676+0000 7fa41ea00640 1 -- 172.21.15.134:0/604216485 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4240050d0 con 0x55dc4deee1b0 2024-07-26T11:20:17.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.679+0000 7fa41ea00640 1 -- 172.21.15.134:0/604216485 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa42404d070 con 0x55dc4deee1b0 2024-07-26T11:20:17.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.682+0000 7fa41ea00640 1 -- 172.21.15.134:0/604216485 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42403fe60 con 0x55dc4deee1b0 2024-07-26T11:20:17.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.683+0000 7fa44028d740 1 -- 172.21.15.134:0/604216485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4deee1b0 msgr2=0x55dc4e2c9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:17.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.683+0000 7fa44028d740 1 --2- 172.21.15.134:0/604216485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4deee1b0 0x55dc4e2c9320 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa42400b230 tx=0x7fa424005fd0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.687+0000 7fa44028d740 1 -- 172.21.15.134:0/604216485 shutdown_connections 2024-07-26T11:20:17.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.687+0000 7fa44028d740 1 --2- 172.21.15.134:0/604216485 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4def6390 0x55dc4e2b9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.687+0000 7fa44028d740 1 --2- 172.21.15.134:0/604216485 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4df17070 0x55dc4e2b9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.687+0000 7fa44028d740 1 --2- 172.21.15.134:0/604216485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4deee1b0 0x55dc4e2c9320 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.688+0000 7fa44028d740 1 -- 172.21.15.134:0/604216485 >> 172.21.15.134:0/604216485 conn(0x55dc4df0b460 msgr2=0x55dc4e2bd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:17.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.689+0000 7fa44028d740 1 -- 172.21.15.134:0/604216485 shutdown_connections 2024-07-26T11:20:17.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.689+0000 7fa44028d740 1 -- 172.21.15.134:0/604216485 wait complete. 2024-07-26T11:20:17.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.690+0000 7fa44028d740 1 -- start start 2024-07-26T11:20:17.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.690+0000 7fa44028d740 1 Processor -- start 2024-07-26T11:20:17.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.692+0000 7fa44028d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4deee1b0 0x55dc4e2cb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.692+0000 7fa44028d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4def6390 0x55dc4e2cb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.692+0000 7fa44028d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4df17070 0x55dc4e2cf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.692+0000 7fa44028d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa4341c0d50 con 0x55dc4deee1b0 2024-07-26T11:20:17.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.692+0000 7fa44028d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa4341c0680 con 0x55dc4df17070 2024-07-26T11:20:17.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.692+0000 7fa44028d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa4341bd8c0 con 0x55dc4def6390 2024-07-26T11:20:17.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.692+0000 7fa44028d740 10 client.0 _ll_register_callbacks cb 0x7fa4200a9b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:17.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.692+0000 7fa432000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4def6390 0x55dc4e2cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:17.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.692+0000 7fa432000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4def6390 0x55dc4e2cb730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57170/0 (socket says 172.21.15.134:57170) 2024-07-26T11:20:17.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.692+0000 7fa432000640 1 -- 172.21.15.134:0/146823765 learned_addr learned my addr 172.21.15.134:0/146823765 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:17.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.693+0000 7fa432000640 1 -- 172.21.15.134:0/146823765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4df17070 msgr2=0x55dc4e2cf620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:20:17.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.693+0000 7fa432000640 1 --2- 172.21.15.134:0/146823765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4df17070 0x55dc4e2cf620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.693+0000 7fa432000640 1 -- 172.21.15.134:0/146823765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4deee1b0 msgr2=0x55dc4e2cb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:17.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.693+0000 7fa432000640 1 --2- 172.21.15.134:0/146823765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4deee1b0 0x55dc4e2cb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.693+0000 7fa432000640 1 -- 172.21.15.134:0/146823765 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dc4e2cfc90 con 0x55dc4def6390 2024-07-26T11:20:17.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.697+0000 7fa432000640 1 --2- 172.21.15.134:0/146823765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4def6390 0x55dc4e2cb730 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa4200a5a10 tx=0x7fa420010870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77c0ff86e08db51b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:17.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.697+0000 7fa413e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:20:17.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.698+0000 7fa413e00640 1 -- 172.21.15.134:0/146823765 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4200e0b60 con 0x55dc4def6390 2024-07-26T11:20:17.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.698+0000 7fa413e00640 1 -- 172.21.15.134:0/146823765 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa4200a6320 con 0x55dc4def6390 2024-07-26T11:20:17.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.699+0000 7fa44028d740 1 -- 172.21.15.134:0/146823765 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa42404d070 con 0x55dc4def6390 2024-07-26T11:20:17.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.699+0000 7fa413e00640 1 -- 172.21.15.134:0/146823765 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa4200ae330 con 0x55dc4def6390 2024-07-26T11:20:17.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.700+0000 7fa44028d740 10 client.26206 fetch_fsmap learned FSMap version 32 2024-07-26T11:20:17.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.700+0000 7fa44028d740 1 -- 172.21.15.134:0/146823765 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55dc4e3527f0 con 0x55dc4def6390 2024-07-26T11:20:17.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.700+0000 7fa413e00640 1 -- 172.21.15.134:0/146823765 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa42404d070 con 0x55dc4def6390 2024-07-26T11:20:17.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.703+0000 7fa413e00640 1 -- 172.21.15.134:0/146823765 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fa4200a5040 con 0x55dc4def6390 2024-07-26T11:20:17.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.703+0000 7fa44028d740 10 client.26206 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:20:17.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.703+0000 7fa44028d740 10 client.26206 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T11:20:17.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.703+0000 7fa44028d740 20 client.26206 populate_metadata read hostname 'smithi134' 2024-07-26T11:20:17.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.703+0000 7fa44028d740 1 --2- 172.21.15.134:0/146823765 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55dc4deaada0 0x55dc4e354e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:17.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.703+0000 7fa44028d740 4 client.26206 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:20:17.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.703+0000 7fa44028d740 1 -- 172.21.15.134:0/146823765 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dc4e2305e0 con 0x55dc4deaada0 2024-07-26T11:20:17.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.705+0000 7fa432a00640 1 --2- 172.21.15.134:0/146823765 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55dc4deaada0 0x55dc4e354e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:17.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.707+0000 7fa432a00640 1 --2- 172.21.15.134:0/146823765 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55dc4deaada0 0x55dc4e354e60 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9c08b0840cd8fdc0 server_cookie=eb10017223618434 in_seq=0 out_seq=0 2024-07-26T11:20:17.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.707+0000 7fa413e00640 10 client.26206 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:20:17.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.713+0000 7fa413e00640 1 -- 172.21.15.134:0/146823765 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dc4e2305e0 con 0x55dc4deaada0 2024-07-26T11:20:17.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.713+0000 7fa413e00640 10 client.26206 handle_command_reply: tid=0 2024-07-26T11:20:17.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.713+0000 7fa44028d740 1 client.26206 shutdown 2024-07-26T11:20:17.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.721+0000 7fa44028d740 1 -- 172.21.15.134:0/146823765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4def6390 msgr2=0x55dc4e2cb730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:17.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.721+0000 7fa44028d740 1 --2- 172.21.15.134:0/146823765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4def6390 0x55dc4e2cb730 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa4200a5a10 tx=0x7fa420010870 comp rx=0 tx=0).stop 2024-07-26T11:20:17.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.722+0000 7fa44028d740 1 -- 172.21.15.134:0/146823765 shutdown_connections 2024-07-26T11:20:17.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.722+0000 7fa44028d740 1 --2- 172.21.15.134:0/146823765 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55dc4deaada0 0x55dc4e354e60 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.722+0000 7fa44028d740 1 --2- 172.21.15.134:0/146823765 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dc4def6390 0x55dc4e2cb730 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.722+0000 7fa44028d740 1 --2- 172.21.15.134:0/146823765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dc4df17070 0x55dc4e2cf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.722+0000 7fa44028d740 1 --2- 172.21.15.134:0/146823765 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dc4deee1b0 0x55dc4e2cb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.722+0000 7fa44028d740 1 -- 172.21.15.134:0/146823765 >> 172.21.15.134:0/146823765 conn(0x55dc4df0b460 msgr2=0x55dc4e2c26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:17.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.726+0000 7fa413e00640 0 client.26206 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:20:17.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.732+0000 7fa44028d740 1 -- 172.21.15.134:0/146823765 shutdown_connections 2024-07-26T11:20:17.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.732+0000 7fa44028d740 1 -- 172.21.15.134:0/146823765 wait complete. 2024-07-26T11:20:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.733+0000 7fa44028d740 20 client.26206 trim_cache size 0 max 16384 2024-07-26T11:20:17.735 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:20:17.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.734+0000 7fa439600640 1 -- 172.21.15.134:0/4228418887 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa4180b40b0 msgr2=0x7fa4180b65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:17.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.734+0000 7fa439600640 1 --2- 172.21.15.134:0/4228418887 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa4180b40b0 0x7fa4180b65c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa42c00bc90 tx=0x7fa42c00d7e0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.734+0000 7fa439600640 1 -- 172.21.15.134:0/4228418887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa434111970 msgr2=0x7fa4341a9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:17.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.734+0000 7fa439600640 1 --2- 172.21.15.134:0/4228418887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa434111970 0x7fa4341a9270 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa43406c110 tx=0x7fa42000b850 comp rx=0 tx=0).stop 2024-07-26T11:20:17.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.735+0000 7fa439600640 1 -- 172.21.15.134:0/4228418887 shutdown_connections 2024-07-26T11:20:17.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.735+0000 7fa439600640 1 --2- 172.21.15.134:0/4228418887 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa4180b40b0 0x7fa4180b65c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.735+0000 7fa439600640 1 --2- 172.21.15.134:0/4228418887 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa434111970 0x7fa4341a9270 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.735+0000 7fa439600640 1 --2- 172.21.15.134:0/4228418887 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa43411c660 0x7fa4341ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.735+0000 7fa439600640 1 --2- 172.21.15.134:0/4228418887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa43410a410 0x7fa4341a8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:17.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.735+0000 7fa439600640 1 -- 172.21.15.134:0/4228418887 >> 172.21.15.134:0/4228418887 conn(0x7fa4340708f0 msgr2=0x7fa43411dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:17.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.737+0000 7fa439600640 1 -- 172.21.15.134:0/4228418887 shutdown_connections 2024-07-26T11:20:17.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:17.737+0000 7fa439600640 1 -- 172.21.15.134:0/4228418887 wait complete. 2024-07-26T11:20:18.641 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.638+0000 7f6338800640 1 -- 172.21.15.134:0/2524106412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6330114190 msgr2=0x7f633011a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:18.641 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.638+0000 7f6338800640 1 --2- 172.21.15.134:0/2524106412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6330114190 0x7f633011a9c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6320009900 tx=0x7f632002a410 comp rx=0 tx=0).stop 2024-07-26T11:20:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.640+0000 7f6338800640 1 -- 172.21.15.134:0/2524106412 shutdown_connections 2024-07-26T11:20:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.640+0000 7f6338800640 1 --2- 172.21.15.134:0/2524106412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6330114190 0x7f633011a9c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.640+0000 7f6338800640 1 --2- 172.21.15.134:0/2524106412 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6330113840 0x7f6330113c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.640+0000 7f6338800640 1 --2- 172.21.15.134:0/2524106412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f633010df10 0x7f633010e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.640+0000 7f6338800640 1 -- 172.21.15.134:0/2524106412 >> 172.21.15.134:0/2524106412 conn(0x7f6330109df0 msgr2=0x7f633010c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:18.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.641+0000 7f6338800640 1 -- 172.21.15.134:0/2524106412 shutdown_connections 2024-07-26T11:20:18.644 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.641+0000 7f6338800640 1 -- 172.21.15.134:0/2524106412 wait complete. 2024-07-26T11:20:18.645 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.643+0000 7f6338800640 1 Processor -- start 2024-07-26T11:20:18.648 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.643+0000 7f6338800640 1 -- start start 2024-07-26T11:20:18.648 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.644+0000 7f6338800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f633010df10 0x7f63301a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.648 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.644+0000 7f6338800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6330113840 0x7f63301a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.648 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.644+0000 7f6338800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6330114190 0x7f63301aa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.648 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.644+0000 7f6338800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f633011deb0 con 0x7f6330114190 2024-07-26T11:20:18.648 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.644+0000 7f6338800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f633011dd30 con 0x7f6330113840 2024-07-26T11:20:18.648 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.644+0000 7f6338800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f633011e030 con 0x7f633010df10 2024-07-26T11:20:18.649 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.645+0000 7f6335400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6330113840 0x7f63301a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:18.649 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.645+0000 7f6335400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6330113840 0x7f63301a4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:57872/0 (socket says 172.21.15.134:57872) 2024-07-26T11:20:18.649 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.645+0000 7f6335400640 1 -- 172.21.15.134:0/772513122 learned_addr learned my addr 172.21.15.134:0/772513122 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:18.649 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.646+0000 7f6336800640 1 --2- 172.21.15.134:0/772513122 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6330114190 0x7f63301aa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:18.650 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.648+0000 7f6335e00640 1 --2- 172.21.15.134:0/772513122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f633010df10 0x7f63301a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:18.651 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.648+0000 7f6335400640 1 -- 172.21.15.134:0/772513122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f633010df10 msgr2=0x7f63301a48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:18.651 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.648+0000 7f6335400640 1 --2- 172.21.15.134:0/772513122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f633010df10 0x7f63301a48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.651 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.648+0000 7f6335400640 1 -- 172.21.15.134:0/772513122 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6330114190 msgr2=0x7f63301aa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:18.651 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.648+0000 7f6335400640 1 --2- 172.21.15.134:0/772513122 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6330114190 0x7f63301aa0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.648+0000 7f6335400640 1 -- 172.21.15.134:0/772513122 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63301aa7e0 con 0x7f6330113840 2024-07-26T11:20:18.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.649+0000 7f6335400640 1 --2- 172.21.15.134:0/772513122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6330113840 0x7f63301a4e50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6324006fd0 tx=0x7f632400b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:18.654 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.652+0000 7f632aa00640 1 -- 172.21.15.134:0/772513122 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6324011830 con 0x7f6330113840 2024-07-26T11:20:18.654 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.652+0000 7f632aa00640 1 -- 172.21.15.134:0/772513122 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6324016070 con 0x7f6330113840 2024-07-26T11:20:18.654 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.652+0000 7f632aa00640 1 -- 172.21.15.134:0/772513122 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6324010e00 con 0x7f6330113840 2024-07-26T11:20:18.654 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.652+0000 7f6338800640 1 -- 172.21.15.134:0/772513122 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63301aaad0 con 0x7f6330113840 2024-07-26T11:20:18.654 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.652+0000 7f6338800640 1 -- 172.21.15.134:0/772513122 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63301b1260 con 0x7f6330113840 2024-07-26T11:20:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.667+0000 7f633f96a740 1 Processor -- start 2024-07-26T11:20:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.668+0000 7f632aa00640 1 -- 172.21.15.134:0/772513122 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f632401b460 con 0x7f6330113840 2024-07-26T11:20:18.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.668+0000 7f633f96a740 1 -- start start 2024-07-26T11:20:18.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.668+0000 7f633f96a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b831b0 0x55e787f26a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.668+0000 7f633f96a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b5a2f0 0x55e787f26f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.668+0000 7f633f96a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b624d0 0x55e787f274c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.668+0000 7f633f96a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e787e9b270 con 0x55e787b831b0 2024-07-26T11:20:18.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.668+0000 7f633f96a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e787e8a770 con 0x55e787b624d0 2024-07-26T11:20:18.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.668+0000 7f633f96a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e787e8e7b0 con 0x55e787b5a2f0 2024-07-26T11:20:18.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.669+0000 7f6335400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b624d0 0x55e787f274c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:18.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.669+0000 7f6335400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b624d0 0x55e787f274c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:57882/0 (socket says 172.21.15.134:57882) 2024-07-26T11:20:18.674 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.669+0000 7f6335400640 1 -- 172.21.15.134:0/1704302822 learned_addr learned my addr 172.21.15.134:0/1704302822 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:18.674 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.670+0000 7f6335400640 1 -- 172.21.15.134:0/1704302822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b5a2f0 msgr2=0x55e787f26f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:20:18.674 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.670+0000 7f6335400640 1 --2- 172.21.15.134:0/1704302822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b5a2f0 0x55e787f26f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.674 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.670+0000 7f6335400640 1 -- 172.21.15.134:0/1704302822 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b831b0 msgr2=0x55e787f26a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:20:18.674 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.670+0000 7f6335400640 1 --2- 172.21.15.134:0/1704302822 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b831b0 0x55e787f26a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.674 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.670+0000 7f6335400640 1 -- 172.21.15.134:0/1704302822 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e787bee790 con 0x55e787b624d0 2024-07-26T11:20:18.674 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.671+0000 7f632aa00640 1 --2- 172.21.15.134:0/772513122 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63180b3e90 0x7f63180b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.671+0000 7f632aa00640 1 -- 172.21.15.134:0/772513122 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f63240e37e0 con 0x7f6330113840 2024-07-26T11:20:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.671+0000 7f6335400640 1 --2- 172.21.15.134:0/1704302822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b624d0 0x55e787f274c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f63240e0c00 tx=0x7f63240e9f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=860b709c99f0f7c7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.672+0000 7f631f000640 1 -- 172.21.15.134:0/1704302822 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63240a6040 con 0x55e787b624d0 2024-07-26T11:20:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.672+0000 7f631f000640 1 -- 172.21.15.134:0/1704302822 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63240a9c90 con 0x55e787b624d0 2024-07-26T11:20:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.672+0000 7f631f000640 1 -- 172.21.15.134:0/1704302822 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63240a98f0 con 0x55e787b624d0 2024-07-26T11:20:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.672+0000 7f6336800640 1 --2- 172.21.15.134:0/772513122 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63180b3e90 0x7f63180b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.674+0000 7f633f96a740 1 -- 172.21.15.134:0/1704302822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b624d0 msgr2=0x55e787f274c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.674+0000 7f633f96a740 1 --2- 172.21.15.134:0/1704302822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b624d0 0x55e787f274c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f63240e0c00 tx=0x7f63240e9f10 comp rx=0 tx=0).stop 2024-07-26T11:20:18.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.676+0000 7f6336800640 1 --2- 172.21.15.134:0/772513122 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63180b3e90 0x7f63180b63a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6320005ab0 tx=0x7f6320005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.680+0000 7f633f96a740 1 -- 172.21.15.134:0/1704302822 shutdown_connections 2024-07-26T11:20:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.680+0000 7f633f96a740 1 --2- 172.21.15.134:0/1704302822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b5a2f0 0x55e787f26f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.680+0000 7f633f96a740 1 --2- 172.21.15.134:0/1704302822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b624d0 0x55e787f274c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.682 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.680+0000 7f633f96a740 1 --2- 172.21.15.134:0/1704302822 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b831b0 0x55e787f26a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.683 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.680+0000 7f633f96a740 1 -- 172.21.15.134:0/1704302822 >> 172.21.15.134:0/1704302822 conn(0x55e787b77300 msgr2=0x55e787f24930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:18.683 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.681+0000 7f633f96a740 1 -- 172.21.15.134:0/1704302822 shutdown_connections 2024-07-26T11:20:18.686 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.681+0000 7f633f96a740 1 -- 172.21.15.134:0/1704302822 wait complete. 2024-07-26T11:20:18.686 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.685+0000 7f633f96a740 1 -- start start 2024-07-26T11:20:18.687 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.685+0000 7f633f96a740 1 Processor -- start 2024-07-26T11:20:18.689 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.687+0000 7f633f96a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b5a2f0 0x55e787f3b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.689 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.687+0000 7f633f96a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b624d0 0x55e787f3bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.689 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.687+0000 7f6335e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b5a2f0 0x55e787f3b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:18.689 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.688+0000 7f633f96a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 0x55e787f40af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.690 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.688+0000 7f6335e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b5a2f0 0x55e787f3b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:42280/0 (socket says 172.21.15.134:42280) 2024-07-26T11:20:18.690 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.688+0000 7f6335e00640 1 -- 172.21.15.134:0/766093010 learned_addr learned my addr 172.21.15.134:0/766093010 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:18.690 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.688+0000 7f633f96a740 1 -- 172.21.15.134:0/766093010 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e787dc4ab0 con 0x55e787b5a2f0 2024-07-26T11:20:18.690 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.688+0000 7f633f96a740 1 -- 172.21.15.134:0/766093010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e787e27d50 con 0x55e787b831b0 2024-07-26T11:20:18.690 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.688+0000 7f633f96a740 1 -- 172.21.15.134:0/766093010 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e787f351c0 con 0x55e787b624d0 2024-07-26T11:20:18.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.688+0000 7f633f96a740 10 client.0 _ll_register_callbacks cb 0x55e787b9aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:18.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.689+0000 7f6335400640 1 --2- 172.21.15.134:0/766093010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b624d0 0x55e787f3bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:18.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.689+0000 7f6336800640 1 --2- 172.21.15.134:0/766093010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 0x55e787f40af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:18.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.689+0000 7f6335400640 1 -- 172.21.15.134:0/766093010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 msgr2=0x55e787f40af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:18.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.689+0000 7f6335400640 1 --2- 172.21.15.134:0/766093010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 0x55e787f40af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.689+0000 7f6335400640 1 -- 172.21.15.134:0/766093010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b5a2f0 msgr2=0x55e787f3b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:18.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.689+0000 7f6335400640 1 --2- 172.21.15.134:0/766093010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b5a2f0 0x55e787f3b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.689+0000 7f6335400640 1 -- 172.21.15.134:0/766093010 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e787c67620 con 0x55e787b624d0 2024-07-26T11:20:18.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.690+0000 7f6335400640 1 --2- 172.21.15.134:0/766093010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b624d0 0x55e787f3bae0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f63240e92f0 tx=0x7f6324010380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c06ee9546178c428 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:18.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.690+0000 7f6313400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:20:18.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:18 smithi189 ceph-mon[36790]: pgmap v610: 129 pgs: 129 active+clean; 27 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 21 KiB/s rd, 150 MiB/s wr, 1.03k op/s 2024-07-26T11:20:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.691+0000 7f6313400640 1 -- 172.21.15.134:0/766093010 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63240a9ad0 con 0x55e787b624d0 2024-07-26T11:20:18.695 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.692+0000 7f6313400640 1 -- 172.21.15.134:0/766093010 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f63240e3400 con 0x55e787b624d0 2024-07-26T11:20:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.694+0000 7f633f96a740 1 -- 172.21.15.134:0/766093010 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63240a9c90 con 0x55e787b624d0 2024-07-26T11:20:18.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.695+0000 7f6313400640 1 -- 172.21.15.134:0/766093010 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f63240a9f50 con 0x55e787b624d0 2024-07-26T11:20:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.696+0000 7f633f96a740 10 client.26215 fetch_fsmap learned FSMap version 32 2024-07-26T11:20:18.698 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.696+0000 7f633f96a740 1 -- 172.21.15.134:0/766093010 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e787f49a40 con 0x55e787b624d0 2024-07-26T11:20:18.700 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.697+0000 7f6313400640 1 -- 172.21.15.134:0/766093010 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63240a9c90 con 0x55e787b624d0 2024-07-26T11:20:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.697+0000 7f6313400640 1 -- 172.21.15.134:0/766093010 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f63240aa3e0 con 0x55e787b624d0 2024-07-26T11:20:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.697+0000 7f633f96a740 10 client.26215 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:20:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.697+0000 7f633f96a740 10 client.26215 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:20:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.697+0000 7f633f96a740 20 client.26215 populate_metadata read hostname 'smithi134' 2024-07-26T11:20:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.697+0000 7f633f96a740 1 --2- 172.21.15.134:0/766093010 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e787b41d80 0x55e787f4cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.697+0000 7f633f96a740 4 client.26215 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:20:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.697+0000 7f633f96a740 1 -- 172.21.15.134:0/766093010 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e787e9d1c0 con 0x55e787b41d80 2024-07-26T11:20:18.701 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.700+0000 7f6335e00640 1 --2- 172.21.15.134:0/766093010 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e787b41d80 0x55e787f4cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:18.706 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.704+0000 7f6313400640 10 client.26215 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:20:18.706 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.704+0000 7f6335e00640 1 --2- 172.21.15.134:0/766093010 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e787b41d80 0x55e787f4cdf0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c468033b7fbbbba7 server_cookie=6260e72b864fc23b in_seq=0 out_seq=0 2024-07-26T11:20:18.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.706+0000 7f6313400640 1 -- 172.21.15.134:0/766093010 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e787e9d1c0 con 0x55e787b41d80 2024-07-26T11:20:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.707+0000 7f6313400640 10 client.26215 handle_command_reply: tid=0 2024-07-26T11:20:18.709 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.707+0000 7f633f96a740 1 client.26215 shutdown 2024-07-26T11:20:18.710 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.708+0000 7f633f96a740 1 -- 172.21.15.134:0/766093010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b624d0 msgr2=0x55e787f3bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:18.710 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.708+0000 7f633f96a740 1 --2- 172.21.15.134:0/766093010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b624d0 0x55e787f3bae0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f63240e92f0 tx=0x7f6324010380 comp rx=0 tx=0).stop 2024-07-26T11:20:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.714+0000 7f633f96a740 1 -- 172.21.15.134:0/766093010 shutdown_connections 2024-07-26T11:20:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.714+0000 7f633f96a740 1 --2- 172.21.15.134:0/766093010 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e787b41d80 0x55e787f4cdf0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.715+0000 7f633f96a740 1 --2- 172.21.15.134:0/766093010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b624d0 0x55e787f3bae0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.715+0000 7f633f96a740 1 --2- 172.21.15.134:0/766093010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 0x55e787f40af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.715+0000 7f633f96a740 1 --2- 172.21.15.134:0/766093010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b5a2f0 0x55e787f3b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.715+0000 7f633f96a740 1 -- 172.21.15.134:0/766093010 >> 172.21.15.134:0/766093010 conn(0x55e787b77300 msgr2=0x55e787f230b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.715+0000 7f6313400640 0 client.26215 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:20:18.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.715+0000 7f633f96a740 1 -- 172.21.15.134:0/766093010 shutdown_connections 2024-07-26T11:20:18.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.716+0000 7f633f96a740 1 -- 172.21.15.134:0/766093010 wait complete. 2024-07-26T11:20:18.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.717+0000 7f633f96a740 20 client.26215 trim_cache size 0 max 16384 2024-07-26T11:20:18.770 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.768+0000 7f633f96a740 1 Processor -- start 2024-07-26T11:20:18.771 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.769+0000 7f633f96a740 1 -- start start 2024-07-26T11:20:18.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.770+0000 7f633f96a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b5a2f0 0x55e787f35ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.770+0000 7f633f96a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b624d0 0x55e787f3a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.770+0000 7f633f96a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 0x55e787f3ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.770+0000 7f633f96a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f63301b7080 con 0x55e787b624d0 2024-07-26T11:20:18.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.770+0000 7f633f96a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f63301b8bc0 con 0x55e787b831b0 2024-07-26T11:20:18.773 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.770+0000 7f633f96a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f63301b7750 con 0x55e787b5a2f0 2024-07-26T11:20:18.773 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.770+0000 7f6336800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 0x55e787f3ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:18.773 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.770+0000 7f6336800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 0x55e787f3ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:57898/0 (socket says 172.21.15.134:57898) 2024-07-26T11:20:18.773 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.770+0000 7f6336800640 1 -- 172.21.15.134:0/2354414302 learned_addr learned my addr 172.21.15.134:0/2354414302 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:18.773 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.771+0000 7f6336800640 1 -- 172.21.15.134:0/2354414302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b5a2f0 msgr2=0x55e787f35ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:20:18.773 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.771+0000 7f6336800640 1 --2- 172.21.15.134:0/2354414302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b5a2f0 0x55e787f35ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.774 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.771+0000 7f6336800640 1 -- 172.21.15.134:0/2354414302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b624d0 msgr2=0x55e787f3a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:18.774 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.771+0000 7f6336800640 1 --2- 172.21.15.134:0/2354414302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b624d0 0x55e787f3a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.774 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.771+0000 7f6336800640 1 -- 172.21.15.134:0/2354414302 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e787f36ad0 con 0x55e787b831b0 2024-07-26T11:20:18.774 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.771+0000 7f6336800640 1 --2- 172.21.15.134:0/2354414302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 0x55e787f3ad00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6320009900 tx=0x7f632003c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8745bf77cb25d557 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:18.774 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.772+0000 7f631e600640 1 -- 172.21.15.134:0/2354414302 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632004d070 con 0x55e787b831b0 2024-07-26T11:20:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:18 smithi193 ceph-mon[37473]: pgmap v610: 129 pgs: 129 active+clean; 27 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 21 KiB/s rd, 150 MiB/s wr, 1.03k op/s 2024-07-26T11:20:18.775 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.772+0000 7f631e600640 1 -- 172.21.15.134:0/2354414302 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63200054f0 con 0x55e787b831b0 2024-07-26T11:20:18.775 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.773+0000 7f631e600640 1 -- 172.21.15.134:0/2354414302 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6320035050 con 0x55e787b831b0 2024-07-26T11:20:18.776 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.775+0000 7f633f96a740 1 -- 172.21.15.134:0/2354414302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 msgr2=0x55e787f3ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:18.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.775+0000 7f633f96a740 1 --2- 172.21.15.134:0/2354414302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 0x55e787f3ad00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6320009900 tx=0x7f632003c000 comp rx=0 tx=0).stop 2024-07-26T11:20:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.779+0000 7f633f96a740 1 -- 172.21.15.134:0/2354414302 shutdown_connections 2024-07-26T11:20:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.779+0000 7f633f96a740 1 --2- 172.21.15.134:0/2354414302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 0x55e787f3ad00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.779+0000 7f633f96a740 1 --2- 172.21.15.134:0/2354414302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b624d0 0x55e787f3a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.779+0000 7f633f96a740 1 --2- 172.21.15.134:0/2354414302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b5a2f0 0x55e787f35ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.779+0000 7f633f96a740 1 -- 172.21.15.134:0/2354414302 >> 172.21.15.134:0/2354414302 conn(0x55e787b77300 msgr2=0x55e787f353d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:18.787 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.785+0000 7f633f96a740 1 -- 172.21.15.134:0/2354414302 shutdown_connections 2024-07-26T11:20:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.786+0000 7f633f96a740 1 -- 172.21.15.134:0/2354414302 wait complete. 2024-07-26T11:20:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.787+0000 7f633f96a740 1 -- start start 2024-07-26T11:20:18.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.787+0000 7f633f96a740 1 Processor -- start 2024-07-26T11:20:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.788+0000 7f633f96a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b5a2f0 0x55e787f52580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.788+0000 7f633f96a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b624d0 0x55e787f52ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.790 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.788+0000 7f633f96a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 0x55e787f3f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.791 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.788+0000 7f633f96a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f63301b84f0 con 0x55e787b5a2f0 2024-07-26T11:20:18.791 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.788+0000 7f633f96a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f63301b7e20 con 0x55e787b831b0 2024-07-26T11:20:18.796 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.788+0000 7f633f96a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f63301b5080 con 0x55e787b624d0 2024-07-26T11:20:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.789+0000 7f6336800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 0x55e787f3f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.789+0000 7f6335400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b624d0 0x55e787f52ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.794+0000 7f6336800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 0x55e787f3f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:57902/0 (socket says 172.21.15.134:57902) 2024-07-26T11:20:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.794+0000 7f6335400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b624d0 0x55e787f52ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57222/0 (socket says 172.21.15.134:57222) 2024-07-26T11:20:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.794+0000 7f6336800640 1 -- 172.21.15.134:0/1159213985 learned_addr learned my addr 172.21.15.134:0/1159213985 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.790+0000 7f6335e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b5a2f0 0x55e787f52580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.794+0000 7f633f96a740 10 client.0 _ll_register_callbacks cb 0x7f63240e9900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.795+0000 7f6336800640 1 -- 172.21.15.134:0/1159213985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b624d0 msgr2=0x55e787f52ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:18.797 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.795+0000 7f6336800640 1 --2- 172.21.15.134:0/1159213985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b624d0 0x55e787f52ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.796+0000 7f6336800640 1 -- 172.21.15.134:0/1159213985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b5a2f0 msgr2=0x55e787f52580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.797+0000 7f6336800640 1 --2- 172.21.15.134:0/1159213985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b5a2f0 0x55e787f52580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:18.799 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.797+0000 7f6336800640 1 -- 172.21.15.134:0/1159213985 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e787f53100 con 0x55e787b831b0 2024-07-26T11:20:18.800 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.798+0000 7f6336800640 1 --2- 172.21.15.134:0/1159213985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 0x55e787f3f950 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f63200092a0 tx=0x7f63200044b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d34b1e86847d381 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:18.801 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.798+0000 7f6313400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:20:18.801 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.799+0000 7f6313400640 1 -- 172.21.15.134:0/1159213985 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632005a030 con 0x55e787b831b0 2024-07-26T11:20:18.802 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.800+0000 7f6313400640 1 -- 172.21.15.134:0/1159213985 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f63200035f0 con 0x55e787b831b0 2024-07-26T11:20:18.802 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.800+0000 7f633f96a740 1 -- 172.21.15.134:0/1159213985 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63200054f0 con 0x55e787b831b0 2024-07-26T11:20:18.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.801+0000 7f6313400640 1 -- 172.21.15.134:0/1159213985 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6320067050 con 0x55e787b831b0 2024-07-26T11:20:18.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.801+0000 7f633f96a740 10 client.26228 fetch_fsmap learned FSMap version 32 2024-07-26T11:20:18.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.804+0000 7f633f96a740 1 -- 172.21.15.134:0/1159213985 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e787f47050 con 0x55e787b831b0 2024-07-26T11:20:18.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.804+0000 7f6313400640 1 -- 172.21.15.134:0/1159213985 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63200054f0 con 0x55e787b831b0 2024-07-26T11:20:18.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.805+0000 7f6313400640 1 -- 172.21.15.134:0/1159213985 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f6320049660 con 0x55e787b831b0 2024-07-26T11:20:18.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.805+0000 7f633f96a740 10 client.26228 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:20:18.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.805+0000 7f633f96a740 10 client.26228 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:20:18.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.805+0000 7f633f96a740 20 client.26228 populate_metadata read hostname 'smithi134' 2024-07-26T11:20:18.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.805+0000 7f633f96a740 1 --2- 172.21.15.134:0/1159213985 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e787b171c0 0x55e787f477e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:18.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.805+0000 7f633f96a740 4 client.26228 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:20:18.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.806+0000 7f633f96a740 1 -- 172.21.15.134:0/1159213985 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e787e98c60 con 0x55e787b171c0 2024-07-26T11:20:18.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.809+0000 7f6335e00640 1 --2- 172.21.15.134:0/1159213985 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e787b171c0 0x55e787f477e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:18.815 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.813+0000 7f6313400640 10 client.26228 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:20:18.815 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:18.813+0000 7f6335e00640 1 --2- 172.21.15.134:0/1159213985 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e787b171c0 0x55e787f477e0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75dd7007e2583b75 server_cookie=de8563319454b732 in_seq=0 out_seq=0 2024-07-26T11:20:18.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:18 smithi134 ceph-mon[31966]: pgmap v610: 129 pgs: 129 active+clean; 27 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 21 KiB/s rd, 150 MiB/s wr, 1.03k op/s 2024-07-26T11:20:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:20 smithi134 ceph-mon[31966]: pgmap v611: 129 pgs: 129 active+clean; 27 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 25 KiB/s rd, 204 MiB/s wr, 1.63k op/s 2024-07-26T11:20:20.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:20 smithi189 ceph-mon[36790]: pgmap v611: 129 pgs: 129 active+clean; 27 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 25 KiB/s rd, 204 MiB/s wr, 1.63k op/s 2024-07-26T11:20:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:20 smithi193 ceph-mon[37473]: pgmap v611: 129 pgs: 129 active+clean; 27 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 25 KiB/s rd, 204 MiB/s wr, 1.63k op/s 2024-07-26T11:20:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.774+0000 7f6313400640 1 -- 172.21.15.134:0/1159213985 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e787e98c60 con 0x55e787b171c0 2024-07-26T11:20:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.774+0000 7f6313400640 10 client.26228 handle_command_reply: tid=0 2024-07-26T11:20:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.774+0000 7f633f96a740 1 client.26228 shutdown 2024-07-26T11:20:20.776 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.775+0000 7f633f96a740 1 -- 172.21.15.134:0/1159213985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 msgr2=0x55e787f3f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:20.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.775+0000 7f633f96a740 1 --2- 172.21.15.134:0/1159213985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 0x55e787f3f950 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f63200092a0 tx=0x7f63200044b0 comp rx=0 tx=0).stop 2024-07-26T11:20:20.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.775+0000 7f633f96a740 1 -- 172.21.15.134:0/1159213985 shutdown_connections 2024-07-26T11:20:20.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.775+0000 7f633f96a740 1 --2- 172.21.15.134:0/1159213985 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e787b171c0 0x55e787f477e0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:20.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.775+0000 7f633f96a740 1 --2- 172.21.15.134:0/1159213985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e787b624d0 0x55e787f52ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:20.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.775+0000 7f633f96a740 1 --2- 172.21.15.134:0/1159213985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e787b831b0 0x55e787f3f950 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:20.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.775+0000 7f633f96a740 1 --2- 172.21.15.134:0/1159213985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e787b5a2f0 0x55e787f52580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:20.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.775+0000 7f633f96a740 1 -- 172.21.15.134:0/1159213985 >> 172.21.15.134:0/1159213985 conn(0x55e787b77300 msgr2=0x55e787f29b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.775+0000 7f6313400640 0 client.26228 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:20:20.778 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.775+0000 7f633f96a740 1 -- 172.21.15.134:0/1159213985 shutdown_connections 2024-07-26T11:20:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.776+0000 7f633f96a740 1 -- 172.21.15.134:0/1159213985 wait complete. 2024-07-26T11:20:20.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.777+0000 7f633f96a740 20 client.26228 trim_cache size 0 max 16384 2024-07-26T11:20:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:20:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:20:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:20:20.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:20:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.780+0000 7f6338800640 1 -- 172.21.15.134:0/772513122 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63180b3e90 msgr2=0x7f63180b63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.780+0000 7f6338800640 1 --2- 172.21.15.134:0/772513122 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63180b3e90 0x7f63180b63a0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6320005ab0 tx=0x7f6320005a20 comp rx=0 tx=0).stop 2024-07-26T11:20:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.780+0000 7f6338800640 1 -- 172.21.15.134:0/772513122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6330113840 msgr2=0x7f63301a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:20.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.780+0000 7f6338800640 1 --2- 172.21.15.134:0/772513122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6330113840 0x7f63301a4e50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6324006fd0 tx=0x7f632400b8e0 comp rx=0 tx=0).stop 2024-07-26T11:20:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.782+0000 7f6338800640 1 -- 172.21.15.134:0/772513122 shutdown_connections 2024-07-26T11:20:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.782+0000 7f6338800640 1 --2- 172.21.15.134:0/772513122 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63180b3e90 0x7f63180b63a0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.782+0000 7f6338800640 1 --2- 172.21.15.134:0/772513122 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6330114190 0x7f63301aa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.782+0000 7f6338800640 1 --2- 172.21.15.134:0/772513122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6330113840 0x7f63301a4e50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.782+0000 7f6338800640 1 --2- 172.21.15.134:0/772513122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f633010df10 0x7f63301a48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:20.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.782+0000 7f6338800640 1 -- 172.21.15.134:0/772513122 >> 172.21.15.134:0/772513122 conn(0x7f6330109df0 msgr2=0x7f633010c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.782+0000 7f6338800640 1 -- 172.21.15.134:0/772513122 shutdown_connections 2024-07-26T11:20:20.785 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:20.784+0000 7f6338800640 1 -- 172.21.15.134:0/772513122 wait complete. 2024-07-26T11:20:20.881 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-26T11:20:21.895 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:20:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:22 smithi134 ceph-mon[31966]: pgmap v612: 129 pgs: 129 active+clean; 27 GiB data, 57 GiB used, 1015 GiB / 1.0 TiB avail; 16 KiB/s rd, 181 MiB/s wr, 1.67k op/s 2024-07-26T11:20:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:22 smithi189 ceph-mon[36790]: pgmap v612: 129 pgs: 129 active+clean; 27 GiB data, 57 GiB used, 1015 GiB / 1.0 TiB avail; 16 KiB/s rd, 181 MiB/s wr, 1.67k op/s 2024-07-26T11:20:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:22 smithi193 ceph-mon[37473]: pgmap v612: 129 pgs: 129 active+clean; 27 GiB data, 57 GiB used, 1015 GiB / 1.0 TiB avail; 16 KiB/s rd, 181 MiB/s wr, 1.67k op/s 2024-07-26T11:20:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:24 smithi189 ceph-mon[36790]: pgmap v613: 129 pgs: 129 active+clean; 27 GiB data, 58 GiB used, 1015 GiB / 1.0 TiB avail; 26 KiB/s rd, 164 MiB/s wr, 1.71k op/s 2024-07-26T11:20:24.762 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:24 smithi134 ceph-mon[31966]: pgmap v613: 129 pgs: 129 active+clean; 27 GiB data, 58 GiB used, 1015 GiB / 1.0 TiB avail; 26 KiB/s rd, 164 MiB/s wr, 1.71k op/s 2024-07-26T11:20:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:24 smithi193 ceph-mon[37473]: pgmap v613: 129 pgs: 129 active+clean; 27 GiB data, 58 GiB used, 1015 GiB / 1.0 TiB avail; 26 KiB/s rd, 164 MiB/s wr, 1.71k op/s 2024-07-26T11:20:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:26 smithi189 ceph-mon[36790]: pgmap v614: 129 pgs: 129 active+clean; 28 GiB data, 58 GiB used, 1015 GiB / 1.0 TiB avail; 18 KiB/s rd, 177 MiB/s wr, 1.95k op/s 2024-07-26T11:20:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:26 smithi193 ceph-mon[37473]: pgmap v614: 129 pgs: 129 active+clean; 28 GiB data, 58 GiB used, 1015 GiB / 1.0 TiB avail; 18 KiB/s rd, 177 MiB/s wr, 1.95k op/s 2024-07-26T11:20:26.889 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:26 smithi134 ceph-mon[31966]: pgmap v614: 129 pgs: 129 active+clean; 28 GiB data, 58 GiB used, 1015 GiB / 1.0 TiB avail; 18 KiB/s rd, 177 MiB/s wr, 1.95k op/s 2024-07-26T11:20:28.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:28 smithi189 ceph-mon[36790]: pgmap v615: 129 pgs: 129 active+clean; 28 GiB data, 58 GiB used, 1014 GiB / 1.0 TiB avail; 19 KiB/s rd, 185 MiB/s wr, 1.95k op/s 2024-07-26T11:20:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:28 smithi193 ceph-mon[37473]: pgmap v615: 129 pgs: 129 active+clean; 28 GiB data, 58 GiB used, 1014 GiB / 1.0 TiB avail; 19 KiB/s rd, 185 MiB/s wr, 1.95k op/s 2024-07-26T11:20:29.030 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:28 smithi134 ceph-mon[31966]: pgmap v615: 129 pgs: 129 active+clean; 28 GiB data, 58 GiB used, 1014 GiB / 1.0 TiB avail; 19 KiB/s rd, 185 MiB/s wr, 1.95k op/s 2024-07-26T11:20:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:30 smithi134 ceph-mon[31966]: pgmap v616: 129 pgs: 129 active+clean; 28 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 22 KiB/s rd, 208 MiB/s wr, 2.24k op/s 2024-07-26T11:20:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:30 smithi189 ceph-mon[36790]: pgmap v616: 129 pgs: 129 active+clean; 28 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 22 KiB/s rd, 208 MiB/s wr, 2.24k op/s 2024-07-26T11:20:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:30 smithi193 ceph-mon[37473]: pgmap v616: 129 pgs: 129 active+clean; 28 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 22 KiB/s rd, 208 MiB/s wr, 2.24k op/s 2024-07-26T11:20:32.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:32 smithi134 ceph-mon[31966]: pgmap v617: 129 pgs: 129 active+clean; 29 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 18 KiB/s rd, 173 MiB/s wr, 1.90k op/s 2024-07-26T11:20:32.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:32 smithi189 ceph-mon[36790]: pgmap v617: 129 pgs: 129 active+clean; 29 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 18 KiB/s rd, 173 MiB/s wr, 1.90k op/s 2024-07-26T11:20:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:32 smithi193 ceph-mon[37473]: pgmap v617: 129 pgs: 129 active+clean; 29 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 18 KiB/s rd, 173 MiB/s wr, 1.90k op/s 2024-07-26T11:20:34.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:34 smithi134 ceph-mon[31966]: pgmap v618: 129 pgs: 129 active+clean; 29 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 179 KiB/s rd, 169 MiB/s wr, 1.83k op/s 2024-07-26T11:20:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:34 smithi189 ceph-mon[36790]: pgmap v618: 129 pgs: 129 active+clean; 29 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 179 KiB/s rd, 169 MiB/s wr, 1.83k op/s 2024-07-26T11:20:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:34 smithi193 ceph-mon[37473]: pgmap v618: 129 pgs: 129 active+clean; 29 GiB data, 61 GiB used, 1012 GiB / 1.0 TiB avail; 179 KiB/s rd, 169 MiB/s wr, 1.83k op/s 2024-07-26T11:20:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:36 smithi134 ceph-mon[31966]: pgmap v619: 129 pgs: 129 active+clean; 29 GiB data, 61 GiB used, 1011 GiB / 1.0 TiB avail; 168 KiB/s rd, 187 MiB/s wr, 1.92k op/s 2024-07-26T11:20:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:36 smithi189 ceph-mon[36790]: pgmap v619: 129 pgs: 129 active+clean; 29 GiB data, 61 GiB used, 1011 GiB / 1.0 TiB avail; 168 KiB/s rd, 187 MiB/s wr, 1.92k op/s 2024-07-26T11:20:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:36 smithi193 ceph-mon[37473]: pgmap v619: 129 pgs: 129 active+clean; 29 GiB data, 61 GiB used, 1011 GiB / 1.0 TiB avail; 168 KiB/s rd, 187 MiB/s wr, 1.92k op/s 2024-07-26T11:20:38.677 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:38 smithi134 ceph-mon[31966]: pgmap v620: 129 pgs: 129 active+clean; 30 GiB data, 62 GiB used, 1010 GiB / 1.0 TiB avail; 170 KiB/s rd, 197 MiB/s wr, 1.90k op/s 2024-07-26T11:20:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:38 smithi189 ceph-mon[36790]: pgmap v620: 129 pgs: 129 active+clean; 30 GiB data, 62 GiB used, 1010 GiB / 1.0 TiB avail; 170 KiB/s rd, 197 MiB/s wr, 1.90k op/s 2024-07-26T11:20:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:38 smithi193 ceph-mon[37473]: pgmap v620: 129 pgs: 129 active+clean; 30 GiB data, 62 GiB used, 1010 GiB / 1.0 TiB avail; 170 KiB/s rd, 197 MiB/s wr, 1.90k op/s 2024-07-26T11:20:39.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:20:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:20:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:20:39.866 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.864+0000 7f140d200640 1 -- 172.21.15.134:0/251527942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1408115fd0 msgr2=0x7f14081184b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.864+0000 7f140d200640 1 --2- 172.21.15.134:0/251527942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1408115fd0 0x7f14081184b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f13f8009900 tx=0x7f13f802a420 comp rx=0 tx=0).stop 2024-07-26T11:20:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.865+0000 7f140d200640 1 -- 172.21.15.134:0/251527942 shutdown_connections 2024-07-26T11:20:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.865+0000 7f140d200640 1 --2- 172.21.15.134:0/251527942 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1408115fd0 0x7f14081184b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.865+0000 7f140d200640 1 --2- 172.21.15.134:0/251527942 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f140810a920 0x7f1408115a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.865+0000 7f140d200640 1 --2- 172.21.15.134:0/251527942 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1408109fd0 0x7f140810a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:39.867 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.865+0000 7f140d200640 1 -- 172.21.15.134:0/251527942 >> 172.21.15.134:0/251527942 conn(0x7f1408070800 msgr2=0x7f1408070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:39.872 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.871+0000 7f140d200640 1 -- 172.21.15.134:0/251527942 shutdown_connections 2024-07-26T11:20:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.871+0000 7f140d200640 1 -- 172.21.15.134:0/251527942 wait complete. 2024-07-26T11:20:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.873+0000 7f140d200640 1 Processor -- start 2024-07-26T11:20:39.875 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.873+0000 7f140d200640 1 -- start start 2024-07-26T11:20:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.874+0000 7f140d200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1408109fd0 0x7f14081a2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:39.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.875+0000 7f140d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f140810a920 0x7f14081a2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.875+0000 7f140d200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1408115fd0 0x7f14081a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.875+0000 7f140d200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f140811bcd0 con 0x7f1408115fd0 2024-07-26T11:20:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.875+0000 7f140d200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f140811bb50 con 0x7f140810a920 2024-07-26T11:20:39.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.875+0000 7f140d200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f140811be50 con 0x7f1408109fd0 2024-07-26T11:20:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.877+0000 7f1406800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1408115fd0 0x7f14081a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:39.879 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.877+0000 7f1406800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1408115fd0 0x7f14081a7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:43770/0 (socket says 172.21.15.134:43770) 2024-07-26T11:20:39.880 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.877+0000 7f1406800640 1 -- 172.21.15.134:0/2886873203 learned_addr learned my addr 172.21.15.134:0/2886873203 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:39.881 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.879+0000 7f1406800640 1 -- 172.21.15.134:0/2886873203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1408109fd0 msgr2=0x7f14081a2610 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T11:20:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.879+0000 7f1406800640 1 --2- 172.21.15.134:0/2886873203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1408109fd0 0x7f14081a2610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.879+0000 7f1406800640 1 -- 172.21.15.134:0/2886873203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f140810a920 msgr2=0x7f14081a2bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-07-26T11:20:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.879+0000 7f1406800640 1 --2- 172.21.15.134:0/2886873203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f140810a920 0x7f14081a2bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.879+0000 7f1406800640 1 -- 172.21.15.134:0/2886873203 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14081a84b0 con 0x7f1408115fd0 2024-07-26T11:20:39.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.880+0000 7f1406800640 1 --2- 172.21.15.134:0/2886873203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1408115fd0 0x7f14081a7e40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f13f8031610 tx=0x7f13f8004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:39.894 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.892+0000 7f13fea00640 1 -- 172.21.15.134:0/2886873203 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f8035070 con 0x7f1408115fd0 2024-07-26T11:20:39.895 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.893+0000 7f13fea00640 1 -- 172.21.15.134:0/2886873203 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13f803dc10 con 0x7f1408115fd0 2024-07-26T11:20:39.895 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.893+0000 7f13fea00640 1 -- 172.21.15.134:0/2886873203 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f80457c0 con 0x7f1408115fd0 2024-07-26T11:20:39.895 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.893+0000 7f140d200640 1 -- 172.21.15.134:0/2886873203 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14081a8740 con 0x7f1408115fd0 2024-07-26T11:20:39.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.895+0000 7f140d200640 1 -- 172.21.15.134:0/2886873203 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14081a8be0 con 0x7f1408115fd0 2024-07-26T11:20:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.916+0000 7f13fea00640 1 -- 172.21.15.134:0/2886873203 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f13f8045960 con 0x7f1408115fd0 2024-07-26T11:20:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.918+0000 7f13fea00640 1 --2- 172.21.15.134:0/2886873203 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f13ec0b3e00 0x7f13ec0b6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:39.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.919+0000 7f13fea00640 1 -- 172.21.15.134:0/2886873203 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f13f80ccfb0 con 0x7f1408115fd0 2024-07-26T11:20:39.925 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:39.923+0000 7f1405e00640 1 --2- 172.21.15.134:0/2886873203 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f13ec0b3e00 0x7f13ec0b6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:40.247 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.245+0000 7f1405e00640 1 --2- 172.21.15.134:0/2886873203 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f13ec0b3e00 0x7f13ec0b6310 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1400007c10 tx=0x7f14000075f0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.247+0000 7f14143a4740 1 Processor -- start 2024-07-26T11:20:40.256 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.250+0000 7f14143a4740 1 -- start start 2024-07-26T11:20:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.250+0000 7f14143a4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d711b0 0x564459114a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.252+0000 7f14143a4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 0x564459114fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.252+0000 7f14143a4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d504d0 0x5644591154e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.252+0000 7f14143a4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564459089270 con 0x564458d711b0 2024-07-26T11:20:40.257 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.253+0000 7f1405400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 0x564459114fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.253+0000 7f1405400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 0x564459114fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:55948/0 (socket says 172.21.15.134:55948) 2024-07-26T11:20:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.253+0000 7f1405400640 1 -- 172.21.15.134:0/1394266251 learned_addr learned my addr 172.21.15.134:0/1394266251 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.253+0000 7f1406800640 1 --2- 172.21.15.134:0/1394266251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d504d0 0x5644591154e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.254+0000 7f14143a4740 1 -- 172.21.15.134:0/1394266251 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564459078770 con 0x564458d482f0 2024-07-26T11:20:40.258 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.254+0000 7f14143a4740 1 -- 172.21.15.134:0/1394266251 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56445907c7b0 con 0x564458d504d0 2024-07-26T11:20:40.260 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.254+0000 7f1405e00640 1 --2- 172.21.15.134:0/1394266251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d711b0 0x564459114a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:40.262 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.255+0000 7f1406800640 1 -- 172.21.15.134:0/1394266251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 msgr2=0x564459114fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:40.268 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.255+0000 7f1406800640 1 --2- 172.21.15.134:0/1394266251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 0x564459114fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.268 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.255+0000 7f1406800640 1 -- 172.21.15.134:0/1394266251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d711b0 msgr2=0x564459114a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:40.268 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.255+0000 7f1406800640 1 --2- 172.21.15.134:0/1394266251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d711b0 0x564459114a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.268 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.255+0000 7f1406800640 1 -- 172.21.15.134:0/1394266251 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564458ddc790 con 0x564458d504d0 2024-07-26T11:20:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.260+0000 7f1406800640 1 --2- 172.21.15.134:0/1394266251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d504d0 0x5644591154e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f13f8107500 tx=0x7f13f80d60e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=808eb5ec305aa9a1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.261+0000 7f13f7400640 1 -- 172.21.15.134:0/1394266251 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f80cd1b0 con 0x564458d504d0 2024-07-26T11:20:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.261+0000 7f13f7400640 1 -- 172.21.15.134:0/1394266251 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13f800b0b0 con 0x564458d504d0 2024-07-26T11:20:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.261+0000 7f13f7400640 1 -- 172.21.15.134:0/1394266251 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f80cd350 con 0x564458d504d0 2024-07-26T11:20:40.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.261+0000 7f1405400640 1 --2- 172.21.15.134:0/1394266251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 0x564459114fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:20:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.263+0000 7f14143a4740 1 -- 172.21.15.134:0/1394266251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d504d0 msgr2=0x5644591154e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.263+0000 7f14143a4740 1 --2- 172.21.15.134:0/1394266251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d504d0 0x5644591154e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f13f8107500 tx=0x7f13f80d60e0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.266+0000 7f14143a4740 1 -- 172.21.15.134:0/1394266251 shutdown_connections 2024-07-26T11:20:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.267+0000 7f14143a4740 1 --2- 172.21.15.134:0/1394266251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d504d0 0x5644591154e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.269+0000 7f14143a4740 1 --2- 172.21.15.134:0/1394266251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 0x564459114fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.269+0000 7f14143a4740 1 --2- 172.21.15.134:0/1394266251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d711b0 0x564459114a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.272 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.269+0000 7f14143a4740 1 -- 172.21.15.134:0/1394266251 >> 172.21.15.134:0/1394266251 conn(0x564458d65300 msgr2=0x564459112950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.270+0000 7f14143a4740 1 -- 172.21.15.134:0/1394266251 shutdown_connections 2024-07-26T11:20:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.270+0000 7f14143a4740 1 -- 172.21.15.134:0/1394266251 wait complete. 2024-07-26T11:20:40.278 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.272+0000 7f14143a4740 1 -- start start 2024-07-26T11:20:40.279 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.276+0000 7f14143a4740 1 Processor -- start 2024-07-26T11:20:40.286 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.284+0000 7f14143a4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d482f0 0x5644591209c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:40.287 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.285+0000 7f14143a4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d504d0 0x564459120f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:40.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.286+0000 7f1405400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d504d0 0x564459120f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.286+0000 7f1405400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d504d0 0x564459120f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:55990/0 (socket says 172.21.15.134:55990) 2024-07-26T11:20:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.287+0000 7f14143a4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d711b0 0x564459125fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.287+0000 7f14143a4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564458fb2ab0 con 0x564458d482f0 2024-07-26T11:20:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.287+0000 7f14143a4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564459015d50 con 0x564458d504d0 2024-07-26T11:20:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.287+0000 7f14143a4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56445907c7b0 con 0x564458d711b0 2024-07-26T11:20:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.287+0000 7f1405e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d482f0 0x5644591209c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.287+0000 7f14143a4740 10 client.0 _ll_register_callbacks cb 0x564458d88a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.287+0000 7f1405e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d482f0 0x5644591209c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:43808/0 (socket says 172.21.15.134:43808) 2024-07-26T11:20:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.287+0000 7f1405e00640 1 -- 172.21.15.134:0/122214902 learned_addr learned my addr 172.21.15.134:0/122214902 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.288+0000 7f1405e00640 1 -- 172.21.15.134:0/122214902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d711b0 msgr2=0x564459125fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:20:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.288+0000 7f1405e00640 1 --2- 172.21.15.134:0/122214902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d711b0 0x564459125fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.288+0000 7f1405e00640 1 -- 172.21.15.134:0/122214902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d504d0 msgr2=0x564459120f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.288+0000 7f1405e00640 1 --2- 172.21.15.134:0/122214902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d504d0 0x564459120f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.288+0000 7f1405e00640 1 -- 172.21.15.134:0/122214902 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564458e55620 con 0x564458d482f0 2024-07-26T11:20:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.291+0000 7f1405e00640 1 --2- 172.21.15.134:0/122214902 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d482f0 0x5644591209c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f140000fe20 tx=0x7f1400004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bc2d23ab8200db server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:40.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.291+0000 7f13ebe00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:20:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.291+0000 7f1405400640 1 --2- 172.21.15.134:0/122214902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d504d0 0x564459120f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-07-26T11:20:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.292+0000 7f13ebe00640 1 -- 172.21.15.134:0/122214902 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1400007d20 con 0x564458d482f0 2024-07-26T11:20:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.292+0000 7f13ebe00640 1 -- 172.21.15.134:0/122214902 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f140000edc0 con 0x564458d482f0 2024-07-26T11:20:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.292+0000 7f14143a4740 1 -- 172.21.15.134:0/122214902 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13f800b0b0 con 0x564458d482f0 2024-07-26T11:20:40.295 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.293+0000 7f13ebe00640 1 -- 172.21.15.134:0/122214902 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f140002d050 con 0x564458d482f0 2024-07-26T11:20:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.293+0000 7f14143a4740 10 client.15879 fetch_fsmap learned FSMap version 32 2024-07-26T11:20:40.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.294+0000 7f14143a4740 1 -- 172.21.15.134:0/122214902 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56445913ff00 con 0x564458d482f0 2024-07-26T11:20:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.295+0000 7f13ebe00640 1 -- 172.21.15.134:0/122214902 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13f800b0b0 con 0x564458d482f0 2024-07-26T11:20:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.295+0000 7f13ebe00640 1 -- 172.21.15.134:0/122214902 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f140001d4f0 con 0x564458d482f0 2024-07-26T11:20:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.298+0000 7f14143a4740 10 client.15879 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:20:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.298+0000 7f14143a4740 10 client.15879 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:20:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.298+0000 7f14143a4740 20 client.15879 populate_metadata read hostname 'smithi134' 2024-07-26T11:20:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.298+0000 7f14143a4740 1 --2- 172.21.15.134:0/122214902 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x564458d2fd80 0x564459143550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.298+0000 7f14143a4740 4 client.15879 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:20:40.301 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.298+0000 7f14143a4740 1 -- 172.21.15.134:0/122214902 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56445908b1c0 con 0x564458d2fd80 2024-07-26T11:20:40.302 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.299+0000 7f1405400640 1 --2- 172.21.15.134:0/122214902 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x564458d2fd80 0x564459143550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:40.303 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.302+0000 7f1405400640 1 --2- 172.21.15.134:0/122214902 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x564458d2fd80 0x564459143550 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f143d37f72d92af8 server_cookie=f22e6732403c76a7 in_seq=0 out_seq=0 2024-07-26T11:20:40.304 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.302+0000 7f13ebe00640 10 client.15879 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:20:40.306 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.304+0000 7f13ebe00640 1 -- 172.21.15.134:0/122214902 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56445908b1c0 con 0x564458d2fd80 2024-07-26T11:20:40.306 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.304+0000 7f13ebe00640 10 client.15879 handle_command_reply: tid=0 2024-07-26T11:20:40.307 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.305+0000 7f14143a4740 1 client.15879 shutdown 2024-07-26T11:20:40.308 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.305+0000 7f14143a4740 1 -- 172.21.15.134:0/122214902 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d482f0 msgr2=0x5644591209c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:40.308 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.305+0000 7f14143a4740 1 --2- 172.21.15.134:0/122214902 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d482f0 0x5644591209c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f140000fe20 tx=0x7f1400004ab0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.309 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.306+0000 7f14143a4740 1 -- 172.21.15.134:0/122214902 shutdown_connections 2024-07-26T11:20:40.309 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.306+0000 7f14143a4740 1 --2- 172.21.15.134:0/122214902 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x564458d2fd80 0x564459143550 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.309 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.306+0000 7f14143a4740 1 --2- 172.21.15.134:0/122214902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d711b0 0x564459125fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.309 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.306+0000 7f14143a4740 1 --2- 172.21.15.134:0/122214902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d504d0 0x564459120f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.309 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.307+0000 7f14143a4740 1 --2- 172.21.15.134:0/122214902 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d482f0 0x5644591209c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.309 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.307+0000 7f14143a4740 1 -- 172.21.15.134:0/122214902 >> 172.21.15.134:0/122214902 conn(0x564458d65300 msgr2=0x564459117bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:40.310 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.307+0000 7f13ebe00640 0 client.15879 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:20:40.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.308+0000 7f14143a4740 1 -- 172.21.15.134:0/122214902 shutdown_connections 2024-07-26T11:20:40.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.309+0000 7f14143a4740 1 -- 172.21.15.134:0/122214902 wait complete. 2024-07-26T11:20:40.312 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.310+0000 7f14143a4740 20 client.15879 trim_cache size 0 max 16384 2024-07-26T11:20:40.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.347+0000 7f14143a4740 1 Processor -- start 2024-07-26T11:20:40.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.347+0000 7f14143a4740 1 -- start start 2024-07-26T11:20:40.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.348+0000 7f14143a4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 0x564459118290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.348+0000 7f14143a4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d504d0 0x564459119d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.348+0000 7f14143a4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d711b0 0x56445911a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.348+0000 7f14143a4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f14081b6d20 con 0x564458d504d0 2024-07-26T11:20:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.348+0000 7f14143a4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f14081b8860 con 0x564458d482f0 2024-07-26T11:20:40.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.348+0000 7f14143a4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f14081b73f0 con 0x564458d711b0 2024-07-26T11:20:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.351+0000 7f1405e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 0x564459118290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.351+0000 7f1406800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d711b0 0x56445911a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.351+0000 7f1406800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d711b0 0x56445911a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53774/0 (socket says 172.21.15.134:53774) 2024-07-26T11:20:40.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.351+0000 7f1406800640 1 -- 172.21.15.134:0/2731152004 learned_addr learned my addr 172.21.15.134:0/2731152004 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.352+0000 7f1406800640 1 -- 172.21.15.134:0/2731152004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 msgr2=0x564459118290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:40.356 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.352+0000 7f1406800640 1 --2- 172.21.15.134:0/2731152004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 0x564459118290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.352+0000 7f1406800640 1 -- 172.21.15.134:0/2731152004 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d504d0 msgr2=0x564459119d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.352+0000 7f1406800640 1 --2- 172.21.15.134:0/2731152004 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d504d0 0x564459119d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.352+0000 7f1406800640 1 -- 172.21.15.134:0/2731152004 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56445910e370 con 0x564458d711b0 2024-07-26T11:20:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.355+0000 7f1406800640 1 --2- 172.21.15.134:0/2731152004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d711b0 0x56445911a290 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f13f8031850 tx=0x7f13f80314a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d815ab638d102c4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.355+0000 7f13f6a00640 1 -- 172.21.15.134:0/2731152004 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f80cd520 con 0x564458d711b0 2024-07-26T11:20:40.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.355+0000 7f13f6a00640 1 -- 172.21.15.134:0/2731152004 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13f802cea0 con 0x564458d711b0 2024-07-26T11:20:40.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.355+0000 7f13f6a00640 1 -- 172.21.15.134:0/2731152004 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f80cd6c0 con 0x564458d711b0 2024-07-26T11:20:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.357+0000 7f14143a4740 1 -- 172.21.15.134:0/2731152004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d711b0 msgr2=0x56445911a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:40.359 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.357+0000 7f14143a4740 1 --2- 172.21.15.134:0/2731152004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d711b0 0x56445911a290 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f13f8031850 tx=0x7f13f80314a0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.358+0000 7f14143a4740 1 -- 172.21.15.134:0/2731152004 shutdown_connections 2024-07-26T11:20:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.358+0000 7f14143a4740 1 --2- 172.21.15.134:0/2731152004 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d711b0 0x56445911a290 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.358+0000 7f14143a4740 1 --2- 172.21.15.134:0/2731152004 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d504d0 0x564459119d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.358+0000 7f14143a4740 1 --2- 172.21.15.134:0/2731152004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 0x564459118290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.358+0000 7f14143a4740 1 -- 172.21.15.134:0/2731152004 >> 172.21.15.134:0/2731152004 conn(0x564458d65300 msgr2=0x564459117b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.359+0000 7f14143a4740 1 -- 172.21.15.134:0/2731152004 shutdown_connections 2024-07-26T11:20:40.362 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.359+0000 7f14143a4740 1 -- 172.21.15.134:0/2731152004 wait complete. 2024-07-26T11:20:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.360+0000 7f14143a4740 1 -- start start 2024-07-26T11:20:40.363 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.360+0000 7f14143a4740 1 Processor -- start 2024-07-26T11:20:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.363+0000 7f14143a4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 0x564459148390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:40.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.364+0000 7f14143a4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d504d0 0x5644591488d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:40.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.364+0000 7f14143a4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d711b0 0x564459153a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.364+0000 7f1405e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 0x564459148390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.364+0000 7f1405e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 0x564459148390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56016/0 (socket says 172.21.15.134:56016) 2024-07-26T11:20:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.364+0000 7f1405400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d504d0 0x5644591488d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.364+0000 7f1405400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d504d0 0x5644591488d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53778/0 (socket says 172.21.15.134:53778) 2024-07-26T11:20:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.364+0000 7f14143a4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f14081b8190 con 0x564458d711b0 2024-07-26T11:20:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.364+0000 7f14143a4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f14081b7ac0 con 0x564458d482f0 2024-07-26T11:20:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.364+0000 7f14143a4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f14081b4d00 con 0x564458d504d0 2024-07-26T11:20:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.364+0000 7f14143a4740 10 client.0 _ll_register_callbacks cb 0x7f140000fad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:40.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.364+0000 7f1405e00640 1 -- 172.21.15.134:0/885619560 learned_addr learned my addr 172.21.15.134:0/885619560 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.366+0000 7f1405e00640 1 -- 172.21.15.134:0/885619560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d504d0 msgr2=0x5644591488d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.366+0000 7f1405e00640 1 --2- 172.21.15.134:0/885619560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d504d0 0x5644591488d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.366+0000 7f1405e00640 1 -- 172.21.15.134:0/885619560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d711b0 msgr2=0x564459153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.366+0000 7f1405e00640 1 --2- 172.21.15.134:0/885619560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d711b0 0x564459153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.366+0000 7f1405e00640 1 -- 172.21.15.134:0/885619560 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564459153fc0 con 0x564458d482f0 2024-07-26T11:20:40.368 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.366+0000 7f1405e00640 1 --2- 172.21.15.134:0/885619560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 0x564459148390 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f140000cc50 tx=0x7f140000ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18ab0b92285daace server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.366+0000 7f13ebe00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:20:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.367+0000 7f13ebe00640 1 -- 172.21.15.134:0/885619560 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1400007d20 con 0x564458d482f0 2024-07-26T11:20:40.369 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.367+0000 7f13ebe00640 1 -- 172.21.15.134:0/885619560 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f1400004b20 con 0x564458d482f0 2024-07-26T11:20:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.367+0000 7f14143a4740 1 -- 172.21.15.134:0/885619560 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13f802cea0 con 0x564458d482f0 2024-07-26T11:20:40.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.368+0000 7f13ebe00640 1 -- 172.21.15.134:0/885619560 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1400040050 con 0x564458d482f0 2024-07-26T11:20:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.369+0000 7f14143a4740 10 client.26243 fetch_fsmap learned FSMap version 32 2024-07-26T11:20:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.369+0000 7f14143a4740 1 -- 172.21.15.134:0/885619560 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5644591551c0 con 0x564458d482f0 2024-07-26T11:20:40.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.372+0000 7f13ebe00640 1 -- 172.21.15.134:0/885619560 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13f802cea0 con 0x564458d482f0 2024-07-26T11:20:40.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.372+0000 7f13ebe00640 1 -- 172.21.15.134:0/885619560 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f1400039070 con 0x564458d482f0 2024-07-26T11:20:40.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.375+0000 7f14143a4740 10 client.26243 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:20:40.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.375+0000 7f14143a4740 10 client.26243 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:20:40.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.375+0000 7f14143a4740 20 client.26243 populate_metadata read hostname 'smithi134' 2024-07-26T11:20:40.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.375+0000 7f14143a4740 1 --2- 172.21.15.134:0/885619560 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x564458d051c0 0x5644591adbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:40.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.375+0000 7f14143a4740 4 client.26243 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:20:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.375+0000 7f14143a4740 1 -- 172.21.15.134:0/885619560 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564459086c60 con 0x564458d051c0 2024-07-26T11:20:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.378+0000 7f1405400640 1 --2- 172.21.15.134:0/885619560 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x564458d051c0 0x5644591adbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.379+0000 7f1405400640 1 --2- 172.21.15.134:0/885619560 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x564458d051c0 0x5644591adbb0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d66ae8a3aff697d6 server_cookie=f6b5713c29688030 in_seq=0 out_seq=0 2024-07-26T11:20:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.380+0000 7f13ebe00640 10 client.26243 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:20:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.629+0000 7f13ebe00640 1 -- 172.21.15.134:0/885619560 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564459086c60 con 0x564458d051c0 2024-07-26T11:20:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.629+0000 7f13ebe00640 10 client.26243 handle_command_reply: tid=0 2024-07-26T11:20:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.629+0000 7f14143a4740 1 client.26243 shutdown 2024-07-26T11:20:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.631+0000 7f14143a4740 1 -- 172.21.15.134:0/885619560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 msgr2=0x564459148390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.631+0000 7f14143a4740 1 --2- 172.21.15.134:0/885619560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 0x564459148390 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f140000cc50 tx=0x7f140000ec90 comp rx=0 tx=0).stop 2024-07-26T11:20:40.634 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.632+0000 7f14143a4740 1 -- 172.21.15.134:0/885619560 shutdown_connections 2024-07-26T11:20:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.632+0000 7f14143a4740 1 --2- 172.21.15.134:0/885619560 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x564458d051c0 0x5644591adbb0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.633+0000 7f14143a4740 1 --2- 172.21.15.134:0/885619560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564458d504d0 0x5644591488d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.633+0000 7f14143a4740 1 --2- 172.21.15.134:0/885619560 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564458d711b0 0x564459153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.633+0000 7f14143a4740 1 --2- 172.21.15.134:0/885619560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564458d482f0 0x564459148390 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.633+0000 7f14143a4740 1 -- 172.21.15.134:0/885619560 >> 172.21.15.134:0/885619560 conn(0x564458d65300 msgr2=0x564459115a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.633+0000 7f13ebe00640 0 client.26243 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:20:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.634+0000 7f14143a4740 1 -- 172.21.15.134:0/885619560 shutdown_connections 2024-07-26T11:20:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.634+0000 7f14143a4740 1 -- 172.21.15.134:0/885619560 wait complete. 2024-07-26T11:20:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.634+0000 7f14143a4740 20 client.26243 trim_cache size 0 max 16384 2024-07-26T11:20:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:20:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:20:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:20:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:20:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.637+0000 7f140d200640 1 -- 172.21.15.134:0/2886873203 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f13ec0b3e00 msgr2=0x7f13ec0b6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.637+0000 7f140d200640 1 --2- 172.21.15.134:0/2886873203 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f13ec0b3e00 0x7f13ec0b6310 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1400007c10 tx=0x7f14000075f0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.637+0000 7f140d200640 1 -- 172.21.15.134:0/2886873203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1408115fd0 msgr2=0x7f14081a7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.637+0000 7f140d200640 1 --2- 172.21.15.134:0/2886873203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1408115fd0 0x7f14081a7e40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f13f8031610 tx=0x7f13f8004580 comp rx=0 tx=0).stop 2024-07-26T11:20:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.637+0000 7f140d200640 1 -- 172.21.15.134:0/2886873203 shutdown_connections 2024-07-26T11:20:40.640 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.637+0000 7f140d200640 1 --2- 172.21.15.134:0/2886873203 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f13ec0b3e00 0x7f13ec0b6310 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.637+0000 7f140d200640 1 --2- 172.21.15.134:0/2886873203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1408115fd0 0x7f14081a7e40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.637+0000 7f140d200640 1 --2- 172.21.15.134:0/2886873203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f140810a920 0x7f14081a2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.638+0000 7f140d200640 1 --2- 172.21.15.134:0/2886873203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1408109fd0 0x7f14081a2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.638+0000 7f140d200640 1 -- 172.21.15.134:0/2886873203 >> 172.21.15.134:0/2886873203 conn(0x7f1408070800 msgr2=0x7f140810d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.638+0000 7f140d200640 1 -- 172.21.15.134:0/2886873203 shutdown_connections 2024-07-26T11:20:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:40.638+0000 7f140d200640 1 -- 172.21.15.134:0/2886873203 wait complete. 2024-07-26T11:20:40.675 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:40 smithi134 ceph-mon[31966]: pgmap v621: 129 pgs: 129 active+clean; 30 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 228 KiB/s rd, 232 MiB/s wr, 2.07k op/s 2024-07-26T11:20:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:40 smithi193 ceph-mon[37473]: pgmap v621: 129 pgs: 129 active+clean; 30 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 228 KiB/s rd, 232 MiB/s wr, 2.07k op/s 2024-07-26T11:20:40.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:40 smithi189 ceph-mon[36790]: pgmap v621: 129 pgs: 129 active+clean; 30 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 228 KiB/s rd, 232 MiB/s wr, 2.07k op/s 2024-07-26T11:20:41.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.268+0000 7fa875600640 1 -- 172.21.15.134:0/1727007048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa870102930 msgr2=0x7fa870106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:41.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.268+0000 7fa875600640 1 --2- 172.21.15.134:0/1727007048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa870102930 0x7fa870106e60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa85c009900 tx=0x7fa85c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.271+0000 7fa875600640 1 -- 172.21.15.134:0/1727007048 shutdown_connections 2024-07-26T11:20:41.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.271+0000 7fa875600640 1 --2- 172.21.15.134:0/1727007048 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa870102930 0x7fa870106e60 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.271+0000 7fa875600640 1 --2- 172.21.15.134:0/1727007048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa870107420 0x7fa870107900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.271+0000 7fa875600640 1 --2- 172.21.15.134:0/1727007048 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa870101fe0 0x7fa8701023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.271+0000 7fa875600640 1 -- 172.21.15.134:0/1727007048 >> 172.21.15.134:0/1727007048 conn(0x7fa8700fdcf0 msgr2=0x7fa870100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:41.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.272+0000 7fa875600640 1 -- 172.21.15.134:0/1727007048 shutdown_connections 2024-07-26T11:20:41.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.272+0000 7fa875600640 1 -- 172.21.15.134:0/1727007048 wait complete. 2024-07-26T11:20:41.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.274+0000 7fa875600640 1 Processor -- start 2024-07-26T11:20:41.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.275+0000 7fa875600640 1 -- start start 2024-07-26T11:20:41.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.276+0000 7fa875600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa870101fe0 0x7fa870116570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.276+0000 7fa875600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa870102930 0x7fa87010f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.276+0000 7fa86fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa870101fe0 0x7fa870116570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:41.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.276+0000 7fa875600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa870107420 0x7fa87010fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.277+0000 7fa86fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa870101fe0 0x7fa870116570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53748/0 (socket says 172.21.15.134:53748) 2024-07-26T11:20:41.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.277+0000 7fa875600640 1 -- 172.21.15.134:0/914661767 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa870119910 con 0x7fa870107420 2024-07-26T11:20:41.159 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.277+0000 7fa875600640 1 -- 172.21.15.134:0/914661767 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa870119790 con 0x7fa870102930 2024-07-26T11:20:41.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.277+0000 7fa875600640 1 -- 172.21.15.134:0/914661767 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa870119a90 con 0x7fa870101fe0 2024-07-26T11:20:41.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.279+0000 7fa86fe00640 1 -- 172.21.15.134:0/914661767 learned_addr learned my addr 172.21.15.134:0/914661767 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:41.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.280+0000 7fa874c00640 1 --2- 172.21.15.134:0/914661767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa870107420 0x7fa87010fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:41.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.281+0000 7fa86f400640 1 --2- 172.21.15.134:0/914661767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa870102930 0x7fa87010f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:41.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.282+0000 7fa86f400640 1 -- 172.21.15.134:0/914661767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa870101fe0 msgr2=0x7fa870116570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:41.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.282+0000 7fa86f400640 1 --2- 172.21.15.134:0/914661767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa870101fe0 0x7fa870116570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.282+0000 7fa86f400640 1 -- 172.21.15.134:0/914661767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa870107420 msgr2=0x7fa87010fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:41.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.282+0000 7fa86f400640 1 --2- 172.21.15.134:0/914661767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa870107420 0x7fa87010fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.282+0000 7fa86f400640 1 -- 172.21.15.134:0/914661767 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa870110670 con 0x7fa870102930 2024-07-26T11:20:41.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.282+0000 7fa86f400640 1 --2- 172.21.15.134:0/914661767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa870102930 0x7fa87010f840 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa85c009230 tx=0x7fa85c003c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:41.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.283+0000 7fa86cc00640 1 -- 172.21.15.134:0/914661767 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c03f070 con 0x7fa870102930 2024-07-26T11:20:41.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.283+0000 7fa875600640 1 -- 172.21.15.134:0/914661767 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8701152a0 con 0x7fa870102930 2024-07-26T11:20:41.160 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.283+0000 7fa875600640 1 -- 172.21.15.134:0/914661767 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8701157a0 con 0x7fa870102930 2024-07-26T11:20:41.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.284+0000 7fa86cc00640 1 -- 172.21.15.134:0/914661767 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa85c03bb10 con 0x7fa870102930 2024-07-26T11:20:41.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.285+0000 7fa86cc00640 1 -- 172.21.15.134:0/914661767 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c03f070 con 0x7fa870102930 2024-07-26T11:20:41.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.295+0000 7fa87d866740 1 Processor -- start 2024-07-26T11:20:41.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.295+0000 7fa87d866740 1 -- start start 2024-07-26T11:20:41.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.295+0000 7fa87d866740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1daa4070 0x560a1de49010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.296+0000 7fa87d866740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1da7b1b0 0x560a1de49550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.296+0000 7fa87d866740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da83390 0x560a1de49a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.296+0000 7fa87d866740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560a1ddbfbd0 con 0x560a1da7b1b0 2024-07-26T11:20:41.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.296+0000 7fa87d866740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560a1ddaf0d0 con 0x560a1da83390 2024-07-26T11:20:41.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.296+0000 7fa87d866740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560a1ddb3110 con 0x560a1daa4070 2024-07-26T11:20:41.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.300+0000 7fa86fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1da7b1b0 0x560a1de49550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:41.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.300+0000 7fa86fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1da7b1b0 0x560a1de49550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:43810/0 (socket says 172.21.15.134:43810) 2024-07-26T11:20:41.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.300+0000 7fa86fe00640 1 -- 172.21.15.134:0/744740388 learned_addr learned my addr 172.21.15.134:0/744740388 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:41.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.302+0000 7fa86fe00640 1 -- 172.21.15.134:0/744740388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1daa4070 msgr2=0x560a1de49010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:20:41.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.302+0000 7fa86fe00640 1 --2- 172.21.15.134:0/744740388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1daa4070 0x560a1de49010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.302+0000 7fa86fe00640 1 -- 172.21.15.134:0/744740388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da83390 msgr2=0x560a1de49a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:20:41.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.302+0000 7fa86fe00640 1 --2- 172.21.15.134:0/744740388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da83390 0x560a1de49a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.302+0000 7fa86fe00640 1 -- 172.21.15.134:0/744740388 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a1db0f650 con 0x560a1da7b1b0 2024-07-26T11:20:41.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.304+0000 7fa86fe00640 1 --2- 172.21.15.134:0/744740388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1da7b1b0 0x560a1de49550 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa8680114b0 tx=0x7fa868011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da686f4213225f99 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:41.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.305+0000 7fa865200640 1 -- 172.21.15.134:0/744740388 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868012770 con 0x560a1da7b1b0 2024-07-26T11:20:41.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.305+0000 7fa865200640 1 -- 172.21.15.134:0/744740388 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa868012dc0 con 0x560a1da7b1b0 2024-07-26T11:20:41.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.305+0000 7fa865200640 1 -- 172.21.15.134:0/744740388 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8680193a0 con 0x560a1da7b1b0 2024-07-26T11:20:41.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.307+0000 7fa87d866740 1 -- 172.21.15.134:0/744740388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1da7b1b0 msgr2=0x560a1de49550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:41.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.307+0000 7fa87d866740 1 --2- 172.21.15.134:0/744740388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1da7b1b0 0x560a1de49550 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa8680114b0 tx=0x7fa868011970 comp rx=0 tx=0).stop 2024-07-26T11:20:41.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.309+0000 7fa87d866740 1 -- 172.21.15.134:0/744740388 shutdown_connections 2024-07-26T11:20:41.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.309+0000 7fa87d866740 1 --2- 172.21.15.134:0/744740388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da83390 0x560a1de49a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.309+0000 7fa87d866740 1 --2- 172.21.15.134:0/744740388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1da7b1b0 0x560a1de49550 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.309+0000 7fa87d866740 1 --2- 172.21.15.134:0/744740388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1daa4070 0x560a1de49010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.309+0000 7fa87d866740 1 -- 172.21.15.134:0/744740388 >> 172.21.15.134:0/744740388 conn(0x560a1da98460 msgr2=0x560a1de46f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:41.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.312+0000 7fa86cc00640 1 -- 172.21.15.134:0/914661767 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fa85c042070 con 0x7fa870102930 2024-07-26T11:20:41.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.315+0000 7fa86cc00640 1 --2- 172.21.15.134:0/914661767 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa8540b3df0 0x7fa8540b6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.316+0000 7fa86cc00640 1 -- 172.21.15.134:0/914661767 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa85c0c7410 con 0x7fa870102930 2024-07-26T11:20:41.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.317+0000 7fa87d866740 1 -- 172.21.15.134:0/744740388 shutdown_connections 2024-07-26T11:20:41.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.317+0000 7fa86fe00640 1 --2- 172.21.15.134:0/914661767 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa8540b3df0 0x7fa8540b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:41.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.318+0000 7fa87d866740 1 -- 172.21.15.134:0/744740388 wait complete. 2024-07-26T11:20:41.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.319+0000 7fa87d866740 1 -- start start 2024-07-26T11:20:41.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.319+0000 7fa87d866740 1 Processor -- start 2024-07-26T11:20:41.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.325+0000 7fa87d866740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1da7b1b0 0x560a1de545b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.325+0000 7fa87d866740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1daa4070 0x560a1de54af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.325+0000 7fa87d866740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da629f0 0x560a1de55030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.325+0000 7fa874c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da629f0 0x560a1de55030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:41.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.325+0000 7fa874c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da629f0 0x560a1de55030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:55996/0 (socket says 172.21.15.134:55996) 2024-07-26T11:20:41.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.326+0000 7fa874c00640 1 -- 172.21.15.134:0/580552529 learned_addr learned my addr 172.21.15.134:0/580552529 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:41.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.325+0000 7fa86fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1da7b1b0 0x560a1de545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:41.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.326+0000 7fa87d866740 1 -- 172.21.15.134:0/580552529 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560a1dc12700 con 0x560a1daa4070 2024-07-26T11:20:41.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.327+0000 7fa87d866740 1 -- 172.21.15.134:0/580552529 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560a1dd4b6a0 con 0x560a1da629f0 2024-07-26T11:20:41.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.327+0000 7fa87d866740 1 -- 172.21.15.134:0/580552529 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560a1de577f0 con 0x560a1da7b1b0 2024-07-26T11:20:41.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.328+0000 7fa87d866740 10 client.0 _ll_register_callbacks cb 0x560a1dd5b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:41.165 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:20:41.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.329+0000 7fa86f400640 1 --2- 172.21.15.134:0/580552529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1daa4070 0x560a1de54af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:41.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.329+0000 7fa874c00640 1 -- 172.21.15.134:0/580552529 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1da7b1b0 msgr2=0x560a1de545b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:41.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.329+0000 7fa874c00640 1 --2- 172.21.15.134:0/580552529 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1da7b1b0 0x560a1de545b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.329+0000 7fa874c00640 1 -- 172.21.15.134:0/580552529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1daa4070 msgr2=0x560a1de54af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:41.165 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.329+0000 7fa874c00640 1 --2- 172.21.15.134:0/580552529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1daa4070 0x560a1de54af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.329+0000 7fa874c00640 1 -- 172.21.15.134:0/580552529 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a1dd62840 con 0x560a1da629f0 2024-07-26T11:20:41.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.329+0000 7fa86fe00640 1 --2- 172.21.15.134:0/914661767 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa8540b3df0 0x7fa8540b6300 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa868005f10 tx=0x7fa868005bd0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:41.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.330+0000 7fa874c00640 1 --2- 172.21.15.134:0/580552529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da629f0 0x560a1de55030 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa8600116c0 tx=0x7fa860011b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c2384eb8f49af8f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:41.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.330+0000 7fa859600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:20:41.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.330+0000 7fa859600640 1 -- 172.21.15.134:0/580552529 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86001e060 con 0x560a1da629f0 2024-07-26T11:20:41.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.330+0000 7fa859600640 1 -- 172.21.15.134:0/580552529 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa86000fdf0 con 0x560a1da629f0 2024-07-26T11:20:41.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.331+0000 7fa87d866740 1 -- 172.21.15.134:0/580552529 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa868012dc0 con 0x560a1da629f0 2024-07-26T11:20:41.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.331+0000 7fa859600640 1 -- 172.21.15.134:0/580552529 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa860019430 con 0x560a1da629f0 2024-07-26T11:20:41.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.331+0000 7fa87d866740 10 client.26237 fetch_fsmap learned FSMap version 32 2024-07-26T11:20:41.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.331+0000 7fa87d866740 1 -- 172.21.15.134:0/580552529 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560a1de634c0 con 0x560a1da629f0 2024-07-26T11:20:41.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.331+0000 7fa859600640 1 -- 172.21.15.134:0/580552529 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa868012dc0 con 0x560a1da629f0 2024-07-26T11:20:41.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.332+0000 7fa859600640 1 -- 172.21.15.134:0/580552529 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fa8600226a0 con 0x560a1da629f0 2024-07-26T11:20:41.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.332+0000 7fa87d866740 10 client.26237 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:20:41.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.332+0000 7fa87d866740 10 client.26237 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T11:20:41.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.332+0000 7fa87d866740 20 client.26237 populate_metadata read hostname 'smithi134' 2024-07-26T11:20:41.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.332+0000 7fa87d866740 1 --2- 172.21.15.134:0/580552529 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x560a1da37da0 0x560a1de66980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.332+0000 7fa87d866740 4 client.26237 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:20:41.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.332+0000 7fa87d866740 1 -- 172.21.15.134:0/580552529 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a1ddc1b20 con 0x560a1da37da0 2024-07-26T11:20:41.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.332+0000 7fa86fe00640 1 --2- 172.21.15.134:0/580552529 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x560a1da37da0 0x560a1de66980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:41.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.334+0000 7fa86fe00640 1 --2- 172.21.15.134:0/580552529 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x560a1da37da0 0x560a1de66980 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5e12c2e04b269d7d server_cookie=a89607a081481003 in_seq=0 out_seq=0 2024-07-26T11:20:41.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.334+0000 7fa859600640 10 client.26237 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:20:41.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.336+0000 7fa859600640 1 -- 172.21.15.134:0/580552529 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560a1ddc1b20 con 0x560a1da37da0 2024-07-26T11:20:41.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.336+0000 7fa859600640 10 client.26237 handle_command_reply: tid=0 2024-07-26T11:20:41.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.336+0000 7fa87d866740 1 client.26237 shutdown 2024-07-26T11:20:41.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.341+0000 7fa87d866740 1 -- 172.21.15.134:0/580552529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da629f0 msgr2=0x560a1de55030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:41.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.341+0000 7fa87d866740 1 --2- 172.21.15.134:0/580552529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da629f0 0x560a1de55030 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa8600116c0 tx=0x7fa860011b80 comp rx=0 tx=0).stop 2024-07-26T11:20:41.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.342+0000 7fa87d866740 1 -- 172.21.15.134:0/580552529 shutdown_connections 2024-07-26T11:20:41.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.342+0000 7fa87d866740 1 --2- 172.21.15.134:0/580552529 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x560a1da37da0 0x560a1de66980 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.342+0000 7fa87d866740 1 --2- 172.21.15.134:0/580552529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da629f0 0x560a1de55030 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.342+0000 7fa87d866740 1 --2- 172.21.15.134:0/580552529 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1daa4070 0x560a1de54af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.342+0000 7fa87d866740 1 --2- 172.21.15.134:0/580552529 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1da7b1b0 0x560a1de545b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.342+0000 7fa87d866740 1 -- 172.21.15.134:0/580552529 >> 172.21.15.134:0/580552529 conn(0x560a1da98460 msgr2=0x560a1de4a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:41.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.343+0000 7fa859600640 0 client.26237 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:20:41.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.343+0000 7fa87d866740 1 -- 172.21.15.134:0/580552529 shutdown_connections 2024-07-26T11:20:41.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.343+0000 7fa87d866740 1 -- 172.21.15.134:0/580552529 wait complete. 2024-07-26T11:20:41.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.344+0000 7fa87d866740 20 client.26237 trim_cache size 0 max 16384 2024-07-26T11:20:41.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.371+0000 7fa87d866740 1 Processor -- start 2024-07-26T11:20:41.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.372+0000 7fa87d866740 1 -- start start 2024-07-26T11:20:41.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.373+0000 7fa87d866740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1da629f0 0x560a1de499b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.373+0000 7fa87d866740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da7b1b0 0x560a1de60980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.373+0000 7fa87d866740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1daa4070 0x560a1de561e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.373+0000 7fa87d866740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa850010010 con 0x560a1daa4070 2024-07-26T11:20:41.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.373+0000 7fa87d866740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa850011b50 con 0x560a1da7b1b0 2024-07-26T11:20:41.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.373+0000 7fa87d866740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa8500106e0 con 0x560a1da629f0 2024-07-26T11:20:41.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.374+0000 7fa86fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1da629f0 0x560a1de499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:41.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.374+0000 7fa86fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1da629f0 0x560a1de499b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53784/0 (socket says 172.21.15.134:53784) 2024-07-26T11:20:41.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.374+0000 7fa86fe00640 1 -- 172.21.15.134:0/413507517 learned_addr learned my addr 172.21.15.134:0/413507517 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:41.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.375+0000 7fa86f400640 1 --2- 172.21.15.134:0/413507517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da7b1b0 0x560a1de60980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:41.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.375+0000 7fa86fe00640 1 -- 172.21.15.134:0/413507517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da7b1b0 msgr2=0x560a1de60980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:41.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.375+0000 7fa86fe00640 1 --2- 172.21.15.134:0/413507517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da7b1b0 0x560a1de60980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.375+0000 7fa86fe00640 1 -- 172.21.15.134:0/413507517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1daa4070 msgr2=0x560a1de561e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:41.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.375+0000 7fa86fe00640 1 --2- 172.21.15.134:0/413507517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1daa4070 0x560a1de561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.375+0000 7fa86fe00640 1 -- 172.21.15.134:0/413507517 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a1de4b300 con 0x560a1da629f0 2024-07-26T11:20:41.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.376+0000 7fa86fe00640 1 --2- 172.21.15.134:0/413507517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1da629f0 0x560a1de499b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa86801ab20 tx=0x7fa868011970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6485463ec179f8eb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:41.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.376+0000 7fa85be00640 1 -- 172.21.15.134:0/413507517 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868036030 con 0x560a1da629f0 2024-07-26T11:20:41.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.378+0000 7fa85be00640 1 -- 172.21.15.134:0/413507517 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa86801a770 con 0x560a1da629f0 2024-07-26T11:20:41.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.378+0000 7fa85be00640 1 -- 172.21.15.134:0/413507517 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868026830 con 0x560a1da629f0 2024-07-26T11:20:41.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.379+0000 7fa87d866740 1 -- 172.21.15.134:0/413507517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1da629f0 msgr2=0x560a1de499b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:41.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.379+0000 7fa87d866740 1 --2- 172.21.15.134:0/413507517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1da629f0 0x560a1de499b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa86801ab20 tx=0x7fa868011970 comp rx=0 tx=0).stop 2024-07-26T11:20:41.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.381+0000 7fa87d866740 1 -- 172.21.15.134:0/413507517 shutdown_connections 2024-07-26T11:20:41.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.381+0000 7fa87d866740 1 --2- 172.21.15.134:0/413507517 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1daa4070 0x560a1de561e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.381+0000 7fa87d866740 1 --2- 172.21.15.134:0/413507517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da7b1b0 0x560a1de60980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.381+0000 7fa87d866740 1 --2- 172.21.15.134:0/413507517 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1da629f0 0x560a1de499b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.382+0000 7fa87d866740 1 -- 172.21.15.134:0/413507517 >> 172.21.15.134:0/413507517 conn(0x560a1da98460 msgr2=0x560a1de55ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:41.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.382+0000 7fa87d866740 1 -- 172.21.15.134:0/413507517 shutdown_connections 2024-07-26T11:20:41.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.383+0000 7fa87d866740 1 -- 172.21.15.134:0/413507517 wait complete. 2024-07-26T11:20:41.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.384+0000 7fa87d866740 1 -- start start 2024-07-26T11:20:41.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.384+0000 7fa87d866740 1 Processor -- start 2024-07-26T11:20:41.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.387+0000 7fa87d866740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da629f0 0x560a1de5ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.388+0000 7fa87d866740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1da7b1b0 0x560a1de5d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.388+0000 7fa87d866740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1daa4070 0x560a1de6b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.389+0000 7fa874c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1daa4070 0x560a1de6b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:41.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.389+0000 7fa874c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1daa4070 0x560a1de6b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:53800/0 (socket says 172.21.15.134:53800) 2024-07-26T11:20:41.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.390+0000 7fa874c00640 1 -- 172.21.15.134:0/2269460633 learned_addr learned my addr 172.21.15.134:0/2269460633 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:41.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.390+0000 7fa87d866740 1 -- 172.21.15.134:0/2269460633 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa850011480 con 0x560a1da7b1b0 2024-07-26T11:20:41.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.390+0000 7fa87d866740 1 -- 172.21.15.134:0/2269460633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa850010db0 con 0x560a1da629f0 2024-07-26T11:20:41.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.390+0000 7fa86fe00640 1 --2- 172.21.15.134:0/2269460633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da629f0 0x560a1de5ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:41.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.390+0000 7fa87d866740 1 -- 172.21.15.134:0/2269460633 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa85000dff0 con 0x560a1daa4070 2024-07-26T11:20:41.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.391+0000 7fa86f400640 1 --2- 172.21.15.134:0/2269460633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1da7b1b0 0x560a1de5d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:41.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.391+0000 7fa86fe00640 1 -- 172.21.15.134:0/2269460633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1daa4070 msgr2=0x560a1de6b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:41.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.391+0000 7fa86fe00640 1 --2- 172.21.15.134:0/2269460633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1daa4070 0x560a1de6b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.391+0000 7fa86fe00640 1 -- 172.21.15.134:0/2269460633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1da7b1b0 msgr2=0x560a1de5d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:41.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.392+0000 7fa86fe00640 1 --2- 172.21.15.134:0/2269460633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1da7b1b0 0x560a1de5d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.392+0000 7fa86fe00640 1 -- 172.21.15.134:0/2269460633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a1de6bec0 con 0x560a1da629f0 2024-07-26T11:20:41.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.392+0000 7fa87d866740 10 client.0 _ll_register_callbacks cb 0x7fa860009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:41.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.392+0000 7fa87d866740 1 -- 172.21.15.134:0/2269460633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa86801a770 con 0x560a1da629f0 2024-07-26T11:20:41.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.392+0000 7fa86f400640 1 --2- 172.21.15.134:0/2269460633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1da7b1b0 0x560a1de5d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:20:41.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.393+0000 7fa86fe00640 1 --2- 172.21.15.134:0/2269460633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da629f0 0x560a1de5ccf0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa868012fd0 tx=0x7fa86801aa80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de1666f3a92c12d4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:41.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.393+0000 7fa859600640 10 client.26246 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:20:41.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.394+0000 7fa859600640 1 -- 172.21.15.134:0/2269460633 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa868049050 con 0x560a1da629f0 2024-07-26T11:20:41.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.394+0000 7fa859600640 1 -- 172.21.15.134:0/2269460633 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa868026770 con 0x560a1da629f0 2024-07-26T11:20:41.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.395+0000 7fa859600640 1 -- 172.21.15.134:0/2269460633 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa86803f070 con 0x560a1da629f0 2024-07-26T11:20:41.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.396+0000 7fa87d866740 10 client.26246 fetch_fsmap learned FSMap version 32 2024-07-26T11:20:41.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.396+0000 7fa87d866740 1 -- 172.21.15.134:0/2269460633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=32}) -- 0x560a1dece8c0 con 0x560a1da629f0 2024-07-26T11:20:41.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.398+0000 7fa859600640 1 -- 172.21.15.134:0/2269460633 <== mon.2 v2:172.21.15.189:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fa86804c020 con 0x560a1da629f0 2024-07-26T11:20:41.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.398+0000 7fa87d866740 10 client.26246 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:20:41.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.398+0000 7fa87d866740 10 client.26246 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T11:20:41.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.398+0000 7fa87d866740 20 client.26246 populate_metadata read hostname 'smithi134' 2024-07-26T11:20:41.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.398+0000 7fa87d866740 1 --2- 172.21.15.134:0/2269460633 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x560a1decec80 0x560a1ded10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:41.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.398+0000 7fa87d866740 4 client.26246 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:20:41.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.398+0000 7fa87d866740 1 -- 172.21.15.134:0/2269460633 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560a1ddbd5c0 con 0x560a1decec80 2024-07-26T11:20:41.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.399+0000 7fa86f400640 1 --2- 172.21.15.134:0/2269460633 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x560a1decec80 0x560a1ded10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:41.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.399+0000 7fa86f400640 1 --2- 172.21.15.134:0/2269460633 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x560a1decec80 0x560a1ded10a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e88654f7f4533856 server_cookie=ddf4283299292be7 in_seq=0 out_seq=0 2024-07-26T11:20:41.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.399+0000 7fa859600640 10 client.26246 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:20:41.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.400+0000 7fa859600640 1 -- 172.21.15.134:0/2269460633 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560a1ddbd5c0 con 0x560a1decec80 2024-07-26T11:20:41.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.400+0000 7fa859600640 10 client.26246 handle_command_reply: tid=0 2024-07-26T11:20:41.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.400+0000 7fa87d866740 1 client.26246 shutdown 2024-07-26T11:20:41.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.401+0000 7fa87d866740 1 -- 172.21.15.134:0/2269460633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da629f0 msgr2=0x560a1de5ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:41.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.401+0000 7fa87d866740 1 --2- 172.21.15.134:0/2269460633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da629f0 0x560a1de5ccf0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa868012fd0 tx=0x7fa86801aa80 comp rx=0 tx=0).stop 2024-07-26T11:20:41.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.403+0000 7fa87d866740 1 -- 172.21.15.134:0/2269460633 shutdown_connections 2024-07-26T11:20:41.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.403+0000 7fa87d866740 1 --2- 172.21.15.134:0/2269460633 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x560a1decec80 0x560a1ded10a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.403+0000 7fa87d866740 1 --2- 172.21.15.134:0/2269460633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560a1daa4070 0x560a1de6b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.403+0000 7fa87d866740 1 --2- 172.21.15.134:0/2269460633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560a1da7b1b0 0x560a1de5d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.403+0000 7fa87d866740 1 --2- 172.21.15.134:0/2269460633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560a1da629f0 0x560a1de5ccf0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.403+0000 7fa859600640 0 client.26246 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:20:41.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.403+0000 7fa87d866740 1 -- 172.21.15.134:0/2269460633 >> 172.21.15.134:0/2269460633 conn(0x560a1da98460 msgr2=0x560a1de486e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:41.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.404+0000 7fa87d866740 1 -- 172.21.15.134:0/2269460633 shutdown_connections 2024-07-26T11:20:41.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.404+0000 7fa87d866740 1 -- 172.21.15.134:0/2269460633 wait complete. 2024-07-26T11:20:41.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.404+0000 7fa87d866740 20 client.26246 trim_cache size 0 max 16384 2024-07-26T11:20:41.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.405+0000 7fa875600640 1 -- 172.21.15.134:0/914661767 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa8540b3df0 msgr2=0x7fa8540b6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:41.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.405+0000 7fa875600640 1 --2- 172.21.15.134:0/914661767 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa8540b3df0 0x7fa8540b6300 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa868005f10 tx=0x7fa868005bd0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.405+0000 7fa875600640 1 -- 172.21.15.134:0/914661767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa870102930 msgr2=0x7fa87010f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:41.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.405+0000 7fa875600640 1 --2- 172.21.15.134:0/914661767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa870102930 0x7fa87010f840 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa85c009230 tx=0x7fa85c003c00 comp rx=0 tx=0).stop 2024-07-26T11:20:41.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.406+0000 7fa875600640 1 -- 172.21.15.134:0/914661767 shutdown_connections 2024-07-26T11:20:41.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.406+0000 7fa875600640 1 --2- 172.21.15.134:0/914661767 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa8540b3df0 0x7fa8540b6300 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.406+0000 7fa875600640 1 --2- 172.21.15.134:0/914661767 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa870107420 0x7fa87010fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.406+0000 7fa875600640 1 --2- 172.21.15.134:0/914661767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa870102930 0x7fa87010f840 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.406+0000 7fa875600640 1 --2- 172.21.15.134:0/914661767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa870101fe0 0x7fa870116570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:41.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.406+0000 7fa875600640 1 -- 172.21.15.134:0/914661767 >> 172.21.15.134:0/914661767 conn(0x7fa8700fdcf0 msgr2=0x7fa870104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:41.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.407+0000 7fa875600640 1 -- 172.21.15.134:0/914661767 shutdown_connections 2024-07-26T11:20:41.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:40.407+0000 7fa875600640 1 -- 172.21.15.134:0/914661767 wait complete. 2024-07-26T11:20:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:42 smithi193 ceph-mon[37473]: pgmap v622: 129 pgs: 129 active+clean; 31 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 228 KiB/s rd, 232 MiB/s wr, 1.90k op/s 2024-07-26T11:20:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:42 smithi134 ceph-mon[31966]: pgmap v622: 129 pgs: 129 active+clean; 31 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 228 KiB/s rd, 232 MiB/s wr, 1.90k op/s 2024-07-26T11:20:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:42 smithi189 ceph-mon[36790]: pgmap v622: 129 pgs: 129 active+clean; 31 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 228 KiB/s rd, 232 MiB/s wr, 1.90k op/s 2024-07-26T11:20:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:44 smithi189 ceph-mon[36790]: pgmap v623: 129 pgs: 129 active+clean; 31 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 242 KiB/s rd, 227 MiB/s wr, 1.76k op/s 2024-07-26T11:20:44.772 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:44 smithi134 ceph-mon[31966]: pgmap v623: 129 pgs: 129 active+clean; 31 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 242 KiB/s rd, 227 MiB/s wr, 1.76k op/s 2024-07-26T11:20:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:44 smithi193 ceph-mon[37473]: pgmap v623: 129 pgs: 129 active+clean; 31 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 242 KiB/s rd, 227 MiB/s wr, 1.76k op/s 2024-07-26T11:20:45.348 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-26T11:20:46.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:20:46.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:20:46.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:20:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:20:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:20:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:20:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:20:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:20:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:20:46.580 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:20:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:47 smithi134 ceph-mon[31966]: pgmap v624: 129 pgs: 129 active+clean; 31 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 80 KiB/s rd, 232 MiB/s wr, 1.75k op/s 2024-07-26T11:20:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:47 smithi189 ceph-mon[36790]: pgmap v624: 129 pgs: 129 active+clean; 31 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 80 KiB/s rd, 232 MiB/s wr, 1.75k op/s 2024-07-26T11:20:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:47 smithi193 ceph-mon[37473]: pgmap v624: 129 pgs: 129 active+clean; 31 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 80 KiB/s rd, 232 MiB/s wr, 1.75k op/s 2024-07-26T11:20:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:48 smithi189 ceph-mon[36790]: pgmap v625: 129 pgs: 129 active+clean; 32 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 90 KiB/s rd, 235 MiB/s wr, 1.72k op/s 2024-07-26T11:20:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:48 smithi193 ceph-mon[37473]: pgmap v625: 129 pgs: 129 active+clean; 32 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 90 KiB/s rd, 235 MiB/s wr, 1.72k op/s 2024-07-26T11:20:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:48 smithi134 ceph-mon[31966]: pgmap v625: 129 pgs: 129 active+clean; 32 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 90 KiB/s rd, 235 MiB/s wr, 1.72k op/s 2024-07-26T11:20:50.680 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:50 smithi134 ceph-mon[31966]: pgmap v626: 129 pgs: 129 active+clean; 32 GiB data, 67 GiB used, 1005 GiB / 1.0 TiB avail; 88 KiB/s rd, 251 MiB/s wr, 1.73k op/s 2024-07-26T11:20:50.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:50 smithi189 ceph-mon[36790]: pgmap v626: 129 pgs: 129 active+clean; 32 GiB data, 67 GiB used, 1005 GiB / 1.0 TiB avail; 88 KiB/s rd, 251 MiB/s wr, 1.73k op/s 2024-07-26T11:20:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:50 smithi193 ceph-mon[37473]: pgmap v626: 129 pgs: 129 active+clean; 32 GiB data, 67 GiB used, 1005 GiB / 1.0 TiB avail; 88 KiB/s rd, 251 MiB/s wr, 1.73k op/s 2024-07-26T11:20:51.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.873+0000 7ff885200640 1 -- 172.21.15.134:0/1968413143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff880113fb0 msgr2=0x7ff8801163d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:51.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.873+0000 7ff885200640 1 --2- 172.21.15.134:0/1968413143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff880113fb0 0x7ff8801163d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff878009900 tx=0x7ff87802a420 comp rx=0 tx=0).stop 2024-07-26T11:20:51.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.875+0000 7ff885200640 1 -- 172.21.15.134:0/1968413143 shutdown_connections 2024-07-26T11:20:51.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.875+0000 7ff885200640 1 --2- 172.21.15.134:0/1968413143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff880113fb0 0x7ff8801163d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.875+0000 7ff885200640 1 --2- 172.21.15.134:0/1968413143 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff880077cc0 0x7ff880113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.875+0000 7ff885200640 1 --2- 172.21.15.134:0/1968413143 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff880077370 0x7ff880077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.875+0000 7ff885200640 1 -- 172.21.15.134:0/1968413143 >> 172.21.15.134:0/1968413143 conn(0x7ff880101060 msgr2=0x7ff8801034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:51.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.877+0000 7ff885200640 1 -- 172.21.15.134:0/1968413143 shutdown_connections 2024-07-26T11:20:51.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.877+0000 7ff885200640 1 -- 172.21.15.134:0/1968413143 wait complete. 2024-07-26T11:20:51.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.879+0000 7ff885200640 1 Processor -- start 2024-07-26T11:20:51.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.879+0000 7ff885200640 1 -- start start 2024-07-26T11:20:51.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.880+0000 7ff885200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff880077370 0x7ff8801137a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.880+0000 7ff885200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff880077cc0 0x7ff88010ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.881+0000 7ff885200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff880113fb0 0x7ff88010cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.881+0000 7ff885200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff880119970 con 0x7ff880077370 2024-07-26T11:20:51.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.881+0000 7ff885200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801197f0 con 0x7ff880077cc0 2024-07-26T11:20:51.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.881+0000 7ff885200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff880119af0 con 0x7ff880113fb0 2024-07-26T11:20:51.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.882+0000 7ff87ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff880077cc0 0x7ff88010ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.882+0000 7ff87fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff880113fb0 0x7ff88010cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.882+0000 7ff87fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff880113fb0 0x7ff88010cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:33364/0 (socket says 172.21.15.134:33364) 2024-07-26T11:20:51.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.882+0000 7ff87fe00640 1 -- 172.21.15.134:0/3915236105 learned_addr learned my addr 172.21.15.134:0/3915236105 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:51.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.883+0000 7ff87ea00640 1 -- 172.21.15.134:0/3915236105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff880113fb0 msgr2=0x7ff88010cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:51.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.883+0000 7ff87ea00640 1 --2- 172.21.15.134:0/3915236105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff880113fb0 0x7ff88010cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.883+0000 7ff87ea00640 1 -- 172.21.15.134:0/3915236105 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff880077370 msgr2=0x7ff8801137a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-07-26T11:20:51.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.884+0000 7ff87ea00640 1 --2- 172.21.15.134:0/3915236105 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff880077370 0x7ff8801137a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.886+0000 7ff87ea00640 1 -- 172.21.15.134:0/3915236105 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88010d7b0 con 0x7ff880077cc0 2024-07-26T11:20:51.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.888+0000 7ff87ea00640 1 --2- 172.21.15.134:0/3915236105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff880077cc0 0x7ff88010ca70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff88006b360 tx=0x7ff86800be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:51.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.888+0000 7ff877e00640 1 -- 172.21.15.134:0/3915236105 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868019070 con 0x7ff880077cc0 2024-07-26T11:20:51.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.888+0000 7ff877e00640 1 -- 172.21.15.134:0/3915236105 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8680148f0 con 0x7ff880077cc0 2024-07-26T11:20:51.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.888+0000 7ff877e00640 1 -- 172.21.15.134:0/3915236105 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff868012880 con 0x7ff880077cc0 2024-07-26T11:20:51.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.889+0000 7ff885200640 1 -- 172.21.15.134:0/3915236105 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff880112470 con 0x7ff880077cc0 2024-07-26T11:20:51.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.889+0000 7ff885200640 1 -- 172.21.15.134:0/3915236105 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8801129e0 con 0x7ff880077cc0 2024-07-26T11:20:51.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.895+0000 7ff88d473740 1 Processor -- start 2024-07-26T11:20:51.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.895+0000 7ff88d473740 1 -- start start 2024-07-26T11:20:51.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.896+0000 7ff88d473740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a775a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.896+0000 7ff88d473740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a738c1b0 0x5615a775a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.896+0000 7ff87fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a775a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.896+0000 7ff87fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a775a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:40438/0 (socket says 172.21.15.134:40438) 2024-07-26T11:20:51.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.897+0000 7ff87fe00640 1 -- 172.21.15.134:0/2348999619 learned_addr learned my addr 172.21.15.134:0/2348999619 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:51.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.897+0000 7ff877e00640 1 -- 172.21.15.134:0/3915236105 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7ff86801c070 con 0x7ff880077cc0 2024-07-26T11:20:51.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.897+0000 7ff88d473740 1 --2- 172.21.15.134:0/2348999619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a7394390 0x5615a775aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.897+0000 7ff88d473740 1 -- 172.21.15.134:0/2348999619 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5615a76d0bf0 con 0x5615a7394390 2024-07-26T11:20:51.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.897+0000 7ff88d473740 1 -- 172.21.15.134:0/2348999619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5615a76c00f0 con 0x5615a73b5070 2024-07-26T11:20:51.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.897+0000 7ff88d473740 1 -- 172.21.15.134:0/2348999619 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5615a76c4130 con 0x5615a738c1b0 2024-07-26T11:20:51.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.898+0000 7ff87f400640 1 --2- 172.21.15.134:0/2348999619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a738c1b0 0x5615a775a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.898+0000 7ff87ea00640 1 --2- 172.21.15.134:0/2348999619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a7394390 0x5615a775aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.898+0000 7ff87fe00640 1 -- 172.21.15.134:0/2348999619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a738c1b0 msgr2=0x5615a775a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:51.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.898+0000 7ff87fe00640 1 --2- 172.21.15.134:0/2348999619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a738c1b0 0x5615a775a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.898+0000 7ff87fe00640 1 -- 172.21.15.134:0/2348999619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a7394390 msgr2=0x5615a775aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:51.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.898+0000 7ff87fe00640 1 --2- 172.21.15.134:0/2348999619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a7394390 0x5615a775aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.898+0000 7ff87fe00640 1 -- 172.21.15.134:0/2348999619 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615a7420650 con 0x5615a73b5070 2024-07-26T11:20:51.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.899+0000 7ff877e00640 1 --2- 172.21.15.134:0/3915236105 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8640b3ed0 0x7ff8640b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.902 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.899+0000 7ff877e00640 1 -- 172.21.15.134:0/3915236105 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff8680e3980 con 0x7ff880077cc0 2024-07-26T11:20:51.902 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.899+0000 7ff87fe00640 1 --2- 172.21.15.134:0/2348999619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a775a030 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff878004280 tx=0x7ff8780042b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=924ff5b9b5dd55be server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:51.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.900+0000 7ff87f400640 1 --2- 172.21.15.134:0/3915236105 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8640b3ed0 0x7ff8640b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.900+0000 7ff874c00640 1 -- 172.21.15.134:0/2348999619 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff878040070 con 0x5615a73b5070 2024-07-26T11:20:51.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.900+0000 7ff874c00640 1 -- 172.21.15.134:0/2348999619 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8780047a0 con 0x5615a73b5070 2024-07-26T11:20:51.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.900+0000 7ff874c00640 1 -- 172.21.15.134:0/2348999619 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff878035040 con 0x5615a73b5070 2024-07-26T11:20:51.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.901+0000 7ff88d473740 1 -- 172.21.15.134:0/2348999619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 msgr2=0x5615a775a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:51.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.901+0000 7ff88d473740 1 --2- 172.21.15.134:0/2348999619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a775a030 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff878004280 tx=0x7ff8780042b0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.901+0000 7ff88d473740 1 -- 172.21.15.134:0/2348999619 shutdown_connections 2024-07-26T11:20:51.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.901+0000 7ff88d473740 1 --2- 172.21.15.134:0/2348999619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a738c1b0 0x5615a775a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.901+0000 7ff88d473740 1 --2- 172.21.15.134:0/2348999619 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a7394390 0x5615a775aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.901+0000 7ff88d473740 1 --2- 172.21.15.134:0/2348999619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a775a030 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.901+0000 7ff88d473740 1 -- 172.21.15.134:0/2348999619 >> 172.21.15.134:0/2348999619 conn(0x5615a73a9460 msgr2=0x5615a7757f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:51.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.902+0000 7ff87f400640 1 --2- 172.21.15.134:0/3915236105 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8640b3ed0 0x7ff8640b63e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x5615a7758cb0 tx=0x7ff87000b390 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:51.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.902+0000 7ff88d473740 1 -- 172.21.15.134:0/2348999619 shutdown_connections 2024-07-26T11:20:51.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.902+0000 7ff88d473740 1 -- 172.21.15.134:0/2348999619 wait complete. 2024-07-26T11:20:51.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.903+0000 7ff88d473740 1 -- start start 2024-07-26T11:20:51.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.903+0000 7ff88d473740 1 Processor -- start 2024-07-26T11:20:51.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.904+0000 7ff88d473740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a738c1b0 0x5615a7765610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.904+0000 7ff88d473740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a7394390 0x5615a7765b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.904+0000 7ff88d473740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a73b5070 0x5615a7766090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.904+0000 7ff87f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a738c1b0 0x5615a7765610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.904+0000 7ff87f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a738c1b0 0x5615a7765610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:40454/0 (socket says 172.21.15.134:40454) 2024-07-26T11:20:51.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.905+0000 7ff87fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a73b5070 0x5615a7766090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.905+0000 7ff87fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a73b5070 0x5615a7766090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:53264/0 (socket says 172.21.15.134:53264) 2024-07-26T11:20:51.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.905+0000 7ff87f400640 1 -- 172.21.15.134:0/943378091 learned_addr learned my addr 172.21.15.134:0/943378091 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:51.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.905+0000 7ff88d473740 1 -- 172.21.15.134:0/943378091 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5615a7523700 con 0x5615a73b5070 2024-07-26T11:20:51.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.905+0000 7ff88d473740 1 -- 172.21.15.134:0/943378091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5615a765c6c0 con 0x5615a738c1b0 2024-07-26T11:20:51.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.905+0000 7ff88d473740 1 -- 172.21.15.134:0/943378091 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5615a77687b0 con 0x5615a7394390 2024-07-26T11:20:51.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.905+0000 7ff87ea00640 1 --2- 172.21.15.134:0/943378091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a7394390 0x5615a7765b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.905+0000 7ff88d473740 10 client.0 _ll_register_callbacks cb 0x5615a766c170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:51.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.906+0000 7ff87f400640 1 -- 172.21.15.134:0/943378091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a7394390 msgr2=0x5615a7765b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:51.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.906+0000 7ff87f400640 1 --2- 172.21.15.134:0/943378091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a7394390 0x5615a7765b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.906+0000 7ff87f400640 1 -- 172.21.15.134:0/943378091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a73b5070 msgr2=0x5615a7766090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:51.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.906+0000 7ff87f400640 1 --2- 172.21.15.134:0/943378091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a73b5070 0x5615a7766090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.906+0000 7ff87f400640 1 -- 172.21.15.134:0/943378091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615a7673860 con 0x5615a738c1b0 2024-07-26T11:20:51.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.906+0000 7ff87fe00640 1 --2- 172.21.15.134:0/943378091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a73b5070 0x5615a7766090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:20:51.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.906+0000 7ff87f400640 1 --2- 172.21.15.134:0/943378091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a738c1b0 0x5615a7765610 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff870004150 tx=0x7ff8700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b84695da6318eadf server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:51.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.906+0000 7ff86d600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:20:51.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.906+0000 7ff86d600640 1 -- 172.21.15.134:0/943378091 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87001e620 con 0x5615a738c1b0 2024-07-26T11:20:51.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.906+0000 7ff86d600640 1 -- 172.21.15.134:0/943378091 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff87001e7c0 con 0x5615a738c1b0 2024-07-26T11:20:51.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.907+0000 7ff88d473740 1 -- 172.21.15.134:0/943378091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8780047a0 con 0x5615a738c1b0 2024-07-26T11:20:51.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.907+0000 7ff86d600640 1 -- 172.21.15.134:0/943378091 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff87001ea00 con 0x5615a738c1b0 2024-07-26T11:20:51.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.907+0000 7ff88d473740 10 client.26258 fetch_fsmap learned FSMap version 32 2024-07-26T11:20:51.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.908+0000 7ff88d473740 1 -- 172.21.15.134:0/943378091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5615a7774470 con 0x5615a738c1b0 2024-07-26T11:20:51.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.909+0000 7ff86d600640 1 -- 172.21.15.134:0/943378091 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8780047a0 con 0x5615a738c1b0 2024-07-26T11:20:51.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.909+0000 7ff86d600640 1 -- 172.21.15.134:0/943378091 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7ff870026c00 con 0x5615a738c1b0 2024-07-26T11:20:51.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.909+0000 7ff88d473740 10 client.26258 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:20:51.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.909+0000 7ff88d473740 10 client.26258 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-07-26T11:20:51.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.909+0000 7ff88d473740 20 client.26258 populate_metadata read hostname 'smithi134' 2024-07-26T11:20:51.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.909+0000 7ff88d473740 1 --2- 172.21.15.134:0/943378091 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5615a73739f0 0x5615a7777a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.909+0000 7ff88d473740 4 client.26258 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:20:51.912 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.909+0000 7ff88d473740 1 -- 172.21.15.134:0/943378091 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615a76d2b40 con 0x5615a73739f0 2024-07-26T11:20:51.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.910+0000 7ff87ea00640 1 --2- 172.21.15.134:0/943378091 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5615a73739f0 0x5615a7777a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.911+0000 7ff86d600640 10 client.26258 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:20:51.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.911+0000 7ff87ea00640 1 --2- 172.21.15.134:0/943378091 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5615a73739f0 0x5615a7777a20 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=27dadf2785766419 server_cookie=9a9a947e4aae1232 in_seq=0 out_seq=0 2024-07-26T11:20:51.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.913+0000 7ff86d600640 1 -- 172.21.15.134:0/943378091 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5615a76d2b40 con 0x5615a73739f0 2024-07-26T11:20:51.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.913+0000 7ff86d600640 10 client.26258 handle_command_reply: tid=0 2024-07-26T11:20:51.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.913+0000 7ff88d473740 1 client.26258 shutdown 2024-07-26T11:20:51.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.913+0000 7ff88d473740 1 -- 172.21.15.134:0/943378091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a738c1b0 msgr2=0x5615a7765610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:51.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.913+0000 7ff88d473740 1 --2- 172.21.15.134:0/943378091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a738c1b0 0x5615a7765610 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff870004150 tx=0x7ff8700046c0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.915+0000 7ff88d473740 1 -- 172.21.15.134:0/943378091 shutdown_connections 2024-07-26T11:20:51.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.915+0000 7ff88d473740 1 --2- 172.21.15.134:0/943378091 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5615a73739f0 0x5615a7777a20 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.915+0000 7ff88d473740 1 --2- 172.21.15.134:0/943378091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a7394390 0x5615a7765b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.915+0000 7ff88d473740 1 --2- 172.21.15.134:0/943378091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a73b5070 0x5615a7766090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.915+0000 7ff88d473740 1 --2- 172.21.15.134:0/943378091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a738c1b0 0x5615a7765610 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.915+0000 7ff86d600640 0 client.26258 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:20:51.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.915+0000 7ff88d473740 1 -- 172.21.15.134:0/943378091 >> 172.21.15.134:0/943378091 conn(0x5615a73a9460 msgr2=0x5615a775d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:51.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.915+0000 7ff88d473740 1 -- 172.21.15.134:0/943378091 shutdown_connections 2024-07-26T11:20:51.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.915+0000 7ff88d473740 1 -- 172.21.15.134:0/943378091 wait complete. 2024-07-26T11:20:51.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.915+0000 7ff88d473740 20 client.26258 trim_cache size 0 max 16384 2024-07-26T11:20:51.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.937+0000 7ff88d473740 1 Processor -- start 2024-07-26T11:20:51.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.937+0000 7ff88d473740 1 -- start start 2024-07-26T11:20:51.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.938+0000 7ff88d473740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a738c1b0 0x5615a7767600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.938+0000 7ff88d473740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a7394390 0x5615a775ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.938+0000 7ff88d473740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a77609c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.938+0000 7ff88d473740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801bf5b0 con 0x5615a7394390 2024-07-26T11:20:51.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.938+0000 7ff88d473740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801c10f0 con 0x5615a73b5070 2024-07-26T11:20:51.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.938+0000 7ff88d473740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801bfc80 con 0x5615a738c1b0 2024-07-26T11:20:51.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.938+0000 7ff87f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a738c1b0 0x5615a7767600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.939+0000 7ff87f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a738c1b0 0x5615a7767600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:33400/0 (socket says 172.21.15.134:33400) 2024-07-26T11:20:51.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.939+0000 7ff87f400640 1 -- 172.21.15.134:0/1240440124 learned_addr learned my addr 172.21.15.134:0/1240440124 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:51.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.939+0000 7ff87fe00640 1 --2- 172.21.15.134:0/1240440124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a77609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.939+0000 7ff87ea00640 1 --2- 172.21.15.134:0/1240440124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a7394390 0x5615a775ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.939+0000 7ff87fe00640 1 -- 172.21.15.134:0/1240440124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a738c1b0 msgr2=0x5615a7767600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:51.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.939+0000 7ff87fe00640 1 --2- 172.21.15.134:0/1240440124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a738c1b0 0x5615a7767600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.939+0000 7ff87fe00640 1 -- 172.21.15.134:0/1240440124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a7394390 msgr2=0x5615a775ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:51.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.940+0000 7ff87fe00640 1 --2- 172.21.15.134:0/1240440124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a7394390 0x5615a775ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.940+0000 7ff87fe00640 1 -- 172.21.15.134:0/1240440124 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615a775c550 con 0x5615a73b5070 2024-07-26T11:20:51.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.940+0000 7ff87f400640 1 --2- 172.21.15.134:0/1240440124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a738c1b0 0x5615a7767600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:20:51.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.940+0000 7ff87ea00640 1 --2- 172.21.15.134:0/1240440124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a7394390 0x5615a775ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:20:51.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.940+0000 7ff87fe00640 1 --2- 172.21.15.134:0/1240440124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a77609c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff878009850 tx=0x7ff878000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3432413c4de989d4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:51.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.941+0000 7ff86fe00640 1 -- 172.21.15.134:0/1240440124 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff878035070 con 0x5615a73b5070 2024-07-26T11:20:51.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.941+0000 7ff86fe00640 1 -- 172.21.15.134:0/1240440124 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8780055b0 con 0x5615a73b5070 2024-07-26T11:20:51.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.941+0000 7ff86fe00640 1 -- 172.21.15.134:0/1240440124 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff878040040 con 0x5615a73b5070 2024-07-26T11:20:51.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.943+0000 7ff88d473740 1 -- 172.21.15.134:0/1240440124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 msgr2=0x5615a77609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:51.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.943+0000 7ff88d473740 1 --2- 172.21.15.134:0/1240440124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a77609c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff878009850 tx=0x7ff878000bc0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.943+0000 7ff88d473740 1 -- 172.21.15.134:0/1240440124 shutdown_connections 2024-07-26T11:20:51.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.943+0000 7ff88d473740 1 --2- 172.21.15.134:0/1240440124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a77609c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.943+0000 7ff88d473740 1 --2- 172.21.15.134:0/1240440124 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a7394390 0x5615a775ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.943+0000 7ff88d473740 1 --2- 172.21.15.134:0/1240440124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a738c1b0 0x5615a7767600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.943+0000 7ff88d473740 1 -- 172.21.15.134:0/1240440124 >> 172.21.15.134:0/1240440124 conn(0x5615a73a9460 msgr2=0x5615a77602b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:51.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.943+0000 7ff88d473740 1 -- 172.21.15.134:0/1240440124 shutdown_connections 2024-07-26T11:20:51.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.944+0000 7ff88d473740 1 -- 172.21.15.134:0/1240440124 wait complete. 2024-07-26T11:20:51.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.945+0000 7ff88d473740 1 -- start start 2024-07-26T11:20:51.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.945+0000 7ff88d473740 1 Processor -- start 2024-07-26T11:20:51.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.946+0000 7ff88d473740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a738c1b0 0x5615a7769f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.946+0000 7ff88d473740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a7394390 0x5615a776a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.946+0000 7ff88d473740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a77df1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.946+0000 7ff88d473740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801c0a20 con 0x5615a738c1b0 2024-07-26T11:20:51.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.946+0000 7ff88d473740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801c0350 con 0x5615a73b5070 2024-07-26T11:20:51.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.946+0000 7ff88d473740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff8801bd590 con 0x5615a7394390 2024-07-26T11:20:51.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.946+0000 7ff88d473740 10 client.0 _ll_register_callbacks cb 0x7ff870003a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:51.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.946+0000 7ff87fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a77df1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.946+0000 7ff87fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a77df1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:40466/0 (socket says 172.21.15.134:40466) 2024-07-26T11:20:51.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.946+0000 7ff87fe00640 1 -- 172.21.15.134:0/1628263053 learned_addr learned my addr 172.21.15.134:0/1628263053 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:51.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.946+0000 7ff87ea00640 1 --2- 172.21.15.134:0/1628263053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a7394390 0x5615a776a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.947+0000 7ff87f400640 1 --2- 172.21.15.134:0/1628263053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a738c1b0 0x5615a7769f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.947+0000 7ff87fe00640 1 -- 172.21.15.134:0/1628263053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a7394390 msgr2=0x5615a776a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:51.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.947+0000 7ff87fe00640 1 --2- 172.21.15.134:0/1628263053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a7394390 0x5615a776a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.947+0000 7ff87fe00640 1 -- 172.21.15.134:0/1628263053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a738c1b0 msgr2=0x5615a7769f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:51.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.947+0000 7ff87fe00640 1 --2- 172.21.15.134:0/1628263053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a738c1b0 0x5615a7769f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.947+0000 7ff87fe00640 1 -- 172.21.15.134:0/1628263053 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615a77df730 con 0x5615a73b5070 2024-07-26T11:20:51.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.947+0000 7ff87fe00640 1 --2- 172.21.15.134:0/1628263053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a77df1f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff878009340 tx=0x7ff878005900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fb020dec9be0cce server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:51.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.947+0000 7ff86d600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:20:51.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.948+0000 7ff86d600640 1 -- 172.21.15.134:0/1628263053 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff87805a060 con 0x5615a73b5070 2024-07-26T11:20:51.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.948+0000 7ff86d600640 1 -- 172.21.15.134:0/1628263053 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7ff878004d60 con 0x5615a73b5070 2024-07-26T11:20:51.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.948+0000 7ff88d473740 1 -- 172.21.15.134:0/1628263053 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8780055b0 con 0x5615a73b5070 2024-07-26T11:20:51.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.949+0000 7ff86d600640 1 -- 172.21.15.134:0/1628263053 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff878061050 con 0x5615a73b5070 2024-07-26T11:20:51.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.951+0000 7ff88d473740 10 client.26264 fetch_fsmap learned FSMap version 32 2024-07-26T11:20:51.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.951+0000 7ff88d473740 1 -- 172.21.15.134:0/1628263053 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5615a77e08f0 con 0x5615a73b5070 2024-07-26T11:20:51.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.951+0000 7ff86d600640 1 -- 172.21.15.134:0/1628263053 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8780055b0 con 0x5615a73b5070 2024-07-26T11:20:51.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.951+0000 7ff86d600640 1 -- 172.21.15.134:0/1628263053 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7ff87803c440 con 0x5615a73b5070 2024-07-26T11:20:51.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.951+0000 7ff88d473740 10 client.26264 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:20:51.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.951+0000 7ff88d473740 10 client.26264 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-07-26T11:20:51.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.952+0000 7ff88d473740 20 client.26264 populate_metadata read hostname 'smithi134' 2024-07-26T11:20:51.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.952+0000 7ff88d473740 1 --2- 172.21.15.134:0/1628263053 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5615a7348da0 0x5615a77e2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:51.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.952+0000 7ff88d473740 4 client.26264 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:20:51.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.952+0000 7ff88d473740 1 -- 172.21.15.134:0/1628263053 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5615a76ce5e0 con 0x5615a7348da0 2024-07-26T11:20:51.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.952+0000 7ff87f400640 1 --2- 172.21.15.134:0/1628263053 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5615a7348da0 0x5615a77e2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:51.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.953+0000 7ff86d600640 10 client.26264 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:20:51.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.953+0000 7ff87f400640 1 --2- 172.21.15.134:0/1628263053 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5615a7348da0 0x5615a77e2b80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5d6f2e9d66ff7877 server_cookie=c68f0354140e4e71 in_seq=0 out_seq=0 2024-07-26T11:20:51.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.953+0000 7ff86d600640 1 -- 172.21.15.134:0/1628263053 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5615a76ce5e0 con 0x5615a7348da0 2024-07-26T11:20:51.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.953+0000 7ff86d600640 10 client.26264 handle_command_reply: tid=0 2024-07-26T11:20:51.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.953+0000 7ff88d473740 1 client.26264 shutdown 2024-07-26T11:20:51.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.955+0000 7ff88d473740 1 -- 172.21.15.134:0/1628263053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 msgr2=0x5615a77df1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:51.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.955+0000 7ff88d473740 1 --2- 172.21.15.134:0/1628263053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a77df1f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff878009340 tx=0x7ff878005900 comp rx=0 tx=0).stop 2024-07-26T11:20:51.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.955+0000 7ff88d473740 1 -- 172.21.15.134:0/1628263053 shutdown_connections 2024-07-26T11:20:51.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.955+0000 7ff88d473740 1 --2- 172.21.15.134:0/1628263053 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x5615a7348da0 0x5615a77e2b80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.955+0000 7ff88d473740 1 --2- 172.21.15.134:0/1628263053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5615a7394390 0x5615a776a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.955+0000 7ff88d473740 1 --2- 172.21.15.134:0/1628263053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615a73b5070 0x5615a77df1f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.955+0000 7ff88d473740 1 --2- 172.21.15.134:0/1628263053 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5615a738c1b0 0x5615a7769f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.955+0000 7ff88d473740 1 -- 172.21.15.134:0/1628263053 >> 172.21.15.134:0/1628263053 conn(0x5615a73a9460 msgr2=0x5615a775d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:51.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.955+0000 7ff86d600640 0 client.26264 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:20:51.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.955+0000 7ff88d473740 1 -- 172.21.15.134:0/1628263053 shutdown_connections 2024-07-26T11:20:51.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.955+0000 7ff88d473740 1 -- 172.21.15.134:0/1628263053 wait complete. 2024-07-26T11:20:51.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.956+0000 7ff88d473740 20 client.26264 trim_cache size 0 max 16384 2024-07-26T11:20:51.959 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:20:51.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.957+0000 7ff885200640 1 -- 172.21.15.134:0/3915236105 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8640b3ed0 msgr2=0x7ff8640b63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:51.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.957+0000 7ff885200640 1 --2- 172.21.15.134:0/3915236105 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8640b3ed0 0x7ff8640b63e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x5615a7758cb0 tx=0x7ff87000b390 comp rx=0 tx=0).stop 2024-07-26T11:20:51.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.957+0000 7ff885200640 1 -- 172.21.15.134:0/3915236105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff880077cc0 msgr2=0x7ff88010ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:51.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.957+0000 7ff885200640 1 --2- 172.21.15.134:0/3915236105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff880077cc0 0x7ff88010ca70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff88006b360 tx=0x7ff86800be00 comp rx=0 tx=0).stop 2024-07-26T11:20:51.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.958+0000 7ff885200640 1 -- 172.21.15.134:0/3915236105 shutdown_connections 2024-07-26T11:20:51.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.958+0000 7ff885200640 1 --2- 172.21.15.134:0/3915236105 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8640b3ed0 0x7ff8640b63e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.958+0000 7ff885200640 1 --2- 172.21.15.134:0/3915236105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff880113fb0 0x7ff88010cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.958+0000 7ff885200640 1 --2- 172.21.15.134:0/3915236105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff880077cc0 0x7ff88010ca70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.958+0000 7ff885200640 1 --2- 172.21.15.134:0/3915236105 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff880077370 0x7ff8801137a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:51.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.958+0000 7ff885200640 1 -- 172.21.15.134:0/3915236105 >> 172.21.15.134:0/3915236105 conn(0x7ff880101060 msgr2=0x7ff880115fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:51.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.958+0000 7ff885200640 1 -- 172.21.15.134:0/3915236105 shutdown_connections 2024-07-26T11:20:51.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:20:51.958+0000 7ff885200640 1 -- 172.21.15.134:0/3915236105 wait complete. 2024-07-26T11:20:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:52 smithi134 ceph-mon[31966]: pgmap v627: 129 pgs: 129 active+clean; 33 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 40 KiB/s rd, 232 MiB/s wr, 1.48k op/s 2024-07-26T11:20:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:52 smithi189 ceph-mon[36790]: pgmap v627: 129 pgs: 129 active+clean; 33 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 40 KiB/s rd, 232 MiB/s wr, 1.48k op/s 2024-07-26T11:20:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:52 smithi193 ceph-mon[37473]: pgmap v627: 129 pgs: 129 active+clean; 33 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 40 KiB/s rd, 232 MiB/s wr, 1.48k op/s 2024-07-26T11:20:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:54 smithi189 ceph-mon[36790]: pgmap v628: 129 pgs: 129 active+clean; 33 GiB data, 69 GiB used, 1004 GiB / 1.0 TiB avail; 38 KiB/s rd, 231 MiB/s wr, 1.33k op/s 2024-07-26T11:20:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:54 smithi193 ceph-mon[37473]: pgmap v628: 129 pgs: 129 active+clean; 33 GiB data, 69 GiB used, 1004 GiB / 1.0 TiB avail; 38 KiB/s rd, 231 MiB/s wr, 1.33k op/s 2024-07-26T11:20:54.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:54 smithi134 ceph-mon[31966]: pgmap v628: 129 pgs: 129 active+clean; 33 GiB data, 69 GiB used, 1004 GiB / 1.0 TiB avail; 38 KiB/s rd, 231 MiB/s wr, 1.33k op/s 2024-07-26T11:20:55.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.064+0000 7f705fa00640 1 -- 172.21.15.134:0/3006033020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f70500a0540 msgr2=0x7f70500a2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:55.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.065+0000 7f705fa00640 1 --2- 172.21.15.134:0/3006033020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f70500a0540 0x7f70500a2960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7058051ee0 tx=0x7f7058097bf0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.070+0000 7f705fa00640 1 -- 172.21.15.134:0/3006033020 shutdown_connections 2024-07-26T11:20:55.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.070+0000 7f705fa00640 1 --2- 172.21.15.134:0/3006033020 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f70500a0540 0x7f70500a2960 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.070+0000 7f705fa00640 1 --2- 172.21.15.134:0/3006033020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f705009dbe0 0x7f70500a0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.070+0000 7f705fa00640 1 --2- 172.21.15.134:0/3006033020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7050094280 0x7f7050094690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.070+0000 7f705fa00640 1 -- 172.21.15.134:0/3006033020 >> 172.21.15.134:0/3006033020 conn(0x7f705000c0b0 msgr2=0x7f705000c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.073+0000 7f705fa00640 1 -- 172.21.15.134:0/3006033020 shutdown_connections 2024-07-26T11:20:55.075 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.074+0000 7f705fa00640 1 -- 172.21.15.134:0/3006033020 wait complete. 2024-07-26T11:20:55.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.076+0000 7f705fa00640 1 Processor -- start 2024-07-26T11:20:55.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.078+0000 7f705fa00640 1 -- start start 2024-07-26T11:20:55.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.081+0000 7f705fa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7050094280 0x7f705013b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.083 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.081+0000 7f705fa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f705009dbe0 0x7f705013bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.083+0000 7f705d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7050094280 0x7f705013b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.083+0000 7f705d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7050094280 0x7f705013b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:33430/0 (socket says 172.21.15.134:33430) 2024-07-26T11:20:55.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.083+0000 7f705d000640 1 -- 172.21.15.134:0/155428064 learned_addr learned my addr 172.21.15.134:0/155428064 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:55.086 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.083+0000 7f705fa00640 1 --2- 172.21.15.134:0/155428064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70500a0540 0x7f7050141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.083+0000 7f705fa00640 1 -- 172.21.15.134:0/155428064 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f70500a3e50 con 0x7f705009dbe0 2024-07-26T11:20:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.083+0000 7f705fa00640 1 -- 172.21.15.134:0/155428064 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f70500a3cd0 con 0x7f70500a0540 2024-07-26T11:20:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.083+0000 7f705fa00640 1 -- 172.21.15.134:0/155428064 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f70500a3fd0 con 0x7f7050094280 2024-07-26T11:20:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.083+0000 7f705da00640 1 --2- 172.21.15.134:0/155428064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70500a0540 0x7f7050141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.084+0000 7f705da00640 1 -- 172.21.15.134:0/155428064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7050094280 msgr2=0x7f705013b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.084+0000 7f705da00640 1 --2- 172.21.15.134:0/155428064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7050094280 0x7f705013b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.084+0000 7f705da00640 1 -- 172.21.15.134:0/155428064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f705009dbe0 msgr2=0x7f705013bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.084+0000 7f705da00640 1 --2- 172.21.15.134:0/155428064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f705009dbe0 0x7f705013bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.084+0000 7f705da00640 1 -- 172.21.15.134:0/155428064 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7050141850 con 0x7f70500a0540 2024-07-26T11:20:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.084+0000 7f705d000640 1 --2- 172.21.15.134:0/155428064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7050094280 0x7f705013b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:20:55.087 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.084+0000 7f705da00640 1 --2- 172.21.15.134:0/155428064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70500a0540 0x7f7050141150 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7058051740 tx=0x7f7058069d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:55.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.085+0000 7f7055600640 1 -- 172.21.15.134:0/155428064 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70580b1070 con 0x7f70500a0540 2024-07-26T11:20:55.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.085+0000 7f705fa00640 1 -- 172.21.15.134:0/155428064 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7050141ae0 con 0x7f70500a0540 2024-07-26T11:20:55.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.085+0000 7f705fa00640 1 -- 172.21.15.134:0/155428064 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70501482d0 con 0x7f70500a0540 2024-07-26T11:20:55.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.088+0000 7f7055600640 1 -- 172.21.15.134:0/155428064 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f705806b520 con 0x7f70500a0540 2024-07-26T11:20:55.090 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.088+0000 7f7055600640 1 -- 172.21.15.134:0/155428064 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70580b1070 con 0x7f70500a0540 2024-07-26T11:20:55.095 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.093+0000 7f7055600640 1 -- 172.21.15.134:0/155428064 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f70580b5430 con 0x7f70500a0540 2024-07-26T11:20:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.094+0000 7f7066b9a740 1 Processor -- start 2024-07-26T11:20:55.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.094+0000 7f7066b9a740 1 -- start start 2024-07-26T11:20:55.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.095+0000 7f7066b9a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7821b0 0x55872db25c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.095+0000 7f7066b9a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 0x55872db26180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.095+0000 7f7057e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7821b0 0x55872db25c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:55.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.096+0000 7f705d000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 0x55872db26180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.096+0000 7f7057e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7821b0 0x55872db25c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:33444/0 (socket says 172.21.15.134:33444) 2024-07-26T11:20:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.096+0000 7f7066b9a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7614d0 0x55872db266c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.096+0000 7f7066b9a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55872da9a270 con 0x55872d7592f0 2024-07-26T11:20:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.096+0000 7f7066b9a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55872da89770 con 0x55872d7614d0 2024-07-26T11:20:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.096+0000 7f7066b9a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55872da8d7b0 con 0x55872d7821b0 2024-07-26T11:20:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.096+0000 7f7055600640 1 --2- 172.21.15.134:0/155428064 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f70400b4180 0x7f70400b6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.096+0000 7f705d000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 0x55872db26180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:53316/0 (socket says 172.21.15.134:53316) 2024-07-26T11:20:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.096+0000 7f705d000640 1 -- 172.21.15.134:0/2605160368 learned_addr learned my addr 172.21.15.134:0/2605160368 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.096+0000 7f7055600640 1 -- 172.21.15.134:0/155428064 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f70580a8030 con 0x7f70500a0540 2024-07-26T11:20:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.097+0000 7f705d000640 1 -- 172.21.15.134:0/2605160368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7821b0 msgr2=0x55872db25c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.097+0000 7f705d000640 1 --2- 172.21.15.134:0/2605160368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7821b0 0x55872db25c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.097+0000 7f705d000640 1 -- 172.21.15.134:0/2605160368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7614d0 msgr2=0x55872db266c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:20:55.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.097+0000 7f705d000640 1 --2- 172.21.15.134:0/2605160368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7614d0 0x55872db266c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.097+0000 7f705d000640 1 -- 172.21.15.134:0/2605160368 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55872d7ed790 con 0x55872d7592f0 2024-07-26T11:20:55.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.097+0000 7f7057e00640 1 --2- 172.21.15.134:0/2605160368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7821b0 0x55872db25c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:20:55.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.097+0000 7f705da00640 1 --2- 172.21.15.134:0/155428064 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f70400b4180 0x7f70400b6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:55.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.097+0000 7f705d000640 1 --2- 172.21.15.134:0/2605160368 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 0x55872db26180 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f704c007930 tx=0x7f704c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8a8279eace8538 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:55.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.098+0000 7f704a000640 1 -- 172.21.15.134:0/2605160368 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704c00da10 con 0x55872d7592f0 2024-07-26T11:20:55.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.098+0000 7f705da00640 1 --2- 172.21.15.134:0/155428064 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f70400b4180 0x7f70400b6690 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7058068b30 tx=0x7f70580694b0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:55.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.098+0000 7f704a000640 1 -- 172.21.15.134:0/2605160368 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f704c018070 con 0x55872d7592f0 2024-07-26T11:20:55.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.098+0000 7f704a000640 1 -- 172.21.15.134:0/2605160368 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704c0123e0 con 0x55872d7592f0 2024-07-26T11:20:55.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.100+0000 7f7066b9a740 1 -- 172.21.15.134:0/2605160368 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 msgr2=0x55872db26180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:55.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.100+0000 7f7066b9a740 1 --2- 172.21.15.134:0/2605160368 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 0x55872db26180 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f704c007930 tx=0x7f704c007df0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.100+0000 7f7066b9a740 1 -- 172.21.15.134:0/2605160368 shutdown_connections 2024-07-26T11:20:55.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.100+0000 7f7066b9a740 1 --2- 172.21.15.134:0/2605160368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7614d0 0x55872db266c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.100+0000 7f7066b9a740 1 --2- 172.21.15.134:0/2605160368 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 0x55872db26180 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.100+0000 7f7066b9a740 1 --2- 172.21.15.134:0/2605160368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7821b0 0x55872db25c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.100+0000 7f7066b9a740 1 -- 172.21.15.134:0/2605160368 >> 172.21.15.134:0/2605160368 conn(0x55872d776300 msgr2=0x55872db23b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:55.115 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.104+0000 7f7066b9a740 1 -- 172.21.15.134:0/2605160368 shutdown_connections 2024-07-26T11:20:55.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.105+0000 7f7066b9a740 1 -- 172.21.15.134:0/2605160368 wait complete. 2024-07-26T11:20:55.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.106+0000 7f7066b9a740 1 -- start start 2024-07-26T11:20:55.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.106+0000 7f7066b9a740 1 Processor -- start 2024-07-26T11:20:55.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.107+0000 7f7066b9a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 0x55872db31790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.108+0000 7f7066b9a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7614d0 0x55872db31cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.108+0000 7f7066b9a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7821b0 0x55872db4e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.109+0000 7f705d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7614d0 0x55872db31cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:55.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.109+0000 7f7066b9a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55872d9c3ab0 con 0x55872d7592f0 2024-07-26T11:20:55.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.109+0000 7f7066b9a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55872da26d50 con 0x55872d7821b0 2024-07-26T11:20:55.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.109+0000 7f7066b9a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55872db343c0 con 0x55872d7614d0 2024-07-26T11:20:55.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.109+0000 7f705d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7614d0 0x55872db31cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:33452/0 (socket says 172.21.15.134:33452) 2024-07-26T11:20:55.117 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.109+0000 7f705d000640 1 -- 172.21.15.134:0/3600746870 learned_addr learned my addr 172.21.15.134:0/3600746870 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:55.117 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.109+0000 7f7057e00640 1 --2- 172.21.15.134:0/3600746870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7821b0 0x55872db4e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:55.117 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.110+0000 7f7066b9a740 10 client.0 _ll_register_callbacks cb 0x55872d799a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:55.117 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.111+0000 7f7057e00640 1 --2- 172.21.15.134:0/3600746870 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 0x55872db31790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:55.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.113+0000 7f705d000640 1 -- 172.21.15.134:0/3600746870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7821b0 msgr2=0x55872db4e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:55.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.113+0000 7f705d000640 1 --2- 172.21.15.134:0/3600746870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7821b0 0x55872db4e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.113+0000 7f705d000640 1 -- 172.21.15.134:0/3600746870 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 msgr2=0x55872db31790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:55.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.113+0000 7f705d000640 1 --2- 172.21.15.134:0/3600746870 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 0x55872db31790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.113+0000 7f705d000640 1 -- 172.21.15.134:0/3600746870 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55872d866620 con 0x55872d7614d0 2024-07-26T11:20:55.122 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.117+0000 7f705d000640 1 --2- 172.21.15.134:0/3600746870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7614d0 0x55872db31cd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f704c007080 tx=0x7f704c00dc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42a92238d16818ce server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:55.122 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.117+0000 7f703ea00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:20:55.123 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.121+0000 7f703ea00640 1 -- 172.21.15.134:0/3600746870 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704c013890 con 0x55872d7614d0 2024-07-26T11:20:55.123 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.122+0000 7f703ea00640 1 -- 172.21.15.134:0/3600746870 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f704c027410 con 0x55872d7614d0 2024-07-26T11:20:55.126 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.124+0000 7f7066b9a740 1 -- 172.21.15.134:0/3600746870 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f704c018070 con 0x55872d7614d0 2024-07-26T11:20:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.149+0000 7f703ea00640 1 -- 172.21.15.134:0/3600746870 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f704c027610 con 0x55872d7614d0 2024-07-26T11:20:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.149+0000 7f7066b9a740 10 client.26269 fetch_fsmap learned FSMap version 32 2024-07-26T11:20:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.149+0000 7f7066b9a740 1 -- 172.21.15.134:0/3600746870 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55872db36d90 con 0x55872d7614d0 2024-07-26T11:20:55.153 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-26T11:20:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.151+0000 7f703ea00640 1 -- 172.21.15.134:0/3600746870 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f704c018070 con 0x55872d7614d0 2024-07-26T11:20:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.151+0000 7f703ea00640 1 -- 172.21.15.134:0/3600746870 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f704c0318c0 con 0x55872d7614d0 2024-07-26T11:20:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.151+0000 7f7066b9a740 10 client.26269 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:20:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.151+0000 7f7066b9a740 10 client.26269 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:20:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.151+0000 7f7066b9a740 20 client.26269 populate_metadata read hostname 'smithi134' 2024-07-26T11:20:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.151+0000 7f7066b9a740 1 --2- 172.21.15.134:0/3600746870 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55872d740d80 0x55872db3a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.151+0000 7f7066b9a740 4 client.26269 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:20:55.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.151+0000 7f7066b9a740 1 -- 172.21.15.134:0/3600746870 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55872da9c1c0 con 0x55872d740d80 2024-07-26T11:20:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.155+0000 7f7057e00640 1 --2- 172.21.15.134:0/3600746870 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55872d740d80 0x55872db3a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:55.171 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.168+0000 7f7057e00640 1 --2- 172.21.15.134:0/3600746870 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55872d740d80 0x55872db3a1b0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12c7f9b64296e72c server_cookie=387d727185d3e68d in_seq=0 out_seq=0 2024-07-26T11:20:55.174 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.173+0000 7f703ea00640 10 client.26269 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:20:55.177 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.175+0000 7f703ea00640 1 -- 172.21.15.134:0/3600746870 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55872da9c1c0 con 0x55872d740d80 2024-07-26T11:20:55.177 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.175+0000 7f703ea00640 10 client.26269 handle_command_reply: tid=0 2024-07-26T11:20:55.177 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.176+0000 7f7066b9a740 1 client.26269 shutdown 2024-07-26T11:20:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.179+0000 7f7066b9a740 1 -- 172.21.15.134:0/3600746870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7614d0 msgr2=0x55872db31cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.179+0000 7f7066b9a740 1 --2- 172.21.15.134:0/3600746870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7614d0 0x55872db31cd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f704c007080 tx=0x7f704c00dc40 comp rx=0 tx=0).stop 2024-07-26T11:20:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.181+0000 7f7066b9a740 1 -- 172.21.15.134:0/3600746870 shutdown_connections 2024-07-26T11:20:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.181+0000 7f7066b9a740 1 --2- 172.21.15.134:0/3600746870 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55872d740d80 0x55872db3a1b0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.181+0000 7f7066b9a740 1 --2- 172.21.15.134:0/3600746870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7614d0 0x55872db31cd0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.181+0000 7f7066b9a740 1 --2- 172.21.15.134:0/3600746870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7821b0 0x55872db4e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.181+0000 7f7066b9a740 1 --2- 172.21.15.134:0/3600746870 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 0x55872db31790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.181+0000 7f703ea00640 0 client.26269 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:20:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.181+0000 7f7066b9a740 1 -- 172.21.15.134:0/3600746870 >> 172.21.15.134:0/3600746870 conn(0x55872d776300 msgr2=0x55872db26c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.182+0000 7f7066b9a740 1 -- 172.21.15.134:0/3600746870 shutdown_connections 2024-07-26T11:20:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.182+0000 7f7066b9a740 1 -- 172.21.15.134:0/3600746870 wait complete. 2024-07-26T11:20:55.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.184+0000 7f7066b9a740 20 client.26269 trim_cache size 0 max 16384 2024-07-26T11:20:55.256 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.254+0000 7f7066b9a740 1 Processor -- start 2024-07-26T11:20:55.256 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.255+0000 7f7066b9a740 1 -- start start 2024-07-26T11:20:55.258 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.255+0000 7f7066b9a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7592f0 0x55872db2bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.255+0000 7f7066b9a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7614d0 0x55872db23a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.255+0000 7f7066b9a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7821b0 0x55872db3f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.255+0000 7f7066b9a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f703800ff70 con 0x55872d7821b0 2024-07-26T11:20:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.255+0000 7f7066b9a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7038011ab0 con 0x55872d7614d0 2024-07-26T11:20:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.256+0000 7f7066b9a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7038010640 con 0x55872d7592f0 2024-07-26T11:20:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.256+0000 7f705d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7614d0 0x55872db23a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.256+0000 7f705d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7614d0 0x55872db23a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:40492/0 (socket says 172.21.15.134:40492) 2024-07-26T11:20:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.256+0000 7f705d000640 1 -- 172.21.15.134:0/4192131853 learned_addr learned my addr 172.21.15.134:0/4192131853 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:55.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.257+0000 7f7057e00640 1 --2- 172.21.15.134:0/4192131853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7592f0 0x55872db2bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.258+0000 7f7057e00640 1 --2- 172.21.15.134:0/4192131853 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7821b0 0x55872db3f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:55.261 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.259+0000 7f7057e00640 1 -- 172.21.15.134:0/4192131853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7614d0 msgr2=0x55872db23a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.259+0000 7f7057e00640 1 --2- 172.21.15.134:0/4192131853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7614d0 0x55872db23a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.259+0000 7f7057e00640 1 -- 172.21.15.134:0/4192131853 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7821b0 msgr2=0x55872db3f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.259+0000 7f7057e00640 1 --2- 172.21.15.134:0/4192131853 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7821b0 0x55872db3f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.259+0000 7f7057e00640 1 -- 172.21.15.134:0/4192131853 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55872db23f70 con 0x55872d7592f0 2024-07-26T11:20:55.262 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.260+0000 7f7057e00640 1 --2- 172.21.15.134:0/4192131853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7592f0 0x55872db2bd80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x55872db3a7c0 tx=0x7f7044004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70a5fe3c0115ed23 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.260+0000 7f7049600640 1 -- 172.21.15.134:0/4192131853 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704404c070 con 0x55872d7592f0 2024-07-26T11:20:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.260+0000 7f7049600640 1 -- 172.21.15.134:0/4192131853 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7044031040 con 0x55872d7592f0 2024-07-26T11:20:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.260+0000 7f7049600640 1 -- 172.21.15.134:0/4192131853 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044030040 con 0x55872d7592f0 2024-07-26T11:20:55.263 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.262+0000 7f7066b9a740 1 -- 172.21.15.134:0/4192131853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7592f0 msgr2=0x55872db2bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:55.264 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.262+0000 7f7066b9a740 1 --2- 172.21.15.134:0/4192131853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7592f0 0x55872db2bd80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x55872db3a7c0 tx=0x7f7044004750 comp rx=0 tx=0).stop 2024-07-26T11:20:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.265+0000 7f7066b9a740 1 -- 172.21.15.134:0/4192131853 shutdown_connections 2024-07-26T11:20:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.265+0000 7f7066b9a740 1 --2- 172.21.15.134:0/4192131853 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7821b0 0x55872db3f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.265+0000 7f7066b9a740 1 --2- 172.21.15.134:0/4192131853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7614d0 0x55872db23a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.265+0000 7f7066b9a740 1 --2- 172.21.15.134:0/4192131853 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7592f0 0x55872db2bd80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.267 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.265+0000 7f7066b9a740 1 -- 172.21.15.134:0/4192131853 >> 172.21.15.134:0/4192131853 conn(0x55872d776300 msgr2=0x55872db23320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:55.268 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.266+0000 7f7066b9a740 1 -- 172.21.15.134:0/4192131853 shutdown_connections 2024-07-26T11:20:55.268 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.266+0000 7f7066b9a740 1 -- 172.21.15.134:0/4192131853 wait complete. 2024-07-26T11:20:55.268 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.266+0000 7f7066b9a740 1 -- start start 2024-07-26T11:20:55.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.267+0000 7f7066b9a740 1 Processor -- start 2024-07-26T11:20:55.270 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.268+0000 7f7066b9a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 0x55872db3e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.268+0000 7f7066b9a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7821b0 0x55872db3e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.268+0000 7f7066b9a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7161c0 0x55872dbc4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.268+0000 7f7066b9a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f70380113e0 con 0x55872d7592f0 2024-07-26T11:20:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.268+0000 7f7066b9a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7038010d10 con 0x55872d7821b0 2024-07-26T11:20:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.268+0000 7f7066b9a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f703800df50 con 0x55872d7161c0 2024-07-26T11:20:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.268+0000 7f7066b9a740 10 client.0 _ll_register_callbacks cb 0x7f704c007780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:20:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.269+0000 7f7057e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7161c0 0x55872dbc4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:55.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.269+0000 7f7057e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7161c0 0x55872dbc4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:33466/0 (socket says 172.21.15.134:33466) 2024-07-26T11:20:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.270+0000 7f705d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7821b0 0x55872db3e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.270+0000 7f705d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7821b0 0x55872db3e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:40504/0 (socket says 172.21.15.134:40504) 2024-07-26T11:20:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.270+0000 7f705d000640 1 -- 172.21.15.134:0/816991937 learned_addr learned my addr 172.21.15.134:0/816991937 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:20:55.272 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.270+0000 7f7057e00640 1 --2- 172.21.15.134:0/816991937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 0x55872db3e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:55.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.273+0000 7f705d000640 1 -- 172.21.15.134:0/816991937 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7161c0 msgr2=0x55872dbc4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:55.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.273+0000 7f705d000640 1 --2- 172.21.15.134:0/816991937 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7161c0 0x55872dbc4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.273+0000 7f705d000640 1 -- 172.21.15.134:0/816991937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 msgr2=0x55872db3e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:20:55.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.273+0000 7f705d000640 1 --2- 172.21.15.134:0/816991937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 0x55872db3e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.273+0000 7f705d000640 1 -- 172.21.15.134:0/816991937 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55872dbc5490 con 0x55872d7821b0 2024-07-26T11:20:55.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.274+0000 7f703ea00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:20:55.277 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.274+0000 7f705d000640 1 --2- 172.21.15.134:0/816991937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7821b0 0x55872db3e810 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f704c031e30 tx=0x7f704c007340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88996f51805339b8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:20:55.278 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.275+0000 7f703ea00640 1 -- 172.21.15.134:0/816991937 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704c007550 con 0x55872d7821b0 2024-07-26T11:20:55.279 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.275+0000 7f703ea00640 1 -- 172.21.15.134:0/816991937 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f704c01d400 con 0x55872d7821b0 2024-07-26T11:20:55.279 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.275+0000 7f7066b9a740 1 -- 172.21.15.134:0/816991937 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7044031040 con 0x55872d7821b0 2024-07-26T11:20:55.279 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.276+0000 7f703ea00640 1 -- 172.21.15.134:0/816991937 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f704c007550 con 0x55872d7821b0 2024-07-26T11:20:55.279 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.276+0000 7f7066b9a740 10 client.26276 fetch_fsmap learned FSMap version 32 2024-07-26T11:20:55.279 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.276+0000 7f7066b9a740 1 -- 172.21.15.134:0/816991937 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55872dbcd9b0 con 0x55872d7821b0 2024-07-26T11:20:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.279+0000 7f703ea00640 1 -- 172.21.15.134:0/816991937 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f704c036070 con 0x55872d7821b0 2024-07-26T11:20:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.279+0000 7f703ea00640 1 -- 172.21.15.134:0/816991937 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f704c027c10 con 0x55872d7821b0 2024-07-26T11:20:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.279+0000 7f7066b9a740 10 client.26276 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:20:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.279+0000 7f7066b9a740 10 client.26276 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:20:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.279+0000 7f7066b9a740 20 client.26276 populate_metadata read hostname 'smithi134' 2024-07-26T11:20:55.281 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.279+0000 7f7066b9a740 1 --2- 172.21.15.134:0/816991937 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55872dbce100 0x55872dbd0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:20:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.280+0000 7f7066b9a740 4 client.26276 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:20:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.280+0000 7f7066b9a740 1 -- 172.21.15.134:0/816991937 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f7044031040 con 0x55872dbce100 2024-07-26T11:20:55.282 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.280+0000 7f7057e00640 1 --2- 172.21.15.134:0/816991937 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55872dbce100 0x55872dbd0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:20:55.285 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.282+0000 7f703ea00640 10 client.26276 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:20:55.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.293+0000 7f7057e00640 1 --2- 172.21.15.134:0/816991937 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55872dbce100 0x55872dbd0520 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=805131c023eff0aa server_cookie=8c870d703aec8e10 in_seq=0 out_seq=0 2024-07-26T11:20:55.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.396+0000 7f703ea00640 1 -- 172.21.15.134:0/816991937 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f7044031040 con 0x55872dbce100 2024-07-26T11:20:55.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.396+0000 7f703ea00640 10 client.26276 handle_command_reply: tid=0 2024-07-26T11:20:55.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.396+0000 7f7066b9a740 1 client.26276 shutdown 2024-07-26T11:20:55.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.397+0000 7f7066b9a740 1 -- 172.21.15.134:0/816991937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7821b0 msgr2=0x55872db3e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:55.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.397+0000 7f7066b9a740 1 --2- 172.21.15.134:0/816991937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7821b0 0x55872db3e810 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f704c031e30 tx=0x7f704c007340 comp rx=0 tx=0).stop 2024-07-26T11:20:55.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.397+0000 7f7066b9a740 1 -- 172.21.15.134:0/816991937 shutdown_connections 2024-07-26T11:20:55.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.397+0000 7f7066b9a740 1 --2- 172.21.15.134:0/816991937 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55872dbce100 0x55872dbd0520 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.397+0000 7f7066b9a740 1 --2- 172.21.15.134:0/816991937 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55872d7161c0 0x55872dbc4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.397+0000 7f7066b9a740 1 --2- 172.21.15.134:0/816991937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55872d7821b0 0x55872db3e810 secure :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f704c031e30 tx=0x7f704c007340 comp rx=0 tx=0).stop 2024-07-26T11:20:55.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.397+0000 7f7066b9a740 1 --2- 172.21.15.134:0/816991937 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55872d7592f0 0x55872db3e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.397+0000 7f703ea00640 0 client.26276 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:20:55.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.397+0000 7f7066b9a740 1 -- 172.21.15.134:0/816991937 >> 172.21.15.134:0/816991937 conn(0x55872d776300 msgr2=0x55872db329e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:55.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.399+0000 7f7066b9a740 1 -- 172.21.15.134:0/816991937 shutdown_connections 2024-07-26T11:20:55.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.399+0000 7f7066b9a740 1 -- 172.21.15.134:0/816991937 wait complete. 2024-07-26T11:20:55.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.400+0000 7f7066b9a740 20 client.26276 trim_cache size 0 max 16384 2024-07-26T11:20:55.403 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:20:55.403 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:20:55.403 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:20:55.403 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:20:55.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.402+0000 7f705fa00640 1 -- 172.21.15.134:0/155428064 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f70400b4180 msgr2=0x7f70400b6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:55.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.402+0000 7f705fa00640 1 --2- 172.21.15.134:0/155428064 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f70400b4180 0x7f70400b6690 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7058068b30 tx=0x7f70580694b0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.402+0000 7f705fa00640 1 -- 172.21.15.134:0/155428064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70500a0540 msgr2=0x7f7050141150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:20:55.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.402+0000 7f705fa00640 1 --2- 172.21.15.134:0/155428064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70500a0540 0x7f7050141150 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7058051740 tx=0x7f7058069d70 comp rx=0 tx=0).stop 2024-07-26T11:20:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.404+0000 7f705fa00640 1 -- 172.21.15.134:0/155428064 shutdown_connections 2024-07-26T11:20:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.404+0000 7f705fa00640 1 --2- 172.21.15.134:0/155428064 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f70400b4180 0x7f70400b6690 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.404+0000 7f705fa00640 1 --2- 172.21.15.134:0/155428064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70500a0540 0x7f7050141150 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.404+0000 7f705fa00640 1 --2- 172.21.15.134:0/155428064 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f705009dbe0 0x7f705013bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.404+0000 7f705fa00640 1 --2- 172.21.15.134:0/155428064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7050094280 0x7f705013b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:20:55.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.404+0000 7f705fa00640 1 -- 172.21.15.134:0/155428064 >> 172.21.15.134:0/155428064 conn(0x7f705000c0b0 msgr2=0x7f7050092df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:20:55.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.404+0000 7f705fa00640 1 -- 172.21.15.134:0/155428064 shutdown_connections 2024-07-26T11:20:55.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:20:55.404+0000 7f705fa00640 1 -- 172.21.15.134:0/155428064 wait complete. 2024-07-26T11:20:56.153 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:20:56.603 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:56 smithi134 ceph-mon[31966]: pgmap v629: 129 pgs: 129 active+clean; 34 GiB data, 70 GiB used, 1003 GiB / 1.0 TiB avail; 23 KiB/s rd, 256 MiB/s wr, 1.18k op/s 2024-07-26T11:20:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:56 smithi193 ceph-mon[37473]: pgmap v629: 129 pgs: 129 active+clean; 34 GiB data, 70 GiB used, 1003 GiB / 1.0 TiB avail; 23 KiB/s rd, 256 MiB/s wr, 1.18k op/s 2024-07-26T11:20:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:56 smithi189 ceph-mon[36790]: pgmap v629: 129 pgs: 129 active+clean; 34 GiB data, 70 GiB used, 1003 GiB / 1.0 TiB avail; 23 KiB/s rd, 256 MiB/s wr, 1.18k op/s 2024-07-26T11:20:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:20:58 smithi189 ceph-mon[36790]: pgmap v630: 129 pgs: 129 active+clean; 34 GiB data, 71 GiB used, 1002 GiB / 1.0 TiB avail; 38 KiB/s rd, 262 MiB/s wr, 1.02k op/s 2024-07-26T11:20:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:20:58 smithi193 ceph-mon[37473]: pgmap v630: 129 pgs: 129 active+clean; 34 GiB data, 71 GiB used, 1002 GiB / 1.0 TiB avail; 38 KiB/s rd, 262 MiB/s wr, 1.02k op/s 2024-07-26T11:20:58.863 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:20:58 smithi134 ceph-mon[31966]: pgmap v630: 129 pgs: 129 active+clean; 34 GiB data, 71 GiB used, 1002 GiB / 1.0 TiB avail; 38 KiB/s rd, 262 MiB/s wr, 1.02k op/s 2024-07-26T11:21:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:00 smithi193 ceph-mon[37473]: pgmap v631: 129 pgs: 129 active+clean; 34 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 28 KiB/s rd, 286 MiB/s wr, 943 op/s 2024-07-26T11:21:00.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:00 smithi134 ceph-mon[31966]: pgmap v631: 129 pgs: 129 active+clean; 34 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 28 KiB/s rd, 286 MiB/s wr, 943 op/s 2024-07-26T11:21:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:00 smithi189 ceph-mon[36790]: pgmap v631: 129 pgs: 129 active+clean; 34 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 28 KiB/s rd, 286 MiB/s wr, 943 op/s 2024-07-26T11:21:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:02 smithi189 ceph-mon[36790]: pgmap v632: 129 pgs: 129 active+clean; 35 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 49 KiB/s rd, 271 MiB/s wr, 794 op/s 2024-07-26T11:21:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:02 smithi193 ceph-mon[37473]: pgmap v632: 129 pgs: 129 active+clean; 35 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 49 KiB/s rd, 271 MiB/s wr, 794 op/s 2024-07-26T11:21:02.870 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:02 smithi134 ceph-mon[31966]: pgmap v632: 129 pgs: 129 active+clean; 35 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 49 KiB/s rd, 271 MiB/s wr, 794 op/s 2024-07-26T11:21:05.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.298+0000 7fa632c00640 1 -- 172.21.15.134:0/2770383380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c10fc40 msgr2=0x7fa62c075090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:05.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.298+0000 7fa632c00640 1 --2- 172.21.15.134:0/2770383380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c10fc40 0x7fa62c075090 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa624009ca0 tx=0x7fa62402a8f0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.299+0000 7fa632c00640 1 -- 172.21.15.134:0/2770383380 shutdown_connections 2024-07-26T11:21:05.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.299+0000 7fa632c00640 1 --2- 172.21.15.134:0/2770383380 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c10fc40 0x7fa62c075090 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.299+0000 7fa632c00640 1 --2- 172.21.15.134:0/2770383380 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa62c10f2f0 0x7fa62c10f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.299+0000 7fa632c00640 1 --2- 172.21.15.134:0/2770383380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa62c10b310 0x7fa62c10b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.299+0000 7fa632c00640 1 -- 172.21.15.134:0/2770383380 >> 172.21.15.134:0/2770383380 conn(0x7fa62c06e420 msgr2=0x7fa62c06e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:05.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.299+0000 7fa632c00640 1 -- 172.21.15.134:0/2770383380 shutdown_connections 2024-07-26T11:21:05.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.299+0000 7fa632c00640 1 -- 172.21.15.134:0/2770383380 wait complete. 2024-07-26T11:21:05.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.301+0000 7fa632c00640 1 Processor -- start 2024-07-26T11:21:05.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.301+0000 7fa632c00640 1 -- start start 2024-07-26T11:21:05.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.303+0000 7fa632c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa62c10b310 0x7fa62c0786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.303+0000 7fa632c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c10f2f0 0x7fa62c078c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.303+0000 7fa632c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa62c10fc40 0x7fa62c1c1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.303+0000 7fa632c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c082a70 con 0x7fa62c10b310 2024-07-26T11:21:05.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.303+0000 7fa632c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c0828f0 con 0x7fa62c10fc40 2024-07-26T11:21:05.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.303+0000 7fa632c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c082bf0 con 0x7fa62c10f2f0 2024-07-26T11:21:05.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.304+0000 7fa631800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa62c10b310 0x7fa62c0786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:05.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.304+0000 7fa631800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa62c10b310 0x7fa62c0786e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58940/0 (socket says 172.21.15.134:58940) 2024-07-26T11:21:05.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.304+0000 7fa631800640 1 -- 172.21.15.134:0/390518198 learned_addr learned my addr 172.21.15.134:0/390518198 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:05.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.304+0000 7fa631800640 1 -- 172.21.15.134:0/390518198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c10f2f0 msgr2=0x7fa62c078c20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T11:21:05.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.305+0000 7fa631800640 1 --2- 172.21.15.134:0/390518198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c10f2f0 0x7fa62c078c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.305+0000 7fa631800640 1 -- 172.21.15.134:0/390518198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa62c10fc40 msgr2=0x7fa62c1c1180 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T11:21:05.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.305+0000 7fa631800640 1 --2- 172.21.15.134:0/390518198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa62c10fc40 0x7fa62c1c1180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.305+0000 7fa631800640 1 -- 172.21.15.134:0/390518198 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c1c16f0 con 0x7fa62c10b310 2024-07-26T11:21:05.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.306+0000 7fa631800640 1 --2- 172.21.15.134:0/390518198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa62c10b310 0x7fa62c0786e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00b450 tx=0x7fa61c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:05.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.306+0000 7fa62a000640 1 -- 172.21.15.134:0/390518198 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c015020 con 0x7fa62c10b310 2024-07-26T11:21:05.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.306+0000 7fa62a000640 1 -- 172.21.15.134:0/390518198 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa61c011d40 con 0x7fa62c10b310 2024-07-26T11:21:05.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.306+0000 7fa62a000640 1 -- 172.21.15.134:0/390518198 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c00f340 con 0x7fa62c10b310 2024-07-26T11:21:05.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.309+0000 7fa632c00640 1 -- 172.21.15.134:0/390518198 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62c1c19e0 con 0x7fa62c10b310 2024-07-26T11:21:05.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.309+0000 7fa632c00640 1 -- 172.21.15.134:0/390518198 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa62c1c1f20 con 0x7fa62c10b310 2024-07-26T11:21:05.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.314+0000 7fa63ae5d740 1 Processor -- start 2024-07-26T11:21:05.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.315+0000 7fa63ae5d740 1 -- start start 2024-07-26T11:21:05.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.316+0000 7fa63ae5d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39c3070 0x55b4a3d70820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.316+0000 7fa63ae5d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 0x55b4a3d70d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.316+0000 7fa63ae5d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a39a2390 0x55b4a3d712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.316+0000 7fa63ae5d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b4a3cdebd0 con 0x55b4a39a2390 2024-07-26T11:21:05.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.316+0000 7fa63ae5d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b4a3cce0d0 con 0x55b4a399a1b0 2024-07-26T11:21:05.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.316+0000 7fa63ae5d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b4a3cd2110 con 0x55b4a39c3070 2024-07-26T11:21:05.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.317+0000 7fa632200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39c3070 0x55b4a3d70820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:05.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.317+0000 7fa632200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39c3070 0x55b4a3d70820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:35888/0 (socket says 172.21.15.134:35888) 2024-07-26T11:21:05.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.317+0000 7fa632200640 1 -- 172.21.15.134:0/3946089186 learned_addr learned my addr 172.21.15.134:0/3946089186 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:05.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.317+0000 7fa632200640 1 -- 172.21.15.134:0/3946089186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 msgr2=0x55b4a3d70d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:21:05.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.317+0000 7fa632200640 1 --2- 172.21.15.134:0/3946089186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 0x55b4a3d70d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.317+0000 7fa632200640 1 -- 172.21.15.134:0/3946089186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a39a2390 msgr2=0x55b4a3d712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:05.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.317+0000 7fa632200640 1 --2- 172.21.15.134:0/3946089186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a39a2390 0x55b4a3d712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.317+0000 7fa632200640 1 -- 172.21.15.134:0/3946089186 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4a3a2e650 con 0x55b4a39c3070 2024-07-26T11:21:05.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.318+0000 7fa62a000640 1 -- 172.21.15.134:0/390518198 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fa61c018020 con 0x7fa62c10b310 2024-07-26T11:21:05.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.318+0000 7fa632200640 1 --2- 172.21.15.134:0/3946089186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39c3070 0x55b4a3d70820 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa62402aee0 tx=0x7fa624001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2df944bdd83bcb61 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:05.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.320+0000 7fa61aa00640 1 -- 172.21.15.134:0/3946089186 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624031cf0 con 0x55b4a39c3070 2024-07-26T11:21:05.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.320+0000 7fa61aa00640 1 -- 172.21.15.134:0/3946089186 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa62403f070 con 0x55b4a39c3070 2024-07-26T11:21:05.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.320+0000 7fa61aa00640 1 -- 172.21.15.134:0/3946089186 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6240042a0 con 0x55b4a39c3070 2024-07-26T11:21:05.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.321+0000 7fa62a000640 1 --2- 172.21.15.134:0/390518198 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6140b3e40 0x7fa6140b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.321+0000 7fa62a000640 1 -- 172.21.15.134:0/390518198 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa61c0e3e20 con 0x7fa62c10b310 2024-07-26T11:21:05.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.324+0000 7fa63ae5d740 1 -- 172.21.15.134:0/3946089186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39c3070 msgr2=0x55b4a3d70820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:05.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.324+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/3946089186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39c3070 0x55b4a3d70820 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa62402aee0 tx=0x7fa624001120 comp rx=0 tx=0).stop 2024-07-26T11:21:05.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.324+0000 7fa631800640 1 --2- 172.21.15.134:0/390518198 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6140b3e40 0x7fa6140b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:05.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.326+0000 7fa631800640 1 --2- 172.21.15.134:0/390518198 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6140b3e40 0x7fa6140b6350 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x55b4a3d6f4a0 tx=0x7fa61c006df0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:05.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.326+0000 7fa63ae5d740 1 -- 172.21.15.134:0/3946089186 shutdown_connections 2024-07-26T11:21:05.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.326+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/3946089186 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a39a2390 0x55b4a3d712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.326+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/3946089186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 0x55b4a3d70d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.326+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/3946089186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39c3070 0x55b4a3d70820 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.326+0000 7fa63ae5d740 1 -- 172.21.15.134:0/3946089186 >> 172.21.15.134:0/3946089186 conn(0x55b4a39b7460 msgr2=0x55b4a3d6e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:05.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.326+0000 7fa63ae5d740 1 -- 172.21.15.134:0/3946089186 shutdown_connections 2024-07-26T11:21:05.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.326+0000 7fa63ae5d740 1 -- 172.21.15.134:0/3946089186 wait complete. 2024-07-26T11:21:05.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.327+0000 7fa63ae5d740 1 -- start start 2024-07-26T11:21:05.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.328+0000 7fa63ae5d740 1 Processor -- start 2024-07-26T11:21:05.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.331+0000 7fa63ae5d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 0x55b4a3d6a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.331+0000 7fa63ae5d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39a2390 0x55b4a3d6aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.331+0000 7fa63ae5d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a39c3070 0x55b4a3d6b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.331+0000 7fa63ae5d740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b4a3b31700 con 0x55b4a39c3070 2024-07-26T11:21:05.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.331+0000 7fa63ae5d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b4a3c6a6a0 con 0x55b4a399a1b0 2024-07-26T11:21:05.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.331+0000 7fa63ae5d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b4a3d6daf0 con 0x55b4a39a2390 2024-07-26T11:21:05.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.331+0000 7fa63ae5d740 10 client.0 _ll_register_callbacks cb 0x55b4a3c7a150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:05.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.331+0000 7fa632200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 0x55b4a3d6a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:05.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.331+0000 7fa632200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 0x55b4a3d6a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46474/0 (socket says 172.21.15.134:46474) 2024-07-26T11:21:05.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.331+0000 7fa632200640 1 -- 172.21.15.134:0/1617106348 learned_addr learned my addr 172.21.15.134:0/1617106348 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:05.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.332+0000 7fa630e00640 1 --2- 172.21.15.134:0/1617106348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39a2390 0x55b4a3d6aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:05.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.332+0000 7fa632200640 1 --2- 172.21.15.134:0/1617106348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a39c3070 0x55b4a3d6b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:05.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.332+0000 7fa632200640 1 -- 172.21.15.134:0/1617106348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39a2390 msgr2=0x55b4a3d6aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:05.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.332+0000 7fa632200640 1 --2- 172.21.15.134:0/1617106348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39a2390 0x55b4a3d6aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.332+0000 7fa632200640 1 -- 172.21.15.134:0/1617106348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a39c3070 msgr2=0x55b4a3d6b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:05.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.332+0000 7fa632200640 1 --2- 172.21.15.134:0/1617106348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a39c3070 0x55b4a3d6b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.332+0000 7fa632200640 1 -- 172.21.15.134:0/1617106348 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4a3c81840 con 0x55b4a399a1b0 2024-07-26T11:21:05.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.333+0000 7fa632200640 1 --2- 172.21.15.134:0/1617106348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 0x55b4a3d6a970 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa624003ee0 tx=0x7fa62403d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39d35883dd63e0c5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:05.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.333+0000 7fa60f400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:21:05.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.333+0000 7fa60f400640 1 -- 172.21.15.134:0/1617106348 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624031af0 con 0x55b4a399a1b0 2024-07-26T11:21:05.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.333+0000 7fa60f400640 1 -- 172.21.15.134:0/1617106348 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa624031c90 con 0x55b4a399a1b0 2024-07-26T11:21:05.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.333+0000 7fa63ae5d740 1 -- 172.21.15.134:0/1617106348 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa62403f070 con 0x55b4a399a1b0 2024-07-26T11:21:05.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.334+0000 7fa60f400640 1 -- 172.21.15.134:0/1617106348 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa624031480 con 0x55b4a399a1b0 2024-07-26T11:21:05.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.335+0000 7fa63ae5d740 10 client.26279 fetch_fsmap learned FSMap version 32 2024-07-26T11:21:05.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.335+0000 7fa63ae5d740 1 -- 172.21.15.134:0/1617106348 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b4a3d93450 con 0x55b4a399a1b0 2024-07-26T11:21:05.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.335+0000 7fa60f400640 1 -- 172.21.15.134:0/1617106348 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa62403f070 con 0x55b4a399a1b0 2024-07-26T11:21:05.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.335+0000 7fa60f400640 1 -- 172.21.15.134:0/1617106348 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fa62402fc90 con 0x55b4a399a1b0 2024-07-26T11:21:05.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.335+0000 7fa63ae5d740 10 client.26279 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:21:05.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.335+0000 7fa63ae5d740 10 client.26279 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:21:05.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.335+0000 7fa63ae5d740 20 client.26279 populate_metadata read hostname 'smithi134' 2024-07-26T11:21:05.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.335+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/1617106348 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55b4a39819f0 0x55b4a3d967b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.335+0000 7fa63ae5d740 4 client.26279 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:21:05.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.335+0000 7fa63ae5d740 1 -- 172.21.15.134:0/1617106348 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4a3ce0b20 con 0x55b4a39819f0 2024-07-26T11:21:05.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.337+0000 7fa630e00640 1 --2- 172.21.15.134:0/1617106348 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55b4a39819f0 0x55b4a3d967b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:05.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.342+0000 7fa630e00640 1 --2- 172.21.15.134:0/1617106348 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55b4a39819f0 0x55b4a3d967b0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7d6d9553554af0b server_cookie=f34371d054c0636f in_seq=0 out_seq=0 2024-07-26T11:21:05.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.342+0000 7fa60f400640 10 client.26279 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:05.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.348+0000 7fa60f400640 1 -- 172.21.15.134:0/1617106348 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b4a3ce0b20 con 0x55b4a39819f0 2024-07-26T11:21:05.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.348+0000 7fa60f400640 10 client.26279 handle_command_reply: tid=0 2024-07-26T11:21:05.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.348+0000 7fa63ae5d740 1 client.26279 shutdown 2024-07-26T11:21:05.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.350+0000 7fa63ae5d740 1 -- 172.21.15.134:0/1617106348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 msgr2=0x55b4a3d6a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:05.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.350+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/1617106348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 0x55b4a3d6a970 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa624003ee0 tx=0x7fa62403d040 comp rx=0 tx=0).stop 2024-07-26T11:21:05.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.350+0000 7fa63ae5d740 1 -- 172.21.15.134:0/1617106348 shutdown_connections 2024-07-26T11:21:05.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.350+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/1617106348 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55b4a39819f0 0x55b4a3d967b0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.350+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/1617106348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39a2390 0x55b4a3d6aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.350+0000 7fa60f400640 0 client.26279 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:05.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.350+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/1617106348 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a39c3070 0x55b4a3d6b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.350+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/1617106348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 0x55b4a3d6a970 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.350+0000 7fa63ae5d740 1 -- 172.21.15.134:0/1617106348 >> 172.21.15.134:0/1617106348 conn(0x55b4a39b7460 msgr2=0x55b4a3d71850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:05.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.350+0000 7fa63ae5d740 1 -- 172.21.15.134:0/1617106348 shutdown_connections 2024-07-26T11:21:05.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.351+0000 7fa63ae5d740 1 -- 172.21.15.134:0/1617106348 wait complete. 2024-07-26T11:21:05.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.352+0000 7fa63ae5d740 20 client.26279 trim_cache size 0 max 16384 2024-07-26T11:21:05.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.383+0000 7fa63ae5d740 1 Processor -- start 2024-07-26T11:21:05.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.383+0000 7fa63ae5d740 1 -- start start 2024-07-26T11:21:05.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.383+0000 7fa63ae5d740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a399a1b0 0x55b4a3d6c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.384+0000 7fa632200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a399a1b0 0x55b4a3d6c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:05.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.384+0000 7fa632200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a399a1b0 0x55b4a3d6c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58964/0 (socket says 172.21.15.134:58964) 2024-07-26T11:21:05.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.384+0000 7fa632200640 1 -- 172.21.15.134:0/1202821631 learned_addr learned my addr 172.21.15.134:0/1202821631 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:05.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.384+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/1202821631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a39a2390 0x55b4a3d67b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.385+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/1202821631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39c3070 0x55b4a3d680a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.385+0000 7fa63ae5d740 1 -- 172.21.15.134:0/1202821631 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1c7ec0 con 0x55b4a399a1b0 2024-07-26T11:21:05.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.385+0000 7fa63ae5d740 1 -- 172.21.15.134:0/1202821631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1c9a00 con 0x55b4a39a2390 2024-07-26T11:21:05.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.385+0000 7fa63ae5d740 1 -- 172.21.15.134:0/1202821631 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1c8590 con 0x55b4a39c3070 2024-07-26T11:21:05.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.385+0000 7fa630e00640 1 --2- 172.21.15.134:0/1202821631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a39a2390 0x55b4a3d67b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:05.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.386+0000 7fa632200640 1 --2- 172.21.15.134:0/1202821631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39c3070 0x55b4a3d680a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:05.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.386+0000 7fa632200640 1 -- 172.21.15.134:0/1202821631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39c3070 msgr2=0x55b4a3d680a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:05.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.386+0000 7fa632200640 1 --2- 172.21.15.134:0/1202821631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39c3070 0x55b4a3d680a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.386+0000 7fa632200640 1 -- 172.21.15.134:0/1202821631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a39a2390 msgr2=0x55b4a3d67b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:05.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.386+0000 7fa632200640 1 --2- 172.21.15.134:0/1202821631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a39a2390 0x55b4a3d67b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.386+0000 7fa632200640 1 -- 172.21.15.134:0/1202821631 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4a3d6a200 con 0x55b4a399a1b0 2024-07-26T11:21:05.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.387+0000 7fa632200640 1 --2- 172.21.15.134:0/1202821631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a399a1b0 0x55b4a3d6c5a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa624005d50 tx=0x7fa624003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1771da340acaa3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:05.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.387+0000 7fa61a000640 1 -- 172.21.15.134:0/1202821631 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624031620 con 0x55b4a399a1b0 2024-07-26T11:21:05.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.388+0000 7fa61a000640 1 -- 172.21.15.134:0/1202821631 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa62403f030 con 0x55b4a399a1b0 2024-07-26T11:21:05.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.388+0000 7fa61a000640 1 -- 172.21.15.134:0/1202821631 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624030e30 con 0x55b4a399a1b0 2024-07-26T11:21:05.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.389+0000 7fa63ae5d740 1 -- 172.21.15.134:0/1202821631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a399a1b0 msgr2=0x55b4a3d6c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:05.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.389+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/1202821631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a399a1b0 0x55b4a3d6c5a0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa624005d50 tx=0x7fa624003f90 comp rx=0 tx=0).stop 2024-07-26T11:21:05.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.389+0000 7fa63ae5d740 1 -- 172.21.15.134:0/1202821631 shutdown_connections 2024-07-26T11:21:05.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.389+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/1202821631 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39c3070 0x55b4a3d680a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.389+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/1202821631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a39a2390 0x55b4a3d67b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.390+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/1202821631 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a399a1b0 0x55b4a3d6c5a0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.390+0000 7fa63ae5d740 1 -- 172.21.15.134:0/1202821631 >> 172.21.15.134:0/1202821631 conn(0x55b4a39b7460 msgr2=0x55b4a3d6be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:05.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.390+0000 7fa63ae5d740 1 -- 172.21.15.134:0/1202821631 shutdown_connections 2024-07-26T11:21:05.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.390+0000 7fa63ae5d740 1 -- 172.21.15.134:0/1202821631 wait complete. 2024-07-26T11:21:05.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.391+0000 7fa63ae5d740 1 -- start start 2024-07-26T11:21:05.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.392+0000 7fa63ae5d740 1 Processor -- start 2024-07-26T11:21:05.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.393+0000 7fa63ae5d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 0x55b4a3d98b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.394+0000 7fa632200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 0x55b4a3d98b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:05.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.394+0000 7fa632200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 0x55b4a3d98b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46484/0 (socket says 172.21.15.134:46484) 2024-07-26T11:21:05.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.394+0000 7fa632200640 1 -- 172.21.15.134:0/3700655111 learned_addr learned my addr 172.21.15.134:0/3700655111 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:05.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.395+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/3700655111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39a2390 0x55b4a3d990a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.395+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/3700655111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a39c3070 0x55b4a3dfcd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.395+0000 7fa63ae5d740 1 -- 172.21.15.134:0/3700655111 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1c9330 con 0x55b4a39c3070 2024-07-26T11:21:05.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.396+0000 7fa63ae5d740 1 -- 172.21.15.134:0/3700655111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1c8c60 con 0x55b4a399a1b0 2024-07-26T11:21:05.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.396+0000 7fa63ae5d740 1 -- 172.21.15.134:0/3700655111 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c1c5ee0 con 0x55b4a39a2390 2024-07-26T11:21:05.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.397+0000 7fa632200640 1 --2- 172.21.15.134:0/3700655111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a39c3070 0x55b4a3dfcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:05.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.397+0000 7fa632200640 1 -- 172.21.15.134:0/3700655111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39a2390 msgr2=0x55b4a3d990a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:05.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.398+0000 7fa63ae5d740 10 client.0 _ll_register_callbacks cb 0x7fa624003b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:05.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.396+0000 7fa630e00640 1 --2- 172.21.15.134:0/3700655111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39a2390 0x55b4a3d990a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:05.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.398+0000 7fa632200640 1 --2- 172.21.15.134:0/3700655111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39a2390 0x55b4a3d990a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.398+0000 7fa632200640 1 -- 172.21.15.134:0/3700655111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a39c3070 msgr2=0x55b4a3dfcd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:05.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.399+0000 7fa632200640 1 --2- 172.21.15.134:0/3700655111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a39c3070 0x55b4a3dfcd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.399+0000 7fa632200640 1 -- 172.21.15.134:0/3700655111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4a3dfd240 con 0x55b4a399a1b0 2024-07-26T11:21:05.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.400+0000 7fa63ae5d740 1 -- 172.21.15.134:0/3700655111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa62403f030 con 0x55b4a399a1b0 2024-07-26T11:21:05.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.400+0000 7fa630e00640 1 --2- 172.21.15.134:0/3700655111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39a2390 0x55b4a3d990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:21:05.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.402+0000 7fa632200640 1 --2- 172.21.15.134:0/3700655111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 0x55b4a3d98b20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa62402f480 tx=0x7fa6240318d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=909d2962399ec48 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:05.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.402+0000 7fa60f400640 10 client.26285 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:21:05.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.402+0000 7fa60f400640 1 -- 172.21.15.134:0/3700655111 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624003de0 con 0x55b4a399a1b0 2024-07-26T11:21:05.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.402+0000 7fa60f400640 1 -- 172.21.15.134:0/3700655111 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa624031d00 con 0x55b4a399a1b0 2024-07-26T11:21:05.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.403+0000 7fa60f400640 1 -- 172.21.15.134:0/3700655111 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa624004350 con 0x55b4a399a1b0 2024-07-26T11:21:05.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.403+0000 7fa63ae5d740 10 client.26285 fetch_fsmap learned FSMap version 32 2024-07-26T11:21:05.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.403+0000 7fa63ae5d740 1 -- 172.21.15.134:0/3700655111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=32}) -- 0x55b4a3dfe2b0 con 0x55b4a399a1b0 2024-07-26T11:21:05.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.406+0000 7fa60f400640 1 -- 172.21.15.134:0/3700655111 <== mon.2 v2:172.21.15.189:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fa624030a10 con 0x55b4a399a1b0 2024-07-26T11:21:05.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.406+0000 7fa63ae5d740 10 client.26285 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:21:05.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.406+0000 7fa63ae5d740 10 client.26285 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:21:05.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.406+0000 7fa63ae5d740 20 client.26285 populate_metadata read hostname 'smithi134' 2024-07-26T11:21:05.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.406+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/3700655111 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55b4a3956da0 0x55b4a3e00660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:05.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.406+0000 7fa63ae5d740 4 client.26285 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:21:05.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.406+0000 7fa63ae5d740 1 -- 172.21.15.134:0/3700655111 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b4a3cdc5c0 con 0x55b4a3956da0 2024-07-26T11:21:05.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.407+0000 7fa630e00640 1 --2- 172.21.15.134:0/3700655111 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55b4a3956da0 0x55b4a3e00660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:05.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.408+0000 7fa630e00640 1 --2- 172.21.15.134:0/3700655111 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55b4a3956da0 0x55b4a3e00660 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5fbc1498eb1d751 server_cookie=dc70eead02caf79d in_seq=0 out_seq=0 2024-07-26T11:21:05.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.408+0000 7fa60f400640 10 client.26285 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:05.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.409+0000 7fa60f400640 1 -- 172.21.15.134:0/3700655111 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b4a3cdc5c0 con 0x55b4a3956da0 2024-07-26T11:21:05.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.409+0000 7fa60f400640 10 client.26285 handle_command_reply: tid=0 2024-07-26T11:21:05.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.409+0000 7fa63ae5d740 1 client.26285 shutdown 2024-07-26T11:21:05.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.412+0000 7fa63ae5d740 1 -- 172.21.15.134:0/3700655111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 msgr2=0x55b4a3d98b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:05.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.412+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/3700655111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 0x55b4a3d98b20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa62402f480 tx=0x7fa6240318d0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.412+0000 7fa63ae5d740 1 -- 172.21.15.134:0/3700655111 shutdown_connections 2024-07-26T11:21:05.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.412+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/3700655111 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55b4a3956da0 0x55b4a3e00660 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.412+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/3700655111 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4a39a2390 0x55b4a3d990a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.194 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.412+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/3700655111 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4a39c3070 0x55b4a3dfcd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.194 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.412+0000 7fa63ae5d740 1 --2- 172.21.15.134:0/3700655111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4a399a1b0 0x55b4a3d98b20 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.194 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.412+0000 7fa63ae5d740 1 -- 172.21.15.134:0/3700655111 >> 172.21.15.134:0/3700655111 conn(0x55b4a39b7460 msgr2=0x55b4a3d8fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:05.194 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.412+0000 7fa60f400640 0 client.26285 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:05.194 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.412+0000 7fa63ae5d740 1 -- 172.21.15.134:0/3700655111 shutdown_connections 2024-07-26T11:21:05.194 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.412+0000 7fa63ae5d740 1 -- 172.21.15.134:0/3700655111 wait complete. 2024-07-26T11:21:05.194 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.413+0000 7fa63ae5d740 20 client.26285 trim_cache size 0 max 16384 2024-07-26T11:21:05.194 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:21:05.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.422+0000 7fa632c00640 1 -- 172.21.15.134:0/390518198 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6140b3e40 msgr2=0x7fa6140b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:05.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.422+0000 7fa632c00640 1 --2- 172.21.15.134:0/390518198 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6140b3e40 0x7fa6140b6350 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x55b4a3d6f4a0 tx=0x7fa61c006df0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.422+0000 7fa632c00640 1 -- 172.21.15.134:0/390518198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa62c10b310 msgr2=0x7fa62c0786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:05.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.422+0000 7fa632c00640 1 --2- 172.21.15.134:0/390518198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa62c10b310 0x7fa62c0786e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00b450 tx=0x7fa61c00b910 comp rx=0 tx=0).stop 2024-07-26T11:21:05.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.422+0000 7fa632c00640 1 -- 172.21.15.134:0/390518198 shutdown_connections 2024-07-26T11:21:05.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.422+0000 7fa632c00640 1 --2- 172.21.15.134:0/390518198 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa6140b3e40 0x7fa6140b6350 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.422+0000 7fa632c00640 1 --2- 172.21.15.134:0/390518198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa62c10f2f0 0x7fa62c078c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.422+0000 7fa632c00640 1 --2- 172.21.15.134:0/390518198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa62c10fc40 0x7fa62c1c1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.422+0000 7fa632c00640 1 --2- 172.21.15.134:0/390518198 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa62c10b310 0x7fa62c0786e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:05.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.422+0000 7fa632c00640 1 -- 172.21.15.134:0/390518198 >> 172.21.15.134:0/390518198 conn(0x7fa62c06e420 msgr2=0x7fa62c10e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:05.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.423+0000 7fa632c00640 1 -- 172.21.15.134:0/390518198 shutdown_connections 2024-07-26T11:21:05.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:04.423+0000 7fa632c00640 1 -- 172.21.15.134:0/390518198 wait complete. 2024-07-26T11:21:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:05 smithi193 ceph-mon[37473]: pgmap v633: 129 pgs: 129 active+clean; 35 GiB data, 72 GiB used, 1000 GiB / 1.0 TiB avail; 39 KiB/s rd, 268 MiB/s wr, 814 op/s 2024-07-26T11:21:05.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:05 smithi134 ceph-mon[31966]: pgmap v633: 129 pgs: 129 active+clean; 35 GiB data, 72 GiB used, 1000 GiB / 1.0 TiB avail; 39 KiB/s rd, 268 MiB/s wr, 814 op/s 2024-07-26T11:21:05.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:05 smithi189 ceph-mon[36790]: pgmap v633: 129 pgs: 129 active+clean; 35 GiB data, 72 GiB used, 1000 GiB / 1.0 TiB avail; 39 KiB/s rd, 268 MiB/s wr, 814 op/s 2024-07-26T11:21:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:07 smithi193 ceph-mon[37473]: pgmap v634: 129 pgs: 129 active+clean; 35 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 38 KiB/s rd, 283 MiB/s wr, 972 op/s 2024-07-26T11:21:07.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:07 smithi134 ceph-mon[31966]: pgmap v634: 129 pgs: 129 active+clean; 35 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 38 KiB/s rd, 283 MiB/s wr, 972 op/s 2024-07-26T11:21:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:07 smithi189 ceph-mon[36790]: pgmap v634: 129 pgs: 129 active+clean; 35 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 38 KiB/s rd, 283 MiB/s wr, 972 op/s 2024-07-26T11:21:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:08 smithi193 ceph-mon[37473]: pgmap v635: 129 pgs: 129 active+clean; 35 GiB data, 73 GiB used, 999 GiB / 1.0 TiB avail; 53 KiB/s rd, 268 MiB/s wr, 1.17k op/s 2024-07-26T11:21:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:08 smithi134 ceph-mon[31966]: pgmap v635: 129 pgs: 129 active+clean; 35 GiB data, 73 GiB used, 999 GiB / 1.0 TiB avail; 53 KiB/s rd, 268 MiB/s wr, 1.17k op/s 2024-07-26T11:21:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:08 smithi189 ceph-mon[36790]: pgmap v635: 129 pgs: 129 active+clean; 35 GiB data, 73 GiB used, 999 GiB / 1.0 TiB avail; 53 KiB/s rd, 268 MiB/s wr, 1.17k op/s 2024-07-26T11:21:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.666+0000 7f4842a00640 1 -- 172.21.15.134:0/2635975302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f483c1101d0 msgr2=0x7f483c116a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.666+0000 7f4842a00640 1 --2- 172.21.15.134:0/2635975302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f483c1101d0 0x7f483c116a00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4834005fd0 tx=0x7f483402b410 comp rx=0 tx=0).stop 2024-07-26T11:21:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.666+0000 7f4842a00640 1 -- 172.21.15.134:0/2635975302 shutdown_connections 2024-07-26T11:21:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.666+0000 7f4842a00640 1 --2- 172.21.15.134:0/2635975302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f483c10f880 0x7f483c10fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.666+0000 7f4842a00640 1 --2- 172.21.15.134:0/2635975302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f483c1101d0 0x7f483c116a00 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.666+0000 7f4842a00640 1 --2- 172.21.15.134:0/2635975302 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f483c10a090 0x7f483c10a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.666+0000 7f4842a00640 1 -- 172.21.15.134:0/2635975302 >> 172.21.15.134:0/2635975302 conn(0x7f483c070800 msgr2=0x7f483c070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:09.669 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.667+0000 7f4842a00640 1 -- 172.21.15.134:0/2635975302 shutdown_connections 2024-07-26T11:21:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.667+0000 7f4842a00640 1 -- 172.21.15.134:0/2635975302 wait complete. 2024-07-26T11:21:09.670 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.668+0000 7f4842a00640 1 Processor -- start 2024-07-26T11:21:09.671 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.669+0000 7f4842a00640 1 -- start start 2024-07-26T11:21:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.670+0000 7f4842a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f483c10a090 0x7f483c1a7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.672 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.670+0000 7f4842a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f483c10f880 0x7f483c1a02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.670+0000 7f4842a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f483c1101d0 0x7f483c1a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.670+0000 7f4842a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f483c119a40 con 0x7f483c10a090 2024-07-26T11:21:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.671+0000 7f4842a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f483c1198c0 con 0x7f483c1101d0 2024-07-26T11:21:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.671+0000 7f4842a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f483c119bc0 con 0x7f483c10f880 2024-07-26T11:21:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.671+0000 7f4840a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f483c1101d0 0x7f483c1a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.671+0000 7f4840a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f483c1101d0 0x7f483c1a0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46500/0 (socket says 172.21.15.134:46500) 2024-07-26T11:21:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.671+0000 7f4840a00640 1 -- 172.21.15.134:0/68247432 learned_addr learned my addr 172.21.15.134:0/68247432 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:09.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.671+0000 7f483be00640 1 --2- 172.21.15.134:0/68247432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f483c10a090 0x7f483c1a7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.674 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.672+0000 7f483b400640 1 --2- 172.21.15.134:0/68247432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f483c10f880 0x7f483c1a02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.673+0000 7f483b400640 1 -- 172.21.15.134:0/68247432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f483c1101d0 msgr2=0x7f483c1a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:09.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.673+0000 7f483b400640 1 --2- 172.21.15.134:0/68247432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f483c1101d0 0x7f483c1a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.674+0000 7f483b400640 1 -- 172.21.15.134:0/68247432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f483c10a090 msgr2=0x7f483c1a7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.674+0000 7f483b400640 1 --2- 172.21.15.134:0/68247432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f483c10a090 0x7f483c1a7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.674+0000 7f483b400640 1 -- 172.21.15.134:0/68247432 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f483c1a5c80 con 0x7f483c10f880 2024-07-26T11:21:09.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.674+0000 7f4840a00640 1 --2- 172.21.15.134:0/68247432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f483c1101d0 0x7f483c1a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:21:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.674+0000 7f483be00640 1 --2- 172.21.15.134:0/68247432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f483c10a090 0x7f483c1a7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:21:09.678 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.675+0000 7f483b400640 1 --2- 172.21.15.134:0/68247432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f483c10f880 0x7f483c1a02e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f483c06c280 tx=0x7f482800db30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.676+0000 7f4838c00640 1 -- 172.21.15.134:0/68247432 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4828017070 con 0x7f483c10f880 2024-07-26T11:21:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.676+0000 7f4838c00640 1 -- 172.21.15.134:0/68247432 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4828009820 con 0x7f483c10f880 2024-07-26T11:21:09.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.676+0000 7f4838c00640 1 -- 172.21.15.134:0/68247432 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4828004b90 con 0x7f483c10f880 2024-07-26T11:21:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.678+0000 7f4842a00640 1 -- 172.21.15.134:0/68247432 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f483c1a5f70 con 0x7f483c10f880 2024-07-26T11:21:09.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.679+0000 7f4842a00640 1 -- 172.21.15.134:0/68247432 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f483c1a6440 con 0x7f483c10f880 2024-07-26T11:21:09.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.694+0000 7f4838c00640 1 -- 172.21.15.134:0/68247432 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f4828004d30 con 0x7f483c10f880 2024-07-26T11:21:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.696+0000 7f4849a63740 1 Processor -- start 2024-07-26T11:21:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.698+0000 7f4838c00640 1 --2- 172.21.15.134:0/68247432 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f48200b3da0 0x7f48200b62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.698+0000 7f4838c00640 1 -- 172.21.15.134:0/68247432 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f48280e3f60 con 0x7f483c10f880 2024-07-26T11:21:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.701+0000 7f4849a63740 1 -- start start 2024-07-26T11:21:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.701+0000 7f4849a63740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c921b0 0x558e01035a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.701+0000 7f4849a63740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 0x558e01035f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.701+0000 7f4849a63740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 0x558e010364c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.701+0000 7f4849a63740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x558e00faa270 con 0x558e00c921b0 2024-07-26T11:21:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.701+0000 7f4849a63740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558e00f99770 con 0x558e00c714d0 2024-07-26T11:21:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.701+0000 7f4849a63740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558e00f9d7b0 con 0x558e00c692f0 2024-07-26T11:21:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.702+0000 7f483b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 0x558e01035f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.702+0000 7f483b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 0x558e01035f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:35934/0 (socket says 172.21.15.134:35934) 2024-07-26T11:21:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.702+0000 7f483b400640 1 -- 172.21.15.134:0/1723952233 learned_addr learned my addr 172.21.15.134:0/1723952233 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.702+0000 7f483be00640 1 --2- 172.21.15.134:0/1723952233 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c921b0 0x558e01035a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.702+0000 7f483b400640 1 -- 172.21.15.134:0/1723952233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 msgr2=0x558e010364c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.702+0000 7f4840a00640 1 --2- 172.21.15.134:0/1723952233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 0x558e010364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.702+0000 7f483b400640 1 --2- 172.21.15.134:0/1723952233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 0x558e010364c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.702+0000 7f483b400640 1 -- 172.21.15.134:0/1723952233 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c921b0 msgr2=0x558e01035a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.702+0000 7f483b400640 1 --2- 172.21.15.134:0/1723952233 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c921b0 0x558e01035a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.702+0000 7f483b400640 1 -- 172.21.15.134:0/1723952233 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e00cfd790 con 0x558e00c692f0 2024-07-26T11:21:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.702+0000 7f4840a00640 1 --2- 172.21.15.134:0/1723952233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 0x558e010364c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:21:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.703+0000 7f4840a00640 1 --2- 172.21.15.134:0/68247432 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f48200b3da0 0x7f48200b62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.703+0000 7f483b400640 1 --2- 172.21.15.134:0/1723952233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 0x558e01035f80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f48280aa780 tx=0x7f48280aa3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70b094266edc4277 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.703+0000 7f4831200640 1 -- 172.21.15.134:0/1723952233 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48280b3040 con 0x558e00c692f0 2024-07-26T11:21:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.704+0000 7f4831200640 1 -- 172.21.15.134:0/1723952233 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48280aa7f0 con 0x558e00c692f0 2024-07-26T11:21:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.704+0000 7f4831200640 1 -- 172.21.15.134:0/1723952233 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48280199d0 con 0x558e00c692f0 2024-07-26T11:21:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.705+0000 7f4849a63740 1 -- 172.21.15.134:0/1723952233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 msgr2=0x558e01035f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.705+0000 7f4849a63740 1 --2- 172.21.15.134:0/1723952233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 0x558e01035f80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f48280aa780 tx=0x7f48280aa3c0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.706+0000 7f4849a63740 1 -- 172.21.15.134:0/1723952233 shutdown_connections 2024-07-26T11:21:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.706+0000 7f4849a63740 1 --2- 172.21.15.134:0/1723952233 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 0x558e01035f80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.706+0000 7f4849a63740 1 --2- 172.21.15.134:0/1723952233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 0x558e010364c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.706+0000 7f4849a63740 1 --2- 172.21.15.134:0/1723952233 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c921b0 0x558e01035a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.706+0000 7f4849a63740 1 -- 172.21.15.134:0/1723952233 >> 172.21.15.134:0/1723952233 conn(0x558e00c86300 msgr2=0x558e01033930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.707+0000 7f4849a63740 1 -- 172.21.15.134:0/1723952233 shutdown_connections 2024-07-26T11:21:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.708+0000 7f4849a63740 1 -- 172.21.15.134:0/1723952233 wait complete. 2024-07-26T11:21:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.708+0000 7f4849a63740 1 -- start start 2024-07-26T11:21:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.709+0000 7f4849a63740 1 Processor -- start 2024-07-26T11:21:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.711+0000 7f4849a63740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 0x558e010416d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.711+0000 7f4849a63740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 0x558e01041cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.711+0000 7f4849a63740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c921b0 0x558e0104d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.711+0000 7f4849a63740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x558e00ed3ab0 con 0x558e00c921b0 2024-07-26T11:21:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.711+0000 7f4849a63740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558e00f36d50 con 0x558e00c714d0 2024-07-26T11:21:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.711+0000 7f4849a63740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558e01044160 con 0x558e00c692f0 2024-07-26T11:21:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.711+0000 7f4849a63740 10 client.0 _ll_register_callbacks cb 0x558e00ca9a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:09.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.715+0000 7f4840a00640 1 --2- 172.21.15.134:0/68247432 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f48200b3da0 0x7f48200b62b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f483400ade0 tx=0x7f4834004a80 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.715+0000 7f483b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 0x558e01041cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.715+0000 7f483b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 0x558e01041cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46526/0 (socket says 172.21.15.134:46526) 2024-07-26T11:21:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.715+0000 7f483b400640 1 -- 172.21.15.134:0/1378561682 learned_addr learned my addr 172.21.15.134:0/1378561682 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.716+0000 7f483be00640 1 --2- 172.21.15.134:0/1378561682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 0x558e010416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.716+0000 7f483b400640 1 -- 172.21.15.134:0/1378561682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 msgr2=0x558e010416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.716+0000 7f483b400640 1 --2- 172.21.15.134:0/1378561682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 0x558e010416d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.716+0000 7f483b400640 1 -- 172.21.15.134:0/1378561682 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c921b0 msgr2=0x558e0104d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.716+0000 7f483b400640 1 --2- 172.21.15.134:0/1378561682 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c921b0 0x558e0104d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.716+0000 7f483b400640 1 -- 172.21.15.134:0/1378561682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e00d76620 con 0x558e00c714d0 2024-07-26T11:21:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.716+0000 7f483b400640 1 --2- 172.21.15.134:0/1378561682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 0x558e01041cb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4828005890 tx=0x7f48280a67e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5c1534d77868b1f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.716+0000 7f4825600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:21:09.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.717+0000 7f4825600640 1 -- 172.21.15.134:0/1378561682 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48280b3040 con 0x558e00c714d0 2024-07-26T11:21:09.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.717+0000 7f4825600640 1 -- 172.21.15.134:0/1378561682 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f48280cd0c0 con 0x558e00c714d0 2024-07-26T11:21:09.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.717+0000 7f4849a63740 1 -- 172.21.15.134:0/1378561682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48280aa7f0 con 0x558e00c714d0 2024-07-26T11:21:09.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.718+0000 7f4825600640 1 -- 172.21.15.134:0/1378561682 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f48280b38a0 con 0x558e00c714d0 2024-07-26T11:21:09.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.718+0000 7f4849a63740 10 client.26291 fetch_fsmap learned FSMap version 32 2024-07-26T11:21:09.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.718+0000 7f4849a63740 1 -- 172.21.15.134:0/1378561682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558e010501a0 con 0x558e00c714d0 2024-07-26T11:21:09.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.718+0000 7f4825600640 1 -- 172.21.15.134:0/1378561682 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48280aa7f0 con 0x558e00c714d0 2024-07-26T11:21:09.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.718+0000 7f4825600640 1 -- 172.21.15.134:0/1378561682 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f48280a9a50 con 0x558e00c714d0 2024-07-26T11:21:09.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.721+0000 7f4849a63740 10 client.26291 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:21:09.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.721+0000 7f4849a63740 10 client.26291 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:21:09.724 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.721+0000 7f4849a63740 20 client.26291 populate_metadata read hostname 'smithi134' 2024-07-26T11:21:09.724 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.721+0000 7f4849a63740 1 --2- 172.21.15.134:0/1378561682 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558e00c50d80 0x558e01053500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.724 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.721+0000 7f4849a63740 4 client.26291 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:21:09.724 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.721+0000 7f4849a63740 1 -- 172.21.15.134:0/1378561682 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558e00fac1c0 con 0x558e00c50d80 2024-07-26T11:21:09.724 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.722+0000 7f483be00640 1 --2- 172.21.15.134:0/1378561682 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558e00c50d80 0x558e01053500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.726+0000 7f483be00640 1 --2- 172.21.15.134:0/1378561682 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558e00c50d80 0x558e01053500 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13540d2619069476 server_cookie=83e8f30755b527df in_seq=0 out_seq=0 2024-07-26T11:21:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.726+0000 7f4825600640 10 client.26291 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:09.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.728+0000 7f4825600640 1 -- 172.21.15.134:0/1378561682 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558e00fac1c0 con 0x558e00c50d80 2024-07-26T11:21:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.728+0000 7f4825600640 10 client.26291 handle_command_reply: tid=0 2024-07-26T11:21:09.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.728+0000 7f4849a63740 1 client.26291 shutdown 2024-07-26T11:21:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.731+0000 7f4849a63740 1 -- 172.21.15.134:0/1378561682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 msgr2=0x558e01041cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:09.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.731+0000 7f4849a63740 1 --2- 172.21.15.134:0/1378561682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 0x558e01041cb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4828005890 tx=0x7f48280a67e0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.731+0000 7f4849a63740 1 -- 172.21.15.134:0/1378561682 shutdown_connections 2024-07-26T11:21:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.732+0000 7f4849a63740 1 --2- 172.21.15.134:0/1378561682 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558e00c50d80 0x558e01053500 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.732+0000 7f4849a63740 1 --2- 172.21.15.134:0/1378561682 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c921b0 0x558e0104d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.732+0000 7f4825600640 0 client.26291 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.732+0000 7f4849a63740 1 --2- 172.21.15.134:0/1378561682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 0x558e01041cb0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.732+0000 7f4849a63740 1 --2- 172.21.15.134:0/1378561682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 0x558e010416d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.733+0000 7f4849a63740 1 -- 172.21.15.134:0/1378561682 >> 172.21.15.134:0/1378561682 conn(0x558e00c86300 msgr2=0x558e01036a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:09.736 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.734+0000 7f4849a63740 1 -- 172.21.15.134:0/1378561682 shutdown_connections 2024-07-26T11:21:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.735+0000 7f4849a63740 1 -- 172.21.15.134:0/1378561682 wait complete. 2024-07-26T11:21:09.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.735+0000 7f4849a63740 20 client.26291 trim_cache size 0 max 16384 2024-07-26T11:21:09.894 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.892+0000 7f4849a63740 1 Processor -- start 2024-07-26T11:21:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.893+0000 7f4849a63740 1 -- start start 2024-07-26T11:21:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.893+0000 7f4849a63740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c50d80 0x558e010370e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.895 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.893+0000 7f4849a63740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c692f0 0x558e010410c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.893+0000 7f4849a63740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c714d0 0x558e0103ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.893+0000 7f4849a63740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f483c1c8080 con 0x558e00c714d0 2024-07-26T11:21:09.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.893+0000 7f4849a63740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f483c1c9bc0 con 0x558e00c692f0 2024-07-26T11:21:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.893+0000 7f4849a63740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f483c1c8750 con 0x558e00c50d80 2024-07-26T11:21:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.893+0000 7f483b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c692f0 0x558e010410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.893+0000 7f483b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c692f0 0x558e010410c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34124/0 (socket says 172.21.15.134:34124) 2024-07-26T11:21:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.893+0000 7f483b400640 1 -- 172.21.15.134:0/3704001069 learned_addr learned my addr 172.21.15.134:0/3704001069 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.894+0000 7f4840a00640 1 --2- 172.21.15.134:0/3704001069 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c714d0 0x558e0103ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.894+0000 7f483be00640 1 --2- 172.21.15.134:0/3704001069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c50d80 0x558e010370e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.894+0000 7f483b400640 1 -- 172.21.15.134:0/3704001069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c50d80 msgr2=0x558e010370e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.894+0000 7f483b400640 1 --2- 172.21.15.134:0/3704001069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c50d80 0x558e010370e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.894+0000 7f483b400640 1 -- 172.21.15.134:0/3704001069 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c714d0 msgr2=0x558e0103ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.894+0000 7f483b400640 1 --2- 172.21.15.134:0/3704001069 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c714d0 0x558e0103ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.894+0000 7f483b400640 1 -- 172.21.15.134:0/3704001069 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558e010346c0 con 0x558e00c692f0 2024-07-26T11:21:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.894+0000 7f483be00640 1 --2- 172.21.15.134:0/3704001069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c50d80 0x558e010370e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:21:09.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.894+0000 7f4840a00640 1 --2- 172.21.15.134:0/3704001069 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c714d0 0x558e0103ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:21:09.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.894+0000 7f483b400640 1 --2- 172.21.15.134:0/3704001069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c692f0 0x558e010410c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f48280a7a60 tx=0x7f48280a60b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dbf3a2ef40d3ec0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:09.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.894+0000 7f4827e00640 1 -- 172.21.15.134:0/3704001069 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48280bd070 con 0x558e00c692f0 2024-07-26T11:21:09.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.895+0000 7f4827e00640 1 -- 172.21.15.134:0/3704001069 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48280aa7f0 con 0x558e00c692f0 2024-07-26T11:21:09.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.895+0000 7f4827e00640 1 -- 172.21.15.134:0/3704001069 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48280199d0 con 0x558e00c692f0 2024-07-26T11:21:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.896+0000 7f4849a63740 1 -- 172.21.15.134:0/3704001069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c692f0 msgr2=0x558e010410c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.896+0000 7f4849a63740 1 --2- 172.21.15.134:0/3704001069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c692f0 0x558e010410c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f48280a7a60 tx=0x7f48280a60b0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.896+0000 7f4849a63740 1 -- 172.21.15.134:0/3704001069 shutdown_connections 2024-07-26T11:21:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.896+0000 7f4849a63740 1 --2- 172.21.15.134:0/3704001069 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c714d0 0x558e0103ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.896+0000 7f4849a63740 1 --2- 172.21.15.134:0/3704001069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c692f0 0x558e010410c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.896+0000 7f4849a63740 1 --2- 172.21.15.134:0/3704001069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c50d80 0x558e010370e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.896+0000 7f4849a63740 1 -- 172.21.15.134:0/3704001069 >> 172.21.15.134:0/3704001069 conn(0x558e00c86300 msgr2=0x558e010409b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.897+0000 7f4849a63740 1 -- 172.21.15.134:0/3704001069 shutdown_connections 2024-07-26T11:21:09.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.897+0000 7f4849a63740 1 -- 172.21.15.134:0/3704001069 wait complete. 2024-07-26T11:21:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.897+0000 7f4849a63740 1 -- start start 2024-07-26T11:21:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.898+0000 7f4849a63740 1 Processor -- start 2024-07-26T11:21:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.898+0000 7f4849a63740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c50d80 0x558e01053fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.898+0000 7f4849a63740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 0x558e01054550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.898+0000 7f4849a63740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 0x558e010b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.898+0000 7f4849a63740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f483c1c94f0 con 0x558e00c50d80 2024-07-26T11:21:09.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.898+0000 7f4849a63740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f483c1c8e20 con 0x558e00c714d0 2024-07-26T11:21:09.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.898+0000 7f4849a63740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f483c1c6060 con 0x558e00c692f0 2024-07-26T11:21:09.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.898+0000 7f4849a63740 10 client.0 _ll_register_callbacks cb 0x7f48280b31e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.899+0000 7f483b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 0x558e01054550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.899+0000 7f483b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 0x558e01054550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57706/0 (socket says 172.21.15.134:57706) 2024-07-26T11:21:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.899+0000 7f483b400640 1 -- 172.21.15.134:0/1853826535 learned_addr learned my addr 172.21.15.134:0/1853826535 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.899+0000 7f483be00640 1 --2- 172.21.15.134:0/1853826535 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c50d80 0x558e01053fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.899+0000 7f4840a00640 1 --2- 172.21.15.134:0/1853826535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 0x558e010b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.899+0000 7f483b400640 1 -- 172.21.15.134:0/1853826535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 msgr2=0x558e010b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.899+0000 7f483b400640 1 --2- 172.21.15.134:0/1853826535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 0x558e010b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.899+0000 7f483b400640 1 -- 172.21.15.134:0/1853826535 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c50d80 msgr2=0x558e01053fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:09.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.899+0000 7f483b400640 1 --2- 172.21.15.134:0/1853826535 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c50d80 0x558e01053fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.899+0000 7f483b400640 1 -- 172.21.15.134:0/1853826535 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558e010b4250 con 0x558e00c692f0 2024-07-26T11:21:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.899+0000 7f4840a00640 1 --2- 172.21.15.134:0/1853826535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 0x558e010b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:21:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.900+0000 7f483b400640 1 --2- 172.21.15.134:0/1853826535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 0x558e01054550 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f48280aa4a0 tx=0x7f48280a9c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2769c230832737d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:09.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.900+0000 7f4825600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:21:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.900+0000 7f4825600640 1 -- 172.21.15.134:0/1853826535 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48280bd070 con 0x558e00c692f0 2024-07-26T11:21:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.900+0000 7f4825600640 1 -- 172.21.15.134:0/1853826535 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4828019be0 con 0x558e00c692f0 2024-07-26T11:21:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.900+0000 7f4849a63740 1 -- 172.21.15.134:0/1853826535 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f48280aa7f0 con 0x558e00c692f0 2024-07-26T11:21:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.901+0000 7f4825600640 1 -- 172.21.15.134:0/1853826535 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f48280cd0b0 con 0x558e00c692f0 2024-07-26T11:21:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.901+0000 7f4849a63740 10 client.26302 fetch_fsmap learned FSMap version 32 2024-07-26T11:21:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.901+0000 7f4849a63740 1 -- 172.21.15.134:0/1853826535 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558e010bb550 con 0x558e00c692f0 2024-07-26T11:21:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.901+0000 7f4825600640 1 -- 172.21.15.134:0/1853826535 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f48280aa7f0 con 0x558e00c692f0 2024-07-26T11:21:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.901+0000 7f4825600640 1 -- 172.21.15.134:0/1853826535 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f48280b3af0 con 0x558e00c692f0 2024-07-26T11:21:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.901+0000 7f4849a63740 10 client.26302 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:21:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.901+0000 7f4849a63740 10 client.26302 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:21:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.901+0000 7f4849a63740 20 client.26302 populate_metadata read hostname 'smithi134' 2024-07-26T11:21:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.901+0000 7f4849a63740 1 --2- 172.21.15.134:0/1853826535 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558e00c921b0 0x558e010bd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:09.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.901+0000 7f4849a63740 4 client.26302 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:21:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.901+0000 7f4849a63740 1 -- 172.21.15.134:0/1853826535 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558e00fa7c60 con 0x558e00c921b0 2024-07-26T11:21:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.902+0000 7f483be00640 1 --2- 172.21.15.134:0/1853826535 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558e00c921b0 0x558e010bd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.903+0000 7f483be00640 1 --2- 172.21.15.134:0/1853826535 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558e00c921b0 0x558e010bd9d0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5ff7ae6d7c18a3e server_cookie=63ee9157f757ba30 in_seq=0 out_seq=0 2024-07-26T11:21:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.903+0000 7f4825600640 10 client.26302 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:09.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.941+0000 7f4825600640 1 -- 172.21.15.134:0/1853826535 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558e00fa7c60 con 0x558e00c921b0 2024-07-26T11:21:09.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.941+0000 7f4825600640 10 client.26302 handle_command_reply: tid=0 2024-07-26T11:21:09.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.941+0000 7f4849a63740 1 client.26302 shutdown 2024-07-26T11:21:09.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.941+0000 7f4849a63740 1 -- 172.21.15.134:0/1853826535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 msgr2=0x558e01054550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:09.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.941+0000 7f4849a63740 1 --2- 172.21.15.134:0/1853826535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 0x558e01054550 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f48280aa4a0 tx=0x7f48280a9c80 comp rx=0 tx=0).stop 2024-07-26T11:21:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.941+0000 7f4849a63740 1 -- 172.21.15.134:0/1853826535 shutdown_connections 2024-07-26T11:21:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.941+0000 7f4849a63740 1 --2- 172.21.15.134:0/1853826535 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558e00c921b0 0x558e010bd9d0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.941+0000 7f4849a63740 1 --2- 172.21.15.134:0/1853826535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558e00c692f0 0x558e01054550 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.944 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.941+0000 7f4849a63740 1 --2- 172.21.15.134:0/1853826535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e00c714d0 0x558e010b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.941+0000 7f4849a63740 1 --2- 172.21.15.134:0/1853826535 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558e00c50d80 0x558e01053fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.941+0000 7f4849a63740 1 -- 172.21.15.134:0/1853826535 >> 172.21.15.134:0/1853826535 conn(0x558e00c86300 msgr2=0x558e0103f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.941+0000 7f4825600640 0 client.26302 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.942+0000 7f4849a63740 1 -- 172.21.15.134:0/1853826535 shutdown_connections 2024-07-26T11:21:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.942+0000 7f4849a63740 1 -- 172.21.15.134:0/1853826535 wait complete. 2024-07-26T11:21:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.942+0000 7f4849a63740 20 client.26302 trim_cache size 0 max 16384 2024-07-26T11:21:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:21:09.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:21:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:21:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:21:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.943+0000 7f4842a00640 1 -- 172.21.15.134:0/68247432 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f48200b3da0 msgr2=0x7f48200b62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.943+0000 7f4842a00640 1 --2- 172.21.15.134:0/68247432 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f48200b3da0 0x7f48200b62b0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f483400ade0 tx=0x7f4834004a80 comp rx=0 tx=0).stop 2024-07-26T11:21:09.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.943+0000 7f4842a00640 1 -- 172.21.15.134:0/68247432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f483c10f880 msgr2=0x7f483c1a02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.943+0000 7f4842a00640 1 --2- 172.21.15.134:0/68247432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f483c10f880 0x7f483c1a02e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f483c06c280 tx=0x7f482800db30 comp rx=0 tx=0).stop 2024-07-26T11:21:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.943+0000 7f4842a00640 1 -- 172.21.15.134:0/68247432 shutdown_connections 2024-07-26T11:21:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.943+0000 7f4842a00640 1 --2- 172.21.15.134:0/68247432 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f48200b3da0 0x7f48200b62b0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.943+0000 7f4842a00640 1 --2- 172.21.15.134:0/68247432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f483c10f880 0x7f483c1a02e0 secure :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f483c06c280 tx=0x7f482800db30 comp rx=0 tx=0).stop 2024-07-26T11:21:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.943+0000 7f4842a00640 1 --2- 172.21.15.134:0/68247432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f483c1101d0 0x7f483c1a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.943+0000 7f4842a00640 1 --2- 172.21.15.134:0/68247432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f483c10a090 0x7f483c1a7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.943+0000 7f4842a00640 1 -- 172.21.15.134:0/68247432 >> 172.21.15.134:0/68247432 conn(0x7f483c070800 msgr2=0x7f483c10d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.943+0000 7f4842a00640 1 -- 172.21.15.134:0/68247432 shutdown_connections 2024-07-26T11:21:09.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:09.943+0000 7f4842a00640 1 -- 172.21.15.134:0/68247432 wait complete. 2024-07-26T11:21:11.051 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T11:21:11.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T11:21:11.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:11 smithi189 ceph-mon[36790]: pgmap v636: 129 pgs: 129 active+clean; 35 GiB data, 74 GiB used, 999 GiB / 1.0 TiB avail; 37 KiB/s rd, 288 MiB/s wr, 1.53k op/s 2024-07-26T11:21:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:11 smithi193 ceph-mon[37473]: pgmap v636: 129 pgs: 129 active+clean; 35 GiB data, 74 GiB used, 999 GiB / 1.0 TiB avail; 37 KiB/s rd, 288 MiB/s wr, 1.53k op/s 2024-07-26T11:21:11.926 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:11 smithi134 ceph-mon[31966]: pgmap v636: 129 pgs: 129 active+clean; 35 GiB data, 74 GiB used, 999 GiB / 1.0 TiB avail; 37 KiB/s rd, 288 MiB/s wr, 1.53k op/s 2024-07-26T11:21:12.081 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:21:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:13 smithi193 ceph-mon[37473]: pgmap v637: 129 pgs: 129 active+clean; 35 GiB data, 74 GiB used, 999 GiB / 1.0 TiB avail; 53 KiB/s rd, 232 MiB/s wr, 1.40k op/s 2024-07-26T11:21:13.926 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:13 smithi134 ceph-mon[31966]: pgmap v637: 129 pgs: 129 active+clean; 35 GiB data, 74 GiB used, 999 GiB / 1.0 TiB avail; 53 KiB/s rd, 232 MiB/s wr, 1.40k op/s 2024-07-26T11:21:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:13 smithi189 ceph-mon[36790]: pgmap v637: 129 pgs: 129 active+clean; 35 GiB data, 74 GiB used, 999 GiB / 1.0 TiB avail; 53 KiB/s rd, 232 MiB/s wr, 1.40k op/s 2024-07-26T11:21:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:15 smithi193 ceph-mon[37473]: pgmap v638: 129 pgs: 129 active+clean; 36 GiB data, 75 GiB used, 998 GiB / 1.0 TiB avail; 35 KiB/s rd, 245 MiB/s wr, 1.66k op/s 2024-07-26T11:21:15.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:15 smithi134 ceph-mon[31966]: pgmap v638: 129 pgs: 129 active+clean; 36 GiB data, 75 GiB used, 998 GiB / 1.0 TiB avail; 35 KiB/s rd, 245 MiB/s wr, 1.66k op/s 2024-07-26T11:21:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:15 smithi189 ceph-mon[36790]: pgmap v638: 129 pgs: 129 active+clean; 36 GiB data, 75 GiB used, 998 GiB / 1.0 TiB avail; 35 KiB/s rd, 245 MiB/s wr, 1.66k op/s 2024-07-26T11:21:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:17 smithi193 ceph-mon[37473]: pgmap v639: 129 pgs: 129 active+clean; 36 GiB data, 75 GiB used, 998 GiB / 1.0 TiB avail; 33 KiB/s rd, 262 MiB/s wr, 1.93k op/s 2024-07-26T11:21:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:17 smithi134 ceph-mon[31966]: pgmap v639: 129 pgs: 129 active+clean; 36 GiB data, 75 GiB used, 998 GiB / 1.0 TiB avail; 33 KiB/s rd, 262 MiB/s wr, 1.93k op/s 2024-07-26T11:21:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:17 smithi189 ceph-mon[36790]: pgmap v639: 129 pgs: 129 active+clean; 36 GiB data, 75 GiB used, 998 GiB / 1.0 TiB avail; 33 KiB/s rd, 262 MiB/s wr, 1.93k op/s 2024-07-26T11:21:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:18 smithi193 ceph-mon[37473]: pgmap v640: 129 pgs: 129 active+clean; 36 GiB data, 75 GiB used, 997 GiB / 1.0 TiB avail; 47 KiB/s rd, 244 MiB/s wr, 1.94k op/s 2024-07-26T11:21:18.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:18 smithi189 ceph-mon[36790]: pgmap v640: 129 pgs: 129 active+clean; 36 GiB data, 75 GiB used, 997 GiB / 1.0 TiB avail; 47 KiB/s rd, 244 MiB/s wr, 1.94k op/s 2024-07-26T11:21:18.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:18 smithi134 ceph-mon[31966]: pgmap v640: 129 pgs: 129 active+clean; 36 GiB data, 75 GiB used, 997 GiB / 1.0 TiB avail; 47 KiB/s rd, 244 MiB/s wr, 1.94k op/s 2024-07-26T11:21:20.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.274+0000 7f497fe00640 1 -- 172.21.15.134:0/3955783922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f498010f740 msgr2=0x7f498010fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:20.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.274+0000 7f497fe00640 1 --2- 172.21.15.134:0/3955783922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f498010f740 0x7f498010fb50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f496c00b9f0 tx=0x7f496c02c0f0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.274+0000 7f497fe00640 1 -- 172.21.15.134:0/3955783922 shutdown_connections 2024-07-26T11:21:20.280 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.274+0000 7f497fe00640 1 --2- 172.21.15.134:0/3955783922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f498010f740 0x7f498010fb50 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.274+0000 7f497fe00640 1 --2- 172.21.15.134:0/3955783922 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4980110090 0x7f49801168c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.274+0000 7f497fe00640 1 --2- 172.21.15.134:0/3955783922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f498010a080 0x7f498010a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.274+0000 7f497fe00640 1 -- 172.21.15.134:0/3955783922 >> 172.21.15.134:0/3955783922 conn(0x7f498006e420 msgr2=0x7f498006e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:20.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.278+0000 7f497fe00640 1 -- 172.21.15.134:0/3955783922 shutdown_connections 2024-07-26T11:21:20.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.278+0000 7f497fe00640 1 -- 172.21.15.134:0/3955783922 wait complete. 2024-07-26T11:21:20.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.281+0000 7f497fe00640 1 Processor -- start 2024-07-26T11:21:20.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.283+0000 7f497fe00640 1 -- start start 2024-07-26T11:21:20.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.286+0000 7f497fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f498010a080 0x7f49801a0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.291+0000 7f497fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f498010f740 0x7f49801a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.291+0000 7f497e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f498010f740 0x7f49801a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:20.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.291+0000 7f497e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f498010f740 0x7f49801a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:47348/0 (socket says 172.21.15.134:47348) 2024-07-26T11:21:20.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.291+0000 7f497fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4980110090 0x7f49801a58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.292+0000 7f497e000640 1 -- 172.21.15.134:0/3971724464 learned_addr learned my addr 172.21.15.134:0/3971724464 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:20.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.294+0000 7f497ea00640 1 --2- 172.21.15.134:0/3971724464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f498010a080 0x7f49801a0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:20.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.294+0000 7f497f400640 1 --2- 172.21.15.134:0/3971724464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4980110090 0x7f49801a58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:20.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.295+0000 7f497fe00640 1 -- 172.21.15.134:0/3971724464 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4980119950 con 0x7f498010a080 2024-07-26T11:21:20.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.295+0000 7f497fe00640 1 -- 172.21.15.134:0/3971724464 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f49801197d0 con 0x7f4980110090 2024-07-26T11:21:20.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.295+0000 7f497fe00640 1 -- 172.21.15.134:0/3971724464 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4980119ad0 con 0x7f498010f740 2024-07-26T11:21:20.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.300+0000 7f497f400640 1 -- 172.21.15.134:0/3971724464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f498010f740 msgr2=0x7f49801a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:20.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.300+0000 7f497f400640 1 --2- 172.21.15.134:0/3971724464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f498010f740 0x7f49801a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.300+0000 7f497f400640 1 -- 172.21.15.134:0/3971724464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f498010a080 msgr2=0x7f49801a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:20.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.300+0000 7f497f400640 1 --2- 172.21.15.134:0/3971724464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f498010a080 0x7f49801a0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.300+0000 7f497f400640 1 -- 172.21.15.134:0/3971724464 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49801a5fe0 con 0x7f4980110090 2024-07-26T11:21:20.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.301+0000 7f497ea00640 1 --2- 172.21.15.134:0/3971724464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f498010a080 0x7f49801a0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:21:20.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.301+0000 7f497e000640 1 --2- 172.21.15.134:0/3971724464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f498010f740 0x7f49801a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:21:20.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.302+0000 7f497f400640 1 --2- 172.21.15.134:0/3971724464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4980110090 0x7f49801a58b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f497800e940 tx=0x7f497800ee00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:20.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.302+0000 7f4977400640 1 -- 172.21.15.134:0/3971724464 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4978004d20 con 0x7f4980110090 2024-07-26T11:21:20.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.303+0000 7f4977400640 1 -- 172.21.15.134:0/3971724464 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4978005020 con 0x7f4980110090 2024-07-26T11:21:20.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.303+0000 7f497fe00640 1 -- 172.21.15.134:0/3971724464 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49801a62d0 con 0x7f4980110090 2024-07-26T11:21:20.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.303+0000 7f4977400640 1 -- 172.21.15.134:0/3971724464 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49780116d0 con 0x7f4980110090 2024-07-26T11:21:20.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.303+0000 7f497fe00640 1 -- 172.21.15.134:0/3971724464 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49801aca30 con 0x7f4980110090 2024-07-26T11:21:20.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.314+0000 7f4977400640 1 -- 172.21.15.134:0/3971724464 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f4978011870 con 0x7f4980110090 2024-07-26T11:21:20.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.324+0000 7f4977400640 1 --2- 172.21.15.134:0/3971724464 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f49640b3e00 0x7f49640b6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.325+0000 7f4977400640 1 -- 172.21.15.134:0/3971724464 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f49780e4460 con 0x7f4980110090 2024-07-26T11:21:20.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.332+0000 7f497ea00640 1 --2- 172.21.15.134:0/3971724464 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f49640b3e00 0x7f49640b6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:20.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.334+0000 7f497ea00640 1 --2- 172.21.15.134:0/3971724464 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f49640b3e00 0x7f49640b6310 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4970009bc0 tx=0x7f4970005f10 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:20.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.342+0000 7f498c5c1740 1 Processor -- start 2024-07-26T11:21:20.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.342+0000 7f498c5c1740 1 -- start start 2024-07-26T11:21:20.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.344+0000 7f498c5c1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8edd350 0x564ce9281f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.344+0000 7f498c5c1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8eb4490 0x564ce9282490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.344+0000 7f498c5c1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8ebc670 0x564ce92829d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.344+0000 7f498c5c1740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564ce91f8be0 con 0x564ce8edd350 2024-07-26T11:21:20.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.344+0000 7f498c5c1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564ce91e80e0 con 0x564ce8ebc670 2024-07-26T11:21:20.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.344+0000 7f498c5c1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564ce91ec120 con 0x564ce8eb4490 2024-07-26T11:21:20.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.346+0000 7f497f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8ebc670 0x564ce92829d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:20.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.346+0000 7f497f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8ebc670 0x564ce92829d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:53822/0 (socket says 172.21.15.134:53822) 2024-07-26T11:21:20.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.346+0000 7f497f400640 1 -- 172.21.15.134:0/1256450566 learned_addr learned my addr 172.21.15.134:0/1256450566 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:20.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.346+0000 7f497e000640 1 --2- 172.21.15.134:0/1256450566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8eb4490 0x564ce9282490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:20.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.347+0000 7f497f400640 1 -- 172.21.15.134:0/1256450566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8eb4490 msgr2=0x564ce9282490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:20.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.347+0000 7f497f400640 1 --2- 172.21.15.134:0/1256450566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8eb4490 0x564ce9282490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.347+0000 7f497f400640 1 -- 172.21.15.134:0/1256450566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8edd350 msgr2=0x564ce9281f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:20.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.347+0000 7f497f400640 1 --2- 172.21.15.134:0/1256450566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8edd350 0x564ce9281f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.349+0000 7f497f400640 1 -- 172.21.15.134:0/1256450566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ce8f48930 con 0x564ce8ebc670 2024-07-26T11:21:20.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.353+0000 7f497f400640 1 --2- 172.21.15.134:0/1256450566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8ebc670 0x564ce92829d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f49780a7cc0 tx=0x7f49780ab4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd7160544713e570 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:20.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.353+0000 7f496be00640 1 -- 172.21.15.134:0/1256450566 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f497801f040 con 0x564ce8ebc670 2024-07-26T11:21:20.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.354+0000 7f496be00640 1 -- 172.21.15.134:0/1256450566 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49780abc40 con 0x564ce8ebc670 2024-07-26T11:21:20.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.354+0000 7f496be00640 1 -- 172.21.15.134:0/1256450566 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4978005460 con 0x564ce8ebc670 2024-07-26T11:21:20.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.356+0000 7f498c5c1740 1 -- 172.21.15.134:0/1256450566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8ebc670 msgr2=0x564ce92829d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:20.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.356+0000 7f498c5c1740 1 --2- 172.21.15.134:0/1256450566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8ebc670 0x564ce92829d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f49780a7cc0 tx=0x7f49780ab4d0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.356+0000 7f498c5c1740 1 -- 172.21.15.134:0/1256450566 shutdown_connections 2024-07-26T11:21:20.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.356+0000 7f498c5c1740 1 --2- 172.21.15.134:0/1256450566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8eb4490 0x564ce9282490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.356+0000 7f498c5c1740 1 --2- 172.21.15.134:0/1256450566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8ebc670 0x564ce92829d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.356+0000 7f498c5c1740 1 --2- 172.21.15.134:0/1256450566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8edd350 0x564ce9281f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.356+0000 7f498c5c1740 1 -- 172.21.15.134:0/1256450566 >> 172.21.15.134:0/1256450566 conn(0x564ce8ed1740 msgr2=0x564ce927fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:20.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.357+0000 7f498c5c1740 1 -- 172.21.15.134:0/1256450566 shutdown_connections 2024-07-26T11:21:20.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.357+0000 7f498c5c1740 1 -- 172.21.15.134:0/1256450566 wait complete. 2024-07-26T11:21:20.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.358+0000 7f498c5c1740 1 -- start start 2024-07-26T11:21:20.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.362+0000 7f498c5c1740 1 Processor -- start 2024-07-26T11:21:20.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.363+0000 7f498c5c1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 0x564ce9289620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.367+0000 7f498c5c1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8ebc670 0x564ce9289b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.368+0000 7f497ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 0x564ce9289620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:20.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.368+0000 7f497ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 0x564ce9289620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:53838/0 (socket says 172.21.15.134:53838) 2024-07-26T11:21:20.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.368+0000 7f498c5c1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8edd350 0x564ce928a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.369+0000 7f497e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8ebc670 0x564ce9289b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:20.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.369+0000 7f497e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8ebc670 0x564ce9289b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:47376/0 (socket says 172.21.15.134:47376) 2024-07-26T11:21:20.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.369+0000 7f498c5c1740 1 -- 172.21.15.134:0/501117927 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564ce904b890 con 0x564ce8edd350 2024-07-26T11:21:20.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.369+0000 7f498c5c1740 1 -- 172.21.15.134:0/501117927 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564ce9184320 con 0x564ce8eb4490 2024-07-26T11:21:20.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.369+0000 7f498c5c1740 1 -- 172.21.15.134:0/501117927 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x564ce9298ec0 con 0x564ce8ebc670 2024-07-26T11:21:20.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.369+0000 7f498c5c1740 10 client.0 _ll_register_callbacks cb 0x564ce91b0000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:20.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.370+0000 7f497f400640 1 --2- 172.21.15.134:0/501117927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8edd350 0x564ce928a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:20.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.369+0000 7f497ea00640 1 -- 172.21.15.134:0/501117927 learned_addr learned my addr 172.21.15.134:0/501117927 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:20.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.376+0000 7f497ea00640 1 -- 172.21.15.134:0/501117927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8ebc670 msgr2=0x564ce9289b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:20.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.376+0000 7f497ea00640 1 --2- 172.21.15.134:0/501117927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8ebc670 0x564ce9289b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.376+0000 7f497ea00640 1 -- 172.21.15.134:0/501117927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8edd350 msgr2=0x564ce928a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:20.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.376+0000 7f497ea00640 1 --2- 172.21.15.134:0/501117927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8edd350 0x564ce928a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.376+0000 7f497ea00640 1 -- 172.21.15.134:0/501117927 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ce92a2440 con 0x564ce8eb4490 2024-07-26T11:21:20.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.377+0000 7f497ea00640 1 --2- 172.21.15.134:0/501117927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 0x564ce9289620 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f49700058c0 tx=0x7f49700039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af076e2de58a2406 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:20.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.377+0000 7f4968c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:21:20.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.378+0000 7f4968c00640 1 -- 172.21.15.134:0/501117927 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4970023420 con 0x564ce8eb4490 2024-07-26T11:21:20.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.378+0000 7f4968c00640 1 -- 172.21.15.134:0/501117927 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4970014410 con 0x564ce8eb4490 2024-07-26T11:21:20.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.378+0000 7f498c5c1740 1 -- 172.21.15.134:0/501117927 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49780abc40 con 0x564ce8eb4490 2024-07-26T11:21:20.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.379+0000 7f4968c00640 1 -- 172.21.15.134:0/501117927 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4970014610 con 0x564ce8eb4490 2024-07-26T11:21:20.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.382+0000 7f498c5c1740 10 client.26309 fetch_fsmap learned FSMap version 32 2024-07-26T11:21:20.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.382+0000 7f498c5c1740 1 -- 172.21.15.134:0/501117927 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x564ce92a4dd0 con 0x564ce8eb4490 2024-07-26T11:21:20.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.382+0000 7f4968c00640 1 -- 172.21.15.134:0/501117927 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49780abc40 con 0x564ce8eb4490 2024-07-26T11:21:20.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.384+0000 7f4968c00640 1 -- 172.21.15.134:0/501117927 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f4970023a30 con 0x564ce8eb4490 2024-07-26T11:21:20.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.386+0000 7f498c5c1740 10 client.26309 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:21:20.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.386+0000 7f498c5c1740 10 client.26309 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:21:20.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.386+0000 7f498c5c1740 20 client.26309 populate_metadata read hostname 'smithi134' 2024-07-26T11:21:20.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.386+0000 7f498c5c1740 1 --2- 172.21.15.134:0/501117927 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x564ce8e9bcd0 0x564ce92a82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.386+0000 7f498c5c1740 4 client.26309 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:21:20.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.386+0000 7f498c5c1740 1 -- 172.21.15.134:0/501117927 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ce91fab30 con 0x564ce8e9bcd0 2024-07-26T11:21:20.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.386+0000 7f497e000640 1 --2- 172.21.15.134:0/501117927 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x564ce8e9bcd0 0x564ce92a82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:20.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.393+0000 7f4968c00640 10 client.26309 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:20.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.393+0000 7f497e000640 1 --2- 172.21.15.134:0/501117927 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x564ce8e9bcd0 0x564ce92a82b0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2e7fb40aca40af8 server_cookie=6215d6bd8734b822 in_seq=0 out_seq=0 2024-07-26T11:21:20.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.395+0000 7f4968c00640 1 -- 172.21.15.134:0/501117927 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564ce91fab30 con 0x564ce8e9bcd0 2024-07-26T11:21:20.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.395+0000 7f4968c00640 10 client.26309 handle_command_reply: tid=0 2024-07-26T11:21:20.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.395+0000 7f498c5c1740 1 client.26309 shutdown 2024-07-26T11:21:20.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.396+0000 7f498c5c1740 1 -- 172.21.15.134:0/501117927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 msgr2=0x564ce9289620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:20.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.396+0000 7f498c5c1740 1 --2- 172.21.15.134:0/501117927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 0x564ce9289620 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f49700058c0 tx=0x7f49700039c0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.397+0000 7f498c5c1740 1 -- 172.21.15.134:0/501117927 shutdown_connections 2024-07-26T11:21:20.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.397+0000 7f498c5c1740 1 --2- 172.21.15.134:0/501117927 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x564ce8e9bcd0 0x564ce92a82b0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.397+0000 7f498c5c1740 1 --2- 172.21.15.134:0/501117927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8ebc670 0x564ce9289b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.397+0000 7f498c5c1740 1 --2- 172.21.15.134:0/501117927 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8edd350 0x564ce928a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.397+0000 7f498c5c1740 1 --2- 172.21.15.134:0/501117927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 0x564ce9289620 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.397+0000 7f498c5c1740 1 -- 172.21.15.134:0/501117927 >> 172.21.15.134:0/501117927 conn(0x564ce8ed1740 msgr2=0x564ce927e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:20.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.397+0000 7f4968c00640 0 client.26309 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:20.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.398+0000 7f498c5c1740 1 -- 172.21.15.134:0/501117927 shutdown_connections 2024-07-26T11:21:20.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.398+0000 7f498c5c1740 1 -- 172.21.15.134:0/501117927 wait complete. 2024-07-26T11:21:20.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.399+0000 7f498c5c1740 20 client.26309 trim_cache size 0 max 16384 2024-07-26T11:21:20.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.434+0000 7f498c5c1740 1 Processor -- start 2024-07-26T11:21:20.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.435+0000 7f498c5c1740 1 -- start start 2024-07-26T11:21:20.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.436+0000 7f498c5c1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 0x564ce9283690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.436+0000 7f497ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 0x564ce9283690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:20.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.437+0000 7f497ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 0x564ce9283690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:53854/0 (socket says 172.21.15.134:53854) 2024-07-26T11:21:20.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.437+0000 7f497ea00640 1 -- 172.21.15.134:0/2480150799 learned_addr learned my addr 172.21.15.134:0/2480150799 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:20.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.437+0000 7f498c5c1740 1 --2- 172.21.15.134:0/2480150799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8ebc670 0x564ce92ada70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.438+0000 7f498c5c1740 1 --2- 172.21.15.134:0/2480150799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8edd350 0x564ce92adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.439+0000 7f497e000640 1 --2- 172.21.15.134:0/2480150799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8ebc670 0x564ce92ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:20.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.439+0000 7f498c5c1740 1 -- 172.21.15.134:0/2480150799 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f49801b6b20 con 0x564ce8edd350 2024-07-26T11:21:20.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.439+0000 7f498c5c1740 1 -- 172.21.15.134:0/2480150799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f49801b8660 con 0x564ce8eb4490 2024-07-26T11:21:20.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.439+0000 7f497f400640 1 --2- 172.21.15.134:0/2480150799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8edd350 0x564ce92adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:20.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.439+0000 7f498c5c1740 1 -- 172.21.15.134:0/2480150799 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f49801b71f0 con 0x564ce8ebc670 2024-07-26T11:21:20.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.440+0000 7f497ea00640 1 -- 172.21.15.134:0/2480150799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8ebc670 msgr2=0x564ce92ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:20.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.441+0000 7f497ea00640 1 --2- 172.21.15.134:0/2480150799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8ebc670 0x564ce92ada70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.441+0000 7f497ea00640 1 -- 172.21.15.134:0/2480150799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8edd350 msgr2=0x564ce92adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:20.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.441+0000 7f497ea00640 1 --2- 172.21.15.134:0/2480150799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8edd350 0x564ce92adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.441+0000 7f497ea00640 1 -- 172.21.15.134:0/2480150799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ce927ef60 con 0x564ce8eb4490 2024-07-26T11:21:20.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.442+0000 7f497e000640 1 --2- 172.21.15.134:0/2480150799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8ebc670 0x564ce92ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:21:20.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.442+0000 7f497ea00640 1 --2- 172.21.15.134:0/2480150799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 0x564ce9283690 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4970000c00 tx=0x7f4970008cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c523aaa83ca6a744 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:20.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.445+0000 7f496b400640 1 -- 172.21.15.134:0/2480150799 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f497002d990 con 0x564ce8eb4490 2024-07-26T11:21:20.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.445+0000 7f496b400640 1 -- 172.21.15.134:0/2480150799 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4970032070 con 0x564ce8eb4490 2024-07-26T11:21:20.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.445+0000 7f496b400640 1 -- 172.21.15.134:0/2480150799 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4970023420 con 0x564ce8eb4490 2024-07-26T11:21:20.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.446+0000 7f498c5c1740 1 -- 172.21.15.134:0/2480150799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 msgr2=0x564ce9283690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:20.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.446+0000 7f498c5c1740 1 --2- 172.21.15.134:0/2480150799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 0x564ce9283690 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4970000c00 tx=0x7f4970008cc0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.449+0000 7f498c5c1740 1 -- 172.21.15.134:0/2480150799 shutdown_connections 2024-07-26T11:21:20.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.449+0000 7f498c5c1740 1 --2- 172.21.15.134:0/2480150799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8ebc670 0x564ce92ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.449+0000 7f498c5c1740 1 --2- 172.21.15.134:0/2480150799 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8edd350 0x564ce92adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.449+0000 7f498c5c1740 1 --2- 172.21.15.134:0/2480150799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 0x564ce9283690 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.449+0000 7f498c5c1740 1 -- 172.21.15.134:0/2480150799 >> 172.21.15.134:0/2480150799 conn(0x564ce8ed1740 msgr2=0x564ce9282f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:20.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.450+0000 7f498c5c1740 1 -- 172.21.15.134:0/2480150799 shutdown_connections 2024-07-26T11:21:20.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.450+0000 7f498c5c1740 1 -- 172.21.15.134:0/2480150799 wait complete. 2024-07-26T11:21:20.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.451+0000 7f498c5c1740 1 -- start start 2024-07-26T11:21:20.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.451+0000 7f498c5c1740 1 Processor -- start 2024-07-26T11:21:20.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.452+0000 7f498c5c1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 0x564ce929e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.453+0000 7f498c5c1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8ebc670 0x564ce929bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.453+0000 7f498c5c1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8edd350 0x564ce929c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.454+0000 7f497f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8edd350 0x564ce929c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:20.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.455+0000 7f497f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8edd350 0x564ce929c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:47390/0 (socket says 172.21.15.134:47390) 2024-07-26T11:21:20.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.455+0000 7f497f400640 1 -- 172.21.15.134:0/2655544490 learned_addr learned my addr 172.21.15.134:0/2655544490 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:20.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.455+0000 7f497ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 0x564ce929e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:20.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.458+0000 7f498c5c1740 1 -- 172.21.15.134:0/2655544490 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f49801b7f90 con 0x564ce8ebc670 2024-07-26T11:21:20.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.458+0000 7f498c5c1740 1 -- 172.21.15.134:0/2655544490 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f49801b78c0 con 0x564ce8eb4490 2024-07-26T11:21:20.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.458+0000 7f498c5c1740 1 -- 172.21.15.134:0/2655544490 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f49801b4b00 con 0x564ce8edd350 2024-07-26T11:21:20.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.460+0000 7f497f400640 1 -- 172.21.15.134:0/2655544490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 msgr2=0x564ce929e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:20.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.460+0000 7f497f400640 1 --2- 172.21.15.134:0/2655544490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 0x564ce929e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.460+0000 7f497f400640 1 -- 172.21.15.134:0/2655544490 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8ebc670 msgr2=0x564ce929bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:20.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.460+0000 7f497f400640 1 --2- 172.21.15.134:0/2655544490 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8ebc670 0x564ce929bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.460+0000 7f497f400640 1 -- 172.21.15.134:0/2655544490 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ce9280380 con 0x564ce8edd350 2024-07-26T11:21:20.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.461+0000 7f498c5c1740 10 client.0 _ll_register_callbacks cb 0x7f49700098f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:20.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.464+0000 7f498c5c1740 1 -- 172.21.15.134:0/2655544490 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4970032070 con 0x564ce8edd350 2024-07-26T11:21:20.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.464+0000 7f497f400640 1 --2- 172.21.15.134:0/2655544490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8edd350 0x564ce929c1a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f497800c640 tx=0x7f49780b0170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=950e0e90c786cd5a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:20.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.464+0000 7f4968c00640 10 client.26320 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:21:20.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.465+0000 7f4968c00640 1 -- 172.21.15.134:0/2655544490 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f497801f040 con 0x564ce8edd350 2024-07-26T11:21:20.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.465+0000 7f4968c00640 1 -- 172.21.15.134:0/2655544490 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4978005460 con 0x564ce8edd350 2024-07-26T11:21:20.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.465+0000 7f4968c00640 1 -- 172.21.15.134:0/2655544490 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f49780b4cf0 con 0x564ce8edd350 2024-07-26T11:21:20.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.467+0000 7f498c5c1740 10 client.26320 fetch_fsmap learned FSMap version 32 2024-07-26T11:21:20.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.467+0000 7f498c5c1740 1 -- 172.21.15.134:0/2655544490 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=32}) -- 0x564ce9284470 con 0x564ce8edd350 2024-07-26T11:21:20.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.468+0000 7f4968c00640 1 -- 172.21.15.134:0/2655544490 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f4978091ce0 con 0x564ce8edd350 2024-07-26T11:21:20.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.468+0000 7f498c5c1740 10 client.26320 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:21:20.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.468+0000 7f498c5c1740 10 client.26320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:21:20.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.468+0000 7f498c5c1740 20 client.26320 populate_metadata read hostname 'smithi134' 2024-07-26T11:21:20.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.468+0000 7f498c5c1740 1 --2- 172.21.15.134:0/2655544490 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x564ce8e71080 0x564ce93126b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:20.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.468+0000 7f498c5c1740 4 client.26320 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T11:21:20.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.468+0000 7f498c5c1740 1 -- 172.21.15.134:0/2655544490 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564ce91f65d0 con 0x564ce8e71080 2024-07-26T11:21:20.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.469+0000 7f497ea00640 1 --2- 172.21.15.134:0/2655544490 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x564ce8e71080 0x564ce93126b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:20.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.480+0000 7f4968c00640 10 client.26320 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:20.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.480+0000 7f497ea00640 1 --2- 172.21.15.134:0/2655544490 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x564ce8e71080 0x564ce93126b0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8888bdbd70a0a5a server_cookie=9e2929784821141c in_seq=0 out_seq=0 2024-07-26T11:21:20.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.495+0000 7f4968c00640 1 -- 172.21.15.134:0/2655544490 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564ce91f65d0 con 0x564ce8e71080 2024-07-26T11:21:20.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.495+0000 7f4968c00640 10 client.26320 handle_command_reply: tid=0 2024-07-26T11:21:20.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.495+0000 7f498c5c1740 1 client.26320 shutdown 2024-07-26T11:21:20.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.496+0000 7f498c5c1740 1 -- 172.21.15.134:0/2655544490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8edd350 msgr2=0x564ce929c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:20.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.496+0000 7f498c5c1740 1 --2- 172.21.15.134:0/2655544490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8edd350 0x564ce929c1a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f497800c640 tx=0x7f49780b0170 comp rx=0 tx=0).stop 2024-07-26T11:21:20.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.496+0000 7f498c5c1740 1 -- 172.21.15.134:0/2655544490 shutdown_connections 2024-07-26T11:21:20.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.496+0000 7f498c5c1740 1 --2- 172.21.15.134:0/2655544490 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x564ce8e71080 0x564ce93126b0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.502+0000 7f498c5c1740 1 --2- 172.21.15.134:0/2655544490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564ce8edd350 0x564ce929c1a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.502+0000 7f498c5c1740 1 --2- 172.21.15.134:0/2655544490 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x564ce8ebc670 0x564ce929bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.502+0000 7f498c5c1740 1 --2- 172.21.15.134:0/2655544490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564ce8eb4490 0x564ce929e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.502+0000 7f498c5c1740 1 -- 172.21.15.134:0/2655544490 >> 172.21.15.134:0/2655544490 conn(0x564ce8ed1740 msgr2=0x564ce9282f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:20.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.502+0000 7f4968c00640 0 client.26320 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:20.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.503+0000 7f498c5c1740 1 -- 172.21.15.134:0/2655544490 shutdown_connections 2024-07-26T11:21:20.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.503+0000 7f498c5c1740 1 -- 172.21.15.134:0/2655544490 wait complete. 2024-07-26T11:21:20.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.503+0000 7f498c5c1740 20 client.26320 trim_cache size 0 max 16384 2024-07-26T11:21:20.507 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:21:20.507 INFO:teuthology.orchestra.run.smithi134.stdout: "return_code": 0, 2024-07-26T11:21:20.507 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_tag": "b8ad51fd-1a31-4d3f-a1e4-26334b2d224e", 2024-07-26T11:21:20.507 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "asynchronous" 2024-07-26T11:21:20.507 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:21:20.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.504+0000 7f497fe00640 1 -- 172.21.15.134:0/3971724464 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f49640b3e00 msgr2=0x7f49640b6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:20.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.504+0000 7f497fe00640 1 --2- 172.21.15.134:0/3971724464 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f49640b3e00 0x7f49640b6310 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4970009bc0 tx=0x7f4970005f10 comp rx=0 tx=0).stop 2024-07-26T11:21:20.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.504+0000 7f497fe00640 1 -- 172.21.15.134:0/3971724464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4980110090 msgr2=0x7f49801a58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:20.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.504+0000 7f497fe00640 1 --2- 172.21.15.134:0/3971724464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4980110090 0x7f49801a58b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f497800e940 tx=0x7f497800ee00 comp rx=0 tx=0).stop 2024-07-26T11:21:20.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.505+0000 7f497fe00640 1 -- 172.21.15.134:0/3971724464 shutdown_connections 2024-07-26T11:21:20.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.505+0000 7f497fe00640 1 --2- 172.21.15.134:0/3971724464 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f49640b3e00 0x7f49640b6310 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.505+0000 7f497fe00640 1 --2- 172.21.15.134:0/3971724464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f498010f740 0x7f49801a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.505+0000 7f497fe00640 1 --2- 172.21.15.134:0/3971724464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4980110090 0x7f49801a58b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.505+0000 7f497fe00640 1 --2- 172.21.15.134:0/3971724464 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f498010a080 0x7f49801a0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:20.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.505+0000 7f497fe00640 1 -- 172.21.15.134:0/3971724464 >> 172.21.15.134:0/3971724464 conn(0x7f498006e420 msgr2=0x7f4980073680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:20.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.505+0000 7f497fe00640 1 -- 172.21.15.134:0/3971724464 shutdown_connections 2024-07-26T11:21:20.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:20.505+0000 7f497fe00640 1 -- 172.21.15.134:0/3971724464 wait complete. 2024-07-26T11:21:20.555 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:20 smithi134 ceph-mon[31966]: pgmap v641: 129 pgs: 129 active+clean; 36 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 32 KiB/s rd, 275 MiB/s wr, 2.22k op/s 2024-07-26T11:21:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:20 smithi193 ceph-mon[37473]: pgmap v641: 129 pgs: 129 active+clean; 36 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 32 KiB/s rd, 275 MiB/s wr, 2.22k op/s 2024-07-26T11:21:20.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:20 smithi189 ceph-mon[36790]: pgmap v641: 129 pgs: 129 active+clean; 36 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 32 KiB/s rd, 275 MiB/s wr, 2.22k op/s 2024-07-26T11:21:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:21 smithi193 ceph-mon[37473]: scrub queued for path: / 2024-07-26T11:21:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:21 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:21:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:21 smithi193 ceph-mon[37473]: scrub summary: active paths [/] 2024-07-26T11:21:21.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:21 smithi134 ceph-mon[31966]: scrub queued for path: / 2024-07-26T11:21:21.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:21 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:21:21.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:21 smithi134 ceph-mon[31966]: scrub summary: active paths [/] 2024-07-26T11:21:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:21 smithi189 ceph-mon[36790]: scrub queued for path: / 2024-07-26T11:21:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:21 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:21:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:21 smithi189 ceph-mon[36790]: scrub summary: active paths [/] 2024-07-26T11:21:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:22 smithi193 ceph-mon[37473]: pgmap v642: 129 pgs: 129 active+clean; 36 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 38 KiB/s rd, 226 MiB/s wr, 1.90k op/s 2024-07-26T11:21:22.927 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:22 smithi134 ceph-mon[31966]: pgmap v642: 129 pgs: 129 active+clean; 36 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 38 KiB/s rd, 226 MiB/s wr, 1.90k op/s 2024-07-26T11:21:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:22 smithi189 ceph-mon[36790]: pgmap v642: 129 pgs: 129 active+clean; 36 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 38 KiB/s rd, 226 MiB/s wr, 1.90k op/s 2024-07-26T11:21:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:23 smithi193 ceph-mon[37473]: scrub summary: active paths [/] 2024-07-26T11:21:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:23 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:21:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:23 smithi134 ceph-mon[31966]: scrub summary: active paths [/] 2024-07-26T11:21:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:23 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:21:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:23 smithi189 ceph-mon[36790]: scrub summary: active paths [/] 2024-07-26T11:21:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:23 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:21:25.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:25 smithi134 ceph-mon[31966]: pgmap v643: 129 pgs: 129 active+clean; 37 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 24 KiB/s rd, 243 MiB/s wr, 2.09k op/s 2024-07-26T11:21:25.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:25 smithi189 ceph-mon[36790]: pgmap v643: 129 pgs: 129 active+clean; 37 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 24 KiB/s rd, 243 MiB/s wr, 2.09k op/s 2024-07-26T11:21:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:25 smithi193 ceph-mon[37473]: pgmap v643: 129 pgs: 129 active+clean; 37 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 24 KiB/s rd, 243 MiB/s wr, 2.09k op/s 2024-07-26T11:21:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.063+0000 7f3d8aa00640 1 -- 172.21.15.134:0/1143814314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d84109df0 msgr2=0x7f3d8410a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.063+0000 7f3d8aa00640 1 --2- 172.21.15.134:0/1143814314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d84109df0 0x7f3d8410a200 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009be0 tx=0x7f3d7c02a6f0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.064+0000 7f3d8aa00640 1 -- 172.21.15.134:0/1143814314 shutdown_connections 2024-07-26T11:21:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.064+0000 7f3d8aa00640 1 --2- 172.21.15.134:0/1143814314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d84115f40 0x7f3d84118360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.064+0000 7f3d8aa00640 1 --2- 172.21.15.134:0/1143814314 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d8410a740 0x7f3d84115a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.064+0000 7f3d8aa00640 1 --2- 172.21.15.134:0/1143814314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d84109df0 0x7f3d8410a200 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.064+0000 7f3d8aa00640 1 -- 172.21.15.134:0/1143814314 >> 172.21.15.134:0/1143814314 conn(0x7f3d84070630 msgr2=0x7f3d84070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.064+0000 7f3d8aa00640 1 -- 172.21.15.134:0/1143814314 shutdown_connections 2024-07-26T11:21:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.064+0000 7f3d8aa00640 1 -- 172.21.15.134:0/1143814314 wait complete. 2024-07-26T11:21:27.067 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.066+0000 7f3d8aa00640 1 Processor -- start 2024-07-26T11:21:27.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.067+0000 7f3d8aa00640 1 -- start start 2024-07-26T11:21:27.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.067+0000 7f3d8aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d84109df0 0x7f3d841a24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.068+0000 7f3d8aa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d8410a740 0x7f3d841a2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.068+0000 7f3d8aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d84115f40 0x7f3d841a7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.068+0000 7f3d8aa00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8411ba50 con 0x7f3d8410a740 2024-07-26T11:21:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.068+0000 7f3d8aa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8411b8d0 con 0x7f3d84109df0 2024-07-26T11:21:27.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.068+0000 7f3d8aa00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8411bbd0 con 0x7f3d84115f40 2024-07-26T11:21:27.071 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.069+0000 7f3d89600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d84109df0 0x7f3d841a24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:27.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.069+0000 7f3d8a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d84115f40 0x7f3d841a7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:27.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.069+0000 7f3d8a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d84115f40 0x7f3d841a7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:47410/0 (socket says 172.21.15.134:47410) 2024-07-26T11:21:27.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.069+0000 7f3d89600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d84109df0 0x7f3d841a24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:53892/0 (socket says 172.21.15.134:53892) 2024-07-26T11:21:27.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.069+0000 7f3d8a000640 1 -- 172.21.15.134:0/3758204327 learned_addr learned my addr 172.21.15.134:0/3758204327 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:27.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.069+0000 7f3d88c00640 1 --2- 172.21.15.134:0/3758204327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d8410a740 0x7f3d841a2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:27.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.070+0000 7f3d8a000640 1 -- 172.21.15.134:0/3758204327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d84109df0 msgr2=0x7f3d841a24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:27.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.070+0000 7f3d8a000640 1 --2- 172.21.15.134:0/3758204327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d84109df0 0x7f3d841a24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.070+0000 7f3d8a000640 1 -- 172.21.15.134:0/3758204327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d8410a740 msgr2=0x7f3d841a2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:27.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.071+0000 7f3d8a000640 1 --2- 172.21.15.134:0/3758204327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d8410a740 0x7f3d841a2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.071+0000 7f3d8a000640 1 -- 172.21.15.134:0/3758204327 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d841a8400 con 0x7f3d84115f40 2024-07-26T11:21:27.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.071+0000 7f3d88c00640 1 --2- 172.21.15.134:0/3758204327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d8410a740 0x7f3d841a2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:21:27.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.071+0000 7f3d89600640 1 --2- 172.21.15.134:0/3758204327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d84109df0 0x7f3d841a24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:21:27.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.072+0000 7f3d8a000640 1 --2- 172.21.15.134:0/3758204327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d84115f40 0x7f3d841a7d00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800e940 tx=0x7f3d7800ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:27.075 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.072+0000 7f3d82000640 1 -- 172.21.15.134:0/3758204327 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7800cd80 con 0x7f3d84115f40 2024-07-26T11:21:27.075 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.072+0000 7f3d82000640 1 -- 172.21.15.134:0/3758204327 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d780035f0 con 0x7f3d84115f40 2024-07-26T11:21:27.075 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.072+0000 7f3d82000640 1 -- 172.21.15.134:0/3758204327 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d78010670 con 0x7f3d84115f40 2024-07-26T11:21:27.075 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.072+0000 7f3d8aa00640 1 -- 172.21.15.134:0/3758204327 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d841a86f0 con 0x7f3d84115f40 2024-07-26T11:21:27.075 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.073+0000 7f3d8aa00640 1 -- 172.21.15.134:0/3758204327 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d8410b790 con 0x7f3d84115f40 2024-07-26T11:21:27.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.076+0000 7f3d82000640 1 -- 172.21.15.134:0/3758204327 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f3d78004030 con 0x7f3d84115f40 2024-07-26T11:21:27.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.078+0000 7f3d82000640 1 --2- 172.21.15.134:0/3758204327 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d6c0b3dc0 0x7f3d6c0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.078+0000 7f3d82000640 1 -- 172.21.15.134:0/3758204327 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3d780e46b0 con 0x7f3d84115f40 2024-07-26T11:21:27.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.086+0000 7f3d89600640 1 --2- 172.21.15.134:0/3758204327 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d6c0b3dc0 0x7f3d6c0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:27.092 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.090+0000 7f3d89600640 1 --2- 172.21.15.134:0/3758204327 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d6c0b3dc0 0x7f3d6c0b62d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009420 tx=0x7f3d7c000f40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.233+0000 7f3d93001740 1 Processor -- start 2024-07-26T11:21:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.235+0000 7f3d93001740 1 -- start start 2024-07-26T11:21:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.235+0000 7f3d93001740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d69481b0 0x55b4d6cf4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.236+0000 7f3d93001740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d691f2f0 0x55b4d6cf4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.236+0000 7f3d93001740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 0x55b4d6cf4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.236+0000 7f3d93001740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b4d6c60270 con 0x55b4d691f2f0 2024-07-26T11:21:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.236+0000 7f3d93001740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b4d6c4f770 con 0x55b4d69274d0 2024-07-26T11:21:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.236+0000 7f3d93001740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b4d6c537b0 con 0x55b4d69481b0 2024-07-26T11:21:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.236+0000 7f3d89600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d69481b0 0x55b4d6cf4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.236+0000 7f3d89600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d69481b0 0x55b4d6cf4430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:47424/0 (socket says 172.21.15.134:47424) 2024-07-26T11:21:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.236+0000 7f3d89600640 1 -- 172.21.15.134:0/411445840 learned_addr learned my addr 172.21.15.134:0/411445840 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.237+0000 7f3d8a000640 1 --2- 172.21.15.134:0/411445840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 0x55b4d6cf4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:27.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.237+0000 7f3d89600640 1 -- 172.21.15.134:0/411445840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 msgr2=0x55b4d6cf4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:27.240 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.237+0000 7f3d89600640 1 --2- 172.21.15.134:0/411445840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 0x55b4d6cf4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.240 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.238+0000 7f3d89600640 1 -- 172.21.15.134:0/411445840 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d691f2f0 msgr2=0x55b4d6cf4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:27.240 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.238+0000 7f3d89600640 1 --2- 172.21.15.134:0/411445840 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d691f2f0 0x55b4d6cf4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.240 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.238+0000 7f3d89600640 1 -- 172.21.15.134:0/411445840 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4d69b3790 con 0x55b4d69481b0 2024-07-26T11:21:27.240 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.238+0000 7f3d8a000640 1 --2- 172.21.15.134:0/411445840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 0x55b4d6cf4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:21:27.243 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.241+0000 7f3d89600640 1 --2- 172.21.15.134:0/411445840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d69481b0 0x55b4d6cf4430 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c030b60 tx=0x7f3d7c03c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9954f2becaf21dc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.242+0000 7f3d76600640 1 -- 172.21.15.134:0/411445840 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c036040 con 0x55b4d69481b0 2024-07-26T11:21:27.244 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.242+0000 7f3d76600640 1 -- 172.21.15.134:0/411445840 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d7c043070 con 0x55b4d69481b0 2024-07-26T11:21:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.244+0000 7f3d93001740 1 -- 172.21.15.134:0/411445840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d69481b0 msgr2=0x55b4d6cf4430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.244+0000 7f3d93001740 1 --2- 172.21.15.134:0/411445840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d69481b0 0x55b4d6cf4430 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c030b60 tx=0x7f3d7c03c000 comp rx=0 tx=0).stop 2024-07-26T11:21:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.247+0000 7f3d76600640 1 -- 172.21.15.134:0/411445840 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c049650 con 0x55b4d69481b0 2024-07-26T11:21:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.247+0000 7f3d93001740 1 -- 172.21.15.134:0/411445840 shutdown_connections 2024-07-26T11:21:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.247+0000 7f3d93001740 1 --2- 172.21.15.134:0/411445840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 0x55b4d6cf4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.247+0000 7f3d93001740 1 --2- 172.21.15.134:0/411445840 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d691f2f0 0x55b4d6cf4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.248+0000 7f3d93001740 1 --2- 172.21.15.134:0/411445840 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d69481b0 0x55b4d6cf4430 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.250 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.248+0000 7f3d93001740 1 -- 172.21.15.134:0/411445840 >> 172.21.15.134:0/411445840 conn(0x55b4d693c300 msgr2=0x55b4d6cf2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.248+0000 7f3d93001740 1 -- 172.21.15.134:0/411445840 shutdown_connections 2024-07-26T11:21:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.249+0000 7f3d93001740 1 -- 172.21.15.134:0/411445840 wait complete. 2024-07-26T11:21:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.250+0000 7f3d93001740 1 -- start start 2024-07-26T11:21:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.251+0000 7f3d93001740 1 Processor -- start 2024-07-26T11:21:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.255+0000 7f3d93001740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d691f2f0 0x55b4d6d00030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.260 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.258+0000 7f3d93001740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d69274d0 0x55b4d6d005b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.258+0000 7f3d89600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d691f2f0 0x55b4d6d00030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.258+0000 7f3d89600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d691f2f0 0x55b4d6d00030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:53912/0 (socket says 172.21.15.134:53912) 2024-07-26T11:21:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.259+0000 7f3d89600640 1 -- 172.21.15.134:0/3743924643 learned_addr learned my addr 172.21.15.134:0/3743924643 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:27.263 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.260+0000 7f3d93001740 1 --2- 172.21.15.134:0/3743924643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d69481b0 0x55b4d6d0bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.264 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.262+0000 7f3d8a000640 1 --2- 172.21.15.134:0/3743924643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d69481b0 0x55b4d6d0bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.263+0000 7f3d93001740 1 -- 172.21.15.134:0/3743924643 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55b4d6b89ab0 con 0x55b4d69274d0 2024-07-26T11:21:27.265 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.263+0000 7f3d93001740 1 -- 172.21.15.134:0/3743924643 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b4d6becd50 con 0x55b4d691f2f0 2024-07-26T11:21:27.266 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.264+0000 7f3d93001740 1 -- 172.21.15.134:0/3743924643 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55b4d6d02ba0 con 0x55b4d69481b0 2024-07-26T11:21:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.265+0000 7f3d8a000640 1 -- 172.21.15.134:0/3743924643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d691f2f0 msgr2=0x55b4d6d00030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.265+0000 7f3d8a000640 1 --2- 172.21.15.134:0/3743924643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d691f2f0 0x55b4d6d00030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.268 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.265+0000 7f3d8a000640 1 -- 172.21.15.134:0/3743924643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d69274d0 msgr2=0x55b4d6d005b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.267+0000 7f3d93001740 10 client.0 _ll_register_callbacks cb 0x55b4d695fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.267+0000 7f3d88c00640 1 --2- 172.21.15.134:0/3743924643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d69274d0 0x55b4d6d005b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.268+0000 7f3d8a000640 1 --2- 172.21.15.134:0/3743924643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d69274d0 0x55b4d6d005b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.268+0000 7f3d8a000640 1 -- 172.21.15.134:0/3743924643 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4d6a2c620 con 0x55b4d69481b0 2024-07-26T11:21:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.270+0000 7f3d8a000640 1 --2- 172.21.15.134:0/3743924643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d69481b0 0x55b4d6d0bf70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3d780e1a20 tx=0x7f3d780aab30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b29af22aeadf3d2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.271+0000 7f3d88c00640 1 --2- 172.21.15.134:0/3743924643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d69274d0 0x55b4d6d005b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:21:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.271+0000 7f3d93001740 1 -- 172.21.15.134:0/3743924643 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d7c043070 con 0x55b4d69481b0 2024-07-26T11:21:27.273 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.271+0000 7f3d6aa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:21:27.274 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.272+0000 7f3d6aa00640 1 -- 172.21.15.134:0/3743924643 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d780e3040 con 0x55b4d69481b0 2024-07-26T11:21:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.273+0000 7f3d6aa00640 1 -- 172.21.15.134:0/3743924643 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3d780a74d0 con 0x55b4d69481b0 2024-07-26T11:21:27.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.274+0000 7f3d6aa00640 1 -- 172.21.15.134:0/3743924643 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3d780b3f90 con 0x55b4d69481b0 2024-07-26T11:21:27.278 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.276+0000 7f3d93001740 10 client.26332 fetch_fsmap learned FSMap version 32 2024-07-26T11:21:27.279 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.276+0000 7f3d93001740 1 -- 172.21.15.134:0/3743924643 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=32}) -- 0x55b4d6d0eaa0 con 0x55b4d69481b0 2024-07-26T11:21:27.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.278+0000 7f3d6aa00640 1 -- 172.21.15.134:0/3743924643 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f3d78004320 con 0x55b4d69481b0 2024-07-26T11:21:27.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.278+0000 7f3d93001740 10 client.26332 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:21:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.279+0000 7f3d93001740 10 client.26332 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:21:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.281+0000 7f3d93001740 20 client.26332 populate_metadata read hostname 'smithi134' 2024-07-26T11:21:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.281+0000 7f3d93001740 1 --2- 172.21.15.134:0/3743924643 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55b4d6906d80 0x55b4d6d12020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.282+0000 7f3d93001740 4 client.26332 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:21:27.283 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.282+0000 7f3d93001740 1 -- 172.21.15.134:0/3743924643 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b4d6c621c0 con 0x55b4d6906d80 2024-07-26T11:21:27.287 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.284+0000 7f3d89600640 1 --2- 172.21.15.134:0/3743924643 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55b4d6906d80 0x55b4d6d12020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:27.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.290+0000 7f3d6aa00640 10 client.26332 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:27.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.290+0000 7f3d89600640 1 --2- 172.21.15.134:0/3743924643 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55b4d6906d80 0x55b4d6d12020 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2567ba8ae49abea server_cookie=728e018047d2ee65 in_seq=0 out_seq=0 2024-07-26T11:21:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.291+0000 7f3d6aa00640 1 -- 172.21.15.134:0/3743924643 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55b4d6c621c0 con 0x55b4d6906d80 2024-07-26T11:21:27.293 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.291+0000 7f3d6aa00640 10 client.26332 handle_command_reply: tid=0 2024-07-26T11:21:27.295 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.292+0000 7f3d93001740 1 client.26332 shutdown 2024-07-26T11:21:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.294+0000 7f3d93001740 1 -- 172.21.15.134:0/3743924643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d69481b0 msgr2=0x55b4d6d0bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.294+0000 7f3d93001740 1 --2- 172.21.15.134:0/3743924643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d69481b0 0x55b4d6d0bf70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3d780e1a20 tx=0x7f3d780aab30 comp rx=0 tx=0).stop 2024-07-26T11:21:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.294+0000 7f3d93001740 1 -- 172.21.15.134:0/3743924643 shutdown_connections 2024-07-26T11:21:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.294+0000 7f3d93001740 1 --2- 172.21.15.134:0/3743924643 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55b4d6906d80 0x55b4d6d12020 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.294+0000 7f3d93001740 1 --2- 172.21.15.134:0/3743924643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d69481b0 0x55b4d6d0bf70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.294+0000 7f3d93001740 1 --2- 172.21.15.134:0/3743924643 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d69274d0 0x55b4d6d005b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.294+0000 7f3d93001740 1 --2- 172.21.15.134:0/3743924643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d691f2f0 0x55b4d6d00030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.294+0000 7f3d93001740 1 -- 172.21.15.134:0/3743924643 >> 172.21.15.134:0/3743924643 conn(0x55b4d693c300 msgr2=0x55b4d6cf5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:27.297 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.295+0000 7f3d6aa00640 0 client.26332 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:27.299 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.295+0000 7f3d93001740 1 -- 172.21.15.134:0/3743924643 shutdown_connections 2024-07-26T11:21:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.298+0000 7f3d93001740 1 -- 172.21.15.134:0/3743924643 wait complete. 2024-07-26T11:21:27.301 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.299+0000 7f3d93001740 20 client.26332 trim_cache size 0 max 16384 2024-07-26T11:21:27.375 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.372+0000 7f3d93001740 1 Processor -- start 2024-07-26T11:21:27.379 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.373+0000 7f3d93001740 1 -- start start 2024-07-26T11:21:27.379 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.373+0000 7f3d93001740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d691f2f0 0x55b4d6cf2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.373+0000 7f3d93001740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 0x55b4d6cfa570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.373+0000 7f3d93001740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d69481b0 0x55b4d6d0bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.374+0000 7f3d93001740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6400ff80 con 0x55b4d69481b0 2024-07-26T11:21:27.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.374+0000 7f3d93001740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3d64011ac0 con 0x55b4d69274d0 2024-07-26T11:21:27.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.374+0000 7f3d93001740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3d64010650 con 0x55b4d691f2f0 2024-07-26T11:21:27.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.374+0000 7f3d88c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 0x55b4d6cfa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:27.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.374+0000 7f3d88c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 0x55b4d6cfa570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:53922/0 (socket says 172.21.15.134:53922) 2024-07-26T11:21:27.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.374+0000 7f3d88c00640 1 -- 172.21.15.134:0/269548860 learned_addr learned my addr 172.21.15.134:0/269548860 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:27.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.375+0000 7f3d88c00640 1 -- 172.21.15.134:0/269548860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d691f2f0 msgr2=0x55b4d6cf2380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:21:27.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.375+0000 7f3d88c00640 1 --2- 172.21.15.134:0/269548860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d691f2f0 0x55b4d6cf2380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.375+0000 7f3d88c00640 1 -- 172.21.15.134:0/269548860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d69481b0 msgr2=0x55b4d6d0bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:27.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.375+0000 7f3d88c00640 1 --2- 172.21.15.134:0/269548860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d69481b0 0x55b4d6d0bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.375+0000 7f3d88c00640 1 -- 172.21.15.134:0/269548860 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4d6cf6cd0 con 0x55b4d69274d0 2024-07-26T11:21:27.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.375+0000 7f3d88c00640 1 --2- 172.21.15.134:0/269548860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 0x55b4d6cfa570 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3d8406c0f0 tx=0x7f3d7000af80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=310c29a066d1df81 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:27.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.375+0000 7f3d75c00640 1 -- 172.21.15.134:0/269548860 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d70004020 con 0x55b4d69274d0 2024-07-26T11:21:27.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.376+0000 7f3d75c00640 1 -- 172.21.15.134:0/269548860 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d700041c0 con 0x55b4d69274d0 2024-07-26T11:21:27.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.376+0000 7f3d75c00640 1 -- 172.21.15.134:0/269548860 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d700116a0 con 0x55b4d69274d0 2024-07-26T11:21:27.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.377+0000 7f3d93001740 1 -- 172.21.15.134:0/269548860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 msgr2=0x55b4d6cfa570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:27.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.377+0000 7f3d93001740 1 --2- 172.21.15.134:0/269548860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 0x55b4d6cfa570 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3d8406c0f0 tx=0x7f3d7000af80 comp rx=0 tx=0).stop 2024-07-26T11:21:27.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.378+0000 7f3d93001740 1 -- 172.21.15.134:0/269548860 shutdown_connections 2024-07-26T11:21:27.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.378+0000 7f3d93001740 1 --2- 172.21.15.134:0/269548860 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d69481b0 0x55b4d6d0bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.378+0000 7f3d93001740 1 --2- 172.21.15.134:0/269548860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 0x55b4d6cfa570 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.378+0000 7f3d93001740 1 --2- 172.21.15.134:0/269548860 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d691f2f0 0x55b4d6cf2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.378+0000 7f3d93001740 1 -- 172.21.15.134:0/269548860 >> 172.21.15.134:0/269548860 conn(0x55b4d693c300 msgr2=0x55b4d6d01340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:27.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.380+0000 7f3d93001740 1 -- 172.21.15.134:0/269548860 shutdown_connections 2024-07-26T11:21:27.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.380+0000 7f3d93001740 1 -- 172.21.15.134:0/269548860 wait complete. 2024-07-26T11:21:27.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.381+0000 7f3d93001740 1 -- start start 2024-07-26T11:21:27.384 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.381+0000 7f3d93001740 1 Processor -- start 2024-07-26T11:21:27.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.385+0000 7f3d93001740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d691f2f0 0x55b4d6d034e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.385+0000 7f3d93001740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 0x55b4d6d03a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.386+0000 7f3d89600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d691f2f0 0x55b4d6d034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.386+0000 7f3d89600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d691f2f0 0x55b4d6d034e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:47430/0 (socket says 172.21.15.134:47430) 2024-07-26T11:21:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.386+0000 7f3d93001740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d69481b0 0x55b4d6d787a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.386+0000 7f3d93001740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f3d640113f0 con 0x55b4d69481b0 2024-07-26T11:21:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.386+0000 7f3d93001740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3d64010d20 con 0x55b4d69274d0 2024-07-26T11:21:27.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.386+0000 7f3d93001740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6400df60 con 0x55b4d691f2f0 2024-07-26T11:21:27.389 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.386+0000 7f3d89600640 1 -- 172.21.15.134:0/66697072 learned_addr learned my addr 172.21.15.134:0/66697072 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:27.389 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.387+0000 7f3d93001740 10 client.0 _ll_register_callbacks cb 0x7f3d780a6da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:27.389 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.387+0000 7f3d88c00640 1 --2- 172.21.15.134:0/66697072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 0x55b4d6d03a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.387+0000 7f3d89600640 1 -- 172.21.15.134:0/66697072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 msgr2=0x55b4d6d03a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.387+0000 7f3d89600640 1 --2- 172.21.15.134:0/66697072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 0x55b4d6d03a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.387+0000 7f3d89600640 1 -- 172.21.15.134:0/66697072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d69481b0 msgr2=0x55b4d6d787a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.387+0000 7f3d89600640 1 --2- 172.21.15.134:0/66697072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d69481b0 0x55b4d6d787a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.387+0000 7f3d89600640 1 -- 172.21.15.134:0/66697072 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b4d6d78ce0 con 0x55b4d691f2f0 2024-07-26T11:21:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.388+0000 7f3d88c00640 1 --2- 172.21.15.134:0/66697072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 0x55b4d6d03a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:21:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.389+0000 7f3d89600640 1 --2- 172.21.15.134:0/66697072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d691f2f0 0x55b4d6d034e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009be0 tx=0x7f3d7c03cbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60782055db46c469 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.389+0000 7f3d6aa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:21:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.390+0000 7f3d6aa00640 1 -- 172.21.15.134:0/66697072 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d7c036040 con 0x55b4d691f2f0 2024-07-26T11:21:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.390+0000 7f3d6aa00640 1 -- 172.21.15.134:0/66697072 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3d7c003970 con 0x55b4d691f2f0 2024-07-26T11:21:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.390+0000 7f3d93001740 1 -- 172.21.15.134:0/66697072 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d700041c0 con 0x55b4d691f2f0 2024-07-26T11:21:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.390+0000 7f3d6aa00640 1 -- 172.21.15.134:0/66697072 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3d7c06c050 con 0x55b4d691f2f0 2024-07-26T11:21:27.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.393+0000 7f3d93001740 10 client.26335 fetch_fsmap learned FSMap version 32 2024-07-26T11:21:27.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.393+0000 7f3d93001740 1 -- 172.21.15.134:0/66697072 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55b4d6d79ea0 con 0x55b4d691f2f0 2024-07-26T11:21:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.396+0000 7f3d6aa00640 1 -- 172.21.15.134:0/66697072 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3d700041c0 con 0x55b4d691f2f0 2024-07-26T11:21:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.397+0000 7f3d6aa00640 1 -- 172.21.15.134:0/66697072 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f3d7c048070 con 0x55b4d691f2f0 2024-07-26T11:21:27.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.399+0000 7f3d93001740 10 client.26335 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:21:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.399+0000 7f3d93001740 10 client.26335 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:21:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.399+0000 7f3d93001740 20 client.26335 populate_metadata read hostname 'smithi134' 2024-07-26T11:21:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.399+0000 7f3d93001740 1 --2- 172.21.15.134:0/66697072 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55b4d68dc1c0 0x55b4d6d7c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.399+0000 7f3d93001740 4 client.26335 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:21:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.399+0000 7f3d93001740 1 -- 172.21.15.134:0/66697072 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b4d6c5dc60 con 0x55b4d68dc1c0 2024-07-26T11:21:27.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.400+0000 7f3d88c00640 1 --2- 172.21.15.134:0/66697072 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55b4d68dc1c0 0x55b4d6d7c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:27.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.406+0000 7f3d6aa00640 10 client.26335 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:27.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:27.406+0000 7f3d88c00640 1 --2- 172.21.15.134:0/66697072 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55b4d68dc1c0 0x55b4d6d7c1e0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b7a56ac0bb87626 server_cookie=e972ee494b2cc2b in_seq=0 out_seq=0 2024-07-26T11:21:27.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:27 smithi134 ceph-mon[31966]: pgmap v644: 129 pgs: 129 active+clean; 37 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 23 KiB/s rd, 227 MiB/s wr, 2.05k op/s 2024-07-26T11:21:27.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:27 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:21:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:27 smithi189 ceph-mon[36790]: pgmap v644: 129 pgs: 129 active+clean; 37 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 23 KiB/s rd, 227 MiB/s wr, 2.05k op/s 2024-07-26T11:21:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:27 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:21:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:27 smithi193 ceph-mon[37473]: pgmap v644: 129 pgs: 129 active+clean; 37 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 23 KiB/s rd, 227 MiB/s wr, 2.05k op/s 2024-07-26T11:21:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:27 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:21:29.368 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:21:29.647 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:29 smithi134 ceph-mon[31966]: pgmap v645: 129 pgs: 129 active+clean; 37 GiB data, 78 GiB used, 995 GiB / 1.0 TiB avail; 28 KiB/s rd, 206 MiB/s wr, 1.86k op/s 2024-07-26T11:21:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:29 smithi189 ceph-mon[36790]: pgmap v645: 129 pgs: 129 active+clean; 37 GiB data, 78 GiB used, 995 GiB / 1.0 TiB avail; 28 KiB/s rd, 206 MiB/s wr, 1.86k op/s 2024-07-26T11:21:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:29 smithi193 ceph-mon[37473]: pgmap v645: 129 pgs: 129 active+clean; 37 GiB data, 78 GiB used, 995 GiB / 1.0 TiB avail; 28 KiB/s rd, 206 MiB/s wr, 1.86k op/s 2024-07-26T11:21:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.421+0000 7f3d6aa00640 1 -- 172.21.15.134:0/66697072 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b4d6c5dc60 con 0x55b4d68dc1c0 2024-07-26T11:21:30.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.421+0000 7f3d6aa00640 10 client.26335 handle_command_reply: tid=0 2024-07-26T11:21:30.427 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.425+0000 7f3d93001740 1 client.26335 shutdown 2024-07-26T11:21:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.427+0000 7f3d93001740 1 -- 172.21.15.134:0/66697072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d691f2f0 msgr2=0x55b4d6d034e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:30.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.427+0000 7f3d93001740 1 --2- 172.21.15.134:0/66697072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d691f2f0 0x55b4d6d034e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009be0 tx=0x7f3d7c03cbb0 comp rx=0 tx=0).stop 2024-07-26T11:21:30.430 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.429+0000 7f3d93001740 1 -- 172.21.15.134:0/66697072 shutdown_connections 2024-07-26T11:21:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.429+0000 7f3d93001740 1 --2- 172.21.15.134:0/66697072 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55b4d68dc1c0 0x55b4d6d7c1e0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.429+0000 7f3d93001740 1 --2- 172.21.15.134:0/66697072 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55b4d69481b0 0x55b4d6d787a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.429+0000 7f3d93001740 1 --2- 172.21.15.134:0/66697072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b4d69274d0 0x55b4d6d03a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.429+0000 7f3d93001740 1 --2- 172.21.15.134:0/66697072 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b4d691f2f0 0x55b4d6d034e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.429+0000 7f3d6aa00640 0 client.26335 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:30.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.429+0000 7f3d93001740 1 -- 172.21.15.134:0/66697072 >> 172.21.15.134:0/66697072 conn(0x55b4d693c300 msgr2=0x55b4d6d01aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:30.435 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.433+0000 7f3d93001740 1 -- 172.21.15.134:0/66697072 shutdown_connections 2024-07-26T11:21:30.435 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.433+0000 7f3d93001740 1 -- 172.21.15.134:0/66697072 wait complete. 2024-07-26T11:21:30.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.435+0000 7f3d93001740 20 client.26335 trim_cache size 0 max 16384 2024-07-26T11:21:30.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:21:30.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:21:30.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:21:30.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:21:30.443 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.441+0000 7f3d8aa00640 1 -- 172.21.15.134:0/3758204327 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d6c0b3dc0 msgr2=0x7f3d6c0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.441+0000 7f3d8aa00640 1 --2- 172.21.15.134:0/3758204327 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d6c0b3dc0 0x7f3d6c0b62d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c009420 tx=0x7f3d7c000f40 comp rx=0 tx=0).stop 2024-07-26T11:21:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.441+0000 7f3d8aa00640 1 -- 172.21.15.134:0/3758204327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d84115f40 msgr2=0x7f3d841a7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:30.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.441+0000 7f3d8aa00640 1 --2- 172.21.15.134:0/3758204327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d84115f40 0x7f3d841a7d00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3d7800e940 tx=0x7f3d7800ee00 comp rx=0 tx=0).stop 2024-07-26T11:21:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.442+0000 7f3d8aa00640 1 -- 172.21.15.134:0/3758204327 shutdown_connections 2024-07-26T11:21:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.442+0000 7f3d8aa00640 1 --2- 172.21.15.134:0/3758204327 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f3d6c0b3dc0 0x7f3d6c0b62d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.442+0000 7f3d8aa00640 1 --2- 172.21.15.134:0/3758204327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d84115f40 0x7f3d841a7d00 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.442+0000 7f3d8aa00640 1 --2- 172.21.15.134:0/3758204327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f3d8410a740 0x7f3d841a2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.442+0000 7f3d8aa00640 1 --2- 172.21.15.134:0/3758204327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d84109df0 0x7f3d841a24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.442+0000 7f3d8aa00640 1 -- 172.21.15.134:0/3758204327 >> 172.21.15.134:0/3758204327 conn(0x7f3d84070630 msgr2=0x7f3d8410fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.443+0000 7f3d8aa00640 1 -- 172.21.15.134:0/3758204327 shutdown_connections 2024-07-26T11:21:30.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:30.443+0000 7f3d8aa00640 1 -- 172.21.15.134:0/3758204327 wait complete. 2024-07-26T11:21:30.850 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:21:30.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:30 smithi134 ceph-mon[31966]: pgmap v646: 129 pgs: 129 active+clean; 38 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 29 KiB/s rd, 239 MiB/s wr, 2.09k op/s 2024-07-26T11:21:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:30 smithi189 ceph-mon[36790]: pgmap v646: 129 pgs: 129 active+clean; 38 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 29 KiB/s rd, 239 MiB/s wr, 2.09k op/s 2024-07-26T11:21:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:30 smithi193 ceph-mon[37473]: pgmap v646: 129 pgs: 129 active+clean; 38 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 29 KiB/s rd, 239 MiB/s wr, 2.09k op/s 2024-07-26T11:21:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:31 smithi134 ceph-mon[31966]: scrub summary: idle 2024-07-26T11:21:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:31 smithi134 ceph-mon[31966]: scrub summary: idle 2024-07-26T11:21:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:31 smithi189 ceph-mon[36790]: scrub summary: idle 2024-07-26T11:21:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:31 smithi189 ceph-mon[36790]: scrub summary: idle 2024-07-26T11:21:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:31 smithi193 ceph-mon[37473]: scrub summary: idle 2024-07-26T11:21:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:31 smithi193 ceph-mon[37473]: scrub summary: idle 2024-07-26T11:21:33.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:33 smithi189 ceph-mon[36790]: pgmap v647: 129 pgs: 129 active+clean; 38 GiB data, 79 GiB used, 993 GiB / 1.0 TiB avail; 274 KiB/s rd, 180 MiB/s wr, 1.64k op/s 2024-07-26T11:21:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:33 smithi193 ceph-mon[37473]: pgmap v647: 129 pgs: 129 active+clean; 38 GiB data, 79 GiB used, 993 GiB / 1.0 TiB avail; 274 KiB/s rd, 180 MiB/s wr, 1.64k op/s 2024-07-26T11:21:34.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:33 smithi134 ceph-mon[31966]: pgmap v647: 129 pgs: 129 active+clean; 38 GiB data, 79 GiB used, 993 GiB / 1.0 TiB avail; 274 KiB/s rd, 180 MiB/s wr, 1.64k op/s 2024-07-26T11:21:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:35 smithi193 ceph-mon[37473]: pgmap v648: 129 pgs: 129 active+clean; 39 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 269 KiB/s rd, 204 MiB/s wr, 1.81k op/s 2024-07-26T11:21:36.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:35 smithi134 ceph-mon[31966]: pgmap v648: 129 pgs: 129 active+clean; 39 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 269 KiB/s rd, 204 MiB/s wr, 1.81k op/s 2024-07-26T11:21:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:35 smithi189 ceph-mon[36790]: pgmap v648: 129 pgs: 129 active+clean; 39 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 269 KiB/s rd, 204 MiB/s wr, 1.81k op/s 2024-07-26T11:21:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:37 smithi193 ceph-mon[37473]: pgmap v649: 129 pgs: 129 active+clean; 39 GiB data, 81 GiB used, 991 GiB / 1.0 TiB avail; 278 KiB/s rd, 231 MiB/s wr, 1.93k op/s 2024-07-26T11:21:38.178 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:37 smithi134 ceph-mon[31966]: pgmap v649: 129 pgs: 129 active+clean; 39 GiB data, 81 GiB used, 991 GiB / 1.0 TiB avail; 278 KiB/s rd, 231 MiB/s wr, 1.93k op/s 2024-07-26T11:21:38.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:37 smithi189 ceph-mon[36790]: pgmap v649: 129 pgs: 129 active+clean; 39 GiB data, 81 GiB used, 991 GiB / 1.0 TiB avail; 278 KiB/s rd, 231 MiB/s wr, 1.93k op/s 2024-07-26T11:21:39.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.793+0000 7f2ffe800640 1 -- 172.21.15.134:0/2032512831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff810a740 msgr2=0x7f2ff8117df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:39.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.793+0000 7f2ffe800640 1 --2- 172.21.15.134:0/2032512831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff810a740 0x7f2ff8117df0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4009900 tx=0x7f2fe402a420 comp rx=0 tx=0).stop 2024-07-26T11:21:39.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.794+0000 7f2ffe800640 1 -- 172.21.15.134:0/2032512831 shutdown_connections 2024-07-26T11:21:39.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.794+0000 7f2ffe800640 1 --2- 172.21.15.134:0/2032512831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ff8118330 0x7f2ff811a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.795+0000 7f2ffe800640 1 --2- 172.21.15.134:0/2032512831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff810a740 0x7f2ff8117df0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.795+0000 7f2ffe800640 1 --2- 172.21.15.134:0/2032512831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ff8109df0 0x7f2ff810a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.795+0000 7f2ffe800640 1 -- 172.21.15.134:0/2032512831 >> 172.21.15.134:0/2032512831 conn(0x7f2ff806e880 msgr2=0x7f2ff806ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:39.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.795+0000 7f2ffe800640 1 -- 172.21.15.134:0/2032512831 shutdown_connections 2024-07-26T11:21:39.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.795+0000 7f2ffe800640 1 -- 172.21.15.134:0/2032512831 wait complete. 2024-07-26T11:21:39.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.797+0000 7f2ffe800640 1 Processor -- start 2024-07-26T11:21:39.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.798+0000 7f2ffe800640 1 -- start start 2024-07-26T11:21:39.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.799+0000 7f2ffe800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8109df0 0x7f2ff81a4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.799+0000 7f2ffe800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ff810a740 0x7f2ff81a4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.799+0000 7f2ffe800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ff8118330 0x7f2ff81a9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.799+0000 7f2ffe800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff811dd30 con 0x7f2ff8118330 2024-07-26T11:21:39.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.799+0000 7f2ffe800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff811dbb0 con 0x7f2ff8109df0 2024-07-26T11:21:39.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.799+0000 7f2ffe800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff811deb0 con 0x7f2ff810a740 2024-07-26T11:21:39.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.799+0000 7f2ffd400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8109df0 0x7f2ff81a4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:39.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.800+0000 7f2ffca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ff810a740 0x7f2ff81a4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:39.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.800+0000 7f2ffca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ff810a740 0x7f2ff81a4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:43770/0 (socket says 172.21.15.134:43770) 2024-07-26T11:21:39.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.800+0000 7f2ffca00640 1 -- 172.21.15.134:0/4104104752 learned_addr learned my addr 172.21.15.134:0/4104104752 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:39.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.800+0000 7f2ffde00640 1 --2- 172.21.15.134:0/4104104752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ff8118330 0x7f2ff81a9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:39.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.801+0000 7f2ffde00640 1 -- 172.21.15.134:0/4104104752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ff810a740 msgr2=0x7f2ff81a4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:39.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.801+0000 7f2ffde00640 1 --2- 172.21.15.134:0/4104104752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ff810a740 0x7f2ff81a4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.801+0000 7f2ffde00640 1 -- 172.21.15.134:0/4104104752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8109df0 msgr2=0x7f2ff81a4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:39.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.801+0000 7f2ffde00640 1 --2- 172.21.15.134:0/4104104752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8109df0 0x7f2ff81a4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.801+0000 7f2ffde00640 1 -- 172.21.15.134:0/4104104752 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff81aa5b0 con 0x7f2ff8118330 2024-07-26T11:21:39.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.801+0000 7f2ffd400640 1 --2- 172.21.15.134:0/4104104752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8109df0 0x7f2ff81a4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:21:39.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.802+0000 7f2ffde00640 1 --2- 172.21.15.134:0/4104104752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ff8118330 0x7f2ff81a9f40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000cab0 tx=0x7f2ff000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:39.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.803+0000 7f2ff6000640 1 -- 172.21.15.134:0/4104104752 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff000ae20 con 0x7f2ff8118330 2024-07-26T11:21:39.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.803+0000 7f2ffe800640 1 -- 172.21.15.134:0/4104104752 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff81aa8a0 con 0x7f2ff8118330 2024-07-26T11:21:39.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.803+0000 7f2ffe800640 1 -- 172.21.15.134:0/4104104752 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ff81b10c0 con 0x7f2ff8118330 2024-07-26T11:21:39.807 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.804+0000 7f2ff6000640 1 -- 172.21.15.134:0/4104104752 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ff000f900 con 0x7f2ff8118330 2024-07-26T11:21:39.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.804+0000 7f2ff6000640 1 -- 172.21.15.134:0/4104104752 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff00189f0 con 0x7f2ff8118330 2024-07-26T11:21:39.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.808+0000 7f3006a55740 1 Processor -- start 2024-07-26T11:21:39.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.820+0000 7f2ff6000640 1 -- 172.21.15.134:0/4104104752 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f2ff0018cd0 con 0x7f2ff8118330 2024-07-26T11:21:39.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.820+0000 7f3006a55740 1 -- start start 2024-07-26T11:21:39.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.821+0000 7f3006a55740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b94070 0x563002f38fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.821+0000 7f3006a55740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b6b1b0 0x563002f39510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.821+0000 7f3006a55740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b73390 0x563002f39a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.821+0000 7f3006a55740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563002eafb90 con 0x563002b94070 2024-07-26T11:21:39.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.821+0000 7f3006a55740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563002e9f090 con 0x563002b6b1b0 2024-07-26T11:21:39.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.821+0000 7f3006a55740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563002ea30d0 con 0x563002b73390 2024-07-26T11:21:39.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.824+0000 7f2ffd400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b6b1b0 0x563002f39510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:39.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.824+0000 7f2ffd400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b6b1b0 0x563002f39510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:57256/0 (socket says 172.21.15.134:57256) 2024-07-26T11:21:39.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.824+0000 7f2ffd400640 1 -- 172.21.15.134:0/1171785594 learned_addr learned my addr 172.21.15.134:0/1171785594 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:39.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.828+0000 7f2ff6000640 1 --2- 172.21.15.134:0/4104104752 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2fe00b3e30 0x7f2fe00b6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.828+0000 7f2ff6000640 1 -- 172.21.15.134:0/4104104752 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2ff00e4470 con 0x7f2ff8118330 2024-07-26T11:21:39.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.834+0000 7f2ffd400640 1 -- 172.21.15.134:0/1171785594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b73390 msgr2=0x563002f39a50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:21:39.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.834+0000 7f2ffd400640 1 --2- 172.21.15.134:0/1171785594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b73390 0x563002f39a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.834+0000 7f2ffd400640 1 -- 172.21.15.134:0/1171785594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b94070 msgr2=0x563002f38fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:39.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.834+0000 7f2ffd400640 1 --2- 172.21.15.134:0/1171785594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b94070 0x563002f38fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.834+0000 7f2ffd400640 1 -- 172.21.15.134:0/1171785594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563002bff650 con 0x563002b6b1b0 2024-07-26T11:21:39.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.836+0000 7f2ffd400640 1 --2- 172.21.15.134:0/1171785594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b6b1b0 0x563002f39510 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2fec011540 tx=0x7f2fec011a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b44afea97e011f9f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:39.838 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.837+0000 7f2ffde00640 1 --2- 172.21.15.134:0/4104104752 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2fe00b3e30 0x7f2fe00b6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:39.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.838+0000 7f2fea600640 1 -- 172.21.15.134:0/1171785594 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec00f640 con 0x563002b6b1b0 2024-07-26T11:21:39.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.838+0000 7f2fea600640 1 -- 172.21.15.134:0/1171785594 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fec00fc90 con 0x563002b6b1b0 2024-07-26T11:21:39.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.838+0000 7f2fea600640 1 -- 172.21.15.134:0/1171785594 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec019c50 con 0x563002b6b1b0 2024-07-26T11:21:39.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.841+0000 7f2ffde00640 1 --2- 172.21.15.134:0/4104104752 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2fe00b3e30 0x7f2fe00b6340 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00aaa00 tx=0x7f2ff000a890 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:39.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.841+0000 7f3006a55740 1 -- 172.21.15.134:0/1171785594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b6b1b0 msgr2=0x563002f39510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:39.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.841+0000 7f3006a55740 1 --2- 172.21.15.134:0/1171785594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b6b1b0 0x563002f39510 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2fec011540 tx=0x7f2fec011a00 comp rx=0 tx=0).stop 2024-07-26T11:21:39.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.843+0000 7f3006a55740 1 -- 172.21.15.134:0/1171785594 shutdown_connections 2024-07-26T11:21:39.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.843+0000 7f3006a55740 1 --2- 172.21.15.134:0/1171785594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b73390 0x563002f39a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.843+0000 7f3006a55740 1 --2- 172.21.15.134:0/1171785594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b6b1b0 0x563002f39510 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.843+0000 7f3006a55740 1 --2- 172.21.15.134:0/1171785594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b94070 0x563002f38fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.843+0000 7f3006a55740 1 -- 172.21.15.134:0/1171785594 >> 172.21.15.134:0/1171785594 conn(0x563002b88460 msgr2=0x563002f36ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:39.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.845+0000 7f3006a55740 1 -- 172.21.15.134:0/1171785594 shutdown_connections 2024-07-26T11:21:39.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.845+0000 7f3006a55740 1 -- 172.21.15.134:0/1171785594 wait complete. 2024-07-26T11:21:39.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.850+0000 7f3006a55740 1 -- start start 2024-07-26T11:21:39.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.851+0000 7f3006a55740 1 Processor -- start 2024-07-26T11:21:39.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.854+0000 7f3006a55740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f44ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.855+0000 7f3006a55740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b73390 0x563002f45480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.855+0000 7f3006a55740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b94070 0x563002f4a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.855+0000 7f3006a55740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563002d02700 con 0x563002b94070 2024-07-26T11:21:39.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.855+0000 7f3006a55740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563002e3b660 con 0x563002b73390 2024-07-26T11:21:39.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.855+0000 7f3006a55740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563002f3f680 con 0x563002b6b1b0 2024-07-26T11:21:39.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.855+0000 7f3006a55740 10 client.0 _ll_register_callbacks cb 0x563002e4b110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:39.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.858+0000 7f2ffca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f44ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:39.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.858+0000 7f2ffca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f44ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:54082/0 (socket says 172.21.15.134:54082) 2024-07-26T11:21:39.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.858+0000 7f2ffca00640 1 -- 172.21.15.134:0/3692370263 learned_addr learned my addr 172.21.15.134:0/3692370263 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:39.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.860+0000 7f2ffca00640 1 -- 172.21.15.134:0/3692370263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b73390 msgr2=0x563002f45480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:39.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.859+0000 7f2ffd400640 1 --2- 172.21.15.134:0/3692370263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b73390 0x563002f45480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:39.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.872+0000 7f2ffca00640 1 --2- 172.21.15.134:0/3692370263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b73390 0x563002f45480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.872+0000 7f2ffca00640 1 -- 172.21.15.134:0/3692370263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b94070 msgr2=0x563002f4a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:39.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.872+0000 7f2ffca00640 1 --2- 172.21.15.134:0/3692370263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b94070 0x563002f4a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.872+0000 7f2ffca00640 1 -- 172.21.15.134:0/3692370263 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563002e52800 con 0x563002b6b1b0 2024-07-26T11:21:39.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.873+0000 7f2ffd400640 1 --2- 172.21.15.134:0/3692370263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b73390 0x563002f45480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:21:39.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.873+0000 7f2ffca00640 1 --2- 172.21.15.134:0/3692370263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f44ec0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4009340 tx=0x7f2fe4003660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5e59a2ab245ab44 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:39.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.875+0000 7f2fdea00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:21:39.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.875+0000 7f2fdea00640 1 -- 172.21.15.134:0/3692370263 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4031a80 con 0x563002b6b1b0 2024-07-26T11:21:39.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.875+0000 7f2fdea00640 1 -- 172.21.15.134:0/3692370263 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2fe4031c20 con 0x563002b6b1b0 2024-07-26T11:21:39.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.875+0000 7f3006a55740 1 -- 172.21.15.134:0/3692370263 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fec00fc90 con 0x563002b6b1b0 2024-07-26T11:21:39.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.876+0000 7f2fdea00640 1 -- 172.21.15.134:0/3692370263 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2fe4050050 con 0x563002b6b1b0 2024-07-26T11:21:39.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.876+0000 7f3006a55740 10 client.26344 fetch_fsmap learned FSMap version 32 2024-07-26T11:21:39.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.876+0000 7f3006a55740 1 -- 172.21.15.134:0/3692370263 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563002f64440 con 0x563002b6b1b0 2024-07-26T11:21:39.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.878+0000 7f2fdea00640 1 -- 172.21.15.134:0/3692370263 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fec00fc90 con 0x563002b6b1b0 2024-07-26T11:21:39.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.882+0000 7f2fdea00640 1 -- 172.21.15.134:0/3692370263 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f2fe403f480 con 0x563002b6b1b0 2024-07-26T11:21:39.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.882+0000 7f3006a55740 10 client.26344 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:21:39.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.882+0000 7f3006a55740 10 client.26344 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:21:39.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.882+0000 7f3006a55740 20 client.26344 populate_metadata read hostname 'smithi134' 2024-07-26T11:21:39.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.882+0000 7f3006a55740 1 --2- 172.21.15.134:0/3692370263 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563002b529f0 0x563002f67910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.882+0000 7f3006a55740 4 client.26344 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:21:39.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.882+0000 7f3006a55740 1 -- 172.21.15.134:0/3692370263 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563002eb1ae0 con 0x563002b529f0 2024-07-26T11:21:39.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.885+0000 7f2ffd400640 1 --2- 172.21.15.134:0/3692370263 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563002b529f0 0x563002f67910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:39.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.886+0000 7f2ffd400640 1 --2- 172.21.15.134:0/3692370263 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563002b529f0 0x563002f67910 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cefe8c021752f13 server_cookie=2261ef208dad6776 in_seq=0 out_seq=0 2024-07-26T11:21:39.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.887+0000 7f2fdea00640 10 client.26344 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:39.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.891+0000 7f2fdea00640 1 -- 172.21.15.134:0/3692370263 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563002eb1ae0 con 0x563002b529f0 2024-07-26T11:21:39.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.891+0000 7f2fdea00640 10 client.26344 handle_command_reply: tid=0 2024-07-26T11:21:39.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.891+0000 7f3006a55740 1 client.26344 shutdown 2024-07-26T11:21:39.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.892+0000 7f3006a55740 1 -- 172.21.15.134:0/3692370263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 msgr2=0x563002f44ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:39.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.892+0000 7f3006a55740 1 --2- 172.21.15.134:0/3692370263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f44ec0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2fe4009340 tx=0x7f2fe4003660 comp rx=0 tx=0).stop 2024-07-26T11:21:39.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.892+0000 7f3006a55740 1 -- 172.21.15.134:0/3692370263 shutdown_connections 2024-07-26T11:21:39.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.892+0000 7f3006a55740 1 --2- 172.21.15.134:0/3692370263 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563002b529f0 0x563002f67910 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.893+0000 7f3006a55740 1 --2- 172.21.15.134:0/3692370263 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b94070 0x563002f4a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.893+0000 7f3006a55740 1 --2- 172.21.15.134:0/3692370263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b73390 0x563002f45480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.893+0000 7f3006a55740 1 --2- 172.21.15.134:0/3692370263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f44ec0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.893+0000 7f3006a55740 1 -- 172.21.15.134:0/3692370263 >> 172.21.15.134:0/3692370263 conn(0x563002b88460 msgr2=0x563002f35500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:39.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.893+0000 7f2fdea00640 0 client.26344 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:39.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.896+0000 7f3006a55740 1 -- 172.21.15.134:0/3692370263 shutdown_connections 2024-07-26T11:21:39.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.897+0000 7f3006a55740 1 -- 172.21.15.134:0/3692370263 wait complete. 2024-07-26T11:21:39.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.897+0000 7f3006a55740 20 client.26344 trim_cache size 0 max 16384 2024-07-26T11:21:39.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.935+0000 7f3006a55740 1 Processor -- start 2024-07-26T11:21:39.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.939+0000 7f3006a55740 1 -- start start 2024-07-26T11:21:39.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.939+0000 7f3006a55740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f36b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.939+0000 7f3006a55740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b73390 0x563002f3a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.939+0000 7f3006a55740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b94070 0x563002f4a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.939+0000 7f3006a55740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd800fff0 con 0x563002b73390 2024-07-26T11:21:39.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.939+0000 7f3006a55740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd8011b30 con 0x563002b94070 2024-07-26T11:21:39.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.939+0000 7f3006a55740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd80106c0 con 0x563002b6b1b0 2024-07-26T11:21:39.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.940+0000 7f2ffca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f36b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:39.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.941+0000 7f2ffca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f36b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:54084/0 (socket says 172.21.15.134:54084) 2024-07-26T11:21:39.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.941+0000 7f2ffca00640 1 -- 172.21.15.134:0/2464859023 learned_addr learned my addr 172.21.15.134:0/2464859023 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:39.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.941+0000 7f2ffca00640 1 --2- 172.21.15.134:0/2464859023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b94070 0x563002f4a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:39.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.941+0000 7f2ffca00640 1 -- 172.21.15.134:0/2464859023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b94070 msgr2=0x563002f4a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:39.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.942+0000 7f2ffca00640 1 --2- 172.21.15.134:0/2464859023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b94070 0x563002f4a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.942+0000 7f2ffca00640 1 -- 172.21.15.134:0/2464859023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b73390 msgr2=0x563002f3a710 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:21:39.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.942+0000 7f2ffca00640 1 --2- 172.21.15.134:0/2464859023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b73390 0x563002f3a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.942+0000 7f2ffca00640 1 -- 172.21.15.134:0/2464859023 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563002b52c20 con 0x563002b6b1b0 2024-07-26T11:21:39.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.942+0000 7f2ffca00640 1 --2- 172.21.15.134:0/2464859023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f36b80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2fe403f880 tx=0x7f2fe4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48ead179bd130b03 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:39.968 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:39 smithi134 ceph-mon[31966]: pgmap v650: 129 pgs: 129 active+clean; 39 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 279 KiB/s rd, 213 MiB/s wr, 1.67k op/s 2024-07-26T11:21:39.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.959+0000 7f2fe9c00640 1 -- 172.21.15.134:0/2464859023 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe403f640 con 0x563002b6b1b0 2024-07-26T11:21:39.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.959+0000 7f2fe9c00640 1 -- 172.21.15.134:0/2464859023 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fe404d070 con 0x563002b6b1b0 2024-07-26T11:21:39.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.959+0000 7f2fe9c00640 1 -- 172.21.15.134:0/2464859023 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe4031ac0 con 0x563002b6b1b0 2024-07-26T11:21:39.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.961+0000 7f3006a55740 1 -- 172.21.15.134:0/2464859023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 msgr2=0x563002f36b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:39.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.961+0000 7f3006a55740 1 --2- 172.21.15.134:0/2464859023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f36b80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2fe403f880 tx=0x7f2fe4000bc0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.962+0000 7f3006a55740 1 -- 172.21.15.134:0/2464859023 shutdown_connections 2024-07-26T11:21:39.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.962+0000 7f3006a55740 1 --2- 172.21.15.134:0/2464859023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b94070 0x563002f4a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.962+0000 7f3006a55740 1 --2- 172.21.15.134:0/2464859023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b73390 0x563002f3a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.962+0000 7f3006a55740 1 --2- 172.21.15.134:0/2464859023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f36b80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.962+0000 7f3006a55740 1 -- 172.21.15.134:0/2464859023 >> 172.21.15.134:0/2464859023 conn(0x563002b88460 msgr2=0x563002f3a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:39.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.963+0000 7f3006a55740 1 -- 172.21.15.134:0/2464859023 shutdown_connections 2024-07-26T11:21:39.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.963+0000 7f3006a55740 1 -- 172.21.15.134:0/2464859023 wait complete. 2024-07-26T11:21:39.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.963+0000 7f3006a55740 1 -- start start 2024-07-26T11:21:39.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.964+0000 7f3006a55740 1 Processor -- start 2024-07-26T11:21:39.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.968+0000 7f3006a55740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f79580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.968+0000 7f3006a55740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b73390 0x563002f74a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.968+0000 7f2ffca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f79580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:39.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.970+0000 7f2ffca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f79580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:54096/0 (socket says 172.21.15.134:54096) 2024-07-26T11:21:39.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.970+0000 7f2ffca00640 1 -- 172.21.15.134:0/4091321398 learned_addr learned my addr 172.21.15.134:0/4091321398 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:39.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.973+0000 7f3006a55740 1 --2- 172.21.15.134:0/4091321398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b94070 0x563002f74fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.973+0000 7f3006a55740 1 -- 172.21.15.134:0/4091321398 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd8011460 con 0x563002b73390 2024-07-26T11:21:39.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.973+0000 7f3006a55740 1 -- 172.21.15.134:0/4091321398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd8010d90 con 0x563002b94070 2024-07-26T11:21:39.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.973+0000 7f3006a55740 1 -- 172.21.15.134:0/4091321398 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd800dfd0 con 0x563002b6b1b0 2024-07-26T11:21:39.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.974+0000 7f3006a55740 10 client.0 _ll_register_callbacks cb 0x7f2fe4009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:39.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.974+0000 7f2ffd400640 1 --2- 172.21.15.134:0/4091321398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b73390 0x563002f74a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:39.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.975+0000 7f2ffd400640 1 -- 172.21.15.134:0/4091321398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 msgr2=0x563002f79580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:39.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.975+0000 7f2ffd400640 1 --2- 172.21.15.134:0/4091321398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f79580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.975+0000 7f2ffca00640 1 --2- 172.21.15.134:0/4091321398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b94070 0x563002f74fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:39.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.975+0000 7f2ffd400640 1 -- 172.21.15.134:0/4091321398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b94070 msgr2=0x563002f74fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:39.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.975+0000 7f2ffd400640 1 --2- 172.21.15.134:0/4091321398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b94070 0x563002f74fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:39.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.975+0000 7f2ffd400640 1 -- 172.21.15.134:0/4091321398 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563002f75760 con 0x563002b73390 2024-07-26T11:21:39.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.976+0000 7f2fdea00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:21:39.981 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.976+0000 7f2ffd400640 1 --2- 172.21.15.134:0/4091321398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b73390 0x563002f74a30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009ec0 tx=0x7f2fec005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b68bedf1df386664 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:39.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.979+0000 7f2fdea00640 1 -- 172.21.15.134:0/4091321398 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fec0055d0 con 0x563002b73390 2024-07-26T11:21:39.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.979+0000 7f3006a55740 1 -- 172.21.15.134:0/4091321398 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fe404d070 con 0x563002b73390 2024-07-26T11:21:39.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.981+0000 7f2fdea00640 1 -- 172.21.15.134:0/4091321398 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f2fec019710 con 0x563002b73390 2024-07-26T11:21:39.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.982+0000 7f2fdea00640 1 -- 172.21.15.134:0/4091321398 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f2fec0199b0 con 0x563002b73390 2024-07-26T11:21:39.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.982+0000 7f3006a55740 10 client.15945 fetch_fsmap learned FSMap version 32 2024-07-26T11:21:39.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.983+0000 7f3006a55740 1 -- 172.21.15.134:0/4091321398 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563002fcff20 con 0x563002b73390 2024-07-26T11:21:39.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.985+0000 7f2fdea00640 1 -- 172.21.15.134:0/4091321398 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fe404d070 con 0x563002b73390 2024-07-26T11:21:39.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.986+0000 7f2fdea00640 1 -- 172.21.15.134:0/4091321398 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f2fec00f640 con 0x563002b73390 2024-07-26T11:21:39.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.986+0000 7f3006a55740 10 client.15945 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:21:39.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.986+0000 7f3006a55740 10 client.15945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:21:39.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.986+0000 7f3006a55740 20 client.15945 populate_metadata read hostname 'smithi134' 2024-07-26T11:21:39.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.986+0000 7f3006a55740 1 --2- 172.21.15.134:0/4091321398 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563002b27da0 0x563002fd24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:39.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.986+0000 7f3006a55740 4 client.15945 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:21:39.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.986+0000 7f3006a55740 1 -- 172.21.15.134:0/4091321398 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563002ead580 con 0x563002b27da0 2024-07-26T11:21:39.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.987+0000 7f2ffca00640 1 --2- 172.21.15.134:0/4091321398 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563002b27da0 0x563002fd24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:39.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.988+0000 7f2fdea00640 10 client.15945 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:39.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.988+0000 7f2ffca00640 1 --2- 172.21.15.134:0/4091321398 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563002b27da0 0x563002fd24d0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c52ee93400ee041 server_cookie=4c9b7f2e3708a279 in_seq=0 out_seq=0 2024-07-26T11:21:39.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.992+0000 7f2fdea00640 1 -- 172.21.15.134:0/4091321398 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563002ead580 con 0x563002b27da0 2024-07-26T11:21:39.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.992+0000 7f2fdea00640 10 client.15945 handle_command_reply: tid=0 2024-07-26T11:21:39.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.992+0000 7f3006a55740 1 client.15945 shutdown 2024-07-26T11:21:39.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.993+0000 7f3006a55740 1 -- 172.21.15.134:0/4091321398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b73390 msgr2=0x563002f74a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:39.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:39.993+0000 7f3006a55740 1 --2- 172.21.15.134:0/4091321398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b73390 0x563002f74a30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009ec0 tx=0x7f2fec005250 comp rx=0 tx=0).stop 2024-07-26T11:21:40.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.003+0000 7f3006a55740 1 -- 172.21.15.134:0/4091321398 shutdown_connections 2024-07-26T11:21:40.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.003+0000 7f3006a55740 1 --2- 172.21.15.134:0/4091321398 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563002b27da0 0x563002fd24d0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:40.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.003+0000 7f2fdea00640 0 client.15945 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:40.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.003+0000 7f3006a55740 1 --2- 172.21.15.134:0/4091321398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563002b94070 0x563002f74fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:40.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.003+0000 7f3006a55740 1 --2- 172.21.15.134:0/4091321398 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563002b73390 0x563002f74a30 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:40.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.003+0000 7f3006a55740 1 --2- 172.21.15.134:0/4091321398 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563002b6b1b0 0x563002f79580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:40.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.003+0000 7f3006a55740 1 -- 172.21.15.134:0/4091321398 >> 172.21.15.134:0/4091321398 conn(0x563002b88460 msgr2=0x563002f368f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:40.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.004+0000 7f3006a55740 1 -- 172.21.15.134:0/4091321398 shutdown_connections 2024-07-26T11:21:40.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.004+0000 7f3006a55740 1 -- 172.21.15.134:0/4091321398 wait complete. 2024-07-26T11:21:40.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.004+0000 7f3006a55740 20 client.15945 trim_cache size 0 max 16384 2024-07-26T11:21:40.007 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:21:40.008 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "no active scrubs running", 2024-07-26T11:21:40.008 INFO:teuthology.orchestra.run.smithi134.stdout: "scrubs": {} 2024-07-26T11:21:40.008 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:21:40.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.011+0000 7f2ffe800640 1 -- 172.21.15.134:0/4104104752 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2fe00b3e30 msgr2=0x7f2fe00b6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:40.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.011+0000 7f2ffe800640 1 --2- 172.21.15.134:0/4104104752 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2fe00b3e30 0x7f2fe00b6340 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00aaa00 tx=0x7f2ff000a890 comp rx=0 tx=0).stop 2024-07-26T11:21:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.011+0000 7f2ffe800640 1 -- 172.21.15.134:0/4104104752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ff8118330 msgr2=0x7f2ff81a9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.011+0000 7f2ffe800640 1 --2- 172.21.15.134:0/4104104752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ff8118330 0x7f2ff81a9f40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000cab0 tx=0x7f2ff000cf70 comp rx=0 tx=0).stop 2024-07-26T11:21:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.011+0000 7f2ffe800640 1 -- 172.21.15.134:0/4104104752 shutdown_connections 2024-07-26T11:21:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.011+0000 7f2ffe800640 1 --2- 172.21.15.134:0/4104104752 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f2fe00b3e30 0x7f2fe00b6340 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.011+0000 7f2ffe800640 1 --2- 172.21.15.134:0/4104104752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ff810a740 0x7f2ff81a4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.011+0000 7f2ffe800640 1 --2- 172.21.15.134:0/4104104752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f2ff8118330 0x7f2ff81a9f40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.012+0000 7f2ffe800640 1 --2- 172.21.15.134:0/4104104752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ff8109df0 0x7f2ff81a4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.012+0000 7f2ffe800640 1 -- 172.21.15.134:0/4104104752 >> 172.21.15.134:0/4104104752 conn(0x7f2ff806e880 msgr2=0x7f2ff8118920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.012+0000 7f2ffe800640 1 -- 172.21.15.134:0/4104104752 shutdown_connections 2024-07-26T11:21:40.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:40.012+0000 7f2ffe800640 1 -- 172.21.15.134:0/4104104752 wait complete. 2024-07-26T11:21:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:39 smithi193 ceph-mon[37473]: pgmap v650: 129 pgs: 129 active+clean; 39 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 279 KiB/s rd, 213 MiB/s wr, 1.67k op/s 2024-07-26T11:21:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:39 smithi189 ceph-mon[36790]: pgmap v650: 129 pgs: 129 active+clean; 39 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 279 KiB/s rd, 213 MiB/s wr, 1.67k op/s 2024-07-26T11:21:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:41 smithi134 ceph-mon[31966]: pgmap v651: 129 pgs: 129 active+clean; 40 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 274 KiB/s rd, 251 MiB/s wr, 1.84k op/s 2024-07-26T11:21:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:41 smithi189 ceph-mon[36790]: pgmap v651: 129 pgs: 129 active+clean; 40 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 274 KiB/s rd, 251 MiB/s wr, 1.84k op/s 2024-07-26T11:21:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:41 smithi193 ceph-mon[37473]: pgmap v651: 129 pgs: 129 active+clean; 40 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 274 KiB/s rd, 251 MiB/s wr, 1.84k op/s 2024-07-26T11:21:43.674 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-26T11:21:43.675 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:21:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.972+0000 7fec44a00640 1 -- 172.21.15.134:0/2118831106 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec300318a0 con 0x7fec40102a70 2024-07-26T11:21:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.976+0000 7fec48800640 1 -- 172.21.15.134:0/2118831106 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec40102a70 msgr2=0x7fec40106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:43.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.976+0000 7fec48800640 1 --2- 172.21.15.134:0/2118831106 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec40102a70 0x7fec40106f90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fec30009900 tx=0x7fec3002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:21:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.980+0000 7fec48800640 1 -- 172.21.15.134:0/2118831106 shutdown_connections 2024-07-26T11:21:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.980+0000 7fec48800640 1 --2- 172.21.15.134:0/2118831106 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec401075e0 0x7fec40107a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.980+0000 7fec48800640 1 --2- 172.21.15.134:0/2118831106 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec40102a70 0x7fec40106f90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.980+0000 7fec48800640 1 --2- 172.21.15.134:0/2118831106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec40102120 0x7fec40102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.980+0000 7fec48800640 1 -- 172.21.15.134:0/2118831106 >> 172.21.15.134:0/2118831106 conn(0x7fec4007a170 msgr2=0x7fec4007a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.983+0000 7fec48800640 1 -- 172.21.15.134:0/2118831106 shutdown_connections 2024-07-26T11:21:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.983+0000 7fec48800640 1 -- 172.21.15.134:0/2118831106 wait complete. 2024-07-26T11:21:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.986+0000 7fec48800640 1 Processor -- start 2024-07-26T11:21:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.988+0000 7fec48800640 1 -- start start 2024-07-26T11:21:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.989+0000 7fec48800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec40102120 0x7fec40116470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.989+0000 7fec48800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec40102a70 0x7fec4010f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.989+0000 7fec48800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec401075e0 0x7fec4010fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.989+0000 7fec48800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fec40119a70 con 0x7fec40102a70 2024-07-26T11:21:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.989+0000 7fec48800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fec401198f0 con 0x7fec40102120 2024-07-26T11:21:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.989+0000 7fec48800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fec40119bf0 con 0x7fec401075e0 2024-07-26T11:21:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.989+0000 7fec45e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec40102120 0x7fec40116470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.990+0000 7fec45400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec40102a70 0x7fec4010f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.990+0000 7fec45400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec40102a70 0x7fec4010f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50282/0 (socket says 172.21.15.134:50282) 2024-07-26T11:21:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.990+0000 7fec45400640 1 -- 172.21.15.134:0/2034771023 learned_addr learned my addr 172.21.15.134:0/2034771023 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.990+0000 7fec45400640 1 -- 172.21.15.134:0/2034771023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec401075e0 msgr2=0x7fec4010fc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-07-26T11:21:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.990+0000 7fec45400640 1 --2- 172.21.15.134:0/2034771023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec401075e0 0x7fec4010fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.990+0000 7fec45400640 1 -- 172.21.15.134:0/2034771023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec40102120 msgr2=0x7fec40116470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.990+0000 7fec45400640 1 --2- 172.21.15.134:0/2034771023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec40102120 0x7fec40116470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.991+0000 7fec45400640 1 -- 172.21.15.134:0/2034771023 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec401102b0 con 0x7fec40102a70 2024-07-26T11:21:43.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.992+0000 7fec45400640 1 --2- 172.21.15.134:0/2034771023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec40102a70 0x7fec4010f730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fec30009850 tx=0x7fec30030940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:43.998 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.996+0000 7fec3aa00640 1 -- 172.21.15.134:0/2034771023 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3003f070 con 0x7fec40102a70 2024-07-26T11:21:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.997+0000 7fec3aa00640 1 -- 172.21.15.134:0/2034771023 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec30030d40 con 0x7fec40102a70 2024-07-26T11:21:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.997+0000 7fec3aa00640 1 -- 172.21.15.134:0/2034771023 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec30035030 con 0x7fec40102a70 2024-07-26T11:21:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.997+0000 7fec48800640 1 -- 172.21.15.134:0/2034771023 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec40110540 con 0x7fec40102a70 2024-07-26T11:21:44.005 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:43.997+0000 7fec48800640 1 -- 172.21.15.134:0/2034771023 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec40115370 con 0x7fec40102a70 2024-07-26T11:21:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.012+0000 7fec4f9b9740 1 Processor -- start 2024-07-26T11:21:44.014 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.012+0000 7fec4f9b9740 1 -- start start 2024-07-26T11:21:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.015+0000 7fec4f9b9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69f121b0 0x563c6a2b5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.015+0000 7fec4f9b9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69ee92f0 0x563c6a2b5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.015+0000 7fec4f9b9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69ef14d0 0x563c6a2b6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.015+0000 7fec4f9b9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563c6a22a270 con 0x563c69ef14d0 2024-07-26T11:21:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.015+0000 7fec4f9b9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563c6a219770 con 0x563c69ee92f0 2024-07-26T11:21:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.015+0000 7fec4f9b9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563c6a21d7b0 con 0x563c69f121b0 2024-07-26T11:21:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.015+0000 7fec45e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69ee92f0 0x563c6a2b5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.016+0000 7fec46800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69f121b0 0x563c6a2b5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.016+0000 7fec46800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69f121b0 0x563c6a2b5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:54104/0 (socket says 172.21.15.134:54104) 2024-07-26T11:21:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.016+0000 7fec46800640 1 -- 172.21.15.134:0/2368105177 learned_addr learned my addr 172.21.15.134:0/2368105177 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.017+0000 7fec45e00640 1 -- 172.21.15.134:0/2368105177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69f121b0 msgr2=0x563c6a2b5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.017+0000 7fec45e00640 1 --2- 172.21.15.134:0/2368105177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69f121b0 0x563c6a2b5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.017+0000 7fec45e00640 1 -- 172.21.15.134:0/2368105177 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69ef14d0 msgr2=0x563c6a2b6500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:21:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.017+0000 7fec45e00640 1 --2- 172.21.15.134:0/2368105177 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69ef14d0 0x563c6a2b6500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.017+0000 7fec45e00640 1 -- 172.21.15.134:0/2368105177 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c69f7d790 con 0x563c69ee92f0 2024-07-26T11:21:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.020+0000 7fec45e00640 1 --2- 172.21.15.134:0/2368105177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69ee92f0 0x563c6a2b5fc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fec3400cbd0 tx=0x7fec3400e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f95d4b4ad282b50 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.022+0000 7fec3aa00640 1 -- 172.21.15.134:0/2034771023 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fec30042070 con 0x7fec40102a70 2024-07-26T11:21:44.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.024+0000 7fec3aa00640 1 --2- 172.21.15.134:0/2034771023 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fec280b3f10 0x7fec280b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:44.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.024+0000 7fec3aa00640 1 -- 172.21.15.134:0/2034771023 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fec30104e50 con 0x7fec40102a70 2024-07-26T11:21:44.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.025+0000 7fec2f000640 1 -- 172.21.15.134:0/2368105177 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec34004230 con 0x563c69ee92f0 2024-07-26T11:21:44.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.025+0000 7fec2f000640 1 -- 172.21.15.134:0/2368105177 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec340043d0 con 0x563c69ee92f0 2024-07-26T11:21:44.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.025+0000 7fec2f000640 1 -- 172.21.15.134:0/2368105177 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec34004d00 con 0x563c69ee92f0 2024-07-26T11:21:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.028+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2368105177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69ee92f0 msgr2=0x563c6a2b5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.028+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/2368105177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69ee92f0 0x563c6a2b5fc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fec3400cbd0 tx=0x7fec3400e720 comp rx=0 tx=0).stop 2024-07-26T11:21:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.030+0000 7fec46800640 1 --2- 172.21.15.134:0/2034771023 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fec280b3f10 0x7fec280b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.030+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2368105177 shutdown_connections 2024-07-26T11:21:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.030+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/2368105177 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69ef14d0 0x563c6a2b6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.030+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/2368105177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69ee92f0 0x563c6a2b5fc0 secure :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fec3400cbd0 tx=0x7fec3400e720 comp rx=0 tx=0).stop 2024-07-26T11:21:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.030+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/2368105177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69f121b0 0x563c6a2b5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.030+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2368105177 >> 172.21.15.134:0/2368105177 conn(0x563c69f06300 msgr2=0x563c6a2b3970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.035+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2368105177 shutdown_connections 2024-07-26T11:21:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.035+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2368105177 wait complete. 2024-07-26T11:21:44.037 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.036+0000 7fec4f9b9740 1 -- start start 2024-07-26T11:21:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.036+0000 7fec4f9b9740 1 Processor -- start 2024-07-26T11:21:44.039 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.036+0000 7fec46800640 1 --2- 172.21.15.134:0/2034771023 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fec280b3f10 0x7fec280b6420 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fec3c007d70 tx=0x7fec3c0074c0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.039+0000 7fec4f9b9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69ef14d0 0x563c6a2c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.039+0000 7fec4f9b9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69f121b0 0x563c6a2c1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:44.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.041+0000 7fec4f9b9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69ed0d80 0x563c6a2c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.042+0000 7fec45400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69f121b0 0x563c6a2c1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.042+0000 7fec45400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69f121b0 0x563c6a2c1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:54108/0 (socket says 172.21.15.134:54108) 2024-07-26T11:21:44.048 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.042+0000 7fec45400640 1 -- 172.21.15.134:0/411344758 learned_addr learned my addr 172.21.15.134:0/411344758 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:44.048 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.042+0000 7fec46800640 1 --2- 172.21.15.134:0/411344758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69ed0d80 0x563c6a2c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:44.048 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.042+0000 7fec45e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69ef14d0 0x563c6a2c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:44.048 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.045+0000 7fec4f9b9740 1 -- 172.21.15.134:0/411344758 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563c6a153ab0 con 0x563c69ef14d0 2024-07-26T11:21:44.048 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.046+0000 7fec4f9b9740 1 -- 172.21.15.134:0/411344758 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563c6a1b6d50 con 0x563c69ed0d80 2024-07-26T11:21:44.048 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.046+0000 7fec4f9b9740 1 -- 172.21.15.134:0/411344758 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563c6a2c4200 con 0x563c69f121b0 2024-07-26T11:21:44.048 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.046+0000 7fec4f9b9740 10 client.0 _ll_register_callbacks cb 0x563c69f29a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.047+0000 7fec46800640 1 -- 172.21.15.134:0/411344758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69f121b0 msgr2=0x563c6a2c1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.047+0000 7fec46800640 1 --2- 172.21.15.134:0/411344758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69f121b0 0x563c6a2c1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.047+0000 7fec46800640 1 -- 172.21.15.134:0/411344758 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69ef14d0 msgr2=0x563c6a2c18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.047+0000 7fec46800640 1 --2- 172.21.15.134:0/411344758 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69ef14d0 0x563c6a2c18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.047+0000 7fec46800640 1 -- 172.21.15.134:0/411344758 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c69ff6620 con 0x563c69ed0d80 2024-07-26T11:21:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.049+0000 7fec46800640 1 --2- 172.21.15.134:0/411344758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69ed0d80 0x563c6a2c75a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fec3c010e00 tx=0x7fec3c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6be6734d1d9f722e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.049+0000 7fec23400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:21:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.049+0000 7fec23400640 1 -- 172.21.15.134:0/411344758 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c026480 con 0x563c69ed0d80 2024-07-26T11:21:44.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.049+0000 7fec23400640 1 -- 172.21.15.134:0/411344758 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fec3c01bd50 con 0x563c69ed0d80 2024-07-26T11:21:44.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.050+0000 7fec4f9b9740 1 -- 172.21.15.134:0/411344758 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec340043d0 con 0x563c69ed0d80 2024-07-26T11:21:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.051+0000 7fec23400640 1 -- 172.21.15.134:0/411344758 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fec3c026480 con 0x563c69ed0d80 2024-07-26T11:21:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.051+0000 7fec4f9b9740 10 client.26351 fetch_fsmap learned FSMap version 32 2024-07-26T11:21:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.051+0000 7fec4f9b9740 1 -- 172.21.15.134:0/411344758 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563c6a2d0480 con 0x563c69ed0d80 2024-07-26T11:21:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.052+0000 7fec23400640 1 -- 172.21.15.134:0/411344758 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec3c0308d0 con 0x563c69ed0d80 2024-07-26T11:21:44.055 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.052+0000 7fec23400640 1 -- 172.21.15.134:0/411344758 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fec3c01d600 con 0x563c69ed0d80 2024-07-26T11:21:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.052+0000 7fec4f9b9740 10 client.26351 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:21:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.052+0000 7fec4f9b9740 10 client.26351 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:21:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.052+0000 7fec4f9b9740 20 client.26351 populate_metadata read hostname 'smithi134' 2024-07-26T11:21:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.052+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/411344758 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563c69ea61c0 0x563c6a2d3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.053+0000 7fec4f9b9740 4 client.26351 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:21:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.053+0000 7fec4f9b9740 1 -- 172.21.15.134:0/411344758 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fec340043d0 con 0x563c69ea61c0 2024-07-26T11:21:44.059 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.057+0000 7fec45e00640 1 --2- 172.21.15.134:0/411344758 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563c69ea61c0 0x563c6a2d3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.062+0000 7fec23400640 10 client.26351 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.062+0000 7fec45e00640 1 --2- 172.21.15.134:0/411344758 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563c69ea61c0 0x563c6a2d3a60 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18160d9cbf57a0be server_cookie=fe2434bdfd7a9ea in_seq=0 out_seq=0 2024-07-26T11:21:44.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.065+0000 7fec23400640 1 -- 172.21.15.134:0/411344758 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fec340043d0 con 0x563c69ea61c0 2024-07-26T11:21:44.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.066+0000 7fec23400640 10 client.26351 handle_command_reply: tid=0 2024-07-26T11:21:44.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.066+0000 7fec4f9b9740 1 client.26351 shutdown 2024-07-26T11:21:44.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.066+0000 7fec4f9b9740 1 -- 172.21.15.134:0/411344758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69ed0d80 msgr2=0x563c6a2c75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:44.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.066+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/411344758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69ed0d80 0x563c6a2c75a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fec3c010e00 tx=0x7fec3c004ab0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.066+0000 7fec4f9b9740 1 -- 172.21.15.134:0/411344758 shutdown_connections 2024-07-26T11:21:44.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.066+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/411344758 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563c69ea61c0 0x563c6a2d3a60 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.066+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/411344758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69f121b0 0x563c6a2c1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.066+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/411344758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69ed0d80 0x563c6a2c75a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.066+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/411344758 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69ef14d0 0x563c6a2c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.066+0000 7fec4f9b9740 1 -- 172.21.15.134:0/411344758 >> 172.21.15.134:0/411344758 conn(0x563c69f06300 msgr2=0x563c6a2b20d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:44.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.066+0000 7fec23400640 0 client.26351 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:44.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.067+0000 7fec4f9b9740 1 -- 172.21.15.134:0/411344758 shutdown_connections 2024-07-26T11:21:44.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.067+0000 7fec4f9b9740 1 -- 172.21.15.134:0/411344758 wait complete. 2024-07-26T11:21:44.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.067+0000 7fec4f9b9740 20 client.26351 trim_cache size 0 max 16384 2024-07-26T11:21:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.115+0000 7fec4f9b9740 1 Processor -- start 2024-07-26T11:21:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.115+0000 7fec4f9b9740 1 -- start start 2024-07-26T11:21:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.115+0000 7fec4f9b9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69ed0d80 0x563c6a2bb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:44.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.115+0000 7fec4f9b9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69ef14d0 0x563c6a2ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.115+0000 7fec4f9b9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69f121b0 0x563c6a2c7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.115+0000 7fec4f9b9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fec401c7da0 con 0x563c69ef14d0 2024-07-26T11:21:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.115+0000 7fec4f9b9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fec401c98e0 con 0x563c69f121b0 2024-07-26T11:21:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.115+0000 7fec4f9b9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fec401c8470 con 0x563c69ed0d80 2024-07-26T11:21:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.116+0000 7fec46800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69f121b0 0x563c6a2c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.116+0000 7fec45e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69ed0d80 0x563c6a2bb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.116+0000 7fec46800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69f121b0 0x563c6a2c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:48842/0 (socket says 172.21.15.134:48842) 2024-07-26T11:21:44.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.116+0000 7fec46800640 1 -- 172.21.15.134:0/2292971678 learned_addr learned my addr 172.21.15.134:0/2292971678 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.116+0000 7fec46800640 1 -- 172.21.15.134:0/2292971678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69ed0d80 msgr2=0x563c6a2bb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.116+0000 7fec45400640 1 --2- 172.21.15.134:0/2292971678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69ef14d0 0x563c6a2ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.116+0000 7fec46800640 1 --2- 172.21.15.134:0/2292971678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69ed0d80 0x563c6a2bb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.116+0000 7fec46800640 1 -- 172.21.15.134:0/2292971678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69ef14d0 msgr2=0x563c6a2ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.117+0000 7fec46800640 1 --2- 172.21.15.134:0/2292971678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69ef14d0 0x563c6a2ce3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.117+0000 7fec46800640 1 -- 172.21.15.134:0/2292971678 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c6a2c2220 con 0x563c69f121b0 2024-07-26T11:21:44.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.117+0000 7fec45400640 1 --2- 172.21.15.134:0/2292971678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69ef14d0 0x563c6a2ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:21:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.117+0000 7fec46800640 1 --2- 172.21.15.134:0/2292971678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69f121b0 0x563c6a2c7430 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00fc20 tx=0x7fec3c01df20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e81878ef1cf9024b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.118+0000 7fec2e600640 1 -- 172.21.15.134:0/2292971678 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c03f020 con 0x563c69f121b0 2024-07-26T11:21:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.118+0000 7fec2e600640 1 -- 172.21.15.134:0/2292971678 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec3c01bda0 con 0x563c69f121b0 2024-07-26T11:21:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.118+0000 7fec2e600640 1 -- 172.21.15.134:0/2292971678 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c035040 con 0x563c69f121b0 2024-07-26T11:21:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.119+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2292971678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69f121b0 msgr2=0x563c6a2c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:44.121 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.119+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/2292971678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69f121b0 0x563c6a2c7430 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fec3c00fc20 tx=0x7fec3c01df20 comp rx=0 tx=0).stop 2024-07-26T11:21:44.123 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.121+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2292971678 shutdown_connections 2024-07-26T11:21:44.123 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.121+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/2292971678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69f121b0 0x563c6a2c7430 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.123 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.121+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/2292971678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69ef14d0 0x563c6a2ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.123 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.121+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/2292971678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69ed0d80 0x563c6a2bb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.123 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.121+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2292971678 >> 172.21.15.134:0/2292971678 conn(0x563c69f06300 msgr2=0x563c6a2c2be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:44.124 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.122+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2292971678 shutdown_connections 2024-07-26T11:21:44.124 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.122+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2292971678 wait complete. 2024-07-26T11:21:44.124 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.123+0000 7fec4f9b9740 1 -- start start 2024-07-26T11:21:44.125 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.123+0000 7fec4f9b9740 1 Processor -- start 2024-07-26T11:21:44.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.126+0000 7fec4f9b9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69ed0d80 0x563c6a2c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:44.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.126+0000 7fec4f9b9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69ef14d0 0x563c6a2c9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:44.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.126+0000 7fec4f9b9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69f121b0 0x563c6a33a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:44.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.126+0000 7fec4f9b9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fec401c9210 con 0x563c69f121b0 2024-07-26T11:21:44.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.126+0000 7fec4f9b9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fec401c8b40 con 0x563c69ef14d0 2024-07-26T11:21:44.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.126+0000 7fec4f9b9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fec40073e50 con 0x563c69ed0d80 2024-07-26T11:21:44.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.126+0000 7fec4f9b9740 10 client.0 _ll_register_callbacks cb 0x7fec3c005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:44.130 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.127+0000 7fec46800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69f121b0 0x563c6a33a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:44.130 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.127+0000 7fec45e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69ed0d80 0x563c6a2c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:44.130 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.127+0000 7fec46800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69f121b0 0x563c6a33a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50314/0 (socket says 172.21.15.134:50314) 2024-07-26T11:21:44.130 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.127+0000 7fec46800640 1 -- 172.21.15.134:0/2636288388 learned_addr learned my addr 172.21.15.134:0/2636288388 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:44.132 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.128+0000 7fec46800640 1 -- 172.21.15.134:0/2636288388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69ed0d80 msgr2=0x563c6a2c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:44.132 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.128+0000 7fec46800640 1 --2- 172.21.15.134:0/2636288388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69ed0d80 0x563c6a2c8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.132 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.128+0000 7fec46800640 1 -- 172.21.15.134:0/2636288388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69ef14d0 msgr2=0x563c6a2c9210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:21:44.132 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.128+0000 7fec46800640 1 --2- 172.21.15.134:0/2636288388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69ef14d0 0x563c6a2c9210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.132 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.128+0000 7fec46800640 1 -- 172.21.15.134:0/2636288388 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c6a33a720 con 0x563c69f121b0 2024-07-26T11:21:44.132 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.130+0000 7fec45e00640 1 --2- 172.21.15.134:0/2636288388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69ed0d80 0x563c6a2c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:21:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.131+0000 7fec23400640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:21:44.133 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.131+0000 7fec46800640 1 --2- 172.21.15.134:0/2636288388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69f121b0 0x563c6a33a1e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fec3c01da70 tx=0x7fec3c007a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ab7c4c5230dc9e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.132+0000 7fec23400640 1 -- 172.21.15.134:0/2636288388 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3c047020 con 0x563c69f121b0 2024-07-26T11:21:44.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.132+0000 7fec23400640 1 -- 172.21.15.134:0/2636288388 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fec3c01d7a0 con 0x563c69f121b0 2024-07-26T11:21:44.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.133+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2636288388 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec3c01bda0 con 0x563c69f121b0 2024-07-26T11:21:44.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.133+0000 7fec23400640 1 -- 172.21.15.134:0/2636288388 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fec3c047020 con 0x563c69f121b0 2024-07-26T11:21:44.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.133+0000 7fec4f9b9740 10 client.15957 fetch_fsmap learned FSMap version 32 2024-07-26T11:21:44.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.133+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2636288388 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563c6a33b790 con 0x563c69f121b0 2024-07-26T11:21:44.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.134+0000 7fec23400640 1 -- 172.21.15.134:0/2636288388 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec3c01d4d0 con 0x563c69f121b0 2024-07-26T11:21:44.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.134+0000 7fec23400640 1 -- 172.21.15.134:0/2636288388 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fec3c026c00 con 0x563c69f121b0 2024-07-26T11:21:44.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.134+0000 7fec4f9b9740 10 client.15957 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:21:44.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.134+0000 7fec4f9b9740 10 client.15957 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:21:44.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.134+0000 7fec4f9b9740 20 client.15957 populate_metadata read hostname 'smithi134' 2024-07-26T11:21:44.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.134+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/2636288388 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563c6a33be40 0x563c6a33e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:44.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.134+0000 7fec4f9b9740 4 client.15957 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:21:44.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.134+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2636288388 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fec3c01bda0 con 0x563c6a33be40 2024-07-26T11:21:44.141 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.139+0000 7fec45e00640 1 --2- 172.21.15.134:0/2636288388 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563c6a33be40 0x563c6a33e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.143+0000 7fec23400640 10 client.15957 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.143+0000 7fec45e00640 1 --2- 172.21.15.134:0/2636288388 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563c6a33be40 0x563c6a33e260 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1399763fb1df4442 server_cookie=13b26f4080828000 in_seq=0 out_seq=0 2024-07-26T11:21:44.178 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:43 smithi134 ceph-mon[31966]: pgmap v652: 129 pgs: 129 active+clean; 40 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 297 KiB/s rd, 219 MiB/s wr, 1.53k op/s 2024-07-26T11:21:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:43 smithi189 ceph-mon[36790]: pgmap v652: 129 pgs: 129 active+clean; 40 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 297 KiB/s rd, 219 MiB/s wr, 1.53k op/s 2024-07-26T11:21:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.242+0000 7fec23400640 1 -- 172.21.15.134:0/2636288388 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fec3c01bda0 con 0x563c6a33be40 2024-07-26T11:21:44.244 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.242+0000 7fec23400640 10 client.15957 handle_command_reply: tid=0 2024-07-26T11:21:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.243+0000 7fec4f9b9740 1 client.15957 shutdown 2024-07-26T11:21:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.243+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2636288388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69f121b0 msgr2=0x563c6a33a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:44.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.243+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/2636288388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69f121b0 0x563c6a33a1e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fec3c01da70 tx=0x7fec3c007a80 comp rx=0 tx=0).stop 2024-07-26T11:21:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.244+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2636288388 shutdown_connections 2024-07-26T11:21:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.244+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/2636288388 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563c6a33be40 0x563c6a33e260 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.244+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/2636288388 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563c69f121b0 0x563c6a33a1e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.244+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/2636288388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c69ef14d0 0x563c6a2c9210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.244+0000 7fec4f9b9740 1 --2- 172.21.15.134:0/2636288388 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c69ed0d80 0x563c6a2c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.244+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2636288388 >> 172.21.15.134:0/2636288388 conn(0x563c69f06300 msgr2=0x563c6a2bb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:44.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.244+0000 7fec23400640 0 client.15957 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:21:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:21:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:21:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:21:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.249+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2636288388 shutdown_connections 2024-07-26T11:21:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.249+0000 7fec4f9b9740 1 -- 172.21.15.134:0/2636288388 wait complete. 2024-07-26T11:21:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.249+0000 7fec4f9b9740 20 client.15957 trim_cache size 0 max 16384 2024-07-26T11:21:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.250+0000 7fec48800640 1 -- 172.21.15.134:0/2034771023 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fec280b3f10 msgr2=0x7fec280b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.250+0000 7fec48800640 1 --2- 172.21.15.134:0/2034771023 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fec280b3f10 0x7fec280b6420 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fec3c007d70 tx=0x7fec3c0074c0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.250+0000 7fec48800640 1 -- 172.21.15.134:0/2034771023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec40102a70 msgr2=0x7fec4010f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.250+0000 7fec48800640 1 --2- 172.21.15.134:0/2034771023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec40102a70 0x7fec4010f730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fec30009850 tx=0x7fec30030940 comp rx=0 tx=0).stop 2024-07-26T11:21:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.251+0000 7fec48800640 1 -- 172.21.15.134:0/2034771023 shutdown_connections 2024-07-26T11:21:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.251+0000 7fec48800640 1 --2- 172.21.15.134:0/2034771023 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fec280b3f10 0x7fec280b6420 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.251+0000 7fec48800640 1 --2- 172.21.15.134:0/2034771023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fec401075e0 0x7fec4010fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.251+0000 7fec48800640 1 --2- 172.21.15.134:0/2034771023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fec40102a70 0x7fec4010f730 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.251+0000 7fec48800640 1 --2- 172.21.15.134:0/2034771023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec40102120 0x7fec40116470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.251+0000 7fec48800640 1 -- 172.21.15.134:0/2034771023 >> 172.21.15.134:0/2034771023 conn(0x7fec4007a170 msgr2=0x7fec40104f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:44.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.253+0000 7fec48800640 1 -- 172.21.15.134:0/2034771023 shutdown_connections 2024-07-26T11:21:44.256 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:44.253+0000 7fec48800640 1 -- 172.21.15.134:0/2034771023 wait complete. 2024-07-26T11:21:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:43 smithi193 ceph-mon[37473]: pgmap v652: 129 pgs: 129 active+clean; 40 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 297 KiB/s rd, 219 MiB/s wr, 1.53k op/s 2024-07-26T11:21:44.810 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:21:46.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:45 smithi189 ceph-mon[36790]: pgmap v653: 129 pgs: 129 active+clean; 41 GiB data, 85 GiB used, 988 GiB / 1.0 TiB avail; 54 KiB/s rd, 248 MiB/s wr, 1.62k op/s 2024-07-26T11:21:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:45 smithi193 ceph-mon[37473]: pgmap v653: 129 pgs: 129 active+clean; 41 GiB data, 85 GiB used, 988 GiB / 1.0 TiB avail; 54 KiB/s rd, 248 MiB/s wr, 1.62k op/s 2024-07-26T11:21:46.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:45 smithi134 ceph-mon[31966]: pgmap v653: 129 pgs: 129 active+clean; 41 GiB data, 85 GiB used, 988 GiB / 1.0 TiB avail; 54 KiB/s rd, 248 MiB/s wr, 1.62k op/s 2024-07-26T11:21:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:21:47.263 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:21:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:21:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:47 smithi189 ceph-mon[36790]: pgmap v654: 129 pgs: 129 active+clean; 41 GiB data, 86 GiB used, 987 GiB / 1.0 TiB avail; 66 KiB/s rd, 257 MiB/s wr, 1.58k op/s 2024-07-26T11:21:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:47 smithi193 ceph-mon[37473]: pgmap v654: 129 pgs: 129 active+clean; 41 GiB data, 86 GiB used, 987 GiB / 1.0 TiB avail; 66 KiB/s rd, 257 MiB/s wr, 1.58k op/s 2024-07-26T11:21:48.433 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:47 smithi134 ceph-mon[31966]: pgmap v654: 129 pgs: 129 active+clean; 41 GiB data, 86 GiB used, 987 GiB / 1.0 TiB avail; 66 KiB/s rd, 257 MiB/s wr, 1.58k op/s 2024-07-26T11:21:49.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:21:49.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:21:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:21:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:21:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:21:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:21:50.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:50 smithi134 ceph-mon[31966]: pgmap v655: 129 pgs: 129 active+clean; 41 GiB data, 87 GiB used, 986 GiB / 1.0 TiB avail; 56 KiB/s rd, 229 MiB/s wr, 1.34k op/s 2024-07-26T11:21:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:50 smithi189 ceph-mon[36790]: pgmap v655: 129 pgs: 129 active+clean; 41 GiB data, 87 GiB used, 986 GiB / 1.0 TiB avail; 56 KiB/s rd, 229 MiB/s wr, 1.34k op/s 2024-07-26T11:21:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:50 smithi193 ceph-mon[37473]: pgmap v655: 129 pgs: 129 active+clean; 41 GiB data, 87 GiB used, 986 GiB / 1.0 TiB avail; 56 KiB/s rd, 229 MiB/s wr, 1.34k op/s 2024-07-26T11:21:52.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:52 smithi134 ceph-mon[31966]: pgmap v656: 129 pgs: 129 active+clean; 42 GiB data, 88 GiB used, 985 GiB / 1.0 TiB avail; 55 KiB/s rd, 266 MiB/s wr, 1.51k op/s 2024-07-26T11:21:52.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:21:52.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:21:52.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:21:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:52 smithi189 ceph-mon[36790]: pgmap v656: 129 pgs: 129 active+clean; 42 GiB data, 88 GiB used, 985 GiB / 1.0 TiB avail; 55 KiB/s rd, 266 MiB/s wr, 1.51k op/s 2024-07-26T11:21:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:21:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:21:52.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:21:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:52 smithi193 ceph-mon[37473]: pgmap v656: 129 pgs: 129 active+clean; 42 GiB data, 88 GiB used, 985 GiB / 1.0 TiB avail; 55 KiB/s rd, 266 MiB/s wr, 1.51k op/s 2024-07-26T11:21:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:21:52.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:21:52.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:21:54.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:54 smithi134 ceph-mon[31966]: pgmap v657: 129 pgs: 129 active+clean; 42 GiB data, 88 GiB used, 985 GiB / 1.0 TiB avail; 67 KiB/s rd, 230 MiB/s wr, 1.26k op/s 2024-07-26T11:21:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:54 smithi189 ceph-mon[36790]: pgmap v657: 129 pgs: 129 active+clean; 42 GiB data, 88 GiB used, 985 GiB / 1.0 TiB avail; 67 KiB/s rd, 230 MiB/s wr, 1.26k op/s 2024-07-26T11:21:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:54 smithi193 ceph-mon[37473]: pgmap v657: 129 pgs: 129 active+clean; 42 GiB data, 88 GiB used, 985 GiB / 1.0 TiB avail; 67 KiB/s rd, 230 MiB/s wr, 1.26k op/s 2024-07-26T11:21:54.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.993+0000 7fafdee00640 1 -- 172.21.15.134:0/3046521280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafd8109fc0 msgr2=0x7fafd810a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:54.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.993+0000 7fafdee00640 1 --2- 172.21.15.134:0/3046521280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafd8109fc0 0x7fafd810a3d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009900 tx=0x7fafc802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:21:54.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.994+0000 7fafdee00640 1 -- 172.21.15.134:0/3046521280 shutdown_connections 2024-07-26T11:21:54.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.994+0000 7fafdee00640 1 --2- 172.21.15.134:0/3046521280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafd8116070 0x7fafd8118580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:54.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.994+0000 7fafdee00640 1 --2- 172.21.15.134:0/3046521280 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafd810a910 0x7fafd8115a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:54.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.994+0000 7fafdee00640 1 --2- 172.21.15.134:0/3046521280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafd8109fc0 0x7fafd810a3d0 secure :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009900 tx=0x7fafc802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:21:54.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.994+0000 7fafdee00640 1 -- 172.21.15.134:0/3046521280 >> 172.21.15.134:0/3046521280 conn(0x7fafd8070d20 msgr2=0x7fafd8073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:54.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.994+0000 7fafdee00640 1 -- 172.21.15.134:0/3046521280 shutdown_connections 2024-07-26T11:21:54.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.994+0000 7fafdee00640 1 -- 172.21.15.134:0/3046521280 wait complete. 2024-07-26T11:21:55.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.996+0000 7fafdee00640 1 Processor -- start 2024-07-26T11:21:55.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.996+0000 7fafdee00640 1 -- start start 2024-07-26T11:21:55.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.997+0000 7fafdee00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafd810a910 0x7fafd81154c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:55.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.997+0000 7fafdee00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafd8116070 0x7fafd8115a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:55.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.997+0000 7fafdee00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafd81104d0 0x7fafd81109d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:55.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.998+0000 7fafd7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafd810a910 0x7fafd81154c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:55.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.998+0000 7fafd7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafd810a910 0x7fafd81154c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43308/0 (socket says 172.21.15.134:43308) 2024-07-26T11:21:55.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.998+0000 7fafd7e00640 1 -- 172.21.15.134:0/2226439091 learned_addr learned my addr 172.21.15.134:0/2226439091 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:55.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.998+0000 7fafdee00640 1 -- 172.21.15.134:0/2226439091 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fafd811bdc0 con 0x7fafd81104d0 2024-07-26T11:21:55.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.998+0000 7fafdee00640 1 -- 172.21.15.134:0/2226439091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fafd811bc40 con 0x7fafd810a910 2024-07-26T11:21:55.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.998+0000 7fafdee00640 1 -- 172.21.15.134:0/2226439091 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fafd811bf40 con 0x7fafd8116070 2024-07-26T11:21:55.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.998+0000 7fafdce00640 1 --2- 172.21.15.134:0/2226439091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafd81104d0 0x7fafd81109d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:55.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.998+0000 7fafd7400640 1 --2- 172.21.15.134:0/2226439091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafd8116070 0x7fafd8115a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:55.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.999+0000 7fafd7e00640 1 -- 172.21.15.134:0/2226439091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafd8116070 msgr2=0x7fafd8115a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:55.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.999+0000 7fafd7e00640 1 --2- 172.21.15.134:0/2226439091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafd8116070 0x7fafd8115a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:55.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.999+0000 7fafd7e00640 1 -- 172.21.15.134:0/2226439091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafd81104d0 msgr2=0x7fafd81109d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:55.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.999+0000 7fafd7e00640 1 --2- 172.21.15.134:0/2226439091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafd81104d0 0x7fafd81109d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:55.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.999+0000 7fafd7e00640 1 -- 172.21.15.134:0/2226439091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd810f190 con 0x7fafd810a910 2024-07-26T11:21:55.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.999+0000 7fafdce00640 1 --2- 172.21.15.134:0/2226439091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafd81104d0 0x7fafd81109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:21:55.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.999+0000 7fafd7e00640 1 --2- 172.21.15.134:0/2226439091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafd810a910 0x7fafd81154c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fafc802a9e0 tx=0x7fafc8003d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:55.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:54.999+0000 7fafd4c00640 1 -- 172.21.15.134:0/2226439091 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc803f020 con 0x7fafd810a910 2024-07-26T11:21:55.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.003+0000 7fafdee00640 1 -- 172.21.15.134:0/2226439091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd810f420 con 0x7fafd810a910 2024-07-26T11:21:55.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.003+0000 7fafdee00640 1 -- 172.21.15.134:0/2226439091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd810f9a0 con 0x7fafd810a910 2024-07-26T11:21:55.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.003+0000 7fafd4c00640 1 -- 172.21.15.134:0/2226439091 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafc8031d60 con 0x7fafd810a910 2024-07-26T11:21:55.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.003+0000 7fafd4c00640 1 -- 172.21.15.134:0/2226439091 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8035070 con 0x7fafd810a910 2024-07-26T11:21:55.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.013+0000 7fafce000640 1 -- 172.21.15.134:0/2226439091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafb0005290 con 0x7fafd810a910 2024-07-26T11:21:55.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.014+0000 7fafd4c00640 1 -- 172.21.15.134:0/2226439091 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fafc8042020 con 0x7fafd810a910 2024-07-26T11:21:55.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.016+0000 7fafd4c00640 1 --2- 172.21.15.134:0/2226439091 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fafbc0b3df0 0x7fafbc0b6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:55.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.017+0000 7fafd4c00640 1 -- 172.21.15.134:0/2226439091 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fafc8104f30 con 0x7fafd810a910 2024-07-26T11:21:55.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.019+0000 7fafd7400640 1 --2- 172.21.15.134:0/2226439091 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fafbc0b3df0 0x7fafbc0b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:55.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.022+0000 7fafd4c00640 1 -- 172.21.15.134:0/2226439091 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fafc8049020 con 0x7fafd810a910 2024-07-26T11:21:55.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.022+0000 7fafd7400640 1 --2- 172.21.15.134:0/2226439091 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fafbc0b3df0 0x7fafbc0b6300 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fafd8074c90 tx=0x7fafc400a4c0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:55.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.503+0000 7fafce000640 1 -- 172.21.15.134:0/2226439091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafb0005850 con 0x7fafd810a910 2024-07-26T11:21:55.508 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:21:55.508 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":32,"btime":"2024-07-26T11:19:25:331870+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:19:25.131276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24721,"mds_1":24719,"mds_2":24703,"mds_3":24688,"mds_4":24686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24635":{"gid":24635,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24641":{"gid":24641,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":32,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24719":{"gid":24719,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":28,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24721,"qdb_cluster":[24721,24719,24703,24688,24686]},"id":1}]} 2024-07-26T11:21:55.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.505+0000 7fafd4c00640 1 -- 172.21.15.134:0/2226439091 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11990 (secure 0 0 0) 0x7fafc80d4130 con 0x7fafd810a910 2024-07-26T11:21:55.509 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 32 2024-07-26T11:21:55.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.512+0000 7fafce000640 1 -- 172.21.15.134:0/2226439091 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fafbc0b3df0 msgr2=0x7fafbc0b6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:55.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.512+0000 7fafce000640 1 --2- 172.21.15.134:0/2226439091 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fafbc0b3df0 0x7fafbc0b6300 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fafd8074c90 tx=0x7fafc400a4c0 comp rx=0 tx=0).stop 2024-07-26T11:21:55.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.512+0000 7fafce000640 1 -- 172.21.15.134:0/2226439091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafd810a910 msgr2=0x7fafd81154c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:55.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.512+0000 7fafce000640 1 --2- 172.21.15.134:0/2226439091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafd810a910 0x7fafd81154c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fafc802a9e0 tx=0x7fafc8003d80 comp rx=0 tx=0).stop 2024-07-26T11:21:55.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.512+0000 7fafce000640 1 -- 172.21.15.134:0/2226439091 shutdown_connections 2024-07-26T11:21:55.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.512+0000 7fafce000640 1 --2- 172.21.15.134:0/2226439091 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fafbc0b3df0 0x7fafbc0b6300 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:55.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.512+0000 7fafce000640 1 --2- 172.21.15.134:0/2226439091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafd8116070 0x7fafd8115a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:55.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.512+0000 7fafce000640 1 --2- 172.21.15.134:0/2226439091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fafd81104d0 0x7fafd81109d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:55.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.512+0000 7fafce000640 1 --2- 172.21.15.134:0/2226439091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafd810a910 0x7fafd81154c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:55.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.512+0000 7fafce000640 1 -- 172.21.15.134:0/2226439091 >> 172.21.15.134:0/2226439091 conn(0x7fafd8070d20 msgr2=0x7fafd8073860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:55.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.513+0000 7fafce000640 1 -- 172.21.15.134:0/2226439091 shutdown_connections 2024-07-26T11:21:55.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:21:55.514+0000 7fafce000640 1 -- 172.21.15.134:0/2226439091 wait complete. 2024-07-26T11:21:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:56 smithi189 ceph-mon[36790]: pgmap v658: 129 pgs: 129 active+clean; 43 GiB data, 89 GiB used, 984 GiB / 1.0 TiB avail; 31 KiB/s rd, 242 MiB/s wr, 1.30k op/s 2024-07-26T11:21:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:56 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2226439091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:21:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:56 smithi193 ceph-mon[37473]: pgmap v658: 129 pgs: 129 active+clean; 43 GiB data, 89 GiB used, 984 GiB / 1.0 TiB avail; 31 KiB/s rd, 242 MiB/s wr, 1.30k op/s 2024-07-26T11:21:56.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:56 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2226439091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:21:56.548 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:56 smithi134 ceph-mon[31966]: pgmap v658: 129 pgs: 129 active+clean; 43 GiB data, 89 GiB used, 984 GiB / 1.0 TiB avail; 31 KiB/s rd, 242 MiB/s wr, 1.30k op/s 2024-07-26T11:21:56.548 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:56 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2226439091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:21:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.804+0000 7fa64a800640 1 -- 172.21.15.134:0/214180030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6441028b0 msgr2=0x7fa644106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.804+0000 7fa64a800640 1 --2- 172.21.15.134:0/214180030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6441028b0 0x7fa644106dd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa630009900 tx=0x7fa63002a440 comp rx=0 tx=0).stop 2024-07-26T11:21:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.805+0000 7fa64a800640 1 -- 172.21.15.134:0/214180030 shutdown_connections 2024-07-26T11:21:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.805+0000 7fa64a800640 1 --2- 172.21.15.134:0/214180030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6441028b0 0x7fa644106dd0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.805+0000 7fa64a800640 1 --2- 172.21.15.134:0/214180030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa644107420 0x7fa644107890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.805+0000 7fa64a800640 1 --2- 172.21.15.134:0/214180030 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa644101f60 0x7fa644102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.805+0000 7fa64a800640 1 -- 172.21.15.134:0/214180030 >> 172.21.15.134:0/214180030 conn(0x7fa6440fdc60 msgr2=0x7fa6441000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:57.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.806+0000 7fa64a800640 1 -- 172.21.15.134:0/214180030 shutdown_connections 2024-07-26T11:21:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.806+0000 7fa64a800640 1 -- 172.21.15.134:0/214180030 wait complete. 2024-07-26T11:21:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.808+0000 7fa64a800640 1 Processor -- start 2024-07-26T11:21:57.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.808+0000 7fa64a800640 1 -- start start 2024-07-26T11:21:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.810+0000 7fa64a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa644101f60 0x7fa644074980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.810+0000 7fa64a800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6441028b0 0x7fa64406dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.810+0000 7fa64a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa644107420 0x7fa64406e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.810+0000 7fa64a800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa6441198f0 con 0x7fa6441028b0 2024-07-26T11:21:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.810+0000 7fa64a800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa644119770 con 0x7fa644101f60 2024-07-26T11:21:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.810+0000 7fa64a800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa644119a70 con 0x7fa644107420 2024-07-26T11:21:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.811+0000 7fa649400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa644101f60 0x7fa644074980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.811+0000 7fa649e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa644107420 0x7fa64406e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.811+0000 7fa649400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa644101f60 0x7fa644074980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43322/0 (socket says 172.21.15.134:43322) 2024-07-26T11:21:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.811+0000 7fa649e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa644107420 0x7fa64406e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59772/0 (socket says 172.21.15.134:59772) 2024-07-26T11:21:57.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.811+0000 7fa649400640 1 -- 172.21.15.134:0/90752555 learned_addr learned my addr 172.21.15.134:0/90752555 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.813+0000 7fa649400640 1 -- 172.21.15.134:0/90752555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa644107420 msgr2=0x7fa64406e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.813+0000 7fa649400640 1 --2- 172.21.15.134:0/90752555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa644107420 0x7fa64406e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.815 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.813+0000 7fa649400640 1 -- 172.21.15.134:0/90752555 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6441028b0 msgr2=0x7fa64406dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.813+0000 7fa649400640 1 --2- 172.21.15.134:0/90752555 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6441028b0 0x7fa64406dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.813+0000 7fa649400640 1 -- 172.21.15.134:0/90752555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6440735e0 con 0x7fa644101f60 2024-07-26T11:21:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.813+0000 7fa649e00640 1 --2- 172.21.15.134:0/90752555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa644107420 0x7fa64406e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:21:57.816 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.814+0000 7fa649400640 1 --2- 172.21.15.134:0/90752555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa644101f60 0x7fa644074980 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa63c009d50 tx=0x7fa63c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.818+0000 7fa642000640 1 -- 172.21.15.134:0/90752555 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c003b70 con 0x7fa644101f60 2024-07-26T11:21:57.820 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.818+0000 7fa64a800640 1 -- 172.21.15.134:0/90752555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6440738d0 con 0x7fa644101f60 2024-07-26T11:21:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.818+0000 7fa64a800640 1 -- 172.21.15.134:0/90752555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa644073e70 con 0x7fa644101f60 2024-07-26T11:21:57.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.818+0000 7fa642000640 1 -- 172.21.15.134:0/90752555 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa63c003d10 con 0x7fa644101f60 2024-07-26T11:21:57.826 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.819+0000 7fa642000640 1 -- 172.21.15.134:0/90752555 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c003b70 con 0x7fa644101f60 2024-07-26T11:21:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.825+0000 7fa652e1c740 1 Processor -- start 2024-07-26T11:21:57.827 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.826+0000 7fa652e1c740 1 -- start start 2024-07-26T11:21:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.827+0000 7fa652e1c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106a31b0 0x55c410a46c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.827+0000 7fa652e1c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c41067a2f0 0x55c410a47160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.829 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.827+0000 7fa652e1c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106824d0 0x55c410a476a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.831 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.829+0000 7fa652e1c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c4109bb270 con 0x55c41067a2f0 2024-07-26T11:21:57.831 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.829+0000 7fa652e1c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c4109aa770 con 0x55c4106a31b0 2024-07-26T11:21:57.831 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.829+0000 7fa652e1c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c4109ae7b0 con 0x55c4106824d0 2024-07-26T11:21:57.831 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.829+0000 7fa649e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106a31b0 0x55c410a46c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:57.831 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.829+0000 7fa649e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106a31b0 0x55c410a46c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43324/0 (socket says 172.21.15.134:43324) 2024-07-26T11:21:57.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.829+0000 7fa649e00640 1 -- 172.21.15.134:0/2696293432 learned_addr learned my addr 172.21.15.134:0/2696293432 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:57.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.829+0000 7fa642000640 1 -- 172.21.15.134:0/90752555 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fa63c004020 con 0x7fa644101f60 2024-07-26T11:21:57.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.828+0000 7fa648a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106824d0 0x55c410a476a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:57.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.829+0000 7fa649400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c41067a2f0 0x55c410a47160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:57.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.830+0000 7fa649e00640 1 -- 172.21.15.134:0/2696293432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106824d0 msgr2=0x55c410a476a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.830+0000 7fa649e00640 1 --2- 172.21.15.134:0/2696293432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106824d0 0x55c410a476a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.831+0000 7fa649e00640 1 -- 172.21.15.134:0/2696293432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c41067a2f0 msgr2=0x55c410a47160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.832+0000 7fa649e00640 1 --2- 172.21.15.134:0/2696293432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c41067a2f0 0x55c410a47160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.834 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.832+0000 7fa649e00640 1 -- 172.21.15.134:0/2696293432 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c41070e790 con 0x55c4106a31b0 2024-07-26T11:21:57.836 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.833+0000 7fa648a00640 1 --2- 172.21.15.134:0/2696293432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106824d0 0x55c410a476a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:21:57.837 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.834+0000 7fa642000640 1 --2- 172.21.15.134:0/90752555 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa62c0b4070 0x7fa62c0b6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.837 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.834+0000 7fa642000640 1 -- 172.21.15.134:0/90752555 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa63c0e44a0 con 0x7fa644101f60 2024-07-26T11:21:57.837 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.835+0000 7fa649400640 1 --2- 172.21.15.134:0/2696293432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c41067a2f0 0x55c410a47160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:21:57.837 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.835+0000 7fa649e00640 1 --2- 172.21.15.134:0/2696293432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106a31b0 0x55c410a46c20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa638009d90 tx=0x7fa63800e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e556dd703e700fc1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:57.839 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.837+0000 7fa649400640 1 --2- 172.21.15.134:0/90752555 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa62c0b4070 0x7fa62c0b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.838+0000 7fa636600640 1 -- 172.21.15.134:0/2696293432 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638017070 con 0x55c4106a31b0 2024-07-26T11:21:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.838+0000 7fa636600640 1 -- 172.21.15.134:0/2696293432 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa638013920 con 0x55c4106a31b0 2024-07-26T11:21:57.840 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.838+0000 7fa636600640 1 -- 172.21.15.134:0/2696293432 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638012910 con 0x55c4106a31b0 2024-07-26T11:21:57.841 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.839+0000 7fa652e1c740 1 -- 172.21.15.134:0/2696293432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106a31b0 msgr2=0x55c410a46c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:57.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.839+0000 7fa649400640 1 --2- 172.21.15.134:0/90752555 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa62c0b4070 0x7fa62c0b6580 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x55c410a458a0 tx=0x7fa63c004930 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:57.844 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.839+0000 7fa652e1c740 1 --2- 172.21.15.134:0/2696293432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106a31b0 0x55c410a46c20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa638009d90 tx=0x7fa63800e5a0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.848+0000 7fa652e1c740 1 -- 172.21.15.134:0/2696293432 shutdown_connections 2024-07-26T11:21:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.848+0000 7fa652e1c740 1 --2- 172.21.15.134:0/2696293432 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106824d0 0x55c410a476a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.848+0000 7fa652e1c740 1 --2- 172.21.15.134:0/2696293432 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c41067a2f0 0x55c410a47160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.850 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.848+0000 7fa652e1c740 1 --2- 172.21.15.134:0/2696293432 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106a31b0 0x55c410a46c20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.848+0000 7fa652e1c740 1 -- 172.21.15.134:0/2696293432 >> 172.21.15.134:0/2696293432 conn(0x55c410697300 msgr2=0x55c410a44b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:57.860 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.858+0000 7fa652e1c740 1 -- 172.21.15.134:0/2696293432 shutdown_connections 2024-07-26T11:21:57.861 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.858+0000 7fa652e1c740 1 -- 172.21.15.134:0/2696293432 wait complete. 2024-07-26T11:21:57.861 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.859+0000 7fa652e1c740 1 -- start start 2024-07-26T11:21:57.861 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.859+0000 7fa652e1c740 1 Processor -- start 2024-07-26T11:21:57.862 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.860+0000 7fa652e1c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c41067a2f0 0x55c410a528a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.863 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.861+0000 7fa652e1c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106824d0 0x55c410a52e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.863 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.861+0000 7fa652e1c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4106a31b0 0x55c410a5e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.861+0000 7fa652e1c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c4108e4ab0 con 0x55c4106a31b0 2024-07-26T11:21:57.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.861+0000 7fa652e1c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c410947d50 con 0x55c4106824d0 2024-07-26T11:21:57.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.861+0000 7fa652e1c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c410a553a0 con 0x55c41067a2f0 2024-07-26T11:21:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.861+0000 7fa652e1c740 10 client.0 _ll_register_callbacks cb 0x55c4106baa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.861+0000 7fa648a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106824d0 0x55c410a52e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.861+0000 7fa648a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106824d0 0x55c410a52e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43330/0 (socket says 172.21.15.134:43330) 2024-07-26T11:21:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.861+0000 7fa648a00640 1 -- 172.21.15.134:0/1900694794 learned_addr learned my addr 172.21.15.134:0/1900694794 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.862+0000 7fa648a00640 1 -- 172.21.15.134:0/1900694794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c41067a2f0 msgr2=0x55c410a528a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:21:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.862+0000 7fa648a00640 1 --2- 172.21.15.134:0/1900694794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c41067a2f0 0x55c410a528a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.862+0000 7fa648a00640 1 -- 172.21.15.134:0/1900694794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4106a31b0 msgr2=0x55c410a5e720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:21:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.862+0000 7fa648a00640 1 --2- 172.21.15.134:0/1900694794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4106a31b0 0x55c410a5e720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.862+0000 7fa648a00640 1 -- 172.21.15.134:0/1900694794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c410787620 con 0x55c4106824d0 2024-07-26T11:21:57.865 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.862+0000 7fa648a00640 1 --2- 172.21.15.134:0/1900694794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106824d0 0x55c410a52e80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa64406c070 tx=0x7fa6300042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d6e9b3c7b6a2b03 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.863+0000 7fa62aa00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:21:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.863+0000 7fa62aa00640 1 -- 172.21.15.134:0/1900694794 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63003f070 con 0x55c4106824d0 2024-07-26T11:21:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.863+0000 7fa62aa00640 1 -- 172.21.15.134:0/1900694794 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa630005240 con 0x55c4106824d0 2024-07-26T11:21:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.864+0000 7fa652e1c740 1 -- 172.21.15.134:0/1900694794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa638013920 con 0x55c4106824d0 2024-07-26T11:21:57.866 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.864+0000 7fa62aa00640 1 -- 172.21.15.134:0/1900694794 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa630049050 con 0x55c4106824d0 2024-07-26T11:21:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.868+0000 7fa652e1c740 10 client.26366 fetch_fsmap learned FSMap version 32 2024-07-26T11:21:57.870 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.868+0000 7fa652e1c740 1 -- 172.21.15.134:0/1900694794 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c410a61410 con 0x55c4106824d0 2024-07-26T11:21:57.873 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.872+0000 7fa62aa00640 1 -- 172.21.15.134:0/1900694794 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa638013920 con 0x55c4106824d0 2024-07-26T11:21:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.872+0000 7fa62aa00640 1 -- 172.21.15.134:0/1900694794 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fa6300036c0 con 0x55c4106824d0 2024-07-26T11:21:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.872+0000 7fa652e1c740 10 client.26366 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:21:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.872+0000 7fa652e1c740 10 client.26366 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:21:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.872+0000 7fa652e1c740 20 client.26366 populate_metadata read hostname 'smithi134' 2024-07-26T11:21:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.872+0000 7fa652e1c740 1 --2- 172.21.15.134:0/1900694794 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c410661d80 0x55c410a64970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.872+0000 7fa652e1c740 4 client.26366 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:21:57.874 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.872+0000 7fa652e1c740 1 -- 172.21.15.134:0/1900694794 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4109bd1c0 con 0x55c410661d80 2024-07-26T11:21:57.876 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.874+0000 7fa649e00640 1 --2- 172.21.15.134:0/1900694794 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c410661d80 0x55c410a64970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:57.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.875+0000 7fa649e00640 1 --2- 172.21.15.134:0/1900694794 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c410661d80 0x55c410a64970 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfb5688e5ef48a3d server_cookie=b59113d5b91c192 in_seq=0 out_seq=0 2024-07-26T11:21:57.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.876+0000 7fa62aa00640 10 client.26366 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:57.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.879+0000 7fa62aa00640 1 -- 172.21.15.134:0/1900694794 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c4109bd1c0 con 0x55c410661d80 2024-07-26T11:21:57.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.879+0000 7fa62aa00640 10 client.26366 handle_command_reply: tid=0 2024-07-26T11:21:57.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.880+0000 7fa652e1c740 1 client.26366 shutdown 2024-07-26T11:21:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.886+0000 7fa652e1c740 1 -- 172.21.15.134:0/1900694794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106824d0 msgr2=0x55c410a52e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.886+0000 7fa652e1c740 1 --2- 172.21.15.134:0/1900694794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106824d0 0x55c410a52e80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa64406c070 tx=0x7fa6300042e0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.886+0000 7fa652e1c740 1 -- 172.21.15.134:0/1900694794 shutdown_connections 2024-07-26T11:21:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.886+0000 7fa652e1c740 1 --2- 172.21.15.134:0/1900694794 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c410661d80 0x55c410a64970 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.886+0000 7fa652e1c740 1 --2- 172.21.15.134:0/1900694794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4106a31b0 0x55c410a5e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.886+0000 7fa652e1c740 1 --2- 172.21.15.134:0/1900694794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106824d0 0x55c410a52e80 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.886+0000 7fa652e1c740 1 --2- 172.21.15.134:0/1900694794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c41067a2f0 0x55c410a528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.886+0000 7fa652e1c740 1 -- 172.21.15.134:0/1900694794 >> 172.21.15.134:0/1900694794 conn(0x55c410697300 msgr2=0x55c410a47c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.886+0000 7fa62aa00640 0 client.26366 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.887+0000 7fa652e1c740 1 -- 172.21.15.134:0/1900694794 shutdown_connections 2024-07-26T11:21:57.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.887+0000 7fa652e1c740 1 -- 172.21.15.134:0/1900694794 wait complete. 2024-07-26T11:21:57.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.888+0000 7fa652e1c740 20 client.26366 trim_cache size 0 max 16384 2024-07-26T11:21:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.930+0000 7fa652e1c740 1 Processor -- start 2024-07-26T11:21:57.933 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.931+0000 7fa652e1c740 1 -- start start 2024-07-26T11:21:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.937+0000 7fa652e1c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c41067a2f0 0x55c410a52270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.939 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.938+0000 7fa652e1c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106824d0 0x55c410a5e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.938+0000 7fa652e1c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106a31b0 0x55c410a4bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.938+0000 7fa652e1c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa624010060 con 0x55c41067a2f0 2024-07-26T11:21:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.938+0000 7fa652e1c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa624011ba0 con 0x55c4106a31b0 2024-07-26T11:21:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.938+0000 7fa652e1c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa624010730 con 0x55c4106824d0 2024-07-26T11:21:57.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.938+0000 7fa648a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106824d0 0x55c410a5e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:57.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.938+0000 7fa648a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106824d0 0x55c410a5e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59780/0 (socket says 172.21.15.134:59780) 2024-07-26T11:21:57.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.938+0000 7fa648a00640 1 -- 172.21.15.134:0/3562115073 learned_addr learned my addr 172.21.15.134:0/3562115073 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:57.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.939+0000 7fa649e00640 1 --2- 172.21.15.134:0/3562115073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106a31b0 0x55c410a4bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:57.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.939+0000 7fa648a00640 1 -- 172.21.15.134:0/3562115073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106a31b0 msgr2=0x55c410a4bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:57.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.939+0000 7fa648a00640 1 --2- 172.21.15.134:0/3562115073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106a31b0 0x55c410a4bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.939+0000 7fa648a00640 1 -- 172.21.15.134:0/3562115073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c41067a2f0 msgr2=0x55c410a52270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:57.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.939+0000 7fa648a00640 1 --2- 172.21.15.134:0/3562115073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c41067a2f0 0x55c410a52270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.939+0000 7fa648a00640 1 -- 172.21.15.134:0/3562115073 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c410a4e1c0 con 0x55c4106824d0 2024-07-26T11:21:57.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.940+0000 7fa648a00640 1 --2- 172.21.15.134:0/3562115073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106824d0 0x55c410a5e510 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa630009900 tx=0x7fa630003f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b907d94d007a023e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:57.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.941+0000 7fa635c00640 1 -- 172.21.15.134:0/3562115073 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63003f040 con 0x55c4106824d0 2024-07-26T11:21:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.941+0000 7fa635c00640 1 -- 172.21.15.134:0/3562115073 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa630052020 con 0x55c4106824d0 2024-07-26T11:21:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.941+0000 7fa635c00640 1 -- 172.21.15.134:0/3562115073 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa630035030 con 0x55c4106824d0 2024-07-26T11:21:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.942+0000 7fa652e1c740 1 -- 172.21.15.134:0/3562115073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106824d0 msgr2=0x55c410a5e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.942+0000 7fa652e1c740 1 --2- 172.21.15.134:0/3562115073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106824d0 0x55c410a5e510 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa630009900 tx=0x7fa630003f60 comp rx=0 tx=0).stop 2024-07-26T11:21:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.943+0000 7fa652e1c740 1 -- 172.21.15.134:0/3562115073 shutdown_connections 2024-07-26T11:21:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.943+0000 7fa652e1c740 1 --2- 172.21.15.134:0/3562115073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106824d0 0x55c410a5e510 secure :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa630009900 tx=0x7fa630003f60 comp rx=0 tx=0).stop 2024-07-26T11:21:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.943+0000 7fa652e1c740 1 --2- 172.21.15.134:0/3562115073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4106a31b0 0x55c410a4bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.943+0000 7fa652e1c740 1 --2- 172.21.15.134:0/3562115073 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c41067a2f0 0x55c410a52270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.943+0000 7fa652e1c740 1 -- 172.21.15.134:0/3562115073 >> 172.21.15.134:0/3562115073 conn(0x55c410697300 msgr2=0x55c410a51b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:57.955 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.953+0000 7fa652e1c740 1 -- 172.21.15.134:0/3562115073 shutdown_connections 2024-07-26T11:21:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.954+0000 7fa652e1c740 1 -- 172.21.15.134:0/3562115073 wait complete. 2024-07-26T11:21:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.955+0000 7fa652e1c740 1 -- start start 2024-07-26T11:21:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.960+0000 7fa652e1c740 1 Processor -- start 2024-07-26T11:21:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.962+0000 7fa652e1c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c41067a2f0 0x55c410a55a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.962+0000 7fa652e1c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106a31b0 0x55c410a56060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.962+0000 7fa652e1c740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4106371c0 0x55c410acb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.962+0000 7fa652e1c740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa6240114d0 con 0x55c4106371c0 2024-07-26T11:21:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.962+0000 7fa652e1c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa624010e00 con 0x55c41067a2f0 2024-07-26T11:21:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.962+0000 7fa652e1c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa62400e040 con 0x55c4106a31b0 2024-07-26T11:21:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.962+0000 7fa652e1c740 10 client.0 _ll_register_callbacks cb 0x7fa63c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:21:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.963+0000 7fa649e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c41067a2f0 0x55c410a55a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.963+0000 7fa649e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c41067a2f0 0x55c410a55a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43354/0 (socket says 172.21.15.134:43354) 2024-07-26T11:21:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.963+0000 7fa649e00640 1 -- 172.21.15.134:0/2926344697 learned_addr learned my addr 172.21.15.134:0/2926344697 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:21:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.963+0000 7fa648a00640 1 --2- 172.21.15.134:0/2926344697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106a31b0 0x55c410a56060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.964+0000 7fa648a00640 1 -- 172.21.15.134:0/2926344697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c41067a2f0 msgr2=0x55c410a55a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.964+0000 7fa648a00640 1 --2- 172.21.15.134:0/2926344697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c41067a2f0 0x55c410a55a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.964+0000 7fa648a00640 1 -- 172.21.15.134:0/2926344697 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4106371c0 msgr2=0x55c410acb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:21:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.964+0000 7fa648a00640 1 --2- 172.21.15.134:0/2926344697 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4106371c0 0x55c410acb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.964+0000 7fa648a00640 1 -- 172.21.15.134:0/2926344697 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c410adc6d0 con 0x55c4106a31b0 2024-07-26T11:21:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.966+0000 7fa648a00640 1 --2- 172.21.15.134:0/2926344697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106a31b0 0x55c410a56060 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x55c410a45560 tx=0x7fa63002ad70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=599a616fe67ddce2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:21:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.966+0000 7fa62aa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:21:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.966+0000 7fa62aa00640 1 -- 172.21.15.134:0/2926344697 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa630035070 con 0x55c4106a31b0 2024-07-26T11:21:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.966+0000 7fa62aa00640 1 -- 172.21.15.134:0/2926344697 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fa6300053f0 con 0x55c4106a31b0 2024-07-26T11:21:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.966+0000 7fa652e1c740 1 -- 172.21.15.134:0/2926344697 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa630052020 con 0x55c4106a31b0 2024-07-26T11:21:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.967+0000 7fa62aa00640 1 -- 172.21.15.134:0/2926344697 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa63003f040 con 0x55c4106a31b0 2024-07-26T11:21:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.967+0000 7fa652e1c740 10 client.26377 fetch_fsmap learned FSMap version 32 2024-07-26T11:21:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.967+0000 7fa652e1c740 1 -- 172.21.15.134:0/2926344697 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c410add890 con 0x55c4106a31b0 2024-07-26T11:21:57.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.967+0000 7fa62aa00640 1 -- 172.21.15.134:0/2926344697 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6300055f0 con 0x55c4106a31b0 2024-07-26T11:21:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.972+0000 7fa62aa00640 1 -- 172.21.15.134:0/2926344697 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fa6300314a0 con 0x55c4106a31b0 2024-07-26T11:21:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.972+0000 7fa652e1c740 10 client.26377 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:21:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.972+0000 7fa652e1c740 10 client.26377 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:21:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.972+0000 7fa652e1c740 20 client.26377 populate_metadata read hostname 'smithi134' 2024-07-26T11:21:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.972+0000 7fa652e1c740 1 --2- 172.21.15.134:0/2926344697 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c410addef0 0x55c410ae0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:21:57.975 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.972+0000 7fa652e1c740 4 client.26377 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:21:57.976 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.972+0000 7fa652e1c740 1 -- 172.21.15.134:0/2926344697 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fa630052020 con 0x55c410addef0 2024-07-26T11:21:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.980+0000 7fa649e00640 1 --2- 172.21.15.134:0/2926344697 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c410addef0 0x55c410ae0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:21:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.987+0000 7fa649e00640 1 --2- 172.21.15.134:0/2926344697 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c410addef0 0x55c410ae0310 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b71ca2c695b3ef server_cookie=d1fa5b1f4f02dd7b in_seq=0 out_seq=0 2024-07-26T11:21:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:57.987+0000 7fa62aa00640 10 client.26377 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.207+0000 7fa62aa00640 1 -- 172.21.15.134:0/2926344697 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fa630052020 con 0x55c410addef0 2024-07-26T11:21:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.207+0000 7fa62aa00640 10 client.26377 handle_command_reply: tid=0 2024-07-26T11:21:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.207+0000 7fa652e1c740 1 client.26377 shutdown 2024-07-26T11:21:58.211 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.209+0000 7fa652e1c740 1 -- 172.21.15.134:0/2926344697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106a31b0 msgr2=0x55c410a56060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:58.212 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.209+0000 7fa652e1c740 1 --2- 172.21.15.134:0/2926344697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106a31b0 0x55c410a56060 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x55c410a45560 tx=0x7fa63002ad70 comp rx=0 tx=0).stop 2024-07-26T11:21:58.212 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.209+0000 7fa652e1c740 1 -- 172.21.15.134:0/2926344697 shutdown_connections 2024-07-26T11:21:58.212 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.209+0000 7fa652e1c740 1 --2- 172.21.15.134:0/2926344697 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c410addef0 0x55c410ae0310 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:58.212 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.209+0000 7fa62aa00640 0 client.26377 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:21:58.212 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.209+0000 7fa652e1c740 1 --2- 172.21.15.134:0/2926344697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c4106a31b0 0x55c410a56060 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:58.212 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.209+0000 7fa652e1c740 1 --2- 172.21.15.134:0/2926344697 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c4106371c0 0x55c410acb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:58.212 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.209+0000 7fa652e1c740 1 --2- 172.21.15.134:0/2926344697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c41067a2f0 0x55c410a55a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:58.212 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.209+0000 7fa652e1c740 1 -- 172.21.15.134:0/2926344697 >> 172.21.15.134:0/2926344697 conn(0x55c410697300 msgr2=0x55c410a4bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:58.216 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.214+0000 7fa652e1c740 1 -- 172.21.15.134:0/2926344697 shutdown_connections 2024-07-26T11:21:58.217 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.216+0000 7fa652e1c740 1 -- 172.21.15.134:0/2926344697 wait complete. 2024-07-26T11:21:58.218 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.216+0000 7fa652e1c740 20 client.26377 trim_cache size 0 max 16384 2024-07-26T11:21:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:21:58.219 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:21:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:21:58.220 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:21:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.221+0000 7fa64a800640 1 -- 172.21.15.134:0/90752555 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa62c0b4070 msgr2=0x7fa62c0b6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.221+0000 7fa64a800640 1 --2- 172.21.15.134:0/90752555 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa62c0b4070 0x7fa62c0b6580 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x55c410a458a0 tx=0x7fa63c004930 comp rx=0 tx=0).stop 2024-07-26T11:21:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.221+0000 7fa64a800640 1 -- 172.21.15.134:0/90752555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa644101f60 msgr2=0x7fa644074980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:21:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.221+0000 7fa64a800640 1 --2- 172.21.15.134:0/90752555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa644101f60 0x7fa644074980 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa63c009d50 tx=0x7fa63c00c6a0 comp rx=0 tx=0).stop 2024-07-26T11:21:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.221+0000 7fa64a800640 1 -- 172.21.15.134:0/90752555 shutdown_connections 2024-07-26T11:21:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.221+0000 7fa64a800640 1 --2- 172.21.15.134:0/90752555 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa62c0b4070 0x7fa62c0b6580 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.221+0000 7fa64a800640 1 --2- 172.21.15.134:0/90752555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa644107420 0x7fa64406e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.221+0000 7fa64a800640 1 --2- 172.21.15.134:0/90752555 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa6441028b0 0x7fa64406dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.221+0000 7fa64a800640 1 --2- 172.21.15.134:0/90752555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa644101f60 0x7fa644074980 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:21:58.223 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.221+0000 7fa64a800640 1 -- 172.21.15.134:0/90752555 >> 172.21.15.134:0/90752555 conn(0x7fa6440fdc60 msgr2=0x7fa644104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:21:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.225+0000 7fa64a800640 1 -- 172.21.15.134:0/90752555 shutdown_connections 2024-07-26T11:21:58.227 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:21:58.225+0000 7fa64a800640 1 -- 172.21.15.134:0/90752555 wait complete. 2024-07-26T11:21:58.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:21:58 smithi134 ceph-mon[31966]: pgmap v659: 129 pgs: 129 active+clean; 43 GiB data, 89 GiB used, 983 GiB / 1.0 TiB avail; 30 KiB/s rd, 231 MiB/s wr, 1.27k op/s 2024-07-26T11:21:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:21:58 smithi193 ceph-mon[37473]: pgmap v659: 129 pgs: 129 active+clean; 43 GiB data, 89 GiB used, 983 GiB / 1.0 TiB avail; 30 KiB/s rd, 231 MiB/s wr, 1.27k op/s 2024-07-26T11:21:58.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-07-26T11:21:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:21:58 smithi189 ceph-mon[36790]: pgmap v659: 129 pgs: 129 active+clean; 43 GiB data, 89 GiB used, 983 GiB / 1.0 TiB avail; 30 KiB/s rd, 231 MiB/s wr, 1.27k op/s 2024-07-26T11:21:59.793 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:22:00.363 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:00 smithi134 ceph-mon[31966]: pgmap v660: 129 pgs: 129 active+clean; 43 GiB data, 90 GiB used, 982 GiB / 1.0 TiB avail; 18 KiB/s rd, 233 MiB/s wr, 1.27k op/s 2024-07-26T11:22:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:00 smithi193 ceph-mon[37473]: pgmap v660: 129 pgs: 129 active+clean; 43 GiB data, 90 GiB used, 982 GiB / 1.0 TiB avail; 18 KiB/s rd, 233 MiB/s wr, 1.27k op/s 2024-07-26T11:22:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:00 smithi189 ceph-mon[36790]: pgmap v660: 129 pgs: 129 active+clean; 43 GiB data, 90 GiB used, 982 GiB / 1.0 TiB avail; 18 KiB/s rd, 233 MiB/s wr, 1.27k op/s 2024-07-26T11:22:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:02 smithi193 ceph-mon[37473]: pgmap v661: 129 pgs: 129 active+clean; 44 GiB data, 91 GiB used, 982 GiB / 1.0 TiB avail; 16 KiB/s rd, 263 MiB/s wr, 1.32k op/s 2024-07-26T11:22:02.675 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:02 smithi134 ceph-mon[31966]: pgmap v661: 129 pgs: 129 active+clean; 44 GiB data, 91 GiB used, 982 GiB / 1.0 TiB avail; 16 KiB/s rd, 263 MiB/s wr, 1.32k op/s 2024-07-26T11:22:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:02 smithi189 ceph-mon[36790]: pgmap v661: 129 pgs: 129 active+clean; 44 GiB data, 91 GiB used, 982 GiB / 1.0 TiB avail; 16 KiB/s rd, 263 MiB/s wr, 1.32k op/s 2024-07-26T11:22:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:04 smithi134 ceph-mon[31966]: pgmap v662: 129 pgs: 129 active+clean; 44 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 16 KiB/s rd, 239 MiB/s wr, 1.08k op/s 2024-07-26T11:22:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:04 smithi189 ceph-mon[36790]: pgmap v662: 129 pgs: 129 active+clean; 44 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 16 KiB/s rd, 239 MiB/s wr, 1.08k op/s 2024-07-26T11:22:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:04 smithi193 ceph-mon[37473]: pgmap v662: 129 pgs: 129 active+clean; 44 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 16 KiB/s rd, 239 MiB/s wr, 1.08k op/s 2024-07-26T11:22:06.677 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:06 smithi134 ceph-mon[31966]: pgmap v663: 129 pgs: 129 active+clean; 44 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 39 KiB/s rd, 262 MiB/s wr, 1.09k op/s 2024-07-26T11:22:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:06 smithi189 ceph-mon[36790]: pgmap v663: 129 pgs: 129 active+clean; 44 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 39 KiB/s rd, 262 MiB/s wr, 1.09k op/s 2024-07-26T11:22:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:06 smithi193 ceph-mon[37473]: pgmap v663: 129 pgs: 129 active+clean; 44 GiB data, 92 GiB used, 981 GiB / 1.0 TiB avail; 39 KiB/s rd, 262 MiB/s wr, 1.09k op/s 2024-07-26T11:22:08.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:08 smithi134 ceph-mon[31966]: pgmap v664: 129 pgs: 129 active+clean; 44 GiB data, 92 GiB used, 980 GiB / 1.0 TiB avail; 38 KiB/s rd, 259 MiB/s wr, 1.07k op/s 2024-07-26T11:22:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:08 smithi189 ceph-mon[36790]: pgmap v664: 129 pgs: 129 active+clean; 44 GiB data, 92 GiB used, 980 GiB / 1.0 TiB avail; 38 KiB/s rd, 259 MiB/s wr, 1.07k op/s 2024-07-26T11:22:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:08 smithi193 ceph-mon[37473]: pgmap v664: 129 pgs: 129 active+clean; 44 GiB data, 92 GiB used, 980 GiB / 1.0 TiB avail; 38 KiB/s rd, 259 MiB/s wr, 1.07k op/s 2024-07-26T11:22:09.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.304+0000 7f6368e00640 1 -- 172.21.15.134:0/1738427864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f636410fe50 msgr2=0x7f6364116680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:09.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.304+0000 7f6368e00640 1 --2- 172.21.15.134:0/1738427864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f636410fe50 0x7f6364116680 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6354009860 tx=0x7f635402a340 comp rx=0 tx=0).stop 2024-07-26T11:22:09.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.312+0000 7f6368e00640 1 -- 172.21.15.134:0/1738427864 shutdown_connections 2024-07-26T11:22:09.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.312+0000 7f6368e00640 1 --2- 172.21.15.134:0/1738427864 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f636410fe50 0x7f6364116680 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.313+0000 7f6368e00640 1 --2- 172.21.15.134:0/1738427864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636410f500 0x7f636410f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.314+0000 7f6368e00640 1 --2- 172.21.15.134:0/1738427864 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f6364109bc0 0x7f6364109fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.314+0000 7f6368e00640 1 -- 172.21.15.134:0/1738427864 >> 172.21.15.134:0/1738427864 conn(0x7f6364077b20 msgr2=0x7f6364107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:09.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.318+0000 7f6368e00640 1 -- 172.21.15.134:0/1738427864 shutdown_connections 2024-07-26T11:22:09.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.318+0000 7f6368e00640 1 -- 172.21.15.134:0/1738427864 wait complete. 2024-07-26T11:22:09.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.329+0000 7f6368e00640 1 Processor -- start 2024-07-26T11:22:09.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.330+0000 7f6368e00640 1 -- start start 2024-07-26T11:22:09.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.331+0000 7f6368e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6364109bc0 0x7f63640745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.332+0000 7f6368e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636410f500 0x7f636406d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.334+0000 7f6361e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6364109bc0 0x7f63640745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.334+0000 7f6361e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6364109bc0 0x7f63640745f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:56004/0 (socket says 172.21.15.134:56004) 2024-07-26T11:22:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.334+0000 7f6361e00640 1 -- 172.21.15.134:0/2053989889 learned_addr learned my addr 172.21.15.134:0/2053989889 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.334+0000 7f6361400640 1 --2- 172.21.15.134:0/2053989889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636410f500 0x7f636406d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.335+0000 7f6368e00640 1 --2- 172.21.15.134:0/2053989889 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f636410fe50 0x7f636406de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.335+0000 7f6368e00640 1 -- 172.21.15.134:0/2053989889 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6364119b10 con 0x7f636410fe50 2024-07-26T11:22:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.335+0000 7f6368e00640 1 -- 172.21.15.134:0/2053989889 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6364119990 con 0x7f636410f500 2024-07-26T11:22:09.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.336+0000 7f6368e00640 1 -- 172.21.15.134:0/2053989889 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6364119c90 con 0x7f6364109bc0 2024-07-26T11:22:09.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.338+0000 7f6361400640 1 -- 172.21.15.134:0/2053989889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6364109bc0 msgr2=0x7f63640745f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:09.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.338+0000 7f6361400640 1 --2- 172.21.15.134:0/2053989889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6364109bc0 0x7f63640745f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.338+0000 7f6361400640 1 -- 172.21.15.134:0/2053989889 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f636410fe50 msgr2=0x7f636406de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:09.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.338+0000 7f6361400640 1 --2- 172.21.15.134:0/2053989889 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f636410fe50 0x7f636406de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.338+0000 7f6361400640 1 -- 172.21.15.134:0/2053989889 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f636406e3c0 con 0x7f636410f500 2024-07-26T11:22:09.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.338+0000 7f6361e00640 1 --2- 172.21.15.134:0/2053989889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6364109bc0 0x7f63640745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:22:09.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.340+0000 7f6361400640 1 --2- 172.21.15.134:0/2053989889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636410f500 0x7f636406d8c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f636406b560 tx=0x7f634c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:09.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.343+0000 7f635aa00640 1 -- 172.21.15.134:0/2053989889 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634c00cc40 con 0x7f636410f500 2024-07-26T11:22:09.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.343+0000 7f635aa00640 1 -- 172.21.15.134:0/2053989889 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f634c00cde0 con 0x7f636410f500 2024-07-26T11:22:09.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.343+0000 7f6368e00640 1 -- 172.21.15.134:0/2053989889 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f636406e6b0 con 0x7f636410f500 2024-07-26T11:22:09.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.343+0000 7f6368e00640 1 -- 172.21.15.134:0/2053989889 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6364073580 con 0x7f636410f500 2024-07-26T11:22:09.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.344+0000 7f635aa00640 1 -- 172.21.15.134:0/2053989889 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634c018780 con 0x7f636410f500 2024-07-26T11:22:09.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.353+0000 7f636fbf6740 1 Processor -- start 2024-07-26T11:22:09.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.354+0000 7f635aa00640 1 -- 172.21.15.134:0/2053989889 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f634c018920 con 0x7f636410f500 2024-07-26T11:22:09.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.357+0000 7f635aa00640 1 --2- 172.21.15.134:0/2053989889 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63480b3f10 0x7f63480b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.357+0000 7f636fbf6740 1 -- start start 2024-07-26T11:22:09.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.358+0000 7f636fbf6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4df070 0x563b2b883e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.358+0000 7f636fbf6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4b61b0 0x563b2b884370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.358+0000 7f636fbf6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4be390 0x563b2b8848b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.358+0000 7f636fbf6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563b2b7fabd0 con 0x563b2b4be390 2024-07-26T11:22:09.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.358+0000 7f636fbf6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563b2b7ea0d0 con 0x563b2b4df070 2024-07-26T11:22:09.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.358+0000 7f636fbf6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563b2b7ee110 con 0x563b2b4b61b0 2024-07-26T11:22:09.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.358+0000 7f6361e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4df070 0x563b2b883e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:09.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.358+0000 7f6361e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4df070 0x563b2b883e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56326/0 (socket says 172.21.15.134:56326) 2024-07-26T11:22:09.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.358+0000 7f6361e00640 1 -- 172.21.15.134:0/3177669777 learned_addr learned my addr 172.21.15.134:0/3177669777 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:09.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.359+0000 7f6361400640 1 --2- 172.21.15.134:0/3177669777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4b61b0 0x563b2b884370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:09.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.359+0000 7f635aa00640 1 -- 172.21.15.134:0/2053989889 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f634c0e4540 con 0x7f636410f500 2024-07-26T11:22:09.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.360+0000 7f6361400640 1 -- 172.21.15.134:0/3177669777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4df070 msgr2=0x563b2b883e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:09.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.360+0000 7f6361400640 1 --2- 172.21.15.134:0/3177669777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4df070 0x563b2b883e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.360+0000 7f6361400640 1 -- 172.21.15.134:0/3177669777 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4be390 msgr2=0x563b2b8848b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:09.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.360+0000 7f6361400640 1 --2- 172.21.15.134:0/3177669777 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4be390 0x563b2b8848b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.360+0000 7f6361400640 1 -- 172.21.15.134:0/3177669777 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b2b54a650 con 0x563b2b4b61b0 2024-07-26T11:22:09.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.362+0000 7f6361400640 1 --2- 172.21.15.134:0/3177669777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4b61b0 0x563b2b884370 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f634c0e18b0 tx=0x7f634c04f520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac37747e28a6e443 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:09.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.363+0000 7f6353400640 1 -- 172.21.15.134:0/3177669777 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634c026040 con 0x563b2b4b61b0 2024-07-26T11:22:09.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.363+0000 7f6362800640 1 --2- 172.21.15.134:0/2053989889 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63480b3f10 0x7f63480b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:09.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.364+0000 7f6353400640 1 -- 172.21.15.134:0/3177669777 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f634c0aad70 con 0x563b2b4b61b0 2024-07-26T11:22:09.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.366+0000 7f636fbf6740 1 -- 172.21.15.134:0/3177669777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4b61b0 msgr2=0x563b2b884370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:09.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.366+0000 7f636fbf6740 1 --2- 172.21.15.134:0/3177669777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4b61b0 0x563b2b884370 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f634c0e18b0 tx=0x7f634c04f520 comp rx=0 tx=0).stop 2024-07-26T11:22:09.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.366+0000 7f636fbf6740 1 -- 172.21.15.134:0/3177669777 shutdown_connections 2024-07-26T11:22:09.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.367+0000 7f636fbf6740 1 --2- 172.21.15.134:0/3177669777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4b61b0 0x563b2b884370 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.367+0000 7f636fbf6740 1 --2- 172.21.15.134:0/3177669777 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4be390 0x563b2b8848b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.367+0000 7f636fbf6740 1 --2- 172.21.15.134:0/3177669777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4df070 0x563b2b883e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.367+0000 7f636fbf6740 1 -- 172.21.15.134:0/3177669777 >> 172.21.15.134:0/3177669777 conn(0x563b2b4d3460 msgr2=0x563b2b881d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:09.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.369+0000 7f636fbf6740 1 -- 172.21.15.134:0/3177669777 shutdown_connections 2024-07-26T11:22:09.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.369+0000 7f636fbf6740 1 -- 172.21.15.134:0/3177669777 wait complete. 2024-07-26T11:22:09.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.370+0000 7f636fbf6740 1 -- start start 2024-07-26T11:22:09.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.370+0000 7f636fbf6740 1 Processor -- start 2024-07-26T11:22:09.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.375+0000 7f6362800640 1 --2- 172.21.15.134:0/2053989889 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63480b3f10 0x7f63480b6420 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f635400b230 tx=0x7f6354005f10 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:09.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.379+0000 7f636fbf6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b88f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.379+0000 7f636fbf6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4be390 0x563b2b88f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.380+0000 7f6361e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b88f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:09.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.380+0000 7f6361e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b88f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56330/0 (socket says 172.21.15.134:56330) 2024-07-26T11:22:09.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.380+0000 7f6361e00640 1 -- 172.21.15.134:0/4239568376 learned_addr learned my addr 172.21.15.134:0/4239568376 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:09.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.380+0000 7f636fbf6740 1 --2- 172.21.15.134:0/4239568376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4df070 0x563b2b88fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.380+0000 7f6361400640 1 --2- 172.21.15.134:0/4239568376 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4be390 0x563b2b88f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:09.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.380+0000 7f636fbf6740 1 -- 172.21.15.134:0/4239568376 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563b2b64d700 con 0x563b2b4be390 2024-07-26T11:22:09.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.380+0000 7f636fbf6740 1 -- 172.21.15.134:0/4239568376 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563b2b7866a0 con 0x563b2b4b61b0 2024-07-26T11:22:09.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.380+0000 7f636fbf6740 1 -- 172.21.15.134:0/4239568376 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563b2b892610 con 0x563b2b4df070 2024-07-26T11:22:09.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.380+0000 7f6362800640 1 --2- 172.21.15.134:0/4239568376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4df070 0x563b2b88fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:09.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.381+0000 7f636fbf6740 10 client.0 _ll_register_callbacks cb 0x563b2b796150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:09.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.382+0000 7f6361e00640 1 -- 172.21.15.134:0/4239568376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4df070 msgr2=0x563b2b88fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:09.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.382+0000 7f6361e00640 1 --2- 172.21.15.134:0/4239568376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4df070 0x563b2b88fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.382+0000 7f6361e00640 1 -- 172.21.15.134:0/4239568376 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4be390 msgr2=0x563b2b88f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:09.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.382+0000 7f6361e00640 1 --2- 172.21.15.134:0/4239568376 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4be390 0x563b2b88f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.382+0000 7f6361e00640 1 -- 172.21.15.134:0/4239568376 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b2b79d840 con 0x563b2b4b61b0 2024-07-26T11:22:09.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.382+0000 7f6362800640 1 --2- 172.21.15.134:0/4239568376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4df070 0x563b2b88fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:22:09.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.383+0000 7f6361e00640 1 --2- 172.21.15.134:0/4239568376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b88f380 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f635c010ec0 tx=0x7f635c00e7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf9aed384a2fe146 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:09.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.383+0000 7f6347e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:22:09.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.383+0000 7f6347e00640 1 -- 172.21.15.134:0/4239568376 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c003970 con 0x563b2b4b61b0 2024-07-26T11:22:09.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.383+0000 7f6347e00640 1 -- 172.21.15.134:0/4239568376 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f635c003b10 con 0x563b2b4b61b0 2024-07-26T11:22:09.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.383+0000 7f636fbf6740 1 -- 172.21.15.134:0/4239568376 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f634c0aad70 con 0x563b2b4b61b0 2024-07-26T11:22:09.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.385+0000 7f6347e00640 1 -- 172.21.15.134:0/4239568376 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f635c003d10 con 0x563b2b4b61b0 2024-07-26T11:22:09.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.385+0000 7f636fbf6740 10 client.26384 fetch_fsmap learned FSMap version 32 2024-07-26T11:22:09.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.385+0000 7f636fbf6740 1 -- 172.21.15.134:0/4239568376 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563b2b89e290 con 0x563b2b4b61b0 2024-07-26T11:22:09.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.386+0000 7f6347e00640 1 -- 172.21.15.134:0/4239568376 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f634c0aad70 con 0x563b2b4b61b0 2024-07-26T11:22:09.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.386+0000 7f6347e00640 1 -- 172.21.15.134:0/4239568376 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f635c00b040 con 0x563b2b4b61b0 2024-07-26T11:22:09.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.386+0000 7f636fbf6740 10 client.26384 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:22:09.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.386+0000 7f636fbf6740 10 client.26384 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-07-26T11:22:09.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.386+0000 7f636fbf6740 20 client.26384 populate_metadata read hostname 'smithi134' 2024-07-26T11:22:09.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.386+0000 7f636fbf6740 1 --2- 172.21.15.134:0/4239568376 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x563b2b49d9f0 0x563b2b8a1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.386+0000 7f636fbf6740 4 client.26384 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:22:09.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.386+0000 7f636fbf6740 1 -- 172.21.15.134:0/4239568376 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b2b7fcb20 con 0x563b2b49d9f0 2024-07-26T11:22:09.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.386+0000 7f6361400640 1 --2- 172.21.15.134:0/4239568376 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x563b2b49d9f0 0x563b2b8a1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:09.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.387+0000 7f6347e00640 10 client.26384 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:22:09.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.387+0000 7f6361400640 1 --2- 172.21.15.134:0/4239568376 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x563b2b49d9f0 0x563b2b8a1800 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=857d16b598c9423 server_cookie=8179e0684b12666a in_seq=0 out_seq=0 2024-07-26T11:22:09.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.389+0000 7f6347e00640 1 -- 172.21.15.134:0/4239568376 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563b2b7fcb20 con 0x563b2b49d9f0 2024-07-26T11:22:09.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.389+0000 7f6347e00640 10 client.26384 handle_command_reply: tid=0 2024-07-26T11:22:09.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.389+0000 7f636fbf6740 1 client.26384 shutdown 2024-07-26T11:22:09.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.389+0000 7f636fbf6740 1 -- 172.21.15.134:0/4239568376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 msgr2=0x563b2b88f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:09.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.389+0000 7f636fbf6740 1 --2- 172.21.15.134:0/4239568376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b88f380 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f635c010ec0 tx=0x7f635c00e7c0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.391+0000 7f636fbf6740 1 -- 172.21.15.134:0/4239568376 shutdown_connections 2024-07-26T11:22:09.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.391+0000 7f636fbf6740 1 --2- 172.21.15.134:0/4239568376 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x563b2b49d9f0 0x563b2b8a1800 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.391+0000 7f636fbf6740 1 --2- 172.21.15.134:0/4239568376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4df070 0x563b2b88fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.391+0000 7f636fbf6740 1 --2- 172.21.15.134:0/4239568376 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4be390 0x563b2b88f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.391+0000 7f636fbf6740 1 --2- 172.21.15.134:0/4239568376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b88f380 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.391+0000 7f636fbf6740 1 -- 172.21.15.134:0/4239568376 >> 172.21.15.134:0/4239568376 conn(0x563b2b4d3460 msgr2=0x563b2b886fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:09.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.391+0000 7f6347e00640 0 client.26384 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:22:09.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.392+0000 7f636fbf6740 1 -- 172.21.15.134:0/4239568376 shutdown_connections 2024-07-26T11:22:09.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.392+0000 7f636fbf6740 1 -- 172.21.15.134:0/4239568376 wait complete. 2024-07-26T11:22:09.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.392+0000 7f636fbf6740 20 client.26384 trim_cache size 0 max 16384 2024-07-26T11:22:09.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.425+0000 7f636fbf6740 1 Processor -- start 2024-07-26T11:22:09.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.426+0000 7f636fbf6740 1 -- start start 2024-07-26T11:22:09.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.427+0000 7f636fbf6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b884e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.427+0000 7f636fbf6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4be390 0x563b2b89b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.427+0000 7f636fbf6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4df070 0x563b2b8a6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.427+0000 7f636fbf6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f634000ff90 con 0x563b2b4df070 2024-07-26T11:22:09.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.427+0000 7f636fbf6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6340011ad0 con 0x563b2b4b61b0 2024-07-26T11:22:09.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.427+0000 7f636fbf6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f6340010660 con 0x563b2b4be390 2024-07-26T11:22:09.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.428+0000 7f6361e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b884e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:09.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.428+0000 7f6361400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4be390 0x563b2b89b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:09.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.430+0000 7f6361400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4be390 0x563b2b89b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:56036/0 (socket says 172.21.15.134:56036) 2024-07-26T11:22:09.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.430+0000 7f6361400640 1 -- 172.21.15.134:0/866928914 learned_addr learned my addr 172.21.15.134:0/866928914 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:09.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.431+0000 7f6361400640 1 -- 172.21.15.134:0/866928914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 msgr2=0x563b2b884e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:09.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.432+0000 7f6361400640 1 --2- 172.21.15.134:0/866928914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b884e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.432+0000 7f6361400640 1 -- 172.21.15.134:0/866928914 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4df070 msgr2=0x563b2b8a6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:09.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.432+0000 7f6361400640 1 --2- 172.21.15.134:0/866928914 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4df070 0x563b2b8a6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.432+0000 7f6361400640 1 -- 172.21.15.134:0/866928914 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b2b886120 con 0x563b2b4be390 2024-07-26T11:22:09.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.432+0000 7f6361e00640 1 --2- 172.21.15.134:0/866928914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b884e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:22:09.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.433+0000 7f6361400640 1 --2- 172.21.15.134:0/866928914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4be390 0x563b2b89b750 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f634c0e1960 tx=0x7f634c00cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f949cf2454595fe5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:09.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.433+0000 7f6352a00640 1 -- 172.21.15.134:0/866928914 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634c0aad70 con 0x563b2b4be390 2024-07-26T11:22:09.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.433+0000 7f6352a00640 1 -- 172.21.15.134:0/866928914 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f634c0af370 con 0x563b2b4be390 2024-07-26T11:22:09.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.433+0000 7f6352a00640 1 -- 172.21.15.134:0/866928914 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634c0a66e0 con 0x563b2b4be390 2024-07-26T11:22:09.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.435+0000 7f636fbf6740 1 -- 172.21.15.134:0/866928914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4be390 msgr2=0x563b2b89b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:09.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.435+0000 7f636fbf6740 1 --2- 172.21.15.134:0/866928914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4be390 0x563b2b89b750 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f634c0e1960 tx=0x7f634c00cf90 comp rx=0 tx=0).stop 2024-07-26T11:22:09.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.435+0000 7f636fbf6740 1 -- 172.21.15.134:0/866928914 shutdown_connections 2024-07-26T11:22:09.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.435+0000 7f636fbf6740 1 --2- 172.21.15.134:0/866928914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4be390 0x563b2b89b750 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.435+0000 7f636fbf6740 1 --2- 172.21.15.134:0/866928914 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4df070 0x563b2b8a6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.435+0000 7f636fbf6740 1 --2- 172.21.15.134:0/866928914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b884e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.436+0000 7f636fbf6740 1 -- 172.21.15.134:0/866928914 >> 172.21.15.134:0/866928914 conn(0x563b2b4d3460 msgr2=0x563b2b88fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:09.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.436+0000 7f636fbf6740 1 -- 172.21.15.134:0/866928914 shutdown_connections 2024-07-26T11:22:09.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.436+0000 7f636fbf6740 1 -- 172.21.15.134:0/866928914 wait complete. 2024-07-26T11:22:09.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.437+0000 7f636fbf6740 1 -- start start 2024-07-26T11:22:09.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.437+0000 7f636fbf6740 1 Processor -- start 2024-07-26T11:22:09.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.440+0000 7f636fbf6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b8931d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.441+0000 7f636fbf6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4be390 0x563b2b8937b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.441+0000 7f636fbf6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4df070 0x563b2b9020a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.441+0000 7f6361e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b8931d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:09.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.442+0000 7f6362800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4df070 0x563b2b9020a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:09.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.442+0000 7f6362800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4df070 0x563b2b9020a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:57252/0 (socket says 172.21.15.134:57252) 2024-07-26T11:22:09.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.442+0000 7f6362800640 1 -- 172.21.15.134:0/3398848910 learned_addr learned my addr 172.21.15.134:0/3398848910 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:09.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.442+0000 7f6361e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b8931d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56360/0 (socket says 172.21.15.134:56360) 2024-07-26T11:22:09.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.442+0000 7f636fbf6740 1 -- 172.21.15.134:0/3398848910 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f6340011400 con 0x563b2b4df070 2024-07-26T11:22:09.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.442+0000 7f636fbf6740 1 -- 172.21.15.134:0/3398848910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6340010d30 con 0x563b2b4b61b0 2024-07-26T11:22:09.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.442+0000 7f636fbf6740 1 -- 172.21.15.134:0/3398848910 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f634000df70 con 0x563b2b4be390 2024-07-26T11:22:09.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.442+0000 7f6361400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4be390 0x563b2b8937b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:09.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.443+0000 7f636fbf6740 10 client.0 _ll_register_callbacks cb 0x7f635c010b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:09.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.443+0000 7f6361e00640 1 -- 172.21.15.134:0/3398848910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4be390 msgr2=0x563b2b8937b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:09.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.444+0000 7f6361e00640 1 --2- 172.21.15.134:0/3398848910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4be390 0x563b2b8937b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.444+0000 7f6361e00640 1 -- 172.21.15.134:0/3398848910 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4df070 msgr2=0x563b2b9020a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:09.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.444+0000 7f6361e00640 1 --2- 172.21.15.134:0/3398848910 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4df070 0x563b2b9020a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.444+0000 7f6361e00640 1 -- 172.21.15.134:0/3398848910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b2b902710 con 0x563b2b4b61b0 2024-07-26T11:22:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.444+0000 7f6361e00640 1 --2- 172.21.15.134:0/3398848910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b8931d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f635c024be0 tx=0x7f635c024700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae8165bfa87b280b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:09.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.444+0000 7f6347e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:22:09.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.445+0000 7f6347e00640 1 -- 172.21.15.134:0/3398848910 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c003eb0 con 0x563b2b4b61b0 2024-07-26T11:22:09.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.445+0000 7f6347e00640 1 -- 172.21.15.134:0/3398848910 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f635c0054e0 con 0x563b2b4b61b0 2024-07-26T11:22:09.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.446+0000 7f636fbf6740 1 -- 172.21.15.134:0/3398848910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563b2b7f85c0 con 0x563b2b4b61b0 2024-07-26T11:22:09.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.446+0000 7f6347e00640 1 -- 172.21.15.134:0/3398848910 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f635c0056e0 con 0x563b2b4b61b0 2024-07-26T11:22:09.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.446+0000 7f636fbf6740 10 client.26390 fetch_fsmap learned FSMap version 32 2024-07-26T11:22:09.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.446+0000 7f636fbf6740 1 -- 172.21.15.134:0/3398848910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563b2b9099f0 con 0x563b2b4b61b0 2024-07-26T11:22:09.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.447+0000 7f6347e00640 1 -- 172.21.15.134:0/3398848910 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x563b2b7f85c0 con 0x563b2b4b61b0 2024-07-26T11:22:09.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.447+0000 7f6347e00640 1 -- 172.21.15.134:0/3398848910 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f635c00b040 con 0x563b2b4b61b0 2024-07-26T11:22:09.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.448+0000 7f636fbf6740 10 client.26390 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:22:09.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.448+0000 7f636fbf6740 10 client.26390 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-07-26T11:22:09.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.448+0000 7f636fbf6740 20 client.26390 populate_metadata read hostname 'smithi134' 2024-07-26T11:22:09.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.448+0000 7f636fbf6740 1 --2- 172.21.15.134:0/3398848910 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x563b2b472da0 0x563b2b90c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:09.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.448+0000 7f636fbf6740 4 client.26390 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:22:09.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.448+0000 7f636fbf6740 1 -- 172.21.15.134:0/3398848910 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563b2b7f5d00 con 0x563b2b472da0 2024-07-26T11:22:09.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.449+0000 7f6361400640 1 --2- 172.21.15.134:0/3398848910 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x563b2b472da0 0x563b2b90c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:09.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.450+0000 7f6361400640 1 --2- 172.21.15.134:0/3398848910 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x563b2b472da0 0x563b2b90c150 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c5a51ba81ee451c9 server_cookie=81c52882e6bacde0 in_seq=0 out_seq=0 2024-07-26T11:22:09.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.450+0000 7f6347e00640 10 client.26390 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:22:09.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.450+0000 7f6347e00640 1 -- 172.21.15.134:0/3398848910 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563b2b7f5d00 con 0x563b2b472da0 2024-07-26T11:22:09.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.450+0000 7f6347e00640 10 client.26390 handle_command_reply: tid=0 2024-07-26T11:22:09.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.450+0000 7f636fbf6740 1 client.26390 shutdown 2024-07-26T11:22:09.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.451+0000 7f636fbf6740 1 -- 172.21.15.134:0/3398848910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 msgr2=0x563b2b8931d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:09.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.451+0000 7f636fbf6740 1 --2- 172.21.15.134:0/3398848910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b8931d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f635c024be0 tx=0x7f635c024700 comp rx=0 tx=0).stop 2024-07-26T11:22:09.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.451+0000 7f636fbf6740 1 -- 172.21.15.134:0/3398848910 shutdown_connections 2024-07-26T11:22:09.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.451+0000 7f636fbf6740 1 --2- 172.21.15.134:0/3398848910 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x563b2b472da0 0x563b2b90c150 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.451+0000 7f636fbf6740 1 --2- 172.21.15.134:0/3398848910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563b2b4be390 0x563b2b8937b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.451+0000 7f636fbf6740 1 --2- 172.21.15.134:0/3398848910 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563b2b4df070 0x563b2b9020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.451+0000 7f636fbf6740 1 --2- 172.21.15.134:0/3398848910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563b2b4b61b0 0x563b2b8931d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.451+0000 7f636fbf6740 1 -- 172.21.15.134:0/3398848910 >> 172.21.15.134:0/3398848910 conn(0x563b2b4d3460 msgr2=0x563b2b88a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:09.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.451+0000 7f6347e00640 0 client.26390 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:22:09.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.455+0000 7f636fbf6740 1 -- 172.21.15.134:0/3398848910 shutdown_connections 2024-07-26T11:22:09.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.455+0000 7f636fbf6740 1 -- 172.21.15.134:0/3398848910 wait complete. 2024-07-26T11:22:09.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.455+0000 7f636fbf6740 20 client.26390 trim_cache size 0 max 16384 2024-07-26T11:22:09.458 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:22:09.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.457+0000 7f6368e00640 1 -- 172.21.15.134:0/2053989889 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63480b3f10 msgr2=0x7f63480b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:09.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.457+0000 7f6368e00640 1 --2- 172.21.15.134:0/2053989889 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63480b3f10 0x7f63480b6420 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f635400b230 tx=0x7f6354005f10 comp rx=0 tx=0).stop 2024-07-26T11:22:09.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.457+0000 7f6368e00640 1 -- 172.21.15.134:0/2053989889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636410f500 msgr2=0x7f636406d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:09.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.457+0000 7f6368e00640 1 --2- 172.21.15.134:0/2053989889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636410f500 0x7f636406d8c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f636406b560 tx=0x7f634c00ee50 comp rx=0 tx=0).stop 2024-07-26T11:22:09.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.457+0000 7f6368e00640 1 -- 172.21.15.134:0/2053989889 shutdown_connections 2024-07-26T11:22:09.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.457+0000 7f6368e00640 1 --2- 172.21.15.134:0/2053989889 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f63480b3f10 0x7f63480b6420 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.457+0000 7f6368e00640 1 --2- 172.21.15.134:0/2053989889 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f636410fe50 0x7f636406de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.457+0000 7f6368e00640 1 --2- 172.21.15.134:0/2053989889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636410f500 0x7f636406d8c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.457+0000 7f6368e00640 1 --2- 172.21.15.134:0/2053989889 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6364109bc0 0x7f63640745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:09.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.458+0000 7f6368e00640 1 -- 172.21.15.134:0/2053989889 >> 172.21.15.134:0/2053989889 conn(0x7f6364077b20 msgr2=0x7f636410d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:09.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.459+0000 7f6368e00640 1 -- 172.21.15.134:0/2053989889 shutdown_connections 2024-07-26T11:22:09.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:09.459+0000 7f6368e00640 1 -- 172.21.15.134:0/2053989889 wait complete. 2024-07-26T11:22:10.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:10 smithi134 ceph-mon[31966]: pgmap v665: 129 pgs: 129 active+clean; 44 GiB data, 93 GiB used, 980 GiB / 1.0 TiB avail; 52 KiB/s rd, 288 MiB/s wr, 1.25k op/s 2024-07-26T11:22:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:10 smithi189 ceph-mon[36790]: pgmap v665: 129 pgs: 129 active+clean; 44 GiB data, 93 GiB used, 980 GiB / 1.0 TiB avail; 52 KiB/s rd, 288 MiB/s wr, 1.25k op/s 2024-07-26T11:22:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:10 smithi193 ceph-mon[37473]: pgmap v665: 129 pgs: 129 active+clean; 44 GiB data, 93 GiB used, 980 GiB / 1.0 TiB avail; 52 KiB/s rd, 288 MiB/s wr, 1.25k op/s 2024-07-26T11:22:12.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:12 smithi134 ceph-mon[31966]: pgmap v666: 129 pgs: 129 active+clean; 45 GiB data, 93 GiB used, 980 GiB / 1.0 TiB avail; 50 KiB/s rd, 292 MiB/s wr, 1.36k op/s 2024-07-26T11:22:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:12 smithi189 ceph-mon[36790]: pgmap v666: 129 pgs: 129 active+clean; 45 GiB data, 93 GiB used, 980 GiB / 1.0 TiB avail; 50 KiB/s rd, 292 MiB/s wr, 1.36k op/s 2024-07-26T11:22:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:12 smithi193 ceph-mon[37473]: pgmap v666: 129 pgs: 129 active+clean; 45 GiB data, 93 GiB used, 980 GiB / 1.0 TiB avail; 50 KiB/s rd, 292 MiB/s wr, 1.36k op/s 2024-07-26T11:22:13.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.054+0000 7f12a9c00640 1 -- 172.21.15.134:0/1489411306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a411c5d0 msgr2=0x7f12a411e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:13.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.054+0000 7f12a9c00640 1 --2- 172.21.15.134:0/1489411306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a411c5d0 0x7f12a411e9f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f129c009ca0 tx=0x7f129c02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.055+0000 7f12a9c00640 1 -- 172.21.15.134:0/1489411306 shutdown_connections 2024-07-26T11:22:13.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.055+0000 7f12a9c00640 1 --2- 172.21.15.134:0/1489411306 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a411c5d0 0x7f12a411e9f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.055+0000 7f12a9c00640 1 --2- 172.21.15.134:0/1489411306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a410a330 0x7f12a410a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.055+0000 7f12a9c00640 1 --2- 172.21.15.134:0/1489411306 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12a41116d0 0x7f12a4109df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.055+0000 7f12a9c00640 1 -- 172.21.15.134:0/1489411306 >> 172.21.15.134:0/1489411306 conn(0x7f12a4070630 msgr2=0x7f12a4070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:13.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.055+0000 7f12a9c00640 1 -- 172.21.15.134:0/1489411306 shutdown_connections 2024-07-26T11:22:13.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.055+0000 7f12a9c00640 1 -- 172.21.15.134:0/1489411306 wait complete. 2024-07-26T11:22:13.060 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.057+0000 7f12a9c00640 1 Processor -- start 2024-07-26T11:22:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.058+0000 7f12a9c00640 1 -- start start 2024-07-26T11:22:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.059+0000 7f12a9c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12a410a330 0x7f12a41a8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.059+0000 7f12a9c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a41116d0 0x7f12a41a9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.059+0000 7f12a9c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a411c5d0 0x7f12a41ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.059+0000 7f12a9c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f12a41220c0 con 0x7f12a410a330 2024-07-26T11:22:13.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.059+0000 7f12a9c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f12a4121f40 con 0x7f12a411c5d0 2024-07-26T11:22:13.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.059+0000 7f12a9c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f12a4122240 con 0x7f12a41116d0 2024-07-26T11:22:13.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.060+0000 7f12a3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12a410a330 0x7f12a41a8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:13.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.061+0000 7f12a3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a41116d0 0x7f12a41a9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:13.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.062+0000 7f12a3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a41116d0 0x7f12a41a9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58236/0 (socket says 172.21.15.134:58236) 2024-07-26T11:22:13.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.062+0000 7f12a3400640 1 -- 172.21.15.134:0/2518097914 learned_addr learned my addr 172.21.15.134:0/2518097914 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:13.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.063+0000 7f12a9200640 1 --2- 172.21.15.134:0/2518097914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a411c5d0 0x7f12a41ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:13.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.064+0000 7f12a3400640 1 -- 172.21.15.134:0/2518097914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a411c5d0 msgr2=0x7f12a41ae2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.064+0000 7f12a3400640 1 --2- 172.21.15.134:0/2518097914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a411c5d0 0x7f12a41ae2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.064+0000 7f12a3400640 1 -- 172.21.15.134:0/2518097914 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12a410a330 msgr2=0x7f12a41a8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.064+0000 7f12a3400640 1 --2- 172.21.15.134:0/2518097914 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12a410a330 0x7f12a41a8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.064+0000 7f12a3400640 1 -- 172.21.15.134:0/2518097914 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a41ae9f0 con 0x7f12a41116d0 2024-07-26T11:22:13.067 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.064+0000 7f12a9200640 1 --2- 172.21.15.134:0/2518097914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a411c5d0 0x7f12a41ae2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:22:13.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.066+0000 7f12a3400640 1 --2- 172.21.15.134:0/2518097914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a41116d0 0x7f12a41a9060 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f128c005fd0 tx=0x7f128c00dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.067+0000 7f12a0c00640 1 -- 172.21.15.134:0/2518097914 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128c017070 con 0x7f12a41116d0 2024-07-26T11:22:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.067+0000 7f12a0c00640 1 -- 172.21.15.134:0/2518097914 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f128c009c30 con 0x7f12a41116d0 2024-07-26T11:22:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.067+0000 7f12a0c00640 1 -- 172.21.15.134:0/2518097914 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128c003d30 con 0x7f12a41116d0 2024-07-26T11:22:13.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.067+0000 7f12a9c00640 1 -- 172.21.15.134:0/2518097914 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12a41aece0 con 0x7f12a41116d0 2024-07-26T11:22:13.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.067+0000 7f12a9c00640 1 -- 172.21.15.134:0/2518097914 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12a4111db0 con 0x7f12a41116d0 2024-07-26T11:22:13.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.079+0000 7f12b23e9740 1 Processor -- start 2024-07-26T11:22:13.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.081+0000 7f12b23e9740 1 -- start start 2024-07-26T11:22:13.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.082+0000 7f12b23e9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 0x561428a41c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.087 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.082+0000 7f12b23e9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5614286752f0 0x561428a42160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.082+0000 7f12b23e9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56142867d4d0 0x561428a426a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.082+0000 7f12b23e9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5614289b6270 con 0x5614286752f0 2024-07-26T11:22:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.082+0000 7f12b23e9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5614289a5770 con 0x56142869e1b0 2024-07-26T11:22:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.082+0000 7f12b23e9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5614289a97b0 con 0x56142867d4d0 2024-07-26T11:22:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.082+0000 7f12a9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 0x561428a41c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.082+0000 7f12a0c00640 1 -- 172.21.15.134:0/2518097914 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f128c003ed0 con 0x7f12a41116d0 2024-07-26T11:22:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.082+0000 7f12a9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 0x561428a41c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38012/0 (socket says 172.21.15.134:38012) 2024-07-26T11:22:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.082+0000 7f12a9200640 1 -- 172.21.15.134:0/3024889489 learned_addr learned my addr 172.21.15.134:0/3024889489 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.082+0000 7f12a3400640 1 --2- 172.21.15.134:0/3024889489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56142867d4d0 0x561428a426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.084+0000 7f12a9200640 1 -- 172.21.15.134:0/3024889489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56142867d4d0 msgr2=0x561428a426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.084+0000 7f12a9200640 1 --2- 172.21.15.134:0/3024889489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56142867d4d0 0x561428a426a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.084+0000 7f12a9200640 1 -- 172.21.15.134:0/3024889489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5614286752f0 msgr2=0x561428a42160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.084+0000 7f12a9200640 1 --2- 172.21.15.134:0/3024889489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5614286752f0 0x561428a42160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.084+0000 7f12a9200640 1 -- 172.21.15.134:0/3024889489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561428709790 con 0x56142869e1b0 2024-07-26T11:22:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.085+0000 7f12a0c00640 1 --2- 172.21.15.134:0/2518097914 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f12880b3f10 0x7f12880b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.085+0000 7f12a9200640 1 --2- 172.21.15.134:0/3024889489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 0x561428a41c20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f129c009a20 tx=0x7f129c031af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd56ddf42eb41c2d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.085+0000 7f12a0c00640 1 -- 172.21.15.134:0/2518097914 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f128c0e3f90 con 0x7f12a41116d0 2024-07-26T11:22:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.086+0000 7f12a3e00640 1 --2- 172.21.15.134:0/2518097914 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f12880b3f10 0x7f12880b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.086+0000 7f1299600640 1 -- 172.21.15.134:0/3024889489 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129c031650 con 0x56142869e1b0 2024-07-26T11:22:13.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.086+0000 7f1299600640 1 -- 172.21.15.134:0/3024889489 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f129c040070 con 0x56142869e1b0 2024-07-26T11:22:13.090 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.086+0000 7f1299600640 1 -- 172.21.15.134:0/3024889489 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129c0046e0 con 0x56142869e1b0 2024-07-26T11:22:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.089+0000 7f12b23e9740 1 -- 172.21.15.134:0/3024889489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 msgr2=0x561428a41c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:13.091 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.089+0000 7f12b23e9740 1 --2- 172.21.15.134:0/3024889489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 0x561428a41c20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f129c009a20 tx=0x7f129c031af0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.090+0000 7f12b23e9740 1 -- 172.21.15.134:0/3024889489 shutdown_connections 2024-07-26T11:22:13.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.090+0000 7f12b23e9740 1 --2- 172.21.15.134:0/3024889489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56142867d4d0 0x561428a426a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.090+0000 7f12a3e00640 1 --2- 172.21.15.134:0/2518097914 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f12880b3f10 0x7f12880b6420 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1294005d90 tx=0x7f1294005d20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:13.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.090+0000 7f12b23e9740 1 --2- 172.21.15.134:0/3024889489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5614286752f0 0x561428a42160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.090+0000 7f12b23e9740 1 --2- 172.21.15.134:0/3024889489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 0x561428a41c20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.090+0000 7f12b23e9740 1 -- 172.21.15.134:0/3024889489 >> 172.21.15.134:0/3024889489 conn(0x561428692300 msgr2=0x561428a3fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:13.095 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.091+0000 7f12b23e9740 1 -- 172.21.15.134:0/3024889489 shutdown_connections 2024-07-26T11:22:13.096 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.091+0000 7f12b23e9740 1 -- 172.21.15.134:0/3024889489 wait complete. 2024-07-26T11:22:13.096 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.094+0000 7f12b23e9740 1 -- start start 2024-07-26T11:22:13.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.095+0000 7f12b23e9740 1 Processor -- start 2024-07-26T11:22:13.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.098+0000 7f12b23e9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5614286752f0 0x561428a4d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.098+0000 7f12b23e9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56142867d4d0 0x561428a4dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.098+0000 7f12b23e9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56142869e1b0 0x561428a53600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.098+0000 7f12b23e9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5614288dfab0 con 0x56142869e1b0 2024-07-26T11:22:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.098+0000 7f12b23e9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561428942d50 con 0x5614286752f0 2024-07-26T11:22:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.098+0000 7f12b23e9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561428a50340 con 0x56142867d4d0 2024-07-26T11:22:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.098+0000 7f12b23e9740 10 client.0 _ll_register_callbacks cb 0x5614286b5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.099+0000 7f12a3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56142867d4d0 0x561428a4dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.099+0000 7f12a3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56142867d4d0 0x561428a4dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58256/0 (socket says 172.21.15.134:58256) 2024-07-26T11:22:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.099+0000 7f12a3400640 1 -- 172.21.15.134:0/824979886 learned_addr learned my addr 172.21.15.134:0/824979886 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.100+0000 7f12a3400640 1 -- 172.21.15.134:0/824979886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5614286752f0 msgr2=0x561428a4d900 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:22:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.100+0000 7f12a3400640 1 --2- 172.21.15.134:0/824979886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5614286752f0 0x561428a4d900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.100+0000 7f12a3400640 1 -- 172.21.15.134:0/824979886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56142869e1b0 msgr2=0x561428a53600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:22:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.100+0000 7f12a3400640 1 --2- 172.21.15.134:0/824979886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56142869e1b0 0x561428a53600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.100+0000 7f12a3400640 1 -- 172.21.15.134:0/824979886 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561428782620 con 0x56142867d4d0 2024-07-26T11:22:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.101+0000 7f12a3400640 1 --2- 172.21.15.134:0/824979886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56142867d4d0 0x561428a4dee0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f128c0a6840 tx=0x7f128c0aa590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=512a4b151aa6ccec server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.101+0000 7f1292000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:22:13.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.104+0000 7f1292000640 1 -- 172.21.15.134:0/824979886 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128c0b31b0 con 0x56142867d4d0 2024-07-26T11:22:13.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.105+0000 7f1292000640 1 -- 172.21.15.134:0/824979886 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f128c0b33e0 con 0x56142867d4d0 2024-07-26T11:22:13.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.105+0000 7f12b23e9740 1 -- 172.21.15.134:0/824979886 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f129c040070 con 0x56142867d4d0 2024-07-26T11:22:13.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.105+0000 7f1292000640 1 -- 172.21.15.134:0/824979886 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f128c0bfe60 con 0x56142867d4d0 2024-07-26T11:22:13.108 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.106+0000 7f12b23e9740 10 client.26404 fetch_fsmap learned FSMap version 32 2024-07-26T11:22:13.108 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.106+0000 7f12b23e9740 1 -- 172.21.15.134:0/824979886 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561428a5c4a0 con 0x56142867d4d0 2024-07-26T11:22:13.108 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.106+0000 7f1292000640 1 -- 172.21.15.134:0/824979886 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f129c040070 con 0x56142867d4d0 2024-07-26T11:22:13.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.107+0000 7f1292000640 1 -- 172.21.15.134:0/824979886 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f128c0cd0d0 con 0x56142867d4d0 2024-07-26T11:22:13.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.107+0000 7f12b23e9740 10 client.26404 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:22:13.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.107+0000 7f12b23e9740 10 client.26404 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:22:13.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.107+0000 7f12b23e9740 20 client.26404 populate_metadata read hostname 'smithi134' 2024-07-26T11:22:13.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.107+0000 7f12b23e9740 1 --2- 172.21.15.134:0/824979886 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56142865cd80 0x561428a5f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.107+0000 7f12b23e9740 4 client.26404 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:22:13.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.107+0000 7f12b23e9740 1 -- 172.21.15.134:0/824979886 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614289b81c0 con 0x56142865cd80 2024-07-26T11:22:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.111+0000 7f12a3e00640 1 --2- 172.21.15.134:0/824979886 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56142865cd80 0x561428a5f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.112+0000 7f12a3e00640 1 --2- 172.21.15.134:0/824979886 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56142865cd80 0x561428a5f9f0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c52012c864e52080 server_cookie=2c246f53e780dadc in_seq=0 out_seq=0 2024-07-26T11:22:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.112+0000 7f1292000640 10 client.26404 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:22:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.113+0000 7f1292000640 1 -- 172.21.15.134:0/824979886 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5614289b81c0 con 0x56142865cd80 2024-07-26T11:22:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.114+0000 7f1292000640 10 client.26404 handle_command_reply: tid=0 2024-07-26T11:22:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.114+0000 7f12b23e9740 1 client.26404 shutdown 2024-07-26T11:22:13.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.117+0000 7f12b23e9740 1 -- 172.21.15.134:0/824979886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56142867d4d0 msgr2=0x561428a4dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:13.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.117+0000 7f12b23e9740 1 --2- 172.21.15.134:0/824979886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56142867d4d0 0x561428a4dee0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f128c0a6840 tx=0x7f128c0aa590 comp rx=0 tx=0).stop 2024-07-26T11:22:13.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.117+0000 7f12b23e9740 1 -- 172.21.15.134:0/824979886 shutdown_connections 2024-07-26T11:22:13.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.117+0000 7f12b23e9740 1 --2- 172.21.15.134:0/824979886 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56142865cd80 0x561428a5f9f0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.117+0000 7f12b23e9740 1 --2- 172.21.15.134:0/824979886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56142867d4d0 0x561428a4dee0 secure :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f128c0a6840 tx=0x7f128c0aa590 comp rx=0 tx=0).stop 2024-07-26T11:22:13.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.117+0000 7f12b23e9740 1 --2- 172.21.15.134:0/824979886 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56142869e1b0 0x561428a53600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.117+0000 7f12b23e9740 1 --2- 172.21.15.134:0/824979886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5614286752f0 0x561428a4d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.117+0000 7f12b23e9740 1 -- 172.21.15.134:0/824979886 >> 172.21.15.134:0/824979886 conn(0x561428692300 msgr2=0x561428a42c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:13.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.117+0000 7f1292000640 0 client.26404 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:22:13.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.118+0000 7f12b23e9740 1 -- 172.21.15.134:0/824979886 shutdown_connections 2024-07-26T11:22:13.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.118+0000 7f12b23e9740 1 -- 172.21.15.134:0/824979886 wait complete. 2024-07-26T11:22:13.121 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.118+0000 7f12b23e9740 20 client.26404 trim_cache size 0 max 16384 2024-07-26T11:22:13.164 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.161+0000 7f12b23e9740 1 Processor -- start 2024-07-26T11:22:13.164 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.161+0000 7f12b23e9740 1 -- start start 2024-07-26T11:22:13.165 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.162+0000 7f12b23e9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614286752f0 0x561428a4dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.165 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.162+0000 7f12b23e9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 0x561428a5a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.162+0000 7f12b23e9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5614286321c0 0x561428a53570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.162+0000 7f12b23e9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f12a41c7e60 con 0x5614286321c0 2024-07-26T11:22:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.162+0000 7f12b23e9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f12a41c99a0 con 0x56142869e1b0 2024-07-26T11:22:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.162+0000 7f12b23e9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f12a41c8530 con 0x5614286752f0 2024-07-26T11:22:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.163+0000 7f12a3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614286752f0 0x561428a4dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:13.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.163+0000 7f12a3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614286752f0 0x561428a4dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58260/0 (socket says 172.21.15.134:58260) 2024-07-26T11:22:13.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.163+0000 7f12a3e00640 1 -- 172.21.15.134:0/116481956 learned_addr learned my addr 172.21.15.134:0/116481956 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.163+0000 7f12a3400640 1 --2- 172.21.15.134:0/116481956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 0x561428a5a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.163+0000 7f12a9200640 1 --2- 172.21.15.134:0/116481956 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5614286321c0 0x561428a53570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.164+0000 7f12a3e00640 1 -- 172.21.15.134:0/116481956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 msgr2=0x561428a5a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.164+0000 7f12a3e00640 1 --2- 172.21.15.134:0/116481956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 0x561428a5a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.164+0000 7f12a3e00640 1 -- 172.21.15.134:0/116481956 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5614286321c0 msgr2=0x561428a53570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.164+0000 7f12a3e00640 1 --2- 172.21.15.134:0/116481956 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5614286321c0 0x561428a53570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.164+0000 7f12a3e00640 1 -- 172.21.15.134:0/116481956 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561428a4bbf0 con 0x5614286752f0 2024-07-26T11:22:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.165+0000 7f12a3e00640 1 --2- 172.21.15.134:0/116481956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614286752f0 0x561428a4dd80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f129400ec10 tx=0x7f1294014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6baf23cabff2aebc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:13.169 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.165+0000 7f1298c00640 1 -- 172.21.15.134:0/116481956 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1294010430 con 0x5614286752f0 2024-07-26T11:22:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.166+0000 7f1298c00640 1 -- 172.21.15.134:0/116481956 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1294010a80 con 0x5614286752f0 2024-07-26T11:22:13.170 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.166+0000 7f1298c00640 1 -- 172.21.15.134:0/116481956 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12940247b0 con 0x5614286752f0 2024-07-26T11:22:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.167+0000 7f12b23e9740 1 -- 172.21.15.134:0/116481956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614286752f0 msgr2=0x561428a4dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:13.171 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.167+0000 7f12b23e9740 1 --2- 172.21.15.134:0/116481956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614286752f0 0x561428a4dd80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f129400ec10 tx=0x7f1294014000 comp rx=0 tx=0).stop 2024-07-26T11:22:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.170+0000 7f12b23e9740 1 -- 172.21.15.134:0/116481956 shutdown_connections 2024-07-26T11:22:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.170+0000 7f12b23e9740 1 --2- 172.21.15.134:0/116481956 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5614286321c0 0x561428a53570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.170+0000 7f12b23e9740 1 --2- 172.21.15.134:0/116481956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 0x561428a5a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.170+0000 7f12b23e9740 1 --2- 172.21.15.134:0/116481956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614286752f0 0x561428a4dd80 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.170+0000 7f12b23e9740 1 -- 172.21.15.134:0/116481956 >> 172.21.15.134:0/116481956 conn(0x561428692300 msgr2=0x561428a46d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.170+0000 7f12b23e9740 1 -- 172.21.15.134:0/116481956 shutdown_connections 2024-07-26T11:22:13.172 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.170+0000 7f12b23e9740 1 -- 172.21.15.134:0/116481956 wait complete. 2024-07-26T11:22:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.173+0000 7f12b23e9740 1 -- start start 2024-07-26T11:22:13.175 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.173+0000 7f12b23e9740 1 Processor -- start 2024-07-26T11:22:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.175+0000 7f12b23e9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614286321c0 0x561428a552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.175+0000 7f12b23e9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5614286752f0 0x561428a55890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.175+0000 7f12b23e9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 0x561428ac0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.175+0000 7f12b23e9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f12a41c92d0 con 0x5614286752f0 2024-07-26T11:22:13.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.175+0000 7f12b23e9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f12a41c8c00 con 0x56142869e1b0 2024-07-26T11:22:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.175+0000 7f12b23e9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f12a4074dc0 con 0x5614286321c0 2024-07-26T11:22:13.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.176+0000 7f12b23e9740 10 client.0 _ll_register_callbacks cb 0x7f128c005bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:13.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.181+0000 7f12a3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614286321c0 0x561428a552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:13.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.181+0000 7f12a9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 0x561428ac0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:13.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.181+0000 7f12a9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 0x561428ac0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38048/0 (socket says 172.21.15.134:38048) 2024-07-26T11:22:13.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.181+0000 7f12a9200640 1 -- 172.21.15.134:0/2950038269 learned_addr learned my addr 172.21.15.134:0/2950038269 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.182+0000 7f12a3e00640 1 -- 172.21.15.134:0/2950038269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 msgr2=0x561428ac0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.183+0000 7f12a3e00640 1 --2- 172.21.15.134:0/2950038269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 0x561428ac0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.183+0000 7f12a3e00640 1 -- 172.21.15.134:0/2950038269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5614286752f0 msgr2=0x561428a55890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.183+0000 7f12a3e00640 1 --2- 172.21.15.134:0/2950038269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5614286752f0 0x561428a55890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.183+0000 7f12a3e00640 1 -- 172.21.15.134:0/2950038269 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561428ac0800 con 0x5614286321c0 2024-07-26T11:22:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.185+0000 7f12a3e00640 1 --2- 172.21.15.134:0/2950038269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614286321c0 0x561428a552d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f129400eaa0 tx=0x7f129400edb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfb29e3a109d225c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:13.189 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.187+0000 7f1292000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:22:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.187+0000 7f1292000640 1 -- 172.21.15.134:0/2950038269 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1294031070 con 0x5614286321c0 2024-07-26T11:22:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.187+0000 7f1292000640 1 -- 172.21.15.134:0/2950038269 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f1294010630 con 0x5614286321c0 2024-07-26T11:22:13.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.187+0000 7f12b23e9740 1 -- 172.21.15.134:0/2950038269 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1294010a80 con 0x5614286321c0 2024-07-26T11:22:13.193 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.191+0000 7f1292000640 1 -- 172.21.15.134:0/2950038269 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1294010830 con 0x5614286321c0 2024-07-26T11:22:13.193 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.191+0000 7f12b23e9740 10 client.26410 fetch_fsmap learned FSMap version 32 2024-07-26T11:22:13.193 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.191+0000 7f12b23e9740 1 -- 172.21.15.134:0/2950038269 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561428ac7b30 con 0x5614286321c0 2024-07-26T11:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.191+0000 7f1292000640 1 -- 172.21.15.134:0/2950038269 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1294010a80 con 0x5614286321c0 2024-07-26T11:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.191+0000 7f1292000640 1 -- 172.21.15.134:0/2950038269 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f1294004b50 con 0x5614286321c0 2024-07-26T11:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.191+0000 7f12b23e9740 10 client.26410 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:22:13.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.191+0000 7f12b23e9740 10 client.26410 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.191+0000 7f12b23e9740 20 client.26410 populate_metadata read hostname 'smithi134' 2024-07-26T11:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.191+0000 7f12b23e9740 1 --2- 172.21.15.134:0/2950038269 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x561428ac8300 0x561428aca720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.192+0000 7f12b23e9740 4 client.26410 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:22:13.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.192+0000 7f12b23e9740 1 -- 172.21.15.134:0/2950038269 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5614289b3c60 con 0x561428ac8300 2024-07-26T11:22:13.202 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.196+0000 7f12a3400640 1 --2- 172.21.15.134:0/2950038269 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x561428ac8300 0x561428aca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.204+0000 7f1292000640 10 client.26410 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:22:13.206 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:13.204+0000 7f12a3400640 1 --2- 172.21.15.134:0/2950038269 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x561428ac8300 0x561428aca720 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13f0e8950aa7d02d server_cookie=30dfcc89a65c31ad in_seq=0 out_seq=0 2024-07-26T11:22:13.324 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-07-26T11:22:13.457 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:13 smithi134 ceph-mon[31966]: pgmap v667: 129 pgs: 129 active+clean; 45 GiB data, 94 GiB used, 979 GiB / 1.0 TiB avail; 51 KiB/s rd, 260 MiB/s wr, 1.33k op/s 2024-07-26T11:22:13.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:13 smithi189 ceph-mon[36790]: pgmap v667: 129 pgs: 129 active+clean; 45 GiB data, 94 GiB used, 979 GiB / 1.0 TiB avail; 51 KiB/s rd, 260 MiB/s wr, 1.33k op/s 2024-07-26T11:22:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:13 smithi193 ceph-mon[37473]: pgmap v667: 129 pgs: 129 active+clean; 45 GiB data, 94 GiB used, 979 GiB / 1.0 TiB avail; 51 KiB/s rd, 260 MiB/s wr, 1.33k op/s 2024-07-26T11:22:14.559 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:22:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:14 smithi193 ceph-mon[37473]: pgmap v668: 129 pgs: 129 active+clean; 45 GiB data, 94 GiB used, 979 GiB / 1.0 TiB avail; 58 KiB/s rd, 272 MiB/s wr, 1.58k op/s 2024-07-26T11:22:14.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:14 smithi134 ceph-mon[31966]: pgmap v668: 129 pgs: 129 active+clean; 45 GiB data, 94 GiB used, 979 GiB / 1.0 TiB avail; 58 KiB/s rd, 272 MiB/s wr, 1.58k op/s 2024-07-26T11:22:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:14 smithi189 ceph-mon[36790]: pgmap v668: 129 pgs: 129 active+clean; 45 GiB data, 94 GiB used, 979 GiB / 1.0 TiB avail; 58 KiB/s rd, 272 MiB/s wr, 1.58k op/s 2024-07-26T11:22:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.932+0000 7f1292000640 1 -- 172.21.15.134:0/2950038269 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5614289b3c60 con 0x561428ac8300 2024-07-26T11:22:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.932+0000 7f1292000640 10 client.26410 handle_command_reply: tid=0 2024-07-26T11:22:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.932+0000 7f12b23e9740 1 client.26410 shutdown 2024-07-26T11:22:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.933+0000 7f12b23e9740 1 -- 172.21.15.134:0/2950038269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614286321c0 msgr2=0x561428a552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.933+0000 7f12b23e9740 1 --2- 172.21.15.134:0/2950038269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614286321c0 0x561428a552d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f129400eaa0 tx=0x7f129400edb0 comp rx=0 tx=0).stop 2024-07-26T11:22:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.933+0000 7f12b23e9740 1 -- 172.21.15.134:0/2950038269 shutdown_connections 2024-07-26T11:22:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.933+0000 7f12b23e9740 1 --2- 172.21.15.134:0/2950038269 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x561428ac8300 0x561428aca720 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.933+0000 7f12b23e9740 1 --2- 172.21.15.134:0/2950038269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56142869e1b0 0x561428ac0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.933+0000 7f12b23e9740 1 --2- 172.21.15.134:0/2950038269 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5614286752f0 0x561428a55890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.933+0000 7f12b23e9740 1 --2- 172.21.15.134:0/2950038269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5614286321c0 0x561428a552d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.933+0000 7f12b23e9740 1 -- 172.21.15.134:0/2950038269 >> 172.21.15.134:0/2950038269 conn(0x561428692300 msgr2=0x561428a3f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.933+0000 7f1292000640 0 client.26410 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:22:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.933+0000 7f12b23e9740 1 -- 172.21.15.134:0/2950038269 shutdown_connections 2024-07-26T11:22:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.934+0000 7f12b23e9740 1 -- 172.21.15.134:0/2950038269 wait complete. 2024-07-26T11:22:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.934+0000 7f12b23e9740 20 client.26410 trim_cache size 0 max 16384 2024-07-26T11:22:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:22:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:22:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:22:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:22:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.938+0000 7f12a9c00640 1 -- 172.21.15.134:0/2518097914 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f12880b3f10 msgr2=0x7f12880b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:15.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.938+0000 7f12a9c00640 1 --2- 172.21.15.134:0/2518097914 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f12880b3f10 0x7f12880b6420 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1294005d90 tx=0x7f1294005d20 comp rx=0 tx=0).stop 2024-07-26T11:22:15.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.938+0000 7f12a9c00640 1 -- 172.21.15.134:0/2518097914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a41116d0 msgr2=0x7f12a41a9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.938+0000 7f12a9c00640 1 --2- 172.21.15.134:0/2518097914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a41116d0 0x7f12a41a9060 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f128c005fd0 tx=0x7f128c00dde0 comp rx=0 tx=0).stop 2024-07-26T11:22:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.938+0000 7f12a9c00640 1 -- 172.21.15.134:0/2518097914 shutdown_connections 2024-07-26T11:22:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.938+0000 7f12a9c00640 1 --2- 172.21.15.134:0/2518097914 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f12880b3f10 0x7f12880b6420 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:15.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.938+0000 7f12a9c00640 1 --2- 172.21.15.134:0/2518097914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f12a41116d0 0x7f12a41a9060 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.938+0000 7f12a9c00640 1 --2- 172.21.15.134:0/2518097914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12a411c5d0 0x7f12a41ae2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.938+0000 7f12a9c00640 1 --2- 172.21.15.134:0/2518097914 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f12a410a330 0x7f12a41a8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.938+0000 7f12a9c00640 1 -- 172.21.15.134:0/2518097914 >> 172.21.15.134:0/2518097914 conn(0x7f12a4070630 msgr2=0x7f12a411a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.939+0000 7f12a9c00640 1 -- 172.21.15.134:0/2518097914 shutdown_connections 2024-07-26T11:22:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:15.939+0000 7f12a9c00640 1 -- 172.21.15.134:0/2518097914 wait complete. 2024-07-26T11:22:16.820 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:16 smithi134 ceph-mon[31966]: pgmap v669: 129 pgs: 129 active+clean; 45 GiB data, 94 GiB used, 979 GiB / 1.0 TiB avail; 22 KiB/s rd, 258 MiB/s wr, 1.70k op/s 2024-07-26T11:22:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:16 smithi189 ceph-mon[36790]: pgmap v669: 129 pgs: 129 active+clean; 45 GiB data, 94 GiB used, 979 GiB / 1.0 TiB avail; 22 KiB/s rd, 258 MiB/s wr, 1.70k op/s 2024-07-26T11:22:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:16 smithi193 ceph-mon[37473]: pgmap v669: 129 pgs: 129 active+clean; 45 GiB data, 94 GiB used, 979 GiB / 1.0 TiB avail; 22 KiB/s rd, 258 MiB/s wr, 1.70k op/s 2024-07-26T11:22:18.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:18 smithi134 ceph-mon[31966]: pgmap v670: 129 pgs: 129 active+clean; 45 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 24 KiB/s rd, 261 MiB/s wr, 1.90k op/s 2024-07-26T11:22:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:18 smithi193 ceph-mon[37473]: pgmap v670: 129 pgs: 129 active+clean; 45 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 24 KiB/s rd, 261 MiB/s wr, 1.90k op/s 2024-07-26T11:22:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:18 smithi189 ceph-mon[36790]: pgmap v670: 129 pgs: 129 active+clean; 45 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 24 KiB/s rd, 261 MiB/s wr, 1.90k op/s 2024-07-26T11:22:20.926 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:20 smithi134 ceph-mon[31966]: pgmap v671: 129 pgs: 129 active+clean; 45 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 19 KiB/s rd, 267 MiB/s wr, 2.13k op/s 2024-07-26T11:22:20.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:20 smithi189 ceph-mon[36790]: pgmap v671: 129 pgs: 129 active+clean; 45 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 19 KiB/s rd, 267 MiB/s wr, 2.13k op/s 2024-07-26T11:22:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:20 smithi193 ceph-mon[37473]: pgmap v671: 129 pgs: 129 active+clean; 45 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 19 KiB/s rd, 267 MiB/s wr, 2.13k op/s 2024-07-26T11:22:22.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:22 smithi134 ceph-mon[31966]: pgmap v672: 129 pgs: 129 active+clean; 45 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 19 KiB/s rd, 217 MiB/s wr, 1.79k op/s 2024-07-26T11:22:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:22 smithi189 ceph-mon[36790]: pgmap v672: 129 pgs: 129 active+clean; 45 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 19 KiB/s rd, 217 MiB/s wr, 1.79k op/s 2024-07-26T11:22:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:22 smithi193 ceph-mon[37473]: pgmap v672: 129 pgs: 129 active+clean; 45 GiB data, 95 GiB used, 978 GiB / 1.0 TiB avail; 19 KiB/s rd, 217 MiB/s wr, 1.79k op/s 2024-07-26T11:22:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:24 smithi134 ceph-mon[31966]: pgmap v673: 129 pgs: 129 active+clean; 46 GiB data, 97 GiB used, 976 GiB / 1.0 TiB avail; 40 KiB/s rd, 255 MiB/s wr, 1.98k op/s 2024-07-26T11:22:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:24 smithi189 ceph-mon[36790]: pgmap v673: 129 pgs: 129 active+clean; 46 GiB data, 97 GiB used, 976 GiB / 1.0 TiB avail; 40 KiB/s rd, 255 MiB/s wr, 1.98k op/s 2024-07-26T11:22:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:24 smithi193 ceph-mon[37473]: pgmap v673: 129 pgs: 129 active+clean; 46 GiB data, 97 GiB used, 976 GiB / 1.0 TiB avail; 40 KiB/s rd, 255 MiB/s wr, 1.98k op/s 2024-07-26T11:22:25.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.221+0000 7f37ae200640 1 -- 172.21.15.134:0/2355331955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37a810c2b0 msgr2=0x7f37a8114800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:25.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.221+0000 7f37ae200640 1 --2- 172.21.15.134:0/2355331955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37a810c2b0 0x7f37a8114800 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f379400dbf0 tx=0x7f379402e2a0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.223+0000 7f37ae200640 1 -- 172.21.15.134:0/2355331955 shutdown_connections 2024-07-26T11:22:25.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.223+0000 7f37ae200640 1 --2- 172.21.15.134:0/2355331955 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37a8114d90 0x7f37a8117280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.223+0000 7f37ae200640 1 --2- 172.21.15.134:0/2355331955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37a810c2b0 0x7f37a8114800 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.223+0000 7f37ae200640 1 --2- 172.21.15.134:0/2355331955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37a810b960 0x7f37a810bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.223+0000 7f37ae200640 1 -- 172.21.15.134:0/2355331955 >> 172.21.15.134:0/2355331955 conn(0x7f37a806e880 msgr2=0x7f37a806ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:25.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.225+0000 7f37ae200640 1 -- 172.21.15.134:0/2355331955 shutdown_connections 2024-07-26T11:22:25.234 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.226+0000 7f37ae200640 1 -- 172.21.15.134:0/2355331955 wait complete. 2024-07-26T11:22:25.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.236+0000 7f37ae200640 1 Processor -- start 2024-07-26T11:22:25.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.237+0000 7f37ae200640 1 -- start start 2024-07-26T11:22:25.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.237+0000 7f37ae200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37a810b960 0x7f37a81a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.238+0000 7f37ae200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37a810c2b0 0x7f37a81a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.238+0000 7f37ae200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37a8114d90 0x7f37a81a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.238+0000 7f37ace00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37a810b960 0x7f37a81a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.238+0000 7f37ace00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37a810b960 0x7f37a81a0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:50666/0 (socket says 172.21.15.134:50666) 2024-07-26T11:22:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.238+0000 7f37a7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37a810c2b0 0x7f37a81a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.238+0000 7f37a7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37a810c2b0 0x7f37a81a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:59096/0 (socket says 172.21.15.134:59096) 2024-07-26T11:22:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.239+0000 7f37ae200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f37a81198d0 con 0x7f37a8114d90 2024-07-26T11:22:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.239+0000 7f37ae200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f37a8119750 con 0x7f37a810c2b0 2024-07-26T11:22:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.240+0000 7f37ace00640 1 -- 172.21.15.134:0/2529691251 learned_addr learned my addr 172.21.15.134:0/2529691251 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.240+0000 7f37ae200640 1 -- 172.21.15.134:0/2529691251 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f37a8119a50 con 0x7f37a810b960 2024-07-26T11:22:25.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.244+0000 7f37ad800640 1 --2- 172.21.15.134:0/2529691251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37a8114d90 0x7f37a81a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:25.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.245+0000 7f37a7e00640 1 -- 172.21.15.134:0/2529691251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37a810b960 msgr2=0x7f37a81a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:25.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.245+0000 7f37a7e00640 1 --2- 172.21.15.134:0/2529691251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37a810b960 0x7f37a81a0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.245+0000 7f37a7e00640 1 -- 172.21.15.134:0/2529691251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37a8114d90 msgr2=0x7f37a81a5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:25.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.245+0000 7f37a7e00640 1 --2- 172.21.15.134:0/2529691251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37a8114d90 0x7f37a81a5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.245+0000 7f37a7e00640 1 -- 172.21.15.134:0/2529691251 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37a81a61f0 con 0x7f37a810c2b0 2024-07-26T11:22:25.248 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.245+0000 7f37a7e00640 1 --2- 172.21.15.134:0/2529691251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37a810c2b0 0x7f37a81a0820 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f379402e7c0 tx=0x7f37940042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:25.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.248+0000 7f37a5600640 1 -- 172.21.15.134:0/2529691251 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37940044e0 con 0x7f37a810c2b0 2024-07-26T11:22:25.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.248+0000 7f37a5600640 1 -- 172.21.15.134:0/2529691251 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3794005500 con 0x7f37a810c2b0 2024-07-26T11:22:25.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.248+0000 7f37a5600640 1 -- 172.21.15.134:0/2529691251 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3794004bd0 con 0x7f37a810c2b0 2024-07-26T11:22:25.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.248+0000 7f37ae200640 1 -- 172.21.15.134:0/2529691251 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37a81a6480 con 0x7f37a810c2b0 2024-07-26T11:22:25.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.248+0000 7f37ae200640 1 -- 172.21.15.134:0/2529691251 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37a81a68e0 con 0x7f37a810c2b0 2024-07-26T11:22:25.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.257+0000 7f37a5600640 1 -- 172.21.15.134:0/2529691251 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f3794046030 con 0x7f37a810c2b0 2024-07-26T11:22:25.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.259+0000 7f37a5600640 1 --2- 172.21.15.134:0/2529691251 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f378c0b3f10 0x7f378c0b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.260+0000 7f37a5600640 1 -- 172.21.15.134:0/2529691251 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f3794107920 con 0x7f37a810c2b0 2024-07-26T11:22:25.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.261+0000 7f37ace00640 1 --2- 172.21.15.134:0/2529691251 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f378c0b3f10 0x7f378c0b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:25.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.265+0000 7f37ace00640 1 --2- 172.21.15.134:0/2529691251 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f378c0b3f10 0x7f378c0b6420 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f37980058c0 tx=0x7f3798009260 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:25.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.344+0000 7f37b64bd740 1 Processor -- start 2024-07-26T11:22:25.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.346+0000 7f37b64bd740 1 -- start start 2024-07-26T11:22:25.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.346+0000 7f37b64bd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 0x55e5fd171030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.346+0000 7f37b64bd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd171570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.346+0000 7f37b64bd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdab390 0x55e5fd171ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.346+0000 7f37b64bd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e5fd0e7bf0 con 0x55e5fcdab390 2024-07-26T11:22:25.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.346+0000 7f37b64bd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e5fd0d70f0 con 0x55e5fcda31b0 2024-07-26T11:22:25.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.346+0000 7f37b64bd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e5fd0db130 con 0x55e5fcdcc070 2024-07-26T11:22:25.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.347+0000 7f37a7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd171570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:25.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.347+0000 7f37a7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd171570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:59098/0 (socket says 172.21.15.134:59098) 2024-07-26T11:22:25.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.347+0000 7f37a7e00640 1 -- 172.21.15.134:0/1726077163 learned_addr learned my addr 172.21.15.134:0/1726077163 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:25.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.347+0000 7f37ad800640 1 --2- 172.21.15.134:0/1726077163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdab390 0x55e5fd171ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:25.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.347+0000 7f37a7e00640 1 -- 172.21.15.134:0/1726077163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 msgr2=0x55e5fd171030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:22:25.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.347+0000 7f37a7e00640 1 --2- 172.21.15.134:0/1726077163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 0x55e5fd171030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.347+0000 7f37a7e00640 1 -- 172.21.15.134:0/1726077163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdab390 msgr2=0x55e5fd171ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:25.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.347+0000 7f37a7e00640 1 --2- 172.21.15.134:0/1726077163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdab390 0x55e5fd171ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.347+0000 7f37a7e00640 1 -- 172.21.15.134:0/1726077163 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5fce37650 con 0x55e5fcda31b0 2024-07-26T11:22:25.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.348+0000 7f37a7e00640 1 --2- 172.21.15.134:0/1726077163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd171570 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f37940ca370 tx=0x7f379402e2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0368879579c6190 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:25.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.348+0000 7f379e000640 1 -- 172.21.15.134:0/1726077163 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37940ce9f0 con 0x55e5fcda31b0 2024-07-26T11:22:25.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.349+0000 7f379e000640 1 -- 172.21.15.134:0/1726077163 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37940ce2f0 con 0x55e5fcda31b0 2024-07-26T11:22:25.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.349+0000 7f379e000640 1 -- 172.21.15.134:0/1726077163 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37940ce9f0 con 0x55e5fcda31b0 2024-07-26T11:22:25.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.351+0000 7f37b64bd740 1 -- 172.21.15.134:0/1726077163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 msgr2=0x55e5fd171570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:25.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.351+0000 7f37b64bd740 1 --2- 172.21.15.134:0/1726077163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd171570 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f37940ca370 tx=0x7f379402e2a0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.352+0000 7f37b64bd740 1 -- 172.21.15.134:0/1726077163 shutdown_connections 2024-07-26T11:22:25.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.352+0000 7f37b64bd740 1 --2- 172.21.15.134:0/1726077163 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdab390 0x55e5fd171ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.352+0000 7f37b64bd740 1 --2- 172.21.15.134:0/1726077163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd171570 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.352+0000 7f37b64bd740 1 --2- 172.21.15.134:0/1726077163 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 0x55e5fd171030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.352+0000 7f37b64bd740 1 -- 172.21.15.134:0/1726077163 >> 172.21.15.134:0/1726077163 conn(0x55e5fcdc0460 msgr2=0x55e5fd16ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:25.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.353+0000 7f37b64bd740 1 -- 172.21.15.134:0/1726077163 shutdown_connections 2024-07-26T11:22:25.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.353+0000 7f37b64bd740 1 -- 172.21.15.134:0/1726077163 wait complete. 2024-07-26T11:22:25.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.354+0000 7f37b64bd740 1 -- start start 2024-07-26T11:22:25.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.355+0000 7f37b64bd740 1 Processor -- start 2024-07-26T11:22:25.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.356+0000 7f37b64bd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd186b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.356+0000 7f37b64bd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdab390 0x55e5fd187120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.357+0000 7f37b64bd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 0x55e5fd17e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.357+0000 7f37b64bd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e5fcf3a700 con 0x55e5fcdab390 2024-07-26T11:22:25.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.357+0000 7f37b64bd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e5fd0736c0 con 0x55e5fcda31b0 2024-07-26T11:22:25.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.357+0000 7f37b64bd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e5fd17f490 con 0x55e5fcdcc070 2024-07-26T11:22:25.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.357+0000 7f37b64bd740 10 client.0 _ll_register_callbacks cb 0x55e5fd083170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:25.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.358+0000 7f37ace00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd186b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:25.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.358+0000 7f37ad800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 0x55e5fd17e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:25.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.358+0000 7f37ad800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 0x55e5fd17e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:50678/0 (socket says 172.21.15.134:50678) 2024-07-26T11:22:25.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.358+0000 7f37ad800640 1 -- 172.21.15.134:0/2895821477 learned_addr learned my addr 172.21.15.134:0/2895821477 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:25.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.359+0000 7f37a7e00640 1 --2- 172.21.15.134:0/2895821477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdab390 0x55e5fd187120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:25.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.359+0000 7f37ace00640 1 -- 172.21.15.134:0/2895821477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 msgr2=0x55e5fd17e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:25.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.359+0000 7f37ace00640 1 --2- 172.21.15.134:0/2895821477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 0x55e5fd17e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.359+0000 7f37ace00640 1 -- 172.21.15.134:0/2895821477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdab390 msgr2=0x55e5fd187120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:25.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.359+0000 7f37ace00640 1 --2- 172.21.15.134:0/2895821477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdab390 0x55e5fd187120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.359+0000 7f37ace00640 1 -- 172.21.15.134:0/2895821477 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5fd08a860 con 0x55e5fcda31b0 2024-07-26T11:22:25.368 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.366+0000 7f37ace00640 1 --2- 172.21.15.134:0/2895821477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd186b60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3798011ca0 tx=0x7f3798004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca0e76444eb65b67 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:25.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.366+0000 7f3792a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:22:25.369 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.367+0000 7f3792a00640 1 -- 172.21.15.134:0/2895821477 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3798004030 con 0x55e5fcda31b0 2024-07-26T11:22:25.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.367+0000 7f37b64bd740 1 -- 172.21.15.134:0/2895821477 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37940ce2f0 con 0x55e5fcda31b0 2024-07-26T11:22:25.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.367+0000 7f3792a00640 1 -- 172.21.15.134:0/2895821477 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f37980041d0 con 0x55e5fcda31b0 2024-07-26T11:22:25.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.367+0000 7f3792a00640 1 -- 172.21.15.134:0/2895821477 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f379802e050 con 0x55e5fcda31b0 2024-07-26T11:22:25.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.367+0000 7f37b64bd740 10 client.26417 fetch_fsmap learned FSMap version 32 2024-07-26T11:22:25.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.367+0000 7f37b64bd740 1 -- 172.21.15.134:0/2895821477 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e5fd177320 con 0x55e5fcda31b0 2024-07-26T11:22:25.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.368+0000 7f3792a00640 1 -- 172.21.15.134:0/2895821477 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37940ce2f0 con 0x55e5fcda31b0 2024-07-26T11:22:25.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.371+0000 7f3792a00640 1 -- 172.21.15.134:0/2895821477 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f3798005340 con 0x55e5fcda31b0 2024-07-26T11:22:25.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.371+0000 7f37b64bd740 10 client.26417 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:22:25.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.371+0000 7f37b64bd740 10 client.26417 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T11:22:25.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.371+0000 7f37b64bd740 20 client.26417 populate_metadata read hostname 'smithi134' 2024-07-26T11:22:25.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.371+0000 7f37b64bd740 1 --2- 172.21.15.134:0/2895821477 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55e5fcd8a9f0 0x55e5fd18e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.371+0000 7f37b64bd740 4 client.26417 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:22:25.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.371+0000 7f37b64bd740 1 -- 172.21.15.134:0/2895821477 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e5fd0e9b40 con 0x55e5fcd8a9f0 2024-07-26T11:22:25.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.372+0000 7f37a7e00640 1 --2- 172.21.15.134:0/2895821477 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55e5fcd8a9f0 0x55e5fd18e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:25.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.372+0000 7f37a7e00640 1 --2- 172.21.15.134:0/2895821477 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55e5fcd8a9f0 0x55e5fd18e630 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=28ca46aacca259ec server_cookie=52487914deea1a12 in_seq=0 out_seq=0 2024-07-26T11:22:25.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.372+0000 7f3792a00640 10 client.26417 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:22:25.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.373+0000 7f3792a00640 1 -- 172.21.15.134:0/2895821477 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e5fd0e9b40 con 0x55e5fcd8a9f0 2024-07-26T11:22:25.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.373+0000 7f3792a00640 10 client.26417 handle_command_reply: tid=0 2024-07-26T11:22:25.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.373+0000 7f37b64bd740 1 client.26417 shutdown 2024-07-26T11:22:25.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.374+0000 7f37b64bd740 1 -- 172.21.15.134:0/2895821477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 msgr2=0x55e5fd186b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:25.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.374+0000 7f37b64bd740 1 --2- 172.21.15.134:0/2895821477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd186b60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3798011ca0 tx=0x7f3798004730 comp rx=0 tx=0).stop 2024-07-26T11:22:25.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.375+0000 7f37b64bd740 1 -- 172.21.15.134:0/2895821477 shutdown_connections 2024-07-26T11:22:25.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.375+0000 7f37b64bd740 1 --2- 172.21.15.134:0/2895821477 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55e5fcd8a9f0 0x55e5fd18e630 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.375+0000 7f3792a00640 0 client.26417 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:22:25.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.375+0000 7f37b64bd740 1 --2- 172.21.15.134:0/2895821477 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 0x55e5fd17e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.375+0000 7f37b64bd740 1 --2- 172.21.15.134:0/2895821477 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdab390 0x55e5fd187120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.375+0000 7f37b64bd740 1 --2- 172.21.15.134:0/2895821477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd186b60 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.375+0000 7f37b64bd740 1 -- 172.21.15.134:0/2895821477 >> 172.21.15.134:0/2895821477 conn(0x55e5fcdc0460 msgr2=0x55e5fd1741c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:25.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.376+0000 7f37b64bd740 1 -- 172.21.15.134:0/2895821477 shutdown_connections 2024-07-26T11:22:25.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.376+0000 7f37b64bd740 1 -- 172.21.15.134:0/2895821477 wait complete. 2024-07-26T11:22:25.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.376+0000 7f37b64bd740 20 client.26417 trim_cache size 0 max 16384 2024-07-26T11:22:25.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.412+0000 7f37b64bd740 1 Processor -- start 2024-07-26T11:22:25.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.417+0000 7f37b64bd740 1 -- start start 2024-07-26T11:22:25.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.417+0000 7f37b64bd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd16dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.417+0000 7f37b64bd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdab390 0x55e5fd193da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.417+0000 7f37b64bd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdcc070 0x55e5fd1942e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.417+0000 7f37b64bd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f37a81b69c0 con 0x55e5fcdcc070 2024-07-26T11:22:25.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.417+0000 7f37b64bd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f37a81b8500 con 0x55e5fcda31b0 2024-07-26T11:22:25.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.417+0000 7f37b64bd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f37a81b7090 con 0x55e5fcdab390 2024-07-26T11:22:25.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.418+0000 7f37ace00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd16dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:25.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.418+0000 7f37a7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdab390 0x55e5fd193da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:25.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.418+0000 7f37ace00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd16dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:59118/0 (socket says 172.21.15.134:59118) 2024-07-26T11:22:25.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.418+0000 7f37a7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdab390 0x55e5fd193da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:50688/0 (socket says 172.21.15.134:50688) 2024-07-26T11:22:25.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.418+0000 7f37a7e00640 1 -- 172.21.15.134:0/3943259852 learned_addr learned my addr 172.21.15.134:0/3943259852 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:25.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.419+0000 7f37ace00640 1 -- 172.21.15.134:0/3943259852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdab390 msgr2=0x55e5fd193da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:25.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.419+0000 7f37ace00640 1 --2- 172.21.15.134:0/3943259852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdab390 0x55e5fd193da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.419+0000 7f37ace00640 1 -- 172.21.15.134:0/3943259852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdcc070 msgr2=0x55e5fd1942e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:25.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.419+0000 7f37ace00640 1 --2- 172.21.15.134:0/3943259852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdcc070 0x55e5fd1942e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.419+0000 7f37ace00640 1 -- 172.21.15.134:0/3943259852 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e5fd184e60 con 0x55e5fcda31b0 2024-07-26T11:22:25.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.419+0000 7f37a7e00640 1 --2- 172.21.15.134:0/3943259852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdab390 0x55e5fd193da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:22:25.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.420+0000 7f37ace00640 1 --2- 172.21.15.134:0/3943259852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd16dc70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3798011ca0 tx=0x7f379801c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99fa8eed399342a4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:25.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.420+0000 7f379d600640 1 -- 172.21.15.134:0/3943259852 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3798037070 con 0x55e5fcda31b0 2024-07-26T11:22:25.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.420+0000 7f379d600640 1 -- 172.21.15.134:0/3943259852 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37980044e0 con 0x55e5fcda31b0 2024-07-26T11:22:25.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.420+0000 7f379d600640 1 -- 172.21.15.134:0/3943259852 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f379800edd0 con 0x55e5fcda31b0 2024-07-26T11:22:25.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.424+0000 7f37b64bd740 1 -- 172.21.15.134:0/3943259852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 msgr2=0x55e5fd16dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:25.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.424+0000 7f37b64bd740 1 --2- 172.21.15.134:0/3943259852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd16dc70 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3798011ca0 tx=0x7f379801c750 comp rx=0 tx=0).stop 2024-07-26T11:22:25.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.425+0000 7f37b64bd740 1 -- 172.21.15.134:0/3943259852 shutdown_connections 2024-07-26T11:22:25.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.425+0000 7f37b64bd740 1 --2- 172.21.15.134:0/3943259852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdab390 0x55e5fd193da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.425+0000 7f37b64bd740 1 --2- 172.21.15.134:0/3943259852 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdcc070 0x55e5fd1942e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.425+0000 7f37b64bd740 1 --2- 172.21.15.134:0/3943259852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd16dc70 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.425+0000 7f37b64bd740 1 -- 172.21.15.134:0/3943259852 >> 172.21.15.134:0/3943259852 conn(0x55e5fcdc0460 msgr2=0x55e5fd16d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:25.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.426+0000 7f37b64bd740 1 -- 172.21.15.134:0/3943259852 shutdown_connections 2024-07-26T11:22:25.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.426+0000 7f37b64bd740 1 -- 172.21.15.134:0/3943259852 wait complete. 2024-07-26T11:22:25.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.426+0000 7f37b64bd740 1 -- start start 2024-07-26T11:22:25.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.427+0000 7f37b64bd740 1 Processor -- start 2024-07-26T11:22:25.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.429+0000 7f37b64bd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd1eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.430+0000 7f37b64bd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdab390 0x55e5fd1ef370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.430+0000 7f37b64bd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 0x55e5fd19f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.430+0000 7f37b64bd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f37a81b7e30 con 0x55e5fcdab390 2024-07-26T11:22:25.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.430+0000 7f37b64bd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f37a81b7760 con 0x55e5fcda31b0 2024-07-26T11:22:25.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.430+0000 7f37b64bd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f37a81b4a50 con 0x55e5fcdcc070 2024-07-26T11:22:25.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.430+0000 7f37b64bd740 10 client.0 _ll_register_callbacks cb 0x7f379801c840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:25.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.431+0000 7f37ad800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 0x55e5fd19f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:25.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.431+0000 7f37ad800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 0x55e5fd19f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:50690/0 (socket says 172.21.15.134:50690) 2024-07-26T11:22:25.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.431+0000 7f37ad800640 1 -- 172.21.15.134:0/1325620858 learned_addr learned my addr 172.21.15.134:0/1325620858 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:25.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.431+0000 7f37ace00640 1 --2- 172.21.15.134:0/1325620858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd1eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:25.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.432+0000 7f37ad800640 1 -- 172.21.15.134:0/1325620858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 msgr2=0x55e5fd1eedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:25.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.432+0000 7f37ad800640 1 --2- 172.21.15.134:0/1325620858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd1eedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.432+0000 7f37ad800640 1 -- 172.21.15.134:0/1325620858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdab390 msgr2=0x55e5fd1ef370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:25.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.432+0000 7f37ad800640 1 --2- 172.21.15.134:0/1325620858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdab390 0x55e5fd1ef370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.432+0000 7f37ad800640 1 -- 172.21.15.134:0/1325620858 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5fd19fe50 con 0x55e5fcdcc070 2024-07-26T11:22:25.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.433+0000 7f37ace00640 1 --2- 172.21.15.134:0/1325620858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd1eedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:22:25.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.434+0000 7f37ad800640 1 --2- 172.21.15.134:0/1325620858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 0x55e5fd19f7e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f37a00042f0 tx=0x7f37a0003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=529f353fda467fd9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:25.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.434+0000 7f3792a00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:22:25.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.434+0000 7f3792a00640 1 -- 172.21.15.134:0/1325620858 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a0023b50 con 0x55e5fcdcc070 2024-07-26T11:22:25.437 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.434+0000 7f3792a00640 1 -- 172.21.15.134:0/1325620858 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f37a0023cf0 con 0x55e5fcdcc070 2024-07-26T11:22:25.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.436+0000 7f37b64bd740 1 -- 172.21.15.134:0/1325620858 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37980044e0 con 0x55e5fcdcc070 2024-07-26T11:22:25.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.437+0000 7f3792a00640 1 -- 172.21.15.134:0/1325620858 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f37a0037050 con 0x55e5fcdcc070 2024-07-26T11:22:25.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.437+0000 7f37b64bd740 10 client.26425 fetch_fsmap learned FSMap version 32 2024-07-26T11:22:25.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.437+0000 7f37b64bd740 1 -- 172.21.15.134:0/1325620858 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e5fd1f65f0 con 0x55e5fcdcc070 2024-07-26T11:22:25.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.437+0000 7f3792a00640 1 -- 172.21.15.134:0/1325620858 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37a002c8b0 con 0x55e5fcdcc070 2024-07-26T11:22:25.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.438+0000 7f3792a00640 1 -- 172.21.15.134:0/1325620858 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f37a00215d0 con 0x55e5fcdcc070 2024-07-26T11:22:25.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.438+0000 7f37b64bd740 10 client.26425 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:22:25.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.438+0000 7f37b64bd740 10 client.26425 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T11:22:25.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.438+0000 7f37b64bd740 20 client.26425 populate_metadata read hostname 'smithi134' 2024-07-26T11:22:25.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.438+0000 7f37b64bd740 1 --2- 172.21.15.134:0/1325620858 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55e5fcd5fda0 0x55e5fd1f8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:25.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.438+0000 7f37b64bd740 4 client.26425 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:22:25.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.438+0000 7f37b64bd740 1 -- 172.21.15.134:0/1325620858 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f37980044e0 con 0x55e5fcd5fda0 2024-07-26T11:22:25.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.439+0000 7f37ace00640 1 --2- 172.21.15.134:0/1325620858 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55e5fcd5fda0 0x55e5fd1f8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:25.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.440+0000 7f37ace00640 1 --2- 172.21.15.134:0/1325620858 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55e5fcd5fda0 0x55e5fd1f8e30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=91bd2a6ced666b8b server_cookie=5e678607c3843dbf in_seq=0 out_seq=0 2024-07-26T11:22:25.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.440+0000 7f3792a00640 10 client.26425 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:22:25.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.441+0000 7f3792a00640 1 -- 172.21.15.134:0/1325620858 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f37980044e0 con 0x55e5fcd5fda0 2024-07-26T11:22:25.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.441+0000 7f3792a00640 10 client.26425 handle_command_reply: tid=0 2024-07-26T11:22:25.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.441+0000 7f37b64bd740 1 client.26425 shutdown 2024-07-26T11:22:25.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.441+0000 7f37b64bd740 1 -- 172.21.15.134:0/1325620858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 msgr2=0x55e5fd19f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:25.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.442+0000 7f37b64bd740 1 --2- 172.21.15.134:0/1325620858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 0x55e5fd19f7e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f37a00042f0 tx=0x7f37a0003f90 comp rx=0 tx=0).stop 2024-07-26T11:22:25.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.442+0000 7f37b64bd740 1 -- 172.21.15.134:0/1325620858 shutdown_connections 2024-07-26T11:22:25.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.442+0000 7f37b64bd740 1 --2- 172.21.15.134:0/1325620858 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x55e5fcd5fda0 0x55e5fd1f8e30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.442+0000 7f37b64bd740 1 --2- 172.21.15.134:0/1325620858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e5fcdcc070 0x55e5fd19f7e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.442+0000 7f37b64bd740 1 --2- 172.21.15.134:0/1325620858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e5fcdab390 0x55e5fd1ef370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.442+0000 7f37b64bd740 1 --2- 172.21.15.134:0/1325620858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e5fcda31b0 0x55e5fd1eedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.442+0000 7f37b64bd740 1 -- 172.21.15.134:0/1325620858 >> 172.21.15.134:0/1325620858 conn(0x55e5fcdc0460 msgr2=0x55e5fd1816b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:25.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.442+0000 7f3792a00640 0 client.26425 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:22:25.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.443+0000 7f37b64bd740 1 -- 172.21.15.134:0/1325620858 shutdown_connections 2024-07-26T11:22:25.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.443+0000 7f37b64bd740 1 -- 172.21.15.134:0/1325620858 wait complete. 2024-07-26T11:22:25.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.443+0000 7f37b64bd740 20 client.26425 trim_cache size 0 max 16384 2024-07-26T11:22:25.446 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:22:25.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.448+0000 7f37ae200640 1 -- 172.21.15.134:0/2529691251 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f378c0b3f10 msgr2=0x7f378c0b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:25.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.448+0000 7f37ae200640 1 --2- 172.21.15.134:0/2529691251 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f378c0b3f10 0x7f378c0b6420 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f37980058c0 tx=0x7f3798009260 comp rx=0 tx=0).stop 2024-07-26T11:22:25.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.448+0000 7f37ae200640 1 -- 172.21.15.134:0/2529691251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37a810c2b0 msgr2=0x7f37a81a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:25.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.448+0000 7f37ae200640 1 --2- 172.21.15.134:0/2529691251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37a810c2b0 0x7f37a81a0820 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f379402e7c0 tx=0x7f37940042f0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.449+0000 7f37ae200640 1 -- 172.21.15.134:0/2529691251 shutdown_connections 2024-07-26T11:22:25.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.449+0000 7f37ae200640 1 --2- 172.21.15.134:0/2529691251 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f378c0b3f10 0x7f378c0b6420 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.449+0000 7f37ae200640 1 --2- 172.21.15.134:0/2529691251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f37a8114d90 0x7f37a81a5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.449+0000 7f37ae200640 1 --2- 172.21.15.134:0/2529691251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37a810c2b0 0x7f37a81a0820 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.449+0000 7f37ae200640 1 --2- 172.21.15.134:0/2529691251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37a810b960 0x7f37a81a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:25.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.449+0000 7f37ae200640 1 -- 172.21.15.134:0/2529691251 >> 172.21.15.134:0/2529691251 conn(0x7f37a806e880 msgr2=0x7f37a8072890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:25.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.449+0000 7f37ae200640 1 -- 172.21.15.134:0/2529691251 shutdown_connections 2024-07-26T11:22:25.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:25.450+0000 7f37ae200640 1 -- 172.21.15.134:0/2529691251 wait complete. 2024-07-26T11:22:26.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:26 smithi193 ceph-mon[37473]: pgmap v674: 129 pgs: 129 active+clean; 46 GiB data, 97 GiB used, 976 GiB / 1.0 TiB avail; 32 KiB/s rd, 243 MiB/s wr, 1.84k op/s 2024-07-26T11:22:26.850 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:26 smithi134 ceph-mon[31966]: pgmap v674: 129 pgs: 129 active+clean; 46 GiB data, 97 GiB used, 976 GiB / 1.0 TiB avail; 32 KiB/s rd, 243 MiB/s wr, 1.84k op/s 2024-07-26T11:22:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:26 smithi189 ceph-mon[36790]: pgmap v674: 129 pgs: 129 active+clean; 46 GiB data, 97 GiB used, 976 GiB / 1.0 TiB avail; 32 KiB/s rd, 243 MiB/s wr, 1.84k op/s 2024-07-26T11:22:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:28 smithi134 ceph-mon[31966]: pgmap v675: 129 pgs: 129 active+clean; 47 GiB data, 98 GiB used, 975 GiB / 1.0 TiB avail; 60 KiB/s rd, 244 MiB/s wr, 1.83k op/s 2024-07-26T11:22:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:28 smithi189 ceph-mon[36790]: pgmap v675: 129 pgs: 129 active+clean; 47 GiB data, 98 GiB used, 975 GiB / 1.0 TiB avail; 60 KiB/s rd, 244 MiB/s wr, 1.83k op/s 2024-07-26T11:22:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:28 smithi193 ceph-mon[37473]: pgmap v675: 129 pgs: 129 active+clean; 47 GiB data, 98 GiB used, 975 GiB / 1.0 TiB avail; 60 KiB/s rd, 244 MiB/s wr, 1.83k op/s 2024-07-26T11:22:29.853 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.851+0000 7f1a7e000640 1 -- 172.21.15.134:0/1864632394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a78077370 msgr2=0x7f1a78077780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:29.854 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.851+0000 7f1a7e000640 1 --2- 172.21.15.134:0/1864632394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a78077370 0x7f1a78077780 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1a64009900 tx=0x7f1a6402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.853+0000 7f1a7e000640 1 -- 172.21.15.134:0/1864632394 shutdown_connections 2024-07-26T11:22:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.853+0000 7f1a7e000640 1 --2- 172.21.15.134:0/1864632394 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a78077cc0 0x7f1a78113b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.853+0000 7f1a7e000640 1 --2- 172.21.15.134:0/1864632394 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a781140b0 0x7f1a781164d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.853+0000 7f1a7e000640 1 --2- 172.21.15.134:0/1864632394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a78077370 0x7f1a78077780 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.855 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.853+0000 7f1a7e000640 1 -- 172.21.15.134:0/1864632394 >> 172.21.15.134:0/1864632394 conn(0x7f1a78101120 msgr2=0x7f1a78103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.857+0000 7f1a7e000640 1 -- 172.21.15.134:0/1864632394 shutdown_connections 2024-07-26T11:22:29.859 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.857+0000 7f1a7e000640 1 -- 172.21.15.134:0/1864632394 wait complete. 2024-07-26T11:22:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.858+0000 7f1a7e000640 1 Processor -- start 2024-07-26T11:22:29.860 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.859+0000 7f1a7e000640 1 -- start start 2024-07-26T11:22:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.859+0000 7f1a7e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a78077370 0x7f1a78113a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.859+0000 7f1a7e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a78077cc0 0x7f1a7810dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.859+0000 7f1a7e000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a781140b0 0x7f1a7810e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.861 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.859+0000 7f1a7e000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1a78105c20 con 0x7f1a781140b0 2024-07-26T11:22:29.862 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.859+0000 7f1a7e000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1a78105aa0 con 0x7f1a78077370 2024-07-26T11:22:29.862 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.859+0000 7f1a7e000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1a78105da0 con 0x7f1a78077cc0 2024-07-26T11:22:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.860+0000 7f1a76a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a78077cc0 0x7f1a7810dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.860+0000 7f1a76a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a78077cc0 0x7f1a7810dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:39578/0 (socket says 172.21.15.134:39578) 2024-07-26T11:22:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.860+0000 7f1a76a00640 1 -- 172.21.15.134:0/3241811923 learned_addr learned my addr 172.21.15.134:0/3241811923 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.860+0000 7f1a77400640 1 --2- 172.21.15.134:0/3241811923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a78077370 0x7f1a78113a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.860+0000 7f1a76a00640 1 -- 172.21.15.134:0/3241811923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a78077370 msgr2=0x7f1a78113a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.860+0000 7f1a76a00640 1 --2- 172.21.15.134:0/3241811923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a78077370 0x7f1a78113a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.863 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.860+0000 7f1a76a00640 1 -- 172.21.15.134:0/3241811923 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a781140b0 msgr2=0x7f1a7810e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.860+0000 7f1a76a00640 1 --2- 172.21.15.134:0/3241811923 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a781140b0 0x7f1a7810e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.860+0000 7f1a76a00640 1 -- 172.21.15.134:0/3241811923 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a781b4f50 con 0x7f1a78077cc0 2024-07-26T11:22:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.861+0000 7f1a77400640 1 --2- 172.21.15.134:0/3241811923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a78077370 0x7f1a78113a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:22:29.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.861+0000 7f1a76a00640 1 --2- 172.21.15.134:0/3241811923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a78077cc0 0x7f1a7810dcc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1a6800e9c0 tx=0x7f1a6800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:29.866 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.864+0000 7f1a6fe00640 1 -- 172.21.15.134:0/3241811923 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6800ccc0 con 0x7f1a78077cc0 2024-07-26T11:22:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.864+0000 7f1a6fe00640 1 -- 172.21.15.134:0/3241811923 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a680054d0 con 0x7f1a78077cc0 2024-07-26T11:22:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.864+0000 7f1a6fe00640 1 -- 172.21.15.134:0/3241811923 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a68010430 con 0x7f1a78077cc0 2024-07-26T11:22:29.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.866+0000 7f1a7e000640 1 -- 172.21.15.134:0/3241811923 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a781b5120 con 0x7f1a78077cc0 2024-07-26T11:22:29.869 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.866+0000 7f1a7e000640 1 -- 172.21.15.134:0/3241811923 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a781b5610 con 0x7f1a78077cc0 2024-07-26T11:22:29.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.875+0000 7f1a8506f740 1 Processor -- start 2024-07-26T11:22:29.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.877+0000 7f1a6fe00640 1 -- 172.21.15.134:0/3241811923 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f1a68005000 con 0x7f1a78077cc0 2024-07-26T11:22:29.879 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.877+0000 7f1a8506f740 1 -- start start 2024-07-26T11:22:29.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.879+0000 7f1a8506f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd908ae1b0 0x55fd90c51a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.880+0000 7f1a8506f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd908852f0 0x55fd90c51f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.880+0000 7f1a8506f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9088d4d0 0x55fd90c524c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.880+0000 7f1a8506f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55fd90bc6270 con 0x55fd908852f0 2024-07-26T11:22:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.880+0000 7f1a8506f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fd90bb5770 con 0x55fd9088d4d0 2024-07-26T11:22:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.880+0000 7f1a8506f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fd90bb97b0 con 0x55fd908ae1b0 2024-07-26T11:22:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.880+0000 7f1a77e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd908ae1b0 0x55fd90c51a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.880+0000 7f1a77e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd908ae1b0 0x55fd90c51a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:39592/0 (socket says 172.21.15.134:39592) 2024-07-26T11:22:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.880+0000 7f1a77e00640 1 -- 172.21.15.134:0/3724256594 learned_addr learned my addr 172.21.15.134:0/3724256594 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.881+0000 7f1a77400640 1 --2- 172.21.15.134:0/3724256594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd908852f0 0x55fd90c51f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.882+0000 7f1a77400640 1 -- 172.21.15.134:0/3724256594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd908ae1b0 msgr2=0x55fd90c51a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.882+0000 7f1a77400640 1 --2- 172.21.15.134:0/3724256594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd908ae1b0 0x55fd90c51a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.882+0000 7f1a77400640 1 -- 172.21.15.134:0/3724256594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9088d4d0 msgr2=0x55fd90c524c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:22:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.882+0000 7f1a77400640 1 --2- 172.21.15.134:0/3724256594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9088d4d0 0x55fd90c524c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.882+0000 7f1a77400640 1 -- 172.21.15.134:0/3724256594 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd90919790 con 0x55fd908852f0 2024-07-26T11:22:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.882+0000 7f1a77e00640 1 --2- 172.21.15.134:0/3724256594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd908ae1b0 0x55fd90c51a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:22:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.883+0000 7f1a77400640 1 --2- 172.21.15.134:0/3724256594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd908852f0 0x55fd90c51f80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1a6402a9e0 tx=0x7f1a64004310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da176bed070c251 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.883+0000 7f1a6fe00640 1 --2- 172.21.15.134:0/3241811923 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1a5c0b3f50 0x7f1a5c0b6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.884+0000 7f1a6fe00640 1 -- 172.21.15.134:0/3241811923 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f1a680e47e0 con 0x7f1a78077cc0 2024-07-26T11:22:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.885+0000 7f1a77e00640 1 --2- 172.21.15.134:0/3241811923 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1a5c0b3f50 0x7f1a5c0b6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:29.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.885+0000 7f1a63e00640 1 -- 172.21.15.134:0/3724256594 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a64035030 con 0x55fd908852f0 2024-07-26T11:22:29.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.886+0000 7f1a77e00640 1 --2- 172.21.15.134:0/3241811923 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1a5c0b3f50 0x7f1a5c0b6460 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1a70007480 tx=0x7f1a70009690 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:29.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.886+0000 7f1a63e00640 1 -- 172.21.15.134:0/3724256594 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a640046c0 con 0x55fd908852f0 2024-07-26T11:22:29.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.887+0000 7f1a8506f740 1 -- 172.21.15.134:0/3724256594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd908852f0 msgr2=0x55fd90c51f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.889+0000 7f1a63e00640 1 -- 172.21.15.134:0/3724256594 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a64003850 con 0x55fd908852f0 2024-07-26T11:22:29.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.887+0000 7f1a8506f740 1 --2- 172.21.15.134:0/3724256594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd908852f0 0x55fd90c51f80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1a6402a9e0 tx=0x7f1a64004310 comp rx=0 tx=0).stop 2024-07-26T11:22:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.890+0000 7f1a8506f740 1 -- 172.21.15.134:0/3724256594 shutdown_connections 2024-07-26T11:22:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.890+0000 7f1a8506f740 1 --2- 172.21.15.134:0/3724256594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9088d4d0 0x55fd90c524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.890+0000 7f1a8506f740 1 --2- 172.21.15.134:0/3724256594 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd908852f0 0x55fd90c51f80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.890+0000 7f1a8506f740 1 --2- 172.21.15.134:0/3724256594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd908ae1b0 0x55fd90c51a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.892 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.890+0000 7f1a8506f740 1 -- 172.21.15.134:0/3724256594 >> 172.21.15.134:0/3724256594 conn(0x55fd908a2300 msgr2=0x55fd90c4f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.890+0000 7f1a8506f740 1 -- 172.21.15.134:0/3724256594 shutdown_connections 2024-07-26T11:22:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.890+0000 7f1a8506f740 1 -- 172.21.15.134:0/3724256594 wait complete. 2024-07-26T11:22:29.893 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.891+0000 7f1a8506f740 1 -- start start 2024-07-26T11:22:29.894 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.892+0000 7f1a8506f740 1 Processor -- start 2024-07-26T11:22:29.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.893+0000 7f1a8506f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd908852f0 0x55fd90c5d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.893+0000 7f1a8506f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd9088d4d0 0x55fd90c5daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.893+0000 7f1a8506f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd908ae1b0 0x55fd90c69540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.893+0000 7f1a8506f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55fd90aefab0 con 0x55fd908852f0 2024-07-26T11:22:29.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.893+0000 7f1a8506f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fd90b52d50 con 0x55fd908ae1b0 2024-07-26T11:22:29.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.893+0000 7f1a8506f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fd90c60200 con 0x55fd9088d4d0 2024-07-26T11:22:29.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.893+0000 7f1a8506f740 10 client.0 _ll_register_callbacks cb 0x55fd908c5a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:29.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.894+0000 7f1a76a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd9088d4d0 0x55fd90c5daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:29.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.894+0000 7f1a76a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd9088d4d0 0x55fd90c5daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:39608/0 (socket says 172.21.15.134:39608) 2024-07-26T11:22:29.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.894+0000 7f1a76a00640 1 -- 172.21.15.134:0/670588753 learned_addr learned my addr 172.21.15.134:0/670588753 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:29.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.894+0000 7f1a76a00640 1 -- 172.21.15.134:0/670588753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd908ae1b0 msgr2=0x55fd90c69540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:22:29.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.895+0000 7f1a76a00640 1 --2- 172.21.15.134:0/670588753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd908ae1b0 0x55fd90c69540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.895+0000 7f1a76a00640 1 -- 172.21.15.134:0/670588753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd908852f0 msgr2=0x55fd90c5d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:29.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.895+0000 7f1a76a00640 1 --2- 172.21.15.134:0/670588753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd908852f0 0x55fd90c5d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.895+0000 7f1a76a00640 1 -- 172.21.15.134:0/670588753 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd90992620 con 0x55fd9088d4d0 2024-07-26T11:22:29.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.895+0000 7f1a76a00640 1 --2- 172.21.15.134:0/670588753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd9088d4d0 0x55fd90c5daf0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1a6800c6a0 tx=0x7f1a68005740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83ddf4afb260dc09 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:29.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.895+0000 7f1a60c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:22:29.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.898+0000 7f1a60c00640 1 -- 172.21.15.134:0/670588753 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a680c4020 con 0x55fd9088d4d0 2024-07-26T11:22:29.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.898+0000 7f1a60c00640 1 -- 172.21.15.134:0/670588753 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f1a680e1dd0 con 0x55fd9088d4d0 2024-07-26T11:22:29.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.901+0000 7f1a8506f740 1 -- 172.21.15.134:0/670588753 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a640046c0 con 0x55fd9088d4d0 2024-07-26T11:22:29.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.902+0000 7f1a60c00640 1 -- 172.21.15.134:0/670588753 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1a680e1fd0 con 0x55fd9088d4d0 2024-07-26T11:22:29.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.902+0000 7f1a8506f740 10 client.26437 fetch_fsmap learned FSMap version 32 2024-07-26T11:22:29.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.902+0000 7f1a8506f740 1 -- 172.21.15.134:0/670588753 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fd90c7d280 con 0x55fd9088d4d0 2024-07-26T11:22:29.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.903+0000 7f1a60c00640 1 -- 172.21.15.134:0/670588753 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a640046c0 con 0x55fd9088d4d0 2024-07-26T11:22:29.909 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.908+0000 7f1a60c00640 1 -- 172.21.15.134:0/670588753 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f1a680e43b0 con 0x55fd9088d4d0 2024-07-26T11:22:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.908+0000 7f1a8506f740 10 client.26437 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:22:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.908+0000 7f1a8506f740 10 client.26437 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:22:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.908+0000 7f1a8506f740 20 client.26437 populate_metadata read hostname 'smithi134' 2024-07-26T11:22:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.908+0000 7f1a8506f740 1 --2- 172.21.15.134:0/670588753 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55fd9086cd80 0x55fd90c807a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.908+0000 7f1a8506f740 4 client.26437 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:22:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.908+0000 7f1a8506f740 1 -- 172.21.15.134:0/670588753 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd90bc81c0 con 0x55fd9086cd80 2024-07-26T11:22:29.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.909+0000 7f1a77400640 1 --2- 172.21.15.134:0/670588753 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55fd9086cd80 0x55fd90c807a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:29.913 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.910+0000 7f1a77400640 1 --2- 172.21.15.134:0/670588753 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55fd9086cd80 0x55fd90c807a0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f613c42f3f36edc server_cookie=9710eabfd3068200 in_seq=0 out_seq=0 2024-07-26T11:22:29.913 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.911+0000 7f1a60c00640 10 client.26437 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:22:29.915 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.913+0000 7f1a60c00640 1 -- 172.21.15.134:0/670588753 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fd90bc81c0 con 0x55fd9086cd80 2024-07-26T11:22:29.915 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.913+0000 7f1a60c00640 10 client.26437 handle_command_reply: tid=0 2024-07-26T11:22:29.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.913+0000 7f1a8506f740 1 client.26437 shutdown 2024-07-26T11:22:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.914+0000 7f1a8506f740 1 -- 172.21.15.134:0/670588753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd9088d4d0 msgr2=0x55fd90c5daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.914+0000 7f1a8506f740 1 --2- 172.21.15.134:0/670588753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd9088d4d0 0x55fd90c5daf0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f1a6800c6a0 tx=0x7f1a68005740 comp rx=0 tx=0).stop 2024-07-26T11:22:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.915+0000 7f1a8506f740 1 -- 172.21.15.134:0/670588753 shutdown_connections 2024-07-26T11:22:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.915+0000 7f1a8506f740 1 --2- 172.21.15.134:0/670588753 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55fd9086cd80 0x55fd90c807a0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.915+0000 7f1a8506f740 1 --2- 172.21.15.134:0/670588753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd9088d4d0 0x55fd90c5daf0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.915+0000 7f1a60c00640 0 client.26437 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:22:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.915+0000 7f1a8506f740 1 --2- 172.21.15.134:0/670588753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd908ae1b0 0x55fd90c69540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.915+0000 7f1a8506f740 1 --2- 172.21.15.134:0/670588753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd908852f0 0x55fd90c5d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.915+0000 7f1a8506f740 1 -- 172.21.15.134:0/670588753 >> 172.21.15.134:0/670588753 conn(0x55fd908a2300 msgr2=0x55fd90c4e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.918+0000 7f1a8506f740 1 -- 172.21.15.134:0/670588753 shutdown_connections 2024-07-26T11:22:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.918+0000 7f1a8506f740 1 -- 172.21.15.134:0/670588753 wait complete. 2024-07-26T11:22:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.918+0000 7f1a8506f740 20 client.26437 trim_cache size 0 max 16384 2024-07-26T11:22:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.964+0000 7f1a8506f740 1 Processor -- start 2024-07-26T11:22:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.965+0000 7f1a8506f740 1 -- start start 2024-07-26T11:22:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.965+0000 7f1a8506f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd9086cd80 0x55fd90c5cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.965+0000 7f1a8506f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd908852f0 0x55fd90c57b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.965+0000 7f1a8506f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9088d4d0 0x55fd90c69330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.966+0000 7f1a8506f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1a781b6e70 con 0x55fd9086cd80 2024-07-26T11:22:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.966+0000 7f1a8506f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1a781b89b0 con 0x55fd9088d4d0 2024-07-26T11:22:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.966+0000 7f1a8506f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1a781b7540 con 0x55fd908852f0 2024-07-26T11:22:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.966+0000 7f1a77e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9088d4d0 0x55fd90c69330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.966+0000 7f1a77e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9088d4d0 0x55fd90c69330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46532/0 (socket says 172.21.15.134:46532) 2024-07-26T11:22:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.966+0000 7f1a77e00640 1 -- 172.21.15.134:0/2608929212 learned_addr learned my addr 172.21.15.134:0/2608929212 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.966+0000 7f1a77400640 1 --2- 172.21.15.134:0/2608929212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd9086cd80 0x55fd90c5cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.967+0000 7f1a77e00640 1 -- 172.21.15.134:0/2608929212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd908852f0 msgr2=0x55fd90c57b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:22:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.967+0000 7f1a77e00640 1 --2- 172.21.15.134:0/2608929212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd908852f0 0x55fd90c57b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.967+0000 7f1a77e00640 1 -- 172.21.15.134:0/2608929212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd9086cd80 msgr2=0x55fd90c5cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.967+0000 7f1a77e00640 1 --2- 172.21.15.134:0/2608929212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd9086cd80 0x55fd90c5cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.967+0000 7f1a77e00640 1 -- 172.21.15.134:0/2608929212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd90c5b880 con 0x55fd9088d4d0 2024-07-26T11:22:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.967+0000 7f1a77e00640 1 --2- 172.21.15.134:0/2608929212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9088d4d0 0x55fd90c69330 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1a700396d0 tx=0x7f1a70011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa9c640af86a5f61 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.968+0000 7f1a63400640 1 -- 172.21.15.134:0/2608929212 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a70049070 con 0x55fd9088d4d0 2024-07-26T11:22:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.968+0000 7f1a63400640 1 -- 172.21.15.134:0/2608929212 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a700159a0 con 0x55fd9088d4d0 2024-07-26T11:22:29.973 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.968+0000 7f1a63400640 1 -- 172.21.15.134:0/2608929212 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a70044ba0 con 0x55fd9088d4d0 2024-07-26T11:22:29.973 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.969+0000 7f1a8506f740 1 -- 172.21.15.134:0/2608929212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9088d4d0 msgr2=0x55fd90c69330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:29.973 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.969+0000 7f1a8506f740 1 --2- 172.21.15.134:0/2608929212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9088d4d0 0x55fd90c69330 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1a700396d0 tx=0x7f1a70011000 comp rx=0 tx=0).stop 2024-07-26T11:22:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.971+0000 7f1a8506f740 1 -- 172.21.15.134:0/2608929212 shutdown_connections 2024-07-26T11:22:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.971+0000 7f1a8506f740 1 --2- 172.21.15.134:0/2608929212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd908852f0 0x55fd90c57b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.971+0000 7f1a8506f740 1 --2- 172.21.15.134:0/2608929212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9088d4d0 0x55fd90c69330 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.971+0000 7f1a8506f740 1 --2- 172.21.15.134:0/2608929212 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd9086cd80 0x55fd90c5cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.971+0000 7f1a8506f740 1 -- 172.21.15.134:0/2608929212 >> 172.21.15.134:0/2608929212 conn(0x55fd908a2300 msgr2=0x55fd90c5c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:29.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.972+0000 7f1a8506f740 1 -- 172.21.15.134:0/2608929212 shutdown_connections 2024-07-26T11:22:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.972+0000 7f1a8506f740 1 -- 172.21.15.134:0/2608929212 wait complete. 2024-07-26T11:22:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.973+0000 7f1a8506f740 1 -- start start 2024-07-26T11:22:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.973+0000 7f1a8506f740 1 Processor -- start 2024-07-26T11:22:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.975+0000 7f1a8506f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd9086cd80 0x55fd90c61350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.975+0000 7f1a8506f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd908852f0 0x55fd90c61890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.975+0000 7f1a8506f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9088d4d0 0x55fd90c61dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.975+0000 7f1a77400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd9086cd80 0x55fd90c61350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.975+0000 7f1a77400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd9086cd80 0x55fd90c61350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:39620/0 (socket says 172.21.15.134:39620) 2024-07-26T11:22:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.976+0000 7f1a77400640 1 -- 172.21.15.134:0/3281477203 learned_addr learned my addr 172.21.15.134:0/3281477203 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.976+0000 7f1a8506f740 1 -- 172.21.15.134:0/3281477203 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1a781b82e0 con 0x55fd908852f0 2024-07-26T11:22:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.976+0000 7f1a8506f740 1 -- 172.21.15.134:0/3281477203 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1a781b7c10 con 0x55fd9088d4d0 2024-07-26T11:22:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.976+0000 7f1a8506f740 1 -- 172.21.15.134:0/3281477203 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1a78074540 con 0x55fd9086cd80 2024-07-26T11:22:29.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.976+0000 7f1a77e00640 1 --2- 172.21.15.134:0/3281477203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9088d4d0 0x55fd90c61dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.976+0000 7f1a8506f740 10 client.0 _ll_register_callbacks cb 0x7f1a680096c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:29.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.977+0000 7f1a76a00640 1 --2- 172.21.15.134:0/3281477203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd908852f0 0x55fd90c61890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.979+0000 7f1a76a00640 1 -- 172.21.15.134:0/3281477203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd9086cd80 msgr2=0x55fd90c61350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.979+0000 7f1a76a00640 1 --2- 172.21.15.134:0/3281477203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd9086cd80 0x55fd90c61350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.979+0000 7f1a76a00640 1 -- 172.21.15.134:0/3281477203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9088d4d0 msgr2=0x55fd90c61dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.979+0000 7f1a76a00640 1 --2- 172.21.15.134:0/3281477203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9088d4d0 0x55fd90c61dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.979+0000 7f1a76a00640 1 -- 172.21.15.134:0/3281477203 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd90ce7f70 con 0x55fd908852f0 2024-07-26T11:22:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.980+0000 7f1a60c00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:22:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.980+0000 7f1a76a00640 1 --2- 172.21.15.134:0/3281477203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd908852f0 0x55fd90c61890 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1a680054a0 tx=0x7f1a680e2000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83772a617b247714 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:29.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.981+0000 7f1a60c00640 1 -- 172.21.15.134:0/3281477203 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6800ce60 con 0x55fd908852f0 2024-07-26T11:22:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.981+0000 7f1a60c00640 1 -- 172.21.15.134:0/3281477203 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f1a680b3e40 con 0x55fd908852f0 2024-07-26T11:22:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.981+0000 7f1a8506f740 1 -- 172.21.15.134:0/3281477203 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a700159a0 con 0x55fd908852f0 2024-07-26T11:22:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.982+0000 7f1a60c00640 1 -- 172.21.15.134:0/3281477203 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1a680ab060 con 0x55fd908852f0 2024-07-26T11:22:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.982+0000 7f1a8506f740 10 client.15993 fetch_fsmap learned FSMap version 32 2024-07-26T11:22:29.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.982+0000 7f1a8506f740 1 -- 172.21.15.134:0/3281477203 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fd90ce91d0 con 0x55fd908852f0 2024-07-26T11:22:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.985+0000 7f1a60c00640 1 -- 172.21.15.134:0/3281477203 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a700159a0 con 0x55fd908852f0 2024-07-26T11:22:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.986+0000 7f1a60c00640 1 -- 172.21.15.134:0/3281477203 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f1a680a6980 con 0x55fd908852f0 2024-07-26T11:22:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.986+0000 7f1a8506f740 10 client.15993 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:22:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.986+0000 7f1a8506f740 10 client.15993 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:22:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.986+0000 7f1a8506f740 20 client.15993 populate_metadata read hostname 'smithi134' 2024-07-26T11:22:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.986+0000 7f1a8506f740 1 --2- 172.21.15.134:0/3281477203 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55fd908ae1b0 0x55fd90ceb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.987+0000 7f1a8506f740 4 client.15993 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:22:29.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.987+0000 7f1a8506f740 1 -- 172.21.15.134:0/3281477203 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fd90bc3c60 con 0x55fd908ae1b0 2024-07-26T11:22:29.990 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.987+0000 7f1a77400640 1 --2- 172.21.15.134:0/3281477203 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55fd908ae1b0 0x55fd90ceb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:29.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.993+0000 7f1a60c00640 10 client.15993 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:22:29.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:29.993+0000 7f1a77400640 1 --2- 172.21.15.134:0/3281477203 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55fd908ae1b0 0x55fd90ceb7b0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7c2098268f8577f server_cookie=7ef84ff5c4b3c998 in_seq=0 out_seq=0 2024-07-26T11:22:30.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.364+0000 7f1a60c00640 1 -- 172.21.15.134:0/3281477203 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fd90bc3c60 con 0x55fd908ae1b0 2024-07-26T11:22:30.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.364+0000 7f1a60c00640 10 client.15993 handle_command_reply: tid=0 2024-07-26T11:22:30.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.364+0000 7f1a8506f740 1 client.15993 shutdown 2024-07-26T11:22:30.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.365+0000 7f1a8506f740 1 -- 172.21.15.134:0/3281477203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd908852f0 msgr2=0x55fd90c61890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:30.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.365+0000 7f1a8506f740 1 --2- 172.21.15.134:0/3281477203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd908852f0 0x55fd90c61890 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1a680054a0 tx=0x7f1a680e2000 comp rx=0 tx=0).stop 2024-07-26T11:22:30.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.365+0000 7f1a8506f740 1 -- 172.21.15.134:0/3281477203 shutdown_connections 2024-07-26T11:22:30.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.365+0000 7f1a8506f740 1 --2- 172.21.15.134:0/3281477203 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55fd908ae1b0 0x55fd90ceb7b0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.365+0000 7f1a8506f740 1 --2- 172.21.15.134:0/3281477203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fd9088d4d0 0x55fd90c61dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.365+0000 7f1a8506f740 1 --2- 172.21.15.134:0/3281477203 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fd908852f0 0x55fd90c61890 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.365+0000 7f1a8506f740 1 --2- 172.21.15.134:0/3281477203 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fd9086cd80 0x55fd90c61350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.365+0000 7f1a8506f740 1 -- 172.21.15.134:0/3281477203 >> 172.21.15.134:0/3281477203 conn(0x55fd908a2300 msgr2=0x55fd90c4f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.365+0000 7f1a60c00640 0 client.15993 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:22:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.366+0000 7f1a8506f740 1 -- 172.21.15.134:0/3281477203 shutdown_connections 2024-07-26T11:22:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.366+0000 7f1a8506f740 1 -- 172.21.15.134:0/3281477203 wait complete. 2024-07-26T11:22:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.368+0000 7f1a8506f740 20 client.15993 trim_cache size 0 max 16384 2024-07-26T11:22:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:22:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:22:30.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:22:30.372 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:22:30.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.378+0000 7f1a7e000640 1 -- 172.21.15.134:0/3241811923 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1a5c0b3f50 msgr2=0x7f1a5c0b6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:30.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.378+0000 7f1a7e000640 1 --2- 172.21.15.134:0/3241811923 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1a5c0b3f50 0x7f1a5c0b6460 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1a70007480 tx=0x7f1a70009690 comp rx=0 tx=0).stop 2024-07-26T11:22:30.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.378+0000 7f1a7e000640 1 -- 172.21.15.134:0/3241811923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a78077cc0 msgr2=0x7f1a7810dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:30.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.378+0000 7f1a7e000640 1 --2- 172.21.15.134:0/3241811923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a78077cc0 0x7f1a7810dcc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1a6800e9c0 tx=0x7f1a6800ee80 comp rx=0 tx=0).stop 2024-07-26T11:22:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.380+0000 7f1a7e000640 1 -- 172.21.15.134:0/3241811923 shutdown_connections 2024-07-26T11:22:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.380+0000 7f1a7e000640 1 --2- 172.21.15.134:0/3241811923 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1a5c0b3f50 0x7f1a5c0b6460 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.380+0000 7f1a7e000640 1 --2- 172.21.15.134:0/3241811923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a78077cc0 0x7f1a7810dcc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.380+0000 7f1a7e000640 1 --2- 172.21.15.134:0/3241811923 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1a781140b0 0x7f1a7810e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.380+0000 7f1a7e000640 1 --2- 172.21.15.134:0/3241811923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a78077370 0x7f1a78113a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:30.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.380+0000 7f1a7e000640 1 -- 172.21.15.134:0/3241811923 >> 172.21.15.134:0/3241811923 conn(0x7f1a78101120 msgr2=0x7f1a78116130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:30.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.381+0000 7f1a7e000640 1 -- 172.21.15.134:0/3241811923 shutdown_connections 2024-07-26T11:22:30.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:30.381+0000 7f1a7e000640 1 -- 172.21.15.134:0/3241811923 wait complete. 2024-07-26T11:22:30.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:30 smithi134 ceph-mon[31966]: pgmap v676: 129 pgs: 129 active+clean; 48 GiB data, 99 GiB used, 973 GiB / 1.0 TiB avail; 241 KiB/s rd, 264 MiB/s wr, 1.89k op/s 2024-07-26T11:22:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:30 smithi189 ceph-mon[36790]: pgmap v676: 129 pgs: 129 active+clean; 48 GiB data, 99 GiB used, 973 GiB / 1.0 TiB avail; 241 KiB/s rd, 264 MiB/s wr, 1.89k op/s 2024-07-26T11:22:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:30 smithi193 ceph-mon[37473]: pgmap v676: 129 pgs: 129 active+clean; 48 GiB data, 99 GiB used, 973 GiB / 1.0 TiB avail; 241 KiB/s rd, 264 MiB/s wr, 1.89k op/s 2024-07-26T11:22:31.428 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-26T11:22:32.601 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:22:32.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:32 smithi134 ceph-mon[31966]: pgmap v677: 129 pgs: 129 active+clean; 48 GiB data, 100 GiB used, 973 GiB / 1.0 TiB avail; 231 KiB/s rd, 202 MiB/s wr, 1.32k op/s 2024-07-26T11:22:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:32 smithi189 ceph-mon[36790]: pgmap v677: 129 pgs: 129 active+clean; 48 GiB data, 100 GiB used, 973 GiB / 1.0 TiB avail; 231 KiB/s rd, 202 MiB/s wr, 1.32k op/s 2024-07-26T11:22:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:32 smithi193 ceph-mon[37473]: pgmap v677: 129 pgs: 129 active+clean; 48 GiB data, 100 GiB used, 973 GiB / 1.0 TiB avail; 231 KiB/s rd, 202 MiB/s wr, 1.32k op/s 2024-07-26T11:22:34.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:34 smithi134 ceph-mon[31966]: pgmap v678: 129 pgs: 129 active+clean; 48 GiB data, 101 GiB used, 971 GiB / 1.0 TiB avail; 251 KiB/s rd, 259 MiB/s wr, 1.66k op/s 2024-07-26T11:22:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:34 smithi189 ceph-mon[36790]: pgmap v678: 129 pgs: 129 active+clean; 48 GiB data, 101 GiB used, 971 GiB / 1.0 TiB avail; 251 KiB/s rd, 259 MiB/s wr, 1.66k op/s 2024-07-26T11:22:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:34 smithi193 ceph-mon[37473]: pgmap v678: 129 pgs: 129 active+clean; 48 GiB data, 101 GiB used, 971 GiB / 1.0 TiB avail; 251 KiB/s rd, 259 MiB/s wr, 1.66k op/s 2024-07-26T11:22:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:36 smithi134 ceph-mon[31966]: pgmap v679: 129 pgs: 129 active+clean; 49 GiB data, 102 GiB used, 971 GiB / 1.0 TiB avail; 231 KiB/s rd, 244 MiB/s wr, 1.57k op/s 2024-07-26T11:22:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:36 smithi189 ceph-mon[36790]: pgmap v679: 129 pgs: 129 active+clean; 49 GiB data, 102 GiB used, 971 GiB / 1.0 TiB avail; 231 KiB/s rd, 244 MiB/s wr, 1.57k op/s 2024-07-26T11:22:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:36 smithi193 ceph-mon[37473]: pgmap v679: 129 pgs: 129 active+clean; 49 GiB data, 102 GiB used, 971 GiB / 1.0 TiB avail; 231 KiB/s rd, 244 MiB/s wr, 1.57k op/s 2024-07-26T11:22:38.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:38 smithi134 ceph-mon[31966]: pgmap v680: 129 pgs: 129 active+clean; 49 GiB data, 103 GiB used, 969 GiB / 1.0 TiB avail; 247 KiB/s rd, 250 MiB/s wr, 1.58k op/s 2024-07-26T11:22:38.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:38 smithi189 ceph-mon[36790]: pgmap v680: 129 pgs: 129 active+clean; 49 GiB data, 103 GiB used, 969 GiB / 1.0 TiB avail; 247 KiB/s rd, 250 MiB/s wr, 1.58k op/s 2024-07-26T11:22:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:38 smithi193 ceph-mon[37473]: pgmap v680: 129 pgs: 129 active+clean; 49 GiB data, 103 GiB used, 969 GiB / 1.0 TiB avail; 247 KiB/s rd, 250 MiB/s wr, 1.58k op/s 2024-07-26T11:22:40.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:40 smithi134 ceph-mon[31966]: pgmap v681: 129 pgs: 129 active+clean; 50 GiB data, 105 GiB used, 968 GiB / 1.0 TiB avail; 220 KiB/s rd, 283 MiB/s wr, 1.64k op/s 2024-07-26T11:22:40.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:40 smithi189 ceph-mon[36790]: pgmap v681: 129 pgs: 129 active+clean; 50 GiB data, 105 GiB used, 968 GiB / 1.0 TiB avail; 220 KiB/s rd, 283 MiB/s wr, 1.64k op/s 2024-07-26T11:22:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:40 smithi193 ceph-mon[37473]: pgmap v681: 129 pgs: 129 active+clean; 50 GiB data, 105 GiB used, 968 GiB / 1.0 TiB avail; 220 KiB/s rd, 283 MiB/s wr, 1.64k op/s 2024-07-26T11:22:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.844+0000 7fe008200640 1 -- 172.21.15.134:0/2538768023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe00010a920 msgr2=0x7fe000115a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.844+0000 7fe008200640 1 --2- 172.21.15.134:0/2538768023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe00010a920 0x7fe000115a00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdfec009900 tx=0x7fdfec02a410 comp rx=0 tx=0).stop 2024-07-26T11:22:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.868+0000 7fe008200640 1 -- 172.21.15.134:0/2538768023 shutdown_connections 2024-07-26T11:22:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.868+0000 7fe008200640 1 --2- 172.21.15.134:0/2538768023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe00010a920 0x7fe000115a00 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.868+0000 7fe008200640 1 --2- 172.21.15.134:0/2538768023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe000115fd0 0x7fe0001184b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.868+0000 7fe008200640 1 --2- 172.21.15.134:0/2538768023 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe000109fd0 0x7fe00010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.870 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.868+0000 7fe008200640 1 -- 172.21.15.134:0/2538768023 >> 172.21.15.134:0/2538768023 conn(0x7fe000070800 msgr2=0x7fe000070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:43.872 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.868+0000 7fe008200640 1 -- 172.21.15.134:0/2538768023 shutdown_connections 2024-07-26T11:22:43.872 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.868+0000 7fe008200640 1 -- 172.21.15.134:0/2538768023 wait complete. 2024-07-26T11:22:43.872 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.870+0000 7fe008200640 1 Processor -- start 2024-07-26T11:22:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.877+0000 7fe008200640 1 -- start start 2024-07-26T11:22:43.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.881+0000 7fe008200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe000109fd0 0x7fe0001a26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.882+0000 7fe008200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe00010a920 0x7fe0001a2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.883+0000 7fe005800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe000109fd0 0x7fe0001a26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:43.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.883+0000 7fe005800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe000109fd0 0x7fe0001a26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:60504/0 (socket says 172.21.15.134:60504) 2024-07-26T11:22:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.883+0000 7fe004e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe00010a920 0x7fe0001a2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.883+0000 7fe004e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe00010a920 0x7fe0001a2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34390/0 (socket says 172.21.15.134:34390) 2024-07-26T11:22:43.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.883+0000 7fe004e00640 1 -- 172.21.15.134:0/2260346647 learned_addr learned my addr 172.21.15.134:0/2260346647 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:43.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.884+0000 7fe008200640 1 --2- 172.21.15.134:0/2260346647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe000115fd0 0x7fe0001a7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.885+0000 7fe008200640 1 -- 172.21.15.134:0/2260346647 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe00011bcd0 con 0x7fe000115fd0 2024-07-26T11:22:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.886+0000 7fe008200640 1 -- 172.21.15.134:0/2260346647 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe00011bb50 con 0x7fe00010a920 2024-07-26T11:22:43.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.886+0000 7fe008200640 1 -- 172.21.15.134:0/2260346647 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe00011be50 con 0x7fe000109fd0 2024-07-26T11:22:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.888+0000 7fe005800640 1 -- 172.21.15.134:0/2260346647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe00010a920 msgr2=0x7fe0001a2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.888+0000 7fe005800640 1 --2- 172.21.15.134:0/2260346647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe00010a920 0x7fe0001a2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.888+0000 7fe005800640 1 -- 172.21.15.134:0/2260346647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe000115fd0 msgr2=0x7fe0001a7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:43.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.888+0000 7fe004e00640 1 --2- 172.21.15.134:0/2260346647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe00010a920 0x7fe0001a2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:22:43.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.888+0000 7fe005800640 1 --2- 172.21.15.134:0/2260346647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe000115fd0 0x7fe0001a7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.888+0000 7fe005800640 1 -- 172.21.15.134:0/2260346647 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0001a85e0 con 0x7fe000109fd0 2024-07-26T11:22:43.892 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.889+0000 7fe005800640 1 --2- 172.21.15.134:0/2260346647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe000109fd0 0x7fe0001a26b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fdff4011f80 tx=0x7fdff4009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:43.893 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.890+0000 7fdffe000640 1 -- 172.21.15.134:0/2260346647 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff401e070 con 0x7fe000109fd0 2024-07-26T11:22:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.891+0000 7fe008200640 1 -- 172.21.15.134:0/2260346647 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0001a88d0 con 0x7fe000109fd0 2024-07-26T11:22:43.894 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.891+0000 7fe008200640 1 -- 172.21.15.134:0/2260346647 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe00010b790 con 0x7fe000109fd0 2024-07-26T11:22:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.892+0000 7fdffe000640 1 -- 172.21.15.134:0/2260346647 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdff401ad00 con 0x7fe000109fd0 2024-07-26T11:22:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.892+0000 7fdffe000640 1 -- 172.21.15.134:0/2260346647 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff401e070 con 0x7fe000109fd0 2024-07-26T11:22:43.908 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:43 smithi134 ceph-mon[31966]: pgmap v682: 129 pgs: 129 active+clean; 50 GiB data, 105 GiB used, 968 GiB / 1.0 TiB avail; 37 KiB/s rd, 224 MiB/s wr, 1.20k op/s 2024-07-26T11:22:43.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.904+0000 7fce1cc00640 1 -- 172.21.15.134:0/188104121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce1810b310 msgr2=0x7fce1810b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:43.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.904+0000 7fce1cc00640 1 --2- 172.21.15.134:0/188104121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce1810b310 0x7fce1810b720 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fce0800dc90 tx=0x7fce0802e390 comp rx=0 tx=0).stop 2024-07-26T11:22:43.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.905+0000 7fce1cc00640 1 -- 172.21.15.134:0/188104121 shutdown_connections 2024-07-26T11:22:43.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.905+0000 7fce1cc00640 1 --2- 172.21.15.134:0/188104121 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce180790b0 0x7fce1807b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.905+0000 7fce1cc00640 1 --2- 172.21.15.134:0/188104121 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce1810f740 0x7fce1810fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.905+0000 7fce1cc00640 1 --2- 172.21.15.134:0/188104121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce1810b310 0x7fce1810b720 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.905+0000 7fce1cc00640 1 -- 172.21.15.134:0/188104121 >> 172.21.15.134:0/188104121 conn(0x7fce1806e420 msgr2=0x7fce1806e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.908+0000 7fdffe000640 1 -- 172.21.15.134:0/2260346647 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fdff40110e0 con 0x7fe000109fd0 2024-07-26T11:22:43.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.907+0000 7fce1cc00640 1 -- 172.21.15.134:0/188104121 shutdown_connections 2024-07-26T11:22:43.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.907+0000 7fce1cc00640 1 -- 172.21.15.134:0/188104121 wait complete. 2024-07-26T11:22:43.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.909+0000 7fce1cc00640 1 Processor -- start 2024-07-26T11:22:43.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.910+0000 7fce1cc00640 1 -- start start 2024-07-26T11:22:43.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.911+0000 7fce1cc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce180790b0 0x7fce18080110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.913+0000 7fce1cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce1810b310 0x7fce18080650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.913+0000 7fce1cc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce1810f740 0x7fce18080bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.914+0000 7fe00f1f1740 1 Processor -- start 2024-07-26T11:22:43.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.914+0000 7fdffe000640 1 --2- 172.21.15.134:0/2260346647 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdfe80b4070 0x7fdfe80b6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.914+0000 7fce1cc00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fce1807e620 con 0x7fce180790b0 2024-07-26T11:22:43.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.914+0000 7fce1cc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fce1807e4a0 con 0x7fce1810b310 2024-07-26T11:22:43.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.914+0000 7fce1cc00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fce1807e7a0 con 0x7fce1810f740 2024-07-26T11:22:43.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.914+0000 7fce16a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce1810b310 0x7fce18080650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:43.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.914+0000 7fce17e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce1810f740 0x7fce18080bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.914+0000 7fdffe000640 1 -- 172.21.15.134:0/2260346647 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fdff40ec010 con 0x7fe000109fd0 2024-07-26T11:22:43.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.914+0000 7fce17e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce1810f740 0x7fce18080bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:60520/0 (socket says 172.21.15.134:60520) 2024-07-26T11:22:43.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.914+0000 7fce17e00640 1 -- 172.21.15.134:0/903632314 learned_addr learned my addr 172.21.15.134:0/903632314 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:43.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.914+0000 7fce16a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce1810b310 0x7fce18080650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34404/0 (socket says 172.21.15.134:34404) 2024-07-26T11:22:43.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.915+0000 7fce17e00640 1 -- 172.21.15.134:0/903632314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce1810b310 msgr2=0x7fce18080650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:43.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.915+0000 7fce17e00640 1 --2- 172.21.15.134:0/903632314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce1810b310 0x7fce18080650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.915+0000 7fce17e00640 1 -- 172.21.15.134:0/903632314 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce180790b0 msgr2=0x7fce18080110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:43.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.915+0000 7fce17e00640 1 --2- 172.21.15.134:0/903632314 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce180790b0 0x7fce18080110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.915+0000 7fce17e00640 1 -- 172.21.15.134:0/903632314 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce181c5ba0 con 0x7fce1810f740 2024-07-26T11:22:43.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.916+0000 7fce17e00640 1 --2- 172.21.15.134:0/903632314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce1810f740 0x7fce18080bd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fce1000cc60 tx=0x7fce1000a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.923+0000 7fe00f1f1740 1 -- start start 2024-07-26T11:22:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.923+0000 7fe00f1f1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d9c1b0 0x55a28413fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.923+0000 7fe00f1f1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d732f0 0x55a28413ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.923+0000 7fe00f1f1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d7b4d0 0x55a2841404e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.923+0000 7fe00f1f1740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a2840b4270 con 0x55a283d732f0 2024-07-26T11:22:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.923+0000 7fe00f1f1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a2840a3770 con 0x55a283d7b4d0 2024-07-26T11:22:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.923+0000 7fe00f1f1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a2840a77b0 con 0x55a283d9c1b0 2024-07-26T11:22:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.924+0000 7fe006200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d7b4d0 0x55a2841404e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:43.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.924+0000 7fe006200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d7b4d0 0x55a2841404e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34408/0 (socket says 172.21.15.134:34408) 2024-07-26T11:22:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.924+0000 7fe006200640 1 -- 172.21.15.134:0/209534591 learned_addr learned my addr 172.21.15.134:0/209534591 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.924+0000 7fe006200640 1 -- 172.21.15.134:0/209534591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d9c1b0 msgr2=0x55a28413fa60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:22:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.924+0000 7fe006200640 1 --2- 172.21.15.134:0/209534591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d9c1b0 0x55a28413fa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.924+0000 7fe006200640 1 -- 172.21.15.134:0/209534591 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d732f0 msgr2=0x55a28413ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:22:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.924+0000 7fe006200640 1 --2- 172.21.15.134:0/209534591 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d732f0 0x55a28413ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.924+0000 7fe006200640 1 -- 172.21.15.134:0/209534591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a283e07790 con 0x55a283d7b4d0 2024-07-26T11:22:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.925+0000 7fe006200640 1 --2- 172.21.15.134:0/209534591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d7b4d0 0x55a2841404e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdff8009ec0 tx=0x7fdff800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f8a0ce5c281ce7d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.925+0000 7fe006200640 1 --2- 172.21.15.134:0/2260346647 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdfe80b4070 0x7fdfe80b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.926+0000 7fdff2a00640 1 -- 172.21.15.134:0/209534591 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff8011720 con 0x55a283d7b4d0 2024-07-26T11:22:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.929+0000 7fdff2a00640 1 -- 172.21.15.134:0/209534591 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdff8011d70 con 0x55a283d7b4d0 2024-07-26T11:22:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.930+0000 7fe00f1f1740 1 -- 172.21.15.134:0/209534591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d7b4d0 msgr2=0x55a2841404e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.930+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/209534591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d7b4d0 0x55a2841404e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fdff8009ec0 tx=0x7fdff800f730 comp rx=0 tx=0).stop 2024-07-26T11:22:43.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.932+0000 7fdff2a00640 1 -- 172.21.15.134:0/209534591 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff800d3e0 con 0x55a283d7b4d0 2024-07-26T11:22:43.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.933+0000 7fe006200640 1 --2- 172.21.15.134:0/2260346647 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdfe80b4070 0x7fdfe80b6580 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdff8009ec0 tx=0x7fdff800f730 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:43 smithi189 ceph-mon[36790]: pgmap v682: 129 pgs: 129 active+clean; 50 GiB data, 105 GiB used, 968 GiB / 1.0 TiB avail; 37 KiB/s rd, 224 MiB/s wr, 1.20k op/s 2024-07-26T11:22:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.945+0000 7fe00f1f1740 1 -- 172.21.15.134:0/209534591 shutdown_connections 2024-07-26T11:22:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.945+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/209534591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d7b4d0 0x55a2841404e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.946+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/209534591 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d732f0 0x55a28413ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.946+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/209534591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d9c1b0 0x55a28413fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.946+0000 7fe00f1f1740 1 -- 172.21.15.134:0/209534591 >> 172.21.15.134:0/209534591 conn(0x55a283d90300 msgr2=0x55a28413d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:43.948 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.946+0000 7fe00f1f1740 1 -- 172.21.15.134:0/209534591 shutdown_connections 2024-07-26T11:22:43.949 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.946+0000 7fe00f1f1740 1 -- 172.21.15.134:0/209534591 wait complete. 2024-07-26T11:22:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.948+0000 7fe00f1f1740 1 -- start start 2024-07-26T11:22:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.948+0000 7fe00f1f1740 1 Processor -- start 2024-07-26T11:22:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.949+0000 7fe00f1f1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 0x55a28414b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.950+0000 7fe00f1f1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d7b4d0 0x55a28414bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.950+0000 7fe005800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 0x55a28414b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:43.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.950+0000 7fe005800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 0x55a28414b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:60532/0 (socket says 172.21.15.134:60532) 2024-07-26T11:22:43.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.950+0000 7fe005800640 1 -- 172.21.15.134:0/2932459571 learned_addr learned my addr 172.21.15.134:0/2932459571 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:43.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.950+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/2932459571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d9c1b0 0x55a284151540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.950+0000 7fe00f1f1740 1 -- 172.21.15.134:0/2932459571 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a283fddab0 con 0x55a283d7b4d0 2024-07-26T11:22:43.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.951+0000 7fe00f1f1740 1 -- 172.21.15.134:0/2932459571 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a284040d50 con 0x55a283d9c1b0 2024-07-26T11:22:43.954 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.951+0000 7fe00f1f1740 1 -- 172.21.15.134:0/2932459571 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a28414e230 con 0x55a283d732f0 2024-07-26T11:22:43.954 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.951+0000 7fe006200640 1 --2- 172.21.15.134:0/2932459571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d9c1b0 0x55a284151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:43.955 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.951+0000 7fe005800640 1 -- 172.21.15.134:0/2932459571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d9c1b0 msgr2=0x55a284151540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.951+0000 7fe005800640 1 --2- 172.21.15.134:0/2932459571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d9c1b0 0x55a284151540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.951+0000 7fe005800640 1 -- 172.21.15.134:0/2932459571 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d7b4d0 msgr2=0x55a28414bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.951+0000 7fe005800640 1 --2- 172.21.15.134:0/2932459571 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d7b4d0 0x55a28414bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.952+0000 7fe005800640 1 -- 172.21.15.134:0/2932459571 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a283e80620 con 0x55a283d732f0 2024-07-26T11:22:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.952+0000 7fe005800640 1 --2- 172.21.15.134:0/2932459571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 0x55a28414b6d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdff40ae8c0 tx=0x7fdff40b2200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58f070993f207ab3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.953+0000 7fe00f1f1740 10 client.0 _ll_register_callbacks cb 0x55a283db3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.953+0000 7fe00f1f1740 1 -- 172.21.15.134:0/2932459571 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdff8011d70 con 0x55a283d732f0 2024-07-26T11:22:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.952+0000 7fdfe7400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:22:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.954+0000 7fdfe7400640 1 -- 172.21.15.134:0/2932459571 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff4027070 con 0x55a283d732f0 2024-07-26T11:22:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.954+0000 7fdfe7400640 1 -- 172.21.15.134:0/2932459571 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fdff40d52f0 con 0x55a283d732f0 2024-07-26T11:22:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.954+0000 7fdfe7400640 1 -- 172.21.15.134:0/2932459571 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdff40bb230 con 0x55a283d732f0 2024-07-26T11:22:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.954+0000 7fe00f1f1740 10 client.26452 fetch_fsmap learned FSMap version 32 2024-07-26T11:22:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.954+0000 7fe00f1f1740 1 -- 172.21.15.134:0/2932459571 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=32}) -- 0x55a28415a300 con 0x55a283d732f0 2024-07-26T11:22:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.955+0000 7fdfe7400640 1 -- 172.21.15.134:0/2932459571 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fdff40b3b70 con 0x55a283d732f0 2024-07-26T11:22:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.955+0000 7fe00f1f1740 10 client.26452 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:22:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.955+0000 7fe00f1f1740 10 client.26452 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:22:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.955+0000 7fe00f1f1740 20 client.26452 populate_metadata read hostname 'smithi134' 2024-07-26T11:22:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.955+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/2932459571 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a283d5ad80 0x55a28415dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.955+0000 7fe00f1f1740 4 client.26452 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:22:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.955+0000 7fe00f1f1740 1 -- 172.21.15.134:0/2932459571 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2840b61c0 con 0x55a283d5ad80 2024-07-26T11:22:43.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.956+0000 7fe004e00640 1 --2- 172.21.15.134:0/2932459571 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a283d5ad80 0x55a28415dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:43.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.957+0000 7fe004e00640 1 --2- 172.21.15.134:0/2932459571 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a283d5ad80 0x55a28415dc40 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef29bd537df63c18 server_cookie=f78fb43618c7d89 in_seq=0 out_seq=0 2024-07-26T11:22:43.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.957+0000 7fdfe7400640 10 client.26452 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:22:43.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.958+0000 7fdfe7400640 1 -- 172.21.15.134:0/2932459571 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55a2840b61c0 con 0x55a283d5ad80 2024-07-26T11:22:43.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.958+0000 7fdfe7400640 10 client.26452 handle_command_reply: tid=0 2024-07-26T11:22:43.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.958+0000 7fe00f1f1740 1 client.26452 shutdown 2024-07-26T11:22:43.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.958+0000 7fe00f1f1740 1 -- 172.21.15.134:0/2932459571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 msgr2=0x55a28414b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:43.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.958+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/2932459571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 0x55a28414b6d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdff40ae8c0 tx=0x7fdff40b2200 comp rx=0 tx=0).stop 2024-07-26T11:22:43.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.916+0000 7fce0fe00640 1 -- 172.21.15.134:0/903632314 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce10003b70 con 0x7fce1810f740 2024-07-26T11:22:43.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.916+0000 7fce0fe00640 1 -- 172.21.15.134:0/903632314 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce10003d10 con 0x7fce1810f740 2024-07-26T11:22:43.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.916+0000 7fce0fe00640 1 -- 172.21.15.134:0/903632314 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1000f810 con 0x7fce1810f740 2024-07-26T11:22:43.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.921+0000 7fce1cc00640 1 -- 172.21.15.134:0/903632314 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce181c5e90 con 0x7fce1810f740 2024-07-26T11:22:43.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.921+0000 7fce1cc00640 1 -- 172.21.15.134:0/903632314 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce181c6370 con 0x7fce1810f740 2024-07-26T11:22:43.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.932+0000 7fce0fe00640 1 -- 172.21.15.134:0/903632314 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fce1000f9b0 con 0x7fce1810f740 2024-07-26T11:22:43.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.935+0000 7fce0fe00640 1 --2- 172.21.15.134:0/903632314 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcdfc0b3e40 0x7fcdfc0b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.935+0000 7fce0fe00640 1 -- 172.21.15.134:0/903632314 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fce100e4850 con 0x7fce1810f740 2024-07-26T11:22:43.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.939+0000 7fce17400640 1 --2- 172.21.15.134:0/903632314 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcdfc0b3e40 0x7fcdfc0b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:43.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.940+0000 7fce17400640 1 --2- 172.21.15.134:0/903632314 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcdfc0b3e40 0x7fcdfc0b6350 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fce0802e980 tx=0x7fce0800d580 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.963+0000 7fe00f1f1740 1 -- 172.21.15.134:0/2932459571 shutdown_connections 2024-07-26T11:22:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.963+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/2932459571 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a283d5ad80 0x55a28415dc40 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.964+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/2932459571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d9c1b0 0x55a284151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.964+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/2932459571 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d7b4d0 0x55a28414bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.964+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/2932459571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 0x55a28414b6d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.964+0000 7fe00f1f1740 1 -- 172.21.15.134:0/2932459571 >> 172.21.15.134:0/2932459571 conn(0x55a283d90300 msgr2=0x55a28413c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:43.966 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.964+0000 7fdfe7400640 0 client.26452 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:22:43.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.982+0000 7fce25008740 1 Processor -- start 2024-07-26T11:22:43.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.982+0000 7fce25008740 1 -- start start 2024-07-26T11:22:43.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.983+0000 7fce25008740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb3a8070 0x55dccb74d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.983+0000 7fce25008740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb37f1b0 0x55dccb74d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.983+0000 7fce17400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb3a8070 0x55dccb74d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:43.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.983+0000 7fce17400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb3a8070 0x55dccb74d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34428/0 (socket says 172.21.15.134:34428) 2024-07-26T11:22:43.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.983+0000 7fce25008740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb387390 0x55dccb74dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:43.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.984+0000 7fce25008740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55dccb6c3bf0 con 0x55dccb387390 2024-07-26T11:22:43.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.984+0000 7fce25008740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dccb6b30f0 con 0x55dccb3a8070 2024-07-26T11:22:43.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.984+0000 7fce25008740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55dccb6b7130 con 0x55dccb37f1b0 2024-07-26T11:22:43.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.984+0000 7fce17400640 1 -- 172.21.15.134:0/3633928719 learned_addr learned my addr 172.21.15.134:0/3633928719 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:43.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.984+0000 7fce16a00640 1 --2- 172.21.15.134:0/3633928719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb37f1b0 0x55dccb74d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.986+0000 7fe00f1f1740 1 -- 172.21.15.134:0/2932459571 shutdown_connections 2024-07-26T11:22:43.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.986+0000 7fce16a00640 1 -- 172.21.15.134:0/3633928719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb3a8070 msgr2=0x55dccb74d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:43.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.986+0000 7fce16a00640 1 --2- 172.21.15.134:0/3633928719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb3a8070 0x55dccb74d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.986+0000 7fce16a00640 1 -- 172.21.15.134:0/3633928719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb387390 msgr2=0x55dccb74dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:43.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.986+0000 7fce16a00640 1 --2- 172.21.15.134:0/3633928719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb387390 0x55dccb74dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.986+0000 7fce16a00640 1 -- 172.21.15.134:0/3633928719 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dccb413650 con 0x55dccb37f1b0 2024-07-26T11:22:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.988+0000 7fe00f1f1740 1 -- 172.21.15.134:0/2932459571 wait complete. 2024-07-26T11:22:43.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.988+0000 7fce16a00640 1 --2- 172.21.15.134:0/3633928719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb37f1b0 0x55dccb74d570 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fce1806c0b0 tx=0x7fce0400bbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b213db69a3072cd9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:43.989+0000 7fe00f1f1740 20 client.26452 trim_cache size 0 max 16384 2024-07-26T11:22:43.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.988+0000 7fce03e00640 1 -- 172.21.15.134:0/3633928719 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0400be10 con 0x55dccb37f1b0 2024-07-26T11:22:43.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.989+0000 7fce03e00640 1 -- 172.21.15.134:0/3633928719 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce04019070 con 0x55dccb37f1b0 2024-07-26T11:22:43.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.989+0000 7fce03e00640 1 -- 172.21.15.134:0/3633928719 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04014da0 con 0x55dccb37f1b0 2024-07-26T11:22:43.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.990+0000 7fce25008740 1 -- 172.21.15.134:0/3633928719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb37f1b0 msgr2=0x55dccb74d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:43.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.990+0000 7fce25008740 1 --2- 172.21.15.134:0/3633928719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb37f1b0 0x55dccb74d570 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fce1806c0b0 tx=0x7fce0400bbc0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.993+0000 7fce25008740 1 -- 172.21.15.134:0/3633928719 shutdown_connections 2024-07-26T11:22:43.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.994+0000 7fce25008740 1 --2- 172.21.15.134:0/3633928719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb37f1b0 0x55dccb74d570 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.994+0000 7fce25008740 1 --2- 172.21.15.134:0/3633928719 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb387390 0x55dccb74dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.994+0000 7fce25008740 1 --2- 172.21.15.134:0/3633928719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb3a8070 0x55dccb74d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:43.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.994+0000 7fce25008740 1 -- 172.21.15.134:0/3633928719 >> 172.21.15.134:0/3633928719 conn(0x55dccb39c460 msgr2=0x55dccb74af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:44.003 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.996+0000 7fce25008740 1 -- 172.21.15.134:0/3633928719 shutdown_connections 2024-07-26T11:22:44.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.996+0000 7fce25008740 1 -- 172.21.15.134:0/3633928719 wait complete. 2024-07-26T11:22:44.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.997+0000 7fce25008740 1 -- start start 2024-07-26T11:22:44.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:43.998+0000 7fce25008740 1 Processor -- start 2024-07-26T11:22:44.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.000+0000 7fce25008740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb37f1b0 0x55dccb758680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.000+0000 7fce25008740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb387390 0x55dccb758bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.001+0000 7fce25008740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb3a8070 0x55dccb764b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.002+0000 7fce16a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb387390 0x55dccb758bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:44.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.002+0000 7fce17400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb37f1b0 0x55dccb758680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:44.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.002+0000 7fce16a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb387390 0x55dccb758bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34430/0 (socket says 172.21.15.134:34430) 2024-07-26T11:22:44.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.002+0000 7fce16a00640 1 -- 172.21.15.134:0/3192267727 learned_addr learned my addr 172.21.15.134:0/3192267727 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:44.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.002+0000 7fce25008740 1 -- 172.21.15.134:0/3192267727 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55dccb516700 con 0x55dccb37f1b0 2024-07-26T11:22:44.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.002+0000 7fce17e00640 1 --2- 172.21.15.134:0/3192267727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb3a8070 0x55dccb764b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:44.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.002+0000 7fce25008740 1 -- 172.21.15.134:0/3192267727 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55dccb64f6c0 con 0x55dccb387390 2024-07-26T11:22:44.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.002+0000 7fce25008740 1 -- 172.21.15.134:0/3192267727 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55dccb75b810 con 0x55dccb3a8070 2024-07-26T11:22:44.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.005+0000 7fce25008740 10 client.0 _ll_register_callbacks cb 0x55dccb65f170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:44.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.005+0000 7fce17e00640 1 -- 172.21.15.134:0/3192267727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb387390 msgr2=0x55dccb758bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:44.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.005+0000 7fce17e00640 1 --2- 172.21.15.134:0/3192267727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb387390 0x55dccb758bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.005+0000 7fce17e00640 1 -- 172.21.15.134:0/3192267727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb37f1b0 msgr2=0x55dccb758680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:44.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.005+0000 7fce17e00640 1 --2- 172.21.15.134:0/3192267727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb37f1b0 0x55dccb758680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.005+0000 7fce17e00640 1 -- 172.21.15.134:0/3192267727 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dccb666860 con 0x55dccb3a8070 2024-07-26T11:22:44.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.006+0000 7fce17e00640 1 --2- 172.21.15.134:0/3192267727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb3a8070 0x55dccb764b00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fce100e1bc0 tx=0x7fce10010fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a651ed08000488e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:44.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.006+0000 7fce00c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:22:44.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.006+0000 7fce00c00640 1 -- 172.21.15.134:0/3192267727 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce100c4020 con 0x55dccb3a8070 2024-07-26T11:22:44.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.006+0000 7fce00c00640 1 -- 172.21.15.134:0/3192267727 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fce100e4390 con 0x55dccb3a8070 2024-07-26T11:22:44.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.007+0000 7fce25008740 1 -- 172.21.15.134:0/3192267727 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce04019070 con 0x55dccb3a8070 2024-07-26T11:22:44.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.007+0000 7fce00c00640 1 -- 172.21.15.134:0/3192267727 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fce100e2050 con 0x55dccb3a8070 2024-07-26T11:22:44.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.007+0000 7fce25008740 10 client.26458 fetch_fsmap learned FSMap version 32 2024-07-26T11:22:44.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.007+0000 7fce25008740 1 -- 172.21.15.134:0/3192267727 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=32}) -- 0x55dccb7673c0 con 0x55dccb3a8070 2024-07-26T11:22:44.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.007+0000 7fce00c00640 1 -- 172.21.15.134:0/3192267727 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fce100cc0d0 con 0x55dccb3a8070 2024-07-26T11:22:44.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.009+0000 7fce25008740 10 client.26458 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:22:44.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.009+0000 7fce25008740 10 client.26458 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T11:22:44.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.009+0000 7fce25008740 20 client.26458 populate_metadata read hostname 'smithi134' 2024-07-26T11:22:44.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.010+0000 7fce25008740 1 --2- 172.21.15.134:0/3192267727 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55dccb3669f0 0x55dccb76a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.010+0000 7fce25008740 4 client.26458 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:22:44.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.010+0000 7fce25008740 1 -- 172.21.15.134:0/3192267727 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dccb6c5b40 con 0x55dccb3669f0 2024-07-26T11:22:44.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.012+0000 7fce17400640 1 --2- 172.21.15.134:0/3192267727 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55dccb3669f0 0x55dccb76a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:44.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.013+0000 7fce17400640 1 --2- 172.21.15.134:0/3192267727 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55dccb3669f0 0x55dccb76a720 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=83176b8ce0d04e1f server_cookie=87b45f7606ff50da in_seq=0 out_seq=0 2024-07-26T11:22:44.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.013+0000 7fce00c00640 10 client.26458 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:22:44.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.015+0000 7fce00c00640 1 -- 172.21.15.134:0/3192267727 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55dccb6c5b40 con 0x55dccb3669f0 2024-07-26T11:22:44.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.015+0000 7fce00c00640 10 client.26458 handle_command_reply: tid=0 2024-07-26T11:22:44.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.015+0000 7fce25008740 1 client.26458 shutdown 2024-07-26T11:22:44.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.018+0000 7fce25008740 1 -- 172.21.15.134:0/3192267727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb3a8070 msgr2=0x55dccb764b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:44.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.018+0000 7fce25008740 1 --2- 172.21.15.134:0/3192267727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb3a8070 0x55dccb764b00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fce100e1bc0 tx=0x7fce10010fb0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.018+0000 7fce25008740 1 -- 172.21.15.134:0/3192267727 shutdown_connections 2024-07-26T11:22:44.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.018+0000 7fce25008740 1 --2- 172.21.15.134:0/3192267727 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55dccb3669f0 0x55dccb76a720 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.018+0000 7fce25008740 1 --2- 172.21.15.134:0/3192267727 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb3a8070 0x55dccb764b00 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.018+0000 7fce25008740 1 --2- 172.21.15.134:0/3192267727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb387390 0x55dccb758bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.018+0000 7fce25008740 1 --2- 172.21.15.134:0/3192267727 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb37f1b0 0x55dccb758680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.018+0000 7fce25008740 1 -- 172.21.15.134:0/3192267727 >> 172.21.15.134:0/3192267727 conn(0x55dccb39c460 msgr2=0x55dccb750210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:44.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.018+0000 7fce00c00640 0 client.26458 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:22:44.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.018+0000 7fce25008740 1 -- 172.21.15.134:0/3192267727 shutdown_connections 2024-07-26T11:22:44.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.020+0000 7fce25008740 1 -- 172.21.15.134:0/3192267727 wait complete. 2024-07-26T11:22:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:43 smithi193 ceph-mon[37473]: pgmap v682: 129 pgs: 129 active+clean; 50 GiB data, 105 GiB used, 968 GiB / 1.0 TiB avail; 37 KiB/s rd, 224 MiB/s wr, 1.20k op/s 2024-07-26T11:22:44.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.022+0000 7fce25008740 20 client.26458 trim_cache size 0 max 16384 2024-07-26T11:22:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.044+0000 7fe00f1f1740 1 Processor -- start 2024-07-26T11:22:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.044+0000 7fe00f1f1740 1 -- start start 2024-07-26T11:22:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.045+0000 7fe00f1f1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 0x55a28413c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.049 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.045+0000 7fe00f1f1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d7b4d0 0x55a2841514b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.045+0000 7fe00f1f1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d9c1b0 0x55a284140030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.046+0000 7fe005800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 0x55a28413c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.046+0000 7fe005800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 0x55a28413c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:60558/0 (socket says 172.21.15.134:60558) 2024-07-26T11:22:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.046+0000 7fe006200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d9c1b0 0x55a284140030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.046+0000 7fe004e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d7b4d0 0x55a2841514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.047+0000 7fe004e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d7b4d0 0x55a2841514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34440/0 (socket says 172.21.15.134:34440) 2024-07-26T11:22:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.047+0000 7fe006200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d9c1b0 0x55a284140030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:59290/0 (socket says 172.21.15.134:59290) 2024-07-26T11:22:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.047+0000 7fe00f1f1740 1 -- 172.21.15.134:0/856868375 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe0001b6fc0 con 0x55a283d9c1b0 2024-07-26T11:22:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.047+0000 7fe00f1f1740 1 -- 172.21.15.134:0/856868375 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe0001b8b00 con 0x55a283d7b4d0 2024-07-26T11:22:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.047+0000 7fe00f1f1740 1 -- 172.21.15.134:0/856868375 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe0001b7690 con 0x55a283d732f0 2024-07-26T11:22:44.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.047+0000 7fe005800640 1 -- 172.21.15.134:0/856868375 learned_addr learned my addr 172.21.15.134:0/856868375 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.048+0000 7fe004e00640 1 -- 172.21.15.134:0/856868375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 msgr2=0x55a28413c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.048+0000 7fe004e00640 1 --2- 172.21.15.134:0/856868375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 0x55a28413c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.048+0000 7fe004e00640 1 -- 172.21.15.134:0/856868375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d9c1b0 msgr2=0x55a284140030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.048+0000 7fe004e00640 1 --2- 172.21.15.134:0/856868375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d9c1b0 0x55a284140030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.048+0000 7fe004e00640 1 -- 172.21.15.134:0/856868375 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a284139370 con 0x55a283d7b4d0 2024-07-26T11:22:44.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.048+0000 7fe005800640 1 --2- 172.21.15.134:0/856868375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 0x55a28413c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:22:44.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.048+0000 7fe006200640 1 --2- 172.21.15.134:0/856868375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d9c1b0 0x55a284140030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:22:44.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.049+0000 7fe004e00640 1 --2- 172.21.15.134:0/856868375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d7b4d0 0x55a2841514b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdfec006010 tx=0x7fdfec005a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c44da7aafb425247 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:44.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.049+0000 7fdff2000640 1 -- 172.21.15.134:0/856868375 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec035040 con 0x55a283d7b4d0 2024-07-26T11:22:44.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.050+0000 7fdff2000640 1 -- 172.21.15.134:0/856868375 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfec0044a0 con 0x55a283d7b4d0 2024-07-26T11:22:44.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.050+0000 7fdff2000640 1 -- 172.21.15.134:0/856868375 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec035040 con 0x55a283d7b4d0 2024-07-26T11:22:44.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.051+0000 7fe00f1f1740 1 -- 172.21.15.134:0/856868375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d7b4d0 msgr2=0x55a2841514b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.051+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/856868375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d7b4d0 0x55a2841514b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdfec006010 tx=0x7fdfec005a00 comp rx=0 tx=0).stop 2024-07-26T11:22:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.051+0000 7fe00f1f1740 1 -- 172.21.15.134:0/856868375 shutdown_connections 2024-07-26T11:22:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.051+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/856868375 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d9c1b0 0x55a284140030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.051+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/856868375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d7b4d0 0x55a2841514b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.052+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/856868375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 0x55a28413c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.052+0000 7fe00f1f1740 1 -- 172.21.15.134:0/856868375 >> 172.21.15.134:0/856868375 conn(0x55a283d90300 msgr2=0x55a28413c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:44.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.054+0000 7fe00f1f1740 1 -- 172.21.15.134:0/856868375 shutdown_connections 2024-07-26T11:22:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.054+0000 7fe00f1f1740 1 -- 172.21.15.134:0/856868375 wait complete. 2024-07-26T11:22:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.056+0000 7fe00f1f1740 1 -- start start 2024-07-26T11:22:44.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.057+0000 7fe00f1f1740 1 Processor -- start 2024-07-26T11:22:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.060+0000 7fe00f1f1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 0x55a284158190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.060+0000 7fe00f1f1740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d7b4d0 0x55a284147b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.060+0000 7fe00f1f1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d9c1b0 0x55a2841481f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.060+0000 7fe00f1f1740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe0001b8430 con 0x55a283d7b4d0 2024-07-26T11:22:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.060+0000 7fe00f1f1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe0001b7d60 con 0x55a283d9c1b0 2024-07-26T11:22:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.060+0000 7fe00f1f1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe0001b4fa0 con 0x55a283d732f0 2024-07-26T11:22:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.061+0000 7fe00f1f1740 10 client.0 _ll_register_callbacks cb 0x7fdff4005bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.062+0000 7fe004e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d7b4d0 0x55a284147b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.062+0000 7fe004e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d7b4d0 0x55a284147b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:59298/0 (socket says 172.21.15.134:59298) 2024-07-26T11:22:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.062+0000 7fe004e00640 1 -- 172.21.15.134:0/3818228557 learned_addr learned my addr 172.21.15.134:0/3818228557 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.062+0000 7fe004e00640 1 -- 172.21.15.134:0/3818228557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 msgr2=0x55a284158190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:22:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.062+0000 7fe004e00640 1 --2- 172.21.15.134:0/3818228557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 0x55a284158190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.062+0000 7fe004e00640 1 -- 172.21.15.134:0/3818228557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d9c1b0 msgr2=0x55a2841481f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:22:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.062+0000 7fe004e00640 1 --2- 172.21.15.134:0/3818228557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d9c1b0 0x55a2841481f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.062+0000 7fe004e00640 1 -- 172.21.15.134:0/3818228557 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a284148960 con 0x55a283d7b4d0 2024-07-26T11:22:44.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.067+0000 7fe004e00640 1 --2- 172.21.15.134:0/3818228557 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d7b4d0 0x55a284147b50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdfec0092a0 tx=0x7fdfec0052e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a4ff875076a91f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:44.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.067+0000 7fdfe7400640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:22:44.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.068+0000 7fdfe7400640 1 -- 172.21.15.134:0/3818228557 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec04f020 con 0x55a283d7b4d0 2024-07-26T11:22:44.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.068+0000 7fe00f1f1740 1 -- 172.21.15.134:0/3818228557 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfec0044a0 con 0x55a283d7b4d0 2024-07-26T11:22:44.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.068+0000 7fdfe7400640 1 -- 172.21.15.134:0/3818228557 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fdfec005500 con 0x55a283d7b4d0 2024-07-26T11:22:44.071 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.069+0000 7fdfe7400640 1 -- 172.21.15.134:0/3818228557 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdfec035040 con 0x55a283d7b4d0 2024-07-26T11:22:44.071 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.069+0000 7fe00f1f1740 10 client.16005 fetch_fsmap learned FSMap version 32 2024-07-26T11:22:44.071 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.069+0000 7fe00f1f1740 1 -- 172.21.15.134:0/3818228557 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a2841c5490 con 0x55a283d7b4d0 2024-07-26T11:22:44.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.070+0000 7fdfe7400640 1 -- 172.21.15.134:0/3818228557 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfec0044a0 con 0x55a283d7b4d0 2024-07-26T11:22:44.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.071+0000 7fdfe7400640 1 -- 172.21.15.134:0/3818228557 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fdfec003a70 con 0x55a283d7b4d0 2024-07-26T11:22:44.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.071+0000 7fe00f1f1740 10 client.16005 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:22:44.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.071+0000 7fe00f1f1740 10 client.16005 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:22:44.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.071+0000 7fe00f1f1740 20 client.16005 populate_metadata read hostname 'smithi134' 2024-07-26T11:22:44.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.071+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/3818228557 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a283d301c0 0x55a2841c7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.074 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.070+0000 7fce25008740 1 Processor -- start 2024-07-26T11:22:44.074 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.071+0000 7fce25008740 1 -- start start 2024-07-26T11:22:44.075 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.072+0000 7fe00f1f1740 4 client.16005 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:22:44.075 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.072+0000 7fe00f1f1740 1 -- 172.21.15.134:0/3818228557 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a2840b1c60 con 0x55a283d301c0 2024-07-26T11:22:44.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.071+0000 7fce25008740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb37f1b0 0x55dccb749f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.071+0000 7fce25008740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb387390 0x55dccb74e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.071+0000 7fce25008740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb3a8070 0x55dccb76fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.071+0000 7fce25008740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fce181d05c0 con 0x55dccb37f1b0 2024-07-26T11:22:44.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.071+0000 7fce25008740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fce181d2100 con 0x55dccb387390 2024-07-26T11:22:44.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.071+0000 7fce25008740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fce181d0c90 con 0x55dccb3a8070 2024-07-26T11:22:44.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.072+0000 7fce16a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb387390 0x55dccb74e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:44.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.073+0000 7fce16a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb387390 0x55dccb74e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34444/0 (socket says 172.21.15.134:34444) 2024-07-26T11:22:44.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.073+0000 7fce16a00640 1 -- 172.21.15.134:0/1844708722 learned_addr learned my addr 172.21.15.134:0/1844708722 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:44.076 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.073+0000 7fe005800640 1 --2- 172.21.15.134:0/3818228557 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a283d301c0 0x55a2841c7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:44.076 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.074+0000 7fe005800640 1 --2- 172.21.15.134:0/3818228557 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a283d301c0 0x55a2841c7ba0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74dac06437b0caa9 server_cookie=fab9cf258ebc753 in_seq=0 out_seq=0 2024-07-26T11:22:44.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.074+0000 7fdfe7400640 10 client.16005 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:22:44.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.072+0000 7fce17400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb37f1b0 0x55dccb749f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:44.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.073+0000 7fce17e00640 1 --2- 172.21.15.134:0/1844708722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb3a8070 0x55dccb76fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:44.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.073+0000 7fce16a00640 1 -- 172.21.15.134:0/1844708722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb3a8070 msgr2=0x55dccb76fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:44.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.073+0000 7fce16a00640 1 --2- 172.21.15.134:0/1844708722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb3a8070 0x55dccb76fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.073+0000 7fce16a00640 1 -- 172.21.15.134:0/1844708722 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb37f1b0 msgr2=0x55dccb749f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:44.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.073+0000 7fce16a00640 1 --2- 172.21.15.134:0/1844708722 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb37f1b0 0x55dccb749f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.073+0000 7fce16a00640 1 -- 172.21.15.134:0/1844708722 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dccb74f320 con 0x55dccb387390 2024-07-26T11:22:44.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.076+0000 7fce16a00640 1 --2- 172.21.15.134:0/1844708722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb387390 0x55dccb74e770 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fce0400b680 tx=0x7fce04014940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb63b303cf6feaa0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:44.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.077+0000 7fce03400640 1 -- 172.21.15.134:0/1844708722 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04014630 con 0x55dccb387390 2024-07-26T11:22:44.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.077+0000 7fce03400640 1 -- 172.21.15.134:0/1844708722 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce0400d950 con 0x55dccb387390 2024-07-26T11:22:44.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.077+0000 7fce03400640 1 -- 172.21.15.134:0/1844708722 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0401e810 con 0x55dccb387390 2024-07-26T11:22:44.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.078+0000 7fce25008740 1 -- 172.21.15.134:0/1844708722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb387390 msgr2=0x55dccb74e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:44.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.078+0000 7fce25008740 1 --2- 172.21.15.134:0/1844708722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb387390 0x55dccb74e770 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fce0400b680 tx=0x7fce04014940 comp rx=0 tx=0).stop 2024-07-26T11:22:44.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.078+0000 7fce25008740 1 -- 172.21.15.134:0/1844708722 shutdown_connections 2024-07-26T11:22:44.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.078+0000 7fce25008740 1 --2- 172.21.15.134:0/1844708722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb3a8070 0x55dccb76fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.078+0000 7fce25008740 1 --2- 172.21.15.134:0/1844708722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb387390 0x55dccb74e770 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.078+0000 7fce25008740 1 --2- 172.21.15.134:0/1844708722 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb37f1b0 0x55dccb749f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.078+0000 7fce25008740 1 -- 172.21.15.134:0/1844708722 >> 172.21.15.134:0/1844708722 conn(0x55dccb39c460 msgr2=0x55dccb74e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:44.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.078+0000 7fce25008740 1 -- 172.21.15.134:0/1844708722 shutdown_connections 2024-07-26T11:22:44.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.080+0000 7fce25008740 1 -- 172.21.15.134:0/1844708722 wait complete. 2024-07-26T11:22:44.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.081+0000 7fce25008740 1 -- start start 2024-07-26T11:22:44.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.081+0000 7fce25008740 1 Processor -- start 2024-07-26T11:22:44.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.083+0000 7fce25008740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb37f1b0 0x55dccb761240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.083+0000 7fce25008740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb387390 0x55dccb75e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.083+0000 7fce25008740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb3a8070 0x55dccb75ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.083+0000 7fce25008740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fce181d1a30 con 0x55dccb3a8070 2024-07-26T11:22:44.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.083+0000 7fce25008740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fce181d1360 con 0x55dccb37f1b0 2024-07-26T11:22:44.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.083+0000 7fce16a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb387390 0x55dccb75e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:44.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.083+0000 7fce16a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb387390 0x55dccb75e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:60584/0 (socket says 172.21.15.134:60584) 2024-07-26T11:22:44.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.083+0000 7fce16a00640 1 -- 172.21.15.134:0/1255287507 learned_addr learned my addr 172.21.15.134:0/1255287507 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:44.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.083+0000 7fce17400640 1 --2- 172.21.15.134:0/1255287507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb37f1b0 0x55dccb761240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:44.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.084+0000 7fce25008740 1 -- 172.21.15.134:0/1255287507 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fce181ce5a0 con 0x55dccb387390 2024-07-26T11:22:44.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.084+0000 7fce16a00640 1 -- 172.21.15.134:0/1255287507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb37f1b0 msgr2=0x55dccb761240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:44.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.084+0000 7fce16a00640 1 --2- 172.21.15.134:0/1255287507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb37f1b0 0x55dccb761240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.084+0000 7fce16a00640 1 -- 172.21.15.134:0/1255287507 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb3a8070 msgr2=0x55dccb75ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:44.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.084+0000 7fce16a00640 1 --2- 172.21.15.134:0/1255287507 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb3a8070 0x55dccb75ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.084+0000 7fce16a00640 1 -- 172.21.15.134:0/1255287507 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dccb75f4f0 con 0x55dccb387390 2024-07-26T11:22:44.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.084+0000 7fce25008740 10 client.0 _ll_register_callbacks cb 0x7fce100aaec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:44.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.084+0000 7fce25008740 1 -- 172.21.15.134:0/1255287507 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce0400d950 con 0x55dccb387390 2024-07-26T11:22:44.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.085+0000 7fce16a00640 1 --2- 172.21.15.134:0/1255287507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb387390 0x55dccb75e710 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fce04006cd0 tx=0x7fce04014660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1914cf1d66005268 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:44.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.085+0000 7fce00c00640 10 client.26467 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:22:44.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.086+0000 7fce00c00640 1 -- 172.21.15.134:0/1255287507 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce040149b0 con 0x55dccb387390 2024-07-26T11:22:44.096 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.086+0000 7fce00c00640 1 -- 172.21.15.134:0/1255287507 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7fce04014b50 con 0x55dccb387390 2024-07-26T11:22:44.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.086+0000 7fce00c00640 1 -- 172.21.15.134:0/1255287507 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fce04014d90 con 0x55dccb387390 2024-07-26T11:22:44.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.088+0000 7fce25008740 10 client.26467 fetch_fsmap learned FSMap version 32 2024-07-26T11:22:44.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.088+0000 7fce25008740 1 -- 172.21.15.134:0/1255287507 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=32}) -- 0x55dccb7d2c80 con 0x55dccb387390 2024-07-26T11:22:44.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.093+0000 7fce00c00640 1 -- 172.21.15.134:0/1255287507 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fce04013070 con 0x55dccb387390 2024-07-26T11:22:44.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.093+0000 7fce25008740 10 client.26467 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:22:44.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.093+0000 7fce25008740 10 client.26467 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T11:22:44.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.093+0000 7fce25008740 20 client.26467 populate_metadata read hostname 'smithi134' 2024-07-26T11:22:44.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.093+0000 7fce25008740 1 --2- 172.21.15.134:0/1255287507 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55dccb33bda0 0x55dccb7d5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:44.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.093+0000 7fce25008740 4 client.26467 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:22:44.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.093+0000 7fce25008740 1 -- 172.21.15.134:0/1255287507 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dccb6c15e0 con 0x55dccb33bda0 2024-07-26T11:22:44.097 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.093+0000 7fce17400640 1 --2- 172.21.15.134:0/1255287507 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55dccb33bda0 0x55dccb7d5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:44.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.094+0000 7fce00c00640 10 client.26467 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:22:44.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.094+0000 7fce17400640 1 --2- 172.21.15.134:0/1255287507 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55dccb33bda0 0x55dccb7d5110 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fdf8cd29baeba2ca server_cookie=6e6f12b742b0c76b in_seq=0 out_seq=0 2024-07-26T11:22:44.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.094+0000 7fce00c00640 1 -- 172.21.15.134:0/1255287507 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dccb6c15e0 con 0x55dccb33bda0 2024-07-26T11:22:44.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.094+0000 7fce00c00640 10 client.26467 handle_command_reply: tid=0 2024-07-26T11:22:44.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.094+0000 7fce25008740 1 client.26467 shutdown 2024-07-26T11:22:44.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.095+0000 7fce25008740 1 -- 172.21.15.134:0/1255287507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb387390 msgr2=0x55dccb75e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:44.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.095+0000 7fce25008740 1 --2- 172.21.15.134:0/1255287507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb387390 0x55dccb75e710 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fce04006cd0 tx=0x7fce04014660 comp rx=0 tx=0).stop 2024-07-26T11:22:44.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.095+0000 7fce25008740 1 -- 172.21.15.134:0/1255287507 shutdown_connections 2024-07-26T11:22:44.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.095+0000 7fce25008740 1 --2- 172.21.15.134:0/1255287507 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55dccb33bda0 0x55dccb7d5110 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.095+0000 7fce25008740 1 --2- 172.21.15.134:0/1255287507 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dccb387390 0x55dccb75e710 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.095+0000 7fce25008740 1 --2- 172.21.15.134:0/1255287507 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55dccb3a8070 0x55dccb75ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.095+0000 7fce25008740 1 --2- 172.21.15.134:0/1255287507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dccb37f1b0 0x55dccb761240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.095+0000 7fce25008740 1 -- 172.21.15.134:0/1255287507 >> 172.21.15.134:0/1255287507 conn(0x55dccb39c460 msgr2=0x55dccb75a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:44.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.095+0000 7fce00c00640 0 client.26467 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:22:44.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.096+0000 7fce25008740 1 -- 172.21.15.134:0/1255287507 shutdown_connections 2024-07-26T11:22:44.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.096+0000 7fce25008740 1 -- 172.21.15.134:0/1255287507 wait complete. 2024-07-26T11:22:44.099 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.096+0000 7fce25008740 20 client.26467 trim_cache size 0 max 16384 2024-07-26T11:22:44.101 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:22:44.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.102+0000 7fce1cc00640 1 -- 172.21.15.134:0/903632314 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcdfc0b3e40 msgr2=0x7fcdfc0b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:44.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.102+0000 7fce1cc00640 1 --2- 172.21.15.134:0/903632314 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcdfc0b3e40 0x7fcdfc0b6350 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fce0802e980 tx=0x7fce0800d580 comp rx=0 tx=0).stop 2024-07-26T11:22:44.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.102+0000 7fce1cc00640 1 -- 172.21.15.134:0/903632314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce1810f740 msgr2=0x7fce18080bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:44.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.102+0000 7fce1cc00640 1 --2- 172.21.15.134:0/903632314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce1810f740 0x7fce18080bd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fce1000cc60 tx=0x7fce1000a620 comp rx=0 tx=0).stop 2024-07-26T11:22:44.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.103+0000 7fce1cc00640 1 -- 172.21.15.134:0/903632314 shutdown_connections 2024-07-26T11:22:44.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.103+0000 7fce1cc00640 1 --2- 172.21.15.134:0/903632314 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fcdfc0b3e40 0x7fcdfc0b6350 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.103+0000 7fce1cc00640 1 --2- 172.21.15.134:0/903632314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fce1810f740 0x7fce18080bd0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.103+0000 7fce1cc00640 1 --2- 172.21.15.134:0/903632314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce1810b310 0x7fce18080650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.103+0000 7fce1cc00640 1 --2- 172.21.15.134:0/903632314 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fce180790b0 0x7fce18080110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.104+0000 7fce1cc00640 1 -- 172.21.15.134:0/903632314 >> 172.21.15.134:0/903632314 conn(0x7fce1806e420 msgr2=0x7fce1810d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:44.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.107+0000 7fce1cc00640 1 -- 172.21.15.134:0/903632314 shutdown_connections 2024-07-26T11:22:44.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:44.107+0000 7fce1cc00640 1 -- 172.21.15.134:0/903632314 wait complete. 2024-07-26T11:22:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.140+0000 7fdfe7400640 1 -- 172.21.15.134:0/3818228557 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a2840b1c60 con 0x55a283d301c0 2024-07-26T11:22:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.140+0000 7fdfe7400640 10 client.16005 handle_command_reply: tid=0 2024-07-26T11:22:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.140+0000 7fe00f1f1740 1 client.16005 shutdown 2024-07-26T11:22:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.140+0000 7fe00f1f1740 1 -- 172.21.15.134:0/3818228557 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d7b4d0 msgr2=0x55a284147b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.140+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/3818228557 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d7b4d0 0x55a284147b50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdfec0092a0 tx=0x7fdfec0052e0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.140+0000 7fe00f1f1740 1 -- 172.21.15.134:0/3818228557 shutdown_connections 2024-07-26T11:22:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.140+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/3818228557 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a283d301c0 0x55a2841c7ba0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.140+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/3818228557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a283d9c1b0 0x55a2841481f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.140+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/3818228557 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a283d7b4d0 0x55a284147b50 secure :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdfec0092a0 tx=0x7fdfec0052e0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.140+0000 7fe00f1f1740 1 --2- 172.21.15.134:0/3818228557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a283d732f0 0x55a284158190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.140+0000 7fe00f1f1740 1 -- 172.21.15.134:0/3818228557 >> 172.21.15.134:0/3818228557 conn(0x55a283d90300 msgr2=0x55a28413d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:44.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.141+0000 7fdfe7400640 0 client.16005 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:22:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.142+0000 7fe00f1f1740 1 -- 172.21.15.134:0/3818228557 shutdown_connections 2024-07-26T11:22:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.142+0000 7fe00f1f1740 1 -- 172.21.15.134:0/3818228557 wait complete. 2024-07-26T11:22:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.142+0000 7fe00f1f1740 20 client.16005 trim_cache size 0 max 16384 2024-07-26T11:22:44.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:22:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:22:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:22:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:22:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.143+0000 7fe008200640 1 -- 172.21.15.134:0/2260346647 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdfe80b4070 msgr2=0x7fdfe80b6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.143+0000 7fe008200640 1 --2- 172.21.15.134:0/2260346647 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdfe80b4070 0x7fdfe80b6580 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fdff8009ec0 tx=0x7fdff800f730 comp rx=0 tx=0).stop 2024-07-26T11:22:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.143+0000 7fe008200640 1 -- 172.21.15.134:0/2260346647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe000109fd0 msgr2=0x7fe0001a26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.143+0000 7fe008200640 1 --2- 172.21.15.134:0/2260346647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe000109fd0 0x7fe0001a26b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fdff4011f80 tx=0x7fdff4009240 comp rx=0 tx=0).stop 2024-07-26T11:22:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.143+0000 7fe008200640 1 -- 172.21.15.134:0/2260346647 shutdown_connections 2024-07-26T11:22:44.145 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.143+0000 7fe008200640 1 --2- 172.21.15.134:0/2260346647 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdfe80b4070 0x7fdfe80b6580 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.143+0000 7fe008200640 1 --2- 172.21.15.134:0/2260346647 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe000115fd0 0x7fe0001a7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.143+0000 7fe008200640 1 --2- 172.21.15.134:0/2260346647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe00010a920 0x7fe0001a2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.143+0000 7fe008200640 1 --2- 172.21.15.134:0/2260346647 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe000109fd0 0x7fe0001a26b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:44.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.143+0000 7fe008200640 1 -- 172.21.15.134:0/2260346647 >> 172.21.15.134:0/2260346647 conn(0x7fe000070800 msgr2=0x7fe000071750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:44.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.143+0000 7fe008200640 1 -- 172.21.15.134:0/2260346647 shutdown_connections 2024-07-26T11:22:44.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:44.143+0000 7fe008200640 1 -- 172.21.15.134:0/2260346647 wait complete. 2024-07-26T11:22:44.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:44 smithi134 ceph-mon[31966]: pgmap v683: 129 pgs: 129 active+clean; 51 GiB data, 106 GiB used, 967 GiB / 1.0 TiB avail; 53 KiB/s rd, 279 MiB/s wr, 1.51k op/s 2024-07-26T11:22:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:44 smithi189 ceph-mon[36790]: pgmap v683: 129 pgs: 129 active+clean; 51 GiB data, 106 GiB used, 967 GiB / 1.0 TiB avail; 53 KiB/s rd, 279 MiB/s wr, 1.51k op/s 2024-07-26T11:22:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:44 smithi193 ceph-mon[37473]: pgmap v683: 129 pgs: 129 active+clean; 51 GiB data, 106 GiB used, 967 GiB / 1.0 TiB avail; 53 KiB/s rd, 279 MiB/s wr, 1.51k op/s 2024-07-26T11:22:46.748 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-26T11:22:46.876 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:46 smithi134 ceph-mon[31966]: pgmap v684: 129 pgs: 129 active+clean; 51 GiB data, 107 GiB used, 966 GiB / 1.0 TiB avail; 33 KiB/s rd, 268 MiB/s wr, 1.39k op/s 2024-07-26T11:22:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:46 smithi189 ceph-mon[36790]: pgmap v684: 129 pgs: 129 active+clean; 51 GiB data, 107 GiB used, 966 GiB / 1.0 TiB avail; 33 KiB/s rd, 268 MiB/s wr, 1.39k op/s 2024-07-26T11:22:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:46 smithi193 ceph-mon[37473]: pgmap v684: 129 pgs: 129 active+clean; 51 GiB data, 107 GiB used, 966 GiB / 1.0 TiB avail; 33 KiB/s rd, 268 MiB/s wr, 1.39k op/s 2024-07-26T11:22:47.664 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:22:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:48 smithi134 ceph-mon[31966]: pgmap v685: 129 pgs: 129 active+clean; 51 GiB data, 108 GiB used, 965 GiB / 1.0 TiB avail; 58 KiB/s rd, 255 MiB/s wr, 1.27k op/s 2024-07-26T11:22:48.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:48 smithi189 ceph-mon[36790]: pgmap v685: 129 pgs: 129 active+clean; 51 GiB data, 108 GiB used, 965 GiB / 1.0 TiB avail; 58 KiB/s rd, 255 MiB/s wr, 1.27k op/s 2024-07-26T11:22:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:48 smithi193 ceph-mon[37473]: pgmap v685: 129 pgs: 129 active+clean; 51 GiB data, 108 GiB used, 965 GiB / 1.0 TiB avail; 58 KiB/s rd, 255 MiB/s wr, 1.27k op/s 2024-07-26T11:22:50.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:50 smithi134 ceph-mon[31966]: pgmap v686: 129 pgs: 129 active+clean; 52 GiB data, 108 GiB used, 964 GiB / 1.0 TiB avail; 42 KiB/s rd, 290 MiB/s wr, 1.45k op/s 2024-07-26T11:22:50.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:50 smithi189 ceph-mon[36790]: pgmap v686: 129 pgs: 129 active+clean; 52 GiB data, 108 GiB used, 964 GiB / 1.0 TiB avail; 42 KiB/s rd, 290 MiB/s wr, 1.45k op/s 2024-07-26T11:22:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:50 smithi193 ceph-mon[37473]: pgmap v686: 129 pgs: 129 active+clean; 52 GiB data, 108 GiB used, 964 GiB / 1.0 TiB avail; 42 KiB/s rd, 290 MiB/s wr, 1.45k op/s 2024-07-26T11:22:52.930 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:22:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:22:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:22:53.861 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:53 smithi189 ceph-mon[36790]: pgmap v687: 129 pgs: 129 active+clean; 52 GiB data, 109 GiB used, 964 GiB / 1.0 TiB avail; 42 KiB/s rd, 227 MiB/s wr, 1.16k op/s 2024-07-26T11:22:53.862 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:53.862 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:53.862 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:53.862 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:53 smithi134 ceph-mon[31966]: pgmap v687: 129 pgs: 129 active+clean; 52 GiB data, 109 GiB used, 964 GiB / 1.0 TiB avail; 42 KiB/s rd, 227 MiB/s wr, 1.16k op/s 2024-07-26T11:22:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:53 smithi193 ceph-mon[37473]: pgmap v687: 129 pgs: 129 active+clean; 52 GiB data, 109 GiB used, 964 GiB / 1.0 TiB avail; 42 KiB/s rd, 227 MiB/s wr, 1.16k op/s 2024-07-26T11:22:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:55.823 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:55 smithi193 ceph-mon[37473]: pgmap v688: 129 pgs: 129 active+clean; 52 GiB data, 109 GiB used, 964 GiB / 1.0 TiB avail; 58 KiB/s rd, 270 MiB/s wr, 1.39k op/s 2024-07-26T11:22:55.823 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:55.824 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:55.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:55 smithi134 ceph-mon[31966]: pgmap v688: 129 pgs: 129 active+clean; 52 GiB data, 109 GiB used, 964 GiB / 1.0 TiB avail; 58 KiB/s rd, 270 MiB/s wr, 1.39k op/s 2024-07-26T11:22:55.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:55.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:55.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:55 smithi189 ceph-mon[36790]: pgmap v688: 129 pgs: 129 active+clean; 52 GiB data, 109 GiB used, 964 GiB / 1.0 TiB avail; 58 KiB/s rd, 270 MiB/s wr, 1.39k op/s 2024-07-26T11:22:55.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:55.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:56.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.835+0000 7f744f600640 1 -- 172.21.15.134:0/4235783011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f744810a740 msgr2=0x7f74481169f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:56.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.835+0000 7f744f600640 1 --2- 172.21.15.134:0/4235783011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f744810a740 0x7f74481169f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7440005fd0 tx=0x7f744002b3e0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.840 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:56 smithi134 ceph-mon[31966]: pgmap v689: 129 pgs: 129 active+clean; 52 GiB data, 110 GiB used, 963 GiB / 1.0 TiB avail; 43 KiB/s rd, 262 MiB/s wr, 1.33k op/s 2024-07-26T11:22:56.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.838+0000 7f744f600640 1 -- 172.21.15.134:0/4235783011 shutdown_connections 2024-07-26T11:22:56.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.838+0000 7f744f600640 1 --2- 172.21.15.134:0/4235783011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f744810a740 0x7f74481169f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.838+0000 7f744f600640 1 --2- 172.21.15.134:0/4235783011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7448109df0 0x7f744810a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.838+0000 7f744f600640 1 --2- 172.21.15.134:0/4235783011 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f744810eb80 0x7f744810ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.839+0000 7f744f600640 1 -- 172.21.15.134:0/4235783011 >> 172.21.15.134:0/4235783011 conn(0x7f7448070d20 msgr2=0x7f7448073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:56.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.839+0000 7f744f600640 1 -- 172.21.15.134:0/4235783011 shutdown_connections 2024-07-26T11:22:56.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.839+0000 7f744f600640 1 -- 172.21.15.134:0/4235783011 wait complete. 2024-07-26T11:22:56.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.842+0000 7f744f600640 1 Processor -- start 2024-07-26T11:22:56.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.850+0000 7f744f600640 1 -- start start 2024-07-26T11:22:56.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.851+0000 7f744f600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7448109df0 0x7f74481a0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:56.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.851+0000 7f744f600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f744810a740 0x7f74481a09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:56.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.851+0000 7f744f600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f744810eb80 0x7f74481a5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:56.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.851+0000 7f7447e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f744810a740 0x7f74481a09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:56.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.851+0000 7f7447e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f744810a740 0x7f74481a09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:45684/0 (socket says 172.21.15.134:45684) 2024-07-26T11:22:56.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.852+0000 7f744cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7448109df0 0x7f74481a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:56.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.852+0000 7f744cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7448109df0 0x7f74481a0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56834/0 (socket says 172.21.15.134:56834) 2024-07-26T11:22:56.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.855+0000 7f7447e00640 1 -- 172.21.15.134:0/1471494071 learned_addr learned my addr 172.21.15.134:0/1471494071 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:56.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.855+0000 7f744f600640 1 -- 172.21.15.134:0/1471494071 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7448119ae0 con 0x7f744810eb80 2024-07-26T11:22:56.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.856+0000 7f744f600640 1 -- 172.21.15.134:0/1471494071 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7448119960 con 0x7f7448109df0 2024-07-26T11:22:56.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.856+0000 7f744f600640 1 -- 172.21.15.134:0/1471494071 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7448119c60 con 0x7f744810a740 2024-07-26T11:22:56.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.856+0000 7f744d600640 1 --2- 172.21.15.134:0/1471494071 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f744810eb80 0x7f74481a5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:56.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.861+0000 7f7447e00640 1 -- 172.21.15.134:0/1471494071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7448109df0 msgr2=0x7f74481a0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:56.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.861+0000 7f7447e00640 1 --2- 172.21.15.134:0/1471494071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7448109df0 0x7f74481a0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.861+0000 7f7447e00640 1 -- 172.21.15.134:0/1471494071 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f744810eb80 msgr2=0x7f74481a5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:56.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.861+0000 7f7447e00640 1 --2- 172.21.15.134:0/1471494071 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f744810eb80 0x7f74481a5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.861+0000 7f7447e00640 1 -- 172.21.15.134:0/1471494071 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74481a63c0 con 0x7f744810a740 2024-07-26T11:22:56.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.862+0000 7f7447e00640 1 --2- 172.21.15.134:0/1471494071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f744810a740 0x7f74481a09f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7434009f70 tx=0x7f743400edd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:56.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.862+0000 7f744cc00640 1 --2- 172.21.15.134:0/1471494071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7448109df0 0x7f74481a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:22:56.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.862+0000 7f744d600640 1 --2- 172.21.15.134:0/1471494071 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f744810eb80 0x7f74481a5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:22:56.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.862+0000 7f7445600640 1 -- 172.21.15.134:0/1471494071 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434004020 con 0x7f744810a740 2024-07-26T11:22:56.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.865+0000 7f7445600640 1 -- 172.21.15.134:0/1471494071 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74340041c0 con 0x7f744810a740 2024-07-26T11:22:56.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.865+0000 7f7445600640 1 -- 172.21.15.134:0/1471494071 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743400ce20 con 0x7f744810a740 2024-07-26T11:22:56.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.866+0000 7f744f600640 1 -- 172.21.15.134:0/1471494071 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74481a66b0 con 0x7f744810a740 2024-07-26T11:22:56.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.866+0000 7f744f600640 1 -- 172.21.15.134:0/1471494071 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74481ace10 con 0x7f744810a740 2024-07-26T11:22:56.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.874+0000 7f7445600640 1 -- 172.21.15.134:0/1471494071 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f74340159e0 con 0x7f744810a740 2024-07-26T11:22:56.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.877+0000 7f7445600640 1 --2- 172.21.15.134:0/1471494071 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f74300b3f10 0x7f74300b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:56.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.877+0000 7f7445600640 1 -- 172.21.15.134:0/1471494071 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f74340e86c0 con 0x7f744810a740 2024-07-26T11:22:56.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.877+0000 7f744cc00640 1 --2- 172.21.15.134:0/1471494071 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f74300b3f10 0x7f74300b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:56.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.878+0000 7f744cc00640 1 --2- 172.21.15.134:0/1471494071 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f74300b3f10 0x7f74300b6420 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f743c0058c0 tx=0x7f743c005ec0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:56 smithi189 ceph-mon[36790]: pgmap v689: 129 pgs: 129 active+clean; 52 GiB data, 110 GiB used, 963 GiB / 1.0 TiB avail; 43 KiB/s rd, 262 MiB/s wr, 1.33k op/s 2024-07-26T11:22:56.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.937+0000 7f74562a7740 1 Processor -- start 2024-07-26T11:22:56.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.948+0000 7f74562a7740 1 -- start start 2024-07-26T11:22:56.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.950+0000 7f74562a7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d960070 0x55e17dd0d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:56.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.950+0000 7f74562a7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d9371b0 0x55e17dd0dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:56.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.951+0000 7f74562a7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d93f390 0x55e17dd0e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:56.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.951+0000 7f74562a7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e17dc7bbf0 con 0x55e17d960070 2024-07-26T11:22:56.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.951+0000 7f74562a7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e17dc6b0f0 con 0x55e17d93f390 2024-07-26T11:22:56.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.951+0000 7f74562a7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e17dc6f130 con 0x55e17d9371b0 2024-07-26T11:22:56.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.953+0000 7f7447e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d9371b0 0x55e17dd0dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:56.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.953+0000 7f7447e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d9371b0 0x55e17dd0dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:45694/0 (socket says 172.21.15.134:45694) 2024-07-26T11:22:56.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.953+0000 7f7447e00640 1 -- 172.21.15.134:0/3864623349 learned_addr learned my addr 172.21.15.134:0/3864623349 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:56.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.953+0000 7f744d600640 1 --2- 172.21.15.134:0/3864623349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d93f390 0x55e17dd0e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:56.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.954+0000 7f744cc00640 1 --2- 172.21.15.134:0/3864623349 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d960070 0x55e17dd0d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:56.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.955+0000 7f7447e00640 1 -- 172.21.15.134:0/3864623349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d93f390 msgr2=0x55e17dd0e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:56.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.955+0000 7f7447e00640 1 --2- 172.21.15.134:0/3864623349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d93f390 0x55e17dd0e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.955+0000 7f7447e00640 1 -- 172.21.15.134:0/3864623349 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d960070 msgr2=0x55e17dd0d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:56.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.955+0000 7f7447e00640 1 --2- 172.21.15.134:0/3864623349 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d960070 0x55e17dd0d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.955+0000 7f7447e00640 1 -- 172.21.15.134:0/3864623349 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e17d9cb650 con 0x55e17d9371b0 2024-07-26T11:22:56.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.956+0000 7f7447e00640 1 --2- 172.21.15.134:0/3864623349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d9371b0 0x55e17dd0dba0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f74340aaff0 tx=0x7f74340aee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8c8316b3b2ac43f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:56.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.957+0000 7f7439c00640 1 -- 172.21.15.134:0/3864623349 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434015070 con 0x55e17d9371b0 2024-07-26T11:22:56.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.957+0000 7f7439c00640 1 -- 172.21.15.134:0/3864623349 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74340e7070 con 0x55e17d9371b0 2024-07-26T11:22:56.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.957+0000 7f7439c00640 1 -- 172.21.15.134:0/3864623349 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74340e5cb0 con 0x55e17d9371b0 2024-07-26T11:22:56.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.958+0000 7f74562a7740 1 -- 172.21.15.134:0/3864623349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d9371b0 msgr2=0x55e17dd0dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:56.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.958+0000 7f74562a7740 1 --2- 172.21.15.134:0/3864623349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d9371b0 0x55e17dd0dba0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f74340aaff0 tx=0x7f74340aee70 comp rx=0 tx=0).stop 2024-07-26T11:22:56.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.959+0000 7f74562a7740 1 -- 172.21.15.134:0/3864623349 shutdown_connections 2024-07-26T11:22:56.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.959+0000 7f74562a7740 1 --2- 172.21.15.134:0/3864623349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d9371b0 0x55e17dd0dba0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.959+0000 7f74562a7740 1 --2- 172.21.15.134:0/3864623349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d93f390 0x55e17dd0e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.959+0000 7f74562a7740 1 --2- 172.21.15.134:0/3864623349 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d960070 0x55e17dd0d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.959+0000 7f74562a7740 1 -- 172.21.15.134:0/3864623349 >> 172.21.15.134:0/3864623349 conn(0x55e17d954460 msgr2=0x55e17dd0b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:56.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.960+0000 7f74562a7740 1 -- 172.21.15.134:0/3864623349 shutdown_connections 2024-07-26T11:22:56.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.961+0000 7f74562a7740 1 -- 172.21.15.134:0/3864623349 wait complete. 2024-07-26T11:22:56.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.962+0000 7f74562a7740 1 -- start start 2024-07-26T11:22:56.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.964+0000 7f74562a7740 1 Processor -- start 2024-07-26T11:22:56.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.970+0000 7f74562a7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d9371b0 0x55e17dd18c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:56.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.971+0000 7f74562a7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d93f390 0x55e17dd19170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:56.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.971+0000 7f74562a7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d960070 0x55e17dd196b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:56.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.971+0000 7f74562a7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e17dace700 con 0x55e17d960070 2024-07-26T11:22:56.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.971+0000 7f74562a7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e17dc076c0 con 0x55e17d93f390 2024-07-26T11:22:56.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.971+0000 7f74562a7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e17dd1be90 con 0x55e17d9371b0 2024-07-26T11:22:56.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.971+0000 7f74562a7740 10 client.0 _ll_register_callbacks cb 0x55e17dc17170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:56.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.973+0000 7f7447e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d93f390 0x55e17dd19170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:56.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.974+0000 7f7447e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d93f390 0x55e17dd19170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56850/0 (socket says 172.21.15.134:56850) 2024-07-26T11:22:56.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.974+0000 7f7447e00640 1 -- 172.21.15.134:0/3353462867 learned_addr learned my addr 172.21.15.134:0/3353462867 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:56.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.974+0000 7f744cc00640 1 --2- 172.21.15.134:0/3353462867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d9371b0 0x55e17dd18c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:56.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.975+0000 7f744cc00640 1 -- 172.21.15.134:0/3353462867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d93f390 msgr2=0x55e17dd19170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:56.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.975+0000 7f744cc00640 1 --2- 172.21.15.134:0/3353462867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d93f390 0x55e17dd19170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.975+0000 7f744cc00640 1 -- 172.21.15.134:0/3353462867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d960070 msgr2=0x55e17dd196b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:56.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.975+0000 7f744cc00640 1 --2- 172.21.15.134:0/3353462867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d960070 0x55e17dd196b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.975+0000 7f744cc00640 1 -- 172.21.15.134:0/3353462867 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e17dc1e860 con 0x55e17d9371b0 2024-07-26T11:22:56.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.977+0000 7f744cc00640 1 --2- 172.21.15.134:0/3353462867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d9371b0 0x55e17dd18c30 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f743c00e7e0 tx=0x7f743c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b514341fae99b58 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:56.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.977+0000 7f742e000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:22:56.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.980+0000 7f742e000640 1 -- 172.21.15.134:0/3353462867 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c00fd50 con 0x55e17d9371b0 2024-07-26T11:22:56.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.980+0000 7f74562a7740 1 -- 172.21.15.134:0/3353462867 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74340e7070 con 0x55e17d9371b0 2024-07-26T11:22:56.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.982+0000 7f742e000640 1 -- 172.21.15.134:0/3353462867 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f743c00fb50 con 0x55e17d9371b0 2024-07-26T11:22:56.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.983+0000 7f742e000640 1 -- 172.21.15.134:0/3353462867 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f743c00fd50 con 0x55e17d9371b0 2024-07-26T11:22:56.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.983+0000 7f74562a7740 10 client.26479 fetch_fsmap learned FSMap version 32 2024-07-26T11:22:56.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.983+0000 7f74562a7740 1 -- 172.21.15.134:0/3353462867 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e17dd27bc0 con 0x55e17d9371b0 2024-07-26T11:22:56.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.983+0000 7f742e000640 1 -- 172.21.15.134:0/3353462867 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74340e7070 con 0x55e17d9371b0 2024-07-26T11:22:56.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.984+0000 7f742e000640 1 -- 172.21.15.134:0/3353462867 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f743c026400 con 0x55e17d9371b0 2024-07-26T11:22:56.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.984+0000 7f74562a7740 10 client.26479 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:22:56.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.984+0000 7f74562a7740 10 client.26479 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-07-26T11:22:56.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.984+0000 7f74562a7740 20 client.26479 populate_metadata read hostname 'smithi134' 2024-07-26T11:22:56.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.984+0000 7f74562a7740 1 --2- 172.21.15.134:0/3353462867 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55e17d91e9f0 0x55e17dd2b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:56.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.984+0000 7f74562a7740 4 client.26479 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:22:56.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.984+0000 7f74562a7740 1 -- 172.21.15.134:0/3353462867 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e17dc7db40 con 0x55e17d91e9f0 2024-07-26T11:22:56.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.985+0000 7f7447e00640 1 --2- 172.21.15.134:0/3353462867 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55e17d91e9f0 0x55e17dd2b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:56.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.986+0000 7f7447e00640 1 --2- 172.21.15.134:0/3353462867 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55e17d91e9f0 0x55e17dd2b070 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=48c9d24ad0f43a4b server_cookie=20d18c54b77840 in_seq=0 out_seq=0 2024-07-26T11:22:56.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.987+0000 7f742e000640 10 client.26479 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:22:56.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.987+0000 7f742e000640 1 -- 172.21.15.134:0/3353462867 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e17dc7db40 con 0x55e17d91e9f0 2024-07-26T11:22:56.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.987+0000 7f742e000640 10 client.26479 handle_command_reply: tid=0 2024-07-26T11:22:56.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.987+0000 7f74562a7740 1 client.26479 shutdown 2024-07-26T11:22:56.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.988+0000 7f74562a7740 1 -- 172.21.15.134:0/3353462867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d9371b0 msgr2=0x55e17dd18c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:56.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.988+0000 7f74562a7740 1 --2- 172.21.15.134:0/3353462867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d9371b0 0x55e17dd18c30 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f743c00e7e0 tx=0x7f743c0046c0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.988+0000 7f74562a7740 1 -- 172.21.15.134:0/3353462867 shutdown_connections 2024-07-26T11:22:56.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.988+0000 7f74562a7740 1 --2- 172.21.15.134:0/3353462867 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55e17d91e9f0 0x55e17dd2b070 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.988+0000 7f742e000640 0 client.26479 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:22:56.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.988+0000 7f74562a7740 1 --2- 172.21.15.134:0/3353462867 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d960070 0x55e17dd196b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.988+0000 7f74562a7740 1 --2- 172.21.15.134:0/3353462867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d93f390 0x55e17dd19170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.988+0000 7f74562a7740 1 --2- 172.21.15.134:0/3353462867 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d9371b0 0x55e17dd18c30 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:56.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.988+0000 7f74562a7740 1 -- 172.21.15.134:0/3353462867 >> 172.21.15.134:0/3353462867 conn(0x55e17d954460 msgr2=0x55e17dd10840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:56.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.989+0000 7f74562a7740 1 -- 172.21.15.134:0/3353462867 shutdown_connections 2024-07-26T11:22:56.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.990+0000 7f74562a7740 1 -- 172.21.15.134:0/3353462867 wait complete. 2024-07-26T11:22:56.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:56.990+0000 7f74562a7740 20 client.26479 trim_cache size 0 max 16384 2024-07-26T11:22:57.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.018+0000 7f74562a7740 1 Processor -- start 2024-07-26T11:22:57.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.018+0000 7f74562a7740 1 -- start start 2024-07-26T11:22:57.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.018+0000 7f74562a7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d9371b0 0x55e17dd1a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.018+0000 7f74562a7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d93f390 0x55e17dd12740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.019+0000 7f74562a7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d960070 0x55e17dd12c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.019+0000 7f74562a7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f74481b6f80 con 0x55e17d960070 2024-07-26T11:22:57.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.019+0000 7f74562a7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f74481b8ac0 con 0x55e17d9371b0 2024-07-26T11:22:57.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.019+0000 7f74562a7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f74481b7650 con 0x55e17d93f390 2024-07-26T11:22:57.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.019+0000 7f744cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d9371b0 0x55e17dd1a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.019+0000 7f7447e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d93f390 0x55e17dd12740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.020+0000 7f744cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d9371b0 0x55e17dd1a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56866/0 (socket says 172.21.15.134:56866) 2024-07-26T11:22:57.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.020+0000 7f744cc00640 1 -- 172.21.15.134:0/3340288510 learned_addr learned my addr 172.21.15.134:0/3340288510 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:57.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.020+0000 7f744cc00640 1 -- 172.21.15.134:0/3340288510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d93f390 msgr2=0x55e17dd12740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:57.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.020+0000 7f744cc00640 1 --2- 172.21.15.134:0/3340288510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d93f390 0x55e17dd12740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.020+0000 7f744cc00640 1 -- 172.21.15.134:0/3340288510 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d960070 msgr2=0x55e17dd12c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:57.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.021+0000 7f744cc00640 1 --2- 172.21.15.134:0/3340288510 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d960070 0x55e17dd12c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.021+0000 7f744cc00640 1 -- 172.21.15.134:0/3340288510 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e17dd17130 con 0x55e17d9371b0 2024-07-26T11:22:57.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.022+0000 7f744cc00640 1 --2- 172.21.15.134:0/3340288510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d9371b0 0x55e17dd1a860 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f743c0268a0 tx=0x7f743c026e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=625b1fb80f940052 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:56 smithi193 ceph-mon[37473]: pgmap v689: 129 pgs: 129 active+clean; 52 GiB data, 110 GiB used, 963 GiB / 1.0 TiB avail; 43 KiB/s rd, 262 MiB/s wr, 1.33k op/s 2024-07-26T11:22:57.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.022+0000 7f7439200640 1 -- 172.21.15.134:0/3340288510 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c039070 con 0x55e17d9371b0 2024-07-26T11:22:57.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.023+0000 7f7439200640 1 -- 172.21.15.134:0/3340288510 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f743c00a3c0 con 0x55e17d9371b0 2024-07-26T11:22:57.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.023+0000 7f7439200640 1 -- 172.21.15.134:0/3340288510 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f743c026cc0 con 0x55e17d9371b0 2024-07-26T11:22:57.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.024+0000 7f74562a7740 1 -- 172.21.15.134:0/3340288510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d9371b0 msgr2=0x55e17dd1a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:57.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.024+0000 7f74562a7740 1 --2- 172.21.15.134:0/3340288510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d9371b0 0x55e17dd1a860 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f743c0268a0 tx=0x7f743c026e60 comp rx=0 tx=0).stop 2024-07-26T11:22:57.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.024+0000 7f74562a7740 1 -- 172.21.15.134:0/3340288510 shutdown_connections 2024-07-26T11:22:57.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.024+0000 7f74562a7740 1 --2- 172.21.15.134:0/3340288510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d93f390 0x55e17dd12740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.024+0000 7f74562a7740 1 --2- 172.21.15.134:0/3340288510 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d960070 0x55e17dd12c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.024+0000 7f74562a7740 1 --2- 172.21.15.134:0/3340288510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d9371b0 0x55e17dd1a860 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.024+0000 7f74562a7740 1 -- 172.21.15.134:0/3340288510 >> 172.21.15.134:0/3340288510 conn(0x55e17d954460 msgr2=0x55e17dd1a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:57.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.024+0000 7f74562a7740 1 -- 172.21.15.134:0/3340288510 shutdown_connections 2024-07-26T11:22:57.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.025+0000 7f74562a7740 1 -- 172.21.15.134:0/3340288510 wait complete. 2024-07-26T11:22:57.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.025+0000 7f74562a7740 1 -- start start 2024-07-26T11:22:57.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.026+0000 7f74562a7740 1 Processor -- start 2024-07-26T11:22:57.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.027+0000 7f74562a7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d9371b0 0x55e17dd1c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.028+0000 7f744cc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d9371b0 0x55e17dd1c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.028+0000 7f744cc00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d9371b0 0x55e17dd1c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:46886/0 (socket says 172.21.15.134:46886) 2024-07-26T11:22:57.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.028+0000 7f744cc00640 1 -- 172.21.15.134:0/654206435 learned_addr learned my addr 172.21.15.134:0/654206435 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:57.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.028+0000 7f74562a7740 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d93f390 0x55e17dd1cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.028+0000 7f74562a7740 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d960070 0x55e17dd07430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.028+0000 7f74562a7740 1 -- 172.21.15.134:0/654206435 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f74481b83f0 con 0x55e17d9371b0 2024-07-26T11:22:57.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.028+0000 7f74562a7740 1 -- 172.21.15.134:0/654206435 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f74481b7d20 con 0x55e17d960070 2024-07-26T11:22:57.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.028+0000 7f74562a7740 1 -- 172.21.15.134:0/654206435 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f74481b4f60 con 0x55e17d93f390 2024-07-26T11:22:57.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.028+0000 7f74562a7740 10 client.0 _ll_register_callbacks cb 0x7f743c010660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:57.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.029+0000 7f744d600640 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d960070 0x55e17dd07430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.030+0000 7f744d600640 1 -- 172.21.15.134:0/654206435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d93f390 msgr2=0x55e17dd1cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:57.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.030+0000 7f7447e00640 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d93f390 0x55e17dd1cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.030+0000 7f744d600640 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d93f390 0x55e17dd1cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.030+0000 7f744d600640 1 -- 172.21.15.134:0/654206435 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d9371b0 msgr2=0x55e17dd1c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:57.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.030+0000 7f744d600640 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d9371b0 0x55e17dd1c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.030+0000 7f744d600640 1 -- 172.21.15.134:0/654206435 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e17dd07970 con 0x55e17d960070 2024-07-26T11:22:57.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.030+0000 7f744cc00640 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d9371b0 0x55e17dd1c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:22:57.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.030+0000 7f7447e00640 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d93f390 0x55e17dd1cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:22:57.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.030+0000 7f744d600640 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d960070 0x55e17dd07430 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7440007780 tx=0x7f7440007c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87332db8c532f70e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:57.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.031+0000 7f742e000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:22:57.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.031+0000 7f742e000640 1 -- 172.21.15.134:0/654206435 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7440037040 con 0x55e17d960070 2024-07-26T11:22:57.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.031+0000 7f742e000640 1 -- 172.21.15.134:0/654206435 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f7440032740 con 0x55e17d960070 2024-07-26T11:22:57.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.031+0000 7f74562a7740 1 -- 172.21.15.134:0/654206435 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f743c00a3c0 con 0x55e17d960070 2024-07-26T11:22:57.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.031+0000 7f742e000640 1 -- 172.21.15.134:0/654206435 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7440032940 con 0x55e17d960070 2024-07-26T11:22:57.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.031+0000 7f74562a7740 10 client.26486 fetch_fsmap learned FSMap version 32 2024-07-26T11:22:57.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.031+0000 7f74562a7740 1 -- 172.21.15.134:0/654206435 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e17dd08bf0 con 0x55e17d960070 2024-07-26T11:22:57.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.032+0000 7f742e000640 1 -- 172.21.15.134:0/654206435 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f743c00a3c0 con 0x55e17d960070 2024-07-26T11:22:57.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.032+0000 7f742e000640 1 -- 172.21.15.134:0/654206435 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f7440008530 con 0x55e17d960070 2024-07-26T11:22:57.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.032+0000 7f74562a7740 10 client.26486 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:22:57.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.032+0000 7f74562a7740 10 client.26486 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-07-26T11:22:57.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.032+0000 7f74562a7740 20 client.26486 populate_metadata read hostname 'smithi134' 2024-07-26T11:22:57.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.032+0000 7f74562a7740 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55e17d8f3da0 0x55e17dd0b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.032+0000 7f74562a7740 4 client.26486 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:22:57.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.032+0000 7f74562a7740 1 -- 172.21.15.134:0/654206435 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e17dc795e0 con 0x55e17d8f3da0 2024-07-26T11:22:57.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.032+0000 7f744cc00640 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55e17d8f3da0 0x55e17dd0b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.033+0000 7f742e000640 10 client.26486 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:22:57.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.034+0000 7f744cc00640 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55e17d8f3da0 0x55e17dd0b360 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1620c19028853d9f server_cookie=2424525888a3bf74 in_seq=0 out_seq=0 2024-07-26T11:22:57.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.034+0000 7f742e000640 1 -- 172.21.15.134:0/654206435 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e17dc795e0 con 0x55e17d8f3da0 2024-07-26T11:22:57.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.034+0000 7f742e000640 10 client.26486 handle_command_reply: tid=0 2024-07-26T11:22:57.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.034+0000 7f74562a7740 1 client.26486 shutdown 2024-07-26T11:22:57.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.035+0000 7f74562a7740 1 -- 172.21.15.134:0/654206435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d960070 msgr2=0x55e17dd07430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:57.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.035+0000 7f74562a7740 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d960070 0x55e17dd07430 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7440007780 tx=0x7f7440007c40 comp rx=0 tx=0).stop 2024-07-26T11:22:57.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.035+0000 7f74562a7740 1 -- 172.21.15.134:0/654206435 shutdown_connections 2024-07-26T11:22:57.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.035+0000 7f74562a7740 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55e17d8f3da0 0x55e17dd0b360 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.035+0000 7f74562a7740 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e17d93f390 0x55e17dd1cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.035+0000 7f74562a7740 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e17d960070 0x55e17dd07430 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.035+0000 7f74562a7740 1 --2- 172.21.15.134:0/654206435 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e17d9371b0 0x55e17dd1c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.035+0000 7f74562a7740 1 -- 172.21.15.134:0/654206435 >> 172.21.15.134:0/654206435 conn(0x55e17d954460 msgr2=0x55e17dd0f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:57.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.035+0000 7f742e000640 0 client.26486 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:22:57.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.035+0000 7f74562a7740 1 -- 172.21.15.134:0/654206435 shutdown_connections 2024-07-26T11:22:57.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.035+0000 7f74562a7740 1 -- 172.21.15.134:0/654206435 wait complete. 2024-07-26T11:22:57.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.036+0000 7f74562a7740 20 client.26486 trim_cache size 0 max 16384 2024-07-26T11:22:57.039 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:22:57.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.036+0000 7f744f600640 1 -- 172.21.15.134:0/1471494071 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f74300b3f10 msgr2=0x7f74300b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:57.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.036+0000 7f744f600640 1 --2- 172.21.15.134:0/1471494071 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f74300b3f10 0x7f74300b6420 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f743c0058c0 tx=0x7f743c005ec0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.036+0000 7f744f600640 1 -- 172.21.15.134:0/1471494071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f744810a740 msgr2=0x7f74481a09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:57.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.036+0000 7f744f600640 1 --2- 172.21.15.134:0/1471494071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f744810a740 0x7f74481a09f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7434009f70 tx=0x7f743400edd0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.037+0000 7f744f600640 1 -- 172.21.15.134:0/1471494071 shutdown_connections 2024-07-26T11:22:57.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.037+0000 7f744f600640 1 --2- 172.21.15.134:0/1471494071 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f74300b3f10 0x7f74300b6420 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.037+0000 7f744f600640 1 --2- 172.21.15.134:0/1471494071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f744810a740 0x7f74481a09f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.037+0000 7f744f600640 1 --2- 172.21.15.134:0/1471494071 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f744810eb80 0x7f74481a5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.037+0000 7f744f600640 1 --2- 172.21.15.134:0/1471494071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7448109df0 0x7f74481a0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.037+0000 7f744f600640 1 -- 172.21.15.134:0/1471494071 >> 172.21.15.134:0/1471494071 conn(0x7f7448070d20 msgr2=0x7f744810ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:57.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.037+0000 7f744f600640 1 -- 172.21.15.134:0/1471494071 shutdown_connections 2024-07-26T11:22:57.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:22:57.037+0000 7f744f600640 1 -- 172.21.15.134:0/1471494071 wait complete. 2024-07-26T11:22:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.540+0000 7f4265200640 1 -- 172.21.15.134:0/1879539685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f426010f6b0 msgr2=0x7f426010fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:57.544 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.540+0000 7f4265200640 1 --2- 172.21.15.134:0/1879539685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f426010f6b0 0x7f426010fac0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f424c008570 tx=0x7f424c02c3e0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.543+0000 7f4265200640 1 -- 172.21.15.134:0/1879539685 shutdown_connections 2024-07-26T11:22:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.543+0000 7f4265200640 1 --2- 172.21.15.134:0/1879539685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4260110000 0x7f4260116830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.543+0000 7f4265200640 1 --2- 172.21.15.134:0/1879539685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f426010f6b0 0x7f426010fac0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.543+0000 7f4265200640 1 --2- 172.21.15.134:0/1879539685 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f426010a090 0x7f426010a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.543+0000 7f4265200640 1 -- 172.21.15.134:0/1879539685 >> 172.21.15.134:0/1879539685 conn(0x7f4260070a80 msgr2=0x7f42600735b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.543+0000 7f4265200640 1 -- 172.21.15.134:0/1879539685 shutdown_connections 2024-07-26T11:22:57.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.543+0000 7f4265200640 1 -- 172.21.15.134:0/1879539685 wait complete. 2024-07-26T11:22:57.550 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.545+0000 7f4265200640 1 Processor -- start 2024-07-26T11:22:57.550 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.548+0000 7f4265200640 1 -- start start 2024-07-26T11:22:57.551 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.549+0000 7f4265200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f426010a090 0x7f42601a0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.551 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.549+0000 7f4265200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f426010f6b0 0x7f42601a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.551 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.549+0000 7f4265200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4260110000 0x7f42601a5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.550+0000 7f425f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f426010a090 0x7f42601a0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.550+0000 7f425f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f426010a090 0x7f42601a0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:45760/0 (socket says 172.21.15.134:45760) 2024-07-26T11:22:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.550+0000 7f425fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4260110000 0x7f42601a5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.550+0000 7f425fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4260110000 0x7f42601a5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56882/0 (socket says 172.21.15.134:56882) 2024-07-26T11:22:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.550+0000 7f4265200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4260119860 con 0x7f426010f6b0 2024-07-26T11:22:57.553 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.550+0000 7f425f400640 1 -- 172.21.15.134:0/4238142752 learned_addr learned my addr 172.21.15.134:0/4238142752 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:57.553 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.550+0000 7f425ea00640 1 --2- 172.21.15.134:0/4238142752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f426010f6b0 0x7f42601a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.553 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.550+0000 7f4265200640 1 -- 172.21.15.134:0/4238142752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f42601196e0 con 0x7f4260110000 2024-07-26T11:22:57.553 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.550+0000 7f4265200640 1 -- 172.21.15.134:0/4238142752 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f42601199e0 con 0x7f426010a090 2024-07-26T11:22:57.554 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.551+0000 7f425fe00640 1 -- 172.21.15.134:0/4238142752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f426010a090 msgr2=0x7f42601a0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:57.554 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.552+0000 7f425fe00640 1 --2- 172.21.15.134:0/4238142752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f426010a090 0x7f42601a0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.554 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.552+0000 7f425fe00640 1 -- 172.21.15.134:0/4238142752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f426010f6b0 msgr2=0x7f42601a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:57.554 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.552+0000 7f425fe00640 1 --2- 172.21.15.134:0/4238142752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f426010f6b0 0x7f42601a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.554 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.552+0000 7f425fe00640 1 -- 172.21.15.134:0/4238142752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42601a60c0 con 0x7f4260110000 2024-07-26T11:22:57.554 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.552+0000 7f425f400640 1 --2- 172.21.15.134:0/4238142752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f426010a090 0x7f42601a0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:22:57.555 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.553+0000 7f425fe00640 1 --2- 172.21.15.134:0/4238142752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4260110000 0x7f42601a5a50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f425000d680 tx=0x7f425000db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:57.555 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.553+0000 7f4257e00640 1 -- 172.21.15.134:0/4238142752 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4250017070 con 0x7f4260110000 2024-07-26T11:22:57.556 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.554+0000 7f4265200640 1 -- 172.21.15.134:0/4238142752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42601a63b0 con 0x7f4260110000 2024-07-26T11:22:57.556 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.554+0000 7f4265200640 1 -- 172.21.15.134:0/4238142752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42601acbd0 con 0x7f4260110000 2024-07-26T11:22:57.556 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.554+0000 7f4257e00640 1 -- 172.21.15.134:0/4238142752 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4250009820 con 0x7f4260110000 2024-07-26T11:22:57.556 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.554+0000 7f4257e00640 1 -- 172.21.15.134:0/4238142752 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4250004ce0 con 0x7f4260110000 2024-07-26T11:22:57.576 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.570+0000 7f4257e00640 1 -- 172.21.15.134:0/4238142752 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f42500099d0 con 0x7f4260110000 2024-07-26T11:22:57.576 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.574+0000 7f426d808740 1 Processor -- start 2024-07-26T11:22:57.576 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.574+0000 7f426d808740 1 -- start start 2024-07-26T11:22:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.575+0000 7f426d808740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22f211b0 0x560d232c4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.575+0000 7f426d808740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22ef82f0 0x560d232c5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.576+0000 7f426d808740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22f004d0 0x560d232c56a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.576+0000 7f426d808740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560d23239270 con 0x560d22ef82f0 2024-07-26T11:22:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.576+0000 7f426d808740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560d23228770 con 0x560d22f004d0 2024-07-26T11:22:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.576+0000 7f426d808740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560d2322c7b0 con 0x560d22f211b0 2024-07-26T11:22:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.576+0000 7f425ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22f211b0 0x560d232c4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.576+0000 7f425ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22f211b0 0x560d232c4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:45764/0 (socket says 172.21.15.134:45764) 2024-07-26T11:22:57.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.576+0000 7f425ea00640 1 -- 172.21.15.134:0/4015242319 learned_addr learned my addr 172.21.15.134:0/4015242319 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:57.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.576+0000 7f425ea00640 1 --2- 172.21.15.134:0/4015242319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22f004d0 0x560d232c56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.577+0000 7f4257e00640 1 --2- 172.21.15.134:0/4238142752 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f42440b3f10 0x7f42440b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.578+0000 7f425fe00640 1 --2- 172.21.15.134:0/4238142752 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f42440b3f10 0x7f42440b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.579+0000 7f425ea00640 1 -- 172.21.15.134:0/4015242319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22f004d0 msgr2=0x560d232c56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.579+0000 7f425ea00640 1 --2- 172.21.15.134:0/4015242319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22f004d0 0x560d232c56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.579+0000 7f425ea00640 1 -- 172.21.15.134:0/4015242319 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22ef82f0 msgr2=0x560d232c5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.579+0000 7f425ea00640 1 --2- 172.21.15.134:0/4015242319 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22ef82f0 0x560d232c5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.579+0000 7f425ea00640 1 -- 172.21.15.134:0/4015242319 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d22f8c790 con 0x560d22f211b0 2024-07-26T11:22:57.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.581+0000 7f425ea00640 1 --2- 172.21.15.134:0/4015242319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22f211b0 0x560d232c4c20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f426006c0f0 tx=0x7f424c00c040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5402eb12c7240944 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.583+0000 7f4254c00640 1 -- 172.21.15.134:0/4015242319 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424c00a0d0 con 0x560d22f211b0 2024-07-26T11:22:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.583+0000 7f4254c00640 1 -- 172.21.15.134:0/4015242319 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f424c00a270 con 0x560d22f211b0 2024-07-26T11:22:57.585 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.583+0000 7f4254c00640 1 -- 172.21.15.134:0/4015242319 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424c005590 con 0x560d22f211b0 2024-07-26T11:22:57.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.583+0000 7f4257e00640 1 -- 172.21.15.134:0/4238142752 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f42500e3c70 con 0x7f4260110000 2024-07-26T11:22:57.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.584+0000 7f426d808740 1 -- 172.21.15.134:0/4015242319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22f211b0 msgr2=0x560d232c4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:57.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.584+0000 7f426d808740 1 --2- 172.21.15.134:0/4015242319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22f211b0 0x560d232c4c20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f426006c0f0 tx=0x7f424c00c040 comp rx=0 tx=0).stop 2024-07-26T11:22:57.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.585+0000 7f426d808740 1 -- 172.21.15.134:0/4015242319 shutdown_connections 2024-07-26T11:22:57.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.585+0000 7f426d808740 1 --2- 172.21.15.134:0/4015242319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22f004d0 0x560d232c56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.588 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.585+0000 7f426d808740 1 --2- 172.21.15.134:0/4015242319 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22ef82f0 0x560d232c5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.588 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.586+0000 7f426d808740 1 --2- 172.21.15.134:0/4015242319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22f211b0 0x560d232c4c20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.588 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.586+0000 7f426d808740 1 -- 172.21.15.134:0/4015242319 >> 172.21.15.134:0/4015242319 conn(0x560d22f15300 msgr2=0x560d232c2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:57.588 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.586+0000 7f425fe00640 1 --2- 172.21.15.134:0/4238142752 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f42440b3f10 0x7f42440b6420 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f42500a7710 tx=0x7f42500aa270 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.587+0000 7f426d808740 1 -- 172.21.15.134:0/4015242319 shutdown_connections 2024-07-26T11:22:57.591 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.589+0000 7f426d808740 1 -- 172.21.15.134:0/4015242319 wait complete. 2024-07-26T11:22:57.592 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.590+0000 7f426d808740 1 -- start start 2024-07-26T11:22:57.592 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.590+0000 7f426d808740 1 Processor -- start 2024-07-26T11:22:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.592+0000 7f426d808740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22ef82f0 0x560d232d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.592+0000 7f426d808740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22f004d0 0x560d232d0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.592+0000 7f426d808740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22f211b0 0x560d232da9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.593+0000 7f426d808740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x560d23162ab0 con 0x560d22f004d0 2024-07-26T11:22:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.593+0000 7f426d808740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560d231c5d50 con 0x560d22ef82f0 2024-07-26T11:22:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.593+0000 7f426d808740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x560d232d76f0 con 0x560d22f211b0 2024-07-26T11:22:57.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.593+0000 7f426d808740 10 client.0 _ll_register_callbacks cb 0x560d22f38a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.593+0000 7f425ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22ef82f0 0x560d232d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.594+0000 7f425f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22f004d0 0x560d232d0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.594+0000 7f425ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22f211b0 0x560d232da9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.594+0000 7f425ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22f211b0 0x560d232da9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:45778/0 (socket says 172.21.15.134:45778) 2024-07-26T11:22:57.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.594+0000 7f425ea00640 1 -- 172.21.15.134:0/3924459467 learned_addr learned my addr 172.21.15.134:0/3924459467 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.596+0000 7f425ea00640 1 -- 172.21.15.134:0/3924459467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22ef82f0 msgr2=0x560d232d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.597+0000 7f425ea00640 1 --2- 172.21.15.134:0/3924459467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22ef82f0 0x560d232d08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.597+0000 7f425ea00640 1 -- 172.21.15.134:0/3924459467 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22f004d0 msgr2=0x560d232d0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.597+0000 7f425ea00640 1 --2- 172.21.15.134:0/3924459467 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22f004d0 0x560d232d0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.599 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.597+0000 7f425ea00640 1 -- 172.21.15.134:0/3924459467 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d23005620 con 0x560d22f211b0 2024-07-26T11:22:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.598+0000 7f425ea00640 1 --2- 172.21.15.134:0/3924459467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22f211b0 0x560d232da9b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f424c00ac40 tx=0x7f424c005780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4908a74c8a1e982 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.598+0000 7f4249600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:22:57.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.599+0000 7f4249600640 1 -- 172.21.15.134:0/3924459467 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424c056060 con 0x560d22f211b0 2024-07-26T11:22:57.601 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.599+0000 7f4249600640 1 -- 172.21.15.134:0/3924459467 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f424c005510 con 0x560d22f211b0 2024-07-26T11:22:57.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.601+0000 7f426d808740 1 -- 172.21.15.134:0/3924459467 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f424c00a270 con 0x560d22f211b0 2024-07-26T11:22:57.603 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.601+0000 7f4249600640 1 -- 172.21.15.134:0/3924459467 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f424c004040 con 0x560d22f211b0 2024-07-26T11:22:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.603+0000 7f426d808740 10 client.26494 fetch_fsmap learned FSMap version 32 2024-07-26T11:22:57.605 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.603+0000 7f426d808740 1 -- 172.21.15.134:0/3924459467 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560d232e37f0 con 0x560d22f211b0 2024-07-26T11:22:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.604+0000 7f4249600640 1 -- 172.21.15.134:0/3924459467 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f424c00a270 con 0x560d22f211b0 2024-07-26T11:22:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.604+0000 7f4249600640 1 -- 172.21.15.134:0/3924459467 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f424c04b050 con 0x560d22f211b0 2024-07-26T11:22:57.606 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.604+0000 7f426d808740 10 client.26494 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:22:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.615+0000 7f426d808740 10 client.26494 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:22:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.615+0000 7f426d808740 20 client.26494 populate_metadata read hostname 'smithi134' 2024-07-26T11:22:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.615+0000 7f426d808740 1 --2- 172.21.15.134:0/3924459467 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560d22edfd80 0x560d232e6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.615+0000 7f426d808740 4 client.26494 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:22:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.615+0000 7f426d808740 1 -- 172.21.15.134:0/3924459467 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d2323b1c0 con 0x560d22edfd80 2024-07-26T11:22:57.618 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.616+0000 7f425f400640 1 --2- 172.21.15.134:0/3924459467 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560d22edfd80 0x560d232e6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.620+0000 7f4249600640 10 client.26494 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:22:57.622 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.620+0000 7f425f400640 1 --2- 172.21.15.134:0/3924459467 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560d22edfd80 0x560d232e6b50 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc173237fd6a9e6f server_cookie=337780338726217a in_seq=0 out_seq=0 2024-07-26T11:22:57.624 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.622+0000 7f4249600640 1 -- 172.21.15.134:0/3924459467 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x560d2323b1c0 con 0x560d22edfd80 2024-07-26T11:22:57.624 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.622+0000 7f4249600640 10 client.26494 handle_command_reply: tid=0 2024-07-26T11:22:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.628+0000 7f426d808740 1 client.26494 shutdown 2024-07-26T11:22:57.630 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.628+0000 7f426d808740 1 -- 172.21.15.134:0/3924459467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22f211b0 msgr2=0x560d232da9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.628+0000 7f426d808740 1 --2- 172.21.15.134:0/3924459467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22f211b0 0x560d232da9b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f424c00ac40 tx=0x7f424c005780 comp rx=0 tx=0).stop 2024-07-26T11:22:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.629+0000 7f426d808740 1 -- 172.21.15.134:0/3924459467 shutdown_connections 2024-07-26T11:22:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.629+0000 7f426d808740 1 --2- 172.21.15.134:0/3924459467 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560d22edfd80 0x560d232e6b50 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.629+0000 7f426d808740 1 --2- 172.21.15.134:0/3924459467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22f211b0 0x560d232da9b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.629+0000 7f426d808740 1 --2- 172.21.15.134:0/3924459467 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22f004d0 0x560d232d0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.631 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.629+0000 7f426d808740 1 --2- 172.21.15.134:0/3924459467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22ef82f0 0x560d232d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.629+0000 7f426d808740 1 -- 172.21.15.134:0/3924459467 >> 172.21.15.134:0/3924459467 conn(0x560d22f15300 msgr2=0x560d232c5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.629+0000 7f4249600640 0 client.26494 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:22:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.629+0000 7f426d808740 1 -- 172.21.15.134:0/3924459467 shutdown_connections 2024-07-26T11:22:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.630+0000 7f426d808740 1 -- 172.21.15.134:0/3924459467 wait complete. 2024-07-26T11:22:57.632 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.630+0000 7f426d808740 20 client.26494 trim_cache size 0 max 16384 2024-07-26T11:22:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.685+0000 7f426d808740 1 Processor -- start 2024-07-26T11:22:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.685+0000 7f426d808740 1 -- start start 2024-07-26T11:22:57.689 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.688+0000 7f426d808740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22ef82f0 0x560d232da920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.688+0000 7f426d808740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22f004d0 0x560d232ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.688+0000 7f426d808740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22eb51c0 0x560d232c2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.688+0000 7f426d808740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f42601b6ca0 con 0x560d22eb51c0 2024-07-26T11:22:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.688+0000 7f426d808740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f42601b87e0 con 0x560d22f004d0 2024-07-26T11:22:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.688+0000 7f426d808740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f42601b7370 con 0x560d22ef82f0 2024-07-26T11:22:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.688+0000 7f425ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22ef82f0 0x560d232da920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.688+0000 7f425ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22ef82f0 0x560d232da920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:45782/0 (socket says 172.21.15.134:45782) 2024-07-26T11:22:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.689+0000 7f425ea00640 1 -- 172.21.15.134:0/3585875795 learned_addr learned my addr 172.21.15.134:0/3585875795 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.689+0000 7f425f400640 1 --2- 172.21.15.134:0/3585875795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22f004d0 0x560d232ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.689+0000 7f425ea00640 1 -- 172.21.15.134:0/3585875795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22f004d0 msgr2=0x560d232ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.689+0000 7f425ea00640 1 --2- 172.21.15.134:0/3585875795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22f004d0 0x560d232ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.689+0000 7f425ea00640 1 -- 172.21.15.134:0/3585875795 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22eb51c0 msgr2=0x560d232c2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.689+0000 7f425ea00640 1 --2- 172.21.15.134:0/3585875795 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22eb51c0 0x560d232c2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.689+0000 7f425ea00640 1 -- 172.21.15.134:0/3585875795 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d232d0c30 con 0x560d22ef82f0 2024-07-26T11:22:57.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.689+0000 7f425f400640 1 --2- 172.21.15.134:0/3585875795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22f004d0 0x560d232ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:22:57.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.689+0000 7f425ea00640 1 --2- 172.21.15.134:0/3585875795 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22ef82f0 0x560d232da920 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f424c0050a0 tx=0x7f424c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=617c5a6750368303 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:57.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.691+0000 7f424be00640 1 -- 172.21.15.134:0/3585875795 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424c056040 con 0x560d22ef82f0 2024-07-26T11:22:57.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.691+0000 7f424be00640 1 -- 172.21.15.134:0/3585875795 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f424c00a270 con 0x560d22ef82f0 2024-07-26T11:22:57.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.691+0000 7f424be00640 1 -- 172.21.15.134:0/3585875795 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424c0044b0 con 0x560d22ef82f0 2024-07-26T11:22:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.693+0000 7f426d808740 1 -- 172.21.15.134:0/3585875795 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22ef82f0 msgr2=0x560d232da920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:57.695 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.693+0000 7f426d808740 1 --2- 172.21.15.134:0/3585875795 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22ef82f0 0x560d232da920 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f424c0050a0 tx=0x7f424c005fd0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.694+0000 7f426d808740 1 -- 172.21.15.134:0/3585875795 shutdown_connections 2024-07-26T11:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.694+0000 7f426d808740 1 --2- 172.21.15.134:0/3585875795 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22eb51c0 0x560d232c2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.694+0000 7f426d808740 1 --2- 172.21.15.134:0/3585875795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22f004d0 0x560d232ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.694+0000 7f426d808740 1 --2- 172.21.15.134:0/3585875795 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22ef82f0 0x560d232da920 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.694+0000 7f426d808740 1 -- 172.21.15.134:0/3585875795 >> 172.21.15.134:0/3585875795 conn(0x560d22f15300 msgr2=0x560d232c9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.695+0000 7f426d808740 1 -- 172.21.15.134:0/3585875795 shutdown_connections 2024-07-26T11:22:57.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.695+0000 7f426d808740 1 -- 172.21.15.134:0/3585875795 wait complete. 2024-07-26T11:22:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.697+0000 7f426d808740 1 -- start start 2024-07-26T11:22:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.697+0000 7f426d808740 1 Processor -- start 2024-07-26T11:22:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.699+0000 7f426d808740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22eb51c0 0x560d232de380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.699+0000 7f426d808740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22ef82f0 0x560d232de8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.699+0000 7f425ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22eb51c0 0x560d232de380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.699+0000 7f425ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22eb51c0 0x560d232de380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:45798/0 (socket says 172.21.15.134:45798) 2024-07-26T11:22:57.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.699+0000 7f426d808740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22f004d0 0x560d2334d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.699+0000 7f426d808740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f42601b8110 con 0x560d22f004d0 2024-07-26T11:22:57.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.699+0000 7f426d808740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f42601b7a40 con 0x560d22ef82f0 2024-07-26T11:22:57.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.699+0000 7f426d808740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f42601b4c80 con 0x560d22eb51c0 2024-07-26T11:22:57.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.700+0000 7f426d808740 10 client.0 _ll_register_callbacks cb 0x7f424c009dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:22:57.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.700+0000 7f425f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22ef82f0 0x560d232de8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.700+0000 7f425f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22ef82f0 0x560d232de8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56918/0 (socket says 172.21.15.134:56918) 2024-07-26T11:22:57.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.700+0000 7f425f400640 1 -- 172.21.15.134:0/2979640746 learned_addr learned my addr 172.21.15.134:0/2979640746 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:22:57.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.701+0000 7f425f400640 1 -- 172.21.15.134:0/2979640746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22eb51c0 msgr2=0x560d232de380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:22:57.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.701+0000 7f425f400640 1 --2- 172.21.15.134:0/2979640746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22eb51c0 0x560d232de380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.701+0000 7f425f400640 1 -- 172.21.15.134:0/2979640746 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22f004d0 msgr2=0x560d2334d090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:22:57.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.701+0000 7f425ea00640 1 --2- 172.21.15.134:0/2979640746 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22f004d0 0x560d2334d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.704 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.702+0000 7f425f400640 1 --2- 172.21.15.134:0/2979640746 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22f004d0 0x560d2334d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.704 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.702+0000 7f425f400640 1 -- 172.21.15.134:0/2979640746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d2334d650 con 0x560d22ef82f0 2024-07-26T11:22:57.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.703+0000 7f425f400640 1 --2- 172.21.15.134:0/2979640746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22ef82f0 0x560d232de8c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4258007770 tx=0x7f4258004410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc860de597df850e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:22:57.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.703+0000 7f4249600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:22:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.705+0000 7f4249600640 1 -- 172.21.15.134:0/2979640746 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4258004d20 con 0x560d22ef82f0 2024-07-26T11:22:57.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.705+0000 7f4249600640 1 -- 172.21.15.134:0/2979640746 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 9423+0+0 (secure 0 0 0) 0x7f4258004b20 con 0x560d22ef82f0 2024-07-26T11:22:57.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.706+0000 7f426d808740 1 -- 172.21.15.134:0/2979640746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f424c00a270 con 0x560d22ef82f0 2024-07-26T11:22:57.712 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.710+0000 7f4249600640 1 -- 172.21.15.134:0/2979640746 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f4258004d20 con 0x560d22ef82f0 2024-07-26T11:22:57.712 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.710+0000 7f426d808740 10 client.26501 fetch_fsmap learned FSMap version 32 2024-07-26T11:22:57.712 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.710+0000 7f426d808740 1 -- 172.21.15.134:0/2979640746 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x560d2334e8c0 con 0x560d22ef82f0 2024-07-26T11:22:57.712 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.710+0000 7f4249600640 1 -- 172.21.15.134:0/2979640746 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f424c00a270 con 0x560d22ef82f0 2024-07-26T11:22:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.711+0000 7f4249600640 1 -- 172.21.15.134:0/2979640746 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f4258013750 con 0x560d22ef82f0 2024-07-26T11:22:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.712+0000 7f426d808740 10 client.26501 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:22:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.712+0000 7f426d808740 10 client.26501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:22:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.712+0000 7f426d808740 20 client.26501 populate_metadata read hostname 'smithi134' 2024-07-26T11:22:57.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.712+0000 7f426d808740 1 --2- 172.21.15.134:0/2979640746 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560d2334f020 0x560d23351470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:22:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.714+0000 7f426d808740 4 client.26501 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:22:57.715 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.714+0000 7f426d808740 1 -- 172.21.15.134:0/2979640746 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560d23236c60 con 0x560d2334f020 2024-07-26T11:22:57.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.715+0000 7f425ea00640 1 --2- 172.21.15.134:0/2979640746 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560d2334f020 0x560d23351470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:22:57.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.718+0000 7f425ea00640 1 --2- 172.21.15.134:0/2979640746 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560d2334f020 0x560d23351470 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea3b2ca37cf65429 server_cookie=26d9ed003f462d26 in_seq=0 out_seq=0 2024-07-26T11:22:57.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.718+0000 7f4249600640 10 client.26501 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:22:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.907+0000 7f4249600640 1 -- 172.21.15.134:0/2979640746 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560d23236c60 con 0x560d2334f020 2024-07-26T11:22:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.907+0000 7f4249600640 10 client.26501 handle_command_reply: tid=0 2024-07-26T11:22:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.907+0000 7f426d808740 1 client.26501 shutdown 2024-07-26T11:22:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.907+0000 7f426d808740 1 -- 172.21.15.134:0/2979640746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22ef82f0 msgr2=0x560d232de8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.907+0000 7f426d808740 1 --2- 172.21.15.134:0/2979640746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22ef82f0 0x560d232de8c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4258007770 tx=0x7f4258004410 comp rx=0 tx=0).stop 2024-07-26T11:22:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.908+0000 7f426d808740 1 -- 172.21.15.134:0/2979640746 shutdown_connections 2024-07-26T11:22:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.908+0000 7f426d808740 1 --2- 172.21.15.134:0/2979640746 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x560d2334f020 0x560d23351470 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.908+0000 7f426d808740 1 --2- 172.21.15.134:0/2979640746 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x560d22f004d0 0x560d2334d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.908+0000 7f426d808740 1 --2- 172.21.15.134:0/2979640746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d22ef82f0 0x560d232de8c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.908+0000 7f426d808740 1 --2- 172.21.15.134:0/2979640746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d22eb51c0 0x560d232de380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.908+0000 7f426d808740 1 -- 172.21.15.134:0/2979640746 >> 172.21.15.134:0/2979640746 conn(0x560d22f15300 msgr2=0x560d232c2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.908+0000 7f4249600640 0 client.26501 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:22:57.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.908+0000 7f426d808740 1 -- 172.21.15.134:0/2979640746 shutdown_connections 2024-07-26T11:22:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.908+0000 7f426d808740 1 -- 172.21.15.134:0/2979640746 wait complete. 2024-07-26T11:22:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.909+0000 7f426d808740 20 client.26501 trim_cache size 0 max 16384 2024-07-26T11:22:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:22:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:22:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:22:57.912 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:22:57.915 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.913+0000 7f4265200640 1 -- 172.21.15.134:0/4238142752 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f42440b3f10 msgr2=0x7f42440b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.913+0000 7f4265200640 1 --2- 172.21.15.134:0/4238142752 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f42440b3f10 0x7f42440b6420 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f42500a7710 tx=0x7f42500aa270 comp rx=0 tx=0).stop 2024-07-26T11:22:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.913+0000 7f4265200640 1 -- 172.21.15.134:0/4238142752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4260110000 msgr2=0x7f42601a5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:22:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.913+0000 7f4265200640 1 --2- 172.21.15.134:0/4238142752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4260110000 0x7f42601a5a50 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f425000d680 tx=0x7f425000db40 comp rx=0 tx=0).stop 2024-07-26T11:22:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.913+0000 7f4265200640 1 -- 172.21.15.134:0/4238142752 shutdown_connections 2024-07-26T11:22:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.913+0000 7f4265200640 1 --2- 172.21.15.134:0/4238142752 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f42440b3f10 0x7f42440b6420 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.913+0000 7f4265200640 1 --2- 172.21.15.134:0/4238142752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4260110000 0x7f42601a5a50 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.913+0000 7f4265200640 1 --2- 172.21.15.134:0/4238142752 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f426010f6b0 0x7f42601a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.913+0000 7f4265200640 1 --2- 172.21.15.134:0/4238142752 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f426010a090 0x7f42601a0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:22:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.913+0000 7f4265200640 1 -- 172.21.15.134:0/4238142752 >> 172.21.15.134:0/4238142752 conn(0x7f4260070a80 msgr2=0x7f426010eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:22:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.914+0000 7f4265200640 1 -- 172.21.15.134:0/4238142752 shutdown_connections 2024-07-26T11:22:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:22:57.915+0000 7f4265200640 1 -- 172.21.15.134:0/4238142752 wait complete. 2024-07-26T11:22:58.679 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:58.679 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:22:59.680 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:22:59 smithi134 ceph-mon[31966]: pgmap v690: 129 pgs: 129 active+clean; 53 GiB data, 110 GiB used, 963 GiB / 1.0 TiB avail; 54 KiB/s rd, 237 MiB/s wr, 1.22k op/s 2024-07-26T11:22:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:22:59 smithi189 ceph-mon[36790]: pgmap v690: 129 pgs: 129 active+clean; 53 GiB data, 110 GiB used, 963 GiB / 1.0 TiB avail; 54 KiB/s rd, 237 MiB/s wr, 1.22k op/s 2024-07-26T11:22:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:22:59 smithi193 ceph-mon[37473]: pgmap v690: 129 pgs: 129 active+clean; 53 GiB data, 110 GiB used, 963 GiB / 1.0 TiB avail; 54 KiB/s rd, 237 MiB/s wr, 1.22k op/s 2024-07-26T11:23:01.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:01 smithi134 ceph-mon[31966]: pgmap v691: 129 pgs: 129 active+clean; 53 GiB data, 111 GiB used, 962 GiB / 1.0 TiB avail; 30 KiB/s rd, 286 MiB/s wr, 1.49k op/s 2024-07-26T11:23:01.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:01 smithi189 ceph-mon[36790]: pgmap v691: 129 pgs: 129 active+clean; 53 GiB data, 111 GiB used, 962 GiB / 1.0 TiB avail; 30 KiB/s rd, 286 MiB/s wr, 1.49k op/s 2024-07-26T11:23:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:01 smithi193 ceph-mon[37473]: pgmap v691: 129 pgs: 129 active+clean; 53 GiB data, 111 GiB used, 962 GiB / 1.0 TiB avail; 30 KiB/s rd, 286 MiB/s wr, 1.49k op/s 2024-07-26T11:23:02.926 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:02 smithi134 ceph-mon[31966]: pgmap v692: 129 pgs: 129 active+clean; 53 GiB data, 111 GiB used, 962 GiB / 1.0 TiB avail; 50 KiB/s rd, 222 MiB/s wr, 1.16k op/s 2024-07-26T11:23:02.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:02 smithi189 ceph-mon[36790]: pgmap v692: 129 pgs: 129 active+clean; 53 GiB data, 111 GiB used, 962 GiB / 1.0 TiB avail; 50 KiB/s rd, 222 MiB/s wr, 1.16k op/s 2024-07-26T11:23:03.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:02 smithi193 ceph-mon[37473]: pgmap v692: 129 pgs: 129 active+clean; 53 GiB data, 111 GiB used, 962 GiB / 1.0 TiB avail; 50 KiB/s rd, 222 MiB/s wr, 1.16k op/s 2024-07-26T11:23:03.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-26T11:23:04.092 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:23:05.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:05 smithi134 ceph-mon[31966]: pgmap v693: 129 pgs: 129 active+clean; 53 GiB data, 111 GiB used, 962 GiB / 1.0 TiB avail; 51 KiB/s rd, 276 MiB/s wr, 1.46k op/s 2024-07-26T11:23:05.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:05 smithi189 ceph-mon[36790]: pgmap v693: 129 pgs: 129 active+clean; 53 GiB data, 111 GiB used, 962 GiB / 1.0 TiB avail; 51 KiB/s rd, 276 MiB/s wr, 1.46k op/s 2024-07-26T11:23:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:05 smithi193 ceph-mon[37473]: pgmap v693: 129 pgs: 129 active+clean; 53 GiB data, 111 GiB used, 962 GiB / 1.0 TiB avail; 51 KiB/s rd, 276 MiB/s wr, 1.46k op/s 2024-07-26T11:23:07.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:07 smithi134 ceph-mon[31966]: pgmap v694: 129 pgs: 129 active+clean; 53 GiB data, 112 GiB used, 961 GiB / 1.0 TiB avail; 44 KiB/s rd, 276 MiB/s wr, 1.50k op/s 2024-07-26T11:23:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:07 smithi189 ceph-mon[36790]: pgmap v694: 129 pgs: 129 active+clean; 53 GiB data, 112 GiB used, 961 GiB / 1.0 TiB avail; 44 KiB/s rd, 276 MiB/s wr, 1.50k op/s 2024-07-26T11:23:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:07 smithi193 ceph-mon[37473]: pgmap v694: 129 pgs: 129 active+clean; 53 GiB data, 112 GiB used, 961 GiB / 1.0 TiB avail; 44 KiB/s rd, 276 MiB/s wr, 1.50k op/s 2024-07-26T11:23:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:09 smithi134 ceph-mon[31966]: pgmap v695: 129 pgs: 129 active+clean; 53 GiB data, 112 GiB used, 961 GiB / 1.0 TiB avail; 58 KiB/s rd, 242 MiB/s wr, 1.33k op/s 2024-07-26T11:23:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:09 smithi189 ceph-mon[36790]: pgmap v695: 129 pgs: 129 active+clean; 53 GiB data, 112 GiB used, 961 GiB / 1.0 TiB avail; 58 KiB/s rd, 242 MiB/s wr, 1.33k op/s 2024-07-26T11:23:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:09 smithi193 ceph-mon[37473]: pgmap v695: 129 pgs: 129 active+clean; 53 GiB data, 112 GiB used, 961 GiB / 1.0 TiB avail; 58 KiB/s rd, 242 MiB/s wr, 1.33k op/s 2024-07-26T11:23:11.356 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.354+0000 7f38f6600640 1 -- 172.21.15.134:0/3694455655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38e8008960 msgr2=0x7f38e8006d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.354+0000 7f38f6600640 1 --2- 172.21.15.134:0/3694455655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38e8008960 0x7f38e8006d80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f38e4009900 tx=0x7f38e402a440 comp rx=0 tx=0).stop 2024-07-26T11:23:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.355+0000 7f38f6600640 1 -- 172.21.15.134:0/3694455655 shutdown_connections 2024-07-26T11:23:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.355+0000 7f38f6600640 1 --2- 172.21.15.134:0/3694455655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38e80a6a30 0x7f38e80a8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.355+0000 7f38f6600640 1 --2- 172.21.15.134:0/3694455655 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38e80073a0 0x7f38e80077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.355+0000 7f38f6600640 1 --2- 172.21.15.134:0/3694455655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38e8008960 0x7f38e8006d80 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.355+0000 7f38f6600640 1 -- 172.21.15.134:0/3694455655 >> 172.21.15.134:0/3694455655 conn(0x7f38e8091b70 msgr2=0x7f38e8093fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.356+0000 7f38f6600640 1 -- 172.21.15.134:0/3694455655 shutdown_connections 2024-07-26T11:23:11.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.356+0000 7f38f6600640 1 -- 172.21.15.134:0/3694455655 wait complete. 2024-07-26T11:23:11.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.358+0000 7f38f6600640 1 Processor -- start 2024-07-26T11:23:11.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.359+0000 7f38f6600640 1 -- start start 2024-07-26T11:23:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.360+0000 7f38f6600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38e80073a0 0x7f38e81330e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.362 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.360+0000 7f38f6600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38e8008960 0x7f38e8133680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.361+0000 7f38f5200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38e80073a0 0x7f38e81330e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.361+0000 7f38f5200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38e80073a0 0x7f38e81330e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50082/0 (socket says 172.21.15.134:50082) 2024-07-26T11:23:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.361+0000 7f38efe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38e8008960 0x7f38e8133680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:11.363 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.361+0000 7f38efe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38e8008960 0x7f38e8133680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:41676/0 (socket says 172.21.15.134:41676) 2024-07-26T11:23:11.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.363+0000 7f38f6600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38e80a6a30 0x7f38e8138910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.363+0000 7f38f5c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38e80a6a30 0x7f38e8138910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:11.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.363+0000 7f38f5c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38e80a6a30 0x7f38e8138910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51208/0 (socket says 172.21.15.134:51208) 2024-07-26T11:23:11.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.363+0000 7f38f5c00640 1 -- 172.21.15.134:0/2619670162 learned_addr learned my addr 172.21.15.134:0/2619670162 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:11.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.363+0000 7f38f6600640 1 -- 172.21.15.134:0/2619670162 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f38e80ac640 con 0x7f38e80073a0 2024-07-26T11:23:11.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.363+0000 7f38f6600640 1 -- 172.21.15.134:0/2619670162 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f38e80ac4c0 con 0x7f38e8008960 2024-07-26T11:23:11.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.363+0000 7f38f6600640 1 -- 172.21.15.134:0/2619670162 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f38e80ac7c0 con 0x7f38e80a6a30 2024-07-26T11:23:11.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.364+0000 7f38f5c00640 1 -- 172.21.15.134:0/2619670162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38e8008960 msgr2=0x7f38e8133680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:11.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.364+0000 7f38f5c00640 1 --2- 172.21.15.134:0/2619670162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38e8008960 0x7f38e8133680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.364+0000 7f38f5c00640 1 -- 172.21.15.134:0/2619670162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38e80073a0 msgr2=0x7f38e81330e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:11.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.364+0000 7f38f5c00640 1 --2- 172.21.15.134:0/2619670162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38e80073a0 0x7f38e81330e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.364+0000 7f38f5200640 1 --2- 172.21.15.134:0/2619670162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38e80073a0 0x7f38e81330e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:23:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.364+0000 7f38f5c00640 1 -- 172.21.15.134:0/2619670162 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38e8139010 con 0x7f38e80a6a30 2024-07-26T11:23:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.365+0000 7f38efe00640 1 --2- 172.21.15.134:0/2619670162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38e8008960 0x7f38e8133680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:23:11.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.365+0000 7f38f5c00640 1 --2- 172.21.15.134:0/2619670162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38e80a6a30 0x7f38e8138910 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f38f0052340 tx=0x7f38f007c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.366+0000 7f38ed600640 1 -- 172.21.15.134:0/2619670162 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f006c170 con 0x7f38e80a6a30 2024-07-26T11:23:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.366+0000 7f38ed600640 1 -- 172.21.15.134:0/2619670162 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38f0083030 con 0x7f38e80a6a30 2024-07-26T11:23:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.366+0000 7f38ed600640 1 -- 172.21.15.134:0/2619670162 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f007f790 con 0x7f38e80a6a30 2024-07-26T11:23:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.366+0000 7f38f6600640 1 -- 172.21.15.134:0/2619670162 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38e8139300 con 0x7f38e80a6a30 2024-07-26T11:23:11.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.366+0000 7f38f6600640 1 -- 172.21.15.134:0/2619670162 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38e813fa90 con 0x7f38e80a6a30 2024-07-26T11:23:11.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.374+0000 7f38ed600640 1 -- 172.21.15.134:0/2619670162 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f38f007f930 con 0x7f38e80a6a30 2024-07-26T11:23:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.375+0000 7f38febd9740 1 Processor -- start 2024-07-26T11:23:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.375+0000 7f38febd9740 1 -- start start 2024-07-26T11:23:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.375+0000 7f38febd9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd88381b0 0x559dd8bdbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.375+0000 7f38febd9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd880f2f0 0x559dd8bdc180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.375+0000 7f38febd9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88174d0 0x559dd8bdc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.375+0000 7f38febd9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559dd8b50270 con 0x559dd88174d0 2024-07-26T11:23:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.375+0000 7f38febd9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559dd8b3f770 con 0x559dd88381b0 2024-07-26T11:23:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.375+0000 7f38febd9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559dd8b437b0 con 0x559dd880f2f0 2024-07-26T11:23:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.377+0000 7f38ed600640 1 --2- 172.21.15.134:0/2619670162 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f38d80b3df0 0x7f38d80b6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.377+0000 7f38ed600640 1 -- 172.21.15.134:0/2619670162 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9599+0+0 (secure 0 0 0) 0x7f38f01516e0 con 0x7f38e80a6a30 2024-07-26T11:23:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.378+0000 7f38efe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd88381b0 0x559dd8bdbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:11.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.378+0000 7f38efe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd88381b0 0x559dd8bdbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:41686/0 (socket says 172.21.15.134:41686) 2024-07-26T11:23:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.378+0000 7f38efe00640 1 -- 172.21.15.134:0/2825773096 learned_addr learned my addr 172.21.15.134:0/2825773096 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.378+0000 7f38f5c00640 1 --2- 172.21.15.134:0/2619670162 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f38d80b3df0 0x7f38d80b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.379+0000 7f38f5200640 1 --2- 172.21.15.134:0/2825773096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd880f2f0 0x559dd8bdc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.379+0000 7f38efe00640 1 -- 172.21.15.134:0/2825773096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd880f2f0 msgr2=0x559dd8bdc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.379+0000 7f38efe00640 1 --2- 172.21.15.134:0/2825773096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd880f2f0 0x559dd8bdc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.379+0000 7f38efe00640 1 -- 172.21.15.134:0/2825773096 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88174d0 msgr2=0x559dd8bdc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.379+0000 7f38efe00640 1 --2- 172.21.15.134:0/2825773096 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88174d0 0x559dd8bdc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.379+0000 7f38efe00640 1 -- 172.21.15.134:0/2825773096 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dd88a3790 con 0x559dd88381b0 2024-07-26T11:23:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.380+0000 7f38efe00640 1 --2- 172.21.15.134:0/2825773096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd88381b0 0x559dd8bdbc40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f38dc00bcd0 tx=0x7f38dc0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ce37806e02dfef6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:11.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.380+0000 7f38f5c00640 1 --2- 172.21.15.134:0/2619670162 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f38d80b3df0 0x7f38d80b6300 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f38f0068b30 tx=0x7f38f0069050 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.381+0000 7f38e2000640 1 -- 172.21.15.134:0/2825773096 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38dc004220 con 0x559dd88381b0 2024-07-26T11:23:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.381+0000 7f38e2000640 1 -- 172.21.15.134:0/2825773096 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38dc015030 con 0x559dd88381b0 2024-07-26T11:23:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.381+0000 7f38e2000640 1 -- 172.21.15.134:0/2825773096 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38dc004d30 con 0x559dd88381b0 2024-07-26T11:23:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.382+0000 7f38febd9740 1 -- 172.21.15.134:0/2825773096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd88381b0 msgr2=0x559dd8bdbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:11.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.383+0000 7f38febd9740 1 --2- 172.21.15.134:0/2825773096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd88381b0 0x559dd8bdbc40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f38dc00bcd0 tx=0x7f38dc0107a0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.385+0000 7f38febd9740 1 -- 172.21.15.134:0/2825773096 shutdown_connections 2024-07-26T11:23:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.385+0000 7f38febd9740 1 --2- 172.21.15.134:0/2825773096 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd880f2f0 0x559dd8bdc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.385+0000 7f38febd9740 1 --2- 172.21.15.134:0/2825773096 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88174d0 0x559dd8bdc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.385+0000 7f38febd9740 1 --2- 172.21.15.134:0/2825773096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd88381b0 0x559dd8bdbc40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.385+0000 7f38febd9740 1 -- 172.21.15.134:0/2825773096 >> 172.21.15.134:0/2825773096 conn(0x559dd882c300 msgr2=0x559dd8bd9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.386+0000 7f38febd9740 1 -- 172.21.15.134:0/2825773096 shutdown_connections 2024-07-26T11:23:11.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.386+0000 7f38febd9740 1 -- 172.21.15.134:0/2825773096 wait complete. 2024-07-26T11:23:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.387+0000 7f38febd9740 1 -- start start 2024-07-26T11:23:11.389 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.387+0000 7f38febd9740 1 Processor -- start 2024-07-26T11:23:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.390+0000 7f38febd9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd880f2f0 0x559dd8be7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.390+0000 7f38febd9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88174d0 0x559dd8be83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.390+0000 7f38febd9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd88381b0 0x559dd8bed400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.390+0000 7f38febd9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559dd8a79ab0 con 0x559dd88174d0 2024-07-26T11:23:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.390+0000 7f38febd9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559dd8adcd50 con 0x559dd880f2f0 2024-07-26T11:23:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.390+0000 7f38febd9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559dd8be2200 con 0x559dd88381b0 2024-07-26T11:23:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.390+0000 7f38febd9740 10 client.0 _ll_register_callbacks cb 0x559dd884fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:23:11.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.391+0000 7f38f5200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88174d0 0x559dd8be83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.391+0000 7f38f5200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88174d0 0x559dd8be83d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50088/0 (socket says 172.21.15.134:50088) 2024-07-26T11:23:11.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.391+0000 7f38f5200640 1 -- 172.21.15.134:0/1108843740 learned_addr learned my addr 172.21.15.134:0/1108843740 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.392+0000 7f38efe00640 1 --2- 172.21.15.134:0/1108843740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd88381b0 0x559dd8bed400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.392+0000 7f38efe00640 1 --2- 172.21.15.134:0/1108843740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd880f2f0 0x559dd8be7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.392+0000 7f38f5200640 1 -- 172.21.15.134:0/1108843740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd88381b0 msgr2=0x559dd8bed400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.392+0000 7f38f5200640 1 --2- 172.21.15.134:0/1108843740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd88381b0 0x559dd8bed400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.392+0000 7f38f5200640 1 -- 172.21.15.134:0/1108843740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd880f2f0 msgr2=0x559dd8be7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.392+0000 7f38f5200640 1 --2- 172.21.15.134:0/1108843740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd880f2f0 0x559dd8be7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.392+0000 7f38f5200640 1 -- 172.21.15.134:0/1108843740 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dd891c620 con 0x559dd88174d0 2024-07-26T11:23:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.393+0000 7f38f5200640 1 --2- 172.21.15.134:0/1108843740 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88174d0 0x559dd8be83d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f38e4009230 tx=0x7f38e40316c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8f56d47ae032a6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.394+0000 7f38d6a00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:23:11.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.394+0000 7f38d6a00640 1 -- 172.21.15.134:0/1108843740 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e4031c40 con 0x559dd88174d0 2024-07-26T11:23:11.397 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.394+0000 7f38d6a00640 1 -- 172.21.15.134:0/1108843740 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9599+0+0 (secure 0 0 0) 0x7f38e4031de0 con 0x559dd88174d0 2024-07-26T11:23:11.397 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.395+0000 7f38febd9740 1 -- 172.21.15.134:0/1108843740 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38dc015030 con 0x559dd88174d0 2024-07-26T11:23:11.398 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.395+0000 7f38d6a00640 1 -- 172.21.15.134:0/1108843740 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f38e4046440 con 0x559dd88174d0 2024-07-26T11:23:11.398 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.396+0000 7f38febd9740 10 client.16023 fetch_fsmap learned FSMap version 32 2024-07-26T11:23:11.398 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.396+0000 7f38febd9740 1 -- 172.21.15.134:0/1108843740 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559dd8bf63a0 con 0x559dd88174d0 2024-07-26T11:23:11.399 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.397+0000 7f38d6a00640 1 -- 172.21.15.134:0/1108843740 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38dc015030 con 0x559dd88174d0 2024-07-26T11:23:11.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.398+0000 7f38d6a00640 1 -- 172.21.15.134:0/1108843740 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f38e4044070 con 0x559dd88174d0 2024-07-26T11:23:11.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.399+0000 7f38febd9740 10 client.16023 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:23:11.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.399+0000 7f38febd9740 10 client.16023 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:23:11.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.399+0000 7f38febd9740 20 client.16023 populate_metadata read hostname 'smithi134' 2024-07-26T11:23:11.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.399+0000 7f38febd9740 1 --2- 172.21.15.134:0/1108843740 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559dd87f6d80 0x559dd8bf98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.399+0000 7f38febd9740 4 client.16023 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:23:11.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.399+0000 7f38febd9740 1 -- 172.21.15.134:0/1108843740 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559dd8b521c0 con 0x559dd87f6d80 2024-07-26T11:23:11.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.401+0000 7f38efe00640 1 --2- 172.21.15.134:0/1108843740 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559dd87f6d80 0x559dd8bf98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:11.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.405+0000 7f38efe00640 1 --2- 172.21.15.134:0/1108843740 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559dd87f6d80 0x559dd8bf98a0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7fe756fe5a90c4c server_cookie=c308c39b5ca43459 in_seq=0 out_seq=0 2024-07-26T11:23:11.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.406+0000 7f38d6a00640 10 client.16023 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.408+0000 7f38d6a00640 1 -- 172.21.15.134:0/1108843740 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559dd8b521c0 con 0x559dd87f6d80 2024-07-26T11:23:11.410 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.408+0000 7f38d6a00640 10 client.16023 handle_command_reply: tid=0 2024-07-26T11:23:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.408+0000 7f38febd9740 1 client.16023 shutdown 2024-07-26T11:23:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.409+0000 7f38febd9740 1 -- 172.21.15.134:0/1108843740 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88174d0 msgr2=0x559dd8be83d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.409+0000 7f38febd9740 1 --2- 172.21.15.134:0/1108843740 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88174d0 0x559dd8be83d0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f38e4009230 tx=0x7f38e40316c0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.409+0000 7f38febd9740 1 -- 172.21.15.134:0/1108843740 shutdown_connections 2024-07-26T11:23:11.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.409+0000 7f38febd9740 1 --2- 172.21.15.134:0/1108843740 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559dd87f6d80 0x559dd8bf98a0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.409+0000 7f38febd9740 1 --2- 172.21.15.134:0/1108843740 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd88381b0 0x559dd8bed400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.409+0000 7f38febd9740 1 --2- 172.21.15.134:0/1108843740 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88174d0 0x559dd8be83d0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.409+0000 7f38febd9740 1 --2- 172.21.15.134:0/1108843740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd880f2f0 0x559dd8be7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.409+0000 7f38febd9740 1 -- 172.21.15.134:0/1108843740 >> 172.21.15.134:0/1108843740 conn(0x559dd882c300 msgr2=0x559dd8bdd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:11.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.410+0000 7f38d6a00640 0 client.16023 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:11.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.412+0000 7f38febd9740 1 -- 172.21.15.134:0/1108843740 shutdown_connections 2024-07-26T11:23:11.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.413+0000 7f38febd9740 1 -- 172.21.15.134:0/1108843740 wait complete. 2024-07-26T11:23:11.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.413+0000 7f38febd9740 20 client.16023 trim_cache size 0 max 16384 2024-07-26T11:23:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.454+0000 7f38febd9740 1 Processor -- start 2024-07-26T11:23:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.454+0000 7f38febd9740 1 -- start start 2024-07-26T11:23:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.455+0000 7f38febd9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd880f2f0 0x559dd8be6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.455+0000 7f38febd9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd88174d0 0x559dd8be69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.455+0000 7f38febd9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88381b0 0x559dd8bfa080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.455+0000 7f38febd9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f38d00101a0 con 0x559dd88381b0 2024-07-26T11:23:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.455+0000 7f38febd9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f38d0011ce0 con 0x559dd88174d0 2024-07-26T11:23:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.455+0000 7f38febd9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f38d0010870 con 0x559dd880f2f0 2024-07-26T11:23:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.456+0000 7f38efe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd880f2f0 0x559dd8be6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.456+0000 7f38efe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd880f2f0 0x559dd8be6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51252/0 (socket says 172.21.15.134:51252) 2024-07-26T11:23:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.456+0000 7f38efe00640 1 -- 172.21.15.134:0/334156097 learned_addr learned my addr 172.21.15.134:0/334156097 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.456+0000 7f38f5200640 1 --2- 172.21.15.134:0/334156097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd88174d0 0x559dd8be69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.456+0000 7f38efe00640 1 -- 172.21.15.134:0/334156097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd88174d0 msgr2=0x559dd8be69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.456+0000 7f38efe00640 1 --2- 172.21.15.134:0/334156097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd88174d0 0x559dd8be69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.456+0000 7f38efe00640 1 -- 172.21.15.134:0/334156097 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88381b0 msgr2=0x559dd8bfa080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.456+0000 7f38efe00640 1 --2- 172.21.15.134:0/334156097 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88381b0 0x559dd8bfa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.456+0000 7f38efe00640 1 -- 172.21.15.134:0/334156097 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559dd8bedf10 con 0x559dd880f2f0 2024-07-26T11:23:11.459 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.457+0000 7f38efe00640 1 --2- 172.21.15.134:0/334156097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd880f2f0 0x559dd8be6460 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f38dc00bfd0 tx=0x7f38dc00b3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d20ebbc0427e5f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.457+0000 7f38e1600640 1 -- 172.21.15.134:0/334156097 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38dc0047a0 con 0x559dd880f2f0 2024-07-26T11:23:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.457+0000 7f38e1600640 1 -- 172.21.15.134:0/334156097 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38dc015030 con 0x559dd880f2f0 2024-07-26T11:23:11.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.457+0000 7f38e1600640 1 -- 172.21.15.134:0/334156097 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38dc011e40 con 0x559dd880f2f0 2024-07-26T11:23:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.459+0000 7f38febd9740 1 -- 172.21.15.134:0/334156097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd880f2f0 msgr2=0x559dd8be6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.459+0000 7f38febd9740 1 --2- 172.21.15.134:0/334156097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd880f2f0 0x559dd8be6460 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f38dc00bfd0 tx=0x7f38dc00b3c0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.459+0000 7f38febd9740 1 -- 172.21.15.134:0/334156097 shutdown_connections 2024-07-26T11:23:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.459+0000 7f38febd9740 1 --2- 172.21.15.134:0/334156097 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88381b0 0x559dd8bfa080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.459+0000 7f38febd9740 1 --2- 172.21.15.134:0/334156097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd88174d0 0x559dd8be69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.459+0000 7f38febd9740 1 --2- 172.21.15.134:0/334156097 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd880f2f0 0x559dd8be6460 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.459+0000 7f38febd9740 1 -- 172.21.15.134:0/334156097 >> 172.21.15.134:0/334156097 conn(0x559dd882c300 msgr2=0x559dd8be2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.460+0000 7f38febd9740 1 -- 172.21.15.134:0/334156097 shutdown_connections 2024-07-26T11:23:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.460+0000 7f38febd9740 1 -- 172.21.15.134:0/334156097 wait complete. 2024-07-26T11:23:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.460+0000 7f38febd9740 1 -- start start 2024-07-26T11:23:11.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.461+0000 7f38febd9740 1 Processor -- start 2024-07-26T11:23:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.461+0000 7f38febd9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd880f2f0 0x559dd8be9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.462+0000 7f38febd9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd88174d0 0x559dd8bd80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.462+0000 7f38febd9740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88381b0 0x559dd8bd8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.462+0000 7f38febd9740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f38d0011610 con 0x559dd88381b0 2024-07-26T11:23:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.462+0000 7f38febd9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f38d0010f40 con 0x559dd880f2f0 2024-07-26T11:23:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.462+0000 7f38febd9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f38d000e230 con 0x559dd88174d0 2024-07-26T11:23:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.462+0000 7f38febd9740 10 client.0 _ll_register_callbacks cb 0x7f38e4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:23:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.462+0000 7f38efe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd880f2f0 0x559dd8be9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.462+0000 7f38f5200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd88174d0 0x559dd8bd80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.462+0000 7f38efe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd880f2f0 0x559dd8be9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:41718/0 (socket says 172.21.15.134:41718) 2024-07-26T11:23:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.462+0000 7f38efe00640 1 -- 172.21.15.134:0/3824811315 learned_addr learned my addr 172.21.15.134:0/3824811315 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.463+0000 7f38efe00640 1 --2- 172.21.15.134:0/3824811315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88381b0 0x559dd8bd8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.463+0000 7f38efe00640 1 -- 172.21.15.134:0/3824811315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd88174d0 msgr2=0x559dd8bd80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.463+0000 7f38efe00640 1 --2- 172.21.15.134:0/3824811315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd88174d0 0x559dd8bd80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.463+0000 7f38efe00640 1 -- 172.21.15.134:0/3824811315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88381b0 msgr2=0x559dd8bd8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.467+0000 7f38efe00640 1 --2- 172.21.15.134:0/3824811315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88381b0 0x559dd8bd8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.467+0000 7f38efe00640 1 -- 172.21.15.134:0/3824811315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559dd8bd8b50 con 0x559dd880f2f0 2024-07-26T11:23:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.467+0000 7f38d6a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:23:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.467+0000 7f38efe00640 1 --2- 172.21.15.134:0/3824811315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd880f2f0 0x559dd8be9a00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f38dc010ca0 tx=0x7f38dc013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6c5f94883cde74f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.468+0000 7f38d6a00640 1 -- 172.21.15.134:0/3824811315 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38dc004020 con 0x559dd880f2f0 2024-07-26T11:23:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.468+0000 7f38d6a00640 1 -- 172.21.15.134:0/3824811315 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9599+0+0 (secure 0 0 0) 0x7f38dc037020 con 0x559dd880f2f0 2024-07-26T11:23:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.468+0000 7f38febd9740 1 -- 172.21.15.134:0/3824811315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38dc015030 con 0x559dd880f2f0 2024-07-26T11:23:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.469+0000 7f38d6a00640 1 -- 172.21.15.134:0/3824811315 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f38dc004020 con 0x559dd880f2f0 2024-07-26T11:23:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.469+0000 7f38febd9740 10 client.26513 fetch_fsmap learned FSMap version 32 2024-07-26T11:23:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.469+0000 7f38febd9740 1 -- 172.21.15.134:0/3824811315 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559dd8c61ca0 con 0x559dd880f2f0 2024-07-26T11:23:11.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.485+0000 7f38d6a00640 1 -- 172.21.15.134:0/3824811315 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38dc0055a0 con 0x559dd880f2f0 2024-07-26T11:23:11.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.485+0000 7f38d6a00640 1 -- 172.21.15.134:0/3824811315 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f38dc011880 con 0x559dd880f2f0 2024-07-26T11:23:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.486+0000 7f38febd9740 10 client.26513 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:23:11.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.486+0000 7f38febd9740 10 client.26513 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:23:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.486+0000 7f38febd9740 20 client.26513 populate_metadata read hostname 'smithi134' 2024-07-26T11:23:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.486+0000 7f38febd9740 1 --2- 172.21.15.134:0/3824811315 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559dd87cc1c0 0x559dd8c641b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.486+0000 7f38febd9740 4 client.26513 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:23:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.486+0000 7f38febd9740 1 -- 172.21.15.134:0/3824811315 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f38dc015030 con 0x559dd87cc1c0 2024-07-26T11:23:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.487+0000 7f38f5200640 1 --2- 172.21.15.134:0/3824811315 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559dd87cc1c0 0x559dd8c641b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.488+0000 7f38f5200640 1 --2- 172.21.15.134:0/3824811315 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559dd87cc1c0 0x559dd8c641b0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ee9940f764b2987 server_cookie=67d320220c7a6292 in_seq=0 out_seq=0 2024-07-26T11:23:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:11.488+0000 7f38d6a00640 10 client.26513 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:11 smithi134 ceph-mon[31966]: pgmap v696: 129 pgs: 129 active+clean; 54 GiB data, 112 GiB used, 961 GiB / 1.0 TiB avail; 47 KiB/s rd, 314 MiB/s wr, 1.84k op/s 2024-07-26T11:23:11.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:11 smithi134 ceph-mon[31966]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T11:23:11.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:23:11.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:23:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:11 smithi189 ceph-mon[36790]: pgmap v696: 129 pgs: 129 active+clean; 54 GiB data, 112 GiB used, 961 GiB / 1.0 TiB avail; 47 KiB/s rd, 314 MiB/s wr, 1.84k op/s 2024-07-26T11:23:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:11 smithi189 ceph-mon[36790]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T11:23:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:23:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:23:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:11 smithi193 ceph-mon[37473]: pgmap v696: 129 pgs: 129 active+clean; 54 GiB data, 112 GiB used, 961 GiB / 1.0 TiB avail; 47 KiB/s rd, 314 MiB/s wr, 1.84k op/s 2024-07-26T11:23:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:11 smithi193 ceph-mon[37473]: osdmap e91: 12 total, 12 up, 12 in 2024-07-26T11:23:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:23:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:23:12.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.406+0000 7fd05e800640 1 -- 172.21.15.134:0/3004277068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd058108530 msgr2=0x7fd05810ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:12.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.406+0000 7fd05e800640 1 --2- 172.21.15.134:0/3004277068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd058108530 0x7fd05810ed60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd050009ca0 tx=0x7fd05002a8e0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.409+0000 7fd05e800640 1 -- 172.21.15.134:0/3004277068 shutdown_connections 2024-07-26T11:23:12.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.409+0000 7fd05e800640 1 --2- 172.21.15.134:0/3004277068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd058108530 0x7fd05810ed60 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.409+0000 7fd05e800640 1 --2- 172.21.15.134:0/3004277068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd058107be0 0x7fd058107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.409+0000 7fd05e800640 1 --2- 172.21.15.134:0/3004277068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd0581022a0 0x7fd0581026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.409+0000 7fd05e800640 1 -- 172.21.15.134:0/3004277068 >> 172.21.15.134:0/3004277068 conn(0x7fd0580fdcf0 msgr2=0x7fd058100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:12.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.410+0000 7fd05e800640 1 -- 172.21.15.134:0/3004277068 shutdown_connections 2024-07-26T11:23:12.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.411+0000 7fd05e800640 1 -- 172.21.15.134:0/3004277068 wait complete. 2024-07-26T11:23:12.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.412+0000 7fd05e800640 1 Processor -- start 2024-07-26T11:23:12.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.413+0000 7fd05e800640 1 -- start start 2024-07-26T11:23:12.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.415+0000 7fd05e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0581022a0 0x7fd058113b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.415+0000 7fd05e800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd058107be0 0x7fd0581140c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.415+0000 7fd05e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd058108530 0x7fd058114600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.415+0000 7fd05d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0581022a0 0x7fd058113b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.415+0000 7fd05d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0581022a0 0x7fd058113b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:41740/0 (socket says 172.21.15.134:41740) 2024-07-26T11:23:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.416+0000 7fd05e800640 1 -- 172.21.15.134:0/1326352912 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd058111170 con 0x7fd058107be0 2024-07-26T11:23:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.416+0000 7fd05e800640 1 -- 172.21.15.134:0/1326352912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd058110ff0 con 0x7fd0581022a0 2024-07-26T11:23:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.416+0000 7fd05d400640 1 -- 172.21.15.134:0/1326352912 learned_addr learned my addr 172.21.15.134:0/1326352912 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.416+0000 7fd05e800640 1 -- 172.21.15.134:0/1326352912 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd0581112f0 con 0x7fd058108530 2024-07-26T11:23:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.416+0000 7fd05de00640 1 --2- 172.21.15.134:0/1326352912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd058108530 0x7fd058114600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.416+0000 7fd05ca00640 1 --2- 172.21.15.134:0/1326352912 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd058107be0 0x7fd0581140c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.417+0000 7fd05de00640 1 -- 172.21.15.134:0/1326352912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0581022a0 msgr2=0x7fd058113b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.417+0000 7fd05de00640 1 --2- 172.21.15.134:0/1326352912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0581022a0 0x7fd058113b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.417+0000 7fd05de00640 1 -- 172.21.15.134:0/1326352912 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd058107be0 msgr2=0x7fd0581140c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.417+0000 7fd05de00640 1 --2- 172.21.15.134:0/1326352912 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd058107be0 0x7fd0581140c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.417+0000 7fd05de00640 1 -- 172.21.15.134:0/1326352912 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd05811ae70 con 0x7fd058108530 2024-07-26T11:23:12.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.418+0000 7fd05de00640 1 --2- 172.21.15.134:0/1326352912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd058108530 0x7fd058114600 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd05002aed0 tx=0x7fd050003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:12.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.418+0000 7fd056000640 1 -- 172.21.15.134:0/1326352912 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd050004040 con 0x7fd058108530 2024-07-26T11:23:12.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.418+0000 7fd056000640 1 -- 172.21.15.134:0/1326352912 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd050004690 con 0x7fd058108530 2024-07-26T11:23:12.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.418+0000 7fd056000640 1 -- 172.21.15.134:0/1326352912 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd050030c80 con 0x7fd058108530 2024-07-26T11:23:12.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.418+0000 7fd05e800640 1 -- 172.21.15.134:0/1326352912 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd05811b100 con 0x7fd058108530 2024-07-26T11:23:12.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.418+0000 7fd05e800640 1 -- 172.21.15.134:0/1326352912 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd05811b670 con 0x7fd058108530 2024-07-26T11:23:12.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.426+0000 7fd056000640 1 -- 172.21.15.134:0/1326352912 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fd0500041e0 con 0x7fd058108530 2024-07-26T11:23:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.430+0000 7f38d6a00640 1 -- 172.21.15.134:0/3824811315 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f38dc015030 con 0x559dd87cc1c0 2024-07-26T11:23:12.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.430+0000 7f38d6a00640 10 client.26513 handle_command_reply: tid=0 2024-07-26T11:23:12.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.431+0000 7f38febd9740 1 client.26513 shutdown 2024-07-26T11:23:12.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.429+0000 7fd056000640 1 --2- 172.21.15.134:0/1326352912 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd0400b3df0 0x7fd0400b6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.430+0000 7fd056000640 1 -- 172.21.15.134:0/1326352912 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9599+0+0 (secure 0 0 0) 0x7fd050105680 con 0x7fd058108530 2024-07-26T11:23:12.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.432+0000 7fd05d400640 1 --2- 172.21.15.134:0/1326352912 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd0400b3df0 0x7fd0400b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.432+0000 7f38febd9740 1 -- 172.21.15.134:0/3824811315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd880f2f0 msgr2=0x559dd8be9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.432+0000 7f38febd9740 1 --2- 172.21.15.134:0/3824811315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd880f2f0 0x559dd8be9a00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f38dc010ca0 tx=0x7f38dc013040 comp rx=0 tx=0).stop 2024-07-26T11:23:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.432+0000 7f38febd9740 1 -- 172.21.15.134:0/3824811315 shutdown_connections 2024-07-26T11:23:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.432+0000 7f38febd9740 1 --2- 172.21.15.134:0/3824811315 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559dd87cc1c0 0x559dd8c641b0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.432+0000 7f38febd9740 1 --2- 172.21.15.134:0/3824811315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559dd88174d0 0x559dd8bd80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.432+0000 7f38febd9740 1 --2- 172.21.15.134:0/3824811315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559dd88381b0 0x559dd8bd8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.432+0000 7f38febd9740 1 --2- 172.21.15.134:0/3824811315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559dd880f2f0 0x559dd8be9a00 secure :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f38dc010ca0 tx=0x7f38dc013040 comp rx=0 tx=0).stop 2024-07-26T11:23:12.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.432+0000 7f38d6a00640 0 client.26513 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.432+0000 7f38febd9740 1 -- 172.21.15.134:0/3824811315 >> 172.21.15.134:0/3824811315 conn(0x559dd882c300 msgr2=0x559dd8be7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.435+0000 7f38febd9740 1 -- 172.21.15.134:0/3824811315 shutdown_connections 2024-07-26T11:23:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.435+0000 7f38febd9740 1 -- 172.21.15.134:0/3824811315 wait complete. 2024-07-26T11:23:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.435+0000 7fd05d400640 1 --2- 172.21.15.134:0/1326352912 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd0400b3df0 0x7fd0400b6300 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd04c005fd0 tx=0x7fd04c00b480 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:12.438 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.435+0000 7f38febd9740 20 client.26513 trim_cache size 0 max 16384 2024-07-26T11:23:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:23:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:23:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:23:12.439 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:23:12.440 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.438+0000 7f38f6600640 1 -- 172.21.15.134:0/2619670162 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f38d80b3df0 msgr2=0x7f38d80b6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.438+0000 7f38f6600640 1 --2- 172.21.15.134:0/2619670162 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f38d80b3df0 0x7f38d80b6300 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f38f0068b30 tx=0x7f38f0069050 comp rx=0 tx=0).stop 2024-07-26T11:23:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.439+0000 7f38f6600640 1 -- 172.21.15.134:0/2619670162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38e80a6a30 msgr2=0x7f38e8138910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.439+0000 7f38f6600640 1 --2- 172.21.15.134:0/2619670162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38e80a6a30 0x7f38e8138910 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f38f0052340 tx=0x7f38f007c3e0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.439+0000 7f38f6600640 1 -- 172.21.15.134:0/2619670162 shutdown_connections 2024-07-26T11:23:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.439+0000 7f38f6600640 1 --2- 172.21.15.134:0/2619670162 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f38d80b3df0 0x7f38d80b6300 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.439+0000 7f38f6600640 1 --2- 172.21.15.134:0/2619670162 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38e80a6a30 0x7f38e8138910 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.439+0000 7f38f6600640 1 --2- 172.21.15.134:0/2619670162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38e8008960 0x7f38e8133680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.439+0000 7f38f6600640 1 --2- 172.21.15.134:0/2619670162 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f38e80073a0 0x7f38e81330e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.439+0000 7f38f6600640 1 -- 172.21.15.134:0/2619670162 >> 172.21.15.134:0/2619670162 conn(0x7f38e8091b70 msgr2=0x7f38e80a2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.440+0000 7f38f6600640 1 -- 172.21.15.134:0/2619670162 shutdown_connections 2024-07-26T11:23:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:12.440+0000 7f38f6600640 1 -- 172.21.15.134:0/2619670162 wait complete. 2024-07-26T11:23:12.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.443+0000 7fd066a1f740 1 Processor -- start 2024-07-26T11:23:12.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.445+0000 7fd066a1f740 1 -- start start 2024-07-26T11:23:12.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.445+0000 7fd066a1f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3294070 0x556aa3639220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.446+0000 7fd066a1f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa326b1b0 0x556aa3639760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.446+0000 7fd066a1f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa3273390 0x556aa3639ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.446+0000 7fd05d400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3294070 0x556aa3639220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.446+0000 7fd066a1f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556aa35afbf0 con 0x556aa3294070 2024-07-26T11:23:12.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.446+0000 7fd066a1f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556aa359f0f0 con 0x556aa326b1b0 2024-07-26T11:23:12.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.446+0000 7fd066a1f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556aa35a3130 con 0x556aa3273390 2024-07-26T11:23:12.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.446+0000 7fd05de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa3273390 0x556aa3639ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.450+0000 7fd05de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa3273390 0x556aa3639ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51288/0 (socket says 172.21.15.134:51288) 2024-07-26T11:23:12.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.450+0000 7fd05de00640 1 -- 172.21.15.134:0/1508452591 learned_addr learned my addr 172.21.15.134:0/1508452591 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:12.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.447+0000 7fd05ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa326b1b0 0x556aa3639760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.448+0000 7fd05d400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3294070 0x556aa3639220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50124/0 (socket says 172.21.15.134:50124) 2024-07-26T11:23:12.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.450+0000 7fd05de00640 1 -- 172.21.15.134:0/1508452591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa326b1b0 msgr2=0x556aa3639760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:12.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.450+0000 7fd05de00640 1 --2- 172.21.15.134:0/1508452591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa326b1b0 0x556aa3639760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.450+0000 7fd05de00640 1 -- 172.21.15.134:0/1508452591 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3294070 msgr2=0x556aa3639220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:12.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.450+0000 7fd05de00640 1 --2- 172.21.15.134:0/1508452591 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3294070 0x556aa3639220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.450+0000 7fd05de00640 1 -- 172.21.15.134:0/1508452591 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556aa32ff650 con 0x556aa3273390 2024-07-26T11:23:12.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.451+0000 7fd05de00640 1 --2- 172.21.15.134:0/1508452591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa3273390 0x556aa3639ca0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd050004af0 tx=0x7fd05002a8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=feb570d9f936046b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.451+0000 7fd04a600640 1 -- 172.21.15.134:0/1508452591 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0500e6070 con 0x556aa3273390 2024-07-26T11:23:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.451+0000 7fd04a600640 1 -- 172.21.15.134:0/1508452591 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd050036030 con 0x556aa3273390 2024-07-26T11:23:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.452+0000 7fd04a600640 1 -- 172.21.15.134:0/1508452591 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0500d5020 con 0x556aa3273390 2024-07-26T11:23:12.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.453+0000 7fd066a1f740 1 -- 172.21.15.134:0/1508452591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa3273390 msgr2=0x556aa3639ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:12.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.453+0000 7fd066a1f740 1 --2- 172.21.15.134:0/1508452591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa3273390 0x556aa3639ca0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd050004af0 tx=0x7fd05002a8e0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.453+0000 7fd066a1f740 1 -- 172.21.15.134:0/1508452591 shutdown_connections 2024-07-26T11:23:12.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.453+0000 7fd066a1f740 1 --2- 172.21.15.134:0/1508452591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa3273390 0x556aa3639ca0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.453+0000 7fd066a1f740 1 --2- 172.21.15.134:0/1508452591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa326b1b0 0x556aa3639760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.453+0000 7fd066a1f740 1 --2- 172.21.15.134:0/1508452591 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3294070 0x556aa3639220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.453+0000 7fd066a1f740 1 -- 172.21.15.134:0/1508452591 >> 172.21.15.134:0/1508452591 conn(0x556aa3288460 msgr2=0x556aa3637110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:12.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.454+0000 7fd066a1f740 1 -- 172.21.15.134:0/1508452591 shutdown_connections 2024-07-26T11:23:12.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.454+0000 7fd066a1f740 1 -- 172.21.15.134:0/1508452591 wait complete. 2024-07-26T11:23:12.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.455+0000 7fd066a1f740 1 -- start start 2024-07-26T11:23:12.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.456+0000 7fd066a1f740 1 Processor -- start 2024-07-26T11:23:12.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.460+0000 7fd066a1f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa326b1b0 0x556aa3644890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.460+0000 7fd066a1f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa3273390 0x556aa3644dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.460+0000 7fd066a1f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa3294070 0x556aa3650a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.461+0000 7fd066a1f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556aa3402700 con 0x556aa326b1b0 2024-07-26T11:23:12.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.461+0000 7fd05de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa3294070 0x556aa3650a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.461+0000 7fd05de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa3294070 0x556aa3650a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51304/0 (socket says 172.21.15.134:51304) 2024-07-26T11:23:12.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.461+0000 7fd05de00640 1 -- 172.21.15.134:0/1132318818 learned_addr learned my addr 172.21.15.134:0/1132318818 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:12.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.461+0000 7fd05ca00640 1 --2- 172.21.15.134:0/1132318818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa3273390 0x556aa3644dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.461+0000 7fd066a1f740 1 -- 172.21.15.134:0/1132318818 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556aa353b6c0 con 0x556aa3273390 2024-07-26T11:23:12.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.461+0000 7fd066a1f740 1 -- 172.21.15.134:0/1132318818 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556aa36477c0 con 0x556aa3294070 2024-07-26T11:23:12.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.461+0000 7fd05d400640 1 --2- 172.21.15.134:0/1132318818 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa326b1b0 0x556aa3644890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.461+0000 7fd05de00640 1 -- 172.21.15.134:0/1132318818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa3273390 msgr2=0x556aa3644dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:12.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.461+0000 7fd05de00640 1 --2- 172.21.15.134:0/1132318818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa3273390 0x556aa3644dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.461+0000 7fd05de00640 1 -- 172.21.15.134:0/1132318818 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa326b1b0 msgr2=0x556aa3644890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:12.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.461+0000 7fd05de00640 1 --2- 172.21.15.134:0/1132318818 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa326b1b0 0x556aa3644890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.462+0000 7fd05de00640 1 -- 172.21.15.134:0/1132318818 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556aa3552860 con 0x556aa3294070 2024-07-26T11:23:12.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.462+0000 7fd05de00640 1 --2- 172.21.15.134:0/1132318818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa3294070 0x556aa3650a20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd0500cbf00 tx=0x7fd0500cbb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee0d628da9533c4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:12.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.462+0000 7fd066a1f740 10 client.0 _ll_register_callbacks cb 0x556aa354b170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:23:12.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.462+0000 7fd066a1f740 1 -- 172.21.15.134:0/1132318818 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd050036030 con 0x556aa3294070 2024-07-26T11:23:12.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.462+0000 7fd03ea00640 10 client.26530 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:23:12.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.466+0000 7fd03ea00640 1 -- 172.21.15.134:0/1132318818 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0500e6070 con 0x556aa3294070 2024-07-26T11:23:12.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.466+0000 7fd03ea00640 1 -- 172.21.15.134:0/1132318818 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9599+0+0 (secure 0 0 0) 0x7fd0500c8560 con 0x556aa3294070 2024-07-26T11:23:12.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.466+0000 7fd03ea00640 1 -- 172.21.15.134:0/1132318818 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd0500cc670 con 0x556aa3294070 2024-07-26T11:23:12.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.466+0000 7fd066a1f740 10 client.26530 fetch_fsmap learned FSMap version 32 2024-07-26T11:23:12.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.466+0000 7fd066a1f740 1 -- 172.21.15.134:0/1132318818 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=32}) -- 0x556aa3652c40 con 0x556aa3294070 2024-07-26T11:23:12.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.467+0000 7fd03ea00640 1 -- 172.21.15.134:0/1132318818 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fd0500c8200 con 0x556aa3294070 2024-07-26T11:23:12.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.467+0000 7fd066a1f740 10 client.26530 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:23:12.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.467+0000 7fd066a1f740 10 client.26530 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:23:12.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.467+0000 7fd066a1f740 20 client.26530 populate_metadata read hostname 'smithi134' 2024-07-26T11:23:12.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.467+0000 7fd066a1f740 1 --2- 172.21.15.134:0/1132318818 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556aa32529f0 0x556aa3656330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.467+0000 7fd066a1f740 4 client.26530 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:23:12.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.467+0000 7fd066a1f740 1 -- 172.21.15.134:0/1132318818 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556aa35b1b40 con 0x556aa32529f0 2024-07-26T11:23:12.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.468+0000 7fd05d400640 1 --2- 172.21.15.134:0/1132318818 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556aa32529f0 0x556aa3656330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.473+0000 7fd05d400640 1 --2- 172.21.15.134:0/1132318818 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556aa32529f0 0x556aa3656330 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d73f70f72f28bb53 server_cookie=1419991768bd9d90 in_seq=0 out_seq=0 2024-07-26T11:23:12.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.473+0000 7fd03ea00640 10 client.26530 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:12.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.477+0000 7fd03ea00640 1 -- 172.21.15.134:0/1132318818 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556aa35b1b40 con 0x556aa32529f0 2024-07-26T11:23:12.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.477+0000 7fd03ea00640 10 client.26530 handle_command_reply: tid=0 2024-07-26T11:23:12.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.477+0000 7fd066a1f740 1 client.26530 shutdown 2024-07-26T11:23:12.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.477+0000 7fd066a1f740 1 -- 172.21.15.134:0/1132318818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa3294070 msgr2=0x556aa3650a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:12.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.477+0000 7fd066a1f740 1 --2- 172.21.15.134:0/1132318818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa3294070 0x556aa3650a20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd0500cbf00 tx=0x7fd0500cbb40 comp rx=0 tx=0).stop 2024-07-26T11:23:12.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.477+0000 7fd066a1f740 1 -- 172.21.15.134:0/1132318818 shutdown_connections 2024-07-26T11:23:12.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.477+0000 7fd066a1f740 1 --2- 172.21.15.134:0/1132318818 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556aa32529f0 0x556aa3656330 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.477+0000 7fd066a1f740 1 --2- 172.21.15.134:0/1132318818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa3294070 0x556aa3650a20 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.477+0000 7fd066a1f740 1 --2- 172.21.15.134:0/1132318818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa3273390 0x556aa3644dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.477+0000 7fd066a1f740 1 --2- 172.21.15.134:0/1132318818 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa326b1b0 0x556aa3644890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.477+0000 7fd066a1f740 1 -- 172.21.15.134:0/1132318818 >> 172.21.15.134:0/1132318818 conn(0x556aa3288460 msgr2=0x556aa363c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:12.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.478+0000 7fd03ea00640 0 client.26530 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:12.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.478+0000 7fd066a1f740 1 -- 172.21.15.134:0/1132318818 shutdown_connections 2024-07-26T11:23:12.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.478+0000 7fd066a1f740 1 -- 172.21.15.134:0/1132318818 wait complete. 2024-07-26T11:23:12.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.478+0000 7fd066a1f740 20 client.26530 trim_cache size 0 max 16384 2024-07-26T11:23:12.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.517+0000 7fd066a1f740 1 Processor -- start 2024-07-26T11:23:12.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.521+0000 7fd066a1f740 1 -- start start 2024-07-26T11:23:12.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.522+0000 7fd066a1f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa326b1b0 0x556aa3636110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.522+0000 7fd066a1f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa3273390 0x556aa3650810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.522+0000 7fd066a1f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3294070 0x556aa363fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.522+0000 7fd066a1f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd0581bf760 con 0x556aa3294070 2024-07-26T11:23:12.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.522+0000 7fd066a1f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd0581c12a0 con 0x556aa3273390 2024-07-26T11:23:12.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.522+0000 7fd066a1f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd0581bfe30 con 0x556aa326b1b0 2024-07-26T11:23:12.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.522+0000 7fd05ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa3273390 0x556aa3650810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.523+0000 7fd05d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa326b1b0 0x556aa3636110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.523+0000 7fd05d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa326b1b0 0x556aa3636110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51316/0 (socket says 172.21.15.134:51316) 2024-07-26T11:23:12.525 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.523+0000 7fd05d400640 1 -- 172.21.15.134:0/3225394362 learned_addr learned my addr 172.21.15.134:0/3225394362 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:12.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.524+0000 7fd05d400640 1 -- 172.21.15.134:0/3225394362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa3273390 msgr2=0x556aa3650810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:12.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.524+0000 7fd05d400640 1 --2- 172.21.15.134:0/3225394362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa3273390 0x556aa3650810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.524+0000 7fd05d400640 1 -- 172.21.15.134:0/3225394362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3294070 msgr2=0x556aa363fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:12.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.524+0000 7fd05d400640 1 --2- 172.21.15.134:0/3225394362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3294070 0x556aa363fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.524+0000 7fd05d400640 1 -- 172.21.15.134:0/3225394362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556aa363b510 con 0x556aa326b1b0 2024-07-26T11:23:12.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.524+0000 7fd05d400640 1 --2- 172.21.15.134:0/3225394362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa326b1b0 0x556aa3636110 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd05810ed80 tx=0x7fd04c013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62d3b1da6535a4a1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:12.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.525+0000 7fd049c00640 1 -- 172.21.15.134:0/3225394362 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04c005020 con 0x556aa326b1b0 2024-07-26T11:23:12.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.525+0000 7fd049c00640 1 -- 172.21.15.134:0/3225394362 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd04c0047a0 con 0x556aa326b1b0 2024-07-26T11:23:12.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.525+0000 7fd049c00640 1 -- 172.21.15.134:0/3225394362 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04c0282d0 con 0x556aa326b1b0 2024-07-26T11:23:12.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.528+0000 7fd066a1f740 1 -- 172.21.15.134:0/3225394362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa326b1b0 msgr2=0x556aa3636110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:12.530 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.528+0000 7fd066a1f740 1 --2- 172.21.15.134:0/3225394362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa326b1b0 0x556aa3636110 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd05810ed80 tx=0x7fd04c013000 comp rx=0 tx=0).stop 2024-07-26T11:23:12.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.529+0000 7fd066a1f740 1 -- 172.21.15.134:0/3225394362 shutdown_connections 2024-07-26T11:23:12.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.529+0000 7fd066a1f740 1 --2- 172.21.15.134:0/3225394362 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3294070 0x556aa363fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.529+0000 7fd066a1f740 1 --2- 172.21.15.134:0/3225394362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa3273390 0x556aa3650810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.529+0000 7fd066a1f740 1 --2- 172.21.15.134:0/3225394362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa326b1b0 0x556aa3636110 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.529+0000 7fd066a1f740 1 -- 172.21.15.134:0/3225394362 >> 172.21.15.134:0/3225394362 conn(0x556aa3288460 msgr2=0x556aa363f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:12.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.531+0000 7fd066a1f740 1 -- 172.21.15.134:0/3225394362 shutdown_connections 2024-07-26T11:23:12.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.531+0000 7fd066a1f740 1 -- 172.21.15.134:0/3225394362 wait complete. 2024-07-26T11:23:12.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.533+0000 7fd066a1f740 1 -- start start 2024-07-26T11:23:12.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.534+0000 7fd066a1f740 1 Processor -- start 2024-07-26T11:23:12.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.537+0000 7fd066a1f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa326b1b0 0x556aa3647d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.537+0000 7fd066a1f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3273390 0x556aa3648270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.538+0000 7fd066a1f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa3294070 0x556aa36bcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.538+0000 7fd05ca00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3273390 0x556aa3648270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.539+0000 7fd05ca00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3273390 0x556aa3648270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50164/0 (socket says 172.21.15.134:50164) 2024-07-26T11:23:12.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.539+0000 7fd05ca00640 1 -- 172.21.15.134:0/2324935266 learned_addr learned my addr 172.21.15.134:0/2324935266 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:12.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.542+0000 7fd05d400640 1 --2- 172.21.15.134:0/2324935266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa326b1b0 0x556aa3647d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.541+0000 7fd066a1f740 1 -- 172.21.15.134:0/2324935266 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd0581c0bd0 con 0x556aa3273390 2024-07-26T11:23:12.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.546+0000 7fd066a1f740 1 -- 172.21.15.134:0/2324935266 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd0581c0500 con 0x556aa3294070 2024-07-26T11:23:12.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.546+0000 7fd066a1f740 1 -- 172.21.15.134:0/2324935266 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd0581bd760 con 0x556aa326b1b0 2024-07-26T11:23:12.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.546+0000 7fd066a1f740 10 client.0 _ll_register_callbacks cb 0x7fd0500cb990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:23:12.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.546+0000 7fd05de00640 1 --2- 172.21.15.134:0/2324935266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa3294070 0x556aa36bcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.547+0000 7fd05ca00640 1 -- 172.21.15.134:0/2324935266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa326b1b0 msgr2=0x556aa3647d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:12.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.547+0000 7fd05ca00640 1 --2- 172.21.15.134:0/2324935266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa326b1b0 0x556aa3647d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.547+0000 7fd05ca00640 1 -- 172.21.15.134:0/2324935266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa3294070 msgr2=0x556aa36bcaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:12.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.547+0000 7fd05ca00640 1 --2- 172.21.15.134:0/2324935266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa3294070 0x556aa36bcaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.547+0000 7fd05ca00640 1 -- 172.21.15.134:0/2324935266 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556aa36bd030 con 0x556aa3273390 2024-07-26T11:23:12.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.549+0000 7fd05ca00640 1 --2- 172.21.15.134:0/2324935266 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3273390 0x556aa3648270 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd044016700 tx=0x7fd044016bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54231225c46e0de3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:12.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.549+0000 7fd03ea00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:23:12.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.549+0000 7fd03ea00640 1 -- 172.21.15.134:0/2324935266 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd044023060 con 0x556aa3273390 2024-07-26T11:23:12.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.549+0000 7fd03ea00640 1 -- 172.21.15.134:0/2324935266 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9599+0+0 (secure 0 0 0) 0x7fd044014b20 con 0x556aa3273390 2024-07-26T11:23:12.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.550+0000 7fd066a1f740 1 -- 172.21.15.134:0/2324935266 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd04c0047a0 con 0x556aa3273390 2024-07-26T11:23:12.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.552+0000 7fd03ea00640 1 -- 172.21.15.134:0/2324935266 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd04402a050 con 0x556aa3273390 2024-07-26T11:23:12.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.552+0000 7fd066a1f740 10 client.16035 fetch_fsmap learned FSMap version 32 2024-07-26T11:23:12.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.552+0000 7fd066a1f740 1 -- 172.21.15.134:0/2324935266 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x556aa36be1c0 con 0x556aa3273390 2024-07-26T11:23:12.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.553+0000 7fd03ea00640 1 -- 172.21.15.134:0/2324935266 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd04c0047a0 con 0x556aa3273390 2024-07-26T11:23:12.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.553+0000 7fd03ea00640 1 -- 172.21.15.134:0/2324935266 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fd04401d030 con 0x556aa3273390 2024-07-26T11:23:12.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.553+0000 7fd066a1f740 10 client.16035 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:23:12.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.553+0000 7fd066a1f740 10 client.16035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:23:12.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.553+0000 7fd066a1f740 20 client.16035 populate_metadata read hostname 'smithi134' 2024-07-26T11:23:12.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.553+0000 7fd066a1f740 1 --2- 172.21.15.134:0/2324935266 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556aa3227da0 0x556aa36c0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:12.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.553+0000 7fd066a1f740 4 client.16035 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:23:12.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.553+0000 7fd066a1f740 1 -- 172.21.15.134:0/2324935266 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556aa35ad5e0 con 0x556aa3227da0 2024-07-26T11:23:12.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.558+0000 7fd05d400640 1 --2- 172.21.15.134:0/2324935266 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556aa3227da0 0x556aa36c0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:12.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.563+0000 7fd03ea00640 10 client.16035 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:12.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.563+0000 7fd05d400640 1 --2- 172.21.15.134:0/2324935266 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556aa3227da0 0x556aa36c0a30 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=203dff698b9f755e server_cookie=e53ddc216b8a8377 in_seq=0 out_seq=0 2024-07-26T11:23:12.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.564+0000 7fd03ea00640 1 -- 172.21.15.134:0/2324935266 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556aa35ad5e0 con 0x556aa3227da0 2024-07-26T11:23:12.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.564+0000 7fd03ea00640 10 client.16035 handle_command_reply: tid=0 2024-07-26T11:23:12.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.564+0000 7fd066a1f740 1 client.16035 shutdown 2024-07-26T11:23:12.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.567+0000 7fd066a1f740 1 -- 172.21.15.134:0/2324935266 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3273390 msgr2=0x556aa3648270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:12.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.567+0000 7fd066a1f740 1 --2- 172.21.15.134:0/2324935266 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3273390 0x556aa3648270 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd044016700 tx=0x7fd044016bc0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.568+0000 7fd066a1f740 1 -- 172.21.15.134:0/2324935266 shutdown_connections 2024-07-26T11:23:12.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.568+0000 7fd066a1f740 1 --2- 172.21.15.134:0/2324935266 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556aa3227da0 0x556aa36c0a30 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.568+0000 7fd066a1f740 1 --2- 172.21.15.134:0/2324935266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556aa3294070 0x556aa36bcaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.568+0000 7fd066a1f740 1 --2- 172.21.15.134:0/2324935266 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556aa3273390 0x556aa3648270 secure :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd044016700 tx=0x7fd044016bc0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.568+0000 7fd066a1f740 1 --2- 172.21.15.134:0/2324935266 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556aa326b1b0 0x556aa3647d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.568+0000 7fd066a1f740 1 -- 172.21.15.134:0/2324935266 >> 172.21.15.134:0/2324935266 conn(0x556aa3288460 msgr2=0x556aa363cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:12.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.568+0000 7fd03ea00640 0 client.16035 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:12.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.569+0000 7fd066a1f740 1 -- 172.21.15.134:0/2324935266 shutdown_connections 2024-07-26T11:23:12.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.569+0000 7fd066a1f740 1 -- 172.21.15.134:0/2324935266 wait complete. 2024-07-26T11:23:12.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.569+0000 7fd066a1f740 20 client.16035 trim_cache size 0 max 16384 2024-07-26T11:23:12.577 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:23:12.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.574+0000 7fd05e800640 1 -- 172.21.15.134:0/1326352912 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd0400b3df0 msgr2=0x7fd0400b6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:12.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.574+0000 7fd05e800640 1 --2- 172.21.15.134:0/1326352912 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd0400b3df0 0x7fd0400b6300 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fd04c005fd0 tx=0x7fd04c00b480 comp rx=0 tx=0).stop 2024-07-26T11:23:12.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.574+0000 7fd05e800640 1 -- 172.21.15.134:0/1326352912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd058108530 msgr2=0x7fd058114600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:12.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.574+0000 7fd05e800640 1 --2- 172.21.15.134:0/1326352912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd058108530 0x7fd058114600 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd05002aed0 tx=0x7fd050003ba0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.584+0000 7fd05e800640 1 -- 172.21.15.134:0/1326352912 shutdown_connections 2024-07-26T11:23:12.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.584+0000 7fd05e800640 1 --2- 172.21.15.134:0/1326352912 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd0400b3df0 0x7fd0400b6300 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.584+0000 7fd05e800640 1 --2- 172.21.15.134:0/1326352912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd058108530 0x7fd058114600 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.584+0000 7fd05e800640 1 --2- 172.21.15.134:0/1326352912 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd058107be0 0x7fd0581140c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.584+0000 7fd05e800640 1 --2- 172.21.15.134:0/1326352912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0581022a0 0x7fd058113b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:12.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.584+0000 7fd05e800640 1 -- 172.21.15.134:0/1326352912 >> 172.21.15.134:0/1326352912 conn(0x7fd0580fdcf0 msgr2=0x7fd0581072a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:12.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.585+0000 7fd05e800640 1 -- 172.21.15.134:0/1326352912 shutdown_connections 2024-07-26T11:23:12.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:12.585+0000 7fd05e800640 1 -- 172.21.15.134:0/1326352912 wait complete. 2024-07-26T11:23:12.664 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:12 smithi134 ceph-mon[31966]: pgmap v698: 129 pgs: 129 active+clean; 54 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 62 KiB/s rd, 270 MiB/s wr, 1.69k op/s 2024-07-26T11:23:12.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:12 smithi189 ceph-mon[36790]: pgmap v698: 129 pgs: 129 active+clean; 54 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 62 KiB/s rd, 270 MiB/s wr, 1.69k op/s 2024-07-26T11:23:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:12 smithi193 ceph-mon[37473]: pgmap v698: 129 pgs: 129 active+clean; 54 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 62 KiB/s rd, 270 MiB/s wr, 1.69k op/s 2024-07-26T11:23:13.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:23:13.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:23:13.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:23:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:23:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:23:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:23:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:23:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:23:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:23:14.797 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T11:23:14.797 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T11:23:14.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:14 smithi134 ceph-mon[31966]: pgmap v699: 129 pgs: 129 active+clean; 54 GiB data, 113 GiB used, 960 GiB / 1.0 TiB avail; 60 KiB/s rd, 248 MiB/s wr, 1.71k op/s 2024-07-26T11:23:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:14 smithi189 ceph-mon[36790]: pgmap v699: 129 pgs: 129 active+clean; 54 GiB data, 113 GiB used, 960 GiB / 1.0 TiB avail; 60 KiB/s rd, 248 MiB/s wr, 1.71k op/s 2024-07-26T11:23:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:14 smithi193 ceph-mon[37473]: pgmap v699: 129 pgs: 129 active+clean; 54 GiB data, 113 GiB used, 960 GiB / 1.0 TiB avail; 60 KiB/s rd, 248 MiB/s wr, 1.71k op/s 2024-07-26T11:23:16.130 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:23:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:17 smithi134 ceph-mon[31966]: pgmap v700: 129 pgs: 129 active+clean; 54 GiB data, 113 GiB used, 960 GiB / 1.0 TiB avail; 50 KiB/s rd, 244 MiB/s wr, 1.84k op/s 2024-07-26T11:23:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:17 smithi189 ceph-mon[36790]: pgmap v700: 129 pgs: 129 active+clean; 54 GiB data, 113 GiB used, 960 GiB / 1.0 TiB avail; 50 KiB/s rd, 244 MiB/s wr, 1.84k op/s 2024-07-26T11:23:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:17 smithi193 ceph-mon[37473]: pgmap v700: 129 pgs: 129 active+clean; 54 GiB data, 113 GiB used, 960 GiB / 1.0 TiB avail; 50 KiB/s rd, 244 MiB/s wr, 1.84k op/s 2024-07-26T11:23:19.926 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:19 smithi134 ceph-mon[31966]: pgmap v701: 129 pgs: 129 active+clean; 54 GiB data, 113 GiB used, 960 GiB / 1.0 TiB avail; 46 KiB/s rd, 243 MiB/s wr, 1.90k op/s 2024-07-26T11:23:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:19 smithi189 ceph-mon[36790]: pgmap v701: 129 pgs: 129 active+clean; 54 GiB data, 113 GiB used, 960 GiB / 1.0 TiB avail; 46 KiB/s rd, 243 MiB/s wr, 1.90k op/s 2024-07-26T11:23:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:19 smithi189 ceph-mon[36790]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T11:23:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:19 smithi193 ceph-mon[37473]: pgmap v701: 129 pgs: 129 active+clean; 54 GiB data, 113 GiB used, 960 GiB / 1.0 TiB avail; 46 KiB/s rd, 243 MiB/s wr, 1.90k op/s 2024-07-26T11:23:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:19 smithi193 ceph-mon[37473]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T11:23:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:20 smithi134 ceph-mon[31966]: osdmap e92: 12 total, 12 up, 12 in 2024-07-26T11:23:21.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:21 smithi134 ceph-mon[31966]: pgmap v703: 129 pgs: 129 active+clean; 54 GiB data, 113 GiB used, 960 GiB / 1.0 TiB avail; 46 KiB/s rd, 210 MiB/s wr, 1.86k op/s 2024-07-26T11:23:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:21 smithi189 ceph-mon[36790]: pgmap v703: 129 pgs: 129 active+clean; 54 GiB data, 113 GiB used, 960 GiB / 1.0 TiB avail; 46 KiB/s rd, 210 MiB/s wr, 1.86k op/s 2024-07-26T11:23:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:21 smithi193 ceph-mon[37473]: pgmap v703: 129 pgs: 129 active+clean; 54 GiB data, 113 GiB used, 960 GiB / 1.0 TiB avail; 46 KiB/s rd, 210 MiB/s wr, 1.86k op/s 2024-07-26T11:23:22.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:22 smithi134 ceph-mon[31966]: pgmap v704: 129 pgs: 129 active+clean; 54 GiB data, 113 GiB used, 959 GiB / 1.0 TiB avail; 35 KiB/s rd, 218 MiB/s wr, 1.96k op/s 2024-07-26T11:23:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:22 smithi189 ceph-mon[36790]: pgmap v704: 129 pgs: 129 active+clean; 54 GiB data, 113 GiB used, 959 GiB / 1.0 TiB avail; 35 KiB/s rd, 218 MiB/s wr, 1.96k op/s 2024-07-26T11:23:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:22 smithi193 ceph-mon[37473]: pgmap v704: 129 pgs: 129 active+clean; 54 GiB data, 113 GiB used, 959 GiB / 1.0 TiB avail; 35 KiB/s rd, 218 MiB/s wr, 1.96k op/s 2024-07-26T11:23:25.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:25 smithi134 ceph-mon[31966]: pgmap v705: 129 pgs: 129 active+clean; 54 GiB data, 114 GiB used, 959 GiB / 1.0 TiB avail; 35 KiB/s rd, 204 MiB/s wr, 1.90k op/s 2024-07-26T11:23:25.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:25 smithi189 ceph-mon[36790]: pgmap v705: 129 pgs: 129 active+clean; 54 GiB data, 114 GiB used, 959 GiB / 1.0 TiB avail; 35 KiB/s rd, 204 MiB/s wr, 1.90k op/s 2024-07-26T11:23:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:25 smithi193 ceph-mon[37473]: pgmap v705: 129 pgs: 129 active+clean; 54 GiB data, 114 GiB used, 959 GiB / 1.0 TiB avail; 35 KiB/s rd, 204 MiB/s wr, 1.90k op/s 2024-07-26T11:23:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.880+0000 7f8fb2c00640 1 -- 172.21.15.134:0/4197701329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fac10a090 msgr2=0x7f8fac10a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.880+0000 7f8fb2c00640 1 --2- 172.21.15.134:0/4197701329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fac10a090 0x7f8fac10a4a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0009370 tx=0x7f8fa002a3f0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.880+0000 7f8fb2c00640 1 -- 172.21.15.134:0/4197701329 shutdown_connections 2024-07-26T11:23:26.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.880+0000 7f8fb2c00640 1 --2- 172.21.15.134:0/4197701329 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8fac10cdc0 0x7f8fac116850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.880+0000 7f8fb2c00640 1 --2- 172.21.15.134:0/4197701329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fac10c470 0x7f8fac10c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.880+0000 7f8fb2c00640 1 --2- 172.21.15.134:0/4197701329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fac10a090 0x7f8fac10a4a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.880+0000 7f8fb2c00640 1 -- 172.21.15.134:0/4197701329 >> 172.21.15.134:0/4197701329 conn(0x7f8fac070620 msgr2=0x7f8fac070a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.880+0000 7f8fb2c00640 1 -- 172.21.15.134:0/4197701329 shutdown_connections 2024-07-26T11:23:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.880+0000 7f8fb2c00640 1 -- 172.21.15.134:0/4197701329 wait complete. 2024-07-26T11:23:26.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.881+0000 7f8fb2c00640 1 Processor -- start 2024-07-26T11:23:26.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.883+0000 7f8fb2c00640 1 -- start start 2024-07-26T11:23:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.884+0000 7f8fb2c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8fac10a090 0x7f8fac1a3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.884+0000 7f8fb2c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fac10c470 0x7f8fac19ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.884+0000 7f8fb2c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fac10cdc0 0x7f8fac1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.884+0000 7f8fb2c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1198b0 con 0x7f8fac10a090 2024-07-26T11:23:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.884+0000 7f8fb2c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac119730 con 0x7f8fac10cdc0 2024-07-26T11:23:26.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.884+0000 7f8fb2c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac119a30 con 0x7f8fac10c470 2024-07-26T11:23:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.885+0000 7f8fb1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fac10cdc0 0x7f8fac1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.885+0000 7f8fb1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fac10cdc0 0x7f8fac1a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38458/0 (socket says 172.21.15.134:38458) 2024-07-26T11:23:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.885+0000 7f8fb1600640 1 -- 172.21.15.134:0/2432501109 learned_addr learned my addr 172.21.15.134:0/2432501109 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.885+0000 7f8fabe00640 1 --2- 172.21.15.134:0/2432501109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8fac10a090 0x7f8fac1a3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.885+0000 7f8fab400640 1 --2- 172.21.15.134:0/2432501109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fac10c470 0x7f8fac19ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.885+0000 7f8fb1600640 1 -- 172.21.15.134:0/2432501109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fac10c470 msgr2=0x7f8fac19ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.885+0000 7f8fb1600640 1 --2- 172.21.15.134:0/2432501109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fac10c470 0x7f8fac19ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.885+0000 7f8fb1600640 1 -- 172.21.15.134:0/2432501109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8fac10a090 msgr2=0x7f8fac1a3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.885+0000 7f8fb1600640 1 --2- 172.21.15.134:0/2432501109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8fac10a090 0x7f8fac1a3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.886+0000 7f8fb1600640 1 -- 172.21.15.134:0/2432501109 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fac1b4a80 con 0x7f8fac10cdc0 2024-07-26T11:23:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.886+0000 7f8fab400640 1 --2- 172.21.15.134:0/2432501109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fac10c470 0x7f8fac19ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:23:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.887+0000 7f8fb1600640 1 --2- 172.21.15.134:0/2432501109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fac10cdc0 0x7f8fac1a0660 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8fa400cec0 tx=0x7f8fa4008650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.887+0000 7f8fa8c00640 1 -- 172.21.15.134:0/2432501109 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa4017430 con 0x7f8fac10cdc0 2024-07-26T11:23:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.888+0000 7f8fa8c00640 1 -- 172.21.15.134:0/2432501109 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fa4016880 con 0x7f8fac10cdc0 2024-07-26T11:23:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.888+0000 7f8fb2c00640 1 -- 172.21.15.134:0/2432501109 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fac1b4cb0 con 0x7f8fac10cdc0 2024-07-26T11:23:26.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.888+0000 7f8fa8c00640 1 -- 172.21.15.134:0/2432501109 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa40097b0 con 0x7f8fac10cdc0 2024-07-26T11:23:26.892 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.888+0000 7f8fb2c00640 1 -- 172.21.15.134:0/2432501109 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fac1b51b0 con 0x7f8fac10cdc0 2024-07-26T11:23:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.893+0000 7f8fb9b92740 1 Processor -- start 2024-07-26T11:23:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.894+0000 7f8fa8c00640 1 -- 172.21.15.134:0/2432501109 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f8fa40163d0 con 0x7f8fac10cdc0 2024-07-26T11:23:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.895+0000 7f8fb9b92740 1 -- start start 2024-07-26T11:23:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.895+0000 7f8fb9b92740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2ac41b0 0x5646a2e67c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.896+0000 7f8fb9b92740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2a9b2f0 0x5646a2e68160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.896+0000 7f8fb9b92740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2aa34d0 0x5646a2e686a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.896+0000 7f8fb9b92740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5646a2ddc270 con 0x5646a2ac41b0 2024-07-26T11:23:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.896+0000 7f8fb9b92740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5646a2dcb770 con 0x5646a2a9b2f0 2024-07-26T11:23:26.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.896+0000 7f8fb9b92740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5646a2dcf7b0 con 0x5646a2aa34d0 2024-07-26T11:23:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.896+0000 7f8fa8c00640 1 --2- 172.21.15.134:0/2432501109 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8f940b3dc0 0x7f8f940b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.897+0000 7f8fabe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2a9b2f0 0x5646a2e68160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.897+0000 7f8fabe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2a9b2f0 0x5646a2e68160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38472/0 (socket says 172.21.15.134:38472) 2024-07-26T11:23:26.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.897+0000 7f8fabe00640 1 -- 172.21.15.134:0/1677499158 learned_addr learned my addr 172.21.15.134:0/1677499158 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.897+0000 7f8fab400640 1 --2- 172.21.15.134:0/1677499158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2aa34d0 0x5646a2e686a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.897+0000 7f8fab400640 1 --2- 172.21.15.134:0/1677499158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2ac41b0 0x5646a2e67c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.897+0000 7f8fa8c00640 1 -- 172.21.15.134:0/2432501109 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f8fa40e4a10 con 0x7f8fac10cdc0 2024-07-26T11:23:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.898+0000 7f8fabe00640 1 -- 172.21.15.134:0/1677499158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2aa34d0 msgr2=0x5646a2e686a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:26.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.898+0000 7f8fb1600640 1 --2- 172.21.15.134:0/2432501109 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8f940b3dc0 0x7f8f940b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.898+0000 7f8fabe00640 1 --2- 172.21.15.134:0/1677499158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2aa34d0 0x5646a2e686a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.898+0000 7f8fabe00640 1 -- 172.21.15.134:0/1677499158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2ac41b0 msgr2=0x5646a2e67c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.898+0000 7f8fabe00640 1 --2- 172.21.15.134:0/1677499158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2ac41b0 0x5646a2e67c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.898+0000 7f8fabe00640 1 -- 172.21.15.134:0/1677499158 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646a2b2f790 con 0x5646a2a9b2f0 2024-07-26T11:23:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.898+0000 7f8fab400640 1 --2- 172.21.15.134:0/1677499158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2aa34d0 0x5646a2e686a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:23:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.899+0000 7f8fabe00640 1 --2- 172.21.15.134:0/1677499158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2a9b2f0 0x5646a2e68160 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0033000 tx=0x7f8fa0031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66e290602d608d87 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.899+0000 7f8fb1600640 1 --2- 172.21.15.134:0/2432501109 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8f940b3dc0 0x7f8f940b62d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40e1de0 tx=0x7f8fa400cef0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.899+0000 7f8f9d200640 1 -- 172.21.15.134:0/1677499158 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0035070 con 0x5646a2a9b2f0 2024-07-26T11:23:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.899+0000 7f8f9d200640 1 -- 172.21.15.134:0/1677499158 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fa0005460 con 0x5646a2a9b2f0 2024-07-26T11:23:26.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.899+0000 7f8f9d200640 1 -- 172.21.15.134:0/1677499158 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0035070 con 0x5646a2a9b2f0 2024-07-26T11:23:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.901+0000 7f8fb9b92740 1 -- 172.21.15.134:0/1677499158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2a9b2f0 msgr2=0x5646a2e68160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:26.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.901+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/1677499158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2a9b2f0 0x5646a2e68160 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0033000 tx=0x7f8fa0031400 comp rx=0 tx=0).stop 2024-07-26T11:23:26.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.902+0000 7f8fb9b92740 1 -- 172.21.15.134:0/1677499158 shutdown_connections 2024-07-26T11:23:26.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.902+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/1677499158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2aa34d0 0x5646a2e686a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.902+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/1677499158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2a9b2f0 0x5646a2e68160 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.902+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/1677499158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2ac41b0 0x5646a2e67c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.904 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.902+0000 7f8fb9b92740 1 -- 172.21.15.134:0/1677499158 >> 172.21.15.134:0/1677499158 conn(0x5646a2ab8300 msgr2=0x5646a2e65b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.903+0000 7f8fb9b92740 1 -- 172.21.15.134:0/1677499158 shutdown_connections 2024-07-26T11:23:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.903+0000 7f8fb9b92740 1 -- 172.21.15.134:0/1677499158 wait complete. 2024-07-26T11:23:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.904+0000 7f8fb9b92740 1 -- start start 2024-07-26T11:23:26.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.905+0000 7f8fb9b92740 1 Processor -- start 2024-07-26T11:23:26.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.906+0000 7f8fb9b92740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2a9b2f0 0x5646a2e73860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.906+0000 7f8fb9b92740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2aa34d0 0x5646a2e73e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.907+0000 7f8fb9b92740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2ac41b0 0x5646a2e90670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.907+0000 7f8fb9b92740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5646a2d05ab0 con 0x5646a2a9b2f0 2024-07-26T11:23:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.907+0000 7f8fb9b92740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5646a2d68d50 con 0x5646a2ac41b0 2024-07-26T11:23:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.907+0000 7f8fb9b92740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5646a2e87350 con 0x5646a2aa34d0 2024-07-26T11:23:26.909 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.907+0000 7f8fb9b92740 10 client.0 _ll_register_callbacks cb 0x5646a2adba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:23:26.910 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.907+0000 7f8fab400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2a9b2f0 0x5646a2e73860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.910 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.907+0000 7f8fabe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2aa34d0 0x5646a2e73e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.912 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.907+0000 7f8fab400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2a9b2f0 0x5646a2e73860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:58610/0 (socket says 172.21.15.134:58610) 2024-07-26T11:23:26.912 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.907+0000 7f8fab400640 1 -- 172.21.15.134:0/3163779635 learned_addr learned my addr 172.21.15.134:0/3163779635 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:26.912 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.910+0000 7f8fab400640 1 --2- 172.21.15.134:0/3163779635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2ac41b0 0x5646a2e90670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.912 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.910+0000 7f8fab400640 1 -- 172.21.15.134:0/3163779635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2aa34d0 msgr2=0x5646a2e73e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.911+0000 7f8fab400640 1 --2- 172.21.15.134:0/3163779635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2aa34d0 0x5646a2e73e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.911+0000 7f8fab400640 1 -- 172.21.15.134:0/3163779635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2a9b2f0 msgr2=0x5646a2e73860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.911+0000 7f8fab400640 1 --2- 172.21.15.134:0/3163779635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2a9b2f0 0x5646a2e73860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.911+0000 7f8fab400640 1 -- 172.21.15.134:0/3163779635 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646a2ba8620 con 0x5646a2ac41b0 2024-07-26T11:23:26.913 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.911+0000 7f8fabe00640 1 --2- 172.21.15.134:0/3163779635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2aa34d0 0x5646a2e73e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:23:26.914 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.912+0000 7f8fab400640 1 --2- 172.21.15.134:0/3163779635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2ac41b0 0x5646a2e90670 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8f98009c50 tx=0x7f8f980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7d791ae73e0384d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:26.914 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.912+0000 7f8f91600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:23:26.914 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.912+0000 7f8f91600640 1 -- 172.21.15.134:0/3163779635 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f98024060 con 0x5646a2ac41b0 2024-07-26T11:23:26.914 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.912+0000 7f8f91600640 1 -- 172.21.15.134:0/3163779635 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f8f9801de10 con 0x5646a2ac41b0 2024-07-26T11:23:26.915 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.912+0000 7f8fb9b92740 1 -- 172.21.15.134:0/3163779635 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fa0005460 con 0x5646a2ac41b0 2024-07-26T11:23:26.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.914+0000 7f8f91600640 1 -- 172.21.15.134:0/3163779635 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8f9802b050 con 0x5646a2ac41b0 2024-07-26T11:23:26.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.914+0000 7f8fb9b92740 10 client.26543 fetch_fsmap learned FSMap version 32 2024-07-26T11:23:26.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.914+0000 7f8fb9b92740 1 -- 172.21.15.134:0/3163779635 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5646a2e93470 con 0x5646a2ac41b0 2024-07-26T11:23:26.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.915+0000 7f8f91600640 1 -- 172.21.15.134:0/3163779635 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f9801f8f0 con 0x5646a2ac41b0 2024-07-26T11:23:26.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.915+0000 7f8f91600640 1 -- 172.21.15.134:0/3163779635 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f8f98019070 con 0x5646a2ac41b0 2024-07-26T11:23:26.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.915+0000 7f8fb9b92740 10 client.26543 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:23:26.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.915+0000 7f8fb9b92740 10 client.26543 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:23:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.915+0000 7f8fb9b92740 20 client.26543 populate_metadata read hostname 'smithi134' 2024-07-26T11:23:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.916+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/3163779635 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5646a2a82d80 0x5646a2e96940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.916+0000 7f8fb9b92740 4 client.26543 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:23:26.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.916+0000 7f8fb9b92740 1 -- 172.21.15.134:0/3163779635 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8fa0005460 con 0x5646a2a82d80 2024-07-26T11:23:26.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.918+0000 7f8fabe00640 1 --2- 172.21.15.134:0/3163779635 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5646a2a82d80 0x5646a2e96940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.929 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.927+0000 7f8fabe00640 1 --2- 172.21.15.134:0/3163779635 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5646a2a82d80 0x5646a2e96940 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90d38a979d13f664 server_cookie=ce722a279895ca2b in_seq=0 out_seq=0 2024-07-26T11:23:26.929 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.927+0000 7f8f91600640 10 client.26543 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.931+0000 7f8f91600640 1 -- 172.21.15.134:0/3163779635 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f8fa0005460 con 0x5646a2a82d80 2024-07-26T11:23:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.932+0000 7f8f91600640 10 client.26543 handle_command_reply: tid=0 2024-07-26T11:23:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.932+0000 7f8fb9b92740 1 client.26543 shutdown 2024-07-26T11:23:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.932+0000 7f8fb9b92740 1 -- 172.21.15.134:0/3163779635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2ac41b0 msgr2=0x5646a2e90670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.932+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/3163779635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2ac41b0 0x5646a2e90670 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8f98009c50 tx=0x7f8f980046c0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.933+0000 7f8fb9b92740 1 -- 172.21.15.134:0/3163779635 shutdown_connections 2024-07-26T11:23:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.933+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/3163779635 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5646a2a82d80 0x5646a2e96940 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.933+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/3163779635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2aa34d0 0x5646a2e73e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.933+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/3163779635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2ac41b0 0x5646a2e90670 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.933+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/3163779635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2a9b2f0 0x5646a2e73860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.933+0000 7f8fb9b92740 1 -- 172.21.15.134:0/3163779635 >> 172.21.15.134:0/3163779635 conn(0x5646a2ab8300 msgr2=0x5646a2e68c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:26.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.933+0000 7f8f91600640 0 client.26543 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:26.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.933+0000 7f8fb9b92740 1 -- 172.21.15.134:0/3163779635 shutdown_connections 2024-07-26T11:23:26.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.934+0000 7f8fb9b92740 1 -- 172.21.15.134:0/3163779635 wait complete. 2024-07-26T11:23:26.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.934+0000 7f8fb9b92740 20 client.26543 trim_cache size 0 max 16384 2024-07-26T11:23:26.972 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.970+0000 7f8fb9b92740 1 Processor -- start 2024-07-26T11:23:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.970+0000 7f8fb9b92740 1 -- start start 2024-07-26T11:23:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.973+0000 7f8fb9b92740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2a9b2f0 0x5646a2e6d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.975 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.973+0000 7f8fb9b92740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2aa34d0 0x5646a2e73230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.974+0000 7f8fb9b92740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2ac41b0 0x5646a2e90460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.974+0000 7f8fab400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2a9b2f0 0x5646a2e6d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.974+0000 7f8fab400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2a9b2f0 0x5646a2e6d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:37120/0 (socket says 172.21.15.134:37120) 2024-07-26T11:23:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.974+0000 7f8fb9b92740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1b90e0 con 0x5646a2aa34d0 2024-07-26T11:23:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.974+0000 7f8fb9b92740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1b6ed0 con 0x5646a2ac41b0 2024-07-26T11:23:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.974+0000 7f8fb9b92740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1b8a10 con 0x5646a2a9b2f0 2024-07-26T11:23:26.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.974+0000 7f8fab400640 1 -- 172.21.15.134:0/362972784 learned_addr learned my addr 172.21.15.134:0/362972784 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:26.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.975+0000 7f8fabe00640 1 --2- 172.21.15.134:0/362972784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2aa34d0 0x5646a2e73230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.978 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.976+0000 7f8fab400640 1 --2- 172.21.15.134:0/362972784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2ac41b0 0x5646a2e90460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.976+0000 7f8fab400640 1 -- 172.21.15.134:0/362972784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2ac41b0 msgr2=0x5646a2e90460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:26.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.976+0000 7f8fab400640 1 --2- 172.21.15.134:0/362972784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2ac41b0 0x5646a2e90460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.976+0000 7f8fab400640 1 -- 172.21.15.134:0/362972784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2aa34d0 msgr2=0x5646a2e73230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:26.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.976+0000 7f8fab400640 1 --2- 172.21.15.134:0/362972784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2aa34d0 0x5646a2e73230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.976+0000 7f8fab400640 1 -- 172.21.15.134:0/362972784 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646a2e66410 con 0x5646a2a9b2f0 2024-07-26T11:23:26.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.977+0000 7f8fab400640 1 --2- 172.21.15.134:0/362972784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2a9b2f0 0x5646a2e6d350 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8f98022040 tx=0x7f8f98005fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2f98a328ddd2295 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:26.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.977+0000 7f8f93e00640 1 -- 172.21.15.134:0/362972784 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f98009a80 con 0x5646a2a9b2f0 2024-07-26T11:23:26.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.977+0000 7f8f93e00640 1 -- 172.21.15.134:0/362972784 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f98036070 con 0x5646a2a9b2f0 2024-07-26T11:23:26.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.977+0000 7f8f93e00640 1 -- 172.21.15.134:0/362972784 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f98005040 con 0x5646a2a9b2f0 2024-07-26T11:23:26.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.978+0000 7f8fb9b92740 1 -- 172.21.15.134:0/362972784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2a9b2f0 msgr2=0x5646a2e6d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:26.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.978+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/362972784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2a9b2f0 0x5646a2e6d350 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8f98022040 tx=0x7f8f98005fb0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.979+0000 7f8fb9b92740 1 -- 172.21.15.134:0/362972784 shutdown_connections 2024-07-26T11:23:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.979+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/362972784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2ac41b0 0x5646a2e90460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.979+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/362972784 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2aa34d0 0x5646a2e73230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.979+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/362972784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2a9b2f0 0x5646a2e6d350 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.979+0000 7f8fb9b92740 1 -- 172.21.15.134:0/362972784 >> 172.21.15.134:0/362972784 conn(0x5646a2ab8300 msgr2=0x5646a2e72b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.981+0000 7f8fb9b92740 1 -- 172.21.15.134:0/362972784 shutdown_connections 2024-07-26T11:23:26.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.982+0000 7f8fb9b92740 1 -- 172.21.15.134:0/362972784 wait complete. 2024-07-26T11:23:26.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.983+0000 7f8fb9b92740 1 -- start start 2024-07-26T11:23:26.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.983+0000 7f8fb9b92740 1 Processor -- start 2024-07-26T11:23:26.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.985+0000 7f8fb9b92740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2a9b2f0 0x5646a2e8cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.988 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.986+0000 7f8fb9b92740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2aa34d0 0x5646a2e8a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.988 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.986+0000 7f8fb9b92740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2ac41b0 0x5646a2e8ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.988 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.986+0000 7f8fb9b92740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1b7c70 con 0x5646a2a9b2f0 2024-07-26T11:23:26.988 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.986+0000 7f8fb9b92740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1b75a0 con 0x5646a2aa34d0 2024-07-26T11:23:26.988 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.986+0000 7f8fb9b92740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8fac1b8340 con 0x5646a2ac41b0 2024-07-26T11:23:26.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.986+0000 7f8fb9b92740 10 client.0 _ll_register_callbacks cb 0x7f8f9800dea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:23:26.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.987+0000 7f8fabe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2aa34d0 0x5646a2e8a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.990 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.987+0000 7f8fabe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2aa34d0 0x5646a2e8a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38478/0 (socket says 172.21.15.134:38478) 2024-07-26T11:23:26.990 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.987+0000 7f8fab400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2ac41b0 0x5646a2e8ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.990 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.988+0000 7f8fabe00640 1 -- 172.21.15.134:0/850350819 learned_addr learned my addr 172.21.15.134:0/850350819 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:26.990 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.988+0000 7f8fab400640 1 --2- 172.21.15.134:0/850350819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2a9b2f0 0x5646a2e8cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.989+0000 7f8fabe00640 1 -- 172.21.15.134:0/850350819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2ac41b0 msgr2=0x5646a2e8ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.989+0000 7f8fabe00640 1 --2- 172.21.15.134:0/850350819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2ac41b0 0x5646a2e8ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.989+0000 7f8fabe00640 1 -- 172.21.15.134:0/850350819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2a9b2f0 msgr2=0x5646a2e8cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.989+0000 7f8fabe00640 1 --2- 172.21.15.134:0/850350819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2a9b2f0 0x5646a2e8cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.989+0000 7f8fabe00640 1 -- 172.21.15.134:0/850350819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646a2e8b5f0 con 0x5646a2aa34d0 2024-07-26T11:23:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.989+0000 7f8fab400640 1 --2- 172.21.15.134:0/850350819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2a9b2f0 0x5646a2e8cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:23:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.989+0000 7f8fabe00640 1 --2- 172.21.15.134:0/850350819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2aa34d0 0x5646a2e8a830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0031510 tx=0x7f8fa0000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=435203c26da2a7ba server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.989+0000 7f8f91600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:23:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.990+0000 7f8f91600640 1 -- 172.21.15.134:0/850350819 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa0035040 con 0x5646a2aa34d0 2024-07-26T11:23:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.990+0000 7f8f91600640 1 -- 172.21.15.134:0/850350819 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f8fa00048d0 con 0x5646a2aa34d0 2024-07-26T11:23:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.990+0000 7f8fb9b92740 1 -- 172.21.15.134:0/850350819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f98036070 con 0x5646a2aa34d0 2024-07-26T11:23:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.990+0000 7f8f91600640 1 -- 172.21.15.134:0/850350819 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8fa0009db0 con 0x5646a2aa34d0 2024-07-26T11:23:26.993 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.991+0000 7f8fb9b92740 10 client.26549 fetch_fsmap learned FSMap version 32 2024-07-26T11:23:26.993 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.991+0000 7f8fb9b92740 1 -- 172.21.15.134:0/850350819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5646a2efe0b0 con 0x5646a2aa34d0 2024-07-26T11:23:26.993 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.992+0000 7f8f91600640 1 -- 172.21.15.134:0/850350819 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f98036070 con 0x5646a2aa34d0 2024-07-26T11:23:26.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.992+0000 7f8f91600640 1 -- 172.21.15.134:0/850350819 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f8fa00315b0 con 0x5646a2aa34d0 2024-07-26T11:23:26.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.993+0000 7f8fb9b92740 10 client.26549 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:23:26.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.993+0000 7f8fb9b92740 10 client.26549 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:23:26.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.993+0000 7f8fb9b92740 20 client.26549 populate_metadata read hostname 'smithi134' 2024-07-26T11:23:26.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.993+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/850350819 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5646a2a581c0 0x5646a2f00400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.993+0000 7f8fb9b92740 4 client.26549 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:23:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.993+0000 7f8fb9b92740 1 -- 172.21.15.134:0/850350819 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5646a2dde1c0 con 0x5646a2a581c0 2024-07-26T11:23:26.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.994+0000 7f8fab400640 1 --2- 172.21.15.134:0/850350819 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5646a2a581c0 0x5646a2f00400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:26.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.995+0000 7f8fab400640 1 --2- 172.21.15.134:0/850350819 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5646a2a581c0 0x5646a2f00400 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc1ff8125d328eb8 server_cookie=a4e90355e8126d8f in_seq=0 out_seq=0 2024-07-26T11:23:26.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:26.996+0000 7f8f91600640 10 client.26549 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.250+0000 7f8f91600640 1 -- 172.21.15.134:0/850350819 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5646a2dde1c0 con 0x5646a2a581c0 2024-07-26T11:23:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.250+0000 7f8f91600640 10 client.26549 handle_command_reply: tid=0 2024-07-26T11:23:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.250+0000 7f8fb9b92740 1 client.26549 shutdown 2024-07-26T11:23:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.250+0000 7f8fb9b92740 1 -- 172.21.15.134:0/850350819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2aa34d0 msgr2=0x5646a2e8a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.250+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/850350819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2aa34d0 0x5646a2e8a830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0031510 tx=0x7f8fa0000bc0 comp rx=0 tx=0).stop 2024-07-26T11:23:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.251+0000 7f8fb9b92740 1 -- 172.21.15.134:0/850350819 shutdown_connections 2024-07-26T11:23:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.251+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/850350819 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5646a2a581c0 0x5646a2f00400 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.251+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/850350819 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5646a2ac41b0 0x5646a2e8ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.251+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/850350819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646a2aa34d0 0x5646a2e8a830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:27.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.251+0000 7f8fb9b92740 1 --2- 172.21.15.134:0/850350819 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5646a2a9b2f0 0x5646a2e8cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.251+0000 7f8fb9b92740 1 -- 172.21.15.134:0/850350819 >> 172.21.15.134:0/850350819 conn(0x5646a2ab8300 msgr2=0x5646a2e74b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.251+0000 7f8f91600640 0 client.26549 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.251+0000 7f8fb9b92740 1 -- 172.21.15.134:0/850350819 shutdown_connections 2024-07-26T11:23:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.251+0000 7f8fb9b92740 1 -- 172.21.15.134:0/850350819 wait complete. 2024-07-26T11:23:27.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.252+0000 7f8fb9b92740 20 client.26549 trim_cache size 0 max 16384 2024-07-26T11:23:27.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:23:27.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:23:27.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:23:27.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:23:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.254+0000 7f8fb2c00640 1 -- 172.21.15.134:0/2432501109 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8f940b3dc0 msgr2=0x7f8f940b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.254+0000 7f8fb2c00640 1 --2- 172.21.15.134:0/2432501109 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8f940b3dc0 0x7f8f940b62d0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40e1de0 tx=0x7f8fa400cef0 comp rx=0 tx=0).stop 2024-07-26T11:23:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.254+0000 7f8fb2c00640 1 -- 172.21.15.134:0/2432501109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fac10cdc0 msgr2=0x7f8fac1a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.254+0000 7f8fb2c00640 1 --2- 172.21.15.134:0/2432501109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fac10cdc0 0x7f8fac1a0660 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8fa400cec0 tx=0x7f8fa4008650 comp rx=0 tx=0).stop 2024-07-26T11:23:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.254+0000 7f8fb2c00640 1 -- 172.21.15.134:0/2432501109 shutdown_connections 2024-07-26T11:23:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.254+0000 7f8fb2c00640 1 --2- 172.21.15.134:0/2432501109 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f8f940b3dc0 0x7f8f940b62d0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.254+0000 7f8fb2c00640 1 --2- 172.21.15.134:0/2432501109 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8fac10c470 0x7f8fac19ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:27.256 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.254+0000 7f8fb2c00640 1 --2- 172.21.15.134:0/2432501109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8fac10cdc0 0x7f8fac1a0660 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.254+0000 7f8fb2c00640 1 --2- 172.21.15.134:0/2432501109 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8fac10a090 0x7f8fac1a3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.254+0000 7f8fb2c00640 1 -- 172.21.15.134:0/2432501109 >> 172.21.15.134:0/2432501109 conn(0x7f8fac070620 msgr2=0x7f8fac073760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:27.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.257+0000 7f8fb2c00640 1 -- 172.21.15.134:0/2432501109 shutdown_connections 2024-07-26T11:23:27.260 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:27.259+0000 7f8fb2c00640 1 -- 172.21.15.134:0/2432501109 wait complete. 2024-07-26T11:23:27.869 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:27 smithi134 ceph-mon[31966]: pgmap v706: 129 pgs: 129 active+clean; 55 GiB data, 114 GiB used, 959 GiB / 1.0 TiB avail; 34 KiB/s rd, 180 MiB/s wr, 1.71k op/s 2024-07-26T11:23:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:27 smithi189 ceph-mon[36790]: pgmap v706: 129 pgs: 129 active+clean; 55 GiB data, 114 GiB used, 959 GiB / 1.0 TiB avail; 34 KiB/s rd, 180 MiB/s wr, 1.71k op/s 2024-07-26T11:23:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:27 smithi193 ceph-mon[37473]: pgmap v706: 129 pgs: 129 active+clean; 55 GiB data, 114 GiB used, 959 GiB / 1.0 TiB avail; 34 KiB/s rd, 180 MiB/s wr, 1.71k op/s 2024-07-26T11:23:29.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:29 smithi193 ceph-mon[37473]: pgmap v707: 129 pgs: 129 active+clean; 55 GiB data, 115 GiB used, 958 GiB / 1.0 TiB avail; 187 KiB/s rd, 193 MiB/s wr, 1.80k op/s 2024-07-26T11:23:29.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:29 smithi134 ceph-mon[31966]: pgmap v707: 129 pgs: 129 active+clean; 55 GiB data, 115 GiB used, 958 GiB / 1.0 TiB avail; 187 KiB/s rd, 193 MiB/s wr, 1.80k op/s 2024-07-26T11:23:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:29 smithi189 ceph-mon[36790]: pgmap v707: 129 pgs: 129 active+clean; 55 GiB data, 115 GiB used, 958 GiB / 1.0 TiB avail; 187 KiB/s rd, 193 MiB/s wr, 1.80k op/s 2024-07-26T11:23:30.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.374+0000 7f8550e00640 1 -- 172.21.15.134:0/1213289291 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f854c10c700 msgr2=0x7f854c10cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:30.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.375+0000 7f8550e00640 1 --2- 172.21.15.134:0/1213289291 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f854c10c700 0x7f854c10cb10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f853c0074e0 tx=0x7f853c02b0a0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.376+0000 7f8550e00640 1 -- 172.21.15.134:0/1213289291 shutdown_connections 2024-07-26T11:23:30.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.377+0000 7f8550e00640 1 --2- 172.21.15.134:0/1213289291 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f854c10d050 0x7f854c116ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.377+0000 7f8550e00640 1 --2- 172.21.15.134:0/1213289291 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f854c10c700 0x7f854c10cb10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.377+0000 7f8550e00640 1 --2- 172.21.15.134:0/1213289291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854c10a080 0x7f854c10a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.377+0000 7f8550e00640 1 -- 172.21.15.134:0/1213289291 >> 172.21.15.134:0/1213289291 conn(0x7f854c0708f0 msgr2=0x7f854c070d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:30.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.377+0000 7f8550e00640 1 -- 172.21.15.134:0/1213289291 shutdown_connections 2024-07-26T11:23:30.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.378+0000 7f8550e00640 1 -- 172.21.15.134:0/1213289291 wait complete. 2024-07-26T11:23:30.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.379+0000 7f8550e00640 1 Processor -- start 2024-07-26T11:23:30.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.381+0000 7f8550e00640 1 -- start start 2024-07-26T11:23:30.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.383+0000 7f8550e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f854c10a080 0x7f854c11b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.383+0000 7f8550e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f854c10c700 0x7f854c11be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.384+0000 7f8550e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854c10d050 0x7f854c126260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.384+0000 7f8550e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f854c119820 con 0x7f854c10a080 2024-07-26T11:23:30.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.384+0000 7f8549400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f854c10c700 0x7f854c11be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:30.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.384+0000 7f8550e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f854c1196a0 con 0x7f854c10d050 2024-07-26T11:23:30.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.384+0000 7f854a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854c10d050 0x7f854c126260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:30.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.384+0000 7f854a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854c10d050 0x7f854c126260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:41392/0 (socket says 172.21.15.134:41392) 2024-07-26T11:23:30.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.384+0000 7f8550e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f854c1199a0 con 0x7f854c10c700 2024-07-26T11:23:30.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.384+0000 7f854a800640 1 -- 172.21.15.134:0/660906563 learned_addr learned my addr 172.21.15.134:0/660906563 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:30.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.384+0000 7f8549400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f854c10c700 0x7f854c11be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:39254/0 (socket says 172.21.15.134:39254) 2024-07-26T11:23:30.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.385+0000 7f854a800640 1 -- 172.21.15.134:0/660906563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f854c10c700 msgr2=0x7f854c11be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:30.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.385+0000 7f854a800640 1 --2- 172.21.15.134:0/660906563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f854c10c700 0x7f854c11be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.385+0000 7f854a800640 1 -- 172.21.15.134:0/660906563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f854c10a080 msgr2=0x7f854c11b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:30.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.385+0000 7f854a800640 1 --2- 172.21.15.134:0/660906563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f854c10a080 0x7f854c11b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.385+0000 7f854a800640 1 -- 172.21.15.134:0/660906563 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854c126990 con 0x7f854c10d050 2024-07-26T11:23:30.389 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.386+0000 7f854a800640 1 --2- 172.21.15.134:0/660906563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854c10d050 0x7f854c126260 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f854400cad0 tx=0x7f854400cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:30.389 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.386+0000 7f8542a00640 1 -- 172.21.15.134:0/660906563 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854400ae20 con 0x7f854c10d050 2024-07-26T11:23:30.389 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.386+0000 7f8542a00640 1 -- 172.21.15.134:0/660906563 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8544005020 con 0x7f854c10d050 2024-07-26T11:23:30.389 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.386+0000 7f8542a00640 1 -- 172.21.15.134:0/660906563 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f854400f6b0 con 0x7f854c10d050 2024-07-26T11:23:30.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.386+0000 7f8550e00640 1 -- 172.21.15.134:0/660906563 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f854c126c20 con 0x7f854c10d050 2024-07-26T11:23:30.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.387+0000 7f8550e00640 1 -- 172.21.15.134:0/660906563 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f854c1213c0 con 0x7f854c10d050 2024-07-26T11:23:30.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.394+0000 7f8542a00640 1 -- 172.21.15.134:0/660906563 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f85440054f0 con 0x7f854c10d050 2024-07-26T11:23:30.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.397+0000 7f8542a00640 1 --2- 172.21.15.134:0/660906563 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f85300b3e40 0x7f85300b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.398+0000 7f8542a00640 1 -- 172.21.15.134:0/660906563 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f85440e4410 con 0x7f854c10d050 2024-07-26T11:23:30.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.422+0000 7f85579d6740 1 Processor -- start 2024-07-26T11:23:30.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.399+0000 7f8549e00640 1 --2- 172.21.15.134:0/660906563 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f85300b3e40 0x7f85300b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:30.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.424+0000 7f8549e00640 1 --2- 172.21.15.134:0/660906563 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f85300b3e40 0x7f85300b6350 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f854c122160 tx=0x7f8538006c40 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:30.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.428+0000 7f85579d6740 1 -- start start 2024-07-26T11:23:30.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.428+0000 7f85579d6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5600a0660350 0x5600a0a04d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.428+0000 7f85579d6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a0637490 0x5600a0a052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.428+0000 7f85579d6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a063f670 0x5600a0a05810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.428+0000 7f85579d6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5600a097bbf0 con 0x5600a0660350 2024-07-26T11:23:30.439 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.428+0000 7f85579d6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5600a096b0f0 con 0x5600a0637490 2024-07-26T11:23:30.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.428+0000 7f85579d6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5600a096f130 con 0x5600a063f670 2024-07-26T11:23:30.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.432+0000 7f854a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a063f670 0x5600a0a05810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:30.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.432+0000 7f854a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a063f670 0x5600a0a05810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:39270/0 (socket says 172.21.15.134:39270) 2024-07-26T11:23:30.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.432+0000 7f854a800640 1 -- 172.21.15.134:0/3882269961 learned_addr learned my addr 172.21.15.134:0/3882269961 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:30.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.432+0000 7f8549400640 1 --2- 172.21.15.134:0/3882269961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a0637490 0x5600a0a052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:30.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.432+0000 7f854a800640 1 -- 172.21.15.134:0/3882269961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a0637490 msgr2=0x5600a0a052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:30.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.432+0000 7f854a800640 1 --2- 172.21.15.134:0/3882269961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a0637490 0x5600a0a052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.432+0000 7f854a800640 1 -- 172.21.15.134:0/3882269961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5600a0660350 msgr2=0x5600a0a04d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:30.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.432+0000 7f854a800640 1 --2- 172.21.15.134:0/3882269961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5600a0660350 0x5600a0a04d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.432+0000 7f854a800640 1 -- 172.21.15.134:0/3882269961 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600a06cb930 con 0x5600a063f670 2024-07-26T11:23:30.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.435+0000 7f854a800640 1 --2- 172.21.15.134:0/3882269961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a063f670 0x5600a0a05810 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f85440a6c60 tx=0x7f85440a6430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78572ed0e62a3f19 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:30.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.436+0000 7f8537400640 1 -- 172.21.15.134:0/3882269961 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85440051d0 con 0x5600a063f670 2024-07-26T11:23:30.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.437+0000 7f8537400640 1 -- 172.21.15.134:0/3882269961 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85440aabe0 con 0x5600a063f670 2024-07-26T11:23:30.440 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.437+0000 7f8537400640 1 -- 172.21.15.134:0/3882269961 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85440051d0 con 0x5600a063f670 2024-07-26T11:23:30.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.439+0000 7f85579d6740 1 -- 172.21.15.134:0/3882269961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a063f670 msgr2=0x5600a0a05810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:30.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.439+0000 7f85579d6740 1 --2- 172.21.15.134:0/3882269961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a063f670 0x5600a0a05810 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f85440a6c60 tx=0x7f85440a6430 comp rx=0 tx=0).stop 2024-07-26T11:23:30.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.444+0000 7f85579d6740 1 -- 172.21.15.134:0/3882269961 shutdown_connections 2024-07-26T11:23:30.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.444+0000 7f85579d6740 1 --2- 172.21.15.134:0/3882269961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a063f670 0x5600a0a05810 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.444+0000 7f85579d6740 1 --2- 172.21.15.134:0/3882269961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a0637490 0x5600a0a052d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.444+0000 7f85579d6740 1 --2- 172.21.15.134:0/3882269961 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5600a0660350 0x5600a0a04d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.444+0000 7f85579d6740 1 -- 172.21.15.134:0/3882269961 >> 172.21.15.134:0/3882269961 conn(0x5600a0654740 msgr2=0x5600a0a02c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:30.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.445+0000 7f85579d6740 1 -- 172.21.15.134:0/3882269961 shutdown_connections 2024-07-26T11:23:30.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.445+0000 7f85579d6740 1 -- 172.21.15.134:0/3882269961 wait complete. 2024-07-26T11:23:30.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.447+0000 7f85579d6740 1 -- start start 2024-07-26T11:23:30.456 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.450+0000 7f85579d6740 1 Processor -- start 2024-07-26T11:23:30.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.454+0000 7f85579d6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a0637490 0x5600a0a103f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.456+0000 7f85579d6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5600a063f670 0x5600a0a10930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.456+0000 7f85579d6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a0660350 0x5600a0a1c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.456+0000 7f85579d6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5600a07ce890 con 0x5600a063f670 2024-07-26T11:23:30.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.456+0000 7f85579d6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5600a0907330 con 0x5600a0660350 2024-07-26T11:23:30.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.456+0000 7f85579d6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5600a0a135c0 con 0x5600a0637490 2024-07-26T11:23:30.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.456+0000 7f85579d6740 10 client.0 _ll_register_callbacks cb 0x5600a0933010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:23:30.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.457+0000 7f8549e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a0637490 0x5600a0a103f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:30.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.457+0000 7f8549e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a0637490 0x5600a0a103f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:39272/0 (socket says 172.21.15.134:39272) 2024-07-26T11:23:30.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.457+0000 7f8549e00640 1 -- 172.21.15.134:0/1043557108 learned_addr learned my addr 172.21.15.134:0/1043557108 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:30.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.458+0000 7f8549e00640 1 -- 172.21.15.134:0/1043557108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a0660350 msgr2=0x5600a0a1c900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:23:30.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.458+0000 7f8549e00640 1 --2- 172.21.15.134:0/1043557108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a0660350 0x5600a0a1c900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.458+0000 7f8549e00640 1 -- 172.21.15.134:0/1043557108 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5600a063f670 msgr2=0x5600a0a10930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:23:30.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.459+0000 7f8549e00640 1 --2- 172.21.15.134:0/1043557108 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5600a063f670 0x5600a0a10930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.459+0000 7f8549e00640 1 -- 172.21.15.134:0/1043557108 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600a0a11120 con 0x5600a0637490 2024-07-26T11:23:30.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.462+0000 7f852be00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:23:30.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.462+0000 7f8549e00640 1 --2- 172.21.15.134:0/1043557108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a0637490 0x5600a0a103f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8538015fa0 tx=0x7f853800b800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4910d14539f41e7d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:30.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.464+0000 7f852be00640 1 -- 172.21.15.134:0/1043557108 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8538023060 con 0x5600a0637490 2024-07-26T11:23:30.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.464+0000 7f852be00640 1 -- 172.21.15.134:0/1043557108 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f853800be40 con 0x5600a0637490 2024-07-26T11:23:30.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.465+0000 7f85579d6740 1 -- 172.21.15.134:0/1043557108 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85440aabe0 con 0x5600a0637490 2024-07-26T11:23:30.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.466+0000 7f852be00640 1 -- 172.21.15.134:0/1043557108 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f853801e410 con 0x5600a0637490 2024-07-26T11:23:30.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.466+0000 7f85579d6740 10 client.26563 fetch_fsmap learned FSMap version 32 2024-07-26T11:23:30.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.466+0000 7f85579d6740 1 -- 172.21.15.134:0/1043557108 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5600a0a1f500 con 0x5600a0637490 2024-07-26T11:23:30.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.467+0000 7f852be00640 1 -- 172.21.15.134:0/1043557108 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f85440aabe0 con 0x5600a0637490 2024-07-26T11:23:30.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.472+0000 7f852be00640 1 -- 172.21.15.134:0/1043557108 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f8538012070 con 0x5600a0637490 2024-07-26T11:23:30.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.472+0000 7f85579d6740 10 client.26563 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:23:30.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.472+0000 7f85579d6740 10 client.26563 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:23:30.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.475+0000 7f85579d6740 20 client.26563 populate_metadata read hostname 'smithi134' 2024-07-26T11:23:30.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.475+0000 7f85579d6740 1 --2- 172.21.15.134:0/1043557108 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5600a061ecd0 0x5600a0a22980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.479+0000 7f85579d6740 4 client.26563 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:23:30.482 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.479+0000 7f85579d6740 1 -- 172.21.15.134:0/1043557108 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600a097db40 con 0x5600a061ecd0 2024-07-26T11:23:30.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.486+0000 7f8549400640 1 --2- 172.21.15.134:0/1043557108 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5600a061ecd0 0x5600a0a22980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:30.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.488+0000 7f8549400640 1 --2- 172.21.15.134:0/1043557108 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5600a061ecd0 0x5600a0a22980 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99150f8d870e4ff8 server_cookie=959df34b132adeb in_seq=0 out_seq=0 2024-07-26T11:23:30.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.488+0000 7f852be00640 10 client.26563 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:30.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.494+0000 7f852be00640 1 -- 172.21.15.134:0/1043557108 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5600a097db40 con 0x5600a061ecd0 2024-07-26T11:23:30.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.494+0000 7f852be00640 10 client.26563 handle_command_reply: tid=0 2024-07-26T11:23:30.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.494+0000 7f85579d6740 1 client.26563 shutdown 2024-07-26T11:23:30.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.496+0000 7f85579d6740 1 -- 172.21.15.134:0/1043557108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a0637490 msgr2=0x5600a0a103f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:30.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.496+0000 7f85579d6740 1 --2- 172.21.15.134:0/1043557108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a0637490 0x5600a0a103f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8538015fa0 tx=0x7f853800b800 comp rx=0 tx=0).stop 2024-07-26T11:23:30.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.496+0000 7f85579d6740 1 -- 172.21.15.134:0/1043557108 shutdown_connections 2024-07-26T11:23:30.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.496+0000 7f85579d6740 1 --2- 172.21.15.134:0/1043557108 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5600a061ecd0 0x5600a0a22980 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.496+0000 7f85579d6740 1 --2- 172.21.15.134:0/1043557108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a0660350 0x5600a0a1c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.496+0000 7f85579d6740 1 --2- 172.21.15.134:0/1043557108 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5600a063f670 0x5600a0a10930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.496+0000 7f85579d6740 1 --2- 172.21.15.134:0/1043557108 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a0637490 0x5600a0a103f0 secure :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f8538015fa0 tx=0x7f853800b800 comp rx=0 tx=0).stop 2024-07-26T11:23:30.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.496+0000 7f85579d6740 1 -- 172.21.15.134:0/1043557108 >> 172.21.15.134:0/1043557108 conn(0x5600a0654740 msgr2=0x5600a0a07f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:30.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.496+0000 7f852be00640 0 client.26563 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:30.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.498+0000 7f85579d6740 1 -- 172.21.15.134:0/1043557108 shutdown_connections 2024-07-26T11:23:30.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.498+0000 7f85579d6740 1 -- 172.21.15.134:0/1043557108 wait complete. 2024-07-26T11:23:30.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.498+0000 7f85579d6740 20 client.26563 trim_cache size 0 max 16384 2024-07-26T11:23:30.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.542+0000 7f85579d6740 1 Processor -- start 2024-07-26T11:23:30.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.546+0000 7f85579d6740 1 -- start start 2024-07-26T11:23:30.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.547+0000 7f85579d6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a063f670 0x5600a0a122d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.547+0000 7f85579d6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5600a0660350 0x5600a0a064d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.547+0000 7f85579d6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a05f4080 0x5600a0a02250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.547+0000 7f85579d6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f854c1c8280 con 0x5600a0660350 2024-07-26T11:23:30.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.547+0000 7f85579d6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f854c1c9dc0 con 0x5600a063f670 2024-07-26T11:23:30.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.547+0000 7f85579d6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f854c1c8950 con 0x5600a05f4080 2024-07-26T11:23:30.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.550+0000 7f8549e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a063f670 0x5600a0a122d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:30.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.550+0000 7f854a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a05f4080 0x5600a0a02250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:30.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.550+0000 7f854a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a05f4080 0x5600a0a02250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:39288/0 (socket says 172.21.15.134:39288) 2024-07-26T11:23:30.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.550+0000 7f854a800640 1 -- 172.21.15.134:0/2187876074 learned_addr learned my addr 172.21.15.134:0/2187876074 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:30.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.551+0000 7f8549e00640 1 -- 172.21.15.134:0/2187876074 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a05f4080 msgr2=0x5600a0a02250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:30.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.551+0000 7f8549e00640 1 --2- 172.21.15.134:0/2187876074 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a05f4080 0x5600a0a02250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.551+0000 7f8549e00640 1 -- 172.21.15.134:0/2187876074 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5600a0660350 msgr2=0x5600a0a064d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:30.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.551+0000 7f8549e00640 1 --2- 172.21.15.134:0/2187876074 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5600a0660350 0x5600a0a064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.551+0000 7f8549e00640 1 -- 172.21.15.134:0/2187876074 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600a0a0bf80 con 0x5600a063f670 2024-07-26T11:23:30.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.552+0000 7f8549e00640 1 --2- 172.21.15.134:0/2187876074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a063f670 0x5600a0a122d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8538014000 tx=0x7f8538006cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3038ce57023a430 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:30.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.552+0000 7f8536a00640 1 -- 172.21.15.134:0/2187876074 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853802d080 con 0x5600a063f670 2024-07-26T11:23:30.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.552+0000 7f8536a00640 1 -- 172.21.15.134:0/2187876074 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8538027830 con 0x5600a063f670 2024-07-26T11:23:30.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.552+0000 7f8536a00640 1 -- 172.21.15.134:0/2187876074 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85380036b0 con 0x5600a063f670 2024-07-26T11:23:30.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.554+0000 7f85579d6740 1 -- 172.21.15.134:0/2187876074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a063f670 msgr2=0x5600a0a122d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:30.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.554+0000 7f85579d6740 1 --2- 172.21.15.134:0/2187876074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a063f670 0x5600a0a122d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8538014000 tx=0x7f8538006cb0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.555+0000 7f85579d6740 1 -- 172.21.15.134:0/2187876074 shutdown_connections 2024-07-26T11:23:30.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.555+0000 7f85579d6740 1 --2- 172.21.15.134:0/2187876074 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a05f4080 0x5600a0a02250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.555+0000 7f85579d6740 1 --2- 172.21.15.134:0/2187876074 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5600a0660350 0x5600a0a064d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.555+0000 7f85579d6740 1 --2- 172.21.15.134:0/2187876074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a063f670 0x5600a0a122d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.555+0000 7f85579d6740 1 -- 172.21.15.134:0/2187876074 >> 172.21.15.134:0/2187876074 conn(0x5600a0654740 msgr2=0x5600a0a05dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:30.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.555+0000 7f85579d6740 1 -- 172.21.15.134:0/2187876074 shutdown_connections 2024-07-26T11:23:30.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.555+0000 7f85579d6740 1 -- 172.21.15.134:0/2187876074 wait complete. 2024-07-26T11:23:30.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.556+0000 7f85579d6740 1 -- start start 2024-07-26T11:23:30.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.556+0000 7f85579d6740 1 Processor -- start 2024-07-26T11:23:30.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.557+0000 7f85579d6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a05f4080 0x5600a0a149e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.557+0000 7f85579d6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a063f670 0x5600a0a14f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.557+0000 7f85579d6740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5600a0660350 0x5600a0a8a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.557+0000 7f85579d6740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f854c1c96f0 con 0x5600a0660350 2024-07-26T11:23:30.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.557+0000 7f85579d6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f854c1c9020 con 0x5600a063f670 2024-07-26T11:23:30.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.557+0000 7f85579d6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f854c1c6280 con 0x5600a05f4080 2024-07-26T11:23:30.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.557+0000 7f85579d6740 10 client.0 _ll_register_callbacks cb 0x7f8538015420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:23:30.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.558+0000 7f8549e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a05f4080 0x5600a0a149e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:30.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.559+0000 7f8549e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a05f4080 0x5600a0a149e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:39298/0 (socket says 172.21.15.134:39298) 2024-07-26T11:23:30.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.559+0000 7f8549400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a063f670 0x5600a0a14f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:30.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.559+0000 7f8549400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a063f670 0x5600a0a14f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:41426/0 (socket says 172.21.15.134:41426) 2024-07-26T11:23:30.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.559+0000 7f8549400640 1 -- 172.21.15.134:0/609761718 learned_addr learned my addr 172.21.15.134:0/609761718 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:30.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.560+0000 7f8549400640 1 -- 172.21.15.134:0/609761718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a05f4080 msgr2=0x5600a0a149e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:30.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.560+0000 7f8549400640 1 --2- 172.21.15.134:0/609761718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a05f4080 0x5600a0a149e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.560+0000 7f8549400640 1 -- 172.21.15.134:0/609761718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5600a0660350 msgr2=0x5600a0a8a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:30.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.560+0000 7f8549400640 1 --2- 172.21.15.134:0/609761718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5600a0660350 0x5600a0a8a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.560+0000 7f8549400640 1 -- 172.21.15.134:0/609761718 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600a0a157c0 con 0x5600a063f670 2024-07-26T11:23:30.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.561+0000 7f8549400640 1 --2- 172.21.15.134:0/609761718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a063f670 0x5600a0a14f20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f854c06c3a0 tx=0x7f853c02bad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2a0714e9c5ea032 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:30.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.561+0000 7f852be00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:23:30.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.561+0000 7f852be00640 1 -- 172.21.15.134:0/609761718 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c005040 con 0x5600a063f670 2024-07-26T11:23:30.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.562+0000 7f852be00640 1 -- 172.21.15.134:0/609761718 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f853c0051e0 con 0x5600a063f670 2024-07-26T11:23:30.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.562+0000 7f85579d6740 1 -- 172.21.15.134:0/609761718 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8538027830 con 0x5600a063f670 2024-07-26T11:23:30.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.563+0000 7f852be00640 1 -- 172.21.15.134:0/609761718 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f853c0053e0 con 0x5600a063f670 2024-07-26T11:23:30.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.563+0000 7f85579d6740 10 client.26561 fetch_fsmap learned FSMap version 32 2024-07-26T11:23:30.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.563+0000 7f85579d6740 1 -- 172.21.15.134:0/609761718 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5600a0a8b680 con 0x5600a063f670 2024-07-26T11:23:30.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.564+0000 7f852be00640 1 -- 172.21.15.134:0/609761718 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8538027830 con 0x5600a063f670 2024-07-26T11:23:30.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.568+0000 7f852be00640 1 -- 172.21.15.134:0/609761718 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f853c00a160 con 0x5600a063f670 2024-07-26T11:23:30.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.570+0000 7f85579d6740 10 client.26561 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:23:30.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.570+0000 7f85579d6740 10 client.26561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:23:30.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.570+0000 7f85579d6740 20 client.26561 populate_metadata read hostname 'smithi134' 2024-07-26T11:23:30.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.570+0000 7f85579d6740 1 --2- 172.21.15.134:0/609761718 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5600a0a8be00 0x5600a0a8e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:30.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.571+0000 7f85579d6740 4 client.26561 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T11:23:30.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.571+0000 7f85579d6740 1 -- 172.21.15.134:0/609761718 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5600a09795e0 con 0x5600a0a8be00 2024-07-26T11:23:30.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.572+0000 7f8549e00640 1 --2- 172.21.15.134:0/609761718 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5600a0a8be00 0x5600a0a8e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:30.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.581+0000 7f852be00640 10 client.26561 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:30.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.581+0000 7f8549e00640 1 --2- 172.21.15.134:0/609761718 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5600a0a8be00 0x5600a0a8e220 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67d9ec1588cef17c server_cookie=4519a270178991da in_seq=0 out_seq=0 2024-07-26T11:23:30.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.585+0000 7f852be00640 1 -- 172.21.15.134:0/609761718 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5600a09795e0 con 0x5600a0a8be00 2024-07-26T11:23:30.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.585+0000 7f852be00640 10 client.26561 handle_command_reply: tid=0 2024-07-26T11:23:30.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.585+0000 7f85579d6740 1 client.26561 shutdown 2024-07-26T11:23:30.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.586+0000 7f85579d6740 1 -- 172.21.15.134:0/609761718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a063f670 msgr2=0x5600a0a14f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:30.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.586+0000 7f85579d6740 1 --2- 172.21.15.134:0/609761718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a063f670 0x5600a0a14f20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f854c06c3a0 tx=0x7f853c02bad0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.586+0000 7f85579d6740 1 -- 172.21.15.134:0/609761718 shutdown_connections 2024-07-26T11:23:30.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.586+0000 7f85579d6740 1 --2- 172.21.15.134:0/609761718 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5600a0a8be00 0x5600a0a8e220 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.586+0000 7f85579d6740 1 --2- 172.21.15.134:0/609761718 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5600a0660350 0x5600a0a8a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.586+0000 7f85579d6740 1 --2- 172.21.15.134:0/609761718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600a063f670 0x5600a0a14f20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.586+0000 7f852be00640 0 client.26561 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:30.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.586+0000 7f85579d6740 1 --2- 172.21.15.134:0/609761718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600a05f4080 0x5600a0a149e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.586+0000 7f85579d6740 1 -- 172.21.15.134:0/609761718 >> 172.21.15.134:0/609761718 conn(0x5600a0654740 msgr2=0x5600a0a12030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:30.591 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.586+0000 7f85579d6740 1 -- 172.21.15.134:0/609761718 shutdown_connections 2024-07-26T11:23:30.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.589+0000 7f85579d6740 1 -- 172.21.15.134:0/609761718 wait complete. 2024-07-26T11:23:30.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.590+0000 7f85579d6740 20 client.26561 trim_cache size 0 max 16384 2024-07-26T11:23:30.592 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:23:30.592 INFO:teuthology.orchestra.run.smithi134.stdout: "return_code": 0, 2024-07-26T11:23:30.592 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_tag": "ea65e344-ca4a-42f1-8503-9eb3f2491fa4", 2024-07-26T11:23:30.593 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "asynchronous" 2024-07-26T11:23:30.593 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:23:30.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.595+0000 7f8550e00640 1 -- 172.21.15.134:0/660906563 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f85300b3e40 msgr2=0x7f85300b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:30.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.595+0000 7f8550e00640 1 --2- 172.21.15.134:0/660906563 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f85300b3e40 0x7f85300b6350 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f854c122160 tx=0x7f8538006c40 comp rx=0 tx=0).stop 2024-07-26T11:23:30.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.595+0000 7f8550e00640 1 -- 172.21.15.134:0/660906563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854c10d050 msgr2=0x7f854c126260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:30.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.595+0000 7f8550e00640 1 --2- 172.21.15.134:0/660906563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854c10d050 0x7f854c126260 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f854400cad0 tx=0x7f854400cf90 comp rx=0 tx=0).stop 2024-07-26T11:23:30.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.595+0000 7f8550e00640 1 -- 172.21.15.134:0/660906563 shutdown_connections 2024-07-26T11:23:30.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.595+0000 7f8550e00640 1 --2- 172.21.15.134:0/660906563 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f85300b3e40 0x7f85300b6350 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.595+0000 7f8550e00640 1 --2- 172.21.15.134:0/660906563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f854c10c700 0x7f854c11be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.595+0000 7f8550e00640 1 --2- 172.21.15.134:0/660906563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854c10d050 0x7f854c126260 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.595+0000 7f8550e00640 1 --2- 172.21.15.134:0/660906563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f854c10a080 0x7f854c11b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:30.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.595+0000 7f8550e00640 1 -- 172.21.15.134:0/660906563 >> 172.21.15.134:0/660906563 conn(0x7f854c0708f0 msgr2=0x7f854c10da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:30.599 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.595+0000 7f8550e00640 1 -- 172.21.15.134:0/660906563 shutdown_connections 2024-07-26T11:23:30.599 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:30.596+0000 7f8550e00640 1 -- 172.21.15.134:0/660906563 wait complete. 2024-07-26T11:23:31.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:31 smithi134 ceph-mon[31966]: pgmap v708: 129 pgs: 129 active+clean; 56 GiB data, 116 GiB used, 956 GiB / 1.0 TiB avail; 501 KiB/s rd, 217 MiB/s wr, 1.95k op/s 2024-07-26T11:23:31.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:31 smithi134 ceph-mon[31966]: scrub queued for path: / 2024-07-26T11:23:31.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:31 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:23:31.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:31 smithi134 ceph-mon[31966]: scrub summary: active paths [/] 2024-07-26T11:23:31.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:31 smithi134 ceph-mon[31966]: scrub summary: active paths [/] 2024-07-26T11:23:31.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:31 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:23:31.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:31 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:23:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:31 smithi189 ceph-mon[36790]: pgmap v708: 129 pgs: 129 active+clean; 56 GiB data, 116 GiB used, 956 GiB / 1.0 TiB avail; 501 KiB/s rd, 217 MiB/s wr, 1.95k op/s 2024-07-26T11:23:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:31 smithi189 ceph-mon[36790]: scrub queued for path: / 2024-07-26T11:23:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:31 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:23:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:31 smithi189 ceph-mon[36790]: scrub summary: active paths [/] 2024-07-26T11:23:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:31 smithi189 ceph-mon[36790]: scrub summary: active paths [/] 2024-07-26T11:23:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:31 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:23:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:31 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:23:32.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:31 smithi193 ceph-mon[37473]: pgmap v708: 129 pgs: 129 active+clean; 56 GiB data, 116 GiB used, 956 GiB / 1.0 TiB avail; 501 KiB/s rd, 217 MiB/s wr, 1.95k op/s 2024-07-26T11:23:32.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:31 smithi193 ceph-mon[37473]: scrub queued for path: / 2024-07-26T11:23:32.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:31 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:23:32.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:31 smithi193 ceph-mon[37473]: scrub summary: active paths [/] 2024-07-26T11:23:32.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:31 smithi193 ceph-mon[37473]: scrub summary: active paths [/] 2024-07-26T11:23:32.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:31 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:23:32.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:31 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:23:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:33 smithi134 ceph-mon[31966]: pgmap v709: 129 pgs: 129 active+clean; 56 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 504 KiB/s rd, 185 MiB/s wr, 1.64k op/s 2024-07-26T11:23:33.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:33 smithi189 ceph-mon[36790]: pgmap v709: 129 pgs: 129 active+clean; 56 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 504 KiB/s rd, 185 MiB/s wr, 1.64k op/s 2024-07-26T11:23:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:33 smithi193 ceph-mon[37473]: pgmap v709: 129 pgs: 129 active+clean; 56 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 504 KiB/s rd, 185 MiB/s wr, 1.64k op/s 2024-07-26T11:23:35.932 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:35 smithi134 ceph-mon[31966]: pgmap v710: 129 pgs: 129 active+clean; 56 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 489 KiB/s rd, 217 MiB/s wr, 1.83k op/s 2024-07-26T11:23:35.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:35 smithi189 ceph-mon[36790]: pgmap v710: 129 pgs: 129 active+clean; 56 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 489 KiB/s rd, 217 MiB/s wr, 1.83k op/s 2024-07-26T11:23:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:35 smithi193 ceph-mon[37473]: pgmap v710: 129 pgs: 129 active+clean; 56 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 489 KiB/s rd, 217 MiB/s wr, 1.83k op/s 2024-07-26T11:23:36.627 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:23:36.787 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:36 smithi134 ceph-mon[31966]: pgmap v711: 129 pgs: 129 active+clean; 57 GiB data, 119 GiB used, 954 GiB / 1.0 TiB avail; 490 KiB/s rd, 227 MiB/s wr, 1.79k op/s 2024-07-26T11:23:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:36 smithi193 ceph-mon[37473]: pgmap v711: 129 pgs: 129 active+clean; 57 GiB data, 119 GiB used, 954 GiB / 1.0 TiB avail; 490 KiB/s rd, 227 MiB/s wr, 1.79k op/s 2024-07-26T11:23:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:36 smithi189 ceph-mon[36790]: pgmap v711: 129 pgs: 129 active+clean; 57 GiB data, 119 GiB used, 954 GiB / 1.0 TiB avail; 490 KiB/s rd, 227 MiB/s wr, 1.79k op/s 2024-07-26T11:23:37.768 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:23:39.794 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:39 smithi134 ceph-mon[31966]: pgmap v712: 129 pgs: 129 active+clean; 57 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 507 KiB/s rd, 227 MiB/s wr, 1.72k op/s 2024-07-26T11:23:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:39 smithi189 ceph-mon[36790]: pgmap v712: 129 pgs: 129 active+clean; 57 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 507 KiB/s rd, 227 MiB/s wr, 1.72k op/s 2024-07-26T11:23:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:39 smithi193 ceph-mon[37473]: pgmap v712: 129 pgs: 129 active+clean; 57 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 507 KiB/s rd, 227 MiB/s wr, 1.72k op/s 2024-07-26T11:23:40.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.714+0000 7f1bf8e00640 1 -- 172.21.15.134:0/750425484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bf4105540 msgr2=0x7f1bf4105950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:40.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.714+0000 7f1bf8e00640 1 --2- 172.21.15.134:0/750425484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bf4105540 0x7f1bf4105950 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1be4009900 tx=0x7f1be402a420 comp rx=0 tx=0).stop 2024-07-26T11:23:40.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.714+0000 7f1bf8e00640 1 -- 172.21.15.134:0/750425484 shutdown_connections 2024-07-26T11:23:40.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.714+0000 7f1bf8e00640 1 --2- 172.21.15.134:0/750425484 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bf410b7c0 0x7f1bf4111ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.714+0000 7f1bf8e00640 1 --2- 172.21.15.134:0/750425484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bf410ae70 0x7f1bf410b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.714+0000 7f1bf8e00640 1 --2- 172.21.15.134:0/750425484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bf4105540 0x7f1bf4105950 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.714+0000 7f1bf8e00640 1 -- 172.21.15.134:0/750425484 >> 172.21.15.134:0/750425484 conn(0x7f1bf4100f20 msgr2=0x7f1bf4103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:40.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.715+0000 7f1bf8e00640 1 -- 172.21.15.134:0/750425484 shutdown_connections 2024-07-26T11:23:40.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.716+0000 7f1bf8e00640 1 -- 172.21.15.134:0/750425484 wait complete. 2024-07-26T11:23:40.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.718+0000 7f1bf8e00640 1 Processor -- start 2024-07-26T11:23:40.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.718+0000 7f1bf8e00640 1 -- start start 2024-07-26T11:23:40.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.722+0000 7f1bf8e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bf410ae70 0x7f1bf4074a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.722+0000 7f1bf8e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bf410b7c0 0x7f1bf406dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.722+0000 7f1bf8e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bf406e2c0 0x7f1bf4073600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.723+0000 7f1bf8e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf41154c0 con 0x7f1bf410b7c0 2024-07-26T11:23:40.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.723+0000 7f1bf8e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf4115340 con 0x7f1bf406e2c0 2024-07-26T11:23:40.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.723+0000 7f1bf8e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf4115640 con 0x7f1bf410ae70 2024-07-26T11:23:40.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.723+0000 7f1bf3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bf410ae70 0x7f1bf4074a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.723+0000 7f1bf2a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bf410b7c0 0x7f1bf406dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.723+0000 7f1bf3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bf410ae70 0x7f1bf4074a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59114/0 (socket says 172.21.15.134:59114) 2024-07-26T11:23:40.726 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.724+0000 7f1bf3400640 1 -- 172.21.15.134:0/146879105 learned_addr learned my addr 172.21.15.134:0/146879105 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:40.727 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.725+0000 7f1bf3e00640 1 --2- 172.21.15.134:0/146879105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bf406e2c0 0x7f1bf4073600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.725+0000 7f1bf3e00640 1 -- 172.21.15.134:0/146879105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bf410ae70 msgr2=0x7f1bf4074a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:40.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.725+0000 7f1bf3e00640 1 --2- 172.21.15.134:0/146879105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bf410ae70 0x7f1bf4074a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.725+0000 7f1bf3e00640 1 -- 172.21.15.134:0/146879105 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bf410b7c0 msgr2=0x7f1bf406dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:40.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.725+0000 7f1bf3e00640 1 --2- 172.21.15.134:0/146879105 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bf410b7c0 0x7f1bf406dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.725+0000 7f1bf3e00640 1 -- 172.21.15.134:0/146879105 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf406ea10 con 0x7f1bf406e2c0 2024-07-26T11:23:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.726+0000 7f1bf3e00640 1 --2- 172.21.15.134:0/146879105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bf406e2c0 0x7f1bf4073600 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1bec00cab0 tx=0x7f1bec00cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.726+0000 7f1bebe00640 1 -- 172.21.15.134:0/146879105 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec00ae20 con 0x7f1bf406e2c0 2024-07-26T11:23:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.726+0000 7f1bebe00640 1 -- 172.21.15.134:0/146879105 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bec00f900 con 0x7f1bf406e2c0 2024-07-26T11:23:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.726+0000 7f1bebe00640 1 -- 172.21.15.134:0/146879105 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec018a10 con 0x7f1bf406e2c0 2024-07-26T11:23:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.727+0000 7f1bf8e00640 1 -- 172.21.15.134:0/146879105 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf4073c00 con 0x7f1bf406e2c0 2024-07-26T11:23:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.728+0000 7f1bf8e00640 1 -- 172.21.15.134:0/146879105 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf4074010 con 0x7f1bf406e2c0 2024-07-26T11:23:40.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.735+0000 7f1bebe00640 1 -- 172.21.15.134:0/146879105 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f1bec018bb0 con 0x7f1bf406e2c0 2024-07-26T11:23:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.737+0000 7f1c015b0740 1 Processor -- start 2024-07-26T11:23:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.737+0000 7f1c015b0740 1 -- start start 2024-07-26T11:23:40.739 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.738+0000 7f1bebe00640 1 --2- 172.21.15.134:0/146879105 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1bd80b3e40 0x7f1bd80b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.740 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.738+0000 7f1bebe00640 1 -- 172.21.15.134:0/146879105 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f1bec0e4c60 con 0x7f1bf406e2c0 2024-07-26T11:23:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.739+0000 7f1c015b0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 0x55fa90c8fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.739+0000 7f1c015b0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908c32f0 0x55fa90c90180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.739+0000 7f1c015b0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 0x55fa90c906c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.739+0000 7f1c015b0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55fa90c04270 con 0x55fa908c32f0 2024-07-26T11:23:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.739+0000 7f1c015b0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fa90bf3770 con 0x55fa908ec1b0 2024-07-26T11:23:40.741 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.739+0000 7f1c015b0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fa90bf77b0 con 0x55fa908cb4d0 2024-07-26T11:23:40.742 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.740+0000 7f1bf3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 0x55fa90c906c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:40.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.740+0000 7f1bf3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 0x55fa90c8fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:40.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.740+0000 7f1bf3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 0x55fa90c906c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59126/0 (socket says 172.21.15.134:59126) 2024-07-26T11:23:40.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.740+0000 7f1bf3e00640 1 -- 172.21.15.134:0/3988946981 learned_addr learned my addr 172.21.15.134:0/3988946981 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:40.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.741+0000 7f1bf3e00640 1 -- 172.21.15.134:0/3988946981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 msgr2=0x55fa90c8fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:40.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.741+0000 7f1bf3e00640 1 --2- 172.21.15.134:0/3988946981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 0x55fa90c8fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.741+0000 7f1bf3e00640 1 -- 172.21.15.134:0/3988946981 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908c32f0 msgr2=0x55fa90c90180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:23:40.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.741+0000 7f1bf3e00640 1 --2- 172.21.15.134:0/3988946981 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908c32f0 0x55fa90c90180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.741+0000 7f1bf3e00640 1 -- 172.21.15.134:0/3988946981 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa90957790 con 0x55fa908cb4d0 2024-07-26T11:23:40.750 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.741+0000 7f1bf2a00640 1 --2- 172.21.15.134:0/146879105 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1bd80b3e40 0x7f1bd80b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:40.750 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.742+0000 7f1bf3e00640 1 --2- 172.21.15.134:0/3988946981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 0x55fa90c906c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1bec0ab3e0 tx=0x7f1bec0aaf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c69c1a8ecab9f4e7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:40.750 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.744+0000 7f1be8c00640 1 -- 172.21.15.134:0/3988946981 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec0ab450 con 0x55fa908cb4d0 2024-07-26T11:23:40.750 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.744+0000 7f1be8c00640 1 -- 172.21.15.134:0/3988946981 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bec0afa50 con 0x55fa908cb4d0 2024-07-26T11:23:40.750 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.744+0000 7f1be8c00640 1 -- 172.21.15.134:0/3988946981 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec0a7860 con 0x55fa908cb4d0 2024-07-26T11:23:40.750 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.744+0000 7f1bf2a00640 1 --2- 172.21.15.134:0/146879105 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1bd80b3e40 0x7f1bd80b6350 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x55fa90c8e8c0 tx=0x7f1be0005b80 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.748+0000 7f1c015b0740 1 -- 172.21.15.134:0/3988946981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 msgr2=0x55fa90c906c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.748+0000 7f1c015b0740 1 --2- 172.21.15.134:0/3988946981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 0x55fa90c906c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1bec0ab3e0 tx=0x7f1bec0aaf60 comp rx=0 tx=0).stop 2024-07-26T11:23:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.748+0000 7f1c015b0740 1 -- 172.21.15.134:0/3988946981 shutdown_connections 2024-07-26T11:23:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.748+0000 7f1c015b0740 1 --2- 172.21.15.134:0/3988946981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 0x55fa90c906c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.748+0000 7f1c015b0740 1 --2- 172.21.15.134:0/3988946981 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908c32f0 0x55fa90c90180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.748+0000 7f1c015b0740 1 --2- 172.21.15.134:0/3988946981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 0x55fa90c8fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.748+0000 7f1c015b0740 1 -- 172.21.15.134:0/3988946981 >> 172.21.15.134:0/3988946981 conn(0x55fa908e0300 msgr2=0x55fa90c8db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.748+0000 7f1c015b0740 1 -- 172.21.15.134:0/3988946981 shutdown_connections 2024-07-26T11:23:40.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.748+0000 7f1c015b0740 1 -- 172.21.15.134:0/3988946981 wait complete. 2024-07-26T11:23:40.752 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.749+0000 7f1c015b0740 1 -- start start 2024-07-26T11:23:40.752 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.749+0000 7f1c015b0740 1 Processor -- start 2024-07-26T11:23:40.752 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.750+0000 7f1c015b0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908c32f0 0x55fa90c9b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.752 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.750+0000 7f1c015b0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908cb4d0 0x55fa90c9bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.750+0000 7f1c015b0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 0x55fa90ca76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.750+0000 7f1c015b0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55fa90b2dab0 con 0x55fa908cb4d0 2024-07-26T11:23:40.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.750+0000 7f1c015b0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fa90b90d50 con 0x55fa908ec1b0 2024-07-26T11:23:40.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.750+0000 7f1c015b0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55fa90c9e360 con 0x55fa908c32f0 2024-07-26T11:23:40.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.750+0000 7f1c015b0740 10 client.0 _ll_register_callbacks cb 0x55fa90903a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:23:40.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.750+0000 7f1bf3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908c32f0 0x55fa90c9b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:40.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.751+0000 7f1bf3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908c32f0 0x55fa90c9b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59140/0 (socket says 172.21.15.134:59140) 2024-07-26T11:23:40.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.751+0000 7f1bf3400640 1 -- 172.21.15.134:0/3190627140 learned_addr learned my addr 172.21.15.134:0/3190627140 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.751+0000 7f1bf3e00640 1 --2- 172.21.15.134:0/3190627140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 0x55fa90ca76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.751+0000 7f1bf2a00640 1 --2- 172.21.15.134:0/3190627140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908cb4d0 0x55fa90c9bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.751+0000 7f1bf3e00640 1 -- 172.21.15.134:0/3190627140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908c32f0 msgr2=0x55fa90c9b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.751+0000 7f1bf3e00640 1 --2- 172.21.15.134:0/3190627140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908c32f0 0x55fa90c9b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.751+0000 7f1bf3e00640 1 -- 172.21.15.134:0/3190627140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908cb4d0 msgr2=0x55fa90c9bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.751+0000 7f1bf3e00640 1 --2- 172.21.15.134:0/3190627140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908cb4d0 0x55fa90c9bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.751+0000 7f1bf3e00640 1 -- 172.21.15.134:0/3190627140 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa909d0620 con 0x55fa908ec1b0 2024-07-26T11:23:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.751+0000 7f1bf2a00640 1 --2- 172.21.15.134:0/3190627140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908cb4d0 0x55fa90c9bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:23:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.752+0000 7f1bf3e00640 1 --2- 172.21.15.134:0/3190627140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 0x55fa90ca76b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1bec0a74d0 tx=0x7f1bec0072d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=858826d1c53ac192 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.752+0000 7f1bdd600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:23:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.752+0000 7f1bdd600640 1 -- 172.21.15.134:0/3190627140 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec0c9070 con 0x55fa908ec1b0 2024-07-26T11:23:40.757 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.752+0000 7f1c015b0740 1 -- 172.21.15.134:0/3190627140 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bec0afa50 con 0x55fa908ec1b0 2024-07-26T11:23:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.752+0000 7f1bdd600640 1 -- 172.21.15.134:0/3190627140 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f1bec0a7a40 con 0x55fa908ec1b0 2024-07-26T11:23:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.752+0000 7f1bdd600640 1 -- 172.21.15.134:0/3190627140 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1bec0ab450 con 0x55fa908ec1b0 2024-07-26T11:23:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.753+0000 7f1c015b0740 10 client.26570 fetch_fsmap learned FSMap version 32 2024-07-26T11:23:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.753+0000 7f1c015b0740 1 -- 172.21.15.134:0/3190627140 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fa90caa400 con 0x55fa908ec1b0 2024-07-26T11:23:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.753+0000 7f1bdd600640 1 -- 172.21.15.134:0/3190627140 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1bec0afa50 con 0x55fa908ec1b0 2024-07-26T11:23:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.754+0000 7f1bdd600640 1 -- 172.21.15.134:0/3190627140 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f1bec020070 con 0x55fa908ec1b0 2024-07-26T11:23:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.755+0000 7f1c015b0740 10 client.26570 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:23:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.755+0000 7f1c015b0740 10 client.26570 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:23:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.755+0000 7f1c015b0740 20 client.26570 populate_metadata read hostname 'smithi134' 2024-07-26T11:23:40.758 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.755+0000 7f1c015b0740 1 --2- 172.21.15.134:0/3190627140 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55fa908aad80 0x55fa90cad760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.755+0000 7f1c015b0740 4 client.26570 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:23:40.759 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.755+0000 7f1c015b0740 1 -- 172.21.15.134:0/3190627140 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa90c061c0 con 0x55fa908aad80 2024-07-26T11:23:40.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.759+0000 7f1bf3400640 1 --2- 172.21.15.134:0/3190627140 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55fa908aad80 0x55fa90cad760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.760+0000 7f1bf3400640 1 --2- 172.21.15.134:0/3190627140 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55fa908aad80 0x55fa90cad760 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9276ce5221edfb7b server_cookie=5f93ac6ab1ec0091 in_seq=0 out_seq=0 2024-07-26T11:23:40.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.760+0000 7f1bdd600640 10 client.26570 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.765+0000 7f1bdd600640 1 -- 172.21.15.134:0/3190627140 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55fa90c061c0 con 0x55fa908aad80 2024-07-26T11:23:40.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.765+0000 7f1bdd600640 10 client.26570 handle_command_reply: tid=0 2024-07-26T11:23:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.766+0000 7f1c015b0740 1 client.26570 shutdown 2024-07-26T11:23:40.768 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.766+0000 7f1c015b0740 1 -- 172.21.15.134:0/3190627140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 msgr2=0x55fa90ca76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.766+0000 7f1c015b0740 1 --2- 172.21.15.134:0/3190627140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 0x55fa90ca76b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1bec0a74d0 tx=0x7f1bec0072d0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.767+0000 7f1c015b0740 1 -- 172.21.15.134:0/3190627140 shutdown_connections 2024-07-26T11:23:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.767+0000 7f1c015b0740 1 --2- 172.21.15.134:0/3190627140 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55fa908aad80 0x55fa90cad760 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.767+0000 7f1c015b0740 1 --2- 172.21.15.134:0/3190627140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 0x55fa90ca76b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.767+0000 7f1c015b0740 1 --2- 172.21.15.134:0/3190627140 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908cb4d0 0x55fa90c9bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.767+0000 7f1c015b0740 1 --2- 172.21.15.134:0/3190627140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908c32f0 0x55fa90c9b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.767+0000 7f1c015b0740 1 -- 172.21.15.134:0/3190627140 >> 172.21.15.134:0/3190627140 conn(0x55fa908e0300 msgr2=0x55fa90c90c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:40.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.767+0000 7f1bdd600640 0 client.26570 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.768+0000 7f1c015b0740 1 -- 172.21.15.134:0/3190627140 shutdown_connections 2024-07-26T11:23:40.770 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.768+0000 7f1c015b0740 1 -- 172.21.15.134:0/3190627140 wait complete. 2024-07-26T11:23:40.771 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.769+0000 7f1c015b0740 20 client.26570 trim_cache size 0 max 16384 2024-07-26T11:23:40.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.801+0000 7f1c015b0740 1 Processor -- start 2024-07-26T11:23:40.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.801+0000 7f1c015b0740 1 -- start start 2024-07-26T11:23:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.802+0000 7f1c015b0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908c32f0 0x55fa90c903f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.802+0000 7f1c015b0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 0x55fa90c9aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.802+0000 7f1c015b0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908ec1b0 0x55fa90c9b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.802+0000 7f1c015b0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf41b72d0 con 0x55fa908ec1b0 2024-07-26T11:23:40.804 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.802+0000 7f1c015b0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf41b8e10 con 0x55fa908c32f0 2024-07-26T11:23:40.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.802+0000 7f1c015b0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf41b79a0 con 0x55fa908cb4d0 2024-07-26T11:23:40.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.802+0000 7f1bf3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908c32f0 0x55fa90c903f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:40.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.802+0000 7f1bf3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908c32f0 0x55fa90c903f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:54070/0 (socket says 172.21.15.134:54070) 2024-07-26T11:23:40.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.802+0000 7f1bf3400640 1 -- 172.21.15.134:0/2988843628 learned_addr learned my addr 172.21.15.134:0/2988843628 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:40.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.802+0000 7f1bf2a00640 1 --2- 172.21.15.134:0/2988843628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 0x55fa90c9aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:40.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.802+0000 7f1bf3400640 1 -- 172.21.15.134:0/2988843628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 msgr2=0x55fa90c9aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:40.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.802+0000 7f1bf3400640 1 --2- 172.21.15.134:0/2988843628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 0x55fa90c9aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.802+0000 7f1bf3400640 1 -- 172.21.15.134:0/2988843628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908ec1b0 msgr2=0x55fa90c9b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:40.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.802+0000 7f1bf3400640 1 --2- 172.21.15.134:0/2988843628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908ec1b0 0x55fa90c9b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.802+0000 7f1bf3400640 1 -- 172.21.15.134:0/2988843628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa90cb3940 con 0x55fa908c32f0 2024-07-26T11:23:40.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.803+0000 7f1bf3400640 1 --2- 172.21.15.134:0/2988843628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908c32f0 0x55fa90c903f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1be4009230 tx=0x7f1be4005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e0b20f201140e7b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:40.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.803+0000 7f1bdfe00640 1 -- 172.21.15.134:0/2988843628 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be4035040 con 0x55fa908c32f0 2024-07-26T11:23:40.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.803+0000 7f1bdfe00640 1 -- 172.21.15.134:0/2988843628 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1be40054d0 con 0x55fa908c32f0 2024-07-26T11:23:40.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.803+0000 7f1bdfe00640 1 -- 172.21.15.134:0/2988843628 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be40035f0 con 0x55fa908c32f0 2024-07-26T11:23:40.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.804+0000 7f1c015b0740 1 -- 172.21.15.134:0/2988843628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908c32f0 msgr2=0x55fa90c903f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:40.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.804+0000 7f1c015b0740 1 --2- 172.21.15.134:0/2988843628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908c32f0 0x55fa90c903f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f1be4009230 tx=0x7f1be4005a20 comp rx=0 tx=0).stop 2024-07-26T11:23:40.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.805+0000 7f1c015b0740 1 -- 172.21.15.134:0/2988843628 shutdown_connections 2024-07-26T11:23:40.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.805+0000 7f1c015b0740 1 --2- 172.21.15.134:0/2988843628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 0x55fa90c9aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.805+0000 7f1c015b0740 1 --2- 172.21.15.134:0/2988843628 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908ec1b0 0x55fa90c9b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.805+0000 7f1c015b0740 1 --2- 172.21.15.134:0/2988843628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908c32f0 0x55fa90c903f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.805+0000 7f1c015b0740 1 -- 172.21.15.134:0/2988843628 >> 172.21.15.134:0/2988843628 conn(0x55fa908e0300 msgr2=0x55fa90c99e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:40.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.805+0000 7f1c015b0740 1 -- 172.21.15.134:0/2988843628 shutdown_connections 2024-07-26T11:23:40.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.805+0000 7f1c015b0740 1 -- 172.21.15.134:0/2988843628 wait complete. 2024-07-26T11:23:40.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.805+0000 7f1c015b0740 1 -- start start 2024-07-26T11:23:40.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.805+0000 7f1c015b0740 1 Processor -- start 2024-07-26T11:23:40.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1c015b0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908c32f0 0x55fa90cadba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1c015b0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 0x55fa90cae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1c015b0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 0x55fa90d13ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1c015b0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf41b8740 con 0x55fa908c32f0 2024-07-26T11:23:40.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1c015b0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf41b8070 con 0x55fa908ec1b0 2024-07-26T11:23:40.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1c015b0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf41b5360 con 0x55fa908cb4d0 2024-07-26T11:23:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1c015b0740 10 client.0 _ll_register_callbacks cb 0x7f1bec0aae30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:23:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1bf2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 0x55fa90cae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1bf3400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908c32f0 0x55fa90cadba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1bf3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 0x55fa90d13ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1bf2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 0x55fa90cae0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59170/0 (socket says 172.21.15.134:59170) 2024-07-26T11:23:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1bf2a00640 1 -- 172.21.15.134:0/2130046311 learned_addr learned my addr 172.21.15.134:0/2130046311 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1bf3e00640 1 -- 172.21.15.134:0/2130046311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 msgr2=0x55fa90cae0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1bf3e00640 1 --2- 172.21.15.134:0/2130046311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 0x55fa90cae0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1bf3e00640 1 -- 172.21.15.134:0/2130046311 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908c32f0 msgr2=0x55fa90cadba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1bf3e00640 1 --2- 172.21.15.134:0/2130046311 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908c32f0 0x55fa90cadba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.806+0000 7f1bf3e00640 1 -- 172.21.15.134:0/2130046311 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa90d141e0 con 0x55fa908ec1b0 2024-07-26T11:23:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.808+0000 7f1bf3e00640 1 --2- 172.21.15.134:0/2130046311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 0x55fa90d13ca0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1bec00ca80 tx=0x7f1bec018400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5aaa77e970d91b9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:40.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.808+0000 7f1bdd600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:23:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.808+0000 7f1bdd600640 1 -- 172.21.15.134:0/2130046311 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec0c9070 con 0x55fa908ec1b0 2024-07-26T11:23:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.808+0000 7f1bdd600640 1 -- 172.21.15.134:0/2130046311 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f1bec0a7a40 con 0x55fa908ec1b0 2024-07-26T11:23:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.808+0000 7f1c015b0740 1 -- 172.21.15.134:0/2130046311 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1be40054d0 con 0x55fa908ec1b0 2024-07-26T11:23:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.808+0000 7f1bdd600640 1 -- 172.21.15.134:0/2130046311 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1bec0afa50 con 0x55fa908ec1b0 2024-07-26T11:23:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.808+0000 7f1c015b0740 10 client.26576 fetch_fsmap learned FSMap version 32 2024-07-26T11:23:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.808+0000 7f1c015b0740 1 -- 172.21.15.134:0/2130046311 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55fa90d15310 con 0x55fa908ec1b0 2024-07-26T11:23:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.808+0000 7f1bdd600640 1 -- 172.21.15.134:0/2130046311 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1be40054d0 con 0x55fa908ec1b0 2024-07-26T11:23:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.808+0000 7f1bdd600640 1 -- 172.21.15.134:0/2130046311 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f1bec0a6dc0 con 0x55fa908ec1b0 2024-07-26T11:23:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.808+0000 7f1c015b0740 10 client.26576 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:23:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.808+0000 7f1c015b0740 10 client.26576 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:23:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.808+0000 7f1c015b0740 20 client.26576 populate_metadata read hostname 'smithi134' 2024-07-26T11:23:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.808+0000 7f1c015b0740 1 --2- 172.21.15.134:0/2130046311 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55fa908801c0 0x55fa90d17610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.808+0000 7f1c015b0740 4 client.26576 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:23:40.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.808+0000 7f1c015b0740 1 -- 172.21.15.134:0/2130046311 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fa90c01c60 con 0x55fa908801c0 2024-07-26T11:23:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.809+0000 7f1bf3400640 1 --2- 172.21.15.134:0/2130046311 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55fa908801c0 0x55fa90d17610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.809+0000 7f1bf3400640 1 --2- 172.21.15.134:0/2130046311 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55fa908801c0 0x55fa90d17610 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16879244e309efb5 server_cookie=e6294be129c7d043 in_seq=0 out_seq=0 2024-07-26T11:23:40.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.809+0000 7f1bdd600640 10 client.26576 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:40.841 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.839+0000 7f1bdd600640 1 -- 172.21.15.134:0/2130046311 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fa90c01c60 con 0x55fa908801c0 2024-07-26T11:23:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.839+0000 7f1bdd600640 10 client.26576 handle_command_reply: tid=0 2024-07-26T11:23:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.839+0000 7f1c015b0740 1 client.26576 shutdown 2024-07-26T11:23:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.840+0000 7f1c015b0740 1 -- 172.21.15.134:0/2130046311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 msgr2=0x55fa90d13ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.840+0000 7f1c015b0740 1 --2- 172.21.15.134:0/2130046311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 0x55fa90d13ca0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1bec00ca80 tx=0x7f1bec018400 comp rx=0 tx=0).stop 2024-07-26T11:23:40.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.841+0000 7f1c015b0740 1 -- 172.21.15.134:0/2130046311 shutdown_connections 2024-07-26T11:23:40.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.841+0000 7f1c015b0740 1 --2- 172.21.15.134:0/2130046311 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55fa908801c0 0x55fa90d17610 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.841+0000 7f1c015b0740 1 --2- 172.21.15.134:0/2130046311 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fa908cb4d0 0x55fa90cae0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.841+0000 7f1c015b0740 1 --2- 172.21.15.134:0/2130046311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa908ec1b0 0x55fa90d13ca0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.841+0000 7f1c015b0740 1 --2- 172.21.15.134:0/2130046311 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55fa908c32f0 0x55fa90cadba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.841+0000 7f1c015b0740 1 -- 172.21.15.134:0/2130046311 >> 172.21.15.134:0/2130046311 conn(0x55fa908e0300 msgr2=0x55fa90c9aae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:40.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.841+0000 7f1bdd600640 0 client.26576 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:40.844 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.842+0000 7f1c015b0740 1 -- 172.21.15.134:0/2130046311 shutdown_connections 2024-07-26T11:23:40.844 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.842+0000 7f1c015b0740 1 -- 172.21.15.134:0/2130046311 wait complete. 2024-07-26T11:23:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.843+0000 7f1c015b0740 20 client.26576 trim_cache size 0 max 16384 2024-07-26T11:23:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:23:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:23:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:23:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:23:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.844+0000 7f1bf8e00640 1 -- 172.21.15.134:0/146879105 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1bd80b3e40 msgr2=0x7f1bd80b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.844+0000 7f1bf8e00640 1 --2- 172.21.15.134:0/146879105 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1bd80b3e40 0x7f1bd80b6350 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x55fa90c8e8c0 tx=0x7f1be0005b80 comp rx=0 tx=0).stop 2024-07-26T11:23:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.844+0000 7f1bf8e00640 1 -- 172.21.15.134:0/146879105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bf406e2c0 msgr2=0x7f1bf4073600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.844+0000 7f1bf8e00640 1 --2- 172.21.15.134:0/146879105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bf406e2c0 0x7f1bf4073600 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1bec00cab0 tx=0x7f1bec00cf70 comp rx=0 tx=0).stop 2024-07-26T11:23:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.844+0000 7f1bf8e00640 1 -- 172.21.15.134:0/146879105 shutdown_connections 2024-07-26T11:23:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.844+0000 7f1bf8e00640 1 --2- 172.21.15.134:0/146879105 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f1bd80b3e40 0x7f1bd80b6350 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.844+0000 7f1bf8e00640 1 --2- 172.21.15.134:0/146879105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bf406e2c0 0x7f1bf4073600 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.844+0000 7f1bf8e00640 1 --2- 172.21.15.134:0/146879105 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1bf410b7c0 0x7f1bf406dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.844+0000 7f1bf8e00640 1 --2- 172.21.15.134:0/146879105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1bf410ae70 0x7f1bf4074a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.844+0000 7f1bf8e00640 1 -- 172.21.15.134:0/146879105 >> 172.21.15.134:0/146879105 conn(0x7f1bf4100f20 msgr2=0x7f1bf4103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.845+0000 7f1bf8e00640 1 -- 172.21.15.134:0/146879105 shutdown_connections 2024-07-26T11:23:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:40.845+0000 7f1bf8e00640 1 -- 172.21.15.134:0/146879105 wait complete. 2024-07-26T11:23:41.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:41 smithi134 ceph-mon[31966]: scrub summary: idle 2024-07-26T11:23:41.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:41 smithi134 ceph-mon[31966]: scrub summary: idle 2024-07-26T11:23:41.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:41 smithi134 ceph-mon[31966]: pgmap v713: 129 pgs: 129 active+clean; 58 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 369 KiB/s rd, 274 MiB/s wr, 1.92k op/s 2024-07-26T11:23:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:41 smithi189 ceph-mon[36790]: scrub summary: idle 2024-07-26T11:23:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:41 smithi189 ceph-mon[36790]: scrub summary: idle 2024-07-26T11:23:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:41 smithi189 ceph-mon[36790]: pgmap v713: 129 pgs: 129 active+clean; 58 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 369 KiB/s rd, 274 MiB/s wr, 1.92k op/s 2024-07-26T11:23:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:41 smithi193 ceph-mon[37473]: scrub summary: idle 2024-07-26T11:23:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:41 smithi193 ceph-mon[37473]: scrub summary: idle 2024-07-26T11:23:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:41 smithi193 ceph-mon[37473]: pgmap v713: 129 pgs: 129 active+clean; 58 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 369 KiB/s rd, 274 MiB/s wr, 1.92k op/s 2024-07-26T11:23:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:42 smithi193 ceph-mon[37473]: pgmap v714: 129 pgs: 129 active+clean; 58 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 51 KiB/s rd, 227 MiB/s wr, 1.48k op/s 2024-07-26T11:23:43.179 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:42 smithi134 ceph-mon[31966]: pgmap v714: 129 pgs: 129 active+clean; 58 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 51 KiB/s rd, 227 MiB/s wr, 1.48k op/s 2024-07-26T11:23:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:42 smithi189 ceph-mon[36790]: pgmap v714: 129 pgs: 129 active+clean; 58 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 51 KiB/s rd, 227 MiB/s wr, 1.48k op/s 2024-07-26T11:23:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:45 smithi134 ceph-mon[31966]: pgmap v715: 129 pgs: 129 active+clean; 59 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 45 KiB/s rd, 246 MiB/s wr, 1.52k op/s 2024-07-26T11:23:45.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:45 smithi189 ceph-mon[36790]: pgmap v715: 129 pgs: 129 active+clean; 59 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 45 KiB/s rd, 246 MiB/s wr, 1.52k op/s 2024-07-26T11:23:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:45 smithi193 ceph-mon[37473]: pgmap v715: 129 pgs: 129 active+clean; 59 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 45 KiB/s rd, 246 MiB/s wr, 1.52k op/s 2024-07-26T11:23:46.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.649+0000 7ff013e00640 1 -- 172.21.15.134:0/1040410327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff00c0a50f0 msgr2=0x7ff00c0a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:46.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.649+0000 7ff013e00640 1 --2- 172.21.15.134:0/1040410327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff00c0a50f0 0x7ff00c0a5500 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff004009900 tx=0x7ff00402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.650+0000 7ff013e00640 1 -- 172.21.15.134:0/1040410327 shutdown_connections 2024-07-26T11:23:46.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.650+0000 7ff013e00640 1 --2- 172.21.15.134:0/1040410327 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff00c0ab1b0 0x7ff00c0ab9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.650+0000 7ff013e00640 1 --2- 172.21.15.134:0/1040410327 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff00c0aa860 0x7ff00c0aac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.650+0000 7ff013e00640 1 --2- 172.21.15.134:0/1040410327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff00c0a50f0 0x7ff00c0a5500 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.650+0000 7ff013e00640 1 -- 172.21.15.134:0/1040410327 >> 172.21.15.134:0/1040410327 conn(0x7ff00c01cc70 msgr2=0x7ff00c01d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:46.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.650+0000 7ff013e00640 1 -- 172.21.15.134:0/1040410327 shutdown_connections 2024-07-26T11:23:46.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.650+0000 7ff013e00640 1 -- 172.21.15.134:0/1040410327 wait complete. 2024-07-26T11:23:46.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.652+0000 7ff013e00640 1 Processor -- start 2024-07-26T11:23:46.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.653+0000 7ff013e00640 1 -- start start 2024-07-26T11:23:46.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.653+0000 7ff013e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff00c0a50f0 0x7ff00c144090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.653+0000 7ff013e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff00c0aa860 0x7ff00c144630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.653+0000 7ff013e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff00c0ab1b0 0x7ff00c1498d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.653+0000 7ff013e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c0bd660 con 0x7ff00c0a50f0 2024-07-26T11:23:46.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.653+0000 7ff013e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c0bd4e0 con 0x7ff00c0aa860 2024-07-26T11:23:46.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.653+0000 7ff013e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c0bd7e0 con 0x7ff00c0ab1b0 2024-07-26T11:23:46.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.655+0000 7ff013400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff00c0ab1b0 0x7ff00c1498d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:46.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.655+0000 7ff013400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff00c0ab1b0 0x7ff00c1498d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59182/0 (socket says 172.21.15.134:59182) 2024-07-26T11:23:46.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.655+0000 7ff013400640 1 -- 172.21.15.134:0/1342922753 learned_addr learned my addr 172.21.15.134:0/1342922753 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.655+0000 7ff013400640 1 -- 172.21.15.134:0/1342922753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff00c0aa860 msgr2=0x7ff00c144630 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T11:23:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.655+0000 7ff013400640 1 --2- 172.21.15.134:0/1342922753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff00c0aa860 0x7ff00c144630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.655+0000 7ff013400640 1 -- 172.21.15.134:0/1342922753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff00c0a50f0 msgr2=0x7ff00c144090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.655+0000 7ff013400640 1 --2- 172.21.15.134:0/1342922753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff00c0a50f0 0x7ff00c144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.655+0000 7ff013400640 1 -- 172.21.15.134:0/1342922753 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff00c14a000 con 0x7ff00c0ab1b0 2024-07-26T11:23:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.656+0000 7ff013400640 1 --2- 172.21.15.134:0/1342922753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff00c0ab1b0 0x7ff00c1498d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff014052290 tx=0x7ff014075a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.656+0000 7ff00b400640 1 -- 172.21.15.134:0/1342922753 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014076a00 con 0x7ff00c0ab1b0 2024-07-26T11:23:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.657+0000 7ff00b400640 1 -- 172.21.15.134:0/1342922753 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff014076ba0 con 0x7ff00c0ab1b0 2024-07-26T11:23:46.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.657+0000 7ff00b400640 1 -- 172.21.15.134:0/1342922753 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01407d330 con 0x7ff00c0ab1b0 2024-07-26T11:23:46.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.658+0000 7ff013e00640 1 -- 172.21.15.134:0/1342922753 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff00c14a2f0 con 0x7ff00c0ab1b0 2024-07-26T11:23:46.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.658+0000 7ff013e00640 1 -- 172.21.15.134:0/1342922753 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff00c150a50 con 0x7ff00c0ab1b0 2024-07-26T11:23:46.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.668+0000 7ff00b400640 1 -- 172.21.15.134:0/1342922753 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7ff014073400 con 0x7ff00c0ab1b0 2024-07-26T11:23:46.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.671+0000 7ff00b400640 1 --2- 172.21.15.134:0/1342922753 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7feff80b3e90 0x7feff80b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.671+0000 7ff00b400640 1 -- 172.21.15.134:0/1342922753 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7ff014148d90 con 0x7ff00c0ab1b0 2024-07-26T11:23:46.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.673+0000 7ff012a00640 1 --2- 172.21.15.134:0/1342922753 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7feff80b3e90 0x7feff80b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:46.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.676+0000 7ff012a00640 1 --2- 172.21.15.134:0/1342922753 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7feff80b3e90 0x7feff80b63a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff004009230 tx=0x7ff004031400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:46.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.680+0000 7ff02067b740 1 Processor -- start 2024-07-26T11:23:46.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.681+0000 7ff02067b740 1 -- start start 2024-07-26T11:23:46.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.681+0000 7ff02067b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fe2070 0x55d957386fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.681+0000 7ff02067b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fb91b0 0x55d957387510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.681+0000 7ff02067b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fc1390 0x55d957387a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.681+0000 7ff02067b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d9572fdb90 con 0x55d956fe2070 2024-07-26T11:23:46.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.681+0000 7ff02067b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d9572ed090 con 0x55d956fb91b0 2024-07-26T11:23:46.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.681+0000 7ff02067b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d9572f10d0 con 0x55d956fc1390 2024-07-26T11:23:46.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.682+0000 7ff013400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fc1390 0x55d957387a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:46.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.682+0000 7ff013400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fc1390 0x55d957387a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59184/0 (socket says 172.21.15.134:59184) 2024-07-26T11:23:46.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.682+0000 7ff013400640 1 -- 172.21.15.134:0/1019079244 learned_addr learned my addr 172.21.15.134:0/1019079244 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:46.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.682+0000 7ff012000640 1 --2- 172.21.15.134:0/1019079244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fb91b0 0x55d957387510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:46.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.682+0000 7ff012a00640 1 --2- 172.21.15.134:0/1019079244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fe2070 0x55d957386fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:46.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.682+0000 7ff013400640 1 -- 172.21.15.134:0/1019079244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fb91b0 msgr2=0x55d957387510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:46.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.682+0000 7ff013400640 1 --2- 172.21.15.134:0/1019079244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fb91b0 0x55d957387510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.682+0000 7ff013400640 1 -- 172.21.15.134:0/1019079244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fe2070 msgr2=0x55d957386fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:46.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.682+0000 7ff013400640 1 --2- 172.21.15.134:0/1019079244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fe2070 0x55d957386fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.682+0000 7ff013400640 1 -- 172.21.15.134:0/1019079244 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d95704d650 con 0x55d956fc1390 2024-07-26T11:23:46.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.682+0000 7ff012a00640 1 --2- 172.21.15.134:0/1019079244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fe2070 0x55d957386fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:23:46.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.685+0000 7ff013400640 1 --2- 172.21.15.134:0/1019079244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fc1390 0x55d957387a50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff01410c940 tx=0x7ff014073b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40c273e1b9832cb5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:46.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.689+0000 7fefffe00640 1 -- 172.21.15.134:0/1019079244 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014113cd0 con 0x55d956fc1390 2024-07-26T11:23:46.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.689+0000 7fefffe00640 1 -- 172.21.15.134:0/1019079244 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff014118780 con 0x55d956fc1390 2024-07-26T11:23:46.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.689+0000 7fefffe00640 1 -- 172.21.15.134:0/1019079244 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014146510 con 0x55d956fc1390 2024-07-26T11:23:46.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.693+0000 7ff02067b740 1 -- 172.21.15.134:0/1019079244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fc1390 msgr2=0x55d957387a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:46.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.693+0000 7ff02067b740 1 --2- 172.21.15.134:0/1019079244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fc1390 0x55d957387a50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff01410c940 tx=0x7ff014073b70 comp rx=0 tx=0).stop 2024-07-26T11:23:46.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.694+0000 7ff02067b740 1 -- 172.21.15.134:0/1019079244 shutdown_connections 2024-07-26T11:23:46.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.694+0000 7ff02067b740 1 --2- 172.21.15.134:0/1019079244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fc1390 0x55d957387a50 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.694+0000 7ff02067b740 1 --2- 172.21.15.134:0/1019079244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fb91b0 0x55d957387510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.694+0000 7ff02067b740 1 --2- 172.21.15.134:0/1019079244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fe2070 0x55d957386fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.694+0000 7ff02067b740 1 -- 172.21.15.134:0/1019079244 >> 172.21.15.134:0/1019079244 conn(0x55d956fd6460 msgr2=0x55d957384ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:46.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.694+0000 7ff02067b740 1 -- 172.21.15.134:0/1019079244 shutdown_connections 2024-07-26T11:23:46.698 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.695+0000 7ff02067b740 1 -- 172.21.15.134:0/1019079244 wait complete. 2024-07-26T11:23:46.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.695+0000 7ff02067b740 1 -- start start 2024-07-26T11:23:46.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.696+0000 7ff02067b740 1 Processor -- start 2024-07-26T11:23:46.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.697+0000 7ff02067b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fb91b0 0x55d9573926f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.697+0000 7ff02067b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fc1390 0x55d957392c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.697+0000 7ff02067b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fe2070 0x55d9573931b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.697+0000 7ff02067b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d957150700 con 0x55d956fe2070 2024-07-26T11:23:46.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.697+0000 7ff02067b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d957289660 con 0x55d956fb91b0 2024-07-26T11:23:46.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.697+0000 7ff02067b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d9573957b0 con 0x55d956fc1390 2024-07-26T11:23:46.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.697+0000 7ff02067b740 10 client.0 _ll_register_callbacks cb 0x55d957299110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:23:46.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.698+0000 7ff012000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fc1390 0x55d957392c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:46.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.698+0000 7ff012000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fc1390 0x55d957392c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59200/0 (socket says 172.21.15.134:59200) 2024-07-26T11:23:46.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.699+0000 7ff012000640 1 -- 172.21.15.134:0/2085762720 learned_addr learned my addr 172.21.15.134:0/2085762720 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:46.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.699+0000 7ff012000640 1 -- 172.21.15.134:0/2085762720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fb91b0 msgr2=0x55d9573926f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:23:46.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.700+0000 7ff013400640 1 --2- 172.21.15.134:0/2085762720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fe2070 0x55d9573931b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:46.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.700+0000 7ff012000640 1 --2- 172.21.15.134:0/2085762720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fb91b0 0x55d9573926f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.700+0000 7ff012000640 1 -- 172.21.15.134:0/2085762720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fe2070 msgr2=0x55d9573931b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:46.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.700+0000 7ff012000640 1 --2- 172.21.15.134:0/2085762720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fe2070 0x55d9573931b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.700+0000 7ff012000640 1 -- 172.21.15.134:0/2085762720 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9572a0800 con 0x55d956fc1390 2024-07-26T11:23:46.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.701+0000 7ff012000640 1 --2- 172.21.15.134:0/2085762720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fc1390 0x55d957392c30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff00000f640 tx=0x7ff00000fb00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59ec77077f44a181 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:46.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.701+0000 7feffcc00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:23:46.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.704+0000 7feffcc00640 1 -- 172.21.15.134:0/2085762720 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff000005220 con 0x55d956fc1390 2024-07-26T11:23:46.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.704+0000 7feffcc00640 1 -- 172.21.15.134:0/2085762720 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7ff0000053c0 con 0x55d956fc1390 2024-07-26T11:23:46.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.704+0000 7ff02067b740 1 -- 172.21.15.134:0/2085762720 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff014118780 con 0x55d956fc1390 2024-07-26T11:23:46.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.705+0000 7feffcc00640 1 -- 172.21.15.134:0/2085762720 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff0000055c0 con 0x55d956fc1390 2024-07-26T11:23:46.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.705+0000 7ff02067b740 10 client.26599 fetch_fsmap learned FSMap version 32 2024-07-26T11:23:46.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.705+0000 7ff02067b740 1 -- 172.21.15.134:0/2085762720 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d9573a1450 con 0x55d956fc1390 2024-07-26T11:23:46.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.707+0000 7feffcc00640 1 -- 172.21.15.134:0/2085762720 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff014118780 con 0x55d956fc1390 2024-07-26T11:23:46.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.707+0000 7feffcc00640 1 -- 172.21.15.134:0/2085762720 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7ff000010ba0 con 0x55d956fc1390 2024-07-26T11:23:46.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.708+0000 7ff02067b740 10 client.26599 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:23:46.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.708+0000 7ff02067b740 10 client.26599 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:23:46.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.708+0000 7ff02067b740 20 client.26599 populate_metadata read hostname 'smithi134' 2024-07-26T11:23:46.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.708+0000 7ff02067b740 1 --2- 172.21.15.134:0/2085762720 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d956fa09f0 0x55d9573a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.709+0000 7ff02067b740 4 client.26599 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:23:46.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.709+0000 7ff02067b740 1 -- 172.21.15.134:0/2085762720 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9572ffae0 con 0x55d956fa09f0 2024-07-26T11:23:46.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.710+0000 7ff012a00640 1 --2- 172.21.15.134:0/2085762720 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d956fa09f0 0x55d9573a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:46.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.711+0000 7ff012a00640 1 --2- 172.21.15.134:0/2085762720 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d956fa09f0 0x55d9573a4a80 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e41ac581a2d2180c server_cookie=37aef3279f003c55 in_seq=0 out_seq=0 2024-07-26T11:23:46.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.712+0000 7feffcc00640 10 client.26599 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:46.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.715+0000 7feffcc00640 1 -- 172.21.15.134:0/2085762720 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d9572ffae0 con 0x55d956fa09f0 2024-07-26T11:23:46.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.715+0000 7feffcc00640 10 client.26599 handle_command_reply: tid=0 2024-07-26T11:23:46.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.715+0000 7ff02067b740 1 client.26599 shutdown 2024-07-26T11:23:46.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.715+0000 7ff02067b740 1 -- 172.21.15.134:0/2085762720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fc1390 msgr2=0x55d957392c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:46.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.715+0000 7ff02067b740 1 --2- 172.21.15.134:0/2085762720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fc1390 0x55d957392c30 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff00000f640 tx=0x7ff00000fb00 comp rx=0 tx=0).stop 2024-07-26T11:23:46.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.716+0000 7ff02067b740 1 -- 172.21.15.134:0/2085762720 shutdown_connections 2024-07-26T11:23:46.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.716+0000 7ff02067b740 1 --2- 172.21.15.134:0/2085762720 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d956fa09f0 0x55d9573a4a80 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.716+0000 7ff02067b740 1 --2- 172.21.15.134:0/2085762720 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fc1390 0x55d957392c30 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.716+0000 7ff02067b740 1 --2- 172.21.15.134:0/2085762720 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fe2070 0x55d9573931b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.716+0000 7ff02067b740 1 --2- 172.21.15.134:0/2085762720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fb91b0 0x55d9573926f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.716+0000 7ff02067b740 1 -- 172.21.15.134:0/2085762720 >> 172.21.15.134:0/2085762720 conn(0x55d956fd6460 msgr2=0x55d95738a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:46.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.716+0000 7feffcc00640 0 client.26599 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:46.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.717+0000 7ff02067b740 1 -- 172.21.15.134:0/2085762720 shutdown_connections 2024-07-26T11:23:46.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.717+0000 7ff02067b740 1 -- 172.21.15.134:0/2085762720 wait complete. 2024-07-26T11:23:46.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.717+0000 7ff02067b740 20 client.26599 trim_cache size 0 max 16384 2024-07-26T11:23:46.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.769+0000 7ff02067b740 1 Processor -- start 2024-07-26T11:23:46.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.769+0000 7ff02067b740 1 -- start start 2024-07-26T11:23:46.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.770+0000 7ff02067b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fb91b0 0x55d95738def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.770+0000 7ff02067b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fc1390 0x55d957384e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.770+0000 7ff02067b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fe2070 0x55d9573aa250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.770+0000 7ff02067b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c00fb20 con 0x55d956fc1390 2024-07-26T11:23:46.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.770+0000 7ff02067b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c011660 con 0x55d956fb91b0 2024-07-26T11:23:46.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.770+0000 7ff02067b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c0101f0 con 0x55d956fe2070 2024-07-26T11:23:46.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.770+0000 7ff012a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fb91b0 0x55d95738def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:46.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.770+0000 7ff013400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fe2070 0x55d9573aa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:46.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.770+0000 7ff013400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fe2070 0x55d9573aa250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59202/0 (socket says 172.21.15.134:59202) 2024-07-26T11:23:46.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.770+0000 7ff013400640 1 -- 172.21.15.134:0/1624207018 learned_addr learned my addr 172.21.15.134:0/1624207018 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:46.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.771+0000 7ff013400640 1 -- 172.21.15.134:0/1624207018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fb91b0 msgr2=0x55d95738def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:46.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.771+0000 7ff013400640 1 --2- 172.21.15.134:0/1624207018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fb91b0 0x55d95738def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.771+0000 7ff013400640 1 -- 172.21.15.134:0/1624207018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fc1390 msgr2=0x55d957384e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:46.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.771+0000 7ff013400640 1 --2- 172.21.15.134:0/1624207018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fc1390 0x55d957384e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.771+0000 7ff013400640 1 -- 172.21.15.134:0/1624207018 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d957393790 con 0x55d956fe2070 2024-07-26T11:23:46.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.771+0000 7ff012a00640 1 --2- 172.21.15.134:0/1624207018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fb91b0 0x55d95738def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:23:46.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.771+0000 7ff013400640 1 --2- 172.21.15.134:0/1624207018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fe2070 0x55d9573aa250 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff01410b750 tx=0x7ff014051090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=961d45dedf39b4fc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:46.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.771+0000 7fefff400640 1 -- 172.21.15.134:0/1624207018 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014118780 con 0x55d956fe2070 2024-07-26T11:23:46.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.772+0000 7fefff400640 1 -- 172.21.15.134:0/1624207018 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff014076d50 con 0x55d956fe2070 2024-07-26T11:23:46.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.772+0000 7fefff400640 1 -- 172.21.15.134:0/1624207018 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0141462f0 con 0x55d956fe2070 2024-07-26T11:23:46.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.773+0000 7ff02067b740 1 -- 172.21.15.134:0/1624207018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fe2070 msgr2=0x55d9573aa250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:46.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.773+0000 7ff02067b740 1 --2- 172.21.15.134:0/1624207018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fe2070 0x55d9573aa250 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff01410b750 tx=0x7ff014051090 comp rx=0 tx=0).stop 2024-07-26T11:23:46.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.773+0000 7ff02067b740 1 -- 172.21.15.134:0/1624207018 shutdown_connections 2024-07-26T11:23:46.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.774+0000 7ff02067b740 1 --2- 172.21.15.134:0/1624207018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fe2070 0x55d9573aa250 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.774+0000 7ff02067b740 1 --2- 172.21.15.134:0/1624207018 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fc1390 0x55d957384e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.774+0000 7ff02067b740 1 --2- 172.21.15.134:0/1624207018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fb91b0 0x55d95738def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.774+0000 7ff02067b740 1 -- 172.21.15.134:0/1624207018 >> 172.21.15.134:0/1624207018 conn(0x55d956fd6460 msgr2=0x55d957384750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:46.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.774+0000 7ff02067b740 1 -- 172.21.15.134:0/1624207018 shutdown_connections 2024-07-26T11:23:46.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.774+0000 7ff02067b740 1 -- 172.21.15.134:0/1624207018 wait complete. 2024-07-26T11:23:46.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.775+0000 7ff02067b740 1 -- start start 2024-07-26T11:23:46.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.776+0000 7ff02067b740 1 Processor -- start 2024-07-26T11:23:46.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.777+0000 7ff02067b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fb91b0 0x55d957405610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.777+0000 7ff02067b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fc1390 0x55d957405b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.777+0000 7ff02067b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fe2070 0x55d957406090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.777+0000 7ff02067b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c010f90 con 0x55d956fc1390 2024-07-26T11:23:46.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.777+0000 7ff02067b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c0108c0 con 0x55d956fe2070 2024-07-26T11:23:46.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.777+0000 7ff02067b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff00c1451c0 con 0x55d956fb91b0 2024-07-26T11:23:46.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.777+0000 7ff02067b740 10 client.0 _ll_register_callbacks cb 0x7ff00000adb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:23:46.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.777+0000 7ff013400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fe2070 0x55d957406090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:46.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.777+0000 7ff013400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fe2070 0x55d957406090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:54128/0 (socket says 172.21.15.134:54128) 2024-07-26T11:23:46.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.777+0000 7ff013400640 1 -- 172.21.15.134:0/3513526971 learned_addr learned my addr 172.21.15.134:0/3513526971 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:46.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.778+0000 7ff012a00640 1 --2- 172.21.15.134:0/3513526971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fb91b0 0x55d957405610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:46.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.778+0000 7ff013400640 1 -- 172.21.15.134:0/3513526971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fb91b0 msgr2=0x55d957405610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:46.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.778+0000 7ff013400640 1 --2- 172.21.15.134:0/3513526971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fb91b0 0x55d957405610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.778+0000 7ff013400640 1 -- 172.21.15.134:0/3513526971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fc1390 msgr2=0x55d957405b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:46.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.778+0000 7ff013400640 1 --2- 172.21.15.134:0/3513526971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fc1390 0x55d957405b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.778+0000 7ff013400640 1 -- 172.21.15.134:0/3513526971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d957399c20 con 0x55d956fe2070 2024-07-26T11:23:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.778+0000 7ff012a00640 1 --2- 172.21.15.134:0/3513526971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fb91b0 0x55d957405610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:23:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.779+0000 7feffcc00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:23:46.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.779+0000 7ff013400640 1 --2- 172.21.15.134:0/3513526971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fe2070 0x55d957406090 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff01410c940 tx=0x7ff014069840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e8eeb2fb7aa4f58 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:46.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.782+0000 7feffcc00640 1 -- 172.21.15.134:0/3513526971 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff01411d060 con 0x55d956fe2070 2024-07-26T11:23:46.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.782+0000 7feffcc00640 1 -- 172.21.15.134:0/3513526971 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7ff01410bcc0 con 0x55d956fe2070 2024-07-26T11:23:46.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.784+0000 7ff02067b740 1 -- 172.21.15.134:0/3513526971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff014076d50 con 0x55d956fe2070 2024-07-26T11:23:46.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.784+0000 7feffcc00640 1 -- 172.21.15.134:0/3513526971 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff01411d060 con 0x55d956fe2070 2024-07-26T11:23:46.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.785+0000 7ff02067b740 10 client.26588 fetch_fsmap learned FSMap version 32 2024-07-26T11:23:46.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.785+0000 7ff02067b740 1 -- 172.21.15.134:0/3513526971 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d95739ae90 con 0x55d956fe2070 2024-07-26T11:23:46.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.785+0000 7feffcc00640 1 -- 172.21.15.134:0/3513526971 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff014076d50 con 0x55d956fe2070 2024-07-26T11:23:46.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.785+0000 7feffcc00640 1 -- 172.21.15.134:0/3513526971 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7ff014130bd0 con 0x55d956fe2070 2024-07-26T11:23:46.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.785+0000 7ff02067b740 10 client.26588 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:23:46.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.785+0000 7ff02067b740 10 client.26588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:23:46.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.785+0000 7ff02067b740 20 client.26588 populate_metadata read hostname 'smithi134' 2024-07-26T11:23:46.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.785+0000 7ff02067b740 1 --2- 172.21.15.134:0/3513526971 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d956f75da0 0x55d95739b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:46.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.785+0000 7ff02067b740 4 client.26588 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:23:46.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.785+0000 7ff02067b740 1 -- 172.21.15.134:0/3513526971 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d9572fb580 con 0x55d956f75da0 2024-07-26T11:23:46.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.792+0000 7ff012a00640 1 --2- 172.21.15.134:0/3513526971 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d956f75da0 0x55d95739b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:46.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.805+0000 7ff012a00640 1 --2- 172.21.15.134:0/3513526971 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d956f75da0 0x55d95739b120 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d29a1c13bd02c98 server_cookie=aab1f0fee5c4e2fe in_seq=0 out_seq=0 2024-07-26T11:23:46.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.805+0000 7feffcc00640 10 client.26588 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:46.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.806+0000 7feffcc00640 1 -- 172.21.15.134:0/3513526971 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d9572fb580 con 0x55d956f75da0 2024-07-26T11:23:46.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.806+0000 7feffcc00640 10 client.26588 handle_command_reply: tid=0 2024-07-26T11:23:46.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.806+0000 7ff02067b740 1 client.26588 shutdown 2024-07-26T11:23:46.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.807+0000 7ff02067b740 1 -- 172.21.15.134:0/3513526971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fe2070 msgr2=0x55d957406090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:46.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.807+0000 7ff02067b740 1 --2- 172.21.15.134:0/3513526971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fe2070 0x55d957406090 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff01410c940 tx=0x7ff014069840 comp rx=0 tx=0).stop 2024-07-26T11:23:46.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.808+0000 7ff02067b740 1 -- 172.21.15.134:0/3513526971 shutdown_connections 2024-07-26T11:23:46.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.808+0000 7ff02067b740 1 --2- 172.21.15.134:0/3513526971 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d956f75da0 0x55d95739b120 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.808+0000 7ff02067b740 1 --2- 172.21.15.134:0/3513526971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d956fe2070 0x55d957406090 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.808+0000 7ff02067b740 1 --2- 172.21.15.134:0/3513526971 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d956fc1390 0x55d957405b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.808+0000 7ff02067b740 1 --2- 172.21.15.134:0/3513526971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d956fb91b0 0x55d957405610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.808+0000 7ff02067b740 1 -- 172.21.15.134:0/3513526971 >> 172.21.15.134:0/3513526971 conn(0x55d956fd6460 msgr2=0x55d957388000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:46.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.808+0000 7feffcc00640 0 client.26588 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:46.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.809+0000 7ff02067b740 1 -- 172.21.15.134:0/3513526971 shutdown_connections 2024-07-26T11:23:46.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.813+0000 7ff02067b740 1 -- 172.21.15.134:0/3513526971 wait complete. 2024-07-26T11:23:46.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.813+0000 7ff02067b740 20 client.26588 trim_cache size 0 max 16384 2024-07-26T11:23:46.822 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:23:46.822 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "no active scrubs running", 2024-07-26T11:23:46.822 INFO:teuthology.orchestra.run.smithi134.stdout: "scrubs": {} 2024-07-26T11:23:46.822 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:23:46.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.819+0000 7ff013e00640 1 -- 172.21.15.134:0/1342922753 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7feff80b3e90 msgr2=0x7feff80b63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:46.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.819+0000 7ff013e00640 1 --2- 172.21.15.134:0/1342922753 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7feff80b3e90 0x7feff80b63a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff004009230 tx=0x7ff004031400 comp rx=0 tx=0).stop 2024-07-26T11:23:46.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.819+0000 7ff013e00640 1 -- 172.21.15.134:0/1342922753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff00c0ab1b0 msgr2=0x7ff00c1498d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:46.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.819+0000 7ff013e00640 1 --2- 172.21.15.134:0/1342922753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff00c0ab1b0 0x7ff00c1498d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff014052290 tx=0x7ff014075a00 comp rx=0 tx=0).stop 2024-07-26T11:23:46.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.820+0000 7ff013e00640 1 -- 172.21.15.134:0/1342922753 shutdown_connections 2024-07-26T11:23:46.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.820+0000 7ff013e00640 1 --2- 172.21.15.134:0/1342922753 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7feff80b3e90 0x7feff80b63a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.820+0000 7ff013e00640 1 --2- 172.21.15.134:0/1342922753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff00c0ab1b0 0x7ff00c1498d0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.820+0000 7ff013e00640 1 --2- 172.21.15.134:0/1342922753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff00c0aa860 0x7ff00c144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.820+0000 7ff013e00640 1 --2- 172.21.15.134:0/1342922753 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff00c0a50f0 0x7ff00c144090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:46.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.820+0000 7ff013e00640 1 -- 172.21.15.134:0/1342922753 >> 172.21.15.134:0/1342922753 conn(0x7ff00c01cc70 msgr2=0x7ff00c0a39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:46.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.820+0000 7ff013e00640 1 -- 172.21.15.134:0/1342922753 shutdown_connections 2024-07-26T11:23:46.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:46.820+0000 7ff013e00640 1 -- 172.21.15.134:0/1342922753 wait complete. 2024-07-26T11:23:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:47 smithi134 ceph-mon[31966]: pgmap v716: 129 pgs: 129 active+clean; 59 GiB data, 123 GiB used, 949 GiB / 1.0 TiB avail; 43 KiB/s rd, 243 MiB/s wr, 1.41k op/s 2024-07-26T11:23:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:47 smithi189 ceph-mon[36790]: pgmap v716: 129 pgs: 129 active+clean; 59 GiB data, 123 GiB used, 949 GiB / 1.0 TiB avail; 43 KiB/s rd, 243 MiB/s wr, 1.41k op/s 2024-07-26T11:23:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:47 smithi193 ceph-mon[37473]: pgmap v716: 129 pgs: 129 active+clean; 59 GiB data, 123 GiB used, 949 GiB / 1.0 TiB avail; 43 KiB/s rd, 243 MiB/s wr, 1.41k op/s 2024-07-26T11:23:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:48 smithi193 ceph-mon[37473]: pgmap v717: 129 pgs: 129 active+clean; 59 GiB data, 124 GiB used, 948 GiB / 1.0 TiB avail; 56 KiB/s rd, 245 MiB/s wr, 1.33k op/s 2024-07-26T11:23:49.141 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:48 smithi134 ceph-mon[31966]: pgmap v717: 129 pgs: 129 active+clean; 59 GiB data, 124 GiB used, 948 GiB / 1.0 TiB avail; 56 KiB/s rd, 245 MiB/s wr, 1.33k op/s 2024-07-26T11:23:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:48 smithi189 ceph-mon[36790]: pgmap v717: 129 pgs: 129 active+clean; 59 GiB data, 124 GiB used, 948 GiB / 1.0 TiB avail; 56 KiB/s rd, 245 MiB/s wr, 1.33k op/s 2024-07-26T11:23:49.746 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-26T11:23:49.747 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:23:50.748 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:23:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:51 smithi134 ceph-mon[31966]: pgmap v718: 129 pgs: 129 active+clean; 60 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 39 KiB/s rd, 285 MiB/s wr, 1.46k op/s 2024-07-26T11:23:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:51 smithi189 ceph-mon[36790]: pgmap v718: 129 pgs: 129 active+clean; 60 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 39 KiB/s rd, 285 MiB/s wr, 1.46k op/s 2024-07-26T11:23:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:51 smithi193 ceph-mon[37473]: pgmap v718: 129 pgs: 129 active+clean; 60 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 39 KiB/s rd, 285 MiB/s wr, 1.46k op/s 2024-07-26T11:23:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:52 smithi134 ceph-mon[31966]: pgmap v719: 129 pgs: 129 active+clean; 60 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 56 KiB/s rd, 231 MiB/s wr, 1.13k op/s 2024-07-26T11:23:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:52 smithi189 ceph-mon[36790]: pgmap v719: 129 pgs: 129 active+clean; 60 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 56 KiB/s rd, 231 MiB/s wr, 1.13k op/s 2024-07-26T11:23:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:52 smithi193 ceph-mon[37473]: pgmap v719: 129 pgs: 129 active+clean; 60 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 56 KiB/s rd, 231 MiB/s wr, 1.13k op/s 2024-07-26T11:23:54.329 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.327+0000 7f095f200640 1 -- 172.21.15.134:0/491633126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0960107800 msgr2=0x7f096010e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:54.329 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.327+0000 7f095f200640 1 --2- 172.21.15.134:0/491633126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0960107800 0x7f096010e030 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0958009be0 tx=0x7f095802a6f0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.329+0000 7f095f200640 1 -- 172.21.15.134:0/491633126 shutdown_connections 2024-07-26T11:23:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.329+0000 7f095f200640 1 --2- 172.21.15.134:0/491633126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0960107800 0x7f096010e030 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.329+0000 7f095f200640 1 --2- 172.21.15.134:0/491633126 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0960106eb0 0x7f09601072c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.329+0000 7f095f200640 1 --2- 172.21.15.134:0/491633126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0960101580 0x7f0960101990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.329+0000 7f095f200640 1 -- 172.21.15.134:0/491633126 >> 172.21.15.134:0/491633126 conn(0x7f09600781a0 msgr2=0x7f09600785b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.332+0000 7f095f200640 1 -- 172.21.15.134:0/491633126 shutdown_connections 2024-07-26T11:23:54.335 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.332+0000 7f095f200640 1 -- 172.21.15.134:0/491633126 wait complete. 2024-07-26T11:23:54.336 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.334+0000 7f095f200640 1 Processor -- start 2024-07-26T11:23:54.336 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.334+0000 7f095f200640 1 -- start start 2024-07-26T11:23:54.337 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.335+0000 7f095f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0960101580 0x7f096010b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.338 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.335+0000 7f095f200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0960107800 0x7f096010ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.338 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.336+0000 7f095f200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f096010bfd0 0x7f0960073860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.338 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.336+0000 7f095f200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f0960110e60 con 0x7f0960107800 2024-07-26T11:23:54.338 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.336+0000 7f095f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0960110ce0 con 0x7f0960101580 2024-07-26T11:23:54.338 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.336+0000 7f095f200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f0960110fe0 con 0x7f096010bfd0 2024-07-26T11:23:54.338 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.337+0000 7f095e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f096010bfd0 0x7f0960073860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.337+0000 7f095e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f096010bfd0 0x7f0960073860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34910/0 (socket says 172.21.15.134:34910) 2024-07-26T11:23:54.339 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.337+0000 7f095e800640 1 -- 172.21.15.134:0/513037152 learned_addr learned my addr 172.21.15.134:0/513037152 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.338+0000 7f095de00640 1 --2- 172.21.15.134:0/513037152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0960101580 0x7f096010b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.338+0000 7f0955400640 1 --2- 172.21.15.134:0/513037152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0960107800 0x7f096010ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.339+0000 7f095e800640 1 -- 172.21.15.134:0/513037152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0960101580 msgr2=0x7f096010b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.339+0000 7f095e800640 1 --2- 172.21.15.134:0/513037152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0960101580 0x7f096010b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.339+0000 7f095e800640 1 -- 172.21.15.134:0/513037152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0960107800 msgr2=0x7f096010ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.339+0000 7f095e800640 1 --2- 172.21.15.134:0/513037152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0960107800 0x7f096010ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.339+0000 7f095e800640 1 -- 172.21.15.134:0/513037152 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0960073ea0 con 0x7f096010bfd0 2024-07-26T11:23:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.341+0000 7f095e800640 1 --2- 172.21.15.134:0/513037152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f096010bfd0 0x7f0960073860 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0958030d40 tx=0x7f09580315d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.342+0000 7f0957400640 1 -- 172.21.15.134:0/513037152 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0958040020 con 0x7f096010bfd0 2024-07-26T11:23:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.342+0000 7f0957400640 1 -- 172.21.15.134:0/513037152 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0958003e20 con 0x7f096010bfd0 2024-07-26T11:23:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.343+0000 7f095f200640 1 -- 172.21.15.134:0/513037152 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0960074130 con 0x7f096010bfd0 2024-07-26T11:23:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.343+0000 7f095f200640 1 -- 172.21.15.134:0/513037152 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09600745f0 con 0x7f096010bfd0 2024-07-26T11:23:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.345+0000 7f0957400640 1 -- 172.21.15.134:0/513037152 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0958036040 con 0x7f096010bfd0 2024-07-26T11:23:54.355 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.352+0000 7f096bb8b740 1 Processor -- start 2024-07-26T11:23:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.358+0000 7f0957400640 1 -- 172.21.15.134:0/513037152 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f0958043020 con 0x7f096010bfd0 2024-07-26T11:23:54.363 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.360+0000 7f0957400640 1 --2- 172.21.15.134:0/513037152 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f09440b3c50 0x7f09440b6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.364 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.362+0000 7f096bb8b740 1 -- start start 2024-07-26T11:23:54.364 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.362+0000 7f096bb8b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483a01b0 0x55e648743c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.364 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.362+0000 7f096bb8b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6483772f0 0x55e648744160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.364 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.362+0000 7f096bb8b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e64837f4d0 0x55e6487446a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.362+0000 7f096bb8b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e6486b8270 con 0x55e6483a01b0 2024-07-26T11:23:54.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.362+0000 7f096bb8b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e6486a7770 con 0x55e6483772f0 2024-07-26T11:23:54.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.362+0000 7f096bb8b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e6486ab7b0 con 0x55e64837f4d0 2024-07-26T11:23:54.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.363+0000 7f0955400640 1 --2- 172.21.15.134:0/513037152 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f09440b3c50 0x7f09440b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.364+0000 7f095de00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483a01b0 0x55e648743c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.364+0000 7f0957400640 1 -- 172.21.15.134:0/513037152 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f0958104590 con 0x7f096010bfd0 2024-07-26T11:23:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.366+0000 7f095e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e64837f4d0 0x55e6487446a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.366+0000 7f0955400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6483772f0 0x55e648744160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.366+0000 7f095e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e64837f4d0 0x55e6487446a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34912/0 (socket says 172.21.15.134:34912) 2024-07-26T11:23:54.368 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.366+0000 7f095e800640 1 -- 172.21.15.134:0/1582352136 learned_addr learned my addr 172.21.15.134:0/1582352136 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:54.369 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.367+0000 7f095de00640 1 -- 172.21.15.134:0/1582352136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e64837f4d0 msgr2=0x55e6487446a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:54.369 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.367+0000 7f095de00640 1 --2- 172.21.15.134:0/1582352136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e64837f4d0 0x55e6487446a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.367+0000 7f095de00640 1 -- 172.21.15.134:0/1582352136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6483772f0 msgr2=0x55e648744160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:54.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.367+0000 7f095de00640 1 --2- 172.21.15.134:0/1582352136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6483772f0 0x55e648744160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.367+0000 7f095de00640 1 -- 172.21.15.134:0/1582352136 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e64840b790 con 0x55e6483a01b0 2024-07-26T11:23:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.369+0000 7f095de00640 1 --2- 172.21.15.134:0/1582352136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483a01b0 0x55e648743c20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0948005fd0 tx=0x7f094800ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a49ebcfad0a5937 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.369+0000 7f0955400640 1 --2- 172.21.15.134:0/513037152 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f09440b3c50 0x7f09440b6160 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x55e6487428a0 tx=0x7f09500094c0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:54.373 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.370+0000 7f094f000640 1 -- 172.21.15.134:0/1582352136 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0948004020 con 0x55e6483a01b0 2024-07-26T11:23:54.373 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.370+0000 7f094f000640 1 -- 172.21.15.134:0/1582352136 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09480041c0 con 0x55e6483a01b0 2024-07-26T11:23:54.373 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.370+0000 7f094f000640 1 -- 172.21.15.134:0/1582352136 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0948003cf0 con 0x55e6483a01b0 2024-07-26T11:23:54.376 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.374+0000 7f096bb8b740 1 -- 172.21.15.134:0/1582352136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483a01b0 msgr2=0x55e648743c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:54.376 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.375+0000 7f096bb8b740 1 --2- 172.21.15.134:0/1582352136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483a01b0 0x55e648743c20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0948005fd0 tx=0x7f094800ac00 comp rx=0 tx=0).stop 2024-07-26T11:23:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.376+0000 7f096bb8b740 1 -- 172.21.15.134:0/1582352136 shutdown_connections 2024-07-26T11:23:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.376+0000 7f096bb8b740 1 --2- 172.21.15.134:0/1582352136 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e64837f4d0 0x55e6487446a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.376+0000 7f096bb8b740 1 --2- 172.21.15.134:0/1582352136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6483772f0 0x55e648744160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.376+0000 7f096bb8b740 1 --2- 172.21.15.134:0/1582352136 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483a01b0 0x55e648743c20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.376+0000 7f096bb8b740 1 -- 172.21.15.134:0/1582352136 >> 172.21.15.134:0/1582352136 conn(0x55e648394300 msgr2=0x55e648741b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.377+0000 7f096bb8b740 1 -- 172.21.15.134:0/1582352136 shutdown_connections 2024-07-26T11:23:54.379 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.377+0000 7f096bb8b740 1 -- 172.21.15.134:0/1582352136 wait complete. 2024-07-26T11:23:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.378+0000 7f096bb8b740 1 -- start start 2024-07-26T11:23:54.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.378+0000 7f096bb8b740 1 Processor -- start 2024-07-26T11:23:54.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.380+0000 7f096bb8b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6483772f0 0x55e64874f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.380+0000 7f096bb8b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e64837f4d0 0x55e64874fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.380+0000 7f096bb8b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6483a01b0 0x55e64875b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.384 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.380+0000 7f095de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6483772f0 0x55e64874f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.384 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.380+0000 7f095de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6483772f0 0x55e64874f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34920/0 (socket says 172.21.15.134:34920) 2024-07-26T11:23:54.384 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.381+0000 7f096bb8b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55e6485e1ab0 con 0x55e64837f4d0 2024-07-26T11:23:54.384 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.381+0000 7f096bb8b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55e648644d50 con 0x55e6483a01b0 2024-07-26T11:23:54.384 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.381+0000 7f096bb8b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55e6487523a0 con 0x55e6483772f0 2024-07-26T11:23:54.384 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.381+0000 7f096bb8b740 10 client.0 _ll_register_callbacks cb 0x55e6483b7a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:23:54.384 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.381+0000 7f095de00640 1 -- 172.21.15.134:0/2227087414 learned_addr learned my addr 172.21.15.134:0/2227087414 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:54.384 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.382+0000 7f095de00640 1 -- 172.21.15.134:0/2227087414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6483a01b0 msgr2=0x55e64875b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:54.384 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.382+0000 7f095de00640 1 --2- 172.21.15.134:0/2227087414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6483a01b0 0x55e64875b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.382+0000 7f0955400640 1 --2- 172.21.15.134:0/2227087414 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e64837f4d0 0x55e64874fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.382+0000 7f095de00640 1 -- 172.21.15.134:0/2227087414 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e64837f4d0 msgr2=0x55e64874fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:54.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.382+0000 7f095de00640 1 --2- 172.21.15.134:0/2227087414 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e64837f4d0 0x55e64874fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.382+0000 7f095de00640 1 -- 172.21.15.134:0/2227087414 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e648484620 con 0x55e6483772f0 2024-07-26T11:23:54.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.382+0000 7f096bb8b740 1 -- 172.21.15.134:0/2227087414 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09480041c0 con 0x55e6483772f0 2024-07-26T11:23:54.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.382+0000 7f0955400640 1 --2- 172.21.15.134:0/2227087414 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e64837f4d0 0x55e64874fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:23:54.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.382+0000 7f095de00640 1 --2- 172.21.15.134:0/2227087414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6483772f0 0x55e64874f830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f09480096d0 tx=0x7f09480045a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe2949939899d7a0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:54.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.382+0000 7f0943400640 10 client.26614 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:23:54.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.384+0000 7f0943400640 1 -- 172.21.15.134:0/2227087414 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094802e050 con 0x55e6483772f0 2024-07-26T11:23:54.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.384+0000 7f0943400640 1 -- 172.21.15.134:0/2227087414 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f0948005040 con 0x55e6483772f0 2024-07-26T11:23:54.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.385+0000 7f0943400640 1 -- 172.21.15.134:0/2227087414 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0948005260 con 0x55e6483772f0 2024-07-26T11:23:54.389 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.388+0000 7f096bb8b740 10 client.26614 fetch_fsmap learned FSMap version 32 2024-07-26T11:23:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.388+0000 7f096bb8b740 1 -- 172.21.15.134:0/2227087414 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=32}) -- 0x55e64875e3b0 con 0x55e6483772f0 2024-07-26T11:23:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.389+0000 7f0943400640 1 -- 172.21.15.134:0/2227087414 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f0948005400 con 0x55e6483772f0 2024-07-26T11:23:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.389+0000 7f096bb8b740 10 client.26614 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:23:54.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.390+0000 7f096bb8b740 10 client.26614 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:23:54.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.390+0000 7f096bb8b740 20 client.26614 populate_metadata read hostname 'smithi134' 2024-07-26T11:23:54.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.390+0000 7f096bb8b740 1 --2- 172.21.15.134:0/2227087414 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e64835ed80 0x55e648761770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.390+0000 7f096bb8b740 4 client.26614 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:23:54.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.390+0000 7f096bb8b740 1 -- 172.21.15.134:0/2227087414 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e6486ba1c0 con 0x55e64835ed80 2024-07-26T11:23:54.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.391+0000 7f0955400640 1 --2- 172.21.15.134:0/2227087414 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e64835ed80 0x55e648761770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.397+0000 7f0943400640 10 client.26614 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:54.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.397+0000 7f0955400640 1 --2- 172.21.15.134:0/2227087414 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e64835ed80 0x55e648761770 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da8dc2a337975e6f server_cookie=28e2b5345442aa44 in_seq=0 out_seq=0 2024-07-26T11:23:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.403+0000 7f0943400640 1 -- 172.21.15.134:0/2227087414 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55e6486ba1c0 con 0x55e64835ed80 2024-07-26T11:23:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.403+0000 7f0943400640 10 client.26614 handle_command_reply: tid=0 2024-07-26T11:23:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.403+0000 7f096bb8b740 1 client.26614 shutdown 2024-07-26T11:23:54.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.404+0000 7f096bb8b740 1 -- 172.21.15.134:0/2227087414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6483772f0 msgr2=0x55e64874f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:54.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.404+0000 7f096bb8b740 1 --2- 172.21.15.134:0/2227087414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6483772f0 0x55e64874f830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f09480096d0 tx=0x7f09480045a0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.404+0000 7f096bb8b740 1 -- 172.21.15.134:0/2227087414 shutdown_connections 2024-07-26T11:23:54.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.404+0000 7f096bb8b740 1 --2- 172.21.15.134:0/2227087414 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e64835ed80 0x55e648761770 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.404+0000 7f096bb8b740 1 --2- 172.21.15.134:0/2227087414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6483a01b0 0x55e64875b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.404+0000 7f096bb8b740 1 --2- 172.21.15.134:0/2227087414 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e64837f4d0 0x55e64874fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.404+0000 7f096bb8b740 1 --2- 172.21.15.134:0/2227087414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6483772f0 0x55e64874f830 secure :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f09480096d0 tx=0x7f09480045a0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.404+0000 7f0943400640 0 client.26614 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.404+0000 7f096bb8b740 1 -- 172.21.15.134:0/2227087414 >> 172.21.15.134:0/2227087414 conn(0x55e648394300 msgr2=0x55e648744c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.412+0000 7f096bb8b740 1 -- 172.21.15.134:0/2227087414 shutdown_connections 2024-07-26T11:23:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.412+0000 7f096bb8b740 1 -- 172.21.15.134:0/2227087414 wait complete. 2024-07-26T11:23:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.412+0000 7f096bb8b740 20 client.26614 trim_cache size 0 max 16384 2024-07-26T11:23:54.471 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.468+0000 7f096bb8b740 1 Processor -- start 2024-07-26T11:23:54.471 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.469+0000 7f096bb8b740 1 -- start start 2024-07-26T11:23:54.472 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.470+0000 7f096bb8b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e64837f4d0 0x55e64875b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.472 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.470+0000 7f096bb8b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6483a01b0 0x55e6487493f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.472 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.470+0000 7f096bb8b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483341c0 0x55e648764790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.472 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.470+0000 7f096bb8b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f09601bf8c0 con 0x55e6483341c0 2024-07-26T11:23:54.472 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.470+0000 7f096bb8b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f09601c1400 con 0x55e64837f4d0 2024-07-26T11:23:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.470+0000 7f096bb8b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f09601bff90 con 0x55e6483a01b0 2024-07-26T11:23:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.470+0000 7f095de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e64837f4d0 0x55e64875b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.470+0000 7f095de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e64837f4d0 0x55e64875b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:39832/0 (socket says 172.21.15.134:39832) 2024-07-26T11:23:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.470+0000 7f095de00640 1 -- 172.21.15.134:0/3747696953 learned_addr learned my addr 172.21.15.134:0/3747696953 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.470+0000 7f0955400640 1 --2- 172.21.15.134:0/3747696953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6483a01b0 0x55e6487493f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.471+0000 7f095e800640 1 --2- 172.21.15.134:0/3747696953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483341c0 0x55e648764790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.471+0000 7f095e800640 1 -- 172.21.15.134:0/3747696953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6483a01b0 msgr2=0x55e6487493f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.471+0000 7f095e800640 1 --2- 172.21.15.134:0/3747696953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6483a01b0 0x55e6487493f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.471+0000 7f095e800640 1 -- 172.21.15.134:0/3747696953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e64837f4d0 msgr2=0x55e64875b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.471+0000 7f095e800640 1 --2- 172.21.15.134:0/3747696953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e64837f4d0 0x55e64875b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.474 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.471+0000 7f095e800640 1 -- 172.21.15.134:0/3747696953 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e6487426d0 con 0x55e6483341c0 2024-07-26T11:23:54.475 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.472+0000 7f095e800640 1 --2- 172.21.15.134:0/3747696953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483341c0 0x55e648764790 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0958101900 tx=0x7f09580c6b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68e608e00934b15 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:54.475 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.473+0000 7f094e600640 1 -- 172.21.15.134:0/3747696953 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09580ca4d0 con 0x55e6483341c0 2024-07-26T11:23:54.475 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.473+0000 7f094e600640 1 -- 172.21.15.134:0/3747696953 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09580ca870 con 0x55e6483341c0 2024-07-26T11:23:54.475 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.473+0000 7f094e600640 1 -- 172.21.15.134:0/3747696953 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0958030d70 con 0x55e6483341c0 2024-07-26T11:23:54.478 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.475+0000 7f096bb8b740 1 -- 172.21.15.134:0/3747696953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483341c0 msgr2=0x55e648764790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:54.478 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.476+0000 7f096bb8b740 1 --2- 172.21.15.134:0/3747696953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483341c0 0x55e648764790 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0958101900 tx=0x7f09580c6b90 comp rx=0 tx=0).stop 2024-07-26T11:23:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.477+0000 7f096bb8b740 1 -- 172.21.15.134:0/3747696953 shutdown_connections 2024-07-26T11:23:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.477+0000 7f096bb8b740 1 --2- 172.21.15.134:0/3747696953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e6483a01b0 0x55e6487493f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.477+0000 7f096bb8b740 1 --2- 172.21.15.134:0/3747696953 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483341c0 0x55e648764790 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.477+0000 7f096bb8b740 1 --2- 172.21.15.134:0/3747696953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e64837f4d0 0x55e64875b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.477+0000 7f096bb8b740 1 -- 172.21.15.134:0/3747696953 >> 172.21.15.134:0/3747696953 conn(0x55e648394300 msgr2=0x55e648748ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.486+0000 7f096bb8b740 1 -- 172.21.15.134:0/3747696953 shutdown_connections 2024-07-26T11:23:54.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.486+0000 7f096bb8b740 1 -- 172.21.15.134:0/3747696953 wait complete. 2024-07-26T11:23:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.487+0000 7f096bb8b740 1 -- start start 2024-07-26T11:23:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.487+0000 7f096bb8b740 1 Processor -- start 2024-07-26T11:23:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.490+0000 7f096bb8b740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483341c0 0x55e648764f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.490+0000 7f096bb8b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e64837f4d0 0x55e648762450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.490+0000 7f096bb8b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6483a01b0 0x55e648762aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.490+0000 7f096bb8b740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f09601c0d30 con 0x55e6483341c0 2024-07-26T11:23:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.490+0000 7f096bb8b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f09601c0660 con 0x55e6483a01b0 2024-07-26T11:23:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.490+0000 7f096bb8b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f09601bd8c0 con 0x55e64837f4d0 2024-07-26T11:23:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.490+0000 7f096bb8b740 10 client.0 _ll_register_callbacks cb 0x7f0950009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:23:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.492+0000 7f095e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6483a01b0 0x55e648762aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.492+0000 7f095e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6483a01b0 0x55e648762aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:39844/0 (socket says 172.21.15.134:39844) 2024-07-26T11:23:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.492+0000 7f095e800640 1 -- 172.21.15.134:0/1523666452 learned_addr learned my addr 172.21.15.134:0/1523666452 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.492+0000 7f0955400640 1 --2- 172.21.15.134:0/1523666452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e64837f4d0 0x55e648762450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.492+0000 7f095de00640 1 --2- 172.21.15.134:0/1523666452 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483341c0 0x55e648764f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.493+0000 7f095de00640 1 -- 172.21.15.134:0/1523666452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e64837f4d0 msgr2=0x55e648762450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.494+0000 7f095de00640 1 --2- 172.21.15.134:0/1523666452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e64837f4d0 0x55e648762450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.494+0000 7f095de00640 1 -- 172.21.15.134:0/1523666452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6483a01b0 msgr2=0x55e648762aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:23:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.494+0000 7f095de00640 1 --2- 172.21.15.134:0/1523666452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6483a01b0 0x55e648762aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.494+0000 7f095de00640 1 -- 172.21.15.134:0/1523666452 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e648763230 con 0x55e6483341c0 2024-07-26T11:23:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.494+0000 7f0955400640 1 --2- 172.21.15.134:0/1523666452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e64837f4d0 0x55e648762450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:23:54.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.494+0000 7f095e800640 1 --2- 172.21.15.134:0/1523666452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6483a01b0 0x55e648762aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:23:54.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.495+0000 7f095de00640 1 --2- 172.21.15.134:0/1523666452 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483341c0 0x55e648764f90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0948000c00 tx=0x7f0948021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26083689324b9cc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:54.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.495+0000 7f0943400640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:23:54.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.495+0000 7f0943400640 1 -- 172.21.15.134:0/1523666452 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f094802e070 con 0x55e6483341c0 2024-07-26T11:23:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.496+0000 7f096bb8b740 1 -- 172.21.15.134:0/1523666452 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09580ca870 con 0x55e6483341c0 2024-07-26T11:23:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.496+0000 7f0943400640 1 -- 172.21.15.134:0/1523666452 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f0948016680 con 0x55e6483341c0 2024-07-26T11:23:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.496+0000 7f0943400640 1 -- 172.21.15.134:0/1523666452 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0948042050 con 0x55e6483341c0 2024-07-26T11:23:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.496+0000 7f096bb8b740 10 client.16071 fetch_fsmap learned FSMap version 32 2024-07-26T11:23:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.496+0000 7f096bb8b740 1 -- 172.21.15.134:0/1523666452 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55e6487c9d20 con 0x55e6483341c0 2024-07-26T11:23:54.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.497+0000 7f0943400640 1 -- 172.21.15.134:0/1523666452 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09580ca870 con 0x55e6483341c0 2024-07-26T11:23:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.497+0000 7f0943400640 1 -- 172.21.15.134:0/1523666452 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f094801f4b0 con 0x55e6483341c0 2024-07-26T11:23:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.500+0000 7f096bb8b740 10 client.16071 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:23:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.500+0000 7f096bb8b740 10 client.16071 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:23:54.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.500+0000 7f096bb8b740 20 client.16071 populate_metadata read hostname 'smithi134' 2024-07-26T11:23:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.500+0000 7f096bb8b740 1 --2- 172.21.15.134:0/1523666452 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e6487ca490 0x55e6487cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.500+0000 7f096bb8b740 4 client.16071 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:23:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.500+0000 7f096bb8b740 1 -- 172.21.15.134:0/1523666452 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e6486b5c60 con 0x55e6487ca490 2024-07-26T11:23:54.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.500+0000 7f0955400640 1 --2- 172.21.15.134:0/1523666452 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e6487ca490 0x55e6487cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:54.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.502+0000 7f0955400640 1 --2- 172.21.15.134:0/1523666452 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e6487ca490 0x55e6487cc8b0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e8b7978f37acb6e server_cookie=c1b2e9238b07e5f0 in_seq=0 out_seq=0 2024-07-26T11:23:54.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.502+0000 7f0943400640 10 client.16071 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.833+0000 7f0943400640 1 -- 172.21.15.134:0/1523666452 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e6486b5c60 con 0x55e6487ca490 2024-07-26T11:23:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.833+0000 7f0943400640 10 client.16071 handle_command_reply: tid=0 2024-07-26T11:23:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.833+0000 7f096bb8b740 1 client.16071 shutdown 2024-07-26T11:23:54.840 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.838+0000 7f096bb8b740 1 -- 172.21.15.134:0/1523666452 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483341c0 msgr2=0x55e648764f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:54.840 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.838+0000 7f096bb8b740 1 --2- 172.21.15.134:0/1523666452 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483341c0 0x55e648764f90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0948000c00 tx=0x7f0948021000 comp rx=0 tx=0).stop 2024-07-26T11:23:54.842 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.840+0000 7f096bb8b740 1 -- 172.21.15.134:0/1523666452 shutdown_connections 2024-07-26T11:23:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.841+0000 7f096bb8b740 1 --2- 172.21.15.134:0/1523666452 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55e6487ca490 0x55e6487cc8b0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.841+0000 7f096bb8b740 1 --2- 172.21.15.134:0/1523666452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e64837f4d0 0x55e648762450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.841+0000 7f096bb8b740 1 --2- 172.21.15.134:0/1523666452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e6483a01b0 0x55e648762aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.841+0000 7f096bb8b740 1 --2- 172.21.15.134:0/1523666452 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55e6483341c0 0x55e648764f90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.841+0000 7f096bb8b740 1 -- 172.21.15.134:0/1523666452 >> 172.21.15.134:0/1523666452 conn(0x55e648394300 msgr2=0x55e64874ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:54.844 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.841+0000 7f0943400640 0 client.16071 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:23:54.845 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.843+0000 7f096bb8b740 1 -- 172.21.15.134:0/1523666452 shutdown_connections 2024-07-26T11:23:54.845 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.843+0000 7f096bb8b740 1 -- 172.21.15.134:0/1523666452 wait complete. 2024-07-26T11:23:54.845 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.843+0000 7f096bb8b740 20 client.16071 trim_cache size 0 max 16384 2024-07-26T11:23:54.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:23:54.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:23:54.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:23:54.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:23:54.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.845+0000 7f095f200640 1 -- 172.21.15.134:0/513037152 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f09440b3c50 msgr2=0x7f09440b6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:54.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.845+0000 7f095f200640 1 --2- 172.21.15.134:0/513037152 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f09440b3c50 0x7f09440b6160 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x55e6487428a0 tx=0x7f09500094c0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.845+0000 7f095f200640 1 -- 172.21.15.134:0/513037152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f096010bfd0 msgr2=0x7f0960073860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:54.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.845+0000 7f095f200640 1 --2- 172.21.15.134:0/513037152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f096010bfd0 0x7f0960073860 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0958030d40 tx=0x7f09580315d0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.846+0000 7f095f200640 1 -- 172.21.15.134:0/513037152 shutdown_connections 2024-07-26T11:23:54.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.846+0000 7f095f200640 1 --2- 172.21.15.134:0/513037152 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f09440b3c50 0x7f09440b6160 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.846+0000 7f095f200640 1 --2- 172.21.15.134:0/513037152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f096010bfd0 0x7f0960073860 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.846+0000 7f095f200640 1 --2- 172.21.15.134:0/513037152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f0960107800 0x7f096010ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.846+0000 7f095f200640 1 --2- 172.21.15.134:0/513037152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0960101580 0x7f096010b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:54.849 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.846+0000 7f095f200640 1 -- 172.21.15.134:0/513037152 >> 172.21.15.134:0/513037152 conn(0x7f09600781a0 msgr2=0x7f0960102e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:54.850 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.849+0000 7f095f200640 1 -- 172.21.15.134:0/513037152 shutdown_connections 2024-07-26T11:23:54.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:23:54.849+0000 7f095f200640 1 -- 172.21.15.134:0/513037152 wait complete. 2024-07-26T11:23:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:55 smithi134 ceph-mon[31966]: pgmap v720: 129 pgs: 129 active+clean; 60 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 32 KiB/s rd, 248 MiB/s wr, 1.21k op/s 2024-07-26T11:23:55.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:55 smithi189 ceph-mon[36790]: pgmap v720: 129 pgs: 129 active+clean; 60 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 32 KiB/s rd, 248 MiB/s wr, 1.21k op/s 2024-07-26T11:23:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:55 smithi193 ceph-mon[37473]: pgmap v720: 129 pgs: 129 active+clean; 60 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 32 KiB/s rd, 248 MiB/s wr, 1.21k op/s 2024-07-26T11:23:57.937 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:57 smithi134 ceph-mon[31966]: pgmap v721: 129 pgs: 129 active+clean; 61 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 32 KiB/s rd, 259 MiB/s wr, 1.26k op/s 2024-07-26T11:23:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:57 smithi189 ceph-mon[36790]: pgmap v721: 129 pgs: 129 active+clean; 61 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 32 KiB/s rd, 259 MiB/s wr, 1.26k op/s 2024-07-26T11:23:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:57 smithi193 ceph-mon[37473]: pgmap v721: 129 pgs: 129 active+clean; 61 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 32 KiB/s rd, 259 MiB/s wr, 1.26k op/s 2024-07-26T11:23:58.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.088+0000 7f195d200640 1 -- 172.21.15.134:0/4067395704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1958075800 msgr2=0x7f1958075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:58.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.088+0000 7f195d200640 1 --2- 172.21.15.134:0/4067395704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1958075800 0x7f1958075c10 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1940009900 tx=0x7f194002a400 comp rx=0 tx=0).stop 2024-07-26T11:23:58.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.088+0000 7f195d200640 1 -- 172.21.15.134:0/4067395704 shutdown_connections 2024-07-26T11:23:58.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.088+0000 7f195d200640 1 --2- 172.21.15.134:0/4067395704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1958075800 0x7f1958075c10 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:58.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.088+0000 7f195d200640 1 --2- 172.21.15.134:0/4067395704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1958113cb0 0x7f1958116180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:58.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.088+0000 7f195d200640 1 --2- 172.21.15.134:0/4067395704 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1958076db0 0x7f19580751e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:58.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.088+0000 7f195d200640 1 -- 172.21.15.134:0/4067395704 >> 172.21.15.134:0/4067395704 conn(0x7f19580fee30 msgr2=0x7f1958101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:58.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.101+0000 7f195d200640 1 -- 172.21.15.134:0/4067395704 shutdown_connections 2024-07-26T11:23:58.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.102+0000 7f195d200640 1 -- 172.21.15.134:0/4067395704 wait complete. 2024-07-26T11:23:58.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.104+0000 7f195d200640 1 Processor -- start 2024-07-26T11:23:58.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.105+0000 7f195d200640 1 -- start start 2024-07-26T11:23:58.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.106+0000 7f195d200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1958075800 0x7f19581a0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:58.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.106+0000 7f195d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1958076db0 0x7f19581a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:58.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.106+0000 7f195d200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1958113cb0 0x7f19581a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:58.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.106+0000 7f195d200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f1958119940 con 0x7f1958075800 2024-07-26T11:23:58.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.106+0000 7f195d200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f19581197c0 con 0x7f1958076db0 2024-07-26T11:23:58.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.106+0000 7f195d200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f1958119ac0 con 0x7f1958113cb0 2024-07-26T11:23:58.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.107+0000 7f1957e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1958113cb0 0x7f19581a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:58.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.107+0000 7f1957e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1958113cb0 0x7f19581a5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34952/0 (socket says 172.21.15.134:34952) 2024-07-26T11:23:58.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.107+0000 7f1957e00640 1 -- 172.21.15.134:0/2991858563 learned_addr learned my addr 172.21.15.134:0/2991858563 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:23:58.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.108+0000 7f194ea00640 1 --2- 172.21.15.134:0/2991858563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1958076db0 0x7f19581a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:58.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.108+0000 7f1957400640 1 --2- 172.21.15.134:0/2991858563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1958075800 0x7f19581a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:58.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.110+0000 7f194ea00640 1 -- 172.21.15.134:0/2991858563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1958113cb0 msgr2=0x7f19581a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:58.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.110+0000 7f194ea00640 1 --2- 172.21.15.134:0/2991858563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1958113cb0 0x7f19581a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:58.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.111+0000 7f194ea00640 1 -- 172.21.15.134:0/2991858563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1958075800 msgr2=0x7f19581a0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:58.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.111+0000 7f194ea00640 1 --2- 172.21.15.134:0/2991858563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1958075800 0x7f19581a0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:58.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.111+0000 7f194ea00640 1 -- 172.21.15.134:0/2991858563 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19581a62a0 con 0x7f1958076db0 2024-07-26T11:23:58.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.112+0000 7f194ea00640 1 --2- 172.21.15.134:0/2991858563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1958076db0 0x7f19581a08d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f194002a920 tx=0x7f1940003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:58.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.113+0000 7f1954c00640 1 -- 172.21.15.134:0/2991858563 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194003f070 con 0x7f1958076db0 2024-07-26T11:23:58.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.113+0000 7f1954c00640 1 -- 172.21.15.134:0/2991858563 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1940031b60 con 0x7f1958076db0 2024-07-26T11:23:58.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.113+0000 7f1954c00640 1 -- 172.21.15.134:0/2991858563 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1940035030 con 0x7f1958076db0 2024-07-26T11:23:58.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.113+0000 7f195d200640 1 -- 172.21.15.134:0/2991858563 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19581a6530 con 0x7f1958076db0 2024-07-26T11:23:58.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.114+0000 7f195d200640 1 -- 172.21.15.134:0/2991858563 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19581a6990 con 0x7f1958076db0 2024-07-26T11:23:58.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.120+0000 7f1954c00640 1 -- 172.21.15.134:0/2991858563 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f1940042070 con 0x7f1958076db0 2024-07-26T11:23:58.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.121+0000 7f195d200640 1 -- 172.21.15.134:0/2991858563 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1958075c10 con 0x7f1958076db0 2024-07-26T11:23:58.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.124+0000 7f1954c00640 1 --2- 172.21.15.134:0/2991858563 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f193c0b3f10 0x7f193c0b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:23:58.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.125+0000 7f1954c00640 1 -- 172.21.15.134:0/2991858563 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f19401059f0 con 0x7f1958076db0 2024-07-26T11:23:58.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.126+0000 7f1957400640 1 --2- 172.21.15.134:0/2991858563 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f193c0b3f10 0x7f193c0b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:23:58.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.127+0000 7f1957400640 1 --2- 172.21.15.134:0/2991858563 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f193c0b3f10 0x7f193c0b6420 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1948009b60 tx=0x7f1948005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:23:58.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.132+0000 7f1954c00640 1 -- 172.21.15.134:0/2991858563 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7f1940049030 con 0x7f1958076db0 2024-07-26T11:23:58.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.653+0000 7f195d200640 1 -- 172.21.15.134:0/2991858563 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19581a1460 con 0x7f1958076db0 2024-07-26T11:23:58.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.655+0000 7f1954c00640 1 -- 172.21.15.134:0/2991858563 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11990 (secure 0 0 0) 0x7f19400cc8c0 con 0x7f1958076db0 2024-07-26T11:23:58.661 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:23:58.661 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":32,"btime":"2024-07-26T11:19:25:331870+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:19:25.131276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24721,"mds_1":24719,"mds_2":24703,"mds_3":24688,"mds_4":24686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24635":{"gid":24635,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24641":{"gid":24641,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":32,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24719":{"gid":24719,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":28,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24721,"qdb_cluster":[24721,24719,24703,24688,24686]},"id":1}]} 2024-07-26T11:23:58.661 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 32 2024-07-26T11:23:58.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.660+0000 7f195d200640 1 -- 172.21.15.134:0/2991858563 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f193c0b3f10 msgr2=0x7f193c0b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:58.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.660+0000 7f195d200640 1 --2- 172.21.15.134:0/2991858563 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f193c0b3f10 0x7f193c0b6420 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1948009b60 tx=0x7f1948005f90 comp rx=0 tx=0).stop 2024-07-26T11:23:58.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.660+0000 7f195d200640 1 -- 172.21.15.134:0/2991858563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1958076db0 msgr2=0x7f19581a08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:23:58.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.660+0000 7f195d200640 1 --2- 172.21.15.134:0/2991858563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1958076db0 0x7f19581a08d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f194002a920 tx=0x7f1940003c10 comp rx=0 tx=0).stop 2024-07-26T11:23:58.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.661+0000 7f195d200640 1 -- 172.21.15.134:0/2991858563 shutdown_connections 2024-07-26T11:23:58.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.661+0000 7f195d200640 1 --2- 172.21.15.134:0/2991858563 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f193c0b3f10 0x7f193c0b6420 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:58.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.661+0000 7f195d200640 1 --2- 172.21.15.134:0/2991858563 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1958113cb0 0x7f19581a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:58.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.661+0000 7f195d200640 1 --2- 172.21.15.134:0/2991858563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1958076db0 0x7f19581a08d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:58.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.661+0000 7f195d200640 1 --2- 172.21.15.134:0/2991858563 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f1958075800 0x7f19581a0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:23:58.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.661+0000 7f195d200640 1 -- 172.21.15.134:0/2991858563 >> 172.21.15.134:0/2991858563 conn(0x7f19580fee30 msgr2=0x7f195810fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:23:58.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.662+0000 7f195d200640 1 -- 172.21.15.134:0/2991858563 shutdown_connections 2024-07-26T11:23:58.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:23:58.664+0000 7f195d200640 1 -- 172.21.15.134:0/2991858563 wait complete. 2024-07-26T11:23:59.755 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:59 smithi134 ceph-mon[31966]: pgmap v722: 129 pgs: 129 active+clean; 61 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 52 KiB/s rd, 265 MiB/s wr, 1.30k op/s 2024-07-26T11:23:59.755 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:23:59 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/2991858563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:23:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:59 smithi189 ceph-mon[36790]: pgmap v722: 129 pgs: 129 active+clean; 61 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 52 KiB/s rd, 265 MiB/s wr, 1.30k op/s 2024-07-26T11:23:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:23:59 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/2991858563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:24:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:59 smithi193 ceph-mon[37473]: pgmap v722: 129 pgs: 129 active+clean; 61 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 52 KiB/s rd, 265 MiB/s wr, 1.30k op/s 2024-07-26T11:24:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:23:59 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/2991858563' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:24:00.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-07-26T11:24:01.924 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:24:01.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:01 smithi189 ceph-mon[36790]: pgmap v723: 129 pgs: 129 active+clean; 61 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 38 KiB/s rd, 284 MiB/s wr, 1.40k op/s 2024-07-26T11:24:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:01 smithi193 ceph-mon[37473]: pgmap v723: 129 pgs: 129 active+clean; 61 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 38 KiB/s rd, 284 MiB/s wr, 1.40k op/s 2024-07-26T11:24:02.121 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:01 smithi134 ceph-mon[31966]: pgmap v723: 129 pgs: 129 active+clean; 61 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 38 KiB/s rd, 284 MiB/s wr, 1.40k op/s 2024-07-26T11:24:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:02 smithi193 ceph-mon[37473]: pgmap v724: 129 pgs: 129 active+clean; 62 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 56 KiB/s rd, 226 MiB/s wr, 1.15k op/s 2024-07-26T11:24:03.112 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:02 smithi134 ceph-mon[31966]: pgmap v724: 129 pgs: 129 active+clean; 62 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 56 KiB/s rd, 226 MiB/s wr, 1.15k op/s 2024-07-26T11:24:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:02 smithi189 ceph-mon[36790]: pgmap v724: 129 pgs: 129 active+clean; 62 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 56 KiB/s rd, 226 MiB/s wr, 1.15k op/s 2024-07-26T11:24:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:05 smithi134 ceph-mon[31966]: pgmap v725: 129 pgs: 129 active+clean; 62 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 49 KiB/s rd, 255 MiB/s wr, 1.34k op/s 2024-07-26T11:24:05.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:05 smithi189 ceph-mon[36790]: pgmap v725: 129 pgs: 129 active+clean; 62 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 49 KiB/s rd, 255 MiB/s wr, 1.34k op/s 2024-07-26T11:24:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:05 smithi193 ceph-mon[37473]: pgmap v725: 129 pgs: 129 active+clean; 62 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 49 KiB/s rd, 255 MiB/s wr, 1.34k op/s 2024-07-26T11:24:07.932 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:07 smithi134 ceph-mon[31966]: pgmap v726: 129 pgs: 129 active+clean; 62 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 49 KiB/s rd, 278 MiB/s wr, 1.49k op/s 2024-07-26T11:24:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:07 smithi189 ceph-mon[36790]: pgmap v726: 129 pgs: 129 active+clean; 62 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 49 KiB/s rd, 278 MiB/s wr, 1.49k op/s 2024-07-26T11:24:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:07 smithi193 ceph-mon[37473]: pgmap v726: 129 pgs: 129 active+clean; 62 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 49 KiB/s rd, 278 MiB/s wr, 1.49k op/s 2024-07-26T11:24:08.270 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.266+0000 7fa5d7200640 1 -- 172.21.15.134:0/3511221790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d8113ef0 msgr2=0x7fa5d8116310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:08.270 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.266+0000 7fa5d7200640 1 --2- 172.21.15.134:0/3511221790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d8113ef0 0x7fa5d8116310 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0009be0 tx=0x7fa5d002a6e0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.273 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.268+0000 7fa5d7200640 1 -- 172.21.15.134:0/3511221790 shutdown_connections 2024-07-26T11:24:08.273 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.271+0000 7fa5d7200640 1 --2- 172.21.15.134:0/3511221790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5d8077cc0 0x7fa5d81139b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.273 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.271+0000 7fa5d7200640 1 --2- 172.21.15.134:0/3511221790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d8113ef0 0x7fa5d8116310 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.273 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.271+0000 7fa5d7200640 1 --2- 172.21.15.134:0/3511221790 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5d8077370 0x7fa5d8077780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.273 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.271+0000 7fa5d7200640 1 -- 172.21.15.134:0/3511221790 >> 172.21.15.134:0/3511221790 conn(0x7fa5d8100f20 msgr2=0x7fa5d8103370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:08.274 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.272+0000 7fa5d7200640 1 -- 172.21.15.134:0/3511221790 shutdown_connections 2024-07-26T11:24:08.274 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.272+0000 7fa5d7200640 1 -- 172.21.15.134:0/3511221790 wait complete. 2024-07-26T11:24:08.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.274+0000 7fa5d7200640 1 Processor -- start 2024-07-26T11:24:08.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.275+0000 7fa5d7200640 1 -- start start 2024-07-26T11:24:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.278+0000 7fa5d7200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5d8077370 0x7fa5d8113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.278+0000 7fa5d7200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5d8077cc0 0x7fa5d810ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.278+0000 7fa5d7200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d8113ef0 0x7fa5d810cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.278+0000 7fa5d7200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d81198d0 con 0x7fa5d8077cc0 2024-07-26T11:24:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.278+0000 7fa5d7200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d8119750 con 0x7fa5d8113ef0 2024-07-26T11:24:08.281 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.278+0000 7fa5d7200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d8119a50 con 0x7fa5d8077370 2024-07-26T11:24:08.283 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.281+0000 7fa5d5e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5d8077370 0x7fa5d8113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.283 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.281+0000 7fa5d5e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5d8077370 0x7fa5d8113780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52040/0 (socket says 172.21.15.134:52040) 2024-07-26T11:24:08.283 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.281+0000 7fa5d5e00640 1 -- 172.21.15.134:0/3970314490 learned_addr learned my addr 172.21.15.134:0/3970314490 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:08.283 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.281+0000 7fa5d6800640 1 --2- 172.21.15.134:0/3970314490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d8113ef0 0x7fa5d810cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.284 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.282+0000 7fa5d5e00640 1 -- 172.21.15.134:0/3970314490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d8113ef0 msgr2=0x7fa5d810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.282+0000 7fa5d5e00640 1 --2- 172.21.15.134:0/3970314490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d8113ef0 0x7fa5d810cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.282+0000 7fa5d5e00640 1 -- 172.21.15.134:0/3970314490 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5d8077cc0 msgr2=0x7fa5d810ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.282+0000 7fa5d5e00640 1 --2- 172.21.15.134:0/3970314490 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5d8077cc0 0x7fa5d810ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.282+0000 7fa5d5e00640 1 -- 172.21.15.134:0/3970314490 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d810d800 con 0x7fa5d8077370 2024-07-26T11:24:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.282+0000 7fa5d5e00640 1 --2- 172.21.15.134:0/3970314490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5d8077370 0x7fa5d8113780 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800b4a0 tx=0x7fa5c800b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.283+0000 7fa5cea00640 1 -- 172.21.15.134:0/3970314490 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8014070 con 0x7fa5d8077370 2024-07-26T11:24:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.283+0000 7fa5cea00640 1 -- 172.21.15.134:0/3970314490 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5c8010ac0 con 0x7fa5d8077370 2024-07-26T11:24:08.285 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.283+0000 7fa5cea00640 1 -- 172.21.15.134:0/3970314490 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8017c30 con 0x7fa5d8077370 2024-07-26T11:24:08.286 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.283+0000 7fa5d7200640 1 -- 172.21.15.134:0/3970314490 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d81124a0 con 0x7fa5d8077370 2024-07-26T11:24:08.286 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.283+0000 7fa5d7200640 1 -- 172.21.15.134:0/3970314490 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d8112a20 con 0x7fa5d8077370 2024-07-26T11:24:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.300+0000 7fa5e37fd740 1 Processor -- start 2024-07-26T11:24:08.302 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.300+0000 7fa5e37fd740 1 -- start start 2024-07-26T11:24:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.301+0000 7fa5e37fd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0481b0 0x55d8df3ebc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.301+0000 7fa5e37fd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 0x55d8df3ec160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.303 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.301+0000 7fa5e37fd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df0274d0 0x55d8df3ec6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.301+0000 7fa5e37fd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d8df360270 con 0x55d8df01f2f0 2024-07-26T11:24:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.301+0000 7fa5e37fd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d8df34f770 con 0x55d8df0274d0 2024-07-26T11:24:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.301+0000 7fa5e37fd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d8df3537b0 con 0x55d8df0481b0 2024-07-26T11:24:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.302+0000 7fa5d6800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0481b0 0x55d8df3ebc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.302+0000 7fa5d6800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0481b0 0x55d8df3ebc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52056/0 (socket says 172.21.15.134:52056) 2024-07-26T11:24:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.302+0000 7fa5d6800640 1 -- 172.21.15.134:0/4277019215 learned_addr learned my addr 172.21.15.134:0/4277019215 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:08.306 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.303+0000 7fa5d5400640 1 --2- 172.21.15.134:0/4277019215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df0274d0 0x55d8df3ec6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.306 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.304+0000 7fa5cea00640 1 -- 172.21.15.134:0/3970314490 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fa5c8017070 con 0x7fa5d8077370 2024-07-26T11:24:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.307+0000 7fa5d5e00640 1 --2- 172.21.15.134:0/4277019215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 0x55d8df3ec160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.307+0000 7fa5d6800640 1 -- 172.21.15.134:0/4277019215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df0274d0 msgr2=0x55d8df3ec6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.307+0000 7fa5d6800640 1 --2- 172.21.15.134:0/4277019215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df0274d0 0x55d8df3ec6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.307+0000 7fa5d6800640 1 -- 172.21.15.134:0/4277019215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 msgr2=0x55d8df3ec160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.307+0000 7fa5d6800640 1 --2- 172.21.15.134:0/4277019215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 0x55d8df3ec160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.307+0000 7fa5d6800640 1 -- 172.21.15.134:0/4277019215 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8df0b3790 con 0x55d8df0481b0 2024-07-26T11:24:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.308+0000 7fa5d6800640 1 --2- 172.21.15.134:0/4277019215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0481b0 0x55d8df3ebc20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0033000 tx=0x7fa5d00305c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab82de910bbb4608 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.308+0000 7fa5c3000640 1 -- 172.21.15.134:0/4277019215 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0043070 con 0x55d8df0481b0 2024-07-26T11:24:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.308+0000 7fa5c3000640 1 -- 172.21.15.134:0/4277019215 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5d0003e20 con 0x55d8df0481b0 2024-07-26T11:24:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.308+0000 7fa5c3000640 1 -- 172.21.15.134:0/4277019215 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0036040 con 0x55d8df0481b0 2024-07-26T11:24:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.310+0000 7fa5e37fd740 1 -- 172.21.15.134:0/4277019215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0481b0 msgr2=0x55d8df3ebc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.310+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/4277019215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0481b0 0x55d8df3ebc20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0033000 tx=0x7fa5d00305c0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.314+0000 7fa5cea00640 1 --2- 172.21.15.134:0/3970314490 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5bc0b3f10 0x7fa5bc0b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.314+0000 7fa5cea00640 1 -- 172.21.15.134:0/3970314490 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7fa5c80e3a10 con 0x7fa5d8077370 2024-07-26T11:24:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.315+0000 7fa5e37fd740 1 -- 172.21.15.134:0/4277019215 shutdown_connections 2024-07-26T11:24:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.315+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/4277019215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df0274d0 0x55d8df3ec6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.315+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/4277019215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 0x55d8df3ec160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.315+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/4277019215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0481b0 0x55d8df3ebc20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.315+0000 7fa5e37fd740 1 -- 172.21.15.134:0/4277019215 >> 172.21.15.134:0/4277019215 conn(0x55d8df03c300 msgr2=0x55d8df3e9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.315+0000 7fa5e37fd740 1 -- 172.21.15.134:0/4277019215 shutdown_connections 2024-07-26T11:24:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.315+0000 7fa5e37fd740 1 -- 172.21.15.134:0/4277019215 wait complete. 2024-07-26T11:24:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.316+0000 7fa5d6800640 1 --2- 172.21.15.134:0/3970314490 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5bc0b3f10 0x7fa5bc0b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.316+0000 7fa5e37fd740 1 -- start start 2024-07-26T11:24:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.317+0000 7fa5e37fd740 1 Processor -- start 2024-07-26T11:24:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.318+0000 7fa5e37fd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 0x55d8df3f78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.319+0000 7fa5d6800640 1 --2- 172.21.15.134:0/3970314490 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5bc0b3f10 0x7fa5bc0b6420 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0031040 tx=0x7fa5d0005fd0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:08.322 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.320+0000 7fa5e37fd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df3f7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.322 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.320+0000 7fa5e37fd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df0481b0 0x55d8df403660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.320+0000 7fa5d5400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df3f7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.320+0000 7fa5d6800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df0481b0 0x55d8df403660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.320+0000 7fa5d5400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df3f7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52060/0 (socket says 172.21.15.134:52060) 2024-07-26T11:24:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.320+0000 7fa5d5400640 1 -- 172.21.15.134:0/1396601513 learned_addr learned my addr 172.21.15.134:0/1396601513 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.320+0000 7fa5e37fd740 1 -- 172.21.15.134:0/1396601513 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d8df289ab0 con 0x55d8df01f2f0 2024-07-26T11:24:08.325 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.320+0000 7fa5e37fd740 1 -- 172.21.15.134:0/1396601513 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d8df2ecd50 con 0x55d8df0481b0 2024-07-26T11:24:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.320+0000 7fa5e37fd740 1 -- 172.21.15.134:0/1396601513 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d8df3fa340 con 0x55d8df0274d0 2024-07-26T11:24:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.321+0000 7fa5e37fd740 10 client.0 _ll_register_callbacks cb 0x55d8df05fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:24:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.321+0000 7fa5d5e00640 1 --2- 172.21.15.134:0/1396601513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 0x55d8df3f78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.321+0000 7fa5d5400640 1 -- 172.21.15.134:0/1396601513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df0481b0 msgr2=0x55d8df403660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.323+0000 7fa5d5400640 1 --2- 172.21.15.134:0/1396601513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df0481b0 0x55d8df403660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.323+0000 7fa5d5400640 1 -- 172.21.15.134:0/1396601513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 msgr2=0x55d8df3f78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.323+0000 7fa5d5400640 1 --2- 172.21.15.134:0/1396601513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 0x55d8df3f78b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.326 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.323+0000 7fa5d5400640 1 -- 172.21.15.134:0/1396601513 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8df12c620 con 0x55d8df0274d0 2024-07-26T11:24:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.324+0000 7fa5d5e00640 1 --2- 172.21.15.134:0/1396601513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 0x55d8df3f78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:24:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.324+0000 7fa5d5400640 1 --2- 172.21.15.134:0/1396601513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df3f7e90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa5c40116f0 tx=0x7fa5c4011bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2607450eb69d693 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.324+0000 7fa5b7400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:24:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.325+0000 7fa5b7400640 1 -- 172.21.15.134:0/1396601513 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c401e060 con 0x55d8df0274d0 2024-07-26T11:24:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.325+0000 7fa5e37fd740 1 -- 172.21.15.134:0/1396601513 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5d0003e20 con 0x55d8df0274d0 2024-07-26T11:24:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.325+0000 7fa5b7400640 1 -- 172.21.15.134:0/1396601513 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7fa5c400fb70 con 0x55d8df0274d0 2024-07-26T11:24:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.325+0000 7fa5b7400640 1 -- 172.21.15.134:0/1396601513 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa5c401e060 con 0x55d8df0274d0 2024-07-26T11:24:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.325+0000 7fa5e37fd740 10 client.26638 fetch_fsmap learned FSMap version 32 2024-07-26T11:24:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.325+0000 7fa5e37fd740 1 -- 172.21.15.134:0/1396601513 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d8df4064d0 con 0x55d8df0274d0 2024-07-26T11:24:08.330 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.326+0000 7fa5b7400640 1 -- 172.21.15.134:0/1396601513 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5d0003e20 con 0x55d8df0274d0 2024-07-26T11:24:08.331 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.329+0000 7fa5b7400640 1 -- 172.21.15.134:0/1396601513 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fa5c4011ce0 con 0x55d8df0274d0 2024-07-26T11:24:08.331 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.329+0000 7fa5e37fd740 10 client.26638 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:24:08.331 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.329+0000 7fa5e37fd740 10 client.26638 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:24:08.331 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.329+0000 7fa5e37fd740 20 client.26638 populate_metadata read hostname 'smithi134' 2024-07-26T11:24:08.331 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.329+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/1396601513 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d8df006d80 0x55d8df409a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.329+0000 7fa5e37fd740 4 client.26638 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:24:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.329+0000 7fa5e37fd740 1 -- 172.21.15.134:0/1396601513 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8df3621c0 con 0x55d8df006d80 2024-07-26T11:24:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.331+0000 7fa5d5e00640 1 --2- 172.21.15.134:0/1396601513 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d8df006d80 0x55d8df409a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.332+0000 7fa5b7400640 10 client.26638 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:24:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.332+0000 7fa5d5e00640 1 --2- 172.21.15.134:0/1396601513 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d8df006d80 0x55d8df409a30 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7f1670a3384f268 server_cookie=d39e7b874509753a in_seq=0 out_seq=0 2024-07-26T11:24:08.344 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.338+0000 7fa5b7400640 1 -- 172.21.15.134:0/1396601513 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d8df3621c0 con 0x55d8df006d80 2024-07-26T11:24:08.344 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.339+0000 7fa5b7400640 10 client.26638 handle_command_reply: tid=0 2024-07-26T11:24:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.343+0000 7fa5e37fd740 1 client.26638 shutdown 2024-07-26T11:24:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.346+0000 7fa5e37fd740 1 -- 172.21.15.134:0/1396601513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 msgr2=0x55d8df3f7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.346+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/1396601513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df3f7e90 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa5c40116f0 tx=0x7fa5c4011bb0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.347+0000 7fa5e37fd740 1 -- 172.21.15.134:0/1396601513 shutdown_connections 2024-07-26T11:24:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.347+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/1396601513 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d8df006d80 0x55d8df409a30 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.347+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/1396601513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df3f7e90 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.347+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/1396601513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df0481b0 0x55d8df403660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.347+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/1396601513 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 0x55d8df3f78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.347+0000 7fa5e37fd740 1 -- 172.21.15.134:0/1396601513 >> 172.21.15.134:0/1396601513 conn(0x55d8df03c300 msgr2=0x55d8df3ecc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.348+0000 7fa5b7400640 0 client.26638 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:24:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.351+0000 7fa5e37fd740 1 -- 172.21.15.134:0/1396601513 shutdown_connections 2024-07-26T11:24:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.353+0000 7fa5e37fd740 1 -- 172.21.15.134:0/1396601513 wait complete. 2024-07-26T11:24:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.353+0000 7fa5e37fd740 20 client.26638 trim_cache size 0 max 16384 2024-07-26T11:24:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.402+0000 7fa5e37fd740 1 Processor -- start 2024-07-26T11:24:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.402+0000 7fa5e37fd740 1 -- start start 2024-07-26T11:24:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.403+0000 7fa5e37fd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df01f2f0 0x55d8df3f1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.403+0000 7fa5e37fd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df403450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.403+0000 7fa5e37fd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df0481b0 0x55d8df3f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.403+0000 7fa5e37fd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d81bf6e0 con 0x55d8df0481b0 2024-07-26T11:24:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.403+0000 7fa5e37fd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d81c1220 con 0x55d8df01f2f0 2024-07-26T11:24:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.403+0000 7fa5e37fd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d81bfdb0 con 0x55d8df0274d0 2024-07-26T11:24:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.403+0000 7fa5d5400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df403450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.404+0000 7fa5d5400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df403450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:52062/0 (socket says 172.21.15.134:52062) 2024-07-26T11:24:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.404+0000 7fa5d5400640 1 -- 172.21.15.134:0/4169597041 learned_addr learned my addr 172.21.15.134:0/4169597041 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.405+0000 7fa5d5400640 1 -- 172.21.15.134:0/4169597041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df01f2f0 msgr2=0x55d8df3f1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.407+0000 7fa5d6800640 1 --2- 172.21.15.134:0/4169597041 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df0481b0 0x55d8df3f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.404+0000 7fa5d5e00640 1 --2- 172.21.15.134:0/4169597041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df01f2f0 0x55d8df3f1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.408+0000 7fa5d5400640 1 --2- 172.21.15.134:0/4169597041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df01f2f0 0x55d8df3f1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.408+0000 7fa5d5400640 1 -- 172.21.15.134:0/4169597041 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df0481b0 msgr2=0x55d8df3f6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.408+0000 7fa5d5400640 1 --2- 172.21.15.134:0/4169597041 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df0481b0 0x55d8df3f6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.408+0000 7fa5d5400640 1 -- 172.21.15.134:0/4169597041 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8df3fa280 con 0x55d8df0274d0 2024-07-26T11:24:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.408+0000 7fa5d6800640 1 --2- 172.21.15.134:0/4169597041 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df0481b0 0x55d8df3f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:24:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.408+0000 7fa5d5e00640 1 --2- 172.21.15.134:0/4169597041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df01f2f0 0x55d8df3f1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:24:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.408+0000 7fa5d5400640 1 --2- 172.21.15.134:0/4169597041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df403450 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa5c400f810 tx=0x7fa5c4009c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76eb65f9553be8d9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.410+0000 7fa5c2600640 1 -- 172.21.15.134:0/4169597041 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c400fd70 con 0x55d8df0274d0 2024-07-26T11:24:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.410+0000 7fa5c2600640 1 -- 172.21.15.134:0/4169597041 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5c4022e50 con 0x55d8df0274d0 2024-07-26T11:24:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.410+0000 7fa5c2600640 1 -- 172.21.15.134:0/4169597041 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c40043c0 con 0x55d8df0274d0 2024-07-26T11:24:08.413 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.411+0000 7fa5e37fd740 1 -- 172.21.15.134:0/4169597041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 msgr2=0x55d8df403450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:08.413 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.411+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/4169597041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df403450 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa5c400f810 tx=0x7fa5c4009c60 comp rx=0 tx=0).stop 2024-07-26T11:24:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.412+0000 7fa5e37fd740 1 -- 172.21.15.134:0/4169597041 shutdown_connections 2024-07-26T11:24:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.412+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/4169597041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df403450 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.412+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/4169597041 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df0481b0 0x55d8df3f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.412+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/4169597041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df01f2f0 0x55d8df3f1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.412+0000 7fa5e37fd740 1 -- 172.21.15.134:0/4169597041 >> 172.21.15.134:0/4169597041 conn(0x55d8df03c300 msgr2=0x55d8df3f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.413+0000 7fa5e37fd740 1 -- 172.21.15.134:0/4169597041 shutdown_connections 2024-07-26T11:24:08.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.413+0000 7fa5e37fd740 1 -- 172.21.15.134:0/4169597041 wait complete. 2024-07-26T11:24:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.415+0000 7fa5e37fd740 1 -- start start 2024-07-26T11:24:08.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.415+0000 7fa5e37fd740 1 Processor -- start 2024-07-26T11:24:08.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.417+0000 7fa5e37fd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 0x55d8df3fadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.419+0000 7fa5e37fd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df3fb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.419+0000 7fa5e37fd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df0481b0 0x55d8df4701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.417+0000 7fa5d5e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 0x55d8df3fadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.420+0000 7fa5d5e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 0x55d8df3fadc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:54024/0 (socket says 172.21.15.134:54024) 2024-07-26T11:24:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.420+0000 7fa5e37fd740 1 -- 172.21.15.134:0/545852416 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d81c0b50 con 0x55d8df01f2f0 2024-07-26T11:24:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.420+0000 7fa5e37fd740 1 -- 172.21.15.134:0/545852416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d81c0480 con 0x55d8df0481b0 2024-07-26T11:24:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.420+0000 7fa5e37fd740 1 -- 172.21.15.134:0/545852416 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d81bd680 con 0x55d8df0274d0 2024-07-26T11:24:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.420+0000 7fa5d5e00640 1 -- 172.21.15.134:0/545852416 learned_addr learned my addr 172.21.15.134:0/545852416 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.420+0000 7fa5d5400640 1 --2- 172.21.15.134:0/545852416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df3fb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.420+0000 7fa5d6800640 1 --2- 172.21.15.134:0/545852416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df0481b0 0x55d8df4701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.420+0000 7fa5e37fd740 10 client.0 _ll_register_callbacks cb 0x7fa5c4009760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:24:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.421+0000 7fa5d5e00640 1 -- 172.21.15.134:0/545852416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 msgr2=0x55d8df3fb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.421+0000 7fa5d5e00640 1 --2- 172.21.15.134:0/545852416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df3fb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.421+0000 7fa5d5e00640 1 -- 172.21.15.134:0/545852416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df0481b0 msgr2=0x55d8df4701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:08.424 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.421+0000 7fa5d5e00640 1 --2- 172.21.15.134:0/545852416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df0481b0 0x55d8df4701f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:08.426 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.424+0000 7fa5d5e00640 1 -- 172.21.15.134:0/545852416 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8df470830 con 0x55d8df01f2f0 2024-07-26T11:24:08.426 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.424+0000 7fa5d5400640 1 --2- 172.21.15.134:0/545852416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df3fb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:24:08.426 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.424+0000 7fa5e37fd740 1 -- 172.21.15.134:0/545852416 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5c4022e50 con 0x55d8df01f2f0 2024-07-26T11:24:08.427 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.425+0000 7fa5b7400640 10 client.16080 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:24:08.427 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.425+0000 7fa5d5e00640 1 --2- 172.21.15.134:0/545852416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 0x55d8df3fadc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80aae60 tx=0x7fa5c8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eef8848fbfaf873d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:08.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.427+0000 7fa5b7400640 1 -- 172.21.15.134:0/545852416 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c801e040 con 0x55d8df01f2f0 2024-07-26T11:24:08.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.427+0000 7fa5b7400640 1 -- 172.21.15.134:0/545852416 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7fa5c80e1090 con 0x55d8df01f2f0 2024-07-26T11:24:08.430 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.427+0000 7fa5b7400640 1 -- 172.21.15.134:0/545852416 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa5c80af760 con 0x55d8df01f2f0 2024-07-26T11:24:08.430 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.427+0000 7fa5e37fd740 10 client.16080 fetch_fsmap learned FSMap version 32 2024-07-26T11:24:08.430 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.428+0000 7fa5e37fd740 1 -- 172.21.15.134:0/545852416 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({fsmap=32}) -- 0x55d8df471920 con 0x55d8df01f2f0 2024-07-26T11:24:08.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.430+0000 7fa5b7400640 1 -- 172.21.15.134:0/545852416 <== mon.0 v2:172.21.15.134:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fa5c80a73f0 con 0x55d8df01f2f0 2024-07-26T11:24:08.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.430+0000 7fa5e37fd740 10 client.16080 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:24:08.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.430+0000 7fa5e37fd740 10 client.16080 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:24:08.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.430+0000 7fa5e37fd740 20 client.16080 populate_metadata read hostname 'smithi134' 2024-07-26T11:24:08.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.430+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/545852416 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d8defdc1c0 0x55d8df474070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:08.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.430+0000 7fa5e37fd740 4 client.16080 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:24:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.431+0000 7fa5e37fd740 1 -- 172.21.15.134:0/545852416 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d8df35dc60 con 0x55d8defdc1c0 2024-07-26T11:24:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.431+0000 7fa5d5400640 1 --2- 172.21.15.134:0/545852416 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d8defdc1c0 0x55d8df474070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.433+0000 7fa5b7400640 10 client.16080 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:24:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:08.433+0000 7fa5d5400640 1 --2- 172.21.15.134:0/545852416 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d8defdc1c0 0x55d8df474070 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=770df4380e03028d server_cookie=3daf2719f5a45e16 in_seq=0 out_seq=0 2024-07-26T11:24:09.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.096+0000 7fa5b7400640 1 -- 172.21.15.134:0/545852416 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d8df35dc60 con 0x55d8defdc1c0 2024-07-26T11:24:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.097+0000 7fa5b7400640 10 client.16080 handle_command_reply: tid=0 2024-07-26T11:24:09.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.097+0000 7fa5e37fd740 1 client.16080 shutdown 2024-07-26T11:24:09.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.098+0000 7fa5e37fd740 1 -- 172.21.15.134:0/545852416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 msgr2=0x55d8df3fadc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:09.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.098+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/545852416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 0x55d8df3fadc0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa5c80aae60 tx=0x7fa5c8000bc0 comp rx=0 tx=0).stop 2024-07-26T11:24:09.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.098+0000 7fa5e37fd740 1 -- 172.21.15.134:0/545852416 shutdown_connections 2024-07-26T11:24:09.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.098+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/545852416 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d8defdc1c0 0x55d8df474070 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:09.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.098+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/545852416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8df0274d0 0x55d8df3fb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:09.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.098+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/545852416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8df0481b0 0x55d8df4701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:09.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.098+0000 7fa5e37fd740 1 --2- 172.21.15.134:0/545852416 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8df01f2f0 0x55d8df3fadc0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:09.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.098+0000 7fa5e37fd740 1 -- 172.21.15.134:0/545852416 >> 172.21.15.134:0/545852416 conn(0x55d8df03c300 msgr2=0x55d8df3f6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:09.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.098+0000 7fa5b7400640 0 client.16080 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:24:09.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.099+0000 7fa5e37fd740 1 -- 172.21.15.134:0/545852416 shutdown_connections 2024-07-26T11:24:09.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.099+0000 7fa5e37fd740 1 -- 172.21.15.134:0/545852416 wait complete. 2024-07-26T11:24:09.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.100+0000 7fa5e37fd740 20 client.16080 trim_cache size 0 max 16384 2024-07-26T11:24:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:24:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:24:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:24:09.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:24:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.103+0000 7fa5d7200640 1 -- 172.21.15.134:0/3970314490 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5bc0b3f10 msgr2=0x7fa5bc0b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.103+0000 7fa5d7200640 1 --2- 172.21.15.134:0/3970314490 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5bc0b3f10 0x7fa5bc0b6420 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0031040 tx=0x7fa5d0005fd0 comp rx=0 tx=0).stop 2024-07-26T11:24:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.103+0000 7fa5d7200640 1 -- 172.21.15.134:0/3970314490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5d8077370 msgr2=0x7fa5d8113780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.103+0000 7fa5d7200640 1 --2- 172.21.15.134:0/3970314490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5d8077370 0x7fa5d8113780 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800b4a0 tx=0x7fa5c800b960 comp rx=0 tx=0).stop 2024-07-26T11:24:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.103+0000 7fa5d7200640 1 -- 172.21.15.134:0/3970314490 shutdown_connections 2024-07-26T11:24:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.103+0000 7fa5d7200640 1 --2- 172.21.15.134:0/3970314490 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5bc0b3f10 0x7fa5bc0b6420 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.103+0000 7fa5d7200640 1 --2- 172.21.15.134:0/3970314490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d8113ef0 0x7fa5d810cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.103+0000 7fa5d7200640 1 --2- 172.21.15.134:0/3970314490 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa5d8077cc0 0x7fa5d810ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.103+0000 7fa5d7200640 1 --2- 172.21.15.134:0/3970314490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5d8077370 0x7fa5d8113780 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.103+0000 7fa5d7200640 1 -- 172.21.15.134:0/3970314490 >> 172.21.15.134:0/3970314490 conn(0x7fa5d8100f20 msgr2=0x7fa5d8103340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:09.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.103+0000 7fa5d7200640 1 -- 172.21.15.134:0/3970314490 shutdown_connections 2024-07-26T11:24:09.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:09.103+0000 7fa5d7200640 1 -- 172.21.15.134:0/3970314490 wait complete. 2024-07-26T11:24:09.727 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:09 smithi134 ceph-mon[31966]: pgmap v727: 129 pgs: 129 active+clean; 62 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 60 KiB/s rd, 254 MiB/s wr, 1.42k op/s 2024-07-26T11:24:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:09 smithi189 ceph-mon[36790]: pgmap v727: 129 pgs: 129 active+clean; 62 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 60 KiB/s rd, 254 MiB/s wr, 1.42k op/s 2024-07-26T11:24:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:09 smithi193 ceph-mon[37473]: pgmap v727: 129 pgs: 129 active+clean; 62 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 60 KiB/s rd, 254 MiB/s wr, 1.42k op/s 2024-07-26T11:24:10.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.255+0000 7f417b400640 1 -- 172.21.15.134:0/1267282659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f417410a910 msgr2=0x7f4174115a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:10.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.255+0000 7f417b400640 1 --2- 172.21.15.134:0/1267282659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f417410a910 0x7f4174115a00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4164009900 tx=0x7f416402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.255+0000 7f417b400640 1 -- 172.21.15.134:0/1267282659 shutdown_connections 2024-07-26T11:24:10.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.255+0000 7f417b400640 1 --2- 172.21.15.134:0/1267282659 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f417410a910 0x7f4174115a00 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.255+0000 7f417b400640 1 --2- 172.21.15.134:0/1267282659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4174116070 0x7f4174118580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.255+0000 7f417b400640 1 --2- 172.21.15.134:0/1267282659 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4174109fc0 0x7f417410a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.255+0000 7f417b400640 1 -- 172.21.15.134:0/1267282659 >> 172.21.15.134:0/1267282659 conn(0x7f41740708f0 msgr2=0x7f4174070d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:10.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.256+0000 7f417b400640 1 -- 172.21.15.134:0/1267282659 shutdown_connections 2024-07-26T11:24:10.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.256+0000 7f417b400640 1 -- 172.21.15.134:0/1267282659 wait complete. 2024-07-26T11:24:10.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.257+0000 7f417b400640 1 Processor -- start 2024-07-26T11:24:10.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.258+0000 7f417b400640 1 -- start start 2024-07-26T11:24:10.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.258+0000 7f417b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4174109fc0 0x7f4174113f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.258+0000 7f417b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f417410a910 0x7f4174114460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.258+0000 7f417b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4174116070 0x7f41741149a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.258+0000 7f417b400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f4174119b30 con 0x7f4174109fc0 2024-07-26T11:24:10.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.258+0000 7f417b400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f41741199b0 con 0x7f4174116070 2024-07-26T11:24:10.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.258+0000 7f417b400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f4174119cb0 con 0x7f417410a910 2024-07-26T11:24:10.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.259+0000 7f4173e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f417410a910 0x7f4174114460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.259+0000 7f4173e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f417410a910 0x7f4174114460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:43408/0 (socket says 172.21.15.134:43408) 2024-07-26T11:24:10.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.259+0000 7f4173e00640 1 -- 172.21.15.134:0/127076633 learned_addr learned my addr 172.21.15.134:0/127076633 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:10.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.259+0000 7f4179400640 1 --2- 172.21.15.134:0/127076633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4174116070 0x7f41741149a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.260+0000 7f4173e00640 1 -- 172.21.15.134:0/127076633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4174116070 msgr2=0x7f41741149a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:10.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.260+0000 7f4173e00640 1 --2- 172.21.15.134:0/127076633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4174116070 0x7f41741149a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.260+0000 7f4173e00640 1 -- 172.21.15.134:0/127076633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4174109fc0 msgr2=0x7f4174113f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-07-26T11:24:10.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.262+0000 7f4178a00640 1 --2- 172.21.15.134:0/127076633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4174109fc0 0x7f4174113f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.264+0000 7f4173e00640 1 --2- 172.21.15.134:0/127076633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4174109fc0 0x7f4174113f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.264+0000 7f4173e00640 1 -- 172.21.15.134:0/127076633 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41741b4d50 con 0x7f417410a910 2024-07-26T11:24:10.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.264+0000 7f4178a00640 1 --2- 172.21.15.134:0/127076633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4174109fc0 0x7f4174113f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:24:10.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.264+0000 7f4173e00640 1 --2- 172.21.15.134:0/127076633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f417410a910 0x7f4174114460 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4164005ab0 tx=0x7f4164003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:10.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.267+0000 7f4171600640 1 -- 172.21.15.134:0/127076633 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416403f070 con 0x7f417410a910 2024-07-26T11:24:10.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.268+0000 7f4171600640 1 -- 172.21.15.134:0/127076633 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4164031b50 con 0x7f417410a910 2024-07-26T11:24:10.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.268+0000 7f4171600640 1 -- 172.21.15.134:0/127076633 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164035030 con 0x7f417410a910 2024-07-26T11:24:10.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.268+0000 7f417b400640 1 -- 172.21.15.134:0/127076633 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41741b4fe0 con 0x7f417410a910 2024-07-26T11:24:10.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.268+0000 7f417b400640 1 -- 172.21.15.134:0/127076633 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41741b5380 con 0x7f417410a910 2024-07-26T11:24:10.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.273+0000 7f4171600640 1 -- 172.21.15.134:0/127076633 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f4164042070 con 0x7f417410a910 2024-07-26T11:24:10.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.276+0000 7f4171600640 1 --2- 172.21.15.134:0/127076633 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f41580b3e40 0x7f41580b6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.276+0000 7f4171600640 1 -- 172.21.15.134:0/127076633 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f4164105730 con 0x7f417410a910 2024-07-26T11:24:10.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.277+0000 7f4178a00640 1 --2- 172.21.15.134:0/127076633 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f41580b3e40 0x7f41580b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.292+0000 7f4178a00640 1 --2- 172.21.15.134:0/127076633 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f41580b3e40 0x7f41580b6350 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4174115970 tx=0x7f416c009500 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:10.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.293+0000 7f4182056740 1 Processor -- start 2024-07-26T11:24:10.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.295+0000 7f4182056740 1 -- start start 2024-07-26T11:24:10.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.296+0000 7f4182056740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564863070 0x561564c07e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.296+0000 7f4182056740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 0x561564c08370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.296+0000 7f4182056740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564842390 0x561564c088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.296+0000 7f4182056740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x561564b7ebd0 con 0x561564842390 2024-07-26T11:24:10.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.296+0000 7f4182056740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561564b6e0d0 con 0x56156483a1b0 2024-07-26T11:24:10.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.296+0000 7f4182056740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561564b72110 con 0x561564863070 2024-07-26T11:24:10.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.296+0000 7f4173e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 0x561564c08370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.296+0000 7f4178a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564863070 0x561564c07e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.296+0000 7f4173e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 0x561564c08370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:53182/0 (socket says 172.21.15.134:53182) 2024-07-26T11:24:10.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.296+0000 7f4173e00640 1 -- 172.21.15.134:0/2840968762 learned_addr learned my addr 172.21.15.134:0/2840968762 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:10.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.297+0000 7f4179400640 1 --2- 172.21.15.134:0/2840968762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564842390 0x561564c088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.297+0000 7f4179400640 1 -- 172.21.15.134:0/2840968762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564863070 msgr2=0x561564c07e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:10.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.297+0000 7f4179400640 1 --2- 172.21.15.134:0/2840968762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564863070 0x561564c07e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.297+0000 7f4179400640 1 -- 172.21.15.134:0/2840968762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 msgr2=0x561564c08370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:10.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.297+0000 7f4179400640 1 --2- 172.21.15.134:0/2840968762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 0x561564c08370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.297+0000 7f4179400640 1 -- 172.21.15.134:0/2840968762 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615648ce650 con 0x561564842390 2024-07-26T11:24:10.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.298+0000 7f4179400640 1 --2- 172.21.15.134:0/2840968762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564842390 0x561564c088b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f415c004640 tx=0x7f415c00ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390925002a9806f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:10.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.298+0000 7f4169c00640 1 -- 172.21.15.134:0/2840968762 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c003bb0 con 0x561564842390 2024-07-26T11:24:10.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.298+0000 7f4169c00640 1 -- 172.21.15.134:0/2840968762 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f415c015070 con 0x561564842390 2024-07-26T11:24:10.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.298+0000 7f4169c00640 1 -- 172.21.15.134:0/2840968762 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415c0106a0 con 0x561564842390 2024-07-26T11:24:10.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.300+0000 7f4182056740 1 -- 172.21.15.134:0/2840968762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564842390 msgr2=0x561564c088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:10.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.300+0000 7f4182056740 1 --2- 172.21.15.134:0/2840968762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564842390 0x561564c088b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f415c004640 tx=0x7f415c00ba90 comp rx=0 tx=0).stop 2024-07-26T11:24:10.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.300+0000 7f4182056740 1 -- 172.21.15.134:0/2840968762 shutdown_connections 2024-07-26T11:24:10.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.300+0000 7f4182056740 1 --2- 172.21.15.134:0/2840968762 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564842390 0x561564c088b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.300+0000 7f4182056740 1 --2- 172.21.15.134:0/2840968762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 0x561564c08370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.300+0000 7f4182056740 1 --2- 172.21.15.134:0/2840968762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564863070 0x561564c07e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.300+0000 7f4182056740 1 -- 172.21.15.134:0/2840968762 >> 172.21.15.134:0/2840968762 conn(0x561564857460 msgr2=0x561564c05d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:10.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.301+0000 7f4182056740 1 -- 172.21.15.134:0/2840968762 shutdown_connections 2024-07-26T11:24:10.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.301+0000 7f4182056740 1 -- 172.21.15.134:0/2840968762 wait complete. 2024-07-26T11:24:10.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.302+0000 7f4182056740 1 -- start start 2024-07-26T11:24:10.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.302+0000 7f4182056740 1 Processor -- start 2024-07-26T11:24:10.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.302+0000 7f4182056740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 0x561564c13440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.302+0000 7f4182056740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c13980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.303+0000 7f4182056740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564863070 0x561564c13ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.303+0000 7f4182056740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5615649d1700 con 0x561564863070 2024-07-26T11:24:10.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.303+0000 7f4182056740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561564b0a6a0 con 0x56156483a1b0 2024-07-26T11:24:10.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.303+0000 7f4182056740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x561564c16610 con 0x561564842390 2024-07-26T11:24:10.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.303+0000 7f4182056740 10 client.0 _ll_register_callbacks cb 0x561564b1a150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:24:10.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.303+0000 7f4173e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c13980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.303+0000 7f4173e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c13980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:43436/0 (socket says 172.21.15.134:43436) 2024-07-26T11:24:10.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.303+0000 7f4173e00640 1 -- 172.21.15.134:0/3725053874 learned_addr learned my addr 172.21.15.134:0/3725053874 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:10.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.303+0000 7f4178a00640 1 --2- 172.21.15.134:0/3725053874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 0x561564c13440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.303+0000 7f4179400640 1 --2- 172.21.15.134:0/3725053874 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564863070 0x561564c13ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.303+0000 7f4173e00640 1 -- 172.21.15.134:0/3725053874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 msgr2=0x561564c13440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:10.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.303+0000 7f4173e00640 1 --2- 172.21.15.134:0/3725053874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 0x561564c13440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.303+0000 7f4173e00640 1 -- 172.21.15.134:0/3725053874 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564863070 msgr2=0x561564c13ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:10.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.303+0000 7f4173e00640 1 --2- 172.21.15.134:0/3725053874 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564863070 0x561564c13ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.303+0000 7f4173e00640 1 -- 172.21.15.134:0/3725053874 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561564b21840 con 0x561564842390 2024-07-26T11:24:10.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.303+0000 7f4179400640 1 --2- 172.21.15.134:0/3725053874 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564863070 0x561564c13ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:24:10.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.305+0000 7f4173e00640 1 --2- 172.21.15.134:0/3725053874 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c13980 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4164031440 tx=0x7f41640cb670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad0be4e19b924c60 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:10.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.305+0000 7f4162000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:24:10.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.305+0000 7f4162000640 1 -- 172.21.15.134:0/3725053874 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41640e6060 con 0x561564842390 2024-07-26T11:24:10.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.305+0000 7f4162000640 1 -- 172.21.15.134:0/3725053874 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f41640c8190 con 0x561564842390 2024-07-26T11:24:10.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.306+0000 7f4182056740 1 -- 172.21.15.134:0/3725053874 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f415c015070 con 0x561564842390 2024-07-26T11:24:10.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.306+0000 7f4162000640 1 -- 172.21.15.134:0/3725053874 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f41640eb050 con 0x561564842390 2024-07-26T11:24:10.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.306+0000 7f4182056740 10 client.26653 fetch_fsmap learned FSMap version 32 2024-07-26T11:24:10.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.306+0000 7f4182056740 1 -- 172.21.15.134:0/3725053874 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561564c22290 con 0x561564842390 2024-07-26T11:24:10.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.307+0000 7f4162000640 1 -- 172.21.15.134:0/3725053874 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f415c015070 con 0x561564842390 2024-07-26T11:24:10.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.307+0000 7f4162000640 1 -- 172.21.15.134:0/3725053874 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f416404c060 con 0x561564842390 2024-07-26T11:24:10.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.309+0000 7f4182056740 10 client.26653 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:24:10.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.309+0000 7f4182056740 10 client.26653 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-07-26T11:24:10.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.309+0000 7f4182056740 20 client.26653 populate_metadata read hostname 'smithi134' 2024-07-26T11:24:10.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.309+0000 7f4182056740 1 --2- 172.21.15.134:0/3725053874 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5615648219f0 0x561564c25760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.310+0000 7f4182056740 4 client.26653 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:24:10.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.310+0000 7f4182056740 1 -- 172.21.15.134:0/3725053874 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561564b80b20 con 0x5615648219f0 2024-07-26T11:24:10.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.310+0000 7f4178a00640 1 --2- 172.21.15.134:0/3725053874 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5615648219f0 0x561564c25760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.311+0000 7f4178a00640 1 --2- 172.21.15.134:0/3725053874 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5615648219f0 0x561564c25760 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=755999884578fb93 server_cookie=d1317711483dc6a1 in_seq=0 out_seq=0 2024-07-26T11:24:10.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.311+0000 7f4162000640 10 client.26653 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:24:10.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.312+0000 7f4162000640 1 -- 172.21.15.134:0/3725053874 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x561564b80b20 con 0x5615648219f0 2024-07-26T11:24:10.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.312+0000 7f4162000640 10 client.26653 handle_command_reply: tid=0 2024-07-26T11:24:10.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.312+0000 7f4182056740 1 client.26653 shutdown 2024-07-26T11:24:10.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.312+0000 7f4182056740 1 -- 172.21.15.134:0/3725053874 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 msgr2=0x561564c13980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:10.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.312+0000 7f4182056740 1 --2- 172.21.15.134:0/3725053874 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c13980 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4164031440 tx=0x7f41640cb670 comp rx=0 tx=0).stop 2024-07-26T11:24:10.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.312+0000 7f4182056740 1 -- 172.21.15.134:0/3725053874 shutdown_connections 2024-07-26T11:24:10.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.313+0000 7f4182056740 1 --2- 172.21.15.134:0/3725053874 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5615648219f0 0x561564c25760 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.313+0000 7f4182056740 1 --2- 172.21.15.134:0/3725053874 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c13980 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.313+0000 7f4182056740 1 --2- 172.21.15.134:0/3725053874 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564863070 0x561564c13ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.313+0000 7f4182056740 1 --2- 172.21.15.134:0/3725053874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 0x561564c13440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.313+0000 7f4182056740 1 -- 172.21.15.134:0/3725053874 >> 172.21.15.134:0/3725053874 conn(0x561564857460 msgr2=0x561564c0b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:10.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.313+0000 7f4162000640 0 client.26653 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:24:10.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.313+0000 7f4182056740 1 -- 172.21.15.134:0/3725053874 shutdown_connections 2024-07-26T11:24:10.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.313+0000 7f4182056740 1 -- 172.21.15.134:0/3725053874 wait complete. 2024-07-26T11:24:10.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.313+0000 7f4182056740 20 client.26653 trim_cache size 0 max 16384 2024-07-26T11:24:10.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.340+0000 7f4182056740 1 Processor -- start 2024-07-26T11:24:10.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.340+0000 7f4182056740 1 -- start start 2024-07-26T11:24:10.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.341+0000 7f4182056740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 0x561564c15430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.341+0000 7f4182056740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c2af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.341+0000 7f4182056740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564863070 0x561564c2b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.341+0000 7f4182056740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f41741bf850 con 0x561564863070 2024-07-26T11:24:10.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.341+0000 7f4182056740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f41741c1390 con 0x56156483a1b0 2024-07-26T11:24:10.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.341+0000 7f4182056740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f41741bff20 con 0x561564842390 2024-07-26T11:24:10.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.341+0000 7f4173e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c2af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.341+0000 7f4173e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c2af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:43448/0 (socket says 172.21.15.134:43448) 2024-07-26T11:24:10.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.341+0000 7f4173e00640 1 -- 172.21.15.134:0/3634479275 learned_addr learned my addr 172.21.15.134:0/3634479275 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:10.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.341+0000 7f4179400640 1 --2- 172.21.15.134:0/3634479275 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564863070 0x561564c2b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.341+0000 7f4178a00640 1 --2- 172.21.15.134:0/3634479275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 0x561564c15430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.342+0000 7f4173e00640 1 -- 172.21.15.134:0/3634479275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 msgr2=0x561564c15430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:10.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.342+0000 7f4173e00640 1 --2- 172.21.15.134:0/3634479275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 0x561564c15430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.342+0000 7f4173e00640 1 -- 172.21.15.134:0/3634479275 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564863070 msgr2=0x561564c2b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:10.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.342+0000 7f4173e00640 1 --2- 172.21.15.134:0/3634479275 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564863070 0x561564c2b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.342+0000 7f4173e00640 1 -- 172.21.15.134:0/3634479275 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561564c13cd0 con 0x561564842390 2024-07-26T11:24:10.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.342+0000 7f4178a00640 1 --2- 172.21.15.134:0/3634479275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 0x561564c15430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:24:10.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.342+0000 7f4179400640 1 --2- 172.21.15.134:0/3634479275 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564863070 0x561564c2b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:24:10.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.343+0000 7f4173e00640 1 --2- 172.21.15.134:0/3634479275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c2af20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f41640cbc00 tx=0x7f416402a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f7b20b2bb537f19 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:10.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.343+0000 7f4169200640 1 -- 172.21.15.134:0/3634479275 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41640e6070 con 0x561564842390 2024-07-26T11:24:10.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.343+0000 7f4169200640 1 -- 172.21.15.134:0/3634479275 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41640cbf00 con 0x561564842390 2024-07-26T11:24:10.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.343+0000 7f4169200640 1 -- 172.21.15.134:0/3634479275 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164049040 con 0x561564842390 2024-07-26T11:24:10.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.344+0000 7f4182056740 1 -- 172.21.15.134:0/3634479275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 msgr2=0x561564c2af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:10.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.344+0000 7f4182056740 1 --2- 172.21.15.134:0/3634479275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c2af20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f41640cbc00 tx=0x7f416402a3f0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.345+0000 7f4182056740 1 -- 172.21.15.134:0/3634479275 shutdown_connections 2024-07-26T11:24:10.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.345+0000 7f4182056740 1 --2- 172.21.15.134:0/3634479275 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c2af20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.345+0000 7f4182056740 1 --2- 172.21.15.134:0/3634479275 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x561564863070 0x561564c2b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.345+0000 7f4182056740 1 --2- 172.21.15.134:0/3634479275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56156483a1b0 0x561564c15430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.345+0000 7f4182056740 1 -- 172.21.15.134:0/3634479275 >> 172.21.15.134:0/3634479275 conn(0x561564857460 msgr2=0x561564c09820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:10.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.345+0000 7f4182056740 1 -- 172.21.15.134:0/3634479275 shutdown_connections 2024-07-26T11:24:10.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.345+0000 7f4182056740 1 -- 172.21.15.134:0/3634479275 wait complete. 2024-07-26T11:24:10.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.345+0000 7f4182056740 1 -- start start 2024-07-26T11:24:10.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.346+0000 7f4182056740 1 Processor -- start 2024-07-26T11:24:10.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.347+0000 7f4182056740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56156483a1b0 0x561564c17030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.347+0000 7f4182056740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c175f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.347+0000 7f4182056740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561564863070 0x561564c85eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.347+0000 7f4182056740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f41741c0cc0 con 0x56156483a1b0 2024-07-26T11:24:10.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.347+0000 7f4182056740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f41741c05f0 con 0x561564863070 2024-07-26T11:24:10.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.347+0000 7f4182056740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f41741bd850 con 0x561564842390 2024-07-26T11:24:10.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.347+0000 7f4182056740 10 client.0 _ll_register_callbacks cb 0x7f41640cbd80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:24:10.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.347+0000 7f4173e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c175f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.347+0000 7f4173e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c175f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:43460/0 (socket says 172.21.15.134:43460) 2024-07-26T11:24:10.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.347+0000 7f4173e00640 1 -- 172.21.15.134:0/613938154 learned_addr learned my addr 172.21.15.134:0/613938154 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:10.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.348+0000 7f4179400640 1 --2- 172.21.15.134:0/613938154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561564863070 0x561564c85eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.348+0000 7f4173e00640 1 -- 172.21.15.134:0/613938154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561564863070 msgr2=0x561564c85eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:10.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.348+0000 7f4178a00640 1 --2- 172.21.15.134:0/613938154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56156483a1b0 0x561564c17030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.348+0000 7f4173e00640 1 --2- 172.21.15.134:0/613938154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561564863070 0x561564c85eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.348+0000 7f4173e00640 1 -- 172.21.15.134:0/613938154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56156483a1b0 msgr2=0x561564c17030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:10.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.348+0000 7f4173e00640 1 --2- 172.21.15.134:0/613938154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56156483a1b0 0x561564c17030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.349+0000 7f4173e00640 1 -- 172.21.15.134:0/613938154 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561564c86520 con 0x561564842390 2024-07-26T11:24:10.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.349+0000 7f4178a00640 1 --2- 172.21.15.134:0/613938154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56156483a1b0 0x561564c17030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:24:10.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.349+0000 7f4173e00640 1 --2- 172.21.15.134:0/613938154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c175f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4164031440 tx=0x7f41640cb7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2a4e149e2f5765c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:10.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.349+0000 7f4162000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:24:10.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.349+0000 7f4162000640 1 -- 172.21.15.134:0/613938154 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164049070 con 0x561564842390 2024-07-26T11:24:10.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.350+0000 7f4182056740 1 -- 172.21.15.134:0/613938154 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41640cbf00 con 0x561564842390 2024-07-26T11:24:10.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.350+0000 7f4162000640 1 -- 172.21.15.134:0/613938154 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) ==== 9511+0+0 (secure 0 0 0) 0x7f41640c8190 con 0x561564842390 2024-07-26T11:24:10.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.351+0000 7f4162000640 1 -- 172.21.15.134:0/613938154 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f41640e6070 con 0x561564842390 2024-07-26T11:24:10.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.351+0000 7f4182056740 10 client.26659 fetch_fsmap learned FSMap version 32 2024-07-26T11:24:10.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.351+0000 7f4182056740 1 -- 172.21.15.134:0/613938154 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561564c86520 con 0x561564842390 2024-07-26T11:24:10.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.352+0000 7f4162000640 1 -- 172.21.15.134:0/613938154 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41640cbf00 con 0x561564842390 2024-07-26T11:24:10.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.352+0000 7f4162000640 1 -- 172.21.15.134:0/613938154 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f41640cb160 con 0x561564842390 2024-07-26T11:24:10.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.352+0000 7f4182056740 10 client.26659 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:24:10.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.352+0000 7f4182056740 10 client.26659 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-07-26T11:24:10.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.352+0000 7f4182056740 20 client.26659 populate_metadata read hostname 'smithi134' 2024-07-26T11:24:10.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.352+0000 7f4182056740 1 --2- 172.21.15.134:0/613938154 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5615647f6da0 0x561564c8fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:10.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.353+0000 7f4182056740 4 client.26659 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:24:10.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.353+0000 7f4182056740 1 -- 172.21.15.134:0/613938154 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561564b7c5c0 con 0x5615647f6da0 2024-07-26T11:24:10.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.353+0000 7f4178a00640 1 --2- 172.21.15.134:0/613938154 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5615647f6da0 0x561564c8fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:10.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.354+0000 7f4162000640 10 client.26659 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:24:10.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.354+0000 7f4178a00640 1 --2- 172.21.15.134:0/613938154 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5615647f6da0 0x561564c8fc40 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a7252287d16a0a4e server_cookie=4c01970848c014df in_seq=0 out_seq=0 2024-07-26T11:24:10.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.354+0000 7f4162000640 1 -- 172.21.15.134:0/613938154 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561564b7c5c0 con 0x5615647f6da0 2024-07-26T11:24:10.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.354+0000 7f4162000640 10 client.26659 handle_command_reply: tid=0 2024-07-26T11:24:10.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.355+0000 7f4182056740 1 client.26659 shutdown 2024-07-26T11:24:10.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.355+0000 7f4182056740 1 -- 172.21.15.134:0/613938154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 msgr2=0x561564c175f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:10.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.355+0000 7f4182056740 1 --2- 172.21.15.134:0/613938154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c175f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4164031440 tx=0x7f41640cb7a0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.356+0000 7f4182056740 1 -- 172.21.15.134:0/613938154 shutdown_connections 2024-07-26T11:24:10.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.356+0000 7f4182056740 1 --2- 172.21.15.134:0/613938154 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x5615647f6da0 0x561564c8fc40 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.356+0000 7f4182056740 1 --2- 172.21.15.134:0/613938154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561564842390 0x561564c175f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.356+0000 7f4182056740 1 --2- 172.21.15.134:0/613938154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561564863070 0x561564c85eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.356+0000 7f4182056740 1 --2- 172.21.15.134:0/613938154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56156483a1b0 0x561564c17030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.356+0000 7f4182056740 1 -- 172.21.15.134:0/613938154 >> 172.21.15.134:0/613938154 conn(0x561564857460 msgr2=0x561564c0e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:10.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.356+0000 7f4162000640 0 client.26659 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:24:10.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.358+0000 7f4182056740 1 -- 172.21.15.134:0/613938154 shutdown_connections 2024-07-26T11:24:10.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.358+0000 7f4182056740 1 -- 172.21.15.134:0/613938154 wait complete. 2024-07-26T11:24:10.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.359+0000 7f4182056740 20 client.26659 trim_cache size 0 max 16384 2024-07-26T11:24:10.361 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:24:10.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.360+0000 7f417b400640 1 -- 172.21.15.134:0/127076633 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f41580b3e40 msgr2=0x7f41580b6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:10.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.360+0000 7f417b400640 1 --2- 172.21.15.134:0/127076633 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f41580b3e40 0x7f41580b6350 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4174115970 tx=0x7f416c009500 comp rx=0 tx=0).stop 2024-07-26T11:24:10.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.360+0000 7f417b400640 1 -- 172.21.15.134:0/127076633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f417410a910 msgr2=0x7f4174114460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:10.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.360+0000 7f417b400640 1 --2- 172.21.15.134:0/127076633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f417410a910 0x7f4174114460 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f4164005ab0 tx=0x7f4164003b90 comp rx=0 tx=0).stop 2024-07-26T11:24:10.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.360+0000 7f417b400640 1 -- 172.21.15.134:0/127076633 shutdown_connections 2024-07-26T11:24:10.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.360+0000 7f417b400640 1 --2- 172.21.15.134:0/127076633 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f41580b3e40 0x7f41580b6350 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.360+0000 7f417b400640 1 --2- 172.21.15.134:0/127076633 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f417410a910 0x7f4174114460 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.360+0000 7f417b400640 1 --2- 172.21.15.134:0/127076633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4174116070 0x7f41741149a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.360+0000 7f417b400640 1 --2- 172.21.15.134:0/127076633 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f4174109fc0 0x7f4174113f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:10.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.360+0000 7f417b400640 1 -- 172.21.15.134:0/127076633 >> 172.21.15.134:0/127076633 conn(0x7f41740708f0 msgr2=0x7f4174071850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:10.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.361+0000 7f417b400640 1 -- 172.21.15.134:0/127076633 shutdown_connections 2024-07-26T11:24:10.364 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:10.361+0000 7f417b400640 1 -- 172.21.15.134:0/127076633 wait complete. 2024-07-26T11:24:11.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-07-26T11:24:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:11 smithi134 ceph-mon[31966]: pgmap v728: 129 pgs: 129 active+clean; 62 GiB data, 130 GiB used, 942 GiB / 1.0 TiB avail; 41 KiB/s rd, 277 MiB/s wr, 1.69k op/s 2024-07-26T11:24:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:11 smithi134 ceph-mon[31966]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T11:24:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:11 smithi189 ceph-mon[36790]: pgmap v728: 129 pgs: 129 active+clean; 62 GiB data, 130 GiB used, 942 GiB / 1.0 TiB avail; 41 KiB/s rd, 277 MiB/s wr, 1.69k op/s 2024-07-26T11:24:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:11 smithi189 ceph-mon[36790]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T11:24:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:11 smithi193 ceph-mon[37473]: pgmap v728: 129 pgs: 129 active+clean; 62 GiB data, 130 GiB used, 942 GiB / 1.0 TiB avail; 41 KiB/s rd, 277 MiB/s wr, 1.69k op/s 2024-07-26T11:24:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:11 smithi193 ceph-mon[37473]: osdmap e93: 12 total, 12 up, 12 in 2024-07-26T11:24:12.167 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:24:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:13 smithi189 ceph-mon[36790]: pgmap v730: 129 pgs: 129 active+clean; 62 GiB data, 129 GiB used, 943 GiB / 1.0 TiB avail; 54 KiB/s rd, 254 MiB/s wr, 1.64k op/s 2024-07-26T11:24:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:24:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:13 smithi193 ceph-mon[37473]: pgmap v730: 129 pgs: 129 active+clean; 62 GiB data, 129 GiB used, 943 GiB / 1.0 TiB avail; 54 KiB/s rd, 254 MiB/s wr, 1.64k op/s 2024-07-26T11:24:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:24:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:13 smithi134 ceph-mon[31966]: pgmap v730: 129 pgs: 129 active+clean; 62 GiB data, 129 GiB used, 943 GiB / 1.0 TiB avail; 54 KiB/s rd, 254 MiB/s wr, 1.64k op/s 2024-07-26T11:24:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:24:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:15 smithi189 ceph-mon[36790]: pgmap v731: 129 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 102 active+clean; 59 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 42 KiB/s rd, 203 MiB/s wr, 1.39k op/s 2024-07-26T11:24:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:24:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:24:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:24:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:15 smithi193 ceph-mon[37473]: pgmap v731: 129 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 102 active+clean; 59 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 42 KiB/s rd, 203 MiB/s wr, 1.39k op/s 2024-07-26T11:24:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:24:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:24:16.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:24:16.156 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:15 smithi134 ceph-mon[31966]: pgmap v731: 129 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 102 active+clean; 59 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 42 KiB/s rd, 203 MiB/s wr, 1.39k op/s 2024-07-26T11:24:16.156 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:24:16.156 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:24:16.157 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:24:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:16 smithi193 ceph-mon[37473]: pgmap v732: 129 pgs: 10 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 93 active+clean; 56 GiB data, 120 GiB used, 952 GiB / 1.0 TiB avail; 42 KiB/s rd, 139 MiB/s wr, 1.06k op/s 2024-07-26T11:24:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:16 smithi134 ceph-mon[31966]: pgmap v732: 129 pgs: 10 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 93 active+clean; 56 GiB data, 120 GiB used, 952 GiB / 1.0 TiB avail; 42 KiB/s rd, 139 MiB/s wr, 1.06k op/s 2024-07-26T11:24:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:16 smithi189 ceph-mon[36790]: pgmap v732: 129 pgs: 10 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 93 active+clean; 56 GiB data, 120 GiB used, 952 GiB / 1.0 TiB avail; 42 KiB/s rd, 139 MiB/s wr, 1.06k op/s 2024-07-26T11:24:19.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:19 smithi134 ceph-mon[31966]: pgmap v733: 129 pgs: 8 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 103 active+clean; 56 GiB data, 118 GiB used, 954 GiB / 1.0 TiB avail; 39 KiB/s rd, 117 MiB/s wr, 947 op/s 2024-07-26T11:24:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:19 smithi189 ceph-mon[36790]: pgmap v733: 129 pgs: 8 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 103 active+clean; 56 GiB data, 118 GiB used, 954 GiB / 1.0 TiB avail; 39 KiB/s rd, 117 MiB/s wr, 947 op/s 2024-07-26T11:24:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:19 smithi193 ceph-mon[37473]: pgmap v733: 129 pgs: 8 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 103 active+clean; 56 GiB data, 118 GiB used, 954 GiB / 1.0 TiB avail; 39 KiB/s rd, 117 MiB/s wr, 947 op/s 2024-07-26T11:24:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:21 smithi134 ceph-mon[31966]: pgmap v734: 129 pgs: 6 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 110 active+clean; 55 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 38 KiB/s rd, 101 MiB/s wr, 953 op/s 2024-07-26T11:24:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:21 smithi189 ceph-mon[36790]: pgmap v734: 129 pgs: 6 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 110 active+clean; 55 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 38 KiB/s rd, 101 MiB/s wr, 953 op/s 2024-07-26T11:24:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:21 smithi193 ceph-mon[37473]: pgmap v734: 129 pgs: 6 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 110 active+clean; 55 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 38 KiB/s rd, 101 MiB/s wr, 953 op/s 2024-07-26T11:24:22.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.213+0000 7fd38b600640 1 -- 172.21.15.134:0/1170164822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd38410c700 msgr2=0x7fd38410cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:22.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.213+0000 7fd38b600640 1 --2- 172.21.15.134:0/1170164822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd38410c700 0x7fd38410cb10 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd37c00bd70 tx=0x7fd37c02c450 comp rx=0 tx=0).stop 2024-07-26T11:24:22.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.213+0000 7fd38b600640 1 -- 172.21.15.134:0/1170164822 shutdown_connections 2024-07-26T11:24:22.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.213+0000 7fd38b600640 1 --2- 172.21.15.134:0/1170164822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd38410c700 0x7fd38410cb10 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.213+0000 7fd38b600640 1 --2- 172.21.15.134:0/1170164822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd38410d050 0x7fd384116ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.213+0000 7fd38b600640 1 --2- 172.21.15.134:0/1170164822 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd38410a080 0x7fd38410a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.213+0000 7fd38b600640 1 -- 172.21.15.134:0/1170164822 >> 172.21.15.134:0/1170164822 conn(0x7fd3840708f0 msgr2=0x7fd384070d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:22.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.216+0000 7fd38b600640 1 -- 172.21.15.134:0/1170164822 shutdown_connections 2024-07-26T11:24:22.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.217+0000 7fd38b600640 1 -- 172.21.15.134:0/1170164822 wait complete. 2024-07-26T11:24:22.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.218+0000 7fd38b600640 1 Processor -- start 2024-07-26T11:24:22.221 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.219+0000 7fd38b600640 1 -- start start 2024-07-26T11:24:22.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.221+0000 7fd38b600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd38410a080 0x7fd3841a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.221+0000 7fd38b600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd38410c700 0x7fd3841a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.221+0000 7fd38b600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd38410d050 0x7fd3841aa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.221+0000 7fd38b600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd384119b70 con 0x7fd38410d050 2024-07-26T11:24:22.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.221+0000 7fd38b600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd3841199f0 con 0x7fd38410c700 2024-07-26T11:24:22.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.221+0000 7fd38b600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd384119cf0 con 0x7fd38410a080 2024-07-26T11:24:22.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.222+0000 7fd383e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd38410c700 0x7fd3841a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.222+0000 7fd388c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd38410a080 0x7fd3841a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.222+0000 7fd383e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd38410c700 0x7fd3841a4f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34592/0 (socket says 172.21.15.134:34592) 2024-07-26T11:24:22.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.222+0000 7fd383e00640 1 -- 172.21.15.134:0/3372788634 learned_addr learned my addr 172.21.15.134:0/3372788634 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:22.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.222+0000 7fd383e00640 1 -- 172.21.15.134:0/3372788634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd38410a080 msgr2=0x7fd3841a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:22.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.222+0000 7fd383e00640 1 --2- 172.21.15.134:0/3372788634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd38410a080 0x7fd3841a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.222+0000 7fd383e00640 1 -- 172.21.15.134:0/3372788634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd38410d050 msgr2=0x7fd3841aa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:22.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.222+0000 7fd383e00640 1 --2- 172.21.15.134:0/3372788634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd38410d050 0x7fd3841aa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.222+0000 7fd383e00640 1 -- 172.21.15.134:0/3372788634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3841aa900 con 0x7fd38410c700 2024-07-26T11:24:22.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.223+0000 7fd383e00640 1 --2- 172.21.15.134:0/3372788634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd38410c700 0x7fd3841a4f30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd37c006010 tx=0x7fd37c003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:22.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.225+0000 7fd381600640 1 -- 172.21.15.134:0/3372788634 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c040020 con 0x7fd38410c700 2024-07-26T11:24:22.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.225+0000 7fd381600640 1 -- 172.21.15.134:0/3372788634 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd37c009a90 con 0x7fd38410c700 2024-07-26T11:24:22.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.225+0000 7fd381600640 1 -- 172.21.15.134:0/3372788634 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c040020 con 0x7fd38410c700 2024-07-26T11:24:22.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.225+0000 7fd38b600640 1 -- 172.21.15.134:0/3372788634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3841aab90 con 0x7fd38410c700 2024-07-26T11:24:22.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.227+0000 7fd38b600640 1 -- 172.21.15.134:0/3372788634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3841aaf30 con 0x7fd38410c700 2024-07-26T11:24:22.233 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.231+0000 7fd381600640 1 -- 172.21.15.134:0/3372788634 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fd37c043020 con 0x7fd38410c700 2024-07-26T11:24:22.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.234+0000 7fd381600640 1 --2- 172.21.15.134:0/3372788634 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd3680b3ff0 0x7fd3680b6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.235+0000 7fd381600640 1 -- 172.21.15.134:0/3372788634 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9599+0+0 (secure 0 0 0) 0x7fd37c1047c0 con 0x7fd38410c700 2024-07-26T11:24:22.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.238+0000 7fd388c00640 1 --2- 172.21.15.134:0/3372788634 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd3680b3ff0 0x7fd3680b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.241+0000 7fd388c00640 1 --2- 172.21.15.134:0/3372788634 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd3680b3ff0 0x7fd3680b6500 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd370005fd0 tx=0x7fd37000c040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:22.263 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.261+0000 7fd392404740 1 Processor -- start 2024-07-26T11:24:22.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.262+0000 7fd392404740 1 -- start start 2024-07-26T11:24:22.264 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.262+0000 7fd392404740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de987070 0x5593ded2be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.262+0000 7fd392404740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593de95e1b0 0x5593ded2c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.262+0000 7fd392404740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de966390 0x5593ded2c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.262+0000 7fd392404740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5593deca2bf0 con 0x5593de95e1b0 2024-07-26T11:24:22.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.262+0000 7fd392404740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5593dec920f0 con 0x5593de966390 2024-07-26T11:24:22.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.263+0000 7fd392404740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5593dec96130 con 0x5593de987070 2024-07-26T11:24:22.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.263+0000 7fd389600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de966390 0x5593ded2c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.263+0000 7fd389600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de966390 0x5593ded2c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34604/0 (socket says 172.21.15.134:34604) 2024-07-26T11:24:22.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.263+0000 7fd389600640 1 -- 172.21.15.134:0/2314099515 learned_addr learned my addr 172.21.15.134:0/2314099515 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:22.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.264+0000 7fd389600640 1 -- 172.21.15.134:0/2314099515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de987070 msgr2=0x5593ded2be50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:24:22.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.264+0000 7fd389600640 1 --2- 172.21.15.134:0/2314099515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de987070 0x5593ded2be50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.264+0000 7fd389600640 1 -- 172.21.15.134:0/2314099515 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593de95e1b0 msgr2=0x5593ded2c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:22.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.264+0000 7fd389600640 1 --2- 172.21.15.134:0/2314099515 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593de95e1b0 0x5593ded2c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.264+0000 7fd389600640 1 -- 172.21.15.134:0/2314099515 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593de9f2650 con 0x5593de966390 2024-07-26T11:24:22.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.264+0000 7fd389600640 1 --2- 172.21.15.134:0/2314099515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de966390 0x5593ded2c8d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd374009790 tx=0x7fd374009c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=951979d922b9a835 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:22.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.265+0000 7fd379c00640 1 -- 172.21.15.134:0/2314099515 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd374015020 con 0x5593de966390 2024-07-26T11:24:22.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.265+0000 7fd379c00640 1 -- 172.21.15.134:0/2314099515 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd374011420 con 0x5593de966390 2024-07-26T11:24:22.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.265+0000 7fd379c00640 1 -- 172.21.15.134:0/2314099515 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3740105c0 con 0x5593de966390 2024-07-26T11:24:22.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.267+0000 7fd392404740 1 -- 172.21.15.134:0/2314099515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de966390 msgr2=0x5593ded2c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:22.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.267+0000 7fd392404740 1 --2- 172.21.15.134:0/2314099515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de966390 0x5593ded2c8d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd374009790 tx=0x7fd374009c50 comp rx=0 tx=0).stop 2024-07-26T11:24:22.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.268+0000 7fd392404740 1 -- 172.21.15.134:0/2314099515 shutdown_connections 2024-07-26T11:24:22.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.268+0000 7fd392404740 1 --2- 172.21.15.134:0/2314099515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de966390 0x5593ded2c8d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.268+0000 7fd392404740 1 --2- 172.21.15.134:0/2314099515 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593de95e1b0 0x5593ded2c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.268+0000 7fd392404740 1 --2- 172.21.15.134:0/2314099515 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de987070 0x5593ded2be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.268+0000 7fd392404740 1 -- 172.21.15.134:0/2314099515 >> 172.21.15.134:0/2314099515 conn(0x5593de97b460 msgr2=0x5593ded29d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:22.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.268+0000 7fd392404740 1 -- 172.21.15.134:0/2314099515 shutdown_connections 2024-07-26T11:24:22.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.269+0000 7fd392404740 1 -- 172.21.15.134:0/2314099515 wait complete. 2024-07-26T11:24:22.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.269+0000 7fd392404740 1 -- start start 2024-07-26T11:24:22.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.269+0000 7fd392404740 1 Processor -- start 2024-07-26T11:24:22.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.273+0000 7fd392404740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593de95e1b0 0x5593ded40590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.273+0000 7fd392404740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de966390 0x5593ded40b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.273+0000 7fd392404740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de987070 0x5593ded45b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.274+0000 7fd392404740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5593deaf5700 con 0x5593de95e1b0 2024-07-26T11:24:22.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.274+0000 7fd383e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de966390 0x5593ded40b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.274+0000 7fd389600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de987070 0x5593ded45b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.274+0000 7fd389600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de987070 0x5593ded45b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34614/0 (socket says 172.21.15.134:34614) 2024-07-26T11:24:22.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.274+0000 7fd389600640 1 -- 172.21.15.134:0/191798625 learned_addr learned my addr 172.21.15.134:0/191798625 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:22.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.274+0000 7fd392404740 1 -- 172.21.15.134:0/191798625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5593dec2e6c0 con 0x5593de987070 2024-07-26T11:24:22.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.274+0000 7fd392404740 1 -- 172.21.15.134:0/191798625 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5593ded3a630 con 0x5593de966390 2024-07-26T11:24:22.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.275+0000 7fd389600640 1 -- 172.21.15.134:0/191798625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de966390 msgr2=0x5593ded40b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:22.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.275+0000 7fd392404740 10 client.0 _ll_register_callbacks cb 0x5593dec3e170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:24:22.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.275+0000 7fd389600640 1 --2- 172.21.15.134:0/191798625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de966390 0x5593ded40b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.275+0000 7fd389600640 1 -- 172.21.15.134:0/191798625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593de95e1b0 msgr2=0x5593ded40590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:22.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.275+0000 7fd389600640 1 --2- 172.21.15.134:0/191798625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593de95e1b0 0x5593ded40590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.275+0000 7fd389600640 1 -- 172.21.15.134:0/191798625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593dec45860 con 0x5593de987070 2024-07-26T11:24:22.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.275+0000 7fd392404740 1 -- 172.21.15.134:0/191798625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd374011420 con 0x5593de987070 2024-07-26T11:24:22.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.276+0000 7fd389600640 1 --2- 172.21.15.134:0/191798625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de987070 0x5593ded45b90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd374011d20 tx=0x7fd3740094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=877d87f1e6dc787a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:22.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.276+0000 7fd36e000640 10 client.26651 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:24:22.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.276+0000 7fd36e000640 1 -- 172.21.15.134:0/191798625 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd374027060 con 0x5593de987070 2024-07-26T11:24:22.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.277+0000 7fd36e000640 1 -- 172.21.15.134:0/191798625 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9599+0+0 (secure 0 0 0) 0x7fd37400b8d0 con 0x5593de987070 2024-07-26T11:24:22.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.277+0000 7fd36e000640 1 -- 172.21.15.134:0/191798625 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd374011960 con 0x5593de987070 2024-07-26T11:24:22.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.277+0000 7fd392404740 10 client.26651 fetch_fsmap learned FSMap version 32 2024-07-26T11:24:22.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.277+0000 7fd392404740 1 -- 172.21.15.134:0/191798625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=32}) -- 0x5593ded4ea20 con 0x5593de987070 2024-07-26T11:24:22.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.278+0000 7fd36e000640 1 -- 172.21.15.134:0/191798625 <== mon.2 v2:172.21.15.189:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fd37400baf0 con 0x5593de987070 2024-07-26T11:24:22.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.278+0000 7fd392404740 10 client.26651 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:24:22.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.278+0000 7fd392404740 10 client.26651 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T11:24:22.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.278+0000 7fd392404740 20 client.26651 populate_metadata read hostname 'smithi134' 2024-07-26T11:24:22.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.278+0000 7fd392404740 1 --2- 172.21.15.134:0/191798625 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5593de9459f0 0x5593ded51d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.278+0000 7fd392404740 4 client.26651 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:24:22.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.278+0000 7fd392404740 1 -- 172.21.15.134:0/191798625 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5593deca4b40 con 0x5593de9459f0 2024-07-26T11:24:22.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.279+0000 7fd388c00640 1 --2- 172.21.15.134:0/191798625 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5593de9459f0 0x5593ded51d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.280+0000 7fd388c00640 1 --2- 172.21.15.134:0/191798625 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5593de9459f0 0x5593ded51d80 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2864191665c670bc server_cookie=2c4a727d661c0182 in_seq=0 out_seq=0 2024-07-26T11:24:22.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.280+0000 7fd36e000640 10 client.26651 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:24:22.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.281+0000 7fd36e000640 1 -- 172.21.15.134:0/191798625 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5593deca4b40 con 0x5593de9459f0 2024-07-26T11:24:22.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.281+0000 7fd36e000640 10 client.26651 handle_command_reply: tid=0 2024-07-26T11:24:22.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.281+0000 7fd392404740 1 client.26651 shutdown 2024-07-26T11:24:22.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.282+0000 7fd392404740 1 -- 172.21.15.134:0/191798625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de987070 msgr2=0x5593ded45b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:22.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.282+0000 7fd392404740 1 --2- 172.21.15.134:0/191798625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de987070 0x5593ded45b90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd374011d20 tx=0x7fd3740094b0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.283+0000 7fd392404740 1 -- 172.21.15.134:0/191798625 shutdown_connections 2024-07-26T11:24:22.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.283+0000 7fd392404740 1 --2- 172.21.15.134:0/191798625 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5593de9459f0 0x5593ded51d80 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.283+0000 7fd392404740 1 --2- 172.21.15.134:0/191798625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de966390 0x5593ded40b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.283+0000 7fd392404740 1 --2- 172.21.15.134:0/191798625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de987070 0x5593ded45b90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.283+0000 7fd392404740 1 --2- 172.21.15.134:0/191798625 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593de95e1b0 0x5593ded40590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.283+0000 7fd392404740 1 -- 172.21.15.134:0/191798625 >> 172.21.15.134:0/191798625 conn(0x5593de97b460 msgr2=0x5593ded2ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:22.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.284+0000 7fd36e000640 0 client.26651 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:24:22.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.284+0000 7fd392404740 1 -- 172.21.15.134:0/191798625 shutdown_connections 2024-07-26T11:24:22.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.284+0000 7fd392404740 1 -- 172.21.15.134:0/191798625 wait complete. 2024-07-26T11:24:22.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.284+0000 7fd392404740 20 client.26651 trim_cache size 0 max 16384 2024-07-26T11:24:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.311+0000 7fd392404740 1 Processor -- start 2024-07-26T11:24:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.311+0000 7fd392404740 1 -- start start 2024-07-26T11:24:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.312+0000 7fd392404740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593de95e1b0 0x5593ded39950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.312+0000 7fd392404740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de966390 0x5593ded29ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.312+0000 7fd392404740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de987070 0x5593ded45a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.312+0000 7fd383e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de966390 0x5593ded29ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.313+0000 7fd383e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de966390 0x5593ded29ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34622/0 (socket says 172.21.15.134:34622) 2024-07-26T11:24:22.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.313+0000 7fd383e00640 1 -- 172.21.15.134:0/3377719884 learned_addr learned my addr 172.21.15.134:0/3377719884 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:22.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.313+0000 7fd392404740 1 -- 172.21.15.134:0/3377719884 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd3841b7180 con 0x5593de95e1b0 2024-07-26T11:24:22.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.313+0000 7fd389600640 1 --2- 172.21.15.134:0/3377719884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de987070 0x5593ded45a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.313+0000 7fd392404740 1 -- 172.21.15.134:0/3377719884 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd3841b8cc0 con 0x5593de966390 2024-07-26T11:24:22.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.313+0000 7fd392404740 1 -- 172.21.15.134:0/3377719884 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd3841b7850 con 0x5593de987070 2024-07-26T11:24:22.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.314+0000 7fd389600640 1 -- 172.21.15.134:0/3377719884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de966390 msgr2=0x5593ded29ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:22.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.314+0000 7fd389600640 1 --2- 172.21.15.134:0/3377719884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de966390 0x5593ded29ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.314+0000 7fd389600640 1 -- 172.21.15.134:0/3377719884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593de95e1b0 msgr2=0x5593ded39950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:22.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.314+0000 7fd389600640 1 --2- 172.21.15.134:0/3377719884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593de95e1b0 0x5593ded39950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.314+0000 7fd389600640 1 -- 172.21.15.134:0/3377719884 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593ded251b0 con 0x5593de987070 2024-07-26T11:24:22.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.315+0000 7fd389600640 1 --2- 172.21.15.134:0/3377719884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de987070 0x5593ded45a00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd374011d20 tx=0x7fd37401d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6744fdfea7ec593d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:22.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.317+0000 7fd379200640 1 -- 172.21.15.134:0/3377719884 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd374015020 con 0x5593de987070 2024-07-26T11:24:22.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.317+0000 7fd379200640 1 -- 172.21.15.134:0/3377719884 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd374011420 con 0x5593de987070 2024-07-26T11:24:22.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.317+0000 7fd379200640 1 -- 172.21.15.134:0/3377719884 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37401a420 con 0x5593de987070 2024-07-26T11:24:22.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.318+0000 7fd392404740 1 -- 172.21.15.134:0/3377719884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de987070 msgr2=0x5593ded45a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:22.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.318+0000 7fd392404740 1 --2- 172.21.15.134:0/3377719884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de987070 0x5593ded45a00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd374011d20 tx=0x7fd37401d000 comp rx=0 tx=0).stop 2024-07-26T11:24:22.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.319+0000 7fd392404740 1 -- 172.21.15.134:0/3377719884 shutdown_connections 2024-07-26T11:24:22.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.319+0000 7fd392404740 1 --2- 172.21.15.134:0/3377719884 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de987070 0x5593ded45a00 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.319+0000 7fd392404740 1 --2- 172.21.15.134:0/3377719884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de966390 0x5593ded29ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.319+0000 7fd392404740 1 --2- 172.21.15.134:0/3377719884 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593de95e1b0 0x5593ded39950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.319+0000 7fd392404740 1 -- 172.21.15.134:0/3377719884 >> 172.21.15.134:0/3377719884 conn(0x5593de97b460 msgr2=0x5593ded2ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:22.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.319+0000 7fd392404740 1 -- 172.21.15.134:0/3377719884 shutdown_connections 2024-07-26T11:24:22.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.319+0000 7fd392404740 1 -- 172.21.15.134:0/3377719884 wait complete. 2024-07-26T11:24:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.320+0000 7fd392404740 1 -- start start 2024-07-26T11:24:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.320+0000 7fd392404740 1 Processor -- start 2024-07-26T11:24:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.321+0000 7fd392404740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593de95e1b0 0x5593ded4c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.321+0000 7fd392404740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de966390 0x5593ded46e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.322+0000 7fd392404740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de91ada0 0x5593ded474c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.322+0000 7fd392404740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd3841b85f0 con 0x5593de95e1b0 2024-07-26T11:24:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.322+0000 7fd392404740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd3841b7f20 con 0x5593de91ada0 2024-07-26T11:24:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.322+0000 7fd392404740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd3841b5180 con 0x5593de966390 2024-07-26T11:24:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.322+0000 7fd392404740 10 client.0 _ll_register_callbacks cb 0x7fd374009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:24:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.323+0000 7fd389600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de91ada0 0x5593ded474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.323+0000 7fd389600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de91ada0 0x5593ded474c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34626/0 (socket says 172.21.15.134:34626) 2024-07-26T11:24:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.323+0000 7fd389600640 1 -- 172.21.15.134:0/455199904 learned_addr learned my addr 172.21.15.134:0/455199904 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.323+0000 7fd383e00640 1 --2- 172.21.15.134:0/455199904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de966390 0x5593ded46e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.324+0000 7fd389600640 1 -- 172.21.15.134:0/455199904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de966390 msgr2=0x5593ded46e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.324+0000 7fd389600640 1 --2- 172.21.15.134:0/455199904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de966390 0x5593ded46e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.324+0000 7fd389600640 1 -- 172.21.15.134:0/455199904 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593de95e1b0 msgr2=0x5593ded4c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:22.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.324+0000 7fd389600640 1 --2- 172.21.15.134:0/455199904 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593de95e1b0 0x5593ded4c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.324+0000 7fd389600640 1 -- 172.21.15.134:0/455199904 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593ded47c30 con 0x5593de91ada0 2024-07-26T11:24:22.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.325+0000 7fd383e00640 1 --2- 172.21.15.134:0/455199904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de966390 0x5593ded46e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:24:22.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.325+0000 7fd36e000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:24:22.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.325+0000 7fd389600640 1 --2- 172.21.15.134:0/455199904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de91ada0 0x5593ded474c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd37400bf20 tx=0x7fd37400bb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94d5ea127c9f850f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:22.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.326+0000 7fd36e000640 1 -- 172.21.15.134:0/455199904 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37403a060 con 0x5593de91ada0 2024-07-26T11:24:22.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.326+0000 7fd36e000640 1 -- 172.21.15.134:0/455199904 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9599+0+0 (secure 0 0 0) 0x7fd37402c640 con 0x5593de91ada0 2024-07-26T11:24:22.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.327+0000 7fd392404740 1 -- 172.21.15.134:0/455199904 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd374011420 con 0x5593de91ada0 2024-07-26T11:24:22.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.328+0000 7fd36e000640 1 -- 172.21.15.134:0/455199904 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd37403a060 con 0x5593de91ada0 2024-07-26T11:24:22.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.330+0000 7fd392404740 10 client.26657 fetch_fsmap learned FSMap version 32 2024-07-26T11:24:22.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.330+0000 7fd392404740 1 -- 172.21.15.134:0/455199904 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5593dedb93f0 con 0x5593de91ada0 2024-07-26T11:24:22.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.330+0000 7fd36e000640 1 -- 172.21.15.134:0/455199904 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd37402ccf0 con 0x5593de91ada0 2024-07-26T11:24:22.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.331+0000 7fd36e000640 1 -- 172.21.15.134:0/455199904 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fd374035990 con 0x5593de91ada0 2024-07-26T11:24:22.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.331+0000 7fd392404740 10 client.26657 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:24:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.331+0000 7fd392404740 10 client.26657 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T11:24:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.331+0000 7fd392404740 20 client.26657 populate_metadata read hostname 'smithi134' 2024-07-26T11:24:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.331+0000 7fd392404740 1 --2- 172.21.15.134:0/455199904 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5593dedb9a90 0x5593dedbbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.331+0000 7fd392404740 4 client.26657 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:24:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.331+0000 7fd392404740 1 -- 172.21.15.134:0/455199904 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fd374011420 con 0x5593dedb9a90 2024-07-26T11:24:22.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.332+0000 7fd388c00640 1 --2- 172.21.15.134:0/455199904 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5593dedb9a90 0x5593dedbbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.336+0000 7fd388c00640 1 --2- 172.21.15.134:0/455199904 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5593dedb9a90 0x5593dedbbeb0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dc06975d79600bb5 server_cookie=d175d288d6ce5df2 in_seq=0 out_seq=0 2024-07-26T11:24:22.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.336+0000 7fd36e000640 10 client.26657 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:24:22.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.336+0000 7fd36e000640 1 -- 172.21.15.134:0/455199904 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fd374011420 con 0x5593dedb9a90 2024-07-26T11:24:22.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.336+0000 7fd36e000640 10 client.26657 handle_command_reply: tid=0 2024-07-26T11:24:22.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.336+0000 7fd392404740 1 client.26657 shutdown 2024-07-26T11:24:22.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.336+0000 7fd392404740 1 -- 172.21.15.134:0/455199904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de91ada0 msgr2=0x5593ded474c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:22.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.336+0000 7fd392404740 1 --2- 172.21.15.134:0/455199904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de91ada0 0x5593ded474c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd37400bf20 tx=0x7fd37400bb00 comp rx=0 tx=0).stop 2024-07-26T11:24:22.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.337+0000 7fd392404740 1 -- 172.21.15.134:0/455199904 shutdown_connections 2024-07-26T11:24:22.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.337+0000 7fd392404740 1 --2- 172.21.15.134:0/455199904 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x5593dedb9a90 0x5593dedbbeb0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.337+0000 7fd392404740 1 --2- 172.21.15.134:0/455199904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5593de966390 0x5593ded46e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.337+0000 7fd392404740 1 --2- 172.21.15.134:0/455199904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5593de91ada0 0x5593ded474c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.337+0000 7fd392404740 1 --2- 172.21.15.134:0/455199904 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5593de95e1b0 0x5593ded4c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.337+0000 7fd392404740 1 -- 172.21.15.134:0/455199904 >> 172.21.15.134:0/455199904 conn(0x5593de97b460 msgr2=0x5593ded2f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:22.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.337+0000 7fd36e000640 0 client.26657 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:24:22.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.337+0000 7fd392404740 1 -- 172.21.15.134:0/455199904 shutdown_connections 2024-07-26T11:24:22.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.337+0000 7fd392404740 1 -- 172.21.15.134:0/455199904 wait complete. 2024-07-26T11:24:22.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.338+0000 7fd392404740 20 client.26657 trim_cache size 0 max 16384 2024-07-26T11:24:22.341 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:24:22.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.339+0000 7fd38b600640 1 -- 172.21.15.134:0/3372788634 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd3680b3ff0 msgr2=0x7fd3680b6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:22.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.339+0000 7fd38b600640 1 --2- 172.21.15.134:0/3372788634 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd3680b3ff0 0x7fd3680b6500 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd370005fd0 tx=0x7fd37000c040 comp rx=0 tx=0).stop 2024-07-26T11:24:22.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.339+0000 7fd38b600640 1 -- 172.21.15.134:0/3372788634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd38410c700 msgr2=0x7fd3841a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:22.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.339+0000 7fd38b600640 1 --2- 172.21.15.134:0/3372788634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd38410c700 0x7fd3841a4f30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd37c006010 tx=0x7fd37c003ba0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.339+0000 7fd38b600640 1 -- 172.21.15.134:0/3372788634 shutdown_connections 2024-07-26T11:24:22.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.339+0000 7fd38b600640 1 --2- 172.21.15.134:0/3372788634 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd3680b3ff0 0x7fd3680b6500 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.339+0000 7fd38b600640 1 --2- 172.21.15.134:0/3372788634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd38410d050 0x7fd3841aa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.339+0000 7fd38b600640 1 --2- 172.21.15.134:0/3372788634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd38410c700 0x7fd3841a4f30 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.339+0000 7fd38b600640 1 --2- 172.21.15.134:0/3372788634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd38410a080 0x7fd3841a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.339+0000 7fd38b600640 1 -- 172.21.15.134:0/3372788634 >> 172.21.15.134:0/3372788634 conn(0x7fd3840708f0 msgr2=0x7fd38410da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:22.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.340+0000 7fd38b600640 1 -- 172.21.15.134:0/3372788634 shutdown_connections 2024-07-26T11:24:22.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:22.342+0000 7fd38b600640 1 -- 172.21.15.134:0/3372788634 wait complete. 2024-07-26T11:24:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.471+0000 7ffa95200640 1 -- 172.21.15.134:0/486184066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa90109fd0 msgr2=0x7ffa9010a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:22.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.471+0000 7ffa95200640 1 --2- 172.21.15.134:0/486184066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa90109fd0 0x7ffa9010a3e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ffa78009900 tx=0x7ffa7802a410 comp rx=0 tx=0).stop 2024-07-26T11:24:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.475+0000 7ffa95200640 1 -- 172.21.15.134:0/486184066 shutdown_connections 2024-07-26T11:24:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.475+0000 7ffa95200640 1 --2- 172.21.15.134:0/486184066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa90115fd0 0x7ffa901184b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.475+0000 7ffa95200640 1 --2- 172.21.15.134:0/486184066 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa9010a920 0x7ffa90115a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.475+0000 7ffa95200640 1 --2- 172.21.15.134:0/486184066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa90109fd0 0x7ffa9010a3e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.475+0000 7ffa95200640 1 -- 172.21.15.134:0/486184066 >> 172.21.15.134:0/486184066 conn(0x7ffa90070800 msgr2=0x7ffa90070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.475+0000 7ffa95200640 1 -- 172.21.15.134:0/486184066 shutdown_connections 2024-07-26T11:24:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.475+0000 7ffa95200640 1 -- 172.21.15.134:0/486184066 wait complete. 2024-07-26T11:24:22.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.477+0000 7ffa95200640 1 Processor -- start 2024-07-26T11:24:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.478+0000 7ffa95200640 1 -- start start 2024-07-26T11:24:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.479+0000 7ffa95200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa90109fd0 0x7ffa901a49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.479+0000 7ffa95200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa9010a920 0x7ffa901a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.479+0000 7ffa95200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa90115fd0 0x7ffa901a6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.479+0000 7ffa95200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9010bd80 con 0x7ffa9010a920 2024-07-26T11:24:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.479+0000 7ffa95200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9010bc00 con 0x7ffa90115fd0 2024-07-26T11:24:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.479+0000 7ffa95200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9010bf00 con 0x7ffa90109fd0 2024-07-26T11:24:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.479+0000 7ffa8de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa90109fd0 0x7ffa901a49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.479+0000 7ffa8de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa90109fd0 0x7ffa901a49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59844/0 (socket says 172.21.15.134:59844) 2024-07-26T11:24:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.479+0000 7ffa8de00640 1 -- 172.21.15.134:0/3321693444 learned_addr learned my addr 172.21.15.134:0/3321693444 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.480+0000 7ffa8de00640 1 -- 172.21.15.134:0/3321693444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa90115fd0 msgr2=0x7ffa901a6c80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T11:24:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.480+0000 7ffa8de00640 1 --2- 172.21.15.134:0/3321693444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa90115fd0 0x7ffa901a6c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.480+0000 7ffa8de00640 1 -- 172.21.15.134:0/3321693444 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa9010a920 msgr2=0x7ffa901a4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.481+0000 7ffa8d400640 1 --2- 172.21.15.134:0/3321693444 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa9010a920 0x7ffa901a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.481+0000 7ffa8de00640 1 --2- 172.21.15.134:0/3321693444 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa9010a920 0x7ffa901a4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.481+0000 7ffa8de00640 1 -- 172.21.15.134:0/3321693444 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa901a73b0 con 0x7ffa90109fd0 2024-07-26T11:24:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.481+0000 7ffa8d400640 1 --2- 172.21.15.134:0/3321693444 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa9010a920 0x7ffa901a4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:24:22.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.481+0000 7ffa8de00640 1 --2- 172.21.15.134:0/3321693444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa90109fd0 0x7ffa901a49c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ffa78009850 tx=0x7ffa78030650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.482+0000 7ffa86a00640 1 -- 172.21.15.134:0/3321693444 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7803f070 con 0x7ffa90109fd0 2024-07-26T11:24:22.484 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.482+0000 7ffa86a00640 1 -- 172.21.15.134:0/3321693444 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa78004460 con 0x7ffa90109fd0 2024-07-26T11:24:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.482+0000 7ffa86a00640 1 -- 172.21.15.134:0/3321693444 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa78035030 con 0x7ffa90109fd0 2024-07-26T11:24:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.482+0000 7ffa95200640 1 -- 172.21.15.134:0/3321693444 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa901a75e0 con 0x7ffa90109fd0 2024-07-26T11:24:22.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.483+0000 7ffa95200640 1 -- 172.21.15.134:0/3321693444 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa901a7a80 con 0x7ffa90109fd0 2024-07-26T11:24:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.491+0000 7ffa9c21f740 1 Processor -- start 2024-07-26T11:24:22.493 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.491+0000 7ffa9c21f740 1 -- start start 2024-07-26T11:24:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.492+0000 7ffa9c21f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b3c1b0 0x562854edfa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.492+0000 7ffa9c21f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b132f0 0x562854edffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.492+0000 7ffa9c21f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b1b4d0 0x562854ee04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.492+0000 7ffa9c21f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562854e54270 con 0x562854b1b4d0 2024-07-26T11:24:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.492+0000 7ffa9c21f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562854e43770 con 0x562854b3c1b0 2024-07-26T11:24:22.494 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.492+0000 7ffa9c21f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562854e477b0 con 0x562854b132f0 2024-07-26T11:24:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.493+0000 7ffa8e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b3c1b0 0x562854edfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.493+0000 7ffa8e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b3c1b0 0x562854edfa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34654/0 (socket says 172.21.15.134:34654) 2024-07-26T11:24:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.493+0000 7ffa8e800640 1 -- 172.21.15.134:0/4038672590 learned_addr learned my addr 172.21.15.134:0/4038672590 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:22.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.493+0000 7ffa8d400640 1 --2- 172.21.15.134:0/4038672590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b1b4d0 0x562854ee04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.494+0000 7ffa8e800640 1 -- 172.21.15.134:0/4038672590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b132f0 msgr2=0x562854edffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:24:22.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.494+0000 7ffa86a00640 1 -- 172.21.15.134:0/3321693444 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7ffa78042070 con 0x7ffa90109fd0 2024-07-26T11:24:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.496+0000 7ffa86a00640 1 --2- 172.21.15.134:0/3321693444 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ffa740b3f10 0x7ffa740b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.496+0000 7ffa86a00640 1 -- 172.21.15.134:0/3321693444 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9599+0+0 (secure 0 0 0) 0x7ffa78104b70 con 0x7ffa90109fd0 2024-07-26T11:24:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.497+0000 7ffa8e800640 1 --2- 172.21.15.134:0/4038672590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b132f0 0x562854edffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.497+0000 7ffa8e800640 1 -- 172.21.15.134:0/4038672590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b1b4d0 msgr2=0x562854ee04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.497+0000 7ffa8e800640 1 --2- 172.21.15.134:0/4038672590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b1b4d0 0x562854ee04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.497+0000 7ffa8e800640 1 -- 172.21.15.134:0/4038672590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562854ba7790 con 0x562854b3c1b0 2024-07-26T11:24:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.497+0000 7ffa8d400640 1 --2- 172.21.15.134:0/4038672590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b1b4d0 0x562854ee04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:24:22.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.497+0000 7ffa8e800640 1 --2- 172.21.15.134:0/4038672590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b3c1b0 0x562854edfa60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ffa88009d00 tx=0x7ffa8800e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=348acf2bb8241fea server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.497+0000 7ffa8e800640 1 --2- 172.21.15.134:0/3321693444 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ffa740b3f10 0x7ffa740b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.498+0000 7ffa7f400640 1 -- 172.21.15.134:0/4038672590 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa88016070 con 0x562854b3c1b0 2024-07-26T11:24:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.498+0000 7ffa7f400640 1 -- 172.21.15.134:0/4038672590 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa88011b20 con 0x562854b3c1b0 2024-07-26T11:24:22.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.498+0000 7ffa7f400640 1 -- 172.21.15.134:0/4038672590 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8801bd30 con 0x562854b3c1b0 2024-07-26T11:24:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.500+0000 7ffa9c21f740 1 -- 172.21.15.134:0/4038672590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b3c1b0 msgr2=0x562854edfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.500+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/4038672590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b3c1b0 0x562854edfa60 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ffa88009d00 tx=0x7ffa8800e6c0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.499+0000 7ffa8e800640 1 --2- 172.21.15.134:0/3321693444 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ffa740b3f10 0x7ffa740b6420 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ffa88009fd0 tx=0x7ffa8801bed0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.501+0000 7ffa9c21f740 1 -- 172.21.15.134:0/4038672590 shutdown_connections 2024-07-26T11:24:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.501+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/4038672590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b132f0 0x562854edffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.501+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/4038672590 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b1b4d0 0x562854ee04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.501+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/4038672590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b3c1b0 0x562854edfa60 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.501+0000 7ffa9c21f740 1 -- 172.21.15.134:0/4038672590 >> 172.21.15.134:0/4038672590 conn(0x562854b30300 msgr2=0x562854edd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.503+0000 7ffa9c21f740 1 -- 172.21.15.134:0/4038672590 shutdown_connections 2024-07-26T11:24:22.505 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.504+0000 7ffa9c21f740 1 -- 172.21.15.134:0/4038672590 wait complete. 2024-07-26T11:24:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.506+0000 7ffa9c21f740 1 -- start start 2024-07-26T11:24:22.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.508+0000 7ffa9c21f740 1 Processor -- start 2024-07-26T11:24:22.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.509+0000 7ffa9c21f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b132f0 0x562854ef3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.509+0000 7ffa9c21f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 0x562854ef43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.509+0000 7ffa8de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b132f0 0x562854ef3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.509+0000 7ffa8de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b132f0 0x562854ef3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34666/0 (socket says 172.21.15.134:34666) 2024-07-26T11:24:22.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.509+0000 7ffa8de00640 1 -- 172.21.15.134:0/586519653 learned_addr learned my addr 172.21.15.134:0/586519653 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:22.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.509+0000 7ffa8d400640 1 --2- 172.21.15.134:0/586519653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 0x562854ef43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.510+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/586519653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b3c1b0 0x562854effcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.510+0000 7ffa9c21f740 1 -- 172.21.15.134:0/586519653 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x562854d7dab0 con 0x562854b3c1b0 2024-07-26T11:24:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.510+0000 7ffa9c21f740 1 -- 172.21.15.134:0/586519653 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562854de0d50 con 0x562854b132f0 2024-07-26T11:24:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.510+0000 7ffa9c21f740 1 -- 172.21.15.134:0/586519653 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x562854ef69a0 con 0x562854b1b4d0 2024-07-26T11:24:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.510+0000 7ffa9c21f740 10 client.0 _ll_register_callbacks cb 0x562854b53a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:24:22.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.511+0000 7ffa8e800640 1 --2- 172.21.15.134:0/586519653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b3c1b0 0x562854effcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.513+0000 7ffa8e800640 1 -- 172.21.15.134:0/586519653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 msgr2=0x562854ef43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.513+0000 7ffa8e800640 1 --2- 172.21.15.134:0/586519653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 0x562854ef43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.513+0000 7ffa8e800640 1 -- 172.21.15.134:0/586519653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b132f0 msgr2=0x562854ef3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.513+0000 7ffa8e800640 1 --2- 172.21.15.134:0/586519653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b132f0 0x562854ef3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.513+0000 7ffa8e800640 1 -- 172.21.15.134:0/586519653 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562854c20620 con 0x562854b3c1b0 2024-07-26T11:24:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.513+0000 7ffa8d400640 1 --2- 172.21.15.134:0/586519653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 0x562854ef43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:24:22.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.514+0000 7ffa8e800640 1 --2- 172.21.15.134:0/586519653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b3c1b0 0x562854effcf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ffa8801b8d0 tx=0x7ffa88019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936d5ac59311b05b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:22.517 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.514+0000 7ffa73e00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:24:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.515+0000 7ffa73e00640 1 -- 172.21.15.134:0/586519653 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa88016050 con 0x562854b3c1b0 2024-07-26T11:24:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.516+0000 7ffa73e00640 1 -- 172.21.15.134:0/586519653 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9599+0+0 (secure 0 0 0) 0x7ffa88003b80 con 0x562854b3c1b0 2024-07-26T11:24:22.518 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.516+0000 7ffa9c21f740 1 -- 172.21.15.134:0/586519653 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa88011b20 con 0x562854b3c1b0 2024-07-26T11:24:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.517+0000 7ffa73e00640 1 -- 172.21.15.134:0/586519653 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffa880104f0 con 0x562854b3c1b0 2024-07-26T11:24:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.517+0000 7ffa9c21f740 10 client.16101 fetch_fsmap learned FSMap version 32 2024-07-26T11:24:22.519 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.517+0000 7ffa9c21f740 1 -- 172.21.15.134:0/586519653 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562854f02a10 con 0x562854b3c1b0 2024-07-26T11:24:22.520 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.518+0000 7ffa73e00640 1 -- 172.21.15.134:0/586519653 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa88011b20 con 0x562854b3c1b0 2024-07-26T11:24:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.519+0000 7ffa73e00640 1 -- 172.21.15.134:0/586519653 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7ffa8802f020 con 0x562854b3c1b0 2024-07-26T11:24:22.521 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.519+0000 7ffa9c21f740 10 client.16101 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:24:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.519+0000 7ffa9c21f740 10 client.16101 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:24:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.519+0000 7ffa9c21f740 20 client.16101 populate_metadata read hostname 'smithi134' 2024-07-26T11:24:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.519+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/586519653 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562854afad80 0x562854f06260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.519+0000 7ffa9c21f740 4 client.16101 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:24:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.519+0000 7ffa9c21f740 1 -- 172.21.15.134:0/586519653 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562854e561c0 con 0x562854afad80 2024-07-26T11:24:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.520+0000 7ffa8de00640 1 --2- 172.21.15.134:0/586519653 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562854afad80 0x562854f06260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.522 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.521+0000 7ffa8de00640 1 --2- 172.21.15.134:0/586519653 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562854afad80 0x562854f06260 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2248eec53b45811 server_cookie=faa72ae5ff49ebba in_seq=0 out_seq=0 2024-07-26T11:24:22.523 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.521+0000 7ffa73e00640 10 client.16101 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:24:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.522+0000 7ffa73e00640 1 -- 172.21.15.134:0/586519653 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x562854e561c0 con 0x562854afad80 2024-07-26T11:24:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.522+0000 7ffa73e00640 10 client.16101 handle_command_reply: tid=0 2024-07-26T11:24:22.524 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.523+0000 7ffa9c21f740 1 client.16101 shutdown 2024-07-26T11:24:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.523+0000 7ffa9c21f740 1 -- 172.21.15.134:0/586519653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b3c1b0 msgr2=0x562854effcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.523+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/586519653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b3c1b0 0x562854effcf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ffa8801b8d0 tx=0x7ffa88019000 comp rx=0 tx=0).stop 2024-07-26T11:24:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.523+0000 7ffa9c21f740 1 -- 172.21.15.134:0/586519653 shutdown_connections 2024-07-26T11:24:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.523+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/586519653 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562854afad80 0x562854f06260 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.523+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/586519653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 0x562854ef43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.523+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/586519653 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b3c1b0 0x562854effcf0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.525 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.523+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/586519653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b132f0 0x562854ef3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.526 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.523+0000 7ffa9c21f740 1 -- 172.21.15.134:0/586519653 >> 172.21.15.134:0/586519653 conn(0x562854b30300 msgr2=0x562854edc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:22.526 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.523+0000 7ffa73e00640 0 client.16101 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:24:22.526 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.524+0000 7ffa9c21f740 1 -- 172.21.15.134:0/586519653 shutdown_connections 2024-07-26T11:24:22.526 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.524+0000 7ffa9c21f740 1 -- 172.21.15.134:0/586519653 wait complete. 2024-07-26T11:24:22.526 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.524+0000 7ffa9c21f740 20 client.16101 trim_cache size 0 max 16384 2024-07-26T11:24:22.565 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.563+0000 7ffa9c21f740 1 Processor -- start 2024-07-26T11:24:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.563+0000 7ffa9c21f740 1 -- start start 2024-07-26T11:24:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.563+0000 7ffa9c21f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b132f0 0x562854edc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.563+0000 7ffa9c21f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 0x562854eef5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.566 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.563+0000 7ffa9c21f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b3c1b0 0x562854edff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.563+0000 7ffa9c21f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901b70d0 con 0x562854b132f0 2024-07-26T11:24:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.563+0000 7ffa9c21f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901b8c10 con 0x562854b3c1b0 2024-07-26T11:24:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.563+0000 7ffa9c21f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901b77a0 con 0x562854b1b4d0 2024-07-26T11:24:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.564+0000 7ffa8d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 0x562854eef5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.564+0000 7ffa8d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 0x562854eef5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59860/0 (socket says 172.21.15.134:59860) 2024-07-26T11:24:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.564+0000 7ffa8d400640 1 -- 172.21.15.134:0/4010583431 learned_addr learned my addr 172.21.15.134:0/4010583431 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.564+0000 7ffa8e800640 1 --2- 172.21.15.134:0/4010583431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b3c1b0 0x562854edff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.564+0000 7ffa8e800640 1 -- 172.21.15.134:0/4010583431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 msgr2=0x562854eef5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.564+0000 7ffa8e800640 1 --2- 172.21.15.134:0/4010583431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 0x562854eef5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.564+0000 7ffa8e800640 1 -- 172.21.15.134:0/4010583431 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b132f0 msgr2=0x562854edc7e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:24:22.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.565+0000 7ffa8de00640 1 --2- 172.21.15.134:0/4010583431 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b132f0 0x562854edc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.565+0000 7ffa8e800640 1 --2- 172.21.15.134:0/4010583431 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b132f0 0x562854edc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.565+0000 7ffa8e800640 1 -- 172.21.15.134:0/4010583431 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562854ede2e0 con 0x562854b3c1b0 2024-07-26T11:24:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.565+0000 7ffa8de00640 1 --2- 172.21.15.134:0/4010583431 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b132f0 0x562854edc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:24:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.566+0000 7ffa8e800640 1 --2- 172.21.15.134:0/4010583431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b3c1b0 0x562854edff90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ffa88011040 tx=0x7ffa8800ec40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89bbe7fc78099d50 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.566+0000 7ffa7ea00640 1 -- 172.21.15.134:0/4010583431 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa88016040 con 0x562854b3c1b0 2024-07-26T11:24:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.567+0000 7ffa7ea00640 1 -- 172.21.15.134:0/4010583431 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa88009900 con 0x562854b3c1b0 2024-07-26T11:24:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.567+0000 7ffa7ea00640 1 -- 172.21.15.134:0/4010583431 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8801b6b0 con 0x562854b3c1b0 2024-07-26T11:24:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.568+0000 7ffa9c21f740 1 -- 172.21.15.134:0/4010583431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b3c1b0 msgr2=0x562854edff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.568+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/4010583431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b3c1b0 0x562854edff90 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ffa88011040 tx=0x7ffa8800ec40 comp rx=0 tx=0).stop 2024-07-26T11:24:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.571+0000 7ffa9c21f740 1 -- 172.21.15.134:0/4010583431 shutdown_connections 2024-07-26T11:24:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.571+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/4010583431 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 0x562854eef5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.571+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/4010583431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b3c1b0 0x562854edff90 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.571+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/4010583431 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b132f0 0x562854edc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.571+0000 7ffa9c21f740 1 -- 172.21.15.134:0/4010583431 >> 172.21.15.134:0/4010583431 conn(0x562854b30300 msgr2=0x562854edc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.571+0000 7ffa9c21f740 1 -- 172.21.15.134:0/4010583431 shutdown_connections 2024-07-26T11:24:22.573 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.571+0000 7ffa9c21f740 1 -- 172.21.15.134:0/4010583431 wait complete. 2024-07-26T11:24:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.572+0000 7ffa9c21f740 1 -- start start 2024-07-26T11:24:22.574 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.572+0000 7ffa9c21f740 1 Processor -- start 2024-07-26T11:24:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.573+0000 7ffa9c21f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b132f0 0x562854f6be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.575 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.573+0000 7ffa9c21f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 0x562854f6c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.574+0000 7ffa9c21f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b3c1b0 0x562854f68b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.574+0000 7ffa8de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b132f0 0x562854f6be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.574+0000 7ffa9c21f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901b8540 con 0x562854b3c1b0 2024-07-26T11:24:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.574+0000 7ffa9c21f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901b7e70 con 0x562854b132f0 2024-07-26T11:24:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.574+0000 7ffa9c21f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ffa901b50d0 con 0x562854b1b4d0 2024-07-26T11:24:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.574+0000 7ffa8de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b132f0 0x562854f6be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:34688/0 (socket says 172.21.15.134:34688) 2024-07-26T11:24:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.574+0000 7ffa8de00640 1 -- 172.21.15.134:0/1177670419 learned_addr learned my addr 172.21.15.134:0/1177670419 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:22.576 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.574+0000 7ffa9c21f740 10 client.0 _ll_register_callbacks cb 0x7ffa88009b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:24:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.574+0000 7ffa8d400640 1 --2- 172.21.15.134:0/1177670419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 0x562854f6c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.575+0000 7ffa8d400640 1 -- 172.21.15.134:0/1177670419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b132f0 msgr2=0x562854f6be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.575+0000 7ffa8d400640 1 --2- 172.21.15.134:0/1177670419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b132f0 0x562854f6be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.575+0000 7ffa8d400640 1 -- 172.21.15.134:0/1177670419 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b3c1b0 msgr2=0x562854f68b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.575+0000 7ffa8d400640 1 --2- 172.21.15.134:0/1177670419 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b3c1b0 0x562854f68b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:22.577 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.575+0000 7ffa8d400640 1 -- 172.21.15.134:0/1177670419 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562854f69280 con 0x562854b1b4d0 2024-07-26T11:24:22.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.575+0000 7ffa8de00640 1 --2- 172.21.15.134:0/1177670419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b132f0 0x562854f6be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:24:22.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.578+0000 7ffa8d400640 1 --2- 172.21.15.134:0/1177670419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 0x562854f6c3f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x562854df8180 tx=0x7ffa800117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=903987d92127af35 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.578+0000 7ffa73e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:24:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.578+0000 7ffa73e00640 1 -- 172.21.15.134:0/1177670419 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa80005020 con 0x562854b1b4d0 2024-07-26T11:24:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.578+0000 7ffa73e00640 1 -- 172.21.15.134:0/1177670419 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9599+0+0 (secure 0 0 0) 0x7ffa800051c0 con 0x562854b1b4d0 2024-07-26T11:24:22.580 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.579+0000 7ffa9c21f740 1 -- 172.21.15.134:0/1177670419 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa88009900 con 0x562854b1b4d0 2024-07-26T11:24:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.579+0000 7ffa73e00640 1 -- 172.21.15.134:0/1177670419 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ffa800053c0 con 0x562854b1b4d0 2024-07-26T11:24:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.579+0000 7ffa9c21f740 10 client.26689 fetch_fsmap learned FSMap version 32 2024-07-26T11:24:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.579+0000 7ffa9c21f740 1 -- 172.21.15.134:0/1177670419 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x562854efb750 con 0x562854b1b4d0 2024-07-26T11:24:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.582+0000 7ffa73e00640 1 -- 172.21.15.134:0/1177670419 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa88009900 con 0x562854b1b4d0 2024-07-26T11:24:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.582+0000 7ffa73e00640 1 -- 172.21.15.134:0/1177670419 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7ffa8001b070 con 0x562854b1b4d0 2024-07-26T11:24:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.582+0000 7ffa9c21f740 10 client.26689 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:24:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.582+0000 7ffa9c21f740 10 client.26689 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:24:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.582+0000 7ffa9c21f740 20 client.26689 populate_metadata read hostname 'smithi134' 2024-07-26T11:24:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.582+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/1177670419 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562854ad01c0 0x562854efbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.582+0000 7ffa9c21f740 4 client.26689 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:24:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.582+0000 7ffa9c21f740 1 -- 172.21.15.134:0/1177670419 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562854e51c60 con 0x562854ad01c0 2024-07-26T11:24:22.585 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.583+0000 7ffa8de00640 1 --2- 172.21.15.134:0/1177670419 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562854ad01c0 0x562854efbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.584+0000 7ffa73e00640 10 client.26689 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:24:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:22.584+0000 7ffa8de00640 1 --2- 172.21.15.134:0/1177670419 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562854ad01c0 0x562854efbf80 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7db20acf710117a8 server_cookie=4447294826b31372 in_seq=0 out_seq=0 2024-07-26T11:24:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.004+0000 7ffa73e00640 1 -- 172.21.15.134:0/1177670419 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562854e51c60 con 0x562854ad01c0 2024-07-26T11:24:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.005+0000 7ffa73e00640 10 client.26689 handle_command_reply: tid=0 2024-07-26T11:24:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.005+0000 7ffa9c21f740 1 client.26689 shutdown 2024-07-26T11:24:23.007 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.005+0000 7ffa9c21f740 1 -- 172.21.15.134:0/1177670419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 msgr2=0x562854f6c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.005+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/1177670419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 0x562854f6c3f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x562854df8180 tx=0x7ffa800117b0 comp rx=0 tx=0).stop 2024-07-26T11:24:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.005+0000 7ffa9c21f740 1 -- 172.21.15.134:0/1177670419 shutdown_connections 2024-07-26T11:24:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.006+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/1177670419 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x562854ad01c0 0x562854efbf80 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.006+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/1177670419 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562854b1b4d0 0x562854f6c3f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.006+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/1177670419 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x562854b3c1b0 0x562854f68b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.006+0000 7ffa9c21f740 1 --2- 172.21.15.134:0/1177670419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562854b132f0 0x562854f6be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.006+0000 7ffa9c21f740 1 -- 172.21.15.134:0/1177670419 >> 172.21.15.134:0/1177670419 conn(0x562854b30300 msgr2=0x562854ee0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:23.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.006+0000 7ffa73e00640 0 client.26689 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:24:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.006+0000 7ffa9c21f740 1 -- 172.21.15.134:0/1177670419 shutdown_connections 2024-07-26T11:24:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.006+0000 7ffa9c21f740 1 -- 172.21.15.134:0/1177670419 wait complete. 2024-07-26T11:24:23.009 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.007+0000 7ffa9c21f740 20 client.26689 trim_cache size 0 max 16384 2024-07-26T11:24:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:24:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:24:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:24:23.010 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:24:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.009+0000 7ffa95200640 1 -- 172.21.15.134:0/3321693444 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ffa740b3f10 msgr2=0x7ffa740b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.009+0000 7ffa95200640 1 --2- 172.21.15.134:0/3321693444 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ffa740b3f10 0x7ffa740b6420 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ffa88009fd0 tx=0x7ffa8801bed0 comp rx=0 tx=0).stop 2024-07-26T11:24:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.009+0000 7ffa95200640 1 -- 172.21.15.134:0/3321693444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa90109fd0 msgr2=0x7ffa901a49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:23.011 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.009+0000 7ffa95200640 1 --2- 172.21.15.134:0/3321693444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa90109fd0 0x7ffa901a49c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ffa78009850 tx=0x7ffa78030650 comp rx=0 tx=0).stop 2024-07-26T11:24:23.013 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.011+0000 7ffa95200640 1 -- 172.21.15.134:0/3321693444 shutdown_connections 2024-07-26T11:24:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.011+0000 7ffa95200640 1 --2- 172.21.15.134:0/3321693444 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ffa740b3f10 0x7ffa740b6420 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.011+0000 7ffa95200640 1 --2- 172.21.15.134:0/3321693444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa90115fd0 0x7ffa901a6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.011+0000 7ffa95200640 1 --2- 172.21.15.134:0/3321693444 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ffa9010a920 0x7ffa901a4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.011+0000 7ffa95200640 1 --2- 172.21.15.134:0/3321693444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffa90109fd0 0x7ffa901a49c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.011+0000 7ffa95200640 1 -- 172.21.15.134:0/3321693444 >> 172.21.15.134:0/3321693444 conn(0x7ffa90070800 msgr2=0x7ffa90117c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.012+0000 7ffa95200640 1 -- 172.21.15.134:0/3321693444 shutdown_connections 2024-07-26T11:24:23.014 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:23.012+0000 7ffa95200640 1 -- 172.21.15.134:0/3321693444 wait complete. 2024-07-26T11:24:23.347 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-26T11:24:23.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:23 smithi134 ceph-mon[31966]: pgmap v735: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 55 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 39 KiB/s rd, 136 MiB/s wr, 1.25k op/s 2024-07-26T11:24:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:23 smithi189 ceph-mon[36790]: pgmap v735: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 55 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 39 KiB/s rd, 136 MiB/s wr, 1.25k op/s 2024-07-26T11:24:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:23 smithi193 ceph-mon[37473]: pgmap v735: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 55 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 39 KiB/s rd, 136 MiB/s wr, 1.25k op/s 2024-07-26T11:24:24.313 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:24:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:25 smithi134 ceph-mon[31966]: pgmap v736: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 55 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 16 KiB/s rd, 152 MiB/s wr, 1.41k op/s 2024-07-26T11:24:25.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:25 smithi189 ceph-mon[36790]: pgmap v736: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 55 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 16 KiB/s rd, 152 MiB/s wr, 1.41k op/s 2024-07-26T11:24:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:25 smithi193 ceph-mon[37473]: pgmap v736: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 55 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 16 KiB/s rd, 152 MiB/s wr, 1.41k op/s 2024-07-26T11:24:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:27 smithi189 ceph-mon[36790]: pgmap v737: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 55 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 16 KiB/s rd, 167 MiB/s wr, 1.55k op/s 2024-07-26T11:24:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:27 smithi193 ceph-mon[37473]: pgmap v737: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 55 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 16 KiB/s rd, 167 MiB/s wr, 1.55k op/s 2024-07-26T11:24:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:27 smithi134 ceph-mon[31966]: pgmap v737: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 55 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 16 KiB/s rd, 167 MiB/s wr, 1.55k op/s 2024-07-26T11:24:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:29 smithi189 ceph-mon[36790]: pgmap v738: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 55 GiB data, 116 GiB used, 957 GiB / 1.0 TiB avail; 110 KiB/s rd, 196 MiB/s wr, 1.84k op/s 2024-07-26T11:24:29.978 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:29 smithi134 ceph-mon[31966]: pgmap v738: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 55 GiB data, 116 GiB used, 957 GiB / 1.0 TiB avail; 110 KiB/s rd, 196 MiB/s wr, 1.84k op/s 2024-07-26T11:24:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:29 smithi193 ceph-mon[37473]: pgmap v738: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 55 GiB data, 116 GiB used, 957 GiB / 1.0 TiB avail; 110 KiB/s rd, 196 MiB/s wr, 1.84k op/s 2024-07-26T11:24:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:31 smithi189 ceph-mon[36790]: pgmap v739: 129 pgs: 129 active+clean; 55 GiB data, 116 GiB used, 957 GiB / 1.0 TiB avail; 102 KiB/s rd, 214 MiB/s wr, 2.11k op/s 2024-07-26T11:24:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:31 smithi193 ceph-mon[37473]: pgmap v739: 129 pgs: 129 active+clean; 55 GiB data, 116 GiB used, 957 GiB / 1.0 TiB avail; 102 KiB/s rd, 214 MiB/s wr, 2.11k op/s 2024-07-26T11:24:32.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:31 smithi134 ceph-mon[31966]: pgmap v739: 129 pgs: 129 active+clean; 55 GiB data, 116 GiB used, 957 GiB / 1.0 TiB avail; 102 KiB/s rd, 214 MiB/s wr, 2.11k op/s 2024-07-26T11:24:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:33 smithi193 ceph-mon[37473]: pgmap v740: 129 pgs: 129 active+clean; 55 GiB data, 116 GiB used, 957 GiB / 1.0 TiB avail; 327 KiB/s rd, 162 MiB/s wr, 1.66k op/s 2024-07-26T11:24:34.177 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:33 smithi134 ceph-mon[31966]: pgmap v740: 129 pgs: 129 active+clean; 55 GiB data, 116 GiB used, 957 GiB / 1.0 TiB avail; 327 KiB/s rd, 162 MiB/s wr, 1.66k op/s 2024-07-26T11:24:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:33 smithi189 ceph-mon[36790]: pgmap v740: 129 pgs: 129 active+clean; 55 GiB data, 116 GiB used, 957 GiB / 1.0 TiB avail; 327 KiB/s rd, 162 MiB/s wr, 1.66k op/s 2024-07-26T11:24:34.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:34 smithi134 ceph-mon[31966]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T11:24:34.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:34 smithi134 ceph-mon[31966]: pgmap v742: 129 pgs: 129 active+clean; 55 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 383 KiB/s rd, 162 MiB/s wr, 1.80k op/s 2024-07-26T11:24:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:34 smithi193 ceph-mon[37473]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T11:24:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:34 smithi193 ceph-mon[37473]: pgmap v742: 129 pgs: 129 active+clean; 55 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 383 KiB/s rd, 162 MiB/s wr, 1.80k op/s 2024-07-26T11:24:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:34 smithi189 ceph-mon[36790]: osdmap e94: 12 total, 12 up, 12 in 2024-07-26T11:24:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:34 smithi189 ceph-mon[36790]: pgmap v742: 129 pgs: 129 active+clean; 55 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 383 KiB/s rd, 162 MiB/s wr, 1.80k op/s 2024-07-26T11:24:36.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.379+0000 7fe881200640 1 -- 172.21.15.134:0/1146513244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe87c109930 msgr2=0x7fe87c109d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:36.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.379+0000 7fe881200640 1 --2- 172.21.15.134:0/1146513244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe87c109930 0x7fe87c109d40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe864009860 tx=0x7fe86402a380 comp rx=0 tx=0).stop 2024-07-26T11:24:36.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.380+0000 7fe881200640 1 -- 172.21.15.134:0/1146513244 shutdown_connections 2024-07-26T11:24:36.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.380+0000 7fe881200640 1 --2- 172.21.15.134:0/1146513244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe87c10fbb0 0x7fe87c1163e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.380+0000 7fe881200640 1 --2- 172.21.15.134:0/1146513244 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe87c10f260 0x7fe87c10f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.380+0000 7fe881200640 1 --2- 172.21.15.134:0/1146513244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe87c109930 0x7fe87c109d40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.380+0000 7fe881200640 1 -- 172.21.15.134:0/1146513244 >> 172.21.15.134:0/1146513244 conn(0x7fe87c079010 msgr2=0x7fe87c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:36.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.380+0000 7fe881200640 1 -- 172.21.15.134:0/1146513244 shutdown_connections 2024-07-26T11:24:36.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.380+0000 7fe881200640 1 -- 172.21.15.134:0/1146513244 wait complete. 2024-07-26T11:24:36.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.381+0000 7fe881200640 1 Processor -- start 2024-07-26T11:24:36.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.382+0000 7fe881200640 1 -- start start 2024-07-26T11:24:36.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.382+0000 7fe881200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe87c109930 0x7fe87c1a0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.382+0000 7fe881200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe87c10f260 0x7fe87c1a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.382+0000 7fe881200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe87c10fbb0 0x7fe87c1a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.383+0000 7fe87b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe87c109930 0x7fe87c1a0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.383+0000 7fe87b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe87c109930 0x7fe87c1a0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:33324/0 (socket says 172.21.15.134:33324) 2024-07-26T11:24:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.383+0000 7fe87be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe87c10fbb0 0x7fe87c1a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.383+0000 7fe87be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe87c10fbb0 0x7fe87c1a5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:50382/0 (socket says 172.21.15.134:50382) 2024-07-26T11:24:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.383+0000 7fe87b400640 1 -- 172.21.15.134:0/1000170923 learned_addr learned my addr 172.21.15.134:0/1000170923 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.383+0000 7fe873e00640 1 --2- 172.21.15.134:0/1000170923 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe87c10f260 0x7fe87c1a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.383+0000 7fe881200640 1 -- 172.21.15.134:0/1000170923 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c119900 con 0x7fe87c10f260 2024-07-26T11:24:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.383+0000 7fe881200640 1 -- 172.21.15.134:0/1000170923 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c119780 con 0x7fe87c10fbb0 2024-07-26T11:24:36.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.383+0000 7fe881200640 1 -- 172.21.15.134:0/1000170923 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c119a80 con 0x7fe87c109930 2024-07-26T11:24:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.384+0000 7fe87b400640 1 -- 172.21.15.134:0/1000170923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe87c10fbb0 msgr2=0x7fe87c1a5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.384+0000 7fe87b400640 1 --2- 172.21.15.134:0/1000170923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe87c10fbb0 0x7fe87c1a5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.384+0000 7fe87b400640 1 -- 172.21.15.134:0/1000170923 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe87c10f260 msgr2=0x7fe87c1a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.384+0000 7fe87b400640 1 --2- 172.21.15.134:0/1000170923 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe87c10f260 0x7fe87c1a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.384+0000 7fe87b400640 1 -- 172.21.15.134:0/1000170923 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87c1a62b0 con 0x7fe87c109930 2024-07-26T11:24:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.384+0000 7fe87b400640 1 --2- 172.21.15.134:0/1000170923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe87c109930 0x7fe87c1a0380 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe864035040 tx=0x7fe864032cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:36.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.385+0000 7fe878c00640 1 -- 172.21.15.134:0/1000170923 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe864037050 con 0x7fe87c109930 2024-07-26T11:24:36.390 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.385+0000 7fe881200640 1 -- 172.21.15.134:0/1000170923 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe87c1a6540 con 0x7fe87c109930 2024-07-26T11:24:36.390 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.385+0000 7fe881200640 1 -- 172.21.15.134:0/1000170923 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe87c1acd30 con 0x7fe87c109930 2024-07-26T11:24:36.392 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.388+0000 7fe878c00640 1 -- 172.21.15.134:0/1000170923 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe864043070 con 0x7fe87c109930 2024-07-26T11:24:36.392 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.388+0000 7fe878c00640 1 -- 172.21.15.134:0/1000170923 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe864003c90 con 0x7fe87c109930 2024-07-26T11:24:36.392 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.390+0000 7fe889980740 1 Processor -- start 2024-07-26T11:24:36.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.391+0000 7fe889980740 1 -- start start 2024-07-26T11:24:36.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.391+0000 7fe889980740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee685c1b0 0x55eee6bffc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.391+0000 7fe889980740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee68332f0 0x55eee6c00180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.391+0000 7fe889980740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee683b4d0 0x55eee6c006c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.391+0000 7fe889980740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55eee6b74270 con 0x55eee683b4d0 2024-07-26T11:24:36.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.391+0000 7fe889980740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55eee6b63770 con 0x55eee685c1b0 2024-07-26T11:24:36.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.391+0000 7fe889980740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55eee6b677b0 con 0x55eee68332f0 2024-07-26T11:24:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.391+0000 7fe87be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee685c1b0 0x55eee6bffc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:36.395 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.391+0000 7fe87be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee685c1b0 0x55eee6bffc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:50384/0 (socket says 172.21.15.134:50384) 2024-07-26T11:24:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.391+0000 7fe87be00640 1 -- 172.21.15.134:0/1285661025 learned_addr learned my addr 172.21.15.134:0/1285661025 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.392+0000 7fe87be00640 1 -- 172.21.15.134:0/1285661025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee68332f0 msgr2=0x55eee6c00180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:24:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.392+0000 7fe87be00640 1 --2- 172.21.15.134:0/1285661025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee68332f0 0x55eee6c00180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.392+0000 7fe87be00640 1 -- 172.21.15.134:0/1285661025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee683b4d0 msgr2=0x55eee6c006c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.392+0000 7fe87be00640 1 --2- 172.21.15.134:0/1285661025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee683b4d0 0x55eee6c006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.392+0000 7fe87be00640 1 -- 172.21.15.134:0/1285661025 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eee68c7790 con 0x55eee685c1b0 2024-07-26T11:24:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.392+0000 7fe87be00640 1 --2- 172.21.15.134:0/1285661025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee685c1b0 0x55eee6bffc40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe87400e5b0 tx=0x7fe8740095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1c892cb471bf900 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.393+0000 7fe870c00640 1 -- 172.21.15.134:0/1285661025 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874003970 con 0x55eee685c1b0 2024-07-26T11:24:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.393+0000 7fe870c00640 1 -- 172.21.15.134:0/1285661025 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe87400d8b0 con 0x55eee685c1b0 2024-07-26T11:24:36.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.393+0000 7fe870c00640 1 -- 172.21.15.134:0/1285661025 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8740044b0 con 0x55eee685c1b0 2024-07-26T11:24:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.396+0000 7fe889980740 1 -- 172.21.15.134:0/1285661025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee685c1b0 msgr2=0x55eee6bffc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.396+0000 7fe889980740 1 --2- 172.21.15.134:0/1285661025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee685c1b0 0x55eee6bffc40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe87400e5b0 tx=0x7fe8740095c0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.396+0000 7fe878c00640 1 -- 172.21.15.134:0/1000170923 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fe86403e450 con 0x7fe87c109930 2024-07-26T11:24:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.398+0000 7fe889980740 1 -- 172.21.15.134:0/1285661025 shutdown_connections 2024-07-26T11:24:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.398+0000 7fe889980740 1 --2- 172.21.15.134:0/1285661025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee68332f0 0x55eee6c00180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.398+0000 7fe889980740 1 --2- 172.21.15.134:0/1285661025 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee683b4d0 0x55eee6c006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.398+0000 7fe889980740 1 --2- 172.21.15.134:0/1285661025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee685c1b0 0x55eee6bffc40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.398+0000 7fe889980740 1 -- 172.21.15.134:0/1285661025 >> 172.21.15.134:0/1285661025 conn(0x55eee6850300 msgr2=0x55eee6bfdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:36.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.399+0000 7fe889980740 1 -- 172.21.15.134:0/1285661025 shutdown_connections 2024-07-26T11:24:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.399+0000 7fe889980740 1 -- 172.21.15.134:0/1285661025 wait complete. 2024-07-26T11:24:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.399+0000 7fe878c00640 1 --2- 172.21.15.134:0/1000170923 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe8600b3d80 0x7fe8600b6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.399+0000 7fe878c00640 1 -- 172.21.15.134:0/1000170923 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fe864104190 con 0x7fe87c109930 2024-07-26T11:24:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.400+0000 7fe889980740 1 -- start start 2024-07-26T11:24:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.400+0000 7fe889980740 1 Processor -- start 2024-07-26T11:24:36.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.400+0000 7fe873e00640 1 --2- 172.21.15.134:0/1000170923 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe8600b3d80 0x7fe8600b6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:36.403 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.401+0000 7fe889980740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee68332f0 0x55eee6c0b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.402+0000 7fe889980740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee683b4d0 0x55eee6c0bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.402+0000 7fe873e00640 1 --2- 172.21.15.134:0/1000170923 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe8600b3d80 0x7fe8600b6290 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe87c06a470 tx=0x7fe86c0094c0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.402+0000 7fe87b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee68332f0 0x55eee6c0b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.402+0000 7fe87b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee68332f0 0x55eee6c0b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:50600/0 (socket says 172.21.15.134:50600) 2024-07-26T11:24:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.402+0000 7fe87b400640 1 -- 172.21.15.134:0/3210680770 learned_addr learned my addr 172.21.15.134:0/3210680770 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.402+0000 7fe889980740 1 --2- 172.21.15.134:0/3210680770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee685c1b0 0x55eee6c11710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.403+0000 7fe889980740 1 -- 172.21.15.134:0/3210680770 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55eee6a9dab0 con 0x55eee68332f0 2024-07-26T11:24:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.403+0000 7fe889980740 1 -- 172.21.15.134:0/3210680770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55eee6b00d50 con 0x55eee685c1b0 2024-07-26T11:24:36.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.403+0000 7fe889980740 1 -- 172.21.15.134:0/3210680770 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55eee6c0e3c0 con 0x55eee683b4d0 2024-07-26T11:24:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.403+0000 7fe889980740 10 client.0 _ll_register_callbacks cb 0x55eee6873a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:24:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.404+0000 7fe87b400640 1 -- 172.21.15.134:0/3210680770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee683b4d0 msgr2=0x55eee6c0bf00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:24:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.404+0000 7fe87b400640 1 --2- 172.21.15.134:0/3210680770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee683b4d0 0x55eee6c0bf00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.404+0000 7fe87b400640 1 -- 172.21.15.134:0/3210680770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee685c1b0 msgr2=0x55eee6c11710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.404+0000 7fe87be00640 1 --2- 172.21.15.134:0/3210680770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee685c1b0 0x55eee6c11710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.404+0000 7fe87b400640 1 --2- 172.21.15.134:0/3210680770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee685c1b0 0x55eee6c11710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.404+0000 7fe87b400640 1 -- 172.21.15.134:0/3210680770 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eee6940620 con 0x55eee68332f0 2024-07-26T11:24:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.404+0000 7fe87be00640 1 --2- 172.21.15.134:0/3210680770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee685c1b0 0x55eee6c11710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:24:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.404+0000 7fe87b400640 1 --2- 172.21.15.134:0/3210680770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee68332f0 0x55eee6c0b920 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe864009980 tx=0x7fe8640ca6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db58ec1ce88a88de server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.405+0000 7fe869600640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:24:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.406+0000 7fe869600640 1 -- 172.21.15.134:0/3210680770 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe864101930 con 0x55eee68332f0 2024-07-26T11:24:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.406+0000 7fe869600640 1 -- 172.21.15.134:0/3210680770 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fe8640d3690 con 0x55eee68332f0 2024-07-26T11:24:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.407+0000 7fe889980740 1 -- 172.21.15.134:0/3210680770 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe87400d8b0 con 0x55eee68332f0 2024-07-26T11:24:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.407+0000 7fe869600640 1 -- 172.21.15.134:0/3210680770 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe8640e2070 con 0x55eee68332f0 2024-07-26T11:24:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.407+0000 7fe889980740 10 client.16107 fetch_fsmap learned FSMap version 32 2024-07-26T11:24:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.407+0000 7fe889980740 1 -- 172.21.15.134:0/3210680770 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55eee6c1a570 con 0x55eee68332f0 2024-07-26T11:24:36.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.412+0000 7fe869600640 1 -- 172.21.15.134:0/3210680770 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe87400d8b0 con 0x55eee68332f0 2024-07-26T11:24:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.415+0000 7fe869600640 1 -- 172.21.15.134:0/3210680770 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fe8640c68b0 con 0x55eee68332f0 2024-07-26T11:24:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.416+0000 7fe889980740 10 client.16107 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:24:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.416+0000 7fe889980740 10 client.16107 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:24:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.416+0000 7fe889980740 20 client.16107 populate_metadata read hostname 'smithi134' 2024-07-26T11:24:36.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.416+0000 7fe889980740 1 --2- 172.21.15.134:0/3210680770 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55eee681ad80 0x55eee6c1daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.417+0000 7fe889980740 4 client.16107 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:24:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.417+0000 7fe889980740 1 -- 172.21.15.134:0/3210680770 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eee6b761c0 con 0x55eee681ad80 2024-07-26T11:24:36.419 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.417+0000 7fe873e00640 1 --2- 172.21.15.134:0/3210680770 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55eee681ad80 0x55eee6c1daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.418+0000 7fe873e00640 1 --2- 172.21.15.134:0/3210680770 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55eee681ad80 0x55eee6c1daf0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3561711a4b0bfb37 server_cookie=678d109f1e49dd6c in_seq=0 out_seq=0 2024-07-26T11:24:36.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.418+0000 7fe869600640 10 client.16107 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:24:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.424+0000 7fe869600640 1 -- 172.21.15.134:0/3210680770 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55eee6b761c0 con 0x55eee681ad80 2024-07-26T11:24:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.425+0000 7fe869600640 10 client.16107 handle_command_reply: tid=0 2024-07-26T11:24:36.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.434+0000 7fe889980740 1 client.16107 shutdown 2024-07-26T11:24:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.436+0000 7fe889980740 1 -- 172.21.15.134:0/3210680770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee68332f0 msgr2=0x55eee6c0b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.436+0000 7fe889980740 1 --2- 172.21.15.134:0/3210680770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee68332f0 0x55eee6c0b920 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe864009980 tx=0x7fe8640ca6b0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.438+0000 7fe889980740 1 -- 172.21.15.134:0/3210680770 shutdown_connections 2024-07-26T11:24:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.438+0000 7fe889980740 1 --2- 172.21.15.134:0/3210680770 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55eee681ad80 0x55eee6c1daf0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.438+0000 7fe889980740 1 --2- 172.21.15.134:0/3210680770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee683b4d0 0x55eee6c0bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.438+0000 7fe889980740 1 --2- 172.21.15.134:0/3210680770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee685c1b0 0x55eee6c11710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.439+0000 7fe889980740 1 --2- 172.21.15.134:0/3210680770 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee68332f0 0x55eee6c0b920 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.439+0000 7fe869600640 0 client.16107 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:24:36.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.439+0000 7fe889980740 1 -- 172.21.15.134:0/3210680770 >> 172.21.15.134:0/3210680770 conn(0x55eee6850300 msgr2=0x55eee6c00c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:36.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.442+0000 7fe889980740 1 -- 172.21.15.134:0/3210680770 shutdown_connections 2024-07-26T11:24:36.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.442+0000 7fe889980740 1 -- 172.21.15.134:0/3210680770 wait complete. 2024-07-26T11:24:36.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.442+0000 7fe889980740 20 client.16107 trim_cache size 0 max 16384 2024-07-26T11:24:36.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.497+0000 7fe889980740 1 Processor -- start 2024-07-26T11:24:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.497+0000 7fe889980740 1 -- start start 2024-07-26T11:24:36.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.498+0000 7fe889980740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee681ad80 0x55eee6c184e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.498+0000 7fe889980740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee68332f0 0x55eee6c09fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.498+0000 7fe889980740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee683b4d0 0x55eee6c0a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.498+0000 7fe889980740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe858010050 con 0x55eee681ad80 2024-07-26T11:24:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.498+0000 7fe889980740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe858011b90 con 0x55eee683b4d0 2024-07-26T11:24:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.500+0000 7fe889980740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe858010720 con 0x55eee68332f0 2024-07-26T11:24:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.500+0000 7fe873e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee68332f0 0x55eee6c09fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.500+0000 7fe873e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee68332f0 0x55eee6c09fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:33356/0 (socket says 172.21.15.134:33356) 2024-07-26T11:24:36.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.500+0000 7fe873e00640 1 -- 172.21.15.134:0/1318519552 learned_addr learned my addr 172.21.15.134:0/1318519552 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.500+0000 7fe873e00640 1 -- 172.21.15.134:0/1318519552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee683b4d0 msgr2=0x55eee6c0a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.500+0000 7fe87be00640 1 --2- 172.21.15.134:0/1318519552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee683b4d0 0x55eee6c0a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.501+0000 7fe87b400640 1 --2- 172.21.15.134:0/1318519552 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee681ad80 0x55eee6c184e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:36.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.501+0000 7fe873e00640 1 --2- 172.21.15.134:0/1318519552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee683b4d0 0x55eee6c0a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.501+0000 7fe873e00640 1 -- 172.21.15.134:0/1318519552 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee681ad80 msgr2=0x55eee6c184e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.502+0000 7fe873e00640 1 --2- 172.21.15.134:0/1318519552 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee681ad80 0x55eee6c184e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.502+0000 7fe873e00640 1 -- 172.21.15.134:0/1318519552 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eee6c0c9f0 con 0x55eee68332f0 2024-07-26T11:24:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.502+0000 7fe87be00640 1 --2- 172.21.15.134:0/1318519552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee683b4d0 0x55eee6c0a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:24:36.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.502+0000 7fe87b400640 1 --2- 172.21.15.134:0/1318519552 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee681ad80 0x55eee6c184e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:24:36.505 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.503+0000 7fe873e00640 1 --2- 172.21.15.134:0/1318519552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee68332f0 0x55eee6c09fb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00eaf0 tx=0x7fe86c011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62c6f505dce651a7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:36.506 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.505+0000 7fe86be00640 1 -- 172.21.15.134:0/1318519552 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c02e070 con 0x55eee68332f0 2024-07-26T11:24:36.507 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.505+0000 7fe86be00640 1 -- 172.21.15.134:0/1318519552 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe86c021420 con 0x55eee68332f0 2024-07-26T11:24:36.507 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.505+0000 7fe86be00640 1 -- 172.21.15.134:0/1318519552 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c01b040 con 0x55eee68332f0 2024-07-26T11:24:36.509 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.508+0000 7fe889980740 1 -- 172.21.15.134:0/1318519552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee68332f0 msgr2=0x55eee6c09fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:36.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.508+0000 7fe889980740 1 --2- 172.21.15.134:0/1318519552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee68332f0 0x55eee6c09fb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00eaf0 tx=0x7fe86c011000 comp rx=0 tx=0).stop 2024-07-26T11:24:36.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.508+0000 7fe889980740 1 -- 172.21.15.134:0/1318519552 shutdown_connections 2024-07-26T11:24:36.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.508+0000 7fe889980740 1 --2- 172.21.15.134:0/1318519552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee68332f0 0x55eee6c09fb0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.508+0000 7fe889980740 1 --2- 172.21.15.134:0/1318519552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee683b4d0 0x55eee6c0a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.508+0000 7fe889980740 1 --2- 172.21.15.134:0/1318519552 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee681ad80 0x55eee6c184e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.508+0000 7fe889980740 1 -- 172.21.15.134:0/1318519552 >> 172.21.15.134:0/1318519552 conn(0x55eee6850300 msgr2=0x55eee6bfc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:36.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.509+0000 7fe889980740 1 -- 172.21.15.134:0/1318519552 shutdown_connections 2024-07-26T11:24:36.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.509+0000 7fe889980740 1 -- 172.21.15.134:0/1318519552 wait complete. 2024-07-26T11:24:36.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.509+0000 7fe889980740 1 -- start start 2024-07-26T11:24:36.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.510+0000 7fe889980740 1 Processor -- start 2024-07-26T11:24:36.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.512+0000 7fe889980740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee681ad80 0x55eee6c125d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.514 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.512+0000 7fe889980740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee68332f0 0x55eee6c12b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.512+0000 7fe889980740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee683b4d0 0x55eee6c17410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.513+0000 7fe87b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee681ad80 0x55eee6c125d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:36.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.513+0000 7fe87b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee681ad80 0x55eee6c125d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:50402/0 (socket says 172.21.15.134:50402) 2024-07-26T11:24:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.513+0000 7fe889980740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe8580114c0 con 0x55eee68332f0 2024-07-26T11:24:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.513+0000 7fe889980740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe858010df0 con 0x55eee681ad80 2024-07-26T11:24:36.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.513+0000 7fe889980740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe85800e050 con 0x55eee683b4d0 2024-07-26T11:24:36.517 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.513+0000 7fe889980740 10 client.0 _ll_register_callbacks cb 0x7fe86402a8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:24:36.517 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.513+0000 7fe87be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee683b4d0 0x55eee6c17410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:36.517 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.513+0000 7fe87be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee683b4d0 0x55eee6c17410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:33368/0 (socket says 172.21.15.134:33368) 2024-07-26T11:24:36.517 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.513+0000 7fe87be00640 1 -- 172.21.15.134:0/2858164077 learned_addr learned my addr 172.21.15.134:0/2858164077 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:36.518 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.515+0000 7fe87be00640 1 -- 172.21.15.134:0/2858164077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee681ad80 msgr2=0x55eee6c125d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:36.519 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.515+0000 7fe87be00640 1 --2- 172.21.15.134:0/2858164077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee681ad80 0x55eee6c125d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.519 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.515+0000 7fe87be00640 1 -- 172.21.15.134:0/2858164077 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee68332f0 msgr2=0x55eee6c12b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:36.519 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.515+0000 7fe873e00640 1 --2- 172.21.15.134:0/2858164077 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee68332f0 0x55eee6c12b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:36.519 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.516+0000 7fe87be00640 1 --2- 172.21.15.134:0/2858164077 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee68332f0 0x55eee6c12b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:36.519 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.516+0000 7fe87be00640 1 -- 172.21.15.134:0/2858164077 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eee6c17950 con 0x55eee683b4d0 2024-07-26T11:24:36.519 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.517+0000 7fe873e00640 1 --2- 172.21.15.134:0/2858164077 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee68332f0 0x55eee6c12b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:24:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.519+0000 7fe87be00640 1 --2- 172.21.15.134:0/2858164077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee683b4d0 0x55eee6c17410 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe8740097f0 tx=0x7fe874004bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e8a5bf7210fa39e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:36.521 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.519+0000 7fe869600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:24:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.520+0000 7fe869600640 1 -- 172.21.15.134:0/2858164077 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe87400d8b0 con 0x55eee683b4d0 2024-07-26T11:24:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.520+0000 7fe869600640 1 -- 172.21.15.134:0/2858164077 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7fe87400da50 con 0x55eee683b4d0 2024-07-26T11:24:36.522 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.520+0000 7fe889980740 1 -- 172.21.15.134:0/2858164077 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe86c021420 con 0x55eee683b4d0 2024-07-26T11:24:36.525 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.522+0000 7fe869600640 1 -- 172.21.15.134:0/2858164077 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe87400dc50 con 0x55eee683b4d0 2024-07-26T11:24:36.525 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.522+0000 7fe889980740 10 client.26701 fetch_fsmap learned FSMap version 32 2024-07-26T11:24:36.525 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.522+0000 7fe889980740 1 -- 172.21.15.134:0/2858164077 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55eee6c972f0 con 0x55eee683b4d0 2024-07-26T11:24:36.526 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.524+0000 7fe869600640 1 -- 172.21.15.134:0/2858164077 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe86c021420 con 0x55eee683b4d0 2024-07-26T11:24:36.526 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.525+0000 7fe869600640 1 -- 172.21.15.134:0/2858164077 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fe87400b040 con 0x55eee683b4d0 2024-07-26T11:24:36.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.526+0000 7fe889980740 10 client.26701 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:24:36.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.526+0000 7fe889980740 10 client.26701 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:24:36.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.526+0000 7fe889980740 20 client.26701 populate_metadata read hostname 'smithi134' 2024-07-26T11:24:36.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.526+0000 7fe889980740 1 --2- 172.21.15.134:0/2858164077 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55eee685c1b0 0x55eee6c99750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:36.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.526+0000 7fe889980740 4 client.26701 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:24:36.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.526+0000 7fe889980740 1 -- 172.21.15.134:0/2858164077 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55eee6b71c60 con 0x55eee685c1b0 2024-07-26T11:24:36.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.527+0000 7fe87b400640 1 --2- 172.21.15.134:0/2858164077 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55eee685c1b0 0x55eee6c99750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.528+0000 7fe87b400640 1 --2- 172.21.15.134:0/2858164077 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55eee685c1b0 0x55eee6c99750 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8760a14894790edd server_cookie=b89a8dc8f33da1a6 in_seq=0 out_seq=0 2024-07-26T11:24:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:36.528+0000 7fe869600640 10 client.26701 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:24:37.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.661+0000 7fe869600640 1 -- 172.21.15.134:0/2858164077 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55eee6b71c60 con 0x55eee685c1b0 2024-07-26T11:24:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.661+0000 7fe869600640 10 client.26701 handle_command_reply: tid=0 2024-07-26T11:24:37.663 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.661+0000 7fe889980740 1 client.26701 shutdown 2024-07-26T11:24:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.662+0000 7fe889980740 1 -- 172.21.15.134:0/2858164077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee683b4d0 msgr2=0x55eee6c17410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.662+0000 7fe889980740 1 --2- 172.21.15.134:0/2858164077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee683b4d0 0x55eee6c17410 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe8740097f0 tx=0x7fe874004bb0 comp rx=0 tx=0).stop 2024-07-26T11:24:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.662+0000 7fe889980740 1 -- 172.21.15.134:0/2858164077 shutdown_connections 2024-07-26T11:24:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.663+0000 7fe889980740 1 --2- 172.21.15.134:0/2858164077 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55eee685c1b0 0x55eee6c99750 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:37.664 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.663+0000 7fe889980740 1 --2- 172.21.15.134:0/2858164077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55eee683b4d0 0x55eee6c17410 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.663+0000 7fe869600640 0 client.26701 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:24:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.663+0000 7fe889980740 1 --2- 172.21.15.134:0/2858164077 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55eee68332f0 0x55eee6c12b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.663+0000 7fe889980740 1 --2- 172.21.15.134:0/2858164077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55eee681ad80 0x55eee6c125d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:37.665 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.663+0000 7fe889980740 1 -- 172.21.15.134:0/2858164077 >> 172.21.15.134:0/2858164077 conn(0x55eee6850300 msgr2=0x55eee6bfc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.664+0000 7fe889980740 1 -- 172.21.15.134:0/2858164077 shutdown_connections 2024-07-26T11:24:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.664+0000 7fe889980740 1 -- 172.21.15.134:0/2858164077 wait complete. 2024-07-26T11:24:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.664+0000 7fe889980740 20 client.26701 trim_cache size 0 max 16384 2024-07-26T11:24:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:24:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:24:37.666 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:24:37.667 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:24:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.668+0000 7fe881200640 1 -- 172.21.15.134:0/1000170923 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe8600b3d80 msgr2=0x7fe8600b6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.668+0000 7fe881200640 1 --2- 172.21.15.134:0/1000170923 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe8600b3d80 0x7fe8600b6290 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe87c06a470 tx=0x7fe86c0094c0 comp rx=0 tx=0).stop 2024-07-26T11:24:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.668+0000 7fe881200640 1 -- 172.21.15.134:0/1000170923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe87c109930 msgr2=0x7fe87c1a0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:37.669 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.668+0000 7fe881200640 1 --2- 172.21.15.134:0/1000170923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe87c109930 0x7fe87c1a0380 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fe864035040 tx=0x7fe864032cd0 comp rx=0 tx=0).stop 2024-07-26T11:24:37.674 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.673+0000 7fe881200640 1 -- 172.21.15.134:0/1000170923 shutdown_connections 2024-07-26T11:24:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.674+0000 7fe881200640 1 --2- 172.21.15.134:0/1000170923 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe8600b3d80 0x7fe8600b6290 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.674+0000 7fe881200640 1 --2- 172.21.15.134:0/1000170923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe87c10fbb0 0x7fe87c1a5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.674+0000 7fe881200640 1 --2- 172.21.15.134:0/1000170923 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe87c10f260 0x7fe87c1a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.674+0000 7fe881200640 1 --2- 172.21.15.134:0/1000170923 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe87c109930 0x7fe87c1a0380 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:37.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.674+0000 7fe881200640 1 -- 172.21.15.134:0/1000170923 >> 172.21.15.134:0/1000170923 conn(0x7fe87c079010 msgr2=0x7fe87c0797e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:37.680 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.678+0000 7fe881200640 1 -- 172.21.15.134:0/1000170923 shutdown_connections 2024-07-26T11:24:37.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:37.680+0000 7fe881200640 1 -- 172.21.15.134:0/1000170923 wait complete. 2024-07-26T11:24:37.926 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:37 smithi134 ceph-mon[31966]: pgmap v743: 129 pgs: 129 active+clean; 56 GiB data, 117 GiB used, 955 GiB / 1.0 TiB avail; 383 KiB/s rd, 175 MiB/s wr, 1.85k op/s 2024-07-26T11:24:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:37 smithi189 ceph-mon[36790]: pgmap v743: 129 pgs: 129 active+clean; 56 GiB data, 117 GiB used, 955 GiB / 1.0 TiB avail; 383 KiB/s rd, 175 MiB/s wr, 1.85k op/s 2024-07-26T11:24:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:37 smithi193 ceph-mon[37473]: pgmap v743: 129 pgs: 129 active+clean; 56 GiB data, 117 GiB used, 955 GiB / 1.0 TiB avail; 383 KiB/s rd, 175 MiB/s wr, 1.85k op/s 2024-07-26T11:24:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:39 smithi193 ceph-mon[37473]: pgmap v744: 129 pgs: 129 active+clean; 56 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 466 KiB/s rd, 163 MiB/s wr, 1.65k op/s 2024-07-26T11:24:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:39 smithi134 ceph-mon[31966]: pgmap v744: 129 pgs: 129 active+clean; 56 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 466 KiB/s rd, 163 MiB/s wr, 1.65k op/s 2024-07-26T11:24:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:39 smithi189 ceph-mon[36790]: pgmap v744: 129 pgs: 129 active+clean; 56 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 466 KiB/s rd, 163 MiB/s wr, 1.65k op/s 2024-07-26T11:24:41.001 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:40 smithi134 ceph-mon[31966]: pgmap v745: 129 pgs: 129 active+clean; 57 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 467 KiB/s rd, 215 MiB/s wr, 1.94k op/s 2024-07-26T11:24:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:40 smithi193 ceph-mon[37473]: pgmap v745: 129 pgs: 129 active+clean; 57 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 467 KiB/s rd, 215 MiB/s wr, 1.94k op/s 2024-07-26T11:24:41.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:40 smithi189 ceph-mon[36790]: pgmap v745: 129 pgs: 129 active+clean; 57 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 467 KiB/s rd, 215 MiB/s wr, 1.94k op/s 2024-07-26T11:24:43.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:43 smithi134 ceph-mon[31966]: pgmap v746: 129 pgs: 129 active+clean; 57 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 214 KiB/s rd, 218 MiB/s wr, 1.93k op/s 2024-07-26T11:24:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:43 smithi189 ceph-mon[36790]: pgmap v746: 129 pgs: 129 active+clean; 57 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 214 KiB/s rd, 218 MiB/s wr, 1.93k op/s 2024-07-26T11:24:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:43 smithi193 ceph-mon[37473]: pgmap v746: 129 pgs: 129 active+clean; 57 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 214 KiB/s rd, 218 MiB/s wr, 1.93k op/s 2024-07-26T11:24:44.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.627+0000 7f8260a00640 1 -- 172.21.15.134:0/4160349374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10a080 msgr2=0x7f825c10a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:44.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.627+0000 7f8260a00640 1 --2- 172.21.15.134:0/4160349374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10a080 0x7f825c10a490 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f824c009900 tx=0x7f824c02b550 comp rx=0 tx=0).stop 2024-07-26T11:24:44.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.627+0000 7f8260a00640 1 -- 172.21.15.134:0/4160349374 shutdown_connections 2024-07-26T11:24:44.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.627+0000 7f8260a00640 1 --2- 172.21.15.134:0/4160349374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c110090 0x7f825c1168c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.627+0000 7f8260a00640 1 --2- 172.21.15.134:0/4160349374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f825c10f740 0x7f825c10fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.627+0000 7f8260a00640 1 --2- 172.21.15.134:0/4160349374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10a080 0x7f825c10a490 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.627+0000 7f8260a00640 1 -- 172.21.15.134:0/4160349374 >> 172.21.15.134:0/4160349374 conn(0x7f825c06e420 msgr2=0x7f825c06e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:44.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.628+0000 7f8260a00640 1 -- 172.21.15.134:0/4160349374 shutdown_connections 2024-07-26T11:24:44.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.628+0000 7f8260a00640 1 -- 172.21.15.134:0/4160349374 wait complete. 2024-07-26T11:24:44.638 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.636+0000 7f8260a00640 1 Processor -- start 2024-07-26T11:24:44.639 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.637+0000 7f8260a00640 1 -- start start 2024-07-26T11:24:44.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.639+0000 7f8260a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10a080 0x7f825c1a03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.639+0000 7f8260a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c10f740 0x7f825c1a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.640+0000 7f8260a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f825c110090 0x7f825c1a5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.640+0000 7f8259e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c10f740 0x7f825c1a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:44.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.640+0000 7f8259e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c10f740 0x7f825c1a0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56768/0 (socket says 172.21.15.134:56768) 2024-07-26T11:24:44.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.641+0000 7f8260a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f825c119990 con 0x7f825c110090 2024-07-26T11:24:44.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.644+0000 7f825a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10a080 0x7f825c1a03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:44.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.644+0000 7f825a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10a080 0x7f825c1a03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:43466/0 (socket says 172.21.15.134:43466) 2024-07-26T11:24:44.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.644+0000 7f825a800640 1 -- 172.21.15.134:0/301361147 learned_addr learned my addr 172.21.15.134:0/301361147 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:44.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.644+0000 7f8260a00640 1 -- 172.21.15.134:0/301361147 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f825c119810 con 0x7f825c10f740 2024-07-26T11:24:44.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.644+0000 7f8260a00640 1 -- 172.21.15.134:0/301361147 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f825c119b10 con 0x7f825c10a080 2024-07-26T11:24:44.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.645+0000 7f8259e00640 1 -- 172.21.15.134:0/301361147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10a080 msgr2=0x7f825c1a03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:44.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.645+0000 7f8259e00640 1 --2- 172.21.15.134:0/301361147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10a080 0x7f825c1a03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.646+0000 7f8259e00640 1 -- 172.21.15.134:0/301361147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f825c110090 msgr2=0x7f825c1a5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:44.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.646+0000 7f8259e00640 1 --2- 172.21.15.134:0/301361147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f825c110090 0x7f825c1a5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.646+0000 7f8259e00640 1 -- 172.21.15.134:0/301361147 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825c1a6330 con 0x7f825c10f740 2024-07-26T11:24:44.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.646+0000 7f825a800640 1 --2- 172.21.15.134:0/301361147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10a080 0x7f825c1a03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:24:44.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.651+0000 7f8259e00640 1 --2- 172.21.15.134:0/301361147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c10f740 0x7f825c1a0960 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f825c06c340 tx=0x7f824800b850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:44.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.652+0000 7f8253400640 1 -- 172.21.15.134:0/301361147 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8248015020 con 0x7f825c10f740 2024-07-26T11:24:44.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.652+0000 7f8253400640 1 -- 172.21.15.134:0/301361147 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8248011ce0 con 0x7f825c10f740 2024-07-26T11:24:44.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.653+0000 7f8260a00640 1 -- 172.21.15.134:0/301361147 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f825c1a6620 con 0x7f825c10f740 2024-07-26T11:24:44.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.654+0000 7f8260a00640 1 -- 172.21.15.134:0/301361147 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f825c1acd80 con 0x7f825c10f740 2024-07-26T11:24:44.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.654+0000 7f8253400640 1 -- 172.21.15.134:0/301361147 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8248010d80 con 0x7f825c10f740 2024-07-26T11:24:44.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.663+0000 7f8253400640 1 -- 172.21.15.134:0/301361147 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f8248018020 con 0x7f825c10f740 2024-07-26T11:24:44.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.669+0000 7f8253400640 1 --2- 172.21.15.134:0/301361147 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82400b3d70 0x7f82400b6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.669+0000 7f8253400640 1 -- 172.21.15.134:0/301361147 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f82480e3860 con 0x7f825c10f740 2024-07-26T11:24:44.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.671+0000 7f825a800640 1 --2- 172.21.15.134:0/301361147 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82400b3d70 0x7f82400b6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:44.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.673+0000 7f825a800640 1 --2- 172.21.15.134:0/301361147 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82400b3d70 0x7f82400b6280 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f824c0096c0 tx=0x7f824c000bc0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:44.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.689+0000 7f82681f2740 1 Processor -- start 2024-07-26T11:24:44.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.689+0000 7f82681f2740 1 -- start start 2024-07-26T11:24:44.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.690+0000 7f82681f2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a60070 0x55c314e05010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.690+0000 7f82681f2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a371b0 0x55c314e05550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.690+0000 7f82681f2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a3f390 0x55c314e05a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.690+0000 7f82681f2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c314d7bbd0 con 0x55c314a3f390 2024-07-26T11:24:44.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.690+0000 7f82681f2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c314d6b0d0 con 0x55c314a60070 2024-07-26T11:24:44.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.690+0000 7f82681f2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c314d6f110 con 0x55c314a371b0 2024-07-26T11:24:44.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.691+0000 7f825b200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a3f390 0x55c314e05a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:44.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.691+0000 7f825b200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a3f390 0x55c314e05a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:41970/0 (socket says 172.21.15.134:41970) 2024-07-26T11:24:44.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.691+0000 7f825b200640 1 -- 172.21.15.134:0/3640969623 learned_addr learned my addr 172.21.15.134:0/3640969623 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:44.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.691+0000 7f8259e00640 1 --2- 172.21.15.134:0/3640969623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a371b0 0x55c314e05550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:44.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.691+0000 7f825a800640 1 --2- 172.21.15.134:0/3640969623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a60070 0x55c314e05010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:44.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.691+0000 7f825b200640 1 -- 172.21.15.134:0/3640969623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a371b0 msgr2=0x55c314e05550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:44.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.691+0000 7f825b200640 1 --2- 172.21.15.134:0/3640969623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a371b0 0x55c314e05550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.691+0000 7f825b200640 1 -- 172.21.15.134:0/3640969623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a60070 msgr2=0x55c314e05010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:44.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.691+0000 7f825b200640 1 --2- 172.21.15.134:0/3640969623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a60070 0x55c314e05010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.692+0000 7f825b200640 1 -- 172.21.15.134:0/3640969623 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c314acb650 con 0x55c314a3f390 2024-07-26T11:24:44.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.692+0000 7f825b200640 1 --2- 172.21.15.134:0/3640969623 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a3f390 0x55c314e05a90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8254009e80 tx=0x7f825400d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7777cfc857f76c3f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:44.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.693+0000 7f8247400640 1 -- 172.21.15.134:0/3640969623 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254016070 con 0x55c314a3f390 2024-07-26T11:24:44.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.693+0000 7f8247400640 1 -- 172.21.15.134:0/3640969623 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8254004d20 con 0x55c314a3f390 2024-07-26T11:24:44.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.693+0000 7f8247400640 1 -- 172.21.15.134:0/3640969623 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254016070 con 0x55c314a3f390 2024-07-26T11:24:44.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.694+0000 7f82681f2740 1 -- 172.21.15.134:0/3640969623 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a3f390 msgr2=0x55c314e05a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:44.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.694+0000 7f82681f2740 1 --2- 172.21.15.134:0/3640969623 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a3f390 0x55c314e05a90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8254009e80 tx=0x7f825400d6b0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.695+0000 7f82681f2740 1 -- 172.21.15.134:0/3640969623 shutdown_connections 2024-07-26T11:24:44.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.695+0000 7f82681f2740 1 --2- 172.21.15.134:0/3640969623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a371b0 0x55c314e05550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.695+0000 7f82681f2740 1 --2- 172.21.15.134:0/3640969623 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a3f390 0x55c314e05a90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.695+0000 7f82681f2740 1 --2- 172.21.15.134:0/3640969623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a60070 0x55c314e05010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.695+0000 7f82681f2740 1 -- 172.21.15.134:0/3640969623 >> 172.21.15.134:0/3640969623 conn(0x55c314a54460 msgr2=0x55c314e02f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:44.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.695+0000 7f82681f2740 1 -- 172.21.15.134:0/3640969623 shutdown_connections 2024-07-26T11:24:44.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.696+0000 7f82681f2740 1 -- 172.21.15.134:0/3640969623 wait complete. 2024-07-26T11:24:44.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.696+0000 7f82681f2740 1 -- start start 2024-07-26T11:24:44.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.697+0000 7f82681f2740 1 Processor -- start 2024-07-26T11:24:44.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.699+0000 7f82681f2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a371b0 0x55c314e10f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.699+0000 7f82681f2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a3f390 0x55c314e114a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.699+0000 7f82681f2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a60070 0x55c314e16560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.699+0000 7f82681f2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c314bce700 con 0x55c314a60070 2024-07-26T11:24:44.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.699+0000 7f82681f2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c314d076a0 con 0x55c314a371b0 2024-07-26T11:24:44.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.699+0000 7f82681f2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c314e0f7d0 con 0x55c314a3f390 2024-07-26T11:24:44.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.700+0000 7f82681f2740 10 client.0 _ll_register_callbacks cb 0x55c314d17150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:24:44.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.703+0000 7f8259e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a3f390 0x55c314e114a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:44.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.703+0000 7f825a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a371b0 0x55c314e10f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:44.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.703+0000 7f825a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a371b0 0x55c314e10f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56778/0 (socket says 172.21.15.134:56778) 2024-07-26T11:24:44.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.703+0000 7f8259e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a3f390 0x55c314e114a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:43484/0 (socket says 172.21.15.134:43484) 2024-07-26T11:24:44.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.704+0000 7f8259e00640 1 -- 172.21.15.134:0/3865484330 learned_addr learned my addr 172.21.15.134:0/3865484330 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:44.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.704+0000 7f8259e00640 1 -- 172.21.15.134:0/3865484330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a371b0 msgr2=0x55c314e10f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:44.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.705+0000 7f8259e00640 1 --2- 172.21.15.134:0/3865484330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a371b0 0x55c314e10f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.705+0000 7f8259e00640 1 -- 172.21.15.134:0/3865484330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a60070 msgr2=0x55c314e16560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:44.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.705+0000 7f8259e00640 1 --2- 172.21.15.134:0/3865484330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a60070 0x55c314e16560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.705+0000 7f8259e00640 1 -- 172.21.15.134:0/3865484330 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c314d1e840 con 0x55c314a3f390 2024-07-26T11:24:44.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.705+0000 7f825a800640 1 --2- 172.21.15.134:0/3865484330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a371b0 0x55c314e10f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:24:44.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.706+0000 7f8259e00640 1 --2- 172.21.15.134:0/3865484330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a3f390 0x55c314e114a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f824800bd50 tx=0x7f82480e1110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f3b0647bb4917b0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:44.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.706+0000 7f823be00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:24:44.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.707+0000 7f823be00640 1 -- 172.21.15.134:0/3865484330 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82480c3070 con 0x55c314a3f390 2024-07-26T11:24:44.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.707+0000 7f823be00640 1 -- 172.21.15.134:0/3865484330 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f82480b2fd0 con 0x55c314a3f390 2024-07-26T11:24:44.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.707+0000 7f82681f2740 1 -- 172.21.15.134:0/3865484330 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8254004d20 con 0x55c314a3f390 2024-07-26T11:24:44.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.708+0000 7f823be00640 1 -- 172.21.15.134:0/3865484330 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f82480ae7a0 con 0x55c314a3f390 2024-07-26T11:24:44.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.708+0000 7f82681f2740 10 client.26713 fetch_fsmap learned FSMap version 32 2024-07-26T11:24:44.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.708+0000 7f82681f2740 1 -- 172.21.15.134:0/3865484330 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c314e1b4c0 con 0x55c314a3f390 2024-07-26T11:24:44.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.711+0000 7f823be00640 1 -- 172.21.15.134:0/3865484330 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8254004d20 con 0x55c314a3f390 2024-07-26T11:24:44.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.711+0000 7f823be00640 1 -- 172.21.15.134:0/3865484330 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f82480c4ce0 con 0x55c314a3f390 2024-07-26T11:24:44.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.711+0000 7f82681f2740 10 client.26713 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:24:44.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.711+0000 7f82681f2740 10 client.26713 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T11:24:44.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.711+0000 7f82681f2740 20 client.26713 populate_metadata read hostname 'smithi134' 2024-07-26T11:24:44.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.711+0000 7f82681f2740 1 --2- 172.21.15.134:0/3865484330 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55c314a1e9f0 0x55c314e1ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.711+0000 7f82681f2740 4 client.26713 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:24:44.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.711+0000 7f82681f2740 1 -- 172.21.15.134:0/3865484330 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c314d7db20 con 0x55c314a1e9f0 2024-07-26T11:24:44.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.712+0000 7f825a800640 1 --2- 172.21.15.134:0/3865484330 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55c314a1e9f0 0x55c314e1ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:44.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.714+0000 7f823be00640 10 client.26713 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:24:44.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.714+0000 7f825a800640 1 --2- 172.21.15.134:0/3865484330 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55c314a1e9f0 0x55c314e1ea70 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=381ad11170bd2068 server_cookie=c2c7c73008b07c29 in_seq=0 out_seq=0 2024-07-26T11:24:44.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.716+0000 7f823be00640 1 -- 172.21.15.134:0/3865484330 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c314d7db20 con 0x55c314a1e9f0 2024-07-26T11:24:44.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.716+0000 7f823be00640 10 client.26713 handle_command_reply: tid=0 2024-07-26T11:24:44.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.716+0000 7f82681f2740 1 client.26713 shutdown 2024-07-26T11:24:44.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.717+0000 7f82681f2740 1 -- 172.21.15.134:0/3865484330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a3f390 msgr2=0x55c314e114a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:44.719 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.717+0000 7f82681f2740 1 --2- 172.21.15.134:0/3865484330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a3f390 0x55c314e114a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f824800bd50 tx=0x7f82480e1110 comp rx=0 tx=0).stop 2024-07-26T11:24:44.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.717+0000 7f82681f2740 1 -- 172.21.15.134:0/3865484330 shutdown_connections 2024-07-26T11:24:44.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.717+0000 7f82681f2740 1 --2- 172.21.15.134:0/3865484330 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55c314a1e9f0 0x55c314e1ea70 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.718+0000 7f82681f2740 1 --2- 172.21.15.134:0/3865484330 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a3f390 0x55c314e114a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.718+0000 7f823be00640 0 client.26713 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:24:44.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.718+0000 7f82681f2740 1 --2- 172.21.15.134:0/3865484330 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a60070 0x55c314e16560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.718+0000 7f82681f2740 1 --2- 172.21.15.134:0/3865484330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a371b0 0x55c314e10f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.718+0000 7f82681f2740 1 -- 172.21.15.134:0/3865484330 >> 172.21.15.134:0/3865484330 conn(0x55c314a54460 msgr2=0x55c314e081f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:44.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.718+0000 7f82681f2740 1 -- 172.21.15.134:0/3865484330 shutdown_connections 2024-07-26T11:24:44.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.719+0000 7f82681f2740 1 -- 172.21.15.134:0/3865484330 wait complete. 2024-07-26T11:24:44.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.719+0000 7f82681f2740 20 client.26713 trim_cache size 0 max 16384 2024-07-26T11:24:44.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.763+0000 7f82681f2740 1 Processor -- start 2024-07-26T11:24:44.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.763+0000 7f82681f2740 1 -- start start 2024-07-26T11:24:44.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.763+0000 7f82681f2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a54460 0x55c314e01f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.763+0000 7f82681f2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a1e9f0 0x55c314e06750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.764+0000 7f82681f2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a371b0 0x55c314e163d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.764+0000 7f82681f2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f825c1b6d30 con 0x55c314a54460 2024-07-26T11:24:44.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.764+0000 7f82681f2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f825c1b8870 con 0x55c314a1e9f0 2024-07-26T11:24:44.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.764+0000 7f82681f2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f825c1b7400 con 0x55c314a371b0 2024-07-26T11:24:44.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.764+0000 7f8259e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a1e9f0 0x55c314e06750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:44.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.764+0000 7f8259e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a1e9f0 0x55c314e06750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56794/0 (socket says 172.21.15.134:56794) 2024-07-26T11:24:44.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.764+0000 7f8259e00640 1 -- 172.21.15.134:0/3360080846 learned_addr learned my addr 172.21.15.134:0/3360080846 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:44.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.765+0000 7f8259e00640 1 -- 172.21.15.134:0/3360080846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a371b0 msgr2=0x55c314e163d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:24:44.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.765+0000 7f8259e00640 1 --2- 172.21.15.134:0/3360080846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a371b0 0x55c314e163d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.765+0000 7f8259e00640 1 -- 172.21.15.134:0/3360080846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a54460 msgr2=0x55c314e01f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:24:44.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.765+0000 7f8259e00640 1 --2- 172.21.15.134:0/3360080846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a54460 0x55c314e01f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.765+0000 7f8259e00640 1 -- 172.21.15.134:0/3360080846 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c314e03870 con 0x55c314a1e9f0 2024-07-26T11:24:44.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.765+0000 7f8259e00640 1 --2- 172.21.15.134:0/3360080846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a1e9f0 0x55c314e06750 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f82480a7410 tx=0x7f82480a6190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b41ba644b61624e2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:44.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.765+0000 7f8246a00640 1 -- 172.21.15.134:0/3360080846 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82480ae7a0 con 0x55c314a1e9f0 2024-07-26T11:24:44.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.765+0000 7f8246a00640 1 -- 172.21.15.134:0/3360080846 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82480b3250 con 0x55c314a1e9f0 2024-07-26T11:24:44.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.765+0000 7f8246a00640 1 -- 172.21.15.134:0/3360080846 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82480cb3d0 con 0x55c314a1e9f0 2024-07-26T11:24:44.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.767+0000 7f82681f2740 1 -- 172.21.15.134:0/3360080846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a1e9f0 msgr2=0x55c314e06750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:44.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.767+0000 7f82681f2740 1 --2- 172.21.15.134:0/3360080846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a1e9f0 0x55c314e06750 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f82480a7410 tx=0x7f82480a6190 comp rx=0 tx=0).stop 2024-07-26T11:24:44.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.767+0000 7f82681f2740 1 -- 172.21.15.134:0/3360080846 shutdown_connections 2024-07-26T11:24:44.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.767+0000 7f82681f2740 1 --2- 172.21.15.134:0/3360080846 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a371b0 0x55c314e163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.767+0000 7f82681f2740 1 --2- 172.21.15.134:0/3360080846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a1e9f0 0x55c314e06750 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.767+0000 7f82681f2740 1 --2- 172.21.15.134:0/3360080846 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a54460 0x55c314e01f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.767+0000 7f82681f2740 1 -- 172.21.15.134:0/3360080846 >> 172.21.15.134:0/3360080846 conn(0x55c314e10520 msgr2=0x55c314e06040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:44.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.768+0000 7f82681f2740 1 -- 172.21.15.134:0/3360080846 shutdown_connections 2024-07-26T11:24:44.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.768+0000 7f82681f2740 1 -- 172.21.15.134:0/3360080846 wait complete. 2024-07-26T11:24:44.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.769+0000 7f82681f2740 1 -- start start 2024-07-26T11:24:44.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.770+0000 7f82681f2740 1 Processor -- start 2024-07-26T11:24:44.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.771+0000 7f82681f2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a1e9f0 0x55c314e25750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.771+0000 7f82681f2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a371b0 0x55c314e1fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.771+0000 7f82681f2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a54460 0x55c314e20180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.771+0000 7f82681f2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f825c1b81a0 con 0x55c314a1e9f0 2024-07-26T11:24:44.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.771+0000 7f82681f2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f825c1b7ad0 con 0x55c314a371b0 2024-07-26T11:24:44.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.771+0000 7f82681f2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f825c1b4cf0 con 0x55c314a54460 2024-07-26T11:24:44.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.771+0000 7f82681f2740 10 client.0 _ll_register_callbacks cb 0x7f8248009220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:24:44.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.772+0000 7f8259e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a371b0 0x55c314e1fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:44.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.772+0000 7f8259e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a371b0 0x55c314e1fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:56802/0 (socket says 172.21.15.134:56802) 2024-07-26T11:24:44.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.772+0000 7f825b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a54460 0x55c314e20180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:44.775 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.772+0000 7f825b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a54460 0x55c314e20180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:43486/0 (socket says 172.21.15.134:43486) 2024-07-26T11:24:44.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.772+0000 7f825b200640 1 -- 172.21.15.134:0/1952298847 learned_addr learned my addr 172.21.15.134:0/1952298847 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:44.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.775+0000 7f825b200640 1 -- 172.21.15.134:0/1952298847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a371b0 msgr2=0x55c314e1fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:44.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.775+0000 7f825b200640 1 --2- 172.21.15.134:0/1952298847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a371b0 0x55c314e1fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.775+0000 7f825b200640 1 -- 172.21.15.134:0/1952298847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a1e9f0 msgr2=0x55c314e25750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:44.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.775+0000 7f825b200640 1 --2- 172.21.15.134:0/1952298847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a1e9f0 0x55c314e25750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.775+0000 7f825b200640 1 -- 172.21.15.134:0/1952298847 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c314e206c0 con 0x55c314a54460 2024-07-26T11:24:44.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.777+0000 7f823be00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:24:44.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.777+0000 7f825b200640 1 --2- 172.21.15.134:0/1952298847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a54460 0x55c314e20180 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8254004d80 tx=0x7f8254004db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=977cfdc8c00369a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:44.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.778+0000 7f823be00640 1 -- 172.21.15.134:0/1952298847 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254016070 con 0x55c314a54460 2024-07-26T11:24:44.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.778+0000 7f823be00640 1 -- 172.21.15.134:0/1952298847 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f8254003b80 con 0x55c314a54460 2024-07-26T11:24:44.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.782+0000 7f82681f2740 1 -- 172.21.15.134:0/1952298847 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82480b3250 con 0x55c314a54460 2024-07-26T11:24:44.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.782+0000 7f823be00640 1 -- 172.21.15.134:0/1952298847 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8254016070 con 0x55c314a54460 2024-07-26T11:24:44.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.783+0000 7f82681f2740 10 client.26716 fetch_fsmap learned FSMap version 32 2024-07-26T11:24:44.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.783+0000 7f82681f2740 1 -- 172.21.15.134:0/1952298847 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55c314e9be10 con 0x55c314a54460 2024-07-26T11:24:44.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.784+0000 7f823be00640 1 -- 172.21.15.134:0/1952298847 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8254003d80 con 0x55c314a54460 2024-07-26T11:24:44.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.784+0000 7f823be00640 1 -- 172.21.15.134:0/1952298847 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f8254022020 con 0x55c314a54460 2024-07-26T11:24:44.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.784+0000 7f82681f2740 10 client.26716 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:24:44.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.784+0000 7f82681f2740 10 client.26716 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T11:24:44.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.784+0000 7f82681f2740 20 client.26716 populate_metadata read hostname 'smithi134' 2024-07-26T11:24:44.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.784+0000 7f82681f2740 1 --2- 172.21.15.134:0/1952298847 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55c314a3f390 0x55c314e9e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:44.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.784+0000 7f82681f2740 4 client.26716 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:24:44.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.784+0000 7f82681f2740 1 -- 172.21.15.134:0/1952298847 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f82480b3250 con 0x55c314a3f390 2024-07-26T11:24:44.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.785+0000 7f825a800640 1 --2- 172.21.15.134:0/1952298847 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55c314a3f390 0x55c314e9e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:44.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.786+0000 7f823be00640 10 client.26716 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:24:44.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.790+0000 7f825a800640 1 --2- 172.21.15.134:0/1952298847 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55c314a3f390 0x55c314e9e2e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=61d1c5939ebf43b8 server_cookie=6f3fcdf7ced92c0c in_seq=0 out_seq=0 2024-07-26T11:24:44.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.790+0000 7f823be00640 1 -- 172.21.15.134:0/1952298847 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f82480b3250 con 0x55c314a3f390 2024-07-26T11:24:44.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.790+0000 7f823be00640 10 client.26716 handle_command_reply: tid=0 2024-07-26T11:24:44.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.790+0000 7f82681f2740 1 client.26716 shutdown 2024-07-26T11:24:44.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.791+0000 7f82681f2740 1 -- 172.21.15.134:0/1952298847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a54460 msgr2=0x55c314e20180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:44.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.791+0000 7f82681f2740 1 --2- 172.21.15.134:0/1952298847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a54460 0x55c314e20180 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8254004d80 tx=0x7f8254004db0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.791+0000 7f82681f2740 1 -- 172.21.15.134:0/1952298847 shutdown_connections 2024-07-26T11:24:44.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.791+0000 7f82681f2740 1 --2- 172.21.15.134:0/1952298847 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x55c314a3f390 0x55c314e9e2e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.791+0000 7f82681f2740 1 --2- 172.21.15.134:0/1952298847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c314a54460 0x55c314e20180 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.791+0000 7f82681f2740 1 --2- 172.21.15.134:0/1952298847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c314a371b0 0x55c314e1fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.791+0000 7f82681f2740 1 --2- 172.21.15.134:0/1952298847 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c314a1e9f0 0x55c314e25750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.791+0000 7f82681f2740 1 -- 172.21.15.134:0/1952298847 >> 172.21.15.134:0/1952298847 conn(0x55c314e10520 msgr2=0x55c314e01cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:44.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.791+0000 7f823be00640 0 client.26716 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:24:44.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.791+0000 7f82681f2740 1 -- 172.21.15.134:0/1952298847 shutdown_connections 2024-07-26T11:24:44.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.792+0000 7f82681f2740 1 -- 172.21.15.134:0/1952298847 wait complete. 2024-07-26T11:24:44.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.792+0000 7f82681f2740 20 client.26716 trim_cache size 0 max 16384 2024-07-26T11:24:44.795 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:24:44.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.793+0000 7f8260a00640 1 -- 172.21.15.134:0/301361147 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82400b3d70 msgr2=0x7f82400b6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:44.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.793+0000 7f8260a00640 1 --2- 172.21.15.134:0/301361147 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82400b3d70 0x7f82400b6280 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f824c0096c0 tx=0x7f824c000bc0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.793+0000 7f8260a00640 1 -- 172.21.15.134:0/301361147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c10f740 msgr2=0x7f825c1a0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:44.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.793+0000 7f8260a00640 1 --2- 172.21.15.134:0/301361147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c10f740 0x7f825c1a0960 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f825c06c340 tx=0x7f824800b850 comp rx=0 tx=0).stop 2024-07-26T11:24:44.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.794+0000 7f8260a00640 1 -- 172.21.15.134:0/301361147 shutdown_connections 2024-07-26T11:24:44.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.794+0000 7f8260a00640 1 --2- 172.21.15.134:0/301361147 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f82400b3d70 0x7f82400b6280 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.794+0000 7f8260a00640 1 --2- 172.21.15.134:0/301361147 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f825c110090 0x7f825c1a5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.794+0000 7f8260a00640 1 --2- 172.21.15.134:0/301361147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f825c10f740 0x7f825c1a0960 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.794+0000 7f8260a00640 1 --2- 172.21.15.134:0/301361147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f825c10a080 0x7f825c1a03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:44.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.794+0000 7f8260a00640 1 -- 172.21.15.134:0/301361147 >> 172.21.15.134:0/301361147 conn(0x7f825c06e420 msgr2=0x7f825c10d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:44.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.794+0000 7f8260a00640 1 -- 172.21.15.134:0/301361147 shutdown_connections 2024-07-26T11:24:44.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:44.795+0000 7f8260a00640 1 -- 172.21.15.134:0/301361147 wait complete. 2024-07-26T11:24:44.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:44 smithi134 ceph-mon[31966]: pgmap v747: 129 pgs: 129 active+clean; 57 GiB data, 121 GiB used, 951 GiB / 1.0 TiB avail; 197 KiB/s rd, 215 MiB/s wr, 1.70k op/s 2024-07-26T11:24:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:44 smithi193 ceph-mon[37473]: pgmap v747: 129 pgs: 129 active+clean; 57 GiB data, 121 GiB used, 951 GiB / 1.0 TiB avail; 197 KiB/s rd, 215 MiB/s wr, 1.70k op/s 2024-07-26T11:24:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:44 smithi189 ceph-mon[36790]: pgmap v747: 129 pgs: 129 active+clean; 57 GiB data, 121 GiB used, 951 GiB / 1.0 TiB avail; 197 KiB/s rd, 215 MiB/s wr, 1.70k op/s 2024-07-26T11:24:47.943 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:47 smithi134 ceph-mon[31966]: pgmap v748: 129 pgs: 129 active+clean; 58 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 178 KiB/s rd, 223 MiB/s wr, 1.73k op/s 2024-07-26T11:24:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:47 smithi189 ceph-mon[36790]: pgmap v748: 129 pgs: 129 active+clean; 58 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 178 KiB/s rd, 223 MiB/s wr, 1.73k op/s 2024-07-26T11:24:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:47 smithi193 ceph-mon[37473]: pgmap v748: 129 pgs: 129 active+clean; 58 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 178 KiB/s rd, 223 MiB/s wr, 1.73k op/s 2024-07-26T11:24:48.055 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-26T11:24:49.018 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:24:49.973 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:49 smithi134 ceph-mon[31966]: pgmap v749: 129 pgs: 129 active+clean; 58 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 197 KiB/s rd, 219 MiB/s wr, 1.66k op/s 2024-07-26T11:24:50.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:49 smithi193 ceph-mon[37473]: pgmap v749: 129 pgs: 129 active+clean; 58 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 197 KiB/s rd, 219 MiB/s wr, 1.66k op/s 2024-07-26T11:24:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:49 smithi189 ceph-mon[36790]: pgmap v749: 129 pgs: 129 active+clean; 58 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 197 KiB/s rd, 219 MiB/s wr, 1.66k op/s 2024-07-26T11:24:50.927 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:50 smithi134 ceph-mon[31966]: pgmap v750: 129 pgs: 129 active+clean; 59 GiB data, 124 GiB used, 949 GiB / 1.0 TiB avail; 34 KiB/s rd, 268 MiB/s wr, 1.91k op/s 2024-07-26T11:24:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:50 smithi193 ceph-mon[37473]: pgmap v750: 129 pgs: 129 active+clean; 59 GiB data, 124 GiB used, 949 GiB / 1.0 TiB avail; 34 KiB/s rd, 268 MiB/s wr, 1.91k op/s 2024-07-26T11:24:51.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:50 smithi189 ceph-mon[36790]: pgmap v750: 129 pgs: 129 active+clean; 59 GiB data, 124 GiB used, 949 GiB / 1.0 TiB avail; 34 KiB/s rd, 268 MiB/s wr, 1.91k op/s 2024-07-26T11:24:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.250+0000 7f9907000640 1 -- 172.21.15.134:0/4294338692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f990010a740 msgr2=0x7f9900115a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.250+0000 7f9907000640 1 --2- 172.21.15.134:0/4294338692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f990010a740 0x7f9900115a00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f98ec009860 tx=0x7f98ec02a340 comp rx=0 tx=0).stop 2024-07-26T11:24:51.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.250+0000 7f9907000640 1 -- 172.21.15.134:0/4294338692 shutdown_connections 2024-07-26T11:24:51.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.250+0000 7f9907000640 1 --2- 172.21.15.134:0/4294338692 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f990010a740 0x7f9900115a00 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.250+0000 7f9907000640 1 --2- 172.21.15.134:0/4294338692 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9900115f40 0x7f9900118360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.250+0000 7f9907000640 1 --2- 172.21.15.134:0/4294338692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9900109df0 0x7f990010a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.250+0000 7f9907000640 1 -- 172.21.15.134:0/4294338692 >> 172.21.15.134:0/4294338692 conn(0x7f9900070630 msgr2=0x7f9900070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.255+0000 7f9907000640 1 -- 172.21.15.134:0/4294338692 shutdown_connections 2024-07-26T11:24:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.256+0000 7f9907000640 1 -- 172.21.15.134:0/4294338692 wait complete. 2024-07-26T11:24:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.258+0000 7f9907000640 1 Processor -- start 2024-07-26T11:24:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.260+0000 7f9907000640 1 -- start start 2024-07-26T11:24:51.262 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.261+0000 7f9907000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9900109df0 0x7f99001a2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.263 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.261+0000 7f9907000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f990010a740 0x7f99001a29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.262+0000 7f9907000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9900115f40 0x7f99001a7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.264 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.262+0000 7f9907000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f990011bb00 con 0x7f990010a740 2024-07-26T11:24:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.262+0000 7f9907000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f990011b980 con 0x7f9900115f40 2024-07-26T11:24:51.265 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.262+0000 7f9907000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f990011bc80 con 0x7f9900109df0 2024-07-26T11:24:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.262+0000 7f9905c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9900109df0 0x7f99001a2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.263+0000 7f9905c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9900109df0 0x7f99001a2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58674/0 (socket says 172.21.15.134:58674) 2024-07-26T11:24:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.263+0000 7f9905c00640 1 -- 172.21.15.134:0/1826921299 learned_addr learned my addr 172.21.15.134:0/1826921299 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.263+0000 7f9905200640 1 --2- 172.21.15.134:0/1826921299 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f990010a740 0x7f99001a29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.264+0000 7f9906600640 1 --2- 172.21.15.134:0/1826921299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9900115f40 0x7f99001a7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.264+0000 7f9905c00640 1 -- 172.21.15.134:0/1826921299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9900115f40 msgr2=0x7f99001a7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.264+0000 7f9905c00640 1 --2- 172.21.15.134:0/1826921299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9900115f40 0x7f99001a7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.265+0000 7f9905c00640 1 -- 172.21.15.134:0/1826921299 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f990010a740 msgr2=0x7f99001a29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.265+0000 7f9905c00640 1 --2- 172.21.15.134:0/1826921299 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f990010a740 0x7f99001a29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.265+0000 7f9905c00640 1 -- 172.21.15.134:0/1826921299 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99001a8350 con 0x7f9900109df0 2024-07-26T11:24:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.266+0000 7f9906600640 1 --2- 172.21.15.134:0/1826921299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9900115f40 0x7f99001a7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:24:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.266+0000 7f9905200640 1 --2- 172.21.15.134:0/1826921299 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f990010a740 0x7f99001a29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:24:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.266+0000 7f9905c00640 1 --2- 172.21.15.134:0/1826921299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9900109df0 0x7f99001a2420 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f98f400d770 tx=0x7f98f400dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.267+0000 7f98fe000640 1 -- 172.21.15.134:0/1826921299 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4017070 con 0x7f9900109df0 2024-07-26T11:24:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.267+0000 7f98fe000640 1 -- 172.21.15.134:0/1826921299 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98f40054d0 con 0x7f9900109df0 2024-07-26T11:24:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.267+0000 7f9907000640 1 -- 172.21.15.134:0/1826921299 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99001a8640 con 0x7f9900109df0 2024-07-26T11:24:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.267+0000 7f98fe000640 1 -- 172.21.15.134:0/1826921299 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4004d50 con 0x7f9900109df0 2024-07-26T11:24:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.267+0000 7f9907000640 1 -- 172.21.15.134:0/1826921299 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f990010b790 con 0x7f9900109df0 2024-07-26T11:24:51.286 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.280+0000 7f990f59a740 1 Processor -- start 2024-07-26T11:24:51.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.287+0000 7f98fe000640 1 -- 172.21.15.134:0/1826921299 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f98f4009c30 con 0x7f9900109df0 2024-07-26T11:24:51.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.290+0000 7f990f59a740 1 -- start start 2024-07-26T11:24:51.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.291+0000 7f990f59a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a55c1b0 0x55636a8ffc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.291+0000 7f990f59a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a5332f0 0x55636a900160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.291+0000 7f990f59a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a53b4d0 0x55636a9006a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.291+0000 7f990f59a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55636a874270 con 0x55636a5332f0 2024-07-26T11:24:51.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.291+0000 7f990f59a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55636a863770 con 0x55636a53b4d0 2024-07-26T11:24:51.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.291+0000 7f990f59a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55636a8677b0 con 0x55636a55c1b0 2024-07-26T11:24:51.295 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.292+0000 7f9906600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a55c1b0 0x55636a8ffc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.295 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.292+0000 7f9906600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a55c1b0 0x55636a8ffc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58678/0 (socket says 172.21.15.134:58678) 2024-07-26T11:24:51.295 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.292+0000 7f9906600640 1 -- 172.21.15.134:0/573338102 learned_addr learned my addr 172.21.15.134:0/573338102 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:51.295 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.293+0000 7f9905c00640 1 --2- 172.21.15.134:0/573338102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a5332f0 0x55636a900160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.295 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.293+0000 7f9905200640 1 --2- 172.21.15.134:0/573338102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a53b4d0 0x55636a9006a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.294+0000 7f9905200640 1 -- 172.21.15.134:0/573338102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a55c1b0 msgr2=0x55636a8ffc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:51.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.294+0000 7f9905200640 1 --2- 172.21.15.134:0/573338102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a55c1b0 0x55636a8ffc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.294+0000 7f9905200640 1 -- 172.21.15.134:0/573338102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a5332f0 msgr2=0x55636a900160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:51.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.294+0000 7f9905200640 1 --2- 172.21.15.134:0/573338102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a5332f0 0x55636a900160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.294+0000 7f9905200640 1 -- 172.21.15.134:0/573338102 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55636a5c7790 con 0x55636a53b4d0 2024-07-26T11:24:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.294+0000 7f98fe000640 1 --2- 172.21.15.134:0/1826921299 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98e80b3df0 0x7f98e80b6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.294+0000 7f98fe000640 1 -- 172.21.15.134:0/1826921299 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f98f40e3e90 con 0x7f9900109df0 2024-07-26T11:24:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.295+0000 7f9905200640 1 --2- 172.21.15.134:0/573338102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a53b4d0 0x55636a9006a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f990006be60 tx=0x7f98ec005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2138f9c74249ade6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.296+0000 7f98f2a00640 1 -- 172.21.15.134:0/573338102 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec041070 con 0x55636a53b4d0 2024-07-26T11:24:51.307 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.303+0000 7f9906600640 1 --2- 172.21.15.134:0/1826921299 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98e80b3df0 0x7f98e80b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.309+0000 7f9906600640 1 --2- 172.21.15.134:0/1826921299 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98e80b3df0 0x7f98e80b6300 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f98f800a510 tx=0x7f98f80096e0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:51.327 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.325+0000 7f98f2a00640 1 -- 172.21.15.134:0/573338102 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98ec030e10 con 0x55636a53b4d0 2024-07-26T11:24:51.328 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.326+0000 7f990f59a740 1 -- 172.21.15.134:0/573338102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a53b4d0 msgr2=0x55636a9006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:51.328 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.326+0000 7f990f59a740 1 --2- 172.21.15.134:0/573338102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a53b4d0 0x55636a9006a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f990006be60 tx=0x7f98ec005c90 comp rx=0 tx=0).stop 2024-07-26T11:24:51.329 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.328+0000 7f990f59a740 1 -- 172.21.15.134:0/573338102 shutdown_connections 2024-07-26T11:24:51.332 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.328+0000 7f990f59a740 1 --2- 172.21.15.134:0/573338102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a53b4d0 0x55636a9006a0 secure :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f990006be60 tx=0x7f98ec005c90 comp rx=0 tx=0).stop 2024-07-26T11:24:51.332 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.328+0000 7f990f59a740 1 --2- 172.21.15.134:0/573338102 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a5332f0 0x55636a900160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.332 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.328+0000 7f990f59a740 1 --2- 172.21.15.134:0/573338102 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a55c1b0 0x55636a8ffc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.332 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.328+0000 7f990f59a740 1 -- 172.21.15.134:0/573338102 >> 172.21.15.134:0/573338102 conn(0x55636a550300 msgr2=0x55636a8fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:51.345 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.342+0000 7f990f59a740 1 -- 172.21.15.134:0/573338102 shutdown_connections 2024-07-26T11:24:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.344+0000 7f990f59a740 1 -- 172.21.15.134:0/573338102 wait complete. 2024-07-26T11:24:51.350 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.349+0000 7f990f59a740 1 -- start start 2024-07-26T11:24:51.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.349+0000 7f990f59a740 1 Processor -- start 2024-07-26T11:24:51.355 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.353+0000 7f990f59a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a5332f0 0x55636a90ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.355 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.353+0000 7f990f59a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a55c1b0 0x55636a90bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.353+0000 7f990f59a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a51ad80 0x55636a911760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.353+0000 7f990f59a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55636a79dab0 con 0x55636a5332f0 2024-07-26T11:24:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.353+0000 7f990f59a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55636a800d50 con 0x55636a51ad80 2024-07-26T11:24:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.353+0000 7f990f59a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55636a90e3e0 con 0x55636a55c1b0 2024-07-26T11:24:51.356 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.354+0000 7f990f59a740 10 client.0 _ll_register_callbacks cb 0x55636a573a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:24:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.354+0000 7f9905200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a55c1b0 0x55636a90bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.354+0000 7f9905200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a55c1b0 0x55636a90bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58684/0 (socket says 172.21.15.134:58684) 2024-07-26T11:24:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.354+0000 7f9905200640 1 -- 172.21.15.134:0/3821881829 learned_addr learned my addr 172.21.15.134:0/3821881829 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.354+0000 7f9906600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a51ad80 0x55636a911760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.357 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.355+0000 7f9905c00640 1 --2- 172.21.15.134:0/3821881829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a5332f0 0x55636a90ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.356+0000 7f9905c00640 1 -- 172.21.15.134:0/3821881829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a55c1b0 msgr2=0x55636a90bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.356+0000 7f9905c00640 1 --2- 172.21.15.134:0/3821881829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a55c1b0 0x55636a90bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.356+0000 7f9905c00640 1 -- 172.21.15.134:0/3821881829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a51ad80 msgr2=0x55636a911760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.356+0000 7f9905c00640 1 --2- 172.21.15.134:0/3821881829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a51ad80 0x55636a911760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.356+0000 7f9905c00640 1 -- 172.21.15.134:0/3821881829 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55636a640620 con 0x55636a5332f0 2024-07-26T11:24:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.357+0000 7f98e7400640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:24:51.359 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.358+0000 7f9905c00640 1 --2- 172.21.15.134:0/3821881829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a5332f0 0x55636a90ba10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f98f40e11a0 tx=0x7f98f4004780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e6cdc6a60438d54 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:51.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.358+0000 7f98e7400640 1 -- 172.21.15.134:0/3821881829 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4020070 con 0x55636a5332f0 2024-07-26T11:24:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.361+0000 7f990f59a740 1 -- 172.21.15.134:0/3821881829 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55636a8761c0 con 0x55636a5332f0 2024-07-26T11:24:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.361+0000 7f98e7400640 1 -- 172.21.15.134:0/3821881829 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f98f40a68f0 con 0x55636a5332f0 2024-07-26T11:24:51.369 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.368+0000 7f98e7400640 1 -- 172.21.15.134:0/3821881829 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f98f40aa430 con 0x55636a5332f0 2024-07-26T11:24:51.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.368+0000 7f990f59a740 10 client.16119 fetch_fsmap learned FSMap version 32 2024-07-26T11:24:51.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.368+0000 7f990f59a740 1 -- 172.21.15.134:0/3821881829 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55636a922f70 con 0x55636a5332f0 2024-07-26T11:24:51.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.369+0000 7f98e7400640 1 -- 172.21.15.134:0/3821881829 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55636a8761c0 con 0x55636a5332f0 2024-07-26T11:24:51.372 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.370+0000 7f98e7400640 1 -- 172.21.15.134:0/3821881829 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f98f40cbb60 con 0x55636a5332f0 2024-07-26T11:24:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.371+0000 7f990f59a740 10 client.16119 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:24:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.371+0000 7f990f59a740 10 client.16119 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:24:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.371+0000 7f990f59a740 20 client.16119 populate_metadata read hostname 'smithi134' 2024-07-26T11:24:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.371+0000 7f990f59a740 1 --2- 172.21.15.134:0/3821881829 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55636a4f01c0 0x55636a926980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.371+0000 7f990f59a740 4 client.16119 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:24:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.371+0000 7f990f59a740 1 -- 172.21.15.134:0/3821881829 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55636a871c60 con 0x55636a4f01c0 2024-07-26T11:24:51.373 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.372+0000 7f9905200640 1 --2- 172.21.15.134:0/3821881829 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55636a4f01c0 0x55636a926980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.373+0000 7f9905200640 1 --2- 172.21.15.134:0/3821881829 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55636a4f01c0 0x55636a926980 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1830822f50a44ab2 server_cookie=74705c3f27b0a6e4 in_seq=0 out_seq=0 2024-07-26T11:24:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.373+0000 7f98e7400640 10 client.16119 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:24:51.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.375+0000 7f98e7400640 1 -- 172.21.15.134:0/3821881829 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55636a871c60 con 0x55636a4f01c0 2024-07-26T11:24:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.376+0000 7f98e7400640 10 client.16119 handle_command_reply: tid=0 2024-07-26T11:24:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.376+0000 7f990f59a740 1 client.16119 shutdown 2024-07-26T11:24:51.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.376+0000 7f990f59a740 1 -- 172.21.15.134:0/3821881829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a5332f0 msgr2=0x55636a90ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.377+0000 7f990f59a740 1 --2- 172.21.15.134:0/3821881829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a5332f0 0x55636a90ba10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f98f40e11a0 tx=0x7f98f4004780 comp rx=0 tx=0).stop 2024-07-26T11:24:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.379+0000 7f990f59a740 1 -- 172.21.15.134:0/3821881829 shutdown_connections 2024-07-26T11:24:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.379+0000 7f990f59a740 1 --2- 172.21.15.134:0/3821881829 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55636a4f01c0 0x55636a926980 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.379+0000 7f990f59a740 1 --2- 172.21.15.134:0/3821881829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a55c1b0 0x55636a90bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.379+0000 7f990f59a740 1 --2- 172.21.15.134:0/3821881829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a51ad80 0x55636a911760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.379+0000 7f990f59a740 1 --2- 172.21.15.134:0/3821881829 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a5332f0 0x55636a90ba10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.379+0000 7f990f59a740 1 -- 172.21.15.134:0/3821881829 >> 172.21.15.134:0/3821881829 conn(0x55636a550300 msgr2=0x55636a900c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.379+0000 7f98e7400640 0 client.16119 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:24:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.380+0000 7f990f59a740 1 -- 172.21.15.134:0/3821881829 shutdown_connections 2024-07-26T11:24:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.380+0000 7f990f59a740 1 -- 172.21.15.134:0/3821881829 wait complete. 2024-07-26T11:24:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.381+0000 7f990f59a740 20 client.16119 trim_cache size 0 max 16384 2024-07-26T11:24:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.418+0000 7f990f59a740 1 Processor -- start 2024-07-26T11:24:51.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.418+0000 7f990f59a740 1 -- start start 2024-07-26T11:24:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.419+0000 7f990f59a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a550300 0x55636a911660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.419+0000 7f990f59a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a51ad80 0x55636a8fdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.419+0000 7f990f59a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a5332f0 0x55636a904d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.419+0000 7f990f59a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f99001b6cc0 con 0x55636a5332f0 2024-07-26T11:24:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.419+0000 7f990f59a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f99001b8800 con 0x55636a51ad80 2024-07-26T11:24:51.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.419+0000 7f990f59a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f99001b7390 con 0x55636a550300 2024-07-26T11:24:51.422 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.420+0000 7f9905c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a550300 0x55636a911660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.422 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.420+0000 7f9905c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a550300 0x55636a911660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58688/0 (socket says 172.21.15.134:58688) 2024-07-26T11:24:51.422 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.420+0000 7f9905200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a51ad80 0x55636a8fdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.422 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.420+0000 7f9905200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a51ad80 0x55636a8fdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46796/0 (socket says 172.21.15.134:46796) 2024-07-26T11:24:51.422 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.420+0000 7f9905200640 1 -- 172.21.15.134:0/3132766447 learned_addr learned my addr 172.21.15.134:0/3132766447 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:51.423 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.421+0000 7f9905200640 1 -- 172.21.15.134:0/3132766447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a550300 msgr2=0x55636a911660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.421+0000 7f9905200640 1 --2- 172.21.15.134:0/3132766447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a550300 0x55636a911660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.421+0000 7f9905200640 1 -- 172.21.15.134:0/3132766447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a5332f0 msgr2=0x55636a904d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.421+0000 7f9905200640 1 --2- 172.21.15.134:0/3132766447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a5332f0 0x55636a904d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.421+0000 7f9905200640 1 -- 172.21.15.134:0/3132766447 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55636a902390 con 0x55636a51ad80 2024-07-26T11:24:51.424 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.422+0000 7f9905c00640 1 --2- 172.21.15.134:0/3132766447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a550300 0x55636a911660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:24:51.426 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.424+0000 7f9905200640 1 --2- 172.21.15.134:0/3132766447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a51ad80 0x55636a8fdb40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f98ec034010 tx=0x7f98ec00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c72e7d92ddb5a0a5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.425+0000 7f98f2000640 1 -- 172.21.15.134:0/3132766447 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec041040 con 0x55636a51ad80 2024-07-26T11:24:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.425+0000 7f98f2000640 1 -- 172.21.15.134:0/3132766447 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98ec055070 con 0x55636a51ad80 2024-07-26T11:24:51.427 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.425+0000 7f98f2000640 1 -- 172.21.15.134:0/3132766447 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec030450 con 0x55636a51ad80 2024-07-26T11:24:51.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.427+0000 7f990f59a740 1 -- 172.21.15.134:0/3132766447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a51ad80 msgr2=0x55636a8fdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:51.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.427+0000 7f990f59a740 1 --2- 172.21.15.134:0/3132766447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a51ad80 0x55636a8fdb40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f98ec034010 tx=0x7f98ec00b040 comp rx=0 tx=0).stop 2024-07-26T11:24:51.430 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.427+0000 7f990f59a740 1 -- 172.21.15.134:0/3132766447 shutdown_connections 2024-07-26T11:24:51.430 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.428+0000 7f990f59a740 1 --2- 172.21.15.134:0/3132766447 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a5332f0 0x55636a904d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.430 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.428+0000 7f990f59a740 1 --2- 172.21.15.134:0/3132766447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a51ad80 0x55636a8fdb40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.430 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.428+0000 7f990f59a740 1 --2- 172.21.15.134:0/3132766447 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a550300 0x55636a911660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.430 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.428+0000 7f990f59a740 1 -- 172.21.15.134:0/3132766447 >> 172.21.15.134:0/3132766447 conn(0x55636a90b3a0 msgr2=0x55636a8fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:51.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.428+0000 7f990f59a740 1 -- 172.21.15.134:0/3132766447 shutdown_connections 2024-07-26T11:24:51.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.428+0000 7f990f59a740 1 -- 172.21.15.134:0/3132766447 wait complete. 2024-07-26T11:24:51.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.429+0000 7f990f59a740 1 -- start start 2024-07-26T11:24:51.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.429+0000 7f990f59a740 1 Processor -- start 2024-07-26T11:24:51.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.431+0000 7f990f59a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a51ad80 0x55636a912a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.431+0000 7f990f59a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a5332f0 0x55636a912f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.431+0000 7f990f59a740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a550300 0x55636a913490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.431+0000 7f9905c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a51ad80 0x55636a912a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.431+0000 7f9905c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a51ad80 0x55636a912a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58704/0 (socket says 172.21.15.134:58704) 2024-07-26T11:24:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.432+0000 7f990f59a740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f99001b8130 con 0x55636a550300 2024-07-26T11:24:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.432+0000 7f990f59a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f99001b7a60 con 0x55636a5332f0 2024-07-26T11:24:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.432+0000 7f990f59a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f99001b4d50 con 0x55636a51ad80 2024-07-26T11:24:51.434 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.432+0000 7f990f59a740 10 client.0 _ll_register_callbacks cb 0x7f98f400de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:24:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.432+0000 7f9905200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a5332f0 0x55636a912f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.432+0000 7f9905c00640 1 -- 172.21.15.134:0/2653357657 learned_addr learned my addr 172.21.15.134:0/2653357657 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:51.435 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.432+0000 7f9906600640 1 --2- 172.21.15.134:0/2653357657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a550300 0x55636a913490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.433+0000 7f9905c00640 1 -- 172.21.15.134:0/2653357657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a5332f0 msgr2=0x55636a912f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:51.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.434+0000 7f9905c00640 1 --2- 172.21.15.134:0/2653357657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a5332f0 0x55636a912f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.434+0000 7f9905c00640 1 -- 172.21.15.134:0/2653357657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a550300 msgr2=0x55636a913490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:51.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.434+0000 7f9905c00640 1 --2- 172.21.15.134:0/2653357657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a550300 0x55636a913490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:51.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.434+0000 7f9905c00640 1 -- 172.21.15.134:0/2653357657 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55636a98d140 con 0x55636a51ad80 2024-07-26T11:24:51.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.434+0000 7f9906600640 1 --2- 172.21.15.134:0/2653357657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a550300 0x55636a913490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:24:51.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.434+0000 7f9905200640 1 --2- 172.21.15.134:0/2653357657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a5332f0 0x55636a912f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:24:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.437+0000 7f9905c00640 1 --2- 172.21.15.134:0/2653357657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a51ad80 0x55636a912a10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f98f4009610 tx=0x7f98f40a6b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6af8d15bbf100ceb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:51.439 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.437+0000 7f98e7400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:24:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.440+0000 7f98e7400640 1 -- 172.21.15.134:0/2653357657 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f4020070 con 0x55636a51ad80 2024-07-26T11:24:51.442 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.440+0000 7f98e7400640 1 -- 172.21.15.134:0/2653357657 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f98f40a62b0 con 0x55636a51ad80 2024-07-26T11:24:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.440+0000 7f990f59a740 1 -- 172.21.15.134:0/2653357657 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98ec055070 con 0x55636a51ad80 2024-07-26T11:24:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.441+0000 7f98e7400640 1 -- 172.21.15.134:0/2653357657 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f98f40a9dd0 con 0x55636a51ad80 2024-07-26T11:24:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.441+0000 7f990f59a740 10 client.26734 fetch_fsmap learned FSMap version 32 2024-07-26T11:24:51.443 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.441+0000 7f990f59a740 1 -- 172.21.15.134:0/2653357657 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55636a98e360 con 0x55636a51ad80 2024-07-26T11:24:51.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.444+0000 7f98e7400640 1 -- 172.21.15.134:0/2653357657 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f98ec055070 con 0x55636a51ad80 2024-07-26T11:24:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.444+0000 7f98e7400640 1 -- 172.21.15.134:0/2653357657 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f98f40a9330 con 0x55636a51ad80 2024-07-26T11:24:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.445+0000 7f990f59a740 10 client.26734 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:24:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.445+0000 7f990f59a740 10 client.26734 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:24:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.445+0000 7f990f59a740 20 client.26734 populate_metadata read hostname 'smithi134' 2024-07-26T11:24:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.445+0000 7f990f59a740 1 --2- 172.21.15.134:0/2653357657 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55636a55c1b0 0x55636a990990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.445+0000 7f990f59a740 4 client.26734 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:24:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.445+0000 7f990f59a740 1 -- 172.21.15.134:0/2653357657 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55636a86f3a0 con 0x55636a55c1b0 2024-07-26T11:24:51.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.445+0000 7f9905200640 1 --2- 172.21.15.134:0/2653357657 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55636a55c1b0 0x55636a990990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:51.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.450+0000 7f9905200640 1 --2- 172.21.15.134:0/2653357657 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55636a55c1b0 0x55636a990990 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=626ef772bcb4cbcf server_cookie=4c0810951ad25ea8 in_seq=0 out_seq=0 2024-07-26T11:24:51.453 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:51.450+0000 7f98e7400640 10 client.26734 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:24:52.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.095+0000 7f98e7400640 1 -- 172.21.15.134:0/2653357657 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55636a86f3a0 con 0x55636a55c1b0 2024-07-26T11:24:52.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.095+0000 7f98e7400640 10 client.26734 handle_command_reply: tid=0 2024-07-26T11:24:52.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.095+0000 7f990f59a740 1 client.26734 shutdown 2024-07-26T11:24:52.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.102+0000 7f990f59a740 1 -- 172.21.15.134:0/2653357657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a51ad80 msgr2=0x55636a912a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:52.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.102+0000 7f990f59a740 1 --2- 172.21.15.134:0/2653357657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a51ad80 0x55636a912a10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f98f4009610 tx=0x7f98f40a6b20 comp rx=0 tx=0).stop 2024-07-26T11:24:52.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.102+0000 7f990f59a740 1 -- 172.21.15.134:0/2653357657 shutdown_connections 2024-07-26T11:24:52.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.102+0000 7f990f59a740 1 --2- 172.21.15.134:0/2653357657 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55636a55c1b0 0x55636a990990 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:52.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.102+0000 7f990f59a740 1 --2- 172.21.15.134:0/2653357657 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55636a550300 0x55636a913490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:52.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.102+0000 7f98e7400640 0 client.26734 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:24:52.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.102+0000 7f990f59a740 1 --2- 172.21.15.134:0/2653357657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55636a5332f0 0x55636a912f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:52.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.102+0000 7f990f59a740 1 --2- 172.21.15.134:0/2653357657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55636a51ad80 0x55636a912a10 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:52.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.102+0000 7f990f59a740 1 -- 172.21.15.134:0/2653357657 >> 172.21.15.134:0/2653357657 conn(0x55636a90b3a0 msgr2=0x55636a8fd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:52.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.103+0000 7f990f59a740 1 -- 172.21.15.134:0/2653357657 shutdown_connections 2024-07-26T11:24:52.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.104+0000 7f990f59a740 1 -- 172.21.15.134:0/2653357657 wait complete. 2024-07-26T11:24:52.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.105+0000 7f990f59a740 20 client.26734 trim_cache size 0 max 16384 2024-07-26T11:24:52.114 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:24:52.114 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:24:52.114 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:24:52.114 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:24:52.115 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.113+0000 7f9907000640 1 -- 172.21.15.134:0/1826921299 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98e80b3df0 msgr2=0x7f98e80b6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:52.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.114+0000 7f9907000640 1 --2- 172.21.15.134:0/1826921299 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98e80b3df0 0x7f98e80b6300 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f98f800a510 tx=0x7f98f80096e0 comp rx=0 tx=0).stop 2024-07-26T11:24:52.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.114+0000 7f9907000640 1 -- 172.21.15.134:0/1826921299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9900109df0 msgr2=0x7f99001a2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:52.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.114+0000 7f9907000640 1 --2- 172.21.15.134:0/1826921299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9900109df0 0x7f99001a2420 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f98f400d770 tx=0x7f98f400dc30 comp rx=0 tx=0).stop 2024-07-26T11:24:52.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.115+0000 7f9907000640 1 -- 172.21.15.134:0/1826921299 shutdown_connections 2024-07-26T11:24:52.117 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.115+0000 7f9907000640 1 --2- 172.21.15.134:0/1826921299 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f98e80b3df0 0x7f98e80b6300 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:52.117 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.115+0000 7f9907000640 1 --2- 172.21.15.134:0/1826921299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9900115f40 0x7f99001a7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:52.117 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.115+0000 7f9907000640 1 --2- 172.21.15.134:0/1826921299 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f990010a740 0x7f99001a29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:52.117 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.115+0000 7f9907000640 1 --2- 172.21.15.134:0/1826921299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9900109df0 0x7f99001a2420 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:52.117 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.115+0000 7f9907000640 1 -- 172.21.15.134:0/1826921299 >> 172.21.15.134:0/1826921299 conn(0x7f9900070630 msgr2=0x7f990010d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.115+0000 7f9907000640 1 -- 172.21.15.134:0/1826921299 shutdown_connections 2024-07-26T11:24:52.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:24:52.116+0000 7f9907000640 1 -- 172.21.15.134:0/1826921299 wait complete. 2024-07-26T11:24:54.001 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:53 smithi134 ceph-mon[31966]: pgmap v751: 129 pgs: 129 active+clean; 59 GiB data, 124 GiB used, 949 GiB / 1.0 TiB avail; 57 KiB/s rd, 204 MiB/s wr, 1.37k op/s 2024-07-26T11:24:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:53 smithi193 ceph-mon[37473]: pgmap v751: 129 pgs: 129 active+clean; 59 GiB data, 124 GiB used, 949 GiB / 1.0 TiB avail; 57 KiB/s rd, 204 MiB/s wr, 1.37k op/s 2024-07-26T11:24:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:53 smithi189 ceph-mon[36790]: pgmap v751: 129 pgs: 129 active+clean; 59 GiB data, 124 GiB used, 949 GiB / 1.0 TiB avail; 57 KiB/s rd, 204 MiB/s wr, 1.37k op/s 2024-07-26T11:24:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:55 smithi134 ceph-mon[31966]: pgmap v752: 129 pgs: 129 active+clean; 59 GiB data, 125 GiB used, 947 GiB / 1.0 TiB avail; 45 KiB/s rd, 247 MiB/s wr, 1.65k op/s 2024-07-26T11:24:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:55 smithi193 ceph-mon[37473]: pgmap v752: 129 pgs: 129 active+clean; 59 GiB data, 125 GiB used, 947 GiB / 1.0 TiB avail; 45 KiB/s rd, 247 MiB/s wr, 1.65k op/s 2024-07-26T11:24:56.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:55 smithi189 ceph-mon[36790]: pgmap v752: 129 pgs: 129 active+clean; 59 GiB data, 125 GiB used, 947 GiB / 1.0 TiB avail; 45 KiB/s rd, 247 MiB/s wr, 1.65k op/s 2024-07-26T11:24:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:56 smithi134 ceph-mon[31966]: pgmap v753: 129 pgs: 129 active+clean; 60 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 44 KiB/s rd, 230 MiB/s wr, 1.47k op/s 2024-07-26T11:24:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:56 smithi189 ceph-mon[36790]: pgmap v753: 129 pgs: 129 active+clean; 60 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 44 KiB/s rd, 230 MiB/s wr, 1.47k op/s 2024-07-26T11:24:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:56 smithi193 ceph-mon[37473]: pgmap v753: 129 pgs: 129 active+clean; 60 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 44 KiB/s rd, 230 MiB/s wr, 1.47k op/s 2024-07-26T11:24:57.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.624+0000 7f59be400640 1 -- 172.21.15.134:0/4049708919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59b81035b0 msgr2=0x7f59b81039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:57.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.624+0000 7f59be400640 1 --2- 172.21.15.134:0/4049708919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59b81035b0 0x7f59b81039c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f59a8009900 tx=0x7f59a802a410 comp rx=0 tx=0).stop 2024-07-26T11:24:57.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.625+0000 7f59be400640 1 -- 172.21.15.134:0/4049708919 shutdown_connections 2024-07-26T11:24:57.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.625+0000 7f59be400640 1 --2- 172.21.15.134:0/4049708919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59b8108ef0 0x7f59b8109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.627 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.625+0000 7f59be400640 1 --2- 172.21.15.134:0/4049708919 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59b8109840 0x7f59b8110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.625+0000 7f59be400640 1 --2- 172.21.15.134:0/4049708919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59b81035b0 0x7f59b81039c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.625+0000 7f59be400640 1 -- 172.21.15.134:0/4049708919 >> 172.21.15.134:0/4049708919 conn(0x7f59b80fefc0 msgr2=0x7f59b8101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:57.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.626+0000 7f59be400640 1 -- 172.21.15.134:0/4049708919 shutdown_connections 2024-07-26T11:24:57.628 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.626+0000 7f59be400640 1 -- 172.21.15.134:0/4049708919 wait complete. 2024-07-26T11:24:57.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.628+0000 7f59be400640 1 Processor -- start 2024-07-26T11:24:57.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.628+0000 7f59be400640 1 -- start start 2024-07-26T11:24:57.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.628+0000 7f59be400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59b81035b0 0x7f59b810d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.629+0000 7f59be400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59b8108ef0 0x7f59b810daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.629+0000 7f59be400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59b8109840 0x7f59b810dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.629+0000 7f59be400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f59b8076b10 con 0x7f59b8109840 2024-07-26T11:24:57.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.629+0000 7f59be400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f59b8076990 con 0x7f59b8108ef0 2024-07-26T11:24:57.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.629+0000 7f59be400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f59b8076c90 con 0x7f59b81035b0 2024-07-26T11:24:57.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.629+0000 7f59b7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59b81035b0 0x7f59b810d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:57.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.629+0000 7f59b7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59b81035b0 0x7f59b810d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58716/0 (socket says 172.21.15.134:58716) 2024-07-26T11:24:57.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.629+0000 7f59b7400640 1 -- 172.21.15.134:0/2727902855 learned_addr learned my addr 172.21.15.134:0/2727902855 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:57.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.630+0000 7f59b7e00640 1 --2- 172.21.15.134:0/2727902855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59b8109840 0x7f59b810dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:57.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.630+0000 7f59b7400640 1 -- 172.21.15.134:0/2727902855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59b8108ef0 msgr2=0x7f59b810daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:57.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.630+0000 7f59b7400640 1 --2- 172.21.15.134:0/2727902855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59b8108ef0 0x7f59b810daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.630+0000 7f59b7400640 1 -- 172.21.15.134:0/2727902855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59b8109840 msgr2=0x7f59b810dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:57.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.630+0000 7f59b7400640 1 --2- 172.21.15.134:0/2727902855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59b8109840 0x7f59b810dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.630+0000 7f59b7400640 1 -- 172.21.15.134:0/2727902855 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b81ac5a0 con 0x7f59b81035b0 2024-07-26T11:24:57.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.631+0000 7f59b7400640 1 --2- 172.21.15.134:0/2727902855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59b81035b0 0x7f59b810d560 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f59a8005ab0 tx=0x7f59a8003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:57.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.632+0000 7f59b4c00640 1 -- 172.21.15.134:0/2727902855 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a803f070 con 0x7f59b81035b0 2024-07-26T11:24:57.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.632+0000 7f59b4c00640 1 -- 172.21.15.134:0/2727902855 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59a8031d60 con 0x7f59b81035b0 2024-07-26T11:24:57.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.632+0000 7f59b4c00640 1 -- 172.21.15.134:0/2727902855 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a8035030 con 0x7f59b81035b0 2024-07-26T11:24:57.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.632+0000 7f59be400640 1 -- 172.21.15.134:0/2727902855 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59b81ac830 con 0x7f59b81035b0 2024-07-26T11:24:57.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.632+0000 7f59be400640 1 -- 172.21.15.134:0/2727902855 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59b81accc0 con 0x7f59b81035b0 2024-07-26T11:24:57.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.638+0000 7f59b4c00640 1 -- 172.21.15.134:0/2727902855 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f59a8042070 con 0x7f59b81035b0 2024-07-26T11:24:57.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.640+0000 7f59b4c00640 1 --2- 172.21.15.134:0/2727902855 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f599c0b3f10 0x7f599c0b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.640+0000 7f59b4c00640 1 -- 172.21.15.134:0/2727902855 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f59a81047c0 con 0x7f59b81035b0 2024-07-26T11:24:57.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.641+0000 7f59aea00640 1 --2- 172.21.15.134:0/2727902855 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f599c0b3f10 0x7f599c0b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:57.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.642+0000 7f59aea00640 1 --2- 172.21.15.134:0/2727902855 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f599c0b3f10 0x7f599c0b6420 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f59b81017c0 tx=0x7f59a0009710 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:57.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.667+0000 7f59c5217740 1 Processor -- start 2024-07-26T11:24:57.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.667+0000 7f59c5217740 1 -- start start 2024-07-26T11:24:57.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.667+0000 7f59c5217740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80cb070 0x55d2b846fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.667+0000 7f59c5217740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80a21b0 0x55d2b8470390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.667+0000 7f59c5217740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80aa390 0x55d2b84708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.668+0000 7f59c5217740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d2b83e6bf0 con 0x55d2b80cb070 2024-07-26T11:24:57.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.668+0000 7f59c5217740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d2b83d60f0 con 0x55d2b80a21b0 2024-07-26T11:24:57.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.668+0000 7f59c5217740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d2b83da130 con 0x55d2b80aa390 2024-07-26T11:24:57.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.668+0000 7f59b7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80aa390 0x55d2b84708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:57.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.668+0000 7f59b7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80aa390 0x55d2b84708d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58724/0 (socket says 172.21.15.134:58724) 2024-07-26T11:24:57.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.668+0000 7f59b7e00640 1 -- 172.21.15.134:0/62308105 learned_addr learned my addr 172.21.15.134:0/62308105 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:57.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.668+0000 7f59b7400640 1 --2- 172.21.15.134:0/62308105 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80cb070 0x55d2b846fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:57.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.668+0000 7f59b7e00640 1 -- 172.21.15.134:0/62308105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80a21b0 msgr2=0x55d2b8470390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:24:57.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.668+0000 7f59b7e00640 1 --2- 172.21.15.134:0/62308105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80a21b0 0x55d2b8470390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.668+0000 7f59b7e00640 1 -- 172.21.15.134:0/62308105 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80cb070 msgr2=0x55d2b846fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:57.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.668+0000 7f59b7e00640 1 --2- 172.21.15.134:0/62308105 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80cb070 0x55d2b846fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.668+0000 7f59b7e00640 1 -- 172.21.15.134:0/62308105 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2b8136650 con 0x55d2b80aa390 2024-07-26T11:24:57.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.669+0000 7f59b7e00640 1 --2- 172.21.15.134:0/62308105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80aa390 0x55d2b84708d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f59b000b6b0 tx=0x7f59b000bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cdcaf88459df637 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:57.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.669+0000 7f59acc00640 1 -- 172.21.15.134:0/62308105 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b0003bd0 con 0x55d2b80aa390 2024-07-26T11:24:57.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.669+0000 7f59acc00640 1 -- 172.21.15.134:0/62308105 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59b0026070 con 0x55d2b80aa390 2024-07-26T11:24:57.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.669+0000 7f59acc00640 1 -- 172.21.15.134:0/62308105 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b00226b0 con 0x55d2b80aa390 2024-07-26T11:24:57.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.670+0000 7f59c5217740 1 -- 172.21.15.134:0/62308105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80aa390 msgr2=0x55d2b84708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:57.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.670+0000 7f59c5217740 1 --2- 172.21.15.134:0/62308105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80aa390 0x55d2b84708d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f59b000b6b0 tx=0x7f59b000bb70 comp rx=0 tx=0).stop 2024-07-26T11:24:57.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.670+0000 7f59c5217740 1 -- 172.21.15.134:0/62308105 shutdown_connections 2024-07-26T11:24:57.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.670+0000 7f59c5217740 1 --2- 172.21.15.134:0/62308105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80aa390 0x55d2b84708d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.670+0000 7f59c5217740 1 --2- 172.21.15.134:0/62308105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80a21b0 0x55d2b8470390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.670+0000 7f59c5217740 1 --2- 172.21.15.134:0/62308105 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80cb070 0x55d2b846fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.670+0000 7f59c5217740 1 -- 172.21.15.134:0/62308105 >> 172.21.15.134:0/62308105 conn(0x55d2b80bf460 msgr2=0x55d2b846dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:57.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.670+0000 7f59c5217740 1 -- 172.21.15.134:0/62308105 shutdown_connections 2024-07-26T11:24:57.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.670+0000 7f59c5217740 1 -- 172.21.15.134:0/62308105 wait complete. 2024-07-26T11:24:57.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.671+0000 7f59c5217740 1 -- start start 2024-07-26T11:24:57.674 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.671+0000 7f59c5217740 1 Processor -- start 2024-07-26T11:24:57.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.672+0000 7f59c5217740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80a21b0 0x55d2b847b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.672+0000 7f59c5217740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80aa390 0x55d2b847ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.672+0000 7f59c5217740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80cb070 0x55d2b847bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.672+0000 7f59c5217740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d2b8239700 con 0x55d2b80a21b0 2024-07-26T11:24:57.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.672+0000 7f59c5217740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d2b83726c0 con 0x55d2b80aa390 2024-07-26T11:24:57.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.672+0000 7f59c5217740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d2b847e680 con 0x55d2b80cb070 2024-07-26T11:24:57.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.672+0000 7f59c5217740 10 client.0 _ll_register_callbacks cb 0x55d2b8382170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:24:57.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.672+0000 7f59aea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80aa390 0x55d2b847ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:57.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.672+0000 7f59aea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80aa390 0x55d2b847ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:46822/0 (socket says 172.21.15.134:46822) 2024-07-26T11:24:57.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.672+0000 7f59aea00640 1 -- 172.21.15.134:0/4291277152 learned_addr learned my addr 172.21.15.134:0/4291277152 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:57.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.672+0000 7f59b7e00640 1 --2- 172.21.15.134:0/4291277152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80cb070 0x55d2b847bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:57.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.672+0000 7f59b7400640 1 --2- 172.21.15.134:0/4291277152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80a21b0 0x55d2b847b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:57.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.673+0000 7f59b7e00640 1 -- 172.21.15.134:0/4291277152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80aa390 msgr2=0x55d2b847ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:57.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.673+0000 7f59b7e00640 1 --2- 172.21.15.134:0/4291277152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80aa390 0x55d2b847ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.673+0000 7f59b7e00640 1 -- 172.21.15.134:0/4291277152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80a21b0 msgr2=0x55d2b847b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:57.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.673+0000 7f59b7e00640 1 --2- 172.21.15.134:0/4291277152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80a21b0 0x55d2b847b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.673+0000 7f59b7e00640 1 -- 172.21.15.134:0/4291277152 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2b8389860 con 0x55d2b80cb070 2024-07-26T11:24:57.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.673+0000 7f59b7e00640 1 --2- 172.21.15.134:0/4291277152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80cb070 0x55d2b847bf70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f59b0000c00 tx=0x7f59b00042d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c446e5ea8428cf1b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:57.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.673+0000 7f59a5600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:24:57.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.673+0000 7f59a5600640 1 -- 172.21.15.134:0/4291277152 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b003b060 con 0x55d2b80cb070 2024-07-26T11:24:57.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.673+0000 7f59c5217740 1 -- 172.21.15.134:0/4291277152 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59b0026070 con 0x55d2b80cb070 2024-07-26T11:24:57.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.673+0000 7f59a5600640 1 -- 172.21.15.134:0/4291277152 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f59b002b470 con 0x55d2b80cb070 2024-07-26T11:24:57.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.674+0000 7f59a5600640 1 -- 172.21.15.134:0/4291277152 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f59b0042050 con 0x55d2b80cb070 2024-07-26T11:24:57.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.674+0000 7f59c5217740 10 client.26746 fetch_fsmap learned FSMap version 32 2024-07-26T11:24:57.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.674+0000 7f59c5217740 1 -- 172.21.15.134:0/4291277152 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d2b848a300 con 0x55d2b80cb070 2024-07-26T11:24:57.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.674+0000 7f59a5600640 1 -- 172.21.15.134:0/4291277152 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59b0026070 con 0x55d2b80cb070 2024-07-26T11:24:57.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.674+0000 7f59a5600640 1 -- 172.21.15.134:0/4291277152 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f59b00229e0 con 0x55d2b80cb070 2024-07-26T11:24:57.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.674+0000 7f59c5217740 10 client.26746 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:24:57.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.674+0000 7f59c5217740 10 client.26746 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-07-26T11:24:57.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.675+0000 7f59c5217740 20 client.26746 populate_metadata read hostname 'smithi134' 2024-07-26T11:24:57.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.675+0000 7f59c5217740 1 --2- 172.21.15.134:0/4291277152 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55d2b80899f0 0x55d2b848d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.675+0000 7f59c5217740 4 client.26746 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:24:57.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.675+0000 7f59c5217740 1 -- 172.21.15.134:0/4291277152 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2b83e8b40 con 0x55d2b80899f0 2024-07-26T11:24:57.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.675+0000 7f59b7400640 1 --2- 172.21.15.134:0/4291277152 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55d2b80899f0 0x55d2b848d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:57.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.676+0000 7f59b7400640 1 --2- 172.21.15.134:0/4291277152 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55d2b80899f0 0x55d2b848d6c0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=963b53edc91b7271 server_cookie=edb2a1639e3c5737 in_seq=0 out_seq=0 2024-07-26T11:24:57.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.676+0000 7f59a5600640 10 client.26746 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:24:57.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.676+0000 7f59a5600640 1 -- 172.21.15.134:0/4291277152 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d2b83e8b40 con 0x55d2b80899f0 2024-07-26T11:24:57.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.676+0000 7f59a5600640 10 client.26746 handle_command_reply: tid=0 2024-07-26T11:24:57.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.676+0000 7f59c5217740 1 client.26746 shutdown 2024-07-26T11:24:57.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.679+0000 7f59c5217740 1 -- 172.21.15.134:0/4291277152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80cb070 msgr2=0x55d2b847bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:57.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.679+0000 7f59c5217740 1 --2- 172.21.15.134:0/4291277152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80cb070 0x55d2b847bf70 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f59b0000c00 tx=0x7f59b00042d0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.679+0000 7f59c5217740 1 -- 172.21.15.134:0/4291277152 shutdown_connections 2024-07-26T11:24:57.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.679+0000 7f59c5217740 1 --2- 172.21.15.134:0/4291277152 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55d2b80899f0 0x55d2b848d6c0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.679+0000 7f59c5217740 1 --2- 172.21.15.134:0/4291277152 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80cb070 0x55d2b847bf70 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.679+0000 7f59c5217740 1 --2- 172.21.15.134:0/4291277152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80aa390 0x55d2b847ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.679+0000 7f59c5217740 1 --2- 172.21.15.134:0/4291277152 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80a21b0 0x55d2b847b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.679+0000 7f59c5217740 1 -- 172.21.15.134:0/4291277152 >> 172.21.15.134:0/4291277152 conn(0x55d2b80bf460 msgr2=0x55d2b8473030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:57.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.679+0000 7f59a5600640 0 client.26746 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:24:57.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.679+0000 7f59c5217740 1 -- 172.21.15.134:0/4291277152 shutdown_connections 2024-07-26T11:24:57.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.679+0000 7f59c5217740 1 -- 172.21.15.134:0/4291277152 wait complete. 2024-07-26T11:24:57.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.679+0000 7f59c5217740 20 client.26746 trim_cache size 0 max 16384 2024-07-26T11:24:57.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.722+0000 7f59c5217740 1 Processor -- start 2024-07-26T11:24:57.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.722+0000 7f59c5217740 1 -- start start 2024-07-26T11:24:57.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.722+0000 7f59c5217740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80a21b0 0x55d2b8471950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.722+0000 7f59c5217740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80aa390 0x55d2b847d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.722+0000 7f59c5217740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80cb070 0x55d2b847b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.722+0000 7f59c5217740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f59b81ae810 con 0x55d2b80a21b0 2024-07-26T11:24:57.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.722+0000 7f59c5217740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f59b81b0350 con 0x55d2b80aa390 2024-07-26T11:24:57.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.722+0000 7f59c5217740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f59b81aeee0 con 0x55d2b80cb070 2024-07-26T11:24:57.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.723+0000 7f59b7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80cb070 0x55d2b847b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:57.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.723+0000 7f59b7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80cb070 0x55d2b847b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:58732/0 (socket says 172.21.15.134:58732) 2024-07-26T11:24:57.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.723+0000 7f59b7e00640 1 -- 172.21.15.134:0/3765582582 learned_addr learned my addr 172.21.15.134:0/3765582582 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:57.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.723+0000 7f59aea00640 1 --2- 172.21.15.134:0/3765582582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80aa390 0x55d2b847d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:57.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.723+0000 7f59b7e00640 1 -- 172.21.15.134:0/3765582582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80aa390 msgr2=0x55d2b847d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:57.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.723+0000 7f59b7e00640 1 --2- 172.21.15.134:0/3765582582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80aa390 0x55d2b847d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.723+0000 7f59b7e00640 1 -- 172.21.15.134:0/3765582582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80a21b0 msgr2=0x55d2b8471950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:57.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.724+0000 7f59b7e00640 1 --2- 172.21.15.134:0/3765582582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80a21b0 0x55d2b8471950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.724+0000 7f59b7e00640 1 -- 172.21.15.134:0/3765582582 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2b847c790 con 0x55d2b80cb070 2024-07-26T11:24:57.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.724+0000 7f59b7e00640 1 --2- 172.21.15.134:0/3765582582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80cb070 0x55d2b847b990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f59b000c010 tx=0x7f59b0036380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7c89b314769fa84 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:57.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.724+0000 7f59a7e00640 1 -- 172.21.15.134:0/3765582582 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b003b020 con 0x55d2b80cb070 2024-07-26T11:24:57.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.725+0000 7f59a7e00640 1 -- 172.21.15.134:0/3765582582 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59b0026040 con 0x55d2b80cb070 2024-07-26T11:24:57.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.725+0000 7f59a7e00640 1 -- 172.21.15.134:0/3765582582 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b00052d0 con 0x55d2b80cb070 2024-07-26T11:24:57.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.726+0000 7f59c5217740 1 -- 172.21.15.134:0/3765582582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80cb070 msgr2=0x55d2b847b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:57.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.726+0000 7f59c5217740 1 --2- 172.21.15.134:0/3765582582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80cb070 0x55d2b847b990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f59b000c010 tx=0x7f59b0036380 comp rx=0 tx=0).stop 2024-07-26T11:24:57.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.727+0000 7f59c5217740 1 -- 172.21.15.134:0/3765582582 shutdown_connections 2024-07-26T11:24:57.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.727+0000 7f59c5217740 1 --2- 172.21.15.134:0/3765582582 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80cb070 0x55d2b847b990 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.727+0000 7f59c5217740 1 --2- 172.21.15.134:0/3765582582 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80aa390 0x55d2b847d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.727+0000 7f59c5217740 1 --2- 172.21.15.134:0/3765582582 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80a21b0 0x55d2b8471950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.727+0000 7f59c5217740 1 -- 172.21.15.134:0/3765582582 >> 172.21.15.134:0/3765582582 conn(0x55d2b80bf460 msgr2=0x55d2b847b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:57.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.728+0000 7f59c5217740 1 -- 172.21.15.134:0/3765582582 shutdown_connections 2024-07-26T11:24:57.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.728+0000 7f59c5217740 1 -- 172.21.15.134:0/3765582582 wait complete. 2024-07-26T11:24:57.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.729+0000 7f59c5217740 1 -- start start 2024-07-26T11:24:57.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.730+0000 7f59c5217740 1 Processor -- start 2024-07-26T11:24:57.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.731+0000 7f59c5217740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80a21b0 0x55d2b8475440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.732+0000 7f59c5217740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80aa390 0x55d2b8475980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.732+0000 7f59c5217740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80cb070 0x55d2b8475ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.733+0000 7f59c5217740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f59b81afc80 con 0x55d2b80cb070 2024-07-26T11:24:57.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.733+0000 7f59c5217740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f59b81af5b0 con 0x55d2b80aa390 2024-07-26T11:24:57.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.733+0000 7f59c5217740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f59b8074910 con 0x55d2b80a21b0 2024-07-26T11:24:57.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.733+0000 7f59c5217740 10 client.0 _ll_register_callbacks cb 0x7f59b002bb80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:24:57.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.733+0000 7f59b7e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80cb070 0x55d2b8475ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:57.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.734+0000 7f59b7e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80cb070 0x55d2b8475ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:43126/0 (socket says 172.21.15.134:43126) 2024-07-26T11:24:57.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.734+0000 7f59b7e00640 1 -- 172.21.15.134:0/218385865 learned_addr learned my addr 172.21.15.134:0/218385865 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:24:57.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.734+0000 7f59b7e00640 1 -- 172.21.15.134:0/218385865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80a21b0 msgr2=0x55d2b8475440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:24:57.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.734+0000 7f59b7400640 1 --2- 172.21.15.134:0/218385865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80a21b0 0x55d2b8475440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:57.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.734+0000 7f59b7e00640 1 --2- 172.21.15.134:0/218385865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80a21b0 0x55d2b8475440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.734+0000 7f59b7e00640 1 -- 172.21.15.134:0/218385865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80aa390 msgr2=0x55d2b8475980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:24:57.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.734+0000 7f59b7e00640 1 --2- 172.21.15.134:0/218385865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80aa390 0x55d2b8475980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.734+0000 7f59b7e00640 1 -- 172.21.15.134:0/218385865 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2b8482a50 con 0x55d2b80cb070 2024-07-26T11:24:57.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.735+0000 7f59b7400640 1 --2- 172.21.15.134:0/218385865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80a21b0 0x55d2b8475440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:24:57.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.735+0000 7f59b7e00640 1 --2- 172.21.15.134:0/218385865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80cb070 0x55d2b8475ec0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f59b002be60 tx=0x7f59b0036680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e638a479888df94 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:24:57.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.736+0000 7f59a5600640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:24:57.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.736+0000 7f59a5600640 1 -- 172.21.15.134:0/218385865 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b004b060 con 0x55d2b80cb070 2024-07-26T11:24:57.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.737+0000 7f59a5600640 1 -- 172.21.15.134:0/218385865 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f59b0033050 con 0x55d2b80cb070 2024-07-26T11:24:57.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.737+0000 7f59c5217740 1 -- 172.21.15.134:0/218385865 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59b0026040 con 0x55d2b80cb070 2024-07-26T11:24:57.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.740+0000 7f59a5600640 1 -- 172.21.15.134:0/218385865 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f59b004b060 con 0x55d2b80cb070 2024-07-26T11:24:57.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.740+0000 7f59c5217740 10 client.16134 fetch_fsmap learned FSMap version 32 2024-07-26T11:24:57.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.740+0000 7f59c5217740 1 -- 172.21.15.134:0/218385865 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d2b8483cd0 con 0x55d2b80cb070 2024-07-26T11:24:57.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.742+0000 7f59a5600640 1 -- 172.21.15.134:0/218385865 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59b0026040 con 0x55d2b80cb070 2024-07-26T11:24:57.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.742+0000 7f59a5600640 1 -- 172.21.15.134:0/218385865 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f59b0005060 con 0x55d2b80cb070 2024-07-26T11:24:57.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.743+0000 7f59c5217740 10 client.16134 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:24:57.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.743+0000 7f59c5217740 10 client.16134 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-07-26T11:24:57.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.744+0000 7f59c5217740 20 client.16134 populate_metadata read hostname 'smithi134' 2024-07-26T11:24:57.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.744+0000 7f59c5217740 1 --2- 172.21.15.134:0/218385865 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55d2b805eda0 0x55d2b84f7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:24:57.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.744+0000 7f59c5217740 4 client.16134 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:24:57.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.744+0000 7f59c5217740 1 -- 172.21.15.134:0/218385865 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d2b83e45e0 con 0x55d2b805eda0 2024-07-26T11:24:57.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.748+0000 7f59b7400640 1 --2- 172.21.15.134:0/218385865 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55d2b805eda0 0x55d2b84f7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:24:57.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.748+0000 7f59b7400640 1 --2- 172.21.15.134:0/218385865 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55d2b805eda0 0x55d2b84f7cd0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3a65a1d1e32ffbe7 server_cookie=b6876d699025656a in_seq=0 out_seq=0 2024-07-26T11:24:57.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.748+0000 7f59a5600640 10 client.16134 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:24:57.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.749+0000 7f59a5600640 1 -- 172.21.15.134:0/218385865 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d2b83e45e0 con 0x55d2b805eda0 2024-07-26T11:24:57.751 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.749+0000 7f59a5600640 10 client.16134 handle_command_reply: tid=0 2024-07-26T11:24:57.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.749+0000 7f59c5217740 1 client.16134 shutdown 2024-07-26T11:24:57.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.749+0000 7f59c5217740 1 -- 172.21.15.134:0/218385865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80cb070 msgr2=0x55d2b8475ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:57.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.749+0000 7f59c5217740 1 --2- 172.21.15.134:0/218385865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80cb070 0x55d2b8475ec0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f59b002be60 tx=0x7f59b0036680 comp rx=0 tx=0).stop 2024-07-26T11:24:57.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.750+0000 7f59c5217740 1 -- 172.21.15.134:0/218385865 shutdown_connections 2024-07-26T11:24:57.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.750+0000 7f59c5217740 1 --2- 172.21.15.134:0/218385865 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55d2b805eda0 0x55d2b84f7cd0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.750+0000 7f59c5217740 1 --2- 172.21.15.134:0/218385865 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d2b80cb070 0x55d2b8475ec0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.750+0000 7f59c5217740 1 --2- 172.21.15.134:0/218385865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2b80aa390 0x55d2b8475980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.750+0000 7f59c5217740 1 --2- 172.21.15.134:0/218385865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d2b80a21b0 0x55d2b8475440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.750+0000 7f59c5217740 1 -- 172.21.15.134:0/218385865 >> 172.21.15.134:0/218385865 conn(0x55d2b80bf460 msgr2=0x55d2b846c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:57.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.750+0000 7f59a5600640 0 client.16134 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:24:57.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.751+0000 7f59c5217740 1 -- 172.21.15.134:0/218385865 shutdown_connections 2024-07-26T11:24:57.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.751+0000 7f59c5217740 1 -- 172.21.15.134:0/218385865 wait complete. 2024-07-26T11:24:57.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.752+0000 7f59c5217740 20 client.16134 trim_cache size 0 max 16384 2024-07-26T11:24:57.754 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:24:57.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.753+0000 7f59be400640 1 -- 172.21.15.134:0/2727902855 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f599c0b3f10 msgr2=0x7f599c0b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:57.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.754+0000 7f59be400640 1 --2- 172.21.15.134:0/2727902855 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f599c0b3f10 0x7f599c0b6420 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f59b81017c0 tx=0x7f59a0009710 comp rx=0 tx=0).stop 2024-07-26T11:24:57.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.754+0000 7f59be400640 1 -- 172.21.15.134:0/2727902855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59b81035b0 msgr2=0x7f59b810d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:24:57.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.754+0000 7f59be400640 1 --2- 172.21.15.134:0/2727902855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59b81035b0 0x7f59b810d560 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f59a8005ab0 tx=0x7f59a8003cd0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.755+0000 7f59be400640 1 -- 172.21.15.134:0/2727902855 shutdown_connections 2024-07-26T11:24:57.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.755+0000 7f59be400640 1 --2- 172.21.15.134:0/2727902855 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f599c0b3f10 0x7f599c0b6420 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.755+0000 7f59be400640 1 --2- 172.21.15.134:0/2727902855 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f59b8109840 0x7f59b810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.755+0000 7f59be400640 1 --2- 172.21.15.134:0/2727902855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59b8108ef0 0x7f59b810daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.755+0000 7f59be400640 1 --2- 172.21.15.134:0/2727902855 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59b81035b0 0x7f59b810d560 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:24:57.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.755+0000 7f59be400640 1 -- 172.21.15.134:0/2727902855 >> 172.21.15.134:0/2727902855 conn(0x7f59b80fefc0 msgr2=0x7f59b8102420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:24:57.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.756+0000 7f59be400640 1 -- 172.21.15.134:0/2727902855 shutdown_connections 2024-07-26T11:24:57.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:24:57.757+0000 7f59be400640 1 -- 172.21.15.134:0/2727902855 wait complete. 2024-07-26T11:24:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:24:59 smithi189 ceph-mon[36790]: pgmap v754: 129 pgs: 129 active+clean; 60 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 56 KiB/s rd, 241 MiB/s wr, 1.51k op/s 2024-07-26T11:24:59.976 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:24:59 smithi193 ceph-mon[37473]: pgmap v754: 129 pgs: 129 active+clean; 60 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 56 KiB/s rd, 241 MiB/s wr, 1.51k op/s 2024-07-26T11:25:00.178 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:24:59 smithi134 ceph-mon[31966]: pgmap v754: 129 pgs: 129 active+clean; 60 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 56 KiB/s rd, 241 MiB/s wr, 1.51k op/s 2024-07-26T11:25:02.003 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:01 smithi134 ceph-mon[31966]: pgmap v755: 129 pgs: 129 active+clean; 60 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 38 KiB/s rd, 276 MiB/s wr, 1.71k op/s 2024-07-26T11:25:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:01 smithi193 ceph-mon[37473]: pgmap v755: 129 pgs: 129 active+clean; 60 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 38 KiB/s rd, 276 MiB/s wr, 1.71k op/s 2024-07-26T11:25:02.109 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-26T11:25:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:01 smithi189 ceph-mon[36790]: pgmap v755: 129 pgs: 129 active+clean; 60 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 38 KiB/s rd, 276 MiB/s wr, 1.71k op/s 2024-07-26T11:25:02.947 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:25:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:03 smithi193 ceph-mon[37473]: pgmap v756: 129 pgs: 129 active+clean; 61 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 57 KiB/s rd, 218 MiB/s wr, 1.38k op/s 2024-07-26T11:25:04.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:03 smithi134 ceph-mon[31966]: pgmap v756: 129 pgs: 129 active+clean; 61 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 57 KiB/s rd, 218 MiB/s wr, 1.38k op/s 2024-07-26T11:25:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:03 smithi189 ceph-mon[36790]: pgmap v756: 129 pgs: 129 active+clean; 61 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 57 KiB/s rd, 218 MiB/s wr, 1.38k op/s 2024-07-26T11:25:05.726 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.724+0000 7f68afe00640 1 -- 172.21.15.134:0/4061298195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68a810b460 msgr2=0x7f68a810b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:05.726 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.724+0000 7f68afe00640 1 --2- 172.21.15.134:0/4061298195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68a810b460 0x7f68a810b870 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6898009860 tx=0x7f689802a340 comp rx=0 tx=0).stop 2024-07-26T11:25:05.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.728+0000 7f68afe00640 1 -- 172.21.15.134:0/4061298195 shutdown_connections 2024-07-26T11:25:05.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.728+0000 7f68afe00640 1 --2- 172.21.15.134:0/4061298195 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68a810f430 0x7f68a807b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.728+0000 7f68afe00640 1 --2- 172.21.15.134:0/4061298195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a810f880 0x7f68a810fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.728+0000 7f68afe00640 1 --2- 172.21.15.134:0/4061298195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68a810b460 0x7f68a810b870 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.731 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.728+0000 7f68afe00640 1 -- 172.21.15.134:0/4061298195 >> 172.21.15.134:0/4061298195 conn(0x7f68a8070800 msgr2=0x7f68a8070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:05.742 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.740+0000 7f68afe00640 1 -- 172.21.15.134:0/4061298195 shutdown_connections 2024-07-26T11:25:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.740+0000 7f68afe00640 1 -- 172.21.15.134:0/4061298195 wait complete. 2024-07-26T11:25:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.742+0000 7f68afe00640 1 Processor -- start 2024-07-26T11:25:05.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.742+0000 7f68afe00640 1 -- start start 2024-07-26T11:25:05.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.743+0000 7f68afe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68a810b460 0x7f68a80824f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.743+0000 7f68afe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a810f430 0x7f68a8082a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.744+0000 7f68ad400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68a810b460 0x7f68a80824f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.744+0000 7f68ad400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68a810b460 0x7f68a80824f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:46344/0 (socket says 172.21.15.134:46344) 2024-07-26T11:25:05.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.744+0000 7f68afe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68a810f880 0x7f68a81bf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.744+0000 7f68aca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a810f430 0x7f68a8082a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:05.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.745+0000 7f68afe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f68a80807d0 con 0x7f68a810f880 2024-07-26T11:25:05.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.745+0000 7f68afe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f68a8080650 con 0x7f68a810f430 2024-07-26T11:25:05.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.745+0000 7f68afe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f68a8080950 con 0x7f68a810b460 2024-07-26T11:25:05.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.745+0000 7f68ad400640 1 -- 172.21.15.134:0/3927752010 learned_addr learned my addr 172.21.15.134:0/3927752010 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:05.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.745+0000 7f68ade00640 1 --2- 172.21.15.134:0/3927752010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68a810f880 0x7f68a81bf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:05.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.746+0000 7f68aca00640 1 -- 172.21.15.134:0/3927752010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68a810b460 msgr2=0x7f68a80824f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.746+0000 7f68aca00640 1 --2- 172.21.15.134:0/3927752010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68a810b460 0x7f68a80824f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.747+0000 7f68aca00640 1 -- 172.21.15.134:0/3927752010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68a810f880 msgr2=0x7f68a81bf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.747+0000 7f68aca00640 1 --2- 172.21.15.134:0/3927752010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68a810f880 0x7f68a81bf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.747+0000 7f68aca00640 1 -- 172.21.15.134:0/3927752010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68a81bf7e0 con 0x7f68a810f430 2024-07-26T11:25:05.749 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.747+0000 7f68ad400640 1 --2- 172.21.15.134:0/3927752010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68a810b460 0x7f68a80824f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:25:05.750 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.748+0000 7f68aca00640 1 --2- 172.21.15.134:0/3927752010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a810f430 0x7f68a8082a30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f689c009990 tx=0x7f689c009e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.748+0000 7f68a6000640 1 -- 172.21.15.134:0/3927752010 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c01f070 con 0x7f68a810f430 2024-07-26T11:25:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.749+0000 7f68a6000640 1 -- 172.21.15.134:0/3927752010 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f689c007cf0 con 0x7f68a810f430 2024-07-26T11:25:05.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.749+0000 7f68a6000640 1 -- 172.21.15.134:0/3927752010 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c01ade0 con 0x7f68a810f430 2024-07-26T11:25:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.749+0000 7f68afe00640 1 -- 172.21.15.134:0/3927752010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68a807d000 con 0x7f68a810f430 2024-07-26T11:25:05.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.749+0000 7f68afe00640 1 -- 172.21.15.134:0/3927752010 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68a807d500 con 0x7f68a810f430 2024-07-26T11:25:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.757+0000 7f68b6de7740 1 Processor -- start 2024-07-26T11:25:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.757+0000 7f68b6de7740 1 -- start start 2024-07-26T11:25:05.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.758+0000 7f68b6de7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4969a61b0 0x55f496d49a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.758+0000 7f68b6de7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f49697d2f0 0x55f496d49fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.758+0000 7f68b6de7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4969854d0 0x55f496d4a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.758+0000 7f68b6de7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f496cbe270 con 0x55f49697d2f0 2024-07-26T11:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.758+0000 7f68b6de7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f496cad770 con 0x55f4969854d0 2024-07-26T11:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.758+0000 7f68b6de7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f496cb17b0 con 0x55f4969a61b0 2024-07-26T11:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.758+0000 7f68ade00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4969a61b0 0x55f496d49a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.758+0000 7f68ade00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4969a61b0 0x55f496d49a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:46352/0 (socket says 172.21.15.134:46352) 2024-07-26T11:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.758+0000 7f68ade00640 1 -- 172.21.15.134:0/2791101370 learned_addr learned my addr 172.21.15.134:0/2791101370 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.759+0000 7f68ade00640 1 -- 172.21.15.134:0/2791101370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4969854d0 msgr2=0x55f496d4a4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.759+0000 7f68ade00640 1 --2- 172.21.15.134:0/2791101370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4969854d0 0x55f496d4a4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.759+0000 7f68ade00640 1 -- 172.21.15.134:0/2791101370 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f49697d2f0 msgr2=0x55f496d49fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.759+0000 7f68ade00640 1 --2- 172.21.15.134:0/2791101370 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f49697d2f0 0x55f496d49fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.759+0000 7f68ade00640 1 -- 172.21.15.134:0/2791101370 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f496a11790 con 0x55f4969a61b0 2024-07-26T11:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.759+0000 7f68ade00640 1 --2- 172.21.15.134:0/2791101370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4969a61b0 0x55f496d49a60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f68a000a460 tx=0x7f68a0009650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9ed87feab444d27 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.760+0000 7f6896600640 1 -- 172.21.15.134:0/2791101370 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0013790 con 0x55f4969a61b0 2024-07-26T11:25:05.763 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.760+0000 7f6896600640 1 -- 172.21.15.134:0/2791101370 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68a0019070 con 0x55f4969a61b0 2024-07-26T11:25:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.760+0000 7f68a6000640 1 -- 172.21.15.134:0/3927752010 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f689c007840 con 0x7f68a810f430 2024-07-26T11:25:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.761+0000 7f6896600640 1 -- 172.21.15.134:0/2791101370 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0013790 con 0x55f4969a61b0 2024-07-26T11:25:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.761+0000 7f68b6de7740 1 -- 172.21.15.134:0/2791101370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4969a61b0 msgr2=0x55f496d49a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.761+0000 7f68b6de7740 1 --2- 172.21.15.134:0/2791101370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4969a61b0 0x55f496d49a60 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f68a000a460 tx=0x7f68a0009650 comp rx=0 tx=0).stop 2024-07-26T11:25:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.762+0000 7f68b6de7740 1 -- 172.21.15.134:0/2791101370 shutdown_connections 2024-07-26T11:25:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.762+0000 7f68b6de7740 1 --2- 172.21.15.134:0/2791101370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4969854d0 0x55f496d4a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.762+0000 7f68b6de7740 1 --2- 172.21.15.134:0/2791101370 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f49697d2f0 0x55f496d49fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.762+0000 7f68b6de7740 1 --2- 172.21.15.134:0/2791101370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4969a61b0 0x55f496d49a60 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.762+0000 7f68b6de7740 1 -- 172.21.15.134:0/2791101370 >> 172.21.15.134:0/2791101370 conn(0x55f49699a300 msgr2=0x55f496d47950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:05.765 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.762+0000 7f68b6de7740 1 -- 172.21.15.134:0/2791101370 shutdown_connections 2024-07-26T11:25:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.762+0000 7f68b6de7740 1 -- 172.21.15.134:0/2791101370 wait complete. 2024-07-26T11:25:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.764+0000 7f68b6de7740 1 -- start start 2024-07-26T11:25:05.766 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.764+0000 7f68b6de7740 1 Processor -- start 2024-07-26T11:25:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.765+0000 7f68a6000640 1 --2- 172.21.15.134:0/3927752010 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f68900b3de0 0x7f68900b62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.765+0000 7f68b6de7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f49697d2f0 0x55f496d5e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.765+0000 7f68b6de7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4969854d0 0x55f496d5eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.765+0000 7f68b6de7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969a61b0 0x55f496d63ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.765+0000 7f68b6de7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f496be7ab0 con 0x55f4969a61b0 2024-07-26T11:25:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.765+0000 7f68b6de7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f496c4ad50 con 0x55f49697d2f0 2024-07-26T11:25:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.766+0000 7f68b6de7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f496d58230 con 0x55f4969854d0 2024-07-26T11:25:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.766+0000 7f68b6de7740 10 client.0 _ll_register_callbacks cb 0x55f4969bda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:25:05.768 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.766+0000 7f68aca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4969854d0 0x55f496d5eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.766+0000 7f68ad400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f49697d2f0 0x55f496d5e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.766+0000 7f68aca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4969854d0 0x55f496d5eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:46368/0 (socket says 172.21.15.134:46368) 2024-07-26T11:25:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.766+0000 7f68aca00640 1 -- 172.21.15.134:0/1912516261 learned_addr learned my addr 172.21.15.134:0/1912516261 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.767+0000 7f68a6000640 1 -- 172.21.15.134:0/3927752010 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f689c0e7a10 con 0x7f68a810f430 2024-07-26T11:25:05.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.767+0000 7f68aca00640 1 -- 172.21.15.134:0/1912516261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f49697d2f0 msgr2=0x55f496d5e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:05.770 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.767+0000 7f68ade00640 1 --2- 172.21.15.134:0/3927752010 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f68900b3de0 0x7f68900b62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:05.770 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.768+0000 7f68aca00640 1 --2- 172.21.15.134:0/1912516261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f49697d2f0 0x55f496d5e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.770 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.768+0000 7f68aca00640 1 -- 172.21.15.134:0/1912516261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969a61b0 msgr2=0x55f496d63ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:05.770 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.768+0000 7f68ade00640 1 --2- 172.21.15.134:0/3927752010 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f68900b3de0 0x7f68900b62f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f68a0014830 tx=0x7f68a0014930 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:05.770 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.769+0000 7f68aca00640 1 --2- 172.21.15.134:0/1912516261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969a61b0 0x55f496d63ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.771 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.769+0000 7f68aca00640 1 -- 172.21.15.134:0/1912516261 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f496a8a620 con 0x55f4969854d0 2024-07-26T11:25:05.771 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.769+0000 7f68ad400640 1 --2- 172.21.15.134:0/1912516261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f49697d2f0 0x55f496d5e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:25:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.773+0000 7f688aa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:25:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.773+0000 7f68aca00640 1 --2- 172.21.15.134:0/1912516261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4969854d0 0x55f496d5eab0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f689c0e4d80 tx=0x7f689c01a6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f0498c19182c29 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.773+0000 7f688aa00640 1 -- 172.21.15.134:0/1912516261 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f689c007b30 con 0x55f4969854d0 2024-07-26T11:25:05.775 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.773+0000 7f68b6de7740 1 -- 172.21.15.134:0/1912516261 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68a0019070 con 0x55f4969854d0 2024-07-26T11:25:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.774+0000 7f688aa00640 1 -- 172.21.15.134:0/1912516261 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f689c0aa900 con 0x55f4969854d0 2024-07-26T11:25:05.776 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.774+0000 7f688aa00640 1 -- 172.21.15.134:0/1912516261 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f689c007b30 con 0x55f4969854d0 2024-07-26T11:25:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.777+0000 7f68b6de7740 10 client.26761 fetch_fsmap learned FSMap version 32 2024-07-26T11:25:05.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.777+0000 7f68b6de7740 1 -- 172.21.15.134:0/1912516261 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f496d6cac0 con 0x55f4969854d0 2024-07-26T11:25:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.778+0000 7f688aa00640 1 -- 172.21.15.134:0/1912516261 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68a0019070 con 0x55f4969854d0 2024-07-26T11:25:05.780 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.778+0000 7f688aa00640 1 -- 172.21.15.134:0/1912516261 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f689c0b5be0 con 0x55f4969854d0 2024-07-26T11:25:05.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.779+0000 7f68b6de7740 10 client.26761 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:25:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.779+0000 7f68b6de7740 10 client.26761 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:25:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.780+0000 7f68b6de7740 20 client.26761 populate_metadata read hostname 'smithi134' 2024-07-26T11:25:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.780+0000 7f68b6de7740 1 --2- 172.21.15.134:0/1912516261 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f496964d80 0x55f496d6ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.780+0000 7f68b6de7740 4 client.26761 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:25:05.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.780+0000 7f68b6de7740 1 -- 172.21.15.134:0/1912516261 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f496cc01c0 con 0x55f496964d80 2024-07-26T11:25:05.785 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.783+0000 7f68ad400640 1 --2- 172.21.15.134:0/1912516261 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f496964d80 0x55f496d6ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.784+0000 7f68ad400640 1 --2- 172.21.15.134:0/1912516261 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f496964d80 0x55f496d6ff40 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8a2a7f85afc2609 server_cookie=d15d2e8f33c60994 in_seq=0 out_seq=0 2024-07-26T11:25:05.786 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.784+0000 7f688aa00640 10 client.26761 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.786+0000 7f688aa00640 1 -- 172.21.15.134:0/1912516261 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7f6898004220 con 0x55f496964d80 2024-07-26T11:25:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.786+0000 7f688aa00640 10 client.26761 handle_command_reply: tid=0 2024-07-26T11:25:05.788 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.786+0000 7f68b6de7740 1 client.26761 shutdown 2024-07-26T11:25:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.789+0000 7f68b6de7740 1 -- 172.21.15.134:0/1912516261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4969854d0 msgr2=0x55f496d5eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.789+0000 7f68b6de7740 1 --2- 172.21.15.134:0/1912516261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4969854d0 0x55f496d5eab0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f689c0e4d80 tx=0x7f689c01a6e0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.789+0000 7f68b6de7740 1 -- 172.21.15.134:0/1912516261 shutdown_connections 2024-07-26T11:25:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.789+0000 7f68b6de7740 1 --2- 172.21.15.134:0/1912516261 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f496964d80 0x55f496d6ff40 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.789+0000 7f68b6de7740 1 --2- 172.21.15.134:0/1912516261 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f4969854d0 0x55f496d5eab0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.791 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.789+0000 7f68b6de7740 1 --2- 172.21.15.134:0/1912516261 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969a61b0 0x55f496d63ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.789+0000 7f68b6de7740 1 --2- 172.21.15.134:0/1912516261 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f49697d2f0 0x55f496d5e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.789+0000 7f68b6de7740 1 -- 172.21.15.134:0/1912516261 >> 172.21.15.134:0/1912516261 conn(0x55f49699a300 msgr2=0x55f496d4cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.789+0000 7f688aa00640 0 client.26761 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.789+0000 7f68b6de7740 1 -- 172.21.15.134:0/1912516261 shutdown_connections 2024-07-26T11:25:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.790+0000 7f68b6de7740 1 -- 172.21.15.134:0/1912516261 wait complete. 2024-07-26T11:25:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.790+0000 7f68b6de7740 20 client.26761 trim_cache size 0 max 16384 2024-07-26T11:25:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.830+0000 7f68b6de7740 1 Processor -- start 2024-07-26T11:25:05.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.830+0000 7f68b6de7740 1 -- start start 2024-07-26T11:25:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.831+0000 7f68b6de7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f49697d2f0 0x55f496d57560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.831+0000 7f68b6de7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4969854d0 0x55f496d4cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.831+0000 7f68b6de7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969a61b0 0x55f496d4d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.831+0000 7f68b6de7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f68a81c8180 con 0x55f4969a61b0 2024-07-26T11:25:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.831+0000 7f68b6de7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f68a81c9cc0 con 0x55f4969854d0 2024-07-26T11:25:05.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.831+0000 7f68b6de7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f68a81c8850 con 0x55f49697d2f0 2024-07-26T11:25:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.831+0000 7f68ad400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f49697d2f0 0x55f496d57560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.831+0000 7f68ade00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969a61b0 0x55f496d4d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.831+0000 7f68ade00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969a61b0 0x55f496d4d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:56734/0 (socket says 172.21.15.134:56734) 2024-07-26T11:25:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.831+0000 7f68ad400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f49697d2f0 0x55f496d57560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:46370/0 (socket says 172.21.15.134:46370) 2024-07-26T11:25:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.831+0000 7f68ade00640 1 -- 172.21.15.134:0/659079928 learned_addr learned my addr 172.21.15.134:0/659079928 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:05.834 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.831+0000 7f68aca00640 1 --2- 172.21.15.134:0/659079928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4969854d0 0x55f496d4cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.832+0000 7f68ade00640 1 -- 172.21.15.134:0/659079928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f49697d2f0 msgr2=0x55f496d57560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.832+0000 7f68ade00640 1 --2- 172.21.15.134:0/659079928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f49697d2f0 0x55f496d57560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.832+0000 7f68ade00640 1 -- 172.21.15.134:0/659079928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4969854d0 msgr2=0x55f496d4cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.832+0000 7f68ade00640 1 --2- 172.21.15.134:0/659079928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4969854d0 0x55f496d4cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.832+0000 7f68ade00640 1 -- 172.21.15.134:0/659079928 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f496d5e040 con 0x55f4969a61b0 2024-07-26T11:25:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.833+0000 7f68ade00640 1 --2- 172.21.15.134:0/659079928 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969a61b0 0x55f496d4d050 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f68a0018000 tx=0x7f68a0009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab627d2ef0becd3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:05.835 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.833+0000 7f6895c00640 1 -- 172.21.15.134:0/659079928 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0013c90 con 0x55f4969a61b0 2024-07-26T11:25:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.833+0000 7f6895c00640 1 -- 172.21.15.134:0/659079928 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68a0019040 con 0x55f4969a61b0 2024-07-26T11:25:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.833+0000 7f6895c00640 1 -- 172.21.15.134:0/659079928 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a0029410 con 0x55f4969a61b0 2024-07-26T11:25:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.835+0000 7f68b6de7740 1 -- 172.21.15.134:0/659079928 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969a61b0 msgr2=0x55f496d4d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:05.836 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.835+0000 7f68b6de7740 1 --2- 172.21.15.134:0/659079928 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969a61b0 0x55f496d4d050 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f68a0018000 tx=0x7f68a0009650 comp rx=0 tx=0).stop 2024-07-26T11:25:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.836+0000 7f68b6de7740 1 -- 172.21.15.134:0/659079928 shutdown_connections 2024-07-26T11:25:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.836+0000 7f68b6de7740 1 --2- 172.21.15.134:0/659079928 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969a61b0 0x55f496d4d050 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.836+0000 7f68b6de7740 1 --2- 172.21.15.134:0/659079928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4969854d0 0x55f496d4cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.836+0000 7f68b6de7740 1 --2- 172.21.15.134:0/659079928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f49697d2f0 0x55f496d57560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.836+0000 7f68b6de7740 1 -- 172.21.15.134:0/659079928 >> 172.21.15.134:0/659079928 conn(0x55f49699a300 msgr2=0x55f496d5cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:05.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.836+0000 7f68b6de7740 1 -- 172.21.15.134:0/659079928 shutdown_connections 2024-07-26T11:25:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.836+0000 7f68b6de7740 1 -- 172.21.15.134:0/659079928 wait complete. 2024-07-26T11:25:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.837+0000 7f68b6de7740 1 -- start start 2024-07-26T11:25:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.838+0000 7f68b6de7740 1 Processor -- start 2024-07-26T11:25:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.838+0000 7f68b6de7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f49697d2f0 0x55f496d6a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.838+0000 7f68b6de7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969854d0 0x55f496d6a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.838+0000 7f68b6de7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4969a61b0 0x55f496d75400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.838+0000 7f68b6de7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f68a81c95f0 con 0x55f4969854d0 2024-07-26T11:25:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.839+0000 7f68b6de7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f68a81c8f20 con 0x55f4969a61b0 2024-07-26T11:25:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.839+0000 7f68b6de7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f68a81c6120 con 0x55f49697d2f0 2024-07-26T11:25:05.841 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.839+0000 7f68b6de7740 10 client.0 _ll_register_callbacks cb 0x7f689c01a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:25:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.839+0000 7f68ade00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4969a61b0 0x55f496d75400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.839+0000 7f68ade00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4969a61b0 0x55f496d75400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:55038/0 (socket says 172.21.15.134:55038) 2024-07-26T11:25:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.840+0000 7f68ad400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f49697d2f0 0x55f496d6a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.840+0000 7f68ad400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f49697d2f0 0x55f496d6a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:46378/0 (socket says 172.21.15.134:46378) 2024-07-26T11:25:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.840+0000 7f68ade00640 1 -- 172.21.15.134:0/3897870444 learned_addr learned my addr 172.21.15.134:0/3897870444 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.840+0000 7f68aca00640 1 --2- 172.21.15.134:0/3897870444 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969854d0 0x55f496d6a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.840+0000 7f68ad400640 1 -- 172.21.15.134:0/3897870444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4969a61b0 msgr2=0x55f496d75400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:05.845 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.841+0000 7f68ad400640 1 --2- 172.21.15.134:0/3897870444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4969a61b0 0x55f496d75400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.841+0000 7f68ad400640 1 -- 172.21.15.134:0/3897870444 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969854d0 msgr2=0x55f496d6a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.841+0000 7f68ad400640 1 --2- 172.21.15.134:0/3897870444 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969854d0 0x55f496d6a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.841+0000 7f68ad400640 1 -- 172.21.15.134:0/3897870444 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f496d75b90 con 0x55f49697d2f0 2024-07-26T11:25:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.842+0000 7f68ad400640 1 --2- 172.21.15.134:0/3897870444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f49697d2f0 0x55f496d6a3d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f68980303f0 tx=0x7f68980044e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcafc5037ca392d7 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.842+0000 7f688aa00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:25:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.842+0000 7f688aa00640 1 -- 172.21.15.134:0/3897870444 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6898037040 con 0x55f49697d2f0 2024-07-26T11:25:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.842+0000 7f688aa00640 1 -- 172.21.15.134:0/3897870444 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9511+0+0 (secure 0 0 0) 0x7f6898030710 con 0x55f49697d2f0 2024-07-26T11:25:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.843+0000 7f68ade00640 1 --2- 172.21.15.134:0/3897870444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4969a61b0 0x55f496d75400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:25:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.843+0000 7f68b6de7740 1 -- 172.21.15.134:0/3897870444 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f68a0019040 con 0x55f49697d2f0 2024-07-26T11:25:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.843+0000 7f688aa00640 1 -- 172.21.15.134:0/3897870444 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f6898005040 con 0x55f49697d2f0 2024-07-26T11:25:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.843+0000 7f68b6de7740 10 client.26767 fetch_fsmap learned FSMap version 32 2024-07-26T11:25:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.843+0000 7f68b6de7740 1 -- 172.21.15.134:0/3897870444 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f496d69c00 con 0x55f49697d2f0 2024-07-26T11:25:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.843+0000 7f688aa00640 1 -- 172.21.15.134:0/3897870444 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68a0019040 con 0x55f49697d2f0 2024-07-26T11:25:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.844+0000 7f68aca00640 1 --2- 172.21.15.134:0/3897870444 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969854d0 0x55f496d6a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:25:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.845+0000 7f688aa00640 1 -- 172.21.15.134:0/3897870444 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f689804c070 con 0x55f49697d2f0 2024-07-26T11:25:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.846+0000 7f68b6de7740 10 client.26767 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:25:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.846+0000 7f68b6de7740 10 client.26767 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:25:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.846+0000 7f68b6de7740 20 client.26767 populate_metadata read hostname 'smithi134' 2024-07-26T11:25:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.846+0000 7f68b6de7740 1 --2- 172.21.15.134:0/3897870444 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f49693a1c0 0x55f496ddaa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.846+0000 7f68b6de7740 4 client.26767 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:25:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.846+0000 7f68b6de7740 1 -- 172.21.15.134:0/3897870444 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f496cbbc60 con 0x55f49693a1c0 2024-07-26T11:25:05.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.846+0000 7f68aca00640 1 --2- 172.21.15.134:0/3897870444 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f49693a1c0 0x55f496ddaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:05.850 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.848+0000 7f688aa00640 10 client.26767 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:05.850 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:05.848+0000 7f68aca00640 1 --2- 172.21.15.134:0/3897870444 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f49693a1c0 0x55f496ddaa30 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca82e0d302e45635 server_cookie=3d463c53272cecca in_seq=0 out_seq=0 2024-07-26T11:25:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:05 smithi193 ceph-mon[37473]: pgmap v757: 129 pgs: 129 active+clean; 61 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 44 KiB/s rd, 266 MiB/s wr, 1.71k op/s 2024-07-26T11:25:06.169 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.166+0000 7f688aa00640 1 -- 172.21.15.134:0/3897870444 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f496cbbc60 con 0x55f49693a1c0 2024-07-26T11:25:06.169 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.166+0000 7f688aa00640 10 client.26767 handle_command_reply: tid=0 2024-07-26T11:25:06.169 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.166+0000 7f68b6de7740 1 client.26767 shutdown 2024-07-26T11:25:06.169 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.167+0000 7f68b6de7740 1 -- 172.21.15.134:0/3897870444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f49697d2f0 msgr2=0x55f496d6a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:06.170 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.167+0000 7f68b6de7740 1 --2- 172.21.15.134:0/3897870444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f49697d2f0 0x55f496d6a3d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f68980303f0 tx=0x7f68980044e0 comp rx=0 tx=0).stop 2024-07-26T11:25:06.170 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.168+0000 7f68b6de7740 1 -- 172.21.15.134:0/3897870444 shutdown_connections 2024-07-26T11:25:06.170 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.168+0000 7f68b6de7740 1 --2- 172.21.15.134:0/3897870444 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55f49693a1c0 0x55f496ddaa30 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.168+0000 7f688aa00640 0 client.26767 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.168+0000 7f68b6de7740 1 --2- 172.21.15.134:0/3897870444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f4969a61b0 0x55f496d75400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.168+0000 7f68b6de7740 1 --2- 172.21.15.134:0/3897870444 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f4969854d0 0x55f496d6a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.168+0000 7f68b6de7740 1 --2- 172.21.15.134:0/3897870444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f49697d2f0 0x55f496d6a3d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:06.171 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.168+0000 7f68b6de7740 1 -- 172.21.15.134:0/3897870444 >> 172.21.15.134:0/3897870444 conn(0x55f49699a300 msgr2=0x55f496d5cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.169+0000 7f68b6de7740 1 -- 172.21.15.134:0/3897870444 shutdown_connections 2024-07-26T11:25:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.169+0000 7f68b6de7740 1 -- 172.21.15.134:0/3897870444 wait complete. 2024-07-26T11:25:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.169+0000 7f68b6de7740 20 client.26767 trim_cache size 0 max 16384 2024-07-26T11:25:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:25:06.172 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:25:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:25:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:25:06.173 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.171+0000 7f68afe00640 1 -- 172.21.15.134:0/3927752010 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f68900b3de0 msgr2=0x7f68900b62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.171+0000 7f68afe00640 1 --2- 172.21.15.134:0/3927752010 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f68900b3de0 0x7f68900b62f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f68a0014830 tx=0x7f68a0014930 comp rx=0 tx=0).stop 2024-07-26T11:25:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.171+0000 7f68afe00640 1 -- 172.21.15.134:0/3927752010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a810f430 msgr2=0x7f68a8082a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:06.174 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.171+0000 7f68afe00640 1 --2- 172.21.15.134:0/3927752010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a810f430 0x7f68a8082a30 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f689c009990 tx=0x7f689c009e50 comp rx=0 tx=0).stop 2024-07-26T11:25:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.172+0000 7f68afe00640 1 -- 172.21.15.134:0/3927752010 shutdown_connections 2024-07-26T11:25:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.172+0000 7f68afe00640 1 --2- 172.21.15.134:0/3927752010 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f68900b3de0 0x7f68900b62f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.172+0000 7f68afe00640 1 --2- 172.21.15.134:0/3927752010 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f68a810f880 0x7f68a81bf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.172+0000 7f68afe00640 1 --2- 172.21.15.134:0/3927752010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68a810f430 0x7f68a8082a30 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.172+0000 7f68afe00640 1 --2- 172.21.15.134:0/3927752010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68a810b460 0x7f68a80824f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.172+0000 7f68afe00640 1 -- 172.21.15.134:0/3927752010 >> 172.21.15.134:0/3927752010 conn(0x7f68a8070800 msgr2=0x7f68a810d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.172+0000 7f68afe00640 1 -- 172.21.15.134:0/3927752010 shutdown_connections 2024-07-26T11:25:06.175 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:06.172+0000 7f68afe00640 1 -- 172.21.15.134:0/3927752010 wait complete. 2024-07-26T11:25:06.177 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:05 smithi134 ceph-mon[31966]: pgmap v757: 129 pgs: 129 active+clean; 61 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 44 KiB/s rd, 266 MiB/s wr, 1.71k op/s 2024-07-26T11:25:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:05 smithi189 ceph-mon[36790]: pgmap v757: 129 pgs: 129 active+clean; 61 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 44 KiB/s rd, 266 MiB/s wr, 1.71k op/s 2024-07-26T11:25:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:07 smithi193 ceph-mon[37473]: pgmap v758: 129 pgs: 129 active+clean; 61 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 43 KiB/s rd, 250 MiB/s wr, 1.58k op/s 2024-07-26T11:25:08.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:07 smithi134 ceph-mon[31966]: pgmap v758: 129 pgs: 129 active+clean; 61 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 43 KiB/s rd, 250 MiB/s wr, 1.58k op/s 2024-07-26T11:25:08.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:07 smithi189 ceph-mon[36790]: pgmap v758: 129 pgs: 129 active+clean; 61 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 43 KiB/s rd, 250 MiB/s wr, 1.58k op/s 2024-07-26T11:25:09.091 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:08 smithi134 ceph-mon[31966]: pgmap v759: 129 pgs: 129 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 61 KiB/s rd, 241 MiB/s wr, 1.57k op/s 2024-07-26T11:25:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:08 smithi189 ceph-mon[36790]: pgmap v759: 129 pgs: 129 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 61 KiB/s rd, 241 MiB/s wr, 1.57k op/s 2024-07-26T11:25:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:08 smithi193 ceph-mon[37473]: pgmap v759: 129 pgs: 129 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 61 KiB/s rd, 241 MiB/s wr, 1.57k op/s 2024-07-26T11:25:11.585 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:11 smithi134 ceph-mon[31966]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T11:25:11.585 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:11 smithi134 ceph-mon[31966]: pgmap v761: 129 pgs: 129 active+clean; 62 GiB data, 131 GiB used, 942 GiB / 1.0 TiB avail; 59 KiB/s rd, 254 MiB/s wr, 1.80k op/s 2024-07-26T11:25:11.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:11 smithi189 ceph-mon[36790]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T11:25:11.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:11 smithi189 ceph-mon[36790]: pgmap v761: 129 pgs: 129 active+clean; 62 GiB data, 131 GiB used, 942 GiB / 1.0 TiB avail; 59 KiB/s rd, 254 MiB/s wr, 1.80k op/s 2024-07-26T11:25:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:11 smithi193 ceph-mon[37473]: osdmap e95: 12 total, 12 up, 12 in 2024-07-26T11:25:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:11 smithi193 ceph-mon[37473]: pgmap v761: 129 pgs: 129 active+clean; 62 GiB data, 131 GiB used, 942 GiB / 1.0 TiB avail; 59 KiB/s rd, 254 MiB/s wr, 1.80k op/s 2024-07-26T11:25:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:13 smithi193 ceph-mon[37473]: pgmap v762: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 62 GiB data, 130 GiB used, 942 GiB / 1.0 TiB avail; 49 KiB/s rd, 253 MiB/s wr, 1.78k op/s 2024-07-26T11:25:14.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:13 smithi134 ceph-mon[31966]: pgmap v762: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 62 GiB data, 130 GiB used, 942 GiB / 1.0 TiB avail; 49 KiB/s rd, 253 MiB/s wr, 1.78k op/s 2024-07-26T11:25:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:13 smithi189 ceph-mon[36790]: pgmap v762: 129 pgs: 3 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 118 active+clean; 62 GiB data, 130 GiB used, 942 GiB / 1.0 TiB avail; 49 KiB/s rd, 253 MiB/s wr, 1.78k op/s 2024-07-26T11:25:15.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.937+0000 7f0124200640 1 -- 172.21.15.134:0/4152512821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f011c10b970 msgr2=0x7f011c10f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:15.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.937+0000 7f011be00640 1 -- 172.21.15.134:0/4152512821 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0110037070 con 0x7f011c10b970 2024-07-26T11:25:15.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.937+0000 7f0124200640 1 --2- 172.21.15.134:0/4152512821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f011c10b970 0x7f011c10f570 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0110009860 tx=0x7f011002a360 comp rx=0 tx=0).stop 2024-07-26T11:25:15.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.941+0000 7f0124200640 1 -- 172.21.15.134:0/4152512821 shutdown_connections 2024-07-26T11:25:15.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.941+0000 7f0124200640 1 --2- 172.21.15.134:0/4152512821 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f011c0790b0 0x7f011c07b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:15.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.941+0000 7f0124200640 1 --2- 172.21.15.134:0/4152512821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f011c10be60 0x7f011c10fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:15.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.941+0000 7f0124200640 1 --2- 172.21.15.134:0/4152512821 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f011c10b970 0x7f011c10f570 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:15.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.941+0000 7f0124200640 1 -- 172.21.15.134:0/4152512821 >> 172.21.15.134:0/4152512821 conn(0x7f011c0708f0 msgr2=0x7f011c070d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:15.944 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.941+0000 7f0124200640 1 -- 172.21.15.134:0/4152512821 shutdown_connections 2024-07-26T11:25:15.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.941+0000 7f0124200640 1 -- 172.21.15.134:0/4152512821 wait complete. 2024-07-26T11:25:15.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.943+0000 7f0124200640 1 Processor -- start 2024-07-26T11:25:15.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.944+0000 7f0124200640 1 -- start start 2024-07-26T11:25:15.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.945+0000 7f0124200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f011c0790b0 0x7f011c07fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:15.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.945+0000 7f0124200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f011c10b970 0x7f011c080390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:15.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.946+0000 7f0121800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f011c0790b0 0x7f011c07fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:15.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.946+0000 7f0121800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f011c0790b0 0x7f011c07fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38078/0 (socket says 172.21.15.134:38078) 2024-07-26T11:25:15.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.947+0000 7f0120e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f011c10b970 0x7f011c080390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:15.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.947+0000 7f0120e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f011c10b970 0x7f011c080390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:54360/0 (socket says 172.21.15.134:54360) 2024-07-26T11:25:15.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.947+0000 7f0121800640 1 -- 172.21.15.134:0/2165286408 learned_addr learned my addr 172.21.15.134:0/2165286408 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:15.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.947+0000 7f0124200640 1 --2- 172.21.15.134:0/2165286408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f011c10be60 0x7f011c0808d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:15.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.948+0000 7f0124200640 1 -- 172.21.15.134:0/2165286408 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f011c07e690 con 0x7f011c0790b0 2024-07-26T11:25:15.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.948+0000 7f0122200640 1 --2- 172.21.15.134:0/2165286408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f011c10be60 0x7f011c0808d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:15.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.948+0000 7f0124200640 1 -- 172.21.15.134:0/2165286408 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f011c07e510 con 0x7f011c10be60 2024-07-26T11:25:15.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.949+0000 7f0124200640 1 -- 172.21.15.134:0/2165286408 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f011c07e810 con 0x7f011c10b970 2024-07-26T11:25:15.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.950+0000 7f0120e00640 1 -- 172.21.15.134:0/2165286408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f011c10be60 msgr2=0x7f011c0808d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:15.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.950+0000 7f0120e00640 1 --2- 172.21.15.134:0/2165286408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f011c10be60 0x7f011c0808d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:15.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.950+0000 7f0120e00640 1 -- 172.21.15.134:0/2165286408 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f011c0790b0 msgr2=0x7f011c07fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:15.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.951+0000 7f0120e00640 1 --2- 172.21.15.134:0/2165286408 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f011c0790b0 0x7f011c07fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:15.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.951+0000 7f0120e00640 1 -- 172.21.15.134:0/2165286408 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f011c1bd440 con 0x7f011c10b970 2024-07-26T11:25:15.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.952+0000 7f0120e00640 1 --2- 172.21.15.134:0/2165286408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f011c10b970 0x7f011c080390 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f011c06c3a0 tx=0x7f010800ddd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:15.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.953+0000 7f011a000640 1 -- 172.21.15.134:0/2165286408 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f010801a070 con 0x7f011c10b970 2024-07-26T11:25:15.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.953+0000 7f011a000640 1 -- 172.21.15.134:0/2165286408 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0108015410 con 0x7f011c10b970 2024-07-26T11:25:15.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.953+0000 7f0124200640 1 -- 172.21.15.134:0/2165286408 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f011c1bd730 con 0x7f011c10b970 2024-07-26T11:25:15.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.954+0000 7f011a000640 1 -- 172.21.15.134:0/2165286408 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0108014610 con 0x7f011c10b970 2024-07-26T11:25:15.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.954+0000 7f0124200640 1 -- 172.21.15.134:0/2165286408 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f011c1bdc70 con 0x7f011c10b970 2024-07-26T11:25:15.975 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:15 smithi134 ceph-mon[31966]: pgmap v763: 129 pgs: 10 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 93 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 37 KiB/s rd, 223 MiB/s wr, 1.50k op/s 2024-07-26T11:25:15.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.968+0000 7f011a000640 1 -- 172.21.15.134:0/2165286408 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f010801d070 con 0x7f011c10b970 2024-07-26T11:25:15.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.970+0000 7f011a000640 1 --2- 172.21.15.134:0/2165286408 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f01040b3ec0 0x7f01040b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:15.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.971+0000 7f011a000640 1 -- 172.21.15.134:0/2165286408 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9599+0+0 (secure 0 0 0) 0x7f01080e37c0 con 0x7f011c10b970 2024-07-26T11:25:15.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.973+0000 7f0121800640 1 --2- 172.21.15.134:0/2165286408 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f01040b3ec0 0x7f01040b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:15.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:15.980+0000 7f0121800640 1 --2- 172.21.15.134:0/2165286408 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f01040b3ec0 0x7f01040b63d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0110030460 tx=0x7f011000b040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:16.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.016+0000 7f012ae73740 1 Processor -- start 2024-07-26T11:25:16.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.017+0000 7f012ae73740 1 -- start start 2024-07-26T11:25:16.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.017+0000 7f012ae73740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074e0b070 0x5590751afe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:16.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.017+0000 7f012ae73740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751b0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:16.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.017+0000 7f012ae73740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074dea390 0x5590751b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:16.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.017+0000 7f012ae73740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559075126bf0 con 0x559074e0b070 2024-07-26T11:25:16.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.017+0000 7f012ae73740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5590751160f0 con 0x559074de21b0 2024-07-26T11:25:16.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.017+0000 7f012ae73740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55907511a130 con 0x559074dea390 2024-07-26T11:25:16.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.020+0000 7f0120e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751b0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:16.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.020+0000 7f0120e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751b0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35546/0 (socket says 172.21.15.134:35546) 2024-07-26T11:25:16.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.020+0000 7f0120e00640 1 -- 172.21.15.134:0/912559899 learned_addr learned my addr 172.21.15.134:0/912559899 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:16.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.020+0000 7f0122200640 1 --2- 172.21.15.134:0/912559899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074dea390 0x5590751b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:16.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.021+0000 7f0122200640 1 -- 172.21.15.134:0/912559899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 msgr2=0x5590751b0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:16.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.021+0000 7f0122200640 1 --2- 172.21.15.134:0/912559899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751b0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.021+0000 7f0122200640 1 -- 172.21.15.134:0/912559899 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074e0b070 msgr2=0x5590751afe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:16.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.021+0000 7f0122200640 1 --2- 172.21.15.134:0/912559899 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074e0b070 0x5590751afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.021+0000 7f0122200640 1 -- 172.21.15.134:0/912559899 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559074e76650 con 0x559074dea390 2024-07-26T11:25:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:15 smithi193 ceph-mon[37473]: pgmap v763: 129 pgs: 10 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 93 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 37 KiB/s rd, 223 MiB/s wr, 1.50k op/s 2024-07-26T11:25:16.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.027+0000 7f0122200640 1 --2- 172.21.15.134:0/912559899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074dea390 0x5590751b08d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0114009f00 tx=0x7f0114011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=654cadc58517fbbe server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:16.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.028+0000 7f010ea00640 1 -- 172.21.15.134:0/912559899 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0114004d20 con 0x559074dea390 2024-07-26T11:25:16.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.028+0000 7f010ea00640 1 -- 172.21.15.134:0/912559899 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0114004030 con 0x559074dea390 2024-07-26T11:25:16.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.028+0000 7f010ea00640 1 -- 172.21.15.134:0/912559899 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0114005280 con 0x559074dea390 2024-07-26T11:25:16.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.036+0000 7f012ae73740 1 -- 172.21.15.134:0/912559899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074dea390 msgr2=0x5590751b08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:16.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.036+0000 7f012ae73740 1 --2- 172.21.15.134:0/912559899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074dea390 0x5590751b08d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0114009f00 tx=0x7f0114011810 comp rx=0 tx=0).stop 2024-07-26T11:25:16.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.036+0000 7f012ae73740 1 -- 172.21.15.134:0/912559899 shutdown_connections 2024-07-26T11:25:16.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.036+0000 7f012ae73740 1 --2- 172.21.15.134:0/912559899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074dea390 0x5590751b08d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.036+0000 7f012ae73740 1 --2- 172.21.15.134:0/912559899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751b0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.036+0000 7f012ae73740 1 --2- 172.21.15.134:0/912559899 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074e0b070 0x5590751afe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.036+0000 7f012ae73740 1 -- 172.21.15.134:0/912559899 >> 172.21.15.134:0/912559899 conn(0x559074dff460 msgr2=0x5590751add40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:16.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.041+0000 7f012ae73740 1 -- 172.21.15.134:0/912559899 shutdown_connections 2024-07-26T11:25:16.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.041+0000 7f012ae73740 1 -- 172.21.15.134:0/912559899 wait complete. 2024-07-26T11:25:16.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.042+0000 7f012ae73740 1 -- start start 2024-07-26T11:25:16.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.042+0000 7f012ae73740 1 Processor -- start 2024-07-26T11:25:16.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.044+0000 7f012ae73740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074de21b0 0x5590751bb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:16.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.044+0000 7f012ae73740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074dea390 0x5590751bb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:16.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.044+0000 7f012ae73740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074e0b070 0x5590751bbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:16.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.045+0000 7f0120e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074dea390 0x5590751bb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:16.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.045+0000 7f0122200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074e0b070 0x5590751bbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:16.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.045+0000 7f0122200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074e0b070 0x5590751bbf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35554/0 (socket says 172.21.15.134:35554) 2024-07-26T11:25:16.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.045+0000 7f0122200640 1 -- 172.21.15.134:0/4163793688 learned_addr learned my addr 172.21.15.134:0/4163793688 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:16.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.045+0000 7f012ae73740 1 -- 172.21.15.134:0/4163793688 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559074f79700 con 0x559074de21b0 2024-07-26T11:25:16.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.045+0000 7f012ae73740 1 -- 172.21.15.134:0/4163793688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5590750b26c0 con 0x559074e0b070 2024-07-26T11:25:16.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.045+0000 7f012ae73740 1 -- 172.21.15.134:0/4163793688 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5590751be630 con 0x559074dea390 2024-07-26T11:25:16.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.045+0000 7f012ae73740 10 client.0 _ll_register_callbacks cb 0x5590750c2170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:25:16.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.045+0000 7f0122200640 1 -- 172.21.15.134:0/4163793688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074dea390 msgr2=0x5590751bb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:16.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.046+0000 7f0122200640 1 --2- 172.21.15.134:0/4163793688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074dea390 0x5590751bb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.046+0000 7f0122200640 1 -- 172.21.15.134:0/4163793688 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074de21b0 msgr2=0x5590751bb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:16.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.046+0000 7f0122200640 1 --2- 172.21.15.134:0/4163793688 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074de21b0 0x5590751bb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.046+0000 7f0122200640 1 -- 172.21.15.134:0/4163793688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590750c9860 con 0x559074e0b070 2024-07-26T11:25:16.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.046+0000 7f012ae73740 1 -- 172.21.15.134:0/4163793688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0114004030 con 0x559074e0b070 2024-07-26T11:25:16.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.046+0000 7f0122200640 1 --2- 172.21.15.134:0/4163793688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074e0b070 0x5590751bbf40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0114009270 tx=0x7f0114003a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8c246fb527e01e8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:16.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.046+0000 7f0103400640 10 client.26750 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:25:16.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.047+0000 7f0103400640 1 -- 172.21.15.134:0/4163793688 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0114027020 con 0x559074e0b070 2024-07-26T11:25:16.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.047+0000 7f0103400640 1 -- 172.21.15.134:0/4163793688 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9599+0+0 (secure 0 0 0) 0x7f01140041e0 con 0x559074e0b070 2024-07-26T11:25:16.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.047+0000 7f0103400640 1 -- 172.21.15.134:0/4163793688 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0114004460 con 0x559074e0b070 2024-07-26T11:25:16.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.047+0000 7f012ae73740 10 client.26750 fetch_fsmap learned FSMap version 32 2024-07-26T11:25:16.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.047+0000 7f012ae73740 1 -- 172.21.15.134:0/4163793688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=32}) -- 0x5590751ca1f0 con 0x559074e0b070 2024-07-26T11:25:16.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.048+0000 7f0103400640 1 -- 172.21.15.134:0/4163793688 <== mon.2 v2:172.21.15.189:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f0114018070 con 0x559074e0b070 2024-07-26T11:25:16.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.048+0000 7f012ae73740 10 client.26750 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:25:16.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.048+0000 7f012ae73740 10 client.26750 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:25:16.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.048+0000 7f012ae73740 20 client.26750 populate_metadata read hostname 'smithi134' 2024-07-26T11:25:16.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.048+0000 7f012ae73740 1 --2- 172.21.15.134:0/4163793688 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559074dc99f0 0x5590751cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:16.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.048+0000 7f012ae73740 4 client.26750 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:25:16.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.048+0000 7f012ae73740 1 -- 172.21.15.134:0/4163793688 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559075128b40 con 0x559074dc99f0 2024-07-26T11:25:16.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.049+0000 7f0121800640 1 --2- 172.21.15.134:0/4163793688 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559074dc99f0 0x5590751cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:16.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.051+0000 7f0103400640 10 client.26750 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:16.055 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.051+0000 7f0121800640 1 --2- 172.21.15.134:0/4163793688 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559074dc99f0 0x5590751cda90 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3de4798dce40133 server_cookie=5fc60405834da493 in_seq=0 out_seq=0 2024-07-26T11:25:16.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.060+0000 7f0103400640 1 -- 172.21.15.134:0/4163793688 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559075128b40 con 0x559074dc99f0 2024-07-26T11:25:16.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.060+0000 7f0103400640 10 client.26750 handle_command_reply: tid=0 2024-07-26T11:25:16.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.061+0000 7f012ae73740 1 client.26750 shutdown 2024-07-26T11:25:16.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.061+0000 7f012ae73740 1 -- 172.21.15.134:0/4163793688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074e0b070 msgr2=0x5590751bbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:16.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.061+0000 7f012ae73740 1 --2- 172.21.15.134:0/4163793688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074e0b070 0x5590751bbf40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0114009270 tx=0x7f0114003a30 comp rx=0 tx=0).stop 2024-07-26T11:25:16.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.061+0000 7f012ae73740 1 -- 172.21.15.134:0/4163793688 shutdown_connections 2024-07-26T11:25:16.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.061+0000 7f012ae73740 1 --2- 172.21.15.134:0/4163793688 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559074dc99f0 0x5590751cda90 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.061+0000 7f012ae73740 1 --2- 172.21.15.134:0/4163793688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074dea390 0x5590751bb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.061+0000 7f012ae73740 1 --2- 172.21.15.134:0/4163793688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074e0b070 0x5590751bbf40 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.061+0000 7f012ae73740 1 --2- 172.21.15.134:0/4163793688 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074de21b0 0x5590751bb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.061+0000 7f012ae73740 1 -- 172.21.15.134:0/4163793688 >> 172.21.15.134:0/4163793688 conn(0x559074dff460 msgr2=0x5590751b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:16.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.061+0000 7f0103400640 0 client.26750 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:16.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.062+0000 7f012ae73740 1 -- 172.21.15.134:0/4163793688 shutdown_connections 2024-07-26T11:25:16.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.062+0000 7f012ae73740 1 -- 172.21.15.134:0/4163793688 wait complete. 2024-07-26T11:25:16.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.062+0000 7f012ae73740 20 client.26750 trim_cache size 0 max 16384 2024-07-26T11:25:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.102+0000 7f012ae73740 1 Processor -- start 2024-07-26T11:25:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.102+0000 7f012ae73740 1 -- start start 2024-07-26T11:25:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.103+0000 7f012ae73740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751b1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.103+0000 7f012ae73740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074dea390 0x5590751bd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.103+0000 7f0121800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.103+0000 7f0121800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751b1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35562/0 (socket says 172.21.15.134:35562) 2024-07-26T11:25:16.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.103+0000 7f0121800640 1 -- 172.21.15.134:0/231116036 learned_addr learned my addr 172.21.15.134:0/231116036 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:16.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.104+0000 7f012ae73740 1 --2- 172.21.15.134:0/231116036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074e0b070 0x5590751b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:16.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.104+0000 7f012ae73740 1 -- 172.21.15.134:0/231116036 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f011c1d91a0 con 0x559074dea390 2024-07-26T11:25:16.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.104+0000 7f012ae73740 1 -- 172.21.15.134:0/231116036 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f011c1dace0 con 0x559074de21b0 2024-07-26T11:25:16.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.104+0000 7f012ae73740 1 -- 172.21.15.134:0/231116036 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f011c1d9870 con 0x559074e0b070 2024-07-26T11:25:16.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.104+0000 7f0122200640 1 --2- 172.21.15.134:0/231116036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074e0b070 0x5590751b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:16.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.104+0000 7f0121800640 1 -- 172.21.15.134:0/231116036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074e0b070 msgr2=0x5590751b4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:16.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.104+0000 7f0121800640 1 --2- 172.21.15.134:0/231116036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074e0b070 0x5590751b4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.104+0000 7f0121800640 1 -- 172.21.15.134:0/231116036 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074dea390 msgr2=0x5590751bd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:16.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.104+0000 7f0121800640 1 --2- 172.21.15.134:0/231116036 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074dea390 0x5590751bd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.104+0000 7f0121800640 1 -- 172.21.15.134:0/231116036 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590751b9980 con 0x559074de21b0 2024-07-26T11:25:16.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.104+0000 7f0121800640 1 --2- 172.21.15.134:0/231116036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751b1840 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f011c0858e0 tx=0x7f011002a360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b8caee174284f98 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:16.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.105+0000 7f010e000640 1 -- 172.21.15.134:0/231116036 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011004f020 con 0x559074de21b0 2024-07-26T11:25:16.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.105+0000 7f010e000640 1 -- 172.21.15.134:0/231116036 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0110009430 con 0x559074de21b0 2024-07-26T11:25:16.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.105+0000 7f010e000640 1 -- 172.21.15.134:0/231116036 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0110037040 con 0x559074de21b0 2024-07-26T11:25:16.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.107+0000 7f012ae73740 1 -- 172.21.15.134:0/231116036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 msgr2=0x5590751b1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:16.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.107+0000 7f012ae73740 1 --2- 172.21.15.134:0/231116036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751b1840 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f011c0858e0 tx=0x7f011002a360 comp rx=0 tx=0).stop 2024-07-26T11:25:16.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.109+0000 7f012ae73740 1 -- 172.21.15.134:0/231116036 shutdown_connections 2024-07-26T11:25:16.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.109+0000 7f012ae73740 1 --2- 172.21.15.134:0/231116036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074e0b070 0x5590751b4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.109+0000 7f012ae73740 1 --2- 172.21.15.134:0/231116036 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074dea390 0x5590751bd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.109+0000 7f012ae73740 1 --2- 172.21.15.134:0/231116036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751b1840 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.109+0000 7f012ae73740 1 -- 172.21.15.134:0/231116036 >> 172.21.15.134:0/231116036 conn(0x559074dff460 msgr2=0x5590751bc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:16.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.110+0000 7f012ae73740 1 -- 172.21.15.134:0/231116036 shutdown_connections 2024-07-26T11:25:16.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.110+0000 7f012ae73740 1 -- 172.21.15.134:0/231116036 wait complete. 2024-07-26T11:25:16.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.111+0000 7f012ae73740 1 -- start start 2024-07-26T11:25:16.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.111+0000 7f012ae73740 1 Processor -- start 2024-07-26T11:25:16.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.113+0000 7f012ae73740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751bf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:16.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.113+0000 7f012ae73740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074dea390 0x5590751bf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:16.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.113+0000 7f0121800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751bf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:16.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.113+0000 7f0121800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751bf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35576/0 (socket says 172.21.15.134:35576) 2024-07-26T11:25:16.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.114+0000 7f0121800640 1 -- 172.21.15.134:0/2866908406 learned_addr learned my addr 172.21.15.134:0/2866908406 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:16.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.114+0000 7f012ae73740 1 --2- 172.21.15.134:0/2866908406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074e0b070 0x559075234250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:16.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.115+0000 7f012ae73740 1 -- 172.21.15.134:0/2866908406 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f011c1da610 con 0x559074e0b070 2024-07-26T11:25:16.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.115+0000 7f0120e00640 1 --2- 172.21.15.134:0/2866908406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074dea390 0x5590751bf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:16.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.115+0000 7f012ae73740 1 -- 172.21.15.134:0/2866908406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f011c1d9f40 con 0x559074de21b0 2024-07-26T11:25:16.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.115+0000 7f012ae73740 1 -- 172.21.15.134:0/2866908406 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f011c1d71a0 con 0x559074dea390 2024-07-26T11:25:16.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.115+0000 7f0122200640 1 --2- 172.21.15.134:0/2866908406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074e0b070 0x559075234250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:16.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.117+0000 7f0121800640 1 -- 172.21.15.134:0/2866908406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074dea390 msgr2=0x5590751bf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:16.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.117+0000 7f0121800640 1 --2- 172.21.15.134:0/2866908406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074dea390 0x5590751bf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.117+0000 7f0121800640 1 -- 172.21.15.134:0/2866908406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074e0b070 msgr2=0x559075234250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:16.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.117+0000 7f0121800640 1 --2- 172.21.15.134:0/2866908406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074e0b070 0x559075234250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.117+0000 7f0121800640 1 -- 172.21.15.134:0/2866908406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590752347e0 con 0x559074de21b0 2024-07-26T11:25:16.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.117+0000 7f012ae73740 10 client.0 _ll_register_callbacks cb 0x7f0114004e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:25:16.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.117+0000 7f012ae73740 1 -- 172.21.15.134:0/2866908406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5590751245e0 con 0x559074de21b0 2024-07-26T11:25:16.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.120+0000 7f0121800640 1 --2- 172.21.15.134:0/2866908406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751bf050 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f011002a950 tx=0x7f0110030ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa29547b56301849 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:16.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.120+0000 7f0103400640 10 client.26756 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:25:16.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.120+0000 7f0103400640 1 -- 172.21.15.134:0/2866908406 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011004f070 con 0x559074de21b0 2024-07-26T11:25:16.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.121+0000 7f0103400640 1 -- 172.21.15.134:0/2866908406 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9599+0+0 (secure 0 0 0) 0x7f0110005440 con 0x559074de21b0 2024-07-26T11:25:16.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.121+0000 7f0103400640 1 -- 172.21.15.134:0/2866908406 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f0110037040 con 0x559074de21b0 2024-07-26T11:25:16.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.121+0000 7f012ae73740 10 client.26756 fetch_fsmap learned FSMap version 32 2024-07-26T11:25:16.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.121+0000 7f012ae73740 1 -- 172.21.15.134:0/2866908406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=32}) -- 0x5590752358a0 con 0x559074de21b0 2024-07-26T11:25:16.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.121+0000 7f0103400640 1 -- 172.21.15.134:0/2866908406 <== mon.2 v2:172.21.15.189:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f01100046c0 con 0x559074de21b0 2024-07-26T11:25:16.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.122+0000 7f012ae73740 10 client.26756 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:25:16.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.122+0000 7f012ae73740 10 client.26756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:25:16.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.122+0000 7f012ae73740 20 client.26756 populate_metadata read hostname 'smithi134' 2024-07-26T11:25:16.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.122+0000 7f012ae73740 1 --2- 172.21.15.134:0/2866908406 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559074d9eda0 0x559075237dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:16.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.123+0000 7f012ae73740 4 client.26756 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:25:16.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.124+0000 7f012ae73740 1 -- 172.21.15.134:0/2866908406 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559075121d20 con 0x559074d9eda0 2024-07-26T11:25:16.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.124+0000 7f0120e00640 1 --2- 172.21.15.134:0/2866908406 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559074d9eda0 0x559075237dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:16.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.125+0000 7f0120e00640 1 --2- 172.21.15.134:0/2866908406 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559074d9eda0 0x559075237dd0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddcd043c06716158 server_cookie=e3a4bc3599f1839b in_seq=0 out_seq=0 2024-07-26T11:25:16.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.126+0000 7f0103400640 10 client.26756 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:16.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.127+0000 7f0103400640 1 -- 172.21.15.134:0/2866908406 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559075121d20 con 0x559074d9eda0 2024-07-26T11:25:16.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.127+0000 7f0103400640 10 client.26756 handle_command_reply: tid=0 2024-07-26T11:25:16.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.127+0000 7f012ae73740 1 client.26756 shutdown 2024-07-26T11:25:16.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.127+0000 7f012ae73740 1 -- 172.21.15.134:0/2866908406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 msgr2=0x5590751bf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:16.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.127+0000 7f012ae73740 1 --2- 172.21.15.134:0/2866908406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751bf050 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f011002a950 tx=0x7f0110030ae0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.127+0000 7f012ae73740 1 -- 172.21.15.134:0/2866908406 shutdown_connections 2024-07-26T11:25:16.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.127+0000 7f012ae73740 1 --2- 172.21.15.134:0/2866908406 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559074d9eda0 0x559075237dd0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.127+0000 7f012ae73740 1 --2- 172.21.15.134:0/2866908406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559074dea390 0x5590751bf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.127+0000 7f012ae73740 1 --2- 172.21.15.134:0/2866908406 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559074e0b070 0x559075234250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.127+0000 7f012ae73740 1 --2- 172.21.15.134:0/2866908406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559074de21b0 0x5590751bf050 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.130 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.127+0000 7f012ae73740 1 -- 172.21.15.134:0/2866908406 >> 172.21.15.134:0/2866908406 conn(0x559074dff460 msgr2=0x5590751b16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:16.131 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:25:16.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.128+0000 7f0103400640 0 client.26756 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:16.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.128+0000 7f012ae73740 1 -- 172.21.15.134:0/2866908406 shutdown_connections 2024-07-26T11:25:16.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.128+0000 7f012ae73740 1 -- 172.21.15.134:0/2866908406 wait complete. 2024-07-26T11:25:16.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.128+0000 7f012ae73740 20 client.26756 trim_cache size 0 max 16384 2024-07-26T11:25:16.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.129+0000 7f0124200640 1 -- 172.21.15.134:0/2165286408 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f01040b3ec0 msgr2=0x7f01040b63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:16.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.129+0000 7f0124200640 1 --2- 172.21.15.134:0/2165286408 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f01040b3ec0 0x7f01040b63d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0110030460 tx=0x7f011000b040 comp rx=0 tx=0).stop 2024-07-26T11:25:16.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.130+0000 7f0124200640 1 -- 172.21.15.134:0/2165286408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f011c10b970 msgr2=0x7f011c080390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:16.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.130+0000 7f0124200640 1 --2- 172.21.15.134:0/2165286408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f011c10b970 0x7f011c080390 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f011c06c3a0 tx=0x7f010800ddd0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.130+0000 7f0124200640 1 -- 172.21.15.134:0/2165286408 shutdown_connections 2024-07-26T11:25:16.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.130+0000 7f0124200640 1 --2- 172.21.15.134:0/2165286408 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f01040b3ec0 0x7f01040b63d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.130+0000 7f0124200640 1 --2- 172.21.15.134:0/2165286408 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f011c10b970 0x7f011c080390 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.130+0000 7f0124200640 1 --2- 172.21.15.134:0/2165286408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f011c10be60 0x7f011c0808d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.130+0000 7f0124200640 1 --2- 172.21.15.134:0/2165286408 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f011c0790b0 0x7f011c07fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:16.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.130+0000 7f0124200640 1 -- 172.21.15.134:0/2165286408 >> 172.21.15.134:0/2165286408 conn(0x7f011c0708f0 msgr2=0x7f011c10e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:16.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.130+0000 7f0124200640 1 -- 172.21.15.134:0/2165286408 shutdown_connections 2024-07-26T11:25:16.133 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:16.130+0000 7f0124200640 1 -- 172.21.15.134:0/2165286408 wait complete. 2024-07-26T11:25:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:15 smithi189 ceph-mon[36790]: pgmap v763: 129 pgs: 10 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 93 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 37 KiB/s rd, 223 MiB/s wr, 1.50k op/s 2024-07-26T11:25:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:25:17.177 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:25:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:25:18.178 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:17 smithi134 ceph-mon[31966]: pgmap v764: 129 pgs: 16 active+clean+snaptrim, 43 active+clean+snaptrim_wait, 70 active+clean; 61 GiB data, 129 GiB used, 943 GiB / 1.0 TiB avail; 37 KiB/s rd, 235 MiB/s wr, 1.48k op/s 2024-07-26T11:25:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:17 smithi189 ceph-mon[36790]: pgmap v764: 129 pgs: 16 active+clean+snaptrim, 43 active+clean+snaptrim_wait, 70 active+clean; 61 GiB data, 129 GiB used, 943 GiB / 1.0 TiB avail; 37 KiB/s rd, 235 MiB/s wr, 1.48k op/s 2024-07-26T11:25:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:17 smithi193 ceph-mon[37473]: pgmap v764: 129 pgs: 16 active+clean+snaptrim, 43 active+clean+snaptrim_wait, 70 active+clean; 61 GiB data, 129 GiB used, 943 GiB / 1.0 TiB avail; 37 KiB/s rd, 235 MiB/s wr, 1.48k op/s 2024-07-26T11:25:19.649 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.647+0000 7f512a600640 1 -- 172.21.15.134:0/1384098009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5124115f40 msgr2=0x7f5124118360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:19.649 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.647+0000 7f512a600640 1 --2- 172.21.15.134:0/1384098009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5124115f40 0x7f5124118360 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f511c009ca0 tx=0x7f511c02a8c0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.650 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.648+0000 7f512a600640 1 -- 172.21.15.134:0/1384098009 shutdown_connections 2024-07-26T11:25:19.650 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.648+0000 7f512a600640 1 --2- 172.21.15.134:0/1384098009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f512410a740 0x7f5124115a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.649+0000 7f512a600640 1 --2- 172.21.15.134:0/1384098009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5124115f40 0x7f5124118360 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.649+0000 7f512a600640 1 --2- 172.21.15.134:0/1384098009 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f5124109df0 0x7f512410a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.651 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.649+0000 7f512a600640 1 -- 172.21.15.134:0/1384098009 >> 172.21.15.134:0/1384098009 conn(0x7f5124070630 msgr2=0x7f5124070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:19.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.650+0000 7f512a600640 1 -- 172.21.15.134:0/1384098009 shutdown_connections 2024-07-26T11:25:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.651+0000 7f512a600640 1 -- 172.21.15.134:0/1384098009 wait complete. 2024-07-26T11:25:19.655 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.653+0000 7f512a600640 1 Processor -- start 2024-07-26T11:25:19.656 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.654+0000 7f512a600640 1 -- start start 2024-07-26T11:25:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.655+0000 7f512a600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5124109df0 0x7f51241a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.655+0000 7f512a600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f512410a740 0x7f51241a14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.657 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.656+0000 7f5129200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5124109df0 0x7f51241a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:19.658 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.656+0000 7f5129200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5124109df0 0x7f51241a0fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:54412/0 (socket says 172.21.15.134:54412) 2024-07-26T11:25:19.658 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.656+0000 7f512a600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5124115f40 0x7f51241a8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.656+0000 7f5129200640 1 -- 172.21.15.134:0/504532731 learned_addr learned my addr 172.21.15.134:0/504532731 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.656+0000 7f512a600640 1 -- 172.21.15.134:0/504532731 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f512411bad0 con 0x7f512410a740 2024-07-26T11:25:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.657+0000 7f512a600640 1 -- 172.21.15.134:0/504532731 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f512411b950 con 0x7f5124115f40 2024-07-26T11:25:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.657+0000 7f512a600640 1 -- 172.21.15.134:0/504532731 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f512411bc50 con 0x7f5124109df0 2024-07-26T11:25:19.659 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.657+0000 7f5129c00640 1 --2- 172.21.15.134:0/504532731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5124115f40 0x7f51241a8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.658+0000 7f5129200640 1 -- 172.21.15.134:0/504532731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5124115f40 msgr2=0x7f51241a8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.658+0000 7f5129200640 1 --2- 172.21.15.134:0/504532731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5124115f40 0x7f51241a8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.658+0000 7f5129200640 1 -- 172.21.15.134:0/504532731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f512410a740 msgr2=0x7f51241a14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:19.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.658+0000 7f5129200640 1 --2- 172.21.15.134:0/504532731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f512410a740 0x7f51241a14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.658+0000 7f5129200640 1 -- 172.21.15.134:0/504532731 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51241a1db0 con 0x7f5124109df0 2024-07-26T11:25:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.658+0000 7f5129c00640 1 --2- 172.21.15.134:0/504532731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5124115f40 0x7f51241a8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:25:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.659+0000 7f5129200640 1 --2- 172.21.15.134:0/504532731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5124109df0 0x7f51241a0fa0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f511800b4a0 tx=0x7f511800b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.659+0000 7f5121600640 1 -- 172.21.15.134:0/504532731 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5118014070 con 0x7f5124109df0 2024-07-26T11:25:19.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.659+0000 7f5121600640 1 -- 172.21.15.134:0/504532731 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5118010ac0 con 0x7f5124109df0 2024-07-26T11:25:19.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.659+0000 7f512a600640 1 -- 172.21.15.134:0/504532731 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51241a8660 con 0x7f5124109df0 2024-07-26T11:25:19.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.660+0000 7f5121600640 1 -- 172.21.15.134:0/504532731 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5118017c10 con 0x7f5124109df0 2024-07-26T11:25:19.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.660+0000 7f512a600640 1 -- 172.21.15.134:0/504532731 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51241a8c50 con 0x7f5124109df0 2024-07-26T11:25:19.669 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.668+0000 7f5121600640 1 -- 172.21.15.134:0/504532731 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f5118017070 con 0x7f5124109df0 2024-07-26T11:25:19.671 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.670+0000 7f5132bdd740 1 Processor -- start 2024-07-26T11:25:19.674 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.672+0000 7f5132bdd740 1 -- start start 2024-07-26T11:25:19.674 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.673+0000 7f5132bdd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 0x563cc4934c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.676 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.674+0000 7f5121600640 1 --2- 172.21.15.134:0/504532731 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f510c0b3df0 0x7f510c0b6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.676 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.675+0000 7f5121600640 1 -- 172.21.15.134:0/504532731 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9599+0+0 (secure 0 0 0) 0x7f51180e36e0 con 0x7f5124109df0 2024-07-26T11:25:19.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.675+0000 7f5132bdd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45682f0 0x563cc4935180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.675+0000 7f5132bdd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45704d0 0x563cc49356c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.675+0000 7f5123e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45682f0 0x563cc4935180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:19.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.675+0000 7f5129c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 0x563cc4934c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:19.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.675+0000 7f5123e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45682f0 0x563cc4935180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35600/0 (socket says 172.21.15.134:35600) 2024-07-26T11:25:19.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.675+0000 7f5129c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 0x563cc4934c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:54418/0 (socket says 172.21.15.134:54418) 2024-07-26T11:25:19.678 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.675+0000 7f5123e00640 1 -- 172.21.15.134:0/2000180279 learned_addr learned my addr 172.21.15.134:0/2000180279 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:19.678 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.675+0000 7f5132bdd740 1 -- 172.21.15.134:0/2000180279 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563cc48a9270 con 0x563cc45704d0 2024-07-26T11:25:19.678 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.675+0000 7f5132bdd740 1 -- 172.21.15.134:0/2000180279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563cc4898770 con 0x563cc45682f0 2024-07-26T11:25:19.678 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.675+0000 7f5132bdd740 1 -- 172.21.15.134:0/2000180279 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563cc489c7b0 con 0x563cc45911b0 2024-07-26T11:25:19.678 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.676+0000 7f5129c00640 1 -- 172.21.15.134:0/2000180279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45682f0 msgr2=0x563cc4935180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:19.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.676+0000 7f5129c00640 1 --2- 172.21.15.134:0/2000180279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45682f0 0x563cc4935180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.676+0000 7f5129c00640 1 -- 172.21.15.134:0/2000180279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45704d0 msgr2=0x563cc49356c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:19.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.676+0000 7f5129c00640 1 --2- 172.21.15.134:0/2000180279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45704d0 0x563cc49356c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.676+0000 7f5129c00640 1 -- 172.21.15.134:0/2000180279 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cc45fc790 con 0x563cc45911b0 2024-07-26T11:25:19.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.676+0000 7f5129c00640 1 --2- 172.21.15.134:0/2000180279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 0x563cc4934c40 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f511c009bf0 tx=0x7f511c02f500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52cf967f2a05d940 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:19.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.677+0000 7f5116000640 1 -- 172.21.15.134:0/2000180279 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c031450 con 0x563cc45911b0 2024-07-26T11:25:19.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.677+0000 7f5116000640 1 -- 172.21.15.134:0/2000180279 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f511c036070 con 0x563cc45911b0 2024-07-26T11:25:19.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.677+0000 7f5116000640 1 -- 172.21.15.134:0/2000180279 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c0043f0 con 0x563cc45911b0 2024-07-26T11:25:19.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.679+0000 7f5132bdd740 1 -- 172.21.15.134:0/2000180279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 msgr2=0x563cc4934c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:19.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.679+0000 7f5132bdd740 1 --2- 172.21.15.134:0/2000180279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 0x563cc4934c40 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f511c009bf0 tx=0x7f511c02f500 comp rx=0 tx=0).stop 2024-07-26T11:25:19.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.679+0000 7f5132bdd740 1 -- 172.21.15.134:0/2000180279 shutdown_connections 2024-07-26T11:25:19.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.679+0000 7f5132bdd740 1 --2- 172.21.15.134:0/2000180279 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45704d0 0x563cc49356c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.679+0000 7f5132bdd740 1 --2- 172.21.15.134:0/2000180279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45682f0 0x563cc4935180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.679+0000 7f5132bdd740 1 --2- 172.21.15.134:0/2000180279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 0x563cc4934c40 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.682 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.679+0000 7f5132bdd740 1 -- 172.21.15.134:0/2000180279 >> 172.21.15.134:0/2000180279 conn(0x563cc4585300 msgr2=0x563cc4932b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:19.682 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.680+0000 7f5132bdd740 1 -- 172.21.15.134:0/2000180279 shutdown_connections 2024-07-26T11:25:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.680+0000 7f5132bdd740 1 -- 172.21.15.134:0/2000180279 wait complete. 2024-07-26T11:25:19.687 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.680+0000 7f5129200640 1 --2- 172.21.15.134:0/504532731 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f510c0b3df0 0x7f510c0b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.683+0000 7f5129200640 1 --2- 172.21.15.134:0/504532731 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f510c0b3df0 0x7f510c0b6300 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f51180e0bc0 tx=0x7f511800bf90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.685+0000 7f5132bdd740 1 -- start start 2024-07-26T11:25:19.688 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.685+0000 7f5132bdd740 1 Processor -- start 2024-07-26T11:25:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.688+0000 7f5132bdd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45682f0 0x563cc49408c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.688+0000 7f5132bdd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc4940ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.688+0000 7f5132bdd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45911b0 0x563cc494c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.688+0000 7f5132bdd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x563cc47d2ab0 con 0x563cc45911b0 2024-07-26T11:25:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.688+0000 7f5132bdd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563cc4835d50 con 0x563cc45704d0 2024-07-26T11:25:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.688+0000 7f5132bdd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x563cc4943360 con 0x563cc45682f0 2024-07-26T11:25:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.689+0000 7f5132bdd740 10 client.0 _ll_register_callbacks cb 0x563cc45a8a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:25:19.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.689+0000 7f5129c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45911b0 0x563cc494c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.689+0000 7f5129c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45911b0 0x563cc494c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:38164/0 (socket says 172.21.15.134:38164) 2024-07-26T11:25:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.689+0000 7f5129c00640 1 -- 172.21.15.134:0/3544928158 learned_addr learned my addr 172.21.15.134:0/3544928158 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.689+0000 7f5129c00640 1 --2- 172.21.15.134:0/3544928158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45682f0 0x563cc49408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.691+0000 7f5129c00640 1 -- 172.21.15.134:0/3544928158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45682f0 msgr2=0x563cc49408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.691+0000 7f5129c00640 1 --2- 172.21.15.134:0/3544928158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45682f0 0x563cc49408c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.691+0000 7f5129c00640 1 -- 172.21.15.134:0/3544928158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 msgr2=0x563cc4940ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:25:19.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.693+0000 7f5123e00640 1 --2- 172.21.15.134:0/3544928158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc4940ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:19.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.693+0000 7f5129c00640 1 --2- 172.21.15.134:0/3544928158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc4940ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.693+0000 7f5129c00640 1 -- 172.21.15.134:0/3544928158 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cc4675620 con 0x563cc45911b0 2024-07-26T11:25:19.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.693+0000 7f5123e00640 1 --2- 172.21.15.134:0/3544928158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc4940ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:25:19.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.694+0000 7f5129c00640 1 --2- 172.21.15.134:0/3544928158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45911b0 0x563cc494c6b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f511c032ff0 tx=0x7f511c0040d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a9e087dd0344cb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:19.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.694+0000 7f510aa00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:25:19.698 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.695+0000 7f510aa00640 1 -- 172.21.15.134:0/3544928158 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511c031890 con 0x563cc45911b0 2024-07-26T11:25:19.698 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.695+0000 7f510aa00640 1 -- 172.21.15.134:0/3544928158 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9599+0+0 (secure 0 0 0) 0x7f511c043040 con 0x563cc45911b0 2024-07-26T11:25:19.698 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.695+0000 7f5132bdd740 1 -- 172.21.15.134:0/3544928158 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f511c036070 con 0x563cc45911b0 2024-07-26T11:25:19.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.699+0000 7f510aa00640 1 -- 172.21.15.134:0/3544928158 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f511c004b40 con 0x563cc45911b0 2024-07-26T11:25:19.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.699+0000 7f5132bdd740 10 client.16155 fetch_fsmap learned FSMap version 32 2024-07-26T11:25:19.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.699+0000 7f5132bdd740 1 -- 172.21.15.134:0/3544928158 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563cc494f490 con 0x563cc45911b0 2024-07-26T11:25:19.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.701+0000 7f510aa00640 1 -- 172.21.15.134:0/3544928158 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f511c036070 con 0x563cc45911b0 2024-07-26T11:25:19.704 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.701+0000 7f510aa00640 1 -- 172.21.15.134:0/3544928158 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f511c02f790 con 0x563cc45911b0 2024-07-26T11:25:19.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.706+0000 7f5132bdd740 10 client.16155 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:25:19.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.706+0000 7f5132bdd740 10 client.16155 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:25:19.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.706+0000 7f5132bdd740 20 client.16155 populate_metadata read hostname 'smithi134' 2024-07-26T11:25:19.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.706+0000 7f5132bdd740 1 --2- 172.21.15.134:0/3544928158 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563cc454fd80 0x563cc49527f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.706+0000 7f5132bdd740 4 client.16155 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:25:19.709 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.706+0000 7f5132bdd740 1 -- 172.21.15.134:0/3544928158 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cc48ab1c0 con 0x563cc454fd80 2024-07-26T11:25:19.711 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.709+0000 7f5123e00640 1 --2- 172.21.15.134:0/3544928158 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563cc454fd80 0x563cc49527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.714+0000 7f510aa00640 10 client.16155 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.714+0000 7f5123e00640 1 --2- 172.21.15.134:0/3544928158 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563cc454fd80 0x563cc49527f0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=816bfb84aaedd219 server_cookie=3f4a681b5ec8eeab in_seq=0 out_seq=0 2024-07-26T11:25:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.721+0000 7f510aa00640 1 -- 172.21.15.134:0/3544928158 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x563cc48ab1c0 con 0x563cc454fd80 2024-07-26T11:25:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.722+0000 7f510aa00640 10 client.16155 handle_command_reply: tid=0 2024-07-26T11:25:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.722+0000 7f5132bdd740 1 client.16155 shutdown 2024-07-26T11:25:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.722+0000 7f5132bdd740 1 -- 172.21.15.134:0/3544928158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45911b0 msgr2=0x563cc494c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.722+0000 7f5132bdd740 1 --2- 172.21.15.134:0/3544928158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45911b0 0x563cc494c6b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f511c032ff0 tx=0x7f511c0040d0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.724+0000 7f5132bdd740 1 -- 172.21.15.134:0/3544928158 shutdown_connections 2024-07-26T11:25:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.724+0000 7f5132bdd740 1 --2- 172.21.15.134:0/3544928158 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563cc454fd80 0x563cc49527f0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.724+0000 7f5132bdd740 1 --2- 172.21.15.134:0/3544928158 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45911b0 0x563cc494c6b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.724+0000 7f5132bdd740 1 --2- 172.21.15.134:0/3544928158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc4940ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.724+0000 7f5132bdd740 1 --2- 172.21.15.134:0/3544928158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45682f0 0x563cc49408c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.724+0000 7f5132bdd740 1 -- 172.21.15.134:0/3544928158 >> 172.21.15.134:0/3544928158 conn(0x563cc4585300 msgr2=0x563cc4935c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.724+0000 7f510aa00640 0 client.16155 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.727+0000 7f5132bdd740 1 -- 172.21.15.134:0/3544928158 shutdown_connections 2024-07-26T11:25:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.727+0000 7f5132bdd740 1 -- 172.21.15.134:0/3544928158 wait complete. 2024-07-26T11:25:19.773 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.728+0000 7f5132bdd740 20 client.16155 trim_cache size 0 max 16384 2024-07-26T11:25:19.795 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.793+0000 7f5132bdd740 1 Processor -- start 2024-07-26T11:25:19.796 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.794+0000 7f5132bdd740 1 -- start start 2024-07-26T11:25:19.797 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.796+0000 7f5132bdd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45682f0 0x563cc494c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.798 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.796+0000 7f5132bdd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc493a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.797+0000 7f5132bdd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 0x563cc4955810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.797+0000 7f5132bdd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f51040101d0 con 0x563cc45682f0 2024-07-26T11:25:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.797+0000 7f5132bdd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5104011d10 con 0x563cc45704d0 2024-07-26T11:25:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.797+0000 7f5132bdd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f51040108a0 con 0x563cc45911b0 2024-07-26T11:25:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.797+0000 7f5123e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc493a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:19.799 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.797+0000 7f5123e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc493a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35614/0 (socket says 172.21.15.134:35614) 2024-07-26T11:25:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.797+0000 7f5123e00640 1 -- 172.21.15.134:0/4212105283 learned_addr learned my addr 172.21.15.134:0/4212105283 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.798+0000 7f5129c00640 1 --2- 172.21.15.134:0/4212105283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 0x563cc4955810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.798+0000 7f5123e00640 1 -- 172.21.15.134:0/4212105283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 msgr2=0x563cc4955810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:19.800 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.798+0000 7f5123e00640 1 --2- 172.21.15.134:0/4212105283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 0x563cc4955810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.798+0000 7f5123e00640 1 -- 172.21.15.134:0/4212105283 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45682f0 msgr2=0x563cc494c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.798+0000 7f5123e00640 1 --2- 172.21.15.134:0/4212105283 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45682f0 0x563cc494c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.798+0000 7f5123e00640 1 -- 172.21.15.134:0/4212105283 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cc493eb90 con 0x563cc45704d0 2024-07-26T11:25:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.799+0000 7f5129c00640 1 --2- 172.21.15.134:0/4212105283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 0x563cc4955810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:25:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.799+0000 7f5123e00640 1 --2- 172.21.15.134:0/4212105283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc493a410 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x563cc48a93a0 tx=0x7f5110009660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30e7af75b1c28ded server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.800+0000 7f5115600640 1 -- 172.21.15.134:0/4212105283 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5110003ca0 con 0x563cc45704d0 2024-07-26T11:25:19.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.800+0000 7f5115600640 1 -- 172.21.15.134:0/4212105283 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5110020070 con 0x563cc45704d0 2024-07-26T11:25:19.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.800+0000 7f5115600640 1 -- 172.21.15.134:0/4212105283 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511001b520 con 0x563cc45704d0 2024-07-26T11:25:19.804 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.802+0000 7f5132bdd740 1 -- 172.21.15.134:0/4212105283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 msgr2=0x563cc493a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:19.804 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.802+0000 7f5132bdd740 1 --2- 172.21.15.134:0/4212105283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc493a410 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x563cc48a93a0 tx=0x7f5110009660 comp rx=0 tx=0).stop 2024-07-26T11:25:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.805+0000 7f5132bdd740 1 -- 172.21.15.134:0/4212105283 shutdown_connections 2024-07-26T11:25:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.805+0000 7f5132bdd740 1 --2- 172.21.15.134:0/4212105283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 0x563cc4955810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.805+0000 7f5132bdd740 1 --2- 172.21.15.134:0/4212105283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc493a410 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.805+0000 7f5132bdd740 1 --2- 172.21.15.134:0/4212105283 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45682f0 0x563cc494c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.805+0000 7f5132bdd740 1 -- 172.21.15.134:0/4212105283 >> 172.21.15.134:0/4212105283 conn(0x563cc4585300 msgr2=0x563cc4939d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.805+0000 7f5132bdd740 1 -- 172.21.15.134:0/4212105283 shutdown_connections 2024-07-26T11:25:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.805+0000 7f5132bdd740 1 -- 172.21.15.134:0/4212105283 wait complete. 2024-07-26T11:25:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.806+0000 7f5132bdd740 1 -- start start 2024-07-26T11:25:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.807+0000 7f5132bdd740 1 Processor -- start 2024-07-26T11:25:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.808+0000 7f5132bdd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45682f0 0x563cc49532c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.808+0000 7f5132bdd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc4953860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.809+0000 7f5132bdd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 0x563cc49b1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.809+0000 7f5132bdd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f5104011640 con 0x563cc45682f0 2024-07-26T11:25:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.809+0000 7f5132bdd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5104010f70 con 0x563cc45704d0 2024-07-26T11:25:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.809+0000 7f5132bdd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f510400e1d0 con 0x563cc45911b0 2024-07-26T11:25:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.809+0000 7f5132bdd740 10 client.0 _ll_register_callbacks cb 0x7f51100095b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:25:19.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.809+0000 7f5123e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc4953860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:19.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.809+0000 7f5123e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc4953860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35618/0 (socket says 172.21.15.134:35618) 2024-07-26T11:25:19.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.810+0000 7f5123e00640 1 -- 172.21.15.134:0/410984050 learned_addr learned my addr 172.21.15.134:0/410984050 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:19.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.810+0000 7f5129c00640 1 --2- 172.21.15.134:0/410984050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 0x563cc49b1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.811+0000 7f5123e00640 1 -- 172.21.15.134:0/410984050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 msgr2=0x563cc49b1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.811+0000 7f5123e00640 1 --2- 172.21.15.134:0/410984050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 0x563cc49b1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.811+0000 7f5123e00640 1 -- 172.21.15.134:0/410984050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45682f0 msgr2=0x563cc49532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.811+0000 7f5123e00640 1 --2- 172.21.15.134:0/410984050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45682f0 0x563cc49532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:19.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.811+0000 7f5123e00640 1 -- 172.21.15.134:0/410984050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cc49b23d0 con 0x563cc45704d0 2024-07-26T11:25:19.814 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.811+0000 7f510aa00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:25:19.814 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.811+0000 7f5123e00640 1 --2- 172.21.15.134:0/410984050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc4953860 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5110009c70 tx=0x7f5110003c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9de84a6483395c11 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:19.814 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.812+0000 7f510aa00640 1 -- 172.21.15.134:0/410984050 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5110012860 con 0x563cc45704d0 2024-07-26T11:25:19.814 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.812+0000 7f510aa00640 1 -- 172.21.15.134:0/410984050 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9599+0+0 (secure 0 0 0) 0x7f5110012a00 con 0x563cc45704d0 2024-07-26T11:25:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.813+0000 7f5132bdd740 1 -- 172.21.15.134:0/410984050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5110020070 con 0x563cc45704d0 2024-07-26T11:25:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.813+0000 7f510aa00640 1 -- 172.21.15.134:0/410984050 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f511001b2c0 con 0x563cc45704d0 2024-07-26T11:25:19.815 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.814+0000 7f5132bdd740 10 client.26771 fetch_fsmap learned FSMap version 32 2024-07-26T11:25:19.816 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.814+0000 7f5132bdd740 1 -- 172.21.15.134:0/410984050 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563cc49ba8e0 con 0x563cc45704d0 2024-07-26T11:25:19.817 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.815+0000 7f510aa00640 1 -- 172.21.15.134:0/410984050 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5110020070 con 0x563cc45704d0 2024-07-26T11:25:19.818 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.816+0000 7f510aa00640 1 -- 172.21.15.134:0/410984050 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f5110038590 con 0x563cc45704d0 2024-07-26T11:25:19.818 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.817+0000 7f5132bdd740 10 client.26771 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:25:19.818 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.817+0000 7f5132bdd740 10 client.26771 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:25:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.817+0000 7f5132bdd740 20 client.26771 populate_metadata read hostname 'smithi134' 2024-07-26T11:25:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.817+0000 7f5132bdd740 1 --2- 172.21.15.134:0/410984050 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563cc45251c0 0x563cc49bcfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.817+0000 7f5132bdd740 4 client.26771 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:25:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.817+0000 7f5132bdd740 1 -- 172.21.15.134:0/410984050 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563cc48a6c60 con 0x563cc45251c0 2024-07-26T11:25:19.819 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.818+0000 7f5129c00640 1 --2- 172.21.15.134:0/410984050 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563cc45251c0 0x563cc49bcfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.819+0000 7f510aa00640 10 client.26771 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:19.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:19.819+0000 7f5129c00640 1 --2- 172.21.15.134:0/410984050 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563cc45251c0 0x563cc49bcfa0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9b29e8a94c685cc server_cookie=30378614d9aa35d4 in_seq=0 out_seq=0 2024-07-26T11:25:20.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:19 smithi134 ceph-mon[31966]: pgmap v765: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 60 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 87 KiB/s rd, 258 MiB/s wr, 1.49k op/s 2024-07-26T11:25:20.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:25:20.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:25:20.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:25:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:19 smithi189 ceph-mon[36790]: pgmap v765: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 60 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 87 KiB/s rd, 258 MiB/s wr, 1.49k op/s 2024-07-26T11:25:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:25:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:25:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:25:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:19 smithi193 ceph-mon[37473]: pgmap v765: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 60 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 87 KiB/s rd, 258 MiB/s wr, 1.49k op/s 2024-07-26T11:25:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:25:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:25:20.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:25:21.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:20 smithi189 ceph-mon[36790]: pgmap v766: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 60 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 371 KiB/s rd, 225 MiB/s wr, 1.14k op/s 2024-07-26T11:25:21.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:20 smithi134 ceph-mon[31966]: pgmap v766: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 60 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 371 KiB/s rd, 225 MiB/s wr, 1.14k op/s 2024-07-26T11:25:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:20 smithi193 ceph-mon[37473]: pgmap v766: 129 pgs: 18 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 61 active+clean; 60 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 371 KiB/s rd, 225 MiB/s wr, 1.14k op/s 2024-07-26T11:25:23.581 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T11:25:23.581 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T11:25:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:23 smithi193 ceph-mon[37473]: pgmap v767: 129 pgs: 17 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 62 active+clean; 59 GiB data, 126 GiB used, 946 GiB / 1.0 TiB avail; 561 KiB/s rd, 218 MiB/s wr, 1.13k op/s 2024-07-26T11:25:24.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:23 smithi134 ceph-mon[31966]: pgmap v767: 129 pgs: 17 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 62 active+clean; 59 GiB data, 126 GiB used, 946 GiB / 1.0 TiB avail; 561 KiB/s rd, 218 MiB/s wr, 1.13k op/s 2024-07-26T11:25:24.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:23 smithi189 ceph-mon[36790]: pgmap v767: 129 pgs: 17 active+clean+snaptrim, 50 active+clean+snaptrim_wait, 62 active+clean; 59 GiB data, 126 GiB used, 946 GiB / 1.0 TiB avail; 561 KiB/s rd, 218 MiB/s wr, 1.13k op/s 2024-07-26T11:25:24.565 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:25:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.191+0000 7f510aa00640 1 -- 172.21.15.134:0/410984050 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563cc48a6c60 con 0x563cc45251c0 2024-07-26T11:25:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.191+0000 7f510aa00640 10 client.26771 handle_command_reply: tid=0 2024-07-26T11:25:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.191+0000 7f5132bdd740 1 client.26771 shutdown 2024-07-26T11:25:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.191+0000 7f5132bdd740 1 -- 172.21.15.134:0/410984050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 msgr2=0x563cc4953860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.191+0000 7f5132bdd740 1 --2- 172.21.15.134:0/410984050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc4953860 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5110009c70 tx=0x7f5110003c00 comp rx=0 tx=0).stop 2024-07-26T11:25:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.191+0000 7f5132bdd740 1 -- 172.21.15.134:0/410984050 shutdown_connections 2024-07-26T11:25:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.191+0000 7f5132bdd740 1 --2- 172.21.15.134:0/410984050 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x563cc45251c0 0x563cc49bcfa0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.191+0000 7f510aa00640 0 client.26771 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.191+0000 7f5132bdd740 1 --2- 172.21.15.134:0/410984050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563cc45911b0 0x563cc49b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.191+0000 7f5132bdd740 1 --2- 172.21.15.134:0/410984050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc45704d0 0x563cc4953860 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.191+0000 7f5132bdd740 1 --2- 172.21.15.134:0/410984050 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x563cc45682f0 0x563cc49532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.191+0000 7f5132bdd740 1 -- 172.21.15.134:0/410984050 >> 172.21.15.134:0/410984050 conn(0x563cc4585300 msgr2=0x563cc4941b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.192+0000 7f5132bdd740 1 -- 172.21.15.134:0/410984050 shutdown_connections 2024-07-26T11:25:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.192+0000 7f5132bdd740 1 -- 172.21.15.134:0/410984050 wait complete. 2024-07-26T11:25:25.195 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.192+0000 7f5132bdd740 20 client.26771 trim_cache size 0 max 16384 2024-07-26T11:25:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:25:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:25:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:25:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:25:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.197+0000 7f512a600640 1 -- 172.21.15.134:0/504532731 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f510c0b3df0 msgr2=0x7f510c0b6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.197+0000 7f512a600640 1 --2- 172.21.15.134:0/504532731 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f510c0b3df0 0x7f510c0b6300 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f51180e0bc0 tx=0x7f511800bf90 comp rx=0 tx=0).stop 2024-07-26T11:25:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.197+0000 7f512a600640 1 -- 172.21.15.134:0/504532731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5124109df0 msgr2=0x7f51241a0fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.197+0000 7f512a600640 1 --2- 172.21.15.134:0/504532731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5124109df0 0x7f51241a0fa0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f511800b4a0 tx=0x7f511800b960 comp rx=0 tx=0).stop 2024-07-26T11:25:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.197+0000 7f512a600640 1 -- 172.21.15.134:0/504532731 shutdown_connections 2024-07-26T11:25:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.197+0000 7f512a600640 1 --2- 172.21.15.134:0/504532731 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f510c0b3df0 0x7f510c0b6300 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.197+0000 7f512a600640 1 --2- 172.21.15.134:0/504532731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5124115f40 0x7f51241a8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.197+0000 7f512a600640 1 --2- 172.21.15.134:0/504532731 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f512410a740 0x7f51241a14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.197+0000 7f512a600640 1 --2- 172.21.15.134:0/504532731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5124109df0 0x7f51241a0fa0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.197+0000 7f512a600640 1 -- 172.21.15.134:0/504532731 >> 172.21.15.134:0/504532731 conn(0x7f5124070630 msgr2=0x7f512410c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.198+0000 7f512a600640 1 -- 172.21.15.134:0/504532731 shutdown_connections 2024-07-26T11:25:25.200 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:25.198+0000 7f512a600640 1 -- 172.21.15.134:0/504532731 wait complete. 2024-07-26T11:25:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:25 smithi193 ceph-mon[37473]: pgmap v768: 129 pgs: 15 active+clean+snaptrim, 44 active+clean+snaptrim_wait, 70 active+clean; 59 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 548 KiB/s rd, 234 MiB/s wr, 1.22k op/s 2024-07-26T11:25:26.179 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:25 smithi134 ceph-mon[31966]: pgmap v768: 129 pgs: 15 active+clean+snaptrim, 44 active+clean+snaptrim_wait, 70 active+clean; 59 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 548 KiB/s rd, 234 MiB/s wr, 1.22k op/s 2024-07-26T11:25:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:25 smithi189 ceph-mon[36790]: pgmap v768: 129 pgs: 15 active+clean+snaptrim, 44 active+clean+snaptrim_wait, 70 active+clean; 59 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 548 KiB/s rd, 234 MiB/s wr, 1.22k op/s 2024-07-26T11:25:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:27 smithi193 ceph-mon[37473]: pgmap v769: 129 pgs: 12 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 83 active+clean; 59 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 640 KiB/s rd, 230 MiB/s wr, 1.24k op/s 2024-07-26T11:25:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:27 smithi134 ceph-mon[31966]: pgmap v769: 129 pgs: 12 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 83 active+clean; 59 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 640 KiB/s rd, 230 MiB/s wr, 1.24k op/s 2024-07-26T11:25:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:27 smithi189 ceph-mon[36790]: pgmap v769: 129 pgs: 12 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 83 active+clean; 59 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 640 KiB/s rd, 230 MiB/s wr, 1.24k op/s 2024-07-26T11:25:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:29 smithi193 ceph-mon[37473]: pgmap v770: 129 pgs: 11 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 86 active+clean; 59 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 692 KiB/s rd, 219 MiB/s wr, 1.25k op/s 2024-07-26T11:25:30.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:29 smithi134 ceph-mon[31966]: pgmap v770: 129 pgs: 11 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 86 active+clean; 59 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 692 KiB/s rd, 219 MiB/s wr, 1.25k op/s 2024-07-26T11:25:30.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:29 smithi189 ceph-mon[36790]: pgmap v770: 129 pgs: 11 active+clean+snaptrim, 32 active+clean+snaptrim_wait, 86 active+clean; 59 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 692 KiB/s rd, 219 MiB/s wr, 1.25k op/s 2024-07-26T11:25:31.832 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:31 smithi134 ceph-mon[31966]: pgmap v771: 129 pgs: 11 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 87 active+clean; 59 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 634 KiB/s rd, 220 MiB/s wr, 1.39k op/s 2024-07-26T11:25:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:31 smithi193 ceph-mon[37473]: pgmap v771: 129 pgs: 11 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 87 active+clean; 59 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 634 KiB/s rd, 220 MiB/s wr, 1.39k op/s 2024-07-26T11:25:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:31 smithi189 ceph-mon[36790]: pgmap v771: 129 pgs: 11 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 87 active+clean; 59 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 634 KiB/s rd, 220 MiB/s wr, 1.39k op/s 2024-07-26T11:25:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:33 smithi134 ceph-mon[31966]: pgmap v772: 129 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 96 active+clean; 59 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 461 KiB/s rd, 185 MiB/s wr, 1.22k op/s 2024-07-26T11:25:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:33 smithi189 ceph-mon[36790]: pgmap v772: 129 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 96 active+clean; 59 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 461 KiB/s rd, 185 MiB/s wr, 1.22k op/s 2024-07-26T11:25:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:33 smithi193 ceph-mon[37473]: pgmap v772: 129 pgs: 10 active+clean+snaptrim, 23 active+clean+snaptrim_wait, 96 active+clean; 59 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 461 KiB/s rd, 185 MiB/s wr, 1.22k op/s 2024-07-26T11:25:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:35 smithi134 ceph-mon[31966]: pgmap v773: 129 pgs: 9 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 105 active+clean; 59 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 221 KiB/s rd, 183 MiB/s wr, 1.25k op/s 2024-07-26T11:25:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:35 smithi189 ceph-mon[36790]: pgmap v773: 129 pgs: 9 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 105 active+clean; 59 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 221 KiB/s rd, 183 MiB/s wr, 1.25k op/s 2024-07-26T11:25:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:35 smithi193 ceph-mon[37473]: pgmap v773: 129 pgs: 9 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 105 active+clean; 59 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 221 KiB/s rd, 183 MiB/s wr, 1.25k op/s 2024-07-26T11:25:37.176 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:36 smithi134 ceph-mon[31966]: pgmap v774: 129 pgs: 9 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 105 active+clean; 60 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 221 KiB/s rd, 209 MiB/s wr, 1.44k op/s 2024-07-26T11:25:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:36 smithi189 ceph-mon[36790]: pgmap v774: 129 pgs: 9 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 105 active+clean; 60 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 221 KiB/s rd, 209 MiB/s wr, 1.44k op/s 2024-07-26T11:25:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:36 smithi193 ceph-mon[37473]: pgmap v774: 129 pgs: 9 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 105 active+clean; 60 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 221 KiB/s rd, 209 MiB/s wr, 1.44k op/s 2024-07-26T11:25:38.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.651+0000 7fb1c8a00640 1 -- 172.21.15.134:0/55965489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1c410a740 msgr2=0x7fb1c4115a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:38.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.651+0000 7fb1c8a00640 1 --2- 172.21.15.134:0/55965489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1c410a740 0x7fb1c4115a00 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009900 tx=0x7fb1b002a440 comp rx=0 tx=0).stop 2024-07-26T11:25:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.652+0000 7fb1c8a00640 1 -- 172.21.15.134:0/55965489 shutdown_connections 2024-07-26T11:25:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.652+0000 7fb1c8a00640 1 --2- 172.21.15.134:0/55965489 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1c4115f40 0x7fb1c4118360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.652+0000 7fb1c8a00640 1 --2- 172.21.15.134:0/55965489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1c410a740 0x7fb1c4115a00 secure :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009900 tx=0x7fb1b002a440 comp rx=0 tx=0).stop 2024-07-26T11:25:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.652+0000 7fb1c8a00640 1 --2- 172.21.15.134:0/55965489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1c4109df0 0x7fb1c410a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.652+0000 7fb1c8a00640 1 -- 172.21.15.134:0/55965489 >> 172.21.15.134:0/55965489 conn(0x7fb1c4070630 msgr2=0x7fb1c4070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:38.656 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.654+0000 7fb1c8a00640 1 -- 172.21.15.134:0/55965489 shutdown_connections 2024-07-26T11:25:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.654+0000 7fb1c8a00640 1 -- 172.21.15.134:0/55965489 wait complete. 2024-07-26T11:25:38.657 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.655+0000 7fb1c8a00640 1 Processor -- start 2024-07-26T11:25:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.658+0000 7fb1c8a00640 1 -- start start 2024-07-26T11:25:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.658+0000 7fb1c8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1c4109df0 0x7fb1c41a2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.659+0000 7fb1c8a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1c4115f40 0x7fb1c41a2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.659+0000 7fb1c8a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1c41a7e00 0x7fb1c41a8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.659+0000 7fb1c2800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1c4109df0 0x7fb1c41a2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.659+0000 7fb1c2800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1c4109df0 0x7fb1c41a2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:33448/0 (socket says 172.21.15.134:33448) 2024-07-26T11:25:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.659+0000 7fb1c1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1c4115f40 0x7fb1c41a2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.659+0000 7fb1c1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1c4115f40 0x7fb1c41a2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:54420/0 (socket says 172.21.15.134:54420) 2024-07-26T11:25:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.659+0000 7fb1c2800640 1 -- 172.21.15.134:0/83702598 learned_addr learned my addr 172.21.15.134:0/83702598 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.660+0000 7fb1c8a00640 1 -- 172.21.15.134:0/83702598 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c411bab0 con 0x7fb1c41a7e00 2024-07-26T11:25:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.660+0000 7fb1c8a00640 1 -- 172.21.15.134:0/83702598 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c411b930 con 0x7fb1c4109df0 2024-07-26T11:25:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.660+0000 7fb1c8a00640 1 -- 172.21.15.134:0/83702598 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c411bc30 con 0x7fb1c4115f40 2024-07-26T11:25:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.661+0000 7fb1c2800640 1 -- 172.21.15.134:0/83702598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1c4115f40 msgr2=0x7fb1c41a2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.661+0000 7fb1c2800640 1 --2- 172.21.15.134:0/83702598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1c4115f40 0x7fb1c41a2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.661+0000 7fb1c2800640 1 -- 172.21.15.134:0/83702598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1c41a7e00 msgr2=0x7fb1c41a8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.661+0000 7fb1c2800640 1 --2- 172.21.15.134:0/83702598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1c41a7e00 0x7fb1c41a8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.661+0000 7fb1c2800640 1 -- 172.21.15.134:0/83702598 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c41a89e0 con 0x7fb1c4109df0 2024-07-26T11:25:38.666 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.662+0000 7fb1c2800640 1 --2- 172.21.15.134:0/83702598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1c4109df0 0x7fb1c41a2690 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb1b400e9f0 tx=0x7fb1b400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:38.667 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.664+0000 7fb1bb400640 1 -- 172.21.15.134:0/83702598 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b400cd80 con 0x7fb1c4109df0 2024-07-26T11:25:38.667 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.664+0000 7fb1bb400640 1 -- 172.21.15.134:0/83702598 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1b40054d0 con 0x7fb1c4109df0 2024-07-26T11:25:38.667 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.664+0000 7fb1c8a00640 1 -- 172.21.15.134:0/83702598 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c410b790 con 0x7fb1c4109df0 2024-07-26T11:25:38.667 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.665+0000 7fb1bb400640 1 -- 172.21.15.134:0/83702598 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b4010690 con 0x7fb1c4109df0 2024-07-26T11:25:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.670+0000 7fb1c8a00640 1 -- 172.21.15.134:0/83702598 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c410bc50 con 0x7fb1c4109df0 2024-07-26T11:25:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.679+0000 7fb1bb400640 1 -- 172.21.15.134:0/83702598 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fb1b4010830 con 0x7fb1c4109df0 2024-07-26T11:25:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.681+0000 7fb1d03eb740 1 Processor -- start 2024-07-26T11:25:38.684 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.682+0000 7fb1bb400640 1 --2- 172.21.15.134:0/83702598 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1a80b3f10 0x7fb1a80b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.684 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.682+0000 7fb1bb400640 1 -- 172.21.15.134:0/83702598 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9599+0+0 (secure 0 0 0) 0x7fb1b40e5750 con 0x7fb1c4109df0 2024-07-26T11:25:38.688 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.685+0000 7fb1c3200640 1 --2- 172.21.15.134:0/83702598 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1a80b3f10 0x7fb1a80b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:38.688 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.686+0000 7fb1c3200640 1 --2- 172.21.15.134:0/83702598 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1a80b3f10 0x7fb1a80b6420 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc00ab10 tx=0x7fb1bc009250 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:38.689 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.687+0000 7fb1d03eb740 1 -- start start 2024-07-26T11:25:38.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.689+0000 7fb1d03eb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe98f1b0 0x55ecbed3b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.690+0000 7fb1d03eb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe9662f0 0x55ecbed3b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.690+0000 7fb1d03eb740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe96e4d0 0x55ecbed3bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.690+0000 7fb1d03eb740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ecbeca7270 con 0x55ecbe96e4d0 2024-07-26T11:25:38.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.690+0000 7fb1d03eb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ecbec96770 con 0x55ecbe9662f0 2024-07-26T11:25:38.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.690+0000 7fb1d03eb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ecbec9a7b0 con 0x55ecbe98f1b0 2024-07-26T11:25:38.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.691+0000 7fb1c1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe9662f0 0x55ecbed3b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:38.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.691+0000 7fb1c1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe9662f0 0x55ecbed3b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:33450/0 (socket says 172.21.15.134:33450) 2024-07-26T11:25:38.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.691+0000 7fb1c1e00640 1 -- 172.21.15.134:0/3380460450 learned_addr learned my addr 172.21.15.134:0/3380460450 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:38.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.692+0000 7fb1c3200640 1 --2- 172.21.15.134:0/3380460450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe96e4d0 0x55ecbed3bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:38.695 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.693+0000 7fb1c1e00640 1 -- 172.21.15.134:0/3380460450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe98f1b0 msgr2=0x55ecbed3b450 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:25:38.695 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.693+0000 7fb1c2800640 1 --2- 172.21.15.134:0/3380460450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe98f1b0 0x55ecbed3b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.694+0000 7fb1c1e00640 1 --2- 172.21.15.134:0/3380460450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe98f1b0 0x55ecbed3b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.694+0000 7fb1c1e00640 1 -- 172.21.15.134:0/3380460450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe96e4d0 msgr2=0x55ecbed3bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.694+0000 7fb1c1e00640 1 --2- 172.21.15.134:0/3380460450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe96e4d0 0x55ecbed3bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.694+0000 7fb1c1e00640 1 -- 172.21.15.134:0/3380460450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecbe9fa790 con 0x55ecbe9662f0 2024-07-26T11:25:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.694+0000 7fb1c1e00640 1 --2- 172.21.15.134:0/3380460450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe9662f0 0x55ecbed3b990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb1b00094b0 tx=0x7fb1b0004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b53e84ece21a7a9c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.695+0000 7fb1c2800640 1 --2- 172.21.15.134:0/3380460450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe98f1b0 0x55ecbed3b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:25:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.695+0000 7fb1af400640 1 -- 172.21.15.134:0/3380460450 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b00046c0 con 0x55ecbe9662f0 2024-07-26T11:25:38.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.695+0000 7fb1af400640 1 -- 172.21.15.134:0/3380460450 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1b0004860 con 0x55ecbe9662f0 2024-07-26T11:25:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.695+0000 7fb1af400640 1 -- 172.21.15.134:0/3380460450 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b0031a50 con 0x55ecbe9662f0 2024-07-26T11:25:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.696+0000 7fb1d03eb740 1 -- 172.21.15.134:0/3380460450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe9662f0 msgr2=0x55ecbed3b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:38.698 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.696+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/3380460450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe9662f0 0x55ecbed3b990 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fb1b00094b0 tx=0x7fb1b0004b40 comp rx=0 tx=0).stop 2024-07-26T11:25:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.697+0000 7fb1d03eb740 1 -- 172.21.15.134:0/3380460450 shutdown_connections 2024-07-26T11:25:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.697+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/3380460450 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe96e4d0 0x55ecbed3bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.697+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/3380460450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe9662f0 0x55ecbed3b990 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.697+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/3380460450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe98f1b0 0x55ecbed3b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.699 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.697+0000 7fb1d03eb740 1 -- 172.21.15.134:0/3380460450 >> 172.21.15.134:0/3380460450 conn(0x55ecbe983300 msgr2=0x55ecbed39340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:38.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.700+0000 7fb1d03eb740 1 -- 172.21.15.134:0/3380460450 shutdown_connections 2024-07-26T11:25:38.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.701+0000 7fb1d03eb740 1 -- 172.21.15.134:0/3380460450 wait complete. 2024-07-26T11:25:38.704 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.702+0000 7fb1d03eb740 1 -- start start 2024-07-26T11:25:38.704 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.702+0000 7fb1d03eb740 1 Processor -- start 2024-07-26T11:25:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.703+0000 7fb1d03eb740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe9662f0 0x55ecbed47150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.703+0000 7fb1d03eb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe96e4d0 0x55ecbed47730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.703+0000 7fb1d03eb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe98f1b0 0x55ecbed52e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.703+0000 7fb1d03eb740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55ecbebd0ab0 con 0x55ecbe9662f0 2024-07-26T11:25:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.703+0000 7fb1d03eb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ecbec33d50 con 0x55ecbe98f1b0 2024-07-26T11:25:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.703+0000 7fb1d03eb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55ecbed49b70 con 0x55ecbe96e4d0 2024-07-26T11:25:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.703+0000 7fb1d03eb740 10 client.0 _ll_register_callbacks cb 0x55ecbe9a6a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:25:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.703+0000 7fb1c3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe98f1b0 0x55ecbed52e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.703+0000 7fb1c3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe98f1b0 0x55ecbed52e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:33452/0 (socket says 172.21.15.134:33452) 2024-07-26T11:25:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.703+0000 7fb1c3200640 1 -- 172.21.15.134:0/334443057 learned_addr learned my addr 172.21.15.134:0/334443057 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.704+0000 7fb1c1e00640 1 --2- 172.21.15.134:0/334443057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe96e4d0 0x55ecbed47730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.704+0000 7fb1c3200640 1 -- 172.21.15.134:0/334443057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe96e4d0 msgr2=0x55ecbed47730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:38.706 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.704+0000 7fb1c3200640 1 --2- 172.21.15.134:0/334443057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe96e4d0 0x55ecbed47730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.704+0000 7fb1c3200640 1 -- 172.21.15.134:0/334443057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe9662f0 msgr2=0x55ecbed47150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.704+0000 7fb1c3200640 1 --2- 172.21.15.134:0/334443057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe9662f0 0x55ecbed47150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.704+0000 7fb1c3200640 1 -- 172.21.15.134:0/334443057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecbea73620 con 0x55ecbe98f1b0 2024-07-26T11:25:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.704+0000 7fb1c1e00640 1 --2- 172.21.15.134:0/334443057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe96e4d0 0x55ecbed47730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:25:38.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.705+0000 7fb1c3200640 1 --2- 172.21.15.134:0/334443057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe98f1b0 0x55ecbed52e60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc01be00 tx=0x7fb1bc004bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eaf10825410b02 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.705+0000 7fb1a3e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:25:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.705+0000 7fb1a3e00640 1 -- 172.21.15.134:0/334443057 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc023410 con 0x55ecbe98f1b0 2024-07-26T11:25:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.705+0000 7fb1a3e00640 1 -- 172.21.15.134:0/334443057 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9599+0+0 (secure 0 0 0) 0x7fb1bc0235b0 con 0x55ecbe98f1b0 2024-07-26T11:25:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.706+0000 7fb1d03eb740 1 -- 172.21.15.134:0/334443057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1b0004860 con 0x55ecbe98f1b0 2024-07-26T11:25:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.707+0000 7fb1a3e00640 1 -- 172.21.15.134:0/334443057 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb1bc0237b0 con 0x55ecbe98f1b0 2024-07-26T11:25:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.707+0000 7fb1d03eb740 10 client.26783 fetch_fsmap learned FSMap version 32 2024-07-26T11:25:38.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.707+0000 7fb1d03eb740 1 -- 172.21.15.134:0/334443057 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ecbed55c60 con 0x55ecbe98f1b0 2024-07-26T11:25:38.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.708+0000 7fb1a3e00640 1 -- 172.21.15.134:0/334443057 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1b0004860 con 0x55ecbe98f1b0 2024-07-26T11:25:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.708+0000 7fb1a3e00640 1 -- 172.21.15.134:0/334443057 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fb1bc016950 con 0x55ecbe98f1b0 2024-07-26T11:25:38.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.712+0000 7fb1d03eb740 10 client.26783 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:25:38.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.712+0000 7fb1d03eb740 10 client.26783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:25:38.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.714+0000 7fb1d03eb740 20 client.26783 populate_metadata read hostname 'smithi134' 2024-07-26T11:25:38.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.714+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/334443057 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55ecbe94dd80 0x55ecbed59190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.714+0000 7fb1d03eb740 4 client.26783 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:25:38.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.714+0000 7fb1d03eb740 1 -- 172.21.15.134:0/334443057 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecbeca91c0 con 0x55ecbe94dd80 2024-07-26T11:25:38.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.719+0000 7fb1c2800640 1 --2- 172.21.15.134:0/334443057 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55ecbe94dd80 0x55ecbed59190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:38.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.720+0000 7fb1a3e00640 10 client.26783 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:38.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.720+0000 7fb1c2800640 1 --2- 172.21.15.134:0/334443057 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55ecbe94dd80 0x55ecbed59190 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4196ef0f31f71136 server_cookie=88f20ac4f17c43f6 in_seq=0 out_seq=0 2024-07-26T11:25:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.728+0000 7fb1a3e00640 1 -- 172.21.15.134:0/334443057 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55ecbeca91c0 con 0x55ecbe94dd80 2024-07-26T11:25:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.728+0000 7fb1a3e00640 10 client.26783 handle_command_reply: tid=0 2024-07-26T11:25:38.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.728+0000 7fb1d03eb740 1 client.26783 shutdown 2024-07-26T11:25:38.738 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.736+0000 7fb1d03eb740 1 -- 172.21.15.134:0/334443057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe98f1b0 msgr2=0x55ecbed52e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:38.738 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.736+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/334443057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe98f1b0 0x55ecbed52e60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc01be00 tx=0x7fb1bc004bc0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.740 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.737+0000 7fb1d03eb740 1 -- 172.21.15.134:0/334443057 shutdown_connections 2024-07-26T11:25:38.740 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.737+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/334443057 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55ecbe94dd80 0x55ecbed59190 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.740 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.737+0000 7fb1a3e00640 0 client.26783 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:38.740 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.737+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/334443057 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe96e4d0 0x55ecbed47730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.740 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.737+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/334443057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe98f1b0 0x55ecbed52e60 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.740 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.737+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/334443057 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe9662f0 0x55ecbed47150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.740 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.737+0000 7fb1d03eb740 1 -- 172.21.15.134:0/334443057 >> 172.21.15.134:0/334443057 conn(0x55ecbe983300 msgr2=0x55ecbed3c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.739+0000 7fb1d03eb740 1 -- 172.21.15.134:0/334443057 shutdown_connections 2024-07-26T11:25:38.741 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.739+0000 7fb1d03eb740 1 -- 172.21.15.134:0/334443057 wait complete. 2024-07-26T11:25:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.740+0000 7fb1d03eb740 20 client.26783 trim_cache size 0 max 16384 2024-07-26T11:25:38.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.780+0000 7fb1d03eb740 1 Processor -- start 2024-07-26T11:25:38.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.780+0000 7fb1d03eb740 1 -- start start 2024-07-26T11:25:38.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.781+0000 7fb1d03eb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe94dd80 0x55ecbed45e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.781+0000 7fb1d03eb740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe9662f0 0x55ecbed46b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.781+0000 7fb1d03eb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe96e4d0 0x55ecbed52c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.781+0000 7fb1d03eb740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c41b74e0 con 0x55ecbe9662f0 2024-07-26T11:25:38.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.781+0000 7fb1d03eb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c41b9020 con 0x55ecbe96e4d0 2024-07-26T11:25:38.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.781+0000 7fb1d03eb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c41b7bb0 con 0x55ecbe94dd80 2024-07-26T11:25:38.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.782+0000 7fb1c3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe96e4d0 0x55ecbed52c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:38.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.782+0000 7fb1c3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe96e4d0 0x55ecbed52c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:33462/0 (socket says 172.21.15.134:33462) 2024-07-26T11:25:38.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.782+0000 7fb1c3200640 1 -- 172.21.15.134:0/195054481 learned_addr learned my addr 172.21.15.134:0/195054481 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:38.785 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.782+0000 7fb1c2800640 1 --2- 172.21.15.134:0/195054481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe94dd80 0x55ecbed45e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.782+0000 7fb1c3200640 1 -- 172.21.15.134:0/195054481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe94dd80 msgr2=0x55ecbed45e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.783+0000 7fb1c3200640 1 --2- 172.21.15.134:0/195054481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe94dd80 0x55ecbed45e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.783+0000 7fb1c3200640 1 -- 172.21.15.134:0/195054481 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe9662f0 msgr2=0x55ecbed46b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.783+0000 7fb1c3200640 1 --2- 172.21.15.134:0/195054481 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe9662f0 0x55ecbed46b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.783+0000 7fb1c3200640 1 -- 172.21.15.134:0/195054481 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecbed3d750 con 0x55ecbe96e4d0 2024-07-26T11:25:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.783+0000 7fb1c2800640 1 --2- 172.21.15.134:0/195054481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe94dd80 0x55ecbed45e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:25:38.786 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.784+0000 7fb1c3200640 1 --2- 172.21.15.134:0/195054481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe96e4d0 0x55ecbed52c50 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc022f00 tx=0x7fb1bc01bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e1433c54856d129 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.785+0000 7fb1aea00640 1 -- 172.21.15.134:0/195054481 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc031070 con 0x55ecbe96e4d0 2024-07-26T11:25:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.785+0000 7fb1aea00640 1 -- 172.21.15.134:0/195054481 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1bc017db0 con 0x55ecbe96e4d0 2024-07-26T11:25:38.787 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.785+0000 7fb1aea00640 1 -- 172.21.15.134:0/195054481 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc0237b0 con 0x55ecbe96e4d0 2024-07-26T11:25:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.787+0000 7fb1d03eb740 1 -- 172.21.15.134:0/195054481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe96e4d0 msgr2=0x55ecbed52c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:38.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.787+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/195054481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe96e4d0 0x55ecbed52c50 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc022f00 tx=0x7fb1bc01bd90 comp rx=0 tx=0).stop 2024-07-26T11:25:38.791 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.787+0000 7fb1d03eb740 1 -- 172.21.15.134:0/195054481 shutdown_connections 2024-07-26T11:25:38.791 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.787+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/195054481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe96e4d0 0x55ecbed52c50 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.791 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.787+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/195054481 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe9662f0 0x55ecbed46b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.791 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.787+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/195054481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe94dd80 0x55ecbed45e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.791 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.787+0000 7fb1d03eb740 1 -- 172.21.15.134:0/195054481 >> 172.21.15.134:0/195054481 conn(0x55ecbe983300 msgr2=0x55ecbed46410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:38.791 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.787+0000 7fb1d03eb740 1 -- 172.21.15.134:0/195054481 shutdown_connections 2024-07-26T11:25:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.787+0000 7fb1d03eb740 1 -- 172.21.15.134:0/195054481 wait complete. 2024-07-26T11:25:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.788+0000 7fb1d03eb740 1 -- start start 2024-07-26T11:25:38.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.788+0000 7fb1d03eb740 1 Processor -- start 2024-07-26T11:25:38.796 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.794+0000 7fb1d03eb740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe94dd80 0x55ecbed4edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.797 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.795+0000 7fb1d03eb740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe9662f0 0x55ecbed4f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.795+0000 7fb1d03eb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe96e4d0 0x55ecbedbad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.795+0000 7fb1d03eb740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c41b8950 con 0x55ecbe9662f0 2024-07-26T11:25:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.795+0000 7fb1d03eb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c41b8280 con 0x55ecbe96e4d0 2024-07-26T11:25:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.795+0000 7fb1d03eb740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c41b54a0 con 0x55ecbe94dd80 2024-07-26T11:25:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.795+0000 7fb1d03eb740 10 client.0 _ll_register_callbacks cb 0x7fb1bc00cd20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:25:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.795+0000 7fb1c3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe96e4d0 0x55ecbedbad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.796+0000 7fb1c3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe96e4d0 0x55ecbedbad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:33472/0 (socket says 172.21.15.134:33472) 2024-07-26T11:25:38.798 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.796+0000 7fb1c3200640 1 -- 172.21.15.134:0/1492015862 learned_addr learned my addr 172.21.15.134:0/1492015862 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:38.799 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.796+0000 7fb1c3200640 1 -- 172.21.15.134:0/1492015862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe94dd80 msgr2=0x55ecbed4edc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:25:38.799 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.796+0000 7fb1c3200640 1 --2- 172.21.15.134:0/1492015862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe94dd80 0x55ecbed4edc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.796+0000 7fb1c3200640 1 -- 172.21.15.134:0/1492015862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe9662f0 msgr2=0x55ecbed4f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.796+0000 7fb1c3200640 1 --2- 172.21.15.134:0/1492015862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe9662f0 0x55ecbed4f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.796+0000 7fb1c3200640 1 -- 172.21.15.134:0/1492015862 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecbedbb2f0 con 0x55ecbe96e4d0 2024-07-26T11:25:38.800 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.797+0000 7fb1c3200640 1 --2- 172.21.15.134:0/1492015862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe96e4d0 0x55ecbedbad80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc01bad0 tx=0x7fb1bc02be70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=262bf00484865941 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:38.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.800+0000 7fb1a3e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:25:38.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.800+0000 7fb1a3e00640 1 -- 172.21.15.134:0/1492015862 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc031040 con 0x55ecbe96e4d0 2024-07-26T11:25:38.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.800+0000 7fb1a3e00640 1 -- 172.21.15.134:0/1492015862 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9599+0+0 (secure 0 0 0) 0x7fb1bc022970 con 0x55ecbe96e4d0 2024-07-26T11:25:38.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.800+0000 7fb1d03eb740 1 -- 172.21.15.134:0/1492015862 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb1bc017db0 con 0x55ecbe96e4d0 2024-07-26T11:25:38.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.800+0000 7fb1a3e00640 1 -- 172.21.15.134:0/1492015862 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fb1bc046050 con 0x55ecbe96e4d0 2024-07-26T11:25:38.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.800+0000 7fb1d03eb740 10 client.26789 fetch_fsmap learned FSMap version 32 2024-07-26T11:25:38.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.800+0000 7fb1d03eb740 1 -- 172.21.15.134:0/1492015862 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ecbed34aa0 con 0x55ecbe96e4d0 2024-07-26T11:25:38.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.801+0000 7fb1a3e00640 1 -- 172.21.15.134:0/1492015862 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1bc017db0 con 0x55ecbe96e4d0 2024-07-26T11:25:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.803+0000 7fb1a3e00640 1 -- 172.21.15.134:0/1492015862 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fb1bc016950 con 0x55ecbe96e4d0 2024-07-26T11:25:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.803+0000 7fb1d03eb740 10 client.26789 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:25:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.803+0000 7fb1d03eb740 10 client.26789 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:25:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.803+0000 7fb1d03eb740 20 client.26789 populate_metadata read hostname 'smithi134' 2024-07-26T11:25:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.803+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/1492015862 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55ecbe98f1b0 0x55ecbed37220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.803+0000 7fb1d03eb740 4 client.26789 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:25:38.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.803+0000 7fb1d03eb740 1 -- 172.21.15.134:0/1492015862 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ecbeca4c60 con 0x55ecbe98f1b0 2024-07-26T11:25:38.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.806+0000 7fb1c2800640 1 --2- 172.21.15.134:0/1492015862 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55ecbe98f1b0 0x55ecbed37220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:38.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.808+0000 7fb1c2800640 1 --2- 172.21.15.134:0/1492015862 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55ecbe98f1b0 0x55ecbed37220 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b846630c490ad153 server_cookie=8137650032b48216 in_seq=0 out_seq=0 2024-07-26T11:25:38.820 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:38.813+0000 7fb1a3e00640 10 client.26789 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:40.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:39 smithi134 ceph-mon[31966]: pgmap v775: 129 pgs: 8 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 106 active+clean; 60 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 176 KiB/s rd, 226 MiB/s wr, 1.57k op/s 2024-07-26T11:25:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:39 smithi189 ceph-mon[36790]: pgmap v775: 129 pgs: 8 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 106 active+clean; 60 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 176 KiB/s rd, 226 MiB/s wr, 1.57k op/s 2024-07-26T11:25:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:39 smithi193 ceph-mon[37473]: pgmap v775: 129 pgs: 8 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 106 active+clean; 60 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 176 KiB/s rd, 226 MiB/s wr, 1.57k op/s 2024-07-26T11:25:40.474 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.467+0000 7fb1a3e00640 1 -- 172.21.15.134:0/1492015862 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ecbeca4c60 con 0x55ecbe98f1b0 2024-07-26T11:25:40.475 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.467+0000 7fb1a3e00640 10 client.26789 handle_command_reply: tid=0 2024-07-26T11:25:40.475 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.471+0000 7fb1d03eb740 1 client.26789 shutdown 2024-07-26T11:25:40.479 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.477+0000 7fb1d03eb740 1 -- 172.21.15.134:0/1492015862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe96e4d0 msgr2=0x55ecbedbad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:40.479 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.477+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/1492015862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe96e4d0 0x55ecbedbad80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc01bad0 tx=0x7fb1bc02be70 comp rx=0 tx=0).stop 2024-07-26T11:25:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.477+0000 7fb1d03eb740 1 -- 172.21.15.134:0/1492015862 shutdown_connections 2024-07-26T11:25:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.477+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/1492015862 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55ecbe98f1b0 0x55ecbed37220 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.477+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/1492015862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ecbe96e4d0 0x55ecbedbad80 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.477+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/1492015862 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55ecbe9662f0 0x55ecbed4f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.477+0000 7fb1a3e00640 0 client.26789 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.477+0000 7fb1d03eb740 1 --2- 172.21.15.134:0/1492015862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ecbe94dd80 0x55ecbed4edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.477+0000 7fb1d03eb740 1 -- 172.21.15.134:0/1492015862 >> 172.21.15.134:0/1492015862 conn(0x55ecbe983300 msgr2=0x55ecbed3c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:40.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.478+0000 7fb1d03eb740 1 -- 172.21.15.134:0/1492015862 shutdown_connections 2024-07-26T11:25:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.479+0000 7fb1d03eb740 1 -- 172.21.15.134:0/1492015862 wait complete. 2024-07-26T11:25:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.479+0000 7fb1d03eb740 20 client.26789 trim_cache size 0 max 16384 2024-07-26T11:25:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:25:40.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:25:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:25:40.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:25:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.488+0000 7fb1c8a00640 1 -- 172.21.15.134:0/83702598 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1a80b3f10 msgr2=0x7fb1a80b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.488+0000 7fb1c8a00640 1 --2- 172.21.15.134:0/83702598 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1a80b3f10 0x7fb1a80b6420 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc00ab10 tx=0x7fb1bc009250 comp rx=0 tx=0).stop 2024-07-26T11:25:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.488+0000 7fb1c8a00640 1 -- 172.21.15.134:0/83702598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1c4109df0 msgr2=0x7fb1c41a2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:40.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.488+0000 7fb1c8a00640 1 --2- 172.21.15.134:0/83702598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1c4109df0 0x7fb1c41a2690 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb1b400e9f0 tx=0x7fb1b400edf0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.490+0000 7fb1c8a00640 1 -- 172.21.15.134:0/83702598 shutdown_connections 2024-07-26T11:25:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.490+0000 7fb1c8a00640 1 --2- 172.21.15.134:0/83702598 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fb1a80b3f10 0x7fb1a80b6420 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.490+0000 7fb1c8a00640 1 --2- 172.21.15.134:0/83702598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb1c4115f40 0x7fb1c41a2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.490+0000 7fb1c8a00640 1 --2- 172.21.15.134:0/83702598 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fb1c41a7e00 0x7fb1c41a8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.490+0000 7fb1c8a00640 1 --2- 172.21.15.134:0/83702598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1c4109df0 0x7fb1c41a2690 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.490+0000 7fb1c8a00640 1 -- 172.21.15.134:0/83702598 >> 172.21.15.134:0/83702598 conn(0x7fb1c4070630 msgr2=0x7fb1c40728f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.490+0000 7fb1c8a00640 1 -- 172.21.15.134:0/83702598 shutdown_connections 2024-07-26T11:25:40.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:40.490+0000 7fb1c8a00640 1 -- 172.21.15.134:0/83702598 wait complete. 2024-07-26T11:25:40.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.637+0000 7f7d1e200640 1 -- 172.21.15.134:0/3802520985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d18108ef0 msgr2=0x7f7d18109300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:40.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.637+0000 7f7d16000640 1 -- 172.21.15.134:0/3802520985 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d04035020 con 0x7f7d18108ef0 2024-07-26T11:25:40.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.637+0000 7f7d1e200640 1 --2- 172.21.15.134:0/3802520985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d18108ef0 0x7f7d18109300 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7d04009900 tx=0x7f7d0402a410 comp rx=0 tx=0).stop 2024-07-26T11:25:40.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.637+0000 7f7d1e200640 1 -- 172.21.15.134:0/3802520985 shutdown_connections 2024-07-26T11:25:40.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.637+0000 7f7d1e200640 1 --2- 172.21.15.134:0/3802520985 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d18108ef0 0x7f7d18109300 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.637+0000 7f7d1e200640 1 --2- 172.21.15.134:0/3802520985 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d18109840 0x7f7d18110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.637+0000 7f7d1e200640 1 --2- 172.21.15.134:0/3802520985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d181035b0 0x7f7d181039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.637+0000 7f7d1e200640 1 -- 172.21.15.134:0/3802520985 >> 172.21.15.134:0/3802520985 conn(0x7f7d180ff040 msgr2=0x7f7d18101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:40.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.638+0000 7f7d1e200640 1 -- 172.21.15.134:0/3802520985 shutdown_connections 2024-07-26T11:25:40.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.638+0000 7f7d1e200640 1 -- 172.21.15.134:0/3802520985 wait complete. 2024-07-26T11:25:40.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.641+0000 7f7d1e200640 1 Processor -- start 2024-07-26T11:25:40.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.641+0000 7f7d1e200640 1 -- start start 2024-07-26T11:25:40.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.642+0000 7f7d1e200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d181035b0 0x7f7d180742b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.643+0000 7f7d1e200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d18108ef0 0x7f7d18074830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.643+0000 7f7d17400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d181035b0 0x7f7d180742b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:40.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.643+0000 7f7d1e200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d18109840 0x7f7d1806e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.643+0000 7f7d16a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d18108ef0 0x7f7d18074830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:40.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.643+0000 7f7d16a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d18108ef0 0x7f7d18074830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38046/0 (socket says 172.21.15.134:38046) 2024-07-26T11:25:40.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.644+0000 7f7d16a00640 1 -- 172.21.15.134:0/3545237583 learned_addr learned my addr 172.21.15.134:0/3545237583 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:40.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.644+0000 7f7d1e200640 1 -- 172.21.15.134:0/3545237583 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1806ccc0 con 0x7f7d18109840 2024-07-26T11:25:40.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.644+0000 7f7d1e200640 1 -- 172.21.15.134:0/3545237583 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1806cb40 con 0x7f7d18108ef0 2024-07-26T11:25:40.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.644+0000 7f7d1e200640 1 -- 172.21.15.134:0/3545237583 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1806ce40 con 0x7f7d181035b0 2024-07-26T11:25:40.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.644+0000 7f7d17e00640 1 --2- 172.21.15.134:0/3545237583 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d18109840 0x7f7d1806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:40.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.644+0000 7f7d16a00640 1 -- 172.21.15.134:0/3545237583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d181035b0 msgr2=0x7f7d180742b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:40.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.644+0000 7f7d16a00640 1 --2- 172.21.15.134:0/3545237583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d181035b0 0x7f7d180742b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.644+0000 7f7d16a00640 1 -- 172.21.15.134:0/3545237583 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d18109840 msgr2=0x7f7d1806e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:40.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.644+0000 7f7d16a00640 1 --2- 172.21.15.134:0/3545237583 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d18109840 0x7f7d1806e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.644+0000 7f7d16a00640 1 -- 172.21.15.134:0/3545237583 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1806ecd0 con 0x7f7d18108ef0 2024-07-26T11:25:40.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.649+0000 7f7d16a00640 1 --2- 172.21.15.134:0/3545237583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d18108ef0 0x7f7d18074830 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7d04000c00 tx=0x7f7d04005590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:40.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.653+0000 7f7d0fe00640 1 -- 172.21.15.134:0/3545237583 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0403f020 con 0x7f7d18108ef0 2024-07-26T11:25:40.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.653+0000 7f7d0fe00640 1 -- 172.21.15.134:0/3545237583 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d04004460 con 0x7f7d18108ef0 2024-07-26T11:25:40.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.653+0000 7f7d0fe00640 1 -- 172.21.15.134:0/3545237583 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d04035070 con 0x7f7d18108ef0 2024-07-26T11:25:40.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.653+0000 7f7d1e200640 1 -- 172.21.15.134:0/3545237583 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d1806ef60 con 0x7f7d18108ef0 2024-07-26T11:25:40.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.653+0000 7f7d1e200640 1 -- 172.21.15.134:0/3545237583 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d181acbe0 con 0x7f7d18108ef0 2024-07-26T11:25:40.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.662+0000 7f7d0fe00640 1 -- 172.21.15.134:0/3545237583 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f7d04042020 con 0x7f7d18108ef0 2024-07-26T11:25:40.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.665+0000 7f7d0fe00640 1 --2- 172.21.15.134:0/3545237583 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7cfc0b3ec0 0x7f7cfc0b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.665+0000 7f7d0fe00640 1 -- 172.21.15.134:0/3545237583 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9599+0+0 (secure 0 0 0) 0x7f7d04104340 con 0x7f7d18108ef0 2024-07-26T11:25:40.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.670+0000 7f7d17400640 1 --2- 172.21.15.134:0/3545237583 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7cfc0b3ec0 0x7f7cfc0b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:40.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.671+0000 7f7d17400640 1 --2- 172.21.15.134:0/3545237583 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7cfc0b3ec0 0x7f7cfc0b63d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7d18074ff0 tx=0x7f7d0800fa80 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:40.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.678+0000 7f7d24fe2740 1 Processor -- start 2024-07-26T11:25:40.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.678+0000 7f7d24fe2740 1 -- start start 2024-07-26T11:25:40.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.679+0000 7f7d24fe2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802c9350 0x55bb8066dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.679+0000 7f7d24fe2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802a0490 0x55bb8066e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.679+0000 7f7d24fe2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802a8670 0x55bb8066e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.679+0000 7f7d24fe2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55bb805e4be0 con 0x55bb802c9350 2024-07-26T11:25:40.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.679+0000 7f7d24fe2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bb805d40e0 con 0x55bb802a8670 2024-07-26T11:25:40.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.679+0000 7f7d24fe2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bb805d8120 con 0x55bb802a0490 2024-07-26T11:25:40.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.680+0000 7f7d17e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802a8670 0x55bb8066e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:40.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.680+0000 7f7d16a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802a0490 0x55bb8066e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:40.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.681+0000 7f7d16a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802a0490 0x55bb8066e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:39318/0 (socket says 172.21.15.134:39318) 2024-07-26T11:25:40.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.681+0000 7f7d17e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802a8670 0x55bb8066e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38062/0 (socket says 172.21.15.134:38062) 2024-07-26T11:25:40.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.682+0000 7f7d17e00640 1 -- 172.21.15.134:0/2961849262 learned_addr learned my addr 172.21.15.134:0/2961849262 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:40.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.681+0000 7f7d17400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802c9350 0x55bb8066dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:40.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.682+0000 7f7d16a00640 1 -- 172.21.15.134:0/2961849262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802a8670 msgr2=0x55bb8066e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:40.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.682+0000 7f7d16a00640 1 --2- 172.21.15.134:0/2961849262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802a8670 0x55bb8066e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.682+0000 7f7d16a00640 1 -- 172.21.15.134:0/2961849262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802c9350 msgr2=0x55bb8066dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:40.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.683+0000 7f7d16a00640 1 --2- 172.21.15.134:0/2961849262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802c9350 0x55bb8066dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.683+0000 7f7d16a00640 1 -- 172.21.15.134:0/2961849262 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb80334930 con 0x55bb802a0490 2024-07-26T11:25:40.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.683+0000 7f7d16a00640 1 --2- 172.21.15.134:0/2961849262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802a0490 0x55bb8066e2c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7d04101660 tx=0x7f7d04004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dadc62620a100750 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:40.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.684+0000 7f7d03e00640 1 -- 172.21.15.134:0/2961849262 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d040cde60 con 0x55bb802a0490 2024-07-26T11:25:40.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.684+0000 7f7d03e00640 1 -- 172.21.15.134:0/2961849262 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d040cbe50 con 0x55bb802a0490 2024-07-26T11:25:40.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.684+0000 7f7d03e00640 1 -- 172.21.15.134:0/2961849262 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d040c6db0 con 0x55bb802a0490 2024-07-26T11:25:40.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.684+0000 7f7d17400640 1 --2- 172.21.15.134:0/2961849262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802c9350 0x55bb8066dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:25:40.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.686+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2961849262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802a0490 msgr2=0x55bb8066e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:40.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.687+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/2961849262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802a0490 0x55bb8066e2c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7d04101660 tx=0x7f7d04004610 comp rx=0 tx=0).stop 2024-07-26T11:25:40.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.691+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2961849262 shutdown_connections 2024-07-26T11:25:40.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.691+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/2961849262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802a0490 0x55bb8066e2c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.691+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/2961849262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802a8670 0x55bb8066e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.691+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/2961849262 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802c9350 0x55bb8066dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.691+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2961849262 >> 172.21.15.134:0/2961849262 conn(0x55bb802bd740 msgr2=0x55bb8066bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:40.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.692+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2961849262 shutdown_connections 2024-07-26T11:25:40.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.692+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2961849262 wait complete. 2024-07-26T11:25:40.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.697+0000 7f7d24fe2740 1 -- start start 2024-07-26T11:25:40.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.697+0000 7f7d24fe2740 1 Processor -- start 2024-07-26T11:25:40.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.700+0000 7f7d24fe2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802a0490 0x55bb80679430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.700+0000 7f7d24fe2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802a8670 0x55bb80679970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.700+0000 7f7d24fe2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802c9350 0x55bb806855d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.700+0000 7f7d24fe2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55bb80437890 con 0x55bb802a8670 2024-07-26T11:25:40.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.700+0000 7f7d24fe2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55bb80570320 con 0x55bb802a0490 2024-07-26T11:25:40.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.700+0000 7f7d24fe2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55bb8067c320 con 0x55bb802c9350 2024-07-26T11:25:40.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.701+0000 7f7d24fe2740 10 client.0 _ll_register_callbacks cb 0x55bb8059c000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:25:40.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.701+0000 7f7d16a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802a8670 0x55bb80679970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:40.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.701+0000 7f7d16a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802a8670 0x55bb80679970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:56680/0 (socket says 172.21.15.134:56680) 2024-07-26T11:25:40.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.701+0000 7f7d16a00640 1 -- 172.21.15.134:0/2852195177 learned_addr learned my addr 172.21.15.134:0/2852195177 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:40.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.702+0000 7f7d16a00640 1 -- 172.21.15.134:0/2852195177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802c9350 msgr2=0x55bb806855d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:25:40.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.702+0000 7f7d16a00640 1 --2- 172.21.15.134:0/2852195177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802c9350 0x55bb806855d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.702+0000 7f7d16a00640 1 -- 172.21.15.134:0/2852195177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802a0490 msgr2=0x55bb80679430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:25:40.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.702+0000 7f7d16a00640 1 --2- 172.21.15.134:0/2852195177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802a0490 0x55bb80679430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.702+0000 7f7d16a00640 1 -- 172.21.15.134:0/2852195177 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb8067a160 con 0x55bb802a8670 2024-07-26T11:25:40.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.702+0000 7f7d16a00640 1 --2- 172.21.15.134:0/2852195177 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802a8670 0x55bb80679970 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7d04101630 tx=0x7f7d040caa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a4e84656873eb21 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:40.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.702+0000 7f7d00c00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:25:40.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.703+0000 7f7d00c00640 1 -- 172.21.15.134:0/2852195177 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d040d3bd0 con 0x55bb802a8670 2024-07-26T11:25:40.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.703+0000 7f7d00c00640 1 -- 172.21.15.134:0/2852195177 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9599+0+0 (secure 0 0 0) 0x7f7d040c7110 con 0x55bb802a8670 2024-07-26T11:25:40.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.707+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2852195177 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d040cbe50 con 0x55bb802a8670 2024-07-26T11:25:40.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.711+0000 7f7d00c00640 1 -- 172.21.15.134:0/2852195177 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7d04101a30 con 0x55bb802a8670 2024-07-26T11:25:40.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.711+0000 7f7d24fe2740 10 client.16164 fetch_fsmap learned FSMap version 32 2024-07-26T11:25:40.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.711+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2852195177 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55bb806880e0 con 0x55bb802a8670 2024-07-26T11:25:40.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.715+0000 7f7d00c00640 1 -- 172.21.15.134:0/2852195177 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d040cbe50 con 0x55bb802a8670 2024-07-26T11:25:40.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.715+0000 7f7d00c00640 1 -- 172.21.15.134:0/2852195177 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f7d04003b50 con 0x55bb802a8670 2024-07-26T11:25:40.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.720+0000 7f7d24fe2740 10 client.16164 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:25:40.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.720+0000 7f7d24fe2740 10 client.16164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:25:40.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.720+0000 7f7d24fe2740 20 client.16164 populate_metadata read hostname 'smithi134' 2024-07-26T11:25:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.720+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/2852195177 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55bb80287cd0 0x55bb8068b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.721+0000 7f7d24fe2740 4 client.16164 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:25:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.721+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2852195177 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb805e6b30 con 0x55bb80287cd0 2024-07-26T11:25:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.721+0000 7f7d17400640 1 --2- 172.21.15.134:0/2852195177 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55bb80287cd0 0x55bb8068b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.721+0000 7f7d17400640 1 --2- 172.21.15.134:0/2852195177 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55bb80287cd0 0x55bb8068b440 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3b7721c126e368 server_cookie=e6f67163df46704e in_seq=0 out_seq=0 2024-07-26T11:25:40.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.721+0000 7f7d00c00640 10 client.16164 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:40.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.724+0000 7f7d00c00640 1 -- 172.21.15.134:0/2852195177 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55bb805e6b30 con 0x55bb80287cd0 2024-07-26T11:25:40.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.724+0000 7f7d00c00640 10 client.16164 handle_command_reply: tid=0 2024-07-26T11:25:40.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.724+0000 7f7d24fe2740 1 client.16164 shutdown 2024-07-26T11:25:40.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.725+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2852195177 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802a8670 msgr2=0x55bb80679970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:40.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.725+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/2852195177 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802a8670 0x55bb80679970 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7d04101630 tx=0x7f7d040caa20 comp rx=0 tx=0).stop 2024-07-26T11:25:40.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.725+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2852195177 shutdown_connections 2024-07-26T11:25:40.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.725+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/2852195177 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55bb80287cd0 0x55bb8068b440 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.725+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/2852195177 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802c9350 0x55bb806855d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.725+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/2852195177 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802a8670 0x55bb80679970 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.725+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/2852195177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802a0490 0x55bb80679430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.725+0000 7f7d00c00640 0 client.16164 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:40.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.725+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2852195177 >> 172.21.15.134:0/2852195177 conn(0x55bb802bd740 msgr2=0x55bb80670f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:40.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.734+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2852195177 shutdown_connections 2024-07-26T11:25:40.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.734+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2852195177 wait complete. 2024-07-26T11:25:40.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.734+0000 7f7d24fe2740 20 client.16164 trim_cache size 0 max 16384 2024-07-26T11:25:40.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.774+0000 7f7d24fe2740 1 Processor -- start 2024-07-26T11:25:40.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.774+0000 7f7d24fe2740 1 -- start start 2024-07-26T11:25:40.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.775+0000 7f7d24fe2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802a0490 0x55bb80674c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.776+0000 7f7d24fe2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802c9350 0x55bb80690b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.776+0000 7f7d24fe2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb8025d080 0x55bb80691060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.776+0000 7f7d17400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802a0490 0x55bb80674c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:40.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.776+0000 7f7d17400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802a0490 0x55bb80674c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:39324/0 (socket says 172.21.15.134:39324) 2024-07-26T11:25:40.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.776+0000 7f7d24fe2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf8021210 con 0x55bb8025d080 2024-07-26T11:25:40.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.776+0000 7f7d24fe2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf8022d50 con 0x55bb802c9350 2024-07-26T11:25:40.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.776+0000 7f7d24fe2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf80218e0 con 0x55bb802a0490 2024-07-26T11:25:40.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.776+0000 7f7d16a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802c9350 0x55bb80690b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:40.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.776+0000 7f7d17400640 1 -- 172.21.15.134:0/2469957104 learned_addr learned my addr 172.21.15.134:0/2469957104 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:40.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.776+0000 7f7d17e00640 1 --2- 172.21.15.134:0/2469957104 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb8025d080 0x55bb80691060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:40.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.777+0000 7f7d17400640 1 -- 172.21.15.134:0/2469957104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802c9350 msgr2=0x55bb80690b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:40.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.777+0000 7f7d17400640 1 --2- 172.21.15.134:0/2469957104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802c9350 0x55bb80690b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.777+0000 7f7d17400640 1 -- 172.21.15.134:0/2469957104 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb8025d080 msgr2=0x55bb80691060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:40.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.777+0000 7f7d17400640 1 --2- 172.21.15.134:0/2469957104 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb8025d080 0x55bb80691060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.777+0000 7f7d17400640 1 -- 172.21.15.134:0/2469957104 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb80679ce0 con 0x55bb802a0490 2024-07-26T11:25:40.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.777+0000 7f7d17400640 1 --2- 172.21.15.134:0/2469957104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802a0490 0x55bb80674c00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x55bb8068bac0 tx=0x7f7d0801b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f99be030eae82a5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:40.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.778+0000 7f7d03400640 1 -- 172.21.15.134:0/2469957104 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0802c070 con 0x55bb802a0490 2024-07-26T11:25:40.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.778+0000 7f7d03400640 1 -- 172.21.15.134:0/2469957104 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d08026360 con 0x55bb802a0490 2024-07-26T11:25:40.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.778+0000 7f7d03400640 1 -- 172.21.15.134:0/2469957104 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0801de60 con 0x55bb802a0490 2024-07-26T11:25:40.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.779+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2469957104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802a0490 msgr2=0x55bb80674c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:40.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.779+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/2469957104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802a0490 0x55bb80674c00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x55bb8068bac0 tx=0x7f7d0801b000 comp rx=0 tx=0).stop 2024-07-26T11:25:40.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.780+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2469957104 shutdown_connections 2024-07-26T11:25:40.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.780+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/2469957104 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb8025d080 0x55bb80691060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.780+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/2469957104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802c9350 0x55bb80690b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.780+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/2469957104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb802a0490 0x55bb80674c00 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.780+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2469957104 >> 172.21.15.134:0/2469957104 conn(0x55bb802bd740 msgr2=0x55bb8066f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:40.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.782+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2469957104 shutdown_connections 2024-07-26T11:25:40.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.782+0000 7f7d24fe2740 1 -- 172.21.15.134:0/2469957104 wait complete. 2024-07-26T11:25:40.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.783+0000 7f7d24fe2740 1 -- start start 2024-07-26T11:25:40.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.783+0000 7f7d24fe2740 1 Processor -- start 2024-07-26T11:25:40.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.785+0000 7f7d24fe2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb8025d080 0x55bb8067df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.786+0000 7f7d24fe2740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802a0490 0x55bb8067e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.786+0000 7f7d24fe2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802c9350 0x55bb806ecc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.786+0000 7f7d24fe2740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf8022680 con 0x55bb802a0490 2024-07-26T11:25:40.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.786+0000 7f7d24fe2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf8021fb0 con 0x55bb802c9350 2024-07-26T11:25:40.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.786+0000 7f7d24fe2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf801f210 con 0x55bb8025d080 2024-07-26T11:25:40.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.786+0000 7f7d17e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802c9350 0x55bb806ecc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:40.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.786+0000 7f7d17e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802c9350 0x55bb806ecc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38076/0 (socket says 172.21.15.134:38076) 2024-07-26T11:25:40.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.786+0000 7f7d17e00640 1 -- 172.21.15.134:0/1760459798 learned_addr learned my addr 172.21.15.134:0/1760459798 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:40.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.787+0000 7f7d17e00640 1 -- 172.21.15.134:0/1760459798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb8025d080 msgr2=0x55bb8067df50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:25:40.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.787+0000 7f7d17e00640 1 --2- 172.21.15.134:0/1760459798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb8025d080 0x55bb8067df50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.787+0000 7f7d17e00640 1 -- 172.21.15.134:0/1760459798 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802a0490 msgr2=0x55bb8067e4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:25:40.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.789+0000 7f7d17e00640 1 --2- 172.21.15.134:0/1760459798 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802a0490 0x55bb8067e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.789+0000 7f7d17e00640 1 -- 172.21.15.134:0/1760459798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb806ed280 con 0x55bb802c9350 2024-07-26T11:25:40.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.789+0000 7f7d17e00640 1 --2- 172.21.15.134:0/1760459798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802c9350 0x55bb806ecc10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7d10012a00 tx=0x7f7d10012ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b4f0ab757555d35 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:40.791 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.789+0000 7f7d24fe2740 10 client.0 _ll_register_callbacks cb 0x7f7d040ca360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:25:40.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.789+0000 7f7d24fe2740 1 -- 172.21.15.134:0/1760459798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d08026360 con 0x55bb802c9350 2024-07-26T11:25:40.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.789+0000 7f7d00c00640 10 client.26804 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:25:40.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.790+0000 7f7d00c00640 1 -- 172.21.15.134:0/1760459798 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d10010c60 con 0x55bb802c9350 2024-07-26T11:25:40.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.790+0000 7f7d00c00640 1 -- 172.21.15.134:0/1760459798 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9599+0+0 (secure 0 0 0) 0x7f7d10010e00 con 0x55bb802c9350 2024-07-26T11:25:40.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.823+0000 7f7d00c00640 1 -- 172.21.15.134:0/1760459798 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7d10003970 con 0x55bb802c9350 2024-07-26T11:25:40.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.823+0000 7f7d24fe2740 10 client.26804 fetch_fsmap learned FSMap version 32 2024-07-26T11:25:40.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.823+0000 7f7d24fe2740 1 -- 172.21.15.134:0/1760459798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=32}) -- 0x55bb806f4650 con 0x55bb802c9350 2024-07-26T11:25:40.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.824+0000 7f7d00c00640 1 -- 172.21.15.134:0/1760459798 <== mon.2 v2:172.21.15.189:3300/0 4 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f7d1001a410 con 0x55bb802c9350 2024-07-26T11:25:40.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.824+0000 7f7d24fe2740 10 client.26804 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:25:40.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.824+0000 7f7d24fe2740 10 client.26804 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:25:40.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.824+0000 7f7d24fe2740 20 client.26804 populate_metadata read hostname 'smithi134' 2024-07-26T11:25:40.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.824+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/1760459798 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55bb806f4c40 0x55bb806f7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:40.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.825+0000 7f7d24fe2740 4 client.26804 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-07-26T11:25:40.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.825+0000 7f7d24fe2740 1 -- 172.21.15.134:0/1760459798 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55bb805e25d0 con 0x55bb806f4c40 2024-07-26T11:25:40.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.826+0000 7f7d17400640 1 --2- 172.21.15.134:0/1760459798 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55bb806f4c40 0x55bb806f7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:40.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.827+0000 7f7d00c00640 10 client.26804 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:40.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.827+0000 7f7d17400640 1 --2- 172.21.15.134:0/1760459798 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55bb806f4c40 0x55bb806f7110 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e0a05c66fdbf7ff server_cookie=485d4806a6d37634 in_seq=0 out_seq=0 2024-07-26T11:25:40.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.831+0000 7f7d00c00640 1 -- 172.21.15.134:0/1760459798 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55bb805e25d0 con 0x55bb806f4c40 2024-07-26T11:25:40.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.831+0000 7f7d00c00640 10 client.26804 handle_command_reply: tid=0 2024-07-26T11:25:40.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.831+0000 7f7d24fe2740 1 client.26804 shutdown 2024-07-26T11:25:40.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.832+0000 7f7d24fe2740 1 -- 172.21.15.134:0/1760459798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802c9350 msgr2=0x55bb806ecc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:40.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.832+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/1760459798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802c9350 0x55bb806ecc10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7d10012a00 tx=0x7f7d10012ec0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.832+0000 7f7d24fe2740 1 -- 172.21.15.134:0/1760459798 shutdown_connections 2024-07-26T11:25:40.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.832+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/1760459798 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55bb806f4c40 0x55bb806f7110 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.832+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/1760459798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bb802c9350 0x55bb806ecc10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.832+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/1760459798 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55bb802a0490 0x55bb8067e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.832+0000 7f7d24fe2740 1 --2- 172.21.15.134:0/1760459798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bb8025d080 0x55bb8067df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.832+0000 7f7d00c00640 0 client.26804 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:40.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.832+0000 7f7d24fe2740 1 -- 172.21.15.134:0/1760459798 >> 172.21.15.134:0/1760459798 conn(0x55bb802bd740 msgr2=0x55bb8066f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:40.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.833+0000 7f7d24fe2740 1 -- 172.21.15.134:0/1760459798 shutdown_connections 2024-07-26T11:25:40.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.833+0000 7f7d24fe2740 1 -- 172.21.15.134:0/1760459798 wait complete. 2024-07-26T11:25:40.836 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.833+0000 7f7d24fe2740 20 client.26804 trim_cache size 0 max 16384 2024-07-26T11:25:40.836 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:25:40.836 INFO:teuthology.orchestra.run.smithi134.stdout: "return_code": 0, 2024-07-26T11:25:40.837 INFO:teuthology.orchestra.run.smithi134.stdout: "scrub_tag": "465d9b68-1461-43c0-ae82-d21b9b99c1b1", 2024-07-26T11:25:40.837 INFO:teuthology.orchestra.run.smithi134.stdout: "mode": "asynchronous" 2024-07-26T11:25:40.837 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:25:40.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.842+0000 7f7d1e200640 1 -- 172.21.15.134:0/3545237583 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7cfc0b3ec0 msgr2=0x7f7cfc0b63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:40.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.842+0000 7f7d1e200640 1 --2- 172.21.15.134:0/3545237583 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7cfc0b3ec0 0x7f7cfc0b63d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7d18074ff0 tx=0x7f7d0800fa80 comp rx=0 tx=0).stop 2024-07-26T11:25:40.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.842+0000 7f7d1e200640 1 -- 172.21.15.134:0/3545237583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d18108ef0 msgr2=0x7f7d18074830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:40.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.842+0000 7f7d1e200640 1 --2- 172.21.15.134:0/3545237583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d18108ef0 0x7f7d18074830 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7d04000c00 tx=0x7f7d04005590 comp rx=0 tx=0).stop 2024-07-26T11:25:40.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.846+0000 7f7d1e200640 1 -- 172.21.15.134:0/3545237583 shutdown_connections 2024-07-26T11:25:40.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.846+0000 7f7d1e200640 1 --2- 172.21.15.134:0/3545237583 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f7cfc0b3ec0 0x7f7cfc0b63d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.846+0000 7f7d1e200640 1 --2- 172.21.15.134:0/3545237583 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f7d18109840 0x7f7d1806e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.846+0000 7f7d1e200640 1 --2- 172.21.15.134:0/3545237583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d18108ef0 0x7f7d18074830 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.846+0000 7f7d1e200640 1 --2- 172.21.15.134:0/3545237583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d181035b0 0x7f7d180742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:40.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.846+0000 7f7d1e200640 1 -- 172.21.15.134:0/3545237583 >> 172.21.15.134:0/3545237583 conn(0x7f7d180ff040 msgr2=0x7f7d18106360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:40.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.846+0000 7f7d1e200640 1 -- 172.21.15.134:0/3545237583 shutdown_connections 2024-07-26T11:25:40.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:40.846+0000 7f7d1e200640 1 -- 172.21.15.134:0/3545237583 wait complete. 2024-07-26T11:25:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:41 smithi134 ceph-mon[31966]: pgmap v776: 129 pgs: 7 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 107 active+clean; 60 GiB data, 127 GiB used, 945 GiB / 1.0 TiB avail; 123 KiB/s rd, 232 MiB/s wr, 1.59k op/s 2024-07-26T11:25:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:41 smithi134 ceph-mon[31966]: scrub queued for path: / 2024-07-26T11:25:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:41 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:25:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:41 smithi134 ceph-mon[31966]: scrub summary: active paths [/] 2024-07-26T11:25:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:41 smithi189 ceph-mon[36790]: pgmap v776: 129 pgs: 7 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 107 active+clean; 60 GiB data, 127 GiB used, 945 GiB / 1.0 TiB avail; 123 KiB/s rd, 232 MiB/s wr, 1.59k op/s 2024-07-26T11:25:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:41 smithi189 ceph-mon[36790]: scrub queued for path: / 2024-07-26T11:25:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:41 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:25:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:41 smithi189 ceph-mon[36790]: scrub summary: active paths [/] 2024-07-26T11:25:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:41 smithi193 ceph-mon[37473]: pgmap v776: 129 pgs: 7 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 107 active+clean; 60 GiB data, 127 GiB used, 945 GiB / 1.0 TiB avail; 123 KiB/s rd, 232 MiB/s wr, 1.59k op/s 2024-07-26T11:25:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:41 smithi193 ceph-mon[37473]: scrub queued for path: / 2024-07-26T11:25:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:41 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:25:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:41 smithi193 ceph-mon[37473]: scrub summary: active paths [/] 2024-07-26T11:25:43.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:42 smithi134 ceph-mon[31966]: scrub summary: active paths [/] 2024-07-26T11:25:43.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:42 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:25:43.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:42 smithi134 ceph-mon[31966]: scrub summary: idle+waiting paths [/] 2024-07-26T11:25:43.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:42 smithi134 ceph-mon[31966]: pgmap v777: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 60 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 188 KiB/s rd, 216 MiB/s wr, 1.39k op/s 2024-07-26T11:25:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:42 smithi189 ceph-mon[36790]: scrub summary: active paths [/] 2024-07-26T11:25:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:42 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:25:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:42 smithi189 ceph-mon[36790]: scrub summary: idle+waiting paths [/] 2024-07-26T11:25:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:42 smithi189 ceph-mon[36790]: pgmap v777: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 60 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 188 KiB/s rd, 216 MiB/s wr, 1.39k op/s 2024-07-26T11:25:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:42 smithi193 ceph-mon[37473]: scrub summary: active paths [/] 2024-07-26T11:25:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:42 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:25:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:42 smithi193 ceph-mon[37473]: scrub summary: idle+waiting paths [/] 2024-07-26T11:25:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:42 smithi193 ceph-mon[37473]: pgmap v777: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 60 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 188 KiB/s rd, 216 MiB/s wr, 1.39k op/s 2024-07-26T11:25:44.244 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub status 2024-07-26T11:25:45.136 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:25:46.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:45 smithi134 ceph-mon[31966]: pgmap v778: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 60 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 113 KiB/s rd, 227 MiB/s wr, 1.39k op/s 2024-07-26T11:25:46.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:45 smithi189 ceph-mon[36790]: pgmap v778: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 60 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 113 KiB/s rd, 227 MiB/s wr, 1.39k op/s 2024-07-26T11:25:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:45 smithi193 ceph-mon[37473]: pgmap v778: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 60 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 113 KiB/s rd, 227 MiB/s wr, 1.39k op/s 2024-07-26T11:25:47.181 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:46 smithi134 ceph-mon[31966]: pgmap v779: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 60 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 114 KiB/s rd, 249 MiB/s wr, 1.48k op/s 2024-07-26T11:25:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:46 smithi189 ceph-mon[36790]: pgmap v779: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 60 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 114 KiB/s rd, 249 MiB/s wr, 1.48k op/s 2024-07-26T11:25:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:46 smithi193 ceph-mon[37473]: pgmap v779: 129 pgs: 6 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 111 active+clean; 60 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 114 KiB/s rd, 249 MiB/s wr, 1.48k op/s 2024-07-26T11:25:50.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:49 smithi134 ceph-mon[31966]: pgmap v780: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 61 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 136 KiB/s rd, 252 MiB/s wr, 1.44k op/s 2024-07-26T11:25:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:49 smithi189 ceph-mon[36790]: pgmap v780: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 61 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 136 KiB/s rd, 252 MiB/s wr, 1.44k op/s 2024-07-26T11:25:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:49 smithi193 ceph-mon[37473]: pgmap v780: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 61 GiB data, 129 GiB used, 944 GiB / 1.0 TiB avail; 136 KiB/s rd, 252 MiB/s wr, 1.44k op/s 2024-07-26T11:25:51.169 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:50 smithi134 ceph-mon[31966]: scrub summary: idle 2024-07-26T11:25:51.169 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:50 smithi134 ceph-mon[31966]: pgmap v781: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 61 GiB data, 129 GiB used, 943 GiB / 1.0 TiB avail; 89 KiB/s rd, 252 MiB/s wr, 1.40k op/s 2024-07-26T11:25:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:50 smithi189 ceph-mon[36790]: scrub summary: idle 2024-07-26T11:25:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:50 smithi189 ceph-mon[36790]: pgmap v781: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 61 GiB data, 129 GiB used, 943 GiB / 1.0 TiB avail; 89 KiB/s rd, 252 MiB/s wr, 1.40k op/s 2024-07-26T11:25:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:50 smithi193 ceph-mon[37473]: scrub summary: idle 2024-07-26T11:25:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:50 smithi193 ceph-mon[37473]: pgmap v781: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 61 GiB data, 129 GiB used, 943 GiB / 1.0 TiB avail; 89 KiB/s rd, 252 MiB/s wr, 1.40k op/s 2024-07-26T11:25:52.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:51 smithi134 ceph-mon[31966]: scrub summary: idle 2024-07-26T11:25:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:51 smithi189 ceph-mon[36790]: scrub summary: idle 2024-07-26T11:25:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:51 smithi193 ceph-mon[37473]: scrub summary: idle 2024-07-26T11:25:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:52 smithi189 ceph-mon[36790]: pgmap v782: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 115 KiB/s rd, 233 MiB/s wr, 1.26k op/s 2024-07-26T11:25:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:52 smithi193 ceph-mon[37473]: pgmap v782: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 115 KiB/s rd, 233 MiB/s wr, 1.26k op/s 2024-07-26T11:25:53.285 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:52 smithi134 ceph-mon[31966]: pgmap v782: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 115 KiB/s rd, 233 MiB/s wr, 1.26k op/s 2024-07-26T11:25:53.942 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.936+0000 7f75c9400640 1 -- 172.21.15.134:0/2840080345 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75c410a740 msgr2=0x7f75c4115a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.937+0000 7f75c9400640 1 --2- 172.21.15.134:0/2840080345 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75c410a740 0x7f75c4115a00 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009900 tx=0x7f75b402a400 comp rx=0 tx=0).stop 2024-07-26T11:25:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.938+0000 7f75c9400640 1 -- 172.21.15.134:0/2840080345 shutdown_connections 2024-07-26T11:25:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.938+0000 7f75c9400640 1 --2- 172.21.15.134:0/2840080345 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75c4115f40 0x7f75c4118360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.938+0000 7f75c9400640 1 --2- 172.21.15.134:0/2840080345 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75c410a740 0x7f75c4115a00 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.938+0000 7f75c9400640 1 --2- 172.21.15.134:0/2840080345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c4109df0 0x7f75c410a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.938+0000 7f75c9400640 1 -- 172.21.15.134:0/2840080345 >> 172.21.15.134:0/2840080345 conn(0x7f75c4070630 msgr2=0x7f75c4070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.941+0000 7f75c9400640 1 -- 172.21.15.134:0/2840080345 shutdown_connections 2024-07-26T11:25:53.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.941+0000 7f75c9400640 1 -- 172.21.15.134:0/2840080345 wait complete. 2024-07-26T11:25:53.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.944+0000 7f75c9400640 1 Processor -- start 2024-07-26T11:25:53.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.946+0000 7f75c9400640 1 -- start start 2024-07-26T11:25:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.950+0000 7f75c9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c4109df0 0x7f75c41a4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.950+0000 7f75c9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75c410a740 0x7f75c41a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.950+0000 7f75c9400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75c4115f40 0x7f75c41a9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.950+0000 7f75c9400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f75c411baa0 con 0x7f75c4115f40 2024-07-26T11:25:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.950+0000 7f75c9400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f75c411b920 con 0x7f75c4109df0 2024-07-26T11:25:53.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.950+0000 7f75c9400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f75c411bc20 con 0x7f75c410a740 2024-07-26T11:25:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.951+0000 7f75c8a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75c4115f40 0x7f75c41a9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.951+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c4109df0 0x7f75c41a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.951+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c4109df0 0x7f75c41a4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38810/0 (socket says 172.21.15.134:38810) 2024-07-26T11:25:53.955 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.951+0000 7f75c3e00640 1 -- 172.21.15.134:0/4030161437 learned_addr learned my addr 172.21.15.134:0/4030161437 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:53.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.954+0000 7f75c3400640 1 --2- 172.21.15.134:0/4030161437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75c410a740 0x7f75c41a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.955+0000 7f75c8a00640 1 -- 172.21.15.134:0/4030161437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75c410a740 msgr2=0x7f75c41a4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.955+0000 7f75c8a00640 1 --2- 172.21.15.134:0/4030161437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75c410a740 0x7f75c41a4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.955+0000 7f75c8a00640 1 -- 172.21.15.134:0/4030161437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c4109df0 msgr2=0x7f75c41a4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.955+0000 7f75c8a00640 1 --2- 172.21.15.134:0/4030161437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c4109df0 0x7f75c41a4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.955+0000 7f75c8a00640 1 -- 172.21.15.134:0/4030161437 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c41aa520 con 0x7f75c4115f40 2024-07-26T11:25:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.955+0000 7f75c8a00640 1 --2- 172.21.15.134:0/4030161437 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75c4115f40 0x7f75c41a9eb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f75b000c970 tx=0x7f75b000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.956+0000 7f75c0c00640 1 -- 172.21.15.134:0/4030161437 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b0007bb0 con 0x7f75c4115f40 2024-07-26T11:25:53.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.956+0000 7f75c0c00640 1 -- 172.21.15.134:0/4030161437 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75b0007d50 con 0x7f75c4115f40 2024-07-26T11:25:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.956+0000 7f75c0c00640 1 -- 172.21.15.134:0/4030161437 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b0018640 con 0x7f75c4115f40 2024-07-26T11:25:53.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.957+0000 7f75c9400640 1 -- 172.21.15.134:0/4030161437 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c41aa810 con 0x7f75c4115f40 2024-07-26T11:25:53.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.957+0000 7f75c9400640 1 -- 172.21.15.134:0/4030161437 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c41aace0 con 0x7f75c4115f40 2024-07-26T11:25:53.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.967+0000 7f75c0c00640 1 -- 172.21.15.134:0/4030161437 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f75b0018860 con 0x7f75c4115f40 2024-07-26T11:25:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.970+0000 7f75c0c00640 1 --2- 172.21.15.134:0/4030161437 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75a80b3dc0 0x7f75a80b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:53.976 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.970+0000 7f75c0c00640 1 -- 172.21.15.134:0/4030161437 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9523+0+0 (secure 0 0 0) 0x7f75b00e52d0 con 0x7f75c4115f40 2024-07-26T11:25:53.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.974+0000 7f75c3e00640 1 --2- 172.21.15.134:0/4030161437 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75a80b3dc0 0x7f75a80b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.976+0000 7f75d198e740 1 Processor -- start 2024-07-26T11:25:53.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.976+0000 7f75c3e00640 1 --2- 172.21.15.134:0/4030161437 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75a80b3dc0 0x7f75a80b62d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f75bc007410 tx=0x7f75bc0073a0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.978+0000 7f75d198e740 1 -- start start 2024-07-26T11:25:53.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.978+0000 7f75d198e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf106c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.979+0000 7f75d198e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced3a2f0 0x55d8cf107160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:53.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.979+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf106c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:53.982 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.979+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf106c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38812/0 (socket says 172.21.15.134:38812) 2024-07-26T11:25:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.979+0000 7f75c3e00640 1 -- 172.21.15.134:0/1636556913 learned_addr learned my addr 172.21.15.134:0/1636556913 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.979+0000 7f75d198e740 1 --2- 172.21.15.134:0/1636556913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced424d0 0x55d8cf1076a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.979+0000 7f75d198e740 1 -- 172.21.15.134:0/1636556913 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d8cf07b270 con 0x55d8ced3a2f0 2024-07-26T11:25:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.979+0000 7f75d198e740 1 -- 172.21.15.134:0/1636556913 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d8cf06a770 con 0x55d8ced631b0 2024-07-26T11:25:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.980+0000 7f75d198e740 1 -- 172.21.15.134:0/1636556913 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d8cf06e7b0 con 0x55d8ced424d0 2024-07-26T11:25:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.980+0000 7f75c8a00640 1 --2- 172.21.15.134:0/1636556913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced424d0 0x55d8cf1076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.980+0000 7f75c8a00640 1 -- 172.21.15.134:0/1636556913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 msgr2=0x55d8cf106c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.980+0000 7f75c8a00640 1 --2- 172.21.15.134:0/1636556913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf106c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:53.983 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.980+0000 7f75c8a00640 1 -- 172.21.15.134:0/1636556913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced3a2f0 msgr2=0x55d8cf107160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.980+0000 7f75c8a00640 1 --2- 172.21.15.134:0/1636556913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced3a2f0 0x55d8cf107160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.980+0000 7f75c8a00640 1 -- 172.21.15.134:0/1636556913 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8cedce790 con 0x55d8ced424d0 2024-07-26T11:25:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.981+0000 7f75c8a00640 1 --2- 172.21.15.134:0/1636556913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced424d0 0x55d8cf1076a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f75b00e2600 tx=0x7f75b00ab6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5aa355eac82f7a80 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.981+0000 7f75c3e00640 1 --2- 172.21.15.134:0/1636556913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf106c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:25:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.981+0000 7f75b9200640 1 -- 172.21.15.134:0/1636556913 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b00b00f0 con 0x55d8ced424d0 2024-07-26T11:25:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.981+0000 7f75b9200640 1 -- 172.21.15.134:0/1636556913 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75b00b4ba0 con 0x55d8ced424d0 2024-07-26T11:25:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.981+0000 7f75b9200640 1 -- 172.21.15.134:0/1636556913 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b0091df0 con 0x55d8ced424d0 2024-07-26T11:25:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.983+0000 7f75d198e740 1 -- 172.21.15.134:0/1636556913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced424d0 msgr2=0x55d8cf1076a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:53.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.983+0000 7f75d198e740 1 --2- 172.21.15.134:0/1636556913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced424d0 0x55d8cf1076a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f75b00e2600 tx=0x7f75b00ab6c0 comp rx=0 tx=0).stop 2024-07-26T11:25:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.983+0000 7f75d198e740 1 -- 172.21.15.134:0/1636556913 shutdown_connections 2024-07-26T11:25:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.983+0000 7f75d198e740 1 --2- 172.21.15.134:0/1636556913 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced424d0 0x55d8cf1076a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.983+0000 7f75d198e740 1 --2- 172.21.15.134:0/1636556913 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced3a2f0 0x55d8cf107160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:53.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.984+0000 7f75d198e740 1 --2- 172.21.15.134:0/1636556913 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf106c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.984+0000 7f75d198e740 1 -- 172.21.15.134:0/1636556913 >> 172.21.15.134:0/1636556913 conn(0x55d8ced57300 msgr2=0x55d8cf104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.984+0000 7f75d198e740 1 -- 172.21.15.134:0/1636556913 shutdown_connections 2024-07-26T11:25:53.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.984+0000 7f75d198e740 1 -- 172.21.15.134:0/1636556913 wait complete. 2024-07-26T11:25:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.986+0000 7f75d198e740 1 -- start start 2024-07-26T11:25:53.988 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.987+0000 7f75d198e740 1 Processor -- start 2024-07-26T11:25:53.990 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.989+0000 7f75d198e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced3a2f0 0x55d8cf1128a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:53.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.989+0000 7f75d198e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced424d0 0x55d8cf112e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:53.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.992+0000 7f75d198e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced631b0 0x55d8cf11e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.992+0000 7f75d198e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55d8cefa4ab0 con 0x55d8ced631b0 2024-07-26T11:25:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.992+0000 7f75d198e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d8cf007d50 con 0x55d8ced3a2f0 2024-07-26T11:25:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.992+0000 7f75d198e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55d8cf1153a0 con 0x55d8ced424d0 2024-07-26T11:25:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.992+0000 7f75d198e740 10 client.0 _ll_register_callbacks cb 0x55d8ced7aa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:25:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.993+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced3a2f0 0x55d8cf1128a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.993+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced3a2f0 0x55d8cf1128a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38814/0 (socket says 172.21.15.134:38814) 2024-07-26T11:25:53.995 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.993+0000 7f75c3e00640 1 -- 172.21.15.134:0/2060294635 learned_addr learned my addr 172.21.15.134:0/2060294635 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.993+0000 7f75c8a00640 1 --2- 172.21.15.134:0/2060294635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced631b0 0x55d8cf11e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.994+0000 7f75c8a00640 1 -- 172.21.15.134:0/2060294635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced424d0 msgr2=0x55d8cf112e80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:25:53.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.994+0000 7f75c8a00640 1 --2- 172.21.15.134:0/2060294635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced424d0 0x55d8cf112e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.994+0000 7f75c8a00640 1 -- 172.21.15.134:0/2060294635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced3a2f0 msgr2=0x55d8cf1128a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.994+0000 7f75c8a00640 1 --2- 172.21.15.134:0/2060294635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced3a2f0 0x55d8cf1128a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.994+0000 7f75c8a00640 1 -- 172.21.15.134:0/2060294635 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8cee47620 con 0x55d8ced631b0 2024-07-26T11:25:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.994+0000 7f75c8a00640 1 --2- 172.21.15.134:0/2060294635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced631b0 0x55d8cf11e690 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f75b00e2630 tx=0x7f75b000a470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa2d88827e06f8f9 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:53.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.994+0000 7f75ad600640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:25:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.998+0000 7f75ad600640 1 -- 172.21.15.134:0/2060294635 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b00ca020 con 0x55d8ced631b0 2024-07-26T11:25:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:53.999+0000 7f75ad600640 1 -- 172.21.15.134:0/2060294635 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9523+0+0 (secure 0 0 0) 0x7f75b0093110 con 0x55d8ced631b0 2024-07-26T11:25:54.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.000+0000 7f75d198e740 1 -- 172.21.15.134:0/2060294635 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75b00b4ba0 con 0x55d8ced631b0 2024-07-26T11:25:54.003 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.001+0000 7f75ad600640 1 -- 172.21.15.134:0/2060294635 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f75b00a7ed0 con 0x55d8ced631b0 2024-07-26T11:25:54.003 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.001+0000 7f75d198e740 10 client.16173 fetch_fsmap learned FSMap version 32 2024-07-26T11:25:54.003 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.001+0000 7f75d198e740 1 -- 172.21.15.134:0/2060294635 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d8cf129bf0 con 0x55d8ced631b0 2024-07-26T11:25:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.002+0000 7f75ad600640 1 -- 172.21.15.134:0/2060294635 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75b00b4ba0 con 0x55d8ced631b0 2024-07-26T11:25:54.004 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.002+0000 7f75ad600640 1 -- 172.21.15.134:0/2060294635 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f75b0020030 con 0x55d8ced631b0 2024-07-26T11:25:54.007 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.006+0000 7f75d198e740 10 client.16173 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:25:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.006+0000 7f75d198e740 10 client.16173 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:25:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.006+0000 7f75d198e740 20 client.16173 populate_metadata read hostname 'smithi134' 2024-07-26T11:25:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.006+0000 7f75d198e740 1 --2- 172.21.15.134:0/2060294635 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d8ced21d80 0x55d8cf12d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.006+0000 7f75d198e740 4 client.16173 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:25:54.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.006+0000 7f75d198e740 1 -- 172.21.15.134:0/2060294635 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d8cf07d1c0 con 0x55d8ced21d80 2024-07-26T11:25:54.009 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.008+0000 7f75c3e00640 1 --2- 172.21.15.134:0/2060294635 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d8ced21d80 0x55d8cf12d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.012+0000 7f75ad600640 10 client.16173 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:54.014 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.012+0000 7f75c3e00640 1 --2- 172.21.15.134:0/2060294635 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d8ced21d80 0x55d8cf12d310 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3e022079037688c server_cookie=2839124f54de6464 in_seq=0 out_seq=0 2024-07-26T11:25:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.013+0000 7f75ad600640 1 -- 172.21.15.134:0/2060294635 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55d8cf07d1c0 con 0x55d8ced21d80 2024-07-26T11:25:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.013+0000 7f75ad600640 10 client.16173 handle_command_reply: tid=0 2024-07-26T11:25:54.015 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.014+0000 7f75d198e740 1 client.16173 shutdown 2024-07-26T11:25:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.014+0000 7f75d198e740 1 -- 172.21.15.134:0/2060294635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced631b0 msgr2=0x55d8cf11e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.014+0000 7f75d198e740 1 --2- 172.21.15.134:0/2060294635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced631b0 0x55d8cf11e690 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f75b00e2630 tx=0x7f75b000a470 comp rx=0 tx=0).stop 2024-07-26T11:25:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.015+0000 7f75d198e740 1 -- 172.21.15.134:0/2060294635 shutdown_connections 2024-07-26T11:25:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.015+0000 7f75d198e740 1 --2- 172.21.15.134:0/2060294635 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d8ced21d80 0x55d8cf12d310 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.015+0000 7f75d198e740 1 --2- 172.21.15.134:0/2060294635 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced424d0 0x55d8cf112e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.015+0000 7f75d198e740 1 --2- 172.21.15.134:0/2060294635 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced631b0 0x55d8cf11e690 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.015+0000 7f75d198e740 1 --2- 172.21.15.134:0/2060294635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced3a2f0 0x55d8cf1128a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.015+0000 7f75d198e740 1 -- 172.21.15.134:0/2060294635 >> 172.21.15.134:0/2060294635 conn(0x55d8ced57300 msgr2=0x55d8cf107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:54.017 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.015+0000 7f75ad600640 0 client.16173 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.016+0000 7f75d198e740 1 -- 172.21.15.134:0/2060294635 shutdown_connections 2024-07-26T11:25:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.016+0000 7f75d198e740 1 -- 172.21.15.134:0/2060294635 wait complete. 2024-07-26T11:25:54.018 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.016+0000 7f75d198e740 20 client.16173 trim_cache size 0 max 16384 2024-07-26T11:25:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.060+0000 7f75d198e740 1 Processor -- start 2024-07-26T11:25:54.062 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.060+0000 7f75d198e740 1 -- start start 2024-07-26T11:25:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.061+0000 7f75d198e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced3a2f0 0x55d8cf10c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.061+0000 7f75d198e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced424d0 0x55d8cf11e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:54.063 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.061+0000 7f75d198e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf111b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.062+0000 7f75d198e740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f75c41b6e40 con 0x55d8ced424d0 2024-07-26T11:25:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.062+0000 7f75d198e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f75c41b8980 con 0x55d8ced631b0 2024-07-26T11:25:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.062+0000 7f75d198e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f75c41b7510 con 0x55d8ced3a2f0 2024-07-26T11:25:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.062+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced3a2f0 0x55d8cf10c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.062+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced3a2f0 0x55d8cf10c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:46766/0 (socket says 172.21.15.134:46766) 2024-07-26T11:25:54.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.062+0000 7f75c8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf111b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:54.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.062+0000 7f75c8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf111b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38828/0 (socket says 172.21.15.134:38828) 2024-07-26T11:25:54.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.062+0000 7f75c8a00640 1 -- 172.21.15.134:0/4262030788 learned_addr learned my addr 172.21.15.134:0/4262030788 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:54.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.063+0000 7f75c8a00640 1 -- 172.21.15.134:0/4262030788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced3a2f0 msgr2=0x55d8cf10c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:54.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.063+0000 7f75c8a00640 1 --2- 172.21.15.134:0/4262030788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced3a2f0 0x55d8cf10c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.063+0000 7f75c8a00640 1 -- 172.21.15.134:0/4262030788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced424d0 msgr2=0x55d8cf11e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:54.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.063+0000 7f75c8a00640 1 --2- 172.21.15.134:0/4262030788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced424d0 0x55d8cf11e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.063+0000 7f75c8a00640 1 -- 172.21.15.134:0/4262030788 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d8cf109150 con 0x55d8ced631b0 2024-07-26T11:25:54.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.063+0000 7f75c8a00640 1 --2- 172.21.15.134:0/4262030788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf111b60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f75b00e55f0 tx=0x7f75b000fca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c83f44ed42929166 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:54.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.064+0000 7f75afe00640 1 -- 172.21.15.134:0/4262030788 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b00b9070 con 0x55d8ced631b0 2024-07-26T11:25:54.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.064+0000 7f75afe00640 1 -- 172.21.15.134:0/4262030788 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75b00b4ba0 con 0x55d8ced631b0 2024-07-26T11:25:54.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.064+0000 7f75afe00640 1 -- 172.21.15.134:0/4262030788 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b00ca070 con 0x55d8ced631b0 2024-07-26T11:25:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.065+0000 7f75d198e740 1 -- 172.21.15.134:0/4262030788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 msgr2=0x55d8cf111b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.065+0000 7f75d198e740 1 --2- 172.21.15.134:0/4262030788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf111b60 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f75b00e55f0 tx=0x7f75b000fca0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.066+0000 7f75d198e740 1 -- 172.21.15.134:0/4262030788 shutdown_connections 2024-07-26T11:25:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.066+0000 7f75d198e740 1 --2- 172.21.15.134:0/4262030788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf111b60 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.066+0000 7f75d198e740 1 --2- 172.21.15.134:0/4262030788 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced424d0 0x55d8cf11e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.066+0000 7f75d198e740 1 --2- 172.21.15.134:0/4262030788 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced3a2f0 0x55d8cf10c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.066+0000 7f75d198e740 1 -- 172.21.15.134:0/4262030788 >> 172.21.15.134:0/4262030788 conn(0x55d8ced57300 msgr2=0x55d8cf10bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.071+0000 7f75d198e740 1 -- 172.21.15.134:0/4262030788 shutdown_connections 2024-07-26T11:25:54.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.071+0000 7f75d198e740 1 -- 172.21.15.134:0/4262030788 wait complete. 2024-07-26T11:25:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.072+0000 7f75d198e740 1 -- start start 2024-07-26T11:25:54.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.072+0000 7f75d198e740 1 Processor -- start 2024-07-26T11:25:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.073+0000 7f75d198e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced3a2f0 0x55d8cf115640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.074+0000 7f75d198e740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced424d0 0x55d8cf115b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.074+0000 7f75d198e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf1160c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.074+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced3a2f0 0x55d8cf115640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.074+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced3a2f0 0x55d8cf115640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:46770/0 (socket says 172.21.15.134:46770) 2024-07-26T11:25:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.075+0000 7f75c3e00640 1 -- 172.21.15.134:0/3087676733 learned_addr learned my addr 172.21.15.134:0/3087676733 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.075+0000 7f75c3400640 1 --2- 172.21.15.134:0/3087676733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced424d0 0x55d8cf115b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.075+0000 7f75c8a00640 1 --2- 172.21.15.134:0/3087676733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf1160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.075+0000 7f75d198e740 1 -- 172.21.15.134:0/3087676733 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f75c41b82b0 con 0x55d8ced424d0 2024-07-26T11:25:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.075+0000 7f75d198e740 1 -- 172.21.15.134:0/3087676733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f75c41b7be0 con 0x55d8ced631b0 2024-07-26T11:25:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.075+0000 7f75d198e740 1 -- 172.21.15.134:0/3087676733 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f75c41b4e40 con 0x55d8ced3a2f0 2024-07-26T11:25:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.075+0000 7f75d198e740 10 client.0 _ll_register_callbacks cb 0x7f75b00a7280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:25:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.076+0000 7f75c8a00640 1 -- 172.21.15.134:0/3087676733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced3a2f0 msgr2=0x55d8cf115640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.076+0000 7f75c8a00640 1 --2- 172.21.15.134:0/3087676733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced3a2f0 0x55d8cf115640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.076+0000 7f75c8a00640 1 -- 172.21.15.134:0/3087676733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced424d0 msgr2=0x55d8cf115b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.076+0000 7f75c8a00640 1 --2- 172.21.15.134:0/3087676733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced424d0 0x55d8cf115b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.076+0000 7f75c8a00640 1 -- 172.21.15.134:0/3087676733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d8cf193c50 con 0x55d8ced631b0 2024-07-26T11:25:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.077+0000 7f75c8a00640 1 --2- 172.21.15.134:0/3087676733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf1160c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f75b00e55f0 tx=0x7f75b00abc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a89dbd3d7571e66d server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.077+0000 7f75ad600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:25:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.078+0000 7f75ad600640 1 -- 172.21.15.134:0/3087676733 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b00b9070 con 0x55d8ced631b0 2024-07-26T11:25:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.078+0000 7f75ad600640 1 -- 172.21.15.134:0/3087676733 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9523+0+0 (secure 0 0 0) 0x7f75b0091b90 con 0x55d8ced631b0 2024-07-26T11:25:54.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.078+0000 7f75d198e740 1 -- 172.21.15.134:0/3087676733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f75b00b4ba0 con 0x55d8ced631b0 2024-07-26T11:25:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.081+0000 7f75ad600640 1 -- 172.21.15.134:0/3087676733 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f75b00ca070 con 0x55d8ced631b0 2024-07-26T11:25:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.081+0000 7f75d198e740 10 client.26816 fetch_fsmap learned FSMap version 32 2024-07-26T11:25:54.083 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.081+0000 7f75d198e740 1 -- 172.21.15.134:0/3087676733 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55d8cf194ee0 con 0x55d8ced631b0 2024-07-26T11:25:54.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.082+0000 7f75ad600640 1 -- 172.21.15.134:0/3087676733 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f75b00b4ba0 con 0x55d8ced631b0 2024-07-26T11:25:54.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.082+0000 7f75ad600640 1 -- 172.21.15.134:0/3087676733 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f75b0020050 con 0x55d8ced631b0 2024-07-26T11:25:54.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.082+0000 7f75d198e740 10 client.26816 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:25:54.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.082+0000 7f75d198e740 10 client.26816 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:25:54.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.082+0000 7f75d198e740 20 client.26816 populate_metadata read hostname 'smithi134' 2024-07-26T11:25:54.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.082+0000 7f75d198e740 1 --2- 172.21.15.134:0/3087676733 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d8cecf71c0 0x55d8cf197410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:54.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.082+0000 7f75d198e740 4 client.26816 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:25:54.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.082+0000 7f75d198e740 1 -- 172.21.15.134:0/3087676733 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d8cf078c60 con 0x55d8cecf71c0 2024-07-26T11:25:54.088 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.086+0000 7f75c3e00640 1 --2- 172.21.15.134:0/3087676733 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d8cecf71c0 0x55d8cf197410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:54.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.087+0000 7f75c3e00640 1 --2- 172.21.15.134:0/3087676733 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d8cecf71c0 0x55d8cf197410 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d50bd2e6a25a5526 server_cookie=a642cb703d6bd61b in_seq=0 out_seq=0 2024-07-26T11:25:54.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.088+0000 7f75ad600640 10 client.26816 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:54.933 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.931+0000 7f75ad600640 1 -- 172.21.15.134:0/3087676733 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d8cf078c60 con 0x55d8cecf71c0 2024-07-26T11:25:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.931+0000 7f75ad600640 10 client.26816 handle_command_reply: tid=0 2024-07-26T11:25:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.931+0000 7f75d198e740 1 client.26816 shutdown 2024-07-26T11:25:54.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.933+0000 7f75d198e740 1 -- 172.21.15.134:0/3087676733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 msgr2=0x55d8cf1160c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:54.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.933+0000 7f75d198e740 1 --2- 172.21.15.134:0/3087676733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf1160c0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f75b00e55f0 tx=0x7f75b00abc70 comp rx=0 tx=0).stop 2024-07-26T11:25:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.938+0000 7f75d198e740 1 -- 172.21.15.134:0/3087676733 shutdown_connections 2024-07-26T11:25:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.938+0000 7f75d198e740 1 --2- 172.21.15.134:0/3087676733 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55d8cecf71c0 0x55d8cf197410 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.938+0000 7f75d198e740 1 --2- 172.21.15.134:0/3087676733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8ced631b0 0x55d8cf1160c0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.938+0000 7f75d198e740 1 --2- 172.21.15.134:0/3087676733 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55d8ced424d0 0x55d8cf115b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.938+0000 7f75d198e740 1 --2- 172.21.15.134:0/3087676733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d8ced3a2f0 0x55d8cf115640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.938+0000 7f75ad600640 0 client.26816 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.938+0000 7f75d198e740 1 -- 172.21.15.134:0/3087676733 >> 172.21.15.134:0/3087676733 conn(0x55d8ced57300 msgr2=0x55d8cf111b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:54.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.941+0000 7f75d198e740 1 -- 172.21.15.134:0/3087676733 shutdown_connections 2024-07-26T11:25:54.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.941+0000 7f75d198e740 1 -- 172.21.15.134:0/3087676733 wait complete. 2024-07-26T11:25:54.943 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.941+0000 7f75d198e740 20 client.26816 trim_cache size 0 max 16384 2024-07-26T11:25:54.944 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:25:54.944 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:25:54.944 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:25:54.944 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:25:54.949 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.947+0000 7f75c9400640 1 -- 172.21.15.134:0/4030161437 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75a80b3dc0 msgr2=0x7f75a80b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:54.949 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.947+0000 7f75c9400640 1 --2- 172.21.15.134:0/4030161437 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75a80b3dc0 0x7f75a80b62d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f75bc007410 tx=0x7f75bc0073a0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.949 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.947+0000 7f75c9400640 1 -- 172.21.15.134:0/4030161437 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75c4115f40 msgr2=0x7f75c41a9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:54.949 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.947+0000 7f75c9400640 1 --2- 172.21.15.134:0/4030161437 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75c4115f40 0x7f75c41a9eb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f75b000c970 tx=0x7f75b000ce30 comp rx=0 tx=0).stop 2024-07-26T11:25:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.947+0000 7f75c9400640 1 -- 172.21.15.134:0/4030161437 shutdown_connections 2024-07-26T11:25:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.947+0000 7f75c9400640 1 --2- 172.21.15.134:0/4030161437 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f75a80b3dc0 0x7f75a80b62d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.947+0000 7f75c9400640 1 --2- 172.21.15.134:0/4030161437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f75c410a740 0x7f75c41a4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.947+0000 7f75c9400640 1 --2- 172.21.15.134:0/4030161437 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f75c4115f40 0x7f75c41a9eb0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.947+0000 7f75c9400640 1 --2- 172.21.15.134:0/4030161437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75c4109df0 0x7f75c41a4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.947+0000 7f75c9400640 1 -- 172.21.15.134:0/4030161437 >> 172.21.15.134:0/4030161437 conn(0x7f75c4070630 msgr2=0x7f75c4073590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.947+0000 7f75c9400640 1 -- 172.21.15.134:0/4030161437 shutdown_connections 2024-07-26T11:25:54.950 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:25:54.947+0000 7f75c9400640 1 -- 172.21.15.134:0/4030161437 wait complete. 2024-07-26T11:25:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:54 smithi134 ceph-mon[31966]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T11:25:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:54 smithi134 ceph-mon[31966]: pgmap v784: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 59 KiB/s rd, 272 MiB/s wr, 1.50k op/s 2024-07-26T11:25:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:54 smithi189 ceph-mon[36790]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T11:25:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:54 smithi189 ceph-mon[36790]: pgmap v784: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 59 KiB/s rd, 272 MiB/s wr, 1.50k op/s 2024-07-26T11:25:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:54 smithi193 ceph-mon[37473]: osdmap e96: 12 total, 12 up, 12 in 2024-07-26T11:25:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:54 smithi193 ceph-mon[37473]: pgmap v784: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 59 KiB/s rd, 272 MiB/s wr, 1.50k op/s 2024-07-26T11:25:55.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.733+0000 7f8066a00640 1 -- 172.21.15.134:0/2297722422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8060072e90 msgr2=0x7f80600736b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:55.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.733+0000 7f8066a00640 1 --2- 172.21.15.134:0/2297722422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8060072e90 0x7f80600736b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8058009ca0 tx=0x7f805802a8c0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.734+0000 7f8066a00640 1 -- 172.21.15.134:0/2297722422 shutdown_connections 2024-07-26T11:25:55.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.734+0000 7f8066a00640 1 --2- 172.21.15.134:0/2297722422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8060072540 0x7f8060072950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.734+0000 7f8066a00640 1 --2- 172.21.15.134:0/2297722422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8060072e90 0x7f80600736b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.734+0000 7f8066a00640 1 --2- 172.21.15.134:0/2297722422 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f806006cc00 0x7f806006d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.734+0000 7f8066a00640 1 -- 172.21.15.134:0/2297722422 >> 172.21.15.134:0/2297722422 conn(0x7f8060079160 msgr2=0x7f8060079570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:55.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.734+0000 7f8066a00640 1 -- 172.21.15.134:0/2297722422 shutdown_connections 2024-07-26T11:25:55.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.735+0000 7f8066a00640 1 -- 172.21.15.134:0/2297722422 wait complete. 2024-07-26T11:25:55.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.739+0000 7f8066a00640 1 Processor -- start 2024-07-26T11:25:55.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.739+0000 7f8066a00640 1 -- start start 2024-07-26T11:25:55.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.741+0000 7f8066a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f806006cc00 0x7f80601a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.742+0000 7f8066a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8060072540 0x7f80601a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.745+0000 7f8066a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8060072e90 0x7f80601a6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.745+0000 7f8066a00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f806011acb0 con 0x7f806006cc00 2024-07-26T11:25:55.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.745+0000 7f8066a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f806011ab30 con 0x7f8060072540 2024-07-26T11:25:55.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.745+0000 7f8066a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f806011ae30 con 0x7f8060072e90 2024-07-26T11:25:55.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.746+0000 7f805fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f806006cc00 0x7f80601a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:55.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.746+0000 7f805fe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f806006cc00 0x7f80601a1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:42206/0 (socket says 172.21.15.134:42206) 2024-07-26T11:25:55.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.747+0000 7f805f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8060072540 0x7f80601a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:55.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.747+0000 7f805f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8060072540 0x7f80601a1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38848/0 (socket says 172.21.15.134:38848) 2024-07-26T11:25:55.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.747+0000 7f805f400640 1 -- 172.21.15.134:0/883877634 learned_addr learned my addr 172.21.15.134:0/883877634 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:55.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.747+0000 7f8064a00640 1 --2- 172.21.15.134:0/883877634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8060072e90 0x7f80601a6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:55.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.751+0000 7f805f400640 1 -- 172.21.15.134:0/883877634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8060072e90 msgr2=0x7f80601a6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:55.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.751+0000 7f805f400640 1 --2- 172.21.15.134:0/883877634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8060072e90 0x7f80601a6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.751+0000 7f805f400640 1 -- 172.21.15.134:0/883877634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f806006cc00 msgr2=0x7f80601a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:55.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.751+0000 7f805f400640 1 --2- 172.21.15.134:0/883877634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f806006cc00 0x7f80601a1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.751+0000 7f805f400640 1 -- 172.21.15.134:0/883877634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80601a75b0 con 0x7f8060072540 2024-07-26T11:25:55.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.751+0000 7f8064a00640 1 --2- 172.21.15.134:0/883877634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8060072e90 0x7f80601a6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:25:55.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.752+0000 7f805f400640 1 --2- 172.21.15.134:0/883877634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8060072540 0x7f80601a1be0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f804c00e9c0 tx=0x7f804c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:55.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.752+0000 7f805cc00640 1 -- 172.21.15.134:0/883877634 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804c00ccc0 con 0x7f8060072540 2024-07-26T11:25:55.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.752+0000 7f805cc00640 1 -- 172.21.15.134:0/883877634 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f804c0054d0 con 0x7f8060072540 2024-07-26T11:25:55.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.752+0000 7f8066a00640 1 -- 172.21.15.134:0/883877634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80601a78a0 con 0x7f8060072540 2024-07-26T11:25:55.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.753+0000 7f805cc00640 1 -- 172.21.15.134:0/883877634 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804c010430 con 0x7f8060072540 2024-07-26T11:25:55.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.753+0000 7f8066a00640 1 -- 172.21.15.134:0/883877634 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80601ae000 con 0x7f8060072540 2024-07-26T11:25:55.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.760+0000 7f805cc00640 1 -- 172.21.15.134:0/883877634 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f804c0105d0 con 0x7f8060072540 2024-07-26T11:25:55.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.762+0000 7f805cc00640 1 --2- 172.21.15.134:0/883877634 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f80440b3dc0 0x7f80440b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.762+0000 7f805cc00640 1 -- 172.21.15.134:0/883877634 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9523+0+0 (secure 0 0 0) 0x7f804c0e5b40 con 0x7f8060072540 2024-07-26T11:25:55.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.763+0000 7f805fe00640 1 --2- 172.21.15.134:0/883877634 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f80440b3dc0 0x7f80440b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:55.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.790+0000 7f805fe00640 1 --2- 172.21.15.134:0/883877634 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f80440b3dc0 0x7f80440b62d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f80600735d0 tx=0x7f8050008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:55.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.792+0000 7f806d5dc740 1 Processor -- start 2024-07-26T11:25:55.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.792+0000 7f806d5dc740 1 -- start start 2024-07-26T11:25:55.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.794+0000 7f806d5dc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f3322070 0x5581f36c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.794+0000 7f806d5dc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f32f91b0 0x5581f36c7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.794+0000 7f806d5dc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3301390 0x5581f36c7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.794+0000 7f806d5dc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5581f363db90 con 0x5581f3322070 2024-07-26T11:25:55.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.794+0000 7f806d5dc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5581f362d090 con 0x5581f3301390 2024-07-26T11:25:55.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.794+0000 7f806d5dc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5581f36310d0 con 0x5581f32f91b0 2024-07-26T11:25:55.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.795+0000 7f8064a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3301390 0x5581f36c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:55.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.795+0000 7f805f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f32f91b0 0x5581f36c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:55.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.795+0000 7f805f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f32f91b0 0x5581f36c7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:46788/0 (socket says 172.21.15.134:46788) 2024-07-26T11:25:55.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.795+0000 7f805f400640 1 -- 172.21.15.134:0/3064359543 learned_addr learned my addr 172.21.15.134:0/3064359543 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:55.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.796+0000 7f805f400640 1 -- 172.21.15.134:0/3064359543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3301390 msgr2=0x5581f36c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:55.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.796+0000 7f805f400640 1 --2- 172.21.15.134:0/3064359543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3301390 0x5581f36c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.796+0000 7f805f400640 1 -- 172.21.15.134:0/3064359543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f3322070 msgr2=0x5581f36c6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:55.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.796+0000 7f805f400640 1 --2- 172.21.15.134:0/3064359543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f3322070 0x5581f36c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.796+0000 7f805f400640 1 -- 172.21.15.134:0/3064359543 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581f338d650 con 0x5581f32f91b0 2024-07-26T11:25:55.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.797+0000 7f805f400640 1 --2- 172.21.15.134:0/3064359543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f32f91b0 0x5581f36c7330 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f804c0a83e0 tx=0x7f804c0abd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbb59f94332bec29 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:55.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.800+0000 7f8055200640 1 -- 172.21.15.134:0/3064359543 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804c0ac360 con 0x5581f32f91b0 2024-07-26T11:25:55.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.800+0000 7f8055200640 1 -- 172.21.15.134:0/3064359543 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f804c0b0960 con 0x5581f32f91b0 2024-07-26T11:25:55.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.800+0000 7f8055200640 1 -- 172.21.15.134:0/3064359543 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804c0ab5c0 con 0x5581f32f91b0 2024-07-26T11:25:55.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.801+0000 7f806d5dc740 1 -- 172.21.15.134:0/3064359543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f32f91b0 msgr2=0x5581f36c7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:55.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.801+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3064359543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f32f91b0 0x5581f36c7330 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f804c0a83e0 tx=0x7f804c0abd70 comp rx=0 tx=0).stop 2024-07-26T11:25:55.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.801+0000 7f806d5dc740 1 -- 172.21.15.134:0/3064359543 shutdown_connections 2024-07-26T11:25:55.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.801+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3064359543 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f32f91b0 0x5581f36c7330 secure :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f804c0a83e0 tx=0x7f804c0abd70 comp rx=0 tx=0).stop 2024-07-26T11:25:55.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.801+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3064359543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3301390 0x5581f36c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.801+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3064359543 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f3322070 0x5581f36c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.801+0000 7f806d5dc740 1 -- 172.21.15.134:0/3064359543 >> 172.21.15.134:0/3064359543 conn(0x5581f3316460 msgr2=0x5581f36c4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:55.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.806+0000 7f806d5dc740 1 -- 172.21.15.134:0/3064359543 shutdown_connections 2024-07-26T11:25:55.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.806+0000 7f806d5dc740 1 -- 172.21.15.134:0/3064359543 wait complete. 2024-07-26T11:25:55.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.807+0000 7f806d5dc740 1 -- start start 2024-07-26T11:25:55.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.807+0000 7f806d5dc740 1 Processor -- start 2024-07-26T11:25:55.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.808+0000 7f806d5dc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f3301390 0x5581f36db030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.808+0000 7f806d5dc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3322070 0x5581f36d4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.808+0000 7f806d5dc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f32e09f0 0x5581f36d74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.808+0000 7f805f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3322070 0x5581f36d4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:55.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.809+0000 7f805f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3322070 0x5581f36d4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:38874/0 (socket says 172.21.15.134:38874) 2024-07-26T11:25:55.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.809+0000 7f805f400640 1 -- 172.21.15.134:0/3313833150 learned_addr learned my addr 172.21.15.134:0/3313833150 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:55.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.809+0000 7f806d5dc740 1 -- 172.21.15.134:0/3313833150 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5581f3490700 con 0x5581f32e09f0 2024-07-26T11:25:55.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.809+0000 7f806d5dc740 1 -- 172.21.15.134:0/3313833150 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5581f35c9660 con 0x5581f3322070 2024-07-26T11:25:55.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.809+0000 7f806d5dc740 1 -- 172.21.15.134:0/3313833150 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5581f36d5510 con 0x5581f3301390 2024-07-26T11:25:55.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.809+0000 7f806d5dc740 10 client.0 _ll_register_callbacks cb 0x5581f35d9110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:25:55.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.809+0000 7f805f400640 1 -- 172.21.15.134:0/3313833150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f3301390 msgr2=0x5581f36db030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:25:55.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.809+0000 7f805f400640 1 --2- 172.21.15.134:0/3313833150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f3301390 0x5581f36db030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.809+0000 7f805f400640 1 -- 172.21.15.134:0/3313833150 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f32e09f0 msgr2=0x5581f36d74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:55.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.809+0000 7f805f400640 1 --2- 172.21.15.134:0/3313833150 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f32e09f0 0x5581f36d74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.810+0000 7f805f400640 1 -- 172.21.15.134:0/3313833150 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581f35e0800 con 0x5581f3322070 2024-07-26T11:25:55.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.810+0000 7f805f400640 1 --2- 172.21.15.134:0/3313833150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3322070 0x5581f36d4900 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f804c00c6a0 tx=0x7f804c00efb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64795464a14381db server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:55.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.810+0000 7f8049600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:25:55.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.810+0000 7f8049600640 1 -- 172.21.15.134:0/3313833150 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804c0cc060 con 0x5581f3322070 2024-07-26T11:25:55.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.811+0000 7f8049600640 1 -- 172.21.15.134:0/3313833150 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9523+0+0 (secure 0 0 0) 0x7f804c005020 con 0x5581f3322070 2024-07-26T11:25:55.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.811+0000 7f806d5dc740 1 -- 172.21.15.134:0/3313833150 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f804c0b0960 con 0x5581f3322070 2024-07-26T11:25:55.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.811+0000 7f8049600640 1 -- 172.21.15.134:0/3313833150 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f804c0ac360 con 0x5581f3322070 2024-07-26T11:25:55.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.812+0000 7f806d5dc740 10 client.26828 fetch_fsmap learned FSMap version 32 2024-07-26T11:25:55.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.812+0000 7f806d5dc740 1 -- 172.21.15.134:0/3313833150 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5581f36dd360 con 0x5581f3322070 2024-07-26T11:25:55.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.812+0000 7f8049600640 1 -- 172.21.15.134:0/3313833150 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f804c0b0960 con 0x5581f3322070 2024-07-26T11:25:55.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.814+0000 7f8049600640 1 -- 172.21.15.134:0/3313833150 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f804c0b3ce0 con 0x5581f3322070 2024-07-26T11:25:55.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.814+0000 7f806d5dc740 10 client.26828 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:25:55.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.815+0000 7f806d5dc740 10 client.26828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:25:55.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.815+0000 7f806d5dc740 20 client.26828 populate_metadata read hostname 'smithi134' 2024-07-26T11:25:55.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.815+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3313833150 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5581f32b5da0 0x5581f36e0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.815+0000 7f806d5dc740 4 client.26828 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:25:55.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.816+0000 7f806d5dc740 1 -- 172.21.15.134:0/3313833150 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581f363fae0 con 0x5581f32b5da0 2024-07-26T11:25:55.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.820+0000 7f805fe00640 1 --2- 172.21.15.134:0/3313833150 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5581f32b5da0 0x5581f36e0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:55.823 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.821+0000 7f805fe00640 1 --2- 172.21.15.134:0/3313833150 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5581f32b5da0 0x5581f36e0780 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc00b8b236bc8fba server_cookie=7cdd11eab8143741 in_seq=0 out_seq=0 2024-07-26T11:25:55.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.821+0000 7f8049600640 10 client.26828 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:55.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.824+0000 7f8049600640 1 -- 172.21.15.134:0/3313833150 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5581f363fae0 con 0x5581f32b5da0 2024-07-26T11:25:55.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.824+0000 7f8049600640 10 client.26828 handle_command_reply: tid=0 2024-07-26T11:25:55.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.824+0000 7f806d5dc740 1 client.26828 shutdown 2024-07-26T11:25:55.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.824+0000 7f806d5dc740 1 -- 172.21.15.134:0/3313833150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3322070 msgr2=0x5581f36d4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:55.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.824+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3313833150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3322070 0x5581f36d4900 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f804c00c6a0 tx=0x7f804c00efb0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.827+0000 7f806d5dc740 1 -- 172.21.15.134:0/3313833150 shutdown_connections 2024-07-26T11:25:55.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.827+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3313833150 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5581f32b5da0 0x5581f36e0780 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.827+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3313833150 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f32e09f0 0x5581f36d74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.827+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3313833150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3322070 0x5581f36d4900 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.827+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3313833150 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f3301390 0x5581f36db030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.827+0000 7f806d5dc740 1 -- 172.21.15.134:0/3313833150 >> 172.21.15.134:0/3313833150 conn(0x5581f3316460 msgr2=0x5581f36c9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:55.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.829+0000 7f8049600640 0 client.26828 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:55.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.829+0000 7f806d5dc740 1 -- 172.21.15.134:0/3313833150 shutdown_connections 2024-07-26T11:25:55.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.830+0000 7f806d5dc740 1 -- 172.21.15.134:0/3313833150 wait complete. 2024-07-26T11:25:55.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.830+0000 7f806d5dc740 20 client.26828 trim_cache size 0 max 16384 2024-07-26T11:25:55.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.890+0000 7f806d5dc740 1 Processor -- start 2024-07-26T11:25:55.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.894+0000 7f806d5dc740 1 -- start start 2024-07-26T11:25:55.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.894+0000 7f806d5dc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f32e09f0 0x5581f36cc140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.896 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.894+0000 7f806d5dc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f3301390 0x5581f36cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.894+0000 7f806d5dc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f3322070 0x5581f36d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.894+0000 7f806d5dc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f80601d9180 con 0x5581f3322070 2024-07-26T11:25:55.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.894+0000 7f806d5dc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f80601dacc0 con 0x5581f32e09f0 2024-07-26T11:25:55.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.894+0000 7f806d5dc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f80601d9850 con 0x5581f3301390 2024-07-26T11:25:55.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.895+0000 7f8064a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f3322070 0x5581f36d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:55.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.896+0000 7f805f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f3301390 0x5581f36cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:55.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.896+0000 7f805f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f3301390 0x5581f36cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:46802/0 (socket says 172.21.15.134:46802) 2024-07-26T11:25:55.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.896+0000 7f805f400640 1 -- 172.21.15.134:0/3210748404 learned_addr learned my addr 172.21.15.134:0/3210748404 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:55.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.896+0000 7f805fe00640 1 --2- 172.21.15.134:0/3210748404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f32e09f0 0x5581f36cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:55.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.897+0000 7f805f400640 1 -- 172.21.15.134:0/3210748404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f32e09f0 msgr2=0x5581f36cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:55.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.897+0000 7f805f400640 1 --2- 172.21.15.134:0/3210748404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f32e09f0 0x5581f36cc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.897+0000 7f805f400640 1 -- 172.21.15.134:0/3210748404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f3322070 msgr2=0x5581f36d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:55.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.897+0000 7f805f400640 1 --2- 172.21.15.134:0/3210748404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f3322070 0x5581f36d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.897+0000 7f805f400640 1 -- 172.21.15.134:0/3210748404 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581f36c55d0 con 0x5581f3301390 2024-07-26T11:25:55.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.897+0000 7f805fe00640 1 --2- 172.21.15.134:0/3210748404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f32e09f0 0x5581f36cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:25:55.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.897+0000 7f805f400640 1 --2- 172.21.15.134:0/3210748404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f3301390 0x5581f36cfaf0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f804c0ac2f0 tx=0x7f804c0ac200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2b1aac9d1c42500 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:55.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.898+0000 7f804be00640 1 -- 172.21.15.134:0/3210748404 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804c0cc020 con 0x5581f3301390 2024-07-26T11:25:55.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.898+0000 7f804be00640 1 -- 172.21.15.134:0/3210748404 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f804c0b5410 con 0x5581f3301390 2024-07-26T11:25:55.902 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.898+0000 7f804be00640 1 -- 172.21.15.134:0/3210748404 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804c0cc020 con 0x5581f3301390 2024-07-26T11:25:55.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.899+0000 7f806d5dc740 1 -- 172.21.15.134:0/3210748404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f3301390 msgr2=0x5581f36cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:55.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.899+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3210748404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f3301390 0x5581f36cfaf0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f804c0ac2f0 tx=0x7f804c0ac200 comp rx=0 tx=0).stop 2024-07-26T11:25:55.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.902+0000 7f806d5dc740 1 -- 172.21.15.134:0/3210748404 shutdown_connections 2024-07-26T11:25:55.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.902+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3210748404 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f3301390 0x5581f36cfaf0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.902+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3210748404 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f3322070 0x5581f36d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.902+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3210748404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f32e09f0 0x5581f36cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.902+0000 7f806d5dc740 1 -- 172.21.15.134:0/3210748404 >> 172.21.15.134:0/3210748404 conn(0x5581f3316460 msgr2=0x5581f36ca9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:55.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.906+0000 7f806d5dc740 1 -- 172.21.15.134:0/3210748404 shutdown_connections 2024-07-26T11:25:55.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.908+0000 7f806d5dc740 1 -- 172.21.15.134:0/3210748404 wait complete. 2024-07-26T11:25:55.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.908+0000 7f806d5dc740 1 -- start start 2024-07-26T11:25:55.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.909+0000 7f806d5dc740 1 Processor -- start 2024-07-26T11:25:55.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.912+0000 7f806d5dc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f32e09f0 0x5581f36e4030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.916+0000 7f806d5dc740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f3301390 0x5581f36df4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.916+0000 7f806d5dc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3322070 0x5581f36dfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.916+0000 7f805fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f32e09f0 0x5581f36e4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:55.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.916+0000 7f805f400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f3301390 0x5581f36df4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:55.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.916+0000 7f806d5dc740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f80601da5f0 con 0x5581f3301390 2024-07-26T11:25:55.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.916+0000 7f806d5dc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f80601d9f20 con 0x5581f3322070 2024-07-26T11:25:55.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.916+0000 7f806d5dc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f80601d7160 con 0x5581f32e09f0 2024-07-26T11:25:55.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.917+0000 7f806d5dc740 10 client.0 _ll_register_callbacks cb 0x7f804c0ac130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:25:55.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.917+0000 7f805fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f32e09f0 0x5581f36e4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:46816/0 (socket says 172.21.15.134:46816) 2024-07-26T11:25:55.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.917+0000 7f805fe00640 1 -- 172.21.15.134:0/3678927696 learned_addr learned my addr 172.21.15.134:0/3678927696 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:25:55.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.918+0000 7f805fe00640 1 -- 172.21.15.134:0/3678927696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3322070 msgr2=0x5581f36dfbd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:25:55.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.919+0000 7f8064a00640 1 --2- 172.21.15.134:0/3678927696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3322070 0x5581f36dfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:55.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.921+0000 7f805fe00640 1 --2- 172.21.15.134:0/3678927696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3322070 0x5581f36dfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.921+0000 7f805fe00640 1 -- 172.21.15.134:0/3678927696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f3301390 msgr2=0x5581f36df4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:25:55.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.921+0000 7f805fe00640 1 --2- 172.21.15.134:0/3678927696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f3301390 0x5581f36df4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.921+0000 7f805fe00640 1 -- 172.21.15.134:0/3678927696 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581f374b000 con 0x5581f32e09f0 2024-07-26T11:25:55.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.921+0000 7f805f400640 1 --2- 172.21.15.134:0/3678927696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f3301390 0x5581f36df4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:25:55.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.921+0000 7f8064a00640 1 --2- 172.21.15.134:0/3678927696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3322070 0x5581f36dfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:25:55.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.922+0000 7f805fe00640 1 --2- 172.21.15.134:0/3678927696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f32e09f0 0x5581f36e4030 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f805000d530 tx=0x7f8050009610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39a1caa2737678e3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:25:55.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.922+0000 7f8049600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:25:55.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.922+0000 7f8049600640 1 -- 172.21.15.134:0/3678927696 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80500059c0 con 0x5581f32e09f0 2024-07-26T11:25:55.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.922+0000 7f8049600640 1 -- 172.21.15.134:0/3678927696 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9523+0+0 (secure 0 0 0) 0x7f8050005b60 con 0x5581f32e09f0 2024-07-26T11:25:55.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.923+0000 7f806d5dc740 1 -- 172.21.15.134:0/3678927696 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f804c0b5410 con 0x5581f32e09f0 2024-07-26T11:25:55.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.923+0000 7f8049600640 1 -- 172.21.15.134:0/3678927696 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8050005da0 con 0x5581f32e09f0 2024-07-26T11:25:55.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.923+0000 7f806d5dc740 10 client.26848 fetch_fsmap learned FSMap version 32 2024-07-26T11:25:55.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.923+0000 7f806d5dc740 1 -- 172.21.15.134:0/3678927696 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5581f374c160 con 0x5581f32e09f0 2024-07-26T11:25:55.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.924+0000 7f8049600640 1 -- 172.21.15.134:0/3678927696 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f804c0b5410 con 0x5581f32e09f0 2024-07-26T11:25:55.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.925+0000 7f8049600640 1 -- 172.21.15.134:0/3678927696 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f8050013070 con 0x5581f32e09f0 2024-07-26T11:25:55.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.925+0000 7f806d5dc740 10 client.26848 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:25:55.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.926+0000 7f806d5dc740 10 client.26848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:25:55.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.926+0000 7f806d5dc740 20 client.26848 populate_metadata read hostname 'smithi134' 2024-07-26T11:25:55.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.926+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3678927696 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5581f374c7c0 0x5581f374ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:25:55.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.926+0000 7f806d5dc740 4 client.26848 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-07-26T11:25:55.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.926+0000 7f806d5dc740 1 -- 172.21.15.134:0/3678927696 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5581f363b580 con 0x5581f374c7c0 2024-07-26T11:25:55.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.926+0000 7f805f400640 1 --2- 172.21.15.134:0/3678927696 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5581f374c7c0 0x5581f374ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:25:55.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.927+0000 7f805f400640 1 --2- 172.21.15.134:0/3678927696 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5581f374c7c0 0x5581f374ebe0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b93ddd83b43f5f5c server_cookie=1e8f3e6fa25bbd60 in_seq=0 out_seq=0 2024-07-26T11:25:55.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.928+0000 7f8049600640 10 client.26848 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:55.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.931+0000 7f8049600640 1 -- 172.21.15.134:0/3678927696 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5581f363b580 con 0x5581f374c7c0 2024-07-26T11:25:55.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.931+0000 7f8049600640 10 client.26848 handle_command_reply: tid=0 2024-07-26T11:25:55.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.931+0000 7f806d5dc740 1 client.26848 shutdown 2024-07-26T11:25:55.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.934+0000 7f806d5dc740 1 -- 172.21.15.134:0/3678927696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f32e09f0 msgr2=0x5581f36e4030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:55.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.934+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3678927696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f32e09f0 0x5581f36e4030 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f805000d530 tx=0x7f8050009610 comp rx=0 tx=0).stop 2024-07-26T11:25:55.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.934+0000 7f806d5dc740 1 -- 172.21.15.134:0/3678927696 shutdown_connections 2024-07-26T11:25:55.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.934+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3678927696 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5581f374c7c0 0x5581f374ebe0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.934+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3678927696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581f3322070 0x5581f36dfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.934+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3678927696 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5581f3301390 0x5581f36df4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.934+0000 7f806d5dc740 1 --2- 172.21.15.134:0/3678927696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5581f32e09f0 0x5581f36e4030 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.934+0000 7f806d5dc740 1 -- 172.21.15.134:0/3678927696 >> 172.21.15.134:0/3678927696 conn(0x5581f3316460 msgr2=0x5581f36ec660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:55.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.934+0000 7f8049600640 0 client.26848 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:25:55.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.935+0000 7f806d5dc740 1 -- 172.21.15.134:0/3678927696 shutdown_connections 2024-07-26T11:25:55.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.935+0000 7f806d5dc740 1 -- 172.21.15.134:0/3678927696 wait complete. 2024-07-26T11:25:55.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.937+0000 7f806d5dc740 20 client.26848 trim_cache size 0 max 16384 2024-07-26T11:25:55.939 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-07-26T11:25:55.939 INFO:teuthology.orchestra.run.smithi134.stdout: "status": "no active scrubs running", 2024-07-26T11:25:55.940 INFO:teuthology.orchestra.run.smithi134.stdout: "scrubs": {} 2024-07-26T11:25:55.940 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-07-26T11:25:55.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.938+0000 7f8066a00640 1 -- 172.21.15.134:0/883877634 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f80440b3dc0 msgr2=0x7f80440b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:55.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.938+0000 7f8066a00640 1 --2- 172.21.15.134:0/883877634 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f80440b3dc0 0x7f80440b62d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f80600735d0 tx=0x7f8050008040 comp rx=0 tx=0).stop 2024-07-26T11:25:55.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.938+0000 7f8066a00640 1 -- 172.21.15.134:0/883877634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8060072540 msgr2=0x7f80601a1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:25:55.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.938+0000 7f8066a00640 1 --2- 172.21.15.134:0/883877634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8060072540 0x7f80601a1be0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f804c00e9c0 tx=0x7f804c00ee80 comp rx=0 tx=0).stop 2024-07-26T11:25:55.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.939+0000 7f8066a00640 1 -- 172.21.15.134:0/883877634 shutdown_connections 2024-07-26T11:25:55.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.939+0000 7f8066a00640 1 --2- 172.21.15.134:0/883877634 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f80440b3dc0 0x7f80440b62d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.939+0000 7f8066a00640 1 --2- 172.21.15.134:0/883877634 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8060072e90 0x7f80601a6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.939+0000 7f8066a00640 1 --2- 172.21.15.134:0/883877634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8060072540 0x7f80601a1be0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.939+0000 7f8066a00640 1 --2- 172.21.15.134:0/883877634 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f806006cc00 0x7f80601a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:25:55.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.939+0000 7f8066a00640 1 -- 172.21.15.134:0/883877634 >> 172.21.15.134:0/883877634 conn(0x7f8060079160 msgr2=0x7f8060100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:25:55.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.939+0000 7f8066a00640 1 -- 172.21.15.134:0/883877634 shutdown_connections 2024-07-26T11:25:55.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:25:55.939+0000 7f8066a00640 1 -- 172.21.15.134:0/883877634 wait complete. 2024-07-26T11:25:58.177 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:57 smithi134 ceph-mon[31966]: pgmap v785: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 58 KiB/s rd, 260 MiB/s wr, 1.41k op/s 2024-07-26T11:25:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:57 smithi189 ceph-mon[36790]: pgmap v785: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 58 KiB/s rd, 260 MiB/s wr, 1.41k op/s 2024-07-26T11:25:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:57 smithi193 ceph-mon[37473]: pgmap v785: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 58 KiB/s rd, 260 MiB/s wr, 1.41k op/s 2024-07-26T11:25:59.023 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-07-26T11:25:59.024 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph fs dump --format=json 2024-07-26T11:26:00.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:25:59 smithi193 ceph-mon[37473]: pgmap v786: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 56 KiB/s rd, 257 MiB/s wr, 1.37k op/s 2024-07-26T11:26:00.091 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:26:00.179 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:25:59 smithi134 ceph-mon[31966]: pgmap v786: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 56 KiB/s rd, 257 MiB/s wr, 1.37k op/s 2024-07-26T11:26:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:25:59 smithi189 ceph-mon[36790]: pgmap v786: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 56 KiB/s rd, 257 MiB/s wr, 1.37k op/s 2024-07-26T11:26:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:01 smithi134 ceph-mon[31966]: pgmap v787: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 56 KiB/s rd, 256 MiB/s wr, 1.35k op/s 2024-07-26T11:26:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:01 smithi189 ceph-mon[36790]: pgmap v787: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 56 KiB/s rd, 256 MiB/s wr, 1.35k op/s 2024-07-26T11:26:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:01 smithi193 ceph-mon[37473]: pgmap v787: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 61 GiB data, 130 GiB used, 943 GiB / 1.0 TiB avail; 56 KiB/s rd, 256 MiB/s wr, 1.35k op/s 2024-07-26T11:26:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:02 smithi134 ceph-mon[31966]: pgmap v788: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 61 GiB data, 130 GiB used, 942 GiB / 1.0 TiB avail; 50 KiB/s rd, 237 MiB/s wr, 1.27k op/s 2024-07-26T11:26:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:02 smithi189 ceph-mon[36790]: pgmap v788: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 61 GiB data, 130 GiB used, 942 GiB / 1.0 TiB avail; 50 KiB/s rd, 237 MiB/s wr, 1.27k op/s 2024-07-26T11:26:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:02 smithi193 ceph-mon[37473]: pgmap v788: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 61 GiB data, 130 GiB used, 942 GiB / 1.0 TiB avail; 50 KiB/s rd, 237 MiB/s wr, 1.27k op/s 2024-07-26T11:26:05.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:04 smithi134 ceph-mon[31966]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T11:26:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:04 smithi189 ceph-mon[36790]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T11:26:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:04 smithi193 ceph-mon[37473]: osdmap e97: 12 total, 12 up, 12 in 2024-07-26T11:26:06.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:05 smithi134 ceph-mon[31966]: pgmap v790: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 61 GiB data, 131 GiB used, 942 GiB / 1.0 TiB avail; 63 KiB/s rd, 244 MiB/s wr, 1.33k op/s 2024-07-26T11:26:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:05 smithi189 ceph-mon[36790]: pgmap v790: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 61 GiB data, 131 GiB used, 942 GiB / 1.0 TiB avail; 63 KiB/s rd, 244 MiB/s wr, 1.33k op/s 2024-07-26T11:26:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:05 smithi193 ceph-mon[37473]: pgmap v790: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 61 GiB data, 131 GiB used, 942 GiB / 1.0 TiB avail; 63 KiB/s rd, 244 MiB/s wr, 1.33k op/s 2024-07-26T11:26:07.181 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:06 smithi134 ceph-mon[31966]: pgmap v791: 129 pgs: 129 active+clean; 61 GiB data, 131 GiB used, 942 GiB / 1.0 TiB avail; 63 KiB/s rd, 260 MiB/s wr, 1.47k op/s 2024-07-26T11:26:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:06 smithi189 ceph-mon[36790]: pgmap v791: 129 pgs: 129 active+clean; 61 GiB data, 131 GiB used, 942 GiB / 1.0 TiB avail; 63 KiB/s rd, 260 MiB/s wr, 1.47k op/s 2024-07-26T11:26:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:06 smithi193 ceph-mon[37473]: pgmap v791: 129 pgs: 129 active+clean; 61 GiB data, 131 GiB used, 942 GiB / 1.0 TiB avail; 63 KiB/s rd, 260 MiB/s wr, 1.47k op/s 2024-07-26T11:26:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.344+0000 7f8368800640 1 -- 172.21.15.134:0/3681169383 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8360102120 msgr2=0x7f8360102530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:08.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.344+0000 7f8368800640 1 --2- 172.21.15.134:0/3681169383 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8360102120 0x7f8360102530 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f8354009900 tx=0x7f835402a420 comp rx=0 tx=0).stop 2024-07-26T11:26:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.344+0000 7f8368800640 1 -- 172.21.15.134:0/3681169383 shutdown_connections 2024-07-26T11:26:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.344+0000 7f8368800640 1 --2- 172.21.15.134:0/3681169383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83601075e0 0x7f8360107a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.344+0000 7f8368800640 1 --2- 172.21.15.134:0/3681169383 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f8360102a70 0x7f8360106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.344+0000 7f8368800640 1 --2- 172.21.15.134:0/3681169383 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8360102120 0x7f8360102530 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.344+0000 7f8368800640 1 -- 172.21.15.134:0/3681169383 >> 172.21.15.134:0/3681169383 conn(0x7f836007a170 msgr2=0x7f836007a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:08.347 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.345+0000 7f8368800640 1 -- 172.21.15.134:0/3681169383 shutdown_connections 2024-07-26T11:26:08.348 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.345+0000 7f8368800640 1 -- 172.21.15.134:0/3681169383 wait complete. 2024-07-26T11:26:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.347+0000 7f8368800640 1 Processor -- start 2024-07-26T11:26:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.347+0000 7f8368800640 1 -- start start 2024-07-26T11:26:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.347+0000 7f8368800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8360102120 0x7f83601a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.347+0000 7f8368800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8360102a70 0x7f83601a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.348+0000 7f8368800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83601075e0 0x7f83601a5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.350 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.348+0000 7f8368800640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f8360119b50 con 0x7f83601075e0 2024-07-26T11:26:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.348+0000 7f8368800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f83601199d0 con 0x7f8360102120 2024-07-26T11:26:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.348+0000 7f8368800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f8360119cd0 con 0x7f8360102a70 2024-07-26T11:26:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.348+0000 7f8365e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8360102120 0x7f83601a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.348+0000 7f8365e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8360102120 0x7f83601a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:40304/0 (socket says 172.21.15.134:40304) 2024-07-26T11:26:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.348+0000 7f8365e00640 1 -- 172.21.15.134:0/2496610251 learned_addr learned my addr 172.21.15.134:0/2496610251 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.348+0000 7f8366800640 1 --2- 172.21.15.134:0/2496610251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83601075e0 0x7f83601a5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.349+0000 7f8365400640 1 --2- 172.21.15.134:0/2496610251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8360102a70 0x7f83601a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.350+0000 7f8365e00640 1 -- 172.21.15.134:0/2496610251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8360102a70 msgr2=0x7f83601a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.350+0000 7f8365e00640 1 --2- 172.21.15.134:0/2496610251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8360102a70 0x7f83601a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.350+0000 7f8365e00640 1 -- 172.21.15.134:0/2496610251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83601075e0 msgr2=0x7f83601a5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.350+0000 7f8365e00640 1 --2- 172.21.15.134:0/2496610251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83601075e0 0x7f83601a5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.350+0000 7f8365e00640 1 -- 172.21.15.134:0/2496610251 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83601a6520 con 0x7f8360102120 2024-07-26T11:26:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.350+0000 7f8365e00640 1 --2- 172.21.15.134:0/2496610251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8360102120 0x7f83601a05f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8354005ab0 tx=0x7f8354003d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.351+0000 7f835aa00640 1 -- 172.21.15.134:0/2496610251 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f835403f070 con 0x7f8360102120 2024-07-26T11:26:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.351+0000 7f835aa00640 1 -- 172.21.15.134:0/2496610251 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8354031d60 con 0x7f8360102120 2024-07-26T11:26:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.351+0000 7f835aa00640 1 -- 172.21.15.134:0/2496610251 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8354035030 con 0x7f8360102120 2024-07-26T11:26:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.351+0000 7f8368800640 1 -- 172.21.15.134:0/2496610251 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83601a67b0 con 0x7f8360102120 2024-07-26T11:26:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.352+0000 7f8368800640 1 -- 172.21.15.134:0/2496610251 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83601acfa0 con 0x7f8360102120 2024-07-26T11:26:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.361+0000 7f836f829740 1 Processor -- start 2024-07-26T11:26:08.363 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.361+0000 7f835aa00640 1 -- 172.21.15.134:0/2496610251 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f8354042070 con 0x7f8360102120 2024-07-26T11:26:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.361+0000 7f836f829740 1 -- start start 2024-07-26T11:26:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.361+0000 7f836f829740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b3f1b0 0x559274ee2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.363+0000 7f836f829740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b162f0 0x559274ee2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.363+0000 7f836f829740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b1e4d0 0x559274ee34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.363+0000 7f8366800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b3f1b0 0x559274ee2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.363+0000 7f8366800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b3f1b0 0x559274ee2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:44864/0 (socket says 172.21.15.134:44864) 2024-07-26T11:26:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.364+0000 7f8366800640 1 -- 172.21.15.134:0/4061248797 learned_addr learned my addr 172.21.15.134:0/4061248797 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.364+0000 7f836f829740 1 -- 172.21.15.134:0/4061248797 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559274e57270 con 0x559274b162f0 2024-07-26T11:26:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.364+0000 7f836f829740 1 -- 172.21.15.134:0/4061248797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559274e46770 con 0x559274b1e4d0 2024-07-26T11:26:08.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.364+0000 7f836f829740 1 -- 172.21.15.134:0/4061248797 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559274e4a7b0 con 0x559274b3f1b0 2024-07-26T11:26:08.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.365+0000 7f8365400640 1 --2- 172.21.15.134:0/4061248797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b1e4d0 0x559274ee34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.365+0000 7f8366800640 1 -- 172.21.15.134:0/4061248797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b1e4d0 msgr2=0x559274ee34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:08.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.365+0000 7f8366800640 1 --2- 172.21.15.134:0/4061248797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b1e4d0 0x559274ee34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.365+0000 7f8366800640 1 -- 172.21.15.134:0/4061248797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b162f0 msgr2=0x559274ee2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:08.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.365+0000 7f8365e00640 1 --2- 172.21.15.134:0/4061248797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b162f0 0x559274ee2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.368 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.365+0000 7f8366800640 1 --2- 172.21.15.134:0/4061248797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b162f0 0x559274ee2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.368 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.365+0000 7f8366800640 1 -- 172.21.15.134:0/4061248797 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559274baa790 con 0x559274b3f1b0 2024-07-26T11:26:08.369 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.367+0000 7f8365400640 1 --2- 172.21.15.134:0/4061248797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b1e4d0 0x559274ee34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:26:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.368+0000 7f8366800640 1 --2- 172.21.15.134:0/4061248797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b3f1b0 0x559274ee2a40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f835c00a430 tx=0x7f835c009650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=783c14a1ffeea067 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.369+0000 7f834f000640 1 -- 172.21.15.134:0/4061248797 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f835c013790 con 0x559274b3f1b0 2024-07-26T11:26:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.369+0000 7f834f000640 1 -- 172.21.15.134:0/4061248797 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f835c019070 con 0x559274b3f1b0 2024-07-26T11:26:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.369+0000 7f834f000640 1 -- 172.21.15.134:0/4061248797 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f835c014d40 con 0x559274b3f1b0 2024-07-26T11:26:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.369+0000 7f835aa00640 1 --2- 172.21.15.134:0/2496610251 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f83480b3f10 0x7f83480b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.370+0000 7f835aa00640 1 -- 172.21.15.134:0/2496610251 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7f8354104e30 con 0x7f8360102120 2024-07-26T11:26:08.372 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.370+0000 7f836f829740 1 -- 172.21.15.134:0/4061248797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b3f1b0 msgr2=0x559274ee2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:08.372 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.370+0000 7f836f829740 1 --2- 172.21.15.134:0/4061248797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b3f1b0 0x559274ee2a40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f835c00a430 tx=0x7f835c009650 comp rx=0 tx=0).stop 2024-07-26T11:26:08.376 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.373+0000 7f8365e00640 1 --2- 172.21.15.134:0/2496610251 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f83480b3f10 0x7f83480b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.375+0000 7f836f829740 1 -- 172.21.15.134:0/4061248797 shutdown_connections 2024-07-26T11:26:08.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.375+0000 7f836f829740 1 --2- 172.21.15.134:0/4061248797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b1e4d0 0x559274ee34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.375+0000 7f836f829740 1 --2- 172.21.15.134:0/4061248797 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b162f0 0x559274ee2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.375+0000 7f836f829740 1 --2- 172.21.15.134:0/4061248797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b3f1b0 0x559274ee2a40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.375+0000 7f836f829740 1 -- 172.21.15.134:0/4061248797 >> 172.21.15.134:0/4061248797 conn(0x559274b33300 msgr2=0x559274ee0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:08.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.375+0000 7f8365e00640 1 --2- 172.21.15.134:0/2496610251 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f83480b3f10 0x7f83480b6420 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x559274ee16c0 tx=0x7f83540d2f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:08.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.376+0000 7f836f829740 1 -- 172.21.15.134:0/4061248797 shutdown_connections 2024-07-26T11:26:08.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.376+0000 7f836f829740 1 -- 172.21.15.134:0/4061248797 wait complete. 2024-07-26T11:26:08.379 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.377+0000 7f836f829740 1 -- start start 2024-07-26T11:26:08.379 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.377+0000 7f836f829740 1 Processor -- start 2024-07-26T11:26:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.378+0000 7f836f829740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b162f0 0x559274eee670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.378+0000 7f836f829740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b1e4d0 0x559274eeec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.378+0000 7f836f829740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b3f1b0 0x559274efa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.378+0000 7f836f829740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x559274d80ab0 con 0x559274b1e4d0 2024-07-26T11:26:08.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.378+0000 7f836f829740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559274de3d50 con 0x559274b162f0 2024-07-26T11:26:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.378+0000 7f836f829740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x559274ef11c0 con 0x559274b3f1b0 2024-07-26T11:26:08.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.378+0000 7f836f829740 10 client.0 _ll_register_callbacks cb 0x559274b56a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:26:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.378+0000 7f8365400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b1e4d0 0x559274eeec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.379+0000 7f8366800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b162f0 0x559274eee670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.379+0000 7f8366800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b162f0 0x559274eee670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:40328/0 (socket says 172.21.15.134:40328) 2024-07-26T11:26:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.379+0000 7f8366800640 1 -- 172.21.15.134:0/3134719574 learned_addr learned my addr 172.21.15.134:0/3134719574 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.379+0000 7f8366800640 1 --2- 172.21.15.134:0/3134719574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b3f1b0 0x559274efa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.379+0000 7f8366800640 1 -- 172.21.15.134:0/3134719574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b3f1b0 msgr2=0x559274efa4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.379+0000 7f8366800640 1 --2- 172.21.15.134:0/3134719574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b3f1b0 0x559274efa4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.379+0000 7f8366800640 1 -- 172.21.15.134:0/3134719574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b1e4d0 msgr2=0x559274eeec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.379+0000 7f8366800640 1 --2- 172.21.15.134:0/3134719574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b1e4d0 0x559274eeec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.379+0000 7f8366800640 1 -- 172.21.15.134:0/3134719574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559274c23620 con 0x559274b162f0 2024-07-26T11:26:08.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.381+0000 7f8343400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:26:08.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.381+0000 7f8366800640 1 --2- 172.21.15.134:0/3134719574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b162f0 0x559274eee670 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f835c0145b0 tx=0x7f835c0148a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fafa891a409804bd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:08.384 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.382+0000 7f8343400640 1 -- 172.21.15.134:0/3134719574 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f835c0137a0 con 0x559274b162f0 2024-07-26T11:26:08.384 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.382+0000 7f836f829740 1 -- 172.21.15.134:0/3134719574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f835c019070 con 0x559274b162f0 2024-07-26T11:26:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.383+0000 7f8343400640 1 -- 172.21.15.134:0/3134719574 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7f835c026030 con 0x559274b162f0 2024-07-26T11:26:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.383+0000 7f8343400640 1 -- 172.21.15.134:0/3134719574 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f835c033050 con 0x559274b162f0 2024-07-26T11:26:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.383+0000 7f836f829740 10 client.26840 fetch_fsmap learned FSMap version 32 2024-07-26T11:26:08.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.383+0000 7f836f829740 1 -- 172.21.15.134:0/3134719574 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559274efd2d0 con 0x559274b162f0 2024-07-26T11:26:08.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.384+0000 7f8343400640 1 -- 172.21.15.134:0/3134719574 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f835c019070 con 0x559274b162f0 2024-07-26T11:26:08.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.384+0000 7f8343400640 1 -- 172.21.15.134:0/3134719574 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f835c020860 con 0x559274b162f0 2024-07-26T11:26:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.384+0000 7f836f829740 10 client.26840 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:26:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.385+0000 7f836f829740 10 client.26840 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:26:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.385+0000 7f836f829740 20 client.26840 populate_metadata read hostname 'smithi134' 2024-07-26T11:26:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.385+0000 7f836f829740 1 --2- 172.21.15.134:0/3134719574 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559274afdd80 0x559274f00750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.386+0000 7f836f829740 4 client.26840 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:26:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.386+0000 7f836f829740 1 -- 172.21.15.134:0/3134719574 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559274e591c0 con 0x559274afdd80 2024-07-26T11:26:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.386+0000 7f8365400640 1 --2- 172.21.15.134:0/3134719574 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559274afdd80 0x559274f00750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.388+0000 7f8365400640 1 --2- 172.21.15.134:0/3134719574 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559274afdd80 0x559274f00750 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=761dd2e6e78ced2d server_cookie=d030986c0c0eb942 in_seq=0 out_seq=0 2024-07-26T11:26:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.388+0000 7f8343400640 10 client.26840 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:26:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.399+0000 7f8343400640 1 -- 172.21.15.134:0/3134719574 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x559274e591c0 con 0x559274afdd80 2024-07-26T11:26:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.399+0000 7f8343400640 10 client.26840 handle_command_reply: tid=0 2024-07-26T11:26:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.400+0000 7f836f829740 1 client.26840 shutdown 2024-07-26T11:26:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.401+0000 7f836f829740 1 -- 172.21.15.134:0/3134719574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b162f0 msgr2=0x559274eee670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.401+0000 7f836f829740 1 --2- 172.21.15.134:0/3134719574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b162f0 0x559274eee670 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f835c0145b0 tx=0x7f835c0148a0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.402+0000 7f836f829740 1 -- 172.21.15.134:0/3134719574 shutdown_connections 2024-07-26T11:26:08.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.402+0000 7f836f829740 1 --2- 172.21.15.134:0/3134719574 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559274afdd80 0x559274f00750 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.403+0000 7f836f829740 1 --2- 172.21.15.134:0/3134719574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b3f1b0 0x559274efa4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.403+0000 7f836f829740 1 --2- 172.21.15.134:0/3134719574 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b1e4d0 0x559274eeec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.403+0000 7f836f829740 1 --2- 172.21.15.134:0/3134719574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b162f0 0x559274eee670 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.403+0000 7f836f829740 1 -- 172.21.15.134:0/3134719574 >> 172.21.15.134:0/3134719574 conn(0x559274b33300 msgr2=0x559274edf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.403+0000 7f8343400640 0 client.26840 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:26:08.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.404+0000 7f836f829740 1 -- 172.21.15.134:0/3134719574 shutdown_connections 2024-07-26T11:26:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.407+0000 7f836f829740 1 -- 172.21.15.134:0/3134719574 wait complete. 2024-07-26T11:26:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.409+0000 7f836f829740 20 client.26840 trim_cache size 0 max 16384 2024-07-26T11:26:08.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.483+0000 7f836f829740 1 Processor -- start 2024-07-26T11:26:08.486 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.484+0000 7f836f829740 1 -- start start 2024-07-26T11:26:08.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.486+0000 7f836f829740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b162f0 0x559274edf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.486+0000 7f836f829740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b1e4d0 0x559274ee8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.486+0000 7f836f829740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b3f1b0 0x559274efa2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.486+0000 7f836f829740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f83601b70e0 con 0x559274b1e4d0 2024-07-26T11:26:08.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.486+0000 7f836f829740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f83601b8c20 con 0x559274b3f1b0 2024-07-26T11:26:08.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.486+0000 7f836f829740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f83601b77b0 con 0x559274b162f0 2024-07-26T11:26:08.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.486+0000 7f8365400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b1e4d0 0x559274ee8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.487+0000 7f8366800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b3f1b0 0x559274efa2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.487+0000 7f8365400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b1e4d0 0x559274ee8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:60176/0 (socket says 172.21.15.134:60176) 2024-07-26T11:26:08.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.487+0000 7f8365400640 1 -- 172.21.15.134:0/1981076248 learned_addr learned my addr 172.21.15.134:0/1981076248 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:08.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.487+0000 7f8366800640 1 --2- 172.21.15.134:0/1981076248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b162f0 0x559274edf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.488+0000 7f8365400640 1 -- 172.21.15.134:0/1981076248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b162f0 msgr2=0x559274edf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:08.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.488+0000 7f8365400640 1 --2- 172.21.15.134:0/1981076248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b162f0 0x559274edf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.488+0000 7f8365400640 1 -- 172.21.15.134:0/1981076248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b3f1b0 msgr2=0x559274efa2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:08.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.488+0000 7f8365400640 1 --2- 172.21.15.134:0/1981076248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b3f1b0 0x559274efa2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.488+0000 7f8365400640 1 -- 172.21.15.134:0/1981076248 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559274ee5850 con 0x559274b1e4d0 2024-07-26T11:26:08.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.494+0000 7f8365400640 1 --2- 172.21.15.134:0/1981076248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b1e4d0 0x559274ee8210 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f836006c3f0 tx=0x7f8350008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c23d76d21995266f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.495+0000 7f834e600640 1 -- 172.21.15.134:0/1981076248 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f835001d250 con 0x559274b1e4d0 2024-07-26T11:26:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.495+0000 7f834e600640 1 -- 172.21.15.134:0/1981076248 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f835001d8a0 con 0x559274b1e4d0 2024-07-26T11:26:08.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.495+0000 7f834e600640 1 -- 172.21.15.134:0/1981076248 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8350015820 con 0x559274b1e4d0 2024-07-26T11:26:08.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.498+0000 7f836f829740 1 -- 172.21.15.134:0/1981076248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b1e4d0 msgr2=0x559274ee8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:08.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.498+0000 7f836f829740 1 --2- 172.21.15.134:0/1981076248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b1e4d0 0x559274ee8210 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f836006c3f0 tx=0x7f8350008040 comp rx=0 tx=0).stop 2024-07-26T11:26:08.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.499+0000 7f836f829740 1 -- 172.21.15.134:0/1981076248 shutdown_connections 2024-07-26T11:26:08.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.499+0000 7f836f829740 1 --2- 172.21.15.134:0/1981076248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b3f1b0 0x559274efa2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.499+0000 7f836f829740 1 --2- 172.21.15.134:0/1981076248 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b1e4d0 0x559274ee8210 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.501 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.499+0000 7f836f829740 1 --2- 172.21.15.134:0/1981076248 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b162f0 0x559274edf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.502 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.499+0000 7f836f829740 1 -- 172.21.15.134:0/1981076248 >> 172.21.15.134:0/1981076248 conn(0x559274b33300 msgr2=0x559274ee7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:08.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.501+0000 7f836f829740 1 -- 172.21.15.134:0/1981076248 shutdown_connections 2024-07-26T11:26:08.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.501+0000 7f836f829740 1 -- 172.21.15.134:0/1981076248 wait complete. 2024-07-26T11:26:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.503+0000 7f836f829740 1 -- start start 2024-07-26T11:26:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.503+0000 7f836f829740 1 Processor -- start 2024-07-26T11:26:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.507+0000 7f836f829740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b162f0 0x559274ef6c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.507+0000 7f836f829740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b1e4d0 0x559274ef1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.507+0000 7f836f829740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b3f1b0 0x559274ef1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.508+0000 7f8365400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b1e4d0 0x559274ef1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.508+0000 7f8366800640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b3f1b0 0x559274ef1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.508+0000 7f8365400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b1e4d0 0x559274ef1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:40334/0 (socket says 172.21.15.134:40334) 2024-07-26T11:26:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.509+0000 7f8365400640 1 -- 172.21.15.134:0/229596773 learned_addr learned my addr 172.21.15.134:0/229596773 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.509+0000 7f8366800640 1 --2- 172.21.15.134:0/229596773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b162f0 0x559274ef6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.509+0000 7f836f829740 1 -- 172.21.15.134:0/229596773 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f83601b8550 con 0x559274b3f1b0 2024-07-26T11:26:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.509+0000 7f836f829740 1 -- 172.21.15.134:0/229596773 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f83601b7e80 con 0x559274b1e4d0 2024-07-26T11:26:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.509+0000 7f836f829740 1 -- 172.21.15.134:0/229596773 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f83601b50c0 con 0x559274b162f0 2024-07-26T11:26:08.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.509+0000 7f836f829740 10 client.0 _ll_register_callbacks cb 0x7f83500093c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:26:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.510+0000 7f8366800640 1 -- 172.21.15.134:0/229596773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b1e4d0 msgr2=0x559274ef1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.510+0000 7f8366800640 1 --2- 172.21.15.134:0/229596773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b1e4d0 0x559274ef1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.510+0000 7f8366800640 1 -- 172.21.15.134:0/229596773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b3f1b0 msgr2=0x559274ef1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.510+0000 7f8366800640 1 --2- 172.21.15.134:0/229596773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b3f1b0 0x559274ef1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.510+0000 7f8366800640 1 -- 172.21.15.134:0/229596773 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559274f66d70 con 0x559274b162f0 2024-07-26T11:26:08.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.511+0000 7f8366800640 1 --2- 172.21.15.134:0/229596773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b162f0 0x559274ef6c80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f835c009ab0 tx=0x7f835c0148a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c29bebc80abcd7ec server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:08.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.512+0000 7f8343400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:26:08.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.512+0000 7f8343400640 1 -- 172.21.15.134:0/229596773 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f835c00ad80 con 0x559274b162f0 2024-07-26T11:26:08.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.512+0000 7f8343400640 1 -- 172.21.15.134:0/229596773 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9495+0+0 (secure 0 0 0) 0x7f835c02a040 con 0x559274b162f0 2024-07-26T11:26:08.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.513+0000 7f836f829740 1 -- 172.21.15.134:0/229596773 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f835001d8a0 con 0x559274b162f0 2024-07-26T11:26:08.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.514+0000 7f8343400640 1 -- 172.21.15.134:0/229596773 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f835c045050 con 0x559274b162f0 2024-07-26T11:26:08.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.514+0000 7f836f829740 10 client.26863 fetch_fsmap learned FSMap version 32 2024-07-26T11:26:08.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.514+0000 7f836f829740 1 -- 172.21.15.134:0/229596773 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x559274f67e50 con 0x559274b162f0 2024-07-26T11:26:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.515+0000 7f8343400640 1 -- 172.21.15.134:0/229596773 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f835001d8a0 con 0x559274b162f0 2024-07-26T11:26:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.515+0000 7f8343400640 1 -- 172.21.15.134:0/229596773 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7f835c013bf0 con 0x559274b162f0 2024-07-26T11:26:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.515+0000 7f836f829740 10 client.26863 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:26:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.515+0000 7f836f829740 10 client.26863 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:26:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.515+0000 7f836f829740 20 client.26863 populate_metadata read hostname 'smithi134' 2024-07-26T11:26:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.515+0000 7f836f829740 1 --2- 172.21.15.134:0/229596773 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559274ad31c0 0x559274f6a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.515+0000 7f836f829740 4 client.26863 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:26:08.517 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.515+0000 7f836f829740 1 -- 172.21.15.134:0/229596773 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559274e54c60 con 0x559274ad31c0 2024-07-26T11:26:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.516+0000 7f8365400640 1 --2- 172.21.15.134:0/229596773 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559274ad31c0 0x559274f6a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.517+0000 7f8343400640 10 client.26863 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:26:08.519 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.517+0000 7f8365400640 1 --2- 172.21.15.134:0/229596773 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559274ad31c0 0x559274f6a4f0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ec9c6d6871494eb server_cookie=95b354b68a32455a in_seq=0 out_seq=0 2024-07-26T11:26:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.721+0000 7f8343400640 1 -- 172.21.15.134:0/229596773 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559274e54c60 con 0x559274ad31c0 2024-07-26T11:26:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.721+0000 7f8343400640 10 client.26863 handle_command_reply: tid=0 2024-07-26T11:26:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.721+0000 7f836f829740 1 client.26863 shutdown 2024-07-26T11:26:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.721+0000 7f836f829740 1 -- 172.21.15.134:0/229596773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b162f0 msgr2=0x559274ef6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.721+0000 7f836f829740 1 --2- 172.21.15.134:0/229596773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b162f0 0x559274ef6c80 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f835c009ab0 tx=0x7f835c0148a0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.721+0000 7f836f829740 1 -- 172.21.15.134:0/229596773 shutdown_connections 2024-07-26T11:26:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.721+0000 7f836f829740 1 --2- 172.21.15.134:0/229596773 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x559274ad31c0 0x559274f6a4f0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.722+0000 7f836f829740 1 --2- 172.21.15.134:0/229596773 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x559274b3f1b0 0x559274ef1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.722+0000 7f836f829740 1 --2- 172.21.15.134:0/229596773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559274b1e4d0 0x559274ef1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.722+0000 7f836f829740 1 --2- 172.21.15.134:0/229596773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559274b162f0 0x559274ef6c80 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.722+0000 7f836f829740 1 -- 172.21.15.134:0/229596773 >> 172.21.15.134:0/229596773 conn(0x559274b33300 msgr2=0x559274ee3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.722+0000 7f8343400640 0 client.26863 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:26:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.722+0000 7f836f829740 1 -- 172.21.15.134:0/229596773 shutdown_connections 2024-07-26T11:26:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.722+0000 7f836f829740 1 -- 172.21.15.134:0/229596773 wait complete. 2024-07-26T11:26:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.722+0000 7f836f829740 20 client.26863 trim_cache size 0 max 16384 2024-07-26T11:26:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:26:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:26:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:26:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:26:08.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.730+0000 7f8368800640 1 -- 172.21.15.134:0/2496610251 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f83480b3f10 msgr2=0x7f83480b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:08.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.730+0000 7f8368800640 1 --2- 172.21.15.134:0/2496610251 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f83480b3f10 0x7f83480b6420 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x559274ee16c0 tx=0x7f83540d2f90 comp rx=0 tx=0).stop 2024-07-26T11:26:08.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.730+0000 7f8368800640 1 -- 172.21.15.134:0/2496610251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8360102120 msgr2=0x7f83601a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:08.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.730+0000 7f8368800640 1 --2- 172.21.15.134:0/2496610251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8360102120 0x7f83601a05f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f8354005ab0 tx=0x7f8354003d50 comp rx=0 tx=0).stop 2024-07-26T11:26:08.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.730+0000 7f8368800640 1 -- 172.21.15.134:0/2496610251 shutdown_connections 2024-07-26T11:26:08.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.730+0000 7f8368800640 1 --2- 172.21.15.134:0/2496610251 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f83480b3f10 0x7f83480b6420 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.730+0000 7f8368800640 1 --2- 172.21.15.134:0/2496610251 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8360102a70 0x7f83601a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.730+0000 7f8368800640 1 --2- 172.21.15.134:0/2496610251 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f83601075e0 0x7f83601a5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.730+0000 7f8368800640 1 --2- 172.21.15.134:0/2496610251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8360102120 0x7f83601a05f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:08.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.730+0000 7f8368800640 1 -- 172.21.15.134:0/2496610251 >> 172.21.15.134:0/2496610251 conn(0x7f836007a170 msgr2=0x7f8360104f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:08.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.731+0000 7f8368800640 1 -- 172.21.15.134:0/2496610251 shutdown_connections 2024-07-26T11:26:08.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:08.731+0000 7f8368800640 1 -- 172.21.15.134:0/2496610251 wait complete. 2024-07-26T11:26:09.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.923+0000 7faea3e00640 1 -- 172.21.15.134:0/3239874982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faea4107420 msgr2=0x7faea4107900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:09.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.923+0000 7faea3e00640 1 --2- 172.21.15.134:0/3239874982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faea4107420 0x7faea4107900 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009900 tx=0x7fae9c02a420 comp rx=0 tx=0).stop 2024-07-26T11:26:09.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.923+0000 7faea3e00640 1 -- 172.21.15.134:0/3239874982 shutdown_connections 2024-07-26T11:26:09.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.923+0000 7faea3e00640 1 --2- 172.21.15.134:0/3239874982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faea4107420 0x7faea4107900 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:09.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.923+0000 7faea3e00640 1 --2- 172.21.15.134:0/3239874982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faea4102930 0x7faea4106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:09.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.923+0000 7faea3e00640 1 --2- 172.21.15.134:0/3239874982 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faea4101fe0 0x7faea41023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:09.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.923+0000 7faea3e00640 1 -- 172.21.15.134:0/3239874982 >> 172.21.15.134:0/3239874982 conn(0x7faea40fdcf0 msgr2=0x7faea4100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:09.927 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.923+0000 7faea3e00640 1 -- 172.21.15.134:0/3239874982 shutdown_connections 2024-07-26T11:26:09.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.923+0000 7faea3e00640 1 -- 172.21.15.134:0/3239874982 wait complete. 2024-07-26T11:26:09.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.926+0000 7faea3e00640 1 Processor -- start 2024-07-26T11:26:09.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.926+0000 7faea3e00640 1 -- start start 2024-07-26T11:26:09.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.927+0000 7faea3e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faea4101fe0 0x7faea41a0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:09.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.927+0000 7faea3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faea4102930 0x7faea41a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:09.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.927+0000 7faea3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faea4107420 0x7faea41a5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:09.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.927+0000 7faea3e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7faea41198c0 con 0x7faea4101fe0 2024-07-26T11:26:09.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.927+0000 7faea3e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faea4119740 con 0x7faea4102930 2024-07-26T11:26:09.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.927+0000 7faea3e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7faea4119a40 con 0x7faea4107420 2024-07-26T11:26:09.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.928+0000 7faea2000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faea4102930 0x7faea41a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:09.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.928+0000 7faea2000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faea4102930 0x7faea41a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:57244/0 (socket says 172.21.15.134:57244) 2024-07-26T11:26:09.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.928+0000 7faea2000640 1 -- 172.21.15.134:0/240744825 learned_addr learned my addr 172.21.15.134:0/240744825 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:09.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.928+0000 7faea3400640 1 --2- 172.21.15.134:0/240744825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faea4107420 0x7faea41a5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:09.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.928+0000 7faea2a00640 1 --2- 172.21.15.134:0/240744825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faea4101fe0 0x7faea41a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:09.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.929+0000 7faea2000640 1 -- 172.21.15.134:0/240744825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faea4107420 msgr2=0x7faea41a5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:09.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.929+0000 7faea2000640 1 --2- 172.21.15.134:0/240744825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faea4107420 0x7faea41a5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:09.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.929+0000 7faea2000640 1 -- 172.21.15.134:0/240744825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faea4101fe0 msgr2=0x7faea41a0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:09.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.929+0000 7faea2000640 1 --2- 172.21.15.134:0/240744825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faea4101fe0 0x7faea41a0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:09.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.929+0000 7faea2000640 1 -- 172.21.15.134:0/240744825 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea41a6120 con 0x7faea4102930 2024-07-26T11:26:09.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.930+0000 7faea2000640 1 --2- 172.21.15.134:0/240744825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faea4102930 0x7faea41a0810 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fae9000c8c0 tx=0x7fae9000cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:09.932 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.930+0000 7faea8a00640 1 -- 172.21.15.134:0/240744825 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90007bb0 con 0x7faea4102930 2024-07-26T11:26:09.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.931+0000 7faea8a00640 1 -- 172.21.15.134:0/240744825 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae90007d50 con 0x7faea4102930 2024-07-26T11:26:09.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.931+0000 7faea3e00640 1 -- 172.21.15.134:0/240744825 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faea41a6410 con 0x7faea4102930 2024-07-26T11:26:09.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.931+0000 7faea3e00640 1 -- 172.21.15.134:0/240744825 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faea41acc30 con 0x7faea4102930 2024-07-26T11:26:09.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.932+0000 7faea8a00640 1 -- 172.21.15.134:0/240744825 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae90007bb0 con 0x7faea4102930 2024-07-26T11:26:09.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.934+0000 7faea3e00640 1 -- 172.21.15.134:0/240744825 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faea4106e60 con 0x7faea4102930 2024-07-26T11:26:09.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.943+0000 7faea8a00640 1 -- 172.21.15.134:0/240744825 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fae9000e4a0 con 0x7faea4102930 2024-07-26T11:26:09.948 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.946+0000 7faea8a00640 1 --2- 172.21.15.134:0/240744825 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fae880b4020 0x7fae880b6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:09.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.947+0000 7faea8a00640 1 -- 172.21.15.134:0/240744825 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9599+0+0 (secure 0 0 0) 0x7fae900e3ff0 con 0x7faea4102930 2024-07-26T11:26:09.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.947+0000 7faea2a00640 1 --2- 172.21.15.134:0/240744825 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fae880b4020 0x7fae880b6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:09.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.952+0000 7faea2a00640 1 --2- 172.21.15.134:0/240744825 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fae880b4020 0x7fae880b6530 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fae94005fd0 tx=0x7fae94009440 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:09.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:09.954+0000 7faea8a00640 1 -- 172.21.15.134:0/240744825 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194024 (secure 0 0 0) 0x7fae9001d060 con 0x7faea4102930 2024-07-26T11:26:10.176 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:09 smithi134 ceph-mon[31966]: pgmap v792: 129 pgs: 129 active+clean; 62 GiB data, 131 GiB used, 941 GiB / 1.0 TiB avail; 51 KiB/s rd, 266 MiB/s wr, 1.63k op/s 2024-07-26T11:26:10.176 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:09 smithi134 ceph-mon[31966]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T11:26:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:09 smithi189 ceph-mon[36790]: pgmap v792: 129 pgs: 129 active+clean; 62 GiB data, 131 GiB used, 941 GiB / 1.0 TiB avail; 51 KiB/s rd, 266 MiB/s wr, 1.63k op/s 2024-07-26T11:26:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:09 smithi189 ceph-mon[36790]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T11:26:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:09 smithi193 ceph-mon[37473]: pgmap v792: 129 pgs: 129 active+clean; 62 GiB data, 131 GiB used, 941 GiB / 1.0 TiB avail; 51 KiB/s rd, 266 MiB/s wr, 1.63k op/s 2024-07-26T11:26:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:09 smithi193 ceph-mon[37473]: osdmap e98: 12 total, 12 up, 12 in 2024-07-26T11:26:10.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:10.444+0000 7faea3e00640 1 -- 172.21.15.134:0/240744825 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faea41a13a0 con 0x7faea4102930 2024-07-26T11:26:10.447 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:10.445+0000 7faea8a00640 1 -- 172.21.15.134:0/240744825 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11990 (secure 0 0 0) 0x7fae90014030 con 0x7faea4102930 2024-07-26T11:26:10.447 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-07-26T11:26:10.448 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":32,"btime":"2024-07-26T11:19:25:331870+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6837/31742259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6836","nonce":31742259},{"type":"v1","addr":"172.21.15.134:6837","nonce":31742259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.134:6839/3307697142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6838","nonce":3307697142},{"type":"v1","addr":"172.21.15.134:6839","nonce":3307697142}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-07-26T11:10:25.500802+0000","modified":"2024-07-26T11:19:25.131276+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24721,"mds_1":24719,"mds_2":24703,"mds_3":24688,"mds_4":24686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24635":{"gid":24635,"name":"k","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/1897636383","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":1897636383},{"type":"v1","addr":"172.21.15.189:6833","nonce":1897636383}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24637":{"gid":24637,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/3093465661","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":3093465661},{"type":"v1","addr":"172.21.15.193:6833","nonce":3093465661}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24641":{"gid":24641,"name":"j","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.134:6835/4071410587","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6834","nonce":4071410587},{"type":"v1","addr":"172.21.15.134:6835","nonce":4071410587}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24665":{"gid":24665,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/1314042896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1314042896},{"type":"v1","addr":"172.21.15.189:6835","nonce":1314042896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24682":{"gid":24682,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6835/97099551","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":97099551},{"type":"v1","addr":"172.21.15.193:6835","nonce":97099551}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24686":{"gid":24686,"name":"h","rank":4,"incarnation":26,"state":"up:active","state_seq":32,"addr":"172.21.15.189:6837/1448732093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1448732093},{"type":"v1","addr":"172.21.15.189:6837","nonce":1448732093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24688":{"gid":24688,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":32,"addr":"172.21.15.193:6837/1968168008","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":1968168008},{"type":"v1","addr":"172.21.15.193:6837","nonce":1968168008}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24703":{"gid":24703,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":31,"addr":"172.21.15.193:6839/3985555366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":3985555366},{"type":"v1","addr":"172.21.15.193:6839","nonce":3985555366}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24719":{"gid":24719,"name":"b","rank":1,"incarnation":20,"state":"up:active","state_seq":28,"addr":"172.21.15.189:6839/790931242","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":790931242},{"type":"v1","addr":"172.21.15.189:6839","nonce":790931242}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24721":{"gid":24721,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":8,"addr":"172.21.15.134:6841/4067549550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6840","nonce":4067549550},{"type":"v1","addr":"172.21.15.134:6841","nonce":4067549550}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24721,"qdb_cluster":[24721,24719,24703,24688,24686]},"id":1}]} 2024-07-26T11:26:10.449 INFO:teuthology.orchestra.run.smithi134.stderr:dumped fsmap epoch 32 2024-07-26T11:26:10.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:10.450+0000 7faea3e00640 1 -- 172.21.15.134:0/240744825 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fae880b4020 msgr2=0x7fae880b6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:10.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:10.450+0000 7faea3e00640 1 --2- 172.21.15.134:0/240744825 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fae880b4020 0x7fae880b6530 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fae94005fd0 tx=0x7fae94009440 comp rx=0 tx=0).stop 2024-07-26T11:26:10.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:10.450+0000 7faea3e00640 1 -- 172.21.15.134:0/240744825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faea4102930 msgr2=0x7faea41a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:10.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:10.450+0000 7faea3e00640 1 --2- 172.21.15.134:0/240744825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faea4102930 0x7faea41a0810 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fae9000c8c0 tx=0x7fae9000cd80 comp rx=0 tx=0).stop 2024-07-26T11:26:10.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:10.450+0000 7faea3e00640 1 -- 172.21.15.134:0/240744825 shutdown_connections 2024-07-26T11:26:10.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:10.450+0000 7faea3e00640 1 --2- 172.21.15.134:0/240744825 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fae880b4020 0x7fae880b6530 secure :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fae94005fd0 tx=0x7fae94009440 comp rx=0 tx=0).stop 2024-07-26T11:26:10.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:10.450+0000 7faea3e00640 1 --2- 172.21.15.134:0/240744825 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faea4107420 0x7faea41a5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:10.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:10.450+0000 7faea3e00640 1 --2- 172.21.15.134:0/240744825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faea4102930 0x7faea41a0810 secure :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fae9000c8c0 tx=0x7fae9000cd80 comp rx=0 tx=0).stop 2024-07-26T11:26:10.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:10.450+0000 7faea3e00640 1 --2- 172.21.15.134:0/240744825 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7faea4101fe0 0x7faea41a0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:10.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:10.450+0000 7faea3e00640 1 -- 172.21.15.134:0/240744825 >> 172.21.15.134:0/240744825 conn(0x7faea40fdcf0 msgr2=0x7faea4104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:10.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:10.451+0000 7faea3e00640 1 -- 172.21.15.134:0/240744825 shutdown_connections 2024-07-26T11:26:10.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:10.451+0000 7faea3e00640 1 -- 172.21.15.134:0/240744825 wait complete. 2024-07-26T11:26:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:10 smithi134 ceph-mon[31966]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T11:26:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:10 smithi134 ceph-mon[31966]: from='client.? 172.21.15.134:0/240744825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:26:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:10 smithi134 ceph-mon[31966]: pgmap v795: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 62 GiB data, 132 GiB used, 941 GiB / 1.0 TiB avail; 19 KiB/s rd, 261 MiB/s wr, 1.77k op/s 2024-07-26T11:26:11.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:10 smithi189 ceph-mon[36790]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T11:26:11.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:10 smithi189 ceph-mon[36790]: from='client.? 172.21.15.134:0/240744825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:26:11.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:10 smithi189 ceph-mon[36790]: pgmap v795: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 62 GiB data, 132 GiB used, 941 GiB / 1.0 TiB avail; 19 KiB/s rd, 261 MiB/s wr, 1.77k op/s 2024-07-26T11:26:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:10 smithi193 ceph-mon[37473]: osdmap e99: 12 total, 12 up, 12 in 2024-07-26T11:26:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:10 smithi193 ceph-mon[37473]: from='client.? 172.21.15.134:0/240744825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-07-26T11:26:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:10 smithi193 ceph-mon[37473]: pgmap v795: 129 pgs: 4 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 113 active+clean; 62 GiB data, 132 GiB used, 941 GiB / 1.0 TiB avail; 19 KiB/s rd, 261 MiB/s wr, 1.77k op/s 2024-07-26T11:26:12.972 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:4 damage ls 2024-07-26T11:26:13.690 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:26:14.084 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:13 smithi134 ceph-mon[31966]: pgmap v796: 129 pgs: 7 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 103 active+clean; 61 GiB data, 131 GiB used, 942 GiB / 1.0 TiB avail; 47 KiB/s rd, 237 MiB/s wr, 1.63k op/s 2024-07-26T11:26:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:13 smithi189 ceph-mon[36790]: pgmap v796: 129 pgs: 7 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 103 active+clean; 61 GiB data, 131 GiB used, 942 GiB / 1.0 TiB avail; 47 KiB/s rd, 237 MiB/s wr, 1.63k op/s 2024-07-26T11:26:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:13 smithi193 ceph-mon[37473]: pgmap v796: 129 pgs: 7 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 103 active+clean; 61 GiB data, 131 GiB used, 942 GiB / 1.0 TiB avail; 47 KiB/s rd, 237 MiB/s wr, 1.63k op/s 2024-07-26T11:26:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:15 smithi134 ceph-mon[31966]: pgmap v797: 129 pgs: 18 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 57 active+clean; 59 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 56 KiB/s rd, 224 MiB/s wr, 1.62k op/s 2024-07-26T11:26:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:15 smithi189 ceph-mon[36790]: pgmap v797: 129 pgs: 18 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 57 active+clean; 59 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 56 KiB/s rd, 224 MiB/s wr, 1.62k op/s 2024-07-26T11:26:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:15 smithi193 ceph-mon[37473]: pgmap v797: 129 pgs: 18 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 57 active+clean; 59 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 56 KiB/s rd, 224 MiB/s wr, 1.62k op/s 2024-07-26T11:26:18.176 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:17 smithi134 ceph-mon[31966]: pgmap v798: 129 pgs: 14 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 74 active+clean; 57 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 47 KiB/s rd, 153 MiB/s wr, 1.10k op/s 2024-07-26T11:26:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:17 smithi189 ceph-mon[36790]: pgmap v798: 129 pgs: 14 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 74 active+clean; 57 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 47 KiB/s rd, 153 MiB/s wr, 1.10k op/s 2024-07-26T11:26:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:17 smithi193 ceph-mon[37473]: pgmap v798: 129 pgs: 14 active+clean+snaptrim, 41 active+clean+snaptrim_wait, 74 active+clean; 57 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 47 KiB/s rd, 153 MiB/s wr, 1.10k op/s 2024-07-26T11:26:18.861 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:18 smithi134 ceph-mon[31966]: pgmap v799: 129 pgs: 7 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 103 active+clean; 56 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 57 KiB/s rd, 154 MiB/s wr, 1.17k op/s 2024-07-26T11:26:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:18 smithi189 ceph-mon[36790]: pgmap v799: 129 pgs: 7 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 103 active+clean; 56 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 57 KiB/s rd, 154 MiB/s wr, 1.17k op/s 2024-07-26T11:26:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:18 smithi193 ceph-mon[37473]: pgmap v799: 129 pgs: 7 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 103 active+clean; 56 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 57 KiB/s rd, 154 MiB/s wr, 1.17k op/s 2024-07-26T11:26:19.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.496+0000 7fa593e00640 1 -- 172.21.15.134:0/1144538436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa59410c0e0 msgr2=0x7fa59410c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:19.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.496+0000 7fa593e00640 1 --2- 172.21.15.134:0/1144538436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa59410c0e0 0x7fa59410c4f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fa584009930 tx=0x7fa58402a440 comp rx=0 tx=0).stop 2024-07-26T11:26:19.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.500+0000 7fa593e00640 1 -- 172.21.15.134:0/1144538436 shutdown_connections 2024-07-26T11:26:19.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.500+0000 7fa593e00640 1 --2- 172.21.15.134:0/1144538436 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa59410ca30 0x7fa5941164c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.500+0000 7fa593e00640 1 --2- 172.21.15.134:0/1144538436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa59410c0e0 0x7fa59410c4f0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.500+0000 7fa593e00640 1 --2- 172.21.15.134:0/1144538436 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa594109ec0 0x7fa59410a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.500+0000 7fa593e00640 1 -- 172.21.15.134:0/1144538436 >> 172.21.15.134:0/1144538436 conn(0x7fa594078e50 msgr2=0x7fa59407b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:19.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.500+0000 7fa593e00640 1 -- 172.21.15.134:0/1144538436 shutdown_connections 2024-07-26T11:26:19.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.500+0000 7fa593e00640 1 -- 172.21.15.134:0/1144538436 wait complete. 2024-07-26T11:26:19.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.502+0000 7fa593e00640 1 Processor -- start 2024-07-26T11:26:19.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.503+0000 7fa593e00640 1 -- start start 2024-07-26T11:26:19.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.505+0000 7fa593e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa594109ec0 0x7fa594085720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.505+0000 7fa593e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa59410c0e0 0x7fa59407e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.506+0000 7fa593e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa59410ca30 0x7fa59407efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.506+0000 7fa592a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa594109ec0 0x7fa594085720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:19.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.506+0000 7fa593e00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa594119950 con 0x7fa59410c0e0 2024-07-26T11:26:19.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.507+0000 7fa593e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa5941197d0 con 0x7fa59410ca30 2024-07-26T11:26:19.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.507+0000 7fa593e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa594119ad0 con 0x7fa594109ec0 2024-07-26T11:26:19.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.507+0000 7fa592000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa59410c0e0 0x7fa59407e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:19.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.507+0000 7fa592000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa59410c0e0 0x7fa59407e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:51704/0 (socket says 172.21.15.134:51704) 2024-07-26T11:26:19.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.507+0000 7fa592000640 1 -- 172.21.15.134:0/878626106 learned_addr learned my addr 172.21.15.134:0/878626106 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:19.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.507+0000 7fa593400640 1 --2- 172.21.15.134:0/878626106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa59410ca30 0x7fa59407efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:19.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.508+0000 7fa593400640 1 -- 172.21.15.134:0/878626106 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa594109ec0 msgr2=0x7fa594085720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:19.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.510+0000 7fa593400640 1 --2- 172.21.15.134:0/878626106 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa594109ec0 0x7fa594085720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.510+0000 7fa593400640 1 -- 172.21.15.134:0/878626106 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa59410c0e0 msgr2=0x7fa59407e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:19.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.510+0000 7fa593400640 1 --2- 172.21.15.134:0/878626106 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa59410c0e0 0x7fa59407e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.510+0000 7fa593400640 1 -- 172.21.15.134:0/878626106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa59407f7d0 con 0x7fa59410ca30 2024-07-26T11:26:19.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.510+0000 7fa592a00640 1 --2- 172.21.15.134:0/878626106 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa594109ec0 0x7fa594085720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:26:19.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.511+0000 7fa592000640 1 --2- 172.21.15.134:0/878626106 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa59410c0e0 0x7fa59407e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:26:19.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.511+0000 7fa593400640 1 --2- 172.21.15.134:0/878626106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa59410ca30 0x7fa59407efb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa58000c710 tx=0x7fa58000cbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:19.514 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.512+0000 7fa58b400640 1 -- 172.21.15.134:0/878626106 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa580004220 con 0x7fa59410ca30 2024-07-26T11:26:19.514 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.512+0000 7fa58b400640 1 -- 172.21.15.134:0/878626106 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa58000ce20 con 0x7fa59410ca30 2024-07-26T11:26:19.514 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.512+0000 7fa58b400640 1 -- 172.21.15.134:0/878626106 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa580003db0 con 0x7fa59410ca30 2024-07-26T11:26:19.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.515+0000 7fa593e00640 1 -- 172.21.15.134:0/878626106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa594084450 con 0x7fa59410ca30 2024-07-26T11:26:19.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.515+0000 7fa593e00640 1 -- 172.21.15.134:0/878626106 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa594084910 con 0x7fa59410ca30 2024-07-26T11:26:19.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.525+0000 7fa58b400640 1 -- 172.21.15.134:0/878626106 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fa5800079b0 con 0x7fa59410ca30 2024-07-26T11:26:19.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.531+0000 7fa58b400640 1 --2- 172.21.15.134:0/878626106 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5780b3dc0 0x7fa5780b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.535+0000 7fa592a00640 1 --2- 172.21.15.134:0/878626106 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5780b3dc0 0x7fa5780b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:19.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.539+0000 7fa58b400640 1 -- 172.21.15.134:0/878626106 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9599+0+0 (secure 0 0 0) 0x7fa5800e43e0 con 0x7fa59410ca30 2024-07-26T11:26:19.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.539+0000 7fa592a00640 1 --2- 172.21.15.134:0/878626106 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5780b3dc0 0x7fa5780b62d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa59407d630 tx=0x7fa58c009400 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:19.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.545+0000 7fa5a01dd740 1 Processor -- start 2024-07-26T11:26:19.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.546+0000 7fa5a01dd740 1 -- start start 2024-07-26T11:26:19.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.547+0000 7fa5a01dd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f087201070 0x55f0875a6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.547+0000 7fa5a01dd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871d81b0 0x55f0875a6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.547+0000 7fa5a01dd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f0871e0390 0x55f0875a6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.547+0000 7fa5a01dd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f08751cbf0 con 0x55f0871e0390 2024-07-26T11:26:19.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.547+0000 7fa5a01dd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f08750c0f0 con 0x55f087201070 2024-07-26T11:26:19.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.547+0000 7fa5a01dd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f087510130 con 0x55f0871d81b0 2024-07-26T11:26:19.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.547+0000 7fa592000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871d81b0 0x55f0875a6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:19.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.547+0000 7fa592000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871d81b0 0x55f0875a6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:39958/0 (socket says 172.21.15.134:39958) 2024-07-26T11:26:19.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.547+0000 7fa592000640 1 -- 172.21.15.134:0/1193097553 learned_addr learned my addr 172.21.15.134:0/1193097553 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:19.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.547+0000 7fa592a00640 1 --2- 172.21.15.134:0/1193097553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f087201070 0x55f0875a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:19.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.548+0000 7fa592000640 1 -- 172.21.15.134:0/1193097553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f087201070 msgr2=0x55f0875a6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:19.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.548+0000 7fa592000640 1 --2- 172.21.15.134:0/1193097553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f087201070 0x55f0875a6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.548+0000 7fa592000640 1 -- 172.21.15.134:0/1193097553 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f0871e0390 msgr2=0x55f0875a6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:19.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.548+0000 7fa592000640 1 --2- 172.21.15.134:0/1193097553 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f0871e0390 0x55f0875a6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.548+0000 7fa592000640 1 -- 172.21.15.134:0/1193097553 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f08726c650 con 0x55f0871d81b0 2024-07-26T11:26:19.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.550+0000 7fa592000640 1 --2- 172.21.15.134:0/1193097553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871d81b0 0x55f0875a6570 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa58402a960 tx=0x7fa584004390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fac6d4493c1ecd1e server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:19.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.551+0000 7fa57f400640 1 -- 172.21.15.134:0/1193097553 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584035030 con 0x55f0871d81b0 2024-07-26T11:26:19.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.551+0000 7fa57f400640 1 -- 172.21.15.134:0/1193097553 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5840046c0 con 0x55f0871d81b0 2024-07-26T11:26:19.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.551+0000 7fa57f400640 1 -- 172.21.15.134:0/1193097553 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584003850 con 0x55f0871d81b0 2024-07-26T11:26:19.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.554+0000 7fa5a01dd740 1 -- 172.21.15.134:0/1193097553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871d81b0 msgr2=0x55f0875a6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:19.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.554+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/1193097553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871d81b0 0x55f0875a6570 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa58402a960 tx=0x7fa584004390 comp rx=0 tx=0).stop 2024-07-26T11:26:19.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.555+0000 7fa5a01dd740 1 -- 172.21.15.134:0/1193097553 shutdown_connections 2024-07-26T11:26:19.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.555+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/1193097553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871d81b0 0x55f0875a6570 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.555+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/1193097553 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f0871e0390 0x55f0875a6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.555+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/1193097553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f087201070 0x55f0875a6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.555+0000 7fa5a01dd740 1 -- 172.21.15.134:0/1193097553 >> 172.21.15.134:0/1193097553 conn(0x55f0871f5460 msgr2=0x55f0875a3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:19.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.555+0000 7fa5a01dd740 1 -- 172.21.15.134:0/1193097553 shutdown_connections 2024-07-26T11:26:19.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.556+0000 7fa5a01dd740 1 -- 172.21.15.134:0/1193097553 wait complete. 2024-07-26T11:26:19.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.557+0000 7fa5a01dd740 1 -- start start 2024-07-26T11:26:19.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.557+0000 7fa5a01dd740 1 Processor -- start 2024-07-26T11:26:19.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.558+0000 7fa5a01dd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871d81b0 0x55f0875b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.558+0000 7fa5a01dd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0871e0390 0x55f0875b1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.558+0000 7fa5a01dd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f087201070 0x55f0875b2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.558+0000 7fa5a01dd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55f08736f700 con 0x55f087201070 2024-07-26T11:26:19.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.558+0000 7fa5a01dd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f0874a86c0 con 0x55f0871e0390 2024-07-26T11:26:19.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.558+0000 7fa5a01dd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55f0875b47b0 con 0x55f0871d81b0 2024-07-26T11:26:19.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.559+0000 7fa5a01dd740 10 client.0 _ll_register_callbacks cb 0x55f0874b8170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:26:19.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.559+0000 7fa593400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f087201070 0x55f0875b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:19.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.559+0000 7fa592000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0871e0390 0x55f0875b1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:19.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.559+0000 7fa592000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0871e0390 0x55f0875b1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:57268/0 (socket says 172.21.15.134:57268) 2024-07-26T11:26:19.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.559+0000 7fa592000640 1 -- 172.21.15.134:0/1311233978 learned_addr learned my addr 172.21.15.134:0/1311233978 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:19.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.560+0000 7fa593400640 1 -- 172.21.15.134:0/1311233978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871d81b0 msgr2=0x55f0875b1660 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:26:19.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.560+0000 7fa593400640 1 --2- 172.21.15.134:0/1311233978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871d81b0 0x55f0875b1660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.560+0000 7fa593400640 1 -- 172.21.15.134:0/1311233978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0871e0390 msgr2=0x55f0875b1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:19.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.560+0000 7fa593400640 1 --2- 172.21.15.134:0/1311233978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0871e0390 0x55f0875b1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.560+0000 7fa593400640 1 -- 172.21.15.134:0/1311233978 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0874bf860 con 0x55f087201070 2024-07-26T11:26:19.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.561+0000 7fa573e00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:26:19.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.561+0000 7fa593400640 1 --2- 172.21.15.134:0/1311233978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f087201070 0x55f0875b2120 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa5800e1700 tx=0x7fa5800aa800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ffe17c765b32ea server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:19.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.564+0000 7fa573e00640 1 -- 172.21.15.134:0/1311233978 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5800af1c0 con 0x55f087201070 2024-07-26T11:26:19.566 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.564+0000 7fa5a01dd740 1 -- 172.21.15.134:0/1311233978 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5840046c0 con 0x55f087201070 2024-07-26T11:26:19.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.565+0000 7fa573e00640 1 -- 172.21.15.134:0/1311233978 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9599+0+0 (secure 0 0 0) 0x7fa5800a71b0 con 0x55f087201070 2024-07-26T11:26:19.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.565+0000 7fa573e00640 1 -- 172.21.15.134:0/1311233978 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa5800af1c0 con 0x55f087201070 2024-07-26T11:26:19.567 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.565+0000 7fa5a01dd740 10 client.16203 fetch_fsmap learned FSMap version 32 2024-07-26T11:26:19.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.566+0000 7fa5a01dd740 1 -- 172.21.15.134:0/1311233978 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f0875d14e0 con 0x55f087201070 2024-07-26T11:26:19.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.567+0000 7fa573e00640 1 -- 172.21.15.134:0/1311233978 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5840046c0 con 0x55f087201070 2024-07-26T11:26:19.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.568+0000 7fa573e00640 1 -- 172.21.15.134:0/1311233978 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fa5800b37e0 con 0x55f087201070 2024-07-26T11:26:19.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.568+0000 7fa5a01dd740 10 client.16203 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:26:19.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.568+0000 7fa5a01dd740 10 client.16203 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-07-26T11:26:19.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.568+0000 7fa5a01dd740 20 client.16203 populate_metadata read hostname 'smithi134' 2024-07-26T11:26:19.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.568+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/1311233978 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x55f0871bf9f0 0x55f0875d4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.568+0000 7fa5a01dd740 4 client.16203 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:26:19.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.568+0000 7fa5a01dd740 1 -- 172.21.15.134:0/1311233978 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f08751eb40 con 0x55f0871bf9f0 2024-07-26T11:26:19.572 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.569+0000 7fa592a00640 1 --2- 172.21.15.134:0/1311233978 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x55f0871bf9f0 0x55f0875d4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:19.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.572+0000 7fa592a00640 1 --2- 172.21.15.134:0/1311233978 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x55f0871bf9f0 0x55f0875d4900 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8d509fdf9b1ff19a server_cookie=706801a7b6eaec76 in_seq=0 out_seq=0 2024-07-26T11:26:19.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.572+0000 7fa573e00640 10 client.16203 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:26:19.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.573+0000 7fa573e00640 1 -- 172.21.15.134:0/1311233978 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55f08751eb40 con 0x55f0871bf9f0 2024-07-26T11:26:19.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.573+0000 7fa573e00640 10 client.16203 handle_command_reply: tid=0 2024-07-26T11:26:19.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.573+0000 7fa5a01dd740 1 client.16203 shutdown 2024-07-26T11:26:19.575 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.573+0000 7fa5a01dd740 1 -- 172.21.15.134:0/1311233978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f087201070 msgr2=0x55f0875b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:19.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.573+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/1311233978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f087201070 0x55f0875b2120 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa5800e1700 tx=0x7fa5800aa800 comp rx=0 tx=0).stop 2024-07-26T11:26:19.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.574+0000 7fa5a01dd740 1 -- 172.21.15.134:0/1311233978 shutdown_connections 2024-07-26T11:26:19.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.574+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/1311233978 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x55f0871bf9f0 0x55f0875d4900 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.574+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/1311233978 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f087201070 0x55f0875b2120 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.574+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/1311233978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0871e0390 0x55f0875b1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.574+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/1311233978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871d81b0 0x55f0875b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.574+0000 7fa5a01dd740 1 -- 172.21.15.134:0/1311233978 >> 172.21.15.134:0/1311233978 conn(0x55f0871f5460 msgr2=0x55f0875a91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:19.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.574+0000 7fa573e00640 0 client.16203 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:26:19.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.574+0000 7fa5a01dd740 1 -- 172.21.15.134:0/1311233978 shutdown_connections 2024-07-26T11:26:19.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.574+0000 7fa5a01dd740 1 -- 172.21.15.134:0/1311233978 wait complete. 2024-07-26T11:26:19.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.575+0000 7fa5a01dd740 20 client.16203 trim_cache size 0 max 16384 2024-07-26T11:26:19.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.619+0000 7fa5a01dd740 1 Processor -- start 2024-07-26T11:26:19.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.619+0000 7fa5a01dd740 1 -- start start 2024-07-26T11:26:19.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.620+0000 7fa5a01dd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0871d81b0 0x55f0875a2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.620+0000 7fa5a01dd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871e0390 0x55f0875da100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.620+0000 7fa5a01dd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f087201070 0x55f0875da640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.623 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.620+0000 7fa592000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871e0390 0x55f0875da100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:19.623 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.620+0000 7fa5a01dd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa5941b6ed0 con 0x55f087201070 2024-07-26T11:26:19.623 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.620+0000 7fa5a01dd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa5941b8a10 con 0x55f0871d81b0 2024-07-26T11:26:19.623 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.620+0000 7fa5a01dd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa5941b75a0 con 0x55f0871e0390 2024-07-26T11:26:19.624 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.621+0000 7fa593400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f087201070 0x55f0875da640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:19.624 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.621+0000 7fa593400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f087201070 0x55f0875da640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:51726/0 (socket says 172.21.15.134:51726) 2024-07-26T11:26:19.624 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.621+0000 7fa593400640 1 -- 172.21.15.134:0/2427790091 learned_addr learned my addr 172.21.15.134:0/2427790091 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:19.624 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.621+0000 7fa592a00640 1 --2- 172.21.15.134:0/2427790091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0871d81b0 0x55f0875a2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:19.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.621+0000 7fa593400640 1 -- 172.21.15.134:0/2427790091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871e0390 msgr2=0x55f0875da100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:19.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.620+0000 7fa592000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871e0390 0x55f0875da100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:39980/0 (socket says 172.21.15.134:39980) 2024-07-26T11:26:19.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.622+0000 7fa593400640 1 --2- 172.21.15.134:0/2427790091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871e0390 0x55f0875da100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.622+0000 7fa593400640 1 -- 172.21.15.134:0/2427790091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0871d81b0 msgr2=0x55f0875a2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:19.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.623+0000 7fa593400640 1 --2- 172.21.15.134:0/2427790091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0871d81b0 0x55f0875a2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.623+0000 7fa593400640 1 -- 172.21.15.134:0/2427790091 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0875d4e60 con 0x55f087201070 2024-07-26T11:26:19.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.623+0000 7fa592a00640 1 --2- 172.21.15.134:0/2427790091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0871d81b0 0x55f0875a2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:26:19.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.623+0000 7fa592000640 1 --2- 172.21.15.134:0/2427790091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871e0390 0x55f0875da100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:26:19.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.627+0000 7fa593400640 1 --2- 172.21.15.134:0/2427790091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f087201070 0x55f0875da640 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa5800aab50 tx=0x7fa580003840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c70aa7cbfd15916 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:19.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.629+0000 7fa57ea00640 1 -- 172.21.15.134:0/2427790091 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5800a6a80 con 0x55f087201070 2024-07-26T11:26:19.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.629+0000 7fa57ea00640 1 -- 172.21.15.134:0/2427790091 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5800b7020 con 0x55f087201070 2024-07-26T11:26:19.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.629+0000 7fa57ea00640 1 -- 172.21.15.134:0/2427790091 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5800b37e0 con 0x55f087201070 2024-07-26T11:26:19.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.631+0000 7fa5a01dd740 1 -- 172.21.15.134:0/2427790091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f087201070 msgr2=0x55f0875da640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:19.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.631+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/2427790091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f087201070 0x55f0875da640 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa5800aab50 tx=0x7fa580003840 comp rx=0 tx=0).stop 2024-07-26T11:26:19.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.631+0000 7fa5a01dd740 1 -- 172.21.15.134:0/2427790091 shutdown_connections 2024-07-26T11:26:19.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.631+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/2427790091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871e0390 0x55f0875da100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.631+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/2427790091 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f087201070 0x55f0875da640 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.631+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/2427790091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0871d81b0 0x55f0875a2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.631+0000 7fa5a01dd740 1 -- 172.21.15.134:0/2427790091 >> 172.21.15.134:0/2427790091 conn(0x55f0871f5460 msgr2=0x55f0875a7a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:19.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.632+0000 7fa5a01dd740 1 -- 172.21.15.134:0/2427790091 shutdown_connections 2024-07-26T11:26:19.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.632+0000 7fa5a01dd740 1 -- 172.21.15.134:0/2427790091 wait complete. 2024-07-26T11:26:19.637 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.635+0000 7fa5a01dd740 1 -- start start 2024-07-26T11:26:19.637 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.635+0000 7fa5a01dd740 1 Processor -- start 2024-07-26T11:26:19.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.638+0000 7fa5a01dd740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f0871d81b0 0x55f0875cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.639+0000 7fa5a01dd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871e0390 0x55f0875c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.639+0000 7fa5a01dd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f087201070 0x55f0875c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.639+0000 7fa5a01dd740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fa5941b8340 con 0x55f0871d81b0 2024-07-26T11:26:19.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.639+0000 7fa5a01dd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa5941b7c70 con 0x55f087201070 2024-07-26T11:26:19.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.640+0000 7fa592000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871e0390 0x55f0875c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:19.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.640+0000 7fa592000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871e0390 0x55f0875c87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:39994/0 (socket says 172.21.15.134:39994) 2024-07-26T11:26:19.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.640+0000 7fa5a01dd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fa59410a2d0 con 0x55f0871e0390 2024-07-26T11:26:19.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.640+0000 7fa5a01dd740 10 client.0 _ll_register_callbacks cb 0x7fa5800aa3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:26:19.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.640+0000 7fa593400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f087201070 0x55f0875c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:19.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.640+0000 7fa593400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f087201070 0x55f0875c8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:57298/0 (socket says 172.21.15.134:57298) 2024-07-26T11:26:19.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.641+0000 7fa593400640 1 -- 172.21.15.134:0/807262340 learned_addr learned my addr 172.21.15.134:0/807262340 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:19.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.643+0000 7fa593400640 1 -- 172.21.15.134:0/807262340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871e0390 msgr2=0x55f0875c87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:19.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.643+0000 7fa593400640 1 --2- 172.21.15.134:0/807262340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871e0390 0x55f0875c87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.643+0000 7fa593400640 1 -- 172.21.15.134:0/807262340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f0871d81b0 msgr2=0x55f0875cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:19.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.643+0000 7fa593400640 1 --2- 172.21.15.134:0/807262340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f0871d81b0 0x55f0875cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.643+0000 7fa593400640 1 -- 172.21.15.134:0/807262340 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0875c95c0 con 0x55f087201070 2024-07-26T11:26:19.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.644+0000 7fa593400640 1 --2- 172.21.15.134:0/807262340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f087201070 0x55f0875c8e30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa580007590 tx=0x7fa5800b3a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f65e9f15a2ffc1 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:19.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.644+0000 7fa573e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:26:19.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.645+0000 7fa573e00640 1 -- 172.21.15.134:0/807262340 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5800a6a80 con 0x55f087201070 2024-07-26T11:26:19.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.645+0000 7fa573e00640 1 -- 172.21.15.134:0/807262340 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9599+0+0 (secure 0 0 0) 0x7fa5800a71b0 con 0x55f087201070 2024-07-26T11:26:19.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.649+0000 7fa5a01dd740 1 -- 172.21.15.134:0/807262340 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5800b7020 con 0x55f087201070 2024-07-26T11:26:19.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.649+0000 7fa573e00640 1 -- 172.21.15.134:0/807262340 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa5800b3c70 con 0x55f087201070 2024-07-26T11:26:19.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.649+0000 7fa5a01dd740 10 client.26864 fetch_fsmap learned FSMap version 32 2024-07-26T11:26:19.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.649+0000 7fa5a01dd740 1 -- 172.21.15.134:0/807262340 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f08763d040 con 0x55f087201070 2024-07-26T11:26:19.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.650+0000 7fa573e00640 1 -- 172.21.15.134:0/807262340 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa5800b7020 con 0x55f087201070 2024-07-26T11:26:19.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.650+0000 7fa573e00640 1 -- 172.21.15.134:0/807262340 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fa5800b0470 con 0x55f087201070 2024-07-26T11:26:19.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.650+0000 7fa5a01dd740 10 client.26864 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:26:19.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.650+0000 7fa5a01dd740 10 client.26864 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.h 2024-07-26T11:26:19.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.650+0000 7fa5a01dd740 20 client.26864 populate_metadata read hostname 'smithi134' 2024-07-26T11:26:19.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.650+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/807262340 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x55f087194da0 0x55f08763f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:19.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.650+0000 7fa5a01dd740 4 client.26864 mds_command: new command op to 24686 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:26:19.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.650+0000 7fa5a01dd740 1 -- 172.21.15.134:0/807262340 --> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f08751a5e0 con 0x55f087194da0 2024-07-26T11:26:19.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.651+0000 7fa592a00640 1 --2- 172.21.15.134:0/807262340 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x55f087194da0 0x55f08763f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:19.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.653+0000 7fa592a00640 1 --2- 172.21.15.134:0/807262340 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x55f087194da0 0x55f08763f5d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d6f92ea626ea1ca1 server_cookie=92497500601d753e in_seq=0 out_seq=0 2024-07-26T11:26:19.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.653+0000 7fa573e00640 10 client.26864 ms_handle_connect on v2:172.21.15.189:6836/1448732093 2024-07-26T11:26:19.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.656+0000 7fa573e00640 1 -- 172.21.15.134:0/807262340 <== mds.4 v2:172.21.15.189:6836/1448732093 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f08751a5e0 con 0x55f087194da0 2024-07-26T11:26:19.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.656+0000 7fa573e00640 10 client.26864 handle_command_reply: tid=0 2024-07-26T11:26:19.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.656+0000 7fa5a01dd740 1 client.26864 shutdown 2024-07-26T11:26:19.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.658+0000 7fa5a01dd740 1 -- 172.21.15.134:0/807262340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f087201070 msgr2=0x55f0875c8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:19.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.658+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/807262340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f087201070 0x55f0875c8e30 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa580007590 tx=0x7fa5800b3a10 comp rx=0 tx=0).stop 2024-07-26T11:26:19.660 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.659+0000 7fa5a01dd740 1 -- 172.21.15.134:0/807262340 shutdown_connections 2024-07-26T11:26:19.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.659+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/807262340 >> [v2:172.21.15.189:6836/1448732093,v1:172.21.15.189:6837/1448732093] conn(0x55f087194da0 0x55f08763f5d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.659+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/807262340 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0871e0390 0x55f0875c87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.659+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/807262340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f087201070 0x55f0875c8e30 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.659+0000 7fa5a01dd740 1 --2- 172.21.15.134:0/807262340 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55f0871d81b0 0x55f0875cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.659+0000 7fa5a01dd740 1 -- 172.21.15.134:0/807262340 >> 172.21.15.134:0/807262340 conn(0x55f0871f5460 msgr2=0x55f0875a7a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:19.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.659+0000 7fa573e00640 0 client.26864 ms_handle_reset on v2:172.21.15.189:6836/1448732093 2024-07-26T11:26:19.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.660+0000 7fa5a01dd740 1 -- 172.21.15.134:0/807262340 shutdown_connections 2024-07-26T11:26:19.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.663+0000 7fa5a01dd740 1 -- 172.21.15.134:0/807262340 wait complete. 2024-07-26T11:26:19.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.663+0000 7fa5a01dd740 20 client.26864 trim_cache size 0 max 16384 2024-07-26T11:26:19.666 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:26:19.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.664+0000 7fa593e00640 1 -- 172.21.15.134:0/878626106 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5780b3dc0 msgr2=0x7fa5780b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:19.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.664+0000 7fa593e00640 1 --2- 172.21.15.134:0/878626106 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5780b3dc0 0x7fa5780b62d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa59407d630 tx=0x7fa58c009400 comp rx=0 tx=0).stop 2024-07-26T11:26:19.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.665+0000 7fa593e00640 1 -- 172.21.15.134:0/878626106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa59410ca30 msgr2=0x7fa59407efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:19.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.665+0000 7fa593e00640 1 --2- 172.21.15.134:0/878626106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa59410ca30 0x7fa59407efb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa58000c710 tx=0x7fa58000cbd0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.665+0000 7fa593e00640 1 -- 172.21.15.134:0/878626106 shutdown_connections 2024-07-26T11:26:19.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.665+0000 7fa593e00640 1 --2- 172.21.15.134:0/878626106 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fa5780b3dc0 0x7fa5780b62d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.667 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.665+0000 7fa593e00640 1 --2- 172.21.15.134:0/878626106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa59410ca30 0x7fa59407efb0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.665+0000 7fa593e00640 1 --2- 172.21.15.134:0/878626106 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fa59410c0e0 0x7fa59407e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.665+0000 7fa593e00640 1 --2- 172.21.15.134:0/878626106 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa594109ec0 0x7fa594085720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:19.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.665+0000 7fa593e00640 1 -- 172.21.15.134:0/878626106 >> 172.21.15.134:0/878626106 conn(0x7fa594078e50 msgr2=0x7fa59407b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:19.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.667+0000 7fa593e00640 1 -- 172.21.15.134:0/878626106 shutdown_connections 2024-07-26T11:26:19.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:19.667+0000 7fa593e00640 1 -- 172.21.15.134:0/878626106 wait complete. 2024-07-26T11:26:20.183 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:26:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:26:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:26:20.839 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:3 damage ls 2024-07-26T11:26:20.959 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:20 smithi134 ceph-mon[31966]: pgmap v800: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 55 GiB data, 118 GiB used, 954 GiB / 1.0 TiB avail; 53 KiB/s rd, 206 MiB/s wr, 1.61k op/s 2024-07-26T11:26:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:20 smithi189 ceph-mon[36790]: pgmap v800: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 55 GiB data, 118 GiB used, 954 GiB / 1.0 TiB avail; 53 KiB/s rd, 206 MiB/s wr, 1.61k op/s 2024-07-26T11:26:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:20 smithi193 ceph-mon[37473]: pgmap v800: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 55 GiB data, 118 GiB used, 954 GiB / 1.0 TiB avail; 53 KiB/s rd, 206 MiB/s wr, 1.61k op/s 2024-07-26T11:26:21.639 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:26:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.069+0000 7fe3fda00640 1 -- 172.21.15.134:0/3055707427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3f810fa60 msgr2=0x7fe3f8111ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.069+0000 7fe3fda00640 1 --2- 172.21.15.134:0/3055707427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3f810fa60 0x7fe3f8111ec0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009900 tx=0x7fe3e402a420 comp rx=0 tx=0).stop 2024-07-26T11:26:22.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.071+0000 7fe3fda00640 1 -- 172.21.15.134:0/3055707427 shutdown_connections 2024-07-26T11:26:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.071+0000 7fe3fda00640 1 --2- 172.21.15.134:0/3055707427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3f8075b30 0x7fe3f810f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.071+0000 7fe3fda00640 1 --2- 172.21.15.134:0/3055707427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3f810fa60 0x7fe3f8111ec0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.071+0000 7fe3fda00640 1 --2- 172.21.15.134:0/3055707427 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3f80751e0 0x7fe3f80755f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.071+0000 7fe3fda00640 1 -- 172.21.15.134:0/3055707427 >> 172.21.15.134:0/3055707427 conn(0x7fe3f80fef50 msgr2=0x7fe3f81013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.072+0000 7fe3fda00640 1 -- 172.21.15.134:0/3055707427 shutdown_connections 2024-07-26T11:26:22.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.072+0000 7fe3fda00640 1 -- 172.21.15.134:0/3055707427 wait complete. 2024-07-26T11:26:22.075 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.073+0000 7fe3fda00640 1 Processor -- start 2024-07-26T11:26:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.075+0000 7fe3fda00640 1 -- start start 2024-07-26T11:26:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.075+0000 7fe3fda00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3f80751e0 0x7fe3f819c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.078 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.075+0000 7fe3fda00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3f8075b30 0x7fe3f819c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.075+0000 7fe3fda00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3f810fa60 0x7fe3f81a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.075+0000 7fe3fda00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f81156e0 con 0x7fe3f8075b30 2024-07-26T11:26:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.075+0000 7fe3fda00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f8115560 con 0x7fe3f810fa60 2024-07-26T11:26:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.075+0000 7fe3fda00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f8115860 con 0x7fe3f80751e0 2024-07-26T11:26:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.077+0000 7fe3f6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3f80751e0 0x7fe3f819c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.077+0000 7fe3f6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3f80751e0 0x7fe3f819c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51028/0 (socket says 172.21.15.134:51028) 2024-07-26T11:26:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.077+0000 7fe3f7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3f810fa60 0x7fe3f81a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.077+0000 7fe3f7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3f810fa60 0x7fe3f81a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:39708/0 (socket says 172.21.15.134:39708) 2024-07-26T11:26:22.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.077+0000 7fe3f7400640 1 -- 172.21.15.134:0/2960218258 learned_addr learned my addr 172.21.15.134:0/2960218258 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.078+0000 7fe3f6000640 1 --2- 172.21.15.134:0/2960218258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3f8075b30 0x7fe3f819c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.078+0000 7fe3f7400640 1 -- 172.21.15.134:0/2960218258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3f80751e0 msgr2=0x7fe3f819c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.078+0000 7fe3f7400640 1 --2- 172.21.15.134:0/2960218258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3f80751e0 0x7fe3f819c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.078+0000 7fe3f7400640 1 -- 172.21.15.134:0/2960218258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3f8075b30 msgr2=0x7fe3f819c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.078+0000 7fe3f7400640 1 --2- 172.21.15.134:0/2960218258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3f8075b30 0x7fe3f819c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.079+0000 7fe3f7400640 1 -- 172.21.15.134:0/2960218258 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f81a1f80 con 0x7fe3f810fa60 2024-07-26T11:26:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.080+0000 7fe3f6000640 1 --2- 172.21.15.134:0/2960218258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3f8075b30 0x7fe3f819c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:26:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.080+0000 7fe3f7400640 1 --2- 172.21.15.134:0/2960218258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3f810fa60 0x7fe3f81a1910 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe3e402a940 tx=0x7fe3e4003cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:22.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.081+0000 7fe3ef400640 1 -- 172.21.15.134:0/2960218258 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e403f070 con 0x7fe3f810fa60 2024-07-26T11:26:22.083 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.081+0000 7fe3ef400640 1 -- 172.21.15.134:0/2960218258 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3e4031d60 con 0x7fe3f810fa60 2024-07-26T11:26:22.083 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.081+0000 7fe3ef400640 1 -- 172.21.15.134:0/2960218258 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4035030 con 0x7fe3f810fa60 2024-07-26T11:26:22.083 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.081+0000 7fe3fda00640 1 -- 172.21.15.134:0/2960218258 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3f81a2210 con 0x7fe3f810fa60 2024-07-26T11:26:22.083 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.081+0000 7fe3fda00640 1 -- 172.21.15.134:0/2960218258 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3f81a2730 con 0x7fe3f810fa60 2024-07-26T11:26:22.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.088+0000 7fe404975740 1 Processor -- start 2024-07-26T11:26:22.092 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.091+0000 7fe3ef400640 1 -- 172.21.15.134:0/2960218258 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fe3e4042070 con 0x7fe3f810fa60 2024-07-26T11:26:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.091+0000 7fe404975740 1 -- start start 2024-07-26T11:26:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.091+0000 7fe404975740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf1a71b0 0x558ccf54aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.091+0000 7fe404975740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf17e2f0 0x558ccf54afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.091+0000 7fe404975740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf1864d0 0x558ccf54b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.091+0000 7fe404975740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x558ccf4bf270 con 0x558ccf1a71b0 2024-07-26T11:26:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.091+0000 7fe404975740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558ccf4ae770 con 0x558ccf1864d0 2024-07-26T11:26:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.091+0000 7fe404975740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558ccf4b27b0 con 0x558ccf17e2f0 2024-07-26T11:26:22.094 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.092+0000 7fe3f6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf17e2f0 0x558ccf54afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.095 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:26:22.096 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:26:22.096 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:26:22.096 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.092+0000 7fe3f6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf17e2f0 0x558ccf54afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51040/0 (socket says 172.21.15.134:51040) 2024-07-26T11:26:22.096 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.092+0000 7fe3f6a00640 1 -- 172.21.15.134:0/1663997372 learned_addr learned my addr 172.21.15.134:0/1663997372 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:22.096 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.093+0000 7fe3f6000640 1 --2- 172.21.15.134:0/1663997372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf1864d0 0x558ccf54b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.096 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.093+0000 7fe3ef400640 1 --2- 172.21.15.134:0/2960218258 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe3dc0b3f10 0x7fe3dc0b6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.094+0000 7fe3f6000640 1 --2- 172.21.15.134:0/1663997372 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf1a71b0 0x558ccf54aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.094+0000 7fe3f6000640 1 -- 172.21.15.134:0/1663997372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf17e2f0 msgr2=0x558ccf54afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.094+0000 7fe3f6000640 1 --2- 172.21.15.134:0/1663997372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf17e2f0 0x558ccf54afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.094+0000 7fe3f6000640 1 -- 172.21.15.134:0/1663997372 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf1a71b0 msgr2=0x558ccf54aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.094+0000 7fe3f6000640 1 --2- 172.21.15.134:0/1663997372 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf1a71b0 0x558ccf54aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.094+0000 7fe3f6000640 1 -- 172.21.15.134:0/1663997372 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ccf212790 con 0x558ccf1864d0 2024-07-26T11:26:22.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.094+0000 7fe3f6000640 1 --2- 172.21.15.134:0/1663997372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf1864d0 0x558ccf54b4e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8077200 tx=0x7fe3e800c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f097c36734f8565 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.095+0000 7fe3e3e00640 1 -- 172.21.15.134:0/1663997372 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e800f040 con 0x558ccf1864d0 2024-07-26T11:26:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.095+0000 7fe3e3e00640 1 -- 172.21.15.134:0/1663997372 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3e80044d0 con 0x558ccf1864d0 2024-07-26T11:26:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.095+0000 7fe3e3e00640 1 -- 172.21.15.134:0/1663997372 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8003e40 con 0x558ccf1864d0 2024-07-26T11:26:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.097+0000 7fe3ef400640 1 -- 172.21.15.134:0/2960218258 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9599+0+0 (secure 0 0 0) 0x7fe3e4104a40 con 0x7fe3f810fa60 2024-07-26T11:26:22.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.098+0000 7fe3f7400640 1 --2- 172.21.15.134:0/2960218258 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe3dc0b3f10 0x7fe3dc0b6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.098+0000 7fe404975740 1 -- 172.21.15.134:0/1663997372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf1864d0 msgr2=0x558ccf54b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:22.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.098+0000 7fe404975740 1 --2- 172.21.15.134:0/1663997372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf1864d0 0x558ccf54b4e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8077200 tx=0x7fe3e800c350 comp rx=0 tx=0).stop 2024-07-26T11:26:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.099+0000 7fe404975740 1 -- 172.21.15.134:0/1663997372 shutdown_connections 2024-07-26T11:26:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.099+0000 7fe404975740 1 --2- 172.21.15.134:0/1663997372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf17e2f0 0x558ccf54afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.099+0000 7fe404975740 1 --2- 172.21.15.134:0/1663997372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf1864d0 0x558ccf54b4e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.099+0000 7fe404975740 1 --2- 172.21.15.134:0/1663997372 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf1a71b0 0x558ccf54aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.099+0000 7fe3f7400640 1 --2- 172.21.15.134:0/2960218258 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe3dc0b3f10 0x7fe3dc0b6420 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4005ab0 tx=0x7fe3e4031f10 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:22.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.099+0000 7fe404975740 1 -- 172.21.15.134:0/1663997372 >> 172.21.15.134:0/1663997372 conn(0x558ccf19b300 msgr2=0x558ccf548950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:22.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.100+0000 7fe404975740 1 -- 172.21.15.134:0/1663997372 shutdown_connections 2024-07-26T11:26:22.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.100+0000 7fe404975740 1 -- 172.21.15.134:0/1663997372 wait complete. 2024-07-26T11:26:22.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.101+0000 7fe404975740 1 -- start start 2024-07-26T11:26:22.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.102+0000 7fe404975740 1 Processor -- start 2024-07-26T11:26:22.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.103+0000 7fe404975740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf17e2f0 0x558ccf5566c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.103+0000 7fe404975740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf1864d0 0x558ccf556c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.104+0000 7fe3f6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf17e2f0 0x558ccf5566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.104+0000 7fe3f6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf17e2f0 0x558ccf5566c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:39728/0 (socket says 172.21.15.134:39728) 2024-07-26T11:26:22.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.104+0000 7fe404975740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1a71b0 0x558ccf562500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.104+0000 7fe404975740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x558ccf3e8ab0 con 0x558ccf1864d0 2024-07-26T11:26:22.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.104+0000 7fe404975740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558ccf44bd50 con 0x558ccf17e2f0 2024-07-26T11:26:22.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.104+0000 7fe404975740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x558ccf5591e0 con 0x558ccf1a71b0 2024-07-26T11:26:22.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.104+0000 7fe404975740 10 client.0 _ll_register_callbacks cb 0x558ccf1bea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:26:22.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.104+0000 7fe3f6000640 1 -- 172.21.15.134:0/3963600566 learned_addr learned my addr 172.21.15.134:0/3963600566 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:22.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.105+0000 7fe3f6a00640 1 --2- 172.21.15.134:0/3963600566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf1864d0 0x558ccf556c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.105+0000 7fe3f6000640 1 --2- 172.21.15.134:0/3963600566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1a71b0 0x558ccf562500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.105+0000 7fe3f6000640 1 -- 172.21.15.134:0/3963600566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1a71b0 msgr2=0x558ccf562500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:22.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.105+0000 7fe3f6000640 1 --2- 172.21.15.134:0/3963600566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1a71b0 0x558ccf562500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.105+0000 7fe3f6000640 1 -- 172.21.15.134:0/3963600566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf1864d0 msgr2=0x558ccf556c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.105+0000 7fe3f6000640 1 --2- 172.21.15.134:0/3963600566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf1864d0 0x558ccf556c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.105+0000 7fe3f6000640 1 -- 172.21.15.134:0/3963600566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ccf28b620 con 0x558ccf17e2f0 2024-07-26T11:26:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.106+0000 7fe3f6000640 1 --2- 172.21.15.134:0/3963600566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf17e2f0 0x558ccf5566c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8003a60 tx=0x7fe3e800c4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50d79bedb31ef4ac server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:22.108 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.106+0000 7fe3e0c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:26:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.106+0000 7fe3e0c00640 1 -- 172.21.15.134:0/3963600566 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e802a060 con 0x558ccf17e2f0 2024-07-26T11:26:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.107+0000 7fe3e0c00640 1 -- 172.21.15.134:0/3963600566 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9599+0+0 (secure 0 0 0) 0x7fe3e8003dd0 con 0x558ccf17e2f0 2024-07-26T11:26:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.107+0000 7fe404975740 1 -- 172.21.15.134:0/3963600566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3e80044d0 con 0x558ccf17e2f0 2024-07-26T11:26:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.107+0000 7fe3e0c00640 1 -- 172.21.15.134:0/3963600566 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe3e8003fd0 con 0x558ccf17e2f0 2024-07-26T11:26:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.107+0000 7fe404975740 10 client.26876 fetch_fsmap learned FSMap version 32 2024-07-26T11:26:22.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.107+0000 7fe404975740 1 -- 172.21.15.134:0/3963600566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558ccf56dae0 con 0x558ccf17e2f0 2024-07-26T11:26:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.108+0000 7fe3e0c00640 1 -- 172.21.15.134:0/3963600566 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3e80044d0 con 0x558ccf17e2f0 2024-07-26T11:26:22.110 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.108+0000 7fe3e0c00640 1 -- 172.21.15.134:0/3963600566 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fe3e8022040 con 0x558ccf17e2f0 2024-07-26T11:26:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.108+0000 7fe404975740 10 client.26876 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:26:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.110+0000 7fe404975740 10 client.26876 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:26:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.110+0000 7fe404975740 20 client.26876 populate_metadata read hostname 'smithi134' 2024-07-26T11:26:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.110+0000 7fe404975740 1 --2- 172.21.15.134:0/3963600566 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558ccf165d80 0x558ccf570f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.110+0000 7fe404975740 4 client.26876 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:26:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.110+0000 7fe404975740 1 -- 172.21.15.134:0/3963600566 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ccf4c11c0 con 0x558ccf165d80 2024-07-26T11:26:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.112+0000 7fe3f6a00640 1 --2- 172.21.15.134:0/3963600566 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558ccf165d80 0x558ccf570f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.115 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.113+0000 7fe3f6a00640 1 --2- 172.21.15.134:0/3963600566 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558ccf165d80 0x558ccf570f00 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=932baa24e2b28033 server_cookie=640319bcff12d677 in_seq=0 out_seq=0 2024-07-26T11:26:22.115 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.113+0000 7fe3e0c00640 10 client.26876 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:26:22.117 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.115+0000 7fe3e0c00640 1 -- 172.21.15.134:0/3963600566 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x558ccf4c11c0 con 0x558ccf165d80 2024-07-26T11:26:22.117 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.116+0000 7fe3e0c00640 10 client.26876 handle_command_reply: tid=0 2024-07-26T11:26:22.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.116+0000 7fe404975740 1 client.26876 shutdown 2024-07-26T11:26:22.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.116+0000 7fe404975740 1 -- 172.21.15.134:0/3963600566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf17e2f0 msgr2=0x558ccf5566c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:22.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.117+0000 7fe404975740 1 --2- 172.21.15.134:0/3963600566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf17e2f0 0x558ccf5566c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8003a60 tx=0x7fe3e800c4f0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.117+0000 7fe404975740 1 -- 172.21.15.134:0/3963600566 shutdown_connections 2024-07-26T11:26:22.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.117+0000 7fe404975740 1 --2- 172.21.15.134:0/3963600566 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558ccf165d80 0x558ccf570f00 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.117+0000 7fe404975740 1 --2- 172.21.15.134:0/3963600566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1a71b0 0x558ccf562500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.117+0000 7fe404975740 1 --2- 172.21.15.134:0/3963600566 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf1864d0 0x558ccf556c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.117+0000 7fe404975740 1 --2- 172.21.15.134:0/3963600566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf17e2f0 0x558ccf5566c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.117+0000 7fe3e0c00640 0 client.26876 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:26:22.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.117+0000 7fe404975740 1 -- 172.21.15.134:0/3963600566 >> 172.21.15.134:0/3963600566 conn(0x558ccf19b300 msgr2=0x558ccf5470d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:22.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.118+0000 7fe404975740 1 -- 172.21.15.134:0/3963600566 shutdown_connections 2024-07-26T11:26:22.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.118+0000 7fe404975740 1 -- 172.21.15.134:0/3963600566 wait complete. 2024-07-26T11:26:22.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.118+0000 7fe404975740 20 client.26876 trim_cache size 0 max 16384 2024-07-26T11:26:22.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.152+0000 7fe404975740 1 Processor -- start 2024-07-26T11:26:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.155+0000 7fe404975740 1 -- start start 2024-07-26T11:26:22.157 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.155+0000 7fe404975740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf17e2f0 0x558ccf562470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.155+0000 7fe404975740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1864d0 0x558ccf548860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.155+0000 7fe404975740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf1a71b0 0x558ccf573f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.155+0000 7fe404975740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f8072620 con 0x558ccf17e2f0 2024-07-26T11:26:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.155+0000 7fe404975740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f8074160 con 0x558ccf1a71b0 2024-07-26T11:26:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.155+0000 7fe404975740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f8072cf0 con 0x558ccf1864d0 2024-07-26T11:26:22.158 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.156+0000 7fe3f6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1864d0 0x558ccf548860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.159 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.156+0000 7fe3f6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf1a71b0 0x558ccf573f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.159 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.156+0000 7fe3f6000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf17e2f0 0x558ccf562470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.159 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.156+0000 7fe3f6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1864d0 0x558ccf548860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51048/0 (socket says 172.21.15.134:51048) 2024-07-26T11:26:22.159 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.156+0000 7fe3f6a00640 1 -- 172.21.15.134:0/935462794 learned_addr learned my addr 172.21.15.134:0/935462794 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:22.159 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.156+0000 7fe3f6000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf17e2f0 0x558ccf562470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:48182/0 (socket says 172.21.15.134:48182) 2024-07-26T11:26:22.159 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.157+0000 7fe3f6000640 1 -- 172.21.15.134:0/935462794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1864d0 msgr2=0x558ccf548860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:22.159 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.157+0000 7fe3f6000640 1 --2- 172.21.15.134:0/935462794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1864d0 0x558ccf548860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.159 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.157+0000 7fe3f6000640 1 -- 172.21.15.134:0/935462794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf1a71b0 msgr2=0x558ccf573f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:22.159 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.157+0000 7fe3f6000640 1 --2- 172.21.15.134:0/935462794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf1a71b0 0x558ccf573f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.157+0000 7fe3f6000640 1 -- 172.21.15.134:0/935462794 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ccf54cf90 con 0x558ccf17e2f0 2024-07-26T11:26:22.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.158+0000 7fe3f6000640 1 --2- 172.21.15.134:0/935462794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf17e2f0 0x558ccf562470 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe3e800d010 tx=0x7fe3e8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8c447e3b9a000f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:22.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.159+0000 7fe3e3400640 1 -- 172.21.15.134:0/935462794 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e802a070 con 0x558ccf17e2f0 2024-07-26T11:26:22.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.159+0000 7fe3e3400640 1 -- 172.21.15.134:0/935462794 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3e8006b90 con 0x558ccf17e2f0 2024-07-26T11:26:22.161 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.159+0000 7fe3e3400640 1 -- 172.21.15.134:0/935462794 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e800c040 con 0x558ccf17e2f0 2024-07-26T11:26:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.160+0000 7fe404975740 1 -- 172.21.15.134:0/935462794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf17e2f0 msgr2=0x558ccf562470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:22.162 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.160+0000 7fe404975740 1 --2- 172.21.15.134:0/935462794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf17e2f0 0x558ccf562470 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe3e800d010 tx=0x7fe3e8000bc0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.161+0000 7fe404975740 1 -- 172.21.15.134:0/935462794 shutdown_connections 2024-07-26T11:26:22.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.161+0000 7fe404975740 1 --2- 172.21.15.134:0/935462794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1864d0 0x558ccf548860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.161+0000 7fe404975740 1 --2- 172.21.15.134:0/935462794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf1a71b0 0x558ccf573f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.161+0000 7fe404975740 1 --2- 172.21.15.134:0/935462794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf17e2f0 0x558ccf562470 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.161+0000 7fe404975740 1 -- 172.21.15.134:0/935462794 >> 172.21.15.134:0/935462794 conn(0x558ccf19b300 msgr2=0x558ccf548150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:22.164 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.162+0000 7fe404975740 1 -- 172.21.15.134:0/935462794 shutdown_connections 2024-07-26T11:26:22.164 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.162+0000 7fe404975740 1 -- 172.21.15.134:0/935462794 wait complete. 2024-07-26T11:26:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.163+0000 7fe404975740 1 -- start start 2024-07-26T11:26:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.163+0000 7fe404975740 1 Processor -- start 2024-07-26T11:26:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.164+0000 7fe404975740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf17e2f0 0x558ccf5742b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.164+0000 7fe404975740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1864d0 0x558ccf574850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.164+0000 7fe404975740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf1a71b0 0x558ccf5d1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.164+0000 7fe404975740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f8073a90 con 0x558ccf1a71b0 2024-07-26T11:26:22.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.164+0000 7fe404975740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f80733c0 con 0x558ccf17e2f0 2024-07-26T11:26:22.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.164+0000 7fe404975740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f8070620 con 0x558ccf1864d0 2024-07-26T11:26:22.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.164+0000 7fe404975740 10 client.0 _ll_register_callbacks cb 0x7fe3f00079b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:26:22.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.165+0000 7fe3f6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1864d0 0x558ccf574850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.165+0000 7fe3f6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1864d0 0x558ccf574850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:51052/0 (socket says 172.21.15.134:51052) 2024-07-26T11:26:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.165+0000 7fe3f6a00640 1 -- 172.21.15.134:0/704812863 learned_addr learned my addr 172.21.15.134:0/704812863 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:22.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.165+0000 7fe3f6000640 1 --2- 172.21.15.134:0/704812863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf1a71b0 0x558ccf5d1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.170+0000 7fe3f6000640 1 --2- 172.21.15.134:0/704812863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf17e2f0 0x558ccf5742b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.171+0000 7fe3f6000640 1 -- 172.21.15.134:0/704812863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1864d0 msgr2=0x558ccf574850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.171+0000 7fe3f6000640 1 --2- 172.21.15.134:0/704812863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1864d0 0x558ccf574850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.173 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.171+0000 7fe3f6000640 1 -- 172.21.15.134:0/704812863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf17e2f0 msgr2=0x558ccf5742b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.171+0000 7fe3f6000640 1 --2- 172.21.15.134:0/704812863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf17e2f0 0x558ccf5742b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.171+0000 7fe3f6000640 1 -- 172.21.15.134:0/704812863 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ccf5d2580 con 0x558ccf1a71b0 2024-07-26T11:26:22.174 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.172+0000 7fe3f6a00640 1 --2- 172.21.15.134:0/704812863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1864d0 0x558ccf574850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:26:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.173+0000 7fe3f6000640 1 --2- 172.21.15.134:0/704812863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf1a71b0 0x558ccf5d1df0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8024cb0 tx=0x7fe3e800c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f668abbdca27add0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:22.175 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.173+0000 7fe3e0c00640 10 client.0 ms_handle_connect on v2:172.21.15.134:3300/0 2024-07-26T11:26:22.177 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.175+0000 7fe3e0c00640 1 -- 172.21.15.134:0/704812863 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e803e030 con 0x558ccf1a71b0 2024-07-26T11:26:22.177 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.175+0000 7fe3e0c00640 1 -- 172.21.15.134:0/704812863 <== mon.0 v2:172.21.15.134:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 9599+0+0 (secure 0 0 0) 0x7fe3e8003dd0 con 0x558ccf1a71b0 2024-07-26T11:26:22.177 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.175+0000 7fe404975740 1 -- 172.21.15.134:0/704812863 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe3e8006b90 con 0x558ccf1a71b0 2024-07-26T11:26:22.178 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.176+0000 7fe3e0c00640 1 -- 172.21.15.134:0/704812863 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe3e802a040 con 0x558ccf1a71b0 2024-07-26T11:26:22.178 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.176+0000 7fe404975740 10 client.16215 fetch_fsmap learned FSMap version 32 2024-07-26T11:26:22.178 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.176+0000 7fe404975740 1 -- 172.21.15.134:0/704812863 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x558ccf573b60 con 0x558ccf1a71b0 2024-07-26T11:26:22.181 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.179+0000 7fe3e0c00640 1 -- 172.21.15.134:0/704812863 <== mon.0 v2:172.21.15.134:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3e8006b90 con 0x558ccf1a71b0 2024-07-26T11:26:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.179+0000 7fe3e0c00640 1 -- 172.21.15.134:0/704812863 <== mon.0 v2:172.21.15.134:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fe3e8015070 con 0x558ccf1a71b0 2024-07-26T11:26:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.179+0000 7fe404975740 10 client.16215 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:26:22.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.179+0000 7fe404975740 10 client.16215 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:26:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.179+0000 7fe404975740 20 client.16215 populate_metadata read hostname 'smithi134' 2024-07-26T11:26:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.179+0000 7fe404975740 1 --2- 172.21.15.134:0/704812863 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558ccf13b1c0 0x558ccf5e4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.179+0000 7fe404975740 4 client.16215 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:26:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.179+0000 7fe404975740 1 -- 172.21.15.134:0/704812863 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558ccf4bcc60 con 0x558ccf13b1c0 2024-07-26T11:26:22.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.180+0000 7fe3f6a00640 1 --2- 172.21.15.134:0/704812863 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558ccf13b1c0 0x558ccf5e4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.183+0000 7fe3f6a00640 1 --2- 172.21.15.134:0/704812863 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558ccf13b1c0 0x558ccf5e4420 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1569a8f34253e6 server_cookie=9f63d41b2867d17d in_seq=0 out_seq=0 2024-07-26T11:26:22.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.183+0000 7fe3e0c00640 10 client.16215 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:26:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:26:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:26:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:26:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:26:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:26:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:26:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.555+0000 7fe3e0c00640 1 -- 172.21.15.134:0/704812863 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558ccf4bcc60 con 0x558ccf13b1c0 2024-07-26T11:26:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.555+0000 7fe3e0c00640 10 client.16215 handle_command_reply: tid=0 2024-07-26T11:26:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.555+0000 7fe404975740 1 client.16215 shutdown 2024-07-26T11:26:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.555+0000 7fe404975740 1 -- 172.21.15.134:0/704812863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf1a71b0 msgr2=0x558ccf5d1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.555+0000 7fe404975740 1 --2- 172.21.15.134:0/704812863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf1a71b0 0x558ccf5d1df0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8024cb0 tx=0x7fe3e800c440 comp rx=0 tx=0).stop 2024-07-26T11:26:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.557+0000 7fe404975740 1 -- 172.21.15.134:0/704812863 shutdown_connections 2024-07-26T11:26:22.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.557+0000 7fe404975740 1 --2- 172.21.15.134:0/704812863 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x558ccf13b1c0 0x558ccf5e4420 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.557+0000 7fe404975740 1 --2- 172.21.15.134:0/704812863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558ccf1864d0 0x558ccf574850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.557+0000 7fe404975740 1 --2- 172.21.15.134:0/704812863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x558ccf1a71b0 0x558ccf5d1df0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.557+0000 7fe404975740 1 --2- 172.21.15.134:0/704812863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558ccf17e2f0 0x558ccf5742b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.557+0000 7fe404975740 1 -- 172.21.15.134:0/704812863 >> 172.21.15.134:0/704812863 conn(0x558ccf19b300 msgr2=0x558ccf54ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:22.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.557+0000 7fe3e0c00640 0 client.16215 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:26:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.566+0000 7fe404975740 1 -- 172.21.15.134:0/704812863 shutdown_connections 2024-07-26T11:26:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.566+0000 7fe404975740 1 -- 172.21.15.134:0/704812863 wait complete. 2024-07-26T11:26:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.566+0000 7fe404975740 20 client.16215 trim_cache size 0 max 16384 2024-07-26T11:26:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:26:22.569 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:26:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:26:22.570 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:26:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.576+0000 7fe3fda00640 1 -- 172.21.15.134:0/2960218258 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe3dc0b3f10 msgr2=0x7fe3dc0b6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.577+0000 7fe3fda00640 1 --2- 172.21.15.134:0/2960218258 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe3dc0b3f10 0x7fe3dc0b6420 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4005ab0 tx=0x7fe3e4031f10 comp rx=0 tx=0).stop 2024-07-26T11:26:22.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.577+0000 7fe3fda00640 1 -- 172.21.15.134:0/2960218258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3f810fa60 msgr2=0x7fe3f81a1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.577+0000 7fe3fda00640 1 --2- 172.21.15.134:0/2960218258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3f810fa60 0x7fe3f81a1910 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe3e402a940 tx=0x7fe3e4003cd0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.581+0000 7fe3fda00640 1 -- 172.21.15.134:0/2960218258 shutdown_connections 2024-07-26T11:26:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.581+0000 7fe3fda00640 1 --2- 172.21.15.134:0/2960218258 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe3dc0b3f10 0x7fe3dc0b6420 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.581+0000 7fe3fda00640 1 --2- 172.21.15.134:0/2960218258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3f810fa60 0x7fe3f81a1910 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.581+0000 7fe3fda00640 1 --2- 172.21.15.134:0/2960218258 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe3f8075b30 0x7fe3f819c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.581+0000 7fe3fda00640 1 --2- 172.21.15.134:0/2960218258 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3f80751e0 0x7fe3f819c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:22.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.581+0000 7fe3fda00640 1 -- 172.21.15.134:0/2960218258 >> 172.21.15.134:0/2960218258 conn(0x7fe3f80fef50 msgr2=0x7fe3f81074c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:22.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.583+0000 7fe3fda00640 1 -- 172.21.15.134:0/2960218258 shutdown_connections 2024-07-26T11:26:22.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:22.583+0000 7fe3fda00640 1 -- 172.21.15.134:0/2960218258 wait complete. 2024-07-26T11:26:23.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:22 smithi134 ceph-mon[31966]: pgmap v801: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 55 GiB data, 118 GiB used, 954 GiB / 1.0 TiB avail; 55 KiB/s rd, 165 MiB/s wr, 1.33k op/s 2024-07-26T11:26:23.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:22 smithi189 ceph-mon[36790]: pgmap v801: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 55 GiB data, 118 GiB used, 954 GiB / 1.0 TiB avail; 55 KiB/s rd, 165 MiB/s wr, 1.33k op/s 2024-07-26T11:26:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:22 smithi193 ceph-mon[37473]: pgmap v801: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 55 GiB data, 118 GiB used, 954 GiB / 1.0 TiB avail; 55 KiB/s rd, 165 MiB/s wr, 1.33k op/s 2024-07-26T11:26:23.674 INFO:journalctl@ceph.osd.9.smithi134.stdout:Jul 26 11:26:23 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-osd-9[71469]: 2024-07-26T11:26:23.341+0000 7f330b800640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 5 after trimming was started 2024-07-26T11:26:26.177 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:25 smithi134 ceph-mon[31966]: pgmap v802: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 55 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 296 KiB/s rd, 177 MiB/s wr, 1.48k op/s 2024-07-26T11:26:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:25 smithi189 ceph-mon[36790]: pgmap v802: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 55 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 296 KiB/s rd, 177 MiB/s wr, 1.48k op/s 2024-07-26T11:26:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:25 smithi193 ceph-mon[37473]: pgmap v802: 129 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 112 active+clean; 55 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 296 KiB/s rd, 177 MiB/s wr, 1.48k op/s 2024-07-26T11:26:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:27 smithi134 ceph-mon[31966]: pgmap v803: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 54 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 711 KiB/s rd, 161 MiB/s wr, 1.49k op/s 2024-07-26T11:26:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:27 smithi189 ceph-mon[36790]: pgmap v803: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 54 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 711 KiB/s rd, 161 MiB/s wr, 1.49k op/s 2024-07-26T11:26:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:27 smithi193 ceph-mon[37473]: pgmap v803: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 54 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 711 KiB/s rd, 161 MiB/s wr, 1.49k op/s 2024-07-26T11:26:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:28 smithi134 ceph-mon[31966]: pgmap v804: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 54 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 879 KiB/s rd, 165 MiB/s wr, 1.63k op/s 2024-07-26T11:26:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:28 smithi134 ceph-mon[31966]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T11:26:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:28 smithi189 ceph-mon[36790]: pgmap v804: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 54 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 879 KiB/s rd, 165 MiB/s wr, 1.63k op/s 2024-07-26T11:26:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:28 smithi189 ceph-mon[36790]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T11:26:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:28 smithi193 ceph-mon[37473]: pgmap v804: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 54 GiB data, 117 GiB used, 956 GiB / 1.0 TiB avail; 879 KiB/s rd, 165 MiB/s wr, 1.63k op/s 2024-07-26T11:26:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:28 smithi193 ceph-mon[37473]: osdmap e100: 12 total, 12 up, 12 in 2024-07-26T11:26:32.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:31 smithi134 ceph-mon[31966]: pgmap v806: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 54 GiB data, 117 GiB used, 955 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 138 MiB/s wr, 1.64k op/s 2024-07-26T11:26:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:31 smithi189 ceph-mon[36790]: pgmap v806: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 54 GiB data, 117 GiB used, 955 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 138 MiB/s wr, 1.64k op/s 2024-07-26T11:26:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:31 smithi193 ceph-mon[37473]: pgmap v806: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 54 GiB data, 117 GiB used, 955 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 138 MiB/s wr, 1.64k op/s 2024-07-26T11:26:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:33 smithi193 ceph-mon[37473]: pgmap v807: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 55 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 154 MiB/s wr, 1.89k op/s 2024-07-26T11:26:34.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:33 smithi134 ceph-mon[31966]: pgmap v807: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 55 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 154 MiB/s wr, 1.89k op/s 2024-07-26T11:26:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:33 smithi189 ceph-mon[36790]: pgmap v807: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 55 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 1.3 MiB/s rd, 154 MiB/s wr, 1.89k op/s 2024-07-26T11:26:35.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:34 smithi134 ceph-mon[31966]: pgmap v808: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 55 GiB data, 119 GiB used, 954 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 139 MiB/s wr, 1.76k op/s 2024-07-26T11:26:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:34 smithi189 ceph-mon[36790]: pgmap v808: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 55 GiB data, 119 GiB used, 954 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 139 MiB/s wr, 1.76k op/s 2024-07-26T11:26:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:34 smithi193 ceph-mon[37473]: pgmap v808: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 55 GiB data, 119 GiB used, 954 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 139 MiB/s wr, 1.76k op/s 2024-07-26T11:26:35.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.950+0000 7fe629400640 1 -- 172.21.15.134:0/3992938068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe61c098350 msgr2=0x7fe61c098760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:35.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.950+0000 7fe629400640 1 --2- 172.21.15.134:0/3992938068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe61c098350 0x7fe61c098760 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe614009900 tx=0x7fe61402a410 comp rx=0 tx=0).stop 2024-07-26T11:26:35.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.950+0000 7fe629400640 1 -- 172.21.15.134:0/3992938068 shutdown_connections 2024-07-26T11:26:35.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.950+0000 7fe629400640 1 --2- 172.21.15.134:0/3992938068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe61c09e5d0 0x7fe61c0a4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:35.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.950+0000 7fe629400640 1 --2- 172.21.15.134:0/3992938068 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe61c09dc80 0x7fe61c09e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:35.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.950+0000 7fe629400640 1 --2- 172.21.15.134:0/3992938068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe61c098350 0x7fe61c098760 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:35.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.950+0000 7fe629400640 1 -- 172.21.15.134:0/3992938068 >> 172.21.15.134:0/3992938068 conn(0x7fe61c093d90 msgr2=0x7fe61c0961e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.950+0000 7fe629400640 1 -- 172.21.15.134:0/3992938068 shutdown_connections 2024-07-26T11:26:35.954 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.950+0000 7fe629400640 1 -- 172.21.15.134:0/3992938068 wait complete. 2024-07-26T11:26:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.954+0000 7fe629400640 1 Processor -- start 2024-07-26T11:26:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.954+0000 7fe629400640 1 -- start start 2024-07-26T11:26:35.956 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.955+0000 7fe629400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe61c098350 0x7fe61c0a2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:35.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.955+0000 7fe629400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe61c09dc80 0x7fe61c0a27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:35.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.955+0000 7fe629400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe61c09e5d0 0x7fe61c0a2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:35.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.955+0000 7fe629400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe61c0a8340 con 0x7fe61c09dc80 2024-07-26T11:26:35.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.955+0000 7fe629400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe61c0a81c0 con 0x7fe61c09e5d0 2024-07-26T11:26:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.955+0000 7fe629400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe61c0a84c0 con 0x7fe61c098350 2024-07-26T11:26:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.955+0000 7fe623200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe61c09e5d0 0x7fe61c0a2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.955+0000 7fe623200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe61c09e5d0 0x7fe61c0a2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:44630/0 (socket says 172.21.15.134:44630) 2024-07-26T11:26:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.955+0000 7fe623200640 1 -- 172.21.15.134:0/138072366 learned_addr learned my addr 172.21.15.134:0/138072366 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.955+0000 7fe622800640 1 --2- 172.21.15.134:0/138072366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe61c098350 0x7fe61c0a2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.956+0000 7fe623200640 1 -- 172.21.15.134:0/138072366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe61c098350 msgr2=0x7fe61c0a2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.956+0000 7fe623200640 1 --2- 172.21.15.134:0/138072366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe61c098350 0x7fe61c0a2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.956+0000 7fe623200640 1 -- 172.21.15.134:0/138072366 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe61c09dc80 msgr2=0x7fe61c0a27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.956+0000 7fe623200640 1 --2- 172.21.15.134:0/138072366 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe61c09dc80 0x7fe61c0a27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:35.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.956+0000 7fe623200640 1 -- 172.21.15.134:0/138072366 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe61c134a50 con 0x7fe61c09e5d0 2024-07-26T11:26:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.957+0000 7fe623200640 1 --2- 172.21.15.134:0/138072366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe61c09e5d0 0x7fe61c0a2d10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe624000b60 tx=0x7fe62407c6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.957+0000 7fe61b400640 1 -- 172.21.15.134:0/138072366 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe624086600 con 0x7fe61c09e5d0 2024-07-26T11:26:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.958+0000 7fe629400640 1 -- 172.21.15.134:0/138072366 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe61c134d40 con 0x7fe61c09e5d0 2024-07-26T11:26:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.958+0000 7fe629400640 1 -- 172.21.15.134:0/138072366 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe61c135210 con 0x7fe61c09e5d0 2024-07-26T11:26:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.958+0000 7fe61b400640 1 -- 172.21.15.134:0/138072366 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe624086c50 con 0x7fe61c09e5d0 2024-07-26T11:26:35.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.958+0000 7fe61b400640 1 -- 172.21.15.134:0/138072366 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe62407eb70 con 0x7fe61c09e5d0 2024-07-26T11:26:35.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.965+0000 7fe61b400640 1 -- 172.21.15.134:0/138072366 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fe62407ed10 con 0x7fe61c09e5d0 2024-07-26T11:26:35.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.966+0000 7fe63039f740 1 Processor -- start 2024-07-26T11:26:35.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.967+0000 7fe63039f740 1 -- start start 2024-07-26T11:26:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.967+0000 7fe63039f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b454b1b0 0x5649b48eec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.967+0000 7fe63039f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b45222f0 0x5649b48ef180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.967+0000 7fe63039f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48ef6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.967+0000 7fe63039f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5649b4863270 con 0x5649b454b1b0 2024-07-26T11:26:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.967+0000 7fe63039f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5649b4852770 con 0x5649b452a4d0 2024-07-26T11:26:35.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.967+0000 7fe63039f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5649b48567b0 con 0x5649b45222f0 2024-07-26T11:26:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.968+0000 7fe622800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b45222f0 0x5649b48ef180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.968+0000 7fe621e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48ef6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.968+0000 7fe621e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48ef6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:44642/0 (socket says 172.21.15.134:44642) 2024-07-26T11:26:35.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.968+0000 7fe621e00640 1 -- 172.21.15.134:0/3388569448 learned_addr learned my addr 172.21.15.134:0/3388569448 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:35.971 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.968+0000 7fe621e00640 1 --2- 172.21.15.134:0/3388569448 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b454b1b0 0x5649b48eec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.970+0000 7fe621e00640 1 -- 172.21.15.134:0/3388569448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b45222f0 msgr2=0x5649b48ef180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.970+0000 7fe621e00640 1 --2- 172.21.15.134:0/3388569448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b45222f0 0x5649b48ef180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.971+0000 7fe621e00640 1 -- 172.21.15.134:0/3388569448 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b454b1b0 msgr2=0x5649b48eec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.971+0000 7fe621e00640 1 --2- 172.21.15.134:0/3388569448 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b454b1b0 0x5649b48eec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:35.973 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.971+0000 7fe621e00640 1 -- 172.21.15.134:0/3388569448 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649b45b6790 con 0x5649b452a4d0 2024-07-26T11:26:35.974 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.972+0000 7fe621e00640 1 --2- 172.21.15.134:0/3388569448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48ef6c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe61000d800 tx=0x7fe61000b4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=248dad1510866fdd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:35.974 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.972+0000 7fe61b400640 1 --2- 172.21.15.134:0/138072366 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6080b3dc0 0x7fe6080b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:35.975 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.973+0000 7fe622800640 1 --2- 172.21.15.134:0/138072366 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6080b3dc0 0x7fe6080b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.974+0000 7fe60f400640 1 -- 172.21.15.134:0/3388569448 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe610004030 con 0x5649b452a4d0 2024-07-26T11:26:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.974+0000 7fe60f400640 1 -- 172.21.15.134:0/3388569448 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe610004d40 con 0x5649b452a4d0 2024-07-26T11:26:35.976 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.974+0000 7fe60f400640 1 -- 172.21.15.134:0/3388569448 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6100065d0 con 0x5649b452a4d0 2024-07-26T11:26:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.975+0000 7fe61b400640 1 -- 172.21.15.134:0/138072366 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9523+0+0 (secure 0 0 0) 0x7fe6241518b0 con 0x7fe61c09e5d0 2024-07-26T11:26:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.975+0000 7fe63039f740 1 -- 172.21.15.134:0/3388569448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 msgr2=0x5649b48ef6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.975+0000 7fe63039f740 1 --2- 172.21.15.134:0/3388569448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48ef6c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe61000d800 tx=0x7fe61000b4a0 comp rx=0 tx=0).stop 2024-07-26T11:26:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.975+0000 7fe63039f740 1 -- 172.21.15.134:0/3388569448 shutdown_connections 2024-07-26T11:26:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.975+0000 7fe63039f740 1 --2- 172.21.15.134:0/3388569448 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b45222f0 0x5649b48ef180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.975+0000 7fe63039f740 1 --2- 172.21.15.134:0/3388569448 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48ef6c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.976+0000 7fe63039f740 1 --2- 172.21.15.134:0/3388569448 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b454b1b0 0x5649b48eec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.976+0000 7fe63039f740 1 -- 172.21.15.134:0/3388569448 >> 172.21.15.134:0/3388569448 conn(0x5649b453f300 msgr2=0x5649b48ecb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:35.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.976+0000 7fe622800640 1 --2- 172.21.15.134:0/138072366 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6080b3dc0 0x7fe6080b62d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe614005ab0 tx=0x7fe614005a20 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:35.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.976+0000 7fe63039f740 1 -- 172.21.15.134:0/3388569448 shutdown_connections 2024-07-26T11:26:35.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.976+0000 7fe63039f740 1 -- 172.21.15.134:0/3388569448 wait complete. 2024-07-26T11:26:35.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.977+0000 7fe63039f740 1 -- start start 2024-07-26T11:26:35.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.978+0000 7fe63039f740 1 Processor -- start 2024-07-26T11:26:35.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.980+0000 7fe63039f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b45222f0 0x5649b48fa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:35.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.982+0000 7fe622800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b45222f0 0x5649b48fa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:35.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.982+0000 7fe622800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b45222f0 0x5649b48fa870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:49124/0 (socket says 172.21.15.134:49124) 2024-07-26T11:26:35.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.982+0000 7fe622800640 1 -- 172.21.15.134:0/3714716341 learned_addr learned my addr 172.21.15.134:0/3714716341 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:35.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.982+0000 7fe63039f740 1 --2- 172.21.15.134:0/3714716341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48fae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:35.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.982+0000 7fe63039f740 1 --2- 172.21.15.134:0/3714716341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b454b1b0 0x5649b49066b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:35.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.982+0000 7fe621e00640 1 --2- 172.21.15.134:0/3714716341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48fae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:35.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.983+0000 7fe63039f740 1 -- 172.21.15.134:0/3714716341 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5649b478cab0 con 0x5649b454b1b0 2024-07-26T11:26:35.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.983+0000 7fe63039f740 1 -- 172.21.15.134:0/3714716341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5649b47efd50 con 0x5649b452a4d0 2024-07-26T11:26:35.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.983+0000 7fe63039f740 1 -- 172.21.15.134:0/3714716341 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5649b48fd360 con 0x5649b45222f0 2024-07-26T11:26:35.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.983+0000 7fe63039f740 10 client.0 _ll_register_callbacks cb 0x5649b4562a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:26:35.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.984+0000 7fe623200640 1 --2- 172.21.15.134:0/3714716341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b454b1b0 0x5649b49066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:35.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.984+0000 7fe621e00640 1 -- 172.21.15.134:0/3714716341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b45222f0 msgr2=0x5649b48fa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:35.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.984+0000 7fe621e00640 1 --2- 172.21.15.134:0/3714716341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b45222f0 0x5649b48fa870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:35.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.984+0000 7fe621e00640 1 -- 172.21.15.134:0/3714716341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b454b1b0 msgr2=0x5649b49066b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:35.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.984+0000 7fe621e00640 1 --2- 172.21.15.134:0/3714716341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b454b1b0 0x5649b49066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:35.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.984+0000 7fe621e00640 1 -- 172.21.15.134:0/3714716341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649b462f620 con 0x5649b452a4d0 2024-07-26T11:26:35.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.985+0000 7fe621e00640 1 --2- 172.21.15.134:0/3714716341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48fae10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe610018000 tx=0x7fe610004aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b298e825df01515f server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:35.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.985+0000 7fe603e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:26:35.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.990+0000 7fe603e00640 1 -- 172.21.15.134:0/3714716341 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe610020020 con 0x5649b452a4d0 2024-07-26T11:26:35.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.990+0000 7fe603e00640 1 -- 172.21.15.134:0/3714716341 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9523+0+0 (secure 0 0 0) 0x7fe610004030 con 0x5649b452a4d0 2024-07-26T11:26:35.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.990+0000 7fe63039f740 1 -- 172.21.15.134:0/3714716341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe610004d40 con 0x5649b452a4d0 2024-07-26T11:26:35.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.994+0000 7fe603e00640 1 -- 172.21.15.134:0/3714716341 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe610004230 con 0x5649b452a4d0 2024-07-26T11:26:35.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.994+0000 7fe63039f740 10 client.26888 fetch_fsmap learned FSMap version 32 2024-07-26T11:26:35.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.994+0000 7fe63039f740 1 -- 172.21.15.134:0/3714716341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5649b4909450 con 0x5649b452a4d0 2024-07-26T11:26:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.996+0000 7fe603e00640 1 -- 172.21.15.134:0/3714716341 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe610004d40 con 0x5649b452a4d0 2024-07-26T11:26:35.999 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.998+0000 7fe603e00640 1 -- 172.21.15.134:0/3714716341 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fe61002d020 con 0x5649b452a4d0 2024-07-26T11:26:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.998+0000 7fe63039f740 10 client.26888 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:26:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.998+0000 7fe63039f740 10 client.26888 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:26:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.998+0000 7fe63039f740 20 client.26888 populate_metadata read hostname 'smithi134' 2024-07-26T11:26:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.998+0000 7fe63039f740 1 --2- 172.21.15.134:0/3714716341 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5649b4509d80 0x5649b490c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.998+0000 7fe63039f740 4 client.26888 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:26:36.000 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:35.998+0000 7fe63039f740 1 -- 172.21.15.134:0/3714716341 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649b48651c0 con 0x5649b4509d80 2024-07-26T11:26:36.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.001+0000 7fe622800640 1 --2- 172.21.15.134:0/3714716341 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5649b4509d80 0x5649b490c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.002+0000 7fe622800640 1 --2- 172.21.15.134:0/3714716341 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5649b4509d80 0x5649b490c870 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8ce23fec8056400 server_cookie=28bc395397bef9b8 in_seq=0 out_seq=0 2024-07-26T11:26:36.004 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.002+0000 7fe603e00640 10 client.26888 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:26:36.007 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.005+0000 7fe603e00640 1 -- 172.21.15.134:0/3714716341 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x5649b48651c0 con 0x5649b4509d80 2024-07-26T11:26:36.009 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.005+0000 7fe603e00640 10 client.26888 handle_command_reply: tid=0 2024-07-26T11:26:36.009 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.008+0000 7fe63039f740 1 client.26888 shutdown 2024-07-26T11:26:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.008+0000 7fe63039f740 1 -- 172.21.15.134:0/3714716341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 msgr2=0x5649b48fae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.008+0000 7fe63039f740 1 --2- 172.21.15.134:0/3714716341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48fae10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe610018000 tx=0x7fe610004aa0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.009+0000 7fe63039f740 1 -- 172.21.15.134:0/3714716341 shutdown_connections 2024-07-26T11:26:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.009+0000 7fe63039f740 1 --2- 172.21.15.134:0/3714716341 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5649b4509d80 0x5649b490c870 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.009+0000 7fe63039f740 1 --2- 172.21.15.134:0/3714716341 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b454b1b0 0x5649b49066b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.009+0000 7fe63039f740 1 --2- 172.21.15.134:0/3714716341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48fae10 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.009+0000 7fe63039f740 1 --2- 172.21.15.134:0/3714716341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b45222f0 0x5649b48fa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.009+0000 7fe63039f740 1 -- 172.21.15.134:0/3714716341 >> 172.21.15.134:0/3714716341 conn(0x5649b453f300 msgr2=0x5649b48efc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.009+0000 7fe603e00640 0 client.26888 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:26:36.011 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.009+0000 7fe63039f740 1 -- 172.21.15.134:0/3714716341 shutdown_connections 2024-07-26T11:26:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.009+0000 7fe63039f740 1 -- 172.21.15.134:0/3714716341 wait complete. 2024-07-26T11:26:36.012 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.010+0000 7fe63039f740 20 client.26888 trim_cache size 0 max 16384 2024-07-26T11:26:36.055 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.050+0000 7fe63039f740 1 Processor -- start 2024-07-26T11:26:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.053+0000 7fe63039f740 1 -- start start 2024-07-26T11:26:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.053+0000 7fe63039f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b45222f0 0x5649b48f0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.053+0000 7fe63039f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.053+0000 7fe63039f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b454b1b0 0x5649b48fc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.053+0000 7fe63039f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe6040100c0 con 0x5649b45222f0 2024-07-26T11:26:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.053+0000 7fe63039f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe604011c00 con 0x5649b452a4d0 2024-07-26T11:26:36.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.053+0000 7fe63039f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe604010790 con 0x5649b454b1b0 2024-07-26T11:26:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.054+0000 7fe621e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.054+0000 7fe623200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b454b1b0 0x5649b48fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.055+0000 7fe621e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48f4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:44656/0 (socket says 172.21.15.134:44656) 2024-07-26T11:26:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.055+0000 7fe623200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b454b1b0 0x5649b48fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:49136/0 (socket says 172.21.15.134:49136) 2024-07-26T11:26:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.055+0000 7fe623200640 1 -- 172.21.15.134:0/356718175 learned_addr learned my addr 172.21.15.134:0/356718175 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.055+0000 7fe623200640 1 -- 172.21.15.134:0/356718175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 msgr2=0x5649b48f4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:36.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.055+0000 7fe623200640 1 --2- 172.21.15.134:0/356718175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48f4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.056+0000 7fe623200640 1 -- 172.21.15.134:0/356718175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b45222f0 msgr2=0x5649b48f0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:36.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.056+0000 7fe623200640 1 --2- 172.21.15.134:0/356718175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b45222f0 0x5649b48f0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.056+0000 7fe623200640 1 -- 172.21.15.134:0/356718175 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649b490cbe0 con 0x5649b454b1b0 2024-07-26T11:26:36.059 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.057+0000 7fe621e00640 1 --2- 172.21.15.134:0/356718175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48f4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:26:36.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.062+0000 7fe623200640 1 --2- 172.21.15.134:0/356718175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b454b1b0 0x5649b48fc290 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fe624116430 tx=0x7fe624068ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b944775396dd03dd server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:36.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.062+0000 7fe60ea00640 1 -- 172.21.15.134:0/356718175 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe624132070 con 0x5649b454b1b0 2024-07-26T11:26:36.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.062+0000 7fe60ea00640 1 -- 172.21.15.134:0/356718175 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6241198c0 con 0x5649b454b1b0 2024-07-26T11:26:36.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.063+0000 7fe60ea00640 1 -- 172.21.15.134:0/356718175 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe624120da0 con 0x5649b454b1b0 2024-07-26T11:26:36.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.064+0000 7fe63039f740 1 -- 172.21.15.134:0/356718175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b454b1b0 msgr2=0x5649b48fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:36.067 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.064+0000 7fe63039f740 1 --2- 172.21.15.134:0/356718175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b454b1b0 0x5649b48fc290 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fe624116430 tx=0x7fe624068ed0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.067+0000 7fe63039f740 1 -- 172.21.15.134:0/356718175 shutdown_connections 2024-07-26T11:26:36.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.067+0000 7fe63039f740 1 --2- 172.21.15.134:0/356718175 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b454b1b0 0x5649b48fc290 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.067+0000 7fe63039f740 1 --2- 172.21.15.134:0/356718175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b452a4d0 0x5649b48f4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.067+0000 7fe63039f740 1 --2- 172.21.15.134:0/356718175 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b45222f0 0x5649b48f0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.067+0000 7fe63039f740 1 -- 172.21.15.134:0/356718175 >> 172.21.15.134:0/356718175 conn(0x5649b453f300 msgr2=0x5649b48fbb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:36.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.067+0000 7fe63039f740 1 -- 172.21.15.134:0/356718175 shutdown_connections 2024-07-26T11:26:36.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.067+0000 7fe63039f740 1 -- 172.21.15.134:0/356718175 wait complete. 2024-07-26T11:26:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.068+0000 7fe63039f740 1 -- start start 2024-07-26T11:26:36.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.068+0000 7fe63039f740 1 Processor -- start 2024-07-26T11:26:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.069+0000 7fe63039f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b45222f0 0x5649b48fdc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.069+0000 7fe63039f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b452a4d0 0x5649b48fe160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.069+0000 7fe63039f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b454b1b0 0x5649b4972f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.069+0000 7fe63039f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fe604011530 con 0x5649b454b1b0 2024-07-26T11:26:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.069+0000 7fe63039f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe604010e60 con 0x5649b45222f0 2024-07-26T11:26:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.069+0000 7fe63039f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fe60400e0a0 con 0x5649b452a4d0 2024-07-26T11:26:36.071 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.069+0000 7fe63039f740 10 client.0 _ll_register_callbacks cb 0x7fe61000be50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:26:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.069+0000 7fe621e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b452a4d0 0x5649b48fe160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.069+0000 7fe621e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b452a4d0 0x5649b48fe160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:49146/0 (socket says 172.21.15.134:49146) 2024-07-26T11:26:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.069+0000 7fe621e00640 1 -- 172.21.15.134:0/91243585 learned_addr learned my addr 172.21.15.134:0/91243585 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:36.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.070+0000 7fe623200640 1 --2- 172.21.15.134:0/91243585 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b454b1b0 0x5649b4972f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:36.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.070+0000 7fe621e00640 1 -- 172.21.15.134:0/91243585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b45222f0 msgr2=0x5649b48fdc20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:26:36.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.070+0000 7fe621e00640 1 --2- 172.21.15.134:0/91243585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b45222f0 0x5649b48fdc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.070+0000 7fe621e00640 1 -- 172.21.15.134:0/91243585 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b454b1b0 msgr2=0x5649b4972f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:36.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.070+0000 7fe621e00640 1 --2- 172.21.15.134:0/91243585 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b454b1b0 0x5649b4972f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.070+0000 7fe621e00640 1 -- 172.21.15.134:0/91243585 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649b49844f0 con 0x5649b452a4d0 2024-07-26T11:26:36.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.070+0000 7fe621e00640 1 --2- 172.21.15.134:0/91243585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b452a4d0 0x5649b48fe160 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe6100090b0 tx=0x7fe61000d790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56b02326b0e87dcb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:36.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.070+0000 7fe603e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:26:36.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.071+0000 7fe603e00640 1 -- 172.21.15.134:0/91243585 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe610020040 con 0x5649b452a4d0 2024-07-26T11:26:36.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.072+0000 7fe603e00640 1 -- 172.21.15.134:0/91243585 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9523+0+0 (secure 0 0 0) 0x7fe610007dc0 con 0x5649b452a4d0 2024-07-26T11:26:36.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.073+0000 7fe63039f740 1 -- 172.21.15.134:0/91243585 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6241198c0 con 0x5649b452a4d0 2024-07-26T11:26:36.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.075+0000 7fe603e00640 1 -- 172.21.15.134:0/91243585 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe610003d60 con 0x5649b452a4d0 2024-07-26T11:26:36.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.075+0000 7fe63039f740 10 client.26908 fetch_fsmap learned FSMap version 32 2024-07-26T11:26:36.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.075+0000 7fe63039f740 1 -- 172.21.15.134:0/91243585 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5649b4985760 con 0x5649b452a4d0 2024-07-26T11:26:36.079 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.076+0000 7fe603e00640 1 -- 172.21.15.134:0/91243585 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6241198c0 con 0x5649b452a4d0 2024-07-26T11:26:36.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.078+0000 7fe603e00640 1 -- 172.21.15.134:0/91243585 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fe61002d040 con 0x5649b452a4d0 2024-07-26T11:26:36.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.078+0000 7fe63039f740 10 client.26908 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:26:36.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.078+0000 7fe63039f740 10 client.26908 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:26:36.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.078+0000 7fe63039f740 20 client.26908 populate_metadata read hostname 'smithi134' 2024-07-26T11:26:36.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.078+0000 7fe63039f740 1 --2- 172.21.15.134:0/91243585 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5649b44df1c0 0x5649b4987ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:36.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.078+0000 7fe63039f740 4 client.26908 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:26:36.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.078+0000 7fe63039f740 1 -- 172.21.15.134:0/91243585 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5649b4860c60 con 0x5649b44df1c0 2024-07-26T11:26:36.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.079+0000 7fe622800640 1 --2- 172.21.15.134:0/91243585 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5649b44df1c0 0x5649b4987ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:36.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.083+0000 7fe603e00640 10 client.26908 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:26:36.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.083+0000 7fe622800640 1 --2- 172.21.15.134:0/91243585 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5649b44df1c0 0x5649b4987ee0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45b3e37d336a073a server_cookie=3d70d550ffc6debc in_seq=0 out_seq=0 2024-07-26T11:26:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.658+0000 7fe603e00640 1 -- 172.21.15.134:0/91243585 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5649b4860c60 con 0x5649b44df1c0 2024-07-26T11:26:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.658+0000 7fe603e00640 10 client.26908 handle_command_reply: tid=0 2024-07-26T11:26:36.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.658+0000 7fe63039f740 1 client.26908 shutdown 2024-07-26T11:26:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.658+0000 7fe63039f740 1 -- 172.21.15.134:0/91243585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b452a4d0 msgr2=0x5649b48fe160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:36.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.659+0000 7fe63039f740 1 --2- 172.21.15.134:0/91243585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b452a4d0 0x5649b48fe160 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe6100090b0 tx=0x7fe61000d790 comp rx=0 tx=0).stop 2024-07-26T11:26:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.659+0000 7fe63039f740 1 -- 172.21.15.134:0/91243585 shutdown_connections 2024-07-26T11:26:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.660+0000 7fe63039f740 1 --2- 172.21.15.134:0/91243585 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5649b44df1c0 0x5649b4987ee0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.660+0000 7fe603e00640 0 client.26908 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:26:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.660+0000 7fe63039f740 1 --2- 172.21.15.134:0/91243585 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5649b452a4d0 0x5649b48fe160 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.660+0000 7fe63039f740 1 --2- 172.21.15.134:0/91243585 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5649b454b1b0 0x5649b4972f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.660+0000 7fe63039f740 1 --2- 172.21.15.134:0/91243585 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649b45222f0 0x5649b48fdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.664 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.660+0000 7fe63039f740 1 -- 172.21.15.134:0/91243585 >> 172.21.15.134:0/91243585 conn(0x5649b453f300 msgr2=0x5649b48f03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.663+0000 7fe63039f740 1 -- 172.21.15.134:0/91243585 shutdown_connections 2024-07-26T11:26:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.663+0000 7fe63039f740 1 -- 172.21.15.134:0/91243585 wait complete. 2024-07-26T11:26:36.665 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.664+0000 7fe63039f740 20 client.26908 trim_cache size 0 max 16384 2024-07-26T11:26:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:26:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:26:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:26:36.666 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:26:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.665+0000 7fe629400640 1 -- 172.21.15.134:0/138072366 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6080b3dc0 msgr2=0x7fe6080b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.665+0000 7fe629400640 1 --2- 172.21.15.134:0/138072366 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6080b3dc0 0x7fe6080b62d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe614005ab0 tx=0x7fe614005a20 comp rx=0 tx=0).stop 2024-07-26T11:26:36.667 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.665+0000 7fe629400640 1 -- 172.21.15.134:0/138072366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe61c09e5d0 msgr2=0x7fe61c0a2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.665+0000 7fe629400640 1 --2- 172.21.15.134:0/138072366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe61c09e5d0 0x7fe61c0a2d10 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe624000b60 tx=0x7fe62407c6f0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.666+0000 7fe629400640 1 -- 172.21.15.134:0/138072366 shutdown_connections 2024-07-26T11:26:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.666+0000 7fe629400640 1 --2- 172.21.15.134:0/138072366 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fe6080b3dc0 0x7fe6080b62d0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.666+0000 7fe629400640 1 --2- 172.21.15.134:0/138072366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe61c09e5d0 0x7fe61c0a2d10 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.666+0000 7fe629400640 1 --2- 172.21.15.134:0/138072366 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fe61c09dc80 0x7fe61c0a27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.666+0000 7fe629400640 1 --2- 172.21.15.134:0/138072366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe61c098350 0x7fe61c0a2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.666+0000 7fe629400640 1 -- 172.21.15.134:0/138072366 >> 172.21.15.134:0/138072366 conn(0x7fe61c093d90 msgr2=0x7fe61c0961b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.668+0000 7fe629400640 1 -- 172.21.15.134:0/138072366 shutdown_connections 2024-07-26T11:26:36.670 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:36.668+0000 7fe629400640 1 -- 172.21.15.134:0/138072366 wait complete. 2024-07-26T11:26:37.088 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:36 smithi134 ceph-mon[31966]: pgmap v809: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 55 GiB data, 119 GiB used, 954 GiB / 1.0 TiB avail; 853 KiB/s rd, 143 MiB/s wr, 1.73k op/s 2024-07-26T11:26:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:36 smithi189 ceph-mon[36790]: pgmap v809: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 55 GiB data, 119 GiB used, 954 GiB / 1.0 TiB avail; 853 KiB/s rd, 143 MiB/s wr, 1.73k op/s 2024-07-26T11:26:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:36 smithi193 ceph-mon[37473]: pgmap v809: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 55 GiB data, 119 GiB used, 954 GiB / 1.0 TiB avail; 853 KiB/s rd, 143 MiB/s wr, 1.73k op/s 2024-07-26T11:26:37.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.880+0000 7fdb52600640 1 -- 172.21.15.134:0/3821007374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb4c1035f0 msgr2=0x7fdb4c103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:37.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.882+0000 7fdb52600640 1 --2- 172.21.15.134:0/3821007374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb4c1035f0 0x7fdb4c103a00 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009900 tx=0x7fdb3402a440 comp rx=0 tx=0).stop 2024-07-26T11:26:37.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.883+0000 7fdb52600640 1 -- 172.21.15.134:0/3821007374 shutdown_connections 2024-07-26T11:26:37.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.883+0000 7fdb52600640 1 --2- 172.21.15.134:0/3821007374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb4c109830 0x7fdb4c110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.885 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.883+0000 7fdb52600640 1 --2- 172.21.15.134:0/3821007374 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb4c108ee0 0x7fdb4c1092f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.883+0000 7fdb52600640 1 --2- 172.21.15.134:0/3821007374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb4c1035f0 0x7fdb4c103a00 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.883+0000 7fdb52600640 1 -- 172.21.15.134:0/3821007374 >> 172.21.15.134:0/3821007374 conn(0x7fdb4c07b340 msgr2=0x7fdb4c07b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:37.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.884+0000 7fdb52600640 1 -- 172.21.15.134:0/3821007374 shutdown_connections 2024-07-26T11:26:37.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.884+0000 7fdb52600640 1 -- 172.21.15.134:0/3821007374 wait complete. 2024-07-26T11:26:37.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.887+0000 7fdb52600640 1 Processor -- start 2024-07-26T11:26:37.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.887+0000 7fdb52600640 1 -- start start 2024-07-26T11:26:37.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.889+0000 7fdb52600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb4c1035f0 0x7fdb4c19c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:37.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.889+0000 7fdb52600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb4c108ee0 0x7fdb4c19c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:37.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.889+0000 7fdb52600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb4c109830 0x7fdb4c1a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:37.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.890+0000 7fdb4b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb4c1035f0 0x7fdb4c19c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:37.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.890+0000 7fdb4b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb4c1035f0 0x7fdb4c19c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:44672/0 (socket says 172.21.15.134:44672) 2024-07-26T11:26:37.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.893+0000 7fdb4aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb4c108ee0 0x7fdb4c19c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:37.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.893+0000 7fdb4aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb4c108ee0 0x7fdb4c19c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:49162/0 (socket says 172.21.15.134:49162) 2024-07-26T11:26:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.898+0000 7fdb4b400640 1 -- 172.21.15.134:0/1303067305 learned_addr learned my addr 172.21.15.134:0/1303067305 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.898+0000 7fdb52600640 1 -- 172.21.15.134:0/1303067305 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4c115780 con 0x7fdb4c109830 2024-07-26T11:26:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.898+0000 7fdb52600640 1 -- 172.21.15.134:0/1303067305 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4c115600 con 0x7fdb4c1035f0 2024-07-26T11:26:37.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.898+0000 7fdb52600640 1 -- 172.21.15.134:0/1303067305 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4c115900 con 0x7fdb4c108ee0 2024-07-26T11:26:37.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.899+0000 7fdb4aa00640 1 -- 172.21.15.134:0/1303067305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb4c1035f0 msgr2=0x7fdb4c19c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:37.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.899+0000 7fdb4aa00640 1 --2- 172.21.15.134:0/1303067305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb4c1035f0 0x7fdb4c19c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.899+0000 7fdb4aa00640 1 -- 172.21.15.134:0/1303067305 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb4c109830 msgr2=0x7fdb4c1a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:37.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.899+0000 7fdb4aa00640 1 --2- 172.21.15.134:0/1303067305 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb4c109830 0x7fdb4c1a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.899+0000 7fdb4aa00640 1 -- 172.21.15.134:0/1303067305 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb4c1a1f40 con 0x7fdb4c108ee0 2024-07-26T11:26:37.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.900+0000 7fdb4aa00640 1 --2- 172.21.15.134:0/1303067305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb4c108ee0 0x7fdb4c19c630 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c06b560 tx=0x7fdb3c00cdd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:37.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.903+0000 7fdb43e00640 1 -- 172.21.15.134:0/1303067305 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c007bb0 con 0x7fdb4c108ee0 2024-07-26T11:26:37.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.904+0000 7fdb43e00640 1 -- 172.21.15.134:0/1303067305 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb3c007d50 con 0x7fdb4c108ee0 2024-07-26T11:26:37.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.904+0000 7fdb52600640 1 -- 172.21.15.134:0/1303067305 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb4c1a2230 con 0x7fdb4c108ee0 2024-07-26T11:26:37.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.905+0000 7fdb52600640 1 -- 172.21.15.134:0/1303067305 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb4c1a8a50 con 0x7fdb4c108ee0 2024-07-26T11:26:37.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.905+0000 7fdb43e00640 1 -- 172.21.15.134:0/1303067305 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c0186f0 con 0x7fdb4c108ee0 2024-07-26T11:26:37.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.911+0000 7fdb43e00640 1 -- 172.21.15.134:0/1303067305 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fdb3c018890 con 0x7fdb4c108ee0 2024-07-26T11:26:37.917 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.915+0000 7fdb43e00640 1 --2- 172.21.15.134:0/1303067305 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdb300b3eb0 0x7fdb300b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:37.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.916+0000 7fdb43e00640 1 -- 172.21.15.134:0/1303067305 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 9523+0+0 (secure 0 0 0) 0x7fdb3c0e42f0 con 0x7fdb4c108ee0 2024-07-26T11:26:37.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.917+0000 7fdb4b400640 1 --2- 172.21.15.134:0/1303067305 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdb300b3eb0 0x7fdb300b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:37.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.918+0000 7fdb4b400640 1 --2- 172.21.15.134:0/1303067305 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdb300b3eb0 0x7fdb300b63c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdb34000c00 tx=0x7fdb3402a940 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:37.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.935+0000 7fdb5926f740 1 Processor -- start 2024-07-26T11:26:37.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.937+0000 7fdb5926f740 1 -- start start 2024-07-26T11:26:37.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.938+0000 7fdb5926f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445c9070 0x557b4496e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:37.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.938+0000 7fdb5926f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a01b0 0x557b4496e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:37.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.938+0000 7fdb5926f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445a8390 0x557b4496eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:37.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.938+0000 7fdb4aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a01b0 0x557b4496e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:37.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.938+0000 7fdb4aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a01b0 0x557b4496e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:49176/0 (socket says 172.21.15.134:49176) 2024-07-26T11:26:37.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.938+0000 7fdb4b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445c9070 0x557b4496e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:37.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.939+0000 7fdb4b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445c9070 0x557b4496e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:42602/0 (socket says 172.21.15.134:42602) 2024-07-26T11:26:37.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.940+0000 7fdb4aa00640 1 -- 172.21.15.134:0/1197258626 learned_addr learned my addr 172.21.15.134:0/1197258626 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:37.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.940+0000 7fdb4be00640 1 --2- 172.21.15.134:0/1197258626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445a8390 0x557b4496eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:37.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.941+0000 7fdb5926f740 1 -- 172.21.15.134:0/1197258626 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x557b448e4bf0 con 0x557b445c9070 2024-07-26T11:26:37.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.941+0000 7fdb5926f740 1 -- 172.21.15.134:0/1197258626 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557b448d40f0 con 0x557b445a8390 2024-07-26T11:26:37.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.941+0000 7fdb5926f740 1 -- 172.21.15.134:0/1197258626 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557b448d8130 con 0x557b445a01b0 2024-07-26T11:26:37.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.942+0000 7fdb4be00640 1 -- 172.21.15.134:0/1197258626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a01b0 msgr2=0x557b4496e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:37.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.942+0000 7fdb4be00640 1 --2- 172.21.15.134:0/1197258626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a01b0 0x557b4496e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.942+0000 7fdb4be00640 1 -- 172.21.15.134:0/1197258626 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445c9070 msgr2=0x557b4496e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:37.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.942+0000 7fdb4be00640 1 --2- 172.21.15.134:0/1197258626 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445c9070 0x557b4496e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.942+0000 7fdb4be00640 1 -- 172.21.15.134:0/1197258626 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b44634650 con 0x557b445a8390 2024-07-26T11:26:37.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.943+0000 7fdb4b400640 1 --2- 172.21.15.134:0/1197258626 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445c9070 0x557b4496e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-07-26T11:26:37.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.946+0000 7fdb4be00640 1 --2- 172.21.15.134:0/1197258626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445a8390 0x557b4496eac0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fdb44009cc0 tx=0x7fdb4400e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39e1faadde9a3ef8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:37.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.949+0000 7fdb40c00640 1 -- 172.21.15.134:0/1197258626 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44016070 con 0x557b445a8390 2024-07-26T11:26:37.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.949+0000 7fdb40c00640 1 -- 172.21.15.134:0/1197258626 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb44005020 con 0x557b445a8390 2024-07-26T11:26:37.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.949+0000 7fdb40c00640 1 -- 172.21.15.134:0/1197258626 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb440116d0 con 0x557b445a8390 2024-07-26T11:26:37.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.950+0000 7fdb5926f740 1 -- 172.21.15.134:0/1197258626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445a8390 msgr2=0x557b4496eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:37.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.950+0000 7fdb5926f740 1 --2- 172.21.15.134:0/1197258626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445a8390 0x557b4496eac0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fdb44009cc0 tx=0x7fdb4400e6c0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.951+0000 7fdb5926f740 1 -- 172.21.15.134:0/1197258626 shutdown_connections 2024-07-26T11:26:37.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.951+0000 7fdb5926f740 1 --2- 172.21.15.134:0/1197258626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a01b0 0x557b4496e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.951+0000 7fdb5926f740 1 --2- 172.21.15.134:0/1197258626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445a8390 0x557b4496eac0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.951+0000 7fdb5926f740 1 --2- 172.21.15.134:0/1197258626 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445c9070 0x557b4496e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.951+0000 7fdb5926f740 1 -- 172.21.15.134:0/1197258626 >> 172.21.15.134:0/1197258626 conn(0x557b445bd460 msgr2=0x557b4496bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:37.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.952+0000 7fdb5926f740 1 -- 172.21.15.134:0/1197258626 shutdown_connections 2024-07-26T11:26:37.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.952+0000 7fdb5926f740 1 -- 172.21.15.134:0/1197258626 wait complete. 2024-07-26T11:26:37.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.953+0000 7fdb5926f740 1 -- start start 2024-07-26T11:26:37.955 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.953+0000 7fdb5926f740 1 Processor -- start 2024-07-26T11:26:37.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.955+0000 7fdb5926f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445a01b0 0x557b44981d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:37.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.955+0000 7fdb5926f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a8390 0x557b44982290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:37.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.956+0000 7fdb5926f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445c9070 0x557b449827d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:37.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.955+0000 7fdb4b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445a01b0 0x557b44981d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:37.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.956+0000 7fdb4b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445a01b0 0x557b44981d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:42610/0 (socket says 172.21.15.134:42610) 2024-07-26T11:26:37.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.956+0000 7fdb4aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a8390 0x557b44982290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:37.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.956+0000 7fdb4aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a8390 0x557b44982290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:49192/0 (socket says 172.21.15.134:49192) 2024-07-26T11:26:37.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.957+0000 7fdb5926f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x557b44737700 con 0x557b445a01b0 2024-07-26T11:26:37.959 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.957+0000 7fdb5926f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557b448706c0 con 0x557b445c9070 2024-07-26T11:26:37.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.957+0000 7fdb4be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445c9070 0x557b449827d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:37.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.957+0000 7fdb4b400640 1 -- 172.21.15.134:0/582050276 learned_addr learned my addr 172.21.15.134:0/582050276 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:37.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.957+0000 7fdb5926f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x557b44984f50 con 0x557b445a8390 2024-07-26T11:26:37.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.958+0000 7fdb5926f740 10 client.0 _ll_register_callbacks cb 0x557b44880170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:26:37.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.958+0000 7fdb4be00640 1 -- 172.21.15.134:0/582050276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a8390 msgr2=0x557b44982290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:37.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.958+0000 7fdb4be00640 1 --2- 172.21.15.134:0/582050276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a8390 0x557b44982290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.958+0000 7fdb4be00640 1 -- 172.21.15.134:0/582050276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445a01b0 msgr2=0x557b44981d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:37.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.958+0000 7fdb4be00640 1 --2- 172.21.15.134:0/582050276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445a01b0 0x557b44981d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.958+0000 7fdb4be00640 1 -- 172.21.15.134:0/582050276 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b44887860 con 0x557b445c9070 2024-07-26T11:26:37.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.960+0000 7fdb4be00640 1 --2- 172.21.15.134:0/582050276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445c9070 0x557b449827d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fdb44004a80 tx=0x7fdb44009fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfdd47d851bdf3ca server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:37.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.960+0000 7fdb39600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:26:37.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.962+0000 7fdb39600640 1 -- 172.21.15.134:0/582050276 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44016040 con 0x557b445c9070 2024-07-26T11:26:37.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.962+0000 7fdb5926f740 1 -- 172.21.15.134:0/582050276 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb44005020 con 0x557b445c9070 2024-07-26T11:26:37.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.962+0000 7fdb39600640 1 -- 172.21.15.134:0/582050276 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9523+0+0 (secure 0 0 0) 0x7fdb440053e0 con 0x557b445c9070 2024-07-26T11:26:37.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.963+0000 7fdb39600640 1 -- 172.21.15.134:0/582050276 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdb44016040 con 0x557b445c9070 2024-07-26T11:26:37.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.964+0000 7fdb5926f740 10 client.26903 fetch_fsmap learned FSMap version 32 2024-07-26T11:26:37.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.964+0000 7fdb5926f740 1 -- 172.21.15.134:0/582050276 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557b44990a70 con 0x557b445c9070 2024-07-26T11:26:37.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.964+0000 7fdb39600640 1 -- 172.21.15.134:0/582050276 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb44005020 con 0x557b445c9070 2024-07-26T11:26:37.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.964+0000 7fdb39600640 1 -- 172.21.15.134:0/582050276 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fdb4400f040 con 0x557b445c9070 2024-07-26T11:26:37.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.965+0000 7fdb5926f740 10 client.26903 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:26:37.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.965+0000 7fdb5926f740 10 client.26903 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T11:26:37.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.965+0000 7fdb5926f740 20 client.26903 populate_metadata read hostname 'smithi134' 2024-07-26T11:26:37.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.965+0000 7fdb5926f740 1 --2- 172.21.15.134:0/582050276 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x557b445879f0 0x557b44993e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:37.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.965+0000 7fdb5926f740 4 client.26903 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:26:37.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.965+0000 7fdb5926f740 1 -- 172.21.15.134:0/582050276 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b448e6b40 con 0x557b445879f0 2024-07-26T11:26:37.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.967+0000 7fdb4b400640 1 --2- 172.21.15.134:0/582050276 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x557b445879f0 0x557b44993e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:37.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.968+0000 7fdb4b400640 1 --2- 172.21.15.134:0/582050276 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x557b445879f0 0x557b44993e30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=85ecc5675f7719b0 server_cookie=981dd1913ade7023 in_seq=0 out_seq=0 2024-07-26T11:26:37.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.968+0000 7fdb39600640 10 client.26903 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:26:37.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.969+0000 7fdb39600640 1 -- 172.21.15.134:0/582050276 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x557b448e6b40 con 0x557b445879f0 2024-07-26T11:26:37.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.969+0000 7fdb39600640 10 client.26903 handle_command_reply: tid=0 2024-07-26T11:26:37.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.969+0000 7fdb5926f740 1 client.26903 shutdown 2024-07-26T11:26:37.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.971+0000 7fdb5926f740 1 -- 172.21.15.134:0/582050276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445c9070 msgr2=0x557b449827d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:37.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.971+0000 7fdb5926f740 1 --2- 172.21.15.134:0/582050276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445c9070 0x557b449827d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fdb44004a80 tx=0x7fdb44009fc0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.971+0000 7fdb5926f740 1 -- 172.21.15.134:0/582050276 shutdown_connections 2024-07-26T11:26:37.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.971+0000 7fdb5926f740 1 --2- 172.21.15.134:0/582050276 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x557b445879f0 0x557b44993e30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.973 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.971+0000 7fdb5926f740 1 --2- 172.21.15.134:0/582050276 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a8390 0x557b44982290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.971+0000 7fdb5926f740 1 --2- 172.21.15.134:0/582050276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445c9070 0x557b449827d0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.971+0000 7fdb5926f740 1 --2- 172.21.15.134:0/582050276 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445a01b0 0x557b44981d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:37.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.971+0000 7fdb39600640 0 client.26903 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:26:37.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.971+0000 7fdb5926f740 1 -- 172.21.15.134:0/582050276 >> 172.21.15.134:0/582050276 conn(0x557b445bd460 msgr2=0x557b4496a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:37.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.972+0000 7fdb5926f740 1 -- 172.21.15.134:0/582050276 shutdown_connections 2024-07-26T11:26:37.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.972+0000 7fdb5926f740 1 -- 172.21.15.134:0/582050276 wait complete. 2024-07-26T11:26:37.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:37.972+0000 7fdb5926f740 20 client.26903 trim_cache size 0 max 16384 2024-07-26T11:26:38.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.009+0000 7fdb5926f740 1 Processor -- start 2024-07-26T11:26:38.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.009+0000 7fdb5926f740 1 -- start start 2024-07-26T11:26:38.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.010+0000 7fdb5926f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445a01b0 0x557b44983270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:38.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.011+0000 7fdb5926f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a8390 0x557b4498e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:38.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.011+0000 7fdb5926f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445c9070 0x557b44999460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:38.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.011+0000 7fdb5926f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fdb28010140 con 0x557b445a01b0 2024-07-26T11:26:38.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.011+0000 7fdb5926f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdb28011c80 con 0x557b445c9070 2024-07-26T11:26:38.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.011+0000 7fdb5926f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdb28010810 con 0x557b445a8390 2024-07-26T11:26:38.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.011+0000 7fdb4b400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445a01b0 0x557b44983270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:38.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.011+0000 7fdb4aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a8390 0x557b4498e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:38.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.012+0000 7fdb4aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a8390 0x557b4498e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:49194/0 (socket says 172.21.15.134:49194) 2024-07-26T11:26:38.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.012+0000 7fdb4be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445c9070 0x557b44999460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:38.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.012+0000 7fdb4be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445c9070 0x557b44999460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:44710/0 (socket says 172.21.15.134:44710) 2024-07-26T11:26:38.014 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.012+0000 7fdb4be00640 1 -- 172.21.15.134:0/3238016277 learned_addr learned my addr 172.21.15.134:0/3238016277 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:38.015 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.013+0000 7fdb4be00640 1 -- 172.21.15.134:0/3238016277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a8390 msgr2=0x557b4498e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:38.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.013+0000 7fdb4be00640 1 --2- 172.21.15.134:0/3238016277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a8390 0x557b4498e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:38.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.013+0000 7fdb4be00640 1 -- 172.21.15.134:0/3238016277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445a01b0 msgr2=0x557b44983270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:38.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.013+0000 7fdb4be00640 1 --2- 172.21.15.134:0/3238016277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445a01b0 0x557b44983270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:38.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.013+0000 7fdb4be00640 1 -- 172.21.15.134:0/3238016277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b44983010 con 0x557b445c9070 2024-07-26T11:26:38.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.014+0000 7fdb4be00640 1 --2- 172.21.15.134:0/3238016277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445c9070 0x557b44999460 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdb44011e00 tx=0x7fdb44000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ac74d6c5a3b960b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:38.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.014+0000 7fdb3be00640 1 -- 172.21.15.134:0/3238016277 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44009860 con 0x557b445c9070 2024-07-26T11:26:38.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.014+0000 7fdb3be00640 1 -- 172.21.15.134:0/3238016277 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb44027070 con 0x557b445c9070 2024-07-26T11:26:38.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.015+0000 7fdb3be00640 1 -- 172.21.15.134:0/3238016277 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44009860 con 0x557b445c9070 2024-07-26T11:26:38.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.018+0000 7fdb5926f740 1 -- 172.21.15.134:0/3238016277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445c9070 msgr2=0x557b44999460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:38.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.018+0000 7fdb5926f740 1 --2- 172.21.15.134:0/3238016277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445c9070 0x557b44999460 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdb44011e00 tx=0x7fdb44000f40 comp rx=0 tx=0).stop 2024-07-26T11:26:38.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.018+0000 7fdb5926f740 1 -- 172.21.15.134:0/3238016277 shutdown_connections 2024-07-26T11:26:38.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.018+0000 7fdb5926f740 1 --2- 172.21.15.134:0/3238016277 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a8390 0x557b4498e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:38.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.018+0000 7fdb5926f740 1 --2- 172.21.15.134:0/3238016277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445c9070 0x557b44999460 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:38.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.018+0000 7fdb5926f740 1 --2- 172.21.15.134:0/3238016277 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445a01b0 0x557b44983270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:38.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.018+0000 7fdb5926f740 1 -- 172.21.15.134:0/3238016277 >> 172.21.15.134:0/3238016277 conn(0x557b445bd460 msgr2=0x557b4496fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:38.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.018+0000 7fdb5926f740 1 -- 172.21.15.134:0/3238016277 shutdown_connections 2024-07-26T11:26:38.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.018+0000 7fdb5926f740 1 -- 172.21.15.134:0/3238016277 wait complete. 2024-07-26T11:26:38.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.019+0000 7fdb5926f740 1 -- start start 2024-07-26T11:26:38.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.020+0000 7fdb5926f740 1 Processor -- start 2024-07-26T11:26:38.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.021+0000 7fdb5926f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a01b0 0x557b4498a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:38.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.021+0000 7fdb5926f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445a8390 0x557b44985d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:38.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.021+0000 7fdb5926f740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445c9070 0x557b449862d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:38.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.021+0000 7fdb5926f740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fdb280115b0 con 0x557b445c9070 2024-07-26T11:26:38.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.021+0000 7fdb5926f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdb28010ee0 con 0x557b445a8390 2024-07-26T11:26:38.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.021+0000 7fdb5926f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fdb2800e120 con 0x557b445a01b0 2024-07-26T11:26:38.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.021+0000 7fdb4b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a01b0 0x557b4498a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:38.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.022+0000 7fdb4b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a01b0 0x557b4498a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:49198/0 (socket says 172.21.15.134:49198) 2024-07-26T11:26:38.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.022+0000 7fdb5926f740 10 client.0 _ll_register_callbacks cb 0x7fdb44004e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:26:38.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.022+0000 7fdb4aa00640 1 --2- 172.21.15.134:0/4270337790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445a8390 0x557b44985d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:38.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.022+0000 7fdb4be00640 1 --2- 172.21.15.134:0/4270337790 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445c9070 0x557b449862d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:38.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.022+0000 7fdb4b400640 1 -- 172.21.15.134:0/4270337790 learned_addr learned my addr 172.21.15.134:0/4270337790 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:38.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.026+0000 7fdb4aa00640 1 -- 172.21.15.134:0/4270337790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a01b0 msgr2=0x557b4498a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:38.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.027+0000 7fdb4aa00640 1 --2- 172.21.15.134:0/4270337790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a01b0 0x557b4498a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:38.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.027+0000 7fdb4aa00640 1 -- 172.21.15.134:0/4270337790 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445c9070 msgr2=0x557b449862d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:38.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.027+0000 7fdb4aa00640 1 --2- 172.21.15.134:0/4270337790 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445c9070 0x557b449862d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:38.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.027+0000 7fdb4aa00640 1 -- 172.21.15.134:0/4270337790 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b44986a40 con 0x557b445a8390 2024-07-26T11:26:38.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.027+0000 7fdb4b400640 1 --2- 172.21.15.134:0/4270337790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a01b0 0x557b4498a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:26:38.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.028+0000 7fdb4aa00640 1 --2- 172.21.15.134:0/4270337790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445a8390 0x557b44985d90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c0a6b30 tx=0x7fdb3c0aa500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d86d4886fbb77a83 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:38.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.028+0000 7fdb39600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:26:38.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.031+0000 7fdb39600640 1 -- 172.21.15.134:0/4270337790 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c0e1a80 con 0x557b445a8390 2024-07-26T11:26:38.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.031+0000 7fdb39600640 1 -- 172.21.15.134:0/4270337790 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(100..100 src has 1..100) ==== 9523+0+0 (secure 0 0 0) 0x7fdb3c0a70a0 con 0x557b445a8390 2024-07-26T11:26:38.033 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.032+0000 7fdb5926f740 1 -- 172.21.15.134:0/4270337790 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb44027070 con 0x557b445a8390 2024-07-26T11:26:38.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.032+0000 7fdb39600640 1 -- 172.21.15.134:0/4270337790 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fdb3c0cb050 con 0x557b445a8390 2024-07-26T11:26:38.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.032+0000 7fdb5926f740 10 client.26909 fetch_fsmap learned FSMap version 32 2024-07-26T11:26:38.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.032+0000 7fdb5926f740 1 -- 172.21.15.134:0/4270337790 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x557b449fc390 con 0x557b445a8390 2024-07-26T11:26:38.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.033+0000 7fdb39600640 1 -- 172.21.15.134:0/4270337790 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb44027070 con 0x557b445a8390 2024-07-26T11:26:38.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.034+0000 7fdb39600640 1 -- 172.21.15.134:0/4270337790 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 32) ==== 8370+0+0 (secure 0 0 0) 0x7fdb3c018b80 con 0x557b445a8390 2024-07-26T11:26:38.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.034+0000 7fdb5926f740 10 client.26909 fetch_fsmap finished waiting for FSMap version 32 2024-07-26T11:26:38.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.034+0000 7fdb5926f740 10 client.26909 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-07-26T11:26:38.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.034+0000 7fdb5926f740 20 client.26909 populate_metadata read hostname 'smithi134' 2024-07-26T11:26:38.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.034+0000 7fdb5926f740 1 --2- 172.21.15.134:0/4270337790 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x557b4455cda0 0x557b449feb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:38.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.034+0000 7fdb5926f740 4 client.26909 mds_command: new command op to 24688 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:26:38.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.034+0000 7fdb5926f740 1 -- 172.21.15.134:0/4270337790 --> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557b448e25e0 con 0x557b4455cda0 2024-07-26T11:26:38.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.034+0000 7fdb4b400640 1 --2- 172.21.15.134:0/4270337790 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x557b4455cda0 0x557b449feb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:38.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.035+0000 7fdb39600640 10 client.26909 ms_handle_connect on v2:172.21.15.193:6836/1968168008 2024-07-26T11:26:38.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.036+0000 7fdb4b400640 1 --2- 172.21.15.134:0/4270337790 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x557b4455cda0 0x557b449feb30 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3234c66e04cc5c98 server_cookie=1e8a6888278071f0 in_seq=0 out_seq=0 2024-07-26T11:26:38.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.039+0000 7fdb39600640 1 -- 172.21.15.134:0/4270337790 <== mds.3 v2:172.21.15.193:6836/1968168008 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557b448e25e0 con 0x557b4455cda0 2024-07-26T11:26:38.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.039+0000 7fdb39600640 10 client.26909 handle_command_reply: tid=0 2024-07-26T11:26:38.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.039+0000 7fdb5926f740 1 client.26909 shutdown 2024-07-26T11:26:38.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.039+0000 7fdb5926f740 1 -- 172.21.15.134:0/4270337790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445a8390 msgr2=0x557b44985d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:38.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.039+0000 7fdb5926f740 1 --2- 172.21.15.134:0/4270337790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445a8390 0x557b44985d90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c0a6b30 tx=0x7fdb3c0aa500 comp rx=0 tx=0).stop 2024-07-26T11:26:38.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.039+0000 7fdb5926f740 1 -- 172.21.15.134:0/4270337790 shutdown_connections 2024-07-26T11:26:38.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.039+0000 7fdb5926f740 1 --2- 172.21.15.134:0/4270337790 >> [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] conn(0x557b4455cda0 0x557b449feb30 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:38.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.039+0000 7fdb5926f740 1 --2- 172.21.15.134:0/4270337790 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x557b445c9070 0x557b449862d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:38.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.039+0000 7fdb5926f740 1 --2- 172.21.15.134:0/4270337790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b445a8390 0x557b44985d90 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:38.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.039+0000 7fdb5926f740 1 --2- 172.21.15.134:0/4270337790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557b445a01b0 0x557b4498a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:38.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.039+0000 7fdb39600640 0 client.26909 ms_handle_reset on v2:172.21.15.193:6836/1968168008 2024-07-26T11:26:38.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.039+0000 7fdb5926f740 1 -- 172.21.15.134:0/4270337790 >> 172.21.15.134:0/4270337790 conn(0x557b445bd460 msgr2=0x557b449824c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:38.042 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:26:38.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.040+0000 7fdb5926f740 1 -- 172.21.15.134:0/4270337790 shutdown_connections 2024-07-26T11:26:38.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.040+0000 7fdb5926f740 1 -- 172.21.15.134:0/4270337790 wait complete. 2024-07-26T11:26:38.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.040+0000 7fdb5926f740 20 client.26909 trim_cache size 0 max 16384 2024-07-26T11:26:38.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.041+0000 7fdb52600640 1 -- 172.21.15.134:0/1303067305 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdb300b3eb0 msgr2=0x7fdb300b63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:38.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.041+0000 7fdb52600640 1 --2- 172.21.15.134:0/1303067305 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdb300b3eb0 0x7fdb300b63c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fdb34000c00 tx=0x7fdb3402a940 comp rx=0 tx=0).stop 2024-07-26T11:26:38.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.041+0000 7fdb52600640 1 -- 172.21.15.134:0/1303067305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb4c108ee0 msgr2=0x7fdb4c19c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:38.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.041+0000 7fdb52600640 1 --2- 172.21.15.134:0/1303067305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb4c108ee0 0x7fdb4c19c630 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c06b560 tx=0x7fdb3c00cdd0 comp rx=0 tx=0).stop 2024-07-26T11:26:38.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.042+0000 7fdb52600640 1 -- 172.21.15.134:0/1303067305 shutdown_connections 2024-07-26T11:26:38.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.042+0000 7fdb52600640 1 --2- 172.21.15.134:0/1303067305 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fdb300b3eb0 0x7fdb300b63c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:38.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.042+0000 7fdb52600640 1 --2- 172.21.15.134:0/1303067305 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdb4c108ee0 0x7fdb4c19c630 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:38.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.042+0000 7fdb52600640 1 --2- 172.21.15.134:0/1303067305 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fdb4c109830 0x7fdb4c1a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:38.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.042+0000 7fdb52600640 1 --2- 172.21.15.134:0/1303067305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb4c1035f0 0x7fdb4c19c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:38.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.042+0000 7fdb52600640 1 -- 172.21.15.134:0/1303067305 >> 172.21.15.134:0/1303067305 conn(0x7fdb4c07b340 msgr2=0x7fdb4c106f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:38.044 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.042+0000 7fdb52600640 1 -- 172.21.15.134:0/1303067305 shutdown_connections 2024-07-26T11:26:38.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:38.042+0000 7fdb52600640 1 -- 172.21.15.134:0/1303067305 wait complete. 2024-07-26T11:26:38.918 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:38 smithi134 ceph-mon[31966]: pgmap v810: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 55 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 1003 KiB/s rd, 165 MiB/s wr, 1.87k op/s 2024-07-26T11:26:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:38 smithi189 ceph-mon[36790]: pgmap v810: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 55 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 1003 KiB/s rd, 165 MiB/s wr, 1.87k op/s 2024-07-26T11:26:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:38 smithi193 ceph-mon[37473]: pgmap v810: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 55 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 1003 KiB/s rd, 165 MiB/s wr, 1.87k op/s 2024-07-26T11:26:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:40 smithi134 ceph-mon[31966]: pgmap v811: 129 pgs: 129 active+clean; 56 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 169 MiB/s wr, 1.85k op/s 2024-07-26T11:26:41.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:40 smithi189 ceph-mon[36790]: pgmap v811: 129 pgs: 129 active+clean; 56 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 169 MiB/s wr, 1.85k op/s 2024-07-26T11:26:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:40 smithi193 ceph-mon[37473]: pgmap v811: 129 pgs: 129 active+clean; 56 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 1.2 MiB/s rd, 169 MiB/s wr, 1.85k op/s 2024-07-26T11:26:43.065 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:2 damage ls 2024-07-26T11:26:43.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:42 smithi134 ceph-mon[31966]: pgmap v812: 129 pgs: 129 active+clean; 56 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 155 MiB/s wr, 1.61k op/s 2024-07-26T11:26:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:42 smithi189 ceph-mon[36790]: pgmap v812: 129 pgs: 129 active+clean; 56 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 155 MiB/s wr, 1.61k op/s 2024-07-26T11:26:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:42 smithi193 ceph-mon[37473]: pgmap v812: 129 pgs: 129 active+clean; 56 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 155 MiB/s wr, 1.61k op/s 2024-07-26T11:26:43.963 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:26:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:45 smithi134 ceph-mon[31966]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T11:26:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:45 smithi134 ceph-mon[31966]: pgmap v814: 129 pgs: 129 active+clean; 56 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 177 MiB/s wr, 1.73k op/s 2024-07-26T11:26:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:45 smithi189 ceph-mon[36790]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T11:26:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:45 smithi189 ceph-mon[36790]: pgmap v814: 129 pgs: 129 active+clean; 56 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 177 MiB/s wr, 1.73k op/s 2024-07-26T11:26:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:45 smithi193 ceph-mon[37473]: osdmap e101: 12 total, 12 up, 12 in 2024-07-26T11:26:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:45 smithi193 ceph-mon[37473]: pgmap v814: 129 pgs: 129 active+clean; 56 GiB data, 121 GiB used, 952 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 177 MiB/s wr, 1.73k op/s 2024-07-26T11:26:47.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:47 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:26:47.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:47 smithi134 ceph-mon[31966]: pgmap v815: 129 pgs: 129 active+clean; 57 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 194 MiB/s wr, 1.89k op/s 2024-07-26T11:26:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:47 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:26:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:47 smithi189 ceph-mon[36790]: pgmap v815: 129 pgs: 129 active+clean; 57 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 194 MiB/s wr, 1.89k op/s 2024-07-26T11:26:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:47 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:26:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:47 smithi193 ceph-mon[37473]: pgmap v815: 129 pgs: 129 active+clean; 57 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 194 MiB/s wr, 1.89k op/s 2024-07-26T11:26:49.035 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:48 smithi134 ceph-mon[31966]: pgmap v816: 129 pgs: 129 active+clean; 57 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 184 MiB/s wr, 1.75k op/s 2024-07-26T11:26:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:48 smithi193 ceph-mon[37473]: pgmap v816: 129 pgs: 129 active+clean; 57 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 184 MiB/s wr, 1.75k op/s 2024-07-26T11:26:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:48 smithi189 ceph-mon[36790]: pgmap v816: 129 pgs: 129 active+clean; 57 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 184 MiB/s wr, 1.75k op/s 2024-07-26T11:26:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.122+0000 7fba7fa00640 1 -- 172.21.15.134:0/1359501632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba78109fd0 msgr2=0x7fba7810a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.122+0000 7fba7fa00640 1 --2- 172.21.15.134:0/1359501632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba78109fd0 0x7fba7810a3e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fba6c009900 tx=0x7fba6c02a440 comp rx=0 tx=0).stop 2024-07-26T11:26:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.125+0000 7fba7fa00640 1 -- 172.21.15.134:0/1359501632 shutdown_connections 2024-07-26T11:26:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.125+0000 7fba7fa00640 1 --2- 172.21.15.134:0/1359501632 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba78115fd0 0x7fba781184b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.125+0000 7fba7fa00640 1 --2- 172.21.15.134:0/1359501632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7810a920 0x7fba78115a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.125+0000 7fba7fa00640 1 --2- 172.21.15.134:0/1359501632 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba78109fd0 0x7fba7810a3e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.125+0000 7fba7fa00640 1 -- 172.21.15.134:0/1359501632 >> 172.21.15.134:0/1359501632 conn(0x7fba78070800 msgr2=0x7fba78070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:50.127 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.125+0000 7fba7fa00640 1 -- 172.21.15.134:0/1359501632 shutdown_connections 2024-07-26T11:26:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.125+0000 7fba7fa00640 1 -- 172.21.15.134:0/1359501632 wait complete. 2024-07-26T11:26:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.129+0000 7fba7fa00640 1 Processor -- start 2024-07-26T11:26:50.132 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.130+0000 7fba7fa00640 1 -- start start 2024-07-26T11:26:50.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.131+0000 7fba7fa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba78109fd0 0x7fba781a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.132+0000 7fba7fa00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba7810a920 0x7fba781a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.132+0000 7fba7d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba78109fd0 0x7fba781a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.132+0000 7fba77e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba7810a920 0x7fba781a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.132+0000 7fba7d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba78109fd0 0x7fba781a1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57458/0 (socket says 172.21.15.134:57458) 2024-07-26T11:26:50.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.132+0000 7fba7fa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba78115fd0 0x7fba781a81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.132+0000 7fba7fa00640 1 -- 172.21.15.134:0/2969532314 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fba7811bca0 con 0x7fba7810a920 2024-07-26T11:26:50.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.132+0000 7fba7fa00640 1 -- 172.21.15.134:0/2969532314 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fba7811bb20 con 0x7fba78115fd0 2024-07-26T11:26:50.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.132+0000 7fba7fa00640 1 -- 172.21.15.134:0/2969532314 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba7811be20 con 0x7fba78109fd0 2024-07-26T11:26:50.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.132+0000 7fba7d000640 1 -- 172.21.15.134:0/2969532314 learned_addr learned my addr 172.21.15.134:0/2969532314 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:50.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.133+0000 7fba7d000640 1 -- 172.21.15.134:0/2969532314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba78115fd0 msgr2=0x7fba781a81d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-07-26T11:26:50.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.133+0000 7fba7d000640 1 --2- 172.21.15.134:0/2969532314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba78115fd0 0x7fba781a81d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.133+0000 7fba7d000640 1 -- 172.21.15.134:0/2969532314 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba7810a920 msgr2=0x7fba781a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:50.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.133+0000 7fba7d000640 1 --2- 172.21.15.134:0/2969532314 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba7810a920 0x7fba781a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.133+0000 7fba7d000640 1 -- 172.21.15.134:0/2969532314 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba781a1f80 con 0x7fba78109fd0 2024-07-26T11:26:50.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.134+0000 7fba7d000640 1 --2- 172.21.15.134:0/2969532314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba78109fd0 0x7fba781a1170 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fba6c005940 tx=0x7fba6c003fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:50.137 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.135+0000 7fba75600640 1 -- 172.21.15.134:0/2969532314 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6c040030 con 0x7fba78109fd0 2024-07-26T11:26:50.137 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.135+0000 7fba75600640 1 -- 172.21.15.134:0/2969532314 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba6c03c8f0 con 0x7fba78109fd0 2024-07-26T11:26:50.137 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.135+0000 7fba7fa00640 1 -- 172.21.15.134:0/2969532314 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba781a87d0 con 0x7fba78109fd0 2024-07-26T11:26:50.138 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.136+0000 7fba75600640 1 -- 172.21.15.134:0/2969532314 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6c031a10 con 0x7fba78109fd0 2024-07-26T11:26:50.138 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.136+0000 7fba7fa00640 1 -- 172.21.15.134:0/2969532314 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba781a8c90 con 0x7fba78109fd0 2024-07-26T11:26:50.153 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.148+0000 7fba75600640 1 -- 172.21.15.134:0/2969532314 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fba6c043030 con 0x7fba78109fd0 2024-07-26T11:26:50.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.150+0000 7fba86bb4740 1 Processor -- start 2024-07-26T11:26:50.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.151+0000 7fba75600640 1 --2- 172.21.15.134:0/2969532314 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fba600b3e90 0x7fba600b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.152+0000 7fba75600640 1 -- 172.21.15.134:0/2969532314 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9495+0+0 (secure 0 0 0) 0x7fba6c0c7e60 con 0x7fba78109fd0 2024-07-26T11:26:50.164 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.162+0000 7fba7da00640 1 --2- 172.21.15.134:0/2969532314 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fba600b3e90 0x7fba600b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.162+0000 7fba86bb4740 1 -- start start 2024-07-26T11:26:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.163+0000 7fba86bb4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 0x5578a182ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.163+0000 7fba86bb4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a14622f0 0x5578a182ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.163+0000 7fba86bb4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a146a4d0 0x5578a182f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.163+0000 7fba86bb4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5578a17a3270 con 0x5578a14622f0 2024-07-26T11:26:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.163+0000 7fba86bb4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5578a1792770 con 0x5578a146a4d0 2024-07-26T11:26:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.163+0000 7fba86bb4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5578a17967b0 con 0x5578a148b1b0 2024-07-26T11:26:50.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.164+0000 7fba7d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 0x5578a182ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.164+0000 7fba7d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 0x5578a182ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57464/0 (socket says 172.21.15.134:57464) 2024-07-26T11:26:50.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.164+0000 7fba7da00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a146a4d0 0x5578a182f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.164+0000 7fba7d000640 1 -- 172.21.15.134:0/4092013723 learned_addr learned my addr 172.21.15.134:0/4092013723 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:50.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.164+0000 7fba7da00640 1 --2- 172.21.15.134:0/2969532314 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fba600b3e90 0x7fba600b63a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fba7000ab10 tx=0x7fba70009250 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:50.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.165+0000 7fba7da00640 1 -- 172.21.15.134:0/4092013723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 msgr2=0x5578a182ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:50.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.166+0000 7fba7da00640 1 --2- 172.21.15.134:0/4092013723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 0x5578a182ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.166+0000 7fba7da00640 1 -- 172.21.15.134:0/4092013723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a14622f0 msgr2=0x5578a182ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:50.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.166+0000 7fba7da00640 1 --2- 172.21.15.134:0/4092013723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a14622f0 0x5578a182ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.166+0000 7fba7da00640 1 -- 172.21.15.134:0/4092013723 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578a14f6790 con 0x5578a146a4d0 2024-07-26T11:26:50.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.166+0000 7fba7d000640 1 --2- 172.21.15.134:0/4092013723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 0x5578a182ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:26:50.170 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.168+0000 7fba7da00640 1 --2- 172.21.15.134:0/4092013723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a146a4d0 0x5578a182f4c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fba7000cf90 tx=0x7fba7000f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68d27e8503985536 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:50.171 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.169+0000 7fba6a000640 1 -- 172.21.15.134:0/4092013723 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7001a830 con 0x5578a146a4d0 2024-07-26T11:26:50.171 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.169+0000 7fba6a000640 1 -- 172.21.15.134:0/4092013723 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba70022400 con 0x5578a146a4d0 2024-07-26T11:26:50.171 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.169+0000 7fba6a000640 1 -- 172.21.15.134:0/4092013723 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba70018610 con 0x5578a146a4d0 2024-07-26T11:26:50.177 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.175+0000 7fba86bb4740 1 -- 172.21.15.134:0/4092013723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a146a4d0 msgr2=0x5578a182f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.175+0000 7fba86bb4740 1 --2- 172.21.15.134:0/4092013723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a146a4d0 0x5578a182f4c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fba7000cf90 tx=0x7fba7000f000 comp rx=0 tx=0).stop 2024-07-26T11:26:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.176+0000 7fba86bb4740 1 -- 172.21.15.134:0/4092013723 shutdown_connections 2024-07-26T11:26:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.176+0000 7fba86bb4740 1 --2- 172.21.15.134:0/4092013723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a146a4d0 0x5578a182f4c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.177+0000 7fba86bb4740 1 --2- 172.21.15.134:0/4092013723 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a14622f0 0x5578a182ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.177+0000 7fba86bb4740 1 --2- 172.21.15.134:0/4092013723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 0x5578a182ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.177+0000 7fba86bb4740 1 -- 172.21.15.134:0/4092013723 >> 172.21.15.134:0/4092013723 conn(0x5578a147f300 msgr2=0x5578a182c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.177+0000 7fba86bb4740 1 -- 172.21.15.134:0/4092013723 shutdown_connections 2024-07-26T11:26:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.177+0000 7fba86bb4740 1 -- 172.21.15.134:0/4092013723 wait complete. 2024-07-26T11:26:50.179 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.178+0000 7fba86bb4740 1 -- start start 2024-07-26T11:26:50.181 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.178+0000 7fba86bb4740 1 Processor -- start 2024-07-26T11:26:50.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.180+0000 7fba86bb4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a14622f0 0x5578a183a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.181+0000 7fba86bb4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a146a4d0 0x5578a183ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.181+0000 7fba86bb4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 0x5578a1840480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.181+0000 7fba86bb4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x5578a16ccab0 con 0x5578a146a4d0 2024-07-26T11:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.181+0000 7fba86bb4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5578a172fd50 con 0x5578a14622f0 2024-07-26T11:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.181+0000 7fba86bb4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5578a183d1c0 con 0x5578a148b1b0 2024-07-26T11:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.182+0000 7fba86bb4740 10 client.0 _ll_register_callbacks cb 0x5578a14a2a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.182+0000 7fba7da00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 0x5578a1840480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.182+0000 7fba7da00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 0x5578a1840480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57466/0 (socket says 172.21.15.134:57466) 2024-07-26T11:26:50.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.182+0000 7fba7da00640 1 -- 172.21.15.134:0/2753619304 learned_addr learned my addr 172.21.15.134:0/2753619304 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.182+0000 7fba7d000640 1 --2- 172.21.15.134:0/2753619304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a14622f0 0x5578a183a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.182+0000 7fba77e00640 1 --2- 172.21.15.134:0/2753619304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a146a4d0 0x5578a183ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.183+0000 7fba7da00640 1 -- 172.21.15.134:0/2753619304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a14622f0 msgr2=0x5578a183a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.183+0000 7fba7da00640 1 --2- 172.21.15.134:0/2753619304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a14622f0 0x5578a183a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.183+0000 7fba7da00640 1 -- 172.21.15.134:0/2753619304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a146a4d0 msgr2=0x5578a183ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.183+0000 7fba7da00640 1 --2- 172.21.15.134:0/2753619304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a146a4d0 0x5578a183ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.183+0000 7fba7da00640 1 -- 172.21.15.134:0/2753619304 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578a156f620 con 0x5578a148b1b0 2024-07-26T11:26:50.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.185+0000 7fba7da00640 1 --2- 172.21.15.134:0/2753619304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 0x5578a1840480 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fba70022cc0 tx=0x7fba7000c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8452e58dab7016ad server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:50.187 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.185+0000 7fba5ea00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:26:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.185+0000 7fba5ea00640 1 -- 172.21.15.134:0/2753619304 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7001aa40 con 0x5578a148b1b0 2024-07-26T11:26:50.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.185+0000 7fba5ea00640 1 -- 172.21.15.134:0/2753619304 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9495+0+0 (secure 0 0 0) 0x7fba7001abe0 con 0x5578a148b1b0 2024-07-26T11:26:50.191 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.186+0000 7fba86bb4740 1 -- 172.21.15.134:0/2753619304 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba70022400 con 0x5578a148b1b0 2024-07-26T11:26:50.191 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.190+0000 7fba5ea00640 1 -- 172.21.15.134:0/2753619304 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fba7001ade0 con 0x5578a148b1b0 2024-07-26T11:26:50.191 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.190+0000 7fba86bb4740 10 client.26938 fetch_fsmap learned FSMap version 33 2024-07-26T11:26:50.192 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.190+0000 7fba86bb4740 1 -- 172.21.15.134:0/2753619304 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5578a1849320 con 0x5578a148b1b0 2024-07-26T11:26:50.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.192+0000 7fba5ea00640 1 -- 172.21.15.134:0/2753619304 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba70032070 con 0x5578a148b1b0 2024-07-26T11:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.192+0000 7fba5ea00640 1 -- 172.21.15.134:0/2753619304 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7fba70037450 con 0x5578a148b1b0 2024-07-26T11:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.194+0000 7fba86bb4740 10 client.26938 fetch_fsmap finished waiting for FSMap version 33 2024-07-26T11:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.194+0000 7fba86bb4740 10 client.26938 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.194+0000 7fba86bb4740 20 client.26938 populate_metadata read hostname 'smithi134' 2024-07-26T11:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.194+0000 7fba86bb4740 1 --2- 172.21.15.134:0/2753619304 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5578a1449d80 0x5578a184cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.194+0000 7fba86bb4740 4 client.26938 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.194+0000 7fba86bb4740 1 -- 172.21.15.134:0/2753619304 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fba70022400 con 0x5578a1449d80 2024-07-26T11:26:50.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.194+0000 7fba7d000640 1 --2- 172.21.15.134:0/2753619304 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5578a1449d80 0x5578a184cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.201+0000 7fba5ea00640 10 client.26938 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:26:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.201+0000 7fba7d000640 1 --2- 172.21.15.134:0/2753619304 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5578a1449d80 0x5578a184cb40 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7b2be1e30f7ac00 server_cookie=9002b4c4034aebeb in_seq=0 out_seq=0 2024-07-26T11:26:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.202+0000 7fba5ea00640 1 -- 172.21.15.134:0/2753619304 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x7fba70022400 con 0x5578a1449d80 2024-07-26T11:26:50.204 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.202+0000 7fba5ea00640 10 client.26938 handle_command_reply: tid=0 2024-07-26T11:26:50.205 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.202+0000 7fba86bb4740 1 client.26938 shutdown 2024-07-26T11:26:50.207 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.205+0000 7fba86bb4740 1 -- 172.21.15.134:0/2753619304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 msgr2=0x5578a1840480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:50.207 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.205+0000 7fba86bb4740 1 --2- 172.21.15.134:0/2753619304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 0x5578a1840480 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fba70022cc0 tx=0x7fba7000c920 comp rx=0 tx=0).stop 2024-07-26T11:26:50.207 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.205+0000 7fba86bb4740 1 -- 172.21.15.134:0/2753619304 shutdown_connections 2024-07-26T11:26:50.207 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.205+0000 7fba86bb4740 1 --2- 172.21.15.134:0/2753619304 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5578a1449d80 0x5578a184cb40 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.207 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.205+0000 7fba5ea00640 0 client.26938 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:26:50.207 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.205+0000 7fba86bb4740 1 --2- 172.21.15.134:0/2753619304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 0x5578a1840480 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.207 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.205+0000 7fba86bb4740 1 --2- 172.21.15.134:0/2753619304 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a146a4d0 0x5578a183ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.207 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.205+0000 7fba86bb4740 1 --2- 172.21.15.134:0/2753619304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a14622f0 0x5578a183a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.207 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.205+0000 7fba86bb4740 1 -- 172.21.15.134:0/2753619304 >> 172.21.15.134:0/2753619304 conn(0x5578a147f300 msgr2=0x5578a182b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:50.208 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.206+0000 7fba86bb4740 1 -- 172.21.15.134:0/2753619304 shutdown_connections 2024-07-26T11:26:50.209 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.207+0000 7fba86bb4740 1 -- 172.21.15.134:0/2753619304 wait complete. 2024-07-26T11:26:50.210 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.208+0000 7fba86bb4740 20 client.26938 trim_cache size 0 max 16384 2024-07-26T11:26:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.270+0000 7fba86bb4740 1 Processor -- start 2024-07-26T11:26:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.271+0000 7fba86bb4740 1 -- start start 2024-07-26T11:26:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.272+0000 7fba86bb4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a14622f0 0x5578a182b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.272+0000 7fba86bb4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a146a4d0 0x5578a1834210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.272+0000 7fba86bb4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 0x5578a18403f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.273+0000 7fba86bb4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fba781c7ff0 con 0x5578a14622f0 2024-07-26T11:26:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.273+0000 7fba86bb4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fba781c9b90 con 0x5578a146a4d0 2024-07-26T11:26:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.273+0000 7fba86bb4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba781c86c0 con 0x5578a148b1b0 2024-07-26T11:26:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.274+0000 7fba77e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a146a4d0 0x5578a1834210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.274+0000 7fba77e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a146a4d0 0x5578a1834210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35264/0 (socket says 172.21.15.134:35264) 2024-07-26T11:26:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.274+0000 7fba77e00640 1 -- 172.21.15.134:0/1587771831 learned_addr learned my addr 172.21.15.134:0/1587771831 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.274+0000 7fba7da00640 1 --2- 172.21.15.134:0/1587771831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 0x5578a18403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.274+0000 7fba7d000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a14622f0 0x5578a182b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.276+0000 7fba77e00640 1 -- 172.21.15.134:0/1587771831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 msgr2=0x5578a18403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.276+0000 7fba77e00640 1 --2- 172.21.15.134:0/1587771831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 0x5578a18403f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.276+0000 7fba77e00640 1 -- 172.21.15.134:0/1587771831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a14622f0 msgr2=0x5578a182b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.276+0000 7fba77e00640 1 --2- 172.21.15.134:0/1587771831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a14622f0 0x5578a182b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.276+0000 7fba77e00640 1 -- 172.21.15.134:0/1587771831 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578a1828350 con 0x5578a146a4d0 2024-07-26T11:26:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.276+0000 7fba77e00640 1 --2- 172.21.15.134:0/1587771831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a146a4d0 0x5578a1834210 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fba6400bab0 tx=0x7fba6400bf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e318515916b1afc server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.277+0000 7fba69600640 1 -- 172.21.15.134:0/1587771831 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba64004230 con 0x5578a146a4d0 2024-07-26T11:26:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.277+0000 7fba69600640 1 -- 172.21.15.134:0/1587771831 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba640043d0 con 0x5578a146a4d0 2024-07-26T11:26:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.277+0000 7fba69600640 1 -- 172.21.15.134:0/1587771831 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba64010910 con 0x5578a146a4d0 2024-07-26T11:26:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.278+0000 7fba86bb4740 1 -- 172.21.15.134:0/1587771831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a146a4d0 msgr2=0x5578a1834210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:50.281 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.278+0000 7fba86bb4740 1 --2- 172.21.15.134:0/1587771831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a146a4d0 0x5578a1834210 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fba6400bab0 tx=0x7fba6400bf70 comp rx=0 tx=0).stop 2024-07-26T11:26:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.281+0000 7fba86bb4740 1 -- 172.21.15.134:0/1587771831 shutdown_connections 2024-07-26T11:26:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.281+0000 7fba86bb4740 1 --2- 172.21.15.134:0/1587771831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a148b1b0 0x5578a18403f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.281+0000 7fba86bb4740 1 --2- 172.21.15.134:0/1587771831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a146a4d0 0x5578a1834210 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.281+0000 7fba86bb4740 1 --2- 172.21.15.134:0/1587771831 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a14622f0 0x5578a182b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.281+0000 7fba86bb4740 1 -- 172.21.15.134:0/1587771831 >> 172.21.15.134:0/1587771831 conn(0x5578a147f300 msgr2=0x5578a1833b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.281+0000 7fba86bb4740 1 -- 172.21.15.134:0/1587771831 shutdown_connections 2024-07-26T11:26:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.281+0000 7fba86bb4740 1 -- 172.21.15.134:0/1587771831 wait complete. 2024-07-26T11:26:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.282+0000 7fba86bb4740 1 -- start start 2024-07-26T11:26:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.282+0000 7fba86bb4740 1 Processor -- start 2024-07-26T11:26:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.283+0000 7fba86bb4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a14622f0 0x5578a1846f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.286 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.283+0000 7fba86bb4740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a146a4d0 0x5578a1841490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.286 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.283+0000 7fba86bb4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a148b1b0 0x5578a18419d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.283+0000 7fba86bb4740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fba781c9460 con 0x5578a146a4d0 2024-07-26T11:26:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.283+0000 7fba86bb4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fba781c8d90 con 0x5578a148b1b0 2024-07-26T11:26:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.283+0000 7fba86bb4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fba781c5ff0 con 0x5578a14622f0 2024-07-26T11:26:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.283+0000 7fba86bb4740 10 client.0 _ll_register_callbacks cb 0x7fba6c0095e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:26:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.286+0000 7fba7d000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a14622f0 0x5578a1846f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.286+0000 7fba77e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a146a4d0 0x5578a1841490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.286+0000 7fba77e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a146a4d0 0x5578a1841490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:39326/0 (socket says 172.21.15.134:39326) 2024-07-26T11:26:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.286+0000 7fba77e00640 1 -- 172.21.15.134:0/3582296678 learned_addr learned my addr 172.21.15.134:0/3582296678 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.287+0000 7fba7da00640 1 --2- 172.21.15.134:0/3582296678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a148b1b0 0x5578a18419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.287+0000 7fba7d000640 1 -- 172.21.15.134:0/3582296678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a148b1b0 msgr2=0x5578a18419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:50.289 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.287+0000 7fba7d000640 1 --2- 172.21.15.134:0/3582296678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a148b1b0 0x5578a18419d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.287+0000 7fba7d000640 1 -- 172.21.15.134:0/3582296678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a146a4d0 msgr2=0x5578a1841490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.287+0000 7fba7d000640 1 --2- 172.21.15.134:0/3582296678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a146a4d0 0x5578a1841490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.287+0000 7fba7d000640 1 -- 172.21.15.134:0/3582296678 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578a1841fc0 con 0x5578a14622f0 2024-07-26T11:26:50.290 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.287+0000 7fba77e00640 1 --2- 172.21.15.134:0/3582296678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a146a4d0 0x5578a1841490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:26:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.288+0000 7fba7d000640 1 --2- 172.21.15.134:0/3582296678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a14622f0 0x5578a1846f10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fba6c0094b0 tx=0x7fba6c0c5f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69cfcd02ebfb80a6 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:50.291 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.288+0000 7fba5ea00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:26:50.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.288+0000 7fba5ea00640 1 -- 172.21.15.134:0/3582296678 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6c04a040 con 0x5578a14622f0 2024-07-26T11:26:50.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.288+0000 7fba86bb4740 1 -- 172.21.15.134:0/3582296678 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba640043d0 con 0x5578a14622f0 2024-07-26T11:26:50.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.288+0000 7fba5ea00640 1 -- 172.21.15.134:0/3582296678 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9495+0+0 (secure 0 0 0) 0x7fba6c0c7b00 con 0x5578a14622f0 2024-07-26T11:26:50.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.288+0000 7fba5ea00640 1 -- 172.21.15.134:0/3582296678 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fba6c0d4920 con 0x5578a14622f0 2024-07-26T11:26:50.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.288+0000 7fba86bb4740 10 client.26944 fetch_fsmap learned FSMap version 33 2024-07-26T11:26:50.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.288+0000 7fba86bb4740 1 -- 172.21.15.134:0/3582296678 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x5578a18b40d0 con 0x5578a14622f0 2024-07-26T11:26:50.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.289+0000 7fba5ea00640 1 -- 172.21.15.134:0/3582296678 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba640043d0 con 0x5578a14622f0 2024-07-26T11:26:50.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.292+0000 7fba5ea00640 1 -- 172.21.15.134:0/3582296678 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 33) ==== 8378+0+0 (secure 0 0 0) 0x7fba6c0d4490 con 0x5578a14622f0 2024-07-26T11:26:50.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.292+0000 7fba86bb4740 10 client.26944 fetch_fsmap finished waiting for FSMap version 33 2024-07-26T11:26:50.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.292+0000 7fba86bb4740 10 client.26944 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:26:50.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.292+0000 7fba86bb4740 20 client.26944 populate_metadata read hostname 'smithi134' 2024-07-26T11:26:50.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.292+0000 7fba86bb4740 1 --2- 172.21.15.134:0/3582296678 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5578a141f1c0 0x5578a18b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.292+0000 7fba86bb4740 4 client.26944 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:26:50.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.292+0000 7fba86bb4740 1 -- 172.21.15.134:0/3582296678 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5578a17a51c0 con 0x5578a141f1c0 2024-07-26T11:26:50.297 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.296+0000 7fba77e00640 1 --2- 172.21.15.134:0/3582296678 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5578a141f1c0 0x5578a18b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.301 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.297+0000 7fba77e00640 1 --2- 172.21.15.134:0/3582296678 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5578a141f1c0 0x5578a18b68a0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=636fa32edec7a52e server_cookie=5b2e750521510838 in_seq=0 out_seq=0 2024-07-26T11:26:50.302 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:50.297+0000 7fba5ea00640 10 client.26944 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:26:50.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.587+0000 7f07ebe00640 1 -- 172.21.15.134:0/2246645935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07e410f360 msgr2=0x7f07e410f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:50.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.587+0000 7f07ebe00640 1 --2- 172.21.15.134:0/2246645935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07e410f360 0x7f07e410f770 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f07d0009900 tx=0x7f07d002a440 comp rx=0 tx=0).stop 2024-07-26T11:26:50.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.590+0000 7f07ebe00640 1 -- 172.21.15.134:0/2246645935 shutdown_connections 2024-07-26T11:26:50.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.590+0000 7f07ebe00640 1 --2- 172.21.15.134:0/2246645935 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07e410fcb0 0x7f07e41164e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.590+0000 7f07ebe00640 1 --2- 172.21.15.134:0/2246645935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07e410f360 0x7f07e410f770 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.590+0000 7f07ebe00640 1 --2- 172.21.15.134:0/2246645935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07e4109a70 0x7f07e4109e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.590+0000 7f07ebe00640 1 -- 172.21.15.134:0/2246645935 >> 172.21.15.134:0/2246645935 conn(0x7f07e4079040 msgr2=0x7f07e4107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:50.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.592+0000 7f07ebe00640 1 -- 172.21.15.134:0/2246645935 shutdown_connections 2024-07-26T11:26:50.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.592+0000 7f07ebe00640 1 -- 172.21.15.134:0/2246645935 wait complete. 2024-07-26T11:26:50.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.595+0000 7f07ebe00640 1 Processor -- start 2024-07-26T11:26:50.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.596+0000 7f07ebe00640 1 -- start start 2024-07-26T11:26:50.599 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.597+0000 7f07ebe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07e4109a70 0x7f07e41a0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.599 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.597+0000 7f07ebe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07e410f360 0x7f07e41a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.597+0000 7f07ebe00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07e410fcb0 0x7f07e41a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.597+0000 7f07ebe00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f07e41199d0 con 0x7f07e410fcb0 2024-07-26T11:26:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.597+0000 7f07ebe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f07e4119850 con 0x7f07e410f360 2024-07-26T11:26:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.597+0000 7f07ebe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f07e4119b50 con 0x7f07e4109a70 2024-07-26T11:26:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.598+0000 7f07e9e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07e410fcb0 0x7f07e41a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.598+0000 7f07e9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07e4109a70 0x7f07e41a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.598+0000 7f07e9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07e4109a70 0x7f07e41a0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57506/0 (socket says 172.21.15.134:57506) 2024-07-26T11:26:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.598+0000 7f07e9400640 1 -- 172.21.15.134:0/2233822680 learned_addr learned my addr 172.21.15.134:0/2233822680 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:50.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.598+0000 7f07e8a00640 1 --2- 172.21.15.134:0/2233822680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07e410f360 0x7f07e41a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.599+0000 7f07e9e00640 1 -- 172.21.15.134:0/2233822680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07e4109a70 msgr2=0x7f07e41a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:50.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.599+0000 7f07e9e00640 1 --2- 172.21.15.134:0/2233822680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07e4109a70 0x7f07e41a0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.599+0000 7f07e9e00640 1 -- 172.21.15.134:0/2233822680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07e410f360 msgr2=0x7f07e41a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:50.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.599+0000 7f07e9e00640 1 --2- 172.21.15.134:0/2233822680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07e410f360 0x7f07e41a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.599+0000 7f07e9e00640 1 -- 172.21.15.134:0/2233822680 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e41a6370 con 0x7f07e410fcb0 2024-07-26T11:26:50.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.600+0000 7f07e9e00640 1 --2- 172.21.15.134:0/2233822680 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07e410fcb0 0x7f07e41a5c40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f07dc00be00 tx=0x7f07dc00bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:50.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.602+0000 7f07e2000640 1 -- 172.21.15.134:0/2233822680 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc019070 con 0x7f07e410fcb0 2024-07-26T11:26:50.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.602+0000 7f07e2000640 1 -- 172.21.15.134:0/2233822680 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07dc0148f0 con 0x7f07e410fcb0 2024-07-26T11:26:50.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.602+0000 7f07e2000640 1 -- 172.21.15.134:0/2233822680 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc012830 con 0x7f07e410fcb0 2024-07-26T11:26:50.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.602+0000 7f07ebe00640 1 -- 172.21.15.134:0/2233822680 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07e41a6660 con 0x7f07e410fcb0 2024-07-26T11:26:50.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.602+0000 7f07ebe00640 1 -- 172.21.15.134:0/2233822680 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07e41acdc0 con 0x7f07e410fcb0 2024-07-26T11:26:50.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.616+0000 7f07e2000640 1 -- 172.21.15.134:0/2233822680 <== mon.0 v2:172.21.15.134:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7f07dc01c070 con 0x7f07e410fcb0 2024-07-26T11:26:50.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.619+0000 7f07e2000640 1 --2- 172.21.15.134:0/2233822680 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f07cc0b3dc0 0x7f07cc0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.622 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.620+0000 7f07e2000640 1 -- 172.21.15.134:0/2233822680 <== mon.0 v2:172.21.15.134:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9495+0+0 (secure 0 0 0) 0x7f07dc0e4a00 con 0x7f07e410fcb0 2024-07-26T11:26:50.624 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.622+0000 7f07e9400640 1 --2- 172.21.15.134:0/2233822680 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f07cc0b3dc0 0x7f07cc0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.649+0000 7f07e9400640 1 --2- 172.21.15.134:0/2233822680 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f07cc0b3dc0 0x7f07cc0b62d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f07d8009c90 tx=0x7f07d8005f90 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:50.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.650+0000 7f07f2aa7740 1 Processor -- start 2024-07-26T11:26:50.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.650+0000 7f07f2aa7740 1 -- start start 2024-07-26T11:26:50.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.651+0000 7f07f2aa7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfe8070 0x56218e38d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.651+0000 7f07f2aa7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfbf1b0 0x56218e38d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.651+0000 7f07f2aa7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfc7390 0x56218e38dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.651+0000 7f07f2aa7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56218e303bf0 con 0x56218dfe8070 2024-07-26T11:26:50.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.651+0000 7f07f2aa7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56218e2f30f0 con 0x56218dfc7390 2024-07-26T11:26:50.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.651+0000 7f07f2aa7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56218e2f7130 con 0x56218dfbf1b0 2024-07-26T11:26:50.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.651+0000 7f07e9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfc7390 0x56218e38dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.651+0000 7f07e9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfc7390 0x56218e38dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35296/0 (socket says 172.21.15.134:35296) 2024-07-26T11:26:50.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.652+0000 7f07e9e00640 1 -- 172.21.15.134:0/2731741682 learned_addr learned my addr 172.21.15.134:0/2731741682 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:50.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.653+0000 7f07e9e00640 1 -- 172.21.15.134:0/2731741682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfbf1b0 msgr2=0x56218e38d570 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:26:50.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.653+0000 7f07e9e00640 1 --2- 172.21.15.134:0/2731741682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfbf1b0 0x56218e38d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.653+0000 7f07e9e00640 1 -- 172.21.15.134:0/2731741682 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfe8070 msgr2=0x56218e38d030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:26:50.655 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.653+0000 7f07e9e00640 1 --2- 172.21.15.134:0/2731741682 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfe8070 0x56218e38d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.653+0000 7f07e9e00640 1 -- 172.21.15.134:0/2731741682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56218e053650 con 0x56218dfc7390 2024-07-26T11:26:50.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.655+0000 7f07e9e00640 1 --2- 172.21.15.134:0/2731741682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfc7390 0x56218e38dab0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f07dc00bdd0 tx=0x7f07dc0ab300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58ecc421dccf79a4 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:50.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.656+0000 7f07d6600640 1 -- 172.21.15.134:0/2731741682 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc0af950 con 0x56218dfc7390 2024-07-26T11:26:50.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.657+0000 7f07d6600640 1 -- 172.21.15.134:0/2731741682 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07dc0b4400 con 0x56218dfc7390 2024-07-26T11:26:50.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.658+0000 7f07f2aa7740 1 -- 172.21.15.134:0/2731741682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfc7390 msgr2=0x56218e38dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:50.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.658+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/2731741682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfc7390 0x56218e38dab0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f07dc00bdd0 tx=0x7f07dc0ab300 comp rx=0 tx=0).stop 2024-07-26T11:26:50.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.661+0000 7f07f2aa7740 1 -- 172.21.15.134:0/2731741682 shutdown_connections 2024-07-26T11:26:50.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.661+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/2731741682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfbf1b0 0x56218e38d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.661+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/2731741682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfc7390 0x56218e38dab0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.661+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/2731741682 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfe8070 0x56218e38d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.661+0000 7f07f2aa7740 1 -- 172.21.15.134:0/2731741682 >> 172.21.15.134:0/2731741682 conn(0x56218dfdc460 msgr2=0x56218e38af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:50.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.662+0000 7f07f2aa7740 1 -- 172.21.15.134:0/2731741682 shutdown_connections 2024-07-26T11:26:50.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.662+0000 7f07f2aa7740 1 -- 172.21.15.134:0/2731741682 wait complete. 2024-07-26T11:26:50.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.663+0000 7f07f2aa7740 1 -- start start 2024-07-26T11:26:50.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.663+0000 7f07f2aa7740 1 Processor -- start 2024-07-26T11:26:50.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.666+0000 7f07f2aa7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfbf1b0 0x56218e3a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.666+0000 7f07f2aa7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfc7390 0x56218e3a9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.670+0000 7f07e8a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfc7390 0x56218e3a9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.670+0000 7f07e8a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfc7390 0x56218e3a9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:57522/0 (socket says 172.21.15.134:57522) 2024-07-26T11:26:50.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.671+0000 7f07e9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfbf1b0 0x56218e3a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.671+0000 7f07e9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfbf1b0 0x56218e3a95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35308/0 (socket says 172.21.15.134:35308) 2024-07-26T11:26:50.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.671+0000 7f07f2aa7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfe8070 0x56218e3aa030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.673+0000 7f07e9e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfe8070 0x56218e3aa030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.673+0000 7f07f2aa7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56218e156700 con 0x56218dfe8070 2024-07-26T11:26:50.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.673+0000 7f07f2aa7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56218e28f6c0 con 0x56218dfbf1b0 2024-07-26T11:26:50.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.673+0000 7f07f2aa7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56218e3ac810 con 0x56218dfc7390 2024-07-26T11:26:50.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.673+0000 7f07f2aa7740 10 client.0 _ll_register_callbacks cb 0x56218e29f170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:26:50.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.673+0000 7f07e8a00640 1 -- 172.21.15.134:0/483524801 learned_addr learned my addr 172.21.15.134:0/483524801 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:50.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.675+0000 7f07e9400640 1 -- 172.21.15.134:0/483524801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfc7390 msgr2=0x56218e3a9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:50.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.675+0000 7f07e9400640 1 --2- 172.21.15.134:0/483524801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfc7390 0x56218e3a9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.675+0000 7f07e9400640 1 -- 172.21.15.134:0/483524801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfe8070 msgr2=0x56218e3aa030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:50.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.675+0000 7f07e9400640 1 --2- 172.21.15.134:0/483524801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfe8070 0x56218e3aa030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.675+0000 7f07e9400640 1 -- 172.21.15.134:0/483524801 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56218e2a6860 con 0x56218dfbf1b0 2024-07-26T11:26:50.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.676+0000 7f07e8a00640 1 --2- 172.21.15.134:0/483524801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfc7390 0x56218e3a9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:26:50.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.677+0000 7f07e9400640 1 --2- 172.21.15.134:0/483524801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfbf1b0 0x56218e3a95b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f07d8011ca0 tx=0x7f07d80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcef8fd6b3dd6824 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:50.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.677+0000 7f07caa00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:26:50.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.678+0000 7f07caa00640 1 -- 172.21.15.134:0/483524801 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d802b060 con 0x56218dfbf1b0 2024-07-26T11:26:50.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.678+0000 7f07caa00640 1 -- 172.21.15.134:0/483524801 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9495+0+0 (secure 0 0 0) 0x7f07d800e8b0 con 0x56218dfbf1b0 2024-07-26T11:26:50.682 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.680+0000 7f07f2aa7740 1 -- 172.21.15.134:0/483524801 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56218e305b40 con 0x56218dfbf1b0 2024-07-26T11:26:50.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.680+0000 7f07caa00640 1 -- 172.21.15.134:0/483524801 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f07d800eab0 con 0x56218dfbf1b0 2024-07-26T11:26:50.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.681+0000 7f07f2aa7740 10 client.26936 fetch_fsmap learned FSMap version 34 2024-07-26T11:26:50.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.681+0000 7f07f2aa7740 1 -- 172.21.15.134:0/483524801 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56218e3b85b0 con 0x56218dfbf1b0 2024-07-26T11:26:50.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.681+0000 7f07caa00640 1 -- 172.21.15.134:0/483524801 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56218e305b40 con 0x56218dfbf1b0 2024-07-26T11:26:50.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.683+0000 7f07caa00640 1 -- 172.21.15.134:0/483524801 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8386+0+0 (secure 0 0 0) 0x7f07d802f3a0 con 0x56218dfbf1b0 2024-07-26T11:26:50.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.683+0000 7f07f2aa7740 10 client.26936 fetch_fsmap finished waiting for FSMap version 34 2024-07-26T11:26:50.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.683+0000 7f07f2aa7740 10 client.26936 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T11:26:50.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.683+0000 7f07f2aa7740 20 client.26936 populate_metadata read hostname 'smithi134' 2024-07-26T11:26:50.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.683+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/483524801 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x56218dfa69f0 0x56218e3bb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.683+0000 7f07f2aa7740 4 client.26936 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:26:50.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.683+0000 7f07f2aa7740 1 -- 172.21.15.134:0/483524801 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56218e3015e0 con 0x56218dfa69f0 2024-07-26T11:26:50.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.684+0000 7f07e8a00640 1 --2- 172.21.15.134:0/483524801 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x56218dfa69f0 0x56218e3bb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.685+0000 7f07e8a00640 1 --2- 172.21.15.134:0/483524801 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x56218dfa69f0 0x56218e3bb9d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cafe1facb7adce12 server_cookie=6d9ae54272190565 in_seq=0 out_seq=0 2024-07-26T11:26:50.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.685+0000 7f07caa00640 10 client.26936 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:26:50.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.689+0000 7f07caa00640 1 -- 172.21.15.134:0/483524801 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x56218e3015e0 con 0x56218dfa69f0 2024-07-26T11:26:50.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.689+0000 7f07caa00640 10 client.26936 handle_command_reply: tid=0 2024-07-26T11:26:50.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.690+0000 7f07f2aa7740 1 client.26936 shutdown 2024-07-26T11:26:50.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.691+0000 7f07f2aa7740 1 -- 172.21.15.134:0/483524801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfbf1b0 msgr2=0x56218e3a95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:50.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.691+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/483524801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfbf1b0 0x56218e3a95b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f07d8011ca0 tx=0x7f07d80046c0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.691+0000 7f07f2aa7740 1 -- 172.21.15.134:0/483524801 shutdown_connections 2024-07-26T11:26:50.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.691+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/483524801 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x56218dfa69f0 0x56218e3bb9d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.691+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/483524801 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfc7390 0x56218e3a9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.691+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/483524801 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfe8070 0x56218e3aa030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.691+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/483524801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfbf1b0 0x56218e3a95b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.691+0000 7f07f2aa7740 1 -- 172.21.15.134:0/483524801 >> 172.21.15.134:0/483524801 conn(0x56218dfdc460 msgr2=0x56218e389560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:50.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.694+0000 7f07caa00640 0 client.26936 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:26:50.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.694+0000 7f07f2aa7740 1 -- 172.21.15.134:0/483524801 shutdown_connections 2024-07-26T11:26:50.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.694+0000 7f07f2aa7740 1 -- 172.21.15.134:0/483524801 wait complete. 2024-07-26T11:26:50.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.695+0000 7f07f2aa7740 20 client.26936 trim_cache size 0 max 16384 2024-07-26T11:26:50.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.730+0000 7f07f2aa7740 1 Processor -- start 2024-07-26T11:26:50.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.730+0000 7f07f2aa7740 1 -- start start 2024-07-26T11:26:50.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.731+0000 7f07f2aa7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfbf1b0 0x56218e38e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.732+0000 7f07f2aa7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfc7390 0x56218e3b5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.733+0000 7f07f2aa7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfe8070 0x56218e3c1170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.733+0000 7f07f2aa7740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f07e41ae700 con 0x56218dfc7390 2024-07-26T11:26:50.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.733+0000 7f07f2aa7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f07e41b0300 con 0x56218dfe8070 2024-07-26T11:26:50.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.733+0000 7f07f2aa7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f07e41aedd0 con 0x56218dfbf1b0 2024-07-26T11:26:50.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.734+0000 7f07e9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfe8070 0x56218e3c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.734+0000 7f07e9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfe8070 0x56218e3c1170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35312/0 (socket says 172.21.15.134:35312) 2024-07-26T11:26:50.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.734+0000 7f07e9e00640 1 -- 172.21.15.134:0/1631351142 learned_addr learned my addr 172.21.15.134:0/1631351142 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:50.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.734+0000 7f07e9400640 1 --2- 172.21.15.134:0/1631351142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfbf1b0 0x56218e38e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.735+0000 7f07e9e00640 1 -- 172.21.15.134:0/1631351142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfbf1b0 msgr2=0x56218e38e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:50.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.735+0000 7f07e9e00640 1 --2- 172.21.15.134:0/1631351142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfbf1b0 0x56218e38e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.735+0000 7f07e9e00640 1 -- 172.21.15.134:0/1631351142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfc7390 msgr2=0x56218e3b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:50.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.735+0000 7f07e9e00640 1 --2- 172.21.15.134:0/1631351142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfc7390 0x56218e3b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.735+0000 7f07e9e00640 1 -- 172.21.15.134:0/1631351142 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56218e38f950 con 0x56218dfe8070 2024-07-26T11:26:50.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.736+0000 7f07e9400640 1 --2- 172.21.15.134:0/1631351142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfbf1b0 0x56218e38e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:26:50.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.736+0000 7f07e9e00640 1 --2- 172.21.15.134:0/1631351142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfe8070 0x56218e3c1170 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f07dc0ce070 tx=0x7f07dc0ab1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=408df7844eb360f8 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:50.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.736+0000 7f07d5c00640 1 -- 172.21.15.134:0/1631351142 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc0ca070 con 0x56218dfe8070 2024-07-26T11:26:50.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.737+0000 7f07d5c00640 1 -- 172.21.15.134:0/1631351142 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07dc0e1f40 con 0x56218dfe8070 2024-07-26T11:26:50.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.737+0000 7f07d5c00640 1 -- 172.21.15.134:0/1631351142 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07dc0aa8a0 con 0x56218dfe8070 2024-07-26T11:26:50.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.739+0000 7f07f2aa7740 1 -- 172.21.15.134:0/1631351142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfe8070 msgr2=0x56218e3c1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:50.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.739+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/1631351142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfe8070 0x56218e3c1170 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f07dc0ce070 tx=0x7f07dc0ab1d0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.739+0000 7f07f2aa7740 1 -- 172.21.15.134:0/1631351142 shutdown_connections 2024-07-26T11:26:50.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.739+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/1631351142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfe8070 0x56218e3c1170 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.739+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/1631351142 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfc7390 0x56218e3b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.739+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/1631351142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfbf1b0 0x56218e38e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.739+0000 7f07f2aa7740 1 -- 172.21.15.134:0/1631351142 >> 172.21.15.134:0/1631351142 conn(0x56218dfdc460 msgr2=0x56218e38e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:50.743 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.741+0000 7f07f2aa7740 1 -- 172.21.15.134:0/1631351142 shutdown_connections 2024-07-26T11:26:50.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.741+0000 7f07f2aa7740 1 -- 172.21.15.134:0/1631351142 wait complete. 2024-07-26T11:26:50.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.742+0000 7f07f2aa7740 1 -- start start 2024-07-26T11:26:50.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.742+0000 7f07f2aa7740 1 Processor -- start 2024-07-26T11:26:50.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.744+0000 7f07f2aa7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfbf1b0 0x56218e3ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.744+0000 7f07f2aa7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfc7390 0x56218e3ade70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.744+0000 7f07f2aa7740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfe8070 0x56218e3ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.745+0000 7f07e8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfc7390 0x56218e3ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.745+0000 7f07e9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfbf1b0 0x56218e3ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.745+0000 7f07e8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfc7390 0x56218e3ade70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:35322/0 (socket says 172.21.15.134:35322) 2024-07-26T11:26:50.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.745+0000 7f07e8a00640 1 -- 172.21.15.134:0/995782804 learned_addr learned my addr 172.21.15.134:0/995782804 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:26:50.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.745+0000 7f07f2aa7740 1 -- 172.21.15.134:0/995782804 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f07e41afb70 con 0x56218dfe8070 2024-07-26T11:26:50.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.745+0000 7f07f2aa7740 1 -- 172.21.15.134:0/995782804 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f07e41af4a0 con 0x56218dfc7390 2024-07-26T11:26:50.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.745+0000 7f07f2aa7740 1 -- 172.21.15.134:0/995782804 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f07e4074790 con 0x56218dfbf1b0 2024-07-26T11:26:50.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.745+0000 7f07f2aa7740 10 client.0 _ll_register_callbacks cb 0x7f07d801c8a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:26:50.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.747+0000 7f07e9e00640 1 --2- 172.21.15.134:0/995782804 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfe8070 0x56218e3ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.747+0000 7f07e8a00640 1 -- 172.21.15.134:0/995782804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfbf1b0 msgr2=0x56218e3ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:50.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.747+0000 7f07e8a00640 1 --2- 172.21.15.134:0/995782804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfbf1b0 0x56218e3ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.747+0000 7f07e8a00640 1 -- 172.21.15.134:0/995782804 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfe8070 msgr2=0x56218e3ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:26:50.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.747+0000 7f07e8a00640 1 --2- 172.21.15.134:0/995782804 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfe8070 0x56218e3ae3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.747+0000 7f07e8a00640 1 -- 172.21.15.134:0/995782804 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56218e38f550 con 0x56218dfc7390 2024-07-26T11:26:50.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.748+0000 7f07e9400640 1 --2- 172.21.15.134:0/995782804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfbf1b0 0x56218e3ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:26:50.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.749+0000 7f07e8a00640 1 --2- 172.21.15.134:0/995782804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfc7390 0x56218e3ade70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f07d0009230 tx=0x7f07d0003730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92215f75d1815b8c server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:26:50.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.749+0000 7f07caa00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:26:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.750+0000 7f07caa00640 1 -- 172.21.15.134:0/995782804 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d0031710 con 0x56218dfc7390 2024-07-26T11:26:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.751+0000 7f07caa00640 1 -- 172.21.15.134:0/995782804 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9495+0+0 (secure 0 0 0) 0x7f07d00318b0 con 0x56218dfc7390 2024-07-26T11:26:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.751+0000 7f07f2aa7740 1 -- 172.21.15.134:0/995782804 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07dc0e1f40 con 0x56218dfc7390 2024-07-26T11:26:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.751+0000 7f07caa00640 1 -- 172.21.15.134:0/995782804 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7f07d0031ab0 con 0x56218dfc7390 2024-07-26T11:26:50.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.752+0000 7f07f2aa7740 10 client.26942 fetch_fsmap learned FSMap version 34 2024-07-26T11:26:50.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.752+0000 7f07f2aa7740 1 -- 172.21.15.134:0/995782804 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x56218e424110 con 0x56218dfc7390 2024-07-26T11:26:50.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.752+0000 7f07caa00640 1 -- 172.21.15.134:0/995782804 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07dc0e1f40 con 0x56218dfc7390 2024-07-26T11:26:50.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.752+0000 7f07caa00640 1 -- 172.21.15.134:0/995782804 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 34) ==== 8386+0+0 (secure 0 0 0) 0x7f07d003d420 con 0x56218dfc7390 2024-07-26T11:26:50.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.752+0000 7f07f2aa7740 10 client.26942 fetch_fsmap finished waiting for FSMap version 34 2024-07-26T11:26:50.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.752+0000 7f07f2aa7740 10 client.26942 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-07-26T11:26:50.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.752+0000 7f07f2aa7740 20 client.26942 populate_metadata read hostname 'smithi134' 2024-07-26T11:26:50.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.753+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/995782804 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x56218df7bda0 0x56218e426950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:26:50.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.753+0000 7f07f2aa7740 4 client.26942 mds_command: new command op to 24703 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:26:50.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.753+0000 7f07f2aa7740 1 -- 172.21.15.134:0/995782804 --> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56218e2fed20 con 0x56218df7bda0 2024-07-26T11:26:50.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.753+0000 7f07e9400640 1 --2- 172.21.15.134:0/995782804 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x56218df7bda0 0x56218e426950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:26:50.756 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.754+0000 7f07e9400640 1 --2- 172.21.15.134:0/995782804 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x56218df7bda0 0x56218e426950 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e007de2348f5d70d server_cookie=517b26e4928185ba in_seq=0 out_seq=0 2024-07-26T11:26:50.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.755+0000 7f07caa00640 10 client.26942 ms_handle_connect on v2:172.21.15.193:6838/3985555366 2024-07-26T11:26:50.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.755+0000 7f07caa00640 1 -- 172.21.15.134:0/995782804 <== mds.2 v2:172.21.15.193:6838/3985555366 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56218e2fed20 con 0x56218df7bda0 2024-07-26T11:26:50.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.755+0000 7f07caa00640 10 client.26942 handle_command_reply: tid=0 2024-07-26T11:26:50.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.756+0000 7f07f2aa7740 1 client.26942 shutdown 2024-07-26T11:26:50.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.756+0000 7f07f2aa7740 1 -- 172.21.15.134:0/995782804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfc7390 msgr2=0x56218e3ade70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:50.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.756+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/995782804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfc7390 0x56218e3ade70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f07d0009230 tx=0x7f07d0003730 comp rx=0 tx=0).stop 2024-07-26T11:26:50.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.757+0000 7f07f2aa7740 1 -- 172.21.15.134:0/995782804 shutdown_connections 2024-07-26T11:26:50.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.757+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/995782804 >> [v2:172.21.15.193:6838/3985555366,v1:172.21.15.193:6839/3985555366] conn(0x56218df7bda0 0x56218e426950 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.757+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/995782804 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56218dfe8070 0x56218e3ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.757+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/995782804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56218dfc7390 0x56218e3ade70 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.757+0000 7f07f2aa7740 1 --2- 172.21.15.134:0/995782804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56218dfbf1b0 0x56218e3ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.759 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.757+0000 7f07f2aa7740 1 -- 172.21.15.134:0/995782804 >> 172.21.15.134:0/995782804 conn(0x56218dfdc460 msgr2=0x56218e38e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:50.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.757+0000 7f07caa00640 0 client.26942 ms_handle_reset on v2:172.21.15.193:6838/3985555366 2024-07-26T11:26:50.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.757+0000 7f07f2aa7740 1 -- 172.21.15.134:0/995782804 shutdown_connections 2024-07-26T11:26:50.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.758+0000 7f07f2aa7740 1 -- 172.21.15.134:0/995782804 wait complete. 2024-07-26T11:26:50.765 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:26:50.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.759+0000 7f07f2aa7740 20 client.26942 trim_cache size 0 max 16384 2024-07-26T11:26:50.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.760+0000 7f07ebe00640 1 -- 172.21.15.134:0/2233822680 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f07cc0b3dc0 msgr2=0x7f07cc0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:50.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.760+0000 7f07ebe00640 1 --2- 172.21.15.134:0/2233822680 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f07cc0b3dc0 0x7f07cc0b62d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f07d8009c90 tx=0x7f07d8005f90 comp rx=0 tx=0).stop 2024-07-26T11:26:50.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.760+0000 7f07ebe00640 1 -- 172.21.15.134:0/2233822680 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07e410fcb0 msgr2=0x7f07e41a5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:50.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.760+0000 7f07ebe00640 1 --2- 172.21.15.134:0/2233822680 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07e410fcb0 0x7f07e41a5c40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f07dc00be00 tx=0x7f07dc00bef0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.760+0000 7f07ebe00640 1 -- 172.21.15.134:0/2233822680 shutdown_connections 2024-07-26T11:26:50.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.760+0000 7f07ebe00640 1 --2- 172.21.15.134:0/2233822680 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f07cc0b3dc0 0x7f07cc0b62d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.760+0000 7f07ebe00640 1 --2- 172.21.15.134:0/2233822680 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f07e410fcb0 0x7f07e41a5c40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.760+0000 7f07ebe00640 1 --2- 172.21.15.134:0/2233822680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07e410f360 0x7f07e41a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.760+0000 7f07ebe00640 1 --2- 172.21.15.134:0/2233822680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f07e4109a70 0x7f07e41a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:50.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.760+0000 7f07ebe00640 1 -- 172.21.15.134:0/2233822680 >> 172.21.15.134:0/2233822680 conn(0x7f07e4079040 msgr2=0x7f07e40796f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:50.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.760+0000 7f07ebe00640 1 -- 172.21.15.134:0/2233822680 shutdown_connections 2024-07-26T11:26:50.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:26:50.760+0000 7f07ebe00640 1 -- 172.21.15.134:0/2233822680 wait complete. 2024-07-26T11:26:51.678 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:51 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:26:51.678 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:51 smithi134 ceph-mon[31966]: pgmap v817: 129 pgs: 129 active+clean; 57 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 188 MiB/s wr, 1.69k op/s 2024-07-26T11:26:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.689+0000 7fba5ea00640 1 -- 172.21.15.134:0/3582296678 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5578a17a51c0 con 0x5578a141f1c0 2024-07-26T11:26:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.689+0000 7fba5ea00640 10 client.26944 handle_command_reply: tid=0 2024-07-26T11:26:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.690+0000 7fba86bb4740 1 client.26944 shutdown 2024-07-26T11:26:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.690+0000 7fba86bb4740 1 -- 172.21.15.134:0/3582296678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a14622f0 msgr2=0x5578a1846f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.690+0000 7fba86bb4740 1 --2- 172.21.15.134:0/3582296678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a14622f0 0x5578a1846f10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fba6c0094b0 tx=0x7fba6c0c5f90 comp rx=0 tx=0).stop 2024-07-26T11:26:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.690+0000 7fba86bb4740 1 -- 172.21.15.134:0/3582296678 shutdown_connections 2024-07-26T11:26:51.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.690+0000 7fba86bb4740 1 --2- 172.21.15.134:0/3582296678 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5578a141f1c0 0x5578a18b68a0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.690+0000 7fba86bb4740 1 --2- 172.21.15.134:0/3582296678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5578a148b1b0 0x5578a18419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.690+0000 7fba86bb4740 1 --2- 172.21.15.134:0/3582296678 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5578a146a4d0 0x5578a1841490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.690+0000 7fba86bb4740 1 --2- 172.21.15.134:0/3582296678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5578a14622f0 0x5578a1846f10 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.690+0000 7fba86bb4740 1 -- 172.21.15.134:0/3582296678 >> 172.21.15.134:0/3582296678 conn(0x5578a147f300 msgr2=0x5578a182c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.690+0000 7fba5ea00640 0 client.26944 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:26:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.691+0000 7fba86bb4740 1 -- 172.21.15.134:0/3582296678 shutdown_connections 2024-07-26T11:26:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.691+0000 7fba86bb4740 1 -- 172.21.15.134:0/3582296678 wait complete. 2024-07-26T11:26:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.691+0000 7fba86bb4740 20 client.26944 trim_cache size 0 max 16384 2024-07-26T11:26:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:51 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:26:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:51 smithi189 ceph-mon[36790]: pgmap v817: 129 pgs: 129 active+clean; 57 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 188 MiB/s wr, 1.69k op/s 2024-07-26T11:26:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:26:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:26:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:26:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:26:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.692+0000 7fba7fa00640 1 -- 172.21.15.134:0/2969532314 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fba600b3e90 msgr2=0x7fba600b63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.692+0000 7fba7fa00640 1 --2- 172.21.15.134:0/2969532314 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fba600b3e90 0x7fba600b63a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fba7000ab10 tx=0x7fba70009250 comp rx=0 tx=0).stop 2024-07-26T11:26:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.692+0000 7fba7fa00640 1 -- 172.21.15.134:0/2969532314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba78109fd0 msgr2=0x7fba781a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:26:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.692+0000 7fba7fa00640 1 --2- 172.21.15.134:0/2969532314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba78109fd0 0x7fba781a1170 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fba6c005940 tx=0x7fba6c003fd0 comp rx=0 tx=0).stop 2024-07-26T11:26:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.693+0000 7fba7fa00640 1 -- 172.21.15.134:0/2969532314 shutdown_connections 2024-07-26T11:26:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.693+0000 7fba7fa00640 1 --2- 172.21.15.134:0/2969532314 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fba600b3e90 0x7fba600b63a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.693+0000 7fba7fa00640 1 --2- 172.21.15.134:0/2969532314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba78115fd0 0x7fba781a81d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.693+0000 7fba7fa00640 1 --2- 172.21.15.134:0/2969532314 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fba7810a920 0x7fba781a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.693+0000 7fba7fa00640 1 --2- 172.21.15.134:0/2969532314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fba78109fd0 0x7fba781a1170 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:26:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.693+0000 7fba7fa00640 1 -- 172.21.15.134:0/2969532314 >> 172.21.15.134:0/2969532314 conn(0x7fba78070800 msgr2=0x7fba78071750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:26:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.693+0000 7fba7fa00640 1 -- 172.21.15.134:0/2969532314 shutdown_connections 2024-07-26T11:26:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:26:51.693+0000 7fba7fa00640 1 -- 172.21.15.134:0/2969532314 wait complete. 2024-07-26T11:26:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:51 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:26:51.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:51 smithi193 ceph-mon[37473]: pgmap v817: 129 pgs: 129 active+clean; 57 GiB data, 123 GiB used, 950 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 188 MiB/s wr, 1.69k op/s 2024-07-26T11:26:52.329 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:52 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:26:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:52 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:26:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:52 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:26:53.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:53 smithi134 ceph-mon[31966]: pgmap v818: 129 pgs: 129 active+clean; 57 GiB data, 124 GiB used, 949 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 192 MiB/s wr, 1.69k op/s 2024-07-26T11:26:53.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:53 smithi189 ceph-mon[36790]: pgmap v818: 129 pgs: 129 active+clean; 57 GiB data, 124 GiB used, 949 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 192 MiB/s wr, 1.69k op/s 2024-07-26T11:26:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:53 smithi193 ceph-mon[37473]: pgmap v818: 129 pgs: 129 active+clean; 57 GiB data, 124 GiB used, 949 GiB / 1.0 TiB avail; 2.0 MiB/s rd, 192 MiB/s wr, 1.69k op/s 2024-07-26T11:26:53.911 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:1 damage ls 2024-07-26T11:26:54.746 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:26:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:54 smithi134 ceph-mon[31966]: pgmap v819: 129 pgs: 129 active+clean; 58 GiB data, 124 GiB used, 948 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 192 MiB/s wr, 1.62k op/s 2024-07-26T11:26:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:54 smithi193 ceph-mon[37473]: pgmap v819: 129 pgs: 129 active+clean; 58 GiB data, 124 GiB used, 948 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 192 MiB/s wr, 1.62k op/s 2024-07-26T11:26:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:54 smithi189 ceph-mon[36790]: pgmap v819: 129 pgs: 129 active+clean; 58 GiB data, 124 GiB used, 948 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 192 MiB/s wr, 1.62k op/s 2024-07-26T11:26:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:56 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:26:57.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:56 smithi193 ceph-mon[37473]: pgmap v820: 129 pgs: 129 active+clean; 58 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 190 MiB/s wr, 1.54k op/s 2024-07-26T11:26:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:56 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:26:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:56 smithi134 ceph-mon[31966]: pgmap v820: 129 pgs: 129 active+clean; 58 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 190 MiB/s wr, 1.54k op/s 2024-07-26T11:26:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:56 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:26:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:56 smithi189 ceph-mon[36790]: pgmap v820: 129 pgs: 129 active+clean; 58 GiB data, 125 GiB used, 948 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 190 MiB/s wr, 1.54k op/s 2024-07-26T11:26:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:26:58 smithi189 ceph-mon[36790]: pgmap v821: 129 pgs: 129 active+clean; 58 GiB data, 125 GiB used, 947 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 169 MiB/s wr, 1.26k op/s 2024-07-26T11:26:59.200 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:26:58 smithi134 ceph-mon[31966]: pgmap v821: 129 pgs: 129 active+clean; 58 GiB data, 125 GiB used, 947 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 169 MiB/s wr, 1.26k op/s 2024-07-26T11:26:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:26:58 smithi193 ceph-mon[37473]: pgmap v821: 129 pgs: 129 active+clean; 58 GiB data, 125 GiB used, 947 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 169 MiB/s wr, 1.26k op/s 2024-07-26T11:27:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:00 smithi189 ceph-mon[36790]: pgmap v822: 129 pgs: 129 active+clean; 59 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 2.1 MiB/s rd, 196 MiB/s wr, 1.35k op/s 2024-07-26T11:27:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:00 smithi193 ceph-mon[37473]: pgmap v822: 129 pgs: 129 active+clean; 59 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 2.1 MiB/s rd, 196 MiB/s wr, 1.35k op/s 2024-07-26T11:27:01.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:00 smithi134 ceph-mon[31966]: pgmap v822: 129 pgs: 129 active+clean; 59 GiB data, 126 GiB used, 947 GiB / 1.0 TiB avail; 2.1 MiB/s rd, 196 MiB/s wr, 1.35k op/s 2024-07-26T11:27:02.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.723+0000 7fd8d4a00640 1 -- 172.21.15.134:0/2787742334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8d0108d20 msgr2=0x7fd8d0109130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:02.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.723+0000 7fd8d4a00640 1 --2- 172.21.15.134:0/2787742334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8d0108d20 0x7fd8d0109130 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc009900 tx=0x7fd8bc02a410 comp rx=0 tx=0).stop 2024-07-26T11:27:02.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.725+0000 7fd8d4a00640 1 -- 172.21.15.134:0/2787742334 shutdown_connections 2024-07-26T11:27:02.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.726+0000 7fd8d4a00640 1 --2- 172.21.15.134:0/2787742334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8d0108d20 0x7fd8d0109130 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.726+0000 7fd8d4a00640 1 --2- 172.21.15.134:0/2787742334 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8d0109670 0x7fd8d010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.726+0000 7fd8d4a00640 1 --2- 172.21.15.134:0/2787742334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d01033e0 0x7fd8d01037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.726+0000 7fd8d4a00640 1 -- 172.21.15.134:0/2787742334 >> 172.21.15.134:0/2787742334 conn(0x7fd8d00fee70 msgr2=0x7fd8d01012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:02.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.726+0000 7fd8d4a00640 1 -- 172.21.15.134:0/2787742334 shutdown_connections 2024-07-26T11:27:02.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.726+0000 7fd8d4a00640 1 -- 172.21.15.134:0/2787742334 wait complete. 2024-07-26T11:27:02.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.728+0000 7fd8d4a00640 1 Processor -- start 2024-07-26T11:27:02.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.729+0000 7fd8d4a00640 1 -- start start 2024-07-26T11:27:02.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.730+0000 7fd8d4a00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8d01033e0 0x7fd8d0198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.730+0000 7fd8d4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d0108d20 0x7fd8d0198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.731+0000 7fd8d4a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8d0109670 0x7fd8d019f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.731+0000 7fd8cde00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d0108d20 0x7fd8d0198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.731+0000 7fd8cde00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d0108d20 0x7fd8d0198d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43974/0 (socket says 172.21.15.134:43974) 2024-07-26T11:27:02.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.731+0000 7fd8cde00640 1 -- 172.21.15.134:0/455240690 learned_addr learned my addr 172.21.15.134:0/455240690 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:02.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.732+0000 7fd8cf200640 1 --2- 172.21.15.134:0/455240690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8d0109670 0x7fd8d019f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.732+0000 7fd8d4a00640 1 -- 172.21.15.134:0/455240690 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d0113350 con 0x7fd8d01033e0 2024-07-26T11:27:02.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.732+0000 7fd8d4a00640 1 -- 172.21.15.134:0/455240690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d01131d0 con 0x7fd8d0108d20 2024-07-26T11:27:02.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.732+0000 7fd8d4a00640 1 -- 172.21.15.134:0/455240690 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d01134d0 con 0x7fd8d0109670 2024-07-26T11:27:02.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.733+0000 7fd8ce800640 1 --2- 172.21.15.134:0/455240690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8d01033e0 0x7fd8d0198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.733+0000 7fd8cf200640 1 -- 172.21.15.134:0/455240690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d0108d20 msgr2=0x7fd8d0198d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:02.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.733+0000 7fd8cf200640 1 --2- 172.21.15.134:0/455240690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d0108d20 0x7fd8d0198d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.734+0000 7fd8cf200640 1 -- 172.21.15.134:0/455240690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8d01033e0 msgr2=0x7fd8d0198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:02.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.734+0000 7fd8cf200640 1 --2- 172.21.15.134:0/455240690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8d01033e0 0x7fd8d0198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.734+0000 7fd8cf200640 1 -- 172.21.15.134:0/455240690 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d019fdc0 con 0x7fd8d0109670 2024-07-26T11:27:02.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.734+0000 7fd8ce800640 1 --2- 172.21.15.134:0/455240690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8d01033e0 0x7fd8d0198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:27:02.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.735+0000 7fd8cf200640 1 --2- 172.21.15.134:0/455240690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8d0109670 0x7fd8d019f880 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800da10 tx=0x7fd8c800ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:02.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.735+0000 7fd8c7400640 1 -- 172.21.15.134:0/455240690 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c800bd70 con 0x7fd8d0109670 2024-07-26T11:27:02.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.735+0000 7fd8d4a00640 1 -- 172.21.15.134:0/455240690 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8d01a00b0 con 0x7fd8d0109670 2024-07-26T11:27:02.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.735+0000 7fd8d4a00640 1 -- 172.21.15.134:0/455240690 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8d01a05c0 con 0x7fd8d0109670 2024-07-26T11:27:02.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.735+0000 7fd8c7400640 1 -- 172.21.15.134:0/455240690 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8c8003e20 con 0x7fd8d0109670 2024-07-26T11:27:02.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.735+0000 7fd8c7400640 1 -- 172.21.15.134:0/455240690 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c80106c0 con 0x7fd8d0109670 2024-07-26T11:27:02.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.743+0000 7fd8c7400640 1 -- 172.21.15.134:0/455240690 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7fd8c8003970 con 0x7fd8d0109670 2024-07-26T11:27:02.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.745+0000 7fd8c7400640 1 --2- 172.21.15.134:0/455240690 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8b40b3e70 0x7fd8b40b6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.746+0000 7fd8c7400640 1 -- 172.21.15.134:0/455240690 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9495+0+0 (secure 0 0 0) 0x7fd8c80e4510 con 0x7fd8d0109670 2024-07-26T11:27:02.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.747+0000 7fd8ce800640 1 --2- 172.21.15.134:0/455240690 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8b40b3e70 0x7fd8b40b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.750+0000 7fd8ce800640 1 --2- 172.21.15.134:0/455240690 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8b40b3e70 0x7fd8b40b6380 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0005fd0 tx=0x7fd8c0005850 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:02.781 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.768+0000 7fd8dc023740 1 Processor -- start 2024-07-26T11:27:02.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.768+0000 7fd8dc023740 1 -- start start 2024-07-26T11:27:02.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.769+0000 7fd8dc023740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cf1070 0x556504096010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.769+0000 7fd8dc023740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cc81b0 0x556504096550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.769+0000 7fd8dc023740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cd0390 0x556504096a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.769+0000 7fd8dc023740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55650400cbd0 con 0x556503cd0390 2024-07-26T11:27:02.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.769+0000 7fd8dc023740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556503ffc0d0 con 0x556503cc81b0 2024-07-26T11:27:02.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.769+0000 7fd8dc023740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556504000110 con 0x556503cf1070 2024-07-26T11:27:02.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.770+0000 7fd8cf200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cd0390 0x556504096a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.770+0000 7fd8ce800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cf1070 0x556504096010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.772+0000 7fd8ce800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cf1070 0x556504096010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34008/0 (socket says 172.21.15.134:34008) 2024-07-26T11:27:02.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.772+0000 7fd8ce800640 1 -- 172.21.15.134:0/430874698 learned_addr learned my addr 172.21.15.134:0/430874698 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:02.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.771+0000 7fd8cde00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cc81b0 0x556504096550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.771+0000 7fd8cf200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cd0390 0x556504096a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:41368/0 (socket says 172.21.15.134:41368) 2024-07-26T11:27:02.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.773+0000 7fd8ce800640 1 -- 172.21.15.134:0/430874698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cc81b0 msgr2=0x556504096550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:02.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.773+0000 7fd8ce800640 1 --2- 172.21.15.134:0/430874698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cc81b0 0x556504096550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.773+0000 7fd8ce800640 1 -- 172.21.15.134:0/430874698 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cd0390 msgr2=0x556504096a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:02.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.773+0000 7fd8ce800640 1 --2- 172.21.15.134:0/430874698 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cd0390 0x556504096a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.773+0000 7fd8ce800640 1 -- 172.21.15.134:0/430874698 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556503d5c650 con 0x556503cf1070 2024-07-26T11:27:02.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.774+0000 7fd8cde00640 1 --2- 172.21.15.134:0/430874698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cc81b0 0x556504096550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:27:02.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.774+0000 7fd8cf200640 1 --2- 172.21.15.134:0/430874698 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cd0390 0x556504096a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:27:02.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.778+0000 7fd8ce800640 1 --2- 172.21.15.134:0/430874698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cf1070 0x556504096010 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd8c000b9f0 tx=0x7fd8c0012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5834415cbc8edcf5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:02.783 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.778+0000 7fd8bb400640 1 -- 172.21.15.134:0/430874698 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c0022070 con 0x556503cf1070 2024-07-26T11:27:02.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.779+0000 7fd8bb400640 1 -- 172.21.15.134:0/430874698 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8c001eb00 con 0x556503cf1070 2024-07-26T11:27:02.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.779+0000 7fd8bb400640 1 -- 172.21.15.134:0/430874698 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c00109d0 con 0x556503cf1070 2024-07-26T11:27:02.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.780+0000 7fd8dc023740 1 -- 172.21.15.134:0/430874698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cf1070 msgr2=0x556504096010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:02.784 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.780+0000 7fd8dc023740 1 --2- 172.21.15.134:0/430874698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cf1070 0x556504096010 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd8c000b9f0 tx=0x7fd8c0012000 comp rx=0 tx=0).stop 2024-07-26T11:27:02.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.781+0000 7fd8dc023740 1 -- 172.21.15.134:0/430874698 shutdown_connections 2024-07-26T11:27:02.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.783+0000 7fd8dc023740 1 --2- 172.21.15.134:0/430874698 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cd0390 0x556504096a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.783+0000 7fd8dc023740 1 --2- 172.21.15.134:0/430874698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cc81b0 0x556504096550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.784+0000 7fd8dc023740 1 --2- 172.21.15.134:0/430874698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cf1070 0x556504096010 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.785+0000 7fd8dc023740 1 -- 172.21.15.134:0/430874698 >> 172.21.15.134:0/430874698 conn(0x556503ce5460 msgr2=0x556504093f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:02.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.785+0000 7fd8dc023740 1 -- 172.21.15.134:0/430874698 shutdown_connections 2024-07-26T11:27:02.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.785+0000 7fd8dc023740 1 -- 172.21.15.134:0/430874698 wait complete. 2024-07-26T11:27:02.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.786+0000 7fd8dc023740 1 -- start start 2024-07-26T11:27:02.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.786+0000 7fd8dc023740 1 Processor -- start 2024-07-26T11:27:02.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.799+0000 7fd8dc023740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cc81b0 0x5565040a1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.801+0000 7fd8dc023740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cd0390 0x5565040a1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.802+0000 7fd8dc023740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cf1070 0x5565040a2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.804+0000 7fd8dc023740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556503e5f700 con 0x556503cd0390 2024-07-26T11:27:02.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.804+0000 7fd8dc023740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556503f986a0 con 0x556503cf1070 2024-07-26T11:27:02.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.805+0000 7fd8cf200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cf1070 0x5565040a2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.806+0000 7fd8cf200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cf1070 0x5565040a2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43984/0 (socket says 172.21.15.134:43984) 2024-07-26T11:27:02.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.806+0000 7fd8cf200640 1 -- 172.21.15.134:0/2689449086 learned_addr learned my addr 172.21.15.134:0/2689449086 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:02.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.801+0000 7fd8ce800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cc81b0 0x5565040a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.809+0000 7fd8dc023740 1 -- 172.21.15.134:0/2689449086 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5565040a47f0 con 0x556503cc81b0 2024-07-26T11:27:02.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.810+0000 7fd8dc023740 10 client.0 _ll_register_callbacks cb 0x556503fa8150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:27:02.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.810+0000 7fd8cf200640 1 -- 172.21.15.134:0/2689449086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cc81b0 msgr2=0x5565040a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:02.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.810+0000 7fd8cf200640 1 --2- 172.21.15.134:0/2689449086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cc81b0 0x5565040a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.810+0000 7fd8cf200640 1 -- 172.21.15.134:0/2689449086 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cd0390 msgr2=0x5565040a1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:02.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.810+0000 7fd8cf200640 1 --2- 172.21.15.134:0/2689449086 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cd0390 0x5565040a1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.810+0000 7fd8cf200640 1 -- 172.21.15.134:0/2689449086 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556503faf840 con 0x556503cf1070 2024-07-26T11:27:02.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.811+0000 7fd8cf200640 1 --2- 172.21.15.134:0/2689449086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cf1070 0x5565040a2090 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd8c80a7f60 tx=0x7fd8c8003d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=154655c6a439b65a server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:02.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.811+0000 7fd8dc023740 1 -- 172.21.15.134:0/2689449086 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8c001eb00 con 0x556503cf1070 2024-07-26T11:27:02.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.811+0000 7fd8afe00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:27:02.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.811+0000 7fd8afe00640 1 -- 172.21.15.134:0/2689449086 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c80aacf0 con 0x556503cf1070 2024-07-26T11:27:02.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.811+0000 7fd8afe00640 1 -- 172.21.15.134:0/2689449086 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9495+0+0 (secure 0 0 0) 0x7fd8c80e1ab0 con 0x556503cf1070 2024-07-26T11:27:02.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.812+0000 7fd8afe00640 1 -- 172.21.15.134:0/2689449086 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd8c80e1cb0 con 0x556503cf1070 2024-07-26T11:27:02.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.812+0000 7fd8dc023740 10 client.26954 fetch_fsmap learned FSMap version 36 2024-07-26T11:27:02.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.812+0000 7fd8dc023740 1 -- 172.21.15.134:0/2689449086 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({fsmap=36}) -- 0x5565040c1450 con 0x556503cf1070 2024-07-26T11:27:02.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.812+0000 7fd8afe00640 1 -- 172.21.15.134:0/2689449086 <== mon.2 v2:172.21.15.189:3300/0 4 ==== fsmap(e 36) ==== 8398+0+0 (secure 0 0 0) 0x7fd8c8024060 con 0x556503cf1070 2024-07-26T11:27:02.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.812+0000 7fd8dc023740 10 client.26954 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T11:27:02.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.812+0000 7fd8dc023740 10 client.26954 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-07-26T11:27:02.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.812+0000 7fd8dc023740 20 client.26954 populate_metadata read hostname 'smithi134' 2024-07-26T11:27:02.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.812+0000 7fd8dc023740 1 --2- 172.21.15.134:0/2689449086 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x556503caf9f0 0x5565040c4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.812+0000 7fd8dc023740 4 client.26954 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:27:02.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.812+0000 7fd8dc023740 1 -- 172.21.15.134:0/2689449086 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55650400eb20 con 0x556503caf9f0 2024-07-26T11:27:02.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.813+0000 7fd8ce800640 1 --2- 172.21.15.134:0/2689449086 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x556503caf9f0 0x5565040c4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.814+0000 7fd8afe00640 10 client.26954 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:27:02.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.814+0000 7fd8ce800640 1 --2- 172.21.15.134:0/2689449086 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x556503caf9f0 0x5565040c4df0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=68081fb7ba95c47f server_cookie=c98351f7f2eaf8e1 in_seq=0 out_seq=0 2024-07-26T11:27:02.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.815+0000 7fd8afe00640 1 -- 172.21.15.134:0/2689449086 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55650400eb20 con 0x556503caf9f0 2024-07-26T11:27:02.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.815+0000 7fd8afe00640 10 client.26954 handle_command_reply: tid=0 2024-07-26T11:27:02.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.815+0000 7fd8dc023740 1 client.26954 shutdown 2024-07-26T11:27:02.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.817+0000 7fd8dc023740 1 -- 172.21.15.134:0/2689449086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cf1070 msgr2=0x5565040a2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:02.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.817+0000 7fd8dc023740 1 --2- 172.21.15.134:0/2689449086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cf1070 0x5565040a2090 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd8c80a7f60 tx=0x7fd8c8003d20 comp rx=0 tx=0).stop 2024-07-26T11:27:02.821 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.818+0000 7fd8dc023740 1 -- 172.21.15.134:0/2689449086 shutdown_connections 2024-07-26T11:27:02.821 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.818+0000 7fd8dc023740 1 --2- 172.21.15.134:0/2689449086 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x556503caf9f0 0x5565040c4df0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.821 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.818+0000 7fd8afe00640 0 client.26954 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:27:02.821 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.818+0000 7fd8dc023740 1 --2- 172.21.15.134:0/2689449086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cf1070 0x5565040a2090 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.818+0000 7fd8dc023740 1 --2- 172.21.15.134:0/2689449086 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cd0390 0x5565040a1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.818+0000 7fd8dc023740 1 --2- 172.21.15.134:0/2689449086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cc81b0 0x5565040a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.818+0000 7fd8dc023740 1 -- 172.21.15.134:0/2689449086 >> 172.21.15.134:0/2689449086 conn(0x556503ce5460 msgr2=0x5565040991a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:02.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.818+0000 7fd8dc023740 1 -- 172.21.15.134:0/2689449086 shutdown_connections 2024-07-26T11:27:02.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.818+0000 7fd8dc023740 1 -- 172.21.15.134:0/2689449086 wait complete. 2024-07-26T11:27:02.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.818+0000 7fd8dc023740 20 client.26954 trim_cache size 0 max 16384 2024-07-26T11:27:02.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.849+0000 7fd8dc023740 1 Processor -- start 2024-07-26T11:27:02.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.850+0000 7fd8dc023740 1 -- start start 2024-07-26T11:27:02.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.850+0000 7fd8dc023740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cc81b0 0x556504093010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.850+0000 7fd8dc023740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cd0390 0x5565040a1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.850+0000 7fd8dc023740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cf1070 0x5565040a3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.851+0000 7fd8dc023740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b0010100 con 0x556503cf1070 2024-07-26T11:27:02.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.851+0000 7fd8dc023740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b0011d00 con 0x556503cd0390 2024-07-26T11:27:02.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.851+0000 7fd8ce800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cc81b0 0x556504093010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.851+0000 7fd8dc023740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b00107d0 con 0x556503cc81b0 2024-07-26T11:27:02.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.851+0000 7fd8cde00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cd0390 0x5565040a1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.852+0000 7fd8cde00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cd0390 0x5565040a1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43996/0 (socket says 172.21.15.134:43996) 2024-07-26T11:27:02.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.852+0000 7fd8cde00640 1 -- 172.21.15.134:0/138475066 learned_addr learned my addr 172.21.15.134:0/138475066 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:02.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.852+0000 7fd8cf200640 1 --2- 172.21.15.134:0/138475066 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cf1070 0x5565040a3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.853+0000 7fd8ce800640 1 -- 172.21.15.134:0/138475066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cd0390 msgr2=0x5565040a1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:02.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.853+0000 7fd8ce800640 1 --2- 172.21.15.134:0/138475066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cd0390 0x5565040a1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.853+0000 7fd8ce800640 1 -- 172.21.15.134:0/138475066 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cf1070 msgr2=0x5565040a3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:02.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.853+0000 7fd8ce800640 1 --2- 172.21.15.134:0/138475066 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cf1070 0x5565040a3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.853+0000 7fd8ce800640 1 -- 172.21.15.134:0/138475066 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556504098e70 con 0x556503cc81b0 2024-07-26T11:27:02.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.855+0000 7fd8ce800640 1 --2- 172.21.15.134:0/138475066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cc81b0 0x556504093010 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0011f50 tx=0x7fd8c000ba60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55554e5651c046a2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:02.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.856+0000 7fd8baa00640 1 -- 172.21.15.134:0/138475066 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c0022040 con 0x556503cc81b0 2024-07-26T11:27:02.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.856+0000 7fd8baa00640 1 -- 172.21.15.134:0/138475066 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8c0005040 con 0x556503cc81b0 2024-07-26T11:27:02.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.856+0000 7fd8baa00640 1 -- 172.21.15.134:0/138475066 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c00049e0 con 0x556503cc81b0 2024-07-26T11:27:02.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.858+0000 7fd8dc023740 1 -- 172.21.15.134:0/138475066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cc81b0 msgr2=0x556504093010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:02.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.858+0000 7fd8dc023740 1 --2- 172.21.15.134:0/138475066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cc81b0 0x556504093010 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0011f50 tx=0x7fd8c000ba60 comp rx=0 tx=0).stop 2024-07-26T11:27:02.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.858+0000 7fd8dc023740 1 -- 172.21.15.134:0/138475066 shutdown_connections 2024-07-26T11:27:02.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.858+0000 7fd8dc023740 1 --2- 172.21.15.134:0/138475066 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cf1070 0x5565040a3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.858+0000 7fd8dc023740 1 --2- 172.21.15.134:0/138475066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503cd0390 0x5565040a1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.858+0000 7fd8dc023740 1 --2- 172.21.15.134:0/138475066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cc81b0 0x556504093010 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.858+0000 7fd8dc023740 1 -- 172.21.15.134:0/138475066 >> 172.21.15.134:0/138475066 conn(0x556503ce5460 msgr2=0x5565040a2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:02.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.858+0000 7fd8dc023740 1 -- 172.21.15.134:0/138475066 shutdown_connections 2024-07-26T11:27:02.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.859+0000 7fd8dc023740 1 -- 172.21.15.134:0/138475066 wait complete. 2024-07-26T11:27:02.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.859+0000 7fd8dc023740 1 -- start start 2024-07-26T11:27:02.862 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.860+0000 7fd8dc023740 1 Processor -- start 2024-07-26T11:27:02.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.862+0000 7fd8dc023740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cd0390 0x5565040aa0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.862+0000 7fd8dc023740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cf1070 0x5565040a75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.862+0000 7fd8dc023740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503c84da0 0x5565040a7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.862+0000 7fd8dc023740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b0011570 con 0x556503cf1070 2024-07-26T11:27:02.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.862+0000 7fd8dc023740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b0010ea0 con 0x556503c84da0 2024-07-26T11:27:02.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.862+0000 7fd8dc023740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b000e100 con 0x556503cd0390 2024-07-26T11:27:02.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.862+0000 7fd8dc023740 10 client.0 _ll_register_callbacks cb 0x7fd8c80aa460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:27:02.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.863+0000 7fd8cf200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503c84da0 0x5565040a7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.865+0000 7fd8cf200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503c84da0 0x5565040a7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:44008/0 (socket says 172.21.15.134:44008) 2024-07-26T11:27:02.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.865+0000 7fd8cf200640 1 -- 172.21.15.134:0/81688794 learned_addr learned my addr 172.21.15.134:0/81688794 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:02.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.864+0000 7fd8cde00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cf1070 0x5565040a75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.864+0000 7fd8ce800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cd0390 0x5565040aa0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.866+0000 7fd8ce800640 1 -- 172.21.15.134:0/81688794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503c84da0 msgr2=0x5565040a7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:02.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.866+0000 7fd8ce800640 1 --2- 172.21.15.134:0/81688794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503c84da0 0x5565040a7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.867+0000 7fd8ce800640 1 -- 172.21.15.134:0/81688794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cf1070 msgr2=0x5565040a75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:02.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.867+0000 7fd8ce800640 1 --2- 172.21.15.134:0/81688794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cf1070 0x5565040a75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.875 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.867+0000 7fd8ce800640 1 -- 172.21.15.134:0/81688794 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565040a8240 con 0x556503cd0390 2024-07-26T11:27:02.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.867+0000 7fd8cf200640 1 --2- 172.21.15.134:0/81688794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503c84da0 0x5565040a7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:27:02.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.867+0000 7fd8ce800640 1 --2- 172.21.15.134:0/81688794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cd0390 0x5565040aa0f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x556504094530 tx=0x7fd8c00054a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f47d5a32d167f8aa server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:02.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.867+0000 7fd8afe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:27:02.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.867+0000 7fd8cde00640 1 --2- 172.21.15.134:0/81688794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cf1070 0x5565040a75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:27:02.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.868+0000 7fd8afe00640 1 -- 172.21.15.134:0/81688794 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c0045060 con 0x556503cd0390 2024-07-26T11:27:02.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.868+0000 7fd8afe00640 1 -- 172.21.15.134:0/81688794 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9495+0+0 (secure 0 0 0) 0x7fd8c001e860 con 0x556503cd0390 2024-07-26T11:27:02.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.868+0000 7fd8dc023740 1 -- 172.21.15.134:0/81688794 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8c0005040 con 0x556503cd0390 2024-07-26T11:27:02.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.869+0000 7fd8afe00640 1 -- 172.21.15.134:0/81688794 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fd8c004c050 con 0x556503cd0390 2024-07-26T11:27:02.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.870+0000 7fd8dc023740 10 client.26974 fetch_fsmap learned FSMap version 36 2024-07-26T11:27:02.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.870+0000 7fd8dc023740 1 -- 172.21.15.134:0/81688794 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55650412d480 con 0x556503cd0390 2024-07-26T11:27:02.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.870+0000 7fd8afe00640 1 -- 172.21.15.134:0/81688794 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8c0004c10 con 0x556503cd0390 2024-07-26T11:27:02.878 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.870+0000 7fd8afe00640 1 -- 172.21.15.134:0/81688794 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 36) ==== 8398+0+0 (secure 0 0 0) 0x7fd8c0010830 con 0x556503cd0390 2024-07-26T11:27:02.878 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.871+0000 7fd8dc023740 10 client.26974 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T11:27:02.878 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.871+0000 7fd8dc023740 10 client.26974 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-07-26T11:27:02.878 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.871+0000 7fd8dc023740 20 client.26974 populate_metadata read hostname 'smithi134' 2024-07-26T11:27:02.878 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.871+0000 7fd8dc023740 1 --2- 172.21.15.134:0/81688794 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55650412d8b0 0x55650412fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:02.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.871+0000 7fd8dc023740 4 client.26974 mds_command: new command op to 24719 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:27:02.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.871+0000 7fd8dc023740 1 -- 172.21.15.134:0/81688794 --> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fd8c0005040 con 0x55650412d8b0 2024-07-26T11:27:02.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.872+0000 7fd8cde00640 1 --2- 172.21.15.134:0/81688794 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55650412d8b0 0x55650412fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:02.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.874+0000 7fd8afe00640 10 client.26974 ms_handle_connect on v2:172.21.15.189:6838/790931242 2024-07-26T11:27:02.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.874+0000 7fd8cde00640 1 --2- 172.21.15.134:0/81688794 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55650412d8b0 0x55650412fcd0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fb8f9d17b4ea851d server_cookie=df8522ca3d4d23cd in_seq=0 out_seq=0 2024-07-26T11:27:02.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.876+0000 7fd8afe00640 1 -- 172.21.15.134:0/81688794 <== mds.1 v2:172.21.15.189:6838/790931242 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fd8c0005040 con 0x55650412d8b0 2024-07-26T11:27:02.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.876+0000 7fd8afe00640 10 client.26974 handle_command_reply: tid=0 2024-07-26T11:27:02.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.876+0000 7fd8dc023740 1 client.26974 shutdown 2024-07-26T11:27:02.885 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:27:02.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.877+0000 7fd8dc023740 1 -- 172.21.15.134:0/81688794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cd0390 msgr2=0x5565040aa0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:02.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.877+0000 7fd8dc023740 1 --2- 172.21.15.134:0/81688794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cd0390 0x5565040aa0f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x556504094530 tx=0x7fd8c00054a0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.878+0000 7fd8dc023740 1 -- 172.21.15.134:0/81688794 shutdown_connections 2024-07-26T11:27:02.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.878+0000 7fd8dc023740 1 --2- 172.21.15.134:0/81688794 >> [v2:172.21.15.189:6838/790931242,v1:172.21.15.189:6839/790931242] conn(0x55650412d8b0 0x55650412fcd0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.878+0000 7fd8dc023740 1 --2- 172.21.15.134:0/81688794 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556503c84da0 0x5565040a7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.878+0000 7fd8dc023740 1 --2- 172.21.15.134:0/81688794 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556503cf1070 0x5565040a75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.878+0000 7fd8dc023740 1 --2- 172.21.15.134:0/81688794 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556503cd0390 0x5565040aa0f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.878+0000 7fd8dc023740 1 -- 172.21.15.134:0/81688794 >> 172.21.15.134:0/81688794 conn(0x556503ce5460 msgr2=0x5565040a1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:02.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.878+0000 7fd8afe00640 0 client.26974 ms_handle_reset on v2:172.21.15.189:6838/790931242 2024-07-26T11:27:02.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.878+0000 7fd8dc023740 1 -- 172.21.15.134:0/81688794 shutdown_connections 2024-07-26T11:27:02.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.878+0000 7fd8dc023740 1 -- 172.21.15.134:0/81688794 wait complete. 2024-07-26T11:27:02.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.878+0000 7fd8dc023740 20 client.26974 trim_cache size 0 max 16384 2024-07-26T11:27:02.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.883+0000 7fd8d4a00640 1 -- 172.21.15.134:0/455240690 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8b40b3e70 msgr2=0x7fd8b40b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:02.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.883+0000 7fd8d4a00640 1 --2- 172.21.15.134:0/455240690 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8b40b3e70 0x7fd8b40b6380 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0005fd0 tx=0x7fd8c0005850 comp rx=0 tx=0).stop 2024-07-26T11:27:02.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.883+0000 7fd8d4a00640 1 -- 172.21.15.134:0/455240690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8d0109670 msgr2=0x7fd8d019f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:02.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.883+0000 7fd8d4a00640 1 --2- 172.21.15.134:0/455240690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8d0109670 0x7fd8d019f880 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800da10 tx=0x7fd8c800ded0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.884+0000 7fd8d4a00640 1 -- 172.21.15.134:0/455240690 shutdown_connections 2024-07-26T11:27:02.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.884+0000 7fd8d4a00640 1 --2- 172.21.15.134:0/455240690 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fd8b40b3e70 0x7fd8b40b6380 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.884+0000 7fd8d4a00640 1 --2- 172.21.15.134:0/455240690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8d0109670 0x7fd8d019f880 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.884+0000 7fd8d4a00640 1 --2- 172.21.15.134:0/455240690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8d0108d20 0x7fd8d0198d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.884+0000 7fd8d4a00640 1 --2- 172.21.15.134:0/455240690 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fd8d01033e0 0x7fd8d0198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:02.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.884+0000 7fd8d4a00640 1 -- 172.21.15.134:0/455240690 >> 172.21.15.134:0/455240690 conn(0x7fd8d00fee70 msgr2=0x7fd8d00ff280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:02.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.884+0000 7fd8d4a00640 1 -- 172.21.15.134:0/455240690 shutdown_connections 2024-07-26T11:27:02.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:02.884+0000 7fd8d4a00640 1 -- 172.21.15.134:0/455240690 wait complete. 2024-07-26T11:27:03.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:02 smithi134 ceph-mon[31966]: pgmap v823: 129 pgs: 129 active+clean; 59 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 184 MiB/s wr, 1.23k op/s 2024-07-26T11:27:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:02 smithi193 ceph-mon[37473]: pgmap v823: 129 pgs: 129 active+clean; 59 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 184 MiB/s wr, 1.23k op/s 2024-07-26T11:27:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:02 smithi189 ceph-mon[36790]: pgmap v823: 129 pgs: 129 active+clean; 59 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 184 MiB/s wr, 1.23k op/s 2024-07-26T11:27:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.059+0000 7ff8f4600640 1 -- 172.21.15.134:0/4283594567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8ec10a090 msgr2=0x7ff8ec10a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:05.061 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.059+0000 7ff8f4600640 1 --2- 172.21.15.134:0/4283594567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8ec10a090 0x7ff8ec10a4a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff8e4009ca0 tx=0x7ff8e402a8c0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.063+0000 7ff8f4600640 1 -- 172.21.15.134:0/4283594567 shutdown_connections 2024-07-26T11:27:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.063+0000 7ff8f4600640 1 --2- 172.21.15.134:0/4283594567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8ec10f430 0x7ff8ec10f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.063+0000 7ff8f4600640 1 --2- 172.21.15.134:0/4283594567 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8ec10fd80 0x7ff8ec1105a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.063+0000 7ff8f4600640 1 --2- 172.21.15.134:0/4283594567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8ec10a090 0x7ff8ec10a4a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.063+0000 7ff8f4600640 1 -- 172.21.15.134:0/4283594567 >> 172.21.15.134:0/4283594567 conn(0x7ff8ec070800 msgr2=0x7ff8ec070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:05.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.066+0000 7ff8f4600640 1 -- 172.21.15.134:0/4283594567 shutdown_connections 2024-07-26T11:27:05.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.067+0000 7ff8f4600640 1 -- 172.21.15.134:0/4283594567 wait complete. 2024-07-26T11:27:05.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.079+0000 7ff8f4600640 1 Processor -- start 2024-07-26T11:27:05.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.081+0000 7ff8f4600640 1 -- start start 2024-07-26T11:27:05.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.087+0000 7ff8f4600640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8ec10a090 0x7ff8ec1a8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.091 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.089+0000 7ff8f4600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8ec10f430 0x7ff8ec1a9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.092 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.090+0000 7ff8f4600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8ec10fd80 0x7ff8ec1ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.092 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.090+0000 7ff8f4600640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec122290 con 0x7ff8ec10a090 2024-07-26T11:27:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.090+0000 7ff8f4600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec122110 con 0x7ff8ec10fd80 2024-07-26T11:27:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.090+0000 7ff8f4600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec122410 con 0x7ff8ec10f430 2024-07-26T11:27:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.091+0000 7ff8f1200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8ec10f430 0x7ff8ec1a9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.091+0000 7ff8f1200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8ec10f430 0x7ff8ec1a9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34064/0 (socket says 172.21.15.134:34064) 2024-07-26T11:27:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.091+0000 7ff8f1200640 1 -- 172.21.15.134:0/1324673184 learned_addr learned my addr 172.21.15.134:0/1324673184 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:05.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.091+0000 7ff8f2600640 1 --2- 172.21.15.134:0/1324673184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8ec10fd80 0x7ff8ec1ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:05.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.092+0000 7ff8f1200640 1 -- 172.21.15.134:0/1324673184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8ec10fd80 msgr2=0x7ff8ec1ae5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:05.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.092+0000 7ff8f1200640 1 --2- 172.21.15.134:0/1324673184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8ec10fd80 0x7ff8ec1ae5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.092+0000 7ff8f1200640 1 -- 172.21.15.134:0/1324673184 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8ec10a090 msgr2=0x7ff8ec1a8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:05.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.092+0000 7ff8f1200640 1 --2- 172.21.15.134:0/1324673184 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8ec10a090 0x7ff8ec1a8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.092+0000 7ff8f1200640 1 -- 172.21.15.134:0/1324673184 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8ec1aeca0 con 0x7ff8ec10f430 2024-07-26T11:27:05.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.097+0000 7ff8f1200640 1 --2- 172.21.15.134:0/1324673184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8ec10f430 0x7ff8ec1a9310 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800e9c0 tx=0x7ff8d800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.097+0000 7ff8ea000640 1 -- 172.21.15.134:0/1324673184 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d800ccc0 con 0x7ff8ec10f430 2024-07-26T11:27:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.097+0000 7ff8ea000640 1 -- 172.21.15.134:0/1324673184 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8d80054d0 con 0x7ff8ec10f430 2024-07-26T11:27:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.097+0000 7ff8ea000640 1 -- 172.21.15.134:0/1324673184 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8010430 con 0x7ff8ec10f430 2024-07-26T11:27:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.097+0000 7ff8f4600640 1 -- 172.21.15.134:0/1324673184 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8ec1aef90 con 0x7ff8ec10f430 2024-07-26T11:27:05.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.097+0000 7ff8f4600640 1 -- 172.21.15.134:0/1324673184 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8ec1b5720 con 0x7ff8ec10f430 2024-07-26T11:27:05.128 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.123+0000 7ff8ea000640 1 -- 172.21.15.134:0/1324673184 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 25) ==== 151191+0+0 (secure 0 0 0) 0x7ff8d80105d0 con 0x7ff8ec10f430 2024-07-26T11:27:05.128 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.125+0000 7ff8fb7e0740 1 Processor -- start 2024-07-26T11:27:05.130 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.128+0000 7ff8ea000640 1 --2- 172.21.15.134:0/1324673184 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8d40b3d40 0x7ff8d40b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.130 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.128+0000 7ff8ea000640 1 -- 172.21.15.134:0/1324673184 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9495+0+0 (secure 0 0 0) 0x7ff8d80e4660 con 0x7ff8ec10f430 2024-07-26T11:27:05.130 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.129+0000 7ff8fb7e0740 1 -- start start 2024-07-26T11:27:05.131 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.129+0000 7ff8fb7e0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c60230fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.132 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.130+0000 7ff8fb7e0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f432f0 0x55c60230ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.132 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.130+0000 7ff8fb7e0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f4b4d0 0x55c6023104e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.132 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.130+0000 7ff8fb7e0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c602284270 con 0x55c601f432f0 2024-07-26T11:27:05.132 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.130+0000 7ff8fb7e0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c602273770 con 0x55c601f4b4d0 2024-07-26T11:27:05.132 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.130+0000 7ff8fb7e0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c6022777b0 con 0x55c601f6c1b0 2024-07-26T11:27:05.132 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.130+0000 7ff8f1c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c60230fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:05.132 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.130+0000 7ff8f1c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c60230fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34076/0 (socket says 172.21.15.134:34076) 2024-07-26T11:27:05.133 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.130+0000 7ff8f1c00640 1 -- 172.21.15.134:0/2094696772 learned_addr learned my addr 172.21.15.134:0/2094696772 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.131+0000 7ff8f2600640 1 --2- 172.21.15.134:0/2094696772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f4b4d0 0x55c6023104e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.131+0000 7ff8f2600640 1 --2- 172.21.15.134:0/1324673184 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8d40b3d40 0x7ff8d40b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.132+0000 7ff8f1c00640 1 -- 172.21.15.134:0/2094696772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f4b4d0 msgr2=0x55c6023104e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.132+0000 7ff8f1c00640 1 --2- 172.21.15.134:0/2094696772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f4b4d0 0x55c6023104e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.132+0000 7ff8f1c00640 1 -- 172.21.15.134:0/2094696772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f432f0 msgr2=0x55c60230ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-07-26T11:27:05.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.133+0000 7ff8f1c00640 1 --2- 172.21.15.134:0/2094696772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f432f0 0x55c60230ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.134+0000 7ff8f1c00640 1 -- 172.21.15.134:0/2094696772 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c601fd7790 con 0x55c601f6c1b0 2024-07-26T11:27:05.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.135+0000 7ff8f2600640 1 --2- 172.21.15.134:0/1324673184 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8d40b3d40 0x7ff8d40b6250 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0005fd0 tx=0x7ff8e0005850 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:05.140 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.137+0000 7ff8f1c00640 1 --2- 172.21.15.134:0/2094696772 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c60230fa60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40097b0 tx=0x7ff8e4009440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdd635c6fd733070 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.145+0000 7ff8dea00640 1 -- 172.21.15.134:0/2094696772 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4031cf0 con 0x55c601f6c1b0 2024-07-26T11:27:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.145+0000 7ff8dea00640 1 -- 172.21.15.134:0/2094696772 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8e403f070 con 0x55c601f6c1b0 2024-07-26T11:27:05.147 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.145+0000 7ff8dea00640 1 -- 172.21.15.134:0/2094696772 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e40042a0 con 0x55c601f6c1b0 2024-07-26T11:27:05.150 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.147+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/2094696772 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 msgr2=0x55c60230fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:05.150 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.147+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/2094696772 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c60230fa60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40097b0 tx=0x7ff8e4009440 comp rx=0 tx=0).stop 2024-07-26T11:27:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.150+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/2094696772 shutdown_connections 2024-07-26T11:27:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.150+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/2094696772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f4b4d0 0x55c6023104e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.150+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/2094696772 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f432f0 0x55c60230ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.150+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/2094696772 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c60230fa60 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.150+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/2094696772 >> 172.21.15.134:0/2094696772 conn(0x55c601f60300 msgr2=0x55c60230d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.151+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/2094696772 shutdown_connections 2024-07-26T11:27:05.153 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.151+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/2094696772 wait complete. 2024-07-26T11:27:05.155 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.153+0000 7ff8fb7e0740 1 -- start start 2024-07-26T11:27:05.159 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.157+0000 7ff8fb7e0740 1 Processor -- start 2024-07-26T11:27:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.158+0000 7ff8fb7e0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f432f0 0x55c60231b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.158+0000 7ff8fb7e0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f4b4d0 0x55c60231bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.159+0000 7ff8fb7e0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f6c1b0 0x55c602327500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.159+0000 7ff8fb7e0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c6021adab0 con 0x55c601f432f0 2024-07-26T11:27:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.159+0000 7ff8fb7e0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c602210d50 con 0x55c601f6c1b0 2024-07-26T11:27:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.159+0000 7ff8fb7e0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c60231e180 con 0x55c601f4b4d0 2024-07-26T11:27:05.161 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.159+0000 7ff8fb7e0740 10 client.0 _ll_register_callbacks cb 0x55c601f83a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:27:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.159+0000 7ff8f1200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f4b4d0 0x55c60231bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.159+0000 7ff8f1200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f4b4d0 0x55c60231bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34078/0 (socket says 172.21.15.134:34078) 2024-07-26T11:27:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.159+0000 7ff8f1200640 1 -- 172.21.15.134:0/1662848400 learned_addr learned my addr 172.21.15.134:0/1662848400 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:05.162 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.160+0000 7ff8f2600640 1 --2- 172.21.15.134:0/1662848400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f6c1b0 0x55c602327500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.161+0000 7ff8f2600640 1 -- 172.21.15.134:0/1662848400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f4b4d0 msgr2=0x55c60231bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.161+0000 7ff8f2600640 1 --2- 172.21.15.134:0/1662848400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f4b4d0 0x55c60231bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.161+0000 7ff8f2600640 1 -- 172.21.15.134:0/1662848400 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f432f0 msgr2=0x55c60231b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:05.165 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.161+0000 7ff8f2600640 1 --2- 172.21.15.134:0/1662848400 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f432f0 0x55c60231b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.161+0000 7ff8f2600640 1 -- 172.21.15.134:0/1662848400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c602050620 con 0x55c601f6c1b0 2024-07-26T11:27:05.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.165+0000 7ff8f2600640 1 --2- 172.21.15.134:0/1662848400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f6c1b0 0x55c602327500 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00058c0 tx=0x7ff8e0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b05545b29b5a2bb server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:05.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.165+0000 7ff8d3400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:27:05.171 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.167+0000 7ff8d3400640 1 -- 172.21.15.134:0/1662848400 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0005220 con 0x55c601f6c1b0 2024-07-26T11:27:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.170+0000 7ff8d3400640 1 -- 172.21.15.134:0/1662848400 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9495+0+0 (secure 0 0 0) 0x7ff8e00053c0 con 0x55c601f6c1b0 2024-07-26T11:27:05.172 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.170+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/1662848400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8e403f070 con 0x55c601f6c1b0 2024-07-26T11:27:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.171+0000 7ff8d3400640 1 -- 172.21.15.134:0/1662848400 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff8e00055c0 con 0x55c601f6c1b0 2024-07-26T11:27:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.171+0000 7ff8fb7e0740 10 client.26972 fetch_fsmap learned FSMap version 36 2024-07-26T11:27:05.173 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.172+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/1662848400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55c60232a310 con 0x55c601f6c1b0 2024-07-26T11:27:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.172+0000 7ff8d3400640 1 -- 172.21.15.134:0/1662848400 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8e403f070 con 0x55c601f6c1b0 2024-07-26T11:27:05.174 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.173+0000 7ff8d3400640 1 -- 172.21.15.134:0/1662848400 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8398+0+0 (secure 0 0 0) 0x7ff8e001c040 con 0x55c601f6c1b0 2024-07-26T11:27:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.173+0000 7ff8fb7e0740 10 client.26972 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T11:27:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.173+0000 7ff8fb7e0740 10 client.26972 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:27:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.173+0000 7ff8fb7e0740 20 client.26972 populate_metadata read hostname 'smithi134' 2024-07-26T11:27:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.173+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/1662848400 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c601f2ad80 0x55c60232da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.173+0000 7ff8fb7e0740 4 client.26972 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:27:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.173+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/1662848400 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6022861c0 con 0x55c601f2ad80 2024-07-26T11:27:05.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.175+0000 7ff8f1c00640 1 --2- 172.21.15.134:0/1662848400 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c601f2ad80 0x55c60232da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:05.180 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.178+0000 7ff8f1c00640 1 --2- 172.21.15.134:0/1662848400 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c601f2ad80 0x55c60232da40 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6439f6d1220ffa77 server_cookie=ff7cc9a13543c563 in_seq=0 out_seq=0 2024-07-26T11:27:05.180 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.178+0000 7ff8d3400640 10 client.26972 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.179+0000 7ff8d3400640 1 -- 172.21.15.134:0/1662848400 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x55c6022861c0 con 0x55c601f2ad80 2024-07-26T11:27:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.179+0000 7ff8d3400640 10 client.26972 handle_command_reply: tid=0 2024-07-26T11:27:05.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.180+0000 7ff8fb7e0740 1 client.26972 shutdown 2024-07-26T11:27:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.181+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/1662848400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f6c1b0 msgr2=0x55c602327500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.182+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/1662848400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f6c1b0 0x55c602327500 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff8e00058c0 tx=0x7ff8e0004730 comp rx=0 tx=0).stop 2024-07-26T11:27:05.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.182+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/1662848400 shutdown_connections 2024-07-26T11:27:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.182+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/1662848400 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c601f2ad80 0x55c60232da40 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.182+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/1662848400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f4b4d0 0x55c60231bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.182+0000 7ff8d3400640 0 client.26972 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.182+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/1662848400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f6c1b0 0x55c602327500 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.182+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/1662848400 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f432f0 0x55c60231b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.182+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/1662848400 >> 172.21.15.134:0/1662848400 conn(0x55c601f60300 msgr2=0x55c602310a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.183+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/1662848400 shutdown_connections 2024-07-26T11:27:05.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.183+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/1662848400 wait complete. 2024-07-26T11:27:05.187 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.185+0000 7ff8fb7e0740 20 client.26972 trim_cache size 0 max 16384 2024-07-26T11:27:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:04 smithi189 ceph-mon[36790]: pgmap v824: 129 pgs: 129 active+clean; 59 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 179 MiB/s wr, 1.20k op/s 2024-07-26T11:27:05.226 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.224+0000 7ff8fb7e0740 1 Processor -- start 2024-07-26T11:27:05.230 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.229+0000 7ff8fb7e0740 1 -- start start 2024-07-26T11:27:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.229+0000 7ff8fb7e0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f432f0 0x55c60231b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.232 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.229+0000 7ff8fb7e0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f4b4d0 0x55c6023272f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.232 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.229+0000 7ff8fb7e0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c60230d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.232 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.229+0000 7ff8fb7e0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc010000 con 0x55c601f4b4d0 2024-07-26T11:27:05.232 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.229+0000 7ff8fb7e0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc011c00 con 0x55c601f432f0 2024-07-26T11:27:05.232 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.229+0000 7ff8fb7e0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc0106d0 con 0x55c601f6c1b0 2024-07-26T11:27:05.233 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.231+0000 7ff8f2600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c60230d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:05.233 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.231+0000 7ff8f2600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c60230d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34086/0 (socket says 172.21.15.134:34086) 2024-07-26T11:27:05.233 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.231+0000 7ff8f2600640 1 -- 172.21.15.134:0/1046774313 learned_addr learned my addr 172.21.15.134:0/1046774313 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:05.234 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.232+0000 7ff8f1200640 1 --2- 172.21.15.134:0/1046774313 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f4b4d0 0x55c6023272f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:05.234 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.232+0000 7ff8f2600640 1 -- 172.21.15.134:0/1046774313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f432f0 msgr2=0x55c60231b0c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-07-26T11:27:05.234 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.233+0000 7ff8f1c00640 1 --2- 172.21.15.134:0/1046774313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f432f0 0x55c60231b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:05.234 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.233+0000 7ff8f2600640 1 --2- 172.21.15.134:0/1046774313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f432f0 0x55c60231b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.236 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.233+0000 7ff8f2600640 1 -- 172.21.15.134:0/1046774313 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f4b4d0 msgr2=0x55c6023272f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:05.236 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.233+0000 7ff8f2600640 1 --2- 172.21.15.134:0/1046774313 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f4b4d0 0x55c6023272f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.236 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.233+0000 7ff8f2600640 1 -- 172.21.15.134:0/1046774313 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c602311980 con 0x55c601f6c1b0 2024-07-26T11:27:05.236 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.233+0000 7ff8f1200640 1 --2- 172.21.15.134:0/1046774313 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f4b4d0 0x55c6023272f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:27:05.236 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.234+0000 7ff8f1c00640 1 --2- 172.21.15.134:0/1046774313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f432f0 0x55c60231b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:27:05.236 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.234+0000 7ff8f2600640 1 --2- 172.21.15.134:0/1046774313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c60230d340 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0011400 tx=0x7ff8e0004640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ac17ed7bfd89621 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:05.237 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.235+0000 7ff8de000640 1 -- 172.21.15.134:0/1046774313 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e001fe60 con 0x55c601f6c1b0 2024-07-26T11:27:05.237 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.235+0000 7ff8de000640 1 -- 172.21.15.134:0/1046774313 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8e0012680 con 0x55c601f6c1b0 2024-07-26T11:27:05.237 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.235+0000 7ff8de000640 1 -- 172.21.15.134:0/1046774313 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0033680 con 0x55c601f6c1b0 2024-07-26T11:27:05.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.237+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/1046774313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 msgr2=0x55c60230d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:05.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.237+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/1046774313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c60230d340 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0011400 tx=0x7ff8e0004640 comp rx=0 tx=0).stop 2024-07-26T11:27:05.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.237+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/1046774313 shutdown_connections 2024-07-26T11:27:05.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.237+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/1046774313 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c60230d340 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.237+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/1046774313 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f4b4d0 0x55c6023272f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.237+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/1046774313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f432f0 0x55c60231b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.237+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/1046774313 >> 172.21.15.134:0/1046774313 conn(0x55c601f60300 msgr2=0x55c60231a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.238+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/1046774313 shutdown_connections 2024-07-26T11:27:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.238+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/1046774313 wait complete. 2024-07-26T11:27:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.239+0000 7ff8fb7e0740 1 -- start start 2024-07-26T11:27:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.241+0000 7ff8fb7e0740 1 Processor -- start 2024-07-26T11:27:05.244 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.242+0000 7ff8fb7e0740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f432f0 0x55c60231e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.243+0000 7ff8fb7e0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f4b4d0 0x55c60231eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.243+0000 7ff8fb7e0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c6023941c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.243+0000 7ff8fb7e0740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc011470 con 0x55c601f432f0 2024-07-26T11:27:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.243+0000 7ff8fb7e0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc010da0 con 0x55c601f4b4d0 2024-07-26T11:27:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.243+0000 7ff8fb7e0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc00e000 con 0x55c601f6c1b0 2024-07-26T11:27:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.243+0000 7ff8fb7e0740 10 client.0 _ll_register_callbacks cb 0x7ff8e000ed60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:27:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.245+0000 7ff8f2600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c6023941c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.245+0000 7ff8f2600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c6023941c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34096/0 (socket says 172.21.15.134:34096) 2024-07-26T11:27:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.245+0000 7ff8f2600640 1 -- 172.21.15.134:0/342830193 learned_addr learned my addr 172.21.15.134:0/342830193 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.245+0000 7ff8f1c00640 1 --2- 172.21.15.134:0/342830193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f432f0 0x55c60231e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.245+0000 7ff8f1200640 1 --2- 172.21.15.134:0/342830193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f4b4d0 0x55c60231eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.246+0000 7ff8f1200640 1 -- 172.21.15.134:0/342830193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 msgr2=0x55c6023941c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.246+0000 7ff8f1200640 1 --2- 172.21.15.134:0/342830193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c6023941c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.246+0000 7ff8f1200640 1 -- 172.21.15.134:0/342830193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f432f0 msgr2=0x55c60231e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:05.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.246+0000 7ff8f1200640 1 --2- 172.21.15.134:0/342830193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f432f0 0x55c60231e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.246+0000 7ff8f1200640 1 -- 172.21.15.134:0/342830193 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c60231f2a0 con 0x55c601f4b4d0 2024-07-26T11:27:05.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.247+0000 7ff8d3400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:27:05.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.247+0000 7ff8f1200640 1 --2- 172.21.15.134:0/342830193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f4b4d0 0x55c60231eab0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80e1fe0 tx=0x7ff8d80aa8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=254f247c488ca969 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:05.250 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.248+0000 7ff8d3400640 1 -- 172.21.15.134:0/342830193 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d80c4070 con 0x55c601f4b4d0 2024-07-26T11:27:05.250 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.248+0000 7ff8d3400640 1 -- 172.21.15.134:0/342830193 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9495+0+0 (secure 0 0 0) 0x7ff8d80a7490 con 0x55c601f4b4d0 2024-07-26T11:27:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.248+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/342830193 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8e0012680 con 0x55c601f4b4d0 2024-07-26T11:27:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.249+0000 7ff8d3400640 1 -- 172.21.15.134:0/342830193 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff8d80aa9b0 con 0x55c601f4b4d0 2024-07-26T11:27:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.249+0000 7ff8fb7e0740 10 client.26975 fetch_fsmap learned FSMap version 36 2024-07-26T11:27:05.251 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.249+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/342830193 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55c602395600 con 0x55c601f4b4d0 2024-07-26T11:27:05.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.249+0000 7ff8d3400640 1 -- 172.21.15.134:0/342830193 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8e0012680 con 0x55c601f4b4d0 2024-07-26T11:27:05.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.250+0000 7ff8d3400640 1 -- 172.21.15.134:0/342830193 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8398+0+0 (secure 0 0 0) 0x7ff8d806d020 con 0x55c601f4b4d0 2024-07-26T11:27:05.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.250+0000 7ff8fb7e0740 10 client.26975 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T11:27:05.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.250+0000 7ff8fb7e0740 10 client.26975 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:27:05.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.250+0000 7ff8fb7e0740 20 client.26975 populate_metadata read hostname 'smithi134' 2024-07-26T11:27:05.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.250+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/342830193 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c601f001c0 0x55c602397e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:05.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.250+0000 7ff8fb7e0740 4 client.26975 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:27:05.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.250+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/342830193 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c602281c60 con 0x55c601f001c0 2024-07-26T11:27:05.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.250+0000 7ff8f1c00640 1 --2- 172.21.15.134:0/342830193 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c601f001c0 0x55c602397e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:05.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.251+0000 7ff8d3400640 10 client.26975 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:05.253 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.252+0000 7ff8f1c00640 1 --2- 172.21.15.134:0/342830193 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c601f001c0 0x55c602397e30 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=176c1eb598f81aa4 server_cookie=efb0572f859a4a3a in_seq=0 out_seq=0 2024-07-26T11:27:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:04 smithi193 ceph-mon[37473]: pgmap v824: 129 pgs: 129 active+clean; 59 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 179 MiB/s wr, 1.20k op/s 2024-07-26T11:27:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:04 smithi134 ceph-mon[31966]: pgmap v824: 129 pgs: 129 active+clean; 59 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 1.8 MiB/s rd, 179 MiB/s wr, 1.20k op/s 2024-07-26T11:27:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.609+0000 7ff8d3400640 1 -- 172.21.15.134:0/342830193 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c602281c60 con 0x55c601f001c0 2024-07-26T11:27:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.609+0000 7ff8d3400640 10 client.26975 handle_command_reply: tid=0 2024-07-26T11:27:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.609+0000 7ff8fb7e0740 1 client.26975 shutdown 2024-07-26T11:27:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.611+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/342830193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f4b4d0 msgr2=0x55c60231eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:05.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.611+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/342830193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f4b4d0 0x55c60231eab0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80e1fe0 tx=0x7ff8d80aa8b0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.611+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/342830193 shutdown_connections 2024-07-26T11:27:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.611+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/342830193 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c601f001c0 0x55c602397e30 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.611+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/342830193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c601f6c1b0 0x55c6023941c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.611+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/342830193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c601f4b4d0 0x55c60231eab0 secure :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80e1fe0 tx=0x7ff8d80aa8b0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.611+0000 7ff8fb7e0740 1 --2- 172.21.15.134:0/342830193 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c601f432f0 0x55c60231e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.611+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/342830193 >> 172.21.15.134:0/342830193 conn(0x55c601f60300 msgr2=0x55c602314b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:05.613 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.611+0000 7ff8d3400640 0 client.26975 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:05.618 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.616+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/342830193 shutdown_connections 2024-07-26T11:27:05.619 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.616+0000 7ff8fb7e0740 1 -- 172.21.15.134:0/342830193 wait complete. 2024-07-26T11:27:05.619 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.617+0000 7ff8fb7e0740 20 client.26975 trim_cache size 0 max 16384 2024-07-26T11:27:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:27:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:27:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:27:05.620 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:27:05.622 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.620+0000 7ff8f4600640 1 -- 172.21.15.134:0/1324673184 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8d40b3d40 msgr2=0x7ff8d40b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:05.622 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.620+0000 7ff8f4600640 1 --2- 172.21.15.134:0/1324673184 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8d40b3d40 0x7ff8d40b6250 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0005fd0 tx=0x7ff8e0005850 comp rx=0 tx=0).stop 2024-07-26T11:27:05.622 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.620+0000 7ff8f4600640 1 -- 172.21.15.134:0/1324673184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8ec10f430 msgr2=0x7ff8ec1a9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.620+0000 7ff8f4600640 1 --2- 172.21.15.134:0/1324673184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8ec10f430 0x7ff8ec1a9310 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800e9c0 tx=0x7ff8d800ee80 comp rx=0 tx=0).stop 2024-07-26T11:27:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.620+0000 7ff8f4600640 1 -- 172.21.15.134:0/1324673184 shutdown_connections 2024-07-26T11:27:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.620+0000 7ff8f4600640 1 --2- 172.21.15.134:0/1324673184 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7ff8d40b3d40 0x7ff8d40b6250 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.620+0000 7ff8f4600640 1 --2- 172.21.15.134:0/1324673184 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8ec10f430 0x7ff8ec1a9310 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.620+0000 7ff8f4600640 1 --2- 172.21.15.134:0/1324673184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8ec10fd80 0x7ff8ec1ae5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.620+0000 7ff8f4600640 1 --2- 172.21.15.134:0/1324673184 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7ff8ec10a090 0x7ff8ec1a8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.620+0000 7ff8f4600640 1 -- 172.21.15.134:0/1324673184 >> 172.21.15.134:0/1324673184 conn(0x7ff8ec070800 msgr2=0x7ff8ec073760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.620+0000 7ff8f4600640 1 -- 172.21.15.134:0/1324673184 shutdown_connections 2024-07-26T11:27:05.623 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:05.621+0000 7ff8f4600640 1 -- 172.21.15.134:0/1324673184 wait complete. 2024-07-26T11:27:05.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 damage ls 2024-07-26T11:27:05.926 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:27:05 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:27:05.659+0000 7f8ff6c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:27:05.926 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:27:05 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:27:05.659+0000 7f8ff6c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:27:05.926 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:27:05 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:27:05.659+0000 7f8ff6c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:27:05.926 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:27:05 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:27:05.659+0000 7f8ff6c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:27:05.926 INFO:journalctl@ceph.mgr.x.smithi134.stdout:Jul 26 11:27:05 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mgr-x[32199]: 2024-07-26T11:27:05.659+0000 7f8ff6c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-07-26T11:27:06.595 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:27:07.139 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:06 smithi134 ceph-mon[31966]: mgrmap e26: x(active, since 24m), standbys: z, y 2024-07-26T11:27:07.139 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:06 smithi134 ceph-mon[31966]: pgmap v825: 129 pgs: 129 active+clean; 59 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 191 MiB/s wr, 1.35k op/s 2024-07-26T11:27:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:06 smithi193 ceph-mon[37473]: mgrmap e26: x(active, since 24m), standbys: z, y 2024-07-26T11:27:07.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:06 smithi193 ceph-mon[37473]: pgmap v825: 129 pgs: 129 active+clean; 59 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 191 MiB/s wr, 1.35k op/s 2024-07-26T11:27:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:06 smithi189 ceph-mon[36790]: mgrmap e26: x(active, since 24m), standbys: z, y 2024-07-26T11:27:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:06 smithi189 ceph-mon[36790]: pgmap v825: 129 pgs: 129 active+clean; 59 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 1.9 MiB/s rd, 191 MiB/s wr, 1.35k op/s 2024-07-26T11:27:09.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.173+0000 7f9b7d400640 1 -- 172.21.15.134:0/3289159365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b7810ba40 msgr2=0x7f9b78112270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:09.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.173+0000 7f9b7d400640 1 --2- 172.21.15.134:0/3289159365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b7810ba40 0x7f9b78112270 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9b70013fd0 tx=0x7f9b70032db0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.174+0000 7f9b7d400640 1 -- 172.21.15.134:0/3289159365 shutdown_connections 2024-07-26T11:27:09.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.174+0000 7f9b7d400640 1 --2- 172.21.15.134:0/3289159365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b7810b0f0 0x7f9b7810b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.174+0000 7f9b7d400640 1 --2- 172.21.15.134:0/3289159365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b7810ba40 0x7f9b78112270 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.174+0000 7f9b7d400640 1 --2- 172.21.15.134:0/3289159365 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b781057b0 0x7f9b78105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.174+0000 7f9b7d400640 1 -- 172.21.15.134:0/3289159365 >> 172.21.15.134:0/3289159365 conn(0x7f9b78101240 msgr2=0x7f9b78103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:09.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.178+0000 7f9b7d400640 1 -- 172.21.15.134:0/3289159365 shutdown_connections 2024-07-26T11:27:09.181 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.178+0000 7f9b7d400640 1 -- 172.21.15.134:0/3289159365 wait complete. 2024-07-26T11:27:09.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.181+0000 7f9b7d400640 1 Processor -- start 2024-07-26T11:27:09.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.182+0000 7f9b7d400640 1 -- start start 2024-07-26T11:27:09.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.183+0000 7f9b7d400640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b781057b0 0x7f9b781a8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.183+0000 7f9b7d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b7810b0f0 0x7f9b781a9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.183+0000 7f9b7d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b7810ba40 0x7f9b781ae590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.183+0000 7f9b7d400640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9b78079ac0 con 0x7f9b781057b0 2024-07-26T11:27:09.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.183+0000 7f9b7d400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9b78079940 con 0x7f9b7810b0f0 2024-07-26T11:27:09.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.183+0000 7f9b7d400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9b78079c40 con 0x7f9b7810ba40 2024-07-26T11:27:09.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.183+0000 7f9b75e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b7810b0f0 0x7f9b781a9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.184+0000 7f9b75e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b7810b0f0 0x7f9b781a9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:44102/0 (socket says 172.21.15.134:44102) 2024-07-26T11:27:09.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.184+0000 7f9b75e00640 1 -- 172.21.15.134:0/875219983 learned_addr learned my addr 172.21.15.134:0/875219983 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:09.186 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.184+0000 7f9b77200640 1 --2- 172.21.15.134:0/875219983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b7810ba40 0x7f9b781ae590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.184+0000 7f9b76800640 1 --2- 172.21.15.134:0/875219983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b781057b0 0x7f9b781a8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.185+0000 7f9b77200640 1 -- 172.21.15.134:0/875219983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b7810b0f0 msgr2=0x7f9b781a9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:09.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.185+0000 7f9b77200640 1 --2- 172.21.15.134:0/875219983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b7810b0f0 0x7f9b781a9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.185+0000 7f9b77200640 1 -- 172.21.15.134:0/875219983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b781057b0 msgr2=0x7f9b781a8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:09.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.185+0000 7f9b77200640 1 --2- 172.21.15.134:0/875219983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b781057b0 0x7f9b781a8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.185+0000 7f9b77200640 1 -- 172.21.15.134:0/875219983 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b781aecc0 con 0x7f9b7810ba40 2024-07-26T11:27:09.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.185+0000 7f9b77200640 1 --2- 172.21.15.134:0/875219983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b7810ba40 0x7f9b781ae590 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9b70013f20 tx=0x7f9b70003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:09.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.186+0000 7f9b6f400640 1 -- 172.21.15.134:0/875219983 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b70040070 con 0x7f9b7810ba40 2024-07-26T11:27:09.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.186+0000 7f9b7d400640 1 -- 172.21.15.134:0/875219983 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b781aef50 con 0x7f9b7810ba40 2024-07-26T11:27:09.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.186+0000 7f9b7d400640 1 -- 172.21.15.134:0/875219983 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b781b5710 con 0x7f9b7810ba40 2024-07-26T11:27:09.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.187+0000 7f9b6f400640 1 -- 172.21.15.134:0/875219983 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b70034da0 con 0x7f9b7810ba40 2024-07-26T11:27:09.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.187+0000 7f9b6f400640 1 -- 172.21.15.134:0/875219983 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b70013790 con 0x7f9b7810ba40 2024-07-26T11:27:09.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.191+0000 7f9b6f400640 1 -- 172.21.15.134:0/875219983 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 27) ==== 151191+0+0 (secure 0 0 0) 0x7f9b70043070 con 0x7f9b7810ba40 2024-07-26T11:27:09.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.195+0000 7f9b83fd5740 1 Processor -- start 2024-07-26T11:27:09.197 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.195+0000 7f9b83fd5740 1 -- start start 2024-07-26T11:27:09.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.196+0000 7f9b83fd5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdd020070 0x556cdd3c4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.196+0000 7f9b83fd5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcff71b0 0x556cdd3c5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.196+0000 7f9b83fd5740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdcfff390 0x556cdd3c58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.196+0000 7f9b83fd5740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556cdd33bbd0 con 0x556cdcfff390 2024-07-26T11:27:09.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.196+0000 7f9b83fd5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556cdd32b0d0 con 0x556cdd020070 2024-07-26T11:27:09.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.196+0000 7f9b83fd5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556cdd32f110 con 0x556cdcff71b0 2024-07-26T11:27:09.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.197+0000 7f9b75e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdd020070 0x556cdd3c4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.197+0000 7f9b75e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdd020070 0x556cdd3c4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:44106/0 (socket says 172.21.15.134:44106) 2024-07-26T11:27:09.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.197+0000 7f9b75e00640 1 -- 172.21.15.134:0/878315858 learned_addr learned my addr 172.21.15.134:0/878315858 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:09.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.197+0000 7f9b76800640 1 --2- 172.21.15.134:0/878315858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcff71b0 0x556cdd3c5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.197+0000 7f9b75e00640 1 --2- 172.21.15.134:0/878315858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdcfff390 0x556cdd3c58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.198+0000 7f9b75e00640 1 -- 172.21.15.134:0/878315858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcff71b0 msgr2=0x556cdd3c5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:09.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.198+0000 7f9b75e00640 1 --2- 172.21.15.134:0/878315858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcff71b0 0x556cdd3c5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.198+0000 7f9b75e00640 1 -- 172.21.15.134:0/878315858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdcfff390 msgr2=0x556cdd3c58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:09.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.198+0000 7f9b75e00640 1 --2- 172.21.15.134:0/878315858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdcfff390 0x556cdd3c58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.198+0000 7f9b75e00640 1 -- 172.21.15.134:0/878315858 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cdd08b650 con 0x556cdd020070 2024-07-26T11:27:09.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.199+0000 7f9b75e00640 1 --2- 172.21.15.134:0/878315858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdd020070 0x556cdd3c4e30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800bef0 tx=0x7f9b6800b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be8eae921897038 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:09.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.199+0000 7f9b6f400640 1 --2- 172.21.15.134:0/875219983 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b5c0b3d40 0x7f9b5c0b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.200+0000 7f9b6f400640 1 -- 172.21.15.134:0/875219983 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 9495+0+0 (secure 0 0 0) 0x7f9b70104670 con 0x7f9b7810ba40 2024-07-26T11:27:09.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.201+0000 7f9b76800640 1 --2- 172.21.15.134:0/875219983 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b5c0b3d40 0x7f9b5c0b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.202+0000 7f9b76800640 1 --2- 172.21.15.134:0/875219983 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b5c0b3d40 0x7f9b5c0b6250 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x556cdd3c3ab0 tx=0x7f9b64008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:09.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.203+0000 7f9b63400640 1 -- 172.21.15.134:0/878315858 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b68010730 con 0x556cdd020070 2024-07-26T11:27:09.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.204+0000 7f9b63400640 1 -- 172.21.15.134:0/878315858 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b68010d80 con 0x556cdd020070 2024-07-26T11:27:09.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.204+0000 7f9b63400640 1 -- 172.21.15.134:0/878315858 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6801dcb0 con 0x556cdd020070 2024-07-26T11:27:09.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.205+0000 7f9b83fd5740 1 -- 172.21.15.134:0/878315858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdd020070 msgr2=0x556cdd3c4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:09.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.205+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/878315858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdd020070 0x556cdd3c4e30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800bef0 tx=0x7f9b6800b800 comp rx=0 tx=0).stop 2024-07-26T11:27:09.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.206+0000 7f9b83fd5740 1 -- 172.21.15.134:0/878315858 shutdown_connections 2024-07-26T11:27:09.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.206+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/878315858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcff71b0 0x556cdd3c5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.210 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.206+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/878315858 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdcfff390 0x556cdd3c58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.206+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/878315858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdd020070 0x556cdd3c4e30 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.206+0000 7f9b83fd5740 1 -- 172.21.15.134:0/878315858 >> 172.21.15.134:0/878315858 conn(0x556cdd014460 msgr2=0x556cdd3c2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:09.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.206+0000 7f9b83fd5740 1 -- 172.21.15.134:0/878315858 shutdown_connections 2024-07-26T11:27:09.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.206+0000 7f9b83fd5740 1 -- 172.21.15.134:0/878315858 wait complete. 2024-07-26T11:27:09.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.207+0000 7f9b83fd5740 1 -- start start 2024-07-26T11:27:09.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.207+0000 7f9b83fd5740 1 Processor -- start 2024-07-26T11:27:09.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.209+0000 7f9b83fd5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcff71b0 0x556cdd3d0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.209+0000 7f9b83fd5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcfff390 0x556cdd3d0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.209+0000 7f9b83fd5740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdd020070 0x556cdd3d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.210+0000 7f9b75e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcfff390 0x556cdd3d0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.210+0000 7f9b77200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdd020070 0x556cdd3d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.210+0000 7f9b77200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdd020070 0x556cdd3d0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:41470/0 (socket says 172.21.15.134:41470) 2024-07-26T11:27:09.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.210+0000 7f9b77200640 1 -- 172.21.15.134:0/2351645871 learned_addr learned my addr 172.21.15.134:0/2351645871 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:09.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.210+0000 7f9b83fd5740 1 -- 172.21.15.134:0/2351645871 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x556cdd18e700 con 0x556cdd020070 2024-07-26T11:27:09.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.210+0000 7f9b83fd5740 1 -- 172.21.15.134:0/2351645871 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x556cdd2c76a0 con 0x556cdcff71b0 2024-07-26T11:27:09.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.210+0000 7f9b83fd5740 1 -- 172.21.15.134:0/2351645871 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x556cdd3d3610 con 0x556cdcfff390 2024-07-26T11:27:09.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.211+0000 7f9b83fd5740 10 client.0 _ll_register_callbacks cb 0x556cdd2d7150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:27:09.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.209+0000 7f9b76800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcff71b0 0x556cdd3d0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.211+0000 7f9b75e00640 1 -- 172.21.15.134:0/2351645871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcff71b0 msgr2=0x556cdd3d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:09.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.211+0000 7f9b75e00640 1 --2- 172.21.15.134:0/2351645871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcff71b0 0x556cdd3d0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.211+0000 7f9b75e00640 1 -- 172.21.15.134:0/2351645871 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdd020070 msgr2=0x556cdd3d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:09.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.211+0000 7f9b75e00640 1 --2- 172.21.15.134:0/2351645871 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdd020070 0x556cdd3d0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.213+0000 7f9b75e00640 1 -- 172.21.15.134:0/2351645871 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cdd2de840 con 0x556cdcfff390 2024-07-26T11:27:09.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.213+0000 7f9b75e00640 1 --2- 172.21.15.134:0/2351645871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcfff390 0x556cdd3d0990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800b6f0 tx=0x7f9b6801d610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d5096a136619580 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:09.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.214+0000 7f9b57e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:27:09.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.214+0000 7f9b57e00640 1 -- 172.21.15.134:0/2351645871 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6800b390 con 0x556cdcfff390 2024-07-26T11:27:09.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.214+0000 7f9b57e00640 1 -- 172.21.15.134:0/2351645871 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9495+0+0 (secure 0 0 0) 0x7f9b6801dc40 con 0x556cdcfff390 2024-07-26T11:27:09.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.214+0000 7f9b83fd5740 1 -- 172.21.15.134:0/2351645871 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b68010d80 con 0x556cdcfff390 2024-07-26T11:27:09.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.214+0000 7f9b57e00640 1 -- 172.21.15.134:0/2351645871 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f9b6801de40 con 0x556cdcfff390 2024-07-26T11:27:09.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.215+0000 7f9b83fd5740 10 client.27007 fetch_fsmap learned FSMap version 36 2024-07-26T11:27:09.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.215+0000 7f9b83fd5740 1 -- 172.21.15.134:0/2351645871 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556cdd3df290 con 0x556cdcfff390 2024-07-26T11:27:09.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.215+0000 7f9b57e00640 1 -- 172.21.15.134:0/2351645871 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b68010d80 con 0x556cdcfff390 2024-07-26T11:27:09.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.217+0000 7f9b57e00640 1 -- 172.21.15.134:0/2351645871 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 36) ==== 8398+0+0 (secure 0 0 0) 0x7f9b68030050 con 0x556cdcfff390 2024-07-26T11:27:09.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.219+0000 7f9b83fd5740 10 client.27007 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T11:27:09.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.219+0000 7f9b83fd5740 10 client.27007 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:27:09.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.219+0000 7f9b83fd5740 20 client.27007 populate_metadata read hostname 'smithi134' 2024-07-26T11:27:09.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.219+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/2351645871 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556cdcfde9f0 0x556cdd3e25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.219+0000 7f9b83fd5740 4 client.27007 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:27:09.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.220+0000 7f9b83fd5740 1 -- 172.21.15.134:0/2351645871 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556cdd33db20 con 0x556cdcfde9f0 2024-07-26T11:27:09.222 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.220+0000 7f9b76800640 1 --2- 172.21.15.134:0/2351645871 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556cdcfde9f0 0x556cdd3e25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.223 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:08 smithi134 ceph-mon[31966]: mgrmap e27: x(active, since 24m), standbys: z, y 2024-07-26T11:27:09.223 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:08 smithi134 ceph-mon[31966]: pgmap v826: 129 pgs: 129 active+clean; 59 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 176 MiB/s wr, 1.26k op/s 2024-07-26T11:27:09.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.221+0000 7f9b57e00640 10 client.27007 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:09.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.221+0000 7f9b76800640 1 --2- 172.21.15.134:0/2351645871 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556cdcfde9f0 0x556cdd3e25f0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64948baa966a4e33 server_cookie=ee8c8a5adf705622 in_seq=0 out_seq=0 2024-07-26T11:27:09.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.222+0000 7f9b57e00640 1 -- 172.21.15.134:0/2351645871 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x556cdd33db20 con 0x556cdcfde9f0 2024-07-26T11:27:09.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.222+0000 7f9b57e00640 10 client.27007 handle_command_reply: tid=0 2024-07-26T11:27:09.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.222+0000 7f9b83fd5740 1 client.27007 shutdown 2024-07-26T11:27:09.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.223+0000 7f9b83fd5740 1 -- 172.21.15.134:0/2351645871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcfff390 msgr2=0x556cdd3d0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:09.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.223+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/2351645871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcfff390 0x556cdd3d0990 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800b6f0 tx=0x7f9b6801d610 comp rx=0 tx=0).stop 2024-07-26T11:27:09.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.223+0000 7f9b83fd5740 1 -- 172.21.15.134:0/2351645871 shutdown_connections 2024-07-26T11:27:09.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.223+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/2351645871 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556cdcfde9f0 0x556cdd3e25f0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.223+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/2351645871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcfff390 0x556cdd3d0990 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.223+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/2351645871 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdd020070 0x556cdd3d0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.223+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/2351645871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcff71b0 0x556cdd3d0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.223+0000 7f9b83fd5740 1 -- 172.21.15.134:0/2351645871 >> 172.21.15.134:0/2351645871 conn(0x556cdd014460 msgr2=0x556cdd3c7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:09.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.223+0000 7f9b57e00640 0 client.27007 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:09.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.223+0000 7f9b83fd5740 1 -- 172.21.15.134:0/2351645871 shutdown_connections 2024-07-26T11:27:09.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.223+0000 7f9b83fd5740 1 -- 172.21.15.134:0/2351645871 wait complete. 2024-07-26T11:27:09.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.223+0000 7f9b83fd5740 20 client.27007 trim_cache size 0 max 16384 2024-07-26T11:27:09.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.250+0000 7f9b83fd5740 1 Processor -- start 2024-07-26T11:27:09.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.250+0000 7f9b83fd5740 1 -- start start 2024-07-26T11:27:09.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.251+0000 7f9b83fd5740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdcff71b0 0x556cdd3cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.251+0000 7f9b83fd5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcfff390 0x556cdd3d2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.251+0000 7f9b83fd5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdd020070 0x556cdd3c6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.251+0000 7f9b83fd5740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9b58010130 con 0x556cdcff71b0 2024-07-26T11:27:09.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.251+0000 7f9b83fd5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9b58011d30 con 0x556cdcfff390 2024-07-26T11:27:09.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.251+0000 7f9b83fd5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9b58010800 con 0x556cdd020070 2024-07-26T11:27:09.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.251+0000 7f9b77200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdd020070 0x556cdd3c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.251+0000 7f9b77200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdd020070 0x556cdd3c6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34152/0 (socket says 172.21.15.134:34152) 2024-07-26T11:27:09.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.251+0000 7f9b77200640 1 -- 172.21.15.134:0/4011072215 learned_addr learned my addr 172.21.15.134:0/4011072215 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:09.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.252+0000 7f9b75e00640 1 --2- 172.21.15.134:0/4011072215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcfff390 0x556cdd3d2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.252+0000 7f9b76800640 1 --2- 172.21.15.134:0/4011072215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdcff71b0 0x556cdd3cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.252+0000 7f9b77200640 1 -- 172.21.15.134:0/4011072215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcfff390 msgr2=0x556cdd3d2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:09.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.252+0000 7f9b77200640 1 --2- 172.21.15.134:0/4011072215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcfff390 0x556cdd3d2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.252+0000 7f9b77200640 1 -- 172.21.15.134:0/4011072215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdcff71b0 msgr2=0x556cdd3cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:09.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.252+0000 7f9b77200640 1 --2- 172.21.15.134:0/4011072215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdcff71b0 0x556cdd3cbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.252+0000 7f9b77200640 1 -- 172.21.15.134:0/4011072215 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556cdd3c7120 con 0x556cdd020070 2024-07-26T11:27:09.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.253+0000 7f9b75e00640 1 --2- 172.21.15.134:0/4011072215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcfff390 0x556cdd3d2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:27:09.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.253+0000 7f9b76800640 1 --2- 172.21.15.134:0/4011072215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdcff71b0 0x556cdd3cbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:27:09.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.253+0000 7f9b77200640 1 --2- 172.21.15.134:0/4011072215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdd020070 0x556cdd3c6570 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9b70005e30 tx=0x7f9b700095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1af4713be09acca5 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:09.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.254+0000 7f9b62a00640 1 -- 172.21.15.134:0/4011072215 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7004a040 con 0x556cdd020070 2024-07-26T11:27:09.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.255+0000 7f9b62a00640 1 -- 172.21.15.134:0/4011072215 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b700cb6f0 con 0x556cdd020070 2024-07-26T11:27:09.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.255+0000 7f9b62a00640 1 -- 172.21.15.134:0/4011072215 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b7004a040 con 0x556cdd020070 2024-07-26T11:27:09.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.257+0000 7f9b83fd5740 1 -- 172.21.15.134:0/4011072215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdd020070 msgr2=0x556cdd3c6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:09.259 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.257+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/4011072215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdd020070 0x556cdd3c6570 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9b70005e30 tx=0x7f9b700095c0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.258+0000 7f9b83fd5740 1 -- 172.21.15.134:0/4011072215 shutdown_connections 2024-07-26T11:27:09.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.258+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/4011072215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdd020070 0x556cdd3c6570 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.258+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/4011072215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcfff390 0x556cdd3d2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.258+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/4011072215 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdcff71b0 0x556cdd3cbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.258+0000 7f9b83fd5740 1 -- 172.21.15.134:0/4011072215 >> 172.21.15.134:0/4011072215 conn(0x556cdd014460 msgr2=0x556cdd3c5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:09.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.258+0000 7f9b83fd5740 1 -- 172.21.15.134:0/4011072215 shutdown_connections 2024-07-26T11:27:09.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.259+0000 7f9b83fd5740 1 -- 172.21.15.134:0/4011072215 wait complete. 2024-07-26T11:27:09.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.259+0000 7f9b83fd5740 1 -- start start 2024-07-26T11:27:09.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.260+0000 7f9b83fd5740 1 Processor -- start 2024-07-26T11:27:09.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.261+0000 7f9b83fd5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcff71b0 0x556cdd3d8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.261+0000 7f9b83fd5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcfff390 0x556cdd3cac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.261+0000 7f9b83fd5740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdd020070 0x556cdd3cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.261+0000 7f9b83fd5740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9b580115a0 con 0x556cdd020070 2024-07-26T11:27:09.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.261+0000 7f9b83fd5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9b58010ed0 con 0x556cdcff71b0 2024-07-26T11:27:09.265 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.261+0000 7f9b83fd5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5800e130 con 0x556cdcfff390 2024-07-26T11:27:09.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.261+0000 7f9b83fd5740 10 client.0 _ll_register_callbacks cb 0x7f9b6800b5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:27:09.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.262+0000 7f9b75e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcfff390 0x556cdd3cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.262+0000 7f9b75e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcfff390 0x556cdd3cac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:34154/0 (socket says 172.21.15.134:34154) 2024-07-26T11:27:09.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.262+0000 7f9b75e00640 1 -- 172.21.15.134:0/3831374038 learned_addr learned my addr 172.21.15.134:0/3831374038 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:09.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.263+0000 7f9b76800640 1 --2- 172.21.15.134:0/3831374038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcff71b0 0x556cdd3d8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.264+0000 7f9b76800640 1 -- 172.21.15.134:0/3831374038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcfff390 msgr2=0x556cdd3cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:09.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.264+0000 7f9b76800640 1 --2- 172.21.15.134:0/3831374038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcfff390 0x556cdd3cac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.264+0000 7f9b76800640 1 -- 172.21.15.134:0/3831374038 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdd020070 msgr2=0x556cdd3cb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:09.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.264+0000 7f9b76800640 1 --2- 172.21.15.134:0/3831374038 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdd020070 0x556cdd3cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.264+0000 7f9b76800640 1 -- 172.21.15.134:0/3831374038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556cdd3cb870 con 0x556cdcff71b0 2024-07-26T11:27:09.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.265+0000 7f9b57e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:27:09.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.265+0000 7f9b76800640 1 --2- 172.21.15.134:0/3831374038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcff71b0 0x556cdd3d8f20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x556cdd3d36e0 tx=0x7f9b64004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58547217becca424 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:09.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.266+0000 7f9b57e00640 1 -- 172.21.15.134:0/3831374038 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6401c040 con 0x556cdcff71b0 2024-07-26T11:27:09.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.266+0000 7f9b57e00640 1 -- 172.21.15.134:0/3831374038 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(101..101 src has 1..101) ==== 9495+0+0 (secure 0 0 0) 0x7f9b64023430 con 0x556cdcff71b0 2024-07-26T11:27:09.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.266+0000 7f9b83fd5740 1 -- 172.21.15.134:0/3831374038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b700cb6f0 con 0x556cdcff71b0 2024-07-26T11:27:09.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.266+0000 7f9b57e00640 1 -- 172.21.15.134:0/3831374038 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f9b64039050 con 0x556cdcff71b0 2024-07-26T11:27:09.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.266+0000 7f9b83fd5740 10 client.26999 fetch_fsmap learned FSMap version 36 2024-07-26T11:27:09.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.266+0000 7f9b83fd5740 1 -- 172.21.15.134:0/3831374038 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x556cdd3d5730 con 0x556cdcff71b0 2024-07-26T11:27:09.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.267+0000 7f9b57e00640 1 -- 172.21.15.134:0/3831374038 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b700cb6f0 con 0x556cdcff71b0 2024-07-26T11:27:09.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.267+0000 7f9b57e00640 1 -- 172.21.15.134:0/3831374038 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8398+0+0 (secure 0 0 0) 0x7f9b64014810 con 0x556cdcff71b0 2024-07-26T11:27:09.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.267+0000 7f9b83fd5740 10 client.26999 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T11:27:09.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.267+0000 7f9b83fd5740 10 client.26999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:27:09.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.267+0000 7f9b83fd5740 20 client.26999 populate_metadata read hostname 'smithi134' 2024-07-26T11:27:09.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.267+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/3831374038 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556cdcfb3da0 0x556cdd44d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:09.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.267+0000 7f9b83fd5740 4 client.26999 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-07-26T11:27:09.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.267+0000 7f9b83fd5740 1 -- 172.21.15.134:0/3831374038 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556cdd3395c0 con 0x556cdcfb3da0 2024-07-26T11:27:09.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.268+0000 7f9b75e00640 1 --2- 172.21.15.134:0/3831374038 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556cdcfb3da0 0x556cdd44d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:09.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.269+0000 7f9b75e00640 1 --2- 172.21.15.134:0/3831374038 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556cdcfb3da0 0x556cdd44d3f0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec99f735ae059acb server_cookie=b7d911cbb8dd52be in_seq=0 out_seq=0 2024-07-26T11:27:09.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.270+0000 7f9b57e00640 10 client.26999 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:09.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.270+0000 7f9b57e00640 1 -- 172.21.15.134:0/3831374038 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556cdd3395c0 con 0x556cdcfb3da0 2024-07-26T11:27:09.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.270+0000 7f9b57e00640 10 client.26999 handle_command_reply: tid=0 2024-07-26T11:27:09.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.270+0000 7f9b83fd5740 1 client.26999 shutdown 2024-07-26T11:27:09.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.272+0000 7f9b83fd5740 1 -- 172.21.15.134:0/3831374038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcff71b0 msgr2=0x556cdd3d8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:09.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.272+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/3831374038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcff71b0 0x556cdd3d8f20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x556cdd3d36e0 tx=0x7f9b64004770 comp rx=0 tx=0).stop 2024-07-26T11:27:09.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:08 smithi193 ceph-mon[37473]: mgrmap e27: x(active, since 24m), standbys: z, y 2024-07-26T11:27:09.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:08 smithi193 ceph-mon[37473]: pgmap v826: 129 pgs: 129 active+clean; 59 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 176 MiB/s wr, 1.26k op/s 2024-07-26T11:27:09.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.272+0000 7f9b83fd5740 1 -- 172.21.15.134:0/3831374038 shutdown_connections 2024-07-26T11:27:09.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.272+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/3831374038 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x556cdcfb3da0 0x556cdd44d3f0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.272+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/3831374038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556cdcfff390 0x556cdd3cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.272+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/3831374038 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x556cdd020070 0x556cdd3cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.272+0000 7f9b83fd5740 1 --2- 172.21.15.134:0/3831374038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556cdcff71b0 0x556cdd3d8f20 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.272+0000 7f9b83fd5740 1 -- 172.21.15.134:0/3831374038 >> 172.21.15.134:0/3831374038 conn(0x556cdd014460 msgr2=0x556cdd3d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:09.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.272+0000 7f9b57e00640 0 client.26999 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:09.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.273+0000 7f9b83fd5740 1 -- 172.21.15.134:0/3831374038 shutdown_connections 2024-07-26T11:27:09.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.273+0000 7f9b83fd5740 1 -- 172.21.15.134:0/3831374038 wait complete. 2024-07-26T11:27:09.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.273+0000 7f9b83fd5740 20 client.26999 trim_cache size 0 max 16384 2024-07-26T11:27:09.276 INFO:teuthology.orchestra.run.smithi134.stdout:[] 2024-07-26T11:27:09.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.274+0000 7f9b7d400640 1 -- 172.21.15.134:0/875219983 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b5c0b3d40 msgr2=0x7f9b5c0b6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:09.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.274+0000 7f9b7d400640 1 --2- 172.21.15.134:0/875219983 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b5c0b3d40 0x7f9b5c0b6250 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x556cdd3c3ab0 tx=0x7f9b64008040 comp rx=0 tx=0).stop 2024-07-26T11:27:09.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.274+0000 7f9b7d400640 1 -- 172.21.15.134:0/875219983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b7810ba40 msgr2=0x7f9b781ae590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:09.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.274+0000 7f9b7d400640 1 --2- 172.21.15.134:0/875219983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b7810ba40 0x7f9b781ae590 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9b70013f20 tx=0x7f9b70003f10 comp rx=0 tx=0).stop 2024-07-26T11:27:09.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.279+0000 7f9b7d400640 1 -- 172.21.15.134:0/875219983 shutdown_connections 2024-07-26T11:27:09.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.279+0000 7f9b7d400640 1 --2- 172.21.15.134:0/875219983 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b5c0b3d40 0x7f9b5c0b6250 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.279+0000 7f9b7d400640 1 --2- 172.21.15.134:0/875219983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b7810ba40 0x7f9b781ae590 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.279+0000 7f9b7d400640 1 --2- 172.21.15.134:0/875219983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b7810b0f0 0x7f9b781a9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.279+0000 7f9b7d400640 1 --2- 172.21.15.134:0/875219983 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b781057b0 0x7f9b781a8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:09.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.279+0000 7f9b7d400640 1 -- 172.21.15.134:0/875219983 >> 172.21.15.134:0/875219983 conn(0x7f9b78101240 msgr2=0x7f9b78109710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:09.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.280+0000 7f9b7d400640 1 -- 172.21.15.134:0/875219983 shutdown_connections 2024-07-26T11:27:09.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:09.280+0000 7f9b7d400640 1 -- 172.21.15.134:0/875219983 wait complete. 2024-07-26T11:27:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:08 smithi189 ceph-mon[36790]: mgrmap e27: x(active, since 24m), standbys: z, y 2024-07-26T11:27:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:08 smithi189 ceph-mon[36790]: pgmap v826: 129 pgs: 129 active+clean; 59 GiB data, 128 GiB used, 945 GiB / 1.0 TiB avail; 1.7 MiB/s rd, 176 MiB/s wr, 1.26k op/s 2024-07-26T11:27:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:11 smithi134 ceph-mon[31966]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T11:27:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:11 smithi134 ceph-mon[31966]: pgmap v828: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 60 GiB data, 128 GiB used, 944 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 143 MiB/s wr, 1.13k op/s 2024-07-26T11:27:11.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:11 smithi189 ceph-mon[36790]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T11:27:11.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:11 smithi189 ceph-mon[36790]: pgmap v828: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 60 GiB data, 128 GiB used, 944 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 143 MiB/s wr, 1.13k op/s 2024-07-26T11:27:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:11 smithi193 ceph-mon[37473]: osdmap e102: 12 total, 12 up, 12 in 2024-07-26T11:27:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:11 smithi193 ceph-mon[37473]: pgmap v828: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 60 GiB data, 128 GiB used, 944 GiB / 1.0 TiB avail; 1.4 MiB/s rd, 143 MiB/s wr, 1.13k op/s 2024-07-26T11:27:12.802 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-07-26T11:27:12.803 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528 shell --fsid 105fbe5c-4b3e-11ef-bcb7-c7b262605968 -- ceph tell mds.1:0 scrub start / force,recursive 2024-07-26T11:27:13.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:12 smithi189 ceph-mon[36790]: pgmap v829: 129 pgs: 8 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 93 active+clean; 59 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 126 MiB/s wr, 1.06k op/s 2024-07-26T11:27:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:12 smithi193 ceph-mon[37473]: pgmap v829: 129 pgs: 8 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 93 active+clean; 59 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 126 MiB/s wr, 1.06k op/s 2024-07-26T11:27:13.427 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:12 smithi134 ceph-mon[31966]: pgmap v829: 129 pgs: 8 active+clean+snaptrim, 28 active+clean+snaptrim_wait, 93 active+clean; 59 GiB data, 127 GiB used, 946 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 126 MiB/s wr, 1.06k op/s 2024-07-26T11:27:13.765 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/105fbe5c-4b3e-11ef-bcb7-c7b262605968/mon.a/config 2024-07-26T11:27:15.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:15 smithi134 ceph-mon[31966]: pgmap v830: 129 pgs: 15 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 69 active+clean; 57 GiB data, 124 GiB used, 949 GiB / 1.0 TiB avail; 856 KiB/s rd, 122 MiB/s wr, 963 op/s 2024-07-26T11:27:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:15 smithi189 ceph-mon[36790]: pgmap v830: 129 pgs: 15 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 69 active+clean; 57 GiB data, 124 GiB used, 949 GiB / 1.0 TiB avail; 856 KiB/s rd, 122 MiB/s wr, 963 op/s 2024-07-26T11:27:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:15 smithi193 ceph-mon[37473]: pgmap v830: 129 pgs: 15 active+clean+snaptrim, 45 active+clean+snaptrim_wait, 69 active+clean; 57 GiB data, 124 GiB used, 949 GiB / 1.0 TiB avail; 856 KiB/s rd, 122 MiB/s wr, 963 op/s 2024-07-26T11:27:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:16 smithi193 ceph-mon[37473]: pgmap v831: 129 pgs: 17 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 61 active+clean; 56 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 492 KiB/s rd, 107 MiB/s wr, 703 op/s 2024-07-26T11:27:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:16 smithi134 ceph-mon[31966]: pgmap v831: 129 pgs: 17 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 61 active+clean; 56 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 492 KiB/s rd, 107 MiB/s wr, 703 op/s 2024-07-26T11:27:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:16 smithi189 ceph-mon[36790]: pgmap v831: 129 pgs: 17 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 61 active+clean; 56 GiB data, 122 GiB used, 951 GiB / 1.0 TiB avail; 492 KiB/s rd, 107 MiB/s wr, 703 op/s 2024-07-26T11:27:18.533 INFO:tasks.workunit.client.0.smithi134.stdout:stopping iogen 2024-07-26T11:27:18.534 INFO:tasks.workunit.client.0.smithi134.stderr:+ echo 'stopping iogen' 2024-07-26T11:27:18.534 INFO:tasks.workunit.client.0.smithi134.stderr:+ ./iogen -k 2024-07-26T11:27:18.642 INFO:tasks.workunit.client.0.smithi134.stdout:OK 2024-07-26T11:27:18.642 INFO:tasks.workunit.client.0.smithi134.stderr:+ echo OK 2024-07-26T11:27:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.064+0000 7fea79c00640 1 -- 172.21.15.134:0/101425301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea741095c0 msgr2=0x7fea7410fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.064+0000 7fea79c00640 1 --2- 172.21.15.134:0/101425301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea741095c0 0x7fea7410fdf0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fea6c009ca0 tx=0x7fea6c02a8b0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.064+0000 7fea79c00640 1 -- 172.21.15.134:0/101425301 shutdown_connections 2024-07-26T11:27:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.064+0000 7fea79c00640 1 --2- 172.21.15.134:0/101425301 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea74108c70 0x7fea74109080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.064+0000 7fea79c00640 1 --2- 172.21.15.134:0/101425301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea741095c0 0x7fea7410fdf0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.064+0000 7fea79c00640 1 --2- 172.21.15.134:0/101425301 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea74103340 0x7fea74103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.064+0000 7fea79c00640 1 -- 172.21.15.134:0/101425301 >> 172.21.15.134:0/101425301 conn(0x7fea740fed60 msgr2=0x7fea741011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:19.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.065+0000 7fea79c00640 1 -- 172.21.15.134:0/101425301 shutdown_connections 2024-07-26T11:27:19.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.065+0000 7fea79c00640 1 -- 172.21.15.134:0/101425301 wait complete. 2024-07-26T11:27:19.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.067+0000 7fea79c00640 1 Processor -- start 2024-07-26T11:27:19.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.069+0000 7fea79c00640 1 -- start start 2024-07-26T11:27:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.069+0000 7fea79c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea74103340 0x7fea74198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.069+0000 7fea79c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea74108c70 0x7fea74198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.070+0000 7fea79c00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea741095c0 0x7fea7419f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.070+0000 7fea79c00640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fea74113330 con 0x7fea741095c0 2024-07-26T11:27:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.070+0000 7fea79c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fea741131b0 con 0x7fea74103340 2024-07-26T11:27:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.070+0000 7fea79c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea741134b0 con 0x7fea74108c70 2024-07-26T11:27:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.070+0000 7fea73e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea74103340 0x7fea74198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.070+0000 7fea73e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea74103340 0x7fea74198800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:52896/0 (socket says 172.21.15.134:52896) 2024-07-26T11:27:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.070+0000 7fea73e00640 1 -- 172.21.15.134:0/1470553863 learned_addr learned my addr 172.21.15.134:0/1470553863 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.070+0000 7fea73400640 1 --2- 172.21.15.134:0/1470553863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea74108c70 0x7fea74198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.071+0000 7fea73e00640 1 -- 172.21.15.134:0/1470553863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea74108c70 msgr2=0x7fea74198d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.071+0000 7fea73e00640 1 --2- 172.21.15.134:0/1470553863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea74108c70 0x7fea74198d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.071+0000 7fea73e00640 1 -- 172.21.15.134:0/1470553863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea741095c0 msgr2=0x7fea7419f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.071+0000 7fea73e00640 1 --2- 172.21.15.134:0/1470553863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea741095c0 0x7fea7419f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.071+0000 7fea73e00640 1 -- 172.21.15.134:0/1470553863 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea741995e0 con 0x7fea74103340 2024-07-26T11:27:19.078 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.076+0000 7fea73e00640 1 --2- 172.21.15.134:0/1470553863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea74103340 0x7fea74198800 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fea6400b4a0 tx=0x7fea6400b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:19.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.079+0000 7fea70c00640 1 -- 172.21.15.134:0/1470553863 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea64014070 con 0x7fea74103340 2024-07-26T11:27:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.079+0000 7fea70c00640 1 -- 172.21.15.134:0/1470553863 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea64010ac0 con 0x7fea74103340 2024-07-26T11:27:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.079+0000 7fea70c00640 1 -- 172.21.15.134:0/1470553863 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea64017c30 con 0x7fea74103340 2024-07-26T11:27:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.079+0000 7fea79c00640 1 -- 172.21.15.134:0/1470553863 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea7419fe60 con 0x7fea74103340 2024-07-26T11:27:19.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.080+0000 7fea79c00640 1 -- 172.21.15.134:0/1470553863 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea741a0360 con 0x7fea74103340 2024-07-26T11:27:19.092 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.089+0000 7fea82237740 1 Processor -- start 2024-07-26T11:27:19.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.091+0000 7fea82237740 1 -- start start 2024-07-26T11:27:19.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.095+0000 7fea70c00640 1 -- 172.21.15.134:0/1470553863 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 151191+0+0 (secure 0 0 0) 0x7fea64017070 con 0x7fea74103340 2024-07-26T11:27:19.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.095+0000 7fea82237740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 0x564265600c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.096+0000 7fea82237740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 0x564265601160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.096+0000 7fea82237740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 0x5642656016a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.096+0000 7fea82237740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x564265575270 con 0x5642652342f0 2024-07-26T11:27:19.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.096+0000 7fea82237740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564265564770 con 0x56426523c4d0 2024-07-26T11:27:19.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.096+0000 7fea82237740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x5642655687b0 con 0x56426525d1b0 2024-07-26T11:27:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.096+0000 7fea79200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 0x564265600c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.096+0000 7fea73e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 0x564265601160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.097+0000 7fea79200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 0x564265600c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:48522/0 (socket says 172.21.15.134:48522) 2024-07-26T11:27:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.097+0000 7fea79200640 1 -- 172.21.15.134:0/3070812315 learned_addr learned my addr 172.21.15.134:0/3070812315 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.096+0000 7fea73400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 0x5642656016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.097+0000 7fea73e00640 1 -- 172.21.15.134:0/3070812315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 msgr2=0x564265600c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:19.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.097+0000 7fea73e00640 1 --2- 172.21.15.134:0/3070812315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 0x564265600c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.097+0000 7fea73e00640 1 -- 172.21.15.134:0/3070812315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 msgr2=0x5642656016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.097+0000 7fea73e00640 1 --2- 172.21.15.134:0/3070812315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 0x5642656016a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.097+0000 7fea73e00640 1 -- 172.21.15.134:0/3070812315 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642652c8790 con 0x5642652342f0 2024-07-26T11:27:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.097+0000 7fea70c00640 1 --2- 172.21.15.134:0/1470553863 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fea580b3e90 0x7fea580b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.098+0000 7fea70c00640 1 -- 172.21.15.134:0/1470553863 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7fea640e3940 con 0x7fea74103340 2024-07-26T11:27:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.099+0000 7fea73e00640 1 --2- 172.21.15.134:0/3070812315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 0x564265601160 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fea640e0d70 tx=0x7fea640a77a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a4b24bb6d8d11b server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.099+0000 7fea79200640 1 --2- 172.21.15.134:0/1470553863 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fea580b3e90 0x7fea580b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.100+0000 7fea69600640 1 -- 172.21.15.134:0/3070812315 <== mon.0 v2:172.21.15.134:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6401e070 con 0x5642652342f0 2024-07-26T11:27:19.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.100+0000 7fea69600640 1 -- 172.21.15.134:0/3070812315 <== mon.0 v2:172.21.15.134:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea640b4200 con 0x5642652342f0 2024-07-26T11:27:19.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.100+0000 7fea69600640 1 -- 172.21.15.134:0/3070812315 <== mon.0 v2:172.21.15.134:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea640aaaa0 con 0x5642652342f0 2024-07-26T11:27:19.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.102+0000 7fea82237740 1 -- 172.21.15.134:0/3070812315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 msgr2=0x564265601160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:19.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.102+0000 7fea82237740 1 --2- 172.21.15.134:0/3070812315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 0x564265601160 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fea640e0d70 tx=0x7fea640a77a0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.105+0000 7fea82237740 1 -- 172.21.15.134:0/3070812315 shutdown_connections 2024-07-26T11:27:19.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.105+0000 7fea82237740 1 --2- 172.21.15.134:0/3070812315 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 0x5642656016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.105+0000 7fea82237740 1 --2- 172.21.15.134:0/3070812315 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 0x564265601160 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.105+0000 7fea82237740 1 --2- 172.21.15.134:0/3070812315 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 0x564265600c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.108 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.105+0000 7fea82237740 1 -- 172.21.15.134:0/3070812315 >> 172.21.15.134:0/3070812315 conn(0x564265251300 msgr2=0x5642655feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:19.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.107+0000 7fea82237740 1 -- 172.21.15.134:0/3070812315 shutdown_connections 2024-07-26T11:27:19.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.107+0000 7fea79200640 1 --2- 172.21.15.134:0/1470553863 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fea580b3e90 0x7fea580b63a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fea6c0094e0 tx=0x7fea6c005da0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:19.110 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.108+0000 7fea82237740 1 -- 172.21.15.134:0/3070812315 wait complete. 2024-07-26T11:27:19.114 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.109+0000 7fea82237740 1 -- start start 2024-07-26T11:27:19.115 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.113+0000 7fea82237740 1 Processor -- start 2024-07-26T11:27:19.117 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.115+0000 7fea82237740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 0x56426560c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.116+0000 7fea73e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 0x56426560c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.116+0000 7fea82237740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 0x56426560cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.116+0000 7fea73e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 0x56426560c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.134:3300/0 says I am v2:172.21.15.134:41916/0 (socket says 172.21.15.134:41916) 2024-07-26T11:27:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.116+0000 7fea73e00640 1 -- 172.21.15.134:0/257692979 learned_addr learned my addr 172.21.15.134:0/257692979 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.116+0000 7fea82237740 1 --2- 172.21.15.134:0/257692979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 0x564265618690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.117+0000 7fea79200640 1 --2- 172.21.15.134:0/257692979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 0x564265618690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.117+0000 7fea82237740 1 -- 172.21.15.134:0/257692979 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x56426549eab0 con 0x5642652342f0 2024-07-26T11:27:19.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.117+0000 7fea82237740 1 -- 172.21.15.134:0/257692979 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564265501d50 con 0x56426523c4d0 2024-07-26T11:27:19.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.118+0000 7fea82237740 1 -- 172.21.15.134:0/257692979 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x56426560f3a0 con 0x56426525d1b0 2024-07-26T11:27:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.119+0000 7fea73400640 1 --2- 172.21.15.134:0/257692979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 0x56426560cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.119+0000 7fea79200640 1 -- 172.21.15.134:0/257692979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 msgr2=0x56426560cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.119+0000 7fea79200640 1 --2- 172.21.15.134:0/257692979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 0x56426560cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.119+0000 7fea79200640 1 -- 172.21.15.134:0/257692979 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 msgr2=0x56426560c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.119+0000 7fea79200640 1 --2- 172.21.15.134:0/257692979 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 0x56426560c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.119+0000 7fea79200640 1 -- 172.21.15.134:0/257692979 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564265341620 con 0x56426525d1b0 2024-07-26T11:27:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.119+0000 7fea82237740 10 client.0 _ll_register_callbacks cb 0x564265274a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:27:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.120+0000 7fea82237740 1 -- 172.21.15.134:0/257692979 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5642655771c0 con 0x56426525d1b0 2024-07-26T11:27:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.120+0000 7fea79200640 1 --2- 172.21.15.134:0/257692979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 0x564265618690 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fea6c009ca0 tx=0x7fea6c0095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=208a0d200ca3671 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.120+0000 7fea5e000640 10 client.27025 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:27:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.122+0000 7fea5e000640 1 -- 172.21.15.134:0/257692979 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6c004450 con 0x56426525d1b0 2024-07-26T11:27:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.122+0000 7fea5e000640 1 -- 172.21.15.134:0/257692979 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7fea6c0045f0 con 0x56426525d1b0 2024-07-26T11:27:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.122+0000 7fea5e000640 1 -- 172.21.15.134:0/257692979 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fea6c0047f0 con 0x56426525d1b0 2024-07-26T11:27:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.122+0000 7fea82237740 10 client.27025 fetch_fsmap learned FSMap version 36 2024-07-26T11:27:19.124 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.122+0000 7fea82237740 1 -- 172.21.15.134:0/257692979 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=36}) -- 0x56426561b250 con 0x56426525d1b0 2024-07-26T11:27:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.124+0000 7fea5e000640 1 -- 172.21.15.134:0/257692979 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 36) ==== 8398+0+0 (secure 0 0 0) 0x7fea6c04e070 con 0x56426525d1b0 2024-07-26T11:27:19.126 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.124+0000 7fea82237740 10 client.27025 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T11:27:19.126 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.124+0000 7fea82237740 10 client.27025 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:27:19.126 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.125+0000 7fea82237740 20 client.27025 populate_metadata read hostname 'smithi134' 2024-07-26T11:27:19.127 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.125+0000 7fea82237740 1 --2- 172.21.15.134:0/257692979 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56426521bd80 0x56426561e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.128 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.126+0000 7fea82237740 4 client.27025 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:27:19.128 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.126+0000 7fea82237740 1 -- 172.21.15.134:0/257692979 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564265572c60 con 0x56426521bd80 2024-07-26T11:27:19.128 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.127+0000 7fea73e00640 1 --2- 172.21.15.134:0/257692979 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56426521bd80 0x56426561e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.130 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.128+0000 7fea73e00640 1 --2- 172.21.15.134:0/257692979 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56426521bd80 0x56426561e790 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90c06618cbdeae40 server_cookie=92b975be50dee69d in_seq=0 out_seq=0 2024-07-26T11:27:19.130 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.128+0000 7fea5e000640 10 client.27025 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.132+0000 7fea5e000640 1 -- 172.21.15.134:0/257692979 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24171 (crc 0 0 0) 0x564265572c60 con 0x56426521bd80 2024-07-26T11:27:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.132+0000 7fea5e000640 10 client.27025 handle_command_reply: tid=0 2024-07-26T11:27:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.133+0000 7fea82237740 1 client.27025 shutdown 2024-07-26T11:27:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.134+0000 7fea82237740 1 -- 172.21.15.134:0/257692979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 msgr2=0x564265618690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.134+0000 7fea82237740 1 --2- 172.21.15.134:0/257692979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 0x564265618690 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fea6c009ca0 tx=0x7fea6c0095c0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.135+0000 7fea82237740 1 -- 172.21.15.134:0/257692979 shutdown_connections 2024-07-26T11:27:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.135+0000 7fea82237740 1 --2- 172.21.15.134:0/257692979 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x56426521bd80 0x56426561e790 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.135+0000 7fea82237740 1 --2- 172.21.15.134:0/257692979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 0x564265618690 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.135+0000 7fea82237740 1 --2- 172.21.15.134:0/257692979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 0x56426560cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.135+0000 7fea82237740 1 --2- 172.21.15.134:0/257692979 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 0x56426560c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.135+0000 7fea82237740 1 -- 172.21.15.134:0/257692979 >> 172.21.15.134:0/257692979 conn(0x564265251300 msgr2=0x564265601c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.135+0000 7fea5e000640 0 client.27025 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.135+0000 7fea82237740 1 -- 172.21.15.134:0/257692979 shutdown_connections 2024-07-26T11:27:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.137+0000 7fea82237740 1 -- 172.21.15.134:0/257692979 wait complete. 2024-07-26T11:27:19.139 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.137+0000 7fea82237740 20 client.27025 trim_cache size 0 max 16384 2024-07-26T11:27:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.180+0000 7fea82237740 1 Processor -- start 2024-07-26T11:27:19.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.180+0000 7fea82237740 1 -- start start 2024-07-26T11:27:19.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.181+0000 7fea82237740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642652342f0 0x564265602610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.181+0000 7fea82237740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426523c4d0 0x564265606cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.181+0000 7fea82237740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56426525d1b0 0x56426560e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.181+0000 7fea82237740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fea741bf870 con 0x56426525d1b0 2024-07-26T11:27:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.181+0000 7fea82237740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fea741c1470 con 0x5642652342f0 2024-07-26T11:27:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.181+0000 7fea82237740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea741bff40 con 0x56426523c4d0 2024-07-26T11:27:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.182+0000 7fea73400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426523c4d0 0x564265606cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.182+0000 7fea73400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426523c4d0 0x564265606cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:48540/0 (socket says 172.21.15.134:48540) 2024-07-26T11:27:19.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.182+0000 7fea73400640 1 -- 172.21.15.134:0/3584338361 learned_addr learned my addr 172.21.15.134:0/3584338361 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:19.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.182+0000 7fea73e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642652342f0 0x564265602610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.183+0000 7fea79200640 1 --2- 172.21.15.134:0/3584338361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56426525d1b0 0x56426560e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.184+0000 7fea73400640 1 -- 172.21.15.134:0/3584338361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642652342f0 msgr2=0x564265602610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.184+0000 7fea73400640 1 --2- 172.21.15.134:0/3584338361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642652342f0 0x564265602610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.187 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.184+0000 7fea73400640 1 -- 172.21.15.134:0/3584338361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56426525d1b0 msgr2=0x56426560e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:19.187 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.184+0000 7fea73400640 1 --2- 172.21.15.134:0/3584338361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56426525d1b0 0x56426560e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.187 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.184+0000 7fea73400640 1 -- 172.21.15.134:0/3584338361 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564265603a30 con 0x56426523c4d0 2024-07-26T11:27:19.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.185+0000 7fea73400640 1 --2- 172.21.15.134:0/3584338361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426523c4d0 0x564265606cf0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fea60009d30 tx=0x7fea60013830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=383cd653e343ad44 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.187+0000 7fea68c00640 1 -- 172.21.15.134:0/3584338361 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea60004050 con 0x56426523c4d0 2024-07-26T11:27:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.187+0000 7fea68c00640 1 -- 172.21.15.134:0/3584338361 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea600041f0 con 0x56426523c4d0 2024-07-26T11:27:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.187+0000 7fea68c00640 1 -- 172.21.15.134:0/3584338361 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea600116a0 con 0x56426523c4d0 2024-07-26T11:27:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.189+0000 7fea82237740 1 -- 172.21.15.134:0/3584338361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426523c4d0 msgr2=0x564265606cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.189+0000 7fea82237740 1 --2- 172.21.15.134:0/3584338361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426523c4d0 0x564265606cf0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fea60009d30 tx=0x7fea60013830 comp rx=0 tx=0).stop 2024-07-26T11:27:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.189+0000 7fea82237740 1 -- 172.21.15.134:0/3584338361 shutdown_connections 2024-07-26T11:27:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.189+0000 7fea82237740 1 --2- 172.21.15.134:0/3584338361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426523c4d0 0x564265606cf0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.189+0000 7fea82237740 1 --2- 172.21.15.134:0/3584338361 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x56426525d1b0 0x56426560e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.189+0000 7fea82237740 1 --2- 172.21.15.134:0/3584338361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5642652342f0 0x564265602610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.189+0000 7fea82237740 1 -- 172.21.15.134:0/3584338361 >> 172.21.15.134:0/3584338361 conn(0x564265251300 msgr2=0x56426560dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.190+0000 7fea82237740 1 -- 172.21.15.134:0/3584338361 shutdown_connections 2024-07-26T11:27:19.192 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.190+0000 7fea82237740 1 -- 172.21.15.134:0/3584338361 wait complete. 2024-07-26T11:27:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.191+0000 7fea82237740 1 -- start start 2024-07-26T11:27:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.191+0000 7fea82237740 1 Processor -- start 2024-07-26T11:27:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.192+0000 7fea82237740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 0x564265614980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.192+0000 7fea82237740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 0x56426560fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.192+0000 7fea82237740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 0x5642656103b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.193+0000 7fea73400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 0x56426560fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.195 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.194+0000 7fea82237740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7fea741c0ce0 con 0x5642652342f0 2024-07-26T11:27:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.194+0000 7fea82237740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fea741c0610 con 0x56426523c4d0 2024-07-26T11:27:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.194+0000 7fea82237740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7fea741bd8c0 con 0x56426525d1b0 2024-07-26T11:27:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.194+0000 7fea82237740 10 client.0 _ll_register_callbacks cb 0x7fea6c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:27:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.194+0000 7fea79200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 0x5642656103b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.194+0000 7fea79200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 0x5642656103b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:48552/0 (socket says 172.21.15.134:48552) 2024-07-26T11:27:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.194+0000 7fea79200640 1 -- 172.21.15.134:0/3833993803 learned_addr learned my addr 172.21.15.134:0/3833993803 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.194+0000 7fea73400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 0x56426560fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:52926/0 (socket says 172.21.15.134:52926) 2024-07-26T11:27:19.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.194+0000 7fea73e00640 1 --2- 172.21.15.134:0/3833993803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 0x564265614980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.196+0000 7fea79200640 1 -- 172.21.15.134:0/3833993803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 msgr2=0x56426560fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.196+0000 7fea79200640 1 --2- 172.21.15.134:0/3833993803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 0x56426560fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.196+0000 7fea79200640 1 -- 172.21.15.134:0/3833993803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 msgr2=0x564265614980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.196+0000 7fea79200640 1 --2- 172.21.15.134:0/3833993803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 0x564265614980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.196+0000 7fea79200640 1 -- 172.21.15.134:0/3833993803 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642656108f0 con 0x56426525d1b0 2024-07-26T11:27:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.196+0000 7fea73e00640 1 --2- 172.21.15.134:0/3833993803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 0x564265614980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:27:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.196+0000 7fea73400640 1 --2- 172.21.15.134:0/3833993803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 0x56426560fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:27:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.197+0000 7fea79200640 1 --2- 172.21.15.134:0/3833993803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 0x5642656103b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fea6c003ff0 tx=0x7fea6c042000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66e4e6fde0602f90 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.197+0000 7fea5e000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:27:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.197+0000 7fea5e000640 1 -- 172.21.15.134:0/3833993803 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6c004650 con 0x56426525d1b0 2024-07-26T11:27:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.198+0000 7fea5e000640 1 -- 172.21.15.134:0/3833993803 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7fea6c045d70 con 0x56426525d1b0 2024-07-26T11:27:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.198+0000 7fea82237740 1 -- 172.21.15.134:0/3833993803 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea600041f0 con 0x56426525d1b0 2024-07-26T11:27:19.200 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.198+0000 7fea5e000640 1 -- 172.21.15.134:0/3833993803 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fea6c050440 con 0x56426525d1b0 2024-07-26T11:27:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.198+0000 7fea82237740 10 client.27031 fetch_fsmap learned FSMap version 36 2024-07-26T11:27:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.198+0000 7fea82237740 1 -- 172.21.15.134:0/3833993803 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x564265686b70 con 0x56426525d1b0 2024-07-26T11:27:19.201 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.200+0000 7fea5e000640 1 -- 172.21.15.134:0/3833993803 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea600041f0 con 0x56426525d1b0 2024-07-26T11:27:19.205 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.203+0000 7fea5e000640 1 -- 172.21.15.134:0/3833993803 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 36) ==== 8398+0+0 (secure 0 0 0) 0x7fea6c04e070 con 0x56426525d1b0 2024-07-26T11:27:19.205 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.203+0000 7fea82237740 10 client.27031 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T11:27:19.206 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.204+0000 7fea82237740 10 client.27031 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:27:19.206 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.204+0000 7fea82237740 20 client.27031 populate_metadata read hostname 'smithi134' 2024-07-26T11:27:19.206 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.204+0000 7fea82237740 1 --2- 172.21.15.134:0/3833993803 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5642651f11c0 0x5642656893f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:19.206 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.204+0000 7fea82237740 4 client.27031 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-07-26T11:27:19.206 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.204+0000 7fea82237740 1 -- 172.21.15.134:0/3833993803 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5642655703a0 con 0x5642651f11c0 2024-07-26T11:27:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.204+0000 7fea73e00640 1 --2- 172.21.15.134:0/3833993803 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5642651f11c0 0x5642656893f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.205+0000 7fea73e00640 1 --2- 172.21.15.134:0/3833993803 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5642651f11c0 0x5642656893f0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f3f18e4f1fb14f5 server_cookie=16decd0c1f54c294 in_seq=0 out_seq=0 2024-07-26T11:27:19.207 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.205+0000 7fea5e000640 10 client.27031 ms_handle_connect on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:18 smithi193 ceph-mon[37473]: pgmap v832: 129 pgs: 17 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 61 active+clean; 55 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 360 KiB/s rd, 105 MiB/s wr, 650 op/s 2024-07-26T11:27:19.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:18 smithi134 ceph-mon[31966]: pgmap v832: 129 pgs: 17 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 61 active+clean; 55 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 360 KiB/s rd, 105 MiB/s wr, 650 op/s 2024-07-26T11:27:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:18 smithi189 ceph-mon[36790]: pgmap v832: 129 pgs: 17 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 61 active+clean; 55 GiB data, 120 GiB used, 953 GiB / 1.0 TiB avail; 360 KiB/s rd, 105 MiB/s wr, 650 op/s 2024-07-26T11:27:19.585 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.583+0000 7fea5e000640 1 -- 172.21.15.134:0/3833993803 <== mds.0 v2:172.21.15.134:6840/4067549550 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5642655703a0 con 0x5642651f11c0 2024-07-26T11:27:19.585 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.583+0000 7fea5e000640 10 client.27031 handle_command_reply: tid=0 2024-07-26T11:27:19.585 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.583+0000 7fea82237740 1 client.27031 shutdown 2024-07-26T11:27:19.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.584+0000 7fea82237740 1 -- 172.21.15.134:0/3833993803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 msgr2=0x5642656103b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:19.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.584+0000 7fea82237740 1 --2- 172.21.15.134:0/3833993803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 0x5642656103b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fea6c003ff0 tx=0x7fea6c042000 comp rx=0 tx=0).stop 2024-07-26T11:27:19.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.585+0000 7fea82237740 1 -- 172.21.15.134:0/3833993803 shutdown_connections 2024-07-26T11:27:19.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.585+0000 7fea82237740 1 --2- 172.21.15.134:0/3833993803 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x5642651f11c0 0x5642656893f0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.585+0000 7fea82237740 1 --2- 172.21.15.134:0/3833993803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56426525d1b0 0x5642656103b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.585+0000 7fea82237740 1 --2- 172.21.15.134:0/3833993803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56426523c4d0 0x56426560fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.585+0000 7fea82237740 1 --2- 172.21.15.134:0/3833993803 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x5642652342f0 0x564265614980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.585+0000 7fea82237740 1 -- 172.21.15.134:0/3833993803 >> 172.21.15.134:0/3833993803 conn(0x564265251300 msgr2=0x56426560e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:19.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.585+0000 7fea5e000640 0 client.27031 ms_handle_reset on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:19.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.585+0000 7fea82237740 1 -- 172.21.15.134:0/3833993803 shutdown_connections 2024-07-26T11:27:19.588 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.585+0000 7fea82237740 1 -- 172.21.15.134:0/3833993803 wait complete. 2024-07-26T11:27:19.588 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.586+0000 7fea82237740 20 client.27031 trim_cache size 0 max 16384 2024-07-26T11:27:19.589 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:{ 2024-07-26T11:27:19.589 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "message": "", 2024-07-26T11:27:19.589 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout: "return_code": 0 2024-07-26T11:27:19.589 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stdout:} 2024-07-26T11:27:19.591 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.589+0000 7fea79c00640 1 -- 172.21.15.134:0/1470553863 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fea580b3e90 msgr2=0x7fea580b63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:19.591 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.589+0000 7fea79c00640 1 --2- 172.21.15.134:0/1470553863 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fea580b3e90 0x7fea580b63a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fea6c0094e0 tx=0x7fea6c005da0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.591 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.589+0000 7fea79c00640 1 -- 172.21.15.134:0/1470553863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea74103340 msgr2=0x7fea74198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:19.591 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.589+0000 7fea79c00640 1 --2- 172.21.15.134:0/1470553863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea74103340 0x7fea74198800 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fea6400b4a0 tx=0x7fea6400b960 comp rx=0 tx=0).stop 2024-07-26T11:27:19.593 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.591+0000 7fea79c00640 1 -- 172.21.15.134:0/1470553863 shutdown_connections 2024-07-26T11:27:19.593 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.591+0000 7fea79c00640 1 --2- 172.21.15.134:0/1470553863 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7fea580b3e90 0x7fea580b63a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.593 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.591+0000 7fea79c00640 1 --2- 172.21.15.134:0/1470553863 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fea74108c70 0x7fea74198d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.593 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.591+0000 7fea79c00640 1 --2- 172.21.15.134:0/1470553863 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7fea741095c0 0x7fea7419f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.593 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.591+0000 7fea79c00640 1 --2- 172.21.15.134:0/1470553863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea74103340 0x7fea74198800 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:19.593 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.591+0000 7fea79c00640 1 -- 172.21.15.134:0/1470553863 >> 172.21.15.134:0/1470553863 conn(0x7fea740fed60 msgr2=0x7fea74102250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:19.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.591+0000 7fea79c00640 1 -- 172.21.15.134:0/1470553863 shutdown_connections 2024-07-26T11:27:19.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:19.592+0000 7fea79c00640 1 -- 172.21.15.134:0/1470553863 wait complete. 2024-07-26T11:27:20.692 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-07-26T11:27:20.692 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-07-26T11:27:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:20 smithi134 ceph-mon[31966]: pgmap v833: 129 pgs: 12 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 81 active+clean; 54 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 189 KiB/s rd, 132 MiB/s wr, 799 op/s 2024-07-26T11:27:21.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:20 smithi189 ceph-mon[36790]: pgmap v833: 129 pgs: 12 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 81 active+clean; 54 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 189 KiB/s rd, 132 MiB/s wr, 799 op/s 2024-07-26T11:27:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:20 smithi193 ceph-mon[37473]: pgmap v833: 129 pgs: 12 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 81 active+clean; 54 GiB data, 118 GiB used, 955 GiB / 1.0 TiB avail; 189 KiB/s rd, 132 MiB/s wr, 799 op/s 2024-07-26T11:27:21.506 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:27:21.506 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: *** Caught signal (Aborted) ** 2024-07-26T11:27:21.506 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: in thread 7f6288823640 thread_name:ms_dispatch 2024-07-26T11:27:21.506 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:21.506 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f628dbfa6f0] 2024-07-26T11:27:21.506 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f628dc4794c] 2024-07-26T11:27:21.506 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 3: raise() 2024-07-26T11:27:21.506 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 4: abort() 2024-07-26T11:27:21.506 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 5: /lib64/libc.so.6(+0x2871b) [0x7f628dbe471b] 2024-07-26T11:27:21.506 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 6: /lib64/libc.so.6(+0x37386) [0x7f628dbf3386] 2024-07-26T11:27:21.507 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56433d0da6b9] 2024-07-26T11:27:21.507 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56433d29880c] 2024-07-26T11:27:21.507 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56433d1d82ca] 2024-07-26T11:27:21.507 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56433d1a8753] 2024-07-26T11:27:21.507 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56433d1a9a1b] 2024-07-26T11:27:21.507 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56433d18d752] 2024-07-26T11:27:21.507 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56433d0dba56] 2024-07-26T11:27:21.507 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56433d0df637] 2024-07-26T11:27:21.507 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56433d0dff5e] 2024-07-26T11:27:21.507 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56433d0c303a] 2024-07-26T11:27:21.507 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f628e4c7407] 2024-07-26T11:27:21.507 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 18: (DispatchQueue::entry()+0x60f) [0x7f628e4c7cf9] 2024-07-26T11:27:21.507 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f628e5a45a1] 2024-07-26T11:27:21.508 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:27:21.508 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: *** Caught signal (Aborted) ** 2024-07-26T11:27:21.508 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: in thread 7f3b46a68640 thread_name:ms_dispatch 2024-07-26T11:27:21.508 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:21.508 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3b4be3f6f0] 2024-07-26T11:27:21.508 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3b4be8c94c] 2024-07-26T11:27:21.509 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 3: raise() 2024-07-26T11:27:21.509 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 4: abort() 2024-07-26T11:27:21.509 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3b4be2971b] 2024-07-26T11:27:21.509 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 6: /lib64/libc.so.6(+0x37386) [0x7f3b4be38386] 2024-07-26T11:27:21.509 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b4f1d706b9] 2024-07-26T11:27:21.509 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b4f1f2e80c] 2024-07-26T11:27:21.509 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b4f1e6e2ca] 2024-07-26T11:27:21.509 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b4f1e3e753] 2024-07-26T11:27:21.509 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b4f1e3fa1b] 2024-07-26T11:27:21.509 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b4f1e23752] 2024-07-26T11:27:21.509 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b4f1d71a56] 2024-07-26T11:27:21.510 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b4f1d75637] 2024-07-26T11:27:21.510 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b4f1d75f5e] 2024-07-26T11:27:21.510 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b4f1d5903a] 2024-07-26T11:27:21.510 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3b4c70c407] 2024-07-26T11:27:21.510 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 18: (DispatchQueue::entry()+0x60f) [0x7f3b4c70ccf9] 2024-07-26T11:27:21.510 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3b4c7e95a1] 2024-07-26T11:27:21.510 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 20: (Thread::entry_wrapper()+0x43) [0x7f628e351cdd] 2024-07-26T11:27:21.510 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 21: (Thread::_entry_func(void*)+0xd) [0x7f628e351cf9] 2024-07-26T11:27:21.510 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 22: /lib64/libc.so.6(+0x89c02) [0x7f628dc45c02] 2024-07-26T11:27:21.510 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f628dccac40] 2024-07-26T11:27:21.511 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 2024-07-26T11:27:21.412+0000 7f6288823640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:21.511 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: in thread 7f6288823640 thread_name:ms_dispatch 2024-07-26T11:27:21.511 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 2024-07-26T11:27:21.511 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:21.511 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f628dbfa6f0] 2024-07-26T11:27:21.511 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f628dc4794c] 2024-07-26T11:27:21.511 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 3: raise() 2024-07-26T11:27:21.511 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 4: abort() 2024-07-26T11:27:21.511 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 5: /lib64/libc.so.6(+0x2871b) [0x7f628dbe471b] 2024-07-26T11:27:21.511 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 6: /lib64/libc.so.6(+0x37386) [0x7f628dbf3386] 2024-07-26T11:27:21.512 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56433d0da6b9] 2024-07-26T11:27:21.512 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56433d29880c] 2024-07-26T11:27:21.512 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56433d1d82ca] 2024-07-26T11:27:21.512 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56433d1a8753] 2024-07-26T11:27:21.512 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56433d1a9a1b] 2024-07-26T11:27:21.512 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56433d18d752] 2024-07-26T11:27:21.512 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56433d0dba56] 2024-07-26T11:27:21.512 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56433d0df637] 2024-07-26T11:27:21.512 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56433d0dff5e] 2024-07-26T11:27:21.512 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56433d0c303a] 2024-07-26T11:27:21.512 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f628e4c7407] 2024-07-26T11:27:21.512 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 18: (DispatchQueue::entry()+0x60f) [0x7f628e4c7cf9] 2024-07-26T11:27:21.512 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f628e5a45a1] 2024-07-26T11:27:21.513 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 20: (Thread::entry_wrapper()+0x43) [0x7f3b4c596cdd] 2024-07-26T11:27:21.513 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3b4c596cf9] 2024-07-26T11:27:21.513 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3b4be8ac02] 2024-07-26T11:27:21.513 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3b4bf0fc40] 2024-07-26T11:27:21.513 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 2024-07-26T11:27:21.412+0000 7f3b46a68640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:21.513 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: in thread 7f3b46a68640 thread_name:ms_dispatch 2024-07-26T11:27:21.513 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 2024-07-26T11:27:21.513 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:21.513 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3b4be3f6f0] 2024-07-26T11:27:21.514 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3b4be8c94c] 2024-07-26T11:27:21.514 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 3: raise() 2024-07-26T11:27:21.514 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 4: abort() 2024-07-26T11:27:21.514 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3b4be2971b] 2024-07-26T11:27:21.514 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 6: /lib64/libc.so.6(+0x37386) [0x7f3b4be38386] 2024-07-26T11:27:21.514 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b4f1d706b9] 2024-07-26T11:27:21.514 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b4f1f2e80c] 2024-07-26T11:27:21.514 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b4f1e6e2ca] 2024-07-26T11:27:21.514 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b4f1e3e753] 2024-07-26T11:27:21.514 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b4f1e3fa1b] 2024-07-26T11:27:21.514 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b4f1e23752] 2024-07-26T11:27:21.514 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b4f1d71a56] 2024-07-26T11:27:21.515 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b4f1d75637] 2024-07-26T11:27:21.515 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b4f1d75f5e] 2024-07-26T11:27:21.515 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b4f1d5903a] 2024-07-26T11:27:21.515 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3b4c70c407] 2024-07-26T11:27:21.515 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 18: (DispatchQueue::entry()+0x60f) [0x7f3b4c70ccf9] 2024-07-26T11:27:21.515 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3b4c7e95a1] 2024-07-26T11:27:21.515 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 20: (Thread::entry_wrapper()+0x43) [0x7f3b4c596cdd] 2024-07-26T11:27:21.515 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3b4c596cf9] 2024-07-26T11:27:21.515 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3b4be8ac02] 2024-07-26T11:27:21.515 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3b4bf0fc40] 2024-07-26T11:27:21.515 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:21.515 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 2024-07-26T11:27:21.515 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 0> 2024-07-26T11:27:21.412+0000 7f3b46a68640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:21.515 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: in thread 7f3b46a68640 thread_name:ms_dispatch 2024-07-26T11:27:21.516 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 2024-07-26T11:27:21.516 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:21.516 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3b4be3f6f0] 2024-07-26T11:27:21.516 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3b4be8c94c] 2024-07-26T11:27:21.516 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 3: raise() 2024-07-26T11:27:21.516 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 4: abort() 2024-07-26T11:27:21.516 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3b4be2971b] 2024-07-26T11:27:21.516 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 6: /lib64/libc.so.6(+0x37386) [0x7f3b4be38386] 2024-07-26T11:27:21.516 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b4f1d706b9] 2024-07-26T11:27:21.516 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b4f1f2e80c] 2024-07-26T11:27:21.516 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b4f1e6e2ca] 2024-07-26T11:27:21.516 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b4f1e3e753] 2024-07-26T11:27:21.516 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b4f1e3fa1b] 2024-07-26T11:27:21.517 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b4f1e23752] 2024-07-26T11:27:21.517 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b4f1d71a56] 2024-07-26T11:27:21.517 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b4f1d75637] 2024-07-26T11:27:21.517 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b4f1d75f5e] 2024-07-26T11:27:21.517 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b4f1d5903a] 2024-07-26T11:27:21.517 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3b4c70c407] 2024-07-26T11:27:21.517 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 20: (Thread::entry_wrapper()+0x43) [0x7f628e351cdd] 2024-07-26T11:27:21.517 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 21: (Thread::_entry_func(void*)+0xd) [0x7f628e351cf9] 2024-07-26T11:27:21.517 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 22: /lib64/libc.so.6(+0x89c02) [0x7f628dc45c02] 2024-07-26T11:27:21.517 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f628dccac40] 2024-07-26T11:27:21.517 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:21.518 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 2024-07-26T11:27:21.518 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 0> 2024-07-26T11:27:21.412+0000 7f6288823640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:21.518 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: in thread 7f6288823640 thread_name:ms_dispatch 2024-07-26T11:27:21.518 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 2024-07-26T11:27:21.518 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:21.518 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f628dbfa6f0] 2024-07-26T11:27:21.518 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f628dc4794c] 2024-07-26T11:27:21.518 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 3: raise() 2024-07-26T11:27:21.518 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 4: abort() 2024-07-26T11:27:21.518 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 5: /lib64/libc.so.6(+0x2871b) [0x7f628dbe471b] 2024-07-26T11:27:21.518 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 6: /lib64/libc.so.6(+0x37386) [0x7f628dbf3386] 2024-07-26T11:27:21.518 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56433d0da6b9] 2024-07-26T11:27:21.519 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56433d29880c] 2024-07-26T11:27:21.519 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56433d1d82ca] 2024-07-26T11:27:21.519 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56433d1a8753] 2024-07-26T11:27:21.519 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56433d1a9a1b] 2024-07-26T11:27:21.519 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56433d18d752] 2024-07-26T11:27:21.519 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56433d0dba56] 2024-07-26T11:27:21.519 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56433d0df637] 2024-07-26T11:27:21.519 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56433d0dff5e] 2024-07-26T11:27:21.519 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56433d0c303a] 2024-07-26T11:27:21.519 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f628e4c7407] 2024-07-26T11:27:21.519 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 18: (DispatchQueue::entry()+0x60f) [0x7f628e4c7cf9] 2024-07-26T11:27:21.520 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f628e5a45a1] 2024-07-26T11:27:21.520 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 20: (Thread::entry_wrapper()+0x43) [0x7f628e351cdd] 2024-07-26T11:27:21.520 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 21: (Thread::_entry_func(void*)+0xd) [0x7f628e351cf9] 2024-07-26T11:27:21.520 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 22: /lib64/libc.so.6(+0x89c02) [0x7f628dc45c02] 2024-07-26T11:27:21.520 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f628dccac40] 2024-07-26T11:27:21.520 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:21.520 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 2024-07-26T11:27:21.520 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 0> 2024-07-26T11:27:21.412+0000 7f6288823640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:21.520 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: in thread 7f6288823640 thread_name:ms_dispatch 2024-07-26T11:27:21.520 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 2024-07-26T11:27:21.520 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:21.520 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f628dbfa6f0] 2024-07-26T11:27:21.521 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f628dc4794c] 2024-07-26T11:27:21.521 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 3: raise() 2024-07-26T11:27:21.521 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 4: abort() 2024-07-26T11:27:21.521 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 5: /lib64/libc.so.6(+0x2871b) [0x7f628dbe471b] 2024-07-26T11:27:21.521 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 6: /lib64/libc.so.6(+0x37386) [0x7f628dbf3386] 2024-07-26T11:27:21.521 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56433d0da6b9] 2024-07-26T11:27:21.521 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56433d29880c] 2024-07-26T11:27:21.521 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56433d1d82ca] 2024-07-26T11:27:21.521 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x56433d1a8753] 2024-07-26T11:27:21.521 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x56433d1a9a1b] 2024-07-26T11:27:21.521 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x56433d18d752] 2024-07-26T11:27:21.522 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56433d0dba56] 2024-07-26T11:27:21.522 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x56433d0df637] 2024-07-26T11:27:21.522 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x56433d0dff5e] 2024-07-26T11:27:21.522 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56433d0c303a] 2024-07-26T11:27:21.522 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f628e4c7407] 2024-07-26T11:27:21.522 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 18: (DispatchQueue::entry()+0x60f) [0x7f628e4c7cf9] 2024-07-26T11:27:21.522 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f628e5a45a1] 2024-07-26T11:27:21.522 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 20: (Thread::entry_wrapper()+0x43) [0x7f628e351cdd] 2024-07-26T11:27:21.522 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 21: (Thread::_entry_func(void*)+0xd) [0x7f628e351cf9] 2024-07-26T11:27:21.522 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 22: /lib64/libc.so.6(+0x89c02) [0x7f628dc45c02] 2024-07-26T11:27:21.522 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f628dccac40] 2024-07-26T11:27:21.522 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:21.522 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[67894]: 2024-07-26T11:27:21.523 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 18: (DispatchQueue::entry()+0x60f) [0x7f3b4c70ccf9] 2024-07-26T11:27:21.523 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3b4c7e95a1] 2024-07-26T11:27:21.523 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 20: (Thread::entry_wrapper()+0x43) [0x7f3b4c596cdd] 2024-07-26T11:27:21.523 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3b4c596cf9] 2024-07-26T11:27:21.523 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3b4be8ac02] 2024-07-26T11:27:21.523 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3b4bf0fc40] 2024-07-26T11:27:21.523 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:21.523 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 2024-07-26T11:27:21.523 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 0> 2024-07-26T11:27:21.412+0000 7f3b46a68640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:21.523 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: in thread 7f3b46a68640 thread_name:ms_dispatch 2024-07-26T11:27:21.523 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 2024-07-26T11:27:21.523 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:21.524 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f3b4be3f6f0] 2024-07-26T11:27:21.524 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f3b4be8c94c] 2024-07-26T11:27:21.524 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 3: raise() 2024-07-26T11:27:21.524 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 4: abort() 2024-07-26T11:27:21.524 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 5: /lib64/libc.so.6(+0x2871b) [0x7f3b4be2971b] 2024-07-26T11:27:21.524 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 6: /lib64/libc.so.6(+0x37386) [0x7f3b4be38386] 2024-07-26T11:27:21.524 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55b4f1d706b9] 2024-07-26T11:27:21.524 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55b4f1f2e80c] 2024-07-26T11:27:21.524 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55b4f1e6e2ca] 2024-07-26T11:27:21.524 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55b4f1e3e753] 2024-07-26T11:27:21.524 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55b4f1e3fa1b] 2024-07-26T11:27:21.524 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55b4f1e23752] 2024-07-26T11:27:21.524 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55b4f1d71a56] 2024-07-26T11:27:21.524 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55b4f1d75637] 2024-07-26T11:27:21.525 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55b4f1d75f5e] 2024-07-26T11:27:21.525 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55b4f1d5903a] 2024-07-26T11:27:21.525 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f3b4c70c407] 2024-07-26T11:27:21.525 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 18: (DispatchQueue::entry()+0x60f) [0x7f3b4c70ccf9] 2024-07-26T11:27:21.525 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f3b4c7e95a1] 2024-07-26T11:27:21.525 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 20: (Thread::entry_wrapper()+0x43) [0x7f3b4c596cdd] 2024-07-26T11:27:21.525 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 21: (Thread::_entry_func(void*)+0xd) [0x7f3b4c596cf9] 2024-07-26T11:27:21.525 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 22: /lib64/libc.so.6(+0x89c02) [0x7f3b4be8ac02] 2024-07-26T11:27:21.525 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f3b4bf0fc40] 2024-07-26T11:27:21.525 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:21.525 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[68199]: 2024-07-26T11:27:21.525 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:21 smithi189 podman[72206]: 2024-07-26 11:27:21.503266137 +0000 UTC m=+0.036265961 container died eacc34abe1c23d2c10dff5a4b08da21c581ce34b7e51da638dcb414d04f80784 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:27:21.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:27:21.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: *** Caught signal (Aborted) ** 2024-07-26T11:27:21.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: in thread 7fa128a00640 thread_name:ms_dispatch 2024-07-26T11:27:21.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:21.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa12ec3e6f0] 2024-07-26T11:27:21.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa12ec8b94c] 2024-07-26T11:27:21.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 3: raise() 2024-07-26T11:27:21.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 4: abort() 2024-07-26T11:27:21.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa12ec2871b] 2024-07-26T11:27:21.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 6: /lib64/libc.so.6(+0x37386) [0x7fa12ec37386] 2024-07-26T11:27:21.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55bca237a6b9] 2024-07-26T11:27:21.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55bca253880c] 2024-07-26T11:27:21.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55bca24782ca] 2024-07-26T11:27:21.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55bca2448753] 2024-07-26T11:27:21.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55bca2449a1b] 2024-07-26T11:27:21.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55bca242d752] 2024-07-26T11:27:21.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55bca237ba56] 2024-07-26T11:27:21.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55bca237f637] 2024-07-26T11:27:21.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55bca237ff5e] 2024-07-26T11:27:21.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55bca236303a] 2024-07-26T11:27:21.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa12f79b407] 2024-07-26T11:27:21.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 18: (DispatchQueue::entry()+0x60f) [0x7fa12f79bcf9] 2024-07-26T11:27:21.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa12f8785a1] 2024-07-26T11:27:21.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 20: (Thread::entry_wrapper()+0x43) [0x7fa12f625cdd] 2024-07-26T11:27:21.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa12f625cf9] 2024-07-26T11:27:21.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa12ec89c02] 2024-07-26T11:27:21.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa12ed0ec40] 2024-07-26T11:27:21.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 2024-07-26T11:27:21.547+0000 7fa128a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:21.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: in thread 7fa128a00640 thread_name:ms_dispatch 2024-07-26T11:27:21.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 2024-07-26T11:27:21.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:21.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa12ec3e6f0] 2024-07-26T11:27:21.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa12ec8b94c] 2024-07-26T11:27:21.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 3: raise() 2024-07-26T11:27:21.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 4: abort() 2024-07-26T11:27:21.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa12ec2871b] 2024-07-26T11:27:21.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 6: /lib64/libc.so.6(+0x37386) [0x7fa12ec37386] 2024-07-26T11:27:21.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55bca237a6b9] 2024-07-26T11:27:21.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55bca253880c] 2024-07-26T11:27:21.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55bca24782ca] 2024-07-26T11:27:21.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55bca2448753] 2024-07-26T11:27:21.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55bca2449a1b] 2024-07-26T11:27:21.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55bca242d752] 2024-07-26T11:27:21.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55bca237ba56] 2024-07-26T11:27:21.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55bca237f637] 2024-07-26T11:27:21.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55bca237ff5e] 2024-07-26T11:27:21.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55bca236303a] 2024-07-26T11:27:21.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa12f79b407] 2024-07-26T11:27:21.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 18: (DispatchQueue::entry()+0x60f) [0x7fa12f79bcf9] 2024-07-26T11:27:21.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa12f8785a1] 2024-07-26T11:27:21.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 20: (Thread::entry_wrapper()+0x43) [0x7fa12f625cdd] 2024-07-26T11:27:21.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa12f625cf9] 2024-07-26T11:27:21.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa12ec89c02] 2024-07-26T11:27:21.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa12ed0ec40] 2024-07-26T11:27:21.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:21.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 2024-07-26T11:27:21.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 0> 2024-07-26T11:27:21.547+0000 7fa128a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:21.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: in thread 7fa128a00640 thread_name:ms_dispatch 2024-07-26T11:27:21.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 2024-07-26T11:27:21.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:21.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa12ec3e6f0] 2024-07-26T11:27:21.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa12ec8b94c] 2024-07-26T11:27:21.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 3: raise() 2024-07-26T11:27:21.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 4: abort() 2024-07-26T11:27:21.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa12ec2871b] 2024-07-26T11:27:21.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 6: /lib64/libc.so.6(+0x37386) [0x7fa12ec37386] 2024-07-26T11:27:21.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55bca237a6b9] 2024-07-26T11:27:21.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55bca253880c] 2024-07-26T11:27:21.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55bca24782ca] 2024-07-26T11:27:21.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55bca2448753] 2024-07-26T11:27:21.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55bca2449a1b] 2024-07-26T11:27:21.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55bca242d752] 2024-07-26T11:27:21.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55bca237ba56] 2024-07-26T11:27:21.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55bca237f637] 2024-07-26T11:27:21.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55bca237ff5e] 2024-07-26T11:27:21.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55bca236303a] 2024-07-26T11:27:21.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa12f79b407] 2024-07-26T11:27:21.687 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 18: (DispatchQueue::entry()+0x60f) [0x7fa12f79bcf9] 2024-07-26T11:27:21.687 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa12f8785a1] 2024-07-26T11:27:21.687 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 20: (Thread::entry_wrapper()+0x43) [0x7fa12f625cdd] 2024-07-26T11:27:21.687 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa12f625cf9] 2024-07-26T11:27:21.687 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa12ec89c02] 2024-07-26T11:27:21.687 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa12ed0ec40] 2024-07-26T11:27:21.687 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:21.687 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 2024-07-26T11:27:21.687 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 0> 2024-07-26T11:27:21.547+0000 7fa128a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:21.688 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: in thread 7fa128a00640 thread_name:ms_dispatch 2024-07-26T11:27:21.688 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 2024-07-26T11:27:21.688 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:21.688 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa12ec3e6f0] 2024-07-26T11:27:21.688 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa12ec8b94c] 2024-07-26T11:27:21.688 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 3: raise() 2024-07-26T11:27:21.688 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 4: abort() 2024-07-26T11:27:21.688 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa12ec2871b] 2024-07-26T11:27:21.688 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 6: /lib64/libc.so.6(+0x37386) [0x7fa12ec37386] 2024-07-26T11:27:21.688 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55bca237a6b9] 2024-07-26T11:27:21.688 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55bca253880c] 2024-07-26T11:27:21.688 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55bca24782ca] 2024-07-26T11:27:21.689 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55bca2448753] 2024-07-26T11:27:21.689 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55bca2449a1b] 2024-07-26T11:27:21.689 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55bca242d752] 2024-07-26T11:27:21.689 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55bca237ba56] 2024-07-26T11:27:21.689 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55bca237f637] 2024-07-26T11:27:21.689 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55bca237ff5e] 2024-07-26T11:27:21.689 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55bca236303a] 2024-07-26T11:27:21.689 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa12f79b407] 2024-07-26T11:27:21.689 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 18: (DispatchQueue::entry()+0x60f) [0x7fa12f79bcf9] 2024-07-26T11:27:21.689 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa12f8785a1] 2024-07-26T11:27:21.689 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 20: (Thread::entry_wrapper()+0x43) [0x7fa12f625cdd] 2024-07-26T11:27:21.690 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa12f625cf9] 2024-07-26T11:27:21.690 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa12ec89c02] 2024-07-26T11:27:21.690 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa12ed0ec40] 2024-07-26T11:27:21.690 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:21.690 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[78601]: 2024-07-26T11:27:21.775 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:27:21.775 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: *** Caught signal (Aborted) ** 2024-07-26T11:27:21.775 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: in thread 7fb9d5a80640 thread_name:ms_dispatch 2024-07-26T11:27:21.775 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:21.775 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb9dae576f0] 2024-07-26T11:27:21.775 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb9daea494c] 2024-07-26T11:27:21.776 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 3: raise() 2024-07-26T11:27:21.776 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 4: abort() 2024-07-26T11:27:21.776 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb9dae4171b] 2024-07-26T11:27:21.776 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 6: /lib64/libc.so.6(+0x37386) [0x7fb9dae50386] 2024-07-26T11:27:21.776 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55e9e2d006b9] 2024-07-26T11:27:21.776 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55e9e2ebe80c] 2024-07-26T11:27:21.776 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55e9e2dfe2ca] 2024-07-26T11:27:21.776 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55e9e2dce753] 2024-07-26T11:27:21.776 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55e9e2dcfa1b] 2024-07-26T11:27:21.776 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55e9e2db3752] 2024-07-26T11:27:21.776 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55e9e2d01a56] 2024-07-26T11:27:21.776 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55e9e2d05637] 2024-07-26T11:27:21.776 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55e9e2d05f5e] 2024-07-26T11:27:21.777 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55e9e2ce903a] 2024-07-26T11:27:21.777 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb9db724407] 2024-07-26T11:27:21.777 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 18: (DispatchQueue::entry()+0x60f) [0x7fb9db724cf9] 2024-07-26T11:27:21.777 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb9db8015a1] 2024-07-26T11:27:21.777 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 20: (Thread::entry_wrapper()+0x43) [0x7fb9db5aecdd] 2024-07-26T11:27:21.778 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb9db5aecf9] 2024-07-26T11:27:21.778 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb9daea2c02] 2024-07-26T11:27:21.778 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb9daf27c40] 2024-07-26T11:27:21.778 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 2024-07-26T11:27:21.407+0000 7fb9d5a80640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:21.778 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: in thread 7fb9d5a80640 thread_name:ms_dispatch 2024-07-26T11:27:21.778 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 2024-07-26T11:27:21.778 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:21.778 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb9dae576f0] 2024-07-26T11:27:21.778 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb9daea494c] 2024-07-26T11:27:21.778 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 3: raise() 2024-07-26T11:27:21.778 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 4: abort() 2024-07-26T11:27:21.778 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb9dae4171b] 2024-07-26T11:27:21.779 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 6: /lib64/libc.so.6(+0x37386) [0x7fb9dae50386] 2024-07-26T11:27:21.779 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55e9e2d006b9] 2024-07-26T11:27:21.779 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55e9e2ebe80c] 2024-07-26T11:27:21.779 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55e9e2dfe2ca] 2024-07-26T11:27:21.779 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55e9e2dce753] 2024-07-26T11:27:21.779 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55e9e2dcfa1b] 2024-07-26T11:27:21.779 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55e9e2db3752] 2024-07-26T11:27:21.779 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55e9e2d01a56] 2024-07-26T11:27:21.779 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55e9e2d05637] 2024-07-26T11:27:21.779 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55e9e2d05f5e] 2024-07-26T11:27:21.779 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55e9e2ce903a] 2024-07-26T11:27:21.779 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb9db724407] 2024-07-26T11:27:21.779 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 18: (DispatchQueue::entry()+0x60f) [0x7fb9db724cf9] 2024-07-26T11:27:21.779 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb9db8015a1] 2024-07-26T11:27:21.780 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 20: (Thread::entry_wrapper()+0x43) [0x7fb9db5aecdd] 2024-07-26T11:27:21.780 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb9db5aecf9] 2024-07-26T11:27:21.780 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb9daea2c02] 2024-07-26T11:27:21.780 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb9daf27c40] 2024-07-26T11:27:21.780 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:21.780 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 2024-07-26T11:27:21.780 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 0> 2024-07-26T11:27:21.407+0000 7fb9d5a80640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:21.780 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: in thread 7fb9d5a80640 thread_name:ms_dispatch 2024-07-26T11:27:21.780 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 2024-07-26T11:27:21.780 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:21.780 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb9dae576f0] 2024-07-26T11:27:21.781 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb9daea494c] 2024-07-26T11:27:21.781 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 3: raise() 2024-07-26T11:27:21.781 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 4: abort() 2024-07-26T11:27:21.781 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb9dae4171b] 2024-07-26T11:27:21.781 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 6: /lib64/libc.so.6(+0x37386) [0x7fb9dae50386] 2024-07-26T11:27:21.781 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55e9e2d006b9] 2024-07-26T11:27:21.781 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55e9e2ebe80c] 2024-07-26T11:27:21.781 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55e9e2dfe2ca] 2024-07-26T11:27:21.781 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55e9e2dce753] 2024-07-26T11:27:21.781 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55e9e2dcfa1b] 2024-07-26T11:27:21.781 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55e9e2db3752] 2024-07-26T11:27:21.781 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55e9e2d01a56] 2024-07-26T11:27:21.781 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55e9e2d05637] 2024-07-26T11:27:21.781 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55e9e2d05f5e] 2024-07-26T11:27:21.782 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55e9e2ce903a] 2024-07-26T11:27:21.782 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb9db724407] 2024-07-26T11:27:21.782 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 18: (DispatchQueue::entry()+0x60f) [0x7fb9db724cf9] 2024-07-26T11:27:21.782 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb9db8015a1] 2024-07-26T11:27:21.782 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 20: (Thread::entry_wrapper()+0x43) [0x7fb9db5aecdd] 2024-07-26T11:27:21.782 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb9db5aecf9] 2024-07-26T11:27:21.782 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb9daea2c02] 2024-07-26T11:27:21.782 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb9daf27c40] 2024-07-26T11:27:21.782 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:21.782 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 2024-07-26T11:27:21.782 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 0> 2024-07-26T11:27:21.407+0000 7fb9d5a80640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:21.782 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: in thread 7fb9d5a80640 thread_name:ms_dispatch 2024-07-26T11:27:21.782 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 2024-07-26T11:27:21.783 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:21.783 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb9dae576f0] 2024-07-26T11:27:21.783 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb9daea494c] 2024-07-26T11:27:21.783 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 3: raise() 2024-07-26T11:27:21.783 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 4: abort() 2024-07-26T11:27:21.783 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb9dae4171b] 2024-07-26T11:27:21.783 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 6: /lib64/libc.so.6(+0x37386) [0x7fb9dae50386] 2024-07-26T11:27:21.783 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55e9e2d006b9] 2024-07-26T11:27:21.783 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55e9e2ebe80c] 2024-07-26T11:27:21.783 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55e9e2dfe2ca] 2024-07-26T11:27:21.783 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55e9e2dce753] 2024-07-26T11:27:21.783 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55e9e2dcfa1b] 2024-07-26T11:27:21.783 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55e9e2db3752] 2024-07-26T11:27:21.783 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55e9e2d01a56] 2024-07-26T11:27:21.784 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55e9e2d05637] 2024-07-26T11:27:21.784 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55e9e2d05f5e] 2024-07-26T11:27:21.784 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55e9e2ce903a] 2024-07-26T11:27:21.784 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb9db724407] 2024-07-26T11:27:21.784 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 18: (DispatchQueue::entry()+0x60f) [0x7fb9db724cf9] 2024-07-26T11:27:21.784 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb9db8015a1] 2024-07-26T11:27:21.784 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 20: (Thread::entry_wrapper()+0x43) [0x7fb9db5aecdd] 2024-07-26T11:27:21.784 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb9db5aecf9] 2024-07-26T11:27:21.784 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb9daea2c02] 2024-07-26T11:27:21.784 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb9daf27c40] 2024-07-26T11:27:21.784 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:21.784 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[69016]: 2024-07-26T11:27:21.784 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:21 smithi193 podman[73288]: 2024-07-26 11:27:21.498898632 +0000 UTC m=+0.027942327 container died ac062bd4a9db69de49819bfc9e105117971087922adad2579bc8244d99c59584 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=) 2024-07-26T11:27:21.963 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:21 smithi134 podman[111136]: 2024-07-26 11:27:21.886555382 +0000 UTC m=+0.123281129 container died f5e056356436200c7fe3b16c72eae01e5184a6ac07bd3d2a4fad7b3f16d8ca63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True) 2024-07-26T11:27:22.266 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:27:22.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.261+0000 7f9b4b200640 1 -- 172.21.15.134:0/2582671638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b4c10f330 msgr2=0x7f9b4c10f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:22.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.261+0000 7f9b4b200640 1 --2- 172.21.15.134:0/2582671638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b4c10f330 0x7f9b4c10f740 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009900 tx=0x7f9b3c02a3f0 comp rx=0 tx=0).stop 2024-07-26T11:27:22.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.261+0000 7f9b4b200640 1 -- 172.21.15.134:0/2582671638 shutdown_connections 2024-07-26T11:27:22.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.261+0000 7f9b4b200640 1 --2- 172.21.15.134:0/2582671638 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b4c10f330 0x7f9b4c10f740 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:22.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.261+0000 7f9b4b200640 1 --2- 172.21.15.134:0/2582671638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b4c10fc80 0x7f9b4c1164b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:22.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.261+0000 7f9b4b200640 1 --2- 172.21.15.134:0/2582671638 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b4c1099f0 0x7f9b4c109e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:22.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.261+0000 7f9b4b200640 1 -- 172.21.15.134:0/2582671638 >> 172.21.15.134:0/2582671638 conn(0x7f9b4c078ff0 msgr2=0x7f9b4c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:22.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.262+0000 7f9b4b200640 1 -- 172.21.15.134:0/2582671638 shutdown_connections 2024-07-26T11:27:22.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.262+0000 7f9b4b200640 1 -- 172.21.15.134:0/2582671638 wait complete. 2024-07-26T11:27:22.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.264+0000 7f9b4b200640 1 Processor -- start 2024-07-26T11:27:22.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.264+0000 7f9b4b200640 1 -- start start 2024-07-26T11:27:22.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.265+0000 7f9b4b200640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b4c1099f0 0x7f9b4c074920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:22.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.265+0000 7f9b4b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b4c10f330 0x7f9b4c06dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:22.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.266+0000 7f9b4b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b4c10fc80 0x7f9b4c06e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:22.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.266+0000 7f9b4b200640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c1199b0 con 0x7f9b4c1099f0 2024-07-26T11:27:22.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.266+0000 7f9b4b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c119830 con 0x7f9b4c10f330 2024-07-26T11:27:22.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.266+0000 7f9b4b200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4c119b30 con 0x7f9b4c10fc80 2024-07-26T11:27:22.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.266+0000 7f9b49e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b4c1099f0 0x7f9b4c074920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:22.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.266+0000 7f9b49400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b4c10f330 0x7f9b4c06dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:22.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.266+0000 7f9b49400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b4c10f330 0x7f9b4c06dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:54676/0 (socket says 172.21.15.134:54676) 2024-07-26T11:27:22.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.266+0000 7f9b49400640 1 -- 172.21.15.134:0/4006618154 learned_addr learned my addr 172.21.15.134:0/4006618154 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:22.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.266+0000 7f9b4a800640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b4c10fc80 0x7f9b4c06e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:22.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.267+0000 7f9b49400640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b4c10fc80 msgr2=0x7f9b4c06e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:22.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.267+0000 7f9b49400640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9b4c10fc80 0x7f9b4c06e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:22.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.267+0000 7f9b49400640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b4c1099f0 msgr2=0x7f9b4c074920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:22.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.267+0000 7f9b49400640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f9b4c1099f0 0x7f9b4c074920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:22.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.267+0000 7f9b49400640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b4c06ea20 con 0x7f9b4c10f330 2024-07-26T11:27:22.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.268+0000 7f9b49400640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b4c10f330 0x7f9b4c06dbf0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c031440 tx=0x7f9b3c003c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:22.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.269+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c03f070 con 0x7f9b4c10f330 2024-07-26T11:27:22.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.269+0000 7f9b4b200640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b4c073650 con 0x7f9b4c10f330 2024-07-26T11:27:22.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.269+0000 7f9b4b200640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b4c073b80 con 0x7f9b4c10f330 2024-07-26T11:27:22.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.270+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b3c031b50 con 0x7f9b4c10f330 2024-07-26T11:27:22.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.270+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c035030 con 0x7f9b4c10f330 2024-07-26T11:27:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:27:22.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.276+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 151191+0+0 (secure 0 0 0) 0x7f9b3c042070 con 0x7f9b4c10f330 2024-07-26T11:27:22.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.278+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300b6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:22.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.278+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f9b3c104920 con 0x7f9b4c10f330 2024-07-26T11:27:22.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.280+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:22.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.282+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300b6300 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009990 tx=0x7f9b38009110 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:22.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.304+0000 7f9b57c86740 1 Processor -- start 2024-07-26T11:27:22.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.307+0000 7f9b57c86740 1 -- start start 2024-07-26T11:27:22.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.309+0000 7f9b57c86740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c65e60f350 0x55c65e9b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:22.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.310+0000 7f9b49e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c65e60f350 0x55c65e9b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:22.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.310+0000 7f9b49e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c65e60f350 0x55c65e9b3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.134:59688/0 (socket says 172.21.15.134:59688) 2024-07-26T11:27:22.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.310+0000 7f9b49e00640 1 -- 172.21.15.134:0/1990778282 learned_addr learned my addr 172.21.15.134:0/1990778282 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:22.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.311+0000 7f9b57c86740 1 --2- 172.21.15.134:0/1990778282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c65e5e6490 0x55c65e9b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:22.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.311+0000 7f9b57c86740 1 --2- 172.21.15.134:0/1990778282 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c65e5ee670 0x55c65e9b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:22.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.311+0000 7f9b57c86740 1 -- 172.21.15.134:0/1990778282 --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c65e92abf0 con 0x55c65e5ee670 2024-07-26T11:27:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.311+0000 7f9b57c86740 1 -- 172.21.15.134:0/1990778282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c65e91a0f0 con 0x55c65e5e6490 2024-07-26T11:27:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.311+0000 7f9b57c86740 1 -- 172.21.15.134:0/1990778282 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c65e91e130 con 0x55c65e60f350 2024-07-26T11:27:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.312+0000 7f9b4a800640 1 --2- 172.21.15.134:0/1990778282 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c65e5ee670 0x55c65e9b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.311+0000 7f9b49400640 1 --2- 172.21.15.134:0/1990778282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c65e5e6490 0x55c65e9b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.313+0000 7f9b49e00640 1 -- 172.21.15.134:0/1990778282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c65e5e6490 msgr2=0x55c65e9b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.313+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1990778282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c65e5e6490 0x55c65e9b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.313+0000 7f9b49e00640 1 -- 172.21.15.134:0/1990778282 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c65e5ee670 msgr2=0x55c65e9b49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:22.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.313+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1990778282 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c65e5ee670 0x55c65e9b49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:22.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.314+0000 7f9b49e00640 1 -- 172.21.15.134:0/1990778282 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c65e67a930 con 0x55c65e60f350 2024-07-26T11:27:22.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.315+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1990778282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c65e60f350 0x55c65e9b3f60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009fd0 tx=0x7f9b38011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a41c1c46529ab3c2 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:22.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.315+0000 7f9b37400640 1 -- 172.21.15.134:0/1990778282 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3801b040 con 0x55c65e60f350 2024-07-26T11:27:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.315+0000 7f9b37400640 1 -- 172.21.15.134:0/1990778282 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9b38024070 con 0x55c65e60f350 2024-07-26T11:27:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.315+0000 7f9b37400640 1 -- 172.21.15.134:0/1990778282 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3801f3f0 con 0x55c65e60f350 2024-07-26T11:27:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.317+0000 7f9b57c86740 1 -- 172.21.15.134:0/1990778282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c65e60f350 msgr2=0x55c65e9b3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.317+0000 7f9b57c86740 1 --2- 172.21.15.134:0/1990778282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c65e60f350 0x55c65e9b3f60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009fd0 tx=0x7f9b38011000 comp rx=0 tx=0).stop 2024-07-26T11:27:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.318+0000 7f9b57c86740 1 -- 172.21.15.134:0/1990778282 shutdown_connections 2024-07-26T11:27:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.318+0000 7f9b57c86740 1 --2- 172.21.15.134:0/1990778282 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c65e5ee670 0x55c65e9b49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.318+0000 7f9b57c86740 1 --2- 172.21.15.134:0/1990778282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c65e5e6490 0x55c65e9b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.318+0000 7f9b57c86740 1 --2- 172.21.15.134:0/1990778282 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c65e60f350 0x55c65e9b3f60 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.318+0000 7f9b57c86740 1 -- 172.21.15.134:0/1990778282 >> 172.21.15.134:0/1990778282 conn(0x55c65e603740 msgr2=0x55c65e9b1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.319+0000 7f9b57c86740 1 -- 172.21.15.134:0/1990778282 shutdown_connections 2024-07-26T11:27:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.319+0000 7f9b57c86740 1 -- 172.21.15.134:0/1990778282 wait complete. 2024-07-26T11:27:22.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.320+0000 7f9b57c86740 1 -- start start 2024-07-26T11:27:22.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.320+0000 7f9b57c86740 1 Processor -- start 2024-07-26T11:27:22.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.321+0000 7f9b57c86740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c65e5e6490 0x55c65e9bf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:22.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.321+0000 7f9b57c86740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c65e5ee670 0x55c65e9bfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:22.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.321+0000 7f9b57c86740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c65e60f350 0x55c65e9cba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:22.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.322+0000 7f9b49e00640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c65e5e6490 0x55c65e9bf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:22.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.322+0000 7f9b57c86740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55c65e77d890 con 0x55c65e5e6490 2024-07-26T11:27:22.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.322+0000 7f9b57c86740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c65e8b6330 con 0x55c65e60f350 2024-07-26T11:27:22.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.322+0000 7f9b57c86740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55c65e9c2740 con 0x55c65e5ee670 2024-07-26T11:27:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.322+0000 7f9b49400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c65e5ee670 0x55c65e9bfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.322+0000 7f9b4a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c65e60f350 0x55c65e9cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.322+0000 7f9b4a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c65e60f350 0x55c65e9cba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:54702/0 (socket says 172.21.15.134:54702) 2024-07-26T11:27:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.322+0000 7f9b4a800640 1 -- 172.21.15.134:0/1872420438 learned_addr learned my addr 172.21.15.134:0/1872420438 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.323+0000 7f9b49400640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c65e60f350 msgr2=0x55c65e9cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.323+0000 7f9b49400640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c65e60f350 0x55c65e9cba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.323+0000 7f9b49400640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c65e5e6490 msgr2=0x55c65e9bf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.323+0000 7f9b57c86740 10 client.0 _ll_register_callbacks cb 0x55c65e8e2010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:27:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.325+0000 7f9b49400640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c65e5e6490 0x55c65e9bf680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.325+0000 7f9b49400640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c65e9c0410 con 0x55c65e5ee670 2024-07-26T11:27:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.325+0000 7f9b57c86740 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b38024070 con 0x55c65e5ee670 2024-07-26T11:27:22.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.325+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55c65e5e6490 0x55c65e9bf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-07-26T11:27:22.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.327+0000 7f9b49400640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c65e5ee670 0x55c65e9bfbc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c101c40 tx=0x7f9b3c0ca930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d828d14d5eee6fe server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:22.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.327+0000 7f9b2be00640 10 client.27043 ms_handle_connect on v2:172.21.15.193:3300/0 2024-07-26T11:27:22.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.327+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c0cf700 con 0x55c65e5ee670 2024-07-26T11:27:22.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.327+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) ==== 9599+0+0 (secure 0 0 0) 0x7f9b3c0c7390 con 0x55c65e5ee670 2024-07-26T11:27:22.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.328+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f9b3c0d41b0 con 0x55c65e5ee670 2024-07-26T11:27:22.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.328+0000 7f9b57c86740 10 client.27043 fetch_fsmap learned FSMap version 36 2024-07-26T11:27:22.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.328+0000 7f9b57c86740 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=36}) -- 0x55c65e9ce530 con 0x55c65e5ee670 2024-07-26T11:27:22.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.328+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 36) ==== 8398+0+0 (secure 0 0 0) 0x7f9b3c0d3d20 con 0x55c65e5ee670 2024-07-26T11:27:22.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.328+0000 7f9b57c86740 10 client.27043 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T11:27:22.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.328+0000 7f9b57c86740 10 client.27043 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.a 2024-07-26T11:27:22.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.328+0000 7f9b57c86740 20 client.27043 populate_metadata read hostname 'smithi134' 2024-07-26T11:27:22.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.328+0000 7f9b57c86740 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:22.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.328+0000 7f9b57c86740 4 client.27043 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:27:22.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.328+0000 7f9b57c86740 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c65e92cb40 con 0x55c65e5cdcd0 2024-07-26T11:27:22.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.329+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:22.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.330+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T11:27:22.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.330+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:27:22.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.531+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:22.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.531+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T11:27:22.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.531+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:22.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.932+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:22.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.932+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:22.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:22.932+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T11:27:22.940 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:22 smithi189 podman[72207]: 2024-07-26 11:27:22.636128479 +0000 UTC m=+1.165093653 container died ce38a5f212ef6631add96e73afc3d155f2f6c6adf7582bd0f2c5723a282783af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:27:22.940 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:22 smithi189 podman[72206]: 2024-07-26 11:27:22.634928554 +0000 UTC m=+1.167928382 container remove eacc34abe1c23d2c10dff5a4b08da21c581ce34b7e51da638dcb414d04f80784 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True) 2024-07-26T11:27:22.940 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:22 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:27:23.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:22 smithi189 ceph-mon[36790]: pgmap v834: 129 pgs: 8 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 103 active+clean; 53 GiB data, 116 GiB used, 957 GiB / 1.0 TiB avail; 95 KiB/s rd, 142 MiB/s wr, 886 op/s 2024-07-26T11:27:23.234 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:22 smithi134 ceph-mon[31966]: pgmap v834: 129 pgs: 8 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 103 active+clean; 53 GiB data, 116 GiB used, 957 GiB / 1.0 TiB avail; 95 KiB/s rd, 142 MiB/s wr, 886 op/s 2024-07-26T11:27:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:22 smithi193 ceph-mon[37473]: pgmap v834: 129 pgs: 8 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 103 active+clean; 53 GiB data, 116 GiB used, 957 GiB / 1.0 TiB avail; 95 KiB/s rd, 142 MiB/s wr, 886 op/s 2024-07-26T11:27:23.608 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:23 smithi189 podman[72207]: 2024-07-26 11:27:23.281878508 +0000 UTC m=+1.810843671 container remove ce38a5f212ef6631add96e73afc3d155f2f6c6adf7582bd0f2c5723a282783af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-07-26T11:27:23.608 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:23 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:27:23.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:23.733+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:23.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:23.733+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:23.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:23.733+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T11:27:23.944 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:23 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-07-26T11:27:23.944 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:23 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.h.service: Consumed 1.983s CPU time. 2024-07-26T11:27:23.944 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:23 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-07-26T11:27:23.944 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:23 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.b.service: Consumed 2.083s CPU time. 2024-07-26T11:27:24.058 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:23 smithi134 podman[111136]: 2024-07-26 11:27:23.803947528 +0000 UTC m=+2.040673285 container remove f5e056356436200c7fe3b16c72eae01e5184a6ac07bd3d2a4fad7b3f16d8ca63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-07-26T11:27:24.058 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:23 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:27:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:27:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:27:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:27:24.504 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:27:24.504 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:27:24.504 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:27:24.504 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:24 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T11:27:24.505 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:24 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Consumed 26.787s CPU time. 2024-07-26T11:27:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:27:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:27:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:27:25.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:25.335+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:25.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:25.335+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:25.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:25.335+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-07-26T11:27:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:25 smithi193 ceph-mon[37473]: pgmap v835: 129 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 112 active+clean; 53 GiB data, 115 GiB used, 957 GiB / 1.0 TiB avail; 32 KiB/s rd, 152 MiB/s wr, 950 op/s 2024-07-26T11:27:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:25 smithi134 ceph-mon[31966]: pgmap v835: 129 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 112 active+clean; 53 GiB data, 115 GiB used, 957 GiB / 1.0 TiB avail; 32 KiB/s rd, 152 MiB/s wr, 950 op/s 2024-07-26T11:27:25.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:25 smithi189 ceph-mon[36790]: pgmap v835: 129 pgs: 6 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 112 active+clean; 53 GiB data, 115 GiB used, 957 GiB / 1.0 TiB avail; 32 KiB/s rd, 152 MiB/s wr, 950 op/s 2024-07-26T11:27:25.911 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:25 smithi193 podman[73288]: 2024-07-26 11:27:25.629967744 +0000 UTC m=+4.159011427 container remove ac062bd4a9db69de49819bfc9e105117971087922adad2579bc8244d99c59584 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:27:25.911 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:25 smithi193 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:27:26.274 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:25 smithi193 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-07-26T11:27:26.275 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:25 smithi193 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.i.service: Consumed 2.033s CPU time. 2024-07-26T11:27:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:26 smithi134 ceph-mon[31966]: pgmap v836: 129 pgs: 3 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 121 active+clean; 52 GiB data, 114 GiB used, 959 GiB / 1.0 TiB avail; 27 KiB/s rd, 154 MiB/s wr, 1.02k op/s 2024-07-26T11:27:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:26 smithi193 ceph-mon[37473]: pgmap v836: 129 pgs: 3 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 121 active+clean; 52 GiB data, 114 GiB used, 959 GiB / 1.0 TiB avail; 27 KiB/s rd, 154 MiB/s wr, 1.02k op/s 2024-07-26T11:27:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:26 smithi189 ceph-mon[36790]: pgmap v836: 129 pgs: 3 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 121 active+clean; 52 GiB data, 114 GiB used, 959 GiB / 1.0 TiB avail; 27 KiB/s rd, 154 MiB/s wr, 1.02k op/s 2024-07-26T11:27:28.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:28.539+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:28.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:28.539+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-07-26T11:27:28.541 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:28.539+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:28 smithi193 ceph-mon[37473]: pgmap v837: 129 pgs: 129 active+clean; 51 GiB data, 113 GiB used, 960 GiB / 1.0 TiB avail; 28 KiB/s rd, 129 MiB/s wr, 903 op/s 2024-07-26T11:27:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:28 smithi134 ceph-mon[31966]: pgmap v837: 129 pgs: 129 active+clean; 51 GiB data, 113 GiB used, 960 GiB / 1.0 TiB avail; 28 KiB/s rd, 129 MiB/s wr, 903 op/s 2024-07-26T11:27:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:28 smithi189 ceph-mon[36790]: pgmap v837: 129 pgs: 129 active+clean; 51 GiB data, 113 GiB used, 960 GiB / 1.0 TiB avail; 28 KiB/s rd, 129 MiB/s wr, 903 op/s 2024-07-26T11:27:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:30 smithi134 ceph-mon[31966]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T11:27:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:30 smithi189 ceph-mon[36790]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T11:27:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:30 smithi193 ceph-mon[37473]: osdmap e103: 12 total, 12 up, 12 in 2024-07-26T11:27:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:31 smithi134 ceph-mon[31966]: pgmap v839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 19 KiB/s rd, 89 MiB/s wr, 701 op/s 2024-07-26T11:27:31.426 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:31 smithi193 ceph-mon[37473]: pgmap v839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 19 KiB/s rd, 89 MiB/s wr, 701 op/s 2024-07-26T11:27:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:31 smithi189 ceph-mon[36790]: pgmap v839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 19 KiB/s rd, 89 MiB/s wr, 701 op/s 2024-07-26T11:27:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.829+0000 7f21cd000640 1 -- 172.21.15.134:0/349642384 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21c8105550 msgr2=0x7f21c8105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.829+0000 7f21cd000640 1 --2- 172.21.15.134:0/349642384 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21c8105550 0x7f21c8105960 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f21b8009900 tx=0x7f21b802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:27:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.830+0000 7f21cd000640 1 -- 172.21.15.134:0/349642384 shutdown_connections 2024-07-26T11:27:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.830+0000 7f21cd000640 1 --2- 172.21.15.134:0/349642384 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21c810ae80 0x7f21c810b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.830+0000 7f21cd000640 1 --2- 172.21.15.134:0/349642384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21c810b7d0 0x7f21c8112000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.830+0000 7f21cd000640 1 --2- 172.21.15.134:0/349642384 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21c8105550 0x7f21c8105960 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.830+0000 7f21cd000640 1 -- 172.21.15.134:0/349642384 >> 172.21.15.134:0/349642384 conn(0x7f21c8100f90 msgr2=0x7f21c81033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:32.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.830+0000 7f21cd000640 1 -- 172.21.15.134:0/349642384 shutdown_connections 2024-07-26T11:27:32.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.830+0000 7f21cd000640 1 -- 172.21.15.134:0/349642384 wait complete. 2024-07-26T11:27:32.835 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.831+0000 7f21cd000640 1 Processor -- start 2024-07-26T11:27:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.831+0000 7f21cd000640 1 -- start start 2024-07-26T11:27:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.831+0000 7f21cd000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21c8105550 0x7f21c819bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.831+0000 7f21cd000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21c810ae80 0x7f21c819c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.831+0000 7f21cd000640 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21c810b7d0 0x7f21c81a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.831+0000 7f21cd000640 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x7f21c81154a0 con 0x7f21c810b7d0 2024-07-26T11:27:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.831+0000 7f21cd000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f21c8115320 con 0x7f21c8105550 2024-07-26T11:27:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.831+0000 7f21cd000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x7f21c8115620 con 0x7f21c810ae80 2024-07-26T11:27:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.832+0000 7f21c7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21c8105550 0x7f21c819bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.832+0000 7f21c7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21c8105550 0x7f21c819bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43776/0 (socket says 172.21.15.134:43776) 2024-07-26T11:27:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.832+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 learned_addr learned my addr 172.21.15.134:0/939165295 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.832+0000 7f21bea00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21c810ae80 0x7f21c819c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.832+0000 7f21c7e00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21c810b7d0 0x7f21c81a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.832+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21c810ae80 msgr2=0x7f21c819c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.832+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21c810ae80 0x7f21c819c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.832+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21c810b7d0 msgr2=0x7f21c81a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.832+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x7f21c810b7d0 0x7f21c81a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.832+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21c81a1e30 con 0x7f21c8105550 2024-07-26T11:27:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.832+0000 7f21bea00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21c810ae80 0x7f21c819c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-07-26T11:27:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.832+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21c8105550 0x7f21c819bf00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f21b8005940 tx=0x7f21b80040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.833+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b803f030 con 0x7f21c8105550 2024-07-26T11:27:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.833+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21b80044c0 con 0x7f21c8105550 2024-07-26T11:27:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.833+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b8035070 con 0x7f21c8105550 2024-07-26T11:27:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.833+0000 7f21cd000640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21c81a20c0 con 0x7f21c8105550 2024-07-26T11:27:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.833+0000 7f21cd000640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21c81a88b0 con 0x7f21c8105550 2024-07-26T11:27:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.836+0000 7f21d5632740 1 Processor -- start 2024-07-26T11:27:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.836+0000 7f21d5632740 1 -- start start 2024-07-26T11:27:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.837+0000 7f21d5632740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a08e2fa1b0 0x55a08e69dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.837+0000 7f21d5632740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a08e2d12f0 0x55a08e69e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.837+0000 7f21d5632740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a08e2d94d0 0x55a08e69e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.837+0000 7f21d5632740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a08e612270 con 0x55a08e2d94d0 2024-07-26T11:27:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.837+0000 7f21d5632740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a08e601770 con 0x55a08e2fa1b0 2024-07-26T11:27:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.837+0000 7f21d5632740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a08e6057b0 con 0x55a08e2d12f0 2024-07-26T11:27:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.837+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 27) ==== 151191+0+0 (secure 0 0 0) 0x7f21b8042030 con 0x7f21c8105550 2024-07-26T11:27:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.837+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a08e2fa1b0 0x55a08e69dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.837+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a08e2fa1b0 0x55a08e69dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43782/0 (socket says 172.21.15.134:43782) 2024-07-26T11:27:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.837+0000 7f21c7e00640 1 -- 172.21.15.134:0/1212735851 learned_addr learned my addr 172.21.15.134:0/1212735851 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.837+0000 7f21bea00640 1 --2- 172.21.15.134:0/1212735851 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a08e2d94d0 0x55a08e69e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.837+0000 7f21c7400640 1 --2- 172.21.15.134:0/1212735851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a08e2d12f0 0x55a08e69e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.837+0000 7f21c7e00640 1 -- 172.21.15.134:0/1212735851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a08e2d12f0 msgr2=0x55a08e69e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.838+0000 7f21c7e00640 1 --2- 172.21.15.134:0/1212735851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a08e2d12f0 0x55a08e69e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.838+0000 7f21c7e00640 1 -- 172.21.15.134:0/1212735851 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a08e2d94d0 msgr2=0x55a08e69e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.838+0000 7f21c7e00640 1 --2- 172.21.15.134:0/1212735851 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a08e2d94d0 0x55a08e69e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.838+0000 7f21c7e00640 1 -- 172.21.15.134:0/1212735851 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a08e365790 con 0x55a08e2fa1b0 2024-07-26T11:27:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.838+0000 7f21bea00640 1 --2- 172.21.15.134:0/1212735851 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a08e2d94d0 0x55a08e69e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:27:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.838+0000 7f21c7e00640 1 --2- 172.21.15.134:0/1212735851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a08e2fa1b0 0x55a08e69dc20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f21c0011440 tx=0x7f21c0011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfaa5ac253b339a3 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.838+0000 7f21bcc00640 1 -- 172.21.15.134:0/1212735851 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c0004d20 con 0x55a08e2fa1b0 2024-07-26T11:27:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.838+0000 7f21bcc00640 1 -- 172.21.15.134:0/1212735851 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21c0004050 con 0x55a08e2fa1b0 2024-07-26T11:27:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.838+0000 7f21bcc00640 1 -- 172.21.15.134:0/1212735851 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c000f7a0 con 0x55a08e2fa1b0 2024-07-26T11:27:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.839+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 0x7f21ac0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.839+0000 7f21d5632740 1 -- 172.21.15.134:0/1212735851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a08e2fa1b0 msgr2=0x55a08e69dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-07-26T11:27:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.839+0000 7f21d5632740 1 --2- 172.21.15.134:0/1212735851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a08e2fa1b0 0x55a08e69dc20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f21c0011440 tx=0x7f21c0011900 comp rx=0 tx=0).stop 2024-07-26T11:27:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.839+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 9511+0+0 (secure 0 0 0) 0x7f21b8104730 con 0x7f21c8105550 2024-07-26T11:27:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.839+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 0x7f21ac0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.839+0000 7f21d5632740 1 -- 172.21.15.134:0/1212735851 shutdown_connections 2024-07-26T11:27:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.839+0000 7f21d5632740 1 --2- 172.21.15.134:0/1212735851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a08e2d12f0 0x55a08e69e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.839+0000 7f21d5632740 1 --2- 172.21.15.134:0/1212735851 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a08e2d94d0 0x55a08e69e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:32.886 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.839+0000 7f21d5632740 1 --2- 172.21.15.134:0/1212735851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a08e2fa1b0 0x55a08e69dc20 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.839+0000 7f21d5632740 1 -- 172.21.15.134:0/1212735851 >> 172.21.15.134:0/1212735851 conn(0x55a08e2ee300 msgr2=0x55a08e69bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-07-26T11:27:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.839+0000 7f21d5632740 1 -- 172.21.15.134:0/1212735851 shutdown_connections 2024-07-26T11:27:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.839+0000 7f21d5632740 1 -- 172.21.15.134:0/1212735851 wait complete. 2024-07-26T11:27:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.840+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 0x7f21ac0b62d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x55a08e69c8a0 tx=0x7f21b802a3f0 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.840+0000 7f21d5632740 1 -- start start 2024-07-26T11:27:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.840+0000 7f21d5632740 1 Processor -- start 2024-07-26T11:27:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21d5632740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a08e2d12f0 0x55a08e6a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21d5632740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a08e2d94d0 0x55a08e6a9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21d5632740 1 --2- >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a08e2fa1b0 0x55a08e6b5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21d5632740 1 -- --> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] -- mon_getmap magic: 0 -- 0x55a08e53bab0 con 0x55a08e2fa1b0 2024-07-26T11:27:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21d5632740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a08e59ed50 con 0x55a08e2d94d0 2024-07-26T11:27:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21d5632740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 -- 0x55a08e6ac3a0 con 0x55a08e2d12f0 2024-07-26T11:27:32.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21d5632740 10 client.0 _ll_register_callbacks cb 0x55a08e311a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-07-26T11:27:32.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21bea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a08e2d94d0 0x55a08e6a9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:32.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21bea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a08e2d94d0 0x55a08e6a9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.134:43796/0 (socket says 172.21.15.134:43796) 2024-07-26T11:27:32.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21bea00640 1 -- 172.21.15.134:0/3456275332 learned_addr learned my addr 172.21.15.134:0/3456275332 (peer_addr_for_me v2:172.21.15.134:0/0) 2024-07-26T11:27:32.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21c7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a08e2d12f0 0x55a08e6a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:32.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a08e2fa1b0 0x55a08e6b5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:32.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21bea00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a08e2d12f0 msgr2=0x55a08e6a97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:32.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21bea00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a08e2d12f0 0x55a08e6a97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:32.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21bea00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a08e2fa1b0 msgr2=0x55a08e6b5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-07-26T11:27:32.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21bea00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] conn(0x55a08e2fa1b0 0x55a08e6b5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:27:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21bea00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a08e3de620 con 0x55a08e2d94d0 2024-07-26T11:27:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.841+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a08e2d12f0 0x55a08e6a97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-07-26T11:27:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.842+0000 7f21bea00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a08e2d94d0 0x55a08e6a9d10 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f21b00129e0 tx=0x7f21b0012ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8863ec7e0ee92dde server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:27:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.842+0000 7f21b5600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-07-26T11:27:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.842+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21b000fa30 con 0x55a08e2d94d0 2024-07-26T11:27:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.842+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(103..103 src has 1..103) ==== 9511+0+0 (secure 0 0 0) 0x7f21b000fbd0 con 0x55a08e2d94d0 2024-07-26T11:27:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.842+0000 7f21d5632740 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21c0004050 con 0x55a08e2d94d0 2024-07-26T11:27:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.842+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f21b000fdd0 con 0x55a08e2d94d0 2024-07-26T11:27:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.842+0000 7f21d5632740 10 client.27041 fetch_fsmap learned FSMap version 36 2024-07-26T11:27:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.842+0000 7f21d5632740 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55a08e6b84a0 con 0x55a08e2d94d0 2024-07-26T11:27:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.843+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21c0004050 con 0x55a08e2d94d0 2024-07-26T11:27:32.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.843+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) ==== 8398+0+0 (secure 0 0 0) 0x7f21b001c4b0 con 0x55a08e2d94d0 2024-07-26T11:27:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.843+0000 7f21d5632740 10 client.27041 fetch_fsmap finished waiting for FSMap version 36 2024-07-26T11:27:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.843+0000 7f21d5632740 10 client.27041 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.a 2024-07-26T11:27:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.843+0000 7f21d5632740 20 client.27041 populate_metadata read hostname 'smithi134' 2024-07-26T11:27:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.843+0000 7f21d5632740 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:27:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.843+0000 7f21d5632740 4 client.27041 mds_command: new command op to 24721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-07-26T11:27:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.843+0000 7f21d5632740 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a08e6141c0 con 0x55a08e2b8d80 2024-07-26T11:27:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.843+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.843+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:32.843+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-07-26T11:27:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:33.044+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:33.044+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-07-26T11:27:33.046 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:33.044+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:32 smithi193 ceph-mon[37473]: pgmap v840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 9.3 KiB/s rd, 48 MiB/s wr, 416 op/s 2024-07-26T11:27:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:32 smithi134 ceph-mon[31966]: pgmap v840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 9.3 KiB/s rd, 48 MiB/s wr, 416 op/s 2024-07-26T11:27:33.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:32 smithi189 ceph-mon[36790]: pgmap v840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 9.3 KiB/s rd, 48 MiB/s wr, 416 op/s 2024-07-26T11:27:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:33.445+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:33.445+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-07-26T11:27:33.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:33.445+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:33.948 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:33 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 1. 2024-07-26T11:27:33.948 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:33 smithi189 systemd[1]: Stopped Ceph mds.b for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:27:33.948 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:33 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.b.service: Consumed 2.083s CPU time. 2024-07-26T11:27:33.949 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:33 smithi189 systemd[1]: Starting Ceph mds.b for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:27:33.949 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:33 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 1. 2024-07-26T11:27:33.949 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:33 smithi189 systemd[1]: Stopped Ceph mds.h for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:27:33.949 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:33 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.h.service: Consumed 1.983s CPU time. 2024-07-26T11:27:33.949 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:33 smithi189 systemd[1]: Starting Ceph mds.h for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:27:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:34.246+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:34.246+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:34.248 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:34.246+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-07-26T11:27:34.258 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:34 smithi189 ceph-mon[36790]: Replacing daemon mds.h as rank 4 with standby daemon mds.k 2024-07-26T11:27:34.258 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:34 smithi189 ceph-mon[36790]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-07-26T11:27:34.258 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:34 smithi189 ceph-mon[36790]: Replacing daemon mds.a as rank 0 with standby daemon mds.f 2024-07-26T11:27:34.258 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:34 smithi189 ceph-mon[36790]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T11:27:34.258 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:34 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:27:34.258 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:34 smithi189 ceph-mon[36790]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T11:27:34.258 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:34 smithi189 ceph-mon[36790]: Replacing daemon mds.b as rank 1 with standby daemon mds.e 2024-07-26T11:27:34.259 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:34 smithi189 ceph-mon[36790]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-07-26T11:27:34.259 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:34 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:replay 3 up:active 2024-07-26T11:27:34.259 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:34 smithi189 ceph-mon[36790]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T11:27:34.259 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:34 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 2 up:standby-replay 2 up:standby 3 up:replay 2 up:active 2024-07-26T11:27:34.259 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:33 smithi189 podman[72517]: 2024-07-26 11:27:33.85293862 +0000 UTC m=+0.018682597 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:27:34.259 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:34 smithi189 podman[72517]: 2024-07-26 11:27:34.092114619 +0000 UTC m=+0.257858586 container create 2c3dce63329e4bd766cb16076b2b134fe5212fe04d14599f94598c22b26b4a91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-07-26T11:27:34.259 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:33 smithi189 podman[72516]: 2024-07-26 11:27:33.850215675 +0000 UTC m=+0.018581193 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:27:34.259 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:33 smithi189 podman[72516]: 2024-07-26 11:27:33.982951683 +0000 UTC m=+0.151317176 container create e7fe7fcba74ec2cea468a51a787946b1e1b4a0a65174e93e48b58370839f162b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-07-26T11:27:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:34 smithi134 ceph-mon[31966]: Replacing daemon mds.h as rank 4 with standby daemon mds.k 2024-07-26T11:27:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:34 smithi134 ceph-mon[31966]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-07-26T11:27:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:34 smithi134 ceph-mon[31966]: Replacing daemon mds.a as rank 0 with standby daemon mds.f 2024-07-26T11:27:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:34 smithi134 ceph-mon[31966]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T11:27:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:34 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:27:34.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:34 smithi134 ceph-mon[31966]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T11:27:34.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:34 smithi134 ceph-mon[31966]: Replacing daemon mds.b as rank 1 with standby daemon mds.e 2024-07-26T11:27:34.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:34 smithi134 ceph-mon[31966]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-07-26T11:27:34.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:34 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:replay 3 up:active 2024-07-26T11:27:34.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:34 smithi134 ceph-mon[31966]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T11:27:34.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:34 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 2 up:standby-replay 2 up:standby 3 up:replay 2 up:active 2024-07-26T11:27:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:34 smithi193 ceph-mon[37473]: Replacing daemon mds.h as rank 4 with standby daemon mds.k 2024-07-26T11:27:34.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:34 smithi193 ceph-mon[37473]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-07-26T11:27:34.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:34 smithi193 ceph-mon[37473]: Replacing daemon mds.a as rank 0 with standby daemon mds.f 2024-07-26T11:27:34.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:34 smithi193 ceph-mon[37473]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T11:27:34.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:34 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:27:34.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:34 smithi193 ceph-mon[37473]: osdmap e104: 12 total, 12 up, 12 in 2024-07-26T11:27:34.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:34 smithi193 ceph-mon[37473]: Replacing daemon mds.b as rank 1 with standby daemon mds.e 2024-07-26T11:27:34.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:34 smithi193 ceph-mon[37473]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-07-26T11:27:34.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:34 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:replay 3 up:active 2024-07-26T11:27:34.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:34 smithi193 ceph-mon[37473]: osdmap e105: 12 total, 12 up, 12 in 2024-07-26T11:27:34.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:34 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 2 up:standby-replay 2 up:standby 3 up:replay 2 up:active 2024-07-26T11:27:34.665 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:34 smithi189 podman[72517]: 2024-07-26 11:27:34.472222688 +0000 UTC m=+0.637966659 container init 2c3dce63329e4bd766cb16076b2b134fe5212fe04d14599f94598c22b26b4a91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-07-26T11:27:34.665 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:34 smithi189 podman[72517]: 2024-07-26 11:27:34.475375263 +0000 UTC m=+0.641119240 container start 2c3dce63329e4bd766cb16076b2b134fe5212fe04d14599f94598c22b26b4a91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_CLEAN=True) 2024-07-26T11:27:34.916 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:34 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 1. 2024-07-26T11:27:34.917 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:34 smithi134 systemd[1]: Stopped Ceph mds.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:27:34.917 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:34 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Consumed 26.787s CPU time. 2024-07-26T11:27:34.917 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:34 smithi134 systemd[1]: Starting Ceph mds.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:27:34.944 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:34 smithi189 podman[72516]: 2024-07-26 11:27:34.664559106 +0000 UTC m=+0.832924599 container init e7fe7fcba74ec2cea468a51a787946b1e1b4a0a65174e93e48b58370839f162b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:27:34.944 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:34 smithi189 podman[72516]: 2024-07-26 11:27:34.667237186 +0000 UTC m=+0.835602680 container start e7fe7fcba74ec2cea468a51a787946b1e1b4a0a65174e93e48b58370839f162b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-07-26T11:27:34.944 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:34 smithi189 bash[72516]: e7fe7fcba74ec2cea468a51a787946b1e1b4a0a65174e93e48b58370839f162b 2024-07-26T11:27:34.944 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:34 smithi189 systemd[1]: Started Ceph mds.h for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:27:34.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:34.945+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:34.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:34.945+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-07-26T11:27:34.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:34.945+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:35.168 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:34 smithi134 podman[111433]: 2024-07-26 11:27:34.818547693 +0000 UTC m=+0.036479280 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:27:35.168 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:34 smithi134 podman[111433]: 2024-07-26 11:27:34.946689048 +0000 UTC m=+0.164620670 container create 55398d1cc7a5c8fb24033b140329ea150261cadfade5f5e24a4013368b5939e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-07-26T11:27:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:35 smithi134 ceph-mon[31966]: pgmap v843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 32 KiB/s wr, 9 op/s 2024-07-26T11:27:35.423 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:35 smithi134 podman[111433]: 2024-07-26 11:27:35.275467718 +0000 UTC m=+0.493399316 container init 55398d1cc7a5c8fb24033b140329ea150261cadfade5f5e24a4013368b5939e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-07-26T11:27:35.423 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:35 smithi134 podman[111433]: 2024-07-26 11:27:35.287510201 +0000 UTC m=+0.505441797 container start 55398d1cc7a5c8fb24033b140329ea150261cadfade5f5e24a4013368b5939e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-07-26T11:27:35.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:35 smithi189 ceph-mon[36790]: pgmap v843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 32 KiB/s wr, 9 op/s 2024-07-26T11:27:35.444 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:35 smithi189 ceph-mds[72555]: -- 172.21.15.189:0/1636487162 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5573e83bf860 con 0x5573ec561600 2024-07-26T11:27:35.444 INFO:journalctl@ceph.mds.h.smithi189.stdout:Jul 26 11:27:35 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-h[72548]: starting mds.h at 2024-07-26T11:27:35.444 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:35 smithi189 bash[72517]: 2c3dce63329e4bd766cb16076b2b134fe5212fe04d14599f94598c22b26b4a91 2024-07-26T11:27:35.444 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:35 smithi189 systemd[1]: Started Ceph mds.b for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:27:35.445 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:35 smithi189 ceph-mds[72553]: -- 172.21.15.189:0/2692802894 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56033fd41860 con 0x560343ee3600 2024-07-26T11:27:35.445 INFO:journalctl@ceph.mds.b.smithi189.stdout:Jul 26 11:27:35 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-b[72543]: starting mds.b at 2024-07-26T11:27:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:35 smithi193 ceph-mon[37473]: pgmap v843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 32 KiB/s wr, 9 op/s 2024-07-26T11:27:35.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:35.849+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:35.850+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T11:27:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:35.850+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:27:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:35.850+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:27:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:35.850+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-07-26T11:27:35.926 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:35 smithi134 bash[111433]: 55398d1cc7a5c8fb24033b140329ea150261cadfade5f5e24a4013368b5939e7 2024-07-26T11:27:35.926 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:35 smithi134 systemd[1]: Started Ceph mds.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:27:35.926 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:35 smithi134 ceph-mds[111452]: -- 172.21.15.134:0/2853556340 <== mon.0 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5612fbd87860 con 0x5612fff29a80 2024-07-26T11:27:35.926 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: starting mds.a at 2024-07-26T11:27:36.291 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:36 smithi193 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 1. 2024-07-26T11:27:36.291 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:36 smithi193 systemd[1]: Stopped Ceph mds.i for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:27:36.291 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:36 smithi193 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.i.service: Consumed 2.033s CPU time. 2024-07-26T11:27:36.291 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:36 smithi193 systemd[1]: Starting Ceph mds.i for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:27:36.291 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:36 smithi193 podman[73507]: 2024-07-26 11:27:36.171514643 +0000 UTC m=+0.018189510 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:27:36.291 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:36 smithi193 podman[73507]: 2024-07-26 11:27:36.289886089 +0000 UTC m=+0.136561016 container create 6a547c17d88fab356e82570cb564e23ff00e8618386dd27f4c6569bd058a1368 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD) 2024-07-26T11:27:36.292 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:36 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/1314042896,v1:172.21.15.189:6835/1314042896] up:resolve 2024-07-26T11:27:36.292 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:36 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:resolve 2024-07-26T11:27:36.292 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:36 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6832/1897636383,v1:172.21.15.189:6833/1897636383] up:resolve 2024-07-26T11:27:36.292 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:36 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6836/3720667957,v1:172.21.15.189:6838/3720667957] up:boot 2024-07-26T11:27:36.292 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:36 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6837/2280137885,v1:172.21.15.189:6839/2280137885] up:boot 2024-07-26T11:27:36.292 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:36 smithi193 ceph-mon[37473]: Replacing daemon mds.i as rank 2 with standby daemon mds.j 2024-07-26T11:27:36.292 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:36 smithi193 ceph-mon[37473]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-07-26T11:27:36.292 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:36 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 3 up:resolve 2 up:active 2024-07-26T11:27:36.292 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T11:27:36.292 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T11:27:36.292 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:36 smithi193 ceph-mon[37473]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T11:27:36.293 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:36 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 1 up:standby-replay 4 up:standby 1 up:replay 3 up:resolve 1 up:active 2024-07-26T11:27:36.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:36 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/1314042896,v1:172.21.15.189:6835/1314042896] up:resolve 2024-07-26T11:27:36.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:36 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:resolve 2024-07-26T11:27:36.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:36 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6832/1897636383,v1:172.21.15.189:6833/1897636383] up:resolve 2024-07-26T11:27:36.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:36 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6836/3720667957,v1:172.21.15.189:6838/3720667957] up:boot 2024-07-26T11:27:36.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:36 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6837/2280137885,v1:172.21.15.189:6839/2280137885] up:boot 2024-07-26T11:27:36.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:36 smithi134 ceph-mon[31966]: Replacing daemon mds.i as rank 2 with standby daemon mds.j 2024-07-26T11:27:36.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:36 smithi134 ceph-mon[31966]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-07-26T11:27:36.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:36 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 3 up:resolve 2 up:active 2024-07-26T11:27:36.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T11:27:36.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T11:27:36.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:36 smithi134 ceph-mon[31966]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T11:27:36.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:36 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 1 up:standby-replay 4 up:standby 1 up:replay 3 up:resolve 1 up:active 2024-07-26T11:27:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:36 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/1314042896,v1:172.21.15.189:6835/1314042896] up:resolve 2024-07-26T11:27:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:36 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:resolve 2024-07-26T11:27:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:36 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6832/1897636383,v1:172.21.15.189:6833/1897636383] up:resolve 2024-07-26T11:27:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:36 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6836/3720667957,v1:172.21.15.189:6838/3720667957] up:boot 2024-07-26T11:27:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:36 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6837/2280137885,v1:172.21.15.189:6839/2280137885] up:boot 2024-07-26T11:27:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:36 smithi189 ceph-mon[36790]: Replacing daemon mds.i as rank 2 with standby daemon mds.j 2024-07-26T11:27:36.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:36 smithi189 ceph-mon[36790]: MDS daemon mds.i is removed because it is dead or otherwise unavailable. 2024-07-26T11:27:36.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:36 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 2 up:standby-replay 4 up:standby 3 up:resolve 2 up:active 2024-07-26T11:27:36.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-07-26T11:27:36.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-07-26T11:27:36.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:36 smithi189 ceph-mon[36790]: osdmap e106: 12 total, 12 up, 12 in 2024-07-26T11:27:36.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:36 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 1 up:standby-replay 4 up:standby 1 up:replay 3 up:resolve 1 up:active 2024-07-26T11:27:37.025 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:36 smithi193 podman[73507]: 2024-07-26 11:27:36.551085465 +0000 UTC m=+0.397760324 container init 6a547c17d88fab356e82570cb564e23ff00e8618386dd27f4c6569bd058a1368 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:27:37.025 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:36 smithi193 podman[73507]: 2024-07-26 11:27:36.553885348 +0000 UTC m=+0.400560207 container start 6a547c17d88fab356e82570cb564e23ff00e8618386dd27f4c6569bd058a1368 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=) 2024-07-26T11:27:37.025 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:36 smithi193 ceph-mds[73527]: -- 172.21.15.193:0/2069202947 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a5107ff860 con 0x55a5149a1600 2024-07-26T11:27:37.025 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:36 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-i[73522]: starting mds.i at 2024-07-26T11:27:37.025 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:36 smithi193 bash[73507]: 6a547c17d88fab356e82570cb564e23ff00e8618386dd27f4c6569bd058a1368 2024-07-26T11:27:37.025 INFO:journalctl@ceph.mds.i.smithi193.stdout:Jul 26 11:27:36 smithi193 systemd[1]: Started Ceph mds.i for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:27:37.451 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:37 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6834/4071410587,v1:172.21.15.134:6835/4071410587] up:resolve 2024-07-26T11:27:37.451 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:37 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/3050637565,v1:172.21.15.134:6841/3050637565] up:boot 2024-07-26T11:27:37.451 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:37 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 1 up:standby-replay 5 up:standby 4 up:resolve 1 up:active 2024-07-26T11:27:37.452 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:27:37.452 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:37 smithi134 ceph-mon[31966]: pgmap v845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:27:37.452 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:37 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6834/4071410587,v1:172.21.15.134:6835/4071410587] up:resolve 2024-07-26T11:27:37.453 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:37 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/3050637565,v1:172.21.15.134:6841/3050637565] up:boot 2024-07-26T11:27:37.453 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:37 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 1 up:standby-replay 5 up:standby 4 up:resolve 1 up:active 2024-07-26T11:27:37.453 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:27:37.453 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:37 smithi193 ceph-mon[37473]: pgmap v845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:27:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:37 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6834/4071410587,v1:172.21.15.134:6835/4071410587] up:resolve 2024-07-26T11:27:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:37 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/3050637565,v1:172.21.15.134:6841/3050637565] up:boot 2024-07-26T11:27:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:37 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 1 up:standby-replay 5 up:standby 4 up:resolve 1 up:active 2024-07-26T11:27:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:27:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:37 smithi189 ceph-mon[36790]: pgmap v845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:27:38.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:38 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00399994 2024-07-26T11:27:38.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:38 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.0049999 2024-07-26T11:27:38.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:38 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00399991 2024-07-26T11:27:38.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:38 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00999981 2024-07-26T11:27:38.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:38 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:reconnect 2024-07-26T11:27:38.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:38 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6832/1897636383,v1:172.21.15.189:6833/1897636383] up:reconnect 2024-07-26T11:27:38.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:38 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/1314042896,v1:172.21.15.189:6835/1314042896] up:reconnect 2024-07-26T11:27:38.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:38 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6834/4071410587,v1:172.21.15.134:6835/4071410587] up:reconnect 2024-07-26T11:27:38.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:38 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.193:6838/355644791,v1:172.21.15.193:6839/355644791] up:boot 2024-07-26T11:27:38.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:38 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-07-26T11:27:38.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:27:38.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:38 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0149998 2024-07-26T11:27:38.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:38 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0379992 2024-07-26T11:27:38.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:38 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.432992 2024-07-26T11:27:38.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:38 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00399994 2024-07-26T11:27:38.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:38 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.0049999 2024-07-26T11:27:38.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:38 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00399991 2024-07-26T11:27:38.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:38 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00999981 2024-07-26T11:27:38.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:38 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:reconnect 2024-07-26T11:27:38.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:38 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6832/1897636383,v1:172.21.15.189:6833/1897636383] up:reconnect 2024-07-26T11:27:38.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:38 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/1314042896,v1:172.21.15.189:6835/1314042896] up:reconnect 2024-07-26T11:27:38.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:38 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6834/4071410587,v1:172.21.15.134:6835/4071410587] up:reconnect 2024-07-26T11:27:38.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:38 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.193:6838/355644791,v1:172.21.15.193:6839/355644791] up:boot 2024-07-26T11:27:38.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:38 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-07-26T11:27:38.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:27:38.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:38 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0149998 2024-07-26T11:27:38.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:38 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0379992 2024-07-26T11:27:38.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:38 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.432992 2024-07-26T11:27:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:38 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00399994 2024-07-26T11:27:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:38 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.0049999 2024-07-26T11:27:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:38 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00399991 2024-07-26T11:27:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:38 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00999981 2024-07-26T11:27:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:38 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:reconnect 2024-07-26T11:27:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:38 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6832/1897636383,v1:172.21.15.189:6833/1897636383] up:reconnect 2024-07-26T11:27:38.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:38 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/1314042896,v1:172.21.15.189:6835/1314042896] up:reconnect 2024-07-26T11:27:38.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:38 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6834/4071410587,v1:172.21.15.134:6835/4071410587] up:reconnect 2024-07-26T11:27:38.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:38 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.193:6838/355644791,v1:172.21.15.193:6839/355644791] up:boot 2024-07-26T11:27:38.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:38 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:reconnect 1 up:active 2024-07-26T11:27:38.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-07-26T11:27:38.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:38 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0149998 2024-07-26T11:27:38.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:38 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0379992 2024-07-26T11:27:38.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:38 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.432992 2024-07-26T11:27:39.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:39.054+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:39.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:39.056+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T11:27:39.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:39.056+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:27:39.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:39.056+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:27:39.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:39.056+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-07-26T11:27:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:39 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6832/1897636383,v1:172.21.15.189:6833/1897636383] up:rejoin 2024-07-26T11:27:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:39 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:rejoin 2024-07-26T11:27:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:39 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6834/4071410587,v1:172.21.15.134:6835/4071410587] up:rejoin 2024-07-26T11:27:39.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:39 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/1314042896,v1:172.21.15.189:6835/1314042896] up:rejoin 2024-07-26T11:27:39.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:39 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-07-26T11:27:39.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:39 smithi193 ceph-mon[37473]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-07-26T11:27:39.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:39 smithi193 ceph-mon[37473]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:27:39.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:39 smithi193 ceph-mon[37473]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-07-26T11:27:39.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:39 smithi193 ceph-mon[37473]: pgmap v846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 9 op/s 2024-07-26T11:27:39.586 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6832/1897636383,v1:172.21.15.189:6833/1897636383] up:rejoin 2024-07-26T11:27:39.586 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:rejoin 2024-07-26T11:27:39.586 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6834/4071410587,v1:172.21.15.134:6835/4071410587] up:rejoin 2024-07-26T11:27:39.586 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/1314042896,v1:172.21.15.189:6835/1314042896] up:rejoin 2024-07-26T11:27:39.586 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-07-26T11:27:39.587 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-mon[36790]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-07-26T11:27:39.587 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-mon[36790]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:27:39.587 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-mon[36790]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-07-26T11:27:39.587 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-mon[36790]: pgmap v846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 9 op/s 2024-07-26T11:27:39.587 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:27:39.588 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: *** Caught signal (Aborted) ** 2024-07-26T11:27:39.588 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: in thread 7fc0b02a7640 thread_name:ms_dispatch 2024-07-26T11:27:39.588 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:39.588 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc0b567e6f0] 2024-07-26T11:27:39.588 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc0b56cb94c] 2024-07-26T11:27:39.588 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 3: raise() 2024-07-26T11:27:39.588 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 4: abort() 2024-07-26T11:27:39.588 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 5: /lib64/libc.so.6(+0x2871b) [0x7fc0b566871b] 2024-07-26T11:27:39.588 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 6: /lib64/libc.so.6(+0x37386) [0x7fc0b5677386] 2024-07-26T11:27:39.588 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55e2bcb1e6b9] 2024-07-26T11:27:39.588 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55e2bccdc80c] 2024-07-26T11:27:39.588 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55e2bcc1c2ca] 2024-07-26T11:27:39.588 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55e2bcbec753] 2024-07-26T11:27:39.589 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55e2bcbeda1b] 2024-07-26T11:27:39.589 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55e2bcbd1752] 2024-07-26T11:27:39.589 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55e2bcb1fa56] 2024-07-26T11:27:39.589 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55e2bcb23637] 2024-07-26T11:27:39.589 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55e2bcb23f5e] 2024-07-26T11:27:39.589 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55e2bcb0703a] 2024-07-26T11:27:39.589 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fc0b5f4b407] 2024-07-26T11:27:39.589 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 18: (DispatchQueue::entry()+0x60f) [0x7fc0b5f4bcf9] 2024-07-26T11:27:39.589 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fc0b60285a1] 2024-07-26T11:27:39.589 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 20: (Thread::entry_wrapper()+0x43) [0x7fc0b5dd5cdd] 2024-07-26T11:27:39.589 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc0b5dd5cf9] 2024-07-26T11:27:39.590 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 22: /lib64/libc.so.6(+0x89c02) [0x7fc0b56c9c02] 2024-07-26T11:27:39.590 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fc0b574ec40] 2024-07-26T11:27:39.590 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 2024-07-26T11:27:39.215+0000 7fc0b02a7640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:39.590 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: in thread 7fc0b02a7640 thread_name:ms_dispatch 2024-07-26T11:27:39.590 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 2024-07-26T11:27:39.590 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:39.590 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc0b567e6f0] 2024-07-26T11:27:39.590 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc0b56cb94c] 2024-07-26T11:27:39.590 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 3: raise() 2024-07-26T11:27:39.590 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 4: abort() 2024-07-26T11:27:39.590 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 5: /lib64/libc.so.6(+0x2871b) [0x7fc0b566871b] 2024-07-26T11:27:39.590 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 6: /lib64/libc.so.6(+0x37386) [0x7fc0b5677386] 2024-07-26T11:27:39.590 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55e2bcb1e6b9] 2024-07-26T11:27:39.590 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55e2bccdc80c] 2024-07-26T11:27:39.591 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55e2bcc1c2ca] 2024-07-26T11:27:39.591 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55e2bcbec753] 2024-07-26T11:27:39.591 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55e2bcbeda1b] 2024-07-26T11:27:39.591 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55e2bcbd1752] 2024-07-26T11:27:39.591 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55e2bcb1fa56] 2024-07-26T11:27:39.591 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55e2bcb23637] 2024-07-26T11:27:39.591 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55e2bcb23f5e] 2024-07-26T11:27:39.591 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55e2bcb0703a] 2024-07-26T11:27:39.591 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fc0b5f4b407] 2024-07-26T11:27:39.591 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 18: (DispatchQueue::entry()+0x60f) [0x7fc0b5f4bcf9] 2024-07-26T11:27:39.591 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fc0b60285a1] 2024-07-26T11:27:39.592 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 20: (Thread::entry_wrapper()+0x43) [0x7fc0b5dd5cdd] 2024-07-26T11:27:39.592 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc0b5dd5cf9] 2024-07-26T11:27:39.592 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 22: /lib64/libc.so.6(+0x89c02) [0x7fc0b56c9c02] 2024-07-26T11:27:39.592 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fc0b574ec40] 2024-07-26T11:27:39.592 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:39.592 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 2024-07-26T11:27:39.592 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 0> 2024-07-26T11:27:39.215+0000 7fc0b02a7640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:39.592 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: in thread 7fc0b02a7640 thread_name:ms_dispatch 2024-07-26T11:27:39.592 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 2024-07-26T11:27:39.593 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:39.593 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc0b567e6f0] 2024-07-26T11:27:39.593 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc0b56cb94c] 2024-07-26T11:27:39.593 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 3: raise() 2024-07-26T11:27:39.593 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 4: abort() 2024-07-26T11:27:39.593 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 5: /lib64/libc.so.6(+0x2871b) [0x7fc0b566871b] 2024-07-26T11:27:39.593 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 6: /lib64/libc.so.6(+0x37386) [0x7fc0b5677386] 2024-07-26T11:27:39.593 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55e2bcb1e6b9] 2024-07-26T11:27:39.593 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55e2bccdc80c] 2024-07-26T11:27:39.593 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55e2bcc1c2ca] 2024-07-26T11:27:39.593 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55e2bcbec753] 2024-07-26T11:27:39.593 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55e2bcbeda1b] 2024-07-26T11:27:39.593 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55e2bcbd1752] 2024-07-26T11:27:39.594 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55e2bcb1fa56] 2024-07-26T11:27:39.594 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55e2bcb23637] 2024-07-26T11:27:39.594 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55e2bcb23f5e] 2024-07-26T11:27:39.594 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55e2bcb0703a] 2024-07-26T11:27:39.594 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fc0b5f4b407] 2024-07-26T11:27:39.594 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 18: (DispatchQueue::entry()+0x60f) [0x7fc0b5f4bcf9] 2024-07-26T11:27:39.594 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fc0b60285a1] 2024-07-26T11:27:39.594 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 20: (Thread::entry_wrapper()+0x43) [0x7fc0b5dd5cdd] 2024-07-26T11:27:39.594 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc0b5dd5cf9] 2024-07-26T11:27:39.594 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 22: /lib64/libc.so.6(+0x89c02) [0x7fc0b56c9c02] 2024-07-26T11:27:39.594 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fc0b574ec40] 2024-07-26T11:27:39.594 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:39.594 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 2024-07-26T11:27:39.594 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 0> 2024-07-26T11:27:39.215+0000 7fc0b02a7640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:39.595 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: in thread 7fc0b02a7640 thread_name:ms_dispatch 2024-07-26T11:27:39.595 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 2024-07-26T11:27:39.595 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:39.595 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fc0b567e6f0] 2024-07-26T11:27:39.595 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fc0b56cb94c] 2024-07-26T11:27:39.595 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 3: raise() 2024-07-26T11:27:39.595 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 4: abort() 2024-07-26T11:27:39.595 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 5: /lib64/libc.so.6(+0x2871b) [0x7fc0b566871b] 2024-07-26T11:27:39.595 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 6: /lib64/libc.so.6(+0x37386) [0x7fc0b5677386] 2024-07-26T11:27:39.595 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55e2bcb1e6b9] 2024-07-26T11:27:39.595 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55e2bccdc80c] 2024-07-26T11:27:39.595 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55e2bcc1c2ca] 2024-07-26T11:27:39.595 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55e2bcbec753] 2024-07-26T11:27:39.596 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55e2bcbeda1b] 2024-07-26T11:27:39.596 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55e2bcbd1752] 2024-07-26T11:27:39.596 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55e2bcb1fa56] 2024-07-26T11:27:39.596 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55e2bcb23637] 2024-07-26T11:27:39.596 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55e2bcb23f5e] 2024-07-26T11:27:39.596 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55e2bcb0703a] 2024-07-26T11:27:39.596 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fc0b5f4b407] 2024-07-26T11:27:39.596 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 18: (DispatchQueue::entry()+0x60f) [0x7fc0b5f4bcf9] 2024-07-26T11:27:39.596 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fc0b60285a1] 2024-07-26T11:27:39.596 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 20: (Thread::entry_wrapper()+0x43) [0x7fc0b5dd5cdd] 2024-07-26T11:27:39.596 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 21: (Thread::_entry_func(void*)+0xd) [0x7fc0b5dd5cf9] 2024-07-26T11:27:39.596 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 22: /lib64/libc.so.6(+0x89c02) [0x7fc0b56c9c02] 2024-07-26T11:27:39.596 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fc0b574ec40] 2024-07-26T11:27:39.596 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:39.597 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[67577]: 2024-07-26T11:27:39.597 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 podman[72611]: 2024-07-26 11:27:39.274716568 +0000 UTC m=+0.020079330 container died ece0fd5f8679b7c3eb22a2ee7a1974018f0b8ae8698d6e2e6ff89d06d89aeb29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:27:39.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:39 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6832/1897636383,v1:172.21.15.189:6833/1897636383] up:rejoin 2024-07-26T11:27:39.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:39 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:rejoin 2024-07-26T11:27:39.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:39 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6834/4071410587,v1:172.21.15.134:6835/4071410587] up:rejoin 2024-07-26T11:27:39.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:39 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/1314042896,v1:172.21.15.189:6835/1314042896] up:rejoin 2024-07-26T11:27:39.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:39 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 4 up:rejoin 1 up:active 2024-07-26T11:27:39.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:39 smithi134 ceph-mon[31966]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-07-26T11:27:39.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:39 smithi134 ceph-mon[31966]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:27:39.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:39 smithi134 ceph-mon[31966]: daemon mds.j is now active in filesystem cephfs as rank 2 2024-07-26T11:27:39.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:39 smithi134 ceph-mon[31966]: pgmap v846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 9 op/s 2024-07-26T11:27:40.206 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 podman[72611]: 2024-07-26 11:27:39.945387916 +0000 UTC m=+0.690750714 container remove ece0fd5f8679b7c3eb22a2ee7a1974018f0b8ae8698d6e2e6ff89d06d89aeb29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:27:40.206 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:39 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:27:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:40 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6832/1897636383,v1:172.21.15.189:6833/1897636383] up:active 2024-07-26T11:27:40.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:40 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/1314042896,v1:172.21.15.189:6835/1314042896] up:active 2024-07-26T11:27:40.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:40 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6834/4071410587,v1:172.21.15.134:6835/4071410587] up:active 2024-07-26T11:27:40.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:40 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:clientreplay 2024-07-26T11:27:40.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:40 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 1 up:clientreplay 4 up:active 2024-07-26T11:27:40.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:40 smithi193 ceph-mon[37473]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T11:27:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:40 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6832/1897636383,v1:172.21.15.189:6833/1897636383] up:active 2024-07-26T11:27:40.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:40 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/1314042896,v1:172.21.15.189:6835/1314042896] up:active 2024-07-26T11:27:40.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:40 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6834/4071410587,v1:172.21.15.134:6835/4071410587] up:active 2024-07-26T11:27:40.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:40 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:clientreplay 2024-07-26T11:27:40.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:40 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 1 up:clientreplay 4 up:active 2024-07-26T11:27:40.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:40 smithi134 ceph-mon[31966]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T11:27:40.694 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:40 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T11:27:40.694 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:40 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Consumed 2.025s CPU time. 2024-07-26T11:27:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:40 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6832/1897636383,v1:172.21.15.189:6833/1897636383] up:active 2024-07-26T11:27:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:40 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/1314042896,v1:172.21.15.189:6835/1314042896] up:active 2024-07-26T11:27:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:40 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6834/4071410587,v1:172.21.15.134:6835/4071410587] up:active 2024-07-26T11:27:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:40 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:clientreplay 2024-07-26T11:27:40.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:40 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 1 up:standby-replay 6 up:standby 1 up:clientreplay 4 up:active 2024-07-26T11:27:40.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:40 smithi189 ceph-mon[36790]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-07-26T11:27:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:41 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:27:41.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:41 smithi193 ceph-mon[37473]: Cluster is now healthy 2024-07-26T11:27:41.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:41 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:active 2024-07-26T11:27:41.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:41 smithi193 ceph-mon[37473]: fsmap cephfs:5 1 up:standby-replay 6 up:standby 5 up:active 2024-07-26T11:27:41.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:41 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:27:41.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:41 smithi193 ceph-mon[37473]: pgmap v847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 26 KiB/s rd, 454 B/s wr, 17 op/s 2024-07-26T11:27:41.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:41 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:27:41.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:41 smithi134 ceph-mon[31966]: Cluster is now healthy 2024-07-26T11:27:41.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:41 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:active 2024-07-26T11:27:41.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:41 smithi134 ceph-mon[31966]: fsmap cephfs:5 1 up:standby-replay 6 up:standby 5 up:active 2024-07-26T11:27:41.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:41 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:27:41.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:41 smithi134 ceph-mon[31966]: pgmap v847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 26 KiB/s rd, 454 B/s wr, 17 op/s 2024-07-26T11:27:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:41 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:27:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:41 smithi189 ceph-mon[36790]: Cluster is now healthy 2024-07-26T11:27:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:41 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:active 2024-07-26T11:27:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:41 smithi189 ceph-mon[36790]: fsmap cephfs:5 1 up:standby-replay 6 up:standby 5 up:active 2024-07-26T11:27:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:41 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:27:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:41 smithi189 ceph-mon[36790]: pgmap v847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 26 KiB/s rd, 454 B/s wr, 17 op/s 2024-07-26T11:27:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:42 smithi193 ceph-mon[37473]: pgmap v848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 55 KiB/s rd, 67 KiB/s wr, 32 op/s 2024-07-26T11:27:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:42 smithi134 ceph-mon[31966]: pgmap v848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 55 KiB/s rd, 67 KiB/s wr, 32 op/s 2024-07-26T11:27:43.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:42 smithi189 ceph-mon[36790]: pgmap v848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 55 KiB/s rd, 67 KiB/s wr, 32 op/s 2024-07-26T11:27:44.525 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:27:44 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:27:44.044+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-07-26T11:27:44.525 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:27:44 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:27:44.044+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:27:44.525 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:27:44 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:27:44.044+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-07-26T11:27:44.525 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:27:44 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:27:44.044+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-07-26T11:27:44.525 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:27:44 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:27:44.044+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-07-26T11:27:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:44 smithi193 ceph-mon[37473]: pgmap v849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 59 KiB/s rd, 85 KiB/s wr, 33 op/s 2024-07-26T11:27:45.292 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:44 smithi134 ceph-mon[31966]: pgmap v849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 59 KiB/s rd, 85 KiB/s wr, 33 op/s 2024-07-26T11:27:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:44 smithi189 ceph-mon[36790]: pgmap v849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 59 KiB/s rd, 85 KiB/s wr, 33 op/s 2024-07-26T11:27:45.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:45.463+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:45.465+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T11:27:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:45.465+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:27:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:45.465+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:27:45.467 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:45.465+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-07-26T11:27:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:46 smithi134 ceph-mon[31966]: pgmap v850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 51 KiB/s rd, 111 KiB/s wr, 30 op/s 2024-07-26T11:27:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:46 smithi189 ceph-mon[36790]: pgmap v850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 51 KiB/s rd, 111 KiB/s wr, 30 op/s 2024-07-26T11:27:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:46 smithi193 ceph-mon[37473]: pgmap v850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 51 KiB/s rd, 111 KiB/s wr, 30 op/s 2024-07-26T11:27:47.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:47.752+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:27:47.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:47.754+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-26T11:27:47.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:47.754+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:27:47.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:47.754+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:27:47.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:27:47.755+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:48 smithi134 ceph-mon[31966]: pgmap v851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 51 KiB/s rd, 108 KiB/s wr, 32 op/s 2024-07-26T11:27:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:48 smithi193 ceph-mon[37473]: pgmap v851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 51 KiB/s rd, 108 KiB/s wr, 32 op/s 2024-07-26T11:27:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:48 smithi189 ceph-mon[36790]: pgmap v851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 51 KiB/s rd, 108 KiB/s wr, 32 op/s 2024-07-26T11:27:50.944 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:50 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 1. 2024-07-26T11:27:50.944 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:50 smithi189 systemd[1]: Stopped Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:27:50.944 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:50 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Consumed 2.025s CPU time. 2024-07-26T11:27:50.944 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:50 smithi189 systemd[1]: Starting Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:27:50.944 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:50 smithi189 podman[72727]: 2024-07-26 11:27:50.578561163 +0000 UTC m=+0.017646179 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:27:51.220 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:50 smithi189 ceph-mon[36790]: pgmap v852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 49 KiB/s rd, 108 KiB/s wr, 30 op/s 2024-07-26T11:27:51.221 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:50 smithi189 podman[72727]: 2024-07-26 11:27:50.946159516 +0000 UTC m=+0.385258143 container create 81b2bf205de143449ee74cf3574f27acabdb4efc584bd312584b20e92367f843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2) 2024-07-26T11:27:51.221 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:51 smithi189 podman[72727]: 2024-07-26 11:27:51.182094263 +0000 UTC m=+0.621179287 container init 81b2bf205de143449ee74cf3574f27acabdb4efc584bd312584b20e92367f843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-07-26T11:27:51.221 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:51 smithi189 podman[72727]: 2024-07-26 11:27:51.184820451 +0000 UTC m=+0.623905463 container start 81b2bf205de143449ee74cf3574f27acabdb4efc584bd312584b20e92367f843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724) 2024-07-26T11:27:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:50 smithi193 ceph-mon[37473]: pgmap v852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 49 KiB/s rd, 108 KiB/s wr, 30 op/s 2024-07-26T11:27:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:50 smithi134 ceph-mon[31966]: pgmap v852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 49 KiB/s rd, 108 KiB/s wr, 30 op/s 2024-07-26T11:27:51.694 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:51 smithi189 ceph-mds[72746]: -- 172.21.15.189:0/1816556011 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559b14dd9860 con 0x559b18f7b180 2024-07-26T11:27:51.694 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:51 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: starting mds.e at 2024-07-26T11:27:51.694 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:51 smithi189 bash[72727]: 81b2bf205de143449ee74cf3574f27acabdb4efc584bd312584b20e92367f843 2024-07-26T11:27:51.694 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:27:51 smithi189 systemd[1]: Started Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:27:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:51 smithi193 ceph-mon[37473]: daemon mds.e restarted 2024-07-26T11:27:52.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:51 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:27:52.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:51 smithi193 ceph-mon[37473]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T11:27:52.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:51 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/1380616145,v1:172.21.15.189:6835/1380616145] up:boot 2024-07-26T11:27:52.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:51 smithi193 ceph-mon[37473]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:27:52.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:51 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:27:52.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:27:52.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:51 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:27:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:51 smithi134 ceph-mon[31966]: daemon mds.e restarted 2024-07-26T11:27:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:51 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:27:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:51 smithi134 ceph-mon[31966]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T11:27:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:51 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/1380616145,v1:172.21.15.189:6835/1380616145] up:boot 2024-07-26T11:27:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:51 smithi134 ceph-mon[31966]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:27:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:51 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:27:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:27:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:51 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:27:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:51 smithi189 ceph-mon[36790]: daemon mds.e restarted 2024-07-26T11:27:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:51 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:27:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:51 smithi189 ceph-mon[36790]: osdmap e107: 12 total, 12 up, 12 in 2024-07-26T11:27:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:51 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/1380616145,v1:172.21.15.189:6835/1380616145] up:boot 2024-07-26T11:27:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:51 smithi189 ceph-mon[36790]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:27:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:51 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:27:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:27:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:51 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:27:53.513 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:53 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/3050637565,v1:172.21.15.134:6841/3050637565] up:resolve 2024-07-26T11:27:53.514 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:53 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:27:53.514 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:53 smithi134 ceph-mon[31966]: pgmap v854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 16 KiB/s rd, 71 KiB/s wr, 16 op/s 2024-07-26T11:27:53.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:53 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/3050637565,v1:172.21.15.134:6841/3050637565] up:resolve 2024-07-26T11:27:53.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:53 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:27:53.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:53 smithi193 ceph-mon[37473]: pgmap v854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 16 KiB/s rd, 71 KiB/s wr, 16 op/s 2024-07-26T11:27:53.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:53 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/3050637565,v1:172.21.15.134:6841/3050637565] up:resolve 2024-07-26T11:27:53.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:53 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:27:53.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:53 smithi189 ceph-mon[36790]: pgmap v854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 16 KiB/s rd, 71 KiB/s wr, 16 op/s 2024-07-26T11:27:54.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:54 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:27:54.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:54 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/3050637565,v1:172.21.15.134:6841/3050637565] up:reconnect 2024-07-26T11:27:54.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:54 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:27:54.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:54 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999979 2024-07-26T11:27:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:54 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:27:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:54 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/3050637565,v1:172.21.15.134:6841/3050637565] up:reconnect 2024-07-26T11:27:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:54 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:27:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:54 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999979 2024-07-26T11:27:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:54 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:27:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:54 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/3050637565,v1:172.21.15.134:6841/3050637565] up:reconnect 2024-07-26T11:27:54.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:54 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:27:54.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:54 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999979 2024-07-26T11:27:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/3050637565,v1:172.21.15.134:6841/3050637565] up:rejoin 2024-07-26T11:27:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:27:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-mon[31966]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:27:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-mon[31966]: pgmap v855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 46 KiB/s wr, 11 op/s 2024-07-26T11:27:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:27:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-mon[31966]: Cluster is now healthy 2024-07-26T11:27:55.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:27:55.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: *** Caught signal (Aborted) ** 2024-07-26T11:27:55.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: in thread 7f21a8a00640 thread_name:ms_dispatch 2024-07-26T11:27:55.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:55.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f21aec3e6f0] 2024-07-26T11:27:55.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f21aec8b94c] 2024-07-26T11:27:55.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 3: raise() 2024-07-26T11:27:55.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 4: abort() 2024-07-26T11:27:55.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 5: /lib64/libc.so.6(+0x2871b) [0x7f21aec2871b] 2024-07-26T11:27:55.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 6: /lib64/libc.so.6(+0x37386) [0x7f21aec37386] 2024-07-26T11:27:55.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5612f59846b9] 2024-07-26T11:27:55.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5612f5b4280c] 2024-07-26T11:27:55.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5612f5a822ca] 2024-07-26T11:27:55.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5612f5a52753] 2024-07-26T11:27:55.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5612f5a53a1b] 2024-07-26T11:27:55.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5612f5a37752] 2024-07-26T11:27:55.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5612f5985a56] 2024-07-26T11:27:55.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5612f5989637] 2024-07-26T11:27:55.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5612f5989f5e] 2024-07-26T11:27:55.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5612f596d03a] 2024-07-26T11:27:55.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f21af79b407] 2024-07-26T11:27:55.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 18: (DispatchQueue::entry()+0x60f) [0x7f21af79bcf9] 2024-07-26T11:27:55.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f21af8785a1] 2024-07-26T11:27:55.676 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 20: (Thread::entry_wrapper()+0x43) [0x7f21af625cdd] 2024-07-26T11:27:55.676 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 21: (Thread::_entry_func(void*)+0xd) [0x7f21af625cf9] 2024-07-26T11:27:55.676 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 22: /lib64/libc.so.6(+0x89c02) [0x7f21aec89c02] 2024-07-26T11:27:55.676 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f21aed0ec40] 2024-07-26T11:27:55.676 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 2024-07-26T11:27:55.277+0000 7f21a8a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:55.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: in thread 7f21a8a00640 thread_name:ms_dispatch 2024-07-26T11:27:55.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 2024-07-26T11:27:55.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:55.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f21aec3e6f0] 2024-07-26T11:27:55.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f21aec8b94c] 2024-07-26T11:27:55.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 3: raise() 2024-07-26T11:27:55.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 4: abort() 2024-07-26T11:27:55.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 5: /lib64/libc.so.6(+0x2871b) [0x7f21aec2871b] 2024-07-26T11:27:55.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 6: /lib64/libc.so.6(+0x37386) [0x7f21aec37386] 2024-07-26T11:27:55.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5612f59846b9] 2024-07-26T11:27:55.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5612f5b4280c] 2024-07-26T11:27:55.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5612f5a822ca] 2024-07-26T11:27:55.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5612f5a52753] 2024-07-26T11:27:55.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5612f5a53a1b] 2024-07-26T11:27:55.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5612f5a37752] 2024-07-26T11:27:55.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5612f5985a56] 2024-07-26T11:27:55.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5612f5989637] 2024-07-26T11:27:55.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5612f5989f5e] 2024-07-26T11:27:55.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5612f596d03a] 2024-07-26T11:27:55.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f21af79b407] 2024-07-26T11:27:55.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 18: (DispatchQueue::entry()+0x60f) [0x7f21af79bcf9] 2024-07-26T11:27:55.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f21af8785a1] 2024-07-26T11:27:55.679 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 20: (Thread::entry_wrapper()+0x43) [0x7f21af625cdd] 2024-07-26T11:27:55.679 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 21: (Thread::_entry_func(void*)+0xd) [0x7f21af625cf9] 2024-07-26T11:27:55.679 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 22: /lib64/libc.so.6(+0x89c02) [0x7f21aec89c02] 2024-07-26T11:27:55.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f21aed0ec40] 2024-07-26T11:27:55.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:55.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 2024-07-26T11:27:55.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 0> 2024-07-26T11:27:55.277+0000 7f21a8a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:55.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: in thread 7f21a8a00640 thread_name:ms_dispatch 2024-07-26T11:27:55.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 2024-07-26T11:27:55.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:55.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f21aec3e6f0] 2024-07-26T11:27:55.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f21aec8b94c] 2024-07-26T11:27:55.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 3: raise() 2024-07-26T11:27:55.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 4: abort() 2024-07-26T11:27:55.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 5: /lib64/libc.so.6(+0x2871b) [0x7f21aec2871b] 2024-07-26T11:27:55.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 6: /lib64/libc.so.6(+0x37386) [0x7f21aec37386] 2024-07-26T11:27:55.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5612f59846b9] 2024-07-26T11:27:55.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5612f5b4280c] 2024-07-26T11:27:55.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5612f5a822ca] 2024-07-26T11:27:55.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5612f5a52753] 2024-07-26T11:27:55.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5612f5a53a1b] 2024-07-26T11:27:55.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5612f5a37752] 2024-07-26T11:27:55.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5612f5985a56] 2024-07-26T11:27:55.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5612f5989637] 2024-07-26T11:27:55.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5612f5989f5e] 2024-07-26T11:27:55.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5612f596d03a] 2024-07-26T11:27:55.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f21af79b407] 2024-07-26T11:27:55.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 18: (DispatchQueue::entry()+0x60f) [0x7f21af79bcf9] 2024-07-26T11:27:55.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f21af8785a1] 2024-07-26T11:27:55.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 20: (Thread::entry_wrapper()+0x43) [0x7f21af625cdd] 2024-07-26T11:27:55.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 21: (Thread::_entry_func(void*)+0xd) [0x7f21af625cf9] 2024-07-26T11:27:55.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 22: /lib64/libc.so.6(+0x89c02) [0x7f21aec89c02] 2024-07-26T11:27:55.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f21aed0ec40] 2024-07-26T11:27:55.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:55.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 2024-07-26T11:27:55.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 0> 2024-07-26T11:27:55.277+0000 7f21a8a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:27:55.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: in thread 7f21a8a00640 thread_name:ms_dispatch 2024-07-26T11:27:55.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 2024-07-26T11:27:55.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:27:55.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f21aec3e6f0] 2024-07-26T11:27:55.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f21aec8b94c] 2024-07-26T11:27:55.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 3: raise() 2024-07-26T11:27:55.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 4: abort() 2024-07-26T11:27:55.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 5: /lib64/libc.so.6(+0x2871b) [0x7f21aec2871b] 2024-07-26T11:27:55.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 6: /lib64/libc.so.6(+0x37386) [0x7f21aec37386] 2024-07-26T11:27:55.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5612f59846b9] 2024-07-26T11:27:55.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5612f5b4280c] 2024-07-26T11:27:55.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5612f5a822ca] 2024-07-26T11:27:55.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5612f5a52753] 2024-07-26T11:27:55.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5612f5a53a1b] 2024-07-26T11:27:55.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5612f5a37752] 2024-07-26T11:27:55.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5612f5985a56] 2024-07-26T11:27:55.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5612f5989637] 2024-07-26T11:27:55.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5612f5989f5e] 2024-07-26T11:27:55.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5612f596d03a] 2024-07-26T11:27:55.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f21af79b407] 2024-07-26T11:27:55.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 18: (DispatchQueue::entry()+0x60f) [0x7f21af79bcf9] 2024-07-26T11:27:55.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f21af8785a1] 2024-07-26T11:27:55.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 20: (Thread::entry_wrapper()+0x43) [0x7f21af625cdd] 2024-07-26T11:27:55.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 21: (Thread::_entry_func(void*)+0xd) [0x7f21af625cf9] 2024-07-26T11:27:55.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 22: /lib64/libc.so.6(+0x89c02) [0x7f21aec89c02] 2024-07-26T11:27:55.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f21aed0ec40] 2024-07-26T11:27:55.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:27:55.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111448]: 2024-07-26T11:27:55.685 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:55 smithi134 podman[111489]: 2024-07-26 11:27:55.428506921 +0000 UTC m=+0.034222372 container died 55398d1cc7a5c8fb24033b140329ea150261cadfade5f5e24a4013368b5939e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-07-26T11:27:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:55 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/3050637565,v1:172.21.15.134:6841/3050637565] up:rejoin 2024-07-26T11:27:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:55 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:27:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:55 smithi189 ceph-mon[36790]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:27:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:55 smithi189 ceph-mon[36790]: pgmap v855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 46 KiB/s wr, 11 op/s 2024-07-26T11:27:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:55 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:27:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:55 smithi189 ceph-mon[36790]: Cluster is now healthy 2024-07-26T11:27:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:55 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/3050637565,v1:172.21.15.134:6841/3050637565] up:rejoin 2024-07-26T11:27:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:55 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:27:55.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:55 smithi193 ceph-mon[37473]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:27:55.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:55 smithi193 ceph-mon[37473]: pgmap v855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 46 KiB/s wr, 11 op/s 2024-07-26T11:27:55.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:55 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:27:55.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:55 smithi193 ceph-mon[37473]: Cluster is now healthy 2024-07-26T11:27:56.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:56 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/3050637565,v1:172.21.15.134:6841/3050637565] up:active 2024-07-26T11:27:56.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:56 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:27:56.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:56 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:27:56.673 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:56 smithi134 podman[111489]: 2024-07-26 11:27:56.421209136 +0000 UTC m=+1.026924637 container remove 55398d1cc7a5c8fb24033b140329ea150261cadfade5f5e24a4013368b5939e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0) 2024-07-26T11:27:56.673 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:56 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:27:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:56 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/3050637565,v1:172.21.15.134:6841/3050637565] up:active 2024-07-26T11:27:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:56 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:27:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:56 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:27:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:56 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/3050637565,v1:172.21.15.134:6841/3050637565] up:active 2024-07-26T11:27:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:56 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:27:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:56 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:27:57.173 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:56 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T11:27:57.173 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:27:56 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Consumed 1.648s CPU time. 2024-07-26T11:27:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:57 smithi134 ceph-mon[31966]: pgmap v856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 818 B/s wr, 11 op/s 2024-07-26T11:27:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:57 smithi189 ceph-mon[36790]: pgmap v856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 818 B/s wr, 11 op/s 2024-07-26T11:27:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:57 smithi193 ceph-mon[37473]: pgmap v856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 818 B/s wr, 11 op/s 2024-07-26T11:27:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:58.278+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:58.278+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:27:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:27:58.278+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:27:59.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:27:58 smithi193 ceph-mon[37473]: pgmap v857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 10 KiB/s rd, 4.9 KiB/s wr, 12 op/s 2024-07-26T11:27:59.275 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:27:59 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:27:59.046+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:27:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:27:58 smithi134 ceph-mon[31966]: pgmap v857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 10 KiB/s rd, 4.9 KiB/s wr, 12 op/s 2024-07-26T11:27:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:27:58 smithi189 ceph-mon[36790]: pgmap v857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 10 KiB/s rd, 4.9 KiB/s wr, 12 op/s 2024-07-26T11:28:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:00 smithi193 ceph-mon[37473]: pgmap v858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 17 KiB/s rd, 4.8 KiB/s wr, 12 op/s 2024-07-26T11:28:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:00 smithi134 ceph-mon[31966]: pgmap v858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 17 KiB/s rd, 4.8 KiB/s wr, 12 op/s 2024-07-26T11:28:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:00 smithi189 ceph-mon[36790]: pgmap v858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 17 KiB/s rd, 4.8 KiB/s wr, 12 op/s 2024-07-26T11:28:02.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:28:02.770+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:28:02.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:28:02.770+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:28:02.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:28:02.770+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:02 smithi134 ceph-mon[31966]: pgmap v859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 17 KiB/s rd, 4.1 KiB/s wr, 14 op/s 2024-07-26T11:28:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:02 smithi193 ceph-mon[37473]: pgmap v859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 17 KiB/s rd, 4.1 KiB/s wr, 14 op/s 2024-07-26T11:28:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:02 smithi189 ceph-mon[36790]: pgmap v859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 17 KiB/s rd, 4.1 KiB/s wr, 14 op/s 2024-07-26T11:28:05.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:04 smithi193 ceph-mon[37473]: pgmap v860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-07-26T11:28:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:04 smithi134 ceph-mon[31966]: pgmap v860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-07-26T11:28:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:04 smithi189 ceph-mon[36790]: pgmap v860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-07-26T11:28:07.202 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:06 smithi134 ceph-mon[31966]: pgmap v861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-07-26T11:28:07.203 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:07 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 2. 2024-07-26T11:28:07.203 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:07 smithi134 systemd[1]: Stopped Ceph mds.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:28:07.203 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:07 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Consumed 1.648s CPU time. 2024-07-26T11:28:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:06 smithi193 ceph-mon[37473]: pgmap v861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-07-26T11:28:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:06 smithi189 ceph-mon[36790]: pgmap v861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-07-26T11:28:07.559 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:07 smithi134 systemd[1]: Starting Ceph mds.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:28:07.911 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:07 smithi134 podman[111584]: 2024-07-26 11:28:07.461872799 +0000 UTC m=+0.031926477 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:28:07.911 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:07 smithi134 podman[111584]: 2024-07-26 11:28:07.621906099 +0000 UTC m=+0.191959799 container create 6ad269dc97502dd70f32935da643e1d09b1d6badf8d302379d25aaf66d098608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, ceph=True, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-07-26T11:28:08.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:08 smithi134 ceph-mon[31966]: daemon mds.a restarted 2024-07-26T11:28:08.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:08 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:28:08.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:08 smithi134 ceph-mon[31966]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T11:28:08.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:08 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/3173924711,v1:172.21.15.134:6841/3173924711] up:boot 2024-07-26T11:28:08.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:08 smithi134 ceph-mon[31966]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T11:28:08.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:08 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:28:08.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:28:08.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:08 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:28:08.174 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:07 smithi134 podman[111584]: 2024-07-26 11:28:07.909079357 +0000 UTC m=+0.479133094 container init 6ad269dc97502dd70f32935da643e1d09b1d6badf8d302379d25aaf66d098608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, ceph=True, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-07-26T11:28:08.174 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:07 smithi134 podman[111584]: 2024-07-26 11:28:07.918308644 +0000 UTC m=+0.488362350 container start 6ad269dc97502dd70f32935da643e1d09b1d6badf8d302379d25aaf66d098608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-07-26T11:28:08.174 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:08 smithi134 bash[111584]: 6ad269dc97502dd70f32935da643e1d09b1d6badf8d302379d25aaf66d098608 2024-07-26T11:28:08.174 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:08 smithi134 systemd[1]: Started Ceph mds.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:28:08.174 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:08 smithi134 ceph-mds[111603]: -- 172.21.15.134:0/3853094982 <== mon.0 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55df26733860 con 0x55df2a8d5a80 2024-07-26T11:28:08.174 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: starting mds.a at 2024-07-26T11:28:08.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:08 smithi189 ceph-mon[36790]: daemon mds.a restarted 2024-07-26T11:28:08.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:08 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:28:08.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:08 smithi189 ceph-mon[36790]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T11:28:08.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:08 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/3173924711,v1:172.21.15.134:6841/3173924711] up:boot 2024-07-26T11:28:08.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:08 smithi189 ceph-mon[36790]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T11:28:08.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:08 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:28:08.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:28:08.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:08 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:28:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:08 smithi193 ceph-mon[37473]: daemon mds.a restarted 2024-07-26T11:28:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:08 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:28:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:08 smithi193 ceph-mon[37473]: osdmap e108: 12 total, 12 up, 12 in 2024-07-26T11:28:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:08 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/3173924711,v1:172.21.15.134:6841/3173924711] up:boot 2024-07-26T11:28:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:08 smithi193 ceph-mon[37473]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T11:28:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:08 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:28:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:28:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:08 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:28:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:09 smithi134 ceph-mon[31966]: pgmap v863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T11:28:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:09 smithi189 ceph-mon[36790]: pgmap v863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T11:28:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:09 smithi193 ceph-mon[37473]: pgmap v863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 11 op/s 2024-07-26T11:28:10.372 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:10 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/1380616145,v1:172.21.15.189:6835/1380616145] up:resolve 2024-07-26T11:28:10.372 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:10 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:28:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:10 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/1380616145,v1:172.21.15.189:6835/1380616145] up:resolve 2024-07-26T11:28:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:10 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:28:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:10 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/1380616145,v1:172.21.15.189:6835/1380616145] up:resolve 2024-07-26T11:28:10.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:10 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:28:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:11 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:28:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:11 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/1380616145,v1:172.21.15.189:6835/1380616145] up:reconnect 2024-07-26T11:28:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:11 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:28:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:11 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999981 2024-07-26T11:28:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:11 smithi134 ceph-mon[31966]: pgmap v864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 0 B/s wr, 11 op/s 2024-07-26T11:28:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:11 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:28:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:11 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/1380616145,v1:172.21.15.189:6835/1380616145] up:reconnect 2024-07-26T11:28:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:11 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:28:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:11 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999981 2024-07-26T11:28:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:11 smithi189 ceph-mon[36790]: pgmap v864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 0 B/s wr, 11 op/s 2024-07-26T11:28:11.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:11 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:28:11.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:11 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/1380616145,v1:172.21.15.189:6835/1380616145] up:reconnect 2024-07-26T11:28:11.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:11 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:28:11.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:11 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999981 2024-07-26T11:28:11.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:11 smithi193 ceph-mon[37473]: pgmap v864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 0 B/s wr, 11 op/s 2024-07-26T11:28:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:12 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/1380616145,v1:172.21.15.189:6835/1380616145] up:rejoin 2024-07-26T11:28:12.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:12 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:28:12.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:12 smithi134 ceph-mon[31966]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:28:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/1380616145,v1:172.21.15.189:6835/1380616145] up:rejoin 2024-07-26T11:28:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:28:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-mon[36790]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:28:12.444 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:28:12.444 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: *** Caught signal (Aborted) ** 2024-07-26T11:28:12.444 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: in thread 7f7f677cf640 thread_name:ms_dispatch 2024-07-26T11:28:12.444 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:28:12.444 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7f6cba66f0] 2024-07-26T11:28:12.444 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7f6cbf394c] 2024-07-26T11:28:12.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 3: raise() 2024-07-26T11:28:12.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 4: abort() 2024-07-26T11:28:12.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7f6cb9071b] 2024-07-26T11:28:12.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 6: /lib64/libc.so.6(+0x37386) [0x7f7f6cb9f386] 2024-07-26T11:28:12.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559b1231f6b9] 2024-07-26T11:28:12.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559b124dd80c] 2024-07-26T11:28:12.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559b1241d2ca] 2024-07-26T11:28:12.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559b123ed753] 2024-07-26T11:28:12.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559b123eea1b] 2024-07-26T11:28:12.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559b123d2752] 2024-07-26T11:28:12.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559b12320a56] 2024-07-26T11:28:12.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559b12324637] 2024-07-26T11:28:12.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559b12324f5e] 2024-07-26T11:28:12.446 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559b1230803a] 2024-07-26T11:28:12.446 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7f6d473407] 2024-07-26T11:28:12.446 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 18: (DispatchQueue::entry()+0x60f) [0x7f7f6d473cf9] 2024-07-26T11:28:12.446 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f7f6d5505a1] 2024-07-26T11:28:12.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 20: (Thread::entry_wrapper()+0x43) [0x7f7f6d2fdcdd] 2024-07-26T11:28:12.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7f6d2fdcf9] 2024-07-26T11:28:12.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7f6cbf1c02] 2024-07-26T11:28:12.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7f6cc76c40] 2024-07-26T11:28:12.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 2024-07-26T11:28:12.142+0000 7f7f677cf640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:28:12.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: in thread 7f7f677cf640 thread_name:ms_dispatch 2024-07-26T11:28:12.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 2024-07-26T11:28:12.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:28:12.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7f6cba66f0] 2024-07-26T11:28:12.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7f6cbf394c] 2024-07-26T11:28:12.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 3: raise() 2024-07-26T11:28:12.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 4: abort() 2024-07-26T11:28:12.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7f6cb9071b] 2024-07-26T11:28:12.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 6: /lib64/libc.so.6(+0x37386) [0x7f7f6cb9f386] 2024-07-26T11:28:12.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559b1231f6b9] 2024-07-26T11:28:12.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559b124dd80c] 2024-07-26T11:28:12.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559b1241d2ca] 2024-07-26T11:28:12.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559b123ed753] 2024-07-26T11:28:12.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559b123eea1b] 2024-07-26T11:28:12.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559b123d2752] 2024-07-26T11:28:12.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559b12320a56] 2024-07-26T11:28:12.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559b12324637] 2024-07-26T11:28:12.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559b12324f5e] 2024-07-26T11:28:12.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559b1230803a] 2024-07-26T11:28:12.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7f6d473407] 2024-07-26T11:28:12.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 18: (DispatchQueue::entry()+0x60f) [0x7f7f6d473cf9] 2024-07-26T11:28:12.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f7f6d5505a1] 2024-07-26T11:28:12.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 20: (Thread::entry_wrapper()+0x43) [0x7f7f6d2fdcdd] 2024-07-26T11:28:12.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7f6d2fdcf9] 2024-07-26T11:28:12.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7f6cbf1c02] 2024-07-26T11:28:12.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7f6cc76c40] 2024-07-26T11:28:12.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:28:12.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 2024-07-26T11:28:12.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 0> 2024-07-26T11:28:12.142+0000 7f7f677cf640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:28:12.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: in thread 7f7f677cf640 thread_name:ms_dispatch 2024-07-26T11:28:12.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 2024-07-26T11:28:12.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:28:12.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7f6cba66f0] 2024-07-26T11:28:12.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7f6cbf394c] 2024-07-26T11:28:12.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 3: raise() 2024-07-26T11:28:12.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 4: abort() 2024-07-26T11:28:12.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7f6cb9071b] 2024-07-26T11:28:12.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 6: /lib64/libc.so.6(+0x37386) [0x7f7f6cb9f386] 2024-07-26T11:28:12.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559b1231f6b9] 2024-07-26T11:28:12.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559b124dd80c] 2024-07-26T11:28:12.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559b1241d2ca] 2024-07-26T11:28:12.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559b123ed753] 2024-07-26T11:28:12.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559b123eea1b] 2024-07-26T11:28:12.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559b123d2752] 2024-07-26T11:28:12.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559b12320a56] 2024-07-26T11:28:12.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559b12324637] 2024-07-26T11:28:12.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559b12324f5e] 2024-07-26T11:28:12.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559b1230803a] 2024-07-26T11:28:12.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7f6d473407] 2024-07-26T11:28:12.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 18: (DispatchQueue::entry()+0x60f) [0x7f7f6d473cf9] 2024-07-26T11:28:12.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f7f6d5505a1] 2024-07-26T11:28:12.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 20: (Thread::entry_wrapper()+0x43) [0x7f7f6d2fdcdd] 2024-07-26T11:28:12.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7f6d2fdcf9] 2024-07-26T11:28:12.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7f6cbf1c02] 2024-07-26T11:28:12.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7f6cc76c40] 2024-07-26T11:28:12.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:28:12.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 2024-07-26T11:28:12.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 0> 2024-07-26T11:28:12.142+0000 7f7f677cf640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:28:12.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: in thread 7f7f677cf640 thread_name:ms_dispatch 2024-07-26T11:28:12.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 2024-07-26T11:28:12.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:28:12.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f7f6cba66f0] 2024-07-26T11:28:12.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f7f6cbf394c] 2024-07-26T11:28:12.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 3: raise() 2024-07-26T11:28:12.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 4: abort() 2024-07-26T11:28:12.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 5: /lib64/libc.so.6(+0x2871b) [0x7f7f6cb9071b] 2024-07-26T11:28:12.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 6: /lib64/libc.so.6(+0x37386) [0x7f7f6cb9f386] 2024-07-26T11:28:12.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559b1231f6b9] 2024-07-26T11:28:12.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x559b124dd80c] 2024-07-26T11:28:12.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559b1241d2ca] 2024-07-26T11:28:12.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559b123ed753] 2024-07-26T11:28:12.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559b123eea1b] 2024-07-26T11:28:12.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559b123d2752] 2024-07-26T11:28:12.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559b12320a56] 2024-07-26T11:28:12.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559b12324637] 2024-07-26T11:28:12.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559b12324f5e] 2024-07-26T11:28:12.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559b1230803a] 2024-07-26T11:28:12.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f7f6d473407] 2024-07-26T11:28:12.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 18: (DispatchQueue::entry()+0x60f) [0x7f7f6d473cf9] 2024-07-26T11:28:12.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f7f6d5505a1] 2024-07-26T11:28:12.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 20: (Thread::entry_wrapper()+0x43) [0x7f7f6d2fdcdd] 2024-07-26T11:28:12.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 21: (Thread::_entry_func(void*)+0xd) [0x7f7f6d2fdcf9] 2024-07-26T11:28:12.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 22: /lib64/libc.so.6(+0x89c02) [0x7f7f6cbf1c02] 2024-07-26T11:28:12.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f7f6cc76c40] 2024-07-26T11:28:12.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:28:12.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72742]: 2024-07-26T11:28:12.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 podman[72781]: 2024-07-26 11:28:12.186207361 +0000 UTC m=+0.019509974 container died 81b2bf205de143449ee74cf3574f27acabdb4efc584bd312584b20e92367f843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-07-26T11:28:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:12 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/1380616145,v1:172.21.15.189:6835/1380616145] up:rejoin 2024-07-26T11:28:12.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:12 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:28:12.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:12 smithi193 ceph-mon[37473]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:28:13.155 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 podman[72781]: 2024-07-26 11:28:12.836567242 +0000 UTC m=+0.669869847 container remove 81b2bf205de143449ee74cf3574f27acabdb4efc584bd312584b20e92367f843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0) 2024-07-26T11:28:13.155 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:12 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:28:13.155 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:13 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T11:28:13.205 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:13 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:28:13.205 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:13 smithi134 ceph-mon[31966]: Cluster is now healthy 2024-07-26T11:28:13.205 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:13 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/1380616145,v1:172.21.15.189:6835/1380616145] up:active 2024-07-26T11:28:13.205 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:13 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:28:13.205 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:13 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:28:13.205 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:13 smithi134 ceph-mon[31966]: pgmap v865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.8 KiB/s wr, 11 op/s 2024-07-26T11:28:13.288 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:13.286+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:28:13.291 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:13.289+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T11:28:13.291 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:13.289+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:28:13.291 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:13.289+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:28:13.291 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:13.289+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:13 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:28:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:13 smithi189 ceph-mon[36790]: Cluster is now healthy 2024-07-26T11:28:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:13 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/1380616145,v1:172.21.15.189:6835/1380616145] up:active 2024-07-26T11:28:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:13 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:28:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:13 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:28:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:13 smithi189 ceph-mon[36790]: pgmap v865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.8 KiB/s wr, 11 op/s 2024-07-26T11:28:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:13 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:28:13.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:13 smithi193 ceph-mon[37473]: Cluster is now healthy 2024-07-26T11:28:13.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:13 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/1380616145,v1:172.21.15.189:6835/1380616145] up:active 2024-07-26T11:28:13.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:13 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:28:13.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:13 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:28:13.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:13 smithi193 ceph-mon[37473]: pgmap v865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 8.8 KiB/s rd, 3.8 KiB/s wr, 11 op/s 2024-07-26T11:28:14.524 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:28:14 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:28:14.047+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:28:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:14 smithi134 ceph-mon[31966]: pgmap v866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.9 KiB/s wr, 11 op/s 2024-07-26T11:28:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:14 smithi193 ceph-mon[37473]: pgmap v866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.9 KiB/s wr, 11 op/s 2024-07-26T11:28:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:14 smithi189 ceph-mon[36790]: pgmap v866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.9 KiB/s wr, 11 op/s 2024-07-26T11:28:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:16 smithi134 ceph-mon[31966]: pgmap v867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 16 KiB/s rd, 3.9 KiB/s wr, 11 op/s 2024-07-26T11:28:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:16 smithi193 ceph-mon[37473]: pgmap v867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 16 KiB/s rd, 3.9 KiB/s wr, 11 op/s 2024-07-26T11:28:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:16 smithi189 ceph-mon[36790]: pgmap v867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 16 KiB/s rd, 3.9 KiB/s wr, 11 op/s 2024-07-26T11:28:17.778 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:28:17.775+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:28:17.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:28:17.776+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-26T11:28:17.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:28:17.777+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:28:17.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:28:17.777+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:28:17.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:28:17.777+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:18 smithi193 ceph-mon[37473]: pgmap v868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.6 KiB/s wr, 10 op/s 2024-07-26T11:28:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:18 smithi134 ceph-mon[31966]: pgmap v868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.6 KiB/s wr, 10 op/s 2024-07-26T11:28:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:18 smithi189 ceph-mon[36790]: pgmap v868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.6 KiB/s wr, 10 op/s 2024-07-26T11:28:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:20 smithi134 ceph-mon[31966]: pgmap v869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-07-26T11:28:21.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:20 smithi189 ceph-mon[36790]: pgmap v869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-07-26T11:28:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:20 smithi193 ceph-mon[37473]: pgmap v869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 3.2 KiB/s wr, 10 op/s 2024-07-26T11:28:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:22 smithi193 ceph-mon[37473]: pgmap v870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-07-26T11:28:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:22 smithi134 ceph-mon[31966]: pgmap v870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-07-26T11:28:23.432 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:22 smithi189 ceph-mon[36790]: pgmap v870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-07-26T11:28:23.432 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:23 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 2. 2024-07-26T11:28:23.432 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:23 smithi189 systemd[1]: Stopped Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:28:23.432 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:23 smithi189 systemd[1]: Starting Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:28:23.691 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:23 smithi189 podman[72875]: 2024-07-26 11:28:23.333958606 +0000 UTC m=+0.017806665 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:28:23.691 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:23 smithi189 podman[72875]: 2024-07-26 11:28:23.445725429 +0000 UTC m=+0.129573529 container create 4e1f27651d0e2e646a6e032455f6db2a105557851c07dbb686504a6b7a97c055 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724) 2024-07-26T11:28:23.691 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:23 smithi189 podman[72875]: 2024-07-26 11:28:23.690141267 +0000 UTC m=+0.373989326 container init 4e1f27651d0e2e646a6e032455f6db2a105557851c07dbb686504a6b7a97c055 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-07-26T11:28:23.944 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:23 smithi189 podman[72875]: 2024-07-26 11:28:23.692944208 +0000 UTC m=+0.376792276 container start 4e1f27651d0e2e646a6e032455f6db2a105557851c07dbb686504a6b7a97c055 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-07-26T11:28:23.944 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:23 smithi189 ceph-mds[72894]: -- 172.21.15.189:0/3558178293 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563808d6b860 con 0x56380cf0d180 2024-07-26T11:28:23.944 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:23 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: starting mds.e at 2024-07-26T11:28:23.944 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:23 smithi189 bash[72875]: 4e1f27651d0e2e646a6e032455f6db2a105557851c07dbb686504a6b7a97c055 2024-07-26T11:28:23.945 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:23 smithi189 systemd[1]: Started Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:28:24.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:23 smithi193 ceph-mon[37473]: daemon mds.e restarted 2024-07-26T11:28:24.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:23 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:28:24.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:23 smithi193 ceph-mon[37473]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T11:28:24.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:23 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/425158378,v1:172.21.15.189:6835/425158378] up:boot 2024-07-26T11:28:24.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:23 smithi193 ceph-mon[37473]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:28:24.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:23 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:28:24.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:28:24.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:23 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:28:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:23 smithi134 ceph-mon[31966]: daemon mds.e restarted 2024-07-26T11:28:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:23 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:28:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:23 smithi134 ceph-mon[31966]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T11:28:24.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:23 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/425158378,v1:172.21.15.189:6835/425158378] up:boot 2024-07-26T11:28:24.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:23 smithi134 ceph-mon[31966]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:28:24.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:23 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:28:24.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:28:24.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:23 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:28:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:23 smithi189 ceph-mon[36790]: daemon mds.e restarted 2024-07-26T11:28:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:23 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:28:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:23 smithi189 ceph-mon[36790]: osdmap e109: 12 total, 12 up, 12 in 2024-07-26T11:28:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:23 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/425158378,v1:172.21.15.189:6835/425158378] up:boot 2024-07-26T11:28:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:23 smithi189 ceph-mon[36790]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:28:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:23 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:28:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:28:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:23 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:28:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:28:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:24 smithi193 ceph-mon[37473]: pgmap v872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-07-26T11:28:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:28:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:24 smithi134 ceph-mon[31966]: pgmap v872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-07-26T11:28:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:28:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:24 smithi189 ceph-mon[36790]: pgmap v872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-07-26T11:28:26.236 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:25 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/3173924711,v1:172.21.15.134:6841/3173924711] up:resolve 2024-07-26T11:28:26.236 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:25 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:28:26.236 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:28:26.236 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:28:26.236 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:28:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:25 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/3173924711,v1:172.21.15.134:6841/3173924711] up:resolve 2024-07-26T11:28:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:25 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:28:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:28:26.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:28:26.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:28:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:25 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/3173924711,v1:172.21.15.134:6841/3173924711] up:resolve 2024-07-26T11:28:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:25 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:28:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:28:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:28:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:28:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:27 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:28:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:27 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/3173924711,v1:172.21.15.134:6841/3173924711] up:reconnect 2024-07-26T11:28:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:27 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:28:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:27 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0119997 2024-07-26T11:28:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:27 smithi193 ceph-mon[37473]: pgmap v873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:28:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:27 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:28:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:27 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/3173924711,v1:172.21.15.134:6841/3173924711] up:reconnect 2024-07-26T11:28:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:27 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:28:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:27 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0119997 2024-07-26T11:28:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:27 smithi134 ceph-mon[31966]: pgmap v873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:28:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:27 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:28:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:27 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/3173924711,v1:172.21.15.134:6841/3173924711] up:reconnect 2024-07-26T11:28:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:27 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:28:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:27 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0119997 2024-07-26T11:28:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:27 smithi189 ceph-mon[36790]: pgmap v873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:28:28.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:28 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/3173924711,v1:172.21.15.134:6841/3173924711] up:rejoin 2024-07-26T11:28:28.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:28 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:28:28.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:28 smithi193 ceph-mon[37473]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:28:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:28.305+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:28:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:28.305+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:28:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:28.305+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/3173924711,v1:172.21.15.134:6841/3173924711] up:rejoin 2024-07-26T11:28:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:28:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-mon[31966]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:28:28.424 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:28:28.424 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: *** Caught signal (Aborted) ** 2024-07-26T11:28:28.424 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: in thread 7fb076a00640 thread_name:ms_dispatch 2024-07-26T11:28:28.424 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:28:28.424 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb07cc3e6f0] 2024-07-26T11:28:28.424 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb07cc8b94c] 2024-07-26T11:28:28.424 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 3: raise() 2024-07-26T11:28:28.424 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 4: abort() 2024-07-26T11:28:28.425 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb07cc2871b] 2024-07-26T11:28:28.425 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 6: /lib64/libc.so.6(+0x37386) [0x7fb07cc37386] 2024-07-26T11:28:28.425 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55df1e9996b9] 2024-07-26T11:28:28.425 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55df1eb5780c] 2024-07-26T11:28:28.425 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55df1ea972ca] 2024-07-26T11:28:28.425 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55df1ea67753] 2024-07-26T11:28:28.425 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55df1ea68a1b] 2024-07-26T11:28:28.425 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55df1ea4c752] 2024-07-26T11:28:28.425 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55df1e99aa56] 2024-07-26T11:28:28.425 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55df1e99e637] 2024-07-26T11:28:28.425 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55df1e99ef5e] 2024-07-26T11:28:28.425 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55df1e98203a] 2024-07-26T11:28:28.425 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb07d79b407] 2024-07-26T11:28:28.426 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 18: (DispatchQueue::entry()+0x60f) [0x7fb07d79bcf9] 2024-07-26T11:28:28.426 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb07d8785a1] 2024-07-26T11:28:28.426 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 20: (Thread::entry_wrapper()+0x43) [0x7fb07d625cdd] 2024-07-26T11:28:28.426 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb07d625cf9] 2024-07-26T11:28:28.426 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb07cc89c02] 2024-07-26T11:28:28.426 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb07cd0ec40] 2024-07-26T11:28:28.426 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 2024-07-26T11:28:28.024+0000 7fb076a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:28:28.426 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: in thread 7fb076a00640 thread_name:ms_dispatch 2024-07-26T11:28:28.427 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 2024-07-26T11:28:28.427 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:28:28.427 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb07cc3e6f0] 2024-07-26T11:28:28.427 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb07cc8b94c] 2024-07-26T11:28:28.427 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 3: raise() 2024-07-26T11:28:28.427 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 4: abort() 2024-07-26T11:28:28.427 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb07cc2871b] 2024-07-26T11:28:28.427 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 6: /lib64/libc.so.6(+0x37386) [0x7fb07cc37386] 2024-07-26T11:28:28.427 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55df1e9996b9] 2024-07-26T11:28:28.427 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55df1eb5780c] 2024-07-26T11:28:28.427 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55df1ea972ca] 2024-07-26T11:28:28.427 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55df1ea67753] 2024-07-26T11:28:28.427 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55df1ea68a1b] 2024-07-26T11:28:28.428 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55df1ea4c752] 2024-07-26T11:28:28.428 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55df1e99aa56] 2024-07-26T11:28:28.428 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55df1e99e637] 2024-07-26T11:28:28.428 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55df1e99ef5e] 2024-07-26T11:28:28.428 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55df1e98203a] 2024-07-26T11:28:28.428 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb07d79b407] 2024-07-26T11:28:28.428 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 18: (DispatchQueue::entry()+0x60f) [0x7fb07d79bcf9] 2024-07-26T11:28:28.428 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb07d8785a1] 2024-07-26T11:28:28.428 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 20: (Thread::entry_wrapper()+0x43) [0x7fb07d625cdd] 2024-07-26T11:28:28.428 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb07d625cf9] 2024-07-26T11:28:28.429 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb07cc89c02] 2024-07-26T11:28:28.429 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb07cd0ec40] 2024-07-26T11:28:28.429 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:28:28.429 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 2024-07-26T11:28:28.429 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 0> 2024-07-26T11:28:28.024+0000 7fb076a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:28:28.429 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: in thread 7fb076a00640 thread_name:ms_dispatch 2024-07-26T11:28:28.429 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 2024-07-26T11:28:28.429 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:28:28.429 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb07cc3e6f0] 2024-07-26T11:28:28.429 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb07cc8b94c] 2024-07-26T11:28:28.429 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 3: raise() 2024-07-26T11:28:28.429 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 4: abort() 2024-07-26T11:28:28.430 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb07cc2871b] 2024-07-26T11:28:28.430 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 6: /lib64/libc.so.6(+0x37386) [0x7fb07cc37386] 2024-07-26T11:28:28.430 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55df1e9996b9] 2024-07-26T11:28:28.430 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55df1eb5780c] 2024-07-26T11:28:28.430 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55df1ea972ca] 2024-07-26T11:28:28.430 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55df1ea67753] 2024-07-26T11:28:28.430 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55df1ea68a1b] 2024-07-26T11:28:28.430 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55df1ea4c752] 2024-07-26T11:28:28.430 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55df1e99aa56] 2024-07-26T11:28:28.430 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55df1e99e637] 2024-07-26T11:28:28.430 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55df1e99ef5e] 2024-07-26T11:28:28.430 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55df1e98203a] 2024-07-26T11:28:28.431 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb07d79b407] 2024-07-26T11:28:28.431 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 18: (DispatchQueue::entry()+0x60f) [0x7fb07d79bcf9] 2024-07-26T11:28:28.431 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb07d8785a1] 2024-07-26T11:28:28.431 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 20: (Thread::entry_wrapper()+0x43) [0x7fb07d625cdd] 2024-07-26T11:28:28.431 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb07d625cf9] 2024-07-26T11:28:28.431 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb07cc89c02] 2024-07-26T11:28:28.432 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb07cd0ec40] 2024-07-26T11:28:28.432 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:28:28.432 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 2024-07-26T11:28:28.432 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: -5> 2024-07-26T11:28:28.024+0000 7fb076a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:28:28.432 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: in thread 7fb076a00640 thread_name:ms_dispatch 2024-07-26T11:28:28.432 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 2024-07-26T11:28:28.432 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:28:28.432 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb07cc3e6f0] 2024-07-26T11:28:28.432 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb07cc8b94c] 2024-07-26T11:28:28.432 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 3: raise() 2024-07-26T11:28:28.432 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 4: abort() 2024-07-26T11:28:28.432 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb07cc2871b] 2024-07-26T11:28:28.433 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 6: /lib64/libc.so.6(+0x37386) [0x7fb07cc37386] 2024-07-26T11:28:28.433 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55df1e9996b9] 2024-07-26T11:28:28.433 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55df1eb5780c] 2024-07-26T11:28:28.433 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55df1ea972ca] 2024-07-26T11:28:28.433 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55df1ea67753] 2024-07-26T11:28:28.433 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55df1ea68a1b] 2024-07-26T11:28:28.433 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55df1ea4c752] 2024-07-26T11:28:28.433 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55df1e99aa56] 2024-07-26T11:28:28.433 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55df1e99e637] 2024-07-26T11:28:28.433 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55df1e99ef5e] 2024-07-26T11:28:28.433 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55df1e98203a] 2024-07-26T11:28:28.433 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb07d79b407] 2024-07-26T11:28:28.433 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 18: (DispatchQueue::entry()+0x60f) [0x7fb07d79bcf9] 2024-07-26T11:28:28.434 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb07d8785a1] 2024-07-26T11:28:28.434 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 20: (Thread::entry_wrapper()+0x43) [0x7fb07d625cdd] 2024-07-26T11:28:28.434 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb07d625cf9] 2024-07-26T11:28:28.434 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb07cc89c02] 2024-07-26T11:28:28.434 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb07cd0ec40] 2024-07-26T11:28:28.434 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:28:28.434 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111599]: 2024-07-26T11:28:28.434 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:28 smithi134 podman[111765]: 2024-07-26 11:28:28.194753412 +0000 UTC m=+0.036181055 container died 6ad269dc97502dd70f32935da643e1d09b1d6badf8d302379d25aaf66d098608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.license=GPLv2) 2024-07-26T11:28:28.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:28 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/3173924711,v1:172.21.15.134:6841/3173924711] up:rejoin 2024-07-26T11:28:28.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:28 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:28:28.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:28 smithi189 ceph-mon[36790]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:28:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:29 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:28:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:29 smithi134 ceph-mon[31966]: Cluster is now healthy 2024-07-26T11:28:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:29 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/3173924711,v1:172.21.15.134:6841/3173924711] up:active 2024-07-26T11:28:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:29 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:28:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:29 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:28:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:29 smithi134 ceph-mon[31966]: pgmap v874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 7.8 KiB/s rd, 3.0 KiB/s wr, 10 op/s 2024-07-26T11:28:29.275 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:28:29 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:28:29.048+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:28:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:29 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:28:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:29 smithi193 ceph-mon[37473]: Cluster is now healthy 2024-07-26T11:28:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:29 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/3173924711,v1:172.21.15.134:6841/3173924711] up:active 2024-07-26T11:28:29.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:29 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:28:29.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:29 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:28:29.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:29 smithi193 ceph-mon[37473]: pgmap v874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 7.8 KiB/s rd, 3.0 KiB/s wr, 10 op/s 2024-07-26T11:28:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:29 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:28:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:29 smithi189 ceph-mon[36790]: Cluster is now healthy 2024-07-26T11:28:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:29 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/3173924711,v1:172.21.15.134:6841/3173924711] up:active 2024-07-26T11:28:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:29 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:28:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:29 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:28:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:29 smithi189 ceph-mon[36790]: pgmap v874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 7.8 KiB/s rd, 3.0 KiB/s wr, 10 op/s 2024-07-26T11:28:29.586 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:29 smithi134 podman[111765]: 2024-07-26 11:28:29.254075851 +0000 UTC m=+1.095503529 container remove 6ad269dc97502dd70f32935da643e1d09b1d6badf8d302379d25aaf66d098608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-07-26T11:28:29.586 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:29 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:28:29.923 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:29 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T11:28:29.923 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:29 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Consumed 1.632s CPU time. 2024-07-26T11:28:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:30 smithi193 ceph-mon[37473]: pgmap v875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-07-26T11:28:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:30 smithi134 ceph-mon[31966]: pgmap v875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-07-26T11:28:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:30 smithi189 ceph-mon[36790]: pgmap v875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-07-26T11:28:32.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:28:32.792+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:28:32.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:28:32.792+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:28:32.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:28:32.792+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:32 smithi134 ceph-mon[31966]: pgmap v876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 16 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-07-26T11:28:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:32 smithi193 ceph-mon[37473]: pgmap v876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 16 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-07-26T11:28:33.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:32 smithi189 ceph-mon[36790]: pgmap v876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 16 KiB/s rd, 3.1 KiB/s wr, 11 op/s 2024-07-26T11:28:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:34 smithi193 ceph-mon[37473]: pgmap v877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-07-26T11:28:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:34 smithi134 ceph-mon[31966]: pgmap v877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-07-26T11:28:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:34 smithi189 ceph-mon[36790]: pgmap v877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-07-26T11:28:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:36 smithi193 ceph-mon[37473]: pgmap v878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 2.6 KiB/s wr, 10 op/s 2024-07-26T11:28:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:36 smithi134 ceph-mon[31966]: pgmap v878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 2.6 KiB/s wr, 10 op/s 2024-07-26T11:28:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:36 smithi189 ceph-mon[36790]: pgmap v878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 2.6 KiB/s wr, 10 op/s 2024-07-26T11:28:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:39 smithi193 ceph-mon[37473]: pgmap v879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-07-26T11:28:40.320 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:39 smithi134 ceph-mon[31966]: pgmap v879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-07-26T11:28:40.320 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:39 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 3. 2024-07-26T11:28:40.320 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:39 smithi134 systemd[1]: Stopped Ceph mds.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:28:40.321 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:39 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Consumed 1.632s CPU time. 2024-07-26T11:28:40.321 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:39 smithi134 systemd[1]: Starting Ceph mds.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:28:40.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:39 smithi189 ceph-mon[36790]: pgmap v879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 2.6 KiB/s wr, 12 op/s 2024-07-26T11:28:40.574 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:40 smithi134 podman[111859]: 2024-07-26 11:28:40.228664317 +0000 UTC m=+0.039016301 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:28:40.574 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:40 smithi134 podman[111859]: 2024-07-26 11:28:40.387948668 +0000 UTC m=+0.198300625 container create 373d2713da3755148f9b6794c7897d8b6e0d710e448cca8400741db1af865bab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-07-26T11:28:40.966 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:40 smithi134 podman[111859]: 2024-07-26 11:28:40.674915503 +0000 UTC m=+0.485267490 container init 373d2713da3755148f9b6794c7897d8b6e0d710e448cca8400741db1af865bab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-07-26T11:28:40.966 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:40 smithi134 podman[111859]: 2024-07-26 11:28:40.683964935 +0000 UTC m=+0.494316865 container start 373d2713da3755148f9b6794c7897d8b6e0d710e448cca8400741db1af865bab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-07-26T11:28:40.966 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:40 smithi134 bash[111859]: 373d2713da3755148f9b6794c7897d8b6e0d710e448cca8400741db1af865bab 2024-07-26T11:28:40.966 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:40 smithi134 systemd[1]: Started Ceph mds.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:28:40.966 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:40 smithi134 ceph-mds[111877]: -- 172.21.15.134:0/2539494557 <== mon.0 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558494707860 con 0x5584988a9600 2024-07-26T11:28:40.966 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:28:40 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: starting mds.a at 2024-07-26T11:28:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:40 smithi193 ceph-mon[37473]: daemon mds.a restarted 2024-07-26T11:28:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:40 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:28:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:40 smithi193 ceph-mon[37473]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T11:28:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:40 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/3576163546,v1:172.21.15.134:6841/3576163546] up:boot 2024-07-26T11:28:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:40 smithi193 ceph-mon[37473]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T11:28:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:40 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:28:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:28:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:40 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:28:41.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:40 smithi134 ceph-mon[31966]: daemon mds.a restarted 2024-07-26T11:28:41.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:40 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:28:41.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:40 smithi134 ceph-mon[31966]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T11:28:41.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:40 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/3576163546,v1:172.21.15.134:6841/3576163546] up:boot 2024-07-26T11:28:41.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:40 smithi134 ceph-mon[31966]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T11:28:41.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:40 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:28:41.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:28:41.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:40 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:28:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:40 smithi189 ceph-mon[36790]: daemon mds.a restarted 2024-07-26T11:28:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:40 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:28:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:40 smithi189 ceph-mon[36790]: osdmap e110: 12 total, 12 up, 12 in 2024-07-26T11:28:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:40 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/3576163546,v1:172.21.15.134:6841/3576163546] up:boot 2024-07-26T11:28:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:40 smithi189 ceph-mon[36790]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T11:28:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:40 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:28:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:28:41.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:40 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:28:43.079 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:42 smithi134 ceph-mon[31966]: pgmap v881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:28:43.079 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:42 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/425158378,v1:172.21.15.189:6835/425158378] up:resolve 2024-07-26T11:28:43.079 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:42 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:28:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:42 smithi189 ceph-mon[36790]: pgmap v881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:28:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:42 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/425158378,v1:172.21.15.189:6835/425158378] up:resolve 2024-07-26T11:28:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:42 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:28:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:42 smithi193 ceph-mon[37473]: pgmap v881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:28:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:42 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/425158378,v1:172.21.15.189:6835/425158378] up:resolve 2024-07-26T11:28:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:42 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:28:43.313 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:43.311+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:28:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:43.312+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T11:28:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:43.312+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:28:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:43.312+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:28:43.314 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:43.312+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:43 smithi193 ceph-mon[37473]: pgmap v882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:28:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:43 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:28:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:43 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/425158378,v1:172.21.15.189:6835/425158378] up:reconnect 2024-07-26T11:28:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:43 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:28:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:43 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999981 2024-07-26T11:28:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:43 smithi134 ceph-mon[31966]: pgmap v882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:28:44.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:43 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:28:44.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:43 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/425158378,v1:172.21.15.189:6835/425158378] up:reconnect 2024-07-26T11:28:44.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:43 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:28:44.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:43 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999981 2024-07-26T11:28:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:43 smithi189 ceph-mon[36790]: pgmap v882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:28:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:43 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:28:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:43 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/425158378,v1:172.21.15.189:6835/425158378] up:reconnect 2024-07-26T11:28:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:43 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:28:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:43 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999981 2024-07-26T11:28:44.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:28:44.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: *** Caught signal (Aborted) ** 2024-07-26T11:28:44.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: in thread 7fa971add640 thread_name:ms_dispatch 2024-07-26T11:28:44.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:28:44.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa976eb46f0] 2024-07-26T11:28:44.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa976f0194c] 2024-07-26T11:28:44.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 3: raise() 2024-07-26T11:28:44.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 4: abort() 2024-07-26T11:28:44.445 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa976e9e71b] 2024-07-26T11:28:44.446 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 6: /lib64/libc.so.6(+0x37386) [0x7fa976ead386] 2024-07-26T11:28:44.446 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5638068ff6b9] 2024-07-26T11:28:44.446 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563806abd80c] 2024-07-26T11:28:44.446 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5638069fd2ca] 2024-07-26T11:28:44.446 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5638069cd753] 2024-07-26T11:28:44.446 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5638069cea1b] 2024-07-26T11:28:44.446 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5638069b2752] 2024-07-26T11:28:44.446 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563806900a56] 2024-07-26T11:28:44.446 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563806904637] 2024-07-26T11:28:44.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563806904f5e] 2024-07-26T11:28:44.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5638068e803a] 2024-07-26T11:28:44.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa977781407] 2024-07-26T11:28:44.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 18: (DispatchQueue::entry()+0x60f) [0x7fa977781cf9] 2024-07-26T11:28:44.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa97785e5a1] 2024-07-26T11:28:44.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 20: (Thread::entry_wrapper()+0x43) [0x7fa97760bcdd] 2024-07-26T11:28:44.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa97760bcf9] 2024-07-26T11:28:44.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa976effc02] 2024-07-26T11:28:44.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa976f84c40] 2024-07-26T11:28:44.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 2024-07-26T11:28:44.145+0000 7fa971add640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:28:44.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: in thread 7fa971add640 thread_name:ms_dispatch 2024-07-26T11:28:44.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 2024-07-26T11:28:44.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:28:44.447 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa976eb46f0] 2024-07-26T11:28:44.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa976f0194c] 2024-07-26T11:28:44.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 3: raise() 2024-07-26T11:28:44.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 4: abort() 2024-07-26T11:28:44.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa976e9e71b] 2024-07-26T11:28:44.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 6: /lib64/libc.so.6(+0x37386) [0x7fa976ead386] 2024-07-26T11:28:44.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5638068ff6b9] 2024-07-26T11:28:44.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563806abd80c] 2024-07-26T11:28:44.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5638069fd2ca] 2024-07-26T11:28:44.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5638069cd753] 2024-07-26T11:28:44.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5638069cea1b] 2024-07-26T11:28:44.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5638069b2752] 2024-07-26T11:28:44.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563806900a56] 2024-07-26T11:28:44.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563806904637] 2024-07-26T11:28:44.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563806904f5e] 2024-07-26T11:28:44.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5638068e803a] 2024-07-26T11:28:44.448 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa977781407] 2024-07-26T11:28:44.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 18: (DispatchQueue::entry()+0x60f) [0x7fa977781cf9] 2024-07-26T11:28:44.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa97785e5a1] 2024-07-26T11:28:44.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 20: (Thread::entry_wrapper()+0x43) [0x7fa97760bcdd] 2024-07-26T11:28:44.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa97760bcf9] 2024-07-26T11:28:44.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa976effc02] 2024-07-26T11:28:44.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa976f84c40] 2024-07-26T11:28:44.449 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:28:44.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 2024-07-26T11:28:44.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 0> 2024-07-26T11:28:44.145+0000 7fa971add640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:28:44.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: in thread 7fa971add640 thread_name:ms_dispatch 2024-07-26T11:28:44.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 2024-07-26T11:28:44.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:28:44.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa976eb46f0] 2024-07-26T11:28:44.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa976f0194c] 2024-07-26T11:28:44.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 3: raise() 2024-07-26T11:28:44.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 4: abort() 2024-07-26T11:28:44.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa976e9e71b] 2024-07-26T11:28:44.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 6: /lib64/libc.so.6(+0x37386) [0x7fa976ead386] 2024-07-26T11:28:44.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5638068ff6b9] 2024-07-26T11:28:44.450 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563806abd80c] 2024-07-26T11:28:44.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5638069fd2ca] 2024-07-26T11:28:44.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5638069cd753] 2024-07-26T11:28:44.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5638069cea1b] 2024-07-26T11:28:44.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5638069b2752] 2024-07-26T11:28:44.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563806900a56] 2024-07-26T11:28:44.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563806904637] 2024-07-26T11:28:44.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563806904f5e] 2024-07-26T11:28:44.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5638068e803a] 2024-07-26T11:28:44.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa977781407] 2024-07-26T11:28:44.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 18: (DispatchQueue::entry()+0x60f) [0x7fa977781cf9] 2024-07-26T11:28:44.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa97785e5a1] 2024-07-26T11:28:44.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 20: (Thread::entry_wrapper()+0x43) [0x7fa97760bcdd] 2024-07-26T11:28:44.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa97760bcf9] 2024-07-26T11:28:44.451 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa976effc02] 2024-07-26T11:28:44.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa976f84c40] 2024-07-26T11:28:44.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:28:44.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 2024-07-26T11:28:44.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 0> 2024-07-26T11:28:44.145+0000 7fa971add640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:28:44.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: in thread 7fa971add640 thread_name:ms_dispatch 2024-07-26T11:28:44.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 2024-07-26T11:28:44.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:28:44.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa976eb46f0] 2024-07-26T11:28:44.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa976f0194c] 2024-07-26T11:28:44.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 3: raise() 2024-07-26T11:28:44.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 4: abort() 2024-07-26T11:28:44.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa976e9e71b] 2024-07-26T11:28:44.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 6: /lib64/libc.so.6(+0x37386) [0x7fa976ead386] 2024-07-26T11:28:44.452 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5638068ff6b9] 2024-07-26T11:28:44.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563806abd80c] 2024-07-26T11:28:44.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5638069fd2ca] 2024-07-26T11:28:44.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5638069cd753] 2024-07-26T11:28:44.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5638069cea1b] 2024-07-26T11:28:44.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5638069b2752] 2024-07-26T11:28:44.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563806900a56] 2024-07-26T11:28:44.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563806904637] 2024-07-26T11:28:44.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563806904f5e] 2024-07-26T11:28:44.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5638068e803a] 2024-07-26T11:28:44.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa977781407] 2024-07-26T11:28:44.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 18: (DispatchQueue::entry()+0x60f) [0x7fa977781cf9] 2024-07-26T11:28:44.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa97785e5a1] 2024-07-26T11:28:44.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 20: (Thread::entry_wrapper()+0x43) [0x7fa97760bcdd] 2024-07-26T11:28:44.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa97760bcf9] 2024-07-26T11:28:44.453 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa976effc02] 2024-07-26T11:28:44.454 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa976f84c40] 2024-07-26T11:28:44.454 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:28:44.454 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[72890]: 2024-07-26T11:28:44.454 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 podman[73058]: 2024-07-26 11:28:44.189608539 +0000 UTC m=+0.020803508 container died 4e1f27651d0e2e646a6e032455f6db2a105557851c07dbb686504a6b7a97c055 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:28:45.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/425158378,v1:172.21.15.189:6835/425158378] up:rejoin 2024-07-26T11:28:45.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:28:45.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-mon[36790]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:28:45.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:28:45.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-mon[36790]: Cluster is now healthy 2024-07-26T11:28:45.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/425158378,v1:172.21.15.189:6835/425158378] up:active 2024-07-26T11:28:45.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:44 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:28:45.105 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 podman[73058]: 2024-07-26 11:28:44.853222299 +0000 UTC m=+0.684417270 container remove 4e1f27651d0e2e646a6e032455f6db2a105557851c07dbb686504a6b7a97c055 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:28:45.106 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:44 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:28:45.106 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:45 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T11:28:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:44 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/425158378,v1:172.21.15.189:6835/425158378] up:rejoin 2024-07-26T11:28:45.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:44 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:28:45.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:44 smithi193 ceph-mon[37473]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:28:45.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:44 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:28:45.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:44 smithi193 ceph-mon[37473]: Cluster is now healthy 2024-07-26T11:28:45.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:44 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/425158378,v1:172.21.15.189:6835/425158378] up:active 2024-07-26T11:28:45.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:44 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:28:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:44 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/425158378,v1:172.21.15.189:6835/425158378] up:rejoin 2024-07-26T11:28:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:44 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:28:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:44 smithi134 ceph-mon[31966]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:28:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:44 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:28:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:44 smithi134 ceph-mon[31966]: Cluster is now healthy 2024-07-26T11:28:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:44 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/425158378,v1:172.21.15.189:6835/425158378] up:active 2024-07-26T11:28:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:44 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:28:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:46 smithi134 ceph-mon[31966]: pgmap v883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2024-07-26T11:28:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:46 smithi189 ceph-mon[36790]: pgmap v883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2024-07-26T11:28:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:46 smithi193 ceph-mon[37473]: pgmap v883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2024-07-26T11:28:47.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:28:47.800+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:28:47.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:28:47.802+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-26T11:28:47.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:28:47.802+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:28:47.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:28:47.802+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:28:47.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:28:47.803+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:48 smithi134 ceph-mon[31966]: pgmap v884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:28:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:48 smithi189 ceph-mon[36790]: pgmap v884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:28:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:48 smithi193 ceph-mon[37473]: pgmap v884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:28:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:50 smithi134 ceph-mon[31966]: pgmap v885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 8.1 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-07-26T11:28:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:50 smithi189 ceph-mon[36790]: pgmap v885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 8.1 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-07-26T11:28:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:50 smithi193 ceph-mon[37473]: pgmap v885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 8.1 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-07-26T11:28:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:52 smithi134 ceph-mon[31966]: pgmap v886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-07-26T11:28:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:52 smithi189 ceph-mon[36790]: pgmap v886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-07-26T11:28:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:52 smithi193 ceph-mon[37473]: pgmap v886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 15 KiB/s rd, 2.3 KiB/s wr, 11 op/s 2024-07-26T11:28:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:54 smithi134 ceph-mon[31966]: pgmap v887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-07-26T11:28:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:54 smithi189 ceph-mon[36790]: pgmap v887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-07-26T11:28:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:54 smithi193 ceph-mon[37473]: pgmap v887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-07-26T11:28:55.471 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:55 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 3. 2024-07-26T11:28:55.472 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:55 smithi189 systemd[1]: Stopped Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:28:55.472 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:55 smithi189 systemd[1]: Starting Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:28:55.472 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:55 smithi189 podman[73163]: 2024-07-26 11:28:55.330483342 +0000 UTC m=+0.017217985 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:28:55.775 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:55 smithi189 podman[73163]: 2024-07-26 11:28:55.470826788 +0000 UTC m=+0.157561420 container create 46c6529c2824e8dc1221088c1081862da79e50c9655f2452eac60af918081c83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-07-26T11:28:55.775 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:55 smithi189 podman[73163]: 2024-07-26 11:28:55.773874195 +0000 UTC m=+0.460608843 container init 46c6529c2824e8dc1221088c1081862da79e50c9655f2452eac60af918081c83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:28:56.178 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:55 smithi189 podman[73163]: 2024-07-26 11:28:55.776684655 +0000 UTC m=+0.463419296 container start 46c6529c2824e8dc1221088c1081862da79e50c9655f2452eac60af918081c83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, org.label-schema.build-date=20240724, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:28:56.178 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:55 smithi189 ceph-mds[73181]: -- 172.21.15.189:0/2428452245 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55913966d860 con 0x55913d80fa80 2024-07-26T11:28:56.178 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:55 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: starting mds.e at 2024-07-26T11:28:56.178 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:55 smithi189 bash[73163]: 46c6529c2824e8dc1221088c1081862da79e50c9655f2452eac60af918081c83 2024-07-26T11:28:56.179 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:28:55 smithi189 systemd[1]: Started Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:28:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:56 smithi189 ceph-mon[36790]: pgmap v888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-07-26T11:28:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:56 smithi189 ceph-mon[36790]: daemon mds.e restarted 2024-07-26T11:28:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:56 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:28:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:56 smithi189 ceph-mon[36790]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T11:28:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:56 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/638423294,v1:172.21.15.189:6835/638423294] up:boot 2024-07-26T11:28:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:56 smithi189 ceph-mon[36790]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:28:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:56 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:28:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:28:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:56 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:28:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:56 smithi193 ceph-mon[37473]: pgmap v888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-07-26T11:28:56.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:56 smithi193 ceph-mon[37473]: daemon mds.e restarted 2024-07-26T11:28:56.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:56 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:28:56.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:56 smithi193 ceph-mon[37473]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T11:28:56.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:56 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/638423294,v1:172.21.15.189:6835/638423294] up:boot 2024-07-26T11:28:56.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:56 smithi193 ceph-mon[37473]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:28:56.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:56 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:28:56.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:28:56.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:56 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:28:56.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:56 smithi134 ceph-mon[31966]: pgmap v888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 1.9 KiB/s wr, 10 op/s 2024-07-26T11:28:56.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:56 smithi134 ceph-mon[31966]: daemon mds.e restarted 2024-07-26T11:28:56.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:56 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:28:56.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:56 smithi134 ceph-mon[31966]: osdmap e111: 12 total, 12 up, 12 in 2024-07-26T11:28:56.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:56 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/638423294,v1:172.21.15.189:6835/638423294] up:boot 2024-07-26T11:28:56.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:56 smithi134 ceph-mon[31966]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:28:56.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:56 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:28:56.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:28:56.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:56 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:28:58.330 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:58.328+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:28:58.331 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:58.330+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T11:28:58.332 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:58.330+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:28:58.332 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:58.330+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:28:58.332 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:28:58.330+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:28:58.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:58 smithi134 ceph-mon[31966]: pgmap v890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-07-26T11:28:58.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:58 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/3576163546,v1:172.21.15.134:6841/3576163546] up:resolve 2024-07-26T11:28:58.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:58 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:28:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:58 smithi193 ceph-mon[37473]: pgmap v890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-07-26T11:28:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:58 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/3576163546,v1:172.21.15.134:6841/3576163546] up:resolve 2024-07-26T11:28:58.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:58 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:28:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:58 smithi189 ceph-mon[36790]: pgmap v890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 2.2 KiB/s wr, 10 op/s 2024-07-26T11:28:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:58 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/3576163546,v1:172.21.15.134:6841/3576163546] up:resolve 2024-07-26T11:28:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:58 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:28:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:59 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00199996 2024-07-26T11:28:59.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:59 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/3576163546,v1:172.21.15.134:6841/3576163546] up:reconnect 2024-07-26T11:28:59.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:59 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:28:59.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:28:59 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0119997 2024-07-26T11:28:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:59 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00199996 2024-07-26T11:28:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:59 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/3576163546,v1:172.21.15.134:6841/3576163546] up:reconnect 2024-07-26T11:28:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:59 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:28:59.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:28:59 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0119997 2024-07-26T11:28:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:59 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00199996 2024-07-26T11:28:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:59 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/3576163546,v1:172.21.15.134:6841/3576163546] up:reconnect 2024-07-26T11:28:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:59 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:28:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:28:59 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0119997 2024-07-26T11:29:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:00 smithi193 ceph-mon[37473]: pgmap v891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 10 op/s 2024-07-26T11:29:00.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:00 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/3576163546,v1:172.21.15.134:6841/3576163546] up:rejoin 2024-07-26T11:29:00.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:00 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:29:00.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:00 smithi193 ceph-mon[37473]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:29:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-mon[31966]: pgmap v891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 10 op/s 2024-07-26T11:29:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/3576163546,v1:172.21.15.134:6841/3576163546] up:rejoin 2024-07-26T11:29:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:29:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-mon[31966]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:29:00.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:29:00.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: *** Caught signal (Aborted) ** 2024-07-26T11:29:00.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: in thread 7f56fd800640 thread_name:ms_dispatch 2024-07-26T11:29:00.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:29:00.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5703a3e6f0] 2024-07-26T11:29:00.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5703a8b94c] 2024-07-26T11:29:00.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 3: raise() 2024-07-26T11:29:00.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 4: abort() 2024-07-26T11:29:00.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5703a2871b] 2024-07-26T11:29:00.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 6: /lib64/libc.so.6(+0x37386) [0x7f5703a37386] 2024-07-26T11:29:00.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55849095e6b9] 2024-07-26T11:29:00.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558490b1c80c] 2024-07-26T11:29:00.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558490a5c2ca] 2024-07-26T11:29:00.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558490a2c753] 2024-07-26T11:29:00.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558490a2da1b] 2024-07-26T11:29:00.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558490a11752] 2024-07-26T11:29:00.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55849095fa56] 2024-07-26T11:29:00.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558490963637] 2024-07-26T11:29:00.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558490963f5e] 2024-07-26T11:29:00.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55849094703a] 2024-07-26T11:29:00.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f570459b407] 2024-07-26T11:29:00.675 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 18: (DispatchQueue::entry()+0x60f) [0x7f570459bcf9] 2024-07-26T11:29:00.676 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f57046785a1] 2024-07-26T11:29:00.676 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 20: (Thread::entry_wrapper()+0x43) [0x7f5704425cdd] 2024-07-26T11:29:00.676 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5704425cf9] 2024-07-26T11:29:00.676 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5703a89c02] 2024-07-26T11:29:00.676 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5703b0ec40] 2024-07-26T11:29:00.676 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 2024-07-26T11:29:00.230+0000 7f56fd800640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:29:00.676 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: in thread 7f56fd800640 thread_name:ms_dispatch 2024-07-26T11:29:00.676 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 2024-07-26T11:29:00.676 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:29:00.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5703a3e6f0] 2024-07-26T11:29:00.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5703a8b94c] 2024-07-26T11:29:00.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 3: raise() 2024-07-26T11:29:00.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 4: abort() 2024-07-26T11:29:00.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5703a2871b] 2024-07-26T11:29:00.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 6: /lib64/libc.so.6(+0x37386) [0x7f5703a37386] 2024-07-26T11:29:00.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55849095e6b9] 2024-07-26T11:29:00.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558490b1c80c] 2024-07-26T11:29:00.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558490a5c2ca] 2024-07-26T11:29:00.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558490a2c753] 2024-07-26T11:29:00.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558490a2da1b] 2024-07-26T11:29:00.677 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558490a11752] 2024-07-26T11:29:00.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55849095fa56] 2024-07-26T11:29:00.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558490963637] 2024-07-26T11:29:00.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558490963f5e] 2024-07-26T11:29:00.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55849094703a] 2024-07-26T11:29:00.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f570459b407] 2024-07-26T11:29:00.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 18: (DispatchQueue::entry()+0x60f) [0x7f570459bcf9] 2024-07-26T11:29:00.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f57046785a1] 2024-07-26T11:29:00.678 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 20: (Thread::entry_wrapper()+0x43) [0x7f5704425cdd] 2024-07-26T11:29:00.679 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5704425cf9] 2024-07-26T11:29:00.679 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5703a89c02] 2024-07-26T11:29:00.679 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5703b0ec40] 2024-07-26T11:29:00.679 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:29:00.679 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 2024-07-26T11:29:00.679 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 0> 2024-07-26T11:29:00.230+0000 7f56fd800640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:29:00.679 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: in thread 7f56fd800640 thread_name:ms_dispatch 2024-07-26T11:29:00.679 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 2024-07-26T11:29:00.679 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:29:00.679 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5703a3e6f0] 2024-07-26T11:29:00.679 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5703a8b94c] 2024-07-26T11:29:00.679 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 3: raise() 2024-07-26T11:29:00.679 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 4: abort() 2024-07-26T11:29:00.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5703a2871b] 2024-07-26T11:29:00.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 6: /lib64/libc.so.6(+0x37386) [0x7f5703a37386] 2024-07-26T11:29:00.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55849095e6b9] 2024-07-26T11:29:00.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558490b1c80c] 2024-07-26T11:29:00.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558490a5c2ca] 2024-07-26T11:29:00.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558490a2c753] 2024-07-26T11:29:00.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558490a2da1b] 2024-07-26T11:29:00.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558490a11752] 2024-07-26T11:29:00.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55849095fa56] 2024-07-26T11:29:00.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558490963637] 2024-07-26T11:29:00.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558490963f5e] 2024-07-26T11:29:00.680 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55849094703a] 2024-07-26T11:29:00.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f570459b407] 2024-07-26T11:29:00.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 18: (DispatchQueue::entry()+0x60f) [0x7f570459bcf9] 2024-07-26T11:29:00.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f57046785a1] 2024-07-26T11:29:00.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 20: (Thread::entry_wrapper()+0x43) [0x7f5704425cdd] 2024-07-26T11:29:00.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5704425cf9] 2024-07-26T11:29:00.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5703a89c02] 2024-07-26T11:29:00.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5703b0ec40] 2024-07-26T11:29:00.681 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:29:00.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 2024-07-26T11:29:00.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: -2> 2024-07-26T11:29:00.230+0000 7f56fd800640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:29:00.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: in thread 7f56fd800640 thread_name:ms_dispatch 2024-07-26T11:29:00.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 2024-07-26T11:29:00.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:29:00.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5703a3e6f0] 2024-07-26T11:29:00.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f5703a8b94c] 2024-07-26T11:29:00.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 3: raise() 2024-07-26T11:29:00.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 4: abort() 2024-07-26T11:29:00.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 5: /lib64/libc.so.6(+0x2871b) [0x7f5703a2871b] 2024-07-26T11:29:00.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 6: /lib64/libc.so.6(+0x37386) [0x7f5703a37386] 2024-07-26T11:29:00.682 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55849095e6b9] 2024-07-26T11:29:00.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558490b1c80c] 2024-07-26T11:29:00.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558490a5c2ca] 2024-07-26T11:29:00.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558490a2c753] 2024-07-26T11:29:00.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558490a2da1b] 2024-07-26T11:29:00.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558490a11752] 2024-07-26T11:29:00.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55849095fa56] 2024-07-26T11:29:00.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558490963637] 2024-07-26T11:29:00.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558490963f5e] 2024-07-26T11:29:00.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55849094703a] 2024-07-26T11:29:00.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f570459b407] 2024-07-26T11:29:00.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 18: (DispatchQueue::entry()+0x60f) [0x7f570459bcf9] 2024-07-26T11:29:00.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f57046785a1] 2024-07-26T11:29:00.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 20: (Thread::entry_wrapper()+0x43) [0x7f5704425cdd] 2024-07-26T11:29:00.683 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 21: (Thread::_entry_func(void*)+0xd) [0x7f5704425cf9] 2024-07-26T11:29:00.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 22: /lib64/libc.so.6(+0x89c02) [0x7f5703a89c02] 2024-07-26T11:29:00.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f5703b0ec40] 2024-07-26T11:29:00.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:29:00.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[111873]: 2024-07-26T11:29:00.684 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:00 smithi134 podman[111923]: 2024-07-26 11:29:00.363940812 +0000 UTC m=+0.033395305 container died 373d2713da3755148f9b6794c7897d8b6e0d710e448cca8400741db1af865bab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-07-26T11:29:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:00 smithi189 ceph-mon[36790]: pgmap v891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 10 op/s 2024-07-26T11:29:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:00 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/3576163546,v1:172.21.15.134:6841/3576163546] up:rejoin 2024-07-26T11:29:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:00 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:29:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:00 smithi189 ceph-mon[36790]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:29:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:01 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:29:01.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:01 smithi193 ceph-mon[37473]: Cluster is now healthy 2024-07-26T11:29:01.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:01 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/3576163546,v1:172.21.15.134:6841/3576163546] up:active 2024-07-26T11:29:01.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:01 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:29:01.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:01 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:29:01.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:01 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:29:01.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:01 smithi134 ceph-mon[31966]: Cluster is now healthy 2024-07-26T11:29:01.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:01 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/3576163546,v1:172.21.15.134:6841/3576163546] up:active 2024-07-26T11:29:01.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:01 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:29:01.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:01 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:29:01.673 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:01 smithi134 podman[111923]: 2024-07-26 11:29:01.24446683 +0000 UTC m=+0.913921363 container remove 373d2713da3755148f9b6794c7897d8b6e0d710e448cca8400741db1af865bab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:29:01.674 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:01 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:29:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:01 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:29:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:01 smithi189 ceph-mon[36790]: Cluster is now healthy 2024-07-26T11:29:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:01 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/3576163546,v1:172.21.15.134:6841/3576163546] up:active 2024-07-26T11:29:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:01 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:29:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:01 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:29:02.090 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:01 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T11:29:02.090 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:01 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Consumed 1.682s CPU time. 2024-07-26T11:29:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:02 smithi193 ceph-mon[37473]: pgmap v892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-07-26T11:29:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:02 smithi134 ceph-mon[31966]: pgmap v892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-07-26T11:29:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:02 smithi189 ceph-mon[36790]: pgmap v892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 10 op/s 2024-07-26T11:29:02.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:29:02.810+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:29:02.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:29:02.810+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:29:02.815 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:29:02.810+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:04 smithi193 ceph-mon[37473]: pgmap v893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-07-26T11:29:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:04 smithi134 ceph-mon[31966]: pgmap v893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-07-26T11:29:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:04 smithi189 ceph-mon[36790]: pgmap v893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-07-26T11:29:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:06 smithi193 ceph-mon[37473]: pgmap v894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-07-26T11:29:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:06 smithi134 ceph-mon[31966]: pgmap v894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-07-26T11:29:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:06 smithi189 ceph-mon[36790]: pgmap v894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-07-26T11:29:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:07 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:29:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:07 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:29:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:07 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:29:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:08 smithi134 ceph-mon[31966]: pgmap v895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-07-26T11:29:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:08 smithi189 ceph-mon[36790]: pgmap v895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-07-26T11:29:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:08 smithi193 ceph-mon[37473]: pgmap v895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-07-26T11:29:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:10 smithi134 ceph-mon[31966]: pgmap v896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T11:29:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:10 smithi189 ceph-mon[36790]: pgmap v896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T11:29:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:10 smithi193 ceph-mon[37473]: pgmap v896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-07-26T11:29:12.291 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:11 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 4. 2024-07-26T11:29:12.292 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:11 smithi134 systemd[1]: Stopped Ceph mds.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:29:12.292 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:11 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Consumed 1.682s CPU time. 2024-07-26T11:29:12.292 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:11 smithi134 systemd[1]: Starting Ceph mds.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:29:12.650 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:12 smithi134 ceph-mon[31966]: pgmap v897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-26T11:29:12.651 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:12 smithi134 podman[112014]: 2024-07-26 11:29:12.2097307 +0000 UTC m=+0.031452976 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:29:12.651 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:12 smithi134 podman[112014]: 2024-07-26 11:29:12.34568476 +0000 UTC m=+0.167407055 container create 24d52cc85f2e8dd14c05a63c69bffee44b9442bbf1def8aecdeeed241085f6cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724) 2024-07-26T11:29:12.651 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:12 smithi134 podman[112014]: 2024-07-26 11:29:12.648865889 +0000 UTC m=+0.470588251 container init 24d52cc85f2e8dd14c05a63c69bffee44b9442bbf1def8aecdeeed241085f6cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:29:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:12 smithi189 ceph-mon[36790]: pgmap v897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-26T11:29:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:12 smithi193 ceph-mon[37473]: pgmap v897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-07-26T11:29:12.923 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:12 smithi134 podman[112014]: 2024-07-26 11:29:12.658545467 +0000 UTC m=+0.480267755 container start 24d52cc85f2e8dd14c05a63c69bffee44b9442bbf1def8aecdeeed241085f6cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-07-26T11:29:12.923 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:12 smithi134 bash[112014]: 24d52cc85f2e8dd14c05a63c69bffee44b9442bbf1def8aecdeeed241085f6cf 2024-07-26T11:29:12.923 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:12 smithi134 systemd[1]: Started Ceph mds.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:29:12.923 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:12 smithi134 ceph-mds[112033]: -- 172.21.15.134:0/786074331 <== mon.0 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ef5e6d3860 con 0x55ef62875600 2024-07-26T11:29:12.923 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:12 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: starting mds.a at 2024-07-26T11:29:13.347 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:29:13.345+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:29:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:29:13.347+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T11:29:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:29:13.347+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:29:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:29:13.347+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:29:13.349 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:29:13.347+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:13.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:13 smithi134 ceph-mon[31966]: daemon mds.a restarted 2024-07-26T11:29:13.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:13 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:29:13.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:13 smithi134 ceph-mon[31966]: osdmap e112: 12 total, 12 up, 12 in 2024-07-26T11:29:13.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:13 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/2284177228,v1:172.21.15.134:6841/2284177228] up:boot 2024-07-26T11:29:13.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:13 smithi134 ceph-mon[31966]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T11:29:13.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:13 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:29:13.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:29:13.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:13 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:29:13.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:13 smithi189 ceph-mon[36790]: daemon mds.a restarted 2024-07-26T11:29:13.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:13 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:29:13.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:13 smithi189 ceph-mon[36790]: osdmap e112: 12 total, 12 up, 12 in 2024-07-26T11:29:13.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:13 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/2284177228,v1:172.21.15.134:6841/2284177228] up:boot 2024-07-26T11:29:13.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:13 smithi189 ceph-mon[36790]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T11:29:13.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:13 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:29:13.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:29:13.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:13 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:29:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:13 smithi193 ceph-mon[37473]: daemon mds.a restarted 2024-07-26T11:29:13.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:13 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:29:13.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:13 smithi193 ceph-mon[37473]: osdmap e112: 12 total, 12 up, 12 in 2024-07-26T11:29:13.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:13 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/2284177228,v1:172.21.15.134:6841/2284177228] up:boot 2024-07-26T11:29:13.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:13 smithi193 ceph-mon[37473]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T11:29:13.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:13 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:29:13.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-07-26T11:29:13.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:13 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:29:15.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:14 smithi134 ceph-mon[31966]: pgmap v899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:15.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:14 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/638423294,v1:172.21.15.189:6835/638423294] up:resolve 2024-07-26T11:29:15.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:14 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:29:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:14 smithi189 ceph-mon[36790]: pgmap v899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:14 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/638423294,v1:172.21.15.189:6835/638423294] up:resolve 2024-07-26T11:29:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:14 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:29:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:14 smithi193 ceph-mon[37473]: pgmap v899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:15.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:14 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/638423294,v1:172.21.15.189:6835/638423294] up:resolve 2024-07-26T11:29:15.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:14 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:29:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:15 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:29:16.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:15 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/638423294,v1:172.21.15.189:6835/638423294] up:reconnect 2024-07-26T11:29:16.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:15 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:29:16.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:15 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999981 2024-07-26T11:29:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:15 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:29:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:15 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/638423294,v1:172.21.15.189:6835/638423294] up:reconnect 2024-07-26T11:29:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:15 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:29:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:15 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999981 2024-07-26T11:29:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:15 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:29:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:15 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/638423294,v1:172.21.15.189:6835/638423294] up:reconnect 2024-07-26T11:29:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:15 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:29:16.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:15 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999981 2024-07-26T11:29:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:16 smithi134 ceph-mon[31966]: pgmap v900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-07-26T11:29:17.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:16 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/638423294,v1:172.21.15.189:6835/638423294] up:rejoin 2024-07-26T11:29:17.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:16 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:29:17.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:16 smithi134 ceph-mon[31966]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:29:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-mon[36790]: pgmap v900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-07-26T11:29:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/638423294,v1:172.21.15.189:6835/638423294] up:rejoin 2024-07-26T11:29:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:29:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-mon[36790]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:29:17.194 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:29:17.194 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: *** Caught signal (Aborted) ** 2024-07-26T11:29:17.194 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: in thread 7fa772a1c640 thread_name:ms_dispatch 2024-07-26T11:29:17.194 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:29:17.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa777df36f0] 2024-07-26T11:29:17.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa777e4094c] 2024-07-26T11:29:17.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 3: raise() 2024-07-26T11:29:17.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 4: abort() 2024-07-26T11:29:17.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa777ddd71b] 2024-07-26T11:29:17.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 6: /lib64/libc.so.6(+0x37386) [0x7fa777dec386] 2024-07-26T11:29:17.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559136ea36b9] 2024-07-26T11:29:17.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55913706180c] 2024-07-26T11:29:17.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559136fa12ca] 2024-07-26T11:29:17.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559136f71753] 2024-07-26T11:29:17.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559136f72a1b] 2024-07-26T11:29:17.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559136f56752] 2024-07-26T11:29:17.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559136ea4a56] 2024-07-26T11:29:17.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559136ea8637] 2024-07-26T11:29:17.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559136ea8f5e] 2024-07-26T11:29:17.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559136e8c03a] 2024-07-26T11:29:17.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa7786c0407] 2024-07-26T11:29:17.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 18: (DispatchQueue::entry()+0x60f) [0x7fa7786c0cf9] 2024-07-26T11:29:17.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa77879d5a1] 2024-07-26T11:29:17.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 20: (Thread::entry_wrapper()+0x43) [0x7fa77854acdd] 2024-07-26T11:29:17.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa77854acf9] 2024-07-26T11:29:17.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa777e3ec02] 2024-07-26T11:29:17.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa777ec3c40] 2024-07-26T11:29:17.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 2024-07-26T11:29:16.846+0000 7fa772a1c640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:29:17.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: in thread 7fa772a1c640 thread_name:ms_dispatch 2024-07-26T11:29:17.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 2024-07-26T11:29:17.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:29:17.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa777df36f0] 2024-07-26T11:29:17.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa777e4094c] 2024-07-26T11:29:17.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 3: raise() 2024-07-26T11:29:17.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 4: abort() 2024-07-26T11:29:17.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa777ddd71b] 2024-07-26T11:29:17.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 6: /lib64/libc.so.6(+0x37386) [0x7fa777dec386] 2024-07-26T11:29:17.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559136ea36b9] 2024-07-26T11:29:17.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55913706180c] 2024-07-26T11:29:17.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559136fa12ca] 2024-07-26T11:29:17.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559136f71753] 2024-07-26T11:29:17.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559136f72a1b] 2024-07-26T11:29:17.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559136f56752] 2024-07-26T11:29:17.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559136ea4a56] 2024-07-26T11:29:17.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559136ea8637] 2024-07-26T11:29:17.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559136ea8f5e] 2024-07-26T11:29:17.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559136e8c03a] 2024-07-26T11:29:17.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa7786c0407] 2024-07-26T11:29:17.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 18: (DispatchQueue::entry()+0x60f) [0x7fa7786c0cf9] 2024-07-26T11:29:17.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa77879d5a1] 2024-07-26T11:29:17.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 20: (Thread::entry_wrapper()+0x43) [0x7fa77854acdd] 2024-07-26T11:29:17.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa77854acf9] 2024-07-26T11:29:17.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa777e3ec02] 2024-07-26T11:29:17.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa777ec3c40] 2024-07-26T11:29:17.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:29:17.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 2024-07-26T11:29:17.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 0> 2024-07-26T11:29:16.846+0000 7fa772a1c640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:29:17.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: in thread 7fa772a1c640 thread_name:ms_dispatch 2024-07-26T11:29:17.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 2024-07-26T11:29:17.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:29:17.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa777df36f0] 2024-07-26T11:29:17.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa777e4094c] 2024-07-26T11:29:17.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 3: raise() 2024-07-26T11:29:17.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 4: abort() 2024-07-26T11:29:17.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa777ddd71b] 2024-07-26T11:29:17.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 6: /lib64/libc.so.6(+0x37386) [0x7fa777dec386] 2024-07-26T11:29:17.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559136ea36b9] 2024-07-26T11:29:17.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55913706180c] 2024-07-26T11:29:17.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559136fa12ca] 2024-07-26T11:29:17.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559136f71753] 2024-07-26T11:29:17.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559136f72a1b] 2024-07-26T11:29:17.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559136f56752] 2024-07-26T11:29:17.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559136ea4a56] 2024-07-26T11:29:17.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559136ea8637] 2024-07-26T11:29:17.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559136ea8f5e] 2024-07-26T11:29:17.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559136e8c03a] 2024-07-26T11:29:17.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa7786c0407] 2024-07-26T11:29:17.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 18: (DispatchQueue::entry()+0x60f) [0x7fa7786c0cf9] 2024-07-26T11:29:17.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa77879d5a1] 2024-07-26T11:29:17.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 20: (Thread::entry_wrapper()+0x43) [0x7fa77854acdd] 2024-07-26T11:29:17.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa77854acf9] 2024-07-26T11:29:17.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa777e3ec02] 2024-07-26T11:29:17.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa777ec3c40] 2024-07-26T11:29:17.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:29:17.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 2024-07-26T11:29:17.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 0> 2024-07-26T11:29:16.846+0000 7fa772a1c640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:29:17.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: in thread 7fa772a1c640 thread_name:ms_dispatch 2024-07-26T11:29:17.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 2024-07-26T11:29:17.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:29:17.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fa777df36f0] 2024-07-26T11:29:17.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fa777e4094c] 2024-07-26T11:29:17.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 3: raise() 2024-07-26T11:29:17.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 4: abort() 2024-07-26T11:29:17.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 5: /lib64/libc.so.6(+0x2871b) [0x7fa777ddd71b] 2024-07-26T11:29:17.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 6: /lib64/libc.so.6(+0x37386) [0x7fa777dec386] 2024-07-26T11:29:17.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x559136ea36b9] 2024-07-26T11:29:17.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55913706180c] 2024-07-26T11:29:17.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x559136fa12ca] 2024-07-26T11:29:17.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x559136f71753] 2024-07-26T11:29:17.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x559136f72a1b] 2024-07-26T11:29:17.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x559136f56752] 2024-07-26T11:29:17.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x559136ea4a56] 2024-07-26T11:29:17.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x559136ea8637] 2024-07-26T11:29:17.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x559136ea8f5e] 2024-07-26T11:29:17.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x559136e8c03a] 2024-07-26T11:29:17.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fa7786c0407] 2024-07-26T11:29:17.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 18: (DispatchQueue::entry()+0x60f) [0x7fa7786c0cf9] 2024-07-26T11:29:17.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fa77879d5a1] 2024-07-26T11:29:17.204 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 20: (Thread::entry_wrapper()+0x43) [0x7fa77854acdd] 2024-07-26T11:29:17.204 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 21: (Thread::_entry_func(void*)+0xd) [0x7fa77854acf9] 2024-07-26T11:29:17.204 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 22: /lib64/libc.so.6(+0x89c02) [0x7fa777e3ec02] 2024-07-26T11:29:17.204 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fa777ec3c40] 2024-07-26T11:29:17.204 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:29:17.204 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73177]: 2024-07-26T11:29:17.204 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:16 smithi189 podman[73215]: 2024-07-26 11:29:16.888042835 +0000 UTC m=+0.019695170 container died 46c6529c2824e8dc1221088c1081862da79e50c9655f2452eac60af918081c83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, io.buildah.version=1.36.0) 2024-07-26T11:29:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:16 smithi193 ceph-mon[37473]: pgmap v900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-07-26T11:29:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:16 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/638423294,v1:172.21.15.189:6835/638423294] up:rejoin 2024-07-26T11:29:17.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:16 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:29:17.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:16 smithi193 ceph-mon[37473]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:29:17.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:29:17.824+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:29:17.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:29:17.826+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 12 2024-07-26T11:29:17.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:29:17.826+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:29:17.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:29:17.826+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:29:17.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:29:17.827+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:17.869 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:17 smithi189 podman[73215]: 2024-07-26 11:29:17.50329892 +0000 UTC m=+0.634951299 container remove 46c6529c2824e8dc1221088c1081862da79e50c9655f2452eac60af918081c83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2) 2024-07-26T11:29:17.869 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:17 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:29:17.869 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:17 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T11:29:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:17 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:29:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:17 smithi134 ceph-mon[31966]: Cluster is now healthy 2024-07-26T11:29:18.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:17 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/638423294,v1:172.21.15.189:6835/638423294] up:active 2024-07-26T11:29:18.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:17 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:29:18.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:17 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:29:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:17 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:29:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:17 smithi189 ceph-mon[36790]: Cluster is now healthy 2024-07-26T11:29:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:17 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/638423294,v1:172.21.15.189:6835/638423294] up:active 2024-07-26T11:29:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:17 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:29:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:17 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:29:18.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:17 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:29:18.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:17 smithi193 ceph-mon[37473]: Cluster is now healthy 2024-07-26T11:29:18.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:17 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/638423294,v1:172.21.15.189:6835/638423294] up:active 2024-07-26T11:29:18.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:17 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:29:18.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:17 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:29:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:18 smithi134 ceph-mon[31966]: pgmap v901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:29:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:18 smithi189 ceph-mon[36790]: pgmap v901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:29:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:18 smithi193 ceph-mon[37473]: pgmap v901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:29:19.275 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:29:19 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:29:19.053+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:29:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:20 smithi134 ceph-mon[31966]: pgmap v902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:20 smithi189 ceph-mon[36790]: pgmap v902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:20 smithi193 ceph-mon[37473]: pgmap v902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:22 smithi134 ceph-mon[31966]: pgmap v903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:29:23.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:22 smithi189 ceph-mon[36790]: pgmap v903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:29:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:22 smithi193 ceph-mon[37473]: pgmap v903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:29:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:24 smithi134 ceph-mon[31966]: pgmap v904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 201 B/s wr, 10 op/s 2024-07-26T11:29:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:24 smithi189 ceph-mon[36790]: pgmap v904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 201 B/s wr, 10 op/s 2024-07-26T11:29:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:24 smithi193 ceph-mon[37473]: pgmap v904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 201 B/s wr, 10 op/s 2024-07-26T11:29:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:29:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:29:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:29:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:26 smithi134 ceph-mon[31966]: pgmap v905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:29:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:29:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:29:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:29:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:29:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:29:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:29:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:29:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:26 smithi189 ceph-mon[36790]: pgmap v905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:29:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:29:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:29:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:29:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:29:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:29:27.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:29:27.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:29:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:26 smithi193 ceph-mon[37473]: pgmap v905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:29:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:29:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:29:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:29:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:29:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:29:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:29:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:29:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:27 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T11:29:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:27 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T11:29:28.182 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:27 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T11:29:28.183 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:27 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T11:29:28.183 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:27 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 4. 2024-07-26T11:29:28.183 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:27 smithi189 systemd[1]: Stopped Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:29:28.183 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:27 smithi189 systemd[1]: Starting Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:29:28.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:27 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T11:29:28.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:27 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T11:29:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:29:28.363+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:29:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:29:28.364+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-07-26T11:29:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:29:28.364+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-07-26T11:29:28.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:29:28.364+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:29:28.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:29:28.364+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:28.444 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:28 smithi189 podman[73436]: 2024-07-26 11:29:28.084053814 +0000 UTC m=+0.018816902 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:29:28.444 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:28 smithi189 podman[73436]: 2024-07-26 11:29:28.204566206 +0000 UTC m=+0.139329334 container create f38729f7ae2f0869a6283da384db0f157158bca9b63f76b5657946e3419e2fe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:29:28.712 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:28 smithi189 podman[73436]: 2024-07-26 11:29:28.464119283 +0000 UTC m=+0.398882367 container init f38729f7ae2f0869a6283da384db0f157158bca9b63f76b5657946e3419e2fe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-07-26T11:29:28.712 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:28 smithi189 podman[73436]: 2024-07-26 11:29:28.467344348 +0000 UTC m=+0.402107434 container start f38729f7ae2f0869a6283da384db0f157158bca9b63f76b5657946e3419e2fe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:29:28.712 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:28 smithi189 ceph-mds[73456]: -- 172.21.15.189:0/249395585 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dcfeb7f860 con 0x55dd02d21a80 2024-07-26T11:29:28.712 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:28 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: starting mds.e at 2024-07-26T11:29:28.712 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:28 smithi189 bash[73436]: f38729f7ae2f0869a6283da384db0f157158bca9b63f76b5657946e3419e2fe6 2024-07-26T11:29:28.712 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:28 smithi189 systemd[1]: Started Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:29:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:28 smithi189 ceph-mon[36790]: pgmap v906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:29:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:28 smithi189 ceph-mon[36790]: daemon mds.e restarted 2024-07-26T11:29:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:28 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:29:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:28 smithi189 ceph-mon[36790]: osdmap e113: 12 total, 12 up, 12 in 2024-07-26T11:29:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:28 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/3821381197,v1:172.21.15.189:6835/3821381197] up:boot 2024-07-26T11:29:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:28 smithi189 ceph-mon[36790]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:29:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:28 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:29:29.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:29:29.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:28 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:29:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:28 smithi193 ceph-mon[37473]: pgmap v906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:29:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:28 smithi193 ceph-mon[37473]: daemon mds.e restarted 2024-07-26T11:29:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:28 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:29:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:28 smithi193 ceph-mon[37473]: osdmap e113: 12 total, 12 up, 12 in 2024-07-26T11:29:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:28 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/3821381197,v1:172.21.15.189:6835/3821381197] up:boot 2024-07-26T11:29:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:28 smithi193 ceph-mon[37473]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:29:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:28 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:29:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:29:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:28 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:29:29.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:28 smithi134 ceph-mon[31966]: pgmap v906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:29:29.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:28 smithi134 ceph-mon[31966]: daemon mds.e restarted 2024-07-26T11:29:29.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:28 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:29:29.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:28 smithi134 ceph-mon[31966]: osdmap e113: 12 total, 12 up, 12 in 2024-07-26T11:29:29.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:28 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/3821381197,v1:172.21.15.189:6835/3821381197] up:boot 2024-07-26T11:29:29.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:28 smithi134 ceph-mon[31966]: Standby daemon mds.a assigned to filesystem cephfs as rank 1 2024-07-26T11:29:29.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:28 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-07-26T11:29:29.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:29:29.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:28 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-07-26T11:29:30.799 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:30 smithi134 ceph-mon[31966]: pgmap v908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:30.799 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:30 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/2284177228,v1:172.21.15.134:6841/2284177228] up:resolve 2024-07-26T11:29:30.799 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:30 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:29:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:30 smithi189 ceph-mon[36790]: pgmap v908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:30 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/2284177228,v1:172.21.15.134:6841/2284177228] up:resolve 2024-07-26T11:29:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:30 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:29:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:30 smithi193 ceph-mon[37473]: pgmap v908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:30 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/2284177228,v1:172.21.15.134:6841/2284177228] up:resolve 2024-07-26T11:29:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:30 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-07-26T11:29:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:31 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:29:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:31 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/2284177228,v1:172.21.15.134:6841/2284177228] up:reconnect 2024-07-26T11:29:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:31 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:29:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:31 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999978 2024-07-26T11:29:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:31 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:29:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:31 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/2284177228,v1:172.21.15.134:6841/2284177228] up:reconnect 2024-07-26T11:29:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:31 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:29:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:31 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999978 2024-07-26T11:29:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:31 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:29:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:31 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/2284177228,v1:172.21.15.134:6841/2284177228] up:reconnect 2024-07-26T11:29:32.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:31 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:29:32.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:31 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999978 2024-07-26T11:29:32.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:29:32.842+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:29:32.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:29:32.842+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:29:32.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:29:32.842+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:32.845 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-mon[31966]: pgmap v909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:29:32.845 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/2284177228,v1:172.21.15.134:6841/2284177228] up:rejoin 2024-07-26T11:29:32.845 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:29:32.845 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-mon[31966]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:29:32.847 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:29:32.848 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: *** Caught signal (Aborted) ** 2024-07-26T11:29:32.848 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: in thread 7f4ef7a00640 thread_name:ms_dispatch 2024-07-26T11:29:32.848 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:29:32.848 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4efdc3e6f0] 2024-07-26T11:29:32.848 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4efdc8b94c] 2024-07-26T11:29:32.848 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 3: raise() 2024-07-26T11:29:32.848 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 4: abort() 2024-07-26T11:29:32.848 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4efdc2871b] 2024-07-26T11:29:32.848 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 6: /lib64/libc.so.6(+0x37386) [0x7f4efdc37386] 2024-07-26T11:29:32.848 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ef2846a6b9] 2024-07-26T11:29:32.848 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ef2862880c] 2024-07-26T11:29:32.848 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ef285682ca] 2024-07-26T11:29:32.849 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ef28538753] 2024-07-26T11:29:32.849 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ef28539a1b] 2024-07-26T11:29:32.849 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ef2851d752] 2024-07-26T11:29:32.849 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ef2846ba56] 2024-07-26T11:29:32.849 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ef2846f637] 2024-07-26T11:29:32.849 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ef2846ff5e] 2024-07-26T11:29:32.849 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ef2845303a] 2024-07-26T11:29:32.849 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4efe79b407] 2024-07-26T11:29:32.849 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 18: (DispatchQueue::entry()+0x60f) [0x7f4efe79bcf9] 2024-07-26T11:29:32.849 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4efe8785a1] 2024-07-26T11:29:32.850 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 20: (Thread::entry_wrapper()+0x43) [0x7f4efe625cdd] 2024-07-26T11:29:32.850 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4efe625cf9] 2024-07-26T11:29:32.850 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4efdc89c02] 2024-07-26T11:29:32.850 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4efdd0ec40] 2024-07-26T11:29:32.850 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 2024-07-26T11:29:32.570+0000 7f4ef7a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:29:32.850 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: in thread 7f4ef7a00640 thread_name:ms_dispatch 2024-07-26T11:29:32.850 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 2024-07-26T11:29:32.850 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:29:32.850 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4efdc3e6f0] 2024-07-26T11:29:32.850 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4efdc8b94c] 2024-07-26T11:29:32.850 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 3: raise() 2024-07-26T11:29:32.850 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 4: abort() 2024-07-26T11:29:32.851 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4efdc2871b] 2024-07-26T11:29:32.851 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 6: /lib64/libc.so.6(+0x37386) [0x7f4efdc37386] 2024-07-26T11:29:32.851 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ef2846a6b9] 2024-07-26T11:29:32.851 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ef2862880c] 2024-07-26T11:29:32.851 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ef285682ca] 2024-07-26T11:29:32.851 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ef28538753] 2024-07-26T11:29:32.851 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ef28539a1b] 2024-07-26T11:29:32.851 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ef2851d752] 2024-07-26T11:29:32.851 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ef2846ba56] 2024-07-26T11:29:32.851 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ef2846f637] 2024-07-26T11:29:32.851 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ef2846ff5e] 2024-07-26T11:29:32.851 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ef2845303a] 2024-07-26T11:29:32.851 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4efe79b407] 2024-07-26T11:29:32.851 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 18: (DispatchQueue::entry()+0x60f) [0x7f4efe79bcf9] 2024-07-26T11:29:32.852 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4efe8785a1] 2024-07-26T11:29:32.852 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 20: (Thread::entry_wrapper()+0x43) [0x7f4efe625cdd] 2024-07-26T11:29:32.852 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4efe625cf9] 2024-07-26T11:29:32.852 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4efdc89c02] 2024-07-26T11:29:32.852 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4efdd0ec40] 2024-07-26T11:29:32.852 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:29:32.852 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 2024-07-26T11:29:32.852 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 0> 2024-07-26T11:29:32.570+0000 7f4ef7a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:29:32.852 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: in thread 7f4ef7a00640 thread_name:ms_dispatch 2024-07-26T11:29:32.853 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 2024-07-26T11:29:32.853 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:29:32.853 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4efdc3e6f0] 2024-07-26T11:29:32.853 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4efdc8b94c] 2024-07-26T11:29:32.853 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 3: raise() 2024-07-26T11:29:32.853 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 4: abort() 2024-07-26T11:29:32.853 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4efdc2871b] 2024-07-26T11:29:32.853 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 6: /lib64/libc.so.6(+0x37386) [0x7f4efdc37386] 2024-07-26T11:29:32.853 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ef2846a6b9] 2024-07-26T11:29:32.853 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ef2862880c] 2024-07-26T11:29:32.853 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ef285682ca] 2024-07-26T11:29:32.853 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ef28538753] 2024-07-26T11:29:32.853 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ef28539a1b] 2024-07-26T11:29:32.854 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ef2851d752] 2024-07-26T11:29:32.854 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ef2846ba56] 2024-07-26T11:29:32.854 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ef2846f637] 2024-07-26T11:29:32.854 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ef2846ff5e] 2024-07-26T11:29:32.854 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ef2845303a] 2024-07-26T11:29:32.854 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4efe79b407] 2024-07-26T11:29:32.854 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 18: (DispatchQueue::entry()+0x60f) [0x7f4efe79bcf9] 2024-07-26T11:29:32.855 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4efe8785a1] 2024-07-26T11:29:32.855 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 20: (Thread::entry_wrapper()+0x43) [0x7f4efe625cdd] 2024-07-26T11:29:32.855 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4efe625cf9] 2024-07-26T11:29:32.855 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4efdc89c02] 2024-07-26T11:29:32.855 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4efdd0ec40] 2024-07-26T11:29:32.855 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:29:32.855 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 2024-07-26T11:29:32.855 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 0> 2024-07-26T11:29:32.570+0000 7f4ef7a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:29:32.855 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: in thread 7f4ef7a00640 thread_name:ms_dispatch 2024-07-26T11:29:32.855 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 2024-07-26T11:29:32.855 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:29:32.855 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4efdc3e6f0] 2024-07-26T11:29:32.855 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4efdc8b94c] 2024-07-26T11:29:32.855 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 3: raise() 2024-07-26T11:29:32.856 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 4: abort() 2024-07-26T11:29:32.856 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4efdc2871b] 2024-07-26T11:29:32.856 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 6: /lib64/libc.so.6(+0x37386) [0x7f4efdc37386] 2024-07-26T11:29:32.856 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55ef2846a6b9] 2024-07-26T11:29:32.856 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55ef2862880c] 2024-07-26T11:29:32.856 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55ef285682ca] 2024-07-26T11:29:32.856 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55ef28538753] 2024-07-26T11:29:32.856 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55ef28539a1b] 2024-07-26T11:29:32.856 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55ef2851d752] 2024-07-26T11:29:32.856 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55ef2846ba56] 2024-07-26T11:29:32.856 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55ef2846f637] 2024-07-26T11:29:32.856 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55ef2846ff5e] 2024-07-26T11:29:32.856 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55ef2845303a] 2024-07-26T11:29:32.857 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4efe79b407] 2024-07-26T11:29:32.857 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 18: (DispatchQueue::entry()+0x60f) [0x7f4efe79bcf9] 2024-07-26T11:29:32.857 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4efe8785a1] 2024-07-26T11:29:32.857 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 20: (Thread::entry_wrapper()+0x43) [0x7f4efe625cdd] 2024-07-26T11:29:32.857 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4efe625cf9] 2024-07-26T11:29:32.857 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4efdc89c02] 2024-07-26T11:29:32.857 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4efdd0ec40] 2024-07-26T11:29:32.857 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:29:32.857 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a[112029]: 2024-07-26T11:29:32.857 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:32 smithi134 podman[112197]: 2024-07-26 11:29:32.688835583 +0000 UTC m=+0.034856132 container died 24d52cc85f2e8dd14c05a63c69bffee44b9442bbf1def8aecdeeed241085f6cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-07-26T11:29:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:32 smithi189 ceph-mon[36790]: pgmap v909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:29:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:32 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/2284177228,v1:172.21.15.134:6841/2284177228] up:rejoin 2024-07-26T11:29:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:32 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:29:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:32 smithi189 ceph-mon[36790]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:29:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:32 smithi193 ceph-mon[37473]: pgmap v909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:29:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:32 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/2284177228,v1:172.21.15.134:6841/2284177228] up:rejoin 2024-07-26T11:29:33.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:32 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:29:33.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:32 smithi193 ceph-mon[37473]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-07-26T11:29:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:33 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:29:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:33 smithi134 ceph-mon[31966]: Cluster is now healthy 2024-07-26T11:29:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:33 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6840/2284177228,v1:172.21.15.134:6841/2284177228] up:active 2024-07-26T11:29:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:33 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:29:33.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:33 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:29:33.923 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:33 smithi134 podman[112197]: 2024-07-26 11:29:33.577760916 +0000 UTC m=+0.923781455 container remove 24d52cc85f2e8dd14c05a63c69bffee44b9442bbf1def8aecdeeed241085f6cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:29:33.923 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:33 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:29:33.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:33 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:29:33.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:33 smithi189 ceph-mon[36790]: Cluster is now healthy 2024-07-26T11:29:33.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:33 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6840/2284177228,v1:172.21.15.134:6841/2284177228] up:active 2024-07-26T11:29:33.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:33 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:29:33.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:33 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:29:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:33 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:29:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:33 smithi193 ceph-mon[37473]: Cluster is now healthy 2024-07-26T11:29:34.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:33 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6840/2284177228,v1:172.21.15.134:6841/2284177228] up:active 2024-07-26T11:29:34.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:33 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-07-26T11:29:34.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:33 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:29:34.423 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:34 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T11:29:34.423 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:34 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Consumed 1.574s CPU time. 2024-07-26T11:29:34.524 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:29:34 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:29:34.054+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:29:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:34 smithi134 ceph-mon[31966]: pgmap v910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:34 smithi189 ceph-mon[36790]: pgmap v910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:34 smithi193 ceph-mon[37473]: pgmap v910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:36 smithi134 ceph-mon[31966]: pgmap v911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:29:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:36 smithi189 ceph-mon[36790]: pgmap v911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:29:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:36 smithi193 ceph-mon[37473]: pgmap v911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:29:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:38 smithi134 ceph-mon[31966]: pgmap v912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:29:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:38 smithi189 ceph-mon[36790]: pgmap v912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:29:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:38 smithi193 ceph-mon[37473]: pgmap v912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:29:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:40 smithi134 ceph-mon[31966]: pgmap v913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 196 B/s wr, 10 op/s 2024-07-26T11:29:40.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:40 smithi189 ceph-mon[36790]: pgmap v913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 196 B/s wr, 10 op/s 2024-07-26T11:29:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:40 smithi193 ceph-mon[37473]: pgmap v913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 196 B/s wr, 10 op/s 2024-07-26T11:29:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:42 smithi134 ceph-mon[31966]: pgmap v914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:29:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:42 smithi189 ceph-mon[36790]: pgmap v914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:29:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:42 smithi193 ceph-mon[37473]: pgmap v914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:29:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:29:43.368+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:29:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:29:43.368+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:29:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:29:43.368+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:44.617 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:44 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Scheduled restart job, restart counter is at 5. 2024-07-26T11:29:44.617 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:44 smithi134 systemd[1]: Stopped Ceph mds.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:29:44.617 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:44 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Consumed 1.574s CPU time. 2024-07-26T11:29:44.617 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:44 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Start request repeated too quickly. 2024-07-26T11:29:44.617 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:44 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.a.service: Failed with result 'exit-code'. 2024-07-26T11:29:44.617 INFO:journalctl@ceph.mds.a.smithi134.stdout:Jul 26 11:29:44 smithi134 systemd[1]: Failed to start Ceph mds.a for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:29:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:44 smithi134 ceph-mon[31966]: pgmap v915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:29:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:44 smithi189 ceph-mon[36790]: pgmap v915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:29:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:44 smithi193 ceph-mon[37473]: pgmap v915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:29:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:46 smithi134 ceph-mon[31966]: pgmap v916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T11:29:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:46 smithi189 ceph-mon[36790]: pgmap v916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T11:29:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:46 smithi193 ceph-mon[37473]: pgmap v916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 85 B/s wr, 9 op/s 2024-07-26T11:29:47.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:29:47.856+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:29:47.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:29:47.856+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:29:47.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:29:47.856+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:48 smithi134 ceph-mon[31966]: pgmap v917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T11:29:48.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:48 smithi189 ceph-mon[36790]: pgmap v917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T11:29:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:48 smithi193 ceph-mon[37473]: pgmap v917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-07-26T11:29:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:49 smithi134 ceph-mon[31966]: Replacing daemon mds.a as rank 1 with standby daemon mds.e 2024-07-26T11:29:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:49 smithi134 ceph-mon[31966]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T11:29:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:49 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:29:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:49 smithi134 ceph-mon[31966]: osdmap e114: 12 total, 12 up, 12 in 2024-07-26T11:29:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:49 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T11:29:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:49 smithi189 ceph-mon[36790]: Replacing daemon mds.a as rank 1 with standby daemon mds.e 2024-07-26T11:29:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:49 smithi189 ceph-mon[36790]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T11:29:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:49 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:29:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:49 smithi189 ceph-mon[36790]: osdmap e114: 12 total, 12 up, 12 in 2024-07-26T11:29:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:49 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T11:29:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:49 smithi193 ceph-mon[37473]: Replacing daemon mds.a as rank 1 with standby daemon mds.e 2024-07-26T11:29:50.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:49 smithi193 ceph-mon[37473]: MDS daemon mds.a is removed because it is dead or otherwise unavailable. 2024-07-26T11:29:50.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:49 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:29:50.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:49 smithi193 ceph-mon[37473]: osdmap e114: 12 total, 12 up, 12 in 2024-07-26T11:29:50.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:49 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T11:29:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:50 smithi134 ceph-mon[31966]: pgmap v918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:29:50.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:50 smithi189 ceph-mon[36790]: pgmap v918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:29:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:50 smithi193 ceph-mon[37473]: pgmap v918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-07-26T11:29:51.911 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:51 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/3821381197,v1:172.21.15.189:6835/3821381197] up:resolve 2024-07-26T11:29:51.912 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:51 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T11:29:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:51 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/3821381197,v1:172.21.15.189:6835/3821381197] up:resolve 2024-07-26T11:29:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:51 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T11:29:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:51 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/3821381197,v1:172.21.15.189:6835/3821381197] up:resolve 2024-07-26T11:29:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:51 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T11:29:52.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:52 smithi134 ceph-mon[31966]: pgmap v920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:29:52.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:52 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:29:52.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:52 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/3821381197,v1:172.21.15.189:6835/3821381197] up:reconnect 2024-07-26T11:29:52.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:52 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:29:52.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:52 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999981 2024-07-26T11:29:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:52 smithi189 ceph-mon[36790]: pgmap v920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:29:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:52 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:29:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:52 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/3821381197,v1:172.21.15.189:6835/3821381197] up:reconnect 2024-07-26T11:29:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:52 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:29:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:52 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999981 2024-07-26T11:29:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:52 smithi193 ceph-mon[37473]: pgmap v920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:29:53.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:52 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:29:53.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:52 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/3821381197,v1:172.21.15.189:6835/3821381197] up:reconnect 2024-07-26T11:29:53.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:52 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:29:53.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:52 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999981 2024-07-26T11:29:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/3821381197,v1:172.21.15.189:6835/3821381197] up:rejoin 2024-07-26T11:29:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:29:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-mon[36790]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:29:53.945 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:29:53.945 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: *** Caught signal (Aborted) ** 2024-07-26T11:29:53.945 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: in thread 7f6e951e6640 thread_name:ms_dispatch 2024-07-26T11:29:53.945 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:29:53.945 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6e9a5bd6f0] 2024-07-26T11:29:53.945 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6e9a60a94c] 2024-07-26T11:29:53.945 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 3: raise() 2024-07-26T11:29:53.945 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 4: abort() 2024-07-26T11:29:53.945 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6e9a5a771b] 2024-07-26T11:29:53.946 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 6: /lib64/libc.so.6(+0x37386) [0x7f6e9a5b6386] 2024-07-26T11:29:53.946 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55dcfd8ad6b9] 2024-07-26T11:29:53.946 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55dcfda6b80c] 2024-07-26T11:29:53.946 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55dcfd9ab2ca] 2024-07-26T11:29:53.946 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55dcfd97b753] 2024-07-26T11:29:53.946 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55dcfd97ca1b] 2024-07-26T11:29:53.946 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55dcfd960752] 2024-07-26T11:29:53.946 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55dcfd8aea56] 2024-07-26T11:29:53.946 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55dcfd8b2637] 2024-07-26T11:29:53.946 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55dcfd8b2f5e] 2024-07-26T11:29:53.947 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55dcfd89603a] 2024-07-26T11:29:53.947 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6e9ae8a407] 2024-07-26T11:29:53.947 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 18: (DispatchQueue::entry()+0x60f) [0x7f6e9ae8acf9] 2024-07-26T11:29:53.947 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6e9af675a1] 2024-07-26T11:29:53.947 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 20: (Thread::entry_wrapper()+0x43) [0x7f6e9ad14cdd] 2024-07-26T11:29:53.947 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6e9ad14cf9] 2024-07-26T11:29:53.947 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6e9a608c02] 2024-07-26T11:29:53.947 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6e9a68dc40] 2024-07-26T11:29:53.947 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 2024-07-26T11:29:53.690+0000 7f6e951e6640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:29:53.947 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: in thread 7f6e951e6640 thread_name:ms_dispatch 2024-07-26T11:29:53.947 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 2024-07-26T11:29:53.947 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:29:53.947 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6e9a5bd6f0] 2024-07-26T11:29:53.948 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6e9a60a94c] 2024-07-26T11:29:53.948 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 3: raise() 2024-07-26T11:29:53.948 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 4: abort() 2024-07-26T11:29:53.948 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6e9a5a771b] 2024-07-26T11:29:53.948 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 6: /lib64/libc.so.6(+0x37386) [0x7f6e9a5b6386] 2024-07-26T11:29:53.948 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55dcfd8ad6b9] 2024-07-26T11:29:53.948 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55dcfda6b80c] 2024-07-26T11:29:53.948 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55dcfd9ab2ca] 2024-07-26T11:29:53.948 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55dcfd97b753] 2024-07-26T11:29:53.948 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55dcfd97ca1b] 2024-07-26T11:29:53.948 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55dcfd960752] 2024-07-26T11:29:53.948 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55dcfd8aea56] 2024-07-26T11:29:53.948 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55dcfd8b2637] 2024-07-26T11:29:53.949 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55dcfd8b2f5e] 2024-07-26T11:29:53.949 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55dcfd89603a] 2024-07-26T11:29:53.949 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6e9ae8a407] 2024-07-26T11:29:53.949 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 18: (DispatchQueue::entry()+0x60f) [0x7f6e9ae8acf9] 2024-07-26T11:29:53.949 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6e9af675a1] 2024-07-26T11:29:53.949 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 20: (Thread::entry_wrapper()+0x43) [0x7f6e9ad14cdd] 2024-07-26T11:29:53.950 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6e9ad14cf9] 2024-07-26T11:29:53.950 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6e9a608c02] 2024-07-26T11:29:53.950 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6e9a68dc40] 2024-07-26T11:29:53.950 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:29:53.950 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 2024-07-26T11:29:53.950 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 0> 2024-07-26T11:29:53.690+0000 7f6e951e6640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:29:53.950 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: in thread 7f6e951e6640 thread_name:ms_dispatch 2024-07-26T11:29:53.950 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 2024-07-26T11:29:53.950 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:29:53.950 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6e9a5bd6f0] 2024-07-26T11:29:53.950 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6e9a60a94c] 2024-07-26T11:29:53.950 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 3: raise() 2024-07-26T11:29:53.951 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 4: abort() 2024-07-26T11:29:53.951 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6e9a5a771b] 2024-07-26T11:29:53.951 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 6: /lib64/libc.so.6(+0x37386) [0x7f6e9a5b6386] 2024-07-26T11:29:53.951 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55dcfd8ad6b9] 2024-07-26T11:29:53.951 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55dcfda6b80c] 2024-07-26T11:29:53.951 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55dcfd9ab2ca] 2024-07-26T11:29:53.951 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55dcfd97b753] 2024-07-26T11:29:53.951 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55dcfd97ca1b] 2024-07-26T11:29:53.951 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55dcfd960752] 2024-07-26T11:29:53.951 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55dcfd8aea56] 2024-07-26T11:29:53.951 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55dcfd8b2637] 2024-07-26T11:29:53.951 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55dcfd8b2f5e] 2024-07-26T11:29:53.951 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55dcfd89603a] 2024-07-26T11:29:53.951 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6e9ae8a407] 2024-07-26T11:29:53.952 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 18: (DispatchQueue::entry()+0x60f) [0x7f6e9ae8acf9] 2024-07-26T11:29:53.952 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6e9af675a1] 2024-07-26T11:29:53.952 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 20: (Thread::entry_wrapper()+0x43) [0x7f6e9ad14cdd] 2024-07-26T11:29:53.952 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6e9ad14cf9] 2024-07-26T11:29:53.952 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6e9a608c02] 2024-07-26T11:29:53.952 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6e9a68dc40] 2024-07-26T11:29:53.952 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:29:53.952 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 2024-07-26T11:29:53.952 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 0> 2024-07-26T11:29:53.690+0000 7f6e951e6640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:29:53.952 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: in thread 7f6e951e6640 thread_name:ms_dispatch 2024-07-26T11:29:53.952 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 2024-07-26T11:29:53.952 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:29:53.952 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6e9a5bd6f0] 2024-07-26T11:29:53.953 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6e9a60a94c] 2024-07-26T11:29:53.953 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 3: raise() 2024-07-26T11:29:53.953 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 4: abort() 2024-07-26T11:29:53.953 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6e9a5a771b] 2024-07-26T11:29:53.953 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 6: /lib64/libc.so.6(+0x37386) [0x7f6e9a5b6386] 2024-07-26T11:29:53.953 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55dcfd8ad6b9] 2024-07-26T11:29:53.953 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55dcfda6b80c] 2024-07-26T11:29:53.953 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55dcfd9ab2ca] 2024-07-26T11:29:53.953 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55dcfd97b753] 2024-07-26T11:29:53.953 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55dcfd97ca1b] 2024-07-26T11:29:53.953 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55dcfd960752] 2024-07-26T11:29:53.953 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55dcfd8aea56] 2024-07-26T11:29:53.953 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55dcfd8b2637] 2024-07-26T11:29:53.953 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55dcfd8b2f5e] 2024-07-26T11:29:53.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55dcfd89603a] 2024-07-26T11:29:53.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f6e9ae8a407] 2024-07-26T11:29:53.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 18: (DispatchQueue::entry()+0x60f) [0x7f6e9ae8acf9] 2024-07-26T11:29:53.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6e9af675a1] 2024-07-26T11:29:53.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 20: (Thread::entry_wrapper()+0x43) [0x7f6e9ad14cdd] 2024-07-26T11:29:53.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6e9ad14cf9] 2024-07-26T11:29:53.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6e9a608c02] 2024-07-26T11:29:53.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6e9a68dc40] 2024-07-26T11:29:53.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:29:53.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73451]: 2024-07-26T11:29:53.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:53 smithi189 podman[73503]: 2024-07-26 11:29:53.735500502 +0000 UTC m=+0.020787854 container died f38729f7ae2f0869a6283da384db0f157158bca9b63f76b5657946e3419e2fe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, GIT_CLEAN=True, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-07-26T11:29:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:53 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/3821381197,v1:172.21.15.189:6835/3821381197] up:rejoin 2024-07-26T11:29:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:53 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:29:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:53 smithi193 ceph-mon[37473]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:29:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:53 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/3821381197,v1:172.21.15.189:6835/3821381197] up:rejoin 2024-07-26T11:29:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:53 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:29:54.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:53 smithi134 ceph-mon[31966]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:29:54.524 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:29:54 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:29:54.055+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:29:54.613 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:54 smithi189 podman[73503]: 2024-07-26 11:29:54.3617433 +0000 UTC m=+0.647030685 container remove f38729f7ae2f0869a6283da384db0f157158bca9b63f76b5657946e3419e2fe6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2) 2024-07-26T11:29:54.613 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:54 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:29:54.944 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:29:54 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T11:29:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:54 smithi189 ceph-mon[36790]: pgmap v921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T11:29:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:54 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:29:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:54 smithi189 ceph-mon[36790]: Cluster is now healthy 2024-07-26T11:29:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:54 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/3821381197,v1:172.21.15.189:6835/3821381197] up:active 2024-07-26T11:29:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:54 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:29:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:54 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:29:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:54 smithi193 ceph-mon[37473]: pgmap v921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T11:29:55.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:54 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:29:55.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:54 smithi193 ceph-mon[37473]: Cluster is now healthy 2024-07-26T11:29:55.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:54 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/3821381197,v1:172.21.15.189:6835/3821381197] up:active 2024-07-26T11:29:55.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:54 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:29:55.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:54 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:29:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:54 smithi134 ceph-mon[31966]: pgmap v921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T11:29:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:54 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:29:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:54 smithi134 ceph-mon[31966]: Cluster is now healthy 2024-07-26T11:29:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:54 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/3821381197,v1:172.21.15.189:6835/3821381197] up:active 2024-07-26T11:29:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:54 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:29:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:54 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:29:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:56 smithi193 ceph-mon[37473]: pgmap v922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:56 smithi134 ceph-mon[31966]: pgmap v922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:56 smithi189 ceph-mon[36790]: pgmap v922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:58.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:29:58.381+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:29:58.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:29:58.381+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:29:58.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:29:58.381+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:29:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:29:58 smithi193 ceph-mon[37473]: pgmap v923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:29:58 smithi134 ceph-mon[31966]: pgmap v923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:29:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:29:58 smithi189 ceph-mon[36790]: pgmap v923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:30:00.889 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:00 smithi134 ceph-mon[31966]: pgmap v924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:30:00.889 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:00 smithi134 ceph-mon[31966]: overall HEALTH_OK 2024-07-26T11:30:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:00 smithi193 ceph-mon[37473]: pgmap v924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:30:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:00 smithi193 ceph-mon[37473]: overall HEALTH_OK 2024-07-26T11:30:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:00 smithi189 ceph-mon[36790]: pgmap v924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:30:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:00 smithi189 ceph-mon[36790]: overall HEALTH_OK 2024-07-26T11:30:02.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:30:02.873+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:30:02.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:30:02.873+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:02.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:30:02.873+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:30:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:02 smithi193 ceph-mon[37473]: pgmap v925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 173 B/s wr, 10 op/s 2024-07-26T11:30:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:02 smithi134 ceph-mon[31966]: pgmap v925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 173 B/s wr, 10 op/s 2024-07-26T11:30:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:02 smithi189 ceph-mon[36790]: pgmap v925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 173 B/s wr, 10 op/s 2024-07-26T11:30:04.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:04 smithi189 ceph-mon[36790]: pgmap v926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:30:04.955 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:04 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 5. 2024-07-26T11:30:04.955 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:04 smithi189 systemd[1]: Stopped Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:30:04.955 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:04 smithi189 systemd[1]: Starting Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:30:04.955 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:04 smithi189 podman[73595]: 2024-07-26 11:30:04.830454076 +0000 UTC m=+0.017692847 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:30:04.955 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:04 smithi189 podman[73595]: 2024-07-26 11:30:04.953972065 +0000 UTC m=+0.141210837 container create 8169e7fab314f11c807efeb4bcbcc0c4d49cf1643f79665bf986f7275caaff56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:30:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:04 smithi193 ceph-mon[37473]: pgmap v926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:30:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:04 smithi134 ceph-mon[31966]: pgmap v926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:30:05.694 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:05 smithi189 podman[73595]: 2024-07-26 11:30:05.214746001 +0000 UTC m=+0.401984788 container init 8169e7fab314f11c807efeb4bcbcc0c4d49cf1643f79665bf986f7275caaff56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:30:05.694 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:05 smithi189 podman[73595]: 2024-07-26 11:30:05.217424642 +0000 UTC m=+0.404663413 container start 8169e7fab314f11c807efeb4bcbcc0c4d49cf1643f79665bf986f7275caaff56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-07-26T11:30:05.694 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:05 smithi189 ceph-mds[73614]: -- 172.21.15.189:0/2579643055 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555cc15d7860 con 0x555cc5779180 2024-07-26T11:30:05.694 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:05 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: starting mds.e at 2024-07-26T11:30:05.694 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:05 smithi189 bash[73595]: 8169e7fab314f11c807efeb4bcbcc0c4d49cf1643f79665bf986f7275caaff56 2024-07-26T11:30:05.694 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:05 smithi189 systemd[1]: Started Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:30:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:05 smithi193 ceph-mon[37473]: daemon mds.e restarted 2024-07-26T11:30:06.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:05 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:30:06.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:05 smithi193 ceph-mon[37473]: osdmap e115: 12 total, 12 up, 12 in 2024-07-26T11:30:06.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:05 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/3552348029,v1:172.21.15.189:6835/3552348029] up:boot 2024-07-26T11:30:06.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:05 smithi193 ceph-mon[37473]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-07-26T11:30:06.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:05 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:30:06.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:30:06.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:05 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T11:30:06.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:05 smithi134 ceph-mon[31966]: daemon mds.e restarted 2024-07-26T11:30:06.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:05 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:30:06.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:05 smithi134 ceph-mon[31966]: osdmap e115: 12 total, 12 up, 12 in 2024-07-26T11:30:06.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:05 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/3552348029,v1:172.21.15.189:6835/3552348029] up:boot 2024-07-26T11:30:06.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:05 smithi134 ceph-mon[31966]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-07-26T11:30:06.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:05 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:30:06.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:30:06.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:05 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T11:30:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:05 smithi189 ceph-mon[36790]: daemon mds.e restarted 2024-07-26T11:30:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:05 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:30:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:05 smithi189 ceph-mon[36790]: osdmap e115: 12 total, 12 up, 12 in 2024-07-26T11:30:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:05 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/3552348029,v1:172.21.15.189:6835/3552348029] up:boot 2024-07-26T11:30:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:05 smithi189 ceph-mon[36790]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-07-26T11:30:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:05 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:30:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-07-26T11:30:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:05 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T11:30:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:06 smithi193 ceph-mon[37473]: pgmap v927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:30:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:06 smithi134 ceph-mon[31966]: pgmap v927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:30:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:06 smithi189 ceph-mon[36790]: pgmap v927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:30:08.009 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:07 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/3307697142,v1:172.21.15.134:6839/3307697142] up:resolve 2024-07-26T11:30:08.010 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:07 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T11:30:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:07 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/3307697142,v1:172.21.15.134:6839/3307697142] up:resolve 2024-07-26T11:30:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:07 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T11:30:08.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:07 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/3307697142,v1:172.21.15.134:6839/3307697142] up:resolve 2024-07-26T11:30:08.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:07 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T11:30:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:08 smithi193 ceph-mon[37473]: pgmap v929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:30:09.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:08 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:30:09.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:08 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/3307697142,v1:172.21.15.134:6839/3307697142] up:reconnect 2024-07-26T11:30:09.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:08 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:30:09.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:08 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0119997 2024-07-26T11:30:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:08 smithi134 ceph-mon[31966]: pgmap v929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:30:09.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:08 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:30:09.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:08 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/3307697142,v1:172.21.15.134:6839/3307697142] up:reconnect 2024-07-26T11:30:09.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:08 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:30:09.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:08 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0119997 2024-07-26T11:30:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:08 smithi189 ceph-mon[36790]: pgmap v929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:30:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:08 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:30:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:08 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/3307697142,v1:172.21.15.134:6839/3307697142] up:reconnect 2024-07-26T11:30:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:08 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:30:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:08 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0119997 2024-07-26T11:30:09.675 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:30:09.675 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: *** Caught signal (Aborted) ** 2024-07-26T11:30:09.675 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: in thread 7fb29bc00640 thread_name:ms_dispatch 2024-07-26T11:30:09.675 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:30:09.675 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb2a1e3e6f0] 2024-07-26T11:30:09.675 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb2a1e8b94c] 2024-07-26T11:30:09.675 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 3: raise() 2024-07-26T11:30:09.675 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 4: abort() 2024-07-26T11:30:09.675 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb2a1e2871b] 2024-07-26T11:30:09.675 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 6: /lib64/libc.so.6(+0x37386) [0x7fb2a1e37386] 2024-07-26T11:30:09.675 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5624c42546b9] 2024-07-26T11:30:09.675 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5624c441280c] 2024-07-26T11:30:09.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5624c43522ca] 2024-07-26T11:30:09.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5624c4322753] 2024-07-26T11:30:09.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5624c4323a1b] 2024-07-26T11:30:09.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5624c4307752] 2024-07-26T11:30:09.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5624c4255a56] 2024-07-26T11:30:09.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5624c4259637] 2024-07-26T11:30:09.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5624c4259f5e] 2024-07-26T11:30:09.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5624c423d03a] 2024-07-26T11:30:09.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb2a299b407] 2024-07-26T11:30:09.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 18: (DispatchQueue::entry()+0x60f) [0x7fb2a299bcf9] 2024-07-26T11:30:09.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb2a2a785a1] 2024-07-26T11:30:09.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 20: (Thread::entry_wrapper()+0x43) [0x7fb2a2825cdd] 2024-07-26T11:30:09.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb2a2825cf9] 2024-07-26T11:30:09.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb2a1e89c02] 2024-07-26T11:30:09.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb2a1f0ec40] 2024-07-26T11:30:09.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 2024-07-26T11:30:09.192+0000 7fb29bc00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:30:09.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: in thread 7fb29bc00640 thread_name:ms_dispatch 2024-07-26T11:30:09.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 2024-07-26T11:30:09.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:30:09.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb2a1e3e6f0] 2024-07-26T11:30:09.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb2a1e8b94c] 2024-07-26T11:30:09.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 3: raise() 2024-07-26T11:30:09.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 4: abort() 2024-07-26T11:30:09.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb2a1e2871b] 2024-07-26T11:30:09.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 6: /lib64/libc.so.6(+0x37386) [0x7fb2a1e37386] 2024-07-26T11:30:09.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5624c42546b9] 2024-07-26T11:30:09.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5624c441280c] 2024-07-26T11:30:09.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5624c43522ca] 2024-07-26T11:30:09.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5624c4322753] 2024-07-26T11:30:09.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5624c4323a1b] 2024-07-26T11:30:09.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5624c4307752] 2024-07-26T11:30:09.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5624c4255a56] 2024-07-26T11:30:09.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5624c4259637] 2024-07-26T11:30:09.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5624c4259f5e] 2024-07-26T11:30:09.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5624c423d03a] 2024-07-26T11:30:09.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb2a299b407] 2024-07-26T11:30:09.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 18: (DispatchQueue::entry()+0x60f) [0x7fb2a299bcf9] 2024-07-26T11:30:09.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb2a2a785a1] 2024-07-26T11:30:09.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 20: (Thread::entry_wrapper()+0x43) [0x7fb2a2825cdd] 2024-07-26T11:30:09.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb2a2825cf9] 2024-07-26T11:30:09.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb2a1e89c02] 2024-07-26T11:30:09.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb2a1f0ec40] 2024-07-26T11:30:09.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:30:09.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 2024-07-26T11:30:09.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 0> 2024-07-26T11:30:09.192+0000 7fb29bc00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:30:09.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: in thread 7fb29bc00640 thread_name:ms_dispatch 2024-07-26T11:30:09.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 2024-07-26T11:30:09.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:30:09.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb2a1e3e6f0] 2024-07-26T11:30:09.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb2a1e8b94c] 2024-07-26T11:30:09.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 3: raise() 2024-07-26T11:30:09.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 4: abort() 2024-07-26T11:30:09.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb2a1e2871b] 2024-07-26T11:30:09.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 6: /lib64/libc.so.6(+0x37386) [0x7fb2a1e37386] 2024-07-26T11:30:09.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5624c42546b9] 2024-07-26T11:30:09.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5624c441280c] 2024-07-26T11:30:09.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5624c43522ca] 2024-07-26T11:30:09.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5624c4322753] 2024-07-26T11:30:09.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5624c4323a1b] 2024-07-26T11:30:09.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5624c4307752] 2024-07-26T11:30:09.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5624c4255a56] 2024-07-26T11:30:09.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5624c4259637] 2024-07-26T11:30:09.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5624c4259f5e] 2024-07-26T11:30:09.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5624c423d03a] 2024-07-26T11:30:09.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb2a299b407] 2024-07-26T11:30:09.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 18: (DispatchQueue::entry()+0x60f) [0x7fb2a299bcf9] 2024-07-26T11:30:09.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb2a2a785a1] 2024-07-26T11:30:09.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 20: (Thread::entry_wrapper()+0x43) [0x7fb2a2825cdd] 2024-07-26T11:30:09.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb2a2825cf9] 2024-07-26T11:30:09.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb2a1e89c02] 2024-07-26T11:30:09.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb2a1f0ec40] 2024-07-26T11:30:09.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:30:09.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 2024-07-26T11:30:09.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 0> 2024-07-26T11:30:09.192+0000 7fb29bc00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:30:09.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: in thread 7fb29bc00640 thread_name:ms_dispatch 2024-07-26T11:30:09.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 2024-07-26T11:30:09.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:30:09.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb2a1e3e6f0] 2024-07-26T11:30:09.683 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb2a1e8b94c] 2024-07-26T11:30:09.683 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 3: raise() 2024-07-26T11:30:09.683 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 4: abort() 2024-07-26T11:30:09.683 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb2a1e2871b] 2024-07-26T11:30:09.683 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 6: /lib64/libc.so.6(+0x37386) [0x7fb2a1e37386] 2024-07-26T11:30:09.683 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5624c42546b9] 2024-07-26T11:30:09.683 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5624c441280c] 2024-07-26T11:30:09.683 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5624c43522ca] 2024-07-26T11:30:09.683 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5624c4322753] 2024-07-26T11:30:09.683 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5624c4323a1b] 2024-07-26T11:30:09.683 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5624c4307752] 2024-07-26T11:30:09.683 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5624c4255a56] 2024-07-26T11:30:09.683 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5624c4259637] 2024-07-26T11:30:09.683 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5624c4259f5e] 2024-07-26T11:30:09.684 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5624c423d03a] 2024-07-26T11:30:09.684 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb2a299b407] 2024-07-26T11:30:09.684 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 18: (DispatchQueue::entry()+0x60f) [0x7fb2a299bcf9] 2024-07-26T11:30:09.684 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb2a2a785a1] 2024-07-26T11:30:09.684 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 20: (Thread::entry_wrapper()+0x43) [0x7fb2a2825cdd] 2024-07-26T11:30:09.684 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb2a2825cf9] 2024-07-26T11:30:09.684 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb2a1e89c02] 2024-07-26T11:30:09.684 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb2a1f0ec40] 2024-07-26T11:30:09.684 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:30:09.684 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[78183]: 2024-07-26T11:30:09.684 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:09 smithi134 podman[112266]: 2024-07-26 11:30:09.352786569 +0000 UTC m=+0.034323443 container died c7b6500323c2247dda218cf14db22f45ed63b84a8ec17e17989195bbaffb25bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:30:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/3307697142,v1:172.21.15.134:6839/3307697142] up:rejoin 2024-07-26T11:30:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:30:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-mon[31966]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:30:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:30:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-mon[31966]: Cluster is now healthy 2024-07-26T11:30:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/3307697142,v1:172.21.15.134:6839/3307697142] up:active 2024-07-26T11:30:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:09 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:30:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:09 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/3307697142,v1:172.21.15.134:6839/3307697142] up:rejoin 2024-07-26T11:30:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:09 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:30:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:09 smithi189 ceph-mon[36790]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:30:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:09 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:30:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:09 smithi189 ceph-mon[36790]: Cluster is now healthy 2024-07-26T11:30:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:09 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/3307697142,v1:172.21.15.134:6839/3307697142] up:active 2024-07-26T11:30:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:09 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:30:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:09 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/3307697142,v1:172.21.15.134:6839/3307697142] up:rejoin 2024-07-26T11:30:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:09 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:30:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:09 smithi193 ceph-mon[37473]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:30:10.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:09 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:30:10.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:09 smithi193 ceph-mon[37473]: Cluster is now healthy 2024-07-26T11:30:10.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:09 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/3307697142,v1:172.21.15.134:6839/3307697142] up:active 2024-07-26T11:30:10.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:09 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:30:10.673 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:10 smithi134 podman[112266]: 2024-07-26 11:30:10.290514369 +0000 UTC m=+0.972051259 container remove c7b6500323c2247dda218cf14db22f45ed63b84a8ec17e17989195bbaffb25bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:30:10.673 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:10 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:30:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:10 smithi134 ceph-mon[31966]: pgmap v930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:11.173 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:10 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-07-26T11:30:11.173 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:10 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Consumed 2.370s CPU time. 2024-07-26T11:30:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:10 smithi189 ceph-mon[36790]: pgmap v930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:10 smithi193 ceph-mon[37473]: pgmap v930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:12 smithi134 ceph-mon[31966]: pgmap v931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:30:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:12 smithi189 ceph-mon[36790]: pgmap v931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:30:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:12 smithi193 ceph-mon[37473]: pgmap v931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:30:13.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:30:13.386+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:30:13.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:30:13.386+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:13.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:30:13.386+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:30:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:14 smithi134 ceph-mon[31966]: pgmap v932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:30:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:14 smithi189 ceph-mon[36790]: pgmap v932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:30:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:14 smithi193 ceph-mon[37473]: pgmap v932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.4 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:30:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:16 smithi134 ceph-mon[31966]: pgmap v933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:30:17.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:16 smithi189 ceph-mon[36790]: pgmap v933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:30:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:16 smithi193 ceph-mon[37473]: pgmap v933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:30:17.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:30:17.875+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:30:17.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:30:17.875+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:30:17.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:30:17.875+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:18 smithi134 ceph-mon[31966]: pgmap v934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 175 B/s wr, 10 op/s 2024-07-26T11:30:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:18 smithi189 ceph-mon[36790]: pgmap v934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 175 B/s wr, 10 op/s 2024-07-26T11:30:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:18 smithi193 ceph-mon[37473]: pgmap v934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 175 B/s wr, 10 op/s 2024-07-26T11:30:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:20 smithi134 ceph-mon[31966]: pgmap v935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:30:21.172 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:20 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 1. 2024-07-26T11:30:21.172 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:20 smithi134 systemd[1]: Stopped Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:30:21.172 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:20 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Consumed 2.370s CPU time. 2024-07-26T11:30:21.173 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:20 smithi134 systemd[1]: Starting Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:30:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:20 smithi189 ceph-mon[36790]: pgmap v935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:30:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:20 smithi193 ceph-mon[37473]: pgmap v935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:30:21.605 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:21 smithi134 podman[112356]: 2024-07-26 11:30:21.211348839 +0000 UTC m=+0.031357781 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:30:21.605 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:21 smithi134 podman[112356]: 2024-07-26 11:30:21.324710955 +0000 UTC m=+0.144719916 container create 380667419a639a6b93060ad9fc1108f6de2cbda129416af6c20e8fec264c669e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-07-26T11:30:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:21 smithi134 ceph-mon[31966]: daemon mds.g restarted 2024-07-26T11:30:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:21 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:30:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:21 smithi134 ceph-mon[31966]: osdmap e116: 12 total, 12 up, 12 in 2024-07-26T11:30:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:21 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/500511838,v1:172.21.15.134:6839/500511838] up:boot 2024-07-26T11:30:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:21 smithi134 ceph-mon[31966]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T11:30:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:21 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:30:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:30:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:21 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T11:30:21.924 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:21 smithi134 podman[112356]: 2024-07-26 11:30:21.603794223 +0000 UTC m=+0.423803241 container init 380667419a639a6b93060ad9fc1108f6de2cbda129416af6c20e8fec264c669e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-07-26T11:30:21.924 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:21 smithi134 podman[112356]: 2024-07-26 11:30:21.613633724 +0000 UTC m=+0.433642691 container start 380667419a639a6b93060ad9fc1108f6de2cbda129416af6c20e8fec264c669e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD) 2024-07-26T11:30:21.924 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:21 smithi134 ceph-mds[112375]: -- 172.21.15.134:0/1852820174 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563b6bfa7c20 con 0x563b70149180 2024-07-26T11:30:21.924 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: starting mds.g at 2024-07-26T11:30:21.924 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:21 smithi134 bash[112356]: 380667419a639a6b93060ad9fc1108f6de2cbda129416af6c20e8fec264c669e 2024-07-26T11:30:21.924 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:21 smithi134 systemd[1]: Started Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:30:22.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:21 smithi189 ceph-mon[36790]: daemon mds.g restarted 2024-07-26T11:30:22.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:21 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:30:22.096 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:21 smithi189 ceph-mon[36790]: osdmap e116: 12 total, 12 up, 12 in 2024-07-26T11:30:22.096 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:21 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/500511838,v1:172.21.15.134:6839/500511838] up:boot 2024-07-26T11:30:22.096 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:21 smithi189 ceph-mon[36790]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T11:30:22.096 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:21 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:30:22.096 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:30:22.096 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:21 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T11:30:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:21 smithi193 ceph-mon[37473]: daemon mds.g restarted 2024-07-26T11:30:22.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:21 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:30:22.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:21 smithi193 ceph-mon[37473]: osdmap e116: 12 total, 12 up, 12 in 2024-07-26T11:30:22.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:21 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/500511838,v1:172.21.15.134:6839/500511838] up:boot 2024-07-26T11:30:22.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:21 smithi193 ceph-mon[37473]: Standby daemon mds.e assigned to filesystem cephfs as rank 1 2024-07-26T11:30:22.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:21 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-07-26T11:30:22.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:30:22.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:21 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-07-26T11:30:22.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:30:22.301+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240034f0 con 0x7f9b4c10f330 2024-07-26T11:30:22.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:30:22.302+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b240034f0 con 0x7f9b4c10f330 2024-07-26T11:30:22.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:30:22.374+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b200034c0 con 0x55c65e5ee670 2024-07-26T11:30:22.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:30:22.375+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 5 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b200034c0 con 0x55c65e5ee670 2024-07-26T11:30:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:22 smithi134 ceph-mon[31966]: pgmap v936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:30:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:22 smithi189 ceph-mon[36790]: pgmap v936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:30:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:22 smithi193 ceph-mon[37473]: pgmap v936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:30:24.111 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:23 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/3552348029,v1:172.21.15.189:6835/3552348029] up:resolve 2024-07-26T11:30:24.111 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:23 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T11:30:24.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:23 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/3552348029,v1:172.21.15.189:6835/3552348029] up:resolve 2024-07-26T11:30:24.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:23 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T11:30:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:23 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/3552348029,v1:172.21.15.189:6835/3552348029] up:resolve 2024-07-26T11:30:24.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:23 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-07-26T11:30:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:24 smithi134 ceph-mon[31966]: pgmap v938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 10 op/s 2024-07-26T11:30:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:24 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:30:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:24 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/3552348029,v1:172.21.15.189:6835/3552348029] up:reconnect 2024-07-26T11:30:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:24 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:30:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:24 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0119998 2024-07-26T11:30:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:24 smithi189 ceph-mon[36790]: pgmap v938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 10 op/s 2024-07-26T11:30:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:24 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:30:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:24 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/3552348029,v1:172.21.15.189:6835/3552348029] up:reconnect 2024-07-26T11:30:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:24 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:30:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:24 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0119998 2024-07-26T11:30:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:24 smithi193 ceph-mon[37473]: pgmap v938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 10 op/s 2024-07-26T11:30:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:24 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:30:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:24 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/3552348029,v1:172.21.15.189:6835/3552348029] up:reconnect 2024-07-26T11:30:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:24 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:30:25.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:24 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0119998 2024-07-26T11:30:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:25 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/3552348029,v1:172.21.15.189:6835/3552348029] up:rejoin 2024-07-26T11:30:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:25 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:30:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:25 smithi134 ceph-mon[31966]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:30:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:25 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:30:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:25 smithi134 ceph-mon[31966]: Cluster is now healthy 2024-07-26T11:30:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/3552348029,v1:172.21.15.189:6835/3552348029] up:rejoin 2024-07-26T11:30:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:30:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-mon[36790]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:30:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:30:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-mon[36790]: Cluster is now healthy 2024-07-26T11:30:26.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:30:26.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: *** Caught signal (Aborted) ** 2024-07-26T11:30:26.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: in thread 7f0a5c854640 thread_name:ms_dispatch 2024-07-26T11:30:26.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:30:26.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0a61c2b6f0] 2024-07-26T11:30:26.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0a61c7894c] 2024-07-26T11:30:26.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 3: raise() 2024-07-26T11:30:26.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 4: abort() 2024-07-26T11:30:26.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0a61c1571b] 2024-07-26T11:30:26.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 6: /lib64/libc.so.6(+0x37386) [0x7f0a61c24386] 2024-07-26T11:30:26.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x555cbec526b9] 2024-07-26T11:30:26.195 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x555cbee1080c] 2024-07-26T11:30:26.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x555cbed502ca] 2024-07-26T11:30:26.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x555cbed20753] 2024-07-26T11:30:26.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x555cbed21a1b] 2024-07-26T11:30:26.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x555cbed05752] 2024-07-26T11:30:26.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x555cbec53a56] 2024-07-26T11:30:26.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x555cbec57637] 2024-07-26T11:30:26.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x555cbec57f5e] 2024-07-26T11:30:26.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x555cbec3b03a] 2024-07-26T11:30:26.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0a624f8407] 2024-07-26T11:30:26.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 18: (DispatchQueue::entry()+0x60f) [0x7f0a624f8cf9] 2024-07-26T11:30:26.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0a625d55a1] 2024-07-26T11:30:26.196 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 20: (Thread::entry_wrapper()+0x43) [0x7f0a62382cdd] 2024-07-26T11:30:26.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0a62382cf9] 2024-07-26T11:30:26.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0a61c76c02] 2024-07-26T11:30:26.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0a61cfbc40] 2024-07-26T11:30:26.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 2024-07-26T11:30:25.866+0000 7f0a5c854640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:30:26.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: in thread 7f0a5c854640 thread_name:ms_dispatch 2024-07-26T11:30:26.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 2024-07-26T11:30:26.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:30:26.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0a61c2b6f0] 2024-07-26T11:30:26.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0a61c7894c] 2024-07-26T11:30:26.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 3: raise() 2024-07-26T11:30:26.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 4: abort() 2024-07-26T11:30:26.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0a61c1571b] 2024-07-26T11:30:26.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 6: /lib64/libc.so.6(+0x37386) [0x7f0a61c24386] 2024-07-26T11:30:26.197 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x555cbec526b9] 2024-07-26T11:30:26.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x555cbee1080c] 2024-07-26T11:30:26.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x555cbed502ca] 2024-07-26T11:30:26.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x555cbed20753] 2024-07-26T11:30:26.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x555cbed21a1b] 2024-07-26T11:30:26.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x555cbed05752] 2024-07-26T11:30:26.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x555cbec53a56] 2024-07-26T11:30:26.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x555cbec57637] 2024-07-26T11:30:26.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x555cbec57f5e] 2024-07-26T11:30:26.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x555cbec3b03a] 2024-07-26T11:30:26.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0a624f8407] 2024-07-26T11:30:26.198 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 18: (DispatchQueue::entry()+0x60f) [0x7f0a624f8cf9] 2024-07-26T11:30:26.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0a625d55a1] 2024-07-26T11:30:26.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 20: (Thread::entry_wrapper()+0x43) [0x7f0a62382cdd] 2024-07-26T11:30:26.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0a62382cf9] 2024-07-26T11:30:26.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0a61c76c02] 2024-07-26T11:30:26.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0a61cfbc40] 2024-07-26T11:30:26.199 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:30:26.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 2024-07-26T11:30:26.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 0> 2024-07-26T11:30:25.866+0000 7f0a5c854640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:30:26.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: in thread 7f0a5c854640 thread_name:ms_dispatch 2024-07-26T11:30:26.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 2024-07-26T11:30:26.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:30:26.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0a61c2b6f0] 2024-07-26T11:30:26.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0a61c7894c] 2024-07-26T11:30:26.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 3: raise() 2024-07-26T11:30:26.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 4: abort() 2024-07-26T11:30:26.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0a61c1571b] 2024-07-26T11:30:26.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 6: /lib64/libc.so.6(+0x37386) [0x7f0a61c24386] 2024-07-26T11:30:26.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x555cbec526b9] 2024-07-26T11:30:26.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x555cbee1080c] 2024-07-26T11:30:26.200 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x555cbed502ca] 2024-07-26T11:30:26.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x555cbed20753] 2024-07-26T11:30:26.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x555cbed21a1b] 2024-07-26T11:30:26.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x555cbed05752] 2024-07-26T11:30:26.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x555cbec53a56] 2024-07-26T11:30:26.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x555cbec57637] 2024-07-26T11:30:26.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x555cbec57f5e] 2024-07-26T11:30:26.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x555cbec3b03a] 2024-07-26T11:30:26.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0a624f8407] 2024-07-26T11:30:26.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 18: (DispatchQueue::entry()+0x60f) [0x7f0a624f8cf9] 2024-07-26T11:30:26.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0a625d55a1] 2024-07-26T11:30:26.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 20: (Thread::entry_wrapper()+0x43) [0x7f0a62382cdd] 2024-07-26T11:30:26.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0a62382cf9] 2024-07-26T11:30:26.201 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0a61c76c02] 2024-07-26T11:30:26.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0a61cfbc40] 2024-07-26T11:30:26.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:30:26.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 2024-07-26T11:30:26.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 0> 2024-07-26T11:30:25.866+0000 7f0a5c854640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:30:26.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: in thread 7f0a5c854640 thread_name:ms_dispatch 2024-07-26T11:30:26.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 2024-07-26T11:30:26.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:30:26.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f0a61c2b6f0] 2024-07-26T11:30:26.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f0a61c7894c] 2024-07-26T11:30:26.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 3: raise() 2024-07-26T11:30:26.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 4: abort() 2024-07-26T11:30:26.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 5: /lib64/libc.so.6(+0x2871b) [0x7f0a61c1571b] 2024-07-26T11:30:26.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 6: /lib64/libc.so.6(+0x37386) [0x7f0a61c24386] 2024-07-26T11:30:26.202 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x555cbec526b9] 2024-07-26T11:30:26.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x555cbee1080c] 2024-07-26T11:30:26.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x555cbed502ca] 2024-07-26T11:30:26.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x555cbed20753] 2024-07-26T11:30:26.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x555cbed21a1b] 2024-07-26T11:30:26.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x555cbed05752] 2024-07-26T11:30:26.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x555cbec53a56] 2024-07-26T11:30:26.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x555cbec57637] 2024-07-26T11:30:26.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x555cbec57f5e] 2024-07-26T11:30:26.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x555cbec3b03a] 2024-07-26T11:30:26.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f0a624f8407] 2024-07-26T11:30:26.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 18: (DispatchQueue::entry()+0x60f) [0x7f0a624f8cf9] 2024-07-26T11:30:26.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f0a625d55a1] 2024-07-26T11:30:26.203 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 20: (Thread::entry_wrapper()+0x43) [0x7f0a62382cdd] 2024-07-26T11:30:26.204 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 21: (Thread::_entry_func(void*)+0xd) [0x7f0a62382cf9] 2024-07-26T11:30:26.204 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 22: /lib64/libc.so.6(+0x89c02) [0x7f0a61c76c02] 2024-07-26T11:30:26.204 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f0a61cfbc40] 2024-07-26T11:30:26.204 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:30:26.204 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e[73610]: 2024-07-26T11:30:26.204 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:25 smithi189 podman[73648]: 2024-07-26 11:30:25.907799693 +0000 UTC m=+0.019123705 container died 8169e7fab314f11c807efeb4bcbcc0c4d49cf1643f79665bf986f7275caaff56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:30:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:25 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/3552348029,v1:172.21.15.189:6835/3552348029] up:rejoin 2024-07-26T11:30:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:25 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:30:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:25 smithi193 ceph-mon[37473]: daemon mds.e is now active in filesystem cephfs as rank 1 2024-07-26T11:30:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:25 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:30:26.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:25 smithi193 ceph-mon[37473]: Cluster is now healthy 2024-07-26T11:30:26.891 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:26 smithi189 podman[73648]: 2024-07-26 11:30:26.55338041 +0000 UTC m=+0.664704468 container remove 8169e7fab314f11c807efeb4bcbcc0c4d49cf1643f79665bf986f7275caaff56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-e, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_CLEAN=True, ceph=True) 2024-07-26T11:30:26.891 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:26 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:30:26.892 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:26 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T11:30:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:26 smithi134 ceph-mon[31966]: pgmap v939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T11:30:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:26 smithi134 ceph-mon[31966]: 1 slow requests, 1 included below; oldest blocked for > 184.462146 secs 2024-07-26T11:30:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:26 smithi134 ceph-mon[31966]: slow request 184.462145 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T11:30:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:26 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.189:6834/3552348029,v1:172.21.15.189:6835/3552348029] up:active 2024-07-26T11:30:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:26 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:30:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:26 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:30:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:30:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:26 smithi189 ceph-mon[36790]: pgmap v939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T11:30:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:26 smithi189 ceph-mon[36790]: 1 slow requests, 1 included below; oldest blocked for > 184.462146 secs 2024-07-26T11:30:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:26 smithi189 ceph-mon[36790]: slow request 184.462145 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T11:30:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:26 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.189:6834/3552348029,v1:172.21.15.189:6835/3552348029] up:active 2024-07-26T11:30:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:26 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:30:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:26 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:30:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:30:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:26 smithi193 ceph-mon[37473]: pgmap v939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T11:30:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:26 smithi193 ceph-mon[37473]: 1 slow requests, 1 included below; oldest blocked for > 184.462146 secs 2024-07-26T11:30:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:26 smithi193 ceph-mon[37473]: slow request 184.462145 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T11:30:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:26 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.189:6834/3552348029,v1:172.21.15.189:6835/3552348029] up:active 2024-07-26T11:30:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:26 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 2 up:standby 5 up:active 2024-07-26T11:30:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:26 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:30:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:30:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:30:28.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:30:28.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:30:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:30:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:30:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:30:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:30:28.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:30:28.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:30:28.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:30:28.399 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:30:28.397+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:30:28.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:30:28.397+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:28.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:30:28.397+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:30:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:28 smithi134 ceph-mon[31966]: pgmap v940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:30:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:28 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T11:30:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:28 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T11:30:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:28 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T11:30:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:28 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T11:30:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:28 smithi189 ceph-mon[36790]: pgmap v940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:30:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:28 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T11:30:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:28 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T11:30:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:28 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T11:30:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:28 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T11:30:29.274 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:30:29 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:30:29.059+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:30:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:28 smithi193 ceph-mon[37473]: pgmap v940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:30:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:28 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T11:30:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:28 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T11:30:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:28 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T11:30:29.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:28 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T11:30:30.155 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:29 smithi134 ceph-mon[31966]: pgmap v941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:30:30.155 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:29 smithi134 ceph-mon[31966]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T11:30:30.155 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:29 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] up:active 2024-07-26T11:30:30.155 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:29 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:30:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:29 smithi193 ceph-mon[37473]: pgmap v941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:30:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:29 smithi193 ceph-mon[37473]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T11:30:30.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:29 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] up:active 2024-07-26T11:30:30.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:29 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:30:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:29 smithi189 ceph-mon[36790]: pgmap v941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:30:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:29 smithi189 ceph-mon[36790]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T11:30:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:29 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.193:6836/1968168008,v1:172.21.15.193:6837/1968168008] up:active 2024-07-26T11:30:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:29 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:30:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:30 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 189.462271 secs 2024-07-26T11:30:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:30 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 189.462271 secs 2024-07-26T11:30:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:30 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 189.462271 secs 2024-07-26T11:30:32.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:31 smithi134 ceph-mon[31966]: pgmap v942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:30:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:31 smithi193 ceph-mon[37473]: pgmap v942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:30:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:31 smithi189 ceph-mon[36790]: pgmap v942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:30:32.863 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:30:32.861+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980034c0 con 0x7f21c8105550 2024-07-26T11:30:32.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:30:32.862+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980034c0 con 0x7f21c8105550 2024-07-26T11:30:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:30:32.868+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21940034c0 con 0x55a08e2d94d0 2024-07-26T11:30:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:30:32.869+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f21940034c0 con 0x55a08e2d94d0 2024-07-26T11:30:32.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:30:32.886+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:30:32.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:30:32.886+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:30:32.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:30:32.886+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:34 smithi134 ceph-mon[31966]: pgmap v943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 182 B/s wr, 12 op/s 2024-07-26T11:30:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:34 smithi189 ceph-mon[36790]: pgmap v943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 182 B/s wr, 12 op/s 2024-07-26T11:30:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:34 smithi193 ceph-mon[37473]: pgmap v943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 182 B/s wr, 12 op/s 2024-07-26T11:30:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:36 smithi134 ceph-mon[31966]: pgmap v944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:30:36.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 194.462387 secs 2024-07-26T11:30:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:36 smithi189 ceph-mon[36790]: pgmap v944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:30:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 194.462387 secs 2024-07-26T11:30:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:36 smithi193 ceph-mon[37473]: pgmap v944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:30:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 194.462387 secs 2024-07-26T11:30:37.444 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:36 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Scheduled restart job, restart counter is at 6. 2024-07-26T11:30:37.444 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:36 smithi189 systemd[1]: Stopped Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:30:37.444 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:36 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Start request repeated too quickly. 2024-07-26T11:30:37.444 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:36 smithi189 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.e.service: Failed with result 'exit-code'. 2024-07-26T11:30:37.444 INFO:journalctl@ceph.mds.e.smithi189.stdout:Jul 26 11:30:36 smithi189 systemd[1]: Failed to start Ceph mds.e for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:30:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:38 smithi134 ceph-mon[31966]: pgmap v945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:30:38.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:38 smithi189 ceph-mon[36790]: pgmap v945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:30:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:38 smithi193 ceph-mon[37473]: pgmap v945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:30:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:39 smithi134 ceph-mon[31966]: 1 slow requests, 1 included below; oldest blocked for > 180.037553 secs 2024-07-26T11:30:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:39 smithi134 ceph-mon[31966]: slow request 180.037552 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T11:30:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:39 smithi189 ceph-mon[36790]: 1 slow requests, 1 included below; oldest blocked for > 180.037553 secs 2024-07-26T11:30:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:39 smithi189 ceph-mon[36790]: slow request 180.037552 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T11:30:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:39 smithi193 ceph-mon[37473]: 1 slow requests, 1 included below; oldest blocked for > 180.037553 secs 2024-07-26T11:30:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:39 smithi193 ceph-mon[37473]: slow request 180.037552 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T11:30:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:40 smithi134 ceph-mon[31966]: pgmap v946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:30:40.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:40 smithi189 ceph-mon[36790]: pgmap v946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:30:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:40 smithi193 ceph-mon[37473]: pgmap v946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:30:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:41 smithi134 ceph-mon[31966]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T11:30:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:41 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:active 2024-07-26T11:30:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:41 smithi134 ceph-mon[31966]: Replacing daemon mds.e as rank 1 with standby daemon mds.g 2024-07-26T11:30:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:41 smithi134 ceph-mon[31966]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-07-26T11:30:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:41 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:30:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:41 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:30:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:41 smithi134 ceph-mon[31966]: osdmap e117: 12 total, 12 up, 12 in 2024-07-26T11:30:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:41 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:30:41.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 199.462479 secs 2024-07-26T11:30:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:41 smithi189 ceph-mon[36790]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T11:30:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:41 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:active 2024-07-26T11:30:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:41 smithi189 ceph-mon[36790]: Replacing daemon mds.e as rank 1 with standby daemon mds.g 2024-07-26T11:30:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:41 smithi189 ceph-mon[36790]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-07-26T11:30:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:41 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:30:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:41 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:30:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:41 smithi189 ceph-mon[36790]: osdmap e117: 12 total, 12 up, 12 in 2024-07-26T11:30:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:41 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:30:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 199.462479 secs 2024-07-26T11:30:41.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:41 smithi193 ceph-mon[37473]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-07-26T11:30:41.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:41 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:active 2024-07-26T11:30:41.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:41 smithi193 ceph-mon[37473]: Replacing daemon mds.e as rank 1 with standby daemon mds.g 2024-07-26T11:30:41.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:41 smithi193 ceph-mon[37473]: MDS daemon mds.e is removed because it is dead or otherwise unavailable. 2024-07-26T11:30:41.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:41 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:30:41.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:41 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:30:41.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:41 smithi193 ceph-mon[37473]: osdmap e117: 12 total, 12 up, 12 in 2024-07-26T11:30:41.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:41 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:30:41.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 199.462479 secs 2024-07-26T11:30:42.357 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:42 smithi134 ceph-mon[31966]: pgmap v948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:42.357 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:42 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/500511838,v1:172.21.15.134:6839/500511838] up:resolve 2024-07-26T11:30:42.357 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:42 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:30:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:42 smithi189 ceph-mon[36790]: pgmap v948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:42 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/500511838,v1:172.21.15.134:6839/500511838] up:resolve 2024-07-26T11:30:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:42 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:30:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:42 smithi193 ceph-mon[37473]: pgmap v948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:42 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/500511838,v1:172.21.15.134:6839/500511838] up:resolve 2024-07-26T11:30:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:42 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:30:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:30:43.413+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:30:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:30:43.413+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:43.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:30:43.413+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:30:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:43 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:30:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:43 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/500511838,v1:172.21.15.134:6839/500511838] up:reconnect 2024-07-26T11:30:43.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:43 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:30:43.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:43 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:30:43.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:43 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:30:43.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:43 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/500511838,v1:172.21.15.134:6839/500511838] up:reconnect 2024-07-26T11:30:43.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:43 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:30:43.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:43 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:30:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:43 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:30:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:43 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/500511838,v1:172.21.15.134:6839/500511838] up:reconnect 2024-07-26T11:30:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:43 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:30:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:43 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:30:44.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-mon[31966]: pgmap v949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:44.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/500511838,v1:172.21.15.134:6839/500511838] up:rejoin 2024-07-26T11:30:44.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:30:44.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-mon[31966]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:30:44.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 185.037694 secs 2024-07-26T11:30:44.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:30:44.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: *** Caught signal (Aborted) ** 2024-07-26T11:30:44.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: in thread 7feb9f200640 thread_name:ms_dispatch 2024-07-26T11:30:44.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:30:44.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7feba563e6f0] 2024-07-26T11:30:44.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 2: /lib64/libc.so.6(+0x8b94c) [0x7feba568b94c] 2024-07-26T11:30:44.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 3: raise() 2024-07-26T11:30:44.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 4: abort() 2024-07-26T11:30:44.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 5: /lib64/libc.so.6(+0x2871b) [0x7feba562871b] 2024-07-26T11:30:44.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 6: /lib64/libc.so.6(+0x37386) [0x7feba5637386] 2024-07-26T11:30:44.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563b399c56b9] 2024-07-26T11:30:44.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563b39b8380c] 2024-07-26T11:30:44.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563b39ac32ca] 2024-07-26T11:30:44.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563b39a93753] 2024-07-26T11:30:44.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563b39a94a1b] 2024-07-26T11:30:44.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563b39a78752] 2024-07-26T11:30:44.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563b399c6a56] 2024-07-26T11:30:44.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563b399ca637] 2024-07-26T11:30:44.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563b399caf5e] 2024-07-26T11:30:44.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563b399ae03a] 2024-07-26T11:30:44.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7feba619b407] 2024-07-26T11:30:44.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 18: (DispatchQueue::entry()+0x60f) [0x7feba619bcf9] 2024-07-26T11:30:44.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7feba62785a1] 2024-07-26T11:30:44.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 20: (Thread::entry_wrapper()+0x43) [0x7feba6025cdd] 2024-07-26T11:30:44.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 21: (Thread::_entry_func(void*)+0xd) [0x7feba6025cf9] 2024-07-26T11:30:44.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 22: /lib64/libc.so.6(+0x89c02) [0x7feba5689c02] 2024-07-26T11:30:44.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 23: /lib64/libc.so.6(+0x10ec40) [0x7feba570ec40] 2024-07-26T11:30:44.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 2024-07-26T11:30:44.136+0000 7feb9f200640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:30:44.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: in thread 7feb9f200640 thread_name:ms_dispatch 2024-07-26T11:30:44.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 2024-07-26T11:30:44.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:30:44.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7feba563e6f0] 2024-07-26T11:30:44.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 2: /lib64/libc.so.6(+0x8b94c) [0x7feba568b94c] 2024-07-26T11:30:44.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 3: raise() 2024-07-26T11:30:44.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 4: abort() 2024-07-26T11:30:44.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 5: /lib64/libc.so.6(+0x2871b) [0x7feba562871b] 2024-07-26T11:30:44.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 6: /lib64/libc.so.6(+0x37386) [0x7feba5637386] 2024-07-26T11:30:44.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563b399c56b9] 2024-07-26T11:30:44.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563b39b8380c] 2024-07-26T11:30:44.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563b39ac32ca] 2024-07-26T11:30:44.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563b39a93753] 2024-07-26T11:30:44.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563b39a94a1b] 2024-07-26T11:30:44.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563b39a78752] 2024-07-26T11:30:44.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563b399c6a56] 2024-07-26T11:30:44.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563b399ca637] 2024-07-26T11:30:44.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563b399caf5e] 2024-07-26T11:30:44.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563b399ae03a] 2024-07-26T11:30:44.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7feba619b407] 2024-07-26T11:30:44.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 18: (DispatchQueue::entry()+0x60f) [0x7feba619bcf9] 2024-07-26T11:30:44.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7feba62785a1] 2024-07-26T11:30:44.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 20: (Thread::entry_wrapper()+0x43) [0x7feba6025cdd] 2024-07-26T11:30:44.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 21: (Thread::_entry_func(void*)+0xd) [0x7feba6025cf9] 2024-07-26T11:30:44.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 22: /lib64/libc.so.6(+0x89c02) [0x7feba5689c02] 2024-07-26T11:30:44.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 23: /lib64/libc.so.6(+0x10ec40) [0x7feba570ec40] 2024-07-26T11:30:44.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:30:44.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 2024-07-26T11:30:44.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 0> 2024-07-26T11:30:44.136+0000 7feb9f200640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:30:44.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: in thread 7feb9f200640 thread_name:ms_dispatch 2024-07-26T11:30:44.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 2024-07-26T11:30:44.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:30:44.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7feba563e6f0] 2024-07-26T11:30:44.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 2: /lib64/libc.so.6(+0x8b94c) [0x7feba568b94c] 2024-07-26T11:30:44.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 3: raise() 2024-07-26T11:30:44.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 4: abort() 2024-07-26T11:30:44.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 5: /lib64/libc.so.6(+0x2871b) [0x7feba562871b] 2024-07-26T11:30:44.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 6: /lib64/libc.so.6(+0x37386) [0x7feba5637386] 2024-07-26T11:30:44.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563b399c56b9] 2024-07-26T11:30:44.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563b39b8380c] 2024-07-26T11:30:44.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563b39ac32ca] 2024-07-26T11:30:44.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563b39a93753] 2024-07-26T11:30:44.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563b39a94a1b] 2024-07-26T11:30:44.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563b39a78752] 2024-07-26T11:30:44.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563b399c6a56] 2024-07-26T11:30:44.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563b399ca637] 2024-07-26T11:30:44.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563b399caf5e] 2024-07-26T11:30:44.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563b399ae03a] 2024-07-26T11:30:44.431 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7feba619b407] 2024-07-26T11:30:44.431 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 18: (DispatchQueue::entry()+0x60f) [0x7feba619bcf9] 2024-07-26T11:30:44.431 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7feba62785a1] 2024-07-26T11:30:44.431 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 20: (Thread::entry_wrapper()+0x43) [0x7feba6025cdd] 2024-07-26T11:30:44.431 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 21: (Thread::_entry_func(void*)+0xd) [0x7feba6025cf9] 2024-07-26T11:30:44.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 22: /lib64/libc.so.6(+0x89c02) [0x7feba5689c02] 2024-07-26T11:30:44.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 23: /lib64/libc.so.6(+0x10ec40) [0x7feba570ec40] 2024-07-26T11:30:44.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:30:44.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 2024-07-26T11:30:44.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 0> 2024-07-26T11:30:44.136+0000 7feb9f200640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:30:44.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: in thread 7feb9f200640 thread_name:ms_dispatch 2024-07-26T11:30:44.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 2024-07-26T11:30:44.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:30:44.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7feba563e6f0] 2024-07-26T11:30:44.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 2: /lib64/libc.so.6(+0x8b94c) [0x7feba568b94c] 2024-07-26T11:30:44.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 3: raise() 2024-07-26T11:30:44.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 4: abort() 2024-07-26T11:30:44.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 5: /lib64/libc.so.6(+0x2871b) [0x7feba562871b] 2024-07-26T11:30:44.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 6: /lib64/libc.so.6(+0x37386) [0x7feba5637386] 2024-07-26T11:30:44.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x563b399c56b9] 2024-07-26T11:30:44.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x563b39b8380c] 2024-07-26T11:30:44.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x563b39ac32ca] 2024-07-26T11:30:44.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x563b39a93753] 2024-07-26T11:30:44.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x563b39a94a1b] 2024-07-26T11:30:44.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x563b39a78752] 2024-07-26T11:30:44.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x563b399c6a56] 2024-07-26T11:30:44.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x563b399ca637] 2024-07-26T11:30:44.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x563b399caf5e] 2024-07-26T11:30:44.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x563b399ae03a] 2024-07-26T11:30:44.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7feba619b407] 2024-07-26T11:30:44.434 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 18: (DispatchQueue::entry()+0x60f) [0x7feba619bcf9] 2024-07-26T11:30:44.434 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7feba62785a1] 2024-07-26T11:30:44.434 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 20: (Thread::entry_wrapper()+0x43) [0x7feba6025cdd] 2024-07-26T11:30:44.434 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 21: (Thread::_entry_func(void*)+0xd) [0x7feba6025cf9] 2024-07-26T11:30:44.434 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 22: /lib64/libc.so.6(+0x89c02) [0x7feba5689c02] 2024-07-26T11:30:44.434 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 23: /lib64/libc.so.6(+0x10ec40) [0x7feba570ec40] 2024-07-26T11:30:44.434 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:30:44.434 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112371]: 2024-07-26T11:30:44.434 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:44 smithi134 podman[112560]: 2024-07-26 11:30:44.283526352 +0000 UTC m=+0.033098757 container died 380667419a639a6b93060ad9fc1108f6de2cbda129416af6c20e8fec264c669e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724) 2024-07-26T11:30:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:44 smithi189 ceph-mon[36790]: pgmap v949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:44 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/500511838,v1:172.21.15.134:6839/500511838] up:rejoin 2024-07-26T11:30:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:44 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:30:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:44 smithi189 ceph-mon[36790]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:30:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:44 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 185.037694 secs 2024-07-26T11:30:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:44 smithi193 ceph-mon[37473]: pgmap v949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:30:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:44 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/500511838,v1:172.21.15.134:6839/500511838] up:rejoin 2024-07-26T11:30:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:44 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:30:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:44 smithi193 ceph-mon[37473]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:30:44.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:44 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 185.037694 secs 2024-07-26T11:30:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:45 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:30:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:45 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/500511838,v1:172.21.15.134:6839/500511838] up:active 2024-07-26T11:30:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:45 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:30:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:45 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:30:45.423 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:45 smithi134 podman[112560]: 2024-07-26 11:30:45.174315051 +0000 UTC m=+0.923887479 container remove 380667419a639a6b93060ad9fc1108f6de2cbda129416af6c20e8fec264c669e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-07-26T11:30:45.423 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:45 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:30:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:45 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:30:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:45 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/500511838,v1:172.21.15.134:6839/500511838] up:active 2024-07-26T11:30:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:45 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:30:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:45 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:30:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:45 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:30:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:45 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/500511838,v1:172.21.15.134:6839/500511838] up:active 2024-07-26T11:30:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:45 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:30:45.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:45 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:30:46.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:45 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-07-26T11:30:46.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:45 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Consumed 1.776s CPU time. 2024-07-26T11:30:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:46 smithi134 ceph-mon[31966]: pgmap v950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2024-07-26T11:30:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 204.462600 secs 2024-07-26T11:30:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:46 smithi189 ceph-mon[36790]: pgmap v950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2024-07-26T11:30:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 204.462600 secs 2024-07-26T11:30:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:46 smithi193 ceph-mon[37473]: pgmap v950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 102 B/s wr, 9 op/s 2024-07-26T11:30:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 204.462600 secs 2024-07-26T11:30:47.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:30:47.896+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:30:47.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:30:47.896+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:30:47.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:30:47.896+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:48 smithi134 ceph-mon[31966]: pgmap v951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:30:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:48 smithi189 ceph-mon[36790]: pgmap v951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:30:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:48 smithi193 ceph-mon[37473]: pgmap v951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:30:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:49 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 190.037831 secs 2024-07-26T11:30:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:49 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 190.037831 secs 2024-07-26T11:30:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:49 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 190.037831 secs 2024-07-26T11:30:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:50 smithi189 ceph-mon[36790]: pgmap v952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.5 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:30:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:50 smithi193 ceph-mon[37473]: pgmap v952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.5 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:30:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:50 smithi134 ceph-mon[31966]: pgmap v952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.5 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:30:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 209.462665 secs 2024-07-26T11:30:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 209.462665 secs 2024-07-26T11:30:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 209.462665 secs 2024-07-26T11:30:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:52 smithi189 ceph-mon[36790]: pgmap v953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 187 B/s wr, 11 op/s 2024-07-26T11:30:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:52 smithi193 ceph-mon[37473]: pgmap v953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 187 B/s wr, 11 op/s 2024-07-26T11:30:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:52 smithi134 ceph-mon[31966]: pgmap v953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 187 B/s wr, 11 op/s 2024-07-26T11:30:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:54 smithi193 ceph-mon[37473]: pgmap v954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:30:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:54 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 195.037913 secs 2024-07-26T11:30:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:54 smithi134 ceph-mon[31966]: pgmap v954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:30:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:54 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 195.037913 secs 2024-07-26T11:30:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:54 smithi189 ceph-mon[36790]: pgmap v954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:30:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:54 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 195.037913 secs 2024-07-26T11:30:56.200 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:55 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 2. 2024-07-26T11:30:56.200 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:55 smithi134 systemd[1]: Stopped Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:30:56.200 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:55 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Consumed 1.776s CPU time. 2024-07-26T11:30:56.201 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:55 smithi134 systemd[1]: Starting Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:30:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:56 smithi193 ceph-mon[37473]: pgmap v955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:30:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 214.462803 secs 2024-07-26T11:30:56.564 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:56 smithi134 ceph-mon[31966]: pgmap v955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:30:56.564 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 214.462803 secs 2024-07-26T11:30:56.564 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:56 smithi134 podman[112649]: 2024-07-26 11:30:56.226200326 +0000 UTC m=+0.035583319 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:30:56.564 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:56 smithi134 podman[112649]: 2024-07-26 11:30:56.374992425 +0000 UTC m=+0.184375454 container create 6a64c45e2785ce9f2bd1e9befb88e657c02f8a8408d52f7624a05511a332437e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-07-26T11:30:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:56 smithi189 ceph-mon[36790]: pgmap v955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:30:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 214.462803 secs 2024-07-26T11:30:57.173 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:56 smithi134 podman[112649]: 2024-07-26 11:30:56.686937092 +0000 UTC m=+0.496320129 container init 6a64c45e2785ce9f2bd1e9befb88e657c02f8a8408d52f7624a05511a332437e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-07-26T11:30:57.173 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:56 smithi134 podman[112649]: 2024-07-26 11:30:56.697996587 +0000 UTC m=+0.507379600 container start 6a64c45e2785ce9f2bd1e9befb88e657c02f8a8408d52f7624a05511a332437e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0) 2024-07-26T11:30:57.174 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:56 smithi134 bash[112649]: 6a64c45e2785ce9f2bd1e9befb88e657c02f8a8408d52f7624a05511a332437e 2024-07-26T11:30:57.174 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:56 smithi134 systemd[1]: Started Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:30:57.174 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:56 smithi134 ceph-mds[112667]: -- 172.21.15.134:0/1182502002 <== mon.0 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565379c5d860 con 0x56537ddffa80 2024-07-26T11:30:57.174 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:30:56 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: starting mds.g at 2024-07-26T11:30:57.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:57 smithi193 ceph-mon[37473]: daemon mds.g restarted 2024-07-26T11:30:57.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:57 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:30:57.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:57 smithi193 ceph-mon[37473]: osdmap e118: 12 total, 12 up, 12 in 2024-07-26T11:30:57.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:57 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/3528951247,v1:172.21.15.134:6839/3528951247] up:boot 2024-07-26T11:30:57.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:57 smithi193 ceph-mon[37473]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:30:57.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:57 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:30:57.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:30:57.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:57 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:30:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:57 smithi134 ceph-mon[31966]: daemon mds.g restarted 2024-07-26T11:30:57.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:57 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:30:57.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:57 smithi134 ceph-mon[31966]: osdmap e118: 12 total, 12 up, 12 in 2024-07-26T11:30:57.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:57 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/3528951247,v1:172.21.15.134:6839/3528951247] up:boot 2024-07-26T11:30:57.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:57 smithi134 ceph-mon[31966]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:30:57.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:57 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:30:57.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:30:57.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:57 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:30:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:57 smithi189 ceph-mon[36790]: daemon mds.g restarted 2024-07-26T11:30:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:57 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:30:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:57 smithi189 ceph-mon[36790]: osdmap e118: 12 total, 12 up, 12 in 2024-07-26T11:30:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:57 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/3528951247,v1:172.21.15.134:6839/3528951247] up:boot 2024-07-26T11:30:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:57 smithi189 ceph-mon[36790]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:30:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:57 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:30:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:30:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:57 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:30:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:30:58.428+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:30:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:30:58.429+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:30:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:30:58.429+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:30:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:58 smithi134 ceph-mon[31966]: pgmap v957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:30:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:58 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/31742259,v1:172.21.15.134:6837/31742259] up:resolve 2024-07-26T11:30:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:58 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:30:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:30:58 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 200.038035 secs 2024-07-26T11:30:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:58 smithi189 ceph-mon[36790]: pgmap v957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:30:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:58 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/31742259,v1:172.21.15.134:6837/31742259] up:resolve 2024-07-26T11:30:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:58 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:30:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:30:58 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 200.038035 secs 2024-07-26T11:30:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:58 smithi193 ceph-mon[37473]: pgmap v957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:30:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:58 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/31742259,v1:172.21.15.134:6837/31742259] up:resolve 2024-07-26T11:30:59.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:58 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:30:59.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:30:58 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 200.038035 secs 2024-07-26T11:31:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:00 smithi193 ceph-mon[37473]: pgmap v958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 10 op/s 2024-07-26T11:31:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:00 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00399991 2024-07-26T11:31:00.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:00 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/31742259,v1:172.21.15.134:6837/31742259] up:reconnect 2024-07-26T11:31:00.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:00 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:31:00.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:00 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:31:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:00 smithi134 ceph-mon[31966]: pgmap v958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 10 op/s 2024-07-26T11:31:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:00 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00399991 2024-07-26T11:31:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:00 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/31742259,v1:172.21.15.134:6837/31742259] up:reconnect 2024-07-26T11:31:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:00 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:31:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:00 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:31:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:00 smithi189 ceph-mon[36790]: pgmap v958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 10 op/s 2024-07-26T11:31:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:00 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00399991 2024-07-26T11:31:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:00 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/31742259,v1:172.21.15.134:6837/31742259] up:reconnect 2024-07-26T11:31:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:00 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:31:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:00 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:31:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:01 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/31742259,v1:172.21.15.134:6837/31742259] up:rejoin 2024-07-26T11:31:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:01 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:31:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:01 smithi193 ceph-mon[37473]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:31:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 219.462867 secs 2024-07-26T11:31:01.580 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/31742259,v1:172.21.15.134:6837/31742259] up:rejoin 2024-07-26T11:31:01.580 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:31:01.581 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-mon[31966]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:31:01.581 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 219.462867 secs 2024-07-26T11:31:01.582 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:31:01.582 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: *** Caught signal (Aborted) ** 2024-07-26T11:31:01.582 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: in thread 7fb011c00640 thread_name:ms_dispatch 2024-07-26T11:31:01.582 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:31:01.582 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb017e3e6f0] 2024-07-26T11:31:01.583 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb017e8b94c] 2024-07-26T11:31:01.583 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 3: raise() 2024-07-26T11:31:01.583 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 4: abort() 2024-07-26T11:31:01.583 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb017e2871b] 2024-07-26T11:31:01.583 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 6: /lib64/libc.so.6(+0x37386) [0x7fb017e37386] 2024-07-26T11:31:01.583 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560210ec06b9] 2024-07-26T11:31:01.583 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56021107e80c] 2024-07-26T11:31:01.583 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560210fbe2ca] 2024-07-26T11:31:01.583 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560210f8e753] 2024-07-26T11:31:01.583 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560210f8fa1b] 2024-07-26T11:31:01.583 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560210f73752] 2024-07-26T11:31:01.584 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560210ec1a56] 2024-07-26T11:31:01.584 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560210ec5637] 2024-07-26T11:31:01.584 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560210ec5f5e] 2024-07-26T11:31:01.584 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560210ea903a] 2024-07-26T11:31:01.584 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb01899b407] 2024-07-26T11:31:01.584 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 18: (DispatchQueue::entry()+0x60f) [0x7fb01899bcf9] 2024-07-26T11:31:01.584 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb018a785a1] 2024-07-26T11:31:01.584 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 20: (Thread::entry_wrapper()+0x43) [0x7fb018825cdd] 2024-07-26T11:31:01.585 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb018825cf9] 2024-07-26T11:31:01.585 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb017e89c02] 2024-07-26T11:31:01.585 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb017f0ec40] 2024-07-26T11:31:01.585 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 2024-07-26T11:31:01.249+0000 7fb011c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:31:01.585 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: in thread 7fb011c00640 thread_name:ms_dispatch 2024-07-26T11:31:01.585 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 2024-07-26T11:31:01.585 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:31:01.585 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb017e3e6f0] 2024-07-26T11:31:01.585 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb017e8b94c] 2024-07-26T11:31:01.585 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 3: raise() 2024-07-26T11:31:01.585 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 4: abort() 2024-07-26T11:31:01.586 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb017e2871b] 2024-07-26T11:31:01.586 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 6: /lib64/libc.so.6(+0x37386) [0x7fb017e37386] 2024-07-26T11:31:01.586 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560210ec06b9] 2024-07-26T11:31:01.586 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56021107e80c] 2024-07-26T11:31:01.586 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560210fbe2ca] 2024-07-26T11:31:01.586 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560210f8e753] 2024-07-26T11:31:01.586 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560210f8fa1b] 2024-07-26T11:31:01.586 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560210f73752] 2024-07-26T11:31:01.586 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560210ec1a56] 2024-07-26T11:31:01.586 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560210ec5637] 2024-07-26T11:31:01.587 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560210ec5f5e] 2024-07-26T11:31:01.587 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560210ea903a] 2024-07-26T11:31:01.587 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb01899b407] 2024-07-26T11:31:01.587 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 18: (DispatchQueue::entry()+0x60f) [0x7fb01899bcf9] 2024-07-26T11:31:01.587 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb018a785a1] 2024-07-26T11:31:01.588 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 20: (Thread::entry_wrapper()+0x43) [0x7fb018825cdd] 2024-07-26T11:31:01.588 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb018825cf9] 2024-07-26T11:31:01.588 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb017e89c02] 2024-07-26T11:31:01.588 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb017f0ec40] 2024-07-26T11:31:01.588 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:31:01.588 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 2024-07-26T11:31:01.588 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 0> 2024-07-26T11:31:01.249+0000 7fb011c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:31:01.588 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: in thread 7fb011c00640 thread_name:ms_dispatch 2024-07-26T11:31:01.588 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 2024-07-26T11:31:01.589 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:31:01.589 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb017e3e6f0] 2024-07-26T11:31:01.589 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb017e8b94c] 2024-07-26T11:31:01.589 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 3: raise() 2024-07-26T11:31:01.589 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 4: abort() 2024-07-26T11:31:01.589 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb017e2871b] 2024-07-26T11:31:01.589 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 6: /lib64/libc.so.6(+0x37386) [0x7fb017e37386] 2024-07-26T11:31:01.589 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560210ec06b9] 2024-07-26T11:31:01.589 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56021107e80c] 2024-07-26T11:31:01.590 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560210fbe2ca] 2024-07-26T11:31:01.590 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560210f8e753] 2024-07-26T11:31:01.590 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560210f8fa1b] 2024-07-26T11:31:01.590 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560210f73752] 2024-07-26T11:31:01.590 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560210ec1a56] 2024-07-26T11:31:01.590 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560210ec5637] 2024-07-26T11:31:01.590 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560210ec5f5e] 2024-07-26T11:31:01.590 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560210ea903a] 2024-07-26T11:31:01.590 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb01899b407] 2024-07-26T11:31:01.590 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 18: (DispatchQueue::entry()+0x60f) [0x7fb01899bcf9] 2024-07-26T11:31:01.590 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb018a785a1] 2024-07-26T11:31:01.591 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 20: (Thread::entry_wrapper()+0x43) [0x7fb018825cdd] 2024-07-26T11:31:01.591 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb018825cf9] 2024-07-26T11:31:01.591 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb017e89c02] 2024-07-26T11:31:01.591 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb017f0ec40] 2024-07-26T11:31:01.591 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:31:01.591 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 2024-07-26T11:31:01.591 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 0> 2024-07-26T11:31:01.249+0000 7fb011c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:31:01.591 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: in thread 7fb011c00640 thread_name:ms_dispatch 2024-07-26T11:31:01.591 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 2024-07-26T11:31:01.591 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:31:01.591 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fb017e3e6f0] 2024-07-26T11:31:01.592 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fb017e8b94c] 2024-07-26T11:31:01.592 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 3: raise() 2024-07-26T11:31:01.592 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 4: abort() 2024-07-26T11:31:01.592 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 5: /lib64/libc.so.6(+0x2871b) [0x7fb017e2871b] 2024-07-26T11:31:01.592 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 6: /lib64/libc.so.6(+0x37386) [0x7fb017e37386] 2024-07-26T11:31:01.592 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x560210ec06b9] 2024-07-26T11:31:01.592 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x56021107e80c] 2024-07-26T11:31:01.592 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x560210fbe2ca] 2024-07-26T11:31:01.592 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x560210f8e753] 2024-07-26T11:31:01.592 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x560210f8fa1b] 2024-07-26T11:31:01.592 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x560210f73752] 2024-07-26T11:31:01.592 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x560210ec1a56] 2024-07-26T11:31:01.593 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x560210ec5637] 2024-07-26T11:31:01.593 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x560210ec5f5e] 2024-07-26T11:31:01.593 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x560210ea903a] 2024-07-26T11:31:01.593 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fb01899b407] 2024-07-26T11:31:01.593 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 18: (DispatchQueue::entry()+0x60f) [0x7fb01899bcf9] 2024-07-26T11:31:01.593 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fb018a785a1] 2024-07-26T11:31:01.593 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 20: (Thread::entry_wrapper()+0x43) [0x7fb018825cdd] 2024-07-26T11:31:01.593 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 21: (Thread::_entry_func(void*)+0xd) [0x7fb018825cf9] 2024-07-26T11:31:01.593 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 22: /lib64/libc.so.6(+0x89c02) [0x7fb017e89c02] 2024-07-26T11:31:01.593 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fb017f0ec40] 2024-07-26T11:31:01.593 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:31:01.593 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[77462]: 2024-07-26T11:31:01.594 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:01 smithi134 podman[112701]: 2024-07-26 11:31:01.408771422 +0000 UTC m=+0.033102164 container died c8e744686ff3fd4e76f5e61c2b46eaaeff3dbecf1ce7e0840f7362504a75ce5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-07-26T11:31:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:01 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/31742259,v1:172.21.15.134:6837/31742259] up:rejoin 2024-07-26T11:31:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:01 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:31:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:01 smithi189 ceph-mon[36790]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:31:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 219.462867 secs 2024-07-26T11:31:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:02 smithi193 ceph-mon[37473]: pgmap v959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:31:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:02 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:31:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:02 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/31742259,v1:172.21.15.134:6837/31742259] up:active 2024-07-26T11:31:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:02 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:31:02.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:02 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:31:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:02 smithi134 ceph-mon[31966]: pgmap v959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:31:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:02 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:31:02.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:02 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/31742259,v1:172.21.15.134:6837/31742259] up:active 2024-07-26T11:31:02.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:02 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:31:02.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:02 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:31:02.673 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:02 smithi134 podman[112701]: 2024-07-26 11:31:02.3996064 +0000 UTC m=+1.023937159 container remove c8e744686ff3fd4e76f5e61c2b46eaaeff3dbecf1ce7e0840f7362504a75ce5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_CLEAN=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:31:02.673 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:02 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:31:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:02 smithi189 ceph-mon[36790]: pgmap v959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:31:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:02 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:31:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:02 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/31742259,v1:172.21.15.134:6837/31742259] up:active 2024-07-26T11:31:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:02 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:31:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:02 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:31:02.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:31:02.911+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:31:02.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:31:02.911+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:02.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:31:02.911+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:31:03.172 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:02 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T11:31:03.172 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:02 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Consumed 2.297s CPU time. 2024-07-26T11:31:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:04 smithi193 ceph-mon[37473]: pgmap v960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:31:04.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:04 smithi193 ceph-mon[37473]: 2 slow requests, 1 included below; oldest blocked for > 205.038105 secs 2024-07-26T11:31:04.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:04 smithi193 ceph-mon[37473]: slow request 182.292822 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:31:04.525 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:31:04 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:31:04.061+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:31:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:04 smithi134 ceph-mon[31966]: pgmap v960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:31:04.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:04 smithi134 ceph-mon[31966]: 2 slow requests, 1 included below; oldest blocked for > 205.038105 secs 2024-07-26T11:31:04.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:04 smithi134 ceph-mon[31966]: slow request 182.292822 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:31:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:04 smithi189 ceph-mon[36790]: pgmap v960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:31:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:04 smithi189 ceph-mon[36790]: 2 slow requests, 1 included below; oldest blocked for > 205.038105 secs 2024-07-26T11:31:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:04 smithi189 ceph-mon[36790]: slow request 182.292822 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:31:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:05 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:active 2024-07-26T11:31:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:05 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:31:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:05 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:active 2024-07-26T11:31:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:05 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:31:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:05 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.193:6834/97099551,v1:172.21.15.193:6835/97099551] up:active 2024-07-26T11:31:05.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:05 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:31:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:06 smithi134 ceph-mon[31966]: pgmap v961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:31:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 224.463020 secs 2024-07-26T11:31:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:06 smithi189 ceph-mon[36790]: pgmap v961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:31:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 224.463020 secs 2024-07-26T11:31:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:06 smithi193 ceph-mon[37473]: pgmap v961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:31:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 224.463020 secs 2024-07-26T11:31:08.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:08 smithi134 ceph-mon[31966]: pgmap v962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 201 B/s wr, 11 op/s 2024-07-26T11:31:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:08 smithi189 ceph-mon[36790]: pgmap v962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 201 B/s wr, 11 op/s 2024-07-26T11:31:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:08 smithi193 ceph-mon[37473]: pgmap v962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 201 B/s wr, 11 op/s 2024-07-26T11:31:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 210.038187 secs 2024-07-26T11:31:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 210.038187 secs 2024-07-26T11:31:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 210.038187 secs 2024-07-26T11:31:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:10 smithi134 ceph-mon[31966]: pgmap v963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:31:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:10 smithi189 ceph-mon[36790]: pgmap v963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:31:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:10 smithi193 ceph-mon[37473]: pgmap v963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:31:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 229.463146 secs 2024-07-26T11:31:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 229.463146 secs 2024-07-26T11:31:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 229.463146 secs 2024-07-26T11:31:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:12 smithi134 ceph-mon[31966]: pgmap v964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:31:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:12 smithi189 ceph-mon[36790]: pgmap v964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:31:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:12 smithi193 ceph-mon[37473]: pgmap v964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:31:13.446 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:31:13.444+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:31:13.446 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:31:13.444+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:31:13.446 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:31:13.444+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:13.574 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:13 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-07-26T11:31:13.574 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:13 smithi134 systemd[1]: Stopped Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:31:13.575 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:13 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Consumed 2.297s CPU time. 2024-07-26T11:31:13.575 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:13 smithi134 systemd[1]: Starting Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:31:13.914 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:13 smithi134 podman[112794]: 2024-07-26 11:31:13.476915324 +0000 UTC m=+0.033074051 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:31:13.914 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:13 smithi134 podman[112794]: 2024-07-26 11:31:13.592124158 +0000 UTC m=+0.148282920 container create 05e7b491d929b8305233938257a825b62eaeae8fcdb4ba7c4122089d2e11e089 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0) 2024-07-26T11:31:14.173 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:13 smithi134 podman[112794]: 2024-07-26 11:31:13.912165528 +0000 UTC m=+0.468324306 container init 05e7b491d929b8305233938257a825b62eaeae8fcdb4ba7c4122089d2e11e089 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-07-26T11:31:14.173 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:13 smithi134 podman[112794]: 2024-07-26 11:31:13.922946275 +0000 UTC m=+0.479105021 container start 05e7b491d929b8305233938257a825b62eaeae8fcdb4ba7c4122089d2e11e089 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0) 2024-07-26T11:31:14.173 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:14 smithi134 bash[112794]: 05e7b491d929b8305233938257a825b62eaeae8fcdb4ba7c4122089d2e11e089 2024-07-26T11:31:14.173 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:14 smithi134 systemd[1]: Started Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:31:14.173 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:14 smithi134 ceph-mds[112813]: -- 172.21.15.134:0/165280895 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f91e4d3c20 con 0x55f922675600 2024-07-26T11:31:14.173 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:14 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: starting mds.d at 2024-07-26T11:31:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:14 smithi134 ceph-mon[31966]: pgmap v965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:31:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 215.038256 secs 2024-07-26T11:31:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:14 smithi134 ceph-mon[31966]: daemon mds.d restarted 2024-07-26T11:31:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:14 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:31:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:14 smithi134 ceph-mon[31966]: osdmap e119: 12 total, 12 up, 12 in 2024-07-26T11:31:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:14 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/351227695,v1:172.21.15.134:6837/351227695] up:boot 2024-07-26T11:31:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:14 smithi134 ceph-mon[31966]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-07-26T11:31:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:14 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:31:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:31:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:14 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:31:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:14 smithi189 ceph-mon[36790]: pgmap v965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:31:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 215.038256 secs 2024-07-26T11:31:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:14 smithi189 ceph-mon[36790]: daemon mds.d restarted 2024-07-26T11:31:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:14 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:31:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:14 smithi189 ceph-mon[36790]: osdmap e119: 12 total, 12 up, 12 in 2024-07-26T11:31:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:14 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/351227695,v1:172.21.15.134:6837/351227695] up:boot 2024-07-26T11:31:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:14 smithi189 ceph-mon[36790]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-07-26T11:31:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:14 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:31:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:31:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:14 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:31:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:14 smithi193 ceph-mon[37473]: pgmap v965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:31:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 215.038256 secs 2024-07-26T11:31:14.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:14 smithi193 ceph-mon[37473]: daemon mds.d restarted 2024-07-26T11:31:14.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:14 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:31:14.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:14 smithi193 ceph-mon[37473]: osdmap e119: 12 total, 12 up, 12 in 2024-07-26T11:31:14.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:14 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/351227695,v1:172.21.15.134:6837/351227695] up:boot 2024-07-26T11:31:14.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:14 smithi193 ceph-mon[37473]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-07-26T11:31:14.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:14 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:31:14.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:31:14.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:14 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:31:16.593 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:16 smithi134 ceph-mon[31966]: pgmap v967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-07-26T11:31:16.593 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:16 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/3528951247,v1:172.21.15.134:6839/3528951247] up:resolve 2024-07-26T11:31:16.593 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:16 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:31:16.593 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 234.463262 secs 2024-07-26T11:31:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:16 smithi189 ceph-mon[36790]: pgmap v967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-07-26T11:31:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:16 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/3528951247,v1:172.21.15.134:6839/3528951247] up:resolve 2024-07-26T11:31:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:16 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:31:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 234.463262 secs 2024-07-26T11:31:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:16 smithi193 ceph-mon[37473]: pgmap v967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-07-26T11:31:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:16 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/3528951247,v1:172.21.15.134:6839/3528951247] up:resolve 2024-07-26T11:31:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:16 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:31:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 234.463262 secs 2024-07-26T11:31:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:17 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00199996 2024-07-26T11:31:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:17 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/3528951247,v1:172.21.15.134:6839/3528951247] up:reconnect 2024-07-26T11:31:17.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:17 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:31:17.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:17 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999978 2024-07-26T11:31:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:17 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00199996 2024-07-26T11:31:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:17 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/3528951247,v1:172.21.15.134:6839/3528951247] up:reconnect 2024-07-26T11:31:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:17 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:31:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:17 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999978 2024-07-26T11:31:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:17 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00199996 2024-07-26T11:31:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:17 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/3528951247,v1:172.21.15.134:6839/3528951247] up:reconnect 2024-07-26T11:31:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:17 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:31:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:17 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999978 2024-07-26T11:31:17.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:31:17.916+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:31:17.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:31:17.916+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:17.918 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:31:17.916+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:31:18.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-mon[31966]: pgmap v968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:31:18.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/3528951247,v1:172.21.15.134:6839/3528951247] up:rejoin 2024-07-26T11:31:18.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:31:18.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-mon[31966]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:31:18.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:31:18.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: *** Caught signal (Aborted) ** 2024-07-26T11:31:18.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: in thread 7f59ab600640 thread_name:ms_dispatch 2024-07-26T11:31:18.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:31:18.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f59b183e6f0] 2024-07-26T11:31:18.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f59b188b94c] 2024-07-26T11:31:18.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 3: raise() 2024-07-26T11:31:18.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 4: abort() 2024-07-26T11:31:18.676 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 5: /lib64/libc.so.6(+0x2871b) [0x7f59b182871b] 2024-07-26T11:31:18.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 6: /lib64/libc.so.6(+0x37386) [0x7f59b1837386] 2024-07-26T11:31:18.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56535972c6b9] 2024-07-26T11:31:18.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5653598ea80c] 2024-07-26T11:31:18.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56535982a2ca] 2024-07-26T11:31:18.677 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5653597fa753] 2024-07-26T11:31:18.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5653597fba1b] 2024-07-26T11:31:18.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5653597df752] 2024-07-26T11:31:18.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56535972da56] 2024-07-26T11:31:18.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x565359731637] 2024-07-26T11:31:18.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x565359731f5e] 2024-07-26T11:31:18.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56535971503a] 2024-07-26T11:31:18.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f59b239b407] 2024-07-26T11:31:18.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 18: (DispatchQueue::entry()+0x60f) [0x7f59b239bcf9] 2024-07-26T11:31:18.678 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f59b24785a1] 2024-07-26T11:31:18.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 20: (Thread::entry_wrapper()+0x43) [0x7f59b2225cdd] 2024-07-26T11:31:18.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 21: (Thread::_entry_func(void*)+0xd) [0x7f59b2225cf9] 2024-07-26T11:31:18.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 22: /lib64/libc.so.6(+0x89c02) [0x7f59b1889c02] 2024-07-26T11:31:18.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f59b190ec40] 2024-07-26T11:31:18.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 2024-07-26T11:31:18.364+0000 7f59ab600640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:31:18.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: in thread 7f59ab600640 thread_name:ms_dispatch 2024-07-26T11:31:18.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 2024-07-26T11:31:18.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:31:18.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f59b183e6f0] 2024-07-26T11:31:18.679 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f59b188b94c] 2024-07-26T11:31:18.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 3: raise() 2024-07-26T11:31:18.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 4: abort() 2024-07-26T11:31:18.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 5: /lib64/libc.so.6(+0x2871b) [0x7f59b182871b] 2024-07-26T11:31:18.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 6: /lib64/libc.so.6(+0x37386) [0x7f59b1837386] 2024-07-26T11:31:18.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56535972c6b9] 2024-07-26T11:31:18.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5653598ea80c] 2024-07-26T11:31:18.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56535982a2ca] 2024-07-26T11:31:18.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5653597fa753] 2024-07-26T11:31:18.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5653597fba1b] 2024-07-26T11:31:18.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5653597df752] 2024-07-26T11:31:18.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56535972da56] 2024-07-26T11:31:18.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x565359731637] 2024-07-26T11:31:18.680 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x565359731f5e] 2024-07-26T11:31:18.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56535971503a] 2024-07-26T11:31:18.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f59b239b407] 2024-07-26T11:31:18.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 18: (DispatchQueue::entry()+0x60f) [0x7f59b239bcf9] 2024-07-26T11:31:18.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f59b24785a1] 2024-07-26T11:31:18.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 20: (Thread::entry_wrapper()+0x43) [0x7f59b2225cdd] 2024-07-26T11:31:18.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 21: (Thread::_entry_func(void*)+0xd) [0x7f59b2225cf9] 2024-07-26T11:31:18.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 22: /lib64/libc.so.6(+0x89c02) [0x7f59b1889c02] 2024-07-26T11:31:18.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f59b190ec40] 2024-07-26T11:31:18.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:31:18.681 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 2024-07-26T11:31:18.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 0> 2024-07-26T11:31:18.364+0000 7f59ab600640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:31:18.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: in thread 7f59ab600640 thread_name:ms_dispatch 2024-07-26T11:31:18.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 2024-07-26T11:31:18.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:31:18.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f59b183e6f0] 2024-07-26T11:31:18.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f59b188b94c] 2024-07-26T11:31:18.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 3: raise() 2024-07-26T11:31:18.682 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 4: abort() 2024-07-26T11:31:18.687 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 5: /lib64/libc.so.6(+0x2871b) [0x7f59b182871b] 2024-07-26T11:31:18.687 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 6: /lib64/libc.so.6(+0x37386) [0x7f59b1837386] 2024-07-26T11:31:18.687 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56535972c6b9] 2024-07-26T11:31:18.687 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5653598ea80c] 2024-07-26T11:31:18.687 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56535982a2ca] 2024-07-26T11:31:18.687 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5653597fa753] 2024-07-26T11:31:18.687 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5653597fba1b] 2024-07-26T11:31:18.688 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5653597df752] 2024-07-26T11:31:18.688 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56535972da56] 2024-07-26T11:31:18.688 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x565359731637] 2024-07-26T11:31:18.688 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x565359731f5e] 2024-07-26T11:31:18.688 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56535971503a] 2024-07-26T11:31:18.689 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f59b239b407] 2024-07-26T11:31:18.689 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 18: (DispatchQueue::entry()+0x60f) [0x7f59b239bcf9] 2024-07-26T11:31:18.689 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f59b24785a1] 2024-07-26T11:31:18.689 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 20: (Thread::entry_wrapper()+0x43) [0x7f59b2225cdd] 2024-07-26T11:31:18.689 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 21: (Thread::_entry_func(void*)+0xd) [0x7f59b2225cf9] 2024-07-26T11:31:18.689 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 22: /lib64/libc.so.6(+0x89c02) [0x7f59b1889c02] 2024-07-26T11:31:18.689 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f59b190ec40] 2024-07-26T11:31:18.689 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:31:18.689 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 2024-07-26T11:31:18.689 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 0> 2024-07-26T11:31:18.364+0000 7f59ab600640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:31:18.689 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: in thread 7f59ab600640 thread_name:ms_dispatch 2024-07-26T11:31:18.690 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 2024-07-26T11:31:18.690 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:31:18.690 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f59b183e6f0] 2024-07-26T11:31:18.690 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f59b188b94c] 2024-07-26T11:31:18.690 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 3: raise() 2024-07-26T11:31:18.690 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 4: abort() 2024-07-26T11:31:18.690 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 5: /lib64/libc.so.6(+0x2871b) [0x7f59b182871b] 2024-07-26T11:31:18.690 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 6: /lib64/libc.so.6(+0x37386) [0x7f59b1837386] 2024-07-26T11:31:18.690 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x56535972c6b9] 2024-07-26T11:31:18.690 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5653598ea80c] 2024-07-26T11:31:18.690 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x56535982a2ca] 2024-07-26T11:31:18.690 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5653597fa753] 2024-07-26T11:31:18.690 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5653597fba1b] 2024-07-26T11:31:18.691 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5653597df752] 2024-07-26T11:31:18.691 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x56535972da56] 2024-07-26T11:31:18.691 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x565359731637] 2024-07-26T11:31:18.691 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x565359731f5e] 2024-07-26T11:31:18.691 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x56535971503a] 2024-07-26T11:31:18.691 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f59b239b407] 2024-07-26T11:31:18.691 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 18: (DispatchQueue::entry()+0x60f) [0x7f59b239bcf9] 2024-07-26T11:31:18.691 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f59b24785a1] 2024-07-26T11:31:18.691 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 20: (Thread::entry_wrapper()+0x43) [0x7f59b2225cdd] 2024-07-26T11:31:18.691 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 21: (Thread::_entry_func(void*)+0xd) [0x7f59b2225cf9] 2024-07-26T11:31:18.691 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 22: /lib64/libc.so.6(+0x89c02) [0x7f59b1889c02] 2024-07-26T11:31:18.691 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f59b190ec40] 2024-07-26T11:31:18.691 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:31:18.692 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[112663]: 2024-07-26T11:31:18.692 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:18 smithi134 podman[112847]: 2024-07-26 11:31:18.474000915 +0000 UTC m=+0.031134481 container died 6a64c45e2785ce9f2bd1e9befb88e657c02f8a8408d52f7624a05511a332437e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-07-26T11:31:18.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:18 smithi189 ceph-mon[36790]: pgmap v968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:31:18.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:18 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/3528951247,v1:172.21.15.134:6839/3528951247] up:rejoin 2024-07-26T11:31:18.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:18 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:31:18.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:18 smithi189 ceph-mon[36790]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:31:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:18 smithi193 ceph-mon[37473]: pgmap v968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:31:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:18 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/3528951247,v1:172.21.15.134:6839/3528951247] up:rejoin 2024-07-26T11:31:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:18 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:31:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:18 smithi193 ceph-mon[37473]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:31:19.386 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:31:19 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:31:19.063+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:31:19.672 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:19 smithi134 podman[112847]: 2024-07-26 11:31:19.405482488 +0000 UTC m=+0.962616094 container remove 6a64c45e2785ce9f2bd1e9befb88e657c02f8a8408d52f7624a05511a332437e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0) 2024-07-26T11:31:19.672 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:19 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:31:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 220.038326 secs 2024-07-26T11:31:19.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:19 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:31:19.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:19 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/3528951247,v1:172.21.15.134:6839/3528951247] up:active 2024-07-26T11:31:19.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:19 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:31:19.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:19 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:31:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 220.038326 secs 2024-07-26T11:31:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:19 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:31:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:19 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/3528951247,v1:172.21.15.134:6839/3528951247] up:active 2024-07-26T11:31:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:19 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:31:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:19 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:31:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 220.038326 secs 2024-07-26T11:31:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:19 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:31:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:19 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/3528951247,v1:172.21.15.134:6839/3528951247] up:active 2024-07-26T11:31:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:19 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:31:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:19 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:31:20.172 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:19 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-07-26T11:31:20.172 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:19 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Consumed 1.560s CPU time. 2024-07-26T11:31:20.593 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:20 smithi134 ceph-mon[31966]: pgmap v969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:31:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:20 smithi189 ceph-mon[36790]: pgmap v969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:31:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:20 smithi193 ceph-mon[37473]: pgmap v969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:31:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 239.463392 secs 2024-07-26T11:31:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 239.463392 secs 2024-07-26T11:31:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 239.463392 secs 2024-07-26T11:31:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:22 smithi134 ceph-mon[31966]: pgmap v970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:31:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:22 smithi189 ceph-mon[36790]: pgmap v970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:31:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:22 smithi193 ceph-mon[37473]: pgmap v970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:31:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 225.038386 secs 2024-07-26T11:31:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 225.038386 secs 2024-07-26T11:31:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 225.038386 secs 2024-07-26T11:31:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:24 smithi134 ceph-mon[31966]: pgmap v971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:31:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:24 smithi189 ceph-mon[36790]: pgmap v971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:31:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:24 smithi193 ceph-mon[37473]: pgmap v971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:31:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:26 smithi189 ceph-mon[36790]: pgmap v972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 187 B/s wr, 10 op/s 2024-07-26T11:31:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 244.463487 secs 2024-07-26T11:31:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:26 smithi193 ceph-mon[37473]: pgmap v972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 187 B/s wr, 10 op/s 2024-07-26T11:31:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 244.463487 secs 2024-07-26T11:31:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:26 smithi134 ceph-mon[31966]: pgmap v972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 187 B/s wr, 10 op/s 2024-07-26T11:31:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 244.463487 secs 2024-07-26T11:31:28.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:31:28.459+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:31:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:31:28.459+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:31:28.461 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:31:28.459+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:28 smithi189 ceph-mon[36790]: pgmap v973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:31:28.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:31:28.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 230.038457 secs 2024-07-26T11:31:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:28 smithi193 ceph-mon[37473]: pgmap v973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:31:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:31:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 230.038457 secs 2024-07-26T11:31:28.908 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:28 smithi134 ceph-mon[31966]: pgmap v973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:31:28.908 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:31:28.908 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 230.038457 secs 2024-07-26T11:31:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:31:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:31:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:31:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:31:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:31:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:31:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:31:29.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:31:29.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:31:30.322 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:29 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 3. 2024-07-26T11:31:30.322 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:29 smithi134 systemd[1]: Stopped Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:31:30.322 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:29 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Consumed 1.560s CPU time. 2024-07-26T11:31:30.322 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:29 smithi134 systemd[1]: Starting Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:31:30.648 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:30 smithi134 ceph-mon[31966]: pgmap v974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:31:30.649 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:30 smithi134 podman[113067]: 2024-07-26 11:31:30.22484776 +0000 UTC m=+0.032086216 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:31:30.649 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:30 smithi134 podman[113067]: 2024-07-26 11:31:30.383506286 +0000 UTC m=+0.190744777 container create 710b299842944119ae6aa869ef8a65302662d412e30d195875a73a70ab1c4275 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True) 2024-07-26T11:31:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:30 smithi189 ceph-mon[36790]: pgmap v974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:31:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:30 smithi193 ceph-mon[37473]: pgmap v974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:31:30.924 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:30 smithi134 podman[113067]: 2024-07-26 11:31:30.662639627 +0000 UTC m=+0.469878116 container init 710b299842944119ae6aa869ef8a65302662d412e30d195875a73a70ab1c4275 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:31:30.924 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:30 smithi134 podman[113067]: 2024-07-26 11:31:30.671652064 +0000 UTC m=+0.478890520 container start 710b299842944119ae6aa869ef8a65302662d412e30d195875a73a70ab1c4275 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724) 2024-07-26T11:31:30.924 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:30 smithi134 bash[113067]: 710b299842944119ae6aa869ef8a65302662d412e30d195875a73a70ab1c4275 2024-07-26T11:31:30.924 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:30 smithi134 systemd[1]: Started Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:31:30.924 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:30 smithi134 ceph-mds[113094]: -- 172.21.15.134:0/413906238 <== mon.0 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55beb9a83860 con 0x55bebdc25180 2024-07-26T11:31:30.924 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:30 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: starting mds.g at 2024-07-26T11:31:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:31 smithi189 ceph-mon[36790]: daemon mds.g restarted 2024-07-26T11:31:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:31 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:31:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:31 smithi189 ceph-mon[36790]: osdmap e120: 12 total, 12 up, 12 in 2024-07-26T11:31:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:31 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/679901285,v1:172.21.15.134:6839/679901285] up:boot 2024-07-26T11:31:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:31 smithi189 ceph-mon[36790]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:31:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:31 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:31:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:31:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 249.463560 secs 2024-07-26T11:31:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:31 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:31:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:31 smithi193 ceph-mon[37473]: daemon mds.g restarted 2024-07-26T11:31:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:31 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:31:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:31 smithi193 ceph-mon[37473]: osdmap e120: 12 total, 12 up, 12 in 2024-07-26T11:31:31.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:31 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/679901285,v1:172.21.15.134:6839/679901285] up:boot 2024-07-26T11:31:31.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:31 smithi193 ceph-mon[37473]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:31:31.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:31 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:31:31.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:31:31.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 249.463560 secs 2024-07-26T11:31:31.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:31 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:31:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:31 smithi134 ceph-mon[31966]: daemon mds.g restarted 2024-07-26T11:31:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:31 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:31:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:31 smithi134 ceph-mon[31966]: osdmap e120: 12 total, 12 up, 12 in 2024-07-26T11:31:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:31 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/679901285,v1:172.21.15.134:6839/679901285] up:boot 2024-07-26T11:31:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:31 smithi134 ceph-mon[31966]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:31:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:31 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:31:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:31:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 249.463560 secs 2024-07-26T11:31:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:31 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:31:32.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:31:32.932+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:31:32.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:31:32.932+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:32.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:31:32.932+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:31:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:32 smithi134 ceph-mon[31966]: pgmap v976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:31:33.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:32 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/351227695,v1:172.21.15.134:6837/351227695] up:resolve 2024-07-26T11:31:33.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:32 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:31:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:32 smithi189 ceph-mon[36790]: pgmap v976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:31:33.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:32 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/351227695,v1:172.21.15.134:6837/351227695] up:resolve 2024-07-26T11:31:33.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:32 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:31:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:32 smithi193 ceph-mon[37473]: pgmap v976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-07-26T11:31:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:32 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/351227695,v1:172.21.15.134:6837/351227695] up:resolve 2024-07-26T11:31:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:32 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:31:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 235.038568 secs 2024-07-26T11:31:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:33 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/351227695,v1:172.21.15.134:6837/351227695] up:reconnect 2024-07-26T11:31:34.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:33 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:31:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 235.038568 secs 2024-07-26T11:31:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:33 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/351227695,v1:172.21.15.134:6837/351227695] up:reconnect 2024-07-26T11:31:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:33 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:31:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 235.038568 secs 2024-07-26T11:31:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:33 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/351227695,v1:172.21.15.134:6837/351227695] up:reconnect 2024-07-26T11:31:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:33 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:31:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:34 smithi134 ceph-mon[31966]: pgmap v977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:35.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:34 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:31:35.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:34 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:31:35.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:34 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/351227695,v1:172.21.15.134:6837/351227695] up:rejoin 2024-07-26T11:31:35.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:34 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:31:35.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:34 smithi134 ceph-mon[31966]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:31:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:34 smithi189 ceph-mon[36790]: pgmap v977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:34 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:31:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:34 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:31:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:34 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/351227695,v1:172.21.15.134:6837/351227695] up:rejoin 2024-07-26T11:31:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:34 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:31:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:34 smithi189 ceph-mon[36790]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:31:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:34 smithi193 ceph-mon[37473]: pgmap v977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:34 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299994 2024-07-26T11:31:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:34 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:31:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:34 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/351227695,v1:172.21.15.134:6837/351227695] up:rejoin 2024-07-26T11:31:35.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:34 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:31:35.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:34 smithi193 ceph-mon[37473]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:31:35.835 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:31:35.835 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: *** Caught signal (Aborted) ** 2024-07-26T11:31:35.835 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: in thread 7f76cba00640 thread_name:ms_dispatch 2024-07-26T11:31:35.835 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:31:35.835 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f76d1c3e6f0] 2024-07-26T11:31:35.835 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f76d1c8b94c] 2024-07-26T11:31:35.835 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 3: raise() 2024-07-26T11:31:35.835 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 4: abort() 2024-07-26T11:31:35.835 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 5: /lib64/libc.so.6(+0x2871b) [0x7f76d1c2871b] 2024-07-26T11:31:35.836 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 6: /lib64/libc.so.6(+0x37386) [0x7f76d1c37386] 2024-07-26T11:31:35.836 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f9111da6b9] 2024-07-26T11:31:35.836 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f91139880c] 2024-07-26T11:31:35.836 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55f9112d82ca] 2024-07-26T11:31:35.836 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55f9112a8753] 2024-07-26T11:31:35.836 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f9112a9a1b] 2024-07-26T11:31:35.836 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f91128d752] 2024-07-26T11:31:35.836 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f9111dba56] 2024-07-26T11:31:35.836 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f9111df637] 2024-07-26T11:31:35.836 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f9111dff5e] 2024-07-26T11:31:35.836 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f9111c303a] 2024-07-26T11:31:35.836 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f76d279b407] 2024-07-26T11:31:35.836 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 18: (DispatchQueue::entry()+0x60f) [0x7f76d279bcf9] 2024-07-26T11:31:35.837 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f76d28785a1] 2024-07-26T11:31:35.837 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 20: (Thread::entry_wrapper()+0x43) [0x7f76d2625cdd] 2024-07-26T11:31:35.837 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 21: (Thread::_entry_func(void*)+0xd) [0x7f76d2625cf9] 2024-07-26T11:31:35.837 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 22: /lib64/libc.so.6(+0x89c02) [0x7f76d1c89c02] 2024-07-26T11:31:35.837 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f76d1d0ec40] 2024-07-26T11:31:35.838 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 2024-07-26T11:31:35.458+0000 7f76cba00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:31:35.838 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: in thread 7f76cba00640 thread_name:ms_dispatch 2024-07-26T11:31:35.838 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 2024-07-26T11:31:35.838 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:31:35.838 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f76d1c3e6f0] 2024-07-26T11:31:35.838 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f76d1c8b94c] 2024-07-26T11:31:35.838 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 3: raise() 2024-07-26T11:31:35.838 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 4: abort() 2024-07-26T11:31:35.838 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 5: /lib64/libc.so.6(+0x2871b) [0x7f76d1c2871b] 2024-07-26T11:31:35.838 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 6: /lib64/libc.so.6(+0x37386) [0x7f76d1c37386] 2024-07-26T11:31:35.838 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f9111da6b9] 2024-07-26T11:31:35.839 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f91139880c] 2024-07-26T11:31:35.839 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55f9112d82ca] 2024-07-26T11:31:35.839 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55f9112a8753] 2024-07-26T11:31:35.839 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f9112a9a1b] 2024-07-26T11:31:35.839 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f91128d752] 2024-07-26T11:31:35.839 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f9111dba56] 2024-07-26T11:31:35.839 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f9111df637] 2024-07-26T11:31:35.839 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f9111dff5e] 2024-07-26T11:31:35.839 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f9111c303a] 2024-07-26T11:31:35.839 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f76d279b407] 2024-07-26T11:31:35.839 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 18: (DispatchQueue::entry()+0x60f) [0x7f76d279bcf9] 2024-07-26T11:31:35.840 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f76d28785a1] 2024-07-26T11:31:35.840 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 20: (Thread::entry_wrapper()+0x43) [0x7f76d2625cdd] 2024-07-26T11:31:35.840 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 21: (Thread::_entry_func(void*)+0xd) [0x7f76d2625cf9] 2024-07-26T11:31:35.840 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 22: /lib64/libc.so.6(+0x89c02) [0x7f76d1c89c02] 2024-07-26T11:31:35.840 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f76d1d0ec40] 2024-07-26T11:31:35.840 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:31:35.840 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 2024-07-26T11:31:35.840 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 0> 2024-07-26T11:31:35.458+0000 7f76cba00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:31:35.840 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: in thread 7f76cba00640 thread_name:ms_dispatch 2024-07-26T11:31:35.840 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 2024-07-26T11:31:35.840 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:31:35.840 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f76d1c3e6f0] 2024-07-26T11:31:35.840 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f76d1c8b94c] 2024-07-26T11:31:35.840 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 3: raise() 2024-07-26T11:31:35.841 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 4: abort() 2024-07-26T11:31:35.841 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 5: /lib64/libc.so.6(+0x2871b) [0x7f76d1c2871b] 2024-07-26T11:31:35.841 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 6: /lib64/libc.so.6(+0x37386) [0x7f76d1c37386] 2024-07-26T11:31:35.841 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f9111da6b9] 2024-07-26T11:31:35.841 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f91139880c] 2024-07-26T11:31:35.841 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55f9112d82ca] 2024-07-26T11:31:35.841 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55f9112a8753] 2024-07-26T11:31:35.841 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f9112a9a1b] 2024-07-26T11:31:35.841 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f91128d752] 2024-07-26T11:31:35.841 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f9111dba56] 2024-07-26T11:31:35.841 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f9111df637] 2024-07-26T11:31:35.841 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f9111dff5e] 2024-07-26T11:31:35.842 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f9111c303a] 2024-07-26T11:31:35.842 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f76d279b407] 2024-07-26T11:31:35.842 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 18: (DispatchQueue::entry()+0x60f) [0x7f76d279bcf9] 2024-07-26T11:31:35.842 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f76d28785a1] 2024-07-26T11:31:35.842 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 20: (Thread::entry_wrapper()+0x43) [0x7f76d2625cdd] 2024-07-26T11:31:35.842 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 21: (Thread::_entry_func(void*)+0xd) [0x7f76d2625cf9] 2024-07-26T11:31:35.842 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 22: /lib64/libc.so.6(+0x89c02) [0x7f76d1c89c02] 2024-07-26T11:31:35.842 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f76d1d0ec40] 2024-07-26T11:31:35.842 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:31:35.842 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 2024-07-26T11:31:35.842 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: -2> 2024-07-26T11:31:35.458+0000 7f76cba00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:31:35.842 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: in thread 7f76cba00640 thread_name:ms_dispatch 2024-07-26T11:31:35.842 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 2024-07-26T11:31:35.843 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:31:35.843 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f76d1c3e6f0] 2024-07-26T11:31:35.843 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f76d1c8b94c] 2024-07-26T11:31:35.843 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 3: raise() 2024-07-26T11:31:35.843 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 4: abort() 2024-07-26T11:31:35.843 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 5: /lib64/libc.so.6(+0x2871b) [0x7f76d1c2871b] 2024-07-26T11:31:35.843 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 6: /lib64/libc.so.6(+0x37386) [0x7f76d1c37386] 2024-07-26T11:31:35.843 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55f9111da6b9] 2024-07-26T11:31:35.843 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55f91139880c] 2024-07-26T11:31:35.843 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55f9112d82ca] 2024-07-26T11:31:35.843 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55f9112a8753] 2024-07-26T11:31:35.843 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55f9112a9a1b] 2024-07-26T11:31:35.843 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55f91128d752] 2024-07-26T11:31:35.843 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55f9111dba56] 2024-07-26T11:31:35.843 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55f9111df637] 2024-07-26T11:31:35.844 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55f9111dff5e] 2024-07-26T11:31:35.844 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55f9111c303a] 2024-07-26T11:31:35.844 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f76d279b407] 2024-07-26T11:31:35.844 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 18: (DispatchQueue::entry()+0x60f) [0x7f76d279bcf9] 2024-07-26T11:31:35.844 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f76d28785a1] 2024-07-26T11:31:35.844 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 20: (Thread::entry_wrapper()+0x43) [0x7f76d2625cdd] 2024-07-26T11:31:35.844 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 21: (Thread::_entry_func(void*)+0xd) [0x7f76d2625cf9] 2024-07-26T11:31:35.844 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 22: /lib64/libc.so.6(+0x89c02) [0x7f76d1c89c02] 2024-07-26T11:31:35.844 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f76d1d0ec40] 2024-07-26T11:31:35.845 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:31:35.845 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[112809]: 2024-07-26T11:31:35.845 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:35 smithi134 podman[113131]: 2024-07-26 11:31:35.563245442 +0000 UTC m=+0.031558890 container died 05e7b491d929b8305233938257a825b62eaeae8fcdb4ba7c4122089d2e11e089 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:31:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:31:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/351227695,v1:172.21.15.134:6837/351227695] up:active 2024-07-26T11:31:36.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:31:36.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:35 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:31:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:35 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:31:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:35 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/351227695,v1:172.21.15.134:6837/351227695] up:active 2024-07-26T11:31:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:35 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:31:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:35 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:31:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:35 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:31:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:35 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/351227695,v1:172.21.15.134:6837/351227695] up:active 2024-07-26T11:31:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:35 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:31:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:35 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:31:36.465 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:36 smithi134 podman[113131]: 2024-07-26 11:31:36.399623054 +0000 UTC m=+0.867936527 container remove 05e7b491d929b8305233938257a825b62eaeae8fcdb4ba7c4122089d2e11e089 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:31:36.465 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:36 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:31:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:36 smithi134 ceph-mon[31966]: pgmap v978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 254.463682 secs 2024-07-26T11:31:37.173 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:36 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T11:31:37.173 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:36 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.549s CPU time. 2024-07-26T11:31:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:36 smithi189 ceph-mon[36790]: pgmap v978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 254.463682 secs 2024-07-26T11:31:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:36 smithi193 ceph-mon[37473]: pgmap v978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:31:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 254.463682 secs 2024-07-26T11:31:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:38 smithi134 ceph-mon[31966]: pgmap v979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-07-26T11:31:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 240.038641 secs 2024-07-26T11:31:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:38 smithi189 ceph-mon[36790]: pgmap v979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-07-26T11:31:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 240.038641 secs 2024-07-26T11:31:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:38 smithi193 ceph-mon[37473]: pgmap v979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-07-26T11:31:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 240.038641 secs 2024-07-26T11:31:39.274 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:31:39 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:31:39.065+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:31:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:40 smithi134 ceph-mon[31966]: pgmap v980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:31:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:40 smithi189 ceph-mon[36790]: pgmap v980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:31:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:40 smithi193 ceph-mon[37473]: pgmap v980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:31:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 259.463802 secs 2024-07-26T11:31:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 259.463802 secs 2024-07-26T11:31:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 259.463802 secs 2024-07-26T11:31:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:42 smithi134 ceph-mon[31966]: pgmap v981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 200 B/s wr, 11 op/s 2024-07-26T11:31:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:42 smithi189 ceph-mon[36790]: pgmap v981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 200 B/s wr, 11 op/s 2024-07-26T11:31:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:42 smithi193 ceph-mon[37473]: pgmap v981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 200 B/s wr, 11 op/s 2024-07-26T11:31:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:31:43.474+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:31:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:31:43.474+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:43.476 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:31:43.474+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:31:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 245.038700 secs 2024-07-26T11:31:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 245.038700 secs 2024-07-26T11:31:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 245.038700 secs 2024-07-26T11:31:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:44 smithi134 ceph-mon[31966]: pgmap v982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:31:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:44 smithi189 ceph-mon[36790]: pgmap v982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:31:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:44 smithi193 ceph-mon[37473]: pgmap v982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:31:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:46 smithi134 ceph-mon[31966]: pgmap v983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:31:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 264.463903 secs 2024-07-26T11:31:47.173 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:46 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 2. 2024-07-26T11:31:47.173 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:46 smithi134 systemd[1]: Stopped Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:31:47.173 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:46 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.549s CPU time. 2024-07-26T11:31:47.173 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:46 smithi134 systemd[1]: Starting Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:31:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:46 smithi189 ceph-mon[36790]: pgmap v983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:31:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 264.463903 secs 2024-07-26T11:31:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:46 smithi193 ceph-mon[37473]: pgmap v983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:31:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 264.463903 secs 2024-07-26T11:31:47.637 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:47 smithi134 podman[113225]: 2024-07-26 11:31:47.210102142 +0000 UTC m=+0.035057874 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:31:47.637 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:47 smithi134 podman[113225]: 2024-07-26 11:31:47.342558344 +0000 UTC m=+0.167514067 container create 5d54e8a881ea7ca42818cdbd7324a4ec9c35a845e26cf6fa26972d482deee077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-07-26T11:31:47.637 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:47 smithi134 podman[113225]: 2024-07-26 11:31:47.635219263 +0000 UTC m=+0.460174993 container init 5d54e8a881ea7ca42818cdbd7324a4ec9c35a845e26cf6fa26972d482deee077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:31:47.896 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:47 smithi134 podman[113225]: 2024-07-26 11:31:47.647207283 +0000 UTC m=+0.472162976 container start 5d54e8a881ea7ca42818cdbd7324a4ec9c35a845e26cf6fa26972d482deee077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-07-26T11:31:47.896 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:47 smithi134 bash[113225]: 5d54e8a881ea7ca42818cdbd7324a4ec9c35a845e26cf6fa26972d482deee077 2024-07-26T11:31:47.896 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:47 smithi134 systemd[1]: Started Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:31:47.896 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:47 smithi134 ceph-mds[113244]: -- 172.21.15.134:0/4091180247 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5573ccd9f860 con 0x5573d0f41180 2024-07-26T11:31:47.896 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:31:47 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: starting mds.d at 2024-07-26T11:31:47.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:31:47.946+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:31:47.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:31:47.946+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:47.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:31:47.946+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:31:48.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:47 smithi134 ceph-mon[31966]: daemon mds.d restarted 2024-07-26T11:31:48.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:47 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:31:48.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:47 smithi134 ceph-mon[31966]: osdmap e121: 12 total, 12 up, 12 in 2024-07-26T11:31:48.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:47 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/3538082494,v1:172.21.15.134:6837/3538082494] up:boot 2024-07-26T11:31:48.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:47 smithi134 ceph-mon[31966]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-07-26T11:31:48.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:47 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:31:48.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:31:48.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:47 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:31:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:47 smithi189 ceph-mon[36790]: daemon mds.d restarted 2024-07-26T11:31:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:47 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:31:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:47 smithi189 ceph-mon[36790]: osdmap e121: 12 total, 12 up, 12 in 2024-07-26T11:31:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:47 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/3538082494,v1:172.21.15.134:6837/3538082494] up:boot 2024-07-26T11:31:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:47 smithi189 ceph-mon[36790]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-07-26T11:31:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:47 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:31:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:31:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:47 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:31:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:47 smithi193 ceph-mon[37473]: daemon mds.d restarted 2024-07-26T11:31:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:47 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:31:48.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:47 smithi193 ceph-mon[37473]: osdmap e121: 12 total, 12 up, 12 in 2024-07-26T11:31:48.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:47 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/3538082494,v1:172.21.15.134:6837/3538082494] up:boot 2024-07-26T11:31:48.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:47 smithi193 ceph-mon[37473]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-07-26T11:31:48.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:47 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:31:48.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:31:48.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:47 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:31:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:48 smithi134 ceph-mon[31966]: pgmap v984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:31:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 250.038775 secs 2024-07-26T11:31:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:48 smithi189 ceph-mon[36790]: pgmap v984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:31:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 250.038775 secs 2024-07-26T11:31:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:48 smithi193 ceph-mon[37473]: pgmap v984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:31:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 250.038775 secs 2024-07-26T11:31:50.179 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:49 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/679901285,v1:172.21.15.134:6839/679901285] up:resolve 2024-07-26T11:31:50.179 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:49 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:31:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:49 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/679901285,v1:172.21.15.134:6839/679901285] up:resolve 2024-07-26T11:31:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:49 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:31:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:49 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/679901285,v1:172.21.15.134:6839/679901285] up:resolve 2024-07-26T11:31:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:49 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:31:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:50 smithi189 ceph-mon[36790]: pgmap v986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:31:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:50 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299993 2024-07-26T11:31:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:50 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/679901285,v1:172.21.15.134:6839/679901285] up:reconnect 2024-07-26T11:31:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:50 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:31:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:50 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:31:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:50 smithi193 ceph-mon[37473]: pgmap v986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:31:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:50 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299993 2024-07-26T11:31:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:50 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/679901285,v1:172.21.15.134:6839/679901285] up:reconnect 2024-07-26T11:31:51.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:50 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:31:51.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:50 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:31:51.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:50 smithi134 ceph-mon[31966]: pgmap v986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:31:51.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:50 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299993 2024-07-26T11:31:51.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:50 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/679901285,v1:172.21.15.134:6839/679901285] up:reconnect 2024-07-26T11:31:51.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:50 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:31:51.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:50 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:31:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 269.464022 secs 2024-07-26T11:31:52.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:51 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/679901285,v1:172.21.15.134:6839/679901285] up:rejoin 2024-07-26T11:31:52.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:51 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:31:52.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:51 smithi193 ceph-mon[37473]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:31:52.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:51 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:31:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 269.464022 secs 2024-07-26T11:31:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/679901285,v1:172.21.15.134:6839/679901285] up:rejoin 2024-07-26T11:31:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:31:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-mon[31966]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:31:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:31:52.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:31:52.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: *** Caught signal (Aborted) ** 2024-07-26T11:31:52.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: in thread 7fe254400640 thread_name:ms_dispatch 2024-07-26T11:31:52.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:31:52.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe25a63e6f0] 2024-07-26T11:31:52.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe25a68b94c] 2024-07-26T11:31:52.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 3: raise() 2024-07-26T11:31:52.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 4: abort() 2024-07-26T11:31:52.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe25a62871b] 2024-07-26T11:31:52.424 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 6: /lib64/libc.so.6(+0x37386) [0x7fe25a637386] 2024-07-26T11:31:52.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55beb5d8f6b9] 2024-07-26T11:31:52.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55beb5f4d80c] 2024-07-26T11:31:52.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55beb5e8d2ca] 2024-07-26T11:31:52.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55beb5e5d753] 2024-07-26T11:31:52.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55beb5e5ea1b] 2024-07-26T11:31:52.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55beb5e42752] 2024-07-26T11:31:52.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55beb5d90a56] 2024-07-26T11:31:52.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55beb5d94637] 2024-07-26T11:31:52.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55beb5d94f5e] 2024-07-26T11:31:52.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55beb5d7803a] 2024-07-26T11:31:52.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe25b19b407] 2024-07-26T11:31:52.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 18: (DispatchQueue::entry()+0x60f) [0x7fe25b19bcf9] 2024-07-26T11:31:52.425 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe25b2785a1] 2024-07-26T11:31:52.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 20: (Thread::entry_wrapper()+0x43) [0x7fe25b025cdd] 2024-07-26T11:31:52.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe25b025cf9] 2024-07-26T11:31:52.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe25a689c02] 2024-07-26T11:31:52.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe25a70ec40] 2024-07-26T11:31:52.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 2024-07-26T11:31:51.937+0000 7fe254400640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:31:52.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: in thread 7fe254400640 thread_name:ms_dispatch 2024-07-26T11:31:52.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 2024-07-26T11:31:52.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:31:52.426 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe25a63e6f0] 2024-07-26T11:31:52.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe25a68b94c] 2024-07-26T11:31:52.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 3: raise() 2024-07-26T11:31:52.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 4: abort() 2024-07-26T11:31:52.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe25a62871b] 2024-07-26T11:31:52.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 6: /lib64/libc.so.6(+0x37386) [0x7fe25a637386] 2024-07-26T11:31:52.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55beb5d8f6b9] 2024-07-26T11:31:52.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55beb5f4d80c] 2024-07-26T11:31:52.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55beb5e8d2ca] 2024-07-26T11:31:52.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55beb5e5d753] 2024-07-26T11:31:52.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55beb5e5ea1b] 2024-07-26T11:31:52.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55beb5e42752] 2024-07-26T11:31:52.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55beb5d90a56] 2024-07-26T11:31:52.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55beb5d94637] 2024-07-26T11:31:52.427 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55beb5d94f5e] 2024-07-26T11:31:52.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55beb5d7803a] 2024-07-26T11:31:52.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe25b19b407] 2024-07-26T11:31:52.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 18: (DispatchQueue::entry()+0x60f) [0x7fe25b19bcf9] 2024-07-26T11:31:52.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe25b2785a1] 2024-07-26T11:31:52.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 20: (Thread::entry_wrapper()+0x43) [0x7fe25b025cdd] 2024-07-26T11:31:52.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe25b025cf9] 2024-07-26T11:31:52.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe25a689c02] 2024-07-26T11:31:52.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe25a70ec40] 2024-07-26T11:31:52.428 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:31:52.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 2024-07-26T11:31:52.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 0> 2024-07-26T11:31:51.937+0000 7fe254400640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:31:52.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: in thread 7fe254400640 thread_name:ms_dispatch 2024-07-26T11:31:52.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 2024-07-26T11:31:52.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:31:52.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe25a63e6f0] 2024-07-26T11:31:52.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe25a68b94c] 2024-07-26T11:31:52.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 3: raise() 2024-07-26T11:31:52.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 4: abort() 2024-07-26T11:31:52.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe25a62871b] 2024-07-26T11:31:52.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 6: /lib64/libc.so.6(+0x37386) [0x7fe25a637386] 2024-07-26T11:31:52.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55beb5d8f6b9] 2024-07-26T11:31:52.429 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55beb5f4d80c] 2024-07-26T11:31:52.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55beb5e8d2ca] 2024-07-26T11:31:52.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55beb5e5d753] 2024-07-26T11:31:52.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55beb5e5ea1b] 2024-07-26T11:31:52.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55beb5e42752] 2024-07-26T11:31:52.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55beb5d90a56] 2024-07-26T11:31:52.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55beb5d94637] 2024-07-26T11:31:52.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55beb5d94f5e] 2024-07-26T11:31:52.430 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55beb5d7803a] 2024-07-26T11:31:52.431 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe25b19b407] 2024-07-26T11:31:52.431 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 18: (DispatchQueue::entry()+0x60f) [0x7fe25b19bcf9] 2024-07-26T11:31:52.431 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe25b2785a1] 2024-07-26T11:31:52.431 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 20: (Thread::entry_wrapper()+0x43) [0x7fe25b025cdd] 2024-07-26T11:31:52.431 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe25b025cf9] 2024-07-26T11:31:52.431 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe25a689c02] 2024-07-26T11:31:52.431 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe25a70ec40] 2024-07-26T11:31:52.431 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:31:52.431 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 2024-07-26T11:31:52.431 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 0> 2024-07-26T11:31:51.937+0000 7fe254400640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:31:52.431 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: in thread 7fe254400640 thread_name:ms_dispatch 2024-07-26T11:31:52.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 2024-07-26T11:31:52.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:31:52.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fe25a63e6f0] 2024-07-26T11:31:52.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fe25a68b94c] 2024-07-26T11:31:52.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 3: raise() 2024-07-26T11:31:52.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 4: abort() 2024-07-26T11:31:52.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 5: /lib64/libc.so.6(+0x2871b) [0x7fe25a62871b] 2024-07-26T11:31:52.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 6: /lib64/libc.so.6(+0x37386) [0x7fe25a637386] 2024-07-26T11:31:52.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55beb5d8f6b9] 2024-07-26T11:31:52.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55beb5f4d80c] 2024-07-26T11:31:52.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55beb5e8d2ca] 2024-07-26T11:31:52.432 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55beb5e5d753] 2024-07-26T11:31:52.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55beb5e5ea1b] 2024-07-26T11:31:52.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55beb5e42752] 2024-07-26T11:31:52.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55beb5d90a56] 2024-07-26T11:31:52.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55beb5d94637] 2024-07-26T11:31:52.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55beb5d94f5e] 2024-07-26T11:31:52.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55beb5d7803a] 2024-07-26T11:31:52.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fe25b19b407] 2024-07-26T11:31:52.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 18: (DispatchQueue::entry()+0x60f) [0x7fe25b19bcf9] 2024-07-26T11:31:52.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fe25b2785a1] 2024-07-26T11:31:52.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 20: (Thread::entry_wrapper()+0x43) [0x7fe25b025cdd] 2024-07-26T11:31:52.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 21: (Thread::_entry_func(void*)+0xd) [0x7fe25b025cf9] 2024-07-26T11:31:52.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 22: /lib64/libc.so.6(+0x89c02) [0x7fe25a689c02] 2024-07-26T11:31:52.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fe25a70ec40] 2024-07-26T11:31:52.433 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:31:52.434 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:51 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113085]: 2024-07-26T11:31:52.434 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:52 smithi134 podman[113278]: 2024-07-26 11:31:52.08226889 +0000 UTC m=+0.031707954 container died 710b299842944119ae6aa869ef8a65302662d412e30d195875a73a70ab1c4275 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.36.0) 2024-07-26T11:31:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 269.464022 secs 2024-07-26T11:31:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:51 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/679901285,v1:172.21.15.134:6839/679901285] up:rejoin 2024-07-26T11:31:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:51 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:31:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:51 smithi189 ceph-mon[36790]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:31:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:51 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:31:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:52 smithi193 ceph-mon[37473]: pgmap v987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:31:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:52 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/679901285,v1:172.21.15.134:6839/679901285] up:active 2024-07-26T11:31:53.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:52 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:31:53.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:52 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:31:53.361 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:52 smithi134 ceph-mon[31966]: pgmap v987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:31:53.361 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:52 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/679901285,v1:172.21.15.134:6839/679901285] up:active 2024-07-26T11:31:53.361 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:52 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:31:53.361 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:52 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:31:53.362 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:52 smithi134 podman[113278]: 2024-07-26 11:31:52.950203244 +0000 UTC m=+0.899642362 container remove 710b299842944119ae6aa869ef8a65302662d412e30d195875a73a70ab1c4275 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD) 2024-07-26T11:31:53.362 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:52 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:31:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:52 smithi189 ceph-mon[36790]: pgmap v987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:31:53.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:52 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/679901285,v1:172.21.15.134:6839/679901285] up:active 2024-07-26T11:31:53.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:52 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:31:53.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:52 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:31:53.672 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:53 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-07-26T11:31:53.672 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:31:53 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Consumed 1.583s CPU time. 2024-07-26T11:31:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:53 smithi134 ceph-mon[31966]: pgmap v988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:31:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 255.038845 secs 2024-07-26T11:31:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:53 smithi193 ceph-mon[37473]: pgmap v988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:31:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 255.038845 secs 2024-07-26T11:31:54.274 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:31:54 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:31:54.065+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:31:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:53 smithi189 ceph-mon[36790]: pgmap v988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:31:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 255.038845 secs 2024-07-26T11:31:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:56 smithi134 ceph-mon[31966]: pgmap v989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:31:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 274.464163 secs 2024-07-26T11:31:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:56 smithi189 ceph-mon[36790]: pgmap v989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:31:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 274.464163 secs 2024-07-26T11:31:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:56 smithi193 ceph-mon[37473]: pgmap v989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:31:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 274.464163 secs 2024-07-26T11:31:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:58 smithi134 ceph-mon[31966]: pgmap v990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:31:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:58 smithi189 ceph-mon[36790]: pgmap v990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:31:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:31:58.490+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:31:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:31:58.490+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:31:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:31:58.490+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:31:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:58 smithi193 ceph-mon[37473]: pgmap v990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:31:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:31:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 260.038921 secs 2024-07-26T11:31:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:31:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 260.038921 secs 2024-07-26T11:31:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:31:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 260.038921 secs 2024-07-26T11:32:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:00 smithi134 ceph-mon[31966]: pgmap v991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 182 B/s wr, 10 op/s 2024-07-26T11:32:00.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:00 smithi189 ceph-mon[36790]: pgmap v991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 182 B/s wr, 10 op/s 2024-07-26T11:32:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:00 smithi193 ceph-mon[37473]: pgmap v991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 182 B/s wr, 10 op/s 2024-07-26T11:32:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 279.464332 secs 2024-07-26T11:32:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 279.464332 secs 2024-07-26T11:32:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 279.464332 secs 2024-07-26T11:32:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:02 smithi134 ceph-mon[31966]: pgmap v992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:02 smithi189 ceph-mon[36790]: pgmap v992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:02 smithi193 ceph-mon[37473]: pgmap v992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:02.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:32:02.958+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:32:02.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:32:02.958+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:32:02.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:32:02.958+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:04.076 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:03 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 4. 2024-07-26T11:32:04.076 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:03 smithi134 systemd[1]: Stopped Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:32:04.076 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:03 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Consumed 1.583s CPU time. 2024-07-26T11:32:04.076 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:03 smithi134 systemd[1]: Starting Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:32:04.414 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:04 smithi134 ceph-mon[31966]: pgmap v993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:32:04.414 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 265.039035 secs 2024-07-26T11:32:04.414 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:04 smithi134 podman[113367]: 2024-07-26 11:32:03.978989498 +0000 UTC m=+0.032603079 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:32:04.414 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:04 smithi134 podman[113367]: 2024-07-26 11:32:04.083416258 +0000 UTC m=+0.137029851 container create 376c9e86ea0473af69587a13844483b5fe5ac63cb19fbdb7e1959d05ea7d8dc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-07-26T11:32:04.415 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:04 smithi134 podman[113367]: 2024-07-26 11:32:04.41242729 +0000 UTC m=+0.466040965 container init 376c9e86ea0473af69587a13844483b5fe5ac63cb19fbdb7e1959d05ea7d8dc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:32:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:04 smithi189 ceph-mon[36790]: pgmap v993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:32:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 265.039035 secs 2024-07-26T11:32:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:04 smithi193 ceph-mon[37473]: pgmap v993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:32:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 265.039035 secs 2024-07-26T11:32:04.673 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:04 smithi134 podman[113367]: 2024-07-26 11:32:04.422924358 +0000 UTC m=+0.476537939 container start 376c9e86ea0473af69587a13844483b5fe5ac63cb19fbdb7e1959d05ea7d8dc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-07-26T11:32:04.673 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:04 smithi134 bash[113367]: 376c9e86ea0473af69587a13844483b5fe5ac63cb19fbdb7e1959d05ea7d8dc6 2024-07-26T11:32:04.673 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:04 smithi134 systemd[1]: Started Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:32:04.673 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:04 smithi134 ceph-mds[113386]: -- 172.21.15.134:0/2554522374 <== mon.0 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558f0fcbd860 con 0x558f13e5fa80 2024-07-26T11:32:04.673 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:04 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: starting mds.g at 2024-07-26T11:32:05.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:05 smithi134 ceph-mon[31966]: daemon mds.g restarted 2024-07-26T11:32:05.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:05 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:32:05.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:05 smithi134 ceph-mon[31966]: osdmap e122: 12 total, 12 up, 12 in 2024-07-26T11:32:05.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:05 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/1942695211,v1:172.21.15.134:6839/1942695211] up:boot 2024-07-26T11:32:05.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:05 smithi134 ceph-mon[31966]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:32:05.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:05 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:32:05.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:32:05.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:05 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:32:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:05 smithi189 ceph-mon[36790]: daemon mds.g restarted 2024-07-26T11:32:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:05 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:32:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:05 smithi189 ceph-mon[36790]: osdmap e122: 12 total, 12 up, 12 in 2024-07-26T11:32:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:05 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/1942695211,v1:172.21.15.134:6839/1942695211] up:boot 2024-07-26T11:32:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:05 smithi189 ceph-mon[36790]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:32:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:05 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:32:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:32:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:05 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:32:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:05 smithi193 ceph-mon[37473]: daemon mds.g restarted 2024-07-26T11:32:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:05 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:32:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:05 smithi193 ceph-mon[37473]: osdmap e122: 12 total, 12 up, 12 in 2024-07-26T11:32:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:05 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/1942695211,v1:172.21.15.134:6839/1942695211] up:boot 2024-07-26T11:32:05.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:05 smithi193 ceph-mon[37473]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:32:05.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:05 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:32:05.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:32:05.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:05 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:32:06.841 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:06 smithi134 ceph-mon[31966]: pgmap v995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-07-26T11:32:06.841 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:06 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/3538082494,v1:172.21.15.134:6837/3538082494] up:resolve 2024-07-26T11:32:06.841 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:06 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:32:06.841 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 284.464429 secs 2024-07-26T11:32:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:06 smithi189 ceph-mon[36790]: pgmap v995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-07-26T11:32:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:06 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/3538082494,v1:172.21.15.134:6837/3538082494] up:resolve 2024-07-26T11:32:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:06 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:32:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 284.464429 secs 2024-07-26T11:32:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:06 smithi193 ceph-mon[37473]: pgmap v995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.0 KiB/s rd, 10 op/s 2024-07-26T11:32:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:06 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/3538082494,v1:172.21.15.134:6837/3538082494] up:resolve 2024-07-26T11:32:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:06 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:32:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 284.464429 secs 2024-07-26T11:32:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:07 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00399991 2024-07-26T11:32:07.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:07 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/3538082494,v1:172.21.15.134:6837/3538082494] up:reconnect 2024-07-26T11:32:07.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:07 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:32:07.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:07 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0129997 2024-07-26T11:32:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:07 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00399991 2024-07-26T11:32:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:07 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/3538082494,v1:172.21.15.134:6837/3538082494] up:reconnect 2024-07-26T11:32:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:07 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:32:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:07 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0129997 2024-07-26T11:32:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:07 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00399991 2024-07-26T11:32:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:07 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/3538082494,v1:172.21.15.134:6837/3538082494] up:reconnect 2024-07-26T11:32:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:07 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:32:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:07 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0129997 2024-07-26T11:32:08.607 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:08 smithi193 ceph-mon[37473]: pgmap v996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T11:32:08.607 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:08 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/3538082494,v1:172.21.15.134:6837/3538082494] up:rejoin 2024-07-26T11:32:08.607 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:08 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:32:08.607 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:08 smithi193 ceph-mon[37473]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:32:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-mon[31966]: pgmap v996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T11:32:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/3538082494,v1:172.21.15.134:6837/3538082494] up:rejoin 2024-07-26T11:32:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:32:08.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-mon[31966]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:32:08.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 270.039132 secs 2024-07-26T11:32:08.924 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:32:08.924 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: *** Caught signal (Aborted) ** 2024-07-26T11:32:08.924 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: in thread 7f2cae800640 thread_name:ms_dispatch 2024-07-26T11:32:08.924 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:32:08.924 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2cb4a3e6f0] 2024-07-26T11:32:08.924 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2cb4a8b94c] 2024-07-26T11:32:08.924 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 3: raise() 2024-07-26T11:32:08.924 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 4: abort() 2024-07-26T11:32:08.924 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2cb4a2871b] 2024-07-26T11:32:08.924 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 6: /lib64/libc.so.6(+0x37386) [0x7f2cb4a37386] 2024-07-26T11:32:08.925 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5573c4fdc6b9] 2024-07-26T11:32:08.925 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5573c519a80c] 2024-07-26T11:32:08.925 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5573c50da2ca] 2024-07-26T11:32:08.925 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5573c50aa753] 2024-07-26T11:32:08.925 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5573c50aba1b] 2024-07-26T11:32:08.925 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5573c508f752] 2024-07-26T11:32:08.925 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5573c4fdda56] 2024-07-26T11:32:08.925 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5573c4fe1637] 2024-07-26T11:32:08.925 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5573c4fe1f5e] 2024-07-26T11:32:08.925 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5573c4fc503a] 2024-07-26T11:32:08.925 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f2cb559b407] 2024-07-26T11:32:08.925 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 18: (DispatchQueue::entry()+0x60f) [0x7f2cb559bcf9] 2024-07-26T11:32:08.926 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2cb56785a1] 2024-07-26T11:32:08.926 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 20: (Thread::entry_wrapper()+0x43) [0x7f2cb5425cdd] 2024-07-26T11:32:08.926 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2cb5425cf9] 2024-07-26T11:32:08.926 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2cb4a89c02] 2024-07-26T11:32:08.926 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2cb4b0ec40] 2024-07-26T11:32:08.926 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 2024-07-26T11:32:08.624+0000 7f2cae800640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:32:08.926 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: in thread 7f2cae800640 thread_name:ms_dispatch 2024-07-26T11:32:08.926 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 2024-07-26T11:32:08.927 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:32:08.927 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2cb4a3e6f0] 2024-07-26T11:32:08.927 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2cb4a8b94c] 2024-07-26T11:32:08.927 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 3: raise() 2024-07-26T11:32:08.927 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 4: abort() 2024-07-26T11:32:08.927 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2cb4a2871b] 2024-07-26T11:32:08.927 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 6: /lib64/libc.so.6(+0x37386) [0x7f2cb4a37386] 2024-07-26T11:32:08.927 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5573c4fdc6b9] 2024-07-26T11:32:08.927 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5573c519a80c] 2024-07-26T11:32:08.927 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5573c50da2ca] 2024-07-26T11:32:08.927 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5573c50aa753] 2024-07-26T11:32:08.927 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5573c50aba1b] 2024-07-26T11:32:08.928 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5573c508f752] 2024-07-26T11:32:08.928 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5573c4fdda56] 2024-07-26T11:32:08.928 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5573c4fe1637] 2024-07-26T11:32:08.928 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5573c4fe1f5e] 2024-07-26T11:32:08.928 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5573c4fc503a] 2024-07-26T11:32:08.928 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f2cb559b407] 2024-07-26T11:32:08.928 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 18: (DispatchQueue::entry()+0x60f) [0x7f2cb559bcf9] 2024-07-26T11:32:08.928 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2cb56785a1] 2024-07-26T11:32:08.929 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 20: (Thread::entry_wrapper()+0x43) [0x7f2cb5425cdd] 2024-07-26T11:32:08.929 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2cb5425cf9] 2024-07-26T11:32:08.929 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2cb4a89c02] 2024-07-26T11:32:08.929 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2cb4b0ec40] 2024-07-26T11:32:08.929 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:32:08.929 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 2024-07-26T11:32:08.929 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 0> 2024-07-26T11:32:08.624+0000 7f2cae800640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:32:08.929 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: in thread 7f2cae800640 thread_name:ms_dispatch 2024-07-26T11:32:08.929 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 2024-07-26T11:32:08.930 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:32:08.930 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2cb4a3e6f0] 2024-07-26T11:32:08.930 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2cb4a8b94c] 2024-07-26T11:32:08.930 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 3: raise() 2024-07-26T11:32:08.930 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 4: abort() 2024-07-26T11:32:08.930 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2cb4a2871b] 2024-07-26T11:32:08.930 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 6: /lib64/libc.so.6(+0x37386) [0x7f2cb4a37386] 2024-07-26T11:32:08.930 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5573c4fdc6b9] 2024-07-26T11:32:08.930 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5573c519a80c] 2024-07-26T11:32:08.930 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5573c50da2ca] 2024-07-26T11:32:08.931 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5573c50aa753] 2024-07-26T11:32:08.931 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5573c50aba1b] 2024-07-26T11:32:08.931 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5573c508f752] 2024-07-26T11:32:08.931 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5573c4fdda56] 2024-07-26T11:32:08.931 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5573c4fe1637] 2024-07-26T11:32:08.931 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5573c4fe1f5e] 2024-07-26T11:32:08.931 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5573c4fc503a] 2024-07-26T11:32:08.931 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f2cb559b407] 2024-07-26T11:32:08.931 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 18: (DispatchQueue::entry()+0x60f) [0x7f2cb559bcf9] 2024-07-26T11:32:08.931 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2cb56785a1] 2024-07-26T11:32:08.931 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 20: (Thread::entry_wrapper()+0x43) [0x7f2cb5425cdd] 2024-07-26T11:32:08.931 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2cb5425cf9] 2024-07-26T11:32:08.932 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2cb4a89c02] 2024-07-26T11:32:08.932 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2cb4b0ec40] 2024-07-26T11:32:08.932 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:32:08.932 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 2024-07-26T11:32:08.932 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: -2> 2024-07-26T11:32:08.624+0000 7f2cae800640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:32:08.932 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: in thread 7f2cae800640 thread_name:ms_dispatch 2024-07-26T11:32:08.932 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 2024-07-26T11:32:08.932 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:32:08.932 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f2cb4a3e6f0] 2024-07-26T11:32:08.932 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f2cb4a8b94c] 2024-07-26T11:32:08.932 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 3: raise() 2024-07-26T11:32:08.932 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 4: abort() 2024-07-26T11:32:08.932 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 5: /lib64/libc.so.6(+0x2871b) [0x7f2cb4a2871b] 2024-07-26T11:32:08.932 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 6: /lib64/libc.so.6(+0x37386) [0x7f2cb4a37386] 2024-07-26T11:32:08.933 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x5573c4fdc6b9] 2024-07-26T11:32:08.933 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x5573c519a80c] 2024-07-26T11:32:08.933 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x5573c50da2ca] 2024-07-26T11:32:08.933 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x5573c50aa753] 2024-07-26T11:32:08.933 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x5573c50aba1b] 2024-07-26T11:32:08.933 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x5573c508f752] 2024-07-26T11:32:08.933 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x5573c4fdda56] 2024-07-26T11:32:08.933 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x5573c4fe1637] 2024-07-26T11:32:08.934 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x5573c4fe1f5e] 2024-07-26T11:32:08.934 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x5573c4fc503a] 2024-07-26T11:32:08.934 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f2cb559b407] 2024-07-26T11:32:08.934 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 18: (DispatchQueue::entry()+0x60f) [0x7f2cb559bcf9] 2024-07-26T11:32:08.934 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f2cb56785a1] 2024-07-26T11:32:08.934 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 20: (Thread::entry_wrapper()+0x43) [0x7f2cb5425cdd] 2024-07-26T11:32:08.934 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 21: (Thread::_entry_func(void*)+0xd) [0x7f2cb5425cf9] 2024-07-26T11:32:08.934 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 22: /lib64/libc.so.6(+0x89c02) [0x7f2cb4a89c02] 2024-07-26T11:32:08.934 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f2cb4b0ec40] 2024-07-26T11:32:08.934 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:32:08.934 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113240]: 2024-07-26T11:32:08.934 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:08 smithi134 podman[113420]: 2024-07-26 11:32:08.74402883 +0000 UTC m=+0.031836617 container died 5d54e8a881ea7ca42818cdbd7324a4ec9c35a845e26cf6fa26972d482deee077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:32:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:08 smithi189 ceph-mon[36790]: pgmap v996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T11:32:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:08 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/3538082494,v1:172.21.15.134:6837/3538082494] up:rejoin 2024-07-26T11:32:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:08 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:32:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:08 smithi189 ceph-mon[36790]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:32:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 270.039132 secs 2024-07-26T11:32:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 270.039132 secs 2024-07-26T11:32:09.524 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:32:09 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:32:09.066+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:32:09.923 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:09 smithi134 podman[113420]: 2024-07-26 11:32:09.724554038 +0000 UTC m=+1.012361876 container remove 5d54e8a881ea7ca42818cdbd7324a4ec9c35a845e26cf6fa26972d482deee077 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-07-26T11:32:09.923 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:09 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:32:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:09 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:32:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:09 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/3538082494,v1:172.21.15.134:6837/3538082494] up:active 2024-07-26T11:32:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:09 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:32:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:09 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:32:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:09 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:32:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:09 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/3538082494,v1:172.21.15.134:6837/3538082494] up:active 2024-07-26T11:32:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:09 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:32:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:09 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:32:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:09 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:32:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:09 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/3538082494,v1:172.21.15.134:6837/3538082494] up:active 2024-07-26T11:32:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:09 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:32:10.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:09 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:32:10.257 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:10 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T11:32:10.257 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:10 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.580s CPU time. 2024-07-26T11:32:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:10 smithi134 ceph-mon[31966]: pgmap v997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T11:32:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:10 smithi189 ceph-mon[36790]: pgmap v997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T11:32:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:10 smithi193 ceph-mon[37473]: pgmap v997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T11:32:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 289.464554 secs 2024-07-26T11:32:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 289.464554 secs 2024-07-26T11:32:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 289.464554 secs 2024-07-26T11:32:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:12 smithi134 ceph-mon[31966]: pgmap v998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:32:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:12 smithi189 ceph-mon[36790]: pgmap v998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:32:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:12 smithi193 ceph-mon[37473]: pgmap v998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:32:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:32:13.493+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:32:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:32:13.493+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:32:13.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:32:13.493+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 275.039200 secs 2024-07-26T11:32:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 275.039200 secs 2024-07-26T11:32:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 275.039200 secs 2024-07-26T11:32:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:14 smithi134 ceph-mon[31966]: pgmap v999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:32:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:14 smithi189 ceph-mon[36790]: pgmap v999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:32:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:14 smithi193 ceph-mon[37473]: pgmap v999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:32:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:16 smithi134 ceph-mon[31966]: pgmap v1000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 195 B/s wr, 10 op/s 2024-07-26T11:32:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 294.464626 secs 2024-07-26T11:32:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:16 smithi189 ceph-mon[36790]: pgmap v1000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 195 B/s wr, 10 op/s 2024-07-26T11:32:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 294.464626 secs 2024-07-26T11:32:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:16 smithi193 ceph-mon[37473]: pgmap v1000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 195 B/s wr, 10 op/s 2024-07-26T11:32:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 294.464626 secs 2024-07-26T11:32:17.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:32:17.970+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:32:17.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:32:17.970+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:17.972 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:32:17.970+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:32:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:18 smithi189 ceph-mon[36790]: pgmap v1001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 280.039296 secs 2024-07-26T11:32:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:18 smithi193 ceph-mon[37473]: pgmap v1001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 280.039296 secs 2024-07-26T11:32:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:18 smithi134 ceph-mon[31966]: pgmap v1001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 280.039296 secs 2024-07-26T11:32:20.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:20 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 3. 2024-07-26T11:32:20.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:20 smithi134 systemd[1]: Stopped Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:32:20.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:20 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.580s CPU time. 2024-07-26T11:32:20.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:20 smithi134 systemd[1]: Starting Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:32:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:20 smithi189 ceph-mon[36790]: pgmap v1002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:32:21.001 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:20 smithi134 ceph-mon[31966]: pgmap v1002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:32:21.001 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:20 smithi134 podman[113512]: 2024-07-26 11:32:20.714603948 +0000 UTC m=+0.030292040 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:32:21.001 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:20 smithi134 podman[113512]: 2024-07-26 11:32:20.833192869 +0000 UTC m=+0.148880964 container create d433ae54cc741068e648dc3b7eef85f7ad25229cad7d914491138c0fbbc5e544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:32:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:20 smithi193 ceph-mon[37473]: pgmap v1002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:32:21.423 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:21 smithi134 podman[113512]: 2024-07-26 11:32:21.094851849 +0000 UTC m=+0.410539954 container init d433ae54cc741068e648dc3b7eef85f7ad25229cad7d914491138c0fbbc5e544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724) 2024-07-26T11:32:21.423 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:21 smithi134 podman[113512]: 2024-07-26 11:32:21.105433343 +0000 UTC m=+0.421121497 container start d433ae54cc741068e648dc3b7eef85f7ad25229cad7d914491138c0fbbc5e544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0) 2024-07-26T11:32:21.423 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:21 smithi134 ceph-mds[113531]: -- 172.21.15.134:0/672456744 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560028b25860 con 0x56002ccc7a80 2024-07-26T11:32:21.423 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:21 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: starting mds.d at 2024-07-26T11:32:21.423 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:21 smithi134 bash[113512]: d433ae54cc741068e648dc3b7eef85f7ad25229cad7d914491138c0fbbc5e544 2024-07-26T11:32:21.423 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:21 smithi134 systemd[1]: Started Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:32:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 299.464776 secs 2024-07-26T11:32:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:21 smithi189 ceph-mon[36790]: daemon mds.d restarted 2024-07-26T11:32:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:21 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:32:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:21 smithi189 ceph-mon[36790]: osdmap e123: 12 total, 12 up, 12 in 2024-07-26T11:32:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:21 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/4263092556,v1:172.21.15.134:6837/4263092556] up:boot 2024-07-26T11:32:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:21 smithi189 ceph-mon[36790]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-07-26T11:32:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:21 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:32:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:32:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:21 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:32:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 299.464776 secs 2024-07-26T11:32:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:21 smithi193 ceph-mon[37473]: daemon mds.d restarted 2024-07-26T11:32:22.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:21 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:32:22.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:21 smithi193 ceph-mon[37473]: osdmap e123: 12 total, 12 up, 12 in 2024-07-26T11:32:22.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:21 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/4263092556,v1:172.21.15.134:6837/4263092556] up:boot 2024-07-26T11:32:22.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:21 smithi193 ceph-mon[37473]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-07-26T11:32:22.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:21 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:32:22.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:32:22.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:21 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:32:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 299.464776 secs 2024-07-26T11:32:22.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:21 smithi134 ceph-mon[31966]: daemon mds.d restarted 2024-07-26T11:32:22.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:21 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:32:22.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:21 smithi134 ceph-mon[31966]: osdmap e123: 12 total, 12 up, 12 in 2024-07-26T11:32:22.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:21 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/4263092556,v1:172.21.15.134:6837/4263092556] up:boot 2024-07-26T11:32:22.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:21 smithi134 ceph-mon[31966]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-07-26T11:32:22.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:21 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:32:22.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:32:22.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:21 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:32:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:22 smithi193 ceph-mon[37473]: pgmap v1003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:22 smithi134 ceph-mon[31966]: pgmap v1003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:22 smithi189 ceph-mon[36790]: pgmap v1003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:23.976 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:23 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/1942695211,v1:172.21.15.134:6839/1942695211] up:resolve 2024-07-26T11:32:23.977 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:23 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:32:23.977 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 285.039406 secs 2024-07-26T11:32:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:23 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/1942695211,v1:172.21.15.134:6839/1942695211] up:resolve 2024-07-26T11:32:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:23 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:32:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 285.039406 secs 2024-07-26T11:32:24.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:23 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/1942695211,v1:172.21.15.134:6839/1942695211] up:resolve 2024-07-26T11:32:24.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:23 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:32:24.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 285.039406 secs 2024-07-26T11:32:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:24 smithi193 ceph-mon[37473]: pgmap v1005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:32:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:24 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00399991 2024-07-26T11:32:25.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:24 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/1942695211,v1:172.21.15.134:6839/1942695211] up:reconnect 2024-07-26T11:32:25.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:24 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:32:25.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:24 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:32:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:24 smithi134 ceph-mon[31966]: pgmap v1005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:32:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:24 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00399991 2024-07-26T11:32:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:24 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/1942695211,v1:172.21.15.134:6839/1942695211] up:reconnect 2024-07-26T11:32:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:24 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:32:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:24 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:32:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:24 smithi189 ceph-mon[36790]: pgmap v1005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:32:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:24 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00399991 2024-07-26T11:32:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:24 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/1942695211,v1:172.21.15.134:6839/1942695211] up:reconnect 2024-07-26T11:32:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:24 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:32:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:24 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:32:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:25 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/1942695211,v1:172.21.15.134:6839/1942695211] up:rejoin 2024-07-26T11:32:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:25 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:32:26.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:25 smithi193 ceph-mon[37473]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:32:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/1942695211,v1:172.21.15.134:6839/1942695211] up:rejoin 2024-07-26T11:32:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:32:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-mon[31966]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:32:26.173 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:32:26.173 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: *** Caught signal (Aborted) ** 2024-07-26T11:32:26.174 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: in thread 7f1309c00640 thread_name:ms_dispatch 2024-07-26T11:32:26.174 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:32:26.174 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f130fe3e6f0] 2024-07-26T11:32:26.174 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f130fe8b94c] 2024-07-26T11:32:26.174 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 3: raise() 2024-07-26T11:32:26.174 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 4: abort() 2024-07-26T11:32:26.174 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 5: /lib64/libc.so.6(+0x2871b) [0x7f130fe2871b] 2024-07-26T11:32:26.174 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 6: /lib64/libc.so.6(+0x37386) [0x7f130fe37386] 2024-07-26T11:32:26.175 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558ecfe2e6b9] 2024-07-26T11:32:26.175 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558ecffec80c] 2024-07-26T11:32:26.175 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558ecff2c2ca] 2024-07-26T11:32:26.175 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558ecfefc753] 2024-07-26T11:32:26.175 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558ecfefda1b] 2024-07-26T11:32:26.175 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558ecfee1752] 2024-07-26T11:32:26.175 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558ecfe2fa56] 2024-07-26T11:32:26.175 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558ecfe33637] 2024-07-26T11:32:26.175 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558ecfe33f5e] 2024-07-26T11:32:26.175 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558ecfe1703a] 2024-07-26T11:32:26.175 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f131099b407] 2024-07-26T11:32:26.175 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 18: (DispatchQueue::entry()+0x60f) [0x7f131099bcf9] 2024-07-26T11:32:26.176 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1310a785a1] 2024-07-26T11:32:26.177 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 20: (Thread::entry_wrapper()+0x43) [0x7f1310825cdd] 2024-07-26T11:32:26.177 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1310825cf9] 2024-07-26T11:32:26.177 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 22: /lib64/libc.so.6(+0x89c02) [0x7f130fe89c02] 2024-07-26T11:32:26.177 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f130ff0ec40] 2024-07-26T11:32:26.177 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 2024-07-26T11:32:25.757+0000 7f1309c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:32:26.177 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: in thread 7f1309c00640 thread_name:ms_dispatch 2024-07-26T11:32:26.177 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 2024-07-26T11:32:26.177 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:32:26.178 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f130fe3e6f0] 2024-07-26T11:32:26.178 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f130fe8b94c] 2024-07-26T11:32:26.178 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 3: raise() 2024-07-26T11:32:26.178 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 4: abort() 2024-07-26T11:32:26.178 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 5: /lib64/libc.so.6(+0x2871b) [0x7f130fe2871b] 2024-07-26T11:32:26.178 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 6: /lib64/libc.so.6(+0x37386) [0x7f130fe37386] 2024-07-26T11:32:26.178 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558ecfe2e6b9] 2024-07-26T11:32:26.178 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558ecffec80c] 2024-07-26T11:32:26.178 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558ecff2c2ca] 2024-07-26T11:32:26.178 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558ecfefc753] 2024-07-26T11:32:26.178 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558ecfefda1b] 2024-07-26T11:32:26.178 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558ecfee1752] 2024-07-26T11:32:26.179 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558ecfe2fa56] 2024-07-26T11:32:26.179 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558ecfe33637] 2024-07-26T11:32:26.179 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558ecfe33f5e] 2024-07-26T11:32:26.179 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558ecfe1703a] 2024-07-26T11:32:26.179 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f131099b407] 2024-07-26T11:32:26.179 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 18: (DispatchQueue::entry()+0x60f) [0x7f131099bcf9] 2024-07-26T11:32:26.179 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1310a785a1] 2024-07-26T11:32:26.179 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 20: (Thread::entry_wrapper()+0x43) [0x7f1310825cdd] 2024-07-26T11:32:26.179 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1310825cf9] 2024-07-26T11:32:26.180 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 22: /lib64/libc.so.6(+0x89c02) [0x7f130fe89c02] 2024-07-26T11:32:26.180 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f130ff0ec40] 2024-07-26T11:32:26.180 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:32:26.180 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 2024-07-26T11:32:26.180 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 0> 2024-07-26T11:32:25.757+0000 7f1309c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:32:26.180 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: in thread 7f1309c00640 thread_name:ms_dispatch 2024-07-26T11:32:26.180 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 2024-07-26T11:32:26.180 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:32:26.180 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f130fe3e6f0] 2024-07-26T11:32:26.180 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f130fe8b94c] 2024-07-26T11:32:26.180 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 3: raise() 2024-07-26T11:32:26.181 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 4: abort() 2024-07-26T11:32:26.181 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 5: /lib64/libc.so.6(+0x2871b) [0x7f130fe2871b] 2024-07-26T11:32:26.181 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 6: /lib64/libc.so.6(+0x37386) [0x7f130fe37386] 2024-07-26T11:32:26.181 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558ecfe2e6b9] 2024-07-26T11:32:26.181 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558ecffec80c] 2024-07-26T11:32:26.181 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558ecff2c2ca] 2024-07-26T11:32:26.181 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558ecfefc753] 2024-07-26T11:32:26.181 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558ecfefda1b] 2024-07-26T11:32:26.181 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558ecfee1752] 2024-07-26T11:32:26.182 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558ecfe2fa56] 2024-07-26T11:32:26.182 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558ecfe33637] 2024-07-26T11:32:26.182 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558ecfe33f5e] 2024-07-26T11:32:26.182 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558ecfe1703a] 2024-07-26T11:32:26.182 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f131099b407] 2024-07-26T11:32:26.182 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 18: (DispatchQueue::entry()+0x60f) [0x7f131099bcf9] 2024-07-26T11:32:26.182 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1310a785a1] 2024-07-26T11:32:26.182 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 20: (Thread::entry_wrapper()+0x43) [0x7f1310825cdd] 2024-07-26T11:32:26.186 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1310825cf9] 2024-07-26T11:32:26.186 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 22: /lib64/libc.so.6(+0x89c02) [0x7f130fe89c02] 2024-07-26T11:32:26.186 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f130ff0ec40] 2024-07-26T11:32:26.186 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:32:26.186 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 2024-07-26T11:32:26.186 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 0> 2024-07-26T11:32:25.757+0000 7f1309c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:32:26.186 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: in thread 7f1309c00640 thread_name:ms_dispatch 2024-07-26T11:32:26.187 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 2024-07-26T11:32:26.187 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:32:26.187 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f130fe3e6f0] 2024-07-26T11:32:26.187 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f130fe8b94c] 2024-07-26T11:32:26.187 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 3: raise() 2024-07-26T11:32:26.187 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 4: abort() 2024-07-26T11:32:26.187 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 5: /lib64/libc.so.6(+0x2871b) [0x7f130fe2871b] 2024-07-26T11:32:26.187 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 6: /lib64/libc.so.6(+0x37386) [0x7f130fe37386] 2024-07-26T11:32:26.188 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558ecfe2e6b9] 2024-07-26T11:32:26.188 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558ecffec80c] 2024-07-26T11:32:26.188 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558ecff2c2ca] 2024-07-26T11:32:26.188 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558ecfefc753] 2024-07-26T11:32:26.188 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558ecfefda1b] 2024-07-26T11:32:26.188 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558ecfee1752] 2024-07-26T11:32:26.188 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558ecfe2fa56] 2024-07-26T11:32:26.188 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558ecfe33637] 2024-07-26T11:32:26.188 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558ecfe33f5e] 2024-07-26T11:32:26.188 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558ecfe1703a] 2024-07-26T11:32:26.188 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f131099b407] 2024-07-26T11:32:26.188 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 18: (DispatchQueue::entry()+0x60f) [0x7f131099bcf9] 2024-07-26T11:32:26.189 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f1310a785a1] 2024-07-26T11:32:26.189 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 20: (Thread::entry_wrapper()+0x43) [0x7f1310825cdd] 2024-07-26T11:32:26.189 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 21: (Thread::_entry_func(void*)+0xd) [0x7f1310825cf9] 2024-07-26T11:32:26.189 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 22: /lib64/libc.so.6(+0x89c02) [0x7f130fe89c02] 2024-07-26T11:32:26.189 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f130ff0ec40] 2024-07-26T11:32:26.189 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:32:26.189 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113382]: 2024-07-26T11:32:26.189 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:25 smithi134 podman[113565]: 2024-07-26 11:32:25.858134869 +0000 UTC m=+0.031435861 container died 376c9e86ea0473af69587a13844483b5fe5ac63cb19fbdb7e1959d05ea7d8dc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724) 2024-07-26T11:32:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:25 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/1942695211,v1:172.21.15.134:6839/1942695211] up:rejoin 2024-07-26T11:32:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:25 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:32:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:25 smithi189 ceph-mon[36790]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:32:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:26 smithi193 ceph-mon[37473]: pgmap v1006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T11:32:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:26 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:32:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:26 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/1942695211,v1:172.21.15.134:6839/1942695211] up:active 2024-07-26T11:32:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:26 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:32:27.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:26 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:32:27.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 304.464837 secs 2024-07-26T11:32:27.153 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:26 smithi134 ceph-mon[31966]: pgmap v1006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T11:32:27.153 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:26 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:32:27.153 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:26 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/1942695211,v1:172.21.15.134:6839/1942695211] up:active 2024-07-26T11:32:27.153 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:26 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:32:27.153 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:26 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:32:27.153 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 304.464837 secs 2024-07-26T11:32:27.154 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:26 smithi134 podman[113565]: 2024-07-26 11:32:26.765605539 +0000 UTC m=+0.938906559 container remove 376c9e86ea0473af69587a13844483b5fe5ac63cb19fbdb7e1959d05ea7d8dc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-07-26T11:32:27.154 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:26 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:32:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:26 smithi189 ceph-mon[36790]: pgmap v1006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T11:32:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:26 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:32:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:26 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/1942695211,v1:172.21.15.134:6839/1942695211] up:active 2024-07-26T11:32:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:26 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:32:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:26 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:32:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 304.464837 secs 2024-07-26T11:32:27.422 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:27 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-07-26T11:32:27.422 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:27 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Consumed 1.577s CPU time. 2024-07-26T11:32:28.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:32:28.508+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:32:28.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:32:28.508+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:28.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:32:28.508+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:32:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:28 smithi193 ceph-mon[37473]: pgmap v1007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:32:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 290.039523 secs 2024-07-26T11:32:29.075 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:28 smithi134 ceph-mon[31966]: pgmap v1007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:32:29.075 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 290.039523 secs 2024-07-26T11:32:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:28 smithi189 ceph-mon[36790]: pgmap v1007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:32:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 290.039523 secs 2024-07-26T11:32:29.524 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:32:29 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:32:29.068+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:32:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:32:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:32:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:32:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:30 smithi134 ceph-mon[31966]: pgmap v1008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:32:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:32:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:32:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:30 smithi189 ceph-mon[36790]: pgmap v1008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:32:31.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:32:31.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:32:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:30 smithi193 ceph-mon[37473]: pgmap v1008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:32:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:32:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:32:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 309.464918 secs 2024-07-26T11:32:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:32:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:32:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:32:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 309.464918 secs 2024-07-26T11:32:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:32:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:32:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:32:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 309.464918 secs 2024-07-26T11:32:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:32:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:32:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:32:32.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:32:32.985+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:32:32.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:32:32.985+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:32.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:32:32.985+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:32:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:32 smithi134 ceph-mon[31966]: pgmap v1009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:32:33.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:32 smithi189 ceph-mon[36790]: pgmap v1009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:32:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:32 smithi193 ceph-mon[37473]: pgmap v1009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:32:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 295.039635 secs 2024-07-26T11:32:34.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 295.039635 secs 2024-07-26T11:32:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 295.039635 secs 2024-07-26T11:32:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:34 smithi134 ceph-mon[31966]: pgmap v1010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 173 B/s wr, 12 op/s 2024-07-26T11:32:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:34 smithi189 ceph-mon[36790]: pgmap v1010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 173 B/s wr, 12 op/s 2024-07-26T11:32:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:34 smithi193 ceph-mon[37473]: pgmap v1010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 173 B/s wr, 12 op/s 2024-07-26T11:32:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:36 smithi134 ceph-mon[31966]: pgmap v1011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 314.465039 secs 2024-07-26T11:32:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:36 smithi189 ceph-mon[36790]: pgmap v1011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 314.465039 secs 2024-07-26T11:32:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:36 smithi193 ceph-mon[37473]: pgmap v1011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 314.465039 secs 2024-07-26T11:32:37.817 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:37 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 5. 2024-07-26T11:32:37.817 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:37 smithi134 systemd[1]: Stopped Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:32:37.817 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:37 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Consumed 1.577s CPU time. 2024-07-26T11:32:37.817 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:37 smithi134 systemd[1]: Starting Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:32:38.121 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:37 smithi134 podman[113873]: 2024-07-26 11:32:37.720944562 +0000 UTC m=+0.033432910 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:32:38.122 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:37 smithi134 podman[113873]: 2024-07-26 11:32:37.863463064 +0000 UTC m=+0.175951430 container create 048fb4767e703fded09ec868c641fc2a34ff7569307405ee819be7d620995eb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2) 2024-07-26T11:32:38.423 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:38 smithi134 podman[113873]: 2024-07-26 11:32:38.120240498 +0000 UTC m=+0.432728917 container init 048fb4767e703fded09ec868c641fc2a34ff7569307405ee819be7d620995eb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:32:38.423 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:38 smithi134 podman[113873]: 2024-07-26 11:32:38.130899048 +0000 UTC m=+0.443387382 container start 048fb4767e703fded09ec868c641fc2a34ff7569307405ee819be7d620995eb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240724, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux ) 2024-07-26T11:32:38.423 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:38 smithi134 ceph-mds[113892]: -- 172.21.15.134:0/367954195 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561dc20b3860 con 0x561dc6255180 2024-07-26T11:32:38.423 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:38 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: starting mds.g at 2024-07-26T11:32:38.423 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:38 smithi134 bash[113873]: 048fb4767e703fded09ec868c641fc2a34ff7569307405ee819be7d620995eb3 2024-07-26T11:32:38.423 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:38 smithi134 systemd[1]: Started Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:32:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:38 smithi189 ceph-mon[36790]: pgmap v1012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 300.039731 secs 2024-07-26T11:32:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:38 smithi189 ceph-mon[36790]: daemon mds.g restarted 2024-07-26T11:32:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:38 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:32:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:38 smithi189 ceph-mon[36790]: osdmap e124: 12 total, 12 up, 12 in 2024-07-26T11:32:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:38 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/1046929576,v1:172.21.15.134:6839/1046929576] up:boot 2024-07-26T11:32:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:38 smithi189 ceph-mon[36790]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:32:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:38 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:32:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:32:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:38 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:32:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:38 smithi193 ceph-mon[37473]: pgmap v1012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 300.039731 secs 2024-07-26T11:32:39.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:38 smithi193 ceph-mon[37473]: daemon mds.g restarted 2024-07-26T11:32:39.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:38 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:32:39.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:38 smithi193 ceph-mon[37473]: osdmap e124: 12 total, 12 up, 12 in 2024-07-26T11:32:39.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:38 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/1046929576,v1:172.21.15.134:6839/1046929576] up:boot 2024-07-26T11:32:39.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:38 smithi193 ceph-mon[37473]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:32:39.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:38 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:32:39.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:32:39.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:38 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:32:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:38 smithi134 ceph-mon[31966]: pgmap v1012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 300.039731 secs 2024-07-26T11:32:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:38 smithi134 ceph-mon[31966]: daemon mds.g restarted 2024-07-26T11:32:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:38 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:32:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:38 smithi134 ceph-mon[31966]: osdmap e124: 12 total, 12 up, 12 in 2024-07-26T11:32:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:38 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/1046929576,v1:172.21.15.134:6839/1046929576] up:boot 2024-07-26T11:32:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:38 smithi134 ceph-mon[31966]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:32:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:38 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:32:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-07-26T11:32:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:38 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:32:40.535 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:40 smithi134 ceph-mon[31966]: pgmap v1014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:32:40.535 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:40 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/4263092556,v1:172.21.15.134:6837/4263092556] up:resolve 2024-07-26T11:32:40.536 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:40 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:32:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:40 smithi189 ceph-mon[36790]: pgmap v1014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:32:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:40 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/4263092556,v1:172.21.15.134:6837/4263092556] up:resolve 2024-07-26T11:32:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:40 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:32:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:40 smithi193 ceph-mon[37473]: pgmap v1014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:32:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:40 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/4263092556,v1:172.21.15.134:6837/4263092556] up:resolve 2024-07-26T11:32:40.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:40 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:32:41.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:41 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299993 2024-07-26T11:32:41.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:41 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/4263092556,v1:172.21.15.134:6837/4263092556] up:reconnect 2024-07-26T11:32:41.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:41 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:32:41.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:41 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0119997 2024-07-26T11:32:41.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 319.465157 secs 2024-07-26T11:32:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:41 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299993 2024-07-26T11:32:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:41 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/4263092556,v1:172.21.15.134:6837/4263092556] up:reconnect 2024-07-26T11:32:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:41 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:32:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:41 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0119997 2024-07-26T11:32:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 319.465157 secs 2024-07-26T11:32:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:41 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299993 2024-07-26T11:32:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:41 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/4263092556,v1:172.21.15.134:6837/4263092556] up:reconnect 2024-07-26T11:32:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:41 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:32:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:41 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0119997 2024-07-26T11:32:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 319.465157 secs 2024-07-26T11:32:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-mon[31966]: pgmap v1015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:32:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/4263092556,v1:172.21.15.134:6837/4263092556] up:rejoin 2024-07-26T11:32:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:32:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-mon[31966]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:32:42.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:32:42.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: *** Caught signal (Aborted) ** 2024-07-26T11:32:42.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: in thread 7f9e08e00640 thread_name:ms_dispatch 2024-07-26T11:32:42.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:32:42.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9e0f03e6f0] 2024-07-26T11:32:42.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9e0f08b94c] 2024-07-26T11:32:42.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 3: raise() 2024-07-26T11:32:42.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 4: abort() 2024-07-26T11:32:42.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9e0f02871b] 2024-07-26T11:32:42.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 6: /lib64/libc.so.6(+0x37386) [0x7f9e0f037386] 2024-07-26T11:32:42.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55fff82576b9] 2024-07-26T11:32:42.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55fff841580c] 2024-07-26T11:32:42.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55fff83552ca] 2024-07-26T11:32:42.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55fff8325753] 2024-07-26T11:32:42.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55fff8326a1b] 2024-07-26T11:32:42.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55fff830a752] 2024-07-26T11:32:42.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55fff8258a56] 2024-07-26T11:32:42.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55fff825c637] 2024-07-26T11:32:42.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55fff825cf5e] 2024-07-26T11:32:42.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55fff824003a] 2024-07-26T11:32:42.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9e0fb9b407] 2024-07-26T11:32:42.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 18: (DispatchQueue::entry()+0x60f) [0x7f9e0fb9bcf9] 2024-07-26T11:32:42.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9e0fc785a1] 2024-07-26T11:32:42.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 20: (Thread::entry_wrapper()+0x43) [0x7f9e0fa25cdd] 2024-07-26T11:32:42.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9e0fa25cf9] 2024-07-26T11:32:42.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9e0f089c02] 2024-07-26T11:32:42.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9e0f10ec40] 2024-07-26T11:32:42.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 2024-07-26T11:32:42.309+0000 7f9e08e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:32:42.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: in thread 7f9e08e00640 thread_name:ms_dispatch 2024-07-26T11:32:42.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 2024-07-26T11:32:42.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:32:42.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9e0f03e6f0] 2024-07-26T11:32:42.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9e0f08b94c] 2024-07-26T11:32:42.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 3: raise() 2024-07-26T11:32:42.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 4: abort() 2024-07-26T11:32:42.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9e0f02871b] 2024-07-26T11:32:42.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 6: /lib64/libc.so.6(+0x37386) [0x7f9e0f037386] 2024-07-26T11:32:42.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55fff82576b9] 2024-07-26T11:32:42.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55fff841580c] 2024-07-26T11:32:42.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55fff83552ca] 2024-07-26T11:32:42.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55fff8325753] 2024-07-26T11:32:42.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55fff8326a1b] 2024-07-26T11:32:42.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55fff830a752] 2024-07-26T11:32:42.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55fff8258a56] 2024-07-26T11:32:42.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55fff825c637] 2024-07-26T11:32:42.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55fff825cf5e] 2024-07-26T11:32:42.678 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55fff824003a] 2024-07-26T11:32:42.678 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9e0fb9b407] 2024-07-26T11:32:42.678 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 18: (DispatchQueue::entry()+0x60f) [0x7f9e0fb9bcf9] 2024-07-26T11:32:42.678 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9e0fc785a1] 2024-07-26T11:32:42.678 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 20: (Thread::entry_wrapper()+0x43) [0x7f9e0fa25cdd] 2024-07-26T11:32:42.678 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9e0fa25cf9] 2024-07-26T11:32:42.678 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9e0f089c02] 2024-07-26T11:32:42.678 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9e0f10ec40] 2024-07-26T11:32:42.678 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:32:42.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 2024-07-26T11:32:42.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 0> 2024-07-26T11:32:42.309+0000 7f9e08e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:32:42.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: in thread 7f9e08e00640 thread_name:ms_dispatch 2024-07-26T11:32:42.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 2024-07-26T11:32:42.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:32:42.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9e0f03e6f0] 2024-07-26T11:32:42.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9e0f08b94c] 2024-07-26T11:32:42.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 3: raise() 2024-07-26T11:32:42.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 4: abort() 2024-07-26T11:32:42.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9e0f02871b] 2024-07-26T11:32:42.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 6: /lib64/libc.so.6(+0x37386) [0x7f9e0f037386] 2024-07-26T11:32:42.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55fff82576b9] 2024-07-26T11:32:42.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55fff841580c] 2024-07-26T11:32:42.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55fff83552ca] 2024-07-26T11:32:42.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55fff8325753] 2024-07-26T11:32:42.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55fff8326a1b] 2024-07-26T11:32:42.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55fff830a752] 2024-07-26T11:32:42.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55fff8258a56] 2024-07-26T11:32:42.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55fff825c637] 2024-07-26T11:32:42.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55fff825cf5e] 2024-07-26T11:32:42.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55fff824003a] 2024-07-26T11:32:42.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9e0fb9b407] 2024-07-26T11:32:42.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 18: (DispatchQueue::entry()+0x60f) [0x7f9e0fb9bcf9] 2024-07-26T11:32:42.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9e0fc785a1] 2024-07-26T11:32:42.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 20: (Thread::entry_wrapper()+0x43) [0x7f9e0fa25cdd] 2024-07-26T11:32:42.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9e0fa25cf9] 2024-07-26T11:32:42.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9e0f089c02] 2024-07-26T11:32:42.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9e0f10ec40] 2024-07-26T11:32:42.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:32:42.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 2024-07-26T11:32:42.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 0> 2024-07-26T11:32:42.309+0000 7f9e08e00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:32:42.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: in thread 7f9e08e00640 thread_name:ms_dispatch 2024-07-26T11:32:42.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 2024-07-26T11:32:42.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:32:42.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f9e0f03e6f0] 2024-07-26T11:32:42.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f9e0f08b94c] 2024-07-26T11:32:42.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 3: raise() 2024-07-26T11:32:42.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 4: abort() 2024-07-26T11:32:42.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 5: /lib64/libc.so.6(+0x2871b) [0x7f9e0f02871b] 2024-07-26T11:32:42.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 6: /lib64/libc.so.6(+0x37386) [0x7f9e0f037386] 2024-07-26T11:32:42.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x55fff82576b9] 2024-07-26T11:32:42.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x55fff841580c] 2024-07-26T11:32:42.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x55fff83552ca] 2024-07-26T11:32:42.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x55fff8325753] 2024-07-26T11:32:42.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x55fff8326a1b] 2024-07-26T11:32:42.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x55fff830a752] 2024-07-26T11:32:42.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x55fff8258a56] 2024-07-26T11:32:42.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x55fff825c637] 2024-07-26T11:32:42.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x55fff825cf5e] 2024-07-26T11:32:42.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x55fff824003a] 2024-07-26T11:32:42.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f9e0fb9b407] 2024-07-26T11:32:42.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 18: (DispatchQueue::entry()+0x60f) [0x7f9e0fb9bcf9] 2024-07-26T11:32:42.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f9e0fc785a1] 2024-07-26T11:32:42.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 20: (Thread::entry_wrapper()+0x43) [0x7f9e0fa25cdd] 2024-07-26T11:32:42.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 21: (Thread::_entry_func(void*)+0xd) [0x7f9e0fa25cf9] 2024-07-26T11:32:42.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 22: /lib64/libc.so.6(+0x89c02) [0x7f9e0f089c02] 2024-07-26T11:32:42.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f9e0f10ec40] 2024-07-26T11:32:42.684 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:32:42.684 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[113527]: 2024-07-26T11:32:42.684 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:42 smithi134 podman[113935]: 2024-07-26 11:32:42.413739649 +0000 UTC m=+0.030864348 container died d433ae54cc741068e648dc3b7eef85f7ad25229cad7d914491138c0fbbc5e544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240724, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-07-26T11:32:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:42 smithi189 ceph-mon[36790]: pgmap v1015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:32:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:42 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/4263092556,v1:172.21.15.134:6837/4263092556] up:rejoin 2024-07-26T11:32:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:42 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:32:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:42 smithi189 ceph-mon[36790]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:32:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:42 smithi193 ceph-mon[37473]: pgmap v1015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:32:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:42 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/4263092556,v1:172.21.15.134:6837/4263092556] up:rejoin 2024-07-26T11:32:42.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:42 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:32:42.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:42 smithi193 ceph-mon[37473]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:32:43.525 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:32:43.523+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:32:43.525 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:32:43.523+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:43.525 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:32:43.523+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:32:43.537 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:43 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:32:43.537 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:43 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/4263092556,v1:172.21.15.134:6837/4263092556] up:active 2024-07-26T11:32:43.537 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:43 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:32:43.537 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:43 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:32:43.537 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:43 smithi134 podman[113935]: 2024-07-26 11:32:43.259763681 +0000 UTC m=+0.876888423 container remove d433ae54cc741068e648dc3b7eef85f7ad25229cad7d914491138c0fbbc5e544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-07-26T11:32:43.537 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:43 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:32:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:43 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:32:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:43 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/4263092556,v1:172.21.15.134:6837/4263092556] up:active 2024-07-26T11:32:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:43 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:32:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:43 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:32:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:43 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:32:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:43 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/4263092556,v1:172.21.15.134:6837/4263092556] up:active 2024-07-26T11:32:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:43 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:32:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:43 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:32:44.172 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:43 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T11:32:44.173 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:43 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.627s CPU time. 2024-07-26T11:32:44.326 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:32:44 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:32:44.069+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:32:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:44 smithi134 ceph-mon[31966]: pgmap v1016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.6 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:32:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 305.039847 secs 2024-07-26T11:32:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:44 smithi189 ceph-mon[36790]: pgmap v1016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.6 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:32:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 305.039847 secs 2024-07-26T11:32:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:44 smithi193 ceph-mon[37473]: pgmap v1016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.6 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:32:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 305.039847 secs 2024-07-26T11:32:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:46 smithi134 ceph-mon[31966]: pgmap v1017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:32:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 324.465322 secs 2024-07-26T11:32:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:46 smithi189 ceph-mon[36790]: pgmap v1017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:32:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 324.465322 secs 2024-07-26T11:32:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:46 smithi193 ceph-mon[37473]: pgmap v1017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:32:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 324.465322 secs 2024-07-26T11:32:48.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:32:48.001+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:32:48.005 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:32:48.001+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:32:48.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:32:48.001+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:48 smithi134 ceph-mon[31966]: pgmap v1018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:32:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:48 smithi189 ceph-mon[36790]: pgmap v1018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:32:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:48 smithi193 ceph-mon[37473]: pgmap v1018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:32:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 310.039979 secs 2024-07-26T11:32:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 310.039979 secs 2024-07-26T11:32:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 310.039979 secs 2024-07-26T11:32:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:50 smithi134 ceph-mon[31966]: pgmap v1019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 189 B/s wr, 10 op/s 2024-07-26T11:32:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:50 smithi189 ceph-mon[36790]: pgmap v1019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 189 B/s wr, 10 op/s 2024-07-26T11:32:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:50 smithi193 ceph-mon[37473]: pgmap v1019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 189 B/s wr, 10 op/s 2024-07-26T11:32:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 329.465373 secs 2024-07-26T11:32:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 329.465373 secs 2024-07-26T11:32:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 329.465373 secs 2024-07-26T11:32:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:52 smithi134 ceph-mon[31966]: pgmap v1020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:52 smithi189 ceph-mon[36790]: pgmap v1020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:52 smithi193 ceph-mon[37473]: pgmap v1020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:32:54.324 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:53 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 4. 2024-07-26T11:32:54.325 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:53 smithi134 systemd[1]: Stopped Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:32:54.325 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:53 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.627s CPU time. 2024-07-26T11:32:54.325 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:53 smithi134 systemd[1]: Starting Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:32:54.665 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:54 smithi134 ceph-mon[31966]: pgmap v1021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:32:54.665 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 315.040103 secs 2024-07-26T11:32:54.666 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:54 smithi134 podman[114032]: 2024-07-26 11:32:54.227020821 +0000 UTC m=+0.032791951 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:32:54.666 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:54 smithi134 podman[114032]: 2024-07-26 11:32:54.357889842 +0000 UTC m=+0.163661001 container create 76b9af7200e0c80b31a36a369ba1b0ea91d60bf329e0eda1d2aceb19040cd6cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:32:54.666 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:54 smithi134 podman[114032]: 2024-07-26 11:32:54.664332044 +0000 UTC m=+0.470103210 container init 76b9af7200e0c80b31a36a369ba1b0ea91d60bf329e0eda1d2aceb19040cd6cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-07-26T11:32:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:54 smithi189 ceph-mon[36790]: pgmap v1021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:32:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 315.040103 secs 2024-07-26T11:32:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:54 smithi193 ceph-mon[37473]: pgmap v1021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:32:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 315.040103 secs 2024-07-26T11:32:54.923 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:54 smithi134 podman[114032]: 2024-07-26 11:32:54.673989157 +0000 UTC m=+0.479760319 container start 76b9af7200e0c80b31a36a369ba1b0ea91d60bf329e0eda1d2aceb19040cd6cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:32:54.923 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:54 smithi134 bash[114032]: 76b9af7200e0c80b31a36a369ba1b0ea91d60bf329e0eda1d2aceb19040cd6cb 2024-07-26T11:32:54.923 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:54 smithi134 systemd[1]: Started Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:32:54.923 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:54 smithi134 ceph-mds[114051]: -- 172.21.15.134:0/3050511012 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558d366bf860 con 0x558d3a861180 2024-07-26T11:32:54.923 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:32:54 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: starting mds.d at 2024-07-26T11:32:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:55 smithi134 ceph-mon[31966]: daemon mds.d restarted 2024-07-26T11:32:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:55 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:32:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:55 smithi134 ceph-mon[31966]: osdmap e125: 12 total, 12 up, 12 in 2024-07-26T11:32:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:55 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/2178607141,v1:172.21.15.134:6837/2178607141] up:boot 2024-07-26T11:32:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:55 smithi134 ceph-mon[31966]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-07-26T11:32:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:55 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:32:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:32:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:55 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:32:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:55 smithi189 ceph-mon[36790]: daemon mds.d restarted 2024-07-26T11:32:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:55 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:32:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:55 smithi189 ceph-mon[36790]: osdmap e125: 12 total, 12 up, 12 in 2024-07-26T11:32:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:55 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/2178607141,v1:172.21.15.134:6837/2178607141] up:boot 2024-07-26T11:32:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:55 smithi189 ceph-mon[36790]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-07-26T11:32:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:55 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:32:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:32:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:55 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:32:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:55 smithi193 ceph-mon[37473]: daemon mds.d restarted 2024-07-26T11:32:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:55 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:32:55.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:55 smithi193 ceph-mon[37473]: osdmap e125: 12 total, 12 up, 12 in 2024-07-26T11:32:55.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:55 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/2178607141,v1:172.21.15.134:6837/2178607141] up:boot 2024-07-26T11:32:55.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:55 smithi193 ceph-mon[37473]: Standby daemon mds.g assigned to filesystem cephfs as rank 1 2024-07-26T11:32:55.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:55 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:32:55.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:32:55.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:55 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-07-26T11:32:57.090 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:56 smithi134 ceph-mon[31966]: pgmap v1023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-07-26T11:32:57.090 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 334.465514 secs 2024-07-26T11:32:57.090 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:56 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/1046929576,v1:172.21.15.134:6839/1046929576] up:resolve 2024-07-26T11:32:57.090 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:56 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:32:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:56 smithi189 ceph-mon[36790]: pgmap v1023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-07-26T11:32:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 334.465514 secs 2024-07-26T11:32:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:56 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/1046929576,v1:172.21.15.134:6839/1046929576] up:resolve 2024-07-26T11:32:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:56 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:32:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:56 smithi193 ceph-mon[37473]: pgmap v1023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 10 op/s 2024-07-26T11:32:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 334.465514 secs 2024-07-26T11:32:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:56 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/1046929576,v1:172.21.15.134:6839/1046929576] up:resolve 2024-07-26T11:32:57.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:56 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-07-26T11:32:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:57 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299993 2024-07-26T11:32:58.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:57 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/1046929576,v1:172.21.15.134:6839/1046929576] up:reconnect 2024-07-26T11:32:58.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:57 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:32:58.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:57 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999978 2024-07-26T11:32:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:57 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299993 2024-07-26T11:32:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:57 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/1046929576,v1:172.21.15.134:6839/1046929576] up:reconnect 2024-07-26T11:32:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:57 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:32:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:57 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999978 2024-07-26T11:32:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:57 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299993 2024-07-26T11:32:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:57 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/1046929576,v1:172.21.15.134:6839/1046929576] up:reconnect 2024-07-26T11:32:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:57 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:reconnect 4 up:active 2024-07-26T11:32:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:57 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.00999978 2024-07-26T11:32:58.540 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:32:58.539+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:32:58.540 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:32:58.539+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:32:58.541 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:32:58.539+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:32:59.152 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-mon[31966]: pgmap v1024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:32:59.152 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/1046929576,v1:172.21.15.134:6839/1046929576] up:rejoin 2024-07-26T11:32:59.153 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:32:59.153 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-mon[31966]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:32:59.153 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 320.040246 secs 2024-07-26T11:32:59.153 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:32:59.153 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: *** Caught signal (Aborted) ** 2024-07-26T11:32:59.153 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: in thread 7f4d07600640 thread_name:ms_dispatch 2024-07-26T11:32:59.153 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:32:59.153 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4d0d83e6f0] 2024-07-26T11:32:59.154 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4d0d88b94c] 2024-07-26T11:32:59.154 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 3: raise() 2024-07-26T11:32:59.154 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 4: abort() 2024-07-26T11:32:59.154 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4d0d82871b] 2024-07-26T11:32:59.154 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 6: /lib64/libc.so.6(+0x37386) [0x7f4d0d837386] 2024-07-26T11:32:59.154 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561d88b236b9] 2024-07-26T11:32:59.154 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561d88ce180c] 2024-07-26T11:32:59.154 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x561d88c212ca] 2024-07-26T11:32:59.154 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x561d88bf1753] 2024-07-26T11:32:59.154 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561d88bf2a1b] 2024-07-26T11:32:59.154 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561d88bd6752] 2024-07-26T11:32:59.154 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561d88b24a56] 2024-07-26T11:32:59.155 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561d88b28637] 2024-07-26T11:32:59.155 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561d88b28f5e] 2024-07-26T11:32:59.155 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561d88b0c03a] 2024-07-26T11:32:59.155 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4d0e39b407] 2024-07-26T11:32:59.155 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 18: (DispatchQueue::entry()+0x60f) [0x7f4d0e39bcf9] 2024-07-26T11:32:59.155 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4d0e4785a1] 2024-07-26T11:32:59.155 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 20: (Thread::entry_wrapper()+0x43) [0x7f4d0e225cdd] 2024-07-26T11:32:59.155 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4d0e225cf9] 2024-07-26T11:32:59.155 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4d0d889c02] 2024-07-26T11:32:59.155 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4d0d90ec40] 2024-07-26T11:32:59.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 2024-07-26T11:32:58.871+0000 7f4d07600640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:32:59.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: in thread 7f4d07600640 thread_name:ms_dispatch 2024-07-26T11:32:59.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 2024-07-26T11:32:59.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:32:59.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4d0d83e6f0] 2024-07-26T11:32:59.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4d0d88b94c] 2024-07-26T11:32:59.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 3: raise() 2024-07-26T11:32:59.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 4: abort() 2024-07-26T11:32:59.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4d0d82871b] 2024-07-26T11:32:59.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 6: /lib64/libc.so.6(+0x37386) [0x7f4d0d837386] 2024-07-26T11:32:59.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561d88b236b9] 2024-07-26T11:32:59.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561d88ce180c] 2024-07-26T11:32:59.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x561d88c212ca] 2024-07-26T11:32:59.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x561d88bf1753] 2024-07-26T11:32:59.156 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561d88bf2a1b] 2024-07-26T11:32:59.157 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561d88bd6752] 2024-07-26T11:32:59.157 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561d88b24a56] 2024-07-26T11:32:59.157 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561d88b28637] 2024-07-26T11:32:59.157 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561d88b28f5e] 2024-07-26T11:32:59.157 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561d88b0c03a] 2024-07-26T11:32:59.157 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4d0e39b407] 2024-07-26T11:32:59.157 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 18: (DispatchQueue::entry()+0x60f) [0x7f4d0e39bcf9] 2024-07-26T11:32:59.157 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4d0e4785a1] 2024-07-26T11:32:59.158 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 20: (Thread::entry_wrapper()+0x43) [0x7f4d0e225cdd] 2024-07-26T11:32:59.158 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4d0e225cf9] 2024-07-26T11:32:59.158 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4d0d889c02] 2024-07-26T11:32:59.158 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4d0d90ec40] 2024-07-26T11:32:59.158 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:32:59.158 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 2024-07-26T11:32:59.158 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 0> 2024-07-26T11:32:58.871+0000 7f4d07600640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:32:59.158 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: in thread 7f4d07600640 thread_name:ms_dispatch 2024-07-26T11:32:59.158 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 2024-07-26T11:32:59.158 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:32:59.158 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4d0d83e6f0] 2024-07-26T11:32:59.159 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4d0d88b94c] 2024-07-26T11:32:59.159 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 3: raise() 2024-07-26T11:32:59.159 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 4: abort() 2024-07-26T11:32:59.159 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4d0d82871b] 2024-07-26T11:32:59.159 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 6: /lib64/libc.so.6(+0x37386) [0x7f4d0d837386] 2024-07-26T11:32:59.159 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561d88b236b9] 2024-07-26T11:32:59.159 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561d88ce180c] 2024-07-26T11:32:59.159 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x561d88c212ca] 2024-07-26T11:32:59.159 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x561d88bf1753] 2024-07-26T11:32:59.159 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561d88bf2a1b] 2024-07-26T11:32:59.159 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561d88bd6752] 2024-07-26T11:32:59.159 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561d88b24a56] 2024-07-26T11:32:59.160 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561d88b28637] 2024-07-26T11:32:59.160 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561d88b28f5e] 2024-07-26T11:32:59.160 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561d88b0c03a] 2024-07-26T11:32:59.160 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4d0e39b407] 2024-07-26T11:32:59.160 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 18: (DispatchQueue::entry()+0x60f) [0x7f4d0e39bcf9] 2024-07-26T11:32:59.160 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4d0e4785a1] 2024-07-26T11:32:59.160 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 20: (Thread::entry_wrapper()+0x43) [0x7f4d0e225cdd] 2024-07-26T11:32:59.160 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4d0e225cf9] 2024-07-26T11:32:59.160 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4d0d889c02] 2024-07-26T11:32:59.160 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4d0d90ec40] 2024-07-26T11:32:59.160 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:32:59.160 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 2024-07-26T11:32:59.160 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 0> 2024-07-26T11:32:58.871+0000 7f4d07600640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:32:59.160 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: in thread 7f4d07600640 thread_name:ms_dispatch 2024-07-26T11:32:59.160 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 2024-07-26T11:32:59.161 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:32:59.161 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f4d0d83e6f0] 2024-07-26T11:32:59.161 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f4d0d88b94c] 2024-07-26T11:32:59.161 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 3: raise() 2024-07-26T11:32:59.161 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 4: abort() 2024-07-26T11:32:59.161 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 5: /lib64/libc.so.6(+0x2871b) [0x7f4d0d82871b] 2024-07-26T11:32:59.161 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 6: /lib64/libc.so.6(+0x37386) [0x7f4d0d837386] 2024-07-26T11:32:59.161 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x561d88b236b9] 2024-07-26T11:32:59.161 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x561d88ce180c] 2024-07-26T11:32:59.161 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x561d88c212ca] 2024-07-26T11:32:59.161 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x561d88bf1753] 2024-07-26T11:32:59.161 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x561d88bf2a1b] 2024-07-26T11:32:59.161 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x561d88bd6752] 2024-07-26T11:32:59.161 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x561d88b24a56] 2024-07-26T11:32:59.161 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x561d88b28637] 2024-07-26T11:32:59.162 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x561d88b28f5e] 2024-07-26T11:32:59.162 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x561d88b0c03a] 2024-07-26T11:32:59.162 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f4d0e39b407] 2024-07-26T11:32:59.162 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 18: (DispatchQueue::entry()+0x60f) [0x7f4d0e39bcf9] 2024-07-26T11:32:59.162 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f4d0e4785a1] 2024-07-26T11:32:59.162 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 20: (Thread::entry_wrapper()+0x43) [0x7f4d0e225cdd] 2024-07-26T11:32:59.162 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 21: (Thread::_entry_func(void*)+0xd) [0x7f4d0e225cf9] 2024-07-26T11:32:59.162 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 22: /lib64/libc.so.6(+0x89c02) [0x7f4d0d889c02] 2024-07-26T11:32:59.162 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f4d0d90ec40] 2024-07-26T11:32:59.163 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:32:59.163 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g[113888]: 2024-07-26T11:32:59.163 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:32:58 smithi134 podman[114085]: 2024-07-26 11:32:58.970520001 +0000 UTC m=+0.031200922 container died 048fb4767e703fded09ec868c641fc2a34ff7569307405ee819be7d620995eb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240724, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb) 2024-07-26T11:32:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:58 smithi189 ceph-mon[36790]: pgmap v1024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:32:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:58 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/1046929576,v1:172.21.15.134:6839/1046929576] up:rejoin 2024-07-26T11:32:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:58 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:32:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:58 smithi189 ceph-mon[36790]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:32:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 320.040246 secs 2024-07-26T11:32:59.274 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:32:59 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:32:59.070+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:32:59.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:58 smithi193 ceph-mon[37473]: pgmap v1024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:32:59.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:58 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/1046929576,v1:172.21.15.134:6839/1046929576] up:rejoin 2024-07-26T11:32:59.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:58 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:rejoin 4 up:active 2024-07-26T11:32:59.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:58 smithi193 ceph-mon[37473]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-07-26T11:32:59.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 320.040246 secs 2024-07-26T11:33:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:59 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:33:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:59 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6838/1046929576,v1:172.21.15.134:6839/1046929576] up:active 2024-07-26T11:33:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:59 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:33:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:32:59 smithi134 ceph-mon[31966]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:33:00.173 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:33:00 smithi134 podman[114085]: 2024-07-26 11:33:00.018210389 +0000 UTC m=+1.078891338 container remove 048fb4767e703fded09ec868c641fc2a34ff7569307405ee819be7d620995eb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-g, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:33:00.173 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:33:00 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:33:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:59 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:33:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:59 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6838/1046929576,v1:172.21.15.134:6839/1046929576] up:active 2024-07-26T11:33:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:59 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:33:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:32:59 smithi189 ceph-mon[36790]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:33:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:59 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:33:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:59 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6838/1046929576,v1:172.21.15.134:6839/1046929576] up:active 2024-07-26T11:33:00.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:59 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 1 up:standby 5 up:active 2024-07-26T11:33:00.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:32:59 smithi193 ceph-mon[37473]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-07-26T11:33:00.874 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:33:00 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-07-26T11:33:00.874 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:33:00 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Consumed 1.558s CPU time. 2024-07-26T11:33:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:00 smithi134 ceph-mon[31966]: pgmap v1025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:33:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:00 smithi189 ceph-mon[36790]: pgmap v1025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:33:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:00 smithi193 ceph-mon[37473]: pgmap v1025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:33:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 339.465600 secs 2024-07-26T11:33:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 339.465600 secs 2024-07-26T11:33:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 339.465600 secs 2024-07-26T11:33:03.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:33:03.017+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:33:03.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:33:03.017+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:03.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:33:03.017+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:33:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:02 smithi134 ceph-mon[31966]: pgmap v1026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:33:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:02 smithi189 ceph-mon[36790]: pgmap v1026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:33:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:02 smithi193 ceph-mon[37473]: pgmap v1026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:33:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 325.040358 secs 2024-07-26T11:33:04.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 325.040358 secs 2024-07-26T11:33:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 325.040358 secs 2024-07-26T11:33:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:04 smithi134 ceph-mon[31966]: pgmap v1027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:33:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:04 smithi189 ceph-mon[36790]: pgmap v1027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:33:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:04 smithi193 ceph-mon[37473]: pgmap v1027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 204 B/s wr, 11 op/s 2024-07-26T11:33:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:06 smithi134 ceph-mon[31966]: pgmap v1028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 199 B/s wr, 11 op/s 2024-07-26T11:33:07.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 344.465717 secs 2024-07-26T11:33:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:06 smithi189 ceph-mon[36790]: pgmap v1028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 199 B/s wr, 11 op/s 2024-07-26T11:33:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 344.465717 secs 2024-07-26T11:33:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:06 smithi193 ceph-mon[37473]: pgmap v1028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 199 B/s wr, 11 op/s 2024-07-26T11:33:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 344.465717 secs 2024-07-26T11:33:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:08 smithi189 ceph-mon[36790]: pgmap v1029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:33:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 330.040488 secs 2024-07-26T11:33:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:08 smithi193 ceph-mon[37473]: pgmap v1029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:33:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 330.040488 secs 2024-07-26T11:33:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:08 smithi134 ceph-mon[31966]: pgmap v1029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:33:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 330.040488 secs 2024-07-26T11:33:10.932 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:33:10 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 6. 2024-07-26T11:33:10.932 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:33:10 smithi134 systemd[1]: Stopped Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:33:10.932 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:33:10 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Consumed 1.558s CPU time. 2024-07-26T11:33:10.933 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:33:10 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Start request repeated too quickly. 2024-07-26T11:33:10.933 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:33:10 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-07-26T11:33:10.933 INFO:journalctl@ceph.mds.g.smithi134.stdout:Jul 26 11:33:10 smithi134 systemd[1]: Failed to start Ceph mds.g for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:33:11.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:10 smithi189 ceph-mon[36790]: pgmap v1030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:33:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:10 smithi193 ceph-mon[37473]: pgmap v1030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:33:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:10 smithi134 ceph-mon[31966]: pgmap v1030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 170 B/s wr, 11 op/s 2024-07-26T11:33:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 349.465853 secs 2024-07-26T11:33:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 349.465853 secs 2024-07-26T11:33:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 349.465853 secs 2024-07-26T11:33:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:12 smithi193 ceph-mon[37473]: pgmap v1031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:33:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:12 smithi134 ceph-mon[31966]: pgmap v1031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:33:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:12 smithi189 ceph-mon[36790]: pgmap v1031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 170 B/s wr, 10 op/s 2024-07-26T11:33:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:33:13.541+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:33:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:33:13.541+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:13.543 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:33:13.541+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:33:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 335.040592 secs 2024-07-26T11:33:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 335.040592 secs 2024-07-26T11:33:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 335.040592 secs 2024-07-26T11:33:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:14 smithi193 ceph-mon[37473]: pgmap v1032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.1 KiB/s rd, 85 B/s wr, 11 op/s 2024-07-26T11:33:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:14 smithi193 ceph-mon[37473]: Replacing daemon mds.g as rank 1 with standby daemon mds.d 2024-07-26T11:33:15.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:14 smithi193 ceph-mon[37473]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-07-26T11:33:15.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:14 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:33:15.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:14 smithi193 ceph-mon[37473]: osdmap e126: 12 total, 12 up, 12 in 2024-07-26T11:33:15.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:14 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T11:33:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:14 smithi134 ceph-mon[31966]: pgmap v1032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.1 KiB/s rd, 85 B/s wr, 11 op/s 2024-07-26T11:33:15.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:14 smithi134 ceph-mon[31966]: Replacing daemon mds.g as rank 1 with standby daemon mds.d 2024-07-26T11:33:15.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:14 smithi134 ceph-mon[31966]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-07-26T11:33:15.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:14 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:33:15.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:14 smithi134 ceph-mon[31966]: osdmap e126: 12 total, 12 up, 12 in 2024-07-26T11:33:15.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:14 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T11:33:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:14 smithi189 ceph-mon[36790]: pgmap v1032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.1 KiB/s rd, 85 B/s wr, 11 op/s 2024-07-26T11:33:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:14 smithi189 ceph-mon[36790]: Replacing daemon mds.g as rank 1 with standby daemon mds.d 2024-07-26T11:33:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:14 smithi189 ceph-mon[36790]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-07-26T11:33:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:14 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:33:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:14 smithi189 ceph-mon[36790]: osdmap e126: 12 total, 12 up, 12 in 2024-07-26T11:33:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:14 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T11:33:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:15 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 354.465978 secs 2024-07-26T11:33:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:15 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 354.465978 secs 2024-07-26T11:33:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:15 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 354.465978 secs 2024-07-26T11:33:17.232 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:16 smithi134 ceph-mon[31966]: pgmap v1034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T11:33:17.232 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:16 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/2178607141,v1:172.21.15.134:6837/2178607141] up:resolve 2024-07-26T11:33:17.232 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:16 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T11:33:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:16 smithi193 ceph-mon[37473]: pgmap v1034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T11:33:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:16 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/2178607141,v1:172.21.15.134:6837/2178607141] up:resolve 2024-07-26T11:33:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:16 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T11:33:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:16 smithi189 ceph-mon[36790]: pgmap v1034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-07-26T11:33:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:16 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/2178607141,v1:172.21.15.134:6837/2178607141] up:resolve 2024-07-26T11:33:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:16 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T11:33:18.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:33:18.017+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:33:18.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:33:18.017+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:18.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:33:18.017+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:33:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:18 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299993 2024-07-26T11:33:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:18 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/2178607141,v1:172.21.15.134:6837/2178607141] up:reconnect 2024-07-26T11:33:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:18 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T11:33:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:18 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:33:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:18 smithi193 ceph-mon[37473]: pgmap v1035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:33:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:18 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299993 2024-07-26T11:33:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:18 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/2178607141,v1:172.21.15.134:6837/2178607141] up:reconnect 2024-07-26T11:33:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:18 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T11:33:18.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:18 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:33:18.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:18 smithi134 ceph-mon[31966]: pgmap v1035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:33:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:18 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299993 2024-07-26T11:33:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:18 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/2178607141,v1:172.21.15.134:6837/2178607141] up:reconnect 2024-07-26T11:33:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:18 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T11:33:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:18 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:33:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:18 smithi189 ceph-mon[36790]: pgmap v1035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-07-26T11:33:19.274 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:33:19 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:33:19.072+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:33:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:19 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/2178607141,v1:172.21.15.134:6837/2178607141] up:rejoin 2024-07-26T11:33:19.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:19 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T11:33:19.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:19 smithi193 ceph-mon[37473]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:33:19.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 340.040716 secs 2024-07-26T11:33:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/2178607141,v1:172.21.15.134:6837/2178607141] up:rejoin 2024-07-26T11:33:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T11:33:19.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-mon[31966]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:33:19.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 340.040716 secs 2024-07-26T11:33:19.424 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:33:19.424 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: *** Caught signal (Aborted) ** 2024-07-26T11:33:19.424 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: in thread 7f6161c00640 thread_name:ms_dispatch 2024-07-26T11:33:19.424 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:33:19.425 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6167e3e6f0] 2024-07-26T11:33:19.425 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6167e8b94c] 2024-07-26T11:33:19.425 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 3: raise() 2024-07-26T11:33:19.425 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 4: abort() 2024-07-26T11:33:19.425 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6167e2871b] 2024-07-26T11:33:19.425 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 6: /lib64/libc.so.6(+0x37386) [0x7f6167e37386] 2024-07-26T11:33:19.425 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558d163ae6b9] 2024-07-26T11:33:19.425 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558d1656c80c] 2024-07-26T11:33:19.425 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558d164ac2ca] 2024-07-26T11:33:19.425 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558d1647c753] 2024-07-26T11:33:19.425 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558d1647da1b] 2024-07-26T11:33:19.425 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558d16461752] 2024-07-26T11:33:19.426 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558d163afa56] 2024-07-26T11:33:19.426 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558d163b3637] 2024-07-26T11:33:19.426 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558d163b3f5e] 2024-07-26T11:33:19.426 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558d1639703a] 2024-07-26T11:33:19.426 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f616899b407] 2024-07-26T11:33:19.426 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 18: (DispatchQueue::entry()+0x60f) [0x7f616899bcf9] 2024-07-26T11:33:19.426 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6168a785a1] 2024-07-26T11:33:19.426 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 20: (Thread::entry_wrapper()+0x43) [0x7f6168825cdd] 2024-07-26T11:33:19.426 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6168825cf9] 2024-07-26T11:33:19.427 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6167e89c02] 2024-07-26T11:33:19.427 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6167f0ec40] 2024-07-26T11:33:19.427 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 2024-07-26T11:33:19.010+0000 7f6161c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:33:19.427 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: in thread 7f6161c00640 thread_name:ms_dispatch 2024-07-26T11:33:19.427 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 2024-07-26T11:33:19.427 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:33:19.427 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6167e3e6f0] 2024-07-26T11:33:19.427 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6167e8b94c] 2024-07-26T11:33:19.427 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 3: raise() 2024-07-26T11:33:19.427 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 4: abort() 2024-07-26T11:33:19.427 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6167e2871b] 2024-07-26T11:33:19.427 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 6: /lib64/libc.so.6(+0x37386) [0x7f6167e37386] 2024-07-26T11:33:19.428 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558d163ae6b9] 2024-07-26T11:33:19.428 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558d1656c80c] 2024-07-26T11:33:19.428 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558d164ac2ca] 2024-07-26T11:33:19.428 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558d1647c753] 2024-07-26T11:33:19.428 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558d1647da1b] 2024-07-26T11:33:19.428 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558d16461752] 2024-07-26T11:33:19.428 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558d163afa56] 2024-07-26T11:33:19.428 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558d163b3637] 2024-07-26T11:33:19.428 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558d163b3f5e] 2024-07-26T11:33:19.428 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558d1639703a] 2024-07-26T11:33:19.428 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f616899b407] 2024-07-26T11:33:19.428 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 18: (DispatchQueue::entry()+0x60f) [0x7f616899bcf9] 2024-07-26T11:33:19.429 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6168a785a1] 2024-07-26T11:33:19.429 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 20: (Thread::entry_wrapper()+0x43) [0x7f6168825cdd] 2024-07-26T11:33:19.429 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6168825cf9] 2024-07-26T11:33:19.429 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6167e89c02] 2024-07-26T11:33:19.429 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6167f0ec40] 2024-07-26T11:33:19.429 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:33:19.429 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 2024-07-26T11:33:19.429 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 0> 2024-07-26T11:33:19.010+0000 7f6161c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:33:19.429 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: in thread 7f6161c00640 thread_name:ms_dispatch 2024-07-26T11:33:19.430 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 2024-07-26T11:33:19.430 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:33:19.430 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6167e3e6f0] 2024-07-26T11:33:19.430 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6167e8b94c] 2024-07-26T11:33:19.430 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 3: raise() 2024-07-26T11:33:19.430 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 4: abort() 2024-07-26T11:33:19.430 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6167e2871b] 2024-07-26T11:33:19.430 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 6: /lib64/libc.so.6(+0x37386) [0x7f6167e37386] 2024-07-26T11:33:19.430 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558d163ae6b9] 2024-07-26T11:33:19.430 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558d1656c80c] 2024-07-26T11:33:19.431 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558d164ac2ca] 2024-07-26T11:33:19.431 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558d1647c753] 2024-07-26T11:33:19.431 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558d1647da1b] 2024-07-26T11:33:19.431 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558d16461752] 2024-07-26T11:33:19.431 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558d163afa56] 2024-07-26T11:33:19.431 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558d163b3637] 2024-07-26T11:33:19.431 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558d163b3f5e] 2024-07-26T11:33:19.431 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558d1639703a] 2024-07-26T11:33:19.432 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f616899b407] 2024-07-26T11:33:19.432 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 18: (DispatchQueue::entry()+0x60f) [0x7f616899bcf9] 2024-07-26T11:33:19.432 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6168a785a1] 2024-07-26T11:33:19.432 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 20: (Thread::entry_wrapper()+0x43) [0x7f6168825cdd] 2024-07-26T11:33:19.432 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6168825cf9] 2024-07-26T11:33:19.432 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6167e89c02] 2024-07-26T11:33:19.432 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6167f0ec40] 2024-07-26T11:33:19.432 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:33:19.432 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 2024-07-26T11:33:19.432 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: -2> 2024-07-26T11:33:19.010+0000 7f6161c00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:33:19.432 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: in thread 7f6161c00640 thread_name:ms_dispatch 2024-07-26T11:33:19.433 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 2024-07-26T11:33:19.433 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:33:19.433 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f6167e3e6f0] 2024-07-26T11:33:19.433 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f6167e8b94c] 2024-07-26T11:33:19.433 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 3: raise() 2024-07-26T11:33:19.433 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 4: abort() 2024-07-26T11:33:19.433 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 5: /lib64/libc.so.6(+0x2871b) [0x7f6167e2871b] 2024-07-26T11:33:19.433 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 6: /lib64/libc.so.6(+0x37386) [0x7f6167e37386] 2024-07-26T11:33:19.433 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x558d163ae6b9] 2024-07-26T11:33:19.433 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x558d1656c80c] 2024-07-26T11:33:19.433 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x558d164ac2ca] 2024-07-26T11:33:19.433 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x558d1647c753] 2024-07-26T11:33:19.433 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x558d1647da1b] 2024-07-26T11:33:19.434 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x558d16461752] 2024-07-26T11:33:19.434 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x558d163afa56] 2024-07-26T11:33:19.434 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x558d163b3637] 2024-07-26T11:33:19.434 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x558d163b3f5e] 2024-07-26T11:33:19.434 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x558d1639703a] 2024-07-26T11:33:19.434 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7f616899b407] 2024-07-26T11:33:19.434 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 18: (DispatchQueue::entry()+0x60f) [0x7f616899bcf9] 2024-07-26T11:33:19.434 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7f6168a785a1] 2024-07-26T11:33:19.434 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 20: (Thread::entry_wrapper()+0x43) [0x7f6168825cdd] 2024-07-26T11:33:19.434 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 21: (Thread::_entry_func(void*)+0xd) [0x7f6168825cf9] 2024-07-26T11:33:19.434 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 22: /lib64/libc.so.6(+0x89c02) [0x7f6167e89c02] 2024-07-26T11:33:19.434 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 23: /lib64/libc.so.6(+0x10ec40) [0x7f6167f0ec40] 2024-07-26T11:33:19.434 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:33:19.435 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114047]: 2024-07-26T11:33:19.435 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 podman[114130]: 2024-07-26 11:33:19.118577532 +0000 UTC m=+0.033958896 container died 76b9af7200e0c80b31a36a369ba1b0ea91d60bf329e0eda1d2aceb19040cd6cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240724, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-07-26T11:33:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:19 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/2178607141,v1:172.21.15.134:6837/2178607141] up:rejoin 2024-07-26T11:33:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:19 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T11:33:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:19 smithi189 ceph-mon[36790]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:33:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 340.040716 secs 2024-07-26T11:33:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:20 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:33:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:20 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/2178607141,v1:172.21.15.134:6837/2178607141] up:active 2024-07-26T11:33:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:20 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T11:33:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:20 smithi134 ceph-mon[31966]: pgmap v1036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T11:33:20.173 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 podman[114130]: 2024-07-26 11:33:19.896968689 +0000 UTC m=+0.812350083 container remove 76b9af7200e0c80b31a36a369ba1b0ea91d60bf329e0eda1d2aceb19040cd6cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-07-26T11:33:20.173 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:19 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:33:20.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:20 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:33:20.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:20 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/2178607141,v1:172.21.15.134:6837/2178607141] up:active 2024-07-26T11:33:20.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:20 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T11:33:20.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:20 smithi189 ceph-mon[36790]: pgmap v1036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T11:33:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:20 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:33:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:20 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/2178607141,v1:172.21.15.134:6837/2178607141] up:active 2024-07-26T11:33:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:20 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T11:33:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:20 smithi193 ceph-mon[37473]: pgmap v1036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-07-26T11:33:20.672 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:20 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T11:33:20.672 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:20 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.607s CPU time. 2024-07-26T11:33:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 359.466074 secs 2024-07-26T11:33:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 359.466074 secs 2024-07-26T11:33:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 359.466074 secs 2024-07-26T11:33:22.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:22 smithi189 ceph-mon[36790]: pgmap v1037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:33:22.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:33:22.364+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24005880 con 0x7f9b4c10f330 2024-07-26T11:33:22.367 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:22 smithi134 ceph-mon[31966]: pgmap v1037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:33:22.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:33:22.365+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24005880 con 0x7f9b4c10f330 2024-07-26T11:33:22.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:33:22.402+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20005820 con 0x55c65e5ee670 2024-07-26T11:33:22.405 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:33:22.403+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20005820 con 0x55c65e5ee670 2024-07-26T11:33:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:22 smithi193 ceph-mon[37473]: pgmap v1037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 102 B/s wr, 10 op/s 2024-07-26T11:33:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:24 smithi134 ceph-mon[31966]: pgmap v1038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:33:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 345.040855 secs 2024-07-26T11:33:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:24 smithi189 ceph-mon[36790]: pgmap v1038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:33:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 345.040855 secs 2024-07-26T11:33:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:24 smithi193 ceph-mon[37473]: pgmap v1038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-07-26T11:33:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 345.040855 secs 2024-07-26T11:33:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:26 smithi134 ceph-mon[31966]: pgmap v1039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 189 B/s wr, 8 op/s 2024-07-26T11:33:26.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:26 smithi134 ceph-mon[31966]: 1 slow requests, 1 included below; oldest blocked for > 364.466211 secs 2024-07-26T11:33:26.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:26 smithi134 ceph-mon[31966]: slow request 364.466210 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T11:33:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:26 smithi189 ceph-mon[36790]: pgmap v1039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 189 B/s wr, 8 op/s 2024-07-26T11:33:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:26 smithi189 ceph-mon[36790]: 1 slow requests, 1 included below; oldest blocked for > 364.466211 secs 2024-07-26T11:33:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:26 smithi189 ceph-mon[36790]: slow request 364.466210 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T11:33:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:26 smithi193 ceph-mon[37473]: pgmap v1039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 189 B/s wr, 8 op/s 2024-07-26T11:33:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:26 smithi193 ceph-mon[37473]: 1 slow requests, 1 included below; oldest blocked for > 364.466211 secs 2024-07-26T11:33:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:26 smithi193 ceph-mon[37473]: slow request 364.466210 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T11:33:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:28 smithi134 ceph-mon[31966]: pgmap v1040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:33:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:28 smithi189 ceph-mon[36790]: pgmap v1040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:33:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:28 smithi193 ceph-mon[37473]: pgmap v1040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:33:28.558 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:33:28.557+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:33:28.559 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:33:28.557+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:28.559 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:33:28.557+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:33:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 350.040986 secs 2024-07-26T11:33:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 350.040986 secs 2024-07-26T11:33:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 350.040986 secs 2024-07-26T11:33:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:30 smithi134 ceph-mon[31966]: pgmap v1041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:33:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:30 smithi189 ceph-mon[36790]: pgmap v1041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:33:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:30 smithi193 ceph-mon[37473]: pgmap v1041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:33:30.797 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:30 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 5. 2024-07-26T11:33:30.797 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:30 smithi134 systemd[1]: Stopped Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:33:30.797 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:30 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.607s CPU time. 2024-07-26T11:33:30.797 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:30 smithi134 systemd[1]: Starting Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968... 2024-07-26T11:33:31.056 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:30 smithi134 podman[114225]: 2024-07-26 11:33:30.706534538 +0000 UTC m=+0.040289531 image pull ae51f432d4fa82622bc253ca9e2da7666c617e096e99d43d0a0e4fdaca8865f8 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2 2024-07-26T11:33:31.056 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:30 smithi134 podman[114225]: 2024-07-26 11:33:30.855999271 +0000 UTC m=+0.189754261 container create 6a312d9ef987852aa435ba0cbf2373bb921f6d02966da6636bf4a17b2a001796 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240724, io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-07-26T11:33:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 369.466361 secs 2024-07-26T11:33:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:33:31.423 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:31 smithi134 podman[114225]: 2024-07-26 11:33:31.167118691 +0000 UTC m=+0.500873662 container init 6a312d9ef987852aa435ba0cbf2373bb921f6d02966da6636bf4a17b2a001796 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, org.label-schema.license=GPLv2, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-07-26T11:33:31.423 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:31 smithi134 podman[114225]: 2024-07-26 11:33:31.179500585 +0000 UTC m=+0.513255538 container start 6a312d9ef987852aa435ba0cbf2373bb921f6d02966da6636bf4a17b2a001796 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240724) 2024-07-26T11:33:31.423 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:31 smithi134 ceph-mds[114260]: -- 172.21.15.134:0/2996978629 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557e79377c20 con 0x557e7d519a80 2024-07-26T11:33:31.423 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:31 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: starting mds.d at 2024-07-26T11:33:31.423 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:31 smithi134 bash[114225]: 6a312d9ef987852aa435ba0cbf2373bb921f6d02966da6636bf4a17b2a001796 2024-07-26T11:33:31.423 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:31 smithi134 systemd[1]: Started Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:33:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 369.466361 secs 2024-07-26T11:33:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:33:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 369.466361 secs 2024-07-26T11:33:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:33:32.418 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:32 smithi134 ceph-mon[31966]: pgmap v1042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:33:32.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:32 smithi134 ceph-mon[31966]: daemon mds.d restarted 2024-07-26T11:33:32.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:32 smithi134 ceph-mon[31966]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:33:32.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:32 smithi134 ceph-mon[31966]: osdmap e127: 12 total, 12 up, 12 in 2024-07-26T11:33:32.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:32 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/4292174368,v1:172.21.15.134:6837/4292174368] up:boot 2024-07-26T11:33:32.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:32 smithi134 ceph-mon[31966]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:33:32.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:32 smithi134 ceph-mon[31966]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:33:32.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:33:32.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:32 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T11:33:32.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:32.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:32.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:32.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:32 smithi189 ceph-mon[36790]: pgmap v1042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:33:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:32 smithi189 ceph-mon[36790]: daemon mds.d restarted 2024-07-26T11:33:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:32 smithi189 ceph-mon[36790]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:33:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:32 smithi189 ceph-mon[36790]: osdmap e127: 12 total, 12 up, 12 in 2024-07-26T11:33:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:32 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/4292174368,v1:172.21.15.134:6837/4292174368] up:boot 2024-07-26T11:33:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:32 smithi189 ceph-mon[36790]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:33:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:32 smithi189 ceph-mon[36790]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:33:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:33:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:32 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T11:33:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:32 smithi193 ceph-mon[37473]: pgmap v1042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 170 B/s wr, 8 op/s 2024-07-26T11:33:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:32 smithi193 ceph-mon[37473]: daemon mds.d restarted 2024-07-26T11:33:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:32 smithi193 ceph-mon[37473]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-07-26T11:33:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:32 smithi193 ceph-mon[37473]: osdmap e127: 12 total, 12 up, 12 in 2024-07-26T11:33:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:32 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/4292174368,v1:172.21.15.134:6837/4292174368] up:boot 2024-07-26T11:33:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:32 smithi193 ceph-mon[37473]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-07-26T11:33:32.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:32 smithi193 ceph-mon[37473]: fsmap cephfs:4/5 {0=f=up:active,2=j=up:active,3=c=up:active,4=k=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-07-26T11:33:32.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-07-26T11:33:32.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:32 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-07-26T11:33:32.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:32.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:32.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:32.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:32.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:33:32.897+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194005850 con 0x55a08e2d94d0 2024-07-26T11:33:32.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:33:32.898+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194005850 con 0x55a08e2d94d0 2024-07-26T11:33:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:33:32.900+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198005850 con 0x7f21c8105550 2024-07-26T11:33:32.902 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:33:32.900+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198005850 con 0x7f21c8105550 2024-07-26T11:33:33.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:33:33.033+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:33:33.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:33:33.033+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:33:33.035 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:33:33.033+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:33.577 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:33 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/4292174368,v1:172.21.15.134:6837/4292174368] up:resolve 2024-07-26T11:33:33.578 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:33 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T11:33:33.578 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:33.578 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:33 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/4292174368,v1:172.21.15.134:6837/4292174368] up:resolve 2024-07-26T11:33:33.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:33 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T11:33:33.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:33.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:33 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/4292174368,v1:172.21.15.134:6837/4292174368] up:resolve 2024-07-26T11:33:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:33 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-07-26T11:33:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:34 smithi134 ceph-mon[31966]: pgmap v1044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 355.041052 secs 2024-07-26T11:33:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:34 smithi134 ceph-mon[31966]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299993 2024-07-26T11:33:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:34 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/4292174368,v1:172.21.15.134:6837/4292174368] up:reconnect 2024-07-26T11:33:34.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:34 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T11:33:34.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:34 smithi134 ceph-mon[31966]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:33:34.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:34 smithi193 ceph-mon[37473]: pgmap v1044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:34.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 355.041052 secs 2024-07-26T11:33:34.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:34 smithi193 ceph-mon[37473]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299993 2024-07-26T11:33:34.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:34 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/4292174368,v1:172.21.15.134:6837/4292174368] up:reconnect 2024-07-26T11:33:34.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:34 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T11:33:34.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:34 smithi193 ceph-mon[37473]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:33:34.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:34 smithi189 ceph-mon[36790]: pgmap v1044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:33:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 355.041052 secs 2024-07-26T11:33:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:34 smithi189 ceph-mon[36790]: reconnect by client.26981 172.21.15.134:0/446159071 after 0.00299993 2024-07-26T11:33:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:34 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/4292174368,v1:172.21.15.134:6837/4292174368] up:reconnect 2024-07-26T11:33:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:34 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:reconnect 4 up:active 2024-07-26T11:33:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:34 smithi189 ceph-mon[36790]: reconnect by client.25409 172.21.15.134:0/4248055675 after 0.0109998 2024-07-26T11:33:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/4292174368,v1:172.21.15.134:6837/4292174368] up:rejoin 2024-07-26T11:33:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-mon[31966]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T11:33:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-mon[31966]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:33:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:35.673 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: ceph-mds: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5320-ge8f88bb1/rpm/el9/BUILD/ceph-19.0.0-5320-ge8f88bb1/redhat-linux-build/boost/include/boost/smart_ptr/intrusive_ptr.hpp:201: T* boost::intrusive_ptr::operator->() const [with T = const Message]: Assertion `px != 0' failed. 2024-07-26T11:33:35.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: *** Caught signal (Aborted) ** 2024-07-26T11:33:35.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: in thread 7fea13a00640 thread_name:ms_dispatch 2024-07-26T11:33:35.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:33:35.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fea19c3e6f0] 2024-07-26T11:33:35.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fea19c8b94c] 2024-07-26T11:33:35.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 3: raise() 2024-07-26T11:33:35.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 4: abort() 2024-07-26T11:33:35.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 5: /lib64/libc.so.6(+0x2871b) [0x7fea19c2871b] 2024-07-26T11:33:35.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 6: /lib64/libc.so.6(+0x37386) [0x7fea19c37386] 2024-07-26T11:33:35.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557e5d8a16b9] 2024-07-26T11:33:35.674 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557e5da5f80c] 2024-07-26T11:33:35.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557e5d99f2ca] 2024-07-26T11:33:35.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x557e5d96f753] 2024-07-26T11:33:35.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557e5d970a1b] 2024-07-26T11:33:35.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557e5d954752] 2024-07-26T11:33:35.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557e5d8a2a56] 2024-07-26T11:33:35.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557e5d8a6637] 2024-07-26T11:33:35.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557e5d8a6f5e] 2024-07-26T11:33:35.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557e5d88a03a] 2024-07-26T11:33:35.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fea1a79b407] 2024-07-26T11:33:35.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 18: (DispatchQueue::entry()+0x60f) [0x7fea1a79bcf9] 2024-07-26T11:33:35.675 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fea1a8785a1] 2024-07-26T11:33:35.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 20: (Thread::entry_wrapper()+0x43) [0x7fea1a625cdd] 2024-07-26T11:33:35.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 21: (Thread::_entry_func(void*)+0xd) [0x7fea1a625cf9] 2024-07-26T11:33:35.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 22: /lib64/libc.so.6(+0x89c02) [0x7fea19c89c02] 2024-07-26T11:33:35.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fea19d0ec40] 2024-07-26T11:33:35.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 2024-07-26T11:33:35.362+0000 7fea13a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:33:35.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: in thread 7fea13a00640 thread_name:ms_dispatch 2024-07-26T11:33:35.676 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 2024-07-26T11:33:35.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:33:35.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fea19c3e6f0] 2024-07-26T11:33:35.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fea19c8b94c] 2024-07-26T11:33:35.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 3: raise() 2024-07-26T11:33:35.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 4: abort() 2024-07-26T11:33:35.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 5: /lib64/libc.so.6(+0x2871b) [0x7fea19c2871b] 2024-07-26T11:33:35.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 6: /lib64/libc.so.6(+0x37386) [0x7fea19c37386] 2024-07-26T11:33:35.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557e5d8a16b9] 2024-07-26T11:33:35.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557e5da5f80c] 2024-07-26T11:33:35.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557e5d99f2ca] 2024-07-26T11:33:35.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x557e5d96f753] 2024-07-26T11:33:35.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557e5d970a1b] 2024-07-26T11:33:35.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557e5d954752] 2024-07-26T11:33:35.677 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557e5d8a2a56] 2024-07-26T11:33:35.678 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557e5d8a6637] 2024-07-26T11:33:35.678 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557e5d8a6f5e] 2024-07-26T11:33:35.678 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557e5d88a03a] 2024-07-26T11:33:35.678 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fea1a79b407] 2024-07-26T11:33:35.678 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 18: (DispatchQueue::entry()+0x60f) [0x7fea1a79bcf9] 2024-07-26T11:33:35.678 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fea1a8785a1] 2024-07-26T11:33:35.678 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 20: (Thread::entry_wrapper()+0x43) [0x7fea1a625cdd] 2024-07-26T11:33:35.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 21: (Thread::_entry_func(void*)+0xd) [0x7fea1a625cf9] 2024-07-26T11:33:35.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 22: /lib64/libc.so.6(+0x89c02) [0x7fea19c89c02] 2024-07-26T11:33:35.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fea19d0ec40] 2024-07-26T11:33:35.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:33:35.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 2024-07-26T11:33:35.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 0> 2024-07-26T11:33:35.362+0000 7fea13a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:33:35.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: in thread 7fea13a00640 thread_name:ms_dispatch 2024-07-26T11:33:35.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 2024-07-26T11:33:35.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:33:35.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fea19c3e6f0] 2024-07-26T11:33:35.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fea19c8b94c] 2024-07-26T11:33:35.679 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 3: raise() 2024-07-26T11:33:35.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 4: abort() 2024-07-26T11:33:35.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 5: /lib64/libc.so.6(+0x2871b) [0x7fea19c2871b] 2024-07-26T11:33:35.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 6: /lib64/libc.so.6(+0x37386) [0x7fea19c37386] 2024-07-26T11:33:35.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557e5d8a16b9] 2024-07-26T11:33:35.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557e5da5f80c] 2024-07-26T11:33:35.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557e5d99f2ca] 2024-07-26T11:33:35.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x557e5d96f753] 2024-07-26T11:33:35.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557e5d970a1b] 2024-07-26T11:33:35.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557e5d954752] 2024-07-26T11:33:35.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557e5d8a2a56] 2024-07-26T11:33:35.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557e5d8a6637] 2024-07-26T11:33:35.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557e5d8a6f5e] 2024-07-26T11:33:35.680 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557e5d88a03a] 2024-07-26T11:33:35.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fea1a79b407] 2024-07-26T11:33:35.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 18: (DispatchQueue::entry()+0x60f) [0x7fea1a79bcf9] 2024-07-26T11:33:35.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fea1a8785a1] 2024-07-26T11:33:35.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 20: (Thread::entry_wrapper()+0x43) [0x7fea1a625cdd] 2024-07-26T11:33:35.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 21: (Thread::_entry_func(void*)+0xd) [0x7fea1a625cf9] 2024-07-26T11:33:35.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 22: /lib64/libc.so.6(+0x89c02) [0x7fea19c89c02] 2024-07-26T11:33:35.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fea19d0ec40] 2024-07-26T11:33:35.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:33:35.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 2024-07-26T11:33:35.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 0> 2024-07-26T11:33:35.362+0000 7fea13a00640 -1 *** Caught signal (Aborted) ** 2024-07-26T11:33:35.681 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: in thread 7fea13a00640 thread_name:ms_dispatch 2024-07-26T11:33:35.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 2024-07-26T11:33:35.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: ceph version 19.0.0-5320-ge8f88bb1 (e8f88bb1fe0f3b5172ca5b6b08c2be739c07e528) squid (dev) 2024-07-26T11:33:35.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fea19c3e6f0] 2024-07-26T11:33:35.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 2: /lib64/libc.so.6(+0x8b94c) [0x7fea19c8b94c] 2024-07-26T11:33:35.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 3: raise() 2024-07-26T11:33:35.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 4: abort() 2024-07-26T11:33:35.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 5: /lib64/libc.so.6(+0x2871b) [0x7fea19c2871b] 2024-07-26T11:33:35.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 6: /lib64/libc.so.6(+0x37386) [0x7fea19c37386] 2024-07-26T11:33:35.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 7: (MDSRank::get_session(boost::intrusive_ptr const&)+0x89) [0x557e5d8a16b9] 2024-07-26T11:33:35.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 8: (MDCache::path_traverse(boost::intrusive_ptr const&, ContextFactory&, filepath const&, int, std::vector >*, CInode**, CDir**)+0x58e) [0x557e5da5f80c] 2024-07-26T11:33:35.682 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 9: (Server::handle_peer_rmdir_prep(boost::intrusive_ptr const&)+0x32a) [0x557e5d99f2ca] 2024-07-26T11:33:35.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 10: (Server::dispatch_peer_request(boost::intrusive_ptr const&)+0x923) [0x557e5d96f753] 2024-07-26T11:33:35.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 11: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x10c3) [0x557e5d970a1b] 2024-07-26T11:33:35.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x9ba) [0x557e5d954752] 2024-07-26T11:33:35.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x81a) [0x557e5d8a2a56] 2024-07-26T11:33:35.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x28f) [0x557e5d8a6637] 2024-07-26T11:33:35.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x94) [0x557e5d8a6f5e] 2024-07-26T11:33:35.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x3f0) [0x557e5d88a03a] 2024-07-26T11:33:35.683 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 17: /usr/lib64/ceph/libceph-common.so.2(+0x39b407) [0x7fea1a79b407] 2024-07-26T11:33:35.684 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 18: (DispatchQueue::entry()+0x60f) [0x7fea1a79bcf9] 2024-07-26T11:33:35.684 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 19: /usr/lib64/ceph/libceph-common.so.2(+0x4785a1) [0x7fea1a8785a1] 2024-07-26T11:33:35.684 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 20: (Thread::entry_wrapper()+0x43) [0x7fea1a625cdd] 2024-07-26T11:33:35.684 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 21: (Thread::_entry_func(void*)+0xd) [0x7fea1a625cf9] 2024-07-26T11:33:35.684 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 22: /lib64/libc.so.6(+0x89c02) [0x7fea19c89c02] 2024-07-26T11:33:35.684 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 23: /lib64/libc.so.6(+0x10ec40) [0x7fea19d0ec40] 2024-07-26T11:33:35.684 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-07-26T11:33:35.684 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d[114240]: 2024-07-26T11:33:35.684 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:35 smithi134 podman[114733]: 2024-07-26 11:33:35.463653479 +0000 UTC m=+0.033499145 container died 6a312d9ef987852aa435ba0cbf2373bb921f6d02966da6636bf4a17b2a001796 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240724, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-07-26T11:33:35.686 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:35 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/4292174368,v1:172.21.15.134:6837/4292174368] up:rejoin 2024-07-26T11:33:35.686 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:35 smithi193 ceph-mon[37473]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T11:33:35.686 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:35 smithi193 ceph-mon[37473]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:33:35.686 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:35.686 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:35 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/4292174368,v1:172.21.15.134:6837/4292174368] up:rejoin 2024-07-26T11:33:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:35 smithi189 ceph-mon[36790]: fsmap cephfs:5/5 4 up:standby-replay 1 up:rejoin 4 up:active 2024-07-26T11:33:35.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:35 smithi189 ceph-mon[36790]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-07-26T11:33:35.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:35.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:36.672 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:36 smithi134 podman[114733]: 2024-07-26 11:33:36.467973951 +0000 UTC m=+1.037819657 container remove 6a312d9ef987852aa435ba0cbf2373bb921f6d02966da6636bf4a17b2a001796 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:54b7054dea190fddefae31e8872b94a8bc32a70716ff596bdd9c7ee47e62e2d2, name=ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, RELEASE=wip-vshankar-testing-20240725.121331-debug-e8f88bb, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240724) 2024-07-26T11:33:36.673 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:36 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-07-26T11:33:36.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:36 smithi134 ceph-mon[31966]: pgmap v1045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 8 op/s 2024-07-26T11:33:36.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:36 smithi134 ceph-mon[31966]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:33:36.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:36 smithi134 ceph-mon[31966]: mds.? [v2:172.21.15.134:6836/4292174368,v1:172.21.15.134:6837/4292174368] up:active 2024-07-26T11:33:36.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:36 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T11:33:36.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 374.466419 secs 2024-07-26T11:33:36.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:36.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:36.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:33:36.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:33:36.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:36 smithi189 ceph-mon[36790]: pgmap v1045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 8 op/s 2024-07-26T11:33:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:36 smithi189 ceph-mon[36790]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:33:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:36 smithi189 ceph-mon[36790]: mds.? [v2:172.21.15.134:6836/4292174368,v1:172.21.15.134:6837/4292174368] up:active 2024-07-26T11:33:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:36 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T11:33:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 374.466419 secs 2024-07-26T11:33:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:33:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:33:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:36 smithi193 ceph-mon[37473]: pgmap v1045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 8 op/s 2024-07-26T11:33:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:36 smithi193 ceph-mon[37473]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-07-26T11:33:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:36 smithi193 ceph-mon[37473]: mds.? [v2:172.21.15.134:6836/4292174368,v1:172.21.15.134:6837/4292174368] up:active 2024-07-26T11:33:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:36 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T11:33:36.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 374.466419 secs 2024-07-26T11:33:36.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:36.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:36.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:33:36.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:33:36.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:33:37.350 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:37 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T11:33:37.351 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:37 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.568s CPU time. 2024-07-26T11:33:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:37 smithi134 ceph-mon[31966]: pgmap v1046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 9 op/s 2024-07-26T11:33:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:37 smithi134 ceph-mon[31966]: Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T11:33:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:37 smithi189 ceph-mon[36790]: pgmap v1046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 9 op/s 2024-07-26T11:33:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:37 smithi189 ceph-mon[36790]: Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T11:33:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:37 smithi193 ceph-mon[37473]: pgmap v1046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 13 KiB/s rd, 9 op/s 2024-07-26T11:33:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:37 smithi193 ceph-mon[37473]: Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T11:33:39.355 INFO:journalctl@ceph.mds.f.smithi193.stdout:Jul 26 11:33:39 smithi193 ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968-mds-f[68404]: 2024-07-26T11:33:39.073+0000 7f092af66640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-07-26T11:33:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:39 smithi134 ceph-mon[31966]: 2 slow requests, 1 included below; oldest blocked for > 360.041151 secs 2024-07-26T11:33:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:39 smithi134 ceph-mon[31966]: slow request 360.041150 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T11:33:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:39 smithi134 ceph-mon[31966]: pgmap v1047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 111 B/s wr, 10 op/s 2024-07-26T11:33:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:39 smithi189 ceph-mon[36790]: 2 slow requests, 1 included below; oldest blocked for > 360.041151 secs 2024-07-26T11:33:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:39 smithi189 ceph-mon[36790]: slow request 360.041150 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T11:33:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:39 smithi189 ceph-mon[36790]: pgmap v1047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 111 B/s wr, 10 op/s 2024-07-26T11:33:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:39 smithi193 ceph-mon[37473]: 2 slow requests, 1 included below; oldest blocked for > 360.041151 secs 2024-07-26T11:33:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:39 smithi193 ceph-mon[37473]: slow request 360.041150 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T11:33:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:39 smithi193 ceph-mon[37473]: pgmap v1047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 111 B/s wr, 10 op/s 2024-07-26T11:33:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:41 smithi134 ceph-mon[31966]: pgmap v1048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 223 B/s wr, 9 op/s 2024-07-26T11:33:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 379.466543 secs 2024-07-26T11:33:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:41 smithi189 ceph-mon[36790]: pgmap v1048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 223 B/s wr, 9 op/s 2024-07-26T11:33:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 379.466543 secs 2024-07-26T11:33:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:41 smithi193 ceph-mon[37473]: pgmap v1048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 14 KiB/s rd, 223 B/s wr, 9 op/s 2024-07-26T11:33:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 379.466543 secs 2024-07-26T11:33:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:33:43.573+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:33:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:33:43.573+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:43.574 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:33:43.573+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:33:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:43 smithi134 ceph-mon[31966]: pgmap v1049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 186 B/s wr, 9 op/s 2024-07-26T11:33:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:43 smithi189 ceph-mon[36790]: pgmap v1049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 186 B/s wr, 9 op/s 2024-07-26T11:33:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:43 smithi193 ceph-mon[37473]: pgmap v1049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 12 KiB/s rd, 186 B/s wr, 9 op/s 2024-07-26T11:33:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 365.041271 secs 2024-07-26T11:33:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 365.041271 secs 2024-07-26T11:33:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 365.041271 secs 2024-07-26T11:33:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:45 smithi134 ceph-mon[31966]: pgmap v1050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 183 B/s wr, 8 op/s 2024-07-26T11:33:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:45 smithi189 ceph-mon[36790]: pgmap v1050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 183 B/s wr, 8 op/s 2024-07-26T11:33:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:45 smithi193 ceph-mon[37473]: pgmap v1050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 11 KiB/s rd, 183 B/s wr, 8 op/s 2024-07-26T11:33:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 384.466669 secs 2024-07-26T11:33:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 384.466669 secs 2024-07-26T11:33:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 384.466669 secs 2024-07-26T11:33:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:47 smithi134 ceph-mon[31966]: pgmap v1051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 183 B/s wr, 9 op/s 2024-07-26T11:33:47.672 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:47 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Scheduled restart job, restart counter is at 6. 2024-07-26T11:33:47.672 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:47 smithi134 systemd[1]: Stopped Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:33:47.673 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:47 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Consumed 1.568s CPU time. 2024-07-26T11:33:47.673 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:47 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Start request repeated too quickly. 2024-07-26T11:33:47.673 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:47 smithi134 systemd[1]: ceph-105fbe5c-4b3e-11ef-bcb7-c7b262605968@mds.d.service: Failed with result 'exit-code'. 2024-07-26T11:33:47.673 INFO:journalctl@ceph.mds.d.smithi134.stdout:Jul 26 11:33:47 smithi134 systemd[1]: Failed to start Ceph mds.d for 105fbe5c-4b3e-11ef-bcb7-c7b262605968. 2024-07-26T11:33:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:47 smithi189 ceph-mon[36790]: pgmap v1051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 183 B/s wr, 9 op/s 2024-07-26T11:33:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:47 smithi193 ceph-mon[37473]: pgmap v1051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 183 B/s wr, 9 op/s 2024-07-26T11:33:48.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:33:48.048+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:33:48.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:33:48.048+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:33:48.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:33:48.048+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 370.041392 secs 2024-07-26T11:33:49.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:49 smithi134 ceph-mon[31966]: pgmap v1052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:33:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 370.041392 secs 2024-07-26T11:33:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:49 smithi189 ceph-mon[36790]: pgmap v1052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:33:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 370.041392 secs 2024-07-26T11:33:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:49 smithi193 ceph-mon[37473]: pgmap v1052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 9 op/s 2024-07-26T11:33:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:51 smithi134 ceph-mon[31966]: pgmap v1053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-07-26T11:33:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 389.466815 secs 2024-07-26T11:33:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:51 smithi189 ceph-mon[36790]: pgmap v1053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-07-26T11:33:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 389.466815 secs 2024-07-26T11:33:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:51 smithi193 ceph-mon[37473]: pgmap v1053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2024-07-26T11:33:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 389.466815 secs 2024-07-26T11:33:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:53 smithi134 ceph-mon[31966]: pgmap v1054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:33:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 375.041509 secs 2024-07-26T11:33:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:53 smithi189 ceph-mon[36790]: pgmap v1054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:33:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 375.041509 secs 2024-07-26T11:33:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:53 smithi193 ceph-mon[37473]: pgmap v1054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:33:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 375.041509 secs 2024-07-26T11:33:55.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:55 smithi134 ceph-mon[31966]: pgmap v1055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:33:55.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:55 smithi134 ceph-mon[31966]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T11:33:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:55 smithi189 ceph-mon[36790]: pgmap v1055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:33:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:55 smithi189 ceph-mon[36790]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T11:33:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:55 smithi193 ceph-mon[37473]: pgmap v1055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:33:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:55 smithi193 ceph-mon[37473]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-07-26T11:33:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 394.466920 secs 2024-07-26T11:33:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 394.466920 secs 2024-07-26T11:33:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 394.466920 secs 2024-07-26T11:33:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:57 smithi134 ceph-mon[31966]: pgmap v1056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:57 smithi189 ceph-mon[36790]: pgmap v1056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:57 smithi193 ceph-mon[37473]: pgmap v1056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:33:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:33:58.576+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:33:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:33:58.576+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:33:58.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:33:58.576+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:33:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 380.041627 secs 2024-07-26T11:33:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:33:59 smithi134 ceph-mon[31966]: pgmap v1057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:33:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 380.041627 secs 2024-07-26T11:33:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:33:59 smithi189 ceph-mon[36790]: pgmap v1057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:33:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 380.041627 secs 2024-07-26T11:33:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:33:59 smithi193 ceph-mon[37473]: pgmap v1057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:01 smithi134 ceph-mon[31966]: pgmap v1058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:34:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 399.467047 secs 2024-07-26T11:34:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:01 smithi189 ceph-mon[36790]: pgmap v1058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:34:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 399.467047 secs 2024-07-26T11:34:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:01 smithi193 ceph-mon[37473]: pgmap v1058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:34:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 399.467047 secs 2024-07-26T11:34:03.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:34:03.063+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:34:03.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:34:03.064+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:34:03.067 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:34:03.064+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:03 smithi134 ceph-mon[31966]: pgmap v1059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:03 smithi189 ceph-mon[36790]: pgmap v1059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:03 smithi193 ceph-mon[37473]: pgmap v1059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:04 smithi134 ceph-mon[31966]: 2 slow requests, 1 included below; oldest blocked for > 385.041747 secs 2024-07-26T11:34:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:04 smithi134 ceph-mon[31966]: slow request 362.296463 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:34:04.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:04 smithi193 ceph-mon[37473]: 2 slow requests, 1 included below; oldest blocked for > 385.041747 secs 2024-07-26T11:34:04.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:04 smithi193 ceph-mon[37473]: slow request 362.296463 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:34:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:04 smithi189 ceph-mon[36790]: 2 slow requests, 1 included below; oldest blocked for > 385.041747 secs 2024-07-26T11:34:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:04 smithi189 ceph-mon[36790]: slow request 362.296463 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:34:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:05 smithi134 ceph-mon[31966]: pgmap v1060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:34:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:05 smithi189 ceph-mon[36790]: pgmap v1060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:34:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:05 smithi193 ceph-mon[37473]: pgmap v1060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:34:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 404.467185 secs 2024-07-26T11:34:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 404.467185 secs 2024-07-26T11:34:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 404.467185 secs 2024-07-26T11:34:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:07 smithi134 ceph-mon[31966]: pgmap v1061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:07 smithi189 ceph-mon[36790]: pgmap v1061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:07 smithi193 ceph-mon[37473]: pgmap v1061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 390.041869 secs 2024-07-26T11:34:09.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:09 smithi134 ceph-mon[31966]: pgmap v1062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 390.041869 secs 2024-07-26T11:34:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:09 smithi189 ceph-mon[36790]: pgmap v1062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 390.041869 secs 2024-07-26T11:34:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:09 smithi193 ceph-mon[37473]: pgmap v1062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:11 smithi134 ceph-mon[31966]: pgmap v1063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:34:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 409.467318 secs 2024-07-26T11:34:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:11 smithi189 ceph-mon[36790]: pgmap v1063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:34:11.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 409.467318 secs 2024-07-26T11:34:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:11 smithi193 ceph-mon[37473]: pgmap v1063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:34:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 409.467318 secs 2024-07-26T11:34:13.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:34:13.584+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:34:13.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:34:13.584+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:34:13.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:34:13.584+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:13 smithi134 ceph-mon[31966]: pgmap v1064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:13 smithi189 ceph-mon[36790]: pgmap v1064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:13 smithi193 ceph-mon[37473]: pgmap v1064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 395.041991 secs 2024-07-26T11:34:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 395.041991 secs 2024-07-26T11:34:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 395.041991 secs 2024-07-26T11:34:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:15 smithi134 ceph-mon[31966]: pgmap v1065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:34:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:15 smithi189 ceph-mon[36790]: pgmap v1065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:34:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:15 smithi193 ceph-mon[37473]: pgmap v1065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:34:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 414.467448 secs 2024-07-26T11:34:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 414.467448 secs 2024-07-26T11:34:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 414.467448 secs 2024-07-26T11:34:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:17 smithi134 ceph-mon[31966]: pgmap v1066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:17 smithi189 ceph-mon[36790]: pgmap v1066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:17 smithi193 ceph-mon[37473]: pgmap v1066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:18.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:34:18.077+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:34:18.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:34:18.077+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:34:18.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:34:18.077+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 400.042116 secs 2024-07-26T11:34:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 400.042116 secs 2024-07-26T11:34:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 400.042116 secs 2024-07-26T11:34:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:19 smithi189 ceph-mon[36790]: pgmap v1067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:19 smithi193 ceph-mon[37473]: pgmap v1067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:19 smithi134 ceph-mon[31966]: pgmap v1067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:21 smithi189 ceph-mon[36790]: pgmap v1068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:34:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 419.467634 secs 2024-07-26T11:34:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:21 smithi193 ceph-mon[37473]: pgmap v1068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:34:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 419.467634 secs 2024-07-26T11:34:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:21 smithi134 ceph-mon[31966]: pgmap v1068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:34:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 419.467634 secs 2024-07-26T11:34:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:23 smithi189 ceph-mon[36790]: pgmap v1069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 405.042181 secs 2024-07-26T11:34:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:23 smithi193 ceph-mon[37473]: pgmap v1069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 405.042181 secs 2024-07-26T11:34:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:23 smithi134 ceph-mon[31966]: pgmap v1069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 405.042181 secs 2024-07-26T11:34:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:25 smithi193 ceph-mon[37473]: pgmap v1070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:34:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:25 smithi134 ceph-mon[31966]: pgmap v1070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:34:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:25 smithi189 ceph-mon[36790]: pgmap v1070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:34:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 424.467717 secs 2024-07-26T11:34:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 424.467717 secs 2024-07-26T11:34:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 424.467717 secs 2024-07-26T11:34:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:27 smithi193 ceph-mon[37473]: pgmap v1071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:27 smithi134 ceph-mon[31966]: pgmap v1071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:27 smithi189 ceph-mon[36790]: pgmap v1071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:28.601 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:34:28.600+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:34:28.601 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:34:28.600+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:34:28.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:34:28.600+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 410.042280 secs 2024-07-26T11:34:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 410.042280 secs 2024-07-26T11:34:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 410.042280 secs 2024-07-26T11:34:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:29 smithi193 ceph-mon[37473]: pgmap v1072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:29.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:29 smithi134 ceph-mon[31966]: pgmap v1072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:29 smithi189 ceph-mon[36790]: pgmap v1072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:31 smithi193 ceph-mon[37473]: pgmap v1073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:34:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 429.467794 secs 2024-07-26T11:34:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:31 smithi134 ceph-mon[31966]: pgmap v1073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:34:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 429.467794 secs 2024-07-26T11:34:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:31 smithi189 ceph-mon[36790]: pgmap v1073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:34:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 429.467794 secs 2024-07-26T11:34:33.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:34:33.093+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:34:33.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:34:33.093+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:33.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:34:33.093+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:34:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:33 smithi193 ceph-mon[37473]: pgmap v1074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 415.042364 secs 2024-07-26T11:34:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:33 smithi134 ceph-mon[31966]: pgmap v1074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 415.042364 secs 2024-07-26T11:34:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:33 smithi189 ceph-mon[36790]: pgmap v1074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 415.042364 secs 2024-07-26T11:34:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:35 smithi193 ceph-mon[37473]: pgmap v1075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:34:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:35 smithi134 ceph-mon[31966]: pgmap v1075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:34:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:35 smithi189 ceph-mon[36790]: pgmap v1075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:34:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 434.467873 secs 2024-07-26T11:34:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:34:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 434.467873 secs 2024-07-26T11:34:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:34:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 434.467873 secs 2024-07-26T11:34:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:34:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:37 smithi134 ceph-mon[31966]: pgmap v1076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:34:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:34:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:34:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:37 smithi189 ceph-mon[36790]: pgmap v1076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:34:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:34:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:34:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:37 smithi193 ceph-mon[37473]: pgmap v1076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:34:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:34:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:34:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:38 smithi134 ceph-mon[31966]: pgmap v1077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:38 smithi134 ceph-mon[31966]: pgmap v1078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:34:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 420.042437 secs 2024-07-26T11:34:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:38 smithi189 ceph-mon[36790]: pgmap v1077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:38 smithi189 ceph-mon[36790]: pgmap v1078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:34:38.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 420.042437 secs 2024-07-26T11:34:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:38 smithi193 ceph-mon[37473]: pgmap v1077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:38 smithi193 ceph-mon[37473]: pgmap v1078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:34:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 420.042437 secs 2024-07-26T11:34:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:40 smithi134 ceph-mon[31966]: pgmap v1079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:40 smithi189 ceph-mon[36790]: pgmap v1079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:40 smithi193 ceph-mon[37473]: pgmap v1079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 439.467998 secs 2024-07-26T11:34:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 439.467998 secs 2024-07-26T11:34:42.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 439.467998 secs 2024-07-26T11:34:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:42 smithi134 ceph-mon[31966]: pgmap v1080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:34:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:42 smithi189 ceph-mon[36790]: pgmap v1080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:34:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:42 smithi193 ceph-mon[37473]: pgmap v1080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:34:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:34:43.604+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:34:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:34:43.604+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:34:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:34:43.604+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 425.042546 secs 2024-07-26T11:34:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 425.042546 secs 2024-07-26T11:34:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 425.042546 secs 2024-07-26T11:34:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:44 smithi134 ceph-mon[31966]: pgmap v1081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:44 smithi189 ceph-mon[36790]: pgmap v1081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:44 smithi193 ceph-mon[37473]: pgmap v1081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:46 smithi134 ceph-mon[31966]: pgmap v1082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 444.468139 secs 2024-07-26T11:34:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:46 smithi189 ceph-mon[36790]: pgmap v1082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 444.468139 secs 2024-07-26T11:34:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:46 smithi193 ceph-mon[37473]: pgmap v1082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:34:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 444.468139 secs 2024-07-26T11:34:48.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:34:48.097+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:34:48.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:34:48.098+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:34:48.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:34:48.098+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:48 smithi134 ceph-mon[31966]: pgmap v1083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:48.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 430.042666 secs 2024-07-26T11:34:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:48 smithi189 ceph-mon[36790]: pgmap v1083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 430.042666 secs 2024-07-26T11:34:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:48 smithi193 ceph-mon[37473]: pgmap v1083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:34:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 430.042666 secs 2024-07-26T11:34:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:50 smithi134 ceph-mon[31966]: pgmap v1084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:34:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:50 smithi189 ceph-mon[36790]: pgmap v1084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:34:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:50 smithi193 ceph-mon[37473]: pgmap v1084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:34:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 449.468254 secs 2024-07-26T11:34:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 449.468254 secs 2024-07-26T11:34:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 449.468254 secs 2024-07-26T11:34:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:52 smithi134 ceph-mon[31966]: pgmap v1085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:52 smithi189 ceph-mon[36790]: pgmap v1085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:52 smithi193 ceph-mon[37473]: pgmap v1085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 435.042777 secs 2024-07-26T11:34:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 435.042777 secs 2024-07-26T11:34:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 435.042777 secs 2024-07-26T11:34:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:54 smithi134 ceph-mon[31966]: pgmap v1086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:54 smithi189 ceph-mon[36790]: pgmap v1086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:54 smithi193 ceph-mon[37473]: pgmap v1086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:34:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:56 smithi134 ceph-mon[31966]: pgmap v1087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 454.468400 secs 2024-07-26T11:34:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:56 smithi189 ceph-mon[36790]: pgmap v1087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 454.468400 secs 2024-07-26T11:34:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:56 smithi193 ceph-mon[37473]: pgmap v1087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:34:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 454.468400 secs 2024-07-26T11:34:58.611 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:34:58.609+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:34:58.611 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:34:58.609+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:34:58.611 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:34:58.609+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:34:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:58 smithi134 ceph-mon[31966]: pgmap v1088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:34:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 440.042905 secs 2024-07-26T11:34:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:58 smithi189 ceph-mon[36790]: pgmap v1088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:34:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 440.042905 secs 2024-07-26T11:34:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:58 smithi193 ceph-mon[37473]: pgmap v1088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:34:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:34:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 440.042905 secs 2024-07-26T11:35:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:00 smithi134 ceph-mon[31966]: pgmap v1089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:35:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:00 smithi189 ceph-mon[36790]: pgmap v1089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:35:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:00 smithi193 ceph-mon[37473]: pgmap v1089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:35:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 459.468522 secs 2024-07-26T11:35:01.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 459.468522 secs 2024-07-26T11:35:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 459.468522 secs 2024-07-26T11:35:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:02 smithi134 ceph-mon[31966]: pgmap v1090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:02 smithi189 ceph-mon[36790]: pgmap v1090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:02 smithi193 ceph-mon[37473]: pgmap v1090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:03.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:35:03.113+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:35:03.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:35:03.113+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:03.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:35:03.113+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:35:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 445.042978 secs 2024-07-26T11:35:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 445.042978 secs 2024-07-26T11:35:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 445.042978 secs 2024-07-26T11:35:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:04 smithi134 ceph-mon[31966]: pgmap v1091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:04 smithi189 ceph-mon[36790]: pgmap v1091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:04 smithi193 ceph-mon[37473]: pgmap v1091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:06 smithi134 ceph-mon[31966]: pgmap v1092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:35:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 464.468642 secs 2024-07-26T11:35:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:06 smithi189 ceph-mon[36790]: pgmap v1092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:35:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 464.468642 secs 2024-07-26T11:35:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:06 smithi193 ceph-mon[37473]: pgmap v1092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:35:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 464.468642 secs 2024-07-26T11:35:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:08 smithi134 ceph-mon[31966]: pgmap v1093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:35:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 450.043036 secs 2024-07-26T11:35:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:08 smithi189 ceph-mon[36790]: pgmap v1093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:35:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 450.043036 secs 2024-07-26T11:35:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:08 smithi193 ceph-mon[37473]: pgmap v1093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:35:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 450.043036 secs 2024-07-26T11:35:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:10 smithi134 ceph-mon[31966]: pgmap v1094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:35:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:10 smithi189 ceph-mon[36790]: pgmap v1094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:35:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:10 smithi193 ceph-mon[37473]: pgmap v1094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:35:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 469.468763 secs 2024-07-26T11:35:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 469.468763 secs 2024-07-26T11:35:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 469.468763 secs 2024-07-26T11:35:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:12 smithi189 ceph-mon[36790]: pgmap v1095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:12 smithi193 ceph-mon[37473]: pgmap v1095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:12 smithi134 ceph-mon[31966]: pgmap v1095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:35:13.619+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:35:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:35:13.619+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:35:13.621 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:35:13.619+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 455.043163 secs 2024-07-26T11:35:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 455.043163 secs 2024-07-26T11:35:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 455.043163 secs 2024-07-26T11:35:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:14 smithi193 ceph-mon[37473]: pgmap v1096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:14 smithi134 ceph-mon[31966]: pgmap v1096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:14 smithi189 ceph-mon[36790]: pgmap v1096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:16 smithi193 ceph-mon[37473]: pgmap v1097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 474.468883 secs 2024-07-26T11:35:17.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:16 smithi134 ceph-mon[31966]: pgmap v1097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:17.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 474.468883 secs 2024-07-26T11:35:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:16 smithi189 ceph-mon[36790]: pgmap v1097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 474.468883 secs 2024-07-26T11:35:18.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:35:18.129+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:35:18.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:35:18.129+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:18.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:35:18.129+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:35:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:18 smithi193 ceph-mon[37473]: pgmap v1098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:35:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 460.043286 secs 2024-07-26T11:35:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:18 smithi134 ceph-mon[31966]: pgmap v1098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:35:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 460.043286 secs 2024-07-26T11:35:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:18 smithi189 ceph-mon[36790]: pgmap v1098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:35:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 460.043286 secs 2024-07-26T11:35:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:20 smithi193 ceph-mon[37473]: pgmap v1099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:35:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:20 smithi134 ceph-mon[31966]: pgmap v1099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:35:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:20 smithi189 ceph-mon[36790]: pgmap v1099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:35:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 479.469002 secs 2024-07-26T11:35:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 479.469002 secs 2024-07-26T11:35:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 479.469002 secs 2024-07-26T11:35:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:22 smithi193 ceph-mon[37473]: pgmap v1100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:23.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:22 smithi189 ceph-mon[36790]: pgmap v1100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:22 smithi134 ceph-mon[31966]: pgmap v1100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 465.043405 secs 2024-07-26T11:35:24.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 465.043405 secs 2024-07-26T11:35:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 465.043405 secs 2024-07-26T11:35:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:24 smithi193 ceph-mon[37473]: pgmap v1101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:24 smithi134 ceph-mon[31966]: pgmap v1101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:24 smithi189 ceph-mon[36790]: pgmap v1101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:35:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:26 smithi193 ceph-mon[37473]: pgmap v1102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 484.469144 secs 2024-07-26T11:35:27.113 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:26 smithi134 ceph-mon[31966]: pgmap v1102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:27.113 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 484.469144 secs 2024-07-26T11:35:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:26 smithi189 ceph-mon[36790]: pgmap v1102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 484.469144 secs 2024-07-26T11:35:28.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:35:28.634+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:35:28.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:35:28.634+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:35:28.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:35:28.635+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:29.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:28 smithi193 ceph-mon[37473]: pgmap v1103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:35:29.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 470.043525 secs 2024-07-26T11:35:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:28 smithi134 ceph-mon[31966]: pgmap v1103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:35:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 470.043525 secs 2024-07-26T11:35:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:28 smithi189 ceph-mon[36790]: pgmap v1103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:35:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 470.043525 secs 2024-07-26T11:35:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:30 smithi193 ceph-mon[37473]: pgmap v1104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:35:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:30 smithi134 ceph-mon[31966]: pgmap v1104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:35:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:30 smithi189 ceph-mon[36790]: pgmap v1104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:35:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 489.469233 secs 2024-07-26T11:35:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 489.469233 secs 2024-07-26T11:35:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 489.469233 secs 2024-07-26T11:35:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:32 smithi193 ceph-mon[37473]: pgmap v1105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:35:33.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:35:33.144+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:35:33.147 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:35:33.144+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:33.147 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:35:33.144+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:35:33.147 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:32 smithi134 ceph-mon[31966]: pgmap v1105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:35:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:32 smithi189 ceph-mon[36790]: pgmap v1105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:35:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 475.043661 secs 2024-07-26T11:35:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 475.043661 secs 2024-07-26T11:35:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 475.043661 secs 2024-07-26T11:35:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:34 smithi134 ceph-mon[31966]: pgmap v1106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:35:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:34 smithi189 ceph-mon[36790]: pgmap v1106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:35:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:34 smithi193 ceph-mon[37473]: pgmap v1106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:35:37.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:36 smithi193 ceph-mon[37473]: pgmap v1107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:35:37.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 494.469354 secs 2024-07-26T11:35:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:36 smithi134 ceph-mon[31966]: pgmap v1107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:35:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 494.469354 secs 2024-07-26T11:35:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:36 smithi189 ceph-mon[36790]: pgmap v1107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:35:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 494.469354 secs 2024-07-26T11:35:37.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:35:37.420+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24005bc0 con 0x7f9b4c10f330 2024-07-26T11:35:37.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:35:37.421+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b24005bc0 con 0x7f9b4c10f330 2024-07-26T11:35:37.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:35:37.423+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20005b00 con 0x55c65e5ee670 2024-07-26T11:35:37.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:35:37.424+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20005b00 con 0x55c65e5ee670 2024-07-26T11:35:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:35:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:35:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:35:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:38 smithi134 ceph-mon[31966]: pgmap v1108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:35:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 480.043768 secs 2024-07-26T11:35:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:35:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:35:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:35:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:38 smithi189 ceph-mon[36790]: pgmap v1108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:35:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 480.043768 secs 2024-07-26T11:35:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:35:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:35:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:35:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:38 smithi193 ceph-mon[37473]: pgmap v1108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:35:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 480.043768 secs 2024-07-26T11:35:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:35:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:35:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:35:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:39 smithi134 ceph-mon[31966]: pgmap v1109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T11:35:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:39 smithi134 ceph-mon[31966]: pgmap v1110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:35:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:39 smithi189 ceph-mon[36790]: pgmap v1109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T11:35:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:39 smithi189 ceph-mon[36790]: pgmap v1110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:35:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:39 smithi193 ceph-mon[37473]: pgmap v1109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T11:35:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:39 smithi193 ceph-mon[37473]: pgmap v1110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:35:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:41 smithi134 ceph-mon[31966]: pgmap v1111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:35:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 499.469473 secs 2024-07-26T11:35:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:41 smithi189 ceph-mon[36790]: pgmap v1111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:35:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 499.469473 secs 2024-07-26T11:35:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:41 smithi193 ceph-mon[37473]: pgmap v1111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:35:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 499.469473 secs 2024-07-26T11:35:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:35:43.650+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:35:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:35:43.650+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:35:43.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:35:43.650+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:43 smithi134 ceph-mon[31966]: pgmap v1112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:35:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 485.043914 secs 2024-07-26T11:35:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:43 smithi189 ceph-mon[36790]: pgmap v1112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:35:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 485.043914 secs 2024-07-26T11:35:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:43 smithi193 ceph-mon[37473]: pgmap v1112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:35:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 485.043914 secs 2024-07-26T11:35:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:45 smithi134 ceph-mon[31966]: pgmap v1113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:35:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:45 smithi189 ceph-mon[36790]: pgmap v1113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:35:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:45 smithi193 ceph-mon[37473]: pgmap v1113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:35:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 504.469596 secs 2024-07-26T11:35:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 504.469596 secs 2024-07-26T11:35:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 504.469596 secs 2024-07-26T11:35:47.925 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:35:47.923+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194005b50 con 0x55a08e2d94d0 2024-07-26T11:35:47.925 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:35:47.923+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198005b10 con 0x7f21c8105550 2024-07-26T11:35:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:35:47.924+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2198005b10 con 0x7f21c8105550 2024-07-26T11:35:47.926 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:35:47.924+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194005b50 con 0x55a08e2d94d0 2024-07-26T11:35:48.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:35:48.159+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:35:48.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:35:48.159+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:35:48.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:35:48.159+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:48.161 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:47 smithi134 ceph-mon[31966]: pgmap v1114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:35:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:47 smithi189 ceph-mon[36790]: pgmap v1114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:35:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:47 smithi193 ceph-mon[37473]: pgmap v1114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:35:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 490.044035 secs 2024-07-26T11:35:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 490.044035 secs 2024-07-26T11:35:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 490.044035 secs 2024-07-26T11:35:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:49 smithi134 ceph-mon[31966]: pgmap v1115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:35:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:49 smithi189 ceph-mon[36790]: pgmap v1115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:35:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:49 smithi193 ceph-mon[37473]: pgmap v1115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:35:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:51 smithi134 ceph-mon[31966]: pgmap v1116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 509.469723 secs 2024-07-26T11:35:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:51 smithi189 ceph-mon[36790]: pgmap v1116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 509.469723 secs 2024-07-26T11:35:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:51 smithi193 ceph-mon[37473]: pgmap v1116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 509.469723 secs 2024-07-26T11:35:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:53 smithi134 ceph-mon[31966]: pgmap v1117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:35:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 495.044150 secs 2024-07-26T11:35:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:53 smithi189 ceph-mon[36790]: pgmap v1117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:35:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 495.044150 secs 2024-07-26T11:35:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:53 smithi193 ceph-mon[37473]: pgmap v1117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:35:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 495.044150 secs 2024-07-26T11:35:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:55 smithi134 ceph-mon[31966]: pgmap v1118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:35:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:55 smithi189 ceph-mon[36790]: pgmap v1118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:35:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:55 smithi193 ceph-mon[37473]: pgmap v1118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:35:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 514.469846 secs 2024-07-26T11:35:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 514.469846 secs 2024-07-26T11:35:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 514.469846 secs 2024-07-26T11:35:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:57 smithi134 ceph-mon[31966]: pgmap v1119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:57 smithi189 ceph-mon[36790]: pgmap v1119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:57 smithi193 ceph-mon[37473]: pgmap v1119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:35:58.667 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:35:58.666+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:35:58.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:35:58.666+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:35:58.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:35:58.666+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:35:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 500.044276 secs 2024-07-26T11:35:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 500.044276 secs 2024-07-26T11:35:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 500.044276 secs 2024-07-26T11:36:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:35:59 smithi134 ceph-mon[31966]: pgmap v1120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:35:59 smithi189 ceph-mon[36790]: pgmap v1120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:35:59 smithi193 ceph-mon[37473]: pgmap v1120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:01 smithi134 ceph-mon[31966]: pgmap v1121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 519.469926 secs 2024-07-26T11:36:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:01 smithi189 ceph-mon[36790]: pgmap v1121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 519.469926 secs 2024-07-26T11:36:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:01 smithi193 ceph-mon[37473]: pgmap v1121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:02.292 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 519.469926 secs 2024-07-26T11:36:03.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:36:03.174+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:36:03.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:36:03.174+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:03.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:36:03.174+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:36:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:03 smithi134 ceph-mon[31966]: pgmap v1122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:36:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 505.044401 secs 2024-07-26T11:36:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:03 smithi189 ceph-mon[36790]: pgmap v1122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:36:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 505.044401 secs 2024-07-26T11:36:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:03 smithi193 ceph-mon[37473]: pgmap v1122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:36:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 505.044401 secs 2024-07-26T11:36:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:05 smithi134 ceph-mon[31966]: pgmap v1123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:36:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:05 smithi189 ceph-mon[36790]: pgmap v1123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:36:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:05 smithi193 ceph-mon[37473]: pgmap v1123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:36:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 524.470008 secs 2024-07-26T11:36:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 524.470008 secs 2024-07-26T11:36:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 524.470008 secs 2024-07-26T11:36:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:07 smithi189 ceph-mon[36790]: pgmap v1124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:07 smithi193 ceph-mon[37473]: pgmap v1124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:07 smithi134 ceph-mon[31966]: pgmap v1124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 510.044523 secs 2024-07-26T11:36:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 510.044523 secs 2024-07-26T11:36:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 510.044523 secs 2024-07-26T11:36:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:09 smithi193 ceph-mon[37473]: pgmap v1125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:09 smithi134 ceph-mon[31966]: pgmap v1125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:09 smithi189 ceph-mon[36790]: pgmap v1125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:11 smithi193 ceph-mon[37473]: pgmap v1126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 529.470124 secs 2024-07-26T11:36:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:11 smithi134 ceph-mon[31966]: pgmap v1126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 529.470124 secs 2024-07-26T11:36:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:11 smithi189 ceph-mon[36790]: pgmap v1126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 529.470124 secs 2024-07-26T11:36:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:36:13.678+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:36:13.680 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:36:13.679+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:36:13.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:36:13.678+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:14.217 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:13 smithi134 ceph-mon[31966]: pgmap v1127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:36:14.217 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 515.044647 secs 2024-07-26T11:36:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:13 smithi193 ceph-mon[37473]: pgmap v1127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:36:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 515.044647 secs 2024-07-26T11:36:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:13 smithi189 ceph-mon[36790]: pgmap v1127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:36:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 515.044647 secs 2024-07-26T11:36:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:15 smithi193 ceph-mon[37473]: pgmap v1128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:36:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:15 smithi134 ceph-mon[31966]: pgmap v1128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:36:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:15 smithi189 ceph-mon[36790]: pgmap v1128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:36:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 534.470208 secs 2024-07-26T11:36:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 534.470208 secs 2024-07-26T11:36:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 534.470208 secs 2024-07-26T11:36:18.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:36:18.190+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:36:18.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:36:18.190+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:36:18.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:36:18.190+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:17 smithi193 ceph-mon[37473]: pgmap v1129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:17 smithi134 ceph-mon[31966]: pgmap v1129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:17 smithi189 ceph-mon[36790]: pgmap v1129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 520.044769 secs 2024-07-26T11:36:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 520.044769 secs 2024-07-26T11:36:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 520.044769 secs 2024-07-26T11:36:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:19 smithi193 ceph-mon[37473]: pgmap v1130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:19 smithi134 ceph-mon[31966]: pgmap v1130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:19 smithi189 ceph-mon[36790]: pgmap v1130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 539.470363 secs 2024-07-26T11:36:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 539.470363 secs 2024-07-26T11:36:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 539.470363 secs 2024-07-26T11:36:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:22 smithi193 ceph-mon[37473]: pgmap v1131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:22 smithi134 ceph-mon[31966]: pgmap v1131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:22.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:36:22.427+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24005f10 con 0x7f9b4c10f330 2024-07-26T11:36:22.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:36:22.428+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24005f10 con 0x7f9b4c10f330 2024-07-26T11:36:22.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:36:22.431+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20005db0 con 0x55c65e5ee670 2024-07-26T11:36:22.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:36:22.431+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20005db0 con 0x55c65e5ee670 2024-07-26T11:36:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:22 smithi189 ceph-mon[36790]: pgmap v1131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:23.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:23 smithi189 ceph-mon[36790]: pgmap v1132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:36:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:23 smithi134 ceph-mon[31966]: pgmap v1132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:36:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:23 smithi193 ceph-mon[37473]: pgmap v1132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:36:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 525.044886 secs 2024-07-26T11:36:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 525.044886 secs 2024-07-26T11:36:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 525.044886 secs 2024-07-26T11:36:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:25 smithi134 ceph-mon[31966]: pgmap v1133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:36:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:25 smithi189 ceph-mon[36790]: pgmap v1133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:36:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:25 smithi193 ceph-mon[37473]: pgmap v1133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:36:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 544.470469 secs 2024-07-26T11:36:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 544.470469 secs 2024-07-26T11:36:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 544.470469 secs 2024-07-26T11:36:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:27 smithi134 ceph-mon[31966]: pgmap v1134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:27 smithi189 ceph-mon[36790]: pgmap v1134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:27 smithi193 ceph-mon[37473]: pgmap v1134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:28.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:36:28.694+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:36:28.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:36:28.694+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:28.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:36:28.694+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:36:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 530.045005 secs 2024-07-26T11:36:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 530.045005 secs 2024-07-26T11:36:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 530.045005 secs 2024-07-26T11:36:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:29 smithi193 ceph-mon[37473]: pgmap v1135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:29 smithi134 ceph-mon[31966]: pgmap v1135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:29 smithi189 ceph-mon[36790]: pgmap v1135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:31 smithi193 ceph-mon[37473]: pgmap v1136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 549.470550 secs 2024-07-26T11:36:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:31 smithi134 ceph-mon[31966]: pgmap v1136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 549.470550 secs 2024-07-26T11:36:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:31 smithi189 ceph-mon[36790]: pgmap v1136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 549.470550 secs 2024-07-26T11:36:32.931 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:36:32.930+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198005e70 con 0x7f21c8105550 2024-07-26T11:36:32.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:36:32.930+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194005e30 con 0x55a08e2d94d0 2024-07-26T11:36:32.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:36:32.931+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198005e70 con 0x7f21c8105550 2024-07-26T11:36:32.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:36:32.931+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194005e30 con 0x55a08e2d94d0 2024-07-26T11:36:33.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:36:33.193+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:36:33.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:36:33.193+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:33.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:36:33.193+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:36:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:33 smithi193 ceph-mon[37473]: pgmap v1137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:36:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 535.045128 secs 2024-07-26T11:36:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:33 smithi134 ceph-mon[31966]: pgmap v1137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:36:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 535.045128 secs 2024-07-26T11:36:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:33 smithi189 ceph-mon[36790]: pgmap v1137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:36:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 535.045128 secs 2024-07-26T11:36:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:35 smithi193 ceph-mon[37473]: pgmap v1138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:36:36.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:35 smithi134 ceph-mon[31966]: pgmap v1138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:36:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:35 smithi189 ceph-mon[36790]: pgmap v1138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:36:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 554.470674 secs 2024-07-26T11:36:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 554.470674 secs 2024-07-26T11:36:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 554.470674 secs 2024-07-26T11:36:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:37 smithi134 ceph-mon[31966]: pgmap v1139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:38.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:37 smithi193 ceph-mon[37473]: pgmap v1139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:37 smithi189 ceph-mon[36790]: pgmap v1139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 540.045193 secs 2024-07-26T11:36:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 540.045193 secs 2024-07-26T11:36:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 540.045193 secs 2024-07-26T11:36:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:39 smithi134 ceph-mon[31966]: pgmap v1140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:36:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:39 smithi189 ceph-mon[36790]: pgmap v1140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:36:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:39 smithi193 ceph-mon[37473]: pgmap v1140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:36:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:36:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:36:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:40 smithi134 ceph-mon[31966]: pgmap v1141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:36:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:40 smithi134 ceph-mon[31966]: pgmap v1142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:36:41.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:36:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:36:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:36:41.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:40 smithi189 ceph-mon[36790]: pgmap v1141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:36:41.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:40 smithi189 ceph-mon[36790]: pgmap v1142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:36:41.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:36:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:36:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:36:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:40 smithi193 ceph-mon[37473]: pgmap v1141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:36:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:40 smithi193 ceph-mon[37473]: pgmap v1142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:36:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:36:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 559.470737 secs 2024-07-26T11:36:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 559.470737 secs 2024-07-26T11:36:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 559.470737 secs 2024-07-26T11:36:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:42 smithi134 ceph-mon[31966]: pgmap v1143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:36:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:42 smithi189 ceph-mon[36790]: pgmap v1143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:36:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:42 smithi193 ceph-mon[37473]: pgmap v1143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:36:43.711 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:36:43.710+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:36:43.711 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:36:43.711+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:36:43.712 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:36:43.711+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 545.045312 secs 2024-07-26T11:36:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 545.045312 secs 2024-07-26T11:36:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 545.045312 secs 2024-07-26T11:36:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:44 smithi134 ceph-mon[31966]: pgmap v1144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:36:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:44 smithi189 ceph-mon[36790]: pgmap v1144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:36:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:44 smithi193 ceph-mon[37473]: pgmap v1144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:36:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 564.470867 secs 2024-07-26T11:36:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:46 smithi193 ceph-mon[37473]: pgmap v1145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:36:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 564.470867 secs 2024-07-26T11:36:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:46 smithi134 ceph-mon[31966]: pgmap v1145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:36:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 564.470867 secs 2024-07-26T11:36:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:46 smithi189 ceph-mon[36790]: pgmap v1145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:36:48.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:36:48.208+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:36:48.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:36:48.209+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:36:48.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:36:48.209+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:48 smithi193 ceph-mon[37473]: pgmap v1146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:36:49.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 550.045386 secs 2024-07-26T11:36:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:48 smithi134 ceph-mon[31966]: pgmap v1146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:36:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 550.045386 secs 2024-07-26T11:36:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:48 smithi189 ceph-mon[36790]: pgmap v1146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:36:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 550.045386 secs 2024-07-26T11:36:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:50 smithi193 ceph-mon[37473]: pgmap v1147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:36:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:50 smithi134 ceph-mon[31966]: pgmap v1147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:36:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:50 smithi189 ceph-mon[36790]: pgmap v1147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:36:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 569.471001 secs 2024-07-26T11:36:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 569.471001 secs 2024-07-26T11:36:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 569.471001 secs 2024-07-26T11:36:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:52 smithi193 ceph-mon[37473]: pgmap v1148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:53.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:52 smithi189 ceph-mon[36790]: pgmap v1148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:52 smithi134 ceph-mon[31966]: pgmap v1148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 555.045482 secs 2024-07-26T11:36:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 555.045482 secs 2024-07-26T11:36:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 555.045482 secs 2024-07-26T11:36:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:55 smithi193 ceph-mon[37473]: pgmap v1149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:55 smithi134 ceph-mon[31966]: pgmap v1149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:55 smithi189 ceph-mon[36790]: pgmap v1149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:36:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 574.471102 secs 2024-07-26T11:36:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 574.471102 secs 2024-07-26T11:36:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 574.471102 secs 2024-07-26T11:36:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:57 smithi193 ceph-mon[37473]: pgmap v1150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:57 smithi134 ceph-mon[31966]: pgmap v1150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:57 smithi189 ceph-mon[36790]: pgmap v1150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:36:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:58 smithi134 ceph-mon[31966]: pgmap v1151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:36:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:58 smithi189 ceph-mon[36790]: pgmap v1151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:36:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:58 smithi193 ceph-mon[37473]: pgmap v1151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:36:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:36:58.715+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:36:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:36:58.715+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:36:58.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:36:58.715+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:36:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:36:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 560.045604 secs 2024-07-26T11:36:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:36:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 560.045604 secs 2024-07-26T11:36:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:36:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 560.045604 secs 2024-07-26T11:37:00.304 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:00 smithi134 ceph-mon[31966]: pgmap v1152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:00 smithi189 ceph-mon[36790]: pgmap v1152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:00 smithi193 ceph-mon[37473]: pgmap v1152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 579.471243 secs 2024-07-26T11:37:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 579.471243 secs 2024-07-26T11:37:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 579.471243 secs 2024-07-26T11:37:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:02 smithi134 ceph-mon[31966]: pgmap v1153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:02 smithi189 ceph-mon[36790]: pgmap v1153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:02 smithi193 ceph-mon[37473]: pgmap v1153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:03.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:37:03.224+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:37:03.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:37:03.224+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:03.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:37:03.224+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:37:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 565.045729 secs 2024-07-26T11:37:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 565.045729 secs 2024-07-26T11:37:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 565.045729 secs 2024-07-26T11:37:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:04 smithi193 ceph-mon[37473]: pgmap v1154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:04 smithi134 ceph-mon[31966]: pgmap v1154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:04 smithi189 ceph-mon[36790]: pgmap v1154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 584.471375 secs 2024-07-26T11:37:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:06 smithi193 ceph-mon[37473]: pgmap v1155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 584.471375 secs 2024-07-26T11:37:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:06 smithi134 ceph-mon[31966]: pgmap v1155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 584.471375 secs 2024-07-26T11:37:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:06 smithi189 ceph-mon[36790]: pgmap v1155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:08 smithi193 ceph-mon[37473]: pgmap v1156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:37:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 570.045852 secs 2024-07-26T11:37:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:08 smithi134 ceph-mon[31966]: pgmap v1156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:37:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 570.045852 secs 2024-07-26T11:37:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:08 smithi189 ceph-mon[36790]: pgmap v1156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:37:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 570.045852 secs 2024-07-26T11:37:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:10 smithi193 ceph-mon[37473]: pgmap v1157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:10 smithi134 ceph-mon[31966]: pgmap v1157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:10 smithi189 ceph-mon[36790]: pgmap v1157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 589.471444 secs 2024-07-26T11:37:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 589.471444 secs 2024-07-26T11:37:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 589.471444 secs 2024-07-26T11:37:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:12 smithi193 ceph-mon[37473]: pgmap v1158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:12 smithi134 ceph-mon[31966]: pgmap v1158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:12 smithi189 ceph-mon[36790]: pgmap v1158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:13.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:37:13.721+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:37:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:37:13.721+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:13.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:37:13.721+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:37:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 575.045967 secs 2024-07-26T11:37:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 575.045967 secs 2024-07-26T11:37:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 575.045967 secs 2024-07-26T11:37:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:15 smithi193 ceph-mon[37473]: pgmap v1159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:15 smithi134 ceph-mon[31966]: pgmap v1159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:15 smithi189 ceph-mon[36790]: pgmap v1159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 594.471571 secs 2024-07-26T11:37:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 594.471571 secs 2024-07-26T11:37:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 594.471571 secs 2024-07-26T11:37:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:17 smithi134 ceph-mon[31966]: pgmap v1160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:17 smithi189 ceph-mon[36790]: pgmap v1160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:17 smithi193 ceph-mon[37473]: pgmap v1160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:18.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:37:18.239+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:37:18.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:37:18.239+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:18.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:37:18.239+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:37:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:18 smithi134 ceph-mon[31966]: pgmap v1161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:37:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:18 smithi189 ceph-mon[36790]: pgmap v1161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:37:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:18 smithi193 ceph-mon[37473]: pgmap v1161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:37:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 580.046036 secs 2024-07-26T11:37:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 580.046036 secs 2024-07-26T11:37:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 580.046036 secs 2024-07-26T11:37:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:20 smithi134 ceph-mon[31966]: pgmap v1162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:20 smithi189 ceph-mon[36790]: pgmap v1162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:20 smithi193 ceph-mon[37473]: pgmap v1162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 599.471656 secs 2024-07-26T11:37:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 599.471656 secs 2024-07-26T11:37:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 599.471656 secs 2024-07-26T11:37:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:22 smithi134 ceph-mon[31966]: pgmap v1163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:22 smithi189 ceph-mon[36790]: pgmap v1163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:22 smithi193 ceph-mon[37473]: pgmap v1163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 585.046174 secs 2024-07-26T11:37:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 585.046174 secs 2024-07-26T11:37:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 585.046174 secs 2024-07-26T11:37:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:24 smithi193 ceph-mon[37473]: pgmap v1164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:24 smithi134 ceph-mon[31966]: pgmap v1164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:24 smithi189 ceph-mon[36790]: pgmap v1164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 604.471751 secs 2024-07-26T11:37:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:26 smithi193 ceph-mon[37473]: pgmap v1165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 604.471751 secs 2024-07-26T11:37:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:26 smithi134 ceph-mon[31966]: pgmap v1165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 604.471751 secs 2024-07-26T11:37:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:26 smithi189 ceph-mon[36790]: pgmap v1165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:28.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:37:28.735+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:37:28.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:37:28.735+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:28.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:37:28.735+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:37:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:28 smithi193 ceph-mon[37473]: pgmap v1166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:37:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 590.046289 secs 2024-07-26T11:37:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:28 smithi134 ceph-mon[31966]: pgmap v1166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:37:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 590.046289 secs 2024-07-26T11:37:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:28 smithi189 ceph-mon[36790]: pgmap v1166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:37:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 590.046289 secs 2024-07-26T11:37:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:30 smithi193 ceph-mon[37473]: pgmap v1167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:30 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 609.471887 secs 2024-07-26T11:37:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:30 smithi134 ceph-mon[31966]: pgmap v1167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:30 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 609.471887 secs 2024-07-26T11:37:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:30 smithi189 ceph-mon[36790]: pgmap v1167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:30 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 609.471887 secs 2024-07-26T11:37:33.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:37:33.242+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:37:33.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:37:33.242+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:33.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:37:33.242+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:37:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:33 smithi193 ceph-mon[37473]: pgmap v1168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:33 smithi134 ceph-mon[31966]: pgmap v1168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:33 smithi189 ceph-mon[36790]: pgmap v1168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 595.046412 secs 2024-07-26T11:37:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 595.046412 secs 2024-07-26T11:37:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 595.046412 secs 2024-07-26T11:37:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:35 smithi193 ceph-mon[37473]: pgmap v1169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:35 smithi134 ceph-mon[31966]: pgmap v1169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:35 smithi189 ceph-mon[36790]: pgmap v1169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:37:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 614.471952 secs 2024-07-26T11:37:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 614.471952 secs 2024-07-26T11:37:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 614.471952 secs 2024-07-26T11:37:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:37 smithi134 ceph-mon[31966]: pgmap v1170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:37 smithi189 ceph-mon[36790]: pgmap v1170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:37 smithi193 ceph-mon[37473]: pgmap v1170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:37:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:38 smithi134 ceph-mon[31966]: pgmap v1171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:37:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:38 smithi189 ceph-mon[36790]: pgmap v1171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:37:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:38 smithi193 ceph-mon[37473]: pgmap v1171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:37:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 600.046551 secs 2024-07-26T11:37:39.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 600.046551 secs 2024-07-26T11:37:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 600.046551 secs 2024-07-26T11:37:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:40 smithi134 ceph-mon[31966]: pgmap v1172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:40 smithi189 ceph-mon[36790]: pgmap v1172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:40 smithi193 ceph-mon[37473]: pgmap v1172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:37:41.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 619.472064 secs 2024-07-26T11:37:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:37:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 619.472064 secs 2024-07-26T11:37:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:37:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 619.472064 secs 2024-07-26T11:37:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:37:42.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:37:42.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:42 smithi134 ceph-mon[31966]: pgmap v1173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:37:42.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:42 smithi134 ceph-mon[31966]: pgmap v1174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:37:42.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:37:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:37:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:37:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:42 smithi189 ceph-mon[36790]: pgmap v1173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:37:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:42 smithi189 ceph-mon[36790]: pgmap v1174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:37:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:37:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:37:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:37:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:42 smithi193 ceph-mon[37473]: pgmap v1173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:37:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:42 smithi193 ceph-mon[37473]: pgmap v1174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:37:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:37:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:37:43.744+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:37:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:37:43.744+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:43.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:37:43.744+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:37:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:44 smithi134 ceph-mon[31966]: pgmap v1175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:37:44.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 605.046670 secs 2024-07-26T11:37:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:44 smithi189 ceph-mon[36790]: pgmap v1175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:37:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 605.046670 secs 2024-07-26T11:37:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:44 smithi193 ceph-mon[37473]: pgmap v1175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:37:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 605.046670 secs 2024-07-26T11:37:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:46 smithi189 ceph-mon[36790]: pgmap v1176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:37:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 624.472215 secs 2024-07-26T11:37:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:46 smithi193 ceph-mon[37473]: pgmap v1176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:37:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 624.472215 secs 2024-07-26T11:37:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:46 smithi134 ceph-mon[31966]: pgmap v1176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:37:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 624.472215 secs 2024-07-26T11:37:48.250 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:37:48.248+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:37:48.250 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:37:48.248+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:37:48.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:37:48.248+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:48 smithi189 ceph-mon[36790]: pgmap v1177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:48 smithi193 ceph-mon[37473]: pgmap v1177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:48 smithi134 ceph-mon[31966]: pgmap v1177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:37:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 610.046794 secs 2024-07-26T11:37:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 610.046794 secs 2024-07-26T11:37:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 610.046794 secs 2024-07-26T11:37:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:50 smithi189 ceph-mon[36790]: pgmap v1178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:37:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:50 smithi193 ceph-mon[37473]: pgmap v1178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:37:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:50 smithi134 ceph-mon[31966]: pgmap v1178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:37:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 629.472343 secs 2024-07-26T11:37:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 629.472343 secs 2024-07-26T11:37:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 629.472343 secs 2024-07-26T11:37:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:52 smithi193 ceph-mon[37473]: pgmap v1179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:37:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:52 smithi134 ceph-mon[31966]: pgmap v1179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:37:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:52 smithi189 ceph-mon[36790]: pgmap v1179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:37:54.496 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:54 smithi189 ceph-mon[36790]: pgmap v1180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:37:54.496 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 615.046931 secs 2024-07-26T11:37:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:54 smithi193 ceph-mon[37473]: pgmap v1180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:37:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 615.046931 secs 2024-07-26T11:37:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:54 smithi134 ceph-mon[31966]: pgmap v1180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:37:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 615.046931 secs 2024-07-26T11:37:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:56 smithi193 ceph-mon[37473]: pgmap v1181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 634.472466 secs 2024-07-26T11:37:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:56 smithi134 ceph-mon[31966]: pgmap v1181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 634.472466 secs 2024-07-26T11:37:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:56 smithi189 ceph-mon[36790]: pgmap v1181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 634.472466 secs 2024-07-26T11:37:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:58 smithi193 ceph-mon[37473]: pgmap v1182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:58 smithi134 ceph-mon[31966]: pgmap v1182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:58 smithi189 ceph-mon[36790]: pgmap v1182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:37:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:37:58.760+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:37:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:37:58.760+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:37:58.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:37:58.760+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:37:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:37:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 620.047040 secs 2024-07-26T11:37:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:37:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 620.047040 secs 2024-07-26T11:37:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:37:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 620.047040 secs 2024-07-26T11:38:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:00 smithi193 ceph-mon[37473]: pgmap v1183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:00 smithi134 ceph-mon[31966]: pgmap v1183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:00 smithi189 ceph-mon[36790]: pgmap v1183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 639.472564 secs 2024-07-26T11:38:01.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 639.472564 secs 2024-07-26T11:38:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 639.472564 secs 2024-07-26T11:38:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:02 smithi193 ceph-mon[37473]: pgmap v1184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:02 smithi134 ceph-mon[31966]: pgmap v1184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:02 smithi189 ceph-mon[36790]: pgmap v1184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:03.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:38:03.258+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:38:03.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:38:03.258+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:38:03.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:38:03.258+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:04 smithi193 ceph-mon[37473]: pgmap v1185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 625.047159 secs 2024-07-26T11:38:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:04 smithi134 ceph-mon[31966]: pgmap v1185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 625.047159 secs 2024-07-26T11:38:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:04 smithi189 ceph-mon[36790]: pgmap v1185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 625.047159 secs 2024-07-26T11:38:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:06 smithi193 ceph-mon[37473]: pgmap v1186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:38:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 644.472693 secs 2024-07-26T11:38:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:06 smithi134 ceph-mon[31966]: pgmap v1186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:38:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 644.472693 secs 2024-07-26T11:38:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:06 smithi189 ceph-mon[36790]: pgmap v1186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:38:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 644.472693 secs 2024-07-26T11:38:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:08 smithi134 ceph-mon[31966]: pgmap v1187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:38:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:08 smithi189 ceph-mon[36790]: pgmap v1187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:38:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:08 smithi193 ceph-mon[37473]: pgmap v1187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:38:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 630.047280 secs 2024-07-26T11:38:09.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 630.047280 secs 2024-07-26T11:38:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 630.047280 secs 2024-07-26T11:38:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:10 smithi134 ceph-mon[31966]: pgmap v1188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:10 smithi189 ceph-mon[36790]: pgmap v1188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:10 smithi193 ceph-mon[37473]: pgmap v1188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 649.472763 secs 2024-07-26T11:38:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 649.472763 secs 2024-07-26T11:38:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 649.472763 secs 2024-07-26T11:38:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:12 smithi134 ceph-mon[31966]: pgmap v1189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:12 smithi189 ceph-mon[36790]: pgmap v1189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:12 smithi193 ceph-mon[37473]: pgmap v1189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:38:13.775+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:38:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:38:13.775+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:13.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:38:13.775+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:38:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:14 smithi134 ceph-mon[31966]: pgmap v1190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 635.047407 secs 2024-07-26T11:38:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:14 smithi189 ceph-mon[36790]: pgmap v1190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 635.047407 secs 2024-07-26T11:38:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:14 smithi193 ceph-mon[37473]: pgmap v1190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 635.047407 secs 2024-07-26T11:38:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:16 smithi134 ceph-mon[31966]: pgmap v1191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:38:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 654.472894 secs 2024-07-26T11:38:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:16 smithi189 ceph-mon[36790]: pgmap v1191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:38:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 654.472894 secs 2024-07-26T11:38:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:16 smithi193 ceph-mon[37473]: pgmap v1191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:38:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 654.472894 secs 2024-07-26T11:38:18.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:38:18.273+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:38:18.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:38:18.273+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:18.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:38:18.273+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:38:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:18 smithi134 ceph-mon[31966]: pgmap v1192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:38:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:18 smithi189 ceph-mon[36790]: pgmap v1192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:38:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:18 smithi193 ceph-mon[37473]: pgmap v1192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:38:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 640.047527 secs 2024-07-26T11:38:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 640.047527 secs 2024-07-26T11:38:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 640.047527 secs 2024-07-26T11:38:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:20 smithi134 ceph-mon[31966]: pgmap v1193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:20 smithi189 ceph-mon[36790]: pgmap v1193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:20 smithi193 ceph-mon[37473]: pgmap v1193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 659.473024 secs 2024-07-26T11:38:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 659.473024 secs 2024-07-26T11:38:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 659.473024 secs 2024-07-26T11:38:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:22 smithi134 ceph-mon[31966]: pgmap v1194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:22 smithi189 ceph-mon[36790]: pgmap v1194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:22 smithi193 ceph-mon[37473]: pgmap v1194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:24 smithi134 ceph-mon[31966]: pgmap v1195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:38:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 645.047660 secs 2024-07-26T11:38:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:24 smithi189 ceph-mon[36790]: pgmap v1195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:38:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 645.047660 secs 2024-07-26T11:38:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:24 smithi193 ceph-mon[37473]: pgmap v1195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:38:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 645.047660 secs 2024-07-26T11:38:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:26 smithi134 ceph-mon[31966]: pgmap v1196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:38:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 664.473123 secs 2024-07-26T11:38:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:26 smithi189 ceph-mon[36790]: pgmap v1196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:38:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 664.473123 secs 2024-07-26T11:38:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:26 smithi193 ceph-mon[37473]: pgmap v1196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:38:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 664.473123 secs 2024-07-26T11:38:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:28 smithi134 ceph-mon[31966]: pgmap v1197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:38:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:28 smithi189 ceph-mon[36790]: pgmap v1197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:38:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:28 smithi193 ceph-mon[37473]: pgmap v1197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:38:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:38:28.791+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:38:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:38:28.791+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:28.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:38:28.791+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:38:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 650.047788 secs 2024-07-26T11:38:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 650.047788 secs 2024-07-26T11:38:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 650.047788 secs 2024-07-26T11:38:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:30 smithi134 ceph-mon[31966]: pgmap v1198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:30 smithi189 ceph-mon[36790]: pgmap v1198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:30 smithi193 ceph-mon[37473]: pgmap v1198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:38:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 669.473241 secs 2024-07-26T11:38:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 669.473241 secs 2024-07-26T11:38:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 669.473241 secs 2024-07-26T11:38:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:32 smithi134 ceph-mon[31966]: pgmap v1199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:32 smithi189 ceph-mon[36790]: pgmap v1199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:32 smithi193 ceph-mon[37473]: pgmap v1199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:33.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:38:33.279+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:38:33.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:38:33.279+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:33.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:38:33.279+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:38:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:34 smithi134 ceph-mon[31966]: pgmap v1200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:38:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 655.047863 secs 2024-07-26T11:38:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:34 smithi189 ceph-mon[36790]: pgmap v1200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:38:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 655.047863 secs 2024-07-26T11:38:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:34 smithi193 ceph-mon[37473]: pgmap v1200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:38:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 655.047863 secs 2024-07-26T11:38:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:36 smithi134 ceph-mon[31966]: pgmap v1201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 674.473402 secs 2024-07-26T11:38:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:36 smithi189 ceph-mon[36790]: pgmap v1201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 674.473402 secs 2024-07-26T11:38:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:36 smithi193 ceph-mon[37473]: pgmap v1201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 674.473402 secs 2024-07-26T11:38:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:38 smithi134 ceph-mon[31966]: pgmap v1202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:38:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:38 smithi189 ceph-mon[36790]: pgmap v1202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:38:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:38 smithi193 ceph-mon[37473]: pgmap v1202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:38:39.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 660.047961 secs 2024-07-26T11:38:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 660.047961 secs 2024-07-26T11:38:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 660.047961 secs 2024-07-26T11:38:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:40 smithi189 ceph-mon[36790]: pgmap v1203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:38:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:40 smithi193 ceph-mon[37473]: pgmap v1203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:38:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:40 smithi134 ceph-mon[31966]: pgmap v1203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:38:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 679.473532 secs 2024-07-26T11:38:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:38:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 679.473532 secs 2024-07-26T11:38:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:38:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 679.473532 secs 2024-07-26T11:38:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:38:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:42 smithi189 ceph-mon[36790]: pgmap v1204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:42 smithi193 ceph-mon[37473]: pgmap v1204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:42 smithi134 ceph-mon[31966]: pgmap v1204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:38:43.806+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:38:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:38:43.806+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:38:43.806+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:38:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:44 smithi134 ceph-mon[31966]: pgmap v1205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:38:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 665.048031 secs 2024-07-26T11:38:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:44 smithi193 ceph-mon[37473]: pgmap v1205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:38:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 665.048031 secs 2024-07-26T11:38:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:44 smithi189 ceph-mon[36790]: pgmap v1205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:38:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 665.048031 secs 2024-07-26T11:38:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:46 smithi193 ceph-mon[37473]: pgmap v1206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 684.473666 secs 2024-07-26T11:38:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:46 smithi134 ceph-mon[31966]: pgmap v1206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:46.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 684.473666 secs 2024-07-26T11:38:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:46 smithi189 ceph-mon[36790]: pgmap v1206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:38:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 684.473666 secs 2024-07-26T11:38:48.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:38:48.294+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:38:48.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:38:48.295+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:38:48.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:38:48.295+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:48.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:48 smithi134 ceph-mon[31966]: pgmap v1207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:38:48.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 670.048163 secs 2024-07-26T11:38:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:48 smithi193 ceph-mon[37473]: pgmap v1207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:38:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 670.048163 secs 2024-07-26T11:38:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:48 smithi189 ceph-mon[36790]: pgmap v1207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:38:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 670.048163 secs 2024-07-26T11:38:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:50 smithi193 ceph-mon[37473]: pgmap v1208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:38:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:38:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:38:50.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:38:50.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:38:50.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:50 smithi193 ceph-mon[37473]: pgmap v1209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:38:50.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:50 smithi193 ceph-mon[37473]: pgmap v1210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:38:50.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:38:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:50 smithi134 ceph-mon[31966]: pgmap v1208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:38:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:38:50.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:38:50.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:38:50.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:38:50.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:50 smithi134 ceph-mon[31966]: pgmap v1209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:38:50.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:50 smithi134 ceph-mon[31966]: pgmap v1210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:38:50.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:38:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:50 smithi189 ceph-mon[36790]: pgmap v1208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:38:50.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:38:50.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:38:50.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:38:50.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:38:50.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:50 smithi189 ceph-mon[36790]: pgmap v1209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:38:50.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:50 smithi189 ceph-mon[36790]: pgmap v1210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:38:50.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:38:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 689.473795 secs 2024-07-26T11:38:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 689.473795 secs 2024-07-26T11:38:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 689.473795 secs 2024-07-26T11:38:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:52 smithi193 ceph-mon[37473]: pgmap v1211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-26T11:38:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:52 smithi134 ceph-mon[31966]: pgmap v1211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-26T11:38:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:52 smithi189 ceph-mon[36790]: pgmap v1211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.3 KiB/s rd, 6 op/s 2024-07-26T11:38:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 675.048296 secs 2024-07-26T11:38:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:54 smithi134 ceph-mon[31966]: pgmap v1212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:38:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 675.048296 secs 2024-07-26T11:38:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:54 smithi189 ceph-mon[36790]: pgmap v1212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:38:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 675.048296 secs 2024-07-26T11:38:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:54 smithi193 ceph-mon[37473]: pgmap v1212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:38:56.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:56 smithi134 ceph-mon[31966]: pgmap v1213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T11:38:56.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 694.473877 secs 2024-07-26T11:38:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:56 smithi189 ceph-mon[36790]: pgmap v1213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T11:38:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 694.473877 secs 2024-07-26T11:38:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:56 smithi193 ceph-mon[37473]: pgmap v1213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T11:38:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 694.473877 secs 2024-07-26T11:38:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:58 smithi134 ceph-mon[31966]: pgmap v1214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:38:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:58 smithi189 ceph-mon[36790]: pgmap v1214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:38:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:58 smithi193 ceph-mon[37473]: pgmap v1214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T11:38:58.817 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:38:58.815+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:38:58.817 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:38:58.815+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:38:58.817 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:38:58.815+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:38:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:38:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 680.048382 secs 2024-07-26T11:38:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:38:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 680.048382 secs 2024-07-26T11:38:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:38:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 680.048382 secs 2024-07-26T11:39:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:00 smithi134 ceph-mon[31966]: pgmap v1215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:39:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:00 smithi189 ceph-mon[36790]: pgmap v1215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:39:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:00 smithi193 ceph-mon[37473]: pgmap v1215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:39:01.609 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 699.474013 secs 2024-07-26T11:39:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 699.474013 secs 2024-07-26T11:39:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 699.474013 secs 2024-07-26T11:39:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:02 smithi134 ceph-mon[31966]: pgmap v1216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:39:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:02 smithi189 ceph-mon[36790]: pgmap v1216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:39:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:02 smithi193 ceph-mon[37473]: pgmap v1216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:39:03.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:39:03.303+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:39:03.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:39:03.303+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:03.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:39:03.303+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:39:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 685.048508 secs 2024-07-26T11:39:04.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:04 smithi134 ceph-mon[31966]: pgmap v1217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:39:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 685.048508 secs 2024-07-26T11:39:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:04 smithi189 ceph-mon[36790]: pgmap v1217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:39:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 685.048508 secs 2024-07-26T11:39:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:04 smithi193 ceph-mon[37473]: pgmap v1217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:39:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:06 smithi193 ceph-mon[37473]: pgmap v1218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:39:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 704.474132 secs 2024-07-26T11:39:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:06 smithi134 ceph-mon[31966]: pgmap v1218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:39:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 704.474132 secs 2024-07-26T11:39:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:06 smithi189 ceph-mon[36790]: pgmap v1218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:39:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 704.474132 secs 2024-07-26T11:39:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:08 smithi193 ceph-mon[37473]: pgmap v1219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:39:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 690.048652 secs 2024-07-26T11:39:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:08 smithi134 ceph-mon[31966]: pgmap v1219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:39:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 690.048652 secs 2024-07-26T11:39:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:08 smithi189 ceph-mon[36790]: pgmap v1219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:39:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 690.048652 secs 2024-07-26T11:39:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:10 smithi193 ceph-mon[37473]: pgmap v1220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:39:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:10 smithi134 ceph-mon[31966]: pgmap v1220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:39:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:10 smithi189 ceph-mon[36790]: pgmap v1220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:39:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 709.474282 secs 2024-07-26T11:39:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 709.474282 secs 2024-07-26T11:39:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 709.474282 secs 2024-07-26T11:39:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:12 smithi193 ceph-mon[37473]: pgmap v1221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:12 smithi134 ceph-mon[31966]: pgmap v1221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:12 smithi189 ceph-mon[36790]: pgmap v1221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 695.048788 secs 2024-07-26T11:39:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 695.048788 secs 2024-07-26T11:39:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:39:13.830+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:39:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:39:13.830+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:39:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:39:13.830+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 695.048788 secs 2024-07-26T11:39:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:14 smithi193 ceph-mon[37473]: pgmap v1222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:14 smithi134 ceph-mon[31966]: pgmap v1222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:14 smithi189 ceph-mon[36790]: pgmap v1222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:16 smithi193 ceph-mon[37473]: pgmap v1223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 714.474423 secs 2024-07-26T11:39:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:16 smithi134 ceph-mon[31966]: pgmap v1223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 714.474423 secs 2024-07-26T11:39:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:16 smithi189 ceph-mon[36790]: pgmap v1223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 714.474423 secs 2024-07-26T11:39:18.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:39:18.304+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:39:18.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:39:18.304+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:39:18.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:39:18.304+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:18 smithi134 ceph-mon[31966]: pgmap v1224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:39:18.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 700.048896 secs 2024-07-26T11:39:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:18 smithi193 ceph-mon[37473]: pgmap v1224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:39:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 700.048896 secs 2024-07-26T11:39:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:18 smithi189 ceph-mon[36790]: pgmap v1224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:39:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 700.048896 secs 2024-07-26T11:39:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:20 smithi193 ceph-mon[37473]: pgmap v1225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:39:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:20 smithi134 ceph-mon[31966]: pgmap v1225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:39:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:20 smithi189 ceph-mon[36790]: pgmap v1225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:39:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 719.474503 secs 2024-07-26T11:39:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 719.474503 secs 2024-07-26T11:39:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 719.474503 secs 2024-07-26T11:39:22.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:39:22.457+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006170 con 0x7f9b4c10f330 2024-07-26T11:39:22.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:39:22.457+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006170 con 0x7f9b4c10f330 2024-07-26T11:39:22.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:39:22.459+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20005ff0 con 0x55c65e5ee670 2024-07-26T11:39:22.462 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:39:22.460+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20005ff0 con 0x55c65e5ee670 2024-07-26T11:39:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:22 smithi134 ceph-mon[31966]: pgmap v1226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:22 smithi189 ceph-mon[36790]: pgmap v1226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:22 smithi193 ceph-mon[37473]: pgmap v1226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 705.049034 secs 2024-07-26T11:39:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 705.049034 secs 2024-07-26T11:39:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 705.049034 secs 2024-07-26T11:39:24.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:24 smithi189 ceph-mon[36790]: pgmap v1227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:24 smithi134 ceph-mon[31966]: pgmap v1227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:24 smithi193 ceph-mon[37473]: pgmap v1227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:26 smithi134 ceph-mon[31966]: pgmap v1228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:26 smithi134 ceph-mon[31966]: 1 slow requests, 1 included below; oldest blocked for > 724.474616 secs 2024-07-26T11:39:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:26 smithi134 ceph-mon[31966]: slow request 724.474615 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T11:39:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:26 smithi189 ceph-mon[36790]: pgmap v1228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:26 smithi189 ceph-mon[36790]: 1 slow requests, 1 included below; oldest blocked for > 724.474616 secs 2024-07-26T11:39:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:26 smithi189 ceph-mon[36790]: slow request 724.474615 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T11:39:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:26 smithi193 ceph-mon[37473]: pgmap v1228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:26 smithi193 ceph-mon[37473]: 1 slow requests, 1 included below; oldest blocked for > 724.474616 secs 2024-07-26T11:39:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:26 smithi193 ceph-mon[37473]: slow request 724.474615 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T11:39:28.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:39:28.846+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:39:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:39:28.846+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:39:28.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:39:28.846+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:28 smithi134 ceph-mon[31966]: pgmap v1229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:39:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 710.049152 secs 2024-07-26T11:39:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:28 smithi189 ceph-mon[36790]: pgmap v1229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:39:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 710.049152 secs 2024-07-26T11:39:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:28 smithi193 ceph-mon[37473]: pgmap v1229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:39:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 710.049152 secs 2024-07-26T11:39:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:30 smithi134 ceph-mon[31966]: pgmap v1230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:39:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:30 smithi189 ceph-mon[36790]: pgmap v1230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:39:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:30 smithi193 ceph-mon[37473]: pgmap v1230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:39:31.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 729.474730 secs 2024-07-26T11:39:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 729.474730 secs 2024-07-26T11:39:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 729.474730 secs 2024-07-26T11:39:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:32 smithi134 ceph-mon[31966]: pgmap v1231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:32 smithi189 ceph-mon[36790]: pgmap v1231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:39:32.955+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980060f0 con 0x7f21c8105550 2024-07-26T11:39:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:39:32.955+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006000 con 0x55a08e2d94d0 2024-07-26T11:39:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:39:32.955+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006000 con 0x55a08e2d94d0 2024-07-26T11:39:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:39:32.955+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980060f0 con 0x7f21c8105550 2024-07-26T11:39:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:32 smithi193 ceph-mon[37473]: pgmap v1231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:33.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:39:33.320+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:39:33.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:39:33.320+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:39:33.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:39:33.320+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 715.049279 secs 2024-07-26T11:39:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 715.049279 secs 2024-07-26T11:39:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 715.049279 secs 2024-07-26T11:39:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:34 smithi134 ceph-mon[31966]: pgmap v1232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:34 smithi189 ceph-mon[36790]: pgmap v1232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:34 smithi193 ceph-mon[37473]: pgmap v1232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:36 smithi134 ceph-mon[31966]: pgmap v1233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 734.474846 secs 2024-07-26T11:39:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:36 smithi189 ceph-mon[36790]: pgmap v1233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 734.474846 secs 2024-07-26T11:39:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:36 smithi193 ceph-mon[37473]: pgmap v1233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 734.474846 secs 2024-07-26T11:39:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:38 smithi134 ceph-mon[31966]: pgmap v1234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:39:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:38 smithi134 ceph-mon[31966]: 2 slow requests, 1 included below; oldest blocked for > 720.049395 secs 2024-07-26T11:39:38.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:38 smithi134 ceph-mon[31966]: slow request 720.049394 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T11:39:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:38 smithi189 ceph-mon[36790]: pgmap v1234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:39:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:38 smithi189 ceph-mon[36790]: 2 slow requests, 1 included below; oldest blocked for > 720.049395 secs 2024-07-26T11:39:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:38 smithi189 ceph-mon[36790]: slow request 720.049394 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T11:39:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:38 smithi193 ceph-mon[37473]: pgmap v1234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:39:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:38 smithi193 ceph-mon[37473]: 2 slow requests, 1 included below; oldest blocked for > 720.049395 secs 2024-07-26T11:39:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:38 smithi193 ceph-mon[37473]: slow request 720.049394 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T11:39:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:40 smithi134 ceph-mon[31966]: pgmap v1235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:39:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:40 smithi189 ceph-mon[36790]: pgmap v1235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:39:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:40 smithi193 ceph-mon[37473]: pgmap v1235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:39:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 739.474965 secs 2024-07-26T11:39:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 739.474965 secs 2024-07-26T11:39:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 739.474965 secs 2024-07-26T11:39:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:42 smithi134 ceph-mon[31966]: pgmap v1236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:42 smithi189 ceph-mon[36790]: pgmap v1236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:42 smithi193 ceph-mon[37473]: pgmap v1236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:39:43.861+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:39:43.863 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:39:43.861+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:43.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:39:43.861+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:39:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 725.049534 secs 2024-07-26T11:39:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 725.049534 secs 2024-07-26T11:39:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 725.049534 secs 2024-07-26T11:39:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:44 smithi134 ceph-mon[31966]: pgmap v1237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:44 smithi189 ceph-mon[36790]: pgmap v1237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:44 smithi193 ceph-mon[37473]: pgmap v1237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:46 smithi134 ceph-mon[31966]: pgmap v1238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 744.475061 secs 2024-07-26T11:39:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:46 smithi189 ceph-mon[36790]: pgmap v1238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 744.475061 secs 2024-07-26T11:39:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:46 smithi193 ceph-mon[37473]: pgmap v1238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 744.475061 secs 2024-07-26T11:39:48.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:39:48.335+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:39:48.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:39:48.335+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:39:48.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:39:48.335+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:48 smithi134 ceph-mon[31966]: pgmap v1239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:39:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 730.049671 secs 2024-07-26T11:39:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:48 smithi189 ceph-mon[36790]: pgmap v1239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:39:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 730.049671 secs 2024-07-26T11:39:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:48 smithi193 ceph-mon[37473]: pgmap v1239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:39:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 730.049671 secs 2024-07-26T11:39:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:39:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:39:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:39:50.700 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:50 smithi193 ceph-mon[37473]: pgmap v1240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:39:50.700 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:39:50.700 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:39:50.700 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:39:50.700 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:39:50.781 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:50 smithi189 ceph-mon[36790]: pgmap v1240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:39:50.781 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:39:50.781 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:39:50.781 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:39:50.782 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:39:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:50 smithi134 ceph-mon[31966]: pgmap v1240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:39:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:39:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:39:50.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:39:50.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:39:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:51 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T11:39:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:51 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T11:39:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 749.475171 secs 2024-07-26T11:39:51.784 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:51 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T11:39:51.784 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:51 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T11:39:51.784 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 749.475171 secs 2024-07-26T11:39:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:51 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T11:39:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:51 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T11:39:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 749.475171 secs 2024-07-26T11:39:52.928 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:52 smithi193 ceph-mon[37473]: pgmap v1241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:52.933 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:52 smithi189 ceph-mon[36790]: pgmap v1241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:52 smithi134 ceph-mon[31966]: pgmap v1241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:39:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 735.049741 secs 2024-07-26T11:39:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:39:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:39:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:39:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 735.049741 secs 2024-07-26T11:39:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:39:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:39:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:39:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 735.049741 secs 2024-07-26T11:39:54.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:39:54.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:39:54.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:39:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:54 smithi193 ceph-mon[37473]: pgmap v1242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:39:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:39:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:39:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:54 smithi193 ceph-mon[37473]: pgmap v1243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:39:55.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:54 smithi193 ceph-mon[37473]: pgmap v1244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:39:55.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:39:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:54 smithi134 ceph-mon[31966]: pgmap v1242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:39:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:39:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:39:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:54 smithi134 ceph-mon[31966]: pgmap v1243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:39:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:54 smithi134 ceph-mon[31966]: pgmap v1244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:39:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:39:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:54 smithi189 ceph-mon[36790]: pgmap v1242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:39:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:39:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:39:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:39:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:54 smithi189 ceph-mon[36790]: pgmap v1243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:39:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:54 smithi189 ceph-mon[36790]: pgmap v1244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:39:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:39:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:56 smithi193 ceph-mon[37473]: pgmap v1245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 754.475334 secs 2024-07-26T11:39:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:56 smithi134 ceph-mon[31966]: pgmap v1245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 754.475334 secs 2024-07-26T11:39:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:56 smithi189 ceph-mon[36790]: pgmap v1245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 754.475334 secs 2024-07-26T11:39:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:39:58.864+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:39:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:39:58.864+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:39:58.866 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:39:58.864+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:39:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:58 smithi193 ceph-mon[37473]: pgmap v1246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:39:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 740.049864 secs 2024-07-26T11:39:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:58 smithi134 ceph-mon[31966]: pgmap v1246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:39:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 740.049864 secs 2024-07-26T11:39:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:58 smithi189 ceph-mon[36790]: pgmap v1246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:39:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:39:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 740.049864 secs 2024-07-26T11:40:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:00 smithi134 ceph-mon[31966]: pgmap v1247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:40:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:00 smithi134 ceph-mon[31966]: Health detail: HEALTH_WARN 3 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T11:40:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:00 smithi134 ceph-mon[31966]: [WRN] CEPHADM_FAILED_DAEMON: 3 failed cephadm daemon(s) 2024-07-26T11:40:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:00 smithi134 ceph-mon[31966]: daemon mds.g on smithi134 is in error state 2024-07-26T11:40:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:00 smithi134 ceph-mon[31966]: daemon mds.a on smithi134 is in error state 2024-07-26T11:40:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:00 smithi134 ceph-mon[31966]: daemon mds.e on smithi189 is in error state 2024-07-26T11:40:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:00 smithi134 ceph-mon[31966]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-07-26T11:40:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:00 smithi134 ceph-mon[31966]: mds.f(mds.0): 2 slow requests are blocked > 180 secs 2024-07-26T11:40:01.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:00 smithi134 ceph-mon[31966]: mds.c(mds.3): 1 slow requests are blocked > 180 secs 2024-07-26T11:40:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:00 smithi189 ceph-mon[36790]: pgmap v1247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:40:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:00 smithi189 ceph-mon[36790]: Health detail: HEALTH_WARN 3 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T11:40:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:00 smithi189 ceph-mon[36790]: [WRN] CEPHADM_FAILED_DAEMON: 3 failed cephadm daemon(s) 2024-07-26T11:40:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:00 smithi189 ceph-mon[36790]: daemon mds.g on smithi134 is in error state 2024-07-26T11:40:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:00 smithi189 ceph-mon[36790]: daemon mds.a on smithi134 is in error state 2024-07-26T11:40:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:00 smithi189 ceph-mon[36790]: daemon mds.e on smithi189 is in error state 2024-07-26T11:40:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:00 smithi189 ceph-mon[36790]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-07-26T11:40:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:00 smithi189 ceph-mon[36790]: mds.f(mds.0): 2 slow requests are blocked > 180 secs 2024-07-26T11:40:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:00 smithi189 ceph-mon[36790]: mds.c(mds.3): 1 slow requests are blocked > 180 secs 2024-07-26T11:40:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:00 smithi193 ceph-mon[37473]: pgmap v1247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:40:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:00 smithi193 ceph-mon[37473]: Health detail: HEALTH_WARN 3 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T11:40:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:00 smithi193 ceph-mon[37473]: [WRN] CEPHADM_FAILED_DAEMON: 3 failed cephadm daemon(s) 2024-07-26T11:40:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:00 smithi193 ceph-mon[37473]: daemon mds.g on smithi134 is in error state 2024-07-26T11:40:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:00 smithi193 ceph-mon[37473]: daemon mds.a on smithi134 is in error state 2024-07-26T11:40:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:00 smithi193 ceph-mon[37473]: daemon mds.e on smithi189 is in error state 2024-07-26T11:40:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:00 smithi193 ceph-mon[37473]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-07-26T11:40:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:00 smithi193 ceph-mon[37473]: mds.f(mds.0): 2 slow requests are blocked > 180 secs 2024-07-26T11:40:01.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:00 smithi193 ceph-mon[37473]: mds.c(mds.3): 1 slow requests are blocked > 180 secs 2024-07-26T11:40:02.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 759.475403 secs 2024-07-26T11:40:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 759.475403 secs 2024-07-26T11:40:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 759.475403 secs 2024-07-26T11:40:03.108 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:02 smithi134 ceph-mon[31966]: pgmap v1248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T11:40:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:02 smithi189 ceph-mon[36790]: pgmap v1248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T11:40:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:02 smithi193 ceph-mon[37473]: pgmap v1248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T11:40:03.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:40:03.337+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:40:03.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:40:03.337+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:03.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:40:03.337+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:40:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:03 smithi193 ceph-mon[37473]: 2 slow requests, 1 included below; oldest blocked for > 745.050005 secs 2024-07-26T11:40:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:03 smithi193 ceph-mon[37473]: slow request 722.304721 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:40:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:03 smithi134 ceph-mon[31966]: 2 slow requests, 1 included below; oldest blocked for > 745.050005 secs 2024-07-26T11:40:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:03 smithi134 ceph-mon[31966]: slow request 722.304721 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:40:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:03 smithi189 ceph-mon[36790]: 2 slow requests, 1 included below; oldest blocked for > 745.050005 secs 2024-07-26T11:40:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:03 smithi189 ceph-mon[36790]: slow request 722.304721 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:40:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:04 smithi134 ceph-mon[31966]: pgmap v1249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:40:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:04 smithi189 ceph-mon[36790]: pgmap v1249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:40:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:04 smithi193 ceph-mon[37473]: pgmap v1249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:40:07.177 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:06 smithi134 ceph-mon[31966]: pgmap v1250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:07.177 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 764.475467 secs 2024-07-26T11:40:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:06 smithi189 ceph-mon[36790]: pgmap v1250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 764.475467 secs 2024-07-26T11:40:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:06 smithi193 ceph-mon[37473]: pgmap v1250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 764.475467 secs 2024-07-26T11:40:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:08 smithi134 ceph-mon[31966]: pgmap v1251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 750.050121 secs 2024-07-26T11:40:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:08 smithi189 ceph-mon[36790]: pgmap v1251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 750.050121 secs 2024-07-26T11:40:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:08 smithi193 ceph-mon[37473]: pgmap v1251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 750.050121 secs 2024-07-26T11:40:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:10 smithi193 ceph-mon[37473]: pgmap v1252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:10 smithi134 ceph-mon[31966]: pgmap v1252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:10 smithi189 ceph-mon[36790]: pgmap v1252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 769.475540 secs 2024-07-26T11:40:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 769.475540 secs 2024-07-26T11:40:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 769.475540 secs 2024-07-26T11:40:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:12 smithi193 ceph-mon[37473]: pgmap v1253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:12 smithi134 ceph-mon[31966]: pgmap v1253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:12 smithi189 ceph-mon[36790]: pgmap v1253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 755.050230 secs 2024-07-26T11:40:13.869 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:40:13.868+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:40:13.869 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:40:13.868+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:40:13.869 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:40:13.868+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 755.050230 secs 2024-07-26T11:40:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 755.050230 secs 2024-07-26T11:40:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:14 smithi134 ceph-mon[31966]: pgmap v1254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:14 smithi189 ceph-mon[36790]: pgmap v1254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:14 smithi193 ceph-mon[37473]: pgmap v1254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:16 smithi134 ceph-mon[31966]: pgmap v1255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 774.475654 secs 2024-07-26T11:40:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:16 smithi189 ceph-mon[36790]: pgmap v1255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 774.475654 secs 2024-07-26T11:40:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:16 smithi193 ceph-mon[37473]: pgmap v1255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 774.475654 secs 2024-07-26T11:40:18.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:40:18.353+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:40:18.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:40:18.353+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:40:18.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:40:18.353+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:18 smithi134 ceph-mon[31966]: pgmap v1256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 760.050343 secs 2024-07-26T11:40:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:18 smithi189 ceph-mon[36790]: pgmap v1256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 760.050343 secs 2024-07-26T11:40:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:18 smithi193 ceph-mon[37473]: pgmap v1256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 760.050343 secs 2024-07-26T11:40:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:20 smithi134 ceph-mon[31966]: pgmap v1257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:20 smithi189 ceph-mon[36790]: pgmap v1257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:20 smithi193 ceph-mon[37473]: pgmap v1257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 779.475777 secs 2024-07-26T11:40:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 779.475777 secs 2024-07-26T11:40:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 779.475777 secs 2024-07-26T11:40:23.056 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:22 smithi134 ceph-mon[31966]: pgmap v1258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:22 smithi189 ceph-mon[36790]: pgmap v1258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:22 smithi193 ceph-mon[37473]: pgmap v1258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:24.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 765.050403 secs 2024-07-26T11:40:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 765.050403 secs 2024-07-26T11:40:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 765.050403 secs 2024-07-26T11:40:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:24 smithi134 ceph-mon[31966]: pgmap v1259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:24 smithi189 ceph-mon[36790]: pgmap v1259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:24 smithi193 ceph-mon[37473]: pgmap v1259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:26 smithi134 ceph-mon[31966]: pgmap v1260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 784.475914 secs 2024-07-26T11:40:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:26 smithi189 ceph-mon[36790]: pgmap v1260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 784.475914 secs 2024-07-26T11:40:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:26 smithi193 ceph-mon[37473]: pgmap v1260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 784.475914 secs 2024-07-26T11:40:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:40:28.883+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:40:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:40:28.883+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:40:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:40:28.883+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:28 smithi134 ceph-mon[31966]: pgmap v1261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 770.050557 secs 2024-07-26T11:40:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:28 smithi189 ceph-mon[36790]: pgmap v1261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 770.050557 secs 2024-07-26T11:40:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:28 smithi193 ceph-mon[37473]: pgmap v1261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 770.050557 secs 2024-07-26T11:40:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:30 smithi134 ceph-mon[31966]: pgmap v1262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:30 smithi189 ceph-mon[36790]: pgmap v1262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:30 smithi193 ceph-mon[37473]: pgmap v1262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 789.476047 secs 2024-07-26T11:40:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 789.476047 secs 2024-07-26T11:40:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 789.476047 secs 2024-07-26T11:40:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:32 smithi134 ceph-mon[31966]: pgmap v1263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:32 smithi189 ceph-mon[36790]: pgmap v1263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:32 smithi193 ceph-mon[37473]: pgmap v1263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:33.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:40:33.368+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:40:33.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:40:33.368+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:40:33.371 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:40:33.368+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 775.050712 secs 2024-07-26T11:40:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 775.050712 secs 2024-07-26T11:40:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 775.050712 secs 2024-07-26T11:40:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:34 smithi134 ceph-mon[31966]: pgmap v1264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:40:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:34 smithi189 ceph-mon[36790]: pgmap v1264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:40:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:34 smithi193 ceph-mon[37473]: pgmap v1264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:40:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:36 smithi134 ceph-mon[31966]: pgmap v1265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:40:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 794.476180 secs 2024-07-26T11:40:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:36 smithi189 ceph-mon[36790]: pgmap v1265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:40:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 794.476180 secs 2024-07-26T11:40:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:36 smithi193 ceph-mon[37473]: pgmap v1265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:40:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 794.476180 secs 2024-07-26T11:40:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:38 smithi134 ceph-mon[31966]: pgmap v1266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:40:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 780.050869 secs 2024-07-26T11:40:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:38 smithi189 ceph-mon[36790]: pgmap v1266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:40:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 780.050869 secs 2024-07-26T11:40:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:38 smithi193 ceph-mon[37473]: pgmap v1266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:40:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 780.050869 secs 2024-07-26T11:40:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:40 smithi134 ceph-mon[31966]: pgmap v1267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:40:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:40 smithi189 ceph-mon[36790]: pgmap v1267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:40:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:40 smithi193 ceph-mon[37473]: pgmap v1267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:40:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 799.476279 secs 2024-07-26T11:40:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 799.476279 secs 2024-07-26T11:40:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 799.476279 secs 2024-07-26T11:40:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:42 smithi134 ceph-mon[31966]: pgmap v1268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:40:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:42 smithi189 ceph-mon[36790]: pgmap v1268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:40:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:42 smithi193 ceph-mon[37473]: pgmap v1268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:40:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:40:43.898+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:40:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:40:43.898+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:40:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:40:43.898+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 785.050989 secs 2024-07-26T11:40:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 785.050989 secs 2024-07-26T11:40:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 785.050989 secs 2024-07-26T11:40:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:44 smithi134 ceph-mon[31966]: pgmap v1269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:40:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:44 smithi189 ceph-mon[36790]: pgmap v1269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:40:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:44 smithi193 ceph-mon[37473]: pgmap v1269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:40:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:46 smithi189 ceph-mon[36790]: pgmap v1270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 804.476415 secs 2024-07-26T11:40:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:46 smithi193 ceph-mon[37473]: pgmap v1270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 804.476415 secs 2024-07-26T11:40:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:46 smithi134 ceph-mon[31966]: pgmap v1270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 804.476415 secs 2024-07-26T11:40:48.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:40:48.373+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:40:48.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:40:48.373+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:48.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:40:48.373+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:40:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:48 smithi189 ceph-mon[36790]: pgmap v1271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 790.051112 secs 2024-07-26T11:40:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:48 smithi193 ceph-mon[37473]: pgmap v1271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 790.051112 secs 2024-07-26T11:40:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:48 smithi134 ceph-mon[31966]: pgmap v1271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:40:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 790.051112 secs 2024-07-26T11:40:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:50 smithi189 ceph-mon[36790]: pgmap v1272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:50 smithi193 ceph-mon[37473]: pgmap v1272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:50 smithi134 ceph-mon[31966]: pgmap v1272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:40:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 809.476529 secs 2024-07-26T11:40:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 809.476529 secs 2024-07-26T11:40:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 809.476529 secs 2024-07-26T11:40:53.257 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:52 smithi134 ceph-mon[31966]: pgmap v1273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:52 smithi193 ceph-mon[37473]: pgmap v1273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:52 smithi189 ceph-mon[36790]: pgmap v1273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:40:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 795.051221 secs 2024-07-26T11:40:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:40:54.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 795.051221 secs 2024-07-26T11:40:54.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:40:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 795.051221 secs 2024-07-26T11:40:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:40:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:54 smithi193 ceph-mon[37473]: pgmap v1274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:40:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:40:55.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:40:55.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:54 smithi134 ceph-mon[31966]: pgmap v1274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:55.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:40:55.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:40:55.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:40:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:54 smithi189 ceph-mon[36790]: pgmap v1274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:40:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:40:55.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:40:55.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:40:55.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:40:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:55 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T11:40:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:55 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T11:40:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:55 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T11:40:56.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:55 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T11:40:56.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:55 smithi193 ceph-mon[37473]: pgmap v1275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:40:56.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:55 smithi193 ceph-mon[37473]: pgmap v1276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:40:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:55 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T11:40:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:55 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T11:40:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:55 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T11:40:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:55 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T11:40:56.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:55 smithi134 ceph-mon[31966]: pgmap v1275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:40:56.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:55 smithi134 ceph-mon[31966]: pgmap v1276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:40:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:55 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T11:40:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:55 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T11:40:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:55 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T11:40:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:55 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T11:40:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:55 smithi189 ceph-mon[36790]: pgmap v1275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:40:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:55 smithi189 ceph-mon[36790]: pgmap v1276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:40:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 814.476593 secs 2024-07-26T11:40:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 814.476593 secs 2024-07-26T11:40:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 814.476593 secs 2024-07-26T11:40:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:57 smithi193 ceph-mon[37473]: pgmap v1277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:40:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:57 smithi134 ceph-mon[31966]: pgmap v1277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:40:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:57 smithi189 ceph-mon[36790]: pgmap v1277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:40:58.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:40:58.915+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:40:58.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:40:58.915+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:40:58.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:40:58.915+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:40:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:40:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 800.051298 secs 2024-07-26T11:40:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:40:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 800.051298 secs 2024-07-26T11:40:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:40:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 800.051298 secs 2024-07-26T11:41:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:00 smithi193 ceph-mon[37473]: pgmap v1278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:41:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:00 smithi134 ceph-mon[31966]: pgmap v1278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:41:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:00 smithi189 ceph-mon[36790]: pgmap v1278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:41:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 819.476717 secs 2024-07-26T11:41:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:02 smithi193 ceph-mon[37473]: pgmap v1279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:41:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 819.476717 secs 2024-07-26T11:41:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:02 smithi134 ceph-mon[31966]: pgmap v1279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:41:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 819.476717 secs 2024-07-26T11:41:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:02 smithi189 ceph-mon[36790]: pgmap v1279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:41:03.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:41:03.389+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:41:03.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:41:03.389+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:41:03.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:41:03.389+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:04 smithi193 ceph-mon[37473]: pgmap v1280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:41:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 805.051378 secs 2024-07-26T11:41:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:04 smithi134 ceph-mon[31966]: pgmap v1280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:41:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 805.051378 secs 2024-07-26T11:41:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:04 smithi189 ceph-mon[36790]: pgmap v1280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:41:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 805.051378 secs 2024-07-26T11:41:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:06 smithi193 ceph-mon[37473]: pgmap v1281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:41:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:06 smithi134 ceph-mon[31966]: pgmap v1281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:41:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:06 smithi189 ceph-mon[36790]: pgmap v1281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:41:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 824.476854 secs 2024-07-26T11:41:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 824.476854 secs 2024-07-26T11:41:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 824.476854 secs 2024-07-26T11:41:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:08 smithi134 ceph-mon[31966]: pgmap v1282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:08 smithi189 ceph-mon[36790]: pgmap v1282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:08 smithi193 ceph-mon[37473]: pgmap v1282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:09.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 810.051516 secs 2024-07-26T11:41:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 810.051516 secs 2024-07-26T11:41:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 810.051516 secs 2024-07-26T11:41:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:10 smithi134 ceph-mon[31966]: pgmap v1283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:10 smithi189 ceph-mon[36790]: pgmap v1283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:10 smithi193 ceph-mon[37473]: pgmap v1283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 829.476931 secs 2024-07-26T11:41:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:11 smithi134 ceph-mon[31966]: pgmap v1284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:11.424 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 829.476931 secs 2024-07-26T11:41:11.424 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:11 smithi193 ceph-mon[37473]: pgmap v1284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 829.476931 secs 2024-07-26T11:41:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:11 smithi189 ceph-mon[36790]: pgmap v1284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:41:13.928+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:41:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:41:13.928+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:13.930 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:41:13.928+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:41:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:13 smithi193 ceph-mon[37473]: pgmap v1285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 815.051634 secs 2024-07-26T11:41:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:13 smithi134 ceph-mon[31966]: pgmap v1285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 815.051634 secs 2024-07-26T11:41:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:13 smithi189 ceph-mon[36790]: pgmap v1285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 815.051634 secs 2024-07-26T11:41:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:15 smithi193 ceph-mon[37473]: pgmap v1286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:15 smithi134 ceph-mon[31966]: pgmap v1286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:15 smithi189 ceph-mon[36790]: pgmap v1286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 834.477048 secs 2024-07-26T11:41:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 834.477048 secs 2024-07-26T11:41:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 834.477048 secs 2024-07-26T11:41:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:18 smithi193 ceph-mon[37473]: pgmap v1287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:18.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:41:18.404+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:41:18.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:41:18.404+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:41:18.407 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:41:18.404+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:18.407 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:18 smithi134 ceph-mon[31966]: pgmap v1287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:18 smithi189 ceph-mon[36790]: pgmap v1287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:19.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 820.051707 secs 2024-07-26T11:41:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 820.051707 secs 2024-07-26T11:41:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 820.051707 secs 2024-07-26T11:41:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:20 smithi134 ceph-mon[31966]: pgmap v1288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:20 smithi189 ceph-mon[36790]: pgmap v1288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:20 smithi193 ceph-mon[37473]: pgmap v1288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 839.477167 secs 2024-07-26T11:41:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 839.477167 secs 2024-07-26T11:41:21.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 839.477167 secs 2024-07-26T11:41:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:22 smithi134 ceph-mon[31966]: pgmap v1289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:22 smithi189 ceph-mon[36790]: pgmap v1289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:22 smithi193 ceph-mon[37473]: pgmap v1289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:24 smithi134 ceph-mon[31966]: pgmap v1290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 825.051843 secs 2024-07-26T11:41:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:24 smithi189 ceph-mon[36790]: pgmap v1290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 825.051843 secs 2024-07-26T11:41:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:24 smithi193 ceph-mon[37473]: pgmap v1290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 825.051843 secs 2024-07-26T11:41:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:26 smithi134 ceph-mon[31966]: pgmap v1291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 844.477330 secs 2024-07-26T11:41:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:26 smithi189 ceph-mon[36790]: pgmap v1291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 844.477330 secs 2024-07-26T11:41:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:26 smithi193 ceph-mon[37473]: pgmap v1291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 844.477330 secs 2024-07-26T11:41:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:28 smithi134 ceph-mon[31966]: pgmap v1292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:28 smithi189 ceph-mon[36790]: pgmap v1292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:28 smithi193 ceph-mon[37473]: pgmap v1292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:28.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:41:28.944+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:41:28.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:41:28.944+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:41:28.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:41:28.944+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 830.052000 secs 2024-07-26T11:41:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 830.052000 secs 2024-07-26T11:41:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 830.052000 secs 2024-07-26T11:41:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:30 smithi134 ceph-mon[31966]: pgmap v1293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:30 smithi189 ceph-mon[36790]: pgmap v1293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:30.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:30 smithi193 ceph-mon[37473]: pgmap v1293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 849.477464 secs 2024-07-26T11:41:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:31 smithi134 ceph-mon[31966]: pgmap v1294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 849.477464 secs 2024-07-26T11:41:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:31 smithi189 ceph-mon[36790]: pgmap v1294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 849.477464 secs 2024-07-26T11:41:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:31 smithi193 ceph-mon[37473]: pgmap v1294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:33.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:41:33.417+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:41:33.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:41:33.417+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:33.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:41:33.417+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:41:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:34 smithi193 ceph-mon[37473]: pgmap v1295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 835.052125 secs 2024-07-26T11:41:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:34 smithi134 ceph-mon[31966]: pgmap v1295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 835.052125 secs 2024-07-26T11:41:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:34 smithi189 ceph-mon[36790]: pgmap v1295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 835.052125 secs 2024-07-26T11:41:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:36 smithi193 ceph-mon[37473]: pgmap v1296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:36 smithi134 ceph-mon[31966]: pgmap v1296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:36 smithi189 ceph-mon[36790]: pgmap v1296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 854.477591 secs 2024-07-26T11:41:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 854.477591 secs 2024-07-26T11:41:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 854.477591 secs 2024-07-26T11:41:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:38 smithi134 ceph-mon[31966]: pgmap v1297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:41:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:38 smithi189 ceph-mon[36790]: pgmap v1297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:41:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:38 smithi193 ceph-mon[37473]: pgmap v1297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:41:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 840.052226 secs 2024-07-26T11:41:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 840.052226 secs 2024-07-26T11:41:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 840.052226 secs 2024-07-26T11:41:40.361 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:40 smithi134 ceph-mon[31966]: pgmap v1298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:40 smithi189 ceph-mon[36790]: pgmap v1298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:40 smithi193 ceph-mon[37473]: pgmap v1298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 859.477677 secs 2024-07-26T11:41:41.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 859.477677 secs 2024-07-26T11:41:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 859.477677 secs 2024-07-26T11:41:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:42 smithi134 ceph-mon[31966]: pgmap v1299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:42 smithi189 ceph-mon[36790]: pgmap v1299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:42 smithi193 ceph-mon[37473]: pgmap v1299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:43.961 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:41:43.959+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:41:43.961 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:41:43.959+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:43.961 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:41:43.959+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:41:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:44 smithi134 ceph-mon[31966]: pgmap v1300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 845.052289 secs 2024-07-26T11:41:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:44 smithi189 ceph-mon[36790]: pgmap v1300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 845.052289 secs 2024-07-26T11:41:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:44 smithi193 ceph-mon[37473]: pgmap v1300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 845.052289 secs 2024-07-26T11:41:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:46 smithi134 ceph-mon[31966]: pgmap v1301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:46.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 864.477818 secs 2024-07-26T11:41:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:46 smithi189 ceph-mon[36790]: pgmap v1301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 864.477818 secs 2024-07-26T11:41:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:46 smithi193 ceph-mon[37473]: pgmap v1301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 864.477818 secs 2024-07-26T11:41:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:48 smithi134 ceph-mon[31966]: pgmap v1302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:41:48.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:41:48.432+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:41:48.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:41:48.432+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:48.434 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:41:48.432+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:41:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:48 smithi189 ceph-mon[36790]: pgmap v1302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:41:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:48 smithi193 ceph-mon[37473]: pgmap v1302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:41:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 850.052357 secs 2024-07-26T11:41:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:49 smithi134 ceph-mon[31966]: pgmap v1303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 850.052357 secs 2024-07-26T11:41:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:49 smithi189 ceph-mon[36790]: pgmap v1303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:49.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 850.052357 secs 2024-07-26T11:41:49.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:49 smithi193 ceph-mon[37473]: pgmap v1303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:41:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 869.477925 secs 2024-07-26T11:41:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:52 smithi193 ceph-mon[37473]: pgmap v1304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 869.477925 secs 2024-07-26T11:41:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:52 smithi134 ceph-mon[31966]: pgmap v1304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 869.477925 secs 2024-07-26T11:41:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:52 smithi189 ceph-mon[36790]: pgmap v1304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:41:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:54 smithi193 ceph-mon[37473]: pgmap v1305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 855.052439 secs 2024-07-26T11:41:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:54 smithi134 ceph-mon[31966]: pgmap v1305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 855.052439 secs 2024-07-26T11:41:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:54 smithi189 ceph-mon[36790]: pgmap v1305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:41:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 855.052439 secs 2024-07-26T11:41:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:56 smithi134 ceph-mon[31966]: pgmap v1306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:41:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:56 smithi189 ceph-mon[36790]: pgmap v1306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:41:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:56 smithi193 ceph-mon[37473]: pgmap v1306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:41:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:41:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 874.477985 secs 2024-07-26T11:41:57.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:41:57.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:41:57.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:41:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 874.477985 secs 2024-07-26T11:41:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:41:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:41:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:41:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 874.477985 secs 2024-07-26T11:41:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:41:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:41:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:41:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:58 smithi134 ceph-mon[31966]: pgmap v1307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:41:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:58 smithi134 ceph-mon[31966]: pgmap v1308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:41:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:58 smithi189 ceph-mon[36790]: pgmap v1307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:41:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:58 smithi189 ceph-mon[36790]: pgmap v1308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:41:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:58 smithi193 ceph-mon[37473]: pgmap v1307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:41:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:58 smithi193 ceph-mon[37473]: pgmap v1308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:41:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:41:58.974+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:41:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:41:58.974+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:41:58.976 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:41:58.974+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:41:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:41:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 860.052559 secs 2024-07-26T11:41:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:41:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 860.052559 secs 2024-07-26T11:41:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:41:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 860.052559 secs 2024-07-26T11:42:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:00 smithi134 ceph-mon[31966]: pgmap v1309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:42:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:00 smithi189 ceph-mon[36790]: pgmap v1309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:42:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:00 smithi193 ceph-mon[37473]: pgmap v1309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:42:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 879.478129 secs 2024-07-26T11:42:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 879.478129 secs 2024-07-26T11:42:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 879.478129 secs 2024-07-26T11:42:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:02 smithi134 ceph-mon[31966]: pgmap v1310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:42:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:02 smithi189 ceph-mon[36790]: pgmap v1310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:42:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:02 smithi193 ceph-mon[37473]: pgmap v1310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:42:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:03 smithi134 ceph-mon[31966]: pgmap v1311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:42:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:03 smithi189 ceph-mon[36790]: pgmap v1311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:42:03.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:03.448+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:42:03.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:03.448+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:42:03.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:03.448+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:03 smithi193 ceph-mon[37473]: pgmap v1311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:42:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 865.052674 secs 2024-07-26T11:42:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 865.052674 secs 2024-07-26T11:42:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 865.052674 secs 2024-07-26T11:42:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:05 smithi134 ceph-mon[31966]: pgmap v1312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:42:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:05 smithi189 ceph-mon[36790]: pgmap v1312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:42:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:05 smithi193 ceph-mon[37473]: pgmap v1312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:42:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 884.478261 secs 2024-07-26T11:42:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 884.478261 secs 2024-07-26T11:42:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 884.478261 secs 2024-07-26T11:42:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:07 smithi134 ceph-mon[31966]: pgmap v1313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:42:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:07 smithi189 ceph-mon[36790]: pgmap v1313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:42:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:07 smithi193 ceph-mon[37473]: pgmap v1313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:42:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:09 smithi193 ceph-mon[37473]: pgmap v1314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 870.052813 secs 2024-07-26T11:42:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:09 smithi134 ceph-mon[31966]: pgmap v1314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 870.052813 secs 2024-07-26T11:42:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:09 smithi189 ceph-mon[36790]: pgmap v1314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 870.052813 secs 2024-07-26T11:42:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:11 smithi193 ceph-mon[37473]: pgmap v1315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 889.478390 secs 2024-07-26T11:42:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:11 smithi134 ceph-mon[31966]: pgmap v1315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 889.478390 secs 2024-07-26T11:42:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:11 smithi189 ceph-mon[36790]: pgmap v1315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 889.478390 secs 2024-07-26T11:42:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:13 smithi193 ceph-mon[37473]: pgmap v1316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:13 smithi134 ceph-mon[31966]: pgmap v1316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:13 smithi189 ceph-mon[36790]: pgmap v1316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:13.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:13.990+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:42:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:13.990+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:42:13.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:13.990+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 875.052945 secs 2024-07-26T11:42:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 875.052945 secs 2024-07-26T11:42:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 875.052945 secs 2024-07-26T11:42:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:15 smithi193 ceph-mon[37473]: pgmap v1317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:15 smithi134 ceph-mon[31966]: pgmap v1317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:15 smithi189 ceph-mon[36790]: pgmap v1317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 894.478488 secs 2024-07-26T11:42:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 894.478488 secs 2024-07-26T11:42:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 894.478488 secs 2024-07-26T11:42:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:17 smithi193 ceph-mon[37473]: pgmap v1318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:17 smithi134 ceph-mon[31966]: pgmap v1318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:17 smithi189 ceph-mon[36790]: pgmap v1318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:18.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:18.464+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:42:18.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:18.464+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:18.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:18.464+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:42:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:19 smithi193 ceph-mon[37473]: pgmap v1319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 880.053065 secs 2024-07-26T11:42:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:19 smithi134 ceph-mon[31966]: pgmap v1319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 880.053065 secs 2024-07-26T11:42:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:19 smithi189 ceph-mon[36790]: pgmap v1319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 880.053065 secs 2024-07-26T11:42:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:21 smithi193 ceph-mon[37473]: pgmap v1320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 899.478615 secs 2024-07-26T11:42:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:21 smithi134 ceph-mon[31966]: pgmap v1320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 899.478615 secs 2024-07-26T11:42:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:21 smithi189 ceph-mon[36790]: pgmap v1320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 899.478615 secs 2024-07-26T11:42:22.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:22.283+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 msgr2=0x7f9b300b6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-07-26T11:42:22.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:22.283+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 msgr2=0x7f9b300b6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:42:22.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:22.283+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300b6300 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009990 tx=0x7f9b38009110 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:42:22.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:22.283+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300b6300 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009990 tx=0x7f9b38009110 comp rx=0 tx=0).stop 2024-07-26T11:42:22.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:22.283+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 msgr2=0x7f9b300b6300 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T11:42:22.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:22.283+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 msgr2=0x7f9b300b6300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:42:22.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:22.283+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300b6300 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:42:22.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:22.283+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bf720 0x7f9b300c1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:42:22.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:22.284+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bf720 0x7f9b300c1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:42:22.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:22.286+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bf720 0x7f9b300c1b40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009960 tx=0x7f9b3802b040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:42:22.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:22.488+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006390 con 0x7f9b4c10f330 2024-07-26T11:42:22.489 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:22.488+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006390 con 0x7f9b4c10f330 2024-07-26T11:42:22.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:22.488+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006260 con 0x55c65e5ee670 2024-07-26T11:42:22.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:22.488+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20006260 con 0x55c65e5ee670 2024-07-26T11:42:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:23 smithi193 ceph-mon[37473]: pgmap v1321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:23 smithi134 ceph-mon[31966]: pgmap v1321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:23 smithi189 ceph-mon[36790]: pgmap v1321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 885.053192 secs 2024-07-26T11:42:24.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 885.053192 secs 2024-07-26T11:42:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 885.053192 secs 2024-07-26T11:42:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:25 smithi193 ceph-mon[37473]: pgmap v1322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:25 smithi134 ceph-mon[31966]: pgmap v1322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:25 smithi189 ceph-mon[36790]: pgmap v1322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 904.478742 secs 2024-07-26T11:42:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 904.478742 secs 2024-07-26T11:42:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 904.478742 secs 2024-07-26T11:42:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:27 smithi134 ceph-mon[31966]: pgmap v1323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:27 smithi189 ceph-mon[36790]: pgmap v1323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:27 smithi193 ceph-mon[37473]: pgmap v1323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:29.005+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:42:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:29.005+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:29.005+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:42:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:29 smithi134 ceph-mon[31966]: pgmap v1324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:29.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 890.053307 secs 2024-07-26T11:42:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:29 smithi189 ceph-mon[36790]: pgmap v1324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 890.053307 secs 2024-07-26T11:42:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:29 smithi193 ceph-mon[37473]: pgmap v1324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 890.053307 secs 2024-07-26T11:42:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:31 smithi134 ceph-mon[31966]: pgmap v1325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 909.478868 secs 2024-07-26T11:42:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:31 smithi189 ceph-mon[36790]: pgmap v1325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 909.478868 secs 2024-07-26T11:42:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:31 smithi193 ceph-mon[37473]: pgmap v1325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 909.478868 secs 2024-07-26T11:42:32.842 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:32.840+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 msgr2=0x7f21ac0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000533) for more than 900000000 us, fault. 2024-07-26T11:42:32.842 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:32.840+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 0x7f21ac0b62d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x55a08e69c8a0 tx=0x7f21b802a3f0 comp rx=0 tx=0).stop 2024-07-26T11:42:32.842 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:32.840+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 msgr2=0x7f21ac0b62d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:42:32.842 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:32.840+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 0x7f21ac0b62d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:42:32.842 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:32.840+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bf5f0 0x7f21ac0c1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:42:32.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:32.841+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bf5f0 0x7f21ac0c1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:42:32.844 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:32.842+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bf5f0 0x7f21ac0c1a10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f21b8006010 tx=0x7f21b8109070 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:42:32.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:32.985+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006270 con 0x55a08e2d94d0 2024-07-26T11:42:32.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:32.985+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980063d0 con 0x7f21c8105550 2024-07-26T11:42:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:32.986+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980063d0 con 0x7f21c8105550 2024-07-26T11:42:32.988 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:32.986+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006270 con 0x55a08e2d94d0 2024-07-26T11:42:33.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:33.475+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:42:33.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:33.475+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:42:33.477 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:33.475+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:33 smithi134 ceph-mon[31966]: pgmap v1326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:33 smithi189 ceph-mon[36790]: pgmap v1326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:33 smithi193 ceph-mon[37473]: pgmap v1326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 895.053408 secs 2024-07-26T11:42:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 895.053408 secs 2024-07-26T11:42:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 895.053408 secs 2024-07-26T11:42:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:35 smithi134 ceph-mon[31966]: pgmap v1327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:35 smithi189 ceph-mon[36790]: pgmap v1327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:35 smithi193 ceph-mon[37473]: pgmap v1327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 914.479010 secs 2024-07-26T11:42:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 914.479010 secs 2024-07-26T11:42:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 914.479010 secs 2024-07-26T11:42:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:37 smithi134 ceph-mon[31966]: pgmap v1328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:37 smithi189 ceph-mon[36790]: pgmap v1328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:37 smithi193 ceph-mon[37473]: pgmap v1328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:39 smithi134 ceph-mon[31966]: pgmap v1329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 900.053533 secs 2024-07-26T11:42:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:39 smithi189 ceph-mon[36790]: pgmap v1329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 900.053533 secs 2024-07-26T11:42:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:39 smithi193 ceph-mon[37473]: pgmap v1329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 900.053533 secs 2024-07-26T11:42:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:41 smithi134 ceph-mon[31966]: pgmap v1330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 919.479123 secs 2024-07-26T11:42:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:41 smithi189 ceph-mon[36790]: pgmap v1330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 919.479123 secs 2024-07-26T11:42:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:41 smithi193 ceph-mon[37473]: pgmap v1330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 919.479123 secs 2024-07-26T11:42:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:43 smithi134 ceph-mon[31966]: pgmap v1331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:43 smithi189 ceph-mon[36790]: pgmap v1331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:43 smithi193 ceph-mon[37473]: pgmap v1331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:44.020+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:42:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:44.020+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:44.022 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:44.020+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:42:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 905.053665 secs 2024-07-26T11:42:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 905.053665 secs 2024-07-26T11:42:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 905.053665 secs 2024-07-26T11:42:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:45 smithi134 ceph-mon[31966]: pgmap v1332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:45 smithi189 ceph-mon[36790]: pgmap v1332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:45 smithi193 ceph-mon[37473]: pgmap v1332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 924.479251 secs 2024-07-26T11:42:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 924.479251 secs 2024-07-26T11:42:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 924.479251 secs 2024-07-26T11:42:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:47 smithi134 ceph-mon[31966]: pgmap v1333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:47 smithi189 ceph-mon[36790]: pgmap v1333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:47 smithi193 ceph-mon[37473]: pgmap v1333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:48.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:48.483+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:42:48.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:48.483+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:42:48.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:42:48.483+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:49 smithi134 ceph-mon[31966]: pgmap v1334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 910.053807 secs 2024-07-26T11:42:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:49 smithi189 ceph-mon[36790]: pgmap v1334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 910.053807 secs 2024-07-26T11:42:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:49 smithi193 ceph-mon[37473]: pgmap v1334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:42:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 910.053807 secs 2024-07-26T11:42:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:51 smithi134 ceph-mon[31966]: pgmap v1335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 929.479380 secs 2024-07-26T11:42:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:51 smithi189 ceph-mon[36790]: pgmap v1335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 929.479380 secs 2024-07-26T11:42:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:51 smithi193 ceph-mon[37473]: pgmap v1335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:42:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 929.479380 secs 2024-07-26T11:42:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:53 smithi134 ceph-mon[31966]: pgmap v1336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:53.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:53 smithi189 ceph-mon[36790]: pgmap v1336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:53 smithi193 ceph-mon[37473]: pgmap v1336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:42:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 915.053918 secs 2024-07-26T11:42:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 915.053918 secs 2024-07-26T11:42:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 915.053918 secs 2024-07-26T11:42:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:55 smithi134 ceph-mon[31966]: pgmap v1337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:55 smithi189 ceph-mon[36790]: pgmap v1337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:55 smithi193 ceph-mon[37473]: pgmap v1337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:42:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 934.479508 secs 2024-07-26T11:42:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:42:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 934.479508 secs 2024-07-26T11:42:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:42:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 934.479508 secs 2024-07-26T11:42:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:42:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:57 smithi193 ceph-mon[37473]: pgmap v1338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:42:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:42:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:57 smithi134 ceph-mon[31966]: pgmap v1338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:42:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:42:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:57 smithi189 ceph-mon[36790]: pgmap v1338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:42:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:42:58.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:42:59.038 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:59.036+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:42:59.038 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:59.036+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:42:59.038 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:42:59.036+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:42:59.217 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:42:59.217 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:42:59.217 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:58 smithi134 ceph-mon[31966]: pgmap v1339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:42:59.217 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:58 smithi134 ceph-mon[31966]: pgmap v1340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:42:59.217 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:42:59.218 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:42:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 920.054051 secs 2024-07-26T11:42:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:42:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:42:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:58 smithi193 ceph-mon[37473]: pgmap v1339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:42:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:58 smithi193 ceph-mon[37473]: pgmap v1340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:42:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:42:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:42:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 920.054051 secs 2024-07-26T11:42:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:42:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:42:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:58 smithi189 ceph-mon[36790]: pgmap v1339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T11:42:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:58 smithi189 ceph-mon[36790]: pgmap v1340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:42:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:42:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:42:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 920.054051 secs 2024-07-26T11:43:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:01 smithi134 ceph-mon[31966]: pgmap v1341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:43:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:01 smithi189 ceph-mon[36790]: pgmap v1341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:43:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:01 smithi193 ceph-mon[37473]: pgmap v1341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:43:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 939.479642 secs 2024-07-26T11:43:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 939.479642 secs 2024-07-26T11:43:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 939.479642 secs 2024-07-26T11:43:03.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:03 smithi134 ceph-mon[31966]: pgmap v1342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:43:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:03 smithi189 ceph-mon[36790]: pgmap v1342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:43:03.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:43:03.488+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:43:03.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:43:03.488+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:43:03.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:43:03.488+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:03 smithi193 ceph-mon[37473]: pgmap v1342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:43:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 925.054208 secs 2024-07-26T11:43:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 925.054208 secs 2024-07-26T11:43:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 925.054208 secs 2024-07-26T11:43:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:05 smithi134 ceph-mon[31966]: pgmap v1343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:43:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:05 smithi189 ceph-mon[36790]: pgmap v1343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:43:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:05 smithi193 ceph-mon[37473]: pgmap v1343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:43:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 944.479721 secs 2024-07-26T11:43:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 944.479721 secs 2024-07-26T11:43:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 944.479721 secs 2024-07-26T11:43:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:07 smithi134 ceph-mon[31966]: pgmap v1344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:43:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:07 smithi189 ceph-mon[36790]: pgmap v1344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:43:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:07 smithi193 ceph-mon[37473]: pgmap v1344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:43:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:08 smithi134 ceph-mon[31966]: pgmap v1345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:43:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:08 smithi189 ceph-mon[36790]: pgmap v1345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:43:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:08 smithi193 ceph-mon[37473]: pgmap v1345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:43:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 930.054292 secs 2024-07-26T11:43:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 930.054292 secs 2024-07-26T11:43:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 930.054292 secs 2024-07-26T11:43:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:10 smithi134 ceph-mon[31966]: pgmap v1346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:10 smithi189 ceph-mon[36790]: pgmap v1346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:10 smithi193 ceph-mon[37473]: pgmap v1346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 949.479848 secs 2024-07-26T11:43:11.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 949.479848 secs 2024-07-26T11:43:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 949.479848 secs 2024-07-26T11:43:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:12 smithi189 ceph-mon[36790]: pgmap v1347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:43:12.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:12 smithi193 ceph-mon[37473]: pgmap v1347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:43:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:12 smithi134 ceph-mon[31966]: pgmap v1347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:43:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 935.054402 secs 2024-07-26T11:43:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 935.054402 secs 2024-07-26T11:43:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 935.054402 secs 2024-07-26T11:43:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:43:14.051+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:43:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:43:14.051+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:43:14.051+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:43:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:14 smithi134 ceph-mon[31966]: pgmap v1348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:14 smithi189 ceph-mon[36790]: pgmap v1348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:14 smithi193 ceph-mon[37473]: pgmap v1348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 954.479959 secs 2024-07-26T11:43:16.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:16 smithi134 ceph-mon[31966]: pgmap v1349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:16.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 954.479959 secs 2024-07-26T11:43:16.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:16 smithi189 ceph-mon[36790]: pgmap v1349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 954.479959 secs 2024-07-26T11:43:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:16 smithi193 ceph-mon[37473]: pgmap v1349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:18.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:43:18.503+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:43:18.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:43:18.503+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:18.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:43:18.503+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:43:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:19 smithi134 ceph-mon[31966]: pgmap v1350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 940.054523 secs 2024-07-26T11:43:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:19 smithi189 ceph-mon[36790]: pgmap v1350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 940.054523 secs 2024-07-26T11:43:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:19 smithi193 ceph-mon[37473]: pgmap v1350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 940.054523 secs 2024-07-26T11:43:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:21 smithi134 ceph-mon[31966]: pgmap v1351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 959.480079 secs 2024-07-26T11:43:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:21 smithi189 ceph-mon[36790]: pgmap v1351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 959.480079 secs 2024-07-26T11:43:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:21 smithi193 ceph-mon[37473]: pgmap v1351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 959.480079 secs 2024-07-26T11:43:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:23 smithi134 ceph-mon[31966]: pgmap v1352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:43:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:23 smithi189 ceph-mon[36790]: pgmap v1352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:43:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:23 smithi193 ceph-mon[37473]: pgmap v1352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:43:24.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 945.054667 secs 2024-07-26T11:43:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 945.054667 secs 2024-07-26T11:43:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 945.054667 secs 2024-07-26T11:43:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:25 smithi134 ceph-mon[31966]: pgmap v1353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:43:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:25 smithi189 ceph-mon[36790]: pgmap v1353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:43:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:25 smithi193 ceph-mon[37473]: pgmap v1353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:43:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 964.480219 secs 2024-07-26T11:43:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 964.480219 secs 2024-07-26T11:43:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 964.480219 secs 2024-07-26T11:43:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:27 smithi134 ceph-mon[31966]: pgmap v1354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:43:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:27 smithi189 ceph-mon[36790]: pgmap v1354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:43:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:27 smithi193 ceph-mon[37473]: pgmap v1354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:43:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:28 smithi134 ceph-mon[31966]: pgmap v1355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:43:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:28 smithi189 ceph-mon[36790]: pgmap v1355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:43:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:28 smithi193 ceph-mon[37473]: pgmap v1355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:43:29.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:43:29.064+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:43:29.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:43:29.064+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:43:29.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:43:29.064+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 950.054806 secs 2024-07-26T11:43:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 950.054806 secs 2024-07-26T11:43:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 950.054806 secs 2024-07-26T11:43:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:30 smithi134 ceph-mon[31966]: pgmap v1356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:30 smithi189 ceph-mon[36790]: pgmap v1356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:30 smithi193 ceph-mon[37473]: pgmap v1356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 969.480363 secs 2024-07-26T11:43:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 969.480363 secs 2024-07-26T11:43:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 969.480363 secs 2024-07-26T11:43:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:32 smithi189 ceph-mon[36790]: pgmap v1357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:43:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:32 smithi193 ceph-mon[37473]: pgmap v1357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:43:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:32 smithi134 ceph-mon[31966]: pgmap v1357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:43:33.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:43:33.519+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:43:33.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:43:33.519+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:33.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:43:33.519+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:43:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 955.054925 secs 2024-07-26T11:43:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 955.054925 secs 2024-07-26T11:43:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 955.054925 secs 2024-07-26T11:43:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:34 smithi134 ceph-mon[31966]: pgmap v1358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:43:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:34 smithi189 ceph-mon[36790]: pgmap v1358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:43:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:34 smithi193 ceph-mon[37473]: pgmap v1358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:43:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 974.480476 secs 2024-07-26T11:43:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:36 smithi134 ceph-mon[31966]: pgmap v1359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 974.480476 secs 2024-07-26T11:43:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:36 smithi189 ceph-mon[36790]: pgmap v1359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 974.480476 secs 2024-07-26T11:43:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:36 smithi193 ceph-mon[37473]: pgmap v1359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:39 smithi134 ceph-mon[31966]: pgmap v1360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 960.054974 secs 2024-07-26T11:43:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:39 smithi189 ceph-mon[36790]: pgmap v1360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 960.054974 secs 2024-07-26T11:43:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:39 smithi193 ceph-mon[37473]: pgmap v1360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 960.054974 secs 2024-07-26T11:43:41.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:41 smithi193 ceph-mon[37473]: pgmap v1361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:41.327 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 979.480600 secs 2024-07-26T11:43:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:41 smithi134 ceph-mon[31966]: pgmap v1361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 979.480600 secs 2024-07-26T11:43:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:41 smithi189 ceph-mon[36790]: pgmap v1361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 979.480600 secs 2024-07-26T11:43:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:43 smithi134 ceph-mon[31966]: pgmap v1362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:43:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:43 smithi189 ceph-mon[36790]: pgmap v1362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:43:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:43 smithi193 ceph-mon[37473]: pgmap v1362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:43:44.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:43:44.079+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:43:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:43:44.080+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:43:44.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:43:44.080+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 965.055087 secs 2024-07-26T11:43:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 965.055087 secs 2024-07-26T11:43:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 965.055087 secs 2024-07-26T11:43:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:45 smithi134 ceph-mon[31966]: pgmap v1363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:45 smithi189 ceph-mon[36790]: pgmap v1363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:45 smithi193 ceph-mon[37473]: pgmap v1363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 984.480759 secs 2024-07-26T11:43:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 984.480759 secs 2024-07-26T11:43:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 984.480759 secs 2024-07-26T11:43:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:47 smithi134 ceph-mon[31966]: pgmap v1364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:47 smithi189 ceph-mon[36790]: pgmap v1364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:47 smithi193 ceph-mon[37473]: pgmap v1364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:48 smithi134 ceph-mon[31966]: pgmap v1365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:48 smithi189 ceph-mon[36790]: pgmap v1365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:48 smithi193 ceph-mon[37473]: pgmap v1365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:48.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:43:48.526+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:43:48.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:43:48.526+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:48.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:43:48.526+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:43:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 970.055210 secs 2024-07-26T11:43:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 970.055210 secs 2024-07-26T11:43:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 970.055210 secs 2024-07-26T11:43:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:50 smithi134 ceph-mon[31966]: pgmap v1366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:50 smithi189 ceph-mon[36790]: pgmap v1366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:50 smithi193 ceph-mon[37473]: pgmap v1366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:43:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 989.480862 secs 2024-07-26T11:43:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 989.480862 secs 2024-07-26T11:43:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 989.480862 secs 2024-07-26T11:43:52.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:43:52.502+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T11:43:52.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:43:52.502+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T11:43:52.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:43:52.502+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T11:43:52.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:43:52.502+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x55c65e9ce530 con 0x7f9b4c10f330 2024-07-26T11:43:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:52 smithi134 ceph-mon[31966]: pgmap v1367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:43:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:52 smithi189 ceph-mon[36790]: pgmap v1367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:43:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:52 smithi193 ceph-mon[37473]: pgmap v1367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:43:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 975.055329 secs 2024-07-26T11:43:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 975.055329 secs 2024-07-26T11:43:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 975.055329 secs 2024-07-26T11:43:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:54 smithi134 ceph-mon[31966]: pgmap v1368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:54 smithi189 ceph-mon[36790]: pgmap v1368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:54 smithi193 ceph-mon[37473]: pgmap v1368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:43:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 994.480993 secs 2024-07-26T11:43:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:56 smithi134 ceph-mon[31966]: pgmap v1369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 994.480993 secs 2024-07-26T11:43:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:56 smithi189 ceph-mon[36790]: pgmap v1369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 994.480993 secs 2024-07-26T11:43:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:56 smithi193 ceph-mon[37473]: pgmap v1369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:43:59.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:43:59.095+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:43:59.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:43:59.095+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:43:59.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:43:59.095+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:43:59.354 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:59 smithi193 ceph-mon[37473]: pgmap v1370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:59.354 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:43:59.354 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 980.055404 secs 2024-07-26T11:43:59.354 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:43:59.354 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:43:59.355 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:43:59.355 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:43:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:43:59.356 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:59 smithi189 ceph-mon[36790]: pgmap v1370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:59.356 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:43:59.357 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 980.055404 secs 2024-07-26T11:43:59.357 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:43:59.357 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:43:59.357 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:43:59.357 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:43:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:43:59.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:59 smithi134 ceph-mon[31966]: pgmap v1370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:43:59.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:43:59.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 980.055404 secs 2024-07-26T11:43:59.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:43:59.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:43:59.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:43:59.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:43:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:01 smithi134 ceph-mon[31966]: pgmap v1371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 999.481120 secs 2024-07-26T11:44:01.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:01 smithi189 ceph-mon[36790]: pgmap v1371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:01.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:01.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:01.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:01.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:01.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 999.481120 secs 2024-07-26T11:44:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:01 smithi193 ceph-mon[37473]: pgmap v1371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:01.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 999.481120 secs 2024-07-26T11:44:03.003 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:44:03.001+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T11:44:03.003 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:44:03.001+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T11:44:03.004 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:44:03.002+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T11:44:03.004 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:44:03.002+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T11:44:03.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:03 smithi134 ceph-mon[31966]: pgmap v1372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:03.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:03.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:03 smithi189 ceph-mon[36790]: pgmap v1372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:03 smithi193 ceph-mon[37473]: pgmap v1372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:03.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:44:03.541+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:44:03.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:44:03.541+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:44:03.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:44:03.541+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:44:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:44:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 985.055464 secs 2024-07-26T11:44:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:44:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:44:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 985.055464 secs 2024-07-26T11:44:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:44:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:44:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:44:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 985.055464 secs 2024-07-26T11:44:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:05 smithi134 ceph-mon[31966]: pgmap v1373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:44:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:05 smithi134 ceph-mon[31966]: pgmap v1374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:44:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:05 smithi134 ceph-mon[31966]: Health check update: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T11:44:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:05 smithi189 ceph-mon[36790]: pgmap v1373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:44:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:05 smithi189 ceph-mon[36790]: pgmap v1374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:44:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:05 smithi189 ceph-mon[36790]: Health check update: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T11:44:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:05 smithi193 ceph-mon[37473]: pgmap v1373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:44:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:05 smithi193 ceph-mon[37473]: pgmap v1374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:44:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:05 smithi193 ceph-mon[37473]: Health check update: 4 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-07-26T11:44:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1004.481253 secs 2024-07-26T11:44:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1004.481253 secs 2024-07-26T11:44:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1004.481253 secs 2024-07-26T11:44:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:07 smithi134 ceph-mon[31966]: pgmap v1375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:44:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:07 smithi189 ceph-mon[36790]: pgmap v1375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:44:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:07 smithi193 ceph-mon[37473]: pgmap v1375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:44:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:08 smithi134 ceph-mon[31966]: pgmap v1376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:44:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:08 smithi189 ceph-mon[36790]: pgmap v1376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:44:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:08 smithi193 ceph-mon[37473]: pgmap v1376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:44:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 990.055613 secs 2024-07-26T11:44:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 990.055613 secs 2024-07-26T11:44:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 990.055613 secs 2024-07-26T11:44:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:10 smithi134 ceph-mon[31966]: pgmap v1377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:44:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:10 smithi189 ceph-mon[36790]: pgmap v1377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:44:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:10 smithi193 ceph-mon[37473]: pgmap v1377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:44:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1009.481405 secs 2024-07-26T11:44:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1009.481405 secs 2024-07-26T11:44:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1009.481405 secs 2024-07-26T11:44:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:12 smithi134 ceph-mon[31966]: pgmap v1378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:44:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:12 smithi189 ceph-mon[36790]: pgmap v1378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:44:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:12 smithi193 ceph-mon[37473]: pgmap v1378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:44:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 995.055693 secs 2024-07-26T11:44:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 995.055693 secs 2024-07-26T11:44:14.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 995.055693 secs 2024-07-26T11:44:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:44:14.111+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:44:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:44:14.111+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:14.113 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:44:14.111+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:44:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:14 smithi134 ceph-mon[31966]: pgmap v1379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:44:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:14 smithi189 ceph-mon[36790]: pgmap v1379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:44:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:14 smithi193 ceph-mon[37473]: pgmap v1379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:44:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:16 smithi134 ceph-mon[31966]: pgmap v1380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1014.481518 secs 2024-07-26T11:44:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:16 smithi189 ceph-mon[36790]: pgmap v1380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1014.481518 secs 2024-07-26T11:44:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:16 smithi193 ceph-mon[37473]: pgmap v1380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1014.481518 secs 2024-07-26T11:44:18.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:44:18.557+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:44:18.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:44:18.557+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:18.559 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:44:18.557+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:44:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:18 smithi134 ceph-mon[31966]: pgmap v1381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1000.055809 secs 2024-07-26T11:44:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:18 smithi189 ceph-mon[36790]: pgmap v1381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1000.055809 secs 2024-07-26T11:44:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:18 smithi193 ceph-mon[37473]: pgmap v1381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1000.055809 secs 2024-07-26T11:44:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:20 smithi134 ceph-mon[31966]: pgmap v1382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:44:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:20 smithi189 ceph-mon[36790]: pgmap v1382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:44:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:20 smithi193 ceph-mon[37473]: pgmap v1382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:44:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1019.481665 secs 2024-07-26T11:44:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1019.481665 secs 2024-07-26T11:44:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1019.481665 secs 2024-07-26T11:44:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:22 smithi189 ceph-mon[36790]: pgmap v1383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:22 smithi193 ceph-mon[37473]: pgmap v1383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:22 smithi134 ceph-mon[31966]: pgmap v1383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1005.055967 secs 2024-07-26T11:44:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1005.055967 secs 2024-07-26T11:44:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1005.055967 secs 2024-07-26T11:44:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:24 smithi193 ceph-mon[37473]: pgmap v1384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:24 smithi134 ceph-mon[31966]: pgmap v1384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:24 smithi189 ceph-mon[36790]: pgmap v1384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:26 smithi193 ceph-mon[37473]: pgmap v1385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1024.481794 secs 2024-07-26T11:44:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:26 smithi134 ceph-mon[31966]: pgmap v1385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1024.481794 secs 2024-07-26T11:44:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:26 smithi189 ceph-mon[36790]: pgmap v1385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1024.481794 secs 2024-07-26T11:44:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:28 smithi193 ceph-mon[37473]: pgmap v1386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:29.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1010.056016 secs 2024-07-26T11:44:29.123 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:44:29.121+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:44:29.123 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:44:29.121+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:44:29.123 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:44:29.121+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:28 smithi134 ceph-mon[31966]: pgmap v1386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1010.056016 secs 2024-07-26T11:44:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:28 smithi189 ceph-mon[36790]: pgmap v1386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1010.056016 secs 2024-07-26T11:44:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:30 smithi193 ceph-mon[37473]: pgmap v1387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:44:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:30 smithi134 ceph-mon[31966]: pgmap v1387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:44:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:30 smithi189 ceph-mon[36790]: pgmap v1387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:44:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1029.481905 secs 2024-07-26T11:44:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1029.481905 secs 2024-07-26T11:44:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1029.481905 secs 2024-07-26T11:44:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:32 smithi193 ceph-mon[37473]: pgmap v1388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:33.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:32 smithi134 ceph-mon[31966]: pgmap v1388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:32 smithi189 ceph-mon[36790]: pgmap v1388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:33.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:44:33.572+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:44:33.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:44:33.572+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:33.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:44:33.572+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:44:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1015.056097 secs 2024-07-26T11:44:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1015.056097 secs 2024-07-26T11:44:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1015.056097 secs 2024-07-26T11:44:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:34 smithi193 ceph-mon[37473]: pgmap v1389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:34 smithi134 ceph-mon[31966]: pgmap v1389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:34 smithi189 ceph-mon[36790]: pgmap v1389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:36 smithi193 ceph-mon[37473]: pgmap v1390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1034.482036 secs 2024-07-26T11:44:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:36 smithi134 ceph-mon[31966]: pgmap v1390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1034.482036 secs 2024-07-26T11:44:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:36 smithi189 ceph-mon[36790]: pgmap v1390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1034.482036 secs 2024-07-26T11:44:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:38 smithi193 ceph-mon[37473]: pgmap v1391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1020.056178 secs 2024-07-26T11:44:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:38 smithi134 ceph-mon[31966]: pgmap v1391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1020.056178 secs 2024-07-26T11:44:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:38 smithi189 ceph-mon[36790]: pgmap v1391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1020.056178 secs 2024-07-26T11:44:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:40 smithi193 ceph-mon[37473]: pgmap v1392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:44:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:40 smithi134 ceph-mon[31966]: pgmap v1392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:44:41.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:40 smithi189 ceph-mon[36790]: pgmap v1392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:44:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1039.482182 secs 2024-07-26T11:44:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1039.482182 secs 2024-07-26T11:44:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1039.482182 secs 2024-07-26T11:44:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:42 smithi193 ceph-mon[37473]: pgmap v1393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:42 smithi134 ceph-mon[31966]: pgmap v1393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:42 smithi189 ceph-mon[36790]: pgmap v1393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:44:44.137+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:44:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:44:44.137+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:44:44.139 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:44:44.137+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1025.056255 secs 2024-07-26T11:44:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1025.056255 secs 2024-07-26T11:44:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1025.056255 secs 2024-07-26T11:44:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:44 smithi134 ceph-mon[31966]: pgmap v1394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:44 smithi189 ceph-mon[36790]: pgmap v1394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:44 smithi193 ceph-mon[37473]: pgmap v1394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:46 smithi134 ceph-mon[31966]: pgmap v1395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1044.482288 secs 2024-07-26T11:44:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:46 smithi189 ceph-mon[36790]: pgmap v1395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1044.482288 secs 2024-07-26T11:44:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:46 smithi193 ceph-mon[37473]: pgmap v1395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1044.482288 secs 2024-07-26T11:44:48.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:44:48.587+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:44:48.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:44:48.587+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:48.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:44:48.587+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:44:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:48 smithi134 ceph-mon[31966]: pgmap v1396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1030.056333 secs 2024-07-26T11:44:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:48 smithi189 ceph-mon[36790]: pgmap v1396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1030.056333 secs 2024-07-26T11:44:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:48 smithi193 ceph-mon[37473]: pgmap v1396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1030.056333 secs 2024-07-26T11:44:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:50 smithi134 ceph-mon[31966]: pgmap v1397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:44:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:50 smithi189 ceph-mon[36790]: pgmap v1397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:44:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:50 smithi193 ceph-mon[37473]: pgmap v1397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:44:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1049.482422 secs 2024-07-26T11:44:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1049.482422 secs 2024-07-26T11:44:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1049.482422 secs 2024-07-26T11:44:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:52 smithi134 ceph-mon[31966]: pgmap v1398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:52 smithi189 ceph-mon[36790]: pgmap v1398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:52 smithi193 ceph-mon[37473]: pgmap v1398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:44:54.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1035.056449 secs 2024-07-26T11:44:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1035.056449 secs 2024-07-26T11:44:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1035.056449 secs 2024-07-26T11:44:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:54 smithi134 ceph-mon[31966]: pgmap v1399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:54 smithi189 ceph-mon[36790]: pgmap v1399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:54 smithi193 ceph-mon[37473]: pgmap v1399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:44:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:56 smithi134 ceph-mon[31966]: pgmap v1400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1054.482546 secs 2024-07-26T11:44:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:56 smithi189 ceph-mon[36790]: pgmap v1400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1054.482546 secs 2024-07-26T11:44:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:56 smithi193 ceph-mon[37473]: pgmap v1400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:44:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1054.482546 secs 2024-07-26T11:44:59.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:44:59.153+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:44:59.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:44:59.153+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:44:59.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:44:59.153+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:44:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:58 smithi134 ceph-mon[31966]: pgmap v1401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:44:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1040.056573 secs 2024-07-26T11:44:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:58 smithi189 ceph-mon[36790]: pgmap v1401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:44:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1040.056573 secs 2024-07-26T11:44:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:58 smithi193 ceph-mon[37473]: pgmap v1401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:44:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:44:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1040.056573 secs 2024-07-26T11:45:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:00 smithi134 ceph-mon[31966]: pgmap v1402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:00 smithi189 ceph-mon[36790]: pgmap v1402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:00 smithi193 ceph-mon[37473]: pgmap v1402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1059.482690 secs 2024-07-26T11:45:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1059.482690 secs 2024-07-26T11:45:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1059.482690 secs 2024-07-26T11:45:03.121 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:02 smithi134 ceph-mon[31966]: pgmap v1403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:02 smithi189 ceph-mon[36790]: pgmap v1403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:02 smithi193 ceph-mon[37473]: pgmap v1403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:03.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:45:03.603+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:45:03.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:45:03.603+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:45:03.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:45:03.603+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1045.056693 secs 2024-07-26T11:45:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:45:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1045.056693 secs 2024-07-26T11:45:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:45:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1045.056693 secs 2024-07-26T11:45:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:45:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:04 smithi134 ceph-mon[31966]: pgmap v1404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:45:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:45:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:45:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:45:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:04 smithi189 ceph-mon[36790]: pgmap v1404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:45:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:45:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:45:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:45:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:04 smithi193 ceph-mon[37473]: pgmap v1404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:45:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:45:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:45:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:45:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:05 smithi134 ceph-mon[31966]: pgmap v1405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:45:06.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:05 smithi134 ceph-mon[31966]: pgmap v1406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:45:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:05 smithi189 ceph-mon[36790]: pgmap v1405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:45:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:05 smithi189 ceph-mon[36790]: pgmap v1406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:45:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:05 smithi193 ceph-mon[37473]: pgmap v1405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:45:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:05 smithi193 ceph-mon[37473]: pgmap v1406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:45:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1064.482789 secs 2024-07-26T11:45:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1064.482789 secs 2024-07-26T11:45:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1064.482789 secs 2024-07-26T11:45:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:07 smithi134 ceph-mon[31966]: pgmap v1407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:07 smithi189 ceph-mon[36790]: pgmap v1407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:07 smithi193 ceph-mon[37473]: pgmap v1407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1050.056847 secs 2024-07-26T11:45:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1050.056847 secs 2024-07-26T11:45:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1050.056847 secs 2024-07-26T11:45:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:09 smithi189 ceph-mon[36790]: pgmap v1408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:09 smithi193 ceph-mon[37473]: pgmap v1408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:09 smithi134 ceph-mon[31966]: pgmap v1408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:11 smithi189 ceph-mon[36790]: pgmap v1409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1069.482933 secs 2024-07-26T11:45:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:11 smithi193 ceph-mon[37473]: pgmap v1409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1069.482933 secs 2024-07-26T11:45:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:11 smithi134 ceph-mon[31966]: pgmap v1409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:45:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1069.482933 secs 2024-07-26T11:45:14.170 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:45:14.169+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:45:14.170 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:45:14.169+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:14.170 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:45:14.169+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:45:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:13 smithi189 ceph-mon[36790]: pgmap v1410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:45:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1055.056901 secs 2024-07-26T11:45:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:13 smithi193 ceph-mon[37473]: pgmap v1410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:45:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1055.056901 secs 2024-07-26T11:45:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:13 smithi134 ceph-mon[31966]: pgmap v1410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:45:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1055.056901 secs 2024-07-26T11:45:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:15 smithi189 ceph-mon[36790]: pgmap v1411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:45:16.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:15 smithi193 ceph-mon[37473]: pgmap v1411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:45:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:15 smithi134 ceph-mon[31966]: pgmap v1411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:45:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1074.483055 secs 2024-07-26T11:45:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1074.483055 secs 2024-07-26T11:45:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1074.483055 secs 2024-07-26T11:45:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:17 smithi193 ceph-mon[37473]: pgmap v1412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:17 smithi134 ceph-mon[31966]: pgmap v1412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:17 smithi189 ceph-mon[36790]: pgmap v1412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:18.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:45:18.616+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:45:18.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:45:18.616+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:45:18.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:45:18.616+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1060.057025 secs 2024-07-26T11:45:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1060.057025 secs 2024-07-26T11:45:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1060.057025 secs 2024-07-26T11:45:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:19 smithi193 ceph-mon[37473]: pgmap v1413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:19 smithi134 ceph-mon[31966]: pgmap v1413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:19 smithi189 ceph-mon[36790]: pgmap v1413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:21 smithi193 ceph-mon[37473]: pgmap v1414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:22.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1079.483199 secs 2024-07-26T11:45:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:21 smithi134 ceph-mon[31966]: pgmap v1414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1079.483199 secs 2024-07-26T11:45:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:21 smithi189 ceph-mon[36790]: pgmap v1414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1079.483199 secs 2024-07-26T11:45:22.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:45:22.517+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T11:45:22.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:45:22.517+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T11:45:22.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:45:22.518+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T11:45:22.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:45:22.518+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T11:45:24.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:23 smithi189 ceph-mon[36790]: pgmap v1415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:24.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1065.057089 secs 2024-07-26T11:45:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:23 smithi193 ceph-mon[37473]: pgmap v1415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1065.057089 secs 2024-07-26T11:45:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:23 smithi134 ceph-mon[31966]: pgmap v1415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1065.057089 secs 2024-07-26T11:45:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:25 smithi193 ceph-mon[37473]: pgmap v1416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:45:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:25 smithi134 ceph-mon[31966]: pgmap v1416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:45:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:25 smithi189 ceph-mon[36790]: pgmap v1416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:45:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1084.483357 secs 2024-07-26T11:45:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1084.483357 secs 2024-07-26T11:45:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1084.483357 secs 2024-07-26T11:45:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:28 smithi193 ceph-mon[37473]: pgmap v1417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:28 smithi134 ceph-mon[31966]: pgmap v1417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:28 smithi189 ceph-mon[36790]: pgmap v1417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:45:29.176+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:45:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:45:29.176+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:45:29.178 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:45:29.176+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1070.057174 secs 2024-07-26T11:45:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1070.057174 secs 2024-07-26T11:45:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1070.057174 secs 2024-07-26T11:45:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:30 smithi193 ceph-mon[37473]: pgmap v1418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:30 smithi134 ceph-mon[31966]: pgmap v1418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:30 smithi189 ceph-mon[36790]: pgmap v1418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:32 smithi193 ceph-mon[37473]: pgmap v1419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1089.483445 secs 2024-07-26T11:45:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:32 smithi134 ceph-mon[31966]: pgmap v1419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1089.483445 secs 2024-07-26T11:45:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:32 smithi189 ceph-mon[36790]: pgmap v1419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1089.483445 secs 2024-07-26T11:45:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:45:33.016+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194003690 con 0x55a08e2d94d0 2024-07-26T11:45:33.018 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:45:33.016+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T11:45:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:45:33.017+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194003690 con 0x55a08e2d94d0 2024-07-26T11:45:33.019 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:45:33.017+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T11:45:33.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:45:33.632+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:45:33.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:45:33.632+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:45:33.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:45:33.632+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:34.345 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:34 smithi134 ceph-mon[31966]: pgmap v1420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:34.345 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1075.057251 secs 2024-07-26T11:45:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:34 smithi189 ceph-mon[36790]: pgmap v1420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1075.057251 secs 2024-07-26T11:45:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:34 smithi193 ceph-mon[37473]: pgmap v1420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1075.057251 secs 2024-07-26T11:45:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:36 smithi193 ceph-mon[37473]: pgmap v1421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T11:45:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:36 smithi134 ceph-mon[31966]: pgmap v1421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T11:45:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:36 smithi189 ceph-mon[36790]: pgmap v1421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T11:45:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1094.483559 secs 2024-07-26T11:45:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1094.483559 secs 2024-07-26T11:45:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1094.483559 secs 2024-07-26T11:45:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:38 smithi134 ceph-mon[31966]: pgmap v1422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:38 smithi189 ceph-mon[36790]: pgmap v1422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:38 smithi193 ceph-mon[37473]: pgmap v1422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1080.057333 secs 2024-07-26T11:45:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1080.057333 secs 2024-07-26T11:45:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1080.057333 secs 2024-07-26T11:45:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:40 smithi134 ceph-mon[31966]: pgmap v1423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:45:40.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:40 smithi189 ceph-mon[36790]: pgmap v1423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:45:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:40 smithi193 ceph-mon[37473]: pgmap v1423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:45:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1099.483626 secs 2024-07-26T11:45:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1099.483626 secs 2024-07-26T11:45:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1099.483626 secs 2024-07-26T11:45:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:42 smithi134 ceph-mon[31966]: pgmap v1424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:45:42.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:42 smithi193 ceph-mon[37473]: pgmap v1424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:45:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:42 smithi189 ceph-mon[36790]: pgmap v1424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:45:43.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:43 smithi189 ceph-mon[36790]: pgmap v1425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:43.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:43 smithi193 ceph-mon[37473]: pgmap v1425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:43 smithi134 ceph-mon[31966]: pgmap v1425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:45:44.191+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:45:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:45:44.191+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:44.193 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:45:44.191+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:45:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1085.057403 secs 2024-07-26T11:45:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1085.057403 secs 2024-07-26T11:45:44.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1085.057403 secs 2024-07-26T11:45:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:45 smithi189 ceph-mon[36790]: pgmap v1426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T11:45:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:45 smithi193 ceph-mon[37473]: pgmap v1426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T11:45:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:45 smithi134 ceph-mon[31966]: pgmap v1426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T11:45:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1104.483721 secs 2024-07-26T11:45:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1104.483721 secs 2024-07-26T11:45:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1104.483721 secs 2024-07-26T11:45:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:47 smithi193 ceph-mon[37473]: pgmap v1427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:45:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:47 smithi134 ceph-mon[31966]: pgmap v1427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:45:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:47 smithi189 ceph-mon[36790]: pgmap v1427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:45:48.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:45:48.648+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:45:48.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:45:48.648+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:48.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:45:48.648+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:45:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1090.057476 secs 2024-07-26T11:45:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:49 smithi193 ceph-mon[37473]: pgmap v1428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:45:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1090.057476 secs 2024-07-26T11:45:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:49 smithi134 ceph-mon[31966]: pgmap v1428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:45:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1090.057476 secs 2024-07-26T11:45:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:49 smithi189 ceph-mon[36790]: pgmap v1428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:45:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:51 smithi193 ceph-mon[37473]: pgmap v1429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1109.483842 secs 2024-07-26T11:45:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:51 smithi134 ceph-mon[31966]: pgmap v1429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1109.483842 secs 2024-07-26T11:45:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:51 smithi189 ceph-mon[36790]: pgmap v1429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1109.483842 secs 2024-07-26T11:45:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:53 smithi193 ceph-mon[37473]: pgmap v1430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:53 smithi134 ceph-mon[31966]: pgmap v1430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:53 smithi189 ceph-mon[36790]: pgmap v1430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:45:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1095.057535 secs 2024-07-26T11:45:54.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1095.057535 secs 2024-07-26T11:45:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1095.057535 secs 2024-07-26T11:45:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:55 smithi193 ceph-mon[37473]: pgmap v1431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T11:45:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:55 smithi134 ceph-mon[31966]: pgmap v1431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T11:45:55.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:55 smithi189 ceph-mon[36790]: pgmap v1431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T11:45:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1114.483972 secs 2024-07-26T11:45:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1114.483972 secs 2024-07-26T11:45:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1114.483972 secs 2024-07-26T11:45:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:57 smithi193 ceph-mon[37473]: pgmap v1432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:57 smithi134 ceph-mon[31966]: pgmap v1432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:57 smithi189 ceph-mon[36790]: pgmap v1432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:45:59.195 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:45:59.193+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:45:59.195 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:45:59.193+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:45:59.195 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:45:59.193+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:45:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1100.057659 secs 2024-07-26T11:45:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:45:59 smithi193 ceph-mon[37473]: pgmap v1433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1100.057659 secs 2024-07-26T11:45:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:45:59 smithi134 ceph-mon[31966]: pgmap v1433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:45:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1100.057659 secs 2024-07-26T11:45:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:45:59 smithi189 ceph-mon[36790]: pgmap v1433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:46:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:01 smithi193 ceph-mon[37473]: pgmap v1434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1119.484121 secs 2024-07-26T11:46:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:01 smithi134 ceph-mon[31966]: pgmap v1434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1119.484121 secs 2024-07-26T11:46:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:01 smithi189 ceph-mon[36790]: pgmap v1434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:01.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1119.484121 secs 2024-07-26T11:46:03.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:46:03.663+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:46:03.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:46:03.663+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:46:03.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:46:03.663+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:03 smithi193 ceph-mon[37473]: pgmap v1435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:03 smithi134 ceph-mon[31966]: pgmap v1435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:03 smithi189 ceph-mon[36790]: pgmap v1435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1105.057771 secs 2024-07-26T11:46:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1105.057771 secs 2024-07-26T11:46:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1105.057771 secs 2024-07-26T11:46:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:05 smithi134 ceph-mon[31966]: pgmap v1436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T11:46:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:46:05.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:05 smithi189 ceph-mon[36790]: pgmap v1436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T11:46:05.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:46:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:05 smithi193 ceph-mon[37473]: pgmap v1436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T11:46:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:46:06.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:46:06.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:46:06.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:06 smithi134 ceph-mon[31966]: pgmap v1437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T11:46:06.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:06 smithi134 ceph-mon[31966]: pgmap v1438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:46:06.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:46:06.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1124.484188 secs 2024-07-26T11:46:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:46:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:46:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:06 smithi189 ceph-mon[36790]: pgmap v1437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T11:46:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:06 smithi189 ceph-mon[36790]: pgmap v1438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:46:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:46:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1124.484188 secs 2024-07-26T11:46:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:46:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:46:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:06 smithi193 ceph-mon[37473]: pgmap v1437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T11:46:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:06 smithi193 ceph-mon[37473]: pgmap v1438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:46:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:46:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1124.484188 secs 2024-07-26T11:46:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:08 smithi134 ceph-mon[31966]: pgmap v1439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:46:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1110.057894 secs 2024-07-26T11:46:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:08 smithi189 ceph-mon[36790]: pgmap v1439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:46:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1110.057894 secs 2024-07-26T11:46:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:08 smithi193 ceph-mon[37473]: pgmap v1439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:46:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1110.057894 secs 2024-07-26T11:46:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:10 smithi189 ceph-mon[36790]: pgmap v1440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:46:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:10 smithi193 ceph-mon[37473]: pgmap v1440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:46:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:10 smithi134 ceph-mon[31966]: pgmap v1440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:46:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1129.484328 secs 2024-07-26T11:46:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1129.484328 secs 2024-07-26T11:46:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1129.484328 secs 2024-07-26T11:46:12.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:12 smithi189 ceph-mon[36790]: pgmap v1441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:46:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:12 smithi193 ceph-mon[37473]: pgmap v1441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:46:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:12 smithi134 ceph-mon[31966]: pgmap v1441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T11:46:14.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1115.057973 secs 2024-07-26T11:46:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1115.057973 secs 2024-07-26T11:46:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1115.057973 secs 2024-07-26T11:46:14.210 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:46:14.208+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:46:14.210 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:46:14.208+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:46:14.210 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:46:14.208+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:14 smithi193 ceph-mon[37473]: pgmap v1442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:46:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:14 smithi134 ceph-mon[31966]: pgmap v1442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:46:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:14 smithi189 ceph-mon[36790]: pgmap v1442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:46:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:16 smithi193 ceph-mon[37473]: pgmap v1443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:46:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1134.484454 secs 2024-07-26T11:46:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:16 smithi134 ceph-mon[31966]: pgmap v1443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:46:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1134.484454 secs 2024-07-26T11:46:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:16 smithi189 ceph-mon[36790]: pgmap v1443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:46:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1134.484454 secs 2024-07-26T11:46:18.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:46:18.667+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:46:18.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:46:18.667+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:18.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:46:18.667+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:46:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:18 smithi134 ceph-mon[31966]: pgmap v1444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1120.058097 secs 2024-07-26T11:46:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:18 smithi193 ceph-mon[37473]: pgmap v1444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1120.058097 secs 2024-07-26T11:46:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:18 smithi189 ceph-mon[36790]: pgmap v1444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1120.058097 secs 2024-07-26T11:46:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:20 smithi193 ceph-mon[37473]: pgmap v1445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:20 smithi134 ceph-mon[31966]: pgmap v1445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:20 smithi189 ceph-mon[36790]: pgmap v1445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1139.484561 secs 2024-07-26T11:46:22.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1139.484561 secs 2024-07-26T11:46:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1139.484561 secs 2024-07-26T11:46:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:22 smithi193 ceph-mon[37473]: pgmap v1446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:22 smithi134 ceph-mon[31966]: pgmap v1446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:23.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:22 smithi189 ceph-mon[36790]: pgmap v1446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1125.058233 secs 2024-07-26T11:46:24.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1125.058233 secs 2024-07-26T11:46:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1125.058233 secs 2024-07-26T11:46:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:24 smithi193 ceph-mon[37473]: pgmap v1447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:46:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:24 smithi134 ceph-mon[31966]: pgmap v1447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:46:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:24 smithi189 ceph-mon[36790]: pgmap v1447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:46:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:26 smithi193 ceph-mon[37473]: pgmap v1448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1144.484648 secs 2024-07-26T11:46:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:26 smithi134 ceph-mon[31966]: pgmap v1448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1144.484648 secs 2024-07-26T11:46:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:26 smithi189 ceph-mon[36790]: pgmap v1448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1144.484648 secs 2024-07-26T11:46:29.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:28 smithi193 ceph-mon[37473]: pgmap v1449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:29.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1130.058372 secs 2024-07-26T11:46:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:28 smithi134 ceph-mon[31966]: pgmap v1449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1130.058372 secs 2024-07-26T11:46:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:28 smithi189 ceph-mon[36790]: pgmap v1449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1130.058372 secs 2024-07-26T11:46:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:46:29.210+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:46:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:46:29.210+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:29.212 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:46:29.210+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:46:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:30 smithi193 ceph-mon[37473]: pgmap v1450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:30 smithi134 ceph-mon[31966]: pgmap v1450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:31.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:30 smithi189 ceph-mon[36790]: pgmap v1450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1149.484788 secs 2024-07-26T11:46:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1149.484788 secs 2024-07-26T11:46:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1149.484788 secs 2024-07-26T11:46:33.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:32 smithi134 ceph-mon[31966]: pgmap v1451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:33.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:32 smithi189 ceph-mon[36790]: pgmap v1451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:32 smithi193 ceph-mon[37473]: pgmap v1451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:33.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:46:33.683+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:46:33.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:46:33.683+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:33.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:46:33.683+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:46:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1135.058505 secs 2024-07-26T11:46:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1135.058505 secs 2024-07-26T11:46:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1135.058505 secs 2024-07-26T11:46:35.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:34 smithi134 ceph-mon[31966]: pgmap v1452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:46:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:34 smithi189 ceph-mon[36790]: pgmap v1452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:46:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:34 smithi193 ceph-mon[37473]: pgmap v1452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:46:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:36 smithi134 ceph-mon[31966]: pgmap v1453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1154.484891 secs 2024-07-26T11:46:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:36 smithi189 ceph-mon[36790]: pgmap v1453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1154.484891 secs 2024-07-26T11:46:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:36 smithi193 ceph-mon[37473]: pgmap v1453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1154.484891 secs 2024-07-26T11:46:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:38 smithi134 ceph-mon[31966]: pgmap v1454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1140.058642 secs 2024-07-26T11:46:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:38 smithi189 ceph-mon[36790]: pgmap v1454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1140.058642 secs 2024-07-26T11:46:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:38 smithi193 ceph-mon[37473]: pgmap v1454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1140.058642 secs 2024-07-26T11:46:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:40 smithi134 ceph-mon[31966]: pgmap v1455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:40 smithi189 ceph-mon[36790]: pgmap v1455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:40 smithi193 ceph-mon[37473]: pgmap v1455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1159.484978 secs 2024-07-26T11:46:42.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1159.484978 secs 2024-07-26T11:46:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1159.484978 secs 2024-07-26T11:46:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:42 smithi134 ceph-mon[31966]: pgmap v1456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:42 smithi189 ceph-mon[36790]: pgmap v1456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:42 smithi193 ceph-mon[37473]: pgmap v1456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1145.058790 secs 2024-07-26T11:46:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1145.058790 secs 2024-07-26T11:46:44.227 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:46:44.226+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:46:44.228 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:46:44.226+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:46:44.228 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:46:44.226+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1145.058790 secs 2024-07-26T11:46:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:44 smithi134 ceph-mon[31966]: pgmap v1457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:46:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:44 smithi189 ceph-mon[36790]: pgmap v1457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:46:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:44 smithi193 ceph-mon[37473]: pgmap v1457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:46:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:46 smithi134 ceph-mon[31966]: pgmap v1458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1164.485099 secs 2024-07-26T11:46:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:46 smithi189 ceph-mon[36790]: pgmap v1458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1164.485099 secs 2024-07-26T11:46:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:46 smithi193 ceph-mon[37473]: pgmap v1458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1164.485099 secs 2024-07-26T11:46:48.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:46:48.698+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:46:48.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:46:48.698+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:48.701 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:46:48.698+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:46:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:48 smithi134 ceph-mon[31966]: pgmap v1459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1150.058928 secs 2024-07-26T11:46:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:48 smithi189 ceph-mon[36790]: pgmap v1459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1150.058928 secs 2024-07-26T11:46:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:48 smithi193 ceph-mon[37473]: pgmap v1459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1150.058928 secs 2024-07-26T11:46:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:50 smithi134 ceph-mon[31966]: pgmap v1460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:50 smithi189 ceph-mon[36790]: pgmap v1460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:50 smithi193 ceph-mon[37473]: pgmap v1460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:46:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1169.485240 secs 2024-07-26T11:46:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1169.485240 secs 2024-07-26T11:46:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1169.485240 secs 2024-07-26T11:46:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:52 smithi134 ceph-mon[31966]: pgmap v1461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:52 smithi189 ceph-mon[36790]: pgmap v1461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:52 smithi193 ceph-mon[37473]: pgmap v1461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1155.059004 secs 2024-07-26T11:46:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1155.059004 secs 2024-07-26T11:46:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1155.059004 secs 2024-07-26T11:46:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:54 smithi134 ceph-mon[31966]: pgmap v1462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:46:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:54 smithi189 ceph-mon[36790]: pgmap v1462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:46:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:54 smithi193 ceph-mon[37473]: pgmap v1462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:46:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:56 smithi134 ceph-mon[31966]: pgmap v1463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1174.485385 secs 2024-07-26T11:46:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:56 smithi189 ceph-mon[36790]: pgmap v1463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1174.485385 secs 2024-07-26T11:46:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:56 smithi193 ceph-mon[37473]: pgmap v1463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:46:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1174.485385 secs 2024-07-26T11:46:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:58 smithi134 ceph-mon[31966]: pgmap v1464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:46:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1160.059139 secs 2024-07-26T11:46:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:58 smithi189 ceph-mon[36790]: pgmap v1464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:46:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1160.059139 secs 2024-07-26T11:46:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:46:59.241+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:46:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:46:59.241+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:46:59.243 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:46:59.241+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:46:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:58 smithi193 ceph-mon[37473]: pgmap v1464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:46:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:46:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1160.059139 secs 2024-07-26T11:47:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:00 smithi134 ceph-mon[31966]: pgmap v1465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:47:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:00 smithi189 ceph-mon[36790]: pgmap v1465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:47:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:00 smithi193 ceph-mon[37473]: pgmap v1465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:47:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1179.485496 secs 2024-07-26T11:47:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1179.485496 secs 2024-07-26T11:47:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1179.485496 secs 2024-07-26T11:47:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:02 smithi189 ceph-mon[36790]: pgmap v1466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:02 smithi193 ceph-mon[37473]: pgmap v1466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:02 smithi134 ceph-mon[31966]: pgmap v1466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:03.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:47:03.714+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:47:03.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:47:03.714+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:47:03.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:47:03.714+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1165.059277 secs 2024-07-26T11:47:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1165.059277 secs 2024-07-26T11:47:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1165.059277 secs 2024-07-26T11:47:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:04 smithi189 ceph-mon[36790]: pgmap v1467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:47:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:04 smithi193 ceph-mon[37473]: pgmap v1467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:47:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:04 smithi134 ceph-mon[31966]: pgmap v1467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:47:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:47:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:47:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:47:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:06 smithi193 ceph-mon[37473]: pgmap v1468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1184.485607 secs 2024-07-26T11:47:07.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:47:07.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:47:07.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:47:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:06 smithi134 ceph-mon[31966]: pgmap v1468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1184.485607 secs 2024-07-26T11:47:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:47:07.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:47:07.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:47:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:06 smithi189 ceph-mon[36790]: pgmap v1468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1184.485607 secs 2024-07-26T11:47:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:47:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:47:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:47:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:07 smithi193 ceph-mon[37473]: pgmap v1469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:47:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:07 smithi193 ceph-mon[37473]: pgmap v1470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:47:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:07 smithi134 ceph-mon[31966]: pgmap v1469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:47:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:07 smithi134 ceph-mon[31966]: pgmap v1470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:47:08.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:07 smithi189 ceph-mon[36790]: pgmap v1469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:47:08.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:07 smithi189 ceph-mon[36790]: pgmap v1470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:47:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1170.059419 secs 2024-07-26T11:47:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1170.059419 secs 2024-07-26T11:47:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1170.059419 secs 2024-07-26T11:47:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:09 smithi193 ceph-mon[37473]: pgmap v1471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:47:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:09 smithi134 ceph-mon[31966]: pgmap v1471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:47:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:09 smithi189 ceph-mon[36790]: pgmap v1471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:47:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1189.485738 secs 2024-07-26T11:47:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:11 smithi193 ceph-mon[37473]: pgmap v1472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:47:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1189.485738 secs 2024-07-26T11:47:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:11 smithi134 ceph-mon[31966]: pgmap v1472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:47:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1189.485738 secs 2024-07-26T11:47:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:11 smithi189 ceph-mon[36790]: pgmap v1472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:47:14.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:47:14.257+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:47:14.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:47:14.257+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:47:14.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:47:14.257+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:13 smithi193 ceph-mon[37473]: pgmap v1473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:47:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1175.059564 secs 2024-07-26T11:47:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:13 smithi134 ceph-mon[31966]: pgmap v1473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:47:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1175.059564 secs 2024-07-26T11:47:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:13 smithi189 ceph-mon[36790]: pgmap v1473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:47:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1175.059564 secs 2024-07-26T11:47:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:15 smithi193 ceph-mon[37473]: pgmap v1474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:47:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:15 smithi134 ceph-mon[31966]: pgmap v1474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:47:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:15 smithi189 ceph-mon[36790]: pgmap v1474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:47:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1194.485855 secs 2024-07-26T11:47:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1194.485855 secs 2024-07-26T11:47:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1194.485855 secs 2024-07-26T11:47:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:18 smithi193 ceph-mon[37473]: pgmap v1475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:47:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:18 smithi134 ceph-mon[31966]: pgmap v1475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:47:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:18 smithi189 ceph-mon[36790]: pgmap v1475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:47:18.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:47:18.721+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:47:18.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:47:18.721+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:47:18.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:47:18.721+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1180.059635 secs 2024-07-26T11:47:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1180.059635 secs 2024-07-26T11:47:19.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1180.059635 secs 2024-07-26T11:47:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:20 smithi193 ceph-mon[37473]: pgmap v1476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:47:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:20 smithi134 ceph-mon[31966]: pgmap v1476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:47:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:20 smithi189 ceph-mon[36790]: pgmap v1476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:47:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1199.485970 secs 2024-07-26T11:47:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:22 smithi193 ceph-mon[37473]: pgmap v1477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1199.485970 secs 2024-07-26T11:47:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:22 smithi134 ceph-mon[31966]: pgmap v1477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1199.485970 secs 2024-07-26T11:47:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:22 smithi189 ceph-mon[36790]: pgmap v1477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:24 smithi134 ceph-mon[31966]: pgmap v1478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:47:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1185.059785 secs 2024-07-26T11:47:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:24 smithi189 ceph-mon[36790]: pgmap v1478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:47:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1185.059785 secs 2024-07-26T11:47:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:24 smithi193 ceph-mon[37473]: pgmap v1478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:47:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1185.059785 secs 2024-07-26T11:47:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:26 smithi134 ceph-mon[31966]: pgmap v1479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:47:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:26 smithi189 ceph-mon[36790]: pgmap v1479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:47:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:26 smithi193 ceph-mon[37473]: pgmap v1479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:47:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1204.486082 secs 2024-07-26T11:47:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1204.486082 secs 2024-07-26T11:47:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1204.486082 secs 2024-07-26T11:47:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:28 smithi134 ceph-mon[31966]: pgmap v1480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:28 smithi189 ceph-mon[36790]: pgmap v1480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:28 smithi193 ceph-mon[37473]: pgmap v1480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:47:29.272+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:47:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:47:29.272+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:47:29.274 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:47:29.272+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1190.059904 secs 2024-07-26T11:47:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1190.059904 secs 2024-07-26T11:47:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1190.059904 secs 2024-07-26T11:47:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:30 smithi134 ceph-mon[31966]: pgmap v1481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:47:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:30 smithi189 ceph-mon[36790]: pgmap v1481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:47:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:30 smithi193 ceph-mon[37473]: pgmap v1481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:47:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1209.486215 secs 2024-07-26T11:47:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:32 smithi134 ceph-mon[31966]: pgmap v1482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1209.486215 secs 2024-07-26T11:47:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:32 smithi189 ceph-mon[36790]: pgmap v1482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1209.486215 secs 2024-07-26T11:47:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:32 smithi193 ceph-mon[37473]: pgmap v1482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:33.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:47:33.734+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:47:33.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:47:33.734+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:33.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:47:33.734+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:47:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:34 smithi134 ceph-mon[31966]: pgmap v1483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:47:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1195.060048 secs 2024-07-26T11:47:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:34 smithi189 ceph-mon[36790]: pgmap v1483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:47:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1195.060048 secs 2024-07-26T11:47:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:34 smithi193 ceph-mon[37473]: pgmap v1483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:47:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1195.060048 secs 2024-07-26T11:47:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:36 smithi134 ceph-mon[31966]: pgmap v1484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:47:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:36 smithi189 ceph-mon[36790]: pgmap v1484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:47:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:36 smithi193 ceph-mon[37473]: pgmap v1484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:47:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1214.486343 secs 2024-07-26T11:47:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1214.486343 secs 2024-07-26T11:47:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1214.486343 secs 2024-07-26T11:47:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:38 smithi134 ceph-mon[31966]: pgmap v1485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:38 smithi189 ceph-mon[36790]: pgmap v1485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:38 smithi193 ceph-mon[37473]: pgmap v1485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1200.060168 secs 2024-07-26T11:47:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1200.060168 secs 2024-07-26T11:47:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1200.060168 secs 2024-07-26T11:47:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:40 smithi134 ceph-mon[31966]: pgmap v1486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:47:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:40 smithi189 ceph-mon[36790]: pgmap v1486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:47:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:40 smithi193 ceph-mon[37473]: pgmap v1486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:47:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1219.486498 secs 2024-07-26T11:47:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1219.486498 secs 2024-07-26T11:47:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1219.486498 secs 2024-07-26T11:47:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:42 smithi134 ceph-mon[31966]: pgmap v1487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:42.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:42 smithi193 ceph-mon[37473]: pgmap v1487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:42 smithi189 ceph-mon[36790]: pgmap v1487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:44.289 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:47:44.288+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:47:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:47:44.288+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:47:44.290 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:47:44.288+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:44 smithi134 ceph-mon[31966]: pgmap v1488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:47:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1205.060305 secs 2024-07-26T11:47:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:44 smithi189 ceph-mon[36790]: pgmap v1488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:47:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1205.060305 secs 2024-07-26T11:47:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:44 smithi193 ceph-mon[37473]: pgmap v1488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:47:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1205.060305 secs 2024-07-26T11:47:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:46 smithi134 ceph-mon[31966]: pgmap v1489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:47:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1224.486634 secs 2024-07-26T11:47:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:46 smithi189 ceph-mon[36790]: pgmap v1489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:47:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1224.486634 secs 2024-07-26T11:47:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:46 smithi193 ceph-mon[37473]: pgmap v1489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:47:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1224.486634 secs 2024-07-26T11:47:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:48 smithi134 ceph-mon[31966]: pgmap v1490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:47:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:48 smithi189 ceph-mon[36790]: pgmap v1490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:47:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:48 smithi193 ceph-mon[37473]: pgmap v1490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:47:48.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:47:48.744+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:47:48.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:47:48.744+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:48.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:47:48.744+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:47:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1210.060455 secs 2024-07-26T11:47:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1210.060455 secs 2024-07-26T11:47:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1210.060455 secs 2024-07-26T11:47:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:50 smithi134 ceph-mon[31966]: pgmap v1491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:47:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:50 smithi189 ceph-mon[36790]: pgmap v1491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:47:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:50 smithi193 ceph-mon[37473]: pgmap v1491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:47:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1229.486772 secs 2024-07-26T11:47:51.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:51 smithi189 ceph-mon[36790]: pgmap v1492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:47:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1229.486772 secs 2024-07-26T11:47:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:51 smithi193 ceph-mon[37473]: pgmap v1492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:47:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1229.486772 secs 2024-07-26T11:47:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:51 smithi134 ceph-mon[31966]: pgmap v1492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:47:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:53 smithi189 ceph-mon[36790]: pgmap v1493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:47:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1215.060571 secs 2024-07-26T11:47:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:53 smithi193 ceph-mon[37473]: pgmap v1493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:47:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1215.060571 secs 2024-07-26T11:47:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:53 smithi134 ceph-mon[31966]: pgmap v1493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:47:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1215.060571 secs 2024-07-26T11:47:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:55 smithi189 ceph-mon[36790]: pgmap v1494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:47:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:55 smithi193 ceph-mon[37473]: pgmap v1494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:47:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:55 smithi134 ceph-mon[31966]: pgmap v1494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:47:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1234.486903 secs 2024-07-26T11:47:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1234.486903 secs 2024-07-26T11:47:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1234.486903 secs 2024-07-26T11:47:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:57 smithi193 ceph-mon[37473]: pgmap v1495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:57 smithi134 ceph-mon[31966]: pgmap v1495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:57 smithi189 ceph-mon[36790]: pgmap v1495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:47:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1220.060725 secs 2024-07-26T11:47:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:47:59.294+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:47:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:47:59.294+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:47:59.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:47:59.294+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:47:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1220.060725 secs 2024-07-26T11:47:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1220.060725 secs 2024-07-26T11:48:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:47:59 smithi193 ceph-mon[37473]: pgmap v1496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:47:59 smithi134 ceph-mon[31966]: pgmap v1496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:47:59 smithi189 ceph-mon[36790]: pgmap v1496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1239.487046 secs 2024-07-26T11:48:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:01 smithi193 ceph-mon[37473]: pgmap v1497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:02.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1239.487046 secs 2024-07-26T11:48:02.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:01 smithi134 ceph-mon[31966]: pgmap v1497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1239.487046 secs 2024-07-26T11:48:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:01 smithi189 ceph-mon[36790]: pgmap v1497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:03.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:48:03.760+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:48:03.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:48:03.760+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:48:03.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:48:03.760+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:03 smithi134 ceph-mon[31966]: pgmap v1498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1225.060866 secs 2024-07-26T11:48:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:03 smithi193 ceph-mon[37473]: pgmap v1498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1225.060866 secs 2024-07-26T11:48:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:03 smithi189 ceph-mon[36790]: pgmap v1498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1225.060866 secs 2024-07-26T11:48:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:05 smithi134 ceph-mon[31966]: pgmap v1499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:05 smithi193 ceph-mon[37473]: pgmap v1499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:05 smithi189 ceph-mon[36790]: pgmap v1499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1244.487176 secs 2024-07-26T11:48:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1244.487176 secs 2024-07-26T11:48:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1244.487176 secs 2024-07-26T11:48:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:07 smithi193 ceph-mon[37473]: pgmap v1500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:48:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:07 smithi134 ceph-mon[31966]: pgmap v1500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:48:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:07 smithi189 ceph-mon[36790]: pgmap v1500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:08.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:48:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:48:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:48:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:09 smithi193 ceph-mon[37473]: pgmap v1501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T11:48:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:09 smithi193 ceph-mon[37473]: pgmap v1502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:48:09.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:48:09.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1230.060975 secs 2024-07-26T11:48:09.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:48:09.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:48:09.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:09 smithi134 ceph-mon[31966]: pgmap v1501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T11:48:09.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:09 smithi134 ceph-mon[31966]: pgmap v1502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:48:09.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:48:09.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1230.060975 secs 2024-07-26T11:48:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:48:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:48:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:09 smithi189 ceph-mon[36790]: pgmap v1501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T11:48:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:09 smithi189 ceph-mon[36790]: pgmap v1502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T11:48:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:48:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1230.060975 secs 2024-07-26T11:48:11.017 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:10 smithi134 ceph-mon[31966]: pgmap v1503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:10 smithi193 ceph-mon[37473]: pgmap v1503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:10 smithi189 ceph-mon[36790]: pgmap v1503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1249.487299 secs 2024-07-26T11:48:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1249.487299 secs 2024-07-26T11:48:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1249.487299 secs 2024-07-26T11:48:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:12 smithi193 ceph-mon[37473]: pgmap v1504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T11:48:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:12 smithi134 ceph-mon[31966]: pgmap v1504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T11:48:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:12 smithi189 ceph-mon[36790]: pgmap v1504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T11:48:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1235.061132 secs 2024-07-26T11:48:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1235.061132 secs 2024-07-26T11:48:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1235.061132 secs 2024-07-26T11:48:14.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:48:14.309+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:48:14.311 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:48:14.310+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:48:14.312 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:48:14.310+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:14 smithi193 ceph-mon[37473]: pgmap v1505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:14 smithi134 ceph-mon[31966]: pgmap v1505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:14 smithi189 ceph-mon[36790]: pgmap v1505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1254.487424 secs 2024-07-26T11:48:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1254.487424 secs 2024-07-26T11:48:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1254.487424 secs 2024-07-26T11:48:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:17 smithi134 ceph-mon[31966]: pgmap v1506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:17 smithi189 ceph-mon[36790]: pgmap v1506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:17 smithi193 ceph-mon[37473]: pgmap v1506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T11:48:18.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:48:18.775+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:48:18.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:48:18.775+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:48:18.779 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:48:18.775+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:19 smithi134 ceph-mon[31966]: pgmap v1507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:48:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1240.061278 secs 2024-07-26T11:48:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:19 smithi189 ceph-mon[36790]: pgmap v1507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:48:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1240.061278 secs 2024-07-26T11:48:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:19 smithi193 ceph-mon[37473]: pgmap v1507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:48:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1240.061278 secs 2024-07-26T11:48:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:21 smithi134 ceph-mon[31966]: pgmap v1508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1259.487492 secs 2024-07-26T11:48:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:21 smithi189 ceph-mon[36790]: pgmap v1508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:21.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1259.487492 secs 2024-07-26T11:48:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:21 smithi193 ceph-mon[37473]: pgmap v1508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1259.487492 secs 2024-07-26T11:48:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:22 smithi134 ceph-mon[31966]: pgmap v1509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:48:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:22 smithi189 ceph-mon[36790]: pgmap v1509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:48:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:22 smithi193 ceph-mon[37473]: pgmap v1509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:48:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:48:22.548+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T11:48:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:48:22.549+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T11:48:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:48:22.549+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T11:48:22.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:48:22.549+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T11:48:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1245.061423 secs 2024-07-26T11:48:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1245.061423 secs 2024-07-26T11:48:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1245.061423 secs 2024-07-26T11:48:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:24 smithi134 ceph-mon[31966]: pgmap v1510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:24 smithi189 ceph-mon[36790]: pgmap v1510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:24 smithi193 ceph-mon[37473]: pgmap v1510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1264.487607 secs 2024-07-26T11:48:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1264.487607 secs 2024-07-26T11:48:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1264.487607 secs 2024-07-26T11:48:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:27 smithi134 ceph-mon[31966]: pgmap v1511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:27 smithi189 ceph-mon[36790]: pgmap v1511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:27 smithi193 ceph-mon[37473]: pgmap v1511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:48:29.325+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:48:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:48:29.325+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:48:29.327 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:48:29.325+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:29 smithi134 ceph-mon[31966]: pgmap v1512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1250.061537 secs 2024-07-26T11:48:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:29 smithi189 ceph-mon[36790]: pgmap v1512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1250.061537 secs 2024-07-26T11:48:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:29 smithi193 ceph-mon[37473]: pgmap v1512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1250.061537 secs 2024-07-26T11:48:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:31 smithi134 ceph-mon[31966]: pgmap v1513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1269.487736 secs 2024-07-26T11:48:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:31 smithi189 ceph-mon[36790]: pgmap v1513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1269.487736 secs 2024-07-26T11:48:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:31 smithi193 ceph-mon[37473]: pgmap v1513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1269.487736 secs 2024-07-26T11:48:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:32 smithi134 ceph-mon[31966]: pgmap v1514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:48:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:32 smithi189 ceph-mon[36790]: pgmap v1514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:48:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:32 smithi193 ceph-mon[37473]: pgmap v1514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:48:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:48:33.048+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194003690 con 0x55a08e2d94d0 2024-07-26T11:48:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:48:33.048+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T11:48:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:48:33.049+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194003690 con 0x55a08e2d94d0 2024-07-26T11:48:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:48:33.049+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T11:48:33.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:48:33.791+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:48:33.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:48:33.791+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:48:33.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:48:33.791+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1255.061677 secs 2024-07-26T11:48:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1255.061677 secs 2024-07-26T11:48:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1255.061677 secs 2024-07-26T11:48:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:34 smithi134 ceph-mon[31966]: pgmap v1515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:34 smithi189 ceph-mon[36790]: pgmap v1515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:34 smithi193 ceph-mon[37473]: pgmap v1515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1274.487877 secs 2024-07-26T11:48:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1274.487877 secs 2024-07-26T11:48:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1274.487877 secs 2024-07-26T11:48:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:37 smithi134 ceph-mon[31966]: pgmap v1516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:37 smithi189 ceph-mon[36790]: pgmap v1516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:37 smithi193 ceph-mon[37473]: pgmap v1516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:39 smithi134 ceph-mon[31966]: pgmap v1517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1260.061834 secs 2024-07-26T11:48:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:39 smithi189 ceph-mon[36790]: pgmap v1517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1260.061834 secs 2024-07-26T11:48:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:39 smithi193 ceph-mon[37473]: pgmap v1517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1260.061834 secs 2024-07-26T11:48:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:41 smithi134 ceph-mon[31966]: pgmap v1518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1279.487989 secs 2024-07-26T11:48:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:41 smithi189 ceph-mon[36790]: pgmap v1518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1279.487989 secs 2024-07-26T11:48:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:41 smithi193 ceph-mon[37473]: pgmap v1518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1279.487989 secs 2024-07-26T11:48:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:42 smithi193 ceph-mon[37473]: pgmap v1519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:48:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:42 smithi134 ceph-mon[31966]: pgmap v1519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:48:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:42 smithi189 ceph-mon[36790]: pgmap v1519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:48:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1265.061973 secs 2024-07-26T11:48:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1265.061973 secs 2024-07-26T11:48:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1265.061973 secs 2024-07-26T11:48:44.343 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:48:44.341+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:48:44.343 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:48:44.341+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:48:44.343 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:48:44.341+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:44 smithi134 ceph-mon[31966]: pgmap v1520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:44 smithi189 ceph-mon[36790]: pgmap v1520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:44 smithi193 ceph-mon[37473]: pgmap v1520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1284.488114 secs 2024-07-26T11:48:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1284.488114 secs 2024-07-26T11:48:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1284.488114 secs 2024-07-26T11:48:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:47 smithi134 ceph-mon[31966]: pgmap v1521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:47 smithi189 ceph-mon[36790]: pgmap v1521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:47 smithi193 ceph-mon[37473]: pgmap v1521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:48.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:48:48.792+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:48:48.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:48:48.792+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:48.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:48:48.792+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:48:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:49 smithi134 ceph-mon[31966]: pgmap v1522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1270.062074 secs 2024-07-26T11:48:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:49 smithi189 ceph-mon[36790]: pgmap v1522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1270.062074 secs 2024-07-26T11:48:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:49 smithi193 ceph-mon[37473]: pgmap v1522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1270.062074 secs 2024-07-26T11:48:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:51 smithi134 ceph-mon[31966]: pgmap v1523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:51.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1289.488240 secs 2024-07-26T11:48:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:51 smithi189 ceph-mon[36790]: pgmap v1523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:51.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1289.488240 secs 2024-07-26T11:48:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:51 smithi193 ceph-mon[37473]: pgmap v1523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:48:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1289.488240 secs 2024-07-26T11:48:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:52 smithi193 ceph-mon[37473]: pgmap v1524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:48:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:52 smithi134 ceph-mon[31966]: pgmap v1524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:48:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:52 smithi189 ceph-mon[36790]: pgmap v1524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:48:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1275.062154 secs 2024-07-26T11:48:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1275.062154 secs 2024-07-26T11:48:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1275.062154 secs 2024-07-26T11:48:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:54 smithi134 ceph-mon[31966]: pgmap v1525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:54 smithi189 ceph-mon[36790]: pgmap v1525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:54 smithi193 ceph-mon[37473]: pgmap v1525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:48:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1294.488368 secs 2024-07-26T11:48:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1294.488368 secs 2024-07-26T11:48:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1294.488368 secs 2024-07-26T11:48:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:57 smithi134 ceph-mon[31966]: pgmap v1526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:57 smithi189 ceph-mon[36790]: pgmap v1526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:57 smithi193 ceph-mon[37473]: pgmap v1526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:48:59.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:48:59.356+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:48:59.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:48:59.356+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:48:59.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:48:59.356+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:48:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:59 smithi134 ceph-mon[31966]: pgmap v1527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:48:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1280.062290 secs 2024-07-26T11:48:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:59 smithi189 ceph-mon[36790]: pgmap v1527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:48:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1280.062290 secs 2024-07-26T11:48:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:59 smithi193 ceph-mon[37473]: pgmap v1527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:48:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:48:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1280.062290 secs 2024-07-26T11:49:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:01 smithi134 ceph-mon[31966]: pgmap v1528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1299.488463 secs 2024-07-26T11:49:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:01 smithi189 ceph-mon[36790]: pgmap v1528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1299.488463 secs 2024-07-26T11:49:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:01 smithi193 ceph-mon[37473]: pgmap v1528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1299.488463 secs 2024-07-26T11:49:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:02 smithi193 ceph-mon[37473]: pgmap v1529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:49:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:02 smithi134 ceph-mon[31966]: pgmap v1529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:49:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:02 smithi189 ceph-mon[36790]: pgmap v1529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:49:03.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:49:03.807+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:49:03.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:49:03.807+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:03.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:49:03.807+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:49:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1285.062448 secs 2024-07-26T11:49:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1285.062448 secs 2024-07-26T11:49:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1285.062448 secs 2024-07-26T11:49:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:04 smithi134 ceph-mon[31966]: pgmap v1530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T11:49:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:04 smithi189 ceph-mon[36790]: pgmap v1530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T11:49:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:04 smithi193 ceph-mon[37473]: pgmap v1530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T11:49:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1304.488586 secs 2024-07-26T11:49:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1304.488586 secs 2024-07-26T11:49:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1304.488586 secs 2024-07-26T11:49:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:07 smithi134 ceph-mon[31966]: pgmap v1531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:07 smithi189 ceph-mon[36790]: pgmap v1531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:07 smithi193 ceph-mon[37473]: pgmap v1531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:09 smithi134 ceph-mon[31966]: pgmap v1532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1290.062561 secs 2024-07-26T11:49:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:49:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:09 smithi189 ceph-mon[36790]: pgmap v1532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1290.062561 secs 2024-07-26T11:49:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:49:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:09 smithi193 ceph-mon[37473]: pgmap v1532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1290.062561 secs 2024-07-26T11:49:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:49:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:49:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:49:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:49:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:49:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:49:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:49:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:49:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:49:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:49:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:11 smithi189 ceph-mon[36790]: pgmap v1533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:49:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:11 smithi189 ceph-mon[36790]: pgmap v1534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:49:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1309.488710 secs 2024-07-26T11:49:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:11 smithi193 ceph-mon[37473]: pgmap v1533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:49:11.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:11 smithi193 ceph-mon[37473]: pgmap v1534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:49:11.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1309.488710 secs 2024-07-26T11:49:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:11 smithi134 ceph-mon[31966]: pgmap v1533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:49:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:11 smithi134 ceph-mon[31966]: pgmap v1534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:49:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1309.488710 secs 2024-07-26T11:49:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:12 smithi134 ceph-mon[31966]: pgmap v1535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:49:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:12 smithi189 ceph-mon[36790]: pgmap v1535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:49:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:12 smithi193 ceph-mon[37473]: pgmap v1535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:49:14.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:49:14.359+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:49:14.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:49:14.359+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:49:14.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:49:14.359+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1295.062680 secs 2024-07-26T11:49:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:14 smithi134 ceph-mon[31966]: pgmap v1536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:49:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1295.062680 secs 2024-07-26T11:49:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:14 smithi189 ceph-mon[36790]: pgmap v1536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:49:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1295.062680 secs 2024-07-26T11:49:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:14 smithi193 ceph-mon[37473]: pgmap v1536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:49:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:16 smithi134 ceph-mon[31966]: pgmap v1537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:49:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1314.488839 secs 2024-07-26T11:49:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:16 smithi189 ceph-mon[36790]: pgmap v1537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:49:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1314.488839 secs 2024-07-26T11:49:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:16 smithi193 ceph-mon[37473]: pgmap v1537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:49:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1314.488839 secs 2024-07-26T11:49:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:18 smithi134 ceph-mon[31966]: pgmap v1538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:49:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:18 smithi189 ceph-mon[36790]: pgmap v1538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:49:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:18 smithi193 ceph-mon[37473]: pgmap v1538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:49:18.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:49:18.823+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:49:18.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:49:18.823+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:18.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:49:18.823+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:49:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1300.062793 secs 2024-07-26T11:49:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1300.062793 secs 2024-07-26T11:49:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1300.062793 secs 2024-07-26T11:49:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:20 smithi134 ceph-mon[31966]: pgmap v1539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:49:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:20 smithi189 ceph-mon[36790]: pgmap v1539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:49:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:20 smithi193 ceph-mon[37473]: pgmap v1539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:49:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1319.488968 secs 2024-07-26T11:49:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1319.488968 secs 2024-07-26T11:49:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1319.488968 secs 2024-07-26T11:49:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:22 smithi134 ceph-mon[31966]: pgmap v1540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:22 smithi189 ceph-mon[36790]: pgmap v1540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:22 smithi193 ceph-mon[37473]: pgmap v1540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1305.062974 secs 2024-07-26T11:49:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:24 smithi134 ceph-mon[31966]: pgmap v1541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1305.062974 secs 2024-07-26T11:49:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:24 smithi189 ceph-mon[36790]: pgmap v1541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1305.062974 secs 2024-07-26T11:49:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:24 smithi193 ceph-mon[37473]: pgmap v1541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:26 smithi134 ceph-mon[31966]: pgmap v1542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1324.489070 secs 2024-07-26T11:49:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:26 smithi189 ceph-mon[36790]: pgmap v1542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1324.489070 secs 2024-07-26T11:49:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:26 smithi193 ceph-mon[37473]: pgmap v1542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1324.489070 secs 2024-07-26T11:49:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:28 smithi134 ceph-mon[31966]: pgmap v1543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:28 smithi189 ceph-mon[36790]: pgmap v1543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:28 smithi193 ceph-mon[37473]: pgmap v1543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:29.376 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:49:29.374+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:49:29.376 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:49:29.375+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:29.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:49:29.375+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:49:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1310.063097 secs 2024-07-26T11:49:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1310.063097 secs 2024-07-26T11:49:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1310.063097 secs 2024-07-26T11:49:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:30 smithi134 ceph-mon[31966]: pgmap v1544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:30 smithi189 ceph-mon[36790]: pgmap v1544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:30 smithi193 ceph-mon[37473]: pgmap v1544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1329.489199 secs 2024-07-26T11:49:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1329.489199 secs 2024-07-26T11:49:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1329.489199 secs 2024-07-26T11:49:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:32 smithi189 ceph-mon[36790]: pgmap v1545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:32 smithi193 ceph-mon[37473]: pgmap v1545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:32 smithi134 ceph-mon[31966]: pgmap v1545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:33.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:49:33.838+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:49:33.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:49:33.838+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:33.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:49:33.838+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:49:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1315.063176 secs 2024-07-26T11:49:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:34 smithi189 ceph-mon[36790]: pgmap v1546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1315.063176 secs 2024-07-26T11:49:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:34 smithi193 ceph-mon[37473]: pgmap v1546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:34.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1315.063176 secs 2024-07-26T11:49:34.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:34 smithi134 ceph-mon[31966]: pgmap v1546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:36 smithi193 ceph-mon[37473]: pgmap v1547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1334.489353 secs 2024-07-26T11:49:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:36 smithi134 ceph-mon[31966]: pgmap v1547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1334.489353 secs 2024-07-26T11:49:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:36 smithi189 ceph-mon[36790]: pgmap v1547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1334.489353 secs 2024-07-26T11:49:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:38 smithi193 ceph-mon[37473]: pgmap v1548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:38 smithi134 ceph-mon[31966]: pgmap v1548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:38 smithi189 ceph-mon[36790]: pgmap v1548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1320.063286 secs 2024-07-26T11:49:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1320.063286 secs 2024-07-26T11:49:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1320.063286 secs 2024-07-26T11:49:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:40 smithi193 ceph-mon[37473]: pgmap v1549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:40 smithi134 ceph-mon[31966]: pgmap v1549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:40 smithi189 ceph-mon[36790]: pgmap v1549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1339.489421 secs 2024-07-26T11:49:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1339.489421 secs 2024-07-26T11:49:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1339.489421 secs 2024-07-26T11:49:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:42 smithi193 ceph-mon[37473]: pgmap v1550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:42 smithi134 ceph-mon[31966]: pgmap v1550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:42 smithi189 ceph-mon[36790]: pgmap v1550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:49:44.376+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:49:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:49:44.376+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:44.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:49:44.376+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:49:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1325.063375 secs 2024-07-26T11:49:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:44 smithi193 ceph-mon[37473]: pgmap v1551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1325.063375 secs 2024-07-26T11:49:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:44 smithi134 ceph-mon[31966]: pgmap v1551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1325.063375 secs 2024-07-26T11:49:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:44 smithi189 ceph-mon[36790]: pgmap v1551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:46 smithi193 ceph-mon[37473]: pgmap v1552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1344.489536 secs 2024-07-26T11:49:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:46 smithi134 ceph-mon[31966]: pgmap v1552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1344.489536 secs 2024-07-26T11:49:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:46 smithi189 ceph-mon[36790]: pgmap v1552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1344.489536 secs 2024-07-26T11:49:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:48 smithi193 ceph-mon[37473]: pgmap v1553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:48.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:49:48.843+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:49:48.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:49:48.843+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:48.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:49:48.843+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:49:48.846 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:48 smithi134 ceph-mon[31966]: pgmap v1553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:48.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:48 smithi189 ceph-mon[36790]: pgmap v1553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1330.063459 secs 2024-07-26T11:49:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1330.063459 secs 2024-07-26T11:49:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1330.063459 secs 2024-07-26T11:49:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:50 smithi193 ceph-mon[37473]: pgmap v1554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:50 smithi134 ceph-mon[31966]: pgmap v1554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:50 smithi189 ceph-mon[36790]: pgmap v1554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:49:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1349.489645 secs 2024-07-26T11:49:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1349.489645 secs 2024-07-26T11:49:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1349.489645 secs 2024-07-26T11:49:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:52 smithi134 ceph-mon[31966]: pgmap v1555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:52 smithi189 ceph-mon[36790]: pgmap v1555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:52 smithi193 ceph-mon[37473]: pgmap v1555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1335.063565 secs 2024-07-26T11:49:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:54 smithi134 ceph-mon[31966]: pgmap v1556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1335.063565 secs 2024-07-26T11:49:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:54 smithi189 ceph-mon[36790]: pgmap v1556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1335.063565 secs 2024-07-26T11:49:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:54 smithi193 ceph-mon[37473]: pgmap v1556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:49:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:56 smithi134 ceph-mon[31966]: pgmap v1557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1354.489789 secs 2024-07-26T11:49:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:56 smithi189 ceph-mon[36790]: pgmap v1557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1354.489789 secs 2024-07-26T11:49:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:56 smithi193 ceph-mon[37473]: pgmap v1557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:49:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1354.489789 secs 2024-07-26T11:49:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:58 smithi134 ceph-mon[31966]: pgmap v1558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:58.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:58 smithi189 ceph-mon[36790]: pgmap v1558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:58 smithi193 ceph-mon[37473]: pgmap v1558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:49:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:49:59.386+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:49:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:49:59.386+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:49:59.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:49:59.386+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:49:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:49:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1340.063687 secs 2024-07-26T11:49:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:49:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1340.063687 secs 2024-07-26T11:50:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:49:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1340.063687 secs 2024-07-26T11:50:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:00 smithi134 ceph-mon[31966]: pgmap v1559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:00.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:00 smithi134 ceph-mon[31966]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T11:50:00.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:00 smithi134 ceph-mon[31966]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-07-26T11:50:00.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:00 smithi134 ceph-mon[31966]: daemon mds.d on smithi134 is in error state 2024-07-26T11:50:00.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:00 smithi134 ceph-mon[31966]: daemon mds.g on smithi134 is in error state 2024-07-26T11:50:00.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:00 smithi134 ceph-mon[31966]: daemon mds.a on smithi134 is in error state 2024-07-26T11:50:00.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:00 smithi134 ceph-mon[31966]: daemon mds.e on smithi189 is in error state 2024-07-26T11:50:00.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:00 smithi134 ceph-mon[31966]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-07-26T11:50:00.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:00 smithi134 ceph-mon[31966]: mds.f(mds.0): 2 slow requests are blocked > 180 secs 2024-07-26T11:50:00.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:00 smithi134 ceph-mon[31966]: mds.c(mds.3): 1 slow requests are blocked > 180 secs 2024-07-26T11:50:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:00 smithi189 ceph-mon[36790]: pgmap v1559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:00 smithi189 ceph-mon[36790]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T11:50:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:00 smithi189 ceph-mon[36790]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-07-26T11:50:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:00 smithi189 ceph-mon[36790]: daemon mds.d on smithi134 is in error state 2024-07-26T11:50:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:00 smithi189 ceph-mon[36790]: daemon mds.g on smithi134 is in error state 2024-07-26T11:50:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:00 smithi189 ceph-mon[36790]: daemon mds.a on smithi134 is in error state 2024-07-26T11:50:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:00 smithi189 ceph-mon[36790]: daemon mds.e on smithi189 is in error state 2024-07-26T11:50:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:00 smithi189 ceph-mon[36790]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-07-26T11:50:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:00 smithi189 ceph-mon[36790]: mds.f(mds.0): 2 slow requests are blocked > 180 secs 2024-07-26T11:50:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:00 smithi189 ceph-mon[36790]: mds.c(mds.3): 1 slow requests are blocked > 180 secs 2024-07-26T11:50:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:00 smithi193 ceph-mon[37473]: pgmap v1559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:00 smithi193 ceph-mon[37473]: Health detail: HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T11:50:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:00 smithi193 ceph-mon[37473]: [WRN] CEPHADM_FAILED_DAEMON: 4 failed cephadm daemon(s) 2024-07-26T11:50:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:00 smithi193 ceph-mon[37473]: daemon mds.d on smithi134 is in error state 2024-07-26T11:50:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:00 smithi193 ceph-mon[37473]: daemon mds.g on smithi134 is in error state 2024-07-26T11:50:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:00 smithi193 ceph-mon[37473]: daemon mds.a on smithi134 is in error state 2024-07-26T11:50:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:00 smithi193 ceph-mon[37473]: daemon mds.e on smithi189 is in error state 2024-07-26T11:50:01.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:00 smithi193 ceph-mon[37473]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2024-07-26T11:50:01.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:00 smithi193 ceph-mon[37473]: mds.f(mds.0): 2 slow requests are blocked > 180 secs 2024-07-26T11:50:01.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:00 smithi193 ceph-mon[37473]: mds.c(mds.3): 1 slow requests are blocked > 180 secs 2024-07-26T11:50:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1359.489867 secs 2024-07-26T11:50:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1359.489867 secs 2024-07-26T11:50:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1359.489867 secs 2024-07-26T11:50:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:02 smithi134 ceph-mon[31966]: pgmap v1560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:02 smithi189 ceph-mon[36790]: pgmap v1560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:02 smithi193 ceph-mon[37473]: pgmap v1560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:03.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:50:03.859+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:50:03.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:50:03.859+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:03.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:50:03.859+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:50:03.861 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1345.063839 secs 2024-07-26T11:50:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1345.063839 secs 2024-07-26T11:50:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1345.063839 secs 2024-07-26T11:50:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:04 smithi134 ceph-mon[31966]: pgmap v1561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:04 smithi189 ceph-mon[36790]: pgmap v1561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:04 smithi193 ceph-mon[37473]: pgmap v1561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:06 smithi134 ceph-mon[31966]: pgmap v1562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:06.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1364.489995 secs 2024-07-26T11:50:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:06 smithi189 ceph-mon[36790]: pgmap v1562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1364.489995 secs 2024-07-26T11:50:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:06 smithi193 ceph-mon[37473]: pgmap v1562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1364.489995 secs 2024-07-26T11:50:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:08 smithi134 ceph-mon[31966]: pgmap v1563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:08.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1350.063984 secs 2024-07-26T11:50:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:08 smithi189 ceph-mon[36790]: pgmap v1563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1350.063984 secs 2024-07-26T11:50:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:08 smithi193 ceph-mon[37473]: pgmap v1563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1350.063984 secs 2024-07-26T11:50:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:50:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:50:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:50:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:10 smithi134 ceph-mon[31966]: pgmap v1564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:50:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:50:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:50:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:50:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:50:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:50:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:50:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:10 smithi189 ceph-mon[36790]: pgmap v1564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:50:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:50:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:50:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:50:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:50:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:50:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:50:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:10 smithi193 ceph-mon[37473]: pgmap v1564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:50:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:50:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:50:11.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:50:11.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:50:11.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:50:11.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:50:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:11 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T11:50:11.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:11 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T11:50:11.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:11 smithi134 ceph-mon[31966]: pgmap v1565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T11:50:11.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:11 smithi134 ceph-mon[31966]: pgmap v1566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:50:11.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1369.490122 secs 2024-07-26T11:50:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:11 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T11:50:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:11 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T11:50:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:11 smithi189 ceph-mon[36790]: pgmap v1565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T11:50:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:11 smithi189 ceph-mon[36790]: pgmap v1566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:50:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1369.490122 secs 2024-07-26T11:50:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:11 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T11:50:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:11 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T11:50:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:11 smithi193 ceph-mon[37473]: pgmap v1565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T11:50:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:11 smithi193 ceph-mon[37473]: pgmap v1566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:50:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1369.490122 secs 2024-07-26T11:50:13.897 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:13 smithi134 ceph-mon[31966]: pgmap v1567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:50:13.897 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1355.064062 secs 2024-07-26T11:50:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:13 smithi189 ceph-mon[36790]: pgmap v1567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:50:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1355.064062 secs 2024-07-26T11:50:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:13 smithi193 ceph-mon[37473]: pgmap v1567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:50:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1355.064062 secs 2024-07-26T11:50:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:50:14.391+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:50:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:50:14.391+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:14.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:50:14.391+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:50:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:15 smithi134 ceph-mon[31966]: pgmap v1568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:50:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:15 smithi189 ceph-mon[36790]: pgmap v1568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:50:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:15 smithi193 ceph-mon[37473]: pgmap v1568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:50:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1374.490266 secs 2024-07-26T11:50:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1374.490266 secs 2024-07-26T11:50:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1374.490266 secs 2024-07-26T11:50:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:17 smithi134 ceph-mon[31966]: pgmap v1569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:50:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:17 smithi189 ceph-mon[36790]: pgmap v1569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:50:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:17 smithi193 ceph-mon[37473]: pgmap v1569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:50:18.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:50:18.864+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:50:18.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:50:18.864+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:18.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:50:18.864+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:50:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1360.064159 secs 2024-07-26T11:50:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1360.064159 secs 2024-07-26T11:50:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1360.064159 secs 2024-07-26T11:50:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:19 smithi134 ceph-mon[31966]: pgmap v1570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:50:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:19 smithi189 ceph-mon[36790]: pgmap v1570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:50:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:19 smithi193 ceph-mon[37473]: pgmap v1570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:50:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:21 smithi134 ceph-mon[31966]: pgmap v1571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:50:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1379.490410 secs 2024-07-26T11:50:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:21 smithi189 ceph-mon[36790]: pgmap v1571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:50:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1379.490410 secs 2024-07-26T11:50:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:21 smithi193 ceph-mon[37473]: pgmap v1571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:50:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1379.490410 secs 2024-07-26T11:50:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:23 smithi134 ceph-mon[31966]: pgmap v1572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1365.064268 secs 2024-07-26T11:50:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:23 smithi189 ceph-mon[36790]: pgmap v1572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1365.064268 secs 2024-07-26T11:50:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:23 smithi193 ceph-mon[37473]: pgmap v1572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1365.064268 secs 2024-07-26T11:50:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:25 smithi189 ceph-mon[36790]: pgmap v1573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:25 smithi193 ceph-mon[37473]: pgmap v1573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:25 smithi134 ceph-mon[31966]: pgmap v1573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1384.490544 secs 2024-07-26T11:50:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1384.490544 secs 2024-07-26T11:50:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1384.490544 secs 2024-07-26T11:50:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:27 smithi189 ceph-mon[36790]: pgmap v1574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:27 smithi193 ceph-mon[37473]: pgmap v1574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:27 smithi134 ceph-mon[31966]: pgmap v1574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1370.064390 secs 2024-07-26T11:50:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1370.064390 secs 2024-07-26T11:50:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1370.064390 secs 2024-07-26T11:50:29.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:50:29.406+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:50:29.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:50:29.407+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:50:29.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:50:29.407+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:29 smithi193 ceph-mon[37473]: pgmap v1575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:29 smithi134 ceph-mon[31966]: pgmap v1575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:29 smithi189 ceph-mon[36790]: pgmap v1575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:31 smithi193 ceph-mon[37473]: pgmap v1576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1389.490634 secs 2024-07-26T11:50:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:31 smithi134 ceph-mon[31966]: pgmap v1576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1389.490634 secs 2024-07-26T11:50:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:31 smithi189 ceph-mon[36790]: pgmap v1576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1389.490634 secs 2024-07-26T11:50:33.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:50:33.879+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:50:33.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:50:33.880+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:50:33.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:50:33.880+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:33 smithi193 ceph-mon[37473]: pgmap v1577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1375.064528 secs 2024-07-26T11:50:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:33 smithi134 ceph-mon[31966]: pgmap v1577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1375.064528 secs 2024-07-26T11:50:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:33 smithi189 ceph-mon[36790]: pgmap v1577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1375.064528 secs 2024-07-26T11:50:35.974 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:35 smithi134 ceph-mon[31966]: pgmap v1578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:50:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:35 smithi193 ceph-mon[37473]: pgmap v1578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:50:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:35 smithi189 ceph-mon[36790]: pgmap v1578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:50:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1394.490772 secs 2024-07-26T11:50:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1394.490772 secs 2024-07-26T11:50:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1394.490772 secs 2024-07-26T11:50:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:37 smithi193 ceph-mon[37473]: pgmap v1579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:37 smithi134 ceph-mon[31966]: pgmap v1579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:37 smithi189 ceph-mon[36790]: pgmap v1579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1380.064645 secs 2024-07-26T11:50:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1380.064645 secs 2024-07-26T11:50:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1380.064645 secs 2024-07-26T11:50:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:39 smithi193 ceph-mon[37473]: pgmap v1580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:39 smithi134 ceph-mon[31966]: pgmap v1580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:39 smithi189 ceph-mon[36790]: pgmap v1580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:41 smithi193 ceph-mon[37473]: pgmap v1581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1399.490918 secs 2024-07-26T11:50:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:41 smithi134 ceph-mon[31966]: pgmap v1581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1399.490918 secs 2024-07-26T11:50:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:41 smithi189 ceph-mon[36790]: pgmap v1581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1399.490918 secs 2024-07-26T11:50:44.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:43 smithi193 ceph-mon[37473]: pgmap v1582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:44.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1385.064785 secs 2024-07-26T11:50:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:43 smithi134 ceph-mon[31966]: pgmap v1582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1385.064785 secs 2024-07-26T11:50:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:43 smithi189 ceph-mon[36790]: pgmap v1582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1385.064785 secs 2024-07-26T11:50:44.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:50:44.410+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:50:44.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:50:44.410+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:50:44.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:50:44.410+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:45 smithi134 ceph-mon[31966]: pgmap v1583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:45 smithi189 ceph-mon[36790]: pgmap v1583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:45 smithi193 ceph-mon[37473]: pgmap v1583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1404.491039 secs 2024-07-26T11:50:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1404.491039 secs 2024-07-26T11:50:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1404.491039 secs 2024-07-26T11:50:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:47 smithi134 ceph-mon[31966]: pgmap v1584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:47 smithi189 ceph-mon[36790]: pgmap v1584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:47 smithi193 ceph-mon[37473]: pgmap v1584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:48.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:50:48.893+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:50:48.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:50:48.893+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:50:48.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:50:48.893+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1390.064925 secs 2024-07-26T11:50:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1390.064925 secs 2024-07-26T11:50:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1390.064925 secs 2024-07-26T11:50:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:49 smithi134 ceph-mon[31966]: pgmap v1585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:49 smithi189 ceph-mon[36790]: pgmap v1585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:49 smithi193 ceph-mon[37473]: pgmap v1585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:50:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:51 smithi134 ceph-mon[31966]: pgmap v1586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1409.491140 secs 2024-07-26T11:50:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:51 smithi189 ceph-mon[36790]: pgmap v1586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1409.491140 secs 2024-07-26T11:50:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:51 smithi193 ceph-mon[37473]: pgmap v1586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1409.491140 secs 2024-07-26T11:50:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:53 smithi134 ceph-mon[31966]: pgmap v1587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1395.065060 secs 2024-07-26T11:50:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:53 smithi189 ceph-mon[36790]: pgmap v1587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1395.065060 secs 2024-07-26T11:50:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:53 smithi193 ceph-mon[37473]: pgmap v1587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:50:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1395.065060 secs 2024-07-26T11:50:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:55 smithi134 ceph-mon[31966]: pgmap v1588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:56.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:55 smithi189 ceph-mon[36790]: pgmap v1588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:55 smithi193 ceph-mon[37473]: pgmap v1588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:50:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1414.491264 secs 2024-07-26T11:50:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1414.491264 secs 2024-07-26T11:50:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1414.491264 secs 2024-07-26T11:50:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:57 smithi134 ceph-mon[31966]: pgmap v1589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:57 smithi189 ceph-mon[36790]: pgmap v1589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:57 smithi193 ceph-mon[37473]: pgmap v1589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:50:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1400.065138 secs 2024-07-26T11:50:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1400.065138 secs 2024-07-26T11:50:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1400.065138 secs 2024-07-26T11:50:59.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:50:59.419+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:50:59.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:50:59.419+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:50:59.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:50:59.419+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:50:59 smithi134 ceph-mon[31966]: pgmap v1590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:50:59 smithi189 ceph-mon[36790]: pgmap v1590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:50:59 smithi193 ceph-mon[37473]: pgmap v1590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:01 smithi134 ceph-mon[31966]: pgmap v1591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1419.491393 secs 2024-07-26T11:51:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:01 smithi189 ceph-mon[36790]: pgmap v1591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1419.491393 secs 2024-07-26T11:51:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:01 smithi193 ceph-mon[37473]: pgmap v1591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1419.491393 secs 2024-07-26T11:51:03.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:51:03.908+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:03.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:51:03.908+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:03.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:51:03.908+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:03 smithi134 ceph-mon[31966]: pgmap v1592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:51:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1405.065257 secs 2024-07-26T11:51:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:03 smithi189 ceph-mon[36790]: pgmap v1592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:51:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1405.065257 secs 2024-07-26T11:51:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:03 smithi193 ceph-mon[37473]: pgmap v1592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:51:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1405.065257 secs 2024-07-26T11:51:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:05 smithi134 ceph-mon[31966]: pgmap v1593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:05 smithi189 ceph-mon[36790]: pgmap v1593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:05 smithi193 ceph-mon[37473]: pgmap v1593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1424.491520 secs 2024-07-26T11:51:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1424.491520 secs 2024-07-26T11:51:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1424.491520 secs 2024-07-26T11:51:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:07 smithi134 ceph-mon[31966]: pgmap v1594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:07 smithi189 ceph-mon[36790]: pgmap v1594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:07 smithi193 ceph-mon[37473]: pgmap v1594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1410.065336 secs 2024-07-26T11:51:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1410.065336 secs 2024-07-26T11:51:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1410.065336 secs 2024-07-26T11:51:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:09 smithi189 ceph-mon[36790]: pgmap v1595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:09 smithi193 ceph-mon[37473]: pgmap v1595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:09 smithi134 ceph-mon[31966]: pgmap v1595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:51:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:51:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:51:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:11 smithi193 ceph-mon[37473]: pgmap v1596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:12.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1429.491610 secs 2024-07-26T11:51:12.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:51:12.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:51:12.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:51:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:11 smithi134 ceph-mon[31966]: pgmap v1596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:12.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1429.491610 secs 2024-07-26T11:51:12.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:51:12.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:51:12.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:51:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:11 smithi189 ceph-mon[36790]: pgmap v1596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1429.491610 secs 2024-07-26T11:51:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T11:51:12.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:51:12.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:51:12.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:51:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:12 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T11:51:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:12 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T11:51:13.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:12 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T11:51:13.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:12 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T11:51:13.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:12 smithi193 ceph-mon[37473]: pgmap v1597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:51:13.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:12 smithi193 ceph-mon[37473]: pgmap v1598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:51:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:12 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T11:51:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:12 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T11:51:13.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:12 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T11:51:13.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:12 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T11:51:13.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:12 smithi134 ceph-mon[31966]: pgmap v1597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:51:13.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:12 smithi134 ceph-mon[31966]: pgmap v1598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:51:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:12 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T11:51:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:12 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T11:51:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:12 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T11:51:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:12 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T11:51:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:12 smithi189 ceph-mon[36790]: pgmap v1597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:51:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:12 smithi189 ceph-mon[36790]: pgmap v1598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:51:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1415.065399 secs 2024-07-26T11:51:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1415.065399 secs 2024-07-26T11:51:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:51:14.435+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:51:14.435+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:51:14.435+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1415.065399 secs 2024-07-26T11:51:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:14 smithi193 ceph-mon[37473]: pgmap v1599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:51:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:14 smithi134 ceph-mon[31966]: pgmap v1599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:51:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:14 smithi189 ceph-mon[36790]: pgmap v1599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:51:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:16 smithi193 ceph-mon[37473]: pgmap v1600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:51:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1434.491740 secs 2024-07-26T11:51:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:16 smithi134 ceph-mon[31966]: pgmap v1600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:51:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1434.491740 secs 2024-07-26T11:51:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:16 smithi189 ceph-mon[36790]: pgmap v1600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:51:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1434.491740 secs 2024-07-26T11:51:18.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:51:18.924+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:18.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:51:18.924+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:18.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:51:18.924+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:18 smithi193 ceph-mon[37473]: pgmap v1601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:51:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1420.065483 secs 2024-07-26T11:51:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:18 smithi134 ceph-mon[31966]: pgmap v1601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:51:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1420.065483 secs 2024-07-26T11:51:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:18 smithi189 ceph-mon[36790]: pgmap v1601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:51:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1420.065483 secs 2024-07-26T11:51:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:20 smithi193 ceph-mon[37473]: pgmap v1602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:51:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:20 smithi134 ceph-mon[31966]: pgmap v1602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:51:21.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:20 smithi189 ceph-mon[36790]: pgmap v1602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:51:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1439.491886 secs 2024-07-26T11:51:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1439.491886 secs 2024-07-26T11:51:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1439.491886 secs 2024-07-26T11:51:22.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:51:22.580+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T11:51:22.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:51:22.580+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T11:51:22.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:51:22.580+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T11:51:22.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:51:22.580+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T11:51:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:22 smithi193 ceph-mon[37473]: pgmap v1603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:51:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:22 smithi134 ceph-mon[31966]: pgmap v1603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:51:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:22 smithi189 ceph-mon[36790]: pgmap v1603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:51:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1425.065558 secs 2024-07-26T11:51:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1425.065558 secs 2024-07-26T11:51:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1425.065558 secs 2024-07-26T11:51:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:25 smithi193 ceph-mon[37473]: pgmap v1604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:25.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:25 smithi189 ceph-mon[36790]: pgmap v1604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:25 smithi134 ceph-mon[31966]: pgmap v1604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:27 smithi193 ceph-mon[37473]: pgmap v1605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:27 smithi193 ceph-mon[37473]: 1 slow requests, 1 included below; oldest blocked for > 1444.491997 secs 2024-07-26T11:51:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:27 smithi193 ceph-mon[37473]: slow request 1444.491996 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T11:51:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:27 smithi134 ceph-mon[31966]: pgmap v1605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:27 smithi134 ceph-mon[31966]: 1 slow requests, 1 included below; oldest blocked for > 1444.491997 secs 2024-07-26T11:51:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:27 smithi134 ceph-mon[31966]: slow request 1444.491996 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T11:51:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:27 smithi189 ceph-mon[36790]: pgmap v1605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:27 smithi189 ceph-mon[36790]: 1 slow requests, 1 included below; oldest blocked for > 1444.491997 secs 2024-07-26T11:51:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:27 smithi189 ceph-mon[36790]: slow request 1444.491996 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T11:51:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:29 smithi193 ceph-mon[37473]: pgmap v1606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:51:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1430.065656 secs 2024-07-26T11:51:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:29 smithi134 ceph-mon[31966]: pgmap v1606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:51:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1430.065656 secs 2024-07-26T11:51:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:29 smithi189 ceph-mon[36790]: pgmap v1606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:51:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1430.065656 secs 2024-07-26T11:51:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:51:29.447+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:51:29.447+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:29.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:51:29.447+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:31 smithi193 ceph-mon[37473]: pgmap v1607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:31 smithi134 ceph-mon[31966]: pgmap v1607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:31 smithi189 ceph-mon[36790]: pgmap v1607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:32.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1449.492142 secs 2024-07-26T11:51:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1449.492142 secs 2024-07-26T11:51:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1449.492142 secs 2024-07-26T11:51:33.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:51:33.081+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194003690 con 0x55a08e2d94d0 2024-07-26T11:51:33.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:51:33.081+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T11:51:33.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:51:33.081+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T11:51:33.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:51:33.081+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194003690 con 0x55a08e2d94d0 2024-07-26T11:51:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:33 smithi134 ceph-mon[31966]: pgmap v1608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:33 smithi189 ceph-mon[36790]: pgmap v1608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:33 smithi193 ceph-mon[37473]: pgmap v1608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:33.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:51:33.940+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:33.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:51:33.940+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:33.942 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:51:33.940+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1435.065718 secs 2024-07-26T11:51:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1435.065718 secs 2024-07-26T11:51:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1435.065718 secs 2024-07-26T11:51:35.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:35 smithi134 ceph-mon[31966]: pgmap v1609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:35 smithi189 ceph-mon[36790]: pgmap v1609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:35 smithi193 ceph-mon[37473]: pgmap v1609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:37 smithi134 ceph-mon[31966]: pgmap v1610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1454.492297 secs 2024-07-26T11:51:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:37 smithi189 ceph-mon[36790]: pgmap v1610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1454.492297 secs 2024-07-26T11:51:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:37 smithi193 ceph-mon[37473]: pgmap v1610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1454.492297 secs 2024-07-26T11:51:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:39 smithi134 ceph-mon[31966]: pgmap v1611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:51:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:39 smithi134 ceph-mon[31966]: 2 slow requests, 1 included below; oldest blocked for > 1440.065834 secs 2024-07-26T11:51:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:39 smithi134 ceph-mon[31966]: slow request 1440.065834 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T11:51:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:39 smithi189 ceph-mon[36790]: pgmap v1611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:51:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:39 smithi189 ceph-mon[36790]: 2 slow requests, 1 included below; oldest blocked for > 1440.065834 secs 2024-07-26T11:51:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:39 smithi189 ceph-mon[36790]: slow request 1440.065834 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T11:51:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:39 smithi193 ceph-mon[37473]: pgmap v1611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:51:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:39 smithi193 ceph-mon[37473]: 2 slow requests, 1 included below; oldest blocked for > 1440.065834 secs 2024-07-26T11:51:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:39 smithi193 ceph-mon[37473]: slow request 1440.065834 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T11:51:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:41 smithi134 ceph-mon[31966]: pgmap v1612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:41 smithi189 ceph-mon[36790]: pgmap v1612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:41 smithi193 ceph-mon[37473]: pgmap v1612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1459.492400 secs 2024-07-26T11:51:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1459.492400 secs 2024-07-26T11:51:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1459.492400 secs 2024-07-26T11:51:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:43 smithi134 ceph-mon[31966]: pgmap v1613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:43.424 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:43 smithi193 ceph-mon[37473]: pgmap v1613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:43 smithi189 ceph-mon[36790]: pgmap v1613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1445.065970 secs 2024-07-26T11:51:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1445.065970 secs 2024-07-26T11:51:44.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:51:44.462+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:44.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:51:44.462+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:44.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:51:44.462+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1445.065970 secs 2024-07-26T11:51:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:45 smithi134 ceph-mon[31966]: pgmap v1614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:45 smithi189 ceph-mon[36790]: pgmap v1614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:45 smithi193 ceph-mon[37473]: pgmap v1614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:47 smithi134 ceph-mon[31966]: pgmap v1615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1464.492465 secs 2024-07-26T11:51:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:47 smithi189 ceph-mon[36790]: pgmap v1615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1464.492465 secs 2024-07-26T11:51:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:47 smithi193 ceph-mon[37473]: pgmap v1615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1464.492465 secs 2024-07-26T11:51:48.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:51:48.949+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:48.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:51:48.949+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:48.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:51:48.949+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:49 smithi134 ceph-mon[31966]: pgmap v1616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:51:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1450.066056 secs 2024-07-26T11:51:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:49 smithi189 ceph-mon[36790]: pgmap v1616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:51:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1450.066056 secs 2024-07-26T11:51:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:49 smithi193 ceph-mon[37473]: pgmap v1616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:51:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1450.066056 secs 2024-07-26T11:51:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:51 smithi134 ceph-mon[31966]: pgmap v1617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:51 smithi189 ceph-mon[36790]: pgmap v1617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:51 smithi193 ceph-mon[37473]: pgmap v1617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:51:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1469.492543 secs 2024-07-26T11:51:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1469.492543 secs 2024-07-26T11:51:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1469.492543 secs 2024-07-26T11:51:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:53 smithi134 ceph-mon[31966]: pgmap v1618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:53 smithi189 ceph-mon[36790]: pgmap v1618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:53 smithi193 ceph-mon[37473]: pgmap v1618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1455.066169 secs 2024-07-26T11:51:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1455.066169 secs 2024-07-26T11:51:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1455.066169 secs 2024-07-26T11:51:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:55 smithi134 ceph-mon[31966]: pgmap v1619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:55 smithi189 ceph-mon[36790]: pgmap v1619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:55 smithi193 ceph-mon[37473]: pgmap v1619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:51:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1474.492669 secs 2024-07-26T11:51:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1474.492669 secs 2024-07-26T11:51:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1474.492669 secs 2024-07-26T11:51:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:57 smithi134 ceph-mon[31966]: pgmap v1620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:57 smithi189 ceph-mon[36790]: pgmap v1620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:57 smithi193 ceph-mon[37473]: pgmap v1620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:51:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:59 smithi189 ceph-mon[36790]: pgmap v1621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:51:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:51:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1460.066284 secs 2024-07-26T11:51:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:51:59.478+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:51:59.478+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:51:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:51:59.478+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:51:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:59 smithi193 ceph-mon[37473]: pgmap v1621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:51:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:51:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1460.066284 secs 2024-07-26T11:51:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:59 smithi134 ceph-mon[31966]: pgmap v1621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:51:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:51:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1460.066284 secs 2024-07-26T11:52:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:01 smithi189 ceph-mon[36790]: pgmap v1622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T11:52:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1479.492812 secs 2024-07-26T11:52:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:01 smithi193 ceph-mon[37473]: pgmap v1622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T11:52:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1479.492812 secs 2024-07-26T11:52:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:01 smithi134 ceph-mon[31966]: pgmap v1622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T11:52:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1479.492812 secs 2024-07-26T11:52:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:03 smithi134 ceph-mon[31966]: pgmap v1623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:52:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:03 smithi189 ceph-mon[36790]: pgmap v1623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:52:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:03 smithi193 ceph-mon[37473]: pgmap v1623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:52:03.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:52:03.964+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:52:03.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:52:03.964+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:52:03.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:52:03.964+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:04 smithi134 ceph-mon[31966]: 2 slow requests, 1 included below; oldest blocked for > 1465.066375 secs 2024-07-26T11:52:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:04 smithi134 ceph-mon[31966]: slow request 1442.321092 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:52:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:04 smithi193 ceph-mon[37473]: 2 slow requests, 1 included below; oldest blocked for > 1465.066375 secs 2024-07-26T11:52:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:04 smithi193 ceph-mon[37473]: slow request 1442.321092 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:52:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:04 smithi189 ceph-mon[36790]: 2 slow requests, 1 included below; oldest blocked for > 1465.066375 secs 2024-07-26T11:52:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:04 smithi189 ceph-mon[36790]: slow request 1442.321092 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T11:52:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:05 smithi193 ceph-mon[37473]: pgmap v1624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:52:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:05 smithi134 ceph-mon[31966]: pgmap v1624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:52:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:05 smithi189 ceph-mon[36790]: pgmap v1624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:52:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1484.492938 secs 2024-07-26T11:52:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1484.492938 secs 2024-07-26T11:52:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1484.492938 secs 2024-07-26T11:52:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:07 smithi193 ceph-mon[37473]: pgmap v1625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:52:07.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:52:07.588+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T11:52:07.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:52:07.588+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T11:52:07.590 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:07 smithi134 ceph-mon[31966]: pgmap v1625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:52:07.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:52:07.588+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T11:52:07.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:52:07.588+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T11:52:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:07 smithi189 ceph-mon[36790]: pgmap v1625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:52:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:08 smithi193 ceph-mon[37473]: pgmap v1626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:52:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:08 smithi134 ceph-mon[31966]: pgmap v1626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:52:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:08 smithi189 ceph-mon[36790]: pgmap v1626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T11:52:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1470.066456 secs 2024-07-26T11:52:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1470.066456 secs 2024-07-26T11:52:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1470.066456 secs 2024-07-26T11:52:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:10 smithi134 ceph-mon[31966]: pgmap v1627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:10 smithi189 ceph-mon[36790]: pgmap v1627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:10 smithi193 ceph-mon[37473]: pgmap v1627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1489.493066 secs 2024-07-26T11:52:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1489.493066 secs 2024-07-26T11:52:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1489.493066 secs 2024-07-26T11:52:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:12 smithi134 ceph-mon[31966]: pgmap v1628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:52:13.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:12 smithi193 ceph-mon[37473]: pgmap v1628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:13.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:52:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:12 smithi189 ceph-mon[36790]: pgmap v1628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:13.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:52:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:52:14.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:52:14.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:13 smithi134 ceph-mon[31966]: pgmap v1629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T11:52:14.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:13 smithi134 ceph-mon[31966]: pgmap v1630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:52:14.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:52:14.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1475.066612 secs 2024-07-26T11:52:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:52:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:52:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:13 smithi189 ceph-mon[36790]: pgmap v1629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T11:52:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:13 smithi189 ceph-mon[36790]: pgmap v1630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:52:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:52:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1475.066612 secs 2024-07-26T11:52:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:52:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:52:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:13 smithi193 ceph-mon[37473]: pgmap v1629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T11:52:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:13 smithi193 ceph-mon[37473]: pgmap v1630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:52:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:52:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1475.066612 secs 2024-07-26T11:52:14.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:52:14.493+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:52:14.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:52:14.493+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:14.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:52:14.493+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:52:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:15 smithi193 ceph-mon[37473]: pgmap v1631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:52:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:15 smithi134 ceph-mon[31966]: pgmap v1631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:52:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:15 smithi189 ceph-mon[36790]: pgmap v1631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:52:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1494.493208 secs 2024-07-26T11:52:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1494.493208 secs 2024-07-26T11:52:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1494.493208 secs 2024-07-26T11:52:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:52:18.089+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194003690 con 0x55a08e2d94d0 2024-07-26T11:52:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:52:18.089+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T11:52:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:52:18.089+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T11:52:18.090 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:52:18.089+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194003690 con 0x55a08e2d94d0 2024-07-26T11:52:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:17 smithi193 ceph-mon[37473]: pgmap v1632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T11:52:18.313 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:17 smithi134 ceph-mon[31966]: pgmap v1632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T11:52:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:17 smithi189 ceph-mon[36790]: pgmap v1632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T11:52:18.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:52:18.980+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:52:18.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:52:18.980+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:52:18.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:52:18.980+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1480.066762 secs 2024-07-26T11:52:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1480.066762 secs 2024-07-26T11:52:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1480.066762 secs 2024-07-26T11:52:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:20 smithi193 ceph-mon[37473]: pgmap v1633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:52:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:20 smithi134 ceph-mon[31966]: pgmap v1633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:52:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:20 smithi189 ceph-mon[36790]: pgmap v1633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:52:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1499.493354 secs 2024-07-26T11:52:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:22 smithi193 ceph-mon[37473]: pgmap v1634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:52:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1499.493354 secs 2024-07-26T11:52:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:22 smithi134 ceph-mon[31966]: pgmap v1634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:52:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1499.493354 secs 2024-07-26T11:52:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:22 smithi189 ceph-mon[36790]: pgmap v1634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:52:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:24 smithi134 ceph-mon[31966]: pgmap v1635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:52:24.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1485.066884 secs 2024-07-26T11:52:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:24 smithi189 ceph-mon[36790]: pgmap v1635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:52:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1485.066884 secs 2024-07-26T11:52:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:24 smithi193 ceph-mon[37473]: pgmap v1635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:52:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1485.066884 secs 2024-07-26T11:52:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:26 smithi134 ceph-mon[31966]: pgmap v1636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:26 smithi189 ceph-mon[36790]: pgmap v1636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:26 smithi193 ceph-mon[37473]: pgmap v1636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1504.493477 secs 2024-07-26T11:52:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1504.493477 secs 2024-07-26T11:52:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1504.493477 secs 2024-07-26T11:52:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:28 smithi134 ceph-mon[31966]: pgmap v1637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:28 smithi189 ceph-mon[36790]: pgmap v1637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:28 smithi193 ceph-mon[37473]: pgmap v1637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1490.067014 secs 2024-07-26T11:52:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1490.067014 secs 2024-07-26T11:52:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:52:29.508+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:52:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:52:29.508+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:29.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:52:29.508+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:52:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1490.067014 secs 2024-07-26T11:52:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:30 smithi134 ceph-mon[31966]: pgmap v1638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:30 smithi189 ceph-mon[36790]: pgmap v1638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:30 smithi193 ceph-mon[37473]: pgmap v1638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1509.493600 secs 2024-07-26T11:52:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:32 smithi134 ceph-mon[31966]: pgmap v1639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1509.493600 secs 2024-07-26T11:52:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:32 smithi189 ceph-mon[36790]: pgmap v1639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1509.493600 secs 2024-07-26T11:52:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:32 smithi193 ceph-mon[37473]: pgmap v1639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:33.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:52:33.992+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:52:33.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:52:33.992+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:33.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:52:33.992+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:52:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:34 smithi134 ceph-mon[31966]: pgmap v1640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:52:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1495.067126 secs 2024-07-26T11:52:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:34 smithi189 ceph-mon[36790]: pgmap v1640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:52:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1495.067126 secs 2024-07-26T11:52:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:34 smithi193 ceph-mon[37473]: pgmap v1640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:52:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1495.067126 secs 2024-07-26T11:52:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:36 smithi134 ceph-mon[31966]: pgmap v1641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:36 smithi189 ceph-mon[36790]: pgmap v1641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:36 smithi193 ceph-mon[37473]: pgmap v1641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1514.493722 secs 2024-07-26T11:52:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1514.493722 secs 2024-07-26T11:52:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1514.493722 secs 2024-07-26T11:52:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:38 smithi134 ceph-mon[31966]: pgmap v1642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:38 smithi189 ceph-mon[36790]: pgmap v1642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:38 smithi193 ceph-mon[37473]: pgmap v1642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1500.067261 secs 2024-07-26T11:52:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1500.067261 secs 2024-07-26T11:52:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1500.067261 secs 2024-07-26T11:52:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:40 smithi134 ceph-mon[31966]: pgmap v1643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:40 smithi189 ceph-mon[36790]: pgmap v1643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:40 smithi193 ceph-mon[37473]: pgmap v1643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1519.493846 secs 2024-07-26T11:52:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:42 smithi134 ceph-mon[31966]: pgmap v1644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1519.493846 secs 2024-07-26T11:52:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:42 smithi189 ceph-mon[36790]: pgmap v1644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1519.493846 secs 2024-07-26T11:52:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:42 smithi193 ceph-mon[37473]: pgmap v1644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:44 smithi134 ceph-mon[31966]: pgmap v1645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:52:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1505.067389 secs 2024-07-26T11:52:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:44 smithi189 ceph-mon[36790]: pgmap v1645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:52:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1505.067389 secs 2024-07-26T11:52:44.523 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:52:44.521+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:52:44.523 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:52:44.521+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:44.523 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:52:44.521+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:52:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:44 smithi193 ceph-mon[37473]: pgmap v1645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:52:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1505.067389 secs 2024-07-26T11:52:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:46 smithi134 ceph-mon[31966]: pgmap v1646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:46 smithi189 ceph-mon[36790]: pgmap v1646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:46 smithi193 ceph-mon[37473]: pgmap v1646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1524.493979 secs 2024-07-26T11:52:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1524.493979 secs 2024-07-26T11:52:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1524.493979 secs 2024-07-26T11:52:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:48 smithi134 ceph-mon[31966]: pgmap v1647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:48 smithi189 ceph-mon[36790]: pgmap v1647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:48 smithi193 ceph-mon[37473]: pgmap v1647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:49.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:52:49.008+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:52:49.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:52:49.008+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:52:49.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:52:49.008+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1510.067546 secs 2024-07-26T11:52:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1510.067546 secs 2024-07-26T11:52:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1510.067546 secs 2024-07-26T11:52:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:50 smithi134 ceph-mon[31966]: pgmap v1648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:50 smithi189 ceph-mon[36790]: pgmap v1648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:50 smithi193 ceph-mon[37473]: pgmap v1648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:52:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1529.494115 secs 2024-07-26T11:52:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1529.494115 secs 2024-07-26T11:52:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1529.494115 secs 2024-07-26T11:52:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:52 smithi134 ceph-mon[31966]: pgmap v1649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:52 smithi189 ceph-mon[36790]: pgmap v1649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:52 smithi193 ceph-mon[37473]: pgmap v1649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:52:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:54 smithi134 ceph-mon[31966]: pgmap v1650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:52:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1515.067700 secs 2024-07-26T11:52:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:54 smithi189 ceph-mon[36790]: pgmap v1650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:52:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1515.067700 secs 2024-07-26T11:52:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:54 smithi193 ceph-mon[37473]: pgmap v1650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:52:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1515.067700 secs 2024-07-26T11:52:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:56 smithi189 ceph-mon[36790]: pgmap v1651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1534.494245 secs 2024-07-26T11:52:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:56 smithi193 ceph-mon[37473]: pgmap v1651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1534.494245 secs 2024-07-26T11:52:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:56 smithi134 ceph-mon[31966]: pgmap v1651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:52:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1534.494245 secs 2024-07-26T11:52:58.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:58 smithi189 ceph-mon[36790]: pgmap v1652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:58 smithi193 ceph-mon[37473]: pgmap v1652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:58 smithi134 ceph-mon[31966]: pgmap v1652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:52:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:52:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1520.067823 secs 2024-07-26T11:52:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:52:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1520.067823 secs 2024-07-26T11:52:59.536 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:52:59.534+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:52:59.536 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:52:59.534+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:52:59.536 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:52:59.534+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:52:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:52:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1520.067823 secs 2024-07-26T11:53:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:00 smithi193 ceph-mon[37473]: pgmap v1653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:53:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:00 smithi134 ceph-mon[31966]: pgmap v1653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:53:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:00 smithi189 ceph-mon[36790]: pgmap v1653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:53:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1539.494371 secs 2024-07-26T11:53:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1539.494371 secs 2024-07-26T11:53:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1539.494371 secs 2024-07-26T11:53:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:02 smithi193 ceph-mon[37473]: pgmap v1654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:02 smithi134 ceph-mon[31966]: pgmap v1654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:02 smithi189 ceph-mon[36790]: pgmap v1654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:03 smithi193 ceph-mon[37473]: pgmap v1655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:03 smithi134 ceph-mon[31966]: pgmap v1655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:03 smithi189 ceph-mon[36790]: pgmap v1655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:04.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:53:04.023+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:53:04.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:53:04.023+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:04.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:53:04.023+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:53:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1525.067949 secs 2024-07-26T11:53:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1525.067949 secs 2024-07-26T11:53:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1525.067949 secs 2024-07-26T11:53:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:05 smithi134 ceph-mon[31966]: pgmap v1656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:53:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:05 smithi189 ceph-mon[36790]: pgmap v1656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:53:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:05 smithi193 ceph-mon[37473]: pgmap v1656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:53:06.577 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1544.494484 secs 2024-07-26T11:53:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1544.494484 secs 2024-07-26T11:53:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1544.494484 secs 2024-07-26T11:53:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:07 smithi134 ceph-mon[31966]: pgmap v1657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:53:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:07 smithi189 ceph-mon[36790]: pgmap v1657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:53:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:07 smithi193 ceph-mon[37473]: pgmap v1657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:53:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1530.068034 secs 2024-07-26T11:53:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1530.068034 secs 2024-07-26T11:53:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1530.068034 secs 2024-07-26T11:53:09.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:09 smithi134 ceph-mon[31966]: pgmap v1658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:53:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:09 smithi189 ceph-mon[36790]: pgmap v1658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:53:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:09 smithi193 ceph-mon[37473]: pgmap v1658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:53:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1549.494555 secs 2024-07-26T11:53:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:12 smithi193 ceph-mon[37473]: pgmap v1659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1549.494555 secs 2024-07-26T11:53:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:12 smithi134 ceph-mon[31966]: pgmap v1659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1549.494555 secs 2024-07-26T11:53:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:12 smithi189 ceph-mon[36790]: pgmap v1659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:14 smithi134 ceph-mon[31966]: pgmap v1660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:53:14.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1535.068104 secs 2024-07-26T11:53:14.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:53:14.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:53:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:14 smithi189 ceph-mon[36790]: pgmap v1660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:53:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1535.068104 secs 2024-07-26T11:53:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:53:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:53:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:14 smithi193 ceph-mon[37473]: pgmap v1660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:53:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1535.068104 secs 2024-07-26T11:53:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:53:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:53:14.550 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:53:14.549+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:53:14.551 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:53:14.549+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:53:14.551 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:53:14.549+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:53:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:53:15.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:53:15.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:53:15.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:53:15.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:53:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:53:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:53:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:53:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:16 smithi134 ceph-mon[31966]: pgmap v1661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:53:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:16 smithi134 ceph-mon[31966]: pgmap v1662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:53:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:16 smithi189 ceph-mon[36790]: pgmap v1661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:53:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:16 smithi189 ceph-mon[36790]: pgmap v1662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:53:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:16 smithi193 ceph-mon[37473]: pgmap v1661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T11:53:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:16 smithi193 ceph-mon[37473]: pgmap v1662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:53:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1554.494688 secs 2024-07-26T11:53:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1554.494688 secs 2024-07-26T11:53:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1554.494688 secs 2024-07-26T11:53:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:18 smithi134 ceph-mon[31966]: pgmap v1663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:53:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:18 smithi189 ceph-mon[36790]: pgmap v1663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:53:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:18 smithi193 ceph-mon[37473]: pgmap v1663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:53:19.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:53:19.039+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:53:19.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:53:19.039+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:53:19.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:53:19.039+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1540.068244 secs 2024-07-26T11:53:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1540.068244 secs 2024-07-26T11:53:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1540.068244 secs 2024-07-26T11:53:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:20 smithi134 ceph-mon[31966]: pgmap v1664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:53:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:20 smithi189 ceph-mon[36790]: pgmap v1664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:53:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:20 smithi193 ceph-mon[37473]: pgmap v1664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:53:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:22 smithi134 ceph-mon[31966]: pgmap v1665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:53:22.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1559.494820 secs 2024-07-26T11:53:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:22 smithi189 ceph-mon[36790]: pgmap v1665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:53:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1559.494820 secs 2024-07-26T11:53:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:22 smithi193 ceph-mon[37473]: pgmap v1665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:53:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1559.494820 secs 2024-07-26T11:53:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:24 smithi134 ceph-mon[31966]: pgmap v1666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:53:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1545.068316 secs 2024-07-26T11:53:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:24 smithi189 ceph-mon[36790]: pgmap v1666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:53:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1545.068316 secs 2024-07-26T11:53:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:24 smithi193 ceph-mon[37473]: pgmap v1666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T11:53:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1545.068316 secs 2024-07-26T11:53:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:26 smithi134 ceph-mon[31966]: pgmap v1667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:53:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:26 smithi189 ceph-mon[36790]: pgmap v1667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:53:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:26 smithi193 ceph-mon[37473]: pgmap v1667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:53:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1564.494896 secs 2024-07-26T11:53:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1564.494896 secs 2024-07-26T11:53:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1564.494896 secs 2024-07-26T11:53:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:28 smithi134 ceph-mon[31966]: pgmap v1668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:28 smithi189 ceph-mon[36790]: pgmap v1668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:28.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:28 smithi193 ceph-mon[37473]: pgmap v1668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1550.068431 secs 2024-07-26T11:53:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1550.068431 secs 2024-07-26T11:53:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1550.068431 secs 2024-07-26T11:53:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:53:29.561+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:53:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:53:29.561+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:29.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:53:29.561+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:53:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:30 smithi134 ceph-mon[31966]: pgmap v1669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:30 smithi189 ceph-mon[36790]: pgmap v1669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:30 smithi193 ceph-mon[37473]: pgmap v1669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:32 smithi134 ceph-mon[31966]: pgmap v1670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:32.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1569.495022 secs 2024-07-26T11:53:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:32 smithi189 ceph-mon[36790]: pgmap v1670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1569.495022 secs 2024-07-26T11:53:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:32 smithi193 ceph-mon[37473]: pgmap v1670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1569.495022 secs 2024-07-26T11:53:34.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:53:34.045+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:53:34.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:53:34.045+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:34.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:53:34.045+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:53:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:34 smithi134 ceph-mon[31966]: pgmap v1671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1555.068554 secs 2024-07-26T11:53:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:34 smithi189 ceph-mon[36790]: pgmap v1671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1555.068554 secs 2024-07-26T11:53:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:34 smithi193 ceph-mon[37473]: pgmap v1671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1555.068554 secs 2024-07-26T11:53:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:36 smithi134 ceph-mon[31966]: pgmap v1672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:53:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:36 smithi189 ceph-mon[36790]: pgmap v1672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:53:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:36 smithi193 ceph-mon[37473]: pgmap v1672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:53:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1574.495153 secs 2024-07-26T11:53:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1574.495153 secs 2024-07-26T11:53:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1574.495153 secs 2024-07-26T11:53:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:38 smithi134 ceph-mon[31966]: pgmap v1673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:38 smithi189 ceph-mon[36790]: pgmap v1673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:38 smithi193 ceph-mon[37473]: pgmap v1673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1560.068657 secs 2024-07-26T11:53:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1560.068657 secs 2024-07-26T11:53:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1560.068657 secs 2024-07-26T11:53:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:40 smithi134 ceph-mon[31966]: pgmap v1674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:40 smithi189 ceph-mon[36790]: pgmap v1674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:40 smithi193 ceph-mon[37473]: pgmap v1674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1579.495292 secs 2024-07-26T11:53:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1579.495292 secs 2024-07-26T11:53:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1579.495292 secs 2024-07-26T11:53:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:42 smithi189 ceph-mon[36790]: pgmap v1675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:42 smithi193 ceph-mon[37473]: pgmap v1675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:42 smithi134 ceph-mon[31966]: pgmap v1675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:44 smithi189 ceph-mon[36790]: pgmap v1676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1565.068776 secs 2024-07-26T11:53:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:44 smithi193 ceph-mon[37473]: pgmap v1676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1565.068776 secs 2024-07-26T11:53:44.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:53:44.576+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:53:44.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:53:44.577+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:53:44.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:53:44.577+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:44 smithi134 ceph-mon[31966]: pgmap v1676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:53:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1565.068776 secs 2024-07-26T11:53:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:46 smithi189 ceph-mon[36790]: pgmap v1677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:53:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1584.495440 secs 2024-07-26T11:53:46.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:46 smithi193 ceph-mon[37473]: pgmap v1677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:53:46.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1584.495440 secs 2024-07-26T11:53:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:46 smithi134 ceph-mon[31966]: pgmap v1677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:53:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1584.495440 secs 2024-07-26T11:53:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:48 smithi193 ceph-mon[37473]: pgmap v1678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:48 smithi134 ceph-mon[31966]: pgmap v1678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:48 smithi189 ceph-mon[36790]: pgmap v1678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:53:49.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:53:49.051+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:53:49.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:53:49.051+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:49.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:53:49.051+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:53:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1570.068893 secs 2024-07-26T11:53:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1570.068893 secs 2024-07-26T11:53:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1570.068893 secs 2024-07-26T11:53:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:50 smithi193 ceph-mon[37473]: pgmap v1679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:53:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:50 smithi134 ceph-mon[31966]: pgmap v1679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:53:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:50 smithi189 ceph-mon[36790]: pgmap v1679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:53:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1589.495592 secs 2024-07-26T11:53:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1589.495592 secs 2024-07-26T11:53:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1589.495592 secs 2024-07-26T11:53:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:52 smithi134 ceph-mon[31966]: pgmap v1680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:53:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:52 smithi193 ceph-mon[37473]: pgmap v1680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:53:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:52 smithi189 ceph-mon[36790]: pgmap v1680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:53:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:53 smithi193 ceph-mon[37473]: pgmap v1681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:53:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:53 smithi134 ceph-mon[31966]: pgmap v1681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:53:53.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:53 smithi189 ceph-mon[36790]: pgmap v1681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:53:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1575.069014 secs 2024-07-26T11:53:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1575.069014 secs 2024-07-26T11:53:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1575.069014 secs 2024-07-26T11:53:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:55 smithi193 ceph-mon[37473]: pgmap v1682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:53:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:55 smithi134 ceph-mon[31966]: pgmap v1682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:53:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:55 smithi189 ceph-mon[36790]: pgmap v1682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:53:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1594.495715 secs 2024-07-26T11:53:56.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1594.495715 secs 2024-07-26T11:53:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1594.495715 secs 2024-07-26T11:53:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:57 smithi193 ceph-mon[37473]: pgmap v1683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:53:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:57 smithi134 ceph-mon[31966]: pgmap v1683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:53:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:57 smithi189 ceph-mon[36790]: pgmap v1683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:53:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1580.069140 secs 2024-07-26T11:53:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1580.069140 secs 2024-07-26T11:53:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1580.069140 secs 2024-07-26T11:53:59.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:53:59.592+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:53:59.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:53:59.592+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:53:59.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:53:59.592+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:53:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:53:59 smithi134 ceph-mon[31966]: pgmap v1684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:53:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:53:59 smithi189 ceph-mon[36790]: pgmap v1684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:54:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:53:59 smithi193 ceph-mon[37473]: pgmap v1684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:54:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:01 smithi134 ceph-mon[31966]: pgmap v1685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1599.495842 secs 2024-07-26T11:54:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:01 smithi189 ceph-mon[36790]: pgmap v1685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1599.495842 secs 2024-07-26T11:54:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:01 smithi193 ceph-mon[37473]: pgmap v1685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1599.495842 secs 2024-07-26T11:54:04.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:54:04.066+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:54:04.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:54:04.066+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:04.068 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:54:04.066+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:54:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:03 smithi134 ceph-mon[31966]: pgmap v1686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:54:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1585.069262 secs 2024-07-26T11:54:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:03 smithi189 ceph-mon[36790]: pgmap v1686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:54:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1585.069262 secs 2024-07-26T11:54:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:03 smithi193 ceph-mon[37473]: pgmap v1686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:54:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1585.069262 secs 2024-07-26T11:54:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:05 smithi134 ceph-mon[31966]: pgmap v1687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:54:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:05 smithi189 ceph-mon[36790]: pgmap v1687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:54:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:05 smithi193 ceph-mon[37473]: pgmap v1687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:54:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1604.495948 secs 2024-07-26T11:54:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1604.495948 secs 2024-07-26T11:54:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1604.495948 secs 2024-07-26T11:54:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:07 smithi134 ceph-mon[31966]: pgmap v1688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:07 smithi189 ceph-mon[36790]: pgmap v1688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:07 smithi193 ceph-mon[37473]: pgmap v1688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1590.069374 secs 2024-07-26T11:54:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1590.069374 secs 2024-07-26T11:54:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1590.069374 secs 2024-07-26T11:54:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:09 smithi134 ceph-mon[31966]: pgmap v1689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:09 smithi189 ceph-mon[36790]: pgmap v1689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:09 smithi193 ceph-mon[37473]: pgmap v1689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:11 smithi134 ceph-mon[31966]: pgmap v1690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1609.496067 secs 2024-07-26T11:54:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:11 smithi189 ceph-mon[36790]: pgmap v1690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1609.496067 secs 2024-07-26T11:54:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:11 smithi193 ceph-mon[37473]: pgmap v1690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1609.496067 secs 2024-07-26T11:54:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:13 smithi134 ceph-mon[31966]: pgmap v1691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:54:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1595.069448 secs 2024-07-26T11:54:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:13 smithi189 ceph-mon[36790]: pgmap v1691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:54:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1595.069448 secs 2024-07-26T11:54:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:13 smithi193 ceph-mon[37473]: pgmap v1691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:54:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1595.069448 secs 2024-07-26T11:54:14.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:54:14.608+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:54:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:54:14.608+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:54:14.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:54:14.608+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:15 smithi134 ceph-mon[31966]: pgmap v1692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:54:16.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:54:16.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:16.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:16.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:16.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:15 smithi189 ceph-mon[36790]: pgmap v1692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:54:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:54:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:16.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:16.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:15 smithi193 ceph-mon[37473]: pgmap v1692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:54:16.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:54:16.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:16.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:16.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:16.257 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1614.496203 secs 2024-07-26T11:54:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1614.496203 secs 2024-07-26T11:54:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1614.496203 secs 2024-07-26T11:54:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:18 smithi193 ceph-mon[37473]: pgmap v1693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:18 smithi134 ceph-mon[31966]: pgmap v1693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:18 smithi189 ceph-mon[36790]: pgmap v1693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:19.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:54:19.081+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:54:19.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:54:19.081+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:19.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:54:19.081+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:54:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1600.069584 secs 2024-07-26T11:54:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1600.069584 secs 2024-07-26T11:54:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1600.069584 secs 2024-07-26T11:54:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:20 smithi193 ceph-mon[37473]: pgmap v1694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:54:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:54:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:20 smithi134 ceph-mon[31966]: pgmap v1694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:54:20.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:54:20.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:20 smithi189 ceph-mon[36790]: pgmap v1694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:20.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:54:20.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:54:20.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:54:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:21 smithi193 ceph-mon[37473]: pgmap v1695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:54:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:21 smithi193 ceph-mon[37473]: pgmap v1696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:54:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:21 smithi134 ceph-mon[31966]: pgmap v1695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:54:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:21 smithi134 ceph-mon[31966]: pgmap v1696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:54:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:21 smithi189 ceph-mon[36790]: pgmap v1695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:54:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:21 smithi189 ceph-mon[36790]: pgmap v1696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:54:22.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1619.496344 secs 2024-07-26T11:54:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1619.496344 secs 2024-07-26T11:54:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1619.496344 secs 2024-07-26T11:54:22.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:54:22.611+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T11:54:22.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:54:22.611+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T11:54:22.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:54:22.611+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T11:54:22.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:54:22.611+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T11:54:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:23 smithi134 ceph-mon[31966]: pgmap v1697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T11:54:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:23 smithi189 ceph-mon[36790]: pgmap v1697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T11:54:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:23 smithi193 ceph-mon[37473]: pgmap v1697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T11:54:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1605.069698 secs 2024-07-26T11:54:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1605.069698 secs 2024-07-26T11:54:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1605.069698 secs 2024-07-26T11:54:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:25 smithi134 ceph-mon[31966]: pgmap v1698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:54:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:25 smithi189 ceph-mon[36790]: pgmap v1698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:54:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:25 smithi193 ceph-mon[37473]: pgmap v1698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:54:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1624.496471 secs 2024-07-26T11:54:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:27 smithi134 ceph-mon[31966]: pgmap v1699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:54:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1624.496471 secs 2024-07-26T11:54:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:27 smithi189 ceph-mon[36790]: pgmap v1699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:54:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1624.496471 secs 2024-07-26T11:54:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:27 smithi193 ceph-mon[37473]: pgmap v1699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:54:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:29 smithi134 ceph-mon[31966]: pgmap v1700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:54:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1610.069820 secs 2024-07-26T11:54:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:29 smithi189 ceph-mon[36790]: pgmap v1700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:54:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1610.069820 secs 2024-07-26T11:54:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:29 smithi193 ceph-mon[37473]: pgmap v1700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:54:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1610.069820 secs 2024-07-26T11:54:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:54:29.617+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:54:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:54:29.617+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:54:29.618 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:54:29.617+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:31 smithi134 ceph-mon[31966]: pgmap v1701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:54:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:31 smithi189 ceph-mon[36790]: pgmap v1701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:54:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:31 smithi193 ceph-mon[37473]: pgmap v1701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:54:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1629.496600 secs 2024-07-26T11:54:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1629.496600 secs 2024-07-26T11:54:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1629.496600 secs 2024-07-26T11:54:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:54:33.111+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T11:54:33.113 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:54:33.111+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194003690 con 0x55a08e2d94d0 2024-07-26T11:54:33.114 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:54:33.112+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T11:54:33.114 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:54:33.112+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194003690 con 0x55a08e2d94d0 2024-07-26T11:54:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:33 smithi134 ceph-mon[31966]: pgmap v1702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:33 smithi189 ceph-mon[36790]: pgmap v1702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:33 smithi193 ceph-mon[37473]: pgmap v1702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:34.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:54:34.096+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:54:34.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:54:34.096+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:34.098 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:54:34.096+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:54:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1615.069955 secs 2024-07-26T11:54:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1615.069955 secs 2024-07-26T11:54:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1615.069955 secs 2024-07-26T11:54:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:35 smithi134 ceph-mon[31966]: pgmap v1703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:35 smithi189 ceph-mon[36790]: pgmap v1703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:35 smithi193 ceph-mon[37473]: pgmap v1703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1634.496702 secs 2024-07-26T11:54:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:37 smithi134 ceph-mon[31966]: pgmap v1704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1634.496702 secs 2024-07-26T11:54:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:37 smithi189 ceph-mon[36790]: pgmap v1704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1634.496702 secs 2024-07-26T11:54:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:37 smithi193 ceph-mon[37473]: pgmap v1704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:39 smithi134 ceph-mon[31966]: pgmap v1705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:54:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1620.070061 secs 2024-07-26T11:54:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:39 smithi189 ceph-mon[36790]: pgmap v1705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:54:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1620.070061 secs 2024-07-26T11:54:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:39 smithi193 ceph-mon[37473]: pgmap v1705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:54:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1620.070061 secs 2024-07-26T11:54:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:41 smithi134 ceph-mon[31966]: pgmap v1706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:54:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:41 smithi189 ceph-mon[36790]: pgmap v1706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:54:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:41 smithi193 ceph-mon[37473]: pgmap v1706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:54:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1639.496835 secs 2024-07-26T11:54:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1639.496835 secs 2024-07-26T11:54:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1639.496835 secs 2024-07-26T11:54:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:43 smithi134 ceph-mon[31966]: pgmap v1707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:43 smithi189 ceph-mon[36790]: pgmap v1707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:43 smithi193 ceph-mon[37473]: pgmap v1707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1625.070159 secs 2024-07-26T11:54:44.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1625.070159 secs 2024-07-26T11:54:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1625.070159 secs 2024-07-26T11:54:44.631 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:54:44.629+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:54:44.631 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:54:44.629+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:44.631 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:54:44.629+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:54:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:45 smithi134 ceph-mon[31966]: pgmap v1708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:45 smithi189 ceph-mon[36790]: pgmap v1708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:45 smithi193 ceph-mon[37473]: pgmap v1708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1644.496955 secs 2024-07-26T11:54:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1644.496955 secs 2024-07-26T11:54:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1644.496955 secs 2024-07-26T11:54:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:47 smithi193 ceph-mon[37473]: pgmap v1709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:47 smithi134 ceph-mon[31966]: pgmap v1709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:47 smithi189 ceph-mon[36790]: pgmap v1709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:49.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:54:49.111+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:54:49.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:54:49.111+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:54:49.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:54:49.111+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:49 smithi193 ceph-mon[37473]: pgmap v1710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:54:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1630.070281 secs 2024-07-26T11:54:49.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:49 smithi134 ceph-mon[31966]: pgmap v1710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:54:49.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1630.070281 secs 2024-07-26T11:54:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:49 smithi189 ceph-mon[36790]: pgmap v1710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:54:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1630.070281 secs 2024-07-26T11:54:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:51 smithi193 ceph-mon[37473]: pgmap v1711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:54:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1649.497023 secs 2024-07-26T11:54:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:51 smithi134 ceph-mon[31966]: pgmap v1711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:54:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1649.497023 secs 2024-07-26T11:54:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:51 smithi189 ceph-mon[36790]: pgmap v1711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:54:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1649.497023 secs 2024-07-26T11:54:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:53 smithi193 ceph-mon[37473]: pgmap v1712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:53 smithi134 ceph-mon[31966]: pgmap v1712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:53 smithi189 ceph-mon[36790]: pgmap v1712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:54:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1635.070392 secs 2024-07-26T11:54:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1635.070392 secs 2024-07-26T11:54:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1635.070392 secs 2024-07-26T11:54:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:55 smithi134 ceph-mon[31966]: pgmap v1713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:55 smithi193 ceph-mon[37473]: pgmap v1713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:55 smithi189 ceph-mon[36790]: pgmap v1713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:54:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1654.497148 secs 2024-07-26T11:54:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1654.497148 secs 2024-07-26T11:54:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1654.497148 secs 2024-07-26T11:54:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:57 smithi193 ceph-mon[37473]: pgmap v1714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:57.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:57 smithi189 ceph-mon[36790]: pgmap v1714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:57 smithi134 ceph-mon[31966]: pgmap v1714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:54:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:58 smithi193 ceph-mon[37473]: pgmap v1715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:54:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:58 smithi134 ceph-mon[31966]: pgmap v1715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:54:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:58 smithi189 ceph-mon[36790]: pgmap v1715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:54:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:54:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1640.070506 secs 2024-07-26T11:54:59.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:54:59.645+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:54:59.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:54:59.645+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:54:59.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:54:59.645+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:54:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:54:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1640.070506 secs 2024-07-26T11:54:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:54:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1640.070506 secs 2024-07-26T11:55:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:00 smithi193 ceph-mon[37473]: pgmap v1716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:55:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:00 smithi134 ceph-mon[31966]: pgmap v1716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:55:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:00 smithi189 ceph-mon[36790]: pgmap v1716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:55:01.651 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1659.497251 secs 2024-07-26T11:55:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1659.497251 secs 2024-07-26T11:55:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1659.497251 secs 2024-07-26T11:55:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:02 smithi134 ceph-mon[31966]: pgmap v1717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:55:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:02 smithi189 ceph-mon[36790]: pgmap v1717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:55:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:02 smithi193 ceph-mon[37473]: pgmap v1717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:55:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1645.070617 secs 2024-07-26T11:55:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1645.070617 secs 2024-07-26T11:55:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1645.070617 secs 2024-07-26T11:55:04.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:55:04.127+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:55:04.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:55:04.127+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:55:04.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:55:04.127+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:04 smithi134 ceph-mon[31966]: pgmap v1718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:55:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:04 smithi189 ceph-mon[36790]: pgmap v1718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:55:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:04 smithi193 ceph-mon[37473]: pgmap v1718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:55:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1664.497373 secs 2024-07-26T11:55:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:07 smithi134 ceph-mon[31966]: pgmap v1719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1664.497373 secs 2024-07-26T11:55:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:07 smithi189 ceph-mon[36790]: pgmap v1719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1664.497373 secs 2024-07-26T11:55:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:07 smithi193 ceph-mon[37473]: pgmap v1719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:09 smithi134 ceph-mon[31966]: pgmap v1720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1650.070744 secs 2024-07-26T11:55:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:09 smithi189 ceph-mon[36790]: pgmap v1720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1650.070744 secs 2024-07-26T11:55:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:09 smithi193 ceph-mon[37473]: pgmap v1720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1650.070744 secs 2024-07-26T11:55:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:10 smithi189 ceph-mon[36790]: pgmap v1721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:55:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:10 smithi193 ceph-mon[37473]: pgmap v1721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:55:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:10 smithi134 ceph-mon[31966]: pgmap v1721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:55:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1669.497496 secs 2024-07-26T11:55:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1669.497496 secs 2024-07-26T11:55:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1669.497496 secs 2024-07-26T11:55:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:12 smithi189 ceph-mon[36790]: pgmap v1722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:55:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:12 smithi193 ceph-mon[37473]: pgmap v1722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:55:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:12 smithi134 ceph-mon[31966]: pgmap v1722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:55:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1655.070820 secs 2024-07-26T11:55:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1655.070820 secs 2024-07-26T11:55:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1655.070820 secs 2024-07-26T11:55:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:55:14.660+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:55:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:55:14.660+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:14.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:55:14.660+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:55:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:14 smithi193 ceph-mon[37473]: pgmap v1723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:55:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:14 smithi134 ceph-mon[31966]: pgmap v1723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:55:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:14 smithi189 ceph-mon[36790]: pgmap v1723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:55:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1674.497623 secs 2024-07-26T11:55:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:17 smithi134 ceph-mon[31966]: pgmap v1724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1674.497623 secs 2024-07-26T11:55:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:17 smithi189 ceph-mon[36790]: pgmap v1724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1674.497623 secs 2024-07-26T11:55:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:17 smithi193 ceph-mon[37473]: pgmap v1724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:19.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:55:19.142+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:55:19.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:55:19.142+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:55:19.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:55:19.142+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:19 smithi134 ceph-mon[31966]: pgmap v1725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:55:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1660.070918 secs 2024-07-26T11:55:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:19 smithi189 ceph-mon[36790]: pgmap v1725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:55:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1660.070918 secs 2024-07-26T11:55:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:19 smithi193 ceph-mon[37473]: pgmap v1725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:55:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1660.070918 secs 2024-07-26T11:55:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:21 smithi134 ceph-mon[31966]: pgmap v1726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:55:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:55:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:21 smithi189 ceph-mon[36790]: pgmap v1726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:55:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:55:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:21 smithi193 ceph-mon[37473]: pgmap v1726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:55:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:55:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1679.497740 secs 2024-07-26T11:55:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:55:22.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:55:22.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:22 smithi134 ceph-mon[31966]: pgmap v1727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:55:22.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:55:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1679.497740 secs 2024-07-26T11:55:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:55:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:55:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:22 smithi189 ceph-mon[36790]: pgmap v1727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:55:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:55:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1679.497740 secs 2024-07-26T11:55:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:55:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:55:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:22 smithi193 ceph-mon[37473]: pgmap v1727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:55:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:55:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:23 smithi134 ceph-mon[31966]: pgmap v1728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T11:55:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:23 smithi189 ceph-mon[36790]: pgmap v1728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T11:55:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:23 smithi193 ceph-mon[37473]: pgmap v1728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T11:55:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1665.070958 secs 2024-07-26T11:55:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1665.070958 secs 2024-07-26T11:55:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1665.070958 secs 2024-07-26T11:55:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:25 smithi134 ceph-mon[31966]: pgmap v1729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:55:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:25 smithi189 ceph-mon[36790]: pgmap v1729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:55:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:25 smithi193 ceph-mon[37473]: pgmap v1729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T11:55:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1684.497877 secs 2024-07-26T11:55:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1684.497877 secs 2024-07-26T11:55:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1684.497877 secs 2024-07-26T11:55:27.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:27 smithi189 ceph-mon[36790]: pgmap v1730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:55:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:27 smithi134 ceph-mon[31966]: pgmap v1730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:55:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:27 smithi193 ceph-mon[37473]: pgmap v1730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T11:55:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1670.071071 secs 2024-07-26T11:55:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1670.071071 secs 2024-07-26T11:55:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1670.071071 secs 2024-07-26T11:55:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:55:29.675+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:55:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:55:29.675+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:29.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:55:29.675+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:55:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:29 smithi134 ceph-mon[31966]: pgmap v1731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:55:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:29 smithi189 ceph-mon[36790]: pgmap v1731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:55:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:29 smithi193 ceph-mon[37473]: pgmap v1731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:55:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1689.498006 secs 2024-07-26T11:55:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1689.498006 secs 2024-07-26T11:55:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1689.498006 secs 2024-07-26T11:55:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:32 smithi193 ceph-mon[37473]: pgmap v1732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:55:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:32 smithi134 ceph-mon[31966]: pgmap v1732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:55:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:32 smithi189 ceph-mon[36790]: pgmap v1732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:55:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:33 smithi134 ceph-mon[31966]: pgmap v1733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:33 smithi189 ceph-mon[36790]: pgmap v1733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:33 smithi193 ceph-mon[37473]: pgmap v1733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:34.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:55:34.157+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:55:34.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:55:34.158+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:55:34.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:55:34.158+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1675.071134 secs 2024-07-26T11:55:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1675.071134 secs 2024-07-26T11:55:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1675.071134 secs 2024-07-26T11:55:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:35 smithi134 ceph-mon[31966]: pgmap v1734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:35.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:35 smithi189 ceph-mon[36790]: pgmap v1734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:35 smithi193 ceph-mon[37473]: pgmap v1734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1694.498135 secs 2024-07-26T11:55:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1694.498135 secs 2024-07-26T11:55:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1694.498135 secs 2024-07-26T11:55:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:37 smithi134 ceph-mon[31966]: pgmap v1735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:55:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:37 smithi189 ceph-mon[36790]: pgmap v1735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:55:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:37 smithi193 ceph-mon[37473]: pgmap v1735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:55:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1680.071215 secs 2024-07-26T11:55:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1680.071215 secs 2024-07-26T11:55:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1680.071215 secs 2024-07-26T11:55:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:39 smithi134 ceph-mon[31966]: pgmap v1736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:55:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:39 smithi189 ceph-mon[36790]: pgmap v1736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:55:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:39 smithi193 ceph-mon[37473]: pgmap v1736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:55:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1699.498259 secs 2024-07-26T11:55:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1699.498259 secs 2024-07-26T11:55:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1699.498259 secs 2024-07-26T11:55:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:42 smithi134 ceph-mon[31966]: pgmap v1737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:42 smithi193 ceph-mon[37473]: pgmap v1737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:42 smithi189 ceph-mon[36790]: pgmap v1737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:43 smithi134 ceph-mon[31966]: pgmap v1738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:43 smithi189 ceph-mon[36790]: pgmap v1738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:43 smithi193 ceph-mon[37473]: pgmap v1738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:55:44.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1685.071353 secs 2024-07-26T11:55:44.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1685.071353 secs 2024-07-26T11:55:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:55:44.691+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:55:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:55:44.691+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:44.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:55:44.691+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:55:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1685.071353 secs 2024-07-26T11:55:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:45 smithi134 ceph-mon[31966]: pgmap v1739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:45 smithi189 ceph-mon[36790]: pgmap v1739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:45 smithi193 ceph-mon[37473]: pgmap v1739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1704.498387 secs 2024-07-26T11:55:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1704.498387 secs 2024-07-26T11:55:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1704.498387 secs 2024-07-26T11:55:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:47 smithi134 ceph-mon[31966]: pgmap v1740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:55:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:47 smithi189 ceph-mon[36790]: pgmap v1740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:55:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:47 smithi193 ceph-mon[37473]: pgmap v1740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:55:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1690.071416 secs 2024-07-26T11:55:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1690.071416 secs 2024-07-26T11:55:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1690.071416 secs 2024-07-26T11:55:49.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:55:49.160+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:55:49.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:55:49.160+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:55:49.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:55:49.160+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:49 smithi134 ceph-mon[31966]: pgmap v1741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:55:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:49 smithi189 ceph-mon[36790]: pgmap v1741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:55:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:49 smithi193 ceph-mon[37473]: pgmap v1741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:55:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1709.498513 secs 2024-07-26T11:55:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1709.498513 secs 2024-07-26T11:55:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1709.498513 secs 2024-07-26T11:55:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:52 smithi193 ceph-mon[37473]: pgmap v1742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:52 smithi134 ceph-mon[31966]: pgmap v1742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:52 smithi189 ceph-mon[36790]: pgmap v1742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:55:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:54 smithi134 ceph-mon[31966]: pgmap v1743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:55:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1695.071541 secs 2024-07-26T11:55:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:54 smithi189 ceph-mon[36790]: pgmap v1743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:55:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1695.071541 secs 2024-07-26T11:55:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:54 smithi193 ceph-mon[37473]: pgmap v1743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:55:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1695.071541 secs 2024-07-26T11:55:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:56 smithi134 ceph-mon[31966]: pgmap v1744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:55:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1714.498646 secs 2024-07-26T11:55:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:56 smithi189 ceph-mon[36790]: pgmap v1744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:55:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1714.498646 secs 2024-07-26T11:55:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:56 smithi193 ceph-mon[37473]: pgmap v1744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:55:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1714.498646 secs 2024-07-26T11:55:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:58 smithi134 ceph-mon[31966]: pgmap v1745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:55:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:58 smithi189 ceph-mon[36790]: pgmap v1745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:55:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:58 smithi193 ceph-mon[37473]: pgmap v1745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T11:55:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:55:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1700.071664 secs 2024-07-26T11:55:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:55:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1700.071664 secs 2024-07-26T11:55:59.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:55:59.706+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:55:59.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:55:59.706+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:55:59.708 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:55:59.706+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:55:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:55:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1700.071664 secs 2024-07-26T11:56:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:00 smithi134 ceph-mon[31966]: pgmap v1746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:56:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:00 smithi189 ceph-mon[36790]: pgmap v1746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:56:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:00 smithi193 ceph-mon[37473]: pgmap v1746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:56:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1719.498773 secs 2024-07-26T11:56:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1719.498773 secs 2024-07-26T11:56:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1719.498773 secs 2024-07-26T11:56:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:02 smithi134 ceph-mon[31966]: pgmap v1747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:56:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:02 smithi189 ceph-mon[36790]: pgmap v1747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:56:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:02 smithi193 ceph-mon[37473]: pgmap v1747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:56:04.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:56:04.175+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:56:04.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:56:04.176+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:56:04.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:56:04.176+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:04 smithi134 ceph-mon[31966]: pgmap v1748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:56:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1705.071774 secs 2024-07-26T11:56:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:04 smithi189 ceph-mon[36790]: pgmap v1748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:56:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1705.071774 secs 2024-07-26T11:56:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:04 smithi193 ceph-mon[37473]: pgmap v1748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:56:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1705.071774 secs 2024-07-26T11:56:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:06 smithi134 ceph-mon[31966]: pgmap v1749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1724.498919 secs 2024-07-26T11:56:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:06 smithi189 ceph-mon[36790]: pgmap v1749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1724.498919 secs 2024-07-26T11:56:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:06 smithi193 ceph-mon[37473]: pgmap v1749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1724.498919 secs 2024-07-26T11:56:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:08 smithi134 ceph-mon[31966]: pgmap v1750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:08 smithi189 ceph-mon[36790]: pgmap v1750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:08 smithi193 ceph-mon[37473]: pgmap v1750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:09.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1710.071893 secs 2024-07-26T11:56:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1710.071893 secs 2024-07-26T11:56:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1710.071893 secs 2024-07-26T11:56:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:10 smithi193 ceph-mon[37473]: pgmap v1751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:10 smithi134 ceph-mon[31966]: pgmap v1751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:10 smithi189 ceph-mon[36790]: pgmap v1751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1729.499044 secs 2024-07-26T11:56:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1729.499044 secs 2024-07-26T11:56:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1729.499044 secs 2024-07-26T11:56:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:12 smithi193 ceph-mon[37473]: pgmap v1752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:13.321 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:12 smithi134 ceph-mon[31966]: pgmap v1752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:12 smithi189 ceph-mon[36790]: pgmap v1752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1715.072020 secs 2024-07-26T11:56:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1715.072020 secs 2024-07-26T11:56:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1715.072020 secs 2024-07-26T11:56:14.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:56:14.721+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:56:14.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:56:14.721+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:14.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:56:14.721+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:56:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:14 smithi193 ceph-mon[37473]: pgmap v1753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:15.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:14 smithi134 ceph-mon[31966]: pgmap v1753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:14 smithi189 ceph-mon[36790]: pgmap v1753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:16 smithi193 ceph-mon[37473]: pgmap v1754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1734.499107 secs 2024-07-26T11:56:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:16 smithi134 ceph-mon[31966]: pgmap v1754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1734.499107 secs 2024-07-26T11:56:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:16 smithi189 ceph-mon[36790]: pgmap v1754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1734.499107 secs 2024-07-26T11:56:19.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:56:19.191+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:56:19.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:56:19.191+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:19.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:56:19.191+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:56:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:18 smithi193 ceph-mon[37473]: pgmap v1755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1720.072115 secs 2024-07-26T11:56:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:18 smithi134 ceph-mon[31966]: pgmap v1755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1720.072115 secs 2024-07-26T11:56:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:18 smithi189 ceph-mon[36790]: pgmap v1755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:56:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1720.072115 secs 2024-07-26T11:56:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:20 smithi193 ceph-mon[37473]: pgmap v1756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:20 smithi134 ceph-mon[31966]: pgmap v1756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:20 smithi189 ceph-mon[36790]: pgmap v1756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:56:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1739.499236 secs 2024-07-26T11:56:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:56:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1739.499236 secs 2024-07-26T11:56:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:56:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1739.499236 secs 2024-07-26T11:56:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:56:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:23 smithi193 ceph-mon[37473]: pgmap v1757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:56:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:56:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:56:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:23 smithi134 ceph-mon[31966]: pgmap v1757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:56:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:56:23.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:56:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:23 smithi189 ceph-mon[36790]: pgmap v1757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:56:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:56:23.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:56:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:24 smithi193 ceph-mon[37473]: pgmap v1758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:56:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:24 smithi193 ceph-mon[37473]: pgmap v1759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1725.072195 secs 2024-07-26T11:56:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:24 smithi134 ceph-mon[31966]: pgmap v1758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:56:24.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:24 smithi134 ceph-mon[31966]: pgmap v1759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:24.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1725.072195 secs 2024-07-26T11:56:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:24 smithi189 ceph-mon[36790]: pgmap v1758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T11:56:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:24 smithi189 ceph-mon[36790]: pgmap v1759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1725.072195 secs 2024-07-26T11:56:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:26 smithi134 ceph-mon[31966]: pgmap v1760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:56:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:26 smithi189 ceph-mon[36790]: pgmap v1760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:56:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:26 smithi193 ceph-mon[37473]: pgmap v1760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T11:56:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1744.499363 secs 2024-07-26T11:56:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1744.499363 secs 2024-07-26T11:56:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1744.499363 secs 2024-07-26T11:56:28.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:28 smithi189 ceph-mon[36790]: pgmap v1761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:56:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:28 smithi134 ceph-mon[31966]: pgmap v1761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:56:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:28 smithi193 ceph-mon[37473]: pgmap v1761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T11:56:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1730.072315 secs 2024-07-26T11:56:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1730.072315 secs 2024-07-26T11:56:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1730.072315 secs 2024-07-26T11:56:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:56:29.729+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:56:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:56:29.729+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:56:29.729+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:56:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:30 smithi134 ceph-mon[31966]: pgmap v1762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:56:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:30 smithi189 ceph-mon[36790]: pgmap v1762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:56:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:30 smithi193 ceph-mon[37473]: pgmap v1762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:56:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:32 smithi134 ceph-mon[31966]: pgmap v1763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:56:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1749.499496 secs 2024-07-26T11:56:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:32 smithi189 ceph-mon[36790]: pgmap v1763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:56:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1749.499496 secs 2024-07-26T11:56:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:32 smithi193 ceph-mon[37473]: pgmap v1763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T11:56:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1749.499496 secs 2024-07-26T11:56:34.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:56:34.206+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:56:34.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:56:34.206+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:56:34.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:56:34.206+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:34 smithi134 ceph-mon[31966]: pgmap v1764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T11:56:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1735.072427 secs 2024-07-26T11:56:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:34 smithi189 ceph-mon[36790]: pgmap v1764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T11:56:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1735.072427 secs 2024-07-26T11:56:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:34 smithi193 ceph-mon[37473]: pgmap v1764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T11:56:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1735.072427 secs 2024-07-26T11:56:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:36 smithi134 ceph-mon[31966]: pgmap v1765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:36 smithi189 ceph-mon[36790]: pgmap v1765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:36 smithi193 ceph-mon[37473]: pgmap v1765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T11:56:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1754.499628 secs 2024-07-26T11:56:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1754.499628 secs 2024-07-26T11:56:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1754.499628 secs 2024-07-26T11:56:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:38 smithi134 ceph-mon[31966]: pgmap v1766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:38 smithi189 ceph-mon[36790]: pgmap v1766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:38 smithi193 ceph-mon[37473]: pgmap v1766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1740.072526 secs 2024-07-26T11:56:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1740.072526 secs 2024-07-26T11:56:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1740.072526 secs 2024-07-26T11:56:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:40 smithi134 ceph-mon[31966]: pgmap v1767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:40 smithi189 ceph-mon[36790]: pgmap v1767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:40 smithi193 ceph-mon[37473]: pgmap v1767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:42 smithi134 ceph-mon[31966]: pgmap v1768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1759.499705 secs 2024-07-26T11:56:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:42 smithi189 ceph-mon[36790]: pgmap v1768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1759.499705 secs 2024-07-26T11:56:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:42 smithi193 ceph-mon[37473]: pgmap v1768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1759.499705 secs 2024-07-26T11:56:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:44 smithi134 ceph-mon[31966]: pgmap v1769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1745.072663 secs 2024-07-26T11:56:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:44 smithi189 ceph-mon[36790]: pgmap v1769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1745.072663 secs 2024-07-26T11:56:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:44 smithi193 ceph-mon[37473]: pgmap v1769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1745.072663 secs 2024-07-26T11:56:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:56:44.744+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:56:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:56:44.744+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:44.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:56:44.744+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:56:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:46 smithi134 ceph-mon[31966]: pgmap v1770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:56:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:46 smithi189 ceph-mon[36790]: pgmap v1770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:56:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:46 smithi193 ceph-mon[37473]: pgmap v1770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:56:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1764.499829 secs 2024-07-26T11:56:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1764.499829 secs 2024-07-26T11:56:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1764.499829 secs 2024-07-26T11:56:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:48 smithi134 ceph-mon[31966]: pgmap v1771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:48 smithi189 ceph-mon[36790]: pgmap v1771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:48 smithi193 ceph-mon[37473]: pgmap v1771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:49.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:56:49.221+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:56:49.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:56:49.222+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:49.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:56:49.222+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:56:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1750.072782 secs 2024-07-26T11:56:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1750.072782 secs 2024-07-26T11:56:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1750.072782 secs 2024-07-26T11:56:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:50 smithi134 ceph-mon[31966]: pgmap v1772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:50 smithi189 ceph-mon[36790]: pgmap v1772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:50 smithi193 ceph-mon[37473]: pgmap v1772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:52 smithi134 ceph-mon[31966]: pgmap v1773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1769.499966 secs 2024-07-26T11:56:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:52 smithi189 ceph-mon[36790]: pgmap v1773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1769.499966 secs 2024-07-26T11:56:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:52 smithi193 ceph-mon[37473]: pgmap v1773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1769.499966 secs 2024-07-26T11:56:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:54 smithi134 ceph-mon[31966]: pgmap v1774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1755.072900 secs 2024-07-26T11:56:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:54 smithi189 ceph-mon[36790]: pgmap v1774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1755.072900 secs 2024-07-26T11:56:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:54 smithi193 ceph-mon[37473]: pgmap v1774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:56:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1755.072900 secs 2024-07-26T11:56:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:56 smithi134 ceph-mon[31966]: pgmap v1775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:56:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:56 smithi189 ceph-mon[36790]: pgmap v1775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:56:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:56 smithi193 ceph-mon[37473]: pgmap v1775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:56:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1774.500092 secs 2024-07-26T11:56:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1774.500092 secs 2024-07-26T11:56:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1774.500092 secs 2024-07-26T11:56:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:58 smithi189 ceph-mon[36790]: pgmap v1776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:58 smithi193 ceph-mon[37473]: pgmap v1776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:58 smithi134 ceph-mon[31966]: pgmap v1776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:56:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:56:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1760.073021 secs 2024-07-26T11:56:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:56:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1760.073021 secs 2024-07-26T11:56:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:56:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1760.073021 secs 2024-07-26T11:56:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:56:59.753+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:56:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:56:59.753+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:56:59.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:56:59.753+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:00.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:00 smithi189 ceph-mon[36790]: pgmap v1777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:00 smithi193 ceph-mon[37473]: pgmap v1777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:00 smithi134 ceph-mon[31966]: pgmap v1777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1779.500218 secs 2024-07-26T11:57:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1779.500218 secs 2024-07-26T11:57:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1779.500218 secs 2024-07-26T11:57:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:02 smithi193 ceph-mon[37473]: pgmap v1778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:02 smithi134 ceph-mon[31966]: pgmap v1778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:02 smithi189 ceph-mon[36790]: pgmap v1778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:04.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:04.237+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:04.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:04.237+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:04.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:04.237+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:04 smithi193 ceph-mon[37473]: pgmap v1779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1765.073134 secs 2024-07-26T11:57:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:04 smithi134 ceph-mon[31966]: pgmap v1779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1765.073134 secs 2024-07-26T11:57:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:04 smithi189 ceph-mon[36790]: pgmap v1779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1765.073134 secs 2024-07-26T11:57:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:06 smithi193 ceph-mon[37473]: pgmap v1780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:57:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1784.500373 secs 2024-07-26T11:57:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:06 smithi134 ceph-mon[31966]: pgmap v1780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:57:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1784.500373 secs 2024-07-26T11:57:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:06 smithi189 ceph-mon[36790]: pgmap v1780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:57:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1784.500373 secs 2024-07-26T11:57:08.475 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:08 smithi134 ceph-mon[31966]: pgmap v1781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:08 smithi193 ceph-mon[37473]: pgmap v1781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:08 smithi189 ceph-mon[36790]: pgmap v1781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1770.073291 secs 2024-07-26T11:57:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1770.073291 secs 2024-07-26T11:57:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1770.073291 secs 2024-07-26T11:57:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:10 smithi193 ceph-mon[37473]: pgmap v1782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:10 smithi134 ceph-mon[31966]: pgmap v1782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:10 smithi189 ceph-mon[36790]: pgmap v1782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1789.500456 secs 2024-07-26T11:57:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1789.500456 secs 2024-07-26T11:57:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1789.500456 secs 2024-07-26T11:57:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:12 smithi193 ceph-mon[37473]: pgmap v1783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:12 smithi134 ceph-mon[31966]: pgmap v1783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:12 smithi189 ceph-mon[36790]: pgmap v1783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:14 smithi193 ceph-mon[37473]: pgmap v1784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1775.073409 secs 2024-07-26T11:57:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:14 smithi134 ceph-mon[31966]: pgmap v1784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1775.073409 secs 2024-07-26T11:57:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:14 smithi189 ceph-mon[36790]: pgmap v1784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1775.073409 secs 2024-07-26T11:57:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:14.762+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:14.762+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:14.762+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:16 smithi193 ceph-mon[37473]: pgmap v1785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:57:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1794.500585 secs 2024-07-26T11:57:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:16 smithi134 ceph-mon[31966]: pgmap v1785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:57:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1794.500585 secs 2024-07-26T11:57:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:16 smithi189 ceph-mon[36790]: pgmap v1785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:57:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1794.500585 secs 2024-07-26T11:57:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:18 smithi193 ceph-mon[37473]: pgmap v1786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:18 smithi134 ceph-mon[31966]: pgmap v1786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:18 smithi189 ceph-mon[36790]: pgmap v1786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:19.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:19.241+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:19.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:19.241+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:19.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:19.241+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1780.073475 secs 2024-07-26T11:57:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:19 smithi134 ceph-mon[31966]: pgmap v1787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1780.073475 secs 2024-07-26T11:57:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:19 smithi189 ceph-mon[36790]: pgmap v1787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1780.073475 secs 2024-07-26T11:57:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:19 smithi193 ceph-mon[37473]: pgmap v1787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:21 smithi134 ceph-mon[31966]: pgmap v1788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1799.500716 secs 2024-07-26T11:57:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:21 smithi189 ceph-mon[36790]: pgmap v1788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1799.500716 secs 2024-07-26T11:57:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:21 smithi193 ceph-mon[37473]: pgmap v1788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1799.500716 secs 2024-07-26T11:57:22.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:22.288+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bf720 msgr2=0x7f9b300c1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T11:57:22.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:22.288+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bf720 msgr2=0x7f9b300c1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:57:22.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:22.288+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bf720 0x7f9b300c1b40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009960 tx=0x7f9b3802b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:57:22.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:22.288+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bf720 0x7f9b300c1b40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009960 tx=0x7f9b3802b040 comp rx=0 tx=0).stop 2024-07-26T11:57:22.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:22.288+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bf720 msgr2=0x7f9b300c1b40 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T11:57:22.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:22.288+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bf720 msgr2=0x7f9b300c1b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:57:22.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:22.288+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bf720 0x7f9b300c1b40 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:57:22.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:22.288+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300c2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:57:22.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:22.289+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300c2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:57:22.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:22.291+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300c2120 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009990 tx=0x7f9b3802b040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:57:22.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:22.641+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T11:57:22.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:22.641+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T11:57:22.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:22.642+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T11:57:22.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:22.642+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T11:57:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:23 smithi134 ceph-mon[31966]: pgmap v1789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:23.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:57:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:23 smithi189 ceph-mon[36790]: pgmap v1789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:57:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:23 smithi193 ceph-mon[37473]: pgmap v1789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:57:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1785.073567 secs 2024-07-26T11:57:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:57:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:57:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:24 smithi134 ceph-mon[31966]: pgmap v1790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:57:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:24 smithi134 ceph-mon[31966]: pgmap v1791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:57:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1785.073567 secs 2024-07-26T11:57:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:57:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:57:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:24 smithi189 ceph-mon[36790]: pgmap v1790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:57:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:24 smithi189 ceph-mon[36790]: pgmap v1791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:57:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1785.073567 secs 2024-07-26T11:57:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:57:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:57:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:24 smithi193 ceph-mon[37473]: pgmap v1790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:57:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:24 smithi193 ceph-mon[37473]: pgmap v1791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:57:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:26 smithi193 ceph-mon[37473]: pgmap v1792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1804.500845 secs 2024-07-26T11:57:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:26 smithi134 ceph-mon[31966]: pgmap v1792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1804.500845 secs 2024-07-26T11:57:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:26 smithi189 ceph-mon[36790]: pgmap v1792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1804.500845 secs 2024-07-26T11:57:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:28 smithi193 ceph-mon[37473]: pgmap v1793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1790.073691 secs 2024-07-26T11:57:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:28 smithi134 ceph-mon[31966]: pgmap v1793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1790.073691 secs 2024-07-26T11:57:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:28 smithi189 ceph-mon[36790]: pgmap v1793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:57:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1790.073691 secs 2024-07-26T11:57:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:29.770+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:29.770+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:29.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:29.770+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:30 smithi193 ceph-mon[37473]: pgmap v1794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:57:31.145 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:30 smithi134 ceph-mon[31966]: pgmap v1794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:57:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:30 smithi189 ceph-mon[36790]: pgmap v1794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:57:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1809.500951 secs 2024-07-26T11:57:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1809.500951 secs 2024-07-26T11:57:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1809.500951 secs 2024-07-26T11:57:32.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:32.844+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bf5f0 msgr2=0x7f21ac0c1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T11:57:32.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:32.844+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bf5f0 msgr2=0x7f21ac0c1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T11:57:32.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:32.844+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bf5f0 0x7f21ac0c1a10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f21b8006010 tx=0x7f21b8109070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T11:57:32.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:32.844+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bf5f0 0x7f21ac0c1a10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f21b8006010 tx=0x7f21b8109070 comp rx=0 tx=0).stop 2024-07-26T11:57:32.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:32.844+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bf5f0 msgr2=0x7f21ac0c1a10 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T11:57:32.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:32.844+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bf5f0 msgr2=0x7f21ac0c1a10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T11:57:32.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:32.844+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bf5f0 0x7f21ac0c1a10 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T11:57:32.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:32.844+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 0x7f21ac0c1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T11:57:32.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:32.845+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 0x7f21ac0c1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T11:57:32.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:32.846+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 0x7f21ac0c1ff0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2198003470 tx=0x7f21b810a000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T11:57:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:33.142+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006660 con 0x55a08e2d94d0 2024-07-26T11:57:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:33.142+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T11:57:33.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:33.143+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T11:57:33.145 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:33.143+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006660 con 0x55a08e2d94d0 2024-07-26T11:57:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:32 smithi134 ceph-mon[31966]: pgmap v1795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T11:57:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:32 smithi189 ceph-mon[36790]: pgmap v1795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T11:57:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:32 smithi193 ceph-mon[37473]: pgmap v1795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T11:57:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1795.073833 secs 2024-07-26T11:57:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1795.073833 secs 2024-07-26T11:57:34.244 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:34.242+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:34.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:34.242+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:34.245 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:34.242+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1795.073833 secs 2024-07-26T11:57:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:34 smithi134 ceph-mon[31966]: pgmap v1796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:57:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:34 smithi189 ceph-mon[36790]: pgmap v1796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:57:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:34 smithi193 ceph-mon[37473]: pgmap v1796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:57:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:36 smithi134 ceph-mon[31966]: pgmap v1797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1814.501081 secs 2024-07-26T11:57:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:36 smithi189 ceph-mon[36790]: pgmap v1797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1814.501081 secs 2024-07-26T11:57:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:36 smithi193 ceph-mon[37473]: pgmap v1797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1814.501081 secs 2024-07-26T11:57:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:38 smithi134 ceph-mon[31966]: pgmap v1798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1800.073979 secs 2024-07-26T11:57:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:38 smithi189 ceph-mon[36790]: pgmap v1798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1800.073979 secs 2024-07-26T11:57:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:38 smithi193 ceph-mon[37473]: pgmap v1798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1800.073979 secs 2024-07-26T11:57:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:40 smithi134 ceph-mon[31966]: pgmap v1799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:57:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:40 smithi189 ceph-mon[36790]: pgmap v1799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:57:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:40 smithi193 ceph-mon[37473]: pgmap v1799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:57:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1819.501212 secs 2024-07-26T11:57:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1819.501212 secs 2024-07-26T11:57:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1819.501212 secs 2024-07-26T11:57:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:42 smithi134 ceph-mon[31966]: pgmap v1800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:42 smithi189 ceph-mon[36790]: pgmap v1800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:42 smithi193 ceph-mon[37473]: pgmap v1800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1805.074095 secs 2024-07-26T11:57:44.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1805.074095 secs 2024-07-26T11:57:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1805.074095 secs 2024-07-26T11:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:44.775+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:44.775+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:44.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:44.775+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:44 smithi134 ceph-mon[31966]: pgmap v1801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:44 smithi189 ceph-mon[36790]: pgmap v1801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:44 smithi193 ceph-mon[37473]: pgmap v1801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:57:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:46 smithi134 ceph-mon[31966]: pgmap v1802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1824.501360 secs 2024-07-26T11:57:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:46 smithi189 ceph-mon[36790]: pgmap v1802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1824.501360 secs 2024-07-26T11:57:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:46 smithi193 ceph-mon[37473]: pgmap v1802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:57:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1824.501360 secs 2024-07-26T11:57:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:48 smithi134 ceph-mon[31966]: pgmap v1803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T11:57:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1810.074222 secs 2024-07-26T11:57:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:48 smithi189 ceph-mon[36790]: pgmap v1803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T11:57:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1810.074222 secs 2024-07-26T11:57:49.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:49.258+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:49.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:49.258+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:57:49.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:57:49.258+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:48 smithi193 ceph-mon[37473]: pgmap v1803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T11:57:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1810.074222 secs 2024-07-26T11:57:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:50 smithi134 ceph-mon[31966]: pgmap v1804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:57:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:50 smithi189 ceph-mon[36790]: pgmap v1804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:57:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:50 smithi193 ceph-mon[37473]: pgmap v1804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T11:57:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1829.501488 secs 2024-07-26T11:57:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1829.501488 secs 2024-07-26T11:57:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1829.501488 secs 2024-07-26T11:57:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:52 smithi134 ceph-mon[31966]: pgmap v1805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:57:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:52 smithi189 ceph-mon[36790]: pgmap v1805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:57:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:52 smithi193 ceph-mon[37473]: pgmap v1805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:57:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1815.074345 secs 2024-07-26T11:57:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1815.074345 secs 2024-07-26T11:57:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1815.074345 secs 2024-07-26T11:57:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:54 smithi134 ceph-mon[31966]: pgmap v1806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:57:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:54 smithi189 ceph-mon[36790]: pgmap v1806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:57:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:54 smithi193 ceph-mon[37473]: pgmap v1806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:57:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:56 smithi134 ceph-mon[31966]: pgmap v1807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:57:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1834.501609 secs 2024-07-26T11:57:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:56 smithi189 ceph-mon[36790]: pgmap v1807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:57:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1834.501609 secs 2024-07-26T11:57:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:56 smithi193 ceph-mon[37473]: pgmap v1807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T11:57:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1834.501609 secs 2024-07-26T11:57:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:58 smithi134 ceph-mon[31966]: pgmap v1808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:57:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:57:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1820.074458 secs 2024-07-26T11:57:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:58 smithi189 ceph-mon[36790]: pgmap v1808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:57:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:57:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1820.074458 secs 2024-07-26T11:57:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:58 smithi193 ceph-mon[37473]: pgmap v1808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:57:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:57:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1820.074458 secs 2024-07-26T11:57:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:59.791+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:59.791+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:57:59.793 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:57:59.791+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:00 smithi134 ceph-mon[31966]: pgmap v1809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:00 smithi189 ceph-mon[36790]: pgmap v1809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:00 smithi193 ceph-mon[37473]: pgmap v1809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1839.501752 secs 2024-07-26T11:58:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1839.501752 secs 2024-07-26T11:58:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1839.501752 secs 2024-07-26T11:58:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:02 smithi134 ceph-mon[31966]: pgmap v1810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:02 smithi189 ceph-mon[36790]: pgmap v1810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:02 smithi193 ceph-mon[37473]: pgmap v1810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1825.074604 secs 2024-07-26T11:58:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1825.074604 secs 2024-07-26T11:58:04.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:58:04.274+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:58:04.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:58:04.274+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:58:04.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:58:04.274+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:04.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1825.074604 secs 2024-07-26T11:58:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:04 smithi189 ceph-mon[36790]: pgmap v1811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:04 smithi193 ceph-mon[37473]: pgmap v1811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:04 smithi134 ceph-mon[31966]: pgmap v1811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:06 smithi189 ceph-mon[36790]: pgmap v1812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1844.501836 secs 2024-07-26T11:58:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:06 smithi193 ceph-mon[37473]: pgmap v1812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1844.501836 secs 2024-07-26T11:58:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:06 smithi134 ceph-mon[31966]: pgmap v1812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1844.501836 secs 2024-07-26T11:58:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:08 smithi189 ceph-mon[36790]: pgmap v1813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1830.074730 secs 2024-07-26T11:58:09.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:08 smithi134 ceph-mon[31966]: pgmap v1813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:09.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1830.074730 secs 2024-07-26T11:58:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:08 smithi193 ceph-mon[37473]: pgmap v1813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1830.074730 secs 2024-07-26T11:58:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:10 smithi193 ceph-mon[37473]: pgmap v1814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:10 smithi134 ceph-mon[31966]: pgmap v1814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:10 smithi189 ceph-mon[36790]: pgmap v1814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1849.501940 secs 2024-07-26T11:58:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1849.501940 secs 2024-07-26T11:58:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1849.501940 secs 2024-07-26T11:58:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:12 smithi193 ceph-mon[37473]: pgmap v1815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:12 smithi134 ceph-mon[31966]: pgmap v1815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:12 smithi189 ceph-mon[36790]: pgmap v1815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1835.074870 secs 2024-07-26T11:58:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1835.074870 secs 2024-07-26T11:58:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1835.074870 secs 2024-07-26T11:58:14.808 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:58:14.806+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:58:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:58:14.807+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:58:14.809 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:58:14.807+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:14 smithi193 ceph-mon[37473]: pgmap v1816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:14 smithi134 ceph-mon[31966]: pgmap v1816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:14 smithi189 ceph-mon[36790]: pgmap v1816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:16 smithi193 ceph-mon[37473]: pgmap v1817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1854.502029 secs 2024-07-26T11:58:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:16 smithi134 ceph-mon[31966]: pgmap v1817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1854.502029 secs 2024-07-26T11:58:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:16 smithi189 ceph-mon[36790]: pgmap v1817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1854.502029 secs 2024-07-26T11:58:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:18 smithi193 ceph-mon[37473]: pgmap v1818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1840.074995 secs 2024-07-26T11:58:19.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:58:19.288+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:58:19.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:58:19.289+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:58:19.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:58:19.289+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:19.291 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:18 smithi134 ceph-mon[31966]: pgmap v1818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:19.291 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1840.074995 secs 2024-07-26T11:58:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:18 smithi189 ceph-mon[36790]: pgmap v1818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1840.074995 secs 2024-07-26T11:58:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:20 smithi193 ceph-mon[37473]: pgmap v1819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:20 smithi134 ceph-mon[31966]: pgmap v1819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:20 smithi189 ceph-mon[36790]: pgmap v1819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1859.502133 secs 2024-07-26T11:58:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1859.502133 secs 2024-07-26T11:58:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1859.502133 secs 2024-07-26T11:58:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:23 smithi193 ceph-mon[37473]: pgmap v1820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:23 smithi134 ceph-mon[31966]: pgmap v1820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:23 smithi189 ceph-mon[36790]: pgmap v1820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1845.075071 secs 2024-07-26T11:58:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:58:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1845.075071 secs 2024-07-26T11:58:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:58:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1845.075071 secs 2024-07-26T11:58:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:58:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:25 smithi134 ceph-mon[31966]: pgmap v1821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:58:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:58:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:58:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:25 smithi189 ceph-mon[36790]: pgmap v1821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:58:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:58:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:58:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:25 smithi193 ceph-mon[37473]: pgmap v1821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:58:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:58:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:58:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:26 smithi134 ceph-mon[31966]: pgmap v1822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:58:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:26 smithi134 ceph-mon[31966]: pgmap v1823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:58:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:26 smithi189 ceph-mon[36790]: pgmap v1822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:58:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:26 smithi189 ceph-mon[36790]: pgmap v1823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:58:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:26 smithi193 ceph-mon[37473]: pgmap v1822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:58:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:26 smithi193 ceph-mon[37473]: pgmap v1823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T11:58:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1864.502280 secs 2024-07-26T11:58:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1864.502280 secs 2024-07-26T11:58:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1864.502280 secs 2024-07-26T11:58:28.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:28 smithi189 ceph-mon[36790]: pgmap v1824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:58:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:28 smithi134 ceph-mon[31966]: pgmap v1824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:58:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:28 smithi193 ceph-mon[37473]: pgmap v1824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T11:58:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1850.075217 secs 2024-07-26T11:58:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1850.075217 secs 2024-07-26T11:58:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1850.075217 secs 2024-07-26T11:58:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:58:29.809+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:58:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:58:29.809+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:58:29.811 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:58:29.809+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:30 smithi134 ceph-mon[31966]: pgmap v1825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:58:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:30 smithi189 ceph-mon[36790]: pgmap v1825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:58:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:30 smithi193 ceph-mon[37473]: pgmap v1825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:58:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1869.502403 secs 2024-07-26T11:58:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:32 smithi134 ceph-mon[31966]: pgmap v1826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:58:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1869.502403 secs 2024-07-26T11:58:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:32 smithi189 ceph-mon[36790]: pgmap v1826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:58:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1869.502403 secs 2024-07-26T11:58:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:32 smithi193 ceph-mon[37473]: pgmap v1826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:58:34.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:58:34.294+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:58:34.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:58:34.294+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:34.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:58:34.294+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:58:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:34 smithi134 ceph-mon[31966]: pgmap v1827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:58:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1855.075339 secs 2024-07-26T11:58:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:34 smithi189 ceph-mon[36790]: pgmap v1827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:58:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1855.075339 secs 2024-07-26T11:58:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:34 smithi193 ceph-mon[37473]: pgmap v1827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T11:58:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1855.075339 secs 2024-07-26T11:58:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:36 smithi134 ceph-mon[31966]: pgmap v1828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:58:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:36 smithi189 ceph-mon[36790]: pgmap v1828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:58:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:36 smithi193 ceph-mon[37473]: pgmap v1828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:58:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1874.502521 secs 2024-07-26T11:58:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1874.502521 secs 2024-07-26T11:58:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1874.502521 secs 2024-07-26T11:58:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:38 smithi134 ceph-mon[31966]: pgmap v1829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:38 smithi189 ceph-mon[36790]: pgmap v1829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:38 smithi193 ceph-mon[37473]: pgmap v1829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1860.075485 secs 2024-07-26T11:58:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1860.075485 secs 2024-07-26T11:58:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1860.075485 secs 2024-07-26T11:58:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:40 smithi134 ceph-mon[31966]: pgmap v1830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:40.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:40 smithi189 ceph-mon[36790]: pgmap v1830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:40 smithi193 ceph-mon[37473]: pgmap v1830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1879.502625 secs 2024-07-26T11:58:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:42 smithi134 ceph-mon[31966]: pgmap v1831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1879.502625 secs 2024-07-26T11:58:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:42 smithi189 ceph-mon[36790]: pgmap v1831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1879.502625 secs 2024-07-26T11:58:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:42 smithi193 ceph-mon[37473]: pgmap v1831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:44 smithi134 ceph-mon[31966]: pgmap v1832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1865.075609 secs 2024-07-26T11:58:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:44 smithi189 ceph-mon[36790]: pgmap v1832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1865.075609 secs 2024-07-26T11:58:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:44 smithi193 ceph-mon[37473]: pgmap v1832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1865.075609 secs 2024-07-26T11:58:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:58:44.811+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:58:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:58:44.811+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:58:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:58:44.811+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:46 smithi134 ceph-mon[31966]: pgmap v1833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:46 smithi189 ceph-mon[36790]: pgmap v1833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:46 smithi193 ceph-mon[37473]: pgmap v1833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1884.502779 secs 2024-07-26T11:58:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1884.502779 secs 2024-07-26T11:58:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1884.502779 secs 2024-07-26T11:58:48.325 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:48 smithi134 ceph-mon[31966]: pgmap v1834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:48 smithi189 ceph-mon[36790]: pgmap v1834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:48 smithi193 ceph-mon[37473]: pgmap v1834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:49.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:58:49.310+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:58:49.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:58:49.310+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:58:49.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:58:49.310+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:58:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1870.075733 secs 2024-07-26T11:58:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1870.075733 secs 2024-07-26T11:58:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1870.075733 secs 2024-07-26T11:58:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:50 smithi134 ceph-mon[31966]: pgmap v1835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:50 smithi189 ceph-mon[36790]: pgmap v1835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:50 smithi193 ceph-mon[37473]: pgmap v1835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:58:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1889.502900 secs 2024-07-26T11:58:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:52 smithi134 ceph-mon[31966]: pgmap v1836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1889.502900 secs 2024-07-26T11:58:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:52 smithi189 ceph-mon[36790]: pgmap v1836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1889.502900 secs 2024-07-26T11:58:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:52 smithi193 ceph-mon[37473]: pgmap v1836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:58:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:54 smithi134 ceph-mon[31966]: pgmap v1837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1875.075838 secs 2024-07-26T11:58:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:54 smithi189 ceph-mon[36790]: pgmap v1837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1875.075838 secs 2024-07-26T11:58:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:54 smithi193 ceph-mon[37473]: pgmap v1837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:58:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1875.075838 secs 2024-07-26T11:58:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:56 smithi134 ceph-mon[31966]: pgmap v1838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:56 smithi189 ceph-mon[36790]: pgmap v1838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:56 smithi193 ceph-mon[37473]: pgmap v1838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:58:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1894.503024 secs 2024-07-26T11:58:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1894.503024 secs 2024-07-26T11:58:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1894.503024 secs 2024-07-26T11:58:58.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:58 smithi189 ceph-mon[36790]: pgmap v1839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:58 smithi193 ceph-mon[37473]: pgmap v1839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:58 smithi134 ceph-mon[31966]: pgmap v1839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:58:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:58:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1880.075903 secs 2024-07-26T11:58:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:58:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1880.075903 secs 2024-07-26T11:58:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:58:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1880.075903 secs 2024-07-26T11:58:59.828 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:58:59.826+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:58:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:58:59.827+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:58:59.829 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:58:59.827+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:00 smithi193 ceph-mon[37473]: pgmap v1840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:00 smithi134 ceph-mon[31966]: pgmap v1840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:00 smithi189 ceph-mon[36790]: pgmap v1840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1899.503126 secs 2024-07-26T11:59:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1899.503126 secs 2024-07-26T11:59:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1899.503126 secs 2024-07-26T11:59:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:02 smithi193 ceph-mon[37473]: pgmap v1841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:02 smithi134 ceph-mon[31966]: pgmap v1841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:02 smithi189 ceph-mon[36790]: pgmap v1841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:04.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:59:04.325+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:59:04.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:59:04.325+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:04.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:59:04.325+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:59:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:04 smithi193 ceph-mon[37473]: pgmap v1842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1885.076035 secs 2024-07-26T11:59:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:04 smithi134 ceph-mon[31966]: pgmap v1842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1885.076035 secs 2024-07-26T11:59:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:04 smithi189 ceph-mon[36790]: pgmap v1842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1885.076035 secs 2024-07-26T11:59:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:06 smithi193 ceph-mon[37473]: pgmap v1843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:59:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1904.503255 secs 2024-07-26T11:59:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:06 smithi134 ceph-mon[31966]: pgmap v1843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:59:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1904.503255 secs 2024-07-26T11:59:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:06 smithi189 ceph-mon[36790]: pgmap v1843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:59:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1904.503255 secs 2024-07-26T11:59:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:08 smithi193 ceph-mon[37473]: pgmap v1844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:08 smithi134 ceph-mon[31966]: pgmap v1844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:08 smithi189 ceph-mon[36790]: pgmap v1844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1890.076161 secs 2024-07-26T11:59:09.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1890.076161 secs 2024-07-26T11:59:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1890.076161 secs 2024-07-26T11:59:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:10 smithi193 ceph-mon[37473]: pgmap v1845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:10 smithi134 ceph-mon[31966]: pgmap v1845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:10 smithi189 ceph-mon[36790]: pgmap v1845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1909.503386 secs 2024-07-26T11:59:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1909.503386 secs 2024-07-26T11:59:11.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1909.503386 secs 2024-07-26T11:59:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:12 smithi193 ceph-mon[37473]: pgmap v1846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:12.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:12 smithi134 ceph-mon[31966]: pgmap v1846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:12 smithi189 ceph-mon[36790]: pgmap v1846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:14 smithi193 ceph-mon[37473]: pgmap v1847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1895.076227 secs 2024-07-26T11:59:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:14 smithi134 ceph-mon[31966]: pgmap v1847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1895.076227 secs 2024-07-26T11:59:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:14 smithi189 ceph-mon[36790]: pgmap v1847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1895.076227 secs 2024-07-26T11:59:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:59:14.836+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:59:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:59:14.836+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:59:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:59:14.836+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:16 smithi193 ceph-mon[37473]: pgmap v1848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:59:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1914.503518 secs 2024-07-26T11:59:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:16 smithi134 ceph-mon[31966]: pgmap v1848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:59:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1914.503518 secs 2024-07-26T11:59:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:16 smithi189 ceph-mon[36790]: pgmap v1848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:59:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1914.503518 secs 2024-07-26T11:59:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:18 smithi193 ceph-mon[37473]: pgmap v1849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:18 smithi134 ceph-mon[31966]: pgmap v1849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:18 smithi189 ceph-mon[36790]: pgmap v1849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:19.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:59:19.334+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:59:19.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:59:19.334+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:19.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:59:19.334+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:59:19.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1900.076298 secs 2024-07-26T11:59:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1900.076298 secs 2024-07-26T11:59:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1900.076298 secs 2024-07-26T11:59:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:20 smithi134 ceph-mon[31966]: pgmap v1850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:20 smithi189 ceph-mon[36790]: pgmap v1850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:20 smithi193 ceph-mon[37473]: pgmap v1850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1919.503601 secs 2024-07-26T11:59:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1919.503601 secs 2024-07-26T11:59:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1919.503601 secs 2024-07-26T11:59:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:22 smithi134 ceph-mon[31966]: pgmap v1851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:22 smithi189 ceph-mon[36790]: pgmap v1851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:22 smithi193 ceph-mon[37473]: pgmap v1851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:23 smithi134 ceph-mon[31966]: pgmap v1852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:23 smithi189 ceph-mon[36790]: pgmap v1852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:23 smithi193 ceph-mon[37473]: pgmap v1852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1905.076425 secs 2024-07-26T11:59:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1905.076425 secs 2024-07-26T11:59:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1905.076425 secs 2024-07-26T11:59:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:25 smithi134 ceph-mon[31966]: pgmap v1853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:59:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:59:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:25 smithi189 ceph-mon[36790]: pgmap v1853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:59:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:59:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:25 smithi193 ceph-mon[37473]: pgmap v1853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T11:59:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T11:59:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1924.503692 secs 2024-07-26T11:59:26.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:59:26.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:59:26.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:26 smithi134 ceph-mon[31966]: pgmap v1854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T11:59:26.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:59:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1924.503692 secs 2024-07-26T11:59:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:59:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:59:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:26 smithi189 ceph-mon[36790]: pgmap v1854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T11:59:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:59:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1924.503692 secs 2024-07-26T11:59:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T11:59:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T11:59:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:26 smithi193 ceph-mon[37473]: pgmap v1854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T11:59:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T11:59:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:29 smithi134 ceph-mon[31966]: pgmap v1855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:59:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1910.076526 secs 2024-07-26T11:59:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:29 smithi189 ceph-mon[36790]: pgmap v1855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:59:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1910.076526 secs 2024-07-26T11:59:29.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:29 smithi193 ceph-mon[37473]: pgmap v1855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T11:59:29.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1910.076526 secs 2024-07-26T11:59:29.853 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:59:29.852+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:59:29.853 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:59:29.853+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:59:29.853 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:59:29.853+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:31 smithi134 ceph-mon[31966]: pgmap v1856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:31 smithi189 ceph-mon[36790]: pgmap v1856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:31 smithi193 ceph-mon[37473]: pgmap v1856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1929.503795 secs 2024-07-26T11:59:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1929.503795 secs 2024-07-26T11:59:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1929.503795 secs 2024-07-26T11:59:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:33 smithi134 ceph-mon[31966]: pgmap v1857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:59:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:33 smithi189 ceph-mon[36790]: pgmap v1857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:59:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:33 smithi193 ceph-mon[37473]: pgmap v1857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T11:59:34.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:59:34.349+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:59:34.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:59:34.350+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:59:34.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:59:34.350+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1915.076671 secs 2024-07-26T11:59:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1915.076671 secs 2024-07-26T11:59:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1915.076671 secs 2024-07-26T11:59:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:35 smithi134 ceph-mon[31966]: pgmap v1858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:35 smithi189 ceph-mon[36790]: pgmap v1858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:35 smithi193 ceph-mon[37473]: pgmap v1858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1934.503911 secs 2024-07-26T11:59:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1934.503911 secs 2024-07-26T11:59:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1934.503911 secs 2024-07-26T11:59:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:37 smithi193 ceph-mon[37473]: pgmap v1859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:37 smithi134 ceph-mon[31966]: pgmap v1859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:37 smithi189 ceph-mon[36790]: pgmap v1859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T11:59:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:39 smithi193 ceph-mon[37473]: pgmap v1860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1920.076799 secs 2024-07-26T11:59:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:39 smithi134 ceph-mon[31966]: pgmap v1860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1920.076799 secs 2024-07-26T11:59:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:39 smithi189 ceph-mon[36790]: pgmap v1860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1920.076799 secs 2024-07-26T11:59:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:41 smithi193 ceph-mon[37473]: pgmap v1861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1939.504028 secs 2024-07-26T11:59:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:41 smithi134 ceph-mon[31966]: pgmap v1861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1939.504028 secs 2024-07-26T11:59:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:41 smithi189 ceph-mon[36790]: pgmap v1861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1939.504028 secs 2024-07-26T11:59:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:43 smithi193 ceph-mon[37473]: pgmap v1862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:43 smithi134 ceph-mon[31966]: pgmap v1862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:43 smithi189 ceph-mon[36790]: pgmap v1862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1925.076926 secs 2024-07-26T11:59:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1925.076926 secs 2024-07-26T11:59:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1925.076926 secs 2024-07-26T11:59:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:59:44.867+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:59:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:59:44.867+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:59:44.869 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:59:44.867+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:45 smithi193 ceph-mon[37473]: pgmap v1863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:45 smithi134 ceph-mon[31966]: pgmap v1863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:45 smithi189 ceph-mon[36790]: pgmap v1863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1944.504134 secs 2024-07-26T11:59:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1944.504134 secs 2024-07-26T11:59:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1944.504134 secs 2024-07-26T11:59:47.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:47 smithi134 ceph-mon[31966]: pgmap v1864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:47 smithi189 ceph-mon[36790]: pgmap v1864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:47 smithi193 ceph-mon[37473]: pgmap v1864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:49.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:59:49.365+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:59:49.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:59:49.365+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T11:59:49.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T11:59:49.365+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:59:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:49 smithi134 ceph-mon[31966]: pgmap v1865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1930.076972 secs 2024-07-26T11:59:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:49 smithi189 ceph-mon[36790]: pgmap v1865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1930.076972 secs 2024-07-26T11:59:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:49 smithi193 ceph-mon[37473]: pgmap v1865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1930.076972 secs 2024-07-26T11:59:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:51 smithi134 ceph-mon[31966]: pgmap v1866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1949.504255 secs 2024-07-26T11:59:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:51 smithi189 ceph-mon[36790]: pgmap v1866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1949.504255 secs 2024-07-26T11:59:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:51 smithi193 ceph-mon[37473]: pgmap v1866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1949.504255 secs 2024-07-26T11:59:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:52 smithi134 ceph-mon[31966]: pgmap v1867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:52 smithi189 ceph-mon[36790]: pgmap v1867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:52 smithi193 ceph-mon[37473]: pgmap v1867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T11:59:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1935.077090 secs 2024-07-26T11:59:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1935.077090 secs 2024-07-26T11:59:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1935.077090 secs 2024-07-26T11:59:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:54 smithi193 ceph-mon[37473]: pgmap v1868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:54 smithi134 ceph-mon[31966]: pgmap v1868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:54 smithi189 ceph-mon[36790]: pgmap v1868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T11:59:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1954.504386 secs 2024-07-26T11:59:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:57 smithi134 ceph-mon[31966]: pgmap v1869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1954.504386 secs 2024-07-26T11:59:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:57 smithi189 ceph-mon[36790]: pgmap v1869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1954.504386 secs 2024-07-26T11:59:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:57 smithi193 ceph-mon[37473]: pgmap v1869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T11:59:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:59 smithi189 ceph-mon[36790]: pgmap v1870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 11:59:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1940.077194 secs 2024-07-26T11:59:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:59 smithi193 ceph-mon[37473]: pgmap v1870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 11:59:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1940.077194 secs 2024-07-26T11:59:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:59 smithi134 ceph-mon[31966]: pgmap v1870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T11:59:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 11:59:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1940.077194 secs 2024-07-26T11:59:59.874 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:59:59.872+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T11:59:59.874 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:59:59.872+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T11:59:59.874 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T11:59:59.872+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:00.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:00:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:00:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:00:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:01 smithi189 ceph-mon[36790]: pgmap v1871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:01 smithi193 ceph-mon[37473]: pgmap v1871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:01 smithi134 ceph-mon[31966]: pgmap v1871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1959.504452 secs 2024-07-26T12:00:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1959.504452 secs 2024-07-26T12:00:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1959.504452 secs 2024-07-26T12:00:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:03 smithi193 ceph-mon[37473]: pgmap v1872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:03 smithi134 ceph-mon[31966]: pgmap v1872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:03 smithi189 ceph-mon[36790]: pgmap v1872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:04.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:04.381+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:00:04.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:04.381+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:04.383 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:04.381+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:00:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1945.077311 secs 2024-07-26T12:00:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1945.077311 secs 2024-07-26T12:00:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1945.077311 secs 2024-07-26T12:00:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:05 smithi193 ceph-mon[37473]: pgmap v1873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:05 smithi134 ceph-mon[31966]: pgmap v1873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:05 smithi189 ceph-mon[36790]: pgmap v1873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1964.504558 secs 2024-07-26T12:00:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1964.504558 secs 2024-07-26T12:00:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1964.504558 secs 2024-07-26T12:00:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:07 smithi193 ceph-mon[37473]: pgmap v1874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:07 smithi134 ceph-mon[31966]: pgmap v1874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:07 smithi189 ceph-mon[36790]: pgmap v1874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:09 smithi193 ceph-mon[37473]: pgmap v1875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1950.077438 secs 2024-07-26T12:00:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:09 smithi134 ceph-mon[31966]: pgmap v1875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1950.077438 secs 2024-07-26T12:00:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:09 smithi189 ceph-mon[36790]: pgmap v1875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1950.077438 secs 2024-07-26T12:00:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:11 smithi134 ceph-mon[31966]: pgmap v1876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:11 smithi189 ceph-mon[36790]: pgmap v1876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:11 smithi193 ceph-mon[37473]: pgmap v1876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1969.504671 secs 2024-07-26T12:00:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1969.504671 secs 2024-07-26T12:00:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1969.504671 secs 2024-07-26T12:00:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:13 smithi134 ceph-mon[31966]: pgmap v1877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:13 smithi189 ceph-mon[36790]: pgmap v1877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:13 smithi193 ceph-mon[37473]: pgmap v1877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:14.357 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1955.077514 secs 2024-07-26T12:00:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1955.077514 secs 2024-07-26T12:00:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1955.077514 secs 2024-07-26T12:00:14.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:14.888+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:00:14.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:14.888+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:00:14.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:14.888+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:15 smithi134 ceph-mon[31966]: pgmap v1878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:15 smithi189 ceph-mon[36790]: pgmap v1878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:15 smithi193 ceph-mon[37473]: pgmap v1878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1974.504801 secs 2024-07-26T12:00:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:17 smithi189 ceph-mon[36790]: pgmap v1879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1974.504801 secs 2024-07-26T12:00:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:17 smithi193 ceph-mon[37473]: pgmap v1879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1974.504801 secs 2024-07-26T12:00:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:17 smithi134 ceph-mon[31966]: pgmap v1879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:19.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:19.388+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:00:19.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:19.388+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:19.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:19.388+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:00:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:19 smithi189 ceph-mon[36790]: pgmap v1880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:19.463 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1960.077639 secs 2024-07-26T12:00:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:19 smithi193 ceph-mon[37473]: pgmap v1880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1960.077639 secs 2024-07-26T12:00:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:19 smithi134 ceph-mon[31966]: pgmap v1880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1960.077639 secs 2024-07-26T12:00:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:21 smithi134 ceph-mon[31966]: pgmap v1881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:21 smithi189 ceph-mon[36790]: pgmap v1881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:21 smithi193 ceph-mon[37473]: pgmap v1881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1979.504866 secs 2024-07-26T12:00:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1979.504866 secs 2024-07-26T12:00:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1979.504866 secs 2024-07-26T12:00:22.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:22.673+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T12:00:22.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:22.673+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T12:00:22.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:22.674+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T12:00:22.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:22.674+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T12:00:22.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:22.674+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x55c65e9ce530 con 0x7f9b4c10f330 2024-07-26T12:00:22.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:22.675+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T12:00:22.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:22.675+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:00:22.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:22.675+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x55c65e9ce530 con 0x55c65e5ee670 2024-07-26T12:00:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:23 smithi134 ceph-mon[31966]: pgmap v1882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:23 smithi189 ceph-mon[36790]: pgmap v1882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:23 smithi193 ceph-mon[37473]: pgmap v1882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1965.077761 secs 2024-07-26T12:00:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1965.077761 secs 2024-07-26T12:00:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1965.077761 secs 2024-07-26T12:00:25.391 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:25 smithi189 ceph-mon[36790]: pgmap v1883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:25 smithi134 ceph-mon[31966]: pgmap v1883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:25 smithi193 ceph-mon[37473]: pgmap v1883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1984.504987 secs 2024-07-26T12:00:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:27 smithi189 ceph-mon[36790]: pgmap v1884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:00:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1984.504987 secs 2024-07-26T12:00:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:27 smithi193 ceph-mon[37473]: pgmap v1884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:00:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1984.504987 secs 2024-07-26T12:00:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:27 smithi134 ceph-mon[31966]: pgmap v1884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:00:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:00:28.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:00:28.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:00:28.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:00:28.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:00:28.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:00:28.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:00:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:00:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:00:28.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:00:28.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:00:28.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:00:28.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:00:28.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:00:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:00:28.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:00:28.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:00:28.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:00:28.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:00:28.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:00:28.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:00:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:29 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:00:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:29 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:00:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:29 smithi193 ceph-mon[37473]: pgmap v1885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:00:29.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1970.077903 secs 2024-07-26T12:00:29.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:29 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:00:29.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:29 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:00:29.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:29 smithi189 ceph-mon[36790]: pgmap v1885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:00:29.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1970.077903 secs 2024-07-26T12:00:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:29 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:00:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:29 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:00:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:29 smithi134 ceph-mon[31966]: pgmap v1885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:00:29.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1970.077903 secs 2024-07-26T12:00:29.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:29.903+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:00:29.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:29.903+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:00:29.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:29.903+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:31 smithi193 ceph-mon[37473]: pgmap v1886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:00:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1989.505108 secs 2024-07-26T12:00:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:31 smithi134 ceph-mon[31966]: pgmap v1886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:00:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1989.505108 secs 2024-07-26T12:00:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:31 smithi189 ceph-mon[36790]: pgmap v1886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:00:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1989.505108 secs 2024-07-26T12:00:33.175 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:33.174+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:00:33.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:33.174+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006660 con 0x55a08e2d94d0 2024-07-26T12:00:33.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:33.174+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f2194006660 con 0x55a08e2d94d0 2024-07-26T12:00:33.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:33.174+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:00:33.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:33.175+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:00:33.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:33.175+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T12:00:33.177 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:33.175+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T12:00:33.177 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:33.175+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:00:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:33 smithi193 ceph-mon[37473]: pgmap v1887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:00:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:33 smithi134 ceph-mon[31966]: pgmap v1887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:00:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:33 smithi189 ceph-mon[36790]: pgmap v1887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:00:34.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:34.392+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:00:34.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:34.392+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:34.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:34.392+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:00:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1975.078040 secs 2024-07-26T12:00:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1975.078040 secs 2024-07-26T12:00:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1975.078040 secs 2024-07-26T12:00:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:35 smithi193 ceph-mon[37473]: pgmap v1888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:00:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:35 smithi134 ceph-mon[31966]: pgmap v1888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:00:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:35 smithi189 ceph-mon[36790]: pgmap v1888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:00:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1994.505260 secs 2024-07-26T12:00:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1994.505260 secs 2024-07-26T12:00:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1994.505260 secs 2024-07-26T12:00:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:37 smithi134 ceph-mon[31966]: pgmap v1889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:00:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:37 smithi193 ceph-mon[37473]: pgmap v1889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:00:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:37 smithi189 ceph-mon[36790]: pgmap v1889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:00:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:39 smithi193 ceph-mon[37473]: pgmap v1890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:00:39.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1980.078108 secs 2024-07-26T12:00:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:39 smithi134 ceph-mon[31966]: pgmap v1890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:00:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1980.078108 secs 2024-07-26T12:00:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:39 smithi189 ceph-mon[36790]: pgmap v1890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:00:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1980.078108 secs 2024-07-26T12:00:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:41 smithi193 ceph-mon[37473]: pgmap v1891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 1999.505347 secs 2024-07-26T12:00:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:41 smithi134 ceph-mon[31966]: pgmap v1891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 1999.505347 secs 2024-07-26T12:00:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:41 smithi189 ceph-mon[36790]: pgmap v1891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 1999.505347 secs 2024-07-26T12:00:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:43 smithi193 ceph-mon[37473]: pgmap v1892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:43 smithi134 ceph-mon[31966]: pgmap v1892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:43 smithi189 ceph-mon[36790]: pgmap v1892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1985.078258 secs 2024-07-26T12:00:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1985.078258 secs 2024-07-26T12:00:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1985.078258 secs 2024-07-26T12:00:44.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:44.915+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:00:44.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:44.915+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:44.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:44.915+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:00:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:45 smithi134 ceph-mon[31966]: pgmap v1893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:45 smithi189 ceph-mon[36790]: pgmap v1893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:45 smithi193 ceph-mon[37473]: pgmap v1893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2004.505398 secs 2024-07-26T12:00:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2004.505398 secs 2024-07-26T12:00:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2004.505398 secs 2024-07-26T12:00:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:47 smithi134 ceph-mon[31966]: pgmap v1894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:47 smithi189 ceph-mon[36790]: pgmap v1894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:47 smithi193 ceph-mon[37473]: pgmap v1894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:49.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:49.408+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:00:49.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:49.408+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:00:49.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:00:49.408+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:49 smithi134 ceph-mon[31966]: pgmap v1895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1990.078376 secs 2024-07-26T12:00:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:49 smithi189 ceph-mon[36790]: pgmap v1895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1990.078376 secs 2024-07-26T12:00:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:49 smithi193 ceph-mon[37473]: pgmap v1895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1990.078376 secs 2024-07-26T12:00:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:50 smithi134 ceph-mon[31966]: pgmap v1896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:50 smithi189 ceph-mon[36790]: pgmap v1896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:50 smithi193 ceph-mon[37473]: pgmap v1896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:00:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2009.505519 secs 2024-07-26T12:00:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2009.505519 secs 2024-07-26T12:00:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2009.505519 secs 2024-07-26T12:00:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:52 smithi134 ceph-mon[31966]: pgmap v1897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:52 smithi189 ceph-mon[36790]: pgmap v1897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:52 smithi193 ceph-mon[37473]: pgmap v1897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 1995.078492 secs 2024-07-26T12:00:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:54 smithi134 ceph-mon[31966]: pgmap v1898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 1995.078492 secs 2024-07-26T12:00:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:54 smithi189 ceph-mon[36790]: pgmap v1898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 1995.078492 secs 2024-07-26T12:00:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:54 smithi193 ceph-mon[37473]: pgmap v1898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:00:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:56 smithi134 ceph-mon[31966]: pgmap v1899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2014.505643 secs 2024-07-26T12:00:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:56 smithi189 ceph-mon[36790]: pgmap v1899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2014.505643 secs 2024-07-26T12:00:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:56 smithi193 ceph-mon[37473]: pgmap v1899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:00:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2014.505643 secs 2024-07-26T12:00:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:58 smithi134 ceph-mon[31966]: pgmap v1900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:58 smithi189 ceph-mon[36790]: pgmap v1900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:58 smithi193 ceph-mon[37473]: pgmap v1900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:00:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:00:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2000.078624 secs 2024-07-26T12:00:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:00:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2000.078624 secs 2024-07-26T12:00:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:00:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2000.078624 secs 2024-07-26T12:00:59.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:59.930+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:00:59.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:59.930+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:00:59.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:00:59.930+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:00 smithi134 ceph-mon[31966]: pgmap v1901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:00 smithi189 ceph-mon[36790]: pgmap v1901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:00 smithi193 ceph-mon[37473]: pgmap v1901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2019.505768 secs 2024-07-26T12:01:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2019.505768 secs 2024-07-26T12:01:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2019.505768 secs 2024-07-26T12:01:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:02 smithi189 ceph-mon[36790]: pgmap v1902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:02 smithi193 ceph-mon[37473]: pgmap v1902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:02 smithi134 ceph-mon[31966]: pgmap v1902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:04.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:01:04.423+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:04.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:01:04.424+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:04.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:01:04.424+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2005.078761 secs 2024-07-26T12:01:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:04 smithi189 ceph-mon[36790]: pgmap v1903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2005.078761 secs 2024-07-26T12:01:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:04 smithi193 ceph-mon[37473]: pgmap v1903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2005.078761 secs 2024-07-26T12:01:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:04 smithi134 ceph-mon[31966]: pgmap v1903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:06 smithi189 ceph-mon[36790]: pgmap v1904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2024.505887 secs 2024-07-26T12:01:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:06 smithi193 ceph-mon[37473]: pgmap v1904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2024.505887 secs 2024-07-26T12:01:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:06 smithi134 ceph-mon[31966]: pgmap v1904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2024.505887 secs 2024-07-26T12:01:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:08 smithi134 ceph-mon[31966]: pgmap v1905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:08 smithi189 ceph-mon[36790]: pgmap v1905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:08 smithi193 ceph-mon[37473]: pgmap v1905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2010.078859 secs 2024-07-26T12:01:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2010.078859 secs 2024-07-26T12:01:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2010.078859 secs 2024-07-26T12:01:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:10 smithi193 ceph-mon[37473]: pgmap v1906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:10 smithi134 ceph-mon[31966]: pgmap v1906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:10 smithi189 ceph-mon[36790]: pgmap v1906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2029.506006 secs 2024-07-26T12:01:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2029.506006 secs 2024-07-26T12:01:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2029.506006 secs 2024-07-26T12:01:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:12 smithi193 ceph-mon[37473]: pgmap v1907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:12 smithi134 ceph-mon[31966]: pgmap v1907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:12 smithi189 ceph-mon[36790]: pgmap v1907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2015.078997 secs 2024-07-26T12:01:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:14 smithi193 ceph-mon[37473]: pgmap v1908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2015.078997 secs 2024-07-26T12:01:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:14 smithi134 ceph-mon[31966]: pgmap v1908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2015.078997 secs 2024-07-26T12:01:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:14 smithi189 ceph-mon[36790]: pgmap v1908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:14.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:01:14.946+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:14.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:01:14.946+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:14.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:01:14.946+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:16 smithi193 ceph-mon[37473]: pgmap v1909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2034.506126 secs 2024-07-26T12:01:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:16 smithi134 ceph-mon[31966]: pgmap v1909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2034.506126 secs 2024-07-26T12:01:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:16 smithi189 ceph-mon[36790]: pgmap v1909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2034.506126 secs 2024-07-26T12:01:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:18 smithi193 ceph-mon[37473]: pgmap v1910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:18 smithi134 ceph-mon[31966]: pgmap v1910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:18 smithi189 ceph-mon[36790]: pgmap v1910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:19.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:01:19.439+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:19.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:01:19.439+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:19.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:01:19.439+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2020.079125 secs 2024-07-26T12:01:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2020.079125 secs 2024-07-26T12:01:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2020.079125 secs 2024-07-26T12:01:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:20 smithi193 ceph-mon[37473]: pgmap v1911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:20 smithi134 ceph-mon[31966]: pgmap v1911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:20 smithi189 ceph-mon[36790]: pgmap v1911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:01:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2039.506249 secs 2024-07-26T12:01:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2039.506249 secs 2024-07-26T12:01:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2039.506249 secs 2024-07-26T12:01:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:22 smithi193 ceph-mon[37473]: pgmap v1912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:22 smithi134 ceph-mon[31966]: pgmap v1912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:22 smithi189 ceph-mon[36790]: pgmap v1912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2025.079242 secs 2024-07-26T12:01:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:24 smithi193 ceph-mon[37473]: pgmap v1913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2025.079242 secs 2024-07-26T12:01:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:24 smithi134 ceph-mon[31966]: pgmap v1913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2025.079242 secs 2024-07-26T12:01:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:24 smithi189 ceph-mon[36790]: pgmap v1913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:26 smithi134 ceph-mon[31966]: pgmap v1914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2044.506379 secs 2024-07-26T12:01:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:26 smithi189 ceph-mon[36790]: pgmap v1914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2044.506379 secs 2024-07-26T12:01:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:26 smithi193 ceph-mon[37473]: pgmap v1914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2044.506379 secs 2024-07-26T12:01:28.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:28 smithi134 ceph-mon[31966]: pgmap v1915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:28.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:01:28.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:28.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:28.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:28.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:28.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:28.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:28.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:28.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:28.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:28 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:01:28.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:28 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:01:28.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:28 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:01:28.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:28 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:01:28.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:01:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:28 smithi189 ceph-mon[36790]: pgmap v1915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:01:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:28 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:01:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:28 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:01:28.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:28 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:01:28.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:28 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:01:28.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:01:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:28 smithi193 ceph-mon[37473]: pgmap v1915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:01:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:01:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:29.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:29.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:29.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:29.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:29.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:01:29.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:28 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:01:29.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:28 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:01:29.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:28 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:01:29.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:28 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:01:29.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:01:29.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2030.079369 secs 2024-07-26T12:01:29.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:01:29.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:29 smithi189 ceph-mon[36790]: pgmap v1916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:01:29.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:29 smithi189 ceph-mon[36790]: pgmap v1917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:01:29.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:01:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2030.079369 secs 2024-07-26T12:01:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:01:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:29 smithi134 ceph-mon[31966]: pgmap v1916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:01:29.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:29 smithi134 ceph-mon[31966]: pgmap v1917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:01:29.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:01:29.961 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:01:29.959+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:29.961 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:01:29.959+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:29.961 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:01:29.959+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2030.079369 secs 2024-07-26T12:01:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:01:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:29 smithi193 ceph-mon[37473]: pgmap v1916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:01:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:29 smithi193 ceph-mon[37473]: pgmap v1917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:01:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:01:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:31 smithi134 ceph-mon[31966]: pgmap v1918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:01:31.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2049.506503 secs 2024-07-26T12:01:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:31 smithi189 ceph-mon[36790]: pgmap v1918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:01:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2049.506503 secs 2024-07-26T12:01:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:31 smithi193 ceph-mon[37473]: pgmap v1918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:01:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2049.506503 secs 2024-07-26T12:01:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:33 smithi134 ceph-mon[31966]: pgmap v1919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:01:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:33 smithi189 ceph-mon[36790]: pgmap v1919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:01:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:33 smithi193 ceph-mon[37473]: pgmap v1919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:01:34.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:01:34.454+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:34.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:01:34.454+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:34.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:01:34.454+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2035.079441 secs 2024-07-26T12:01:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2035.079441 secs 2024-07-26T12:01:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2035.079441 secs 2024-07-26T12:01:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:35 smithi134 ceph-mon[31966]: pgmap v1920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:01:35.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:35 smithi189 ceph-mon[36790]: pgmap v1920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:01:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:35 smithi193 ceph-mon[37473]: pgmap v1920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:01:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2054.506632 secs 2024-07-26T12:01:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2054.506632 secs 2024-07-26T12:01:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2054.506632 secs 2024-07-26T12:01:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:37 smithi134 ceph-mon[31966]: pgmap v1921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:01:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:37 smithi189 ceph-mon[36790]: pgmap v1921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:01:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:37 smithi193 ceph-mon[37473]: pgmap v1921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:01:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2040.079561 secs 2024-07-26T12:01:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:39 smithi134 ceph-mon[31966]: pgmap v1922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:01:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2040.079561 secs 2024-07-26T12:01:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:39 smithi189 ceph-mon[36790]: pgmap v1922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:01:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2040.079561 secs 2024-07-26T12:01:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:39 smithi193 ceph-mon[37473]: pgmap v1922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:01:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:41 smithi134 ceph-mon[31966]: pgmap v1923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2059.506764 secs 2024-07-26T12:01:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:41 smithi189 ceph-mon[36790]: pgmap v1923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2059.506764 secs 2024-07-26T12:01:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:41 smithi193 ceph-mon[37473]: pgmap v1923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2059.506764 secs 2024-07-26T12:01:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:43 smithi134 ceph-mon[31966]: pgmap v1924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:43 smithi189 ceph-mon[36790]: pgmap v1924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:43 smithi193 ceph-mon[37473]: pgmap v1924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:01:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2045.079678 secs 2024-07-26T12:01:44.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2045.079678 secs 2024-07-26T12:01:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2045.079678 secs 2024-07-26T12:01:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:01:44.975+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:01:44.975+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:44.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:01:44.975+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:45 smithi134 ceph-mon[31966]: pgmap v1925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:01:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:45 smithi189 ceph-mon[36790]: pgmap v1925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:01:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:45 smithi193 ceph-mon[37473]: pgmap v1925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:01:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2064.506933 secs 2024-07-26T12:01:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2064.506933 secs 2024-07-26T12:01:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2064.506933 secs 2024-07-26T12:01:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:47 smithi134 ceph-mon[31966]: pgmap v1926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:47 smithi189 ceph-mon[36790]: pgmap v1926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:47 smithi193 ceph-mon[37473]: pgmap v1926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:01:49.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:01:49.470+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:49.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:01:49.470+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:49.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:01:49.470+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:01:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2050.079827 secs 2024-07-26T12:01:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:49 smithi134 ceph-mon[31966]: pgmap v1927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:01:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2050.079827 secs 2024-07-26T12:01:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:49 smithi189 ceph-mon[36790]: pgmap v1927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:01:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2050.079827 secs 2024-07-26T12:01:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:49 smithi193 ceph-mon[37473]: pgmap v1927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:01:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:51 smithi134 ceph-mon[31966]: pgmap v1928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:01:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2069.507008 secs 2024-07-26T12:01:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:51 smithi189 ceph-mon[36790]: pgmap v1928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:01:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2069.507008 secs 2024-07-26T12:01:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:51 smithi193 ceph-mon[37473]: pgmap v1928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:01:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2069.507008 secs 2024-07-26T12:01:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:53 smithi134 ceph-mon[31966]: pgmap v1929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:01:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:53 smithi189 ceph-mon[36790]: pgmap v1929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:01:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:53 smithi193 ceph-mon[37473]: pgmap v1929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:01:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2055.079923 secs 2024-07-26T12:01:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2055.079923 secs 2024-07-26T12:01:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2055.079923 secs 2024-07-26T12:01:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:55 smithi134 ceph-mon[31966]: pgmap v1930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:01:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:55 smithi189 ceph-mon[36790]: pgmap v1930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:01:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:55 smithi193 ceph-mon[37473]: pgmap v1930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:01:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2074.507130 secs 2024-07-26T12:01:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2074.507130 secs 2024-07-26T12:01:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2074.507130 secs 2024-07-26T12:01:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:57 smithi189 ceph-mon[36790]: pgmap v1931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:01:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:57 smithi193 ceph-mon[37473]: pgmap v1931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:01:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:57 smithi134 ceph-mon[31966]: pgmap v1931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:01:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2060.080045 secs 2024-07-26T12:01:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2060.080045 secs 2024-07-26T12:01:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2060.080045 secs 2024-07-26T12:01:59.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:01:59.991+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:59.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:01:59.991+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:01:59.993 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:01:59.991+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:01:59 smithi193 ceph-mon[37473]: pgmap v1932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:02:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:01:59 smithi134 ceph-mon[31966]: pgmap v1932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:02:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:01:59 smithi189 ceph-mon[36790]: pgmap v1932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:02:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:01 smithi193 ceph-mon[37473]: pgmap v1933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2079.507258 secs 2024-07-26T12:02:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:01 smithi134 ceph-mon[31966]: pgmap v1933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2079.507258 secs 2024-07-26T12:02:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:01 smithi189 ceph-mon[36790]: pgmap v1933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2079.507258 secs 2024-07-26T12:02:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:03 smithi193 ceph-mon[37473]: pgmap v1934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2065.080162 secs 2024-07-26T12:02:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:03 smithi134 ceph-mon[31966]: pgmap v1934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2065.080162 secs 2024-07-26T12:02:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:03 smithi189 ceph-mon[36790]: pgmap v1934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2065.080162 secs 2024-07-26T12:02:04.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:02:04.485+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:02:04.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:02:04.485+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:04.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:02:04.485+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:02:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:05 smithi193 ceph-mon[37473]: pgmap v1935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:02:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:05 smithi134 ceph-mon[31966]: pgmap v1935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:02:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:05 smithi189 ceph-mon[36790]: pgmap v1935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:02:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2084.507404 secs 2024-07-26T12:02:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2084.507404 secs 2024-07-26T12:02:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2084.507404 secs 2024-07-26T12:02:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:07 smithi193 ceph-mon[37473]: pgmap v1936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:07 smithi134 ceph-mon[31966]: pgmap v1936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:07 smithi189 ceph-mon[36790]: pgmap v1936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2070.080282 secs 2024-07-26T12:02:09.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2070.080282 secs 2024-07-26T12:02:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2070.080282 secs 2024-07-26T12:02:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:09 smithi193 ceph-mon[37473]: pgmap v1937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:09 smithi134 ceph-mon[31966]: pgmap v1937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:09 smithi189 ceph-mon[36790]: pgmap v1937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:12 smithi134 ceph-mon[31966]: pgmap v1938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2089.507491 secs 2024-07-26T12:02:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:12 smithi189 ceph-mon[36790]: pgmap v1938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2089.507491 secs 2024-07-26T12:02:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:12 smithi193 ceph-mon[37473]: pgmap v1938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2089.507491 secs 2024-07-26T12:02:14.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:14 smithi134 ceph-mon[31966]: pgmap v1939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:14.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2075.080373 secs 2024-07-26T12:02:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:14 smithi189 ceph-mon[36790]: pgmap v1939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2075.080373 secs 2024-07-26T12:02:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:14 smithi193 ceph-mon[37473]: pgmap v1939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2075.080373 secs 2024-07-26T12:02:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:02:15.007+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:02:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:02:15.007+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:02:15.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:02:15.007+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:16 smithi134 ceph-mon[31966]: pgmap v1940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:02:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:16 smithi189 ceph-mon[36790]: pgmap v1940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:02:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:16 smithi193 ceph-mon[37473]: pgmap v1940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:02:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2094.507580 secs 2024-07-26T12:02:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2094.507580 secs 2024-07-26T12:02:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2094.507580 secs 2024-07-26T12:02:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:18 smithi134 ceph-mon[31966]: pgmap v1941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:18 smithi189 ceph-mon[36790]: pgmap v1941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:18 smithi193 ceph-mon[37473]: pgmap v1941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2080.080441 secs 2024-07-26T12:02:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2080.080441 secs 2024-07-26T12:02:19.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:02:19.499+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:02:19.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:02:19.499+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:19.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:02:19.499+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:02:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2080.080441 secs 2024-07-26T12:02:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:20 smithi134 ceph-mon[31966]: pgmap v1942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:20 smithi189 ceph-mon[36790]: pgmap v1942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:20 smithi193 ceph-mon[37473]: pgmap v1942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:22 smithi134 ceph-mon[31966]: pgmap v1943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2099.507717 secs 2024-07-26T12:02:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:22 smithi189 ceph-mon[36790]: pgmap v1943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2099.507717 secs 2024-07-26T12:02:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:22 smithi193 ceph-mon[37473]: pgmap v1943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2099.507717 secs 2024-07-26T12:02:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:24 smithi134 ceph-mon[31966]: pgmap v1944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2085.080517 secs 2024-07-26T12:02:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:24 smithi189 ceph-mon[36790]: pgmap v1944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2085.080517 secs 2024-07-26T12:02:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:24 smithi193 ceph-mon[37473]: pgmap v1944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2085.080517 secs 2024-07-26T12:02:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:26 smithi189 ceph-mon[36790]: pgmap v1945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:02:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:26 smithi193 ceph-mon[37473]: pgmap v1945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:02:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:26 smithi134 ceph-mon[31966]: pgmap v1945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:02:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2104.507779 secs 2024-07-26T12:02:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2104.507779 secs 2024-07-26T12:02:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2104.507779 secs 2024-07-26T12:02:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:28 smithi189 ceph-mon[36790]: pgmap v1946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:28 smithi193 ceph-mon[37473]: pgmap v1946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:28 smithi134 ceph-mon[31966]: pgmap v1946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2090.080665 secs 2024-07-26T12:02:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:02:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2090.080665 secs 2024-07-26T12:02:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:02:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2090.080665 secs 2024-07-26T12:02:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:02:30.023 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:02:30.022+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:02:30.024 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:02:30.022+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:30.024 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:02:30.022+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:02:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:30 smithi193 ceph-mon[37473]: pgmap v1947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:02:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:02:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:02:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:30 smithi134 ceph-mon[31966]: pgmap v1947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:02:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:02:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:02:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:30 smithi189 ceph-mon[36790]: pgmap v1947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:02:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:02:30.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:02:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:31 smithi193 ceph-mon[37473]: pgmap v1948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:02:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:31 smithi193 ceph-mon[37473]: pgmap v1949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:02:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2109.507907 secs 2024-07-26T12:02:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:31 smithi134 ceph-mon[31966]: pgmap v1948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:02:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:31 smithi134 ceph-mon[31966]: pgmap v1949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:02:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2109.507907 secs 2024-07-26T12:02:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:31 smithi189 ceph-mon[36790]: pgmap v1948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:02:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:31 smithi189 ceph-mon[36790]: pgmap v1949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:02:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2109.507907 secs 2024-07-26T12:02:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:33 smithi193 ceph-mon[37473]: pgmap v1950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:02:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:33 smithi134 ceph-mon[31966]: pgmap v1950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:02:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:33 smithi189 ceph-mon[36790]: pgmap v1950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:02:34.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:02:34.502+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:02:34.504 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:02:34.502+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:34.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:02:34.502+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:02:34.505 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2095.080803 secs 2024-07-26T12:02:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2095.080803 secs 2024-07-26T12:02:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2095.080803 secs 2024-07-26T12:02:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:35 smithi193 ceph-mon[37473]: pgmap v1951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:35 smithi134 ceph-mon[31966]: pgmap v1951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:35 smithi189 ceph-mon[36790]: pgmap v1951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2114.508032 secs 2024-07-26T12:02:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2114.508032 secs 2024-07-26T12:02:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2114.508032 secs 2024-07-26T12:02:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:37 smithi193 ceph-mon[37473]: pgmap v1952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:37 smithi134 ceph-mon[31966]: pgmap v1952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:37 smithi189 ceph-mon[36790]: pgmap v1952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:39 smithi193 ceph-mon[37473]: pgmap v1953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2100.080925 secs 2024-07-26T12:02:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:39 smithi134 ceph-mon[31966]: pgmap v1953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2100.080925 secs 2024-07-26T12:02:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:39 smithi189 ceph-mon[36790]: pgmap v1953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:02:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2100.080925 secs 2024-07-26T12:02:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:41 smithi193 ceph-mon[37473]: pgmap v1954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:02:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2119.508162 secs 2024-07-26T12:02:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:41 smithi134 ceph-mon[31966]: pgmap v1954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:02:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2119.508162 secs 2024-07-26T12:02:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:41 smithi189 ceph-mon[36790]: pgmap v1954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:02:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2119.508162 secs 2024-07-26T12:02:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:43 smithi193 ceph-mon[37473]: pgmap v1955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:43 smithi134 ceph-mon[31966]: pgmap v1955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:43 smithi189 ceph-mon[36790]: pgmap v1955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2105.081050 secs 2024-07-26T12:02:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2105.081050 secs 2024-07-26T12:02:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2105.081050 secs 2024-07-26T12:02:45.026 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:02:45.024+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:02:45.026 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:02:45.024+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:02:45.026 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:02:45.024+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:45 smithi134 ceph-mon[31966]: pgmap v1956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:45 smithi189 ceph-mon[36790]: pgmap v1956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:45 smithi193 ceph-mon[37473]: pgmap v1956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2124.508335 secs 2024-07-26T12:02:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2124.508335 secs 2024-07-26T12:02:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2124.508335 secs 2024-07-26T12:02:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:47 smithi134 ceph-mon[31966]: pgmap v1957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:47 smithi189 ceph-mon[36790]: pgmap v1957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:47 smithi193 ceph-mon[37473]: pgmap v1957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:49.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:02:49.503+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:02:49.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:02:49.503+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:02:49.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:02:49.503+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:02:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:49 smithi134 ceph-mon[31966]: pgmap v1958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2110.081155 secs 2024-07-26T12:02:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:49 smithi189 ceph-mon[36790]: pgmap v1958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2110.081155 secs 2024-07-26T12:02:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:49 smithi193 ceph-mon[37473]: pgmap v1958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2110.081155 secs 2024-07-26T12:02:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:51 smithi134 ceph-mon[31966]: pgmap v1959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:02:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2129.508444 secs 2024-07-26T12:02:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:51 smithi189 ceph-mon[36790]: pgmap v1959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:02:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2129.508444 secs 2024-07-26T12:02:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:51 smithi193 ceph-mon[37473]: pgmap v1959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:02:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2129.508444 secs 2024-07-26T12:02:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:53 smithi134 ceph-mon[31966]: pgmap v1960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:53 smithi189 ceph-mon[36790]: pgmap v1960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:53 smithi193 ceph-mon[37473]: pgmap v1960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2115.081276 secs 2024-07-26T12:02:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2115.081276 secs 2024-07-26T12:02:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2115.081276 secs 2024-07-26T12:02:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:55 smithi134 ceph-mon[31966]: pgmap v1961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:55 smithi189 ceph-mon[36790]: pgmap v1961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:55 smithi193 ceph-mon[37473]: pgmap v1961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2134.508560 secs 2024-07-26T12:02:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2134.508560 secs 2024-07-26T12:02:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2134.508560 secs 2024-07-26T12:02:57.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:57 smithi134 ceph-mon[31966]: pgmap v1962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:57 smithi189 ceph-mon[36790]: pgmap v1962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:57 smithi193 ceph-mon[37473]: pgmap v1962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:02:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:58 smithi134 ceph-mon[31966]: pgmap v1963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:58 smithi189 ceph-mon[36790]: pgmap v1963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:58 smithi193 ceph-mon[37473]: pgmap v1963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:02:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:02:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2120.081411 secs 2024-07-26T12:02:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:02:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2120.081411 secs 2024-07-26T12:02:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:02:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2120.081411 secs 2024-07-26T12:03:00.042 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:03:00.040+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:03:00.042 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:03:00.040+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:03:00.042 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:03:00.040+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:00 smithi134 ceph-mon[31966]: pgmap v1964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:03:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:00 smithi189 ceph-mon[36790]: pgmap v1964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:03:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:00 smithi193 ceph-mon[37473]: pgmap v1964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:03:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2139.508663 secs 2024-07-26T12:03:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2139.508663 secs 2024-07-26T12:03:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2139.508663 secs 2024-07-26T12:03:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:02 smithi134 ceph-mon[31966]: pgmap v1965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:02 smithi189 ceph-mon[36790]: pgmap v1965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:02 smithi193 ceph-mon[37473]: pgmap v1965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2125.081531 secs 2024-07-26T12:03:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2125.081531 secs 2024-07-26T12:03:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2125.081531 secs 2024-07-26T12:03:04.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:03:04.520+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:03:04.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:03:04.520+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:04.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:03:04.520+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:03:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:04 smithi134 ceph-mon[31966]: pgmap v1966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:04 smithi193 ceph-mon[37473]: pgmap v1966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:04 smithi189 ceph-mon[36790]: pgmap v1966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:06 smithi134 ceph-mon[31966]: pgmap v1967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:07.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2144.508777 secs 2024-07-26T12:03:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:06 smithi189 ceph-mon[36790]: pgmap v1967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2144.508777 secs 2024-07-26T12:03:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:06 smithi193 ceph-mon[37473]: pgmap v1967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2144.508777 secs 2024-07-26T12:03:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:08 smithi134 ceph-mon[31966]: pgmap v1968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2130.081673 secs 2024-07-26T12:03:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:08 smithi189 ceph-mon[36790]: pgmap v1968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2130.081673 secs 2024-07-26T12:03:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:08 smithi193 ceph-mon[37473]: pgmap v1968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2130.081673 secs 2024-07-26T12:03:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:10 smithi134 ceph-mon[31966]: pgmap v1969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:03:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:10 smithi189 ceph-mon[36790]: pgmap v1969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:03:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:10 smithi193 ceph-mon[37473]: pgmap v1969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:03:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2149.508866 secs 2024-07-26T12:03:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2149.508866 secs 2024-07-26T12:03:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2149.508866 secs 2024-07-26T12:03:13.161 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:12 smithi134 ceph-mon[31966]: pgmap v1970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:12 smithi189 ceph-mon[36790]: pgmap v1970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:12 smithi193 ceph-mon[37473]: pgmap v1970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2135.081802 secs 2024-07-26T12:03:14.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2135.081802 secs 2024-07-26T12:03:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2135.081802 secs 2024-07-26T12:03:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:03:15.055+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:03:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:03:15.055+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:03:15.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:03:15.055+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:14 smithi134 ceph-mon[31966]: pgmap v1971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:14 smithi189 ceph-mon[36790]: pgmap v1971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:14 smithi193 ceph-mon[37473]: pgmap v1971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:17.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:16 smithi134 ceph-mon[31966]: pgmap v1972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:17.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2154.508985 secs 2024-07-26T12:03:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:16 smithi189 ceph-mon[36790]: pgmap v1972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2154.508985 secs 2024-07-26T12:03:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:16 smithi193 ceph-mon[37473]: pgmap v1972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2154.508985 secs 2024-07-26T12:03:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:18 smithi134 ceph-mon[31966]: pgmap v1973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2140.081967 secs 2024-07-26T12:03:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:18 smithi189 ceph-mon[36790]: pgmap v1973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2140.081967 secs 2024-07-26T12:03:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:18 smithi193 ceph-mon[37473]: pgmap v1973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2140.081967 secs 2024-07-26T12:03:19.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:03:19.534+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:03:19.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:03:19.534+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:03:19.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:03:19.534+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:20 smithi189 ceph-mon[36790]: pgmap v1974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:03:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:20 smithi193 ceph-mon[37473]: pgmap v1974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:03:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:20 smithi134 ceph-mon[31966]: pgmap v1974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:03:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2159.509108 secs 2024-07-26T12:03:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2159.509108 secs 2024-07-26T12:03:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2159.509108 secs 2024-07-26T12:03:22.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:03:22.704+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:03:22.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:03:22.704+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:03:22.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:03:22.705+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T12:03:22.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:03:22.705+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T12:03:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:22 smithi134 ceph-mon[31966]: pgmap v1975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:22 smithi189 ceph-mon[36790]: pgmap v1975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:22 smithi193 ceph-mon[37473]: pgmap v1975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2145.082060 secs 2024-07-26T12:03:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2145.082060 secs 2024-07-26T12:03:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2145.082060 secs 2024-07-26T12:03:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:24 smithi193 ceph-mon[37473]: pgmap v1976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:24 smithi134 ceph-mon[31966]: pgmap v1976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:24 smithi189 ceph-mon[36790]: pgmap v1976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:26 smithi193 ceph-mon[37473]: pgmap v1977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2164.509229 secs 2024-07-26T12:03:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:26 smithi134 ceph-mon[31966]: pgmap v1977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2164.509229 secs 2024-07-26T12:03:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:26 smithi189 ceph-mon[36790]: pgmap v1977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2164.509229 secs 2024-07-26T12:03:29.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:28 smithi134 ceph-mon[31966]: pgmap v1978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:29.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2150.082159 secs 2024-07-26T12:03:29.245 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:28 smithi189 ceph-mon[36790]: pgmap v1978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:29.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2150.082159 secs 2024-07-26T12:03:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:28 smithi193 ceph-mon[37473]: pgmap v1978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2150.082159 secs 2024-07-26T12:03:30.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:03:30.067+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:03:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:03:30.067+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:03:30.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:03:30.067+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:30 smithi193 ceph-mon[37473]: pgmap v1979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:03:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:03:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:03:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:03:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:30 smithi134 ceph-mon[31966]: pgmap v1979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:03:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:03:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:03:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:03:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:30 smithi189 ceph-mon[36790]: pgmap v1979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:03:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:03:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:03:31.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:03:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2169.509297 secs 2024-07-26T12:03:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:03:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:03:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:03:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2169.509297 secs 2024-07-26T12:03:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:03:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:03:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:03:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2169.509297 secs 2024-07-26T12:03:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:03:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:03:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:03:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:03:33.206+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:03:33.207 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:03:33.206+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T12:03:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:03:33.206+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:03:33.208 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:03:33.206+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T12:03:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:32 smithi193 ceph-mon[37473]: pgmap v1980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:03:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:32 smithi193 ceph-mon[37473]: pgmap v1981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:03:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:32 smithi134 ceph-mon[31966]: pgmap v1980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:03:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:32 smithi134 ceph-mon[31966]: pgmap v1981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:03:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:32 smithi189 ceph-mon[36790]: pgmap v1980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:03:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:32 smithi189 ceph-mon[36790]: pgmap v1981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:03:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2155.082258 secs 2024-07-26T12:03:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2155.082258 secs 2024-07-26T12:03:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2155.082258 secs 2024-07-26T12:03:34.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:03:34.550+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:03:34.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:03:34.550+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:34.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:03:34.550+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:03:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:34 smithi193 ceph-mon[37473]: pgmap v1982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:03:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:34 smithi134 ceph-mon[31966]: pgmap v1982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:03:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:34 smithi189 ceph-mon[36790]: pgmap v1982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:03:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:37 smithi193 ceph-mon[37473]: pgmap v1983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:03:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2174.509422 secs 2024-07-26T12:03:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:37 smithi134 ceph-mon[31966]: pgmap v1983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:03:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2174.509422 secs 2024-07-26T12:03:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:37 smithi189 ceph-mon[36790]: pgmap v1983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:03:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2174.509422 secs 2024-07-26T12:03:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:39 smithi193 ceph-mon[37473]: pgmap v1984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:03:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2160.082381 secs 2024-07-26T12:03:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:39 smithi134 ceph-mon[31966]: pgmap v1984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:03:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2160.082381 secs 2024-07-26T12:03:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:39 smithi189 ceph-mon[36790]: pgmap v1984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:03:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2160.082381 secs 2024-07-26T12:03:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:41 smithi193 ceph-mon[37473]: pgmap v1985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:03:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:41 smithi134 ceph-mon[31966]: pgmap v1985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:03:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:41 smithi189 ceph-mon[36790]: pgmap v1985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:03:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2179.509548 secs 2024-07-26T12:03:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2179.509548 secs 2024-07-26T12:03:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2179.509548 secs 2024-07-26T12:03:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:43 smithi134 ceph-mon[31966]: pgmap v1986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:03:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:43 smithi189 ceph-mon[36790]: pgmap v1986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:03:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:43 smithi193 ceph-mon[37473]: pgmap v1986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:03:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2165.082486 secs 2024-07-26T12:03:44.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2165.082486 secs 2024-07-26T12:03:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2165.082486 secs 2024-07-26T12:03:45.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:03:45.082+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:03:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:03:45.082+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:03:45.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:03:45.082+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:45 smithi134 ceph-mon[31966]: pgmap v1987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:03:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:45 smithi189 ceph-mon[36790]: pgmap v1987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:03:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:45 smithi193 ceph-mon[37473]: pgmap v1987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:03:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:47 smithi134 ceph-mon[31966]: pgmap v1988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2184.509669 secs 2024-07-26T12:03:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:47 smithi189 ceph-mon[36790]: pgmap v1988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2184.509669 secs 2024-07-26T12:03:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:47 smithi193 ceph-mon[37473]: pgmap v1988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2184.509669 secs 2024-07-26T12:03:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:49 smithi134 ceph-mon[31966]: pgmap v1989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2170.082607 secs 2024-07-26T12:03:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:49 smithi189 ceph-mon[36790]: pgmap v1989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2170.082607 secs 2024-07-26T12:03:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:49 smithi193 ceph-mon[37473]: pgmap v1989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2170.082607 secs 2024-07-26T12:03:49.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:03:49.556+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:03:49.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:03:49.556+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:03:49.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:03:49.556+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:03:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:51 smithi134 ceph-mon[31966]: pgmap v1990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:03:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:51 smithi189 ceph-mon[36790]: pgmap v1990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:03:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:51 smithi193 ceph-mon[37473]: pgmap v1990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:03:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2189.509817 secs 2024-07-26T12:03:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2189.509817 secs 2024-07-26T12:03:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2189.509817 secs 2024-07-26T12:03:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:53 smithi134 ceph-mon[31966]: pgmap v1991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:53 smithi189 ceph-mon[36790]: pgmap v1991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:53 smithi193 ceph-mon[37473]: pgmap v1991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2175.082706 secs 2024-07-26T12:03:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2175.082706 secs 2024-07-26T12:03:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2175.082706 secs 2024-07-26T12:03:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:55 smithi134 ceph-mon[31966]: pgmap v1992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:55 smithi189 ceph-mon[36790]: pgmap v1992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:55 smithi193 ceph-mon[37473]: pgmap v1992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:57 smithi134 ceph-mon[31966]: pgmap v1993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2194.509933 secs 2024-07-26T12:03:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:57 smithi189 ceph-mon[36790]: pgmap v1993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2194.509933 secs 2024-07-26T12:03:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:57 smithi193 ceph-mon[37473]: pgmap v1993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:03:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2194.509933 secs 2024-07-26T12:03:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:59 smithi134 ceph-mon[31966]: pgmap v1994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:03:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2180.082824 secs 2024-07-26T12:03:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:59 smithi189 ceph-mon[36790]: pgmap v1994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:03:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2180.082824 secs 2024-07-26T12:03:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:59 smithi193 ceph-mon[37473]: pgmap v1994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:03:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:03:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2180.082824 secs 2024-07-26T12:04:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:04:00.098+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:04:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:04:00.098+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:04:00.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:04:00.098+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:01 smithi134 ceph-mon[31966]: pgmap v1995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:04:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:01 smithi189 ceph-mon[36790]: pgmap v1995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:04:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:01 smithi193 ceph-mon[37473]: pgmap v1995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:04:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2199.510061 secs 2024-07-26T12:04:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2199.510061 secs 2024-07-26T12:04:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2199.510061 secs 2024-07-26T12:04:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:03 smithi134 ceph-mon[31966]: pgmap v1996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:03 smithi189 ceph-mon[36790]: pgmap v1996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:03 smithi193 ceph-mon[37473]: pgmap v1996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2185.082975 secs 2024-07-26T12:04:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2185.082975 secs 2024-07-26T12:04:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2185.082975 secs 2024-07-26T12:04:04.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:04:04.572+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:04:04.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:04:04.572+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:04:04.574 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:04:04.572+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:05 smithi134 ceph-mon[31966]: pgmap v1997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:04:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:05 smithi189 ceph-mon[36790]: pgmap v1997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:04:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:05 smithi193 ceph-mon[37473]: pgmap v1997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:04:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:07 smithi134 ceph-mon[31966]: pgmap v1998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2204.510176 secs 2024-07-26T12:04:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:07 smithi189 ceph-mon[36790]: pgmap v1998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2204.510176 secs 2024-07-26T12:04:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:07 smithi193 ceph-mon[37473]: pgmap v1998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2204.510176 secs 2024-07-26T12:04:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:09 smithi134 ceph-mon[31966]: pgmap v1999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:04:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2190.083039 secs 2024-07-26T12:04:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:09 smithi189 ceph-mon[36790]: pgmap v1999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:04:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2190.083039 secs 2024-07-26T12:04:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:09 smithi193 ceph-mon[37473]: pgmap v1999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:04:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2190.083039 secs 2024-07-26T12:04:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:11 smithi134 ceph-mon[31966]: pgmap v2000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:04:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:11 smithi189 ceph-mon[36790]: pgmap v2000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:04:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:11 smithi193 ceph-mon[37473]: pgmap v2000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:04:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2209.510345 secs 2024-07-26T12:04:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2209.510345 secs 2024-07-26T12:04:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2209.510345 secs 2024-07-26T12:04:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:13 smithi189 ceph-mon[36790]: pgmap v2001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:13 smithi193 ceph-mon[37473]: pgmap v2001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:13 smithi134 ceph-mon[31966]: pgmap v2001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2195.083178 secs 2024-07-26T12:04:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2195.083178 secs 2024-07-26T12:04:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2195.083178 secs 2024-07-26T12:04:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:04:15.103+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:04:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:04:15.103+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:04:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:04:15.103+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:15 smithi134 ceph-mon[31966]: pgmap v2002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:04:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:15 smithi193 ceph-mon[37473]: pgmap v2002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:04:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:15 smithi189 ceph-mon[36790]: pgmap v2002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:04:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:17 smithi193 ceph-mon[37473]: pgmap v2003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2214.510488 secs 2024-07-26T12:04:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:17 smithi134 ceph-mon[31966]: pgmap v2003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2214.510488 secs 2024-07-26T12:04:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:17 smithi189 ceph-mon[36790]: pgmap v2003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2214.510488 secs 2024-07-26T12:04:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:19 smithi193 ceph-mon[37473]: pgmap v2004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:04:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2200.083300 secs 2024-07-26T12:04:19.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:04:19.587+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:04:19.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:04:19.587+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:04:19.590 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:04:19.587+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:19.590 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:19 smithi134 ceph-mon[31966]: pgmap v2004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:04:19.590 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2200.083300 secs 2024-07-26T12:04:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:19 smithi189 ceph-mon[36790]: pgmap v2004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:04:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2200.083300 secs 2024-07-26T12:04:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:21 smithi193 ceph-mon[37473]: pgmap v2005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:04:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:21 smithi134 ceph-mon[31966]: pgmap v2005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:04:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:21 smithi189 ceph-mon[36790]: pgmap v2005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:04:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2219.510590 secs 2024-07-26T12:04:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2219.510590 secs 2024-07-26T12:04:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2219.510590 secs 2024-07-26T12:04:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:23 smithi193 ceph-mon[37473]: pgmap v2006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:23 smithi134 ceph-mon[31966]: pgmap v2006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:23 smithi189 ceph-mon[36790]: pgmap v2006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2205.083418 secs 2024-07-26T12:04:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2205.083418 secs 2024-07-26T12:04:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2205.083418 secs 2024-07-26T12:04:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:25 smithi193 ceph-mon[37473]: pgmap v2007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:04:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:25 smithi134 ceph-mon[31966]: pgmap v2007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:04:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:25 smithi189 ceph-mon[36790]: pgmap v2007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:04:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2224.510715 secs 2024-07-26T12:04:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2224.510715 secs 2024-07-26T12:04:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2224.510715 secs 2024-07-26T12:04:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:27 smithi193 ceph-mon[37473]: pgmap v2008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:04:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:27 smithi134 ceph-mon[31966]: pgmap v2008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:04:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:27 smithi189 ceph-mon[36790]: pgmap v2008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:04:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:29 smithi193 ceph-mon[37473]: pgmap v2009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:04:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2210.083531 secs 2024-07-26T12:04:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:29 smithi134 ceph-mon[31966]: pgmap v2009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:04:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2210.083531 secs 2024-07-26T12:04:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:29 smithi189 ceph-mon[36790]: pgmap v2009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:04:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2210.083531 secs 2024-07-26T12:04:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:04:30.119+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:04:30.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:04:30.119+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:04:30.121 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:04:30.119+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:31 smithi193 ceph-mon[37473]: pgmap v2010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:04:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2229.510812 secs 2024-07-26T12:04:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:31 smithi134 ceph-mon[31966]: pgmap v2010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:04:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2229.510812 secs 2024-07-26T12:04:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:31 smithi189 ceph-mon[36790]: pgmap v2010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:04:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2229.510812 secs 2024-07-26T12:04:32.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:04:32.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:04:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:04:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:33 smithi134 ceph-mon[31966]: pgmap v2011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:04:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:33 smithi189 ceph-mon[36790]: pgmap v2011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:04:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:33.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:33 smithi193 ceph-mon[37473]: pgmap v2011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:04:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:34.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:04:34.602+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:04:34.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:04:34.602+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:04:34.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:04:34.602+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:34.605 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2215.083606 secs 2024-07-26T12:04:34.605 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:34.605 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:34.605 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:34.605 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2215.083606 secs 2024-07-26T12:04:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2215.083606 secs 2024-07-26T12:04:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:35 smithi134 ceph-mon[31966]: pgmap v2012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:04:35.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:35.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:35.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:35.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:35 smithi189 ceph-mon[36790]: pgmap v2012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:04:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:35.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:35 smithi193 ceph-mon[37473]: pgmap v2012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:04:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:36 smithi193 ceph-mon[37473]: pgmap v2013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2234.510939 secs 2024-07-26T12:04:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:36 smithi134 ceph-mon[31966]: pgmap v2013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2234.510939 secs 2024-07-26T12:04:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:36 smithi189 ceph-mon[36790]: pgmap v2013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2234.510939 secs 2024-07-26T12:04:38.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:38.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:38.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:04:38.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:04:38.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:38 smithi134 ceph-mon[31966]: pgmap v2014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:04:38.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:38 smithi134 ceph-mon[31966]: pgmap v2015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T12:04:38.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:04:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:04:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:38 smithi189 ceph-mon[36790]: pgmap v2014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:04:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:38 smithi189 ceph-mon[36790]: pgmap v2015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T12:04:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:04:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:04:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:38 smithi193 ceph-mon[37473]: pgmap v2014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:04:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:38 smithi193 ceph-mon[37473]: pgmap v2015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T12:04:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:04:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2220.083740 secs 2024-07-26T12:04:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2220.083740 secs 2024-07-26T12:04:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2220.083740 secs 2024-07-26T12:04:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:40 smithi134 ceph-mon[31966]: pgmap v2016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:40 smithi189 ceph-mon[36790]: pgmap v2016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:40 smithi193 ceph-mon[37473]: pgmap v2016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2239.511013 secs 2024-07-26T12:04:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2239.511013 secs 2024-07-26T12:04:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2239.511013 secs 2024-07-26T12:04:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:42 smithi134 ceph-mon[31966]: pgmap v2017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:42 smithi189 ceph-mon[36790]: pgmap v2017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:42 smithi193 ceph-mon[37473]: pgmap v2017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:43 smithi134 ceph-mon[31966]: pgmap v2018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:43 smithi189 ceph-mon[36790]: pgmap v2018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:43 smithi193 ceph-mon[37473]: pgmap v2018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2225.083843 secs 2024-07-26T12:04:44.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2225.083843 secs 2024-07-26T12:04:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2225.083843 secs 2024-07-26T12:04:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:04:45.134+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:04:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:04:45.134+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:04:45.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:04:45.134+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:45 smithi134 ceph-mon[31966]: pgmap v2019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:45.690 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:45 smithi193 ceph-mon[37473]: pgmap v2019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:45 smithi189 ceph-mon[36790]: pgmap v2019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:04:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2244.511105 secs 2024-07-26T12:04:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2244.511105 secs 2024-07-26T12:04:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2244.511105 secs 2024-07-26T12:04:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:47 smithi134 ceph-mon[31966]: pgmap v2020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:04:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:47 smithi189 ceph-mon[36790]: pgmap v2020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:04:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:47 smithi193 ceph-mon[37473]: pgmap v2020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:04:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2230.083987 secs 2024-07-26T12:04:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2230.083987 secs 2024-07-26T12:04:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2230.083987 secs 2024-07-26T12:04:49.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:04:49.611+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:04:49.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:04:49.611+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:04:49.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:04:49.611+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:04:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:50 smithi193 ceph-mon[37473]: pgmap v2021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:04:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:50 smithi134 ceph-mon[31966]: pgmap v2021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:04:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:50 smithi189 ceph-mon[36790]: pgmap v2021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:04:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2249.511231 secs 2024-07-26T12:04:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2249.511231 secs 2024-07-26T12:04:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2249.511231 secs 2024-07-26T12:04:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:52 smithi193 ceph-mon[37473]: pgmap v2022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:52 smithi134 ceph-mon[31966]: pgmap v2022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:52 smithi189 ceph-mon[36790]: pgmap v2022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2235.084085 secs 2024-07-26T12:04:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2235.084085 secs 2024-07-26T12:04:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2235.084085 secs 2024-07-26T12:04:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:54 smithi193 ceph-mon[37473]: pgmap v2023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:04:55.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:54 smithi134 ceph-mon[31966]: pgmap v2023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:04:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:54 smithi189 ceph-mon[36790]: pgmap v2023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:04:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:56 smithi134 ceph-mon[31966]: pgmap v2024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2254.511339 secs 2024-07-26T12:04:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:56 smithi189 ceph-mon[36790]: pgmap v2024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2254.511339 secs 2024-07-26T12:04:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:56 smithi193 ceph-mon[37473]: pgmap v2024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:04:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2254.511339 secs 2024-07-26T12:04:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:58 smithi134 ceph-mon[31966]: pgmap v2025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:04:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2240.084221 secs 2024-07-26T12:04:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:58 smithi189 ceph-mon[36790]: pgmap v2025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:04:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2240.084221 secs 2024-07-26T12:04:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:58 smithi193 ceph-mon[37473]: pgmap v2025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:04:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:04:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2240.084221 secs 2024-07-26T12:05:00.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:05:00.150+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:05:00.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:05:00.150+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:05:00.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:05:00.150+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:00 smithi134 ceph-mon[31966]: pgmap v2026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:00 smithi189 ceph-mon[36790]: pgmap v2026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:00 smithi193 ceph-mon[37473]: pgmap v2026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2259.511469 secs 2024-07-26T12:05:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2259.511469 secs 2024-07-26T12:05:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2259.511469 secs 2024-07-26T12:05:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:02 smithi134 ceph-mon[31966]: pgmap v2027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:02 smithi189 ceph-mon[36790]: pgmap v2027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:02 smithi193 ceph-mon[37473]: pgmap v2027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:04.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2245.084312 secs 2024-07-26T12:05:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2245.084312 secs 2024-07-26T12:05:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2245.084312 secs 2024-07-26T12:05:04.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:05:04.614+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:05:04.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:05:04.614+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:04.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:05:04.614+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:05:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:04 smithi134 ceph-mon[31966]: pgmap v2028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:04 smithi189 ceph-mon[36790]: pgmap v2028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:04 smithi193 ceph-mon[37473]: pgmap v2028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:06 smithi134 ceph-mon[31966]: pgmap v2029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2264.511601 secs 2024-07-26T12:05:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:06 smithi189 ceph-mon[36790]: pgmap v2029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2264.511601 secs 2024-07-26T12:05:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:06 smithi193 ceph-mon[37473]: pgmap v2029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2264.511601 secs 2024-07-26T12:05:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:08 smithi134 ceph-mon[31966]: pgmap v2030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2250.084414 secs 2024-07-26T12:05:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:08 smithi189 ceph-mon[36790]: pgmap v2030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2250.084414 secs 2024-07-26T12:05:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:08 smithi193 ceph-mon[37473]: pgmap v2030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2250.084414 secs 2024-07-26T12:05:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:10 smithi134 ceph-mon[31966]: pgmap v2031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:10 smithi189 ceph-mon[36790]: pgmap v2031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:10 smithi193 ceph-mon[37473]: pgmap v2031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2269.511698 secs 2024-07-26T12:05:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2269.511698 secs 2024-07-26T12:05:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2269.511698 secs 2024-07-26T12:05:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:12 smithi134 ceph-mon[31966]: pgmap v2032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:12 smithi189 ceph-mon[36790]: pgmap v2032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:12 smithi193 ceph-mon[37473]: pgmap v2032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2255.084484 secs 2024-07-26T12:05:14.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2255.084484 secs 2024-07-26T12:05:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2255.084484 secs 2024-07-26T12:05:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:05:15.165+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:05:15.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:05:15.165+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:15.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:05:15.165+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:05:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:14 smithi134 ceph-mon[31966]: pgmap v2033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:14 smithi189 ceph-mon[36790]: pgmap v2033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:14 smithi193 ceph-mon[37473]: pgmap v2033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:16 smithi134 ceph-mon[31966]: pgmap v2034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2274.511849 secs 2024-07-26T12:05:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:16 smithi189 ceph-mon[36790]: pgmap v2034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2274.511849 secs 2024-07-26T12:05:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:16 smithi193 ceph-mon[37473]: pgmap v2034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2274.511849 secs 2024-07-26T12:05:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:18 smithi134 ceph-mon[31966]: pgmap v2035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2260.084615 secs 2024-07-26T12:05:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:18 smithi189 ceph-mon[36790]: pgmap v2035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2260.084615 secs 2024-07-26T12:05:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:18 smithi193 ceph-mon[37473]: pgmap v2035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2260.084615 secs 2024-07-26T12:05:19.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:05:19.629+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:05:19.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:05:19.629+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:19.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:05:19.629+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:05:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:20 smithi134 ceph-mon[31966]: pgmap v2036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:20 smithi189 ceph-mon[36790]: pgmap v2036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:20 smithi193 ceph-mon[37473]: pgmap v2036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2279.511998 secs 2024-07-26T12:05:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2279.511998 secs 2024-07-26T12:05:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2279.511998 secs 2024-07-26T12:05:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:22 smithi134 ceph-mon[31966]: pgmap v2037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:22 smithi189 ceph-mon[36790]: pgmap v2037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:22 smithi193 ceph-mon[37473]: pgmap v2037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2265.084740 secs 2024-07-26T12:05:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2265.084740 secs 2024-07-26T12:05:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2265.084740 secs 2024-07-26T12:05:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:24 smithi134 ceph-mon[31966]: pgmap v2038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:24 smithi189 ceph-mon[36790]: pgmap v2038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:24 smithi193 ceph-mon[37473]: pgmap v2038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:26 smithi134 ceph-mon[31966]: pgmap v2039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2284.512127 secs 2024-07-26T12:05:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:26 smithi189 ceph-mon[36790]: pgmap v2039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2284.512127 secs 2024-07-26T12:05:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:26 smithi193 ceph-mon[37473]: pgmap v2039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2284.512127 secs 2024-07-26T12:05:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:28 smithi189 ceph-mon[36790]: pgmap v2040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2270.084870 secs 2024-07-26T12:05:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:28 smithi193 ceph-mon[37473]: pgmap v2040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2270.084870 secs 2024-07-26T12:05:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:28 smithi134 ceph-mon[31966]: pgmap v2040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2270.084870 secs 2024-07-26T12:05:30.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:05:30.181+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:05:30.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:05:30.181+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:05:30.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:05:30.181+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:30 smithi189 ceph-mon[36790]: pgmap v2041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:30 smithi193 ceph-mon[37473]: pgmap v2041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:30 smithi134 ceph-mon[31966]: pgmap v2041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:05:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2289.512239 secs 2024-07-26T12:05:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2289.512239 secs 2024-07-26T12:05:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2289.512239 secs 2024-07-26T12:05:33.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:32 smithi189 ceph-mon[36790]: pgmap v2042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:32 smithi193 ceph-mon[37473]: pgmap v2042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:33.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:32 smithi134 ceph-mon[31966]: pgmap v2042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2275.084994 secs 2024-07-26T12:05:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2275.084994 secs 2024-07-26T12:05:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2275.084994 secs 2024-07-26T12:05:34.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:05:34.639+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:05:34.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:05:34.639+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:34.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:05:34.639+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:05:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:34 smithi193 ceph-mon[37473]: pgmap v2043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:34 smithi134 ceph-mon[31966]: pgmap v2043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:34 smithi189 ceph-mon[36790]: pgmap v2043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:36 smithi193 ceph-mon[37473]: pgmap v2044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2294.512336 secs 2024-07-26T12:05:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:36 smithi134 ceph-mon[31966]: pgmap v2044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2294.512336 secs 2024-07-26T12:05:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:36 smithi189 ceph-mon[36790]: pgmap v2044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2294.512336 secs 2024-07-26T12:05:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:05:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:05:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:05:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:38 smithi193 ceph-mon[37473]: pgmap v2045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2280.085116 secs 2024-07-26T12:05:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:05:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:05:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:05:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:38 smithi134 ceph-mon[31966]: pgmap v2045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2280.085116 secs 2024-07-26T12:05:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:05:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:05:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:05:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:38 smithi189 ceph-mon[36790]: pgmap v2045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:05:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2280.085116 secs 2024-07-26T12:05:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:05:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:05:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:05:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:39 smithi193 ceph-mon[37473]: pgmap v2046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:05:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:39 smithi193 ceph-mon[37473]: pgmap v2047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:39 smithi134 ceph-mon[31966]: pgmap v2046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:05:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:39 smithi134 ceph-mon[31966]: pgmap v2047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:39 smithi189 ceph-mon[36790]: pgmap v2046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:05:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:39 smithi189 ceph-mon[36790]: pgmap v2047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:41 smithi193 ceph-mon[37473]: pgmap v2048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:42.292 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2299.512457 secs 2024-07-26T12:05:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:41 smithi134 ceph-mon[31966]: pgmap v2048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2299.512457 secs 2024-07-26T12:05:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:41 smithi189 ceph-mon[36790]: pgmap v2048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2299.512457 secs 2024-07-26T12:05:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:44 smithi193 ceph-mon[37473]: pgmap v2049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2285.085211 secs 2024-07-26T12:05:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:44 smithi134 ceph-mon[31966]: pgmap v2049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2285.085211 secs 2024-07-26T12:05:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:44 smithi189 ceph-mon[36790]: pgmap v2049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:05:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2285.085211 secs 2024-07-26T12:05:45.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:05:45.197+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:05:45.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:05:45.197+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:45.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:05:45.197+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:05:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:46 smithi193 ceph-mon[37473]: pgmap v2050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:05:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:46 smithi134 ceph-mon[31966]: pgmap v2050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:05:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:46 smithi189 ceph-mon[36790]: pgmap v2050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:05:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2304.512574 secs 2024-07-26T12:05:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2304.512574 secs 2024-07-26T12:05:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2304.512574 secs 2024-07-26T12:05:48.297 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:48 smithi134 ceph-mon[31966]: pgmap v2051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:05:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:48 smithi189 ceph-mon[36790]: pgmap v2051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:05:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:48 smithi193 ceph-mon[37473]: pgmap v2051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:05:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2290.085280 secs 2024-07-26T12:05:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2290.085280 secs 2024-07-26T12:05:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2290.085280 secs 2024-07-26T12:05:49.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:05:49.642+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:05:49.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:05:49.642+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:05:49.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:05:49.642+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:05:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:50 smithi134 ceph-mon[31966]: pgmap v2052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:05:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:50 smithi189 ceph-mon[36790]: pgmap v2052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:05:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:50 smithi193 ceph-mon[37473]: pgmap v2052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:05:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:52 smithi134 ceph-mon[31966]: pgmap v2053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2309.512677 secs 2024-07-26T12:05:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:52 smithi189 ceph-mon[36790]: pgmap v2053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2309.512677 secs 2024-07-26T12:05:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:52 smithi193 ceph-mon[37473]: pgmap v2053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2309.512677 secs 2024-07-26T12:05:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:54 smithi134 ceph-mon[31966]: pgmap v2054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2295.085410 secs 2024-07-26T12:05:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:54 smithi189 ceph-mon[36790]: pgmap v2054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2295.085410 secs 2024-07-26T12:05:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:54 smithi193 ceph-mon[37473]: pgmap v2054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:05:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2295.085410 secs 2024-07-26T12:05:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:56 smithi134 ceph-mon[31966]: pgmap v2055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:05:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:56 smithi189 ceph-mon[36790]: pgmap v2055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:05:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:56 smithi193 ceph-mon[37473]: pgmap v2055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:05:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2314.512742 secs 2024-07-26T12:05:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2314.512742 secs 2024-07-26T12:05:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2314.512742 secs 2024-07-26T12:05:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:58 smithi134 ceph-mon[31966]: pgmap v2056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:58 smithi189 ceph-mon[36790]: pgmap v2056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:58 smithi193 ceph-mon[37473]: pgmap v2056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:05:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:05:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2300.085536 secs 2024-07-26T12:05:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:05:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2300.085536 secs 2024-07-26T12:05:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:05:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2300.085536 secs 2024-07-26T12:06:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:06:00.212+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:06:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:06:00.212+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:06:00.212+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:06:00.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:00 smithi189 ceph-mon[36790]: pgmap v2057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:00 smithi134 ceph-mon[31966]: pgmap v2057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:00 smithi193 ceph-mon[37473]: pgmap v2057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:02 smithi134 ceph-mon[31966]: pgmap v2058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2319.512854 secs 2024-07-26T12:06:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:02 smithi189 ceph-mon[36790]: pgmap v2058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2319.512854 secs 2024-07-26T12:06:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:02 smithi193 ceph-mon[37473]: pgmap v2058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2319.512854 secs 2024-07-26T12:06:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:04 smithi134 ceph-mon[31966]: pgmap v2059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2305.085670 secs 2024-07-26T12:06:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:04 smithi189 ceph-mon[36790]: pgmap v2059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2305.085670 secs 2024-07-26T12:06:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:04 smithi193 ceph-mon[37473]: pgmap v2059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2305.085670 secs 2024-07-26T12:06:04.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:06:04.655+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:06:04.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:06:04.655+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:06:04.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:06:04.655+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:06 smithi134 ceph-mon[31966]: pgmap v2060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:06 smithi189 ceph-mon[36790]: pgmap v2060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:06 smithi193 ceph-mon[37473]: pgmap v2060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2324.512970 secs 2024-07-26T12:06:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2324.512970 secs 2024-07-26T12:06:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2324.512970 secs 2024-07-26T12:06:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:08 smithi134 ceph-mon[31966]: pgmap v2061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:08 smithi189 ceph-mon[36790]: pgmap v2061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:08 smithi193 ceph-mon[37473]: pgmap v2061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2310.085815 secs 2024-07-26T12:06:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2310.085815 secs 2024-07-26T12:06:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2310.085815 secs 2024-07-26T12:06:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:10 smithi134 ceph-mon[31966]: pgmap v2062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:10 smithi189 ceph-mon[36790]: pgmap v2062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:10 smithi193 ceph-mon[37473]: pgmap v2062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:12 smithi193 ceph-mon[37473]: pgmap v2063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2329.513093 secs 2024-07-26T12:06:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:12 smithi134 ceph-mon[31966]: pgmap v2063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2329.513093 secs 2024-07-26T12:06:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:12 smithi189 ceph-mon[36790]: pgmap v2063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2329.513093 secs 2024-07-26T12:06:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:14 smithi193 ceph-mon[37473]: pgmap v2064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2315.085959 secs 2024-07-26T12:06:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:14 smithi134 ceph-mon[31966]: pgmap v2064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2315.085959 secs 2024-07-26T12:06:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:14 smithi189 ceph-mon[36790]: pgmap v2064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2315.085959 secs 2024-07-26T12:06:15.229 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:06:15.227+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:06:15.229 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:06:15.228+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:06:15.229 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:06:15.228+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:16 smithi193 ceph-mon[37473]: pgmap v2065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:16 smithi134 ceph-mon[31966]: pgmap v2065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:16 smithi189 ceph-mon[36790]: pgmap v2065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2334.513212 secs 2024-07-26T12:06:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2334.513212 secs 2024-07-26T12:06:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2334.513212 secs 2024-07-26T12:06:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:18 smithi193 ceph-mon[37473]: pgmap v2066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:18 smithi134 ceph-mon[31966]: pgmap v2066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:18 smithi189 ceph-mon[36790]: pgmap v2066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2320.086031 secs 2024-07-26T12:06:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2320.086031 secs 2024-07-26T12:06:19.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:06:19.671+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:06:19.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:06:19.671+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:06:19.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:06:19.671+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2320.086031 secs 2024-07-26T12:06:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:20 smithi134 ceph-mon[31966]: pgmap v2067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:20 smithi193 ceph-mon[37473]: pgmap v2067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:20 smithi189 ceph-mon[36790]: pgmap v2067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2339.513368 secs 2024-07-26T12:06:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2339.513368 secs 2024-07-26T12:06:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2339.513368 secs 2024-07-26T12:06:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:22 smithi134 ceph-mon[31966]: pgmap v2068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:22 smithi189 ceph-mon[36790]: pgmap v2068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:22.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:06:22.734+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:06:22.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:06:22.734+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:06:22.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:06:22.735+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T12:06:22.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:06:22.735+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T12:06:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:22 smithi193 ceph-mon[37473]: pgmap v2068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:24 smithi134 ceph-mon[31966]: pgmap v2069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2325.086100 secs 2024-07-26T12:06:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:24 smithi189 ceph-mon[36790]: pgmap v2069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2325.086100 secs 2024-07-26T12:06:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:24 smithi193 ceph-mon[37473]: pgmap v2069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2325.086100 secs 2024-07-26T12:06:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:26 smithi134 ceph-mon[31966]: pgmap v2070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2344.513447 secs 2024-07-26T12:06:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:26 smithi189 ceph-mon[36790]: pgmap v2070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2344.513447 secs 2024-07-26T12:06:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:26 smithi193 ceph-mon[37473]: pgmap v2070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2344.513447 secs 2024-07-26T12:06:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:28 smithi134 ceph-mon[31966]: pgmap v2071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:28 smithi189 ceph-mon[36790]: pgmap v2071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:28 smithi193 ceph-mon[37473]: pgmap v2071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2330.086169 secs 2024-07-26T12:06:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2330.086169 secs 2024-07-26T12:06:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2330.086169 secs 2024-07-26T12:06:30.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:06:30.243+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:06:30.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:06:30.243+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:30.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:06:30.243+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:06:30.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:30 smithi189 ceph-mon[36790]: pgmap v2072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:30 smithi134 ceph-mon[31966]: pgmap v2072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:30 smithi193 ceph-mon[37473]: pgmap v2072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2349.513513 secs 2024-07-26T12:06:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2349.513513 secs 2024-07-26T12:06:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2349.513513 secs 2024-07-26T12:06:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:32 smithi134 ceph-mon[31966]: pgmap v2073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:32 smithi189 ceph-mon[36790]: pgmap v2073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:32 smithi193 ceph-mon[37473]: pgmap v2073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:33.238 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:06:33.237+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:06:33.238 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:06:33.237+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T12:06:33.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:06:33.237+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:06:33.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:06:33.237+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T12:06:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:34 smithi134 ceph-mon[31966]: pgmap v2074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2335.086290 secs 2024-07-26T12:06:34.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:06:34.687+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:06:34.688 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:06:34.687+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:34.689 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:06:34.687+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:06:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:34 smithi189 ceph-mon[36790]: pgmap v2074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2335.086290 secs 2024-07-26T12:06:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:34 smithi193 ceph-mon[37473]: pgmap v2074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2335.086290 secs 2024-07-26T12:06:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:36 smithi134 ceph-mon[31966]: pgmap v2075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2354.513607 secs 2024-07-26T12:06:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:36 smithi189 ceph-mon[36790]: pgmap v2075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2354.513607 secs 2024-07-26T12:06:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:36 smithi193 ceph-mon[37473]: pgmap v2075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:06:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2354.513607 secs 2024-07-26T12:06:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:38 smithi134 ceph-mon[31966]: pgmap v2076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:38 smithi189 ceph-mon[36790]: pgmap v2076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:38 smithi193 ceph-mon[37473]: pgmap v2076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2340.086404 secs 2024-07-26T12:06:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:06:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2340.086404 secs 2024-07-26T12:06:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:06:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2340.086404 secs 2024-07-26T12:06:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:06:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:40 smithi134 ceph-mon[31966]: pgmap v2077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:06:40.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:06:40.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:06:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:40 smithi189 ceph-mon[36790]: pgmap v2077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:06:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:06:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:06:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:40 smithi193 ceph-mon[37473]: pgmap v2077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:06:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:06:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:06:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:41 smithi134 ceph-mon[31966]: pgmap v2078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:06:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:41 smithi134 ceph-mon[31966]: pgmap v2079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:06:41.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2359.513727 secs 2024-07-26T12:06:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:41 smithi189 ceph-mon[36790]: pgmap v2078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:06:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:41 smithi189 ceph-mon[36790]: pgmap v2079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:06:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2359.513727 secs 2024-07-26T12:06:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:41 smithi193 ceph-mon[37473]: pgmap v2078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:06:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:41 smithi193 ceph-mon[37473]: pgmap v2079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:06:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2359.513727 secs 2024-07-26T12:06:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:43 smithi134 ceph-mon[31966]: pgmap v2080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:06:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:43 smithi189 ceph-mon[36790]: pgmap v2080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:06:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:43 smithi193 ceph-mon[37473]: pgmap v2080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:06:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2345.086522 secs 2024-07-26T12:06:44.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2345.086522 secs 2024-07-26T12:06:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2345.086522 secs 2024-07-26T12:06:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:06:45.245+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:06:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:06:45.245+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:45.247 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:06:45.245+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:06:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:45 smithi134 ceph-mon[31966]: pgmap v2081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:06:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:45 smithi189 ceph-mon[36790]: pgmap v2081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:06:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:45 smithi193 ceph-mon[37473]: pgmap v2081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:06:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2364.513859 secs 2024-07-26T12:06:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2364.513859 secs 2024-07-26T12:06:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2364.513859 secs 2024-07-26T12:06:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:47 smithi134 ceph-mon[31966]: pgmap v2082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:06:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:47 smithi189 ceph-mon[36790]: pgmap v2082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:06:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:47 smithi193 ceph-mon[37473]: pgmap v2082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:06:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:48 smithi134 ceph-mon[31966]: pgmap v2083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:06:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:48 smithi189 ceph-mon[36790]: pgmap v2083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:06:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:48 smithi193 ceph-mon[37473]: pgmap v2083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:06:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2350.086623 secs 2024-07-26T12:06:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2350.086623 secs 2024-07-26T12:06:49.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:06:49.702+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:06:49.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:06:49.702+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:06:49.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:06:49.702+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:06:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2350.086623 secs 2024-07-26T12:06:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:50 smithi134 ceph-mon[31966]: pgmap v2084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:06:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:50 smithi189 ceph-mon[36790]: pgmap v2084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:06:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:50 smithi193 ceph-mon[37473]: pgmap v2084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:06:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2369.513984 secs 2024-07-26T12:06:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2369.513984 secs 2024-07-26T12:06:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2369.513984 secs 2024-07-26T12:06:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:52 smithi189 ceph-mon[36790]: pgmap v2085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:52 smithi193 ceph-mon[37473]: pgmap v2085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:52 smithi134 ceph-mon[31966]: pgmap v2085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2355.086739 secs 2024-07-26T12:06:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2355.086739 secs 2024-07-26T12:06:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2355.086739 secs 2024-07-26T12:06:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:54 smithi193 ceph-mon[37473]: pgmap v2086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:54 smithi134 ceph-mon[31966]: pgmap v2086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:54 smithi189 ceph-mon[36790]: pgmap v2086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:56 smithi134 ceph-mon[31966]: pgmap v2087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2374.514102 secs 2024-07-26T12:06:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:56 smithi189 ceph-mon[36790]: pgmap v2087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2374.514102 secs 2024-07-26T12:06:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:56 smithi193 ceph-mon[37473]: pgmap v2087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:06:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2374.514102 secs 2024-07-26T12:06:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:58 smithi134 ceph-mon[31966]: pgmap v2088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:06:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2360.086867 secs 2024-07-26T12:06:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:58 smithi189 ceph-mon[36790]: pgmap v2088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:06:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2360.086867 secs 2024-07-26T12:06:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:58 smithi193 ceph-mon[37473]: pgmap v2088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:06:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:06:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2360.086867 secs 2024-07-26T12:07:00.263 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:07:00.261+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:07:00.263 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:07:00.261+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:07:00.263 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:07:00.261+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:00 smithi134 ceph-mon[31966]: pgmap v2089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:07:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:00 smithi189 ceph-mon[36790]: pgmap v2089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:07:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:00 smithi193 ceph-mon[37473]: pgmap v2089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:07:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2379.514244 secs 2024-07-26T12:07:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2379.514244 secs 2024-07-26T12:07:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2379.514244 secs 2024-07-26T12:07:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:02 smithi134 ceph-mon[31966]: pgmap v2090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:02 smithi189 ceph-mon[36790]: pgmap v2090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:02 smithi193 ceph-mon[37473]: pgmap v2090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2365.086982 secs 2024-07-26T12:07:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2365.086982 secs 2024-07-26T12:07:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2365.086982 secs 2024-07-26T12:07:04.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:07:04.704+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:07:04.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:07:04.704+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:04.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:07:04.704+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:07:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:04 smithi134 ceph-mon[31966]: pgmap v2091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:04 smithi189 ceph-mon[36790]: pgmap v2091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:04 smithi193 ceph-mon[37473]: pgmap v2091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:06 smithi134 ceph-mon[31966]: pgmap v2092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2384.514362 secs 2024-07-26T12:07:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:06 smithi189 ceph-mon[36790]: pgmap v2092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2384.514362 secs 2024-07-26T12:07:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:06 smithi193 ceph-mon[37473]: pgmap v2092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2384.514362 secs 2024-07-26T12:07:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:08 smithi189 ceph-mon[36790]: pgmap v2093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2370.087050 secs 2024-07-26T12:07:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:08 smithi193 ceph-mon[37473]: pgmap v2093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2370.087050 secs 2024-07-26T12:07:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:08 smithi134 ceph-mon[31966]: pgmap v2093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2370.087050 secs 2024-07-26T12:07:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:10 smithi189 ceph-mon[36790]: pgmap v2094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:07:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:10 smithi193 ceph-mon[37473]: pgmap v2094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:07:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:10 smithi134 ceph-mon[31966]: pgmap v2094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:07:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2389.514426 secs 2024-07-26T12:07:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2389.514426 secs 2024-07-26T12:07:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2389.514426 secs 2024-07-26T12:07:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:12 smithi193 ceph-mon[37473]: pgmap v2095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:13.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:12 smithi134 ceph-mon[31966]: pgmap v2095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:12 smithi189 ceph-mon[36790]: pgmap v2095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2375.087207 secs 2024-07-26T12:07:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2375.087207 secs 2024-07-26T12:07:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2375.087207 secs 2024-07-26T12:07:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:14 smithi193 ceph-mon[37473]: pgmap v2096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:15.278 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:07:15.277+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:07:15.278 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:07:15.277+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:15.278 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:07:15.277+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:07:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:14 smithi134 ceph-mon[31966]: pgmap v2096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:14 smithi189 ceph-mon[36790]: pgmap v2096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:16 smithi193 ceph-mon[37473]: pgmap v2097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2394.514554 secs 2024-07-26T12:07:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:16 smithi134 ceph-mon[31966]: pgmap v2097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2394.514554 secs 2024-07-26T12:07:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:16 smithi189 ceph-mon[36790]: pgmap v2097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2394.514554 secs 2024-07-26T12:07:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:18 smithi193 ceph-mon[37473]: pgmap v2098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2380.087331 secs 2024-07-26T12:07:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:18 smithi134 ceph-mon[31966]: pgmap v2098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2380.087331 secs 2024-07-26T12:07:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:18 smithi189 ceph-mon[36790]: pgmap v2098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2380.087331 secs 2024-07-26T12:07:19.721 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:07:19.720+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:07:19.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:07:19.720+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:19.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:07:19.720+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:07:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:20 smithi193 ceph-mon[37473]: pgmap v2099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:07:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:20 smithi134 ceph-mon[31966]: pgmap v2099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:07:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:20 smithi189 ceph-mon[36790]: pgmap v2099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:07:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2399.514684 secs 2024-07-26T12:07:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2399.514684 secs 2024-07-26T12:07:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2399.514684 secs 2024-07-26T12:07:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:23 smithi193 ceph-mon[37473]: pgmap v2100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:23 smithi134 ceph-mon[31966]: pgmap v2100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:23 smithi189 ceph-mon[36790]: pgmap v2100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2385.087473 secs 2024-07-26T12:07:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2385.087473 secs 2024-07-26T12:07:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2385.087473 secs 2024-07-26T12:07:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:25 smithi193 ceph-mon[37473]: pgmap v2101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:25 smithi134 ceph-mon[31966]: pgmap v2101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:25 smithi189 ceph-mon[36790]: pgmap v2101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:27 smithi134 ceph-mon[31966]: pgmap v2102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2404.514751 secs 2024-07-26T12:07:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:27 smithi189 ceph-mon[36790]: pgmap v2102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2404.514751 secs 2024-07-26T12:07:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:27 smithi193 ceph-mon[37473]: pgmap v2102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2404.514751 secs 2024-07-26T12:07:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:29 smithi134 ceph-mon[31966]: pgmap v2103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2390.087599 secs 2024-07-26T12:07:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:29 smithi189 ceph-mon[36790]: pgmap v2103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2390.087599 secs 2024-07-26T12:07:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:29 smithi193 ceph-mon[37473]: pgmap v2103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2390.087599 secs 2024-07-26T12:07:30.293 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:07:30.292+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:07:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:07:30.292+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:07:30.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:07:30.292+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:31 smithi134 ceph-mon[31966]: pgmap v2104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:07:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:31 smithi189 ceph-mon[36790]: pgmap v2104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:07:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:31 smithi193 ceph-mon[37473]: pgmap v2104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:07:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2409.514877 secs 2024-07-26T12:07:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2409.514877 secs 2024-07-26T12:07:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2409.514877 secs 2024-07-26T12:07:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:33 smithi134 ceph-mon[31966]: pgmap v2105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:33 smithi189 ceph-mon[36790]: pgmap v2105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:33 smithi193 ceph-mon[37473]: pgmap v2105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2395.087718 secs 2024-07-26T12:07:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2395.087718 secs 2024-07-26T12:07:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2395.087718 secs 2024-07-26T12:07:34.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:07:34.735+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:07:34.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:07:34.735+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:34.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:07:34.735+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:07:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:35 smithi134 ceph-mon[31966]: pgmap v2106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:35 smithi189 ceph-mon[36790]: pgmap v2106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:35 smithi193 ceph-mon[37473]: pgmap v2106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:37.353 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:37 smithi134 ceph-mon[31966]: pgmap v2107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:37.353 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2414.514977 secs 2024-07-26T12:07:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:37 smithi189 ceph-mon[36790]: pgmap v2107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2414.514977 secs 2024-07-26T12:07:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:37 smithi193 ceph-mon[37473]: pgmap v2107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:07:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2414.514977 secs 2024-07-26T12:07:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:39 smithi134 ceph-mon[31966]: pgmap v2108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2400.087863 secs 2024-07-26T12:07:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:39 smithi189 ceph-mon[36790]: pgmap v2108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2400.087863 secs 2024-07-26T12:07:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:39 smithi193 ceph-mon[37473]: pgmap v2108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:07:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2400.087863 secs 2024-07-26T12:07:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:07:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:07:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:07:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:41 smithi134 ceph-mon[31966]: pgmap v2109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:07:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:07:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:07:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:07:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:41 smithi189 ceph-mon[36790]: pgmap v2109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:07:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:07:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:07:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:07:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:41 smithi193 ceph-mon[37473]: pgmap v2109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:07:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:07:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:07:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:07:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2419.515103 secs 2024-07-26T12:07:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:42 smithi134 ceph-mon[31966]: pgmap v2110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:07:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:42 smithi134 ceph-mon[31966]: pgmap v2111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:07:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2419.515103 secs 2024-07-26T12:07:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:42 smithi189 ceph-mon[36790]: pgmap v2110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:07:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:42 smithi189 ceph-mon[36790]: pgmap v2111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:07:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2419.515103 secs 2024-07-26T12:07:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:42 smithi193 ceph-mon[37473]: pgmap v2110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:07:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:42 smithi193 ceph-mon[37473]: pgmap v2111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:07:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:44 smithi134 ceph-mon[31966]: pgmap v2112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:07:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2405.088008 secs 2024-07-26T12:07:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:44 smithi189 ceph-mon[36790]: pgmap v2112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:07:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2405.088008 secs 2024-07-26T12:07:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:44 smithi193 ceph-mon[37473]: pgmap v2112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:07:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2405.088008 secs 2024-07-26T12:07:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:07:45.307+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:07:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:07:45.307+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:45.309 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:07:45.307+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:07:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:46 smithi134 ceph-mon[31966]: pgmap v2113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:07:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:46 smithi189 ceph-mon[36790]: pgmap v2113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:07:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:46 smithi193 ceph-mon[37473]: pgmap v2113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:07:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2424.515243 secs 2024-07-26T12:07:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2424.515243 secs 2024-07-26T12:07:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2424.515243 secs 2024-07-26T12:07:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:48 smithi134 ceph-mon[31966]: pgmap v2114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:07:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:48 smithi189 ceph-mon[36790]: pgmap v2114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:07:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:48 smithi193 ceph-mon[37473]: pgmap v2114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:07:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2410.088079 secs 2024-07-26T12:07:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2410.088079 secs 2024-07-26T12:07:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2410.088079 secs 2024-07-26T12:07:49.752 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:07:49.750+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:07:49.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:07:49.750+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:07:49.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:07:49.750+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:07:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:50 smithi134 ceph-mon[31966]: pgmap v2115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:07:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:50 smithi189 ceph-mon[36790]: pgmap v2115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:07:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:50 smithi193 ceph-mon[37473]: pgmap v2115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:07:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2429.515411 secs 2024-07-26T12:07:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:52 smithi134 ceph-mon[31966]: pgmap v2116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:07:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2429.515411 secs 2024-07-26T12:07:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:52 smithi189 ceph-mon[36790]: pgmap v2116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:07:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2429.515411 secs 2024-07-26T12:07:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:52 smithi193 ceph-mon[37473]: pgmap v2116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:07:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:54 smithi134 ceph-mon[31966]: pgmap v2117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:07:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2415.088180 secs 2024-07-26T12:07:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:54 smithi189 ceph-mon[36790]: pgmap v2117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:07:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2415.088180 secs 2024-07-26T12:07:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:54 smithi193 ceph-mon[37473]: pgmap v2117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:07:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2415.088180 secs 2024-07-26T12:07:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:56 smithi193 ceph-mon[37473]: pgmap v2118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:07:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2434.515529 secs 2024-07-26T12:07:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:56 smithi134 ceph-mon[31966]: pgmap v2118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:07:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2434.515529 secs 2024-07-26T12:07:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:56 smithi189 ceph-mon[36790]: pgmap v2118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:07:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2434.515529 secs 2024-07-26T12:07:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:58 smithi134 ceph-mon[31966]: pgmap v2119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:07:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:58 smithi189 ceph-mon[36790]: pgmap v2119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:07:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:58 smithi193 ceph-mon[37473]: pgmap v2119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:07:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:07:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2420.088288 secs 2024-07-26T12:07:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:07:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2420.088288 secs 2024-07-26T12:07:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:07:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2420.088288 secs 2024-07-26T12:08:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:08:00.313+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:08:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:08:00.313+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:00.315 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:08:00.313+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:08:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:00 smithi134 ceph-mon[31966]: pgmap v2120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:08:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:00 smithi189 ceph-mon[36790]: pgmap v2120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:08:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:00 smithi193 ceph-mon[37473]: pgmap v2120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:08:01.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2439.515656 secs 2024-07-26T12:08:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2439.515656 secs 2024-07-26T12:08:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2439.515656 secs 2024-07-26T12:08:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:02 smithi134 ceph-mon[31966]: pgmap v2121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:02 smithi189 ceph-mon[36790]: pgmap v2121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:02 smithi193 ceph-mon[37473]: pgmap v2121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:04 smithi134 ceph-mon[31966]: pgmap v2122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:08:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2425.088364 secs 2024-07-26T12:08:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:04 smithi189 ceph-mon[36790]: pgmap v2122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:08:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2425.088364 secs 2024-07-26T12:08:04.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:08:04.763+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:08:04.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:08:04.763+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:04.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:08:04.763+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:08:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:04 smithi193 ceph-mon[37473]: pgmap v2122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:08:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2425.088364 secs 2024-07-26T12:08:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:06 smithi134 ceph-mon[31966]: pgmap v2123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2444.515780 secs 2024-07-26T12:08:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:06 smithi189 ceph-mon[36790]: pgmap v2123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2444.515780 secs 2024-07-26T12:08:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:06 smithi193 ceph-mon[37473]: pgmap v2123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2444.515780 secs 2024-07-26T12:08:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:08 smithi134 ceph-mon[31966]: pgmap v2124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:08:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:08 smithi189 ceph-mon[36790]: pgmap v2124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:08:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:08 smithi193 ceph-mon[37473]: pgmap v2124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:08:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2430.088500 secs 2024-07-26T12:08:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2430.088500 secs 2024-07-26T12:08:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2430.088500 secs 2024-07-26T12:08:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:10 smithi134 ceph-mon[31966]: pgmap v2125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:08:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:10 smithi189 ceph-mon[36790]: pgmap v2125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:08:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:10 smithi193 ceph-mon[37473]: pgmap v2125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:08:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2449.515904 secs 2024-07-26T12:08:11.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2449.515904 secs 2024-07-26T12:08:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2449.515904 secs 2024-07-26T12:08:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:12 smithi134 ceph-mon[31966]: pgmap v2126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:12 smithi189 ceph-mon[36790]: pgmap v2126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:12 smithi193 ceph-mon[37473]: pgmap v2126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:14 smithi134 ceph-mon[31966]: pgmap v2127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2435.088647 secs 2024-07-26T12:08:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:14 smithi189 ceph-mon[36790]: pgmap v2127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2435.088647 secs 2024-07-26T12:08:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:14 smithi193 ceph-mon[37473]: pgmap v2127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2435.088647 secs 2024-07-26T12:08:15.319 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:08:15.318+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:08:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:08:15.318+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:08:15.320 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:08:15.318+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:16 smithi134 ceph-mon[31966]: pgmap v2128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2454.516026 secs 2024-07-26T12:08:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:16 smithi189 ceph-mon[36790]: pgmap v2128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2454.516026 secs 2024-07-26T12:08:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:16 smithi193 ceph-mon[37473]: pgmap v2128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2454.516026 secs 2024-07-26T12:08:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:18 smithi134 ceph-mon[31966]: pgmap v2129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:08:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:18 smithi189 ceph-mon[36790]: pgmap v2129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:08:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:18 smithi193 ceph-mon[37473]: pgmap v2129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:08:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2440.088789 secs 2024-07-26T12:08:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2440.088789 secs 2024-07-26T12:08:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2440.088789 secs 2024-07-26T12:08:19.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:08:19.778+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:08:19.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:08:19.779+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:19.780 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:08:19.779+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:08:20.675 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:20 smithi134 ceph-mon[31966]: pgmap v2130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:08:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:20 smithi189 ceph-mon[36790]: pgmap v2130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:08:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:20 smithi193 ceph-mon[37473]: pgmap v2130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:08:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2459.516150 secs 2024-07-26T12:08:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2459.516150 secs 2024-07-26T12:08:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2459.516150 secs 2024-07-26T12:08:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:22 smithi134 ceph-mon[31966]: pgmap v2131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:08:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:22 smithi189 ceph-mon[36790]: pgmap v2131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:08:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:22 smithi193 ceph-mon[37473]: pgmap v2131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:08:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:24 smithi134 ceph-mon[31966]: pgmap v2132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:08:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2445.088918 secs 2024-07-26T12:08:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:24 smithi189 ceph-mon[36790]: pgmap v2132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:08:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2445.088918 secs 2024-07-26T12:08:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:24 smithi193 ceph-mon[37473]: pgmap v2132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:08:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2445.088918 secs 2024-07-26T12:08:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:25 smithi193 ceph-mon[37473]: pgmap v2133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:08:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:25 smithi134 ceph-mon[31966]: pgmap v2133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:08:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:25 smithi189 ceph-mon[36790]: pgmap v2133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:08:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2464.516271 secs 2024-07-26T12:08:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2464.516271 secs 2024-07-26T12:08:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2464.516271 secs 2024-07-26T12:08:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:27 smithi193 ceph-mon[37473]: pgmap v2134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:08:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:27 smithi134 ceph-mon[31966]: pgmap v2134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:08:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:27 smithi189 ceph-mon[36790]: pgmap v2134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:08:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2450.088958 secs 2024-07-26T12:08:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2450.088958 secs 2024-07-26T12:08:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2450.088958 secs 2024-07-26T12:08:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:29 smithi134 ceph-mon[31966]: pgmap v2135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:08:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:29 smithi189 ceph-mon[36790]: pgmap v2135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:08:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:29 smithi193 ceph-mon[37473]: pgmap v2135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:08:30.331 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:08:30.329+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:08:30.331 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:08:30.330+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:08:30.332 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:08:30.330+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2469.516435 secs 2024-07-26T12:08:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:32 smithi134 ceph-mon[31966]: pgmap v2136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2469.516435 secs 2024-07-26T12:08:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:32 smithi189 ceph-mon[36790]: pgmap v2136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2469.516435 secs 2024-07-26T12:08:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:32 smithi193 ceph-mon[37473]: pgmap v2136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:34 smithi134 ceph-mon[31966]: pgmap v2137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:08:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2455.089056 secs 2024-07-26T12:08:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:34 smithi189 ceph-mon[36790]: pgmap v2137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:08:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2455.089056 secs 2024-07-26T12:08:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:34 smithi193 ceph-mon[37473]: pgmap v2137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:08:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2455.089056 secs 2024-07-26T12:08:34.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:08:34.794+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:08:34.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:08:34.794+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:34.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:08:34.794+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:08:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:36 smithi134 ceph-mon[31966]: pgmap v2138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:08:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2474.516557 secs 2024-07-26T12:08:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:36 smithi189 ceph-mon[36790]: pgmap v2138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:08:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2474.516557 secs 2024-07-26T12:08:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:36 smithi193 ceph-mon[37473]: pgmap v2138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:08:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2474.516557 secs 2024-07-26T12:08:37.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:08:37.758+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:08:37.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:08:37.758+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:08:37.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:08:37.759+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T12:08:37.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:08:37.759+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T12:08:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:38 smithi134 ceph-mon[31966]: pgmap v2139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:08:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:38 smithi189 ceph-mon[36790]: pgmap v2139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:08:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:38 smithi193 ceph-mon[37473]: pgmap v2139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:08:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2460.089168 secs 2024-07-26T12:08:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2460.089168 secs 2024-07-26T12:08:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2460.089168 secs 2024-07-26T12:08:40.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:40 smithi134 ceph-mon[31966]: pgmap v2140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:08:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:40 smithi189 ceph-mon[36790]: pgmap v2140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:08:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:40 smithi193 ceph-mon[37473]: pgmap v2140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:08:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2479.516642 secs 2024-07-26T12:08:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:08:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2479.516642 secs 2024-07-26T12:08:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:08:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2479.516642 secs 2024-07-26T12:08:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:08:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:42 smithi134 ceph-mon[31966]: pgmap v2141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:08:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:08:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:08:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:08:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:42 smithi189 ceph-mon[36790]: pgmap v2141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:08:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:08:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:08:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:08:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:42 smithi193 ceph-mon[37473]: pgmap v2141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:08:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:08:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:08:42.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:08:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:43 smithi134 ceph-mon[31966]: pgmap v2142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:08:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:43 smithi134 ceph-mon[31966]: pgmap v2143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:08:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:43 smithi189 ceph-mon[36790]: pgmap v2142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:08:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:43 smithi189 ceph-mon[36790]: pgmap v2143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:08:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:43 smithi193 ceph-mon[37473]: pgmap v2142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:08:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:43 smithi193 ceph-mon[37473]: pgmap v2143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:08:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2465.089294 secs 2024-07-26T12:08:44.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2465.089294 secs 2024-07-26T12:08:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2465.089294 secs 2024-07-26T12:08:45.347 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:08:45.346+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:08:45.347 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:08:45.346+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:45.347 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:08:45.346+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:08:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:45 smithi134 ceph-mon[31966]: pgmap v2144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:08:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:45 smithi189 ceph-mon[36790]: pgmap v2144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:08:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:45 smithi193 ceph-mon[37473]: pgmap v2144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:08:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2484.516773 secs 2024-07-26T12:08:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2484.516773 secs 2024-07-26T12:08:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2484.516773 secs 2024-07-26T12:08:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:47 smithi134 ceph-mon[31966]: pgmap v2145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:08:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:47 smithi189 ceph-mon[36790]: pgmap v2145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:08:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:47 smithi193 ceph-mon[37473]: pgmap v2145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:08:48.262 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:08:48.261+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T12:08:48.262 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:08:48.261+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:08:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:08:48.262+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:08:48.263 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:08:48.262+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T12:08:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:49 smithi134 ceph-mon[31966]: pgmap v2146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:08:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2470.089422 secs 2024-07-26T12:08:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:49 smithi189 ceph-mon[36790]: pgmap v2146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:08:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2470.089422 secs 2024-07-26T12:08:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:49 smithi193 ceph-mon[37473]: pgmap v2146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:08:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2470.089422 secs 2024-07-26T12:08:49.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:08:49.810+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:08:49.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:08:49.810+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:08:49.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:08:49.810+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:08:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:51 smithi134 ceph-mon[31966]: pgmap v2147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:08:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2489.516890 secs 2024-07-26T12:08:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:51 smithi189 ceph-mon[36790]: pgmap v2147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:08:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2489.516890 secs 2024-07-26T12:08:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:51 smithi193 ceph-mon[37473]: pgmap v2147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:08:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2489.516890 secs 2024-07-26T12:08:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:53 smithi134 ceph-mon[31966]: pgmap v2148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:08:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:53 smithi189 ceph-mon[36790]: pgmap v2148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:08:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:53 smithi193 ceph-mon[37473]: pgmap v2148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:08:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2475.089572 secs 2024-07-26T12:08:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2475.089572 secs 2024-07-26T12:08:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2475.089572 secs 2024-07-26T12:08:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:55 smithi134 ceph-mon[31966]: pgmap v2149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:55 smithi189 ceph-mon[36790]: pgmap v2149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:55 smithi193 ceph-mon[37473]: pgmap v2149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:08:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2494.517036 secs 2024-07-26T12:08:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2494.517036 secs 2024-07-26T12:08:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2494.517036 secs 2024-07-26T12:08:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:57 smithi134 ceph-mon[31966]: pgmap v2150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:57 smithi189 ceph-mon[36790]: pgmap v2150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:57 smithi193 ceph-mon[37473]: pgmap v2150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:08:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:58 smithi189 ceph-mon[36790]: pgmap v2151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:58 smithi193 ceph-mon[37473]: pgmap v2151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:58 smithi134 ceph-mon[31966]: pgmap v2151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:08:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:08:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2480.089704 secs 2024-07-26T12:08:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:08:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2480.089704 secs 2024-07-26T12:08:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:08:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2480.089704 secs 2024-07-26T12:09:00.359 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:09:00.357+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:09:00.359 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:09:00.357+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:09:00.359 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:09:00.357+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:00 smithi193 ceph-mon[37473]: pgmap v2152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:00 smithi134 ceph-mon[31966]: pgmap v2152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:00 smithi189 ceph-mon[36790]: pgmap v2152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2499.517165 secs 2024-07-26T12:09:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2499.517165 secs 2024-07-26T12:09:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2499.517165 secs 2024-07-26T12:09:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:02 smithi193 ceph-mon[37473]: pgmap v2153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:02.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:02 smithi189 ceph-mon[36790]: pgmap v2153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:02 smithi134 ceph-mon[31966]: pgmap v2153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2485.089834 secs 2024-07-26T12:09:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2485.089834 secs 2024-07-26T12:09:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2485.089834 secs 2024-07-26T12:09:04.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:09:04.825+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:09:04.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:09:04.825+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:04.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:09:04.825+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:09:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:04 smithi134 ceph-mon[31966]: pgmap v2154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:04 smithi189 ceph-mon[36790]: pgmap v2154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:04 smithi193 ceph-mon[37473]: pgmap v2154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2504.517298 secs 2024-07-26T12:09:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:07 smithi193 ceph-mon[37473]: pgmap v2155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2504.517298 secs 2024-07-26T12:09:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:07 smithi134 ceph-mon[31966]: pgmap v2155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2504.517298 secs 2024-07-26T12:09:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:07 smithi189 ceph-mon[36790]: pgmap v2155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:09 smithi193 ceph-mon[37473]: pgmap v2156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2490.089901 secs 2024-07-26T12:09:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:09 smithi134 ceph-mon[31966]: pgmap v2156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2490.089901 secs 2024-07-26T12:09:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:09 smithi189 ceph-mon[36790]: pgmap v2156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2490.089901 secs 2024-07-26T12:09:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:11 smithi134 ceph-mon[31966]: pgmap v2157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2509.517426 secs 2024-07-26T12:09:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:11 smithi189 ceph-mon[36790]: pgmap v2157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2509.517426 secs 2024-07-26T12:09:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:11 smithi193 ceph-mon[37473]: pgmap v2157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2509.517426 secs 2024-07-26T12:09:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:13 smithi134 ceph-mon[31966]: pgmap v2158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:13 smithi189 ceph-mon[36790]: pgmap v2158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:13 smithi193 ceph-mon[37473]: pgmap v2158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:14.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2495.090026 secs 2024-07-26T12:09:14.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2495.090026 secs 2024-07-26T12:09:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2495.090026 secs 2024-07-26T12:09:15.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:09:15.359+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:09:15.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:09:15.359+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:09:15.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:09:15.359+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:15 smithi134 ceph-mon[31966]: pgmap v2159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:15 smithi189 ceph-mon[36790]: pgmap v2159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:15 smithi193 ceph-mon[37473]: pgmap v2159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2514.517551 secs 2024-07-26T12:09:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2514.517551 secs 2024-07-26T12:09:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2514.517551 secs 2024-07-26T12:09:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:17 smithi134 ceph-mon[31966]: pgmap v2160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:17 smithi189 ceph-mon[36790]: pgmap v2160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:17 smithi193 ceph-mon[37473]: pgmap v2160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:19 smithi134 ceph-mon[31966]: pgmap v2161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2500.090104 secs 2024-07-26T12:09:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:19 smithi189 ceph-mon[36790]: pgmap v2161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2500.090104 secs 2024-07-26T12:09:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:19 smithi193 ceph-mon[37473]: pgmap v2161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2500.090104 secs 2024-07-26T12:09:19.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:09:19.841+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:09:19.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:09:19.841+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:09:19.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:09:19.841+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:21 smithi134 ceph-mon[31966]: pgmap v2162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2519.517697 secs 2024-07-26T12:09:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:21 smithi189 ceph-mon[36790]: pgmap v2162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2519.517697 secs 2024-07-26T12:09:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:21 smithi193 ceph-mon[37473]: pgmap v2162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2519.517697 secs 2024-07-26T12:09:22.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:09:22.765+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:09:22.767 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:09:22.766+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:09:22.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:09:22.766+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T12:09:22.768 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:09:22.766+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T12:09:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:23 smithi134 ceph-mon[31966]: pgmap v2163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:23 smithi189 ceph-mon[36790]: pgmap v2163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:23 smithi193 ceph-mon[37473]: pgmap v2163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2505.090225 secs 2024-07-26T12:09:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2505.090225 secs 2024-07-26T12:09:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2505.090225 secs 2024-07-26T12:09:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:25 smithi134 ceph-mon[31966]: pgmap v2164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:25 smithi189 ceph-mon[36790]: pgmap v2164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:25 smithi193 ceph-mon[37473]: pgmap v2164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2524.517826 secs 2024-07-26T12:09:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2524.517826 secs 2024-07-26T12:09:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2524.517826 secs 2024-07-26T12:09:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:27 smithi134 ceph-mon[31966]: pgmap v2165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:27 smithi189 ceph-mon[36790]: pgmap v2165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:27 smithi193 ceph-mon[37473]: pgmap v2165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:29 smithi134 ceph-mon[31966]: pgmap v2166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2510.090356 secs 2024-07-26T12:09:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:29 smithi189 ceph-mon[36790]: pgmap v2166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2510.090356 secs 2024-07-26T12:09:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:29 smithi193 ceph-mon[37473]: pgmap v2166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2510.090356 secs 2024-07-26T12:09:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:09:30.360+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:09:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:09:30.360+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:09:30.362 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:09:30.360+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:31 smithi134 ceph-mon[31966]: pgmap v2167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2529.517908 secs 2024-07-26T12:09:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:31 smithi189 ceph-mon[36790]: pgmap v2167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2529.517908 secs 2024-07-26T12:09:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:31 smithi193 ceph-mon[37473]: pgmap v2167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2529.517908 secs 2024-07-26T12:09:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:09:33.270+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:09:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:09:33.270+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T12:09:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:09:33.271+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T12:09:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:09:33.271+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:09:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:33 smithi134 ceph-mon[31966]: pgmap v2168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:33 smithi189 ceph-mon[36790]: pgmap v2168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:33 smithi193 ceph-mon[37473]: pgmap v2168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2515.090439 secs 2024-07-26T12:09:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2515.090439 secs 2024-07-26T12:09:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2515.090439 secs 2024-07-26T12:09:34.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:09:34.856+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:09:34.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:09:34.856+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:34.858 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:09:34.856+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:09:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:35 smithi134 ceph-mon[31966]: pgmap v2169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:35 smithi189 ceph-mon[36790]: pgmap v2169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:35 smithi193 ceph-mon[37473]: pgmap v2169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2534.518009 secs 2024-07-26T12:09:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2534.518009 secs 2024-07-26T12:09:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2534.518009 secs 2024-07-26T12:09:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:37 smithi134 ceph-mon[31966]: pgmap v2170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:37 smithi189 ceph-mon[36790]: pgmap v2170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:37 smithi193 ceph-mon[37473]: pgmap v2170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:39 smithi134 ceph-mon[31966]: pgmap v2171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2520.090580 secs 2024-07-26T12:09:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:39 smithi189 ceph-mon[36790]: pgmap v2171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2520.090580 secs 2024-07-26T12:09:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:39 smithi193 ceph-mon[37473]: pgmap v2171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2520.090580 secs 2024-07-26T12:09:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:41 smithi134 ceph-mon[31966]: pgmap v2172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2539.518135 secs 2024-07-26T12:09:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:41 smithi189 ceph-mon[36790]: pgmap v2172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2539.518135 secs 2024-07-26T12:09:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:41 smithi193 ceph-mon[37473]: pgmap v2172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2539.518135 secs 2024-07-26T12:09:42.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:09:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:09:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:09:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:43 smithi189 ceph-mon[36790]: pgmap v2173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:43 smithi193 ceph-mon[37473]: pgmap v2173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:43 smithi134 ceph-mon[31966]: pgmap v2173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:09:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2525.090731 secs 2024-07-26T12:09:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:44 smithi134 ceph-mon[31966]: pgmap v2174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2525.090731 secs 2024-07-26T12:09:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:44 smithi189 ceph-mon[36790]: pgmap v2174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2525.090731 secs 2024-07-26T12:09:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:44 smithi193 ceph-mon[37473]: pgmap v2174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:09:45.372 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:09:45.370+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:09:45.372 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:09:45.370+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:45.373 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:09:45.370+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:09:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2544.518264 secs 2024-07-26T12:09:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2544.518264 secs 2024-07-26T12:09:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2544.518264 secs 2024-07-26T12:09:47.632 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:47 smithi134 ceph-mon[31966]: pgmap v2175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:47 smithi189 ceph-mon[36790]: pgmap v2175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:47 smithi193 ceph-mon[37473]: pgmap v2175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:09:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:49 smithi134 ceph-mon[31966]: pgmap v2176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2530.090799 secs 2024-07-26T12:09:49.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:09:49.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:09:49.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:09:49.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:09:49.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:09:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:49 smithi189 ceph-mon[36790]: pgmap v2176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2530.090799 secs 2024-07-26T12:09:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:09:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:09:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:09:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:09:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:09:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:49 smithi193 ceph-mon[37473]: pgmap v2176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:09:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2530.090799 secs 2024-07-26T12:09:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:09:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:09:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:09:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:09:49.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:09:49.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:09:49.872+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:09:49.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:09:49.872+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:09:49.874 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:09:49.872+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:09:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:50 smithi134 ceph-mon[31966]: pgmap v2177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:09:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:50 smithi134 ceph-mon[31966]: pgmap v2178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:09:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:50 smithi189 ceph-mon[36790]: pgmap v2177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:09:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:50 smithi189 ceph-mon[36790]: pgmap v2178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:09:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:50 smithi193 ceph-mon[37473]: pgmap v2177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:09:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:50 smithi193 ceph-mon[37473]: pgmap v2178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:09:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2549.518384 secs 2024-07-26T12:09:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2549.518384 secs 2024-07-26T12:09:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2549.518384 secs 2024-07-26T12:09:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:52 smithi134 ceph-mon[31966]: pgmap v2179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:09:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:52 smithi189 ceph-mon[36790]: pgmap v2179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:09:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:52 smithi193 ceph-mon[37473]: pgmap v2179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:09:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:54 smithi134 ceph-mon[31966]: pgmap v2180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T12:09:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2535.090920 secs 2024-07-26T12:09:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:54 smithi189 ceph-mon[36790]: pgmap v2180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T12:09:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2535.090920 secs 2024-07-26T12:09:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:54 smithi193 ceph-mon[37473]: pgmap v2180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T12:09:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2535.090920 secs 2024-07-26T12:09:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:56 smithi134 ceph-mon[31966]: pgmap v2181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:09:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2554.518546 secs 2024-07-26T12:09:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:56 smithi189 ceph-mon[36790]: pgmap v2181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:09:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2554.518546 secs 2024-07-26T12:09:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:56 smithi193 ceph-mon[37473]: pgmap v2181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:09:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2554.518546 secs 2024-07-26T12:09:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:58 smithi134 ceph-mon[31966]: pgmap v2182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T12:09:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:58 smithi189 ceph-mon[36790]: pgmap v2182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T12:09:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:58 smithi193 ceph-mon[37473]: pgmap v2182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T12:09:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:09:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2540.091048 secs 2024-07-26T12:09:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:09:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2540.091048 secs 2024-07-26T12:09:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:09:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2540.091048 secs 2024-07-26T12:10:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:10:00.376+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:10:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:10:00.376+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:10:00.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:10:00.376+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:00 smithi134 ceph-mon[31966]: pgmap v2183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:10:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:10:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:00 smithi189 ceph-mon[36790]: pgmap v2183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:10:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:10:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:00 smithi193 ceph-mon[37473]: pgmap v2183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:10:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:10:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2559.518672 secs 2024-07-26T12:10:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2559.518672 secs 2024-07-26T12:10:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2559.518672 secs 2024-07-26T12:10:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:02 smithi134 ceph-mon[31966]: pgmap v2184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:02 smithi189 ceph-mon[36790]: pgmap v2184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:02 smithi193 ceph-mon[37473]: pgmap v2184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:04 smithi134 ceph-mon[31966]: pgmap v2185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2545.091192 secs 2024-07-26T12:10:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:04 smithi189 ceph-mon[36790]: pgmap v2185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2545.091192 secs 2024-07-26T12:10:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:04 smithi193 ceph-mon[37473]: pgmap v2185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2545.091192 secs 2024-07-26T12:10:04.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:10:04.888+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:10:04.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:10:04.888+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:10:04.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:10:04.888+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:06 smithi134 ceph-mon[31966]: pgmap v2186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:10:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2564.518747 secs 2024-07-26T12:10:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:06 smithi189 ceph-mon[36790]: pgmap v2186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:10:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2564.518747 secs 2024-07-26T12:10:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:06 smithi193 ceph-mon[37473]: pgmap v2186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:10:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2564.518747 secs 2024-07-26T12:10:08.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:08 smithi134 ceph-mon[31966]: pgmap v2187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:09.496 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:08 smithi193 ceph-mon[37473]: pgmap v2187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:09.496 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:08 smithi189 ceph-mon[36790]: pgmap v2187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2550.091322 secs 2024-07-26T12:10:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2550.091322 secs 2024-07-26T12:10:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2550.091322 secs 2024-07-26T12:10:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:10 smithi189 ceph-mon[36790]: pgmap v2188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:10 smithi193 ceph-mon[37473]: pgmap v2188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:10 smithi134 ceph-mon[31966]: pgmap v2188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:11 smithi189 ceph-mon[36790]: pgmap v2189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:11.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2569.518870 secs 2024-07-26T12:10:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:11 smithi193 ceph-mon[37473]: pgmap v2189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2569.518870 secs 2024-07-26T12:10:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:11 smithi134 ceph-mon[31966]: pgmap v2189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2569.518870 secs 2024-07-26T12:10:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:13 smithi134 ceph-mon[31966]: pgmap v2190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:13 smithi189 ceph-mon[36790]: pgmap v2190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:13 smithi193 ceph-mon[37473]: pgmap v2190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2555.091452 secs 2024-07-26T12:10:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2555.091452 secs 2024-07-26T12:10:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2555.091452 secs 2024-07-26T12:10:15.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:10:15.392+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:10:15.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:10:15.392+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:10:15.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:10:15.392+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:15 smithi134 ceph-mon[31966]: pgmap v2191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:10:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:15 smithi189 ceph-mon[36790]: pgmap v2191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:10:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:15 smithi193 ceph-mon[37473]: pgmap v2191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:10:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2574.519017 secs 2024-07-26T12:10:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2574.519017 secs 2024-07-26T12:10:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2574.519017 secs 2024-07-26T12:10:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:17 smithi189 ceph-mon[36790]: pgmap v2192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:18.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:17 smithi193 ceph-mon[37473]: pgmap v2192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:17 smithi134 ceph-mon[31966]: pgmap v2192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:19.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:10:19.896+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:10:19.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:10:19.896+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:19.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:10:19.896+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:10:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2560.091581 secs 2024-07-26T12:10:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:19 smithi189 ceph-mon[36790]: pgmap v2193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2560.091581 secs 2024-07-26T12:10:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:19 smithi193 ceph-mon[37473]: pgmap v2193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2560.091581 secs 2024-07-26T12:10:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:19 smithi134 ceph-mon[31966]: pgmap v2193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:21 smithi189 ceph-mon[36790]: pgmap v2194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2579.519113 secs 2024-07-26T12:10:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:21 smithi193 ceph-mon[37473]: pgmap v2194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2579.519113 secs 2024-07-26T12:10:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:21 smithi134 ceph-mon[31966]: pgmap v2194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2579.519113 secs 2024-07-26T12:10:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:23 smithi193 ceph-mon[37473]: pgmap v2195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:23 smithi134 ceph-mon[31966]: pgmap v2195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:23 smithi189 ceph-mon[36790]: pgmap v2195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2565.091741 secs 2024-07-26T12:10:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2565.091741 secs 2024-07-26T12:10:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2565.091741 secs 2024-07-26T12:10:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:25 smithi193 ceph-mon[37473]: pgmap v2196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:10:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:25 smithi134 ceph-mon[31966]: pgmap v2196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:10:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:25 smithi189 ceph-mon[36790]: pgmap v2196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:10:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2584.519213 secs 2024-07-26T12:10:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2584.519213 secs 2024-07-26T12:10:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2584.519213 secs 2024-07-26T12:10:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:27 smithi193 ceph-mon[37473]: pgmap v2197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:27 smithi134 ceph-mon[31966]: pgmap v2197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:27 smithi189 ceph-mon[36790]: pgmap v2197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2570.091810 secs 2024-07-26T12:10:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2570.091810 secs 2024-07-26T12:10:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2570.091810 secs 2024-07-26T12:10:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:29 smithi134 ceph-mon[31966]: pgmap v2198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:29 smithi189 ceph-mon[36790]: pgmap v2198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:29 smithi193 ceph-mon[37473]: pgmap v2198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:10:30.407+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:10:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:10:30.407+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:30.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:10:30.407+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:10:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:31 smithi134 ceph-mon[31966]: pgmap v2199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2589.519348 secs 2024-07-26T12:10:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:31 smithi189 ceph-mon[36790]: pgmap v2199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2589.519348 secs 2024-07-26T12:10:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:31 smithi193 ceph-mon[37473]: pgmap v2199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2589.519348 secs 2024-07-26T12:10:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:33 smithi134 ceph-mon[31966]: pgmap v2200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2575.091879 secs 2024-07-26T12:10:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:33 smithi189 ceph-mon[36790]: pgmap v2200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2575.091879 secs 2024-07-26T12:10:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:33 smithi193 ceph-mon[37473]: pgmap v2200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2575.091879 secs 2024-07-26T12:10:34.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:10:34.911+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:10:34.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:10:34.911+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:34.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:10:34.912+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:10:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:35 smithi134 ceph-mon[31966]: pgmap v2201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:10:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:35 smithi189 ceph-mon[36790]: pgmap v2201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:10:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:35 smithi193 ceph-mon[37473]: pgmap v2201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:10:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2594.519474 secs 2024-07-26T12:10:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2594.519474 secs 2024-07-26T12:10:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2594.519474 secs 2024-07-26T12:10:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:37 smithi134 ceph-mon[31966]: pgmap v2202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:37 smithi189 ceph-mon[36790]: pgmap v2202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:37 smithi193 ceph-mon[37473]: pgmap v2202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2580.092048 secs 2024-07-26T12:10:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2580.092048 secs 2024-07-26T12:10:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2580.092048 secs 2024-07-26T12:10:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:39 smithi134 ceph-mon[31966]: pgmap v2203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:39 smithi189 ceph-mon[36790]: pgmap v2203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:39 smithi193 ceph-mon[37473]: pgmap v2203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:41 smithi134 ceph-mon[31966]: pgmap v2204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2599.519630 secs 2024-07-26T12:10:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:41 smithi189 ceph-mon[36790]: pgmap v2204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2599.519630 secs 2024-07-26T12:10:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:41 smithi193 ceph-mon[37473]: pgmap v2204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2599.519630 secs 2024-07-26T12:10:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:43 smithi134 ceph-mon[31966]: pgmap v2205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2585.092183 secs 2024-07-26T12:10:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:43 smithi189 ceph-mon[36790]: pgmap v2205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2585.092183 secs 2024-07-26T12:10:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:43 smithi193 ceph-mon[37473]: pgmap v2205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2585.092183 secs 2024-07-26T12:10:45.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:10:45.420+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:10:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:10:45.420+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:45.422 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:10:45.420+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:10:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:45 smithi134 ceph-mon[31966]: pgmap v2206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:10:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:45 smithi189 ceph-mon[36790]: pgmap v2206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:10:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:45 smithi193 ceph-mon[37473]: pgmap v2206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:10:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2604.519728 secs 2024-07-26T12:10:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2604.519728 secs 2024-07-26T12:10:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2604.519728 secs 2024-07-26T12:10:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:47 smithi134 ceph-mon[31966]: pgmap v2207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:47 smithi189 ceph-mon[36790]: pgmap v2207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:47 smithi193 ceph-mon[37473]: pgmap v2207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2590.092325 secs 2024-07-26T12:10:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:10:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2590.092325 secs 2024-07-26T12:10:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:10:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2590.092325 secs 2024-07-26T12:10:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:10:49.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:10:49.913+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:10:49.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:10:49.913+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:10:49.916 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:10:49.913+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:10:50.014 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:49 smithi193 ceph-mon[37473]: pgmap v2208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:50.014 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:10:50.014 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:10:50.014 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:10:50.014 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:10:50.069 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:49 smithi189 ceph-mon[36790]: pgmap v2208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:50.069 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:10:50.069 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:10:50.069 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:10:50.069 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:10:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:49 smithi134 ceph-mon[31966]: pgmap v2208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:10:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:10:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:10:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:10:51.062 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:50 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:10:51.062 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:50 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:10:51.106 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:50 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:10:51.107 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:50 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:10:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:50 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:10:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:50 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:10:52.111 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:51 smithi193 ceph-mon[37473]: pgmap v2209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:52.111 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2609.519882 secs 2024-07-26T12:10:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:51 smithi134 ceph-mon[31966]: pgmap v2209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2609.519882 secs 2024-07-26T12:10:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:51 smithi189 ceph-mon[36790]: pgmap v2209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:10:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2609.519882 secs 2024-07-26T12:10:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:10:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:10:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:10:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:53 smithi193 ceph-mon[37473]: pgmap v2210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:10:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:10:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:10:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:53 smithi193 ceph-mon[37473]: pgmap v2211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:10:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:53 smithi193 ceph-mon[37473]: pgmap v2212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:10:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:10:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:10:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:10:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:53 smithi134 ceph-mon[31966]: pgmap v2210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:10:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:10:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:10:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:53 smithi134 ceph-mon[31966]: pgmap v2211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:10:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:53 smithi134 ceph-mon[31966]: pgmap v2212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:10:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:10:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:10:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:10:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:53 smithi189 ceph-mon[36790]: pgmap v2210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:10:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:10:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:10:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:53 smithi189 ceph-mon[36790]: pgmap v2211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:10:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:53 smithi189 ceph-mon[36790]: pgmap v2212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:10:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:10:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2595.092427 secs 2024-07-26T12:10:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2595.092427 secs 2024-07-26T12:10:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2595.092427 secs 2024-07-26T12:10:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:55 smithi134 ceph-mon[31966]: pgmap v2213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:10:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:55 smithi189 ceph-mon[36790]: pgmap v2213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:10:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:55 smithi193 ceph-mon[37473]: pgmap v2213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:10:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2614.520014 secs 2024-07-26T12:10:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2614.520014 secs 2024-07-26T12:10:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2614.520014 secs 2024-07-26T12:10:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:57 smithi134 ceph-mon[31966]: pgmap v2214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T12:10:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:57 smithi189 ceph-mon[36790]: pgmap v2214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T12:10:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:57 smithi193 ceph-mon[37473]: pgmap v2214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T12:11:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2600.092567 secs 2024-07-26T12:11:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:10:59 smithi193 ceph-mon[37473]: pgmap v2215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:11:00.105 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2600.092567 secs 2024-07-26T12:11:00.105 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:10:59 smithi134 ceph-mon[31966]: pgmap v2215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:11:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2600.092567 secs 2024-07-26T12:11:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:10:59 smithi189 ceph-mon[36790]: pgmap v2215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:11:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:11:00.435+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:11:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:11:00.436+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:11:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:11:00.436+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:01 smithi193 ceph-mon[37473]: pgmap v2216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T12:11:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2619.520144 secs 2024-07-26T12:11:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:01 smithi134 ceph-mon[31966]: pgmap v2216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T12:11:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2619.520144 secs 2024-07-26T12:11:02.172 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:01 smithi189 ceph-mon[36790]: pgmap v2216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T12:11:02.172 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2619.520144 secs 2024-07-26T12:11:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:03 smithi193 ceph-mon[37473]: pgmap v2217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:11:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:03 smithi134 ceph-mon[31966]: pgmap v2217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:11:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:03 smithi189 ceph-mon[36790]: pgmap v2217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:11:04.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:11:04.929+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:11:04.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:11:04.929+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:04.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:11:04.929+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:11:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2605.092693 secs 2024-07-26T12:11:05.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2605.092693 secs 2024-07-26T12:11:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2605.092693 secs 2024-07-26T12:11:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:05 smithi193 ceph-mon[37473]: pgmap v2218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:11:06.107 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:05 smithi189 ceph-mon[36790]: pgmap v2218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:11:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:05 smithi134 ceph-mon[31966]: pgmap v2218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:11:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2624.520268 secs 2024-07-26T12:11:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2624.520268 secs 2024-07-26T12:11:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2624.520268 secs 2024-07-26T12:11:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:07 smithi193 ceph-mon[37473]: pgmap v2219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:07 smithi134 ceph-mon[31966]: pgmap v2219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:07 smithi189 ceph-mon[36790]: pgmap v2219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2610.092836 secs 2024-07-26T12:11:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2610.092836 secs 2024-07-26T12:11:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2610.092836 secs 2024-07-26T12:11:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:09 smithi134 ceph-mon[31966]: pgmap v2220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:09 smithi189 ceph-mon[36790]: pgmap v2220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:09 smithi193 ceph-mon[37473]: pgmap v2220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:11 smithi134 ceph-mon[31966]: pgmap v2221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2629.520409 secs 2024-07-26T12:11:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:11 smithi189 ceph-mon[36790]: pgmap v2221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2629.520409 secs 2024-07-26T12:11:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:11 smithi193 ceph-mon[37473]: pgmap v2221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2629.520409 secs 2024-07-26T12:11:14.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:13 smithi193 ceph-mon[37473]: pgmap v2222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:14.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2615.092970 secs 2024-07-26T12:11:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:13 smithi134 ceph-mon[31966]: pgmap v2222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2615.092970 secs 2024-07-26T12:11:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:13 smithi189 ceph-mon[36790]: pgmap v2222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2615.092970 secs 2024-07-26T12:11:15.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:11:15.445+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:11:15.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:11:15.445+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:11:15.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:11:15.445+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:15 smithi134 ceph-mon[31966]: pgmap v2223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:11:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:15 smithi189 ceph-mon[36790]: pgmap v2223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:11:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:15 smithi193 ceph-mon[37473]: pgmap v2223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:11:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2634.520513 secs 2024-07-26T12:11:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2634.520513 secs 2024-07-26T12:11:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2634.520513 secs 2024-07-26T12:11:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:17 smithi134 ceph-mon[31966]: pgmap v2224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:17 smithi189 ceph-mon[36790]: pgmap v2224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:17 smithi193 ceph-mon[37473]: pgmap v2224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2620.093084 secs 2024-07-26T12:11:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2620.093084 secs 2024-07-26T12:11:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2620.093084 secs 2024-07-26T12:11:19.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:11:19.939+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:11:19.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:11:19.939+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:19.943 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:11:19.939+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:11:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:19 smithi134 ceph-mon[31966]: pgmap v2225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:19 smithi189 ceph-mon[36790]: pgmap v2225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:19 smithi193 ceph-mon[37473]: pgmap v2225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:21 smithi134 ceph-mon[31966]: pgmap v2226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2639.520603 secs 2024-07-26T12:11:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:21 smithi189 ceph-mon[36790]: pgmap v2226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2639.520603 secs 2024-07-26T12:11:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:21 smithi193 ceph-mon[37473]: pgmap v2226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2639.520603 secs 2024-07-26T12:11:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:23 smithi134 ceph-mon[31966]: pgmap v2227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2625.093210 secs 2024-07-26T12:11:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:23 smithi189 ceph-mon[36790]: pgmap v2227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2625.093210 secs 2024-07-26T12:11:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:23 smithi193 ceph-mon[37473]: pgmap v2227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2625.093210 secs 2024-07-26T12:11:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:25 smithi134 ceph-mon[31966]: pgmap v2228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:11:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:25 smithi189 ceph-mon[36790]: pgmap v2228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:11:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:25 smithi193 ceph-mon[37473]: pgmap v2228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:11:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2644.520723 secs 2024-07-26T12:11:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2644.520723 secs 2024-07-26T12:11:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2644.520723 secs 2024-07-26T12:11:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:27 smithi134 ceph-mon[31966]: pgmap v2229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:27 smithi189 ceph-mon[36790]: pgmap v2229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:27 smithi193 ceph-mon[37473]: pgmap v2229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2630.093325 secs 2024-07-26T12:11:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2630.093325 secs 2024-07-26T12:11:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2630.093325 secs 2024-07-26T12:11:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:29 smithi134 ceph-mon[31966]: pgmap v2230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:29 smithi189 ceph-mon[36790]: pgmap v2230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:29 smithi193 ceph-mon[37473]: pgmap v2230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:11:30.453+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:11:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:11:30.453+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:11:30.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:11:30.453+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:31 smithi134 ceph-mon[31966]: pgmap v2231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2649.520864 secs 2024-07-26T12:11:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:31 smithi189 ceph-mon[36790]: pgmap v2231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2649.520864 secs 2024-07-26T12:11:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:31 smithi193 ceph-mon[37473]: pgmap v2231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2649.520864 secs 2024-07-26T12:11:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:33 smithi134 ceph-mon[31966]: pgmap v2232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2635.093466 secs 2024-07-26T12:11:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:33 smithi189 ceph-mon[36790]: pgmap v2232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2635.093466 secs 2024-07-26T12:11:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:33 smithi193 ceph-mon[37473]: pgmap v2232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2635.093466 secs 2024-07-26T12:11:34.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:11:34.948+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:11:34.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:11:34.948+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:11:34.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:11:34.948+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:35 smithi134 ceph-mon[31966]: pgmap v2233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:11:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:35 smithi189 ceph-mon[36790]: pgmap v2233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:11:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:35 smithi193 ceph-mon[37473]: pgmap v2233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:11:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2654.521000 secs 2024-07-26T12:11:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2654.521000 secs 2024-07-26T12:11:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2654.521000 secs 2024-07-26T12:11:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:37 smithi189 ceph-mon[36790]: pgmap v2234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:37 smithi193 ceph-mon[37473]: pgmap v2234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:37 smithi134 ceph-mon[31966]: pgmap v2234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2640.093604 secs 2024-07-26T12:11:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2640.093604 secs 2024-07-26T12:11:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2640.093604 secs 2024-07-26T12:11:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:39 smithi189 ceph-mon[36790]: pgmap v2235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:39 smithi193 ceph-mon[37473]: pgmap v2235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:39 smithi134 ceph-mon[31966]: pgmap v2235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:41 smithi193 ceph-mon[37473]: pgmap v2236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2659.521154 secs 2024-07-26T12:11:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:41 smithi134 ceph-mon[31966]: pgmap v2236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2659.521154 secs 2024-07-26T12:11:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:41 smithi189 ceph-mon[36790]: pgmap v2236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2659.521154 secs 2024-07-26T12:11:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:43 smithi193 ceph-mon[37473]: pgmap v2237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2645.093743 secs 2024-07-26T12:11:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:43 smithi134 ceph-mon[31966]: pgmap v2237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2645.093743 secs 2024-07-26T12:11:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:43 smithi189 ceph-mon[36790]: pgmap v2237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2645.093743 secs 2024-07-26T12:11:45.468 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:11:45.467+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:11:45.469 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:11:45.467+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:11:45.469 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:11:45.467+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:46.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:45 smithi134 ceph-mon[31966]: pgmap v2238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:11:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:45 smithi193 ceph-mon[37473]: pgmap v2238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:11:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:45 smithi189 ceph-mon[36790]: pgmap v2238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:11:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2664.521282 secs 2024-07-26T12:11:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2664.521282 secs 2024-07-26T12:11:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2664.521282 secs 2024-07-26T12:11:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:47 smithi193 ceph-mon[37473]: pgmap v2239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:47 smithi134 ceph-mon[31966]: pgmap v2239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:47 smithi189 ceph-mon[36790]: pgmap v2239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2650.093881 secs 2024-07-26T12:11:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2650.093881 secs 2024-07-26T12:11:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2650.093881 secs 2024-07-26T12:11:49.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:11:49.963+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:11:49.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:11:49.963+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:11:49.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:11:49.963+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:11:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:49 smithi193 ceph-mon[37473]: pgmap v2240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:49 smithi134 ceph-mon[31966]: pgmap v2240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:49 smithi189 ceph-mon[36790]: pgmap v2240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:51 smithi193 ceph-mon[37473]: pgmap v2241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2669.521403 secs 2024-07-26T12:11:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:51 smithi134 ceph-mon[31966]: pgmap v2241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2669.521403 secs 2024-07-26T12:11:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:51 smithi189 ceph-mon[36790]: pgmap v2241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:11:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2669.521403 secs 2024-07-26T12:11:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:11:53.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:11:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:11:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: pgmap v2242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2655.094020 secs 2024-07-26T12:11:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:11:54.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:11:54.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:11:54.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:11:54.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:11:54.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:11:54.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:11:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: pgmap v2242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2655.094020 secs 2024-07-26T12:11:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:11:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:11:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:11:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:11:54.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:11:54.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:11:54.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:11:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: pgmap v2242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:11:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2655.094020 secs 2024-07-26T12:11:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:11:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:11:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:11:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:11:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:11:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:11:54.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:11:54.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:11:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:55 smithi193 ceph-mon[37473]: pgmap v2243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:11:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:55 smithi193 ceph-mon[37473]: pgmap v2244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:11:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:55 smithi134 ceph-mon[31966]: pgmap v2243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:11:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:55 smithi134 ceph-mon[31966]: pgmap v2244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:11:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:55 smithi189 ceph-mon[36790]: pgmap v2243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:11:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:55 smithi189 ceph-mon[36790]: pgmap v2244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:11:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2674.521522 secs 2024-07-26T12:11:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:57 smithi193 ceph-mon[37473]: pgmap v2245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:11:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2674.521522 secs 2024-07-26T12:11:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:57 smithi134 ceph-mon[31966]: pgmap v2245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:11:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2674.521522 secs 2024-07-26T12:11:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:57 smithi189 ceph-mon[36790]: pgmap v2245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:11:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:59 smithi193 ceph-mon[37473]: pgmap v2246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:11:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:11:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2660.094144 secs 2024-07-26T12:11:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:59 smithi134 ceph-mon[31966]: pgmap v2246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:11:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:11:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2660.094144 secs 2024-07-26T12:11:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:59 smithi189 ceph-mon[36790]: pgmap v2246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:11:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:11:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2660.094144 secs 2024-07-26T12:12:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:00.482+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:12:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:00.482+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:12:00.484 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:00.482+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:01 smithi134 ceph-mon[31966]: pgmap v2247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:12:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:01 smithi189 ceph-mon[36790]: pgmap v2247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:12:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:01 smithi193 ceph-mon[37473]: pgmap v2247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:12:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2679.521644 secs 2024-07-26T12:12:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2679.521644 secs 2024-07-26T12:12:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2679.521644 secs 2024-07-26T12:12:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:03 smithi134 ceph-mon[31966]: pgmap v2248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:12:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:03 smithi189 ceph-mon[36790]: pgmap v2248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:12:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:03 smithi193 ceph-mon[37473]: pgmap v2248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:12:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2665.094268 secs 2024-07-26T12:12:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2665.094268 secs 2024-07-26T12:12:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2665.094268 secs 2024-07-26T12:12:04.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:04.979+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:12:04.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:04.979+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:12:04.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:04.979+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:05.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:05 smithi189 ceph-mon[36790]: pgmap v2249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:12:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:05 smithi134 ceph-mon[31966]: pgmap v2249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:12:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:05 smithi193 ceph-mon[37473]: pgmap v2249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:12:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2684.521763 secs 2024-07-26T12:12:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:07 smithi134 ceph-mon[31966]: pgmap v2250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2684.521763 secs 2024-07-26T12:12:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:07 smithi189 ceph-mon[36790]: pgmap v2250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2684.521763 secs 2024-07-26T12:12:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:07 smithi193 ceph-mon[37473]: pgmap v2250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:09 smithi134 ceph-mon[31966]: pgmap v2251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2670.094385 secs 2024-07-26T12:12:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:09 smithi189 ceph-mon[36790]: pgmap v2251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2670.094385 secs 2024-07-26T12:12:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:09 smithi193 ceph-mon[37473]: pgmap v2251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2670.094385 secs 2024-07-26T12:12:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:11 smithi134 ceph-mon[31966]: pgmap v2252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2689.521858 secs 2024-07-26T12:12:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:11 smithi189 ceph-mon[36790]: pgmap v2252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2689.521858 secs 2024-07-26T12:12:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:11 smithi193 ceph-mon[37473]: pgmap v2252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2689.521858 secs 2024-07-26T12:12:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:13 smithi134 ceph-mon[31966]: pgmap v2253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:13 smithi189 ceph-mon[36790]: pgmap v2253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:13 smithi193 ceph-mon[37473]: pgmap v2253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2675.094523 secs 2024-07-26T12:12:14.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2675.094523 secs 2024-07-26T12:12:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2675.094523 secs 2024-07-26T12:12:15.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:15.498+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:12:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:15.498+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:12:15.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:15.498+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:15 smithi134 ceph-mon[31966]: pgmap v2254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:15 smithi189 ceph-mon[36790]: pgmap v2254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:15 smithi193 ceph-mon[37473]: pgmap v2254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2694.522001 secs 2024-07-26T12:12:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2694.522001 secs 2024-07-26T12:12:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2694.522001 secs 2024-07-26T12:12:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:17 smithi189 ceph-mon[36790]: pgmap v2255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:17 smithi193 ceph-mon[37473]: pgmap v2255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:17 smithi134 ceph-mon[31966]: pgmap v2255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:19 smithi189 ceph-mon[36790]: pgmap v2256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2680.094671 secs 2024-07-26T12:12:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:19 smithi193 ceph-mon[37473]: pgmap v2256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2680.094671 secs 2024-07-26T12:12:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:19 smithi134 ceph-mon[31966]: pgmap v2256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2680.094671 secs 2024-07-26T12:12:19.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:19.995+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:12:19.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:19.995+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:19.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:19.995+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:12:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:21 smithi189 ceph-mon[36790]: pgmap v2257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2699.522098 secs 2024-07-26T12:12:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:21 smithi193 ceph-mon[37473]: pgmap v2257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2699.522098 secs 2024-07-26T12:12:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:21 smithi134 ceph-mon[31966]: pgmap v2257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2699.522098 secs 2024-07-26T12:12:22.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:22.292+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 msgr2=0x7f9b300c2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900001102) for more than 900000000 us, fault. 2024-07-26T12:12:22.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:22.292+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300c2120 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009990 tx=0x7f9b3802b040 comp rx=0 tx=0).stop 2024-07-26T12:12:22.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:22.292+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 msgr2=0x7f9b300c2120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:12:22.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:22.292+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300c2120 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:12:22.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:22.292+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bfb30 0x7f9b300bff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:12:22.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:22.293+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bfb30 0x7f9b300bff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:12:22.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:22.295+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bfb30 0x7f9b300bff40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b3802b040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:12:22.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:22.797+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:12:22.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:22.797+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:12:22.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:22.797+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T12:12:22.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:22.797+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T12:12:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:23 smithi189 ceph-mon[36790]: pgmap v2258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:23 smithi193 ceph-mon[37473]: pgmap v2258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:23 smithi134 ceph-mon[31966]: pgmap v2258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2685.094811 secs 2024-07-26T12:12:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2685.094811 secs 2024-07-26T12:12:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2685.094811 secs 2024-07-26T12:12:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:25 smithi193 ceph-mon[37473]: pgmap v2259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:25 smithi134 ceph-mon[31966]: pgmap v2259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:25.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:25 smithi189 ceph-mon[36790]: pgmap v2259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2704.522183 secs 2024-07-26T12:12:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:26 smithi193 ceph-mon[37473]: pgmap v2260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2704.522183 secs 2024-07-26T12:12:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:26 smithi134 ceph-mon[31966]: pgmap v2260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2704.522183 secs 2024-07-26T12:12:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:26 smithi189 ceph-mon[36790]: pgmap v2260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:28 smithi193 ceph-mon[37473]: pgmap v2261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2690.094985 secs 2024-07-26T12:12:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:28 smithi134 ceph-mon[31966]: pgmap v2261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2690.094985 secs 2024-07-26T12:12:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:28 smithi189 ceph-mon[36790]: pgmap v2261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2690.094985 secs 2024-07-26T12:12:30.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:30.508+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:12:30.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:30.508+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:30.510 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:30.508+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:12:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:30 smithi193 ceph-mon[37473]: pgmap v2262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:30 smithi134 ceph-mon[31966]: pgmap v2262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:30 smithi189 ceph-mon[36790]: pgmap v2262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2709.522322 secs 2024-07-26T12:12:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2709.522322 secs 2024-07-26T12:12:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2709.522322 secs 2024-07-26T12:12:32.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:32.846+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 msgr2=0x7f21ac0c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T12:12:32.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:32.846+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 msgr2=0x7f21ac0c1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:12:32.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:32.846+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 0x7f21ac0c1ff0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2198003470 tx=0x7f21b810a000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:12:32.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:32.846+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 0x7f21ac0c1ff0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2198003470 tx=0x7f21b810a000 comp rx=0 tx=0).stop 2024-07-26T12:12:32.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:32.846+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 msgr2=0x7f21ac0c1ff0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T12:12:32.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:32.846+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 msgr2=0x7f21ac0c1ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:12:32.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:32.846+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0b3dc0 0x7f21ac0c1ff0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:12:32.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:32.846+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bfa00 0x7f21ac0bfe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:12:32.849 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:32.847+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bfa00 0x7f21ac0bfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:12:32.850 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:32.849+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bfa00 0x7f21ac0bfe10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f21980063a0 tx=0x7f21b80aa000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:12:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:32 smithi193 ceph-mon[37473]: pgmap v2263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:33.288 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:32 smithi134 ceph-mon[31966]: pgmap v2263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:33.304 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:33.303+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T12:12:33.304 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:33.303+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:12:33.305 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:33.304+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T12:12:33.305 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:33.304+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:12:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:32 smithi189 ceph-mon[36790]: pgmap v2263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2695.095102 secs 2024-07-26T12:12:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2695.095102 secs 2024-07-26T12:12:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2695.095102 secs 2024-07-26T12:12:35.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:35.008+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:12:35.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:35.008+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:35.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:35.008+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:12:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:34 smithi193 ceph-mon[37473]: pgmap v2264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:35.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:34 smithi189 ceph-mon[36790]: pgmap v2264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:34 smithi134 ceph-mon[31966]: pgmap v2264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2714.522427 secs 2024-07-26T12:12:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:37 smithi193 ceph-mon[37473]: pgmap v2265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2714.522427 secs 2024-07-26T12:12:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:36 smithi134 ceph-mon[31966]: pgmap v2265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2714.522427 secs 2024-07-26T12:12:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:37 smithi189 ceph-mon[36790]: pgmap v2265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:39 smithi193 ceph-mon[37473]: pgmap v2266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2700.095247 secs 2024-07-26T12:12:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:39 smithi134 ceph-mon[31966]: pgmap v2266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2700.095247 secs 2024-07-26T12:12:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:39 smithi189 ceph-mon[36790]: pgmap v2266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2700.095247 secs 2024-07-26T12:12:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:41 smithi193 ceph-mon[37473]: pgmap v2267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:41 smithi134 ceph-mon[31966]: pgmap v2267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:41 smithi189 ceph-mon[36790]: pgmap v2267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2719.522548 secs 2024-07-26T12:12:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2719.522548 secs 2024-07-26T12:12:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2719.522548 secs 2024-07-26T12:12:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:43 smithi134 ceph-mon[31966]: pgmap v2268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:43 smithi189 ceph-mon[36790]: pgmap v2268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:43 smithi193 ceph-mon[37473]: pgmap v2268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2705.095357 secs 2024-07-26T12:12:44.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2705.095357 secs 2024-07-26T12:12:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2705.095357 secs 2024-07-26T12:12:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:45 smithi134 ceph-mon[31966]: pgmap v2269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:45 smithi189 ceph-mon[36790]: pgmap v2269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:45.522+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:12:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:45.522+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:45.524 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:12:45.522+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:12:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:45 smithi193 ceph-mon[37473]: pgmap v2269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2724.522664 secs 2024-07-26T12:12:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:47 smithi134 ceph-mon[31966]: pgmap v2270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2724.522664 secs 2024-07-26T12:12:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:47 smithi189 ceph-mon[36790]: pgmap v2270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2724.522664 secs 2024-07-26T12:12:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:47 smithi193 ceph-mon[37473]: pgmap v2270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:49 smithi134 ceph-mon[31966]: pgmap v2271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2710.095489 secs 2024-07-26T12:12:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:49 smithi189 ceph-mon[36790]: pgmap v2271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2710.095489 secs 2024-07-26T12:12:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:49 smithi193 ceph-mon[37473]: pgmap v2271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:12:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2710.095489 secs 2024-07-26T12:12:50.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:50.011+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:12:50.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:50.011+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:12:50.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:12:50.011+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:12:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:51 smithi134 ceph-mon[31966]: pgmap v2272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:51 smithi189 ceph-mon[36790]: pgmap v2272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:51 smithi193 ceph-mon[37473]: pgmap v2272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:12:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2729.522762 secs 2024-07-26T12:12:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2729.522762 secs 2024-07-26T12:12:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2729.522762 secs 2024-07-26T12:12:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:53 smithi134 ceph-mon[31966]: pgmap v2273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:53 smithi189 ceph-mon[36790]: pgmap v2273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:53 smithi193 ceph-mon[37473]: pgmap v2273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:12:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2715.095628 secs 2024-07-26T12:12:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2715.095628 secs 2024-07-26T12:12:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2715.095628 secs 2024-07-26T12:12:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:55 smithi134 ceph-mon[31966]: pgmap v2274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:12:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:12:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:55 smithi189 ceph-mon[36790]: pgmap v2274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:12:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:12:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:55 smithi193 ceph-mon[37473]: pgmap v2274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:12:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:12:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:12:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:12:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:56 smithi134 ceph-mon[31966]: pgmap v2275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:12:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:56 smithi134 ceph-mon[31966]: pgmap v2276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:56.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:12:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:12:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:12:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:56 smithi189 ceph-mon[36790]: pgmap v2275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:12:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:56 smithi189 ceph-mon[36790]: pgmap v2276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:12:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:12:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:12:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:56 smithi193 ceph-mon[37473]: pgmap v2275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:12:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:56 smithi193 ceph-mon[37473]: pgmap v2276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:12:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:12:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2734.522888 secs 2024-07-26T12:12:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2734.522888 secs 2024-07-26T12:12:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2734.522888 secs 2024-07-26T12:12:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:58 smithi134 ceph-mon[31966]: pgmap v2277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T12:12:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:58 smithi189 ceph-mon[36790]: pgmap v2277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T12:12:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:58 smithi193 ceph-mon[37473]: pgmap v2277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T12:12:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:12:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2720.095783 secs 2024-07-26T12:12:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:12:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2720.095783 secs 2024-07-26T12:12:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:12:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2720.095783 secs 2024-07-26T12:13:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:00 smithi134 ceph-mon[31966]: pgmap v2278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:00 smithi189 ceph-mon[36790]: pgmap v2278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:00 smithi193 ceph-mon[37473]: pgmap v2278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:00.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:13:00.528+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:13:00.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:13:00.528+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:00.530 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:13:00.528+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:13:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2739.523036 secs 2024-07-26T12:13:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:02 smithi134 ceph-mon[31966]: pgmap v2279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2739.523036 secs 2024-07-26T12:13:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:02 smithi189 ceph-mon[36790]: pgmap v2279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2739.523036 secs 2024-07-26T12:13:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:02 smithi193 ceph-mon[37473]: pgmap v2279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:13:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:04 smithi134 ceph-mon[31966]: pgmap v2280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:13:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2725.095906 secs 2024-07-26T12:13:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:04 smithi189 ceph-mon[36790]: pgmap v2280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:13:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2725.095906 secs 2024-07-26T12:13:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:04 smithi193 ceph-mon[37473]: pgmap v2280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:13:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2725.095906 secs 2024-07-26T12:13:05.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:13:05.024+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:13:05.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:13:05.024+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:13:05.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:13:05.024+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:06 smithi134 ceph-mon[31966]: pgmap v2281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:13:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:06 smithi189 ceph-mon[36790]: pgmap v2281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:13:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:06 smithi193 ceph-mon[37473]: pgmap v2281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:13:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2744.523155 secs 2024-07-26T12:13:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2744.523155 secs 2024-07-26T12:13:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2744.523155 secs 2024-07-26T12:13:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:08 smithi134 ceph-mon[31966]: pgmap v2282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:08 smithi189 ceph-mon[36790]: pgmap v2282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:08 smithi193 ceph-mon[37473]: pgmap v2282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2730.096036 secs 2024-07-26T12:13:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2730.096036 secs 2024-07-26T12:13:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2730.096036 secs 2024-07-26T12:13:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:10 smithi189 ceph-mon[36790]: pgmap v2283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:10 smithi193 ceph-mon[37473]: pgmap v2283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:10 smithi134 ceph-mon[31966]: pgmap v2283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2749.523281 secs 2024-07-26T12:13:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:12 smithi189 ceph-mon[36790]: pgmap v2284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:12.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2749.523281 secs 2024-07-26T12:13:12.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:12 smithi193 ceph-mon[37473]: pgmap v2284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2749.523281 secs 2024-07-26T12:13:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:12 smithi134 ceph-mon[31966]: pgmap v2284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:14 smithi189 ceph-mon[36790]: pgmap v2285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2735.096164 secs 2024-07-26T12:13:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:14 smithi193 ceph-mon[37473]: pgmap v2285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2735.096164 secs 2024-07-26T12:13:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:14 smithi134 ceph-mon[31966]: pgmap v2285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2735.096164 secs 2024-07-26T12:13:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:13:15.537+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:13:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:13:15.537+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:13:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:13:15.538+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:16 smithi189 ceph-mon[36790]: pgmap v2286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:16 smithi193 ceph-mon[37473]: pgmap v2286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:16 smithi134 ceph-mon[31966]: pgmap v2286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2754.523409 secs 2024-07-26T12:13:17.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2754.523409 secs 2024-07-26T12:13:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2754.523409 secs 2024-07-26T12:13:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:18 smithi193 ceph-mon[37473]: pgmap v2287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:18 smithi134 ceph-mon[31966]: pgmap v2287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:18 smithi189 ceph-mon[36790]: pgmap v2287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2740.096306 secs 2024-07-26T12:13:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2740.096306 secs 2024-07-26T12:13:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2740.096306 secs 2024-07-26T12:13:20.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:13:20.036+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:13:20.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:13:20.036+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:13:20.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:13:20.036+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:20 smithi193 ceph-mon[37473]: pgmap v2288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:20 smithi134 ceph-mon[31966]: pgmap v2288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:20 smithi189 ceph-mon[36790]: pgmap v2288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2759.523534 secs 2024-07-26T12:13:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:22 smithi193 ceph-mon[37473]: pgmap v2289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2759.523534 secs 2024-07-26T12:13:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:22 smithi134 ceph-mon[31966]: pgmap v2289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2759.523534 secs 2024-07-26T12:13:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:22 smithi189 ceph-mon[36790]: pgmap v2289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:24 smithi193 ceph-mon[37473]: pgmap v2290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2745.096390 secs 2024-07-26T12:13:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:24 smithi134 ceph-mon[31966]: pgmap v2290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2745.096390 secs 2024-07-26T12:13:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:24 smithi189 ceph-mon[36790]: pgmap v2290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2745.096390 secs 2024-07-26T12:13:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:26 smithi193 ceph-mon[37473]: pgmap v2291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:26 smithi134 ceph-mon[31966]: pgmap v2291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:26 smithi189 ceph-mon[36790]: pgmap v2291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2764.523680 secs 2024-07-26T12:13:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2764.523680 secs 2024-07-26T12:13:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2764.523680 secs 2024-07-26T12:13:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:28 smithi193 ceph-mon[37473]: pgmap v2292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:28 smithi134 ceph-mon[31966]: pgmap v2292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:28 smithi189 ceph-mon[36790]: pgmap v2292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2750.096510 secs 2024-07-26T12:13:29.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2750.096510 secs 2024-07-26T12:13:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2750.096510 secs 2024-07-26T12:13:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:30 smithi193 ceph-mon[37473]: pgmap v2293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:13:30.544+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:13:30.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:13:30.544+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:30.546 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:13:30.544+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:13:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:30 smithi134 ceph-mon[31966]: pgmap v2293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:30 smithi189 ceph-mon[36790]: pgmap v2293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2769.523800 secs 2024-07-26T12:13:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:32 smithi193 ceph-mon[37473]: pgmap v2294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2769.523800 secs 2024-07-26T12:13:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:32 smithi134 ceph-mon[31966]: pgmap v2294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2769.523800 secs 2024-07-26T12:13:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:32 smithi189 ceph-mon[36790]: pgmap v2294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:34 smithi134 ceph-mon[31966]: pgmap v2295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2755.096668 secs 2024-07-26T12:13:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:34 smithi189 ceph-mon[36790]: pgmap v2295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2755.096668 secs 2024-07-26T12:13:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:34 smithi193 ceph-mon[37473]: pgmap v2295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2755.096668 secs 2024-07-26T12:13:35.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:13:35.049+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:13:35.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:13:35.049+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:35.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:13:35.049+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:13:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:36 smithi134 ceph-mon[31966]: pgmap v2296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:36 smithi189 ceph-mon[36790]: pgmap v2296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:36 smithi193 ceph-mon[37473]: pgmap v2296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2774.523864 secs 2024-07-26T12:13:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2774.523864 secs 2024-07-26T12:13:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2774.523864 secs 2024-07-26T12:13:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:38 smithi134 ceph-mon[31966]: pgmap v2297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:38 smithi189 ceph-mon[36790]: pgmap v2297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:38 smithi193 ceph-mon[37473]: pgmap v2297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2760.096820 secs 2024-07-26T12:13:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2760.096820 secs 2024-07-26T12:13:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2760.096820 secs 2024-07-26T12:13:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:40 smithi134 ceph-mon[31966]: pgmap v2298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:40 smithi189 ceph-mon[36790]: pgmap v2298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:40 smithi193 ceph-mon[37473]: pgmap v2298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2779.524012 secs 2024-07-26T12:13:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2779.524012 secs 2024-07-26T12:13:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2779.524012 secs 2024-07-26T12:13:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:42 smithi134 ceph-mon[31966]: pgmap v2299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:42 smithi189 ceph-mon[36790]: pgmap v2299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:42 smithi193 ceph-mon[37473]: pgmap v2299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:44 smithi134 ceph-mon[31966]: pgmap v2300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2765.096989 secs 2024-07-26T12:13:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:44 smithi189 ceph-mon[36790]: pgmap v2300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2765.096989 secs 2024-07-26T12:13:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:44 smithi193 ceph-mon[37473]: pgmap v2300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2765.096989 secs 2024-07-26T12:13:45.559 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:13:45.558+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:13:45.559 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:13:45.558+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:13:45.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:13:45.558+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:46 smithi189 ceph-mon[36790]: pgmap v2301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2784.524155 secs 2024-07-26T12:13:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:46 smithi193 ceph-mon[37473]: pgmap v2301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2784.524155 secs 2024-07-26T12:13:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:46 smithi134 ceph-mon[31966]: pgmap v2301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2784.524155 secs 2024-07-26T12:13:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:48 smithi189 ceph-mon[36790]: pgmap v2302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:48.724 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:48 smithi134 ceph-mon[31966]: pgmap v2302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:48 smithi193 ceph-mon[37473]: pgmap v2302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:13:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2770.097129 secs 2024-07-26T12:13:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2770.097129 secs 2024-07-26T12:13:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2770.097129 secs 2024-07-26T12:13:50.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:13:50.063+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:13:50.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:13:50.063+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:13:50.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:13:50.063+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:13:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:50 smithi193 ceph-mon[37473]: pgmap v2303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:50 smithi134 ceph-mon[31966]: pgmap v2303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:50 smithi189 ceph-mon[36790]: pgmap v2303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:13:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2789.524286 secs 2024-07-26T12:13:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2789.524286 secs 2024-07-26T12:13:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2789.524286 secs 2024-07-26T12:13:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:52 smithi193 ceph-mon[37473]: pgmap v2304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:52 smithi134 ceph-mon[31966]: pgmap v2304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:52.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:52 smithi189 ceph-mon[36790]: pgmap v2304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:54 smithi193 ceph-mon[37473]: pgmap v2305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2775.097206 secs 2024-07-26T12:13:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:54 smithi134 ceph-mon[31966]: pgmap v2305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2775.097206 secs 2024-07-26T12:13:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:54 smithi189 ceph-mon[36790]: pgmap v2305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:13:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2775.097206 secs 2024-07-26T12:13:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:55 smithi193 ceph-mon[37473]: pgmap v2306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:13:55.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:55 smithi134 ceph-mon[31966]: pgmap v2306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:13:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:55 smithi189 ceph-mon[36790]: pgmap v2306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:13:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:13:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2794.524332 secs 2024-07-26T12:13:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:13:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:13:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2794.524332 secs 2024-07-26T12:13:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:13:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:13:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2794.524332 secs 2024-07-26T12:13:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:13:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:13:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:13:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:13:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:57 smithi193 ceph-mon[37473]: pgmap v2307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T12:13:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:13:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:13:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:13:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:57 smithi134 ceph-mon[31966]: pgmap v2307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T12:13:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:13:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:13:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:13:58.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:57 smithi189 ceph-mon[36790]: pgmap v2307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T12:13:58.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:13:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2780.097290 secs 2024-07-26T12:13:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2780.097290 secs 2024-07-26T12:13:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2780.097290 secs 2024-07-26T12:14:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:13:59 smithi193 ceph-mon[37473]: pgmap v2308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T12:14:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:13:59 smithi134 ceph-mon[31966]: pgmap v2308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T12:14:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:13:59 smithi189 ceph-mon[36790]: pgmap v2308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T12:14:00.565 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:14:00.564+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:14:00.566 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:14:00.564+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:00.566 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:14:00.564+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:14:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2799.524407 secs 2024-07-26T12:14:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:01 smithi193 ceph-mon[37473]: pgmap v2309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:14:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2799.524407 secs 2024-07-26T12:14:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:01 smithi134 ceph-mon[31966]: pgmap v2309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:14:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2799.524407 secs 2024-07-26T12:14:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:01 smithi189 ceph-mon[36790]: pgmap v2309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:14:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:03 smithi193 ceph-mon[37473]: pgmap v2310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T12:14:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2785.097375 secs 2024-07-26T12:14:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:03 smithi134 ceph-mon[31966]: pgmap v2310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T12:14:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2785.097375 secs 2024-07-26T12:14:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:03 smithi189 ceph-mon[36790]: pgmap v2310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T12:14:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2785.097375 secs 2024-07-26T12:14:05.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:14:05.074+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:14:05.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:14:05.074+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:14:05.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:14:05.074+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:06.245 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:05 smithi189 ceph-mon[36790]: pgmap v2311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:14:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:05 smithi193 ceph-mon[37473]: pgmap v2311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:14:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:05 smithi134 ceph-mon[31966]: pgmap v2311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:14:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2804.524491 secs 2024-07-26T12:14:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2804.524491 secs 2024-07-26T12:14:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2804.524491 secs 2024-07-26T12:14:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:08 smithi193 ceph-mon[37473]: pgmap v2312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T12:14:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:08 smithi134 ceph-mon[31966]: pgmap v2312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T12:14:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:08 smithi189 ceph-mon[36790]: pgmap v2312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T12:14:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2790.097456 secs 2024-07-26T12:14:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2790.097456 secs 2024-07-26T12:14:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2790.097456 secs 2024-07-26T12:14:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:10 smithi193 ceph-mon[37473]: pgmap v2313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:10 smithi134 ceph-mon[31966]: pgmap v2313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:10 smithi189 ceph-mon[36790]: pgmap v2313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2809.524560 secs 2024-07-26T12:14:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:12 smithi134 ceph-mon[31966]: pgmap v2314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2809.524560 secs 2024-07-26T12:14:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:12 smithi189 ceph-mon[36790]: pgmap v2314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2809.524560 secs 2024-07-26T12:14:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:12 smithi193 ceph-mon[37473]: pgmap v2314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:14.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:14 smithi193 ceph-mon[37473]: pgmap v2315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:14.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2795.097576 secs 2024-07-26T12:14:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:14 smithi134 ceph-mon[31966]: pgmap v2315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2795.097576 secs 2024-07-26T12:14:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:14 smithi189 ceph-mon[36790]: pgmap v2315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2795.097576 secs 2024-07-26T12:14:15.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:14:15.579+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:14:15.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:14:15.580+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:14:15.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:14:15.580+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:16 smithi134 ceph-mon[31966]: pgmap v2316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:16 smithi189 ceph-mon[36790]: pgmap v2316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:16 smithi193 ceph-mon[37473]: pgmap v2316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2814.524676 secs 2024-07-26T12:14:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2814.524676 secs 2024-07-26T12:14:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2814.524676 secs 2024-07-26T12:14:18.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:18 smithi134 ceph-mon[31966]: pgmap v2317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:18 smithi189 ceph-mon[36790]: pgmap v2317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:18 smithi193 ceph-mon[37473]: pgmap v2317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2800.097676 secs 2024-07-26T12:14:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2800.097676 secs 2024-07-26T12:14:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2800.097676 secs 2024-07-26T12:14:20.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:14:20.089+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:14:20.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:14:20.089+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:20.091 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:14:20.089+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:14:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:20 smithi134 ceph-mon[31966]: pgmap v2318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:20 smithi189 ceph-mon[36790]: pgmap v2318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:20 smithi193 ceph-mon[37473]: pgmap v2318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:22.345 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2819.524769 secs 2024-07-26T12:14:22.345 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:22 smithi134 ceph-mon[31966]: pgmap v2319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2819.524769 secs 2024-07-26T12:14:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:22 smithi189 ceph-mon[36790]: pgmap v2319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2819.524769 secs 2024-07-26T12:14:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:22 smithi193 ceph-mon[37473]: pgmap v2319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:24 smithi134 ceph-mon[31966]: pgmap v2320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2805.097796 secs 2024-07-26T12:14:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:24 smithi189 ceph-mon[36790]: pgmap v2320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2805.097796 secs 2024-07-26T12:14:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:24 smithi193 ceph-mon[37473]: pgmap v2320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2805.097796 secs 2024-07-26T12:14:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:26 smithi134 ceph-mon[31966]: pgmap v2321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:26 smithi189 ceph-mon[36790]: pgmap v2321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:26 smithi193 ceph-mon[37473]: pgmap v2321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2824.524887 secs 2024-07-26T12:14:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2824.524887 secs 2024-07-26T12:14:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2824.524887 secs 2024-07-26T12:14:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:28 smithi134 ceph-mon[31966]: pgmap v2322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:28 smithi189 ceph-mon[36790]: pgmap v2322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:28 smithi193 ceph-mon[37473]: pgmap v2322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2810.097923 secs 2024-07-26T12:14:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2810.097923 secs 2024-07-26T12:14:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2810.097923 secs 2024-07-26T12:14:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:30 smithi134 ceph-mon[31966]: pgmap v2323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:30 smithi189 ceph-mon[36790]: pgmap v2323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:30 smithi193 ceph-mon[37473]: pgmap v2323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:14:30.596+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:14:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:14:30.596+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:30.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:14:30.596+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:14:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2829.525010 secs 2024-07-26T12:14:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:32 smithi134 ceph-mon[31966]: pgmap v2324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2829.525010 secs 2024-07-26T12:14:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:32 smithi189 ceph-mon[36790]: pgmap v2324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2829.525010 secs 2024-07-26T12:14:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:32 smithi193 ceph-mon[37473]: pgmap v2324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:34 smithi134 ceph-mon[31966]: pgmap v2325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2815.098067 secs 2024-07-26T12:14:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:34 smithi189 ceph-mon[36790]: pgmap v2325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2815.098067 secs 2024-07-26T12:14:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:34 smithi193 ceph-mon[37473]: pgmap v2325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2815.098067 secs 2024-07-26T12:14:35.101 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:14:35.100+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:14:35.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:14:35.100+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:14:35.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:14:35.100+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:36 smithi134 ceph-mon[31966]: pgmap v2326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:36 smithi189 ceph-mon[36790]: pgmap v2326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:36 smithi193 ceph-mon[37473]: pgmap v2326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2834.525128 secs 2024-07-26T12:14:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2834.525128 secs 2024-07-26T12:14:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2834.525128 secs 2024-07-26T12:14:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:38 smithi134 ceph-mon[31966]: pgmap v2327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:38 smithi189 ceph-mon[36790]: pgmap v2327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:38 smithi193 ceph-mon[37473]: pgmap v2327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2820.098211 secs 2024-07-26T12:14:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2820.098211 secs 2024-07-26T12:14:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2820.098211 secs 2024-07-26T12:14:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:40 smithi134 ceph-mon[31966]: pgmap v2328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:40 smithi189 ceph-mon[36790]: pgmap v2328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:40 smithi193 ceph-mon[37473]: pgmap v2328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2839.525268 secs 2024-07-26T12:14:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:42 smithi134 ceph-mon[31966]: pgmap v2329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2839.525268 secs 2024-07-26T12:14:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:42 smithi189 ceph-mon[36790]: pgmap v2329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2839.525268 secs 2024-07-26T12:14:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:42 smithi193 ceph-mon[37473]: pgmap v2329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:44 smithi189 ceph-mon[36790]: pgmap v2330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2825.098341 secs 2024-07-26T12:14:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:44 smithi193 ceph-mon[37473]: pgmap v2330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2825.098341 secs 2024-07-26T12:14:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:44 smithi134 ceph-mon[31966]: pgmap v2330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2825.098341 secs 2024-07-26T12:14:45.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:14:45.611+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:14:45.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:14:45.611+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:14:45.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:14:45.611+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:46 smithi189 ceph-mon[36790]: pgmap v2331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:46 smithi193 ceph-mon[37473]: pgmap v2331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:46 smithi134 ceph-mon[31966]: pgmap v2331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2844.525412 secs 2024-07-26T12:14:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2844.525412 secs 2024-07-26T12:14:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2844.525412 secs 2024-07-26T12:14:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:48 smithi193 ceph-mon[37473]: pgmap v2332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:48.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:48 smithi134 ceph-mon[31966]: pgmap v2332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:48 smithi189 ceph-mon[36790]: pgmap v2332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2830.098470 secs 2024-07-26T12:14:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2830.098470 secs 2024-07-26T12:14:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2830.098470 secs 2024-07-26T12:14:50.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:14:50.115+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:14:50.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:14:50.115+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:14:50.117 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:14:50.115+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:14:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:50 smithi134 ceph-mon[31966]: pgmap v2333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:50 smithi193 ceph-mon[37473]: pgmap v2333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:50 smithi189 ceph-mon[36790]: pgmap v2333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2849.525535 secs 2024-07-26T12:14:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:52 smithi193 ceph-mon[37473]: pgmap v2334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2849.525535 secs 2024-07-26T12:14:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:52 smithi134 ceph-mon[31966]: pgmap v2334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2849.525535 secs 2024-07-26T12:14:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:52 smithi189 ceph-mon[36790]: pgmap v2334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:54 smithi134 ceph-mon[31966]: pgmap v2335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2835.098529 secs 2024-07-26T12:14:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:54 smithi193 ceph-mon[37473]: pgmap v2335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2835.098529 secs 2024-07-26T12:14:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:54 smithi189 ceph-mon[36790]: pgmap v2335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:14:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2835.098529 secs 2024-07-26T12:14:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:56 smithi193 ceph-mon[37473]: pgmap v2336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:56 smithi134 ceph-mon[31966]: pgmap v2336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:56 smithi189 ceph-mon[36790]: pgmap v2336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:14:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2854.525656 secs 2024-07-26T12:14:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:14:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2854.525656 secs 2024-07-26T12:14:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:14:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2854.525656 secs 2024-07-26T12:14:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:14:58.489 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:58 smithi134 ceph-mon[31966]: pgmap v2337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:58.490 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:58.490 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:58.490 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:58.490 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:58 smithi193 ceph-mon[37473]: pgmap v2337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:58 smithi189 ceph-mon[36790]: pgmap v2337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:14:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2840.098611 secs 2024-07-26T12:14:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:14:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2840.098611 secs 2024-07-26T12:14:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:14:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2840.098611 secs 2024-07-26T12:14:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:14:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:14:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:15:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:00 smithi193 ceph-mon[37473]: pgmap v2338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:15:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:15:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:15:00.624 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:15:00.622+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:15:00.624 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:15:00.622+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:15:00.624 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:15:00.622+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:00 smithi134 ceph-mon[31966]: pgmap v2338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:15:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:15:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:15:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:00 smithi189 ceph-mon[36790]: pgmap v2338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:15:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:15:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:15:02.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2859.525779 secs 2024-07-26T12:15:02.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:02 smithi189 ceph-mon[36790]: pgmap v2339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:15:02.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:15:02.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:15:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2859.525779 secs 2024-07-26T12:15:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:02 smithi134 ceph-mon[31966]: pgmap v2339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:15:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:15:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:15:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2859.525779 secs 2024-07-26T12:15:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:02 smithi193 ceph-mon[37473]: pgmap v2339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:15:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:15:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:15:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:15:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:15:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:03 smithi134 ceph-mon[31966]: pgmap v2340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:15:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:03 smithi134 ceph-mon[31966]: pgmap v2341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-07-26T12:15:03.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:15:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:15:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:15:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:03 smithi189 ceph-mon[36790]: pgmap v2340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:15:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:03 smithi189 ceph-mon[36790]: pgmap v2341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-07-26T12:15:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:15:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:15:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:15:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:03 smithi193 ceph-mon[37473]: pgmap v2340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T12:15:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:03 smithi193 ceph-mon[37473]: pgmap v2341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-07-26T12:15:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:15:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2845.098737 secs 2024-07-26T12:15:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2845.098737 secs 2024-07-26T12:15:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2845.098737 secs 2024-07-26T12:15:05.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:15:05.117+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:15:05.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:15:05.117+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:15:05.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:15:05.117+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:05 smithi134 ceph-mon[31966]: pgmap v2342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:15:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:05 smithi189 ceph-mon[36790]: pgmap v2342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:15:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:05 smithi193 ceph-mon[37473]: pgmap v2342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:15:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2864.525901 secs 2024-07-26T12:15:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:07 smithi134 ceph-mon[31966]: pgmap v2343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:15:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2864.525901 secs 2024-07-26T12:15:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:07 smithi189 ceph-mon[36790]: pgmap v2343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:15:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2864.525901 secs 2024-07-26T12:15:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:07 smithi193 ceph-mon[37473]: pgmap v2343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:15:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:09 smithi134 ceph-mon[31966]: pgmap v2344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:15:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2850.098866 secs 2024-07-26T12:15:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:09 smithi189 ceph-mon[36790]: pgmap v2344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:15:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2850.098866 secs 2024-07-26T12:15:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:09 smithi193 ceph-mon[37473]: pgmap v2344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:15:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2850.098866 secs 2024-07-26T12:15:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:11 smithi134 ceph-mon[31966]: pgmap v2345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:15:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:11 smithi189 ceph-mon[36790]: pgmap v2345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:15:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:11 smithi193 ceph-mon[37473]: pgmap v2345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:15:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2869.526014 secs 2024-07-26T12:15:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2869.526014 secs 2024-07-26T12:15:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2869.526014 secs 2024-07-26T12:15:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:13 smithi134 ceph-mon[31966]: pgmap v2346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:15:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:13 smithi189 ceph-mon[36790]: pgmap v2346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:15:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:13 smithi193 ceph-mon[37473]: pgmap v2346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:15:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2855.099014 secs 2024-07-26T12:15:14.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2855.099014 secs 2024-07-26T12:15:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2855.099014 secs 2024-07-26T12:15:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:15 smithi134 ceph-mon[31966]: pgmap v2347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:15 smithi189 ceph-mon[36790]: pgmap v2347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:15 smithi193 ceph-mon[37473]: pgmap v2347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:15.638 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:15:15.637+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:15:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:15:15.637+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:15.639 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:15:15.637+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:15:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2874.526124 secs 2024-07-26T12:15:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:17 smithi134 ceph-mon[31966]: pgmap v2348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2874.526124 secs 2024-07-26T12:15:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:17 smithi189 ceph-mon[36790]: pgmap v2348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2874.526124 secs 2024-07-26T12:15:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:17 smithi193 ceph-mon[37473]: pgmap v2348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:19 smithi134 ceph-mon[31966]: pgmap v2349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2860.099134 secs 2024-07-26T12:15:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:19 smithi189 ceph-mon[36790]: pgmap v2349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2860.099134 secs 2024-07-26T12:15:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:19 smithi193 ceph-mon[37473]: pgmap v2349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2860.099134 secs 2024-07-26T12:15:20.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:15:20.118+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:15:20.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:15:20.118+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:15:20.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:15:20.118+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:21 smithi134 ceph-mon[31966]: pgmap v2350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:21 smithi189 ceph-mon[36790]: pgmap v2350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:21 smithi193 ceph-mon[37473]: pgmap v2350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2879.526245 secs 2024-07-26T12:15:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2879.526245 secs 2024-07-26T12:15:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2879.526245 secs 2024-07-26T12:15:22.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:15:22.827+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:15:22.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:15:22.827+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:15:22.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:15:22.828+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:15:22.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:15:22.829+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:15:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:23 smithi134 ceph-mon[31966]: pgmap v2351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:15:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:23 smithi189 ceph-mon[36790]: pgmap v2351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:15:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:23 smithi193 ceph-mon[37473]: pgmap v2351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:15:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2865.099235 secs 2024-07-26T12:15:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2865.099235 secs 2024-07-26T12:15:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2865.099235 secs 2024-07-26T12:15:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:25 smithi134 ceph-mon[31966]: pgmap v2352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:15:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:25 smithi189 ceph-mon[36790]: pgmap v2352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:15:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:25 smithi193 ceph-mon[37473]: pgmap v2352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:15:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:27 smithi134 ceph-mon[31966]: 1 slow requests, 1 included below; oldest blocked for > 2884.526335 secs 2024-07-26T12:15:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:27 smithi134 ceph-mon[31966]: slow request 2884.526334 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T12:15:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:27 smithi134 ceph-mon[31966]: pgmap v2353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:27 smithi189 ceph-mon[36790]: 1 slow requests, 1 included below; oldest blocked for > 2884.526335 secs 2024-07-26T12:15:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:27 smithi189 ceph-mon[36790]: slow request 2884.526334 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T12:15:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:27 smithi189 ceph-mon[36790]: pgmap v2353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:27.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:27 smithi193 ceph-mon[37473]: 1 slow requests, 1 included below; oldest blocked for > 2884.526335 secs 2024-07-26T12:15:27.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:27 smithi193 ceph-mon[37473]: slow request 2884.526334 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T12:15:27.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:27 smithi193 ceph-mon[37473]: pgmap v2353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:29 smithi134 ceph-mon[31966]: pgmap v2354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2870.099337 secs 2024-07-26T12:15:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:29 smithi189 ceph-mon[36790]: pgmap v2354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2870.099337 secs 2024-07-26T12:15:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:29 smithi193 ceph-mon[37473]: pgmap v2354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2870.099337 secs 2024-07-26T12:15:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:15:30.640+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:15:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:15:30.640+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:30.642 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:15:30.640+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:15:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:31 smithi134 ceph-mon[31966]: pgmap v2355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:31 smithi189 ceph-mon[36790]: pgmap v2355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:31 smithi193 ceph-mon[37473]: pgmap v2355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2889.526451 secs 2024-07-26T12:15:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2889.526451 secs 2024-07-26T12:15:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2889.526451 secs 2024-07-26T12:15:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:15:33.335+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T12:15:33.336 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:15:33.335+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:15:33.337 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:15:33.336+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T12:15:33.337 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:15:33.336+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:15:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:33 smithi189 ceph-mon[36790]: pgmap v2356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:15:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:33 smithi193 ceph-mon[37473]: pgmap v2356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:15:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:33 smithi134 ceph-mon[31966]: pgmap v2356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:15:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2875.099480 secs 2024-07-26T12:15:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2875.099480 secs 2024-07-26T12:15:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2875.099480 secs 2024-07-26T12:15:35.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:15:35.134+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:15:35.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:15:35.134+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:15:35.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:15:35.134+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:35 smithi134 ceph-mon[31966]: pgmap v2357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:15:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:35 smithi193 ceph-mon[37473]: pgmap v2357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:15:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:35 smithi189 ceph-mon[36790]: pgmap v2357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:15:37.495 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2894.526561 secs 2024-07-26T12:15:37.496 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:37 smithi189 ceph-mon[36790]: pgmap v2358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2894.526561 secs 2024-07-26T12:15:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:37 smithi193 ceph-mon[37473]: pgmap v2358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2894.526561 secs 2024-07-26T12:15:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:37 smithi134 ceph-mon[31966]: pgmap v2358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:39 smithi193 ceph-mon[37473]: pgmap v2359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:39 smithi193 ceph-mon[37473]: 2 slow requests, 1 included below; oldest blocked for > 2880.099567 secs 2024-07-26T12:15:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:39 smithi193 ceph-mon[37473]: slow request 2880.099566 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T12:15:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:39 smithi134 ceph-mon[31966]: pgmap v2359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:39 smithi134 ceph-mon[31966]: 2 slow requests, 1 included below; oldest blocked for > 2880.099567 secs 2024-07-26T12:15:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:39 smithi134 ceph-mon[31966]: slow request 2880.099566 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T12:15:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:39 smithi189 ceph-mon[36790]: pgmap v2359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:39 smithi189 ceph-mon[36790]: 2 slow requests, 1 included below; oldest blocked for > 2880.099567 secs 2024-07-26T12:15:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:39 smithi189 ceph-mon[36790]: slow request 2880.099566 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T12:15:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:41 smithi193 ceph-mon[37473]: pgmap v2360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:41 smithi134 ceph-mon[31966]: pgmap v2360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:41 smithi189 ceph-mon[36790]: pgmap v2360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2899.526681 secs 2024-07-26T12:15:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2899.526681 secs 2024-07-26T12:15:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2899.526681 secs 2024-07-26T12:15:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:43 smithi193 ceph-mon[37473]: pgmap v2361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:15:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:43 smithi134 ceph-mon[31966]: pgmap v2361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:15:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:43 smithi189 ceph-mon[36790]: pgmap v2361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:15:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2885.099706 secs 2024-07-26T12:15:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2885.099706 secs 2024-07-26T12:15:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2885.099706 secs 2024-07-26T12:15:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:45 smithi193 ceph-mon[37473]: pgmap v2362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:15:45.657 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:15:45.656+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:15:45.658 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:15:45.656+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:45.658 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:15:45.656+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:15:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:45 smithi134 ceph-mon[31966]: pgmap v2362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:15:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:45 smithi189 ceph-mon[36790]: pgmap v2362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:15:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2904.526803 secs 2024-07-26T12:15:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:47 smithi193 ceph-mon[37473]: pgmap v2363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2904.526803 secs 2024-07-26T12:15:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:47 smithi134 ceph-mon[31966]: pgmap v2363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2904.526803 secs 2024-07-26T12:15:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:47 smithi189 ceph-mon[36790]: pgmap v2363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:49 smithi193 ceph-mon[37473]: pgmap v2364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2890.099806 secs 2024-07-26T12:15:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:49 smithi134 ceph-mon[31966]: pgmap v2364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2890.099806 secs 2024-07-26T12:15:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:49 smithi189 ceph-mon[36790]: pgmap v2364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:15:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2890.099806 secs 2024-07-26T12:15:50.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:15:50.150+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:15:50.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:15:50.150+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:15:50.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:15:50.150+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:15:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:51 smithi193 ceph-mon[37473]: pgmap v2365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:51 smithi134 ceph-mon[31966]: pgmap v2365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:51 smithi189 ceph-mon[36790]: pgmap v2365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:15:52.657 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2909.526931 secs 2024-07-26T12:15:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2909.526931 secs 2024-07-26T12:15:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2909.526931 secs 2024-07-26T12:15:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:53 smithi134 ceph-mon[31966]: pgmap v2366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:15:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:53 smithi189 ceph-mon[36790]: pgmap v2366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:15:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:53 smithi193 ceph-mon[37473]: pgmap v2366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:15:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2895.099884 secs 2024-07-26T12:15:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2895.099884 secs 2024-07-26T12:15:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2895.099884 secs 2024-07-26T12:15:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:55 smithi134 ceph-mon[31966]: pgmap v2367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:15:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:55 smithi189 ceph-mon[36790]: pgmap v2367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:15:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:55 smithi193 ceph-mon[37473]: pgmap v2367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:15:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2914.527064 secs 2024-07-26T12:15:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:57 smithi134 ceph-mon[31966]: pgmap v2368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:15:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2914.527064 secs 2024-07-26T12:15:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:57 smithi189 ceph-mon[36790]: pgmap v2368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:15:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2914.527064 secs 2024-07-26T12:15:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:57 smithi193 ceph-mon[37473]: pgmap v2368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:15:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:59 smithi134 ceph-mon[31966]: pgmap v2369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:15:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:15:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2900.099956 secs 2024-07-26T12:15:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:59 smithi189 ceph-mon[36790]: pgmap v2369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:15:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:15:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2900.099956 secs 2024-07-26T12:15:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:59 smithi193 ceph-mon[37473]: pgmap v2369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:15:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:15:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2900.099956 secs 2024-07-26T12:16:00.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:16:00.672+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:16:00.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:16:00.672+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:00.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:16:00.672+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:16:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:01 smithi134 ceph-mon[31966]: pgmap v2370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:16:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:01 smithi189 ceph-mon[36790]: pgmap v2370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:16:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:01 smithi193 ceph-mon[37473]: pgmap v2370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:16:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2919.527144 secs 2024-07-26T12:16:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:16:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2919.527144 secs 2024-07-26T12:16:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:16:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2919.527144 secs 2024-07-26T12:16:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:16:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:03 smithi134 ceph-mon[31966]: pgmap v2371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:16:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:16:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:16:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:16:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:03 smithi189 ceph-mon[36790]: pgmap v2371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:16:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:16:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:16:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:16:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:03 smithi193 ceph-mon[37473]: pgmap v2371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:16:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:16:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:16:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:16:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:04 smithi134 ceph-mon[31966]: pgmap v2372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T12:16:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:04 smithi134 ceph-mon[31966]: 2 slow requests, 1 included below; oldest blocked for > 2905.100036 secs 2024-07-26T12:16:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:04 smithi134 ceph-mon[31966]: slow request 2882.354753 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T12:16:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:04 smithi189 ceph-mon[36790]: pgmap v2372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T12:16:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:04 smithi189 ceph-mon[36790]: 2 slow requests, 1 included below; oldest blocked for > 2905.100036 secs 2024-07-26T12:16:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:04 smithi189 ceph-mon[36790]: slow request 2882.354753 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T12:16:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:04 smithi193 ceph-mon[37473]: pgmap v2372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T12:16:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:04 smithi193 ceph-mon[37473]: 2 slow requests, 1 included below; oldest blocked for > 2905.100036 secs 2024-07-26T12:16:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:04 smithi193 ceph-mon[37473]: slow request 2882.354753 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T12:16:05.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:16:05.165+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:16:05.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:16:05.165+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:16:05.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:16:05.165+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:06 smithi134 ceph-mon[31966]: pgmap v2373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:16:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2924.527210 secs 2024-07-26T12:16:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:06 smithi189 ceph-mon[36790]: pgmap v2373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:16:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2924.527210 secs 2024-07-26T12:16:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:06 smithi193 ceph-mon[37473]: pgmap v2373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:16:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2924.527210 secs 2024-07-26T12:16:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:08 smithi134 ceph-mon[31966]: pgmap v2374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:16:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:08 smithi189 ceph-mon[36790]: pgmap v2374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:16:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:08 smithi193 ceph-mon[37473]: pgmap v2374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:16:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2910.100131 secs 2024-07-26T12:16:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2910.100131 secs 2024-07-26T12:16:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2910.100131 secs 2024-07-26T12:16:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:10 smithi134 ceph-mon[31966]: pgmap v2375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:16:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:10 smithi189 ceph-mon[36790]: pgmap v2375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:16:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:10 smithi193 ceph-mon[37473]: pgmap v2375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:16:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2929.527355 secs 2024-07-26T12:16:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2929.527355 secs 2024-07-26T12:16:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2929.527355 secs 2024-07-26T12:16:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:12 smithi193 ceph-mon[37473]: pgmap v2376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:16:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:12 smithi134 ceph-mon[31966]: pgmap v2376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:16:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:12 smithi189 ceph-mon[36790]: pgmap v2376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:16:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:13 smithi193 ceph-mon[37473]: pgmap v2377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:16:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:13 smithi134 ceph-mon[31966]: pgmap v2377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:16:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:13 smithi189 ceph-mon[36790]: pgmap v2377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:16:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2915.100209 secs 2024-07-26T12:16:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2915.100209 secs 2024-07-26T12:16:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2915.100209 secs 2024-07-26T12:16:15.688 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:16:15.686+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:16:15.688 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:16:15.687+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:16:15.688 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:16:15.687+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:15 smithi134 ceph-mon[31966]: pgmap v2378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:15 smithi189 ceph-mon[36790]: pgmap v2378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:15 smithi193 ceph-mon[37473]: pgmap v2378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2934.527473 secs 2024-07-26T12:16:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2934.527473 secs 2024-07-26T12:16:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2934.527473 secs 2024-07-26T12:16:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:17 smithi134 ceph-mon[31966]: pgmap v2379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:17 smithi189 ceph-mon[36790]: pgmap v2379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:17 smithi193 ceph-mon[37473]: pgmap v2379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2920.100326 secs 2024-07-26T12:16:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2920.100326 secs 2024-07-26T12:16:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2920.100326 secs 2024-07-26T12:16:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:19 smithi134 ceph-mon[31966]: pgmap v2380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:20.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:16:20.172+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:16:20.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:16:20.172+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:16:20.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:16:20.172+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:19 smithi189 ceph-mon[36790]: pgmap v2380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:19 smithi193 ceph-mon[37473]: pgmap v2380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2939.527538 secs 2024-07-26T12:16:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:22 smithi134 ceph-mon[31966]: pgmap v2381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2939.527538 secs 2024-07-26T12:16:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:22 smithi189 ceph-mon[36790]: pgmap v2381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2939.527538 secs 2024-07-26T12:16:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:22 smithi193 ceph-mon[37473]: pgmap v2381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:24 smithi134 ceph-mon[31966]: pgmap v2382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:16:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2925.100443 secs 2024-07-26T12:16:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:24 smithi189 ceph-mon[36790]: pgmap v2382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:16:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2925.100443 secs 2024-07-26T12:16:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:24 smithi193 ceph-mon[37473]: pgmap v2382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:16:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2925.100443 secs 2024-07-26T12:16:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:26 smithi134 ceph-mon[31966]: pgmap v2383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:16:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2944.527631 secs 2024-07-26T12:16:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:26 smithi189 ceph-mon[36790]: pgmap v2383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:16:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2944.527631 secs 2024-07-26T12:16:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:26 smithi193 ceph-mon[37473]: pgmap v2383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:16:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2944.527631 secs 2024-07-26T12:16:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:28 smithi134 ceph-mon[31966]: pgmap v2384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:16:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:28 smithi189 ceph-mon[36790]: pgmap v2384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:16:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:28 smithi193 ceph-mon[37473]: pgmap v2384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:16:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2930.100567 secs 2024-07-26T12:16:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2930.100567 secs 2024-07-26T12:16:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2930.100567 secs 2024-07-26T12:16:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:30 smithi134 ceph-mon[31966]: pgmap v2385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:16:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:30 smithi189 ceph-mon[36790]: pgmap v2385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:16:30.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:16:30.702+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:16:30.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:16:30.702+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:16:30.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:16:30.702+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:30 smithi193 ceph-mon[37473]: pgmap v2385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:16:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2949.527759 secs 2024-07-26T12:16:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2949.527759 secs 2024-07-26T12:16:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2949.527759 secs 2024-07-26T12:16:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:32 smithi134 ceph-mon[31966]: pgmap v2386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:16:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:32 smithi189 ceph-mon[36790]: pgmap v2386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:16:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:32 smithi193 ceph-mon[37473]: pgmap v2386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:16:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:34 smithi134 ceph-mon[31966]: pgmap v2387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:16:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2935.100699 secs 2024-07-26T12:16:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:34 smithi189 ceph-mon[36790]: pgmap v2387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:16:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2935.100699 secs 2024-07-26T12:16:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:34 smithi193 ceph-mon[37473]: pgmap v2387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:16:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2935.100699 secs 2024-07-26T12:16:35.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:16:35.188+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:16:35.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:16:35.188+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:16:35.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:16:35.188+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:36 smithi134 ceph-mon[31966]: pgmap v2388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2954.527881 secs 2024-07-26T12:16:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:36 smithi189 ceph-mon[36790]: pgmap v2388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2954.527881 secs 2024-07-26T12:16:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:36 smithi193 ceph-mon[37473]: pgmap v2388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2954.527881 secs 2024-07-26T12:16:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:38 smithi134 ceph-mon[31966]: pgmap v2389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:38 smithi189 ceph-mon[36790]: pgmap v2389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:38 smithi193 ceph-mon[37473]: pgmap v2389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2940.100835 secs 2024-07-26T12:16:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2940.100835 secs 2024-07-26T12:16:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2940.100835 secs 2024-07-26T12:16:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:40 smithi134 ceph-mon[31966]: pgmap v2390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:40 smithi189 ceph-mon[36790]: pgmap v2390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:40 smithi193 ceph-mon[37473]: pgmap v2390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2959.527999 secs 2024-07-26T12:16:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2959.527999 secs 2024-07-26T12:16:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2959.527999 secs 2024-07-26T12:16:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:42 smithi134 ceph-mon[31966]: pgmap v2391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:42 smithi189 ceph-mon[36790]: pgmap v2391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:42 smithi193 ceph-mon[37473]: pgmap v2391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:44.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:44 smithi134 ceph-mon[31966]: pgmap v2392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:16:44.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2945.100999 secs 2024-07-26T12:16:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:44 smithi189 ceph-mon[36790]: pgmap v2392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:16:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2945.100999 secs 2024-07-26T12:16:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:44 smithi193 ceph-mon[37473]: pgmap v2392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:16:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2945.100999 secs 2024-07-26T12:16:45.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:16:45.713+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:16:45.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:16:45.713+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:45.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:16:45.713+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:16:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:46 smithi189 ceph-mon[36790]: pgmap v2393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2964.528112 secs 2024-07-26T12:16:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:46 smithi193 ceph-mon[37473]: pgmap v2393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2964.528112 secs 2024-07-26T12:16:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:46 smithi134 ceph-mon[31966]: pgmap v2393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2964.528112 secs 2024-07-26T12:16:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:48 smithi189 ceph-mon[36790]: pgmap v2394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:48 smithi193 ceph-mon[37473]: pgmap v2394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:48 smithi134 ceph-mon[31966]: pgmap v2394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2950.101126 secs 2024-07-26T12:16:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2950.101126 secs 2024-07-26T12:16:49.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2950.101126 secs 2024-07-26T12:16:50.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:16:50.203+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:16:50.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:16:50.203+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:16:50.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:16:50.203+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:16:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:50 smithi134 ceph-mon[31966]: pgmap v2395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:50 smithi193 ceph-mon[37473]: pgmap v2395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:50 smithi189 ceph-mon[36790]: pgmap v2395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:16:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2969.528233 secs 2024-07-26T12:16:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:51 smithi193 ceph-mon[37473]: pgmap v2396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2969.528233 secs 2024-07-26T12:16:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:51 smithi134 ceph-mon[31966]: pgmap v2396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2969.528233 secs 2024-07-26T12:16:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:51 smithi189 ceph-mon[36790]: pgmap v2396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:52.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:16:52.843+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:16:52.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:16:52.843+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:16:52.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:16:52.844+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T12:16:52.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:16:52.844+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T12:16:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:54 smithi134 ceph-mon[31966]: pgmap v2397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:16:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2955.101241 secs 2024-07-26T12:16:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:54 smithi189 ceph-mon[36790]: pgmap v2397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:16:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2955.101241 secs 2024-07-26T12:16:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:54 smithi193 ceph-mon[37473]: pgmap v2397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:16:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2955.101241 secs 2024-07-26T12:16:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:56 smithi134 ceph-mon[31966]: pgmap v2398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2974.528347 secs 2024-07-26T12:16:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:56 smithi189 ceph-mon[36790]: pgmap v2398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2974.528347 secs 2024-07-26T12:16:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:56 smithi193 ceph-mon[37473]: pgmap v2398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:16:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2974.528347 secs 2024-07-26T12:16:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:58 smithi134 ceph-mon[31966]: pgmap v2399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:58 smithi189 ceph-mon[36790]: pgmap v2399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:58 smithi193 ceph-mon[37473]: pgmap v2399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:16:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:16:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2960.101336 secs 2024-07-26T12:16:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:16:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2960.101336 secs 2024-07-26T12:16:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:16:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2960.101336 secs 2024-07-26T12:17:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:00 smithi134 ceph-mon[31966]: pgmap v2400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:00 smithi189 ceph-mon[36790]: pgmap v2400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:00.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:17:00.719+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:17:00.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:17:00.719+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:00.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:17:00.719+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:17:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:00 smithi193 ceph-mon[37473]: pgmap v2400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:17:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2979.528431 secs 2024-07-26T12:17:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2979.528431 secs 2024-07-26T12:17:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2979.528431 secs 2024-07-26T12:17:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:02 smithi134 ceph-mon[31966]: pgmap v2401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:02 smithi189 ceph-mon[36790]: pgmap v2401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:02 smithi193 ceph-mon[37473]: pgmap v2401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:03.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:17:03.352+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:17:03.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:17:03.352+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T12:17:03.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:17:03.352+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:17:03.354 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:17:03.352+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006490 con 0x55a08e2d94d0 2024-07-26T12:17:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:04 smithi134 ceph-mon[31966]: pgmap v2402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2965.101490 secs 2024-07-26T12:17:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:17:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:04 smithi189 ceph-mon[36790]: pgmap v2402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2965.101490 secs 2024-07-26T12:17:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:17:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:04 smithi193 ceph-mon[37473]: pgmap v2402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2965.101490 secs 2024-07-26T12:17:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:17:05.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:17:05.211+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:17:05.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:17:05.211+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:05.214 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:17:05.211+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:17:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:17:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:17:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:05 smithi134 ceph-mon[31966]: pgmap v2403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:17:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:05 smithi134 ceph-mon[31966]: pgmap v2404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:17:05.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:17:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:17:05.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:17:05.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:05 smithi189 ceph-mon[36790]: pgmap v2403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:17:05.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:05 smithi189 ceph-mon[36790]: pgmap v2404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:17:05.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:17:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:17:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:17:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:05 smithi193 ceph-mon[37473]: pgmap v2403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:17:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:05 smithi193 ceph-mon[37473]: pgmap v2404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:17:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:17:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2984.528554 secs 2024-07-26T12:17:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2984.528554 secs 2024-07-26T12:17:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2984.528554 secs 2024-07-26T12:17:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:07 smithi189 ceph-mon[36790]: pgmap v2405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:07 smithi193 ceph-mon[37473]: pgmap v2405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:07 smithi134 ceph-mon[31966]: pgmap v2405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:08 smithi134 ceph-mon[31966]: pgmap v2406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:17:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:08 smithi189 ceph-mon[36790]: pgmap v2406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:17:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:08 smithi193 ceph-mon[37473]: pgmap v2406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:17:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2970.101626 secs 2024-07-26T12:17:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2970.101626 secs 2024-07-26T12:17:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2970.101626 secs 2024-07-26T12:17:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:10 smithi134 ceph-mon[31966]: pgmap v2407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:17:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:10 smithi189 ceph-mon[36790]: pgmap v2407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:17:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:10 smithi193 ceph-mon[37473]: pgmap v2407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:17:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2989.528675 secs 2024-07-26T12:17:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2989.528675 secs 2024-07-26T12:17:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2989.528675 secs 2024-07-26T12:17:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:12 smithi134 ceph-mon[31966]: pgmap v2408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:17:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:12 smithi189 ceph-mon[36790]: pgmap v2408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:17:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:12 smithi193 ceph-mon[37473]: pgmap v2408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:17:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2975.101763 secs 2024-07-26T12:17:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2975.101763 secs 2024-07-26T12:17:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2975.101763 secs 2024-07-26T12:17:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:14 smithi134 ceph-mon[31966]: pgmap v2409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:17:15.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:14 smithi193 ceph-mon[37473]: pgmap v2409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:17:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:14 smithi189 ceph-mon[36790]: pgmap v2409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:17:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:17:15.735+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:17:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:17:15.735+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:17:15.735+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:17:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2994.528796 secs 2024-07-26T12:17:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2994.528796 secs 2024-07-26T12:17:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2994.528796 secs 2024-07-26T12:17:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:17 smithi134 ceph-mon[31966]: pgmap v2410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:17 smithi189 ceph-mon[36790]: pgmap v2410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:17 smithi193 ceph-mon[37473]: pgmap v2410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:18 smithi134 ceph-mon[31966]: pgmap v2411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:18.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:18 smithi189 ceph-mon[36790]: pgmap v2411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:18 smithi193 ceph-mon[37473]: pgmap v2411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:19.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2980.101896 secs 2024-07-26T12:17:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2980.101896 secs 2024-07-26T12:17:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2980.101896 secs 2024-07-26T12:17:20.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:17:20.227+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:17:20.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:17:20.227+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:20.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:17:20.227+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:17:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:20 smithi134 ceph-mon[31966]: pgmap v2412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:20 smithi189 ceph-mon[36790]: pgmap v2412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:20 smithi193 ceph-mon[37473]: pgmap v2412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 2999.528926 secs 2024-07-26T12:17:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 2999.528926 secs 2024-07-26T12:17:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 2999.528926 secs 2024-07-26T12:17:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:22 smithi134 ceph-mon[31966]: pgmap v2413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:22 smithi189 ceph-mon[36790]: pgmap v2413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:22 smithi193 ceph-mon[37473]: pgmap v2413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2985.102018 secs 2024-07-26T12:17:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2985.102018 secs 2024-07-26T12:17:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2985.102018 secs 2024-07-26T12:17:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:24 smithi134 ceph-mon[31966]: pgmap v2414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:17:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:24 smithi189 ceph-mon[36790]: pgmap v2414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:17:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:24 smithi193 ceph-mon[37473]: pgmap v2414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:17:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3004.529075 secs 2024-07-26T12:17:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3004.529075 secs 2024-07-26T12:17:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3004.529075 secs 2024-07-26T12:17:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:27 smithi134 ceph-mon[31966]: pgmap v2415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:27 smithi189 ceph-mon[36790]: pgmap v2415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:27 smithi193 ceph-mon[37473]: pgmap v2415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:28 smithi134 ceph-mon[31966]: pgmap v2416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:28 smithi189 ceph-mon[36790]: pgmap v2416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:29.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:28 smithi193 ceph-mon[37473]: pgmap v2416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2990.102126 secs 2024-07-26T12:17:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2990.102126 secs 2024-07-26T12:17:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2990.102126 secs 2024-07-26T12:17:30.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:17:30.744+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:17:30.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:17:30.744+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:30.746 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:17:30.744+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:17:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:30 smithi134 ceph-mon[31966]: pgmap v2417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:30 smithi189 ceph-mon[36790]: pgmap v2417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:30 smithi193 ceph-mon[37473]: pgmap v2417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3009.529206 secs 2024-07-26T12:17:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3009.529206 secs 2024-07-26T12:17:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3009.529206 secs 2024-07-26T12:17:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:32 smithi134 ceph-mon[31966]: pgmap v2418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:32 smithi189 ceph-mon[36790]: pgmap v2418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:32 smithi193 ceph-mon[37473]: pgmap v2418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 2995.102261 secs 2024-07-26T12:17:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 2995.102261 secs 2024-07-26T12:17:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 2995.102261 secs 2024-07-26T12:17:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:34 smithi134 ceph-mon[31966]: pgmap v2419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:17:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:34 smithi189 ceph-mon[36790]: pgmap v2419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:17:35.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:17:35.238+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:17:35.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:17:35.238+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:17:35.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:17:35.238+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:34 smithi193 ceph-mon[37473]: pgmap v2419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:17:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3014.529288 secs 2024-07-26T12:17:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3014.529288 secs 2024-07-26T12:17:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3014.529288 secs 2024-07-26T12:17:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:37 smithi134 ceph-mon[31966]: pgmap v2420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:37 smithi189 ceph-mon[36790]: pgmap v2420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:37 smithi193 ceph-mon[37473]: pgmap v2420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:38 smithi134 ceph-mon[31966]: pgmap v2421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:38 smithi189 ceph-mon[36790]: pgmap v2421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:38 smithi193 ceph-mon[37473]: pgmap v2421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3000.102390 secs 2024-07-26T12:17:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3000.102390 secs 2024-07-26T12:17:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3000.102390 secs 2024-07-26T12:17:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:40 smithi134 ceph-mon[31966]: pgmap v2422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:40 smithi189 ceph-mon[36790]: pgmap v2422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:40 smithi193 ceph-mon[37473]: pgmap v2422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3019.529405 secs 2024-07-26T12:17:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3019.529405 secs 2024-07-26T12:17:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3019.529405 secs 2024-07-26T12:17:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:42 smithi134 ceph-mon[31966]: pgmap v2423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:42 smithi189 ceph-mon[36790]: pgmap v2423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:42 smithi193 ceph-mon[37473]: pgmap v2423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3005.102519 secs 2024-07-26T12:17:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3005.102519 secs 2024-07-26T12:17:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3005.102519 secs 2024-07-26T12:17:45.103 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:44 smithi134 ceph-mon[31966]: pgmap v2424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:17:45.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:44 smithi193 ceph-mon[37473]: pgmap v2424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:17:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:44 smithi189 ceph-mon[36790]: pgmap v2424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:17:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:17:45.759+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:17:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:17:45.760+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:17:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:17:45.760+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3024.529528 secs 2024-07-26T12:17:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3024.529528 secs 2024-07-26T12:17:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3024.529528 secs 2024-07-26T12:17:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:47 smithi134 ceph-mon[31966]: pgmap v2425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:47 smithi189 ceph-mon[36790]: pgmap v2425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:47 smithi193 ceph-mon[37473]: pgmap v2425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:48 smithi134 ceph-mon[31966]: pgmap v2426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:48 smithi189 ceph-mon[36790]: pgmap v2426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:48 smithi193 ceph-mon[37473]: pgmap v2426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:49.825 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3010.102636 secs 2024-07-26T12:17:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3010.102636 secs 2024-07-26T12:17:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3010.102636 secs 2024-07-26T12:17:50.253 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:17:50.251+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:17:50.253 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:17:50.251+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:17:50.253 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:17:50.251+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:17:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:50 smithi134 ceph-mon[31966]: pgmap v2427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:50 smithi189 ceph-mon[36790]: pgmap v2427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:50 smithi193 ceph-mon[37473]: pgmap v2427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3029.529684 secs 2024-07-26T12:17:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3029.529684 secs 2024-07-26T12:17:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3029.529684 secs 2024-07-26T12:17:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:52 smithi134 ceph-mon[31966]: pgmap v2428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:52 smithi189 ceph-mon[36790]: pgmap v2428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:52 smithi193 ceph-mon[37473]: pgmap v2428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3015.102758 secs 2024-07-26T12:17:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3015.102758 secs 2024-07-26T12:17:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3015.102758 secs 2024-07-26T12:17:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:54 smithi134 ceph-mon[31966]: pgmap v2429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:17:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:54 smithi189 ceph-mon[36790]: pgmap v2429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:17:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:54 smithi193 ceph-mon[37473]: pgmap v2429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:17:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3034.529806 secs 2024-07-26T12:17:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3034.529806 secs 2024-07-26T12:17:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3034.529806 secs 2024-07-26T12:17:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:57 smithi134 ceph-mon[31966]: pgmap v2430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:57 smithi189 ceph-mon[36790]: pgmap v2430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:57 smithi193 ceph-mon[37473]: pgmap v2430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:17:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:58 smithi134 ceph-mon[31966]: pgmap v2431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:58 smithi189 ceph-mon[36790]: pgmap v2431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:58 smithi193 ceph-mon[37473]: pgmap v2431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:17:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:17:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3020.102897 secs 2024-07-26T12:17:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:17:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3020.102897 secs 2024-07-26T12:18:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:17:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3020.102897 secs 2024-07-26T12:18:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:18:00.775+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:18:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:18:00.775+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:00.777 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:18:00.775+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:18:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:00 smithi134 ceph-mon[31966]: pgmap v2432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:00 smithi189 ceph-mon[36790]: pgmap v2432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:00 smithi193 ceph-mon[37473]: pgmap v2432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3039.529931 secs 2024-07-26T12:18:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3039.529931 secs 2024-07-26T12:18:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3039.529931 secs 2024-07-26T12:18:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:02 smithi134 ceph-mon[31966]: pgmap v2433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:02 smithi189 ceph-mon[36790]: pgmap v2433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:02 smithi193 ceph-mon[37473]: pgmap v2433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3025.103035 secs 2024-07-26T12:18:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3025.103035 secs 2024-07-26T12:18:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3025.103035 secs 2024-07-26T12:18:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:04 smithi134 ceph-mon[31966]: pgmap v2434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:18:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:04 smithi189 ceph-mon[36790]: pgmap v2434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:18:05.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:18:05.266+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:18:05.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:18:05.266+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:18:05.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:18:05.266+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:04 smithi193 ceph-mon[37473]: pgmap v2434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:18:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:18:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:18:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:18:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:18:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:18:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:18:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:18:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:18:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:18:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:06 smithi134 ceph-mon[31966]: pgmap v2435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:18:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:06 smithi134 ceph-mon[31966]: pgmap v2436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:18:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3044.530019 secs 2024-07-26T12:18:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:06 smithi189 ceph-mon[36790]: pgmap v2435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:18:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:06 smithi189 ceph-mon[36790]: pgmap v2436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:18:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3044.530019 secs 2024-07-26T12:18:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:06 smithi193 ceph-mon[37473]: pgmap v2435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:18:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:06 smithi193 ceph-mon[37473]: pgmap v2436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:18:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3044.530019 secs 2024-07-26T12:18:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:08 smithi134 ceph-mon[31966]: pgmap v2437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:18:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3030.103182 secs 2024-07-26T12:18:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:08 smithi189 ceph-mon[36790]: pgmap v2437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:18:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3030.103182 secs 2024-07-26T12:18:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:08 smithi193 ceph-mon[37473]: pgmap v2437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:18:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3030.103182 secs 2024-07-26T12:18:11.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:10 smithi134 ceph-mon[31966]: pgmap v2438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:18:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:10 smithi189 ceph-mon[36790]: pgmap v2438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:18:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:10 smithi193 ceph-mon[37473]: pgmap v2438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:18:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3049.530144 secs 2024-07-26T12:18:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3049.530144 secs 2024-07-26T12:18:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3049.530144 secs 2024-07-26T12:18:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:12 smithi134 ceph-mon[31966]: pgmap v2439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:18:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:12 smithi189 ceph-mon[36790]: pgmap v2439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:18:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:12 smithi193 ceph-mon[37473]: pgmap v2439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:18:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3035.103317 secs 2024-07-26T12:18:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3035.103317 secs 2024-07-26T12:18:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3035.103317 secs 2024-07-26T12:18:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:14 smithi134 ceph-mon[31966]: pgmap v2440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:18:15.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:14 smithi193 ceph-mon[37473]: pgmap v2440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:18:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:14 smithi189 ceph-mon[36790]: pgmap v2440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:18:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:18:15.791+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:18:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:18:15.791+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:18:15.791+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:18:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:16 smithi134 ceph-mon[31966]: pgmap v2441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:18:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3054.530285 secs 2024-07-26T12:18:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:16 smithi189 ceph-mon[36790]: pgmap v2441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:18:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3054.530285 secs 2024-07-26T12:18:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:16 smithi193 ceph-mon[37473]: pgmap v2441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:18:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3054.530285 secs 2024-07-26T12:18:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:18 smithi189 ceph-mon[36790]: pgmap v2442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3040.103454 secs 2024-07-26T12:18:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:18 smithi193 ceph-mon[37473]: pgmap v2442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3040.103454 secs 2024-07-26T12:18:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:18 smithi134 ceph-mon[31966]: pgmap v2442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3040.103454 secs 2024-07-26T12:18:20.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:18:20.280+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:18:20.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:18:20.280+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:18:20.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:18:20.280+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:20 smithi189 ceph-mon[36790]: pgmap v2443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:20 smithi193 ceph-mon[37473]: pgmap v2443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:20 smithi134 ceph-mon[31966]: pgmap v2443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3059.530397 secs 2024-07-26T12:18:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3059.530397 secs 2024-07-26T12:18:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3059.530397 secs 2024-07-26T12:18:22.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:18:22.858+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:18:22.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:18:22.858+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:18:22.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:18:22.859+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:18:22.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:18:22.859+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:18:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:22 smithi134 ceph-mon[31966]: pgmap v2444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:22 smithi193 ceph-mon[37473]: pgmap v2444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:22 smithi189 ceph-mon[36790]: pgmap v2444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3045.103585 secs 2024-07-26T12:18:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3045.103585 secs 2024-07-26T12:18:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3045.103585 secs 2024-07-26T12:18:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:24 smithi193 ceph-mon[37473]: pgmap v2445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:24 smithi134 ceph-mon[31966]: pgmap v2445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:24 smithi189 ceph-mon[36790]: pgmap v2445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:26 smithi193 ceph-mon[37473]: pgmap v2446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3064.530458 secs 2024-07-26T12:18:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:26 smithi134 ceph-mon[31966]: pgmap v2446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3064.530458 secs 2024-07-26T12:18:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:26 smithi189 ceph-mon[36790]: pgmap v2446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3064.530458 secs 2024-07-26T12:18:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:28 smithi193 ceph-mon[37473]: pgmap v2447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3050.103710 secs 2024-07-26T12:18:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:28 smithi134 ceph-mon[31966]: pgmap v2447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3050.103710 secs 2024-07-26T12:18:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:28 smithi189 ceph-mon[36790]: pgmap v2447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3050.103710 secs 2024-07-26T12:18:30.793 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:18:30.792+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:18:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:18:30.792+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:18:30.792+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:18:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:30 smithi193 ceph-mon[37473]: pgmap v2448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:30 smithi134 ceph-mon[31966]: pgmap v2448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:30 smithi189 ceph-mon[36790]: pgmap v2448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3069.530573 secs 2024-07-26T12:18:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3069.530573 secs 2024-07-26T12:18:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3069.530573 secs 2024-07-26T12:18:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:32 smithi193 ceph-mon[37473]: pgmap v2449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:33.369 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:18:33.368+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T12:18:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:18:33.368+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:18:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:18:33.369+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T12:18:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:18:33.369+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:18:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:32 smithi134 ceph-mon[31966]: pgmap v2449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:32 smithi189 ceph-mon[36790]: pgmap v2449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3055.103840 secs 2024-07-26T12:18:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3055.103840 secs 2024-07-26T12:18:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3055.103840 secs 2024-07-26T12:18:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:35 smithi193 ceph-mon[37473]: pgmap v2450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:35.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:18:35.295+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:18:35.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:18:35.295+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:35.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:18:35.295+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:18:35.298 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:35 smithi134 ceph-mon[31966]: pgmap v2450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:35 smithi189 ceph-mon[36790]: pgmap v2450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:37 smithi193 ceph-mon[37473]: pgmap v2451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3074.530675 secs 2024-07-26T12:18:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:37 smithi134 ceph-mon[31966]: pgmap v2451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3074.530675 secs 2024-07-26T12:18:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:37 smithi189 ceph-mon[36790]: pgmap v2451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3074.530675 secs 2024-07-26T12:18:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:39 smithi193 ceph-mon[37473]: pgmap v2452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3060.103986 secs 2024-07-26T12:18:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:39 smithi134 ceph-mon[31966]: pgmap v2452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3060.103986 secs 2024-07-26T12:18:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:39 smithi189 ceph-mon[36790]: pgmap v2452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3060.103986 secs 2024-07-26T12:18:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:41 smithi193 ceph-mon[37473]: pgmap v2453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:41 smithi134 ceph-mon[31966]: pgmap v2453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:41.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:41 smithi189 ceph-mon[36790]: pgmap v2453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3079.530769 secs 2024-07-26T12:18:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3079.530769 secs 2024-07-26T12:18:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3079.530769 secs 2024-07-26T12:18:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:43 smithi134 ceph-mon[31966]: pgmap v2454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:43 smithi189 ceph-mon[36790]: pgmap v2454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:43 smithi193 ceph-mon[37473]: pgmap v2454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3065.104103 secs 2024-07-26T12:18:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3065.104103 secs 2024-07-26T12:18:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3065.104103 secs 2024-07-26T12:18:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:45 smithi134 ceph-mon[31966]: pgmap v2455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:45.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:45 smithi193 ceph-mon[37473]: pgmap v2455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:45 smithi189 ceph-mon[36790]: pgmap v2455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:18:45.808+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:18:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:18:45.808+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:18:45.809 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:18:45.808+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:47 smithi134 ceph-mon[31966]: pgmap v2456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3084.530880 secs 2024-07-26T12:18:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:47 smithi189 ceph-mon[36790]: pgmap v2456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3084.530880 secs 2024-07-26T12:18:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:47 smithi193 ceph-mon[37473]: pgmap v2456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3084.530880 secs 2024-07-26T12:18:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:49 smithi134 ceph-mon[31966]: pgmap v2457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3070.104216 secs 2024-07-26T12:18:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:49 smithi189 ceph-mon[36790]: pgmap v2457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3070.104216 secs 2024-07-26T12:18:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:49 smithi193 ceph-mon[37473]: pgmap v2457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3070.104216 secs 2024-07-26T12:18:50.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:18:50.307+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:18:50.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:18:50.307+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:18:50.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:18:50.307+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:18:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:51 smithi134 ceph-mon[31966]: pgmap v2458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:51 smithi189 ceph-mon[36790]: pgmap v2458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:51 smithi193 ceph-mon[37473]: pgmap v2458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:18:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3089.530974 secs 2024-07-26T12:18:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3089.530974 secs 2024-07-26T12:18:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3089.530974 secs 2024-07-26T12:18:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:53 smithi134 ceph-mon[31966]: pgmap v2459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:53 smithi189 ceph-mon[36790]: pgmap v2459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:53 smithi193 ceph-mon[37473]: pgmap v2459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3075.104340 secs 2024-07-26T12:18:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3075.104340 secs 2024-07-26T12:18:54.527 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3075.104340 secs 2024-07-26T12:18:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:55 smithi134 ceph-mon[31966]: pgmap v2460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:55 smithi189 ceph-mon[36790]: pgmap v2460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:55 smithi193 ceph-mon[37473]: pgmap v2460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:57 smithi134 ceph-mon[31966]: pgmap v2461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3094.531122 secs 2024-07-26T12:18:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:57 smithi189 ceph-mon[36790]: pgmap v2461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3094.531122 secs 2024-07-26T12:18:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:57 smithi193 ceph-mon[37473]: pgmap v2461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:18:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3094.531122 secs 2024-07-26T12:18:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:59 smithi134 ceph-mon[31966]: pgmap v2462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:18:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3080.104464 secs 2024-07-26T12:18:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:59 smithi189 ceph-mon[36790]: pgmap v2462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:18:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3080.104464 secs 2024-07-26T12:18:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:59 smithi193 ceph-mon[37473]: pgmap v2462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:18:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:18:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3080.104464 secs 2024-07-26T12:19:00.825 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:19:00.823+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:19:00.825 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:19:00.823+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:00.825 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:19:00.823+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:19:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:01 smithi134 ceph-mon[31966]: pgmap v2463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:01 smithi189 ceph-mon[36790]: pgmap v2463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:01.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:01 smithi193 ceph-mon[37473]: pgmap v2463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3099.531262 secs 2024-07-26T12:19:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3099.531262 secs 2024-07-26T12:19:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3099.531262 secs 2024-07-26T12:19:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:03 smithi134 ceph-mon[31966]: pgmap v2464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:03 smithi189 ceph-mon[36790]: pgmap v2464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:03.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:03 smithi193 ceph-mon[37473]: pgmap v2464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3085.104601 secs 2024-07-26T12:19:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3085.104601 secs 2024-07-26T12:19:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3085.104601 secs 2024-07-26T12:19:05.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:19:05.322+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:19:05.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:19:05.322+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:05.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:19:05.322+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:19:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:05 smithi134 ceph-mon[31966]: pgmap v2465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:05 smithi189 ceph-mon[36790]: pgmap v2465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:05 smithi193 ceph-mon[37473]: pgmap v2465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:19:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:19:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:19:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:07 smithi134 ceph-mon[31966]: pgmap v2466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3104.531378 secs 2024-07-26T12:19:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:19:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:19:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:19:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:07 smithi189 ceph-mon[36790]: pgmap v2466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3104.531378 secs 2024-07-26T12:19:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:19:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:19:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:19:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:07 smithi193 ceph-mon[37473]: pgmap v2466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3104.531378 secs 2024-07-26T12:19:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:19:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:19:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:19:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:08 smithi134 ceph-mon[31966]: pgmap v2467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:19:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:08 smithi134 ceph-mon[31966]: pgmap v2468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:19:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:08 smithi189 ceph-mon[36790]: pgmap v2467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:19:08.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:08 smithi189 ceph-mon[36790]: pgmap v2468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:19:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:08 smithi193 ceph-mon[37473]: pgmap v2467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:19:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:08 smithi193 ceph-mon[37473]: pgmap v2468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:19:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3090.104727 secs 2024-07-26T12:19:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3090.104727 secs 2024-07-26T12:19:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3090.104727 secs 2024-07-26T12:19:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:10 smithi189 ceph-mon[36790]: pgmap v2469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:19:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:10 smithi193 ceph-mon[37473]: pgmap v2469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:19:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:10 smithi134 ceph-mon[31966]: pgmap v2469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:19:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3109.531492 secs 2024-07-26T12:19:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:12 smithi189 ceph-mon[36790]: pgmap v2470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:19:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3109.531492 secs 2024-07-26T12:19:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:12 smithi193 ceph-mon[37473]: pgmap v2470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:19:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3109.531492 secs 2024-07-26T12:19:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:12 smithi134 ceph-mon[31966]: pgmap v2470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:19:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:14 smithi193 ceph-mon[37473]: pgmap v2471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:19:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3095.104852 secs 2024-07-26T12:19:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:14 smithi134 ceph-mon[31966]: pgmap v2471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:19:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3095.104852 secs 2024-07-26T12:19:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:14 smithi189 ceph-mon[36790]: pgmap v2471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:19:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3095.104852 secs 2024-07-26T12:19:15.840 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:19:15.839+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:19:15.840 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:19:15.839+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:19:15.840 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:19:15.839+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:16 smithi193 ceph-mon[37473]: pgmap v2472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:19:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:16 smithi134 ceph-mon[31966]: pgmap v2472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:19:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:16 smithi189 ceph-mon[36790]: pgmap v2472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:19:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3114.531626 secs 2024-07-26T12:19:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3114.531626 secs 2024-07-26T12:19:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3114.531626 secs 2024-07-26T12:19:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:18 smithi193 ceph-mon[37473]: pgmap v2473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:19:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:18 smithi134 ceph-mon[31966]: pgmap v2473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:19:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:18 smithi189 ceph-mon[36790]: pgmap v2473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:19:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3100.104982 secs 2024-07-26T12:19:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3100.104982 secs 2024-07-26T12:19:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3100.104982 secs 2024-07-26T12:19:20.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:19:20.337+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:19:20.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:19:20.337+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:20.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:19:20.337+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:19:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:20 smithi193 ceph-mon[37473]: pgmap v2474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:20.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:20 smithi134 ceph-mon[31966]: pgmap v2474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:20 smithi189 ceph-mon[36790]: pgmap v2474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3119.531755 secs 2024-07-26T12:19:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:21 smithi134 ceph-mon[31966]: pgmap v2475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3119.531755 secs 2024-07-26T12:19:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:21 smithi189 ceph-mon[36790]: pgmap v2475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3119.531755 secs 2024-07-26T12:19:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:21 smithi193 ceph-mon[37473]: pgmap v2475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:24 smithi193 ceph-mon[37473]: pgmap v2476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3105.105115 secs 2024-07-26T12:19:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:24 smithi134 ceph-mon[31966]: pgmap v2476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3105.105115 secs 2024-07-26T12:19:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:24 smithi189 ceph-mon[36790]: pgmap v2476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3105.105115 secs 2024-07-26T12:19:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:26 smithi193 ceph-mon[37473]: pgmap v2477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:26 smithi134 ceph-mon[31966]: pgmap v2477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:26 smithi189 ceph-mon[36790]: pgmap v2477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3124.531818 secs 2024-07-26T12:19:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3124.531818 secs 2024-07-26T12:19:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3124.531818 secs 2024-07-26T12:19:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:28 smithi134 ceph-mon[31966]: pgmap v2478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:28 smithi189 ceph-mon[36790]: pgmap v2478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:28 smithi193 ceph-mon[37473]: pgmap v2478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3110.105250 secs 2024-07-26T12:19:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3110.105250 secs 2024-07-26T12:19:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3110.105250 secs 2024-07-26T12:19:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:30 smithi134 ceph-mon[31966]: pgmap v2479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:30 smithi189 ceph-mon[36790]: pgmap v2479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:30.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:30 smithi193 ceph-mon[37473]: pgmap v2479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:19:30.855+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:19:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:19:30.855+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:30.856 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:19:30.855+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:19:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3129.531954 secs 2024-07-26T12:19:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:32 smithi134 ceph-mon[31966]: pgmap v2480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3129.531954 secs 2024-07-26T12:19:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:32 smithi189 ceph-mon[36790]: pgmap v2480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3129.531954 secs 2024-07-26T12:19:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:32 smithi193 ceph-mon[37473]: pgmap v2480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:34 smithi134 ceph-mon[31966]: pgmap v2481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3115.105369 secs 2024-07-26T12:19:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:34 smithi189 ceph-mon[36790]: pgmap v2481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3115.105369 secs 2024-07-26T12:19:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:34 smithi193 ceph-mon[37473]: pgmap v2481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3115.105369 secs 2024-07-26T12:19:35.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:19:35.351+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:19:35.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:19:35.351+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:35.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:19:35.351+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:19:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:36 smithi134 ceph-mon[31966]: pgmap v2482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:36.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:36 smithi189 ceph-mon[36790]: pgmap v2482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:36 smithi193 ceph-mon[37473]: pgmap v2482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3134.532072 secs 2024-07-26T12:19:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3134.532072 secs 2024-07-26T12:19:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3134.532072 secs 2024-07-26T12:19:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:38 smithi134 ceph-mon[31966]: pgmap v2483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:38 smithi189 ceph-mon[36790]: pgmap v2483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:38 smithi193 ceph-mon[37473]: pgmap v2483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:39.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3120.105488 secs 2024-07-26T12:19:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3120.105488 secs 2024-07-26T12:19:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3120.105488 secs 2024-07-26T12:19:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:40 smithi134 ceph-mon[31966]: pgmap v2484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:40 smithi189 ceph-mon[36790]: pgmap v2484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:40 smithi193 ceph-mon[37473]: pgmap v2484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3139.532202 secs 2024-07-26T12:19:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:42 smithi134 ceph-mon[31966]: pgmap v2485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3139.532202 secs 2024-07-26T12:19:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:42 smithi189 ceph-mon[36790]: pgmap v2485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3139.532202 secs 2024-07-26T12:19:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:42 smithi193 ceph-mon[37473]: pgmap v2485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:44 smithi134 ceph-mon[31966]: pgmap v2486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3125.105572 secs 2024-07-26T12:19:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:44 smithi189 ceph-mon[36790]: pgmap v2486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3125.105572 secs 2024-07-26T12:19:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:44 smithi193 ceph-mon[37473]: pgmap v2486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3125.105572 secs 2024-07-26T12:19:45.867 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:19:45.866+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:19:45.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:19:45.866+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:45.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:19:45.866+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:19:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:46 smithi134 ceph-mon[31966]: pgmap v2487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:46 smithi189 ceph-mon[36790]: pgmap v2487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:46 smithi193 ceph-mon[37473]: pgmap v2487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3144.532333 secs 2024-07-26T12:19:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3144.532333 secs 2024-07-26T12:19:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3144.532333 secs 2024-07-26T12:19:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:48 smithi134 ceph-mon[31966]: pgmap v2488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:48 smithi189 ceph-mon[36790]: pgmap v2488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:48 smithi193 ceph-mon[37473]: pgmap v2488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3130.105662 secs 2024-07-26T12:19:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3130.105662 secs 2024-07-26T12:19:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3130.105662 secs 2024-07-26T12:19:50.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:19:50.361+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:19:50.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:19:50.361+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:19:50.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:19:50.361+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:19:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:50 smithi134 ceph-mon[31966]: pgmap v2489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:50 smithi189 ceph-mon[36790]: pgmap v2489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:50 smithi193 ceph-mon[37473]: pgmap v2489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3149.532421 secs 2024-07-26T12:19:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:52 smithi134 ceph-mon[31966]: pgmap v2490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3149.532421 secs 2024-07-26T12:19:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:52 smithi189 ceph-mon[36790]: pgmap v2490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3149.532421 secs 2024-07-26T12:19:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:52 smithi193 ceph-mon[37473]: pgmap v2490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:54 smithi134 ceph-mon[31966]: pgmap v2491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3135.105797 secs 2024-07-26T12:19:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:54 smithi189 ceph-mon[36790]: pgmap v2491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3135.105797 secs 2024-07-26T12:19:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:54 smithi193 ceph-mon[37473]: pgmap v2491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:19:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3135.105797 secs 2024-07-26T12:19:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:56 smithi134 ceph-mon[31966]: pgmap v2492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:56 smithi189 ceph-mon[36790]: pgmap v2492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:56 smithi193 ceph-mon[37473]: pgmap v2492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:19:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3154.532500 secs 2024-07-26T12:19:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3154.532500 secs 2024-07-26T12:19:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3154.532500 secs 2024-07-26T12:19:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:58 smithi189 ceph-mon[36790]: pgmap v2493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:58 smithi193 ceph-mon[37473]: pgmap v2493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:58 smithi134 ceph-mon[31966]: pgmap v2493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:19:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:19:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3140.105899 secs 2024-07-26T12:19:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:19:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3140.105899 secs 2024-07-26T12:19:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:19:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3140.105899 secs 2024-07-26T12:20:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:00 smithi189 ceph-mon[36790]: pgmap v2494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:20:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:00 smithi193 ceph-mon[37473]: pgmap v2494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:20:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:00 smithi134 ceph-mon[31966]: pgmap v2494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:20:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:20:00.880+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:20:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:20:00.880+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:20:00.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:20:00.880+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3159.532565 secs 2024-07-26T12:20:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:02 smithi193 ceph-mon[37473]: pgmap v2495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3159.532565 secs 2024-07-26T12:20:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:02 smithi134 ceph-mon[31966]: pgmap v2495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3159.532565 secs 2024-07-26T12:20:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:02 smithi189 ceph-mon[36790]: pgmap v2495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:04.475 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:04 smithi134 ceph-mon[31966]: pgmap v2496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:04.475 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3145.106015 secs 2024-07-26T12:20:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:04 smithi193 ceph-mon[37473]: pgmap v2496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3145.106015 secs 2024-07-26T12:20:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:04 smithi189 ceph-mon[36790]: pgmap v2496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3145.106015 secs 2024-07-26T12:20:05.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:20:05.376+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:20:05.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:20:05.376+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:20:05.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:20:05.376+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:06.464 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:06 smithi134 ceph-mon[31966]: pgmap v2497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:20:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:06 smithi193 ceph-mon[37473]: pgmap v2497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:20:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:06 smithi189 ceph-mon[36790]: pgmap v2497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:20:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3164.532687 secs 2024-07-26T12:20:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:20:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3164.532687 secs 2024-07-26T12:20:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:20:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3164.532687 secs 2024-07-26T12:20:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:20:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:08 smithi193 ceph-mon[37473]: pgmap v2498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:20:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:20:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:20:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:08 smithi134 ceph-mon[31966]: pgmap v2498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:20:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:20:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:20:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:08 smithi189 ceph-mon[36790]: pgmap v2498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:20:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:20:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:20:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:09 smithi193 ceph-mon[37473]: pgmap v2499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T12:20:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:09 smithi193 ceph-mon[37473]: pgmap v2500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:20:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3150.106082 secs 2024-07-26T12:20:09.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:09 smithi189 ceph-mon[36790]: pgmap v2499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T12:20:09.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:09 smithi189 ceph-mon[36790]: pgmap v2500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:20:09.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3150.106082 secs 2024-07-26T12:20:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:09 smithi134 ceph-mon[31966]: pgmap v2499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T12:20:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:09 smithi134 ceph-mon[31966]: pgmap v2500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:20:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3150.106082 secs 2024-07-26T12:20:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:11 smithi193 ceph-mon[37473]: pgmap v2501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:20:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:11 smithi134 ceph-mon[31966]: pgmap v2501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:20:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:11 smithi189 ceph-mon[36790]: pgmap v2501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:20:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3169.532810 secs 2024-07-26T12:20:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3169.532810 secs 2024-07-26T12:20:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3169.532810 secs 2024-07-26T12:20:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:13 smithi193 ceph-mon[37473]: pgmap v2502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:20:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:13 smithi134 ceph-mon[31966]: pgmap v2502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:20:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:13 smithi189 ceph-mon[36790]: pgmap v2502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:20:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3155.106151 secs 2024-07-26T12:20:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3155.106151 secs 2024-07-26T12:20:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3155.106151 secs 2024-07-26T12:20:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:15 smithi193 ceph-mon[37473]: pgmap v2503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:20:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:15 smithi134 ceph-mon[31966]: pgmap v2503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:20:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:15 smithi189 ceph-mon[36790]: pgmap v2503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:20:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:20:15.888+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:20:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:20:15.888+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:20:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:20:15.888+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3174.532909 secs 2024-07-26T12:20:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:17 smithi193 ceph-mon[37473]: pgmap v2504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:20:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3174.532909 secs 2024-07-26T12:20:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:17 smithi134 ceph-mon[31966]: pgmap v2504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:20:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3174.532909 secs 2024-07-26T12:20:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:17 smithi189 ceph-mon[36790]: pgmap v2504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:20:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:19 smithi193 ceph-mon[37473]: pgmap v2505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:20:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3160.106216 secs 2024-07-26T12:20:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:19 smithi134 ceph-mon[31966]: pgmap v2505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:20:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3160.106216 secs 2024-07-26T12:20:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:19 smithi189 ceph-mon[36790]: pgmap v2505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:20:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3160.106216 secs 2024-07-26T12:20:20.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:20:20.385+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:20:20.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:20:20.385+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:20.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:20:20.385+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:20:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:21 smithi134 ceph-mon[31966]: pgmap v2506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:21 smithi189 ceph-mon[36790]: pgmap v2506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:21 smithi193 ceph-mon[37473]: pgmap v2506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3179.533036 secs 2024-07-26T12:20:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3179.533036 secs 2024-07-26T12:20:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3179.533036 secs 2024-07-26T12:20:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:23 smithi134 ceph-mon[31966]: pgmap v2507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:23 smithi189 ceph-mon[36790]: pgmap v2507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:23 smithi193 ceph-mon[37473]: pgmap v2507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3165.106302 secs 2024-07-26T12:20:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3165.106302 secs 2024-07-26T12:20:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3165.106302 secs 2024-07-26T12:20:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:25 smithi134 ceph-mon[31966]: pgmap v2508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:25 smithi189 ceph-mon[36790]: pgmap v2508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:25 smithi193 ceph-mon[37473]: pgmap v2508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3184.533172 secs 2024-07-26T12:20:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:27 smithi134 ceph-mon[31966]: pgmap v2509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3184.533172 secs 2024-07-26T12:20:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:27 smithi189 ceph-mon[36790]: pgmap v2509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3184.533172 secs 2024-07-26T12:20:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:27 smithi193 ceph-mon[37473]: pgmap v2509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:29 smithi134 ceph-mon[31966]: pgmap v2510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3170.106370 secs 2024-07-26T12:20:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:29 smithi189 ceph-mon[36790]: pgmap v2510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3170.106370 secs 2024-07-26T12:20:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:29 smithi193 ceph-mon[37473]: pgmap v2510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3170.106370 secs 2024-07-26T12:20:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:20:30.895+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:20:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:20:30.895+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:30.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:20:30.895+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:20:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:31 smithi134 ceph-mon[31966]: pgmap v2511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:31 smithi189 ceph-mon[36790]: pgmap v2511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:31 smithi193 ceph-mon[37473]: pgmap v2511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3189.533266 secs 2024-07-26T12:20:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3189.533266 secs 2024-07-26T12:20:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3189.533266 secs 2024-07-26T12:20:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:33 smithi134 ceph-mon[31966]: pgmap v2512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:33 smithi189 ceph-mon[36790]: pgmap v2512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:33 smithi193 ceph-mon[37473]: pgmap v2512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3175.106500 secs 2024-07-26T12:20:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3175.106500 secs 2024-07-26T12:20:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3175.106500 secs 2024-07-26T12:20:35.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:20:35.396+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:20:35.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:20:35.396+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:20:35.398 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:20:35.396+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:35 smithi134 ceph-mon[31966]: pgmap v2513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:35 smithi189 ceph-mon[36790]: pgmap v2513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:35 smithi193 ceph-mon[37473]: pgmap v2513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3194.533392 secs 2024-07-26T12:20:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:37 smithi134 ceph-mon[31966]: pgmap v2514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3194.533392 secs 2024-07-26T12:20:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:37 smithi189 ceph-mon[36790]: pgmap v2514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3194.533392 secs 2024-07-26T12:20:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:37 smithi193 ceph-mon[37473]: pgmap v2514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:39 smithi134 ceph-mon[31966]: pgmap v2515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3180.106568 secs 2024-07-26T12:20:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:39 smithi189 ceph-mon[36790]: pgmap v2515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3180.106568 secs 2024-07-26T12:20:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:39 smithi193 ceph-mon[37473]: pgmap v2515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3180.106568 secs 2024-07-26T12:20:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:41 smithi134 ceph-mon[31966]: pgmap v2516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:41 smithi189 ceph-mon[36790]: pgmap v2516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:41 smithi193 ceph-mon[37473]: pgmap v2516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3199.533489 secs 2024-07-26T12:20:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3199.533489 secs 2024-07-26T12:20:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3199.533489 secs 2024-07-26T12:20:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:43 smithi134 ceph-mon[31966]: pgmap v2517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:43 smithi189 ceph-mon[36790]: pgmap v2517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:43 smithi193 ceph-mon[37473]: pgmap v2517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3185.106712 secs 2024-07-26T12:20:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3185.106712 secs 2024-07-26T12:20:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3185.106712 secs 2024-07-26T12:20:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:45 smithi134 ceph-mon[31966]: pgmap v2518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:45 smithi189 ceph-mon[36790]: pgmap v2518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:45 smithi193 ceph-mon[37473]: pgmap v2518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:45.906 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:20:45.905+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:20:45.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:20:45.905+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:20:45.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:20:45.905+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3204.533619 secs 2024-07-26T12:20:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3204.533619 secs 2024-07-26T12:20:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3204.533619 secs 2024-07-26T12:20:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:47 smithi134 ceph-mon[31966]: pgmap v2519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:47.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:47 smithi193 ceph-mon[37473]: pgmap v2519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:47 smithi189 ceph-mon[36790]: pgmap v2519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:49 smithi134 ceph-mon[31966]: pgmap v2520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3190.106840 secs 2024-07-26T12:20:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:49 smithi189 ceph-mon[36790]: pgmap v2520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3190.106840 secs 2024-07-26T12:20:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:49 smithi193 ceph-mon[37473]: pgmap v2520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3190.106840 secs 2024-07-26T12:20:50.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:20:50.411+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:20:50.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:20:50.411+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:20:50.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:20:50.411+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:20:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:51 smithi134 ceph-mon[31966]: pgmap v2521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3209.533747 secs 2024-07-26T12:20:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:51 smithi189 ceph-mon[36790]: pgmap v2521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3209.533747 secs 2024-07-26T12:20:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:51 smithi193 ceph-mon[37473]: pgmap v2521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3209.533747 secs 2024-07-26T12:20:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:53 smithi189 ceph-mon[36790]: pgmap v2522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:53 smithi193 ceph-mon[37473]: pgmap v2522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:53 smithi134 ceph-mon[31966]: pgmap v2522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:20:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3195.106917 secs 2024-07-26T12:20:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3195.106917 secs 2024-07-26T12:20:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3195.106917 secs 2024-07-26T12:20:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:55 smithi189 ceph-mon[36790]: pgmap v2523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:55 smithi193 ceph-mon[37473]: pgmap v2523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:55 smithi134 ceph-mon[31966]: pgmap v2523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:20:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3214.533894 secs 2024-07-26T12:20:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3214.533894 secs 2024-07-26T12:20:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3214.533894 secs 2024-07-26T12:20:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:57 smithi193 ceph-mon[37473]: pgmap v2524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:57 smithi134 ceph-mon[31966]: pgmap v2524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:57 smithi189 ceph-mon[36790]: pgmap v2524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:20:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:59 smithi193 ceph-mon[37473]: pgmap v2525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:20:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3200.107045 secs 2024-07-26T12:20:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:59 smithi134 ceph-mon[31966]: pgmap v2525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:20:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3200.107045 secs 2024-07-26T12:20:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:59 smithi189 ceph-mon[36790]: pgmap v2525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:20:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:20:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3200.107045 secs 2024-07-26T12:21:00.915 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:21:00.914+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:21:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:21:00.914+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:00.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:21:00.914+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:21:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:01 smithi193 ceph-mon[37473]: pgmap v2526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3219.534017 secs 2024-07-26T12:21:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:01 smithi134 ceph-mon[31966]: pgmap v2526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3219.534017 secs 2024-07-26T12:21:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:01 smithi189 ceph-mon[36790]: pgmap v2526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3219.534017 secs 2024-07-26T12:21:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:03 smithi193 ceph-mon[37473]: pgmap v2527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:21:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:03 smithi134 ceph-mon[31966]: pgmap v2527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:21:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:03 smithi189 ceph-mon[36790]: pgmap v2527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:21:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3205.107172 secs 2024-07-26T12:21:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3205.107172 secs 2024-07-26T12:21:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3205.107172 secs 2024-07-26T12:21:05.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:21:05.426+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:21:05.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:21:05.426+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:21:05.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:21:05.427+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:05 smithi193 ceph-mon[37473]: pgmap v2528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:21:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:05 smithi134 ceph-mon[31966]: pgmap v2528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:21:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:05 smithi189 ceph-mon[36790]: pgmap v2528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:21:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3224.534139 secs 2024-07-26T12:21:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3224.534139 secs 2024-07-26T12:21:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3224.534139 secs 2024-07-26T12:21:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:07 smithi193 ceph-mon[37473]: pgmap v2529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:07 smithi134 ceph-mon[31966]: pgmap v2529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:07 smithi189 ceph-mon[36790]: pgmap v2529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:21:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:21:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:21:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:09 smithi193 ceph-mon[37473]: pgmap v2530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:21:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3210.107256 secs 2024-07-26T12:21:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:21:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:21:09.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:21:09.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:09 smithi189 ceph-mon[36790]: pgmap v2530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:21:09.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3210.107256 secs 2024-07-26T12:21:09.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:09.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:09.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:09.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:09.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:21:09.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:21:09.846 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:21:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:09 smithi134 ceph-mon[31966]: pgmap v2530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:21:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3210.107256 secs 2024-07-26T12:21:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:21:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:21:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:21:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:21:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:10 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:21:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:10 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:21:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:10 smithi193 ceph-mon[37473]: pgmap v2531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:21:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:10 smithi193 ceph-mon[37473]: pgmap v2532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:21:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:10 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:21:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:10 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:21:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:10 smithi134 ceph-mon[31966]: pgmap v2531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:21:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:10 smithi134 ceph-mon[31966]: pgmap v2532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:21:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:10 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:21:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:10 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:21:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:10 smithi189 ceph-mon[36790]: pgmap v2531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:21:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:10 smithi189 ceph-mon[36790]: pgmap v2532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:21:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3229.534284 secs 2024-07-26T12:21:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3229.534284 secs 2024-07-26T12:21:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3229.534284 secs 2024-07-26T12:21:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:12 smithi134 ceph-mon[31966]: pgmap v2533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:21:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:12 smithi189 ceph-mon[36790]: pgmap v2533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:21:13.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:12 smithi193 ceph-mon[37473]: pgmap v2533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:21:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:14 smithi134 ceph-mon[31966]: pgmap v2534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:21:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3215.107378 secs 2024-07-26T12:21:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:14 smithi189 ceph-mon[36790]: pgmap v2534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:21:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3215.107378 secs 2024-07-26T12:21:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:14 smithi193 ceph-mon[37473]: pgmap v2534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:21:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3215.107378 secs 2024-07-26T12:21:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:21:15.922+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:21:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:21:15.922+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:15.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:21:15.922+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:21:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:16 smithi134 ceph-mon[31966]: pgmap v2535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:21:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3234.534415 secs 2024-07-26T12:21:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:16 smithi189 ceph-mon[36790]: pgmap v2535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:21:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3234.534415 secs 2024-07-26T12:21:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:16 smithi193 ceph-mon[37473]: pgmap v2535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:21:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3234.534415 secs 2024-07-26T12:21:18.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:18 smithi193 ceph-mon[37473]: pgmap v2536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:21:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:18 smithi134 ceph-mon[31966]: pgmap v2536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:21:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:18 smithi189 ceph-mon[36790]: pgmap v2536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:21:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3220.107523 secs 2024-07-26T12:21:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3220.107523 secs 2024-07-26T12:21:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3220.107523 secs 2024-07-26T12:21:20.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:21:20.442+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:21:20.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:21:20.442+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:20.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:21:20.442+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:21:20.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:20 smithi134 ceph-mon[31966]: pgmap v2537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:21:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:20 smithi189 ceph-mon[36790]: pgmap v2537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:21:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:20 smithi193 ceph-mon[37473]: pgmap v2537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:21:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3239.534548 secs 2024-07-26T12:21:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:21 smithi134 ceph-mon[31966]: pgmap v2538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3239.534548 secs 2024-07-26T12:21:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:21 smithi189 ceph-mon[36790]: pgmap v2538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3239.534548 secs 2024-07-26T12:21:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:21 smithi193 ceph-mon[37473]: pgmap v2538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:22.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:21:22.889+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:21:22.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:21:22.889+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:21:22.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:21:22.890+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T12:21:22.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:21:22.890+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T12:21:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:24 smithi134 ceph-mon[31966]: pgmap v2539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:21:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3225.107645 secs 2024-07-26T12:21:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:24 smithi189 ceph-mon[36790]: pgmap v2539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:21:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3225.107645 secs 2024-07-26T12:21:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:24 smithi193 ceph-mon[37473]: pgmap v2539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:21:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3225.107645 secs 2024-07-26T12:21:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:26 smithi134 ceph-mon[31966]: pgmap v2540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:26 smithi189 ceph-mon[36790]: pgmap v2540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:26 smithi193 ceph-mon[37473]: pgmap v2540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3244.534682 secs 2024-07-26T12:21:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3244.534682 secs 2024-07-26T12:21:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3244.534682 secs 2024-07-26T12:21:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:28 smithi134 ceph-mon[31966]: pgmap v2541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:21:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:28 smithi189 ceph-mon[36790]: pgmap v2541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:21:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:28 smithi193 ceph-mon[37473]: pgmap v2541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:21:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3230.107775 secs 2024-07-26T12:21:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3230.107775 secs 2024-07-26T12:21:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3230.107775 secs 2024-07-26T12:21:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:30 smithi134 ceph-mon[31966]: pgmap v2542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:21:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:30 smithi189 ceph-mon[36790]: pgmap v2542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:21:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:30 smithi193 ceph-mon[37473]: pgmap v2542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:21:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:21:30.929+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:21:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:21:30.929+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:21:30.931 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:21:30.929+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3249.534767 secs 2024-07-26T12:21:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3249.534767 secs 2024-07-26T12:21:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3249.534767 secs 2024-07-26T12:21:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:32 smithi134 ceph-mon[31966]: pgmap v2543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:32 smithi189 ceph-mon[36790]: pgmap v2543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:32 smithi193 ceph-mon[37473]: pgmap v2543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:33.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:21:33.400+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:21:33.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:21:33.400+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T12:21:33.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:21:33.400+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:21:33.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:21:33.400+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T12:21:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:34 smithi134 ceph-mon[31966]: pgmap v2544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:21:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3235.107915 secs 2024-07-26T12:21:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:34 smithi189 ceph-mon[36790]: pgmap v2544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:21:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3235.107915 secs 2024-07-26T12:21:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:34 smithi193 ceph-mon[37473]: pgmap v2544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:21:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3235.107915 secs 2024-07-26T12:21:35.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:21:35.451+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:21:35.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:21:35.451+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:21:35.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:21:35.451+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:36 smithi134 ceph-mon[31966]: pgmap v2545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3254.534895 secs 2024-07-26T12:21:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:36 smithi189 ceph-mon[36790]: pgmap v2545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3254.534895 secs 2024-07-26T12:21:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:36 smithi193 ceph-mon[37473]: pgmap v2545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3254.534895 secs 2024-07-26T12:21:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:38 smithi134 ceph-mon[31966]: pgmap v2546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:21:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:38 smithi189 ceph-mon[36790]: pgmap v2546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:21:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:38 smithi193 ceph-mon[37473]: pgmap v2546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:21:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3240.108058 secs 2024-07-26T12:21:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3240.108058 secs 2024-07-26T12:21:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3240.108058 secs 2024-07-26T12:21:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:40 smithi189 ceph-mon[36790]: pgmap v2547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:21:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:40 smithi193 ceph-mon[37473]: pgmap v2547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:21:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:40 smithi134 ceph-mon[31966]: pgmap v2547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:21:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3259.535044 secs 2024-07-26T12:21:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3259.535044 secs 2024-07-26T12:21:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3259.535044 secs 2024-07-26T12:21:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:42 smithi193 ceph-mon[37473]: pgmap v2548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:42 smithi134 ceph-mon[31966]: pgmap v2548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:42 smithi189 ceph-mon[36790]: pgmap v2548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:44 smithi193 ceph-mon[37473]: pgmap v2549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:21:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3245.108208 secs 2024-07-26T12:21:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:44 smithi134 ceph-mon[31966]: pgmap v2549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:21:44.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3245.108208 secs 2024-07-26T12:21:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:44 smithi189 ceph-mon[36790]: pgmap v2549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:21:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3245.108208 secs 2024-07-26T12:21:45.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:21:45.937+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:21:45.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:21:45.937+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:21:45.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:21:45.937+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:46 smithi193 ceph-mon[37473]: pgmap v2550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:21:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3264.535191 secs 2024-07-26T12:21:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:46 smithi134 ceph-mon[31966]: pgmap v2550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:21:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3264.535191 secs 2024-07-26T12:21:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:46 smithi189 ceph-mon[36790]: pgmap v2550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:21:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3264.535191 secs 2024-07-26T12:21:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:48 smithi193 ceph-mon[37473]: pgmap v2551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:21:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:48 smithi134 ceph-mon[31966]: pgmap v2551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:21:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:48 smithi189 ceph-mon[36790]: pgmap v2551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:21:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3250.108339 secs 2024-07-26T12:21:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3250.108339 secs 2024-07-26T12:21:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3250.108339 secs 2024-07-26T12:21:50.468 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:21:50.467+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:21:50.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:21:50.467+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:21:50.469 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:21:50.467+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:21:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:50 smithi193 ceph-mon[37473]: pgmap v2552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:21:50.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:50 smithi134 ceph-mon[31966]: pgmap v2552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:21:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:50 smithi189 ceph-mon[36790]: pgmap v2552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:21:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3269.535339 secs 2024-07-26T12:21:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3269.535339 secs 2024-07-26T12:21:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3269.535339 secs 2024-07-26T12:21:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:52 smithi193 ceph-mon[37473]: pgmap v2553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:21:52.906 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:52 smithi134 ceph-mon[31966]: pgmap v2553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:21:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:52 smithi189 ceph-mon[36790]: pgmap v2553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:21:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:54 smithi193 ceph-mon[37473]: pgmap v2554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:21:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3255.108518 secs 2024-07-26T12:21:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:54 smithi134 ceph-mon[31966]: pgmap v2554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:21:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3255.108518 secs 2024-07-26T12:21:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:54 smithi189 ceph-mon[36790]: pgmap v2554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:21:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3255.108518 secs 2024-07-26T12:21:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:56 smithi193 ceph-mon[37473]: pgmap v2555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3274.535461 secs 2024-07-26T12:21:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:56 smithi134 ceph-mon[31966]: pgmap v2555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3274.535461 secs 2024-07-26T12:21:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:56 smithi189 ceph-mon[36790]: pgmap v2555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:21:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3274.535461 secs 2024-07-26T12:21:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:58 smithi193 ceph-mon[37473]: pgmap v2556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:21:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:58 smithi134 ceph-mon[31966]: pgmap v2556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:21:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:58 smithi189 ceph-mon[36790]: pgmap v2556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:21:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:21:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3260.108611 secs 2024-07-26T12:21:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:21:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3260.108611 secs 2024-07-26T12:22:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:21:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3260.108611 secs 2024-07-26T12:22:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:00 smithi134 ceph-mon[31966]: pgmap v2557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:00 smithi189 ceph-mon[36790]: pgmap v2557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:22:00.950+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:22:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:22:00.950+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:22:00.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:22:00.950+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:00 smithi193 ceph-mon[37473]: pgmap v2557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3279.535527 secs 2024-07-26T12:22:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3279.535527 secs 2024-07-26T12:22:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3279.535527 secs 2024-07-26T12:22:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:02 smithi134 ceph-mon[31966]: pgmap v2558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:02 smithi189 ceph-mon[36790]: pgmap v2558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:02 smithi193 ceph-mon[37473]: pgmap v2558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3265.108743 secs 2024-07-26T12:22:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:04 smithi134 ceph-mon[31966]: pgmap v2559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3265.108743 secs 2024-07-26T12:22:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:04 smithi189 ceph-mon[36790]: pgmap v2559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3265.108743 secs 2024-07-26T12:22:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:04 smithi193 ceph-mon[37473]: pgmap v2559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:05.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:22:05.481+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:22:05.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:22:05.481+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:22:05.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:22:05.481+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:06 smithi134 ceph-mon[31966]: pgmap v2560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3284.535607 secs 2024-07-26T12:22:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:06 smithi189 ceph-mon[36790]: pgmap v2560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3284.535607 secs 2024-07-26T12:22:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:06 smithi193 ceph-mon[37473]: pgmap v2560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3284.535607 secs 2024-07-26T12:22:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:08 smithi134 ceph-mon[31966]: pgmap v2561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:08 smithi189 ceph-mon[36790]: pgmap v2561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:08 smithi193 ceph-mon[37473]: pgmap v2561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:22:09.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3270.108838 secs 2024-07-26T12:22:09.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:22:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3270.108838 secs 2024-07-26T12:22:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:22:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3270.108838 secs 2024-07-26T12:22:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:22:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:10 smithi134 ceph-mon[31966]: pgmap v2562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:22:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:22:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:22:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:10 smithi189 ceph-mon[36790]: pgmap v2562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:22:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:22:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:22:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:10 smithi193 ceph-mon[37473]: pgmap v2562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:22:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:22:11.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:22:11.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:22:11.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:22:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:11 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:22:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:11 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:22:11.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:11 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:22:11.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:11 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:22:11.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:11 smithi134 ceph-mon[31966]: pgmap v2563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:22:11.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:11 smithi134 ceph-mon[31966]: pgmap v2564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:11.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3289.535682 secs 2024-07-26T12:22:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:11 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:22:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:11 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:22:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:11 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:22:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:11 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:22:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:11 smithi189 ceph-mon[36790]: pgmap v2563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:22:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:11 smithi189 ceph-mon[36790]: pgmap v2564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3289.535682 secs 2024-07-26T12:22:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:11 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:22:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:11 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:22:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:11 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:22:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:11 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:22:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:11 smithi193 ceph-mon[37473]: pgmap v2563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:22:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:11 smithi193 ceph-mon[37473]: pgmap v2564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3289.535682 secs 2024-07-26T12:22:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:12 smithi134 ceph-mon[31966]: pgmap v2565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:12 smithi189 ceph-mon[36790]: pgmap v2565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:12 smithi193 ceph-mon[37473]: pgmap v2565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3275.108999 secs 2024-07-26T12:22:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3275.108999 secs 2024-07-26T12:22:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3275.108999 secs 2024-07-26T12:22:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:14 smithi134 ceph-mon[31966]: pgmap v2566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:22:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:14 smithi189 ceph-mon[36790]: pgmap v2566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:22:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:14 smithi193 ceph-mon[37473]: pgmap v2566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:22:15.967 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:22:15.966+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:22:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:22:15.966+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:22:15.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:22:15.966+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3294.535808 secs 2024-07-26T12:22:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3294.535808 secs 2024-07-26T12:22:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3294.535808 secs 2024-07-26T12:22:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:17 smithi193 ceph-mon[37473]: pgmap v2567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:17 smithi134 ceph-mon[31966]: pgmap v2567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:17 smithi189 ceph-mon[36790]: pgmap v2567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3280.109150 secs 2024-07-26T12:22:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:19 smithi134 ceph-mon[31966]: pgmap v2568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3280.109150 secs 2024-07-26T12:22:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:19 smithi189 ceph-mon[36790]: pgmap v2568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3280.109150 secs 2024-07-26T12:22:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:19 smithi193 ceph-mon[37473]: pgmap v2568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:22:20.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:22:20.496+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:22:20.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:22:20.496+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:22:20.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:22:20.496+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:21 smithi134 ceph-mon[31966]: pgmap v2569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:22:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3299.535936 secs 2024-07-26T12:22:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:21 smithi189 ceph-mon[36790]: pgmap v2569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:22:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3299.535936 secs 2024-07-26T12:22:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:21 smithi193 ceph-mon[37473]: pgmap v2569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:22:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3299.535936 secs 2024-07-26T12:22:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:23 smithi134 ceph-mon[31966]: pgmap v2570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:23 smithi189 ceph-mon[36790]: pgmap v2570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:23 smithi193 ceph-mon[37473]: pgmap v2570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3285.109221 secs 2024-07-26T12:22:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3285.109221 secs 2024-07-26T12:22:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3285.109221 secs 2024-07-26T12:22:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:25 smithi134 ceph-mon[31966]: pgmap v2571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:22:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:25 smithi189 ceph-mon[36790]: pgmap v2571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:22:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:25 smithi193 ceph-mon[37473]: pgmap v2571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:22:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3304.536085 secs 2024-07-26T12:22:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3304.536085 secs 2024-07-26T12:22:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3304.536085 secs 2024-07-26T12:22:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:27 smithi134 ceph-mon[31966]: pgmap v2572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:27 smithi189 ceph-mon[36790]: pgmap v2572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:27 smithi193 ceph-mon[37473]: pgmap v2572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3290.109343 secs 2024-07-26T12:22:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:29 smithi134 ceph-mon[31966]: pgmap v2573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3290.109343 secs 2024-07-26T12:22:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:29 smithi189 ceph-mon[36790]: pgmap v2573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3290.109343 secs 2024-07-26T12:22:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:29 smithi193 ceph-mon[37473]: pgmap v2573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:30 smithi134 ceph-mon[31966]: pgmap v2574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:30 smithi189 ceph-mon[36790]: pgmap v2574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:30.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:22:30.968+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:22:30.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:22:30.968+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:30.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:22:30.968+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:22:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:30 smithi193 ceph-mon[37473]: pgmap v2574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3309.536216 secs 2024-07-26T12:22:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3309.536216 secs 2024-07-26T12:22:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3309.536216 secs 2024-07-26T12:22:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:32 smithi134 ceph-mon[31966]: pgmap v2575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:32 smithi189 ceph-mon[36790]: pgmap v2575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:32 smithi193 ceph-mon[37473]: pgmap v2575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3295.109475 secs 2024-07-26T12:22:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3295.109475 secs 2024-07-26T12:22:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3295.109475 secs 2024-07-26T12:22:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:34 smithi134 ceph-mon[31966]: pgmap v2576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:22:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:34 smithi189 ceph-mon[36790]: pgmap v2576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:22:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:34 smithi193 ceph-mon[37473]: pgmap v2576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:22:35.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:22:35.504+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:22:35.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:22:35.504+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:35.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:22:35.504+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:22:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3314.536275 secs 2024-07-26T12:22:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3314.536275 secs 2024-07-26T12:22:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3314.536275 secs 2024-07-26T12:22:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:37 smithi193 ceph-mon[37473]: pgmap v2577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:37 smithi134 ceph-mon[31966]: pgmap v2577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:37 smithi189 ceph-mon[36790]: pgmap v2577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3300.109577 secs 2024-07-26T12:22:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:39 smithi134 ceph-mon[31966]: pgmap v2578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3300.109577 secs 2024-07-26T12:22:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:39 smithi189 ceph-mon[36790]: pgmap v2578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3300.109577 secs 2024-07-26T12:22:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:39 smithi193 ceph-mon[37473]: pgmap v2578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:41 smithi134 ceph-mon[31966]: pgmap v2579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3319.536401 secs 2024-07-26T12:22:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:41 smithi189 ceph-mon[36790]: pgmap v2579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3319.536401 secs 2024-07-26T12:22:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:41 smithi193 ceph-mon[37473]: pgmap v2579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3319.536401 secs 2024-07-26T12:22:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:43 smithi134 ceph-mon[31966]: pgmap v2580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:43 smithi189 ceph-mon[36790]: pgmap v2580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:43 smithi193 ceph-mon[37473]: pgmap v2580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3305.109662 secs 2024-07-26T12:22:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3305.109662 secs 2024-07-26T12:22:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3305.109662 secs 2024-07-26T12:22:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:45 smithi134 ceph-mon[31966]: pgmap v2581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:22:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:45 smithi189 ceph-mon[36790]: pgmap v2581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:22:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:22:45.971+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:22:45.972 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:22:45.971+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:22:45.973 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:22:45.971+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:45 smithi193 ceph-mon[37473]: pgmap v2581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:22:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3324.536496 secs 2024-07-26T12:22:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3324.536496 secs 2024-07-26T12:22:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3324.536496 secs 2024-07-26T12:22:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:47 smithi134 ceph-mon[31966]: pgmap v2582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:47 smithi189 ceph-mon[36790]: pgmap v2582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:47 smithi193 ceph-mon[37473]: pgmap v2582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:48 smithi134 ceph-mon[31966]: pgmap v2583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:48 smithi189 ceph-mon[36790]: pgmap v2583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:48 smithi193 ceph-mon[37473]: pgmap v2583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3310.109764 secs 2024-07-26T12:22:49.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3310.109764 secs 2024-07-26T12:22:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3310.109764 secs 2024-07-26T12:22:50.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:22:50.507+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:22:50.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:22:50.507+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:22:50.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:22:50.507+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:22:50.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:50 smithi134 ceph-mon[31966]: pgmap v2584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:50 smithi189 ceph-mon[36790]: pgmap v2584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:50 smithi193 ceph-mon[37473]: pgmap v2584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3329.536616 secs 2024-07-26T12:22:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3329.536616 secs 2024-07-26T12:22:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3329.536616 secs 2024-07-26T12:22:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:52 smithi134 ceph-mon[31966]: pgmap v2585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:52 smithi189 ceph-mon[36790]: pgmap v2585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:52 smithi193 ceph-mon[37473]: pgmap v2585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3315.109893 secs 2024-07-26T12:22:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3315.109893 secs 2024-07-26T12:22:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3315.109893 secs 2024-07-26T12:22:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:54 smithi134 ceph-mon[31966]: pgmap v2586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:22:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:54 smithi189 ceph-mon[36790]: pgmap v2586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:22:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:54 smithi193 ceph-mon[37473]: pgmap v2586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:22:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3334.536695 secs 2024-07-26T12:22:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3334.536695 secs 2024-07-26T12:22:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3334.536695 secs 2024-07-26T12:22:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:57 smithi193 ceph-mon[37473]: pgmap v2587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:57 smithi134 ceph-mon[31966]: pgmap v2587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:57 smithi189 ceph-mon[36790]: pgmap v2587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:22:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3320.110021 secs 2024-07-26T12:22:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:22:59 smithi134 ceph-mon[31966]: pgmap v2588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:22:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3320.110021 secs 2024-07-26T12:22:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:22:59 smithi189 ceph-mon[36790]: pgmap v2588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3320.110021 secs 2024-07-26T12:23:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:22:59 smithi193 ceph-mon[37473]: pgmap v2588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:23:00.979+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:23:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:23:00.979+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:00.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:23:00.979+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:23:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:01 smithi134 ceph-mon[31966]: pgmap v2589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3339.536831 secs 2024-07-26T12:23:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:01 smithi189 ceph-mon[36790]: pgmap v2589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3339.536831 secs 2024-07-26T12:23:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:01 smithi193 ceph-mon[37473]: pgmap v2589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3339.536831 secs 2024-07-26T12:23:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:03 smithi134 ceph-mon[31966]: pgmap v2590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:03 smithi189 ceph-mon[36790]: pgmap v2590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:03 smithi193 ceph-mon[37473]: pgmap v2590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3325.110097 secs 2024-07-26T12:23:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3325.110097 secs 2024-07-26T12:23:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3325.110097 secs 2024-07-26T12:23:05.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:23:05.510+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:23:05.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:23:05.510+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:23:05.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:23:05.510+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:05 smithi134 ceph-mon[31966]: pgmap v2591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:05 smithi189 ceph-mon[36790]: pgmap v2591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:05 smithi193 ceph-mon[37473]: pgmap v2591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3344.536975 secs 2024-07-26T12:23:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:06 smithi134 ceph-mon[31966]: pgmap v2592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3344.536975 secs 2024-07-26T12:23:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:06 smithi189 ceph-mon[36790]: pgmap v2592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3344.536975 secs 2024-07-26T12:23:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:06 smithi193 ceph-mon[37473]: pgmap v2592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3330.110224 secs 2024-07-26T12:23:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:09 smithi134 ceph-mon[31966]: pgmap v2593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3330.110224 secs 2024-07-26T12:23:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:09 smithi189 ceph-mon[36790]: pgmap v2593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3330.110224 secs 2024-07-26T12:23:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:09 smithi193 ceph-mon[37473]: pgmap v2593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:11 smithi134 ceph-mon[31966]: pgmap v2594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:23:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3349.537037 secs 2024-07-26T12:23:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:11 smithi189 ceph-mon[36790]: pgmap v2594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:23:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3349.537037 secs 2024-07-26T12:23:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:11 smithi193 ceph-mon[37473]: pgmap v2594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:23:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3349.537037 secs 2024-07-26T12:23:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:23:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:23:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:23:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:23:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:23:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:23:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:23:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:23:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:23:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:13 smithi134 ceph-mon[31966]: pgmap v2595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:23:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:13 smithi134 ceph-mon[31966]: pgmap v2596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:23:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:13 smithi189 ceph-mon[36790]: pgmap v2595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:23:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:13 smithi189 ceph-mon[36790]: pgmap v2596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:23:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:13 smithi193 ceph-mon[37473]: pgmap v2595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:23:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:13 smithi193 ceph-mon[37473]: pgmap v2596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:23:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3335.110341 secs 2024-07-26T12:23:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:14 smithi134 ceph-mon[31966]: pgmap v2597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:23:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3335.110341 secs 2024-07-26T12:23:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:14 smithi189 ceph-mon[36790]: pgmap v2597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:23:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3335.110341 secs 2024-07-26T12:23:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:14 smithi193 ceph-mon[37473]: pgmap v2597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:23:15.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:23:15.987+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:23:15.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:23:15.987+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:15.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:23:15.987+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:23:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:16 smithi193 ceph-mon[37473]: pgmap v2598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:23:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3354.537121 secs 2024-07-26T12:23:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:16 smithi134 ceph-mon[31966]: pgmap v2598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:23:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3354.537121 secs 2024-07-26T12:23:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:16 smithi189 ceph-mon[36790]: pgmap v2598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:23:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3354.537121 secs 2024-07-26T12:23:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:18 smithi193 ceph-mon[37473]: pgmap v2599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:23:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:18 smithi134 ceph-mon[31966]: pgmap v2599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:23:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:18 smithi189 ceph-mon[36790]: pgmap v2599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:23:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3340.110477 secs 2024-07-26T12:23:20.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3340.110477 secs 2024-07-26T12:23:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3340.110477 secs 2024-07-26T12:23:20.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:23:20.519+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:23:20.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:23:20.519+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:20.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:23:20.519+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:23:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:20 smithi134 ceph-mon[31966]: pgmap v2600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:23:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:20 smithi193 ceph-mon[37473]: pgmap v2600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:23:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:20 smithi189 ceph-mon[36790]: pgmap v2600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:23:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3359.537237 secs 2024-07-26T12:23:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3359.537237 secs 2024-07-26T12:23:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3359.537237 secs 2024-07-26T12:23:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:22 smithi134 ceph-mon[31966]: pgmap v2601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:23:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:22 smithi189 ceph-mon[36790]: pgmap v2601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:23:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:22 smithi193 ceph-mon[37473]: pgmap v2601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:23:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3345.110621 secs 2024-07-26T12:23:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:24 smithi134 ceph-mon[31966]: pgmap v2602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3345.110621 secs 2024-07-26T12:23:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:24 smithi189 ceph-mon[36790]: pgmap v2602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3345.110621 secs 2024-07-26T12:23:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:24 smithi193 ceph-mon[37473]: pgmap v2602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:26 smithi134 ceph-mon[31966]: pgmap v2603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3364.537339 secs 2024-07-26T12:23:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:26 smithi189 ceph-mon[36790]: pgmap v2603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3364.537339 secs 2024-07-26T12:23:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:26 smithi193 ceph-mon[37473]: pgmap v2603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3364.537339 secs 2024-07-26T12:23:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:28 smithi134 ceph-mon[31966]: pgmap v2604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:28 smithi189 ceph-mon[36790]: pgmap v2604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:28 smithi193 ceph-mon[37473]: pgmap v2604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3350.110787 secs 2024-07-26T12:23:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3350.110787 secs 2024-07-26T12:23:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3350.110787 secs 2024-07-26T12:23:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:23:31.003+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:23:31.004 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:23:31.003+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:23:31.005 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:23:31.003+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:30 smithi134 ceph-mon[31966]: pgmap v2605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:30 smithi189 ceph-mon[36790]: pgmap v2605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:30 smithi193 ceph-mon[37473]: pgmap v2605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3369.537461 secs 2024-07-26T12:23:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3369.537461 secs 2024-07-26T12:23:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3369.537461 secs 2024-07-26T12:23:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:32 smithi134 ceph-mon[31966]: pgmap v2606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:32 smithi189 ceph-mon[36790]: pgmap v2606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:32 smithi193 ceph-mon[37473]: pgmap v2606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3355.110924 secs 2024-07-26T12:23:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:34 smithi134 ceph-mon[31966]: pgmap v2607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:23:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3355.110924 secs 2024-07-26T12:23:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:34 smithi189 ceph-mon[36790]: pgmap v2607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:23:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3355.110924 secs 2024-07-26T12:23:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:34 smithi193 ceph-mon[37473]: pgmap v2607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:23:35.535 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:23:35.532+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:23:35.535 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:23:35.532+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:23:35.535 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:23:35.532+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:36 smithi134 ceph-mon[31966]: pgmap v2608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:23:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3374.537564 secs 2024-07-26T12:23:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:36 smithi189 ceph-mon[36790]: pgmap v2608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:23:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3374.537564 secs 2024-07-26T12:23:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:36 smithi193 ceph-mon[37473]: pgmap v2608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:23:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3374.537564 secs 2024-07-26T12:23:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:38 smithi134 ceph-mon[31966]: pgmap v2609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:23:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:38 smithi189 ceph-mon[36790]: pgmap v2609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:23:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:38 smithi193 ceph-mon[37473]: pgmap v2609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:23:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3360.111082 secs 2024-07-26T12:23:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3360.111082 secs 2024-07-26T12:23:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3360.111082 secs 2024-07-26T12:23:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:40 smithi134 ceph-mon[31966]: pgmap v2610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:23:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:40 smithi189 ceph-mon[36790]: pgmap v2610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:23:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:40 smithi193 ceph-mon[37473]: pgmap v2610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:23:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3379.537689 secs 2024-07-26T12:23:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3379.537689 secs 2024-07-26T12:23:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3379.537689 secs 2024-07-26T12:23:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:42 smithi134 ceph-mon[31966]: pgmap v2611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:23:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:42 smithi189 ceph-mon[36790]: pgmap v2611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:23:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:42 smithi193 ceph-mon[37473]: pgmap v2611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:23:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3365.111197 secs 2024-07-26T12:23:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3365.111197 secs 2024-07-26T12:23:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3365.111197 secs 2024-07-26T12:23:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:44 smithi134 ceph-mon[31966]: pgmap v2612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:23:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:44 smithi189 ceph-mon[36790]: pgmap v2612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:23:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:44 smithi193 ceph-mon[37473]: pgmap v2612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:23:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:23:46.007+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:23:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:23:46.007+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:23:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:23:46.007+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:46 smithi134 ceph-mon[31966]: pgmap v2613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3384.537777 secs 2024-07-26T12:23:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:46 smithi189 ceph-mon[36790]: pgmap v2613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3384.537777 secs 2024-07-26T12:23:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:46 smithi193 ceph-mon[37473]: pgmap v2613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3384.537777 secs 2024-07-26T12:23:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:48 smithi134 ceph-mon[31966]: pgmap v2614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3370.111337 secs 2024-07-26T12:23:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:48 smithi189 ceph-mon[36790]: pgmap v2614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3370.111337 secs 2024-07-26T12:23:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:48 smithi193 ceph-mon[37473]: pgmap v2614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3370.111337 secs 2024-07-26T12:23:50.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:23:50.540+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:23:50.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:23:50.540+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:23:50.543 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:23:50.540+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:23:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:50 smithi134 ceph-mon[31966]: pgmap v2615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:50 smithi189 ceph-mon[36790]: pgmap v2615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:50 smithi193 ceph-mon[37473]: pgmap v2615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:23:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3389.537894 secs 2024-07-26T12:23:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3389.537894 secs 2024-07-26T12:23:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3389.537894 secs 2024-07-26T12:23:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:52 smithi189 ceph-mon[36790]: pgmap v2616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:52 smithi193 ceph-mon[37473]: pgmap v2616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:52 smithi134 ceph-mon[31966]: pgmap v2616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3375.111484 secs 2024-07-26T12:23:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3375.111484 secs 2024-07-26T12:23:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3375.111484 secs 2024-07-26T12:23:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:54 smithi189 ceph-mon[36790]: pgmap v2617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:54 smithi193 ceph-mon[37473]: pgmap v2617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:54 smithi134 ceph-mon[31966]: pgmap v2617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:23:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:56 smithi189 ceph-mon[36790]: pgmap v2618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3394.538005 secs 2024-07-26T12:23:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:56 smithi193 ceph-mon[37473]: pgmap v2618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3394.538005 secs 2024-07-26T12:23:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:56 smithi134 ceph-mon[31966]: pgmap v2618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:23:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3394.538005 secs 2024-07-26T12:23:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:58 smithi193 ceph-mon[37473]: pgmap v2619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:23:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:23:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3380.111628 secs 2024-07-26T12:23:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:58 smithi134 ceph-mon[31966]: pgmap v2619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:23:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:23:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3380.111628 secs 2024-07-26T12:23:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:58 smithi189 ceph-mon[36790]: pgmap v2619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:23:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:23:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3380.111628 secs 2024-07-26T12:24:01.024 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:24:01.022+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:24:01.025 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:24:01.023+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:24:01.025 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:24:01.023+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:00 smithi193 ceph-mon[37473]: pgmap v2620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:24:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:00 smithi134 ceph-mon[31966]: pgmap v2620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:24:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:00 smithi189 ceph-mon[36790]: pgmap v2620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:24:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3399.538126 secs 2024-07-26T12:24:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3399.538126 secs 2024-07-26T12:24:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3399.538126 secs 2024-07-26T12:24:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:02 smithi193 ceph-mon[37473]: pgmap v2621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:24:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:02 smithi134 ceph-mon[31966]: pgmap v2621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:24:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:02 smithi189 ceph-mon[36790]: pgmap v2621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:24:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3385.111757 secs 2024-07-26T12:24:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3385.111757 secs 2024-07-26T12:24:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3385.111757 secs 2024-07-26T12:24:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:04 smithi193 ceph-mon[37473]: pgmap v2622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:24:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:04 smithi134 ceph-mon[31966]: pgmap v2622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:24:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:04 smithi189 ceph-mon[36790]: pgmap v2622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:24:05.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:24:05.556+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:24:05.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:24:05.556+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:05.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:24:05.556+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:24:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:06 smithi193 ceph-mon[37473]: pgmap v2623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:24:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3404.538205 secs 2024-07-26T12:24:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:06 smithi134 ceph-mon[31966]: pgmap v2623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:24:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3404.538205 secs 2024-07-26T12:24:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:06 smithi189 ceph-mon[36790]: pgmap v2623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:24:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3404.538205 secs 2024-07-26T12:24:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:08 smithi193 ceph-mon[37473]: pgmap v2624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:24:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3390.111841 secs 2024-07-26T12:24:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:08 smithi134 ceph-mon[31966]: pgmap v2624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:24:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3390.111841 secs 2024-07-26T12:24:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:08 smithi189 ceph-mon[36790]: pgmap v2624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:24:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3390.111841 secs 2024-07-26T12:24:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:11 smithi193 ceph-mon[37473]: pgmap v2625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:24:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:11 smithi134 ceph-mon[31966]: pgmap v2625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:24:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:11 smithi189 ceph-mon[36790]: pgmap v2625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:24:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3409.538322 secs 2024-07-26T12:24:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:24:12.322 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3409.538322 secs 2024-07-26T12:24:12.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:24:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3409.538322 secs 2024-07-26T12:24:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:24:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:13 smithi193 ceph-mon[37473]: pgmap v2626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:24:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:24:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:24:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:24:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:13 smithi134 ceph-mon[31966]: pgmap v2626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:24:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:24:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:24:13.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:24:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:13 smithi189 ceph-mon[36790]: pgmap v2626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:24:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:24:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:24:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:24:14.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3395.112009 secs 2024-07-26T12:24:14.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:14 smithi193 ceph-mon[37473]: pgmap v2627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T12:24:14.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:14 smithi193 ceph-mon[37473]: pgmap v2628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:24:14.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:24:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3395.112009 secs 2024-07-26T12:24:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:14 smithi134 ceph-mon[31966]: pgmap v2627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T12:24:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:14 smithi134 ceph-mon[31966]: pgmap v2628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:24:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:24:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3395.112009 secs 2024-07-26T12:24:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:14 smithi189 ceph-mon[36790]: pgmap v2627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T12:24:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:14 smithi189 ceph-mon[36790]: pgmap v2628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:24:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:24:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:24:16.029+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:24:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:24:16.029+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:24:16.030 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:24:16.029+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:16 smithi193 ceph-mon[37473]: pgmap v2629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:24:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3414.538413 secs 2024-07-26T12:24:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:16 smithi134 ceph-mon[31966]: pgmap v2629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:24:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3414.538413 secs 2024-07-26T12:24:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:16 smithi189 ceph-mon[36790]: pgmap v2629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:24:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3414.538413 secs 2024-07-26T12:24:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:18 smithi193 ceph-mon[37473]: pgmap v2630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T12:24:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:18 smithi134 ceph-mon[31966]: pgmap v2630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T12:24:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:18 smithi189 ceph-mon[36790]: pgmap v2630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T12:24:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3400.112096 secs 2024-07-26T12:24:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3400.112096 secs 2024-07-26T12:24:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3400.112096 secs 2024-07-26T12:24:20.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:24:20.571+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:24:20.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:24:20.571+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:24:20.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:24:20.571+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:20 smithi193 ceph-mon[37473]: pgmap v2631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:24:20.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:20 smithi134 ceph-mon[31966]: pgmap v2631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:24:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:20 smithi189 ceph-mon[36790]: pgmap v2631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:24:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3419.538533 secs 2024-07-26T12:24:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3419.538533 secs 2024-07-26T12:24:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3419.538533 secs 2024-07-26T12:24:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:22 smithi193 ceph-mon[37473]: pgmap v2632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:24:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:22 smithi134 ceph-mon[31966]: pgmap v2632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:24:22.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:24:22.920+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:24:22.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:24:22.921+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:24:22.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:24:22.921+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:24:22.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:24:22.921+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:24:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:22 smithi189 ceph-mon[36790]: pgmap v2632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:24:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3405.112163 secs 2024-07-26T12:24:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:24 smithi193 ceph-mon[37473]: pgmap v2633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:24:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3405.112163 secs 2024-07-26T12:24:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:24 smithi134 ceph-mon[31966]: pgmap v2633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:24:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3405.112163 secs 2024-07-26T12:24:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:24 smithi189 ceph-mon[36790]: pgmap v2633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:24:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:26 smithi193 ceph-mon[37473]: pgmap v2634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3424.538648 secs 2024-07-26T12:24:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:26 smithi134 ceph-mon[31966]: pgmap v2634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3424.538648 secs 2024-07-26T12:24:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:26 smithi189 ceph-mon[36790]: pgmap v2634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3424.538648 secs 2024-07-26T12:24:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:28 smithi134 ceph-mon[31966]: pgmap v2635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:28 smithi189 ceph-mon[36790]: pgmap v2635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:28 smithi193 ceph-mon[37473]: pgmap v2635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3410.112250 secs 2024-07-26T12:24:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3410.112250 secs 2024-07-26T12:24:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3410.112250 secs 2024-07-26T12:24:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:30 smithi134 ceph-mon[31966]: pgmap v2636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:30 smithi189 ceph-mon[36790]: pgmap v2636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:30 smithi193 ceph-mon[37473]: pgmap v2636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:31.045 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:24:31.044+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:24:31.046 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:24:31.044+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:31.046 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:24:31.044+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:24:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3429.538716 secs 2024-07-26T12:24:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3429.538716 secs 2024-07-26T12:24:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3429.538716 secs 2024-07-26T12:24:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:32 smithi134 ceph-mon[31966]: pgmap v2637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:32 smithi189 ceph-mon[36790]: pgmap v2637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:32 smithi193 ceph-mon[37473]: pgmap v2637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:33.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:24:33.431+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:24:33.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:24:33.431+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T12:24:33.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:24:33.432+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:24:33.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:24:33.432+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T12:24:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3415.112365 secs 2024-07-26T12:24:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:34 smithi134 ceph-mon[31966]: pgmap v2638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3415.112365 secs 2024-07-26T12:24:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:34 smithi189 ceph-mon[36790]: pgmap v2638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3415.112365 secs 2024-07-26T12:24:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:34 smithi193 ceph-mon[37473]: pgmap v2638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:35.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:24:35.587+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:24:35.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:24:35.587+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:24:35.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:24:35.587+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:36 smithi134 ceph-mon[31966]: pgmap v2639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3434.538815 secs 2024-07-26T12:24:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:36 smithi189 ceph-mon[36790]: pgmap v2639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3434.538815 secs 2024-07-26T12:24:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:36 smithi193 ceph-mon[37473]: pgmap v2639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3434.538815 secs 2024-07-26T12:24:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:38 smithi134 ceph-mon[31966]: pgmap v2640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:38 smithi189 ceph-mon[36790]: pgmap v2640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:38 smithi193 ceph-mon[37473]: pgmap v2640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3420.112497 secs 2024-07-26T12:24:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3420.112497 secs 2024-07-26T12:24:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3420.112497 secs 2024-07-26T12:24:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:40 smithi134 ceph-mon[31966]: pgmap v2641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:40 smithi189 ceph-mon[36790]: pgmap v2641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:40 smithi193 ceph-mon[37473]: pgmap v2641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3439.538959 secs 2024-07-26T12:24:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3439.538959 secs 2024-07-26T12:24:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3439.538959 secs 2024-07-26T12:24:42.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:42 smithi134 ceph-mon[31966]: pgmap v2642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:42 smithi189 ceph-mon[36790]: pgmap v2642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:42 smithi193 ceph-mon[37473]: pgmap v2642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3425.112633 secs 2024-07-26T12:24:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:44 smithi134 ceph-mon[31966]: pgmap v2643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3425.112633 secs 2024-07-26T12:24:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:44 smithi189 ceph-mon[36790]: pgmap v2643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3425.112633 secs 2024-07-26T12:24:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:44 smithi193 ceph-mon[37473]: pgmap v2643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:45 smithi134 ceph-mon[31966]: pgmap v2644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:45 smithi189 ceph-mon[36790]: pgmap v2644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:45 smithi193 ceph-mon[37473]: pgmap v2644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:46.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:24:46.051+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:24:46.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:24:46.051+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:46.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:24:46.051+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:24:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3444.539086 secs 2024-07-26T12:24:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3444.539086 secs 2024-07-26T12:24:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3444.539086 secs 2024-07-26T12:24:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:47 smithi134 ceph-mon[31966]: pgmap v2645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:47 smithi189 ceph-mon[36790]: pgmap v2645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:47 smithi193 ceph-mon[37473]: pgmap v2645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3430.112754 secs 2024-07-26T12:24:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3430.112754 secs 2024-07-26T12:24:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3430.112754 secs 2024-07-26T12:24:50.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:49 smithi193 ceph-mon[37473]: pgmap v2646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:49 smithi189 ceph-mon[36790]: pgmap v2646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:49 smithi134 ceph-mon[31966]: pgmap v2646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:24:50.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:24:50.602+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:24:50.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:24:50.602+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:24:50.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:24:50.602+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:24:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3449.539205 secs 2024-07-26T12:24:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3449.539205 secs 2024-07-26T12:24:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3449.539205 secs 2024-07-26T12:24:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:52 smithi193 ceph-mon[37473]: pgmap v2647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:52 smithi134 ceph-mon[31966]: pgmap v2647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:52 smithi189 ceph-mon[36790]: pgmap v2647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3435.112910 secs 2024-07-26T12:24:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:54 smithi134 ceph-mon[31966]: pgmap v2648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3435.112910 secs 2024-07-26T12:24:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:54 smithi189 ceph-mon[36790]: pgmap v2648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3435.112910 secs 2024-07-26T12:24:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:54 smithi193 ceph-mon[37473]: pgmap v2648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:24:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:56 smithi134 ceph-mon[31966]: pgmap v2649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3454.539359 secs 2024-07-26T12:24:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:56 smithi189 ceph-mon[36790]: pgmap v2649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3454.539359 secs 2024-07-26T12:24:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:56 smithi193 ceph-mon[37473]: pgmap v2649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:24:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3454.539359 secs 2024-07-26T12:24:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:58 smithi134 ceph-mon[31966]: pgmap v2650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:58 smithi189 ceph-mon[36790]: pgmap v2650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:58 smithi193 ceph-mon[37473]: pgmap v2650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:24:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:24:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3440.113056 secs 2024-07-26T12:24:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:24:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3440.113056 secs 2024-07-26T12:25:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:24:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3440.113056 secs 2024-07-26T12:25:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:00 smithi134 ceph-mon[31966]: pgmap v2651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:00 smithi189 ceph-mon[36790]: pgmap v2651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:00 smithi193 ceph-mon[37473]: pgmap v2651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:01.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:25:01.066+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:25:01.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:25:01.066+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:01.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:25:01.066+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:25:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3459.539491 secs 2024-07-26T12:25:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3459.539491 secs 2024-07-26T12:25:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3459.539491 secs 2024-07-26T12:25:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:02 smithi134 ceph-mon[31966]: pgmap v2652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:02 smithi189 ceph-mon[36790]: pgmap v2652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:02 smithi193 ceph-mon[37473]: pgmap v2652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3445.113197 secs 2024-07-26T12:25:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:04 smithi134 ceph-mon[31966]: pgmap v2653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3445.113197 secs 2024-07-26T12:25:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:04 smithi189 ceph-mon[36790]: pgmap v2653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3445.113197 secs 2024-07-26T12:25:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:04 smithi193 ceph-mon[37473]: pgmap v2653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:05.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:25:05.618+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:25:05.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:25:05.618+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:05.620 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:25:05.618+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:25:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:06 smithi134 ceph-mon[31966]: pgmap v2654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3464.539585 secs 2024-07-26T12:25:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:06 smithi189 ceph-mon[36790]: pgmap v2654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3464.539585 secs 2024-07-26T12:25:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:06 smithi193 ceph-mon[37473]: pgmap v2654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3464.539585 secs 2024-07-26T12:25:07.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:25:07.929+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:25:07.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:25:07.929+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:25:07.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:25:07.929+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:25:07.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:25:07.929+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:25:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:08 smithi134 ceph-mon[31966]: pgmap v2655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:08 smithi189 ceph-mon[36790]: pgmap v2655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:08 smithi193 ceph-mon[37473]: pgmap v2655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3450.113331 secs 2024-07-26T12:25:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3450.113331 secs 2024-07-26T12:25:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3450.113331 secs 2024-07-26T12:25:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:10 smithi134 ceph-mon[31966]: pgmap v2656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:10 smithi189 ceph-mon[36790]: pgmap v2656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:10 smithi193 ceph-mon[37473]: pgmap v2656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:25:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3469.539713 secs 2024-07-26T12:25:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:11 smithi193 ceph-mon[37473]: pgmap v2657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3469.539713 secs 2024-07-26T12:25:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:11 smithi134 ceph-mon[31966]: pgmap v2657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3469.539713 secs 2024-07-26T12:25:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:11 smithi189 ceph-mon[36790]: pgmap v2657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:14.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3455.113461 secs 2024-07-26T12:25:14.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:14 smithi134 ceph-mon[31966]: pgmap v2658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:14.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:25:14.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:14.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:14.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:14.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:14.824 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3455.113461 secs 2024-07-26T12:25:14.825 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:14 smithi193 ceph-mon[37473]: pgmap v2658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:14.825 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:25:14.825 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:14.825 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:14.825 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:14.825 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:14.889 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3455.113461 secs 2024-07-26T12:25:14.889 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:14 smithi189 ceph-mon[36790]: pgmap v2658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:14.889 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:25:14.890 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:14.890 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:14.890 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:14.890 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:16.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:25:16.082+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:25:16.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:25:16.082+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:25:16.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:25:16.082+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:16 smithi134 ceph-mon[31966]: pgmap v2659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:16.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3474.539780 secs 2024-07-26T12:25:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:16 smithi189 ceph-mon[36790]: pgmap v2659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3474.539780 secs 2024-07-26T12:25:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:16 smithi193 ceph-mon[37473]: pgmap v2659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3474.539780 secs 2024-07-26T12:25:18.440 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:25:18.439+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T12:25:18.440 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:25:18.439+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:25:18.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:25:18.439+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T12:25:18.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:25:18.440+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:25:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:18 smithi134 ceph-mon[31966]: pgmap v2660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:18 smithi189 ceph-mon[36790]: pgmap v2660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:18 smithi193 ceph-mon[37473]: pgmap v2660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3460.113592 secs 2024-07-26T12:25:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:25:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:25:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:19 smithi134 ceph-mon[31966]: pgmap v2661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:25:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:19 smithi134 ceph-mon[31966]: pgmap v2662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:25:19.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3460.113592 secs 2024-07-26T12:25:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:25:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:25:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:19 smithi189 ceph-mon[36790]: pgmap v2661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:25:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:19 smithi189 ceph-mon[36790]: pgmap v2662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:25:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3460.113592 secs 2024-07-26T12:25:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:25:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:25:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:19 smithi193 ceph-mon[37473]: pgmap v2661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:25:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:19 smithi193 ceph-mon[37473]: pgmap v2662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:25:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:25:20.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:25:20.623+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:25:20.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:25:20.623+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:25:20.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:25:20.623+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:21 smithi134 ceph-mon[31966]: pgmap v2663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:25:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3479.539924 secs 2024-07-26T12:25:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:21 smithi189 ceph-mon[36790]: pgmap v2663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:25:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3479.539924 secs 2024-07-26T12:25:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:21 smithi193 ceph-mon[37473]: pgmap v2663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:25:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3479.539924 secs 2024-07-26T12:25:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:23 smithi134 ceph-mon[31966]: pgmap v2664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:25:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:23 smithi189 ceph-mon[36790]: pgmap v2664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:25:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:23 smithi193 ceph-mon[37473]: pgmap v2664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:25:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3465.113722 secs 2024-07-26T12:25:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3465.113722 secs 2024-07-26T12:25:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3465.113722 secs 2024-07-26T12:25:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:25 smithi134 ceph-mon[31966]: pgmap v2665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:25:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:25 smithi189 ceph-mon[36790]: pgmap v2665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:25:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:25 smithi193 ceph-mon[37473]: pgmap v2665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:25:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3484.540049 secs 2024-07-26T12:25:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:26 smithi134 ceph-mon[31966]: pgmap v2666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:25:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3484.540049 secs 2024-07-26T12:25:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:26 smithi189 ceph-mon[36790]: pgmap v2666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:25:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3484.540049 secs 2024-07-26T12:25:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:26 smithi193 ceph-mon[37473]: pgmap v2666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:25:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3470.113860 secs 2024-07-26T12:25:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:29 smithi134 ceph-mon[31966]: pgmap v2667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:25:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3470.113860 secs 2024-07-26T12:25:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:29 smithi189 ceph-mon[36790]: pgmap v2667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:25:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3470.113860 secs 2024-07-26T12:25:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:29 smithi193 ceph-mon[37473]: pgmap v2667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:25:31.091 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:25:31.089+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:25:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:25:31.090+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:25:31.092 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:25:31.090+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:31 smithi134 ceph-mon[31966]: pgmap v2668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3489.540200 secs 2024-07-26T12:25:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:31 smithi189 ceph-mon[36790]: pgmap v2668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3489.540200 secs 2024-07-26T12:25:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:31 smithi193 ceph-mon[37473]: pgmap v2668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3489.540200 secs 2024-07-26T12:25:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:33 smithi134 ceph-mon[31966]: pgmap v2669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:33 smithi189 ceph-mon[36790]: pgmap v2669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:33 smithi193 ceph-mon[37473]: pgmap v2669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3475.114008 secs 2024-07-26T12:25:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3475.114008 secs 2024-07-26T12:25:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3475.114008 secs 2024-07-26T12:25:35.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:25:35.627+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:25:35.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:25:35.627+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:25:35.629 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:25:35.627+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:35 smithi134 ceph-mon[31966]: pgmap v2670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:25:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:35 smithi189 ceph-mon[36790]: pgmap v2670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:25:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:35 smithi193 ceph-mon[37473]: pgmap v2670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:25:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3494.540345 secs 2024-07-26T12:25:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:36 smithi134 ceph-mon[31966]: pgmap v2671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3494.540345 secs 2024-07-26T12:25:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:36 smithi189 ceph-mon[36790]: pgmap v2671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3494.540345 secs 2024-07-26T12:25:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:36 smithi193 ceph-mon[37473]: pgmap v2671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3480.114122 secs 2024-07-26T12:25:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:39 smithi134 ceph-mon[31966]: pgmap v2672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3480.114122 secs 2024-07-26T12:25:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:39 smithi189 ceph-mon[36790]: pgmap v2672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3480.114122 secs 2024-07-26T12:25:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:39 smithi193 ceph-mon[37473]: pgmap v2672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:41 smithi134 ceph-mon[31966]: pgmap v2673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3499.540414 secs 2024-07-26T12:25:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:41 smithi189 ceph-mon[36790]: pgmap v2673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3499.540414 secs 2024-07-26T12:25:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:41 smithi193 ceph-mon[37473]: pgmap v2673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3499.540414 secs 2024-07-26T12:25:42.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:42 smithi189 ceph-mon[36790]: pgmap v2674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:42 smithi193 ceph-mon[37473]: pgmap v2674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:42 smithi134 ceph-mon[31966]: pgmap v2674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3485.114248 secs 2024-07-26T12:25:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3485.114248 secs 2024-07-26T12:25:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3485.114248 secs 2024-07-26T12:25:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:44 smithi189 ceph-mon[36790]: pgmap v2675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:25:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:44 smithi193 ceph-mon[37473]: pgmap v2675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:25:45.353 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:44 smithi134 ceph-mon[31966]: pgmap v2675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:25:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:25:46.105+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:25:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:25:46.105+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:25:46.107 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:25:46.105+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3504.540537 secs 2024-07-26T12:25:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3504.540537 secs 2024-07-26T12:25:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3504.540537 secs 2024-07-26T12:25:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:47 smithi193 ceph-mon[37473]: pgmap v2676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:47 smithi134 ceph-mon[31966]: pgmap v2676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:47 smithi189 ceph-mon[36790]: pgmap v2676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3490.114357 secs 2024-07-26T12:25:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:49 smithi134 ceph-mon[31966]: pgmap v2677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3490.114357 secs 2024-07-26T12:25:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:49 smithi189 ceph-mon[36790]: pgmap v2677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3490.114357 secs 2024-07-26T12:25:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:49 smithi193 ceph-mon[37473]: pgmap v2677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:50.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:25:50.642+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:25:50.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:25:50.643+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:25:50.645 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:25:50.643+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:25:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:50 smithi134 ceph-mon[31966]: pgmap v2678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:50 smithi189 ceph-mon[36790]: pgmap v2678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:50 smithi193 ceph-mon[37473]: pgmap v2678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3509.540628 secs 2024-07-26T12:25:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3509.540628 secs 2024-07-26T12:25:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3509.540628 secs 2024-07-26T12:25:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:52 smithi189 ceph-mon[36790]: pgmap v2679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:52 smithi193 ceph-mon[37473]: pgmap v2679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:52 smithi134 ceph-mon[31966]: pgmap v2679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:25:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3495.114492 secs 2024-07-26T12:25:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3495.114492 secs 2024-07-26T12:25:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3495.114492 secs 2024-07-26T12:25:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:54 smithi189 ceph-mon[36790]: pgmap v2680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:25:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:54 smithi193 ceph-mon[37473]: pgmap v2680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:25:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:54 smithi134 ceph-mon[31966]: pgmap v2680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:25:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3514.540760 secs 2024-07-26T12:25:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3514.540760 secs 2024-07-26T12:25:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3514.540760 secs 2024-07-26T12:25:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:57 smithi193 ceph-mon[37473]: pgmap v2681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:57 smithi134 ceph-mon[31966]: pgmap v2681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:57 smithi189 ceph-mon[36790]: pgmap v2681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:25:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3500.114624 secs 2024-07-26T12:25:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:25:59 smithi134 ceph-mon[31966]: pgmap v2682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:25:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3500.114624 secs 2024-07-26T12:25:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:25:59 smithi189 ceph-mon[36790]: pgmap v2682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:26:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3500.114624 secs 2024-07-26T12:26:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:25:59 smithi193 ceph-mon[37473]: pgmap v2682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:26:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:00 smithi134 ceph-mon[31966]: pgmap v2683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:26:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:00 smithi189 ceph-mon[36790]: pgmap v2683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:26:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:00 smithi193 ceph-mon[37473]: pgmap v2683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:26:01.122 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:26:01.121+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:26:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:26:01.121+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:26:01.123 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:26:01.121+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3519.540916 secs 2024-07-26T12:26:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3519.540916 secs 2024-07-26T12:26:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3519.540916 secs 2024-07-26T12:26:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:02 smithi134 ceph-mon[31966]: pgmap v2684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:26:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:02 smithi189 ceph-mon[36790]: pgmap v2684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:26:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:02 smithi193 ceph-mon[37473]: pgmap v2684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:26:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3505.114759 secs 2024-07-26T12:26:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3505.114759 secs 2024-07-26T12:26:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3505.114759 secs 2024-07-26T12:26:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:04 smithi189 ceph-mon[36790]: pgmap v2685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:26:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:04 smithi193 ceph-mon[37473]: pgmap v2685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:26:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:04 smithi134 ceph-mon[31966]: pgmap v2685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:26:05.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:26:05.656+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:26:05.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:26:05.656+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:05.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:26:05.656+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:26:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3524.541058 secs 2024-07-26T12:26:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3524.541058 secs 2024-07-26T12:26:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3524.541058 secs 2024-07-26T12:26:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:07 smithi193 ceph-mon[37473]: pgmap v2686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:26:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:07 smithi134 ceph-mon[31966]: pgmap v2686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:26:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:07 smithi189 ceph-mon[36790]: pgmap v2686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:26:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:08 smithi189 ceph-mon[36790]: pgmap v2687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:26:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:08 smithi193 ceph-mon[37473]: pgmap v2687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:26:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:08 smithi134 ceph-mon[31966]: pgmap v2687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:26:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3510.114901 secs 2024-07-26T12:26:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3510.114901 secs 2024-07-26T12:26:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3510.114901 secs 2024-07-26T12:26:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:10 smithi189 ceph-mon[36790]: pgmap v2688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:10 smithi193 ceph-mon[37473]: pgmap v2688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:10 smithi134 ceph-mon[31966]: pgmap v2688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3529.541187 secs 2024-07-26T12:26:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3529.541187 secs 2024-07-26T12:26:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3529.541187 secs 2024-07-26T12:26:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:12 smithi193 ceph-mon[37473]: pgmap v2689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:12 smithi134 ceph-mon[31966]: pgmap v2689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:12 smithi189 ceph-mon[36790]: pgmap v2689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3515.115048 secs 2024-07-26T12:26:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3515.115048 secs 2024-07-26T12:26:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3515.115048 secs 2024-07-26T12:26:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:14 smithi189 ceph-mon[36790]: pgmap v2690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:26:15.323 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:14 smithi193 ceph-mon[37473]: pgmap v2690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:26:15.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:14 smithi134 ceph-mon[31966]: pgmap v2690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:26:16.138 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:26:16.136+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:26:16.138 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:26:16.136+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:26:16.138 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:26:16.136+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3534.541336 secs 2024-07-26T12:26:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3534.541336 secs 2024-07-26T12:26:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3534.541336 secs 2024-07-26T12:26:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:17 smithi193 ceph-mon[37473]: pgmap v2691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:17 smithi134 ceph-mon[31966]: pgmap v2691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:17 smithi189 ceph-mon[36790]: pgmap v2691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:18 smithi189 ceph-mon[36790]: pgmap v2692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:18 smithi193 ceph-mon[37473]: pgmap v2692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:18 smithi134 ceph-mon[31966]: pgmap v2692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3520.115197 secs 2024-07-26T12:26:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:26:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3520.115197 secs 2024-07-26T12:26:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:26:20.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3520.115197 secs 2024-07-26T12:26:20.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:26:20.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:26:20.666+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:26:20.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:26:20.666+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:20.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:26:20.666+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:26:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:26:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:26:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:20 smithi134 ceph-mon[31966]: pgmap v2693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:26:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:20 smithi134 ceph-mon[31966]: pgmap v2694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:26:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:26:21.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:26:21.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:26:21.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:20 smithi193 ceph-mon[37473]: pgmap v2693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:26:21.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:20 smithi193 ceph-mon[37473]: pgmap v2694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:26:21.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:26:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:26:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:26:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:20 smithi189 ceph-mon[36790]: pgmap v2693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:26:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:20 smithi189 ceph-mon[36790]: pgmap v2694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:26:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:26:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3539.541458 secs 2024-07-26T12:26:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3539.541458 secs 2024-07-26T12:26:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3539.541458 secs 2024-07-26T12:26:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:22 smithi193 ceph-mon[37473]: pgmap v2695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:26:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:22 smithi134 ceph-mon[31966]: pgmap v2695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:26:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:22 smithi189 ceph-mon[36790]: pgmap v2695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:26:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3525.115336 secs 2024-07-26T12:26:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:24 smithi134 ceph-mon[31966]: pgmap v2696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:26:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3525.115336 secs 2024-07-26T12:26:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:24 smithi189 ceph-mon[36790]: pgmap v2696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:26:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3525.115336 secs 2024-07-26T12:26:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:24 smithi193 ceph-mon[37473]: pgmap v2696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:26:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:26 smithi134 ceph-mon[31966]: pgmap v2697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:26:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3544.541560 secs 2024-07-26T12:26:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:26 smithi189 ceph-mon[36790]: pgmap v2697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:26:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3544.541560 secs 2024-07-26T12:26:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:26 smithi193 ceph-mon[37473]: pgmap v2697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:26:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3544.541560 secs 2024-07-26T12:26:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:28 smithi134 ceph-mon[31966]: pgmap v2698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:26:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:28 smithi189 ceph-mon[36790]: pgmap v2698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:26:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:28 smithi193 ceph-mon[37473]: pgmap v2698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:26:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3530.115415 secs 2024-07-26T12:26:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3530.115415 secs 2024-07-26T12:26:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3530.115415 secs 2024-07-26T12:26:31.153 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:26:31.152+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:26:31.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:26:31.152+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:26:31.154 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:26:31.152+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:30 smithi134 ceph-mon[31966]: pgmap v2699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:26:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:30 smithi189 ceph-mon[36790]: pgmap v2699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:26:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:30 smithi193 ceph-mon[37473]: pgmap v2699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:26:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3549.541677 secs 2024-07-26T12:26:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3549.541677 secs 2024-07-26T12:26:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3549.541677 secs 2024-07-26T12:26:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:32 smithi134 ceph-mon[31966]: pgmap v2700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:32 smithi189 ceph-mon[36790]: pgmap v2700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:32 smithi193 ceph-mon[37473]: pgmap v2700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3535.115568 secs 2024-07-26T12:26:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:34 smithi134 ceph-mon[31966]: pgmap v2701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3535.115568 secs 2024-07-26T12:26:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:34 smithi189 ceph-mon[36790]: pgmap v2701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3535.115568 secs 2024-07-26T12:26:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:34 smithi193 ceph-mon[37473]: pgmap v2701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:35.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:26:35.682+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:26:35.683 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:26:35.682+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:26:35.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:26:35.682+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:36 smithi134 ceph-mon[31966]: pgmap v2702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3554.541750 secs 2024-07-26T12:26:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:36 smithi189 ceph-mon[36790]: pgmap v2702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3554.541750 secs 2024-07-26T12:26:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:36 smithi193 ceph-mon[37473]: pgmap v2702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3554.541750 secs 2024-07-26T12:26:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:38 smithi134 ceph-mon[31966]: pgmap v2703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:38 smithi189 ceph-mon[36790]: pgmap v2703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:38 smithi193 ceph-mon[37473]: pgmap v2703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3540.115710 secs 2024-07-26T12:26:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3540.115710 secs 2024-07-26T12:26:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3540.115710 secs 2024-07-26T12:26:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:40 smithi134 ceph-mon[31966]: pgmap v2704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:26:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:40 smithi189 ceph-mon[36790]: pgmap v2704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:26:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:40 smithi193 ceph-mon[37473]: pgmap v2704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:26:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3559.541877 secs 2024-07-26T12:26:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3559.541877 secs 2024-07-26T12:26:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3559.541877 secs 2024-07-26T12:26:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:42 smithi134 ceph-mon[31966]: pgmap v2705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:42 smithi189 ceph-mon[36790]: pgmap v2705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:42 smithi193 ceph-mon[37473]: pgmap v2705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3545.115866 secs 2024-07-26T12:26:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3545.115866 secs 2024-07-26T12:26:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3545.115866 secs 2024-07-26T12:26:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:44 smithi189 ceph-mon[36790]: pgmap v2706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:44 smithi193 ceph-mon[37473]: pgmap v2706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:44 smithi134 ceph-mon[31966]: pgmap v2706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:26:46.168+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:26:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:26:46.168+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:26:46.169 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:26:46.168+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:46 smithi189 ceph-mon[36790]: pgmap v2707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3564.542027 secs 2024-07-26T12:26:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:46 smithi193 ceph-mon[37473]: pgmap v2707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3564.542027 secs 2024-07-26T12:26:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:46 smithi134 ceph-mon[31966]: pgmap v2707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3564.542027 secs 2024-07-26T12:26:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:48 smithi189 ceph-mon[36790]: pgmap v2708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3550.115995 secs 2024-07-26T12:26:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:48 smithi193 ceph-mon[37473]: pgmap v2708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3550.115995 secs 2024-07-26T12:26:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:48 smithi134 ceph-mon[31966]: pgmap v2708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3550.115995 secs 2024-07-26T12:26:50.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:26:50.697+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:26:50.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:26:50.697+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:26:50.699 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:26:50.697+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:26:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:50 smithi134 ceph-mon[31966]: pgmap v2709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:26:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:50 smithi189 ceph-mon[36790]: pgmap v2709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:26:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:50 smithi193 ceph-mon[37473]: pgmap v2709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:26:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3569.542128 secs 2024-07-26T12:26:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3569.542128 secs 2024-07-26T12:26:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3569.542128 secs 2024-07-26T12:26:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:52 smithi193 ceph-mon[37473]: pgmap v2710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:52 smithi134 ceph-mon[31966]: pgmap v2710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:52 smithi189 ceph-mon[36790]: pgmap v2710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3555.116110 secs 2024-07-26T12:26:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3555.116110 secs 2024-07-26T12:26:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3555.116110 secs 2024-07-26T12:26:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:54 smithi193 ceph-mon[37473]: pgmap v2711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:54 smithi134 ceph-mon[31966]: pgmap v2711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:54 smithi189 ceph-mon[36790]: pgmap v2711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:56 smithi193 ceph-mon[37473]: pgmap v2712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3574.542204 secs 2024-07-26T12:26:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:56 smithi134 ceph-mon[31966]: pgmap v2712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3574.542204 secs 2024-07-26T12:26:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:56 smithi189 ceph-mon[36790]: pgmap v2712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:26:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3574.542204 secs 2024-07-26T12:26:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:58 smithi193 ceph-mon[37473]: pgmap v2713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:26:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3560.116245 secs 2024-07-26T12:26:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:58 smithi134 ceph-mon[31966]: pgmap v2713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:26:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3560.116245 secs 2024-07-26T12:26:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:58 smithi189 ceph-mon[36790]: pgmap v2713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:26:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:26:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3560.116245 secs 2024-07-26T12:27:01.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:01.183+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:27:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:01.183+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:27:01.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:01.183+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:00 smithi193 ceph-mon[37473]: pgmap v2714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:00 smithi134 ceph-mon[31966]: pgmap v2714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:00 smithi189 ceph-mon[36790]: pgmap v2714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3579.542267 secs 2024-07-26T12:27:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3579.542267 secs 2024-07-26T12:27:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3579.542267 secs 2024-07-26T12:27:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:03 smithi193 ceph-mon[37473]: pgmap v2715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:03 smithi134 ceph-mon[31966]: pgmap v2715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:03 smithi189 ceph-mon[36790]: pgmap v2715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3565.116345 secs 2024-07-26T12:27:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3565.116345 secs 2024-07-26T12:27:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3565.116345 secs 2024-07-26T12:27:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:05 smithi193 ceph-mon[37473]: pgmap v2716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:05 smithi134 ceph-mon[31966]: pgmap v2716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:05 smithi189 ceph-mon[36790]: pgmap v2716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:05.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:05.713+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:27:05.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:05.713+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:27:05.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:05.713+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:07 smithi193 ceph-mon[37473]: pgmap v2717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:07.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3584.542388 secs 2024-07-26T12:27:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:07 smithi134 ceph-mon[31966]: pgmap v2717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3584.542388 secs 2024-07-26T12:27:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:07 smithi189 ceph-mon[36790]: pgmap v2717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3584.542388 secs 2024-07-26T12:27:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:09 smithi134 ceph-mon[31966]: pgmap v2718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3570.116470 secs 2024-07-26T12:27:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:09 smithi189 ceph-mon[36790]: pgmap v2718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3570.116470 secs 2024-07-26T12:27:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:09 smithi193 ceph-mon[37473]: pgmap v2718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3570.116470 secs 2024-07-26T12:27:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:11 smithi134 ceph-mon[31966]: pgmap v2719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:11 smithi189 ceph-mon[36790]: pgmap v2719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:11 smithi193 ceph-mon[37473]: pgmap v2719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3589.542453 secs 2024-07-26T12:27:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3589.542453 secs 2024-07-26T12:27:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3589.542453 secs 2024-07-26T12:27:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:13 smithi134 ceph-mon[31966]: pgmap v2720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:13 smithi189 ceph-mon[36790]: pgmap v2720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:13 smithi193 ceph-mon[37473]: pgmap v2720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3575.116599 secs 2024-07-26T12:27:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3575.116599 secs 2024-07-26T12:27:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3575.116599 secs 2024-07-26T12:27:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:15 smithi134 ceph-mon[31966]: pgmap v2721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:15 smithi189 ceph-mon[36790]: pgmap v2721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:15 smithi193 ceph-mon[37473]: pgmap v2721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:16.186+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:27:16.187 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:16.186+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:27:16.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:16.186+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:17 smithi134 ceph-mon[31966]: pgmap v2722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3594.542598 secs 2024-07-26T12:27:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:17 smithi189 ceph-mon[36790]: pgmap v2722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3594.542598 secs 2024-07-26T12:27:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:17 smithi193 ceph-mon[37473]: pgmap v2722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3594.542598 secs 2024-07-26T12:27:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:19 smithi134 ceph-mon[31966]: pgmap v2723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3580.116724 secs 2024-07-26T12:27:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:19 smithi189 ceph-mon[36790]: pgmap v2723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3580.116724 secs 2024-07-26T12:27:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:19 smithi193 ceph-mon[37473]: pgmap v2723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3580.116724 secs 2024-07-26T12:27:20.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:27:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:27:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:27:20.700 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() TimeoutError 2024-07-26T12:27:20.702 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_53ce1462e129f6eb4071986336534c740fdebd31/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() TimeoutError 2024-07-26T12:27:20.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:20.728+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:27:20.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:20.728+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:27:20.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:20.728+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:21 smithi134 ceph-mon[31966]: pgmap v2724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:27:21.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:27:21.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:27:21.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:21 smithi193 ceph-mon[37473]: pgmap v2724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:21.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:27:21.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:27:21.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:27:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:21 smithi189 ceph-mon[36790]: pgmap v2724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:27:21.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:27:21.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:27:22.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:22.295+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bfb30 msgr2=0x7f9b300bff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T12:27:22.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:22.295+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bfb30 msgr2=0x7f9b300bff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:27:22.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:22.295+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bfb30 0x7f9b300bff40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b3802b040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:27:22.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:22.295+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bfb30 0x7f9b300bff40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b3802b040 comp rx=0 tx=0).stop 2024-07-26T12:27:22.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:22.295+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bfb30 msgr2=0x7f9b300bff40 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T12:27:22.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:22.295+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bfb30 msgr2=0x7f9b300bff40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:27:22.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:22.295+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300bfb30 0x7f9b300bff40 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:27:22.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:22.295+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300c0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:27:22.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:22.296+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300c0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:27:22.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:22.298+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300c0480 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800e640 tx=0x7f9b38008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:27:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3599.542720 secs 2024-07-26T12:27:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:22 smithi134 ceph-mon[31966]: pgmap v2725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:27:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:22 smithi134 ceph-mon[31966]: pgmap v2726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:27:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3599.542720 secs 2024-07-26T12:27:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:22 smithi189 ceph-mon[36790]: pgmap v2725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:27:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:22 smithi189 ceph-mon[36790]: pgmap v2726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:27:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3599.542720 secs 2024-07-26T12:27:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:22 smithi193 ceph-mon[37473]: pgmap v2725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:27:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:22 smithi193 ceph-mon[37473]: pgmap v2726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:27:22.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:22.952+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:27:22.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:22.952+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:27:22.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:22.952+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T12:27:22.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:22.952+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T12:27:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:24 smithi134 ceph-mon[31966]: pgmap v2727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:27:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3585.116866 secs 2024-07-26T12:27:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:24 smithi189 ceph-mon[36790]: pgmap v2727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:27:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3585.116866 secs 2024-07-26T12:27:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:24 smithi193 ceph-mon[37473]: pgmap v2727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:27:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3585.116866 secs 2024-07-26T12:27:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:26 smithi134 ceph-mon[31966]: pgmap v2728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:27:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:26 smithi189 ceph-mon[36790]: pgmap v2728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:27:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:26 smithi193 ceph-mon[37473]: pgmap v2728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:27:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3604.542789 secs 2024-07-26T12:27:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3604.542789 secs 2024-07-26T12:27:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3604.542789 secs 2024-07-26T12:27:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:28 smithi134 ceph-mon[31966]: pgmap v2729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:27:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:28 smithi189 ceph-mon[36790]: pgmap v2729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:27:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:28 smithi193 ceph-mon[37473]: pgmap v2729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:27:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3590.116985 secs 2024-07-26T12:27:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3590.116985 secs 2024-07-26T12:27:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3590.116985 secs 2024-07-26T12:27:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:30 smithi134 ceph-mon[31966]: pgmap v2730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:27:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:30 smithi189 ceph-mon[36790]: pgmap v2730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:27:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:30 smithi193 ceph-mon[37473]: pgmap v2730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:27:31.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:31.196+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:27:31.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:31.196+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:27:31.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:31.196+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3609.542939 secs 2024-07-26T12:27:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:32 smithi134 ceph-mon[31966]: pgmap v2731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:27:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3609.542939 secs 2024-07-26T12:27:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:32 smithi189 ceph-mon[36790]: pgmap v2731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:27:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3609.542939 secs 2024-07-26T12:27:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:32 smithi193 ceph-mon[37473]: pgmap v2731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:27:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:32.849+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bfa00 msgr2=0x7f21ac0bfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T12:27:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:32.849+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bfa00 msgr2=0x7f21ac0bfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:27:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:32.849+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bfa00 0x7f21ac0bfe10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f21980063a0 tx=0x7f21b80aa000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:27:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:32.849+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bfa00 0x7f21ac0bfe10 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f21980063a0 tx=0x7f21b80aa000 comp rx=0 tx=0).stop 2024-07-26T12:27:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:32.849+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bfa00 msgr2=0x7f21ac0bfe10 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T12:27:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:32.849+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bfa00 msgr2=0x7f21ac0bfe10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:27:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:32.849+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0bfa00 0x7f21ac0bfe10 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:27:32.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:32.849+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 0x7f21ac0c0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:27:32.852 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:32.850+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 0x7f21ac0c0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:27:32.852 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:32.851+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 0x7f21ac0c0760 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2198005820 tx=0x7f21b80c3000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:27:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:33.463+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:27:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:33.463+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T12:27:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:33.464+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:27:33.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:33.464+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T12:27:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:34 smithi134 ceph-mon[31966]: pgmap v2732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3595.117120 secs 2024-07-26T12:27:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:34 smithi189 ceph-mon[36790]: pgmap v2732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3595.117120 secs 2024-07-26T12:27:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:34 smithi193 ceph-mon[37473]: pgmap v2732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3595.117120 secs 2024-07-26T12:27:35.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:35.742+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:27:35.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:35.742+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:35.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:35.742+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:27:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:36 smithi134 ceph-mon[31966]: pgmap v2733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:36 smithi189 ceph-mon[36790]: pgmap v2733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:36 smithi193 ceph-mon[37473]: pgmap v2733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3614.543002 secs 2024-07-26T12:27:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3614.543002 secs 2024-07-26T12:27:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3614.543002 secs 2024-07-26T12:27:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:38 smithi134 ceph-mon[31966]: pgmap v2734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:38 smithi189 ceph-mon[36790]: pgmap v2734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:38 smithi193 ceph-mon[37473]: pgmap v2734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3600.117275 secs 2024-07-26T12:27:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3600.117275 secs 2024-07-26T12:27:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3600.117275 secs 2024-07-26T12:27:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:40 smithi189 ceph-mon[36790]: pgmap v2735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:40 smithi193 ceph-mon[37473]: pgmap v2735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:40 smithi134 ceph-mon[31966]: pgmap v2735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3619.543131 secs 2024-07-26T12:27:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:42 smithi189 ceph-mon[36790]: pgmap v2736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3619.543131 secs 2024-07-26T12:27:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:42 smithi193 ceph-mon[37473]: pgmap v2736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3619.543131 secs 2024-07-26T12:27:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:42 smithi134 ceph-mon[31966]: pgmap v2736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:44 smithi193 ceph-mon[37473]: pgmap v2737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3605.117361 secs 2024-07-26T12:27:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:44 smithi134 ceph-mon[31966]: pgmap v2737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3605.117361 secs 2024-07-26T12:27:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:44 smithi189 ceph-mon[36790]: pgmap v2737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3605.117361 secs 2024-07-26T12:27:46.213 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:46.211+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:27:46.213 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:46.211+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:27:46.213 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:27:46.211+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:46 smithi193 ceph-mon[37473]: pgmap v2738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:46 smithi134 ceph-mon[31966]: pgmap v2738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:46 smithi189 ceph-mon[36790]: pgmap v2738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3624.543255 secs 2024-07-26T12:27:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3624.543255 secs 2024-07-26T12:27:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3624.543255 secs 2024-07-26T12:27:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:48 smithi193 ceph-mon[37473]: pgmap v2739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:48 smithi134 ceph-mon[31966]: pgmap v2739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:48 smithi189 ceph-mon[36790]: pgmap v2739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3610.117490 secs 2024-07-26T12:27:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3610.117490 secs 2024-07-26T12:27:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3610.117490 secs 2024-07-26T12:27:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:50 smithi193 ceph-mon[37473]: pgmap v2740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:50.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:50 smithi134 ceph-mon[31966]: pgmap v2740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:50 smithi189 ceph-mon[36790]: pgmap v2740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:27:50.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:50.753+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:27:50.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:50.753+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:27:50.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:27:50.753+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:27:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3629.543387 secs 2024-07-26T12:27:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:52 smithi193 ceph-mon[37473]: pgmap v2741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3629.543387 secs 2024-07-26T12:27:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:52 smithi134 ceph-mon[31966]: pgmap v2741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3629.543387 secs 2024-07-26T12:27:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:52 smithi189 ceph-mon[36790]: pgmap v2741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:27:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:54 smithi193 ceph-mon[37473]: pgmap v2742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3615.117611 secs 2024-07-26T12:27:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:54 smithi134 ceph-mon[31966]: pgmap v2742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3615.117611 secs 2024-07-26T12:27:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:54 smithi189 ceph-mon[36790]: pgmap v2742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:27:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3615.117611 secs 2024-07-26T12:27:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:56 smithi193 ceph-mon[37473]: pgmap v2743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:56 smithi134 ceph-mon[31966]: pgmap v2743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:56 smithi189 ceph-mon[36790]: pgmap v2743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:27:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3634.543545 secs 2024-07-26T12:27:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3634.543545 secs 2024-07-26T12:27:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3634.543545 secs 2024-07-26T12:27:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:58 smithi193 ceph-mon[37473]: pgmap v2744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:58 smithi134 ceph-mon[31966]: pgmap v2744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:58 smithi189 ceph-mon[36790]: pgmap v2744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:27:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:27:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3620.117738 secs 2024-07-26T12:27:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:27:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3620.117738 secs 2024-07-26T12:27:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:27:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3620.117738 secs 2024-07-26T12:28:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:00 smithi134 ceph-mon[31966]: pgmap v2745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:00 smithi189 ceph-mon[36790]: pgmap v2745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:00 smithi193 ceph-mon[37473]: pgmap v2745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:28:01.227+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:28:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:28:01.227+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:28:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:28:01.227+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3639.543673 secs 2024-07-26T12:28:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:02 smithi134 ceph-mon[31966]: pgmap v2746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3639.543673 secs 2024-07-26T12:28:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:02 smithi189 ceph-mon[36790]: pgmap v2746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3639.543673 secs 2024-07-26T12:28:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:02 smithi193 ceph-mon[37473]: pgmap v2746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:04 smithi134 ceph-mon[31966]: pgmap v2747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3625.117862 secs 2024-07-26T12:28:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:04 smithi189 ceph-mon[36790]: pgmap v2747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3625.117862 secs 2024-07-26T12:28:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:04 smithi193 ceph-mon[37473]: pgmap v2747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3625.117862 secs 2024-07-26T12:28:05.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:28:05.760+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:28:05.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:28:05.760+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:05.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:28:05.760+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:28:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:06 smithi134 ceph-mon[31966]: pgmap v2748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:28:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:06 smithi189 ceph-mon[36790]: pgmap v2748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:28:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:06 smithi193 ceph-mon[37473]: pgmap v2748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:28:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3644.543806 secs 2024-07-26T12:28:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3644.543806 secs 2024-07-26T12:28:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3644.543806 secs 2024-07-26T12:28:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:08 smithi134 ceph-mon[31966]: pgmap v2749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:08 smithi189 ceph-mon[36790]: pgmap v2749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:08 smithi193 ceph-mon[37473]: pgmap v2749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3630.117988 secs 2024-07-26T12:28:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3630.117988 secs 2024-07-26T12:28:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3630.117988 secs 2024-07-26T12:28:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:10 smithi134 ceph-mon[31966]: pgmap v2750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:10 smithi189 ceph-mon[36790]: pgmap v2750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:10 smithi193 ceph-mon[37473]: pgmap v2750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3649.543912 secs 2024-07-26T12:28:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:11 smithi134 ceph-mon[31966]: pgmap v2751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3649.543912 secs 2024-07-26T12:28:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:11 smithi189 ceph-mon[36790]: pgmap v2751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3649.543912 secs 2024-07-26T12:28:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:11 smithi193 ceph-mon[37473]: pgmap v2751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:14 smithi134 ceph-mon[31966]: pgmap v2752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3635.118057 secs 2024-07-26T12:28:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:14 smithi189 ceph-mon[36790]: pgmap v2752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3635.118057 secs 2024-07-26T12:28:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:14 smithi193 ceph-mon[37473]: pgmap v2752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3635.118057 secs 2024-07-26T12:28:16.237 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:28:16.235+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:28:16.237 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:28:16.235+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:28:16.237 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:28:16.235+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:16 smithi134 ceph-mon[31966]: pgmap v2753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:28:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:16 smithi189 ceph-mon[36790]: pgmap v2753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:28:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:16 smithi193 ceph-mon[37473]: pgmap v2753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:28:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3654.544068 secs 2024-07-26T12:28:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3654.544068 secs 2024-07-26T12:28:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3654.544068 secs 2024-07-26T12:28:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:18 smithi134 ceph-mon[31966]: pgmap v2754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:18 smithi189 ceph-mon[36790]: pgmap v2754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:18 smithi193 ceph-mon[37473]: pgmap v2754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3640.118215 secs 2024-07-26T12:28:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3640.118215 secs 2024-07-26T12:28:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3640.118215 secs 2024-07-26T12:28:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:20 smithi134 ceph-mon[31966]: pgmap v2755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:20 smithi189 ceph-mon[36790]: pgmap v2755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:20 smithi193 ceph-mon[37473]: pgmap v2755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:20.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:28:20.769+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:28:20.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:28:20.769+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:20.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:28:20.769+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:28:22.325 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3659.544196 secs 2024-07-26T12:28:22.325 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:22 smithi193 ceph-mon[37473]: pgmap v2756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:22.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:28:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3659.544196 secs 2024-07-26T12:28:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:22 smithi134 ceph-mon[31966]: pgmap v2756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:28:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3659.544196 secs 2024-07-26T12:28:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:22 smithi189 ceph-mon[36790]: pgmap v2756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:28:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:28:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:28:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:23 smithi134 ceph-mon[31966]: pgmap v2757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:28:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:23 smithi134 ceph-mon[31966]: pgmap v2758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:28:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:28:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:28:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:28:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:23 smithi189 ceph-mon[36790]: pgmap v2757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:28:23.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:23 smithi189 ceph-mon[36790]: pgmap v2758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:28:23.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:28:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:28:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:28:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:23 smithi193 ceph-mon[37473]: pgmap v2757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:28:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:23 smithi193 ceph-mon[37473]: pgmap v2758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:28:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:28:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3645.118338 secs 2024-07-26T12:28:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3645.118338 secs 2024-07-26T12:28:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3645.118338 secs 2024-07-26T12:28:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:25 smithi134 ceph-mon[31966]: pgmap v2759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:25 smithi189 ceph-mon[36790]: pgmap v2759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:25 smithi193 ceph-mon[37473]: pgmap v2759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3664.544362 secs 2024-07-26T12:28:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:27 smithi193 ceph-mon[37473]: pgmap v2760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:28:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3664.544362 secs 2024-07-26T12:28:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:27 smithi134 ceph-mon[31966]: pgmap v2760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:28:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3664.544362 secs 2024-07-26T12:28:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:27 smithi189 ceph-mon[36790]: pgmap v2760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:28:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:29 smithi193 ceph-mon[37473]: pgmap v2761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3650.118416 secs 2024-07-26T12:28:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:29 smithi134 ceph-mon[31966]: pgmap v2761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3650.118416 secs 2024-07-26T12:28:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:29 smithi189 ceph-mon[36790]: pgmap v2761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3650.118416 secs 2024-07-26T12:28:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:28:31.250+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:28:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:28:31.250+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:28:31.252 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:28:31.250+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:31 smithi193 ceph-mon[37473]: pgmap v2762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:31 smithi134 ceph-mon[31966]: pgmap v2762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:31 smithi189 ceph-mon[36790]: pgmap v2762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:28:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3669.544489 secs 2024-07-26T12:28:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3669.544489 secs 2024-07-26T12:28:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3669.544489 secs 2024-07-26T12:28:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:33 smithi193 ceph-mon[37473]: pgmap v2763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:28:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:33 smithi134 ceph-mon[31966]: pgmap v2763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:28:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:33 smithi189 ceph-mon[36790]: pgmap v2763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:28:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3655.118571 secs 2024-07-26T12:28:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3655.118571 secs 2024-07-26T12:28:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3655.118571 secs 2024-07-26T12:28:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:35 smithi134 ceph-mon[31966]: pgmap v2764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:35 smithi189 ceph-mon[36790]: pgmap v2764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:35 smithi193 ceph-mon[37473]: pgmap v2764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:35.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:28:35.785+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:28:35.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:28:35.785+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:35.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:28:35.785+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:28:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3674.544555 secs 2024-07-26T12:28:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:37 smithi134 ceph-mon[31966]: pgmap v2765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3674.544555 secs 2024-07-26T12:28:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:37 smithi189 ceph-mon[36790]: pgmap v2765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3674.544555 secs 2024-07-26T12:28:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:37 smithi193 ceph-mon[37473]: pgmap v2765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:39 smithi134 ceph-mon[31966]: pgmap v2766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3660.118694 secs 2024-07-26T12:28:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:39 smithi189 ceph-mon[36790]: pgmap v2766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3660.118694 secs 2024-07-26T12:28:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:39 smithi193 ceph-mon[37473]: pgmap v2766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3660.118694 secs 2024-07-26T12:28:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:41 smithi134 ceph-mon[31966]: pgmap v2767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:41 smithi189 ceph-mon[36790]: pgmap v2767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:41 smithi193 ceph-mon[37473]: pgmap v2767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:42.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3679.544686 secs 2024-07-26T12:28:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3679.544686 secs 2024-07-26T12:28:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3679.544686 secs 2024-07-26T12:28:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:43 smithi134 ceph-mon[31966]: pgmap v2768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:43 smithi189 ceph-mon[36790]: pgmap v2768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:43 smithi193 ceph-mon[37473]: pgmap v2768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3665.118786 secs 2024-07-26T12:28:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3665.118786 secs 2024-07-26T12:28:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3665.118786 secs 2024-07-26T12:28:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:45 smithi134 ceph-mon[31966]: pgmap v2769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:45 smithi189 ceph-mon[36790]: pgmap v2769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:45 smithi193 ceph-mon[37473]: pgmap v2769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:46.257 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:28:46.256+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:28:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:28:46.256+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:28:46.258 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:28:46.256+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3684.544743 secs 2024-07-26T12:28:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:47 smithi134 ceph-mon[31966]: pgmap v2770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3684.544743 secs 2024-07-26T12:28:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:47 smithi189 ceph-mon[36790]: pgmap v2770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3684.544743 secs 2024-07-26T12:28:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:47 smithi193 ceph-mon[37473]: pgmap v2770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:49 smithi134 ceph-mon[31966]: pgmap v2771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3670.118897 secs 2024-07-26T12:28:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:49 smithi189 ceph-mon[36790]: pgmap v2771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3670.118897 secs 2024-07-26T12:28:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:49 smithi193 ceph-mon[37473]: pgmap v2771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3670.118897 secs 2024-07-26T12:28:50.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:28:50.795+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:28:50.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:28:50.795+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:28:50.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:28:50.795+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:28:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:51 smithi134 ceph-mon[31966]: pgmap v2772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:51 smithi189 ceph-mon[36790]: pgmap v2772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:51 smithi193 ceph-mon[37473]: pgmap v2772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:52.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3689.544888 secs 2024-07-26T12:28:52.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3689.544888 secs 2024-07-26T12:28:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3689.544888 secs 2024-07-26T12:28:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:53 smithi134 ceph-mon[31966]: pgmap v2773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:53 smithi189 ceph-mon[36790]: pgmap v2773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:53 smithi193 ceph-mon[37473]: pgmap v2773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:28:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3675.119018 secs 2024-07-26T12:28:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3675.119018 secs 2024-07-26T12:28:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3675.119018 secs 2024-07-26T12:28:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:55 smithi134 ceph-mon[31966]: pgmap v2774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:55 smithi189 ceph-mon[36790]: pgmap v2774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:55 smithi193 ceph-mon[37473]: pgmap v2774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:28:57.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3694.545048 secs 2024-07-26T12:28:57.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:57 smithi134 ceph-mon[31966]: pgmap v2775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3694.545048 secs 2024-07-26T12:28:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:57 smithi189 ceph-mon[36790]: pgmap v2775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3694.545048 secs 2024-07-26T12:28:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:57 smithi193 ceph-mon[37473]: pgmap v2775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:28:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:59 smithi134 ceph-mon[31966]: pgmap v2776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:28:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3680.119131 secs 2024-07-26T12:28:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:59 smithi189 ceph-mon[36790]: pgmap v2776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:28:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3680.119131 secs 2024-07-26T12:28:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:59 smithi193 ceph-mon[37473]: pgmap v2776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:28:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:28:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3680.119131 secs 2024-07-26T12:29:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:29:01.267+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:29:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:29:01.267+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:29:01.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:29:01.267+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:01 smithi134 ceph-mon[31966]: pgmap v2777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:01 smithi189 ceph-mon[36790]: pgmap v2777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:01 smithi193 ceph-mon[37473]: pgmap v2777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3699.545200 secs 2024-07-26T12:29:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3699.545200 secs 2024-07-26T12:29:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3699.545200 secs 2024-07-26T12:29:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:03 smithi134 ceph-mon[31966]: pgmap v2778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:03 smithi189 ceph-mon[36790]: pgmap v2778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:03 smithi193 ceph-mon[37473]: pgmap v2778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3685.119205 secs 2024-07-26T12:29:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3685.119205 secs 2024-07-26T12:29:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3685.119205 secs 2024-07-26T12:29:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:05 smithi134 ceph-mon[31966]: pgmap v2779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:05 smithi189 ceph-mon[36790]: pgmap v2779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:05 smithi193 ceph-mon[37473]: pgmap v2779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:05.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:29:05.810+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:29:05.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:29:05.810+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:05.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:29:05.810+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:29:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3704.545311 secs 2024-07-26T12:29:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:07 smithi134 ceph-mon[31966]: pgmap v2780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3704.545311 secs 2024-07-26T12:29:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:07 smithi189 ceph-mon[36790]: pgmap v2780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3704.545311 secs 2024-07-26T12:29:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:07 smithi193 ceph-mon[37473]: pgmap v2780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:09 smithi189 ceph-mon[36790]: pgmap v2781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3690.119335 secs 2024-07-26T12:29:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:09 smithi193 ceph-mon[37473]: pgmap v2781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3690.119335 secs 2024-07-26T12:29:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:09 smithi134 ceph-mon[31966]: pgmap v2781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3690.119335 secs 2024-07-26T12:29:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:11 smithi189 ceph-mon[36790]: pgmap v2782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:11 smithi193 ceph-mon[37473]: pgmap v2782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:11 smithi134 ceph-mon[31966]: pgmap v2782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3709.545400 secs 2024-07-26T12:29:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3709.545400 secs 2024-07-26T12:29:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3709.545400 secs 2024-07-26T12:29:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:13 smithi193 ceph-mon[37473]: pgmap v2783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:13 smithi134 ceph-mon[31966]: pgmap v2783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:13 smithi189 ceph-mon[36790]: pgmap v2783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3695.119455 secs 2024-07-26T12:29:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3695.119455 secs 2024-07-26T12:29:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3695.119455 secs 2024-07-26T12:29:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:15 smithi193 ceph-mon[37473]: pgmap v2784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:15 smithi134 ceph-mon[31966]: pgmap v2784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:15 smithi189 ceph-mon[36790]: pgmap v2784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:16.281 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:29:16.280+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:29:16.282 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:29:16.280+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:29:16.282 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:29:16.280+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3714.545470 secs 2024-07-26T12:29:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3714.545470 secs 2024-07-26T12:29:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3714.545470 secs 2024-07-26T12:29:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:17 smithi193 ceph-mon[37473]: pgmap v2785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:17 smithi134 ceph-mon[31966]: pgmap v2785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:17 smithi189 ceph-mon[36790]: pgmap v2785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:19 smithi193 ceph-mon[37473]: pgmap v2786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3700.119599 secs 2024-07-26T12:29:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:19 smithi134 ceph-mon[31966]: pgmap v2786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3700.119599 secs 2024-07-26T12:29:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:19 smithi189 ceph-mon[36790]: pgmap v2786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3700.119599 secs 2024-07-26T12:29:20.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:29:20.825+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:29:20.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:29:20.825+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:20.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:29:20.825+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:29:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:21 smithi193 ceph-mon[37473]: pgmap v2787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3719.545595 secs 2024-07-26T12:29:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:21 smithi134 ceph-mon[31966]: pgmap v2787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3719.545595 secs 2024-07-26T12:29:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:21 smithi189 ceph-mon[36790]: pgmap v2787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3719.545595 secs 2024-07-26T12:29:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:29:22.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:29:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:29:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:23 smithi193 ceph-mon[37473]: pgmap v2788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:29:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:29:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:29:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:29:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:23 smithi134 ceph-mon[31966]: pgmap v2788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:29:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:29:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:29:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:29:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:23 smithi189 ceph-mon[36790]: pgmap v2788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:29:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:29:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:29:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:29:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3705.119753 secs 2024-07-26T12:29:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:24 smithi193 ceph-mon[37473]: pgmap v2789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:29:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3705.119753 secs 2024-07-26T12:29:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:24 smithi134 ceph-mon[31966]: pgmap v2789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:29:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3705.119753 secs 2024-07-26T12:29:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:24 smithi189 ceph-mon[36790]: pgmap v2789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:29:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:26 smithi193 ceph-mon[37473]: pgmap v2790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:29:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3724.545721 secs 2024-07-26T12:29:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:26 smithi134 ceph-mon[31966]: pgmap v2790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:29:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3724.545721 secs 2024-07-26T12:29:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:26 smithi189 ceph-mon[36790]: pgmap v2790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:29:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3724.545721 secs 2024-07-26T12:29:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:28 smithi193 ceph-mon[37473]: pgmap v2791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:29:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:28 smithi134 ceph-mon[31966]: pgmap v2791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:29:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:28 smithi189 ceph-mon[36790]: pgmap v2791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:29:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3710.119886 secs 2024-07-26T12:29:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3710.119886 secs 2024-07-26T12:29:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3710.119886 secs 2024-07-26T12:29:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:30 smithi134 ceph-mon[31966]: pgmap v2792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:29:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:30 smithi189 ceph-mon[36790]: pgmap v2792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:29:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:30 smithi193 ceph-mon[37473]: pgmap v2792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:29:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:29:31.296+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:29:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:29:31.296+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:29:31.297 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:29:31.296+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3729.545850 secs 2024-07-26T12:29:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3729.545850 secs 2024-07-26T12:29:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3729.545850 secs 2024-07-26T12:29:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:32 smithi134 ceph-mon[31966]: pgmap v2793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:29:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:32 smithi189 ceph-mon[36790]: pgmap v2793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:29:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:32 smithi193 ceph-mon[37473]: pgmap v2793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:29:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3715.119986 secs 2024-07-26T12:29:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:34 smithi134 ceph-mon[31966]: pgmap v2794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:29:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3715.119986 secs 2024-07-26T12:29:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:34 smithi189 ceph-mon[36790]: pgmap v2794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:29:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3715.119986 secs 2024-07-26T12:29:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:34 smithi193 ceph-mon[37473]: pgmap v2794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:29:35.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:29:35.841+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:29:35.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:29:35.841+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:35.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:29:35.841+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:29:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:36 smithi134 ceph-mon[31966]: pgmap v2795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3734.545976 secs 2024-07-26T12:29:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:36 smithi189 ceph-mon[36790]: pgmap v2795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3734.545976 secs 2024-07-26T12:29:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:36 smithi193 ceph-mon[37473]: pgmap v2795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3734.545976 secs 2024-07-26T12:29:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:38 smithi134 ceph-mon[31966]: pgmap v2796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:38 smithi189 ceph-mon[36790]: pgmap v2796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:38 smithi193 ceph-mon[37473]: pgmap v2796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3720.120106 secs 2024-07-26T12:29:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3720.120106 secs 2024-07-26T12:29:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3720.120106 secs 2024-07-26T12:29:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:40 smithi134 ceph-mon[31966]: pgmap v2797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:40 smithi189 ceph-mon[36790]: pgmap v2797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:40 smithi193 ceph-mon[37473]: pgmap v2797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3739.546112 secs 2024-07-26T12:29:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3739.546112 secs 2024-07-26T12:29:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3739.546112 secs 2024-07-26T12:29:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:42 smithi134 ceph-mon[31966]: pgmap v2798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:42 smithi189 ceph-mon[36790]: pgmap v2798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:42 smithi193 ceph-mon[37473]: pgmap v2798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3725.120229 secs 2024-07-26T12:29:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:44 smithi134 ceph-mon[31966]: pgmap v2799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3725.120229 secs 2024-07-26T12:29:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:44 smithi189 ceph-mon[36790]: pgmap v2799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3725.120229 secs 2024-07-26T12:29:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:44 smithi193 ceph-mon[37473]: pgmap v2799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:46.313 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:29:46.311+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:29:46.313 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:29:46.311+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:29:46.313 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:29:46.311+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:46 smithi134 ceph-mon[31966]: pgmap v2800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3744.546240 secs 2024-07-26T12:29:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:46 smithi189 ceph-mon[36790]: pgmap v2800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3744.546240 secs 2024-07-26T12:29:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:46 smithi193 ceph-mon[37473]: pgmap v2800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3744.546240 secs 2024-07-26T12:29:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:48 smithi134 ceph-mon[31966]: pgmap v2801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:48 smithi189 ceph-mon[36790]: pgmap v2801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:48 smithi193 ceph-mon[37473]: pgmap v2801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3730.120303 secs 2024-07-26T12:29:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3730.120303 secs 2024-07-26T12:29:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3730.120303 secs 2024-07-26T12:29:50.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:29:50.843+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:29:50.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:29:50.843+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:29:50.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:29:50.843+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:29:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:50 smithi134 ceph-mon[31966]: pgmap v2802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:50 smithi189 ceph-mon[36790]: pgmap v2802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:50 smithi193 ceph-mon[37473]: pgmap v2802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:29:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3749.546386 secs 2024-07-26T12:29:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3749.546386 secs 2024-07-26T12:29:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3749.546386 secs 2024-07-26T12:29:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:52 smithi134 ceph-mon[31966]: pgmap v2803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:52.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:52 smithi193 ceph-mon[37473]: pgmap v2803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:52 smithi189 ceph-mon[36790]: pgmap v2803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3735.120366 secs 2024-07-26T12:29:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:54 smithi134 ceph-mon[31966]: pgmap v2804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:54.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3735.120366 secs 2024-07-26T12:29:54.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:54 smithi189 ceph-mon[36790]: pgmap v2804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3735.120366 secs 2024-07-26T12:29:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:54 smithi193 ceph-mon[37473]: pgmap v2804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:29:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:56 smithi134 ceph-mon[31966]: pgmap v2805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3754.546507 secs 2024-07-26T12:29:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:56 smithi189 ceph-mon[36790]: pgmap v2805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3754.546507 secs 2024-07-26T12:29:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:56 smithi193 ceph-mon[37473]: pgmap v2805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:29:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3754.546507 secs 2024-07-26T12:29:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:58 smithi134 ceph-mon[31966]: pgmap v2806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:58 smithi189 ceph-mon[36790]: pgmap v2806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:58 smithi193 ceph-mon[37473]: pgmap v2806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:29:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:29:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3740.120493 secs 2024-07-26T12:30:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:29:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3740.120493 secs 2024-07-26T12:30:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:29:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3740.120493 secs 2024-07-26T12:30:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:00 smithi189 ceph-mon[36790]: pgmap v2807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:30:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:00 smithi193 ceph-mon[37473]: pgmap v2807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:30:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:00 smithi134 ceph-mon[31966]: pgmap v2807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:30:01.328 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:30:01.326+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:30:01.328 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:30:01.326+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:30:01.328 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:30:01.326+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3759.546575 secs 2024-07-26T12:30:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:01 smithi193 ceph-mon[37473]: pgmap v2808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3759.546575 secs 2024-07-26T12:30:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:01 smithi134 ceph-mon[31966]: pgmap v2808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3759.546575 secs 2024-07-26T12:30:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:01 smithi189 ceph-mon[36790]: pgmap v2808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3745.120636 secs 2024-07-26T12:30:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:04 smithi134 ceph-mon[31966]: pgmap v2809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3745.120636 secs 2024-07-26T12:30:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:04 smithi189 ceph-mon[36790]: pgmap v2809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3745.120636 secs 2024-07-26T12:30:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:04 smithi193 ceph-mon[37473]: pgmap v2809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:05.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:30:05.858+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:30:05.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:30:05.859+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:30:05.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:30:05.859+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:06 smithi189 ceph-mon[36790]: pgmap v2810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:06 smithi193 ceph-mon[37473]: pgmap v2810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:06 smithi134 ceph-mon[31966]: pgmap v2810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3764.546701 secs 2024-07-26T12:30:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3764.546701 secs 2024-07-26T12:30:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3764.546701 secs 2024-07-26T12:30:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:08 smithi189 ceph-mon[36790]: pgmap v2811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:08 smithi193 ceph-mon[37473]: pgmap v2811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:08 smithi134 ceph-mon[31966]: pgmap v2811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3750.120773 secs 2024-07-26T12:30:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3750.120773 secs 2024-07-26T12:30:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3750.120773 secs 2024-07-26T12:30:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:10 smithi193 ceph-mon[37473]: pgmap v2812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:10 smithi134 ceph-mon[31966]: pgmap v2812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:10 smithi189 ceph-mon[36790]: pgmap v2812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3769.546843 secs 2024-07-26T12:30:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3769.546843 secs 2024-07-26T12:30:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3769.546843 secs 2024-07-26T12:30:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:12 smithi193 ceph-mon[37473]: pgmap v2813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:12.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:12 smithi189 ceph-mon[36790]: pgmap v2813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:12 smithi134 ceph-mon[31966]: pgmap v2813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3755.120862 secs 2024-07-26T12:30:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:14 smithi193 ceph-mon[37473]: pgmap v2814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3755.120862 secs 2024-07-26T12:30:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:14 smithi134 ceph-mon[31966]: pgmap v2814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3755.120862 secs 2024-07-26T12:30:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:14 smithi189 ceph-mon[36790]: pgmap v2814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:16.343 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:30:16.342+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:30:16.343 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:30:16.342+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:30:16.344 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:30:16.342+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:16 smithi193 ceph-mon[37473]: pgmap v2815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3774.546985 secs 2024-07-26T12:30:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:16 smithi134 ceph-mon[31966]: pgmap v2815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3774.546985 secs 2024-07-26T12:30:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:16 smithi189 ceph-mon[36790]: pgmap v2815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3774.546985 secs 2024-07-26T12:30:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:18 smithi193 ceph-mon[37473]: pgmap v2816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:18 smithi134 ceph-mon[31966]: pgmap v2816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:18 smithi189 ceph-mon[36790]: pgmap v2816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:30:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3760.120984 secs 2024-07-26T12:30:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3760.120984 secs 2024-07-26T12:30:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3760.120984 secs 2024-07-26T12:30:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:20 smithi193 ceph-mon[37473]: pgmap v2817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:20.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:30:20.865+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:30:20.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:30:20.865+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:30:20.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:30:20.865+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:20.867 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:20 smithi134 ceph-mon[31966]: pgmap v2817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:20 smithi189 ceph-mon[36790]: pgmap v2817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:30:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3779.547105 secs 2024-07-26T12:30:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3779.547105 secs 2024-07-26T12:30:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3779.547105 secs 2024-07-26T12:30:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:22 smithi193 ceph-mon[37473]: pgmap v2818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:22.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:22 smithi134 ceph-mon[31966]: pgmap v2818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:22 smithi189 ceph-mon[36790]: pgmap v2818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:22.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:30:22.982+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:30:22.984 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:30:22.982+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:30:22.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:30:22.983+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T12:30:22.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:30:22.983+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T12:30:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3765.121055 secs 2024-07-26T12:30:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:24 smithi193 ceph-mon[37473]: pgmap v2819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:30:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3765.121055 secs 2024-07-26T12:30:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:24 smithi134 ceph-mon[31966]: pgmap v2819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:30:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3765.121055 secs 2024-07-26T12:30:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:24 smithi189 ceph-mon[36790]: pgmap v2819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:30:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:30:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:30:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:25 smithi134 ceph-mon[31966]: pgmap v2820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:30:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:25 smithi134 ceph-mon[31966]: pgmap v2821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:30:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:30:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:30:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:30:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:25 smithi189 ceph-mon[36790]: pgmap v2820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:30:25.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:25 smithi189 ceph-mon[36790]: pgmap v2821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:30:25.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:30:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:30:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:30:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:25 smithi193 ceph-mon[37473]: pgmap v2820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:30:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:25 smithi193 ceph-mon[37473]: pgmap v2821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:30:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:30:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3784.547239 secs 2024-07-26T12:30:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3784.547239 secs 2024-07-26T12:30:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3784.547239 secs 2024-07-26T12:30:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:27 smithi134 ceph-mon[31966]: pgmap v2822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:30:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:27 smithi189 ceph-mon[36790]: pgmap v2822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:30:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:27 smithi193 ceph-mon[37473]: pgmap v2822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:30:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3770.121122 secs 2024-07-26T12:30:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:29 smithi134 ceph-mon[31966]: pgmap v2823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:30:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3770.121122 secs 2024-07-26T12:30:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:29 smithi189 ceph-mon[36790]: pgmap v2823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:30:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3770.121122 secs 2024-07-26T12:30:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:29 smithi193 ceph-mon[37473]: pgmap v2823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:30:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:30:31.357+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:30:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:30:31.357+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:31.359 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:30:31.357+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:30:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:31 smithi134 ceph-mon[31966]: pgmap v2824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:30:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3789.547361 secs 2024-07-26T12:30:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:31 smithi189 ceph-mon[36790]: pgmap v2824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:30:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3789.547361 secs 2024-07-26T12:30:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:31 smithi193 ceph-mon[37473]: pgmap v2824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:30:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3789.547361 secs 2024-07-26T12:30:33.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:30:33.496+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T12:30:33.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:30:33.496+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:30:33.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:30:33.497+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T12:30:33.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:30:33.497+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:30:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:33 smithi193 ceph-mon[37473]: pgmap v2825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:30:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:33 smithi134 ceph-mon[31966]: pgmap v2825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:30:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:33 smithi189 ceph-mon[36790]: pgmap v2825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:30:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3775.121243 secs 2024-07-26T12:30:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3775.121243 secs 2024-07-26T12:30:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3775.121243 secs 2024-07-26T12:30:35.881 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:30:35.880+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:30:35.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:30:35.880+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:35.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:30:35.880+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:30:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:35 smithi134 ceph-mon[31966]: pgmap v2826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:30:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:35 smithi189 ceph-mon[36790]: pgmap v2826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:30:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:35 smithi193 ceph-mon[37473]: pgmap v2826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:30:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3794.547484 secs 2024-07-26T12:30:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3794.547484 secs 2024-07-26T12:30:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3794.547484 secs 2024-07-26T12:30:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:37 smithi134 ceph-mon[31966]: pgmap v2827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:37 smithi189 ceph-mon[36790]: pgmap v2827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:37 smithi193 ceph-mon[37473]: pgmap v2827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3780.121357 secs 2024-07-26T12:30:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:39 smithi134 ceph-mon[31966]: pgmap v2828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3780.121357 secs 2024-07-26T12:30:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:39 smithi189 ceph-mon[36790]: pgmap v2828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3780.121357 secs 2024-07-26T12:30:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:39 smithi193 ceph-mon[37473]: pgmap v2828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:41 smithi134 ceph-mon[31966]: pgmap v2829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3799.547604 secs 2024-07-26T12:30:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:41 smithi189 ceph-mon[36790]: pgmap v2829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3799.547604 secs 2024-07-26T12:30:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:41 smithi193 ceph-mon[37473]: pgmap v2829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3799.547604 secs 2024-07-26T12:30:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:43 smithi134 ceph-mon[31966]: pgmap v2830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:43 smithi189 ceph-mon[36790]: pgmap v2830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:43 smithi193 ceph-mon[37473]: pgmap v2830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3785.121479 secs 2024-07-26T12:30:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3785.121479 secs 2024-07-26T12:30:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3785.121479 secs 2024-07-26T12:30:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:45 smithi189 ceph-mon[36790]: pgmap v2831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:30:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:45 smithi193 ceph-mon[37473]: pgmap v2831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:30:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:45 smithi134 ceph-mon[31966]: pgmap v2831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:30:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:30:46.369+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:30:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:30:46.369+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:46.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:30:46.369+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:30:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3804.547665 secs 2024-07-26T12:30:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3804.547665 secs 2024-07-26T12:30:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3804.547665 secs 2024-07-26T12:30:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:47 smithi193 ceph-mon[37473]: pgmap v2832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:47 smithi134 ceph-mon[31966]: pgmap v2832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:47 smithi189 ceph-mon[36790]: pgmap v2832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:48 smithi193 ceph-mon[37473]: pgmap v2833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:48 smithi134 ceph-mon[31966]: pgmap v2833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:48 smithi189 ceph-mon[36790]: pgmap v2833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3790.121623 secs 2024-07-26T12:30:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3790.121623 secs 2024-07-26T12:30:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3790.121623 secs 2024-07-26T12:30:50.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:30:50.895+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:30:50.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:30:50.895+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:30:50.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:30:50.895+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:30:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:50 smithi193 ceph-mon[37473]: pgmap v2834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:50 smithi134 ceph-mon[31966]: pgmap v2834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:50 smithi189 ceph-mon[36790]: pgmap v2834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3809.547750 secs 2024-07-26T12:30:52.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3809.547750 secs 2024-07-26T12:30:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3809.547750 secs 2024-07-26T12:30:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:52 smithi193 ceph-mon[37473]: pgmap v2835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:52 smithi134 ceph-mon[31966]: pgmap v2835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:52 smithi189 ceph-mon[36790]: pgmap v2835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3795.121786 secs 2024-07-26T12:30:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3795.121786 secs 2024-07-26T12:30:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3795.121786 secs 2024-07-26T12:30:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:54 smithi193 ceph-mon[37473]: pgmap v2836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:30:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:54 smithi134 ceph-mon[31966]: pgmap v2836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:30:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:54 smithi189 ceph-mon[36790]: pgmap v2836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:30:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3814.547887 secs 2024-07-26T12:30:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3814.547887 secs 2024-07-26T12:30:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3814.547887 secs 2024-07-26T12:30:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:57 smithi193 ceph-mon[37473]: pgmap v2837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:57 smithi134 ceph-mon[31966]: pgmap v2837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:57 smithi189 ceph-mon[36790]: pgmap v2837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:30:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3800.121917 secs 2024-07-26T12:30:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:30:59 smithi134 ceph-mon[31966]: pgmap v2838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:30:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3800.121917 secs 2024-07-26T12:30:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:30:59 smithi189 ceph-mon[36790]: pgmap v2838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3800.121917 secs 2024-07-26T12:31:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:30:59 smithi193 ceph-mon[37473]: pgmap v2838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:31:01.379+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:31:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:31:01.379+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:31:01.379+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:31:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:01 smithi134 ceph-mon[31966]: pgmap v2839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3819.548036 secs 2024-07-26T12:31:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:01 smithi189 ceph-mon[36790]: pgmap v2839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3819.548036 secs 2024-07-26T12:31:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:01 smithi193 ceph-mon[37473]: pgmap v2839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3819.548036 secs 2024-07-26T12:31:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:03 smithi134 ceph-mon[31966]: pgmap v2840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:03 smithi189 ceph-mon[36790]: pgmap v2840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:03 smithi193 ceph-mon[37473]: pgmap v2840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3805.122043 secs 2024-07-26T12:31:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3805.122043 secs 2024-07-26T12:31:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3805.122043 secs 2024-07-26T12:31:05.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:31:05.911+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:31:05.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:31:05.911+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:05.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:31:05.911+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:31:05.913 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:05 smithi134 ceph-mon[31966]: pgmap v2841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:31:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:05 smithi189 ceph-mon[36790]: pgmap v2841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:31:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:05 smithi193 ceph-mon[37473]: pgmap v2841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:31:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3824.548186 secs 2024-07-26T12:31:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3824.548186 secs 2024-07-26T12:31:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3824.548186 secs 2024-07-26T12:31:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:07 smithi189 ceph-mon[36790]: pgmap v2842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:07 smithi193 ceph-mon[37473]: pgmap v2842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:07 smithi134 ceph-mon[31966]: pgmap v2842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3810.122167 secs 2024-07-26T12:31:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:09 smithi189 ceph-mon[36790]: pgmap v2843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3810.122167 secs 2024-07-26T12:31:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:09 smithi193 ceph-mon[37473]: pgmap v2843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3810.122167 secs 2024-07-26T12:31:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:09 smithi134 ceph-mon[31966]: pgmap v2843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:11 smithi189 ceph-mon[36790]: pgmap v2844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3829.548349 secs 2024-07-26T12:31:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:11 smithi193 ceph-mon[37473]: pgmap v2844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3829.548349 secs 2024-07-26T12:31:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:11 smithi134 ceph-mon[31966]: pgmap v2844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3829.548349 secs 2024-07-26T12:31:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:13 smithi189 ceph-mon[36790]: pgmap v2845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:13 smithi193 ceph-mon[37473]: pgmap v2845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:13 smithi134 ceph-mon[31966]: pgmap v2845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3815.122291 secs 2024-07-26T12:31:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3815.122291 secs 2024-07-26T12:31:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3815.122291 secs 2024-07-26T12:31:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:15 smithi193 ceph-mon[37473]: pgmap v2846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:31:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:15 smithi134 ceph-mon[31966]: pgmap v2846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:31:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:15 smithi189 ceph-mon[36790]: pgmap v2846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:31:16.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:31:16.394+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:31:16.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:31:16.394+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:31:16.396 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:31:16.394+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3834.548438 secs 2024-07-26T12:31:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:16 smithi193 ceph-mon[37473]: pgmap v2847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3834.548438 secs 2024-07-26T12:31:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:16 smithi134 ceph-mon[31966]: pgmap v2847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3834.548438 secs 2024-07-26T12:31:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:16 smithi189 ceph-mon[36790]: pgmap v2847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3820.122421 secs 2024-07-26T12:31:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:19 smithi134 ceph-mon[31966]: pgmap v2848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3820.122421 secs 2024-07-26T12:31:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:19 smithi189 ceph-mon[36790]: pgmap v2848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3820.122421 secs 2024-07-26T12:31:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:19 smithi193 ceph-mon[37473]: pgmap v2848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:20.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:31:20.918+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:31:20.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:31:20.918+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:31:20.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:31:20.918+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:21 smithi134 ceph-mon[31966]: pgmap v2849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3839.548561 secs 2024-07-26T12:31:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:21 smithi189 ceph-mon[36790]: pgmap v2849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3839.548561 secs 2024-07-26T12:31:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:21 smithi193 ceph-mon[37473]: pgmap v2849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3839.548561 secs 2024-07-26T12:31:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:23 smithi134 ceph-mon[31966]: pgmap v2850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:23 smithi189 ceph-mon[36790]: pgmap v2850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:24.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:23 smithi193 ceph-mon[37473]: pgmap v2850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3825.122550 secs 2024-07-26T12:31:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3825.122550 secs 2024-07-26T12:31:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3825.122550 secs 2024-07-26T12:31:25.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:25 smithi134 ceph-mon[31966]: pgmap v2851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:31:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:31:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:25 smithi189 ceph-mon[36790]: pgmap v2851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:31:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:31:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:25 smithi193 ceph-mon[37473]: pgmap v2851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:31:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:31:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:31:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:31:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:31:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:31:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:31:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:31:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:31:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3844.548679 secs 2024-07-26T12:31:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:31:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:31:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:31:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:31:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:31:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:31:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:31:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3844.548679 secs 2024-07-26T12:31:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:31:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:31:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:31:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:31:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:31:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:31:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:31:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3844.548679 secs 2024-07-26T12:31:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:27 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:31:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:27 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:31:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:27 smithi189 ceph-mon[36790]: pgmap v2852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:31:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:27 smithi189 ceph-mon[36790]: pgmap v2853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:31:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:27 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:31:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:27 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:31:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:27 smithi193 ceph-mon[37473]: pgmap v2852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:31:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:27 smithi193 ceph-mon[37473]: pgmap v2853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:31:28.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:27 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:31:28.075 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:27 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:31:28.075 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:27 smithi134 ceph-mon[31966]: pgmap v2852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:31:28.075 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:27 smithi134 ceph-mon[31966]: pgmap v2853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:31:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:29 smithi189 ceph-mon[36790]: pgmap v2854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:31:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3830.122674 secs 2024-07-26T12:31:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:29 smithi193 ceph-mon[37473]: pgmap v2854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:31:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3830.122674 secs 2024-07-26T12:31:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:29 smithi134 ceph-mon[31966]: pgmap v2854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:31:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3830.122674 secs 2024-07-26T12:31:31.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:31:31.410+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:31:31.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:31:31.410+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:31:31.411 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:31:31.410+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:31 smithi189 ceph-mon[36790]: pgmap v2855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:31:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3849.548811 secs 2024-07-26T12:31:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:31 smithi193 ceph-mon[37473]: pgmap v2855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:31:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3849.548811 secs 2024-07-26T12:31:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:31 smithi134 ceph-mon[31966]: pgmap v2855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:31:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3849.548811 secs 2024-07-26T12:31:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:32 smithi193 ceph-mon[37473]: pgmap v2856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:31:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:32 smithi134 ceph-mon[31966]: pgmap v2856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:31:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:32 smithi189 ceph-mon[36790]: pgmap v2856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:31:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3835.122800 secs 2024-07-26T12:31:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:34 smithi134 ceph-mon[31966]: pgmap v2857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3835.122800 secs 2024-07-26T12:31:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:34 smithi189 ceph-mon[36790]: pgmap v2857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3835.122800 secs 2024-07-26T12:31:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:34 smithi193 ceph-mon[37473]: pgmap v2857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:31:35.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:31:35.928+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:31:35.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:31:35.928+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:31:35.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:31:35.928+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:36 smithi134 ceph-mon[31966]: pgmap v2858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:31:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3854.548935 secs 2024-07-26T12:31:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:36 smithi189 ceph-mon[36790]: pgmap v2858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:31:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3854.548935 secs 2024-07-26T12:31:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:36 smithi193 ceph-mon[37473]: pgmap v2858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:31:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3854.548935 secs 2024-07-26T12:31:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:38 smithi134 ceph-mon[31966]: pgmap v2859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:31:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:38 smithi189 ceph-mon[36790]: pgmap v2859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:31:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:38 smithi193 ceph-mon[37473]: pgmap v2859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:31:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3840.122923 secs 2024-07-26T12:31:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3840.122923 secs 2024-07-26T12:31:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3840.122923 secs 2024-07-26T12:31:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:40 smithi134 ceph-mon[31966]: pgmap v2860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:31:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:40 smithi189 ceph-mon[36790]: pgmap v2860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:31:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:40 smithi193 ceph-mon[37473]: pgmap v2860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:31:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3859.549036 secs 2024-07-26T12:31:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3859.549036 secs 2024-07-26T12:31:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3859.549036 secs 2024-07-26T12:31:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:42 smithi134 ceph-mon[31966]: pgmap v2861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:31:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:42 smithi189 ceph-mon[36790]: pgmap v2861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:31:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:42 smithi193 ceph-mon[37473]: pgmap v2861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:31:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3845.123064 secs 2024-07-26T12:31:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:44 smithi134 ceph-mon[31966]: pgmap v2862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:31:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3845.123064 secs 2024-07-26T12:31:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:44 smithi189 ceph-mon[36790]: pgmap v2862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:31:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3845.123064 secs 2024-07-26T12:31:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:44 smithi193 ceph-mon[37473]: pgmap v2862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:31:46.414 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:31:46.413+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:31:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:31:46.413+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:31:46.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:31:46.413+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:46 smithi134 ceph-mon[31966]: pgmap v2863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3864.549133 secs 2024-07-26T12:31:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:46 smithi189 ceph-mon[36790]: pgmap v2863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3864.549133 secs 2024-07-26T12:31:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:46 smithi193 ceph-mon[37473]: pgmap v2863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3864.549133 secs 2024-07-26T12:31:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:48 smithi134 ceph-mon[31966]: pgmap v2864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:48 smithi189 ceph-mon[36790]: pgmap v2864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:48 smithi193 ceph-mon[37473]: pgmap v2864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3850.123221 secs 2024-07-26T12:31:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3850.123221 secs 2024-07-26T12:31:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3850.123221 secs 2024-07-26T12:31:50.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:31:50.936+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:31:50.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:31:50.936+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:31:50.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:31:50.936+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:31:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:50 smithi134 ceph-mon[31966]: pgmap v2865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:31:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:50 smithi189 ceph-mon[36790]: pgmap v2865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:31:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:50 smithi193 ceph-mon[37473]: pgmap v2865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:31:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3869.549237 secs 2024-07-26T12:31:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3869.549237 secs 2024-07-26T12:31:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3869.549237 secs 2024-07-26T12:31:53.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:52 smithi193 ceph-mon[37473]: pgmap v2866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:52 smithi189 ceph-mon[36790]: pgmap v2866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:52 smithi134 ceph-mon[31966]: pgmap v2866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3855.123371 secs 2024-07-26T12:31:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3855.123371 secs 2024-07-26T12:31:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3855.123371 secs 2024-07-26T12:31:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:54 smithi193 ceph-mon[37473]: pgmap v2867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:54 smithi134 ceph-mon[31966]: pgmap v2867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:54 smithi189 ceph-mon[36790]: pgmap v2867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:56 smithi193 ceph-mon[37473]: pgmap v2868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3874.549357 secs 2024-07-26T12:31:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:56 smithi134 ceph-mon[31966]: pgmap v2868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3874.549357 secs 2024-07-26T12:31:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:56 smithi189 ceph-mon[36790]: pgmap v2868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:31:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3874.549357 secs 2024-07-26T12:31:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:58 smithi193 ceph-mon[37473]: pgmap v2869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:31:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3860.123435 secs 2024-07-26T12:31:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:58 smithi134 ceph-mon[31966]: pgmap v2869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:31:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3860.123435 secs 2024-07-26T12:31:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:58 smithi189 ceph-mon[36790]: pgmap v2869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:31:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:31:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3860.123435 secs 2024-07-26T12:32:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:00 smithi193 ceph-mon[37473]: pgmap v2870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:32:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:00 smithi134 ceph-mon[31966]: pgmap v2870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:32:01.424 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:32:01.423+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:32:01.424 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:32:01.423+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:32:01.424 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:32:01.423+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:00 smithi189 ceph-mon[36790]: pgmap v2870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:32:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3879.549487 secs 2024-07-26T12:32:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3879.549487 secs 2024-07-26T12:32:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3879.549487 secs 2024-07-26T12:32:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:03 smithi193 ceph-mon[37473]: pgmap v2871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:03 smithi134 ceph-mon[31966]: pgmap v2871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:03 smithi189 ceph-mon[36790]: pgmap v2871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3865.123565 secs 2024-07-26T12:32:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3865.123565 secs 2024-07-26T12:32:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3865.123565 secs 2024-07-26T12:32:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:05 smithi134 ceph-mon[31966]: pgmap v2872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:05 smithi189 ceph-mon[36790]: pgmap v2872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:05 smithi193 ceph-mon[37473]: pgmap v2872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:05.953 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:32:05.952+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:32:05.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:32:05.952+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:32:05.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:32:05.952+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:07 smithi134 ceph-mon[31966]: pgmap v2873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3884.549608 secs 2024-07-26T12:32:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:07 smithi189 ceph-mon[36790]: pgmap v2873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3884.549608 secs 2024-07-26T12:32:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:07 smithi193 ceph-mon[37473]: pgmap v2873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3884.549608 secs 2024-07-26T12:32:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:09 smithi134 ceph-mon[31966]: pgmap v2874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3870.123687 secs 2024-07-26T12:32:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:09 smithi189 ceph-mon[36790]: pgmap v2874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3870.123687 secs 2024-07-26T12:32:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:09 smithi193 ceph-mon[37473]: pgmap v2874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3870.123687 secs 2024-07-26T12:32:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:11 smithi134 ceph-mon[31966]: pgmap v2875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:32:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:11 smithi189 ceph-mon[36790]: pgmap v2875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:32:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:11 smithi193 ceph-mon[37473]: pgmap v2875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:32:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3889.549693 secs 2024-07-26T12:32:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3889.549693 secs 2024-07-26T12:32:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3889.549693 secs 2024-07-26T12:32:13.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:13 smithi189 ceph-mon[36790]: pgmap v2876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:13 smithi134 ceph-mon[31966]: pgmap v2876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:13 smithi193 ceph-mon[37473]: pgmap v2876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3875.123816 secs 2024-07-26T12:32:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3875.123816 secs 2024-07-26T12:32:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3875.123816 secs 2024-07-26T12:32:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:15 smithi134 ceph-mon[31966]: pgmap v2877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:15 smithi189 ceph-mon[36790]: pgmap v2877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:15 smithi193 ceph-mon[37473]: pgmap v2877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:32:16.428+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:32:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:32:16.428+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:32:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:32:16.428+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:17 smithi134 ceph-mon[31966]: pgmap v2878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3894.549821 secs 2024-07-26T12:32:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:17 smithi189 ceph-mon[36790]: pgmap v2878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3894.549821 secs 2024-07-26T12:32:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:17 smithi193 ceph-mon[37473]: pgmap v2878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3894.549821 secs 2024-07-26T12:32:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:19 smithi134 ceph-mon[31966]: pgmap v2879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3880.123934 secs 2024-07-26T12:32:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:19 smithi189 ceph-mon[36790]: pgmap v2879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3880.123934 secs 2024-07-26T12:32:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:19 smithi193 ceph-mon[37473]: pgmap v2879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3880.123934 secs 2024-07-26T12:32:20.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:32:20.967+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:32:20.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:32:20.967+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:32:20.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:32:20.968+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:21 smithi134 ceph-mon[31966]: pgmap v2880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:32:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:21 smithi189 ceph-mon[36790]: pgmap v2880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:32:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:21 smithi193 ceph-mon[37473]: pgmap v2880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:32:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3899.549977 secs 2024-07-26T12:32:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3899.549977 secs 2024-07-26T12:32:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3899.549977 secs 2024-07-26T12:32:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:23 smithi134 ceph-mon[31966]: pgmap v2881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:23.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:23 smithi193 ceph-mon[37473]: pgmap v2881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:23 smithi189 ceph-mon[36790]: pgmap v2881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3885.124056 secs 2024-07-26T12:32:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3885.124056 secs 2024-07-26T12:32:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3885.124056 secs 2024-07-26T12:32:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:25 smithi134 ceph-mon[31966]: pgmap v2882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:25 smithi189 ceph-mon[36790]: pgmap v2882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:25 smithi193 ceph-mon[37473]: pgmap v2882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:26.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:32:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:32:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:32:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: pgmap v2883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3904.550103 secs 2024-07-26T12:32:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:32:27.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:32:27.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:32:27.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:32:27.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:32:27.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:32:27.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:32:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: pgmap v2883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3904.550103 secs 2024-07-26T12:32:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:32:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:32:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:32:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:32:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:32:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:32:27.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:32:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: pgmap v2883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3904.550103 secs 2024-07-26T12:32:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:32:27.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:32:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:28 smithi134 ceph-mon[31966]: pgmap v2884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:32:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:28 smithi134 ceph-mon[31966]: pgmap v2885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:32:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:28 smithi189 ceph-mon[36790]: pgmap v2884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:32:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:28 smithi189 ceph-mon[36790]: pgmap v2885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:32:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:28 smithi193 ceph-mon[37473]: pgmap v2884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:32:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:28 smithi193 ceph-mon[37473]: pgmap v2885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:32:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3890.124184 secs 2024-07-26T12:32:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3890.124184 secs 2024-07-26T12:32:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3890.124184 secs 2024-07-26T12:32:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:30 smithi134 ceph-mon[31966]: pgmap v2886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:32:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:30 smithi189 ceph-mon[36790]: pgmap v2886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:32:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:30 smithi193 ceph-mon[37473]: pgmap v2886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:32:31.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:32:31.430+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:32:31.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:32:31.430+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:32:31.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:32:31.430+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3909.550241 secs 2024-07-26T12:32:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:32 smithi134 ceph-mon[31966]: pgmap v2887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:32:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3909.550241 secs 2024-07-26T12:32:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:32 smithi189 ceph-mon[36790]: pgmap v2887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:32:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3909.550241 secs 2024-07-26T12:32:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:32 smithi193 ceph-mon[37473]: pgmap v2887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:32:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:34 smithi134 ceph-mon[31966]: pgmap v2888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:32:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3895.124320 secs 2024-07-26T12:32:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:34 smithi189 ceph-mon[36790]: pgmap v2888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:32:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3895.124320 secs 2024-07-26T12:32:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:34 smithi193 ceph-mon[37473]: pgmap v2888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:32:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3895.124320 secs 2024-07-26T12:32:35.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:32:35.981+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:32:35.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:32:35.981+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:32:35.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:32:35.981+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:36 smithi134 ceph-mon[31966]: pgmap v2889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:32:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:36 smithi189 ceph-mon[36790]: pgmap v2889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:32:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:36 smithi193 ceph-mon[37473]: pgmap v2889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:32:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3914.550384 secs 2024-07-26T12:32:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3914.550384 secs 2024-07-26T12:32:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3914.550384 secs 2024-07-26T12:32:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:38 smithi189 ceph-mon[36790]: pgmap v2890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:32:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:38 smithi193 ceph-mon[37473]: pgmap v2890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:32:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:38 smithi134 ceph-mon[31966]: pgmap v2890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:32:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3900.124456 secs 2024-07-26T12:32:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3900.124456 secs 2024-07-26T12:32:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3900.124456 secs 2024-07-26T12:32:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:40 smithi193 ceph-mon[37473]: pgmap v2891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:32:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:40 smithi134 ceph-mon[31966]: pgmap v2891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:32:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:40 smithi189 ceph-mon[36790]: pgmap v2891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:32:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3919.550499 secs 2024-07-26T12:32:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:42 smithi193 ceph-mon[37473]: pgmap v2892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:32:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3919.550499 secs 2024-07-26T12:32:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:42 smithi134 ceph-mon[31966]: pgmap v2892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:32:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3919.550499 secs 2024-07-26T12:32:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:42 smithi189 ceph-mon[36790]: pgmap v2892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:32:44.496 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:44 smithi189 ceph-mon[36790]: pgmap v2893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:32:44.496 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3905.124600 secs 2024-07-26T12:32:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:44 smithi193 ceph-mon[37473]: pgmap v2893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:32:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3905.124600 secs 2024-07-26T12:32:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:44 smithi134 ceph-mon[31966]: pgmap v2893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:32:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3905.124600 secs 2024-07-26T12:32:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:32:46.432+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:32:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:32:46.432+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:32:46.434 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:32:46.432+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:46 smithi193 ceph-mon[37473]: pgmap v2894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:32:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:46 smithi134 ceph-mon[31966]: pgmap v2894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:32:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:46 smithi189 ceph-mon[36790]: pgmap v2894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:32:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3924.550572 secs 2024-07-26T12:32:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3924.550572 secs 2024-07-26T12:32:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3924.550572 secs 2024-07-26T12:32:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:48 smithi193 ceph-mon[37473]: pgmap v2895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:48 smithi134 ceph-mon[31966]: pgmap v2895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:48 smithi189 ceph-mon[36790]: pgmap v2895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3910.124732 secs 2024-07-26T12:32:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3910.124732 secs 2024-07-26T12:32:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3910.124732 secs 2024-07-26T12:32:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:50 smithi193 ceph-mon[37473]: pgmap v2896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:32:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:50 smithi134 ceph-mon[31966]: pgmap v2896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:32:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:50 smithi189 ceph-mon[36790]: pgmap v2896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:32:50.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:32:50.989+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:32:50.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:32:50.989+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:32:50.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:32:50.989+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:32:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3929.550700 secs 2024-07-26T12:32:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:52 smithi193 ceph-mon[37473]: pgmap v2897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3929.550700 secs 2024-07-26T12:32:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:52 smithi134 ceph-mon[31966]: pgmap v2897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3929.550700 secs 2024-07-26T12:32:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:52 smithi189 ceph-mon[36790]: pgmap v2897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:32:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:54 smithi193 ceph-mon[37473]: pgmap v2898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3915.124881 secs 2024-07-26T12:32:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:54 smithi134 ceph-mon[31966]: pgmap v2898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3915.124881 secs 2024-07-26T12:32:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:54 smithi189 ceph-mon[36790]: pgmap v2898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:32:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3915.124881 secs 2024-07-26T12:32:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:56 smithi193 ceph-mon[37473]: pgmap v2899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:32:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:56 smithi134 ceph-mon[31966]: pgmap v2899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:32:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:56 smithi189 ceph-mon[36790]: pgmap v2899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:32:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3934.550820 secs 2024-07-26T12:32:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3934.550820 secs 2024-07-26T12:32:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3934.550820 secs 2024-07-26T12:32:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:58 smithi134 ceph-mon[31966]: pgmap v2900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:58 smithi189 ceph-mon[36790]: pgmap v2900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:58 smithi193 ceph-mon[37473]: pgmap v2900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:32:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:32:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3920.125020 secs 2024-07-26T12:32:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:32:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3920.125020 secs 2024-07-26T12:32:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:32:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3920.125020 secs 2024-07-26T12:33:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:00 smithi134 ceph-mon[31966]: pgmap v2901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:00 smithi189 ceph-mon[36790]: pgmap v2901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:00 smithi193 ceph-mon[37473]: pgmap v2901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:01.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:01.448+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:33:01.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:01.448+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:33:01.450 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:01.448+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3939.550946 secs 2024-07-26T12:33:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:02 smithi134 ceph-mon[31966]: pgmap v2902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3939.550946 secs 2024-07-26T12:33:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:02 smithi189 ceph-mon[36790]: pgmap v2902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3939.550946 secs 2024-07-26T12:33:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:02 smithi193 ceph-mon[37473]: pgmap v2902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:04 smithi134 ceph-mon[31966]: pgmap v2903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3925.125155 secs 2024-07-26T12:33:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:04 smithi189 ceph-mon[36790]: pgmap v2903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3925.125155 secs 2024-07-26T12:33:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:04 smithi193 ceph-mon[37473]: pgmap v2903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3925.125155 secs 2024-07-26T12:33:06.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:06.005+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:33:06.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:06.005+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:33:06.007 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:06.005+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:06 smithi134 ceph-mon[31966]: pgmap v2904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:33:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:06 smithi189 ceph-mon[36790]: pgmap v2904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:33:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:06 smithi193 ceph-mon[37473]: pgmap v2904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:33:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3944.551023 secs 2024-07-26T12:33:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3944.551023 secs 2024-07-26T12:33:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3944.551023 secs 2024-07-26T12:33:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:08 smithi134 ceph-mon[31966]: pgmap v2905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:08 smithi189 ceph-mon[36790]: pgmap v2905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:08 smithi193 ceph-mon[37473]: pgmap v2905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3930.125287 secs 2024-07-26T12:33:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3930.125287 secs 2024-07-26T12:33:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3930.125287 secs 2024-07-26T12:33:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:10 smithi134 ceph-mon[31966]: pgmap v2906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:10 smithi189 ceph-mon[36790]: pgmap v2906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:10 smithi193 ceph-mon[37473]: pgmap v2906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3949.551159 secs 2024-07-26T12:33:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:12 smithi134 ceph-mon[31966]: pgmap v2907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3949.551159 secs 2024-07-26T12:33:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:12 smithi189 ceph-mon[36790]: pgmap v2907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3949.551159 secs 2024-07-26T12:33:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:12 smithi193 ceph-mon[37473]: pgmap v2907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:14 smithi134 ceph-mon[31966]: pgmap v2908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3935.125423 secs 2024-07-26T12:33:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:14 smithi189 ceph-mon[36790]: pgmap v2908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3935.125423 secs 2024-07-26T12:33:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:14 smithi193 ceph-mon[37473]: pgmap v2908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3935.125423 secs 2024-07-26T12:33:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:16.463+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:33:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:16.463+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:33:16.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:16.463+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:16 smithi134 ceph-mon[31966]: pgmap v2909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:33:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:16 smithi189 ceph-mon[36790]: pgmap v2909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:33:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:16 smithi193 ceph-mon[37473]: pgmap v2909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:33:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3954.551283 secs 2024-07-26T12:33:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3954.551283 secs 2024-07-26T12:33:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3954.551283 secs 2024-07-26T12:33:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:18 smithi134 ceph-mon[31966]: pgmap v2910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:18 smithi189 ceph-mon[36790]: pgmap v2910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:18 smithi193 ceph-mon[37473]: pgmap v2910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3940.125570 secs 2024-07-26T12:33:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3940.125570 secs 2024-07-26T12:33:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3940.125570 secs 2024-07-26T12:33:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:20 smithi134 ceph-mon[31966]: pgmap v2911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:20.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:20 smithi189 ceph-mon[36790]: pgmap v2911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:20 smithi193 ceph-mon[37473]: pgmap v2911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:21.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:21.013+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:33:21.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:21.014+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:33:21.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:21.014+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3959.551326 secs 2024-07-26T12:33:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:22 smithi193 ceph-mon[37473]: pgmap v2912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3959.551326 secs 2024-07-26T12:33:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:22 smithi134 ceph-mon[31966]: pgmap v2912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3959.551326 secs 2024-07-26T12:33:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:22 smithi189 ceph-mon[36790]: pgmap v2912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:23.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:23.014+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T12:33:23.016 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:23.014+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T12:33:23.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:23.015+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T12:33:23.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:23.015+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T12:33:23.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:23.015+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T12:33:23.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:23.015+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T12:33:23.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:23.015+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T12:33:23.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:23.016+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T12:33:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:24 smithi134 ceph-mon[31966]: pgmap v2913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3945.125669 secs 2024-07-26T12:33:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:24 smithi189 ceph-mon[36790]: pgmap v2913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3945.125669 secs 2024-07-26T12:33:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:24 smithi193 ceph-mon[37473]: pgmap v2913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3945.125669 secs 2024-07-26T12:33:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:26 smithi134 ceph-mon[31966]: pgmap v2914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:33:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:26 smithi189 ceph-mon[36790]: pgmap v2914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:33:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:26 smithi193 ceph-mon[37473]: pgmap v2914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:33:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3964.551449 secs 2024-07-26T12:33:27.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3964.551449 secs 2024-07-26T12:33:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3964.551449 secs 2024-07-26T12:33:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:28 smithi134 ceph-mon[31966]: pgmap v2915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:33:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:28 smithi189 ceph-mon[36790]: pgmap v2915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:33:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:28 smithi193 ceph-mon[37473]: pgmap v2915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:33:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:33:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:33:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:29 smithi134 ceph-mon[31966]: pgmap v2916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:33:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:29 smithi134 ceph-mon[31966]: pgmap v2917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:33:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:33:29.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3950.125763 secs 2024-07-26T12:33:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:33:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:33:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:29 smithi189 ceph-mon[36790]: pgmap v2916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:33:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:29 smithi189 ceph-mon[36790]: pgmap v2917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:33:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:33:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3950.125763 secs 2024-07-26T12:33:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:33:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:33:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:29 smithi193 ceph-mon[37473]: pgmap v2916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:33:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:29 smithi193 ceph-mon[37473]: pgmap v2917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:33:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:33:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3950.125763 secs 2024-07-26T12:33:31.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:31.479+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:33:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:31.479+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:33:31.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:31.479+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:31 smithi193 ceph-mon[37473]: pgmap v2918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:33:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:31 smithi134 ceph-mon[31966]: pgmap v2918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:33:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:31 smithi189 ceph-mon[36790]: pgmap v2918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:33:32.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3969.551515 secs 2024-07-26T12:33:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3969.551515 secs 2024-07-26T12:33:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3969.551515 secs 2024-07-26T12:33:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:33.528+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:33:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:33.528+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T12:33:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:33.529+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T12:33:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:33.529+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T12:33:33.530 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:33.529+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:33:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:33.529+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006660 con 0x55a08e2d94d0 2024-07-26T12:33:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:33.530+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:33:33.531 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:33.530+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006660 con 0x55a08e2d94d0 2024-07-26T12:33:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:33 smithi134 ceph-mon[31966]: pgmap v2919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:33:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:33 smithi189 ceph-mon[36790]: pgmap v2919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:33:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:33 smithi193 ceph-mon[37473]: pgmap v2919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:33:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3955.125922 secs 2024-07-26T12:33:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3955.125922 secs 2024-07-26T12:33:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3955.125922 secs 2024-07-26T12:33:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:35 smithi134 ceph-mon[31966]: pgmap v2920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:33:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:35 smithi189 ceph-mon[36790]: pgmap v2920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:33:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:35 smithi193 ceph-mon[37473]: pgmap v2920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:33:36.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:36.025+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:33:36.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:36.025+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:36.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:36.025+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:33:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3974.551644 secs 2024-07-26T12:33:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:37 smithi134 ceph-mon[31966]: pgmap v2921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:33:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3974.551644 secs 2024-07-26T12:33:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:37 smithi189 ceph-mon[36790]: pgmap v2921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:33:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3974.551644 secs 2024-07-26T12:33:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:37 smithi193 ceph-mon[37473]: pgmap v2921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:33:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:39 smithi134 ceph-mon[31966]: pgmap v2922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:33:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3960.126055 secs 2024-07-26T12:33:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:39 smithi189 ceph-mon[36790]: pgmap v2922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:33:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3960.126055 secs 2024-07-26T12:33:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:39 smithi193 ceph-mon[37473]: pgmap v2922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:33:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3960.126055 secs 2024-07-26T12:33:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:41 smithi134 ceph-mon[31966]: pgmap v2923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:41 smithi189 ceph-mon[36790]: pgmap v2923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:41 smithi193 ceph-mon[37473]: pgmap v2923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3979.551760 secs 2024-07-26T12:33:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3979.551760 secs 2024-07-26T12:33:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3979.551760 secs 2024-07-26T12:33:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:43 smithi134 ceph-mon[31966]: pgmap v2924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:43 smithi189 ceph-mon[36790]: pgmap v2924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:43 smithi193 ceph-mon[37473]: pgmap v2924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:44.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3965.126136 secs 2024-07-26T12:33:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3965.126136 secs 2024-07-26T12:33:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3965.126136 secs 2024-07-26T12:33:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:45 smithi134 ceph-mon[31966]: pgmap v2925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:45 smithi189 ceph-mon[36790]: pgmap v2925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:45 smithi193 ceph-mon[37473]: pgmap v2925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:46.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:46.495+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:33:46.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:46.495+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:33:46.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:33:46.495+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3984.551905 secs 2024-07-26T12:33:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:47 smithi134 ceph-mon[31966]: pgmap v2926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3984.551905 secs 2024-07-26T12:33:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:47 smithi189 ceph-mon[36790]: pgmap v2926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3984.551905 secs 2024-07-26T12:33:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:47 smithi193 ceph-mon[37473]: pgmap v2926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:49 smithi134 ceph-mon[31966]: pgmap v2927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3970.126266 secs 2024-07-26T12:33:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:49 smithi189 ceph-mon[36790]: pgmap v2927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3970.126266 secs 2024-07-26T12:33:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:49 smithi193 ceph-mon[37473]: pgmap v2927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3970.126266 secs 2024-07-26T12:33:51.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:51.041+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:33:51.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:51.041+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:33:51.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:33:51.041+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:33:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:51 smithi134 ceph-mon[31966]: pgmap v2928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:51 smithi189 ceph-mon[36790]: pgmap v2928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:51 smithi193 ceph-mon[37473]: pgmap v2928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3989.552053 secs 2024-07-26T12:33:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3989.552053 secs 2024-07-26T12:33:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3989.552053 secs 2024-07-26T12:33:53.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:53 smithi134 ceph-mon[31966]: pgmap v2929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:53 smithi189 ceph-mon[36790]: pgmap v2929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:53 smithi193 ceph-mon[37473]: pgmap v2929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:33:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3975.126364 secs 2024-07-26T12:33:54.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3975.126364 secs 2024-07-26T12:33:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3975.126364 secs 2024-07-26T12:33:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:55 smithi134 ceph-mon[31966]: pgmap v2930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:55 smithi189 ceph-mon[36790]: pgmap v2930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:55 smithi193 ceph-mon[37473]: pgmap v2930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:33:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3994.552199 secs 2024-07-26T12:33:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:57 smithi134 ceph-mon[31966]: pgmap v2931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3994.552199 secs 2024-07-26T12:33:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:57 smithi189 ceph-mon[36790]: pgmap v2931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3994.552199 secs 2024-07-26T12:33:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:57 smithi193 ceph-mon[37473]: pgmap v2931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:33:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:59 smithi134 ceph-mon[31966]: pgmap v2932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:33:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3980.126440 secs 2024-07-26T12:33:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:59 smithi189 ceph-mon[36790]: pgmap v2932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:33:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3980.126440 secs 2024-07-26T12:33:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:59 smithi193 ceph-mon[37473]: pgmap v2932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:33:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:33:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3980.126440 secs 2024-07-26T12:34:01.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:34:01.510+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:34:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:34:01.510+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:34:01.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:34:01.510+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:01 smithi134 ceph-mon[31966]: pgmap v2933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:01 smithi189 ceph-mon[36790]: pgmap v2933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:01 smithi193 ceph-mon[37473]: pgmap v2933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 3999.552325 secs 2024-07-26T12:34:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 3999.552325 secs 2024-07-26T12:34:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 3999.552325 secs 2024-07-26T12:34:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:03 smithi134 ceph-mon[31966]: pgmap v2934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:34:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:03 smithi189 ceph-mon[36790]: pgmap v2934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:34:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:03 smithi193 ceph-mon[37473]: pgmap v2934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:34:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3985.126560 secs 2024-07-26T12:34:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3985.126560 secs 2024-07-26T12:34:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3985.126560 secs 2024-07-26T12:34:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:05 smithi189 ceph-mon[36790]: pgmap v2935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:05 smithi193 ceph-mon[37473]: pgmap v2935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:05 smithi134 ceph-mon[31966]: pgmap v2935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:06.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:34:06.056+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:34:06.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:34:06.056+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:06.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:34:06.056+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:34:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4004.552463 secs 2024-07-26T12:34:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:07 smithi189 ceph-mon[36790]: pgmap v2936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4004.552463 secs 2024-07-26T12:34:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:07 smithi193 ceph-mon[37473]: pgmap v2936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4004.552463 secs 2024-07-26T12:34:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:07 smithi134 ceph-mon[31966]: pgmap v2936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:09 smithi193 ceph-mon[37473]: pgmap v2937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:34:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3990.126689 secs 2024-07-26T12:34:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:09 smithi134 ceph-mon[31966]: pgmap v2937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:34:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3990.126689 secs 2024-07-26T12:34:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:09 smithi189 ceph-mon[36790]: pgmap v2937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:34:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3990.126689 secs 2024-07-26T12:34:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:11 smithi193 ceph-mon[37473]: pgmap v2938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:34:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:11 smithi134 ceph-mon[31966]: pgmap v2938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:34:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:11 smithi189 ceph-mon[36790]: pgmap v2938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:34:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4009.552595 secs 2024-07-26T12:34:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4009.552595 secs 2024-07-26T12:34:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4009.552595 secs 2024-07-26T12:34:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:13 smithi193 ceph-mon[37473]: pgmap v2939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:34:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:13 smithi134 ceph-mon[31966]: pgmap v2939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:34:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:13 smithi189 ceph-mon[36790]: pgmap v2939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:34:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 3995.126812 secs 2024-07-26T12:34:14.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 3995.126812 secs 2024-07-26T12:34:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 3995.126812 secs 2024-07-26T12:34:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:15 smithi193 ceph-mon[37473]: pgmap v2940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:34:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:15 smithi134 ceph-mon[31966]: pgmap v2940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:34:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:15 smithi189 ceph-mon[36790]: pgmap v2940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:34:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:34:16.526+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:34:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:34:16.526+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:34:16.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:34:16.526+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4014.552724 secs 2024-07-26T12:34:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:17 smithi193 ceph-mon[37473]: pgmap v2941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:34:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4014.552724 secs 2024-07-26T12:34:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:17 smithi134 ceph-mon[31966]: pgmap v2941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:34:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4014.552724 secs 2024-07-26T12:34:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:17 smithi189 ceph-mon[36790]: pgmap v2941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:34:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:19 smithi193 ceph-mon[37473]: pgmap v2942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:34:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4000.126913 secs 2024-07-26T12:34:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:19 smithi134 ceph-mon[31966]: pgmap v2942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:34:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4000.126913 secs 2024-07-26T12:34:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:19 smithi189 ceph-mon[36790]: pgmap v2942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:34:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4000.126913 secs 2024-07-26T12:34:21.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:34:21.072+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:34:21.074 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:34:21.072+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:34:21.074 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:34:21.072+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:21 smithi193 ceph-mon[37473]: pgmap v2943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4019.552867 secs 2024-07-26T12:34:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:21 smithi134 ceph-mon[31966]: pgmap v2943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4019.552867 secs 2024-07-26T12:34:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:21 smithi189 ceph-mon[36790]: pgmap v2943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4019.552867 secs 2024-07-26T12:34:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:23 smithi193 ceph-mon[37473]: pgmap v2944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:23 smithi134 ceph-mon[31966]: pgmap v2944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:23 smithi189 ceph-mon[36790]: pgmap v2944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4005.127011 secs 2024-07-26T12:34:24.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4005.127011 secs 2024-07-26T12:34:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4005.127011 secs 2024-07-26T12:34:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:25 smithi193 ceph-mon[37473]: pgmap v2945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:25 smithi134 ceph-mon[31966]: pgmap v2945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:25 smithi189 ceph-mon[36790]: pgmap v2945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4024.552932 secs 2024-07-26T12:34:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4024.552932 secs 2024-07-26T12:34:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4024.552932 secs 2024-07-26T12:34:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:27 smithi134 ceph-mon[31966]: pgmap v2946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:27 smithi189 ceph-mon[36790]: pgmap v2946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:27 smithi193 ceph-mon[37473]: pgmap v2946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:34:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:29 smithi134 ceph-mon[31966]: pgmap v2947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:34:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4010.127113 secs 2024-07-26T12:34:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:34:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:34:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:29 smithi189 ceph-mon[36790]: pgmap v2947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:34:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4010.127113 secs 2024-07-26T12:34:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:34:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:34:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:29 smithi193 ceph-mon[37473]: pgmap v2947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:34:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4010.127113 secs 2024-07-26T12:34:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:34:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:34:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:34:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:34:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:34:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:34:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:34:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:34:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:34:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:34:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:34:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:34:31.540+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:34:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:34:31.540+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:31.542 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:34:31.540+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:34:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:31 smithi134 ceph-mon[31966]: pgmap v2948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T12:34:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:31 smithi134 ceph-mon[31966]: pgmap v2949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:34:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4029.553060 secs 2024-07-26T12:34:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:31 smithi189 ceph-mon[36790]: pgmap v2948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T12:34:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:31 smithi189 ceph-mon[36790]: pgmap v2949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:34:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4029.553060 secs 2024-07-26T12:34:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:31 smithi193 ceph-mon[37473]: pgmap v2948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T12:34:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:31 smithi193 ceph-mon[37473]: pgmap v2949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:34:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4029.553060 secs 2024-07-26T12:34:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:33 smithi134 ceph-mon[31966]: pgmap v2950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:33 smithi189 ceph-mon[36790]: pgmap v2950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:33 smithi193 ceph-mon[37473]: pgmap v2950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4015.127238 secs 2024-07-26T12:34:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4015.127238 secs 2024-07-26T12:34:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4015.127238 secs 2024-07-26T12:34:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:35 smithi134 ceph-mon[31966]: pgmap v2951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:35 smithi189 ceph-mon[36790]: pgmap v2951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:35 smithi193 ceph-mon[37473]: pgmap v2951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:36.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:34:36.088+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:34:36.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:34:36.088+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:36.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:34:36.088+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:34:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4034.553187 secs 2024-07-26T12:34:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4034.553187 secs 2024-07-26T12:34:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4034.553187 secs 2024-07-26T12:34:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:37 smithi134 ceph-mon[31966]: pgmap v2952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:37 smithi189 ceph-mon[36790]: pgmap v2952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:37 smithi193 ceph-mon[37473]: pgmap v2952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:39 smithi134 ceph-mon[31966]: pgmap v2953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4020.127321 secs 2024-07-26T12:34:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:39 smithi189 ceph-mon[36790]: pgmap v2953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4020.127321 secs 2024-07-26T12:34:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:39 smithi193 ceph-mon[37473]: pgmap v2953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:34:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4020.127321 secs 2024-07-26T12:34:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:41 smithi134 ceph-mon[31966]: pgmap v2954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:34:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4039.553279 secs 2024-07-26T12:34:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:41 smithi189 ceph-mon[36790]: pgmap v2954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:34:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4039.553279 secs 2024-07-26T12:34:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:41 smithi193 ceph-mon[37473]: pgmap v2954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:34:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4039.553279 secs 2024-07-26T12:34:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:43 smithi134 ceph-mon[31966]: pgmap v2955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:43 smithi189 ceph-mon[36790]: pgmap v2955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:43 smithi193 ceph-mon[37473]: pgmap v2955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4025.127420 secs 2024-07-26T12:34:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4025.127420 secs 2024-07-26T12:34:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4025.127420 secs 2024-07-26T12:34:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:45 smithi134 ceph-mon[31966]: pgmap v2956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:45 smithi189 ceph-mon[36790]: pgmap v2956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:45 smithi193 ceph-mon[37473]: pgmap v2956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:46.549 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:34:46.548+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:34:46.550 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:34:46.548+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:34:46.550 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:34:46.548+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4044.553407 secs 2024-07-26T12:34:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4044.553407 secs 2024-07-26T12:34:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4044.553407 secs 2024-07-26T12:34:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:47 smithi134 ceph-mon[31966]: pgmap v2957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:47 smithi189 ceph-mon[36790]: pgmap v2957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:47 smithi193 ceph-mon[37473]: pgmap v2957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:49 smithi134 ceph-mon[31966]: pgmap v2958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:34:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4030.127541 secs 2024-07-26T12:34:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:49 smithi189 ceph-mon[36790]: pgmap v2958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:34:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4030.127541 secs 2024-07-26T12:34:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:49 smithi193 ceph-mon[37473]: pgmap v2958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:34:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4030.127541 secs 2024-07-26T12:34:51.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:34:51.103+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:34:51.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:34:51.103+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:34:51.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:34:51.103+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:34:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:51 smithi134 ceph-mon[31966]: pgmap v2959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:34:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4049.553508 secs 2024-07-26T12:34:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:51 smithi189 ceph-mon[36790]: pgmap v2959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:34:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4049.553508 secs 2024-07-26T12:34:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:51 smithi193 ceph-mon[37473]: pgmap v2959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:34:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4049.553508 secs 2024-07-26T12:34:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:53 smithi134 ceph-mon[31966]: pgmap v2960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:53 smithi189 ceph-mon[36790]: pgmap v2960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:53 smithi193 ceph-mon[37473]: pgmap v2960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:34:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4035.127667 secs 2024-07-26T12:34:54.924 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4035.127667 secs 2024-07-26T12:34:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4035.127667 secs 2024-07-26T12:34:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:55 smithi134 ceph-mon[31966]: pgmap v2961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:55 smithi189 ceph-mon[36790]: pgmap v2961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:55 smithi193 ceph-mon[37473]: pgmap v2961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:34:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4054.553684 secs 2024-07-26T12:34:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4054.553684 secs 2024-07-26T12:34:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4054.553684 secs 2024-07-26T12:34:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:57 smithi189 ceph-mon[36790]: pgmap v2962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:57 smithi193 ceph-mon[37473]: pgmap v2962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:57 smithi134 ceph-mon[31966]: pgmap v2962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:34:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:59 smithi189 ceph-mon[36790]: pgmap v2963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:34:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:34:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4040.127795 secs 2024-07-26T12:35:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:59 smithi193 ceph-mon[37473]: pgmap v2963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:00.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:34:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4040.127795 secs 2024-07-26T12:35:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:59 smithi134 ceph-mon[31966]: pgmap v2963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:34:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4040.127795 secs 2024-07-26T12:35:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:35:01.563+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:35:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:35:01.563+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:01.565 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:35:01.563+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:35:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:01 smithi193 ceph-mon[37473]: pgmap v2964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:02.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4059.553809 secs 2024-07-26T12:35:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:01 smithi134 ceph-mon[31966]: pgmap v2964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4059.553809 secs 2024-07-26T12:35:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:01 smithi189 ceph-mon[36790]: pgmap v2964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4059.553809 secs 2024-07-26T12:35:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:03 smithi193 ceph-mon[37473]: pgmap v2965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:03 smithi134 ceph-mon[31966]: pgmap v2965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:03 smithi189 ceph-mon[36790]: pgmap v2965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4045.127924 secs 2024-07-26T12:35:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4045.127924 secs 2024-07-26T12:35:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4045.127924 secs 2024-07-26T12:35:05.993 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:05 smithi134 ceph-mon[31966]: pgmap v2966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:05 smithi193 ceph-mon[37473]: pgmap v2966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:06.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:35:06.118+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:35:06.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:35:06.118+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:06.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:35:06.118+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:35:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:05 smithi189 ceph-mon[36790]: pgmap v2966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4064.553935 secs 2024-07-26T12:35:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4064.553935 secs 2024-07-26T12:35:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4064.553935 secs 2024-07-26T12:35:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:07 smithi193 ceph-mon[37473]: pgmap v2967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:07 smithi134 ceph-mon[31966]: pgmap v2967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:07 smithi189 ceph-mon[36790]: pgmap v2967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:09 smithi193 ceph-mon[37473]: pgmap v2968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4050.128040 secs 2024-07-26T12:35:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:09 smithi134 ceph-mon[31966]: pgmap v2968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4050.128040 secs 2024-07-26T12:35:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:09 smithi189 ceph-mon[36790]: pgmap v2968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4050.128040 secs 2024-07-26T12:35:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:11 smithi193 ceph-mon[37473]: pgmap v2969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4069.554081 secs 2024-07-26T12:35:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:11 smithi134 ceph-mon[31966]: pgmap v2969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4069.554081 secs 2024-07-26T12:35:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:11 smithi189 ceph-mon[36790]: pgmap v2969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4069.554081 secs 2024-07-26T12:35:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:12 smithi193 ceph-mon[37473]: pgmap v2970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:12 smithi134 ceph-mon[31966]: pgmap v2970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:12 smithi189 ceph-mon[36790]: pgmap v2970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4055.128188 secs 2024-07-26T12:35:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4055.128188 secs 2024-07-26T12:35:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4055.128188 secs 2024-07-26T12:35:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:15 smithi193 ceph-mon[37473]: pgmap v2971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:15 smithi134 ceph-mon[31966]: pgmap v2971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:15 smithi189 ceph-mon[36790]: pgmap v2971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:35:16.579+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:35:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:35:16.579+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:35:16.580 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:35:16.579+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4074.554260 secs 2024-07-26T12:35:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:17 smithi134 ceph-mon[31966]: pgmap v2972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4074.554260 secs 2024-07-26T12:35:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:17 smithi189 ceph-mon[36790]: pgmap v2972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4074.554260 secs 2024-07-26T12:35:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:17 smithi193 ceph-mon[37473]: pgmap v2972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:19 smithi134 ceph-mon[31966]: pgmap v2973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4060.128328 secs 2024-07-26T12:35:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:19 smithi189 ceph-mon[36790]: pgmap v2973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4060.128328 secs 2024-07-26T12:35:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:19 smithi193 ceph-mon[37473]: pgmap v2973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:19.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4060.128328 secs 2024-07-26T12:35:21.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:35:21.131+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:35:21.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:35:21.131+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:35:21.132 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:35:21.131+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:21 smithi134 ceph-mon[31966]: pgmap v2974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:21 smithi189 ceph-mon[36790]: pgmap v2974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:21 smithi193 ceph-mon[37473]: pgmap v2974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4079.554350 secs 2024-07-26T12:35:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4079.554350 secs 2024-07-26T12:35:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4079.554350 secs 2024-07-26T12:35:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:23 smithi189 ceph-mon[36790]: pgmap v2975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:23 smithi193 ceph-mon[37473]: pgmap v2975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:23 smithi134 ceph-mon[31966]: pgmap v2975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4065.128457 secs 2024-07-26T12:35:24.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4065.128457 secs 2024-07-26T12:35:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4065.128457 secs 2024-07-26T12:35:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:25 smithi193 ceph-mon[37473]: pgmap v2976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:25 smithi134 ceph-mon[31966]: pgmap v2976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:25 smithi189 ceph-mon[36790]: pgmap v2976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4084.554415 secs 2024-07-26T12:35:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:27 smithi193 ceph-mon[37473]: pgmap v2977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4084.554415 secs 2024-07-26T12:35:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:27 smithi134 ceph-mon[31966]: pgmap v2977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4084.554415 secs 2024-07-26T12:35:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:27 smithi189 ceph-mon[36790]: pgmap v2977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:29 smithi193 ceph-mon[37473]: pgmap v2978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4070.128590 secs 2024-07-26T12:35:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:29 smithi134 ceph-mon[31966]: pgmap v2978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4070.128590 secs 2024-07-26T12:35:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:29 smithi189 ceph-mon[36790]: pgmap v2978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4070.128590 secs 2024-07-26T12:35:31.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:31 smithi193 ceph-mon[37473]: pgmap v2979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:31.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:31.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:31.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:31.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:31.499 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:31.533 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:31 smithi189 ceph-mon[36790]: pgmap v2979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:31.533 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:31.533 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:31.533 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:31.533 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:31.533 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:31.534 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:31 smithi134 ceph-mon[31966]: pgmap v2979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:31.534 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:35:31.534 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:31.534 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:31.535 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:31.535 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:31.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:35:31.585+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:35:31.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:35:31.585+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:35:31.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:35:31.585+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4089.554483 secs 2024-07-26T12:35:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4089.554483 secs 2024-07-26T12:35:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4089.554483 secs 2024-07-26T12:35:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:33 smithi134 ceph-mon[31966]: pgmap v2980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:33 smithi189 ceph-mon[36790]: pgmap v2980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:33 smithi193 ceph-mon[37473]: pgmap v2980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:35:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4075.128728 secs 2024-07-26T12:35:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4075.128728 secs 2024-07-26T12:35:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4075.128728 secs 2024-07-26T12:35:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:35 smithi134 ceph-mon[31966]: pgmap v2981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:35 smithi189 ceph-mon[36790]: pgmap v2981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:35 smithi193 ceph-mon[37473]: pgmap v2981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:35:36.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:35:36.133+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:35:36.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:35:36.133+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:35:36.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:35:36.133+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:36 smithi193 ceph-mon[37473]: pgmap v2982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:35:37.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:36 smithi193 ceph-mon[37473]: pgmap v2983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:37.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:37.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4094.554605 secs 2024-07-26T12:35:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:36 smithi134 ceph-mon[31966]: pgmap v2982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:35:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:36 smithi134 ceph-mon[31966]: pgmap v2983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4094.554605 secs 2024-07-26T12:35:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:35:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:35:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:36 smithi189 ceph-mon[36790]: pgmap v2982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:35:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:36 smithi189 ceph-mon[36790]: pgmap v2983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:35:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4094.554605 secs 2024-07-26T12:35:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:38 smithi134 ceph-mon[31966]: pgmap v2984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:38 smithi189 ceph-mon[36790]: pgmap v2984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:38 smithi193 ceph-mon[37473]: pgmap v2984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4080.128849 secs 2024-07-26T12:35:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4080.128849 secs 2024-07-26T12:35:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4080.128849 secs 2024-07-26T12:35:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:40 smithi134 ceph-mon[31966]: pgmap v2985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:35:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:40 smithi189 ceph-mon[36790]: pgmap v2985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:35:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:40 smithi193 ceph-mon[37473]: pgmap v2985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:35:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4099.554728 secs 2024-07-26T12:35:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4099.554728 secs 2024-07-26T12:35:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4099.554728 secs 2024-07-26T12:35:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:42 smithi134 ceph-mon[31966]: pgmap v2986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:35:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:42 smithi189 ceph-mon[36790]: pgmap v2986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:35:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:42 smithi193 ceph-mon[37473]: pgmap v2986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:35:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4085.128973 secs 2024-07-26T12:35:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:44 smithi134 ceph-mon[31966]: pgmap v2987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4085.128973 secs 2024-07-26T12:35:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:44 smithi189 ceph-mon[36790]: pgmap v2987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4085.128973 secs 2024-07-26T12:35:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:44 smithi193 ceph-mon[37473]: pgmap v2987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:35:46.601 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:35:46.600+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:35:46.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:35:46.600+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:35:46.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:35:46.600+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:46 smithi134 ceph-mon[31966]: pgmap v2988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:35:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4104.554860 secs 2024-07-26T12:35:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:46 smithi189 ceph-mon[36790]: pgmap v2988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:35:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4104.554860 secs 2024-07-26T12:35:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:46 smithi193 ceph-mon[37473]: pgmap v2988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:35:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4104.554860 secs 2024-07-26T12:35:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:48 smithi134 ceph-mon[31966]: pgmap v2989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:48 smithi189 ceph-mon[36790]: pgmap v2989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:48 smithi193 ceph-mon[37473]: pgmap v2989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4090.129088 secs 2024-07-26T12:35:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4090.129088 secs 2024-07-26T12:35:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4090.129088 secs 2024-07-26T12:35:51.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:35:51.138+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:35:51.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:35:51.138+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:35:51.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:35:51.138+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:35:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:50 smithi134 ceph-mon[31966]: pgmap v2990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:35:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:50 smithi189 ceph-mon[36790]: pgmap v2990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:35:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:50 smithi193 ceph-mon[37473]: pgmap v2990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:35:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4109.554992 secs 2024-07-26T12:35:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4109.554992 secs 2024-07-26T12:35:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4109.554992 secs 2024-07-26T12:35:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:52 smithi134 ceph-mon[31966]: pgmap v2991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:52 smithi189 ceph-mon[36790]: pgmap v2991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:52 smithi193 ceph-mon[37473]: pgmap v2991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4095.129170 secs 2024-07-26T12:35:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:54 smithi134 ceph-mon[31966]: pgmap v2992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4095.129170 secs 2024-07-26T12:35:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:54 smithi189 ceph-mon[36790]: pgmap v2992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4095.129170 secs 2024-07-26T12:35:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:54 smithi193 ceph-mon[37473]: pgmap v2992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:35:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:56 smithi134 ceph-mon[31966]: pgmap v2993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4114.555117 secs 2024-07-26T12:35:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:56 smithi189 ceph-mon[36790]: pgmap v2993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4114.555117 secs 2024-07-26T12:35:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:56 smithi193 ceph-mon[37473]: pgmap v2993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:35:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4114.555117 secs 2024-07-26T12:35:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:58 smithi189 ceph-mon[36790]: pgmap v2994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:35:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:58 smithi193 ceph-mon[37473]: pgmap v2994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:35:59.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:58 smithi134 ceph-mon[31966]: pgmap v2994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:36:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:35:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4100.129279 secs 2024-07-26T12:36:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:35:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4100.129279 secs 2024-07-26T12:36:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:35:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4100.129279 secs 2024-07-26T12:36:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:00 smithi189 ceph-mon[36790]: pgmap v2995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:36:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:00 smithi193 ceph-mon[37473]: pgmap v2995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:36:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:00 smithi134 ceph-mon[31966]: pgmap v2995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:36:01.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:36:01.616+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:36:01.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:36:01.616+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:36:01.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:36:01.616+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4119.555240 secs 2024-07-26T12:36:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4119.555240 secs 2024-07-26T12:36:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4119.555240 secs 2024-07-26T12:36:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:02 smithi193 ceph-mon[37473]: pgmap v2996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:36:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:02 smithi134 ceph-mon[31966]: pgmap v2996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:36:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:02 smithi189 ceph-mon[36790]: pgmap v2996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:36:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4105.129405 secs 2024-07-26T12:36:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:04 smithi193 ceph-mon[37473]: pgmap v2997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:36:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4105.129405 secs 2024-07-26T12:36:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:04 smithi134 ceph-mon[31966]: pgmap v2997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:36:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4105.129405 secs 2024-07-26T12:36:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:04 smithi189 ceph-mon[36790]: pgmap v2997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:36:06.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:36:06.153+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:36:06.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:36:06.153+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:36:06.155 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:36:06.153+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:06 smithi193 ceph-mon[37473]: pgmap v2998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:36:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4124.555347 secs 2024-07-26T12:36:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:06 smithi134 ceph-mon[31966]: pgmap v2998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:36:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4124.555347 secs 2024-07-26T12:36:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:06 smithi189 ceph-mon[36790]: pgmap v2998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:36:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4124.555347 secs 2024-07-26T12:36:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:08 smithi193 ceph-mon[37473]: pgmap v2999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:36:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4110.129470 secs 2024-07-26T12:36:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:08 smithi134 ceph-mon[31966]: pgmap v2999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:36:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4110.129470 secs 2024-07-26T12:36:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:08 smithi189 ceph-mon[36790]: pgmap v2999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:36:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4110.129470 secs 2024-07-26T12:36:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:10 smithi193 ceph-mon[37473]: pgmap v3000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:36:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:10 smithi134 ceph-mon[31966]: pgmap v3000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:36:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:10 smithi189 ceph-mon[36790]: pgmap v3000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:36:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4129.555460 secs 2024-07-26T12:36:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4129.555460 secs 2024-07-26T12:36:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4129.555460 secs 2024-07-26T12:36:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:13 smithi134 ceph-mon[31966]: pgmap v3001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:13 smithi189 ceph-mon[36790]: pgmap v3001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:13 smithi193 ceph-mon[37473]: pgmap v3001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:14.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4115.129569 secs 2024-07-26T12:36:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4115.129569 secs 2024-07-26T12:36:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4115.129569 secs 2024-07-26T12:36:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:15 smithi134 ceph-mon[31966]: pgmap v3002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:15 smithi189 ceph-mon[36790]: pgmap v3002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:15 smithi193 ceph-mon[37473]: pgmap v3002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:36:16.620+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:36:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:36:16.620+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:16.622 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:36:16.620+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:36:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:17 smithi134 ceph-mon[31966]: pgmap v3003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4134.555589 secs 2024-07-26T12:36:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:17 smithi189 ceph-mon[36790]: pgmap v3003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4134.555589 secs 2024-07-26T12:36:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:17 smithi193 ceph-mon[37473]: pgmap v3003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4134.555589 secs 2024-07-26T12:36:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:19 smithi134 ceph-mon[31966]: pgmap v3004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4120.129717 secs 2024-07-26T12:36:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:19 smithi189 ceph-mon[36790]: pgmap v3004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4120.129717 secs 2024-07-26T12:36:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:19 smithi193 ceph-mon[37473]: pgmap v3004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4120.129717 secs 2024-07-26T12:36:21.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:36:21.169+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:36:21.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:36:21.169+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:36:21.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:36:21.169+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:21 smithi134 ceph-mon[31966]: pgmap v3005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:36:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:21 smithi189 ceph-mon[36790]: pgmap v3005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:36:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:21 smithi193 ceph-mon[37473]: pgmap v3005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:36:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4139.555716 secs 2024-07-26T12:36:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4139.555716 secs 2024-07-26T12:36:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4139.555716 secs 2024-07-26T12:36:23.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:36:23.045+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T12:36:23.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:36:23.045+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T12:36:23.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:36:23.045+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T12:36:23.047 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:36:23.046+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T12:36:23.305 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:23 smithi134 ceph-mon[31966]: pgmap v3006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:23 smithi189 ceph-mon[36790]: pgmap v3006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:23 smithi193 ceph-mon[37473]: pgmap v3006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4125.129829 secs 2024-07-26T12:36:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4125.129829 secs 2024-07-26T12:36:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4125.129829 secs 2024-07-26T12:36:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:25 smithi134 ceph-mon[31966]: pgmap v3007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:25.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:25 smithi193 ceph-mon[37473]: pgmap v3007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:25 smithi189 ceph-mon[36790]: pgmap v3007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:27 smithi134 ceph-mon[31966]: pgmap v3008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4144.555835 secs 2024-07-26T12:36:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:27 smithi189 ceph-mon[36790]: pgmap v3008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4144.555835 secs 2024-07-26T12:36:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:27 smithi193 ceph-mon[37473]: pgmap v3008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4144.555835 secs 2024-07-26T12:36:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:29 smithi134 ceph-mon[31966]: pgmap v3009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4130.129971 secs 2024-07-26T12:36:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:29 smithi189 ceph-mon[36790]: pgmap v3009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4130.129971 secs 2024-07-26T12:36:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:29 smithi193 ceph-mon[37473]: pgmap v3009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4130.129971 secs 2024-07-26T12:36:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:31 smithi134 ceph-mon[31966]: pgmap v3010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:36:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:31 smithi189 ceph-mon[36790]: pgmap v3010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:36:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:31 smithi193 ceph-mon[37473]: pgmap v3010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:36:31.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:36:31.635+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:36:31.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:36:31.635+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:36:31.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:36:31.635+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4149.555972 secs 2024-07-26T12:36:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4149.555972 secs 2024-07-26T12:36:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4149.555972 secs 2024-07-26T12:36:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:33 smithi189 ceph-mon[36790]: pgmap v3011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:33 smithi193 ceph-mon[37473]: pgmap v3011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:36:33.560+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:36:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:36:33.560+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:36:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:36:33.560+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:36:33.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:36:33.560+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:36:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:33 smithi134 ceph-mon[31966]: pgmap v3011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4135.130093 secs 2024-07-26T12:36:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4135.130093 secs 2024-07-26T12:36:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4135.130093 secs 2024-07-26T12:36:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:35 smithi193 ceph-mon[37473]: pgmap v3012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:35 smithi134 ceph-mon[31966]: pgmap v3012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:35 smithi189 ceph-mon[36790]: pgmap v3012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:36.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:36:36.184+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:36:36.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:36:36.184+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:36.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:36:36.184+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:36:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:36:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:37 smithi193 ceph-mon[37473]: pgmap v3013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4154.556086 secs 2024-07-26T12:36:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:36:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:37 smithi134 ceph-mon[31966]: pgmap v3013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4154.556086 secs 2024-07-26T12:36:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:36:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:37 smithi189 ceph-mon[36790]: pgmap v3013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4154.556086 secs 2024-07-26T12:36:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:36:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:36:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:36:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:38 smithi193 ceph-mon[37473]: pgmap v3014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:36:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:38 smithi193 ceph-mon[37473]: pgmap v3015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:36:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:38 smithi134 ceph-mon[31966]: pgmap v3014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:36:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:38 smithi134 ceph-mon[31966]: pgmap v3015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:36:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:38 smithi189 ceph-mon[36790]: pgmap v3014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:36:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:38 smithi189 ceph-mon[36790]: pgmap v3015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:36:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4140.130220 secs 2024-07-26T12:36:39.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4140.130220 secs 2024-07-26T12:36:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4140.130220 secs 2024-07-26T12:36:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:40 smithi193 ceph-mon[37473]: pgmap v3016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:40.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:40 smithi134 ceph-mon[31966]: pgmap v3016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:40 smithi189 ceph-mon[36790]: pgmap v3016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4159.556153 secs 2024-07-26T12:36:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:42 smithi193 ceph-mon[37473]: pgmap v3017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4159.556153 secs 2024-07-26T12:36:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:42 smithi134 ceph-mon[31966]: pgmap v3017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4159.556153 secs 2024-07-26T12:36:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:42 smithi189 ceph-mon[36790]: pgmap v3017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:44 smithi193 ceph-mon[37473]: pgmap v3018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4145.130342 secs 2024-07-26T12:36:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:44 smithi134 ceph-mon[31966]: pgmap v3018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4145.130342 secs 2024-07-26T12:36:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:44 smithi189 ceph-mon[36790]: pgmap v3018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4145.130342 secs 2024-07-26T12:36:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:46 smithi193 ceph-mon[37473]: pgmap v3019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:36:46.646+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:36:46.646+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:46.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:36:46.646+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:36:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:46 smithi134 ceph-mon[31966]: pgmap v3019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:46 smithi189 ceph-mon[36790]: pgmap v3019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:36:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4164.556277 secs 2024-07-26T12:36:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4164.556277 secs 2024-07-26T12:36:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4164.556277 secs 2024-07-26T12:36:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:48 smithi193 ceph-mon[37473]: pgmap v3020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:36:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:48 smithi134 ceph-mon[31966]: pgmap v3020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:36:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:48 smithi189 ceph-mon[36790]: pgmap v3020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:36:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4150.130461 secs 2024-07-26T12:36:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4150.130461 secs 2024-07-26T12:36:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4150.130461 secs 2024-07-26T12:36:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:50 smithi134 ceph-mon[31966]: pgmap v3021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:36:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:50 smithi189 ceph-mon[36790]: pgmap v3021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:36:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:50 smithi193 ceph-mon[37473]: pgmap v3021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:36:51.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:36:51.199+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:36:51.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:36:51.199+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:36:51.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:36:51.199+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:36:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4169.556409 secs 2024-07-26T12:36:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:52 smithi134 ceph-mon[31966]: pgmap v3022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4169.556409 secs 2024-07-26T12:36:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:52 smithi189 ceph-mon[36790]: pgmap v3022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4169.556409 secs 2024-07-26T12:36:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:52 smithi193 ceph-mon[37473]: pgmap v3022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:54 smithi134 ceph-mon[31966]: pgmap v3023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4155.130615 secs 2024-07-26T12:36:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:54 smithi189 ceph-mon[36790]: pgmap v3023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4155.130615 secs 2024-07-26T12:36:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:54 smithi193 ceph-mon[37473]: pgmap v3023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:36:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4155.130615 secs 2024-07-26T12:36:56.576 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:56 smithi193 ceph-mon[37473]: pgmap v3024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:56 smithi134 ceph-mon[31966]: pgmap v3024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:56 smithi189 ceph-mon[36790]: pgmap v3024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:36:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4174.556491 secs 2024-07-26T12:36:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4174.556491 secs 2024-07-26T12:36:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4174.556491 secs 2024-07-26T12:36:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:58 smithi134 ceph-mon[31966]: pgmap v3025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:36:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:58 smithi189 ceph-mon[36790]: pgmap v3025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:36:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:58 smithi193 ceph-mon[37473]: pgmap v3025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:36:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:36:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4160.130754 secs 2024-07-26T12:36:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:36:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4160.130754 secs 2024-07-26T12:36:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:36:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4160.130754 secs 2024-07-26T12:37:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:00 smithi134 ceph-mon[31966]: pgmap v3026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:00 smithi189 ceph-mon[36790]: pgmap v3026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:00 smithi193 ceph-mon[37473]: pgmap v3026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:01.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:37:01.660+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:37:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:37:01.661+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:37:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:37:01.661+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4179.556648 secs 2024-07-26T12:37:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:02 smithi134 ceph-mon[31966]: pgmap v3027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4179.556648 secs 2024-07-26T12:37:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:02 smithi189 ceph-mon[36790]: pgmap v3027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4179.556648 secs 2024-07-26T12:37:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:02 smithi193 ceph-mon[37473]: pgmap v3027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:04 smithi134 ceph-mon[31966]: pgmap v3028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:37:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4165.130885 secs 2024-07-26T12:37:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:04 smithi189 ceph-mon[36790]: pgmap v3028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:37:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4165.130885 secs 2024-07-26T12:37:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:04 smithi193 ceph-mon[37473]: pgmap v3028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:37:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4165.130885 secs 2024-07-26T12:37:06.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:37:06.215+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:37:06.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:37:06.215+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:37:06.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:37:06.215+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:06 smithi134 ceph-mon[31966]: pgmap v3029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:06 smithi189 ceph-mon[36790]: pgmap v3029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:06 smithi193 ceph-mon[37473]: pgmap v3029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4184.556738 secs 2024-07-26T12:37:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4184.556738 secs 2024-07-26T12:37:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4184.556738 secs 2024-07-26T12:37:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:08 smithi134 ceph-mon[31966]: pgmap v3030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:08 smithi189 ceph-mon[36790]: pgmap v3030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:08 smithi193 ceph-mon[37473]: pgmap v3030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4170.131005 secs 2024-07-26T12:37:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4170.131005 secs 2024-07-26T12:37:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4170.131005 secs 2024-07-26T12:37:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:10 smithi134 ceph-mon[31966]: pgmap v3031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:10 smithi189 ceph-mon[36790]: pgmap v3031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:10 smithi193 ceph-mon[37473]: pgmap v3031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4189.556865 secs 2024-07-26T12:37:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:12 smithi134 ceph-mon[31966]: pgmap v3032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4189.556865 secs 2024-07-26T12:37:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:12 smithi189 ceph-mon[36790]: pgmap v3032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4189.556865 secs 2024-07-26T12:37:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:12 smithi193 ceph-mon[37473]: pgmap v3032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:14 smithi134 ceph-mon[31966]: pgmap v3033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:37:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4175.131124 secs 2024-07-26T12:37:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:14 smithi189 ceph-mon[36790]: pgmap v3033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:37:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4175.131124 secs 2024-07-26T12:37:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:14 smithi193 ceph-mon[37473]: pgmap v3033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:37:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4175.131124 secs 2024-07-26T12:37:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:16 smithi134 ceph-mon[31966]: pgmap v3034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:16.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:37:16.674+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:37:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:37:16.674+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:37:16.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:37:16.674+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:16 smithi189 ceph-mon[36790]: pgmap v3034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:16 smithi193 ceph-mon[37473]: pgmap v3034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4194.556975 secs 2024-07-26T12:37:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4194.556975 secs 2024-07-26T12:37:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4194.556975 secs 2024-07-26T12:37:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:18 smithi134 ceph-mon[31966]: pgmap v3035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:18 smithi189 ceph-mon[36790]: pgmap v3035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:18 smithi193 ceph-mon[37473]: pgmap v3035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4180.131189 secs 2024-07-26T12:37:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4180.131189 secs 2024-07-26T12:37:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4180.131189 secs 2024-07-26T12:37:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:20 smithi134 ceph-mon[31966]: pgmap v3036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:20 smithi189 ceph-mon[36790]: pgmap v3036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:20 smithi193 ceph-mon[37473]: pgmap v3036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:21.232 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:37:21.230+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:37:21.233 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:37:21.230+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:37:21.233 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:37:21.230+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4199.557089 secs 2024-07-26T12:37:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:22 smithi189 ceph-mon[36790]: pgmap v3037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4199.557089 secs 2024-07-26T12:37:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:22 smithi193 ceph-mon[37473]: pgmap v3037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4199.557089 secs 2024-07-26T12:37:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:22 smithi134 ceph-mon[31966]: pgmap v3037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:24 smithi189 ceph-mon[36790]: pgmap v3038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:37:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4185.131322 secs 2024-07-26T12:37:24.724 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:24 smithi134 ceph-mon[31966]: pgmap v3038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:37:24.724 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4185.131322 secs 2024-07-26T12:37:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:24 smithi193 ceph-mon[37473]: pgmap v3038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:37:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4185.131322 secs 2024-07-26T12:37:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:26 smithi189 ceph-mon[36790]: pgmap v3039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:26 smithi193 ceph-mon[37473]: pgmap v3039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:26 smithi134 ceph-mon[31966]: pgmap v3039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4204.557219 secs 2024-07-26T12:37:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4204.557219 secs 2024-07-26T12:37:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4204.557219 secs 2024-07-26T12:37:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:28 smithi193 ceph-mon[37473]: pgmap v3040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:28 smithi134 ceph-mon[31966]: pgmap v3040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:28 smithi189 ceph-mon[36790]: pgmap v3040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:37:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4190.131425 secs 2024-07-26T12:37:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4190.131425 secs 2024-07-26T12:37:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4190.131425 secs 2024-07-26T12:37:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:30 smithi193 ceph-mon[37473]: pgmap v3041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:30 smithi134 ceph-mon[31966]: pgmap v3041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:30 smithi189 ceph-mon[36790]: pgmap v3041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:31.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:37:31.690+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:37:31.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:37:31.690+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:37:31.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:37:31.690+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4209.557351 secs 2024-07-26T12:37:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:32 smithi193 ceph-mon[37473]: pgmap v3042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4209.557351 secs 2024-07-26T12:37:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:32 smithi134 ceph-mon[31966]: pgmap v3042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4209.557351 secs 2024-07-26T12:37:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:32 smithi189 ceph-mon[36790]: pgmap v3042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:34 smithi193 ceph-mon[37473]: pgmap v3043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:37:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4195.131551 secs 2024-07-26T12:37:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:34 smithi134 ceph-mon[31966]: pgmap v3043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:37:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4195.131551 secs 2024-07-26T12:37:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:34 smithi189 ceph-mon[36790]: pgmap v3043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:37:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4195.131551 secs 2024-07-26T12:37:36.248 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:37:36.246+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:37:36.248 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:37:36.246+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:37:36.249 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:37:36.246+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:36 smithi134 ceph-mon[31966]: pgmap v3044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:36 smithi193 ceph-mon[37473]: pgmap v3044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:36 smithi189 ceph-mon[36790]: pgmap v3044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4214.557508 secs 2024-07-26T12:37:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4214.557508 secs 2024-07-26T12:37:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4214.557508 secs 2024-07-26T12:37:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:37:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:38 smithi193 ceph-mon[37473]: pgmap v3045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:37:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:37:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:38 smithi134 ceph-mon[31966]: pgmap v3045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:37:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:37:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:38 smithi189 ceph-mon[36790]: pgmap v3045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:37:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:37:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:37:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:37:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:39 smithi193 ceph-mon[37473]: pgmap v3046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:37:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:39 smithi193 ceph-mon[37473]: pgmap v3047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:37:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4200.131666 secs 2024-07-26T12:37:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:39 smithi134 ceph-mon[31966]: pgmap v3046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:37:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:39 smithi134 ceph-mon[31966]: pgmap v3047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:37:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4200.131666 secs 2024-07-26T12:37:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:39 smithi189 ceph-mon[36790]: pgmap v3046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:37:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:39 smithi189 ceph-mon[36790]: pgmap v3047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:37:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4200.131666 secs 2024-07-26T12:37:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:41 smithi193 ceph-mon[37473]: pgmap v3048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:37:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:41 smithi134 ceph-mon[31966]: pgmap v3048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:37:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:41 smithi189 ceph-mon[36790]: pgmap v3048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:37:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4219.557639 secs 2024-07-26T12:37:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4219.557639 secs 2024-07-26T12:37:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4219.557639 secs 2024-07-26T12:37:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:43 smithi134 ceph-mon[31966]: pgmap v3049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:37:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:43 smithi189 ceph-mon[36790]: pgmap v3049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:37:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:43 smithi193 ceph-mon[37473]: pgmap v3049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:37:44.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4205.131785 secs 2024-07-26T12:37:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4205.131785 secs 2024-07-26T12:37:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4205.131785 secs 2024-07-26T12:37:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:45 smithi134 ceph-mon[31966]: pgmap v3050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:37:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:45 smithi189 ceph-mon[36790]: pgmap v3050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:37:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:45 smithi193 ceph-mon[37473]: pgmap v3050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:37:46.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:37:46.705+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:37:46.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:37:46.705+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:37:46.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:37:46.705+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4224.557765 secs 2024-07-26T12:37:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4224.557765 secs 2024-07-26T12:37:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4224.557765 secs 2024-07-26T12:37:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:47 smithi134 ceph-mon[31966]: pgmap v3051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:37:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:47 smithi189 ceph-mon[36790]: pgmap v3051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:37:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:47 smithi193 ceph-mon[37473]: pgmap v3051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:37:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:49 smithi134 ceph-mon[31966]: pgmap v3052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:37:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4210.131928 secs 2024-07-26T12:37:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:49 smithi189 ceph-mon[36790]: pgmap v3052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:37:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4210.131928 secs 2024-07-26T12:37:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:49 smithi193 ceph-mon[37473]: pgmap v3052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:37:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4210.131928 secs 2024-07-26T12:37:51.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:37:51.259+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:37:51.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:37:51.259+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:37:51.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:37:51.259+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:37:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:51 smithi134 ceph-mon[31966]: pgmap v3053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4229.557863 secs 2024-07-26T12:37:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:51 smithi189 ceph-mon[36790]: pgmap v3053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4229.557863 secs 2024-07-26T12:37:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:51 smithi193 ceph-mon[37473]: pgmap v3053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4229.557863 secs 2024-07-26T12:37:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:53 smithi134 ceph-mon[31966]: pgmap v3054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:37:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:53 smithi189 ceph-mon[36790]: pgmap v3054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:37:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:53 smithi193 ceph-mon[37473]: pgmap v3054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:37:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4215.132052 secs 2024-07-26T12:37:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4215.132052 secs 2024-07-26T12:37:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4215.132052 secs 2024-07-26T12:37:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:55 smithi134 ceph-mon[31966]: pgmap v3055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:55 smithi189 ceph-mon[36790]: pgmap v3055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:55 smithi193 ceph-mon[37473]: pgmap v3055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:37:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4234.557984 secs 2024-07-26T12:37:56.924 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4234.557984 secs 2024-07-26T12:37:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4234.557984 secs 2024-07-26T12:37:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:57 smithi134 ceph-mon[31966]: pgmap v3056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:57 smithi189 ceph-mon[36790]: pgmap v3056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:57 smithi193 ceph-mon[37473]: pgmap v3056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:37:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:59 smithi134 ceph-mon[31966]: pgmap v3057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:37:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:37:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4220.132167 secs 2024-07-26T12:37:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:59 smithi189 ceph-mon[36790]: pgmap v3057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:37:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:37:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4220.132167 secs 2024-07-26T12:38:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:59 smithi193 ceph-mon[37473]: pgmap v3057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:38:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:37:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4220.132167 secs 2024-07-26T12:38:01.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:38:01.717+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:38:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:38:01.717+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:01.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:38:01.717+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:38:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:01 smithi134 ceph-mon[31966]: pgmap v3058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4239.558105 secs 2024-07-26T12:38:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:01 smithi189 ceph-mon[36790]: pgmap v3058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:01.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4239.558105 secs 2024-07-26T12:38:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:01 smithi193 ceph-mon[37473]: pgmap v3058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4239.558105 secs 2024-07-26T12:38:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:03 smithi134 ceph-mon[31966]: pgmap v3059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:03 smithi189 ceph-mon[36790]: pgmap v3059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:03 smithi193 ceph-mon[37473]: pgmap v3059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4225.132291 secs 2024-07-26T12:38:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4225.132291 secs 2024-07-26T12:38:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4225.132291 secs 2024-07-26T12:38:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:05 smithi134 ceph-mon[31966]: pgmap v3060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:38:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:05 smithi189 ceph-mon[36790]: pgmap v3060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:38:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:05 smithi193 ceph-mon[37473]: pgmap v3060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:38:06.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:38:06.275+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:38:06.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:38:06.275+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:38:06.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:38:06.275+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4244.558236 secs 2024-07-26T12:38:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4244.558236 secs 2024-07-26T12:38:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4244.558236 secs 2024-07-26T12:38:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:07 smithi134 ceph-mon[31966]: pgmap v3061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:07 smithi189 ceph-mon[36790]: pgmap v3061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:07 smithi193 ceph-mon[37473]: pgmap v3061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:09 smithi134 ceph-mon[31966]: pgmap v3062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:38:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4230.132446 secs 2024-07-26T12:38:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:09 smithi189 ceph-mon[36790]: pgmap v3062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:38:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4230.132446 secs 2024-07-26T12:38:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:09 smithi193 ceph-mon[37473]: pgmap v3062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:38:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4230.132446 secs 2024-07-26T12:38:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:11 smithi134 ceph-mon[31966]: pgmap v3063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4249.558382 secs 2024-07-26T12:38:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:11 smithi189 ceph-mon[36790]: pgmap v3063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4249.558382 secs 2024-07-26T12:38:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:11 smithi193 ceph-mon[37473]: pgmap v3063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4249.558382 secs 2024-07-26T12:38:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:13 smithi134 ceph-mon[31966]: pgmap v3064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:13 smithi189 ceph-mon[36790]: pgmap v3064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:13 smithi193 ceph-mon[37473]: pgmap v3064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:38:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4235.132560 secs 2024-07-26T12:38:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4235.132560 secs 2024-07-26T12:38:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4235.132560 secs 2024-07-26T12:38:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:15 smithi189 ceph-mon[36790]: pgmap v3065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:38:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:15 smithi193 ceph-mon[37473]: pgmap v3065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:38:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:15 smithi134 ceph-mon[31966]: pgmap v3065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:38:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:38:16.732+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:38:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:38:16.732+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:38:16.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:38:16.732+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4254.558493 secs 2024-07-26T12:38:17.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4254.558493 secs 2024-07-26T12:38:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4254.558493 secs 2024-07-26T12:38:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:17 smithi193 ceph-mon[37473]: pgmap v3066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:17 smithi134 ceph-mon[31966]: pgmap v3066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:17 smithi189 ceph-mon[36790]: pgmap v3066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:19 smithi193 ceph-mon[37473]: pgmap v3067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:38:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4240.132682 secs 2024-07-26T12:38:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:19 smithi134 ceph-mon[31966]: pgmap v3067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:38:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4240.132682 secs 2024-07-26T12:38:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:19 smithi189 ceph-mon[36790]: pgmap v3067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:38:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4240.132682 secs 2024-07-26T12:38:21.292 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:38:21.290+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:38:21.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:38:21.290+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:21.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:38:21.290+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:38:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:21 smithi193 ceph-mon[37473]: pgmap v3068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4259.558614 secs 2024-07-26T12:38:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:21 smithi134 ceph-mon[31966]: pgmap v3068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4259.558614 secs 2024-07-26T12:38:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:21 smithi189 ceph-mon[36790]: pgmap v3068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4259.558614 secs 2024-07-26T12:38:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:23 smithi193 ceph-mon[37473]: pgmap v3069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:38:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:23 smithi134 ceph-mon[31966]: pgmap v3069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:38:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:23 smithi189 ceph-mon[36790]: pgmap v3069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:38:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4245.132802 secs 2024-07-26T12:38:25.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4245.132802 secs 2024-07-26T12:38:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4245.132802 secs 2024-07-26T12:38:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:25 smithi193 ceph-mon[37473]: pgmap v3070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:38:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:25 smithi134 ceph-mon[31966]: pgmap v3070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:38:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:25 smithi189 ceph-mon[36790]: pgmap v3070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:38:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4264.558689 secs 2024-07-26T12:38:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4264.558689 secs 2024-07-26T12:38:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4264.558689 secs 2024-07-26T12:38:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:27 smithi193 ceph-mon[37473]: pgmap v3071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:38:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:27 smithi134 ceph-mon[31966]: pgmap v3071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:38:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:27 smithi189 ceph-mon[36790]: pgmap v3071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:38:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:29 smithi193 ceph-mon[37473]: pgmap v3072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:38:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4250.132926 secs 2024-07-26T12:38:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:29 smithi134 ceph-mon[31966]: pgmap v3072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:38:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4250.132926 secs 2024-07-26T12:38:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:29 smithi189 ceph-mon[36790]: pgmap v3072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:38:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4250.132926 secs 2024-07-26T12:38:31.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:38:31.742+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:38:31.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:38:31.742+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:31.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:38:31.742+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:38:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:31 smithi193 ceph-mon[37473]: pgmap v3073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:38:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4269.558816 secs 2024-07-26T12:38:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:31 smithi134 ceph-mon[31966]: pgmap v3073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:38:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4269.558816 secs 2024-07-26T12:38:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:31 smithi189 ceph-mon[36790]: pgmap v3073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:38:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4269.558816 secs 2024-07-26T12:38:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:33 smithi193 ceph-mon[37473]: pgmap v3074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:38:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:33 smithi134 ceph-mon[31966]: pgmap v3074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:38:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:33 smithi189 ceph-mon[36790]: pgmap v3074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:38:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4255.133078 secs 2024-07-26T12:38:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4255.133078 secs 2024-07-26T12:38:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4255.133078 secs 2024-07-26T12:38:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:35 smithi134 ceph-mon[31966]: pgmap v3075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:35 smithi189 ceph-mon[36790]: pgmap v3075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:35 smithi193 ceph-mon[37473]: pgmap v3075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:38:36.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:38:36.306+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:38:36.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:38:36.306+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:38:36.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:38:36.306+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4274.558939 secs 2024-07-26T12:38:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4274.558939 secs 2024-07-26T12:38:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4274.558939 secs 2024-07-26T12:38:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:37 smithi134 ceph-mon[31966]: pgmap v3076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:37 smithi189 ceph-mon[36790]: pgmap v3076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:37 smithi193 ceph-mon[37473]: pgmap v3076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:38:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:39 smithi134 ceph-mon[31966]: pgmap v3077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:38:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4260.133198 secs 2024-07-26T12:38:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:39 smithi134 ceph-mon[31966]: pgmap v3078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:38:40.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:39 smithi134 ceph-mon[31966]: pgmap v3079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:38:40.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:38:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:39 smithi189 ceph-mon[36790]: pgmap v3077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:38:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4260.133198 secs 2024-07-26T12:38:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:39 smithi189 ceph-mon[36790]: pgmap v3078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:38:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:39 smithi189 ceph-mon[36790]: pgmap v3079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:38:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:38:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:39 smithi193 ceph-mon[37473]: pgmap v3077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:38:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4260.133198 secs 2024-07-26T12:38:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:38:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:38:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:39 smithi193 ceph-mon[37473]: pgmap v3078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:38:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:39 smithi193 ceph-mon[37473]: pgmap v3079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:38:40.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:38:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4279.559055 secs 2024-07-26T12:38:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:42 smithi193 ceph-mon[37473]: pgmap v3080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4279.559055 secs 2024-07-26T12:38:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:42 smithi134 ceph-mon[31966]: pgmap v3080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4279.559055 secs 2024-07-26T12:38:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:42 smithi189 ceph-mon[36790]: pgmap v3080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4265.133337 secs 2024-07-26T12:38:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:44 smithi193 ceph-mon[37473]: pgmap v3081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:38:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4265.133337 secs 2024-07-26T12:38:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:44 smithi134 ceph-mon[31966]: pgmap v3081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:38:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4265.133337 secs 2024-07-26T12:38:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:44 smithi189 ceph-mon[36790]: pgmap v3081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:38:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:38:46.758+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:38:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:38:46.758+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:38:46.759 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:38:46.758+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:46 smithi193 ceph-mon[37473]: pgmap v3082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:38:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:46 smithi134 ceph-mon[31966]: pgmap v3082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:38:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:46 smithi189 ceph-mon[36790]: pgmap v3082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:38:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4284.559174 secs 2024-07-26T12:38:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4284.559174 secs 2024-07-26T12:38:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4284.559174 secs 2024-07-26T12:38:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:48 smithi134 ceph-mon[31966]: pgmap v3083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:38:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:48 smithi189 ceph-mon[36790]: pgmap v3083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:38:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:48 smithi193 ceph-mon[37473]: pgmap v3083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:38:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4270.133448 secs 2024-07-26T12:38:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4270.133448 secs 2024-07-26T12:38:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4270.133448 secs 2024-07-26T12:38:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:50 smithi134 ceph-mon[31966]: pgmap v3084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:38:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:50 smithi189 ceph-mon[36790]: pgmap v3084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:38:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:50 smithi193 ceph-mon[37473]: pgmap v3084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:38:51.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:38:51.322+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:38:51.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:38:51.322+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:38:51.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:38:51.322+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:38:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4289.559296 secs 2024-07-26T12:38:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4289.559296 secs 2024-07-26T12:38:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4289.559296 secs 2024-07-26T12:38:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:52 smithi134 ceph-mon[31966]: pgmap v3085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:52 smithi189 ceph-mon[36790]: pgmap v3085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:52 smithi193 ceph-mon[37473]: pgmap v3085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4275.133581 secs 2024-07-26T12:38:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:54 smithi134 ceph-mon[31966]: pgmap v3086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:38:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4275.133581 secs 2024-07-26T12:38:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:54 smithi189 ceph-mon[36790]: pgmap v3086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:38:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4275.133581 secs 2024-07-26T12:38:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:54 smithi193 ceph-mon[37473]: pgmap v3086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:38:56.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:56 smithi193 ceph-mon[37473]: pgmap v3087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:56.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4294.559365 secs 2024-07-26T12:38:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:56 smithi134 ceph-mon[31966]: pgmap v3087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4294.559365 secs 2024-07-26T12:38:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:56 smithi189 ceph-mon[36790]: pgmap v3087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:38:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4294.559365 secs 2024-07-26T12:38:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:58 smithi134 ceph-mon[31966]: pgmap v3088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:38:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:58 smithi189 ceph-mon[36790]: pgmap v3088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:38:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:58 smithi193 ceph-mon[37473]: pgmap v3088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:38:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:38:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4280.133730 secs 2024-07-26T12:38:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:38:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4280.133730 secs 2024-07-26T12:39:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:38:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4280.133730 secs 2024-07-26T12:39:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:00 smithi134 ceph-mon[31966]: pgmap v3089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:39:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:00 smithi189 ceph-mon[36790]: pgmap v3089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:39:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:00 smithi193 ceph-mon[37473]: pgmap v3089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:39:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:39:01.761+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:39:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:39:01.761+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:01.763 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:39:01.761+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:39:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4299.559428 secs 2024-07-26T12:39:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4299.559428 secs 2024-07-26T12:39:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4299.559428 secs 2024-07-26T12:39:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:02 smithi134 ceph-mon[31966]: pgmap v3090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:02 smithi189 ceph-mon[36790]: pgmap v3090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:02 smithi193 ceph-mon[37473]: pgmap v3090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4285.133848 secs 2024-07-26T12:39:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:04 smithi134 ceph-mon[31966]: pgmap v3091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4285.133848 secs 2024-07-26T12:39:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:04 smithi189 ceph-mon[36790]: pgmap v3091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4285.133848 secs 2024-07-26T12:39:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:04 smithi193 ceph-mon[37473]: pgmap v3091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:06.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:39:06.329+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:39:06.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:39:06.329+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:06.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:39:06.329+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:39:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:06 smithi134 ceph-mon[31966]: pgmap v3092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4304.559530 secs 2024-07-26T12:39:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:06 smithi189 ceph-mon[36790]: pgmap v3092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4304.559530 secs 2024-07-26T12:39:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:06 smithi193 ceph-mon[37473]: pgmap v3092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4304.559530 secs 2024-07-26T12:39:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:08 smithi134 ceph-mon[31966]: pgmap v3093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:39:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:08 smithi189 ceph-mon[36790]: pgmap v3093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:39:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:08 smithi193 ceph-mon[37473]: pgmap v3093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:39:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4290.133996 secs 2024-07-26T12:39:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4290.133996 secs 2024-07-26T12:39:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4290.133996 secs 2024-07-26T12:39:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:10 smithi134 ceph-mon[31966]: pgmap v3094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:39:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:10 smithi189 ceph-mon[36790]: pgmap v3094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:39:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:10 smithi193 ceph-mon[37473]: pgmap v3094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:39:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4309.559657 secs 2024-07-26T12:39:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:12 smithi134 ceph-mon[31966]: pgmap v3095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:39:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4309.559657 secs 2024-07-26T12:39:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:12 smithi189 ceph-mon[36790]: pgmap v3095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:39:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4309.559657 secs 2024-07-26T12:39:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:12 smithi193 ceph-mon[37473]: pgmap v3095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:39:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4295.134107 secs 2024-07-26T12:39:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:14 smithi134 ceph-mon[31966]: pgmap v3096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:39:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4295.134107 secs 2024-07-26T12:39:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:14 smithi189 ceph-mon[36790]: pgmap v3096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:39:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4295.134107 secs 2024-07-26T12:39:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:14 smithi193 ceph-mon[37473]: pgmap v3096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:39:16.778 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:39:16.777+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:39:16.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:39:16.777+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:39:16.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:39:16.777+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:16 smithi134 ceph-mon[31966]: pgmap v3097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:39:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4314.559782 secs 2024-07-26T12:39:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:16 smithi189 ceph-mon[36790]: pgmap v3097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:39:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4314.559782 secs 2024-07-26T12:39:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:16 smithi193 ceph-mon[37473]: pgmap v3097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:39:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4314.559782 secs 2024-07-26T12:39:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:18 smithi134 ceph-mon[31966]: pgmap v3098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:39:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:18 smithi189 ceph-mon[36790]: pgmap v3098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:39:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:18 smithi193 ceph-mon[37473]: pgmap v3098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:39:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4300.134254 secs 2024-07-26T12:39:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4300.134254 secs 2024-07-26T12:39:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4300.134254 secs 2024-07-26T12:39:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:20 smithi134 ceph-mon[31966]: pgmap v3099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:39:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:20 smithi189 ceph-mon[36790]: pgmap v3099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:39:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:20 smithi193 ceph-mon[37473]: pgmap v3099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:39:21.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:39:21.337+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:39:21.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:39:21.337+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:21.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:39:21.337+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:39:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4319.559905 secs 2024-07-26T12:39:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4319.559905 secs 2024-07-26T12:39:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4319.559905 secs 2024-07-26T12:39:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:22 smithi134 ceph-mon[31966]: pgmap v3100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:22 smithi189 ceph-mon[36790]: pgmap v3100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:22 smithi193 ceph-mon[37473]: pgmap v3100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:23.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:39:23.077+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T12:39:23.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:39:23.077+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T12:39:23.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:39:23.078+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T12:39:23.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:39:23.078+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T12:39:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4305.134381 secs 2024-07-26T12:39:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:24 smithi134 ceph-mon[31966]: pgmap v3101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4305.134381 secs 2024-07-26T12:39:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:24 smithi189 ceph-mon[36790]: pgmap v3101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4305.134381 secs 2024-07-26T12:39:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:24 smithi193 ceph-mon[37473]: pgmap v3101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:26 smithi134 ceph-mon[31966]: pgmap v3102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4324.560043 secs 2024-07-26T12:39:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:26 smithi189 ceph-mon[36790]: pgmap v3102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4324.560043 secs 2024-07-26T12:39:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:26 smithi193 ceph-mon[37473]: pgmap v3102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4324.560043 secs 2024-07-26T12:39:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:28 smithi134 ceph-mon[31966]: pgmap v3103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:28 smithi189 ceph-mon[36790]: pgmap v3103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:28 smithi193 ceph-mon[37473]: pgmap v3103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4310.134508 secs 2024-07-26T12:39:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4310.134508 secs 2024-07-26T12:39:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4310.134508 secs 2024-07-26T12:39:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:30 smithi134 ceph-mon[31966]: pgmap v3104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:39:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:30 smithi189 ceph-mon[36790]: pgmap v3104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:39:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:30 smithi193 ceph-mon[37473]: pgmap v3104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:39:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:39:31.793+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:39:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:39:31.793+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:39:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:39:31.793+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4329.560195 secs 2024-07-26T12:39:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4329.560195 secs 2024-07-26T12:39:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4329.560195 secs 2024-07-26T12:39:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:32 smithi134 ceph-mon[31966]: pgmap v3105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:32 smithi189 ceph-mon[36790]: pgmap v3105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:32 smithi193 ceph-mon[37473]: pgmap v3105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:33.593 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:39:33.592+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:39:33.593 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:39:33.592+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:39:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:39:33.593+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:39:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:39:33.593+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:39:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4315.134647 secs 2024-07-26T12:39:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:34 smithi134 ceph-mon[31966]: pgmap v3106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4315.134647 secs 2024-07-26T12:39:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:34 smithi189 ceph-mon[36790]: pgmap v3106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4315.134647 secs 2024-07-26T12:39:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:34 smithi193 ceph-mon[37473]: pgmap v3106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:36.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:39:36.346+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:39:36.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:39:36.346+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:39:36.348 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:39:36.346+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:36 smithi134 ceph-mon[31966]: pgmap v3107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4334.560338 secs 2024-07-26T12:39:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:36 smithi189 ceph-mon[36790]: pgmap v3107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4334.560338 secs 2024-07-26T12:39:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:36 smithi193 ceph-mon[37473]: pgmap v3107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4334.560338 secs 2024-07-26T12:39:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:38 smithi134 ceph-mon[31966]: pgmap v3108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:38 smithi189 ceph-mon[36790]: pgmap v3108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:38 smithi193 ceph-mon[37473]: pgmap v3108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4320.134764 secs 2024-07-26T12:39:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4320.134764 secs 2024-07-26T12:39:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4320.134764 secs 2024-07-26T12:39:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:40 smithi134 ceph-mon[31966]: pgmap v3109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:39:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:40.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:40.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:39:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:40 smithi189 ceph-mon[36790]: pgmap v3109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:39:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:39:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:40 smithi193 ceph-mon[37473]: pgmap v3109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:39:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:39:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:39:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:39:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:39:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:41 smithi134 ceph-mon[31966]: pgmap v3110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:39:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:41 smithi134 ceph-mon[31966]: pgmap v3111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:39:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4339.560462 secs 2024-07-26T12:39:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:41 smithi189 ceph-mon[36790]: pgmap v3110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:39:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:41 smithi189 ceph-mon[36790]: pgmap v3111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:39:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4339.560462 secs 2024-07-26T12:39:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:41 smithi193 ceph-mon[37473]: pgmap v3110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:39:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:41 smithi193 ceph-mon[37473]: pgmap v3111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:39:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4339.560462 secs 2024-07-26T12:39:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:43 smithi134 ceph-mon[31966]: pgmap v3112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:39:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:43 smithi189 ceph-mon[36790]: pgmap v3112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:39:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:43 smithi193 ceph-mon[37473]: pgmap v3112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:39:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4325.134883 secs 2024-07-26T12:39:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4325.134883 secs 2024-07-26T12:39:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4325.134883 secs 2024-07-26T12:39:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:45 smithi189 ceph-mon[36790]: pgmap v3113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:39:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:45 smithi193 ceph-mon[37473]: pgmap v3113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:39:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:45 smithi134 ceph-mon[31966]: pgmap v3113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:39:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:39:46.808+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:39:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:39:46.808+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:39:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:39:46.808+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4344.560576 secs 2024-07-26T12:39:47.080 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4344.560576 secs 2024-07-26T12:39:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4344.560576 secs 2024-07-26T12:39:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:47 smithi193 ceph-mon[37473]: pgmap v3114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:39:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:47 smithi134 ceph-mon[31966]: pgmap v3114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:39:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:47 smithi189 ceph-mon[36790]: pgmap v3114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:39:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4330.135011 secs 2024-07-26T12:39:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:49 smithi193 ceph-mon[37473]: pgmap v3115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:39:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4330.135011 secs 2024-07-26T12:39:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:49 smithi134 ceph-mon[31966]: pgmap v3115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:39:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4330.135011 secs 2024-07-26T12:39:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:49 smithi189 ceph-mon[36790]: pgmap v3115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:39:51.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:39:51.351+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:39:51.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:39:51.351+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:39:51.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:39:51.351+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:39:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:51 smithi193 ceph-mon[37473]: pgmap v3116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:39:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4349.560708 secs 2024-07-26T12:39:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:51 smithi134 ceph-mon[31966]: pgmap v3116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:39:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4349.560708 secs 2024-07-26T12:39:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:51 smithi189 ceph-mon[36790]: pgmap v3116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:39:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4349.560708 secs 2024-07-26T12:39:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:53 smithi193 ceph-mon[37473]: pgmap v3117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:53 smithi134 ceph-mon[31966]: pgmap v3117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:53 smithi189 ceph-mon[36790]: pgmap v3117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:39:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4335.135155 secs 2024-07-26T12:39:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4335.135155 secs 2024-07-26T12:39:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4335.135155 secs 2024-07-26T12:39:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:55 smithi193 ceph-mon[37473]: pgmap v3118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:39:56.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:55 smithi134 ceph-mon[31966]: pgmap v3118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:39:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:55 smithi189 ceph-mon[36790]: pgmap v3118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:39:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4354.560830 secs 2024-07-26T12:39:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4354.560830 secs 2024-07-26T12:39:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4354.560830 secs 2024-07-26T12:39:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:57 smithi193 ceph-mon[37473]: pgmap v3119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:57 smithi134 ceph-mon[31966]: pgmap v3119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:39:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:57 smithi189 ceph-mon[36790]: pgmap v3119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4340.135276 secs 2024-07-26T12:40:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:39:59 smithi193 ceph-mon[37473]: pgmap v3120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4340.135276 secs 2024-07-26T12:40:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:39:59 smithi134 ceph-mon[31966]: pgmap v3120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4340.135276 secs 2024-07-26T12:40:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:39:59 smithi189 ceph-mon[36790]: pgmap v3120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:40:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:40:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:40:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:40:01.810+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:40:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:40:01.810+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:01.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:40:01.810+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:40:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:01 smithi134 ceph-mon[31966]: pgmap v3121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4359.560952 secs 2024-07-26T12:40:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:01 smithi189 ceph-mon[36790]: pgmap v3121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4359.560952 secs 2024-07-26T12:40:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:01 smithi193 ceph-mon[37473]: pgmap v3121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4359.560952 secs 2024-07-26T12:40:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:03 smithi134 ceph-mon[31966]: pgmap v3122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:03 smithi189 ceph-mon[36790]: pgmap v3122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:03 smithi193 ceph-mon[37473]: pgmap v3122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4345.135407 secs 2024-07-26T12:40:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4345.135407 secs 2024-07-26T12:40:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4345.135407 secs 2024-07-26T12:40:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:05 smithi134 ceph-mon[31966]: pgmap v3123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:40:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:05 smithi189 ceph-mon[36790]: pgmap v3123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:40:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:05 smithi193 ceph-mon[37473]: pgmap v3123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:40:06.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:40:06.365+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:40:06.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:40:06.365+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:06.367 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:40:06.365+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:40:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4364.561014 secs 2024-07-26T12:40:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4364.561014 secs 2024-07-26T12:40:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4364.561014 secs 2024-07-26T12:40:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:07 smithi134 ceph-mon[31966]: pgmap v3124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:07 smithi189 ceph-mon[36790]: pgmap v3124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:07 smithi193 ceph-mon[37473]: pgmap v3124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:08 smithi134 ceph-mon[31966]: pgmap v3125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:08 smithi189 ceph-mon[36790]: pgmap v3125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:08 smithi193 ceph-mon[37473]: pgmap v3125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4350.135549 secs 2024-07-26T12:40:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4350.135549 secs 2024-07-26T12:40:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4350.135549 secs 2024-07-26T12:40:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:10 smithi134 ceph-mon[31966]: pgmap v3126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:10 smithi189 ceph-mon[36790]: pgmap v3126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:10 smithi193 ceph-mon[37473]: pgmap v3126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4369.561139 secs 2024-07-26T12:40:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4369.561139 secs 2024-07-26T12:40:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4369.561139 secs 2024-07-26T12:40:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:13 smithi193 ceph-mon[37473]: pgmap v3127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:13 smithi134 ceph-mon[31966]: pgmap v3127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:13 smithi189 ceph-mon[36790]: pgmap v3127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4355.135639 secs 2024-07-26T12:40:14.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4355.135639 secs 2024-07-26T12:40:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4355.135639 secs 2024-07-26T12:40:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:15 smithi193 ceph-mon[37473]: pgmap v3128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:40:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:15 smithi134 ceph-mon[31966]: pgmap v3128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:40:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:15 smithi189 ceph-mon[36790]: pgmap v3128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:40:16.817 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:40:16.816+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:40:16.817 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:40:16.816+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:40:16.818 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:40:16.816+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4374.561214 secs 2024-07-26T12:40:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4374.561214 secs 2024-07-26T12:40:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4374.561214 secs 2024-07-26T12:40:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:17 smithi134 ceph-mon[31966]: pgmap v3129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:17 smithi189 ceph-mon[36790]: pgmap v3129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:17 smithi193 ceph-mon[37473]: pgmap v3129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4360.135787 secs 2024-07-26T12:40:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:19 smithi193 ceph-mon[37473]: pgmap v3130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4360.135787 secs 2024-07-26T12:40:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:19 smithi134 ceph-mon[31966]: pgmap v3130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4360.135787 secs 2024-07-26T12:40:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:19 smithi189 ceph-mon[36790]: pgmap v3130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:21.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:40:21.380+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:40:21.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:40:21.380+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:21.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:40:21.380+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:40:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:21 smithi193 ceph-mon[37473]: pgmap v3131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4379.561349 secs 2024-07-26T12:40:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:21 smithi134 ceph-mon[31966]: pgmap v3131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4379.561349 secs 2024-07-26T12:40:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:21 smithi189 ceph-mon[36790]: pgmap v3131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4379.561349 secs 2024-07-26T12:40:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:23 smithi193 ceph-mon[37473]: pgmap v3132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:23 smithi134 ceph-mon[31966]: pgmap v3132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:23 smithi189 ceph-mon[36790]: pgmap v3132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4365.135894 secs 2024-07-26T12:40:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4365.135894 secs 2024-07-26T12:40:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4365.135894 secs 2024-07-26T12:40:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:25 smithi193 ceph-mon[37473]: pgmap v3133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:40:26.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:25 smithi134 ceph-mon[31966]: pgmap v3133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:40:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:25 smithi189 ceph-mon[36790]: pgmap v3133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:40:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4384.561401 secs 2024-07-26T12:40:27.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4384.561401 secs 2024-07-26T12:40:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4384.561401 secs 2024-07-26T12:40:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:27 smithi193 ceph-mon[37473]: pgmap v3134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:27 smithi134 ceph-mon[31966]: pgmap v3134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:27 smithi189 ceph-mon[36790]: pgmap v3134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4370.136024 secs 2024-07-26T12:40:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:29 smithi193 ceph-mon[37473]: pgmap v3135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4370.136024 secs 2024-07-26T12:40:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:29 smithi134 ceph-mon[31966]: pgmap v3135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4370.136024 secs 2024-07-26T12:40:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:29 smithi189 ceph-mon[36790]: pgmap v3135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:40:31.831+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:40:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:40:31.831+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:31.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:40:31.831+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:40:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:31 smithi193 ceph-mon[37473]: pgmap v3136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4389.561482 secs 2024-07-26T12:40:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:31 smithi134 ceph-mon[31966]: pgmap v3136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4389.561482 secs 2024-07-26T12:40:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:31 smithi189 ceph-mon[36790]: pgmap v3136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4389.561482 secs 2024-07-26T12:40:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:33 smithi134 ceph-mon[31966]: pgmap v3137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:33 smithi189 ceph-mon[36790]: pgmap v3137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:33 smithi193 ceph-mon[37473]: pgmap v3137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4375.136117 secs 2024-07-26T12:40:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4375.136117 secs 2024-07-26T12:40:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4375.136117 secs 2024-07-26T12:40:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:35 smithi134 ceph-mon[31966]: pgmap v3138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:40:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:35 smithi189 ceph-mon[36790]: pgmap v3138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:40:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:35 smithi193 ceph-mon[37473]: pgmap v3138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:40:36.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:40:36.384+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:40:36.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:40:36.384+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:36.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:40:36.384+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:40:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4394.561584 secs 2024-07-26T12:40:37.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4394.561584 secs 2024-07-26T12:40:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4394.561584 secs 2024-07-26T12:40:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:37 smithi193 ceph-mon[37473]: pgmap v3139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:37 smithi134 ceph-mon[31966]: pgmap v3139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:37 smithi189 ceph-mon[36790]: pgmap v3139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4380.136202 secs 2024-07-26T12:40:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:39 smithi134 ceph-mon[31966]: pgmap v3140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4380.136202 secs 2024-07-26T12:40:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:39 smithi189 ceph-mon[36790]: pgmap v3140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4380.136202 secs 2024-07-26T12:40:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:39 smithi193 ceph-mon[37473]: pgmap v3140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:40 smithi134 ceph-mon[31966]: pgmap v3141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:40 smithi189 ceph-mon[36790]: pgmap v3141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:41.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:40 smithi193 ceph-mon[37473]: pgmap v3141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4399.561658 secs 2024-07-26T12:40:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4399.561658 secs 2024-07-26T12:40:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:40:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4399.561658 secs 2024-07-26T12:40:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:42 smithi134 ceph-mon[31966]: pgmap v3142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:42 smithi189 ceph-mon[36790]: pgmap v3142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:42 smithi193 ceph-mon[37473]: pgmap v3142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:40:44.334 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4385.136321 secs 2024-07-26T12:40:44.368 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4385.136321 secs 2024-07-26T12:40:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4385.136321 secs 2024-07-26T12:40:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:45 smithi134 ceph-mon[31966]: pgmap v3143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:40:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:45 smithi189 ceph-mon[36790]: pgmap v3143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:40:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:45 smithi193 ceph-mon[37473]: pgmap v3143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:40:46.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:40:46.846+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:40:46.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:40:46.846+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:46.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:40:46.846+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:40:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4404.561778 secs 2024-07-26T12:40:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4404.561778 secs 2024-07-26T12:40:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4404.561778 secs 2024-07-26T12:40:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:47 smithi134 ceph-mon[31966]: pgmap v3144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:40:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:40:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:47.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:40:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:47 smithi189 ceph-mon[36790]: pgmap v3144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:40:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:40:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:40:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:47 smithi193 ceph-mon[37473]: pgmap v3144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:40:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:40:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:40:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:40:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:40:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:40:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:48 smithi134 ceph-mon[31966]: pgmap v3145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:40:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:48 smithi134 ceph-mon[31966]: pgmap v3146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T12:40:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:48 smithi189 ceph-mon[36790]: pgmap v3145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:40:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:48 smithi189 ceph-mon[36790]: pgmap v3146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T12:40:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:48 smithi193 ceph-mon[37473]: pgmap v3145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:40:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:48 smithi193 ceph-mon[37473]: pgmap v3146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T12:40:49.807 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4390.136407 secs 2024-07-26T12:40:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4390.136407 secs 2024-07-26T12:40:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4390.136407 secs 2024-07-26T12:40:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:50 smithi134 ceph-mon[31966]: pgmap v3147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:40:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:50 smithi189 ceph-mon[36790]: pgmap v3147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:40:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:50 smithi193 ceph-mon[37473]: pgmap v3147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:40:51.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:40:51.400+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:40:51.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:40:51.400+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:40:51.403 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:40:51.400+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:40:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4409.561892 secs 2024-07-26T12:40:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4409.561892 secs 2024-07-26T12:40:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4409.561892 secs 2024-07-26T12:40:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:52 smithi134 ceph-mon[31966]: pgmap v3148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T12:40:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:52 smithi189 ceph-mon[36790]: pgmap v3148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T12:40:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:52 smithi193 ceph-mon[37473]: pgmap v3148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T12:40:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:54 smithi134 ceph-mon[31966]: pgmap v3149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T12:40:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4395.136557 secs 2024-07-26T12:40:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:54 smithi189 ceph-mon[36790]: pgmap v3149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T12:40:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4395.136557 secs 2024-07-26T12:40:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:54 smithi193 ceph-mon[37473]: pgmap v3149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T12:40:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4395.136557 secs 2024-07-26T12:40:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:56 smithi134 ceph-mon[31966]: pgmap v3150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T12:40:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4414.561958 secs 2024-07-26T12:40:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:56 smithi189 ceph-mon[36790]: pgmap v3150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T12:40:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4414.561958 secs 2024-07-26T12:40:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:56 smithi193 ceph-mon[37473]: pgmap v3150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T12:40:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4414.561958 secs 2024-07-26T12:40:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:58 smithi134 ceph-mon[31966]: pgmap v3151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:58 smithi189 ceph-mon[36790]: pgmap v3151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:58 smithi193 ceph-mon[37473]: pgmap v3151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:40:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:40:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4400.136627 secs 2024-07-26T12:40:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:40:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4400.136627 secs 2024-07-26T12:41:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:40:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4400.136627 secs 2024-07-26T12:41:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:00 smithi134 ceph-mon[31966]: pgmap v3152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:00 smithi189 ceph-mon[36790]: pgmap v3152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:00 smithi193 ceph-mon[37473]: pgmap v3152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:41:01.858+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:41:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:41:01.858+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:41:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:41:01.858+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4419.562080 secs 2024-07-26T12:41:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4419.562080 secs 2024-07-26T12:41:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4419.562080 secs 2024-07-26T12:41:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:02 smithi189 ceph-mon[36790]: pgmap v3153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:02 smithi193 ceph-mon[37473]: pgmap v3153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:02 smithi134 ceph-mon[31966]: pgmap v3153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:04 smithi189 ceph-mon[36790]: pgmap v3154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4405.136743 secs 2024-07-26T12:41:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:04 smithi193 ceph-mon[37473]: pgmap v3154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4405.136743 secs 2024-07-26T12:41:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:04 smithi134 ceph-mon[31966]: pgmap v3154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4405.136743 secs 2024-07-26T12:41:06.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:41:06.415+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:41:06.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:41:06.415+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:06.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:41:06.415+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:41:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:06 smithi189 ceph-mon[36790]: pgmap v3155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4424.562198 secs 2024-07-26T12:41:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:06 smithi193 ceph-mon[37473]: pgmap v3155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4424.562198 secs 2024-07-26T12:41:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:06 smithi134 ceph-mon[31966]: pgmap v3155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4424.562198 secs 2024-07-26T12:41:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:08 smithi193 ceph-mon[37473]: pgmap v3156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:08 smithi134 ceph-mon[31966]: pgmap v3156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:08 smithi189 ceph-mon[36790]: pgmap v3156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4410.136855 secs 2024-07-26T12:41:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4410.136855 secs 2024-07-26T12:41:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4410.136855 secs 2024-07-26T12:41:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:10 smithi193 ceph-mon[37473]: pgmap v3157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:10 smithi134 ceph-mon[31966]: pgmap v3157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:10 smithi189 ceph-mon[36790]: pgmap v3157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4429.562264 secs 2024-07-26T12:41:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4429.562264 secs 2024-07-26T12:41:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4429.562264 secs 2024-07-26T12:41:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:12 smithi193 ceph-mon[37473]: pgmap v3158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:12 smithi134 ceph-mon[31966]: pgmap v3158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:12 smithi189 ceph-mon[36790]: pgmap v3158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:14.996 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:14 smithi189 ceph-mon[36790]: pgmap v3159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:14.996 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4415.136992 secs 2024-07-26T12:41:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:14 smithi193 ceph-mon[37473]: pgmap v3159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4415.136992 secs 2024-07-26T12:41:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:14 smithi134 ceph-mon[31966]: pgmap v3159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4415.136992 secs 2024-07-26T12:41:16.875 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:41:16.873+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:41:16.875 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:41:16.873+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:41:16.875 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:41:16.873+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:16 smithi193 ceph-mon[37473]: pgmap v3160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4434.562389 secs 2024-07-26T12:41:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:16 smithi134 ceph-mon[31966]: pgmap v3160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4434.562389 secs 2024-07-26T12:41:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:16 smithi189 ceph-mon[36790]: pgmap v3160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4434.562389 secs 2024-07-26T12:41:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:18 smithi193 ceph-mon[37473]: pgmap v3161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:18 smithi134 ceph-mon[31966]: pgmap v3161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:18 smithi189 ceph-mon[36790]: pgmap v3161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4420.137134 secs 2024-07-26T12:41:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4420.137134 secs 2024-07-26T12:41:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4420.137134 secs 2024-07-26T12:41:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:20 smithi193 ceph-mon[37473]: pgmap v3162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:20 smithi134 ceph-mon[31966]: pgmap v3162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:20 smithi189 ceph-mon[36790]: pgmap v3162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:21.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:41:21.431+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:41:21.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:41:21.431+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:21.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:41:21.431+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:41:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4439.562506 secs 2024-07-26T12:41:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4439.562506 secs 2024-07-26T12:41:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4439.562506 secs 2024-07-26T12:41:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:22 smithi193 ceph-mon[37473]: pgmap v3163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:22 smithi134 ceph-mon[31966]: pgmap v3163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:22 smithi189 ceph-mon[36790]: pgmap v3163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:24 smithi134 ceph-mon[31966]: pgmap v3164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4425.137205 secs 2024-07-26T12:41:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:24 smithi189 ceph-mon[36790]: pgmap v3164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4425.137205 secs 2024-07-26T12:41:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:24 smithi193 ceph-mon[37473]: pgmap v3164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4425.137205 secs 2024-07-26T12:41:27.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:26 smithi193 ceph-mon[37473]: pgmap v3165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:27.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4444.562626 secs 2024-07-26T12:41:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:26 smithi134 ceph-mon[31966]: pgmap v3165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4444.562626 secs 2024-07-26T12:41:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:26 smithi189 ceph-mon[36790]: pgmap v3165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4444.562626 secs 2024-07-26T12:41:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:28 smithi134 ceph-mon[31966]: pgmap v3166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:28 smithi189 ceph-mon[36790]: pgmap v3166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:28 smithi193 ceph-mon[37473]: pgmap v3166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4430.137313 secs 2024-07-26T12:41:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4430.137313 secs 2024-07-26T12:41:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4430.137313 secs 2024-07-26T12:41:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:30 smithi134 ceph-mon[31966]: pgmap v3167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:30 smithi189 ceph-mon[36790]: pgmap v3167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:30 smithi193 ceph-mon[37473]: pgmap v3167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:41:31.877+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:41:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:41:31.877+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:41:31.879 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:41:31.877+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4449.562745 secs 2024-07-26T12:41:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4449.562745 secs 2024-07-26T12:41:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4449.562745 secs 2024-07-26T12:41:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:32 smithi134 ceph-mon[31966]: pgmap v3168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:32 smithi189 ceph-mon[36790]: pgmap v3168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:32 smithi193 ceph-mon[37473]: pgmap v3168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:33 smithi134 ceph-mon[31966]: pgmap v3169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:33 smithi189 ceph-mon[36790]: pgmap v3169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:33 smithi193 ceph-mon[37473]: pgmap v3169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:35.114 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4435.137427 secs 2024-07-26T12:41:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4435.137427 secs 2024-07-26T12:41:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4435.137427 secs 2024-07-26T12:41:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:35 smithi134 ceph-mon[31966]: pgmap v3170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:35 smithi189 ceph-mon[36790]: pgmap v3170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:35 smithi193 ceph-mon[37473]: pgmap v3170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:36.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:41:36.439+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:41:36.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:41:36.440+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:41:36.442 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:41:36.440+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4454.562816 secs 2024-07-26T12:41:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4454.562816 secs 2024-07-26T12:41:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4454.562816 secs 2024-07-26T12:41:38.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:41:38.101+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T12:41:38.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:41:38.101+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T12:41:38.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:41:38.102+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T12:41:38.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:41:38.102+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T12:41:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:38 smithi134 ceph-mon[31966]: pgmap v3171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:38 smithi193 ceph-mon[37473]: pgmap v3171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:38 smithi189 ceph-mon[36790]: pgmap v3171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4440.137497 secs 2024-07-26T12:41:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4440.137497 secs 2024-07-26T12:41:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4440.137497 secs 2024-07-26T12:41:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:40 smithi193 ceph-mon[37473]: pgmap v3172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:40 smithi134 ceph-mon[31966]: pgmap v3172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:40 smithi189 ceph-mon[36790]: pgmap v3172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4459.562938 secs 2024-07-26T12:41:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:42 smithi193 ceph-mon[37473]: pgmap v3173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4459.562938 secs 2024-07-26T12:41:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:42 smithi134 ceph-mon[31966]: pgmap v3173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4459.562938 secs 2024-07-26T12:41:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:42 smithi189 ceph-mon[36790]: pgmap v3173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:44 smithi193 ceph-mon[37473]: pgmap v3174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4445.137564 secs 2024-07-26T12:41:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:44 smithi134 ceph-mon[31966]: pgmap v3174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4445.137564 secs 2024-07-26T12:41:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:44 smithi189 ceph-mon[36790]: pgmap v3174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4445.137564 secs 2024-07-26T12:41:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:46 smithi193 ceph-mon[37473]: pgmap v3175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:46 smithi134 ceph-mon[31966]: pgmap v3175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:46 smithi189 ceph-mon[36790]: pgmap v3175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:41:46.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:41:46.889+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:41:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:41:46.889+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:41:46.889+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:41:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4464.563056 secs 2024-07-26T12:41:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4464.563056 secs 2024-07-26T12:41:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4464.563056 secs 2024-07-26T12:41:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:41:48.340 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:48 smithi193 ceph-mon[37473]: pgmap v3176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:48.340 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:48.340 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:48.340 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:48.340 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:48.343 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:48 smithi189 ceph-mon[36790]: pgmap v3176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:48.343 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:48.343 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:48.343 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:48.343 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:48.616 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:41:48.616+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:41:48.616 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:41:48.616+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:41:48.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:41:48.617+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:41:48.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:41:48.617+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:41:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:48 smithi134 ceph-mon[31966]: pgmap v3176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:41:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:41:49.386 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:49 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:41:49.386 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:49 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:41:49.386 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4450.137641 secs 2024-07-26T12:41:49.458 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:49 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:41:49.458 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:49 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:41:49.458 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4450.137641 secs 2024-07-26T12:41:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:49 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:41:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:49 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:41:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4450.137641 secs 2024-07-26T12:41:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:50 smithi189 ceph-mon[36790]: pgmap v3177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:50.452 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:50 smithi193 ceph-mon[37473]: pgmap v3177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:50 smithi134 ceph-mon[31966]: pgmap v3177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:41:51.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:41:51.455+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:41:51.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:41:51.455+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:41:51.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:41:51.455+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:41:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:41:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:41:52.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:41:52.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:41:52.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:52.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:52.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:51 smithi134 ceph-mon[31966]: pgmap v3178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:41:52.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:51 smithi134 ceph-mon[31966]: pgmap v3179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:41:52.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4469.563127 secs 2024-07-26T12:41:52.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:41:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:41:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:41:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:41:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:41:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:51 smithi189 ceph-mon[36790]: pgmap v3178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:41:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:51 smithi189 ceph-mon[36790]: pgmap v3179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:41:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4469.563127 secs 2024-07-26T12:41:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:41:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:41:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:41:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:41:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:41:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:41:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:41:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:51 smithi193 ceph-mon[37473]: pgmap v3178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:41:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:51 smithi193 ceph-mon[37473]: pgmap v3179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:41:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4469.563127 secs 2024-07-26T12:41:52.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:41:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:53 smithi193 ceph-mon[37473]: pgmap v3180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:41:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:53 smithi134 ceph-mon[31966]: pgmap v3180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:41:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:53 smithi189 ceph-mon[36790]: pgmap v3180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:41:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4455.137759 secs 2024-07-26T12:41:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4455.137759 secs 2024-07-26T12:41:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4455.137759 secs 2024-07-26T12:41:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:55 smithi193 ceph-mon[37473]: pgmap v3181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:41:56.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:55 smithi134 ceph-mon[31966]: pgmap v3181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:41:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:55 smithi189 ceph-mon[36790]: pgmap v3181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:41:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4474.563252 secs 2024-07-26T12:41:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4474.563252 secs 2024-07-26T12:41:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4474.563252 secs 2024-07-26T12:41:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:57 smithi193 ceph-mon[37473]: pgmap v3182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:41:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:57 smithi134 ceph-mon[31966]: pgmap v3182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:41:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:57 smithi189 ceph-mon[36790]: pgmap v3182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:42:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4460.137878 secs 2024-07-26T12:42:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:41:59 smithi193 ceph-mon[37473]: pgmap v3183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:42:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4460.137878 secs 2024-07-26T12:42:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:41:59 smithi134 ceph-mon[31966]: pgmap v3183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:42:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4460.137878 secs 2024-07-26T12:42:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:41:59 smithi189 ceph-mon[36790]: pgmap v3183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T12:42:01.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:01.901+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:42:01.914 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:01.901+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:42:01.914 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:01.901+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:01 smithi193 ceph-mon[37473]: pgmap v3184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:42:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4479.563372 secs 2024-07-26T12:42:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:01 smithi134 ceph-mon[31966]: pgmap v3184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:42:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4479.563372 secs 2024-07-26T12:42:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:01 smithi189 ceph-mon[36790]: pgmap v3184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:42:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4479.563372 secs 2024-07-26T12:42:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:03 smithi193 ceph-mon[37473]: pgmap v3185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:03 smithi134 ceph-mon[31966]: pgmap v3185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:03 smithi189 ceph-mon[36790]: pgmap v3185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4465.137981 secs 2024-07-26T12:42:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4465.137981 secs 2024-07-26T12:42:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4465.137981 secs 2024-07-26T12:42:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:06 smithi193 ceph-mon[37473]: pgmap v3186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:42:06.345 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:06 smithi134 ceph-mon[31966]: pgmap v3186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:42:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:06 smithi189 ceph-mon[36790]: pgmap v3186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:42:06.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:06.471+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:42:06.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:06.471+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:42:06.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:06.471+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4484.563489 secs 2024-07-26T12:42:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4484.563489 secs 2024-07-26T12:42:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4484.563489 secs 2024-07-26T12:42:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:08 smithi134 ceph-mon[31966]: pgmap v3187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:08 smithi189 ceph-mon[36790]: pgmap v3187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:08 smithi193 ceph-mon[37473]: pgmap v3187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4470.138122 secs 2024-07-26T12:42:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4470.138122 secs 2024-07-26T12:42:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4470.138122 secs 2024-07-26T12:42:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:10 smithi134 ceph-mon[31966]: pgmap v3188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:10 smithi189 ceph-mon[36790]: pgmap v3188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:10 smithi193 ceph-mon[37473]: pgmap v3188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4489.563605 secs 2024-07-26T12:42:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:12 smithi134 ceph-mon[31966]: pgmap v3189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4489.563605 secs 2024-07-26T12:42:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:12 smithi189 ceph-mon[36790]: pgmap v3189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4489.563605 secs 2024-07-26T12:42:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:12 smithi193 ceph-mon[37473]: pgmap v3189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:14 smithi134 ceph-mon[31966]: pgmap v3190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4475.138264 secs 2024-07-26T12:42:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:14 smithi189 ceph-mon[36790]: pgmap v3190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4475.138264 secs 2024-07-26T12:42:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:14 smithi193 ceph-mon[37473]: pgmap v3190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4475.138264 secs 2024-07-26T12:42:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:16 smithi134 ceph-mon[31966]: pgmap v3191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:42:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:16 smithi189 ceph-mon[36790]: pgmap v3191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:42:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:16 smithi193 ceph-mon[37473]: pgmap v3191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:42:16.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:16.906+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:42:16.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:16.906+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:42:16.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:16.906+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4494.563713 secs 2024-07-26T12:42:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4494.563713 secs 2024-07-26T12:42:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4494.563713 secs 2024-07-26T12:42:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:18 smithi134 ceph-mon[31966]: pgmap v3192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:18 smithi189 ceph-mon[36790]: pgmap v3192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:18 smithi193 ceph-mon[37473]: pgmap v3192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4480.138334 secs 2024-07-26T12:42:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4480.138334 secs 2024-07-26T12:42:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4480.138334 secs 2024-07-26T12:42:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:20 smithi134 ceph-mon[31966]: pgmap v3193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:20 smithi189 ceph-mon[36790]: pgmap v3193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:20 smithi193 ceph-mon[37473]: pgmap v3193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:21.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:21.487+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:42:21.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:21.487+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:42:21.488 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:21.487+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:22.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:22.298+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 msgr2=0x7f9b300c0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T12:42:22.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:22.298+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 msgr2=0x7f9b300c0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:42:22.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:22.298+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300c0480 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800e640 tx=0x7f9b38008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:42:22.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:22.298+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300c0480 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800e640 tx=0x7f9b38008040 comp rx=0 tx=0).stop 2024-07-26T12:42:22.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:22.298+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 msgr2=0x7f9b300c0480 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T12:42:22.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:22.298+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 msgr2=0x7f9b300c0480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:42:22.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:22.298+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300b3df0 0x7f9b300c0480 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:22.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:22.298+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:22.300 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:22.298+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:22.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:22.301+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b38008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4499.563848 secs 2024-07-26T12:42:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:22 smithi134 ceph-mon[31966]: pgmap v3194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4499.563848 secs 2024-07-26T12:42:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:22 smithi189 ceph-mon[36790]: pgmap v3194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4499.563848 secs 2024-07-26T12:42:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:22 smithi193 ceph-mon[37473]: pgmap v3194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:23.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:23.109+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T12:42:23.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:23.109+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T12:42:23.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:23.109+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T12:42:23.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:23.109+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T12:42:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:24 smithi134 ceph-mon[31966]: pgmap v3195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4485.138454 secs 2024-07-26T12:42:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:24 smithi189 ceph-mon[36790]: pgmap v3195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4485.138454 secs 2024-07-26T12:42:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:24 smithi193 ceph-mon[37473]: pgmap v3195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4485.138454 secs 2024-07-26T12:42:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:26 smithi134 ceph-mon[31966]: pgmap v3196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:42:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:26 smithi189 ceph-mon[36790]: pgmap v3196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:42:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:26 smithi193 ceph-mon[37473]: pgmap v3196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:42:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4504.563988 secs 2024-07-26T12:42:27.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4504.563988 secs 2024-07-26T12:42:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4504.563988 secs 2024-07-26T12:42:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:28 smithi134 ceph-mon[31966]: pgmap v3197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:28 smithi189 ceph-mon[36790]: pgmap v3197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:28 smithi193 ceph-mon[37473]: pgmap v3197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4490.138578 secs 2024-07-26T12:42:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4490.138578 secs 2024-07-26T12:42:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4490.138578 secs 2024-07-26T12:42:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:30 smithi134 ceph-mon[31966]: pgmap v3198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:42:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:30 smithi189 ceph-mon[36790]: pgmap v3198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:42:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:30 smithi193 ceph-mon[37473]: pgmap v3198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:42:31.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:31.916+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:42:31.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:31.916+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:31.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:31.916+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:42:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4509.564064 secs 2024-07-26T12:42:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:32 smithi134 ceph-mon[31966]: pgmap v3199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:42:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4509.564064 secs 2024-07-26T12:42:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:32 smithi189 ceph-mon[36790]: pgmap v3199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:42:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4509.564064 secs 2024-07-26T12:42:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:32 smithi193 ceph-mon[37473]: pgmap v3199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:42:32.853 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:32.852+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 msgr2=0x7f21ac0c0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T12:42:32.854 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:32.852+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 msgr2=0x7f21ac0c0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:42:32.854 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:32.852+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 0x7f21ac0c0760 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2198005820 tx=0x7f21b80c3000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:42:32.854 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:32.852+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 0x7f21ac0c0760 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f2198005820 tx=0x7f21b80c3000 comp rx=0 tx=0).stop 2024-07-26T12:42:32.854 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:32.852+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 msgr2=0x7f21ac0c0760 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T12:42:32.854 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:32.852+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 msgr2=0x7f21ac0c0760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:42:32.854 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:32.852+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 0x7f21ac0c0760 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:42:32.854 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:32.852+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0ca0 0x7f21ac0c10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:42:32.855 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:32.853+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0ca0 0x7f21ac0c10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:42:32.855 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:32.854+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0ca0 0x7f21ac0c10b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f21b80c7030 tx=0x7f21b80d9000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:42:33.624 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:33.623+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:42:33.625 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:33.623+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:42:33.625 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:33.624+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:42:33.625 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:33.624+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:42:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:34 smithi134 ceph-mon[31966]: pgmap v3200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:42:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4495.138700 secs 2024-07-26T12:42:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:34 smithi189 ceph-mon[36790]: pgmap v3200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:42:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4495.138700 secs 2024-07-26T12:42:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:34 smithi193 ceph-mon[37473]: pgmap v3200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:42:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4495.138700 secs 2024-07-26T12:42:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:36 smithi134 ceph-mon[31966]: pgmap v3201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:42:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:36 smithi189 ceph-mon[36790]: pgmap v3201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:42:36.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:36.499+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:42:36.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:36.499+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:42:36.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:36.499+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:36 smithi193 ceph-mon[37473]: pgmap v3201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:42:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4514.564202 secs 2024-07-26T12:42:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4514.564202 secs 2024-07-26T12:42:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4514.564202 secs 2024-07-26T12:42:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:38 smithi134 ceph-mon[31966]: pgmap v3202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:42:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:38 smithi189 ceph-mon[36790]: pgmap v3202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:42:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:38 smithi193 ceph-mon[37473]: pgmap v3202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:42:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4500.138825 secs 2024-07-26T12:42:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4500.138825 secs 2024-07-26T12:42:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4500.138825 secs 2024-07-26T12:42:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:40 smithi189 ceph-mon[36790]: pgmap v3203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:42:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:40 smithi193 ceph-mon[37473]: pgmap v3203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:42:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:40 smithi134 ceph-mon[31966]: pgmap v3203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:42:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4519.564347 secs 2024-07-26T12:42:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:42 smithi193 ceph-mon[37473]: pgmap v3204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4519.564347 secs 2024-07-26T12:42:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:42 smithi134 ceph-mon[31966]: pgmap v3204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4519.564347 secs 2024-07-26T12:42:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:42 smithi189 ceph-mon[36790]: pgmap v3204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:44 smithi193 ceph-mon[37473]: pgmap v3205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4505.138971 secs 2024-07-26T12:42:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:44 smithi134 ceph-mon[31966]: pgmap v3205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4505.138971 secs 2024-07-26T12:42:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:44 smithi189 ceph-mon[36790]: pgmap v3205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4505.138971 secs 2024-07-26T12:42:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:46 smithi134 ceph-mon[31966]: pgmap v3206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:42:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:46 smithi189 ceph-mon[36790]: pgmap v3206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:42:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:46 smithi193 ceph-mon[37473]: pgmap v3206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:42:46.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:46.931+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:42:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:46.931+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:42:46.931+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:42:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4524.564428 secs 2024-07-26T12:42:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4524.564428 secs 2024-07-26T12:42:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4524.564428 secs 2024-07-26T12:42:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:48 smithi134 ceph-mon[31966]: pgmap v3207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:48 smithi189 ceph-mon[36790]: pgmap v3207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:48 smithi193 ceph-mon[37473]: pgmap v3207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4510.139093 secs 2024-07-26T12:42:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4510.139093 secs 2024-07-26T12:42:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4510.139093 secs 2024-07-26T12:42:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:50 smithi134 ceph-mon[31966]: pgmap v3208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:50 smithi189 ceph-mon[36790]: pgmap v3208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:50 smithi193 ceph-mon[37473]: pgmap v3208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:42:51.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:51.514+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:42:51.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:51.514+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:42:51.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:42:51.514+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:42:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:42:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4529.564546 secs 2024-07-26T12:42:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: pgmap v3209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:42:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:42:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:42:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:42:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:52.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:52.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:42:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4529.564546 secs 2024-07-26T12:42:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: pgmap v3209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:42:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:42:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:42:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:42:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:42:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4529.564546 secs 2024-07-26T12:42:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: pgmap v3209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:42:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:42:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:42:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:42:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:42:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:42:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:42:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:42:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:42:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:53 smithi134 ceph-mon[31966]: pgmap v3210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:42:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:53 smithi134 ceph-mon[31966]: pgmap v3211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:42:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:53 smithi189 ceph-mon[36790]: pgmap v3210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:42:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:53 smithi189 ceph-mon[36790]: pgmap v3211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:42:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:53 smithi193 ceph-mon[37473]: pgmap v3210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:42:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:53 smithi193 ceph-mon[37473]: pgmap v3211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T12:42:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4515.139160 secs 2024-07-26T12:42:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4515.139160 secs 2024-07-26T12:42:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4515.139160 secs 2024-07-26T12:42:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:55 smithi134 ceph-mon[31966]: pgmap v3212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:42:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:55 smithi189 ceph-mon[36790]: pgmap v3212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:42:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:55 smithi193 ceph-mon[37473]: pgmap v3212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:42:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4534.564658 secs 2024-07-26T12:42:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:57 smithi134 ceph-mon[31966]: pgmap v3213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:42:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4534.564658 secs 2024-07-26T12:42:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:57 smithi189 ceph-mon[36790]: pgmap v3213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:42:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4534.564658 secs 2024-07-26T12:42:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:57 smithi193 ceph-mon[37473]: pgmap v3213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:42:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:59 smithi134 ceph-mon[31966]: pgmap v3214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:42:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:42:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4520.139223 secs 2024-07-26T12:42:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:59 smithi189 ceph-mon[36790]: pgmap v3214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:42:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:42:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4520.139223 secs 2024-07-26T12:42:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:59 smithi193 ceph-mon[37473]: pgmap v3214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:42:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:42:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4520.139223 secs 2024-07-26T12:43:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:01 smithi134 ceph-mon[31966]: pgmap v3215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:43:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:01 smithi189 ceph-mon[36790]: pgmap v3215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:43:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:01 smithi193 ceph-mon[37473]: pgmap v3215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:43:01.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:43:01.939+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:43:01.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:43:01.939+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:43:01.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:43:01.939+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4539.564801 secs 2024-07-26T12:43:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4539.564801 secs 2024-07-26T12:43:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4539.564801 secs 2024-07-26T12:43:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:03 smithi134 ceph-mon[31966]: pgmap v3216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:43:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:03 smithi189 ceph-mon[36790]: pgmap v3216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:43:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:03 smithi193 ceph-mon[37473]: pgmap v3216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:43:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4525.139345 secs 2024-07-26T12:43:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4525.139345 secs 2024-07-26T12:43:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4525.139345 secs 2024-07-26T12:43:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:05 smithi134 ceph-mon[31966]: pgmap v3217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:05 smithi189 ceph-mon[36790]: pgmap v3217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:05 smithi193 ceph-mon[37473]: pgmap v3217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:06.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:43:06.525+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:43:06.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:43:06.525+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:43:06.527 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:43:06.525+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4544.564871 secs 2024-07-26T12:43:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:07 smithi134 ceph-mon[31966]: pgmap v3218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4544.564871 secs 2024-07-26T12:43:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:07 smithi189 ceph-mon[36790]: pgmap v3218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4544.564871 secs 2024-07-26T12:43:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:07 smithi193 ceph-mon[37473]: pgmap v3218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:09 smithi134 ceph-mon[31966]: pgmap v3219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:43:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4530.139501 secs 2024-07-26T12:43:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:09 smithi189 ceph-mon[36790]: pgmap v3219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:43:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4530.139501 secs 2024-07-26T12:43:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:09 smithi193 ceph-mon[37473]: pgmap v3219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:43:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4530.139501 secs 2024-07-26T12:43:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:11 smithi134 ceph-mon[31966]: pgmap v3220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:11 smithi189 ceph-mon[36790]: pgmap v3220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:11 smithi193 ceph-mon[37473]: pgmap v3220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4549.564937 secs 2024-07-26T12:43:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4549.564937 secs 2024-07-26T12:43:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4549.564937 secs 2024-07-26T12:43:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:13 smithi189 ceph-mon[36790]: pgmap v3221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:13 smithi193 ceph-mon[37473]: pgmap v3221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:13 smithi134 ceph-mon[31966]: pgmap v3221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4535.139586 secs 2024-07-26T12:43:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4535.139586 secs 2024-07-26T12:43:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4535.139586 secs 2024-07-26T12:43:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:15 smithi193 ceph-mon[37473]: pgmap v3222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:15.870 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:15 smithi189 ceph-mon[36790]: pgmap v3222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:15 smithi134 ceph-mon[31966]: pgmap v3222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:16.956 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:43:16.955+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:43:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:43:16.955+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:43:16.957 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:43:16.955+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4554.565034 secs 2024-07-26T12:43:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:17 smithi193 ceph-mon[37473]: pgmap v3223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4554.565034 secs 2024-07-26T12:43:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:17 smithi134 ceph-mon[31966]: pgmap v3223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4554.565034 secs 2024-07-26T12:43:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:17 smithi189 ceph-mon[36790]: pgmap v3223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:19 smithi193 ceph-mon[37473]: pgmap v3224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:43:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4540.139704 secs 2024-07-26T12:43:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:19 smithi134 ceph-mon[31966]: pgmap v3224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:43:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4540.139704 secs 2024-07-26T12:43:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:19 smithi189 ceph-mon[36790]: pgmap v3224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:43:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4540.139704 secs 2024-07-26T12:43:21.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:43:21.540+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:43:21.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:43:21.540+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:21.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:43:21.540+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:43:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:21 smithi193 ceph-mon[37473]: pgmap v3225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:21 smithi134 ceph-mon[31966]: pgmap v3225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:21 smithi189 ceph-mon[36790]: pgmap v3225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4559.565121 secs 2024-07-26T12:43:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4559.565121 secs 2024-07-26T12:43:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4559.565121 secs 2024-07-26T12:43:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:23 smithi193 ceph-mon[37473]: pgmap v3226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:23 smithi134 ceph-mon[31966]: pgmap v3226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:23 smithi189 ceph-mon[36790]: pgmap v3226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4545.139852 secs 2024-07-26T12:43:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4545.139852 secs 2024-07-26T12:43:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4545.139852 secs 2024-07-26T12:43:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:25 smithi193 ceph-mon[37473]: pgmap v3227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:25 smithi134 ceph-mon[31966]: pgmap v3227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:25 smithi189 ceph-mon[36790]: pgmap v3227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4564.565239 secs 2024-07-26T12:43:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:27 smithi193 ceph-mon[37473]: pgmap v3228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4564.565239 secs 2024-07-26T12:43:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:27 smithi134 ceph-mon[31966]: pgmap v3228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4564.565239 secs 2024-07-26T12:43:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:27 smithi189 ceph-mon[36790]: pgmap v3228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:29 smithi193 ceph-mon[37473]: pgmap v3229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:43:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4550.139971 secs 2024-07-26T12:43:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:29 smithi134 ceph-mon[31966]: pgmap v3229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:43:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4550.139971 secs 2024-07-26T12:43:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:29 smithi189 ceph-mon[36790]: pgmap v3229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:43:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4550.139971 secs 2024-07-26T12:43:31.778 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:31 smithi193 ceph-mon[37473]: pgmap v3230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:31 smithi134 ceph-mon[31966]: pgmap v3230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:31 smithi189 ceph-mon[36790]: pgmap v3230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:43:31.970+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:43:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:43:31.970+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:43:31.972 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:43:31.970+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4569.565374 secs 2024-07-26T12:43:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4569.565374 secs 2024-07-26T12:43:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4569.565374 secs 2024-07-26T12:43:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:33 smithi193 ceph-mon[37473]: pgmap v3231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:43:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:33 smithi134 ceph-mon[31966]: pgmap v3231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:43:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:33 smithi189 ceph-mon[36790]: pgmap v3231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:43:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4555.140089 secs 2024-07-26T12:43:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4555.140089 secs 2024-07-26T12:43:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4555.140089 secs 2024-07-26T12:43:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:35 smithi134 ceph-mon[31966]: pgmap v3232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:35 smithi189 ceph-mon[36790]: pgmap v3232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:35 smithi193 ceph-mon[37473]: pgmap v3232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:36.557 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:43:36.556+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:43:36.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:43:36.556+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:43:36.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:43:36.556+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4574.565495 secs 2024-07-26T12:43:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:37 smithi134 ceph-mon[31966]: pgmap v3233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4574.565495 secs 2024-07-26T12:43:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:37 smithi189 ceph-mon[36790]: pgmap v3233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4574.565495 secs 2024-07-26T12:43:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:37 smithi193 ceph-mon[37473]: pgmap v3233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:39 smithi134 ceph-mon[31966]: pgmap v3234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:43:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4560.140158 secs 2024-07-26T12:43:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:39 smithi189 ceph-mon[36790]: pgmap v3234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:43:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4560.140158 secs 2024-07-26T12:43:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:39 smithi193 ceph-mon[37473]: pgmap v3234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:43:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4560.140158 secs 2024-07-26T12:43:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:41 smithi134 ceph-mon[31966]: pgmap v3235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:41 smithi189 ceph-mon[36790]: pgmap v3235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:41 smithi193 ceph-mon[37473]: pgmap v3235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4579.565597 secs 2024-07-26T12:43:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4579.565597 secs 2024-07-26T12:43:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4579.565597 secs 2024-07-26T12:43:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:43 smithi134 ceph-mon[31966]: pgmap v3236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:43 smithi189 ceph-mon[36790]: pgmap v3236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:43 smithi193 ceph-mon[37473]: pgmap v3236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:43:44.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4565.140260 secs 2024-07-26T12:43:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4565.140260 secs 2024-07-26T12:43:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4565.140260 secs 2024-07-26T12:43:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:45 smithi134 ceph-mon[31966]: pgmap v3237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:45 smithi189 ceph-mon[36790]: pgmap v3237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:45 smithi193 ceph-mon[37473]: pgmap v3237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:43:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4584.565721 secs 2024-07-26T12:43:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4584.565721 secs 2024-07-26T12:43:46.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:43:46.986+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:43:46.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:43:46.986+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:43:46.987 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:43:46.986+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4584.565721 secs 2024-07-26T12:43:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:47 smithi134 ceph-mon[31966]: pgmap v3238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:47 smithi189 ceph-mon[36790]: pgmap v3238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:47 smithi193 ceph-mon[37473]: pgmap v3238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:43:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:49 smithi134 ceph-mon[31966]: pgmap v3239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:43:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4570.140355 secs 2024-07-26T12:43:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:49 smithi189 ceph-mon[36790]: pgmap v3239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:43:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4570.140355 secs 2024-07-26T12:43:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:49 smithi193 ceph-mon[37473]: pgmap v3239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:43:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4570.140355 secs 2024-07-26T12:43:51.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:43:51.572+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:43:51.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:43:51.572+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:43:51.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:43:51.572+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:43:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:51 smithi134 ceph-mon[31966]: pgmap v3240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:43:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4589.565846 secs 2024-07-26T12:43:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:51 smithi189 ceph-mon[36790]: pgmap v3240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:43:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4589.565846 secs 2024-07-26T12:43:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:51 smithi193 ceph-mon[37473]: pgmap v3240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:43:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4589.565846 secs 2024-07-26T12:43:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:43:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:43:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:43:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:53 smithi134 ceph-mon[31966]: pgmap v3241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:43:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:43:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:53 smithi189 ceph-mon[36790]: pgmap v3241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:43:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:43:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:53 smithi193 ceph-mon[37473]: pgmap v3241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:43:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:43:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:43:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:43:54.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:54 smithi134 ceph-mon[31966]: pgmap v3242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T12:43:54.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:54 smithi134 ceph-mon[31966]: pgmap v3243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:54.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4575.140513 secs 2024-07-26T12:43:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:54 smithi189 ceph-mon[36790]: pgmap v3242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T12:43:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:54 smithi189 ceph-mon[36790]: pgmap v3243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4575.140513 secs 2024-07-26T12:43:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:54 smithi193 ceph-mon[37473]: pgmap v3242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T12:43:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:54 smithi193 ceph-mon[37473]: pgmap v3243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4575.140513 secs 2024-07-26T12:43:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:56 smithi134 ceph-mon[31966]: pgmap v3244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4594.565945 secs 2024-07-26T12:43:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:56 smithi189 ceph-mon[36790]: pgmap v3244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4594.565945 secs 2024-07-26T12:43:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:56 smithi193 ceph-mon[37473]: pgmap v3244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:43:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4594.565945 secs 2024-07-26T12:43:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:58 smithi134 ceph-mon[31966]: pgmap v3245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:43:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:58 smithi189 ceph-mon[36790]: pgmap v3245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:43:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:58 smithi193 ceph-mon[37473]: pgmap v3245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:43:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:43:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4580.140632 secs 2024-07-26T12:43:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:43:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4580.140632 secs 2024-07-26T12:44:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:43:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4580.140632 secs 2024-07-26T12:44:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:00 smithi134 ceph-mon[31966]: pgmap v3246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:44:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:00 smithi189 ceph-mon[36790]: pgmap v3246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:44:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:00 smithi193 ceph-mon[37473]: pgmap v3246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:44:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4599.566068 secs 2024-07-26T12:44:01.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:44:01.996+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:44:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:44:01.996+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:44:01.998 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:44:01.996+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4599.566068 secs 2024-07-26T12:44:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4599.566068 secs 2024-07-26T12:44:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:02 smithi189 ceph-mon[36790]: pgmap v3247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:44:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:02 smithi193 ceph-mon[37473]: pgmap v3247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:44:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:02 smithi134 ceph-mon[31966]: pgmap v3247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:44:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:04 smithi189 ceph-mon[36790]: pgmap v3248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:44:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4585.140774 secs 2024-07-26T12:44:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:04 smithi193 ceph-mon[37473]: pgmap v3248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:44:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4585.140774 secs 2024-07-26T12:44:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:04 smithi134 ceph-mon[31966]: pgmap v3248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:44:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4585.140774 secs 2024-07-26T12:44:06.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:44:06.587+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:44:06.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:44:06.587+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:44:06.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:44:06.587+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:06 smithi134 ceph-mon[31966]: pgmap v3249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4604.566222 secs 2024-07-26T12:44:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:06 smithi189 ceph-mon[36790]: pgmap v3249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4604.566222 secs 2024-07-26T12:44:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:06 smithi193 ceph-mon[37473]: pgmap v3249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4604.566222 secs 2024-07-26T12:44:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:08 smithi193 ceph-mon[37473]: pgmap v3250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:44:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:08 smithi134 ceph-mon[31966]: pgmap v3250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:44:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:08 smithi189 ceph-mon[36790]: pgmap v3250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:44:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4590.140889 secs 2024-07-26T12:44:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4590.140889 secs 2024-07-26T12:44:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4590.140889 secs 2024-07-26T12:44:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:10 smithi193 ceph-mon[37473]: pgmap v3251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:44:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:10 smithi134 ceph-mon[31966]: pgmap v3251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:44:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:10 smithi189 ceph-mon[36790]: pgmap v3251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:44:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4609.566339 secs 2024-07-26T12:44:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:12 smithi134 ceph-mon[31966]: pgmap v3252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:44:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4609.566339 secs 2024-07-26T12:44:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:12 smithi189 ceph-mon[36790]: pgmap v3252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:44:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4609.566339 secs 2024-07-26T12:44:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:12 smithi193 ceph-mon[37473]: pgmap v3252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:44:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:14 smithi134 ceph-mon[31966]: pgmap v3253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:44:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4595.141012 secs 2024-07-26T12:44:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:14 smithi189 ceph-mon[36790]: pgmap v3253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:44:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4595.141012 secs 2024-07-26T12:44:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:14 smithi193 ceph-mon[37473]: pgmap v3253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:44:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4595.141012 secs 2024-07-26T12:44:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:16 smithi134 ceph-mon[31966]: pgmap v3254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:44:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:16 smithi189 ceph-mon[36790]: pgmap v3254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:44:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:16 smithi193 ceph-mon[37473]: pgmap v3254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:44:17.012 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:44:17.011+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:44:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:44:17.011+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:44:17.013 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:44:17.011+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4614.566460 secs 2024-07-26T12:44:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4614.566460 secs 2024-07-26T12:44:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4614.566460 secs 2024-07-26T12:44:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:18 smithi134 ceph-mon[31966]: pgmap v3255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:44:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:18 smithi189 ceph-mon[36790]: pgmap v3255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:44:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:18 smithi193 ceph-mon[37473]: pgmap v3255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:44:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4600.141088 secs 2024-07-26T12:44:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4600.141088 secs 2024-07-26T12:44:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4600.141088 secs 2024-07-26T12:44:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:20 smithi134 ceph-mon[31966]: pgmap v3256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:20 smithi189 ceph-mon[36790]: pgmap v3256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:20 smithi193 ceph-mon[37473]: pgmap v3256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:21.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:44:21.603+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:44:21.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:44:21.603+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:21.604 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:44:21.603+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:44:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4619.566543 secs 2024-07-26T12:44:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:22 smithi134 ceph-mon[31966]: pgmap v3257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4619.566543 secs 2024-07-26T12:44:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:22 smithi189 ceph-mon[36790]: pgmap v3257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4619.566543 secs 2024-07-26T12:44:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:22 smithi193 ceph-mon[37473]: pgmap v3257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:24 smithi134 ceph-mon[31966]: pgmap v3258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4605.141155 secs 2024-07-26T12:44:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:24 smithi189 ceph-mon[36790]: pgmap v3258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4605.141155 secs 2024-07-26T12:44:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:24 smithi193 ceph-mon[37473]: pgmap v3258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4605.141155 secs 2024-07-26T12:44:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:26 smithi134 ceph-mon[31966]: pgmap v3259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:26 smithi189 ceph-mon[36790]: pgmap v3259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:26 smithi193 ceph-mon[37473]: pgmap v3259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4624.566659 secs 2024-07-26T12:44:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4624.566659 secs 2024-07-26T12:44:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4624.566659 secs 2024-07-26T12:44:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:28 smithi134 ceph-mon[31966]: pgmap v3260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:28.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:28 smithi193 ceph-mon[37473]: pgmap v3260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:28 smithi189 ceph-mon[36790]: pgmap v3260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4610.141240 secs 2024-07-26T12:44:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4610.141240 secs 2024-07-26T12:44:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4610.141240 secs 2024-07-26T12:44:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:30 smithi134 ceph-mon[31966]: pgmap v3261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:30 smithi189 ceph-mon[36790]: pgmap v3261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:30 smithi193 ceph-mon[37473]: pgmap v3261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:44:32.027+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:44:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:44:32.027+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:44:32.028 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:44:32.027+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4629.566784 secs 2024-07-26T12:44:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:32 smithi134 ceph-mon[31966]: pgmap v3262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4629.566784 secs 2024-07-26T12:44:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:32 smithi189 ceph-mon[36790]: pgmap v3262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4629.566784 secs 2024-07-26T12:44:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:32 smithi193 ceph-mon[37473]: pgmap v3262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:34 smithi134 ceph-mon[31966]: pgmap v3263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4615.141365 secs 2024-07-26T12:44:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:34 smithi189 ceph-mon[36790]: pgmap v3263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4615.141365 secs 2024-07-26T12:44:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:34 smithi193 ceph-mon[37473]: pgmap v3263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4615.141365 secs 2024-07-26T12:44:36.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:44:36.608+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:44:36.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:44:36.608+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:36.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:44:36.608+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:44:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:36 smithi134 ceph-mon[31966]: pgmap v3264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:36 smithi189 ceph-mon[36790]: pgmap v3264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:36 smithi193 ceph-mon[37473]: pgmap v3264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4634.566901 secs 2024-07-26T12:44:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4634.566901 secs 2024-07-26T12:44:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4634.566901 secs 2024-07-26T12:44:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:38 smithi189 ceph-mon[36790]: pgmap v3265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:38 smithi193 ceph-mon[37473]: pgmap v3265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:38 smithi134 ceph-mon[31966]: pgmap v3265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4620.141505 secs 2024-07-26T12:44:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4620.141505 secs 2024-07-26T12:44:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4620.141505 secs 2024-07-26T12:44:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:40 smithi193 ceph-mon[37473]: pgmap v3266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:40.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:40 smithi134 ceph-mon[31966]: pgmap v3266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:40 smithi189 ceph-mon[36790]: pgmap v3266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4639.567002 secs 2024-07-26T12:44:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:42 smithi193 ceph-mon[37473]: pgmap v3267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4639.567002 secs 2024-07-26T12:44:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:42 smithi134 ceph-mon[31966]: pgmap v3267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4639.567002 secs 2024-07-26T12:44:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:42 smithi189 ceph-mon[36790]: pgmap v3267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:44.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:44 smithi189 ceph-mon[36790]: pgmap v3268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:44.746 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4625.141620 secs 2024-07-26T12:44:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:44 smithi193 ceph-mon[37473]: pgmap v3268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4625.141620 secs 2024-07-26T12:44:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:44 smithi134 ceph-mon[31966]: pgmap v3268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4625.141620 secs 2024-07-26T12:44:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:46 smithi193 ceph-mon[37473]: pgmap v3269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:46 smithi134 ceph-mon[31966]: pgmap v3269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:46 smithi189 ceph-mon[36790]: pgmap v3269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:44:47.042+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:44:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:44:47.042+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:44:47.044 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:44:47.042+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4644.567125 secs 2024-07-26T12:44:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4644.567125 secs 2024-07-26T12:44:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4644.567125 secs 2024-07-26T12:44:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:48 smithi193 ceph-mon[37473]: pgmap v3270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:48 smithi134 ceph-mon[31966]: pgmap v3270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:48 smithi189 ceph-mon[36790]: pgmap v3270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:44:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4630.141738 secs 2024-07-26T12:44:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4630.141738 secs 2024-07-26T12:44:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4630.141738 secs 2024-07-26T12:44:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:50 smithi193 ceph-mon[37473]: pgmap v3271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:50 smithi134 ceph-mon[31966]: pgmap v3271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:50 smithi189 ceph-mon[36790]: pgmap v3271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:44:51.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:44:51.623+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:44:51.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:44:51.624+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:44:51.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:44:51.624+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:44:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4649.567189 secs 2024-07-26T12:44:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:52 smithi193 ceph-mon[37473]: pgmap v3272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4649.567189 secs 2024-07-26T12:44:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:52 smithi134 ceph-mon[31966]: pgmap v3272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4649.567189 secs 2024-07-26T12:44:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:52 smithi189 ceph-mon[36790]: pgmap v3272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:44:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:54 smithi193 ceph-mon[37473]: pgmap v3273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4635.141863 secs 2024-07-26T12:44:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:44:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:44:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:44:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:54 smithi134 ceph-mon[31966]: pgmap v3273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4635.141863 secs 2024-07-26T12:44:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:44:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:44:54.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:44:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:54 smithi189 ceph-mon[36790]: pgmap v3273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:44:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4635.141863 secs 2024-07-26T12:44:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:44:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:44:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:44:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:44:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:44:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:44:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:44:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:44:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:44:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:44:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:44:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:44:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:56 smithi193 ceph-mon[37473]: pgmap v3274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:44:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:56 smithi193 ceph-mon[37473]: pgmap v3275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:44:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:56 smithi134 ceph-mon[31966]: pgmap v3274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:44:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:56 smithi134 ceph-mon[31966]: pgmap v3275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:44:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:56 smithi189 ceph-mon[36790]: pgmap v3274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T12:44:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:56 smithi189 ceph-mon[36790]: pgmap v3275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:44:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4654.567346 secs 2024-07-26T12:44:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4654.567346 secs 2024-07-26T12:44:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4654.567346 secs 2024-07-26T12:44:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:58 smithi134 ceph-mon[31966]: pgmap v3276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:44:58.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:58 smithi193 ceph-mon[37473]: pgmap v3276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:44:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:58 smithi189 ceph-mon[36790]: pgmap v3276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:44:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:44:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4640.141929 secs 2024-07-26T12:44:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:44:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4640.141929 secs 2024-07-26T12:45:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:44:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4640.141929 secs 2024-07-26T12:45:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:00 smithi134 ceph-mon[31966]: pgmap v3277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:00 smithi189 ceph-mon[36790]: pgmap v3277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:00 smithi193 ceph-mon[37473]: pgmap v3277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:45:02.056+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:45:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:45:02.056+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:45:02.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:45:02.056+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4659.567469 secs 2024-07-26T12:45:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:02 smithi134 ceph-mon[31966]: pgmap v3278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4659.567469 secs 2024-07-26T12:45:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:02 smithi189 ceph-mon[36790]: pgmap v3278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4659.567469 secs 2024-07-26T12:45:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:02 smithi193 ceph-mon[37473]: pgmap v3278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:04 smithi134 ceph-mon[31966]: pgmap v3279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4645.142049 secs 2024-07-26T12:45:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:04 smithi189 ceph-mon[36790]: pgmap v3279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4645.142049 secs 2024-07-26T12:45:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:04 smithi193 ceph-mon[37473]: pgmap v3279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:45:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4645.142049 secs 2024-07-26T12:45:06.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:45:06.630+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:45:06.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:45:06.630+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:06.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:45:06.630+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:45:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:06 smithi134 ceph-mon[31966]: pgmap v3280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:45:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:06 smithi189 ceph-mon[36790]: pgmap v3280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:45:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:06 smithi193 ceph-mon[37473]: pgmap v3280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:45:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4664.567566 secs 2024-07-26T12:45:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4664.567566 secs 2024-07-26T12:45:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4664.567566 secs 2024-07-26T12:45:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:08 smithi134 ceph-mon[31966]: pgmap v3281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:45:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:08 smithi189 ceph-mon[36790]: pgmap v3281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:45:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:08 smithi193 ceph-mon[37473]: pgmap v3281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:45:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4650.142170 secs 2024-07-26T12:45:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4650.142170 secs 2024-07-26T12:45:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4650.142170 secs 2024-07-26T12:45:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:10 smithi134 ceph-mon[31966]: pgmap v3282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:10 smithi189 ceph-mon[36790]: pgmap v3282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:10 smithi193 ceph-mon[37473]: pgmap v3282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4669.567650 secs 2024-07-26T12:45:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4669.567650 secs 2024-07-26T12:45:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4669.567650 secs 2024-07-26T12:45:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:12 smithi134 ceph-mon[31966]: pgmap v3283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:12 smithi189 ceph-mon[36790]: pgmap v3283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:12 smithi193 ceph-mon[37473]: pgmap v3283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:14 smithi134 ceph-mon[31966]: pgmap v3284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:45:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4655.142279 secs 2024-07-26T12:45:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:14 smithi189 ceph-mon[36790]: pgmap v3284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:45:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4655.142279 secs 2024-07-26T12:45:15.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:14 smithi193 ceph-mon[37473]: pgmap v3284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:45:15.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4655.142279 secs 2024-07-26T12:45:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:16 smithi134 ceph-mon[31966]: pgmap v3285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4674.567765 secs 2024-07-26T12:45:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:16 smithi189 ceph-mon[36790]: pgmap v3285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4674.567765 secs 2024-07-26T12:45:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:16 smithi193 ceph-mon[37473]: pgmap v3285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4674.567765 secs 2024-07-26T12:45:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:45:17.072+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:45:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:45:17.072+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:45:17.072+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:45:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:18 smithi134 ceph-mon[31966]: pgmap v3286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:45:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:18 smithi189 ceph-mon[36790]: pgmap v3286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:45:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:18 smithi193 ceph-mon[37473]: pgmap v3286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:45:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4660.142401 secs 2024-07-26T12:45:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4660.142401 secs 2024-07-26T12:45:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4660.142401 secs 2024-07-26T12:45:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:20 smithi134 ceph-mon[31966]: pgmap v3287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:20 smithi189 ceph-mon[36790]: pgmap v3287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:20 smithi193 ceph-mon[37473]: pgmap v3287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:21.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:45:21.633+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:45:21.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:45:21.633+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:45:21.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:45:21.633+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4679.567882 secs 2024-07-26T12:45:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4679.567882 secs 2024-07-26T12:45:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4679.567882 secs 2024-07-26T12:45:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:22 smithi134 ceph-mon[31966]: pgmap v3288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:22 smithi189 ceph-mon[36790]: pgmap v3288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:22 smithi193 ceph-mon[37473]: pgmap v3288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:23.143 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:45:23.142+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T12:45:23.143 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:45:23.142+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T12:45:23.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:45:23.142+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T12:45:23.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:45:23.142+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T12:45:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:24 smithi134 ceph-mon[31966]: pgmap v3289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4665.142520 secs 2024-07-26T12:45:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:24 smithi189 ceph-mon[36790]: pgmap v3289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4665.142520 secs 2024-07-26T12:45:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:24 smithi193 ceph-mon[37473]: pgmap v3289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4665.142520 secs 2024-07-26T12:45:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:26 smithi189 ceph-mon[36790]: pgmap v3290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4684.568004 secs 2024-07-26T12:45:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:26 smithi193 ceph-mon[37473]: pgmap v3290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4684.568004 secs 2024-07-26T12:45:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:26 smithi134 ceph-mon[31966]: pgmap v3290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4684.568004 secs 2024-07-26T12:45:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:28 smithi189 ceph-mon[36790]: pgmap v3291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:45:28.974 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:28 smithi134 ceph-mon[31966]: pgmap v3291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:45:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:28 smithi193 ceph-mon[37473]: pgmap v3291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:45:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4670.142640 secs 2024-07-26T12:45:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4670.142640 secs 2024-07-26T12:45:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4670.142640 secs 2024-07-26T12:45:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:30 smithi193 ceph-mon[37473]: pgmap v3292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:30 smithi134 ceph-mon[31966]: pgmap v3292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:30 smithi189 ceph-mon[36790]: pgmap v3292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4689.568150 secs 2024-07-26T12:45:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:45:32.087+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:45:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:45:32.087+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:45:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:45:32.087+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4689.568150 secs 2024-07-26T12:45:32.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4689.568150 secs 2024-07-26T12:45:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:32 smithi193 ceph-mon[37473]: pgmap v3293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:32 smithi134 ceph-mon[31966]: pgmap v3293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:32 smithi189 ceph-mon[36790]: pgmap v3293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:33.656 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:45:33.655+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:45:33.657 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:45:33.655+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:45:33.657 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:45:33.656+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:45:33.657 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:45:33.656+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:45:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:34 smithi193 ceph-mon[37473]: pgmap v3294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4675.142763 secs 2024-07-26T12:45:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:34 smithi134 ceph-mon[31966]: pgmap v3294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4675.142763 secs 2024-07-26T12:45:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:34 smithi189 ceph-mon[36790]: pgmap v3294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4675.142763 secs 2024-07-26T12:45:36.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:45:36.647+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:45:36.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:45:36.648+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:45:36.650 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:45:36.648+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:36 smithi134 ceph-mon[31966]: pgmap v3295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4694.568274 secs 2024-07-26T12:45:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:36 smithi193 ceph-mon[37473]: pgmap v3295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4694.568274 secs 2024-07-26T12:45:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:36 smithi189 ceph-mon[36790]: pgmap v3295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4694.568274 secs 2024-07-26T12:45:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:38 smithi193 ceph-mon[37473]: pgmap v3296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:45:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:38 smithi134 ceph-mon[31966]: pgmap v3296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:45:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:38 smithi189 ceph-mon[36790]: pgmap v3296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:45:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4680.142852 secs 2024-07-26T12:45:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4680.142852 secs 2024-07-26T12:45:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4680.142852 secs 2024-07-26T12:45:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:40 smithi193 ceph-mon[37473]: pgmap v3297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:45:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:40 smithi134 ceph-mon[31966]: pgmap v3297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:45:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:40 smithi189 ceph-mon[36790]: pgmap v3297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:45:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4699.568394 secs 2024-07-26T12:45:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4699.568394 secs 2024-07-26T12:45:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4699.568394 secs 2024-07-26T12:45:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:42 smithi193 ceph-mon[37473]: pgmap v3298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:45:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:42 smithi134 ceph-mon[31966]: pgmap v3298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:45:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:42 smithi189 ceph-mon[36790]: pgmap v3298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:45:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:44 smithi193 ceph-mon[37473]: pgmap v3299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:45:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4685.143004 secs 2024-07-26T12:45:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:44 smithi134 ceph-mon[31966]: pgmap v3299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:45:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4685.143004 secs 2024-07-26T12:45:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:44 smithi189 ceph-mon[36790]: pgmap v3299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:45:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4685.143004 secs 2024-07-26T12:45:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:46 smithi193 ceph-mon[37473]: pgmap v3300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:45:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4704.568489 secs 2024-07-26T12:45:47.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:45:47.103+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:45:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:45:47.103+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:45:47.103+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:45:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:46 smithi134 ceph-mon[31966]: pgmap v3300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:45:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4704.568489 secs 2024-07-26T12:45:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:46 smithi189 ceph-mon[36790]: pgmap v3300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:45:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4704.568489 secs 2024-07-26T12:45:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:48 smithi193 ceph-mon[37473]: pgmap v3301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:45:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:48 smithi134 ceph-mon[31966]: pgmap v3301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:45:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:48 smithi189 ceph-mon[36790]: pgmap v3301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:45:50.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4690.143119 secs 2024-07-26T12:45:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4690.143119 secs 2024-07-26T12:45:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4690.143119 secs 2024-07-26T12:45:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:50 smithi134 ceph-mon[31966]: pgmap v3302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:50 smithi189 ceph-mon[36790]: pgmap v3302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:50 smithi193 ceph-mon[37473]: pgmap v3302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:45:51.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:45:51.663+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:45:51.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:45:51.663+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:45:51.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:45:51.663+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:45:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4709.568618 secs 2024-07-26T12:45:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4709.568618 secs 2024-07-26T12:45:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4709.568618 secs 2024-07-26T12:45:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:52 smithi134 ceph-mon[31966]: pgmap v3303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:52 smithi189 ceph-mon[36790]: pgmap v3303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:52 smithi193 ceph-mon[37473]: pgmap v3303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:54 smithi134 ceph-mon[31966]: pgmap v3304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4695.143256 secs 2024-07-26T12:45:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:54 smithi189 ceph-mon[36790]: pgmap v3304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4695.143256 secs 2024-07-26T12:45:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:54 smithi193 ceph-mon[37473]: pgmap v3304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:45:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4695.143256 secs 2024-07-26T12:45:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:45:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:45:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:45:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:56 smithi193 ceph-mon[37473]: pgmap v3305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4714.568682 secs 2024-07-26T12:45:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:56 smithi134 ceph-mon[31966]: pgmap v3305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4714.568682 secs 2024-07-26T12:45:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:56 smithi189 ceph-mon[36790]: pgmap v3305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:45:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4714.568682 secs 2024-07-26T12:45:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:58 smithi193 ceph-mon[37473]: pgmap v3306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:45:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:58 smithi134 ceph-mon[31966]: pgmap v3306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:45:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:58 smithi189 ceph-mon[36790]: pgmap v3306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:45:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:45:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:45:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4700.143377 secs 2024-07-26T12:45:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:45:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4700.143377 secs 2024-07-26T12:45:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:45:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4700.143377 secs 2024-07-26T12:46:00.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:00 smithi134 ceph-mon[31966]: pgmap v3307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:00.575 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:46:00.575 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:46:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:00 smithi189 ceph-mon[36790]: pgmap v3307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:46:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:46:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:00 smithi193 ceph-mon[37473]: pgmap v3307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:46:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:46:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:46:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:46:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:46:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:01 smithi134 ceph-mon[31966]: pgmap v3308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:46:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:01 smithi134 ceph-mon[31966]: pgmap v3309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:46:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:46:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:46:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:01 smithi189 ceph-mon[36790]: pgmap v3308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:46:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:01 smithi189 ceph-mon[36790]: pgmap v3309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:46:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:46:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:46:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:01 smithi193 ceph-mon[37473]: pgmap v3308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:46:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:01 smithi193 ceph-mon[37473]: pgmap v3309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:46:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:46:02.119+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:46:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:46:02.119+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:46:02.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:46:02.119+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4719.568803 secs 2024-07-26T12:46:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4719.568803 secs 2024-07-26T12:46:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4719.568803 secs 2024-07-26T12:46:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:03 smithi134 ceph-mon[31966]: pgmap v3310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:46:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:03 smithi189 ceph-mon[36790]: pgmap v3310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:46:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:03 smithi193 ceph-mon[37473]: pgmap v3310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:46:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4705.143444 secs 2024-07-26T12:46:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4705.143444 secs 2024-07-26T12:46:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4705.143444 secs 2024-07-26T12:46:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:05 smithi134 ceph-mon[31966]: pgmap v3311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:46:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:05 smithi189 ceph-mon[36790]: pgmap v3311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:46:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:05 smithi193 ceph-mon[37473]: pgmap v3311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:46:06.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:46:06.674+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:46:06.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:46:06.674+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:46:06.675 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:46:06.674+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4724.568917 secs 2024-07-26T12:46:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:07 smithi193 ceph-mon[37473]: pgmap v3312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:46:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4724.568917 secs 2024-07-26T12:46:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:07 smithi134 ceph-mon[31966]: pgmap v3312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:46:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4724.568917 secs 2024-07-26T12:46:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:07 smithi189 ceph-mon[36790]: pgmap v3312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:46:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4710.143589 secs 2024-07-26T12:46:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:09 smithi193 ceph-mon[37473]: pgmap v3313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4710.143589 secs 2024-07-26T12:46:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:09 smithi134 ceph-mon[31966]: pgmap v3313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4710.143589 secs 2024-07-26T12:46:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:09 smithi189 ceph-mon[36790]: pgmap v3313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:46:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:11 smithi193 ceph-mon[37473]: pgmap v3314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:46:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:11 smithi134 ceph-mon[31966]: pgmap v3314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:46:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:11 smithi189 ceph-mon[36790]: pgmap v3314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:46:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4729.569017 secs 2024-07-26T12:46:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4729.569017 secs 2024-07-26T12:46:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4729.569017 secs 2024-07-26T12:46:13.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:13 smithi193 ceph-mon[37473]: pgmap v3315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:13 smithi134 ceph-mon[31966]: pgmap v3315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:13 smithi189 ceph-mon[36790]: pgmap v3315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4715.143711 secs 2024-07-26T12:46:14.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4715.143711 secs 2024-07-26T12:46:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4715.143711 secs 2024-07-26T12:46:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:15 smithi193 ceph-mon[37473]: pgmap v3316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:15 smithi134 ceph-mon[31966]: pgmap v3316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:15 smithi189 ceph-mon[36790]: pgmap v3316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:17.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:46:17.134+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:46:17.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:46:17.134+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:46:17.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:46:17.134+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4734.569155 secs 2024-07-26T12:46:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:17 smithi193 ceph-mon[37473]: pgmap v3317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4734.569155 secs 2024-07-26T12:46:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:17 smithi134 ceph-mon[31966]: pgmap v3317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4734.569155 secs 2024-07-26T12:46:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:17 smithi189 ceph-mon[36790]: pgmap v3317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4720.143857 secs 2024-07-26T12:46:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:19 smithi193 ceph-mon[37473]: pgmap v3318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:46:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4720.143857 secs 2024-07-26T12:46:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:19 smithi134 ceph-mon[31966]: pgmap v3318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:46:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4720.143857 secs 2024-07-26T12:46:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:19 smithi189 ceph-mon[36790]: pgmap v3318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:46:21.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:46:21.683+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:46:21.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:46:21.683+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:46:21.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:46:21.683+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:21 smithi193 ceph-mon[37473]: pgmap v3319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:21 smithi134 ceph-mon[31966]: pgmap v3319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:21 smithi189 ceph-mon[36790]: pgmap v3319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4739.569280 secs 2024-07-26T12:46:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4739.569280 secs 2024-07-26T12:46:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4739.569280 secs 2024-07-26T12:46:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:23 smithi134 ceph-mon[31966]: pgmap v3320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:23 smithi189 ceph-mon[36790]: pgmap v3320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:23 smithi193 ceph-mon[37473]: pgmap v3320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4725.143958 secs 2024-07-26T12:46:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4725.143958 secs 2024-07-26T12:46:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4725.143958 secs 2024-07-26T12:46:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:25 smithi134 ceph-mon[31966]: pgmap v3321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:25 smithi189 ceph-mon[36790]: pgmap v3321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:25 smithi193 ceph-mon[37473]: pgmap v3321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4744.569395 secs 2024-07-26T12:46:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:27 smithi134 ceph-mon[31966]: pgmap v3322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4744.569395 secs 2024-07-26T12:46:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:27 smithi189 ceph-mon[36790]: pgmap v3322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4744.569395 secs 2024-07-26T12:46:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:27 smithi193 ceph-mon[37473]: pgmap v3322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:29.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4730.144089 secs 2024-07-26T12:46:29.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:29 smithi193 ceph-mon[37473]: pgmap v3323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:46:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4730.144089 secs 2024-07-26T12:46:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:29 smithi134 ceph-mon[31966]: pgmap v3323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:46:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4730.144089 secs 2024-07-26T12:46:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:29 smithi189 ceph-mon[36790]: pgmap v3323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:46:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:31 smithi134 ceph-mon[31966]: pgmap v3324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:31 smithi189 ceph-mon[36790]: pgmap v3324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:31 smithi193 ceph-mon[37473]: pgmap v3324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:32.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:46:32.145+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:46:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:46:32.145+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:46:32.147 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:46:32.145+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4749.569479 secs 2024-07-26T12:46:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4749.569479 secs 2024-07-26T12:46:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4749.569479 secs 2024-07-26T12:46:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:33 smithi134 ceph-mon[31966]: pgmap v3325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:33 smithi189 ceph-mon[36790]: pgmap v3325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:33 smithi193 ceph-mon[37473]: pgmap v3325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4735.144222 secs 2024-07-26T12:46:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4735.144222 secs 2024-07-26T12:46:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4735.144222 secs 2024-07-26T12:46:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:35 smithi134 ceph-mon[31966]: pgmap v3326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:35 smithi189 ceph-mon[36790]: pgmap v3326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:35 smithi193 ceph-mon[37473]: pgmap v3326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:36.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:46:36.699+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:46:36.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:46:36.699+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:46:36.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:46:36.699+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4754.569600 secs 2024-07-26T12:46:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4754.569600 secs 2024-07-26T12:46:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4754.569600 secs 2024-07-26T12:46:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:37 smithi134 ceph-mon[31966]: pgmap v3327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:37 smithi189 ceph-mon[36790]: pgmap v3327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:37 smithi193 ceph-mon[37473]: pgmap v3327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4740.144349 secs 2024-07-26T12:46:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:39 smithi134 ceph-mon[31966]: pgmap v3328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:46:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4740.144349 secs 2024-07-26T12:46:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:39 smithi189 ceph-mon[36790]: pgmap v3328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:46:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4740.144349 secs 2024-07-26T12:46:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:39 smithi193 ceph-mon[37473]: pgmap v3328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:46:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:41 smithi134 ceph-mon[31966]: pgmap v3329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4759.569736 secs 2024-07-26T12:46:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:41 smithi189 ceph-mon[36790]: pgmap v3329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4759.569736 secs 2024-07-26T12:46:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:41 smithi193 ceph-mon[37473]: pgmap v3329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4759.569736 secs 2024-07-26T12:46:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:43 smithi134 ceph-mon[31966]: pgmap v3330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:43 smithi189 ceph-mon[36790]: pgmap v3330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:43 smithi193 ceph-mon[37473]: pgmap v3330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4745.144501 secs 2024-07-26T12:46:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4745.144501 secs 2024-07-26T12:46:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4745.144501 secs 2024-07-26T12:46:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:45 smithi134 ceph-mon[31966]: pgmap v3331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:45 smithi189 ceph-mon[36790]: pgmap v3331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:45 smithi193 ceph-mon[37473]: pgmap v3331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4764.569861 secs 2024-07-26T12:46:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4764.569861 secs 2024-07-26T12:46:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4764.569861 secs 2024-07-26T12:46:47.157 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:46:47.157+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:46:47.158 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:46:47.157+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:47.158 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:46:47.157+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:46:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:47 smithi134 ceph-mon[31966]: pgmap v3332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:47 smithi189 ceph-mon[36790]: pgmap v3332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:47 smithi193 ceph-mon[37473]: pgmap v3332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4750.144624 secs 2024-07-26T12:46:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:49 smithi134 ceph-mon[31966]: pgmap v3333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:46:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4750.144624 secs 2024-07-26T12:46:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:49 smithi189 ceph-mon[36790]: pgmap v3333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:46:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4750.144624 secs 2024-07-26T12:46:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:49 smithi193 ceph-mon[37473]: pgmap v3333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:46:51.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:46:51.712+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:46:51.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:46:51.712+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:46:51.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:46:51.712+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:46:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:51 smithi134 ceph-mon[31966]: pgmap v3334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4769.569960 secs 2024-07-26T12:46:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:51 smithi189 ceph-mon[36790]: pgmap v3334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4769.569960 secs 2024-07-26T12:46:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:51 smithi193 ceph-mon[37473]: pgmap v3334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4769.569960 secs 2024-07-26T12:46:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:53 smithi134 ceph-mon[31966]: pgmap v3335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:53.991 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:53 smithi189 ceph-mon[36790]: pgmap v3335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:53 smithi193 ceph-mon[37473]: pgmap v3335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T12:46:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4755.144746 secs 2024-07-26T12:46:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4755.144746 secs 2024-07-26T12:46:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4755.144746 secs 2024-07-26T12:46:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:55 smithi189 ceph-mon[36790]: pgmap v3336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:55 smithi193 ceph-mon[37473]: pgmap v3336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:55 smithi134 ceph-mon[31966]: pgmap v3336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T12:46:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4774.570058 secs 2024-07-26T12:46:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4774.570058 secs 2024-07-26T12:46:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4774.570058 secs 2024-07-26T12:46:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:57 smithi193 ceph-mon[37473]: pgmap v3337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:57 smithi134 ceph-mon[31966]: pgmap v3337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:57 smithi189 ceph-mon[36790]: pgmap v3337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:46:59.974 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4760.144883 secs 2024-07-26T12:46:59.975 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:46:59 smithi134 ceph-mon[31966]: pgmap v3338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4760.144883 secs 2024-07-26T12:47:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:46:59 smithi193 ceph-mon[37473]: pgmap v3338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4760.144883 secs 2024-07-26T12:47:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:46:59 smithi189 ceph-mon[36790]: pgmap v3338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:47:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:47:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:47:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:01 smithi193 ceph-mon[37473]: pgmap v3339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4779.570144 secs 2024-07-26T12:47:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:47:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:47:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:47:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:01 smithi134 ceph-mon[31966]: pgmap v3339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4779.570144 secs 2024-07-26T12:47:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:47:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:47:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:47:02.173 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:47:02.171+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:47:02.173 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:47:02.171+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:02.173 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:47:02.171+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:47:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:01 smithi189 ceph-mon[36790]: pgmap v3339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4779.570144 secs 2024-07-26T12:47:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:47:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:47:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:47:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:02 smithi193 ceph-mon[37473]: pgmap v3340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:47:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:02 smithi193 ceph-mon[37473]: pgmap v3341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:47:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:02 smithi134 ceph-mon[31966]: pgmap v3340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:47:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:02 smithi134 ceph-mon[31966]: pgmap v3341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:47:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:02 smithi189 ceph-mon[36790]: pgmap v3340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:47:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:02 smithi189 ceph-mon[36790]: pgmap v3341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:47:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4765.145018 secs 2024-07-26T12:47:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:04 smithi193 ceph-mon[37473]: pgmap v3342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4765.145018 secs 2024-07-26T12:47:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:04 smithi134 ceph-mon[31966]: pgmap v3342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4765.145018 secs 2024-07-26T12:47:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:04 smithi189 ceph-mon[36790]: pgmap v3342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:06.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:47:06.713+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:47:06.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:47:06.713+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:06.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:47:06.713+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:47:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:06 smithi193 ceph-mon[37473]: pgmap v3343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4784.570263 secs 2024-07-26T12:47:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:06 smithi134 ceph-mon[31966]: pgmap v3343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4784.570263 secs 2024-07-26T12:47:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:06 smithi189 ceph-mon[36790]: pgmap v3343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4784.570263 secs 2024-07-26T12:47:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:08 smithi193 ceph-mon[37473]: pgmap v3344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:08 smithi134 ceph-mon[31966]: pgmap v3344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:08 smithi189 ceph-mon[36790]: pgmap v3344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:47:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4770.145137 secs 2024-07-26T12:47:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4770.145137 secs 2024-07-26T12:47:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4770.145137 secs 2024-07-26T12:47:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:10 smithi193 ceph-mon[37473]: pgmap v3345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:47:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:10 smithi134 ceph-mon[31966]: pgmap v3345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:47:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:10 smithi189 ceph-mon[36790]: pgmap v3345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:47:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4789.570377 secs 2024-07-26T12:47:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4789.570377 secs 2024-07-26T12:47:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4789.570377 secs 2024-07-26T12:47:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:12 smithi134 ceph-mon[31966]: pgmap v3346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:47:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:12 smithi189 ceph-mon[36790]: pgmap v3346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:47:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:12 smithi193 ceph-mon[37473]: pgmap v3346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:47:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4775.145262 secs 2024-07-26T12:47:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:14 smithi134 ceph-mon[31966]: pgmap v3347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4775.145262 secs 2024-07-26T12:47:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:14 smithi189 ceph-mon[36790]: pgmap v3347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4775.145262 secs 2024-07-26T12:47:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:14 smithi193 ceph-mon[37473]: pgmap v3347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:16 smithi134 ceph-mon[31966]: pgmap v3348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4794.570500 secs 2024-07-26T12:47:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:47:17.187+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:47:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:47:17.187+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:17.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:47:17.187+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:47:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:16 smithi189 ceph-mon[36790]: pgmap v3348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4794.570500 secs 2024-07-26T12:47:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:16 smithi193 ceph-mon[37473]: pgmap v3348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4794.570500 secs 2024-07-26T12:47:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:18 smithi134 ceph-mon[31966]: pgmap v3349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:18 smithi189 ceph-mon[36790]: pgmap v3349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:18 smithi193 ceph-mon[37473]: pgmap v3349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4780.145380 secs 2024-07-26T12:47:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4780.145380 secs 2024-07-26T12:47:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4780.145380 secs 2024-07-26T12:47:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:20 smithi134 ceph-mon[31966]: pgmap v3350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:47:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:20 smithi189 ceph-mon[36790]: pgmap v3350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:47:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:20 smithi193 ceph-mon[37473]: pgmap v3350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:47:21.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:47:21.728+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:47:21.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:47:21.728+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:21.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:47:21.728+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:47:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4799.570608 secs 2024-07-26T12:47:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4799.570608 secs 2024-07-26T12:47:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4799.570608 secs 2024-07-26T12:47:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:22 smithi134 ceph-mon[31966]: pgmap v3351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:22 smithi189 ceph-mon[36790]: pgmap v3351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:22 smithi193 ceph-mon[37473]: pgmap v3351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4785.145499 secs 2024-07-26T12:47:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:24 smithi134 ceph-mon[31966]: pgmap v3352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4785.145499 secs 2024-07-26T12:47:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:24 smithi189 ceph-mon[36790]: pgmap v3352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4785.145499 secs 2024-07-26T12:47:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:24 smithi193 ceph-mon[37473]: pgmap v3352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:26 smithi134 ceph-mon[31966]: pgmap v3353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4804.570690 secs 2024-07-26T12:47:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:26 smithi189 ceph-mon[36790]: pgmap v3353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4804.570690 secs 2024-07-26T12:47:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:26 smithi193 ceph-mon[37473]: pgmap v3353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4804.570690 secs 2024-07-26T12:47:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:28 smithi134 ceph-mon[31966]: pgmap v3354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:28 smithi189 ceph-mon[36790]: pgmap v3354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:28 smithi193 ceph-mon[37473]: pgmap v3354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4790.145617 secs 2024-07-26T12:47:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4790.145617 secs 2024-07-26T12:47:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4790.145617 secs 2024-07-26T12:47:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:30 smithi134 ceph-mon[31966]: pgmap v3355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:47:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:30 smithi189 ceph-mon[36790]: pgmap v3355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:47:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:30 smithi193 ceph-mon[37473]: pgmap v3355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:47:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4809.570840 secs 2024-07-26T12:47:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4809.570840 secs 2024-07-26T12:47:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:47:32.199+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:47:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:47:32.199+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:47:32.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:47:32.199+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4809.570840 secs 2024-07-26T12:47:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:32 smithi134 ceph-mon[31966]: pgmap v3356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:32 smithi189 ceph-mon[36790]: pgmap v3356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:32 smithi193 ceph-mon[37473]: pgmap v3356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4795.145758 secs 2024-07-26T12:47:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:34 smithi134 ceph-mon[31966]: pgmap v3357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4795.145758 secs 2024-07-26T12:47:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:34 smithi189 ceph-mon[36790]: pgmap v3357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4795.145758 secs 2024-07-26T12:47:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:34 smithi193 ceph-mon[37473]: pgmap v3357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:36.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:47:36.743+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:47:36.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:47:36.744+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:47:36.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:47:36.744+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:36 smithi134 ceph-mon[31966]: pgmap v3358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4814.570941 secs 2024-07-26T12:47:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:36 smithi189 ceph-mon[36790]: pgmap v3358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4814.570941 secs 2024-07-26T12:47:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:36 smithi193 ceph-mon[37473]: pgmap v3358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4814.570941 secs 2024-07-26T12:47:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:38 smithi134 ceph-mon[31966]: pgmap v3359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:38 smithi189 ceph-mon[36790]: pgmap v3359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:38 smithi193 ceph-mon[37473]: pgmap v3359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4800.145836 secs 2024-07-26T12:47:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4800.145836 secs 2024-07-26T12:47:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4800.145836 secs 2024-07-26T12:47:41.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:40 smithi134 ceph-mon[31966]: pgmap v3360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:47:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:40 smithi189 ceph-mon[36790]: pgmap v3360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:47:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:40 smithi193 ceph-mon[37473]: pgmap v3360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:47:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4819.571038 secs 2024-07-26T12:47:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:41 smithi134 ceph-mon[31966]: pgmap v3361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4819.571038 secs 2024-07-26T12:47:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:41 smithi189 ceph-mon[36790]: pgmap v3361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4819.571038 secs 2024-07-26T12:47:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:41 smithi193 ceph-mon[37473]: pgmap v3361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4805.145990 secs 2024-07-26T12:47:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:44 smithi193 ceph-mon[37473]: pgmap v3362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4805.145990 secs 2024-07-26T12:47:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:44 smithi134 ceph-mon[31966]: pgmap v3362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4805.145990 secs 2024-07-26T12:47:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:44 smithi189 ceph-mon[36790]: pgmap v3362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:46 smithi193 ceph-mon[37473]: pgmap v3363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4824.571185 secs 2024-07-26T12:47:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:46 smithi134 ceph-mon[31966]: pgmap v3363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4824.571185 secs 2024-07-26T12:47:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:46 smithi189 ceph-mon[36790]: pgmap v3363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4824.571185 secs 2024-07-26T12:47:47.210 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:47:47.208+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:47:47.210 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:47:47.208+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:47.210 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:47:47.208+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:47:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:48 smithi193 ceph-mon[37473]: pgmap v3364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:48 smithi134 ceph-mon[31966]: pgmap v3364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:48 smithi189 ceph-mon[36790]: pgmap v3364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4810.146090 secs 2024-07-26T12:47:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4810.146090 secs 2024-07-26T12:47:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4810.146090 secs 2024-07-26T12:47:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:50 smithi193 ceph-mon[37473]: pgmap v3365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:47:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:50 smithi134 ceph-mon[31966]: pgmap v3365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:47:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:50 smithi189 ceph-mon[36790]: pgmap v3365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:47:51.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:47:51.759+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:47:51.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:47:51.759+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:47:51.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:47:51.759+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:47:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4829.571255 secs 2024-07-26T12:47:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4829.571255 secs 2024-07-26T12:47:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4829.571255 secs 2024-07-26T12:47:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:52 smithi193 ceph-mon[37473]: pgmap v3366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:52 smithi134 ceph-mon[31966]: pgmap v3366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:52 smithi189 ceph-mon[36790]: pgmap v3366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4815.146215 secs 2024-07-26T12:47:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:54 smithi193 ceph-mon[37473]: pgmap v3367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4815.146215 secs 2024-07-26T12:47:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:54 smithi134 ceph-mon[31966]: pgmap v3367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4815.146215 secs 2024-07-26T12:47:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:54 smithi189 ceph-mon[36790]: pgmap v3367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:56 smithi193 ceph-mon[37473]: pgmap v3368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4834.571405 secs 2024-07-26T12:47:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:56 smithi134 ceph-mon[31966]: pgmap v3368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4834.571405 secs 2024-07-26T12:47:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:56 smithi189 ceph-mon[36790]: pgmap v3368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:47:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4834.571405 secs 2024-07-26T12:47:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:58 smithi193 ceph-mon[37473]: pgmap v3369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:58 smithi134 ceph-mon[31966]: pgmap v3369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:47:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:58 smithi189 ceph-mon[36790]: pgmap v3369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:00.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:47:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4820.146333 secs 2024-07-26T12:48:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:47:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4820.146333 secs 2024-07-26T12:48:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:47:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4820.146333 secs 2024-07-26T12:48:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:00 smithi134 ceph-mon[31966]: pgmap v3370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:00 smithi189 ceph-mon[36790]: pgmap v3370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:00 smithi193 ceph-mon[37473]: pgmap v3370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4839.571530 secs 2024-07-26T12:48:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4839.571530 secs 2024-07-26T12:48:02.224 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:48:02.223+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:48:02.224 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:48:02.223+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:48:02.225 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:48:02.223+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4839.571530 secs 2024-07-26T12:48:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:02 smithi134 ceph-mon[31966]: pgmap v3371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:48:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:48:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:48:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:02 smithi189 ceph-mon[36790]: pgmap v3371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:48:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:48:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:48:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:02 smithi193 ceph-mon[37473]: pgmap v3371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:48:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:48:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:48:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:03 smithi134 ceph-mon[31966]: pgmap v3372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T12:48:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:03 smithi134 ceph-mon[31966]: pgmap v3373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:48:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:48:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:03 smithi189 ceph-mon[36790]: pgmap v3372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T12:48:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:03 smithi189 ceph-mon[36790]: pgmap v3373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:48:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:48:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:03 smithi193 ceph-mon[37473]: pgmap v3372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T12:48:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:03 smithi193 ceph-mon[37473]: pgmap v3373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:48:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:48:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4825.146461 secs 2024-07-26T12:48:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4825.146461 secs 2024-07-26T12:48:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4825.146461 secs 2024-07-26T12:48:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:05 smithi134 ceph-mon[31966]: pgmap v3374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:48:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:05 smithi189 ceph-mon[36790]: pgmap v3374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:48:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:05 smithi193 ceph-mon[37473]: pgmap v3374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:48:06.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:48:06.768+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:48:06.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:48:06.768+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:06.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:48:06.768+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:48:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4844.571618 secs 2024-07-26T12:48:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4844.571618 secs 2024-07-26T12:48:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4844.571618 secs 2024-07-26T12:48:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:07 smithi134 ceph-mon[31966]: pgmap v3375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:48:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:07 smithi189 ceph-mon[36790]: pgmap v3375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:48:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:07 smithi193 ceph-mon[37473]: pgmap v3375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:48:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4830.146544 secs 2024-07-26T12:48:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:09 smithi134 ceph-mon[31966]: pgmap v3376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:48:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4830.146544 secs 2024-07-26T12:48:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:09 smithi189 ceph-mon[36790]: pgmap v3376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:48:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4830.146544 secs 2024-07-26T12:48:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:09 smithi193 ceph-mon[37473]: pgmap v3376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:48:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:11 smithi134 ceph-mon[31966]: pgmap v3377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:48:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4849.571719 secs 2024-07-26T12:48:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:11 smithi189 ceph-mon[36790]: pgmap v3377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:48:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4849.571719 secs 2024-07-26T12:48:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:11 smithi193 ceph-mon[37473]: pgmap v3377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:48:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4849.571719 secs 2024-07-26T12:48:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:13 smithi134 ceph-mon[31966]: pgmap v3378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:48:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:13 smithi189 ceph-mon[36790]: pgmap v3378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:48:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:13 smithi193 ceph-mon[37473]: pgmap v3378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:48:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4835.146653 secs 2024-07-26T12:48:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4835.146653 secs 2024-07-26T12:48:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4835.146653 secs 2024-07-26T12:48:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:15 smithi134 ceph-mon[31966]: pgmap v3379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:15 smithi189 ceph-mon[36790]: pgmap v3379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:15 smithi193 ceph-mon[37473]: pgmap v3379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4854.571849 secs 2024-07-26T12:48:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4854.571849 secs 2024-07-26T12:48:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:48:17.232+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:48:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:48:17.232+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:48:17.232+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:48:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4854.571849 secs 2024-07-26T12:48:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:17 smithi134 ceph-mon[31966]: pgmap v3380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:17 smithi189 ceph-mon[36790]: pgmap v3380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:17 smithi193 ceph-mon[37473]: pgmap v3380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4840.146800 secs 2024-07-26T12:48:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:19 smithi134 ceph-mon[31966]: pgmap v3381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4840.146800 secs 2024-07-26T12:48:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:19 smithi189 ceph-mon[36790]: pgmap v3381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4840.146800 secs 2024-07-26T12:48:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:19 smithi193 ceph-mon[37473]: pgmap v3381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:21.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:48:21.770+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:48:21.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:48:21.770+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:21.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:48:21.770+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:48:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:21 smithi134 ceph-mon[31966]: pgmap v3382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4859.571952 secs 2024-07-26T12:48:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:21 smithi189 ceph-mon[36790]: pgmap v3382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4859.571952 secs 2024-07-26T12:48:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:21 smithi193 ceph-mon[37473]: pgmap v3382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4859.571952 secs 2024-07-26T12:48:23.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:48:23.172+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T12:48:23.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:48:23.172+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T12:48:23.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:48:23.173+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T12:48:23.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:48:23.173+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T12:48:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:23 smithi134 ceph-mon[31966]: pgmap v3383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:23 smithi189 ceph-mon[36790]: pgmap v3383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:23 smithi193 ceph-mon[37473]: pgmap v3383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4845.146921 secs 2024-07-26T12:48:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4845.146921 secs 2024-07-26T12:48:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4845.146921 secs 2024-07-26T12:48:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:25 smithi134 ceph-mon[31966]: pgmap v3384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:25 smithi189 ceph-mon[36790]: pgmap v3384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:25 smithi193 ceph-mon[37473]: pgmap v3384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4864.572094 secs 2024-07-26T12:48:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:26 smithi189 ceph-mon[36790]: pgmap v3385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4864.572094 secs 2024-07-26T12:48:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:26 smithi193 ceph-mon[37473]: pgmap v3385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4864.572094 secs 2024-07-26T12:48:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:26 smithi134 ceph-mon[31966]: pgmap v3385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:30.125 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4850.147035 secs 2024-07-26T12:48:30.126 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:29 smithi134 ceph-mon[31966]: pgmap v3386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4850.147035 secs 2024-07-26T12:48:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:29 smithi189 ceph-mon[36790]: pgmap v3386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4850.147035 secs 2024-07-26T12:48:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:29 smithi193 ceph-mon[37473]: pgmap v3386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:31 smithi134 ceph-mon[31966]: pgmap v3387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4869.572255 secs 2024-07-26T12:48:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:31 smithi189 ceph-mon[36790]: pgmap v3387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4869.572255 secs 2024-07-26T12:48:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:48:32.247+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:48:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:48:32.247+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:48:32.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:48:32.247+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:31 smithi193 ceph-mon[37473]: pgmap v3387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4869.572255 secs 2024-07-26T12:48:33.687 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:48:33.686+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:48:33.688 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:48:33.686+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:48:33.688 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:48:33.687+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:48:33.689 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:48:33.687+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:48:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:33 smithi134 ceph-mon[31966]: pgmap v3388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:33 smithi189 ceph-mon[36790]: pgmap v3388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:33 smithi193 ceph-mon[37473]: pgmap v3388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4855.147116 secs 2024-07-26T12:48:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4855.147116 secs 2024-07-26T12:48:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4855.147116 secs 2024-07-26T12:48:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:35 smithi134 ceph-mon[31966]: pgmap v3389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:35 smithi189 ceph-mon[36790]: pgmap v3389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:35 smithi193 ceph-mon[37473]: pgmap v3389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:36.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:48:36.773+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:48:36.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:48:36.773+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:36.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:48:36.773+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:48:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4874.572324 secs 2024-07-26T12:48:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4874.572324 secs 2024-07-26T12:48:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4874.572324 secs 2024-07-26T12:48:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:37 smithi134 ceph-mon[31966]: pgmap v3390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:37 smithi189 ceph-mon[36790]: pgmap v3390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:37 smithi193 ceph-mon[37473]: pgmap v3390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:38 smithi189 ceph-mon[36790]: pgmap v3391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:38 smithi193 ceph-mon[37473]: pgmap v3391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:38 smithi134 ceph-mon[31966]: pgmap v3391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4860.147236 secs 2024-07-26T12:48:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4860.147236 secs 2024-07-26T12:48:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4860.147236 secs 2024-07-26T12:48:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:40 smithi189 ceph-mon[36790]: pgmap v3392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:40 smithi193 ceph-mon[37473]: pgmap v3392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:40 smithi134 ceph-mon[31966]: pgmap v3392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4879.572353 secs 2024-07-26T12:48:42.218 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4879.572353 secs 2024-07-26T12:48:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4879.572353 secs 2024-07-26T12:48:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:42 smithi134 ceph-mon[31966]: pgmap v3393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:42 smithi193 ceph-mon[37473]: pgmap v3393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:42 smithi189 ceph-mon[36790]: pgmap v3393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4865.147359 secs 2024-07-26T12:48:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4865.147359 secs 2024-07-26T12:48:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4865.147359 secs 2024-07-26T12:48:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:45 smithi134 ceph-mon[31966]: pgmap v3394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:45 smithi189 ceph-mon[36790]: pgmap v3394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:45 smithi193 ceph-mon[37473]: pgmap v3394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4884.572471 secs 2024-07-26T12:48:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4884.572471 secs 2024-07-26T12:48:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4884.572471 secs 2024-07-26T12:48:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:48:47.257+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:48:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:48:47.257+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:48:47.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:48:47.257+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:47 smithi189 ceph-mon[36790]: pgmap v3395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:47 smithi193 ceph-mon[37473]: pgmap v3395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:47 smithi134 ceph-mon[31966]: pgmap v3395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4870.147480 secs 2024-07-26T12:48:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:49 smithi134 ceph-mon[31966]: pgmap v3396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4870.147480 secs 2024-07-26T12:48:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:49 smithi189 ceph-mon[36790]: pgmap v3396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4870.147480 secs 2024-07-26T12:48:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:49 smithi193 ceph-mon[37473]: pgmap v3396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:51.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:48:51.785+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:48:51.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:48:51.785+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:48:51.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:48:51.785+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:48:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:51 smithi134 ceph-mon[31966]: pgmap v3397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4889.572587 secs 2024-07-26T12:48:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:51 smithi189 ceph-mon[36790]: pgmap v3397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4889.572587 secs 2024-07-26T12:48:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:51 smithi193 ceph-mon[37473]: pgmap v3397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4889.572587 secs 2024-07-26T12:48:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:53 smithi134 ceph-mon[31966]: pgmap v3398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:53 smithi189 ceph-mon[36790]: pgmap v3398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:53 smithi193 ceph-mon[37473]: pgmap v3398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4875.147604 secs 2024-07-26T12:48:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4875.147604 secs 2024-07-26T12:48:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4875.147604 secs 2024-07-26T12:48:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:55 smithi134 ceph-mon[31966]: pgmap v3399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:55 smithi189 ceph-mon[36790]: pgmap v3399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:55 smithi193 ceph-mon[37473]: pgmap v3399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:48:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4894.572653 secs 2024-07-26T12:48:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4894.572653 secs 2024-07-26T12:48:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4894.572653 secs 2024-07-26T12:48:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:57 smithi134 ceph-mon[31966]: pgmap v3400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:57 smithi189 ceph-mon[36790]: pgmap v3400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:57 smithi193 ceph-mon[37473]: pgmap v3400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:48:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:58 smithi189 ceph-mon[36790]: pgmap v3401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:58 smithi193 ceph-mon[37473]: pgmap v3401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:48:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:58 smithi134 ceph-mon[31966]: pgmap v3401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:48:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4880.147723 secs 2024-07-26T12:49:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:48:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4880.147723 secs 2024-07-26T12:49:00.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:48:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4880.147723 secs 2024-07-26T12:49:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:00 smithi193 ceph-mon[37473]: pgmap v3402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:00 smithi134 ceph-mon[31966]: pgmap v3402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:00 smithi189 ceph-mon[36790]: pgmap v3402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4899.572773 secs 2024-07-26T12:49:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:49:02.273+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:49:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:49:02.273+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:49:02.275 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:49:02.273+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4899.572773 secs 2024-07-26T12:49:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4899.572773 secs 2024-07-26T12:49:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:02 smithi193 ceph-mon[37473]: pgmap v3403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:02 smithi134 ceph-mon[31966]: pgmap v3403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:02 smithi189 ceph-mon[36790]: pgmap v3403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:49:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:49:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:49:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4885.147792 secs 2024-07-26T12:49:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:49:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:49:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:04 smithi193 ceph-mon[37473]: pgmap v3404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:49:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:04 smithi193 ceph-mon[37473]: pgmap v3405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:49:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:49:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4885.147792 secs 2024-07-26T12:49:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:49:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:49:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:04 smithi134 ceph-mon[31966]: pgmap v3404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:49:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:04 smithi134 ceph-mon[31966]: pgmap v3405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:49:05.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:49:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4885.147792 secs 2024-07-26T12:49:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:49:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:49:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:04 smithi189 ceph-mon[36790]: pgmap v3404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:49:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:04 smithi189 ceph-mon[36790]: pgmap v3405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:49:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:49:06.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:49:06.801+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:49:06.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:49:06.801+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:49:06.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:49:06.801+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4904.572887 secs 2024-07-26T12:49:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:07 smithi134 ceph-mon[31966]: pgmap v3406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:49:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4904.572887 secs 2024-07-26T12:49:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:07 smithi189 ceph-mon[36790]: pgmap v3406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:49:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4904.572887 secs 2024-07-26T12:49:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:07 smithi193 ceph-mon[37473]: pgmap v3406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:49:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:09 smithi134 ceph-mon[31966]: pgmap v3407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:49:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:09 smithi189 ceph-mon[36790]: pgmap v3407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:49:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:09 smithi193 ceph-mon[37473]: pgmap v3407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:49:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4890.147934 secs 2024-07-26T12:49:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4890.147934 secs 2024-07-26T12:49:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4890.147934 secs 2024-07-26T12:49:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:11 smithi134 ceph-mon[31966]: pgmap v3408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:49:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:11 smithi189 ceph-mon[36790]: pgmap v3408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:49:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:11 smithi193 ceph-mon[37473]: pgmap v3408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:49:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4909.573002 secs 2024-07-26T12:49:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4909.573002 secs 2024-07-26T12:49:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4909.573002 secs 2024-07-26T12:49:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:13 smithi134 ceph-mon[31966]: pgmap v3409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:49:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:13 smithi189 ceph-mon[36790]: pgmap v3409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:49:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:13 smithi193 ceph-mon[37473]: pgmap v3409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:49:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4895.148072 secs 2024-07-26T12:49:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4895.148072 secs 2024-07-26T12:49:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4895.148072 secs 2024-07-26T12:49:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:15 smithi134 ceph-mon[31966]: pgmap v3410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:49:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:15 smithi189 ceph-mon[36790]: pgmap v3410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:49:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:15 smithi193 ceph-mon[37473]: pgmap v3410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:49:17.290 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:49:17.288+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:49:17.290 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:49:17.288+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:49:17.290 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:49:17.288+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4914.573121 secs 2024-07-26T12:49:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:17 smithi134 ceph-mon[31966]: pgmap v3411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4914.573121 secs 2024-07-26T12:49:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:17 smithi189 ceph-mon[36790]: pgmap v3411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4914.573121 secs 2024-07-26T12:49:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:17 smithi193 ceph-mon[37473]: pgmap v3411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:19 smithi134 ceph-mon[31966]: pgmap v3412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4900.148222 secs 2024-07-26T12:49:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:19 smithi189 ceph-mon[36790]: pgmap v3412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4900.148222 secs 2024-07-26T12:49:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:19 smithi193 ceph-mon[37473]: pgmap v3412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4900.148222 secs 2024-07-26T12:49:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:21 smithi134 ceph-mon[31966]: pgmap v3413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:21 smithi189 ceph-mon[36790]: pgmap v3413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:21 smithi193 ceph-mon[37473]: pgmap v3413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:21.818 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:49:21.816+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:49:21.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:49:21.816+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:49:21.819 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:49:21.816+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4919.573241 secs 2024-07-26T12:49:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4919.573241 secs 2024-07-26T12:49:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4919.573241 secs 2024-07-26T12:49:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:23 smithi134 ceph-mon[31966]: pgmap v3414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:23 smithi189 ceph-mon[36790]: pgmap v3414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:23 smithi193 ceph-mon[37473]: pgmap v3414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4905.148373 secs 2024-07-26T12:49:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4905.148373 secs 2024-07-26T12:49:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4905.148373 secs 2024-07-26T12:49:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:25 smithi134 ceph-mon[31966]: pgmap v3415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:25 smithi189 ceph-mon[36790]: pgmap v3415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:25 smithi193 ceph-mon[37473]: pgmap v3415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4924.573392 secs 2024-07-26T12:49:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:27 smithi134 ceph-mon[31966]: pgmap v3416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4924.573392 secs 2024-07-26T12:49:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:27 smithi189 ceph-mon[36790]: pgmap v3416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4924.573392 secs 2024-07-26T12:49:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:27 smithi193 ceph-mon[37473]: pgmap v3416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:29 smithi189 ceph-mon[36790]: pgmap v3417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4910.148500 secs 2024-07-26T12:49:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:29 smithi193 ceph-mon[37473]: pgmap v3417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4910.148500 secs 2024-07-26T12:49:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:29 smithi134 ceph-mon[31966]: pgmap v3417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4910.148500 secs 2024-07-26T12:49:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:31 smithi189 ceph-mon[36790]: pgmap v3418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:31.474 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:31 smithi134 ceph-mon[31966]: pgmap v3418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:31 smithi193 ceph-mon[37473]: pgmap v3418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:49:32.304+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:49:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:49:32.304+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:49:32.305 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:49:32.304+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4929.573514 secs 2024-07-26T12:49:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4929.573514 secs 2024-07-26T12:49:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4929.573514 secs 2024-07-26T12:49:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:33 smithi193 ceph-mon[37473]: pgmap v3419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:33 smithi134 ceph-mon[31966]: pgmap v3419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:33 smithi189 ceph-mon[36790]: pgmap v3419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4915.148648 secs 2024-07-26T12:49:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4915.148648 secs 2024-07-26T12:49:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4915.148648 secs 2024-07-26T12:49:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:35 smithi193 ceph-mon[37473]: pgmap v3420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:35 smithi134 ceph-mon[31966]: pgmap v3420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:35 smithi189 ceph-mon[36790]: pgmap v3420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:36.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:49:36.832+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:49:36.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:49:36.832+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:49:36.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:49:36.832+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4934.573600 secs 2024-07-26T12:49:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:37 smithi193 ceph-mon[37473]: pgmap v3421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4934.573600 secs 2024-07-26T12:49:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:37 smithi134 ceph-mon[31966]: pgmap v3421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4934.573600 secs 2024-07-26T12:49:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:37 smithi189 ceph-mon[36790]: pgmap v3421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:39 smithi193 ceph-mon[37473]: pgmap v3422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4920.148786 secs 2024-07-26T12:49:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:39 smithi134 ceph-mon[31966]: pgmap v3422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4920.148786 secs 2024-07-26T12:49:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:39 smithi189 ceph-mon[36790]: pgmap v3422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4920.148786 secs 2024-07-26T12:49:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:41 smithi193 ceph-mon[37473]: pgmap v3423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:41 smithi134 ceph-mon[31966]: pgmap v3423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:41 smithi189 ceph-mon[36790]: pgmap v3423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4939.573734 secs 2024-07-26T12:49:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4939.573734 secs 2024-07-26T12:49:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4939.573734 secs 2024-07-26T12:49:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:43 smithi193 ceph-mon[37473]: pgmap v3424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:49:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:43 smithi134 ceph-mon[31966]: pgmap v3424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:49:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:43 smithi189 ceph-mon[36790]: pgmap v3424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:49:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4925.148936 secs 2024-07-26T12:49:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4925.148936 secs 2024-07-26T12:49:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4925.148936 secs 2024-07-26T12:49:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:45 smithi193 ceph-mon[37473]: pgmap v3425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:45.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:45 smithi189 ceph-mon[36790]: pgmap v3425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:45 smithi134 ceph-mon[31966]: pgmap v3425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:47.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:49:47.319+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:49:47.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:49:47.319+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:49:47.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:49:47.319+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4944.573854 secs 2024-07-26T12:49:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:47 smithi193 ceph-mon[37473]: pgmap v3426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4944.573854 secs 2024-07-26T12:49:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:47 smithi134 ceph-mon[31966]: pgmap v3426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4944.573854 secs 2024-07-26T12:49:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:47 smithi189 ceph-mon[36790]: pgmap v3426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:49 smithi193 ceph-mon[37473]: pgmap v3427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4930.149020 secs 2024-07-26T12:49:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:49 smithi134 ceph-mon[31966]: pgmap v3427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4930.149020 secs 2024-07-26T12:49:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:49 smithi189 ceph-mon[36790]: pgmap v3427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4930.149020 secs 2024-07-26T12:49:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:51 smithi193 ceph-mon[37473]: pgmap v3428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:51 smithi134 ceph-mon[31966]: pgmap v3428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:51 smithi189 ceph-mon[36790]: pgmap v3428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:51.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:49:51.839+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:49:51.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:49:51.839+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:49:51.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:49:51.839+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:49:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4949.573999 secs 2024-07-26T12:49:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4949.573999 secs 2024-07-26T12:49:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4949.573999 secs 2024-07-26T12:49:53.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:49:53.188+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T12:49:53.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:49:53.188+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T12:49:53.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:49:53.188+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T12:49:53.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:49:53.188+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T12:49:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:53 smithi193 ceph-mon[37473]: pgmap v3429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:49:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:53 smithi134 ceph-mon[31966]: pgmap v3429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:49:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:53 smithi189 ceph-mon[36790]: pgmap v3429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:49:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4935.149131 secs 2024-07-26T12:49:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4935.149131 secs 2024-07-26T12:49:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4935.149131 secs 2024-07-26T12:49:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:55 smithi193 ceph-mon[37473]: pgmap v3430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:55.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:55 smithi134 ceph-mon[31966]: pgmap v3430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:55 smithi189 ceph-mon[36790]: pgmap v3430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4954.574119 secs 2024-07-26T12:49:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:57 smithi193 ceph-mon[37473]: pgmap v3431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:57.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4954.574119 secs 2024-07-26T12:49:57.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:57 smithi134 ceph-mon[31966]: pgmap v3431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4954.574119 secs 2024-07-26T12:49:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:57 smithi189 ceph-mon[36790]: pgmap v3431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:49:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:59 smithi193 ceph-mon[37473]: pgmap v3432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:49:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4940.149244 secs 2024-07-26T12:49:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:59 smithi134 ceph-mon[31966]: pgmap v3432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:49:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4940.149244 secs 2024-07-26T12:49:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:59 smithi189 ceph-mon[36790]: pgmap v3432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:49:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:49:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4940.149244 secs 2024-07-26T12:50:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:50:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:50:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T12:50:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:01 smithi193 ceph-mon[37473]: pgmap v3433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:01.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:01 smithi134 ceph-mon[31966]: pgmap v3433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:01 smithi189 ceph-mon[36790]: pgmap v3433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:50:02.328+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:50:02.329 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:50:02.328+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:50:02.330 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:50:02.328+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4959.574243 secs 2024-07-26T12:50:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4959.574243 secs 2024-07-26T12:50:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4959.574243 secs 2024-07-26T12:50:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:50:03.703+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:50:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:50:03.703+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:50:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:50:03.703+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:50:03.704 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:50:03.703+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:50:03.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:03 smithi134 ceph-mon[31966]: pgmap v3434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:50:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:03 smithi189 ceph-mon[36790]: pgmap v3434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:50:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:03 smithi193 ceph-mon[37473]: pgmap v3434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:50:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4945.149361 secs 2024-07-26T12:50:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:50:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4945.149361 secs 2024-07-26T12:50:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:50:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4945.149361 secs 2024-07-26T12:50:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:50:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:05 smithi134 ceph-mon[31966]: pgmap v3435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:50:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:50:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:50:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:50:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:05 smithi189 ceph-mon[36790]: pgmap v3435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:50:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:50:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:50:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:50:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:05 smithi193 ceph-mon[37473]: pgmap v3435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:50:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:50:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:50:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:50:06.857 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:06 smithi134 ceph-mon[31966]: pgmap v3436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:50:06.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:50:06.855+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:50:06.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:50:06.855+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:50:06.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:50:06.855+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:06 smithi189 ceph-mon[36790]: pgmap v3436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:50:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:06 smithi193 ceph-mon[37473]: pgmap v3436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:50:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4964.574395 secs 2024-07-26T12:50:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4964.574395 secs 2024-07-26T12:50:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4964.574395 secs 2024-07-26T12:50:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:08 smithi134 ceph-mon[31966]: pgmap v3437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:50:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:08 smithi189 ceph-mon[36790]: pgmap v3437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:50:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:08 smithi193 ceph-mon[37473]: pgmap v3437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:50:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4950.149462 secs 2024-07-26T12:50:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4950.149462 secs 2024-07-26T12:50:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4950.149462 secs 2024-07-26T12:50:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:10 smithi134 ceph-mon[31966]: pgmap v3438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:50:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:10 smithi189 ceph-mon[36790]: pgmap v3438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:50:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:10 smithi193 ceph-mon[37473]: pgmap v3438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:50:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4969.574546 secs 2024-07-26T12:50:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:12 smithi134 ceph-mon[31966]: pgmap v3439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:50:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4969.574546 secs 2024-07-26T12:50:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:12 smithi189 ceph-mon[36790]: pgmap v3439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:50:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4969.574546 secs 2024-07-26T12:50:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:12 smithi193 ceph-mon[37473]: pgmap v3439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:50:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:14 smithi193 ceph-mon[37473]: pgmap v3440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T12:50:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4955.149611 secs 2024-07-26T12:50:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:14 smithi134 ceph-mon[31966]: pgmap v3440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T12:50:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4955.149611 secs 2024-07-26T12:50:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:14 smithi189 ceph-mon[36790]: pgmap v3440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T12:50:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4955.149611 secs 2024-07-26T12:50:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:16 smithi134 ceph-mon[31966]: pgmap v3441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:50:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:16 smithi189 ceph-mon[36790]: pgmap v3441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:50:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:16 smithi193 ceph-mon[37473]: pgmap v3441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:50:17.345 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:50:17.344+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:50:17.345 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:50:17.344+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:50:17.345 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:50:17.344+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4974.574699 secs 2024-07-26T12:50:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4974.574699 secs 2024-07-26T12:50:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4974.574699 secs 2024-07-26T12:50:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:18 smithi134 ceph-mon[31966]: pgmap v3442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:18 smithi189 ceph-mon[36790]: pgmap v3442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:18 smithi193 ceph-mon[37473]: pgmap v3442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4960.149729 secs 2024-07-26T12:50:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4960.149729 secs 2024-07-26T12:50:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4960.149729 secs 2024-07-26T12:50:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:20 smithi134 ceph-mon[31966]: pgmap v3443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:20 smithi189 ceph-mon[36790]: pgmap v3443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:20 smithi193 ceph-mon[37473]: pgmap v3443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:21.873 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:50:21.871+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:50:21.873 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:50:21.871+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:21.873 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:50:21.871+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:50:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4979.574785 secs 2024-07-26T12:50:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4979.574785 secs 2024-07-26T12:50:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4979.574785 secs 2024-07-26T12:50:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:22 smithi134 ceph-mon[31966]: pgmap v3444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:22 smithi189 ceph-mon[36790]: pgmap v3444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:22 smithi193 ceph-mon[37473]: pgmap v3444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:24 smithi134 ceph-mon[31966]: pgmap v3445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:50:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4965.149832 secs 2024-07-26T12:50:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:24 smithi189 ceph-mon[36790]: pgmap v3445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:50:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4965.149832 secs 2024-07-26T12:50:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:24 smithi193 ceph-mon[37473]: pgmap v3445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:50:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4965.149832 secs 2024-07-26T12:50:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:26 smithi134 ceph-mon[31966]: pgmap v3446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4984.574911 secs 2024-07-26T12:50:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:26 smithi189 ceph-mon[36790]: pgmap v3446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4984.574911 secs 2024-07-26T12:50:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:26 smithi193 ceph-mon[37473]: pgmap v3446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4984.574911 secs 2024-07-26T12:50:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:28 smithi134 ceph-mon[31966]: pgmap v3447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:28 smithi189 ceph-mon[36790]: pgmap v3447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:28 smithi193 ceph-mon[37473]: pgmap v3447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4970.149990 secs 2024-07-26T12:50:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4970.149990 secs 2024-07-26T12:50:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4970.149990 secs 2024-07-26T12:50:30.935 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:30 smithi193 ceph-mon[37473]: pgmap v3448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:30 smithi189 ceph-mon[36790]: pgmap v3448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:31.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:30 smithi134 ceph-mon[31966]: pgmap v3448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4989.575002 secs 2024-07-26T12:50:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4989.575002 secs 2024-07-26T12:50:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4989.575002 secs 2024-07-26T12:50:32.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:50:32.352+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:50:32.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:50:32.352+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:50:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:50:32.352+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:32 smithi193 ceph-mon[37473]: pgmap v3449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:32 smithi134 ceph-mon[31966]: pgmap v3449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:32 smithi189 ceph-mon[36790]: pgmap v3449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:34 smithi193 ceph-mon[37473]: pgmap v3450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:50:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4975.150110 secs 2024-07-26T12:50:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:34 smithi134 ceph-mon[31966]: pgmap v3450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:50:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4975.150110 secs 2024-07-26T12:50:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:34 smithi189 ceph-mon[36790]: pgmap v3450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:50:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4975.150110 secs 2024-07-26T12:50:36.887 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:50:36.886+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:50:36.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:50:36.886+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:50:36.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:50:36.886+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:36 smithi193 ceph-mon[37473]: pgmap v3451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4994.575061 secs 2024-07-26T12:50:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:36 smithi134 ceph-mon[31966]: pgmap v3451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4994.575061 secs 2024-07-26T12:50:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:36 smithi189 ceph-mon[36790]: pgmap v3451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4994.575061 secs 2024-07-26T12:50:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:38 smithi193 ceph-mon[37473]: pgmap v3452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:38 smithi134 ceph-mon[31966]: pgmap v3452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:38 smithi189 ceph-mon[36790]: pgmap v3452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4980.150176 secs 2024-07-26T12:50:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4980.150176 secs 2024-07-26T12:50:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4980.150176 secs 2024-07-26T12:50:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:40 smithi193 ceph-mon[37473]: pgmap v3453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:40 smithi134 ceph-mon[31966]: pgmap v3453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:40 smithi189 ceph-mon[36790]: pgmap v3453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 4999.575183 secs 2024-07-26T12:50:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 4999.575183 secs 2024-07-26T12:50:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 4999.575183 secs 2024-07-26T12:50:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:42 smithi193 ceph-mon[37473]: pgmap v3454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:42 smithi134 ceph-mon[31966]: pgmap v3454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:42 smithi189 ceph-mon[36790]: pgmap v3454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:44 smithi193 ceph-mon[37473]: pgmap v3455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:50:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4985.150298 secs 2024-07-26T12:50:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:44 smithi134 ceph-mon[31966]: pgmap v3455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:50:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4985.150298 secs 2024-07-26T12:50:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:44 smithi189 ceph-mon[36790]: pgmap v3455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:50:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4985.150298 secs 2024-07-26T12:50:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:46 smithi193 ceph-mon[37473]: pgmap v3456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5004.575322 secs 2024-07-26T12:50:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:46 smithi134 ceph-mon[31966]: pgmap v3456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5004.575322 secs 2024-07-26T12:50:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:46 smithi189 ceph-mon[36790]: pgmap v3456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5004.575322 secs 2024-07-26T12:50:47.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:50:47.367+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:50:47.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:50:47.368+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:50:47.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:50:47.368+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:48 smithi193 ceph-mon[37473]: pgmap v3457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:48 smithi134 ceph-mon[31966]: pgmap v3457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:48 smithi189 ceph-mon[36790]: pgmap v3457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4990.150376 secs 2024-07-26T12:50:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4990.150376 secs 2024-07-26T12:50:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4990.150376 secs 2024-07-26T12:50:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:50 smithi134 ceph-mon[31966]: pgmap v3458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:50 smithi189 ceph-mon[36790]: pgmap v3458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:50 smithi193 ceph-mon[37473]: pgmap v3458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:50:51.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:50:51.893+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:50:51.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:50:51.893+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:50:51.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:50:51.893+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:50:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5009.575450 secs 2024-07-26T12:50:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5009.575450 secs 2024-07-26T12:50:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5009.575450 secs 2024-07-26T12:50:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:52 smithi134 ceph-mon[31966]: pgmap v3459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:52 smithi189 ceph-mon[36790]: pgmap v3459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:52 smithi193 ceph-mon[37473]: pgmap v3459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:54 smithi134 ceph-mon[31966]: pgmap v3460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:50:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 4995.150474 secs 2024-07-26T12:50:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:54 smithi189 ceph-mon[36790]: pgmap v3460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:50:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 4995.150474 secs 2024-07-26T12:50:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:54 smithi193 ceph-mon[37473]: pgmap v3460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:50:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 4995.150474 secs 2024-07-26T12:50:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:56 smithi134 ceph-mon[31966]: pgmap v3461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5014.575582 secs 2024-07-26T12:50:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:56 smithi189 ceph-mon[36790]: pgmap v3461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5014.575582 secs 2024-07-26T12:50:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:56 smithi193 ceph-mon[37473]: pgmap v3461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:50:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5014.575582 secs 2024-07-26T12:50:59.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:58 smithi193 ceph-mon[37473]: pgmap v3462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:58 smithi134 ceph-mon[31966]: pgmap v3462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:50:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:58 smithi189 ceph-mon[36790]: pgmap v3462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:51:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:50:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5000.150619 secs 2024-07-26T12:51:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:50:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5000.150619 secs 2024-07-26T12:51:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:50:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5000.150619 secs 2024-07-26T12:51:01.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:00 smithi134 ceph-mon[31966]: pgmap v3463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:51:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:00 smithi189 ceph-mon[36790]: pgmap v3463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:51:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:00 smithi193 ceph-mon[37473]: pgmap v3463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:51:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5019.575721 secs 2024-07-26T12:51:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5019.575721 secs 2024-07-26T12:51:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5019.575721 secs 2024-07-26T12:51:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:51:02.383+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:51:02.384 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:51:02.383+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:51:02.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:51:02.383+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:02 smithi134 ceph-mon[31966]: pgmap v3464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:02 smithi189 ceph-mon[36790]: pgmap v3464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:02 smithi193 ceph-mon[37473]: pgmap v3464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:04 smithi134 ceph-mon[31966]: pgmap v3465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5005.150758 secs 2024-07-26T12:51:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:04 smithi189 ceph-mon[36790]: pgmap v3465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5005.150758 secs 2024-07-26T12:51:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:04 smithi193 ceph-mon[37473]: pgmap v3465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5005.150758 secs 2024-07-26T12:51:06.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:51:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:51:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:51:06.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:51:06.909+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:51:06.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:51:06.909+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:51:06.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:51:06.909+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:06 smithi134 ceph-mon[31966]: pgmap v3466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:07.260 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5024.575864 secs 2024-07-26T12:51:07.260 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:51:07.260 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:51:07.260 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:51:07.261 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:06 smithi189 ceph-mon[36790]: pgmap v3466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:07.261 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5024.575864 secs 2024-07-26T12:51:07.261 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:51:07.261 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:51:07.262 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:51:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:06 smithi193 ceph-mon[37473]: pgmap v3466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5024.575864 secs 2024-07-26T12:51:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:51:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:51:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:51:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:07 smithi134 ceph-mon[31966]: pgmap v3467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:51:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:07 smithi189 ceph-mon[36790]: pgmap v3467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:51:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:07 smithi193 ceph-mon[37473]: pgmap v3467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:51:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5010.150855 secs 2024-07-26T12:51:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:09 smithi134 ceph-mon[31966]: pgmap v3468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:51:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5010.150855 secs 2024-07-26T12:51:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:09 smithi189 ceph-mon[36790]: pgmap v3468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:51:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5010.150855 secs 2024-07-26T12:51:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:09 smithi193 ceph-mon[37473]: pgmap v3468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:51:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:11 smithi134 ceph-mon[31966]: pgmap v3469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:51:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5029.575988 secs 2024-07-26T12:51:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:11 smithi189 ceph-mon[36790]: pgmap v3469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:51:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5029.575988 secs 2024-07-26T12:51:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:11 smithi193 ceph-mon[37473]: pgmap v3469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:51:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5029.575988 secs 2024-07-26T12:51:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:13 smithi134 ceph-mon[31966]: pgmap v3470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T12:51:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:13 smithi189 ceph-mon[36790]: pgmap v3470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T12:51:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:13 smithi193 ceph-mon[37473]: pgmap v3470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T12:51:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5015.150978 secs 2024-07-26T12:51:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5015.150978 secs 2024-07-26T12:51:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5015.150978 secs 2024-07-26T12:51:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:15 smithi134 ceph-mon[31966]: pgmap v3471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:51:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:15 smithi189 ceph-mon[36790]: pgmap v3471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:51:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:15 smithi193 ceph-mon[37473]: pgmap v3471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:51:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5034.576109 secs 2024-07-26T12:51:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5034.576109 secs 2024-07-26T12:51:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5034.576109 secs 2024-07-26T12:51:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:51:17.398+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:51:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:51:17.399+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:51:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:51:17.399+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:17 smithi134 ceph-mon[31966]: pgmap v3472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:51:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:17 smithi189 ceph-mon[36790]: pgmap v3472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:51:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:17 smithi193 ceph-mon[37473]: pgmap v3472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:51:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5020.151051 secs 2024-07-26T12:51:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:19 smithi134 ceph-mon[31966]: pgmap v3473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5020.151051 secs 2024-07-26T12:51:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:19 smithi189 ceph-mon[36790]: pgmap v3473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5020.151051 secs 2024-07-26T12:51:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:19 smithi193 ceph-mon[37473]: pgmap v3473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:21.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:51:21.924+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:51:21.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:51:21.924+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:51:21.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:51:21.924+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:21 smithi189 ceph-mon[36790]: pgmap v3474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5039.576247 secs 2024-07-26T12:51:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:21 smithi193 ceph-mon[37473]: pgmap v3474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5039.576247 secs 2024-07-26T12:51:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:21 smithi134 ceph-mon[31966]: pgmap v3474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5039.576247 secs 2024-07-26T12:51:23.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:51:23.203+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T12:51:23.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:51:23.203+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T12:51:23.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:51:23.204+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T12:51:23.206 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:51:23.204+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T12:51:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:23 smithi189 ceph-mon[36790]: pgmap v3475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:23 smithi193 ceph-mon[37473]: pgmap v3475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:23 smithi134 ceph-mon[31966]: pgmap v3475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5025.151173 secs 2024-07-26T12:51:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5025.151173 secs 2024-07-26T12:51:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5025.151173 secs 2024-07-26T12:51:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:25 smithi193 ceph-mon[37473]: pgmap v3476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:51:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:25 smithi134 ceph-mon[31966]: pgmap v3476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:51:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:25 smithi189 ceph-mon[36790]: pgmap v3476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:51:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5044.576349 secs 2024-07-26T12:51:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:26 smithi189 ceph-mon[36790]: pgmap v3477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5044.576349 secs 2024-07-26T12:51:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:26 smithi193 ceph-mon[37473]: pgmap v3477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5044.576349 secs 2024-07-26T12:51:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:26 smithi134 ceph-mon[31966]: pgmap v3477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5030.151300 secs 2024-07-26T12:51:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:29 smithi134 ceph-mon[31966]: pgmap v3478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5030.151300 secs 2024-07-26T12:51:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:29 smithi189 ceph-mon[36790]: pgmap v3478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:29.977 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5030.151300 secs 2024-07-26T12:51:29.977 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:29 smithi193 ceph-mon[37473]: pgmap v3478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:31 smithi134 ceph-mon[31966]: pgmap v3479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:31 smithi189 ceph-mon[36790]: pgmap v3479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:31 smithi193 ceph-mon[37473]: pgmap v3479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:32.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:51:32.414+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:51:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:51:32.414+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:51:32.416 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:51:32.414+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5049.576417 secs 2024-07-26T12:51:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5049.576417 secs 2024-07-26T12:51:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5049.576417 secs 2024-07-26T12:51:33.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:51:33.718+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:51:33.719 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:51:33.718+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:51:33.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:51:33.719+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:51:33.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:51:33.719+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:51:33.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:33 smithi134 ceph-mon[31966]: pgmap v3480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:33 smithi189 ceph-mon[36790]: pgmap v3480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:33 smithi193 ceph-mon[37473]: pgmap v3480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5035.151421 secs 2024-07-26T12:51:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5035.151421 secs 2024-07-26T12:51:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5035.151421 secs 2024-07-26T12:51:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:35 smithi134 ceph-mon[31966]: pgmap v3481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:51:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:35 smithi189 ceph-mon[36790]: pgmap v3481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:51:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:35 smithi193 ceph-mon[37473]: pgmap v3481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:51:36.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:51:36.939+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:51:36.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:51:36.939+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:36.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:51:36.939+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:51:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5054.576489 secs 2024-07-26T12:51:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:37 smithi134 ceph-mon[31966]: pgmap v3482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5054.576489 secs 2024-07-26T12:51:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:37 smithi189 ceph-mon[36790]: pgmap v3482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5054.576489 secs 2024-07-26T12:51:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:37 smithi193 ceph-mon[37473]: pgmap v3482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5040.151503 secs 2024-07-26T12:51:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:39 smithi134 ceph-mon[31966]: pgmap v3483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5040.151503 secs 2024-07-26T12:51:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:39 smithi189 ceph-mon[36790]: pgmap v3483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5040.151503 secs 2024-07-26T12:51:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:39 smithi193 ceph-mon[37473]: pgmap v3483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:41 smithi134 ceph-mon[31966]: pgmap v3484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:41 smithi189 ceph-mon[36790]: pgmap v3484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:41 smithi193 ceph-mon[37473]: pgmap v3484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5059.576610 secs 2024-07-26T12:51:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5059.576610 secs 2024-07-26T12:51:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5059.576610 secs 2024-07-26T12:51:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:43 smithi134 ceph-mon[31966]: pgmap v3485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:43 smithi189 ceph-mon[36790]: pgmap v3485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:43 smithi193 ceph-mon[37473]: pgmap v3485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5045.151622 secs 2024-07-26T12:51:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5045.151622 secs 2024-07-26T12:51:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5045.151622 secs 2024-07-26T12:51:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:45 smithi134 ceph-mon[31966]: pgmap v3486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:51:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:45 smithi189 ceph-mon[36790]: pgmap v3486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:51:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:45 smithi193 ceph-mon[37473]: pgmap v3486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:51:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:51:47.426+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:51:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:51:47.426+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:51:47.427 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:51:47.426+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5064.576726 secs 2024-07-26T12:51:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:47 smithi134 ceph-mon[31966]: pgmap v3487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5064.576726 secs 2024-07-26T12:51:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:47 smithi189 ceph-mon[36790]: pgmap v3487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5064.576726 secs 2024-07-26T12:51:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:47 smithi193 ceph-mon[37473]: pgmap v3487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5050.151688 secs 2024-07-26T12:51:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:49 smithi134 ceph-mon[31966]: pgmap v3488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5050.151688 secs 2024-07-26T12:51:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:49 smithi189 ceph-mon[36790]: pgmap v3488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5050.151688 secs 2024-07-26T12:51:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:49 smithi193 ceph-mon[37473]: pgmap v3488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:51 smithi134 ceph-mon[31966]: pgmap v3489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:51 smithi189 ceph-mon[36790]: pgmap v3489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:51.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:51:51.950+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:51:51.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:51:51.950+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:51:51.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:51:51.950+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:51:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:51 smithi193 ceph-mon[37473]: pgmap v3489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5069.576866 secs 2024-07-26T12:51:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5069.576866 secs 2024-07-26T12:51:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5069.576866 secs 2024-07-26T12:51:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:53 smithi134 ceph-mon[31966]: pgmap v3490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:53 smithi189 ceph-mon[36790]: pgmap v3490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:53 smithi193 ceph-mon[37473]: pgmap v3490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:51:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5055.151828 secs 2024-07-26T12:51:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5055.151828 secs 2024-07-26T12:51:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5055.151828 secs 2024-07-26T12:51:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:55 smithi134 ceph-mon[31966]: pgmap v3491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:51:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:55 smithi189 ceph-mon[36790]: pgmap v3491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:51:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:55 smithi193 ceph-mon[37473]: pgmap v3491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:51:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5074.576979 secs 2024-07-26T12:51:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5074.576979 secs 2024-07-26T12:51:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5074.576979 secs 2024-07-26T12:51:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:57 smithi134 ceph-mon[31966]: pgmap v3492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:57 smithi189 ceph-mon[36790]: pgmap v3492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:57 smithi193 ceph-mon[37473]: pgmap v3492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:51:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5060.151979 secs 2024-07-26T12:51:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:51:59 smithi189 ceph-mon[36790]: pgmap v3493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:52:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5060.151979 secs 2024-07-26T12:52:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:51:59 smithi193 ceph-mon[37473]: pgmap v3493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:52:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5060.151979 secs 2024-07-26T12:52:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:51:59 smithi134 ceph-mon[31966]: pgmap v3493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:52:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:01 smithi189 ceph-mon[36790]: pgmap v3494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:52:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5079.577106 secs 2024-07-26T12:52:01.975 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:01 smithi134 ceph-mon[31966]: pgmap v3494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:52:01.975 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5079.577106 secs 2024-07-26T12:52:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:01 smithi193 ceph-mon[37473]: pgmap v3494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:52:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5079.577106 secs 2024-07-26T12:52:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:52:02.441+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:52:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:52:02.441+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:02.443 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:52:02.441+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:52:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:03 smithi189 ceph-mon[36790]: pgmap v3495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:52:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:03 smithi193 ceph-mon[37473]: pgmap v3495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:52:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:03 smithi134 ceph-mon[31966]: pgmap v3495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T12:52:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5065.152061 secs 2024-07-26T12:52:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5065.152061 secs 2024-07-26T12:52:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5065.152061 secs 2024-07-26T12:52:06.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:05 smithi193 ceph-mon[37473]: pgmap v3496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:52:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:05 smithi134 ceph-mon[31966]: pgmap v3496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:52:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:05 smithi189 ceph-mon[36790]: pgmap v3496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:52:06.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:52:06.965+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:52:06.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:52:06.965+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:06.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:52:06.965+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:52:06.967 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5084.577250 secs 2024-07-26T12:52:06.968 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:52:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5084.577250 secs 2024-07-26T12:52:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:52:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5084.577250 secs 2024-07-26T12:52:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:52:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:07 smithi193 ceph-mon[37473]: pgmap v3497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:52:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:52:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:52:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:52:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:07 smithi134 ceph-mon[31966]: pgmap v3497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:52:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:52:08.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:52:08.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:52:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:07 smithi189 ceph-mon[36790]: pgmap v3497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:52:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:08.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:08.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:52:08.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:52:08.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:52:08.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:52:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:08 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:52:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:08 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:52:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:08 smithi193 ceph-mon[37473]: pgmap v3498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T12:52:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:08 smithi193 ceph-mon[37473]: pgmap v3499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:52:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:08 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:52:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:08 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:52:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:08 smithi134 ceph-mon[31966]: pgmap v3498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T12:52:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:08 smithi134 ceph-mon[31966]: pgmap v3499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:52:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:08 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T12:52:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:08 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T12:52:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:08 smithi189 ceph-mon[36790]: pgmap v3498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T12:52:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:08 smithi189 ceph-mon[36790]: pgmap v3499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:52:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5070.152182 secs 2024-07-26T12:52:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5070.152182 secs 2024-07-26T12:52:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5070.152182 secs 2024-07-26T12:52:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:10 smithi193 ceph-mon[37473]: pgmap v3500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:52:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:10 smithi134 ceph-mon[31966]: pgmap v3500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:52:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:10 smithi189 ceph-mon[36790]: pgmap v3500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T12:52:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5089.577365 secs 2024-07-26T12:52:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:12 smithi134 ceph-mon[31966]: pgmap v3501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:52:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5089.577365 secs 2024-07-26T12:52:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:12 smithi189 ceph-mon[36790]: pgmap v3501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:52:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5089.577365 secs 2024-07-26T12:52:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:12 smithi193 ceph-mon[37473]: pgmap v3501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:52:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5075.152305 secs 2024-07-26T12:52:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:14 smithi193 ceph-mon[37473]: pgmap v3502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:52:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5075.152305 secs 2024-07-26T12:52:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:14 smithi134 ceph-mon[31966]: pgmap v3502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:52:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5075.152305 secs 2024-07-26T12:52:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:14 smithi189 ceph-mon[36790]: pgmap v3502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:52:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:16 smithi193 ceph-mon[37473]: pgmap v3503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:52:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5094.577507 secs 2024-07-26T12:52:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:16 smithi134 ceph-mon[31966]: pgmap v3503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:52:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5094.577507 secs 2024-07-26T12:52:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:16 smithi189 ceph-mon[36790]: pgmap v3503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:52:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5094.577507 secs 2024-07-26T12:52:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:52:17.453+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:52:17.454 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:52:17.453+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:52:17.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:52:17.453+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:18 smithi193 ceph-mon[37473]: pgmap v3504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:52:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:18 smithi134 ceph-mon[31966]: pgmap v3504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:52:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:18 smithi189 ceph-mon[36790]: pgmap v3504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:52:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5080.152379 secs 2024-07-26T12:52:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5080.152379 secs 2024-07-26T12:52:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5080.152379 secs 2024-07-26T12:52:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:20 smithi193 ceph-mon[37473]: pgmap v3505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:20 smithi134 ceph-mon[31966]: pgmap v3505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:20 smithi189 ceph-mon[36790]: pgmap v3505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:21.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:52:21.980+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:52:21.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:52:21.981+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:52:21.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:52:21.981+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5099.577590 secs 2024-07-26T12:52:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5099.577590 secs 2024-07-26T12:52:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5099.577590 secs 2024-07-26T12:52:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:22 smithi134 ceph-mon[31966]: pgmap v3506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:22 smithi189 ceph-mon[36790]: pgmap v3506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:22 smithi193 ceph-mon[37473]: pgmap v3506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5085.152458 secs 2024-07-26T12:52:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:24 smithi134 ceph-mon[31966]: pgmap v3507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5085.152458 secs 2024-07-26T12:52:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:24 smithi189 ceph-mon[36790]: pgmap v3507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5085.152458 secs 2024-07-26T12:52:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:24 smithi193 ceph-mon[37473]: pgmap v3507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:26 smithi134 ceph-mon[31966]: pgmap v3508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5104.577710 secs 2024-07-26T12:52:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:26 smithi189 ceph-mon[36790]: pgmap v3508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5104.577710 secs 2024-07-26T12:52:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:26 smithi193 ceph-mon[37473]: pgmap v3508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5104.577710 secs 2024-07-26T12:52:29.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:28 smithi193 ceph-mon[37473]: pgmap v3509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:28 smithi134 ceph-mon[31966]: pgmap v3509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:28 smithi189 ceph-mon[36790]: pgmap v3509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5090.152578 secs 2024-07-26T12:52:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5090.152578 secs 2024-07-26T12:52:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5090.152578 secs 2024-07-26T12:52:31.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:30 smithi134 ceph-mon[31966]: pgmap v3510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:30 smithi189 ceph-mon[36790]: pgmap v3510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:30 smithi193 ceph-mon[37473]: pgmap v3510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5109.577832 secs 2024-07-26T12:52:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5109.577832 secs 2024-07-26T12:52:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5109.577832 secs 2024-07-26T12:52:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:52:32.469+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:52:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:52:32.469+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:52:32.470 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:52:32.469+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:32 smithi134 ceph-mon[31966]: pgmap v3511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:32 smithi189 ceph-mon[36790]: pgmap v3511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:32 smithi193 ceph-mon[37473]: pgmap v3511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5095.152698 secs 2024-07-26T12:52:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:34 smithi134 ceph-mon[31966]: pgmap v3512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:52:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5095.152698 secs 2024-07-26T12:52:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:34 smithi189 ceph-mon[36790]: pgmap v3512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:52:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5095.152698 secs 2024-07-26T12:52:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:34 smithi193 ceph-mon[37473]: pgmap v3512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T12:52:36.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:52:36.996+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:52:36.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:52:36.996+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:36.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:52:36.996+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:52:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:36 smithi134 ceph-mon[31966]: pgmap v3513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5114.577952 secs 2024-07-26T12:52:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:36 smithi189 ceph-mon[36790]: pgmap v3513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5114.577952 secs 2024-07-26T12:52:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:36 smithi193 ceph-mon[37473]: pgmap v3513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5114.577952 secs 2024-07-26T12:52:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:38 smithi134 ceph-mon[31966]: pgmap v3514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:38 smithi189 ceph-mon[36790]: pgmap v3514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:38 smithi193 ceph-mon[37473]: pgmap v3514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5100.152808 secs 2024-07-26T12:52:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5100.152808 secs 2024-07-26T12:52:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5100.152808 secs 2024-07-26T12:52:41.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:40 smithi134 ceph-mon[31966]: pgmap v3515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:40 smithi189 ceph-mon[36790]: pgmap v3515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:40 smithi193 ceph-mon[37473]: pgmap v3515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5119.578073 secs 2024-07-26T12:52:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5119.578073 secs 2024-07-26T12:52:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5119.578073 secs 2024-07-26T12:52:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:42 smithi134 ceph-mon[31966]: pgmap v3516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:42 smithi189 ceph-mon[36790]: pgmap v3516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:42 smithi193 ceph-mon[37473]: pgmap v3516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5105.152965 secs 2024-07-26T12:52:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:44 smithi134 ceph-mon[31966]: pgmap v3517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5105.152965 secs 2024-07-26T12:52:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:44 smithi189 ceph-mon[36790]: pgmap v3517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5105.152965 secs 2024-07-26T12:52:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:44 smithi193 ceph-mon[37473]: pgmap v3517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:46 smithi134 ceph-mon[31966]: pgmap v3518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5124.578195 secs 2024-07-26T12:52:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:46 smithi189 ceph-mon[36790]: pgmap v3518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5124.578195 secs 2024-07-26T12:52:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:46 smithi193 ceph-mon[37473]: pgmap v3518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5124.578195 secs 2024-07-26T12:52:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:52:47.479+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:52:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:52:47.479+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:52:47.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:52:47.479+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:48 smithi134 ceph-mon[31966]: pgmap v3519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:48 smithi189 ceph-mon[36790]: pgmap v3519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:48 smithi193 ceph-mon[37473]: pgmap v3519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5110.153033 secs 2024-07-26T12:52:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5110.153033 secs 2024-07-26T12:52:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5110.153033 secs 2024-07-26T12:52:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:50 smithi189 ceph-mon[36790]: pgmap v3520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:50 smithi193 ceph-mon[37473]: pgmap v3520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:50 smithi134 ceph-mon[31966]: pgmap v3520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:52:52.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:52:52.009+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:52:52.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:52:52.009+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:52:52.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:52:52.009+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:52:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5129.578354 secs 2024-07-26T12:52:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5129.578354 secs 2024-07-26T12:52:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5129.578354 secs 2024-07-26T12:52:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:52 smithi189 ceph-mon[36790]: pgmap v3521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:52 smithi193 ceph-mon[37473]: pgmap v3521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:52 smithi134 ceph-mon[31966]: pgmap v3521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5115.153156 secs 2024-07-26T12:52:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:54 smithi193 ceph-mon[37473]: pgmap v3522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5115.153156 secs 2024-07-26T12:52:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:54 smithi134 ceph-mon[31966]: pgmap v3522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5115.153156 secs 2024-07-26T12:52:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:54 smithi189 ceph-mon[36790]: pgmap v3522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:56 smithi193 ceph-mon[37473]: pgmap v3523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5134.578470 secs 2024-07-26T12:52:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:56 smithi134 ceph-mon[31966]: pgmap v3523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5134.578470 secs 2024-07-26T12:52:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:56 smithi189 ceph-mon[36790]: pgmap v3523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:52:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5134.578470 secs 2024-07-26T12:52:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:58 smithi193 ceph-mon[37473]: pgmap v3524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:59.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:58 smithi134 ceph-mon[31966]: pgmap v3524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:52:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:58 smithi189 ceph-mon[36790]: pgmap v3524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:52:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5120.153223 secs 2024-07-26T12:53:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:52:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5120.153223 secs 2024-07-26T12:53:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:52:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5120.153223 secs 2024-07-26T12:53:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:00 smithi193 ceph-mon[37473]: pgmap v3525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:53:01.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:00 smithi134 ceph-mon[31966]: pgmap v3525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:53:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:00 smithi189 ceph-mon[36790]: pgmap v3525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:53:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5139.578584 secs 2024-07-26T12:53:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:01 smithi193 ceph-mon[37473]: pgmap v3526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5139.578584 secs 2024-07-26T12:53:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:01 smithi134 ceph-mon[31966]: pgmap v3526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5139.578584 secs 2024-07-26T12:53:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:01 smithi189 ceph-mon[36790]: pgmap v3526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:02.483 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:53:02.482+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:53:02.484 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:53:02.482+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:53:02.484 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:53:02.482+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5125.153321 secs 2024-07-26T12:53:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:04 smithi134 ceph-mon[31966]: pgmap v3527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5125.153321 secs 2024-07-26T12:53:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:04 smithi189 ceph-mon[36790]: pgmap v3527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5125.153321 secs 2024-07-26T12:53:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:04 smithi193 ceph-mon[37473]: pgmap v3527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:07.026 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:53:07.025+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:53:07.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:53:07.025+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:53:07.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:53:07.025+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:06 smithi134 ceph-mon[31966]: pgmap v3528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5144.578709 secs 2024-07-26T12:53:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:06 smithi189 ceph-mon[36790]: pgmap v3528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5144.578709 secs 2024-07-26T12:53:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:06 smithi193 ceph-mon[37473]: pgmap v3528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5144.578709 secs 2024-07-26T12:53:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:08 smithi134 ceph-mon[31966]: pgmap v3529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:53:09.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:08 smithi189 ceph-mon[36790]: pgmap v3529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:53:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:08 smithi193 ceph-mon[37473]: pgmap v3529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:53:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:09.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T12:53:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5130.153474 secs 2024-07-26T12:53:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:09 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:53:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:09 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:53:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:09 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:53:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:09 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:53:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:53:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:53:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:09 smithi134 ceph-mon[31966]: pgmap v3530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:53:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:09 smithi134 ceph-mon[31966]: pgmap v3531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:53:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:53:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5130.153474 secs 2024-07-26T12:53:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:09 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:53:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:09 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:53:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:09 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:53:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:09 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:53:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:53:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:53:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:09 smithi189 ceph-mon[36790]: pgmap v3530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:53:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:09 smithi189 ceph-mon[36790]: pgmap v3531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:53:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:53:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5130.153474 secs 2024-07-26T12:53:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:09 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T12:53:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:09 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T12:53:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:09 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T12:53:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:09 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T12:53:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:53:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:53:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:09 smithi193 ceph-mon[37473]: pgmap v3530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:53:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:09 smithi193 ceph-mon[37473]: pgmap v3531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:53:10.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:53:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:11 smithi193 ceph-mon[37473]: pgmap v3532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:53:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5149.578809 secs 2024-07-26T12:53:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:11 smithi134 ceph-mon[31966]: pgmap v3532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:53:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5149.578809 secs 2024-07-26T12:53:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:11 smithi189 ceph-mon[36790]: pgmap v3532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:53:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5149.578809 secs 2024-07-26T12:53:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:13 smithi193 ceph-mon[37473]: pgmap v3533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:53:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:13 smithi134 ceph-mon[31966]: pgmap v3533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:53:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:13 smithi189 ceph-mon[36790]: pgmap v3533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:53:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5135.153597 secs 2024-07-26T12:53:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5135.153597 secs 2024-07-26T12:53:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5135.153597 secs 2024-07-26T12:53:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:15 smithi189 ceph-mon[36790]: pgmap v3534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:53:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:15 smithi193 ceph-mon[37473]: pgmap v3534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:53:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:15 smithi134 ceph-mon[31966]: pgmap v3534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:53:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5154.578933 secs 2024-07-26T12:53:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5154.578933 secs 2024-07-26T12:53:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5154.578933 secs 2024-07-26T12:53:17.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:53:17.498+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:53:17.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:53:17.498+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:17.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:53:17.498+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:53:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:17 smithi193 ceph-mon[37473]: pgmap v3535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:53:18.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:17 smithi189 ceph-mon[36790]: pgmap v3535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:53:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:17 smithi134 ceph-mon[31966]: pgmap v3535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:53:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:19 smithi134 ceph-mon[31966]: pgmap v3536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:53:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:19 smithi189 ceph-mon[36790]: pgmap v3536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:53:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:19 smithi193 ceph-mon[37473]: pgmap v3536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T12:53:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5140.153726 secs 2024-07-26T12:53:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5140.153726 secs 2024-07-26T12:53:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5140.153726 secs 2024-07-26T12:53:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:21 smithi134 ceph-mon[31966]: pgmap v3537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:53:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:21 smithi189 ceph-mon[36790]: pgmap v3537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:53:21.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:21 smithi193 ceph-mon[37473]: pgmap v3537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:53:22.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:53:22.040+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:53:22.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:53:22.041+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:53:22.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:53:22.041+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5159.579051 secs 2024-07-26T12:53:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5159.579051 secs 2024-07-26T12:53:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5159.579051 secs 2024-07-26T12:53:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:23 smithi134 ceph-mon[31966]: pgmap v3538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:53:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:23 smithi189 ceph-mon[36790]: pgmap v3538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:53:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:23 smithi193 ceph-mon[37473]: pgmap v3538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:53:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5145.153860 secs 2024-07-26T12:53:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5145.153860 secs 2024-07-26T12:53:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5145.153860 secs 2024-07-26T12:53:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:25 smithi189 ceph-mon[36790]: pgmap v3539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:53:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:25 smithi193 ceph-mon[37473]: pgmap v3539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:53:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:25 smithi134 ceph-mon[31966]: pgmap v3539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:53:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5164.579146 secs 2024-07-26T12:53:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5164.579146 secs 2024-07-26T12:53:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5164.579146 secs 2024-07-26T12:53:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:27 smithi193 ceph-mon[37473]: pgmap v3540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:27 smithi134 ceph-mon[31966]: pgmap v3540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:27 smithi189 ceph-mon[36790]: pgmap v3540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:29 smithi134 ceph-mon[31966]: pgmap v3541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:53:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:29 smithi189 ceph-mon[36790]: pgmap v3541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:53:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:29 smithi193 ceph-mon[37473]: pgmap v3541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:53:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5150.154018 secs 2024-07-26T12:53:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5150.154018 secs 2024-07-26T12:53:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5150.154018 secs 2024-07-26T12:53:31.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:31 smithi134 ceph-mon[31966]: pgmap v3542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:31 smithi189 ceph-mon[36790]: pgmap v3542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:31 smithi193 ceph-mon[37473]: pgmap v3542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5169.579265 secs 2024-07-26T12:53:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5169.579265 secs 2024-07-26T12:53:32.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:53:32.513+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:53:32.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:53:32.513+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:53:32.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:53:32.513+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5169.579265 secs 2024-07-26T12:53:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:33 smithi134 ceph-mon[31966]: pgmap v3543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:33 smithi189 ceph-mon[36790]: pgmap v3543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:33 smithi193 ceph-mon[37473]: pgmap v3543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5155.154082 secs 2024-07-26T12:53:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5155.154082 secs 2024-07-26T12:53:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5155.154082 secs 2024-07-26T12:53:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:35 smithi189 ceph-mon[36790]: pgmap v3544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:53:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:35 smithi193 ceph-mon[37473]: pgmap v3544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:53:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:35 smithi134 ceph-mon[31966]: pgmap v3544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:53:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5174.579362 secs 2024-07-26T12:53:37.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:53:37.056+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:53:37.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:53:37.056+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:53:37.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:53:37.056+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:37.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5174.579362 secs 2024-07-26T12:53:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5174.579362 secs 2024-07-26T12:53:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:37 smithi193 ceph-mon[37473]: pgmap v3545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:37 smithi134 ceph-mon[31966]: pgmap v3545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:37 smithi189 ceph-mon[36790]: pgmap v3545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:39 smithi134 ceph-mon[31966]: pgmap v3546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:39 smithi189 ceph-mon[36790]: pgmap v3546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:39 smithi193 ceph-mon[37473]: pgmap v3546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5160.154154 secs 2024-07-26T12:53:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5160.154154 secs 2024-07-26T12:53:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5160.154154 secs 2024-07-26T12:53:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:41 smithi134 ceph-mon[31966]: pgmap v3547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:41 smithi189 ceph-mon[36790]: pgmap v3547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:41 smithi193 ceph-mon[37473]: pgmap v3547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5179.579424 secs 2024-07-26T12:53:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5179.579424 secs 2024-07-26T12:53:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5179.579424 secs 2024-07-26T12:53:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:43 smithi134 ceph-mon[31966]: pgmap v3548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:43 smithi189 ceph-mon[36790]: pgmap v3548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:43 smithi193 ceph-mon[37473]: pgmap v3548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5165.154221 secs 2024-07-26T12:53:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5165.154221 secs 2024-07-26T12:53:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5165.154221 secs 2024-07-26T12:53:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:45 smithi189 ceph-mon[36790]: pgmap v3549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:53:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:45 smithi193 ceph-mon[37473]: pgmap v3549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:53:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:45 smithi134 ceph-mon[31966]: pgmap v3549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:53:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5184.579548 secs 2024-07-26T12:53:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5184.579548 secs 2024-07-26T12:53:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5184.579548 secs 2024-07-26T12:53:47.530 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:53:47.529+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:53:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:53:47.529+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:47.531 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:53:47.529+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:53:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:47 smithi193 ceph-mon[37473]: pgmap v3550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:48.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:47 smithi189 ceph-mon[36790]: pgmap v3550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:47 smithi134 ceph-mon[31966]: pgmap v3550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:49 smithi134 ceph-mon[31966]: pgmap v3551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:49 smithi189 ceph-mon[36790]: pgmap v3551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:49 smithi193 ceph-mon[37473]: pgmap v3551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:53:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5170.154334 secs 2024-07-26T12:53:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5170.154334 secs 2024-07-26T12:53:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5170.154334 secs 2024-07-26T12:53:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:51 smithi134 ceph-mon[31966]: pgmap v3552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:51 smithi189 ceph-mon[36790]: pgmap v3552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:51 smithi193 ceph-mon[37473]: pgmap v3552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:53:52.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:53:52.072+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:53:52.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:53:52.072+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:53:52.074 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:53:52.072+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:53:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5189.579691 secs 2024-07-26T12:53:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5189.579691 secs 2024-07-26T12:53:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5189.579691 secs 2024-07-26T12:53:53.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:53 smithi134 ceph-mon[31966]: pgmap v3553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:53 smithi189 ceph-mon[36790]: pgmap v3553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:53 smithi193 ceph-mon[37473]: pgmap v3553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:53:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5175.154437 secs 2024-07-26T12:53:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5175.154437 secs 2024-07-26T12:53:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5175.154437 secs 2024-07-26T12:53:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:55 smithi189 ceph-mon[36790]: pgmap v3554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:53:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:55 smithi193 ceph-mon[37473]: pgmap v3554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:53:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:55 smithi134 ceph-mon[31966]: pgmap v3554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:53:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5194.579806 secs 2024-07-26T12:53:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5194.579806 secs 2024-07-26T12:53:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5194.579806 secs 2024-07-26T12:53:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:57 smithi193 ceph-mon[37473]: pgmap v3555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:57 smithi134 ceph-mon[31966]: pgmap v3555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:57 smithi189 ceph-mon[36790]: pgmap v3555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:53:59.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:53:59 smithi134 ceph-mon[31966]: pgmap v3556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:53:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:53:59 smithi189 ceph-mon[36790]: pgmap v3556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:53:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:53:59 smithi193 ceph-mon[37473]: pgmap v3556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:54:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5180.154567 secs 2024-07-26T12:54:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5180.154567 secs 2024-07-26T12:54:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5180.154567 secs 2024-07-26T12:54:01.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:01 smithi134 ceph-mon[31966]: pgmap v3557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:54:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:01 smithi189 ceph-mon[36790]: pgmap v3557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:54:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:01 smithi193 ceph-mon[37473]: pgmap v3557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:54:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5199.579923 secs 2024-07-26T12:54:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5199.579923 secs 2024-07-26T12:54:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5199.579923 secs 2024-07-26T12:54:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:54:02.539+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:54:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:54:02.539+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:54:02.541 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:54:02.539+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:03 smithi134 ceph-mon[31966]: pgmap v3558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:54:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:03 smithi189 ceph-mon[36790]: pgmap v3558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:54:03.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:03 smithi193 ceph-mon[37473]: pgmap v3558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:54:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5185.154702 secs 2024-07-26T12:54:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5185.154702 secs 2024-07-26T12:54:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5185.154702 secs 2024-07-26T12:54:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:05 smithi189 ceph-mon[36790]: pgmap v3559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:54:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:05 smithi193 ceph-mon[37473]: pgmap v3559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:54:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:05 smithi134 ceph-mon[31966]: pgmap v3559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:54:07.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:54:07.087+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:54:07.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:54:07.087+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:07.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:54:07.087+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:54:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5204.580006 secs 2024-07-26T12:54:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:07 smithi134 ceph-mon[31966]: pgmap v3560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5204.580006 secs 2024-07-26T12:54:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:07 smithi189 ceph-mon[36790]: pgmap v3560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5204.580006 secs 2024-07-26T12:54:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:07 smithi193 ceph-mon[37473]: pgmap v3560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:09 smithi134 ceph-mon[31966]: pgmap v3561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:54:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:09 smithi189 ceph-mon[36790]: pgmap v3561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:54:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:09 smithi193 ceph-mon[37473]: pgmap v3561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T12:54:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5190.154849 secs 2024-07-26T12:54:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:54:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5190.154849 secs 2024-07-26T12:54:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:54:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5190.154849 secs 2024-07-26T12:54:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:54:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:54:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:54:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:11 smithi134 ceph-mon[31966]: pgmap v3562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:54:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:11 smithi134 ceph-mon[31966]: pgmap v3563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:54:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:54:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:54:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:54:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:11 smithi189 ceph-mon[36790]: pgmap v3562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:54:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:11 smithi189 ceph-mon[36790]: pgmap v3563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:54:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:54:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:54:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:54:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:11 smithi193 ceph-mon[37473]: pgmap v3562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T12:54:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:11 smithi193 ceph-mon[37473]: pgmap v3563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:54:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:54:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5209.580144 secs 2024-07-26T12:54:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5209.580144 secs 2024-07-26T12:54:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5209.580144 secs 2024-07-26T12:54:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:13 smithi134 ceph-mon[31966]: pgmap v3564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:54:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:13 smithi189 ceph-mon[36790]: pgmap v3564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:54:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:13 smithi193 ceph-mon[37473]: pgmap v3564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:54:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5195.154983 secs 2024-07-26T12:54:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5195.154983 secs 2024-07-26T12:54:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5195.154983 secs 2024-07-26T12:54:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:15 smithi189 ceph-mon[36790]: pgmap v3565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:54:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:15 smithi193 ceph-mon[37473]: pgmap v3565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:54:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:15 smithi134 ceph-mon[31966]: pgmap v3565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:54:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5214.580264 secs 2024-07-26T12:54:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:17 smithi189 ceph-mon[36790]: pgmap v3566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:54:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5214.580264 secs 2024-07-26T12:54:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:17 smithi193 ceph-mon[37473]: pgmap v3566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:54:17.550 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:54:17.549+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:54:17.550 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:54:17.549+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:54:17.551 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:54:17.549+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5214.580264 secs 2024-07-26T12:54:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:17 smithi134 ceph-mon[31966]: pgmap v3566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:54:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:19 smithi189 ceph-mon[36790]: pgmap v3567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:54:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5200.155094 secs 2024-07-26T12:54:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:19 smithi193 ceph-mon[37473]: pgmap v3567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:54:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5200.155094 secs 2024-07-26T12:54:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:19 smithi134 ceph-mon[31966]: pgmap v3567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:54:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5200.155094 secs 2024-07-26T12:54:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:21 smithi193 ceph-mon[37473]: pgmap v3568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:54:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:21 smithi134 ceph-mon[31966]: pgmap v3568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:54:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:21 smithi189 ceph-mon[36790]: pgmap v3568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:54:22.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:54:22.103+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:54:22.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:54:22.103+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:54:22.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:54:22.103+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5219.580388 secs 2024-07-26T12:54:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5219.580388 secs 2024-07-26T12:54:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5219.580388 secs 2024-07-26T12:54:23.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:54:23.235+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T12:54:23.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:54:23.235+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T12:54:23.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:54:23.235+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T12:54:23.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:54:23.235+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T12:54:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:23 smithi193 ceph-mon[37473]: pgmap v3569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:23 smithi134 ceph-mon[31966]: pgmap v3569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:23 smithi189 ceph-mon[36790]: pgmap v3569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5205.155222 secs 2024-07-26T12:54:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5205.155222 secs 2024-07-26T12:54:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5205.155222 secs 2024-07-26T12:54:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:25 smithi193 ceph-mon[37473]: pgmap v3570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:54:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:25 smithi134 ceph-mon[31966]: pgmap v3570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:54:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:25 smithi189 ceph-mon[36790]: pgmap v3570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:54:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5224.580502 secs 2024-07-26T12:54:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:27 smithi193 ceph-mon[37473]: pgmap v3571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5224.580502 secs 2024-07-26T12:54:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:27 smithi134 ceph-mon[31966]: pgmap v3571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5224.580502 secs 2024-07-26T12:54:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:27 smithi189 ceph-mon[36790]: pgmap v3571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:29 smithi193 ceph-mon[37473]: pgmap v3572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5210.155360 secs 2024-07-26T12:54:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:29 smithi134 ceph-mon[31966]: pgmap v3572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5210.155360 secs 2024-07-26T12:54:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:29 smithi189 ceph-mon[36790]: pgmap v3572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5210.155360 secs 2024-07-26T12:54:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:31 smithi193 ceph-mon[37473]: pgmap v3573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:31 smithi134 ceph-mon[31966]: pgmap v3573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:31 smithi189 ceph-mon[36790]: pgmap v3573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5229.580628 secs 2024-07-26T12:54:32.566 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:54:32.565+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:54:32.566 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:54:32.565+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:54:32.566 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:54:32.565+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5229.580628 secs 2024-07-26T12:54:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5229.580628 secs 2024-07-26T12:54:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:33 smithi193 ceph-mon[37473]: pgmap v3574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:33 smithi134 ceph-mon[31966]: pgmap v3574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:33 smithi189 ceph-mon[36790]: pgmap v3574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:33.750 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:54:33.750+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:54:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:54:33.750+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:54:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:54:33.750+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:54:33.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:54:33.750+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:54:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5215.155480 secs 2024-07-26T12:54:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5215.155480 secs 2024-07-26T12:54:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5215.155480 secs 2024-07-26T12:54:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:35 smithi134 ceph-mon[31966]: pgmap v3575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:54:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:35 smithi189 ceph-mon[36790]: pgmap v3575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:54:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:35 smithi193 ceph-mon[37473]: pgmap v3575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:54:37.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:54:37.105+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:54:37.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:54:37.105+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:37.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:54:37.105+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:54:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5234.580749 secs 2024-07-26T12:54:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:37 smithi134 ceph-mon[31966]: pgmap v3576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5234.580749 secs 2024-07-26T12:54:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:37 smithi189 ceph-mon[36790]: pgmap v3576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5234.580749 secs 2024-07-26T12:54:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:37 smithi193 ceph-mon[37473]: pgmap v3576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:39 smithi134 ceph-mon[31966]: pgmap v3577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5220.155625 secs 2024-07-26T12:54:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:39 smithi189 ceph-mon[36790]: pgmap v3577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5220.155625 secs 2024-07-26T12:54:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:39 smithi193 ceph-mon[37473]: pgmap v3577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5220.155625 secs 2024-07-26T12:54:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:41 smithi134 ceph-mon[31966]: pgmap v3578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:41 smithi189 ceph-mon[36790]: pgmap v3578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:41 smithi193 ceph-mon[37473]: pgmap v3578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5239.580872 secs 2024-07-26T12:54:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5239.580872 secs 2024-07-26T12:54:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5239.580872 secs 2024-07-26T12:54:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:43 smithi134 ceph-mon[31966]: pgmap v3579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:43 smithi189 ceph-mon[36790]: pgmap v3579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:43 smithi193 ceph-mon[37473]: pgmap v3579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5225.155754 secs 2024-07-26T12:54:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5225.155754 secs 2024-07-26T12:54:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5225.155754 secs 2024-07-26T12:54:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:45 smithi134 ceph-mon[31966]: pgmap v3580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:54:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:45 smithi189 ceph-mon[36790]: pgmap v3580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:54:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:45 smithi193 ceph-mon[37473]: pgmap v3580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:54:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:54:47.580+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:54:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:54:47.580+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:54:47.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:54:47.580+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5244.580956 secs 2024-07-26T12:54:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:47 smithi134 ceph-mon[31966]: pgmap v3581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5244.580956 secs 2024-07-26T12:54:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:47 smithi189 ceph-mon[36790]: pgmap v3581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5244.580956 secs 2024-07-26T12:54:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:47 smithi193 ceph-mon[37473]: pgmap v3581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:49.475 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:49 smithi134 ceph-mon[31966]: pgmap v3582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:49.475 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5230.155892 secs 2024-07-26T12:54:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:49 smithi189 ceph-mon[36790]: pgmap v3582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5230.155892 secs 2024-07-26T12:54:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:49 smithi193 ceph-mon[37473]: pgmap v3582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5230.155892 secs 2024-07-26T12:54:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:51 smithi134 ceph-mon[31966]: pgmap v3583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:51 smithi189 ceph-mon[36790]: pgmap v3583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:51 smithi193 ceph-mon[37473]: pgmap v3583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:52.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:54:52.120+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:54:52.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:54:52.121+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:54:52.123 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:54:52.121+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:54:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5249.581080 secs 2024-07-26T12:54:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5249.581080 secs 2024-07-26T12:54:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5249.581080 secs 2024-07-26T12:54:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:53 smithi134 ceph-mon[31966]: pgmap v3584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:53 smithi189 ceph-mon[36790]: pgmap v3584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:53 smithi193 ceph-mon[37473]: pgmap v3584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:54:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5235.156026 secs 2024-07-26T12:54:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5235.156026 secs 2024-07-26T12:54:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5235.156026 secs 2024-07-26T12:54:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:55 smithi134 ceph-mon[31966]: pgmap v3585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:54:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:55 smithi189 ceph-mon[36790]: pgmap v3585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:54:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:55 smithi193 ceph-mon[37473]: pgmap v3585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:54:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5254.581177 secs 2024-07-26T12:54:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:57 smithi134 ceph-mon[31966]: pgmap v3586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5254.581177 secs 2024-07-26T12:54:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:57 smithi189 ceph-mon[36790]: pgmap v3586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5254.581177 secs 2024-07-26T12:54:57.803 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:57 smithi193 ceph-mon[37473]: pgmap v3586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:54:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:59 smithi134 ceph-mon[31966]: pgmap v3587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:54:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5240.156092 secs 2024-07-26T12:54:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:59 smithi189 ceph-mon[36790]: pgmap v3587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:54:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5240.156092 secs 2024-07-26T12:54:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:59 smithi193 ceph-mon[37473]: pgmap v3587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:54:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:54:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5240.156092 secs 2024-07-26T12:55:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:01 smithi134 ceph-mon[31966]: pgmap v3588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:01 smithi189 ceph-mon[36790]: pgmap v3588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:01 smithi193 ceph-mon[37473]: pgmap v3588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:55:02.596+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:55:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:55:02.596+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:55:02.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:55:02.596+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5259.581298 secs 2024-07-26T12:55:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5259.581298 secs 2024-07-26T12:55:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5259.581298 secs 2024-07-26T12:55:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:03 smithi134 ceph-mon[31966]: pgmap v3589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:55:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:03 smithi189 ceph-mon[36790]: pgmap v3589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:55:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:03 smithi193 ceph-mon[37473]: pgmap v3589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:55:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5245.156214 secs 2024-07-26T12:55:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5245.156214 secs 2024-07-26T12:55:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5245.156214 secs 2024-07-26T12:55:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:05 smithi134 ceph-mon[31966]: pgmap v3590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:55:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:05 smithi189 ceph-mon[36790]: pgmap v3590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:55:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:05 smithi193 ceph-mon[37473]: pgmap v3590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:55:07.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:55:07.136+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:55:07.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:55:07.136+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:07.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:55:07.136+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:55:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5264.581413 secs 2024-07-26T12:55:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:07 smithi134 ceph-mon[31966]: pgmap v3591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5264.581413 secs 2024-07-26T12:55:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:07 smithi189 ceph-mon[36790]: pgmap v3591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5264.581413 secs 2024-07-26T12:55:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:07 smithi193 ceph-mon[37473]: pgmap v3591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:09 smithi189 ceph-mon[36790]: pgmap v3592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5250.156295 secs 2024-07-26T12:55:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:09 smithi193 ceph-mon[37473]: pgmap v3592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5250.156295 secs 2024-07-26T12:55:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:09 smithi134 ceph-mon[31966]: pgmap v3592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5250.156295 secs 2024-07-26T12:55:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:55:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:55:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:55:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:11 smithi134 ceph-mon[31966]: pgmap v3593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5269.581504 secs 2024-07-26T12:55:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:55:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:55:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:11 smithi189 ceph-mon[36790]: pgmap v3593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5269.581504 secs 2024-07-26T12:55:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:55:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:55:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:11 smithi193 ceph-mon[37473]: pgmap v3593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5269.581504 secs 2024-07-26T12:55:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:55:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:55:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:55:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:55:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:12 smithi189 ceph-mon[36790]: pgmap v3594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:55:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:12 smithi189 ceph-mon[36790]: pgmap v3595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:13.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:55:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:55:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:55:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:12 smithi193 ceph-mon[37473]: pgmap v3594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:55:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:12 smithi193 ceph-mon[37473]: pgmap v3595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:55:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:55:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:55:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:12 smithi134 ceph-mon[31966]: pgmap v3594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T12:55:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:12 smithi134 ceph-mon[31966]: pgmap v3595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:55:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:14 smithi134 ceph-mon[31966]: pgmap v3596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:14 smithi189 ceph-mon[36790]: pgmap v3596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:14 smithi193 ceph-mon[37473]: pgmap v3596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5255.156415 secs 2024-07-26T12:55:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5255.156415 secs 2024-07-26T12:55:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5255.156415 secs 2024-07-26T12:55:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:16 smithi134 ceph-mon[31966]: pgmap v3597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:16 smithi189 ceph-mon[36790]: pgmap v3597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:16 smithi193 ceph-mon[37473]: pgmap v3597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5274.581620 secs 2024-07-26T12:55:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5274.581620 secs 2024-07-26T12:55:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5274.581620 secs 2024-07-26T12:55:17.613 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:55:17.612+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:55:17.613 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:55:17.612+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:55:17.613 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:55:17.612+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:18 smithi134 ceph-mon[31966]: pgmap v3598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:18 smithi189 ceph-mon[36790]: pgmap v3598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:18.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:18 smithi193 ceph-mon[37473]: pgmap v3598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T12:55:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5260.156540 secs 2024-07-26T12:55:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5260.156540 secs 2024-07-26T12:55:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5260.156540 secs 2024-07-26T12:55:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:20 smithi193 ceph-mon[37473]: pgmap v3599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:55:20.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:20 smithi134 ceph-mon[31966]: pgmap v3599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:55:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:20 smithi189 ceph-mon[36790]: pgmap v3599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T12:55:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5279.581735 secs 2024-07-26T12:55:22.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:55:22.152+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:55:22.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:55:22.152+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:55:22.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:55:22.152+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:22.154 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5279.581735 secs 2024-07-26T12:55:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5279.581735 secs 2024-07-26T12:55:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:22 smithi193 ceph-mon[37473]: pgmap v3600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:55:23.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:22 smithi134 ceph-mon[31966]: pgmap v3600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:55:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:22 smithi189 ceph-mon[36790]: pgmap v3600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:55:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:24 smithi134 ceph-mon[31966]: pgmap v3601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:55:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:24 smithi189 ceph-mon[36790]: pgmap v3601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:55:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:24 smithi193 ceph-mon[37473]: pgmap v3601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:55:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5265.156661 secs 2024-07-26T12:55:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5265.156661 secs 2024-07-26T12:55:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5265.156661 secs 2024-07-26T12:55:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:26 smithi134 ceph-mon[31966]: pgmap v3602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:26 smithi189 ceph-mon[36790]: pgmap v3602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:26 smithi193 ceph-mon[37473]: pgmap v3602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5284.581845 secs 2024-07-26T12:55:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5284.581845 secs 2024-07-26T12:55:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5284.581845 secs 2024-07-26T12:55:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:28 smithi134 ceph-mon[31966]: pgmap v3603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:28 smithi189 ceph-mon[36790]: pgmap v3603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:28 smithi193 ceph-mon[37473]: pgmap v3603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5270.156774 secs 2024-07-26T12:55:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5270.156774 secs 2024-07-26T12:55:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5270.156774 secs 2024-07-26T12:55:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:30 smithi193 ceph-mon[37473]: pgmap v3604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:55:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:30 smithi134 ceph-mon[31966]: pgmap v3604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:55:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:30 smithi189 ceph-mon[36790]: pgmap v3604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:55:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5289.581962 secs 2024-07-26T12:55:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5289.581962 secs 2024-07-26T12:55:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5289.581962 secs 2024-07-26T12:55:32.628 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:55:32.627+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:55:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:55:32.627+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:55:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:55:32.627+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:32 smithi193 ceph-mon[37473]: pgmap v3605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:55:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:32 smithi134 ceph-mon[31966]: pgmap v3605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:55:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:32 smithi189 ceph-mon[36790]: pgmap v3605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:55:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:34 smithi134 ceph-mon[31966]: pgmap v3606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:55:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:34 smithi189 ceph-mon[36790]: pgmap v3606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:55:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:34 smithi193 ceph-mon[37473]: pgmap v3606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:55:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5275.156915 secs 2024-07-26T12:55:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5275.156915 secs 2024-07-26T12:55:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5275.156915 secs 2024-07-26T12:55:36.329 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:36 smithi134 ceph-mon[31966]: pgmap v3607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:36 smithi189 ceph-mon[36790]: pgmap v3607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:36 smithi193 ceph-mon[37473]: pgmap v3607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:37.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:55:37.166+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:55:37.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:55:37.166+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:37.168 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:55:37.166+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:55:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5294.582089 secs 2024-07-26T12:55:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5294.582089 secs 2024-07-26T12:55:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5294.582089 secs 2024-07-26T12:55:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:38 smithi134 ceph-mon[31966]: pgmap v3608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:38 smithi189 ceph-mon[36790]: pgmap v3608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:38 smithi193 ceph-mon[37473]: pgmap v3608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5280.157048 secs 2024-07-26T12:55:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5280.157048 secs 2024-07-26T12:55:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5280.157048 secs 2024-07-26T12:55:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:40 smithi193 ceph-mon[37473]: pgmap v3609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:55:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:40 smithi134 ceph-mon[31966]: pgmap v3609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:55:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:40 smithi189 ceph-mon[36790]: pgmap v3609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:55:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5299.582215 secs 2024-07-26T12:55:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5299.582215 secs 2024-07-26T12:55:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5299.582215 secs 2024-07-26T12:55:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:42 smithi193 ceph-mon[37473]: pgmap v3610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:55:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:42 smithi134 ceph-mon[31966]: pgmap v3610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:55:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:42 smithi189 ceph-mon[36790]: pgmap v3610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:55:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:44 smithi134 ceph-mon[31966]: pgmap v3611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:55:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:44 smithi189 ceph-mon[36790]: pgmap v3611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:55:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:44 smithi193 ceph-mon[37473]: pgmap v3611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:55:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5285.157154 secs 2024-07-26T12:55:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5285.157154 secs 2024-07-26T12:55:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5285.157154 secs 2024-07-26T12:55:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:46 smithi134 ceph-mon[31966]: pgmap v3612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:46 smithi189 ceph-mon[36790]: pgmap v3612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:46 smithi193 ceph-mon[37473]: pgmap v3612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5304.582322 secs 2024-07-26T12:55:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5304.582322 secs 2024-07-26T12:55:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5304.582322 secs 2024-07-26T12:55:47.632 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:55:47.631+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:55:47.632 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:55:47.631+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:55:47.632 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:55:47.631+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:48 smithi134 ceph-mon[31966]: pgmap v3613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:48 smithi189 ceph-mon[36790]: pgmap v3613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:48 smithi193 ceph-mon[37473]: pgmap v3613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5290.157252 secs 2024-07-26T12:55:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5290.157252 secs 2024-07-26T12:55:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5290.157252 secs 2024-07-26T12:55:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:50 smithi193 ceph-mon[37473]: pgmap v3614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:55:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:50 smithi134 ceph-mon[31966]: pgmap v3614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:55:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:50 smithi189 ceph-mon[36790]: pgmap v3614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:55:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5309.582442 secs 2024-07-26T12:55:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5309.582442 secs 2024-07-26T12:55:52.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:55:52.179+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:55:52.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:55:52.179+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:55:52.180 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:55:52.179+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:55:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5309.582442 secs 2024-07-26T12:55:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:52 smithi193 ceph-mon[37473]: pgmap v3615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:55:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:52 smithi134 ceph-mon[31966]: pgmap v3615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:55:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:52 smithi189 ceph-mon[36790]: pgmap v3615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:55:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:54 smithi134 ceph-mon[31966]: pgmap v3616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:55:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:54 smithi189 ceph-mon[36790]: pgmap v3616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:55:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:54 smithi193 ceph-mon[37473]: pgmap v3616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:55:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5295.157380 secs 2024-07-26T12:55:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5295.157380 secs 2024-07-26T12:55:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5295.157380 secs 2024-07-26T12:55:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:56 smithi134 ceph-mon[31966]: pgmap v3617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:56 smithi189 ceph-mon[36790]: pgmap v3617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:56 smithi193 ceph-mon[37473]: pgmap v3617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:55:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5314.582551 secs 2024-07-26T12:55:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5314.582551 secs 2024-07-26T12:55:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5314.582551 secs 2024-07-26T12:55:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:58 smithi134 ceph-mon[31966]: pgmap v3618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:58 smithi189 ceph-mon[36790]: pgmap v3618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:58 smithi193 ceph-mon[37473]: pgmap v3618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:55:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:55:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5300.157502 secs 2024-07-26T12:55:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:55:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5300.157502 secs 2024-07-26T12:55:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:55:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5300.157502 secs 2024-07-26T12:56:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:00 smithi193 ceph-mon[37473]: pgmap v3619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:56:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:00 smithi134 ceph-mon[31966]: pgmap v3619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:56:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:00 smithi189 ceph-mon[36790]: pgmap v3619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:56:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5319.582632 secs 2024-07-26T12:56:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5319.582632 secs 2024-07-26T12:56:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5319.582632 secs 2024-07-26T12:56:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:56:02.647+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:56:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:56:02.647+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:56:02.648 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:56:02.647+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:02 smithi193 ceph-mon[37473]: pgmap v3620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:02 smithi134 ceph-mon[31966]: pgmap v3620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:02 smithi189 ceph-mon[36790]: pgmap v3620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:04 smithi134 ceph-mon[31966]: pgmap v3621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:04 smithi189 ceph-mon[36790]: pgmap v3621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:04 smithi193 ceph-mon[37473]: pgmap v3621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5305.157629 secs 2024-07-26T12:56:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5305.157629 secs 2024-07-26T12:56:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5305.157629 secs 2024-07-26T12:56:07.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:56:07.194+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:56:07.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:56:07.194+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:07.196 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:56:07.194+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:56:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5324.582697 secs 2024-07-26T12:56:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:07 smithi134 ceph-mon[31966]: pgmap v3622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5324.582697 secs 2024-07-26T12:56:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:07 smithi189 ceph-mon[36790]: pgmap v3622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5324.582697 secs 2024-07-26T12:56:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:07 smithi193 ceph-mon[37473]: pgmap v3622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:09 smithi134 ceph-mon[31966]: pgmap v3623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:09 smithi189 ceph-mon[36790]: pgmap v3623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:09 smithi193 ceph-mon[37473]: pgmap v3623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5310.157741 secs 2024-07-26T12:56:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5310.157741 secs 2024-07-26T12:56:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5310.157741 secs 2024-07-26T12:56:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:11 smithi134 ceph-mon[31966]: pgmap v3624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:56:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:11 smithi189 ceph-mon[36790]: pgmap v3624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:56:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:11 smithi193 ceph-mon[37473]: pgmap v3624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:56:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5329.582781 secs 2024-07-26T12:56:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:56:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5329.582781 secs 2024-07-26T12:56:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:56:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5329.582781 secs 2024-07-26T12:56:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:56:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:13 smithi134 ceph-mon[31966]: pgmap v3625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:13 smithi189 ceph-mon[36790]: pgmap v3625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:13 smithi193 ceph-mon[37473]: pgmap v3625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5315.157834 secs 2024-07-26T12:56:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:14.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5315.157834 secs 2024-07-26T12:56:14.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:14.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:14.947 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:14.947 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5315.157834 secs 2024-07-26T12:56:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:15 smithi134 ceph-mon[31966]: pgmap v3626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:15 smithi189 ceph-mon[36790]: pgmap v3626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:15 smithi193 ceph-mon[37473]: pgmap v3626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5334.582901 secs 2024-07-26T12:56:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:17 smithi134 ceph-mon[31966]: pgmap v3627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5334.582901 secs 2024-07-26T12:56:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:17 smithi189 ceph-mon[36790]: pgmap v3627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5334.582901 secs 2024-07-26T12:56:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:17 smithi193 ceph-mon[37473]: pgmap v3627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:56:17.662+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:56:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:56:17.662+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:56:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:56:17.662+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:56:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:56:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:18 smithi134 ceph-mon[31966]: pgmap v3628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:56:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:18 smithi134 ceph-mon[31966]: pgmap v3629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T12:56:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:56:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:56:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:18 smithi189 ceph-mon[36790]: pgmap v3628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:56:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:18 smithi189 ceph-mon[36790]: pgmap v3629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T12:56:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:18.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:56:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:56:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:18 smithi193 ceph-mon[37473]: pgmap v3628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:56:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:18 smithi193 ceph-mon[37473]: pgmap v3629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T12:56:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:56:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5320.157979 secs 2024-07-26T12:56:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5320.157979 secs 2024-07-26T12:56:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5320.157979 secs 2024-07-26T12:56:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:20 smithi193 ceph-mon[37473]: pgmap v3630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:20 smithi134 ceph-mon[31966]: pgmap v3630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:20 smithi189 ceph-mon[36790]: pgmap v3630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:22.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:56:22.209+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:56:22.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:56:22.209+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:22.211 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:56:22.209+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:56:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5339.583013 secs 2024-07-26T12:56:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:22 smithi193 ceph-mon[37473]: pgmap v3631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:56:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5339.583013 secs 2024-07-26T12:56:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:22 smithi134 ceph-mon[31966]: pgmap v3631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:56:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5339.583013 secs 2024-07-26T12:56:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:22 smithi189 ceph-mon[36790]: pgmap v3631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T12:56:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:24 smithi193 ceph-mon[37473]: pgmap v3632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5325.158111 secs 2024-07-26T12:56:24.550 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:24 smithi134 ceph-mon[31966]: pgmap v3632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:24.551 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5325.158111 secs 2024-07-26T12:56:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:24 smithi189 ceph-mon[36790]: pgmap v3632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5325.158111 secs 2024-07-26T12:56:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:26 smithi193 ceph-mon[37473]: pgmap v3633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:26 smithi134 ceph-mon[31966]: pgmap v3633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:26 smithi189 ceph-mon[36790]: pgmap v3633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T12:56:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5344.583129 secs 2024-07-26T12:56:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5344.583129 secs 2024-07-26T12:56:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5344.583129 secs 2024-07-26T12:56:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:28 smithi193 ceph-mon[37473]: pgmap v3634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:56:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:28 smithi134 ceph-mon[31966]: pgmap v3634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:56:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:28 smithi189 ceph-mon[36790]: pgmap v3634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:56:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5330.158256 secs 2024-07-26T12:56:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5330.158256 secs 2024-07-26T12:56:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5330.158256 secs 2024-07-26T12:56:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:30 smithi193 ceph-mon[37473]: pgmap v3635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:30 smithi134 ceph-mon[31966]: pgmap v3635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:30 smithi189 ceph-mon[36790]: pgmap v3635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5349.583246 secs 2024-07-26T12:56:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:32 smithi193 ceph-mon[37473]: pgmap v3636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5349.583246 secs 2024-07-26T12:56:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:32 smithi134 ceph-mon[31966]: pgmap v3636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:56:32.677+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:56:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:56:32.677+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:56:32.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:56:32.678+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5349.583246 secs 2024-07-26T12:56:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:32 smithi189 ceph-mon[36790]: pgmap v3636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:34 smithi193 ceph-mon[37473]: pgmap v3637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5335.158346 secs 2024-07-26T12:56:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:34 smithi134 ceph-mon[31966]: pgmap v3637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5335.158346 secs 2024-07-26T12:56:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:34 smithi189 ceph-mon[36790]: pgmap v3637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5335.158346 secs 2024-07-26T12:56:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:36 smithi193 ceph-mon[37473]: pgmap v3638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:36 smithi134 ceph-mon[31966]: pgmap v3638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:36 smithi189 ceph-mon[36790]: pgmap v3638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:37.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:56:37.210+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:56:37.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:56:37.210+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:37.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:56:37.210+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:56:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5354.583400 secs 2024-07-26T12:56:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5354.583400 secs 2024-07-26T12:56:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5354.583400 secs 2024-07-26T12:56:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:38 smithi193 ceph-mon[37473]: pgmap v3639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:38 smithi134 ceph-mon[31966]: pgmap v3639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:38 smithi189 ceph-mon[36790]: pgmap v3639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5340.158496 secs 2024-07-26T12:56:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5340.158496 secs 2024-07-26T12:56:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5340.158496 secs 2024-07-26T12:56:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:40 smithi134 ceph-mon[31966]: pgmap v3640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:40 smithi189 ceph-mon[36790]: pgmap v3640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:40 smithi193 ceph-mon[37473]: pgmap v3640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5359.583519 secs 2024-07-26T12:56:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:42 smithi134 ceph-mon[31966]: pgmap v3641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5359.583519 secs 2024-07-26T12:56:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:42 smithi189 ceph-mon[36790]: pgmap v3641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5359.583519 secs 2024-07-26T12:56:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:42 smithi193 ceph-mon[37473]: pgmap v3641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:44 smithi134 ceph-mon[31966]: pgmap v3642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5345.158626 secs 2024-07-26T12:56:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:44 smithi189 ceph-mon[36790]: pgmap v3642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5345.158626 secs 2024-07-26T12:56:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:44 smithi193 ceph-mon[37473]: pgmap v3642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5345.158626 secs 2024-07-26T12:56:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:46 smithi134 ceph-mon[31966]: pgmap v3643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:46 smithi189 ceph-mon[36790]: pgmap v3643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:46 smithi193 ceph-mon[37473]: pgmap v3643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5364.583643 secs 2024-07-26T12:56:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:56:47.685+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:56:47.685 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:56:47.685+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:56:47.686 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:56:47.685+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5364.583643 secs 2024-07-26T12:56:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5364.583643 secs 2024-07-26T12:56:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:48 smithi134 ceph-mon[31966]: pgmap v3644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:48 smithi189 ceph-mon[36790]: pgmap v3644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:48 smithi193 ceph-mon[37473]: pgmap v3644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:49.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5350.158785 secs 2024-07-26T12:56:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5350.158785 secs 2024-07-26T12:56:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5350.158785 secs 2024-07-26T12:56:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:50 smithi134 ceph-mon[31966]: pgmap v3645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:50 smithi189 ceph-mon[36790]: pgmap v3645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:50 smithi193 ceph-mon[37473]: pgmap v3645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:56:52.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:56:52.225+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:56:52.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:56:52.225+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:56:52.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:56:52.225+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:56:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5369.583764 secs 2024-07-26T12:56:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:52 smithi134 ceph-mon[31966]: pgmap v3646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5369.583764 secs 2024-07-26T12:56:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:52 smithi189 ceph-mon[36790]: pgmap v3646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5369.583764 secs 2024-07-26T12:56:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:52 smithi193 ceph-mon[37473]: pgmap v3646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:54 smithi134 ceph-mon[31966]: pgmap v3647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5355.158913 secs 2024-07-26T12:56:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:54 smithi189 ceph-mon[36790]: pgmap v3647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5355.158913 secs 2024-07-26T12:56:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:54 smithi193 ceph-mon[37473]: pgmap v3647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:56:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5355.158913 secs 2024-07-26T12:56:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:56 smithi134 ceph-mon[31966]: pgmap v3648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:56 smithi189 ceph-mon[36790]: pgmap v3648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:56 smithi193 ceph-mon[37473]: pgmap v3648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:56:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5374.583883 secs 2024-07-26T12:56:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5374.583883 secs 2024-07-26T12:56:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5374.583883 secs 2024-07-26T12:56:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:58 smithi134 ceph-mon[31966]: pgmap v3649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:58 smithi189 ceph-mon[36790]: pgmap v3649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:58 smithi193 ceph-mon[37473]: pgmap v3649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:56:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:56:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5360.159000 secs 2024-07-26T12:56:59.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:56:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5360.159000 secs 2024-07-26T12:56:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:56:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5360.159000 secs 2024-07-26T12:57:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:00 smithi134 ceph-mon[31966]: pgmap v3650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:00 smithi189 ceph-mon[36790]: pgmap v3650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:00 smithi193 ceph-mon[37473]: pgmap v3650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5379.583999 secs 2024-07-26T12:57:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:02 smithi134 ceph-mon[31966]: pgmap v3651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5379.583999 secs 2024-07-26T12:57:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:02 smithi189 ceph-mon[36790]: pgmap v3651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:02.700+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:57:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:02.700+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:57:02.701 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:02.700+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5379.583999 secs 2024-07-26T12:57:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:02 smithi193 ceph-mon[37473]: pgmap v3651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:04 smithi134 ceph-mon[31966]: pgmap v3652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5365.159109 secs 2024-07-26T12:57:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:04 smithi189 ceph-mon[36790]: pgmap v3652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5365.159109 secs 2024-07-26T12:57:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:04 smithi193 ceph-mon[37473]: pgmap v3652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5365.159109 secs 2024-07-26T12:57:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:06 smithi134 ceph-mon[31966]: pgmap v3653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:06 smithi189 ceph-mon[36790]: pgmap v3653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:06 smithi193 ceph-mon[37473]: pgmap v3653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:07.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:07.241+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:57:07.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:07.241+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:07.243 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:07.241+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:57:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5384.584116 secs 2024-07-26T12:57:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5384.584116 secs 2024-07-26T12:57:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5384.584116 secs 2024-07-26T12:57:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:08 smithi134 ceph-mon[31966]: pgmap v3654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:08 smithi189 ceph-mon[36790]: pgmap v3654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:08 smithi193 ceph-mon[37473]: pgmap v3654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5370.159188 secs 2024-07-26T12:57:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5370.159188 secs 2024-07-26T12:57:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5370.159188 secs 2024-07-26T12:57:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:10 smithi189 ceph-mon[36790]: pgmap v3655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:10 smithi193 ceph-mon[37473]: pgmap v3655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:10 smithi134 ceph-mon[31966]: pgmap v3655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5389.584248 secs 2024-07-26T12:57:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:12 smithi189 ceph-mon[36790]: pgmap v3656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5389.584248 secs 2024-07-26T12:57:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:12 smithi193 ceph-mon[37473]: pgmap v3656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5389.584248 secs 2024-07-26T12:57:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:12 smithi134 ceph-mon[31966]: pgmap v3656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:14 smithi189 ceph-mon[36790]: pgmap v3657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5375.159311 secs 2024-07-26T12:57:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:14 smithi193 ceph-mon[37473]: pgmap v3657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5375.159311 secs 2024-07-26T12:57:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:14 smithi134 ceph-mon[31966]: pgmap v3657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5375.159311 secs 2024-07-26T12:57:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:16 smithi193 ceph-mon[37473]: pgmap v3658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:16 smithi134 ceph-mon[31966]: pgmap v3658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:16 smithi189 ceph-mon[36790]: pgmap v3658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:17.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:17.715+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:57:17.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:17.715+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:57:17.717 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:17.715+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5394.584389 secs 2024-07-26T12:57:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:57:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5394.584389 secs 2024-07-26T12:57:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:57:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5394.584389 secs 2024-07-26T12:57:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:57:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:18 smithi193 ceph-mon[37473]: pgmap v3659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:57:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:57:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:57:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:18 smithi134 ceph-mon[31966]: pgmap v3659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:57:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:57:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:57:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:18 smithi189 ceph-mon[36790]: pgmap v3659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:57:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:57:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:57:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:19 smithi193 ceph-mon[37473]: pgmap v3660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:57:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:19 smithi193 ceph-mon[37473]: pgmap v3661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5380.159376 secs 2024-07-26T12:57:19.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:19 smithi189 ceph-mon[36790]: pgmap v3660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:57:19.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:19 smithi189 ceph-mon[36790]: pgmap v3661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:19.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5380.159376 secs 2024-07-26T12:57:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:19 smithi134 ceph-mon[31966]: pgmap v3660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:57:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:19 smithi134 ceph-mon[31966]: pgmap v3661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5380.159376 secs 2024-07-26T12:57:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:21 smithi193 ceph-mon[37473]: pgmap v3662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:21 smithi134 ceph-mon[31966]: pgmap v3662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:21 smithi189 ceph-mon[36790]: pgmap v3662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:57:22.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:22.257+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:57:22.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:22.257+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:22.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:22.257+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:57:22.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:22.302+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000997) for more than 900000000 us, fault. 2024-07-26T12:57:22.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:22.302+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b38008040 comp rx=0 tx=0).stop 2024-07-26T12:57:22.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:22.302+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:57:22.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:22.302+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:57:22.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:22.302+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0360 0x7f9b300c1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:57:22.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:22.303+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0360 0x7f9b300c1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:57:22.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:22.305+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0360 0x7f9b300c1530 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9b38013930 tx=0x7f9b3804e040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:57:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5399.584531 secs 2024-07-26T12:57:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5399.584531 secs 2024-07-26T12:57:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5399.584531 secs 2024-07-26T12:57:23.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:23.266+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T12:57:23.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:23.266+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T12:57:23.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:23.266+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T12:57:23.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:23.266+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T12:57:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:23 smithi134 ceph-mon[31966]: pgmap v3663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:57:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:23 smithi193 ceph-mon[37473]: pgmap v3663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:57:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:23 smithi189 ceph-mon[36790]: pgmap v3663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T12:57:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5385.159471 secs 2024-07-26T12:57:24.873 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5385.159471 secs 2024-07-26T12:57:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5385.159471 secs 2024-07-26T12:57:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:25 smithi193 ceph-mon[37473]: pgmap v3664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:57:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:25 smithi134 ceph-mon[31966]: pgmap v3664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:57:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:25 smithi189 ceph-mon[36790]: pgmap v3664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:57:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5404.584610 secs 2024-07-26T12:57:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:27 smithi193 ceph-mon[37473]: pgmap v3665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:57:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5404.584610 secs 2024-07-26T12:57:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:27 smithi134 ceph-mon[31966]: pgmap v3665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:57:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5404.584610 secs 2024-07-26T12:57:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:27 smithi189 ceph-mon[36790]: pgmap v3665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:57:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:29 smithi193 ceph-mon[37473]: pgmap v3666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5390.159594 secs 2024-07-26T12:57:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:29 smithi134 ceph-mon[31966]: pgmap v3666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5390.159594 secs 2024-07-26T12:57:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:29 smithi189 ceph-mon[36790]: pgmap v3666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:57:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5390.159594 secs 2024-07-26T12:57:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:31 smithi193 ceph-mon[37473]: pgmap v3667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:31 smithi134 ceph-mon[31966]: pgmap v3667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:31 smithi189 ceph-mon[36790]: pgmap v3667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:32.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:32.730+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:57:32.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:32.730+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:57:32.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:32.731+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:32.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5409.584680 secs 2024-07-26T12:57:32.857 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:32.855+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0ca0 msgr2=0x7f21ac0c10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T12:57:32.857 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:32.855+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0ca0 msgr2=0x7f21ac0c10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T12:57:32.857 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:32.855+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0ca0 0x7f21ac0c10b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f21b80c7030 tx=0x7f21b80d9000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T12:57:32.857 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:32.855+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0ca0 0x7f21ac0c10b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f21b80c7030 tx=0x7f21b80d9000 comp rx=0 tx=0).stop 2024-07-26T12:57:32.857 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:32.855+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0ca0 msgr2=0x7f21ac0c10b0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T12:57:32.857 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:32.856+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0ca0 msgr2=0x7f21ac0c10b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T12:57:32.858 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:32.856+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0ca0 0x7f21ac0c10b0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T12:57:32.858 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:32.856+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 0x7f21ac0c0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T12:57:32.858 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:32.857+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 0x7f21ac0c0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T12:57:32.859 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:32.858+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 0x7f21ac0c0760 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f21980063a0 tx=0x7f21b80c9000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T12:57:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5409.584680 secs 2024-07-26T12:57:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5409.584680 secs 2024-07-26T12:57:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:33.781+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:57:33.783 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:33.781+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:57:33.783 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:33.782+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:57:33.783 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:33.782+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:57:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:33 smithi134 ceph-mon[31966]: pgmap v3668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:33 smithi189 ceph-mon[36790]: pgmap v3668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:33 smithi193 ceph-mon[37473]: pgmap v3668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5395.159717 secs 2024-07-26T12:57:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5395.159717 secs 2024-07-26T12:57:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5395.159717 secs 2024-07-26T12:57:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:35 smithi134 ceph-mon[31966]: pgmap v3669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:35 smithi189 ceph-mon[36790]: pgmap v3669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:35 smithi193 ceph-mon[37473]: pgmap v3669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:37.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:37.268+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:57:37.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:37.268+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:57:37.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:37.268+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5414.584806 secs 2024-07-26T12:57:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:37 smithi134 ceph-mon[31966]: pgmap v3670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5414.584806 secs 2024-07-26T12:57:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:37 smithi189 ceph-mon[36790]: pgmap v3670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5414.584806 secs 2024-07-26T12:57:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:37 smithi193 ceph-mon[37473]: pgmap v3670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:39 smithi134 ceph-mon[31966]: pgmap v3671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5400.159856 secs 2024-07-26T12:57:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:39 smithi189 ceph-mon[36790]: pgmap v3671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5400.159856 secs 2024-07-26T12:57:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:39 smithi193 ceph-mon[37473]: pgmap v3671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5400.159856 secs 2024-07-26T12:57:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:41 smithi134 ceph-mon[31966]: pgmap v3672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:41 smithi189 ceph-mon[36790]: pgmap v3672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:41 smithi193 ceph-mon[37473]: pgmap v3672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5419.584925 secs 2024-07-26T12:57:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5419.584925 secs 2024-07-26T12:57:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5419.584925 secs 2024-07-26T12:57:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:43 smithi134 ceph-mon[31966]: pgmap v3673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:43 smithi189 ceph-mon[36790]: pgmap v3673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:43 smithi193 ceph-mon[37473]: pgmap v3673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:57:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5405.159987 secs 2024-07-26T12:57:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5405.159987 secs 2024-07-26T12:57:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5405.159987 secs 2024-07-26T12:57:45.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:45 smithi134 ceph-mon[31966]: pgmap v3674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:45 smithi189 ceph-mon[36790]: pgmap v3674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:45 smithi193 ceph-mon[37473]: pgmap v3674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:47.734+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:57:47.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:47.734+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:57:47.736 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:57:47.734+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5424.584988 secs 2024-07-26T12:57:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:47 smithi134 ceph-mon[31966]: pgmap v3675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5424.584988 secs 2024-07-26T12:57:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:47 smithi189 ceph-mon[36790]: pgmap v3675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5424.584988 secs 2024-07-26T12:57:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:47 smithi193 ceph-mon[37473]: pgmap v3675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:49 smithi134 ceph-mon[31966]: pgmap v3676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5410.160120 secs 2024-07-26T12:57:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:49 smithi189 ceph-mon[36790]: pgmap v3676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5410.160120 secs 2024-07-26T12:57:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:49 smithi193 ceph-mon[37473]: pgmap v3676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5410.160120 secs 2024-07-26T12:57:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:51 smithi134 ceph-mon[31966]: pgmap v3677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:51 smithi189 ceph-mon[36790]: pgmap v3677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:51 smithi193 ceph-mon[37473]: pgmap v3677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:52.285 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:52.284+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:57:52.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:52.284+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:57:52.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:57:52.284+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:57:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5429.585145 secs 2024-07-26T12:57:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5429.585145 secs 2024-07-26T12:57:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5429.585145 secs 2024-07-26T12:57:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:53 smithi134 ceph-mon[31966]: pgmap v3678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:57:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:53 smithi189 ceph-mon[36790]: pgmap v3678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:57:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:53 smithi193 ceph-mon[37473]: pgmap v3678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T12:57:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5415.160209 secs 2024-07-26T12:57:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5415.160209 secs 2024-07-26T12:57:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5415.160209 secs 2024-07-26T12:57:55.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:55 smithi134 ceph-mon[31966]: pgmap v3679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:55 smithi189 ceph-mon[36790]: pgmap v3679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:55 smithi193 ceph-mon[37473]: pgmap v3679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:57:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5434.585268 secs 2024-07-26T12:57:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:57 smithi134 ceph-mon[31966]: pgmap v3680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5434.585268 secs 2024-07-26T12:57:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:57 smithi189 ceph-mon[36790]: pgmap v3680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5434.585268 secs 2024-07-26T12:57:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:57 smithi193 ceph-mon[37473]: pgmap v3680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:57:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:59 smithi134 ceph-mon[31966]: pgmap v3681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:57:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5420.160329 secs 2024-07-26T12:57:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:59 smithi189 ceph-mon[36790]: pgmap v3681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:57:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:57:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5420.160329 secs 2024-07-26T12:58:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:59 smithi193 ceph-mon[37473]: pgmap v3681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:57:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5420.160329 secs 2024-07-26T12:58:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:01 smithi134 ceph-mon[31966]: pgmap v3682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:01 smithi189 ceph-mon[36790]: pgmap v3682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:01 smithi193 ceph-mon[37473]: pgmap v3682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:02.744 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:58:02.743+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:58:02.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:58:02.743+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:58:02.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:58:02.743+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5439.585397 secs 2024-07-26T12:58:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5439.585397 secs 2024-07-26T12:58:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5439.585397 secs 2024-07-26T12:58:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:03 smithi189 ceph-mon[36790]: pgmap v3683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:03 smithi193 ceph-mon[37473]: pgmap v3683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:03 smithi134 ceph-mon[31966]: pgmap v3683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5425.160450 secs 2024-07-26T12:58:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5425.160450 secs 2024-07-26T12:58:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5425.160450 secs 2024-07-26T12:58:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:05 smithi189 ceph-mon[36790]: pgmap v3684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:05 smithi193 ceph-mon[37473]: pgmap v3684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:05 smithi134 ceph-mon[31966]: pgmap v3684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:07.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:58:07.300+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:58:07.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:58:07.300+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:58:07.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:58:07.300+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5444.585524 secs 2024-07-26T12:58:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:07 smithi189 ceph-mon[36790]: pgmap v3685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5444.585524 secs 2024-07-26T12:58:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:07 smithi193 ceph-mon[37473]: pgmap v3685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5444.585524 secs 2024-07-26T12:58:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:07 smithi134 ceph-mon[31966]: pgmap v3685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:08.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:58:08.274+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T12:58:08.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:58:08.274+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T12:58:08.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:58:08.274+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T12:58:08.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:58:08.274+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T12:58:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:09 smithi193 ceph-mon[37473]: pgmap v3686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5430.160571 secs 2024-07-26T12:58:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:09 smithi134 ceph-mon[31966]: pgmap v3686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5430.160571 secs 2024-07-26T12:58:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:09 smithi189 ceph-mon[36790]: pgmap v3686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5430.160571 secs 2024-07-26T12:58:11.976 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:11 smithi134 ceph-mon[31966]: pgmap v3687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:11.977 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5449.585647 secs 2024-07-26T12:58:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:11 smithi193 ceph-mon[37473]: pgmap v3687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5449.585647 secs 2024-07-26T12:58:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:11 smithi189 ceph-mon[36790]: pgmap v3687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5449.585647 secs 2024-07-26T12:58:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:13 smithi193 ceph-mon[37473]: pgmap v3688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:14.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:13 smithi134 ceph-mon[31966]: pgmap v3688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:13 smithi189 ceph-mon[36790]: pgmap v3688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T12:58:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5435.160703 secs 2024-07-26T12:58:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5435.160703 secs 2024-07-26T12:58:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5435.160703 secs 2024-07-26T12:58:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:15 smithi193 ceph-mon[37473]: pgmap v3689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:16.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:15 smithi134 ceph-mon[31966]: pgmap v3689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:15 smithi189 ceph-mon[36790]: pgmap v3689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T12:58:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5454.585772 secs 2024-07-26T12:58:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5454.585772 secs 2024-07-26T12:58:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5454.585772 secs 2024-07-26T12:58:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:58:17.745+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:58:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:58:17.745+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:58:17.747 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:58:17.745+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:17 smithi193 ceph-mon[37473]: pgmap v3690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:17 smithi134 ceph-mon[31966]: pgmap v3690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:17 smithi189 ceph-mon[36790]: pgmap v3690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:18.791 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:58:18.789+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:58:18.791 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:58:18.789+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:58:18.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:58:18.790+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T12:58:18.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:58:18.790+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T12:58:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:58:19.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:58:19.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:58:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:19 smithi193 ceph-mon[37473]: pgmap v3691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5440.160839 secs 2024-07-26T12:58:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:58:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:58:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:58:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:19 smithi134 ceph-mon[31966]: pgmap v3691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5440.160839 secs 2024-07-26T12:58:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:58:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:58:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:58:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:19 smithi189 ceph-mon[36790]: pgmap v3691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5440.160839 secs 2024-07-26T12:58:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:58:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:58:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:58:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:20 smithi193 ceph-mon[37473]: pgmap v3692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:58:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:20 smithi193 ceph-mon[37473]: pgmap v3693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:58:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:20 smithi134 ceph-mon[31966]: pgmap v3692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:58:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:20 smithi134 ceph-mon[31966]: pgmap v3693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:58:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:20 smithi189 ceph-mon[36790]: pgmap v3692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:58:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:20 smithi189 ceph-mon[36790]: pgmap v3693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:58:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5459.585895 secs 2024-07-26T12:58:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5459.585895 secs 2024-07-26T12:58:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5459.585895 secs 2024-07-26T12:58:22.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:58:22.315+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:58:22.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:58:22.315+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:58:22.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:58:22.315+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:22 smithi193 ceph-mon[37473]: pgmap v3694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:58:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:22 smithi134 ceph-mon[31966]: pgmap v3694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:58:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:22 smithi189 ceph-mon[36790]: pgmap v3694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T12:58:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5445.160972 secs 2024-07-26T12:58:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:24 smithi134 ceph-mon[31966]: pgmap v3695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:58:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5445.160972 secs 2024-07-26T12:58:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:24 smithi189 ceph-mon[36790]: pgmap v3695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:58:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5445.160972 secs 2024-07-26T12:58:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:24 smithi193 ceph-mon[37473]: pgmap v3695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:58:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:26 smithi134 ceph-mon[31966]: pgmap v3696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:58:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5464.586020 secs 2024-07-26T12:58:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:26 smithi189 ceph-mon[36790]: pgmap v3696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:58:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5464.586020 secs 2024-07-26T12:58:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:26 smithi193 ceph-mon[37473]: pgmap v3696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:58:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5464.586020 secs 2024-07-26T12:58:29.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:28 smithi193 ceph-mon[37473]: pgmap v3697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:58:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:28 smithi134 ceph-mon[31966]: pgmap v3697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:58:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:28 smithi189 ceph-mon[36790]: pgmap v3697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:58:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5450.161061 secs 2024-07-26T12:58:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5450.161061 secs 2024-07-26T12:58:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5450.161061 secs 2024-07-26T12:58:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:30 smithi134 ceph-mon[31966]: pgmap v3698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:58:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:30 smithi189 ceph-mon[36790]: pgmap v3698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:58:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:30 smithi193 ceph-mon[37473]: pgmap v3698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:58:32.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5469.586149 secs 2024-07-26T12:58:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5469.586149 secs 2024-07-26T12:58:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5469.586149 secs 2024-07-26T12:58:32.762 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:58:32.761+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:58:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:58:32.761+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:58:32.763 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:58:32.761+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:32 smithi134 ceph-mon[31966]: pgmap v3699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:32 smithi189 ceph-mon[36790]: pgmap v3699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:32 smithi193 ceph-mon[37473]: pgmap v3699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5455.161180 secs 2024-07-26T12:58:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:34 smithi134 ceph-mon[31966]: pgmap v3700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5455.161180 secs 2024-07-26T12:58:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:34 smithi189 ceph-mon[36790]: pgmap v3700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5455.161180 secs 2024-07-26T12:58:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:34 smithi193 ceph-mon[37473]: pgmap v3700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:36 smithi134 ceph-mon[31966]: pgmap v3701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5474.586274 secs 2024-07-26T12:58:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:36 smithi189 ceph-mon[36790]: pgmap v3701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5474.586274 secs 2024-07-26T12:58:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:36 smithi193 ceph-mon[37473]: pgmap v3701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5474.586274 secs 2024-07-26T12:58:37.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:58:37.319+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:58:37.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:58:37.319+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:37.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:58:37.319+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:58:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:38 smithi134 ceph-mon[31966]: pgmap v3702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:38 smithi189 ceph-mon[36790]: pgmap v3702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:38 smithi193 ceph-mon[37473]: pgmap v3702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5460.161301 secs 2024-07-26T12:58:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5460.161301 secs 2024-07-26T12:58:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5460.161301 secs 2024-07-26T12:58:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:40 smithi134 ceph-mon[31966]: pgmap v3703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:40 smithi189 ceph-mon[36790]: pgmap v3703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:40 smithi193 ceph-mon[37473]: pgmap v3703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5479.586364 secs 2024-07-26T12:58:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5479.586364 secs 2024-07-26T12:58:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5479.586364 secs 2024-07-26T12:58:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:42 smithi134 ceph-mon[31966]: pgmap v3704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:42 smithi189 ceph-mon[36790]: pgmap v3704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:42 smithi193 ceph-mon[37473]: pgmap v3704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5465.161417 secs 2024-07-26T12:58:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:44 smithi134 ceph-mon[31966]: pgmap v3705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5465.161417 secs 2024-07-26T12:58:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:44 smithi189 ceph-mon[36790]: pgmap v3705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5465.161417 secs 2024-07-26T12:58:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:44 smithi193 ceph-mon[37473]: pgmap v3705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:46 smithi134 ceph-mon[31966]: pgmap v3706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5484.586494 secs 2024-07-26T12:58:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:46 smithi189 ceph-mon[36790]: pgmap v3706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5484.586494 secs 2024-07-26T12:58:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:46 smithi193 ceph-mon[37473]: pgmap v3706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5484.586494 secs 2024-07-26T12:58:47.778 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:58:47.777+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:58:47.778 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:58:47.777+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:47.778 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:58:47.777+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:58:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:48 smithi134 ceph-mon[31966]: pgmap v3707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:48 smithi189 ceph-mon[36790]: pgmap v3707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:48 smithi193 ceph-mon[37473]: pgmap v3707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5470.161528 secs 2024-07-26T12:58:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5470.161528 secs 2024-07-26T12:58:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5470.161528 secs 2024-07-26T12:58:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:50 smithi134 ceph-mon[31966]: pgmap v3708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:50 smithi189 ceph-mon[36790]: pgmap v3708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:50 smithi193 ceph-mon[37473]: pgmap v3708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:58:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5489.586565 secs 2024-07-26T12:58:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5489.586565 secs 2024-07-26T12:58:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5489.586565 secs 2024-07-26T12:58:52.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:58:52.324+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:58:52.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:58:52.324+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:58:52.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:58:52.324+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:58:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:52 smithi134 ceph-mon[31966]: pgmap v3709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:52 smithi189 ceph-mon[36790]: pgmap v3709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:52 smithi193 ceph-mon[37473]: pgmap v3709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5475.161639 secs 2024-07-26T12:58:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:54 smithi134 ceph-mon[31966]: pgmap v3710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5475.161639 secs 2024-07-26T12:58:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:54 smithi189 ceph-mon[36790]: pgmap v3710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5475.161639 secs 2024-07-26T12:58:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:54 smithi193 ceph-mon[37473]: pgmap v3710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:56 smithi134 ceph-mon[31966]: pgmap v3711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5494.586629 secs 2024-07-26T12:58:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:56 smithi189 ceph-mon[36790]: pgmap v3711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5494.586629 secs 2024-07-26T12:58:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:56 smithi193 ceph-mon[37473]: pgmap v3711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:58:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5494.586629 secs 2024-07-26T12:58:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:58 smithi189 ceph-mon[36790]: pgmap v3712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:58 smithi193 ceph-mon[37473]: pgmap v3712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:58:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:58 smithi134 ceph-mon[31966]: pgmap v3712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:58:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5480.161784 secs 2024-07-26T12:59:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:58:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5480.161784 secs 2024-07-26T12:59:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:58:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5480.161784 secs 2024-07-26T12:59:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:00 smithi189 ceph-mon[36790]: pgmap v3713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:00 smithi193 ceph-mon[37473]: pgmap v3713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:01.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:00 smithi134 ceph-mon[31966]: pgmap v3713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5499.586730 secs 2024-07-26T12:59:02.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5499.586730 secs 2024-07-26T12:59:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5499.586730 secs 2024-07-26T12:59:02.794 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:59:02.792+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:59:02.794 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:59:02.792+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:59:02.794 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:59:02.792+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:02 smithi193 ceph-mon[37473]: pgmap v3714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:02 smithi134 ceph-mon[31966]: pgmap v3714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:02 smithi189 ceph-mon[36790]: pgmap v3714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5485.161917 secs 2024-07-26T12:59:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:04 smithi193 ceph-mon[37473]: pgmap v3715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5485.161917 secs 2024-07-26T12:59:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:04 smithi134 ceph-mon[31966]: pgmap v3715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5485.161917 secs 2024-07-26T12:59:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:04 smithi189 ceph-mon[36790]: pgmap v3715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:06 smithi193 ceph-mon[37473]: pgmap v3716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5504.586844 secs 2024-07-26T12:59:07.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:59:07.339+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:59:07.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:59:07.340+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:59:07.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:59:07.340+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:07.342 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:06 smithi134 ceph-mon[31966]: pgmap v3716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:07.342 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5504.586844 secs 2024-07-26T12:59:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:06 smithi189 ceph-mon[36790]: pgmap v3716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5504.586844 secs 2024-07-26T12:59:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:08 smithi193 ceph-mon[37473]: pgmap v3717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:08 smithi134 ceph-mon[31966]: pgmap v3717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:08 smithi189 ceph-mon[36790]: pgmap v3717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5490.162025 secs 2024-07-26T12:59:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5490.162025 secs 2024-07-26T12:59:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5490.162025 secs 2024-07-26T12:59:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:11 smithi193 ceph-mon[37473]: pgmap v3718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:11 smithi134 ceph-mon[31966]: pgmap v3718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:11 smithi189 ceph-mon[36790]: pgmap v3718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5509.586972 secs 2024-07-26T12:59:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5509.586972 secs 2024-07-26T12:59:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5509.586972 secs 2024-07-26T12:59:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:13 smithi193 ceph-mon[37473]: pgmap v3719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:13 smithi134 ceph-mon[31966]: pgmap v3719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:13 smithi189 ceph-mon[36790]: pgmap v3719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5495.162145 secs 2024-07-26T12:59:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:15 smithi134 ceph-mon[31966]: pgmap v3720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5495.162145 secs 2024-07-26T12:59:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:15 smithi189 ceph-mon[36790]: pgmap v3720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5495.162145 secs 2024-07-26T12:59:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:15 smithi193 ceph-mon[37473]: pgmap v3720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:17 smithi134 ceph-mon[31966]: pgmap v3721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5514.587130 secs 2024-07-26T12:59:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:17 smithi189 ceph-mon[36790]: pgmap v3721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5514.587130 secs 2024-07-26T12:59:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:17 smithi193 ceph-mon[37473]: pgmap v3721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5514.587130 secs 2024-07-26T12:59:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:59:17.803+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:59:17.804 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:59:17.803+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:59:17.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:59:17.803+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:19 smithi134 ceph-mon[31966]: pgmap v3722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:19 smithi189 ceph-mon[36790]: pgmap v3722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:19 smithi193 ceph-mon[37473]: pgmap v3722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:20.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5500.162257 secs 2024-07-26T12:59:20.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:59:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5500.162257 secs 2024-07-26T12:59:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:59:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5500.162257 secs 2024-07-26T12:59:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T12:59:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:21 smithi134 ceph-mon[31966]: pgmap v3723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:59:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:59:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:59:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:21 smithi189 ceph-mon[36790]: pgmap v3723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:59:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:59:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:59:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:21 smithi193 ceph-mon[37473]: pgmap v3723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T12:59:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T12:59:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T12:59:22.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:59:22.353+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:59:22.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:59:22.353+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:59:22.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:59:22.353+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:22.356 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:22 smithi134 ceph-mon[31966]: pgmap v3724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:59:22.356 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:22 smithi134 ceph-mon[31966]: pgmap v3725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:59:22.356 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5519.587210 secs 2024-07-26T12:59:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:22 smithi189 ceph-mon[36790]: pgmap v3724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:59:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:22 smithi189 ceph-mon[36790]: pgmap v3725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:59:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5519.587210 secs 2024-07-26T12:59:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:22 smithi193 ceph-mon[37473]: pgmap v3724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T12:59:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:22 smithi193 ceph-mon[37473]: pgmap v3725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:59:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5519.587210 secs 2024-07-26T12:59:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:24 smithi134 ceph-mon[31966]: pgmap v3726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:59:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:24 smithi189 ceph-mon[36790]: pgmap v3726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:59:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:24 smithi193 ceph-mon[37473]: pgmap v3726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:59:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5505.162399 secs 2024-07-26T12:59:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:25 smithi134 ceph-mon[31966]: pgmap v3727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:59:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5505.162399 secs 2024-07-26T12:59:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:25 smithi189 ceph-mon[36790]: pgmap v3727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:59:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5505.162399 secs 2024-07-26T12:59:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:25 smithi193 ceph-mon[37473]: pgmap v3727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T12:59:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5524.587368 secs 2024-07-26T12:59:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5524.587368 secs 2024-07-26T12:59:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5524.587368 secs 2024-07-26T12:59:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:27 smithi134 ceph-mon[31966]: pgmap v3728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:59:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:27 smithi189 ceph-mon[36790]: pgmap v3728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:59:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:27 smithi193 ceph-mon[37473]: pgmap v3728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T12:59:30.075 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5510.162514 secs 2024-07-26T12:59:30.075 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:29 smithi193 ceph-mon[37473]: pgmap v3729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:59:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5510.162514 secs 2024-07-26T12:59:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:29 smithi134 ceph-mon[31966]: pgmap v3729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:59:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5510.162514 secs 2024-07-26T12:59:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:29 smithi189 ceph-mon[36790]: pgmap v3729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T12:59:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:31 smithi134 ceph-mon[31966]: pgmap v3730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:59:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5529.587500 secs 2024-07-26T12:59:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:31 smithi189 ceph-mon[36790]: pgmap v3730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:59:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5529.587500 secs 2024-07-26T12:59:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:31 smithi193 ceph-mon[37473]: pgmap v3730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T12:59:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5529.587500 secs 2024-07-26T12:59:32.819 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:59:32.818+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:59:32.820 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:59:32.818+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:59:32.820 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:59:32.818+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:33 smithi134 ceph-mon[31966]: pgmap v3731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:33 smithi189 ceph-mon[36790]: pgmap v3731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:33 smithi193 ceph-mon[37473]: pgmap v3731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5515.162633 secs 2024-07-26T12:59:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5515.162633 secs 2024-07-26T12:59:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5515.162633 secs 2024-07-26T12:59:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:35 smithi134 ceph-mon[31966]: pgmap v3732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:35 smithi189 ceph-mon[36790]: pgmap v3732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:35 smithi193 ceph-mon[37473]: pgmap v3732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5534.587625 secs 2024-07-26T12:59:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5534.587625 secs 2024-07-26T12:59:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5534.587625 secs 2024-07-26T12:59:37.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:59:37.364+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:59:37.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:59:37.364+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:37.366 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:59:37.364+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:59:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:37 smithi134 ceph-mon[31966]: pgmap v3733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:37 smithi189 ceph-mon[36790]: pgmap v3733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:37 smithi193 ceph-mon[37473]: pgmap v3733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5520.162747 secs 2024-07-26T12:59:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:39 smithi134 ceph-mon[31966]: pgmap v3734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5520.162747 secs 2024-07-26T12:59:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:39 smithi189 ceph-mon[36790]: pgmap v3734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5520.162747 secs 2024-07-26T12:59:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:39 smithi193 ceph-mon[37473]: pgmap v3734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:41 smithi134 ceph-mon[31966]: pgmap v3735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5539.587755 secs 2024-07-26T12:59:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:41 smithi189 ceph-mon[36790]: pgmap v3735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5539.587755 secs 2024-07-26T12:59:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:41 smithi193 ceph-mon[37473]: pgmap v3735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5539.587755 secs 2024-07-26T12:59:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:43 smithi134 ceph-mon[31966]: pgmap v3736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:43 smithi189 ceph-mon[36790]: pgmap v3736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:43 smithi193 ceph-mon[37473]: pgmap v3736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T12:59:45.163 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5525.162880 secs 2024-07-26T12:59:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5525.162880 secs 2024-07-26T12:59:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5525.162880 secs 2024-07-26T12:59:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:45 smithi134 ceph-mon[31966]: pgmap v3737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:45 smithi189 ceph-mon[36790]: pgmap v3737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:45 smithi193 ceph-mon[37473]: pgmap v3737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T12:59:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5544.587916 secs 2024-07-26T12:59:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5544.587916 secs 2024-07-26T12:59:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5544.587916 secs 2024-07-26T12:59:47.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:59:47.832+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:59:47.834 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:59:47.832+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:59:47.834 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T12:59:47.832+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:47 smithi134 ceph-mon[31966]: pgmap v3738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:47 smithi189 ceph-mon[36790]: pgmap v3738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:47 smithi193 ceph-mon[37473]: pgmap v3738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T12:59:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5530.163033 secs 2024-07-26T12:59:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:49 smithi134 ceph-mon[31966]: pgmap v3739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:59:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5530.163033 secs 2024-07-26T12:59:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:49 smithi189 ceph-mon[36790]: pgmap v3739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:59:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5530.163033 secs 2024-07-26T12:59:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:49 smithi193 ceph-mon[37473]: pgmap v3739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:59:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:51 smithi134 ceph-mon[31966]: pgmap v3740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:59:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5549.588060 secs 2024-07-26T12:59:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:51 smithi189 ceph-mon[36790]: pgmap v3740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:59:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5549.588060 secs 2024-07-26T12:59:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:51 smithi193 ceph-mon[37473]: pgmap v3740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:59:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5549.588060 secs 2024-07-26T12:59:52.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:59:52.379+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T12:59:52.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:59:52.380+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T12:59:52.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T12:59:52.380+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T12:59:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:53 smithi134 ceph-mon[31966]: pgmap v3741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:59:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:53 smithi189 ceph-mon[36790]: pgmap v3741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:59:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:53 smithi193 ceph-mon[37473]: pgmap v3741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T12:59:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5535.163147 secs 2024-07-26T12:59:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5535.163147 secs 2024-07-26T12:59:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5535.163147 secs 2024-07-26T12:59:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:55 smithi134 ceph-mon[31966]: pgmap v3742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:59:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:55 smithi189 ceph-mon[36790]: pgmap v3742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:59:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:55 smithi193 ceph-mon[37473]: pgmap v3742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T12:59:57.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5554.588190 secs 2024-07-26T12:59:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5554.588190 secs 2024-07-26T12:59:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5554.588190 secs 2024-07-26T12:59:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:57 smithi189 ceph-mon[36790]: pgmap v3743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:59:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:57 smithi193 ceph-mon[37473]: pgmap v3743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T12:59:58.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:57 smithi134 ceph-mon[31966]: pgmap v3743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:00:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5540.163272 secs 2024-07-26T13:00:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 12:59:59 smithi189 ceph-mon[36790]: pgmap v3744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:00:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5540.163272 secs 2024-07-26T13:00:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 12:59:59 smithi193 ceph-mon[37473]: pgmap v3744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:00:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5540.163272 secs 2024-07-26T13:00:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 12:59:59 smithi134 ceph-mon[31966]: pgmap v3744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:00:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:00:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:00:01.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:00:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:01 smithi189 ceph-mon[36790]: pgmap v3745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5559.588334 secs 2024-07-26T13:00:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:01 smithi193 ceph-mon[37473]: pgmap v3745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5559.588334 secs 2024-07-26T13:00:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:01 smithi134 ceph-mon[31966]: pgmap v3745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5559.588334 secs 2024-07-26T13:00:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:00:02.848+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:00:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:00:02.848+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:00:02.849 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:00:02.848+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:04.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:03 smithi134 ceph-mon[31966]: pgmap v3746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:03 smithi193 ceph-mon[37473]: pgmap v3746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:03 smithi189 ceph-mon[36790]: pgmap v3746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5545.163396 secs 2024-07-26T13:00:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5545.163396 secs 2024-07-26T13:00:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5545.163396 secs 2024-07-26T13:00:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:05 smithi193 ceph-mon[37473]: pgmap v3747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:05 smithi134 ceph-mon[31966]: pgmap v3747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:05 smithi189 ceph-mon[36790]: pgmap v3747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5564.588456 secs 2024-07-26T13:00:07.395 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5564.588456 secs 2024-07-26T13:00:07.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:00:07.392+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:00:07.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:00:07.392+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:07.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:00:07.392+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:00:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5564.588456 secs 2024-07-26T13:00:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:07 smithi193 ceph-mon[37473]: pgmap v3748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:07 smithi134 ceph-mon[31966]: pgmap v3748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:07 smithi189 ceph-mon[36790]: pgmap v3748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5550.163518 secs 2024-07-26T13:00:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:09 smithi193 ceph-mon[37473]: pgmap v3749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5550.163518 secs 2024-07-26T13:00:10.445 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:09 smithi134 ceph-mon[31966]: pgmap v3749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:10.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5550.163518 secs 2024-07-26T13:00:10.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:09 smithi189 ceph-mon[36790]: pgmap v3749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:11 smithi193 ceph-mon[37473]: pgmap v3750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5569.588573 secs 2024-07-26T13:00:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:11 smithi134 ceph-mon[31966]: pgmap v3750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5569.588573 secs 2024-07-26T13:00:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:11 smithi189 ceph-mon[36790]: pgmap v3750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5569.588573 secs 2024-07-26T13:00:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:13 smithi193 ceph-mon[37473]: pgmap v3751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:13 smithi134 ceph-mon[31966]: pgmap v3751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:13 smithi189 ceph-mon[36790]: pgmap v3751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5555.163640 secs 2024-07-26T13:00:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5555.163640 secs 2024-07-26T13:00:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5555.163640 secs 2024-07-26T13:00:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:16 smithi134 ceph-mon[31966]: pgmap v3752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:16 smithi193 ceph-mon[37473]: pgmap v3752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:16 smithi189 ceph-mon[36790]: pgmap v3752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5574.588695 secs 2024-07-26T13:00:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5574.588695 secs 2024-07-26T13:00:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5574.588695 secs 2024-07-26T13:00:17.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:00:17.864+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:00:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:00:17.864+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:00:17.865 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:00:17.864+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:18.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:18 smithi134 ceph-mon[31966]: pgmap v3753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:18 smithi189 ceph-mon[36790]: pgmap v3753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:18 smithi193 ceph-mon[37473]: pgmap v3753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5560.163745 secs 2024-07-26T13:00:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:20 smithi134 ceph-mon[31966]: pgmap v3754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5560.163745 secs 2024-07-26T13:00:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:20 smithi189 ceph-mon[36790]: pgmap v3754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5560.163745 secs 2024-07-26T13:00:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:20 smithi193 ceph-mon[37473]: pgmap v3754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:21.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:00:21.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:00:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:00:22.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:00:22.407+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:00:22.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:00:22.408+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:00:22.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:00:22.408+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:22.410 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:22 smithi134 ceph-mon[31966]: pgmap v3755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:22.410 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5579.588845 secs 2024-07-26T13:00:22.410 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:00:22.410 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:00:22.410 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:00:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:22 smithi189 ceph-mon[36790]: pgmap v3755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5579.588845 secs 2024-07-26T13:00:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:00:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:00:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:00:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:22 smithi193 ceph-mon[37473]: pgmap v3755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5579.588845 secs 2024-07-26T13:00:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:00:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:00:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:00:23.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:00:23.296+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T13:00:23.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:00:23.296+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T13:00:23.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:00:23.297+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T13:00:23.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:00:23.297+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T13:00:23.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:23 smithi134 ceph-mon[31966]: pgmap v3756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:00:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:23 smithi134 ceph-mon[31966]: pgmap v3757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T13:00:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:23 smithi189 ceph-mon[36790]: pgmap v3756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:00:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:23 smithi189 ceph-mon[36790]: pgmap v3757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T13:00:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:23 smithi193 ceph-mon[37473]: pgmap v3756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:00:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:23 smithi193 ceph-mon[37473]: pgmap v3757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T13:00:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:24 smithi134 ceph-mon[31966]: pgmap v3758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:24 smithi189 ceph-mon[36790]: pgmap v3758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:24 smithi193 ceph-mon[37473]: pgmap v3758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5565.163860 secs 2024-07-26T13:00:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5565.163860 secs 2024-07-26T13:00:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5565.163860 secs 2024-07-26T13:00:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:26 smithi134 ceph-mon[31966]: pgmap v3759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:26 smithi189 ceph-mon[36790]: pgmap v3759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:26 smithi193 ceph-mon[37473]: pgmap v3759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5584.588966 secs 2024-07-26T13:00:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5584.588966 secs 2024-07-26T13:00:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5584.588966 secs 2024-07-26T13:00:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:28 smithi134 ceph-mon[31966]: pgmap v3760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:28 smithi189 ceph-mon[36790]: pgmap v3760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:28 smithi193 ceph-mon[37473]: pgmap v3760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:00:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5570.163980 secs 2024-07-26T13:00:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5570.163980 secs 2024-07-26T13:00:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5570.163980 secs 2024-07-26T13:00:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:30 smithi193 ceph-mon[37473]: pgmap v3761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:00:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:30 smithi134 ceph-mon[31966]: pgmap v3761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:00:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:30 smithi189 ceph-mon[36790]: pgmap v3761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:00:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5589.589052 secs 2024-07-26T13:00:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5589.589052 secs 2024-07-26T13:00:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5589.589052 secs 2024-07-26T13:00:32.880 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:00:32.879+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:00:32.880 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:00:32.879+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:00:32.880 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:00:32.879+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:32 smithi134 ceph-mon[31966]: pgmap v3762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:00:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:32 smithi189 ceph-mon[36790]: pgmap v3762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:00:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:32 smithi193 ceph-mon[37473]: pgmap v3762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:00:33.814 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:00:33.813+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T13:00:33.814 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:00:33.813+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:00:33.815 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:00:33.814+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:00:33.815 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:00:33.814+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T13:00:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5575.164137 secs 2024-07-26T13:00:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:35 smithi134 ceph-mon[31966]: pgmap v3763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5575.164137 secs 2024-07-26T13:00:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:35 smithi189 ceph-mon[36790]: pgmap v3763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5575.164137 secs 2024-07-26T13:00:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:35 smithi193 ceph-mon[37473]: pgmap v3763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:37.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:00:37.412+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:00:37.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:00:37.412+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:00:37.414 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:00:37.412+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:37.414 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5594.589176 secs 2024-07-26T13:00:37.415 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:37 smithi134 ceph-mon[31966]: pgmap v3764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5594.589176 secs 2024-07-26T13:00:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:37 smithi189 ceph-mon[36790]: pgmap v3764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5594.589176 secs 2024-07-26T13:00:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:37 smithi193 ceph-mon[37473]: pgmap v3764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:39 smithi134 ceph-mon[31966]: pgmap v3765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:39 smithi189 ceph-mon[36790]: pgmap v3765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:39 smithi193 ceph-mon[37473]: pgmap v3765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5580.164203 secs 2024-07-26T13:00:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5580.164203 secs 2024-07-26T13:00:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5580.164203 secs 2024-07-26T13:00:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:41 smithi134 ceph-mon[31966]: pgmap v3766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:41 smithi189 ceph-mon[36790]: pgmap v3766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:41 smithi193 ceph-mon[37473]: pgmap v3766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5599.589333 secs 2024-07-26T13:00:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5599.589333 secs 2024-07-26T13:00:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5599.589333 secs 2024-07-26T13:00:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:43 smithi134 ceph-mon[31966]: pgmap v3767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:43 smithi189 ceph-mon[36790]: pgmap v3767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:43 smithi193 ceph-mon[37473]: pgmap v3767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:44 smithi134 ceph-mon[31966]: pgmap v3768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:44 smithi189 ceph-mon[36790]: pgmap v3768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:44 smithi193 ceph-mon[37473]: pgmap v3768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5585.164354 secs 2024-07-26T13:00:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5585.164354 secs 2024-07-26T13:00:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5585.164354 secs 2024-07-26T13:00:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:46 smithi134 ceph-mon[31966]: pgmap v3769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:46 smithi189 ceph-mon[36790]: pgmap v3769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:46 smithi193 ceph-mon[37473]: pgmap v3769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5604.589489 secs 2024-07-26T13:00:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5604.589489 secs 2024-07-26T13:00:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5604.589489 secs 2024-07-26T13:00:47.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:00:47.894+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:00:47.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:00:47.894+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:00:47.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:00:47.894+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:48 smithi134 ceph-mon[31966]: pgmap v3770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:48 smithi189 ceph-mon[36790]: pgmap v3770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:48 smithi193 ceph-mon[37473]: pgmap v3770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5590.164473 secs 2024-07-26T13:00:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5590.164473 secs 2024-07-26T13:00:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5590.164473 secs 2024-07-26T13:00:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:50 smithi193 ceph-mon[37473]: pgmap v3771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:50 smithi134 ceph-mon[31966]: pgmap v3771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:50 smithi189 ceph-mon[36790]: pgmap v3771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:00:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5609.589639 secs 2024-07-26T13:00:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5609.589639 secs 2024-07-26T13:00:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5609.589639 secs 2024-07-26T13:00:52.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:00:52.428+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:00:52.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:00:52.428+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:00:52.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:00:52.428+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:00:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:52 smithi134 ceph-mon[31966]: pgmap v3772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:52 smithi189 ceph-mon[36790]: pgmap v3772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:52 smithi193 ceph-mon[37473]: pgmap v3772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:00:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5595.164606 secs 2024-07-26T13:00:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:55 smithi134 ceph-mon[31966]: pgmap v3773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5595.164606 secs 2024-07-26T13:00:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:55 smithi189 ceph-mon[36790]: pgmap v3773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5595.164606 secs 2024-07-26T13:00:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:55 smithi193 ceph-mon[37473]: pgmap v3773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:00:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5614.589764 secs 2024-07-26T13:00:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:57 smithi134 ceph-mon[31966]: pgmap v3774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5614.589764 secs 2024-07-26T13:00:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:57 smithi189 ceph-mon[36790]: pgmap v3774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5614.589764 secs 2024-07-26T13:00:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:57 smithi193 ceph-mon[37473]: pgmap v3774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:00:59.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:00:59 smithi134 ceph-mon[31966]: pgmap v3775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:00:59 smithi189 ceph-mon[36790]: pgmap v3775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:00:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:00:59 smithi193 ceph-mon[37473]: pgmap v3775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5600.164748 secs 2024-07-26T13:01:00.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5600.164748 secs 2024-07-26T13:01:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5600.164748 secs 2024-07-26T13:01:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:01 smithi134 ceph-mon[31966]: pgmap v3776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:01:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:01 smithi189 ceph-mon[36790]: pgmap v3776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:01:01.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:01 smithi193 ceph-mon[37473]: pgmap v3776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:01:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5619.589908 secs 2024-07-26T13:01:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:02 smithi134 ceph-mon[31966]: pgmap v3777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5619.589908 secs 2024-07-26T13:01:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:02 smithi189 ceph-mon[36790]: pgmap v3777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5619.589908 secs 2024-07-26T13:01:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:02 smithi193 ceph-mon[37473]: pgmap v3777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:01:02.910+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:01:02.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:01:02.910+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:01:02.912 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:01:02.910+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5605.164867 secs 2024-07-26T13:01:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:05 smithi134 ceph-mon[31966]: pgmap v3778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5605.164867 secs 2024-07-26T13:01:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:05 smithi189 ceph-mon[36790]: pgmap v3778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5605.164867 secs 2024-07-26T13:01:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:05 smithi193 ceph-mon[37473]: pgmap v3778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5624.590036 secs 2024-07-26T13:01:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:07 smithi134 ceph-mon[31966]: pgmap v3779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5624.590036 secs 2024-07-26T13:01:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:07 smithi189 ceph-mon[36790]: pgmap v3779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:07.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:01:07.442+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:01:07.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:01:07.442+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:01:07.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:01:07.442+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5624.590036 secs 2024-07-26T13:01:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:07 smithi193 ceph-mon[37473]: pgmap v3779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:09 smithi134 ceph-mon[31966]: pgmap v3780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:09 smithi189 ceph-mon[36790]: pgmap v3780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:09 smithi193 ceph-mon[37473]: pgmap v3780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5610.164996 secs 2024-07-26T13:01:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5610.164996 secs 2024-07-26T13:01:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5610.164996 secs 2024-07-26T13:01:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:11 smithi134 ceph-mon[31966]: pgmap v3781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:01:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:11 smithi189 ceph-mon[36790]: pgmap v3781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:01:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:11 smithi193 ceph-mon[37473]: pgmap v3781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:01:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5629.590175 secs 2024-07-26T13:01:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:12 smithi134 ceph-mon[31966]: pgmap v3782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5629.590175 secs 2024-07-26T13:01:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:12 smithi189 ceph-mon[36790]: pgmap v3782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5629.590175 secs 2024-07-26T13:01:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:12 smithi193 ceph-mon[37473]: pgmap v3782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5615.165115 secs 2024-07-26T13:01:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:15 smithi134 ceph-mon[31966]: pgmap v3783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5615.165115 secs 2024-07-26T13:01:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:15 smithi189 ceph-mon[36790]: pgmap v3783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5615.165115 secs 2024-07-26T13:01:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:15 smithi193 ceph-mon[37473]: pgmap v3783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5634.590353 secs 2024-07-26T13:01:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:17 smithi134 ceph-mon[31966]: pgmap v3784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5634.590353 secs 2024-07-26T13:01:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:17 smithi189 ceph-mon[36790]: pgmap v3784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5634.590353 secs 2024-07-26T13:01:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:17 smithi193 ceph-mon[37473]: pgmap v3784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:17.926 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:01:17.925+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:01:17.927 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:01:17.925+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:17.927 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:01:17.925+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:01:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:19 smithi134 ceph-mon[31966]: pgmap v3785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:19 smithi189 ceph-mon[36790]: pgmap v3785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:19 smithi193 ceph-mon[37473]: pgmap v3785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5620.165242 secs 2024-07-26T13:01:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5620.165242 secs 2024-07-26T13:01:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5620.165242 secs 2024-07-26T13:01:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:21 smithi134 ceph-mon[31966]: pgmap v3786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:01:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:21 smithi189 ceph-mon[36790]: pgmap v3786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:01:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:21 smithi193 ceph-mon[37473]: pgmap v3786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:01:22.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5639.590481 secs 2024-07-26T13:01:22.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:22 smithi189 ceph-mon[36790]: pgmap v3787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5639.590481 secs 2024-07-26T13:01:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:22 smithi134 ceph-mon[31966]: pgmap v3787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:22.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:01:22.457+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:01:22.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:01:22.457+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:01:22.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:01:22.457+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5639.590481 secs 2024-07-26T13:01:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:22 smithi193 ceph-mon[37473]: pgmap v3787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:01:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:01:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:01:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:01:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:01:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:24 smithi134 ceph-mon[31966]: pgmap v3788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:01:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:01:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:01:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:01:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:24 smithi189 ceph-mon[36790]: pgmap v3788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:01:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:01:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:01:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:01:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:24 smithi193 ceph-mon[37473]: pgmap v3788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:01:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:01:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5625.165310 secs 2024-07-26T13:01:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5625.165310 secs 2024-07-26T13:01:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5625.165310 secs 2024-07-26T13:01:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:26 smithi134 ceph-mon[31966]: pgmap v3789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:01:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:26 smithi189 ceph-mon[36790]: pgmap v3789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:01:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:26 smithi193 ceph-mon[37473]: pgmap v3789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:01:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5644.590612 secs 2024-07-26T13:01:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5644.590612 secs 2024-07-26T13:01:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5644.590612 secs 2024-07-26T13:01:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:28 smithi134 ceph-mon[31966]: pgmap v3790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:01:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:28 smithi189 ceph-mon[36790]: pgmap v3790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:01:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:28 smithi193 ceph-mon[37473]: pgmap v3790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:01:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5630.165416 secs 2024-07-26T13:01:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5630.165416 secs 2024-07-26T13:01:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5630.165416 secs 2024-07-26T13:01:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:30 smithi193 ceph-mon[37473]: pgmap v3791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:01:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:30 smithi134 ceph-mon[31966]: pgmap v3791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:01:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:30 smithi189 ceph-mon[36790]: pgmap v3791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:01:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5649.590748 secs 2024-07-26T13:01:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:32 smithi193 ceph-mon[37473]: pgmap v3792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:01:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5649.590748 secs 2024-07-26T13:01:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:32 smithi134 ceph-mon[31966]: pgmap v3792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:01:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5649.590748 secs 2024-07-26T13:01:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:32 smithi189 ceph-mon[36790]: pgmap v3792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:01:32.928 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:01:32.927+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:01:32.928 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:01:32.927+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:32.929 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:01:32.927+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:01:34.474 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:34 smithi134 ceph-mon[31966]: pgmap v3793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:01:34.474 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5635.165492 secs 2024-07-26T13:01:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:34 smithi193 ceph-mon[37473]: pgmap v3793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:01:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5635.165492 secs 2024-07-26T13:01:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:34 smithi189 ceph-mon[36790]: pgmap v3793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:01:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5635.165492 secs 2024-07-26T13:01:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:36 smithi193 ceph-mon[37473]: pgmap v3794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:36 smithi134 ceph-mon[31966]: pgmap v3794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:36 smithi189 ceph-mon[36790]: pgmap v3794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:37.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:01:37.464+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:01:37.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:01:37.464+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:01:37.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:01:37.464+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5654.590865 secs 2024-07-26T13:01:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5654.590865 secs 2024-07-26T13:01:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5654.590865 secs 2024-07-26T13:01:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:38 smithi134 ceph-mon[31966]: pgmap v3795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:38.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:38 smithi189 ceph-mon[36790]: pgmap v3795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:38 smithi193 ceph-mon[37473]: pgmap v3795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5640.165613 secs 2024-07-26T13:01:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5640.165613 secs 2024-07-26T13:01:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5640.165613 secs 2024-07-26T13:01:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:40 smithi134 ceph-mon[31966]: pgmap v3796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:40 smithi189 ceph-mon[36790]: pgmap v3796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:40 smithi193 ceph-mon[37473]: pgmap v3796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5659.590997 secs 2024-07-26T13:01:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:42 smithi134 ceph-mon[31966]: pgmap v3797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5659.590997 secs 2024-07-26T13:01:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:42 smithi189 ceph-mon[36790]: pgmap v3797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5659.590997 secs 2024-07-26T13:01:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:42 smithi193 ceph-mon[37473]: pgmap v3797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:44 smithi134 ceph-mon[31966]: pgmap v3798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:44.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5645.165733 secs 2024-07-26T13:01:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:44 smithi189 ceph-mon[36790]: pgmap v3798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5645.165733 secs 2024-07-26T13:01:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:44 smithi193 ceph-mon[37473]: pgmap v3798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5645.165733 secs 2024-07-26T13:01:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:46 smithi134 ceph-mon[31966]: pgmap v3799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:46 smithi189 ceph-mon[36790]: pgmap v3799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:46 smithi193 ceph-mon[37473]: pgmap v3799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5664.591140 secs 2024-07-26T13:01:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5664.591140 secs 2024-07-26T13:01:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5664.591140 secs 2024-07-26T13:01:47.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:01:47.936+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:01:47.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:01:47.936+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:01:47.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:01:47.936+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:48 smithi134 ceph-mon[31966]: pgmap v3800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:48 smithi189 ceph-mon[36790]: pgmap v3800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:48 smithi193 ceph-mon[37473]: pgmap v3800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5650.165847 secs 2024-07-26T13:01:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5650.165847 secs 2024-07-26T13:01:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5650.165847 secs 2024-07-26T13:01:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:50 smithi134 ceph-mon[31966]: pgmap v3801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:50 smithi189 ceph-mon[36790]: pgmap v3801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:50 smithi193 ceph-mon[37473]: pgmap v3801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:01:52.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:01:52.470+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:01:52.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:01:52.470+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:01:52.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:01:52.470+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:01:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5669.591264 secs 2024-07-26T13:01:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:52 smithi134 ceph-mon[31966]: pgmap v3802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5669.591264 secs 2024-07-26T13:01:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:52 smithi189 ceph-mon[36790]: pgmap v3802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5669.591264 secs 2024-07-26T13:01:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:52 smithi193 ceph-mon[37473]: pgmap v3802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:54 smithi134 ceph-mon[31966]: pgmap v3803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5655.165997 secs 2024-07-26T13:01:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:54 smithi189 ceph-mon[36790]: pgmap v3803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5655.165997 secs 2024-07-26T13:01:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:54 smithi193 ceph-mon[37473]: pgmap v3803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:01:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5655.165997 secs 2024-07-26T13:01:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:56 smithi134 ceph-mon[31966]: pgmap v3804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:56 smithi189 ceph-mon[36790]: pgmap v3804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:56 smithi193 ceph-mon[37473]: pgmap v3804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:01:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5674.591401 secs 2024-07-26T13:01:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5674.591401 secs 2024-07-26T13:01:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5674.591401 secs 2024-07-26T13:01:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:58 smithi134 ceph-mon[31966]: pgmap v3805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:58 smithi189 ceph-mon[36790]: pgmap v3805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:58 smithi193 ceph-mon[37473]: pgmap v3805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:01:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:01:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5660.166071 secs 2024-07-26T13:01:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:01:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5660.166071 secs 2024-07-26T13:01:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:01:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5660.166071 secs 2024-07-26T13:02:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:00 smithi134 ceph-mon[31966]: pgmap v3806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:00.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:00 smithi193 ceph-mon[37473]: pgmap v3806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:00 smithi189 ceph-mon[36790]: pgmap v3806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5679.591532 secs 2024-07-26T13:02:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:02 smithi134 ceph-mon[31966]: pgmap v3807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5679.591532 secs 2024-07-26T13:02:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:02 smithi189 ceph-mon[36790]: pgmap v3807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5679.591532 secs 2024-07-26T13:02:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:02 smithi193 ceph-mon[37473]: pgmap v3807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:02:02.951+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:02:02.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:02:02.952+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:02:02.954 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:02:02.952+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:04 smithi134 ceph-mon[31966]: pgmap v3808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5665.166200 secs 2024-07-26T13:02:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:04 smithi189 ceph-mon[36790]: pgmap v3808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5665.166200 secs 2024-07-26T13:02:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:04 smithi193 ceph-mon[37473]: pgmap v3808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5665.166200 secs 2024-07-26T13:02:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:06 smithi134 ceph-mon[31966]: pgmap v3809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:06 smithi189 ceph-mon[36790]: pgmap v3809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:06 smithi193 ceph-mon[37473]: pgmap v3809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:07.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:02:07.483+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:02:07.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:02:07.483+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:07.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:02:07.483+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:02:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5684.591660 secs 2024-07-26T13:02:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5684.591660 secs 2024-07-26T13:02:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5684.591660 secs 2024-07-26T13:02:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:08 smithi134 ceph-mon[31966]: pgmap v3810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:08 smithi189 ceph-mon[36790]: pgmap v3810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:08 smithi193 ceph-mon[37473]: pgmap v3810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5670.166282 secs 2024-07-26T13:02:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5670.166282 secs 2024-07-26T13:02:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5670.166282 secs 2024-07-26T13:02:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:10 smithi189 ceph-mon[36790]: pgmap v3811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:10 smithi193 ceph-mon[37473]: pgmap v3811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:10 smithi134 ceph-mon[31966]: pgmap v3811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5689.591795 secs 2024-07-26T13:02:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:12 smithi189 ceph-mon[36790]: pgmap v3812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5689.591795 secs 2024-07-26T13:02:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:12 smithi193 ceph-mon[37473]: pgmap v3812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5689.591795 secs 2024-07-26T13:02:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:12 smithi134 ceph-mon[31966]: pgmap v3812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:14 smithi193 ceph-mon[37473]: pgmap v3813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5675.166414 secs 2024-07-26T13:02:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:14 smithi134 ceph-mon[31966]: pgmap v3813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5675.166414 secs 2024-07-26T13:02:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:14 smithi189 ceph-mon[36790]: pgmap v3813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5675.166414 secs 2024-07-26T13:02:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:16 smithi193 ceph-mon[37473]: pgmap v3814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:16 smithi134 ceph-mon[31966]: pgmap v3814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:16 smithi189 ceph-mon[36790]: pgmap v3814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5694.591923 secs 2024-07-26T13:02:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5694.591923 secs 2024-07-26T13:02:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5694.591923 secs 2024-07-26T13:02:17.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:02:17.967+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:02:17.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:02:17.967+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:17.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:02:17.967+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:02:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:18 smithi193 ceph-mon[37473]: pgmap v3815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:18 smithi134 ceph-mon[31966]: pgmap v3815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:18 smithi189 ceph-mon[36790]: pgmap v3815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5680.166491 secs 2024-07-26T13:02:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5680.166491 secs 2024-07-26T13:02:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5680.166491 secs 2024-07-26T13:02:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:20 smithi134 ceph-mon[31966]: pgmap v3816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:20 smithi193 ceph-mon[37473]: pgmap v3816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:20 smithi189 ceph-mon[36790]: pgmap v3816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:02:22.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:02:22.489+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:02:22.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:02:22.489+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:02:22.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:02:22.489+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:22.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5699.592055 secs 2024-07-26T13:02:22.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:22 smithi189 ceph-mon[36790]: pgmap v3817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5699.592055 secs 2024-07-26T13:02:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:22 smithi193 ceph-mon[37473]: pgmap v3817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5699.592055 secs 2024-07-26T13:02:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:22 smithi134 ceph-mon[31966]: pgmap v3817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:02:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:02:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:02:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:24 smithi193 ceph-mon[37473]: pgmap v3818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5685.166613 secs 2024-07-26T13:02:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:02:24.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:02:24.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:02:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:24 smithi134 ceph-mon[31966]: pgmap v3818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5685.166613 secs 2024-07-26T13:02:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:02:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:02:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:02:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:24 smithi189 ceph-mon[36790]: pgmap v3818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5685.166613 secs 2024-07-26T13:02:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:02:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:02:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:02:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:02:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:25 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:02:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:25 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:02:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:25 smithi193 ceph-mon[37473]: pgmap v3819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:02:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:25 smithi193 ceph-mon[37473]: pgmap v3820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:02:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:25 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:02:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:25 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:02:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:25 smithi134 ceph-mon[31966]: pgmap v3819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:02:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:25 smithi134 ceph-mon[31966]: pgmap v3820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:02:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:25 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:02:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:25 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:02:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:25 smithi189 ceph-mon[36790]: pgmap v3819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:02:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:25 smithi189 ceph-mon[36790]: pgmap v3820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:02:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5704.592183 secs 2024-07-26T13:02:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:27 smithi134 ceph-mon[31966]: pgmap v3821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:02:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5704.592183 secs 2024-07-26T13:02:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:27 smithi189 ceph-mon[36790]: pgmap v3821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:02:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5704.592183 secs 2024-07-26T13:02:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:27 smithi193 ceph-mon[37473]: pgmap v3821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:02:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5690.166752 secs 2024-07-26T13:02:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:29 smithi134 ceph-mon[31966]: pgmap v3822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:02:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5690.166752 secs 2024-07-26T13:02:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:29 smithi189 ceph-mon[36790]: pgmap v3822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:02:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5690.166752 secs 2024-07-26T13:02:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:29 smithi193 ceph-mon[37473]: pgmap v3822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:02:31.825 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:31 smithi193 ceph-mon[37473]: pgmap v3823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:02:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:31 smithi134 ceph-mon[31966]: pgmap v3823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:02:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:31 smithi189 ceph-mon[36790]: pgmap v3823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:02:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5709.592354 secs 2024-07-26T13:02:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5709.592354 secs 2024-07-26T13:02:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:02:32.983+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:02:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:02:32.983+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:02:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:02:32.983+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5709.592354 secs 2024-07-26T13:02:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:33 smithi134 ceph-mon[31966]: pgmap v3824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2024-07-26T13:02:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:33 smithi189 ceph-mon[36790]: pgmap v3824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2024-07-26T13:02:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:33 smithi193 ceph-mon[37473]: pgmap v3824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2024-07-26T13:02:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5695.166873 secs 2024-07-26T13:02:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5695.166873 secs 2024-07-26T13:02:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5695.166873 secs 2024-07-26T13:02:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:35 smithi134 ceph-mon[31966]: pgmap v3825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:02:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:35 smithi189 ceph-mon[36790]: pgmap v3825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:02:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:35 smithi193 ceph-mon[37473]: pgmap v3825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:02:37.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:02:37.504+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:02:37.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:02:37.505+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:02:37.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:02:37.505+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5714.592497 secs 2024-07-26T13:02:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:37 smithi134 ceph-mon[31966]: pgmap v3826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5714.592497 secs 2024-07-26T13:02:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:37 smithi189 ceph-mon[36790]: pgmap v3826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5714.592497 secs 2024-07-26T13:02:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:37 smithi193 ceph-mon[37473]: pgmap v3826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5700.166970 secs 2024-07-26T13:02:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:39 smithi134 ceph-mon[31966]: pgmap v3827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5700.166970 secs 2024-07-26T13:02:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:39 smithi189 ceph-mon[36790]: pgmap v3827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5700.166970 secs 2024-07-26T13:02:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:39 smithi193 ceph-mon[37473]: pgmap v3827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:41 smithi134 ceph-mon[31966]: pgmap v3828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:41 smithi189 ceph-mon[36790]: pgmap v3828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:41 smithi193 ceph-mon[37473]: pgmap v3828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5719.592595 secs 2024-07-26T13:02:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5719.592595 secs 2024-07-26T13:02:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5719.592595 secs 2024-07-26T13:02:43.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:43 smithi134 ceph-mon[31966]: pgmap v3829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:43 smithi189 ceph-mon[36790]: pgmap v3829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:43 smithi193 ceph-mon[37473]: pgmap v3829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5705.167091 secs 2024-07-26T13:02:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5705.167091 secs 2024-07-26T13:02:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5705.167091 secs 2024-07-26T13:02:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:45 smithi134 ceph-mon[31966]: pgmap v3830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:02:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:45 smithi189 ceph-mon[36790]: pgmap v3830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:02:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:45 smithi193 ceph-mon[37473]: pgmap v3830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:02:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5724.592732 secs 2024-07-26T13:02:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:47 smithi134 ceph-mon[31966]: pgmap v3831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5724.592732 secs 2024-07-26T13:02:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:47 smithi189 ceph-mon[36790]: pgmap v3831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:47.999 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:02:47.998+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:02:48.000 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:02:47.998+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:48.000 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:02:47.998+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:02:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5724.592732 secs 2024-07-26T13:02:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:47 smithi193 ceph-mon[37473]: pgmap v3831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5710.167194 secs 2024-07-26T13:02:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:49 smithi134 ceph-mon[31966]: pgmap v3832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5710.167194 secs 2024-07-26T13:02:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:49 smithi189 ceph-mon[36790]: pgmap v3832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5710.167194 secs 2024-07-26T13:02:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:49 smithi193 ceph-mon[37473]: pgmap v3832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:02:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:51 smithi134 ceph-mon[31966]: pgmap v3833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:51 smithi189 ceph-mon[36790]: pgmap v3833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:51 smithi193 ceph-mon[37473]: pgmap v3833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:52.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:02:52.520+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:02:52.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:02:52.520+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:02:52.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:02:52.520+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:02:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5729.592815 secs 2024-07-26T13:02:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5729.592815 secs 2024-07-26T13:02:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5729.592815 secs 2024-07-26T13:02:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:53 smithi134 ceph-mon[31966]: pgmap v3834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:53 smithi189 ceph-mon[36790]: pgmap v3834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:53 smithi193 ceph-mon[37473]: pgmap v3834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:02:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5715.167313 secs 2024-07-26T13:02:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5715.167313 secs 2024-07-26T13:02:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5715.167313 secs 2024-07-26T13:02:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:55 smithi134 ceph-mon[31966]: pgmap v3835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:02:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:55 smithi189 ceph-mon[36790]: pgmap v3835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:02:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:55 smithi193 ceph-mon[37473]: pgmap v3835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:02:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5734.592959 secs 2024-07-26T13:02:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:57 smithi134 ceph-mon[31966]: pgmap v3836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5734.592959 secs 2024-07-26T13:02:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:57 smithi189 ceph-mon[36790]: pgmap v3836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5734.592959 secs 2024-07-26T13:02:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:57 smithi193 ceph-mon[37473]: pgmap v3836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:02:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5720.167437 secs 2024-07-26T13:02:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:02:59 smithi134 ceph-mon[31966]: pgmap v3837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:02:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5720.167437 secs 2024-07-26T13:02:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:02:59 smithi189 ceph-mon[36790]: pgmap v3837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:03:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5720.167437 secs 2024-07-26T13:03:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:02:59 smithi193 ceph-mon[37473]: pgmap v3837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:03:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:01 smithi189 ceph-mon[36790]: pgmap v3838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:03:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:01 smithi193 ceph-mon[37473]: pgmap v3838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:03:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:01 smithi134 ceph-mon[31966]: pgmap v3838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:03:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5739.593037 secs 2024-07-26T13:03:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:03:03.013+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:03:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:03:03.013+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:03:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:03:03.013+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5739.593037 secs 2024-07-26T13:03:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5739.593037 secs 2024-07-26T13:03:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:03 smithi189 ceph-mon[36790]: pgmap v3839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:03 smithi193 ceph-mon[37473]: pgmap v3839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:03 smithi134 ceph-mon[31966]: pgmap v3839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5725.167561 secs 2024-07-26T13:03:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5725.167561 secs 2024-07-26T13:03:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5725.167561 secs 2024-07-26T13:03:06.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:05 smithi193 ceph-mon[37473]: pgmap v3840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T13:03:06.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:05 smithi134 ceph-mon[31966]: pgmap v3840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T13:03:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:05 smithi189 ceph-mon[36790]: pgmap v3840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T13:03:07.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:03:07.536+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:03:07.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:03:07.536+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:03:07.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:03:07.536+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5744.593173 secs 2024-07-26T13:03:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:07 smithi134 ceph-mon[31966]: pgmap v3841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:03:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5744.593173 secs 2024-07-26T13:03:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:07 smithi193 ceph-mon[37473]: pgmap v3841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:03:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5744.593173 secs 2024-07-26T13:03:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:07 smithi189 ceph-mon[36790]: pgmap v3841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:03:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5730.167710 secs 2024-07-26T13:03:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:09 smithi193 ceph-mon[37473]: pgmap v3842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:03:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5730.167710 secs 2024-07-26T13:03:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:09 smithi134 ceph-mon[31966]: pgmap v3842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:03:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5730.167710 secs 2024-07-26T13:03:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:09 smithi189 ceph-mon[36790]: pgmap v3842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:03:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:11 smithi193 ceph-mon[37473]: pgmap v3843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:11 smithi134 ceph-mon[31966]: pgmap v3843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:11 smithi189 ceph-mon[36790]: pgmap v3843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5749.593246 secs 2024-07-26T13:03:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5749.593246 secs 2024-07-26T13:03:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5749.593246 secs 2024-07-26T13:03:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:13 smithi193 ceph-mon[37473]: pgmap v3844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:13 smithi134 ceph-mon[31966]: pgmap v3844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:13 smithi189 ceph-mon[36790]: pgmap v3844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5735.167852 secs 2024-07-26T13:03:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5735.167852 secs 2024-07-26T13:03:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5735.167852 secs 2024-07-26T13:03:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:15 smithi193 ceph-mon[37473]: pgmap v3845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:03:16.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:15 smithi134 ceph-mon[31966]: pgmap v3845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:03:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:15 smithi189 ceph-mon[36790]: pgmap v3845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:03:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5754.593369 secs 2024-07-26T13:03:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5754.593369 secs 2024-07-26T13:03:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5754.593369 secs 2024-07-26T13:03:18.017 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:03:18.016+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:03:18.018 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:03:18.016+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:03:18.018 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:03:18.016+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:17 smithi193 ceph-mon[37473]: pgmap v3846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:17 smithi134 ceph-mon[31966]: pgmap v3846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:17 smithi189 ceph-mon[36790]: pgmap v3846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5740.167971 secs 2024-07-26T13:03:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:19 smithi134 ceph-mon[31966]: pgmap v3847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5740.167971 secs 2024-07-26T13:03:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:19 smithi189 ceph-mon[36790]: pgmap v3847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5740.167971 secs 2024-07-26T13:03:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:19 smithi193 ceph-mon[37473]: pgmap v3847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:21 smithi134 ceph-mon[31966]: pgmap v3848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5759.593490 secs 2024-07-26T13:03:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:21 smithi189 ceph-mon[36790]: pgmap v3848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5759.593490 secs 2024-07-26T13:03:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:21 smithi193 ceph-mon[37473]: pgmap v3848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5759.593490 secs 2024-07-26T13:03:22.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:03:22.551+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:03:22.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:03:22.551+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:22.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:03:22.551+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:03:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:03:23.327+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T13:03:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:03:23.327+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T13:03:23.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:03:23.328+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T13:03:23.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:03:23.328+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T13:03:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:23 smithi134 ceph-mon[31966]: pgmap v3849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:23 smithi189 ceph-mon[36790]: pgmap v3849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:23 smithi193 ceph-mon[37473]: pgmap v3849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:03:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5745.168062 secs 2024-07-26T13:03:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:03:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5745.168062 secs 2024-07-26T13:03:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:03:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5745.168062 secs 2024-07-26T13:03:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:03:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:25 smithi134 ceph-mon[31966]: pgmap v3850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:03:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:03:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:03:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:03:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:25 smithi189 ceph-mon[36790]: pgmap v3850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:03:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:03:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:03:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:03:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:25 smithi193 ceph-mon[37473]: pgmap v3850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:03:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:03:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:03:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:03:26.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:03:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:26 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:03:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:26 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:03:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:26 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:03:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:26 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:03:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:26 smithi134 ceph-mon[31966]: pgmap v3851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:03:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:26 smithi134 ceph-mon[31966]: pgmap v3852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:03:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:26 smithi134 ceph-mon[31966]: 1 slow requests, 1 included below; oldest blocked for > 5764.593570 secs 2024-07-26T13:03:27.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:26 smithi134 ceph-mon[31966]: slow request 5764.593570 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T13:03:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:26 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:03:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:26 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:03:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:26 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:03:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:26 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:03:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:26 smithi189 ceph-mon[36790]: pgmap v3851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:03:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:26 smithi189 ceph-mon[36790]: pgmap v3852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:03:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:26 smithi189 ceph-mon[36790]: 1 slow requests, 1 included below; oldest blocked for > 5764.593570 secs 2024-07-26T13:03:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:26 smithi189 ceph-mon[36790]: slow request 5764.593570 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T13:03:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:26 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:03:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:26 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:03:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:26 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:03:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:26 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:03:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:26 smithi193 ceph-mon[37473]: pgmap v3851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:03:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:26 smithi193 ceph-mon[37473]: pgmap v3852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:03:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:26 smithi193 ceph-mon[37473]: 1 slow requests, 1 included below; oldest blocked for > 5764.593570 secs 2024-07-26T13:03:27.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:26 smithi193 ceph-mon[37473]: slow request 5764.593570 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T13:03:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:28 smithi134 ceph-mon[31966]: pgmap v3853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:03:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:28 smithi189 ceph-mon[36790]: pgmap v3853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:03:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:28 smithi193 ceph-mon[37473]: pgmap v3853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:03:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5750.168161 secs 2024-07-26T13:03:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5750.168161 secs 2024-07-26T13:03:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5750.168161 secs 2024-07-26T13:03:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:30 smithi134 ceph-mon[31966]: pgmap v3854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:03:31.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:30 smithi193 ceph-mon[37473]: pgmap v3854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:03:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:30 smithi189 ceph-mon[36790]: pgmap v3854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:03:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5769.593728 secs 2024-07-26T13:03:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5769.593728 secs 2024-07-26T13:03:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5769.593728 secs 2024-07-26T13:03:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:03:33.032+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:03:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:03:33.032+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:03:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:03:33.032+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:32 smithi134 ceph-mon[31966]: pgmap v3855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:03:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:32 smithi189 ceph-mon[36790]: pgmap v3855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:03:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:32 smithi193 ceph-mon[37473]: pgmap v3855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:03:33.845 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:03:33.844+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T13:03:33.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:03:33.844+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:03:33.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:03:33.844+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T13:03:33.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:03:33.845+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:03:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5755.168274 secs 2024-07-26T13:03:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:34 smithi134 ceph-mon[31966]: pgmap v3856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:03:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5755.168274 secs 2024-07-26T13:03:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:34 smithi189 ceph-mon[36790]: pgmap v3856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:03:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5755.168274 secs 2024-07-26T13:03:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:34 smithi193 ceph-mon[37473]: pgmap v3856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:03:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:36 smithi134 ceph-mon[31966]: pgmap v3857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:03:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5774.593808 secs 2024-07-26T13:03:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:36 smithi189 ceph-mon[36790]: pgmap v3857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:03:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5774.593808 secs 2024-07-26T13:03:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:36 smithi193 ceph-mon[37473]: pgmap v3857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:03:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5774.593808 secs 2024-07-26T13:03:37.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:03:37.554+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:03:37.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:03:37.554+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:03:37.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:03:37.554+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:38 smithi134 ceph-mon[31966]: pgmap v3858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:38 smithi189 ceph-mon[36790]: pgmap v3858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:38 smithi193 ceph-mon[37473]: pgmap v3858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:39 smithi134 ceph-mon[31966]: 2 slow requests, 1 included below; oldest blocked for > 5760.168400 secs 2024-07-26T13:03:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:39 smithi134 ceph-mon[31966]: slow request 5760.168399 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T13:03:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:39 smithi189 ceph-mon[36790]: 2 slow requests, 1 included below; oldest blocked for > 5760.168400 secs 2024-07-26T13:03:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:39 smithi189 ceph-mon[36790]: slow request 5760.168399 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T13:03:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:39 smithi193 ceph-mon[37473]: 2 slow requests, 1 included below; oldest blocked for > 5760.168400 secs 2024-07-26T13:03:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:39 smithi193 ceph-mon[37473]: slow request 5760.168399 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T13:03:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:40 smithi134 ceph-mon[31966]: pgmap v3859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:40 smithi189 ceph-mon[36790]: pgmap v3859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:40 smithi193 ceph-mon[37473]: pgmap v3859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5779.593951 secs 2024-07-26T13:03:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5779.593951 secs 2024-07-26T13:03:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5779.593951 secs 2024-07-26T13:03:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:42 smithi134 ceph-mon[31966]: pgmap v3860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:42 smithi189 ceph-mon[36790]: pgmap v3860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:42 smithi193 ceph-mon[37473]: pgmap v3860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5765.168519 secs 2024-07-26T13:03:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:44 smithi134 ceph-mon[31966]: pgmap v3861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5765.168519 secs 2024-07-26T13:03:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:44 smithi189 ceph-mon[36790]: pgmap v3861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5765.168519 secs 2024-07-26T13:03:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:44 smithi193 ceph-mon[37473]: pgmap v3861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:46 smithi134 ceph-mon[31966]: pgmap v3862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5784.594070 secs 2024-07-26T13:03:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:46 smithi189 ceph-mon[36790]: pgmap v3862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5784.594070 secs 2024-07-26T13:03:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:46 smithi193 ceph-mon[37473]: pgmap v3862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5784.594070 secs 2024-07-26T13:03:48.042 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:03:48.041+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:03:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:03:48.041+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:03:48.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:03:48.041+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:48 smithi134 ceph-mon[31966]: pgmap v3863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:48 smithi189 ceph-mon[36790]: pgmap v3863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:48 smithi193 ceph-mon[37473]: pgmap v3863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5770.168632 secs 2024-07-26T13:03:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5770.168632 secs 2024-07-26T13:03:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5770.168632 secs 2024-07-26T13:03:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:50 smithi189 ceph-mon[36790]: pgmap v3864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:50 smithi193 ceph-mon[37473]: pgmap v3864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:50 smithi134 ceph-mon[31966]: pgmap v3864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:03:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5789.594166 secs 2024-07-26T13:03:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5789.594166 secs 2024-07-26T13:03:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5789.594166 secs 2024-07-26T13:03:52.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:03:52.570+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:03:52.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:03:52.570+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:03:52.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:03:52.570+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:03:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:52 smithi189 ceph-mon[36790]: pgmap v3865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:52 smithi193 ceph-mon[37473]: pgmap v3865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:52 smithi134 ceph-mon[31966]: pgmap v3865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5775.168746 secs 2024-07-26T13:03:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:54 smithi193 ceph-mon[37473]: pgmap v3866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5775.168746 secs 2024-07-26T13:03:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:54 smithi134 ceph-mon[31966]: pgmap v3866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5775.168746 secs 2024-07-26T13:03:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:54 smithi189 ceph-mon[36790]: pgmap v3866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:56 smithi193 ceph-mon[37473]: pgmap v3867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5794.594282 secs 2024-07-26T13:03:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:56 smithi134 ceph-mon[31966]: pgmap v3867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5794.594282 secs 2024-07-26T13:03:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:56 smithi189 ceph-mon[36790]: pgmap v3867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:03:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5794.594282 secs 2024-07-26T13:03:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:58 smithi193 ceph-mon[37473]: pgmap v3868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:58 smithi134 ceph-mon[31966]: pgmap v3868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:03:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:58 smithi189 ceph-mon[36790]: pgmap v3868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:03:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5780.168884 secs 2024-07-26T13:04:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:03:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5780.168884 secs 2024-07-26T13:04:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:03:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5780.168884 secs 2024-07-26T13:04:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:00 smithi193 ceph-mon[37473]: pgmap v3869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:01.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:00 smithi134 ceph-mon[31966]: pgmap v3869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:00 smithi189 ceph-mon[36790]: pgmap v3869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5799.594342 secs 2024-07-26T13:04:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5799.594342 secs 2024-07-26T13:04:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5799.594342 secs 2024-07-26T13:04:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:04:03.056+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:04:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:04:03.057+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:04:03.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:04:03.057+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:02 smithi193 ceph-mon[37473]: pgmap v3870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:02 smithi134 ceph-mon[31966]: pgmap v3870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:02 smithi189 ceph-mon[36790]: pgmap v3870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:04 smithi193 ceph-mon[37473]: 2 slow requests, 1 included below; oldest blocked for > 5785.169008 secs 2024-07-26T13:04:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:04 smithi193 ceph-mon[37473]: slow request 5762.423725 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T13:04:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:04 smithi193 ceph-mon[37473]: pgmap v3871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:04:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:04 smithi134 ceph-mon[31966]: 2 slow requests, 1 included below; oldest blocked for > 5785.169008 secs 2024-07-26T13:04:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:04 smithi134 ceph-mon[31966]: slow request 5762.423725 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T13:04:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:04 smithi134 ceph-mon[31966]: pgmap v3871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:04:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:04 smithi189 ceph-mon[36790]: 2 slow requests, 1 included below; oldest blocked for > 5785.169008 secs 2024-07-26T13:04:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:04 smithi189 ceph-mon[36790]: slow request 5762.423725 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T13:04:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:04 smithi189 ceph-mon[36790]: pgmap v3871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:04:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:07 smithi193 ceph-mon[37473]: pgmap v3872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:04:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5804.594441 secs 2024-07-26T13:04:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:07 smithi134 ceph-mon[31966]: pgmap v3872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:04:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5804.594441 secs 2024-07-26T13:04:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:07 smithi189 ceph-mon[36790]: pgmap v3872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:04:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5804.594441 secs 2024-07-26T13:04:07.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:04:07.581+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:04:07.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:04:07.581+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:04:07.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:04:07.581+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:09 smithi193 ceph-mon[37473]: pgmap v3873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:04:09.352 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:09 smithi134 ceph-mon[31966]: pgmap v3873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:04:09.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:09 smithi189 ceph-mon[36790]: pgmap v3873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:04:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5790.169129 secs 2024-07-26T13:04:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5790.169129 secs 2024-07-26T13:04:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5790.169129 secs 2024-07-26T13:04:11.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:11 smithi193 ceph-mon[37473]: pgmap v3874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:04:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:11 smithi134 ceph-mon[31966]: pgmap v3874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:04:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:11 smithi189 ceph-mon[36790]: pgmap v3874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:04:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5809.594581 secs 2024-07-26T13:04:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5809.594581 secs 2024-07-26T13:04:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5809.594581 secs 2024-07-26T13:04:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:13 smithi134 ceph-mon[31966]: pgmap v3875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:04:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:13 smithi189 ceph-mon[36790]: pgmap v3875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:04:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:13 smithi193 ceph-mon[37473]: pgmap v3875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:04:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5795.169268 secs 2024-07-26T13:04:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:15 smithi134 ceph-mon[31966]: pgmap v3876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:04:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5795.169268 secs 2024-07-26T13:04:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:15 smithi189 ceph-mon[36790]: pgmap v3876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:04:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5795.169268 secs 2024-07-26T13:04:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:15 smithi193 ceph-mon[37473]: pgmap v3876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:04:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:17 smithi134 ceph-mon[31966]: pgmap v3877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5814.594699 secs 2024-07-26T13:04:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:17 smithi189 ceph-mon[36790]: pgmap v3877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5814.594699 secs 2024-07-26T13:04:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:17 smithi193 ceph-mon[37473]: pgmap v3877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5814.594699 secs 2024-07-26T13:04:18.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:04:18.064+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:04:18.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:04:18.064+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:04:18.066 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:04:18.064+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:19 smithi134 ceph-mon[31966]: pgmap v3878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:19 smithi189 ceph-mon[36790]: pgmap v3878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:19 smithi193 ceph-mon[37473]: pgmap v3878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5800.169394 secs 2024-07-26T13:04:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5800.169394 secs 2024-07-26T13:04:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5800.169394 secs 2024-07-26T13:04:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:21 smithi134 ceph-mon[31966]: pgmap v3879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:21 smithi189 ceph-mon[36790]: pgmap v3879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:21 smithi193 ceph-mon[37473]: pgmap v3879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5819.594823 secs 2024-07-26T13:04:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5819.594823 secs 2024-07-26T13:04:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5819.594823 secs 2024-07-26T13:04:22.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:04:22.592+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:04:22.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:04:22.592+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:04:22.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:04:22.592+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:23.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:23 smithi189 ceph-mon[36790]: pgmap v3880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:23 smithi134 ceph-mon[31966]: pgmap v3880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:23 smithi193 ceph-mon[37473]: pgmap v3880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5805.169511 secs 2024-07-26T13:04:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:25 smithi134 ceph-mon[31966]: pgmap v3881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5805.169511 secs 2024-07-26T13:04:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:25 smithi189 ceph-mon[36790]: pgmap v3881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5805.169511 secs 2024-07-26T13:04:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:25 smithi193 ceph-mon[37473]: pgmap v3881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:26.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:04:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:04:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:04:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:27 smithi134 ceph-mon[31966]: pgmap v3882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5824.594905 secs 2024-07-26T13:04:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:04:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:04:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:04:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:27 smithi189 ceph-mon[36790]: pgmap v3882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5824.594905 secs 2024-07-26T13:04:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:04:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:04:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:04:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:27 smithi193 ceph-mon[37473]: pgmap v3882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5824.594905 secs 2024-07-26T13:04:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:04:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:04:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:04:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:28 smithi134 ceph-mon[31966]: pgmap v3883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:04:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:28 smithi134 ceph-mon[31966]: pgmap v3884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:04:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:28 smithi189 ceph-mon[36790]: pgmap v3883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:04:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:28 smithi189 ceph-mon[36790]: pgmap v3884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:04:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:28 smithi193 ceph-mon[37473]: pgmap v3883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:04:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:28 smithi193 ceph-mon[37473]: pgmap v3884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:04:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:29 smithi134 ceph-mon[31966]: pgmap v3885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:04:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:29 smithi189 ceph-mon[36790]: pgmap v3885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:04:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:29 smithi193 ceph-mon[37473]: pgmap v3885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:04:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5810.169628 secs 2024-07-26T13:04:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5810.169628 secs 2024-07-26T13:04:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5810.169628 secs 2024-07-26T13:04:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:31 smithi134 ceph-mon[31966]: pgmap v3886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:04:31.424 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:31 smithi193 ceph-mon[37473]: pgmap v3886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:04:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:31 smithi189 ceph-mon[36790]: pgmap v3886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:04:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5829.595026 secs 2024-07-26T13:04:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5829.595026 secs 2024-07-26T13:04:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5829.595026 secs 2024-07-26T13:04:33.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:04:33.080+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:04:33.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:04:33.080+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:04:33.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:04:33.080+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:33 smithi134 ceph-mon[31966]: pgmap v3887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:04:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:33 smithi189 ceph-mon[36790]: pgmap v3887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:04:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:33 smithi193 ceph-mon[37473]: pgmap v3887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:04:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5815.169729 secs 2024-07-26T13:04:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5815.169729 secs 2024-07-26T13:04:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5815.169729 secs 2024-07-26T13:04:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:35 smithi134 ceph-mon[31966]: pgmap v3888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:04:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:35 smithi189 ceph-mon[36790]: pgmap v3888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:04:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:35 smithi193 ceph-mon[37473]: pgmap v3888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:04:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5834.595141 secs 2024-07-26T13:04:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5834.595141 secs 2024-07-26T13:04:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5834.595141 secs 2024-07-26T13:04:37.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:04:37.598+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:04:37.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:04:37.598+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:37.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:04:37.598+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:04:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:37 smithi134 ceph-mon[31966]: pgmap v3889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:04:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:37 smithi189 ceph-mon[36790]: pgmap v3889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:04:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:37 smithi193 ceph-mon[37473]: pgmap v3889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:04:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5820.169798 secs 2024-07-26T13:04:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:39 smithi134 ceph-mon[31966]: pgmap v3890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5820.169798 secs 2024-07-26T13:04:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:39 smithi189 ceph-mon[36790]: pgmap v3890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5820.169798 secs 2024-07-26T13:04:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:39 smithi193 ceph-mon[37473]: pgmap v3890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:41 smithi134 ceph-mon[31966]: pgmap v3891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5839.595219 secs 2024-07-26T13:04:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:41 smithi189 ceph-mon[36790]: pgmap v3891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5839.595219 secs 2024-07-26T13:04:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:41 smithi193 ceph-mon[37473]: pgmap v3891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5839.595219 secs 2024-07-26T13:04:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:43 smithi134 ceph-mon[31966]: pgmap v3892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:43 smithi189 ceph-mon[36790]: pgmap v3892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:43 smithi193 ceph-mon[37473]: pgmap v3892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5825.169917 secs 2024-07-26T13:04:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5825.169917 secs 2024-07-26T13:04:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5825.169917 secs 2024-07-26T13:04:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:45 smithi189 ceph-mon[36790]: pgmap v3893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:45 smithi193 ceph-mon[37473]: pgmap v3893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:46.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:45 smithi134 ceph-mon[31966]: pgmap v3893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5844.595367 secs 2024-07-26T13:04:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5844.595367 secs 2024-07-26T13:04:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5844.595367 secs 2024-07-26T13:04:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:04:48.095+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:04:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:04:48.095+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:04:48.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:04:48.095+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:47 smithi189 ceph-mon[36790]: pgmap v3894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:47 smithi193 ceph-mon[37473]: pgmap v3894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:47 smithi134 ceph-mon[31966]: pgmap v3894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5830.170035 secs 2024-07-26T13:04:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:49 smithi193 ceph-mon[37473]: pgmap v3895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5830.170035 secs 2024-07-26T13:04:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:49 smithi134 ceph-mon[31966]: pgmap v3895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5830.170035 secs 2024-07-26T13:04:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:49 smithi189 ceph-mon[36790]: pgmap v3895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:51 smithi193 ceph-mon[37473]: pgmap v3896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5849.595458 secs 2024-07-26T13:04:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:51 smithi134 ceph-mon[31966]: pgmap v3896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5849.595458 secs 2024-07-26T13:04:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:51 smithi189 ceph-mon[36790]: pgmap v3896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5849.595458 secs 2024-07-26T13:04:52.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:04:52.613+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:04:52.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:04:52.613+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:04:52.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:04:52.613+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:04:54.245 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:53 smithi189 ceph-mon[36790]: pgmap v3897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:53 smithi193 ceph-mon[37473]: pgmap v3897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:53 smithi134 ceph-mon[31966]: pgmap v3897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:04:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5835.170137 secs 2024-07-26T13:04:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5835.170137 secs 2024-07-26T13:04:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5835.170137 secs 2024-07-26T13:04:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:55 smithi193 ceph-mon[37473]: pgmap v3898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:56.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:55 smithi134 ceph-mon[31966]: pgmap v3898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:55 smithi189 ceph-mon[36790]: pgmap v3898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:04:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5854.595529 secs 2024-07-26T13:04:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5854.595529 secs 2024-07-26T13:04:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5854.595529 secs 2024-07-26T13:04:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:57 smithi193 ceph-mon[37473]: pgmap v3899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:57 smithi134 ceph-mon[31966]: pgmap v3899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:04:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:57 smithi189 ceph-mon[36790]: pgmap v3899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5840.170258 secs 2024-07-26T13:05:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:04:59 smithi193 ceph-mon[37473]: pgmap v3900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5840.170258 secs 2024-07-26T13:05:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:04:59 smithi134 ceph-mon[31966]: pgmap v3900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5840.170258 secs 2024-07-26T13:05:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:04:59 smithi189 ceph-mon[36790]: pgmap v3900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:01 smithi193 ceph-mon[37473]: pgmap v3901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5859.595672 secs 2024-07-26T13:05:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:01 smithi134 ceph-mon[31966]: pgmap v3901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5859.595672 secs 2024-07-26T13:05:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:01 smithi189 ceph-mon[36790]: pgmap v3901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5859.595672 secs 2024-07-26T13:05:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:05:03.111+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:05:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:05:03.111+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:05:03.112 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:05:03.111+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:04 smithi193 ceph-mon[37473]: pgmap v3902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:04 smithi134 ceph-mon[31966]: pgmap v3902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:04 smithi189 ceph-mon[36790]: pgmap v3902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5845.170378 secs 2024-07-26T13:05:05.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5845.170378 secs 2024-07-26T13:05:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5845.170378 secs 2024-07-26T13:05:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:06 smithi193 ceph-mon[37473]: pgmap v3903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:06 smithi134 ceph-mon[31966]: pgmap v3903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:06 smithi189 ceph-mon[36790]: pgmap v3903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:07.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5864.595790 secs 2024-07-26T13:05:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5864.595790 secs 2024-07-26T13:05:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5864.595790 secs 2024-07-26T13:05:07.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:05:07.629+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:05:07.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:05:07.629+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:07.631 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:05:07.629+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:05:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:08 smithi134 ceph-mon[31966]: pgmap v3904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:08 smithi189 ceph-mon[36790]: pgmap v3904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:08 smithi193 ceph-mon[37473]: pgmap v3904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5850.170535 secs 2024-07-26T13:05:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:10 smithi134 ceph-mon[31966]: pgmap v3905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5850.170535 secs 2024-07-26T13:05:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:10 smithi189 ceph-mon[36790]: pgmap v3905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5850.170535 secs 2024-07-26T13:05:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:10 smithi193 ceph-mon[37473]: pgmap v3905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:12.330 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:12 smithi134 ceph-mon[31966]: pgmap v3906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:12.330 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5869.595928 secs 2024-07-26T13:05:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:12 smithi189 ceph-mon[36790]: pgmap v3906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5869.595928 secs 2024-07-26T13:05:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:12 smithi193 ceph-mon[37473]: pgmap v3906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5869.595928 secs 2024-07-26T13:05:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:14 smithi134 ceph-mon[31966]: pgmap v3907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:14 smithi189 ceph-mon[36790]: pgmap v3907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:14 smithi193 ceph-mon[37473]: pgmap v3907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5855.170653 secs 2024-07-26T13:05:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5855.170653 secs 2024-07-26T13:05:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5855.170653 secs 2024-07-26T13:05:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:16 smithi134 ceph-mon[31966]: pgmap v3908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:16 smithi189 ceph-mon[36790]: pgmap v3908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:16 smithi193 ceph-mon[37473]: pgmap v3908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5874.596064 secs 2024-07-26T13:05:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5874.596064 secs 2024-07-26T13:05:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5874.596064 secs 2024-07-26T13:05:18.121 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:05:18.121+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:05:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:05:18.121+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:05:18.122 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:05:18.121+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:18 smithi134 ceph-mon[31966]: pgmap v3909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:18 smithi189 ceph-mon[36790]: pgmap v3909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:18.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:18 smithi193 ceph-mon[37473]: pgmap v3909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5860.170772 secs 2024-07-26T13:05:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:20 smithi134 ceph-mon[31966]: pgmap v3910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5860.170772 secs 2024-07-26T13:05:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:20 smithi189 ceph-mon[36790]: pgmap v3910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5860.170772 secs 2024-07-26T13:05:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:20 smithi193 ceph-mon[37473]: pgmap v3910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:22 smithi134 ceph-mon[31966]: pgmap v3911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5879.596179 secs 2024-07-26T13:05:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:22 smithi189 ceph-mon[36790]: pgmap v3911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5879.596179 secs 2024-07-26T13:05:22.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:22 smithi193 ceph-mon[37473]: pgmap v3911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:22.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5879.596179 secs 2024-07-26T13:05:22.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:05:22.639+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:05:22.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:05:22.639+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:22.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:05:22.639+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:05:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:24 smithi134 ceph-mon[31966]: pgmap v3912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:24 smithi189 ceph-mon[36790]: pgmap v3912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:24 smithi193 ceph-mon[37473]: pgmap v3912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5865.170915 secs 2024-07-26T13:05:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5865.170915 secs 2024-07-26T13:05:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5865.170915 secs 2024-07-26T13:05:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:26 smithi134 ceph-mon[31966]: pgmap v3913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:26 smithi189 ceph-mon[36790]: pgmap v3913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:26 smithi193 ceph-mon[37473]: pgmap v3913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5884.596324 secs 2024-07-26T13:05:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:05:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5884.596324 secs 2024-07-26T13:05:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:05:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5884.596324 secs 2024-07-26T13:05:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:05:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:28 smithi134 ceph-mon[31966]: pgmap v3914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:05:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:05:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:28 smithi189 ceph-mon[36790]: pgmap v3914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:05:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:05:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:28 smithi193 ceph-mon[37473]: pgmap v3914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:05:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:05:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:05:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:05:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:05:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:05:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:05:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:05:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:05:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:05:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:05:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5870.171052 secs 2024-07-26T13:05:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:30 smithi134 ceph-mon[31966]: pgmap v3915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T13:05:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:30 smithi134 ceph-mon[31966]: pgmap v3916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5870.171052 secs 2024-07-26T13:05:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:30 smithi189 ceph-mon[36790]: pgmap v3915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T13:05:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:30 smithi189 ceph-mon[36790]: pgmap v3916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5870.171052 secs 2024-07-26T13:05:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:30 smithi193 ceph-mon[37473]: pgmap v3915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T13:05:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:30 smithi193 ceph-mon[37473]: pgmap v3916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:31 smithi134 ceph-mon[31966]: pgmap v3917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:31 smithi189 ceph-mon[36790]: pgmap v3917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:31 smithi193 ceph-mon[37473]: pgmap v3917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5889.596444 secs 2024-07-26T13:05:32.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5889.596444 secs 2024-07-26T13:05:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5889.596444 secs 2024-07-26T13:05:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:05:33.132+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:05:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:05:33.132+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:05:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:05:33.132+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:33 smithi134 ceph-mon[31966]: pgmap v3918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:33 smithi189 ceph-mon[36790]: pgmap v3918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:33 smithi193 ceph-mon[37473]: pgmap v3918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5875.171171 secs 2024-07-26T13:05:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5875.171171 secs 2024-07-26T13:05:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5875.171171 secs 2024-07-26T13:05:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:35 smithi134 ceph-mon[31966]: pgmap v3919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:35 smithi189 ceph-mon[36790]: pgmap v3919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:35 smithi193 ceph-mon[37473]: pgmap v3919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:37.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:05:37.655+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:05:37.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:05:37.655+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:37.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:05:37.655+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:05:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5894.596572 secs 2024-07-26T13:05:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:37 smithi134 ceph-mon[31966]: pgmap v3920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5894.596572 secs 2024-07-26T13:05:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:37 smithi193 ceph-mon[37473]: pgmap v3920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5894.596572 secs 2024-07-26T13:05:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:37 smithi189 ceph-mon[36790]: pgmap v3920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:05:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5880.171289 secs 2024-07-26T13:05:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:39 smithi193 ceph-mon[37473]: pgmap v3921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:05:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5880.171289 secs 2024-07-26T13:05:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:39 smithi134 ceph-mon[31966]: pgmap v3921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:05:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5880.171289 secs 2024-07-26T13:05:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:39 smithi189 ceph-mon[36790]: pgmap v3921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:05:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:41 smithi193 ceph-mon[37473]: pgmap v3922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:42.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:41 smithi134 ceph-mon[31966]: pgmap v3922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:41 smithi189 ceph-mon[36790]: pgmap v3922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5899.596698 secs 2024-07-26T13:05:43.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5899.596698 secs 2024-07-26T13:05:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5899.596698 secs 2024-07-26T13:05:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:43 smithi193 ceph-mon[37473]: pgmap v3923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:44.072 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:43 smithi134 ceph-mon[31966]: pgmap v3923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:43 smithi189 ceph-mon[36790]: pgmap v3923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5885.171400 secs 2024-07-26T13:05:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5885.171400 secs 2024-07-26T13:05:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5885.171400 secs 2024-07-26T13:05:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:45 smithi193 ceph-mon[37473]: pgmap v3924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:45 smithi134 ceph-mon[31966]: pgmap v3924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:45 smithi189 ceph-mon[36790]: pgmap v3924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5904.596826 secs 2024-07-26T13:05:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5904.596826 secs 2024-07-26T13:05:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5904.596826 secs 2024-07-26T13:05:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:05:48.148+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:05:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:05:48.148+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:05:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:05:48.148+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:47 smithi134 ceph-mon[31966]: pgmap v3925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:47 smithi189 ceph-mon[36790]: pgmap v3925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:47 smithi193 ceph-mon[37473]: pgmap v3925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5890.171513 secs 2024-07-26T13:05:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:49 smithi134 ceph-mon[31966]: pgmap v3926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5890.171513 secs 2024-07-26T13:05:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:49 smithi189 ceph-mon[36790]: pgmap v3926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5890.171513 secs 2024-07-26T13:05:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:49 smithi193 ceph-mon[37473]: pgmap v3926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:51 smithi134 ceph-mon[31966]: pgmap v3927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5909.596966 secs 2024-07-26T13:05:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:51 smithi189 ceph-mon[36790]: pgmap v3927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5909.596966 secs 2024-07-26T13:05:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:51 smithi193 ceph-mon[37473]: pgmap v3927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5909.596966 secs 2024-07-26T13:05:52.668 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:05:52.667+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:05:52.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:05:52.667+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:05:52.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:05:52.667+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:05:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:53 smithi134 ceph-mon[31966]: pgmap v3928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:53 smithi189 ceph-mon[36790]: pgmap v3928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:53 smithi193 ceph-mon[37473]: pgmap v3928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:05:55.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5895.171625 secs 2024-07-26T13:05:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5895.171625 secs 2024-07-26T13:05:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5895.171625 secs 2024-07-26T13:05:56.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:55 smithi134 ceph-mon[31966]: pgmap v3929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:55 smithi189 ceph-mon[36790]: pgmap v3929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:55 smithi193 ceph-mon[37473]: pgmap v3929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:05:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5914.597098 secs 2024-07-26T13:05:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5914.597098 secs 2024-07-26T13:05:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5914.597098 secs 2024-07-26T13:05:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:57 smithi134 ceph-mon[31966]: pgmap v3930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:57 smithi189 ceph-mon[36790]: pgmap v3930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:05:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:57 smithi193 ceph-mon[37473]: pgmap v3930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5900.171749 secs 2024-07-26T13:06:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:05:59 smithi134 ceph-mon[31966]: pgmap v3931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5900.171749 secs 2024-07-26T13:06:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:05:59 smithi189 ceph-mon[36790]: pgmap v3931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5900.171749 secs 2024-07-26T13:06:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:05:59 smithi193 ceph-mon[37473]: pgmap v3931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:01 smithi134 ceph-mon[31966]: pgmap v3932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5919.597257 secs 2024-07-26T13:06:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:01 smithi189 ceph-mon[36790]: pgmap v3932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5919.597257 secs 2024-07-26T13:06:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:01 smithi193 ceph-mon[37473]: pgmap v3932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5919.597257 secs 2024-07-26T13:06:03.159 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:03.158+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:06:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:03.158+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:03.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:03.158+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:06:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:03 smithi134 ceph-mon[31966]: pgmap v3933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:03 smithi189 ceph-mon[36790]: pgmap v3933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:03 smithi193 ceph-mon[37473]: pgmap v3933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5905.171869 secs 2024-07-26T13:06:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5905.171869 secs 2024-07-26T13:06:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5905.171869 secs 2024-07-26T13:06:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:05 smithi134 ceph-mon[31966]: pgmap v3934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:05 smithi189 ceph-mon[36790]: pgmap v3934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:05 smithi193 ceph-mon[37473]: pgmap v3934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5924.597368 secs 2024-07-26T13:06:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5924.597368 secs 2024-07-26T13:06:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5924.597368 secs 2024-07-26T13:06:07.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:07.682+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:06:07.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:07.682+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:06:07.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:07.682+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:07 smithi134 ceph-mon[31966]: pgmap v3935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:07 smithi189 ceph-mon[36790]: pgmap v3935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:07 smithi193 ceph-mon[37473]: pgmap v3935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5910.171971 secs 2024-07-26T13:06:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:09 smithi134 ceph-mon[31966]: pgmap v3936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5910.171971 secs 2024-07-26T13:06:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:09 smithi189 ceph-mon[36790]: pgmap v3936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5910.171971 secs 2024-07-26T13:06:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:09 smithi193 ceph-mon[37473]: pgmap v3936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:11 smithi134 ceph-mon[31966]: pgmap v3937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5929.597492 secs 2024-07-26T13:06:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:11 smithi189 ceph-mon[36790]: pgmap v3937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5929.597492 secs 2024-07-26T13:06:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:11 smithi193 ceph-mon[37473]: pgmap v3937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5929.597492 secs 2024-07-26T13:06:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:13 smithi134 ceph-mon[31966]: pgmap v3938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:13 smithi189 ceph-mon[36790]: pgmap v3938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:13 smithi193 ceph-mon[37473]: pgmap v3938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5915.172115 secs 2024-07-26T13:06:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5915.172115 secs 2024-07-26T13:06:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5915.172115 secs 2024-07-26T13:06:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:15 smithi134 ceph-mon[31966]: pgmap v3939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:15 smithi189 ceph-mon[36790]: pgmap v3939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:15 smithi193 ceph-mon[37473]: pgmap v3939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5934.597619 secs 2024-07-26T13:06:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5934.597619 secs 2024-07-26T13:06:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5934.597619 secs 2024-07-26T13:06:18.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:18.167+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:06:18.169 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:18.167+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:18.169 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:18.167+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:06:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:17 smithi189 ceph-mon[36790]: pgmap v3940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:17 smithi193 ceph-mon[37473]: pgmap v3940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:17 smithi134 ceph-mon[31966]: pgmap v3940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5920.172235 secs 2024-07-26T13:06:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:19 smithi189 ceph-mon[36790]: pgmap v3941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5920.172235 secs 2024-07-26T13:06:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:19 smithi193 ceph-mon[37473]: pgmap v3941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5920.172235 secs 2024-07-26T13:06:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:19 smithi134 ceph-mon[31966]: pgmap v3941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:21 smithi189 ceph-mon[36790]: pgmap v3942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5939.597711 secs 2024-07-26T13:06:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:21 smithi193 ceph-mon[37473]: pgmap v3942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5939.597711 secs 2024-07-26T13:06:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:21 smithi134 ceph-mon[31966]: pgmap v3942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5939.597711 secs 2024-07-26T13:06:22.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:22.698+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:06:22.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:22.698+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:06:22.700 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:22.698+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:23.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:23.359+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T13:06:23.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:23.359+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T13:06:23.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:23.359+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T13:06:23.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:23.359+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T13:06:23.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:23.359+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T13:06:23.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:23.360+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T13:06:23.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:23.360+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T13:06:23.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:23.361+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T13:06:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:23 smithi189 ceph-mon[36790]: pgmap v3943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:23 smithi193 ceph-mon[37473]: pgmap v3943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:23 smithi134 ceph-mon[31966]: pgmap v3943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5925.172357 secs 2024-07-26T13:06:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5925.172357 secs 2024-07-26T13:06:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5925.172357 secs 2024-07-26T13:06:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:25 smithi193 ceph-mon[37473]: pgmap v3944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:25 smithi134 ceph-mon[31966]: pgmap v3944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:25 smithi189 ceph-mon[36790]: pgmap v3944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:06:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5944.597850 secs 2024-07-26T13:06:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5944.597850 secs 2024-07-26T13:06:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5944.597850 secs 2024-07-26T13:06:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:27 smithi193 ceph-mon[37473]: pgmap v3945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:27 smithi134 ceph-mon[31966]: pgmap v3945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:27 smithi189 ceph-mon[36790]: pgmap v3945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:06:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:06:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:06:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5930.172479 secs 2024-07-26T13:06:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:29 smithi189 ceph-mon[36790]: pgmap v3946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5930.172479 secs 2024-07-26T13:06:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:29 smithi193 ceph-mon[37473]: pgmap v3946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5930.172479 secs 2024-07-26T13:06:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:29 smithi134 ceph-mon[31966]: pgmap v3946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:30.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:30.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:31 smithi193 ceph-mon[37473]: pgmap v3947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:31 smithi134 ceph-mon[31966]: pgmap v3947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:31 smithi189 ceph-mon[36790]: pgmap v3947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5949.597920 secs 2024-07-26T13:06:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5949.597920 secs 2024-07-26T13:06:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:33.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:33.182+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:06:33.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:33.182+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:06:33.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:33.182+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5949.597920 secs 2024-07-26T13:06:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:33.875+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:06:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:33.875+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T13:06:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:33.875+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:06:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:33.875+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T13:06:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:33.875+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:06:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:33.875+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:06:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:33.876+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:06:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:33.876+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:06:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:33 smithi193 ceph-mon[37473]: pgmap v3948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:33 smithi134 ceph-mon[31966]: pgmap v3948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:33 smithi189 ceph-mon[36790]: pgmap v3948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5935.172538 secs 2024-07-26T13:06:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:06:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:06:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5935.172538 secs 2024-07-26T13:06:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:06:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:06:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5935.172538 secs 2024-07-26T13:06:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:06:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:06:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:06:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:35 smithi193 ceph-mon[37473]: pgmap v3949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T13:06:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:35 smithi193 ceph-mon[37473]: pgmap v3950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:35 smithi134 ceph-mon[31966]: pgmap v3949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T13:06:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:35 smithi134 ceph-mon[31966]: pgmap v3950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:35 smithi189 ceph-mon[36790]: pgmap v3949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T13:06:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:35 smithi189 ceph-mon[36790]: pgmap v3950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:37.715 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:37.713+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:06:37.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:37.713+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:06:37.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:37.713+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:37.974 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5954.598062 secs 2024-07-26T13:06:37.975 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:37 smithi134 ceph-mon[31966]: pgmap v3951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5954.598062 secs 2024-07-26T13:06:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:37 smithi193 ceph-mon[37473]: pgmap v3951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5954.598062 secs 2024-07-26T13:06:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:37 smithi189 ceph-mon[36790]: pgmap v3951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:39 smithi134 ceph-mon[31966]: pgmap v3952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5940.172675 secs 2024-07-26T13:06:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:39 smithi193 ceph-mon[37473]: pgmap v3952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5940.172675 secs 2024-07-26T13:06:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:39 smithi189 ceph-mon[36790]: pgmap v3952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5940.172675 secs 2024-07-26T13:06:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:41 smithi193 ceph-mon[37473]: pgmap v3953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:41 smithi134 ceph-mon[31966]: pgmap v3953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:41 smithi189 ceph-mon[36790]: pgmap v3953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:06:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5959.598204 secs 2024-07-26T13:06:43.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5959.598204 secs 2024-07-26T13:06:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5959.598204 secs 2024-07-26T13:06:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:43 smithi134 ceph-mon[31966]: pgmap v3954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T13:06:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:43 smithi189 ceph-mon[36790]: pgmap v3954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T13:06:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:43 smithi193 ceph-mon[37473]: pgmap v3954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T13:06:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5945.172802 secs 2024-07-26T13:06:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5945.172802 secs 2024-07-26T13:06:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5945.172802 secs 2024-07-26T13:06:46.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:45 smithi134 ceph-mon[31966]: pgmap v3955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:45 smithi189 ceph-mon[36790]: pgmap v3955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:45 smithi193 ceph-mon[37473]: pgmap v3955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:06:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5964.598351 secs 2024-07-26T13:06:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5964.598351 secs 2024-07-26T13:06:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5964.598351 secs 2024-07-26T13:06:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:47 smithi134 ceph-mon[31966]: pgmap v3956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:48.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:48.184+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:06:48.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:48.184+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:06:48.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:06:48.184+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:47 smithi189 ceph-mon[36790]: pgmap v3956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:47 smithi193 ceph-mon[37473]: pgmap v3956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:49 smithi134 ceph-mon[31966]: pgmap v3957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5950.172898 secs 2024-07-26T13:06:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:49 smithi189 ceph-mon[36790]: pgmap v3957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5950.172898 secs 2024-07-26T13:06:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:49 smithi193 ceph-mon[37473]: pgmap v3957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:06:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5950.172898 secs 2024-07-26T13:06:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:51 smithi134 ceph-mon[31966]: pgmap v3958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5969.598407 secs 2024-07-26T13:06:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:51 smithi189 ceph-mon[36790]: pgmap v3958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5969.598407 secs 2024-07-26T13:06:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:51 smithi193 ceph-mon[37473]: pgmap v3958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5969.598407 secs 2024-07-26T13:06:52.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:52.721+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:06:52.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:52.721+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:06:52.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:06:52.721+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:06:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:53 smithi134 ceph-mon[31966]: pgmap v3959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:06:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:53 smithi189 ceph-mon[36790]: pgmap v3959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:06:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:53 smithi193 ceph-mon[37473]: pgmap v3959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:06:55.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5955.173021 secs 2024-07-26T13:06:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5955.173021 secs 2024-07-26T13:06:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5955.173021 secs 2024-07-26T13:06:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:55 smithi134 ceph-mon[31966]: pgmap v3960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:55 smithi189 ceph-mon[36790]: pgmap v3960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:55 smithi193 ceph-mon[37473]: pgmap v3960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:06:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5974.598526 secs 2024-07-26T13:06:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5974.598526 secs 2024-07-26T13:06:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5974.598526 secs 2024-07-26T13:06:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:57 smithi134 ceph-mon[31966]: pgmap v3961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:57 smithi189 ceph-mon[36790]: pgmap v3961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:06:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:57 smithi193 ceph-mon[37473]: pgmap v3961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:59 smithi134 ceph-mon[31966]: pgmap v3962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:06:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5960.173117 secs 2024-07-26T13:07:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:59 smithi189 ceph-mon[36790]: pgmap v3962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:06:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5960.173117 secs 2024-07-26T13:07:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:59 smithi193 ceph-mon[37473]: pgmap v3962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:06:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5960.173117 secs 2024-07-26T13:07:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:01 smithi134 ceph-mon[31966]: pgmap v3963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5979.598639 secs 2024-07-26T13:07:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:01 smithi189 ceph-mon[36790]: pgmap v3963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5979.598639 secs 2024-07-26T13:07:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:01 smithi193 ceph-mon[37473]: pgmap v3963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5979.598639 secs 2024-07-26T13:07:03.186 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:07:03.185+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:07:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:07:03.185+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:07:03.187 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:07:03.185+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:03 smithi134 ceph-mon[31966]: pgmap v3964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:03 smithi189 ceph-mon[36790]: pgmap v3964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:03 smithi193 ceph-mon[37473]: pgmap v3964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5965.173184 secs 2024-07-26T13:07:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5965.173184 secs 2024-07-26T13:07:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5965.173184 secs 2024-07-26T13:07:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:05 smithi134 ceph-mon[31966]: pgmap v3965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:05 smithi189 ceph-mon[36790]: pgmap v3965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:05 smithi193 ceph-mon[37473]: pgmap v3965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5984.598782 secs 2024-07-26T13:07:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5984.598782 secs 2024-07-26T13:07:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5984.598782 secs 2024-07-26T13:07:07.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:07:07.737+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:07:07.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:07:07.737+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:07:07.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:07:07.737+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:07 smithi134 ceph-mon[31966]: pgmap v3966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:07 smithi189 ceph-mon[36790]: pgmap v3966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:07 smithi193 ceph-mon[37473]: pgmap v3966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:10.176 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:09 smithi134 ceph-mon[31966]: pgmap v3967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:10.176 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5970.173306 secs 2024-07-26T13:07:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:09 smithi189 ceph-mon[36790]: pgmap v3967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5970.173306 secs 2024-07-26T13:07:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:09 smithi193 ceph-mon[37473]: pgmap v3967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5970.173306 secs 2024-07-26T13:07:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:11 smithi134 ceph-mon[31966]: pgmap v3968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5989.598860 secs 2024-07-26T13:07:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:11 smithi189 ceph-mon[36790]: pgmap v3968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5989.598860 secs 2024-07-26T13:07:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:11 smithi193 ceph-mon[37473]: pgmap v3968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5989.598860 secs 2024-07-26T13:07:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:13 smithi134 ceph-mon[31966]: pgmap v3969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:13 smithi189 ceph-mon[36790]: pgmap v3969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:13 smithi193 ceph-mon[37473]: pgmap v3969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5975.173422 secs 2024-07-26T13:07:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5975.173422 secs 2024-07-26T13:07:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5975.173422 secs 2024-07-26T13:07:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:15 smithi189 ceph-mon[36790]: pgmap v3970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:16.232 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:15 smithi134 ceph-mon[31966]: pgmap v3970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:15 smithi193 ceph-mon[37473]: pgmap v3970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:07:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5994.598927 secs 2024-07-26T13:07:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5994.598927 secs 2024-07-26T13:07:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5994.598927 secs 2024-07-26T13:07:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:07:18.200+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:07:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:07:18.201+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:07:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:07:18.201+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:17 smithi193 ceph-mon[37473]: pgmap v3971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:18.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:17 smithi134 ceph-mon[31966]: pgmap v3971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:17 smithi189 ceph-mon[36790]: pgmap v3971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:19 smithi193 ceph-mon[37473]: pgmap v3972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5980.173545 secs 2024-07-26T13:07:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:19 smithi134 ceph-mon[31966]: pgmap v3972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5980.173545 secs 2024-07-26T13:07:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:19 smithi189 ceph-mon[36790]: pgmap v3972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5980.173545 secs 2024-07-26T13:07:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:21 smithi193 ceph-mon[37473]: pgmap v3973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 5999.599008 secs 2024-07-26T13:07:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:21 smithi134 ceph-mon[31966]: pgmap v3973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 5999.599008 secs 2024-07-26T13:07:22.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:21 smithi189 ceph-mon[36790]: pgmap v3973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:22.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 5999.599008 secs 2024-07-26T13:07:22.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:07:22.752+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:07:22.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:07:22.752+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:07:22.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:07:22.752+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:23 smithi193 ceph-mon[37473]: pgmap v3974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:07:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:23 smithi134 ceph-mon[31966]: pgmap v3974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:07:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:23 smithi189 ceph-mon[36790]: pgmap v3974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:07:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5985.173663 secs 2024-07-26T13:07:25.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5985.173663 secs 2024-07-26T13:07:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5985.173663 secs 2024-07-26T13:07:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:25 smithi193 ceph-mon[37473]: pgmap v3975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:25 smithi134 ceph-mon[31966]: pgmap v3975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:25 smithi189 ceph-mon[36790]: pgmap v3975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6004.599102 secs 2024-07-26T13:07:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6004.599102 secs 2024-07-26T13:07:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6004.599102 secs 2024-07-26T13:07:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:27 smithi193 ceph-mon[37473]: pgmap v3976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:07:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:27 smithi134 ceph-mon[31966]: pgmap v3976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:07:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:27 smithi189 ceph-mon[36790]: pgmap v3976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:07:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:30 smithi193 ceph-mon[37473]: pgmap v3977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:07:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5990.173788 secs 2024-07-26T13:07:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:30 smithi134 ceph-mon[31966]: pgmap v3977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:07:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5990.173788 secs 2024-07-26T13:07:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:30 smithi189 ceph-mon[36790]: pgmap v3977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:07:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5990.173788 secs 2024-07-26T13:07:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:32 smithi193 ceph-mon[37473]: pgmap v3978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:07:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6009.599226 secs 2024-07-26T13:07:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:32 smithi134 ceph-mon[31966]: pgmap v3978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:07:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6009.599226 secs 2024-07-26T13:07:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:32 smithi189 ceph-mon[36790]: pgmap v3978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:07:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6009.599226 secs 2024-07-26T13:07:33.217 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:07:33.216+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:07:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:07:33.216+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:07:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:07:33.216+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:34 smithi193 ceph-mon[37473]: pgmap v3979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:07:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:34 smithi134 ceph-mon[31966]: pgmap v3979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:07:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:34 smithi189 ceph-mon[36790]: pgmap v3979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:07:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 5995.173917 secs 2024-07-26T13:07:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:07:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 5995.173917 secs 2024-07-26T13:07:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:07:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 5995.173917 secs 2024-07-26T13:07:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:07:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:36 smithi134 ceph-mon[31966]: pgmap v3980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:07:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:07:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:07:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:36 smithi189 ceph-mon[36790]: pgmap v3980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:07:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:07:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:07:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:36 smithi193 ceph-mon[37473]: pgmap v3980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:07:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:07:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:07:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:07:37.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:37 smithi134 ceph-mon[31966]: pgmap v3981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:07:37.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:37 smithi134 ceph-mon[31966]: pgmap v3982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:07:37.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6014.599376 secs 2024-07-26T13:07:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:37 smithi189 ceph-mon[36790]: pgmap v3981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:07:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:37 smithi189 ceph-mon[36790]: pgmap v3982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:07:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6014.599376 secs 2024-07-26T13:07:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:37 smithi193 ceph-mon[37473]: pgmap v3981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:07:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:37 smithi193 ceph-mon[37473]: pgmap v3982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:07:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6014.599376 secs 2024-07-26T13:07:37.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:07:37.767+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:07:37.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:07:37.768+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:07:37.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:07:37.768+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:39 smithi134 ceph-mon[31966]: pgmap v3983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2024-07-26T13:07:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:39 smithi189 ceph-mon[36790]: pgmap v3983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2024-07-26T13:07:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:39 smithi193 ceph-mon[37473]: pgmap v3983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2024-07-26T13:07:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6000.174036 secs 2024-07-26T13:07:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6000.174036 secs 2024-07-26T13:07:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6000.174036 secs 2024-07-26T13:07:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:41 smithi134 ceph-mon[31966]: pgmap v3984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:07:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:41 smithi189 ceph-mon[36790]: pgmap v3984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:07:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:41 smithi193 ceph-mon[37473]: pgmap v3984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:07:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6019.599443 secs 2024-07-26T13:07:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6019.599443 secs 2024-07-26T13:07:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6019.599443 secs 2024-07-26T13:07:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:43 smithi134 ceph-mon[31966]: pgmap v3985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:07:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:43 smithi189 ceph-mon[36790]: pgmap v3985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:07:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:43 smithi193 ceph-mon[37473]: pgmap v3985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:07:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6005.174147 secs 2024-07-26T13:07:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:45 smithi134 ceph-mon[31966]: pgmap v3986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:07:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6005.174147 secs 2024-07-26T13:07:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:45 smithi189 ceph-mon[36790]: pgmap v3986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:07:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6005.174147 secs 2024-07-26T13:07:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:45 smithi193 ceph-mon[37473]: pgmap v3986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:07:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:47 smithi134 ceph-mon[31966]: pgmap v3987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:07:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6024.599575 secs 2024-07-26T13:07:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:47 smithi189 ceph-mon[36790]: pgmap v3987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:07:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6024.599575 secs 2024-07-26T13:07:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:47 smithi193 ceph-mon[37473]: pgmap v3987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:07:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6024.599575 secs 2024-07-26T13:07:48.229 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:07:48.228+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:07:48.229 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:07:48.228+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:07:48.229 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:07:48.228+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:49 smithi134 ceph-mon[31966]: pgmap v3988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:49 smithi189 ceph-mon[36790]: pgmap v3988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:49 smithi193 ceph-mon[37473]: pgmap v3988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6010.174242 secs 2024-07-26T13:07:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6010.174242 secs 2024-07-26T13:07:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6010.174242 secs 2024-07-26T13:07:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:51 smithi134 ceph-mon[31966]: pgmap v3989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:07:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:51 smithi189 ceph-mon[36790]: pgmap v3989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:07:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:51 smithi193 ceph-mon[37473]: pgmap v3989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:07:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6029.599640 secs 2024-07-26T13:07:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6029.599640 secs 2024-07-26T13:07:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6029.599640 secs 2024-07-26T13:07:52.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:07:52.770+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:07:52.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:07:52.770+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:07:52.772 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:07:52.770+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:07:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:53 smithi134 ceph-mon[31966]: pgmap v3990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:53 smithi189 ceph-mon[36790]: pgmap v3990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:53 smithi193 ceph-mon[37473]: pgmap v3990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6015.174325 secs 2024-07-26T13:07:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:55 smithi134 ceph-mon[31966]: pgmap v3991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6015.174325 secs 2024-07-26T13:07:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:55 smithi189 ceph-mon[36790]: pgmap v3991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6015.174325 secs 2024-07-26T13:07:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:55 smithi193 ceph-mon[37473]: pgmap v3991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:07:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:57 smithi134 ceph-mon[31966]: pgmap v3992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6034.599766 secs 2024-07-26T13:07:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:57 smithi189 ceph-mon[36790]: pgmap v3992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6034.599766 secs 2024-07-26T13:07:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:57 smithi193 ceph-mon[37473]: pgmap v3992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:07:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6034.599766 secs 2024-07-26T13:07:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:07:59 smithi134 ceph-mon[31966]: pgmap v3993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:07:59 smithi189 ceph-mon[36790]: pgmap v3993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:07:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:07:59 smithi193 ceph-mon[37473]: pgmap v3993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6020.174445 secs 2024-07-26T13:08:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6020.174445 secs 2024-07-26T13:08:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6020.174445 secs 2024-07-26T13:08:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:01 smithi134 ceph-mon[31966]: pgmap v3994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:08:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:01 smithi189 ceph-mon[36790]: pgmap v3994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:08:01.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:01 smithi193 ceph-mon[37473]: pgmap v3994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:08:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6039.599886 secs 2024-07-26T13:08:02.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6039.599886 secs 2024-07-26T13:08:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6039.599886 secs 2024-07-26T13:08:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:08:03.243+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:08:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:08:03.243+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:03.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:08:03.243+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:08:03.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:03 smithi134 ceph-mon[31966]: pgmap v3995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:03 smithi189 ceph-mon[36790]: pgmap v3995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:03 smithi193 ceph-mon[37473]: pgmap v3995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6025.174573 secs 2024-07-26T13:08:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:05 smithi134 ceph-mon[31966]: pgmap v3996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6025.174573 secs 2024-07-26T13:08:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:05 smithi189 ceph-mon[36790]: pgmap v3996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6025.174573 secs 2024-07-26T13:08:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:05 smithi193 ceph-mon[37473]: pgmap v3996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:07 smithi134 ceph-mon[31966]: pgmap v3997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6044.600026 secs 2024-07-26T13:08:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:07 smithi189 ceph-mon[36790]: pgmap v3997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6044.600026 secs 2024-07-26T13:08:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:07 smithi193 ceph-mon[37473]: pgmap v3997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6044.600026 secs 2024-07-26T13:08:07.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:08:07.771+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:08:07.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:08:07.771+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:08:07.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:08:07.771+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:08 smithi134 ceph-mon[31966]: pgmap v3998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:08 smithi189 ceph-mon[36790]: pgmap v3998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:08 smithi193 ceph-mon[37473]: pgmap v3998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6030.174673 secs 2024-07-26T13:08:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6030.174673 secs 2024-07-26T13:08:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6030.174673 secs 2024-07-26T13:08:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:10 smithi134 ceph-mon[31966]: pgmap v3999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:08:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:10 smithi189 ceph-mon[36790]: pgmap v3999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:08:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:10 smithi193 ceph-mon[37473]: pgmap v3999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:08:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6049.600168 secs 2024-07-26T13:08:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:12 smithi134 ceph-mon[31966]: pgmap v4000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6049.600168 secs 2024-07-26T13:08:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:12 smithi189 ceph-mon[36790]: pgmap v4000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6049.600168 secs 2024-07-26T13:08:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:12 smithi193 ceph-mon[37473]: pgmap v4000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6035.174798 secs 2024-07-26T13:08:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:14 smithi134 ceph-mon[31966]: pgmap v4001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6035.174798 secs 2024-07-26T13:08:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:14 smithi189 ceph-mon[36790]: pgmap v4001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6035.174798 secs 2024-07-26T13:08:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:14 smithi193 ceph-mon[37473]: pgmap v4001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:16 smithi134 ceph-mon[31966]: pgmap v4002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:16 smithi189 ceph-mon[36790]: pgmap v4002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:16 smithi193 ceph-mon[37473]: pgmap v4002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6054.600272 secs 2024-07-26T13:08:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6054.600272 secs 2024-07-26T13:08:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6054.600272 secs 2024-07-26T13:08:18.260 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:08:18.258+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:08:18.260 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:08:18.259+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:08:18.260 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:08:18.259+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:18 smithi134 ceph-mon[31966]: pgmap v4003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:18 smithi189 ceph-mon[36790]: pgmap v4003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:18 smithi193 ceph-mon[37473]: pgmap v4003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:08:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6040.174957 secs 2024-07-26T13:08:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6040.174957 secs 2024-07-26T13:08:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6040.174957 secs 2024-07-26T13:08:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:20 smithi134 ceph-mon[31966]: pgmap v4004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:08:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:20 smithi189 ceph-mon[36790]: pgmap v4004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:08:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:20 smithi193 ceph-mon[37473]: pgmap v4004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:08:22.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:08:22.783+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:08:22.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:08:22.783+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:08:22.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:08:22.783+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6059.600395 secs 2024-07-26T13:08:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:22 smithi134 ceph-mon[31966]: pgmap v4005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6059.600395 secs 2024-07-26T13:08:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:22 smithi189 ceph-mon[36790]: pgmap v4005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6059.600395 secs 2024-07-26T13:08:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:22 smithi193 ceph-mon[37473]: pgmap v4005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6045.175059 secs 2024-07-26T13:08:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:24 smithi134 ceph-mon[31966]: pgmap v4006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6045.175059 secs 2024-07-26T13:08:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:24 smithi189 ceph-mon[36790]: pgmap v4006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6045.175059 secs 2024-07-26T13:08:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:24 smithi193 ceph-mon[37473]: pgmap v4006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:26 smithi134 ceph-mon[31966]: pgmap v4007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:26 smithi189 ceph-mon[36790]: pgmap v4007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:26 smithi193 ceph-mon[37473]: pgmap v4007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6064.600515 secs 2024-07-26T13:08:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6064.600515 secs 2024-07-26T13:08:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6064.600515 secs 2024-07-26T13:08:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:28 smithi134 ceph-mon[31966]: pgmap v4008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:28 smithi189 ceph-mon[36790]: pgmap v4008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:28 smithi193 ceph-mon[37473]: pgmap v4008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:08:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6050.175151 secs 2024-07-26T13:08:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6050.175151 secs 2024-07-26T13:08:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6050.175151 secs 2024-07-26T13:08:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:30 smithi134 ceph-mon[31966]: pgmap v4009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:08:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:30 smithi189 ceph-mon[36790]: pgmap v4009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:08:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:30 smithi193 ceph-mon[37473]: pgmap v4009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T13:08:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6069.600631 secs 2024-07-26T13:08:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:32 smithi134 ceph-mon[31966]: pgmap v4010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6069.600631 secs 2024-07-26T13:08:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:32 smithi189 ceph-mon[36790]: pgmap v4010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6069.600631 secs 2024-07-26T13:08:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:32 smithi193 ceph-mon[37473]: pgmap v4010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:33.268 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:08:33.267+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:08:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:08:33.267+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:08:33.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:08:33.267+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6055.175273 secs 2024-07-26T13:08:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:34 smithi189 ceph-mon[36790]: pgmap v4011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6055.175273 secs 2024-07-26T13:08:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:34 smithi193 ceph-mon[37473]: pgmap v4011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:35.081 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6055.175273 secs 2024-07-26T13:08:35.081 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:34 smithi134 ceph-mon[31966]: pgmap v4011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:36 smithi189 ceph-mon[36790]: pgmap v4012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:08:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:36 smithi193 ceph-mon[37473]: pgmap v4012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:08:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:36 smithi134 ceph-mon[31966]: pgmap v4012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:08:37.796 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:08:37.795+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:08:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:08:37.795+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:08:37.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:08:37.795+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6074.600696 secs 2024-07-26T13:08:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:08:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:08:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:37 smithi189 ceph-mon[36790]: pgmap v4013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:08:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:37 smithi189 ceph-mon[36790]: pgmap v4014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:08:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:08:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6074.600696 secs 2024-07-26T13:08:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:08:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:08:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:37 smithi193 ceph-mon[37473]: pgmap v4013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:08:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:37 smithi193 ceph-mon[37473]: pgmap v4014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:08:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:08:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6074.600696 secs 2024-07-26T13:08:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:08:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:08:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:37 smithi134 ceph-mon[31966]: pgmap v4013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:08:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:37 smithi134 ceph-mon[31966]: pgmap v4014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:08:38.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:08:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6060.175388 secs 2024-07-26T13:08:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:39 smithi134 ceph-mon[31966]: pgmap v4015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6060.175388 secs 2024-07-26T13:08:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:39 smithi189 ceph-mon[36790]: pgmap v4015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6060.175388 secs 2024-07-26T13:08:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:39 smithi193 ceph-mon[37473]: pgmap v4015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:41 smithi134 ceph-mon[31966]: pgmap v4016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6079.600819 secs 2024-07-26T13:08:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:41 smithi189 ceph-mon[36790]: pgmap v4016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6079.600819 secs 2024-07-26T13:08:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:41 smithi193 ceph-mon[37473]: pgmap v4016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6079.600819 secs 2024-07-26T13:08:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:43 smithi134 ceph-mon[31966]: pgmap v4017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:43 smithi189 ceph-mon[36790]: pgmap v4017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:43 smithi193 ceph-mon[37473]: pgmap v4017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:08:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6065.175502 secs 2024-07-26T13:08:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6065.175502 secs 2024-07-26T13:08:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6065.175502 secs 2024-07-26T13:08:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:45 smithi134 ceph-mon[31966]: pgmap v4018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:08:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:45 smithi189 ceph-mon[36790]: pgmap v4018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:08:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:45 smithi193 ceph-mon[37473]: pgmap v4018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:08:47.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6084.600940 secs 2024-07-26T13:08:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6084.600940 secs 2024-07-26T13:08:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6084.600940 secs 2024-07-26T13:08:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:47 smithi134 ceph-mon[31966]: pgmap v4019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:08:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:47 smithi189 ceph-mon[36790]: pgmap v4019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:08:48.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:47 smithi193 ceph-mon[37473]: pgmap v4019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:08:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:08:48.283+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:08:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:08:48.283+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:08:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:08:48.283+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6070.175614 secs 2024-07-26T13:08:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:49 smithi134 ceph-mon[31966]: pgmap v4020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6070.175614 secs 2024-07-26T13:08:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:49 smithi189 ceph-mon[36790]: pgmap v4020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6070.175614 secs 2024-07-26T13:08:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:49 smithi193 ceph-mon[37473]: pgmap v4020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:51 smithi134 ceph-mon[31966]: pgmap v4021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6089.601069 secs 2024-07-26T13:08:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:51 smithi189 ceph-mon[36790]: pgmap v4021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6089.601069 secs 2024-07-26T13:08:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:51 smithi193 ceph-mon[37473]: pgmap v4021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6089.601069 secs 2024-07-26T13:08:52.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:08:52.811+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:08:52.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:08:52.811+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:08:52.812 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:08:52.811+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:08:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:53 smithi134 ceph-mon[31966]: pgmap v4022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:53 smithi189 ceph-mon[36790]: pgmap v4022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:53 smithi193 ceph-mon[37473]: pgmap v4022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:08:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6075.175733 secs 2024-07-26T13:08:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6075.175733 secs 2024-07-26T13:08:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6075.175733 secs 2024-07-26T13:08:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:55 smithi134 ceph-mon[31966]: pgmap v4023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:08:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:55 smithi189 ceph-mon[36790]: pgmap v4023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:08:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:55 smithi193 ceph-mon[37473]: pgmap v4023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:08:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6094.601215 secs 2024-07-26T13:08:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6094.601215 secs 2024-07-26T13:08:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6094.601215 secs 2024-07-26T13:08:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:57 smithi134 ceph-mon[31966]: pgmap v4024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:57 smithi189 ceph-mon[36790]: pgmap v4024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:08:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:57 smithi193 ceph-mon[37473]: pgmap v4024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6080.175854 secs 2024-07-26T13:09:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:08:59 smithi189 ceph-mon[36790]: pgmap v4025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6080.175854 secs 2024-07-26T13:09:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:08:59 smithi193 ceph-mon[37473]: pgmap v4025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6080.175854 secs 2024-07-26T13:09:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:08:59 smithi134 ceph-mon[31966]: pgmap v4025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:01 smithi189 ceph-mon[36790]: pgmap v4026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6099.601277 secs 2024-07-26T13:09:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:01 smithi193 ceph-mon[37473]: pgmap v4026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6099.601277 secs 2024-07-26T13:09:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:01 smithi134 ceph-mon[31966]: pgmap v4026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6099.601277 secs 2024-07-26T13:09:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:09:03.298+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:09:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:09:03.298+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:09:03.300 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:09:03.298+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:03 smithi189 ceph-mon[36790]: pgmap v4027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:03 smithi193 ceph-mon[37473]: pgmap v4027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:03 smithi134 ceph-mon[31966]: pgmap v4027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6085.176007 secs 2024-07-26T13:09:05.288 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6085.176007 secs 2024-07-26T13:09:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6085.176007 secs 2024-07-26T13:09:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:05 smithi193 ceph-mon[37473]: pgmap v4028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:06.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:05 smithi134 ceph-mon[31966]: pgmap v4028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:05 smithi189 ceph-mon[36790]: pgmap v4028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6104.601374 secs 2024-07-26T13:09:07.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6104.601374 secs 2024-07-26T13:09:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6104.601374 secs 2024-07-26T13:09:07.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:09:07.818+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:09:07.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:09:07.818+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:09:07.820 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:09:07.818+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:07 smithi134 ceph-mon[31966]: pgmap v4029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:07 smithi193 ceph-mon[37473]: pgmap v4029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:07 smithi189 ceph-mon[36790]: pgmap v4029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6090.176119 secs 2024-07-26T13:09:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:09 smithi193 ceph-mon[37473]: pgmap v4030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6090.176119 secs 2024-07-26T13:09:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:09 smithi134 ceph-mon[31966]: pgmap v4030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6090.176119 secs 2024-07-26T13:09:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:09 smithi189 ceph-mon[36790]: pgmap v4030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:11 smithi193 ceph-mon[37473]: pgmap v4031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6109.601497 secs 2024-07-26T13:09:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:11 smithi134 ceph-mon[31966]: pgmap v4031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6109.601497 secs 2024-07-26T13:09:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:11 smithi189 ceph-mon[36790]: pgmap v4031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6109.601497 secs 2024-07-26T13:09:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:13 smithi193 ceph-mon[37473]: pgmap v4032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:13 smithi134 ceph-mon[31966]: pgmap v4032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:13 smithi189 ceph-mon[36790]: pgmap v4032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6095.176212 secs 2024-07-26T13:09:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6095.176212 secs 2024-07-26T13:09:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6095.176212 secs 2024-07-26T13:09:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:15 smithi193 ceph-mon[37473]: pgmap v4033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:16.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:15 smithi134 ceph-mon[31966]: pgmap v4033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:15 smithi189 ceph-mon[36790]: pgmap v4033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6114.601562 secs 2024-07-26T13:09:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6114.601562 secs 2024-07-26T13:09:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6114.601562 secs 2024-07-26T13:09:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:18 smithi193 ceph-mon[37473]: pgmap v4034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:18.315 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:09:18.314+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:09:18.315 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:09:18.314+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:09:18.315 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:09:18.314+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:18 smithi134 ceph-mon[31966]: pgmap v4034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:18 smithi189 ceph-mon[36790]: pgmap v4034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6100.176296 secs 2024-07-26T13:09:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:20 smithi193 ceph-mon[37473]: pgmap v4035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6100.176296 secs 2024-07-26T13:09:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:20 smithi134 ceph-mon[31966]: pgmap v4035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6100.176296 secs 2024-07-26T13:09:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:20 smithi189 ceph-mon[36790]: pgmap v4035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:22 smithi134 ceph-mon[31966]: pgmap v4036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6119.601646 secs 2024-07-26T13:09:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:22 smithi189 ceph-mon[36790]: pgmap v4036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6119.601646 secs 2024-07-26T13:09:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:22 smithi193 ceph-mon[37473]: pgmap v4036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6119.601646 secs 2024-07-26T13:09:22.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:09:22.830+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:09:22.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:09:22.830+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:22.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:09:22.830+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:09:23.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:09:23.390+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:09:23.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:09:23.390+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:09:23.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:09:23.390+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:09:23.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:09:23.391+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:09:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:24 smithi134 ceph-mon[31966]: pgmap v4037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:24 smithi189 ceph-mon[36790]: pgmap v4037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:24 smithi193 ceph-mon[37473]: pgmap v4037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6105.176420 secs 2024-07-26T13:09:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6105.176420 secs 2024-07-26T13:09:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6105.176420 secs 2024-07-26T13:09:26.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:26 smithi189 ceph-mon[36790]: pgmap v4038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:26 smithi134 ceph-mon[31966]: pgmap v4038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:26 smithi193 ceph-mon[37473]: pgmap v4038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6124.601730 secs 2024-07-26T13:09:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6124.601730 secs 2024-07-26T13:09:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6124.601730 secs 2024-07-26T13:09:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:28 smithi134 ceph-mon[31966]: pgmap v4039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:28 smithi189 ceph-mon[36790]: pgmap v4039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:28 smithi193 ceph-mon[37473]: pgmap v4039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6110.176558 secs 2024-07-26T13:09:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:30 smithi134 ceph-mon[31966]: pgmap v4040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6110.176558 secs 2024-07-26T13:09:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:30 smithi189 ceph-mon[36790]: pgmap v4040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6110.176558 secs 2024-07-26T13:09:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:30 smithi193 ceph-mon[37473]: pgmap v4040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:32.325 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:32 smithi193 ceph-mon[37473]: pgmap v4041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:32.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6129.601874 secs 2024-07-26T13:09:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:32 smithi134 ceph-mon[31966]: pgmap v4041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6129.601874 secs 2024-07-26T13:09:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:32 smithi189 ceph-mon[36790]: pgmap v4041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6129.601874 secs 2024-07-26T13:09:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:09:33.315+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:09:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:09:33.315+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:09:33.316 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:09:33.315+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:33.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:09:33.907+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:09:33.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:09:33.907+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:09:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:09:33.908+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:09:33.909 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:09:33.908+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:09:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:34 smithi134 ceph-mon[31966]: pgmap v4042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:34 smithi189 ceph-mon[36790]: pgmap v4042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:34.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:34 smithi193 ceph-mon[37473]: pgmap v4042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6115.176698 secs 2024-07-26T13:09:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6115.176698 secs 2024-07-26T13:09:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6115.176698 secs 2024-07-26T13:09:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:36 smithi134 ceph-mon[31966]: pgmap v4043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:36 smithi189 ceph-mon[36790]: pgmap v4043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:36 smithi193 ceph-mon[37473]: pgmap v4043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:37.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6134.601982 secs 2024-07-26T13:09:37.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:09:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6134.601982 secs 2024-07-26T13:09:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:09:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6134.601982 secs 2024-07-26T13:09:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:09:37.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:09:37.845+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:09:37.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:09:37.845+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:37.847 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:09:37.845+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:09:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:38 smithi134 ceph-mon[31966]: pgmap v4044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:09:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:09:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:09:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:38 smithi189 ceph-mon[36790]: pgmap v4044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:09:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:09:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:09:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:38 smithi193 ceph-mon[37473]: pgmap v4044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:09:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:09:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:09:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:39 smithi134 ceph-mon[31966]: pgmap v4045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T13:09:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:39 smithi189 ceph-mon[36790]: pgmap v4045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T13:09:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:39 smithi193 ceph-mon[37473]: pgmap v4045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T13:09:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6120.176837 secs 2024-07-26T13:09:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6120.176837 secs 2024-07-26T13:09:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6120.176837 secs 2024-07-26T13:09:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:41 smithi134 ceph-mon[31966]: pgmap v4046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:09:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:41 smithi189 ceph-mon[36790]: pgmap v4046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:09:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:41 smithi193 ceph-mon[37473]: pgmap v4046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:09:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6139.602107 secs 2024-07-26T13:09:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6139.602107 secs 2024-07-26T13:09:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6139.602107 secs 2024-07-26T13:09:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:43 smithi134 ceph-mon[31966]: pgmap v4047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:09:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:43 smithi189 ceph-mon[36790]: pgmap v4047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:09:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:43 smithi193 ceph-mon[37473]: pgmap v4047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:09:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6125.176988 secs 2024-07-26T13:09:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:45 smithi134 ceph-mon[31966]: pgmap v4048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:09:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6125.176988 secs 2024-07-26T13:09:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:45 smithi189 ceph-mon[36790]: pgmap v4048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:09:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6125.176988 secs 2024-07-26T13:09:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:45 smithi193 ceph-mon[37473]: pgmap v4048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:09:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6144.602208 secs 2024-07-26T13:09:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:47 smithi134 ceph-mon[31966]: pgmap v4049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:09:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6144.602208 secs 2024-07-26T13:09:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:47 smithi189 ceph-mon[36790]: pgmap v4049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:09:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6144.602208 secs 2024-07-26T13:09:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:47 smithi193 ceph-mon[37473]: pgmap v4049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:09:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:09:48.330+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:09:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:09:48.330+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:09:48.331 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:09:48.330+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:49 smithi134 ceph-mon[31966]: pgmap v4050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T13:09:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:49 smithi189 ceph-mon[36790]: pgmap v4050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T13:09:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:49 smithi193 ceph-mon[37473]: pgmap v4050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T13:09:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6130.177116 secs 2024-07-26T13:09:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6130.177116 secs 2024-07-26T13:09:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6130.177116 secs 2024-07-26T13:09:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:51 smithi189 ceph-mon[36790]: pgmap v4051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:51 smithi193 ceph-mon[37473]: pgmap v4051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:51 smithi134 ceph-mon[31966]: pgmap v4051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6149.602337 secs 2024-07-26T13:09:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6149.602337 secs 2024-07-26T13:09:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6149.602337 secs 2024-07-26T13:09:52.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:09:52.849+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:09:52.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:09:52.849+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:09:52.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:09:52.849+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:09:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:53 smithi189 ceph-mon[36790]: pgmap v4052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:53 smithi193 ceph-mon[37473]: pgmap v4052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:53 smithi134 ceph-mon[31966]: pgmap v4052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:09:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:54 smithi193 ceph-mon[37473]: pgmap v4053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:54 smithi134 ceph-mon[31966]: pgmap v4053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:54 smithi189 ceph-mon[36790]: pgmap v4053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6135.177231 secs 2024-07-26T13:09:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6135.177231 secs 2024-07-26T13:09:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6135.177231 secs 2024-07-26T13:09:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:56 smithi193 ceph-mon[37473]: pgmap v4054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:56.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:56 smithi189 ceph-mon[36790]: pgmap v4054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:56 smithi134 ceph-mon[31966]: pgmap v4054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:09:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6154.602427 secs 2024-07-26T13:09:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6154.602427 secs 2024-07-26T13:09:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6154.602427 secs 2024-07-26T13:09:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:58 smithi193 ceph-mon[37473]: pgmap v4055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:58.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:58 smithi134 ceph-mon[31966]: pgmap v4055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:58 smithi189 ceph-mon[36790]: pgmap v4055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:09:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:09:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6140.177315 secs 2024-07-26T13:09:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:09:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6140.177315 secs 2024-07-26T13:09:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:09:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6140.177315 secs 2024-07-26T13:10:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:00 smithi134 ceph-mon[31966]: pgmap v4056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:10:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:00 smithi189 ceph-mon[36790]: pgmap v4056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:10:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:00 smithi193 ceph-mon[37473]: pgmap v4056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:10:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6159.602566 secs 2024-07-26T13:10:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6159.602566 secs 2024-07-26T13:10:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6159.602566 secs 2024-07-26T13:10:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:02 smithi134 ceph-mon[31966]: pgmap v4057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:02 smithi189 ceph-mon[36790]: pgmap v4057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:02 smithi193 ceph-mon[37473]: pgmap v4057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:10:03.342+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:10:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:10:03.342+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:03.343 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:10:03.342+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:10:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6145.177378 secs 2024-07-26T13:10:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:05 smithi134 ceph-mon[31966]: pgmap v4058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6145.177378 secs 2024-07-26T13:10:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:05 smithi189 ceph-mon[36790]: pgmap v4058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6145.177378 secs 2024-07-26T13:10:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:05 smithi193 ceph-mon[37473]: pgmap v4058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6164.602688 secs 2024-07-26T13:10:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:07 smithi134 ceph-mon[31966]: pgmap v4059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6164.602688 secs 2024-07-26T13:10:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:07 smithi189 ceph-mon[36790]: pgmap v4059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6164.602688 secs 2024-07-26T13:10:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:07 smithi193 ceph-mon[37473]: pgmap v4059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:07.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:10:07.864+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:10:07.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:10:07.865+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:10:07.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:10:07.865+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:09 smithi134 ceph-mon[31966]: pgmap v4060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:09 smithi189 ceph-mon[36790]: pgmap v4060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:09 smithi193 ceph-mon[37473]: pgmap v4060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6150.177501 secs 2024-07-26T13:10:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6150.177501 secs 2024-07-26T13:10:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6150.177501 secs 2024-07-26T13:10:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:11 smithi134 ceph-mon[31966]: pgmap v4061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:11 smithi189 ceph-mon[36790]: pgmap v4061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:11 smithi193 ceph-mon[37473]: pgmap v4061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6169.602819 secs 2024-07-26T13:10:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6169.602819 secs 2024-07-26T13:10:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6169.602819 secs 2024-07-26T13:10:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:13 smithi134 ceph-mon[31966]: pgmap v4062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:13 smithi189 ceph-mon[36790]: pgmap v4062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:13 smithi193 ceph-mon[37473]: pgmap v4062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6155.177636 secs 2024-07-26T13:10:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:15 smithi134 ceph-mon[31966]: pgmap v4063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6155.177636 secs 2024-07-26T13:10:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:15 smithi189 ceph-mon[36790]: pgmap v4063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6155.177636 secs 2024-07-26T13:10:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:15 smithi193 ceph-mon[37473]: pgmap v4063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6174.602948 secs 2024-07-26T13:10:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:17 smithi134 ceph-mon[31966]: pgmap v4064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6174.602948 secs 2024-07-26T13:10:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:17 smithi189 ceph-mon[36790]: pgmap v4064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6174.602948 secs 2024-07-26T13:10:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:17 smithi193 ceph-mon[37473]: pgmap v4064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:10:18.359+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:10:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:10:18.359+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:10:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:10:18.359+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:19 smithi134 ceph-mon[31966]: pgmap v4065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:19 smithi189 ceph-mon[36790]: pgmap v4065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:19 smithi193 ceph-mon[37473]: pgmap v4065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6160.177724 secs 2024-07-26T13:10:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6160.177724 secs 2024-07-26T13:10:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6160.177724 secs 2024-07-26T13:10:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:21 smithi134 ceph-mon[31966]: pgmap v4066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:21 smithi189 ceph-mon[36790]: pgmap v4066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:21 smithi193 ceph-mon[37473]: pgmap v4066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6179.603068 secs 2024-07-26T13:10:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6179.603068 secs 2024-07-26T13:10:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6179.603068 secs 2024-07-26T13:10:22.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:10:22.880+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:10:22.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:10:22.880+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:22.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:10:22.880+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:10:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:23 smithi134 ceph-mon[31966]: pgmap v4067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:23 smithi189 ceph-mon[36790]: pgmap v4067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:23 smithi193 ceph-mon[37473]: pgmap v4067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6165.177848 secs 2024-07-26T13:10:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:25 smithi134 ceph-mon[31966]: pgmap v4068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6165.177848 secs 2024-07-26T13:10:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:25 smithi189 ceph-mon[36790]: pgmap v4068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6165.177848 secs 2024-07-26T13:10:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:25 smithi193 ceph-mon[37473]: pgmap v4068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6184.603196 secs 2024-07-26T13:10:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:27 smithi134 ceph-mon[31966]: pgmap v4069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6184.603196 secs 2024-07-26T13:10:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:27 smithi189 ceph-mon[36790]: pgmap v4069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6184.603196 secs 2024-07-26T13:10:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:27 smithi193 ceph-mon[37473]: pgmap v4069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:29 smithi134 ceph-mon[31966]: pgmap v4070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:29 smithi189 ceph-mon[36790]: pgmap v4070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:29 smithi193 ceph-mon[37473]: pgmap v4070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6170.177967 secs 2024-07-26T13:10:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6170.177967 secs 2024-07-26T13:10:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6170.177967 secs 2024-07-26T13:10:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:31 smithi189 ceph-mon[36790]: pgmap v4071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:31 smithi193 ceph-mon[37473]: pgmap v4071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:31 smithi134 ceph-mon[31966]: pgmap v4071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6189.603344 secs 2024-07-26T13:10:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6189.603344 secs 2024-07-26T13:10:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6189.603344 secs 2024-07-26T13:10:33.375 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:10:33.373+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:10:33.375 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:10:33.374+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:10:33.375 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:10:33.374+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:33 smithi189 ceph-mon[36790]: pgmap v4072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:33 smithi193 ceph-mon[37473]: pgmap v4072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:33 smithi134 ceph-mon[31966]: pgmap v4072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:10:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:34 smithi193 ceph-mon[37473]: pgmap v4073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:34 smithi134 ceph-mon[31966]: pgmap v4073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:34 smithi189 ceph-mon[36790]: pgmap v4073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6175.178036 secs 2024-07-26T13:10:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6175.178036 secs 2024-07-26T13:10:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6175.178036 secs 2024-07-26T13:10:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:36 smithi193 ceph-mon[37473]: pgmap v4074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:36 smithi134 ceph-mon[31966]: pgmap v4074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:36 smithi189 ceph-mon[36790]: pgmap v4074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6194.603400 secs 2024-07-26T13:10:37.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6194.603400 secs 2024-07-26T13:10:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6194.603400 secs 2024-07-26T13:10:37.890 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:10:37.888+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:10:37.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:10:37.888+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:37.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:10:37.888+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:10:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:38 smithi193 ceph-mon[37473]: pgmap v4075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:10:38.581 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:38 smithi134 ceph-mon[31966]: pgmap v4075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:38.581 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:10:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:38 smithi189 ceph-mon[36790]: pgmap v4075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:10:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:10:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:10:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:39 smithi134 ceph-mon[31966]: pgmap v4076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:10:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:39 smithi134 ceph-mon[31966]: pgmap v4077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:10:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:10:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:10:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:10:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:39 smithi189 ceph-mon[36790]: pgmap v4076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:10:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:39 smithi189 ceph-mon[36790]: pgmap v4077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:10:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:10:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:10:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:10:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:39 smithi193 ceph-mon[37473]: pgmap v4076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:10:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:39 smithi193 ceph-mon[37473]: pgmap v4077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:10:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:10:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6180.178103 secs 2024-07-26T13:10:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6180.178103 secs 2024-07-26T13:10:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6180.178103 secs 2024-07-26T13:10:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:41 smithi134 ceph-mon[31966]: pgmap v4078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:10:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:41 smithi189 ceph-mon[36790]: pgmap v4078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:10:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:41 smithi193 ceph-mon[37473]: pgmap v4078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:10:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6199.603489 secs 2024-07-26T13:10:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6199.603489 secs 2024-07-26T13:10:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6199.603489 secs 2024-07-26T13:10:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:43 smithi134 ceph-mon[31966]: pgmap v4079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:10:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:43 smithi189 ceph-mon[36790]: pgmap v4079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:10:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:43 smithi193 ceph-mon[37473]: pgmap v4079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:10:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6185.178191 secs 2024-07-26T13:10:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6185.178191 secs 2024-07-26T13:10:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6185.178191 secs 2024-07-26T13:10:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:45 smithi134 ceph-mon[31966]: pgmap v4080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:10:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:45 smithi189 ceph-mon[36790]: pgmap v4080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:10:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:45 smithi193 ceph-mon[37473]: pgmap v4080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:10:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6204.603564 secs 2024-07-26T13:10:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6204.603564 secs 2024-07-26T13:10:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6204.603564 secs 2024-07-26T13:10:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:47 smithi134 ceph-mon[31966]: pgmap v4081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:10:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:47 smithi189 ceph-mon[36790]: pgmap v4081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:10:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:47 smithi193 ceph-mon[37473]: pgmap v4081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:10:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:10:48.376+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:10:48.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:10:48.376+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:10:48.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:10:48.376+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6190.178290 secs 2024-07-26T13:10:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:50 smithi189 ceph-mon[36790]: pgmap v4082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:10:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6190.178290 secs 2024-07-26T13:10:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:50 smithi193 ceph-mon[37473]: pgmap v4082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:10:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6190.178290 secs 2024-07-26T13:10:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:50 smithi134 ceph-mon[31966]: pgmap v4082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:10:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6209.603631 secs 2024-07-26T13:10:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:52 smithi189 ceph-mon[36790]: pgmap v4083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6209.603631 secs 2024-07-26T13:10:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:52 smithi193 ceph-mon[37473]: pgmap v4083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6209.603631 secs 2024-07-26T13:10:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:52 smithi134 ceph-mon[31966]: pgmap v4083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:52.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:10:52.903+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:10:52.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:10:52.903+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:10:52.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:10:52.903+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:10:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:54 smithi189 ceph-mon[36790]: pgmap v4084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:54 smithi193 ceph-mon[37473]: pgmap v4084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:54 smithi134 ceph-mon[31966]: pgmap v4084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:10:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6195.178411 secs 2024-07-26T13:10:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6195.178411 secs 2024-07-26T13:10:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6195.178411 secs 2024-07-26T13:10:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:56 smithi193 ceph-mon[37473]: pgmap v4085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:56 smithi134 ceph-mon[31966]: pgmap v4085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:56 smithi189 ceph-mon[36790]: pgmap v4085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:10:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6214.603712 secs 2024-07-26T13:10:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:10:57 smithi193 ceph-mon[37473]: pgmap v4086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:57.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6214.603712 secs 2024-07-26T13:10:57.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:10:57 smithi189 ceph-mon[36790]: pgmap v4086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:10:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6214.603712 secs 2024-07-26T13:10:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:10:57 smithi134 ceph-mon[31966]: pgmap v4086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6200.178506 secs 2024-07-26T13:11:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:00 smithi189 ceph-mon[36790]: pgmap v4087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6200.178506 secs 2024-07-26T13:11:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:00 smithi193 ceph-mon[37473]: pgmap v4087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6200.178506 secs 2024-07-26T13:11:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:00 smithi134 ceph-mon[31966]: pgmap v4087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6219.603830 secs 2024-07-26T13:11:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:02 smithi189 ceph-mon[36790]: pgmap v4088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6219.603830 secs 2024-07-26T13:11:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:02 smithi193 ceph-mon[37473]: pgmap v4088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6219.603830 secs 2024-07-26T13:11:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:02 smithi134 ceph-mon[31966]: pgmap v4088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:03.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:11:03.392+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:11:03.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:11:03.392+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:11:03.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:11:03.392+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:04 smithi189 ceph-mon[36790]: pgmap v4089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:04 smithi193 ceph-mon[37473]: pgmap v4089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:04 smithi134 ceph-mon[31966]: pgmap v4089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6205.178645 secs 2024-07-26T13:11:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6205.178645 secs 2024-07-26T13:11:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6205.178645 secs 2024-07-26T13:11:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:06 smithi193 ceph-mon[37473]: pgmap v4090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:06 smithi134 ceph-mon[31966]: pgmap v4090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:06 smithi189 ceph-mon[36790]: pgmap v4090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6224.603925 secs 2024-07-26T13:11:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:07 smithi193 ceph-mon[37473]: pgmap v4091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:07.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6224.603925 secs 2024-07-26T13:11:07.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:07 smithi134 ceph-mon[31966]: pgmap v4091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6224.603925 secs 2024-07-26T13:11:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:07 smithi189 ceph-mon[36790]: pgmap v4091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:07.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:11:07.904+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:11:07.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:11:07.904+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:11:07.906 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:11:07.904+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6210.178745 secs 2024-07-26T13:11:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:10 smithi189 ceph-mon[36790]: pgmap v4092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6210.178745 secs 2024-07-26T13:11:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:10 smithi193 ceph-mon[37473]: pgmap v4092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6210.178745 secs 2024-07-26T13:11:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:10 smithi134 ceph-mon[31966]: pgmap v4092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6229.604041 secs 2024-07-26T13:11:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:12 smithi189 ceph-mon[36790]: pgmap v4093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6229.604041 secs 2024-07-26T13:11:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:12 smithi193 ceph-mon[37473]: pgmap v4093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6229.604041 secs 2024-07-26T13:11:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:12 smithi134 ceph-mon[31966]: pgmap v4093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:14 smithi193 ceph-mon[37473]: pgmap v4094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:14 smithi134 ceph-mon[31966]: pgmap v4094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:14 smithi189 ceph-mon[36790]: pgmap v4094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6215.178865 secs 2024-07-26T13:11:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:15 smithi193 ceph-mon[37473]: pgmap v4095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6215.178865 secs 2024-07-26T13:11:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:15 smithi134 ceph-mon[31966]: pgmap v4095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6215.178865 secs 2024-07-26T13:11:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:15 smithi189 ceph-mon[36790]: pgmap v4095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6234.604174 secs 2024-07-26T13:11:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6234.604174 secs 2024-07-26T13:11:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6234.604174 secs 2024-07-26T13:11:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:17 smithi134 ceph-mon[31966]: pgmap v4096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:17 smithi189 ceph-mon[36790]: pgmap v4096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:17 smithi193 ceph-mon[37473]: pgmap v4096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:18.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:11:18.407+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:11:18.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:11:18.407+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:11:18.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:11:18.407+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6220.178994 secs 2024-07-26T13:11:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:20 smithi189 ceph-mon[36790]: pgmap v4097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6220.178994 secs 2024-07-26T13:11:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:20 smithi193 ceph-mon[37473]: pgmap v4097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:20.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6220.178994 secs 2024-07-26T13:11:20.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:20 smithi134 ceph-mon[31966]: pgmap v4097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6239.604343 secs 2024-07-26T13:11:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:22 smithi193 ceph-mon[37473]: pgmap v4098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6239.604343 secs 2024-07-26T13:11:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:22 smithi134 ceph-mon[31966]: pgmap v4098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6239.604343 secs 2024-07-26T13:11:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:22 smithi189 ceph-mon[36790]: pgmap v4098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:22.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:11:22.920+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:11:22.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:11:22.920+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:11:22.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:11:22.920+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:23 smithi193 ceph-mon[37473]: pgmap v4099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:23 smithi134 ceph-mon[31966]: pgmap v4099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:23 smithi189 ceph-mon[36790]: pgmap v4099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6225.179057 secs 2024-07-26T13:11:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6225.179057 secs 2024-07-26T13:11:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6225.179057 secs 2024-07-26T13:11:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:25 smithi134 ceph-mon[31966]: pgmap v4100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:25 smithi189 ceph-mon[36790]: pgmap v4100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:25 smithi193 ceph-mon[37473]: pgmap v4100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6244.604431 secs 2024-07-26T13:11:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6244.604431 secs 2024-07-26T13:11:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6244.604431 secs 2024-07-26T13:11:28.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:27 smithi134 ceph-mon[31966]: pgmap v4101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:27 smithi189 ceph-mon[36790]: pgmap v4101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:27 smithi193 ceph-mon[37473]: pgmap v4101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6230.179179 secs 2024-07-26T13:11:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:30 smithi193 ceph-mon[37473]: pgmap v4102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6230.179179 secs 2024-07-26T13:11:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:30 smithi134 ceph-mon[31966]: pgmap v4102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6230.179179 secs 2024-07-26T13:11:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:30 smithi189 ceph-mon[36790]: pgmap v4102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:31 smithi193 ceph-mon[37473]: pgmap v4103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:31 smithi134 ceph-mon[31966]: pgmap v4103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:31 smithi189 ceph-mon[36790]: pgmap v4103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6249.604555 secs 2024-07-26T13:11:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6249.604555 secs 2024-07-26T13:11:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6249.604555 secs 2024-07-26T13:11:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:11:33.408+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:11:33.410 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:11:33.408+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:33.410 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:11:33.408+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:11:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:33 smithi193 ceph-mon[37473]: pgmap v4104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:33.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:33 smithi134 ceph-mon[31966]: pgmap v4104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:33 smithi189 ceph-mon[36790]: pgmap v4104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6235.179333 secs 2024-07-26T13:11:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6235.179333 secs 2024-07-26T13:11:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6235.179333 secs 2024-07-26T13:11:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:35 smithi134 ceph-mon[31966]: pgmap v4105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:35 smithi189 ceph-mon[36790]: pgmap v4105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:35 smithi193 ceph-mon[37473]: pgmap v4105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6254.604643 secs 2024-07-26T13:11:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6254.604643 secs 2024-07-26T13:11:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6254.604643 secs 2024-07-26T13:11:37.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:11:37.929+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:11:37.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:11:37.929+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:11:37.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:11:37.929+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:37 smithi134 ceph-mon[31966]: pgmap v4106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:37 smithi189 ceph-mon[36790]: pgmap v4106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:37 smithi193 ceph-mon[37473]: pgmap v4106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:11:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:39 smithi193 ceph-mon[37473]: pgmap v4107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:39 smithi134 ceph-mon[31966]: pgmap v4107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:39 smithi189 ceph-mon[36790]: pgmap v4107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:11:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6240.179397 secs 2024-07-26T13:11:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:11:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6240.179397 secs 2024-07-26T13:11:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:11:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6240.179397 secs 2024-07-26T13:11:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:11:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:41 smithi193 ceph-mon[37473]: pgmap v4108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:41.581 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:41 smithi134 ceph-mon[31966]: pgmap v4108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:41 smithi189 ceph-mon[36790]: pgmap v4108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6259.604794 secs 2024-07-26T13:11:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6259.604794 secs 2024-07-26T13:11:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6259.604794 secs 2024-07-26T13:11:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:43 smithi134 ceph-mon[31966]: pgmap v4109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:43 smithi189 ceph-mon[36790]: pgmap v4109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:43 smithi193 ceph-mon[37473]: pgmap v4109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6245.179516 secs 2024-07-26T13:11:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6245.179516 secs 2024-07-26T13:11:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6245.179516 secs 2024-07-26T13:11:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:45 smithi134 ceph-mon[31966]: pgmap v4110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:45 smithi189 ceph-mon[36790]: pgmap v4110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:45 smithi193 ceph-mon[37473]: pgmap v4110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:11:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:11:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:11:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:11:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:11:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:46 smithi134 ceph-mon[31966]: pgmap v4111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:11:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:46 smithi134 ceph-mon[31966]: pgmap v4112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:11:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:11:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:11:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:11:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:11:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:11:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:46 smithi189 ceph-mon[36790]: pgmap v4111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:11:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:46 smithi189 ceph-mon[36790]: pgmap v4112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:11:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:11:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:11:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:11:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:11:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:11:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:46 smithi193 ceph-mon[37473]: pgmap v4111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:11:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:46 smithi193 ceph-mon[37473]: pgmap v4112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:11:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:11:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6264.604919 secs 2024-07-26T13:11:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6264.604919 secs 2024-07-26T13:11:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6264.604919 secs 2024-07-26T13:11:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:11:48.423+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:11:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:11:48.424+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:11:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:11:48.424+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:48 smithi134 ceph-mon[31966]: pgmap v4113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T13:11:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:48 smithi189 ceph-mon[36790]: pgmap v4113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T13:11:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:48 smithi193 ceph-mon[37473]: pgmap v4113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T13:11:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6250.179651 secs 2024-07-26T13:11:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6250.179651 secs 2024-07-26T13:11:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6250.179651 secs 2024-07-26T13:11:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:50 smithi134 ceph-mon[31966]: pgmap v4114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:11:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:50 smithi189 ceph-mon[36790]: pgmap v4114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:11:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:50 smithi193 ceph-mon[37473]: pgmap v4114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:11:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6269.605064 secs 2024-07-26T13:11:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:52 smithi134 ceph-mon[31966]: pgmap v4115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T13:11:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6269.605064 secs 2024-07-26T13:11:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:52 smithi189 ceph-mon[36790]: pgmap v4115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T13:11:52.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:11:52.945+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:11:52.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:11:52.945+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:11:52.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:11:52.945+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:11:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6269.605064 secs 2024-07-26T13:11:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:52 smithi193 ceph-mon[37473]: pgmap v4115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T13:11:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6255.179786 secs 2024-07-26T13:11:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:54 smithi134 ceph-mon[31966]: pgmap v4116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:11:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6255.179786 secs 2024-07-26T13:11:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:54 smithi189 ceph-mon[36790]: pgmap v4116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:11:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6255.179786 secs 2024-07-26T13:11:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:54 smithi193 ceph-mon[37473]: pgmap v4116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:11:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:56 smithi134 ceph-mon[31966]: pgmap v4117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:56 smithi189 ceph-mon[36790]: pgmap v4117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:56 smithi193 ceph-mon[37473]: pgmap v4117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:11:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6274.605153 secs 2024-07-26T13:11:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6274.605153 secs 2024-07-26T13:11:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6274.605153 secs 2024-07-26T13:11:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:58 smithi134 ceph-mon[31966]: pgmap v4118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:58 smithi189 ceph-mon[36790]: pgmap v4118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:58 smithi193 ceph-mon[37473]: pgmap v4118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:11:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:11:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6260.179908 secs 2024-07-26T13:11:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:11:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6260.179908 secs 2024-07-26T13:12:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:11:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6260.179908 secs 2024-07-26T13:12:00.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:00 smithi134 ceph-mon[31966]: pgmap v4119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:12:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:00 smithi189 ceph-mon[36790]: pgmap v4119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:12:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:00 smithi193 ceph-mon[37473]: pgmap v4119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:12:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6279.605337 secs 2024-07-26T13:12:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:02 smithi134 ceph-mon[31966]: pgmap v4120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6279.605337 secs 2024-07-26T13:12:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:02 smithi189 ceph-mon[36790]: pgmap v4120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6279.605337 secs 2024-07-26T13:12:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:02 smithi193 ceph-mon[37473]: pgmap v4120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:03.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:03.429+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:12:03.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:03.429+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:12:03.431 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:03.429+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6265.179964 secs 2024-07-26T13:12:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:04 smithi189 ceph-mon[36790]: pgmap v4121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:05.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6265.179964 secs 2024-07-26T13:12:05.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:04 smithi193 ceph-mon[37473]: pgmap v4121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6265.179964 secs 2024-07-26T13:12:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:04 smithi134 ceph-mon[31966]: pgmap v4121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:06 smithi193 ceph-mon[37473]: pgmap v4122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:06 smithi134 ceph-mon[31966]: pgmap v4122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:06 smithi189 ceph-mon[36790]: pgmap v4122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:07.963 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6284.605498 secs 2024-07-26T13:12:07.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:07.960+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:12:07.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:07.960+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:12:07.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:07.960+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6284.605498 secs 2024-07-26T13:12:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6284.605498 secs 2024-07-26T13:12:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:08 smithi193 ceph-mon[37473]: pgmap v4123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:12:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:08 smithi134 ceph-mon[31966]: pgmap v4123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:12:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:08 smithi189 ceph-mon[36790]: pgmap v4123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:12:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6270.180094 secs 2024-07-26T13:12:10.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6270.180094 secs 2024-07-26T13:12:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6270.180094 secs 2024-07-26T13:12:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:10 smithi193 ceph-mon[37473]: pgmap v4124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:12:11.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:10 smithi134 ceph-mon[31966]: pgmap v4124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:12:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:10 smithi189 ceph-mon[36790]: pgmap v4124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T13:12:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6289.605634 secs 2024-07-26T13:12:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:12 smithi193 ceph-mon[37473]: pgmap v4125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:12:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6289.605634 secs 2024-07-26T13:12:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:12 smithi134 ceph-mon[31966]: pgmap v4125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:12:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6289.605634 secs 2024-07-26T13:12:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:12 smithi189 ceph-mon[36790]: pgmap v4125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:12:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6275.180161 secs 2024-07-26T13:12:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:14 smithi193 ceph-mon[37473]: pgmap v4126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:12:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6275.180161 secs 2024-07-26T13:12:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:14 smithi134 ceph-mon[31966]: pgmap v4126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:12:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6275.180161 secs 2024-07-26T13:12:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:14 smithi189 ceph-mon[36790]: pgmap v4126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:12:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:16 smithi193 ceph-mon[37473]: pgmap v4127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:12:17.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:16 smithi134 ceph-mon[31966]: pgmap v4127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:12:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:16 smithi189 ceph-mon[36790]: pgmap v4127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T13:12:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6294.605759 secs 2024-07-26T13:12:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6294.605759 secs 2024-07-26T13:12:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6294.605759 secs 2024-07-26T13:12:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:18.445+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:12:18.446 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:18.445+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:18.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:18.445+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:12:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:18 smithi193 ceph-mon[37473]: pgmap v4128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:12:19.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:18 smithi134 ceph-mon[31966]: pgmap v4128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:12:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:18 smithi189 ceph-mon[36790]: pgmap v4128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T13:12:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6280.180224 secs 2024-07-26T13:12:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6280.180224 secs 2024-07-26T13:12:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6280.180224 secs 2024-07-26T13:12:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:20 smithi193 ceph-mon[37473]: pgmap v4129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:12:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:20 smithi134 ceph-mon[31966]: pgmap v4129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:12:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:20 smithi189 ceph-mon[36790]: pgmap v4129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:12:22.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:22.307+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0360 msgr2=0x7f9b300c1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T13:12:22.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:22.307+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0360 msgr2=0x7f9b300c1530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:12:22.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:22.307+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0360 0x7f9b300c1530 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9b38013930 tx=0x7f9b3804e040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:12:22.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:22.307+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0360 0x7f9b300c1530 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9b38013930 tx=0x7f9b3804e040 comp rx=0 tx=0).stop 2024-07-26T13:12:22.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:22.307+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0360 msgr2=0x7f9b300c1530 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T13:12:22.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:22.307+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0360 msgr2=0x7f9b300c1530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:12:22.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:22.307+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0360 0x7f9b300c1530 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:12:22.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:22.307+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:12:22.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:22.308+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:12:22.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:22.310+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b3804e040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:12:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:22.961+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:12:22.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:22.961+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:22.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:22.961+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:12:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6299.605883 secs 2024-07-26T13:12:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:22 smithi193 ceph-mon[37473]: pgmap v4130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6299.605883 secs 2024-07-26T13:12:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:22 smithi134 ceph-mon[31966]: pgmap v4130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6299.605883 secs 2024-07-26T13:12:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:22 smithi189 ceph-mon[36790]: pgmap v4130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:23.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:23.420+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:12:23.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:23.421+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:12:23.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:23.422+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T13:12:23.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:23.422+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T13:12:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6285.180379 secs 2024-07-26T13:12:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:24 smithi134 ceph-mon[31966]: pgmap v4131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6285.180379 secs 2024-07-26T13:12:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:24 smithi189 ceph-mon[36790]: pgmap v4131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6285.180379 secs 2024-07-26T13:12:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:24 smithi193 ceph-mon[37473]: pgmap v4131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:26 smithi134 ceph-mon[31966]: pgmap v4132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:26 smithi189 ceph-mon[36790]: pgmap v4132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:26 smithi193 ceph-mon[37473]: pgmap v4132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:28.048 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6304.606028 secs 2024-07-26T13:12:28.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6304.606028 secs 2024-07-26T13:12:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6304.606028 secs 2024-07-26T13:12:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:28 smithi134 ceph-mon[31966]: pgmap v4133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:28 smithi189 ceph-mon[36790]: pgmap v4133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:28 smithi193 ceph-mon[37473]: pgmap v4133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6290.180522 secs 2024-07-26T13:12:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6290.180522 secs 2024-07-26T13:12:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6290.180522 secs 2024-07-26T13:12:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:30 smithi134 ceph-mon[31966]: pgmap v4134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:12:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:30 smithi189 ceph-mon[36790]: pgmap v4134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:12:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:30 smithi193 ceph-mon[37473]: pgmap v4134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:12:32.861 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:32.859+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 msgr2=0x7f21ac0c0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T13:12:32.861 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:32.859+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 msgr2=0x7f21ac0c0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:12:32.861 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:32.859+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 0x7f21ac0c0760 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f21980063a0 tx=0x7f21b80c9000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:12:32.861 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:32.859+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 0x7f21ac0c0760 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f21980063a0 tx=0x7f21b80c9000 comp rx=0 tx=0).stop 2024-07-26T13:12:32.861 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:32.859+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 msgr2=0x7f21ac0c0760 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T13:12:32.862 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:32.859+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 msgr2=0x7f21ac0c0760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:12:32.862 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:32.859+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0350 0x7f21ac0c0760 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:12:32.862 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:32.859+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c12c0 0x7f21ac0c16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:12:32.862 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:32.861+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c12c0 0x7f21ac0c16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:12:32.863 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:32.862+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c12c0 0x7f21ac0c16d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f2198005820 tx=0x7f21b80fc000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:12:33.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6309.606174 secs 2024-07-26T13:12:33.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:32 smithi193 ceph-mon[37473]: pgmap v4135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6309.606174 secs 2024-07-26T13:12:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:32 smithi134 ceph-mon[31966]: pgmap v4135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6309.606174 secs 2024-07-26T13:12:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:32 smithi189 ceph-mon[36790]: pgmap v4135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:33.447+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:12:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:33.447+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:12:33.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:33.447+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:33.939+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:12:33.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:33.939+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:12:33.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:33.940+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:12:33.941 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:33.940+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:12:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6295.180649 secs 2024-07-26T13:12:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:34 smithi134 ceph-mon[31966]: pgmap v4136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6295.180649 secs 2024-07-26T13:12:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:34 smithi189 ceph-mon[36790]: pgmap v4136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6295.180649 secs 2024-07-26T13:12:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:34 smithi193 ceph-mon[37473]: pgmap v4136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:36 smithi134 ceph-mon[31966]: pgmap v4137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6314.606326 secs 2024-07-26T13:12:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:36 smithi189 ceph-mon[36790]: pgmap v4137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6314.606326 secs 2024-07-26T13:12:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:36 smithi193 ceph-mon[37473]: pgmap v4137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6314.606326 secs 2024-07-26T13:12:37.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:37.965+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:12:37.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:37.965+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:12:37.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:37.965+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:38 smithi134 ceph-mon[31966]: pgmap v4138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:38 smithi189 ceph-mon[36790]: pgmap v4138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:38 smithi193 ceph-mon[37473]: pgmap v4138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6300.180778 secs 2024-07-26T13:12:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6300.180778 secs 2024-07-26T13:12:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6300.180778 secs 2024-07-26T13:12:41.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:40 smithi134 ceph-mon[31966]: pgmap v4139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:12:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:40 smithi189 ceph-mon[36790]: pgmap v4139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:12:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:40 smithi193 ceph-mon[37473]: pgmap v4139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:12:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6319.606407 secs 2024-07-26T13:12:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6319.606407 secs 2024-07-26T13:12:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6319.606407 secs 2024-07-26T13:12:43.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:42 smithi134 ceph-mon[31966]: pgmap v4140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:42 smithi189 ceph-mon[36790]: pgmap v4140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:42 smithi193 ceph-mon[37473]: pgmap v4140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6305.180903 secs 2024-07-26T13:12:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:44 smithi134 ceph-mon[31966]: pgmap v4141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6305.180903 secs 2024-07-26T13:12:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:44 smithi189 ceph-mon[36790]: pgmap v4141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6305.180903 secs 2024-07-26T13:12:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:44 smithi193 ceph-mon[37473]: pgmap v4141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:12:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:12:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:12:47.014 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:46 smithi189 ceph-mon[36790]: pgmap v4142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:47.014 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6324.606483 secs 2024-07-26T13:12:47.014 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:47.014 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:47.014 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:47.014 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:46 smithi193 ceph-mon[37473]: pgmap v4142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6324.606483 secs 2024-07-26T13:12:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:46 smithi134 ceph-mon[31966]: pgmap v4142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:12:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6324.606483 secs 2024-07-26T13:12:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:12:48.015 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:47 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:12:48.015 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:47 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:12:48.081 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:47 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:12:48.081 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:47 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:12:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:47 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:12:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:47 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:12:48.463 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:48.462+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:12:48.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:48.462+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:12:48.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:12:48.462+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:49.146 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:48 smithi189 ceph-mon[36790]: pgmap v4143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:49.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:48 smithi134 ceph-mon[31966]: pgmap v4143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:49.207 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:48 smithi193 ceph-mon[37473]: pgmap v4143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:12:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6310.180989 secs 2024-07-26T13:12:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:12:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:12:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:12:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:12:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:12:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:12:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:12:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6310.180989 secs 2024-07-26T13:12:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:12:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:12:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:12:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:12:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:12:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:12:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:12:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6310.180989 secs 2024-07-26T13:12:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:12:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:12:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:12:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:12:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:12:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:12:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:12:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:50 smithi134 ceph-mon[31966]: pgmap v4144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:12:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:50 smithi134 ceph-mon[31966]: pgmap v4145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:50 smithi189 ceph-mon[36790]: pgmap v4144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:12:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:50 smithi189 ceph-mon[36790]: pgmap v4145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:50 smithi193 ceph-mon[37473]: pgmap v4144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:12:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:50 smithi193 ceph-mon[37473]: pgmap v4145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6329.606563 secs 2024-07-26T13:12:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6329.606563 secs 2024-07-26T13:12:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6329.606563 secs 2024-07-26T13:12:52.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:52.978+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:12:52.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:52.978+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:12:52.980 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:12:52.978+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:12:53.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:52 smithi134 ceph-mon[31966]: pgmap v4146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:52 smithi189 ceph-mon[36790]: pgmap v4146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:52 smithi193 ceph-mon[37473]: pgmap v4146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6315.181094 secs 2024-07-26T13:12:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:54 smithi189 ceph-mon[36790]: pgmap v4147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6315.181094 secs 2024-07-26T13:12:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:54 smithi193 ceph-mon[37473]: pgmap v4147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6315.181094 secs 2024-07-26T13:12:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:54 smithi134 ceph-mon[31966]: pgmap v4147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:56 smithi189 ceph-mon[36790]: pgmap v4148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6334.606682 secs 2024-07-26T13:12:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:56 smithi193 ceph-mon[37473]: pgmap v4148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6334.606682 secs 2024-07-26T13:12:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:56 smithi134 ceph-mon[31966]: pgmap v4148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6334.606682 secs 2024-07-26T13:12:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:58 smithi189 ceph-mon[36790]: pgmap v4149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:58 smithi193 ceph-mon[37473]: pgmap v4149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:12:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:58 smithi134 ceph-mon[31966]: pgmap v4149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:13:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:12:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6320.181221 secs 2024-07-26T13:13:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:12:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6320.181221 secs 2024-07-26T13:13:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:12:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6320.181221 secs 2024-07-26T13:13:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:00 smithi193 ceph-mon[37473]: pgmap v4150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:13:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:00 smithi134 ceph-mon[31966]: pgmap v4150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:13:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:00 smithi189 ceph-mon[36790]: pgmap v4150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:13:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6339.606802 secs 2024-07-26T13:13:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6339.606802 secs 2024-07-26T13:13:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6339.606802 secs 2024-07-26T13:13:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:02 smithi193 ceph-mon[37473]: pgmap v4151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:02 smithi134 ceph-mon[31966]: pgmap v4151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:02 smithi189 ceph-mon[36790]: pgmap v4151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:13:03.478+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:13:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:13:03.478+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:13:03.479 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:13:03.478+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6325.181325 secs 2024-07-26T13:13:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:04 smithi193 ceph-mon[37473]: pgmap v4152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6325.181325 secs 2024-07-26T13:13:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:04 smithi134 ceph-mon[31966]: pgmap v4152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6325.181325 secs 2024-07-26T13:13:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:04 smithi189 ceph-mon[36790]: pgmap v4152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:06 smithi193 ceph-mon[37473]: pgmap v4153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6344.606862 secs 2024-07-26T13:13:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:06 smithi134 ceph-mon[31966]: pgmap v4153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6344.606862 secs 2024-07-26T13:13:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:06 smithi189 ceph-mon[36790]: pgmap v4153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6344.606862 secs 2024-07-26T13:13:07.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:13:07.994+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:13:07.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:13:07.994+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:07.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:13:07.994+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:13:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:08 smithi193 ceph-mon[37473]: pgmap v4154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:08 smithi134 ceph-mon[31966]: pgmap v4154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:08 smithi189 ceph-mon[36790]: pgmap v4154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6330.181438 secs 2024-07-26T13:13:10.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6330.181438 secs 2024-07-26T13:13:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6330.181438 secs 2024-07-26T13:13:11.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:10 smithi193 ceph-mon[37473]: pgmap v4155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:10 smithi134 ceph-mon[31966]: pgmap v4155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:10 smithi189 ceph-mon[36790]: pgmap v4155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6349.607020 secs 2024-07-26T13:13:12.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6349.607020 secs 2024-07-26T13:13:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6349.607020 secs 2024-07-26T13:13:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:13 smithi193 ceph-mon[37473]: pgmap v4156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:13 smithi134 ceph-mon[31966]: pgmap v4156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:13 smithi189 ceph-mon[36790]: pgmap v4156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6335.181586 secs 2024-07-26T13:13:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:15 smithi193 ceph-mon[37473]: pgmap v4157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6335.181586 secs 2024-07-26T13:13:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:15 smithi134 ceph-mon[31966]: pgmap v4157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6335.181586 secs 2024-07-26T13:13:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:15 smithi189 ceph-mon[36790]: pgmap v4157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:17 smithi193 ceph-mon[37473]: pgmap v4158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6354.607161 secs 2024-07-26T13:13:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:17 smithi134 ceph-mon[31966]: pgmap v4158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6354.607161 secs 2024-07-26T13:13:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:17 smithi189 ceph-mon[36790]: pgmap v4158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6354.607161 secs 2024-07-26T13:13:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:13:18.493+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:13:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:13:18.493+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:13:18.495 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:13:18.493+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:19 smithi134 ceph-mon[31966]: pgmap v4159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:19 smithi189 ceph-mon[36790]: pgmap v4159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:19 smithi193 ceph-mon[37473]: pgmap v4159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6340.181756 secs 2024-07-26T13:13:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6340.181756 secs 2024-07-26T13:13:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6340.181756 secs 2024-07-26T13:13:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:21 smithi134 ceph-mon[31966]: pgmap v4160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:21 smithi189 ceph-mon[36790]: pgmap v4160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:21 smithi193 ceph-mon[37473]: pgmap v4160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6359.607283 secs 2024-07-26T13:13:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6359.607283 secs 2024-07-26T13:13:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6359.607283 secs 2024-07-26T13:13:23.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:13:23.009+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:13:23.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:13:23.009+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:13:23.011 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:13:23.009+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:23.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:23 smithi134 ceph-mon[31966]: pgmap v4161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:23 smithi189 ceph-mon[36790]: pgmap v4161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:23 smithi193 ceph-mon[37473]: pgmap v4161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6345.181853 secs 2024-07-26T13:13:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:25 smithi134 ceph-mon[31966]: pgmap v4162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6345.181853 secs 2024-07-26T13:13:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:25 smithi189 ceph-mon[36790]: pgmap v4162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6345.181853 secs 2024-07-26T13:13:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:25 smithi193 ceph-mon[37473]: pgmap v4162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:27 smithi134 ceph-mon[31966]: pgmap v4163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6364.607366 secs 2024-07-26T13:13:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:27 smithi189 ceph-mon[36790]: pgmap v4163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6364.607366 secs 2024-07-26T13:13:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:27 smithi193 ceph-mon[37473]: pgmap v4163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6364.607366 secs 2024-07-26T13:13:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:29 smithi134 ceph-mon[31966]: pgmap v4164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:29 smithi189 ceph-mon[36790]: pgmap v4164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:29 smithi193 ceph-mon[37473]: pgmap v4164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:30.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6350.181973 secs 2024-07-26T13:13:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6350.181973 secs 2024-07-26T13:13:30.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6350.181973 secs 2024-07-26T13:13:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:31 smithi134 ceph-mon[31966]: pgmap v4165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:31 smithi189 ceph-mon[36790]: pgmap v4165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:31 smithi193 ceph-mon[37473]: pgmap v4165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6369.607493 secs 2024-07-26T13:13:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6369.607493 secs 2024-07-26T13:13:32.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6369.607493 secs 2024-07-26T13:13:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:33 smithi134 ceph-mon[31966]: pgmap v4166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:33.424 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:33 smithi193 ceph-mon[37473]: pgmap v4166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:33 smithi189 ceph-mon[36790]: pgmap v4166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:13:33.499+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:13:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:13:33.499+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:13:33.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:13:33.499+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6355.182046 secs 2024-07-26T13:13:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:35 smithi134 ceph-mon[31966]: pgmap v4167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6355.182046 secs 2024-07-26T13:13:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:35 smithi189 ceph-mon[36790]: pgmap v4167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6355.182046 secs 2024-07-26T13:13:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:35 smithi193 ceph-mon[37473]: pgmap v4167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:37 smithi134 ceph-mon[31966]: pgmap v4168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6374.607625 secs 2024-07-26T13:13:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:37 smithi189 ceph-mon[36790]: pgmap v4168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6374.607625 secs 2024-07-26T13:13:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:37 smithi193 ceph-mon[37473]: pgmap v4168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6374.607625 secs 2024-07-26T13:13:38.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:13:38.025+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:13:38.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:13:38.025+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:38.027 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:13:38.025+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:13:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:39 smithi134 ceph-mon[31966]: pgmap v4169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:39 smithi189 ceph-mon[36790]: pgmap v4169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:39 smithi193 ceph-mon[37473]: pgmap v4169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6360.182181 secs 2024-07-26T13:13:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6360.182181 secs 2024-07-26T13:13:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6360.182181 secs 2024-07-26T13:13:41.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:41 smithi134 ceph-mon[31966]: pgmap v4170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:41 smithi189 ceph-mon[36790]: pgmap v4170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:41 smithi193 ceph-mon[37473]: pgmap v4170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6379.607750 secs 2024-07-26T13:13:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6379.607750 secs 2024-07-26T13:13:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6379.607750 secs 2024-07-26T13:13:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:43 smithi134 ceph-mon[31966]: pgmap v4171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:43 smithi189 ceph-mon[36790]: pgmap v4171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:43 smithi193 ceph-mon[37473]: pgmap v4171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6365.182296 secs 2024-07-26T13:13:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:45 smithi134 ceph-mon[31966]: pgmap v4172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6365.182296 secs 2024-07-26T13:13:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:45 smithi189 ceph-mon[36790]: pgmap v4172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6365.182296 secs 2024-07-26T13:13:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:45 smithi193 ceph-mon[37473]: pgmap v4172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:47 smithi134 ceph-mon[31966]: pgmap v4173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6384.607879 secs 2024-07-26T13:13:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:47 smithi189 ceph-mon[36790]: pgmap v4173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6384.607879 secs 2024-07-26T13:13:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:47 smithi193 ceph-mon[37473]: pgmap v4173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:13:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6384.607879 secs 2024-07-26T13:13:48.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:13:48.515+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:13:48.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:13:48.515+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:13:48.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:13:48.515+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:49 smithi189 ceph-mon[36790]: pgmap v4174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:49 smithi193 ceph-mon[37473]: pgmap v4174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:49 smithi134 ceph-mon[31966]: pgmap v4174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:13:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6370.182383 secs 2024-07-26T13:13:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:13:50.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6370.182383 secs 2024-07-26T13:13:50.527 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:13:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6370.182383 secs 2024-07-26T13:13:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:13:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:51 smithi193 ceph-mon[37473]: pgmap v4175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:13:51.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:13:51.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:13:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:51 smithi134 ceph-mon[31966]: pgmap v4175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:51.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:13:51.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:13:51.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:13:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:51 smithi189 ceph-mon[36790]: pgmap v4175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:13:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:13:51.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:13:51.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:13:51.696 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:13:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:52 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:13:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:52 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:13:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:52 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:13:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:52 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:13:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:52 smithi193 ceph-mon[37473]: pgmap v4176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:13:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6389.607991 secs 2024-07-26T13:13:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:52 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:13:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:52 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:13:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:52 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:13:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:52 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:13:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:52 smithi134 ceph-mon[31966]: pgmap v4176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:13:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6389.607991 secs 2024-07-26T13:13:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:52 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:13:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:52 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:13:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:52 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:13:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:52 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:13:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:52 smithi189 ceph-mon[36790]: pgmap v4176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:13:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6389.607991 secs 2024-07-26T13:13:53.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:13:53.029+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:13:53.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:13:53.029+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:13:53.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:13:53.029+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:13:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:54 smithi193 ceph-mon[37473]: pgmap v4177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:13:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:54 smithi134 ceph-mon[31966]: pgmap v4177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:13:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:54 smithi189 ceph-mon[36790]: pgmap v4177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:13:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6375.182518 secs 2024-07-26T13:13:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6375.182518 secs 2024-07-26T13:13:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6375.182518 secs 2024-07-26T13:13:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:56 smithi193 ceph-mon[37473]: pgmap v4178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:13:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:56 smithi134 ceph-mon[31966]: pgmap v4178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:13:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:56 smithi189 ceph-mon[36790]: pgmap v4178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:13:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6394.608118 secs 2024-07-26T13:13:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6394.608118 secs 2024-07-26T13:13:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6394.608118 secs 2024-07-26T13:13:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:58 smithi193 ceph-mon[37473]: pgmap v4179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:13:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:58 smithi134 ceph-mon[31966]: pgmap v4179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:13:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:58 smithi189 ceph-mon[36790]: pgmap v4179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:13:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:13:59 smithi193 ceph-mon[37473]: pgmap v4180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:13:59.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:13:59 smithi189 ceph-mon[36790]: pgmap v4180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:13:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:13:59 smithi134 ceph-mon[31966]: pgmap v4180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:14:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6380.182665 secs 2024-07-26T13:14:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6380.182665 secs 2024-07-26T13:14:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6380.182665 secs 2024-07-26T13:14:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:01 smithi193 ceph-mon[37473]: pgmap v4181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:14:01.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:01 smithi134 ceph-mon[31966]: pgmap v4181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:14:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:01 smithi189 ceph-mon[36790]: pgmap v4181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:14:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6399.608260 secs 2024-07-26T13:14:02.597 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6399.608260 secs 2024-07-26T13:14:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6399.608260 secs 2024-07-26T13:14:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:14:03.531+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:14:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:14:03.531+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:14:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:14:03.531+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:03.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:03 smithi134 ceph-mon[31966]: pgmap v4182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:03 smithi189 ceph-mon[36790]: pgmap v4182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:03 smithi193 ceph-mon[37473]: pgmap v4182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6385.182810 secs 2024-07-26T13:14:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6385.182810 secs 2024-07-26T13:14:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6385.182810 secs 2024-07-26T13:14:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:05 smithi134 ceph-mon[31966]: pgmap v4183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:05 smithi189 ceph-mon[36790]: pgmap v4183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:05 smithi193 ceph-mon[37473]: pgmap v4183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6404.608378 secs 2024-07-26T13:14:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6404.608378 secs 2024-07-26T13:14:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6404.608378 secs 2024-07-26T13:14:08.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:14:08.044+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:14:08.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:14:08.044+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:08.046 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:14:08.044+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:14:08.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:07 smithi134 ceph-mon[31966]: pgmap v4184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:07 smithi189 ceph-mon[36790]: pgmap v4184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:07 smithi193 ceph-mon[37473]: pgmap v4184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6390.182925 secs 2024-07-26T13:14:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:09 smithi134 ceph-mon[31966]: pgmap v4185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6390.182925 secs 2024-07-26T13:14:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:09 smithi189 ceph-mon[36790]: pgmap v4185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6390.182925 secs 2024-07-26T13:14:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:09 smithi193 ceph-mon[37473]: pgmap v4185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:11 smithi134 ceph-mon[31966]: pgmap v4186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6409.608522 secs 2024-07-26T13:14:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:11 smithi189 ceph-mon[36790]: pgmap v4186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6409.608522 secs 2024-07-26T13:14:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:11 smithi193 ceph-mon[37473]: pgmap v4186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6409.608522 secs 2024-07-26T13:14:14.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:13 smithi134 ceph-mon[31966]: pgmap v4187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:13 smithi189 ceph-mon[36790]: pgmap v4187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:13 smithi193 ceph-mon[37473]: pgmap v4187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6395.183075 secs 2024-07-26T13:14:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6395.183075 secs 2024-07-26T13:14:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6395.183075 secs 2024-07-26T13:14:16.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:15 smithi134 ceph-mon[31966]: pgmap v4188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:15 smithi189 ceph-mon[36790]: pgmap v4188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:15 smithi193 ceph-mon[37473]: pgmap v4188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6414.608668 secs 2024-07-26T13:14:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6414.608668 secs 2024-07-26T13:14:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6414.608668 secs 2024-07-26T13:14:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:17 smithi189 ceph-mon[36790]: pgmap v4189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:17 smithi193 ceph-mon[37473]: pgmap v4189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:18.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:17 smithi134 ceph-mon[31966]: pgmap v4189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:18.547 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:14:18.546+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:14:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:14:18.546+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:14:18.548 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:14:18.546+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6400.183227 secs 2024-07-26T13:14:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:19 smithi189 ceph-mon[36790]: pgmap v4190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6400.183227 secs 2024-07-26T13:14:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:19 smithi193 ceph-mon[37473]: pgmap v4190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6400.183227 secs 2024-07-26T13:14:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:19 smithi134 ceph-mon[31966]: pgmap v4190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:21 smithi189 ceph-mon[36790]: pgmap v4191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6419.608769 secs 2024-07-26T13:14:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:21 smithi193 ceph-mon[37473]: pgmap v4191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6419.608769 secs 2024-07-26T13:14:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:21 smithi134 ceph-mon[31966]: pgmap v4191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6419.608769 secs 2024-07-26T13:14:23.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:14:23.047+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:14:23.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:14:23.047+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:23.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:14:23.047+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:14:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:23 smithi193 ceph-mon[37473]: pgmap v4192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:23 smithi134 ceph-mon[31966]: pgmap v4192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:23 smithi189 ceph-mon[36790]: pgmap v4192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6405.183350 secs 2024-07-26T13:14:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6405.183350 secs 2024-07-26T13:14:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6405.183350 secs 2024-07-26T13:14:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:25 smithi193 ceph-mon[37473]: pgmap v4193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:25 smithi134 ceph-mon[31966]: pgmap v4193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:25 smithi189 ceph-mon[36790]: pgmap v4193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6424.608894 secs 2024-07-26T13:14:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6424.608894 secs 2024-07-26T13:14:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6424.608894 secs 2024-07-26T13:14:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:27 smithi193 ceph-mon[37473]: pgmap v4194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:27 smithi134 ceph-mon[31966]: pgmap v4194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:27 smithi189 ceph-mon[36790]: pgmap v4194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:30.245 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6410.183467 secs 2024-07-26T13:14:30.245 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:29 smithi189 ceph-mon[36790]: pgmap v4195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6410.183467 secs 2024-07-26T13:14:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:29 smithi193 ceph-mon[37473]: pgmap v4195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6410.183467 secs 2024-07-26T13:14:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:29 smithi134 ceph-mon[31966]: pgmap v4195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:32.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:31 smithi134 ceph-mon[31966]: pgmap v4196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:32.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6429.609021 secs 2024-07-26T13:14:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:31 smithi193 ceph-mon[37473]: pgmap v4196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6429.609021 secs 2024-07-26T13:14:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:31 smithi189 ceph-mon[36790]: pgmap v4196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6429.609021 secs 2024-07-26T13:14:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:14:33.561+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:14:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:14:33.561+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:14:33.562 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:14:33.561+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:33 smithi193 ceph-mon[37473]: pgmap v4197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:33 smithi134 ceph-mon[31966]: pgmap v4197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:33 smithi189 ceph-mon[36790]: pgmap v4197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6415.183612 secs 2024-07-26T13:14:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6415.183612 secs 2024-07-26T13:14:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6415.183612 secs 2024-07-26T13:14:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:36 smithi193 ceph-mon[37473]: pgmap v4198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:36 smithi134 ceph-mon[31966]: pgmap v4198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:36 smithi189 ceph-mon[36790]: pgmap v4198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6434.609164 secs 2024-07-26T13:14:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6434.609164 secs 2024-07-26T13:14:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6434.609164 secs 2024-07-26T13:14:38.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:14:38.062+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:14:38.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:14:38.063+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:14:38.064 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:14:38.063+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:38.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:38 smithi134 ceph-mon[31966]: pgmap v4199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:38 smithi189 ceph-mon[36790]: pgmap v4199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:38.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:14:38.444+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:14:38.445 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:14:38.444+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:14:38.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:14:38.445+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T13:14:38.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:14:38.445+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T13:14:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:38 smithi193 ceph-mon[37473]: pgmap v4199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6420.183729 secs 2024-07-26T13:14:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:40 smithi134 ceph-mon[31966]: pgmap v4200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6420.183729 secs 2024-07-26T13:14:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:40 smithi189 ceph-mon[36790]: pgmap v4200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6420.183729 secs 2024-07-26T13:14:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:40 smithi193 ceph-mon[37473]: pgmap v4200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:42 smithi134 ceph-mon[31966]: pgmap v4201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6439.609332 secs 2024-07-26T13:14:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:42 smithi189 ceph-mon[36790]: pgmap v4201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6439.609332 secs 2024-07-26T13:14:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:42 smithi193 ceph-mon[37473]: pgmap v4201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6439.609332 secs 2024-07-26T13:14:44.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:44 smithi134 ceph-mon[31966]: pgmap v4202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:44 smithi189 ceph-mon[36790]: pgmap v4202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:44 smithi193 ceph-mon[37473]: pgmap v4202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6425.183871 secs 2024-07-26T13:14:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6425.183871 secs 2024-07-26T13:14:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6425.183871 secs 2024-07-26T13:14:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:46 smithi189 ceph-mon[36790]: pgmap v4203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:46 smithi193 ceph-mon[37473]: pgmap v4203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:46 smithi134 ceph-mon[31966]: pgmap v4203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:14:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6444.609456 secs 2024-07-26T13:14:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6444.609456 secs 2024-07-26T13:14:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6444.609456 secs 2024-07-26T13:14:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:48 smithi193 ceph-mon[37473]: pgmap v4204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:14:48.566+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:14:48.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:14:48.566+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:48.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:14:48.566+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:14:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:48 smithi134 ceph-mon[31966]: pgmap v4204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:48 smithi189 ceph-mon[36790]: pgmap v4204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:14:48.963+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:14:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:14:48.963+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:14:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:14:48.963+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:14:48.964 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:14:48.963+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:14:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6430.183992 secs 2024-07-26T13:14:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:50 smithi193 ceph-mon[37473]: pgmap v4205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6430.183992 secs 2024-07-26T13:14:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:50 smithi134 ceph-mon[31966]: pgmap v4205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6430.183992 secs 2024-07-26T13:14:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:50 smithi189 ceph-mon[36790]: pgmap v4205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:14:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:14:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:14:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:14:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:52 smithi193 ceph-mon[37473]: pgmap v4206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6449.609586 secs 2024-07-26T13:14:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:14:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:14:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:14:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:52 smithi134 ceph-mon[31966]: pgmap v4206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6449.609586 secs 2024-07-26T13:14:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:14:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:14:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:14:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:52 smithi189 ceph-mon[36790]: pgmap v4206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:14:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6449.609586 secs 2024-07-26T13:14:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:14:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:14:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:14:53.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:14:53.064+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:14:53.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:14:53.064+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:14:53.066 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:14:53.064+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:14:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:53 smithi134 ceph-mon[31966]: pgmap v4207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:14:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:53 smithi134 ceph-mon[31966]: pgmap v4208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:14:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:53 smithi193 ceph-mon[37473]: pgmap v4207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:14:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:53 smithi193 ceph-mon[37473]: pgmap v4208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:14:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:53 smithi189 ceph-mon[36790]: pgmap v4207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:14:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:53 smithi189 ceph-mon[36790]: pgmap v4208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:14:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6435.184075 secs 2024-07-26T13:14:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:55 smithi193 ceph-mon[37473]: pgmap v4209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:14:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6435.184075 secs 2024-07-26T13:14:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:55 smithi134 ceph-mon[31966]: pgmap v4209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:14:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6435.184075 secs 2024-07-26T13:14:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:55 smithi189 ceph-mon[36790]: pgmap v4209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:14:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6454.609707 secs 2024-07-26T13:14:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:57 smithi193 ceph-mon[37473]: pgmap v4210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:14:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6454.609707 secs 2024-07-26T13:14:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:57 smithi134 ceph-mon[31966]: pgmap v4210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:14:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6454.609707 secs 2024-07-26T13:14:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:57 smithi189 ceph-mon[36790]: pgmap v4210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:14:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:14:59 smithi193 ceph-mon[37473]: pgmap v4211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:14:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:14:59 smithi134 ceph-mon[31966]: pgmap v4211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:14:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:14:59 smithi189 ceph-mon[36790]: pgmap v4211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:15:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6440.184191 secs 2024-07-26T13:15:00.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6440.184191 secs 2024-07-26T13:15:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6440.184191 secs 2024-07-26T13:15:01.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:01 smithi193 ceph-mon[37473]: pgmap v4212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:15:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:01 smithi134 ceph-mon[31966]: pgmap v4212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:15:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:01 smithi189 ceph-mon[36790]: pgmap v4212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:15:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6459.609842 secs 2024-07-26T13:15:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:02 smithi193 ceph-mon[37473]: pgmap v4213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6459.609842 secs 2024-07-26T13:15:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:02 smithi134 ceph-mon[31966]: pgmap v4213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6459.609842 secs 2024-07-26T13:15:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:02 smithi189 ceph-mon[36790]: pgmap v4213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:15:03.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:15:03.581+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:15:03.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:15:03.582+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:15:03.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:15:03.582+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6445.184333 secs 2024-07-26T13:15:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:05 smithi134 ceph-mon[31966]: pgmap v4214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6445.184333 secs 2024-07-26T13:15:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:05 smithi189 ceph-mon[36790]: pgmap v4214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6445.184333 secs 2024-07-26T13:15:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:05 smithi193 ceph-mon[37473]: pgmap v4214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6464.609987 secs 2024-07-26T13:15:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:07 smithi134 ceph-mon[31966]: pgmap v4215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6464.609987 secs 2024-07-26T13:15:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:07 smithi189 ceph-mon[36790]: pgmap v4215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6464.609987 secs 2024-07-26T13:15:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:07 smithi193 ceph-mon[37473]: pgmap v4215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:08.074 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:15:08.072+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:15:08.074 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:15:08.072+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:08.074 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:15:08.072+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:15:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:09 smithi134 ceph-mon[31966]: pgmap v4216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:09 smithi189 ceph-mon[36790]: pgmap v4216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:09 smithi193 ceph-mon[37473]: pgmap v4216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:10.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6450.184374 secs 2024-07-26T13:15:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6450.184374 secs 2024-07-26T13:15:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6450.184374 secs 2024-07-26T13:15:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:11 smithi134 ceph-mon[31966]: pgmap v4217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:11 smithi189 ceph-mon[36790]: pgmap v4217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:11 smithi193 ceph-mon[37473]: pgmap v4217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6469.610146 secs 2024-07-26T13:15:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6469.610146 secs 2024-07-26T13:15:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6469.610146 secs 2024-07-26T13:15:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:13 smithi134 ceph-mon[31966]: pgmap v4218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:15:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:13 smithi189 ceph-mon[36790]: pgmap v4218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:15:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:13 smithi193 ceph-mon[37473]: pgmap v4218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:15:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6455.184513 secs 2024-07-26T13:15:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:15 smithi134 ceph-mon[31966]: pgmap v4219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6455.184513 secs 2024-07-26T13:15:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:15 smithi189 ceph-mon[36790]: pgmap v4219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6455.184513 secs 2024-07-26T13:15:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:15 smithi193 ceph-mon[37473]: pgmap v4219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6474.610284 secs 2024-07-26T13:15:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:17 smithi134 ceph-mon[31966]: pgmap v4220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6474.610284 secs 2024-07-26T13:15:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:17 smithi189 ceph-mon[36790]: pgmap v4220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6474.610284 secs 2024-07-26T13:15:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:17 smithi193 ceph-mon[37473]: pgmap v4220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:15:18.586+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:15:18.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:15:18.586+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:18.588 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:15:18.586+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:15:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:19 smithi134 ceph-mon[31966]: pgmap v4221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:19 smithi189 ceph-mon[36790]: pgmap v4221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:19 smithi193 ceph-mon[37473]: pgmap v4221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6460.184647 secs 2024-07-26T13:15:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6460.184647 secs 2024-07-26T13:15:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6460.184647 secs 2024-07-26T13:15:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:21 smithi134 ceph-mon[31966]: pgmap v4222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:21 smithi189 ceph-mon[36790]: pgmap v4222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:21 smithi193 ceph-mon[37473]: pgmap v4222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6479.610412 secs 2024-07-26T13:15:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6479.610412 secs 2024-07-26T13:15:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6479.610412 secs 2024-07-26T13:15:23.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:15:23.076+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:15:23.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:15:23.076+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:15:23.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:15:23.076+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:23 smithi134 ceph-mon[31966]: pgmap v4223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:15:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:23 smithi189 ceph-mon[36790]: pgmap v4223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:15:23.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:15:23.452+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:15:23.453 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:15:23.452+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:15:23.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:15:23.452+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T13:15:23.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:15:23.452+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T13:15:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:23 smithi193 ceph-mon[37473]: pgmap v4223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:15:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6465.184766 secs 2024-07-26T13:15:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:25 smithi134 ceph-mon[31966]: pgmap v4224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6465.184766 secs 2024-07-26T13:15:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:25 smithi189 ceph-mon[36790]: pgmap v4224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6465.184766 secs 2024-07-26T13:15:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:25 smithi193 ceph-mon[37473]: pgmap v4224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6484.610493 secs 2024-07-26T13:15:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:27 smithi134 ceph-mon[31966]: pgmap v4225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6484.610493 secs 2024-07-26T13:15:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:27 smithi189 ceph-mon[36790]: pgmap v4225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6484.610493 secs 2024-07-26T13:15:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:27 smithi193 ceph-mon[37473]: pgmap v4225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:29 smithi134 ceph-mon[31966]: pgmap v4226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:29 smithi189 ceph-mon[36790]: pgmap v4226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:29 smithi193 ceph-mon[37473]: pgmap v4226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6470.184881 secs 2024-07-26T13:15:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6470.184881 secs 2024-07-26T13:15:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6470.184881 secs 2024-07-26T13:15:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:31 smithi189 ceph-mon[36790]: pgmap v4227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:31 smithi193 ceph-mon[37473]: pgmap v4227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:31 smithi134 ceph-mon[31966]: pgmap v4227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6489.610576 secs 2024-07-26T13:15:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6489.610576 secs 2024-07-26T13:15:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6489.610576 secs 2024-07-26T13:15:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:33 smithi193 ceph-mon[37473]: pgmap v4228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:15:33.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:15:33.595+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:15:33.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:15:33.595+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:33.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:15:33.595+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:15:33.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:33 smithi134 ceph-mon[31966]: pgmap v4228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:15:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:33 smithi189 ceph-mon[36790]: pgmap v4228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:15:33.971 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:15:33.970+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:15:33.972 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:15:33.970+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:15:33.972 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:15:33.971+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:15:33.972 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:15:33.971+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:15:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6475.184991 secs 2024-07-26T13:15:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:35 smithi193 ceph-mon[37473]: pgmap v4229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6475.184991 secs 2024-07-26T13:15:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:35 smithi134 ceph-mon[31966]: pgmap v4229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6475.184991 secs 2024-07-26T13:15:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:35 smithi189 ceph-mon[36790]: pgmap v4229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6494.610686 secs 2024-07-26T13:15:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:37 smithi193 ceph-mon[37473]: pgmap v4230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6494.610686 secs 2024-07-26T13:15:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:37 smithi134 ceph-mon[31966]: pgmap v4230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6494.610686 secs 2024-07-26T13:15:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:37 smithi189 ceph-mon[36790]: pgmap v4230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:38.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:15:38.092+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:15:38.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:15:38.092+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:15:38.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:15:38.092+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:39 smithi193 ceph-mon[37473]: pgmap v4231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:39 smithi134 ceph-mon[31966]: pgmap v4231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:39 smithi189 ceph-mon[36790]: pgmap v4231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6480.185112 secs 2024-07-26T13:15:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6480.185112 secs 2024-07-26T13:15:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6480.185112 secs 2024-07-26T13:15:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:41 smithi193 ceph-mon[37473]: pgmap v4232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:41 smithi134 ceph-mon[31966]: pgmap v4232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:41 smithi189 ceph-mon[36790]: pgmap v4232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6499.610848 secs 2024-07-26T13:15:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6499.610848 secs 2024-07-26T13:15:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6499.610848 secs 2024-07-26T13:15:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:43 smithi193 ceph-mon[37473]: pgmap v4233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:15:43.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:43 smithi134 ceph-mon[31966]: pgmap v4233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:15:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:43 smithi189 ceph-mon[36790]: pgmap v4233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:15:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6485.185238 secs 2024-07-26T13:15:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:45 smithi193 ceph-mon[37473]: pgmap v4234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6485.185238 secs 2024-07-26T13:15:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:45 smithi134 ceph-mon[31966]: pgmap v4234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6485.185238 secs 2024-07-26T13:15:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:45 smithi189 ceph-mon[36790]: pgmap v4234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:46 smithi193 ceph-mon[37473]: pgmap v4235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:46 smithi134 ceph-mon[31966]: pgmap v4235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:46 smithi189 ceph-mon[36790]: pgmap v4235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6504.610971 secs 2024-07-26T13:15:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6504.610971 secs 2024-07-26T13:15:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6504.610971 secs 2024-07-26T13:15:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:15:48.611+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:15:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:15:48.611+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:15:48.611+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:15:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:48 smithi134 ceph-mon[31966]: pgmap v4236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:48 smithi189 ceph-mon[36790]: pgmap v4236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:48 smithi193 ceph-mon[37473]: pgmap v4236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:15:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6490.185353 secs 2024-07-26T13:15:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6490.185353 secs 2024-07-26T13:15:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6490.185353 secs 2024-07-26T13:15:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:50 smithi134 ceph-mon[31966]: pgmap v4237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:50 smithi189 ceph-mon[36790]: pgmap v4237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:50 smithi193 ceph-mon[37473]: pgmap v4237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:15:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6509.611096 secs 2024-07-26T13:15:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6509.611096 secs 2024-07-26T13:15:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6509.611096 secs 2024-07-26T13:15:53.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:15:53.107+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:15:53.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:15:53.107+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:15:53.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:15:53.107+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:15:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:52 smithi134 ceph-mon[31966]: pgmap v4238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:15:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:15:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:15:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:15:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:52 smithi189 ceph-mon[36790]: pgmap v4238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:15:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:15:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:15:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:15:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:52 smithi193 ceph-mon[37473]: pgmap v4238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:15:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:15:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:15:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:15:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:15:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:15:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:15:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:15:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:15:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:15:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:15:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:15:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:15:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6495.185471 secs 2024-07-26T13:15:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:54 smithi134 ceph-mon[31966]: pgmap v4239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T13:15:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:54 smithi134 ceph-mon[31966]: pgmap v4240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6495.185471 secs 2024-07-26T13:15:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:54 smithi189 ceph-mon[36790]: pgmap v4239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T13:15:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:54 smithi189 ceph-mon[36790]: pgmap v4240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6495.185471 secs 2024-07-26T13:15:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:54 smithi193 ceph-mon[37473]: pgmap v4239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T13:15:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:54 smithi193 ceph-mon[37473]: pgmap v4240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:56 smithi134 ceph-mon[31966]: pgmap v4241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6514.611223 secs 2024-07-26T13:15:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:56 smithi189 ceph-mon[36790]: pgmap v4241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6514.611223 secs 2024-07-26T13:15:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:56 smithi193 ceph-mon[37473]: pgmap v4241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6514.611223 secs 2024-07-26T13:15:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:58 smithi189 ceph-mon[36790]: pgmap v4242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:58 smithi193 ceph-mon[37473]: pgmap v4242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:15:59.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:58 smithi134 ceph-mon[31966]: pgmap v4242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:16:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:15:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6500.185577 secs 2024-07-26T13:16:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:15:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6500.185577 secs 2024-07-26T13:16:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:15:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6500.185577 secs 2024-07-26T13:16:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:00 smithi189 ceph-mon[36790]: pgmap v4243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:16:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:00 smithi193 ceph-mon[37473]: pgmap v4243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:16:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:00 smithi134 ceph-mon[31966]: pgmap v4243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T13:16:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6519.611289 secs 2024-07-26T13:16:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6519.611289 secs 2024-07-26T13:16:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6519.611289 secs 2024-07-26T13:16:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:02 smithi193 ceph-mon[37473]: pgmap v4244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:16:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:02 smithi134 ceph-mon[31966]: pgmap v4244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:16:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:02 smithi189 ceph-mon[36790]: pgmap v4244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:16:03.612 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:16:03.611+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:16:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:16:03.611+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:03.613 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:16:03.611+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:16:05.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6505.185676 secs 2024-07-26T13:16:05.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:04 smithi134 ceph-mon[31966]: pgmap v4245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:16:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6505.185676 secs 2024-07-26T13:16:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:04 smithi193 ceph-mon[37473]: pgmap v4245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:16:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6505.185676 secs 2024-07-26T13:16:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:04 smithi189 ceph-mon[36790]: pgmap v4245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:16:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:06 smithi193 ceph-mon[37473]: pgmap v4246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6524.611408 secs 2024-07-26T13:16:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:06 smithi134 ceph-mon[31966]: pgmap v4246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6524.611408 secs 2024-07-26T13:16:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:06 smithi189 ceph-mon[36790]: pgmap v4246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6524.611408 secs 2024-07-26T13:16:08.124 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:16:08.123+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:16:08.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:16:08.123+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:08.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:16:08.123+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:16:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:08 smithi193 ceph-mon[37473]: pgmap v4247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:09.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:08 smithi134 ceph-mon[31966]: pgmap v4247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:08 smithi189 ceph-mon[36790]: pgmap v4247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6510.185821 secs 2024-07-26T13:16:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6510.185821 secs 2024-07-26T13:16:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6510.185821 secs 2024-07-26T13:16:11.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:10 smithi193 ceph-mon[37473]: pgmap v4248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:16:11.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:10 smithi134 ceph-mon[31966]: pgmap v4248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:16:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:10 smithi189 ceph-mon[36790]: pgmap v4248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:16:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6529.611541 secs 2024-07-26T13:16:12.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6529.611541 secs 2024-07-26T13:16:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6529.611541 secs 2024-07-26T13:16:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:13 smithi193 ceph-mon[37473]: pgmap v4249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:13.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:13 smithi134 ceph-mon[31966]: pgmap v4249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:13 smithi189 ceph-mon[36790]: pgmap v4249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6515.185969 secs 2024-07-26T13:16:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:15 smithi193 ceph-mon[37473]: pgmap v4250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6515.185969 secs 2024-07-26T13:16:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:15 smithi134 ceph-mon[31966]: pgmap v4250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6515.185969 secs 2024-07-26T13:16:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:15 smithi189 ceph-mon[36790]: pgmap v4250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:17 smithi193 ceph-mon[37473]: pgmap v4251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6534.611621 secs 2024-07-26T13:16:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:17 smithi134 ceph-mon[31966]: pgmap v4251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6534.611621 secs 2024-07-26T13:16:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:17 smithi189 ceph-mon[36790]: pgmap v4251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6534.611621 secs 2024-07-26T13:16:18.627 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:16:18.626+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:16:18.627 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:16:18.626+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:16:18.628 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:16:18.626+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:19 smithi193 ceph-mon[37473]: pgmap v4252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:16:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:19 smithi134 ceph-mon[31966]: pgmap v4252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:16:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:19 smithi189 ceph-mon[36790]: pgmap v4252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:16:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6520.186095 secs 2024-07-26T13:16:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6520.186095 secs 2024-07-26T13:16:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6520.186095 secs 2024-07-26T13:16:21.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:21 smithi134 ceph-mon[31966]: pgmap v4253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:16:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:21 smithi193 ceph-mon[37473]: pgmap v4253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:16:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:21 smithi189 ceph-mon[36790]: pgmap v4253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:16:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6539.611743 secs 2024-07-26T13:16:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6539.611743 secs 2024-07-26T13:16:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6539.611743 secs 2024-07-26T13:16:23.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:16:23.138+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:16:23.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:16:23.138+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:16:23.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:16:23.138+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:23 smithi134 ceph-mon[31966]: pgmap v4254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:16:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:23 smithi193 ceph-mon[37473]: pgmap v4254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:16:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:23 smithi189 ceph-mon[36790]: pgmap v4254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:16:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6525.186180 secs 2024-07-26T13:16:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:25 smithi193 ceph-mon[37473]: pgmap v4255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:16:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6525.186180 secs 2024-07-26T13:16:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:25 smithi134 ceph-mon[31966]: pgmap v4255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:16:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6525.186180 secs 2024-07-26T13:16:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:25 smithi189 ceph-mon[36790]: pgmap v4255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:16:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:26 smithi134 ceph-mon[31966]: pgmap v4256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:16:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:26 smithi189 ceph-mon[36790]: pgmap v4256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:16:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:26 smithi193 ceph-mon[37473]: pgmap v4256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:16:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6544.611822 secs 2024-07-26T13:16:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6544.611822 secs 2024-07-26T13:16:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6544.611822 secs 2024-07-26T13:16:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:28 smithi134 ceph-mon[31966]: pgmap v4257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:16:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:28 smithi189 ceph-mon[36790]: pgmap v4257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:16:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:28 smithi193 ceph-mon[37473]: pgmap v4257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:16:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6530.186310 secs 2024-07-26T13:16:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6530.186310 secs 2024-07-26T13:16:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6530.186310 secs 2024-07-26T13:16:30.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:30 smithi189 ceph-mon[36790]: pgmap v4258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:16:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:30 smithi134 ceph-mon[31966]: pgmap v4258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:16:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:30 smithi193 ceph-mon[37473]: pgmap v4258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:16:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6549.611933 secs 2024-07-26T13:16:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6549.611933 secs 2024-07-26T13:16:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6549.611933 secs 2024-07-26T13:16:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:32 smithi134 ceph-mon[31966]: pgmap v4259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:32 smithi189 ceph-mon[36790]: pgmap v4259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:32 smithi193 ceph-mon[37473]: pgmap v4259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:16:33.642+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:16:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:16:33.642+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:16:33.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:16:33.642+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6535.186463 secs 2024-07-26T13:16:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:34 smithi189 ceph-mon[36790]: pgmap v4260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6535.186463 secs 2024-07-26T13:16:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:34 smithi193 ceph-mon[37473]: pgmap v4260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6535.186463 secs 2024-07-26T13:16:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:34 smithi134 ceph-mon[31966]: pgmap v4260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:36 smithi189 ceph-mon[36790]: pgmap v4261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6554.612050 secs 2024-07-26T13:16:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:36 smithi193 ceph-mon[37473]: pgmap v4261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6554.612050 secs 2024-07-26T13:16:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:36 smithi134 ceph-mon[31966]: pgmap v4261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6554.612050 secs 2024-07-26T13:16:38.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:16:38.144+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:16:38.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:16:38.144+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:38.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:16:38.144+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:16:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:38 smithi189 ceph-mon[36790]: pgmap v4262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:38 smithi193 ceph-mon[37473]: pgmap v4262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:38 smithi134 ceph-mon[31966]: pgmap v4262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6540.186582 secs 2024-07-26T13:16:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6540.186582 secs 2024-07-26T13:16:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6540.186582 secs 2024-07-26T13:16:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:40 smithi193 ceph-mon[37473]: pgmap v4263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:16:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:40 smithi134 ceph-mon[31966]: pgmap v4263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:16:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:40 smithi189 ceph-mon[36790]: pgmap v4263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:16:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6559.612184 secs 2024-07-26T13:16:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6559.612184 secs 2024-07-26T13:16:42.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6559.612184 secs 2024-07-26T13:16:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:42 smithi193 ceph-mon[37473]: pgmap v4264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:42 smithi134 ceph-mon[31966]: pgmap v4264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:42 smithi189 ceph-mon[36790]: pgmap v4264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6545.186733 secs 2024-07-26T13:16:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:44 smithi193 ceph-mon[37473]: pgmap v4265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6545.186733 secs 2024-07-26T13:16:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:44 smithi134 ceph-mon[31966]: pgmap v4265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6545.186733 secs 2024-07-26T13:16:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:44 smithi189 ceph-mon[36790]: pgmap v4265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:46 smithi193 ceph-mon[37473]: pgmap v4266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6564.612320 secs 2024-07-26T13:16:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:46 smithi134 ceph-mon[31966]: pgmap v4266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6564.612320 secs 2024-07-26T13:16:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:46 smithi189 ceph-mon[36790]: pgmap v4266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6564.612320 secs 2024-07-26T13:16:48.645 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:16:48.643+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:16:48.645 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:16:48.643+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:16:48.645 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:16:48.643+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:48 smithi193 ceph-mon[37473]: pgmap v4267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:48 smithi134 ceph-mon[31966]: pgmap v4267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:48 smithi189 ceph-mon[36790]: pgmap v4267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6550.186856 secs 2024-07-26T13:16:50.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6550.186856 secs 2024-07-26T13:16:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6550.186856 secs 2024-07-26T13:16:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:50 smithi193 ceph-mon[37473]: pgmap v4268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:16:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:50 smithi134 ceph-mon[31966]: pgmap v4268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:16:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:50 smithi189 ceph-mon[36790]: pgmap v4268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:16:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6569.612394 secs 2024-07-26T13:16:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6569.612394 secs 2024-07-26T13:16:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6569.612394 secs 2024-07-26T13:16:53.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:16:53.160+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:16:53.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:16:53.160+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:16:53.162 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:16:53.160+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:16:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:53 smithi193 ceph-mon[37473]: pgmap v4269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:53.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:53 smithi134 ceph-mon[31966]: pgmap v4269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:53 smithi189 ceph-mon[36790]: pgmap v4269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6555.186977 secs 2024-07-26T13:16:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:55 smithi193 ceph-mon[37473]: pgmap v4270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:16:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:55.291 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6555.186977 secs 2024-07-26T13:16:55.291 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:55 smithi189 ceph-mon[36790]: pgmap v4270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:55.291 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:16:55.291 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:55.291 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:55.291 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:55.291 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:55.330 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6555.186977 secs 2024-07-26T13:16:55.330 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:55 smithi134 ceph-mon[31966]: pgmap v4270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:55.330 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:16:55.330 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:55.330 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:55.330 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:55.330 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:56 smithi134 ceph-mon[31966]: pgmap v4271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6574.612482 secs 2024-07-26T13:16:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:56 smithi189 ceph-mon[36790]: pgmap v4271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6574.612482 secs 2024-07-26T13:16:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:56 smithi193 ceph-mon[37473]: pgmap v4271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:16:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6574.612482 secs 2024-07-26T13:16:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:58 smithi193 ceph-mon[37473]: pgmap v4272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:58 smithi134 ceph-mon[31966]: pgmap v4272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:58 smithi189 ceph-mon[36790]: pgmap v4272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:16:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:16:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:17:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6560.187094 secs 2024-07-26T13:17:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:17:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:17:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:59 smithi193 ceph-mon[37473]: pgmap v4273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:17:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:59 smithi193 ceph-mon[37473]: pgmap v4274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:17:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:16:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:17:00.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6560.187094 secs 2024-07-26T13:17:00.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:17:00.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:17:00.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:59 smithi189 ceph-mon[36790]: pgmap v4273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:17:00.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:59 smithi189 ceph-mon[36790]: pgmap v4274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:17:00.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:16:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:17:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6560.187094 secs 2024-07-26T13:17:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:17:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:17:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:59 smithi134 ceph-mon[31966]: pgmap v4273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:17:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:59 smithi134 ceph-mon[31966]: pgmap v4274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:17:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:16:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:17:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6579.612606 secs 2024-07-26T13:17:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:02 smithi134 ceph-mon[31966]: pgmap v4275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:17:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6579.612606 secs 2024-07-26T13:17:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:02 smithi189 ceph-mon[36790]: pgmap v4275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:17:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6579.612606 secs 2024-07-26T13:17:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:02 smithi193 ceph-mon[37473]: pgmap v4275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:17:03.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:17:03.659+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:17:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:17:03.659+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:03.661 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:17:03.659+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:17:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:04 smithi134 ceph-mon[31966]: pgmap v4276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:17:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:04 smithi189 ceph-mon[36790]: pgmap v4276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:17:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:04 smithi193 ceph-mon[37473]: pgmap v4276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:17:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6565.187214 secs 2024-07-26T13:17:05.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6565.187214 secs 2024-07-26T13:17:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6565.187214 secs 2024-07-26T13:17:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:06 smithi134 ceph-mon[31966]: pgmap v4277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:17:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:06 smithi189 ceph-mon[36790]: pgmap v4277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:17:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:06 smithi193 ceph-mon[37473]: pgmap v4277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:17:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6584.612720 secs 2024-07-26T13:17:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6584.612720 secs 2024-07-26T13:17:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6584.612720 secs 2024-07-26T13:17:08.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:17:08.175+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:17:08.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:17:08.175+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:08.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:17:08.175+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:17:08.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:08 smithi134 ceph-mon[31966]: pgmap v4278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:17:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:08 smithi193 ceph-mon[37473]: pgmap v4278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:17:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:08 smithi189 ceph-mon[36790]: pgmap v4278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:17:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6570.187333 secs 2024-07-26T13:17:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:10 smithi193 ceph-mon[37473]: pgmap v4279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6570.187333 secs 2024-07-26T13:17:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:10 smithi134 ceph-mon[31966]: pgmap v4279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6570.187333 secs 2024-07-26T13:17:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:10 smithi189 ceph-mon[36790]: pgmap v4279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:17:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6589.612843 secs 2024-07-26T13:17:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:12 smithi193 ceph-mon[37473]: pgmap v4280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6589.612843 secs 2024-07-26T13:17:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:12 smithi134 ceph-mon[31966]: pgmap v4280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6589.612843 secs 2024-07-26T13:17:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:12 smithi189 ceph-mon[36790]: pgmap v4280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:14 smithi193 ceph-mon[37473]: pgmap v4281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:14 smithi134 ceph-mon[31966]: pgmap v4281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:14 smithi189 ceph-mon[36790]: pgmap v4281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6575.187468 secs 2024-07-26T13:17:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6575.187468 secs 2024-07-26T13:17:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6575.187468 secs 2024-07-26T13:17:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:16 smithi193 ceph-mon[37473]: pgmap v4282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:16 smithi134 ceph-mon[31966]: pgmap v4282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:16 smithi189 ceph-mon[36790]: pgmap v4282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6594.612973 secs 2024-07-26T13:17:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6594.612973 secs 2024-07-26T13:17:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6594.612973 secs 2024-07-26T13:17:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:18 smithi134 ceph-mon[31966]: pgmap v4283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:17:18.674+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:17:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:17:18.674+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:18.675 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:17:18.674+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:17:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:18 smithi189 ceph-mon[36790]: pgmap v4283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:18 smithi193 ceph-mon[37473]: pgmap v4283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:19 smithi134 ceph-mon[31966]: pgmap v4284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:19 smithi189 ceph-mon[36790]: pgmap v4284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:19 smithi193 ceph-mon[37473]: pgmap v4284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6580.187587 secs 2024-07-26T13:17:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6580.187587 secs 2024-07-26T13:17:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6580.187587 secs 2024-07-26T13:17:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:21 smithi134 ceph-mon[31966]: pgmap v4285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:21 smithi189 ceph-mon[36790]: pgmap v4285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:21 smithi193 ceph-mon[37473]: pgmap v4285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6599.613101 secs 2024-07-26T13:17:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6599.613101 secs 2024-07-26T13:17:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6599.613101 secs 2024-07-26T13:17:23.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:17:23.191+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:17:23.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:17:23.191+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:23.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:17:23.191+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:17:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:23 smithi134 ceph-mon[31966]: pgmap v4286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:23 smithi189 ceph-mon[36790]: pgmap v4286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:23 smithi193 ceph-mon[37473]: pgmap v4286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6585.187727 secs 2024-07-26T13:17:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6585.187727 secs 2024-07-26T13:17:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6585.187727 secs 2024-07-26T13:17:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:25 smithi134 ceph-mon[31966]: pgmap v4287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:25 smithi189 ceph-mon[36790]: pgmap v4287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:25 smithi193 ceph-mon[37473]: pgmap v4287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6604.613163 secs 2024-07-26T13:17:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6604.613163 secs 2024-07-26T13:17:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6604.613163 secs 2024-07-26T13:17:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:27 smithi189 ceph-mon[36790]: pgmap v4288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:27 smithi193 ceph-mon[37473]: pgmap v4288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:28.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:27 smithi134 ceph-mon[31966]: pgmap v4288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:30.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6590.187860 secs 2024-07-26T13:17:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:30 smithi134 ceph-mon[31966]: pgmap v4289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6590.187860 secs 2024-07-26T13:17:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:30 smithi189 ceph-mon[36790]: pgmap v4289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6590.187860 secs 2024-07-26T13:17:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:30 smithi193 ceph-mon[37473]: pgmap v4289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6609.613285 secs 2024-07-26T13:17:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:32 smithi134 ceph-mon[31966]: pgmap v4290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6609.613285 secs 2024-07-26T13:17:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:32 smithi189 ceph-mon[36790]: pgmap v4290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6609.613285 secs 2024-07-26T13:17:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:32 smithi193 ceph-mon[37473]: pgmap v4290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:17:33.690+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:17:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:17:33.690+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:17:33.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:17:33.690+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:34 smithi134 ceph-mon[31966]: pgmap v4291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:34 smithi189 ceph-mon[36790]: pgmap v4291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:34 smithi193 ceph-mon[37473]: pgmap v4291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6595.187969 secs 2024-07-26T13:17:35.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6595.187969 secs 2024-07-26T13:17:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6595.187969 secs 2024-07-26T13:17:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:36 smithi134 ceph-mon[31966]: pgmap v4292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:36 smithi189 ceph-mon[36790]: pgmap v4292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:36 smithi193 ceph-mon[37473]: pgmap v4292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6614.613378 secs 2024-07-26T13:17:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6614.613378 secs 2024-07-26T13:17:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6614.613378 secs 2024-07-26T13:17:38.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:17:38.206+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:17:38.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:17:38.206+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:38.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:17:38.206+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:17:38.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:38 smithi134 ceph-mon[31966]: pgmap v4293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:38 smithi189 ceph-mon[36790]: pgmap v4293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:38 smithi193 ceph-mon[37473]: pgmap v4293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6600.188067 secs 2024-07-26T13:17:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:40 smithi134 ceph-mon[31966]: pgmap v4294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6600.188067 secs 2024-07-26T13:17:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:40 smithi189 ceph-mon[36790]: pgmap v4294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6600.188067 secs 2024-07-26T13:17:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:40 smithi193 ceph-mon[37473]: pgmap v4294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6619.613447 secs 2024-07-26T13:17:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:42 smithi189 ceph-mon[36790]: pgmap v4295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6619.613447 secs 2024-07-26T13:17:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:42 smithi193 ceph-mon[37473]: pgmap v4295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6619.613447 secs 2024-07-26T13:17:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:42 smithi134 ceph-mon[31966]: pgmap v4295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:44 smithi189 ceph-mon[36790]: pgmap v4296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:44 smithi193 ceph-mon[37473]: pgmap v4296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:44 smithi134 ceph-mon[31966]: pgmap v4296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6605.188185 secs 2024-07-26T13:17:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6605.188185 secs 2024-07-26T13:17:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6605.188185 secs 2024-07-26T13:17:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:46 smithi193 ceph-mon[37473]: pgmap v4297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:46 smithi134 ceph-mon[31966]: pgmap v4297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:46 smithi189 ceph-mon[36790]: pgmap v4297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6624.613546 secs 2024-07-26T13:17:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6624.613546 secs 2024-07-26T13:17:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6624.613546 secs 2024-07-26T13:17:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:48 smithi193 ceph-mon[37473]: pgmap v4298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:48 smithi134 ceph-mon[31966]: pgmap v4298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:48 smithi189 ceph-mon[36790]: pgmap v4298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:17:48.704+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:17:48.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:17:48.704+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:48.706 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:17:48.704+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:17:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6610.188329 secs 2024-07-26T13:17:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:50 smithi193 ceph-mon[37473]: pgmap v4299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6610.188329 secs 2024-07-26T13:17:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:50 smithi134 ceph-mon[31966]: pgmap v4299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6610.188329 secs 2024-07-26T13:17:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:50 smithi189 ceph-mon[36790]: pgmap v4299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6629.613633 secs 2024-07-26T13:17:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:52 smithi193 ceph-mon[37473]: pgmap v4300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6629.613633 secs 2024-07-26T13:17:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:52 smithi134 ceph-mon[31966]: pgmap v4300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6629.613633 secs 2024-07-26T13:17:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:52 smithi189 ceph-mon[36790]: pgmap v4300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:53.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:17:53.221+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:17:53.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:17:53.221+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:17:53.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:17:53.221+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:17:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:54 smithi193 ceph-mon[37473]: pgmap v4301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:54 smithi134 ceph-mon[31966]: pgmap v4301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:54 smithi189 ceph-mon[36790]: pgmap v4301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:17:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6615.188492 secs 2024-07-26T13:17:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6615.188492 secs 2024-07-26T13:17:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6615.188492 secs 2024-07-26T13:17:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:56 smithi193 ceph-mon[37473]: pgmap v4302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:56 smithi134 ceph-mon[31966]: pgmap v4302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:56 smithi189 ceph-mon[36790]: pgmap v4302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:17:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6634.613753 secs 2024-07-26T13:17:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6634.613753 secs 2024-07-26T13:17:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6634.613753 secs 2024-07-26T13:17:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:17:58 smithi193 ceph-mon[37473]: pgmap v4303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:17:58 smithi134 ceph-mon[31966]: pgmap v4303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:17:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:17:58 smithi189 ceph-mon[36790]: pgmap v4303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:18:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6620.188603 secs 2024-07-26T13:18:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:00 smithi193 ceph-mon[37473]: pgmap v4304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:18:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6620.188603 secs 2024-07-26T13:18:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:00 smithi134 ceph-mon[31966]: pgmap v4304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:18:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6620.188603 secs 2024-07-26T13:18:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:00 smithi189 ceph-mon[36790]: pgmap v4304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:18:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:18:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:18:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:18:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:18:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:18:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:18:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:18:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:18:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:18:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:02 smithi134 ceph-mon[31966]: pgmap v4305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:18:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:02 smithi134 ceph-mon[31966]: pgmap v4306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6639.613877 secs 2024-07-26T13:18:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:02 smithi189 ceph-mon[36790]: pgmap v4305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:18:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:02 smithi189 ceph-mon[36790]: pgmap v4306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6639.613877 secs 2024-07-26T13:18:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:02 smithi193 ceph-mon[37473]: pgmap v4305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:18:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:02 smithi193 ceph-mon[37473]: pgmap v4306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6639.613877 secs 2024-07-26T13:18:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:03 smithi134 ceph-mon[31966]: pgmap v4307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:18:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:03 smithi189 ceph-mon[36790]: pgmap v4307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:18:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:18:03.720+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:18:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:18:03.720+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:18:03.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:18:03.720+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:03 smithi193 ceph-mon[37473]: pgmap v4307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:18:05.576 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6625.188729 secs 2024-07-26T13:18:05.576 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:05 smithi193 ceph-mon[37473]: pgmap v4308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:18:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6625.188729 secs 2024-07-26T13:18:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:05 smithi134 ceph-mon[31966]: pgmap v4308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:18:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6625.188729 secs 2024-07-26T13:18:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:05 smithi189 ceph-mon[36790]: pgmap v4308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:18:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6644.614019 secs 2024-07-26T13:18:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:07 smithi134 ceph-mon[31966]: pgmap v4309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6644.614019 secs 2024-07-26T13:18:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:07 smithi189 ceph-mon[36790]: pgmap v4309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6644.614019 secs 2024-07-26T13:18:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:07 smithi193 ceph-mon[37473]: pgmap v4309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:08.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:18:08.225+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:18:08.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:18:08.225+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:08.226 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:18:08.225+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:18:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:09 smithi134 ceph-mon[31966]: pgmap v4310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:09 smithi189 ceph-mon[36790]: pgmap v4310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:09 smithi193 ceph-mon[37473]: pgmap v4310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:18:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6630.188850 secs 2024-07-26T13:18:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6630.188850 secs 2024-07-26T13:18:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6630.188850 secs 2024-07-26T13:18:11.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:11 smithi134 ceph-mon[31966]: pgmap v4311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:18:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:11 smithi189 ceph-mon[36790]: pgmap v4311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:18:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:11 smithi193 ceph-mon[37473]: pgmap v4311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:18:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6649.614165 secs 2024-07-26T13:18:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6649.614165 secs 2024-07-26T13:18:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6649.614165 secs 2024-07-26T13:18:13.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:13 smithi134 ceph-mon[31966]: pgmap v4312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:13 smithi189 ceph-mon[36790]: pgmap v4312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:13.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:13 smithi193 ceph-mon[37473]: pgmap v4312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6635.188979 secs 2024-07-26T13:18:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6635.188979 secs 2024-07-26T13:18:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6635.188979 secs 2024-07-26T13:18:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:15 smithi134 ceph-mon[31966]: pgmap v4313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:15 smithi189 ceph-mon[36790]: pgmap v4313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:15 smithi193 ceph-mon[37473]: pgmap v4313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6654.614284 secs 2024-07-26T13:18:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:17 smithi134 ceph-mon[31966]: pgmap v4314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6654.614284 secs 2024-07-26T13:18:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:17 smithi189 ceph-mon[36790]: pgmap v4314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6654.614284 secs 2024-07-26T13:18:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:17 smithi193 ceph-mon[37473]: pgmap v4314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:18.722 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:18:18.721+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:18:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:18:18.721+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:18.723 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:18:18.721+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:18:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:19 smithi134 ceph-mon[31966]: pgmap v4315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6640.189122 secs 2024-07-26T13:18:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:19 smithi189 ceph-mon[36790]: pgmap v4315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6640.189122 secs 2024-07-26T13:18:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:19 smithi193 ceph-mon[37473]: pgmap v4315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6640.189122 secs 2024-07-26T13:18:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:21 smithi134 ceph-mon[31966]: pgmap v4316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:21 smithi189 ceph-mon[36790]: pgmap v4316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:21 smithi193 ceph-mon[37473]: pgmap v4316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:22.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6659.614415 secs 2024-07-26T13:18:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6659.614415 secs 2024-07-26T13:18:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6659.614415 secs 2024-07-26T13:18:23.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:18:23.239+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:18:23.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:18:23.239+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:18:23.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:18:23.240+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:23.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:18:23.483+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:18:23.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:18:23.483+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:18:23.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:18:23.484+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T13:18:23.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:18:23.484+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T13:18:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:23 smithi134 ceph-mon[31966]: pgmap v4317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:23 smithi189 ceph-mon[36790]: pgmap v4317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:23 smithi193 ceph-mon[37473]: pgmap v4317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6645.189245 secs 2024-07-26T13:18:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6645.189245 secs 2024-07-26T13:18:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6645.189245 secs 2024-07-26T13:18:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:25 smithi134 ceph-mon[31966]: pgmap v4318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:25 smithi189 ceph-mon[36790]: pgmap v4318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:25 smithi193 ceph-mon[37473]: pgmap v4318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6664.614543 secs 2024-07-26T13:18:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:27 smithi134 ceph-mon[31966]: pgmap v4319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6664.614543 secs 2024-07-26T13:18:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:27 smithi189 ceph-mon[36790]: pgmap v4319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6664.614543 secs 2024-07-26T13:18:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:27 smithi193 ceph-mon[37473]: pgmap v4319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:29 smithi134 ceph-mon[31966]: pgmap v4320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6650.189392 secs 2024-07-26T13:18:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:29 smithi189 ceph-mon[36790]: pgmap v4320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6650.189392 secs 2024-07-26T13:18:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:29 smithi193 ceph-mon[37473]: pgmap v4320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6650.189392 secs 2024-07-26T13:18:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:31 smithi189 ceph-mon[36790]: pgmap v4321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:31 smithi193 ceph-mon[37473]: pgmap v4321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:31 smithi134 ceph-mon[31966]: pgmap v4321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6669.614666 secs 2024-07-26T13:18:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6669.614666 secs 2024-07-26T13:18:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6669.614666 secs 2024-07-26T13:18:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:18:33.735+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:18:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:18:33.735+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:18:33.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:18:33.735+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:33 smithi193 ceph-mon[37473]: pgmap v4322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:33.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:33 smithi134 ceph-mon[31966]: pgmap v4322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:33 smithi189 ceph-mon[36790]: pgmap v4322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:18:34.003+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:18:34.004 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:18:34.003+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:18:34.005 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:18:34.004+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:18:34.005 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:18:34.004+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:18:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6655.189532 secs 2024-07-26T13:18:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6655.189532 secs 2024-07-26T13:18:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6655.189532 secs 2024-07-26T13:18:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:35 smithi193 ceph-mon[37473]: pgmap v4323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:35 smithi134 ceph-mon[31966]: pgmap v4323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:35 smithi189 ceph-mon[36790]: pgmap v4323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6674.614786 secs 2024-07-26T13:18:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:37 smithi193 ceph-mon[37473]: pgmap v4324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6674.614786 secs 2024-07-26T13:18:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:37 smithi134 ceph-mon[31966]: pgmap v4324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6674.614786 secs 2024-07-26T13:18:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:37 smithi189 ceph-mon[36790]: pgmap v4324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:38.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:18:38.255+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:18:38.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:18:38.255+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:38.257 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:18:38.255+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:18:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:39 smithi193 ceph-mon[37473]: pgmap v4325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6660.189670 secs 2024-07-26T13:18:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:39 smithi134 ceph-mon[31966]: pgmap v4325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6660.189670 secs 2024-07-26T13:18:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:39 smithi189 ceph-mon[36790]: pgmap v4325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6660.189670 secs 2024-07-26T13:18:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:41 smithi134 ceph-mon[31966]: pgmap v4326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:41 smithi193 ceph-mon[37473]: pgmap v4326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:41 smithi189 ceph-mon[36790]: pgmap v4326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6679.614908 secs 2024-07-26T13:18:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6679.614908 secs 2024-07-26T13:18:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6679.614908 secs 2024-07-26T13:18:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:43 smithi193 ceph-mon[37473]: pgmap v4327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:43.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:43 smithi134 ceph-mon[31966]: pgmap v4327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:43 smithi189 ceph-mon[36790]: pgmap v4327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6665.189786 secs 2024-07-26T13:18:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6665.189786 secs 2024-07-26T13:18:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6665.189786 secs 2024-07-26T13:18:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:45 smithi193 ceph-mon[37473]: pgmap v4328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:45.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:45 smithi134 ceph-mon[31966]: pgmap v4328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:45 smithi189 ceph-mon[36790]: pgmap v4328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6684.615029 secs 2024-07-26T13:18:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:47 smithi193 ceph-mon[37473]: pgmap v4329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6684.615029 secs 2024-07-26T13:18:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:47 smithi134 ceph-mon[31966]: pgmap v4329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6684.615029 secs 2024-07-26T13:18:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:47 smithi189 ceph-mon[36790]: pgmap v4329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:18:48.751+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:18:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:18:48.751+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:18:48.752 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:18:48.751+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:49 smithi193 ceph-mon[37473]: pgmap v4330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6670.189911 secs 2024-07-26T13:18:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:49 smithi134 ceph-mon[31966]: pgmap v4330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6670.189911 secs 2024-07-26T13:18:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:49 smithi189 ceph-mon[36790]: pgmap v4330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6670.189911 secs 2024-07-26T13:18:51.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:51 smithi134 ceph-mon[31966]: pgmap v4331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:51 smithi189 ceph-mon[36790]: pgmap v4331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:51 smithi193 ceph-mon[37473]: pgmap v4331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6689.615152 secs 2024-07-26T13:18:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6689.615152 secs 2024-07-26T13:18:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6689.615152 secs 2024-07-26T13:18:53.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:18:53.256+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:18:53.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:18:53.256+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:18:53.258 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:18:53.256+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:18:53.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:53 smithi134 ceph-mon[31966]: pgmap v4332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:53 smithi189 ceph-mon[36790]: pgmap v4332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:53 smithi193 ceph-mon[37473]: pgmap v4332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:18:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6675.190030 secs 2024-07-26T13:18:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6675.190030 secs 2024-07-26T13:18:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6675.190030 secs 2024-07-26T13:18:55.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:55 smithi134 ceph-mon[31966]: pgmap v4333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:55 smithi189 ceph-mon[36790]: pgmap v4333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:55 smithi193 ceph-mon[37473]: pgmap v4333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:18:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6694.615267 secs 2024-07-26T13:18:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:57 smithi134 ceph-mon[31966]: pgmap v4334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6694.615267 secs 2024-07-26T13:18:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:57 smithi189 ceph-mon[36790]: pgmap v4334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6694.615267 secs 2024-07-26T13:18:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:57 smithi193 ceph-mon[37473]: pgmap v4334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:18:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:59 smithi134 ceph-mon[31966]: pgmap v4335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:18:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6680.190098 secs 2024-07-26T13:18:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:59 smithi189 ceph-mon[36790]: pgmap v4335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:18:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:18:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6680.190098 secs 2024-07-26T13:19:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:59 smithi193 ceph-mon[37473]: pgmap v4335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:18:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6680.190098 secs 2024-07-26T13:19:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:19:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:19:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:19:01.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:01 smithi189 ceph-mon[36790]: pgmap v4336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:01.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:19:01.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:19:01.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:19:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:01 smithi134 ceph-mon[31966]: pgmap v4336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:19:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:19:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:19:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:01 smithi193 ceph-mon[37473]: pgmap v4336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:19:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:19:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:19:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6699.615384 secs 2024-07-26T13:19:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:02 smithi134 ceph-mon[31966]: pgmap v4337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:19:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6699.615384 secs 2024-07-26T13:19:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:02 smithi189 ceph-mon[36790]: pgmap v4337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:19:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6699.615384 secs 2024-07-26T13:19:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:02 smithi193 ceph-mon[37473]: pgmap v4337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:19:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:19:03.766+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:19:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:19:03.766+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:19:03.768 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:19:03.766+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6685.190185 secs 2024-07-26T13:19:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:04 smithi134 ceph-mon[31966]: pgmap v4338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:19:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6685.190185 secs 2024-07-26T13:19:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:04 smithi189 ceph-mon[36790]: pgmap v4338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:19:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6685.190185 secs 2024-07-26T13:19:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:04 smithi193 ceph-mon[37473]: pgmap v4338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:19:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:06 smithi134 ceph-mon[31966]: pgmap v4339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:19:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:06 smithi189 ceph-mon[36790]: pgmap v4339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:19:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:06 smithi193 ceph-mon[37473]: pgmap v4339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:19:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6704.615482 secs 2024-07-26T13:19:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6704.615482 secs 2024-07-26T13:19:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6704.615482 secs 2024-07-26T13:19:08.273 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:19:08.272+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:19:08.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:19:08.272+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:08.274 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:19:08.272+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:19:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:08 smithi134 ceph-mon[31966]: pgmap v4340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:19:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:08 smithi189 ceph-mon[36790]: pgmap v4340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:19:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:08 smithi193 ceph-mon[37473]: pgmap v4340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:19:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6690.190268 secs 2024-07-26T13:19:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6690.190268 secs 2024-07-26T13:19:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6690.190268 secs 2024-07-26T13:19:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:10 smithi134 ceph-mon[31966]: pgmap v4341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:19:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:10 smithi189 ceph-mon[36790]: pgmap v4341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:19:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:10 smithi193 ceph-mon[37473]: pgmap v4341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:19:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6709.615636 secs 2024-07-26T13:19:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:12 smithi134 ceph-mon[31966]: pgmap v4342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:19:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6709.615636 secs 2024-07-26T13:19:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:12 smithi189 ceph-mon[36790]: pgmap v4342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:19:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6709.615636 secs 2024-07-26T13:19:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:12 smithi193 ceph-mon[37473]: pgmap v4342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:19:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6695.190344 secs 2024-07-26T13:19:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:14 smithi134 ceph-mon[31966]: pgmap v4343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6695.190344 secs 2024-07-26T13:19:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:14 smithi189 ceph-mon[36790]: pgmap v4343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6695.190344 secs 2024-07-26T13:19:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:14 smithi193 ceph-mon[37473]: pgmap v4343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:16 smithi134 ceph-mon[31966]: pgmap v4344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:16 smithi189 ceph-mon[36790]: pgmap v4344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:16 smithi193 ceph-mon[37473]: pgmap v4344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:17.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6714.615755 secs 2024-07-26T13:19:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6714.615755 secs 2024-07-26T13:19:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6714.615755 secs 2024-07-26T13:19:18.778 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:19:18.777+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:19:18.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:19:18.777+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:19:18.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:19:18.777+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:18 smithi134 ceph-mon[31966]: pgmap v4345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:18 smithi189 ceph-mon[36790]: pgmap v4345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:18 smithi193 ceph-mon[37473]: pgmap v4345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6700.190456 secs 2024-07-26T13:19:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6700.190456 secs 2024-07-26T13:19:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6700.190456 secs 2024-07-26T13:19:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:20 smithi189 ceph-mon[36790]: pgmap v4346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:19:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:20 smithi193 ceph-mon[37473]: pgmap v4346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:19:21.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:20 smithi134 ceph-mon[31966]: pgmap v4346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:19:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6719.615876 secs 2024-07-26T13:19:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:22 smithi189 ceph-mon[36790]: pgmap v4347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6719.615876 secs 2024-07-26T13:19:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:22 smithi193 ceph-mon[37473]: pgmap v4347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:23.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6719.615876 secs 2024-07-26T13:19:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:22 smithi134 ceph-mon[31966]: pgmap v4347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:23.288 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:19:23.287+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:19:23.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:19:23.287+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:23.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:19:23.287+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:19:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6705.190592 secs 2024-07-26T13:19:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:24 smithi189 ceph-mon[36790]: pgmap v4348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6705.190592 secs 2024-07-26T13:19:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:24 smithi193 ceph-mon[37473]: pgmap v4348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6705.190592 secs 2024-07-26T13:19:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:24 smithi134 ceph-mon[31966]: pgmap v4348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:26 smithi189 ceph-mon[36790]: pgmap v4349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:26 smithi193 ceph-mon[37473]: pgmap v4349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:26 smithi134 ceph-mon[31966]: pgmap v4349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6724.616000 secs 2024-07-26T13:19:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6724.616000 secs 2024-07-26T13:19:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6724.616000 secs 2024-07-26T13:19:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:28 smithi193 ceph-mon[37473]: pgmap v4350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:29.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:28 smithi134 ceph-mon[31966]: pgmap v4350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:28 smithi189 ceph-mon[36790]: pgmap v4350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6710.190714 secs 2024-07-26T13:19:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6710.190714 secs 2024-07-26T13:19:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6710.190714 secs 2024-07-26T13:19:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:30 smithi193 ceph-mon[37473]: pgmap v4351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:19:31.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:30 smithi189 ceph-mon[36790]: pgmap v4351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:19:31.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:30 smithi134 ceph-mon[31966]: pgmap v4351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:19:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6729.616146 secs 2024-07-26T13:19:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:32 smithi193 ceph-mon[37473]: pgmap v4352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6729.616146 secs 2024-07-26T13:19:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:32 smithi134 ceph-mon[31966]: pgmap v4352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6729.616146 secs 2024-07-26T13:19:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:32 smithi189 ceph-mon[36790]: pgmap v4352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:19:33.781+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:19:33.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:19:33.781+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:19:33.783 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:19:33.781+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6715.190838 secs 2024-07-26T13:19:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:34 smithi193 ceph-mon[37473]: pgmap v4353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6715.190838 secs 2024-07-26T13:19:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:34 smithi134 ceph-mon[31966]: pgmap v4353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6715.190838 secs 2024-07-26T13:19:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:34 smithi189 ceph-mon[36790]: pgmap v4353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:36 smithi193 ceph-mon[37473]: pgmap v4354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:36 smithi134 ceph-mon[31966]: pgmap v4354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:36 smithi189 ceph-mon[36790]: pgmap v4354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6734.616299 secs 2024-07-26T13:19:38.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6734.616299 secs 2024-07-26T13:19:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6734.616299 secs 2024-07-26T13:19:38.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:19:38.297+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:19:38.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:19:38.297+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:19:38.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:19:38.297+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:38 smithi193 ceph-mon[37473]: pgmap v4355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:38 smithi134 ceph-mon[31966]: pgmap v4355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:38 smithi189 ceph-mon[36790]: pgmap v4355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6720.190955 secs 2024-07-26T13:19:40.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6720.190955 secs 2024-07-26T13:19:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6720.190955 secs 2024-07-26T13:19:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:40 smithi193 ceph-mon[37473]: pgmap v4356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:19:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:40 smithi134 ceph-mon[31966]: pgmap v4356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:19:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:40 smithi189 ceph-mon[36790]: pgmap v4356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:19:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6739.616426 secs 2024-07-26T13:19:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:42 smithi193 ceph-mon[37473]: pgmap v4357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6739.616426 secs 2024-07-26T13:19:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:42 smithi134 ceph-mon[31966]: pgmap v4357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6739.616426 secs 2024-07-26T13:19:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:42 smithi189 ceph-mon[36790]: pgmap v4357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6725.191050 secs 2024-07-26T13:19:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:44 smithi134 ceph-mon[31966]: pgmap v4358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6725.191050 secs 2024-07-26T13:19:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:44 smithi189 ceph-mon[36790]: pgmap v4358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6725.191050 secs 2024-07-26T13:19:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:44 smithi193 ceph-mon[37473]: pgmap v4358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:19:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:46 smithi134 ceph-mon[31966]: pgmap v4359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:46 smithi189 ceph-mon[36790]: pgmap v4359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:46 smithi193 ceph-mon[37473]: pgmap v4359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:19:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6744.616527 secs 2024-07-26T13:19:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6744.616527 secs 2024-07-26T13:19:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6744.616527 secs 2024-07-26T13:19:48.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:19:48.783+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:19:48.784 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:19:48.783+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:19:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:19:48.783+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:48 smithi134 ceph-mon[31966]: pgmap v4360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:19:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:48 smithi189 ceph-mon[36790]: pgmap v4360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:19:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:48 smithi193 ceph-mon[37473]: pgmap v4360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:19:50.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6730.191192 secs 2024-07-26T13:19:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6730.191192 secs 2024-07-26T13:19:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6730.191192 secs 2024-07-26T13:19:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:50 smithi134 ceph-mon[31966]: pgmap v4361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:19:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:50 smithi189 ceph-mon[36790]: pgmap v4361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:19:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:50 smithi193 ceph-mon[37473]: pgmap v4361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:19:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6749.616654 secs 2024-07-26T13:19:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:52 smithi134 ceph-mon[31966]: pgmap v4362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:19:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6749.616654 secs 2024-07-26T13:19:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:52 smithi189 ceph-mon[36790]: pgmap v4362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:19:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6749.616654 secs 2024-07-26T13:19:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:52 smithi193 ceph-mon[37473]: pgmap v4362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:19:53.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:19:53.310+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:19:53.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:19:53.311+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:19:53.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:19:53.311+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:19:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6735.191311 secs 2024-07-26T13:19:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:54 smithi134 ceph-mon[31966]: pgmap v4363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:19:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6735.191311 secs 2024-07-26T13:19:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:54 smithi189 ceph-mon[36790]: pgmap v4363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:19:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6735.191311 secs 2024-07-26T13:19:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:54 smithi193 ceph-mon[37473]: pgmap v4363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:19:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:56 smithi134 ceph-mon[31966]: pgmap v4364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:19:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:56 smithi189 ceph-mon[36790]: pgmap v4364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:19:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:56 smithi193 ceph-mon[37473]: pgmap v4364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:19:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6754.616720 secs 2024-07-26T13:19:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6754.616720 secs 2024-07-26T13:19:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6754.616720 secs 2024-07-26T13:19:59.112 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:58 smithi134 ceph-mon[31966]: pgmap v4365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:19:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:58 smithi189 ceph-mon[36790]: pgmap v4365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:19:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:58 smithi193 ceph-mon[37473]: pgmap v4365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:20:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:19:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6740.191455 secs 2024-07-26T13:20:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:19:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6740.191455 secs 2024-07-26T13:20:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:19:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6740.191455 secs 2024-07-26T13:20:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:00 smithi134 ceph-mon[31966]: pgmap v4366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:20:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:00 smithi189 ceph-mon[36790]: pgmap v4366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:20:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:00 smithi193 ceph-mon[37473]: pgmap v4366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:20:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:20:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:20:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:20:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6759.616848 secs 2024-07-26T13:20:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:02 smithi134 ceph-mon[31966]: pgmap v4367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:20:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:20:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:20:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6759.616848 secs 2024-07-26T13:20:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:02 smithi189 ceph-mon[36790]: pgmap v4367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:20:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:20:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:20:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6759.616848 secs 2024-07-26T13:20:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:02 smithi193 ceph-mon[37473]: pgmap v4367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:20:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:20:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:20:03.785 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:20:03.784+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:20:03.785 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:20:03.784+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:20:03.786 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:20:03.784+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:04.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:03 smithi134 ceph-mon[31966]: pgmap v4368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:20:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:03 smithi134 ceph-mon[31966]: pgmap v4369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:03 smithi189 ceph-mon[36790]: pgmap v4368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:20:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:03 smithi189 ceph-mon[36790]: pgmap v4369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:03 smithi193 ceph-mon[37473]: pgmap v4368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:20:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:03 smithi193 ceph-mon[37473]: pgmap v4369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6745.191575 secs 2024-07-26T13:20:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6745.191575 secs 2024-07-26T13:20:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6745.191575 secs 2024-07-26T13:20:06.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:05 smithi134 ceph-mon[31966]: pgmap v4370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:20:06.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:05 smithi193 ceph-mon[37473]: pgmap v4370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:20:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:05 smithi189 ceph-mon[36790]: pgmap v4370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:20:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6764.616979 secs 2024-07-26T13:20:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6764.616979 secs 2024-07-26T13:20:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6764.616979 secs 2024-07-26T13:20:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:07 smithi134 ceph-mon[31966]: pgmap v4371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:20:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:07 smithi189 ceph-mon[36790]: pgmap v4371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:20:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:07 smithi193 ceph-mon[37473]: pgmap v4371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:20:08.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:20:08.324+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:20:08.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:20:08.324+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:08.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:20:08.324+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:20:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6750.191700 secs 2024-07-26T13:20:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:09 smithi134 ceph-mon[31966]: pgmap v4372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6750.191700 secs 2024-07-26T13:20:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:09 smithi189 ceph-mon[36790]: pgmap v4372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6750.191700 secs 2024-07-26T13:20:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:09 smithi193 ceph-mon[37473]: pgmap v4372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:11 smithi189 ceph-mon[36790]: pgmap v4373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6769.617120 secs 2024-07-26T13:20:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:11 smithi193 ceph-mon[37473]: pgmap v4373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6769.617120 secs 2024-07-26T13:20:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:11 smithi134 ceph-mon[31966]: pgmap v4373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:20:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6769.617120 secs 2024-07-26T13:20:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:13 smithi134 ceph-mon[31966]: pgmap v4374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:20:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:13 smithi189 ceph-mon[36790]: pgmap v4374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:20:13.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:13 smithi193 ceph-mon[37473]: pgmap v4374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:20:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6755.191850 secs 2024-07-26T13:20:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6755.191850 secs 2024-07-26T13:20:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6755.191850 secs 2024-07-26T13:20:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:15 smithi134 ceph-mon[31966]: pgmap v4375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:15 smithi189 ceph-mon[36790]: pgmap v4375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:15 smithi193 ceph-mon[37473]: pgmap v4375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6774.617250 secs 2024-07-26T13:20:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:17 smithi134 ceph-mon[31966]: pgmap v4376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6774.617250 secs 2024-07-26T13:20:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:17 smithi189 ceph-mon[36790]: pgmap v4376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6774.617250 secs 2024-07-26T13:20:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:17 smithi193 ceph-mon[37473]: pgmap v4376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:18.801 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:20:18.801+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:20:18.801 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:20:18.801+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:20:18.801 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:20:18.801+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6760.191981 secs 2024-07-26T13:20:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:19 smithi134 ceph-mon[31966]: pgmap v4377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6760.191981 secs 2024-07-26T13:20:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:19 smithi189 ceph-mon[36790]: pgmap v4377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6760.191981 secs 2024-07-26T13:20:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:19 smithi193 ceph-mon[37473]: pgmap v4377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:21 smithi134 ceph-mon[31966]: pgmap v4378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:21 smithi189 ceph-mon[36790]: pgmap v4378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:21 smithi193 ceph-mon[37473]: pgmap v4378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6779.617373 secs 2024-07-26T13:20:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6779.617373 secs 2024-07-26T13:20:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6779.617373 secs 2024-07-26T13:20:23.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:20:23.340+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:20:23.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:20:23.340+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:23.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:20:23.340+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:20:24.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:23 smithi134 ceph-mon[31966]: pgmap v4379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:23 smithi189 ceph-mon[36790]: pgmap v4379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:23 smithi193 ceph-mon[37473]: pgmap v4379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6765.192102 secs 2024-07-26T13:20:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6765.192102 secs 2024-07-26T13:20:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6765.192102 secs 2024-07-26T13:20:26.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:25 smithi134 ceph-mon[31966]: pgmap v4380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:25 smithi189 ceph-mon[36790]: pgmap v4380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:25 smithi193 ceph-mon[37473]: pgmap v4380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6784.617501 secs 2024-07-26T13:20:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:27 smithi134 ceph-mon[31966]: pgmap v4381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6784.617501 secs 2024-07-26T13:20:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:27 smithi189 ceph-mon[36790]: pgmap v4381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6784.617501 secs 2024-07-26T13:20:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:27 smithi193 ceph-mon[37473]: pgmap v4381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6770.192168 secs 2024-07-26T13:20:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:29 smithi134 ceph-mon[31966]: pgmap v4382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6770.192168 secs 2024-07-26T13:20:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:29 smithi189 ceph-mon[36790]: pgmap v4382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6770.192168 secs 2024-07-26T13:20:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:29 smithi193 ceph-mon[37473]: pgmap v4382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:31 smithi134 ceph-mon[31966]: pgmap v4383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:31 smithi189 ceph-mon[36790]: pgmap v4383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:31 smithi193 ceph-mon[37473]: pgmap v4383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6789.617638 secs 2024-07-26T13:20:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6789.617638 secs 2024-07-26T13:20:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6789.617638 secs 2024-07-26T13:20:33.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:20:33.811+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:20:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:20:33.811+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:20:33.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:20:33.811+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:33 smithi134 ceph-mon[31966]: pgmap v4384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:33 smithi189 ceph-mon[36790]: pgmap v4384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:33 smithi193 ceph-mon[37473]: pgmap v4384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6775.192254 secs 2024-07-26T13:20:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6775.192254 secs 2024-07-26T13:20:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6775.192254 secs 2024-07-26T13:20:36.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:35 smithi134 ceph-mon[31966]: pgmap v4385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:36.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:35 smithi193 ceph-mon[37473]: pgmap v4385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:35 smithi189 ceph-mon[36790]: pgmap v4385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6794.617782 secs 2024-07-26T13:20:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:37 smithi134 ceph-mon[31966]: pgmap v4386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6794.617782 secs 2024-07-26T13:20:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:37 smithi189 ceph-mon[36790]: pgmap v4386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6794.617782 secs 2024-07-26T13:20:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:37 smithi193 ceph-mon[37473]: pgmap v4386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:38.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:20:38.355+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:20:38.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:20:38.355+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:38.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:20:38.355+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:20:40.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6780.192403 secs 2024-07-26T13:20:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:39 smithi134 ceph-mon[31966]: pgmap v4387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6780.192403 secs 2024-07-26T13:20:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:39 smithi189 ceph-mon[36790]: pgmap v4387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6780.192403 secs 2024-07-26T13:20:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:39 smithi193 ceph-mon[37473]: pgmap v4387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:41 smithi134 ceph-mon[31966]: pgmap v4388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6799.617909 secs 2024-07-26T13:20:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:41 smithi189 ceph-mon[36790]: pgmap v4388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6799.617909 secs 2024-07-26T13:20:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:41 smithi193 ceph-mon[37473]: pgmap v4388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6799.617909 secs 2024-07-26T13:20:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:43 smithi134 ceph-mon[31966]: pgmap v4389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:43 smithi189 ceph-mon[36790]: pgmap v4389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:43 smithi193 ceph-mon[37473]: pgmap v4389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6785.192542 secs 2024-07-26T13:20:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6785.192542 secs 2024-07-26T13:20:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6785.192542 secs 2024-07-26T13:20:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:45 smithi134 ceph-mon[31966]: pgmap v4390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:45 smithi189 ceph-mon[36790]: pgmap v4390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:45 smithi193 ceph-mon[37473]: pgmap v4390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6804.618036 secs 2024-07-26T13:20:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6804.618036 secs 2024-07-26T13:20:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6804.618036 secs 2024-07-26T13:20:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:47 smithi189 ceph-mon[36790]: pgmap v4391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:47 smithi193 ceph-mon[37473]: pgmap v4391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:47 smithi134 ceph-mon[31966]: pgmap v4391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:20:48.827+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:20:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:20:48.827+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:20:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:20:48.827+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6790.192679 secs 2024-07-26T13:20:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:49 smithi189 ceph-mon[36790]: pgmap v4392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6790.192679 secs 2024-07-26T13:20:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:49 smithi193 ceph-mon[37473]: pgmap v4392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:50.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6790.192679 secs 2024-07-26T13:20:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:49 smithi134 ceph-mon[31966]: pgmap v4392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:51 smithi189 ceph-mon[36790]: pgmap v4393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6809.618166 secs 2024-07-26T13:20:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:51 smithi193 ceph-mon[37473]: pgmap v4393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6809.618166 secs 2024-07-26T13:20:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:51 smithi134 ceph-mon[31966]: pgmap v4393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6809.618166 secs 2024-07-26T13:20:53.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:20:53.371+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:20:53.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:20:53.371+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:20:53.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:20:53.371+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:20:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:53 smithi189 ceph-mon[36790]: pgmap v4394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:53 smithi193 ceph-mon[37473]: pgmap v4394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:53 smithi134 ceph-mon[31966]: pgmap v4394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:20:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6795.192800 secs 2024-07-26T13:20:55.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6795.192800 secs 2024-07-26T13:20:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6795.192800 secs 2024-07-26T13:20:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:55 smithi193 ceph-mon[37473]: pgmap v4395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:55 smithi134 ceph-mon[31966]: pgmap v4395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:55 smithi189 ceph-mon[36790]: pgmap v4395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:20:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6814.618270 secs 2024-07-26T13:20:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6814.618270 secs 2024-07-26T13:20:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6814.618270 secs 2024-07-26T13:20:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:57 smithi193 ceph-mon[37473]: pgmap v4396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:58.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:57 smithi134 ceph-mon[31966]: pgmap v4396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:20:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:57 smithi189 ceph-mon[36790]: pgmap v4396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6800.192910 secs 2024-07-26T13:21:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:20:59 smithi193 ceph-mon[37473]: pgmap v4397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6800.192910 secs 2024-07-26T13:21:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:20:59 smithi134 ceph-mon[31966]: pgmap v4397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6800.192910 secs 2024-07-26T13:21:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:20:59 smithi189 ceph-mon[36790]: pgmap v4397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:02.245 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:01 smithi189 ceph-mon[36790]: pgmap v4398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:02.245 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6819.618413 secs 2024-07-26T13:21:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:01 smithi193 ceph-mon[37473]: pgmap v4398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6819.618413 secs 2024-07-26T13:21:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:01 smithi134 ceph-mon[31966]: pgmap v4398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6819.618413 secs 2024-07-26T13:21:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:21:03.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:21:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:21:03.843 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:21:03.842+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:21:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:21:03.842+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:03.844 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:21:03.842+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:21:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:04 smithi193 ceph-mon[37473]: pgmap v4399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:21:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:21:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:21:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:04 smithi134 ceph-mon[31966]: pgmap v4399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:21:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:21:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:21:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:04 smithi189 ceph-mon[36790]: pgmap v4399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:21:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:21:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:21:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6805.193023 secs 2024-07-26T13:21:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:05 smithi193 ceph-mon[37473]: pgmap v4400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:21:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:05 smithi193 ceph-mon[37473]: pgmap v4401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6805.193023 secs 2024-07-26T13:21:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:05 smithi134 ceph-mon[31966]: pgmap v4400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:21:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:05 smithi134 ceph-mon[31966]: pgmap v4401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6805.193023 secs 2024-07-26T13:21:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:05 smithi189 ceph-mon[36790]: pgmap v4400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:21:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:05 smithi189 ceph-mon[36790]: pgmap v4401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:07 smithi193 ceph-mon[37473]: pgmap v4402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6824.618510 secs 2024-07-26T13:21:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:07 smithi134 ceph-mon[31966]: pgmap v4402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6824.618510 secs 2024-07-26T13:21:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:07 smithi189 ceph-mon[36790]: pgmap v4402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6824.618510 secs 2024-07-26T13:21:08.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:21:08.386+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:21:08.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:21:08.386+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:08.388 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:21:08.386+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:21:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:09 smithi193 ceph-mon[37473]: pgmap v4403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:09 smithi134 ceph-mon[31966]: pgmap v4403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:09 smithi189 ceph-mon[36790]: pgmap v4403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6810.193114 secs 2024-07-26T13:21:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6810.193114 secs 2024-07-26T13:21:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6810.193114 secs 2024-07-26T13:21:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:11 smithi134 ceph-mon[31966]: pgmap v4404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:11 smithi189 ceph-mon[36790]: pgmap v4404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:11 smithi193 ceph-mon[37473]: pgmap v4404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:21:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6829.618596 secs 2024-07-26T13:21:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6829.618596 secs 2024-07-26T13:21:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6829.618596 secs 2024-07-26T13:21:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:13 smithi134 ceph-mon[31966]: pgmap v4405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:21:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:13 smithi189 ceph-mon[36790]: pgmap v4405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:21:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:13 smithi193 ceph-mon[37473]: pgmap v4405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:21:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6815.193182 secs 2024-07-26T13:21:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:15 smithi134 ceph-mon[31966]: pgmap v4406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:21:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6815.193182 secs 2024-07-26T13:21:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:15 smithi189 ceph-mon[36790]: pgmap v4406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:21:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6815.193182 secs 2024-07-26T13:21:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:15 smithi193 ceph-mon[37473]: pgmap v4406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:21:17.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:17 smithi134 ceph-mon[31966]: pgmap v4407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:17.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6834.618718 secs 2024-07-26T13:21:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:17 smithi189 ceph-mon[36790]: pgmap v4407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6834.618718 secs 2024-07-26T13:21:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:17 smithi193 ceph-mon[37473]: pgmap v4407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6834.618718 secs 2024-07-26T13:21:18.860 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:21:18.858+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:21:18.860 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:21:18.858+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:21:18.861 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:21:18.858+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:19 smithi134 ceph-mon[31966]: pgmap v4408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:19 smithi189 ceph-mon[36790]: pgmap v4408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:19 smithi193 ceph-mon[37473]: pgmap v4408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6820.193281 secs 2024-07-26T13:21:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6820.193281 secs 2024-07-26T13:21:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6820.193281 secs 2024-07-26T13:21:21.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:21 smithi134 ceph-mon[31966]: pgmap v4409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:21 smithi189 ceph-mon[36790]: pgmap v4409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:21 smithi193 ceph-mon[37473]: pgmap v4409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6839.618818 secs 2024-07-26T13:21:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6839.618818 secs 2024-07-26T13:21:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6839.618818 secs 2024-07-26T13:21:23.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:21:23.402+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:21:23.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:21:23.402+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:23.404 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:21:23.402+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:21:23.404 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:23 smithi134 ceph-mon[31966]: pgmap v4410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:23 smithi189 ceph-mon[36790]: pgmap v4410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:23.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:21:23.514+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:21:23.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:21:23.514+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:21:23.515 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:21:23.514+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:21:23.516 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:21:23.514+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:21:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:23 smithi193 ceph-mon[37473]: pgmap v4410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6825.193348 secs 2024-07-26T13:21:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:25 smithi134 ceph-mon[31966]: pgmap v4411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6825.193348 secs 2024-07-26T13:21:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:25 smithi189 ceph-mon[36790]: pgmap v4411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6825.193348 secs 2024-07-26T13:21:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:25 smithi193 ceph-mon[37473]: pgmap v4411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:27 smithi134 ceph-mon[31966]: pgmap v4412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6844.618915 secs 2024-07-26T13:21:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:27 smithi189 ceph-mon[36790]: pgmap v4412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6844.618915 secs 2024-07-26T13:21:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:27 smithi193 ceph-mon[37473]: pgmap v4412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6844.618915 secs 2024-07-26T13:21:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:29 smithi134 ceph-mon[31966]: pgmap v4413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:29 smithi189 ceph-mon[36790]: pgmap v4413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:29 smithi193 ceph-mon[37473]: pgmap v4413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:30.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6830.193490 secs 2024-07-26T13:21:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6830.193490 secs 2024-07-26T13:21:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6830.193490 secs 2024-07-26T13:21:31.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:31 smithi134 ceph-mon[31966]: pgmap v4414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:31 smithi189 ceph-mon[36790]: pgmap v4414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:31 smithi193 ceph-mon[37473]: pgmap v4414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6849.619052 secs 2024-07-26T13:21:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6849.619052 secs 2024-07-26T13:21:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6849.619052 secs 2024-07-26T13:21:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:33 smithi134 ceph-mon[31966]: pgmap v4415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:33 smithi189 ceph-mon[36790]: pgmap v4415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:33 smithi193 ceph-mon[37473]: pgmap v4415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:21:33.874+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:21:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:21:33.874+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:21:33.874+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:21:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:21:34.035+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:21:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:21:34.035+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:21:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:21:34.035+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:21:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:21:34.035+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:21:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6835.193562 secs 2024-07-26T13:21:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:35 smithi134 ceph-mon[31966]: pgmap v4416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6835.193562 secs 2024-07-26T13:21:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:35 smithi189 ceph-mon[36790]: pgmap v4416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6835.193562 secs 2024-07-26T13:21:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:35 smithi193 ceph-mon[37473]: pgmap v4416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:37 smithi134 ceph-mon[31966]: pgmap v4417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6854.619193 secs 2024-07-26T13:21:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:37 smithi189 ceph-mon[36790]: pgmap v4417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6854.619193 secs 2024-07-26T13:21:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:37 smithi193 ceph-mon[37473]: pgmap v4417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6854.619193 secs 2024-07-26T13:21:38.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:21:38.418+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:21:38.419 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:21:38.418+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:38.420 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:21:38.418+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:21:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:39 smithi134 ceph-mon[31966]: pgmap v4418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:39 smithi189 ceph-mon[36790]: pgmap v4418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:39 smithi193 ceph-mon[37473]: pgmap v4418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6840.193702 secs 2024-07-26T13:21:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6840.193702 secs 2024-07-26T13:21:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6840.193702 secs 2024-07-26T13:21:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:41 smithi189 ceph-mon[36790]: pgmap v4419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:41 smithi193 ceph-mon[37473]: pgmap v4419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:41 smithi134 ceph-mon[31966]: pgmap v4419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6859.619339 secs 2024-07-26T13:21:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6859.619339 secs 2024-07-26T13:21:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6859.619339 secs 2024-07-26T13:21:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:43 smithi189 ceph-mon[36790]: pgmap v4420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:43 smithi193 ceph-mon[37473]: pgmap v4420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:43.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:43 smithi134 ceph-mon[31966]: pgmap v4420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6845.193775 secs 2024-07-26T13:21:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:45 smithi189 ceph-mon[36790]: pgmap v4421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6845.193775 secs 2024-07-26T13:21:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:45 smithi193 ceph-mon[37473]: pgmap v4421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6845.193775 secs 2024-07-26T13:21:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:45 smithi134 ceph-mon[31966]: pgmap v4421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:47 smithi193 ceph-mon[37473]: pgmap v4422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6864.619451 secs 2024-07-26T13:21:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:47 smithi134 ceph-mon[31966]: pgmap v4422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6864.619451 secs 2024-07-26T13:21:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:47 smithi189 ceph-mon[36790]: pgmap v4422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6864.619451 secs 2024-07-26T13:21:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:21:48.878+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:21:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:21:48.879+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:21:48.880 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:21:48.879+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:49 smithi193 ceph-mon[37473]: pgmap v4423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:49 smithi134 ceph-mon[31966]: pgmap v4423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:49 smithi189 ceph-mon[36790]: pgmap v4423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6850.193901 secs 2024-07-26T13:21:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6850.193901 secs 2024-07-26T13:21:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6850.193901 secs 2024-07-26T13:21:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:51 smithi193 ceph-mon[37473]: pgmap v4424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:51.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:51 smithi134 ceph-mon[31966]: pgmap v4424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:51 smithi189 ceph-mon[36790]: pgmap v4424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6869.619536 secs 2024-07-26T13:21:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6869.619536 secs 2024-07-26T13:21:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6869.619536 secs 2024-07-26T13:21:53.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:21:53.433+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:21:53.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:21:53.433+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:21:53.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:21:53.433+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:21:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:53 smithi193 ceph-mon[37473]: pgmap v4425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:53.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:53 smithi134 ceph-mon[31966]: pgmap v4425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:53 smithi189 ceph-mon[36790]: pgmap v4425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:21:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6855.194015 secs 2024-07-26T13:21:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:55 smithi193 ceph-mon[37473]: pgmap v4426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6855.194015 secs 2024-07-26T13:21:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:55 smithi134 ceph-mon[31966]: pgmap v4426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6855.194015 secs 2024-07-26T13:21:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:55 smithi189 ceph-mon[36790]: pgmap v4426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:21:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:57 smithi193 ceph-mon[37473]: pgmap v4427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6874.619657 secs 2024-07-26T13:21:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:57 smithi134 ceph-mon[31966]: pgmap v4427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6874.619657 secs 2024-07-26T13:21:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:57 smithi189 ceph-mon[36790]: pgmap v4427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:21:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6874.619657 secs 2024-07-26T13:21:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:21:59 smithi193 ceph-mon[37473]: pgmap v4428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:21:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:21:59 smithi134 ceph-mon[31966]: pgmap v4428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:21:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:21:59 smithi189 ceph-mon[36790]: pgmap v4428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:22:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6860.194132 secs 2024-07-26T13:22:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6860.194132 secs 2024-07-26T13:22:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6860.194132 secs 2024-07-26T13:22:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:01 smithi193 ceph-mon[37473]: pgmap v4429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:22:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:01 smithi134 ceph-mon[31966]: pgmap v4429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:22:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:01 smithi189 ceph-mon[36790]: pgmap v4429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:22:02.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6879.619774 secs 2024-07-26T13:22:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6879.619774 secs 2024-07-26T13:22:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6879.619774 secs 2024-07-26T13:22:03.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:03 smithi134 ceph-mon[31966]: pgmap v4430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:22:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:03 smithi189 ceph-mon[36790]: pgmap v4430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:22:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:03 smithi193 ceph-mon[37473]: pgmap v4430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:22:03.895 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:22:03.894+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:22:03.895 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:22:03.894+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:22:03.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:22:03.894+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:22:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:22:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:22:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6865.194252 secs 2024-07-26T13:22:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:05 smithi134 ceph-mon[31966]: pgmap v4431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:22:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:22:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:22:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:22:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6865.194252 secs 2024-07-26T13:22:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:05 smithi189 ceph-mon[36790]: pgmap v4431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:22:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:22:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:22:05.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:22:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6865.194252 secs 2024-07-26T13:22:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:05 smithi193 ceph-mon[37473]: pgmap v4431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:22:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:22:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:22:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:22:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:06 smithi134 ceph-mon[31966]: pgmap v4432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:22:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:06 smithi134 ceph-mon[31966]: pgmap v4433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:22:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:06 smithi189 ceph-mon[36790]: pgmap v4432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:22:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:06 smithi189 ceph-mon[36790]: pgmap v4433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:22:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:06 smithi193 ceph-mon[37473]: pgmap v4432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:22:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:06 smithi193 ceph-mon[37473]: pgmap v4433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:22:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6884.619892 secs 2024-07-26T13:22:07.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6884.619892 secs 2024-07-26T13:22:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6884.619892 secs 2024-07-26T13:22:08.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:22:08.448+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:22:08.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:22:08.448+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:08.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:22:08.448+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:22:08.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:08 smithi134 ceph-mon[31966]: pgmap v4434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:22:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:08 smithi189 ceph-mon[36790]: pgmap v4434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:22:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:08 smithi193 ceph-mon[37473]: pgmap v4434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:22:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6870.194365 secs 2024-07-26T13:22:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:10 smithi134 ceph-mon[31966]: pgmap v4435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:22:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6870.194365 secs 2024-07-26T13:22:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:10 smithi189 ceph-mon[36790]: pgmap v4435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:22:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6870.194365 secs 2024-07-26T13:22:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:10 smithi193 ceph-mon[37473]: pgmap v4435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:22:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6889.620007 secs 2024-07-26T13:22:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:12 smithi134 ceph-mon[31966]: pgmap v4436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:22:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6889.620007 secs 2024-07-26T13:22:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:12 smithi189 ceph-mon[36790]: pgmap v4436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:22:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6889.620007 secs 2024-07-26T13:22:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:12 smithi193 ceph-mon[37473]: pgmap v4436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:22:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:14 smithi134 ceph-mon[31966]: pgmap v4437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:22:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:14 smithi189 ceph-mon[36790]: pgmap v4437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:22:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:14 smithi193 ceph-mon[37473]: pgmap v4437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:22:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6875.194490 secs 2024-07-26T13:22:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:15 smithi134 ceph-mon[31966]: pgmap v4438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6875.194490 secs 2024-07-26T13:22:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:15 smithi189 ceph-mon[36790]: pgmap v4438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6875.194490 secs 2024-07-26T13:22:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:15 smithi193 ceph-mon[37473]: pgmap v4438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:22:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6894.620141 secs 2024-07-26T13:22:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6894.620141 secs 2024-07-26T13:22:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6894.620141 secs 2024-07-26T13:22:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:17 smithi193 ceph-mon[37473]: pgmap v4439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:22:18.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:17 smithi134 ceph-mon[31966]: pgmap v4439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:22:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:17 smithi189 ceph-mon[36790]: pgmap v4439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:22:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:22:18.909+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:22:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:22:18.909+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:22:18.910 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:22:18.909+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6880.194627 secs 2024-07-26T13:22:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:20 smithi134 ceph-mon[31966]: pgmap v4440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6880.194627 secs 2024-07-26T13:22:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:20 smithi189 ceph-mon[36790]: pgmap v4440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6880.194627 secs 2024-07-26T13:22:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:20 smithi193 ceph-mon[37473]: pgmap v4440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6899.620266 secs 2024-07-26T13:22:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:22 smithi189 ceph-mon[36790]: pgmap v4441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6899.620266 secs 2024-07-26T13:22:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:22 smithi193 ceph-mon[37473]: pgmap v4441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6899.620266 secs 2024-07-26T13:22:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:22 smithi134 ceph-mon[31966]: pgmap v4441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:23.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:22:23.464+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:22:23.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:22:23.464+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:23.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:22:23.464+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:22:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:24 smithi189 ceph-mon[36790]: pgmap v4442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:22:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:24 smithi193 ceph-mon[37473]: pgmap v4442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:22:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:24 smithi134 ceph-mon[31966]: pgmap v4442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:22:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6885.194750 secs 2024-07-26T13:22:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6885.194750 secs 2024-07-26T13:22:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6885.194750 secs 2024-07-26T13:22:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:26 smithi193 ceph-mon[37473]: pgmap v4443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:26 smithi134 ceph-mon[31966]: pgmap v4443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:26 smithi189 ceph-mon[36790]: pgmap v4443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6904.620386 secs 2024-07-26T13:22:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6904.620386 secs 2024-07-26T13:22:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6904.620386 secs 2024-07-26T13:22:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:28 smithi193 ceph-mon[37473]: pgmap v4444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:28.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:28 smithi134 ceph-mon[31966]: pgmap v4444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:28 smithi189 ceph-mon[36790]: pgmap v4444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6890.194894 secs 2024-07-26T13:22:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:30 smithi193 ceph-mon[37473]: pgmap v4445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:22:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6890.194894 secs 2024-07-26T13:22:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:30 smithi134 ceph-mon[31966]: pgmap v4445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:22:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6890.194894 secs 2024-07-26T13:22:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:30 smithi189 ceph-mon[36790]: pgmap v4445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:22:32.495 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6909.620470 secs 2024-07-26T13:22:32.495 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:32 smithi189 ceph-mon[36790]: pgmap v4446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:22:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6909.620470 secs 2024-07-26T13:22:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:32 smithi193 ceph-mon[37473]: pgmap v4446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:22:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6909.620470 secs 2024-07-26T13:22:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:32 smithi134 ceph-mon[31966]: pgmap v4446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:22:33.916 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:22:33.915+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:22:33.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:22:33.915+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:33.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:22:33.915+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:22:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:34 smithi193 ceph-mon[37473]: pgmap v4447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:22:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:34 smithi134 ceph-mon[31966]: pgmap v4447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:22:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:34 smithi189 ceph-mon[36790]: pgmap v4447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:22:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6895.195006 secs 2024-07-26T13:22:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6895.195006 secs 2024-07-26T13:22:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6895.195006 secs 2024-07-26T13:22:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:36 smithi193 ceph-mon[37473]: pgmap v4448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:22:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:36 smithi134 ceph-mon[31966]: pgmap v4448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:22:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:36 smithi189 ceph-mon[36790]: pgmap v4448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:22:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6914.620599 secs 2024-07-26T13:22:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6914.620599 secs 2024-07-26T13:22:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6914.620599 secs 2024-07-26T13:22:38.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:22:38.479+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:22:38.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:22:38.479+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:38.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:22:38.479+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:22:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:38 smithi193 ceph-mon[37473]: pgmap v4449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T13:22:38.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:38 smithi134 ceph-mon[31966]: pgmap v4449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T13:22:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:38 smithi189 ceph-mon[36790]: pgmap v4449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T13:22:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6900.195091 secs 2024-07-26T13:22:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:40 smithi193 ceph-mon[37473]: pgmap v4450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:22:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6900.195091 secs 2024-07-26T13:22:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:40 smithi134 ceph-mon[31966]: pgmap v4450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:22:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6900.195091 secs 2024-07-26T13:22:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:40 smithi189 ceph-mon[36790]: pgmap v4450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:22:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6919.620711 secs 2024-07-26T13:22:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:42 smithi193 ceph-mon[37473]: pgmap v4451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6919.620711 secs 2024-07-26T13:22:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:42 smithi134 ceph-mon[31966]: pgmap v4451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6919.620711 secs 2024-07-26T13:22:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:42 smithi189 ceph-mon[36790]: pgmap v4451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:44 smithi193 ceph-mon[37473]: pgmap v4452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:22:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:44 smithi134 ceph-mon[31966]: pgmap v4452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:22:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:44 smithi189 ceph-mon[36790]: pgmap v4452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:22:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6905.195173 secs 2024-07-26T13:22:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6905.195173 secs 2024-07-26T13:22:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6905.195173 secs 2024-07-26T13:22:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:46 smithi134 ceph-mon[31966]: pgmap v4453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:46 smithi189 ceph-mon[36790]: pgmap v4453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:46 smithi193 ceph-mon[37473]: pgmap v4453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6924.620826 secs 2024-07-26T13:22:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6924.620826 secs 2024-07-26T13:22:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6924.620826 secs 2024-07-26T13:22:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:48 smithi134 ceph-mon[31966]: pgmap v4454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:48 smithi189 ceph-mon[36790]: pgmap v4454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:48 smithi193 ceph-mon[37473]: pgmap v4454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:22:48.931+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:22:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:22:48.931+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:48.932 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:22:48.931+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:22:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6910.195287 secs 2024-07-26T13:22:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:50 smithi134 ceph-mon[31966]: pgmap v4455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6910.195287 secs 2024-07-26T13:22:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:50 smithi189 ceph-mon[36790]: pgmap v4455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6910.195287 secs 2024-07-26T13:22:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:50 smithi193 ceph-mon[37473]: pgmap v4455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:22:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6929.620959 secs 2024-07-26T13:22:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:52 smithi134 ceph-mon[31966]: pgmap v4456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6929.620959 secs 2024-07-26T13:22:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:52 smithi189 ceph-mon[36790]: pgmap v4456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6929.620959 secs 2024-07-26T13:22:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:52 smithi193 ceph-mon[37473]: pgmap v4456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:53.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:22:53.492+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:22:53.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:22:53.492+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:22:53.493 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:22:53.492+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:22:53.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:22:53.530+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:22:53.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:22:53.530+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:22:53.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:22:53.530+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T13:22:53.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:22:53.530+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T13:22:53.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:53 smithi134 ceph-mon[31966]: pgmap v4457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:22:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:53 smithi189 ceph-mon[36790]: pgmap v4457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:22:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:53 smithi193 ceph-mon[37473]: pgmap v4457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:22:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6915.195356 secs 2024-07-26T13:22:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6915.195356 secs 2024-07-26T13:22:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6915.195356 secs 2024-07-26T13:22:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:55 smithi134 ceph-mon[31966]: pgmap v4458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:55 smithi189 ceph-mon[36790]: pgmap v4458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:55 smithi193 ceph-mon[37473]: pgmap v4458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:22:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6934.621076 secs 2024-07-26T13:22:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6934.621076 secs 2024-07-26T13:22:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6934.621076 secs 2024-07-26T13:22:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:22:57 smithi193 ceph-mon[37473]: pgmap v4459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:22:57 smithi134 ceph-mon[31966]: pgmap v4459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:22:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:22:57 smithi189 ceph-mon[36790]: pgmap v4459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6920.195424 secs 2024-07-26T13:23:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:00 smithi193 ceph-mon[37473]: pgmap v4460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6920.195424 secs 2024-07-26T13:23:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:00 smithi134 ceph-mon[31966]: pgmap v4460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6920.195424 secs 2024-07-26T13:23:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:00 smithi189 ceph-mon[36790]: pgmap v4460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:02.496 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6939.621197 secs 2024-07-26T13:23:02.496 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:02 smithi189 ceph-mon[36790]: pgmap v4461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6939.621197 secs 2024-07-26T13:23:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:02 smithi193 ceph-mon[37473]: pgmap v4461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6939.621197 secs 2024-07-26T13:23:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:02 smithi134 ceph-mon[31966]: pgmap v4461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:03.947 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:23:03.947+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:23:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:23:03.947+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:23:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:23:03.947+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:04.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:23:04.051+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:23:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:23:04.051+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:23:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:23:04.051+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:23:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:23:04.051+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:23:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:04 smithi193 ceph-mon[37473]: pgmap v4462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:04 smithi134 ceph-mon[31966]: pgmap v4462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:04 smithi189 ceph-mon[36790]: pgmap v4462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6925.195547 secs 2024-07-26T13:23:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6925.195547 secs 2024-07-26T13:23:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6925.195547 secs 2024-07-26T13:23:06.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:06 smithi134 ceph-mon[31966]: pgmap v4463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:23:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:06 smithi193 ceph-mon[37473]: pgmap v4463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:23:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:06 smithi189 ceph-mon[36790]: pgmap v4463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:23:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6944.621326 secs 2024-07-26T13:23:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:07 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:23:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:07 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:23:07.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:23:07.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:23:07.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:07 smithi193 ceph-mon[37473]: pgmap v4464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:23:07.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:07 smithi193 ceph-mon[37473]: pgmap v4465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:23:07.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:23:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6944.621326 secs 2024-07-26T13:23:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:07 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:23:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:07 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:23:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:23:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:23:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:07 smithi134 ceph-mon[31966]: pgmap v4464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:23:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:07 smithi134 ceph-mon[31966]: pgmap v4465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:23:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:23:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6944.621326 secs 2024-07-26T13:23:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:23:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:07 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:23:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:07 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:23:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:23:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:23:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:07 smithi189 ceph-mon[36790]: pgmap v4464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:23:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:07 smithi189 ceph-mon[36790]: pgmap v4465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:23:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:23:08.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:23:08.507+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:23:08.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:23:08.507+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:08.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:23:08.507+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:23:08.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:08 smithi134 ceph-mon[31966]: pgmap v4466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:23:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:08 smithi189 ceph-mon[36790]: pgmap v4466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:23:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:08 smithi193 ceph-mon[37473]: pgmap v4466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:23:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6930.195690 secs 2024-07-26T13:23:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6930.195690 secs 2024-07-26T13:23:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6930.195690 secs 2024-07-26T13:23:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:10 smithi134 ceph-mon[31966]: pgmap v4467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:23:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:10 smithi189 ceph-mon[36790]: pgmap v4467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:23:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:10 smithi193 ceph-mon[37473]: pgmap v4467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:23:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6949.621443 secs 2024-07-26T13:23:12.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6949.621443 secs 2024-07-26T13:23:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6949.621443 secs 2024-07-26T13:23:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:12 smithi193 ceph-mon[37473]: pgmap v4468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:23:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:12 smithi134 ceph-mon[31966]: pgmap v4468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:23:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:12 smithi189 ceph-mon[36790]: pgmap v4468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:23:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6935.195825 secs 2024-07-26T13:23:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:14 smithi134 ceph-mon[31966]: pgmap v4469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:23:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6935.195825 secs 2024-07-26T13:23:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:14 smithi189 ceph-mon[36790]: pgmap v4469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:23:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6935.195825 secs 2024-07-26T13:23:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:14 smithi193 ceph-mon[37473]: pgmap v4469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:23:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:16 smithi134 ceph-mon[31966]: pgmap v4470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:23:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:16 smithi189 ceph-mon[36790]: pgmap v4470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:23:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:16 smithi193 ceph-mon[37473]: pgmap v4470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:23:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6954.621595 secs 2024-07-26T13:23:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6954.621595 secs 2024-07-26T13:23:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6954.621595 secs 2024-07-26T13:23:18.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:18 smithi134 ceph-mon[31966]: pgmap v4471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:18 smithi189 ceph-mon[36790]: pgmap v4471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:18 smithi193 ceph-mon[37473]: pgmap v4471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:23:18.955+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:23:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:23:18.955+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:18.956 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:23:18.955+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:23:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6940.195875 secs 2024-07-26T13:23:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6940.195875 secs 2024-07-26T13:23:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6940.195875 secs 2024-07-26T13:23:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:20 smithi189 ceph-mon[36790]: pgmap v4472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:20 smithi193 ceph-mon[37473]: pgmap v4472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:20 smithi134 ceph-mon[31966]: pgmap v4472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6959.621716 secs 2024-07-26T13:23:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6959.621716 secs 2024-07-26T13:23:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6959.621716 secs 2024-07-26T13:23:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:22 smithi193 ceph-mon[37473]: pgmap v4473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:23.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:22 smithi134 ceph-mon[31966]: pgmap v4473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:22 smithi189 ceph-mon[36790]: pgmap v4473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:23.523 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:23:23.522+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:23:23.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:23:23.522+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:23.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:23:23.522+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:23:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6945.196017 secs 2024-07-26T13:23:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:24 smithi134 ceph-mon[31966]: pgmap v4474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6945.196017 secs 2024-07-26T13:23:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:24 smithi189 ceph-mon[36790]: pgmap v4474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6945.196017 secs 2024-07-26T13:23:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:24 smithi193 ceph-mon[37473]: pgmap v4474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:26 smithi134 ceph-mon[31966]: pgmap v4475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:26 smithi189 ceph-mon[36790]: pgmap v4475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:26 smithi193 ceph-mon[37473]: pgmap v4475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6964.621821 secs 2024-07-26T13:23:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6964.621821 secs 2024-07-26T13:23:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6964.621821 secs 2024-07-26T13:23:28.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:28 smithi134 ceph-mon[31966]: pgmap v4476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:28 smithi189 ceph-mon[36790]: pgmap v4476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:28 smithi193 ceph-mon[37473]: pgmap v4476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6950.196118 secs 2024-07-26T13:23:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6950.196118 secs 2024-07-26T13:23:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6950.196118 secs 2024-07-26T13:23:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:30 smithi189 ceph-mon[36790]: pgmap v4477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:30 smithi193 ceph-mon[37473]: pgmap v4477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:30 smithi134 ceph-mon[31966]: pgmap v4477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6969.621969 secs 2024-07-26T13:23:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6969.621969 secs 2024-07-26T13:23:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6969.621969 secs 2024-07-26T13:23:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:32 smithi193 ceph-mon[37473]: pgmap v4478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:32 smithi134 ceph-mon[31966]: pgmap v4478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:32 smithi189 ceph-mon[36790]: pgmap v4478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:33.971 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:23:33.970+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:23:33.972 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:23:33.970+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:23:33.972 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:23:33.970+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6955.196236 secs 2024-07-26T13:23:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:34 smithi134 ceph-mon[31966]: pgmap v4479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6955.196236 secs 2024-07-26T13:23:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:34 smithi189 ceph-mon[36790]: pgmap v4479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6955.196236 secs 2024-07-26T13:23:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:34 smithi193 ceph-mon[37473]: pgmap v4479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:36 smithi134 ceph-mon[31966]: pgmap v4480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:36 smithi189 ceph-mon[36790]: pgmap v4480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:36 smithi193 ceph-mon[37473]: pgmap v4480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6974.622099 secs 2024-07-26T13:23:37.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6974.622099 secs 2024-07-26T13:23:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6974.622099 secs 2024-07-26T13:23:38.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:23:38.538+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:23:38.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:23:38.538+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:23:38.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:23:38.538+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:38.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:38 smithi134 ceph-mon[31966]: pgmap v4481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:38 smithi189 ceph-mon[36790]: pgmap v4481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:38 smithi193 ceph-mon[37473]: pgmap v4481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6960.196336 secs 2024-07-26T13:23:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6960.196336 secs 2024-07-26T13:23:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6960.196336 secs 2024-07-26T13:23:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:40 smithi189 ceph-mon[36790]: pgmap v4482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:40 smithi193 ceph-mon[37473]: pgmap v4482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:40 smithi134 ceph-mon[31966]: pgmap v4482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6979.622224 secs 2024-07-26T13:23:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6979.622224 secs 2024-07-26T13:23:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6979.622224 secs 2024-07-26T13:23:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:42 smithi193 ceph-mon[37473]: pgmap v4483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:43.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:42 smithi134 ceph-mon[31966]: pgmap v4483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:42 smithi189 ceph-mon[36790]: pgmap v4483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6965.196461 secs 2024-07-26T13:23:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:44 smithi134 ceph-mon[31966]: pgmap v4484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6965.196461 secs 2024-07-26T13:23:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:44 smithi189 ceph-mon[36790]: pgmap v4484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6965.196461 secs 2024-07-26T13:23:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:44 smithi193 ceph-mon[37473]: pgmap v4484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:46 smithi134 ceph-mon[31966]: pgmap v4485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:46 smithi189 ceph-mon[36790]: pgmap v4485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:46 smithi193 ceph-mon[37473]: pgmap v4485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6984.622392 secs 2024-07-26T13:23:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6984.622392 secs 2024-07-26T13:23:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6984.622392 secs 2024-07-26T13:23:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:48 smithi134 ceph-mon[31966]: pgmap v4486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:48 smithi189 ceph-mon[36790]: pgmap v4486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:48 smithi193 ceph-mon[37473]: pgmap v4486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:48.973 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:23:48.972+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:23:48.974 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:23:48.972+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:48.974 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:23:48.972+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:23:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6970.196527 secs 2024-07-26T13:23:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6970.196527 secs 2024-07-26T13:23:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6970.196527 secs 2024-07-26T13:23:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:50 smithi189 ceph-mon[36790]: pgmap v4487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:50 smithi193 ceph-mon[37473]: pgmap v4487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:50 smithi134 ceph-mon[31966]: pgmap v4487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6989.622517 secs 2024-07-26T13:23:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6989.622517 secs 2024-07-26T13:23:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6989.622517 secs 2024-07-26T13:23:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:52 smithi193 ceph-mon[37473]: pgmap v4488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:53.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:52 smithi134 ceph-mon[31966]: pgmap v4488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:52 smithi189 ceph-mon[36790]: pgmap v4488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:23:53.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:23:53.553+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:23:53.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:23:53.553+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:23:53.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:23:53.553+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:23:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6975.196651 secs 2024-07-26T13:23:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:54 smithi134 ceph-mon[31966]: pgmap v4489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6975.196651 secs 2024-07-26T13:23:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:54 smithi189 ceph-mon[36790]: pgmap v4489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6975.196651 secs 2024-07-26T13:23:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:54 smithi193 ceph-mon[37473]: pgmap v4489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:23:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:56 smithi134 ceph-mon[31966]: pgmap v4490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:56 smithi189 ceph-mon[36790]: pgmap v4490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:56 smithi193 ceph-mon[37473]: pgmap v4490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:23:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6994.622621 secs 2024-07-26T13:23:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6994.622621 secs 2024-07-26T13:23:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6994.622621 secs 2024-07-26T13:23:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:58 smithi134 ceph-mon[31966]: pgmap v4491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:58.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:58 smithi189 ceph-mon[36790]: pgmap v4491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:58 smithi193 ceph-mon[37473]: pgmap v4491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:23:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:23:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6980.196762 secs 2024-07-26T13:23:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:23:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6980.196762 secs 2024-07-26T13:23:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:23:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6980.196762 secs 2024-07-26T13:24:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:00 smithi189 ceph-mon[36790]: pgmap v4492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:00 smithi193 ceph-mon[37473]: pgmap v4492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:00 smithi134 ceph-mon[31966]: pgmap v4492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 6999.622748 secs 2024-07-26T13:24:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 6999.622748 secs 2024-07-26T13:24:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 6999.622748 secs 2024-07-26T13:24:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:02 smithi193 ceph-mon[37473]: pgmap v4493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:03.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:02 smithi134 ceph-mon[31966]: pgmap v4493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:02 smithi189 ceph-mon[36790]: pgmap v4493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:24:03.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:24:03.987+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:24:03.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:24:03.988+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:24:03.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:24:03.988+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6985.196883 secs 2024-07-26T13:24:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:04 smithi134 ceph-mon[31966]: pgmap v4494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6985.196883 secs 2024-07-26T13:24:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:04 smithi189 ceph-mon[36790]: pgmap v4494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:04.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6985.196883 secs 2024-07-26T13:24:04.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:04 smithi193 ceph-mon[37473]: pgmap v4494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:24:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:06 smithi134 ceph-mon[31966]: pgmap v4495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:06 smithi189 ceph-mon[36790]: pgmap v4495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:06 smithi193 ceph-mon[37473]: pgmap v4495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7004.622872 secs 2024-07-26T13:24:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:24:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:07 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:24:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:07 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:24:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:07 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:24:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:07 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:24:07.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7004.622872 secs 2024-07-26T13:24:07.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:24:07.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:07 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:24:07.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:07 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:24:07.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:07 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:24:07.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:07 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:24:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7004.622872 secs 2024-07-26T13:24:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:24:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:24:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:07 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:24:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:07 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:24:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:07 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:24:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:07 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:24:08.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:24:08.554+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:24:08.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:24:08.554+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:24:08.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:24:08.554+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:24:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:24:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:08 smithi134 ceph-mon[31966]: pgmap v4496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:24:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:24:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:24:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:24:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:08 smithi189 ceph-mon[36790]: pgmap v4496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:24:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:24:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:24:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:24:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:08 smithi193 ceph-mon[37473]: pgmap v4496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:24:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:24:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6990.196969 secs 2024-07-26T13:24:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6990.196969 secs 2024-07-26T13:24:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6990.196969 secs 2024-07-26T13:24:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:10 smithi189 ceph-mon[36790]: pgmap v4497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:24:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:10 smithi193 ceph-mon[37473]: pgmap v4497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:24:10.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:10 smithi134 ceph-mon[31966]: pgmap v4497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:24:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7009.623000 secs 2024-07-26T13:24:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:12 smithi134 ceph-mon[31966]: pgmap v4498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:24:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7009.623000 secs 2024-07-26T13:24:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:12 smithi189 ceph-mon[36790]: pgmap v4498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:24:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7009.623000 secs 2024-07-26T13:24:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:12 smithi193 ceph-mon[37473]: pgmap v4498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:24:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 6995.197086 secs 2024-07-26T13:24:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:14 smithi134 ceph-mon[31966]: pgmap v4499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:24:14.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 6995.197086 secs 2024-07-26T13:24:14.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:14 smithi189 ceph-mon[36790]: pgmap v4499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:24:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 6995.197086 secs 2024-07-26T13:24:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:14 smithi193 ceph-mon[37473]: pgmap v4499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:24:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:16 smithi134 ceph-mon[31966]: pgmap v4500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:24:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:16 smithi189 ceph-mon[36790]: pgmap v4500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:24:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:16 smithi193 ceph-mon[37473]: pgmap v4500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:24:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7014.623126 secs 2024-07-26T13:24:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7014.623126 secs 2024-07-26T13:24:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7014.623126 secs 2024-07-26T13:24:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:18 smithi134 ceph-mon[31966]: pgmap v4501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:24:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:18 smithi189 ceph-mon[36790]: pgmap v4501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:24:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:24:19.003+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:24:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:24:19.003+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:24:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:24:19.003+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:19.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:18 smithi193 ceph-mon[37473]: pgmap v4501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:24:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7000.197226 secs 2024-07-26T13:24:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7000.197226 secs 2024-07-26T13:24:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7000.197226 secs 2024-07-26T13:24:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:20 smithi134 ceph-mon[31966]: pgmap v4502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:24:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:20 smithi189 ceph-mon[36790]: pgmap v4502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:24:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:20 smithi193 ceph-mon[37473]: pgmap v4502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:24:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7019.623253 secs 2024-07-26T13:24:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:22 smithi134 ceph-mon[31966]: pgmap v4503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7019.623253 secs 2024-07-26T13:24:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:22 smithi189 ceph-mon[36790]: pgmap v4503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7019.623253 secs 2024-07-26T13:24:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:22 smithi193 ceph-mon[37473]: pgmap v4503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:23.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:24:23.545+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:24:23.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:24:23.545+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:24:23.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:24:23.545+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T13:24:23.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:24:23.545+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T13:24:23.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:24:23.563+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:24:23.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:24:23.563+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:24:23.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:24:23.563+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7005.197360 secs 2024-07-26T13:24:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:24 smithi134 ceph-mon[31966]: pgmap v4504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7005.197360 secs 2024-07-26T13:24:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:24 smithi189 ceph-mon[36790]: pgmap v4504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7005.197360 secs 2024-07-26T13:24:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:24 smithi193 ceph-mon[37473]: pgmap v4504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:26 smithi134 ceph-mon[31966]: pgmap v4505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:26 smithi189 ceph-mon[36790]: pgmap v4505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:26 smithi193 ceph-mon[37473]: pgmap v4505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7024.623329 secs 2024-07-26T13:24:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7024.623329 secs 2024-07-26T13:24:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7024.623329 secs 2024-07-26T13:24:28.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:28 smithi134 ceph-mon[31966]: pgmap v4506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:28 smithi189 ceph-mon[36790]: pgmap v4506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:28 smithi193 ceph-mon[37473]: pgmap v4506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7010.197501 secs 2024-07-26T13:24:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7010.197501 secs 2024-07-26T13:24:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7010.197501 secs 2024-07-26T13:24:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:30 smithi134 ceph-mon[31966]: pgmap v4507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:24:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:30 smithi189 ceph-mon[36790]: pgmap v4507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:24:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:30 smithi193 ceph-mon[37473]: pgmap v4507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:24:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7029.623441 secs 2024-07-26T13:24:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:32 smithi134 ceph-mon[31966]: pgmap v4508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7029.623441 secs 2024-07-26T13:24:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:32 smithi189 ceph-mon[36790]: pgmap v4508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7029.623441 secs 2024-07-26T13:24:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:32 smithi193 ceph-mon[37473]: pgmap v4508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:24:34.014+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:24:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:24:34.014+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:24:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:24:34.014+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:24:34.067+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:24:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:24:34.067+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:24:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:24:34.067+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:24:34.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:24:34.068+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:24:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7015.197621 secs 2024-07-26T13:24:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:34 smithi134 ceph-mon[31966]: pgmap v4509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7015.197621 secs 2024-07-26T13:24:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:34 smithi189 ceph-mon[36790]: pgmap v4509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7015.197621 secs 2024-07-26T13:24:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:34 smithi193 ceph-mon[37473]: pgmap v4509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:36 smithi134 ceph-mon[31966]: pgmap v4510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:36 smithi189 ceph-mon[36790]: pgmap v4510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:36 smithi193 ceph-mon[37473]: pgmap v4510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:37.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7034.623551 secs 2024-07-26T13:24:37.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7034.623551 secs 2024-07-26T13:24:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7034.623551 secs 2024-07-26T13:24:38.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:24:38.567+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:24:38.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:24:38.567+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:24:38.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:24:38.567+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:38.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:38 smithi134 ceph-mon[31966]: pgmap v4511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:38 smithi189 ceph-mon[36790]: pgmap v4511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:38 smithi193 ceph-mon[37473]: pgmap v4511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7020.197745 secs 2024-07-26T13:24:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7020.197745 secs 2024-07-26T13:24:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7020.197745 secs 2024-07-26T13:24:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:40 smithi189 ceph-mon[36790]: pgmap v4512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:24:41.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:40 smithi193 ceph-mon[37473]: pgmap v4512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:24:41.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:40 smithi134 ceph-mon[31966]: pgmap v4512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:24:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7039.623674 secs 2024-07-26T13:24:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:42 smithi189 ceph-mon[36790]: pgmap v4513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7039.623674 secs 2024-07-26T13:24:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:42 smithi193 ceph-mon[37473]: pgmap v4513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:43.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7039.623674 secs 2024-07-26T13:24:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:42 smithi134 ceph-mon[31966]: pgmap v4513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7025.197868 secs 2024-07-26T13:24:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:44 smithi193 ceph-mon[37473]: pgmap v4514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7025.197868 secs 2024-07-26T13:24:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:44 smithi134 ceph-mon[31966]: pgmap v4514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7025.197868 secs 2024-07-26T13:24:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:44 smithi189 ceph-mon[36790]: pgmap v4514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:46 smithi193 ceph-mon[37473]: pgmap v4515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:47.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:46 smithi134 ceph-mon[31966]: pgmap v4515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:46 smithi189 ceph-mon[36790]: pgmap v4515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7044.623805 secs 2024-07-26T13:24:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7044.623805 secs 2024-07-26T13:24:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7044.623805 secs 2024-07-26T13:24:49.022 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:24:49.019+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:24:49.022 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:24:49.019+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:24:49.022 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:24:49.019+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:48 smithi193 ceph-mon[37473]: pgmap v4516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:48 smithi134 ceph-mon[31966]: pgmap v4516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:48 smithi189 ceph-mon[36790]: pgmap v4516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7030.197994 secs 2024-07-26T13:24:50.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7030.197994 secs 2024-07-26T13:24:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7030.197994 secs 2024-07-26T13:24:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:50 smithi193 ceph-mon[37473]: pgmap v4517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:24:51.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:50 smithi134 ceph-mon[31966]: pgmap v4517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:24:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:50 smithi189 ceph-mon[36790]: pgmap v4517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:24:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7049.623924 secs 2024-07-26T13:24:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:52 smithi193 ceph-mon[37473]: pgmap v4518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:53.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7049.623924 secs 2024-07-26T13:24:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:52 smithi134 ceph-mon[31966]: pgmap v4518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7049.623924 secs 2024-07-26T13:24:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:52 smithi189 ceph-mon[36790]: pgmap v4518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:53.582 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:24:53.581+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:24:53.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:24:53.581+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:24:53.583 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:24:53.581+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:24:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7035.198094 secs 2024-07-26T13:24:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:54 smithi193 ceph-mon[37473]: pgmap v4519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7035.198094 secs 2024-07-26T13:24:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:54 smithi134 ceph-mon[31966]: pgmap v4519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7035.198094 secs 2024-07-26T13:24:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:54 smithi189 ceph-mon[36790]: pgmap v4519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:56 smithi193 ceph-mon[37473]: pgmap v4520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:56 smithi134 ceph-mon[31966]: pgmap v4520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:56 smithi189 ceph-mon[36790]: pgmap v4520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:24:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7054.624077 secs 2024-07-26T13:24:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7054.624077 secs 2024-07-26T13:24:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7054.624077 secs 2024-07-26T13:24:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:58 smithi193 ceph-mon[37473]: pgmap v4521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:58 smithi134 ceph-mon[31966]: pgmap v4521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:24:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:58 smithi189 ceph-mon[36790]: pgmap v4521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:24:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7040.198175 secs 2024-07-26T13:25:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:24:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7040.198175 secs 2024-07-26T13:25:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:24:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7040.198175 secs 2024-07-26T13:25:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:00 smithi134 ceph-mon[31966]: pgmap v4522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:25:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:00 smithi189 ceph-mon[36790]: pgmap v4522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:25:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:00 smithi193 ceph-mon[37473]: pgmap v4522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:25:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7059.624221 secs 2024-07-26T13:25:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:02 smithi134 ceph-mon[31966]: pgmap v4523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7059.624221 secs 2024-07-26T13:25:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:02 smithi189 ceph-mon[36790]: pgmap v4523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7059.624221 secs 2024-07-26T13:25:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:02 smithi193 ceph-mon[37473]: pgmap v4523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:25:04.035+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:25:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:25:04.035+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:25:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:25:04.035+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7045.198276 secs 2024-07-26T13:25:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:04 smithi134 ceph-mon[31966]: pgmap v4524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7045.198276 secs 2024-07-26T13:25:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:04 smithi189 ceph-mon[36790]: pgmap v4524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7045.198276 secs 2024-07-26T13:25:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:04 smithi193 ceph-mon[37473]: pgmap v4524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:07.075 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:06 smithi193 ceph-mon[37473]: pgmap v4525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:06 smithi134 ceph-mon[31966]: pgmap v4525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:06 smithi189 ceph-mon[36790]: pgmap v4525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:08.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7064.624365 secs 2024-07-26T13:25:08.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:25:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7064.624365 secs 2024-07-26T13:25:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:25:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7064.624365 secs 2024-07-26T13:25:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:25:08.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:25:08.585+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:25:08.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:25:08.585+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:08.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:25:08.585+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:25:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:08 smithi134 ceph-mon[31966]: pgmap v4526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:25:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:25:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:25:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:08 smithi189 ceph-mon[36790]: pgmap v4526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:25:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:25:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:25:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:08 smithi193 ceph-mon[37473]: pgmap v4526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:25:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:25:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:25:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7050.198350 secs 2024-07-26T13:25:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:09 smithi134 ceph-mon[31966]: pgmap v4527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:25:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:09 smithi134 ceph-mon[31966]: pgmap v4528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7050.198350 secs 2024-07-26T13:25:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:09 smithi189 ceph-mon[36790]: pgmap v4527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:25:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:09 smithi189 ceph-mon[36790]: pgmap v4528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7050.198350 secs 2024-07-26T13:25:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:09 smithi193 ceph-mon[37473]: pgmap v4527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:25:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:09 smithi193 ceph-mon[37473]: pgmap v4528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:11 smithi134 ceph-mon[31966]: pgmap v4529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:11 smithi189 ceph-mon[36790]: pgmap v4529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:11 smithi193 ceph-mon[37473]: pgmap v4529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7069.624494 secs 2024-07-26T13:25:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7069.624494 secs 2024-07-26T13:25:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7069.624494 secs 2024-07-26T13:25:14.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:13 smithi134 ceph-mon[31966]: pgmap v4530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:13 smithi189 ceph-mon[36790]: pgmap v4530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:13 smithi193 ceph-mon[37473]: pgmap v4530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:25:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7055.198431 secs 2024-07-26T13:25:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7055.198431 secs 2024-07-26T13:25:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7055.198431 secs 2024-07-26T13:25:16.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:15 smithi134 ceph-mon[31966]: pgmap v4531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:25:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:15 smithi189 ceph-mon[36790]: pgmap v4531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:25:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:15 smithi193 ceph-mon[37473]: pgmap v4531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:25:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7074.624629 secs 2024-07-26T13:25:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:17 smithi134 ceph-mon[31966]: pgmap v4532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:25:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7074.624629 secs 2024-07-26T13:25:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:17 smithi189 ceph-mon[36790]: pgmap v4532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:25:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7074.624629 secs 2024-07-26T13:25:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:17 smithi193 ceph-mon[37473]: pgmap v4532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:25:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:25:19.050+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:25:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:25:19.050+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:25:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:25:19.050+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7060.198553 secs 2024-07-26T13:25:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:19 smithi134 ceph-mon[31966]: pgmap v4533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:25:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7060.198553 secs 2024-07-26T13:25:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:19 smithi189 ceph-mon[36790]: pgmap v4533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:25:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7060.198553 secs 2024-07-26T13:25:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:19 smithi193 ceph-mon[37473]: pgmap v4533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:25:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:21 smithi134 ceph-mon[31966]: pgmap v4534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:21 smithi189 ceph-mon[36790]: pgmap v4534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:21 smithi193 ceph-mon[37473]: pgmap v4534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7079.624753 secs 2024-07-26T13:25:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7079.624753 secs 2024-07-26T13:25:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7079.624753 secs 2024-07-26T13:25:23.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:25:23.601+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:25:23.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:25:23.601+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:23.603 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:25:23.601+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:25:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:23 smithi134 ceph-mon[31966]: pgmap v4535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:23 smithi189 ceph-mon[36790]: pgmap v4535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:23 smithi193 ceph-mon[37473]: pgmap v4535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7065.198674 secs 2024-07-26T13:25:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7065.198674 secs 2024-07-26T13:25:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7065.198674 secs 2024-07-26T13:25:26.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:25 smithi134 ceph-mon[31966]: pgmap v4536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:25:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:25 smithi189 ceph-mon[36790]: pgmap v4536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:25:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:25 smithi193 ceph-mon[37473]: pgmap v4536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:25:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7084.624875 secs 2024-07-26T13:25:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:27 smithi134 ceph-mon[31966]: pgmap v4537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7084.624875 secs 2024-07-26T13:25:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:27 smithi189 ceph-mon[36790]: pgmap v4537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7084.624875 secs 2024-07-26T13:25:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:27 smithi193 ceph-mon[37473]: pgmap v4537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7070.198796 secs 2024-07-26T13:25:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:29 smithi134 ceph-mon[31966]: pgmap v4538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7070.198796 secs 2024-07-26T13:25:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:29 smithi189 ceph-mon[36790]: pgmap v4538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7070.198796 secs 2024-07-26T13:25:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:29 smithi193 ceph-mon[37473]: pgmap v4538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:31 smithi134 ceph-mon[31966]: pgmap v4539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:31 smithi189 ceph-mon[36790]: pgmap v4539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:31 smithi193 ceph-mon[37473]: pgmap v4539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7089.624996 secs 2024-07-26T13:25:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7089.624996 secs 2024-07-26T13:25:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7089.624996 secs 2024-07-26T13:25:34.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:25:34.051+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:25:34.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:25:34.052+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:25:34.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:25:34.052+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:33 smithi189 ceph-mon[36790]: pgmap v4540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:33 smithi193 ceph-mon[37473]: pgmap v4540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:34.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:33 smithi134 ceph-mon[31966]: pgmap v4540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7075.198927 secs 2024-07-26T13:25:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7075.198927 secs 2024-07-26T13:25:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7075.198927 secs 2024-07-26T13:25:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:35 smithi189 ceph-mon[36790]: pgmap v4541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:25:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:35 smithi193 ceph-mon[37473]: pgmap v4541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:25:36.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:35 smithi134 ceph-mon[31966]: pgmap v4541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:25:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7094.625130 secs 2024-07-26T13:25:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7094.625130 secs 2024-07-26T13:25:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7094.625130 secs 2024-07-26T13:25:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:37 smithi193 ceph-mon[37473]: pgmap v4542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:38.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:37 smithi134 ceph-mon[31966]: pgmap v4542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:37 smithi189 ceph-mon[36790]: pgmap v4542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:38.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:25:38.616+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:25:38.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:25:38.616+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:38.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:25:38.616+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:25:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7080.199052 secs 2024-07-26T13:25:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:39 smithi193 ceph-mon[37473]: pgmap v4543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7080.199052 secs 2024-07-26T13:25:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:39 smithi134 ceph-mon[31966]: pgmap v4543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7080.199052 secs 2024-07-26T13:25:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:39 smithi189 ceph-mon[36790]: pgmap v4543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:41 smithi193 ceph-mon[37473]: pgmap v4544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7099.625224 secs 2024-07-26T13:25:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:41 smithi134 ceph-mon[31966]: pgmap v4544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7099.625224 secs 2024-07-26T13:25:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:41 smithi189 ceph-mon[36790]: pgmap v4544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7099.625224 secs 2024-07-26T13:25:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:43 smithi193 ceph-mon[37473]: pgmap v4545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:43 smithi134 ceph-mon[31966]: pgmap v4545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:43 smithi189 ceph-mon[36790]: pgmap v4545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7085.199166 secs 2024-07-26T13:25:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7085.199166 secs 2024-07-26T13:25:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7085.199166 secs 2024-07-26T13:25:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:45 smithi193 ceph-mon[37473]: pgmap v4546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:25:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:45 smithi134 ceph-mon[31966]: pgmap v4546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:25:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:45 smithi189 ceph-mon[36790]: pgmap v4546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:25:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7104.625326 secs 2024-07-26T13:25:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7104.625326 secs 2024-07-26T13:25:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7104.625326 secs 2024-07-26T13:25:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:48 smithi193 ceph-mon[37473]: pgmap v4547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:48 smithi134 ceph-mon[31966]: pgmap v4547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:48 smithi189 ceph-mon[36790]: pgmap v4547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:25:49.067+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:25:49.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:25:49.067+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:25:49.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:25:49.067+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7090.199257 secs 2024-07-26T13:25:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:50 smithi193 ceph-mon[37473]: pgmap v4548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7090.199257 secs 2024-07-26T13:25:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:50 smithi134 ceph-mon[31966]: pgmap v4548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7090.199257 secs 2024-07-26T13:25:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:50 smithi189 ceph-mon[36790]: pgmap v4548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:52 smithi193 ceph-mon[37473]: pgmap v4549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7109.625465 secs 2024-07-26T13:25:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:52 smithi134 ceph-mon[31966]: pgmap v4549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7109.625465 secs 2024-07-26T13:25:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:52 smithi189 ceph-mon[36790]: pgmap v4549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7109.625465 secs 2024-07-26T13:25:53.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:25:53.632+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:25:53.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:25:53.632+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:25:53.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:25:53.632+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:25:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:54 smithi193 ceph-mon[37473]: pgmap v4550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:54 smithi134 ceph-mon[31966]: pgmap v4550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:54 smithi189 ceph-mon[36790]: pgmap v4550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:25:55.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7095.199383 secs 2024-07-26T13:25:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7095.199383 secs 2024-07-26T13:25:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7095.199383 secs 2024-07-26T13:25:56.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:56 smithi134 ceph-mon[31966]: pgmap v4551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:25:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:56 smithi189 ceph-mon[36790]: pgmap v4551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:25:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:56 smithi193 ceph-mon[37473]: pgmap v4551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:25:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7114.625584 secs 2024-07-26T13:25:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7114.625584 secs 2024-07-26T13:25:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7114.625584 secs 2024-07-26T13:25:58.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:25:58 smithi134 ceph-mon[31966]: pgmap v4552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:25:58 smithi189 ceph-mon[36790]: pgmap v4552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:25:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:25:58 smithi193 ceph-mon[37473]: pgmap v4552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7100.199482 secs 2024-07-26T13:26:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:00 smithi134 ceph-mon[31966]: pgmap v4553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7100.199482 secs 2024-07-26T13:26:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:00 smithi189 ceph-mon[36790]: pgmap v4553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7100.199482 secs 2024-07-26T13:26:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:00 smithi193 ceph-mon[37473]: pgmap v4553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:02 smithi134 ceph-mon[31966]: pgmap v4554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7119.625671 secs 2024-07-26T13:26:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:02 smithi189 ceph-mon[36790]: pgmap v4554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7119.625671 secs 2024-07-26T13:26:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:02 smithi193 ceph-mon[37473]: pgmap v4554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7119.625671 secs 2024-07-26T13:26:04.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:26:04.083+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:26:04.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:26:04.083+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:26:04.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:26:04.083+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:04 smithi134 ceph-mon[31966]: pgmap v4555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:04 smithi189 ceph-mon[36790]: pgmap v4555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:04 smithi193 ceph-mon[37473]: pgmap v4555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:05.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7105.199604 secs 2024-07-26T13:26:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7105.199604 secs 2024-07-26T13:26:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7105.199604 secs 2024-07-26T13:26:06.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:06 smithi134 ceph-mon[31966]: pgmap v4556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:26:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:06 smithi189 ceph-mon[36790]: pgmap v4556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:26:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:06 smithi193 ceph-mon[37473]: pgmap v4556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:26:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7124.625813 secs 2024-07-26T13:26:07.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7124.625813 secs 2024-07-26T13:26:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7124.625813 secs 2024-07-26T13:26:08.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:08 smithi134 ceph-mon[31966]: pgmap v4557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:08 smithi189 ceph-mon[36790]: pgmap v4557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:08 smithi193 ceph-mon[37473]: pgmap v4557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:08.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:26:08.647+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:26:08.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:26:08.647+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:08.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:26:08.647+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:26:09.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:26:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:26:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:26:10.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7110.199720 secs 2024-07-26T13:26:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:10 smithi134 ceph-mon[31966]: pgmap v4558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:26:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:26:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7110.199720 secs 2024-07-26T13:26:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:10 smithi189 ceph-mon[36790]: pgmap v4558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:26:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:26:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7110.199720 secs 2024-07-26T13:26:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:10 smithi193 ceph-mon[37473]: pgmap v4558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:26:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:26:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:26:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:26:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:26:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:26:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:26:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:26:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:26:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:26:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:26:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:12 smithi134 ceph-mon[31966]: pgmap v4559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:26:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:12 smithi134 ceph-mon[31966]: pgmap v4560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:26:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7129.625932 secs 2024-07-26T13:26:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:12 smithi189 ceph-mon[36790]: pgmap v4559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:26:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:12 smithi189 ceph-mon[36790]: pgmap v4560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:26:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7129.625932 secs 2024-07-26T13:26:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:12 smithi193 ceph-mon[37473]: pgmap v4559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T13:26:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:12 smithi193 ceph-mon[37473]: pgmap v4560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:26:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7129.625932 secs 2024-07-26T13:26:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:14 smithi134 ceph-mon[31966]: pgmap v4561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:26:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:14 smithi189 ceph-mon[36790]: pgmap v4561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:26:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:14 smithi193 ceph-mon[37473]: pgmap v4561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:26:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7115.199821 secs 2024-07-26T13:26:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7115.199821 secs 2024-07-26T13:26:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7115.199821 secs 2024-07-26T13:26:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:16 smithi134 ceph-mon[31966]: pgmap v4562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:26:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:16 smithi189 ceph-mon[36790]: pgmap v4562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:26:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:16 smithi193 ceph-mon[37473]: pgmap v4562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:26:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7134.626048 secs 2024-07-26T13:26:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7134.626048 secs 2024-07-26T13:26:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7134.626048 secs 2024-07-26T13:26:18.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:18 smithi134 ceph-mon[31966]: pgmap v4563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:26:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:18 smithi189 ceph-mon[36790]: pgmap v4563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:26:18.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:18 smithi193 ceph-mon[37473]: pgmap v4563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:26:19.096 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:26:19.094+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:26:19.096 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:26:19.094+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:26:19.096 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:26:19.094+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7120.199981 secs 2024-07-26T13:26:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:20 smithi134 ceph-mon[31966]: pgmap v4564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:26:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7120.199981 secs 2024-07-26T13:26:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:20 smithi189 ceph-mon[36790]: pgmap v4564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:26:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7120.199981 secs 2024-07-26T13:26:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:20 smithi193 ceph-mon[37473]: pgmap v4564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:26:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:22 smithi134 ceph-mon[31966]: pgmap v4565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:26:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7139.626168 secs 2024-07-26T13:26:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:22 smithi189 ceph-mon[36790]: pgmap v4565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:26:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7139.626168 secs 2024-07-26T13:26:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:22 smithi193 ceph-mon[37473]: pgmap v4565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:26:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7139.626168 secs 2024-07-26T13:26:23.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:26:23.663+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:26:23.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:26:23.663+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:23.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:26:23.663+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:26:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:24 smithi134 ceph-mon[31966]: pgmap v4566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:24 smithi189 ceph-mon[36790]: pgmap v4566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:24 smithi193 ceph-mon[37473]: pgmap v4566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7125.200078 secs 2024-07-26T13:26:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7125.200078 secs 2024-07-26T13:26:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7125.200078 secs 2024-07-26T13:26:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:26 smithi189 ceph-mon[36790]: pgmap v4567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:26:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:26 smithi193 ceph-mon[37473]: pgmap v4567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:26:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:26 smithi134 ceph-mon[31966]: pgmap v4567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:26:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7144.626262 secs 2024-07-26T13:26:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7144.626262 secs 2024-07-26T13:26:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7144.626262 secs 2024-07-26T13:26:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:28 smithi193 ceph-mon[37473]: pgmap v4568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:28.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:28 smithi134 ceph-mon[31966]: pgmap v4568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:28 smithi189 ceph-mon[36790]: pgmap v4568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7130.200194 secs 2024-07-26T13:26:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:30 smithi193 ceph-mon[37473]: pgmap v4569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7130.200194 secs 2024-07-26T13:26:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:30 smithi134 ceph-mon[31966]: pgmap v4569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7130.200194 secs 2024-07-26T13:26:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:30 smithi189 ceph-mon[36790]: pgmap v4569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:32 smithi193 ceph-mon[37473]: pgmap v4570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7149.626382 secs 2024-07-26T13:26:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:32 smithi134 ceph-mon[31966]: pgmap v4570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7149.626382 secs 2024-07-26T13:26:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:32 smithi189 ceph-mon[36790]: pgmap v4570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7149.626382 secs 2024-07-26T13:26:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:26:34.110+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:26:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:26:34.110+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:26:34.111 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:26:34.110+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:34.495 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:34 smithi189 ceph-mon[36790]: pgmap v4571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:34 smithi193 ceph-mon[37473]: pgmap v4571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:34 smithi134 ceph-mon[31966]: pgmap v4571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7135.200251 secs 2024-07-26T13:26:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7135.200251 secs 2024-07-26T13:26:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7135.200251 secs 2024-07-26T13:26:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:36 smithi193 ceph-mon[37473]: pgmap v4572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:26:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:36 smithi134 ceph-mon[31966]: pgmap v4572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:26:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:36 smithi189 ceph-mon[36790]: pgmap v4572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:26:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7154.626506 secs 2024-07-26T13:26:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7154.626506 secs 2024-07-26T13:26:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7154.626506 secs 2024-07-26T13:26:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:38 smithi193 ceph-mon[37473]: pgmap v4573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:38.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:38 smithi134 ceph-mon[31966]: pgmap v4573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:38.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:26:38.677+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:26:38.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:26:38.677+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:26:38.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:26:38.677+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:38 smithi189 ceph-mon[36790]: pgmap v4573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7140.200368 secs 2024-07-26T13:26:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:40 smithi193 ceph-mon[37473]: pgmap v4574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7140.200368 secs 2024-07-26T13:26:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:40 smithi134 ceph-mon[31966]: pgmap v4574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7140.200368 secs 2024-07-26T13:26:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:40 smithi189 ceph-mon[36790]: pgmap v4574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:42.527 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:42 smithi193 ceph-mon[37473]: pgmap v4575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:42.527 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7159.626625 secs 2024-07-26T13:26:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:42 smithi134 ceph-mon[31966]: pgmap v4575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7159.626625 secs 2024-07-26T13:26:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:42 smithi189 ceph-mon[36790]: pgmap v4575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7159.626625 secs 2024-07-26T13:26:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:44 smithi134 ceph-mon[31966]: pgmap v4576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:44 smithi193 ceph-mon[37473]: pgmap v4576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:44 smithi189 ceph-mon[36790]: pgmap v4576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7145.200497 secs 2024-07-26T13:26:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7145.200497 secs 2024-07-26T13:26:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7145.200497 secs 2024-07-26T13:26:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:46 smithi193 ceph-mon[37473]: pgmap v4577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:26:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:46 smithi134 ceph-mon[31966]: pgmap v4577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:26:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:46 smithi189 ceph-mon[36790]: pgmap v4577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:26:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7164.626779 secs 2024-07-26T13:26:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7164.626779 secs 2024-07-26T13:26:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7164.626779 secs 2024-07-26T13:26:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:48 smithi134 ceph-mon[31966]: pgmap v4578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:48 smithi189 ceph-mon[36790]: pgmap v4578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:48 smithi193 ceph-mon[37473]: pgmap v4578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:49.126 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:26:49.125+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:26:49.126 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:26:49.125+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:26:49.127 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:26:49.125+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7150.200590 secs 2024-07-26T13:26:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:50 smithi134 ceph-mon[31966]: pgmap v4579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7150.200590 secs 2024-07-26T13:26:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:50 smithi189 ceph-mon[36790]: pgmap v4579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7150.200590 secs 2024-07-26T13:26:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:50 smithi193 ceph-mon[37473]: pgmap v4579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:52 smithi134 ceph-mon[31966]: pgmap v4580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7169.626917 secs 2024-07-26T13:26:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:52 smithi189 ceph-mon[36790]: pgmap v4580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7169.626917 secs 2024-07-26T13:26:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:52 smithi193 ceph-mon[37473]: pgmap v4580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7169.626917 secs 2024-07-26T13:26:53.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:26:53.692+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:26:53.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:26:53.692+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:26:53.694 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:26:53.692+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:26:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:54 smithi134 ceph-mon[31966]: pgmap v4581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:54 smithi189 ceph-mon[36790]: pgmap v4581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:54 smithi193 ceph-mon[37473]: pgmap v4581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:26:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7155.200732 secs 2024-07-26T13:26:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7155.200732 secs 2024-07-26T13:26:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7155.200732 secs 2024-07-26T13:26:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:56 smithi134 ceph-mon[31966]: pgmap v4582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:26:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:56 smithi189 ceph-mon[36790]: pgmap v4582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:26:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:56 smithi193 ceph-mon[37473]: pgmap v4582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:26:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7174.627056 secs 2024-07-26T13:26:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7174.627056 secs 2024-07-26T13:26:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7174.627056 secs 2024-07-26T13:26:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:26:58 smithi134 ceph-mon[31966]: pgmap v4583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:26:58 smithi189 ceph-mon[36790]: pgmap v4583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:26:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:26:58 smithi193 ceph-mon[37473]: pgmap v4583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7160.200877 secs 2024-07-26T13:27:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:00 smithi134 ceph-mon[31966]: pgmap v4584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7160.200877 secs 2024-07-26T13:27:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:00 smithi189 ceph-mon[36790]: pgmap v4584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7160.200877 secs 2024-07-26T13:27:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:00 smithi193 ceph-mon[37473]: pgmap v4584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:02 smithi134 ceph-mon[31966]: pgmap v4585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7179.627181 secs 2024-07-26T13:27:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:02 smithi189 ceph-mon[36790]: pgmap v4585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7179.627181 secs 2024-07-26T13:27:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:02 smithi193 ceph-mon[37473]: pgmap v4585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7179.627181 secs 2024-07-26T13:27:04.131 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:04.130+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:27:04.132 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:04.130+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:04.132 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:04.130+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:27:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:04 smithi134 ceph-mon[31966]: pgmap v4586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:04 smithi189 ceph-mon[36790]: pgmap v4586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:04 smithi193 ceph-mon[37473]: pgmap v4586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7165.201025 secs 2024-07-26T13:27:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7165.201025 secs 2024-07-26T13:27:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7165.201025 secs 2024-07-26T13:27:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:06 smithi134 ceph-mon[31966]: pgmap v4587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:27:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:06 smithi189 ceph-mon[36790]: pgmap v4587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:27:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:06 smithi193 ceph-mon[37473]: pgmap v4587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:27:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7184.627244 secs 2024-07-26T13:27:07.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7184.627244 secs 2024-07-26T13:27:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7184.627244 secs 2024-07-26T13:27:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:08 smithi134 ceph-mon[31966]: pgmap v4588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:27:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:08 smithi189 ceph-mon[36790]: pgmap v4588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:27:08.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:08.708+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:27:08.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:08.708+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:08.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:08.708+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:27:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:08 smithi193 ceph-mon[37473]: pgmap v4588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:27:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7170.201159 secs 2024-07-26T13:27:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:10 smithi134 ceph-mon[31966]: pgmap v4589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7170.201159 secs 2024-07-26T13:27:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:10 smithi189 ceph-mon[36790]: pgmap v4589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7170.201159 secs 2024-07-26T13:27:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:10 smithi193 ceph-mon[37473]: pgmap v4589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:27:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:27:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:11.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:11.748 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:27:11.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:11.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:11.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:11.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:12 smithi134 ceph-mon[31966]: pgmap v4590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:27:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7189.627351 secs 2024-07-26T13:27:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:12 smithi189 ceph-mon[36790]: pgmap v4590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:27:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7189.627351 secs 2024-07-26T13:27:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:12 smithi193 ceph-mon[37473]: pgmap v4590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:27:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7189.627351 secs 2024-07-26T13:27:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:13 smithi134 ceph-mon[31966]: pgmap v4591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:13 smithi189 ceph-mon[36790]: pgmap v4591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:13 smithi193 ceph-mon[37473]: pgmap v4591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7175.201222 secs 2024-07-26T13:27:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7175.201222 secs 2024-07-26T13:27:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7175.201222 secs 2024-07-26T13:27:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:15 smithi134 ceph-mon[31966]: pgmap v4592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:15 smithi189 ceph-mon[36790]: pgmap v4592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:15 smithi193 ceph-mon[37473]: pgmap v4592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:27:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:27:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:16 smithi134 ceph-mon[31966]: pgmap v4593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:27:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:16 smithi134 ceph-mon[31966]: pgmap v4594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:27:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:27:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:16 smithi189 ceph-mon[36790]: pgmap v4593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:27:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:16 smithi189 ceph-mon[36790]: pgmap v4594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:27:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:27:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:16 smithi193 ceph-mon[37473]: pgmap v4593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:27:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:16 smithi193 ceph-mon[37473]: pgmap v4594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:27:17.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7194.627491 secs 2024-07-26T13:27:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7194.627491 secs 2024-07-26T13:27:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7194.627491 secs 2024-07-26T13:27:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:18 smithi134 ceph-mon[31966]: pgmap v4595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:18 smithi189 ceph-mon[36790]: pgmap v4595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:18 smithi193 ceph-mon[37473]: pgmap v4595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:19.139 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:19.138+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:27:19.139 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:19.138+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:19.139 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:19.138+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:27:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7180.201351 secs 2024-07-26T13:27:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7180.201351 secs 2024-07-26T13:27:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7180.201351 secs 2024-07-26T13:27:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:20 smithi189 ceph-mon[36790]: pgmap v4596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:27:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:20 smithi193 ceph-mon[37473]: pgmap v4596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:27:21.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:20 smithi134 ceph-mon[31966]: pgmap v4596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:27:22.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:22.310+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T13:27:22.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:22.310+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:27:22.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:22.310+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b3804e040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:27:22.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:22.310+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b3804e040 comp rx=0 tx=0).stop 2024-07-26T13:27:22.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:22.310+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T13:27:22.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:22.310+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:27:22.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:22.310+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:27:22.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:22.310+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 0x7f9b300c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:27:22.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:22.311+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 0x7f9b300c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:27:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:22.313+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 0x7f9b300c1940 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009770 tx=0x7f9b38015000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:27:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7199.627616 secs 2024-07-26T13:27:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:22 smithi193 ceph-mon[37473]: pgmap v4597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:27:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7199.627616 secs 2024-07-26T13:27:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:22 smithi134 ceph-mon[31966]: pgmap v4597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:27:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7199.627616 secs 2024-07-26T13:27:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:22 smithi189 ceph-mon[36790]: pgmap v4597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:27:23.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:23.576+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:27:23.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:23.576+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:27:23.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:23.576+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T13:27:23.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:23.576+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T13:27:23.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:23.710+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:27:23.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:23.710+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:23.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:23.710+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:27:25.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7185.201441 secs 2024-07-26T13:27:25.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:24 smithi193 ceph-mon[37473]: pgmap v4598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7185.201441 secs 2024-07-26T13:27:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:24 smithi134 ceph-mon[31966]: pgmap v4598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7185.201441 secs 2024-07-26T13:27:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:24 smithi189 ceph-mon[36790]: pgmap v4598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:27:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:26 smithi193 ceph-mon[37473]: pgmap v4599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:27:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:26 smithi134 ceph-mon[31966]: pgmap v4599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:27:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:26 smithi189 ceph-mon[36790]: pgmap v4599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:27:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7204.627715 secs 2024-07-26T13:27:28.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7204.627715 secs 2024-07-26T13:27:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7204.627715 secs 2024-07-26T13:27:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:28 smithi193 ceph-mon[37473]: pgmap v4600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:29.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:28 smithi134 ceph-mon[31966]: pgmap v4600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:28 smithi189 ceph-mon[36790]: pgmap v4600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7190.201526 secs 2024-07-26T13:27:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7190.201526 secs 2024-07-26T13:27:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7190.201526 secs 2024-07-26T13:27:31.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:30 smithi134 ceph-mon[31966]: pgmap v4601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:30 smithi189 ceph-mon[36790]: pgmap v4601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:30 smithi193 ceph-mon[37473]: pgmap v4601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:32.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:32.862+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c12c0 msgr2=0x7f21ac0c16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000047) for more than 900000000 us, fault. 2024-07-26T13:27:32.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:32.862+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c12c0 0x7f21ac0c16d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f2198005820 tx=0x7f21b80fc000 comp rx=0 tx=0).stop 2024-07-26T13:27:32.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:32.862+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c12c0 msgr2=0x7f21ac0c16d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:27:32.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:32.863+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c12c0 0x7f21ac0c16d0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:27:32.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:32.863+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 0x7f21ac0c0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:27:32.865 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:32.863+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 0x7f21ac0c0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:27:32.866 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:32.865+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 0x7f21ac0c0640 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2198005e40 tx=0x7f21b805c000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:27:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7209.627841 secs 2024-07-26T13:27:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:32 smithi134 ceph-mon[31966]: pgmap v4602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7209.627841 secs 2024-07-26T13:27:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:32 smithi189 ceph-mon[36790]: pgmap v4602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7209.627841 secs 2024-07-26T13:27:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:32 smithi193 ceph-mon[37473]: pgmap v4602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:34.099+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:27:34.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:34.099+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:27:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:34.100+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:27:34.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:34.100+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:27:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:34.153+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:27:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:34.153+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:34.153+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:27:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7195.201647 secs 2024-07-26T13:27:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:34 smithi134 ceph-mon[31966]: pgmap v4603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7195.201647 secs 2024-07-26T13:27:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:34 smithi189 ceph-mon[36790]: pgmap v4603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7195.201647 secs 2024-07-26T13:27:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:34 smithi193 ceph-mon[37473]: pgmap v4603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:37.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:36 smithi193 ceph-mon[37473]: pgmap v4604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:27:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:36 smithi134 ceph-mon[31966]: pgmap v4604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:27:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:36 smithi189 ceph-mon[36790]: pgmap v4604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:27:38.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7214.627967 secs 2024-07-26T13:27:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7214.627967 secs 2024-07-26T13:27:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7214.627967 secs 2024-07-26T13:27:38.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:38.723+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:27:38.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:38.723+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:27:38.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:38.723+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:38 smithi134 ceph-mon[31966]: pgmap v4605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:38 smithi189 ceph-mon[36790]: pgmap v4605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:38 smithi193 ceph-mon[37473]: pgmap v4605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7200.201708 secs 2024-07-26T13:27:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7200.201708 secs 2024-07-26T13:27:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7200.201708 secs 2024-07-26T13:27:41.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:40 smithi134 ceph-mon[31966]: pgmap v4606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:27:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:40 smithi189 ceph-mon[36790]: pgmap v4606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:27:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:40 smithi193 ceph-mon[37473]: pgmap v4606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:27:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7219.628104 secs 2024-07-26T13:27:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:42 smithi134 ceph-mon[31966]: pgmap v4607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:27:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7219.628104 secs 2024-07-26T13:27:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:42 smithi189 ceph-mon[36790]: pgmap v4607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:27:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7219.628104 secs 2024-07-26T13:27:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:42 smithi193 ceph-mon[37473]: pgmap v4607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:27:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7205.201834 secs 2024-07-26T13:27:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7205.201834 secs 2024-07-26T13:27:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7205.201834 secs 2024-07-26T13:27:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:44 smithi134 ceph-mon[31966]: pgmap v4608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:44 smithi189 ceph-mon[36790]: pgmap v4608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:44 smithi193 ceph-mon[37473]: pgmap v4608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:27:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:46 smithi134 ceph-mon[31966]: pgmap v4609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:46 smithi189 ceph-mon[36790]: pgmap v4609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:46 smithi193 ceph-mon[37473]: pgmap v4609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7224.628246 secs 2024-07-26T13:27:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7224.628246 secs 2024-07-26T13:27:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7224.628246 secs 2024-07-26T13:27:49.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:49.166+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:27:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:49.166+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:49.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:27:49.166+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:27:49.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:48 smithi134 ceph-mon[31966]: pgmap v4610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:49.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7210.201922 secs 2024-07-26T13:27:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:48 smithi189 ceph-mon[36790]: pgmap v4610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7210.201922 secs 2024-07-26T13:27:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:48 smithi193 ceph-mon[37473]: pgmap v4610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7210.201922 secs 2024-07-26T13:27:51.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:50 smithi134 ceph-mon[31966]: pgmap v4611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:50 smithi189 ceph-mon[36790]: pgmap v4611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:50 smithi193 ceph-mon[37473]: pgmap v4611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:27:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7229.628368 secs 2024-07-26T13:27:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:52 smithi134 ceph-mon[31966]: pgmap v4612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7229.628368 secs 2024-07-26T13:27:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:52 smithi189 ceph-mon[36790]: pgmap v4612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7229.628368 secs 2024-07-26T13:27:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:52 smithi193 ceph-mon[37473]: pgmap v4612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:53.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:53.728+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:27:53.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:53.728+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:27:53.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:27:53.728+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:27:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7215.202054 secs 2024-07-26T13:27:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7215.202054 secs 2024-07-26T13:27:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7215.202054 secs 2024-07-26T13:27:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:54 smithi134 ceph-mon[31966]: pgmap v4613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:54 smithi189 ceph-mon[36790]: pgmap v4613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:54 smithi193 ceph-mon[37473]: pgmap v4613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:56 smithi134 ceph-mon[31966]: pgmap v4614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:56 smithi189 ceph-mon[36790]: pgmap v4614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:56 smithi193 ceph-mon[37473]: pgmap v4614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:27:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7234.628499 secs 2024-07-26T13:27:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7234.628499 secs 2024-07-26T13:27:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7234.628499 secs 2024-07-26T13:27:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:58 smithi134 ceph-mon[31966]: pgmap v4615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:27:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7220.202202 secs 2024-07-26T13:27:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:58 smithi189 ceph-mon[36790]: pgmap v4615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:27:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7220.202202 secs 2024-07-26T13:27:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:58 smithi193 ceph-mon[37473]: pgmap v4615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:27:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:27:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7220.202202 secs 2024-07-26T13:28:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:00 smithi134 ceph-mon[31966]: pgmap v4616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:28:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:00 smithi189 ceph-mon[36790]: pgmap v4616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:28:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:00 smithi193 ceph-mon[37473]: pgmap v4616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:28:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7239.628585 secs 2024-07-26T13:28:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:02 smithi189 ceph-mon[36790]: pgmap v4617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7239.628585 secs 2024-07-26T13:28:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:02 smithi193 ceph-mon[37473]: pgmap v4617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7239.628585 secs 2024-07-26T13:28:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:02 smithi134 ceph-mon[31966]: pgmap v4617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:04.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:28:04.182+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:28:04.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:28:04.182+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:04.183 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:28:04.182+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:28:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7225.202350 secs 2024-07-26T13:28:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7225.202350 secs 2024-07-26T13:28:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7225.202350 secs 2024-07-26T13:28:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:04 smithi189 ceph-mon[36790]: pgmap v4618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:04 smithi193 ceph-mon[37473]: pgmap v4618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:04 smithi134 ceph-mon[31966]: pgmap v4618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:06 smithi193 ceph-mon[37473]: pgmap v4619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:06 smithi134 ceph-mon[31966]: pgmap v4619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:06 smithi189 ceph-mon[36790]: pgmap v4619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7244.628713 secs 2024-07-26T13:28:08.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7244.628713 secs 2024-07-26T13:28:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7244.628713 secs 2024-07-26T13:28:08.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:28:08.743+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:28:08.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:28:08.743+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:08.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:28:08.743+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:28:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:08 smithi134 ceph-mon[31966]: pgmap v4620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7230.202470 secs 2024-07-26T13:28:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:08 smithi193 ceph-mon[37473]: pgmap v4620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7230.202470 secs 2024-07-26T13:28:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:08 smithi189 ceph-mon[36790]: pgmap v4620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7230.202470 secs 2024-07-26T13:28:11.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:10 smithi193 ceph-mon[37473]: pgmap v4621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:28:11.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:10 smithi134 ceph-mon[31966]: pgmap v4621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:28:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:10 smithi189 ceph-mon[36790]: pgmap v4621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:28:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7249.628780 secs 2024-07-26T13:28:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7249.628780 secs 2024-07-26T13:28:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7249.628780 secs 2024-07-26T13:28:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:12 smithi193 ceph-mon[37473]: pgmap v4622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:12 smithi134 ceph-mon[31966]: pgmap v4622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:12 smithi189 ceph-mon[36790]: pgmap v4622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7235.202603 secs 2024-07-26T13:28:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7235.202603 secs 2024-07-26T13:28:14.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7235.202603 secs 2024-07-26T13:28:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:14 smithi193 ceph-mon[37473]: pgmap v4623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:14 smithi134 ceph-mon[31966]: pgmap v4623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:14 smithi189 ceph-mon[36790]: pgmap v4623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:28:16.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:28:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:28:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:17 smithi193 ceph-mon[37473]: pgmap v4624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7254.628848 secs 2024-07-26T13:28:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:28:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:28:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:28:17.289 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:17 smithi134 ceph-mon[31966]: pgmap v4624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:17.289 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7254.628848 secs 2024-07-26T13:28:17.289 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:28:17.289 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:28:17.289 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:28:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:17 smithi189 ceph-mon[36790]: pgmap v4624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7254.628848 secs 2024-07-26T13:28:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:28:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:28:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:28:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:18 smithi193 ceph-mon[37473]: pgmap v4625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:28:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:18 smithi134 ceph-mon[31966]: pgmap v4625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:28:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:18 smithi189 ceph-mon[36790]: pgmap v4625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:28:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:28:19.197+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:28:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:28:19.197+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:28:19.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:28:19.197+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7240.202726 secs 2024-07-26T13:28:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7240.202726 secs 2024-07-26T13:28:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7240.202726 secs 2024-07-26T13:28:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:20 smithi134 ceph-mon[31966]: pgmap v4626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:28:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:20 smithi189 ceph-mon[36790]: pgmap v4626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:28:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:20 smithi193 ceph-mon[37473]: pgmap v4626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:28:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:22 smithi134 ceph-mon[31966]: pgmap v4627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:28:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7259.628969 secs 2024-07-26T13:28:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:22 smithi189 ceph-mon[36790]: pgmap v4627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:28:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7259.628969 secs 2024-07-26T13:28:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:22 smithi193 ceph-mon[37473]: pgmap v4627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:28:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7259.628969 secs 2024-07-26T13:28:23.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:28:23.752+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:28:23.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:28:23.752+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:23.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:28:23.752+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:28:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:24 smithi134 ceph-mon[31966]: pgmap v4628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T13:28:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7245.202852 secs 2024-07-26T13:28:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:24 smithi189 ceph-mon[36790]: pgmap v4628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T13:28:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7245.202852 secs 2024-07-26T13:28:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:24 smithi193 ceph-mon[37473]: pgmap v4628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T13:28:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7245.202852 secs 2024-07-26T13:28:26.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:26 smithi134 ceph-mon[31966]: pgmap v4629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:28:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:26 smithi189 ceph-mon[36790]: pgmap v4629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:28:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:26 smithi193 ceph-mon[37473]: pgmap v4629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:28:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7264.629121 secs 2024-07-26T13:28:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7264.629121 secs 2024-07-26T13:28:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7264.629121 secs 2024-07-26T13:28:28.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:28 smithi134 ceph-mon[31966]: pgmap v4630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:28:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:28 smithi189 ceph-mon[36790]: pgmap v4630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:28:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:28 smithi193 ceph-mon[37473]: pgmap v4630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:28:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7250.202997 secs 2024-07-26T13:28:29.464 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7250.202997 secs 2024-07-26T13:28:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7250.202997 secs 2024-07-26T13:28:30.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:30 smithi134 ceph-mon[31966]: pgmap v4631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:30 smithi189 ceph-mon[36790]: pgmap v4631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:30 smithi193 ceph-mon[37473]: pgmap v4631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:32 smithi134 ceph-mon[31966]: pgmap v4632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7269.629265 secs 2024-07-26T13:28:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:32 smithi189 ceph-mon[36790]: pgmap v4632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7269.629265 secs 2024-07-26T13:28:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:32 smithi193 ceph-mon[37473]: pgmap v4632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7269.629265 secs 2024-07-26T13:28:34.214 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:28:34.213+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:28:34.215 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:28:34.213+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:28:34.215 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:28:34.213+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:34.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:34 smithi134 ceph-mon[31966]: pgmap v4633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7255.203127 secs 2024-07-26T13:28:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:34 smithi189 ceph-mon[36790]: pgmap v4633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7255.203127 secs 2024-07-26T13:28:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:34 smithi193 ceph-mon[37473]: pgmap v4633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7255.203127 secs 2024-07-26T13:28:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:36 smithi134 ceph-mon[31966]: pgmap v4634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:28:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:36 smithi189 ceph-mon[36790]: pgmap v4634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:28:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:36 smithi193 ceph-mon[37473]: pgmap v4634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:28:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7274.629380 secs 2024-07-26T13:28:37.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7274.629380 secs 2024-07-26T13:28:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7274.629380 secs 2024-07-26T13:28:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:38 smithi134 ceph-mon[31966]: pgmap v4635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:38 smithi189 ceph-mon[36790]: pgmap v4635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:38 smithi193 ceph-mon[37473]: pgmap v4635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:38.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:28:38.768+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:28:38.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:28:38.768+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:28:38.769 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:28:38.768+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7260.203208 secs 2024-07-26T13:28:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7260.203208 secs 2024-07-26T13:28:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7260.203208 secs 2024-07-26T13:28:40.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:40 smithi134 ceph-mon[31966]: pgmap v4636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:40 smithi189 ceph-mon[36790]: pgmap v4636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:40 smithi193 ceph-mon[37473]: pgmap v4636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:42 smithi134 ceph-mon[31966]: pgmap v4637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7279.629497 secs 2024-07-26T13:28:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:42 smithi189 ceph-mon[36790]: pgmap v4637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7279.629497 secs 2024-07-26T13:28:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:42 smithi193 ceph-mon[37473]: pgmap v4637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7279.629497 secs 2024-07-26T13:28:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:44 smithi134 ceph-mon[31966]: pgmap v4638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7265.203334 secs 2024-07-26T13:28:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:44 smithi189 ceph-mon[36790]: pgmap v4638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7265.203334 secs 2024-07-26T13:28:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:44 smithi193 ceph-mon[37473]: pgmap v4638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7265.203334 secs 2024-07-26T13:28:46.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:46 smithi134 ceph-mon[31966]: pgmap v4639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:28:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:46 smithi189 ceph-mon[36790]: pgmap v4639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:28:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:46 smithi193 ceph-mon[37473]: pgmap v4639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:28:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7284.629623 secs 2024-07-26T13:28:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7284.629623 secs 2024-07-26T13:28:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7284.629623 secs 2024-07-26T13:28:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:48 smithi134 ceph-mon[31966]: pgmap v4640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:48 smithi189 ceph-mon[36790]: pgmap v4640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:48 smithi193 ceph-mon[37473]: pgmap v4640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:49.229 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:28:49.229+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:28:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:28:49.229+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:28:49.230 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:28:49.229+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7270.203469 secs 2024-07-26T13:28:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7270.203469 secs 2024-07-26T13:28:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7270.203469 secs 2024-07-26T13:28:50.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:50 smithi134 ceph-mon[31966]: pgmap v4641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:50 smithi189 ceph-mon[36790]: pgmap v4641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:50 smithi193 ceph-mon[37473]: pgmap v4641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:28:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:52 smithi189 ceph-mon[36790]: pgmap v4642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7289.629684 secs 2024-07-26T13:28:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:52 smithi193 ceph-mon[37473]: pgmap v4642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7289.629684 secs 2024-07-26T13:28:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:52 smithi134 ceph-mon[31966]: pgmap v4642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:28:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7289.629684 secs 2024-07-26T13:28:53.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:28:53.784+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:28:53.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:28:53.784+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:28:53.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:28:53.784+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:28:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:54 smithi189 ceph-mon[36790]: pgmap v4643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7275.203573 secs 2024-07-26T13:28:54.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:54 smithi193 ceph-mon[37473]: pgmap v4643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:54.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7275.203573 secs 2024-07-26T13:28:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:54 smithi134 ceph-mon[31966]: pgmap v4643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:28:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7275.203573 secs 2024-07-26T13:28:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:56 smithi189 ceph-mon[36790]: pgmap v4644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:28:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:56 smithi193 ceph-mon[37473]: pgmap v4644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:28:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:56 smithi134 ceph-mon[31966]: pgmap v4644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:28:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7294.629804 secs 2024-07-26T13:28:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7294.629804 secs 2024-07-26T13:28:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7294.629804 secs 2024-07-26T13:28:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:58 smithi193 ceph-mon[37473]: pgmap v4645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:58 smithi134 ceph-mon[31966]: pgmap v4645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:58 smithi189 ceph-mon[36790]: pgmap v4645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:28:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:28:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7280.203731 secs 2024-07-26T13:28:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:28:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7280.203731 secs 2024-07-26T13:28:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:28:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7280.203731 secs 2024-07-26T13:29:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:00 smithi193 ceph-mon[37473]: pgmap v4646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:00 smithi134 ceph-mon[31966]: pgmap v4646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:00 smithi189 ceph-mon[36790]: pgmap v4646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:02 smithi193 ceph-mon[37473]: pgmap v4647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7299.629929 secs 2024-07-26T13:29:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:02 smithi134 ceph-mon[31966]: pgmap v4647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7299.629929 secs 2024-07-26T13:29:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:02 smithi189 ceph-mon[36790]: pgmap v4647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7299.629929 secs 2024-07-26T13:29:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:29:04.237+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:29:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:29:04.237+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:29:04.238 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:29:04.237+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:04.495 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:04 smithi189 ceph-mon[36790]: pgmap v4648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:04.496 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7285.203859 secs 2024-07-26T13:29:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:04 smithi193 ceph-mon[37473]: pgmap v4648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7285.203859 secs 2024-07-26T13:29:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:04 smithi134 ceph-mon[31966]: pgmap v4648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7285.203859 secs 2024-07-26T13:29:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:06 smithi193 ceph-mon[37473]: pgmap v4649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:06 smithi134 ceph-mon[31966]: pgmap v4649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:06 smithi189 ceph-mon[36790]: pgmap v4649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7304.630072 secs 2024-07-26T13:29:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7304.630072 secs 2024-07-26T13:29:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7304.630072 secs 2024-07-26T13:29:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:08 smithi193 ceph-mon[37473]: pgmap v4650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:08.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:08 smithi134 ceph-mon[31966]: pgmap v4650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:08 smithi189 ceph-mon[36790]: pgmap v4650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:08.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:29:08.799+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:29:08.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:29:08.799+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:29:08.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:29:08.799+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7290.204002 secs 2024-07-26T13:29:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7290.204002 secs 2024-07-26T13:29:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7290.204002 secs 2024-07-26T13:29:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:10 smithi193 ceph-mon[37473]: pgmap v4651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:10 smithi134 ceph-mon[31966]: pgmap v4651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:10 smithi189 ceph-mon[36790]: pgmap v4651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:12.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:12 smithi193 ceph-mon[37473]: pgmap v4652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:12.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7309.630193 secs 2024-07-26T13:29:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:12 smithi134 ceph-mon[31966]: pgmap v4652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7309.630193 secs 2024-07-26T13:29:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:12 smithi189 ceph-mon[36790]: pgmap v4652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7309.630193 secs 2024-07-26T13:29:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:14 smithi134 ceph-mon[31966]: pgmap v4653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7295.204103 secs 2024-07-26T13:29:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:14 smithi189 ceph-mon[36790]: pgmap v4653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7295.204103 secs 2024-07-26T13:29:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:14 smithi193 ceph-mon[37473]: pgmap v4653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7295.204103 secs 2024-07-26T13:29:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:16 smithi134 ceph-mon[31966]: pgmap v4654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:16 smithi189 ceph-mon[36790]: pgmap v4654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:16 smithi193 ceph-mon[37473]: pgmap v4654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7314.630338 secs 2024-07-26T13:29:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:29:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7314.630338 secs 2024-07-26T13:29:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:29:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7314.630338 secs 2024-07-26T13:29:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:29:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:18 smithi134 ceph-mon[31966]: pgmap v4655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:29:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:29:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:29:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:18 smithi189 ceph-mon[36790]: pgmap v4655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:29:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:29:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:29:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:18 smithi193 ceph-mon[37473]: pgmap v4655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:29:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:29:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:29:19.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:29:19.252+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:29:19.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:29:19.252+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:29:19.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:29:19.252+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:19 smithi134 ceph-mon[31966]: pgmap v4656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:29:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:19 smithi134 ceph-mon[31966]: pgmap v4657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7300.204248 secs 2024-07-26T13:29:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:19 smithi189 ceph-mon[36790]: pgmap v4656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:29:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:19 smithi189 ceph-mon[36790]: pgmap v4657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7300.204248 secs 2024-07-26T13:29:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:19 smithi193 ceph-mon[37473]: pgmap v4656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:29:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:19 smithi193 ceph-mon[37473]: pgmap v4657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7300.204248 secs 2024-07-26T13:29:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:21 smithi134 ceph-mon[31966]: pgmap v4658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:21 smithi189 ceph-mon[36790]: pgmap v4658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:21 smithi193 ceph-mon[37473]: pgmap v4658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7319.630403 secs 2024-07-26T13:29:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7319.630403 secs 2024-07-26T13:29:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7319.630403 secs 2024-07-26T13:29:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:23 smithi134 ceph-mon[31966]: pgmap v4659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:29:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:23 smithi189 ceph-mon[36790]: pgmap v4659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:29:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:23 smithi193 ceph-mon[37473]: pgmap v4659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:29:23.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:29:23.800+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:29:23.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:29:23.800+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:23.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:29:23.800+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:29:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7305.204329 secs 2024-07-26T13:29:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7305.204329 secs 2024-07-26T13:29:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7305.204329 secs 2024-07-26T13:29:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:25 smithi134 ceph-mon[31966]: pgmap v4660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:25 smithi189 ceph-mon[36790]: pgmap v4660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:25 smithi193 ceph-mon[37473]: pgmap v4660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:29:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7324.630536 secs 2024-07-26T13:29:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:27 smithi134 ceph-mon[31966]: pgmap v4661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:29:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7324.630536 secs 2024-07-26T13:29:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:27 smithi189 ceph-mon[36790]: pgmap v4661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:29:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7324.630536 secs 2024-07-26T13:29:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:27 smithi193 ceph-mon[37473]: pgmap v4661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:29:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:29 smithi134 ceph-mon[31966]: pgmap v4662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:29:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7310.204408 secs 2024-07-26T13:29:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:29 smithi189 ceph-mon[36790]: pgmap v4662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:29:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7310.204408 secs 2024-07-26T13:29:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:29 smithi193 ceph-mon[37473]: pgmap v4662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:29:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7310.204408 secs 2024-07-26T13:29:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:31 smithi134 ceph-mon[31966]: pgmap v4663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:31 smithi189 ceph-mon[36790]: pgmap v4663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:31 smithi193 ceph-mon[37473]: pgmap v4663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:32.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7329.630669 secs 2024-07-26T13:29:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7329.630669 secs 2024-07-26T13:29:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7329.630669 secs 2024-07-26T13:29:33.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:33 smithi134 ceph-mon[31966]: pgmap v4664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:33 smithi189 ceph-mon[36790]: pgmap v4664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:33 smithi193 ceph-mon[37473]: pgmap v4664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:34.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:29:34.268+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:29:34.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:29:34.268+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:29:34.270 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:29:34.268+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7315.204536 secs 2024-07-26T13:29:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7315.204536 secs 2024-07-26T13:29:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7315.204536 secs 2024-07-26T13:29:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:35 smithi134 ceph-mon[31966]: pgmap v4665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:35 smithi189 ceph-mon[36790]: pgmap v4665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:35 smithi193 ceph-mon[37473]: pgmap v4665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7334.630789 secs 2024-07-26T13:29:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:37 smithi134 ceph-mon[31966]: pgmap v4666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7334.630789 secs 2024-07-26T13:29:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:37 smithi189 ceph-mon[36790]: pgmap v4666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7334.630789 secs 2024-07-26T13:29:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:37 smithi193 ceph-mon[37473]: pgmap v4666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:38.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:29:38.815+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:29:38.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:29:38.815+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:38.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:29:38.815+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:29:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:39 smithi134 ceph-mon[31966]: pgmap v4667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7320.204630 secs 2024-07-26T13:29:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:39 smithi189 ceph-mon[36790]: pgmap v4667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7320.204630 secs 2024-07-26T13:29:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:39 smithi193 ceph-mon[37473]: pgmap v4667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7320.204630 secs 2024-07-26T13:29:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:41 smithi134 ceph-mon[31966]: pgmap v4668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:41 smithi189 ceph-mon[36790]: pgmap v4668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:41 smithi193 ceph-mon[37473]: pgmap v4668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7339.630929 secs 2024-07-26T13:29:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7339.630929 secs 2024-07-26T13:29:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7339.630929 secs 2024-07-26T13:29:43.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:43 smithi134 ceph-mon[31966]: pgmap v4669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:43 smithi189 ceph-mon[36790]: pgmap v4669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:43 smithi193 ceph-mon[37473]: pgmap v4669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7325.204725 secs 2024-07-26T13:29:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7325.204725 secs 2024-07-26T13:29:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7325.204725 secs 2024-07-26T13:29:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:45 smithi134 ceph-mon[31966]: pgmap v4670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:45 smithi189 ceph-mon[36790]: pgmap v4670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:45 smithi193 ceph-mon[37473]: pgmap v4670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7344.631028 secs 2024-07-26T13:29:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:47 smithi189 ceph-mon[36790]: pgmap v4671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7344.631028 secs 2024-07-26T13:29:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:47 smithi193 ceph-mon[37473]: pgmap v4671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7344.631028 secs 2024-07-26T13:29:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:47 smithi134 ceph-mon[31966]: pgmap v4671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:48 smithi193 ceph-mon[37473]: pgmap v4672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7330.204868 secs 2024-07-26T13:29:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:48 smithi134 ceph-mon[31966]: pgmap v4672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7330.204868 secs 2024-07-26T13:29:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:48 smithi189 ceph-mon[36790]: pgmap v4672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:48.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7330.204868 secs 2024-07-26T13:29:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:29:49.283+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:29:49.285 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:29:49.284+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:29:49.286 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:29:49.284+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:51.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:51 smithi134 ceph-mon[31966]: pgmap v4673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:51 smithi189 ceph-mon[36790]: pgmap v4673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:51 smithi193 ceph-mon[37473]: pgmap v4673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7349.631173 secs 2024-07-26T13:29:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7349.631173 secs 2024-07-26T13:29:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7349.631173 secs 2024-07-26T13:29:53.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:53 smithi134 ceph-mon[31966]: pgmap v4674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:53 smithi189 ceph-mon[36790]: pgmap v4674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:53 smithi193 ceph-mon[37473]: pgmap v4674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:29:53.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:29:53.831+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:29:53.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:29:53.831+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:29:53.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:29:53.831+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:29:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7335.204997 secs 2024-07-26T13:29:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7335.204997 secs 2024-07-26T13:29:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7335.204997 secs 2024-07-26T13:29:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:55 smithi189 ceph-mon[36790]: pgmap v4675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:55 smithi193 ceph-mon[37473]: pgmap v4675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:55 smithi134 ceph-mon[31966]: pgmap v4675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:29:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7354.631277 secs 2024-07-26T13:29:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:57 smithi189 ceph-mon[36790]: pgmap v4676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7354.631277 secs 2024-07-26T13:29:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:57 smithi193 ceph-mon[37473]: pgmap v4676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7354.631277 secs 2024-07-26T13:29:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:57 smithi134 ceph-mon[31966]: pgmap v4676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:29:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:59 smithi189 ceph-mon[36790]: pgmap v4677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:29:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7340.205091 secs 2024-07-26T13:29:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:59 smithi193 ceph-mon[37473]: pgmap v4677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:29:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7340.205091 secs 2024-07-26T13:29:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:59 smithi134 ceph-mon[31966]: pgmap v4677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:29:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:29:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7340.205091 secs 2024-07-26T13:30:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:30:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:30:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:30:01.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:01 smithi193 ceph-mon[37473]: pgmap v4678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:01 smithi134 ceph-mon[31966]: pgmap v4678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:01 smithi189 ceph-mon[36790]: pgmap v4678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7359.631370 secs 2024-07-26T13:30:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7359.631370 secs 2024-07-26T13:30:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7359.631370 secs 2024-07-26T13:30:03.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:03 smithi193 ceph-mon[37473]: pgmap v4679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:03.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:03 smithi134 ceph-mon[31966]: pgmap v4679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:03 smithi189 ceph-mon[36790]: pgmap v4679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:04.289 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:30:04.288+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:30:04.289 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:30:04.288+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:30:04.289 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:30:04.288+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7345.205203 secs 2024-07-26T13:30:04.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7345.205203 secs 2024-07-26T13:30:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7345.205203 secs 2024-07-26T13:30:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:05 smithi193 ceph-mon[37473]: pgmap v4680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:05 smithi134 ceph-mon[31966]: pgmap v4680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:05 smithi189 ceph-mon[36790]: pgmap v4680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7364.631496 secs 2024-07-26T13:30:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:07 smithi193 ceph-mon[37473]: pgmap v4681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7364.631496 secs 2024-07-26T13:30:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:07 smithi134 ceph-mon[31966]: pgmap v4681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7364.631496 secs 2024-07-26T13:30:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:07 smithi189 ceph-mon[36790]: pgmap v4681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:08.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:30:08.847+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:30:08.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:30:08.847+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:08.848 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:30:08.847+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:30:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:09 smithi193 ceph-mon[37473]: pgmap v4682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7350.205332 secs 2024-07-26T13:30:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:09 smithi134 ceph-mon[31966]: pgmap v4682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7350.205332 secs 2024-07-26T13:30:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:09 smithi189 ceph-mon[36790]: pgmap v4682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7350.205332 secs 2024-07-26T13:30:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:11 smithi193 ceph-mon[37473]: pgmap v4683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:11.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:11 smithi134 ceph-mon[31966]: pgmap v4683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:11 smithi189 ceph-mon[36790]: pgmap v4683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7369.631577 secs 2024-07-26T13:30:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7369.631577 secs 2024-07-26T13:30:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7369.631577 secs 2024-07-26T13:30:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:13 smithi193 ceph-mon[37473]: pgmap v4684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:13.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:13 smithi134 ceph-mon[31966]: pgmap v4684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:13 smithi189 ceph-mon[36790]: pgmap v4684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7355.205465 secs 2024-07-26T13:30:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7355.205465 secs 2024-07-26T13:30:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7355.205465 secs 2024-07-26T13:30:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:15 smithi134 ceph-mon[31966]: pgmap v4685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:15 smithi189 ceph-mon[36790]: pgmap v4685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:15 smithi193 ceph-mon[37473]: pgmap v4685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7374.631662 secs 2024-07-26T13:30:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:17 smithi134 ceph-mon[31966]: pgmap v4686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7374.631662 secs 2024-07-26T13:30:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:17 smithi189 ceph-mon[36790]: pgmap v4686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7374.631662 secs 2024-07-26T13:30:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:17 smithi193 ceph-mon[37473]: pgmap v4686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:18.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:30:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:30:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:30:19.304 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:30:19.303+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:30:19.304 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:30:19.303+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:30:19.304 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:30:19.303+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:19 smithi134 ceph-mon[31966]: pgmap v4687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7360.205567 secs 2024-07-26T13:30:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:30:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:30:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:30:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:19 smithi189 ceph-mon[36790]: pgmap v4687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7360.205567 secs 2024-07-26T13:30:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:30:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:30:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:30:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:19 smithi193 ceph-mon[37473]: pgmap v4687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7360.205567 secs 2024-07-26T13:30:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:30:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:30:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:30:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:20 smithi134 ceph-mon[31966]: pgmap v4688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:30:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:20 smithi134 ceph-mon[31966]: pgmap v4689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:30:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:20 smithi189 ceph-mon[36790]: pgmap v4688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:30:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:20 smithi189 ceph-mon[36790]: pgmap v4689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:30:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:20 smithi193 ceph-mon[37473]: pgmap v4688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:30:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:20 smithi193 ceph-mon[37473]: pgmap v4689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:30:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7379.631727 secs 2024-07-26T13:30:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:22 smithi134 ceph-mon[31966]: pgmap v4690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:30:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7379.631727 secs 2024-07-26T13:30:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:22 smithi189 ceph-mon[36790]: pgmap v4690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:30:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7379.631727 secs 2024-07-26T13:30:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:22 smithi193 ceph-mon[37473]: pgmap v4690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:30:23.608 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:30:23.607+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:30:23.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:30:23.607+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:30:23.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:30:23.608+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T13:30:23.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:30:23.608+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T13:30:23.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:30:23.862+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:30:23.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:30:23.862+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:23.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:30:23.862+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:30:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:24 smithi134 ceph-mon[31966]: pgmap v4691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:30:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7365.205699 secs 2024-07-26T13:30:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:24 smithi189 ceph-mon[36790]: pgmap v4691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:30:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7365.205699 secs 2024-07-26T13:30:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:24 smithi193 ceph-mon[37473]: pgmap v4691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:30:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7365.205699 secs 2024-07-26T13:30:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:26 smithi134 ceph-mon[31966]: pgmap v4692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:30:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:26 smithi189 ceph-mon[36790]: pgmap v4692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:30:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:26 smithi193 ceph-mon[37473]: pgmap v4692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:30:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7384.631844 secs 2024-07-26T13:30:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7384.631844 secs 2024-07-26T13:30:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7384.631844 secs 2024-07-26T13:30:28.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:28 smithi134 ceph-mon[31966]: pgmap v4693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:30:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:28 smithi189 ceph-mon[36790]: pgmap v4693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:30:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:28 smithi193 ceph-mon[37473]: pgmap v4693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:30:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7370.205836 secs 2024-07-26T13:30:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7370.205836 secs 2024-07-26T13:30:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7370.205836 secs 2024-07-26T13:30:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:30 smithi134 ceph-mon[31966]: pgmap v4694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:30:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:30 smithi189 ceph-mon[36790]: pgmap v4694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:30:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:30 smithi193 ceph-mon[37473]: pgmap v4694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:30:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7389.631866 secs 2024-07-26T13:30:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7389.631866 secs 2024-07-26T13:30:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7389.631866 secs 2024-07-26T13:30:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:32 smithi134 ceph-mon[31966]: pgmap v4695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:32 smithi189 ceph-mon[36790]: pgmap v4695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:32 smithi193 ceph-mon[37473]: pgmap v4695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:34.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:30:34.133+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:30:34.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:30:34.133+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:30:34.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:30:34.133+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:30:34.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:30:34.133+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:30:34.319 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:30:34.318+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:30:34.320 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:30:34.318+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:30:34.320 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:30:34.318+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:34 smithi134 ceph-mon[31966]: pgmap v4696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7375.205982 secs 2024-07-26T13:30:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:34 smithi189 ceph-mon[36790]: pgmap v4696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7375.205982 secs 2024-07-26T13:30:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:34 smithi193 ceph-mon[37473]: pgmap v4696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7375.205982 secs 2024-07-26T13:30:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:36 smithi134 ceph-mon[31966]: pgmap v4697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7394.631984 secs 2024-07-26T13:30:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:36 smithi189 ceph-mon[36790]: pgmap v4697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7394.631984 secs 2024-07-26T13:30:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:36 smithi193 ceph-mon[37473]: pgmap v4697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7394.631984 secs 2024-07-26T13:30:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:38 smithi134 ceph-mon[31966]: pgmap v4698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:38 smithi189 ceph-mon[36790]: pgmap v4698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:38 smithi193 ceph-mon[37473]: pgmap v4698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:38.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:30:38.868+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:30:38.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:30:38.869+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:38.871 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:30:38.869+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:30:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7380.206118 secs 2024-07-26T13:30:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7380.206118 secs 2024-07-26T13:30:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7380.206118 secs 2024-07-26T13:30:40.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:40 smithi134 ceph-mon[31966]: pgmap v4699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:40 smithi189 ceph-mon[36790]: pgmap v4699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:40 smithi193 ceph-mon[37473]: pgmap v4699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7399.632102 secs 2024-07-26T13:30:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7399.632102 secs 2024-07-26T13:30:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7399.632102 secs 2024-07-26T13:30:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:42 smithi134 ceph-mon[31966]: pgmap v4700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:42 smithi189 ceph-mon[36790]: pgmap v4700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:42 smithi193 ceph-mon[37473]: pgmap v4700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:44 smithi189 ceph-mon[36790]: pgmap v4701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7385.206232 secs 2024-07-26T13:30:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:44 smithi193 ceph-mon[37473]: pgmap v4701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7385.206232 secs 2024-07-26T13:30:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:44 smithi134 ceph-mon[31966]: pgmap v4701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7385.206232 secs 2024-07-26T13:30:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:46 smithi189 ceph-mon[36790]: pgmap v4702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7404.632232 secs 2024-07-26T13:30:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:46 smithi193 ceph-mon[37473]: pgmap v4702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7404.632232 secs 2024-07-26T13:30:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:46 smithi134 ceph-mon[31966]: pgmap v4702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7404.632232 secs 2024-07-26T13:30:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:48 smithi189 ceph-mon[36790]: pgmap v4703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:48 smithi193 ceph-mon[37473]: pgmap v4703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:48 smithi134 ceph-mon[31966]: pgmap v4703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:30:49.334+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:30:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:30:49.334+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:30:49.335 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:30:49.334+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7390.206350 secs 2024-07-26T13:30:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7390.206350 secs 2024-07-26T13:30:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7390.206350 secs 2024-07-26T13:30:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:50 smithi193 ceph-mon[37473]: pgmap v4704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:50 smithi134 ceph-mon[31966]: pgmap v4704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:50 smithi189 ceph-mon[36790]: pgmap v4704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:30:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7409.632348 secs 2024-07-26T13:30:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:51 smithi193 ceph-mon[37473]: pgmap v4705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7409.632348 secs 2024-07-26T13:30:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:51 smithi134 ceph-mon[31966]: pgmap v4705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7409.632348 secs 2024-07-26T13:30:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:51 smithi189 ceph-mon[36790]: pgmap v4705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:53.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:30:53.884+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:30:53.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:30:53.884+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:30:53.886 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:30:53.884+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:30:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:54 smithi134 ceph-mon[31966]: pgmap v4706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7395.206468 secs 2024-07-26T13:30:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:54 smithi189 ceph-mon[36790]: pgmap v4706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7395.206468 secs 2024-07-26T13:30:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:54 smithi193 ceph-mon[37473]: pgmap v4706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:30:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7395.206468 secs 2024-07-26T13:30:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:56 smithi134 ceph-mon[31966]: pgmap v4707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7414.632446 secs 2024-07-26T13:30:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:56 smithi189 ceph-mon[36790]: pgmap v4707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7414.632446 secs 2024-07-26T13:30:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:56 smithi193 ceph-mon[37473]: pgmap v4707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:30:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7414.632446 secs 2024-07-26T13:30:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:58 smithi134 ceph-mon[31966]: pgmap v4708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:58 smithi189 ceph-mon[36790]: pgmap v4708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:58 smithi193 ceph-mon[37473]: pgmap v4708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:30:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:30:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7400.206548 secs 2024-07-26T13:30:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:30:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7400.206548 secs 2024-07-26T13:30:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:30:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7400.206548 secs 2024-07-26T13:31:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:00 smithi134 ceph-mon[31966]: pgmap v4709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:00 smithi189 ceph-mon[36790]: pgmap v4709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:00 smithi193 ceph-mon[37473]: pgmap v4709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7419.632568 secs 2024-07-26T13:31:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7419.632568 secs 2024-07-26T13:31:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7419.632568 secs 2024-07-26T13:31:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:02 smithi134 ceph-mon[31966]: pgmap v4710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:02 smithi189 ceph-mon[36790]: pgmap v4710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:02 smithi193 ceph-mon[37473]: pgmap v4710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:04.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:31:04.350+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:31:04.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:31:04.350+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:31:04.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:31:04.351+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:04 smithi134 ceph-mon[31966]: pgmap v4711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7405.206646 secs 2024-07-26T13:31:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:04 smithi189 ceph-mon[36790]: pgmap v4711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7405.206646 secs 2024-07-26T13:31:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:04 smithi193 ceph-mon[37473]: pgmap v4711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7405.206646 secs 2024-07-26T13:31:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:06 smithi134 ceph-mon[31966]: pgmap v4712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7424.632679 secs 2024-07-26T13:31:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:06 smithi189 ceph-mon[36790]: pgmap v4712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7424.632679 secs 2024-07-26T13:31:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:06 smithi193 ceph-mon[37473]: pgmap v4712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7424.632679 secs 2024-07-26T13:31:08.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:31:08.615+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:31:08.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:31:08.615+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:31:08.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:31:08.616+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:31:08.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:31:08.616+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:31:08.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:08 smithi134 ceph-mon[31966]: pgmap v4713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:08 smithi189 ceph-mon[36790]: pgmap v4713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:08 smithi193 ceph-mon[37473]: pgmap v4713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:08.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:31:08.899+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:31:08.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:31:08.900+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:31:08.901 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:31:08.899+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7410.206790 secs 2024-07-26T13:31:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7410.206790 secs 2024-07-26T13:31:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7410.206790 secs 2024-07-26T13:31:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:10 smithi134 ceph-mon[31966]: pgmap v4714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:10 smithi189 ceph-mon[36790]: pgmap v4714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:10 smithi193 ceph-mon[37473]: pgmap v4714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7429.632738 secs 2024-07-26T13:31:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7429.632738 secs 2024-07-26T13:31:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7429.632738 secs 2024-07-26T13:31:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:12 smithi189 ceph-mon[36790]: pgmap v4715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:12 smithi193 ceph-mon[37473]: pgmap v4715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:12 smithi134 ceph-mon[31966]: pgmap v4715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:14 smithi189 ceph-mon[36790]: pgmap v4716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7415.206906 secs 2024-07-26T13:31:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:14 smithi193 ceph-mon[37473]: pgmap v4716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7415.206906 secs 2024-07-26T13:31:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:14 smithi134 ceph-mon[31966]: pgmap v4716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7415.206906 secs 2024-07-26T13:31:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:15 smithi193 ceph-mon[37473]: pgmap v4717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:15.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:15 smithi134 ceph-mon[31966]: pgmap v4717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:15 smithi189 ceph-mon[36790]: pgmap v4717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7434.632856 secs 2024-07-26T13:31:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7434.632856 secs 2024-07-26T13:31:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7434.632856 secs 2024-07-26T13:31:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:17 smithi193 ceph-mon[37473]: pgmap v4718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:17.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:17 smithi134 ceph-mon[31966]: pgmap v4718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:17 smithi189 ceph-mon[36790]: pgmap v4718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:31:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7420.207025 secs 2024-07-26T13:31:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7420.207025 secs 2024-07-26T13:31:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7420.207025 secs 2024-07-26T13:31:19.142 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:31:19.141+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:31:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:31:19.141+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:31:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:31:19.142+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:31:19.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:31:19.142+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:31:19.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:31:19.365+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:31:19.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:31:19.365+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:19.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:31:19.365+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:31:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:19 smithi193 ceph-mon[37473]: pgmap v4719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:31:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:19 smithi134 ceph-mon[31966]: pgmap v4719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:31:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:19 smithi189 ceph-mon[36790]: pgmap v4719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:31:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:31:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:31:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:31:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:31:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:31:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:31:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:31:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:31:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:31:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:31:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:21 smithi134 ceph-mon[31966]: pgmap v4720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:31:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:21 smithi134 ceph-mon[31966]: pgmap v4721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:31:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7439.632949 secs 2024-07-26T13:31:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:21 smithi189 ceph-mon[36790]: pgmap v4720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:31:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:21 smithi189 ceph-mon[36790]: pgmap v4721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:31:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7439.632949 secs 2024-07-26T13:31:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:21 smithi193 ceph-mon[37473]: pgmap v4720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:31:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:21 smithi193 ceph-mon[37473]: pgmap v4721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:31:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7439.632949 secs 2024-07-26T13:31:23.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:31:23.902+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:31:23.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:31:23.902+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:31:23.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:31:23.902+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:23.904 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:23 smithi134 ceph-mon[31966]: pgmap v4722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:31:23.904 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7425.207164 secs 2024-07-26T13:31:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:23 smithi189 ceph-mon[36790]: pgmap v4722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:31:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7425.207164 secs 2024-07-26T13:31:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:23 smithi193 ceph-mon[37473]: pgmap v4722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:31:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7425.207164 secs 2024-07-26T13:31:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:25 smithi134 ceph-mon[31966]: pgmap v4723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:31:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:25 smithi189 ceph-mon[36790]: pgmap v4723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:31:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:25 smithi193 ceph-mon[37473]: pgmap v4723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:31:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7444.633070 secs 2024-07-26T13:31:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7444.633070 secs 2024-07-26T13:31:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7444.633070 secs 2024-07-26T13:31:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:27 smithi134 ceph-mon[31966]: pgmap v4724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:31:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:27 smithi189 ceph-mon[36790]: pgmap v4724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:31:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:27 smithi193 ceph-mon[37473]: pgmap v4724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:31:28.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7430.207255 secs 2024-07-26T13:31:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7430.207255 secs 2024-07-26T13:31:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7430.207255 secs 2024-07-26T13:31:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:29 smithi134 ceph-mon[31966]: pgmap v4725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:31:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:29 smithi189 ceph-mon[36790]: pgmap v4725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:31:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:29 smithi193 ceph-mon[37473]: pgmap v4725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:31:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:31 smithi134 ceph-mon[31966]: pgmap v4726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:31:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7449.633133 secs 2024-07-26T13:31:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:31 smithi189 ceph-mon[36790]: pgmap v4726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:31:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7449.633133 secs 2024-07-26T13:31:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:31 smithi193 ceph-mon[37473]: pgmap v4726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:31:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7449.633133 secs 2024-07-26T13:31:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:33 smithi134 ceph-mon[31966]: pgmap v4727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7435.207376 secs 2024-07-26T13:31:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:33 smithi189 ceph-mon[36790]: pgmap v4727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7435.207376 secs 2024-07-26T13:31:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:33 smithi193 ceph-mon[37473]: pgmap v4727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7435.207376 secs 2024-07-26T13:31:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:31:34.376+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:31:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:31:34.376+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:31:34.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:31:34.376+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:35.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:35 smithi134 ceph-mon[31966]: pgmap v4728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:35 smithi189 ceph-mon[36790]: pgmap v4728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:35 smithi193 ceph-mon[37473]: pgmap v4728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7454.633262 secs 2024-07-26T13:31:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7454.633262 secs 2024-07-26T13:31:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7454.633262 secs 2024-07-26T13:31:37.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:37 smithi134 ceph-mon[31966]: pgmap v4729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:37 smithi189 ceph-mon[36790]: pgmap v4729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:37 smithi193 ceph-mon[37473]: pgmap v4729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:38.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:31:38.917+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:31:38.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:31:38.917+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:38.919 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:31:38.917+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:31:38.920 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7440.207443 secs 2024-07-26T13:31:38.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7440.207443 secs 2024-07-26T13:31:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7440.207443 secs 2024-07-26T13:31:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:39 smithi134 ceph-mon[31966]: pgmap v4730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:39 smithi189 ceph-mon[36790]: pgmap v4730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:39 smithi193 ceph-mon[37473]: pgmap v4730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:41 smithi134 ceph-mon[31966]: pgmap v4731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7459.633328 secs 2024-07-26T13:31:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:41 smithi189 ceph-mon[36790]: pgmap v4731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7459.633328 secs 2024-07-26T13:31:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:41 smithi193 ceph-mon[37473]: pgmap v4731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7459.633328 secs 2024-07-26T13:31:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:43 smithi134 ceph-mon[31966]: pgmap v4732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7445.207583 secs 2024-07-26T13:31:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:43 smithi189 ceph-mon[36790]: pgmap v4732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7445.207583 secs 2024-07-26T13:31:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:43 smithi193 ceph-mon[37473]: pgmap v4732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7445.207583 secs 2024-07-26T13:31:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:45 smithi134 ceph-mon[31966]: pgmap v4733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:45 smithi189 ceph-mon[36790]: pgmap v4733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:45 smithi193 ceph-mon[37473]: pgmap v4733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7464.633446 secs 2024-07-26T13:31:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7464.633446 secs 2024-07-26T13:31:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7464.633446 secs 2024-07-26T13:31:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:47 smithi134 ceph-mon[31966]: pgmap v4734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:47 smithi189 ceph-mon[36790]: pgmap v4734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:47 smithi193 ceph-mon[37473]: pgmap v4734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7450.207702 secs 2024-07-26T13:31:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7450.207702 secs 2024-07-26T13:31:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7450.207702 secs 2024-07-26T13:31:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:31:49.384+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:31:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:31:49.384+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:31:49.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:31:49.384+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:49 smithi134 ceph-mon[31966]: pgmap v4735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:49 smithi189 ceph-mon[36790]: pgmap v4735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:49 smithi193 ceph-mon[37473]: pgmap v4735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:31:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:51 smithi189 ceph-mon[36790]: pgmap v4736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7469.633569 secs 2024-07-26T13:31:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:51 smithi193 ceph-mon[37473]: pgmap v4736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7469.633569 secs 2024-07-26T13:31:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:51 smithi134 ceph-mon[31966]: pgmap v4736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7469.633569 secs 2024-07-26T13:31:53.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:31:53.933+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:31:53.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:31:53.933+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:31:53.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:31:53.933+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:31:53.935 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:53 smithi134 ceph-mon[31966]: pgmap v4737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:53.935 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7455.207827 secs 2024-07-26T13:31:53.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:53 smithi189 ceph-mon[36790]: pgmap v4737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:53.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7455.207827 secs 2024-07-26T13:31:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:53 smithi193 ceph-mon[37473]: pgmap v4737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:31:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7455.207827 secs 2024-07-26T13:31:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:55 smithi189 ceph-mon[36790]: pgmap v4738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:55 smithi193 ceph-mon[37473]: pgmap v4738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:56.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:55 smithi134 ceph-mon[31966]: pgmap v4738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:31:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7474.633689 secs 2024-07-26T13:31:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7474.633689 secs 2024-07-26T13:31:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7474.633689 secs 2024-07-26T13:31:58.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:57 smithi193 ceph-mon[37473]: pgmap v4739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:57 smithi134 ceph-mon[31966]: pgmap v4739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:57 smithi189 ceph-mon[36790]: pgmap v4739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:31:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7460.207965 secs 2024-07-26T13:31:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7460.207965 secs 2024-07-26T13:31:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7460.207965 secs 2024-07-26T13:32:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:31:59 smithi193 ceph-mon[37473]: pgmap v4740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:31:59 smithi134 ceph-mon[31966]: pgmap v4740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:31:59 smithi189 ceph-mon[36790]: pgmap v4740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:01 smithi193 ceph-mon[37473]: pgmap v4741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7479.633804 secs 2024-07-26T13:32:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:01 smithi134 ceph-mon[31966]: pgmap v4741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7479.633804 secs 2024-07-26T13:32:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:01 smithi189 ceph-mon[36790]: pgmap v4741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7479.633804 secs 2024-07-26T13:32:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:03 smithi193 ceph-mon[37473]: pgmap v4742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7465.208092 secs 2024-07-26T13:32:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:03 smithi134 ceph-mon[31966]: pgmap v4742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7465.208092 secs 2024-07-26T13:32:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:03 smithi189 ceph-mon[36790]: pgmap v4742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7465.208092 secs 2024-07-26T13:32:04.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:32:04.399+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:32:04.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:32:04.400+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:32:04.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:32:04.400+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:05.995 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:05 smithi189 ceph-mon[36790]: pgmap v4743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:06.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:05 smithi193 ceph-mon[37473]: pgmap v4743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:06.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:05 smithi134 ceph-mon[31966]: pgmap v4743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7484.633970 secs 2024-07-26T13:32:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7484.633970 secs 2024-07-26T13:32:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7484.633970 secs 2024-07-26T13:32:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:07 smithi193 ceph-mon[37473]: pgmap v4744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:08.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:07 smithi134 ceph-mon[31966]: pgmap v4744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:07 smithi189 ceph-mon[36790]: pgmap v4744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:08.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:32:08.948+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:32:08.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:32:08.948+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:08.950 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:32:08.948+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:32:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7470.208212 secs 2024-07-26T13:32:09.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7470.208212 secs 2024-07-26T13:32:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7470.208212 secs 2024-07-26T13:32:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:09 smithi193 ceph-mon[37473]: pgmap v4745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:10.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:09 smithi134 ceph-mon[31966]: pgmap v4745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:09 smithi189 ceph-mon[36790]: pgmap v4745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:11 smithi193 ceph-mon[37473]: pgmap v4746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7489.634086 secs 2024-07-26T13:32:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:11 smithi134 ceph-mon[31966]: pgmap v4746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7489.634086 secs 2024-07-26T13:32:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:11 smithi189 ceph-mon[36790]: pgmap v4746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7489.634086 secs 2024-07-26T13:32:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:13 smithi193 ceph-mon[37473]: pgmap v4747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7475.208334 secs 2024-07-26T13:32:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:13 smithi134 ceph-mon[31966]: pgmap v4747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7475.208334 secs 2024-07-26T13:32:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:13 smithi189 ceph-mon[36790]: pgmap v4747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:32:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7475.208334 secs 2024-07-26T13:32:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:15 smithi134 ceph-mon[31966]: pgmap v4748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:15 smithi189 ceph-mon[36790]: pgmap v4748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:15 smithi193 ceph-mon[37473]: pgmap v4748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:32:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7494.634182 secs 2024-07-26T13:32:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7494.634182 secs 2024-07-26T13:32:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7494.634182 secs 2024-07-26T13:32:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:17 smithi134 ceph-mon[31966]: pgmap v4749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:17 smithi189 ceph-mon[36790]: pgmap v4749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:17 smithi193 ceph-mon[37473]: pgmap v4749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:19.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7480.208454 secs 2024-07-26T13:32:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7480.208454 secs 2024-07-26T13:32:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7480.208454 secs 2024-07-26T13:32:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:32:19.415+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:32:19.416 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:32:19.415+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:32:19.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:32:19.415+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:20.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:19 smithi134 ceph-mon[31966]: pgmap v4750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:19 smithi189 ceph-mon[36790]: pgmap v4750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:19 smithi193 ceph-mon[37473]: pgmap v4750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:21.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:32:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:32:21.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:32:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:21 smithi134 ceph-mon[31966]: pgmap v4751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7499.634275 secs 2024-07-26T13:32:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:32:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:32:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:32:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:21 smithi189 ceph-mon[36790]: pgmap v4751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7499.634275 secs 2024-07-26T13:32:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:32:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:32:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:32:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:21 smithi193 ceph-mon[37473]: pgmap v4751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7499.634275 secs 2024-07-26T13:32:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:32:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:32:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:32:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:22 smithi134 ceph-mon[31966]: pgmap v4752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:32:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:22 smithi134 ceph-mon[31966]: pgmap v4753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:22 smithi189 ceph-mon[36790]: pgmap v4752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:32:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:22 smithi189 ceph-mon[36790]: pgmap v4753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:22 smithi193 ceph-mon[37473]: pgmap v4752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:32:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:22 smithi193 ceph-mon[37473]: pgmap v4753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:23.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:32:23.964+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:32:23.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:32:23.964+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:23.968 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:32:23.964+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:32:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7485.208594 secs 2024-07-26T13:32:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7485.208594 secs 2024-07-26T13:32:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7485.208594 secs 2024-07-26T13:32:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:24 smithi134 ceph-mon[31966]: pgmap v4754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:32:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:24 smithi189 ceph-mon[36790]: pgmap v4754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:32:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:24 smithi193 ceph-mon[37473]: pgmap v4754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:32:27.144 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:26 smithi134 ceph-mon[31966]: pgmap v4755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:32:27.145 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7504.634373 secs 2024-07-26T13:32:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:26 smithi189 ceph-mon[36790]: pgmap v4755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:32:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7504.634373 secs 2024-07-26T13:32:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:26 smithi193 ceph-mon[37473]: pgmap v4755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:32:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7504.634373 secs 2024-07-26T13:32:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:28 smithi134 ceph-mon[31966]: pgmap v4756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:32:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7490.208715 secs 2024-07-26T13:32:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:28 smithi189 ceph-mon[36790]: pgmap v4756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:32:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7490.208715 secs 2024-07-26T13:32:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:28 smithi193 ceph-mon[37473]: pgmap v4756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:32:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7490.208715 secs 2024-07-26T13:32:31.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:30 smithi134 ceph-mon[31966]: pgmap v4757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:32:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:30 smithi189 ceph-mon[36790]: pgmap v4757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:32:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:30 smithi193 ceph-mon[37473]: pgmap v4757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:32:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7509.634496 secs 2024-07-26T13:32:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7509.634496 secs 2024-07-26T13:32:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7509.634496 secs 2024-07-26T13:32:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:32 smithi134 ceph-mon[31966]: pgmap v4758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:32:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:32 smithi189 ceph-mon[36790]: pgmap v4758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:32:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:32 smithi193 ceph-mon[37473]: pgmap v4758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:32:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7495.208777 secs 2024-07-26T13:32:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7495.208777 secs 2024-07-26T13:32:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7495.208777 secs 2024-07-26T13:32:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:32:34.431+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:32:34.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:32:34.431+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:32:34.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:32:34.431+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:34 smithi134 ceph-mon[31966]: pgmap v4759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:34 smithi189 ceph-mon[36790]: pgmap v4759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:34 smithi193 ceph-mon[37473]: pgmap v4759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:36 smithi134 ceph-mon[31966]: pgmap v4760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7514.634617 secs 2024-07-26T13:32:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:36 smithi189 ceph-mon[36790]: pgmap v4760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7514.634617 secs 2024-07-26T13:32:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:36 smithi193 ceph-mon[37473]: pgmap v4760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7514.634617 secs 2024-07-26T13:32:38.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:32:38.976+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:32:38.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:32:38.976+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:38.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:32:38.976+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:32:39.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:38 smithi134 ceph-mon[31966]: pgmap v4761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7500.208922 secs 2024-07-26T13:32:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:38 smithi189 ceph-mon[36790]: pgmap v4761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7500.208922 secs 2024-07-26T13:32:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:38 smithi193 ceph-mon[37473]: pgmap v4761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7500.208922 secs 2024-07-26T13:32:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:40 smithi134 ceph-mon[31966]: pgmap v4762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:32:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:40 smithi189 ceph-mon[36790]: pgmap v4762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:32:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:40 smithi193 ceph-mon[37473]: pgmap v4762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:32:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7519.634722 secs 2024-07-26T13:32:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7519.634722 secs 2024-07-26T13:32:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7519.634722 secs 2024-07-26T13:32:43.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:42 smithi134 ceph-mon[31966]: pgmap v4763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:42 smithi189 ceph-mon[36790]: pgmap v4763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:42 smithi193 ceph-mon[37473]: pgmap v4763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7505.209040 secs 2024-07-26T13:32:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7505.209040 secs 2024-07-26T13:32:44.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7505.209040 secs 2024-07-26T13:32:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:44 smithi189 ceph-mon[36790]: pgmap v4764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:44 smithi193 ceph-mon[37473]: pgmap v4764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:44 smithi134 ceph-mon[31966]: pgmap v4764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:32:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:46 smithi189 ceph-mon[36790]: pgmap v4765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7524.634840 secs 2024-07-26T13:32:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:46 smithi193 ceph-mon[37473]: pgmap v4765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7524.634840 secs 2024-07-26T13:32:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:46 smithi134 ceph-mon[31966]: pgmap v4765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:32:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7524.634840 secs 2024-07-26T13:32:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:48 smithi189 ceph-mon[36790]: pgmap v4766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:32:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7510.209128 secs 2024-07-26T13:32:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:48 smithi193 ceph-mon[37473]: pgmap v4766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:32:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7510.209128 secs 2024-07-26T13:32:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:48 smithi134 ceph-mon[31966]: pgmap v4766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:32:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7510.209128 secs 2024-07-26T13:32:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:32:49.446+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:32:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:32:49.446+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:32:49.447 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:32:49.446+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:50 smithi193 ceph-mon[37473]: pgmap v4767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:32:51.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:50 smithi134 ceph-mon[31966]: pgmap v4767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:32:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:50 smithi189 ceph-mon[36790]: pgmap v4767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:32:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7529.634962 secs 2024-07-26T13:32:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7529.634962 secs 2024-07-26T13:32:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7529.634962 secs 2024-07-26T13:32:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:52 smithi193 ceph-mon[37473]: pgmap v4768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:32:53.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:52 smithi134 ceph-mon[31966]: pgmap v4768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:32:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:52 smithi189 ceph-mon[36790]: pgmap v4768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:32:53.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:32:53.992+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:32:53.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:32:53.992+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:32:53.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:32:53.992+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:32:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7515.209217 secs 2024-07-26T13:32:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7515.209217 secs 2024-07-26T13:32:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7515.209217 secs 2024-07-26T13:32:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:54 smithi193 ceph-mon[37473]: pgmap v4769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:32:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:54 smithi134 ceph-mon[31966]: pgmap v4769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:32:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:54 smithi189 ceph-mon[36790]: pgmap v4769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:32:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:56 smithi193 ceph-mon[37473]: pgmap v4770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:32:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7534.635096 secs 2024-07-26T13:32:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:56 smithi134 ceph-mon[31966]: pgmap v4770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:32:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7534.635096 secs 2024-07-26T13:32:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:56 smithi189 ceph-mon[36790]: pgmap v4770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:32:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7534.635096 secs 2024-07-26T13:32:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:58 smithi193 ceph-mon[37473]: pgmap v4771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:32:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:32:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7520.209339 secs 2024-07-26T13:32:59.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:58 smithi134 ceph-mon[31966]: pgmap v4771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:32:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:32:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7520.209339 secs 2024-07-26T13:32:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:58 smithi189 ceph-mon[36790]: pgmap v4771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:32:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:32:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7520.209339 secs 2024-07-26T13:33:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:00 smithi193 ceph-mon[37473]: pgmap v4772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:33:01.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:00 smithi134 ceph-mon[31966]: pgmap v4772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:33:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:00 smithi189 ceph-mon[36790]: pgmap v4772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:33:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7539.635216 secs 2024-07-26T13:33:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7539.635216 secs 2024-07-26T13:33:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7539.635216 secs 2024-07-26T13:33:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:03 smithi193 ceph-mon[37473]: pgmap v4773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:03.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:03 smithi134 ceph-mon[31966]: pgmap v4773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:03 smithi189 ceph-mon[36790]: pgmap v4773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7525.209460 secs 2024-07-26T13:33:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7525.209460 secs 2024-07-26T13:33:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7525.209460 secs 2024-07-26T13:33:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:33:04.462+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:33:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:33:04.462+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:33:04.462+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:33:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:05 smithi193 ceph-mon[37473]: pgmap v4774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:33:05.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:05 smithi189 ceph-mon[36790]: pgmap v4774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:33:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:05 smithi134 ceph-mon[31966]: pgmap v4774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:33:06.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7544.635339 secs 2024-07-26T13:33:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7544.635339 secs 2024-07-26T13:33:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7544.635339 secs 2024-07-26T13:33:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:07 smithi134 ceph-mon[31966]: pgmap v4775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:07 smithi189 ceph-mon[36790]: pgmap v4775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:07 smithi193 ceph-mon[37473]: pgmap v4775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:09.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:33:09.008+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:33:09.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:33:09.008+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:09.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:33:09.008+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:33:09.325 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:09 smithi193 ceph-mon[37473]: pgmap v4776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:33:09.325 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7530.209587 secs 2024-07-26T13:33:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:09 smithi134 ceph-mon[31966]: pgmap v4776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:33:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7530.209587 secs 2024-07-26T13:33:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:09 smithi189 ceph-mon[36790]: pgmap v4776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:33:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7530.209587 secs 2024-07-26T13:33:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:11 smithi134 ceph-mon[31966]: pgmap v4777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:33:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7549.635454 secs 2024-07-26T13:33:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:11 smithi189 ceph-mon[36790]: pgmap v4777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:33:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7549.635454 secs 2024-07-26T13:33:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:11 smithi193 ceph-mon[37473]: pgmap v4777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:33:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7549.635454 secs 2024-07-26T13:33:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:13 smithi134 ceph-mon[31966]: pgmap v4778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:13 smithi189 ceph-mon[36790]: pgmap v4778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:13 smithi193 ceph-mon[37473]: pgmap v4778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7535.209730 secs 2024-07-26T13:33:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7535.209730 secs 2024-07-26T13:33:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7535.209730 secs 2024-07-26T13:33:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:15 smithi134 ceph-mon[31966]: pgmap v4779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:33:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:15 smithi189 ceph-mon[36790]: pgmap v4779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:33:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:15 smithi193 ceph-mon[37473]: pgmap v4779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:33:16.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7554.635571 secs 2024-07-26T13:33:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7554.635571 secs 2024-07-26T13:33:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7554.635571 secs 2024-07-26T13:33:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:17 smithi134 ceph-mon[31966]: pgmap v4780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:17 smithi189 ceph-mon[36790]: pgmap v4780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:17 smithi193 ceph-mon[37473]: pgmap v4780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:19 smithi134 ceph-mon[31966]: pgmap v4781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:33:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7540.209865 secs 2024-07-26T13:33:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:19 smithi189 ceph-mon[36790]: pgmap v4781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:33:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7540.209865 secs 2024-07-26T13:33:19.478 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:33:19.477+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:33:19.478 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:33:19.477+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:19.478 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:33:19.477+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:33:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:19 smithi193 ceph-mon[37473]: pgmap v4781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:33:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7540.209865 secs 2024-07-26T13:33:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:21 smithi134 ceph-mon[31966]: pgmap v4782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:33:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7559.635667 secs 2024-07-26T13:33:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:21 smithi189 ceph-mon[36790]: pgmap v4782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:33:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7559.635667 secs 2024-07-26T13:33:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:21 smithi193 ceph-mon[37473]: pgmap v4782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:33:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7559.635667 secs 2024-07-26T13:33:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:33:22.447 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:33:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:33:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:23 smithi134 ceph-mon[31966]: pgmap v4783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:23.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:23.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:33:23.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:33:23.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:33:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:23 smithi189 ceph-mon[36790]: pgmap v4783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:23.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:23.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:33:23.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:33:23.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:33:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:23 smithi193 ceph-mon[37473]: pgmap v4783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:33:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:33:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:33:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:33:23.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:33:23.638+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:33:23.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:33:23.638+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:33:23.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:33:23.639+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T13:33:23.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:33:23.639+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T13:33:24.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:33:24.023+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:33:24.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:33:24.023+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:24.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:33:24.023+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:33:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:24 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:33:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:24 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:33:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:24 smithi134 ceph-mon[31966]: pgmap v4784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:33:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:24 smithi134 ceph-mon[31966]: pgmap v4785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:33:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7545.210021 secs 2024-07-26T13:33:24.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:24 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:33:24.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:24 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:33:24.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:24 smithi189 ceph-mon[36790]: pgmap v4784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:33:24.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:24 smithi189 ceph-mon[36790]: pgmap v4785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:33:24.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7545.210021 secs 2024-07-26T13:33:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:24 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:33:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:24 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:33:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:24 smithi193 ceph-mon[37473]: pgmap v4784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:33:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:24 smithi193 ceph-mon[37473]: pgmap v4785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:33:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7545.210021 secs 2024-07-26T13:33:26.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:26 smithi134 ceph-mon[31966]: pgmap v4786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:33:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7564.635795 secs 2024-07-26T13:33:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:26 smithi189 ceph-mon[36790]: pgmap v4786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:33:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7564.635795 secs 2024-07-26T13:33:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:26 smithi193 ceph-mon[37473]: pgmap v4786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:33:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7564.635795 secs 2024-07-26T13:33:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:28 smithi134 ceph-mon[31966]: pgmap v4787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:33:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:28 smithi189 ceph-mon[36790]: pgmap v4787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:33:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:28 smithi193 ceph-mon[37473]: pgmap v4787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:33:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7550.210122 secs 2024-07-26T13:33:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7550.210122 secs 2024-07-26T13:33:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7550.210122 secs 2024-07-26T13:33:30.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:30 smithi134 ceph-mon[31966]: pgmap v4788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:33:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:30 smithi189 ceph-mon[36790]: pgmap v4788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:33:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:30 smithi193 ceph-mon[37473]: pgmap v4788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:33:31.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7569.635922 secs 2024-07-26T13:33:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7569.635922 secs 2024-07-26T13:33:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7569.635922 secs 2024-07-26T13:33:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:32 smithi134 ceph-mon[31966]: pgmap v4789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:33:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:32 smithi189 ceph-mon[36790]: pgmap v4789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:33:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:32 smithi193 ceph-mon[37473]: pgmap v4789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:33:34.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:33:34.165+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:33:34.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:33:34.165+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:33:34.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:33:34.175+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:33:34.176 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:33:34.175+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006460 con 0x55a08e2d94d0 2024-07-26T13:33:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:34 smithi134 ceph-mon[31966]: pgmap v4790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:33:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7555.210258 secs 2024-07-26T13:33:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:34 smithi189 ceph-mon[36790]: pgmap v4790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:33:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7555.210258 secs 2024-07-26T13:33:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:33:34.486+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:33:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:33:34.486+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:33:34.486+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:33:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:34 smithi193 ceph-mon[37473]: pgmap v4790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:33:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7555.210258 secs 2024-07-26T13:33:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:36 smithi189 ceph-mon[36790]: pgmap v4791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:33:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7574.636056 secs 2024-07-26T13:33:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:36 smithi193 ceph-mon[37473]: pgmap v4791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:33:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7574.636056 secs 2024-07-26T13:33:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:36 smithi134 ceph-mon[31966]: pgmap v4791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:33:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7574.636056 secs 2024-07-26T13:33:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:38 smithi189 ceph-mon[36790]: pgmap v4792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:38 smithi193 ceph-mon[37473]: pgmap v4792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:38.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:38 smithi134 ceph-mon[31966]: pgmap v4792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:39.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:33:39.035+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:33:39.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:33:39.035+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:33:39.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:33:39.035+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7560.210402 secs 2024-07-26T13:33:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7560.210402 secs 2024-07-26T13:33:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7560.210402 secs 2024-07-26T13:33:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:40 smithi193 ceph-mon[37473]: pgmap v4793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:33:40.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:40 smithi134 ceph-mon[31966]: pgmap v4793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:33:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:40 smithi189 ceph-mon[36790]: pgmap v4793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:33:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7579.636176 secs 2024-07-26T13:33:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7579.636176 secs 2024-07-26T13:33:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7579.636176 secs 2024-07-26T13:33:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:42 smithi193 ceph-mon[37473]: pgmap v4794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:33:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:42 smithi134 ceph-mon[31966]: pgmap v4794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:33:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:42 smithi189 ceph-mon[36790]: pgmap v4794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:33:44.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:44 smithi193 ceph-mon[37473]: pgmap v4795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:33:44.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7565.210508 secs 2024-07-26T13:33:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:44 smithi134 ceph-mon[31966]: pgmap v4795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:33:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7565.210508 secs 2024-07-26T13:33:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:44 smithi189 ceph-mon[36790]: pgmap v4795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:33:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7565.210508 secs 2024-07-26T13:33:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:46 smithi193 ceph-mon[37473]: pgmap v4796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:33:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7584.636294 secs 2024-07-26T13:33:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:46 smithi134 ceph-mon[31966]: pgmap v4796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:33:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7584.636294 secs 2024-07-26T13:33:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:46 smithi189 ceph-mon[36790]: pgmap v4796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:33:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7584.636294 secs 2024-07-26T13:33:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:48 smithi193 ceph-mon[37473]: pgmap v4797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:33:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:48 smithi134 ceph-mon[31966]: pgmap v4797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:33:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:48 smithi189 ceph-mon[36790]: pgmap v4797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:33:49.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:33:49.502+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:33:49.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:33:49.502+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:33:49.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:33:49.502+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7570.210620 secs 2024-07-26T13:33:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7570.210620 secs 2024-07-26T13:33:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7570.210620 secs 2024-07-26T13:33:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:50 smithi193 ceph-mon[37473]: pgmap v4798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:33:50.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:50 smithi134 ceph-mon[31966]: pgmap v4798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:33:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:50 smithi189 ceph-mon[36790]: pgmap v4798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:33:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7589.636395 secs 2024-07-26T13:33:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7589.636395 secs 2024-07-26T13:33:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7589.636395 secs 2024-07-26T13:33:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:52 smithi193 ceph-mon[37473]: pgmap v4799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:52 smithi134 ceph-mon[31966]: pgmap v4799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:52 smithi189 ceph-mon[36790]: pgmap v4799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:54.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:33:54.051+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:33:54.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:33:54.051+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:33:54.053 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:33:54.051+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:33:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:54 smithi134 ceph-mon[31966]: pgmap v4800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:33:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7575.210749 secs 2024-07-26T13:33:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:54 smithi193 ceph-mon[37473]: pgmap v4800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:33:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7575.210749 secs 2024-07-26T13:33:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:54 smithi189 ceph-mon[36790]: pgmap v4800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:33:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7575.210749 secs 2024-07-26T13:33:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:56 smithi193 ceph-mon[37473]: pgmap v4801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:33:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7594.636515 secs 2024-07-26T13:33:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:56 smithi134 ceph-mon[31966]: pgmap v4801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:33:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7594.636515 secs 2024-07-26T13:33:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:56 smithi189 ceph-mon[36790]: pgmap v4801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:33:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7594.636515 secs 2024-07-26T13:33:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:58 smithi193 ceph-mon[37473]: pgmap v4802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:58 smithi134 ceph-mon[31966]: pgmap v4802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:58 smithi189 ceph-mon[36790]: pgmap v4802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:33:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:33:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7580.210877 secs 2024-07-26T13:33:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:33:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7580.210877 secs 2024-07-26T13:33:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:33:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7580.210877 secs 2024-07-26T13:34:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:00 smithi134 ceph-mon[31966]: pgmap v4803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:00 smithi189 ceph-mon[36790]: pgmap v4803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:00 smithi193 ceph-mon[37473]: pgmap v4803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7599.636643 secs 2024-07-26T13:34:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7599.636643 secs 2024-07-26T13:34:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7599.636643 secs 2024-07-26T13:34:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:02 smithi134 ceph-mon[31966]: pgmap v4804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:02 smithi189 ceph-mon[36790]: pgmap v4804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:02 smithi193 ceph-mon[37473]: pgmap v4804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:04.518 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:34:04.517+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:34:04.519 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:34:04.517+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:34:04.519 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:34:04.517+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:04 smithi134 ceph-mon[31966]: pgmap v4805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7585.211007 secs 2024-07-26T13:34:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:04 smithi189 ceph-mon[36790]: pgmap v4805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7585.211007 secs 2024-07-26T13:34:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:04 smithi193 ceph-mon[37473]: pgmap v4805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7585.211007 secs 2024-07-26T13:34:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:06 smithi134 ceph-mon[31966]: pgmap v4806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:34:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7604.636731 secs 2024-07-26T13:34:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:06 smithi189 ceph-mon[36790]: pgmap v4806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:34:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7604.636731 secs 2024-07-26T13:34:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:06 smithi193 ceph-mon[37473]: pgmap v4806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:34:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7604.636731 secs 2024-07-26T13:34:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:08 smithi134 ceph-mon[31966]: pgmap v4807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:08 smithi189 ceph-mon[36790]: pgmap v4807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:08 smithi193 ceph-mon[37473]: pgmap v4807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:09.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:34:09.055+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:34:09.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:34:09.055+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:34:09.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:34:09.055+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7590.211137 secs 2024-07-26T13:34:09.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7590.211137 secs 2024-07-26T13:34:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7590.211137 secs 2024-07-26T13:34:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:10 smithi134 ceph-mon[31966]: pgmap v4808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:10 smithi189 ceph-mon[36790]: pgmap v4808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:10 smithi193 ceph-mon[37473]: pgmap v4808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:11.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7609.636850 secs 2024-07-26T13:34:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7609.636850 secs 2024-07-26T13:34:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7609.636850 secs 2024-07-26T13:34:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:12 smithi134 ceph-mon[31966]: pgmap v4809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:12 smithi189 ceph-mon[36790]: pgmap v4809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:12 smithi193 ceph-mon[37473]: pgmap v4809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:13 smithi193 ceph-mon[37473]: pgmap v4810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7595.211234 secs 2024-07-26T13:34:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:13 smithi134 ceph-mon[31966]: pgmap v4810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7595.211234 secs 2024-07-26T13:34:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:13 smithi189 ceph-mon[36790]: pgmap v4810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7595.211234 secs 2024-07-26T13:34:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:15 smithi193 ceph-mon[37473]: pgmap v4811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:34:16.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:15 smithi134 ceph-mon[31966]: pgmap v4811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:34:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:15 smithi189 ceph-mon[36790]: pgmap v4811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:34:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7614.636971 secs 2024-07-26T13:34:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7614.636971 secs 2024-07-26T13:34:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7614.636971 secs 2024-07-26T13:34:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:18 smithi193 ceph-mon[37473]: pgmap v4812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:18.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:18 smithi134 ceph-mon[31966]: pgmap v4812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:18 smithi189 ceph-mon[36790]: pgmap v4812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7600.211356 secs 2024-07-26T13:34:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7600.211356 secs 2024-07-26T13:34:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7600.211356 secs 2024-07-26T13:34:19.534 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:34:19.532+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:34:19.534 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:34:19.533+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:19.534 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:34:19.533+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:34:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:20 smithi193 ceph-mon[37473]: pgmap v4813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:20 smithi134 ceph-mon[31966]: pgmap v4813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:20 smithi189 ceph-mon[36790]: pgmap v4813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:21.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7619.637071 secs 2024-07-26T13:34:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7619.637071 secs 2024-07-26T13:34:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7619.637071 secs 2024-07-26T13:34:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:22 smithi134 ceph-mon[31966]: pgmap v4814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:22 smithi189 ceph-mon[36790]: pgmap v4814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:22 smithi193 ceph-mon[37473]: pgmap v4814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:24.071 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:34:24.070+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:34:24.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:34:24.070+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:24.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:34:24.070+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:34:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:24 smithi134 ceph-mon[31966]: pgmap v4815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:34:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7605.211443 secs 2024-07-26T13:34:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:24 smithi189 ceph-mon[36790]: pgmap v4815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:34:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7605.211443 secs 2024-07-26T13:34:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:24 smithi193 ceph-mon[37473]: pgmap v4815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:34:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7605.211443 secs 2024-07-26T13:34:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:24.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:34:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:25 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:34:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:25 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:34:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:25 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:34:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:25 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:34:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:34:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:34:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:25 smithi134 ceph-mon[31966]: pgmap v4816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:34:25.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:34:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:25 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:34:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:25 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:34:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:25 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:34:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:25 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:34:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:34:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:34:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:25 smithi189 ceph-mon[36790]: pgmap v4816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:34:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:34:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:25 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:34:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:25 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:34:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:25 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:34:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:25 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:34:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:34:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:34:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:25 smithi193 ceph-mon[37473]: pgmap v4816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:34:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:34:26.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7624.637210 secs 2024-07-26T13:34:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7624.637210 secs 2024-07-26T13:34:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7624.637210 secs 2024-07-26T13:34:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:27 smithi134 ceph-mon[31966]: pgmap v4817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:34:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:27 smithi189 ceph-mon[36790]: pgmap v4817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:34:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:27 smithi193 ceph-mon[37473]: pgmap v4817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:34:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:28 smithi193 ceph-mon[37473]: pgmap v4818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:34:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7610.211586 secs 2024-07-26T13:34:29.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:28 smithi134 ceph-mon[31966]: pgmap v4818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:34:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7610.211586 secs 2024-07-26T13:34:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:28 smithi189 ceph-mon[36790]: pgmap v4818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:34:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7610.211586 secs 2024-07-26T13:34:31.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:31 smithi134 ceph-mon[31966]: pgmap v4819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:34:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7629.637333 secs 2024-07-26T13:34:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:31 smithi189 ceph-mon[36790]: pgmap v4819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:34:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7629.637333 secs 2024-07-26T13:34:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:31 smithi193 ceph-mon[37473]: pgmap v4819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:34:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7629.637333 secs 2024-07-26T13:34:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:33 smithi189 ceph-mon[36790]: pgmap v4820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:34:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:33 smithi193 ceph-mon[37473]: pgmap v4820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:34:33.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:33 smithi134 ceph-mon[31966]: pgmap v4820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:34:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7615.211731 secs 2024-07-26T13:34:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7615.211731 secs 2024-07-26T13:34:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:34:34.549+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:34:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:34:34.549+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:34:34.549+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:34:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7615.211731 secs 2024-07-26T13:34:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:35 smithi193 ceph-mon[37473]: pgmap v4821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:34:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:35 smithi134 ceph-mon[31966]: pgmap v4821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:34:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:35 smithi189 ceph-mon[36790]: pgmap v4821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:34:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7634.637408 secs 2024-07-26T13:34:36.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7634.637408 secs 2024-07-26T13:34:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7634.637408 secs 2024-07-26T13:34:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:37 smithi193 ceph-mon[37473]: pgmap v4822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:37 smithi134 ceph-mon[31966]: pgmap v4822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:37 smithi189 ceph-mon[36790]: pgmap v4822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:39.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:34:39.080+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:34:39.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:34:39.080+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:39.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:34:39.080+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:34:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:39 smithi134 ceph-mon[31966]: pgmap v4823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7620.211852 secs 2024-07-26T13:34:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:39 smithi193 ceph-mon[37473]: pgmap v4823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7620.211852 secs 2024-07-26T13:34:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:39 smithi189 ceph-mon[36790]: pgmap v4823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7620.211852 secs 2024-07-26T13:34:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:41 smithi193 ceph-mon[37473]: pgmap v4824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7639.637472 secs 2024-07-26T13:34:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:41 smithi134 ceph-mon[31966]: pgmap v4824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7639.637472 secs 2024-07-26T13:34:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:41 smithi189 ceph-mon[36790]: pgmap v4824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7639.637472 secs 2024-07-26T13:34:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:43 smithi193 ceph-mon[37473]: pgmap v4825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:43.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:43 smithi134 ceph-mon[31966]: pgmap v4825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:43 smithi189 ceph-mon[36790]: pgmap v4825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7625.211985 secs 2024-07-26T13:34:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7625.211985 secs 2024-07-26T13:34:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7625.211985 secs 2024-07-26T13:34:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:45 smithi193 ceph-mon[37473]: pgmap v4826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:45 smithi134 ceph-mon[31966]: pgmap v4826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:45 smithi189 ceph-mon[36790]: pgmap v4826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7644.637598 secs 2024-07-26T13:34:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7644.637598 secs 2024-07-26T13:34:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7644.637598 secs 2024-07-26T13:34:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:47 smithi193 ceph-mon[37473]: pgmap v4827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:47 smithi134 ceph-mon[31966]: pgmap v4827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:47 smithi189 ceph-mon[36790]: pgmap v4827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:49 smithi193 ceph-mon[37473]: pgmap v4828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7630.212142 secs 2024-07-26T13:34:49.565 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:34:49.564+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:34:49.565 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:34:49.564+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:49.565 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:34:49.564+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:34:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:49 smithi134 ceph-mon[31966]: pgmap v4828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7630.212142 secs 2024-07-26T13:34:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:49 smithi189 ceph-mon[36790]: pgmap v4828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7630.212142 secs 2024-07-26T13:34:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:51 smithi193 ceph-mon[37473]: pgmap v4829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7649.637717 secs 2024-07-26T13:34:51.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:51 smithi134 ceph-mon[31966]: pgmap v4829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:51.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7649.637717 secs 2024-07-26T13:34:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:51 smithi189 ceph-mon[36790]: pgmap v4829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7649.637717 secs 2024-07-26T13:34:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:53 smithi193 ceph-mon[37473]: pgmap v4830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:53.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:53 smithi134 ceph-mon[31966]: pgmap v4830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:53 smithi189 ceph-mon[36790]: pgmap v4830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:34:54.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:34:54.080+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:34:54.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:34:54.080+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:34:54.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:34:54.080+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:34:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7635.212303 secs 2024-07-26T13:34:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7635.212303 secs 2024-07-26T13:34:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7635.212303 secs 2024-07-26T13:34:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:55 smithi134 ceph-mon[31966]: pgmap v4831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:55 smithi189 ceph-mon[36790]: pgmap v4831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:55 smithi193 ceph-mon[37473]: pgmap v4831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:34:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7654.637862 secs 2024-07-26T13:34:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7654.637862 secs 2024-07-26T13:34:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7654.637862 secs 2024-07-26T13:34:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:57 smithi134 ceph-mon[31966]: pgmap v4832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:57 smithi189 ceph-mon[36790]: pgmap v4832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:57 smithi193 ceph-mon[37473]: pgmap v4832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:34:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:59 smithi134 ceph-mon[31966]: pgmap v4833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:34:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7640.212455 secs 2024-07-26T13:34:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:59 smithi189 ceph-mon[36790]: pgmap v4833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:34:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7640.212455 secs 2024-07-26T13:34:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:59 smithi193 ceph-mon[37473]: pgmap v4833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:34:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:34:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7640.212455 secs 2024-07-26T13:35:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:01 smithi134 ceph-mon[31966]: pgmap v4834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7659.637938 secs 2024-07-26T13:35:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:01 smithi189 ceph-mon[36790]: pgmap v4834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7659.637938 secs 2024-07-26T13:35:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:01 smithi193 ceph-mon[37473]: pgmap v4834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7659.637938 secs 2024-07-26T13:35:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:03 smithi134 ceph-mon[31966]: pgmap v4835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:03 smithi189 ceph-mon[36790]: pgmap v4835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:03 smithi193 ceph-mon[37473]: pgmap v4835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:04.580 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:35:04.579+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:35:04.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:35:04.579+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:04.581 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:35:04.579+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:35:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7645.212586 secs 2024-07-26T13:35:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7645.212586 secs 2024-07-26T13:35:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7645.212586 secs 2024-07-26T13:35:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:05 smithi134 ceph-mon[31966]: pgmap v4836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:05 smithi189 ceph-mon[36790]: pgmap v4836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:05 smithi193 ceph-mon[37473]: pgmap v4836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7664.638062 secs 2024-07-26T13:35:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7664.638062 secs 2024-07-26T13:35:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7664.638062 secs 2024-07-26T13:35:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:07 smithi134 ceph-mon[31966]: pgmap v4837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:07 smithi189 ceph-mon[36790]: pgmap v4837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:07 smithi193 ceph-mon[37473]: pgmap v4837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:09.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:35:09.093+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:35:09.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:35:09.093+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:35:09.095 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:35:09.093+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:09 smithi134 ceph-mon[31966]: pgmap v4838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:35:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7650.212734 secs 2024-07-26T13:35:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:09 smithi189 ceph-mon[36790]: pgmap v4838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:35:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7650.212734 secs 2024-07-26T13:35:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:09 smithi193 ceph-mon[37473]: pgmap v4838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:35:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7650.212734 secs 2024-07-26T13:35:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:11 smithi134 ceph-mon[31966]: pgmap v4839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:35:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7669.638183 secs 2024-07-26T13:35:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:11 smithi189 ceph-mon[36790]: pgmap v4839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:35:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7669.638183 secs 2024-07-26T13:35:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:11 smithi193 ceph-mon[37473]: pgmap v4839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:35:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7669.638183 secs 2024-07-26T13:35:13.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:13 smithi134 ceph-mon[31966]: pgmap v4840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:35:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:13 smithi189 ceph-mon[36790]: pgmap v4840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:35:13.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:13 smithi193 ceph-mon[37473]: pgmap v4840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:35:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7655.212891 secs 2024-07-26T13:35:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7655.212891 secs 2024-07-26T13:35:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7655.212891 secs 2024-07-26T13:35:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:15 smithi134 ceph-mon[31966]: pgmap v4841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:35:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:15 smithi189 ceph-mon[36790]: pgmap v4841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:35:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:15 smithi193 ceph-mon[37473]: pgmap v4841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:35:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7674.638321 secs 2024-07-26T13:35:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7674.638321 secs 2024-07-26T13:35:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7674.638321 secs 2024-07-26T13:35:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:17 smithi134 ceph-mon[31966]: pgmap v4842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:35:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:17 smithi189 ceph-mon[36790]: pgmap v4842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:35:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:17 smithi193 ceph-mon[37473]: pgmap v4842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:35:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:35:19.595+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:35:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:35:19.595+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:19.596 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:35:19.595+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:35:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:19 smithi134 ceph-mon[31966]: pgmap v4843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:35:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7660.213018 secs 2024-07-26T13:35:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:19 smithi189 ceph-mon[36790]: pgmap v4843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:35:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7660.213018 secs 2024-07-26T13:35:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:19 smithi193 ceph-mon[37473]: pgmap v4843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:35:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7660.213018 secs 2024-07-26T13:35:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:21 smithi134 ceph-mon[31966]: pgmap v4844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7679.638429 secs 2024-07-26T13:35:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:21 smithi189 ceph-mon[36790]: pgmap v4844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7679.638429 secs 2024-07-26T13:35:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:21 smithi193 ceph-mon[37473]: pgmap v4844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7679.638429 secs 2024-07-26T13:35:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:23 smithi134 ceph-mon[31966]: pgmap v4845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:23 smithi189 ceph-mon[36790]: pgmap v4845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:23 smithi193 ceph-mon[37473]: pgmap v4845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:24.102 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:35:24.101+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:35:24.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:35:24.101+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:24.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:35:24.101+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:35:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7665.213101 secs 2024-07-26T13:35:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:35:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7665.213101 secs 2024-07-26T13:35:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:35:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7665.213101 secs 2024-07-26T13:35:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:35:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:25 smithi189 ceph-mon[36790]: pgmap v4846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:35:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:35:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:35:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:25 smithi193 ceph-mon[37473]: pgmap v4846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:35:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:35:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:35:25.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:25 smithi134 ceph-mon[31966]: pgmap v4846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:35:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:35:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:35:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:26 smithi189 ceph-mon[36790]: pgmap v4847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:35:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:26 smithi189 ceph-mon[36790]: pgmap v4848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:35:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7684.638509 secs 2024-07-26T13:35:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:26 smithi193 ceph-mon[37473]: pgmap v4847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:35:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:26 smithi193 ceph-mon[37473]: pgmap v4848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:35:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7684.638509 secs 2024-07-26T13:35:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:26 smithi134 ceph-mon[31966]: pgmap v4847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:35:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:26 smithi134 ceph-mon[31966]: pgmap v4848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:35:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7684.638509 secs 2024-07-26T13:35:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:28 smithi189 ceph-mon[36790]: pgmap v4849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:35:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:28 smithi193 ceph-mon[37473]: pgmap v4849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:35:28.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:28 smithi134 ceph-mon[31966]: pgmap v4849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:35:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7670.213198 secs 2024-07-26T13:35:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7670.213198 secs 2024-07-26T13:35:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7670.213198 secs 2024-07-26T13:35:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:30 smithi193 ceph-mon[37473]: pgmap v4850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:35:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:30 smithi134 ceph-mon[31966]: pgmap v4850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:35:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:30 smithi189 ceph-mon[36790]: pgmap v4850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:35:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7689.638631 secs 2024-07-26T13:35:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7689.638631 secs 2024-07-26T13:35:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7689.638631 secs 2024-07-26T13:35:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:32 smithi193 ceph-mon[37473]: pgmap v4851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:35:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:32 smithi134 ceph-mon[31966]: pgmap v4851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:35:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:32 smithi189 ceph-mon[36790]: pgmap v4851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:35:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:35:34.599+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:35:34.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:35:34.599+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:34.601 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:35:34.599+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:35:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:34 smithi193 ceph-mon[37473]: pgmap v4852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:35:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7675.213320 secs 2024-07-26T13:35:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:34 smithi134 ceph-mon[31966]: pgmap v4852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:35:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7675.213320 secs 2024-07-26T13:35:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:34 smithi189 ceph-mon[36790]: pgmap v4852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:35:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7675.213320 secs 2024-07-26T13:35:36.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:36 smithi189 ceph-mon[36790]: pgmap v4853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:35:36.746 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7694.638743 secs 2024-07-26T13:35:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:36 smithi193 ceph-mon[37473]: pgmap v4853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:35:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7694.638743 secs 2024-07-26T13:35:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:36 smithi134 ceph-mon[31966]: pgmap v4853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:35:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7694.638743 secs 2024-07-26T13:35:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:38 smithi193 ceph-mon[37473]: pgmap v4854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:35:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:38 smithi134 ceph-mon[31966]: pgmap v4854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:35:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:38 smithi189 ceph-mon[36790]: pgmap v4854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:35:39.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:35:39.109+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:35:39.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:35:39.109+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:39.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:35:39.109+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:35:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7680.213440 secs 2024-07-26T13:35:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7680.213440 secs 2024-07-26T13:35:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7680.213440 secs 2024-07-26T13:35:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:40 smithi193 ceph-mon[37473]: pgmap v4855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:40 smithi134 ceph-mon[31966]: pgmap v4855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:40 smithi189 ceph-mon[36790]: pgmap v4855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7699.638886 secs 2024-07-26T13:35:41.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7699.638886 secs 2024-07-26T13:35:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7699.638886 secs 2024-07-26T13:35:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:42 smithi193 ceph-mon[37473]: pgmap v4856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:42 smithi134 ceph-mon[31966]: pgmap v4856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:42 smithi189 ceph-mon[36790]: pgmap v4856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:43.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:43 smithi134 ceph-mon[31966]: pgmap v4857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:35:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7685.213556 secs 2024-07-26T13:35:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:43 smithi189 ceph-mon[36790]: pgmap v4857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:35:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7685.213556 secs 2024-07-26T13:35:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:43 smithi193 ceph-mon[37473]: pgmap v4857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:35:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7685.213556 secs 2024-07-26T13:35:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:46 smithi134 ceph-mon[31966]: pgmap v4858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7704.639022 secs 2024-07-26T13:35:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:46 smithi189 ceph-mon[36790]: pgmap v4858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7704.639022 secs 2024-07-26T13:35:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:46 smithi193 ceph-mon[37473]: pgmap v4858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7704.639022 secs 2024-07-26T13:35:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:48 smithi134 ceph-mon[31966]: pgmap v4859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:48 smithi189 ceph-mon[36790]: pgmap v4859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:48 smithi193 ceph-mon[37473]: pgmap v4859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:49.616 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:35:49.615+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:35:49.616 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:35:49.615+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:35:49.616 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:35:49.615+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7690.213681 secs 2024-07-26T13:35:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7690.213681 secs 2024-07-26T13:35:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7690.213681 secs 2024-07-26T13:35:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:50 smithi134 ceph-mon[31966]: pgmap v4860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:50 smithi189 ceph-mon[36790]: pgmap v4860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:50 smithi193 ceph-mon[37473]: pgmap v4860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:35:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7709.639137 secs 2024-07-26T13:35:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7709.639137 secs 2024-07-26T13:35:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7709.639137 secs 2024-07-26T13:35:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:52 smithi134 ceph-mon[31966]: pgmap v4861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:52 smithi189 ceph-mon[36790]: pgmap v4861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:52 smithi193 ceph-mon[37473]: pgmap v4861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:54.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:35:54.112+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:35:54.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:35:54.112+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:35:54.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:35:54.112+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:35:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:54 smithi134 ceph-mon[31966]: pgmap v4862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:35:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7695.213807 secs 2024-07-26T13:35:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:54 smithi189 ceph-mon[36790]: pgmap v4862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:35:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7695.213807 secs 2024-07-26T13:35:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:54 smithi193 ceph-mon[37473]: pgmap v4862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:35:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7695.213807 secs 2024-07-26T13:35:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:56 smithi134 ceph-mon[31966]: pgmap v4863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7714.639261 secs 2024-07-26T13:35:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:56 smithi189 ceph-mon[36790]: pgmap v4863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7714.639261 secs 2024-07-26T13:35:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:56 smithi193 ceph-mon[37473]: pgmap v4863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:35:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7714.639261 secs 2024-07-26T13:35:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:58 smithi134 ceph-mon[31966]: pgmap v4864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:58 smithi189 ceph-mon[36790]: pgmap v4864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:58 smithi193 ceph-mon[37473]: pgmap v4864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:35:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:35:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7700.213966 secs 2024-07-26T13:35:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:35:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7700.213966 secs 2024-07-26T13:35:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:35:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7700.213966 secs 2024-07-26T13:36:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:00 smithi189 ceph-mon[36790]: pgmap v4865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:00 smithi193 ceph-mon[37473]: pgmap v4865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:00.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:00 smithi134 ceph-mon[31966]: pgmap v4865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7719.639361 secs 2024-07-26T13:36:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7719.639361 secs 2024-07-26T13:36:01.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7719.639361 secs 2024-07-26T13:36:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:02 smithi193 ceph-mon[37473]: pgmap v4866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:02 smithi134 ceph-mon[31966]: pgmap v4866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:02 smithi189 ceph-mon[36790]: pgmap v4866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:36:04.617+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:36:04.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:36:04.617+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:04.618 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:36:04.617+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:36:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:04 smithi134 ceph-mon[31966]: pgmap v4867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:36:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7705.214086 secs 2024-07-26T13:36:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:04 smithi193 ceph-mon[37473]: pgmap v4867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:36:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7705.214086 secs 2024-07-26T13:36:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:04 smithi189 ceph-mon[36790]: pgmap v4867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:36:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7705.214086 secs 2024-07-26T13:36:06.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:06 smithi189 ceph-mon[36790]: pgmap v4868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:06.746 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7724.639482 secs 2024-07-26T13:36:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:06 smithi193 ceph-mon[37473]: pgmap v4868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7724.639482 secs 2024-07-26T13:36:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:06 smithi134 ceph-mon[31966]: pgmap v4868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7724.639482 secs 2024-07-26T13:36:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:08 smithi193 ceph-mon[37473]: pgmap v4869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:08 smithi134 ceph-mon[31966]: pgmap v4869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:08 smithi189 ceph-mon[36790]: pgmap v4869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:09.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:36:09.127+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:36:09.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:36:09.128+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:36:09.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:36:09.128+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7710.214206 secs 2024-07-26T13:36:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7710.214206 secs 2024-07-26T13:36:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7710.214206 secs 2024-07-26T13:36:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:10 smithi193 ceph-mon[37473]: pgmap v4870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:10.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:10 smithi134 ceph-mon[31966]: pgmap v4870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:10 smithi189 ceph-mon[36790]: pgmap v4870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7729.639549 secs 2024-07-26T13:36:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7729.639549 secs 2024-07-26T13:36:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7729.639549 secs 2024-07-26T13:36:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:12 smithi193 ceph-mon[37473]: pgmap v4871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:12 smithi134 ceph-mon[31966]: pgmap v4871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:12 smithi189 ceph-mon[36790]: pgmap v4871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:14.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:13 smithi134 ceph-mon[31966]: pgmap v4872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:36:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7715.214339 secs 2024-07-26T13:36:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:13 smithi189 ceph-mon[36790]: pgmap v4872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:36:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7715.214339 secs 2024-07-26T13:36:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:13 smithi193 ceph-mon[37473]: pgmap v4872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:36:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7715.214339 secs 2024-07-26T13:36:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:16 smithi134 ceph-mon[31966]: pgmap v4873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7734.639673 secs 2024-07-26T13:36:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:16 smithi189 ceph-mon[36790]: pgmap v4873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7734.639673 secs 2024-07-26T13:36:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:16 smithi193 ceph-mon[37473]: pgmap v4873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7734.639673 secs 2024-07-26T13:36:18.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:18 smithi134 ceph-mon[31966]: pgmap v4874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:18 smithi189 ceph-mon[36790]: pgmap v4874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:18 smithi193 ceph-mon[37473]: pgmap v4874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:36:19.631+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:36:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:36:19.632+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:19.633 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:36:19.632+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:36:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7720.214474 secs 2024-07-26T13:36:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7720.214474 secs 2024-07-26T13:36:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7720.214474 secs 2024-07-26T13:36:20.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:20 smithi134 ceph-mon[31966]: pgmap v4875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:20 smithi189 ceph-mon[36790]: pgmap v4875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:20 smithi193 ceph-mon[37473]: pgmap v4875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7739.639794 secs 2024-07-26T13:36:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7739.639794 secs 2024-07-26T13:36:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7739.639794 secs 2024-07-26T13:36:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:22 smithi134 ceph-mon[31966]: pgmap v4876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:22 smithi189 ceph-mon[36790]: pgmap v4876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:22 smithi193 ceph-mon[37473]: pgmap v4876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:23.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:36:23.670+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:36:23.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:36:23.670+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:36:23.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:36:23.671+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:36:23.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:36:23.671+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:36:24.138 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:36:24.137+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:36:24.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:36:24.137+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:36:24.139 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:36:24.137+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:24 smithi189 ceph-mon[36790]: pgmap v4877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:36:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7725.214577 secs 2024-07-26T13:36:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:24 smithi193 ceph-mon[37473]: pgmap v4877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:36:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7725.214577 secs 2024-07-26T13:36:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:24 smithi134 ceph-mon[31966]: pgmap v4877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:36:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7725.214577 secs 2024-07-26T13:36:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:26 smithi189 ceph-mon[36790]: pgmap v4878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:36:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7744.639870 secs 2024-07-26T13:36:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:36:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:36:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:26 smithi193 ceph-mon[37473]: pgmap v4878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:36:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7744.639870 secs 2024-07-26T13:36:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:36:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:36:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:26 smithi134 ceph-mon[31966]: pgmap v4878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:36:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7744.639870 secs 2024-07-26T13:36:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:36:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:36:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:36:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:36:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:36:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:36:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:36:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:36:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:36:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:36:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:36:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:28 smithi193 ceph-mon[37473]: pgmap v4879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:36:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:28 smithi193 ceph-mon[37473]: pgmap v4880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:36:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:28 smithi134 ceph-mon[31966]: pgmap v4879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:36:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:28 smithi134 ceph-mon[31966]: pgmap v4880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:36:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:28 smithi189 ceph-mon[36790]: pgmap v4879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T13:36:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:28 smithi189 ceph-mon[36790]: pgmap v4880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:36:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7730.214703 secs 2024-07-26T13:36:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7730.214703 secs 2024-07-26T13:36:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7730.214703 secs 2024-07-26T13:36:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:30 smithi193 ceph-mon[37473]: pgmap v4881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:36:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:30 smithi134 ceph-mon[31966]: pgmap v4881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:36:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:30 smithi189 ceph-mon[36790]: pgmap v4881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:36:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7749.639986 secs 2024-07-26T13:36:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7749.639986 secs 2024-07-26T13:36:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7749.639986 secs 2024-07-26T13:36:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:32 smithi193 ceph-mon[37473]: pgmap v4882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:36:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:32 smithi134 ceph-mon[31966]: pgmap v4882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:36:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:32 smithi189 ceph-mon[36790]: pgmap v4882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:36:34.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:36:34.196+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:36:34.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:36:34.196+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T13:36:34.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:36:34.197+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T13:36:34.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:36:34.197+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:36:34.648 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:36:34.647+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:36:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:36:34.647+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:36:34.649 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:36:34.647+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:34 smithi193 ceph-mon[37473]: pgmap v4883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:36:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7735.214790 secs 2024-07-26T13:36:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:34 smithi134 ceph-mon[31966]: pgmap v4883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:36:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7735.214790 secs 2024-07-26T13:36:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:34 smithi189 ceph-mon[36790]: pgmap v4883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:36:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7735.214790 secs 2024-07-26T13:36:36.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:36 smithi189 ceph-mon[36790]: pgmap v4884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:36:36.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7754.640107 secs 2024-07-26T13:36:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:36 smithi193 ceph-mon[37473]: pgmap v4884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:36:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7754.640107 secs 2024-07-26T13:36:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:36 smithi134 ceph-mon[31966]: pgmap v4884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:36:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7754.640107 secs 2024-07-26T13:36:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:38 smithi193 ceph-mon[37473]: pgmap v4885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:36:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:38 smithi134 ceph-mon[31966]: pgmap v4885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:36:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:38 smithi189 ceph-mon[36790]: pgmap v4885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:36:39.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:36:39.139+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:36:39.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:36:39.139+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:39.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:36:39.139+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:36:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7740.214886 secs 2024-07-26T13:36:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7740.214886 secs 2024-07-26T13:36:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7740.214886 secs 2024-07-26T13:36:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:40 smithi193 ceph-mon[37473]: pgmap v4886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:40 smithi134 ceph-mon[31966]: pgmap v4886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:40 smithi189 ceph-mon[36790]: pgmap v4886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7759.640223 secs 2024-07-26T13:36:41.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7759.640223 secs 2024-07-26T13:36:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7759.640223 secs 2024-07-26T13:36:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:42 smithi193 ceph-mon[37473]: pgmap v4887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:42 smithi134 ceph-mon[31966]: pgmap v4887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:42 smithi189 ceph-mon[36790]: pgmap v4887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:43 smithi134 ceph-mon[31966]: pgmap v4888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:36:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7745.215013 secs 2024-07-26T13:36:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:43 smithi189 ceph-mon[36790]: pgmap v4888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:36:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7745.215013 secs 2024-07-26T13:36:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:43 smithi193 ceph-mon[37473]: pgmap v4888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:36:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7745.215013 secs 2024-07-26T13:36:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:46 smithi193 ceph-mon[37473]: pgmap v4889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7764.640346 secs 2024-07-26T13:36:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:46 smithi134 ceph-mon[31966]: pgmap v4889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7764.640346 secs 2024-07-26T13:36:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:46 smithi189 ceph-mon[36790]: pgmap v4889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7764.640346 secs 2024-07-26T13:36:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:48 smithi193 ceph-mon[37473]: pgmap v4890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:48 smithi134 ceph-mon[31966]: pgmap v4890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:48 smithi189 ceph-mon[36790]: pgmap v4890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7750.215133 secs 2024-07-26T13:36:49.657 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:36:49.657+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:36:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:36:49.657+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:49.658 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:36:49.657+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:36:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7750.215133 secs 2024-07-26T13:36:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7750.215133 secs 2024-07-26T13:36:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:50 smithi193 ceph-mon[37473]: pgmap v4891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:50 smithi134 ceph-mon[31966]: pgmap v4891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:50 smithi189 ceph-mon[36790]: pgmap v4891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:36:51.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7769.640411 secs 2024-07-26T13:36:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7769.640411 secs 2024-07-26T13:36:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7769.640411 secs 2024-07-26T13:36:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:52 smithi134 ceph-mon[31966]: pgmap v4892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:52 smithi189 ceph-mon[36790]: pgmap v4892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:52 smithi193 ceph-mon[37473]: pgmap v4892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:54.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:36:54.151+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:36:54.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:36:54.151+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:36:54.153 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:36:54.151+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:36:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:54 smithi134 ceph-mon[31966]: pgmap v4893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:36:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7755.215252 secs 2024-07-26T13:36:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:54 smithi189 ceph-mon[36790]: pgmap v4893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:36:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7755.215252 secs 2024-07-26T13:36:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:54 smithi193 ceph-mon[37473]: pgmap v4893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:36:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7755.215252 secs 2024-07-26T13:36:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:56 smithi134 ceph-mon[31966]: pgmap v4894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7774.640537 secs 2024-07-26T13:36:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:56 smithi189 ceph-mon[36790]: pgmap v4894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7774.640537 secs 2024-07-26T13:36:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:56 smithi193 ceph-mon[37473]: pgmap v4894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:36:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7774.640537 secs 2024-07-26T13:36:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:58 smithi134 ceph-mon[31966]: pgmap v4895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:58 smithi189 ceph-mon[36790]: pgmap v4895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:58 smithi193 ceph-mon[37473]: pgmap v4895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:36:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:36:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7760.215372 secs 2024-07-26T13:36:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:36:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7760.215372 secs 2024-07-26T13:36:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:36:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7760.215372 secs 2024-07-26T13:37:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:00 smithi134 ceph-mon[31966]: pgmap v4896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:00 smithi189 ceph-mon[36790]: pgmap v4896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:00 smithi193 ceph-mon[37473]: pgmap v4896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7779.640658 secs 2024-07-26T13:37:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7779.640658 secs 2024-07-26T13:37:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7779.640658 secs 2024-07-26T13:37:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:02 smithi134 ceph-mon[31966]: pgmap v4897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:02 smithi189 ceph-mon[36790]: pgmap v4897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:02 smithi193 ceph-mon[37473]: pgmap v4897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:04 smithi134 ceph-mon[31966]: pgmap v4898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7765.215491 secs 2024-07-26T13:37:04.672 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:37:04.671+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:37:04.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:37:04.671+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:37:04.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:37:04.671+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:04 smithi189 ceph-mon[36790]: pgmap v4898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7765.215491 secs 2024-07-26T13:37:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:04 smithi193 ceph-mon[37473]: pgmap v4898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7765.215491 secs 2024-07-26T13:37:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:06 smithi134 ceph-mon[31966]: pgmap v4899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7784.640776 secs 2024-07-26T13:37:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:06 smithi189 ceph-mon[36790]: pgmap v4899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7784.640776 secs 2024-07-26T13:37:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:06 smithi193 ceph-mon[37473]: pgmap v4899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7784.640776 secs 2024-07-26T13:37:08.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:08 smithi134 ceph-mon[31966]: pgmap v4900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:08 smithi189 ceph-mon[36790]: pgmap v4900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:08 smithi193 ceph-mon[37473]: pgmap v4900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:09.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:37:09.159+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:37:09.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:37:09.159+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:09.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:37:09.159+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:37:09.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7770.215647 secs 2024-07-26T13:37:09.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7770.215647 secs 2024-07-26T13:37:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7770.215647 secs 2024-07-26T13:37:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:10 smithi134 ceph-mon[31966]: pgmap v4901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:10 smithi189 ceph-mon[36790]: pgmap v4901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:10 smithi193 ceph-mon[37473]: pgmap v4901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7789.640859 secs 2024-07-26T13:37:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7789.640859 secs 2024-07-26T13:37:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7789.640859 secs 2024-07-26T13:37:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:12 smithi134 ceph-mon[31966]: pgmap v4902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:12 smithi189 ceph-mon[36790]: pgmap v4902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:12 smithi193 ceph-mon[37473]: pgmap v4902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:14 smithi134 ceph-mon[31966]: pgmap v4903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7775.215768 secs 2024-07-26T13:37:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:14 smithi189 ceph-mon[36790]: pgmap v4903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7775.215768 secs 2024-07-26T13:37:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:14 smithi193 ceph-mon[37473]: pgmap v4903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7775.215768 secs 2024-07-26T13:37:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:16 smithi134 ceph-mon[31966]: pgmap v4904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7794.640982 secs 2024-07-26T13:37:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:16 smithi189 ceph-mon[36790]: pgmap v4904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7794.640982 secs 2024-07-26T13:37:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:16 smithi193 ceph-mon[37473]: pgmap v4904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7794.640982 secs 2024-07-26T13:37:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:18 smithi134 ceph-mon[31966]: pgmap v4905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:37:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:18 smithi189 ceph-mon[36790]: pgmap v4905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:37:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:18 smithi193 ceph-mon[37473]: pgmap v4905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:37:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7780.215879 secs 2024-07-26T13:37:19.676 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:37:19.675+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:37:19.676 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:37:19.675+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:19.676 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:37:19.675+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:37:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7780.215879 secs 2024-07-26T13:37:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7780.215879 secs 2024-07-26T13:37:20.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:20 smithi134 ceph-mon[31966]: pgmap v4906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:20 smithi189 ceph-mon[36790]: pgmap v4906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:20 smithi193 ceph-mon[37473]: pgmap v4906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7799.641108 secs 2024-07-26T13:37:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7799.641108 secs 2024-07-26T13:37:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7799.641108 secs 2024-07-26T13:37:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:22 smithi189 ceph-mon[36790]: pgmap v4907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:22 smithi193 ceph-mon[37473]: pgmap v4907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:22 smithi134 ceph-mon[31966]: pgmap v4907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:24.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:37:24.175+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:37:24.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:37:24.175+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:24.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:37:24.175+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:37:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:24 smithi189 ceph-mon[36790]: pgmap v4908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7785.216018 secs 2024-07-26T13:37:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:24 smithi193 ceph-mon[37473]: pgmap v4908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7785.216018 secs 2024-07-26T13:37:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:24 smithi134 ceph-mon[31966]: pgmap v4908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7785.216018 secs 2024-07-26T13:37:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:26 smithi189 ceph-mon[36790]: pgmap v4909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7804.641243 secs 2024-07-26T13:37:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:26 smithi193 ceph-mon[37473]: pgmap v4909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7804.641243 secs 2024-07-26T13:37:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:26 smithi134 ceph-mon[31966]: pgmap v4909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7804.641243 secs 2024-07-26T13:37:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:37:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:37:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:37:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:28 smithi189 ceph-mon[36790]: pgmap v4910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:28 smithi193 ceph-mon[37473]: pgmap v4910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:28 smithi134 ceph-mon[31966]: pgmap v4910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:37:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7790.216089 secs 2024-07-26T13:37:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7790.216089 secs 2024-07-26T13:37:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7790.216089 secs 2024-07-26T13:37:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:30 smithi193 ceph-mon[37473]: pgmap v4911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:30 smithi134 ceph-mon[31966]: pgmap v4911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:30 smithi189 ceph-mon[36790]: pgmap v4911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:37:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7809.641366 secs 2024-07-26T13:37:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7809.641366 secs 2024-07-26T13:37:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7809.641366 secs 2024-07-26T13:37:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:32 smithi193 ceph-mon[37473]: pgmap v4912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:32 smithi134 ceph-mon[31966]: pgmap v4912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:32 smithi189 ceph-mon[36790]: pgmap v4912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:37:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:37:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:33 smithi193 ceph-mon[37473]: pgmap v4913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T13:37:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7795.216223 secs 2024-07-26T13:37:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:37:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:37:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:33 smithi134 ceph-mon[31966]: pgmap v4913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T13:37:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7795.216223 secs 2024-07-26T13:37:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:37:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:37:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:33 smithi189 ceph-mon[36790]: pgmap v4913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T13:37:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:37:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7795.216223 secs 2024-07-26T13:37:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:37:34.691+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:37:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:37:34.691+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:37:34.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:37:34.691+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:36.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:35 smithi134 ceph-mon[31966]: pgmap v4914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T13:37:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:35 smithi189 ceph-mon[36790]: pgmap v4914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T13:37:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:35 smithi193 ceph-mon[37473]: pgmap v4914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T13:37:37.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7814.641428 secs 2024-07-26T13:37:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7814.641428 secs 2024-07-26T13:37:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7814.641428 secs 2024-07-26T13:37:38.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:37 smithi134 ceph-mon[31966]: pgmap v4915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T13:37:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:37 smithi189 ceph-mon[36790]: pgmap v4915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T13:37:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:37 smithi193 ceph-mon[37473]: pgmap v4915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T13:37:39.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7800.216382 secs 2024-07-26T13:37:39.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7800.216382 secs 2024-07-26T13:37:39.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:37:39.185+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:37:39.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:37:39.185+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:39.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:37:39.185+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:37:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7800.216382 secs 2024-07-26T13:37:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:39 smithi134 ceph-mon[31966]: pgmap v4916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T13:37:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:39 smithi189 ceph-mon[36790]: pgmap v4916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T13:37:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:39 smithi193 ceph-mon[37473]: pgmap v4916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T13:37:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:41 smithi134 ceph-mon[31966]: pgmap v4917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:37:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7819.641549 secs 2024-07-26T13:37:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:41 smithi189 ceph-mon[36790]: pgmap v4917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:37:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7819.641549 secs 2024-07-26T13:37:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:41 smithi193 ceph-mon[37473]: pgmap v4917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:37:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7819.641549 secs 2024-07-26T13:37:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:43 smithi134 ceph-mon[31966]: pgmap v4918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T13:37:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7805.216546 secs 2024-07-26T13:37:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:43 smithi189 ceph-mon[36790]: pgmap v4918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T13:37:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7805.216546 secs 2024-07-26T13:37:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:43 smithi193 ceph-mon[37473]: pgmap v4918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 9 op/s 2024-07-26T13:37:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7805.216546 secs 2024-07-26T13:37:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:45 smithi134 ceph-mon[31966]: pgmap v4919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:37:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:45 smithi189 ceph-mon[36790]: pgmap v4919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:37:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:45 smithi193 ceph-mon[37473]: pgmap v4919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:37:47.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7824.641668 secs 2024-07-26T13:37:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7824.641668 secs 2024-07-26T13:37:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7824.641668 secs 2024-07-26T13:37:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:47 smithi134 ceph-mon[31966]: pgmap v4920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:47 smithi189 ceph-mon[36790]: pgmap v4920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:47 smithi193 ceph-mon[37473]: pgmap v4920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:49.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7810.216684 secs 2024-07-26T13:37:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7810.216684 secs 2024-07-26T13:37:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7810.216684 secs 2024-07-26T13:37:49.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:37:49.701+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:37:49.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:37:49.701+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:49.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:37:49.701+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:37:50.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:49 smithi134 ceph-mon[31966]: pgmap v4921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:49 smithi189 ceph-mon[36790]: pgmap v4921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:49 smithi193 ceph-mon[37473]: pgmap v4921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:51 smithi134 ceph-mon[31966]: pgmap v4922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7829.641787 secs 2024-07-26T13:37:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:51 smithi189 ceph-mon[36790]: pgmap v4922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7829.641787 secs 2024-07-26T13:37:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:51 smithi193 ceph-mon[37473]: pgmap v4922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7829.641787 secs 2024-07-26T13:37:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:53 smithi134 ceph-mon[31966]: pgmap v4923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7815.216813 secs 2024-07-26T13:37:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:53 smithi189 ceph-mon[36790]: pgmap v4923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7815.216813 secs 2024-07-26T13:37:54.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:37:54.201+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:37:54.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:37:54.201+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:37:54.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:37:54.201+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:37:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:53 smithi193 ceph-mon[37473]: pgmap v4923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:37:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7815.216813 secs 2024-07-26T13:37:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:55 smithi134 ceph-mon[31966]: pgmap v4924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:37:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:55 smithi189 ceph-mon[36790]: pgmap v4924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:37:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:55 smithi193 ceph-mon[37473]: pgmap v4924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:37:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7834.641925 secs 2024-07-26T13:37:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7834.641925 secs 2024-07-26T13:37:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7834.641925 secs 2024-07-26T13:37:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:57 smithi134 ceph-mon[31966]: pgmap v4925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:57 smithi189 ceph-mon[36790]: pgmap v4925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:57 smithi193 ceph-mon[37473]: pgmap v4925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:37:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7820.216972 secs 2024-07-26T13:37:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7820.216972 secs 2024-07-26T13:37:59.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7820.216972 secs 2024-07-26T13:38:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:37:59 smithi189 ceph-mon[36790]: pgmap v4926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:37:59 smithi193 ceph-mon[37473]: pgmap v4926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:37:59 smithi134 ceph-mon[31966]: pgmap v4926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:01 smithi189 ceph-mon[36790]: pgmap v4927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7839.642040 secs 2024-07-26T13:38:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:01 smithi193 ceph-mon[37473]: pgmap v4927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7839.642040 secs 2024-07-26T13:38:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:01 smithi134 ceph-mon[31966]: pgmap v4927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7839.642040 secs 2024-07-26T13:38:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:03 smithi189 ceph-mon[36790]: pgmap v4928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7825.217014 secs 2024-07-26T13:38:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:03 smithi193 ceph-mon[37473]: pgmap v4928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7825.217014 secs 2024-07-26T13:38:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:03 smithi134 ceph-mon[31966]: pgmap v4928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7825.217014 secs 2024-07-26T13:38:04.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:38:04.712+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:38:04.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:38:04.712+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:04.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:38:04.712+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:38:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:05 smithi193 ceph-mon[37473]: pgmap v4929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:38:06.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:05 smithi134 ceph-mon[31966]: pgmap v4929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:38:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:05 smithi189 ceph-mon[36790]: pgmap v4929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:38:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7844.642169 secs 2024-07-26T13:38:07.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7844.642169 secs 2024-07-26T13:38:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7844.642169 secs 2024-07-26T13:38:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:07 smithi193 ceph-mon[37473]: pgmap v4930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:08.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:07 smithi134 ceph-mon[31966]: pgmap v4930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:07 smithi189 ceph-mon[36790]: pgmap v4930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:09.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:38:09.213+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:38:09.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:38:09.213+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:09.215 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:38:09.213+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:38:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7830.217131 secs 2024-07-26T13:38:09.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7830.217131 secs 2024-07-26T13:38:09.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7830.217131 secs 2024-07-26T13:38:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:09 smithi193 ceph-mon[37473]: pgmap v4931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:10.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:09 smithi134 ceph-mon[31966]: pgmap v4931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:09 smithi189 ceph-mon[36790]: pgmap v4931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:11 smithi193 ceph-mon[37473]: pgmap v4932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7849.642284 secs 2024-07-26T13:38:12.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:11 smithi134 ceph-mon[31966]: pgmap v4932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7849.642284 secs 2024-07-26T13:38:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:11 smithi189 ceph-mon[36790]: pgmap v4932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7849.642284 secs 2024-07-26T13:38:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:13 smithi193 ceph-mon[37473]: pgmap v4933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7835.217226 secs 2024-07-26T13:38:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:13 smithi134 ceph-mon[31966]: pgmap v4933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7835.217226 secs 2024-07-26T13:38:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:13 smithi189 ceph-mon[36790]: pgmap v4933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7835.217226 secs 2024-07-26T13:38:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:16 smithi193 ceph-mon[37473]: pgmap v4934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:38:16.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:16 smithi134 ceph-mon[31966]: pgmap v4934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:38:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:16 smithi189 ceph-mon[36790]: pgmap v4934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:38:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7854.642399 secs 2024-07-26T13:38:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7854.642399 secs 2024-07-26T13:38:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7854.642399 secs 2024-07-26T13:38:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:18 smithi193 ceph-mon[37473]: pgmap v4935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:18.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:18 smithi134 ceph-mon[31966]: pgmap v4935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:18 smithi189 ceph-mon[36790]: pgmap v4935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7840.217367 secs 2024-07-26T13:38:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7840.217367 secs 2024-07-26T13:38:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7840.217367 secs 2024-07-26T13:38:19.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:38:19.717+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:38:19.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:38:19.717+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:19.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:38:19.717+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:38:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:20 smithi134 ceph-mon[31966]: pgmap v4936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:20 smithi189 ceph-mon[36790]: pgmap v4936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:20 smithi193 ceph-mon[37473]: pgmap v4936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:21.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7859.642517 secs 2024-07-26T13:38:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7859.642517 secs 2024-07-26T13:38:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7859.642517 secs 2024-07-26T13:38:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:22 smithi134 ceph-mon[31966]: pgmap v4937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:22 smithi189 ceph-mon[36790]: pgmap v4937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:22 smithi193 ceph-mon[37473]: pgmap v4937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:24.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:38:24.228+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:38:24.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:38:24.228+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:24.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:38:24.228+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:38:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:24 smithi134 ceph-mon[31966]: pgmap v4938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7845.217511 secs 2024-07-26T13:38:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:24 smithi189 ceph-mon[36790]: pgmap v4938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7845.217511 secs 2024-07-26T13:38:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:24 smithi193 ceph-mon[37473]: pgmap v4938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7845.217511 secs 2024-07-26T13:38:26.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:26 smithi134 ceph-mon[31966]: pgmap v4939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:38:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7864.642639 secs 2024-07-26T13:38:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:26 smithi189 ceph-mon[36790]: pgmap v4939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:38:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7864.642639 secs 2024-07-26T13:38:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:26 smithi193 ceph-mon[37473]: pgmap v4939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:38:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7864.642639 secs 2024-07-26T13:38:28.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:28 smithi134 ceph-mon[31966]: pgmap v4940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:28 smithi189 ceph-mon[36790]: pgmap v4940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:28 smithi193 ceph-mon[37473]: pgmap v4940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7850.217625 secs 2024-07-26T13:38:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7850.217625 secs 2024-07-26T13:38:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7850.217625 secs 2024-07-26T13:38:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:30 smithi134 ceph-mon[31966]: pgmap v4941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:30 smithi189 ceph-mon[36790]: pgmap v4941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:30 smithi193 ceph-mon[37473]: pgmap v4941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:31.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7869.642764 secs 2024-07-26T13:38:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7869.642764 secs 2024-07-26T13:38:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7869.642764 secs 2024-07-26T13:38:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:32 smithi134 ceph-mon[31966]: pgmap v4942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:32 smithi189 ceph-mon[36790]: pgmap v4942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:32 smithi193 ceph-mon[37473]: pgmap v4942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:38:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:38:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:38:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:34 smithi134 ceph-mon[31966]: pgmap v4943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7855.217756 secs 2024-07-26T13:38:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:38:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:38:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:38:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:34 smithi189 ceph-mon[36790]: pgmap v4943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7855.217756 secs 2024-07-26T13:38:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:38:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:38:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:38:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:34 smithi193 ceph-mon[37473]: pgmap v4943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7855.217756 secs 2024-07-26T13:38:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:38:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:38:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:38:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:38:34.727+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:38:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:38:34.727+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:38:34.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:38:34.727+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:35 smithi189 ceph-mon[36790]: pgmap v4944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:38:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:35 smithi189 ceph-mon[36790]: pgmap v4945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:38:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:35 smithi193 ceph-mon[37473]: pgmap v4944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:38:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:35 smithi193 ceph-mon[37473]: pgmap v4945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:38:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:35 smithi134 ceph-mon[31966]: pgmap v4944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:38:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:35 smithi134 ceph-mon[31966]: pgmap v4945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:38:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7874.642845 secs 2024-07-26T13:38:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7874.642845 secs 2024-07-26T13:38:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7874.642845 secs 2024-07-26T13:38:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:37 smithi193 ceph-mon[37473]: pgmap v4946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:38:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:37 smithi134 ceph-mon[31966]: pgmap v4946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:38:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:37 smithi189 ceph-mon[36790]: pgmap v4946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:38:39.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:38:39.235+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:38:39.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:38:39.235+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:38:39.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:38:39.235+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:39.473 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:39 smithi134 ceph-mon[31966]: pgmap v4947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:38:39.474 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7860.217887 secs 2024-07-26T13:38:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:39 smithi193 ceph-mon[37473]: pgmap v4947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:38:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7860.217887 secs 2024-07-26T13:38:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:39 smithi189 ceph-mon[36790]: pgmap v4947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:38:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7860.217887 secs 2024-07-26T13:38:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:41 smithi193 ceph-mon[37473]: pgmap v4948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:38:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7879.642964 secs 2024-07-26T13:38:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:41 smithi134 ceph-mon[31966]: pgmap v4948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:38:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7879.642964 secs 2024-07-26T13:38:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:41 smithi189 ceph-mon[36790]: pgmap v4948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:38:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7879.642964 secs 2024-07-26T13:38:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:43 smithi193 ceph-mon[37473]: pgmap v4949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:38:43.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:43 smithi134 ceph-mon[31966]: pgmap v4949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:38:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:43 smithi189 ceph-mon[36790]: pgmap v4949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:38:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7865.217989 secs 2024-07-26T13:38:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7865.217989 secs 2024-07-26T13:38:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7865.217989 secs 2024-07-26T13:38:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:45 smithi193 ceph-mon[37473]: pgmap v4950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:38:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:45 smithi134 ceph-mon[31966]: pgmap v4950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:38:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:45 smithi189 ceph-mon[36790]: pgmap v4950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:38:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7884.643080 secs 2024-07-26T13:38:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7884.643080 secs 2024-07-26T13:38:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7884.643080 secs 2024-07-26T13:38:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:47 smithi193 ceph-mon[37473]: pgmap v4951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:47 smithi134 ceph-mon[31966]: pgmap v4951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:47 smithi189 ceph-mon[36790]: pgmap v4951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:49 smithi193 ceph-mon[37473]: pgmap v4952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7870.218112 secs 2024-07-26T13:38:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:49 smithi134 ceph-mon[31966]: pgmap v4952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7870.218112 secs 2024-07-26T13:38:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:49 smithi189 ceph-mon[36790]: pgmap v4952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7870.218112 secs 2024-07-26T13:38:49.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:38:49.733+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:38:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:38:49.733+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:38:49.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:38:49.733+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:51 smithi193 ceph-mon[37473]: pgmap v4953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:38:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7889.643206 secs 2024-07-26T13:38:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:51 smithi134 ceph-mon[31966]: pgmap v4953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:38:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7889.643206 secs 2024-07-26T13:38:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:51 smithi189 ceph-mon[36790]: pgmap v4953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:38:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7889.643206 secs 2024-07-26T13:38:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:53 smithi193 ceph-mon[37473]: pgmap v4954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:53.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:53 smithi134 ceph-mon[31966]: pgmap v4954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:53 smithi189 ceph-mon[36790]: pgmap v4954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:54.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:38:54.236+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:38:54.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:38:54.236+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:38:54.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:38:54.236+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:38:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7875.218180 secs 2024-07-26T13:38:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7875.218180 secs 2024-07-26T13:38:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7875.218180 secs 2024-07-26T13:38:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:55 smithi193 ceph-mon[37473]: pgmap v4955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:55.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:55 smithi134 ceph-mon[31966]: pgmap v4955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:55 smithi189 ceph-mon[36790]: pgmap v4955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7894.643299 secs 2024-07-26T13:38:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7894.643299 secs 2024-07-26T13:38:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7894.643299 secs 2024-07-26T13:38:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:57 smithi134 ceph-mon[31966]: pgmap v4956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:57 smithi189 ceph-mon[36790]: pgmap v4956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:57 smithi193 ceph-mon[37473]: pgmap v4956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:38:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:59 smithi134 ceph-mon[31966]: pgmap v4957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:38:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7880.218302 secs 2024-07-26T13:38:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:59 smithi189 ceph-mon[36790]: pgmap v4957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:38:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7880.218302 secs 2024-07-26T13:38:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:59 smithi193 ceph-mon[37473]: pgmap v4957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:38:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:38:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7880.218302 secs 2024-07-26T13:39:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:01 smithi134 ceph-mon[31966]: pgmap v4958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:39:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7899.643375 secs 2024-07-26T13:39:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:01 smithi189 ceph-mon[36790]: pgmap v4958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:39:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7899.643375 secs 2024-07-26T13:39:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:01 smithi193 ceph-mon[37473]: pgmap v4958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:39:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7899.643375 secs 2024-07-26T13:39:03.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:03 smithi134 ceph-mon[31966]: pgmap v4959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:03 smithi189 ceph-mon[36790]: pgmap v4959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:03 smithi193 ceph-mon[37473]: pgmap v4959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7885.218418 secs 2024-07-26T13:39:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7885.218418 secs 2024-07-26T13:39:04.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:04.744+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:39:04.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:04.744+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:39:04.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:04.744+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7885.218418 secs 2024-07-26T13:39:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:05 smithi134 ceph-mon[31966]: pgmap v4960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:05 smithi189 ceph-mon[36790]: pgmap v4960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:05 smithi193 ceph-mon[37473]: pgmap v4960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7904.643462 secs 2024-07-26T13:39:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7904.643462 secs 2024-07-26T13:39:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7904.643462 secs 2024-07-26T13:39:07.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:07 smithi189 ceph-mon[36790]: pgmap v4961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:07 smithi134 ceph-mon[31966]: pgmap v4961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:07 smithi193 ceph-mon[37473]: pgmap v4961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:09.253 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:09.251+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:39:09.253 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:09.252+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:39:09.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:09.252+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:09 smithi134 ceph-mon[31966]: pgmap v4962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7890.218566 secs 2024-07-26T13:39:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:09 smithi189 ceph-mon[36790]: pgmap v4962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7890.218566 secs 2024-07-26T13:39:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:09 smithi193 ceph-mon[37473]: pgmap v4962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7890.218566 secs 2024-07-26T13:39:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:11 smithi134 ceph-mon[31966]: pgmap v4963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:39:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7909.643588 secs 2024-07-26T13:39:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:11 smithi189 ceph-mon[36790]: pgmap v4963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:39:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7909.643588 secs 2024-07-26T13:39:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:11 smithi193 ceph-mon[37473]: pgmap v4963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:39:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7909.643588 secs 2024-07-26T13:39:13.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:13 smithi134 ceph-mon[31966]: pgmap v4964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:13 smithi189 ceph-mon[36790]: pgmap v4964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:13.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:13 smithi193 ceph-mon[37473]: pgmap v4964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7895.218693 secs 2024-07-26T13:39:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7895.218693 secs 2024-07-26T13:39:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7895.218693 secs 2024-07-26T13:39:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:15 smithi134 ceph-mon[31966]: pgmap v4965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:15 smithi189 ceph-mon[36790]: pgmap v4965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:15 smithi193 ceph-mon[37473]: pgmap v4965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7914.643703 secs 2024-07-26T13:39:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7914.643703 secs 2024-07-26T13:39:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7914.643703 secs 2024-07-26T13:39:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:17 smithi134 ceph-mon[31966]: pgmap v4966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:17 smithi189 ceph-mon[36790]: pgmap v4966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:17 smithi193 ceph-mon[37473]: pgmap v4966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:19 smithi134 ceph-mon[31966]: pgmap v4967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7900.218806 secs 2024-07-26T13:39:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:19 smithi189 ceph-mon[36790]: pgmap v4967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7900.218806 secs 2024-07-26T13:39:19.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:19.755+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:39:19.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:19.755+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:39:19.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:19.755+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:19 smithi193 ceph-mon[37473]: pgmap v4967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7900.218806 secs 2024-07-26T13:39:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:21 smithi134 ceph-mon[31966]: pgmap v4968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:39:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7919.643837 secs 2024-07-26T13:39:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:21 smithi189 ceph-mon[36790]: pgmap v4968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:39:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7919.643837 secs 2024-07-26T13:39:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:21 smithi193 ceph-mon[37473]: pgmap v4968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:39:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7919.643837 secs 2024-07-26T13:39:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:23 smithi134 ceph-mon[31966]: pgmap v4969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:23 smithi189 ceph-mon[36790]: pgmap v4969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:23.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:23.702+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T13:39:23.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:23.702+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T13:39:23.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:23.702+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T13:39:23.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:23.702+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T13:39:23.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:23.703+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T13:39:23.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:23.703+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T13:39:23.704 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:23.703+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T13:39:23.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:23.703+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T13:39:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:23 smithi193 ceph-mon[37473]: pgmap v4969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:24.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:24.267+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:39:24.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:24.267+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:24.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:24.267+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:39:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7905.218979 secs 2024-07-26T13:39:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7905.218979 secs 2024-07-26T13:39:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7905.218979 secs 2024-07-26T13:39:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:25 smithi134 ceph-mon[31966]: pgmap v4970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:25 smithi189 ceph-mon[36790]: pgmap v4970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:25 smithi193 ceph-mon[37473]: pgmap v4970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7924.643974 secs 2024-07-26T13:39:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7924.643974 secs 2024-07-26T13:39:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7924.643974 secs 2024-07-26T13:39:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:27 smithi134 ceph-mon[31966]: pgmap v4971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:27 smithi189 ceph-mon[36790]: pgmap v4971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:27 smithi193 ceph-mon[37473]: pgmap v4971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:29 smithi189 ceph-mon[36790]: pgmap v4972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7910.219098 secs 2024-07-26T13:39:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:29 smithi193 ceph-mon[37473]: pgmap v4972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7910.219098 secs 2024-07-26T13:39:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:29 smithi134 ceph-mon[31966]: pgmap v4972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7910.219098 secs 2024-07-26T13:39:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:31 smithi189 ceph-mon[36790]: pgmap v4973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:39:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7929.644110 secs 2024-07-26T13:39:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:31 smithi193 ceph-mon[37473]: pgmap v4973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:39:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7929.644110 secs 2024-07-26T13:39:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:31 smithi134 ceph-mon[31966]: pgmap v4973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:39:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7929.644110 secs 2024-07-26T13:39:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:33 smithi189 ceph-mon[36790]: pgmap v4974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:33 smithi193 ceph-mon[37473]: pgmap v4974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:33.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:33 smithi134 ceph-mon[31966]: pgmap v4974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:34.227+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:39:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:34.227+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T13:39:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:34.228+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T13:39:34.229 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:34.228+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T13:39:34.230 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:34.228+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:39:34.230 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:34.228+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T13:39:34.230 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:34.228+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:39:34.230 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:34.228+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T13:39:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:34.770+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:39:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:34.771+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:39:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:34.771+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7915.219216 secs 2024-07-26T13:39:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:39:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7915.219216 secs 2024-07-26T13:39:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:39:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7915.219216 secs 2024-07-26T13:39:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:39:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:35 smithi193 ceph-mon[37473]: pgmap v4975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:39:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:39:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:39:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:35 smithi134 ceph-mon[31966]: pgmap v4975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:39:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:39:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:39:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:35 smithi189 ceph-mon[36790]: pgmap v4975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:39:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:39:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:39:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:36 smithi193 ceph-mon[37473]: pgmap v4976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:39:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:36 smithi193 ceph-mon[37473]: pgmap v4977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:39:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7934.644223 secs 2024-07-26T13:39:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:36 smithi134 ceph-mon[31966]: pgmap v4976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:39:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:36 smithi134 ceph-mon[31966]: pgmap v4977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:39:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7934.644223 secs 2024-07-26T13:39:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:36 smithi189 ceph-mon[36790]: pgmap v4976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:39:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:36 smithi189 ceph-mon[36790]: pgmap v4977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:39:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7934.644223 secs 2024-07-26T13:39:38.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:38 smithi189 ceph-mon[36790]: pgmap v4978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:39:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:38 smithi193 ceph-mon[37473]: pgmap v4978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:39:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:38 smithi134 ceph-mon[31966]: pgmap v4978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:39:39.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:39.283+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:39:39.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:39.283+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:39.284 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:39.283+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:39:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7920.219291 secs 2024-07-26T13:39:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7920.219291 secs 2024-07-26T13:39:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7920.219291 secs 2024-07-26T13:39:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:40 smithi193 ceph-mon[37473]: pgmap v4979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:39:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:40 smithi134 ceph-mon[31966]: pgmap v4979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:39:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:40 smithi189 ceph-mon[36790]: pgmap v4979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:39:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7939.644350 secs 2024-07-26T13:39:41.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7939.644350 secs 2024-07-26T13:39:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7939.644350 secs 2024-07-26T13:39:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:42 smithi134 ceph-mon[31966]: pgmap v4980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:39:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:42 smithi193 ceph-mon[37473]: pgmap v4980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:39:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:42 smithi189 ceph-mon[36790]: pgmap v4980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:39:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:44 smithi193 ceph-mon[37473]: pgmap v4981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:39:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7925.219377 secs 2024-07-26T13:39:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:44 smithi134 ceph-mon[31966]: pgmap v4981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:39:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7925.219377 secs 2024-07-26T13:39:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:44 smithi189 ceph-mon[36790]: pgmap v4981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:39:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7925.219377 secs 2024-07-26T13:39:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:46 smithi193 ceph-mon[37473]: pgmap v4982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:39:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7944.644477 secs 2024-07-26T13:39:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:46 smithi134 ceph-mon[31966]: pgmap v4982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:39:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7944.644477 secs 2024-07-26T13:39:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:46 smithi189 ceph-mon[36790]: pgmap v4982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:39:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7944.644477 secs 2024-07-26T13:39:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:48 smithi193 ceph-mon[37473]: pgmap v4983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:48 smithi134 ceph-mon[31966]: pgmap v4983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:48 smithi189 ceph-mon[36790]: pgmap v4983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:49.786+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:39:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:49.786+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:49.788 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:39:49.786+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:39:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7930.219500 secs 2024-07-26T13:39:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7930.219500 secs 2024-07-26T13:39:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7930.219500 secs 2024-07-26T13:39:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:50 smithi134 ceph-mon[31966]: pgmap v4984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:50 smithi189 ceph-mon[36790]: pgmap v4984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:50 smithi193 ceph-mon[37473]: pgmap v4984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:51.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7949.644579 secs 2024-07-26T13:39:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7949.644579 secs 2024-07-26T13:39:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7949.644579 secs 2024-07-26T13:39:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:52 smithi134 ceph-mon[31966]: pgmap v4985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:52 smithi189 ceph-mon[36790]: pgmap v4985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:52 smithi193 ceph-mon[37473]: pgmap v4985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:53.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7935.219627 secs 2024-07-26T13:39:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7935.219627 secs 2024-07-26T13:39:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7935.219627 secs 2024-07-26T13:39:54.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:54.296+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:39:54.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:54.296+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:39:54.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:39:54.296+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:39:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:54 smithi134 ceph-mon[31966]: pgmap v4986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:54 smithi189 ceph-mon[36790]: pgmap v4986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:54 smithi193 ceph-mon[37473]: pgmap v4986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:39:55.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:55 smithi134 ceph-mon[31966]: pgmap v4987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:55 smithi189 ceph-mon[36790]: pgmap v4987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:55 smithi193 ceph-mon[37473]: pgmap v4987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:39:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7954.644729 secs 2024-07-26T13:39:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7954.644729 secs 2024-07-26T13:39:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7954.644729 secs 2024-07-26T13:39:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:57 smithi134 ceph-mon[31966]: pgmap v4988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:57 smithi189 ceph-mon[36790]: pgmap v4988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:57 smithi193 ceph-mon[37473]: pgmap v4988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:39:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7940.219771 secs 2024-07-26T13:39:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7940.219771 secs 2024-07-26T13:39:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7940.219771 secs 2024-07-26T13:39:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:39:59 smithi134 ceph-mon[31966]: pgmap v4989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:39:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:39:59 smithi189 ceph-mon[36790]: pgmap v4989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:39:59 smithi193 ceph-mon[37473]: pgmap v4989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:00.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:40:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:40:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:40:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7959.644852 secs 2024-07-26T13:40:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:01 smithi134 ceph-mon[31966]: pgmap v4990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7959.644852 secs 2024-07-26T13:40:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:01 smithi189 ceph-mon[36790]: pgmap v4990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7959.644852 secs 2024-07-26T13:40:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:01 smithi193 ceph-mon[37473]: pgmap v4990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:04 smithi193 ceph-mon[37473]: pgmap v4991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:40:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7945.219923 secs 2024-07-26T13:40:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:04 smithi134 ceph-mon[31966]: pgmap v4991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:40:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7945.219923 secs 2024-07-26T13:40:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:04 smithi189 ceph-mon[36790]: pgmap v4991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:40:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7945.219923 secs 2024-07-26T13:40:04.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:40:04.791+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:40:04.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:40:04.791+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:40:04.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:40:04.791+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:06 smithi193 ceph-mon[37473]: pgmap v4992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:40:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7964.644961 secs 2024-07-26T13:40:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:06 smithi134 ceph-mon[31966]: pgmap v4992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:40:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7964.644961 secs 2024-07-26T13:40:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:06 smithi189 ceph-mon[36790]: pgmap v4992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:40:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7964.644961 secs 2024-07-26T13:40:08.495 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:08 smithi189 ceph-mon[36790]: pgmap v4993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T13:40:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:08 smithi193 ceph-mon[37473]: pgmap v4993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T13:40:08.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:08 smithi134 ceph-mon[31966]: pgmap v4993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T13:40:09.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:40:09.311+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:40:09.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:40:09.311+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:09.313 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:40:09.311+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:40:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7950.220067 secs 2024-07-26T13:40:09.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7950.220067 secs 2024-07-26T13:40:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7950.220067 secs 2024-07-26T13:40:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:10 smithi193 ceph-mon[37473]: pgmap v4994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:10 smithi134 ceph-mon[31966]: pgmap v4994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:10 smithi189 ceph-mon[36790]: pgmap v4994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7969.645081 secs 2024-07-26T13:40:11.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7969.645081 secs 2024-07-26T13:40:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7969.645081 secs 2024-07-26T13:40:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:12 smithi193 ceph-mon[37473]: pgmap v4995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:40:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:12 smithi134 ceph-mon[31966]: pgmap v4995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:40:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:12 smithi189 ceph-mon[36790]: pgmap v4995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:40:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:13 smithi134 ceph-mon[31966]: pgmap v4996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:40:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7955.220218 secs 2024-07-26T13:40:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:13 smithi189 ceph-mon[36790]: pgmap v4996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:40:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7955.220218 secs 2024-07-26T13:40:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:13 smithi193 ceph-mon[37473]: pgmap v4996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:40:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7955.220218 secs 2024-07-26T13:40:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:16 smithi193 ceph-mon[37473]: pgmap v4997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:40:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7974.645216 secs 2024-07-26T13:40:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:16 smithi134 ceph-mon[31966]: pgmap v4997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:40:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7974.645216 secs 2024-07-26T13:40:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:16 smithi189 ceph-mon[36790]: pgmap v4997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:40:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7974.645216 secs 2024-07-26T13:40:18.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:18 smithi193 ceph-mon[37473]: pgmap v4998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:40:18.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:18 smithi134 ceph-mon[31966]: pgmap v4998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:40:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:18 smithi189 ceph-mon[36790]: pgmap v4998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:40:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7960.220339 secs 2024-07-26T13:40:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7960.220339 secs 2024-07-26T13:40:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7960.220339 secs 2024-07-26T13:40:19.793 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:40:19.792+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:40:19.793 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:40:19.792+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:19.794 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:40:19.792+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:40:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:20 smithi193 ceph-mon[37473]: pgmap v4999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:20.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:20 smithi134 ceph-mon[31966]: pgmap v4999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:20 smithi189 ceph-mon[36790]: pgmap v4999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7979.645352 secs 2024-07-26T13:40:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7979.645352 secs 2024-07-26T13:40:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7979.645352 secs 2024-07-26T13:40:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:22 smithi193 ceph-mon[37473]: pgmap v5000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:22 smithi134 ceph-mon[31966]: pgmap v5000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:22 smithi189 ceph-mon[36790]: pgmap v5000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:24.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:40:24.326+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:40:24.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:40:24.326+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:24.329 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:40:24.327+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:40:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:24 smithi193 ceph-mon[37473]: pgmap v5001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:40:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7965.220403 secs 2024-07-26T13:40:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:24 smithi134 ceph-mon[31966]: pgmap v5001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:40:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7965.220403 secs 2024-07-26T13:40:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:24 smithi189 ceph-mon[36790]: pgmap v5001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:40:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7965.220403 secs 2024-07-26T13:40:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:26 smithi134 ceph-mon[31966]: pgmap v5002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7984.645466 secs 2024-07-26T13:40:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:26 smithi189 ceph-mon[36790]: pgmap v5002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7984.645466 secs 2024-07-26T13:40:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:26 smithi193 ceph-mon[37473]: pgmap v5002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7984.645466 secs 2024-07-26T13:40:28.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:28 smithi134 ceph-mon[31966]: pgmap v5003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:40:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:28 smithi189 ceph-mon[36790]: pgmap v5003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:40:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:28 smithi193 ceph-mon[37473]: pgmap v5003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:40:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7970.220560 secs 2024-07-26T13:40:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7970.220560 secs 2024-07-26T13:40:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7970.220560 secs 2024-07-26T13:40:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:30 smithi134 ceph-mon[31966]: pgmap v5004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:30 smithi189 ceph-mon[36790]: pgmap v5004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:30 smithi193 ceph-mon[37473]: pgmap v5004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:40:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7989.645543 secs 2024-07-26T13:40:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7989.645543 secs 2024-07-26T13:40:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7989.645543 secs 2024-07-26T13:40:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:32 smithi134 ceph-mon[31966]: pgmap v5005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:32 smithi189 ceph-mon[36790]: pgmap v5005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:32 smithi193 ceph-mon[37473]: pgmap v5005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:34 smithi134 ceph-mon[31966]: pgmap v5006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:40:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7975.220685 secs 2024-07-26T13:40:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:34 smithi189 ceph-mon[36790]: pgmap v5006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:40:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7975.220685 secs 2024-07-26T13:40:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:34 smithi193 ceph-mon[37473]: pgmap v5006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:40:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7975.220685 secs 2024-07-26T13:40:34.809 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:40:34.807+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:40:34.809 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:40:34.808+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:40:34.809 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:40:34.808+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:36 smithi134 ceph-mon[31966]: pgmap v5007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:40:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7994.645616 secs 2024-07-26T13:40:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:36 smithi189 ceph-mon[36790]: pgmap v5007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:40:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7994.645616 secs 2024-07-26T13:40:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:36 smithi193 ceph-mon[37473]: pgmap v5007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:40:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7994.645616 secs 2024-07-26T13:40:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:40:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:40:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:37 smithi134 ceph-mon[31966]: pgmap v5008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:40:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:37 smithi134 ceph-mon[31966]: pgmap v5009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:40:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:40:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:40:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:40:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:37 smithi189 ceph-mon[36790]: pgmap v5008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:40:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:37 smithi189 ceph-mon[36790]: pgmap v5009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:40:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:40:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:40:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:40:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:37 smithi193 ceph-mon[37473]: pgmap v5008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:40:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:37 smithi193 ceph-mon[37473]: pgmap v5009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:40:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:40:39.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:40:39.342+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:40:39.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:40:39.342+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:39.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:40:39.342+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:40:39.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:39 smithi134 ceph-mon[31966]: pgmap v5010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:39.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7980.220787 secs 2024-07-26T13:40:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:39 smithi189 ceph-mon[36790]: pgmap v5010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7980.220787 secs 2024-07-26T13:40:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:39 smithi193 ceph-mon[37473]: pgmap v5010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7980.220787 secs 2024-07-26T13:40:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:41 smithi189 ceph-mon[36790]: pgmap v5011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 7999.645711 secs 2024-07-26T13:40:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:41 smithi193 ceph-mon[37473]: pgmap v5011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 7999.645711 secs 2024-07-26T13:40:41.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:41 smithi134 ceph-mon[31966]: pgmap v5011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:40:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 7999.645711 secs 2024-07-26T13:40:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:43 smithi189 ceph-mon[36790]: pgmap v5012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:43 smithi193 ceph-mon[37473]: pgmap v5012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:43.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:43 smithi134 ceph-mon[31966]: pgmap v5012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7985.220927 secs 2024-07-26T13:40:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7985.220927 secs 2024-07-26T13:40:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7985.220927 secs 2024-07-26T13:40:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:45 smithi193 ceph-mon[37473]: pgmap v5013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:40:45.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:45 smithi134 ceph-mon[31966]: pgmap v5013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:40:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:45 smithi189 ceph-mon[36790]: pgmap v5013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:40:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8004.645803 secs 2024-07-26T13:40:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8004.645803 secs 2024-07-26T13:40:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8004.645803 secs 2024-07-26T13:40:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:47 smithi193 ceph-mon[37473]: pgmap v5014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:40:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:47 smithi134 ceph-mon[31966]: pgmap v5014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:40:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:47 smithi189 ceph-mon[36790]: pgmap v5014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:40:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:49 smithi193 ceph-mon[37473]: pgmap v5015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:40:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7990.221068 secs 2024-07-26T13:40:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:40:49.812+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:40:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:40:49.812+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:40:49.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:40:49.812+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:49 smithi134 ceph-mon[31966]: pgmap v5015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:40:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7990.221068 secs 2024-07-26T13:40:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:49 smithi189 ceph-mon[36790]: pgmap v5015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:40:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7990.221068 secs 2024-07-26T13:40:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:51 smithi193 ceph-mon[37473]: pgmap v5016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8009.645866 secs 2024-07-26T13:40:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:51 smithi134 ceph-mon[31966]: pgmap v5016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8009.645866 secs 2024-07-26T13:40:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:51 smithi189 ceph-mon[36790]: pgmap v5016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:40:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8009.645866 secs 2024-07-26T13:40:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:53 smithi193 ceph-mon[37473]: pgmap v5017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:53.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:53 smithi134 ceph-mon[31966]: pgmap v5017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:53 smithi189 ceph-mon[36790]: pgmap v5017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:40:54.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:40:54.344+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:40:54.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:40:54.344+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:40:54.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:40:54.344+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:40:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 7995.221196 secs 2024-07-26T13:40:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 7995.221196 secs 2024-07-26T13:40:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 7995.221196 secs 2024-07-26T13:40:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:55 smithi193 ceph-mon[37473]: pgmap v5018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:40:55.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:55 smithi134 ceph-mon[31966]: pgmap v5018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:40:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:55 smithi189 ceph-mon[36790]: pgmap v5018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:40:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8014.645971 secs 2024-07-26T13:40:56.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8014.645971 secs 2024-07-26T13:40:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8014.645971 secs 2024-07-26T13:40:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:57 smithi193 ceph-mon[37473]: pgmap v5019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:40:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:57 smithi134 ceph-mon[31966]: pgmap v5019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:40:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:57 smithi189 ceph-mon[36790]: pgmap v5019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:40:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:59 smithi193 ceph-mon[37473]: pgmap v5020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:40:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:40:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8000.221340 secs 2024-07-26T13:40:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:59 smithi134 ceph-mon[31966]: pgmap v5020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:40:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:40:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8000.221340 secs 2024-07-26T13:40:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:59 smithi189 ceph-mon[36790]: pgmap v5020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:40:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:40:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8000.221340 secs 2024-07-26T13:41:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:01 smithi134 ceph-mon[31966]: pgmap v5021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:41:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8019.646092 secs 2024-07-26T13:41:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:01 smithi189 ceph-mon[36790]: pgmap v5021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:41:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8019.646092 secs 2024-07-26T13:41:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:01 smithi193 ceph-mon[37473]: pgmap v5021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:41:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8019.646092 secs 2024-07-26T13:41:03.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:03 smithi134 ceph-mon[31966]: pgmap v5022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:41:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:03 smithi189 ceph-mon[36790]: pgmap v5022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:41:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:03 smithi193 ceph-mon[37473]: pgmap v5022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T13:41:04.828 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:41:04.828+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:41:04.829 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:41:04.828+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:04.829 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:41:04.828+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:41:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8005.221420 secs 2024-07-26T13:41:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8005.221420 secs 2024-07-26T13:41:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8005.221420 secs 2024-07-26T13:41:05.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:05 smithi134 ceph-mon[31966]: pgmap v5023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:41:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:05 smithi189 ceph-mon[36790]: pgmap v5023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:41:06.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:05 smithi193 ceph-mon[37473]: pgmap v5023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:41:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8024.646228 secs 2024-07-26T13:41:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8024.646228 secs 2024-07-26T13:41:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8024.646228 secs 2024-07-26T13:41:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:07 smithi134 ceph-mon[31966]: pgmap v5024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:07 smithi189 ceph-mon[36790]: pgmap v5024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:07 smithi193 ceph-mon[37473]: pgmap v5024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8010.221576 secs 2024-07-26T13:41:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8010.221576 secs 2024-07-26T13:41:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8010.221576 secs 2024-07-26T13:41:09.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:41:09.359+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:41:09.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:41:09.360+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:41:09.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:41:09.360+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:09 smithi134 ceph-mon[31966]: pgmap v5025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:09.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:09 smithi189 ceph-mon[36790]: pgmap v5025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:09.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:09 smithi193 ceph-mon[37473]: pgmap v5025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:10 smithi134 ceph-mon[31966]: pgmap v5026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:10 smithi189 ceph-mon[36790]: pgmap v5026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:10 smithi193 ceph-mon[37473]: pgmap v5026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8029.646379 secs 2024-07-26T13:41:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8029.646379 secs 2024-07-26T13:41:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8029.646379 secs 2024-07-26T13:41:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:12 smithi134 ceph-mon[31966]: pgmap v5027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:12 smithi189 ceph-mon[36790]: pgmap v5027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:12 smithi193 ceph-mon[37473]: pgmap v5027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:13.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8015.221715 secs 2024-07-26T13:41:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8015.221715 secs 2024-07-26T13:41:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8015.221715 secs 2024-07-26T13:41:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:14 smithi134 ceph-mon[31966]: pgmap v5028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:14 smithi189 ceph-mon[36790]: pgmap v5028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:14 smithi193 ceph-mon[37473]: pgmap v5028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:41:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8034.646496 secs 2024-07-26T13:41:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8034.646496 secs 2024-07-26T13:41:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8034.646496 secs 2024-07-26T13:41:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:17 smithi134 ceph-mon[31966]: pgmap v5029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:17 smithi189 ceph-mon[36790]: pgmap v5029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:17 smithi193 ceph-mon[37473]: pgmap v5029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:19 smithi193 ceph-mon[37473]: pgmap v5030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8020.221852 secs 2024-07-26T13:41:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:41:19.843+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:41:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:41:19.843+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:41:19.844 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:41:19.843+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:19 smithi134 ceph-mon[31966]: pgmap v5030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8020.221852 secs 2024-07-26T13:41:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:19 smithi189 ceph-mon[36790]: pgmap v5030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8020.221852 secs 2024-07-26T13:41:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:21 smithi193 ceph-mon[37473]: pgmap v5031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8039.646622 secs 2024-07-26T13:41:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:21 smithi134 ceph-mon[31966]: pgmap v5031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8039.646622 secs 2024-07-26T13:41:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:21 smithi189 ceph-mon[36790]: pgmap v5031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8039.646622 secs 2024-07-26T13:41:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:23 smithi193 ceph-mon[37473]: pgmap v5032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:23.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:23 smithi134 ceph-mon[31966]: pgmap v5032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:23 smithi189 ceph-mon[36790]: pgmap v5032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:24.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:41:24.375+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:41:24.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:41:24.375+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:24.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:41:24.375+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:41:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8025.221987 secs 2024-07-26T13:41:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8025.221987 secs 2024-07-26T13:41:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8025.221987 secs 2024-07-26T13:41:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:25 smithi193 ceph-mon[37473]: pgmap v5033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:25.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:25 smithi134 ceph-mon[31966]: pgmap v5033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:25 smithi189 ceph-mon[36790]: pgmap v5033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8044.646742 secs 2024-07-26T13:41:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8044.646742 secs 2024-07-26T13:41:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8044.646742 secs 2024-07-26T13:41:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:27 smithi193 ceph-mon[37473]: pgmap v5034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:27 smithi134 ceph-mon[31966]: pgmap v5034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:27 smithi189 ceph-mon[36790]: pgmap v5034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:29 smithi193 ceph-mon[37473]: pgmap v5035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8030.222054 secs 2024-07-26T13:41:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:29 smithi134 ceph-mon[31966]: pgmap v5035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8030.222054 secs 2024-07-26T13:41:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:29 smithi189 ceph-mon[36790]: pgmap v5035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8030.222054 secs 2024-07-26T13:41:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:31 smithi193 ceph-mon[37473]: pgmap v5036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8049.646857 secs 2024-07-26T13:41:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:31 smithi134 ceph-mon[31966]: pgmap v5036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8049.646857 secs 2024-07-26T13:41:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:31 smithi189 ceph-mon[36790]: pgmap v5036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8049.646857 secs 2024-07-26T13:41:33.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:33 smithi134 ceph-mon[31966]: pgmap v5037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:33 smithi189 ceph-mon[36790]: pgmap v5037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:33 smithi193 ceph-mon[37473]: pgmap v5037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:41:34.859+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:41:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:41:34.859+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:41:34.860 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:41:34.859+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8035.222186 secs 2024-07-26T13:41:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8035.222186 secs 2024-07-26T13:41:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8035.222186 secs 2024-07-26T13:41:35.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:35 smithi134 ceph-mon[31966]: pgmap v5038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:35 smithi189 ceph-mon[36790]: pgmap v5038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:35 smithi193 ceph-mon[37473]: pgmap v5038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8054.646983 secs 2024-07-26T13:41:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8054.646983 secs 2024-07-26T13:41:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8054.646983 secs 2024-07-26T13:41:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:37 smithi134 ceph-mon[31966]: pgmap v5039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:41:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:37 smithi189 ceph-mon[36790]: pgmap v5039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:41:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:37 smithi193 ceph-mon[37473]: pgmap v5039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:41:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:41:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:41:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:41:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8040.222310 secs 2024-07-26T13:41:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:41:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:41:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:41:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8040.222310 secs 2024-07-26T13:41:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:41:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:41:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:41:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8040.222310 secs 2024-07-26T13:41:39.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:41:39.391+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:41:39.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:41:39.391+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:39.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:41:39.391+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:41:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:39 smithi134 ceph-mon[31966]: pgmap v5040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:41:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:39 smithi134 ceph-mon[31966]: pgmap v5041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:41:39.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:39 smithi189 ceph-mon[36790]: pgmap v5040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:41:39.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:39 smithi189 ceph-mon[36790]: pgmap v5041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:41:39.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:39 smithi193 ceph-mon[37473]: pgmap v5040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:41:39.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:39 smithi193 ceph-mon[37473]: pgmap v5041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:41:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:41 smithi134 ceph-mon[31966]: pgmap v5042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:41:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8059.647103 secs 2024-07-26T13:41:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:41 smithi189 ceph-mon[36790]: pgmap v5042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:41:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8059.647103 secs 2024-07-26T13:41:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:41 smithi193 ceph-mon[37473]: pgmap v5042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:41:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8059.647103 secs 2024-07-26T13:41:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:42 smithi134 ceph-mon[31966]: pgmap v5043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:41:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:42 smithi189 ceph-mon[36790]: pgmap v5043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:41:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:42 smithi193 ceph-mon[37473]: pgmap v5043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:41:43.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8045.222458 secs 2024-07-26T13:41:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8045.222458 secs 2024-07-26T13:41:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8045.222458 secs 2024-07-26T13:41:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:44 smithi134 ceph-mon[31966]: pgmap v5044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:41:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:44 smithi189 ceph-mon[36790]: pgmap v5044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:41:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:44 smithi193 ceph-mon[37473]: pgmap v5044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:41:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:46 smithi134 ceph-mon[31966]: pgmap v5045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:41:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8064.647237 secs 2024-07-26T13:41:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:46 smithi189 ceph-mon[36790]: pgmap v5045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:41:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8064.647237 secs 2024-07-26T13:41:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:46 smithi193 ceph-mon[37473]: pgmap v5045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:41:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8064.647237 secs 2024-07-26T13:41:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:48 smithi134 ceph-mon[31966]: pgmap v5046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:41:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8050.222594 secs 2024-07-26T13:41:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:48 smithi189 ceph-mon[36790]: pgmap v5046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:41:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8050.222594 secs 2024-07-26T13:41:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:48 smithi193 ceph-mon[37473]: pgmap v5046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:41:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8050.222594 secs 2024-07-26T13:41:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:41:49.872+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:41:49.873 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:41:49.872+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:41:49.874 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:41:49.872+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:50 smithi134 ceph-mon[31966]: pgmap v5047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:50 smithi189 ceph-mon[36790]: pgmap v5047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:50 smithi193 ceph-mon[37473]: pgmap v5047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:41:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8069.647396 secs 2024-07-26T13:41:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8069.647396 secs 2024-07-26T13:41:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8069.647396 secs 2024-07-26T13:41:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:52 smithi189 ceph-mon[36790]: pgmap v5048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:52 smithi193 ceph-mon[37473]: pgmap v5048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:53.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:52 smithi134 ceph-mon[31966]: pgmap v5048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8055.222672 secs 2024-07-26T13:41:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8055.222672 secs 2024-07-26T13:41:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8055.222672 secs 2024-07-26T13:41:54.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:41:54.406+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:41:54.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:41:54.406+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:41:54.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:41:54.406+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:41:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:54 smithi189 ceph-mon[36790]: pgmap v5049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:54 smithi193 ceph-mon[37473]: pgmap v5049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:54 smithi134 ceph-mon[31966]: pgmap v5049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:56 smithi193 ceph-mon[37473]: pgmap v5050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8074.647476 secs 2024-07-26T13:41:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:56 smithi134 ceph-mon[31966]: pgmap v5050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8074.647476 secs 2024-07-26T13:41:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:56 smithi189 ceph-mon[36790]: pgmap v5050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:41:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8074.647476 secs 2024-07-26T13:41:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:58 smithi193 ceph-mon[37473]: pgmap v5051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:41:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8060.222754 secs 2024-07-26T13:41:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:58 smithi134 ceph-mon[31966]: pgmap v5051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:41:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8060.222754 secs 2024-07-26T13:41:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:58 smithi189 ceph-mon[36790]: pgmap v5051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:41:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:41:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8060.222754 secs 2024-07-26T13:42:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:00 smithi193 ceph-mon[37473]: pgmap v5052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:42:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:00 smithi134 ceph-mon[31966]: pgmap v5052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:42:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:00 smithi189 ceph-mon[36790]: pgmap v5052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:42:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8079.647576 secs 2024-07-26T13:42:02.120 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8079.647576 secs 2024-07-26T13:42:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8079.647576 secs 2024-07-26T13:42:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:02 smithi193 ceph-mon[37473]: pgmap v5053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:03.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:02 smithi134 ceph-mon[31966]: pgmap v5053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:02 smithi189 ceph-mon[36790]: pgmap v5053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8065.222853 secs 2024-07-26T13:42:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8065.222853 secs 2024-07-26T13:42:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8065.222853 secs 2024-07-26T13:42:04.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:04.876+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:42:04.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:04.876+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:42:04.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:04.876+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:04 smithi193 ceph-mon[37473]: pgmap v5054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:04 smithi134 ceph-mon[31966]: pgmap v5054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:04 smithi189 ceph-mon[36790]: pgmap v5054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:06 smithi193 ceph-mon[37473]: pgmap v5055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8084.647657 secs 2024-07-26T13:42:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:06 smithi134 ceph-mon[31966]: pgmap v5055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8084.647657 secs 2024-07-26T13:42:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:06 smithi189 ceph-mon[36790]: pgmap v5055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8084.647657 secs 2024-07-26T13:42:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:08 smithi193 ceph-mon[37473]: pgmap v5056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8070.222934 secs 2024-07-26T13:42:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:08 smithi134 ceph-mon[31966]: pgmap v5056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8070.222934 secs 2024-07-26T13:42:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:08 smithi189 ceph-mon[36790]: pgmap v5056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8070.222934 secs 2024-07-26T13:42:09.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:09.414+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:42:09.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:09.414+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:42:09.416 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:09.414+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:10 smithi193 ceph-mon[37473]: pgmap v5057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:42:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:10 smithi134 ceph-mon[31966]: pgmap v5057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:42:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:10 smithi189 ceph-mon[36790]: pgmap v5057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:42:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8089.647781 secs 2024-07-26T13:42:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8089.647781 secs 2024-07-26T13:42:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8089.647781 secs 2024-07-26T13:42:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:12 smithi193 ceph-mon[37473]: pgmap v5058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:12 smithi134 ceph-mon[31966]: pgmap v5058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:12 smithi189 ceph-mon[36790]: pgmap v5058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8075.223070 secs 2024-07-26T13:42:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8075.223070 secs 2024-07-26T13:42:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8075.223070 secs 2024-07-26T13:42:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:14 smithi134 ceph-mon[31966]: pgmap v5059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:14 smithi189 ceph-mon[36790]: pgmap v5059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:14 smithi193 ceph-mon[37473]: pgmap v5059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:17.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:16 smithi134 ceph-mon[31966]: pgmap v5060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:17.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8094.647864 secs 2024-07-26T13:42:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:16 smithi189 ceph-mon[36790]: pgmap v5060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8094.647864 secs 2024-07-26T13:42:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:16 smithi193 ceph-mon[37473]: pgmap v5060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8094.647864 secs 2024-07-26T13:42:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:18 smithi134 ceph-mon[31966]: pgmap v5061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8080.223174 secs 2024-07-26T13:42:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:18 smithi189 ceph-mon[36790]: pgmap v5061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8080.223174 secs 2024-07-26T13:42:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:18 smithi193 ceph-mon[37473]: pgmap v5061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8080.223174 secs 2024-07-26T13:42:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:19.892+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:42:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:19.892+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:42:19.893 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:19.892+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:21.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:20 smithi134 ceph-mon[31966]: pgmap v5062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:42:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:20 smithi189 ceph-mon[36790]: pgmap v5062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:42:21.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:20 smithi193 ceph-mon[37473]: pgmap v5062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:42:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8099.647990 secs 2024-07-26T13:42:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8099.647990 secs 2024-07-26T13:42:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8099.647990 secs 2024-07-26T13:42:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:22.313+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 msgr2=0x7f9b300c1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T13:42:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:22.313+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 msgr2=0x7f9b300c1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:42:22.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:22.313+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 0x7f9b300c1940 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009770 tx=0x7f9b38015000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:42:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:22.313+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 0x7f9b300c1940 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009770 tx=0x7f9b38015000 comp rx=0 tx=0).stop 2024-07-26T13:42:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:22.313+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 msgr2=0x7f9b300c1940 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T13:42:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:22.313+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 msgr2=0x7f9b300c1940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:42:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:22.313+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 0x7f9b300c1940 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:42:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:22.313+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:42:22.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:22.313+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:42:22.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:22.315+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b3805d040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:42:23.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:22 smithi134 ceph-mon[31966]: pgmap v5063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:22 smithi189 ceph-mon[36790]: pgmap v5063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:22 smithi193 ceph-mon[37473]: pgmap v5063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:23.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:23.732+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T13:42:23.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:23.732+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T13:42:23.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:23.732+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T13:42:23.734 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:23.732+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T13:42:24.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8085.223262 secs 2024-07-26T13:42:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8085.223262 secs 2024-07-26T13:42:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8085.223262 secs 2024-07-26T13:42:24.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:24.416+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:42:24.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:24.416+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:42:24.417 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:24.416+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:24 smithi134 ceph-mon[31966]: pgmap v5064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:24 smithi189 ceph-mon[36790]: pgmap v5064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:24 smithi193 ceph-mon[37473]: pgmap v5064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:26 smithi134 ceph-mon[31966]: pgmap v5065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8104.648128 secs 2024-07-26T13:42:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:26 smithi189 ceph-mon[36790]: pgmap v5065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8104.648128 secs 2024-07-26T13:42:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:26 smithi193 ceph-mon[37473]: pgmap v5065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8104.648128 secs 2024-07-26T13:42:29.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:28 smithi134 ceph-mon[31966]: pgmap v5066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8090.223384 secs 2024-07-26T13:42:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:28 smithi189 ceph-mon[36790]: pgmap v5066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8090.223384 secs 2024-07-26T13:42:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:28 smithi193 ceph-mon[37473]: pgmap v5066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8090.223384 secs 2024-07-26T13:42:31.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:30 smithi134 ceph-mon[31966]: pgmap v5067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:42:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:30 smithi189 ceph-mon[36790]: pgmap v5067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:42:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:30 smithi193 ceph-mon[37473]: pgmap v5067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:42:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8109.648224 secs 2024-07-26T13:42:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8109.648224 secs 2024-07-26T13:42:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8109.648224 secs 2024-07-26T13:42:32.867 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:32.866+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 msgr2=0x7f21ac0c0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900001077) for more than 900000000 us, fault. 2024-07-26T13:42:32.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:32.866+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 0x7f21ac0c0640 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f2198005e40 tx=0x7f21b805c000 comp rx=0 tx=0).stop 2024-07-26T13:42:32.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:32.866+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 msgr2=0x7f21ac0c0640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:42:32.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:32.866+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 0x7f21ac0c0640 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:42:32.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:32.866+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:42:32.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:32.867+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:42:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:32.869+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f21980063a0 tx=0x7f21b80ba000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:42:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:32 smithi134 ceph-mon[31966]: pgmap v5068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:32 smithi189 ceph-mon[36790]: pgmap v5068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:32 smithi193 ceph-mon[37473]: pgmap v5068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8095.223516 secs 2024-07-26T13:42:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8095.223516 secs 2024-07-26T13:42:34.260 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:34.259+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:42:34.260 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:34.259+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T13:42:34.261 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:34.260+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:42:34.261 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:34.260+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006630 con 0x55a08e2d94d0 2024-07-26T13:42:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8095.223516 secs 2024-07-26T13:42:34.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:34.906+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:42:34.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:34.906+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:42:34.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:34.906+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:34 smithi134 ceph-mon[31966]: pgmap v5069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:34 smithi189 ceph-mon[36790]: pgmap v5069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:34 smithi193 ceph-mon[37473]: pgmap v5069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:36 smithi134 ceph-mon[31966]: pgmap v5070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8114.648346 secs 2024-07-26T13:42:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:36 smithi189 ceph-mon[36790]: pgmap v5070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8114.648346 secs 2024-07-26T13:42:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:36 smithi193 ceph-mon[37473]: pgmap v5070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8114.648346 secs 2024-07-26T13:42:38.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:42:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:42:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:42:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:38 smithi134 ceph-mon[31966]: pgmap v5071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8100.223665 secs 2024-07-26T13:42:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:38 smithi189 ceph-mon[36790]: pgmap v5071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8100.223665 secs 2024-07-26T13:42:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:38 smithi193 ceph-mon[37473]: pgmap v5071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8100.223665 secs 2024-07-26T13:42:39.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:39.431+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:42:39.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:39.431+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:39.432 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:39.431+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:42:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:40 smithi134 ceph-mon[31966]: pgmap v5072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:42:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:40 smithi189 ceph-mon[36790]: pgmap v5072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:42:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:40 smithi193 ceph-mon[37473]: pgmap v5072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:42:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8119.648465 secs 2024-07-26T13:42:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8119.648465 secs 2024-07-26T13:42:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8119.648465 secs 2024-07-26T13:42:43.159 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:42 smithi134 ceph-mon[31966]: pgmap v5073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:42 smithi189 ceph-mon[36790]: pgmap v5073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:42 smithi193 ceph-mon[37473]: pgmap v5073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8105.223792 secs 2024-07-26T13:42:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8105.223792 secs 2024-07-26T13:42:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8105.223792 secs 2024-07-26T13:42:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:44 smithi193 ceph-mon[37473]: pgmap v5074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:42:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:42:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:42:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:42:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:44 smithi193 ceph-mon[37473]: pgmap v5075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:42:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:44 smithi193 ceph-mon[37473]: pgmap v5076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:42:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:42:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:44 smithi134 ceph-mon[31966]: pgmap v5074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:42:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:42:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:42:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:42:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:44 smithi134 ceph-mon[31966]: pgmap v5075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:42:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:44 smithi134 ceph-mon[31966]: pgmap v5076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:42:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:42:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:44 smithi189 ceph-mon[36790]: pgmap v5074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:42:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:42:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:42:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:42:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:44 smithi189 ceph-mon[36790]: pgmap v5075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:42:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:44 smithi189 ceph-mon[36790]: pgmap v5076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:42:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:42:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8124.648555 secs 2024-07-26T13:42:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:47 smithi134 ceph-mon[31966]: pgmap v5077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T13:42:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8124.648555 secs 2024-07-26T13:42:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:47 smithi189 ceph-mon[36790]: pgmap v5077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T13:42:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8124.648555 secs 2024-07-26T13:42:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:47 smithi193 ceph-mon[37473]: pgmap v5077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T13:42:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:49 smithi134 ceph-mon[31966]: pgmap v5078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T13:42:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8110.223920 secs 2024-07-26T13:42:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:49 smithi189 ceph-mon[36790]: pgmap v5078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T13:42:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8110.223920 secs 2024-07-26T13:42:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:49 smithi193 ceph-mon[37473]: pgmap v5078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T13:42:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8110.223920 secs 2024-07-26T13:42:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:49.920+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:42:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:49.920+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:42:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:42:49.920+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:51.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:51 smithi134 ceph-mon[31966]: pgmap v5079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:42:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8129.648696 secs 2024-07-26T13:42:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:51 smithi189 ceph-mon[36790]: pgmap v5079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:42:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8129.648696 secs 2024-07-26T13:42:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:51 smithi193 ceph-mon[37473]: pgmap v5079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:42:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8129.648696 secs 2024-07-26T13:42:53.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:53 smithi134 ceph-mon[31966]: pgmap v5080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T13:42:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:53 smithi189 ceph-mon[36790]: pgmap v5080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T13:42:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:53 smithi193 ceph-mon[37473]: pgmap v5080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T13:42:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8115.224019 secs 2024-07-26T13:42:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8115.224019 secs 2024-07-26T13:42:54.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:54.446+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:42:54.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:54.446+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:42:54.449 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:42:54.446+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:42:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8115.224019 secs 2024-07-26T13:42:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:55 smithi134 ceph-mon[31966]: pgmap v5081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:42:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:55 smithi189 ceph-mon[36790]: pgmap v5081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:42:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:55 smithi193 ceph-mon[37473]: pgmap v5081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:42:56.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8134.648835 secs 2024-07-26T13:42:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8134.648835 secs 2024-07-26T13:42:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8134.648835 secs 2024-07-26T13:42:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:57 smithi134 ceph-mon[31966]: pgmap v5082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:57 smithi189 ceph-mon[36790]: pgmap v5082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:57 smithi193 ceph-mon[37473]: pgmap v5082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:42:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:59 smithi134 ceph-mon[31966]: pgmap v5083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:42:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8120.224110 secs 2024-07-26T13:42:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:59 smithi189 ceph-mon[36790]: pgmap v5083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:42:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8120.224110 secs 2024-07-26T13:42:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:59 smithi193 ceph-mon[37473]: pgmap v5083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:42:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:42:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8120.224110 secs 2024-07-26T13:43:01.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:01 smithi134 ceph-mon[31966]: pgmap v5084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:43:01.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8139.648965 secs 2024-07-26T13:43:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:01 smithi189 ceph-mon[36790]: pgmap v5084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:43:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8139.648965 secs 2024-07-26T13:43:01.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:01 smithi193 ceph-mon[37473]: pgmap v5084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:43:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8139.648965 secs 2024-07-26T13:43:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:03 smithi134 ceph-mon[31966]: pgmap v5085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:03 smithi189 ceph-mon[36790]: pgmap v5085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:03.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:03 smithi193 ceph-mon[37473]: pgmap v5085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8125.224234 secs 2024-07-26T13:43:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8125.224234 secs 2024-07-26T13:43:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8125.224234 secs 2024-07-26T13:43:04.928 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:43:04.927+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:43:04.928 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:43:04.927+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:43:04.928 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:43:04.927+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:05.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:05 smithi134 ceph-mon[31966]: pgmap v5086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:05 smithi189 ceph-mon[36790]: pgmap v5086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:05 smithi193 ceph-mon[37473]: pgmap v5086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:06.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8144.649099 secs 2024-07-26T13:43:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8144.649099 secs 2024-07-26T13:43:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8144.649099 secs 2024-07-26T13:43:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:07 smithi134 ceph-mon[31966]: pgmap v5087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:07 smithi189 ceph-mon[36790]: pgmap v5087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:07 smithi193 ceph-mon[37473]: pgmap v5087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:09 smithi189 ceph-mon[36790]: pgmap v5088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8130.224351 secs 2024-07-26T13:43:09.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:43:09.462+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:43:09.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:43:09.462+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:43:09.464 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:43:09.462+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:09.464 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:09 smithi134 ceph-mon[31966]: pgmap v5088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:09.464 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8130.224351 secs 2024-07-26T13:43:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:09 smithi193 ceph-mon[37473]: pgmap v5088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8130.224351 secs 2024-07-26T13:43:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:11 smithi189 ceph-mon[36790]: pgmap v5089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:43:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8149.649221 secs 2024-07-26T13:43:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:11 smithi193 ceph-mon[37473]: pgmap v5089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:43:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8149.649221 secs 2024-07-26T13:43:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:11 smithi134 ceph-mon[31966]: pgmap v5089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:43:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8149.649221 secs 2024-07-26T13:43:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:13 smithi189 ceph-mon[36790]: pgmap v5090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:13 smithi193 ceph-mon[37473]: pgmap v5090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:13.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:13 smithi134 ceph-mon[31966]: pgmap v5090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8135.224478 secs 2024-07-26T13:43:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8135.224478 secs 2024-07-26T13:43:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8135.224478 secs 2024-07-26T13:43:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:15 smithi193 ceph-mon[37473]: pgmap v5091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:15 smithi134 ceph-mon[31966]: pgmap v5091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:15 smithi189 ceph-mon[36790]: pgmap v5091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8154.649337 secs 2024-07-26T13:43:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8154.649337 secs 2024-07-26T13:43:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8154.649337 secs 2024-07-26T13:43:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:17 smithi193 ceph-mon[37473]: pgmap v5092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:17 smithi134 ceph-mon[31966]: pgmap v5092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:17 smithi189 ceph-mon[36790]: pgmap v5092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:19 smithi193 ceph-mon[37473]: pgmap v5093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8140.224640 secs 2024-07-26T13:43:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:19 smithi134 ceph-mon[31966]: pgmap v5093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8140.224640 secs 2024-07-26T13:43:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:19 smithi189 ceph-mon[36790]: pgmap v5093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8140.224640 secs 2024-07-26T13:43:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:43:19.933+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:43:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:43:19.933+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:43:19.935 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:43:19.933+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:21 smithi193 ceph-mon[37473]: pgmap v5094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:43:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8159.649434 secs 2024-07-26T13:43:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:21 smithi134 ceph-mon[31966]: pgmap v5094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:43:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8159.649434 secs 2024-07-26T13:43:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:21 smithi189 ceph-mon[36790]: pgmap v5094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:43:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8159.649434 secs 2024-07-26T13:43:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:23 smithi193 ceph-mon[37473]: pgmap v5095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:23 smithi134 ceph-mon[31966]: pgmap v5095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:23 smithi189 ceph-mon[36790]: pgmap v5095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:24.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:43:24.477+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:43:24.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:43:24.477+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:24.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:43:24.477+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:43:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8145.224707 secs 2024-07-26T13:43:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8145.224707 secs 2024-07-26T13:43:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8145.224707 secs 2024-07-26T13:43:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:25 smithi193 ceph-mon[37473]: pgmap v5096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:25 smithi134 ceph-mon[31966]: pgmap v5096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:25 smithi189 ceph-mon[36790]: pgmap v5096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8164.649553 secs 2024-07-26T13:43:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8164.649553 secs 2024-07-26T13:43:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8164.649553 secs 2024-07-26T13:43:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:27 smithi193 ceph-mon[37473]: pgmap v5097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:27 smithi134 ceph-mon[31966]: pgmap v5097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:27 smithi189 ceph-mon[36790]: pgmap v5097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:29 smithi193 ceph-mon[37473]: pgmap v5098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8150.224853 secs 2024-07-26T13:43:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:29 smithi134 ceph-mon[31966]: pgmap v5098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8150.224853 secs 2024-07-26T13:43:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:29 smithi189 ceph-mon[36790]: pgmap v5098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8150.224853 secs 2024-07-26T13:43:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:31 smithi193 ceph-mon[37473]: pgmap v5099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:43:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8169.649671 secs 2024-07-26T13:43:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:31 smithi134 ceph-mon[31966]: pgmap v5099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:43:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8169.649671 secs 2024-07-26T13:43:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:31 smithi189 ceph-mon[36790]: pgmap v5099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:43:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8169.649671 secs 2024-07-26T13:43:33.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:33 smithi134 ceph-mon[31966]: pgmap v5100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:33 smithi189 ceph-mon[36790]: pgmap v5100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:33 smithi193 ceph-mon[37473]: pgmap v5100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8155.224985 secs 2024-07-26T13:43:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8155.224985 secs 2024-07-26T13:43:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8155.224985 secs 2024-07-26T13:43:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:43:34.936+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:43:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:43:34.936+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:43:34.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:43:34.936+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:35 smithi134 ceph-mon[31966]: pgmap v5101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:35 smithi189 ceph-mon[36790]: pgmap v5101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:35 smithi193 ceph-mon[37473]: pgmap v5101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:36.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8174.649784 secs 2024-07-26T13:43:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8174.649784 secs 2024-07-26T13:43:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8174.649784 secs 2024-07-26T13:43:37.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:37 smithi134 ceph-mon[31966]: pgmap v5102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:37 smithi189 ceph-mon[36790]: pgmap v5102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:37 smithi193 ceph-mon[37473]: pgmap v5102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:39.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:43:39.492+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:43:39.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:43:39.492+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:39.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:43:39.492+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:43:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:39 smithi134 ceph-mon[31966]: pgmap v5103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8160.225055 secs 2024-07-26T13:43:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:39 smithi189 ceph-mon[36790]: pgmap v5103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8160.225055 secs 2024-07-26T13:43:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:39 smithi193 ceph-mon[37473]: pgmap v5103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8160.225055 secs 2024-07-26T13:43:41.576 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:41 smithi193 ceph-mon[37473]: pgmap v5104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:43:41.576 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8179.649912 secs 2024-07-26T13:43:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:41 smithi134 ceph-mon[31966]: pgmap v5104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:43:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8179.649912 secs 2024-07-26T13:43:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:41 smithi189 ceph-mon[36790]: pgmap v5104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:43:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8179.649912 secs 2024-07-26T13:43:43.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:43 smithi134 ceph-mon[31966]: pgmap v5105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:43 smithi189 ceph-mon[36790]: pgmap v5105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:43 smithi193 ceph-mon[37473]: pgmap v5105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8165.225122 secs 2024-07-26T13:43:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:43:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8165.225122 secs 2024-07-26T13:43:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:43:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8165.225122 secs 2024-07-26T13:43:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:43:45.423 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:45 smithi193 ceph-mon[37473]: pgmap v5106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:45.423 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:45.423 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:45.423 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:45.423 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:45 smithi189 ceph-mon[36790]: pgmap v5106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:45 smithi134 ceph-mon[31966]: pgmap v5106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:43:46.470 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:46 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:43:46.470 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:46 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:43:46.470 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8184.650029 secs 2024-07-26T13:43:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:46 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:43:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:46 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:43:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8184.650029 secs 2024-07-26T13:43:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:46 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:43:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:46 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:43:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8184.650029 secs 2024-07-26T13:43:47.450 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:47 smithi189 ceph-mon[36790]: pgmap v5107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:47 smithi134 ceph-mon[31966]: pgmap v5107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:47 smithi193 ceph-mon[37473]: pgmap v5107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:43:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:49 smithi134 ceph-mon[31966]: pgmap v5108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:43:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:43:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:43:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:43:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:43:49.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:43:49.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:49 smithi134 ceph-mon[31966]: pgmap v5109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:43:49.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:49 smithi134 ceph-mon[31966]: pgmap v5110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:43:49.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:43:49.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8170.225187 secs 2024-07-26T13:43:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:49 smithi189 ceph-mon[36790]: pgmap v5108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:43:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:43:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:43:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:43:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:43:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:43:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:49 smithi189 ceph-mon[36790]: pgmap v5109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:43:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:49 smithi189 ceph-mon[36790]: pgmap v5110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:43:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:43:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8170.225187 secs 2024-07-26T13:43:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:49 smithi193 ceph-mon[37473]: pgmap v5108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:43:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:43:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:43:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:43:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:43:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:43:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:43:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:49 smithi193 ceph-mon[37473]: pgmap v5109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T13:43:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:49 smithi193 ceph-mon[37473]: pgmap v5110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:43:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:43:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8170.225187 secs 2024-07-26T13:43:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:43:49.951+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:43:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:43:49.951+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:49.954 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:43:49.951+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:43:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:51 smithi134 ceph-mon[31966]: pgmap v5111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-26T13:43:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8189.650158 secs 2024-07-26T13:43:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:51 smithi189 ceph-mon[36790]: pgmap v5111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-26T13:43:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8189.650158 secs 2024-07-26T13:43:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:51 smithi193 ceph-mon[37473]: pgmap v5111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-07-26T13:43:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8189.650158 secs 2024-07-26T13:43:53.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:53 smithi134 ceph-mon[31966]: pgmap v5112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:53 smithi189 ceph-mon[36790]: pgmap v5112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:53 smithi193 ceph-mon[37473]: pgmap v5112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:43:54.509 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:43:54.508+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:43:54.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:43:54.508+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:43:54.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:43:54.508+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:43:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8175.225271 secs 2024-07-26T13:43:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8175.225271 secs 2024-07-26T13:43:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8175.225271 secs 2024-07-26T13:43:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:55 smithi134 ceph-mon[31966]: pgmap v5113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:43:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:55 smithi189 ceph-mon[36790]: pgmap v5113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:43:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:55 smithi193 ceph-mon[37473]: pgmap v5113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:43:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8194.650281 secs 2024-07-26T13:43:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8194.650281 secs 2024-07-26T13:43:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8194.650281 secs 2024-07-26T13:43:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:57 smithi134 ceph-mon[31966]: pgmap v5114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T13:43:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:57 smithi189 ceph-mon[36790]: pgmap v5114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T13:43:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:57 smithi193 ceph-mon[37473]: pgmap v5114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T13:43:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:59 smithi134 ceph-mon[31966]: pgmap v5115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:43:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:43:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8180.225391 secs 2024-07-26T13:43:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:59 smithi189 ceph-mon[36790]: pgmap v5115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:43:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:43:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8180.225391 secs 2024-07-26T13:43:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:59 smithi193 ceph-mon[37473]: pgmap v5115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:43:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:43:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8180.225391 secs 2024-07-26T13:44:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:01 smithi134 ceph-mon[31966]: pgmap v5116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8199.650385 secs 2024-07-26T13:44:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:01 smithi189 ceph-mon[36790]: pgmap v5116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8199.650385 secs 2024-07-26T13:44:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:01 smithi193 ceph-mon[37473]: pgmap v5116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8199.650385 secs 2024-07-26T13:44:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:03 smithi134 ceph-mon[31966]: pgmap v5117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:44:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:03 smithi189 ceph-mon[36790]: pgmap v5117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:44:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:03 smithi193 ceph-mon[37473]: pgmap v5117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:44:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8185.225488 secs 2024-07-26T13:44:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8185.225488 secs 2024-07-26T13:44:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8185.225488 secs 2024-07-26T13:44:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:44:04.958+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:44:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:44:04.958+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:04.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:44:04.958+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:44:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:05 smithi134 ceph-mon[31966]: pgmap v5118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:05 smithi189 ceph-mon[36790]: pgmap v5118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:05 smithi193 ceph-mon[37473]: pgmap v5118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8204.650508 secs 2024-07-26T13:44:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8204.650508 secs 2024-07-26T13:44:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8204.650508 secs 2024-07-26T13:44:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:07 smithi134 ceph-mon[31966]: pgmap v5119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:07 smithi189 ceph-mon[36790]: pgmap v5119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:07 smithi193 ceph-mon[37473]: pgmap v5119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:09.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:44:09.522+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:44:09.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:44:09.522+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:44:09.524 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:44:09.522+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:09 smithi134 ceph-mon[31966]: pgmap v5120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8190.225645 secs 2024-07-26T13:44:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:09 smithi189 ceph-mon[36790]: pgmap v5120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8190.225645 secs 2024-07-26T13:44:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:09 smithi193 ceph-mon[37473]: pgmap v5120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8190.225645 secs 2024-07-26T13:44:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:11 smithi134 ceph-mon[31966]: pgmap v5121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8209.650634 secs 2024-07-26T13:44:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:11 smithi189 ceph-mon[36790]: pgmap v5121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8209.650634 secs 2024-07-26T13:44:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:11 smithi193 ceph-mon[37473]: pgmap v5121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8209.650634 secs 2024-07-26T13:44:13.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:13 smithi134 ceph-mon[31966]: pgmap v5122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:44:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:13 smithi189 ceph-mon[36790]: pgmap v5122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:44:13.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:13 smithi193 ceph-mon[37473]: pgmap v5122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:44:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8195.225768 secs 2024-07-26T13:44:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8195.225768 secs 2024-07-26T13:44:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8195.225768 secs 2024-07-26T13:44:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:15 smithi134 ceph-mon[31966]: pgmap v5123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:15 smithi189 ceph-mon[36790]: pgmap v5123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:15 smithi193 ceph-mon[37473]: pgmap v5123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8214.650760 secs 2024-07-26T13:44:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8214.650760 secs 2024-07-26T13:44:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8214.650760 secs 2024-07-26T13:44:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:17 smithi189 ceph-mon[36790]: pgmap v5124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:17 smithi193 ceph-mon[37473]: pgmap v5124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:17.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:17 smithi134 ceph-mon[31966]: pgmap v5124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:19 smithi189 ceph-mon[36790]: pgmap v5125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8200.225880 secs 2024-07-26T13:44:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:19 smithi193 ceph-mon[37473]: pgmap v5125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8200.225880 secs 2024-07-26T13:44:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:19 smithi134 ceph-mon[31966]: pgmap v5125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8200.225880 secs 2024-07-26T13:44:19.974 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:44:19.973+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:44:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:44:19.973+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:19.975 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:44:19.973+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:44:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:21 smithi189 ceph-mon[36790]: pgmap v5126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8219.650881 secs 2024-07-26T13:44:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:21 smithi193 ceph-mon[37473]: pgmap v5126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8219.650881 secs 2024-07-26T13:44:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:21 smithi134 ceph-mon[31966]: pgmap v5126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8219.650881 secs 2024-07-26T13:44:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:23 smithi189 ceph-mon[36790]: pgmap v5127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:44:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:23 smithi193 ceph-mon[37473]: pgmap v5127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:44:23.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:23 smithi134 ceph-mon[31966]: pgmap v5127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:44:24.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:44:24.537+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:44:24.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:44:24.537+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:24.539 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:44:24.537+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:44:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8205.226007 secs 2024-07-26T13:44:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8205.226007 secs 2024-07-26T13:44:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8205.226007 secs 2024-07-26T13:44:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:25 smithi193 ceph-mon[37473]: pgmap v5128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:25.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:25 smithi134 ceph-mon[31966]: pgmap v5128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:25 smithi189 ceph-mon[36790]: pgmap v5128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8224.651018 secs 2024-07-26T13:44:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8224.651018 secs 2024-07-26T13:44:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8224.651018 secs 2024-07-26T13:44:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:27 smithi193 ceph-mon[37473]: pgmap v5129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:27 smithi134 ceph-mon[31966]: pgmap v5129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:27 smithi189 ceph-mon[36790]: pgmap v5129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:29 smithi193 ceph-mon[37473]: pgmap v5130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8210.226087 secs 2024-07-26T13:44:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:29 smithi134 ceph-mon[31966]: pgmap v5130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8210.226087 secs 2024-07-26T13:44:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:29 smithi189 ceph-mon[36790]: pgmap v5130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8210.226087 secs 2024-07-26T13:44:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:31 smithi193 ceph-mon[37473]: pgmap v5131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8229.651154 secs 2024-07-26T13:44:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:31 smithi134 ceph-mon[31966]: pgmap v5131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8229.651154 secs 2024-07-26T13:44:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:31 smithi189 ceph-mon[36790]: pgmap v5131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8229.651154 secs 2024-07-26T13:44:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:33 smithi193 ceph-mon[37473]: pgmap v5132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:44:33.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:33 smithi134 ceph-mon[31966]: pgmap v5132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:44:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:33 smithi189 ceph-mon[36790]: pgmap v5132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:44:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8215.226214 secs 2024-07-26T13:44:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8215.226214 secs 2024-07-26T13:44:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8215.226214 secs 2024-07-26T13:44:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:44:34.989+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:44:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:44:34.989+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:44:34.990 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:44:34.989+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:35 smithi193 ceph-mon[37473]: pgmap v5133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:35.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:35 smithi134 ceph-mon[31966]: pgmap v5133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:35 smithi189 ceph-mon[36790]: pgmap v5133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8234.651277 secs 2024-07-26T13:44:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8234.651277 secs 2024-07-26T13:44:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8234.651277 secs 2024-07-26T13:44:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:37 smithi193 ceph-mon[37473]: pgmap v5134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:37.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:37 smithi134 ceph-mon[31966]: pgmap v5134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:37 smithi189 ceph-mon[36790]: pgmap v5134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:39.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:44:39.553+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:44:39.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:44:39.553+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:39.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:44:39.553+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:44:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:39 smithi193 ceph-mon[37473]: pgmap v5135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8220.226290 secs 2024-07-26T13:44:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:39 smithi134 ceph-mon[31966]: pgmap v5135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8220.226290 secs 2024-07-26T13:44:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:39 smithi189 ceph-mon[36790]: pgmap v5135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8220.226290 secs 2024-07-26T13:44:41.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:41 smithi134 ceph-mon[31966]: pgmap v5136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8239.651404 secs 2024-07-26T13:44:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:41 smithi189 ceph-mon[36790]: pgmap v5136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8239.651404 secs 2024-07-26T13:44:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:41 smithi193 ceph-mon[37473]: pgmap v5136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8239.651404 secs 2024-07-26T13:44:43.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:43 smithi134 ceph-mon[31966]: pgmap v5137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:44:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:43 smithi189 ceph-mon[36790]: pgmap v5137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:44:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:43 smithi193 ceph-mon[37473]: pgmap v5137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:44:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8225.226408 secs 2024-07-26T13:44:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8225.226408 secs 2024-07-26T13:44:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8225.226408 secs 2024-07-26T13:44:45.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:45 smithi134 ceph-mon[31966]: pgmap v5138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:45 smithi189 ceph-mon[36790]: pgmap v5138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:45 smithi193 ceph-mon[37473]: pgmap v5138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:44:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8244.651526 secs 2024-07-26T13:44:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8244.651526 secs 2024-07-26T13:44:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8244.651526 secs 2024-07-26T13:44:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:47 smithi134 ceph-mon[31966]: pgmap v5139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:47 smithi189 ceph-mon[36790]: pgmap v5139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:47 smithi193 ceph-mon[37473]: pgmap v5139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8230.226531 secs 2024-07-26T13:44:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:44:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8230.226531 secs 2024-07-26T13:44:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:44:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8230.226531 secs 2024-07-26T13:44:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:44:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:49 smithi134 ceph-mon[31966]: pgmap v5140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:44:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:44:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:44:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:49 smithi189 ceph-mon[36790]: pgmap v5140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:44:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:44:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:44:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:44:50.004+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:44:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:44:50.004+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:50.006 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:44:50.004+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:44:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:49 smithi193 ceph-mon[37473]: pgmap v5140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:44:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:44:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:44:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:44:50.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:44:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:50 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:44:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:50 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:44:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:50 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:44:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:50 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:44:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:50 smithi134 ceph-mon[31966]: pgmap v5141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:44:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:50 smithi134 ceph-mon[31966]: pgmap v5142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:44:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:50 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:44:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:50 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:44:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:50 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:44:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:50 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:44:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:50 smithi189 ceph-mon[36790]: pgmap v5141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:44:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:50 smithi189 ceph-mon[36790]: pgmap v5142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:44:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:50 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:44:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:50 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:44:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:50 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:44:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:50 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:44:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:50 smithi193 ceph-mon[37473]: pgmap v5141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:44:51.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:50 smithi193 ceph-mon[37473]: pgmap v5142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:44:51.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8249.651638 secs 2024-07-26T13:44:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8249.651638 secs 2024-07-26T13:44:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8249.651638 secs 2024-07-26T13:44:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:52 smithi134 ceph-mon[31966]: pgmap v5143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:52 smithi189 ceph-mon[36790]: pgmap v5143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:52 smithi193 ceph-mon[37473]: pgmap v5143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:44:53.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8235.226597 secs 2024-07-26T13:44:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8235.226597 secs 2024-07-26T13:44:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8235.226597 secs 2024-07-26T13:44:54.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:44:54.553+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:44:54.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:44:54.554+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:44:54.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:44:54.554+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:44:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:54 smithi134 ceph-mon[31966]: pgmap v5144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:44:54.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:54 smithi189 ceph-mon[36790]: pgmap v5144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:44:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:54 smithi193 ceph-mon[37473]: pgmap v5144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:44:55.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:55 smithi134 ceph-mon[31966]: pgmap v5145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:44:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:55 smithi189 ceph-mon[36790]: pgmap v5145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:44:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:55 smithi193 ceph-mon[37473]: pgmap v5145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:44:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8254.651762 secs 2024-07-26T13:44:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8254.651762 secs 2024-07-26T13:44:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8254.651762 secs 2024-07-26T13:44:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:57 smithi134 ceph-mon[31966]: pgmap v5146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:44:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:57 smithi189 ceph-mon[36790]: pgmap v5146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:44:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:57 smithi193 ceph-mon[37473]: pgmap v5146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:44:58.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8240.226718 secs 2024-07-26T13:44:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8240.226718 secs 2024-07-26T13:44:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8240.226718 secs 2024-07-26T13:44:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:44:59 smithi134 ceph-mon[31966]: pgmap v5147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:44:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:44:59 smithi189 ceph-mon[36790]: pgmap v5147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:45:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:44:59 smithi193 ceph-mon[37473]: pgmap v5147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:45:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8259.651843 secs 2024-07-26T13:45:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8259.651843 secs 2024-07-26T13:45:01.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8259.651843 secs 2024-07-26T13:45:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:02 smithi134 ceph-mon[31966]: pgmap v5148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:02 smithi189 ceph-mon[36790]: pgmap v5148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:02 smithi193 ceph-mon[37473]: pgmap v5148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8245.226807 secs 2024-07-26T13:45:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:04 smithi134 ceph-mon[31966]: pgmap v5149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8245.226807 secs 2024-07-26T13:45:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:04 smithi189 ceph-mon[36790]: pgmap v5149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:45:05.020+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:45:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:45:05.020+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:05.021 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:45:05.020+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:45:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8245.226807 secs 2024-07-26T13:45:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:04 smithi193 ceph-mon[37473]: pgmap v5149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:06 smithi134 ceph-mon[31966]: pgmap v5150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8264.651962 secs 2024-07-26T13:45:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:06 smithi189 ceph-mon[36790]: pgmap v5150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8264.651962 secs 2024-07-26T13:45:07.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:06 smithi193 ceph-mon[37473]: pgmap v5150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:07.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8264.651962 secs 2024-07-26T13:45:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:08 smithi134 ceph-mon[31966]: pgmap v5151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:08 smithi189 ceph-mon[36790]: pgmap v5151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:08 smithi193 ceph-mon[37473]: pgmap v5151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:09.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:45:09.569+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:45:09.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:45:09.569+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:09.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:45:09.569+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:45:09.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8250.226934 secs 2024-07-26T13:45:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8250.226934 secs 2024-07-26T13:45:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8250.226934 secs 2024-07-26T13:45:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:10 smithi134 ceph-mon[31966]: pgmap v5152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:45:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:10 smithi189 ceph-mon[36790]: pgmap v5152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:45:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:10 smithi193 ceph-mon[37473]: pgmap v5152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:45:11.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8269.652089 secs 2024-07-26T13:45:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8269.652089 secs 2024-07-26T13:45:11.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8269.652089 secs 2024-07-26T13:45:12.860 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:12 smithi189 ceph-mon[36790]: pgmap v5153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:12 smithi134 ceph-mon[31966]: pgmap v5153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:12 smithi193 ceph-mon[37473]: pgmap v5153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8255.227077 secs 2024-07-26T13:45:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:14 smithi134 ceph-mon[31966]: pgmap v5154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8255.227077 secs 2024-07-26T13:45:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:14 smithi189 ceph-mon[36790]: pgmap v5154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8255.227077 secs 2024-07-26T13:45:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:14 smithi193 ceph-mon[37473]: pgmap v5154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:16 smithi134 ceph-mon[31966]: pgmap v5155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8274.652179 secs 2024-07-26T13:45:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:16 smithi189 ceph-mon[36790]: pgmap v5155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8274.652179 secs 2024-07-26T13:45:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:16 smithi193 ceph-mon[37473]: pgmap v5155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8274.652179 secs 2024-07-26T13:45:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:18 smithi134 ceph-mon[31966]: pgmap v5156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:18 smithi189 ceph-mon[36790]: pgmap v5156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:18 smithi193 ceph-mon[37473]: pgmap v5156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8260.227194 secs 2024-07-26T13:45:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8260.227194 secs 2024-07-26T13:45:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8260.227194 secs 2024-07-26T13:45:20.025 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:45:20.024+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:45:20.025 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:45:20.024+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:20.025 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:45:20.024+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:45:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:20 smithi134 ceph-mon[31966]: pgmap v5157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:45:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:20 smithi189 ceph-mon[36790]: pgmap v5157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:45:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:20 smithi193 ceph-mon[37473]: pgmap v5157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:45:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8279.652278 secs 2024-07-26T13:45:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8279.652278 secs 2024-07-26T13:45:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8279.652278 secs 2024-07-26T13:45:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:22 smithi134 ceph-mon[31966]: pgmap v5158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:22 smithi189 ceph-mon[36790]: pgmap v5158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:22 smithi193 ceph-mon[37473]: pgmap v5158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:23.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:45:23.765+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T13:45:23.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:45:23.765+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T13:45:23.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:45:23.766+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T13:45:23.766 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:45:23.766+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T13:45:23.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8265.227344 secs 2024-07-26T13:45:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8265.227344 secs 2024-07-26T13:45:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8265.227344 secs 2024-07-26T13:45:24.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:45:24.584+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:45:24.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:45:24.584+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:24.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:45:24.584+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:45:24.872 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:24 smithi134 ceph-mon[31966]: pgmap v5159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:24 smithi189 ceph-mon[36790]: pgmap v5159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:24 smithi193 ceph-mon[37473]: pgmap v5159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:26 smithi134 ceph-mon[31966]: pgmap v5160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8284.652394 secs 2024-07-26T13:45:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:26 smithi189 ceph-mon[36790]: pgmap v5160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8284.652394 secs 2024-07-26T13:45:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:26 smithi193 ceph-mon[37473]: pgmap v5160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8284.652394 secs 2024-07-26T13:45:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:28 smithi134 ceph-mon[31966]: pgmap v5161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8270.227409 secs 2024-07-26T13:45:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:28 smithi189 ceph-mon[36790]: pgmap v5161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8270.227409 secs 2024-07-26T13:45:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:28 smithi193 ceph-mon[37473]: pgmap v5161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8270.227409 secs 2024-07-26T13:45:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:30 smithi134 ceph-mon[31966]: pgmap v5162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:45:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:30 smithi189 ceph-mon[36790]: pgmap v5162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:45:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:30 smithi193 ceph-mon[37473]: pgmap v5162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:45:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8289.652533 secs 2024-07-26T13:45:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8289.652533 secs 2024-07-26T13:45:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8289.652533 secs 2024-07-26T13:45:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:32 smithi134 ceph-mon[31966]: pgmap v5163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:32 smithi189 ceph-mon[36790]: pgmap v5163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:32 smithi193 ceph-mon[37473]: pgmap v5163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8275.227494 secs 2024-07-26T13:45:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:33 smithi134 ceph-mon[31966]: pgmap v5164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8275.227494 secs 2024-07-26T13:45:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:33 smithi189 ceph-mon[36790]: pgmap v5164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8275.227494 secs 2024-07-26T13:45:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:33 smithi193 ceph-mon[37473]: pgmap v5164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:45:34.290+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T13:45:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:45:34.290+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:45:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:45:34.291+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T13:45:34.293 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:45:34.291+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:45:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:45:35.039+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:45:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:45:35.040+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:45:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:45:35.040+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:36 smithi134 ceph-mon[31966]: pgmap v5165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8294.652651 secs 2024-07-26T13:45:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:36 smithi189 ceph-mon[36790]: pgmap v5165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8294.652651 secs 2024-07-26T13:45:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:36 smithi193 ceph-mon[37473]: pgmap v5165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8294.652651 secs 2024-07-26T13:45:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:38 smithi134 ceph-mon[31966]: pgmap v5166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:38 smithi189 ceph-mon[36790]: pgmap v5166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:38 smithi193 ceph-mon[37473]: pgmap v5166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:39.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:45:39.599+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:45:39.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:45:39.599+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:39.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:45:39.599+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:45:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8280.227636 secs 2024-07-26T13:45:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8280.227636 secs 2024-07-26T13:45:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8280.227636 secs 2024-07-26T13:45:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:40 smithi134 ceph-mon[31966]: pgmap v5167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:45:40.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:40 smithi189 ceph-mon[36790]: pgmap v5167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:45:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:40 smithi193 ceph-mon[37473]: pgmap v5167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:45:41.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8299.652772 secs 2024-07-26T13:45:41.924 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8299.652772 secs 2024-07-26T13:45:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8299.652772 secs 2024-07-26T13:45:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:42 smithi134 ceph-mon[31966]: pgmap v5168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:42 smithi189 ceph-mon[36790]: pgmap v5168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:42 smithi193 ceph-mon[37473]: pgmap v5168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:43.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8285.227782 secs 2024-07-26T13:45:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8285.227782 secs 2024-07-26T13:45:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8285.227782 secs 2024-07-26T13:45:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:44 smithi134 ceph-mon[31966]: pgmap v5169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:44 smithi189 ceph-mon[36790]: pgmap v5169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:44 smithi193 ceph-mon[37473]: pgmap v5169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:46 smithi134 ceph-mon[31966]: pgmap v5170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8304.652893 secs 2024-07-26T13:45:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:46 smithi189 ceph-mon[36790]: pgmap v5170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8304.652893 secs 2024-07-26T13:45:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:46 smithi193 ceph-mon[37473]: pgmap v5170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:45:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8304.652893 secs 2024-07-26T13:45:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:48 smithi134 ceph-mon[31966]: pgmap v5171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8290.227909 secs 2024-07-26T13:45:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:48 smithi189 ceph-mon[36790]: pgmap v5171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8290.227909 secs 2024-07-26T13:45:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:48 smithi193 ceph-mon[37473]: pgmap v5171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:45:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8290.227909 secs 2024-07-26T13:45:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:49 smithi189 ceph-mon[36790]: pgmap v5172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:45:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:49 smithi193 ceph-mon[37473]: pgmap v5172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:45:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:45:50.055+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:45:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:45:50.055+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:50.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:45:50.055+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:45:50.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:49 smithi134 ceph-mon[31966]: pgmap v5172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:45:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:45:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:45:51.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:45:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:45:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:45:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:51 smithi193 ceph-mon[37473]: pgmap v5173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:45:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:45:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8309.653017 secs 2024-07-26T13:45:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:45:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:45:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:51 smithi134 ceph-mon[31966]: pgmap v5173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:45:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:45:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8309.653017 secs 2024-07-26T13:45:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:45:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:45:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:51 smithi189 ceph-mon[36790]: pgmap v5173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:45:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:45:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8309.653017 secs 2024-07-26T13:45:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:53 smithi193 ceph-mon[37473]: pgmap v5174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:45:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8295.228038 secs 2024-07-26T13:45:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:53 smithi134 ceph-mon[31966]: pgmap v5174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:45:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8295.228038 secs 2024-07-26T13:45:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:53 smithi189 ceph-mon[36790]: pgmap v5174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T13:45:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8295.228038 secs 2024-07-26T13:45:54.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:45:54.603+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:45:54.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:45:54.603+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:45:54.605 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:45:54.603+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:45:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:55 smithi193 ceph-mon[37473]: pgmap v5175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:45:56.104 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:55 smithi134 ceph-mon[31966]: pgmap v5175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:45:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:55 smithi189 ceph-mon[36790]: pgmap v5175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:45:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8314.653114 secs 2024-07-26T13:45:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8314.653114 secs 2024-07-26T13:45:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8314.653114 secs 2024-07-26T13:45:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:57 smithi134 ceph-mon[31966]: pgmap v5176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:45:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:57 smithi189 ceph-mon[36790]: pgmap v5176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:45:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:57 smithi193 ceph-mon[37473]: pgmap v5176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:45:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8300.228168 secs 2024-07-26T13:45:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8300.228168 secs 2024-07-26T13:45:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8300.228168 secs 2024-07-26T13:46:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:45:59 smithi134 ceph-mon[31966]: pgmap v5177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:46:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:45:59 smithi189 ceph-mon[36790]: pgmap v5177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:46:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:45:59 smithi193 ceph-mon[37473]: pgmap v5177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:46:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:01 smithi134 ceph-mon[31966]: pgmap v5178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:46:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8319.653235 secs 2024-07-26T13:46:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:01 smithi189 ceph-mon[36790]: pgmap v5178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:46:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8319.653235 secs 2024-07-26T13:46:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:01 smithi193 ceph-mon[37473]: pgmap v5178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:46:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8319.653235 secs 2024-07-26T13:46:04.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:03 smithi134 ceph-mon[31966]: pgmap v5179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8305.228294 secs 2024-07-26T13:46:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:03 smithi189 ceph-mon[36790]: pgmap v5179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8305.228294 secs 2024-07-26T13:46:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:03 smithi193 ceph-mon[37473]: pgmap v5179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8305.228294 secs 2024-07-26T13:46:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:46:05.056+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:46:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:46:05.056+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:46:05.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:46:05.056+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:06.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:05 smithi134 ceph-mon[31966]: pgmap v5180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:46:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:05 smithi189 ceph-mon[36790]: pgmap v5180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:46:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:05 smithi193 ceph-mon[37473]: pgmap v5180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:46:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8324.653355 secs 2024-07-26T13:46:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8324.653355 secs 2024-07-26T13:46:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8324.653355 secs 2024-07-26T13:46:08.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:07 smithi134 ceph-mon[31966]: pgmap v5181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:07 smithi189 ceph-mon[36790]: pgmap v5181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:07 smithi193 ceph-mon[37473]: pgmap v5181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:09.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8310.228376 secs 2024-07-26T13:46:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8310.228376 secs 2024-07-26T13:46:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8310.228376 secs 2024-07-26T13:46:09.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:46:09.616+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:46:09.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:46:09.616+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:09.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:46:09.616+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:46:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:10 smithi134 ceph-mon[31966]: pgmap v5182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:10 smithi189 ceph-mon[36790]: pgmap v5182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:10 smithi193 ceph-mon[37473]: pgmap v5182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:12.226 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:12 smithi134 ceph-mon[31966]: pgmap v5183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:12.227 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8329.653477 secs 2024-07-26T13:46:12.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:12 smithi193 ceph-mon[37473]: pgmap v5183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:12.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8329.653477 secs 2024-07-26T13:46:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:12 smithi189 ceph-mon[36790]: pgmap v5183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8329.653477 secs 2024-07-26T13:46:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:14 smithi134 ceph-mon[31966]: pgmap v5184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8315.228503 secs 2024-07-26T13:46:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:14 smithi189 ceph-mon[36790]: pgmap v5184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8315.228503 secs 2024-07-26T13:46:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:14 smithi193 ceph-mon[37473]: pgmap v5184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8315.228503 secs 2024-07-26T13:46:16.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:16 smithi134 ceph-mon[31966]: pgmap v5185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:46:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:16 smithi189 ceph-mon[36790]: pgmap v5185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:46:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:16 smithi193 ceph-mon[37473]: pgmap v5185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:46:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8334.653562 secs 2024-07-26T13:46:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8334.653562 secs 2024-07-26T13:46:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8334.653562 secs 2024-07-26T13:46:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:18 smithi134 ceph-mon[31966]: pgmap v5186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:18 smithi189 ceph-mon[36790]: pgmap v5186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:18.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:18 smithi193 ceph-mon[37473]: pgmap v5186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8320.228632 secs 2024-07-26T13:46:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8320.228632 secs 2024-07-26T13:46:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8320.228632 secs 2024-07-26T13:46:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:46:20.066+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:46:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:46:20.066+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:46:20.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:46:20.066+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:20 smithi134 ceph-mon[31966]: pgmap v5187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:20 smithi189 ceph-mon[36790]: pgmap v5187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:20 smithi193 ceph-mon[37473]: pgmap v5187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8339.653701 secs 2024-07-26T13:46:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8339.653701 secs 2024-07-26T13:46:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8339.653701 secs 2024-07-26T13:46:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:22 smithi134 ceph-mon[31966]: pgmap v5188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:22 smithi189 ceph-mon[36790]: pgmap v5188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:22 smithi193 ceph-mon[37473]: pgmap v5188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:24 smithi134 ceph-mon[31966]: pgmap v5189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8325.228759 secs 2024-07-26T13:46:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:24 smithi189 ceph-mon[36790]: pgmap v5189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8325.228759 secs 2024-07-26T13:46:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:24 smithi193 ceph-mon[37473]: pgmap v5189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8325.228759 secs 2024-07-26T13:46:24.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:46:24.631+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:46:24.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:46:24.631+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:24.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:46:24.631+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:46:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:26 smithi189 ceph-mon[36790]: pgmap v5190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:46:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8344.653838 secs 2024-07-26T13:46:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:26 smithi193 ceph-mon[37473]: pgmap v5190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:46:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8344.653838 secs 2024-07-26T13:46:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:26 smithi134 ceph-mon[31966]: pgmap v5190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:46:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8344.653838 secs 2024-07-26T13:46:28.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:28 smithi189 ceph-mon[36790]: pgmap v5191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:28 smithi193 ceph-mon[37473]: pgmap v5191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:28.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:28 smithi134 ceph-mon[31966]: pgmap v5191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8330.228917 secs 2024-07-26T13:46:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8330.228917 secs 2024-07-26T13:46:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8330.228917 secs 2024-07-26T13:46:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:30 smithi193 ceph-mon[37473]: pgmap v5192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:30 smithi134 ceph-mon[31966]: pgmap v5192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:30 smithi189 ceph-mon[36790]: pgmap v5192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8349.653959 secs 2024-07-26T13:46:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8349.653959 secs 2024-07-26T13:46:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8349.653959 secs 2024-07-26T13:46:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:32 smithi193 ceph-mon[37473]: pgmap v5193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:32 smithi134 ceph-mon[31966]: pgmap v5193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:32 smithi189 ceph-mon[36790]: pgmap v5193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:34 smithi193 ceph-mon[37473]: pgmap v5194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8335.229034 secs 2024-07-26T13:46:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:34 smithi134 ceph-mon[31966]: pgmap v5194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8335.229034 secs 2024-07-26T13:46:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:34 smithi189 ceph-mon[36790]: pgmap v5194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:46:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8335.229034 secs 2024-07-26T13:46:35.083 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:46:35.082+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:46:35.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:46:35.082+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:35.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:46:35.082+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:46:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:36 smithi193 ceph-mon[37473]: pgmap v5195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:46:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8354.654097 secs 2024-07-26T13:46:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:36 smithi134 ceph-mon[31966]: pgmap v5195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:46:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8354.654097 secs 2024-07-26T13:46:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:36 smithi189 ceph-mon[36790]: pgmap v5195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:46:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8354.654097 secs 2024-07-26T13:46:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:38 smithi193 ceph-mon[37473]: pgmap v5196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:38 smithi134 ceph-mon[31966]: pgmap v5196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:38 smithi189 ceph-mon[36790]: pgmap v5196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8340.229131 secs 2024-07-26T13:46:39.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8340.229131 secs 2024-07-26T13:46:39.646 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:46:39.645+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:46:39.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:46:39.645+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:46:39.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:46:39.645+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8340.229131 secs 2024-07-26T13:46:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:40 smithi193 ceph-mon[37473]: pgmap v5197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:46:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:40 smithi134 ceph-mon[31966]: pgmap v5197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:46:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:40 smithi189 ceph-mon[36790]: pgmap v5197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:46:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8359.654245 secs 2024-07-26T13:46:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8359.654245 secs 2024-07-26T13:46:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8359.654245 secs 2024-07-26T13:46:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:42 smithi193 ceph-mon[37473]: pgmap v5198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:42.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:42 smithi189 ceph-mon[36790]: pgmap v5198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:42 smithi134 ceph-mon[31966]: pgmap v5198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:44 smithi193 ceph-mon[37473]: pgmap v5199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:46:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8345.229202 secs 2024-07-26T13:46:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:44 smithi134 ceph-mon[31966]: pgmap v5199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:46:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8345.229202 secs 2024-07-26T13:46:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:44 smithi189 ceph-mon[36790]: pgmap v5199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:46:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8345.229202 secs 2024-07-26T13:46:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:46 smithi193 ceph-mon[37473]: pgmap v5200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:46:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8364.654372 secs 2024-07-26T13:46:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:46 smithi134 ceph-mon[31966]: pgmap v5200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:46:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8364.654372 secs 2024-07-26T13:46:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:46 smithi189 ceph-mon[36790]: pgmap v5200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:46:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8364.654372 secs 2024-07-26T13:46:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:48 smithi193 ceph-mon[37473]: pgmap v5201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:48 smithi134 ceph-mon[31966]: pgmap v5201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:48 smithi189 ceph-mon[36790]: pgmap v5201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:46:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8350.229321 secs 2024-07-26T13:46:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8350.229321 secs 2024-07-26T13:46:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8350.229321 secs 2024-07-26T13:46:50.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:46:50.084+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:46:50.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:46:50.084+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:50.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:46:50.084+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:46:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:50 smithi134 ceph-mon[31966]: pgmap v5202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:46:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:50 smithi189 ceph-mon[36790]: pgmap v5202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:46:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:50 smithi193 ceph-mon[37473]: pgmap v5202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T13:46:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:46:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8369.654464 secs 2024-07-26T13:46:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:46:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8369.654464 secs 2024-07-26T13:46:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:46:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8369.654464 secs 2024-07-26T13:46:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:52 smithi193 ceph-mon[37473]: pgmap v5203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:46:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:46:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:52 smithi134 ceph-mon[31966]: pgmap v5203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:46:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:46:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:52 smithi189 ceph-mon[36790]: pgmap v5203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:46:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:46:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:46:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:46:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:46:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:53 smithi193 ceph-mon[37473]: pgmap v5204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T13:46:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:53 smithi193 ceph-mon[37473]: pgmap v5205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:46:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:46:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:46:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:46:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:53 smithi134 ceph-mon[31966]: pgmap v5204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T13:46:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:53 smithi134 ceph-mon[31966]: pgmap v5205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:46:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:46:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:46:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:46:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:53 smithi189 ceph-mon[36790]: pgmap v5204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T13:46:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:53 smithi189 ceph-mon[36790]: pgmap v5205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:46:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:46:54.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:46:54.660+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:46:54.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:46:54.660+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:46:54.662 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:46:54.660+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:46:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8355.229398 secs 2024-07-26T13:46:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8355.229398 secs 2024-07-26T13:46:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8355.229398 secs 2024-07-26T13:46:55.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:55 smithi134 ceph-mon[31966]: pgmap v5206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:46:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:55 smithi189 ceph-mon[36790]: pgmap v5206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:46:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:55 smithi193 ceph-mon[37473]: pgmap v5206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:46:56.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8374.654604 secs 2024-07-26T13:46:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8374.654604 secs 2024-07-26T13:46:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8374.654604 secs 2024-07-26T13:46:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:57 smithi134 ceph-mon[31966]: pgmap v5207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:46:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:57 smithi189 ceph-mon[36790]: pgmap v5207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:46:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:57 smithi193 ceph-mon[37473]: pgmap v5207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:46:58.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8360.229537 secs 2024-07-26T13:46:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8360.229537 secs 2024-07-26T13:46:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8360.229537 secs 2024-07-26T13:46:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:46:59 smithi134 ceph-mon[31966]: pgmap v5208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:46:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:46:59 smithi189 ceph-mon[36790]: pgmap v5208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:47:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:46:59 smithi193 ceph-mon[37473]: pgmap v5208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:47:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:00 smithi193 ceph-mon[37473]: pgmap v5209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:47:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:00 smithi134 ceph-mon[31966]: pgmap v5209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:47:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:00 smithi189 ceph-mon[36790]: pgmap v5209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:47:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8379.654716 secs 2024-07-26T13:47:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8379.654716 secs 2024-07-26T13:47:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8379.654716 secs 2024-07-26T13:47:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:02 smithi193 ceph-mon[37473]: pgmap v5210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:47:03.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:02 smithi134 ceph-mon[31966]: pgmap v5210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:47:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:02 smithi189 ceph-mon[36790]: pgmap v5210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:47:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8365.229668 secs 2024-07-26T13:47:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8365.229668 secs 2024-07-26T13:47:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8365.229668 secs 2024-07-26T13:47:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:04 smithi193 ceph-mon[37473]: pgmap v5211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:05.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:47:05.100+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:47:05.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:47:05.100+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:05.101 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:47:05.100+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:47:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:04 smithi134 ceph-mon[31966]: pgmap v5211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:04 smithi189 ceph-mon[36790]: pgmap v5211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:06.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8384.654835 secs 2024-07-26T13:47:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8384.654835 secs 2024-07-26T13:47:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8384.654835 secs 2024-07-26T13:47:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:07 smithi189 ceph-mon[36790]: pgmap v5212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:07 smithi193 ceph-mon[37473]: pgmap v5212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:07 smithi134 ceph-mon[31966]: pgmap v5212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8370.229748 secs 2024-07-26T13:47:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8370.229748 secs 2024-07-26T13:47:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8370.229748 secs 2024-07-26T13:47:09.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:47:09.676+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:47:09.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:47:09.676+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:09.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:47:09.676+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:47:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:09 smithi134 ceph-mon[31966]: pgmap v5213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:09 smithi189 ceph-mon[36790]: pgmap v5213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:09 smithi193 ceph-mon[37473]: pgmap v5213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:10 smithi193 ceph-mon[37473]: pgmap v5214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:11.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:10 smithi134 ceph-mon[31966]: pgmap v5214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:10 smithi189 ceph-mon[36790]: pgmap v5214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8389.654957 secs 2024-07-26T13:47:12.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8389.654957 secs 2024-07-26T13:47:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8389.654957 secs 2024-07-26T13:47:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:12 smithi193 ceph-mon[37473]: pgmap v5215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:12 smithi134 ceph-mon[31966]: pgmap v5215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:12 smithi189 ceph-mon[36790]: pgmap v5215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8375.229882 secs 2024-07-26T13:47:14.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8375.229882 secs 2024-07-26T13:47:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8375.229882 secs 2024-07-26T13:47:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:14 smithi193 ceph-mon[37473]: pgmap v5216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:14 smithi134 ceph-mon[31966]: pgmap v5216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:14 smithi189 ceph-mon[36790]: pgmap v5216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:16.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8394.655053 secs 2024-07-26T13:47:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8394.655053 secs 2024-07-26T13:47:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8394.655053 secs 2024-07-26T13:47:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:17 smithi189 ceph-mon[36790]: pgmap v5217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:17 smithi193 ceph-mon[37473]: pgmap v5217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:17 smithi134 ceph-mon[31966]: pgmap v5217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8380.229975 secs 2024-07-26T13:47:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8380.229975 secs 2024-07-26T13:47:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8380.229975 secs 2024-07-26T13:47:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:19 smithi134 ceph-mon[31966]: pgmap v5218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:19 smithi189 ceph-mon[36790]: pgmap v5218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:19 smithi193 ceph-mon[37473]: pgmap v5218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:47:20.108+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:47:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:47:20.108+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:47:20.109 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:47:20.108+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:20 smithi134 ceph-mon[31966]: pgmap v5219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:20 smithi189 ceph-mon[36790]: pgmap v5219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:20 smithi193 ceph-mon[37473]: pgmap v5219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8399.655178 secs 2024-07-26T13:47:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8399.655178 secs 2024-07-26T13:47:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8399.655178 secs 2024-07-26T13:47:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:22 smithi189 ceph-mon[36790]: pgmap v5220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:22 smithi193 ceph-mon[37473]: pgmap v5220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:23.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:22 smithi134 ceph-mon[31966]: pgmap v5220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8385.230106 secs 2024-07-26T13:47:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8385.230106 secs 2024-07-26T13:47:24.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8385.230106 secs 2024-07-26T13:47:24.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:47:24.691+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:47:24.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:47:24.691+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:24.693 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:47:24.691+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:47:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:24 smithi189 ceph-mon[36790]: pgmap v5221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:24 smithi193 ceph-mon[37473]: pgmap v5221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:24 smithi134 ceph-mon[31966]: pgmap v5221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:26.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8404.655293 secs 2024-07-26T13:47:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8404.655293 secs 2024-07-26T13:47:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8404.655293 secs 2024-07-26T13:47:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:27 smithi189 ceph-mon[36790]: pgmap v5222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:27 smithi193 ceph-mon[37473]: pgmap v5222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:27 smithi134 ceph-mon[31966]: pgmap v5222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:28.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8390.230233 secs 2024-07-26T13:47:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8390.230233 secs 2024-07-26T13:47:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8390.230233 secs 2024-07-26T13:47:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:29 smithi134 ceph-mon[31966]: pgmap v5223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:29 smithi189 ceph-mon[36790]: pgmap v5223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:29 smithi193 ceph-mon[37473]: pgmap v5223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:30 smithi134 ceph-mon[31966]: pgmap v5224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:30 smithi189 ceph-mon[36790]: pgmap v5224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:30 smithi193 ceph-mon[37473]: pgmap v5224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8409.655422 secs 2024-07-26T13:47:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8409.655422 secs 2024-07-26T13:47:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8409.655422 secs 2024-07-26T13:47:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:32 smithi189 ceph-mon[36790]: pgmap v5225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:32 smithi193 ceph-mon[37473]: pgmap v5225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:32 smithi134 ceph-mon[31966]: pgmap v5225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8395.230321 secs 2024-07-26T13:47:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8395.230321 secs 2024-07-26T13:47:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8395.230321 secs 2024-07-26T13:47:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:34 smithi193 ceph-mon[37473]: pgmap v5226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:47:35.123+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:47:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:47:35.123+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:47:35.123+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:47:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:34 smithi134 ceph-mon[31966]: pgmap v5226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:34 smithi189 ceph-mon[36790]: pgmap v5226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:36.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8414.655516 secs 2024-07-26T13:47:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8414.655516 secs 2024-07-26T13:47:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8414.655516 secs 2024-07-26T13:47:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:37 smithi189 ceph-mon[36790]: pgmap v5227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:37 smithi193 ceph-mon[37473]: pgmap v5227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:37 smithi134 ceph-mon[31966]: pgmap v5227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:38.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:47:38.788+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T13:47:38.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:47:38.788+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T13:47:38.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:47:38.789+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T13:47:38.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:47:38.789+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T13:47:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8400.230452 secs 2024-07-26T13:47:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:39 smithi134 ceph-mon[31966]: pgmap v5228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8400.230452 secs 2024-07-26T13:47:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:39 smithi189 ceph-mon[36790]: pgmap v5228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8400.230452 secs 2024-07-26T13:47:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:39 smithi193 ceph-mon[37473]: pgmap v5228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:39.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:47:39.701+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:47:39.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:47:39.701+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:39.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:47:39.701+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:47:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:41 smithi134 ceph-mon[31966]: pgmap v5229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8419.655658 secs 2024-07-26T13:47:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:41 smithi189 ceph-mon[36790]: pgmap v5229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8419.655658 secs 2024-07-26T13:47:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:41 smithi193 ceph-mon[37473]: pgmap v5229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8419.655658 secs 2024-07-26T13:47:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:42 smithi134 ceph-mon[31966]: pgmap v5230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:42 smithi189 ceph-mon[36790]: pgmap v5230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:42 smithi193 ceph-mon[37473]: pgmap v5230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8405.230596 secs 2024-07-26T13:47:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8405.230596 secs 2024-07-26T13:47:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8405.230596 secs 2024-07-26T13:47:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:44 smithi189 ceph-mon[36790]: pgmap v5231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:44 smithi193 ceph-mon[37473]: pgmap v5231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:45.160 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:44 smithi134 ceph-mon[31966]: pgmap v5231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:47:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8424.655774 secs 2024-07-26T13:47:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8424.655774 secs 2024-07-26T13:47:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8424.655774 secs 2024-07-26T13:47:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:47 smithi189 ceph-mon[36790]: pgmap v5232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:47 smithi193 ceph-mon[37473]: pgmap v5232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:47 smithi134 ceph-mon[31966]: pgmap v5232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8410.230677 secs 2024-07-26T13:47:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8410.230677 secs 2024-07-26T13:47:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8410.230677 secs 2024-07-26T13:47:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:47:49.315+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:47:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:47:49.315+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T13:47:49.316 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:47:49.315+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:47:49.317 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:47:49.315+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T13:47:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:49 smithi134 ceph-mon[31966]: pgmap v5233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:49 smithi189 ceph-mon[36790]: pgmap v5233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:49 smithi193 ceph-mon[37473]: pgmap v5233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:47:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:47:50.128+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:47:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:47:50.128+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:47:50.128+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:47:51.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:51 smithi134 ceph-mon[31966]: pgmap v5234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8429.655862 secs 2024-07-26T13:47:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:51 smithi189 ceph-mon[36790]: pgmap v5234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8429.655862 secs 2024-07-26T13:47:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:51 smithi193 ceph-mon[37473]: pgmap v5234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:47:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8429.655862 secs 2024-07-26T13:47:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:52 smithi134 ceph-mon[31966]: pgmap v5235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:47:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:52 smithi189 ceph-mon[36790]: pgmap v5235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:47:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:52 smithi193 ceph-mon[37473]: pgmap v5235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:47:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:47:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8415.230769 secs 2024-07-26T13:47:53.924 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:47:53.924 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:53.924 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:53.925 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:53.925 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:53.925 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8415.230769 secs 2024-07-26T13:47:53.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:47:53.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:53.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:53.994 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:53.995 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:53.995 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8415.230769 secs 2024-07-26T13:47:54.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:47:54.716+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:47:54.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:47:54.716+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:47:54.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:47:54.716+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:47:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:54 smithi189 ceph-mon[36790]: pgmap v5236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:47:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:54 smithi193 ceph-mon[37473]: pgmap v5236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:47:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:54 smithi134 ceph-mon[31966]: pgmap v5236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:47:56.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:56.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:56.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:56.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:56.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8434.655973 secs 2024-07-26T13:47:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:56 smithi193 ceph-mon[37473]: pgmap v5237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:47:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8434.655973 secs 2024-07-26T13:47:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:56 smithi134 ceph-mon[31966]: pgmap v5237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:47:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:57.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8434.655973 secs 2024-07-26T13:47:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:56 smithi189 ceph-mon[36790]: pgmap v5237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:47:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:47:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:47:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:47:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:47:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:47:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:47:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:47:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:58 smithi193 ceph-mon[37473]: pgmap v5238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T13:47:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:58 smithi193 ceph-mon[37473]: pgmap v5239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:47:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:47:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8420.230902 secs 2024-07-26T13:47:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:58 smithi134 ceph-mon[31966]: pgmap v5238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T13:47:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:58 smithi134 ceph-mon[31966]: pgmap v5239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:47:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:47:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8420.230902 secs 2024-07-26T13:47:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:58 smithi189 ceph-mon[36790]: pgmap v5238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T13:47:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:58 smithi189 ceph-mon[36790]: pgmap v5239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T13:47:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:47:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8420.230902 secs 2024-07-26T13:48:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:00 smithi193 ceph-mon[37473]: pgmap v5240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:48:01.032 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:00 smithi134 ceph-mon[31966]: pgmap v5240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:48:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:00 smithi189 ceph-mon[36790]: pgmap v5240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:48:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8439.656101 secs 2024-07-26T13:48:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8439.656101 secs 2024-07-26T13:48:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8439.656101 secs 2024-07-26T13:48:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:02 smithi193 ceph-mon[37473]: pgmap v5241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:48:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:02 smithi134 ceph-mon[31966]: pgmap v5241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:48:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:02 smithi189 ceph-mon[36790]: pgmap v5241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:48:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8425.231018 secs 2024-07-26T13:48:04.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8425.231018 secs 2024-07-26T13:48:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8425.231018 secs 2024-07-26T13:48:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:04 smithi193 ceph-mon[37473]: pgmap v5242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:48:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:48:05.133+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:48:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:48:05.133+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:05.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:48:05.133+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:48:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:04 smithi134 ceph-mon[31966]: pgmap v5242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:48:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:04 smithi189 ceph-mon[36790]: pgmap v5242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:48:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:06 smithi134 ceph-mon[31966]: pgmap v5243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:48:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8444.656227 secs 2024-07-26T13:48:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:06 smithi189 ceph-mon[36790]: pgmap v5243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:48:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8444.656227 secs 2024-07-26T13:48:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:06 smithi193 ceph-mon[37473]: pgmap v5243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:48:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8444.656227 secs 2024-07-26T13:48:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:08 smithi134 ceph-mon[31966]: pgmap v5244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:48:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8430.231135 secs 2024-07-26T13:48:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:08 smithi189 ceph-mon[36790]: pgmap v5244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:48:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8430.231135 secs 2024-07-26T13:48:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:08 smithi193 ceph-mon[37473]: pgmap v5244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:48:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8430.231135 secs 2024-07-26T13:48:09.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:48:09.728+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:48:09.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:48:09.728+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:09.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:48:09.728+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:48:11.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:10 smithi134 ceph-mon[31966]: pgmap v5245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:10 smithi189 ceph-mon[36790]: pgmap v5245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:11.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:10 smithi193 ceph-mon[37473]: pgmap v5245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:12.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8449.656379 secs 2024-07-26T13:48:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8449.656379 secs 2024-07-26T13:48:12.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8449.656379 secs 2024-07-26T13:48:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:12 smithi134 ceph-mon[31966]: pgmap v5246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:12 smithi189 ceph-mon[36790]: pgmap v5246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:12 smithi193 ceph-mon[37473]: pgmap v5246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8435.231248 secs 2024-07-26T13:48:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8435.231248 secs 2024-07-26T13:48:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8435.231248 secs 2024-07-26T13:48:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:15 smithi134 ceph-mon[31966]: pgmap v5247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:15 smithi189 ceph-mon[36790]: pgmap v5247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:15 smithi193 ceph-mon[37473]: pgmap v5247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8454.656445 secs 2024-07-26T13:48:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8454.656445 secs 2024-07-26T13:48:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8454.656445 secs 2024-07-26T13:48:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:17 smithi189 ceph-mon[36790]: pgmap v5248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:17 smithi193 ceph-mon[37473]: pgmap v5248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:17 smithi134 ceph-mon[31966]: pgmap v5248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:19 smithi189 ceph-mon[36790]: pgmap v5249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8440.231359 secs 2024-07-26T13:48:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:19 smithi193 ceph-mon[37473]: pgmap v5249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8440.231359 secs 2024-07-26T13:48:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:19 smithi134 ceph-mon[31966]: pgmap v5249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8440.231359 secs 2024-07-26T13:48:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:48:20.135+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:48:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:48:20.135+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:20.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:48:20.135+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:48:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:21 smithi189 ceph-mon[36790]: pgmap v5250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8459.656573 secs 2024-07-26T13:48:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:21 smithi193 ceph-mon[37473]: pgmap v5250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8459.656573 secs 2024-07-26T13:48:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:21 smithi134 ceph-mon[31966]: pgmap v5250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8459.656573 secs 2024-07-26T13:48:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:23 smithi193 ceph-mon[37473]: pgmap v5251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:23 smithi134 ceph-mon[31966]: pgmap v5251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:23 smithi189 ceph-mon[36790]: pgmap v5251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:23.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:48:23.796+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T13:48:23.797 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:48:23.796+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T13:48:23.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:48:23.796+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T13:48:23.798 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:48:23.797+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T13:48:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8445.231500 secs 2024-07-26T13:48:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8445.231500 secs 2024-07-26T13:48:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8445.231500 secs 2024-07-26T13:48:24.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:48:24.743+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:48:24.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:48:24.743+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:24.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:48:24.744+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:48:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:25 smithi193 ceph-mon[37473]: pgmap v5252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:25 smithi134 ceph-mon[31966]: pgmap v5252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:25 smithi189 ceph-mon[36790]: pgmap v5252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8464.656699 secs 2024-07-26T13:48:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8464.656699 secs 2024-07-26T13:48:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8464.656699 secs 2024-07-26T13:48:27.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:27 smithi193 ceph-mon[37473]: pgmap v5253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:27 smithi134 ceph-mon[31966]: pgmap v5253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:27 smithi189 ceph-mon[36790]: pgmap v5253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:29 smithi193 ceph-mon[37473]: pgmap v5254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8450.231578 secs 2024-07-26T13:48:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:29 smithi134 ceph-mon[31966]: pgmap v5254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8450.231578 secs 2024-07-26T13:48:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:29 smithi189 ceph-mon[36790]: pgmap v5254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8450.231578 secs 2024-07-26T13:48:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:31 smithi193 ceph-mon[37473]: pgmap v5255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8469.656781 secs 2024-07-26T13:48:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:31 smithi134 ceph-mon[31966]: pgmap v5255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8469.656781 secs 2024-07-26T13:48:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:31 smithi189 ceph-mon[36790]: pgmap v5255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8469.656781 secs 2024-07-26T13:48:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:33 smithi193 ceph-mon[37473]: pgmap v5256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:33.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:33 smithi134 ceph-mon[31966]: pgmap v5256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:33 smithi189 ceph-mon[36790]: pgmap v5256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:34.323 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:48:34.322+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T13:48:34.323 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:48:34.322+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:48:34.324 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:48:34.323+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:48:34.324 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:48:34.323+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T13:48:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8455.231681 secs 2024-07-26T13:48:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8455.231681 secs 2024-07-26T13:48:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8455.231681 secs 2024-07-26T13:48:35.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:48:35.151+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:48:35.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:48:35.151+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:48:35.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:48:35.151+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:35 smithi193 ceph-mon[37473]: pgmap v5257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:35 smithi134 ceph-mon[31966]: pgmap v5257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:35 smithi189 ceph-mon[36790]: pgmap v5257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8474.656897 secs 2024-07-26T13:48:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8474.656897 secs 2024-07-26T13:48:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8474.656897 secs 2024-07-26T13:48:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:37 smithi134 ceph-mon[31966]: pgmap v5258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:37 smithi189 ceph-mon[36790]: pgmap v5258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:37 smithi193 ceph-mon[37473]: pgmap v5258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:39 smithi134 ceph-mon[31966]: pgmap v5259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8460.231808 secs 2024-07-26T13:48:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:39 smithi189 ceph-mon[36790]: pgmap v5259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8460.231808 secs 2024-07-26T13:48:39.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:48:39.759+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:48:39.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:48:39.759+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:39.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:48:39.759+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:48:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:39 smithi193 ceph-mon[37473]: pgmap v5259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8460.231808 secs 2024-07-26T13:48:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:41 smithi134 ceph-mon[31966]: pgmap v5260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8479.657016 secs 2024-07-26T13:48:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:41 smithi189 ceph-mon[36790]: pgmap v5260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8479.657016 secs 2024-07-26T13:48:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:41 smithi193 ceph-mon[37473]: pgmap v5260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8479.657016 secs 2024-07-26T13:48:43.576 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:43 smithi193 ceph-mon[37473]: pgmap v5261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:43.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:43 smithi134 ceph-mon[31966]: pgmap v5261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:43 smithi189 ceph-mon[36790]: pgmap v5261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8465.231927 secs 2024-07-26T13:48:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8465.231927 secs 2024-07-26T13:48:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8465.231927 secs 2024-07-26T13:48:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:45 smithi134 ceph-mon[31966]: pgmap v5262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:45 smithi189 ceph-mon[36790]: pgmap v5262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:45 smithi193 ceph-mon[37473]: pgmap v5262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8484.657133 secs 2024-07-26T13:48:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8484.657133 secs 2024-07-26T13:48:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8484.657133 secs 2024-07-26T13:48:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:47 smithi134 ceph-mon[31966]: pgmap v5263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:47 smithi189 ceph-mon[36790]: pgmap v5263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:47 smithi193 ceph-mon[37473]: pgmap v5263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:49 smithi134 ceph-mon[31966]: pgmap v5264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8470.232027 secs 2024-07-26T13:48:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:49 smithi189 ceph-mon[36790]: pgmap v5264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8470.232027 secs 2024-07-26T13:48:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:49 smithi193 ceph-mon[37473]: pgmap v5264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8470.232027 secs 2024-07-26T13:48:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:48:50.164+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:48:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:48:50.164+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:48:50.165 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:48:50.164+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:51 smithi134 ceph-mon[31966]: pgmap v5265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8489.657269 secs 2024-07-26T13:48:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:51 smithi189 ceph-mon[36790]: pgmap v5265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8489.657269 secs 2024-07-26T13:48:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:51 smithi193 ceph-mon[37473]: pgmap v5265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:48:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8489.657269 secs 2024-07-26T13:48:53.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:53 smithi134 ceph-mon[31966]: pgmap v5266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:53 smithi189 ceph-mon[36790]: pgmap v5266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:53 smithi193 ceph-mon[37473]: pgmap v5266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8475.232153 secs 2024-07-26T13:48:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8475.232153 secs 2024-07-26T13:48:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8475.232153 secs 2024-07-26T13:48:54.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:48:54.775+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:48:54.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:48:54.775+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:48:54.776 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:48:54.775+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:48:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:55 smithi134 ceph-mon[31966]: pgmap v5267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:55 smithi189 ceph-mon[36790]: pgmap v5267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:55 smithi193 ceph-mon[37473]: pgmap v5267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8494.657342 secs 2024-07-26T13:48:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8494.657342 secs 2024-07-26T13:48:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8494.657342 secs 2024-07-26T13:48:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:57 smithi134 ceph-mon[31966]: pgmap v5268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:57 smithi189 ceph-mon[36790]: pgmap v5268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:57 smithi193 ceph-mon[37473]: pgmap v5268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:48:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:48:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:48:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:48:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:59 smithi134 ceph-mon[31966]: pgmap v5269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8480.232238 secs 2024-07-26T13:48:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:48:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:48:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:48:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:48:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:59 smithi189 ceph-mon[36790]: pgmap v5269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8480.232238 secs 2024-07-26T13:48:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:48:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:48:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:48:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:48:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:59 smithi193 ceph-mon[37473]: pgmap v5269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:48:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8480.232238 secs 2024-07-26T13:48:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:48:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:48:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:48:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:49:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:00 smithi134 ceph-mon[31966]: pgmap v5270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:49:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:00 smithi134 ceph-mon[31966]: pgmap v5271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:00 smithi189 ceph-mon[36790]: pgmap v5270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:49:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:00 smithi189 ceph-mon[36790]: pgmap v5271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:00 smithi193 ceph-mon[37473]: pgmap v5270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:49:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:00 smithi193 ceph-mon[37473]: pgmap v5271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8499.657461 secs 2024-07-26T13:49:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8499.657461 secs 2024-07-26T13:49:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8499.657461 secs 2024-07-26T13:49:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:02 smithi134 ceph-mon[31966]: pgmap v5272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:02 smithi189 ceph-mon[36790]: pgmap v5272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:02 smithi193 ceph-mon[37473]: pgmap v5272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:04 smithi134 ceph-mon[31966]: pgmap v5273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8485.232323 secs 2024-07-26T13:49:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:04 smithi189 ceph-mon[36790]: pgmap v5273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8485.232323 secs 2024-07-26T13:49:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:04 smithi193 ceph-mon[37473]: pgmap v5273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:49:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8485.232323 secs 2024-07-26T13:49:05.180 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:49:05.179+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:49:05.181 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:49:05.179+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:05.181 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:49:05.179+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:49:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:06 smithi134 ceph-mon[31966]: pgmap v5274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:49:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8504.657573 secs 2024-07-26T13:49:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:06 smithi189 ceph-mon[36790]: pgmap v5274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:49:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8504.657573 secs 2024-07-26T13:49:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:06 smithi193 ceph-mon[37473]: pgmap v5274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:49:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8504.657573 secs 2024-07-26T13:49:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:08 smithi189 ceph-mon[36790]: pgmap v5275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:49:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:08 smithi193 ceph-mon[37473]: pgmap v5275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:49:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:08 smithi134 ceph-mon[31966]: pgmap v5275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:49:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8490.232434 secs 2024-07-26T13:49:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8490.232434 secs 2024-07-26T13:49:09.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:49:09.790+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:49:09.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:49:09.790+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:09.792 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:49:09.790+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:49:09.792 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8490.232434 secs 2024-07-26T13:49:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:10 smithi189 ceph-mon[36790]: pgmap v5276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:49:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:10 smithi193 ceph-mon[37473]: pgmap v5276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:49:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:10 smithi134 ceph-mon[31966]: pgmap v5276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:49:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8509.657663 secs 2024-07-26T13:49:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8509.657663 secs 2024-07-26T13:49:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8509.657663 secs 2024-07-26T13:49:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:12 smithi193 ceph-mon[37473]: pgmap v5277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:12 smithi134 ceph-mon[31966]: pgmap v5277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:12 smithi189 ceph-mon[36790]: pgmap v5277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:14.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:14 smithi189 ceph-mon[36790]: pgmap v5278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:14.746 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8495.232556 secs 2024-07-26T13:49:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:14 smithi193 ceph-mon[37473]: pgmap v5278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8495.232556 secs 2024-07-26T13:49:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:14 smithi134 ceph-mon[31966]: pgmap v5278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8495.232556 secs 2024-07-26T13:49:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:16 smithi193 ceph-mon[37473]: pgmap v5279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:49:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8514.657782 secs 2024-07-26T13:49:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:16 smithi134 ceph-mon[31966]: pgmap v5279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:49:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8514.657782 secs 2024-07-26T13:49:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:16 smithi189 ceph-mon[36790]: pgmap v5279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:49:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8514.657782 secs 2024-07-26T13:49:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:18 smithi134 ceph-mon[31966]: pgmap v5280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:18 smithi193 ceph-mon[37473]: pgmap v5280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:18 smithi189 ceph-mon[36790]: pgmap v5280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8500.232678 secs 2024-07-26T13:49:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8500.232678 secs 2024-07-26T13:49:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8500.232678 secs 2024-07-26T13:49:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:49:20.194+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:49:20.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:49:20.194+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:49:20.196 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:49:20.194+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:20 smithi193 ceph-mon[37473]: pgmap v5281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:20 smithi134 ceph-mon[31966]: pgmap v5281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:20 smithi189 ceph-mon[36790]: pgmap v5281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8519.657901 secs 2024-07-26T13:49:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8519.657901 secs 2024-07-26T13:49:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8519.657901 secs 2024-07-26T13:49:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:22 smithi193 ceph-mon[37473]: pgmap v5282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:22 smithi134 ceph-mon[31966]: pgmap v5282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:22 smithi189 ceph-mon[36790]: pgmap v5282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:24 smithi193 ceph-mon[37473]: pgmap v5283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8505.232775 secs 2024-07-26T13:49:24.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:49:24.797+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:49:24.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:49:24.797+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:24.804 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:49:24.797+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:49:24.804 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:24 smithi134 ceph-mon[31966]: pgmap v5283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:24.805 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8505.232775 secs 2024-07-26T13:49:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:24 smithi189 ceph-mon[36790]: pgmap v5283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8505.232775 secs 2024-07-26T13:49:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:26 smithi193 ceph-mon[37473]: pgmap v5284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:49:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8524.658016 secs 2024-07-26T13:49:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:26 smithi134 ceph-mon[31966]: pgmap v5284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:49:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8524.658016 secs 2024-07-26T13:49:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:26 smithi189 ceph-mon[36790]: pgmap v5284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:49:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8524.658016 secs 2024-07-26T13:49:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:28 smithi193 ceph-mon[37473]: pgmap v5285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:28.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:28 smithi134 ceph-mon[31966]: pgmap v5285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:28 smithi189 ceph-mon[36790]: pgmap v5285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8510.232852 secs 2024-07-26T13:49:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8510.232852 secs 2024-07-26T13:49:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8510.232852 secs 2024-07-26T13:49:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:30 smithi134 ceph-mon[31966]: pgmap v5286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:30 smithi189 ceph-mon[36790]: pgmap v5286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:30 smithi193 ceph-mon[37473]: pgmap v5286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8529.658165 secs 2024-07-26T13:49:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8529.658165 secs 2024-07-26T13:49:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8529.658165 secs 2024-07-26T13:49:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:32 smithi134 ceph-mon[31966]: pgmap v5287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:32 smithi189 ceph-mon[36790]: pgmap v5287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:32 smithi193 ceph-mon[37473]: pgmap v5287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:34 smithi134 ceph-mon[31966]: pgmap v5288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8515.232958 secs 2024-07-26T13:49:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:34 smithi189 ceph-mon[36790]: pgmap v5288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8515.232958 secs 2024-07-26T13:49:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:34 smithi193 ceph-mon[37473]: pgmap v5288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8515.232958 secs 2024-07-26T13:49:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:49:35.210+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:49:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:49:35.210+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:49:35.211 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:49:35.210+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:36 smithi134 ceph-mon[31966]: pgmap v5289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:49:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8534.658288 secs 2024-07-26T13:49:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:36 smithi189 ceph-mon[36790]: pgmap v5289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:49:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8534.658288 secs 2024-07-26T13:49:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:36 smithi193 ceph-mon[37473]: pgmap v5289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:49:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8534.658288 secs 2024-07-26T13:49:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:38 smithi134 ceph-mon[31966]: pgmap v5290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:38 smithi189 ceph-mon[36790]: pgmap v5290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:38 smithi193 ceph-mon[37473]: pgmap v5290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:39.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:49:39.813+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:49:39.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:49:39.813+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:49:39.814 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:49:39.813+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8520.233051 secs 2024-07-26T13:49:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8520.233051 secs 2024-07-26T13:49:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8520.233051 secs 2024-07-26T13:49:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:40 smithi134 ceph-mon[31966]: pgmap v5291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:40 smithi189 ceph-mon[36790]: pgmap v5291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:40 smithi193 ceph-mon[37473]: pgmap v5291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:41.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8539.658406 secs 2024-07-26T13:49:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8539.658406 secs 2024-07-26T13:49:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8539.658406 secs 2024-07-26T13:49:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:42 smithi134 ceph-mon[31966]: pgmap v5292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:42 smithi189 ceph-mon[36790]: pgmap v5292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:42 smithi193 ceph-mon[37473]: pgmap v5292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:44 smithi134 ceph-mon[31966]: pgmap v5293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8525.233170 secs 2024-07-26T13:49:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:44 smithi189 ceph-mon[36790]: pgmap v5293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8525.233170 secs 2024-07-26T13:49:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:44 smithi193 ceph-mon[37473]: pgmap v5293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8525.233170 secs 2024-07-26T13:49:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:46 smithi134 ceph-mon[31966]: pgmap v5294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:49:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8544.658545 secs 2024-07-26T13:49:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:46 smithi189 ceph-mon[36790]: pgmap v5294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:49:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8544.658545 secs 2024-07-26T13:49:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:46 smithi193 ceph-mon[37473]: pgmap v5294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:49:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8544.658545 secs 2024-07-26T13:49:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:48 smithi134 ceph-mon[31966]: pgmap v5295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:48 smithi189 ceph-mon[36790]: pgmap v5295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:48 smithi193 ceph-mon[37473]: pgmap v5295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8530.233257 secs 2024-07-26T13:49:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8530.233257 secs 2024-07-26T13:49:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8530.233257 secs 2024-07-26T13:49:50.222 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:49:50.221+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:49:50.223 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:49:50.221+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:49:50.223 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:49:50.221+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:50 smithi134 ceph-mon[31966]: pgmap v5296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:50 smithi189 ceph-mon[36790]: pgmap v5296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:50 smithi193 ceph-mon[37473]: pgmap v5296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:51.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8549.658700 secs 2024-07-26T13:49:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8549.658700 secs 2024-07-26T13:49:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8549.658700 secs 2024-07-26T13:49:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:52 smithi134 ceph-mon[31966]: pgmap v5297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:52 smithi189 ceph-mon[36790]: pgmap v5297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:52 smithi193 ceph-mon[37473]: pgmap v5297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:53.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8535.233375 secs 2024-07-26T13:49:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8535.233375 secs 2024-07-26T13:49:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8535.233375 secs 2024-07-26T13:49:54.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:49:54.828+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:49:54.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:49:54.828+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:49:54.830 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:49:54.828+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:49:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:54 smithi134 ceph-mon[31966]: pgmap v5298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:54 smithi189 ceph-mon[36790]: pgmap v5298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:54 smithi193 ceph-mon[37473]: pgmap v5298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:49:56.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:56 smithi134 ceph-mon[31966]: pgmap v5299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:49:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8554.658842 secs 2024-07-26T13:49:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:56 smithi189 ceph-mon[36790]: pgmap v5299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:49:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8554.658842 secs 2024-07-26T13:49:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:56 smithi193 ceph-mon[37473]: pgmap v5299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:49:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8554.658842 secs 2024-07-26T13:49:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:58 smithi134 ceph-mon[31966]: pgmap v5300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8540.233469 secs 2024-07-26T13:49:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:58 smithi189 ceph-mon[36790]: pgmap v5300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8540.233469 secs 2024-07-26T13:49:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:58 smithi193 ceph-mon[37473]: pgmap v5300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:49:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8540.233469 secs 2024-07-26T13:49:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:49:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:50:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:49:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:50:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:49:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:50:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:00 smithi189 ceph-mon[36790]: pgmap v5301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:50:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:50:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:50:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:50:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:00 smithi193 ceph-mon[37473]: pgmap v5301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:50:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:50:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:50:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:50:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:00 smithi134 ceph-mon[31966]: pgmap v5301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T13:50:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:50:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:50:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:50:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:01 smithi189 ceph-mon[36790]: pgmap v5302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:50:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:01 smithi189 ceph-mon[36790]: pgmap v5303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:50:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8559.658941 secs 2024-07-26T13:50:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:01 smithi193 ceph-mon[37473]: pgmap v5302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:50:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:01 smithi193 ceph-mon[37473]: pgmap v5303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:50:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8559.658941 secs 2024-07-26T13:50:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:01 smithi134 ceph-mon[31966]: pgmap v5302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:50:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:01 smithi134 ceph-mon[31966]: pgmap v5303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:50:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8559.658941 secs 2024-07-26T13:50:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:02 smithi193 ceph-mon[37473]: pgmap v5304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:50:03.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:02 smithi134 ceph-mon[31966]: pgmap v5304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:50:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:02 smithi189 ceph-mon[36790]: pgmap v5304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T13:50:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8545.233586 secs 2024-07-26T13:50:04.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8545.233586 secs 2024-07-26T13:50:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8545.233586 secs 2024-07-26T13:50:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:04 smithi193 ceph-mon[37473]: pgmap v5305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:50:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:04 smithi134 ceph-mon[31966]: pgmap v5305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:50:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:04 smithi189 ceph-mon[36790]: pgmap v5305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:50:05.238 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:50:05.237+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:50:05.238 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:50:05.237+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:50:05.238 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:50:05.237+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8564.659039 secs 2024-07-26T13:50:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:07 smithi134 ceph-mon[31966]: pgmap v5306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:50:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8564.659039 secs 2024-07-26T13:50:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:07 smithi189 ceph-mon[36790]: pgmap v5306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:50:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8564.659039 secs 2024-07-26T13:50:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:07 smithi193 ceph-mon[37473]: pgmap v5306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:50:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:09 smithi134 ceph-mon[31966]: pgmap v5307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:50:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8550.233684 secs 2024-07-26T13:50:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:09 smithi189 ceph-mon[36790]: pgmap v5307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:50:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8550.233684 secs 2024-07-26T13:50:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:09 smithi193 ceph-mon[37473]: pgmap v5307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:50:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8550.233684 secs 2024-07-26T13:50:09.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:50:09.840+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:50:09.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:50:09.840+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:09.843 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:50:09.840+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:50:11.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:11 smithi134 ceph-mon[31966]: pgmap v5308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:50:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:11 smithi189 ceph-mon[36790]: pgmap v5308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:50:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:11 smithi193 ceph-mon[37473]: pgmap v5308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:50:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8569.659169 secs 2024-07-26T13:50:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8569.659169 secs 2024-07-26T13:50:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8569.659169 secs 2024-07-26T13:50:13.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:13 smithi134 ceph-mon[31966]: pgmap v5309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:13 smithi189 ceph-mon[36790]: pgmap v5309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:13 smithi193 ceph-mon[37473]: pgmap v5309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8555.233829 secs 2024-07-26T13:50:14.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8555.233829 secs 2024-07-26T13:50:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8555.233829 secs 2024-07-26T13:50:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:15 smithi134 ceph-mon[31966]: pgmap v5310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:15 smithi189 ceph-mon[36790]: pgmap v5310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:15 smithi193 ceph-mon[37473]: pgmap v5310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8574.659294 secs 2024-07-26T13:50:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8574.659294 secs 2024-07-26T13:50:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8574.659294 secs 2024-07-26T13:50:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:17 smithi189 ceph-mon[36790]: pgmap v5311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:17 smithi193 ceph-mon[37473]: pgmap v5311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:17 smithi134 ceph-mon[31966]: pgmap v5311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:19 smithi189 ceph-mon[36790]: pgmap v5312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8560.233972 secs 2024-07-26T13:50:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:19 smithi193 ceph-mon[37473]: pgmap v5312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8560.233972 secs 2024-07-26T13:50:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:19 smithi134 ceph-mon[31966]: pgmap v5312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8560.233972 secs 2024-07-26T13:50:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:50:20.253+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:50:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:50:20.253+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:20.254 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:50:20.253+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:50:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:21 smithi189 ceph-mon[36790]: pgmap v5313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8579.659359 secs 2024-07-26T13:50:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:21 smithi193 ceph-mon[37473]: pgmap v5313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8579.659359 secs 2024-07-26T13:50:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:21 smithi134 ceph-mon[31966]: pgmap v5313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8579.659359 secs 2024-07-26T13:50:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:23 smithi189 ceph-mon[36790]: pgmap v5314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:23 smithi193 ceph-mon[37473]: pgmap v5314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:23 smithi134 ceph-mon[31966]: pgmap v5314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8565.234087 secs 2024-07-26T13:50:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8565.234087 secs 2024-07-26T13:50:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8565.234087 secs 2024-07-26T13:50:24.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:50:24.850+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:50:24.852 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:50:24.850+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:50:24.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:50:24.850+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:25 smithi193 ceph-mon[37473]: pgmap v5315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:25 smithi134 ceph-mon[31966]: pgmap v5315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:25 smithi189 ceph-mon[36790]: pgmap v5315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8584.659456 secs 2024-07-26T13:50:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8584.659456 secs 2024-07-26T13:50:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8584.659456 secs 2024-07-26T13:50:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:27 smithi193 ceph-mon[37473]: pgmap v5316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:27 smithi134 ceph-mon[31966]: pgmap v5316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:27 smithi189 ceph-mon[36790]: pgmap v5316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:29 smithi193 ceph-mon[37473]: pgmap v5317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8570.234168 secs 2024-07-26T13:50:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:29 smithi134 ceph-mon[31966]: pgmap v5317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8570.234168 secs 2024-07-26T13:50:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:29 smithi189 ceph-mon[36790]: pgmap v5317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8570.234168 secs 2024-07-26T13:50:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:31 smithi193 ceph-mon[37473]: pgmap v5318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8589.659544 secs 2024-07-26T13:50:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:31 smithi134 ceph-mon[31966]: pgmap v5318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8589.659544 secs 2024-07-26T13:50:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:31 smithi189 ceph-mon[36790]: pgmap v5318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8589.659544 secs 2024-07-26T13:50:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:33 smithi193 ceph-mon[37473]: pgmap v5319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:33 smithi134 ceph-mon[31966]: pgmap v5319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:33 smithi189 ceph-mon[36790]: pgmap v5319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8575.234297 secs 2024-07-26T13:50:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8575.234297 secs 2024-07-26T13:50:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8575.234297 secs 2024-07-26T13:50:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:50:35.268+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:50:35.269 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:50:35.268+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:35.270 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:50:35.268+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:50:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:35 smithi193 ceph-mon[37473]: pgmap v5320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:35 smithi134 ceph-mon[31966]: pgmap v5320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:35 smithi189 ceph-mon[36790]: pgmap v5320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8594.659665 secs 2024-07-26T13:50:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8594.659665 secs 2024-07-26T13:50:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8594.659665 secs 2024-07-26T13:50:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:37 smithi193 ceph-mon[37473]: pgmap v5321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:37 smithi134 ceph-mon[31966]: pgmap v5321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:37 smithi189 ceph-mon[36790]: pgmap v5321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:39 smithi193 ceph-mon[37473]: pgmap v5322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8580.234417 secs 2024-07-26T13:50:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:39 smithi134 ceph-mon[31966]: pgmap v5322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8580.234417 secs 2024-07-26T13:50:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:39 smithi189 ceph-mon[36790]: pgmap v5322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8580.234417 secs 2024-07-26T13:50:39.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:50:39.865+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:50:39.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:50:39.865+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:39.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:50:39.865+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:50:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:41 smithi134 ceph-mon[31966]: pgmap v5323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8599.659804 secs 2024-07-26T13:50:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:41 smithi189 ceph-mon[36790]: pgmap v5323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8599.659804 secs 2024-07-26T13:50:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:41 smithi193 ceph-mon[37473]: pgmap v5323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8599.659804 secs 2024-07-26T13:50:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:43 smithi134 ceph-mon[31966]: pgmap v5324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:43 smithi189 ceph-mon[36790]: pgmap v5324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:43 smithi193 ceph-mon[37473]: pgmap v5324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:44.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8585.234545 secs 2024-07-26T13:50:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8585.234545 secs 2024-07-26T13:50:44.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8585.234545 secs 2024-07-26T13:50:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:45 smithi134 ceph-mon[31966]: pgmap v5325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:45 smithi189 ceph-mon[36790]: pgmap v5325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:45 smithi193 ceph-mon[37473]: pgmap v5325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8604.659940 secs 2024-07-26T13:50:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8604.659940 secs 2024-07-26T13:50:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8604.659940 secs 2024-07-26T13:50:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:47 smithi134 ceph-mon[31966]: pgmap v5326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:47 smithi189 ceph-mon[36790]: pgmap v5326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:47 smithi193 ceph-mon[37473]: pgmap v5326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:49 smithi134 ceph-mon[31966]: pgmap v5327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8590.234693 secs 2024-07-26T13:50:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:49 smithi189 ceph-mon[36790]: pgmap v5327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8590.234693 secs 2024-07-26T13:50:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:49 smithi193 ceph-mon[37473]: pgmap v5327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8590.234693 secs 2024-07-26T13:50:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:50:50.284+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:50:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:50:50.284+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:50:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:50:50.284+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:51 smithi134 ceph-mon[31966]: pgmap v5328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8609.660063 secs 2024-07-26T13:50:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:51 smithi189 ceph-mon[36790]: pgmap v5328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8609.660063 secs 2024-07-26T13:50:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:51 smithi193 ceph-mon[37473]: pgmap v5328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8609.660063 secs 2024-07-26T13:50:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:53 smithi134 ceph-mon[31966]: pgmap v5329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:53 smithi189 ceph-mon[36790]: pgmap v5329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:53 smithi193 ceph-mon[37473]: pgmap v5329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8595.234814 secs 2024-07-26T13:50:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8595.234814 secs 2024-07-26T13:50:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8595.234814 secs 2024-07-26T13:50:54.873 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:50:54.872+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:50:54.873 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:50:54.872+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:50:54.873 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:50:54.872+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:50:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:55 smithi134 ceph-mon[31966]: pgmap v5330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:55 smithi189 ceph-mon[36790]: pgmap v5330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:55 smithi193 ceph-mon[37473]: pgmap v5330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8614.660187 secs 2024-07-26T13:50:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8614.660187 secs 2024-07-26T13:50:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8614.660187 secs 2024-07-26T13:50:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:57 smithi134 ceph-mon[31966]: pgmap v5331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:57 smithi189 ceph-mon[36790]: pgmap v5331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:57 smithi193 ceph-mon[37473]: pgmap v5331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:50:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:59 smithi134 ceph-mon[31966]: pgmap v5332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:50:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8600.234878 secs 2024-07-26T13:50:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:59 smithi189 ceph-mon[36790]: pgmap v5332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:50:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8600.234878 secs 2024-07-26T13:50:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:59 smithi193 ceph-mon[37473]: pgmap v5332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:50:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:50:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8600.234878 secs 2024-07-26T13:51:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:51:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:51:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:51:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:01 smithi134 ceph-mon[31966]: pgmap v5333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8619.660243 secs 2024-07-26T13:51:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:51:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:51:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:51:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:01 smithi189 ceph-mon[36790]: pgmap v5333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8619.660243 secs 2024-07-26T13:51:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:51:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:51:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:51:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:01 smithi193 ceph-mon[37473]: pgmap v5333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8619.660243 secs 2024-07-26T13:51:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:51:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:51:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:51:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:02 smithi134 ceph-mon[31966]: pgmap v5334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:51:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:02 smithi134 ceph-mon[31966]: pgmap v5335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:51:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:02 smithi189 ceph-mon[36790]: pgmap v5334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:51:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:02 smithi189 ceph-mon[36790]: pgmap v5335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:51:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:02 smithi193 ceph-mon[37473]: pgmap v5334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:51:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:02 smithi193 ceph-mon[37473]: pgmap v5335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:51:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:04 smithi134 ceph-mon[31966]: pgmap v5336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:51:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8605.235012 secs 2024-07-26T13:51:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:04 smithi189 ceph-mon[36790]: pgmap v5336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:51:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8605.235012 secs 2024-07-26T13:51:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:04 smithi193 ceph-mon[37473]: pgmap v5336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:51:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8605.235012 secs 2024-07-26T13:51:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:51:05.287+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:51:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:51:05.287+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:51:05.289 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:51:05.287+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:06 smithi134 ceph-mon[31966]: pgmap v5337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:51:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8624.660371 secs 2024-07-26T13:51:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:06 smithi189 ceph-mon[36790]: pgmap v5337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:51:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8624.660371 secs 2024-07-26T13:51:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:06 smithi193 ceph-mon[37473]: pgmap v5337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:51:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8624.660371 secs 2024-07-26T13:51:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:08 smithi134 ceph-mon[31966]: pgmap v5338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:51:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:08 smithi189 ceph-mon[36790]: pgmap v5338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:51:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:08 smithi193 ceph-mon[37473]: pgmap v5338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:51:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8610.235082 secs 2024-07-26T13:51:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8610.235082 secs 2024-07-26T13:51:09.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:51:09.878+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:51:09.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:51:09.878+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:51:09.880 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:51:09.878+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:09.880 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8610.235082 secs 2024-07-26T13:51:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:10 smithi189 ceph-mon[36790]: pgmap v5339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:51:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:10 smithi193 ceph-mon[37473]: pgmap v5339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:51:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:10 smithi134 ceph-mon[31966]: pgmap v5339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:51:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8629.660503 secs 2024-07-26T13:51:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8629.660503 secs 2024-07-26T13:51:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8629.660503 secs 2024-07-26T13:51:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:12 smithi193 ceph-mon[37473]: pgmap v5340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:51:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:12 smithi134 ceph-mon[31966]: pgmap v5340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:51:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:12 smithi189 ceph-mon[36790]: pgmap v5340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:51:14.746 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:14 smithi189 ceph-mon[36790]: pgmap v5341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:51:14.746 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8615.235150 secs 2024-07-26T13:51:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:14 smithi193 ceph-mon[37473]: pgmap v5341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:51:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8615.235150 secs 2024-07-26T13:51:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:14 smithi134 ceph-mon[31966]: pgmap v5341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:51:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8615.235150 secs 2024-07-26T13:51:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:16 smithi193 ceph-mon[37473]: pgmap v5342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:51:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8634.660630 secs 2024-07-26T13:51:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:16 smithi134 ceph-mon[31966]: pgmap v5342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:51:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8634.660630 secs 2024-07-26T13:51:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:16 smithi189 ceph-mon[36790]: pgmap v5342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:51:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8634.660630 secs 2024-07-26T13:51:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:18 smithi193 ceph-mon[37473]: pgmap v5343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T13:51:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:18 smithi134 ceph-mon[31966]: pgmap v5343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T13:51:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:18 smithi189 ceph-mon[36790]: pgmap v5343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T13:51:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8620.235288 secs 2024-07-26T13:51:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8620.235288 secs 2024-07-26T13:51:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8620.235288 secs 2024-07-26T13:51:20.298 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:51:20.297+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:51:20.298 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:51:20.297+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:20.298 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:51:20.297+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:51:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:20 smithi193 ceph-mon[37473]: pgmap v5344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:51:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:20 smithi134 ceph-mon[31966]: pgmap v5344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:51:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:20 smithi189 ceph-mon[36790]: pgmap v5344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:51:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8639.660718 secs 2024-07-26T13:51:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8639.660718 secs 2024-07-26T13:51:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8639.660718 secs 2024-07-26T13:51:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:22 smithi134 ceph-mon[31966]: pgmap v5345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:51:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:22 smithi193 ceph-mon[37473]: pgmap v5345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:51:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:22 smithi189 ceph-mon[36790]: pgmap v5345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:51:23.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:51:23.827+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T13:51:23.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:51:23.827+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T13:51:23.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:51:23.827+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T13:51:23.829 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:51:23.827+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T13:51:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:24 smithi193 ceph-mon[37473]: pgmap v5346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:51:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8625.235425 secs 2024-07-26T13:51:24.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:51:24.889+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:51:24.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:51:24.889+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:24.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:51:24.889+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:51:24.892 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:24 smithi134 ceph-mon[31966]: pgmap v5346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:51:24.892 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8625.235425 secs 2024-07-26T13:51:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:24 smithi189 ceph-mon[36790]: pgmap v5346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T13:51:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8625.235425 secs 2024-07-26T13:51:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:26 smithi193 ceph-mon[37473]: pgmap v5347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8644.660802 secs 2024-07-26T13:51:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:26 smithi134 ceph-mon[31966]: pgmap v5347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8644.660802 secs 2024-07-26T13:51:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:26 smithi189 ceph-mon[36790]: pgmap v5347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8644.660802 secs 2024-07-26T13:51:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:28 smithi193 ceph-mon[37473]: pgmap v5348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:28 smithi134 ceph-mon[31966]: pgmap v5348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:28 smithi189 ceph-mon[36790]: pgmap v5348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8630.235575 secs 2024-07-26T13:51:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8630.235575 secs 2024-07-26T13:51:29.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8630.235575 secs 2024-07-26T13:51:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:30 smithi193 ceph-mon[37473]: pgmap v5349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:30 smithi134 ceph-mon[31966]: pgmap v5349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:30 smithi189 ceph-mon[36790]: pgmap v5349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8649.660933 secs 2024-07-26T13:51:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8649.660933 secs 2024-07-26T13:51:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8649.660933 secs 2024-07-26T13:51:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:32 smithi134 ceph-mon[31966]: pgmap v5350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:32 smithi189 ceph-mon[36790]: pgmap v5350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:32 smithi193 ceph-mon[37473]: pgmap v5350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:34.355 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:51:34.354+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T13:51:34.355 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:51:34.354+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:51:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:51:34.355+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:51:34.356 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:51:34.355+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T13:51:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:34 smithi134 ceph-mon[31966]: pgmap v5351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:51:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8635.235703 secs 2024-07-26T13:51:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:34 smithi189 ceph-mon[36790]: pgmap v5351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:51:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8635.235703 secs 2024-07-26T13:51:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:34 smithi193 ceph-mon[37473]: pgmap v5351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:51:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8635.235703 secs 2024-07-26T13:51:35.312 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:51:35.311+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:51:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:51:35.311+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:51:35.311+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:51:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:36 smithi134 ceph-mon[31966]: pgmap v5352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8654.661011 secs 2024-07-26T13:51:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:36 smithi189 ceph-mon[36790]: pgmap v5352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8654.661011 secs 2024-07-26T13:51:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:36 smithi193 ceph-mon[37473]: pgmap v5352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8654.661011 secs 2024-07-26T13:51:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:38 smithi134 ceph-mon[31966]: pgmap v5353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:38 smithi189 ceph-mon[36790]: pgmap v5353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:38 smithi193 ceph-mon[37473]: pgmap v5353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:39.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:51:39.903+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:51:39.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:51:39.903+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:51:39.905 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:51:39.904+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:39.906 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8640.235850 secs 2024-07-26T13:51:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8640.235850 secs 2024-07-26T13:51:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8640.235850 secs 2024-07-26T13:51:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:40 smithi134 ceph-mon[31966]: pgmap v5354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:40 smithi189 ceph-mon[36790]: pgmap v5354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:40 smithi193 ceph-mon[37473]: pgmap v5354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8659.661135 secs 2024-07-26T13:51:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8659.661135 secs 2024-07-26T13:51:42.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8659.661135 secs 2024-07-26T13:51:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:42 smithi134 ceph-mon[31966]: pgmap v5355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:42 smithi189 ceph-mon[36790]: pgmap v5355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:42 smithi193 ceph-mon[37473]: pgmap v5355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:44 smithi193 ceph-mon[37473]: pgmap v5356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:51:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8645.236022 secs 2024-07-26T13:51:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:44 smithi134 ceph-mon[31966]: pgmap v5356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:51:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8645.236022 secs 2024-07-26T13:51:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:44 smithi189 ceph-mon[36790]: pgmap v5356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:51:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8645.236022 secs 2024-07-26T13:51:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:46 smithi134 ceph-mon[31966]: pgmap v5357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8664.661222 secs 2024-07-26T13:51:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:46 smithi189 ceph-mon[36790]: pgmap v5357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8664.661222 secs 2024-07-26T13:51:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:46 smithi193 ceph-mon[37473]: pgmap v5357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8664.661222 secs 2024-07-26T13:51:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:48 smithi134 ceph-mon[31966]: pgmap v5358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:48 smithi189 ceph-mon[36790]: pgmap v5358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:48 smithi193 ceph-mon[37473]: pgmap v5358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:51:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8650.236106 secs 2024-07-26T13:51:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8650.236106 secs 2024-07-26T13:51:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8650.236106 secs 2024-07-26T13:51:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:51:50.327+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:51:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:51:50.327+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:51:50.328 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:51:50.327+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:50 smithi134 ceph-mon[31966]: pgmap v5359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:50 smithi189 ceph-mon[36790]: pgmap v5359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:50 smithi193 ceph-mon[37473]: pgmap v5359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:51:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8669.661372 secs 2024-07-26T13:51:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8669.661372 secs 2024-07-26T13:51:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8669.661372 secs 2024-07-26T13:51:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:52 smithi134 ceph-mon[31966]: pgmap v5360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:52 smithi189 ceph-mon[36790]: pgmap v5360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:52 smithi193 ceph-mon[37473]: pgmap v5360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:54 smithi134 ceph-mon[31966]: pgmap v5361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:51:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8655.236192 secs 2024-07-26T13:51:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:54 smithi189 ceph-mon[36790]: pgmap v5361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:51:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8655.236192 secs 2024-07-26T13:51:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:54 smithi193 ceph-mon[37473]: pgmap v5361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:51:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8655.236192 secs 2024-07-26T13:51:54.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:51:54.906+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:51:54.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:51:54.906+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:51:54.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:51:54.906+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:51:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:56 smithi134 ceph-mon[31966]: pgmap v5362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8674.661503 secs 2024-07-26T13:51:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:56 smithi189 ceph-mon[36790]: pgmap v5362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8674.661503 secs 2024-07-26T13:51:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:56 smithi193 ceph-mon[37473]: pgmap v5362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:51:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8674.661503 secs 2024-07-26T13:51:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:58 smithi134 ceph-mon[31966]: pgmap v5363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:51:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:58 smithi189 ceph-mon[36790]: pgmap v5363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:51:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:58 smithi193 ceph-mon[37473]: pgmap v5363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:51:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:51:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8660.236278 secs 2024-07-26T13:51:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:51:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8660.236278 secs 2024-07-26T13:51:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:51:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8660.236278 secs 2024-07-26T13:52:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:00 smithi134 ceph-mon[31966]: pgmap v5364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:52:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:00 smithi189 ceph-mon[36790]: pgmap v5364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:52:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:00 smithi193 ceph-mon[37473]: pgmap v5364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:52:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8679.661624 secs 2024-07-26T13:52:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8679.661624 secs 2024-07-26T13:52:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8679.661624 secs 2024-07-26T13:52:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:02 smithi134 ceph-mon[31966]: pgmap v5365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:52:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:52:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:02 smithi189 ceph-mon[36790]: pgmap v5365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:52:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:52:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:02 smithi193 ceph-mon[37473]: pgmap v5365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:52:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:52:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:52:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:52:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:03 smithi134 ceph-mon[31966]: pgmap v5366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:52:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:03 smithi134 ceph-mon[31966]: pgmap v5367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:52:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:52:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:52:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:52:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:03 smithi189 ceph-mon[36790]: pgmap v5366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:52:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:03 smithi189 ceph-mon[36790]: pgmap v5367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:52:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:52:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:52:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:52:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:03 smithi193 ceph-mon[37473]: pgmap v5366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:52:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:03 smithi193 ceph-mon[37473]: pgmap v5367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:52:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:52:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8665.236387 secs 2024-07-26T13:52:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8665.236387 secs 2024-07-26T13:52:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8665.236387 secs 2024-07-26T13:52:05.335 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:52:05.334+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:52:05.336 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:52:05.334+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:52:05.336 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:52:05.334+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:05 smithi189 ceph-mon[36790]: pgmap v5368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:52:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:05 smithi193 ceph-mon[37473]: pgmap v5368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:52:05.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:05 smithi134 ceph-mon[31966]: pgmap v5368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:52:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8684.661750 secs 2024-07-26T13:52:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8684.661750 secs 2024-07-26T13:52:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8684.661750 secs 2024-07-26T13:52:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:07 smithi193 ceph-mon[37473]: pgmap v5369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:52:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:07 smithi134 ceph-mon[31966]: pgmap v5369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:52:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:07 smithi189 ceph-mon[36790]: pgmap v5369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:52:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:09 smithi134 ceph-mon[31966]: pgmap v5370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:52:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8670.236456 secs 2024-07-26T13:52:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:09 smithi193 ceph-mon[37473]: pgmap v5370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:52:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8670.236456 secs 2024-07-26T13:52:09.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:52:09.920+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:52:09.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:52:09.920+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:52:09.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:52:09.920+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:09 smithi189 ceph-mon[36790]: pgmap v5370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:52:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8670.236456 secs 2024-07-26T13:52:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:11 smithi193 ceph-mon[37473]: pgmap v5371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:52:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8689.661877 secs 2024-07-26T13:52:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:11 smithi134 ceph-mon[31966]: pgmap v5371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:52:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8689.661877 secs 2024-07-26T13:52:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:11 smithi189 ceph-mon[36790]: pgmap v5371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:52:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8689.661877 secs 2024-07-26T13:52:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:13 smithi193 ceph-mon[37473]: pgmap v5372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:52:13.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:13 smithi134 ceph-mon[31966]: pgmap v5372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:52:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:13 smithi189 ceph-mon[36790]: pgmap v5372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:52:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8675.236588 secs 2024-07-26T13:52:14.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8675.236588 secs 2024-07-26T13:52:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8675.236588 secs 2024-07-26T13:52:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:15 smithi193 ceph-mon[37473]: pgmap v5373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:15.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:15 smithi134 ceph-mon[31966]: pgmap v5373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:15 smithi189 ceph-mon[36790]: pgmap v5373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8694.661939 secs 2024-07-26T13:52:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8694.661939 secs 2024-07-26T13:52:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8694.661939 secs 2024-07-26T13:52:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:17 smithi193 ceph-mon[37473]: pgmap v5374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:17.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:17 smithi134 ceph-mon[31966]: pgmap v5374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:17 smithi189 ceph-mon[36790]: pgmap v5374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:19 smithi193 ceph-mon[37473]: pgmap v5375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8680.236718 secs 2024-07-26T13:52:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:19 smithi134 ceph-mon[31966]: pgmap v5375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8680.236718 secs 2024-07-26T13:52:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:19 smithi189 ceph-mon[36790]: pgmap v5375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8680.236718 secs 2024-07-26T13:52:20.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:52:20.350+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:52:20.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:52:20.350+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:20.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:52:20.350+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:52:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:21 smithi193 ceph-mon[37473]: pgmap v5376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8699.662068 secs 2024-07-26T13:52:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:21 smithi134 ceph-mon[31966]: pgmap v5376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8699.662068 secs 2024-07-26T13:52:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:21 smithi189 ceph-mon[36790]: pgmap v5376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8699.662068 secs 2024-07-26T13:52:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:23 smithi193 ceph-mon[37473]: pgmap v5377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:52:23.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:23 smithi134 ceph-mon[31966]: pgmap v5377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:52:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:23 smithi189 ceph-mon[36790]: pgmap v5377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:52:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8685.236845 secs 2024-07-26T13:52:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8685.236845 secs 2024-07-26T13:52:24.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:52:24.936+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:52:24.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:52:24.936+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:52:24.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:52:24.936+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8685.236845 secs 2024-07-26T13:52:25.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:25 smithi134 ceph-mon[31966]: pgmap v5378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:25 smithi189 ceph-mon[36790]: pgmap v5378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:25 smithi193 ceph-mon[37473]: pgmap v5378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8704.662200 secs 2024-07-26T13:52:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8704.662200 secs 2024-07-26T13:52:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8704.662200 secs 2024-07-26T13:52:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:27 smithi134 ceph-mon[31966]: pgmap v5379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:27 smithi189 ceph-mon[36790]: pgmap v5379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:27 smithi193 ceph-mon[37473]: pgmap v5379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8690.236952 secs 2024-07-26T13:52:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:29 smithi134 ceph-mon[31966]: pgmap v5380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8690.236952 secs 2024-07-26T13:52:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:29 smithi189 ceph-mon[36790]: pgmap v5380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8690.236952 secs 2024-07-26T13:52:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:29 smithi193 ceph-mon[37473]: pgmap v5380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:31 smithi134 ceph-mon[31966]: pgmap v5381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8709.662352 secs 2024-07-26T13:52:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:31 smithi189 ceph-mon[36790]: pgmap v5381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8709.662352 secs 2024-07-26T13:52:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:31 smithi193 ceph-mon[37473]: pgmap v5381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8709.662352 secs 2024-07-26T13:52:33.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:33 smithi134 ceph-mon[31966]: pgmap v5382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:33 smithi189 ceph-mon[36790]: pgmap v5382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:33 smithi193 ceph-mon[37473]: pgmap v5382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8695.237093 secs 2024-07-26T13:52:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8695.237093 secs 2024-07-26T13:52:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8695.237093 secs 2024-07-26T13:52:35.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:52:35.365+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:52:35.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:52:35.365+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:35.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:52:35.365+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:52:35.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:35 smithi134 ceph-mon[31966]: pgmap v5383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:35 smithi189 ceph-mon[36790]: pgmap v5383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:35 smithi193 ceph-mon[37473]: pgmap v5383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8714.662472 secs 2024-07-26T13:52:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8714.662472 secs 2024-07-26T13:52:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8714.662472 secs 2024-07-26T13:52:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:37 smithi134 ceph-mon[31966]: pgmap v5384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:37 smithi189 ceph-mon[36790]: pgmap v5384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:37 smithi193 ceph-mon[37473]: pgmap v5384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8700.237183 secs 2024-07-26T13:52:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:39 smithi134 ceph-mon[31966]: pgmap v5385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8700.237183 secs 2024-07-26T13:52:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:39 smithi189 ceph-mon[36790]: pgmap v5385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:39.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:52:39.951+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:52:39.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:52:39.951+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:52:39.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:52:39.951+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8700.237183 secs 2024-07-26T13:52:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:39 smithi193 ceph-mon[37473]: pgmap v5385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:41 smithi134 ceph-mon[31966]: pgmap v5386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8719.662532 secs 2024-07-26T13:52:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:41 smithi189 ceph-mon[36790]: pgmap v5386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8719.662532 secs 2024-07-26T13:52:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:41 smithi193 ceph-mon[37473]: pgmap v5386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8719.662532 secs 2024-07-26T13:52:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:43 smithi134 ceph-mon[31966]: pgmap v5387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:43 smithi189 ceph-mon[36790]: pgmap v5387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:43 smithi193 ceph-mon[37473]: pgmap v5387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8705.237296 secs 2024-07-26T13:52:44.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8705.237296 secs 2024-07-26T13:52:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8705.237296 secs 2024-07-26T13:52:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:45 smithi134 ceph-mon[31966]: pgmap v5388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:52:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:45 smithi189 ceph-mon[36790]: pgmap v5388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:52:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:45 smithi193 ceph-mon[37473]: pgmap v5388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:52:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8724.662638 secs 2024-07-26T13:52:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8724.662638 secs 2024-07-26T13:52:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8724.662638 secs 2024-07-26T13:52:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:47 smithi134 ceph-mon[31966]: pgmap v5389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:47 smithi189 ceph-mon[36790]: pgmap v5389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:47 smithi193 ceph-mon[37473]: pgmap v5389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8710.237417 secs 2024-07-26T13:52:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8710.237417 secs 2024-07-26T13:52:49.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8710.237417 secs 2024-07-26T13:52:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:49 smithi189 ceph-mon[36790]: pgmap v5390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:49 smithi193 ceph-mon[37473]: pgmap v5390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:50.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:49 smithi134 ceph-mon[31966]: pgmap v5390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:50.369 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:52:50.367+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:52:50.369 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:52:50.368+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:52:50.369 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:52:50.368+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:51 smithi193 ceph-mon[37473]: pgmap v5391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8729.662763 secs 2024-07-26T13:52:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:51 smithi134 ceph-mon[31966]: pgmap v5391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8729.662763 secs 2024-07-26T13:52:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:51 smithi189 ceph-mon[36790]: pgmap v5391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8729.662763 secs 2024-07-26T13:52:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:53 smithi193 ceph-mon[37473]: pgmap v5392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8715.237482 secs 2024-07-26T13:52:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:53 smithi134 ceph-mon[31966]: pgmap v5392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8715.237482 secs 2024-07-26T13:52:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:53 smithi189 ceph-mon[36790]: pgmap v5392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:52:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8715.237482 secs 2024-07-26T13:52:54.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:52:54.952+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:52:54.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:52:54.952+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:52:54.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:52:54.952+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:52:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:55 smithi193 ceph-mon[37473]: pgmap v5393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:52:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:55 smithi134 ceph-mon[31966]: pgmap v5393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:52:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:55 smithi189 ceph-mon[36790]: pgmap v5393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:52:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8734.662823 secs 2024-07-26T13:52:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8734.662823 secs 2024-07-26T13:52:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8734.662823 secs 2024-07-26T13:52:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:57 smithi193 ceph-mon[37473]: pgmap v5394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:57 smithi134 ceph-mon[31966]: pgmap v5394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:57 smithi189 ceph-mon[36790]: pgmap v5394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:52:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8720.237600 secs 2024-07-26T13:52:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:52:58 smithi193 ceph-mon[37473]: pgmap v5395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8720.237600 secs 2024-07-26T13:52:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:52:58 smithi134 ceph-mon[31966]: pgmap v5395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:52:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8720.237600 secs 2024-07-26T13:52:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:52:58 smithi189 ceph-mon[36790]: pgmap v5395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:01 smithi134 ceph-mon[31966]: pgmap v5396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8739.662962 secs 2024-07-26T13:53:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:01 smithi189 ceph-mon[36790]: pgmap v5396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8739.662962 secs 2024-07-26T13:53:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:01 smithi193 ceph-mon[37473]: pgmap v5396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8739.662962 secs 2024-07-26T13:53:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:03 smithi134 ceph-mon[31966]: pgmap v5397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:53:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:03 smithi189 ceph-mon[36790]: pgmap v5397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:53:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:03 smithi193 ceph-mon[37473]: pgmap v5397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:53:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:53:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8725.237748 secs 2024-07-26T13:53:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:53:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:53:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:04 smithi134 ceph-mon[31966]: pgmap v5398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:53:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:04 smithi134 ceph-mon[31966]: pgmap v5399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:53:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:53:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8725.237748 secs 2024-07-26T13:53:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:53:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:53:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:04 smithi189 ceph-mon[36790]: pgmap v5398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:53:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:04 smithi189 ceph-mon[36790]: pgmap v5399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:53:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:53:05.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8725.237748 secs 2024-07-26T13:53:05.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:53:05.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:53:05.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:04 smithi193 ceph-mon[37473]: pgmap v5398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:53:05.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:04 smithi193 ceph-mon[37473]: pgmap v5399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:53:05.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:53:05.384 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:53:05.384+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:53:05.384 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:53:05.384+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:53:05.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:53:05.384+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:06 smithi189 ceph-mon[36790]: pgmap v5400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:53:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8744.663101 secs 2024-07-26T13:53:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:06 smithi193 ceph-mon[37473]: pgmap v5400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:53:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8744.663101 secs 2024-07-26T13:53:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:06 smithi134 ceph-mon[31966]: pgmap v5400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:53:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8744.663101 secs 2024-07-26T13:53:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:08 smithi189 ceph-mon[36790]: pgmap v5401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:53:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8730.237818 secs 2024-07-26T13:53:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:08 smithi193 ceph-mon[37473]: pgmap v5401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:53:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8730.237818 secs 2024-07-26T13:53:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:08 smithi134 ceph-mon[31966]: pgmap v5401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:53:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8730.237818 secs 2024-07-26T13:53:09.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:53:09.963+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:53:09.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:53:09.963+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:53:09.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:53:09.963+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:10 smithi193 ceph-mon[37473]: pgmap v5402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:53:11.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:10 smithi134 ceph-mon[31966]: pgmap v5402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:53:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:10 smithi189 ceph-mon[36790]: pgmap v5402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:53:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8749.663225 secs 2024-07-26T13:53:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8749.663225 secs 2024-07-26T13:53:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8749.663225 secs 2024-07-26T13:53:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:12 smithi193 ceph-mon[37473]: pgmap v5403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:53:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:12 smithi134 ceph-mon[31966]: pgmap v5403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:53:13.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:12 smithi189 ceph-mon[36790]: pgmap v5403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:53:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8735.237961 secs 2024-07-26T13:53:14.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8735.237961 secs 2024-07-26T13:53:14.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8735.237961 secs 2024-07-26T13:53:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:14 smithi193 ceph-mon[37473]: pgmap v5404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:53:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:14 smithi134 ceph-mon[31966]: pgmap v5404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:53:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:14 smithi189 ceph-mon[36790]: pgmap v5404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:53:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:15 smithi193 ceph-mon[37473]: pgmap v5405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:16.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:15 smithi134 ceph-mon[31966]: pgmap v5405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:15 smithi189 ceph-mon[36790]: pgmap v5405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8754.663340 secs 2024-07-26T13:53:17.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8754.663340 secs 2024-07-26T13:53:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8754.663340 secs 2024-07-26T13:53:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:17 smithi193 ceph-mon[37473]: pgmap v5406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:17 smithi134 ceph-mon[31966]: pgmap v5406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:17 smithi189 ceph-mon[36790]: pgmap v5406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8740.238064 secs 2024-07-26T13:53:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8740.238064 secs 2024-07-26T13:53:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8740.238064 secs 2024-07-26T13:53:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:19 smithi193 ceph-mon[37473]: pgmap v5407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:53:20.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:19 smithi134 ceph-mon[31966]: pgmap v5407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:53:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:19 smithi189 ceph-mon[36790]: pgmap v5407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:53:20.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:53:20.387+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:53:20.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:53:20.387+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:53:20.389 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:53:20.387+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8759.663479 secs 2024-07-26T13:53:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8759.663479 secs 2024-07-26T13:53:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8759.663479 secs 2024-07-26T13:53:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:22 smithi193 ceph-mon[37473]: pgmap v5408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:22 smithi134 ceph-mon[31966]: pgmap v5408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:22 smithi189 ceph-mon[36790]: pgmap v5408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8745.238185 secs 2024-07-26T13:53:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8745.238185 secs 2024-07-26T13:53:24.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8745.238185 secs 2024-07-26T13:53:24.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:53:24.968+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:53:24.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:53:24.968+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:53:24.970 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:53:24.968+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:24.971 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:24 smithi134 ceph-mon[31966]: pgmap v5409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:24 smithi193 ceph-mon[37473]: pgmap v5409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:24 smithi189 ceph-mon[36790]: pgmap v5409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:26 smithi193 ceph-mon[37473]: pgmap v5410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8764.663614 secs 2024-07-26T13:53:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:26 smithi134 ceph-mon[31966]: pgmap v5410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8764.663614 secs 2024-07-26T13:53:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:26 smithi189 ceph-mon[36790]: pgmap v5410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8764.663614 secs 2024-07-26T13:53:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:28 smithi193 ceph-mon[37473]: pgmap v5411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8750.238311 secs 2024-07-26T13:53:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:28 smithi134 ceph-mon[31966]: pgmap v5411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8750.238311 secs 2024-07-26T13:53:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:28 smithi189 ceph-mon[36790]: pgmap v5411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8750.238311 secs 2024-07-26T13:53:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:30 smithi193 ceph-mon[37473]: pgmap v5412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:53:31.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:30 smithi134 ceph-mon[31966]: pgmap v5412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:53:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:30 smithi189 ceph-mon[36790]: pgmap v5412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:53:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8769.663734 secs 2024-07-26T13:53:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:31 smithi193 ceph-mon[37473]: pgmap v5413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8769.663734 secs 2024-07-26T13:53:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:31 smithi134 ceph-mon[31966]: pgmap v5413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8769.663734 secs 2024-07-26T13:53:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:31 smithi189 ceph-mon[36790]: pgmap v5413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8755.238369 secs 2024-07-26T13:53:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8755.238369 secs 2024-07-26T13:53:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8755.238369 secs 2024-07-26T13:53:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:34 smithi193 ceph-mon[37473]: pgmap v5414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:34 smithi134 ceph-mon[31966]: pgmap v5414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:34 smithi189 ceph-mon[36790]: pgmap v5414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:53:35.403+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:53:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:53:35.403+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:53:35.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:53:35.403+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:36 smithi193 ceph-mon[37473]: pgmap v5415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8774.663862 secs 2024-07-26T13:53:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:36 smithi134 ceph-mon[31966]: pgmap v5415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8774.663862 secs 2024-07-26T13:53:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:36 smithi189 ceph-mon[36790]: pgmap v5415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8774.663862 secs 2024-07-26T13:53:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:38 smithi193 ceph-mon[37473]: pgmap v5416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8760.238514 secs 2024-07-26T13:53:39.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:38 smithi134 ceph-mon[31966]: pgmap v5416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8760.238514 secs 2024-07-26T13:53:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:38 smithi189 ceph-mon[36790]: pgmap v5416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8760.238514 secs 2024-07-26T13:53:39.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:53:39.972+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:53:39.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:53:39.972+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:39.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:53:39.972+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:53:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:40 smithi193 ceph-mon[37473]: pgmap v5417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:53:41.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:40 smithi134 ceph-mon[31966]: pgmap v5417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:53:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:40 smithi189 ceph-mon[36790]: pgmap v5417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:53:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8779.663987 secs 2024-07-26T13:53:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8779.663987 secs 2024-07-26T13:53:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8779.663987 secs 2024-07-26T13:53:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:42 smithi193 ceph-mon[37473]: pgmap v5418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:43.048 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:42 smithi134 ceph-mon[31966]: pgmap v5418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:42 smithi189 ceph-mon[36790]: pgmap v5418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8765.238643 secs 2024-07-26T13:53:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:43 smithi193 ceph-mon[37473]: pgmap v5419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:44.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8765.238643 secs 2024-07-26T13:53:44.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:43 smithi189 ceph-mon[36790]: pgmap v5419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8765.238643 secs 2024-07-26T13:53:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:43 smithi134 ceph-mon[31966]: pgmap v5419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:46 smithi193 ceph-mon[37473]: pgmap v5420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8784.664083 secs 2024-07-26T13:53:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:46 smithi134 ceph-mon[31966]: pgmap v5420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8784.664083 secs 2024-07-26T13:53:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:46 smithi189 ceph-mon[36790]: pgmap v5420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8784.664083 secs 2024-07-26T13:53:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:48 smithi193 ceph-mon[37473]: pgmap v5421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8770.238708 secs 2024-07-26T13:53:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:48 smithi134 ceph-mon[31966]: pgmap v5421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8770.238708 secs 2024-07-26T13:53:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:48 smithi189 ceph-mon[36790]: pgmap v5421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8770.238708 secs 2024-07-26T13:53:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:53:50.404+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:53:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:53:50.404+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:50.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:53:50.404+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:53:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:50 smithi193 ceph-mon[37473]: pgmap v5422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:53:51.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:50 smithi134 ceph-mon[31966]: pgmap v5422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:53:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:50 smithi189 ceph-mon[36790]: pgmap v5422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:53:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8789.664209 secs 2024-07-26T13:53:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8789.664209 secs 2024-07-26T13:53:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8789.664209 secs 2024-07-26T13:53:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:52 smithi193 ceph-mon[37473]: pgmap v5423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:52 smithi134 ceph-mon[31966]: pgmap v5423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:52 smithi189 ceph-mon[36790]: pgmap v5423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8775.238839 secs 2024-07-26T13:53:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:53 smithi193 ceph-mon[37473]: pgmap v5424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8775.238839 secs 2024-07-26T13:53:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:53 smithi134 ceph-mon[31966]: pgmap v5424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8775.238839 secs 2024-07-26T13:53:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:53 smithi189 ceph-mon[36790]: pgmap v5424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:53:54.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:53:54.988+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:53:54.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:53:54.988+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:53:54.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:53:54.988+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:53:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:56 smithi193 ceph-mon[37473]: pgmap v5425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8794.664342 secs 2024-07-26T13:53:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:56 smithi134 ceph-mon[31966]: pgmap v5425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8794.664342 secs 2024-07-26T13:53:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:56 smithi189 ceph-mon[36790]: pgmap v5425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:53:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8794.664342 secs 2024-07-26T13:53:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:58 smithi193 ceph-mon[37473]: pgmap v5426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:53:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:53:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8780.238985 secs 2024-07-26T13:53:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:58 smithi134 ceph-mon[31966]: pgmap v5426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:53:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:53:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8780.238985 secs 2024-07-26T13:53:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:58 smithi189 ceph-mon[36790]: pgmap v5426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:53:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:53:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8780.238985 secs 2024-07-26T13:54:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:00 smithi193 ceph-mon[37473]: pgmap v5427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:54:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:00 smithi134 ceph-mon[31966]: pgmap v5427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:54:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:00 smithi189 ceph-mon[36790]: pgmap v5427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T13:54:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8799.664493 secs 2024-07-26T13:54:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:01 smithi193 ceph-mon[37473]: pgmap v5428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:54:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8799.664493 secs 2024-07-26T13:54:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:01 smithi134 ceph-mon[31966]: pgmap v5428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:54:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8799.664493 secs 2024-07-26T13:54:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:01 smithi189 ceph-mon[36790]: pgmap v5428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:54:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8785.239051 secs 2024-07-26T13:54:04.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8785.239051 secs 2024-07-26T13:54:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8785.239051 secs 2024-07-26T13:54:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:04 smithi193 ceph-mon[37473]: pgmap v5429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:54:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:54:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:04 smithi134 ceph-mon[31966]: pgmap v5429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:54:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:54:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:04 smithi189 ceph-mon[36790]: pgmap v5429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:54:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:54:05.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:54:05.419+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:54:05.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:54:05.419+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:05.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:54:05.419+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:54:06.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:54:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:54:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:54:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:54:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:54:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:54:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:54:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:54:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:54:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:54:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:06 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:54:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:06 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:54:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:06 smithi193 ceph-mon[37473]: pgmap v5430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:54:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:06 smithi193 ceph-mon[37473]: pgmap v5431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:54:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8804.664613 secs 2024-07-26T13:54:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:06 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:54:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:06 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:54:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:06 smithi134 ceph-mon[31966]: pgmap v5430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:54:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:06 smithi134 ceph-mon[31966]: pgmap v5431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:54:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8804.664613 secs 2024-07-26T13:54:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:06 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T13:54:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:06 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T13:54:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:06 smithi189 ceph-mon[36790]: pgmap v5430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T13:54:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:06 smithi189 ceph-mon[36790]: pgmap v5431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T13:54:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8804.664613 secs 2024-07-26T13:54:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:07 smithi134 ceph-mon[31966]: pgmap v5432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:54:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:07 smithi189 ceph-mon[36790]: pgmap v5432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:54:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:07 smithi193 ceph-mon[37473]: pgmap v5432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T13:54:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8790.239190 secs 2024-07-26T13:54:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8790.239190 secs 2024-07-26T13:54:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8790.239190 secs 2024-07-26T13:54:09.998 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:54:09.997+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:54:09.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:54:09.997+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:09.999 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:54:09.997+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:54:10.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:09 smithi134 ceph-mon[31966]: pgmap v5433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:54:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:09 smithi189 ceph-mon[36790]: pgmap v5433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:54:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:09 smithi193 ceph-mon[37473]: pgmap v5433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:54:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:11 smithi134 ceph-mon[31966]: pgmap v5434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:54:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8809.664708 secs 2024-07-26T13:54:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:11 smithi189 ceph-mon[36790]: pgmap v5434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:54:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8809.664708 secs 2024-07-26T13:54:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:11 smithi193 ceph-mon[37473]: pgmap v5434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:54:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8809.664708 secs 2024-07-26T13:54:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:13 smithi134 ceph-mon[31966]: pgmap v5435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:54:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8795.239328 secs 2024-07-26T13:54:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:13 smithi189 ceph-mon[36790]: pgmap v5435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:54:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8795.239328 secs 2024-07-26T13:54:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:13 smithi193 ceph-mon[37473]: pgmap v5435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:54:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8795.239328 secs 2024-07-26T13:54:16.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:15 smithi134 ceph-mon[31966]: pgmap v5436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:54:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:15 smithi189 ceph-mon[36790]: pgmap v5436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:54:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:15 smithi193 ceph-mon[37473]: pgmap v5436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:54:17.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8814.664791 secs 2024-07-26T13:54:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8814.664791 secs 2024-07-26T13:54:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8814.664791 secs 2024-07-26T13:54:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:17 smithi134 ceph-mon[31966]: pgmap v5437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:17 smithi189 ceph-mon[36790]: pgmap v5437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:17 smithi193 ceph-mon[37473]: pgmap v5437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8800.239486 secs 2024-07-26T13:54:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8800.239486 secs 2024-07-26T13:54:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8800.239486 secs 2024-07-26T13:54:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:19 smithi189 ceph-mon[36790]: pgmap v5438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:19 smithi193 ceph-mon[37473]: pgmap v5438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:19 smithi134 ceph-mon[31966]: pgmap v5438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:54:20.434+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:54:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:54:20.434+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:54:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:54:20.434+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:21 smithi193 ceph-mon[37473]: pgmap v5439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8819.664908 secs 2024-07-26T13:54:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:21 smithi134 ceph-mon[31966]: pgmap v5439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8819.664908 secs 2024-07-26T13:54:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:21 smithi189 ceph-mon[36790]: pgmap v5439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8819.664908 secs 2024-07-26T13:54:23.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:54:23.858+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T13:54:23.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:54:23.858+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T13:54:23.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:54:23.859+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T13:54:23.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:54:23.859+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T13:54:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:23 smithi134 ceph-mon[31966]: pgmap v5440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8805.239633 secs 2024-07-26T13:54:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:23 smithi193 ceph-mon[37473]: pgmap v5440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8805.239633 secs 2024-07-26T13:54:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:23 smithi189 ceph-mon[36790]: pgmap v5440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8805.239633 secs 2024-07-26T13:54:25.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:54:24.998+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:54:25.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:54:24.998+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:25.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:54:24.998+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:54:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:25 smithi193 ceph-mon[37473]: pgmap v5441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:26.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:25 smithi134 ceph-mon[31966]: pgmap v5441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:25 smithi189 ceph-mon[36790]: pgmap v5441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8824.665017 secs 2024-07-26T13:54:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8824.665017 secs 2024-07-26T13:54:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8824.665017 secs 2024-07-26T13:54:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:27 smithi193 ceph-mon[37473]: pgmap v5442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:28.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:27 smithi134 ceph-mon[31966]: pgmap v5442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:27 smithi189 ceph-mon[36790]: pgmap v5442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8810.239698 secs 2024-07-26T13:54:29.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8810.239698 secs 2024-07-26T13:54:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8810.239698 secs 2024-07-26T13:54:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:29 smithi193 ceph-mon[37473]: pgmap v5443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:29 smithi134 ceph-mon[31966]: pgmap v5443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:29 smithi189 ceph-mon[36790]: pgmap v5443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:31 smithi193 ceph-mon[37473]: pgmap v5444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8829.665156 secs 2024-07-26T13:54:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:31 smithi134 ceph-mon[31966]: pgmap v5444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8829.665156 secs 2024-07-26T13:54:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:31 smithi189 ceph-mon[36790]: pgmap v5444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8829.665156 secs 2024-07-26T13:54:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:33 smithi193 ceph-mon[37473]: pgmap v5445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8815.239818 secs 2024-07-26T13:54:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:54:34.386+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T13:54:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:54:34.386+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:54:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:54:34.387+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T13:54:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:54:34.387+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:54:34.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:33 smithi134 ceph-mon[31966]: pgmap v5445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8815.239818 secs 2024-07-26T13:54:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:33 smithi189 ceph-mon[36790]: pgmap v5445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8815.239818 secs 2024-07-26T13:54:35.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:54:35.443+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:54:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:54:35.443+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:54:35.445 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:54:35.443+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:35 smithi193 ceph-mon[37473]: pgmap v5446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:35 smithi134 ceph-mon[31966]: pgmap v5446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:35 smithi189 ceph-mon[36790]: pgmap v5446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8834.665249 secs 2024-07-26T13:54:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8834.665249 secs 2024-07-26T13:54:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8834.665249 secs 2024-07-26T13:54:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:38 smithi193 ceph-mon[37473]: pgmap v5447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:38 smithi134 ceph-mon[31966]: pgmap v5447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:38 smithi189 ceph-mon[36790]: pgmap v5447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8820.239933 secs 2024-07-26T13:54:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8820.239933 secs 2024-07-26T13:54:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8820.239933 secs 2024-07-26T13:54:40.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:54:40.000+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:54:40.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:54:40.000+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:40.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:54:40.000+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:54:40.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:40 smithi134 ceph-mon[31966]: pgmap v5448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:40 smithi189 ceph-mon[36790]: pgmap v5448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:40 smithi193 ceph-mon[37473]: pgmap v5448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:42 smithi134 ceph-mon[31966]: pgmap v5449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8839.665369 secs 2024-07-26T13:54:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:42 smithi189 ceph-mon[36790]: pgmap v5449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8839.665369 secs 2024-07-26T13:54:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:42 smithi193 ceph-mon[37473]: pgmap v5449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8839.665369 secs 2024-07-26T13:54:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:44 smithi134 ceph-mon[31966]: pgmap v5450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8825.240046 secs 2024-07-26T13:54:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:44 smithi189 ceph-mon[36790]: pgmap v5450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8825.240046 secs 2024-07-26T13:54:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:44 smithi193 ceph-mon[37473]: pgmap v5450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8825.240046 secs 2024-07-26T13:54:46.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:46 smithi193 ceph-mon[37473]: pgmap v5451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:46.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:46 smithi134 ceph-mon[31966]: pgmap v5451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:46 smithi189 ceph-mon[36790]: pgmap v5451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8844.665466 secs 2024-07-26T13:54:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8844.665466 secs 2024-07-26T13:54:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8844.665466 secs 2024-07-26T13:54:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:48 smithi134 ceph-mon[31966]: pgmap v5452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:48 smithi189 ceph-mon[36790]: pgmap v5452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:48 smithi193 ceph-mon[37473]: pgmap v5452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8830.240143 secs 2024-07-26T13:54:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8830.240143 secs 2024-07-26T13:54:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8830.240143 secs 2024-07-26T13:54:50.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:50 smithi134 ceph-mon[31966]: pgmap v5453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:50 smithi189 ceph-mon[36790]: pgmap v5453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:54:50.459+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:54:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:54:50.459+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:50.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:54:50.459+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:54:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:50 smithi193 ceph-mon[37473]: pgmap v5453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:54:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:52 smithi134 ceph-mon[31966]: pgmap v5454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8849.665533 secs 2024-07-26T13:54:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:52 smithi189 ceph-mon[36790]: pgmap v5454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8849.665533 secs 2024-07-26T13:54:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:52 smithi193 ceph-mon[37473]: pgmap v5454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:54:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8849.665533 secs 2024-07-26T13:54:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:54 smithi134 ceph-mon[31966]: pgmap v5455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8835.240211 secs 2024-07-26T13:54:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:54 smithi189 ceph-mon[36790]: pgmap v5455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8835.240211 secs 2024-07-26T13:54:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:54 smithi193 ceph-mon[37473]: pgmap v5455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:54:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8835.240211 secs 2024-07-26T13:54:55.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:54:55.015+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:54:55.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:54:55.015+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:54:55.017 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:54:55.016+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:54:56.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:56 smithi134 ceph-mon[31966]: pgmap v5456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:56 smithi189 ceph-mon[36790]: pgmap v5456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:56 smithi193 ceph-mon[37473]: pgmap v5456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:54:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8854.665655 secs 2024-07-26T13:54:57.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8854.665655 secs 2024-07-26T13:54:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8854.665655 secs 2024-07-26T13:54:58.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:58 smithi134 ceph-mon[31966]: pgmap v5457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:58 smithi189 ceph-mon[36790]: pgmap v5457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:58 smithi193 ceph-mon[37473]: pgmap v5457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:54:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:54:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8840.240328 secs 2024-07-26T13:54:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:54:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8840.240328 secs 2024-07-26T13:54:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:54:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8840.240328 secs 2024-07-26T13:55:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:00 smithi134 ceph-mon[31966]: pgmap v5458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:00 smithi189 ceph-mon[36790]: pgmap v5458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:00 smithi193 ceph-mon[37473]: pgmap v5458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:02 smithi134 ceph-mon[31966]: pgmap v5459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8859.665770 secs 2024-07-26T13:55:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:02 smithi189 ceph-mon[36790]: pgmap v5459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8859.665770 secs 2024-07-26T13:55:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:02 smithi193 ceph-mon[37473]: pgmap v5459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:02.539 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8859.665770 secs 2024-07-26T13:55:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:04 smithi134 ceph-mon[31966]: pgmap v5460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8845.240441 secs 2024-07-26T13:55:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:04 smithi189 ceph-mon[36790]: pgmap v5460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8845.240441 secs 2024-07-26T13:55:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:04 smithi193 ceph-mon[37473]: pgmap v5460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8845.240441 secs 2024-07-26T13:55:05.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:55:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:55:05.478 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:55:05.474+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:55:05.478 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:55:05.474+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:05.478 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:55:05.474+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:55:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:55:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:06 smithi134 ceph-mon[31966]: pgmap v5461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:55:06.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:06 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:55:06.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:06 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:55:06.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:06 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:55:06.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:06 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:55:06.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:55:06.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:55:06.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:55:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:06 smithi189 ceph-mon[36790]: pgmap v5461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:55:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:06 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:55:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:06 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:55:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:06 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:55:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:06 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:55:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:55:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:55:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:55:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:06 smithi193 ceph-mon[37473]: pgmap v5461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:55:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T13:55:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:06 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T13:55:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:06 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T13:55:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:06 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T13:55:06.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:06 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T13:55:06.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:55:06.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:55:06.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:55:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8864.665863 secs 2024-07-26T13:55:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:07 smithi134 ceph-mon[31966]: pgmap v5462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:55:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:07 smithi134 ceph-mon[31966]: pgmap v5463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:55:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8864.665863 secs 2024-07-26T13:55:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:07 smithi189 ceph-mon[36790]: pgmap v5462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:55:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:07 smithi189 ceph-mon[36790]: pgmap v5463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:55:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8864.665863 secs 2024-07-26T13:55:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:07 smithi193 ceph-mon[37473]: pgmap v5462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:55:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:07 smithi193 ceph-mon[37473]: pgmap v5463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:55:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:09 smithi134 ceph-mon[31966]: pgmap v5464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8850.240559 secs 2024-07-26T13:55:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:09 smithi189 ceph-mon[36790]: pgmap v5464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8850.240559 secs 2024-07-26T13:55:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:09 smithi193 ceph-mon[37473]: pgmap v5464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8850.240559 secs 2024-07-26T13:55:10.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:55:10.030+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:55:10.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:55:10.030+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:10.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:55:10.030+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:55:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:11 smithi134 ceph-mon[31966]: pgmap v5465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:11 smithi189 ceph-mon[36790]: pgmap v5465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:11 smithi193 ceph-mon[37473]: pgmap v5465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8869.665981 secs 2024-07-26T13:55:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8869.665981 secs 2024-07-26T13:55:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8869.665981 secs 2024-07-26T13:55:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:13 smithi189 ceph-mon[36790]: pgmap v5466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T13:55:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:13 smithi193 ceph-mon[37473]: pgmap v5466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T13:55:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:13 smithi134 ceph-mon[31966]: pgmap v5466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T13:55:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8855.240636 secs 2024-07-26T13:55:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8855.240636 secs 2024-07-26T13:55:14.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8855.240636 secs 2024-07-26T13:55:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:15 smithi189 ceph-mon[36790]: pgmap v5467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:15 smithi193 ceph-mon[37473]: pgmap v5467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:15 smithi134 ceph-mon[31966]: pgmap v5467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:55:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8874.666105 secs 2024-07-26T13:55:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8874.666105 secs 2024-07-26T13:55:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8874.666105 secs 2024-07-26T13:55:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:17 smithi193 ceph-mon[37473]: pgmap v5468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:55:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:17 smithi134 ceph-mon[31966]: pgmap v5468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:55:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:17 smithi189 ceph-mon[36790]: pgmap v5468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:55:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:19 smithi193 ceph-mon[37473]: pgmap v5469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8860.240731 secs 2024-07-26T13:55:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:19 smithi134 ceph-mon[31966]: pgmap v5469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8860.240731 secs 2024-07-26T13:55:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:19 smithi189 ceph-mon[36790]: pgmap v5469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8860.240731 secs 2024-07-26T13:55:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:55:20.486+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:55:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:55:20.486+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:55:20.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:55:20.486+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:21 smithi193 ceph-mon[37473]: pgmap v5470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8879.666224 secs 2024-07-26T13:55:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:21 smithi134 ceph-mon[31966]: pgmap v5470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8879.666224 secs 2024-07-26T13:55:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:21 smithi189 ceph-mon[36790]: pgmap v5470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8879.666224 secs 2024-07-26T13:55:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:23 smithi193 ceph-mon[37473]: pgmap v5471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:23 smithi134 ceph-mon[31966]: pgmap v5471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:23 smithi189 ceph-mon[36790]: pgmap v5471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8865.240867 secs 2024-07-26T13:55:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8865.240867 secs 2024-07-26T13:55:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8865.240867 secs 2024-07-26T13:55:25.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:55:25.040+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:55:25.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:55:25.040+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:25.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:55:25.040+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:55:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:25 smithi134 ceph-mon[31966]: pgmap v5472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:25 smithi193 ceph-mon[37473]: pgmap v5472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:25 smithi189 ceph-mon[36790]: pgmap v5472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8884.666384 secs 2024-07-26T13:55:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8884.666384 secs 2024-07-26T13:55:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8884.666384 secs 2024-07-26T13:55:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:27 smithi193 ceph-mon[37473]: pgmap v5473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:27 smithi134 ceph-mon[31966]: pgmap v5473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:27 smithi189 ceph-mon[36790]: pgmap v5473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:29 smithi193 ceph-mon[37473]: pgmap v5474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8870.240988 secs 2024-07-26T13:55:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:29 smithi134 ceph-mon[31966]: pgmap v5474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8870.240988 secs 2024-07-26T13:55:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:29 smithi189 ceph-mon[36790]: pgmap v5474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8870.240988 secs 2024-07-26T13:55:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:31 smithi193 ceph-mon[37473]: pgmap v5475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8889.666474 secs 2024-07-26T13:55:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:31 smithi134 ceph-mon[31966]: pgmap v5475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8889.666474 secs 2024-07-26T13:55:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:31 smithi189 ceph-mon[36790]: pgmap v5475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8889.666474 secs 2024-07-26T13:55:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:33 smithi193 ceph-mon[37473]: pgmap v5476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:33.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:33 smithi134 ceph-mon[31966]: pgmap v5476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:33 smithi189 ceph-mon[36790]: pgmap v5476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8875.241109 secs 2024-07-26T13:55:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8875.241109 secs 2024-07-26T13:55:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8875.241109 secs 2024-07-26T13:55:35.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:55:35.501+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:55:35.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:55:35.502+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:55:35.503 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:55:35.502+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:35 smithi134 ceph-mon[31966]: pgmap v5477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:35 smithi189 ceph-mon[36790]: pgmap v5477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:35 smithi193 ceph-mon[37473]: pgmap v5477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8894.666587 secs 2024-07-26T13:55:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8894.666587 secs 2024-07-26T13:55:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8894.666587 secs 2024-07-26T13:55:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:37 smithi134 ceph-mon[31966]: pgmap v5478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:37 smithi189 ceph-mon[36790]: pgmap v5478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:37 smithi193 ceph-mon[37473]: pgmap v5478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:39 smithi134 ceph-mon[31966]: pgmap v5479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8880.241177 secs 2024-07-26T13:55:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:39 smithi189 ceph-mon[36790]: pgmap v5479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8880.241177 secs 2024-07-26T13:55:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:39 smithi193 ceph-mon[37473]: pgmap v5479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8880.241177 secs 2024-07-26T13:55:40.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:55:40.055+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:55:40.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:55:40.056+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:55:40.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:55:40.056+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:41 smithi134 ceph-mon[31966]: pgmap v5480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8899.666654 secs 2024-07-26T13:55:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:41 smithi189 ceph-mon[36790]: pgmap v5480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8899.666654 secs 2024-07-26T13:55:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:41 smithi193 ceph-mon[37473]: pgmap v5480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8899.666654 secs 2024-07-26T13:55:43.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:43 smithi134 ceph-mon[31966]: pgmap v5481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:55:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:43 smithi189 ceph-mon[36790]: pgmap v5481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:55:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:43 smithi193 ceph-mon[37473]: pgmap v5481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:55:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8885.241307 secs 2024-07-26T13:55:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8885.241307 secs 2024-07-26T13:55:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8885.241307 secs 2024-07-26T13:55:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:45 smithi134 ceph-mon[31966]: pgmap v5482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:45 smithi189 ceph-mon[36790]: pgmap v5482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:45 smithi193 ceph-mon[37473]: pgmap v5482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:46.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8904.666770 secs 2024-07-26T13:55:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8904.666770 secs 2024-07-26T13:55:46.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8904.666770 secs 2024-07-26T13:55:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:47 smithi134 ceph-mon[31966]: pgmap v5483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:47 smithi189 ceph-mon[36790]: pgmap v5483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:47 smithi193 ceph-mon[37473]: pgmap v5483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:49 smithi134 ceph-mon[31966]: pgmap v5484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8890.241399 secs 2024-07-26T13:55:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:49 smithi189 ceph-mon[36790]: pgmap v5484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8890.241399 secs 2024-07-26T13:55:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:49 smithi193 ceph-mon[37473]: pgmap v5484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8890.241399 secs 2024-07-26T13:55:50.514 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:55:50.513+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:55:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:55:50.513+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:55:50.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:55:50.513+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:51 smithi134 ceph-mon[31966]: pgmap v5485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8909.666847 secs 2024-07-26T13:55:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:51 smithi189 ceph-mon[36790]: pgmap v5485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8909.666847 secs 2024-07-26T13:55:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:51 smithi193 ceph-mon[37473]: pgmap v5485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8909.666847 secs 2024-07-26T13:55:53.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:53 smithi134 ceph-mon[31966]: pgmap v5486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:53 smithi189 ceph-mon[36790]: pgmap v5486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:53 smithi193 ceph-mon[37473]: pgmap v5486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:55:53.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:55:53.875+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T13:55:53.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:55:53.875+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T13:55:53.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:55:53.875+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T13:55:53.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:55:53.875+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T13:55:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8895.241515 secs 2024-07-26T13:55:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8895.241515 secs 2024-07-26T13:55:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8895.241515 secs 2024-07-26T13:55:55.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:55:55.071+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:55:55.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:55:55.071+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:55:55.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:55:55.071+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:55:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:55 smithi134 ceph-mon[31966]: pgmap v5487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:55 smithi189 ceph-mon[36790]: pgmap v5487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:55 smithi193 ceph-mon[37473]: pgmap v5487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:55:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8914.666964 secs 2024-07-26T13:55:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8914.666964 secs 2024-07-26T13:55:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8914.666964 secs 2024-07-26T13:55:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:57 smithi134 ceph-mon[31966]: pgmap v5488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:57 smithi189 ceph-mon[36790]: pgmap v5488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:57 smithi193 ceph-mon[37473]: pgmap v5488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:55:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:59 smithi134 ceph-mon[31966]: pgmap v5489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:55:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8900.241637 secs 2024-07-26T13:55:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:59 smithi189 ceph-mon[36790]: pgmap v5489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:55:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8900.241637 secs 2024-07-26T13:55:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:59 smithi193 ceph-mon[37473]: pgmap v5489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:55:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:55:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8900.241637 secs 2024-07-26T13:56:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:01 smithi134 ceph-mon[31966]: pgmap v5490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8919.667083 secs 2024-07-26T13:56:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:01 smithi189 ceph-mon[36790]: pgmap v5490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8919.667083 secs 2024-07-26T13:56:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:01 smithi193 ceph-mon[37473]: pgmap v5490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8919.667083 secs 2024-07-26T13:56:03.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:03 smithi134 ceph-mon[31966]: pgmap v5491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:03 smithi189 ceph-mon[36790]: pgmap v5491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:03 smithi193 ceph-mon[37473]: pgmap v5491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:56:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:56:04.402+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:56:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:56:04.402+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T13:56:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:56:04.402+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:56:04.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:56:04.402+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T13:56:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8905.241780 secs 2024-07-26T13:56:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8905.241780 secs 2024-07-26T13:56:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8905.241780 secs 2024-07-26T13:56:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:56:05.519+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:56:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:56:05.519+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:56:05.519+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:56:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:05 smithi189 ceph-mon[36790]: pgmap v5492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:05 smithi193 ceph-mon[37473]: pgmap v5492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:05.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:05 smithi134 ceph-mon[31966]: pgmap v5492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:56:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8924.667200 secs 2024-07-26T13:56:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:56:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8924.667200 secs 2024-07-26T13:56:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:56:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8924.667200 secs 2024-07-26T13:56:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:56:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:07 smithi193 ceph-mon[37473]: pgmap v5493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:56:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:56:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:56:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:07 smithi134 ceph-mon[31966]: pgmap v5493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:56:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:56:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:56:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:07 smithi189 ceph-mon[36790]: pgmap v5493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:56:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:56:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:56:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:08 smithi193 ceph-mon[37473]: pgmap v5494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T13:56:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:08 smithi193 ceph-mon[37473]: pgmap v5495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:56:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:08 smithi134 ceph-mon[31966]: pgmap v5494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T13:56:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:08 smithi134 ceph-mon[31966]: pgmap v5495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:56:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:08 smithi189 ceph-mon[36790]: pgmap v5494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T13:56:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:08 smithi189 ceph-mon[36790]: pgmap v5495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:56:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8910.241866 secs 2024-07-26T13:56:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8910.241866 secs 2024-07-26T13:56:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8910.241866 secs 2024-07-26T13:56:10.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:56:10.077+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:56:10.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:56:10.077+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:56:10.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:56:10.077+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:10 smithi193 ceph-mon[37473]: pgmap v5496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:56:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:10 smithi134 ceph-mon[31966]: pgmap v5496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:56:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:10 smithi189 ceph-mon[36790]: pgmap v5496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:56:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8929.667334 secs 2024-07-26T13:56:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8929.667334 secs 2024-07-26T13:56:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8929.667334 secs 2024-07-26T13:56:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:12 smithi193 ceph-mon[37473]: pgmap v5497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:56:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:12 smithi134 ceph-mon[31966]: pgmap v5497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:56:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:12 smithi189 ceph-mon[36790]: pgmap v5497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T13:56:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:14 smithi193 ceph-mon[37473]: pgmap v5498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:56:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8915.242013 secs 2024-07-26T13:56:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:14 smithi134 ceph-mon[31966]: pgmap v5498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:56:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8915.242013 secs 2024-07-26T13:56:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:14 smithi189 ceph-mon[36790]: pgmap v5498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:56:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8915.242013 secs 2024-07-26T13:56:16.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:16 smithi189 ceph-mon[36790]: pgmap v5499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:56:16.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8934.667416 secs 2024-07-26T13:56:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:16 smithi193 ceph-mon[37473]: pgmap v5499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:56:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8934.667416 secs 2024-07-26T13:56:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:16 smithi134 ceph-mon[31966]: pgmap v5499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:56:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8934.667416 secs 2024-07-26T13:56:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:18 smithi193 ceph-mon[37473]: pgmap v5500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:56:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:18 smithi134 ceph-mon[31966]: pgmap v5500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:56:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:18 smithi189 ceph-mon[36790]: pgmap v5500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:56:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8920.242091 secs 2024-07-26T13:56:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8920.242091 secs 2024-07-26T13:56:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8920.242091 secs 2024-07-26T13:56:20.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:56:20.526+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:56:20.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:56:20.526+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:20.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:56:20.526+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:56:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:20 smithi193 ceph-mon[37473]: pgmap v5501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:20 smithi134 ceph-mon[31966]: pgmap v5501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:20 smithi189 ceph-mon[36790]: pgmap v5501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8939.667479 secs 2024-07-26T13:56:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8939.667479 secs 2024-07-26T13:56:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8939.667479 secs 2024-07-26T13:56:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:22 smithi193 ceph-mon[37473]: pgmap v5502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:22 smithi134 ceph-mon[31966]: pgmap v5502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:22 smithi189 ceph-mon[36790]: pgmap v5502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:24 smithi134 ceph-mon[31966]: pgmap v5503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:56:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8925.242211 secs 2024-07-26T13:56:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:24 smithi189 ceph-mon[36790]: pgmap v5503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:56:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8925.242211 secs 2024-07-26T13:56:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:24 smithi193 ceph-mon[37473]: pgmap v5503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:56:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8925.242211 secs 2024-07-26T13:56:25.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:56:25.093+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:56:25.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:56:25.093+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:56:25.094 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:56:25.093+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:26 smithi134 ceph-mon[31966]: pgmap v5504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8944.667569 secs 2024-07-26T13:56:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:26 smithi189 ceph-mon[36790]: pgmap v5504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8944.667569 secs 2024-07-26T13:56:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:26 smithi193 ceph-mon[37473]: pgmap v5504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8944.667569 secs 2024-07-26T13:56:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:28 smithi134 ceph-mon[31966]: pgmap v5505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:28 smithi189 ceph-mon[36790]: pgmap v5505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:28 smithi193 ceph-mon[37473]: pgmap v5505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8930.242277 secs 2024-07-26T13:56:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8930.242277 secs 2024-07-26T13:56:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8930.242277 secs 2024-07-26T13:56:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:30 smithi134 ceph-mon[31966]: pgmap v5506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:30 smithi189 ceph-mon[36790]: pgmap v5506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:30 smithi193 ceph-mon[37473]: pgmap v5506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8949.667692 secs 2024-07-26T13:56:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8949.667692 secs 2024-07-26T13:56:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8949.667692 secs 2024-07-26T13:56:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:32 smithi134 ceph-mon[31966]: pgmap v5507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:32 smithi189 ceph-mon[36790]: pgmap v5507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:32 smithi193 ceph-mon[37473]: pgmap v5507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:34 smithi134 ceph-mon[31966]: pgmap v5508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:56:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8935.242396 secs 2024-07-26T13:56:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:34 smithi189 ceph-mon[36790]: pgmap v5508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:56:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8935.242396 secs 2024-07-26T13:56:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:34 smithi193 ceph-mon[37473]: pgmap v5508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:56:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8935.242396 secs 2024-07-26T13:56:35.541 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:56:35.540+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:56:35.541 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:56:35.540+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:35.541 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:56:35.540+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:56:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:36 smithi134 ceph-mon[31966]: pgmap v5509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8954.667818 secs 2024-07-26T13:56:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:36 smithi189 ceph-mon[36790]: pgmap v5509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8954.667818 secs 2024-07-26T13:56:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:36 smithi193 ceph-mon[37473]: pgmap v5509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8954.667818 secs 2024-07-26T13:56:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:38 smithi134 ceph-mon[31966]: pgmap v5510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:38 smithi189 ceph-mon[36790]: pgmap v5510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:38 smithi193 ceph-mon[37473]: pgmap v5510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8940.242517 secs 2024-07-26T13:56:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8940.242517 secs 2024-07-26T13:56:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8940.242517 secs 2024-07-26T13:56:40.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:56:40.108+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:56:40.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:56:40.108+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:56:40.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:56:40.108+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:40 smithi134 ceph-mon[31966]: pgmap v5511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:40 smithi189 ceph-mon[36790]: pgmap v5511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:40 smithi193 ceph-mon[37473]: pgmap v5511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:41.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8959.667955 secs 2024-07-26T13:56:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8959.667955 secs 2024-07-26T13:56:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8959.667955 secs 2024-07-26T13:56:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:42 smithi134 ceph-mon[31966]: pgmap v5512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:42 smithi189 ceph-mon[36790]: pgmap v5512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:42 smithi193 ceph-mon[37473]: pgmap v5512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:44 smithi134 ceph-mon[31966]: pgmap v5513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:56:44.941 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8945.242623 secs 2024-07-26T13:56:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:44 smithi189 ceph-mon[36790]: pgmap v5513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:56:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8945.242623 secs 2024-07-26T13:56:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:44 smithi193 ceph-mon[37473]: pgmap v5513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:56:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8945.242623 secs 2024-07-26T13:56:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:46 smithi134 ceph-mon[31966]: pgmap v5514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8964.668068 secs 2024-07-26T13:56:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:46 smithi189 ceph-mon[36790]: pgmap v5514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8964.668068 secs 2024-07-26T13:56:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:46 smithi193 ceph-mon[37473]: pgmap v5514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8964.668068 secs 2024-07-26T13:56:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:48 smithi134 ceph-mon[31966]: pgmap v5515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:48 smithi189 ceph-mon[36790]: pgmap v5515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:48 smithi193 ceph-mon[37473]: pgmap v5515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8950.242767 secs 2024-07-26T13:56:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8950.242767 secs 2024-07-26T13:56:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8950.242767 secs 2024-07-26T13:56:50.547 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:56:50.547+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:56:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:56:50.547+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:50.548 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:56:50.547+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:56:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:50 smithi134 ceph-mon[31966]: pgmap v5516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:50 smithi189 ceph-mon[36790]: pgmap v5516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:50 smithi193 ceph-mon[37473]: pgmap v5516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:56:51.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8969.668188 secs 2024-07-26T13:56:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8969.668188 secs 2024-07-26T13:56:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8969.668188 secs 2024-07-26T13:56:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:52 smithi134 ceph-mon[31966]: pgmap v5517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:52 smithi189 ceph-mon[36790]: pgmap v5517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:52 smithi193 ceph-mon[37473]: pgmap v5517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:54 smithi134 ceph-mon[31966]: pgmap v5518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:56:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8955.242887 secs 2024-07-26T13:56:54.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:54 smithi189 ceph-mon[36790]: pgmap v5518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:56:54.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8955.242887 secs 2024-07-26T13:56:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:54 smithi193 ceph-mon[37473]: pgmap v5518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:56:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8955.242887 secs 2024-07-26T13:56:55.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:56:55.114+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:56:55.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:56:55.114+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:56:55.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:56:55.114+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:56:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:56 smithi134 ceph-mon[31966]: pgmap v5519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8974.668261 secs 2024-07-26T13:56:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:56 smithi189 ceph-mon[36790]: pgmap v5519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8974.668261 secs 2024-07-26T13:56:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:56 smithi193 ceph-mon[37473]: pgmap v5519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:56:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8974.668261 secs 2024-07-26T13:56:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:58 smithi189 ceph-mon[36790]: pgmap v5520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:56:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8960.243027 secs 2024-07-26T13:56:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:58 smithi193 ceph-mon[37473]: pgmap v5520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:56:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8960.243027 secs 2024-07-26T13:56:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:58 smithi134 ceph-mon[31966]: pgmap v5520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:56:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:56:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8960.243027 secs 2024-07-26T13:57:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:00 smithi189 ceph-mon[36790]: pgmap v5521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:00 smithi193 ceph-mon[37473]: pgmap v5521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:00 smithi134 ceph-mon[31966]: pgmap v5521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8979.668381 secs 2024-07-26T13:57:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8979.668381 secs 2024-07-26T13:57:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8979.668381 secs 2024-07-26T13:57:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:02 smithi193 ceph-mon[37473]: pgmap v5522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:02 smithi134 ceph-mon[31966]: pgmap v5522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:02 smithi189 ceph-mon[36790]: pgmap v5522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8965.243142 secs 2024-07-26T13:57:04.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8965.243142 secs 2024-07-26T13:57:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8965.243142 secs 2024-07-26T13:57:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:04 smithi193 ceph-mon[37473]: pgmap v5523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:04 smithi134 ceph-mon[31966]: pgmap v5523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:04 smithi189 ceph-mon[36790]: pgmap v5523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:05.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:05.562+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:57:05.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:05.562+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:05.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:05.562+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:57:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:06 smithi193 ceph-mon[37473]: pgmap v5524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8984.668515 secs 2024-07-26T13:57:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:06 smithi134 ceph-mon[31966]: pgmap v5524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8984.668515 secs 2024-07-26T13:57:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:06 smithi189 ceph-mon[36790]: pgmap v5524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8984.668515 secs 2024-07-26T13:57:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:57:08.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:57:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:57:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:08 smithi193 ceph-mon[37473]: pgmap v5525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:57:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:57:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:57:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8970.243265 secs 2024-07-26T13:57:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:08 smithi134 ceph-mon[31966]: pgmap v5525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:57:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:57:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:57:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8970.243265 secs 2024-07-26T13:57:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:08 smithi189 ceph-mon[36790]: pgmap v5525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:57:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:57:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:57:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8970.243265 secs 2024-07-26T13:57:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:57:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:57:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:57:10.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:57:10.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:57:10.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:57:10.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:10.130+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:57:10.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:10.130+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:57:10.131 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:10.130+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:57:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:57:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:57:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:10 smithi193 ceph-mon[37473]: pgmap v5526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:10 smithi193 ceph-mon[37473]: pgmap v5527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:57:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:10 smithi134 ceph-mon[31966]: pgmap v5526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:10 smithi134 ceph-mon[31966]: pgmap v5527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:57:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:10 smithi189 ceph-mon[36790]: pgmap v5526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:57:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:10 smithi189 ceph-mon[36790]: pgmap v5527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:57:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8989.668631 secs 2024-07-26T13:57:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8989.668631 secs 2024-07-26T13:57:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8989.668631 secs 2024-07-26T13:57:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:12 smithi193 ceph-mon[37473]: pgmap v5528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:57:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:12 smithi134 ceph-mon[31966]: pgmap v5528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:57:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:12 smithi189 ceph-mon[36790]: pgmap v5528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:57:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8975.243420 secs 2024-07-26T13:57:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8975.243420 secs 2024-07-26T13:57:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8975.243420 secs 2024-07-26T13:57:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:14 smithi193 ceph-mon[37473]: pgmap v5529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:57:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:14 smithi134 ceph-mon[31966]: pgmap v5529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:57:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:14 smithi189 ceph-mon[36790]: pgmap v5529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:57:16.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:15 smithi189 ceph-mon[36790]: pgmap v5530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:57:16.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:15 smithi134 ceph-mon[31966]: pgmap v5530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:57:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:15 smithi193 ceph-mon[37473]: pgmap v5530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T13:57:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8994.668751 secs 2024-07-26T13:57:17.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8994.668751 secs 2024-07-26T13:57:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8994.668751 secs 2024-07-26T13:57:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:17 smithi134 ceph-mon[31966]: pgmap v5531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:57:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:17 smithi189 ceph-mon[36790]: pgmap v5531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:57:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:17 smithi193 ceph-mon[37473]: pgmap v5531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:57:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8980.243548 secs 2024-07-26T13:57:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8980.243548 secs 2024-07-26T13:57:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8980.243548 secs 2024-07-26T13:57:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:19 smithi134 ceph-mon[31966]: pgmap v5532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:57:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:19 smithi189 ceph-mon[36790]: pgmap v5532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:57:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:19 smithi193 ceph-mon[37473]: pgmap v5532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:57:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:20.578+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:57:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:20.578+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:20.582 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:20.578+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:57:22.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:22.315+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000021) for more than 900000000 us, fault. 2024-07-26T13:57:22.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:22.315+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b3805d040 comp rx=0 tx=0).stop 2024-07-26T13:57:22.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:22.315+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:57:22.316 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:22.315+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:57:22.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:22.315+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 0x7f9b300c1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:57:22.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:22.315+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 0x7f9b300c1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:57:22.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:22.317+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 0x7f9b300c1940 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009770 tx=0x7f9b38008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:57:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 8999.668869 secs 2024-07-26T13:57:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:22 smithi193 ceph-mon[37473]: pgmap v5533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 8999.668869 secs 2024-07-26T13:57:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:22 smithi134 ceph-mon[31966]: pgmap v5533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 8999.668869 secs 2024-07-26T13:57:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:22 smithi189 ceph-mon[36790]: pgmap v5533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:23.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:23.892+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T13:57:23.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:23.892+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T13:57:23.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:23.892+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T13:57:23.893 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:23.892+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T13:57:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:24 smithi193 ceph-mon[37473]: pgmap v5534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8985.243679 secs 2024-07-26T13:57:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:24 smithi134 ceph-mon[31966]: pgmap v5534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8985.243679 secs 2024-07-26T13:57:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:24 smithi189 ceph-mon[36790]: pgmap v5534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8985.243679 secs 2024-07-26T13:57:25.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:25.133+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:57:25.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:25.133+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:25.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:25.133+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:57:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:26 smithi193 ceph-mon[37473]: pgmap v5535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9004.668987 secs 2024-07-26T13:57:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:26 smithi134 ceph-mon[31966]: pgmap v5535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9004.668987 secs 2024-07-26T13:57:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:26 smithi189 ceph-mon[36790]: pgmap v5535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9004.668987 secs 2024-07-26T13:57:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:28 smithi193 ceph-mon[37473]: pgmap v5536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:28 smithi134 ceph-mon[31966]: pgmap v5536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:28 smithi189 ceph-mon[36790]: pgmap v5536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8990.243797 secs 2024-07-26T13:57:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8990.243797 secs 2024-07-26T13:57:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8990.243797 secs 2024-07-26T13:57:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:30 smithi193 ceph-mon[37473]: pgmap v5537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:30 smithi134 ceph-mon[31966]: pgmap v5537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:30 smithi189 ceph-mon[36790]: pgmap v5537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9009.669122 secs 2024-07-26T13:57:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9009.669122 secs 2024-07-26T13:57:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9009.669122 secs 2024-07-26T13:57:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:32 smithi193 ceph-mon[37473]: pgmap v5538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:32.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:32 smithi134 ceph-mon[31966]: pgmap v5538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:32 smithi189 ceph-mon[36790]: pgmap v5538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:32.869+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 msgr2=0x7f21ac0c14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T13:57:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:32.869+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 msgr2=0x7f21ac0c14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T13:57:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:32.869+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f21980063a0 tx=0x7f21b80ba000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T13:57:32.870 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:32.869+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f21980063a0 tx=0x7f21b80ba000 comp rx=0 tx=0).stop 2024-07-26T13:57:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:32.870+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 msgr2=0x7f21ac0c14c0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T13:57:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:32.870+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 msgr2=0x7f21ac0c14c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T13:57:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:32.870+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T13:57:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:32.870+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 0x7f21ac0c0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T13:57:32.871 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:32.871+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 0x7f21ac0c0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T13:57:32.873 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:32.873+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 0x7f21ac0c0640 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2198005820 tx=0x7f21b8063000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T13:57:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:34.417+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:57:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:34.417+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T13:57:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:34.418+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T13:57:34.419 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:34.418+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T13:57:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:34 smithi193 ceph-mon[37473]: pgmap v5539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 8995.243934 secs 2024-07-26T13:57:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:34 smithi134 ceph-mon[31966]: pgmap v5539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 8995.243934 secs 2024-07-26T13:57:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:34 smithi189 ceph-mon[36790]: pgmap v5539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 8995.243934 secs 2024-07-26T13:57:35.591 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:35.590+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:57:35.591 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:35.590+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:35.592 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:35.590+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:57:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:36 smithi193 ceph-mon[37473]: pgmap v5540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9014.669217 secs 2024-07-26T13:57:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:36 smithi134 ceph-mon[31966]: pgmap v5540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9014.669217 secs 2024-07-26T13:57:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:36 smithi189 ceph-mon[36790]: pgmap v5540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9014.669217 secs 2024-07-26T13:57:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:38 smithi193 ceph-mon[37473]: pgmap v5541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:38.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:38 smithi134 ceph-mon[31966]: pgmap v5541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:38 smithi189 ceph-mon[36790]: pgmap v5541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:57:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9000.244083 secs 2024-07-26T13:57:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9000.244083 secs 2024-07-26T13:57:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9000.244083 secs 2024-07-26T13:57:40.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:40.149+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:57:40.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:40.149+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:57:40.150 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:40.149+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:40 smithi134 ceph-mon[31966]: pgmap v5542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:40 smithi189 ceph-mon[36790]: pgmap v5542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:40 smithi193 ceph-mon[37473]: pgmap v5542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:57:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9019.669338 secs 2024-07-26T13:57:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9019.669338 secs 2024-07-26T13:57:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9019.669338 secs 2024-07-26T13:57:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:42 smithi134 ceph-mon[31966]: pgmap v5543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:42 smithi189 ceph-mon[36790]: pgmap v5543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:42 smithi193 ceph-mon[37473]: pgmap v5543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:44 smithi134 ceph-mon[31966]: pgmap v5544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9005.244179 secs 2024-07-26T13:57:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:44 smithi189 ceph-mon[36790]: pgmap v5544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9005.244179 secs 2024-07-26T13:57:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:44 smithi193 ceph-mon[37473]: pgmap v5544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:57:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9005.244179 secs 2024-07-26T13:57:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:46 smithi134 ceph-mon[31966]: pgmap v5545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9024.669452 secs 2024-07-26T13:57:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:46 smithi189 ceph-mon[36790]: pgmap v5545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9024.669452 secs 2024-07-26T13:57:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:46 smithi193 ceph-mon[37473]: pgmap v5545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:57:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9024.669452 secs 2024-07-26T13:57:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:48 smithi189 ceph-mon[36790]: pgmap v5546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:57:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:48 smithi193 ceph-mon[37473]: pgmap v5546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:57:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:48 smithi134 ceph-mon[31966]: pgmap v5546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:57:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9010.244309 secs 2024-07-26T13:57:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9010.244309 secs 2024-07-26T13:57:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9010.244309 secs 2024-07-26T13:57:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:50.606+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:57:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:50.606+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:57:50.607 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:57:50.606+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:50 smithi189 ceph-mon[36790]: pgmap v5547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:57:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:50 smithi193 ceph-mon[37473]: pgmap v5547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:57:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:50 smithi134 ceph-mon[31966]: pgmap v5547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T13:57:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9029.669573 secs 2024-07-26T13:57:51.880 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9029.669573 secs 2024-07-26T13:57:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9029.669573 secs 2024-07-26T13:57:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:52 smithi193 ceph-mon[37473]: pgmap v5548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:57:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:52 smithi134 ceph-mon[31966]: pgmap v5548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:57:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:52 smithi189 ceph-mon[36790]: pgmap v5548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:57:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:54 smithi193 ceph-mon[37473]: pgmap v5549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:57:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9015.244403 secs 2024-07-26T13:57:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:54 smithi134 ceph-mon[31966]: pgmap v5549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:57:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9015.244403 secs 2024-07-26T13:57:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:54 smithi189 ceph-mon[36790]: pgmap v5549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T13:57:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9015.244403 secs 2024-07-26T13:57:55.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:55.159+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:57:55.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:55.159+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:57:55.161 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:57:55.159+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:57:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:56 smithi193 ceph-mon[37473]: pgmap v5550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:57:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9034.669692 secs 2024-07-26T13:57:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:56 smithi134 ceph-mon[31966]: pgmap v5550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:57:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9034.669692 secs 2024-07-26T13:57:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:56 smithi189 ceph-mon[36790]: pgmap v5550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T13:57:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9034.669692 secs 2024-07-26T13:57:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:58 smithi193 ceph-mon[37473]: pgmap v5551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:57:58.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:58 smithi134 ceph-mon[31966]: pgmap v5551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:57:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:58 smithi189 ceph-mon[36790]: pgmap v5551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:57:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:57:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9020.244527 secs 2024-07-26T13:57:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:57:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9020.244527 secs 2024-07-26T13:57:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:57:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9020.244527 secs 2024-07-26T13:58:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:00 smithi193 ceph-mon[37473]: pgmap v5552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:00 smithi134 ceph-mon[31966]: pgmap v5552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:00 smithi189 ceph-mon[36790]: pgmap v5552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9039.669834 secs 2024-07-26T13:58:01.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9039.669834 secs 2024-07-26T13:58:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9039.669834 secs 2024-07-26T13:58:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:02 smithi193 ceph-mon[37473]: pgmap v5553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:02 smithi134 ceph-mon[31966]: pgmap v5553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:02 smithi189 ceph-mon[36790]: pgmap v5553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:04 smithi193 ceph-mon[37473]: pgmap v5554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9025.244650 secs 2024-07-26T13:58:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:04 smithi134 ceph-mon[31966]: pgmap v5554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9025.244650 secs 2024-07-26T13:58:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:04 smithi189 ceph-mon[36790]: pgmap v5554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9025.244650 secs 2024-07-26T13:58:05.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:58:05.609+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:58:05.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:58:05.609+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:05.611 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:58:05.609+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:58:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:06 smithi193 ceph-mon[37473]: pgmap v5555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9044.669952 secs 2024-07-26T13:58:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:06 smithi134 ceph-mon[31966]: pgmap v5555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9044.669952 secs 2024-07-26T13:58:06.947 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:06 smithi189 ceph-mon[36790]: pgmap v5555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:06.947 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9044.669952 secs 2024-07-26T13:58:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:08 smithi193 ceph-mon[37473]: pgmap v5556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:08 smithi134 ceph-mon[31966]: pgmap v5556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:08 smithi189 ceph-mon[36790]: pgmap v5556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9030.244771 secs 2024-07-26T13:58:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:58:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9030.244771 secs 2024-07-26T13:58:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:58:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9030.244771 secs 2024-07-26T13:58:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:58:10.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:58:10.172+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:58:10.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:58:10.172+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:10.174 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:58:10.172+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:58:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:10 smithi134 ceph-mon[31966]: pgmap v5557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:10.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:10 smithi193 ceph-mon[37473]: pgmap v5557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:10 smithi189 ceph-mon[36790]: pgmap v5557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9049.670024 secs 2024-07-26T13:58:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9049.670024 secs 2024-07-26T13:58:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9049.670024 secs 2024-07-26T13:58:12.213 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:12.213 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:12.213 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:12.213 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:12 smithi193 ceph-mon[37473]: pgmap v5558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:12 smithi134 ceph-mon[31966]: pgmap v5558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:12 smithi189 ceph-mon[36790]: pgmap v5558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:14 smithi134 ceph-mon[31966]: pgmap v5559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9035.244893 secs 2024-07-26T13:58:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:14 smithi189 ceph-mon[36790]: pgmap v5559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9035.244893 secs 2024-07-26T13:58:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:14 smithi193 ceph-mon[37473]: pgmap v5559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9035.244893 secs 2024-07-26T13:58:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:58:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:58:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:15 smithi189 ceph-mon[36790]: pgmap v5560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:58:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:15 smithi189 ceph-mon[36790]: pgmap v5561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:58:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:58:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:58:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:15 smithi193 ceph-mon[37473]: pgmap v5560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:58:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:15 smithi193 ceph-mon[37473]: pgmap v5561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:58:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:58:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:58:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:15 smithi134 ceph-mon[31966]: pgmap v5560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T13:58:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:15 smithi134 ceph-mon[31966]: pgmap v5561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:58:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:58:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9054.670165 secs 2024-07-26T13:58:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9054.670165 secs 2024-07-26T13:58:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9054.670165 secs 2024-07-26T13:58:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:17 smithi134 ceph-mon[31966]: pgmap v5562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T13:58:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:17 smithi189 ceph-mon[36790]: pgmap v5562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T13:58:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:17 smithi193 ceph-mon[37473]: pgmap v5562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T13:58:19.576 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:19 smithi193 ceph-mon[37473]: pgmap v5563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:58:19.576 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9040.245055 secs 2024-07-26T13:58:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:19 smithi134 ceph-mon[31966]: pgmap v5563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:58:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9040.245055 secs 2024-07-26T13:58:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:19 smithi189 ceph-mon[36790]: pgmap v5563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:58:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9040.245055 secs 2024-07-26T13:58:20.624 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:58:20.623+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:58:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:58:20.623+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:20.625 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:58:20.623+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:58:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:21 smithi134 ceph-mon[31966]: pgmap v5564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:58:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9059.670288 secs 2024-07-26T13:58:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:21 smithi189 ceph-mon[36790]: pgmap v5564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:58:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9059.670288 secs 2024-07-26T13:58:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:21 smithi193 ceph-mon[37473]: pgmap v5564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:58:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9059.670288 secs 2024-07-26T13:58:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:23 smithi134 ceph-mon[31966]: pgmap v5565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T13:58:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:23 smithi189 ceph-mon[36790]: pgmap v5565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T13:58:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:23 smithi193 ceph-mon[37473]: pgmap v5565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T13:58:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9045.245194 secs 2024-07-26T13:58:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9045.245194 secs 2024-07-26T13:58:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9045.245194 secs 2024-07-26T13:58:25.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:58:25.180+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:58:25.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:58:25.180+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:58:25.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:58:25.180+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:25 smithi134 ceph-mon[31966]: pgmap v5566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:58:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:25 smithi189 ceph-mon[36790]: pgmap v5566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:58:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:25 smithi193 ceph-mon[37473]: pgmap v5566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:58:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9064.670402 secs 2024-07-26T13:58:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9064.670402 secs 2024-07-26T13:58:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9064.670402 secs 2024-07-26T13:58:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:27 smithi134 ceph-mon[31966]: pgmap v5567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:27 smithi189 ceph-mon[36790]: pgmap v5567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:27 smithi193 ceph-mon[37473]: pgmap v5567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:29 smithi134 ceph-mon[31966]: pgmap v5568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9050.245342 secs 2024-07-26T13:58:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:29 smithi189 ceph-mon[36790]: pgmap v5568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9050.245342 secs 2024-07-26T13:58:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:29 smithi193 ceph-mon[37473]: pgmap v5568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9050.245342 secs 2024-07-26T13:58:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:31 smithi134 ceph-mon[31966]: pgmap v5569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9069.670469 secs 2024-07-26T13:58:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:31 smithi189 ceph-mon[36790]: pgmap v5569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9069.670469 secs 2024-07-26T13:58:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:31 smithi193 ceph-mon[37473]: pgmap v5569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9069.670469 secs 2024-07-26T13:58:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:33 smithi134 ceph-mon[31966]: pgmap v5570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:58:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:33 smithi189 ceph-mon[36790]: pgmap v5570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:58:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:33 smithi193 ceph-mon[37473]: pgmap v5570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T13:58:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9055.245469 secs 2024-07-26T13:58:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9055.245469 secs 2024-07-26T13:58:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9055.245469 secs 2024-07-26T13:58:35.633 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:58:35.632+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:58:35.633 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:58:35.632+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:35.634 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:58:35.632+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:58:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:35 smithi134 ceph-mon[31966]: pgmap v5571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:35 smithi189 ceph-mon[36790]: pgmap v5571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:35 smithi193 ceph-mon[37473]: pgmap v5571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9074.670581 secs 2024-07-26T13:58:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9074.670581 secs 2024-07-26T13:58:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9074.670581 secs 2024-07-26T13:58:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:37 smithi134 ceph-mon[31966]: pgmap v5572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:37 smithi189 ceph-mon[36790]: pgmap v5572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:37 smithi193 ceph-mon[37473]: pgmap v5572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:39 smithi134 ceph-mon[31966]: pgmap v5573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9060.245617 secs 2024-07-26T13:58:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:39 smithi189 ceph-mon[36790]: pgmap v5573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9060.245617 secs 2024-07-26T13:58:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:39 smithi193 ceph-mon[37473]: pgmap v5573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9060.245617 secs 2024-07-26T13:58:40.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:58:40.193+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:58:40.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:58:40.193+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:58:40.195 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:58:40.193+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:41 smithi134 ceph-mon[31966]: pgmap v5574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9079.670671 secs 2024-07-26T13:58:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:41 smithi189 ceph-mon[36790]: pgmap v5574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9079.670671 secs 2024-07-26T13:58:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:41 smithi193 ceph-mon[37473]: pgmap v5574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9079.670671 secs 2024-07-26T13:58:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:43 smithi134 ceph-mon[31966]: pgmap v5575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:43 smithi189 ceph-mon[36790]: pgmap v5575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:43 smithi193 ceph-mon[37473]: pgmap v5575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T13:58:44.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9065.245757 secs 2024-07-26T13:58:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9065.245757 secs 2024-07-26T13:58:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9065.245757 secs 2024-07-26T13:58:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:45 smithi134 ceph-mon[31966]: pgmap v5576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:45 smithi189 ceph-mon[36790]: pgmap v5576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:45 smithi193 ceph-mon[37473]: pgmap v5576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9084.670791 secs 2024-07-26T13:58:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9084.670791 secs 2024-07-26T13:58:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9084.670791 secs 2024-07-26T13:58:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:47 smithi134 ceph-mon[31966]: pgmap v5577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:47 smithi189 ceph-mon[36790]: pgmap v5577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:47 smithi193 ceph-mon[37473]: pgmap v5577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:49 smithi189 ceph-mon[36790]: pgmap v5578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9070.245890 secs 2024-07-26T13:58:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:49 smithi193 ceph-mon[37473]: pgmap v5578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9070.245890 secs 2024-07-26T13:58:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:49 smithi134 ceph-mon[31966]: pgmap v5578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9070.245890 secs 2024-07-26T13:58:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:58:50.635+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:58:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:58:50.635+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:50.636 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:58:50.635+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:58:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:51 smithi134 ceph-mon[31966]: pgmap v5579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9089.670914 secs 2024-07-26T13:58:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:51 smithi189 ceph-mon[36790]: pgmap v5579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9089.670914 secs 2024-07-26T13:58:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:51 smithi193 ceph-mon[37473]: pgmap v5579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9089.670914 secs 2024-07-26T13:58:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:53 smithi189 ceph-mon[36790]: pgmap v5580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:58:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:53 smithi193 ceph-mon[37473]: pgmap v5580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:58:53.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:53 smithi134 ceph-mon[31966]: pgmap v5580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:58:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9075.246030 secs 2024-07-26T13:58:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9075.246030 secs 2024-07-26T13:58:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9075.246030 secs 2024-07-26T13:58:55.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:58:55.197+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:58:55.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:58:55.197+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:58:55.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:58:55.197+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:58:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:55 smithi134 ceph-mon[31966]: pgmap v5581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:55 smithi193 ceph-mon[37473]: pgmap v5581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:55 smithi189 ceph-mon[36790]: pgmap v5581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T13:58:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9094.671038 secs 2024-07-26T13:58:56.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9094.671038 secs 2024-07-26T13:58:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9094.671038 secs 2024-07-26T13:58:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:57 smithi193 ceph-mon[37473]: pgmap v5582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:57 smithi134 ceph-mon[31966]: pgmap v5582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:57 smithi189 ceph-mon[36790]: pgmap v5582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:58:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:59 smithi193 ceph-mon[37473]: pgmap v5583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:58:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9080.246111 secs 2024-07-26T13:58:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:59 smithi134 ceph-mon[31966]: pgmap v5583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:58:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9080.246111 secs 2024-07-26T13:58:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:59 smithi189 ceph-mon[36790]: pgmap v5583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:58:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:58:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9080.246111 secs 2024-07-26T13:59:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:01 smithi193 ceph-mon[37473]: pgmap v5584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9099.671125 secs 2024-07-26T13:59:01.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:01 smithi134 ceph-mon[31966]: pgmap v5584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9099.671125 secs 2024-07-26T13:59:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:01 smithi189 ceph-mon[36790]: pgmap v5584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9099.671125 secs 2024-07-26T13:59:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:03 smithi193 ceph-mon[37473]: pgmap v5585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:03.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:03 smithi134 ceph-mon[31966]: pgmap v5585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:03 smithi189 ceph-mon[36790]: pgmap v5585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9085.246232 secs 2024-07-26T13:59:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9085.246232 secs 2024-07-26T13:59:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9085.246232 secs 2024-07-26T13:59:05.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:59:05.636+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:59:05.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:59:05.636+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:05.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:59:05.636+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:59:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:05 smithi193 ceph-mon[37473]: pgmap v5586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:05.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:05 smithi134 ceph-mon[31966]: pgmap v5586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:05 smithi189 ceph-mon[36790]: pgmap v5586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9104.671289 secs 2024-07-26T13:59:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9104.671289 secs 2024-07-26T13:59:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9104.671289 secs 2024-07-26T13:59:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:07 smithi193 ceph-mon[37473]: pgmap v5587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:07 smithi134 ceph-mon[31966]: pgmap v5587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:07 smithi189 ceph-mon[36790]: pgmap v5587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:09 smithi193 ceph-mon[37473]: pgmap v5588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9090.246355 secs 2024-07-26T13:59:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:09 smithi134 ceph-mon[31966]: pgmap v5588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9090.246355 secs 2024-07-26T13:59:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:09 smithi189 ceph-mon[36790]: pgmap v5588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9090.246355 secs 2024-07-26T13:59:10.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:59:10.211+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:59:10.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:59:10.211+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:59:10.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:59:10.211+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:11 smithi193 ceph-mon[37473]: pgmap v5589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9109.671410 secs 2024-07-26T13:59:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:11 smithi134 ceph-mon[31966]: pgmap v5589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9109.671410 secs 2024-07-26T13:59:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:11 smithi189 ceph-mon[36790]: pgmap v5589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9109.671410 secs 2024-07-26T13:59:13.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:13 smithi134 ceph-mon[31966]: pgmap v5590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:13 smithi189 ceph-mon[36790]: pgmap v5590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:13 smithi193 ceph-mon[37473]: pgmap v5590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9095.246467 secs 2024-07-26T13:59:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:59:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9095.246467 secs 2024-07-26T13:59:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:59:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9095.246467 secs 2024-07-26T13:59:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T13:59:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:15 smithi134 ceph-mon[31966]: pgmap v5591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:59:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:59:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:59:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:15 smithi189 ceph-mon[36790]: pgmap v5591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:59:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:59:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:59:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:15 smithi193 ceph-mon[37473]: pgmap v5591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T13:59:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T13:59:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T13:59:16.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:16 smithi189 ceph-mon[36790]: pgmap v5592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:59:16.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:16 smithi189 ceph-mon[36790]: pgmap v5593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:59:16.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9114.671527 secs 2024-07-26T13:59:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:16 smithi134 ceph-mon[31966]: pgmap v5592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:59:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:16 smithi134 ceph-mon[31966]: pgmap v5593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:59:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9114.671527 secs 2024-07-26T13:59:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:16 smithi193 ceph-mon[37473]: pgmap v5592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T13:59:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:16 smithi193 ceph-mon[37473]: pgmap v5593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:59:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9114.671527 secs 2024-07-26T13:59:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:18 smithi134 ceph-mon[31966]: pgmap v5594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:18 smithi189 ceph-mon[36790]: pgmap v5594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:18 smithi193 ceph-mon[37473]: pgmap v5594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T13:59:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9100.246562 secs 2024-07-26T13:59:19.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9100.246562 secs 2024-07-26T13:59:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9100.246562 secs 2024-07-26T13:59:20.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:59:20.651+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:59:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:59:20.651+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:20.653 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:59:20.651+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:59:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:20 smithi134 ceph-mon[31966]: pgmap v5595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:59:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:20 smithi189 ceph-mon[36790]: pgmap v5595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:59:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:20 smithi193 ceph-mon[37473]: pgmap v5595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T13:59:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9119.671651 secs 2024-07-26T13:59:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9119.671651 secs 2024-07-26T13:59:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9119.671651 secs 2024-07-26T13:59:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:22 smithi134 ceph-mon[31966]: pgmap v5596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:22 smithi189 ceph-mon[36790]: pgmap v5596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:22 smithi193 ceph-mon[37473]: pgmap v5596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:24 smithi134 ceph-mon[31966]: pgmap v5597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:59:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9105.246685 secs 2024-07-26T13:59:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:24 smithi189 ceph-mon[36790]: pgmap v5597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:59:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9105.246685 secs 2024-07-26T13:59:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:24 smithi193 ceph-mon[37473]: pgmap v5597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T13:59:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9105.246685 secs 2024-07-26T13:59:25.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:59:25.227+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:59:25.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:59:25.227+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:59:25.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:59:25.227+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:26 smithi134 ceph-mon[31966]: pgmap v5598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:59:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9124.671745 secs 2024-07-26T13:59:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:26 smithi189 ceph-mon[36790]: pgmap v5598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:59:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9124.671745 secs 2024-07-26T13:59:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:26 smithi193 ceph-mon[37473]: pgmap v5598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T13:59:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9124.671745 secs 2024-07-26T13:59:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:28 smithi134 ceph-mon[31966]: pgmap v5599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:28 smithi189 ceph-mon[36790]: pgmap v5599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:28 smithi193 ceph-mon[37473]: pgmap v5599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9110.246836 secs 2024-07-26T13:59:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9110.246836 secs 2024-07-26T13:59:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9110.246836 secs 2024-07-26T13:59:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:30 smithi134 ceph-mon[31966]: pgmap v5600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:30 smithi189 ceph-mon[36790]: pgmap v5600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:30 smithi193 ceph-mon[37473]: pgmap v5600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9129.671865 secs 2024-07-26T13:59:31.947 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9129.671865 secs 2024-07-26T13:59:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9129.671865 secs 2024-07-26T13:59:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:32 smithi134 ceph-mon[31966]: pgmap v5601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:32 smithi189 ceph-mon[36790]: pgmap v5601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:32 smithi193 ceph-mon[37473]: pgmap v5601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:34 smithi134 ceph-mon[31966]: pgmap v5602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9115.246988 secs 2024-07-26T13:59:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:34 smithi189 ceph-mon[36790]: pgmap v5602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9115.246988 secs 2024-07-26T13:59:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:34 smithi193 ceph-mon[37473]: pgmap v5602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9115.246988 secs 2024-07-26T13:59:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:59:35.667+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:59:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:59:35.667+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:59:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:59:35.667+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:36 smithi134 ceph-mon[31966]: pgmap v5603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9134.671984 secs 2024-07-26T13:59:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:36 smithi189 ceph-mon[36790]: pgmap v5603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9134.671984 secs 2024-07-26T13:59:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:36 smithi193 ceph-mon[37473]: pgmap v5603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9134.671984 secs 2024-07-26T13:59:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:38 smithi134 ceph-mon[31966]: pgmap v5604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:38 smithi189 ceph-mon[36790]: pgmap v5604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:38 smithi193 ceph-mon[37473]: pgmap v5604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9120.247139 secs 2024-07-26T13:59:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9120.247139 secs 2024-07-26T13:59:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9120.247139 secs 2024-07-26T13:59:40.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:59:40.238+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:59:40.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:59:40.238+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:59:40.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:59:40.238+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:40 smithi134 ceph-mon[31966]: pgmap v5605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:40 smithi189 ceph-mon[36790]: pgmap v5605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:40 smithi193 ceph-mon[37473]: pgmap v5605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9139.672138 secs 2024-07-26T13:59:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9139.672138 secs 2024-07-26T13:59:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9139.672138 secs 2024-07-26T13:59:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:42 smithi189 ceph-mon[36790]: pgmap v5606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:42 smithi193 ceph-mon[37473]: pgmap v5606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:43.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:42 smithi134 ceph-mon[31966]: pgmap v5606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:44 smithi189 ceph-mon[36790]: pgmap v5607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9125.247263 secs 2024-07-26T13:59:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:44 smithi193 ceph-mon[37473]: pgmap v5607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9125.247263 secs 2024-07-26T13:59:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:44 smithi134 ceph-mon[31966]: pgmap v5607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9125.247263 secs 2024-07-26T13:59:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:46 smithi189 ceph-mon[36790]: pgmap v5608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9144.672277 secs 2024-07-26T13:59:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:46 smithi193 ceph-mon[37473]: pgmap v5608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9144.672277 secs 2024-07-26T13:59:47.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:46 smithi134 ceph-mon[31966]: pgmap v5608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9144.672277 secs 2024-07-26T13:59:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:48 smithi193 ceph-mon[37473]: pgmap v5609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:49.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:48 smithi134 ceph-mon[31966]: pgmap v5609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:48 smithi189 ceph-mon[36790]: pgmap v5609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9130.247324 secs 2024-07-26T13:59:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9130.247324 secs 2024-07-26T13:59:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9130.247324 secs 2024-07-26T13:59:50.684 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:59:50.682+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:59:50.684 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:59:50.682+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:50.684 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T13:59:50.682+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:59:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:50 smithi193 ceph-mon[37473]: pgmap v5610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:50 smithi134 ceph-mon[31966]: pgmap v5610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:50 smithi189 ceph-mon[36790]: pgmap v5610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T13:59:52.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9149.672404 secs 2024-07-26T13:59:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9149.672404 secs 2024-07-26T13:59:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9149.672404 secs 2024-07-26T13:59:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:52 smithi193 ceph-mon[37473]: pgmap v5611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:53.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:52 smithi134 ceph-mon[31966]: pgmap v5611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:52 smithi189 ceph-mon[36790]: pgmap v5611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9135.247413 secs 2024-07-26T13:59:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9135.247413 secs 2024-07-26T13:59:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9135.247413 secs 2024-07-26T13:59:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:54 smithi193 ceph-mon[37473]: pgmap v5612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:54 smithi134 ceph-mon[31966]: pgmap v5612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:54 smithi189 ceph-mon[36790]: pgmap v5612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T13:59:55.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:59:55.253+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T13:59:55.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:59:55.253+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T13:59:55.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T13:59:55.253+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T13:59:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:56 smithi193 ceph-mon[37473]: pgmap v5613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9154.672514 secs 2024-07-26T13:59:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:56 smithi134 ceph-mon[31966]: pgmap v5613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9154.672514 secs 2024-07-26T13:59:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:56 smithi189 ceph-mon[36790]: pgmap v5613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T13:59:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9154.672514 secs 2024-07-26T13:59:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:58 smithi193 ceph-mon[37473]: pgmap v5614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 13:59:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9140.247533 secs 2024-07-26T13:59:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:58 smithi134 ceph-mon[31966]: pgmap v5614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 13:59:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9140.247533 secs 2024-07-26T13:59:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:58 smithi189 ceph-mon[36790]: pgmap v5614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T13:59:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 13:59:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9140.247533 secs 2024-07-26T14:00:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:00 smithi193 ceph-mon[37473]: pgmap v5615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:00:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:00 smithi134 ceph-mon[31966]: pgmap v5615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:00:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:00 smithi189 ceph-mon[36790]: pgmap v5615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:00:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9159.672636 secs 2024-07-26T14:00:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9159.672636 secs 2024-07-26T14:00:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9159.672636 secs 2024-07-26T14:00:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:02 smithi193 ceph-mon[37473]: pgmap v5616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:03.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:02 smithi134 ceph-mon[31966]: pgmap v5616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:02 smithi189 ceph-mon[36790]: pgmap v5616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:04.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9145.247655 secs 2024-07-26T14:00:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9145.247655 secs 2024-07-26T14:00:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9145.247655 secs 2024-07-26T14:00:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:04 smithi134 ceph-mon[31966]: pgmap v5617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:04 smithi189 ceph-mon[36790]: pgmap v5617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:04 smithi193 ceph-mon[37473]: pgmap v5617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:05.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:00:05.690+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:00:05.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:00:05.690+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:00:05.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:00:05.690+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:06 smithi134 ceph-mon[31966]: pgmap v5618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9164.672756 secs 2024-07-26T14:00:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:06 smithi189 ceph-mon[36790]: pgmap v5618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9164.672756 secs 2024-07-26T14:00:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:06 smithi193 ceph-mon[37473]: pgmap v5618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9164.672756 secs 2024-07-26T14:00:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:08 smithi134 ceph-mon[31966]: pgmap v5619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:00:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9150.247790 secs 2024-07-26T14:00:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:08 smithi189 ceph-mon[36790]: pgmap v5619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:00:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9150.247790 secs 2024-07-26T14:00:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:08 smithi193 ceph-mon[37473]: pgmap v5619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:00:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9150.247790 secs 2024-07-26T14:00:10.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:00:10.266+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:00:10.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:00:10.266+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:00:10.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:00:10.266+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:11.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:10 smithi134 ceph-mon[31966]: pgmap v5620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:10 smithi189 ceph-mon[36790]: pgmap v5620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:11.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:10 smithi193 ceph-mon[37473]: pgmap v5620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:00:12.168 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9169.672882 secs 2024-07-26T14:00:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9169.672882 secs 2024-07-26T14:00:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9169.672882 secs 2024-07-26T14:00:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:12 smithi134 ceph-mon[31966]: pgmap v5621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:12 smithi189 ceph-mon[36790]: pgmap v5621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:12 smithi193 ceph-mon[37473]: pgmap v5621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9155.247918 secs 2024-07-26T14:00:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:14 smithi193 ceph-mon[37473]: pgmap v5622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9155.247918 secs 2024-07-26T14:00:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:14 smithi134 ceph-mon[31966]: pgmap v5622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9155.247918 secs 2024-07-26T14:00:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:14 smithi189 ceph-mon[36790]: pgmap v5622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:16 smithi193 ceph-mon[37473]: pgmap v5623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:00:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9174.672949 secs 2024-07-26T14:00:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:16 smithi134 ceph-mon[31966]: pgmap v5623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:00:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9174.672949 secs 2024-07-26T14:00:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:16 smithi189 ceph-mon[36790]: pgmap v5623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:00:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9174.672949 secs 2024-07-26T14:00:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:00:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:00:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:17 smithi134 ceph-mon[31966]: pgmap v5624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:00:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:17 smithi134 ceph-mon[31966]: pgmap v5625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:00:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:00:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:00:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:17 smithi189 ceph-mon[36790]: pgmap v5624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:00:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:17 smithi189 ceph-mon[36790]: pgmap v5625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:00:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:00:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:00:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:17 smithi193 ceph-mon[37473]: pgmap v5624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:00:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:17 smithi193 ceph-mon[37473]: pgmap v5625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:00:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9160.248035 secs 2024-07-26T14:00:19.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9160.248035 secs 2024-07-26T14:00:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9160.248035 secs 2024-07-26T14:00:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:19 smithi193 ceph-mon[37473]: pgmap v5626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:00:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:19 smithi134 ceph-mon[31966]: pgmap v5626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:00:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:19 smithi189 ceph-mon[36790]: pgmap v5626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:00:20.706 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:00:20.706+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:00:20.706 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:00:20.706+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:20.707 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:00:20.706+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:00:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:21 smithi193 ceph-mon[37473]: pgmap v5627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:00:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9179.673075 secs 2024-07-26T14:00:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:21 smithi134 ceph-mon[31966]: pgmap v5627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:00:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9179.673075 secs 2024-07-26T14:00:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:21 smithi189 ceph-mon[36790]: pgmap v5627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:00:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9179.673075 secs 2024-07-26T14:00:23.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:00:23.924+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T14:00:23.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:00:23.924+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T14:00:23.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:00:23.924+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T14:00:23.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:00:23.924+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T14:00:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:23 smithi193 ceph-mon[37473]: pgmap v5628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:00:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9165.248145 secs 2024-07-26T14:00:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:23 smithi134 ceph-mon[31966]: pgmap v5628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:00:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9165.248145 secs 2024-07-26T14:00:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:23 smithi189 ceph-mon[36790]: pgmap v5628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:00:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9165.248145 secs 2024-07-26T14:00:25.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:00:25.282+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:00:25.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:00:25.283+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:00:25.283 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:00:25.283+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:25 smithi134 ceph-mon[31966]: pgmap v5629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:00:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:25 smithi189 ceph-mon[36790]: pgmap v5629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:00:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:25 smithi193 ceph-mon[37473]: pgmap v5629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:00:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9184.673194 secs 2024-07-26T14:00:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9184.673194 secs 2024-07-26T14:00:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9184.673194 secs 2024-07-26T14:00:28.040 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:27 smithi134 ceph-mon[31966]: pgmap v5630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:00:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:27 smithi189 ceph-mon[36790]: pgmap v5630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:00:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:27 smithi193 ceph-mon[37473]: pgmap v5630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:00:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9170.248234 secs 2024-07-26T14:00:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9170.248234 secs 2024-07-26T14:00:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9170.248234 secs 2024-07-26T14:00:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:29 smithi134 ceph-mon[31966]: pgmap v5631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:29 smithi189 ceph-mon[36790]: pgmap v5631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:29 smithi193 ceph-mon[37473]: pgmap v5631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:31 smithi134 ceph-mon[31966]: pgmap v5632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9189.673333 secs 2024-07-26T14:00:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:31 smithi189 ceph-mon[36790]: pgmap v5632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9189.673333 secs 2024-07-26T14:00:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:31 smithi193 ceph-mon[37473]: pgmap v5632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9189.673333 secs 2024-07-26T14:00:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:32 smithi134 ceph-mon[31966]: pgmap v5633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:32 smithi189 ceph-mon[36790]: pgmap v5633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:32 smithi193 ceph-mon[37473]: pgmap v5633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9175.248364 secs 2024-07-26T14:00:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9175.248364 secs 2024-07-26T14:00:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9175.248364 secs 2024-07-26T14:00:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:00:34.448+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:00:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:00:34.447+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:00:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:00:34.448+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:00:34.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:00:34.448+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:00:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:34 smithi134 ceph-mon[31966]: pgmap v5634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:00:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:34 smithi189 ceph-mon[36790]: pgmap v5634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:00:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:34 smithi193 ceph-mon[37473]: pgmap v5634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:00:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:00:35.712+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:00:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:00:35.712+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:00:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:00:35.712+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9194.673355 secs 2024-07-26T14:00:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9194.673355 secs 2024-07-26T14:00:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9194.673355 secs 2024-07-26T14:00:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:37 smithi134 ceph-mon[31966]: pgmap v5635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:37 smithi189 ceph-mon[36790]: pgmap v5635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:37 smithi193 ceph-mon[37473]: pgmap v5635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9180.248456 secs 2024-07-26T14:00:39.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9180.248456 secs 2024-07-26T14:00:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9180.248456 secs 2024-07-26T14:00:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:39 smithi193 ceph-mon[37473]: pgmap v5636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:40.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:39 smithi134 ceph-mon[31966]: pgmap v5636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:39 smithi189 ceph-mon[36790]: pgmap v5636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:40.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:00:40.297+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:00:40.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:00:40.297+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:40.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:00:40.297+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:00:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:41 smithi193 ceph-mon[37473]: pgmap v5637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9199.673497 secs 2024-07-26T14:00:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:41 smithi134 ceph-mon[31966]: pgmap v5637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9199.673497 secs 2024-07-26T14:00:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:41 smithi189 ceph-mon[36790]: pgmap v5637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9199.673497 secs 2024-07-26T14:00:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:43 smithi193 ceph-mon[37473]: pgmap v5638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9185.248596 secs 2024-07-26T14:00:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:43 smithi134 ceph-mon[31966]: pgmap v5638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9185.248596 secs 2024-07-26T14:00:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:43 smithi189 ceph-mon[36790]: pgmap v5638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9185.248596 secs 2024-07-26T14:00:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:45 smithi134 ceph-mon[31966]: pgmap v5639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:00:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:45 smithi189 ceph-mon[36790]: pgmap v5639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:00:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:45 smithi193 ceph-mon[37473]: pgmap v5639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:00:47.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9204.673625 secs 2024-07-26T14:00:47.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9204.673625 secs 2024-07-26T14:00:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9204.673625 secs 2024-07-26T14:00:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:47 smithi134 ceph-mon[31966]: pgmap v5640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:47 smithi189 ceph-mon[36790]: pgmap v5640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:47 smithi193 ceph-mon[37473]: pgmap v5640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:49.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9190.248665 secs 2024-07-26T14:00:49.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9190.248665 secs 2024-07-26T14:00:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9190.248665 secs 2024-07-26T14:00:50.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:49 smithi134 ceph-mon[31966]: pgmap v5641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:49 smithi189 ceph-mon[36790]: pgmap v5641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:49 smithi193 ceph-mon[37473]: pgmap v5641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:50.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:00:50.728+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:00:50.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:00:50.728+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:00:50.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:00:50.728+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:51 smithi134 ceph-mon[31966]: pgmap v5642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9209.673790 secs 2024-07-26T14:00:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:51 smithi189 ceph-mon[36790]: pgmap v5642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9209.673790 secs 2024-07-26T14:00:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:51 smithi193 ceph-mon[37473]: pgmap v5642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9209.673790 secs 2024-07-26T14:00:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:53 smithi134 ceph-mon[31966]: pgmap v5643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9195.248766 secs 2024-07-26T14:00:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:53 smithi189 ceph-mon[36790]: pgmap v5643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9195.248766 secs 2024-07-26T14:00:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:53 smithi193 ceph-mon[37473]: pgmap v5643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:00:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9195.248766 secs 2024-07-26T14:00:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:54 smithi134 ceph-mon[31966]: pgmap v5644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:00:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:54 smithi189 ceph-mon[36790]: pgmap v5644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:00:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:54 smithi193 ceph-mon[37473]: pgmap v5644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:00:55.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:00:55.313+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:00:55.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:00:55.313+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:00:55.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:00:55.313+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:00:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9214.673875 secs 2024-07-26T14:00:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9214.673875 secs 2024-07-26T14:00:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9214.673875 secs 2024-07-26T14:00:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:57 smithi134 ceph-mon[31966]: pgmap v5645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:57 smithi189 ceph-mon[36790]: pgmap v5645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:57 smithi193 ceph-mon[37473]: pgmap v5645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:00:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9200.248844 secs 2024-07-26T14:00:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9200.248844 secs 2024-07-26T14:00:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9200.248844 secs 2024-07-26T14:01:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:00:59 smithi193 ceph-mon[37473]: pgmap v5646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:00:59 smithi134 ceph-mon[31966]: pgmap v5646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:00:59 smithi189 ceph-mon[36790]: pgmap v5646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:01 smithi193 ceph-mon[37473]: pgmap v5647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9219.673958 secs 2024-07-26T14:01:02.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:01 smithi134 ceph-mon[31966]: pgmap v5647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:02.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9219.673958 secs 2024-07-26T14:01:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:01 smithi189 ceph-mon[36790]: pgmap v5647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9219.673958 secs 2024-07-26T14:01:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:03 smithi134 ceph-mon[31966]: pgmap v5648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9205.248966 secs 2024-07-26T14:01:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:03 smithi189 ceph-mon[36790]: pgmap v5648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9205.248966 secs 2024-07-26T14:01:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:03 smithi193 ceph-mon[37473]: pgmap v5648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9205.248966 secs 2024-07-26T14:01:05.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:01:05.731+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:01:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:01:05.731+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:01:05.733 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:01:05.731+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:06.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:05 smithi134 ceph-mon[31966]: pgmap v5649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:01:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:05 smithi189 ceph-mon[36790]: pgmap v5649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:01:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:05 smithi193 ceph-mon[37473]: pgmap v5649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:01:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9224.674098 secs 2024-07-26T14:01:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9224.674098 secs 2024-07-26T14:01:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9224.674098 secs 2024-07-26T14:01:08.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:07 smithi134 ceph-mon[31966]: pgmap v5650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:07 smithi189 ceph-mon[36790]: pgmap v5650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:07 smithi193 ceph-mon[37473]: pgmap v5650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9210.249093 secs 2024-07-26T14:01:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9210.249093 secs 2024-07-26T14:01:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9210.249093 secs 2024-07-26T14:01:10.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:09 smithi134 ceph-mon[31966]: pgmap v5651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:09 smithi189 ceph-mon[36790]: pgmap v5651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:09 smithi193 ceph-mon[37473]: pgmap v5651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:10.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:01:10.328+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:01:10.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:01:10.328+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:10.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:01:10.328+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:01:12.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:11 smithi134 ceph-mon[31966]: pgmap v5652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:12.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9229.674237 secs 2024-07-26T14:01:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:11 smithi189 ceph-mon[36790]: pgmap v5652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9229.674237 secs 2024-07-26T14:01:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:11 smithi193 ceph-mon[37473]: pgmap v5652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9229.674237 secs 2024-07-26T14:01:14.121 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:13 smithi134 ceph-mon[31966]: pgmap v5653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:14.122 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9215.249216 secs 2024-07-26T14:01:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:13 smithi189 ceph-mon[36790]: pgmap v5653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9215.249216 secs 2024-07-26T14:01:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:13 smithi193 ceph-mon[37473]: pgmap v5653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9215.249216 secs 2024-07-26T14:01:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:14 smithi134 ceph-mon[31966]: pgmap v5654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:01:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:14 smithi189 ceph-mon[36790]: pgmap v5654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:01:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:14 smithi193 ceph-mon[37473]: pgmap v5654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:01:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9234.674370 secs 2024-07-26T14:01:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9234.674370 secs 2024-07-26T14:01:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9234.674370 secs 2024-07-26T14:01:17.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:17 smithi189 ceph-mon[36790]: pgmap v5655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:17.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:01:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:17 smithi134 ceph-mon[31966]: pgmap v5655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:01:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:17 smithi193 ceph-mon[37473]: pgmap v5655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:01:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:01:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:01:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:18 smithi189 ceph-mon[36790]: pgmap v5656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:01:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:18 smithi189 ceph-mon[36790]: pgmap v5657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:01:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:01:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:01:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:01:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:18 smithi134 ceph-mon[31966]: pgmap v5656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:01:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:18 smithi134 ceph-mon[31966]: pgmap v5657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:01:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:01:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:01:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:01:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:18 smithi193 ceph-mon[37473]: pgmap v5656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:01:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:18 smithi193 ceph-mon[37473]: pgmap v5657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:01:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:01:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9220.249281 secs 2024-07-26T14:01:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9220.249281 secs 2024-07-26T14:01:19.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9220.249281 secs 2024-07-26T14:01:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:20 smithi134 ceph-mon[31966]: pgmap v5658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:01:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:20 smithi189 ceph-mon[36790]: pgmap v5658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:01:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:01:20.747+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:01:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:01:20.747+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:01:20.748 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:01:20.747+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:20 smithi193 ceph-mon[37473]: pgmap v5658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:01:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9239.674460 secs 2024-07-26T14:01:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9239.674460 secs 2024-07-26T14:01:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9239.674460 secs 2024-07-26T14:01:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:22 smithi134 ceph-mon[31966]: pgmap v5659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:01:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:22 smithi189 ceph-mon[36790]: pgmap v5659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:01:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:22 smithi193 ceph-mon[37473]: pgmap v5659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:01:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9225.249402 secs 2024-07-26T14:01:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9225.249402 secs 2024-07-26T14:01:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9225.249402 secs 2024-07-26T14:01:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:24 smithi134 ceph-mon[31966]: pgmap v5660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:01:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:24 smithi189 ceph-mon[36790]: pgmap v5660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:01:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:24 smithi193 ceph-mon[37473]: pgmap v5660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:01:25.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:01:25.344+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:01:25.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:01:25.344+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:25.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:01:25.344+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:01:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:25 smithi134 ceph-mon[31966]: pgmap v5661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:01:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:25 smithi189 ceph-mon[36790]: pgmap v5661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:01:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:25 smithi193 ceph-mon[37473]: pgmap v5661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:01:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9244.674592 secs 2024-07-26T14:01:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9244.674592 secs 2024-07-26T14:01:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9244.674592 secs 2024-07-26T14:01:28.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:27 smithi134 ceph-mon[31966]: pgmap v5662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:01:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:27 smithi189 ceph-mon[36790]: pgmap v5662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:01:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:27 smithi193 ceph-mon[37473]: pgmap v5662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:01:29.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9230.249522 secs 2024-07-26T14:01:29.190 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9230.249522 secs 2024-07-26T14:01:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9230.249522 secs 2024-07-26T14:01:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:29 smithi134 ceph-mon[31966]: pgmap v5663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:01:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:29 smithi189 ceph-mon[36790]: pgmap v5663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:01:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:29 smithi193 ceph-mon[37473]: pgmap v5663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:01:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9249.674685 secs 2024-07-26T14:01:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9249.674685 secs 2024-07-26T14:01:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9249.674685 secs 2024-07-26T14:01:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:32 smithi134 ceph-mon[31966]: pgmap v5664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:01:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:32 smithi189 ceph-mon[36790]: pgmap v5664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:01:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:32 smithi193 ceph-mon[37473]: pgmap v5664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:01:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9235.249643 secs 2024-07-26T14:01:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9235.249643 secs 2024-07-26T14:01:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9235.249643 secs 2024-07-26T14:01:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:34 smithi134 ceph-mon[31966]: pgmap v5665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:01:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:34 smithi189 ceph-mon[36790]: pgmap v5665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:01:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:34 smithi193 ceph-mon[37473]: pgmap v5665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:01:35.763 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:01:35.762+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:01:35.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:01:35.762+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:01:35.764 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:01:35.762+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:36.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:35 smithi134 ceph-mon[31966]: pgmap v5666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:01:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:35 smithi189 ceph-mon[36790]: pgmap v5666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:01:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:35 smithi193 ceph-mon[37473]: pgmap v5666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:01:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9254.674820 secs 2024-07-26T14:01:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9254.674820 secs 2024-07-26T14:01:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9254.674820 secs 2024-07-26T14:01:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:37 smithi134 ceph-mon[31966]: pgmap v5667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:01:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:37 smithi189 ceph-mon[36790]: pgmap v5667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:01:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:37 smithi193 ceph-mon[37473]: pgmap v5667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:01:39.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9240.249711 secs 2024-07-26T14:01:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9240.249711 secs 2024-07-26T14:01:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9240.249711 secs 2024-07-26T14:01:40.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:39 smithi134 ceph-mon[31966]: pgmap v5668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:01:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:39 smithi189 ceph-mon[36790]: pgmap v5668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:01:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:39 smithi193 ceph-mon[37473]: pgmap v5668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:01:40.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:01:40.360+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:01:40.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:01:40.360+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:40.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:01:40.360+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:01:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9259.674909 secs 2024-07-26T14:01:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9259.674909 secs 2024-07-26T14:01:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9259.674909 secs 2024-07-26T14:01:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:42 smithi134 ceph-mon[31966]: pgmap v5669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:01:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:42 smithi189 ceph-mon[36790]: pgmap v5669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:01:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:42 smithi193 ceph-mon[37473]: pgmap v5669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:01:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9245.249854 secs 2024-07-26T14:01:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9245.249854 secs 2024-07-26T14:01:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9245.249854 secs 2024-07-26T14:01:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:44 smithi134 ceph-mon[31966]: pgmap v5670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:01:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:44 smithi189 ceph-mon[36790]: pgmap v5670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:01:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:44 smithi193 ceph-mon[37473]: pgmap v5670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:01:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:45 smithi134 ceph-mon[31966]: pgmap v5671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:01:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:45 smithi189 ceph-mon[36790]: pgmap v5671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:01:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:45 smithi193 ceph-mon[37473]: pgmap v5671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:01:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9264.675032 secs 2024-07-26T14:01:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9264.675032 secs 2024-07-26T14:01:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9264.675032 secs 2024-07-26T14:01:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:47 smithi134 ceph-mon[31966]: pgmap v5672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:01:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:47 smithi189 ceph-mon[36790]: pgmap v5672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:01:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:47 smithi193 ceph-mon[37473]: pgmap v5672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:01:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9250.249975 secs 2024-07-26T14:01:49.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9250.249975 secs 2024-07-26T14:01:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9250.249975 secs 2024-07-26T14:01:50.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:49 smithi134 ceph-mon[31966]: pgmap v5673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:01:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:49 smithi189 ceph-mon[36790]: pgmap v5673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:01:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:49 smithi193 ceph-mon[37473]: pgmap v5673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:01:50.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:01:50.778+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:01:50.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:01:50.778+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:01:50.779 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:01:50.778+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9269.675099 secs 2024-07-26T14:01:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9269.675099 secs 2024-07-26T14:01:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9269.675099 secs 2024-07-26T14:01:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:52 smithi134 ceph-mon[31966]: pgmap v5674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:52 smithi189 ceph-mon[36790]: pgmap v5674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:52 smithi193 ceph-mon[37473]: pgmap v5674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9255.250095 secs 2024-07-26T14:01:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9255.250095 secs 2024-07-26T14:01:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9255.250095 secs 2024-07-26T14:01:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:54 smithi134 ceph-mon[31966]: pgmap v5675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:54 smithi189 ceph-mon[36790]: pgmap v5675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:54 smithi193 ceph-mon[37473]: pgmap v5675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:55.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:01:55.375+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:01:55.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:01:55.375+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:01:55.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:01:55.375+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:01:56.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:55 smithi134 ceph-mon[31966]: pgmap v5676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:55 smithi189 ceph-mon[36790]: pgmap v5676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:55 smithi193 ceph-mon[37473]: pgmap v5676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:01:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9274.675229 secs 2024-07-26T14:01:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9274.675229 secs 2024-07-26T14:01:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9274.675229 secs 2024-07-26T14:01:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:57 smithi134 ceph-mon[31966]: pgmap v5677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:57 smithi189 ceph-mon[36790]: pgmap v5677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:57 smithi193 ceph-mon[37473]: pgmap v5677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:01:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9260.250215 secs 2024-07-26T14:01:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9260.250215 secs 2024-07-26T14:01:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9260.250215 secs 2024-07-26T14:02:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:01:59 smithi134 ceph-mon[31966]: pgmap v5678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:02:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:01:59 smithi189 ceph-mon[36790]: pgmap v5678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:02:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:01:59 smithi193 ceph-mon[37473]: pgmap v5678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:02:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9279.675352 secs 2024-07-26T14:02:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9279.675352 secs 2024-07-26T14:02:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9279.675352 secs 2024-07-26T14:02:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:02 smithi134 ceph-mon[31966]: pgmap v5679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:02 smithi189 ceph-mon[36790]: pgmap v5679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:02 smithi193 ceph-mon[37473]: pgmap v5679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9265.250337 secs 2024-07-26T14:02:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9265.250337 secs 2024-07-26T14:02:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9265.250337 secs 2024-07-26T14:02:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:04 smithi134 ceph-mon[31966]: pgmap v5680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:04 smithi189 ceph-mon[36790]: pgmap v5680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:04 smithi193 ceph-mon[37473]: pgmap v5680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:02:05.791+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:02:05.792 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:02:05.791+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:05.793 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:02:05.791+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:02:06.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:05 smithi134 ceph-mon[31966]: pgmap v5681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:05 smithi189 ceph-mon[36790]: pgmap v5681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:05 smithi193 ceph-mon[37473]: pgmap v5681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9284.675453 secs 2024-07-26T14:02:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9284.675453 secs 2024-07-26T14:02:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9284.675453 secs 2024-07-26T14:02:08.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:07 smithi134 ceph-mon[31966]: pgmap v5682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:07 smithi189 ceph-mon[36790]: pgmap v5682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:07 smithi193 ceph-mon[37473]: pgmap v5682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9270.250459 secs 2024-07-26T14:02:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9270.250459 secs 2024-07-26T14:02:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9270.250459 secs 2024-07-26T14:02:10.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:09 smithi134 ceph-mon[31966]: pgmap v5683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:02:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:09 smithi189 ceph-mon[36790]: pgmap v5683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:02:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:09 smithi193 ceph-mon[37473]: pgmap v5683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:02:10.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:02:10.390+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:02:10.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:02:10.390+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:10.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:02:10.390+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:02:11.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9289.675556 secs 2024-07-26T14:02:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9289.675556 secs 2024-07-26T14:02:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9289.675556 secs 2024-07-26T14:02:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:12 smithi134 ceph-mon[31966]: pgmap v5684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:12 smithi189 ceph-mon[36790]: pgmap v5684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:12 smithi193 ceph-mon[37473]: pgmap v5684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9275.250555 secs 2024-07-26T14:02:14.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9275.250555 secs 2024-07-26T14:02:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9275.250555 secs 2024-07-26T14:02:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:14 smithi134 ceph-mon[31966]: pgmap v5685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:14 smithi189 ceph-mon[36790]: pgmap v5685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:14 smithi193 ceph-mon[37473]: pgmap v5685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:15 smithi134 ceph-mon[31966]: pgmap v5686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:16.805 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:15 smithi193 ceph-mon[37473]: pgmap v5686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:16.805 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:15 smithi189 ceph-mon[36790]: pgmap v5686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:17.189 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9294.675684 secs 2024-07-26T14:02:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9294.675684 secs 2024-07-26T14:02:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9294.675684 secs 2024-07-26T14:02:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:17 smithi134 ceph-mon[31966]: pgmap v5687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:17 smithi189 ceph-mon[36790]: pgmap v5687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:17 smithi193 ceph-mon[37473]: pgmap v5687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:02:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9280.250703 secs 2024-07-26T14:02:19.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:02:19.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9280.250703 secs 2024-07-26T14:02:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:02:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9280.250703 secs 2024-07-26T14:02:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:02:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:02:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:19 smithi134 ceph-mon[31966]: pgmap v5688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:02:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:19 smithi134 ceph-mon[31966]: pgmap v5689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:02:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:02:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:02:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:02:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:19 smithi189 ceph-mon[36790]: pgmap v5688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:02:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:19 smithi189 ceph-mon[36790]: pgmap v5689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:02:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:02:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:02:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:02:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:19 smithi193 ceph-mon[37473]: pgmap v5688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:02:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:19 smithi193 ceph-mon[37473]: pgmap v5689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:02:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:02:20.800 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:02:20.799+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:02:20.800 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:02:20.799+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:02:20.800 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:02:20.799+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:22 smithi134 ceph-mon[31966]: pgmap v5690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:02:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9299.675842 secs 2024-07-26T14:02:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:22 smithi189 ceph-mon[36790]: pgmap v5690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:02:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9299.675842 secs 2024-07-26T14:02:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:22 smithi193 ceph-mon[37473]: pgmap v5690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:02:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9299.675842 secs 2024-07-26T14:02:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:24 smithi134 ceph-mon[31966]: pgmap v5691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:02:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9285.250822 secs 2024-07-26T14:02:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:24 smithi189 ceph-mon[36790]: pgmap v5691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:02:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9285.250822 secs 2024-07-26T14:02:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:24 smithi193 ceph-mon[37473]: pgmap v5691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:02:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9285.250822 secs 2024-07-26T14:02:25.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:02:25.406+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:02:25.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:02:25.406+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:25.408 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:02:25.406+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:02:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:26 smithi134 ceph-mon[31966]: pgmap v5692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:02:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:26 smithi189 ceph-mon[36790]: pgmap v5692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:02:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:26 smithi193 ceph-mon[37473]: pgmap v5692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:02:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9304.675964 secs 2024-07-26T14:02:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9304.675964 secs 2024-07-26T14:02:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9304.675964 secs 2024-07-26T14:02:28.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:28 smithi134 ceph-mon[31966]: pgmap v5693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:02:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:28 smithi189 ceph-mon[36790]: pgmap v5693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:02:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:28 smithi193 ceph-mon[37473]: pgmap v5693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:02:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9290.250963 secs 2024-07-26T14:02:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9290.250963 secs 2024-07-26T14:02:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9290.250963 secs 2024-07-26T14:02:30.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:30 smithi134 ceph-mon[31966]: pgmap v5694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:02:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:30 smithi189 ceph-mon[36790]: pgmap v5694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:02:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:30 smithi193 ceph-mon[37473]: pgmap v5694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:02:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:32 smithi134 ceph-mon[31966]: pgmap v5695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9309.676106 secs 2024-07-26T14:02:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:32 smithi189 ceph-mon[36790]: pgmap v5695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9309.676106 secs 2024-07-26T14:02:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:32 smithi193 ceph-mon[37473]: pgmap v5695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9309.676106 secs 2024-07-26T14:02:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:34 smithi134 ceph-mon[31966]: pgmap v5696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9295.251083 secs 2024-07-26T14:02:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:34 smithi189 ceph-mon[36790]: pgmap v5696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9295.251083 secs 2024-07-26T14:02:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:34 smithi193 ceph-mon[37473]: pgmap v5696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9295.251083 secs 2024-07-26T14:02:35.816 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:02:35.814+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:02:35.816 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:02:35.814+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:02:35.816 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:02:35.814+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:36.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:36 smithi134 ceph-mon[31966]: pgmap v5697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:36 smithi189 ceph-mon[36790]: pgmap v5697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:36 smithi193 ceph-mon[37473]: pgmap v5697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9314.676231 secs 2024-07-26T14:02:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9314.676231 secs 2024-07-26T14:02:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9314.676231 secs 2024-07-26T14:02:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:38 smithi134 ceph-mon[31966]: pgmap v5698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:38 smithi189 ceph-mon[36790]: pgmap v5698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:38 smithi193 ceph-mon[37473]: pgmap v5698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9300.251203 secs 2024-07-26T14:02:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9300.251203 secs 2024-07-26T14:02:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9300.251203 secs 2024-07-26T14:02:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:40 smithi134 ceph-mon[31966]: pgmap v5699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:40.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:02:40.421+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:02:40.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:02:40.421+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:40.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:02:40.421+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:02:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:40 smithi189 ceph-mon[36790]: pgmap v5699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:40 smithi193 ceph-mon[37473]: pgmap v5699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9319.676361 secs 2024-07-26T14:02:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:42 smithi134 ceph-mon[31966]: pgmap v5700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9319.676361 secs 2024-07-26T14:02:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:42 smithi189 ceph-mon[36790]: pgmap v5700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9319.676361 secs 2024-07-26T14:02:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:42 smithi193 ceph-mon[37473]: pgmap v5700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:44 smithi134 ceph-mon[31966]: pgmap v5701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9305.251336 secs 2024-07-26T14:02:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:44 smithi189 ceph-mon[36790]: pgmap v5701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9305.251336 secs 2024-07-26T14:02:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:44 smithi193 ceph-mon[37473]: pgmap v5701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9305.251336 secs 2024-07-26T14:02:46.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:46 smithi134 ceph-mon[31966]: pgmap v5702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:46 smithi189 ceph-mon[36790]: pgmap v5702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:46 smithi193 ceph-mon[37473]: pgmap v5702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9324.676487 secs 2024-07-26T14:02:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9324.676487 secs 2024-07-26T14:02:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9324.676487 secs 2024-07-26T14:02:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:48 smithi189 ceph-mon[36790]: pgmap v5703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:48 smithi193 ceph-mon[37473]: pgmap v5703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:48 smithi134 ceph-mon[31966]: pgmap v5703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9310.251404 secs 2024-07-26T14:02:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9310.251404 secs 2024-07-26T14:02:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9310.251404 secs 2024-07-26T14:02:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:50 smithi189 ceph-mon[36790]: pgmap v5704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:50 smithi193 ceph-mon[37473]: pgmap v5704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:50 smithi134 ceph-mon[31966]: pgmap v5704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:02:50.831 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:02:50.830+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:02:50.831 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:02:50.830+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:50.831 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:02:50.830+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:02:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9329.676579 secs 2024-07-26T14:02:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:52 smithi189 ceph-mon[36790]: pgmap v5705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9329.676579 secs 2024-07-26T14:02:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:52 smithi193 ceph-mon[37473]: pgmap v5705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9329.676579 secs 2024-07-26T14:02:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:52 smithi134 ceph-mon[31966]: pgmap v5705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:54 smithi193 ceph-mon[37473]: pgmap v5706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9315.251525 secs 2024-07-26T14:02:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:54 smithi134 ceph-mon[31966]: pgmap v5706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9315.251525 secs 2024-07-26T14:02:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:54 smithi189 ceph-mon[36790]: pgmap v5706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:02:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9315.251525 secs 2024-07-26T14:02:55.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:02:55.437+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:02:55.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:02:55.437+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:02:55.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:02:55.437+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:02:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:56 smithi193 ceph-mon[37473]: pgmap v5707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:56 smithi134 ceph-mon[31966]: pgmap v5707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:56 smithi189 ceph-mon[36790]: pgmap v5707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:02:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9334.676698 secs 2024-07-26T14:02:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9334.676698 secs 2024-07-26T14:02:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9334.676698 secs 2024-07-26T14:02:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:58 smithi193 ceph-mon[37473]: pgmap v5708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:58 smithi134 ceph-mon[31966]: pgmap v5708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:58 smithi189 ceph-mon[36790]: pgmap v5708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:02:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:02:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9320.251621 secs 2024-07-26T14:02:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:02:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9320.251621 secs 2024-07-26T14:02:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:02:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9320.251621 secs 2024-07-26T14:03:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:00 smithi193 ceph-mon[37473]: pgmap v5709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:00 smithi134 ceph-mon[31966]: pgmap v5709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:00 smithi189 ceph-mon[36790]: pgmap v5709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9339.676781 secs 2024-07-26T14:03:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:02 smithi193 ceph-mon[37473]: pgmap v5710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9339.676781 secs 2024-07-26T14:03:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:02 smithi134 ceph-mon[31966]: pgmap v5710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9339.676781 secs 2024-07-26T14:03:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:02 smithi189 ceph-mon[36790]: pgmap v5710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:04 smithi193 ceph-mon[37473]: pgmap v5711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9325.251763 secs 2024-07-26T14:03:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:04 smithi134 ceph-mon[31966]: pgmap v5711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9325.251763 secs 2024-07-26T14:03:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:04 smithi189 ceph-mon[36790]: pgmap v5711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9325.251763 secs 2024-07-26T14:03:05.846 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:03:05.845+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:03:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:03:05.845+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:05.847 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:03:05.845+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:03:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:06 smithi193 ceph-mon[37473]: pgmap v5712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:06 smithi134 ceph-mon[31966]: pgmap v5712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:06 smithi189 ceph-mon[36790]: pgmap v5712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9344.676904 secs 2024-07-26T14:03:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9344.676904 secs 2024-07-26T14:03:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9344.676904 secs 2024-07-26T14:03:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:08 smithi193 ceph-mon[37473]: pgmap v5713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:08.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:08 smithi134 ceph-mon[31966]: pgmap v5713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:08 smithi189 ceph-mon[36790]: pgmap v5713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9330.251884 secs 2024-07-26T14:03:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9330.251884 secs 2024-07-26T14:03:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9330.251884 secs 2024-07-26T14:03:10.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:03:10.452+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:03:10.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:03:10.452+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:10.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:03:10.452+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:03:10.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:10 smithi134 ceph-mon[31966]: pgmap v5714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:10 smithi189 ceph-mon[36790]: pgmap v5714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:10 smithi193 ceph-mon[37473]: pgmap v5714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9349.677033 secs 2024-07-26T14:03:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9349.677033 secs 2024-07-26T14:03:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9349.677033 secs 2024-07-26T14:03:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:12 smithi134 ceph-mon[31966]: pgmap v5715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:12 smithi189 ceph-mon[36790]: pgmap v5715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:12 smithi193 ceph-mon[37473]: pgmap v5715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:14 smithi134 ceph-mon[31966]: pgmap v5716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9335.252016 secs 2024-07-26T14:03:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:14 smithi189 ceph-mon[36790]: pgmap v5716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9335.252016 secs 2024-07-26T14:03:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:14 smithi193 ceph-mon[37473]: pgmap v5716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9335.252016 secs 2024-07-26T14:03:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:16 smithi134 ceph-mon[31966]: pgmap v5717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9354.677178 secs 2024-07-26T14:03:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:16 smithi189 ceph-mon[36790]: pgmap v5717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9354.677178 secs 2024-07-26T14:03:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:16 smithi193 ceph-mon[37473]: pgmap v5717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9354.677178 secs 2024-07-26T14:03:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:18 smithi134 ceph-mon[31966]: pgmap v5718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:18 smithi189 ceph-mon[36790]: pgmap v5718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:18 smithi193 ceph-mon[37473]: pgmap v5718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9340.252145 secs 2024-07-26T14:03:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:03:19.924 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9340.252145 secs 2024-07-26T14:03:19.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:03:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9340.252145 secs 2024-07-26T14:03:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:03:20.862 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:03:20.860+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:03:20.862 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:03:20.861+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:03:20.862 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:03:20.861+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:20 smithi134 ceph-mon[31966]: pgmap v5719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:03:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:03:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:03:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:20 smithi189 ceph-mon[36790]: pgmap v5719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:03:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:03:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:03:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:20 smithi193 ceph-mon[37473]: pgmap v5719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:03:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:03:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:03:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:21 smithi134 ceph-mon[31966]: pgmap v5720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:03:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:21 smithi134 ceph-mon[31966]: pgmap v5721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:03:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9359.677263 secs 2024-07-26T14:03:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:21 smithi189 ceph-mon[36790]: pgmap v5720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:03:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:21 smithi189 ceph-mon[36790]: pgmap v5721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:03:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9359.677263 secs 2024-07-26T14:03:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:21 smithi193 ceph-mon[37473]: pgmap v5720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:03:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:21 smithi193 ceph-mon[37473]: pgmap v5721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:03:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9359.677263 secs 2024-07-26T14:03:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:23 smithi134 ceph-mon[31966]: pgmap v5722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:03:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:23 smithi189 ceph-mon[36790]: pgmap v5722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:03:23.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:03:23.955+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T14:03:23.956 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:03:23.955+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T14:03:23.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:03:23.955+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T14:03:23.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:03:23.955+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T14:03:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:23 smithi193 ceph-mon[37473]: pgmap v5722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:03:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9345.252228 secs 2024-07-26T14:03:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9345.252228 secs 2024-07-26T14:03:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9345.252228 secs 2024-07-26T14:03:25.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:03:25.465+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:03:25.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:03:25.465+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:25.467 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:03:25.465+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:03:25.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:25 smithi134 ceph-mon[31966]: pgmap v5723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:03:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:25 smithi189 ceph-mon[36790]: pgmap v5723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:03:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:25 smithi193 ceph-mon[37473]: pgmap v5723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:03:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9364.677385 secs 2024-07-26T14:03:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9364.677385 secs 2024-07-26T14:03:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9364.677385 secs 2024-07-26T14:03:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:27 smithi134 ceph-mon[31966]: pgmap v5724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:03:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:27 smithi189 ceph-mon[36790]: pgmap v5724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:03:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:27 smithi193 ceph-mon[37473]: pgmap v5724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:03:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:29 smithi134 ceph-mon[31966]: pgmap v5725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:03:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9350.252338 secs 2024-07-26T14:03:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:29 smithi189 ceph-mon[36790]: pgmap v5725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:03:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9350.252338 secs 2024-07-26T14:03:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:29 smithi193 ceph-mon[37473]: pgmap v5725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:03:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9350.252338 secs 2024-07-26T14:03:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:31 smithi134 ceph-mon[31966]: pgmap v5726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:03:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9369.677513 secs 2024-07-26T14:03:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:31 smithi189 ceph-mon[36790]: pgmap v5726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:03:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9369.677513 secs 2024-07-26T14:03:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:31 smithi193 ceph-mon[37473]: pgmap v5726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:03:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9369.677513 secs 2024-07-26T14:03:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:33 smithi134 ceph-mon[31966]: pgmap v5727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:33 smithi189 ceph-mon[36790]: pgmap v5727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:33 smithi193 ceph-mon[37473]: pgmap v5727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:03:34.479+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:03:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:03:34.479+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:03:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:03:34.479+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:03:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:03:34.480+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:03:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9355.252407 secs 2024-07-26T14:03:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9355.252407 secs 2024-07-26T14:03:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9355.252407 secs 2024-07-26T14:03:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:03:35.867+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:03:35.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:03:35.867+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:03:35.869 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:03:35.867+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:35.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:35 smithi134 ceph-mon[31966]: pgmap v5728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:35 smithi189 ceph-mon[36790]: pgmap v5728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:35 smithi193 ceph-mon[37473]: pgmap v5728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9374.677629 secs 2024-07-26T14:03:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9374.677629 secs 2024-07-26T14:03:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9374.677629 secs 2024-07-26T14:03:37.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:37 smithi134 ceph-mon[31966]: pgmap v5729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:37 smithi189 ceph-mon[36790]: pgmap v5729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:37 smithi193 ceph-mon[37473]: pgmap v5729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:39 smithi134 ceph-mon[31966]: pgmap v5730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9360.252528 secs 2024-07-26T14:03:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:39 smithi189 ceph-mon[36790]: pgmap v5730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9360.252528 secs 2024-07-26T14:03:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:39 smithi193 ceph-mon[37473]: pgmap v5730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9360.252528 secs 2024-07-26T14:03:40.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:03:40.480+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:03:40.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:03:40.480+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:03:40.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:03:40.480+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:41 smithi134 ceph-mon[31966]: pgmap v5731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9379.677692 secs 2024-07-26T14:03:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:41 smithi189 ceph-mon[36790]: pgmap v5731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9379.677692 secs 2024-07-26T14:03:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:41 smithi193 ceph-mon[37473]: pgmap v5731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9379.677692 secs 2024-07-26T14:03:43.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:43 smithi134 ceph-mon[31966]: pgmap v5732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:43 smithi189 ceph-mon[36790]: pgmap v5732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:43 smithi193 ceph-mon[37473]: pgmap v5732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9365.252648 secs 2024-07-26T14:03:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9365.252648 secs 2024-07-26T14:03:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9365.252648 secs 2024-07-26T14:03:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:45 smithi189 ceph-mon[36790]: pgmap v5733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:45 smithi193 ceph-mon[37473]: pgmap v5733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:45 smithi134 ceph-mon[31966]: pgmap v5733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9384.677819 secs 2024-07-26T14:03:47.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9384.677819 secs 2024-07-26T14:03:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9384.677819 secs 2024-07-26T14:03:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:47 smithi193 ceph-mon[37473]: pgmap v5734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:48.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:47 smithi189 ceph-mon[36790]: pgmap v5734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:47 smithi134 ceph-mon[31966]: pgmap v5734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:49 smithi193 ceph-mon[37473]: pgmap v5735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9370.252800 secs 2024-07-26T14:03:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:49 smithi134 ceph-mon[31966]: pgmap v5735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9370.252800 secs 2024-07-26T14:03:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:49 smithi189 ceph-mon[36790]: pgmap v5735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:03:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9370.252800 secs 2024-07-26T14:03:50.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:03:50.883+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:03:50.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:03:50.883+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:03:50.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:03:50.883+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:51 smithi193 ceph-mon[37473]: pgmap v5736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9389.677948 secs 2024-07-26T14:03:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:51 smithi134 ceph-mon[31966]: pgmap v5736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9389.677948 secs 2024-07-26T14:03:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:51 smithi189 ceph-mon[36790]: pgmap v5736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9389.677948 secs 2024-07-26T14:03:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:53 smithi193 ceph-mon[37473]: pgmap v5737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:53 smithi134 ceph-mon[31966]: pgmap v5737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:53 smithi189 ceph-mon[36790]: pgmap v5737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:03:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9375.252928 secs 2024-07-26T14:03:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9375.252928 secs 2024-07-26T14:03:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9375.252928 secs 2024-07-26T14:03:55.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:03:55.482+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:03:55.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:03:55.483+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:03:55.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:03:55.483+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:03:55.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:55 smithi134 ceph-mon[31966]: pgmap v5738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:55 smithi193 ceph-mon[37473]: pgmap v5738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:55 smithi189 ceph-mon[36790]: pgmap v5738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:03:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9394.678069 secs 2024-07-26T14:03:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9394.678069 secs 2024-07-26T14:03:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9394.678069 secs 2024-07-26T14:03:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:57 smithi193 ceph-mon[37473]: pgmap v5739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:57 smithi134 ceph-mon[31966]: pgmap v5739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:57 smithi189 ceph-mon[36790]: pgmap v5739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:03:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9380.253060 secs 2024-07-26T14:03:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9380.253060 secs 2024-07-26T14:03:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9380.253060 secs 2024-07-26T14:04:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:03:59 smithi193 ceph-mon[37473]: pgmap v5740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:03:59 smithi134 ceph-mon[31966]: pgmap v5740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:03:59 smithi189 ceph-mon[36790]: pgmap v5740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:01 smithi193 ceph-mon[37473]: pgmap v5741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9399.678216 secs 2024-07-26T14:04:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:01 smithi134 ceph-mon[31966]: pgmap v5741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9399.678216 secs 2024-07-26T14:04:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:01 smithi189 ceph-mon[36790]: pgmap v5741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9399.678216 secs 2024-07-26T14:04:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:03 smithi193 ceph-mon[37473]: pgmap v5742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:04:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9385.253190 secs 2024-07-26T14:04:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:03 smithi134 ceph-mon[31966]: pgmap v5742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:04:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9385.253190 secs 2024-07-26T14:04:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:03 smithi189 ceph-mon[36790]: pgmap v5742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:04:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9385.253190 secs 2024-07-26T14:04:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:04:05.898+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:04:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:04:05.898+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:04:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:04:05.898+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:06.152 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:05 smithi134 ceph-mon[31966]: pgmap v5743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:05 smithi189 ceph-mon[36790]: pgmap v5743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:05 smithi193 ceph-mon[37473]: pgmap v5743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9404.678357 secs 2024-07-26T14:04:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9404.678357 secs 2024-07-26T14:04:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9404.678357 secs 2024-07-26T14:04:08.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:07 smithi134 ceph-mon[31966]: pgmap v5744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:07 smithi189 ceph-mon[36790]: pgmap v5744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:07 smithi193 ceph-mon[37473]: pgmap v5744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:08.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:04:08.963+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T14:04:08.964 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:04:08.963+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T14:04:08.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:04:08.963+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T14:04:08.965 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:04:08.963+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T14:04:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:09 smithi134 ceph-mon[31966]: pgmap v5745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9390.253279 secs 2024-07-26T14:04:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:09 smithi189 ceph-mon[36790]: pgmap v5745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9390.253279 secs 2024-07-26T14:04:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:09 smithi193 ceph-mon[37473]: pgmap v5745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9390.253279 secs 2024-07-26T14:04:10.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:04:10.498+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:04:10.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:04:10.498+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:10.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:04:10.498+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:04:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:11 smithi134 ceph-mon[31966]: pgmap v5746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9409.678456 secs 2024-07-26T14:04:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:11 smithi189 ceph-mon[36790]: pgmap v5746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9409.678456 secs 2024-07-26T14:04:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:11 smithi193 ceph-mon[37473]: pgmap v5746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9409.678456 secs 2024-07-26T14:04:13.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:13 smithi134 ceph-mon[31966]: pgmap v5747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:04:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:13 smithi189 ceph-mon[36790]: pgmap v5747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:04:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:13 smithi193 ceph-mon[37473]: pgmap v5747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:04:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9395.253411 secs 2024-07-26T14:04:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9395.253411 secs 2024-07-26T14:04:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9395.253411 secs 2024-07-26T14:04:15.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:15 smithi134 ceph-mon[31966]: pgmap v5748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:15 smithi189 ceph-mon[36790]: pgmap v5748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:15 smithi193 ceph-mon[37473]: pgmap v5748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:04:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9414.678557 secs 2024-07-26T14:04:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9414.678557 secs 2024-07-26T14:04:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9414.678557 secs 2024-07-26T14:04:17.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:17 smithi134 ceph-mon[31966]: pgmap v5749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:17 smithi189 ceph-mon[36790]: pgmap v5749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:17 smithi193 ceph-mon[37473]: pgmap v5749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:19.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:04:19.487+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:04:19.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:04:19.487+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:04:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:04:19.488+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:04:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:04:19.488+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:04:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:19 smithi134 ceph-mon[31966]: pgmap v5750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9400.253542 secs 2024-07-26T14:04:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:19 smithi189 ceph-mon[36790]: pgmap v5750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9400.253542 secs 2024-07-26T14:04:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:19 smithi193 ceph-mon[37473]: pgmap v5750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9400.253542 secs 2024-07-26T14:04:20.914 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:04:20.913+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:04:20.914 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:04:20.913+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:20.914 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:04:20.913+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:04:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:04:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:04:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:04:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:21 smithi134 ceph-mon[31966]: pgmap v5751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9419.678619 secs 2024-07-26T14:04:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:04:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:04:21.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:04:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:21 smithi189 ceph-mon[36790]: pgmap v5751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9419.678619 secs 2024-07-26T14:04:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:04:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:04:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:04:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:21 smithi193 ceph-mon[37473]: pgmap v5751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9419.678619 secs 2024-07-26T14:04:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:04:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:04:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:04:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:04:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:22 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:04:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:22 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:04:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:22 smithi134 ceph-mon[31966]: pgmap v5752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:04:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:22 smithi134 ceph-mon[31966]: pgmap v5753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:04:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:22 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:04:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:22 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:04:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:22 smithi189 ceph-mon[36790]: pgmap v5752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:04:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:22 smithi189 ceph-mon[36790]: pgmap v5753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:04:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:22 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:04:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:22 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:04:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:22 smithi193 ceph-mon[37473]: pgmap v5752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:04:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:22 smithi193 ceph-mon[37473]: pgmap v5753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:04:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:24 smithi134 ceph-mon[31966]: pgmap v5754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9405.253663 secs 2024-07-26T14:04:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:24 smithi189 ceph-mon[36790]: pgmap v5754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9405.253663 secs 2024-07-26T14:04:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:24 smithi193 ceph-mon[37473]: pgmap v5754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9405.253663 secs 2024-07-26T14:04:25.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:04:25.499+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:04:25.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:04:25.499+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:04:25.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:04:25.499+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:26 smithi134 ceph-mon[31966]: pgmap v5755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9424.678743 secs 2024-07-26T14:04:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:26 smithi189 ceph-mon[36790]: pgmap v5755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9424.678743 secs 2024-07-26T14:04:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:26 smithi193 ceph-mon[37473]: pgmap v5755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9424.678743 secs 2024-07-26T14:04:28.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:28 smithi134 ceph-mon[31966]: pgmap v5756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:04:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:28 smithi189 ceph-mon[36790]: pgmap v5756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:04:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:28 smithi193 ceph-mon[37473]: pgmap v5756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:04:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9410.253787 secs 2024-07-26T14:04:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9410.253787 secs 2024-07-26T14:04:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9410.253787 secs 2024-07-26T14:04:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:30 smithi134 ceph-mon[31966]: pgmap v5757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:30 smithi189 ceph-mon[36790]: pgmap v5757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:30 smithi193 ceph-mon[37473]: pgmap v5757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9429.678869 secs 2024-07-26T14:04:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9429.678869 secs 2024-07-26T14:04:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9429.678869 secs 2024-07-26T14:04:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:32 smithi189 ceph-mon[36790]: pgmap v5758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:04:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:32 smithi193 ceph-mon[37473]: pgmap v5758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:04:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:32 smithi134 ceph-mon[31966]: pgmap v5758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:04:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:34 smithi189 ceph-mon[36790]: pgmap v5759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:04:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9415.253914 secs 2024-07-26T14:04:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:34 smithi193 ceph-mon[37473]: pgmap v5759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:04:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9415.253914 secs 2024-07-26T14:04:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:34 smithi134 ceph-mon[31966]: pgmap v5759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:04:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9415.253914 secs 2024-07-26T14:04:35.917 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:04:35.916+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:04:35.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:04:35.916+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:35.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:04:35.916+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:04:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:36 smithi189 ceph-mon[36790]: pgmap v5760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9434.678965 secs 2024-07-26T14:04:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:36 smithi193 ceph-mon[37473]: pgmap v5760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9434.678965 secs 2024-07-26T14:04:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:36 smithi134 ceph-mon[31966]: pgmap v5760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9434.678965 secs 2024-07-26T14:04:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:38 smithi189 ceph-mon[36790]: pgmap v5761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:38 smithi193 ceph-mon[37473]: pgmap v5761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:39.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:38 smithi134 ceph-mon[31966]: pgmap v5761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9420.254056 secs 2024-07-26T14:04:40.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9420.254056 secs 2024-07-26T14:04:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9420.254056 secs 2024-07-26T14:04:40.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:04:40.500+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:04:40.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:04:40.500+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:04:40.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:04:40.500+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:40 smithi134 ceph-mon[31966]: pgmap v5762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:40 smithi193 ceph-mon[37473]: pgmap v5762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:40 smithi189 ceph-mon[36790]: pgmap v5762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9439.679102 secs 2024-07-26T14:04:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9439.679102 secs 2024-07-26T14:04:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9439.679102 secs 2024-07-26T14:04:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:42 smithi193 ceph-mon[37473]: pgmap v5763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:43.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:42 smithi134 ceph-mon[31966]: pgmap v5763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:42 smithi189 ceph-mon[36790]: pgmap v5763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:44 smithi193 ceph-mon[37473]: pgmap v5764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9425.254155 secs 2024-07-26T14:04:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:44 smithi134 ceph-mon[31966]: pgmap v5764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9425.254155 secs 2024-07-26T14:04:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:44 smithi189 ceph-mon[36790]: pgmap v5764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9425.254155 secs 2024-07-26T14:04:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:46 smithi193 ceph-mon[37473]: pgmap v5765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9444.679231 secs 2024-07-26T14:04:47.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:46 smithi134 ceph-mon[31966]: pgmap v5765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9444.679231 secs 2024-07-26T14:04:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:46 smithi189 ceph-mon[36790]: pgmap v5765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9444.679231 secs 2024-07-26T14:04:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:48 smithi193 ceph-mon[37473]: pgmap v5766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:49.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:48 smithi134 ceph-mon[31966]: pgmap v5766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:48 smithi189 ceph-mon[36790]: pgmap v5766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9430.254289 secs 2024-07-26T14:04:50.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9430.254289 secs 2024-07-26T14:04:50.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9430.254289 secs 2024-07-26T14:04:50.933 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:04:50.931+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:04:50.933 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:04:50.932+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:04:50.933 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:04:50.932+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:50 smithi193 ceph-mon[37473]: pgmap v5767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:51.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:50 smithi134 ceph-mon[31966]: pgmap v5767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:50 smithi189 ceph-mon[36790]: pgmap v5767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:04:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9449.679356 secs 2024-07-26T14:04:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9449.679356 secs 2024-07-26T14:04:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9449.679356 secs 2024-07-26T14:04:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:52 smithi193 ceph-mon[37473]: pgmap v5768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:52 smithi134 ceph-mon[31966]: pgmap v5768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:52 smithi189 ceph-mon[36790]: pgmap v5768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9435.254427 secs 2024-07-26T14:04:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9435.254427 secs 2024-07-26T14:04:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9435.254427 secs 2024-07-26T14:04:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:54 smithi134 ceph-mon[31966]: pgmap v5769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:54 smithi189 ceph-mon[36790]: pgmap v5769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:54 smithi193 ceph-mon[37473]: pgmap v5769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:04:55.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:04:55.504+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:04:55.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:04:55.504+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:04:55.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:04:55.504+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:04:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:56 smithi134 ceph-mon[31966]: pgmap v5770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9454.679440 secs 2024-07-26T14:04:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:56 smithi189 ceph-mon[36790]: pgmap v5770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9454.679440 secs 2024-07-26T14:04:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:56 smithi193 ceph-mon[37473]: pgmap v5770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:04:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9454.679440 secs 2024-07-26T14:04:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:58 smithi134 ceph-mon[31966]: pgmap v5771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:04:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9440.254494 secs 2024-07-26T14:04:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:58 smithi189 ceph-mon[36790]: pgmap v5771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:04:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9440.254494 secs 2024-07-26T14:04:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:58 smithi193 ceph-mon[37473]: pgmap v5771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:04:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:04:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9440.254494 secs 2024-07-26T14:05:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:00 smithi134 ceph-mon[31966]: pgmap v5772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:00 smithi189 ceph-mon[36790]: pgmap v5772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:00 smithi193 ceph-mon[37473]: pgmap v5772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9459.679561 secs 2024-07-26T14:05:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9459.679561 secs 2024-07-26T14:05:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9459.679561 secs 2024-07-26T14:05:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:02 smithi134 ceph-mon[31966]: pgmap v5773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:02 smithi189 ceph-mon[36790]: pgmap v5773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:02 smithi193 ceph-mon[37473]: pgmap v5773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:04.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9445.254638 secs 2024-07-26T14:05:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9445.254638 secs 2024-07-26T14:05:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9445.254638 secs 2024-07-26T14:05:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:04 smithi134 ceph-mon[31966]: pgmap v5774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:04 smithi189 ceph-mon[36790]: pgmap v5774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:04 smithi193 ceph-mon[37473]: pgmap v5774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:05.948 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:05:05.947+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:05:05.949 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:05:05.947+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:05.949 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:05:05.947+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:05:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:06 smithi134 ceph-mon[31966]: pgmap v5775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9464.679654 secs 2024-07-26T14:05:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:06 smithi189 ceph-mon[36790]: pgmap v5775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9464.679654 secs 2024-07-26T14:05:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:06 smithi193 ceph-mon[37473]: pgmap v5775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9464.679654 secs 2024-07-26T14:05:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:08 smithi134 ceph-mon[31966]: pgmap v5776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9450.254747 secs 2024-07-26T14:05:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:08 smithi189 ceph-mon[36790]: pgmap v5776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9450.254747 secs 2024-07-26T14:05:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:08 smithi193 ceph-mon[37473]: pgmap v5776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9450.254747 secs 2024-07-26T14:05:10.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:05:10.520+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:05:10.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:05:10.520+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:05:10.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:05:10.520+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:10 smithi134 ceph-mon[31966]: pgmap v5777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:10 smithi189 ceph-mon[36790]: pgmap v5777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:11.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:10 smithi193 ceph-mon[37473]: pgmap v5777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9469.679780 secs 2024-07-26T14:05:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9469.679780 secs 2024-07-26T14:05:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9469.679780 secs 2024-07-26T14:05:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:12 smithi134 ceph-mon[31966]: pgmap v5778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:12 smithi189 ceph-mon[36790]: pgmap v5778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:12 smithi193 ceph-mon[37473]: pgmap v5778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:14.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9455.254890 secs 2024-07-26T14:05:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9455.254890 secs 2024-07-26T14:05:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9455.254890 secs 2024-07-26T14:05:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:14 smithi134 ceph-mon[31966]: pgmap v5779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:14 smithi189 ceph-mon[36790]: pgmap v5779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:14 smithi193 ceph-mon[37473]: pgmap v5779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:15 smithi189 ceph-mon[36790]: pgmap v5780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:15 smithi193 ceph-mon[37473]: pgmap v5780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:16.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:15 smithi134 ceph-mon[31966]: pgmap v5780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9474.679909 secs 2024-07-26T14:05:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9474.679909 secs 2024-07-26T14:05:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9474.679909 secs 2024-07-26T14:05:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:17 smithi189 ceph-mon[36790]: pgmap v5781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:17 smithi193 ceph-mon[37473]: pgmap v5781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:18.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:17 smithi134 ceph-mon[31966]: pgmap v5781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:05:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9460.254989 secs 2024-07-26T14:05:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9460.254989 secs 2024-07-26T14:05:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9460.254989 secs 2024-07-26T14:05:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:19 smithi193 ceph-mon[37473]: pgmap v5782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:19 smithi134 ceph-mon[31966]: pgmap v5782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:19 smithi189 ceph-mon[36790]: pgmap v5782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:05:20.961 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:05:20.960+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:05:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:05:20.960+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:05:20.962 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:05:20.960+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9479.680033 secs 2024-07-26T14:05:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9479.680033 secs 2024-07-26T14:05:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9479.680033 secs 2024-07-26T14:05:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:22 smithi134 ceph-mon[31966]: pgmap v5783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:05:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:22 smithi189 ceph-mon[36790]: pgmap v5783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:05:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:22 smithi193 ceph-mon[37473]: pgmap v5783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:05:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:05:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:23 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:05:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:23 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:05:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:23 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:05:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:23 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:05:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:05:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:05:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:23 smithi134 ceph-mon[31966]: pgmap v5784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:05:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:23 smithi134 ceph-mon[31966]: pgmap v5785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:05:23.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:05:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:23 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:05:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:23 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:05:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:23 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:05:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:23 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:05:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:05:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:05:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:23 smithi189 ceph-mon[36790]: pgmap v5784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:05:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:23 smithi189 ceph-mon[36790]: pgmap v5785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:05:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:05:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:23 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:05:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:23 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:05:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:23 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:05:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:23 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:05:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:05:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:05:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:23 smithi193 ceph-mon[37473]: pgmap v5784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:05:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:23 smithi193 ceph-mon[37473]: pgmap v5785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:05:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:05:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9465.255112 secs 2024-07-26T14:05:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9465.255112 secs 2024-07-26T14:05:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9465.255112 secs 2024-07-26T14:05:25.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:05:25.528+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:05:25.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:05:25.528+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:25.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:05:25.528+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:05:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:25 smithi134 ceph-mon[31966]: pgmap v5786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:05:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:25 smithi189 ceph-mon[36790]: pgmap v5786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:05:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:25 smithi193 ceph-mon[37473]: pgmap v5786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:05:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9484.680126 secs 2024-07-26T14:05:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9484.680126 secs 2024-07-26T14:05:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9484.680126 secs 2024-07-26T14:05:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:27 smithi134 ceph-mon[31966]: pgmap v5787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:27 smithi189 ceph-mon[36790]: pgmap v5787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:27 smithi193 ceph-mon[37473]: pgmap v5787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9470.255237 secs 2024-07-26T14:05:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:29 smithi193 ceph-mon[37473]: pgmap v5788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:05:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9470.255237 secs 2024-07-26T14:05:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:29 smithi134 ceph-mon[31966]: pgmap v5788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:05:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9470.255237 secs 2024-07-26T14:05:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:29 smithi189 ceph-mon[36790]: pgmap v5788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:05:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:31 smithi193 ceph-mon[37473]: pgmap v5789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:05:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9489.680268 secs 2024-07-26T14:05:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:31 smithi134 ceph-mon[31966]: pgmap v5789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:05:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9489.680268 secs 2024-07-26T14:05:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:31 smithi189 ceph-mon[36790]: pgmap v5789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:05:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9489.680268 secs 2024-07-26T14:05:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:33 smithi193 ceph-mon[37473]: pgmap v5790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:05:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:33 smithi134 ceph-mon[31966]: pgmap v5790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:05:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:33 smithi189 ceph-mon[36790]: pgmap v5790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:05:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9475.255370 secs 2024-07-26T14:05:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9475.255370 secs 2024-07-26T14:05:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9475.255370 secs 2024-07-26T14:05:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:05:35.976+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:05:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:05:35.976+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:05:35.976+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:05:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:35 smithi193 ceph-mon[37473]: pgmap v5791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:05:36.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:35 smithi134 ceph-mon[31966]: pgmap v5791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:05:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:35 smithi189 ceph-mon[36790]: pgmap v5791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:05:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9494.680387 secs 2024-07-26T14:05:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9494.680387 secs 2024-07-26T14:05:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9494.680387 secs 2024-07-26T14:05:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:37 smithi193 ceph-mon[37473]: pgmap v5792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:38.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:37 smithi134 ceph-mon[31966]: pgmap v5792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:37 smithi189 ceph-mon[36790]: pgmap v5792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9480.255502 secs 2024-07-26T14:05:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:39 smithi193 ceph-mon[37473]: pgmap v5793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:40.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9480.255502 secs 2024-07-26T14:05:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:39 smithi134 ceph-mon[31966]: pgmap v5793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9480.255502 secs 2024-07-26T14:05:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:39 smithi189 ceph-mon[36790]: pgmap v5793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:40.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:05:40.530+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:05:40.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:05:40.530+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:40.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:05:40.530+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:05:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:41 smithi193 ceph-mon[37473]: pgmap v5794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9499.680507 secs 2024-07-26T14:05:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:41 smithi134 ceph-mon[31966]: pgmap v5794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9499.680507 secs 2024-07-26T14:05:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:41 smithi189 ceph-mon[36790]: pgmap v5794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9499.680507 secs 2024-07-26T14:05:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:43 smithi193 ceph-mon[37473]: pgmap v5795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9485.255637 secs 2024-07-26T14:05:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:43 smithi134 ceph-mon[31966]: pgmap v5795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9485.255637 secs 2024-07-26T14:05:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:43 smithi189 ceph-mon[36790]: pgmap v5795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9485.255637 secs 2024-07-26T14:05:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:45 smithi193 ceph-mon[37473]: pgmap v5796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:05:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:45 smithi134 ceph-mon[31966]: pgmap v5796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:05:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:45 smithi189 ceph-mon[36790]: pgmap v5796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:05:47.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9504.680627 secs 2024-07-26T14:05:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9504.680627 secs 2024-07-26T14:05:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9504.680627 secs 2024-07-26T14:05:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:47 smithi134 ceph-mon[31966]: pgmap v5797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:47 smithi189 ceph-mon[36790]: pgmap v5797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:47 smithi193 ceph-mon[37473]: pgmap v5797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:49.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9490.255768 secs 2024-07-26T14:05:49.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9490.255768 secs 2024-07-26T14:05:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9490.255768 secs 2024-07-26T14:05:50.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:49 smithi189 ceph-mon[36790]: pgmap v5798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:50.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:49 smithi134 ceph-mon[31966]: pgmap v5798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:49 smithi193 ceph-mon[37473]: pgmap v5798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:50.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:05:50.991+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:05:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:05:50.991+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:50.993 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:05:50.991+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:05:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:51 smithi134 ceph-mon[31966]: pgmap v5799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9509.680750 secs 2024-07-26T14:05:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:51 smithi189 ceph-mon[36790]: pgmap v5799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9509.680750 secs 2024-07-26T14:05:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:51 smithi193 ceph-mon[37473]: pgmap v5799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9509.680750 secs 2024-07-26T14:05:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:53 smithi134 ceph-mon[31966]: pgmap v5800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9495.255902 secs 2024-07-26T14:05:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:53 smithi189 ceph-mon[36790]: pgmap v5800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9495.255902 secs 2024-07-26T14:05:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:53 smithi193 ceph-mon[37473]: pgmap v5800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:05:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9495.255902 secs 2024-07-26T14:05:55.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:05:55.546+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:05:55.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:05:55.546+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:05:55.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:05:55.546+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:05:56.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:55 smithi134 ceph-mon[31966]: pgmap v5801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:05:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:55 smithi189 ceph-mon[36790]: pgmap v5801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:05:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:55 smithi193 ceph-mon[37473]: pgmap v5801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:05:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9514.680866 secs 2024-07-26T14:05:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9514.680866 secs 2024-07-26T14:05:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9514.680866 secs 2024-07-26T14:05:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:57 smithi134 ceph-mon[31966]: pgmap v5802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:57 smithi189 ceph-mon[36790]: pgmap v5802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:57 smithi193 ceph-mon[37473]: pgmap v5802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:05:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9500.256040 secs 2024-07-26T14:05:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9500.256040 secs 2024-07-26T14:05:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9500.256040 secs 2024-07-26T14:06:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:05:59 smithi134 ceph-mon[31966]: pgmap v5803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:05:59 smithi189 ceph-mon[36790]: pgmap v5803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:05:59 smithi193 ceph-mon[37473]: pgmap v5803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:01 smithi134 ceph-mon[31966]: pgmap v5804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9519.680943 secs 2024-07-26T14:06:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:01 smithi189 ceph-mon[36790]: pgmap v5804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9519.680943 secs 2024-07-26T14:06:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:01 smithi193 ceph-mon[37473]: pgmap v5804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9519.680943 secs 2024-07-26T14:06:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:03 smithi134 ceph-mon[31966]: pgmap v5805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9505.256107 secs 2024-07-26T14:06:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:03 smithi189 ceph-mon[36790]: pgmap v5805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9505.256107 secs 2024-07-26T14:06:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:03 smithi193 ceph-mon[37473]: pgmap v5805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9505.256107 secs 2024-07-26T14:06:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:06:06.007+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:06:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:06:06.007+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:06:06.007+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:06:06.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:05 smithi134 ceph-mon[31966]: pgmap v5806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:06:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:05 smithi189 ceph-mon[36790]: pgmap v5806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:06:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:05 smithi193 ceph-mon[37473]: pgmap v5806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:06:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9524.681062 secs 2024-07-26T14:06:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9524.681062 secs 2024-07-26T14:06:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9524.681062 secs 2024-07-26T14:06:08.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:07 smithi134 ceph-mon[31966]: pgmap v5807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:07 smithi189 ceph-mon[36790]: pgmap v5807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:07 smithi193 ceph-mon[37473]: pgmap v5807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9510.256229 secs 2024-07-26T14:06:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9510.256229 secs 2024-07-26T14:06:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9510.256229 secs 2024-07-26T14:06:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:09 smithi134 ceph-mon[31966]: pgmap v5808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:09 smithi189 ceph-mon[36790]: pgmap v5808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:09 smithi193 ceph-mon[37473]: pgmap v5808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:10.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:06:10.559+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:06:10.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:06:10.559+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:10.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:06:10.559+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:06:11.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:10 smithi134 ceph-mon[31966]: pgmap v5809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:10 smithi189 ceph-mon[36790]: pgmap v5809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:11.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:10 smithi193 ceph-mon[37473]: pgmap v5809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9529.681126 secs 2024-07-26T14:06:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9529.681126 secs 2024-07-26T14:06:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9529.681126 secs 2024-07-26T14:06:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:12 smithi134 ceph-mon[31966]: pgmap v5810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:12 smithi189 ceph-mon[36790]: pgmap v5810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:12 smithi193 ceph-mon[37473]: pgmap v5810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:14.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9515.256345 secs 2024-07-26T14:06:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9515.256345 secs 2024-07-26T14:06:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9515.256345 secs 2024-07-26T14:06:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:14 smithi189 ceph-mon[36790]: pgmap v5811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:06:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:14 smithi193 ceph-mon[37473]: pgmap v5811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:06:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:14 smithi134 ceph-mon[31966]: pgmap v5811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:06:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9534.681281 secs 2024-07-26T14:06:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9534.681281 secs 2024-07-26T14:06:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9534.681281 secs 2024-07-26T14:06:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:17 smithi134 ceph-mon[31966]: pgmap v5812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:17 smithi189 ceph-mon[36790]: pgmap v5812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:17 smithi193 ceph-mon[37473]: pgmap v5812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9520.256461 secs 2024-07-26T14:06:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:19 smithi193 ceph-mon[37473]: pgmap v5813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:20.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9520.256461 secs 2024-07-26T14:06:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:19 smithi134 ceph-mon[31966]: pgmap v5813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9520.256461 secs 2024-07-26T14:06:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:19 smithi189 ceph-mon[36790]: pgmap v5813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:06:21.009+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:06:21.010 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:06:21.009+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:21.011 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:06:21.009+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:06:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:21 smithi193 ceph-mon[37473]: pgmap v5814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9539.681410 secs 2024-07-26T14:06:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:21 smithi134 ceph-mon[31966]: pgmap v5814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9539.681410 secs 2024-07-26T14:06:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:21 smithi189 ceph-mon[36790]: pgmap v5814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9539.681410 secs 2024-07-26T14:06:23.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:06:23.986+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T14:06:23.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:06:23.986+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T14:06:23.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:06:23.987+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T14:06:23.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:06:23.987+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T14:06:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:23 smithi193 ceph-mon[37473]: pgmap v5815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:06:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:23 smithi134 ceph-mon[31966]: pgmap v5815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:06:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:23 smithi189 ceph-mon[36790]: pgmap v5815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:06:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9525.256559 secs 2024-07-26T14:06:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:06:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:06:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:24 smithi193 ceph-mon[37473]: pgmap v5816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:06:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:24 smithi193 ceph-mon[37473]: pgmap v5817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:06:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:06:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9525.256559 secs 2024-07-26T14:06:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:06:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:06:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:24 smithi134 ceph-mon[31966]: pgmap v5816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:06:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:24 smithi134 ceph-mon[31966]: pgmap v5817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:06:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:06:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9525.256559 secs 2024-07-26T14:06:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:06:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:06:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:24 smithi189 ceph-mon[36790]: pgmap v5816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:06:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:24 smithi189 ceph-mon[36790]: pgmap v5817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:06:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:06:25.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:06:25.575+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:06:25.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:06:25.575+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:06:25.576 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:06:25.575+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:25 smithi189 ceph-mon[36790]: pgmap v5818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:06:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:25 smithi193 ceph-mon[37473]: pgmap v5818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:06:26.436 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:25 smithi134 ceph-mon[31966]: pgmap v5818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:06:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9544.681528 secs 2024-07-26T14:06:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9544.681528 secs 2024-07-26T14:06:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9544.681528 secs 2024-07-26T14:06:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:27 smithi193 ceph-mon[37473]: pgmap v5819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:06:28.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:27 smithi134 ceph-mon[31966]: pgmap v5819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:06:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:27 smithi189 ceph-mon[36790]: pgmap v5819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:06:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9530.256680 secs 2024-07-26T14:06:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9530.256680 secs 2024-07-26T14:06:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9530.256680 secs 2024-07-26T14:06:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:29 smithi193 ceph-mon[37473]: pgmap v5820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:06:30.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:29 smithi134 ceph-mon[31966]: pgmap v5820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:06:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:29 smithi189 ceph-mon[36790]: pgmap v5820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:06:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9549.681652 secs 2024-07-26T14:06:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9549.681652 secs 2024-07-26T14:06:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9549.681652 secs 2024-07-26T14:06:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:32 smithi134 ceph-mon[31966]: pgmap v5821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:06:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:32 smithi189 ceph-mon[36790]: pgmap v5821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:06:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:32 smithi193 ceph-mon[37473]: pgmap v5821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:06:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9535.256829 secs 2024-07-26T14:06:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9535.256829 secs 2024-07-26T14:06:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9535.256829 secs 2024-07-26T14:06:34.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:06:34.511+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:06:34.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:06:34.511+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:06:34.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:06:34.512+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:06:34.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:06:34.512+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:06:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:34 smithi134 ceph-mon[31966]: pgmap v5822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:06:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:34 smithi189 ceph-mon[36790]: pgmap v5822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:06:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:34 smithi193 ceph-mon[37473]: pgmap v5822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:06:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:06:36.025+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:06:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:06:36.025+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:06:36.026 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:06:36.025+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:35 smithi189 ceph-mon[36790]: pgmap v5823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:35 smithi193 ceph-mon[37473]: pgmap v5823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:36.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:35 smithi134 ceph-mon[31966]: pgmap v5823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9554.681753 secs 2024-07-26T14:06:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9554.681753 secs 2024-07-26T14:06:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9554.681753 secs 2024-07-26T14:06:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:37 smithi193 ceph-mon[37473]: pgmap v5824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:38.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:37 smithi134 ceph-mon[31966]: pgmap v5824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:37 smithi189 ceph-mon[36790]: pgmap v5824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9540.256993 secs 2024-07-26T14:06:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9540.256993 secs 2024-07-26T14:06:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9540.256993 secs 2024-07-26T14:06:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:39 smithi193 ceph-mon[37473]: pgmap v5825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:06:40.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:39 smithi134 ceph-mon[31966]: pgmap v5825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:06:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:39 smithi189 ceph-mon[36790]: pgmap v5825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:06:40.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:06:40.587+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:06:40.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:06:40.587+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:06:40.588 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:06:40.587+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9559.681848 secs 2024-07-26T14:06:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9559.681848 secs 2024-07-26T14:06:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9559.681848 secs 2024-07-26T14:06:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:42 smithi134 ceph-mon[31966]: pgmap v5826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:42 smithi189 ceph-mon[36790]: pgmap v5826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:42 smithi193 ceph-mon[37473]: pgmap v5826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9545.257128 secs 2024-07-26T14:06:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9545.257128 secs 2024-07-26T14:06:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9545.257128 secs 2024-07-26T14:06:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:44 smithi134 ceph-mon[31966]: pgmap v5827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:44 smithi189 ceph-mon[36790]: pgmap v5827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:44 smithi193 ceph-mon[37473]: pgmap v5827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:45 smithi189 ceph-mon[36790]: pgmap v5828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:45 smithi193 ceph-mon[37473]: pgmap v5828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:46.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:45 smithi134 ceph-mon[31966]: pgmap v5828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9564.681976 secs 2024-07-26T14:06:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9564.681976 secs 2024-07-26T14:06:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9564.681976 secs 2024-07-26T14:06:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:47 smithi193 ceph-mon[37473]: pgmap v5829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:47 smithi134 ceph-mon[31966]: pgmap v5829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:47 smithi189 ceph-mon[36790]: pgmap v5829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9550.257267 secs 2024-07-26T14:06:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9550.257267 secs 2024-07-26T14:06:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9550.257267 secs 2024-07-26T14:06:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:49 smithi193 ceph-mon[37473]: pgmap v5830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:06:50.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:49 smithi189 ceph-mon[36790]: pgmap v5830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:06:50.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:49 smithi134 ceph-mon[31966]: pgmap v5830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:06:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:06:51.041+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:06:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:06:51.041+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:51.042 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:06:51.041+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:06:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9569.682094 secs 2024-07-26T14:06:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9569.682094 secs 2024-07-26T14:06:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9569.682094 secs 2024-07-26T14:06:52.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:52 smithi134 ceph-mon[31966]: pgmap v5831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:52 smithi189 ceph-mon[36790]: pgmap v5831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:52 smithi193 ceph-mon[37473]: pgmap v5831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9555.257396 secs 2024-07-26T14:06:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9555.257396 secs 2024-07-26T14:06:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9555.257396 secs 2024-07-26T14:06:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:54 smithi134 ceph-mon[31966]: pgmap v5832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:54 smithi189 ceph-mon[36790]: pgmap v5832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:54 smithi193 ceph-mon[37473]: pgmap v5832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:55.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:06:55.597+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:06:55.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:06:55.597+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:06:55.598 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:06:55.597+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:06:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:55 smithi189 ceph-mon[36790]: pgmap v5833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:55 smithi193 ceph-mon[37473]: pgmap v5833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:56.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:55 smithi134 ceph-mon[31966]: pgmap v5833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:06:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9574.682249 secs 2024-07-26T14:06:57.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9574.682249 secs 2024-07-26T14:06:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9574.682249 secs 2024-07-26T14:06:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:57 smithi193 ceph-mon[37473]: pgmap v5834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:58.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:57 smithi134 ceph-mon[31966]: pgmap v5834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:57 smithi189 ceph-mon[36790]: pgmap v5834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:06:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9560.257549 secs 2024-07-26T14:06:59.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9560.257549 secs 2024-07-26T14:06:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9560.257549 secs 2024-07-26T14:07:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:06:59 smithi193 ceph-mon[37473]: pgmap v5835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:07:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:06:59 smithi134 ceph-mon[31966]: pgmap v5835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:07:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:06:59 smithi189 ceph-mon[36790]: pgmap v5835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:07:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9579.682321 secs 2024-07-26T14:07:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9579.682321 secs 2024-07-26T14:07:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9579.682321 secs 2024-07-26T14:07:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:02 smithi134 ceph-mon[31966]: pgmap v5836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:02 smithi189 ceph-mon[36790]: pgmap v5836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:02 smithi193 ceph-mon[37473]: pgmap v5836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:04.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9565.257685 secs 2024-07-26T14:07:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9565.257685 secs 2024-07-26T14:07:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9565.257685 secs 2024-07-26T14:07:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:04 smithi134 ceph-mon[31966]: pgmap v5837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:04 smithi189 ceph-mon[36790]: pgmap v5837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:04 smithi193 ceph-mon[37473]: pgmap v5837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:06.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:07:06.056+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:07:06.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:07:06.056+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:07:06.057 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:07:06.056+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:05 smithi193 ceph-mon[37473]: pgmap v5838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:06.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:05 smithi134 ceph-mon[31966]: pgmap v5838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:05 smithi189 ceph-mon[36790]: pgmap v5838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9584.682442 secs 2024-07-26T14:07:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9584.682442 secs 2024-07-26T14:07:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9584.682442 secs 2024-07-26T14:07:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:07 smithi193 ceph-mon[37473]: pgmap v5839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:07 smithi134 ceph-mon[31966]: pgmap v5839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:07 smithi189 ceph-mon[36790]: pgmap v5839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9570.257816 secs 2024-07-26T14:07:09.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9570.257816 secs 2024-07-26T14:07:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9570.257816 secs 2024-07-26T14:07:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:09 smithi193 ceph-mon[37473]: pgmap v5840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:07:10.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:09 smithi134 ceph-mon[31966]: pgmap v5840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:07:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:09 smithi189 ceph-mon[36790]: pgmap v5840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:07:10.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:07:10.600+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:07:10.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:07:10.600+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:07:10.602 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:07:10.600+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:11.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9589.682567 secs 2024-07-26T14:07:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9589.682567 secs 2024-07-26T14:07:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9589.682567 secs 2024-07-26T14:07:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:12 smithi134 ceph-mon[31966]: pgmap v5841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:12 smithi189 ceph-mon[36790]: pgmap v5841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:12 smithi193 ceph-mon[37473]: pgmap v5841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9575.257971 secs 2024-07-26T14:07:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:14 smithi134 ceph-mon[31966]: pgmap v5842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9575.257971 secs 2024-07-26T14:07:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:14 smithi189 ceph-mon[36790]: pgmap v5842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9575.257971 secs 2024-07-26T14:07:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:14 smithi193 ceph-mon[37473]: pgmap v5842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:15 smithi193 ceph-mon[37473]: pgmap v5843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:16.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:15 smithi134 ceph-mon[31966]: pgmap v5843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:15 smithi189 ceph-mon[36790]: pgmap v5843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9594.682690 secs 2024-07-26T14:07:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9594.682690 secs 2024-07-26T14:07:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9594.682690 secs 2024-07-26T14:07:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:17 smithi193 ceph-mon[37473]: pgmap v5844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:18.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:17 smithi134 ceph-mon[31966]: pgmap v5844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:17 smithi189 ceph-mon[36790]: pgmap v5844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9580.258104 secs 2024-07-26T14:07:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9580.258104 secs 2024-07-26T14:07:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9580.258104 secs 2024-07-26T14:07:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:19 smithi193 ceph-mon[37473]: pgmap v5845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:07:20.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:19 smithi189 ceph-mon[36790]: pgmap v5845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:07:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:19 smithi134 ceph-mon[31966]: pgmap v5845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:07:21.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:07:21.071+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:07:21.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:07:21.071+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:21.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:07:21.071+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:07:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9599.682765 secs 2024-07-26T14:07:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9599.682765 secs 2024-07-26T14:07:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9599.682765 secs 2024-07-26T14:07:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:22 smithi134 ceph-mon[31966]: pgmap v5846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:22 smithi189 ceph-mon[36790]: pgmap v5846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:22 smithi193 ceph-mon[37473]: pgmap v5846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:24.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9585.258190 secs 2024-07-26T14:07:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9585.258190 secs 2024-07-26T14:07:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9585.258190 secs 2024-07-26T14:07:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:07:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:07:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:07:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:07:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:07:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:07:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:07:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:07:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:07:25.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:07:25.615+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:07:25.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:07:25.616+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:07:25.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:07:25.616+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:25 smithi134 ceph-mon[31966]: pgmap v5847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:07:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:07:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:07:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:25 smithi189 ceph-mon[36790]: pgmap v5847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:07:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:07:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:07:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:25 smithi193 ceph-mon[37473]: pgmap v5847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:07:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:07:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:07:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:26 smithi134 ceph-mon[31966]: pgmap v5848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:07:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:26 smithi134 ceph-mon[31966]: pgmap v5849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:07:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9604.682882 secs 2024-07-26T14:07:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:26 smithi189 ceph-mon[36790]: pgmap v5848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:07:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:26 smithi189 ceph-mon[36790]: pgmap v5849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:07:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9604.682882 secs 2024-07-26T14:07:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:26 smithi193 ceph-mon[37473]: pgmap v5848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:07:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:26 smithi193 ceph-mon[37473]: pgmap v5849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:07:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9604.682882 secs 2024-07-26T14:07:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:28 smithi134 ceph-mon[31966]: pgmap v5850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:07:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9590.258314 secs 2024-07-26T14:07:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:28 smithi189 ceph-mon[36790]: pgmap v5850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:07:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9590.258314 secs 2024-07-26T14:07:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:28 smithi193 ceph-mon[37473]: pgmap v5850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:07:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9590.258314 secs 2024-07-26T14:07:31.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:30 smithi134 ceph-mon[31966]: pgmap v5851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:07:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:30 smithi189 ceph-mon[36790]: pgmap v5851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:07:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:30 smithi193 ceph-mon[37473]: pgmap v5851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:07:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9609.682982 secs 2024-07-26T14:07:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9609.682982 secs 2024-07-26T14:07:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9609.682982 secs 2024-07-26T14:07:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:32 smithi134 ceph-mon[31966]: pgmap v5852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:07:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:32 smithi189 ceph-mon[36790]: pgmap v5852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:07:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:32 smithi193 ceph-mon[37473]: pgmap v5852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:07:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9595.258433 secs 2024-07-26T14:07:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9595.258433 secs 2024-07-26T14:07:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9595.258433 secs 2024-07-26T14:07:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:34 smithi134 ceph-mon[31966]: pgmap v5853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:07:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:34 smithi189 ceph-mon[36790]: pgmap v5853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:07:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:34 smithi193 ceph-mon[37473]: pgmap v5853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:07:36.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:07:36.080+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:07:36.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:07:36.080+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:36.081 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:07:36.080+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:07:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:36 smithi134 ceph-mon[31966]: pgmap v5854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:07:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9614.683061 secs 2024-07-26T14:07:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:36 smithi189 ceph-mon[36790]: pgmap v5854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:07:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9614.683061 secs 2024-07-26T14:07:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:36 smithi193 ceph-mon[37473]: pgmap v5854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:07:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9614.683061 secs 2024-07-26T14:07:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:38 smithi134 ceph-mon[31966]: pgmap v5855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9600.258553 secs 2024-07-26T14:07:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:38 smithi189 ceph-mon[36790]: pgmap v5855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9600.258553 secs 2024-07-26T14:07:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:38 smithi193 ceph-mon[37473]: pgmap v5855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9600.258553 secs 2024-07-26T14:07:40.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:07:40.631+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:07:40.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:07:40.631+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:07:40.633 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:07:40.631+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:41.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:40 smithi134 ceph-mon[31966]: pgmap v5856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:07:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:40 smithi189 ceph-mon[36790]: pgmap v5856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:07:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:40 smithi193 ceph-mon[37473]: pgmap v5856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:07:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9619.683200 secs 2024-07-26T14:07:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9619.683200 secs 2024-07-26T14:07:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9619.683200 secs 2024-07-26T14:07:43.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:42 smithi134 ceph-mon[31966]: pgmap v5857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:42 smithi189 ceph-mon[36790]: pgmap v5857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:42 smithi193 ceph-mon[37473]: pgmap v5857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9605.258651 secs 2024-07-26T14:07:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:43 smithi134 ceph-mon[31966]: pgmap v5858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9605.258651 secs 2024-07-26T14:07:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:43 smithi189 ceph-mon[36790]: pgmap v5858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9605.258651 secs 2024-07-26T14:07:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:43 smithi193 ceph-mon[37473]: pgmap v5858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:46 smithi193 ceph-mon[37473]: pgmap v5859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9624.683335 secs 2024-07-26T14:07:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:46 smithi134 ceph-mon[31966]: pgmap v5859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9624.683335 secs 2024-07-26T14:07:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:46 smithi189 ceph-mon[36790]: pgmap v5859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9624.683335 secs 2024-07-26T14:07:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:48 smithi193 ceph-mon[37473]: pgmap v5860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9610.258766 secs 2024-07-26T14:07:49.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:48 smithi134 ceph-mon[31966]: pgmap v5860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9610.258766 secs 2024-07-26T14:07:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:48 smithi189 ceph-mon[36790]: pgmap v5860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9610.258766 secs 2024-07-26T14:07:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:07:51.095+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:07:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:07:51.096+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:07:51.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:07:51.096+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:51.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:50 smithi134 ceph-mon[31966]: pgmap v5861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:07:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:50 smithi189 ceph-mon[36790]: pgmap v5861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:07:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:50 smithi193 ceph-mon[37473]: pgmap v5861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:07:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9629.683450 secs 2024-07-26T14:07:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9629.683450 secs 2024-07-26T14:07:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9629.683450 secs 2024-07-26T14:07:53.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:52 smithi134 ceph-mon[31966]: pgmap v5862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:52 smithi189 ceph-mon[36790]: pgmap v5862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:52 smithi193 ceph-mon[37473]: pgmap v5862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9615.258923 secs 2024-07-26T14:07:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9615.258923 secs 2024-07-26T14:07:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9615.258923 secs 2024-07-26T14:07:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:54 smithi134 ceph-mon[31966]: pgmap v5863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:54 smithi189 ceph-mon[36790]: pgmap v5863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:54 smithi193 ceph-mon[37473]: pgmap v5863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:55.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:07:55.647+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:07:55.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:07:55.647+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:07:55.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:07:55.647+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:07:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:56 smithi134 ceph-mon[31966]: pgmap v5864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9634.683513 secs 2024-07-26T14:07:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:56 smithi189 ceph-mon[36790]: pgmap v5864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9634.683513 secs 2024-07-26T14:07:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:56 smithi193 ceph-mon[37473]: pgmap v5864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:07:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9634.683513 secs 2024-07-26T14:07:59.114 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:58 smithi134 ceph-mon[31966]: pgmap v5865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:59.114 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:07:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9620.259055 secs 2024-07-26T14:07:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:58 smithi189 ceph-mon[36790]: pgmap v5865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:07:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9620.259055 secs 2024-07-26T14:07:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:58 smithi193 ceph-mon[37473]: pgmap v5865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:07:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:07:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9620.259055 secs 2024-07-26T14:08:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:00 smithi134 ceph-mon[31966]: pgmap v5866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:08:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:00 smithi189 ceph-mon[36790]: pgmap v5866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:08:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:00 smithi193 ceph-mon[37473]: pgmap v5866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:08:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9639.683626 secs 2024-07-26T14:08:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9639.683626 secs 2024-07-26T14:08:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9639.683626 secs 2024-07-26T14:08:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:02 smithi134 ceph-mon[31966]: pgmap v5867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:02 smithi189 ceph-mon[36790]: pgmap v5867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:02 smithi193 ceph-mon[37473]: pgmap v5867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:04.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9625.259192 secs 2024-07-26T14:08:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9625.259192 secs 2024-07-26T14:08:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9625.259192 secs 2024-07-26T14:08:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:04 smithi134 ceph-mon[31966]: pgmap v5868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:04 smithi189 ceph-mon[36790]: pgmap v5868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:04 smithi193 ceph-mon[37473]: pgmap v5868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:08:06.111+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:08:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:08:06.111+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:08:06.111+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:08:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:06 smithi134 ceph-mon[31966]: pgmap v5869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9644.683723 secs 2024-07-26T14:08:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:06 smithi189 ceph-mon[36790]: pgmap v5869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9644.683723 secs 2024-07-26T14:08:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:06 smithi193 ceph-mon[37473]: pgmap v5869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9644.683723 secs 2024-07-26T14:08:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:08 smithi134 ceph-mon[31966]: pgmap v5870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9630.259350 secs 2024-07-26T14:08:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:08 smithi189 ceph-mon[36790]: pgmap v5870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9630.259350 secs 2024-07-26T14:08:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:08 smithi193 ceph-mon[37473]: pgmap v5870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9630.259350 secs 2024-07-26T14:08:10.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:08:10.663+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:08:10.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:08:10.663+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:08:10.664 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:08:10.663+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:10.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:10 smithi134 ceph-mon[31966]: pgmap v5871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:08:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:10 smithi189 ceph-mon[36790]: pgmap v5871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:08:11.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:10 smithi193 ceph-mon[37473]: pgmap v5871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:08:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9649.683806 secs 2024-07-26T14:08:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9649.683806 secs 2024-07-26T14:08:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9649.683806 secs 2024-07-26T14:08:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:12 smithi134 ceph-mon[31966]: pgmap v5872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:12 smithi189 ceph-mon[36790]: pgmap v5872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:12 smithi193 ceph-mon[37473]: pgmap v5872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9635.259470 secs 2024-07-26T14:08:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:14 smithi189 ceph-mon[36790]: pgmap v5873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:14.984 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9635.259470 secs 2024-07-26T14:08:14.984 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:14 smithi134 ceph-mon[31966]: pgmap v5873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9635.259470 secs 2024-07-26T14:08:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:14 smithi193 ceph-mon[37473]: pgmap v5873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:17.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:16 smithi134 ceph-mon[31966]: pgmap v5874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9654.683946 secs 2024-07-26T14:08:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:16 smithi189 ceph-mon[36790]: pgmap v5874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9654.683946 secs 2024-07-26T14:08:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:16 smithi193 ceph-mon[37473]: pgmap v5874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9654.683946 secs 2024-07-26T14:08:19.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:18 smithi134 ceph-mon[31966]: pgmap v5875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9640.259571 secs 2024-07-26T14:08:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:18 smithi189 ceph-mon[36790]: pgmap v5875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9640.259571 secs 2024-07-26T14:08:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:18 smithi193 ceph-mon[37473]: pgmap v5875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9640.259571 secs 2024-07-26T14:08:21.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:20 smithi193 ceph-mon[37473]: pgmap v5876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:08:21.128 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:08:21.127+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:08:21.128 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:08:21.127+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:08:21.128 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:08:21.127+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:21.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:20 smithi134 ceph-mon[31966]: pgmap v5876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:08:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:20 smithi189 ceph-mon[36790]: pgmap v5876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:08:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9659.684063 secs 2024-07-26T14:08:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9659.684063 secs 2024-07-26T14:08:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9659.684063 secs 2024-07-26T14:08:23.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:22 smithi134 ceph-mon[31966]: pgmap v5877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:22 smithi189 ceph-mon[36790]: pgmap v5877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:22 smithi193 ceph-mon[37473]: pgmap v5877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:24.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9645.259723 secs 2024-07-26T14:08:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9645.259723 secs 2024-07-26T14:08:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9645.259723 secs 2024-07-26T14:08:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:24 smithi134 ceph-mon[31966]: pgmap v5878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:24 smithi189 ceph-mon[36790]: pgmap v5878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:24 smithi193 ceph-mon[37473]: pgmap v5878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:25.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:08:25.678+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:08:25.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:08:25.678+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:08:25.680 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:08:25.678+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:27.089 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:26 smithi134 ceph-mon[31966]: pgmap v5879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:27.089 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:08:27.089 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9664.684174 secs 2024-07-26T14:08:27.090 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:27.090 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:27.090 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:27.090 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:27.118 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:26 smithi189 ceph-mon[36790]: pgmap v5879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:27.118 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:08:27.118 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9664.684174 secs 2024-07-26T14:08:27.118 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:27.118 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:27.119 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:27.119 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:27.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:26 smithi193 ceph-mon[37473]: pgmap v5879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:27.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:08:27.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9664.684174 secs 2024-07-26T14:08:27.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:27.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:27.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:27.157 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:28 smithi193 ceph-mon[37473]: pgmap v5880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:28 smithi134 ceph-mon[31966]: pgmap v5880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:28 smithi189 ceph-mon[36790]: pgmap v5880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9650.259852 secs 2024-07-26T14:08:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9650.259852 secs 2024-07-26T14:08:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9650.259852 secs 2024-07-26T14:08:31.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:30 smithi134 ceph-mon[31966]: pgmap v5881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:08:31.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:31.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:30 smithi189 ceph-mon[36790]: pgmap v5881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:08:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:30 smithi193 ceph-mon[37473]: pgmap v5881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:08:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9669.684327 secs 2024-07-26T14:08:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:08:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:08:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9669.684327 secs 2024-07-26T14:08:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:08:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:08:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9669.684327 secs 2024-07-26T14:08:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:08:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:08:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:08:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:32 smithi134 ceph-mon[31966]: pgmap v5882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:08:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:32 smithi134 ceph-mon[31966]: pgmap v5883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:08:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:32 smithi189 ceph-mon[36790]: pgmap v5882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:08:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:32 smithi189 ceph-mon[36790]: pgmap v5883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:08:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:32 smithi193 ceph-mon[37473]: pgmap v5882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:08:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:32 smithi193 ceph-mon[37473]: pgmap v5883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:08:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9655.260003 secs 2024-07-26T14:08:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9655.260003 secs 2024-07-26T14:08:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9655.260003 secs 2024-07-26T14:08:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:34 smithi134 ceph-mon[31966]: pgmap v5884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:08:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:34 smithi189 ceph-mon[36790]: pgmap v5884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:08:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:34 smithi193 ceph-mon[37473]: pgmap v5884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:08:36.133 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:08:36.132+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:08:36.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:08:36.132+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:36.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:08:36.132+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:08:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:36 smithi134 ceph-mon[31966]: pgmap v5885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:08:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9674.684421 secs 2024-07-26T14:08:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:36 smithi189 ceph-mon[36790]: pgmap v5885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:08:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9674.684421 secs 2024-07-26T14:08:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:36 smithi193 ceph-mon[37473]: pgmap v5885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:08:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9674.684421 secs 2024-07-26T14:08:39.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:38 smithi134 ceph-mon[31966]: pgmap v5886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T14:08:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9660.260120 secs 2024-07-26T14:08:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:38 smithi189 ceph-mon[36790]: pgmap v5886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T14:08:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9660.260120 secs 2024-07-26T14:08:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:38 smithi193 ceph-mon[37473]: pgmap v5886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T14:08:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9660.260120 secs 2024-07-26T14:08:40.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:08:40.694+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:08:40.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:08:40.694+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:08:40.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:08:40.694+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:40.974 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:40 smithi134 ceph-mon[31966]: pgmap v5887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:08:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:40 smithi189 ceph-mon[36790]: pgmap v5887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:08:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:40 smithi193 ceph-mon[37473]: pgmap v5887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:08:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9679.684517 secs 2024-07-26T14:08:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9679.684517 secs 2024-07-26T14:08:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9679.684517 secs 2024-07-26T14:08:43.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:42 smithi134 ceph-mon[31966]: pgmap v5888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:08:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:42 smithi189 ceph-mon[36790]: pgmap v5888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:08:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:42 smithi193 ceph-mon[37473]: pgmap v5888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:08:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9665.260197 secs 2024-07-26T14:08:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9665.260197 secs 2024-07-26T14:08:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9665.260197 secs 2024-07-26T14:08:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:44 smithi134 ceph-mon[31966]: pgmap v5889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:44 smithi189 ceph-mon[36790]: pgmap v5889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:44 smithi193 ceph-mon[37473]: pgmap v5889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:46 smithi134 ceph-mon[31966]: pgmap v5890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9684.684633 secs 2024-07-26T14:08:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:46 smithi189 ceph-mon[36790]: pgmap v5890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9684.684633 secs 2024-07-26T14:08:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:46 smithi193 ceph-mon[37473]: pgmap v5890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9684.684633 secs 2024-07-26T14:08:49.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:48 smithi134 ceph-mon[31966]: pgmap v5891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9670.260317 secs 2024-07-26T14:08:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:48 smithi189 ceph-mon[36790]: pgmap v5891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9670.260317 secs 2024-07-26T14:08:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:48 smithi193 ceph-mon[37473]: pgmap v5891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9670.260317 secs 2024-07-26T14:08:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:08:51.142+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:08:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:08:51.142+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:51.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:08:51.142+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:08:51.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:50 smithi134 ceph-mon[31966]: pgmap v5892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:08:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:50 smithi189 ceph-mon[36790]: pgmap v5892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:08:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:50 smithi193 ceph-mon[37473]: pgmap v5892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:08:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9689.684759 secs 2024-07-26T14:08:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9689.684759 secs 2024-07-26T14:08:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9689.684759 secs 2024-07-26T14:08:53.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:52 smithi134 ceph-mon[31966]: pgmap v5893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:52 smithi189 ceph-mon[36790]: pgmap v5893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:52 smithi193 ceph-mon[37473]: pgmap v5893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9675.260382 secs 2024-07-26T14:08:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:54 smithi193 ceph-mon[37473]: pgmap v5894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9675.260382 secs 2024-07-26T14:08:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:54 smithi134 ceph-mon[31966]: pgmap v5894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9675.260382 secs 2024-07-26T14:08:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:54 smithi189 ceph-mon[36790]: pgmap v5894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:55.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:08:55.709+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:08:55.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:08:55.709+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:08:55.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:08:55.709+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:08:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:56 smithi193 ceph-mon[37473]: pgmap v5895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9694.684874 secs 2024-07-26T14:08:57.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:56 smithi134 ceph-mon[31966]: pgmap v5895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:57.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9694.684874 secs 2024-07-26T14:08:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:56 smithi189 ceph-mon[36790]: pgmap v5895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:08:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9694.684874 secs 2024-07-26T14:08:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:58 smithi193 ceph-mon[37473]: pgmap v5896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:58 smithi134 ceph-mon[31966]: pgmap v5896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:08:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:58 smithi189 ceph-mon[36790]: pgmap v5896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:09:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:08:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9680.260502 secs 2024-07-26T14:09:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:08:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9680.260502 secs 2024-07-26T14:09:00.219 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:08:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9680.260502 secs 2024-07-26T14:09:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:00 smithi193 ceph-mon[37473]: pgmap v5897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:09:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:00 smithi134 ceph-mon[31966]: pgmap v5897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:09:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:00 smithi189 ceph-mon[36790]: pgmap v5897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:09:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9699.685018 secs 2024-07-26T14:09:02.088 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9699.685018 secs 2024-07-26T14:09:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9699.685018 secs 2024-07-26T14:09:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:02 smithi193 ceph-mon[37473]: pgmap v5898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:02 smithi134 ceph-mon[31966]: pgmap v5898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:02 smithi189 ceph-mon[36790]: pgmap v5898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9685.260622 secs 2024-07-26T14:09:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:04 smithi193 ceph-mon[37473]: pgmap v5899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:09:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9685.260622 secs 2024-07-26T14:09:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:04 smithi134 ceph-mon[31966]: pgmap v5899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:09:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9685.260622 secs 2024-07-26T14:09:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:04 smithi189 ceph-mon[36790]: pgmap v5899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:09:06.147 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:09:06.146+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:09:06.147 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:09:06.146+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:06.147 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:09:06.146+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:09:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:06 smithi193 ceph-mon[37473]: pgmap v5900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:09:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9704.685141 secs 2024-07-26T14:09:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:06 smithi134 ceph-mon[31966]: pgmap v5900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:09:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9704.685141 secs 2024-07-26T14:09:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:06 smithi189 ceph-mon[36790]: pgmap v5900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:09:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9704.685141 secs 2024-07-26T14:09:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:08 smithi193 ceph-mon[37473]: pgmap v5901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:09:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:08 smithi134 ceph-mon[31966]: pgmap v5901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:09:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:08 smithi189 ceph-mon[36790]: pgmap v5901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:09:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9690.260766 secs 2024-07-26T14:09:10.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9690.260766 secs 2024-07-26T14:09:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9690.260766 secs 2024-07-26T14:09:10.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:09:10.724+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:09:10.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:09:10.725+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:09:10.726 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:09:10.725+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:10 smithi193 ceph-mon[37473]: pgmap v5902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:09:11.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:10 smithi134 ceph-mon[31966]: pgmap v5902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:09:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:10 smithi189 ceph-mon[36790]: pgmap v5902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:09:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9709.685284 secs 2024-07-26T14:09:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9709.685284 secs 2024-07-26T14:09:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9709.685284 secs 2024-07-26T14:09:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:12 smithi134 ceph-mon[31966]: pgmap v5903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:09:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:12 smithi189 ceph-mon[36790]: pgmap v5903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:09:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:12 smithi193 ceph-mon[37473]: pgmap v5903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:09:14.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9695.260900 secs 2024-07-26T14:09:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9695.260900 secs 2024-07-26T14:09:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9695.260900 secs 2024-07-26T14:09:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:14 smithi134 ceph-mon[31966]: pgmap v5904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:09:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:14 smithi189 ceph-mon[36790]: pgmap v5904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:09:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:14 smithi193 ceph-mon[37473]: pgmap v5904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:09:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:16 smithi134 ceph-mon[31966]: pgmap v5905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9714.685376 secs 2024-07-26T14:09:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:16 smithi189 ceph-mon[36790]: pgmap v5905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9714.685376 secs 2024-07-26T14:09:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:16 smithi193 ceph-mon[37473]: pgmap v5905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9714.685376 secs 2024-07-26T14:09:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:18 smithi134 ceph-mon[31966]: pgmap v5906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:09:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9700.261044 secs 2024-07-26T14:09:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:18 smithi189 ceph-mon[36790]: pgmap v5906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:09:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9700.261044 secs 2024-07-26T14:09:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:18 smithi193 ceph-mon[37473]: pgmap v5906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:09:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9700.261044 secs 2024-07-26T14:09:21.075 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:20 smithi193 ceph-mon[37473]: pgmap v5907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:09:21.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:09:21.162+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:09:21.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:09:21.162+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:09:21.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:09:21.162+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:21.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:20 smithi134 ceph-mon[31966]: pgmap v5907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:09:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:20 smithi189 ceph-mon[36790]: pgmap v5907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:09:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9719.685490 secs 2024-07-26T14:09:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9719.685490 secs 2024-07-26T14:09:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9719.685490 secs 2024-07-26T14:09:23.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:22 smithi134 ceph-mon[31966]: pgmap v5908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:22 smithi189 ceph-mon[36790]: pgmap v5908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:22 smithi193 ceph-mon[37473]: pgmap v5908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:24.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:09:24.018+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T14:09:24.019 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:09:24.018+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T14:09:24.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:09:24.018+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T14:09:24.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:09:24.018+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T14:09:24.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9705.261186 secs 2024-07-26T14:09:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9705.261186 secs 2024-07-26T14:09:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9705.261186 secs 2024-07-26T14:09:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:24 smithi134 ceph-mon[31966]: pgmap v5909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:09:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:24 smithi189 ceph-mon[36790]: pgmap v5909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:09:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:24 smithi193 ceph-mon[37473]: pgmap v5909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:09:25.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:09:25.740+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:09:25.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:09:25.740+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:09:25.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:09:25.740+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:26 smithi134 ceph-mon[31966]: pgmap v5910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9724.685601 secs 2024-07-26T14:09:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:26 smithi189 ceph-mon[36790]: pgmap v5910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9724.685601 secs 2024-07-26T14:09:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:26 smithi193 ceph-mon[37473]: pgmap v5910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9724.685601 secs 2024-07-26T14:09:29.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:28 smithi134 ceph-mon[31966]: pgmap v5911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:09:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9710.261293 secs 2024-07-26T14:09:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:28 smithi189 ceph-mon[36790]: pgmap v5911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:09:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9710.261293 secs 2024-07-26T14:09:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:28 smithi193 ceph-mon[37473]: pgmap v5911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:09:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9710.261293 secs 2024-07-26T14:09:31.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:30 smithi134 ceph-mon[31966]: pgmap v5912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:09:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:30 smithi189 ceph-mon[36790]: pgmap v5912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:09:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:30 smithi193 ceph-mon[37473]: pgmap v5912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:09:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9729.685720 secs 2024-07-26T14:09:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:09:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9729.685720 secs 2024-07-26T14:09:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:09:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9729.685720 secs 2024-07-26T14:09:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:09:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:32 smithi134 ceph-mon[31966]: pgmap v5913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:09:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:09:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:09:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:32 smithi189 ceph-mon[36790]: pgmap v5913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:09:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:09:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:09:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:32 smithi193 ceph-mon[37473]: pgmap v5913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:09:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:09:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:09:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:33 smithi134 ceph-mon[31966]: pgmap v5914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:09:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:33 smithi134 ceph-mon[31966]: pgmap v5915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:09:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9715.261423 secs 2024-07-26T14:09:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:33 smithi189 ceph-mon[36790]: pgmap v5914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:09:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:33 smithi189 ceph-mon[36790]: pgmap v5915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:09:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9715.261423 secs 2024-07-26T14:09:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:33 smithi193 ceph-mon[37473]: pgmap v5914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:09:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:33 smithi193 ceph-mon[37473]: pgmap v5915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:09:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9715.261423 secs 2024-07-26T14:09:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:09:34.543+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:09:34.544 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:09:34.543+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:09:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:09:34.544+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:09:34.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:09:34.544+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:09:36.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:35 smithi134 ceph-mon[31966]: pgmap v5916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:09:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:09:36.177+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:09:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:09:36.177+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:36.178 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:09:36.177+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:09:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:35 smithi189 ceph-mon[36790]: pgmap v5916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:09:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:35 smithi193 ceph-mon[37473]: pgmap v5916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:09:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9734.685863 secs 2024-07-26T14:09:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9734.685863 secs 2024-07-26T14:09:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9734.685863 secs 2024-07-26T14:09:38.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:37 smithi134 ceph-mon[31966]: pgmap v5917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:09:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:37 smithi189 ceph-mon[36790]: pgmap v5917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:09:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:37 smithi193 ceph-mon[37473]: pgmap v5917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:09:39.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9720.261556 secs 2024-07-26T14:09:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9720.261556 secs 2024-07-26T14:09:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9720.261556 secs 2024-07-26T14:09:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:39 smithi134 ceph-mon[31966]: pgmap v5918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:09:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:39 smithi189 ceph-mon[36790]: pgmap v5918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:09:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:39 smithi193 ceph-mon[37473]: pgmap v5918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:09:40.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:09:40.753+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:09:40.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:09:40.753+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:09:40.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:09:40.753+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:41 smithi134 ceph-mon[31966]: pgmap v5919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:09:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9739.685962 secs 2024-07-26T14:09:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:41 smithi189 ceph-mon[36790]: pgmap v5919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:09:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9739.685962 secs 2024-07-26T14:09:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:41 smithi193 ceph-mon[37473]: pgmap v5919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:09:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9739.685962 secs 2024-07-26T14:09:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:43 smithi189 ceph-mon[36790]: pgmap v5920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:09:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9725.261691 secs 2024-07-26T14:09:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:43 smithi193 ceph-mon[37473]: pgmap v5920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:09:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9725.261691 secs 2024-07-26T14:09:44.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:43 smithi134 ceph-mon[31966]: pgmap v5920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:09:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9725.261691 secs 2024-07-26T14:09:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:44 smithi189 ceph-mon[36790]: pgmap v5921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:09:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:44 smithi193 ceph-mon[37473]: pgmap v5921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:09:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:44 smithi134 ceph-mon[31966]: pgmap v5921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:09:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9744.686087 secs 2024-07-26T14:09:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9744.686087 secs 2024-07-26T14:09:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9744.686087 secs 2024-07-26T14:09:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:47 smithi134 ceph-mon[31966]: pgmap v5922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:09:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:47 smithi189 ceph-mon[36790]: pgmap v5922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:09:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:47 smithi193 ceph-mon[37473]: pgmap v5922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:09:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9730.261766 secs 2024-07-26T14:09:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:49 smithi134 ceph-mon[31966]: pgmap v5923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:09:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9730.261766 secs 2024-07-26T14:09:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:49 smithi189 ceph-mon[36790]: pgmap v5923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:09:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9730.261766 secs 2024-07-26T14:09:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:49 smithi193 ceph-mon[37473]: pgmap v5923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:09:51.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:09:51.192+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:09:51.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:09:51.193+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:09:51.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:09:51.193+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:52.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:51 smithi193 ceph-mon[37473]: pgmap v5924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:09:52.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9749.686203 secs 2024-07-26T14:09:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:51 smithi134 ceph-mon[31966]: pgmap v5924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:09:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9749.686203 secs 2024-07-26T14:09:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:51 smithi189 ceph-mon[36790]: pgmap v5924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:09:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9749.686203 secs 2024-07-26T14:09:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:53 smithi134 ceph-mon[31966]: pgmap v5925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:09:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9735.261918 secs 2024-07-26T14:09:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:53 smithi189 ceph-mon[36790]: pgmap v5925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:09:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9735.261918 secs 2024-07-26T14:09:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:53 smithi193 ceph-mon[37473]: pgmap v5925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:09:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9735.261918 secs 2024-07-26T14:09:55.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:09:55.756+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:09:55.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:09:55.756+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:09:55.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:09:55.756+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:09:56.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:55 smithi134 ceph-mon[31966]: pgmap v5926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:09:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:55 smithi189 ceph-mon[36790]: pgmap v5926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:09:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:55 smithi193 ceph-mon[37473]: pgmap v5926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:09:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9754.686339 secs 2024-07-26T14:09:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9754.686339 secs 2024-07-26T14:09:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9754.686339 secs 2024-07-26T14:09:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:57 smithi134 ceph-mon[31966]: pgmap v5927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:57 smithi189 ceph-mon[36790]: pgmap v5927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:57 smithi193 ceph-mon[37473]: pgmap v5927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:09:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9740.262037 secs 2024-07-26T14:09:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9740.262037 secs 2024-07-26T14:09:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9740.262037 secs 2024-07-26T14:10:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:09:59 smithi134 ceph-mon[31966]: pgmap v5928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:09:59 smithi189 ceph-mon[36790]: pgmap v5928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:09:59 smithi193 ceph-mon[37473]: pgmap v5928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:10:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:10:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:10:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:01 smithi134 ceph-mon[31966]: pgmap v5929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9759.686461 secs 2024-07-26T14:10:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:01 smithi189 ceph-mon[36790]: pgmap v5929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9759.686461 secs 2024-07-26T14:10:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:01 smithi193 ceph-mon[37473]: pgmap v5929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9759.686461 secs 2024-07-26T14:10:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:03 smithi134 ceph-mon[31966]: pgmap v5930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9745.262144 secs 2024-07-26T14:10:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:03 smithi189 ceph-mon[36790]: pgmap v5930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9745.262144 secs 2024-07-26T14:10:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:03 smithi193 ceph-mon[37473]: pgmap v5930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9745.262144 secs 2024-07-26T14:10:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:05 smithi134 ceph-mon[31966]: pgmap v5931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:05 smithi189 ceph-mon[36790]: pgmap v5931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:10:06.208+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:10:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:10:06.208+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:06.209 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:10:06.208+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:10:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:05 smithi193 ceph-mon[37473]: pgmap v5931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9764.686578 secs 2024-07-26T14:10:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9764.686578 secs 2024-07-26T14:10:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9764.686578 secs 2024-07-26T14:10:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:07 smithi134 ceph-mon[31966]: pgmap v5932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:07 smithi189 ceph-mon[36790]: pgmap v5932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:07 smithi193 ceph-mon[37473]: pgmap v5932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9750.262276 secs 2024-07-26T14:10:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9750.262276 secs 2024-07-26T14:10:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9750.262276 secs 2024-07-26T14:10:10.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:09 smithi134 ceph-mon[31966]: pgmap v5933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:09 smithi189 ceph-mon[36790]: pgmap v5933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:09 smithi193 ceph-mon[37473]: pgmap v5933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:10.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:10:10.762+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:10:10.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:10:10.762+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:10:10.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:10:10.762+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:11 smithi134 ceph-mon[31966]: pgmap v5934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9769.686688 secs 2024-07-26T14:10:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:11 smithi189 ceph-mon[36790]: pgmap v5934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9769.686688 secs 2024-07-26T14:10:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:11 smithi193 ceph-mon[37473]: pgmap v5934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9769.686688 secs 2024-07-26T14:10:14.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:13 smithi134 ceph-mon[31966]: pgmap v5935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9755.262400 secs 2024-07-26T14:10:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:13 smithi189 ceph-mon[36790]: pgmap v5935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9755.262400 secs 2024-07-26T14:10:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:13 smithi193 ceph-mon[37473]: pgmap v5935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9755.262400 secs 2024-07-26T14:10:16.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:15 smithi134 ceph-mon[31966]: pgmap v5936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:15 smithi189 ceph-mon[36790]: pgmap v5936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:15 smithi193 ceph-mon[37473]: pgmap v5936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9774.686779 secs 2024-07-26T14:10:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9774.686779 secs 2024-07-26T14:10:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9774.686779 secs 2024-07-26T14:10:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:17 smithi134 ceph-mon[31966]: pgmap v5937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:17 smithi189 ceph-mon[36790]: pgmap v5937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:17 smithi193 ceph-mon[37473]: pgmap v5937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:19.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9760.262537 secs 2024-07-26T14:10:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9760.262537 secs 2024-07-26T14:10:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9760.262537 secs 2024-07-26T14:10:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:19 smithi189 ceph-mon[36790]: pgmap v5938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:19 smithi193 ceph-mon[37473]: pgmap v5938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:19 smithi134 ceph-mon[31966]: pgmap v5938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:10:21.213+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:10:21.214 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:10:21.213+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:21.215 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:10:21.213+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:10:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:20 smithi193 ceph-mon[37473]: pgmap v5939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:21.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:20 smithi134 ceph-mon[31966]: pgmap v5939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:20 smithi189 ceph-mon[36790]: pgmap v5939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9779.686908 secs 2024-07-26T14:10:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9779.686908 secs 2024-07-26T14:10:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9779.686908 secs 2024-07-26T14:10:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:22 smithi193 ceph-mon[37473]: pgmap v5940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:23.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:22 smithi134 ceph-mon[31966]: pgmap v5940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:22 smithi189 ceph-mon[36790]: pgmap v5940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9765.262669 secs 2024-07-26T14:10:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9765.262669 secs 2024-07-26T14:10:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9765.262669 secs 2024-07-26T14:10:25.249 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:24 smithi189 ceph-mon[36790]: pgmap v5941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:24 smithi193 ceph-mon[37473]: pgmap v5941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:24 smithi134 ceph-mon[31966]: pgmap v5941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:25.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:10:25.772+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:10:25.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:10:25.772+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:10:25.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:10:25.772+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9784.686987 secs 2024-07-26T14:10:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9784.686987 secs 2024-07-26T14:10:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9784.686987 secs 2024-07-26T14:10:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:27 smithi134 ceph-mon[31966]: pgmap v5942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:27 smithi189 ceph-mon[36790]: pgmap v5942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:27 smithi193 ceph-mon[37473]: pgmap v5942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:29.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9770.262827 secs 2024-07-26T14:10:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9770.262827 secs 2024-07-26T14:10:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9770.262827 secs 2024-07-26T14:10:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:29 smithi134 ceph-mon[31966]: pgmap v5943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:29 smithi189 ceph-mon[36790]: pgmap v5943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:29 smithi193 ceph-mon[37473]: pgmap v5943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:31 smithi134 ceph-mon[31966]: pgmap v5944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9789.687108 secs 2024-07-26T14:10:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:31 smithi189 ceph-mon[36790]: pgmap v5944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9789.687108 secs 2024-07-26T14:10:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:31 smithi193 ceph-mon[37473]: pgmap v5944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9789.687108 secs 2024-07-26T14:10:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:33 smithi134 ceph-mon[31966]: pgmap v5945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:10:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9775.262967 secs 2024-07-26T14:10:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:33 smithi189 ceph-mon[36790]: pgmap v5945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:10:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9775.262967 secs 2024-07-26T14:10:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:33 smithi193 ceph-mon[37473]: pgmap v5945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:10:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9775.262967 secs 2024-07-26T14:10:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:10:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:10:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:34 smithi134 ceph-mon[31966]: pgmap v5946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:10:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:34 smithi134 ceph-mon[31966]: pgmap v5947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:10:35.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:10:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:10:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:10:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:34 smithi189 ceph-mon[36790]: pgmap v5946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:10:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:34 smithi189 ceph-mon[36790]: pgmap v5947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:10:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:10:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:10:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:10:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:34 smithi193 ceph-mon[37473]: pgmap v5946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:10:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:34 smithi193 ceph-mon[37473]: pgmap v5947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:10:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:10:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:10:36.228+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:10:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:10:36.229+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:10:36.230 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:10:36.229+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:35 smithi193 ceph-mon[37473]: pgmap v5948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:10:36.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:35 smithi134 ceph-mon[31966]: pgmap v5948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:10:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:35 smithi189 ceph-mon[36790]: pgmap v5948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:10:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9794.687226 secs 2024-07-26T14:10:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9794.687226 secs 2024-07-26T14:10:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9794.687226 secs 2024-07-26T14:10:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:38 smithi193 ceph-mon[37473]: pgmap v5949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:10:38.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:38 smithi134 ceph-mon[31966]: pgmap v5949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:10:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:38 smithi189 ceph-mon[36790]: pgmap v5949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:10:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9780.263103 secs 2024-07-26T14:10:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9780.263103 secs 2024-07-26T14:10:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9780.263103 secs 2024-07-26T14:10:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:40 smithi193 ceph-mon[37473]: pgmap v5950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:10:40.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:40 smithi134 ceph-mon[31966]: pgmap v5950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:10:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:40 smithi189 ceph-mon[36790]: pgmap v5950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:10:40.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:10:40.783+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:10:40.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:10:40.783+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:10:40.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:10:40.783+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9799.687341 secs 2024-07-26T14:10:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9799.687341 secs 2024-07-26T14:10:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9799.687341 secs 2024-07-26T14:10:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:42 smithi134 ceph-mon[31966]: pgmap v5951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:10:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:42 smithi189 ceph-mon[36790]: pgmap v5951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:10:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:42 smithi193 ceph-mon[37473]: pgmap v5951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:10:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9785.263233 secs 2024-07-26T14:10:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9785.263233 secs 2024-07-26T14:10:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9785.263233 secs 2024-07-26T14:10:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:44 smithi134 ceph-mon[31966]: pgmap v5952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:10:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:44 smithi189 ceph-mon[36790]: pgmap v5952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:10:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:44 smithi193 ceph-mon[37473]: pgmap v5952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:10:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:45 smithi193 ceph-mon[37473]: pgmap v5953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:45 smithi134 ceph-mon[31966]: pgmap v5953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:46.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:45 smithi189 ceph-mon[36790]: pgmap v5953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9804.687404 secs 2024-07-26T14:10:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9804.687404 secs 2024-07-26T14:10:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9804.687404 secs 2024-07-26T14:10:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:48 smithi193 ceph-mon[37473]: pgmap v5954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:48 smithi134 ceph-mon[31966]: pgmap v5954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:48 smithi189 ceph-mon[36790]: pgmap v5954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9790.263363 secs 2024-07-26T14:10:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9790.263363 secs 2024-07-26T14:10:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9790.263363 secs 2024-07-26T14:10:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:50 smithi193 ceph-mon[37473]: pgmap v5955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:50.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:50 smithi189 ceph-mon[36790]: pgmap v5955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:50 smithi134 ceph-mon[31966]: pgmap v5955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:10:51.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:10:51.244+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:10:51.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:10:51.244+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:51.246 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:10:51.244+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:10:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9809.687525 secs 2024-07-26T14:10:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9809.687525 secs 2024-07-26T14:10:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9809.687525 secs 2024-07-26T14:10:53.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:53 smithi134 ceph-mon[31966]: pgmap v5956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:53 smithi189 ceph-mon[36790]: pgmap v5956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:53 smithi193 ceph-mon[37473]: pgmap v5956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9795.263505 secs 2024-07-26T14:10:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9795.263505 secs 2024-07-26T14:10:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9795.263505 secs 2024-07-26T14:10:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:55 smithi134 ceph-mon[31966]: pgmap v5957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:55 smithi189 ceph-mon[36790]: pgmap v5957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:55 smithi193 ceph-mon[37473]: pgmap v5957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:55.789 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:10:55.788+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:10:55.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:10:55.788+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:10:55.790 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:10:55.788+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:10:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9814.687647 secs 2024-07-26T14:10:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9814.687647 secs 2024-07-26T14:10:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9814.687647 secs 2024-07-26T14:10:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:57 smithi134 ceph-mon[31966]: pgmap v5958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:57 smithi189 ceph-mon[36790]: pgmap v5958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:57 smithi193 ceph-mon[37473]: pgmap v5958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:10:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:59 smithi134 ceph-mon[31966]: pgmap v5959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:10:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9800.263641 secs 2024-07-26T14:10:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:59 smithi189 ceph-mon[36790]: pgmap v5959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:10:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9800.263641 secs 2024-07-26T14:10:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:59 smithi193 ceph-mon[37473]: pgmap v5959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:10:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:10:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9800.263641 secs 2024-07-26T14:11:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:01 smithi134 ceph-mon[31966]: pgmap v5960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:11:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9819.687769 secs 2024-07-26T14:11:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:01 smithi189 ceph-mon[36790]: pgmap v5960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:11:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9819.687769 secs 2024-07-26T14:11:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:01 smithi193 ceph-mon[37473]: pgmap v5960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:11:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9819.687769 secs 2024-07-26T14:11:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:03 smithi189 ceph-mon[36790]: pgmap v5961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:03 smithi193 ceph-mon[37473]: pgmap v5961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:03.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:03 smithi134 ceph-mon[31966]: pgmap v5961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9805.263764 secs 2024-07-26T14:11:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9805.263764 secs 2024-07-26T14:11:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9805.263764 secs 2024-07-26T14:11:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:05 smithi189 ceph-mon[36790]: pgmap v5962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:05 smithi193 ceph-mon[37473]: pgmap v5962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:05.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:05 smithi134 ceph-mon[31966]: pgmap v5962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:06.251 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:11:06.250+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:11:06.251 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:11:06.250+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:11:06.251 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:11:06.250+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9824.687886 secs 2024-07-26T14:11:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9824.687886 secs 2024-07-26T14:11:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9824.687886 secs 2024-07-26T14:11:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:07 smithi193 ceph-mon[37473]: pgmap v5963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:07 smithi134 ceph-mon[31966]: pgmap v5963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:07 smithi189 ceph-mon[36790]: pgmap v5963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:09 smithi193 ceph-mon[37473]: pgmap v5964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:11:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9810.263897 secs 2024-07-26T14:11:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:09 smithi134 ceph-mon[31966]: pgmap v5964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:11:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9810.263897 secs 2024-07-26T14:11:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:09 smithi189 ceph-mon[36790]: pgmap v5964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:11:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9810.263897 secs 2024-07-26T14:11:10.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:11:10.798+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:11:10.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:11:10.798+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:10.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:11:10.798+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:11:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:11 smithi193 ceph-mon[37473]: pgmap v5965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:11:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9829.688022 secs 2024-07-26T14:11:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:11 smithi134 ceph-mon[31966]: pgmap v5965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:11:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9829.688022 secs 2024-07-26T14:11:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:11 smithi189 ceph-mon[36790]: pgmap v5965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:11:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9829.688022 secs 2024-07-26T14:11:13.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:13 smithi193 ceph-mon[37473]: pgmap v5966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:11:13.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:13 smithi134 ceph-mon[31966]: pgmap v5966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:11:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:13 smithi189 ceph-mon[36790]: pgmap v5966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:11:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9815.264007 secs 2024-07-26T14:11:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9815.264007 secs 2024-07-26T14:11:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9815.264007 secs 2024-07-26T14:11:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:15 smithi193 ceph-mon[37473]: pgmap v5967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:11:15.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:15 smithi134 ceph-mon[31966]: pgmap v5967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:11:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:15 smithi189 ceph-mon[36790]: pgmap v5967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:11:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9834.688157 secs 2024-07-26T14:11:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9834.688157 secs 2024-07-26T14:11:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9834.688157 secs 2024-07-26T14:11:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:17 smithi193 ceph-mon[37473]: pgmap v5968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:11:17.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:17 smithi134 ceph-mon[31966]: pgmap v5968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:11:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:17 smithi189 ceph-mon[36790]: pgmap v5968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:11:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:19 smithi193 ceph-mon[37473]: pgmap v5969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:11:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9820.264116 secs 2024-07-26T14:11:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:19 smithi134 ceph-mon[31966]: pgmap v5969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:11:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9820.264116 secs 2024-07-26T14:11:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:19 smithi189 ceph-mon[36790]: pgmap v5969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:11:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9820.264116 secs 2024-07-26T14:11:21.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:11:21.258+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:11:21.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:11:21.258+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:21.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:11:21.258+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:11:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:21 smithi134 ceph-mon[31966]: pgmap v5970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:11:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9839.688329 secs 2024-07-26T14:11:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:21 smithi193 ceph-mon[37473]: pgmap v5970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:11:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9839.688329 secs 2024-07-26T14:11:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:21 smithi189 ceph-mon[36790]: pgmap v5970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:11:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9839.688329 secs 2024-07-26T14:11:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:23 smithi193 ceph-mon[37473]: pgmap v5971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:23.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:23 smithi134 ceph-mon[31966]: pgmap v5971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:23 smithi189 ceph-mon[36790]: pgmap v5971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9825.264215 secs 2024-07-26T14:11:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9825.264215 secs 2024-07-26T14:11:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9825.264215 secs 2024-07-26T14:11:25.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:11:25.807+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:11:25.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:11:25.807+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:25.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:11:25.807+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:11:25.809 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:25 smithi134 ceph-mon[31966]: pgmap v5972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:25 smithi189 ceph-mon[36790]: pgmap v5972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:25 smithi193 ceph-mon[37473]: pgmap v5972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9844.688394 secs 2024-07-26T14:11:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9844.688394 secs 2024-07-26T14:11:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9844.688394 secs 2024-07-26T14:11:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:27 smithi134 ceph-mon[31966]: pgmap v5973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:27 smithi189 ceph-mon[36790]: pgmap v5973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:27 smithi193 ceph-mon[37473]: pgmap v5973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:29 smithi134 ceph-mon[31966]: pgmap v5974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9830.264283 secs 2024-07-26T14:11:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:29 smithi189 ceph-mon[36790]: pgmap v5974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9830.264283 secs 2024-07-26T14:11:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:29 smithi193 ceph-mon[37473]: pgmap v5974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9830.264283 secs 2024-07-26T14:11:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:31 smithi134 ceph-mon[31966]: pgmap v5975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:11:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9849.688490 secs 2024-07-26T14:11:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:31 smithi189 ceph-mon[36790]: pgmap v5975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:11:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9849.688490 secs 2024-07-26T14:11:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:31 smithi193 ceph-mon[37473]: pgmap v5975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:11:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9849.688490 secs 2024-07-26T14:11:33.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:33 smithi134 ceph-mon[31966]: pgmap v5976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:33 smithi189 ceph-mon[36790]: pgmap v5976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:33 smithi193 ceph-mon[37473]: pgmap v5976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9835.264406 secs 2024-07-26T14:11:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:11:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9835.264406 secs 2024-07-26T14:11:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:11:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9835.264406 secs 2024-07-26T14:11:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:11:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:35 smithi134 ceph-mon[31966]: pgmap v5977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:11:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:11:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:11:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:35 smithi189 ceph-mon[36790]: pgmap v5977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:11:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:11:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:11:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:35 smithi193 ceph-mon[37473]: pgmap v5977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:11:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:11:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:11:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:11:36.273+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:11:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:11:36.273+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:36.274 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:11:36.273+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:11:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:36 smithi134 ceph-mon[31966]: pgmap v5978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:11:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:36 smithi134 ceph-mon[31966]: pgmap v5979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:11:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9854.688553 secs 2024-07-26T14:11:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:36 smithi189 ceph-mon[36790]: pgmap v5978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:11:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:36 smithi189 ceph-mon[36790]: pgmap v5979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:11:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9854.688553 secs 2024-07-26T14:11:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:36 smithi193 ceph-mon[37473]: pgmap v5978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:11:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:36 smithi193 ceph-mon[37473]: pgmap v5979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:11:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9854.688553 secs 2024-07-26T14:11:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:38 smithi134 ceph-mon[31966]: pgmap v5980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:11:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:38 smithi189 ceph-mon[36790]: pgmap v5980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:11:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:38 smithi193 ceph-mon[37473]: pgmap v5980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:11:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9840.264519 secs 2024-07-26T14:11:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9840.264519 secs 2024-07-26T14:11:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9840.264519 secs 2024-07-26T14:11:40.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:11:40.822+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:11:40.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:11:40.823+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:11:40.824 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:11:40.823+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:40 smithi134 ceph-mon[31966]: pgmap v5981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:11:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:40 smithi189 ceph-mon[36790]: pgmap v5981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:11:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:40 smithi193 ceph-mon[37473]: pgmap v5981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:11:41.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9859.688622 secs 2024-07-26T14:11:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9859.688622 secs 2024-07-26T14:11:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9859.688622 secs 2024-07-26T14:11:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:42 smithi134 ceph-mon[31966]: pgmap v5982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:11:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:42 smithi189 ceph-mon[36790]: pgmap v5982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:11:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:42 smithi193 ceph-mon[37473]: pgmap v5982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:11:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:44 smithi134 ceph-mon[31966]: pgmap v5983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:11:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9845.264649 secs 2024-07-26T14:11:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:44 smithi189 ceph-mon[36790]: pgmap v5983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:11:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9845.264649 secs 2024-07-26T14:11:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:44 smithi193 ceph-mon[37473]: pgmap v5983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:11:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9845.264649 secs 2024-07-26T14:11:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:46 smithi134 ceph-mon[31966]: pgmap v5984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:11:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9864.688740 secs 2024-07-26T14:11:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:46 smithi189 ceph-mon[36790]: pgmap v5984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:11:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9864.688740 secs 2024-07-26T14:11:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:46 smithi193 ceph-mon[37473]: pgmap v5984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:11:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9864.688740 secs 2024-07-26T14:11:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:48 smithi134 ceph-mon[31966]: pgmap v5985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:11:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:48 smithi189 ceph-mon[36790]: pgmap v5985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:11:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:48 smithi193 ceph-mon[37473]: pgmap v5985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:11:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9850.264709 secs 2024-07-26T14:11:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9850.264709 secs 2024-07-26T14:11:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9850.264709 secs 2024-07-26T14:11:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:50 smithi134 ceph-mon[31966]: pgmap v5986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:50 smithi189 ceph-mon[36790]: pgmap v5986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:50 smithi193 ceph-mon[37473]: pgmap v5986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:11:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:11:51.280+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:11:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:11:51.280+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:11:51.280+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:11:51.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9869.688820 secs 2024-07-26T14:11:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9869.688820 secs 2024-07-26T14:11:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9869.688820 secs 2024-07-26T14:11:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:52 smithi134 ceph-mon[31966]: pgmap v5987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:52 smithi189 ceph-mon[36790]: pgmap v5987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:52 smithi193 ceph-mon[37473]: pgmap v5987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:54 smithi189 ceph-mon[36790]: pgmap v5988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9855.264814 secs 2024-07-26T14:11:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:54 smithi193 ceph-mon[37473]: pgmap v5988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9855.264814 secs 2024-07-26T14:11:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:54 smithi134 ceph-mon[31966]: pgmap v5988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:11:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9855.264814 secs 2024-07-26T14:11:55.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:11:55.829+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:11:55.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:11:55.829+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:11:55.831 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:11:55.829+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:11:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:56 smithi189 ceph-mon[36790]: pgmap v5989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9874.688936 secs 2024-07-26T14:11:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:56 smithi193 ceph-mon[37473]: pgmap v5989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9874.688936 secs 2024-07-26T14:11:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:56 smithi134 ceph-mon[31966]: pgmap v5989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:11:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9874.688936 secs 2024-07-26T14:11:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:58 smithi189 ceph-mon[36790]: pgmap v5990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:11:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:58 smithi193 ceph-mon[37473]: pgmap v5990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:11:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:58 smithi134 ceph-mon[31966]: pgmap v5990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:11:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9860.264935 secs 2024-07-26T14:12:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:11:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9860.264935 secs 2024-07-26T14:12:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:11:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9860.264935 secs 2024-07-26T14:12:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:00 smithi193 ceph-mon[37473]: pgmap v5991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:00 smithi134 ceph-mon[31966]: pgmap v5991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:00 smithi189 ceph-mon[36790]: pgmap v5991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9879.689015 secs 2024-07-26T14:12:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9879.689015 secs 2024-07-26T14:12:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9879.689015 secs 2024-07-26T14:12:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:02 smithi193 ceph-mon[37473]: pgmap v5992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:03.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:02 smithi134 ceph-mon[31966]: pgmap v5992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:02 smithi189 ceph-mon[36790]: pgmap v5992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:04 smithi193 ceph-mon[37473]: pgmap v5993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9865.265065 secs 2024-07-26T14:12:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:04 smithi134 ceph-mon[31966]: pgmap v5993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9865.265065 secs 2024-07-26T14:12:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:04 smithi189 ceph-mon[36790]: pgmap v5993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9865.265065 secs 2024-07-26T14:12:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:06.295+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:12:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:06.296+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:12:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:06.296+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:06 smithi193 ceph-mon[37473]: pgmap v5994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9884.689129 secs 2024-07-26T14:12:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:06 smithi134 ceph-mon[31966]: pgmap v5994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9884.689129 secs 2024-07-26T14:12:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:06 smithi189 ceph-mon[36790]: pgmap v5994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9884.689129 secs 2024-07-26T14:12:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:08 smithi193 ceph-mon[37473]: pgmap v5995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:08 smithi134 ceph-mon[31966]: pgmap v5995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:08 smithi189 ceph-mon[36790]: pgmap v5995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9870.265128 secs 2024-07-26T14:12:10.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9870.265128 secs 2024-07-26T14:12:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9870.265128 secs 2024-07-26T14:12:10.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:10.844+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:12:10.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:10.845+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:12:10.846 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:10.845+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:10 smithi193 ceph-mon[37473]: pgmap v5996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:11.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:10 smithi134 ceph-mon[31966]: pgmap v5996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:10 smithi189 ceph-mon[36790]: pgmap v5996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9889.689280 secs 2024-07-26T14:12:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9889.689280 secs 2024-07-26T14:12:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9889.689280 secs 2024-07-26T14:12:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:12 smithi193 ceph-mon[37473]: pgmap v5997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:12 smithi134 ceph-mon[31966]: pgmap v5997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:12 smithi189 ceph-mon[36790]: pgmap v5997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:14 smithi134 ceph-mon[31966]: pgmap v5998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9875.265265 secs 2024-07-26T14:12:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:14 smithi189 ceph-mon[36790]: pgmap v5998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9875.265265 secs 2024-07-26T14:12:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:14 smithi193 ceph-mon[37473]: pgmap v5998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9875.265265 secs 2024-07-26T14:12:17.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:16 smithi134 ceph-mon[31966]: pgmap v5999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9894.689396 secs 2024-07-26T14:12:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:16 smithi189 ceph-mon[36790]: pgmap v5999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9894.689396 secs 2024-07-26T14:12:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:16 smithi193 ceph-mon[37473]: pgmap v5999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9894.689396 secs 2024-07-26T14:12:19.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:18 smithi134 ceph-mon[31966]: pgmap v6000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:18 smithi189 ceph-mon[36790]: pgmap v6000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:18 smithi193 ceph-mon[37473]: pgmap v6000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:20.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9880.265376 secs 2024-07-26T14:12:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9880.265376 secs 2024-07-26T14:12:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9880.265376 secs 2024-07-26T14:12:21.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:20 smithi189 ceph-mon[36790]: pgmap v6001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:21.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:20 smithi134 ceph-mon[31966]: pgmap v6001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:21.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:20 smithi193 ceph-mon[37473]: pgmap v6001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:21.312 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:21.311+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:12:21.312 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:21.311+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:21.313 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:21.311+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:12:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9899.689514 secs 2024-07-26T14:12:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9899.689514 secs 2024-07-26T14:12:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9899.689514 secs 2024-07-26T14:12:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:22.318+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 msgr2=0x7f9b300c1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T14:12:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:22.318+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 msgr2=0x7f9b300c1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T14:12:22.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:22.318+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 0x7f9b300c1940 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009770 tx=0x7f9b38008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:12:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:22.318+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 0x7f9b300c1940 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009770 tx=0x7f9b38008040 comp rx=0 tx=0).stop 2024-07-26T14:12:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:22.318+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 msgr2=0x7f9b300c1940 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T14:12:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:22.318+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 msgr2=0x7f9b300c1940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:12:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:22.318+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1530 0x7f9b300c1940 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:12:22.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:22.318+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:12:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:22.319+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:12:22.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:22.320+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b3803a040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:12:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:22 smithi134 ceph-mon[31966]: pgmap v6002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:22 smithi189 ceph-mon[36790]: pgmap v6002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:22 smithi193 ceph-mon[37473]: pgmap v6002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:24.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:24.050+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T14:12:24.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:24.050+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T14:12:24.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:24.050+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T14:12:24.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:24.050+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T14:12:24.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:24.050+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T14:12:24.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:24.050+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T14:12:24.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:24.051+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:12:24.052 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:24.051+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:12:25.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:24 smithi193 ceph-mon[37473]: pgmap v6003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:25.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9885.265494 secs 2024-07-26T14:12:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:24 smithi134 ceph-mon[31966]: pgmap v6003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9885.265494 secs 2024-07-26T14:12:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:24 smithi189 ceph-mon[36790]: pgmap v6003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9885.265494 secs 2024-07-26T14:12:25.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:25.849+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:12:25.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:25.849+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:25.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:25.849+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:12:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:26 smithi134 ceph-mon[31966]: pgmap v6004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9904.689580 secs 2024-07-26T14:12:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:26 smithi189 ceph-mon[36790]: pgmap v6004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9904.689580 secs 2024-07-26T14:12:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:26 smithi193 ceph-mon[37473]: pgmap v6004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9904.689580 secs 2024-07-26T14:12:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:28 smithi134 ceph-mon[31966]: pgmap v6005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:28 smithi189 ceph-mon[36790]: pgmap v6005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:28 smithi193 ceph-mon[37473]: pgmap v6005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9890.265557 secs 2024-07-26T14:12:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9890.265557 secs 2024-07-26T14:12:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9890.265557 secs 2024-07-26T14:12:31.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:30 smithi134 ceph-mon[31966]: pgmap v6006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:30 smithi189 ceph-mon[36790]: pgmap v6006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:30 smithi193 ceph-mon[37473]: pgmap v6006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9909.689680 secs 2024-07-26T14:12:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9909.689680 secs 2024-07-26T14:12:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9909.689680 secs 2024-07-26T14:12:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:32.874+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 msgr2=0x7f21ac0c0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T14:12:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:32.874+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 msgr2=0x7f21ac0c0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T14:12:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:32.874+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 0x7f21ac0c0640 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2198005820 tx=0x7f21b8063000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:12:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:32.874+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 0x7f21ac0c0640 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2198005820 tx=0x7f21b8063000 comp rx=0 tx=0).stop 2024-07-26T14:12:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:32.874+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 msgr2=0x7f21ac0c0640 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T14:12:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:32.874+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 msgr2=0x7f21ac0c0640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:12:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:32.874+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0230 0x7f21ac0c0640 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:12:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:32.874+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:12:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:32.875+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:12:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:32.876+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f2198005e40 tx=0x7f21b80c3000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:12:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:32 smithi134 ceph-mon[31966]: pgmap v6007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:32 smithi189 ceph-mon[36790]: pgmap v6007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:32 smithi193 ceph-mon[37473]: pgmap v6007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9895.265685 secs 2024-07-26T14:12:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9895.265685 secs 2024-07-26T14:12:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9895.265685 secs 2024-07-26T14:12:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:34.575+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:12:34.576 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:34.575+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:12:34.577 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:34.575+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:12:34.577 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:34.575+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:12:34.577 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:34.575+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:12:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:34.575+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:12:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:34.576+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:12:34.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:34.576+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:12:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:34 smithi134 ceph-mon[31966]: pgmap v6008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:35.201 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:34 smithi189 ceph-mon[36790]: pgmap v6008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:34 smithi193 ceph-mon[37473]: pgmap v6008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:36.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:12:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:12:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:12:36.328 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:36.327+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:12:36.328 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:36.327+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:12:36.328 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:36.327+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:36 smithi134 ceph-mon[31966]: pgmap v6009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9914.689761 secs 2024-07-26T14:12:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:12:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:12:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:12:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:36 smithi189 ceph-mon[36790]: pgmap v6009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9914.689761 secs 2024-07-26T14:12:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:12:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:12:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:12:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:36 smithi193 ceph-mon[37473]: pgmap v6009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9914.689761 secs 2024-07-26T14:12:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:12:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:12:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:12:38.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:37 smithi134 ceph-mon[31966]: pgmap v6010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:12:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:37 smithi134 ceph-mon[31966]: pgmap v6011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:12:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:37 smithi189 ceph-mon[36790]: pgmap v6010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:12:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:37 smithi189 ceph-mon[36790]: pgmap v6011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:12:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:37 smithi193 ceph-mon[37473]: pgmap v6010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:12:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:37 smithi193 ceph-mon[37473]: pgmap v6011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:12:39.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9900.265810 secs 2024-07-26T14:12:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9900.265810 secs 2024-07-26T14:12:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9900.265810 secs 2024-07-26T14:12:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:39 smithi134 ceph-mon[31966]: pgmap v6012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:39 smithi189 ceph-mon[36790]: pgmap v6012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:39 smithi193 ceph-mon[37473]: pgmap v6012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:40.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:40.859+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:12:40.860 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:40.859+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:12:40.861 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:40.859+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:41 smithi134 ceph-mon[31966]: pgmap v6013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9919.689868 secs 2024-07-26T14:12:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:41 smithi189 ceph-mon[36790]: pgmap v6013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9919.689868 secs 2024-07-26T14:12:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:41 smithi193 ceph-mon[37473]: pgmap v6013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:12:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9919.689868 secs 2024-07-26T14:12:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:43 smithi134 ceph-mon[31966]: pgmap v6014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:12:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9905.265973 secs 2024-07-26T14:12:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:43 smithi189 ceph-mon[36790]: pgmap v6014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:12:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9905.265973 secs 2024-07-26T14:12:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:43 smithi193 ceph-mon[37473]: pgmap v6014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:12:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9905.265973 secs 2024-07-26T14:12:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:45 smithi134 ceph-mon[31966]: pgmap v6015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:12:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:45 smithi189 ceph-mon[36790]: pgmap v6015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:12:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:45 smithi193 ceph-mon[37473]: pgmap v6015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:12:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9924.689994 secs 2024-07-26T14:12:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9924.689994 secs 2024-07-26T14:12:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9924.689994 secs 2024-07-26T14:12:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:47 smithi189 ceph-mon[36790]: pgmap v6016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:12:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:47 smithi193 ceph-mon[37473]: pgmap v6016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:12:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:47 smithi134 ceph-mon[31966]: pgmap v6016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:12:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9910.266084 secs 2024-07-26T14:12:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9910.266084 secs 2024-07-26T14:12:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9910.266084 secs 2024-07-26T14:12:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:49 smithi193 ceph-mon[37473]: pgmap v6017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:50.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:49 smithi134 ceph-mon[31966]: pgmap v6017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:49 smithi189 ceph-mon[36790]: pgmap v6017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:51.333+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:12:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:51.334+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:12:51.335 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:12:51.334+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:52.245 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:51 smithi189 ceph-mon[36790]: pgmap v6018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:52.245 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9929.690115 secs 2024-07-26T14:12:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:51 smithi193 ceph-mon[37473]: pgmap v6018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9929.690115 secs 2024-07-26T14:12:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:51 smithi134 ceph-mon[31966]: pgmap v6018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9929.690115 secs 2024-07-26T14:12:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:53 smithi193 ceph-mon[37473]: pgmap v6019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9915.266212 secs 2024-07-26T14:12:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:53 smithi134 ceph-mon[31966]: pgmap v6019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9915.266212 secs 2024-07-26T14:12:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:53 smithi189 ceph-mon[36790]: pgmap v6019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:12:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9915.266212 secs 2024-07-26T14:12:55.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:55.874+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:12:55.876 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:55.874+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:12:55.877 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:12:55.874+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:12:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:55 smithi134 ceph-mon[31966]: pgmap v6020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:55 smithi193 ceph-mon[37473]: pgmap v6020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:55 smithi189 ceph-mon[36790]: pgmap v6020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:12:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9934.690231 secs 2024-07-26T14:12:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9934.690231 secs 2024-07-26T14:12:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9934.690231 secs 2024-07-26T14:12:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:57 smithi193 ceph-mon[37473]: pgmap v6021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:57 smithi134 ceph-mon[31966]: pgmap v6021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:57 smithi189 ceph-mon[36790]: pgmap v6021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:12:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:58 smithi193 ceph-mon[37473]: pgmap v6022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:12:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9920.266290 secs 2024-07-26T14:12:59.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:58 smithi134 ceph-mon[31966]: pgmap v6022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:12:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9920.266290 secs 2024-07-26T14:12:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:58 smithi189 ceph-mon[36790]: pgmap v6022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:12:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:12:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9920.266290 secs 2024-07-26T14:13:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:01 smithi189 ceph-mon[36790]: pgmap v6023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9939.690318 secs 2024-07-26T14:13:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:01 smithi193 ceph-mon[37473]: pgmap v6023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9939.690318 secs 2024-07-26T14:13:01.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:01 smithi134 ceph-mon[31966]: pgmap v6023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9939.690318 secs 2024-07-26T14:13:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:03 smithi189 ceph-mon[36790]: pgmap v6024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:03 smithi193 ceph-mon[37473]: pgmap v6024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:03.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:03 smithi134 ceph-mon[31966]: pgmap v6024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9925.266410 secs 2024-07-26T14:13:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9925.266410 secs 2024-07-26T14:13:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9925.266410 secs 2024-07-26T14:13:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:05 smithi193 ceph-mon[37473]: pgmap v6025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:05.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:05 smithi134 ceph-mon[31966]: pgmap v6025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:05 smithi189 ceph-mon[36790]: pgmap v6025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:13:06.349+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:13:06.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:13:06.349+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:06.351 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:13:06.349+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:13:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9944.690432 secs 2024-07-26T14:13:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9944.690432 secs 2024-07-26T14:13:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9944.690432 secs 2024-07-26T14:13:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:07 smithi193 ceph-mon[37473]: pgmap v6026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:07 smithi134 ceph-mon[31966]: pgmap v6026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:07 smithi189 ceph-mon[36790]: pgmap v6026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:09 smithi193 ceph-mon[37473]: pgmap v6027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:13:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9930.266515 secs 2024-07-26T14:13:09.896 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:09 smithi134 ceph-mon[31966]: pgmap v6027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:13:09.896 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9930.266515 secs 2024-07-26T14:13:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:09 smithi189 ceph-mon[36790]: pgmap v6027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:13:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9930.266515 secs 2024-07-26T14:13:10.891 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:13:10.890+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:13:10.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:13:10.890+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:13:10.892 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:13:10.890+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:11 smithi193 ceph-mon[37473]: pgmap v6028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9949.690493 secs 2024-07-26T14:13:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:11 smithi134 ceph-mon[31966]: pgmap v6028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9949.690493 secs 2024-07-26T14:13:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:11 smithi189 ceph-mon[36790]: pgmap v6028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9949.690493 secs 2024-07-26T14:13:13.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:13 smithi193 ceph-mon[37473]: pgmap v6029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:13.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:13 smithi134 ceph-mon[31966]: pgmap v6029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:13 smithi189 ceph-mon[36790]: pgmap v6029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9935.266618 secs 2024-07-26T14:13:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9935.266618 secs 2024-07-26T14:13:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9935.266618 secs 2024-07-26T14:13:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:15 smithi193 ceph-mon[37473]: pgmap v6030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:15.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:15 smithi134 ceph-mon[31966]: pgmap v6030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:15 smithi189 ceph-mon[36790]: pgmap v6030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9954.690616 secs 2024-07-26T14:13:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9954.690616 secs 2024-07-26T14:13:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9954.690616 secs 2024-07-26T14:13:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:17 smithi193 ceph-mon[37473]: pgmap v6031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:17.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:17 smithi134 ceph-mon[31966]: pgmap v6031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:17 smithi189 ceph-mon[36790]: pgmap v6031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:19 smithi193 ceph-mon[37473]: pgmap v6032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:13:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9940.266770 secs 2024-07-26T14:13:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:19 smithi134 ceph-mon[31966]: pgmap v6032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:13:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9940.266770 secs 2024-07-26T14:13:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:19 smithi189 ceph-mon[36790]: pgmap v6032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:13:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9940.266770 secs 2024-07-26T14:13:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:13:21.365+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:13:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:13:21.365+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:13:21.366 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:13:21.365+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:21 smithi193 ceph-mon[37473]: pgmap v6033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9959.690734 secs 2024-07-26T14:13:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:21 smithi134 ceph-mon[31966]: pgmap v6033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9959.690734 secs 2024-07-26T14:13:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:21 smithi189 ceph-mon[36790]: pgmap v6033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9959.690734 secs 2024-07-26T14:13:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:23 smithi134 ceph-mon[31966]: pgmap v6034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:23 smithi189 ceph-mon[36790]: pgmap v6034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:23 smithi193 ceph-mon[37473]: pgmap v6034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9945.266887 secs 2024-07-26T14:13:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9945.266887 secs 2024-07-26T14:13:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9945.266887 secs 2024-07-26T14:13:25.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:13:25.905+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:13:25.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:13:25.905+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:13:25.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:13:25.905+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:25.907 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:25 smithi134 ceph-mon[31966]: pgmap v6035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:25 smithi189 ceph-mon[36790]: pgmap v6035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:25 smithi193 ceph-mon[37473]: pgmap v6035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:13:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9964.690816 secs 2024-07-26T14:13:26.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9964.690816 secs 2024-07-26T14:13:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9964.690816 secs 2024-07-26T14:13:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:27 smithi134 ceph-mon[31966]: pgmap v6036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:27 smithi189 ceph-mon[36790]: pgmap v6036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:27 smithi193 ceph-mon[37473]: pgmap v6036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:29 smithi134 ceph-mon[31966]: pgmap v6037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:13:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9950.267014 secs 2024-07-26T14:13:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:29 smithi189 ceph-mon[36790]: pgmap v6037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:13:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9950.267014 secs 2024-07-26T14:13:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:29 smithi193 ceph-mon[37473]: pgmap v6037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:13:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9950.267014 secs 2024-07-26T14:13:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:31 smithi134 ceph-mon[31966]: pgmap v6038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9969.690971 secs 2024-07-26T14:13:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:31 smithi189 ceph-mon[36790]: pgmap v6038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9969.690971 secs 2024-07-26T14:13:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:31 smithi193 ceph-mon[37473]: pgmap v6038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9969.690971 secs 2024-07-26T14:13:33.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:33 smithi134 ceph-mon[31966]: pgmap v6039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:33 smithi189 ceph-mon[36790]: pgmap v6039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:33 smithi193 ceph-mon[37473]: pgmap v6039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9955.267139 secs 2024-07-26T14:13:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9955.267139 secs 2024-07-26T14:13:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9955.267139 secs 2024-07-26T14:13:35.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:35 smithi134 ceph-mon[31966]: pgmap v6040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T14:13:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:35 smithi189 ceph-mon[36790]: pgmap v6040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T14:13:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:35 smithi193 ceph-mon[37473]: pgmap v6040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T14:13:36.381 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:13:36.380+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:13:36.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:13:36.380+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:36.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:13:36.380+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:13:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9974.691105 secs 2024-07-26T14:13:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:36 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:13:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9974.691105 secs 2024-07-26T14:13:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:36 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:13:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9974.691105 secs 2024-07-26T14:13:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:36 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:13:37.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:37 smithi134 ceph-mon[31966]: pgmap v6041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:37 smithi189 ceph-mon[36790]: pgmap v6041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:37 smithi193 ceph-mon[37473]: pgmap v6041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:39 smithi134 ceph-mon[31966]: pgmap v6042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:13:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9960.267261 secs 2024-07-26T14:13:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:39 smithi189 ceph-mon[36790]: pgmap v6042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:13:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9960.267261 secs 2024-07-26T14:13:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:39 smithi193 ceph-mon[37473]: pgmap v6042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:13:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9960.267261 secs 2024-07-26T14:13:40.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:13:40.921+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:13:40.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:13:40.921+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:13:40.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:13:40.921+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:41 smithi134 ceph-mon[31966]: pgmap v6043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9979.691265 secs 2024-07-26T14:13:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:41 smithi189 ceph-mon[36790]: pgmap v6043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9979.691265 secs 2024-07-26T14:13:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:41 smithi193 ceph-mon[37473]: pgmap v6043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9979.691265 secs 2024-07-26T14:13:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:43 smithi193 ceph-mon[37473]: pgmap v6044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:13:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:13:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:13:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:13:43.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:43 smithi193 ceph-mon[37473]: pgmap v6045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:13:43.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:43 smithi193 ceph-mon[37473]: pgmap v6046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:13:43.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:13:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:43 smithi134 ceph-mon[31966]: pgmap v6044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:13:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:13:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:13:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:13:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:43 smithi134 ceph-mon[31966]: pgmap v6045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:13:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:43 smithi134 ceph-mon[31966]: pgmap v6046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:13:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:13:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:43 smithi189 ceph-mon[36790]: pgmap v6044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:13:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:13:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:13:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:13:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:13:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:43 smithi189 ceph-mon[36790]: pgmap v6045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:13:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:43 smithi189 ceph-mon[36790]: pgmap v6046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:13:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:13:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9965.267397 secs 2024-07-26T14:13:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9965.267397 secs 2024-07-26T14:13:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9965.267397 secs 2024-07-26T14:13:45.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:45 smithi134 ceph-mon[31966]: pgmap v6047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:13:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:45 smithi189 ceph-mon[36790]: pgmap v6047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:13:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:45 smithi193 ceph-mon[37473]: pgmap v6047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:13:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9984.691393 secs 2024-07-26T14:13:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9984.691393 secs 2024-07-26T14:13:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9984.691393 secs 2024-07-26T14:13:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:47 smithi134 ceph-mon[31966]: pgmap v6048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T14:13:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:47 smithi189 ceph-mon[36790]: pgmap v6048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T14:13:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:47 smithi193 ceph-mon[37473]: pgmap v6048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T14:13:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9970.267524 secs 2024-07-26T14:13:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:49 smithi134 ceph-mon[31966]: pgmap v6049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:13:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9970.267524 secs 2024-07-26T14:13:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:49 smithi189 ceph-mon[36790]: pgmap v6049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:13:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9970.267524 secs 2024-07-26T14:13:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:49 smithi193 ceph-mon[37473]: pgmap v6049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:13:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:13:51.385+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:13:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:13:51.385+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:51.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:13:51.385+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:13:51.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:51 smithi134 ceph-mon[31966]: pgmap v6050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:13:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9989.691522 secs 2024-07-26T14:13:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:51 smithi189 ceph-mon[36790]: pgmap v6050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:13:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9989.691522 secs 2024-07-26T14:13:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:51 smithi193 ceph-mon[37473]: pgmap v6050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:13:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9989.691522 secs 2024-07-26T14:13:53.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:53 smithi134 ceph-mon[31966]: pgmap v6051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:13:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:53 smithi189 ceph-mon[36790]: pgmap v6051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:13:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:53 smithi193 ceph-mon[37473]: pgmap v6051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:13:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9975.267647 secs 2024-07-26T14:13:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9975.267647 secs 2024-07-26T14:13:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9975.267647 secs 2024-07-26T14:13:55.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:55 smithi134 ceph-mon[31966]: pgmap v6052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:13:55.933 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:13:55.932+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:13:55.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:13:55.932+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:13:55.934 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:13:55.932+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:13:55.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:55 smithi189 ceph-mon[36790]: pgmap v6052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:13:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:55 smithi193 ceph-mon[37473]: pgmap v6052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:13:56.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9994.691606 secs 2024-07-26T14:13:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9994.691606 secs 2024-07-26T14:13:57.000 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9994.691606 secs 2024-07-26T14:13:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:57 smithi189 ceph-mon[36790]: pgmap v6053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:57 smithi193 ceph-mon[37473]: pgmap v6053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:57 smithi134 ceph-mon[31966]: pgmap v6053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:13:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9980.267766 secs 2024-07-26T14:13:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:13:59 smithi189 ceph-mon[36790]: pgmap v6054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9980.267766 secs 2024-07-26T14:14:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:13:59 smithi193 ceph-mon[37473]: pgmap v6054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9980.267766 secs 2024-07-26T14:14:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:13:59 smithi134 ceph-mon[31966]: pgmap v6054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:01 smithi189 ceph-mon[36790]: pgmap v6055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 9999.691683 secs 2024-07-26T14:14:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:01 smithi193 ceph-mon[37473]: pgmap v6055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 9999.691683 secs 2024-07-26T14:14:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:01 smithi134 ceph-mon[31966]: pgmap v6055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 9999.691683 secs 2024-07-26T14:14:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:03 smithi189 ceph-mon[36790]: pgmap v6056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:03 smithi193 ceph-mon[37473]: pgmap v6056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:04.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:03 smithi134 ceph-mon[31966]: pgmap v6056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9985.267908 secs 2024-07-26T14:14:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9985.267908 secs 2024-07-26T14:14:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9985.267908 secs 2024-07-26T14:14:06.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:05 smithi193 ceph-mon[37473]: pgmap v6057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:14:06.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:05 smithi134 ceph-mon[31966]: pgmap v6057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:14:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:05 smithi189 ceph-mon[36790]: pgmap v6057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:14:06.392 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:14:06.391+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:14:06.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:14:06.391+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:14:06.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:14:06.391+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10004.691749 secs 2024-07-26T14:14:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10004.691749 secs 2024-07-26T14:14:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10004.691749 secs 2024-07-26T14:14:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:07 smithi193 ceph-mon[37473]: pgmap v6058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:08.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:07 smithi134 ceph-mon[31966]: pgmap v6058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:07 smithi189 ceph-mon[36790]: pgmap v6058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9990.268039 secs 2024-07-26T14:14:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:09 smithi193 ceph-mon[37473]: pgmap v6059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9990.268039 secs 2024-07-26T14:14:10.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:09 smithi134 ceph-mon[31966]: pgmap v6059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9990.268039 secs 2024-07-26T14:14:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:09 smithi189 ceph-mon[36790]: pgmap v6059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:10.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:14:10.934+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:14:10.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:14:10.934+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:14:10.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:14:10.934+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:11 smithi193 ceph-mon[37473]: pgmap v6060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10009.691893 secs 2024-07-26T14:14:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:11 smithi134 ceph-mon[31966]: pgmap v6060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10009.691893 secs 2024-07-26T14:14:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:11 smithi189 ceph-mon[36790]: pgmap v6060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10009.691893 secs 2024-07-26T14:14:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:13 smithi193 ceph-mon[37473]: pgmap v6061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:14.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:13 smithi134 ceph-mon[31966]: pgmap v6061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:13 smithi189 ceph-mon[36790]: pgmap v6061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 9995.268155 secs 2024-07-26T14:14:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 9995.268155 secs 2024-07-26T14:14:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 9995.268155 secs 2024-07-26T14:14:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:15 smithi193 ceph-mon[37473]: pgmap v6062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:14:16.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:15 smithi134 ceph-mon[31966]: pgmap v6062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:14:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:15 smithi189 ceph-mon[36790]: pgmap v6062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:14:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10014.691961 secs 2024-07-26T14:14:17.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10014.691961 secs 2024-07-26T14:14:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10014.691961 secs 2024-07-26T14:14:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:17 smithi193 ceph-mon[37473]: pgmap v6063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:17 smithi134 ceph-mon[31966]: pgmap v6063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:17 smithi189 ceph-mon[36790]: pgmap v6063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:20.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10000.268281 secs 2024-07-26T14:14:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:19 smithi134 ceph-mon[31966]: pgmap v6064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10000.268281 secs 2024-07-26T14:14:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:19 smithi189 ceph-mon[36790]: pgmap v6064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10000.268281 secs 2024-07-26T14:14:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:19 smithi193 ceph-mon[37473]: pgmap v6064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:21.397 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:14:21.396+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:14:21.397 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:14:21.396+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:14:21.398 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:14:21.396+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:21 smithi134 ceph-mon[31966]: pgmap v6065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10019.692049 secs 2024-07-26T14:14:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:21 smithi189 ceph-mon[36790]: pgmap v6065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10019.692049 secs 2024-07-26T14:14:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:21 smithi193 ceph-mon[37473]: pgmap v6065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:22.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10019.692049 secs 2024-07-26T14:14:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:23 smithi134 ceph-mon[31966]: pgmap v6066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:23 smithi189 ceph-mon[36790]: pgmap v6066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:23 smithi193 ceph-mon[37473]: pgmap v6066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10005.268374 secs 2024-07-26T14:14:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10005.268374 secs 2024-07-26T14:14:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10005.268374 secs 2024-07-26T14:14:25.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:14:25.936+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:14:25.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:14:25.936+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:25.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:14:25.936+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:14:26.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:25 smithi134 ceph-mon[31966]: pgmap v6067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:14:26.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:25 smithi193 ceph-mon[37473]: pgmap v6067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:14:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:25 smithi189 ceph-mon[36790]: pgmap v6067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:14:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10024.692194 secs 2024-07-26T14:14:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10024.692194 secs 2024-07-26T14:14:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10024.692194 secs 2024-07-26T14:14:28.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:27 smithi134 ceph-mon[31966]: pgmap v6068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:27 smithi189 ceph-mon[36790]: pgmap v6068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:27 smithi193 ceph-mon[37473]: pgmap v6068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10010.268497 secs 2024-07-26T14:14:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:29 smithi134 ceph-mon[31966]: pgmap v6069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10010.268497 secs 2024-07-26T14:14:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:29 smithi189 ceph-mon[36790]: pgmap v6069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10010.268497 secs 2024-07-26T14:14:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:29 smithi193 ceph-mon[37473]: pgmap v6069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:31 smithi134 ceph-mon[31966]: pgmap v6070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10029.692334 secs 2024-07-26T14:14:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:31 smithi189 ceph-mon[36790]: pgmap v6070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10029.692334 secs 2024-07-26T14:14:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:31 smithi193 ceph-mon[37473]: pgmap v6070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10029.692334 secs 2024-07-26T14:14:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:33 smithi134 ceph-mon[31966]: pgmap v6071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10015.268618 secs 2024-07-26T14:14:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:33 smithi189 ceph-mon[36790]: pgmap v6071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10015.268618 secs 2024-07-26T14:14:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:33 smithi193 ceph-mon[37473]: pgmap v6071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10015.268618 secs 2024-07-26T14:14:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:35 smithi134 ceph-mon[31966]: pgmap v6072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:14:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:35 smithi189 ceph-mon[36790]: pgmap v6072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:14:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:35 smithi193 ceph-mon[37473]: pgmap v6072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:14:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:14:36.412+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:14:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:14:36.412+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:14:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:14:36.412+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10034.692378 secs 2024-07-26T14:14:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10034.692378 secs 2024-07-26T14:14:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10034.692378 secs 2024-07-26T14:14:38.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:37 smithi134 ceph-mon[31966]: pgmap v6073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:37 smithi189 ceph-mon[36790]: pgmap v6073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:37 smithi193 ceph-mon[37473]: pgmap v6073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:39.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10020.268744 secs 2024-07-26T14:14:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10020.268744 secs 2024-07-26T14:14:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10020.268744 secs 2024-07-26T14:14:40.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:39 smithi134 ceph-mon[31966]: pgmap v6074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:39 smithi189 ceph-mon[36790]: pgmap v6074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:39 smithi193 ceph-mon[37473]: pgmap v6074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:40.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:14:40.940+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:14:40.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:14:40.940+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:14:40.941 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:14:40.940+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:41 smithi134 ceph-mon[31966]: pgmap v6075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10039.692490 secs 2024-07-26T14:14:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:41 smithi189 ceph-mon[36790]: pgmap v6075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10039.692490 secs 2024-07-26T14:14:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:41 smithi193 ceph-mon[37473]: pgmap v6075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10039.692490 secs 2024-07-26T14:14:43.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:14:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:14:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:14:43.988 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:43 smithi193 ceph-mon[37473]: pgmap v6076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:43.988 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10025.268831 secs 2024-07-26T14:14:43.988 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:43.988 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:43.988 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:43.988 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:43.996 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:43 smithi189 ceph-mon[36790]: pgmap v6076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:43.996 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10025.268831 secs 2024-07-26T14:14:43.996 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:43.996 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:43.997 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:43.997 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:43 smithi134 ceph-mon[31966]: pgmap v6076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:14:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10025.268831 secs 2024-07-26T14:14:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:14:45.014 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:44 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:14:45.014 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:44 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:14:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:44 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:14:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:44 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:14:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:44 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:14:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:44 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:14:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:45 smithi189 ceph-mon[36790]: pgmap v6077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:14:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:45 smithi134 ceph-mon[31966]: pgmap v6077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:14:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:45 smithi193 ceph-mon[37473]: pgmap v6077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:14:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10044.692566 secs 2024-07-26T14:14:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:14:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:14:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:14:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:14:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:14:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:14:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:14:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10044.692566 secs 2024-07-26T14:14:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:14:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:14:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:14:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:14:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:14:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:14:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:14:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10044.692566 secs 2024-07-26T14:14:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:14:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:14:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:14:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:14:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:14:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:14:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:14:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:47 smithi189 ceph-mon[36790]: pgmap v6078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:47 smithi189 ceph-mon[36790]: pgmap v6079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:14:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:47 smithi189 ceph-mon[36790]: pgmap v6080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T14:14:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:47 smithi193 ceph-mon[37473]: pgmap v6078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:47 smithi193 ceph-mon[37473]: pgmap v6079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:14:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:47 smithi193 ceph-mon[37473]: pgmap v6080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T14:14:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:47 smithi134 ceph-mon[31966]: pgmap v6078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:14:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:47 smithi134 ceph-mon[31966]: pgmap v6079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:14:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:47 smithi134 ceph-mon[31966]: pgmap v6080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T14:14:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10030.269000 secs 2024-07-26T14:14:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10030.269000 secs 2024-07-26T14:14:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10030.269000 secs 2024-07-26T14:14:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:49 smithi189 ceph-mon[36790]: pgmap v6081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:14:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:49 smithi193 ceph-mon[37473]: pgmap v6081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:14:50.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:49 smithi134 ceph-mon[31966]: pgmap v6081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:14:51.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:14:51.427+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:14:51.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:14:51.427+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:51.429 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:14:51.427+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:14:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:51 smithi189 ceph-mon[36790]: pgmap v6082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T14:14:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10049.692705 secs 2024-07-26T14:14:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:51 smithi193 ceph-mon[37473]: pgmap v6082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T14:14:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10049.692705 secs 2024-07-26T14:14:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:51 smithi134 ceph-mon[31966]: pgmap v6082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T14:14:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10049.692705 secs 2024-07-26T14:14:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:53 smithi193 ceph-mon[37473]: pgmap v6083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:14:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10035.269090 secs 2024-07-26T14:14:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:53 smithi134 ceph-mon[31966]: pgmap v6083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:14:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10035.269090 secs 2024-07-26T14:14:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:53 smithi189 ceph-mon[36790]: pgmap v6083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:14:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10035.269090 secs 2024-07-26T14:14:55.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:14:55.954+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:14:55.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:14:55.954+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:14:55.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:14:55.954+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:14:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:55 smithi193 ceph-mon[37473]: pgmap v6084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:14:56.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:55 smithi134 ceph-mon[31966]: pgmap v6084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:14:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:55 smithi189 ceph-mon[36790]: pgmap v6084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:14:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10054.692805 secs 2024-07-26T14:14:57.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10054.692805 secs 2024-07-26T14:14:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10054.692805 secs 2024-07-26T14:14:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:57 smithi189 ceph-mon[36790]: pgmap v6085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:14:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:57 smithi193 ceph-mon[37473]: pgmap v6085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:14:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:57 smithi134 ceph-mon[31966]: pgmap v6085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:14:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10040.269212 secs 2024-07-26T14:14:59.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10040.269212 secs 2024-07-26T14:14:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10040.269212 secs 2024-07-26T14:15:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:14:59 smithi193 ceph-mon[37473]: pgmap v6086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:14:59 smithi134 ceph-mon[31966]: pgmap v6086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:14:59 smithi189 ceph-mon[36790]: pgmap v6086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:01 smithi193 ceph-mon[37473]: pgmap v6087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10059.692955 secs 2024-07-26T14:15:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:01 smithi134 ceph-mon[31966]: pgmap v6087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10059.692955 secs 2024-07-26T14:15:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:01 smithi189 ceph-mon[36790]: pgmap v6087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10059.692955 secs 2024-07-26T14:15:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:03 smithi193 ceph-mon[37473]: pgmap v6088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:03.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:03 smithi134 ceph-mon[31966]: pgmap v6088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:03 smithi189 ceph-mon[36790]: pgmap v6088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10045.269338 secs 2024-07-26T14:15:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10045.269338 secs 2024-07-26T14:15:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10045.269338 secs 2024-07-26T14:15:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:05 smithi193 ceph-mon[37473]: pgmap v6089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:15:05.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:05 smithi134 ceph-mon[31966]: pgmap v6089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:15:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:05 smithi189 ceph-mon[36790]: pgmap v6089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:15:06.434 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:15:06.433+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:15:06.435 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:15:06.433+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:06.435 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:15:06.433+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:15:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10064.693087 secs 2024-07-26T14:15:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10064.693087 secs 2024-07-26T14:15:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10064.693087 secs 2024-07-26T14:15:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:07 smithi189 ceph-mon[36790]: pgmap v6090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:07 smithi193 ceph-mon[37473]: pgmap v6090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:07 smithi134 ceph-mon[31966]: pgmap v6090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10050.269510 secs 2024-07-26T14:15:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:09 smithi134 ceph-mon[31966]: pgmap v6091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10050.269510 secs 2024-07-26T14:15:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:09 smithi189 ceph-mon[36790]: pgmap v6091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10050.269510 secs 2024-07-26T14:15:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:09 smithi193 ceph-mon[37473]: pgmap v6091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:10.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:15:10.969+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:15:10.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:15:10.969+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:10.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:15:10.969+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:15:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:11 smithi134 ceph-mon[31966]: pgmap v6092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10069.693246 secs 2024-07-26T14:15:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:11 smithi189 ceph-mon[36790]: pgmap v6092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10069.693246 secs 2024-07-26T14:15:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:11 smithi193 ceph-mon[37473]: pgmap v6092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10069.693246 secs 2024-07-26T14:15:14.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:13 smithi134 ceph-mon[31966]: pgmap v6093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:13 smithi189 ceph-mon[36790]: pgmap v6093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:13 smithi193 ceph-mon[37473]: pgmap v6093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10055.269641 secs 2024-07-26T14:15:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10055.269641 secs 2024-07-26T14:15:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10055.269641 secs 2024-07-26T14:15:16.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:15 smithi134 ceph-mon[31966]: pgmap v6094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:15:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:15 smithi189 ceph-mon[36790]: pgmap v6094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:15:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:15 smithi193 ceph-mon[37473]: pgmap v6094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:15:17.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10074.693376 secs 2024-07-26T14:15:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10074.693376 secs 2024-07-26T14:15:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10074.693376 secs 2024-07-26T14:15:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:17 smithi134 ceph-mon[31966]: pgmap v6095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:17 smithi189 ceph-mon[36790]: pgmap v6095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:17 smithi193 ceph-mon[37473]: pgmap v6095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:19.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10060.269796 secs 2024-07-26T14:15:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10060.269796 secs 2024-07-26T14:15:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10060.269796 secs 2024-07-26T14:15:20.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:19 smithi134 ceph-mon[31966]: pgmap v6096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:19 smithi189 ceph-mon[36790]: pgmap v6096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:19 smithi193 ceph-mon[37473]: pgmap v6096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:21.450 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:15:21.449+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:15:21.450 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:15:21.449+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:15:21.450 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:15:21.449+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:21 smithi134 ceph-mon[31966]: pgmap v6097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10079.693509 secs 2024-07-26T14:15:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:21 smithi189 ceph-mon[36790]: pgmap v6097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10079.693509 secs 2024-07-26T14:15:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:21 smithi193 ceph-mon[37473]: pgmap v6097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10079.693509 secs 2024-07-26T14:15:24.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:15:24.081+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:15:24.082 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:15:24.081+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:15:24.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:15:24.082+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T14:15:24.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:15:24.082+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T14:15:24.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:23 smithi134 ceph-mon[31966]: pgmap v6098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10065.269929 secs 2024-07-26T14:15:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:23 smithi189 ceph-mon[36790]: pgmap v6098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10065.269929 secs 2024-07-26T14:15:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:23 smithi193 ceph-mon[37473]: pgmap v6098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10065.269929 secs 2024-07-26T14:15:25.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:15:25.982+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:15:25.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:15:25.982+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:15:25.983 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:15:25.982+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:26.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:25 smithi134 ceph-mon[31966]: pgmap v6099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:15:26.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:25 smithi189 ceph-mon[36790]: pgmap v6099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:15:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:25 smithi193 ceph-mon[37473]: pgmap v6099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:15:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10084.693637 secs 2024-07-26T14:15:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10084.693637 secs 2024-07-26T14:15:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10084.693637 secs 2024-07-26T14:15:28.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:27 smithi134 ceph-mon[31966]: pgmap v6100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:27 smithi189 ceph-mon[36790]: pgmap v6100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:27 smithi193 ceph-mon[37473]: pgmap v6100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:29.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10070.270077 secs 2024-07-26T14:15:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10070.270077 secs 2024-07-26T14:15:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10070.270077 secs 2024-07-26T14:15:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:29 smithi134 ceph-mon[31966]: pgmap v6101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:29 smithi189 ceph-mon[36790]: pgmap v6101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:29 smithi193 ceph-mon[37473]: pgmap v6101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:31 smithi134 ceph-mon[31966]: pgmap v6102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10089.693784 secs 2024-07-26T14:15:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:31 smithi189 ceph-mon[36790]: pgmap v6102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10089.693784 secs 2024-07-26T14:15:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:31 smithi193 ceph-mon[37473]: pgmap v6102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10089.693784 secs 2024-07-26T14:15:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:33 smithi134 ceph-mon[31966]: pgmap v6103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10075.270149 secs 2024-07-26T14:15:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:33 smithi189 ceph-mon[36790]: pgmap v6103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10075.270149 secs 2024-07-26T14:15:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:33 smithi193 ceph-mon[37473]: pgmap v6103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10075.270149 secs 2024-07-26T14:15:34.607 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:15:34.606+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:15:34.607 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:15:34.606+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:15:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:15:34.607+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:15:34.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:15:34.607+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:15:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:35 smithi189 ceph-mon[36790]: pgmap v6104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:15:36.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:35 smithi134 ceph-mon[31966]: pgmap v6104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:15:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:35 smithi193 ceph-mon[37473]: pgmap v6104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:15:36.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:15:36.454+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:15:36.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:15:36.454+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:15:36.455 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:15:36.454+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10094.693916 secs 2024-07-26T14:15:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10094.693916 secs 2024-07-26T14:15:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10094.693916 secs 2024-07-26T14:15:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:37 smithi189 ceph-mon[36790]: pgmap v6105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:37 smithi193 ceph-mon[37473]: pgmap v6105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:38.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:37 smithi134 ceph-mon[31966]: pgmap v6105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10080.270218 secs 2024-07-26T14:15:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10080.270218 secs 2024-07-26T14:15:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10080.270218 secs 2024-07-26T14:15:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:39 smithi193 ceph-mon[37473]: pgmap v6106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:39 smithi134 ceph-mon[31966]: pgmap v6106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:39 smithi189 ceph-mon[36790]: pgmap v6106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:40.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:15:40.995+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:15:40.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:15:40.995+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:15:40.997 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:15:40.995+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:41 smithi193 ceph-mon[37473]: pgmap v6107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10099.694073 secs 2024-07-26T14:15:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:41 smithi134 ceph-mon[31966]: pgmap v6107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10099.694073 secs 2024-07-26T14:15:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:41 smithi189 ceph-mon[36790]: pgmap v6107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10099.694073 secs 2024-07-26T14:15:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:43 smithi193 ceph-mon[37473]: pgmap v6108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10085.270297 secs 2024-07-26T14:15:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:43 smithi134 ceph-mon[31966]: pgmap v6108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10085.270297 secs 2024-07-26T14:15:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:43 smithi189 ceph-mon[36790]: pgmap v6108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:15:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10085.270297 secs 2024-07-26T14:15:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:45 smithi193 ceph-mon[37473]: pgmap v6109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:15:46.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:45 smithi134 ceph-mon[31966]: pgmap v6109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:15:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:45 smithi189 ceph-mon[36790]: pgmap v6109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:15:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10104.694190 secs 2024-07-26T14:15:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:15:47.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10104.694190 secs 2024-07-26T14:15:47.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:15:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10104.694190 secs 2024-07-26T14:15:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:15:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:47 smithi193 ceph-mon[37473]: pgmap v6110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:15:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:15:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:15:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:47 smithi134 ceph-mon[31966]: pgmap v6110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:15:48.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:15:48.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:15:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:47 smithi189 ceph-mon[36790]: pgmap v6110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:15:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:15:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:15:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:15:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:15:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:48 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:15:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:48 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:15:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:48 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:15:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:48 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:15:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:48 smithi193 ceph-mon[37473]: pgmap v6111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:15:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:48 smithi193 ceph-mon[37473]: pgmap v6112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:15:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10090.270409 secs 2024-07-26T14:15:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:48 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:15:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:48 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:15:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:48 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:15:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:48 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:15:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:48 smithi134 ceph-mon[31966]: pgmap v6111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:15:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:48 smithi134 ceph-mon[31966]: pgmap v6112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:15:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10090.270409 secs 2024-07-26T14:15:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:48 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:15:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:48 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:15:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:48 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:15:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:48 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:15:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:48 smithi189 ceph-mon[36790]: pgmap v6111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:15:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:48 smithi189 ceph-mon[36790]: pgmap v6112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:15:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10090.270409 secs 2024-07-26T14:15:50.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:50 smithi134 ceph-mon[31966]: pgmap v6113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:15:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:50 smithi189 ceph-mon[36790]: pgmap v6113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:15:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:50 smithi193 ceph-mon[37473]: pgmap v6113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:15:51.469 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:15:51.468+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:15:51.469 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:15:51.468+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:15:51.469 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:15:51.468+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10109.694343 secs 2024-07-26T14:15:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10109.694343 secs 2024-07-26T14:15:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10109.694343 secs 2024-07-26T14:15:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:52 smithi189 ceph-mon[36790]: pgmap v6114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:15:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:52 smithi193 ceph-mon[37473]: pgmap v6114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:15:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:52 smithi134 ceph-mon[31966]: pgmap v6114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:15:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10095.270539 secs 2024-07-26T14:15:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10095.270539 secs 2024-07-26T14:15:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10095.270539 secs 2024-07-26T14:15:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:54 smithi134 ceph-mon[31966]: pgmap v6115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:15:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:54 smithi189 ceph-mon[36790]: pgmap v6115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:15:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:54 smithi193 ceph-mon[37473]: pgmap v6115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:15:56.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:15:56.011+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:15:56.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:15:56.011+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:15:56.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:15:56.011+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:15:56.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:56 smithi134 ceph-mon[31966]: pgmap v6116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:15:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:56 smithi189 ceph-mon[36790]: pgmap v6116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:15:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:56 smithi193 ceph-mon[37473]: pgmap v6116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:15:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10114.694404 secs 2024-07-26T14:15:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10114.694404 secs 2024-07-26T14:15:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10114.694404 secs 2024-07-26T14:15:58.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:58 smithi134 ceph-mon[31966]: pgmap v6117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:15:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:58 smithi189 ceph-mon[36790]: pgmap v6117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:15:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:58 smithi193 ceph-mon[37473]: pgmap v6117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:15:59.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:15:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10100.270675 secs 2024-07-26T14:15:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:15:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10100.270675 secs 2024-07-26T14:15:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:15:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10100.270675 secs 2024-07-26T14:16:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:00 smithi134 ceph-mon[31966]: pgmap v6118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:00 smithi189 ceph-mon[36790]: pgmap v6118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:00 smithi193 ceph-mon[37473]: pgmap v6118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10119.694520 secs 2024-07-26T14:16:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10119.694520 secs 2024-07-26T14:16:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10119.694520 secs 2024-07-26T14:16:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:02 smithi189 ceph-mon[36790]: pgmap v6119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:02 smithi193 ceph-mon[37473]: pgmap v6119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:02 smithi134 ceph-mon[31966]: pgmap v6119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:04.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10105.270793 secs 2024-07-26T14:16:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10105.270793 secs 2024-07-26T14:16:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10105.270793 secs 2024-07-26T14:16:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:04 smithi134 ceph-mon[31966]: pgmap v6120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:04 smithi189 ceph-mon[36790]: pgmap v6120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:04 smithi193 ceph-mon[37473]: pgmap v6120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:06.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:06 smithi134 ceph-mon[31966]: pgmap v6121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:06 smithi189 ceph-mon[36790]: pgmap v6121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:06.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:16:06.472+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:16:06.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:16:06.472+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:16:06.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:16:06.472+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:06 smithi193 ceph-mon[37473]: pgmap v6121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10124.694659 secs 2024-07-26T14:16:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10124.694659 secs 2024-07-26T14:16:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10124.694659 secs 2024-07-26T14:16:08.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:08 smithi134 ceph-mon[31966]: pgmap v6122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:08 smithi189 ceph-mon[36790]: pgmap v6122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:08 smithi193 ceph-mon[37473]: pgmap v6122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:09.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10110.270913 secs 2024-07-26T14:16:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10110.270913 secs 2024-07-26T14:16:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10110.270913 secs 2024-07-26T14:16:10.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:10 smithi134 ceph-mon[31966]: pgmap v6123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:10 smithi189 ceph-mon[36790]: pgmap v6123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:10 smithi193 ceph-mon[37473]: pgmap v6123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:11.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:16:11.026+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:16:11.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:16:11.026+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:11.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:16:11.026+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:16:11.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10129.694789 secs 2024-07-26T14:16:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10129.694789 secs 2024-07-26T14:16:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10129.694789 secs 2024-07-26T14:16:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:12 smithi189 ceph-mon[36790]: pgmap v6124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:12 smithi193 ceph-mon[37473]: pgmap v6124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:12 smithi134 ceph-mon[31966]: pgmap v6124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10115.271039 secs 2024-07-26T14:16:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10115.271039 secs 2024-07-26T14:16:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10115.271039 secs 2024-07-26T14:16:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:14 smithi189 ceph-mon[36790]: pgmap v6125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:14 smithi193 ceph-mon[37473]: pgmap v6125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:14 smithi134 ceph-mon[31966]: pgmap v6125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:16.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:16 smithi134 ceph-mon[31966]: pgmap v6126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:16 smithi189 ceph-mon[36790]: pgmap v6126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:16 smithi193 ceph-mon[37473]: pgmap v6126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:17.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10134.694939 secs 2024-07-26T14:16:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10134.694939 secs 2024-07-26T14:16:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10134.694939 secs 2024-07-26T14:16:18.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:18 smithi134 ceph-mon[31966]: pgmap v6127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:18 smithi189 ceph-mon[36790]: pgmap v6127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:18.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:18 smithi193 ceph-mon[37473]: pgmap v6127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10120.271160 secs 2024-07-26T14:16:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10120.271160 secs 2024-07-26T14:16:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10120.271160 secs 2024-07-26T14:16:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:20 smithi134 ceph-mon[31966]: pgmap v6128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:20 smithi189 ceph-mon[36790]: pgmap v6128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:20 smithi193 ceph-mon[37473]: pgmap v6128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:21.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:16:21.486+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:16:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:16:21.486+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:16:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:16:21.486+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10139.695077 secs 2024-07-26T14:16:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10139.695077 secs 2024-07-26T14:16:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10139.695077 secs 2024-07-26T14:16:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:22 smithi189 ceph-mon[36790]: pgmap v6129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:22 smithi193 ceph-mon[37473]: pgmap v6129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:22 smithi134 ceph-mon[31966]: pgmap v6129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10125.271292 secs 2024-07-26T14:16:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10125.271292 secs 2024-07-26T14:16:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10125.271292 secs 2024-07-26T14:16:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:24 smithi134 ceph-mon[31966]: pgmap v6130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:24 smithi189 ceph-mon[36790]: pgmap v6130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:24 smithi193 ceph-mon[37473]: pgmap v6130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:26.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:16:26.041+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:16:26.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:16:26.041+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:16:26.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:16:26.041+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:26.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:26 smithi134 ceph-mon[31966]: pgmap v6131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:26 smithi189 ceph-mon[36790]: pgmap v6131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:26 smithi193 ceph-mon[37473]: pgmap v6131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10144.695189 secs 2024-07-26T14:16:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10144.695189 secs 2024-07-26T14:16:27.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10144.695189 secs 2024-07-26T14:16:28.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:28 smithi134 ceph-mon[31966]: pgmap v6132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:28 smithi189 ceph-mon[36790]: pgmap v6132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:28 smithi193 ceph-mon[37473]: pgmap v6132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10130.271417 secs 2024-07-26T14:16:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10130.271417 secs 2024-07-26T14:16:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10130.271417 secs 2024-07-26T14:16:30.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:30 smithi134 ceph-mon[31966]: pgmap v6133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:30 smithi189 ceph-mon[36790]: pgmap v6133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:30 smithi193 ceph-mon[37473]: pgmap v6133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10149.695331 secs 2024-07-26T14:16:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10149.695331 secs 2024-07-26T14:16:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10149.695331 secs 2024-07-26T14:16:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:32 smithi189 ceph-mon[36790]: pgmap v6134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:32 smithi193 ceph-mon[37473]: pgmap v6134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:32 smithi134 ceph-mon[31966]: pgmap v6134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10135.271537 secs 2024-07-26T14:16:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10135.271537 secs 2024-07-26T14:16:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10135.271537 secs 2024-07-26T14:16:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:34 smithi134 ceph-mon[31966]: pgmap v6135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:34 smithi189 ceph-mon[36790]: pgmap v6135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:34 smithi193 ceph-mon[37473]: pgmap v6135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:36.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:36 smithi134 ceph-mon[31966]: pgmap v6136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:36 smithi189 ceph-mon[36790]: pgmap v6136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:16:36.490+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:16:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:16:36.490+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:16:36.490+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:16:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:36 smithi193 ceph-mon[37473]: pgmap v6136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10154.695410 secs 2024-07-26T14:16:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10154.695410 secs 2024-07-26T14:16:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10154.695410 secs 2024-07-26T14:16:38.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:38 smithi134 ceph-mon[31966]: pgmap v6137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:38 smithi189 ceph-mon[36790]: pgmap v6137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:38 smithi193 ceph-mon[37473]: pgmap v6137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:16:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10140.271669 secs 2024-07-26T14:16:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10140.271669 secs 2024-07-26T14:16:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10140.271669 secs 2024-07-26T14:16:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:40 smithi134 ceph-mon[31966]: pgmap v6138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:40 smithi189 ceph-mon[36790]: pgmap v6138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:40 smithi193 ceph-mon[37473]: pgmap v6138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:16:41.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:16:41.057+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:16:41.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:16:41.057+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:41.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:16:41.057+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:16:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10159.695491 secs 2024-07-26T14:16:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10159.695491 secs 2024-07-26T14:16:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10159.695491 secs 2024-07-26T14:16:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:42 smithi189 ceph-mon[36790]: pgmap v6139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:42 smithi193 ceph-mon[37473]: pgmap v6139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:42 smithi134 ceph-mon[31966]: pgmap v6139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:16:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10145.271816 secs 2024-07-26T14:16:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10145.271816 secs 2024-07-26T14:16:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10145.271816 secs 2024-07-26T14:16:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:44 smithi134 ceph-mon[31966]: pgmap v6140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:16:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:44 smithi189 ceph-mon[36790]: pgmap v6140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:16:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:44 smithi193 ceph-mon[37473]: pgmap v6140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:16:46.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:46 smithi134 ceph-mon[31966]: pgmap v6141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:16:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:46 smithi189 ceph-mon[36790]: pgmap v6141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:16:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:46 smithi193 ceph-mon[37473]: pgmap v6141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:16:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10164.695611 secs 2024-07-26T14:16:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10164.695611 secs 2024-07-26T14:16:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10164.695611 secs 2024-07-26T14:16:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:48 smithi134 ceph-mon[31966]: pgmap v6142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:16:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:48 smithi189 ceph-mon[36790]: pgmap v6142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:16:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:48 smithi193 ceph-mon[37473]: pgmap v6142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:16:49.106 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:16:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10150.271924 secs 2024-07-26T14:16:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:16:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10150.271924 secs 2024-07-26T14:16:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:16:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10150.271924 secs 2024-07-26T14:16:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:16:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:16:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:50 smithi134 ceph-mon[31966]: pgmap v6143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:16:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:16:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:16:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:16:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:50 smithi189 ceph-mon[36790]: pgmap v6143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:16:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:16:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:16:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:16:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:50 smithi193 ceph-mon[37473]: pgmap v6143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:16:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:16:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:16:51.506+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:16:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:16:51.506+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:16:51.507 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:16:51.506+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10169.695677 secs 2024-07-26T14:16:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:52 smithi193 ceph-mon[37473]: pgmap v6144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:16:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10169.695677 secs 2024-07-26T14:16:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:52 smithi134 ceph-mon[31966]: pgmap v6144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:16:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10169.695677 secs 2024-07-26T14:16:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:52 smithi189 ceph-mon[36790]: pgmap v6144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:16:54.495 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:54 smithi189 ceph-mon[36790]: pgmap v6145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T14:16:54.495 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10155.272050 secs 2024-07-26T14:16:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:54 smithi193 ceph-mon[37473]: pgmap v6145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T14:16:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10155.272050 secs 2024-07-26T14:16:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:54 smithi134 ceph-mon[31966]: pgmap v6145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T14:16:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10155.272050 secs 2024-07-26T14:16:56.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:16:56.072+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:16:56.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:16:56.072+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:16:56.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:16:56.072+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:16:56.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:56 smithi134 ceph-mon[31966]: pgmap v6146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:16:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:56 smithi193 ceph-mon[37473]: pgmap v6146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:16:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:56 smithi189 ceph-mon[36790]: pgmap v6146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:16:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10174.695803 secs 2024-07-26T14:16:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10174.695803 secs 2024-07-26T14:16:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10174.695803 secs 2024-07-26T14:16:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:58 smithi193 ceph-mon[37473]: pgmap v6147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:16:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:58 smithi134 ceph-mon[31966]: pgmap v6147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:16:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:58 smithi189 ceph-mon[36790]: pgmap v6147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:16:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:16:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10160.272193 secs 2024-07-26T14:16:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:16:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10160.272193 secs 2024-07-26T14:16:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:16:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10160.272193 secs 2024-07-26T14:17:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:00 smithi193 ceph-mon[37473]: pgmap v6148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:17:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:00 smithi134 ceph-mon[31966]: pgmap v6148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:17:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:00 smithi189 ceph-mon[36790]: pgmap v6148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:17:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10179.695951 secs 2024-07-26T14:17:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:02 smithi193 ceph-mon[37473]: pgmap v6149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10179.695951 secs 2024-07-26T14:17:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:02 smithi134 ceph-mon[31966]: pgmap v6149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10179.695951 secs 2024-07-26T14:17:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:02 smithi189 ceph-mon[36790]: pgmap v6149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:04 smithi193 ceph-mon[37473]: pgmap v6150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:17:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10165.272270 secs 2024-07-26T14:17:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:04 smithi134 ceph-mon[31966]: pgmap v6150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:17:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10165.272270 secs 2024-07-26T14:17:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:04 smithi189 ceph-mon[36790]: pgmap v6150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:17:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10165.272270 secs 2024-07-26T14:17:06.522 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:17:06.521+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:17:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:17:06.521+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:06.523 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:17:06.521+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:17:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:06 smithi193 ceph-mon[37473]: pgmap v6151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:06 smithi134 ceph-mon[31966]: pgmap v6151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:06 smithi189 ceph-mon[36790]: pgmap v6151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10184.696075 secs 2024-07-26T14:17:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10184.696075 secs 2024-07-26T14:17:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10184.696075 secs 2024-07-26T14:17:08.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:08 smithi134 ceph-mon[31966]: pgmap v6152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:08 smithi189 ceph-mon[36790]: pgmap v6152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:08 smithi193 ceph-mon[37473]: pgmap v6152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10170.272387 secs 2024-07-26T14:17:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10170.272387 secs 2024-07-26T14:17:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10170.272387 secs 2024-07-26T14:17:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:10 smithi134 ceph-mon[31966]: pgmap v6153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:10 smithi189 ceph-mon[36790]: pgmap v6153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:10 smithi193 ceph-mon[37473]: pgmap v6153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:11.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:17:11.074+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:17:11.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:17:11.074+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:17:11.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:17:11.074+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10189.696208 secs 2024-07-26T14:17:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:12 smithi134 ceph-mon[31966]: pgmap v6154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10189.696208 secs 2024-07-26T14:17:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:12 smithi189 ceph-mon[36790]: pgmap v6154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10189.696208 secs 2024-07-26T14:17:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:12 smithi193 ceph-mon[37473]: pgmap v6154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:14 smithi134 ceph-mon[31966]: pgmap v6155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:17:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10175.272526 secs 2024-07-26T14:17:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:14 smithi189 ceph-mon[36790]: pgmap v6155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:17:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10175.272526 secs 2024-07-26T14:17:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:14 smithi193 ceph-mon[37473]: pgmap v6155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:17:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10175.272526 secs 2024-07-26T14:17:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:16 smithi134 ceph-mon[31966]: pgmap v6156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:16 smithi189 ceph-mon[36790]: pgmap v6156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:16 smithi193 ceph-mon[37473]: pgmap v6156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10194.696336 secs 2024-07-26T14:17:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10194.696336 secs 2024-07-26T14:17:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10194.696336 secs 2024-07-26T14:17:18.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:18 smithi134 ceph-mon[31966]: pgmap v6157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:18 smithi189 ceph-mon[36790]: pgmap v6157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:18 smithi193 ceph-mon[37473]: pgmap v6157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:19.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10180.272652 secs 2024-07-26T14:17:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10180.272652 secs 2024-07-26T14:17:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10180.272652 secs 2024-07-26T14:17:20.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:20 smithi134 ceph-mon[31966]: pgmap v6158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:20 smithi189 ceph-mon[36790]: pgmap v6158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:20 smithi193 ceph-mon[37473]: pgmap v6158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:17:21.537+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:17:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:17:21.537+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:17:21.538 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:17:21.537+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10199.696466 secs 2024-07-26T14:17:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:22 smithi134 ceph-mon[31966]: pgmap v6159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10199.696466 secs 2024-07-26T14:17:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:22 smithi189 ceph-mon[36790]: pgmap v6159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10199.696466 secs 2024-07-26T14:17:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:22 smithi193 ceph-mon[37473]: pgmap v6159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:24 smithi134 ceph-mon[31966]: pgmap v6160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:17:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10185.272770 secs 2024-07-26T14:17:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:24 smithi189 ceph-mon[36790]: pgmap v6160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:17:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10185.272770 secs 2024-07-26T14:17:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:24 smithi193 ceph-mon[37473]: pgmap v6160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:17:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10185.272770 secs 2024-07-26T14:17:26.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:17:26.083+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:17:26.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:17:26.083+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:26.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:17:26.083+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:17:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:26 smithi134 ceph-mon[31966]: pgmap v6161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:26 smithi189 ceph-mon[36790]: pgmap v6161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:26 smithi193 ceph-mon[37473]: pgmap v6161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10204.696585 secs 2024-07-26T14:17:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10204.696585 secs 2024-07-26T14:17:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10204.696585 secs 2024-07-26T14:17:28.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:28 smithi134 ceph-mon[31966]: pgmap v6162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:28 smithi189 ceph-mon[36790]: pgmap v6162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:28 smithi193 ceph-mon[37473]: pgmap v6162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:17:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10190.272911 secs 2024-07-26T14:17:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10190.272911 secs 2024-07-26T14:17:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10190.272911 secs 2024-07-26T14:17:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:30 smithi134 ceph-mon[31966]: pgmap v6163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:30 smithi189 ceph-mon[36790]: pgmap v6163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:30 smithi193 ceph-mon[37473]: pgmap v6163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10209.696745 secs 2024-07-26T14:17:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10209.696745 secs 2024-07-26T14:17:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10209.696745 secs 2024-07-26T14:17:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:32 smithi134 ceph-mon[31966]: pgmap v6164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:32 smithi189 ceph-mon[36790]: pgmap v6164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:32 smithi193 ceph-mon[37473]: pgmap v6164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:34 smithi134 ceph-mon[31966]: pgmap v6165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:17:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10195.273010 secs 2024-07-26T14:17:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:34 smithi189 ceph-mon[36790]: pgmap v6165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:17:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10195.273010 secs 2024-07-26T14:17:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:34 smithi193 ceph-mon[37473]: pgmap v6165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:17:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10195.273010 secs 2024-07-26T14:17:36.553 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:17:36.552+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:17:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:17:36.552+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:17:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:17:36.552+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:36 smithi134 ceph-mon[31966]: pgmap v6166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10214.696892 secs 2024-07-26T14:17:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:36 smithi189 ceph-mon[36790]: pgmap v6166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10214.696892 secs 2024-07-26T14:17:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:36 smithi193 ceph-mon[37473]: pgmap v6166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:17:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10214.696892 secs 2024-07-26T14:17:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:38 smithi189 ceph-mon[36790]: pgmap v6167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:17:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:38 smithi193 ceph-mon[37473]: pgmap v6167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:17:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:38 smithi134 ceph-mon[31966]: pgmap v6167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:17:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10200.273135 secs 2024-07-26T14:17:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10200.273135 secs 2024-07-26T14:17:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10200.273135 secs 2024-07-26T14:17:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:40 smithi189 ceph-mon[36790]: pgmap v6168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:17:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:40 smithi193 ceph-mon[37473]: pgmap v6168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:17:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:40 smithi134 ceph-mon[31966]: pgmap v6168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:17:41.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:17:41.098+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:17:41.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:17:41.099+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:17:41.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:17:41.099+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10219.697021 secs 2024-07-26T14:17:41.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10219.697021 secs 2024-07-26T14:17:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10219.697021 secs 2024-07-26T14:17:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:42 smithi193 ceph-mon[37473]: pgmap v6169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:17:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:42 smithi134 ceph-mon[31966]: pgmap v6169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:17:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:42 smithi189 ceph-mon[36790]: pgmap v6169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:17:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:44 smithi193 ceph-mon[37473]: pgmap v6170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:17:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10205.273260 secs 2024-07-26T14:17:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:44 smithi134 ceph-mon[31966]: pgmap v6170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:17:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10205.273260 secs 2024-07-26T14:17:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:44 smithi189 ceph-mon[36790]: pgmap v6170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:17:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10205.273260 secs 2024-07-26T14:17:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:46 smithi193 ceph-mon[37473]: pgmap v6171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:17:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10224.697148 secs 2024-07-26T14:17:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:46 smithi134 ceph-mon[31966]: pgmap v6171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:17:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10224.697148 secs 2024-07-26T14:17:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:46 smithi189 ceph-mon[36790]: pgmap v6171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:17:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10224.697148 secs 2024-07-26T14:17:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:48 smithi193 ceph-mon[37473]: pgmap v6172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:17:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:48 smithi134 ceph-mon[31966]: pgmap v6172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:17:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:48 smithi189 ceph-mon[36790]: pgmap v6172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:17:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10210.273342 secs 2024-07-26T14:17:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:17:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10210.273342 secs 2024-07-26T14:17:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:17:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10210.273342 secs 2024-07-26T14:17:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:17:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:50 smithi193 ceph-mon[37473]: pgmap v6173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:17:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:17:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:50 smithi134 ceph-mon[31966]: pgmap v6173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:17:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:17:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:50 smithi189 ceph-mon[36790]: pgmap v6173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:17:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:17:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:17:51.569 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:17:51.568+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:17:51.569 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:17:51.568+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:51.569 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:17:51.568+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:17:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10229.697277 secs 2024-07-26T14:17:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:17:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:17:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:17:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10229.697277 secs 2024-07-26T14:17:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:17:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:17:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:17:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10229.697277 secs 2024-07-26T14:17:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:17:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:17:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:17:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:53 smithi193 ceph-mon[37473]: pgmap v6174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:17:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:53 smithi193 ceph-mon[37473]: pgmap v6175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:17:53.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:53 smithi134 ceph-mon[31966]: pgmap v6174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:17:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:53 smithi134 ceph-mon[31966]: pgmap v6175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:17:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:53 smithi189 ceph-mon[36790]: pgmap v6174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:17:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:53 smithi189 ceph-mon[36790]: pgmap v6175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:17:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10215.273457 secs 2024-07-26T14:17:54.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10215.273457 secs 2024-07-26T14:17:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10215.273457 secs 2024-07-26T14:17:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:55 smithi193 ceph-mon[37473]: pgmap v6176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:17:55.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:55 smithi134 ceph-mon[31966]: pgmap v6176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:17:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:55 smithi189 ceph-mon[36790]: pgmap v6176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:17:56.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:17:56.114+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:17:56.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:17:56.114+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:17:56.116 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:17:56.114+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:17:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:57 smithi193 ceph-mon[37473]: pgmap v6177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:17:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10234.697428 secs 2024-07-26T14:17:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:57 smithi134 ceph-mon[31966]: pgmap v6177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:17:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10234.697428 secs 2024-07-26T14:17:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:57 smithi189 ceph-mon[36790]: pgmap v6177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:17:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10234.697428 secs 2024-07-26T14:17:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:59 smithi134 ceph-mon[31966]: pgmap v6178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:17:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:17:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10220.273578 secs 2024-07-26T14:17:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:59 smithi189 ceph-mon[36790]: pgmap v6178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:17:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:17:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10220.273578 secs 2024-07-26T14:17:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:59 smithi193 ceph-mon[37473]: pgmap v6178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:17:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:17:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10220.273578 secs 2024-07-26T14:18:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:00 smithi134 ceph-mon[31966]: pgmap v6179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:18:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:00 smithi189 ceph-mon[36790]: pgmap v6179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:18:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:00 smithi193 ceph-mon[37473]: pgmap v6179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:18:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10239.697558 secs 2024-07-26T14:18:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:02 smithi134 ceph-mon[31966]: pgmap v6180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:18:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10239.697558 secs 2024-07-26T14:18:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:02 smithi189 ceph-mon[36790]: pgmap v6180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:18:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10239.697558 secs 2024-07-26T14:18:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:02 smithi193 ceph-mon[37473]: pgmap v6180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:18:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:04 smithi134 ceph-mon[31966]: pgmap v6181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10225.273706 secs 2024-07-26T14:18:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:04 smithi189 ceph-mon[36790]: pgmap v6181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10225.273706 secs 2024-07-26T14:18:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:04 smithi193 ceph-mon[37473]: pgmap v6181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10225.273706 secs 2024-07-26T14:18:06.352 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:06 smithi134 ceph-mon[31966]: pgmap v6182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:06 smithi189 ceph-mon[36790]: pgmap v6182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:06 smithi193 ceph-mon[37473]: pgmap v6182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:18:06.583+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:18:06.585 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:18:06.583+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:06.585 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:18:06.583+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:18:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10244.697685 secs 2024-07-26T14:18:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10244.697685 secs 2024-07-26T14:18:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10244.697685 secs 2024-07-26T14:18:08.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:08 smithi134 ceph-mon[31966]: pgmap v6183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:08 smithi189 ceph-mon[36790]: pgmap v6183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:08 smithi193 ceph-mon[37473]: pgmap v6183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:09.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10230.273827 secs 2024-07-26T14:18:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10230.273827 secs 2024-07-26T14:18:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10230.273827 secs 2024-07-26T14:18:10.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:10 smithi134 ceph-mon[31966]: pgmap v6184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:10 smithi189 ceph-mon[36790]: pgmap v6184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:10 smithi193 ceph-mon[37473]: pgmap v6184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:11.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:18:11.119+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:18:11.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:18:11.119+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:18:11.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:18:11.119+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:12.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10249.697814 secs 2024-07-26T14:18:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:12 smithi134 ceph-mon[31966]: pgmap v6185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10249.697814 secs 2024-07-26T14:18:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:12 smithi189 ceph-mon[36790]: pgmap v6185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10249.697814 secs 2024-07-26T14:18:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:12 smithi193 ceph-mon[37473]: pgmap v6185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:14 smithi134 ceph-mon[31966]: pgmap v6186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10235.273963 secs 2024-07-26T14:18:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:14 smithi189 ceph-mon[36790]: pgmap v6186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10235.273963 secs 2024-07-26T14:18:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:14 smithi193 ceph-mon[37473]: pgmap v6186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10235.273963 secs 2024-07-26T14:18:16.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:16 smithi134 ceph-mon[31966]: pgmap v6187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:16.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:16 smithi189 ceph-mon[36790]: pgmap v6187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:16 smithi193 ceph-mon[37473]: pgmap v6187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10254.697977 secs 2024-07-26T14:18:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10254.697977 secs 2024-07-26T14:18:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10254.697977 secs 2024-07-26T14:18:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:18 smithi134 ceph-mon[31966]: pgmap v6188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:18 smithi189 ceph-mon[36790]: pgmap v6188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:18.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:18 smithi193 ceph-mon[37473]: pgmap v6188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10240.274013 secs 2024-07-26T14:18:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10240.274013 secs 2024-07-26T14:18:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10240.274013 secs 2024-07-26T14:18:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:20 smithi189 ceph-mon[36790]: pgmap v6189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:20 smithi193 ceph-mon[37473]: pgmap v6189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:20.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:20 smithi134 ceph-mon[31966]: pgmap v6189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:21.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:18:21.599+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:18:21.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:18:21.599+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:21.600 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:18:21.599+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:18:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10259.698063 secs 2024-07-26T14:18:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:22 smithi189 ceph-mon[36790]: pgmap v6190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:22.474 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10259.698063 secs 2024-07-26T14:18:22.475 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:22 smithi134 ceph-mon[31966]: pgmap v6190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10259.698063 secs 2024-07-26T14:18:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:22 smithi193 ceph-mon[37473]: pgmap v6190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:24.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:18:24.112+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:18:24.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:18:24.112+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:18:24.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:18:24.112+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T14:18:24.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:18:24.112+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T14:18:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:24 smithi134 ceph-mon[31966]: pgmap v6191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10245.274101 secs 2024-07-26T14:18:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:24 smithi189 ceph-mon[36790]: pgmap v6191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10245.274101 secs 2024-07-26T14:18:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:24 smithi193 ceph-mon[37473]: pgmap v6191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10245.274101 secs 2024-07-26T14:18:26.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:18:26.134+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:18:26.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:18:26.135+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:18:26.136 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:18:26.135+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:26.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:26 smithi134 ceph-mon[31966]: pgmap v6192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:26 smithi189 ceph-mon[36790]: pgmap v6192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:26 smithi193 ceph-mon[37473]: pgmap v6192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10264.698166 secs 2024-07-26T14:18:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10264.698166 secs 2024-07-26T14:18:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10264.698166 secs 2024-07-26T14:18:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:28 smithi193 ceph-mon[37473]: pgmap v6193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:28.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:28 smithi134 ceph-mon[31966]: pgmap v6193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:28 smithi189 ceph-mon[36790]: pgmap v6193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10250.274203 secs 2024-07-26T14:18:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10250.274203 secs 2024-07-26T14:18:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10250.274203 secs 2024-07-26T14:18:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:30 smithi193 ceph-mon[37473]: pgmap v6194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:30 smithi134 ceph-mon[31966]: pgmap v6194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:30 smithi189 ceph-mon[36790]: pgmap v6194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10269.698329 secs 2024-07-26T14:18:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:32 smithi193 ceph-mon[37473]: pgmap v6195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10269.698329 secs 2024-07-26T14:18:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:32 smithi134 ceph-mon[31966]: pgmap v6195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10269.698329 secs 2024-07-26T14:18:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:32 smithi189 ceph-mon[36790]: pgmap v6195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:34 smithi193 ceph-mon[37473]: pgmap v6196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10255.274313 secs 2024-07-26T14:18:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:18:34.638+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:18:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:18:34.638+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:18:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:18:34.638+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:18:34.639 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:18:34.638+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:18:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:34 smithi134 ceph-mon[31966]: pgmap v6196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10255.274313 secs 2024-07-26T14:18:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:34 smithi189 ceph-mon[36790]: pgmap v6196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10255.274313 secs 2024-07-26T14:18:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:36 smithi193 ceph-mon[37473]: pgmap v6197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:18:36.600+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:18:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:18:36.600+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:18:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:18:36.600+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:36 smithi134 ceph-mon[31966]: pgmap v6197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:36 smithi189 ceph-mon[36790]: pgmap v6197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10274.698458 secs 2024-07-26T14:18:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10274.698458 secs 2024-07-26T14:18:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10274.698458 secs 2024-07-26T14:18:38.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:38 smithi134 ceph-mon[31966]: pgmap v6198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:38 smithi193 ceph-mon[37473]: pgmap v6198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:38 smithi189 ceph-mon[36790]: pgmap v6198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10260.274396 secs 2024-07-26T14:18:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10260.274396 secs 2024-07-26T14:18:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10260.274396 secs 2024-07-26T14:18:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:40 smithi193 ceph-mon[37473]: pgmap v6199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:40.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:40 smithi134 ceph-mon[31966]: pgmap v6199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:40 smithi189 ceph-mon[36790]: pgmap v6199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:41.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:18:41.150+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:18:41.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:18:41.150+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:41.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:18:41.150+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:18:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10279.698573 secs 2024-07-26T14:18:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:42 smithi193 ceph-mon[37473]: pgmap v6200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10279.698573 secs 2024-07-26T14:18:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:42 smithi134 ceph-mon[31966]: pgmap v6200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10279.698573 secs 2024-07-26T14:18:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:42 smithi189 ceph-mon[36790]: pgmap v6200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:44 smithi193 ceph-mon[37473]: pgmap v6201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10265.274513 secs 2024-07-26T14:18:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:44 smithi134 ceph-mon[31966]: pgmap v6201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10265.274513 secs 2024-07-26T14:18:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:44 smithi189 ceph-mon[36790]: pgmap v6201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10265.274513 secs 2024-07-26T14:18:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:46 smithi134 ceph-mon[31966]: pgmap v6202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:46 smithi189 ceph-mon[36790]: pgmap v6202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:46 smithi193 ceph-mon[37473]: pgmap v6202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10284.698691 secs 2024-07-26T14:18:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10284.698691 secs 2024-07-26T14:18:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10284.698691 secs 2024-07-26T14:18:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:48 smithi134 ceph-mon[31966]: pgmap v6203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:48 smithi189 ceph-mon[36790]: pgmap v6203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:48 smithi193 ceph-mon[37473]: pgmap v6203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:49.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10270.274655 secs 2024-07-26T14:18:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10270.274655 secs 2024-07-26T14:18:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10270.274655 secs 2024-07-26T14:18:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:50 smithi134 ceph-mon[31966]: pgmap v6204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:50 smithi189 ceph-mon[36790]: pgmap v6204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:50 smithi193 ceph-mon[37473]: pgmap v6204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:18:51.615+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:18:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:18:51.616+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:18:51.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:18:51.616+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:18:51.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:18:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:18:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10289.698812 secs 2024-07-26T14:18:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:52 smithi193 ceph-mon[37473]: pgmap v6205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:52.588 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10289.698812 secs 2024-07-26T14:18:52.588 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:52 smithi189 ceph-mon[36790]: pgmap v6205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:52.588 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:52.589 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:52.589 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:52.589 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10289.698812 secs 2024-07-26T14:18:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:52 smithi134 ceph-mon[31966]: pgmap v6205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:18:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:54 smithi193 ceph-mon[37473]: pgmap v6206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10275.274745 secs 2024-07-26T14:18:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:54 smithi134 ceph-mon[31966]: pgmap v6206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10275.274745 secs 2024-07-26T14:18:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:54 smithi189 ceph-mon[36790]: pgmap v6206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:18:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10275.274745 secs 2024-07-26T14:18:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:18:56.165+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:18:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:18:56.165+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:18:56.166 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:18:56.165+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:18:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:56 smithi134 ceph-mon[31966]: pgmap v6207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:56 smithi189 ceph-mon[36790]: pgmap v6207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:56 smithi193 ceph-mon[37473]: pgmap v6207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:18:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10294.698925 secs 2024-07-26T14:18:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:18:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:18:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:57 smithi134 ceph-mon[31966]: pgmap v6208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:18:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:57 smithi134 ceph-mon[31966]: pgmap v6209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:18:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:57.424 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10294.698925 secs 2024-07-26T14:18:57.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:18:57.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:18:57.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:57 smithi193 ceph-mon[37473]: pgmap v6208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:18:57.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:57 smithi193 ceph-mon[37473]: pgmap v6209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:18:57.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10294.698925 secs 2024-07-26T14:18:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:18:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:18:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:57 smithi189 ceph-mon[36790]: pgmap v6208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:18:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:57 smithi189 ceph-mon[36790]: pgmap v6209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:18:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:18:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:59 smithi134 ceph-mon[31966]: pgmap v6210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:18:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:18:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10280.274883 secs 2024-07-26T14:18:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:59 smithi189 ceph-mon[36790]: pgmap v6210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:18:59.487 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:18:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10280.274883 secs 2024-07-26T14:18:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:59 smithi193 ceph-mon[37473]: pgmap v6210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:18:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:18:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10280.274883 secs 2024-07-26T14:19:01.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:01 smithi134 ceph-mon[31966]: pgmap v6211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:19:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:01 smithi189 ceph-mon[36790]: pgmap v6211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:19:01.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:01 smithi193 ceph-mon[37473]: pgmap v6211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:19:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10299.699044 secs 2024-07-26T14:19:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10299.699044 secs 2024-07-26T14:19:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10299.699044 secs 2024-07-26T14:19:03.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:03 smithi134 ceph-mon[31966]: pgmap v6212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:19:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:03 smithi189 ceph-mon[36790]: pgmap v6212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:19:03.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:03 smithi193 ceph-mon[37473]: pgmap v6212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:19:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10285.275008 secs 2024-07-26T14:19:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10285.275008 secs 2024-07-26T14:19:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10285.275008 secs 2024-07-26T14:19:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:05 smithi189 ceph-mon[36790]: pgmap v6213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:19:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:05 smithi193 ceph-mon[37473]: pgmap v6213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:19:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:05 smithi134 ceph-mon[31966]: pgmap v6213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:19:06.622 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:19:06.621+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:19:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:19:06.621+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:06.623 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:19:06.621+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:19:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10304.699169 secs 2024-07-26T14:19:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:07 smithi189 ceph-mon[36790]: pgmap v6214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:19:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10304.699169 secs 2024-07-26T14:19:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:07 smithi193 ceph-mon[37473]: pgmap v6214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:19:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10304.699169 secs 2024-07-26T14:19:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:07 smithi134 ceph-mon[31966]: pgmap v6214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:19:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:09 smithi189 ceph-mon[36790]: pgmap v6215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10290.275106 secs 2024-07-26T14:19:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:09 smithi193 ceph-mon[37473]: pgmap v6215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10290.275106 secs 2024-07-26T14:19:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:09 smithi134 ceph-mon[31966]: pgmap v6215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10290.275106 secs 2024-07-26T14:19:11.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:19:11.180+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:19:11.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:19:11.180+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:11.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:19:11.180+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:19:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:11 smithi189 ceph-mon[36790]: pgmap v6216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:19:11.474 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:11 smithi134 ceph-mon[31966]: pgmap v6216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:19:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:11 smithi193 ceph-mon[37473]: pgmap v6216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:19:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10309.699299 secs 2024-07-26T14:19:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10309.699299 secs 2024-07-26T14:19:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10309.699299 secs 2024-07-26T14:19:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:13 smithi193 ceph-mon[37473]: pgmap v6217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:13.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:13 smithi134 ceph-mon[31966]: pgmap v6217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:13 smithi189 ceph-mon[36790]: pgmap v6217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10295.275224 secs 2024-07-26T14:19:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10295.275224 secs 2024-07-26T14:19:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10295.275224 secs 2024-07-26T14:19:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:15 smithi193 ceph-mon[37473]: pgmap v6218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:15 smithi134 ceph-mon[31966]: pgmap v6218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:15 smithi189 ceph-mon[36790]: pgmap v6218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10314.699406 secs 2024-07-26T14:19:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:17 smithi193 ceph-mon[37473]: pgmap v6219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10314.699406 secs 2024-07-26T14:19:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:17 smithi134 ceph-mon[31966]: pgmap v6219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10314.699406 secs 2024-07-26T14:19:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:17 smithi189 ceph-mon[36790]: pgmap v6219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:19 smithi193 ceph-mon[37473]: pgmap v6220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10300.275346 secs 2024-07-26T14:19:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:19 smithi134 ceph-mon[31966]: pgmap v6220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10300.275346 secs 2024-07-26T14:19:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:19 smithi189 ceph-mon[36790]: pgmap v6220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10300.275346 secs 2024-07-26T14:19:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:21 smithi193 ceph-mon[37473]: pgmap v6221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:19:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:19:21.634+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:19:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:19:21.634+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:21.636 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:19:21.634+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:19:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:21 smithi134 ceph-mon[31966]: pgmap v6221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:19:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:21 smithi189 ceph-mon[36790]: pgmap v6221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:19:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10319.699528 secs 2024-07-26T14:19:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10319.699528 secs 2024-07-26T14:19:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10319.699528 secs 2024-07-26T14:19:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:23 smithi193 ceph-mon[37473]: pgmap v6222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:23 smithi134 ceph-mon[31966]: pgmap v6222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:23 smithi189 ceph-mon[36790]: pgmap v6222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10305.275469 secs 2024-07-26T14:19:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10305.275469 secs 2024-07-26T14:19:24.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10305.275469 secs 2024-07-26T14:19:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:25 smithi193 ceph-mon[37473]: pgmap v6223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:25 smithi134 ceph-mon[31966]: pgmap v6223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:25 smithi189 ceph-mon[36790]: pgmap v6223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:26.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:19:26.186+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:19:26.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:19:26.186+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:19:26.187 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:19:26.186+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10324.699657 secs 2024-07-26T14:19:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:27 smithi193 ceph-mon[37473]: pgmap v6224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10324.699657 secs 2024-07-26T14:19:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:27 smithi134 ceph-mon[31966]: pgmap v6224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10324.699657 secs 2024-07-26T14:19:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:27 smithi189 ceph-mon[36790]: pgmap v6224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:29 smithi134 ceph-mon[31966]: pgmap v6225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10310.275533 secs 2024-07-26T14:19:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:29 smithi189 ceph-mon[36790]: pgmap v6225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10310.275533 secs 2024-07-26T14:19:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:29 smithi193 ceph-mon[37473]: pgmap v6225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10310.275533 secs 2024-07-26T14:19:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:31 smithi134 ceph-mon[31966]: pgmap v6226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:19:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:31 smithi189 ceph-mon[36790]: pgmap v6226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:19:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:31 smithi193 ceph-mon[37473]: pgmap v6226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:19:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10329.699752 secs 2024-07-26T14:19:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10329.699752 secs 2024-07-26T14:19:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10329.699752 secs 2024-07-26T14:19:33.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:33 smithi134 ceph-mon[31966]: pgmap v6227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:33 smithi189 ceph-mon[36790]: pgmap v6227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:33 smithi193 ceph-mon[37473]: pgmap v6227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10315.275651 secs 2024-07-26T14:19:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10315.275651 secs 2024-07-26T14:19:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10315.275651 secs 2024-07-26T14:19:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:35 smithi134 ceph-mon[31966]: pgmap v6228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:35 smithi189 ceph-mon[36790]: pgmap v6228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:35 smithi193 ceph-mon[37473]: pgmap v6228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:36.641 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:19:36.640+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:19:36.641 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:19:36.640+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:36.641 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:19:36.640+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:19:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10334.699880 secs 2024-07-26T14:19:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:37 smithi134 ceph-mon[31966]: pgmap v6229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10334.699880 secs 2024-07-26T14:19:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:37 smithi189 ceph-mon[36790]: pgmap v6229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10334.699880 secs 2024-07-26T14:19:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:37 smithi193 ceph-mon[37473]: pgmap v6229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:39 smithi134 ceph-mon[31966]: pgmap v6230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10320.275750 secs 2024-07-26T14:19:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:39 smithi189 ceph-mon[36790]: pgmap v6230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10320.275750 secs 2024-07-26T14:19:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:39 smithi193 ceph-mon[37473]: pgmap v6230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10320.275750 secs 2024-07-26T14:19:41.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:19:41.202+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:19:41.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:19:41.202+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:41.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:19:41.202+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:19:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:41 smithi134 ceph-mon[31966]: pgmap v6231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:19:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:41 smithi189 ceph-mon[36790]: pgmap v6231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:19:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:41 smithi193 ceph-mon[37473]: pgmap v6231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:19:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10339.700013 secs 2024-07-26T14:19:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10339.700013 secs 2024-07-26T14:19:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10339.700013 secs 2024-07-26T14:19:43.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:43 smithi134 ceph-mon[31966]: pgmap v6232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:43 smithi189 ceph-mon[36790]: pgmap v6232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:43 smithi193 ceph-mon[37473]: pgmap v6232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10325.275893 secs 2024-07-26T14:19:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10325.275893 secs 2024-07-26T14:19:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10325.275893 secs 2024-07-26T14:19:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:45 smithi134 ceph-mon[31966]: pgmap v6233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:45 smithi189 ceph-mon[36790]: pgmap v6233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:45 smithi193 ceph-mon[37473]: pgmap v6233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10344.700131 secs 2024-07-26T14:19:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:47 smithi134 ceph-mon[31966]: pgmap v6234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10344.700131 secs 2024-07-26T14:19:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:47 smithi189 ceph-mon[36790]: pgmap v6234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10344.700131 secs 2024-07-26T14:19:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:47 smithi193 ceph-mon[37473]: pgmap v6234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:49 smithi134 ceph-mon[31966]: pgmap v6235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10330.276029 secs 2024-07-26T14:19:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:49 smithi189 ceph-mon[36790]: pgmap v6235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10330.276029 secs 2024-07-26T14:19:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:49 smithi193 ceph-mon[37473]: pgmap v6235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10330.276029 secs 2024-07-26T14:19:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:19:51.642+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:19:51.643 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:19:51.642+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:51.644 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:19:51.642+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:19:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:51 smithi134 ceph-mon[31966]: pgmap v6236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:19:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:51 smithi189 ceph-mon[36790]: pgmap v6236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:19:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:51 smithi193 ceph-mon[37473]: pgmap v6236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:19:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10349.700278 secs 2024-07-26T14:19:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10349.700278 secs 2024-07-26T14:19:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10349.700278 secs 2024-07-26T14:19:53.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:53 smithi134 ceph-mon[31966]: pgmap v6237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:53 smithi189 ceph-mon[36790]: pgmap v6237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:53 smithi193 ceph-mon[37473]: pgmap v6237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10335.276154 secs 2024-07-26T14:19:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10335.276154 secs 2024-07-26T14:19:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10335.276154 secs 2024-07-26T14:19:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:55 smithi134 ceph-mon[31966]: pgmap v6238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:55 smithi189 ceph-mon[36790]: pgmap v6238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:55 smithi193 ceph-mon[37473]: pgmap v6238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:19:56.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:19:56.216+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:19:56.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:19:56.217+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:19:56.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:19:56.217+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:19:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10354.700408 secs 2024-07-26T14:19:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:19:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10354.700408 secs 2024-07-26T14:19:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:19:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10354.700408 secs 2024-07-26T14:19:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:19:57.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:57 smithi193 ceph-mon[37473]: pgmap v6239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:57.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:19:57.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:19:57.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:19:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:57 smithi189 ceph-mon[36790]: pgmap v6239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:19:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:19:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:19:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:57 smithi134 ceph-mon[31966]: pgmap v6239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:19:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:19:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:19:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:19:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:58 smithi189 ceph-mon[36790]: pgmap v6240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:19:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:58 smithi189 ceph-mon[36790]: pgmap v6241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:19:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:58 smithi193 ceph-mon[37473]: pgmap v6240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:19:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:58 smithi193 ceph-mon[37473]: pgmap v6241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:19:58.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:58 smithi134 ceph-mon[31966]: pgmap v6240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:19:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:58 smithi134 ceph-mon[31966]: pgmap v6241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:19:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:19:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10340.276278 secs 2024-07-26T14:19:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:19:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10340.276278 secs 2024-07-26T14:19:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:19:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10340.276278 secs 2024-07-26T14:20:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:00 smithi193 ceph-mon[37473]: pgmap v6242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:20:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:20:00.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:00 smithi134 ceph-mon[31966]: pgmap v6242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:20:00.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:20:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:00 smithi189 ceph-mon[36790]: pgmap v6242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:20:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:20:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10359.700483 secs 2024-07-26T14:20:01.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10359.700483 secs 2024-07-26T14:20:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10359.700483 secs 2024-07-26T14:20:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:02 smithi193 ceph-mon[37473]: pgmap v6243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:20:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:02 smithi134 ceph-mon[31966]: pgmap v6243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:20:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:02 smithi189 ceph-mon[36790]: pgmap v6243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:20:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:04 smithi193 ceph-mon[37473]: pgmap v6244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:20:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10345.276417 secs 2024-07-26T14:20:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:04 smithi134 ceph-mon[31966]: pgmap v6244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:20:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10345.276417 secs 2024-07-26T14:20:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:04 smithi189 ceph-mon[36790]: pgmap v6244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:20:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10345.276417 secs 2024-07-26T14:20:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:20:06.650+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:20:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:20:06.650+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:06.654 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:20:06.650+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:20:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:06 smithi193 ceph-mon[37473]: pgmap v6245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:20:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10364.700571 secs 2024-07-26T14:20:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:06 smithi134 ceph-mon[31966]: pgmap v6245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:20:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10364.700571 secs 2024-07-26T14:20:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:06 smithi189 ceph-mon[36790]: pgmap v6245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:20:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10364.700571 secs 2024-07-26T14:20:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:08 smithi193 ceph-mon[37473]: pgmap v6246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:20:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:08 smithi134 ceph-mon[31966]: pgmap v6246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:20:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:08 smithi189 ceph-mon[36790]: pgmap v6246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:20:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10350.276508 secs 2024-07-26T14:20:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10350.276508 secs 2024-07-26T14:20:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10350.276508 secs 2024-07-26T14:20:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:10 smithi193 ceph-mon[37473]: pgmap v6247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:20:10.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:10 smithi134 ceph-mon[31966]: pgmap v6247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:20:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:10 smithi189 ceph-mon[36790]: pgmap v6247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:20:11.234 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:20:11.232+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:20:11.234 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:20:11.232+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:11.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:20:11.232+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:20:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10369.700658 secs 2024-07-26T14:20:11.891 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10369.700658 secs 2024-07-26T14:20:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10369.700658 secs 2024-07-26T14:20:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:12 smithi193 ceph-mon[37473]: pgmap v6248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:20:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:12 smithi134 ceph-mon[31966]: pgmap v6248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:20:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:12 smithi189 ceph-mon[36790]: pgmap v6248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:20:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:14 smithi193 ceph-mon[37473]: pgmap v6249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:20:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10355.276654 secs 2024-07-26T14:20:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:14 smithi134 ceph-mon[31966]: pgmap v6249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:20:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10355.276654 secs 2024-07-26T14:20:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:14 smithi189 ceph-mon[36790]: pgmap v6249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:20:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10355.276654 secs 2024-07-26T14:20:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:16 smithi134 ceph-mon[31966]: pgmap v6250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10374.700749 secs 2024-07-26T14:20:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:16 smithi189 ceph-mon[36790]: pgmap v6250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10374.700749 secs 2024-07-26T14:20:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:16 smithi193 ceph-mon[37473]: pgmap v6250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10374.700749 secs 2024-07-26T14:20:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:18 smithi134 ceph-mon[31966]: pgmap v6251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:18 smithi189 ceph-mon[36790]: pgmap v6251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:18 smithi193 ceph-mon[37473]: pgmap v6251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10360.276770 secs 2024-07-26T14:20:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10360.276770 secs 2024-07-26T14:20:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10360.276770 secs 2024-07-26T14:20:21.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:21 smithi134 ceph-mon[31966]: pgmap v6252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:21 smithi189 ceph-mon[36790]: pgmap v6252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:21 smithi193 ceph-mon[37473]: pgmap v6252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:20:21.651+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:20:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:20:21.651+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:20:21.651+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:20:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10379.700832 secs 2024-07-26T14:20:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10379.700832 secs 2024-07-26T14:20:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10379.700832 secs 2024-07-26T14:20:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:23 smithi134 ceph-mon[31966]: pgmap v6253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:23 smithi189 ceph-mon[36790]: pgmap v6253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:23 smithi193 ceph-mon[37473]: pgmap v6253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:24.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10365.276915 secs 2024-07-26T14:20:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10365.276915 secs 2024-07-26T14:20:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10365.276915 secs 2024-07-26T14:20:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:25 smithi134 ceph-mon[31966]: pgmap v6254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:20:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:25 smithi189 ceph-mon[36790]: pgmap v6254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:20:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:25 smithi193 ceph-mon[37473]: pgmap v6254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:20:26.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:20:26.244+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:20:26.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:20:26.244+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:20:26.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:20:26.244+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:26.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:26 smithi134 ceph-mon[31966]: pgmap v6255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:26 smithi189 ceph-mon[36790]: pgmap v6255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:26 smithi193 ceph-mon[37473]: pgmap v6255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10384.700956 secs 2024-07-26T14:20:27.424 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10384.700956 secs 2024-07-26T14:20:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10384.700956 secs 2024-07-26T14:20:28.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:28 smithi134 ceph-mon[31966]: pgmap v6256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:28 smithi189 ceph-mon[36790]: pgmap v6256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:28 smithi193 ceph-mon[37473]: pgmap v6256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10370.277038 secs 2024-07-26T14:20:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10370.277038 secs 2024-07-26T14:20:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10370.277038 secs 2024-07-26T14:20:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:30 smithi134 ceph-mon[31966]: pgmap v6257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:30 smithi189 ceph-mon[36790]: pgmap v6257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:30 smithi193 ceph-mon[37473]: pgmap v6257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10389.701093 secs 2024-07-26T14:20:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:32 smithi134 ceph-mon[31966]: pgmap v6258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10389.701093 secs 2024-07-26T14:20:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:32 smithi189 ceph-mon[36790]: pgmap v6258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10389.701093 secs 2024-07-26T14:20:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:32 smithi193 ceph-mon[37473]: pgmap v6258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:34 smithi134 ceph-mon[31966]: pgmap v6259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:20:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10375.277107 secs 2024-07-26T14:20:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:34 smithi189 ceph-mon[36790]: pgmap v6259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:20:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10375.277107 secs 2024-07-26T14:20:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:34 smithi193 ceph-mon[37473]: pgmap v6259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:20:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10375.277107 secs 2024-07-26T14:20:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:20:36.667+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:20:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:20:36.667+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:20:36.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:20:36.667+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:36 smithi134 ceph-mon[31966]: pgmap v6260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10394.701188 secs 2024-07-26T14:20:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:36 smithi189 ceph-mon[36790]: pgmap v6260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10394.701188 secs 2024-07-26T14:20:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:36 smithi193 ceph-mon[37473]: pgmap v6260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10394.701188 secs 2024-07-26T14:20:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:38 smithi189 ceph-mon[36790]: pgmap v6261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:38 smithi193 ceph-mon[37473]: pgmap v6261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:38 smithi134 ceph-mon[31966]: pgmap v6261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:39.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:20:39.135+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:20:39.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:20:39.135+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:20:39.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:20:39.136+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T14:20:39.137 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:20:39.136+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T14:20:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10380.277230 secs 2024-07-26T14:20:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10380.277230 secs 2024-07-26T14:20:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10380.277230 secs 2024-07-26T14:20:41.260 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:20:41.259+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:20:41.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:20:41.259+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:20:41.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:20:41.259+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:41.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:41 smithi134 ceph-mon[31966]: pgmap v6262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:41 smithi189 ceph-mon[36790]: pgmap v6262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:41 smithi193 ceph-mon[37473]: pgmap v6262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10399.701329 secs 2024-07-26T14:20:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10399.701329 secs 2024-07-26T14:20:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10399.701329 secs 2024-07-26T14:20:43.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:43 smithi134 ceph-mon[31966]: pgmap v6263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:43 smithi189 ceph-mon[36790]: pgmap v6263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:43 smithi193 ceph-mon[37473]: pgmap v6263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:44 smithi134 ceph-mon[31966]: pgmap v6264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:20:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10385.277353 secs 2024-07-26T14:20:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:44 smithi189 ceph-mon[36790]: pgmap v6264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:20:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10385.277353 secs 2024-07-26T14:20:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:44 smithi193 ceph-mon[37473]: pgmap v6264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:20:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10385.277353 secs 2024-07-26T14:20:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:46 smithi134 ceph-mon[31966]: pgmap v6265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10404.701450 secs 2024-07-26T14:20:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:46 smithi189 ceph-mon[36790]: pgmap v6265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10404.701450 secs 2024-07-26T14:20:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:46 smithi193 ceph-mon[37473]: pgmap v6265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10404.701450 secs 2024-07-26T14:20:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:48 smithi189 ceph-mon[36790]: pgmap v6266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:48 smithi193 ceph-mon[37473]: pgmap v6266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:48 smithi134 ceph-mon[31966]: pgmap v6266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:49.663 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:20:49.662+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:20:49.663 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:20:49.662+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:20:49.664 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:20:49.663+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:20:49.664 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:20:49.663+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:20:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10390.277472 secs 2024-07-26T14:20:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10390.277472 secs 2024-07-26T14:20:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10390.277472 secs 2024-07-26T14:20:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:50 smithi189 ceph-mon[36790]: pgmap v6267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:50 smithi193 ceph-mon[37473]: pgmap v6267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:50 smithi134 ceph-mon[31966]: pgmap v6267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:20:51.680 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:20:51.679+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:20:51.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:20:51.679+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:20:51.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:20:51.679+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10409.701568 secs 2024-07-26T14:20:51.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10409.701568 secs 2024-07-26T14:20:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10409.701568 secs 2024-07-26T14:20:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:52 smithi193 ceph-mon[37473]: pgmap v6268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:52 smithi134 ceph-mon[31966]: pgmap v6268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:52 smithi189 ceph-mon[36790]: pgmap v6268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:54.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:54 smithi189 ceph-mon[36790]: pgmap v6269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:20:54.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10395.277628 secs 2024-07-26T14:20:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:54 smithi193 ceph-mon[37473]: pgmap v6269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:20:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10395.277628 secs 2024-07-26T14:20:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:54 smithi134 ceph-mon[31966]: pgmap v6269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:20:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10395.277628 secs 2024-07-26T14:20:56.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:20:56.260+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:20:56.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:20:56.260+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:20:56.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:20:56.260+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:20:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:56 smithi134 ceph-mon[31966]: pgmap v6270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10414.701679 secs 2024-07-26T14:20:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:56 smithi193 ceph-mon[37473]: pgmap v6270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10414.701679 secs 2024-07-26T14:20:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:56 smithi189 ceph-mon[36790]: pgmap v6270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:20:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10414.701679 secs 2024-07-26T14:20:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:58 smithi193 ceph-mon[37473]: pgmap v6271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:20:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:58 smithi134 ceph-mon[31966]: pgmap v6271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:20:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:58 smithi189 ceph-mon[36790]: pgmap v6271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:20:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:20:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10400.277789 secs 2024-07-26T14:20:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:20:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:20:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:59 smithi193 ceph-mon[37473]: pgmap v6272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:20:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:20:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:20:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10400.277789 secs 2024-07-26T14:20:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:20:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:20:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:59 smithi134 ceph-mon[31966]: pgmap v6272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:20:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:20:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:20:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10400.277789 secs 2024-07-26T14:20:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:20:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:20:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:59 smithi189 ceph-mon[36790]: pgmap v6272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:20:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:20:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:21:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:01 smithi134 ceph-mon[31966]: pgmap v6273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:21:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10419.701766 secs 2024-07-26T14:21:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:01 smithi189 ceph-mon[36790]: pgmap v6273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:21:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10419.701766 secs 2024-07-26T14:21:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:01 smithi193 ceph-mon[37473]: pgmap v6273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:21:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10419.701766 secs 2024-07-26T14:21:03.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:03 smithi134 ceph-mon[31966]: pgmap v6274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:21:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:03 smithi189 ceph-mon[36790]: pgmap v6274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:21:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:03 smithi193 ceph-mon[37473]: pgmap v6274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:21:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10405.277925 secs 2024-07-26T14:21:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10405.277925 secs 2024-07-26T14:21:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10405.277925 secs 2024-07-26T14:21:05.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:05 smithi134 ceph-mon[31966]: pgmap v6275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:21:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:05 smithi189 ceph-mon[36790]: pgmap v6275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:21:06.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:05 smithi193 ceph-mon[37473]: pgmap v6275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:21:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:21:06.695+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:21:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:21:06.695+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:21:06.696 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:21:06.695+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10424.701916 secs 2024-07-26T14:21:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10424.701916 secs 2024-07-26T14:21:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10424.701916 secs 2024-07-26T14:21:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:07 smithi134 ceph-mon[31966]: pgmap v6276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:21:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:07 smithi189 ceph-mon[36790]: pgmap v6276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:21:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:07 smithi193 ceph-mon[37473]: pgmap v6276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:21:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10410.278023 secs 2024-07-26T14:21:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:09 smithi134 ceph-mon[31966]: pgmap v6277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:21:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10410.278023 secs 2024-07-26T14:21:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:09 smithi189 ceph-mon[36790]: pgmap v6277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:21:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10410.278023 secs 2024-07-26T14:21:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:09 smithi193 ceph-mon[37473]: pgmap v6277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:21:11.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:21:11.276+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:21:11.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:21:11.276+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:21:11.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:21:11.276+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:11 smithi134 ceph-mon[31966]: pgmap v6278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10429.702041 secs 2024-07-26T14:21:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:11 smithi189 ceph-mon[36790]: pgmap v6278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10429.702041 secs 2024-07-26T14:21:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:11 smithi193 ceph-mon[37473]: pgmap v6278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10429.702041 secs 2024-07-26T14:21:13.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:13 smithi134 ceph-mon[31966]: pgmap v6279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:13 smithi189 ceph-mon[36790]: pgmap v6279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:13 smithi193 ceph-mon[37473]: pgmap v6279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10415.278123 secs 2024-07-26T14:21:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10415.278123 secs 2024-07-26T14:21:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10415.278123 secs 2024-07-26T14:21:15.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:15 smithi134 ceph-mon[31966]: pgmap v6280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:15 smithi189 ceph-mon[36790]: pgmap v6280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:15 smithi193 ceph-mon[37473]: pgmap v6280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10434.702180 secs 2024-07-26T14:21:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10434.702180 secs 2024-07-26T14:21:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10434.702180 secs 2024-07-26T14:21:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:17 smithi189 ceph-mon[36790]: pgmap v6281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:17 smithi193 ceph-mon[37473]: pgmap v6281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:17 smithi134 ceph-mon[31966]: pgmap v6281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10420.278252 secs 2024-07-26T14:21:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:19 smithi189 ceph-mon[36790]: pgmap v6282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10420.278252 secs 2024-07-26T14:21:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:19 smithi193 ceph-mon[37473]: pgmap v6282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:20.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10420.278252 secs 2024-07-26T14:21:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:19 smithi134 ceph-mon[31966]: pgmap v6282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:21.711 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:21:21.710+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:21:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:21:21.710+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:21.712 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:21:21.710+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:21:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:21 smithi189 ceph-mon[36790]: pgmap v6283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10439.702297 secs 2024-07-26T14:21:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:21 smithi193 ceph-mon[37473]: pgmap v6283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10439.702297 secs 2024-07-26T14:21:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:21 smithi134 ceph-mon[31966]: pgmap v6283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10439.702297 secs 2024-07-26T14:21:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:23 smithi189 ceph-mon[36790]: pgmap v6284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:23 smithi193 ceph-mon[37473]: pgmap v6284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:24.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:21:24.144+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:21:24.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:21:24.144+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:21:24.146 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:23 smithi134 ceph-mon[31966]: pgmap v6284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:24.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:21:24.144+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T14:21:24.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:21:24.144+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 83 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T14:21:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10425.278375 secs 2024-07-26T14:21:25.124 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10425.278375 secs 2024-07-26T14:21:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10425.278375 secs 2024-07-26T14:21:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:25 smithi193 ceph-mon[37473]: pgmap v6285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:26.055 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:25 smithi189 ceph-mon[36790]: pgmap v6285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:26.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:25 smithi134 ceph-mon[31966]: pgmap v6285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:26.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:21:26.291+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:21:26.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:21:26.291+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:26.293 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:21:26.291+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:21:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10444.702416 secs 2024-07-26T14:21:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10444.702416 secs 2024-07-26T14:21:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10444.702416 secs 2024-07-26T14:21:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:27 smithi193 ceph-mon[37473]: pgmap v6286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:28.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:27 smithi134 ceph-mon[31966]: pgmap v6286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:27 smithi189 ceph-mon[36790]: pgmap v6286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10430.278512 secs 2024-07-26T14:21:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:29 smithi193 ceph-mon[37473]: pgmap v6287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10430.278512 secs 2024-07-26T14:21:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:29 smithi134 ceph-mon[31966]: pgmap v6287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10430.278512 secs 2024-07-26T14:21:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:29 smithi189 ceph-mon[36790]: pgmap v6287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:31 smithi193 ceph-mon[37473]: pgmap v6288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10449.702483 secs 2024-07-26T14:21:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:31 smithi134 ceph-mon[31966]: pgmap v6288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10449.702483 secs 2024-07-26T14:21:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:31 smithi189 ceph-mon[36790]: pgmap v6288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10449.702483 secs 2024-07-26T14:21:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:33 smithi193 ceph-mon[37473]: pgmap v6289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:33 smithi134 ceph-mon[31966]: pgmap v6289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:33 smithi189 ceph-mon[36790]: pgmap v6289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:34.671 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:21:34.669+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:21:34.671 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:21:34.670+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:21:34.671 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:21:34.671+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:21:34.672 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:21:34.671+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:21:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10435.278651 secs 2024-07-26T14:21:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10435.278651 secs 2024-07-26T14:21:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10435.278651 secs 2024-07-26T14:21:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:35 smithi189 ceph-mon[36790]: pgmap v6290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:35 smithi193 ceph-mon[37473]: pgmap v6290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:36.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:35 smithi134 ceph-mon[31966]: pgmap v6290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:21:36.726+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:21:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:21:36.726+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:21:36.727 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:21:36.726+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10454.702604 secs 2024-07-26T14:21:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10454.702604 secs 2024-07-26T14:21:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10454.702604 secs 2024-07-26T14:21:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:37 smithi193 ceph-mon[37473]: pgmap v6291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:38.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:37 smithi134 ceph-mon[31966]: pgmap v6291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:37 smithi189 ceph-mon[36790]: pgmap v6291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10440.278778 secs 2024-07-26T14:21:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:39 smithi193 ceph-mon[37473]: pgmap v6292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:40.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10440.278778 secs 2024-07-26T14:21:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:39 smithi134 ceph-mon[31966]: pgmap v6292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10440.278778 secs 2024-07-26T14:21:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:39 smithi189 ceph-mon[36790]: pgmap v6292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:41.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:21:41.300+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:21:41.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:21:41.300+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:41.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:21:41.300+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:21:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:41 smithi134 ceph-mon[31966]: pgmap v6293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10459.702711 secs 2024-07-26T14:21:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:41 smithi189 ceph-mon[36790]: pgmap v6293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10459.702711 secs 2024-07-26T14:21:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:41 smithi193 ceph-mon[37473]: pgmap v6293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10459.702711 secs 2024-07-26T14:21:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:43 smithi134 ceph-mon[31966]: pgmap v6294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:43 smithi189 ceph-mon[36790]: pgmap v6294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:43 smithi193 ceph-mon[37473]: pgmap v6294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10445.278911 secs 2024-07-26T14:21:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10445.278911 secs 2024-07-26T14:21:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10445.278911 secs 2024-07-26T14:21:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:45 smithi134 ceph-mon[31966]: pgmap v6295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:45 smithi189 ceph-mon[36790]: pgmap v6295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:45 smithi193 ceph-mon[37473]: pgmap v6295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:47.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10464.702792 secs 2024-07-26T14:21:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10464.702792 secs 2024-07-26T14:21:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10464.702792 secs 2024-07-26T14:21:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:47 smithi134 ceph-mon[31966]: pgmap v6296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:47 smithi189 ceph-mon[36790]: pgmap v6296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:47 smithi193 ceph-mon[37473]: pgmap v6296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10450.279041 secs 2024-07-26T14:21:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:49 smithi134 ceph-mon[31966]: pgmap v6297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10450.279041 secs 2024-07-26T14:21:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:49 smithi189 ceph-mon[36790]: pgmap v6297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10450.279041 secs 2024-07-26T14:21:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:49 smithi193 ceph-mon[37473]: pgmap v6297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:51.743 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:21:51.742+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:21:51.743 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:21:51.742+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:51.743 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:21:51.742+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:21:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:51 smithi134 ceph-mon[31966]: pgmap v6298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10469.702943 secs 2024-07-26T14:21:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:51 smithi189 ceph-mon[36790]: pgmap v6298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10469.702943 secs 2024-07-26T14:21:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:51 smithi193 ceph-mon[37473]: pgmap v6298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10469.702943 secs 2024-07-26T14:21:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:53 smithi134 ceph-mon[31966]: pgmap v6299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:53 smithi189 ceph-mon[36790]: pgmap v6299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:53 smithi193 ceph-mon[37473]: pgmap v6299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:21:55.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10455.279193 secs 2024-07-26T14:21:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10455.279193 secs 2024-07-26T14:21:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10455.279193 secs 2024-07-26T14:21:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:55 smithi134 ceph-mon[31966]: pgmap v6300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:55 smithi189 ceph-mon[36790]: pgmap v6300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:55 smithi193 ceph-mon[37473]: pgmap v6300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:21:56.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:21:56.306+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:21:56.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:21:56.306+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:21:56.308 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:21:56.306+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:21:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10474.703086 secs 2024-07-26T14:21:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10474.703086 secs 2024-07-26T14:21:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10474.703086 secs 2024-07-26T14:21:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:57 smithi134 ceph-mon[31966]: pgmap v6301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:57 smithi189 ceph-mon[36790]: pgmap v6301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:57 smithi193 ceph-mon[37473]: pgmap v6301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:21:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:21:59.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:21:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:22:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10460.279335 secs 2024-07-26T14:22:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:59 smithi134 ceph-mon[31966]: pgmap v6302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:22:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:22:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:22:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:21:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:22:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10460.279335 secs 2024-07-26T14:22:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:59 smithi189 ceph-mon[36790]: pgmap v6302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:22:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:22:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:22:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:21:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:22:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10460.279335 secs 2024-07-26T14:22:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:59 smithi193 ceph-mon[37473]: pgmap v6302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:22:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:22:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:22:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:21:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:22:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:00 smithi134 ceph-mon[31966]: pgmap v6303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:22:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:00 smithi134 ceph-mon[31966]: pgmap v6304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:22:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:00 smithi189 ceph-mon[36790]: pgmap v6303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:22:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:00 smithi189 ceph-mon[36790]: pgmap v6304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:22:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:00 smithi193 ceph-mon[37473]: pgmap v6303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T14:22:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:00 smithi193 ceph-mon[37473]: pgmap v6304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:22:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10479.703204 secs 2024-07-26T14:22:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10479.703204 secs 2024-07-26T14:22:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10479.703204 secs 2024-07-26T14:22:03.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:02 smithi134 ceph-mon[31966]: pgmap v6305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-07-26T14:22:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:02 smithi189 ceph-mon[36790]: pgmap v6305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-07-26T14:22:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:02 smithi193 ceph-mon[37473]: pgmap v6305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-07-26T14:22:04.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10465.279473 secs 2024-07-26T14:22:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10465.279473 secs 2024-07-26T14:22:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10465.279473 secs 2024-07-26T14:22:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:04 smithi134 ceph-mon[31966]: pgmap v6306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:22:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:04 smithi189 ceph-mon[36790]: pgmap v6306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:22:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:04 smithi193 ceph-mon[37473]: pgmap v6306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:22:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:22:06.757+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:22:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:22:06.757+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:06.758 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:22:06.757+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:22:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:06 smithi189 ceph-mon[36790]: pgmap v6307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:22:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10484.703315 secs 2024-07-26T14:22:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:06 smithi193 ceph-mon[37473]: pgmap v6307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:22:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10484.703315 secs 2024-07-26T14:22:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:06 smithi134 ceph-mon[31966]: pgmap v6307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:22:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10484.703315 secs 2024-07-26T14:22:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:08 smithi189 ceph-mon[36790]: pgmap v6308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:22:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10470.279617 secs 2024-07-26T14:22:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:08 smithi193 ceph-mon[37473]: pgmap v6308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:22:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10470.279617 secs 2024-07-26T14:22:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:08 smithi134 ceph-mon[31966]: pgmap v6308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:22:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10470.279617 secs 2024-07-26T14:22:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:10 smithi189 ceph-mon[36790]: pgmap v6309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:22:11.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:10 smithi134 ceph-mon[31966]: pgmap v6309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:22:11.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:10 smithi193 ceph-mon[37473]: pgmap v6309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:22:11.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:22:11.316+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:22:11.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:22:11.316+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:11.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:22:11.316+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:22:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10489.703459 secs 2024-07-26T14:22:12.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10489.703459 secs 2024-07-26T14:22:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10489.703459 secs 2024-07-26T14:22:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:12 smithi193 ceph-mon[37473]: pgmap v6310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:13.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:12 smithi134 ceph-mon[31966]: pgmap v6310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:12 smithi189 ceph-mon[36790]: pgmap v6310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10475.279753 secs 2024-07-26T14:22:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10475.279753 secs 2024-07-26T14:22:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10475.279753 secs 2024-07-26T14:22:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:14 smithi193 ceph-mon[37473]: pgmap v6311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:14 smithi134 ceph-mon[31966]: pgmap v6311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:14 smithi189 ceph-mon[36790]: pgmap v6311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:16 smithi193 ceph-mon[37473]: pgmap v6312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10494.703532 secs 2024-07-26T14:22:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:16 smithi134 ceph-mon[31966]: pgmap v6312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10494.703532 secs 2024-07-26T14:22:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:16 smithi189 ceph-mon[36790]: pgmap v6312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10494.703532 secs 2024-07-26T14:22:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:18 smithi193 ceph-mon[37473]: pgmap v6313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10480.279878 secs 2024-07-26T14:22:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:18 smithi134 ceph-mon[31966]: pgmap v6313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10480.279878 secs 2024-07-26T14:22:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:18 smithi189 ceph-mon[36790]: pgmap v6313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10480.279878 secs 2024-07-26T14:22:21.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:20 smithi193 ceph-mon[37473]: pgmap v6314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:22:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:20 smithi134 ceph-mon[31966]: pgmap v6314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:22:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:20 smithi189 ceph-mon[36790]: pgmap v6314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:22:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:22:21.773+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:22:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:22:21.773+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:21.774 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:22:21.773+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:22:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10499.703617 secs 2024-07-26T14:22:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10499.703617 secs 2024-07-26T14:22:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10499.703617 secs 2024-07-26T14:22:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:22 smithi193 ceph-mon[37473]: pgmap v6315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:23.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:22 smithi134 ceph-mon[31966]: pgmap v6315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:22 smithi189 ceph-mon[36790]: pgmap v6315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10485.280020 secs 2024-07-26T14:22:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10485.280020 secs 2024-07-26T14:22:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10485.280020 secs 2024-07-26T14:22:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:25 smithi193 ceph-mon[37473]: pgmap v6316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:25.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:25 smithi189 ceph-mon[36790]: pgmap v6316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:25 smithi134 ceph-mon[31966]: pgmap v6316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:26.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:22:26.329+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:22:26.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:22:26.329+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:26.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:22:26.329+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:22:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:27 smithi193 ceph-mon[37473]: pgmap v6317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10504.703711 secs 2024-07-26T14:22:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:27 smithi134 ceph-mon[31966]: pgmap v6317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10504.703711 secs 2024-07-26T14:22:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:27 smithi189 ceph-mon[36790]: pgmap v6317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10504.703711 secs 2024-07-26T14:22:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:29 smithi193 ceph-mon[37473]: pgmap v6318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10490.280084 secs 2024-07-26T14:22:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:29 smithi134 ceph-mon[31966]: pgmap v6318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10490.280084 secs 2024-07-26T14:22:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:29 smithi189 ceph-mon[36790]: pgmap v6318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10490.280084 secs 2024-07-26T14:22:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:31 smithi134 ceph-mon[31966]: pgmap v6319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:22:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:31 smithi189 ceph-mon[36790]: pgmap v6319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:22:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:31 smithi193 ceph-mon[37473]: pgmap v6319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:22:32.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10509.703838 secs 2024-07-26T14:22:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10509.703838 secs 2024-07-26T14:22:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10509.703838 secs 2024-07-26T14:22:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:33 smithi134 ceph-mon[31966]: pgmap v6320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:33 smithi189 ceph-mon[36790]: pgmap v6320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:33 smithi193 ceph-mon[37473]: pgmap v6320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10495.280198 secs 2024-07-26T14:22:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10495.280198 secs 2024-07-26T14:22:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10495.280198 secs 2024-07-26T14:22:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:35 smithi134 ceph-mon[31966]: pgmap v6321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:35 smithi189 ceph-mon[36790]: pgmap v6321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:35 smithi193 ceph-mon[37473]: pgmap v6321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:22:36.789+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:22:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:22:36.789+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:36.789 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:22:36.789+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:22:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:37 smithi134 ceph-mon[31966]: pgmap v6322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10514.703950 secs 2024-07-26T14:22:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:37 smithi189 ceph-mon[36790]: pgmap v6322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10514.703950 secs 2024-07-26T14:22:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:37 smithi193 ceph-mon[37473]: pgmap v6322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10514.703950 secs 2024-07-26T14:22:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:39 smithi134 ceph-mon[31966]: pgmap v6323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10500.280290 secs 2024-07-26T14:22:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:39 smithi189 ceph-mon[36790]: pgmap v6323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10500.280290 secs 2024-07-26T14:22:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:39 smithi193 ceph-mon[37473]: pgmap v6323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10500.280290 secs 2024-07-26T14:22:41.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:22:41.344+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:22:41.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:22:41.345+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:22:41.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:22:41.345+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:41.347 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:41 smithi134 ceph-mon[31966]: pgmap v6324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:22:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:41 smithi189 ceph-mon[36790]: pgmap v6324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:22:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:41 smithi193 ceph-mon[37473]: pgmap v6324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:22:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10519.704026 secs 2024-07-26T14:22:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10519.704026 secs 2024-07-26T14:22:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10519.704026 secs 2024-07-26T14:22:43.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:43 smithi134 ceph-mon[31966]: pgmap v6325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:43 smithi189 ceph-mon[36790]: pgmap v6325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:43 smithi193 ceph-mon[37473]: pgmap v6325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:44.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10505.280367 secs 2024-07-26T14:22:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10505.280367 secs 2024-07-26T14:22:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10505.280367 secs 2024-07-26T14:22:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:45 smithi134 ceph-mon[31966]: pgmap v6326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:45 smithi189 ceph-mon[36790]: pgmap v6326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:45 smithi193 ceph-mon[37473]: pgmap v6326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:46.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:46 smithi134 ceph-mon[31966]: pgmap v6327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:46 smithi189 ceph-mon[36790]: pgmap v6327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:46 smithi193 ceph-mon[37473]: pgmap v6327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10524.704150 secs 2024-07-26T14:22:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10524.704150 secs 2024-07-26T14:22:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10524.704150 secs 2024-07-26T14:22:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:48 smithi134 ceph-mon[31966]: pgmap v6328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:48 smithi189 ceph-mon[36790]: pgmap v6328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:48 smithi193 ceph-mon[37473]: pgmap v6328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10510.280508 secs 2024-07-26T14:22:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10510.280508 secs 2024-07-26T14:22:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10510.280508 secs 2024-07-26T14:22:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:50 smithi134 ceph-mon[31966]: pgmap v6329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:22:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:50 smithi189 ceph-mon[36790]: pgmap v6329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:22:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:50 smithi193 ceph-mon[37473]: pgmap v6329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:22:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10529.704294 secs 2024-07-26T14:22:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:22:51.803+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:22:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:22:51.803+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:22:51.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:22:51.804+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:51.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10529.704294 secs 2024-07-26T14:22:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10529.704294 secs 2024-07-26T14:22:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:52 smithi193 ceph-mon[37473]: pgmap v6330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:52 smithi134 ceph-mon[31966]: pgmap v6330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:52 smithi189 ceph-mon[36790]: pgmap v6330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10515.280626 secs 2024-07-26T14:22:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:54 smithi134 ceph-mon[31966]: pgmap v6331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10515.280626 secs 2024-07-26T14:22:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:54 smithi189 ceph-mon[36790]: pgmap v6331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10515.280626 secs 2024-07-26T14:22:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:54 smithi193 ceph-mon[37473]: pgmap v6331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:56.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:22:56.360+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:22:56.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:22:56.360+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:22:56.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:22:56.360+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:22:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:56 smithi134 ceph-mon[31966]: pgmap v6332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10534.704408 secs 2024-07-26T14:22:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:56 smithi189 ceph-mon[36790]: pgmap v6332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10534.704408 secs 2024-07-26T14:22:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:56 smithi193 ceph-mon[37473]: pgmap v6332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:22:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10534.704408 secs 2024-07-26T14:22:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:58 smithi134 ceph-mon[31966]: pgmap v6333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:58 smithi189 ceph-mon[36790]: pgmap v6333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:22:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:58 smithi193 ceph-mon[37473]: pgmap v6333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:23:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:22:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10520.280702 secs 2024-07-26T14:23:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:22:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10520.280702 secs 2024-07-26T14:23:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:22:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10520.280702 secs 2024-07-26T14:23:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:00 smithi134 ceph-mon[31966]: pgmap v6334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:23:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:23:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:00 smithi189 ceph-mon[36790]: pgmap v6334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:23:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:23:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:00 smithi193 ceph-mon[37473]: pgmap v6334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:23:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:23:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:23:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:23:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:01 smithi134 ceph-mon[31966]: pgmap v6335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:23:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:01 smithi134 ceph-mon[31966]: pgmap v6336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:23:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:23:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10539.704533 secs 2024-07-26T14:23:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:23:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:23:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:01 smithi189 ceph-mon[36790]: pgmap v6335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:23:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:01 smithi189 ceph-mon[36790]: pgmap v6336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:23:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:23:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10539.704533 secs 2024-07-26T14:23:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:23:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:23:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:01 smithi193 ceph-mon[37473]: pgmap v6335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:23:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:01 smithi193 ceph-mon[37473]: pgmap v6336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:23:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:23:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10539.704533 secs 2024-07-26T14:23:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:04 smithi193 ceph-mon[37473]: pgmap v6337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:23:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10525.280837 secs 2024-07-26T14:23:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:04 smithi134 ceph-mon[31966]: pgmap v6337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:23:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10525.280837 secs 2024-07-26T14:23:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:04 smithi189 ceph-mon[36790]: pgmap v6337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:23:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10525.280837 secs 2024-07-26T14:23:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:06 smithi193 ceph-mon[37473]: pgmap v6338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:23:06.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:06 smithi134 ceph-mon[31966]: pgmap v6338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:23:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:06 smithi189 ceph-mon[36790]: pgmap v6338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:23:06.809 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:23:06.808+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:23:06.809 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:23:06.808+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:23:06.809 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:23:06.808+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10544.704659 secs 2024-07-26T14:23:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10544.704659 secs 2024-07-26T14:23:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10544.704659 secs 2024-07-26T14:23:08.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:08 smithi134 ceph-mon[31966]: pgmap v6339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:23:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:08 smithi189 ceph-mon[36790]: pgmap v6339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:23:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:08 smithi193 ceph-mon[37473]: pgmap v6339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:23:09.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10530.280965 secs 2024-07-26T14:23:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10530.280965 secs 2024-07-26T14:23:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10530.280965 secs 2024-07-26T14:23:10.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:10 smithi134 ceph-mon[31966]: pgmap v6340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:23:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:10 smithi189 ceph-mon[36790]: pgmap v6340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:23:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:10 smithi193 ceph-mon[37473]: pgmap v6340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:23:11.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:23:11.371+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:23:11.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:23:11.371+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:11.372 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:23:11.371+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:23:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:12 smithi134 ceph-mon[31966]: pgmap v6341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:23:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10549.704770 secs 2024-07-26T14:23:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:12 smithi189 ceph-mon[36790]: pgmap v6341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:23:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10549.704770 secs 2024-07-26T14:23:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:12 smithi193 ceph-mon[37473]: pgmap v6341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:23:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10549.704770 secs 2024-07-26T14:23:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:14 smithi134 ceph-mon[31966]: pgmap v6342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:23:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10535.281060 secs 2024-07-26T14:23:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:14 smithi189 ceph-mon[36790]: pgmap v6342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:23:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10535.281060 secs 2024-07-26T14:23:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:14 smithi193 ceph-mon[37473]: pgmap v6342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:23:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10535.281060 secs 2024-07-26T14:23:16.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:16 smithi134 ceph-mon[31966]: pgmap v6343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:16 smithi189 ceph-mon[36790]: pgmap v6343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:16 smithi193 ceph-mon[37473]: pgmap v6343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10554.704842 secs 2024-07-26T14:23:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10554.704842 secs 2024-07-26T14:23:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10554.704842 secs 2024-07-26T14:23:18.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:18 smithi134 ceph-mon[31966]: pgmap v6344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:18 smithi189 ceph-mon[36790]: pgmap v6344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:18.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:18 smithi193 ceph-mon[37473]: pgmap v6344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10540.281186 secs 2024-07-26T14:23:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10540.281186 secs 2024-07-26T14:23:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10540.281186 secs 2024-07-26T14:23:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:20 smithi134 ceph-mon[31966]: pgmap v6345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:20 smithi189 ceph-mon[36790]: pgmap v6345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:20 smithi193 ceph-mon[37473]: pgmap v6345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:21.825 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:23:21.823+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:23:21.825 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:23:21.824+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:23:21.825 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:23:21.824+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:22 smithi134 ceph-mon[31966]: pgmap v6346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10559.704917 secs 2024-07-26T14:23:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:22 smithi189 ceph-mon[36790]: pgmap v6346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10559.704917 secs 2024-07-26T14:23:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:22 smithi193 ceph-mon[37473]: pgmap v6346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10559.704917 secs 2024-07-26T14:23:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:23 smithi134 ceph-mon[31966]: pgmap v6347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:23:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:23 smithi189 ceph-mon[36790]: pgmap v6347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:23:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:23 smithi193 ceph-mon[37473]: pgmap v6347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:23:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10545.281310 secs 2024-07-26T14:23:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10545.281310 secs 2024-07-26T14:23:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10545.281310 secs 2024-07-26T14:23:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:25 smithi134 ceph-mon[31966]: pgmap v6348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:23:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:25 smithi189 ceph-mon[36790]: pgmap v6348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:23:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:25 smithi193 ceph-mon[37473]: pgmap v6348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:23:26.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:23:26.384+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:23:26.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:23:26.384+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:26.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:23:26.384+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:23:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10564.705036 secs 2024-07-26T14:23:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10564.705036 secs 2024-07-26T14:23:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10564.705036 secs 2024-07-26T14:23:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:27 smithi193 ceph-mon[37473]: pgmap v6349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:23:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:27 smithi134 ceph-mon[31966]: pgmap v6349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:23:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:27 smithi189 ceph-mon[36790]: pgmap v6349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:23:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10550.281424 secs 2024-07-26T14:23:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10550.281424 secs 2024-07-26T14:23:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10550.281424 secs 2024-07-26T14:23:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:29 smithi193 ceph-mon[37473]: pgmap v6350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:23:30.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:29 smithi134 ceph-mon[31966]: pgmap v6350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:23:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:29 smithi189 ceph-mon[36790]: pgmap v6350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:23:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:32 smithi134 ceph-mon[31966]: pgmap v6351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:23:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10569.705157 secs 2024-07-26T14:23:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:32 smithi189 ceph-mon[36790]: pgmap v6351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:23:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10569.705157 secs 2024-07-26T14:23:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:32 smithi193 ceph-mon[37473]: pgmap v6351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:23:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10569.705157 secs 2024-07-26T14:23:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:34 smithi134 ceph-mon[31966]: pgmap v6352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:23:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10555.281554 secs 2024-07-26T14:23:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:34 smithi189 ceph-mon[36790]: pgmap v6352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:23:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10555.281554 secs 2024-07-26T14:23:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:34 smithi193 ceph-mon[37473]: pgmap v6352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:23:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10555.281554 secs 2024-07-26T14:23:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:36 smithi134 ceph-mon[31966]: pgmap v6353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:36 smithi189 ceph-mon[36790]: pgmap v6353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:36 smithi193 ceph-mon[37473]: pgmap v6353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:36.840 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:23:36.839+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:23:36.841 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:23:36.839+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:23:36.841 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:23:36.839+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10574.705269 secs 2024-07-26T14:23:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10574.705269 secs 2024-07-26T14:23:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10574.705269 secs 2024-07-26T14:23:38.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:38 smithi134 ceph-mon[31966]: pgmap v6354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:38 smithi189 ceph-mon[36790]: pgmap v6354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:38 smithi193 ceph-mon[37473]: pgmap v6354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10560.281705 secs 2024-07-26T14:23:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10560.281705 secs 2024-07-26T14:23:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10560.281705 secs 2024-07-26T14:23:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:40 smithi134 ceph-mon[31966]: pgmap v6355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:40 smithi189 ceph-mon[36790]: pgmap v6355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:40 smithi193 ceph-mon[37473]: pgmap v6355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:41.401 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:23:41.400+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:23:41.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:23:41.400+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:23:41.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:23:41.400+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:42 smithi134 ceph-mon[31966]: pgmap v6356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10579.705370 secs 2024-07-26T14:23:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:42 smithi189 ceph-mon[36790]: pgmap v6356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10579.705370 secs 2024-07-26T14:23:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:42 smithi193 ceph-mon[37473]: pgmap v6356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10579.705370 secs 2024-07-26T14:23:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:44 smithi134 ceph-mon[31966]: pgmap v6357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:23:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10565.281847 secs 2024-07-26T14:23:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:44 smithi189 ceph-mon[36790]: pgmap v6357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:23:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10565.281847 secs 2024-07-26T14:23:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:44 smithi193 ceph-mon[37473]: pgmap v6357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:23:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10565.281847 secs 2024-07-26T14:23:46.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:46 smithi134 ceph-mon[31966]: pgmap v6358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:46 smithi189 ceph-mon[36790]: pgmap v6358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:46 smithi193 ceph-mon[37473]: pgmap v6358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10584.705488 secs 2024-07-26T14:23:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10584.705488 secs 2024-07-26T14:23:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10584.705488 secs 2024-07-26T14:23:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:48 smithi134 ceph-mon[31966]: pgmap v6359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:23:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:48 smithi189 ceph-mon[36790]: pgmap v6359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:23:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:48 smithi193 ceph-mon[37473]: pgmap v6359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:23:49.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10570.281912 secs 2024-07-26T14:23:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10570.281912 secs 2024-07-26T14:23:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10570.281912 secs 2024-07-26T14:23:50.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:50 smithi134 ceph-mon[31966]: pgmap v6360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:50 smithi189 ceph-mon[36790]: pgmap v6360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:50 smithi193 ceph-mon[37473]: pgmap v6360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:23:51.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:51 smithi134 ceph-mon[31966]: pgmap v6361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:51 smithi189 ceph-mon[36790]: pgmap v6361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:51 smithi193 ceph-mon[37473]: pgmap v6361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:51.856 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:23:51.855+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:23:51.856 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:23:51.855+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:23:51.856 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:23:51.855+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10589.705611 secs 2024-07-26T14:23:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10589.705611 secs 2024-07-26T14:23:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10589.705611 secs 2024-07-26T14:23:53.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:53 smithi134 ceph-mon[31966]: pgmap v6362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:23:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:53 smithi189 ceph-mon[36790]: pgmap v6362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:23:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:53 smithi193 ceph-mon[37473]: pgmap v6362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:23:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10575.281980 secs 2024-07-26T14:23:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10575.281980 secs 2024-07-26T14:23:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10575.281980 secs 2024-07-26T14:23:55.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:55 smithi134 ceph-mon[31966]: pgmap v6363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:55 smithi189 ceph-mon[36790]: pgmap v6363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:55 smithi193 ceph-mon[37473]: pgmap v6363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:23:56.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:23:56.411+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:23:56.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:23:56.411+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:23:56.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:23:56.411+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:23:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10594.705722 secs 2024-07-26T14:23:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10594.705722 secs 2024-07-26T14:23:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10594.705722 secs 2024-07-26T14:23:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:57 smithi193 ceph-mon[37473]: pgmap v6364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:57 smithi134 ceph-mon[31966]: pgmap v6364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:57 smithi189 ceph-mon[36790]: pgmap v6364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:23:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10580.282093 secs 2024-07-26T14:23:59.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10580.282093 secs 2024-07-26T14:23:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10580.282093 secs 2024-07-26T14:24:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:23:59 smithi193 ceph-mon[37473]: pgmap v6365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:23:59 smithi134 ceph-mon[31966]: pgmap v6365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:23:59 smithi189 ceph-mon[36790]: pgmap v6365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:02 smithi134 ceph-mon[31966]: pgmap v6366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10599.705803 secs 2024-07-26T14:24:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:24:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:02 smithi189 ceph-mon[36790]: pgmap v6366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10599.705803 secs 2024-07-26T14:24:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:24:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:02 smithi193 ceph-mon[37473]: pgmap v6366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10599.705803 secs 2024-07-26T14:24:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:24:03.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:24:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:24:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:24:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:24:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:24:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:24:03.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:24:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:24:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:24:04.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:04 smithi134 ceph-mon[31966]: pgmap v6367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:24:04.325 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:04 smithi134 ceph-mon[31966]: pgmap v6368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:24:04.325 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10585.282211 secs 2024-07-26T14:24:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:04 smithi189 ceph-mon[36790]: pgmap v6367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:24:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:04 smithi189 ceph-mon[36790]: pgmap v6368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:24:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10585.282211 secs 2024-07-26T14:24:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:04 smithi193 ceph-mon[37473]: pgmap v6367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:24:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:04 smithi193 ceph-mon[37473]: pgmap v6368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:24:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10585.282211 secs 2024-07-26T14:24:06.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:06 smithi134 ceph-mon[31966]: pgmap v6369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:24:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:06 smithi189 ceph-mon[36790]: pgmap v6369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:24:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:06 smithi193 ceph-mon[37473]: pgmap v6369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:24:06.869 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:24:06.868+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:24:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:24:06.868+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:24:06.870 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:24:06.868+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10604.705939 secs 2024-07-26T14:24:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10604.705939 secs 2024-07-26T14:24:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10604.705939 secs 2024-07-26T14:24:08.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:08 smithi134 ceph-mon[31966]: pgmap v6370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:24:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:08 smithi189 ceph-mon[36790]: pgmap v6370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:24:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:08 smithi193 ceph-mon[37473]: pgmap v6370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:24:09.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10590.282377 secs 2024-07-26T14:24:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10590.282377 secs 2024-07-26T14:24:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10590.282377 secs 2024-07-26T14:24:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:10 smithi134 ceph-mon[31966]: pgmap v6371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:24:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:10 smithi189 ceph-mon[36790]: pgmap v6371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:24:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:10 smithi193 ceph-mon[37473]: pgmap v6371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:24:11.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:24:11.426+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:24:11.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:24:11.426+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:11.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:24:11.426+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:24:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:12 smithi134 ceph-mon[31966]: pgmap v6372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:24:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10609.706060 secs 2024-07-26T14:24:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:12 smithi189 ceph-mon[36790]: pgmap v6372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:24:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10609.706060 secs 2024-07-26T14:24:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:12 smithi193 ceph-mon[37473]: pgmap v6372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:24:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10609.706060 secs 2024-07-26T14:24:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:14 smithi134 ceph-mon[31966]: pgmap v6373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:24:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10595.282464 secs 2024-07-26T14:24:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:14 smithi189 ceph-mon[36790]: pgmap v6373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:24:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10595.282464 secs 2024-07-26T14:24:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:14 smithi193 ceph-mon[37473]: pgmap v6373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:24:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10595.282464 secs 2024-07-26T14:24:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:15 smithi134 ceph-mon[31966]: pgmap v6374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:15 smithi189 ceph-mon[36790]: pgmap v6374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:15 smithi193 ceph-mon[37473]: pgmap v6374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10614.706209 secs 2024-07-26T14:24:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:17 smithi134 ceph-mon[31966]: pgmap v6375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10614.706209 secs 2024-07-26T14:24:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:17 smithi189 ceph-mon[36790]: pgmap v6375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10614.706209 secs 2024-07-26T14:24:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:17 smithi193 ceph-mon[37473]: pgmap v6375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:19 smithi134 ceph-mon[31966]: pgmap v6376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:24:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10600.282594 secs 2024-07-26T14:24:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:19 smithi189 ceph-mon[36790]: pgmap v6376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:24:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10600.282594 secs 2024-07-26T14:24:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:19 smithi193 ceph-mon[37473]: pgmap v6376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:24:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10600.282594 secs 2024-07-26T14:24:21.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:21 smithi134 ceph-mon[31966]: pgmap v6377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:21 smithi189 ceph-mon[36790]: pgmap v6377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:21 smithi193 ceph-mon[37473]: pgmap v6377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:24:21.882+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:24:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:24:21.882+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:24:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:24:21.882+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10619.706332 secs 2024-07-26T14:24:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10619.706332 secs 2024-07-26T14:24:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10619.706332 secs 2024-07-26T14:24:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:23 smithi189 ceph-mon[36790]: pgmap v6378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:23 smithi193 ceph-mon[37473]: pgmap v6378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:23 smithi134 ceph-mon[31966]: pgmap v6378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:24.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:24:24.177+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:24:24.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:24:24.177+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:24:24.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:24:24.177+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 84 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:24:24.178 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:24:24.177+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:24:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10605.282706 secs 2024-07-26T14:24:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10605.282706 secs 2024-07-26T14:24:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10605.282706 secs 2024-07-26T14:24:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:25 smithi193 ceph-mon[37473]: pgmap v6379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:25.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:25 smithi189 ceph-mon[36790]: pgmap v6379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:25 smithi134 ceph-mon[31966]: pgmap v6379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:26.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:24:26.442+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:24:26.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:24:26.442+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:26.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:24:26.442+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:24:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10624.706448 secs 2024-07-26T14:24:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:27 smithi193 ceph-mon[37473]: pgmap v6380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10624.706448 secs 2024-07-26T14:24:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:27 smithi134 ceph-mon[31966]: pgmap v6380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10624.706448 secs 2024-07-26T14:24:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:27 smithi189 ceph-mon[36790]: pgmap v6380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:29 smithi193 ceph-mon[37473]: pgmap v6381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:24:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10610.282824 secs 2024-07-26T14:24:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:29 smithi134 ceph-mon[31966]: pgmap v6381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:24:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10610.282824 secs 2024-07-26T14:24:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:29 smithi189 ceph-mon[36790]: pgmap v6381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:24:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10610.282824 secs 2024-07-26T14:24:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:31 smithi193 ceph-mon[37473]: pgmap v6382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:31 smithi134 ceph-mon[31966]: pgmap v6382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:31 smithi189 ceph-mon[36790]: pgmap v6382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10629.706592 secs 2024-07-26T14:24:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10629.706592 secs 2024-07-26T14:24:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10629.706592 secs 2024-07-26T14:24:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:33 smithi193 ceph-mon[37473]: pgmap v6383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:24:33.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:33 smithi134 ceph-mon[31966]: pgmap v6383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:24:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:33 smithi189 ceph-mon[36790]: pgmap v6383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:24:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10615.282923 secs 2024-07-26T14:24:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10615.282923 secs 2024-07-26T14:24:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10615.282923 secs 2024-07-26T14:24:34.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:24:34.701+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:24:34.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:24:34.701+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:24:34.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:24:34.702+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:24:34.703 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:24:34.702+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:24:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:35 smithi193 ceph-mon[37473]: pgmap v6384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:35 smithi134 ceph-mon[31966]: pgmap v6384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:35 smithi189 ceph-mon[36790]: pgmap v6384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:36.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:24:36.888+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:24:36.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:24:36.888+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:36.889 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:24:36.888+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:24:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10634.706671 secs 2024-07-26T14:24:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:37 smithi193 ceph-mon[37473]: pgmap v6385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10634.706671 secs 2024-07-26T14:24:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:37 smithi134 ceph-mon[31966]: pgmap v6385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10634.706671 secs 2024-07-26T14:24:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:37 smithi189 ceph-mon[36790]: pgmap v6385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:39 smithi193 ceph-mon[37473]: pgmap v6386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:24:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10620.283039 secs 2024-07-26T14:24:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:39 smithi134 ceph-mon[31966]: pgmap v6386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:24:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10620.283039 secs 2024-07-26T14:24:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:39 smithi189 ceph-mon[36790]: pgmap v6386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:24:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10620.283039 secs 2024-07-26T14:24:41.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:24:41.453+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:24:41.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:24:41.453+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:24:41.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:24:41.453+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:41 smithi193 ceph-mon[37473]: pgmap v6387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:41 smithi134 ceph-mon[31966]: pgmap v6387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:41 smithi189 ceph-mon[36790]: pgmap v6387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10639.706761 secs 2024-07-26T14:24:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10639.706761 secs 2024-07-26T14:24:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10639.706761 secs 2024-07-26T14:24:43.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:43 smithi134 ceph-mon[31966]: pgmap v6388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:43 smithi189 ceph-mon[36790]: pgmap v6388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:43 smithi193 ceph-mon[37473]: pgmap v6388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10625.283160 secs 2024-07-26T14:24:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10625.283160 secs 2024-07-26T14:24:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10625.283160 secs 2024-07-26T14:24:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:45 smithi134 ceph-mon[31966]: pgmap v6389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:45 smithi189 ceph-mon[36790]: pgmap v6389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:45 smithi193 ceph-mon[37473]: pgmap v6389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10644.706881 secs 2024-07-26T14:24:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:47 smithi134 ceph-mon[31966]: pgmap v6390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10644.706881 secs 2024-07-26T14:24:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:47 smithi189 ceph-mon[36790]: pgmap v6390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10644.706881 secs 2024-07-26T14:24:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:47 smithi193 ceph-mon[37473]: pgmap v6390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:49 smithi134 ceph-mon[31966]: pgmap v6391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:24:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10630.283242 secs 2024-07-26T14:24:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:49 smithi189 ceph-mon[36790]: pgmap v6391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:24:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10630.283242 secs 2024-07-26T14:24:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:49 smithi193 ceph-mon[37473]: pgmap v6391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:24:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10630.283242 secs 2024-07-26T14:24:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:51 smithi134 ceph-mon[31966]: pgmap v6392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:51 smithi189 ceph-mon[36790]: pgmap v6392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:51 smithi193 ceph-mon[37473]: pgmap v6392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:24:51.903+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:24:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:24:51.904+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:24:51.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:24:51.904+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10649.707001 secs 2024-07-26T14:24:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10649.707001 secs 2024-07-26T14:24:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10649.707001 secs 2024-07-26T14:24:53.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:53 smithi134 ceph-mon[31966]: pgmap v6393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:53 smithi189 ceph-mon[36790]: pgmap v6393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:53 smithi193 ceph-mon[37473]: pgmap v6393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:24:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10635.283359 secs 2024-07-26T14:24:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10635.283359 secs 2024-07-26T14:24:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10635.283359 secs 2024-07-26T14:24:55.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:55 smithi189 ceph-mon[36790]: pgmap v6394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:55 smithi134 ceph-mon[31966]: pgmap v6394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:55 smithi193 ceph-mon[37473]: pgmap v6394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:24:56.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:24:56.468+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:24:56.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:24:56.468+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:24:56.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:24:56.468+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:24:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10654.707138 secs 2024-07-26T14:24:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:57 smithi134 ceph-mon[31966]: pgmap v6395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10654.707138 secs 2024-07-26T14:24:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:57 smithi189 ceph-mon[36790]: pgmap v6395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10654.707138 secs 2024-07-26T14:24:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:57 smithi193 ceph-mon[37473]: pgmap v6395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:24:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:59 smithi134 ceph-mon[31966]: pgmap v6396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:24:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:24:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10640.283483 secs 2024-07-26T14:24:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:59 smithi189 ceph-mon[36790]: pgmap v6396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:24:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:24:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10640.283483 secs 2024-07-26T14:24:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:59 smithi193 ceph-mon[37473]: pgmap v6396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:24:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:24:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10640.283483 secs 2024-07-26T14:25:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:01 smithi134 ceph-mon[31966]: pgmap v6397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:01 smithi189 ceph-mon[36790]: pgmap v6397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:01 smithi193 ceph-mon[37473]: pgmap v6397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10659.707259 secs 2024-07-26T14:25:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:25:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10659.707259 secs 2024-07-26T14:25:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:25:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10659.707259 secs 2024-07-26T14:25:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:25:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:03 smithi134 ceph-mon[31966]: pgmap v6398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:25:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:25:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:25:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:03 smithi189 ceph-mon[36790]: pgmap v6398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:25:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:25:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:25:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:03 smithi193 ceph-mon[37473]: pgmap v6398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:25:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:25:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:25:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:25:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:25:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:04 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:25:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:04 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:25:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:04 smithi134 ceph-mon[31966]: pgmap v6399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:25:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:04 smithi134 ceph-mon[31966]: pgmap v6400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:25:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10645.283601 secs 2024-07-26T14:25:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:04 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:25:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:04 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:25:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:04 smithi189 ceph-mon[36790]: pgmap v6399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:25:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:04 smithi189 ceph-mon[36790]: pgmap v6400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:25:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10645.283601 secs 2024-07-26T14:25:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:04 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:25:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:04 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:25:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:04 smithi193 ceph-mon[37473]: pgmap v6399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:25:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:04 smithi193 ceph-mon[37473]: pgmap v6400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:25:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10645.283601 secs 2024-07-26T14:25:06.474 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:06 smithi134 ceph-mon[31966]: pgmap v6401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:25:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:06 smithi189 ceph-mon[36790]: pgmap v6401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:25:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:06 smithi193 ceph-mon[37473]: pgmap v6401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:25:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:25:06.905+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:25:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:25:06.905+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:06.906 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:25:06.905+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:25:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10664.707341 secs 2024-07-26T14:25:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10664.707341 secs 2024-07-26T14:25:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10664.707341 secs 2024-07-26T14:25:08.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:08 smithi134 ceph-mon[31966]: pgmap v6402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:25:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:08 smithi189 ceph-mon[36790]: pgmap v6402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:25:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:08 smithi193 ceph-mon[37473]: pgmap v6402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:25:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10650.283724 secs 2024-07-26T14:25:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10650.283724 secs 2024-07-26T14:25:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10650.283724 secs 2024-07-26T14:25:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:10 smithi134 ceph-mon[31966]: pgmap v6403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:25:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:10 smithi189 ceph-mon[36790]: pgmap v6403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:25:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:10 smithi193 ceph-mon[37473]: pgmap v6403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:25:11.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:25:11.484+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:25:11.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:25:11.484+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:11.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:25:11.484+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:25:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10669.707397 secs 2024-07-26T14:25:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:12 smithi189 ceph-mon[36790]: pgmap v6404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:25:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10669.707397 secs 2024-07-26T14:25:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:12 smithi193 ceph-mon[37473]: pgmap v6404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:25:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10669.707397 secs 2024-07-26T14:25:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:12 smithi134 ceph-mon[31966]: pgmap v6404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:25:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:14 smithi189 ceph-mon[36790]: pgmap v6405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:25:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10655.283845 secs 2024-07-26T14:25:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:14 smithi193 ceph-mon[37473]: pgmap v6405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:25:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10655.283845 secs 2024-07-26T14:25:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:14 smithi134 ceph-mon[31966]: pgmap v6405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:25:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10655.283845 secs 2024-07-26T14:25:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:16 smithi189 ceph-mon[36790]: pgmap v6406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:16 smithi193 ceph-mon[37473]: pgmap v6406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:16 smithi134 ceph-mon[31966]: pgmap v6406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10674.707481 secs 2024-07-26T14:25:17.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10674.707481 secs 2024-07-26T14:25:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10674.707481 secs 2024-07-26T14:25:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:18 smithi193 ceph-mon[37473]: pgmap v6407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:25:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:18 smithi134 ceph-mon[31966]: pgmap v6407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:25:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:18 smithi189 ceph-mon[36790]: pgmap v6407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:25:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10660.283986 secs 2024-07-26T14:25:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10660.283986 secs 2024-07-26T14:25:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10660.283986 secs 2024-07-26T14:25:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:20 smithi193 ceph-mon[37473]: pgmap v6408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:25:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:20 smithi134 ceph-mon[31966]: pgmap v6408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:25:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:20 smithi189 ceph-mon[36790]: pgmap v6408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:25:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10679.707547 secs 2024-07-26T14:25:21.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:25:21.920+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:25:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:25:21.920+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:25:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10679.707547 secs 2024-07-26T14:25:21.922 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:25:21.920+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10679.707547 secs 2024-07-26T14:25:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:22 smithi193 ceph-mon[37473]: pgmap v6409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:22 smithi134 ceph-mon[31966]: pgmap v6409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:22 smithi189 ceph-mon[36790]: pgmap v6409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:24 smithi193 ceph-mon[37473]: pgmap v6410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:25:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10665.284116 secs 2024-07-26T14:25:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:24 smithi134 ceph-mon[31966]: pgmap v6410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:25:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10665.284116 secs 2024-07-26T14:25:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:24 smithi189 ceph-mon[36790]: pgmap v6410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:25:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10665.284116 secs 2024-07-26T14:25:26.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:25:26.499+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:25:26.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:25:26.499+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:25:26.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:25:26.499+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:26.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:26 smithi189 ceph-mon[36790]: pgmap v6411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:26.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10684.707685 secs 2024-07-26T14:25:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:26 smithi193 ceph-mon[37473]: pgmap v6411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10684.707685 secs 2024-07-26T14:25:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:26 smithi134 ceph-mon[31966]: pgmap v6411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10684.707685 secs 2024-07-26T14:25:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:28 smithi193 ceph-mon[37473]: pgmap v6412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:25:28.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:28 smithi134 ceph-mon[31966]: pgmap v6412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:25:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:28 smithi189 ceph-mon[36790]: pgmap v6412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:25:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10670.284193 secs 2024-07-26T14:25:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10670.284193 secs 2024-07-26T14:25:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10670.284193 secs 2024-07-26T14:25:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:30 smithi193 ceph-mon[37473]: pgmap v6413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:25:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:30 smithi134 ceph-mon[31966]: pgmap v6413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:25:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:30 smithi189 ceph-mon[36790]: pgmap v6413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:25:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10689.707825 secs 2024-07-26T14:25:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10689.707825 secs 2024-07-26T14:25:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10689.707825 secs 2024-07-26T14:25:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:32 smithi134 ceph-mon[31966]: pgmap v6414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:32 smithi189 ceph-mon[36790]: pgmap v6414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:32 smithi193 ceph-mon[37473]: pgmap v6414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:34 smithi134 ceph-mon[31966]: pgmap v6415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:25:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10675.284311 secs 2024-07-26T14:25:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:34 smithi189 ceph-mon[36790]: pgmap v6415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:25:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10675.284311 secs 2024-07-26T14:25:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:34 smithi193 ceph-mon[37473]: pgmap v6415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:25:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10675.284311 secs 2024-07-26T14:25:36.904 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:36 smithi134 ceph-mon[31966]: pgmap v6416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:36.904 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10694.707964 secs 2024-07-26T14:25:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:25:36.936+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:25:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:25:36.936+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:36.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:25:36.936+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:25:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:36 smithi189 ceph-mon[36790]: pgmap v6416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10694.707964 secs 2024-07-26T14:25:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:36 smithi193 ceph-mon[37473]: pgmap v6416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10694.707964 secs 2024-07-26T14:25:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:38 smithi134 ceph-mon[31966]: pgmap v6417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:25:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:38 smithi189 ceph-mon[36790]: pgmap v6417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:25:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:38 smithi193 ceph-mon[37473]: pgmap v6417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:25:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10680.284435 secs 2024-07-26T14:25:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10680.284435 secs 2024-07-26T14:25:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10680.284435 secs 2024-07-26T14:25:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:40 smithi134 ceph-mon[31966]: pgmap v6418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:25:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:40 smithi189 ceph-mon[36790]: pgmap v6418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:25:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:40 smithi193 ceph-mon[37473]: pgmap v6418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:25:41.505 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:25:41.504+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:25:41.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:25:41.504+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:25:41.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:25:41.504+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:41.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10699.708091 secs 2024-07-26T14:25:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10699.708091 secs 2024-07-26T14:25:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10699.708091 secs 2024-07-26T14:25:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:42 smithi134 ceph-mon[31966]: pgmap v6419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:42 smithi189 ceph-mon[36790]: pgmap v6419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:42 smithi193 ceph-mon[37473]: pgmap v6419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:44 smithi134 ceph-mon[31966]: pgmap v6420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:25:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10685.284550 secs 2024-07-26T14:25:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:44 smithi189 ceph-mon[36790]: pgmap v6420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:25:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10685.284550 secs 2024-07-26T14:25:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:44 smithi193 ceph-mon[37473]: pgmap v6420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:25:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10685.284550 secs 2024-07-26T14:25:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:46 smithi134 ceph-mon[31966]: pgmap v6421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10704.708185 secs 2024-07-26T14:25:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:46 smithi189 ceph-mon[36790]: pgmap v6421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10704.708185 secs 2024-07-26T14:25:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:46 smithi193 ceph-mon[37473]: pgmap v6421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10704.708185 secs 2024-07-26T14:25:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:48 smithi134 ceph-mon[31966]: pgmap v6422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:25:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:48 smithi189 ceph-mon[36790]: pgmap v6422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:25:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:48 smithi193 ceph-mon[37473]: pgmap v6422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:25:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10690.284671 secs 2024-07-26T14:25:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10690.284671 secs 2024-07-26T14:25:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10690.284671 secs 2024-07-26T14:25:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:50 smithi134 ceph-mon[31966]: pgmap v6423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:25:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:50 smithi189 ceph-mon[36790]: pgmap v6423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:25:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:50 smithi193 ceph-mon[37473]: pgmap v6423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:25:51.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10709.708263 secs 2024-07-26T14:25:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10709.708263 secs 2024-07-26T14:25:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:25:51.951+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:25:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:25:51.951+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:25:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:25:51.951+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10709.708263 secs 2024-07-26T14:25:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:52 smithi134 ceph-mon[31966]: pgmap v6424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:52 smithi189 ceph-mon[36790]: pgmap v6424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:52 smithi193 ceph-mon[37473]: pgmap v6424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:54 smithi134 ceph-mon[31966]: pgmap v6425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:25:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10695.284806 secs 2024-07-26T14:25:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:54 smithi189 ceph-mon[36790]: pgmap v6425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:25:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10695.284806 secs 2024-07-26T14:25:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:54 smithi193 ceph-mon[37473]: pgmap v6425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:25:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10695.284806 secs 2024-07-26T14:25:56.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:25:56.520+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:25:56.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:25:56.520+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:25:56.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:25:56.520+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:25:56.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:56 smithi134 ceph-mon[31966]: pgmap v6426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10714.708381 secs 2024-07-26T14:25:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:56 smithi189 ceph-mon[36790]: pgmap v6426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10714.708381 secs 2024-07-26T14:25:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:56 smithi193 ceph-mon[37473]: pgmap v6426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:25:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10714.708381 secs 2024-07-26T14:25:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:58 smithi134 ceph-mon[31966]: pgmap v6427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:25:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:58 smithi189 ceph-mon[36790]: pgmap v6427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:25:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:58 smithi193 ceph-mon[37473]: pgmap v6427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:25:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:25:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10700.284916 secs 2024-07-26T14:25:59.924 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:25:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10700.284916 secs 2024-07-26T14:25:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:25:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10700.284916 secs 2024-07-26T14:26:00.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:00 smithi134 ceph-mon[31966]: pgmap v6428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:26:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:00 smithi189 ceph-mon[36790]: pgmap v6428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:26:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:00 smithi193 ceph-mon[37473]: pgmap v6428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:26:01.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10719.708505 secs 2024-07-26T14:26:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10719.708505 secs 2024-07-26T14:26:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10719.708505 secs 2024-07-26T14:26:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:02 smithi189 ceph-mon[36790]: pgmap v6429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:02 smithi193 ceph-mon[37473]: pgmap v6429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:03.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:02 smithi134 ceph-mon[31966]: pgmap v6429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:26:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:26:04.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:26:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: pgmap v6430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10705.285015 secs 2024-07-26T14:26:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:26:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:26:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:26:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:26:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:26:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:26:04.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:26:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: pgmap v6430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10705.285015 secs 2024-07-26T14:26:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:26:05.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:26:05.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:26:05.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:26:05.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:26:05.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:26:05.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:26:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: pgmap v6430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10705.285015 secs 2024-07-26T14:26:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:26:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:26:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:26:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:26:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:26:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:26:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:26:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:26:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:05 smithi189 ceph-mon[36790]: pgmap v6431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:26:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:05 smithi189 ceph-mon[36790]: pgmap v6432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:26:06.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:05 smithi193 ceph-mon[37473]: pgmap v6431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:26:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:05 smithi193 ceph-mon[37473]: pgmap v6432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:26:06.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:05 smithi134 ceph-mon[31966]: pgmap v6431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:26:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:05 smithi134 ceph-mon[31966]: pgmap v6432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:26:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:26:06.967+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:26:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:26:06.967+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:06.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:26:06.967+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:26:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10724.708620 secs 2024-07-26T14:26:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10724.708620 secs 2024-07-26T14:26:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10724.708620 secs 2024-07-26T14:26:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:07 smithi193 ceph-mon[37473]: pgmap v6433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:08.136 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:07 smithi134 ceph-mon[31966]: pgmap v6433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:07 smithi189 ceph-mon[36790]: pgmap v6433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:09 smithi193 ceph-mon[37473]: pgmap v6434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:26:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10710.285082 secs 2024-07-26T14:26:10.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:09 smithi134 ceph-mon[31966]: pgmap v6434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:26:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10710.285082 secs 2024-07-26T14:26:10.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:09 smithi189 ceph-mon[36790]: pgmap v6434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:26:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10710.285082 secs 2024-07-26T14:26:11.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:26:11.526+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:26:11.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:26:11.526+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:11.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:26:11.526+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:26:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:11 smithi134 ceph-mon[31966]: pgmap v6435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:26:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10729.708734 secs 2024-07-26T14:26:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:11 smithi193 ceph-mon[37473]: pgmap v6435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:26:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10729.708734 secs 2024-07-26T14:26:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:11 smithi189 ceph-mon[36790]: pgmap v6435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:26:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10729.708734 secs 2024-07-26T14:26:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:13 smithi193 ceph-mon[37473]: pgmap v6436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:26:14.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:13 smithi134 ceph-mon[31966]: pgmap v6436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:26:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:13 smithi189 ceph-mon[36790]: pgmap v6436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:26:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10715.285175 secs 2024-07-26T14:26:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10715.285175 secs 2024-07-26T14:26:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10715.285175 secs 2024-07-26T14:26:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:15 smithi193 ceph-mon[37473]: pgmap v6437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:26:16.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:15 smithi134 ceph-mon[31966]: pgmap v6437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:26:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:15 smithi189 ceph-mon[36790]: pgmap v6437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:26:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10734.708832 secs 2024-07-26T14:26:17.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10734.708832 secs 2024-07-26T14:26:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10734.708832 secs 2024-07-26T14:26:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:17 smithi193 ceph-mon[37473]: pgmap v6438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:17 smithi134 ceph-mon[31966]: pgmap v6438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:17 smithi189 ceph-mon[36790]: pgmap v6438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:19 smithi193 ceph-mon[37473]: pgmap v6439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10720.285272 secs 2024-07-26T14:26:20.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:19 smithi134 ceph-mon[31966]: pgmap v6439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10720.285272 secs 2024-07-26T14:26:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:19 smithi189 ceph-mon[36790]: pgmap v6439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10720.285272 secs 2024-07-26T14:26:21.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:26:21.968+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:26:21.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:26:21.968+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:21.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:26:21.968+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:26:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:21 smithi193 ceph-mon[37473]: pgmap v6440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10739.708955 secs 2024-07-26T14:26:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:21 smithi134 ceph-mon[31966]: pgmap v6440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10739.708955 secs 2024-07-26T14:26:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:21 smithi189 ceph-mon[36790]: pgmap v6440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10739.708955 secs 2024-07-26T14:26:24.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:23 smithi134 ceph-mon[31966]: pgmap v6441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T14:26:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:23 smithi193 ceph-mon[37473]: pgmap v6441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T14:26:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:23 smithi189 ceph-mon[36790]: pgmap v6441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T14:26:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10725.285388 secs 2024-07-26T14:26:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10725.285388 secs 2024-07-26T14:26:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10725.285388 secs 2024-07-26T14:26:26.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:25 smithi189 ceph-mon[36790]: pgmap v6442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:26.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:25 smithi134 ceph-mon[31966]: pgmap v6442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:25 smithi193 ceph-mon[37473]: pgmap v6442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:26.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:26:26.539+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:26:26.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:26:26.539+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:26.540 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:26:26.539+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:26:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10744.709076 secs 2024-07-26T14:26:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10744.709076 secs 2024-07-26T14:26:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10744.709076 secs 2024-07-26T14:26:28.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:27 smithi134 ceph-mon[31966]: pgmap v6443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:27 smithi189 ceph-mon[36790]: pgmap v6443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:27 smithi193 ceph-mon[37473]: pgmap v6443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:30.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:29 smithi193 ceph-mon[37473]: pgmap v6444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:30.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10730.285508 secs 2024-07-26T14:26:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:29 smithi134 ceph-mon[31966]: pgmap v6444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10730.285508 secs 2024-07-26T14:26:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:29 smithi189 ceph-mon[36790]: pgmap v6444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10730.285508 secs 2024-07-26T14:26:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:31 smithi134 ceph-mon[31966]: pgmap v6445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10749.709208 secs 2024-07-26T14:26:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:31 smithi189 ceph-mon[36790]: pgmap v6445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10749.709208 secs 2024-07-26T14:26:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:31 smithi193 ceph-mon[37473]: pgmap v6445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10749.709208 secs 2024-07-26T14:26:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:33 smithi134 ceph-mon[31966]: pgmap v6446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:33 smithi189 ceph-mon[36790]: pgmap v6446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:33 smithi193 ceph-mon[37473]: pgmap v6446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10735.285650 secs 2024-07-26T14:26:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10735.285650 secs 2024-07-26T14:26:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10735.285650 secs 2024-07-26T14:26:36.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:35 smithi134 ceph-mon[31966]: pgmap v6447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:35 smithi189 ceph-mon[36790]: pgmap v6447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:35 smithi193 ceph-mon[37473]: pgmap v6447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:26:36.976+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:26:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:26:36.976+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:26:36.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:26:36.976+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10754.709358 secs 2024-07-26T14:26:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10754.709358 secs 2024-07-26T14:26:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10754.709358 secs 2024-07-26T14:26:38.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:37 smithi134 ceph-mon[31966]: pgmap v6448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:37 smithi189 ceph-mon[36790]: pgmap v6448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:37 smithi193 ceph-mon[37473]: pgmap v6448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:39 smithi134 ceph-mon[31966]: pgmap v6449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10740.285770 secs 2024-07-26T14:26:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:39 smithi189 ceph-mon[36790]: pgmap v6449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10740.285770 secs 2024-07-26T14:26:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:39 smithi193 ceph-mon[37473]: pgmap v6449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10740.285770 secs 2024-07-26T14:26:41.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:26:41.555+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:26:41.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:26:41.555+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:26:41.556 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:26:41.555+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:41 smithi134 ceph-mon[31966]: pgmap v6450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10759.709479 secs 2024-07-26T14:26:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:41 smithi189 ceph-mon[36790]: pgmap v6450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10759.709479 secs 2024-07-26T14:26:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:41 smithi193 ceph-mon[37473]: pgmap v6450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10759.709479 secs 2024-07-26T14:26:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:43 smithi134 ceph-mon[31966]: pgmap v6451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:43 smithi189 ceph-mon[36790]: pgmap v6451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:43 smithi193 ceph-mon[37473]: pgmap v6451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10745.285889 secs 2024-07-26T14:26:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10745.285889 secs 2024-07-26T14:26:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10745.285889 secs 2024-07-26T14:26:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:45 smithi134 ceph-mon[31966]: pgmap v6452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:26:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:45 smithi189 ceph-mon[36790]: pgmap v6452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:26:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:45 smithi193 ceph-mon[37473]: pgmap v6452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:26:47.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10764.709552 secs 2024-07-26T14:26:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10764.709552 secs 2024-07-26T14:26:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10764.709552 secs 2024-07-26T14:26:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:47 smithi134 ceph-mon[31966]: pgmap v6453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:47 smithi189 ceph-mon[36790]: pgmap v6453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:47 smithi193 ceph-mon[37473]: pgmap v6453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:50.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10750.285957 secs 2024-07-26T14:26:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:49 smithi134 ceph-mon[31966]: pgmap v6454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10750.285957 secs 2024-07-26T14:26:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:49 smithi189 ceph-mon[36790]: pgmap v6454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10750.285957 secs 2024-07-26T14:26:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:49 smithi193 ceph-mon[37473]: pgmap v6454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:26:51.992+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:26:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:26:51.992+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:26:51.993 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:26:51.992+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:51 smithi134 ceph-mon[31966]: pgmap v6455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10769.709678 secs 2024-07-26T14:26:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:51 smithi189 ceph-mon[36790]: pgmap v6455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10769.709678 secs 2024-07-26T14:26:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:51 smithi193 ceph-mon[37473]: pgmap v6455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10769.709678 secs 2024-07-26T14:26:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:53 smithi134 ceph-mon[31966]: pgmap v6456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:53 smithi189 ceph-mon[36790]: pgmap v6456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:53 smithi193 ceph-mon[37473]: pgmap v6456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:26:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10755.286077 secs 2024-07-26T14:26:55.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10755.286077 secs 2024-07-26T14:26:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10755.286077 secs 2024-07-26T14:26:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:55 smithi189 ceph-mon[36790]: pgmap v6457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:26:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:55 smithi193 ceph-mon[37473]: pgmap v6457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:26:56.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:55 smithi134 ceph-mon[31966]: pgmap v6457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:26:56.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:26:56.570+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:26:56.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:26:56.570+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:26:56.571 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:26:56.570+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:26:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10774.709819 secs 2024-07-26T14:26:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10774.709819 secs 2024-07-26T14:26:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10774.709819 secs 2024-07-26T14:26:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:57 smithi193 ceph-mon[37473]: pgmap v6458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:58.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:57 smithi134 ceph-mon[31966]: pgmap v6458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:57 smithi189 ceph-mon[36790]: pgmap v6458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:26:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10760.286162 secs 2024-07-26T14:26:59.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10760.286162 secs 2024-07-26T14:26:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10760.286162 secs 2024-07-26T14:27:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:26:59 smithi193 ceph-mon[37473]: pgmap v6459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:26:59 smithi134 ceph-mon[31966]: pgmap v6459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:26:59 smithi189 ceph-mon[36790]: pgmap v6459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:01 smithi193 ceph-mon[37473]: pgmap v6460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10779.709899 secs 2024-07-26T14:27:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:01 smithi134 ceph-mon[31966]: pgmap v6460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10779.709899 secs 2024-07-26T14:27:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:01 smithi189 ceph-mon[36790]: pgmap v6460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10779.709899 secs 2024-07-26T14:27:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:03 smithi193 ceph-mon[37473]: pgmap v6461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10765.286286 secs 2024-07-26T14:27:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:03 smithi134 ceph-mon[31966]: pgmap v6461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10765.286286 secs 2024-07-26T14:27:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:03 smithi189 ceph-mon[36790]: pgmap v6461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10765.286286 secs 2024-07-26T14:27:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:27:05.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:27:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:27:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:05 smithi193 ceph-mon[37473]: pgmap v6462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:27:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:27:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:27:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:05 smithi134 ceph-mon[31966]: pgmap v6462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:27:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:27:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:27:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:05 smithi189 ceph-mon[36790]: pgmap v6462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:27:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:27:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:27:07.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:07.007+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:27:07.009 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:07.007+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:27:07.009 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:07.007+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:07 smithi193 ceph-mon[37473]: pgmap v6463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:27:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:07 smithi193 ceph-mon[37473]: pgmap v6464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:27:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10784.710025 secs 2024-07-26T14:27:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:07 smithi134 ceph-mon[31966]: pgmap v6463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:27:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:07 smithi134 ceph-mon[31966]: pgmap v6464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:27:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10784.710025 secs 2024-07-26T14:27:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:07 smithi189 ceph-mon[36790]: pgmap v6463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:27:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:07 smithi189 ceph-mon[36790]: pgmap v6464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:27:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10784.710025 secs 2024-07-26T14:27:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:09 smithi193 ceph-mon[37473]: pgmap v6465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:27:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10770.286398 secs 2024-07-26T14:27:09.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:09 smithi134 ceph-mon[31966]: pgmap v6465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:27:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10770.286398 secs 2024-07-26T14:27:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:09 smithi189 ceph-mon[36790]: pgmap v6465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:27:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10770.286398 secs 2024-07-26T14:27:11.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:11 smithi193 ceph-mon[37473]: pgmap v6466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:27:11.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:11 smithi134 ceph-mon[31966]: pgmap v6466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:27:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:11 smithi189 ceph-mon[36790]: pgmap v6466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:27:11.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:11.585+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:27:11.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:11.585+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:27:11.587 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:11.585+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10789.710154 secs 2024-07-26T14:27:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10789.710154 secs 2024-07-26T14:27:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10789.710154 secs 2024-07-26T14:27:13.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:13 smithi134 ceph-mon[31966]: pgmap v6467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:27:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:13 smithi189 ceph-mon[36790]: pgmap v6467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:27:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:13 smithi193 ceph-mon[37473]: pgmap v6467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:27:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10775.286482 secs 2024-07-26T14:27:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10775.286482 secs 2024-07-26T14:27:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10775.286482 secs 2024-07-26T14:27:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:15 smithi134 ceph-mon[31966]: pgmap v6468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:27:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:15 smithi189 ceph-mon[36790]: pgmap v6468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:27:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:15 smithi193 ceph-mon[37473]: pgmap v6468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:27:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:17 smithi134 ceph-mon[31966]: pgmap v6469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:27:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10794.710219 secs 2024-07-26T14:27:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:17 smithi189 ceph-mon[36790]: pgmap v6469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:27:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10794.710219 secs 2024-07-26T14:27:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:17 smithi193 ceph-mon[37473]: pgmap v6469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:27:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10794.710219 secs 2024-07-26T14:27:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:19 smithi134 ceph-mon[31966]: pgmap v6470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10780.286601 secs 2024-07-26T14:27:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:19 smithi189 ceph-mon[36790]: pgmap v6470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10780.286601 secs 2024-07-26T14:27:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:19 smithi193 ceph-mon[37473]: pgmap v6470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10780.286601 secs 2024-07-26T14:27:21.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:21 smithi134 ceph-mon[31966]: pgmap v6471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:21 smithi189 ceph-mon[36790]: pgmap v6471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:21 smithi193 ceph-mon[37473]: pgmap v6471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:22.025 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:22.023+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:27:22.025 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:22.023+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:22.025 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:22.023+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:27:22.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10799.710341 secs 2024-07-26T14:27:22.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:22.321+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T14:27:22.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:22.321+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T14:27:22.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:22.321+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b3803a040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:27:22.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:22.321+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b3803a040 comp rx=0 tx=0).stop 2024-07-26T14:27:22.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:22.321+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T14:27:22.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:22.321+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:27:22.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:22.321+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:27:22.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:22.321+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1740 0x7f9b300c4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:27:22.324 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:22.322+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1740 0x7f9b300c4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:27:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:22.324+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1740 0x7f9b300c4130 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009770 tx=0x7f9b3805d040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:27:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10799.710341 secs 2024-07-26T14:27:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10799.710341 secs 2024-07-26T14:27:23.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:23 smithi134 ceph-mon[31966]: pgmap v6472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:23 smithi189 ceph-mon[36790]: pgmap v6472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:23 smithi193 ceph-mon[37473]: pgmap v6472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:24.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:24.207+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:27:24.209 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:24.208+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 85 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:27:24.212 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:24.211+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:27:24.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:24.212+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:27:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10785.286749 secs 2024-07-26T14:27:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10785.286749 secs 2024-07-26T14:27:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10785.286749 secs 2024-07-26T14:27:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:25 smithi134 ceph-mon[31966]: pgmap v6473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:25 smithi189 ceph-mon[36790]: pgmap v6473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:25 smithi193 ceph-mon[37473]: pgmap v6473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:26.589 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:26.588+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:27:26.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:26.588+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:26.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:26.588+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:27:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:27 smithi134 ceph-mon[31966]: pgmap v6474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10804.710460 secs 2024-07-26T14:27:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:27 smithi189 ceph-mon[36790]: pgmap v6474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10804.710460 secs 2024-07-26T14:27:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:27 smithi193 ceph-mon[37473]: pgmap v6474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10804.710460 secs 2024-07-26T14:27:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:29 smithi134 ceph-mon[31966]: pgmap v6475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10790.286814 secs 2024-07-26T14:27:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:29 smithi189 ceph-mon[36790]: pgmap v6475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10790.286814 secs 2024-07-26T14:27:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:29 smithi193 ceph-mon[37473]: pgmap v6475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10790.286814 secs 2024-07-26T14:27:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:31 smithi134 ceph-mon[31966]: pgmap v6476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:31 smithi189 ceph-mon[36790]: pgmap v6476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:31 smithi193 ceph-mon[37473]: pgmap v6476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10809.710578 secs 2024-07-26T14:27:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10809.710578 secs 2024-07-26T14:27:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10809.710578 secs 2024-07-26T14:27:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:32.876+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 msgr2=0x7f21ac0c14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000040) for more than 900000000 us, fault. 2024-07-26T14:27:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:32.877+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f2198005e40 tx=0x7f21b80c3000 comp rx=0 tx=0).stop 2024-07-26T14:27:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:32.877+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 msgr2=0x7f21ac0c14c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:27:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:32.877+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:27:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:32.877+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:27:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:32.877+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:27:32.880 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:32.879+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f21b80d3f20 tx=0x7f21b80aa000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:27:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:33 smithi134 ceph-mon[31966]: pgmap v6477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:33 smithi189 ceph-mon[36790]: pgmap v6477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:33 smithi193 ceph-mon[37473]: pgmap v6477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:34.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10795.286954 secs 2024-07-26T14:27:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10795.286954 secs 2024-07-26T14:27:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10795.286954 secs 2024-07-26T14:27:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:34.734+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:27:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:34.734+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:27:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:34.735+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:27:34.736 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:34.735+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:27:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:35 smithi134 ceph-mon[31966]: pgmap v6478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:35 smithi189 ceph-mon[36790]: pgmap v6478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:35 smithi193 ceph-mon[37473]: pgmap v6478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:37.025+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:27:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:37.025+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:37.026 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:37.025+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:27:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:37 smithi134 ceph-mon[31966]: pgmap v6479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10814.710681 secs 2024-07-26T14:27:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:37 smithi189 ceph-mon[36790]: pgmap v6479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10814.710681 secs 2024-07-26T14:27:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:37 smithi193 ceph-mon[37473]: pgmap v6479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10814.710681 secs 2024-07-26T14:27:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:39 smithi134 ceph-mon[31966]: pgmap v6480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10800.287053 secs 2024-07-26T14:27:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:39 smithi189 ceph-mon[36790]: pgmap v6480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10800.287053 secs 2024-07-26T14:27:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:39 smithi193 ceph-mon[37473]: pgmap v6480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10800.287053 secs 2024-07-26T14:27:41.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:41 smithi134 ceph-mon[31966]: pgmap v6481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:41 smithi189 ceph-mon[36790]: pgmap v6481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:41 smithi193 ceph-mon[37473]: pgmap v6481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:41.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:41.600+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:27:41.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:41.600+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:27:41.601 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:41.600+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10819.710802 secs 2024-07-26T14:27:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10819.710802 secs 2024-07-26T14:27:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10819.710802 secs 2024-07-26T14:27:43.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:43 smithi134 ceph-mon[31966]: pgmap v6482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:43 smithi189 ceph-mon[36790]: pgmap v6482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:43 smithi193 ceph-mon[37473]: pgmap v6482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10805.287170 secs 2024-07-26T14:27:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:44 smithi189 ceph-mon[36790]: pgmap v6483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10805.287170 secs 2024-07-26T14:27:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:44 smithi193 ceph-mon[37473]: pgmap v6483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10805.287170 secs 2024-07-26T14:27:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:44 smithi134 ceph-mon[31966]: pgmap v6483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:46 smithi134 ceph-mon[31966]: pgmap v6484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10824.710945 secs 2024-07-26T14:27:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:46 smithi189 ceph-mon[36790]: pgmap v6484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10824.710945 secs 2024-07-26T14:27:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:46 smithi193 ceph-mon[37473]: pgmap v6484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10824.710945 secs 2024-07-26T14:27:49.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:48 smithi134 ceph-mon[31966]: pgmap v6485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:48 smithi189 ceph-mon[36790]: pgmap v6485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:48 smithi193 ceph-mon[37473]: pgmap v6485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:50.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10810.287250 secs 2024-07-26T14:27:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10810.287250 secs 2024-07-26T14:27:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10810.287250 secs 2024-07-26T14:27:51.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:50 smithi134 ceph-mon[31966]: pgmap v6486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:50 smithi189 ceph-mon[36790]: pgmap v6486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:50 smithi193 ceph-mon[37473]: pgmap v6486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:27:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:52.039+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:27:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:52.040+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:27:52.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:27:52.040+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10829.711089 secs 2024-07-26T14:27:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10829.711089 secs 2024-07-26T14:27:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10829.711089 secs 2024-07-26T14:27:53.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:52 smithi134 ceph-mon[31966]: pgmap v6487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:52 smithi189 ceph-mon[36790]: pgmap v6487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:52 smithi193 ceph-mon[37473]: pgmap v6487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10815.287316 secs 2024-07-26T14:27:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:54 smithi134 ceph-mon[31966]: pgmap v6488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10815.287316 secs 2024-07-26T14:27:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:54 smithi189 ceph-mon[36790]: pgmap v6488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10815.287316 secs 2024-07-26T14:27:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:54 smithi193 ceph-mon[37473]: pgmap v6488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:56.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:56.608+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:27:56.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:56.608+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:27:56.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:27:56.608+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:27:56.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:56 smithi134 ceph-mon[31966]: pgmap v6489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10834.711223 secs 2024-07-26T14:27:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:56 smithi189 ceph-mon[36790]: pgmap v6489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10834.711223 secs 2024-07-26T14:27:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:56 smithi193 ceph-mon[37473]: pgmap v6489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:27:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10834.711223 secs 2024-07-26T14:27:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:58 smithi134 ceph-mon[31966]: pgmap v6490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:58 smithi189 ceph-mon[36790]: pgmap v6490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:27:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:58 smithi193 ceph-mon[37473]: pgmap v6490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:27:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10820.287442 secs 2024-07-26T14:28:00.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:27:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10820.287442 secs 2024-07-26T14:28:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:27:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10820.287442 secs 2024-07-26T14:28:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:00 smithi134 ceph-mon[31966]: pgmap v6491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:28:01.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:00 smithi193 ceph-mon[37473]: pgmap v6491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:28:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:00 smithi189 ceph-mon[36790]: pgmap v6491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:28:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10839.711357 secs 2024-07-26T14:28:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10839.711357 secs 2024-07-26T14:28:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10839.711357 secs 2024-07-26T14:28:03.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:02 smithi134 ceph-mon[31966]: pgmap v6492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:02 smithi189 ceph-mon[36790]: pgmap v6492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:02 smithi193 ceph-mon[37473]: pgmap v6492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10825.287565 secs 2024-07-26T14:28:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:04 smithi134 ceph-mon[31966]: pgmap v6493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10825.287565 secs 2024-07-26T14:28:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:04 smithi189 ceph-mon[36790]: pgmap v6493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10825.287565 secs 2024-07-26T14:28:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:04 smithi193 ceph-mon[37473]: pgmap v6493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:07.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:28:07.055+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:28:07.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:28:07.055+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:07.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:28:07.055+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:28:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:06 smithi134 ceph-mon[31966]: pgmap v6494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:28:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10844.711421 secs 2024-07-26T14:28:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:28:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:28:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:06 smithi189 ceph-mon[36790]: pgmap v6494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:28:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10844.711421 secs 2024-07-26T14:28:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:28:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:28:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:06 smithi193 ceph-mon[37473]: pgmap v6494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:28:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10844.711421 secs 2024-07-26T14:28:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:28:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:28:08.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:28:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:28:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:28:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:28:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:28:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:28:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:28:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:28:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:28:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:08 smithi134 ceph-mon[31966]: pgmap v6495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T14:28:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:08 smithi134 ceph-mon[31966]: pgmap v6496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:28:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:08 smithi189 ceph-mon[36790]: pgmap v6495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T14:28:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:08 smithi189 ceph-mon[36790]: pgmap v6496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:28:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:08 smithi193 ceph-mon[37473]: pgmap v6495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T14:28:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:08 smithi193 ceph-mon[37473]: pgmap v6496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:28:10.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10830.287707 secs 2024-07-26T14:28:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10830.287707 secs 2024-07-26T14:28:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10830.287707 secs 2024-07-26T14:28:11.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:10 smithi134 ceph-mon[31966]: pgmap v6497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:28:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:10 smithi189 ceph-mon[36790]: pgmap v6497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:28:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:10 smithi193 ceph-mon[37473]: pgmap v6497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:28:11.624 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:28:11.623+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:28:11.624 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:28:11.623+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:28:11.625 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:28:11.623+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10849.711544 secs 2024-07-26T14:28:12.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10849.711544 secs 2024-07-26T14:28:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10849.711544 secs 2024-07-26T14:28:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:12 smithi134 ceph-mon[31966]: pgmap v6498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:28:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:12 smithi189 ceph-mon[36790]: pgmap v6498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:28:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:12 smithi193 ceph-mon[37473]: pgmap v6498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:28:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10835.287859 secs 2024-07-26T14:28:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:14 smithi134 ceph-mon[31966]: pgmap v6499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:28:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10835.287859 secs 2024-07-26T14:28:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:14 smithi189 ceph-mon[36790]: pgmap v6499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:28:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10835.287859 secs 2024-07-26T14:28:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:14 smithi193 ceph-mon[37473]: pgmap v6499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:28:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:16 smithi189 ceph-mon[36790]: pgmap v6500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:28:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10854.711644 secs 2024-07-26T14:28:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:16 smithi193 ceph-mon[37473]: pgmap v6500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:28:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10854.711644 secs 2024-07-26T14:28:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:16 smithi134 ceph-mon[31966]: pgmap v6500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:28:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10854.711644 secs 2024-07-26T14:28:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:18 smithi189 ceph-mon[36790]: pgmap v6501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:28:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:18 smithi193 ceph-mon[37473]: pgmap v6501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:28:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:18 smithi134 ceph-mon[31966]: pgmap v6501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:28:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10840.287963 secs 2024-07-26T14:28:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10840.287963 secs 2024-07-26T14:28:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10840.287963 secs 2024-07-26T14:28:21.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:20 smithi193 ceph-mon[37473]: pgmap v6502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:28:21.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:20 smithi134 ceph-mon[31966]: pgmap v6502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:28:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:20 smithi189 ceph-mon[36790]: pgmap v6502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:28:22.060 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:28:22.059+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:28:22.061 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:28:22.059+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:28:22.061 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:28:22.059+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10859.711739 secs 2024-07-26T14:28:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10859.711739 secs 2024-07-26T14:28:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10859.711739 secs 2024-07-26T14:28:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:22 smithi193 ceph-mon[37473]: pgmap v6503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:28:23.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:22 smithi134 ceph-mon[31966]: pgmap v6503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:28:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:22 smithi189 ceph-mon[36790]: pgmap v6503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:28:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10845.288012 secs 2024-07-26T14:28:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10845.288012 secs 2024-07-26T14:28:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10845.288012 secs 2024-07-26T14:28:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:24 smithi193 ceph-mon[37473]: pgmap v6504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:28:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:24 smithi134 ceph-mon[31966]: pgmap v6504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:28:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:24 smithi189 ceph-mon[36790]: pgmap v6504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:28:26.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:28:26.638+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:28:26.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:28:26.639+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:28:26.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:28:26.639+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:26 smithi193 ceph-mon[37473]: pgmap v6505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:28:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10864.711863 secs 2024-07-26T14:28:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:26 smithi134 ceph-mon[31966]: pgmap v6505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:28:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10864.711863 secs 2024-07-26T14:28:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:26 smithi189 ceph-mon[36790]: pgmap v6505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:28:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10864.711863 secs 2024-07-26T14:28:29.245 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:28 smithi189 ceph-mon[36790]: pgmap v6506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:28:29.245 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10850.288129 secs 2024-07-26T14:28:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:28 smithi193 ceph-mon[37473]: pgmap v6506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:28:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10850.288129 secs 2024-07-26T14:28:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:28 smithi134 ceph-mon[31966]: pgmap v6506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:28:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10850.288129 secs 2024-07-26T14:28:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:30 smithi193 ceph-mon[37473]: pgmap v6507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:28:31.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:30 smithi134 ceph-mon[31966]: pgmap v6507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:28:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:30 smithi189 ceph-mon[36790]: pgmap v6507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:28:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10869.711992 secs 2024-07-26T14:28:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10869.711992 secs 2024-07-26T14:28:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10869.711992 secs 2024-07-26T14:28:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:33 smithi193 ceph-mon[37473]: pgmap v6508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:33 smithi134 ceph-mon[31966]: pgmap v6508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:33 smithi189 ceph-mon[36790]: pgmap v6508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:34.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10855.288193 secs 2024-07-26T14:28:34.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10855.288193 secs 2024-07-26T14:28:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10855.288193 secs 2024-07-26T14:28:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:35 smithi193 ceph-mon[37473]: pgmap v6509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:35 smithi134 ceph-mon[31966]: pgmap v6509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:35 smithi189 ceph-mon[36790]: pgmap v6509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:37.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:28:37.072+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:28:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:28:37.072+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:28:37.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:28:37.072+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:37 smithi134 ceph-mon[31966]: pgmap v6510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10874.712118 secs 2024-07-26T14:28:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:37 smithi189 ceph-mon[36790]: pgmap v6510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10874.712118 secs 2024-07-26T14:28:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:37 smithi193 ceph-mon[37473]: pgmap v6510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10874.712118 secs 2024-07-26T14:28:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:39 smithi134 ceph-mon[31966]: pgmap v6511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10860.288312 secs 2024-07-26T14:28:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:39 smithi189 ceph-mon[36790]: pgmap v6511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10860.288312 secs 2024-07-26T14:28:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:39 smithi193 ceph-mon[37473]: pgmap v6511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10860.288312 secs 2024-07-26T14:28:41.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:41 smithi134 ceph-mon[31966]: pgmap v6512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:28:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:41 smithi189 ceph-mon[36790]: pgmap v6512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:28:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:41 smithi193 ceph-mon[37473]: pgmap v6512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:28:41.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:28:41.652+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:28:41.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:28:41.652+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:28:41.654 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:28:41.652+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10879.712246 secs 2024-07-26T14:28:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10879.712246 secs 2024-07-26T14:28:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10879.712246 secs 2024-07-26T14:28:43.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:43 smithi134 ceph-mon[31966]: pgmap v6513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:43 smithi189 ceph-mon[36790]: pgmap v6513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:43 smithi193 ceph-mon[37473]: pgmap v6513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:44.304 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10865.288374 secs 2024-07-26T14:28:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10865.288374 secs 2024-07-26T14:28:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10865.288374 secs 2024-07-26T14:28:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:45 smithi134 ceph-mon[31966]: pgmap v6514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:45 smithi189 ceph-mon[36790]: pgmap v6514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:45 smithi193 ceph-mon[37473]: pgmap v6514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:47 smithi134 ceph-mon[31966]: pgmap v6515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10884.712373 secs 2024-07-26T14:28:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:47 smithi189 ceph-mon[36790]: pgmap v6515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10884.712373 secs 2024-07-26T14:28:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:47 smithi193 ceph-mon[37473]: pgmap v6515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10884.712373 secs 2024-07-26T14:28:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:49 smithi134 ceph-mon[31966]: pgmap v6516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10870.288496 secs 2024-07-26T14:28:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:49 smithi189 ceph-mon[36790]: pgmap v6516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10870.288496 secs 2024-07-26T14:28:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:49 smithi193 ceph-mon[37473]: pgmap v6516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10870.288496 secs 2024-07-26T14:28:51.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:51 smithi134 ceph-mon[31966]: pgmap v6517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:28:51.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:51 smithi189 ceph-mon[36790]: pgmap v6517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:28:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:51 smithi193 ceph-mon[37473]: pgmap v6517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:28:52.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:28:52.088+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:28:52.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:28:52.088+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:28:52.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:28:52.088+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10889.712527 secs 2024-07-26T14:28:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10889.712527 secs 2024-07-26T14:28:52.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10889.712527 secs 2024-07-26T14:28:53.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:53 smithi134 ceph-mon[31966]: pgmap v6518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:53 smithi189 ceph-mon[36790]: pgmap v6518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:53 smithi193 ceph-mon[37473]: pgmap v6518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:54.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:28:54.223+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:28:54.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:28:54.223+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 86 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:28:54.227 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:28:54.226+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:28:54.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:28:54.227+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:28:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10875.288559 secs 2024-07-26T14:28:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10875.288559 secs 2024-07-26T14:28:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10875.288559 secs 2024-07-26T14:28:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:55 smithi134 ceph-mon[31966]: pgmap v6519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:55 smithi189 ceph-mon[36790]: pgmap v6519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:55 smithi193 ceph-mon[37473]: pgmap v6519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:56.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:28:56.668+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:28:56.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:28:56.668+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:28:56.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:28:56.668+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:28:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:57 smithi134 ceph-mon[31966]: pgmap v6520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10894.712674 secs 2024-07-26T14:28:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:57 smithi189 ceph-mon[36790]: pgmap v6520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10894.712674 secs 2024-07-26T14:28:57.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:57 smithi193 ceph-mon[37473]: pgmap v6520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:28:57.527 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10894.712674 secs 2024-07-26T14:28:59.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:59 smithi134 ceph-mon[31966]: pgmap v6521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:28:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10880.288702 secs 2024-07-26T14:28:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:59 smithi189 ceph-mon[36790]: pgmap v6521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:28:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10880.288702 secs 2024-07-26T14:28:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:59 smithi193 ceph-mon[37473]: pgmap v6521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:28:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:28:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10880.288702 secs 2024-07-26T14:29:01.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:01 smithi134 ceph-mon[31966]: pgmap v6522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:29:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:01 smithi189 ceph-mon[36790]: pgmap v6522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:29:01.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:01 smithi193 ceph-mon[37473]: pgmap v6522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:29:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10899.712835 secs 2024-07-26T14:29:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10899.712835 secs 2024-07-26T14:29:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10899.712835 secs 2024-07-26T14:29:03.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:03 smithi134 ceph-mon[31966]: pgmap v6523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:03 smithi189 ceph-mon[36790]: pgmap v6523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:03.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:03 smithi193 ceph-mon[37473]: pgmap v6523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10885.288799 secs 2024-07-26T14:29:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10885.288799 secs 2024-07-26T14:29:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10885.288799 secs 2024-07-26T14:29:04.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:29:04.750+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:29:04.752 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:29:04.750+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:29:04.752 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:29:04.751+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:29:04.752 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:29:04.751+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:29:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:05 smithi134 ceph-mon[31966]: pgmap v6524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:29:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:05 smithi189 ceph-mon[36790]: pgmap v6524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:29:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:05 smithi193 ceph-mon[37473]: pgmap v6524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:29:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:29:07.103+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:29:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:29:07.103+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:29:07.105 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:29:07.104+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:07 smithi134 ceph-mon[31966]: pgmap v6525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:29:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10904.712959 secs 2024-07-26T14:29:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:07 smithi189 ceph-mon[36790]: pgmap v6525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:29:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10904.712959 secs 2024-07-26T14:29:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:07 smithi193 ceph-mon[37473]: pgmap v6525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:29:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10904.712959 secs 2024-07-26T14:29:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:29:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:29:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:08.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:29:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:09 smithi189 ceph-mon[36790]: pgmap v6526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:29:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10890.288899 secs 2024-07-26T14:29:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:09 smithi193 ceph-mon[37473]: pgmap v6526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:29:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10890.288899 secs 2024-07-26T14:29:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:09 smithi134 ceph-mon[31966]: pgmap v6526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:29:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10890.288899 secs 2024-07-26T14:29:10.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:10 smithi134 ceph-mon[31966]: pgmap v6527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:29:10.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:10.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:10.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:10.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:10 smithi189 ceph-mon[36790]: pgmap v6527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:29:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:10 smithi193 ceph-mon[37473]: pgmap v6527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:29:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:11.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:29:11.683+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:29:11.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:29:11.683+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:29:11.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:29:11.683+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:11.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10909.713088 secs 2024-07-26T14:29:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10909.713088 secs 2024-07-26T14:29:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10909.713088 secs 2024-07-26T14:29:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:12 smithi134 ceph-mon[31966]: pgmap v6528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:29:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:29:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:29:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:12 smithi189 ceph-mon[36790]: pgmap v6528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:29:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:29:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:29:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:12 smithi193 ceph-mon[37473]: pgmap v6528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:29:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:29:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:29:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:29:13.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:13 smithi134 ceph-mon[31966]: pgmap v6529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T14:29:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:13 smithi134 ceph-mon[31966]: pgmap v6530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:29:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:13 smithi189 ceph-mon[36790]: pgmap v6529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T14:29:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:13 smithi189 ceph-mon[36790]: pgmap v6530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:29:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:13 smithi193 ceph-mon[37473]: pgmap v6529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T14:29:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:13 smithi193 ceph-mon[37473]: pgmap v6530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:29:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10895.289026 secs 2024-07-26T14:29:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10895.289026 secs 2024-07-26T14:29:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10895.289026 secs 2024-07-26T14:29:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:15 smithi189 ceph-mon[36790]: pgmap v6531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:29:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:15 smithi193 ceph-mon[37473]: pgmap v6531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:29:16.042 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:15 smithi134 ceph-mon[31966]: pgmap v6531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:29:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10914.713205 secs 2024-07-26T14:29:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10914.713205 secs 2024-07-26T14:29:17.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10914.713205 secs 2024-07-26T14:29:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:17 smithi193 ceph-mon[37473]: pgmap v6532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:29:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:17 smithi134 ceph-mon[31966]: pgmap v6532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:29:18.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:17 smithi189 ceph-mon[36790]: pgmap v6532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:29:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10900.289146 secs 2024-07-26T14:29:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:19 smithi193 ceph-mon[37473]: pgmap v6533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:29:20.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10900.289146 secs 2024-07-26T14:29:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:19 smithi134 ceph-mon[31966]: pgmap v6533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:29:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10900.289146 secs 2024-07-26T14:29:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:19 smithi189 ceph-mon[36790]: pgmap v6533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:29:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:21 smithi193 ceph-mon[37473]: pgmap v6534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:29:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10919.713332 secs 2024-07-26T14:29:22.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:29:22.119+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:29:22.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:29:22.119+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:22.120 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:29:22.119+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:29:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:21 smithi134 ceph-mon[31966]: pgmap v6534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:29:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10919.713332 secs 2024-07-26T14:29:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:21 smithi189 ceph-mon[36790]: pgmap v6534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:29:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10919.713332 secs 2024-07-26T14:29:23.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:23 smithi134 ceph-mon[31966]: pgmap v6535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:29:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:23 smithi193 ceph-mon[37473]: pgmap v6535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:29:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:23 smithi189 ceph-mon[36790]: pgmap v6535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:29:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10905.289223 secs 2024-07-26T14:29:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10905.289223 secs 2024-07-26T14:29:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10905.289223 secs 2024-07-26T14:29:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:25 smithi193 ceph-mon[37473]: pgmap v6536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:29:26.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:25 smithi134 ceph-mon[31966]: pgmap v6536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:29:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:25 smithi189 ceph-mon[36790]: pgmap v6536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:29:26.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:29:26.699+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:29:26.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:29:26.699+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:29:26.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:29:26.699+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10924.713456 secs 2024-07-26T14:29:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10924.713456 secs 2024-07-26T14:29:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10924.713456 secs 2024-07-26T14:29:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:27 smithi193 ceph-mon[37473]: pgmap v6537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:29:28.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:27 smithi134 ceph-mon[31966]: pgmap v6537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:29:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:27 smithi189 ceph-mon[36790]: pgmap v6537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:29:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10910.289344 secs 2024-07-26T14:29:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:29 smithi193 ceph-mon[37473]: pgmap v6538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:29:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10910.289344 secs 2024-07-26T14:29:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:29 smithi134 ceph-mon[31966]: pgmap v6538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:29:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10910.289344 secs 2024-07-26T14:29:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:29 smithi189 ceph-mon[36790]: pgmap v6538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:29:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:31 smithi193 ceph-mon[37473]: pgmap v6539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10929.713581 secs 2024-07-26T14:29:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:31 smithi134 ceph-mon[31966]: pgmap v6539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10929.713581 secs 2024-07-26T14:29:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:31 smithi189 ceph-mon[36790]: pgmap v6539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10929.713581 secs 2024-07-26T14:29:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:33 smithi193 ceph-mon[37473]: pgmap v6540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:29:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:33 smithi134 ceph-mon[31966]: pgmap v6540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:29:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:33 smithi189 ceph-mon[36790]: pgmap v6540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:29:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10915.289479 secs 2024-07-26T14:29:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10915.289479 secs 2024-07-26T14:29:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10915.289479 secs 2024-07-26T14:29:36.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:35 smithi134 ceph-mon[31966]: pgmap v6541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:29:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:35 smithi189 ceph-mon[36790]: pgmap v6541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:29:36.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:35 smithi193 ceph-mon[37473]: pgmap v6541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:29:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:29:37.123+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:29:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:29:37.123+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:29:37.124 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:29:37.123+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10934.713661 secs 2024-07-26T14:29:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10934.713661 secs 2024-07-26T14:29:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10934.713661 secs 2024-07-26T14:29:38.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:37 smithi134 ceph-mon[31966]: pgmap v6542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:37 smithi189 ceph-mon[36790]: pgmap v6542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:37 smithi193 ceph-mon[37473]: pgmap v6542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:40.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10920.289616 secs 2024-07-26T14:29:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:39 smithi134 ceph-mon[31966]: pgmap v6543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:29:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10920.289616 secs 2024-07-26T14:29:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:39 smithi189 ceph-mon[36790]: pgmap v6543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:29:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10920.289616 secs 2024-07-26T14:29:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:39 smithi193 ceph-mon[37473]: pgmap v6543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:29:41.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:29:41.715+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:29:41.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:29:41.715+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:41.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:29:41.715+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:29:41.974 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:41 smithi134 ceph-mon[31966]: pgmap v6544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:41.975 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10939.713805 secs 2024-07-26T14:29:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:41 smithi189 ceph-mon[36790]: pgmap v6544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10939.713805 secs 2024-07-26T14:29:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:41 smithi193 ceph-mon[37473]: pgmap v6544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10939.713805 secs 2024-07-26T14:29:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:43 smithi134 ceph-mon[31966]: pgmap v6545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:29:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:43 smithi189 ceph-mon[36790]: pgmap v6545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:29:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:43 smithi193 ceph-mon[37473]: pgmap v6545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:29:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10925.289709 secs 2024-07-26T14:29:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10925.289709 secs 2024-07-26T14:29:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10925.289709 secs 2024-07-26T14:29:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:45 smithi134 ceph-mon[31966]: pgmap v6546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:29:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:45 smithi189 ceph-mon[36790]: pgmap v6546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:29:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:45 smithi193 ceph-mon[37473]: pgmap v6546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:29:47.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10944.713929 secs 2024-07-26T14:29:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10944.713929 secs 2024-07-26T14:29:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10944.713929 secs 2024-07-26T14:29:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:47 smithi134 ceph-mon[31966]: pgmap v6547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:47 smithi189 ceph-mon[36790]: pgmap v6547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:47 smithi193 ceph-mon[37473]: pgmap v6547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:50.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10930.289835 secs 2024-07-26T14:29:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:49 smithi134 ceph-mon[31966]: pgmap v6548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:29:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10930.289835 secs 2024-07-26T14:29:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:49 smithi189 ceph-mon[36790]: pgmap v6548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:29:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10930.289835 secs 2024-07-26T14:29:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:49 smithi193 ceph-mon[37473]: pgmap v6548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:29:52.127 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:29:52.126+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:29:52.127 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:29:52.126+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:29:52.127 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:29:52.126+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:51 smithi134 ceph-mon[31966]: pgmap v6549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10949.714069 secs 2024-07-26T14:29:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:51 smithi189 ceph-mon[36790]: pgmap v6549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10949.714069 secs 2024-07-26T14:29:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:51 smithi193 ceph-mon[37473]: pgmap v6549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10949.714069 secs 2024-07-26T14:29:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:53 smithi134 ceph-mon[31966]: pgmap v6550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:29:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:53 smithi189 ceph-mon[36790]: pgmap v6550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:29:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:53 smithi193 ceph-mon[37473]: pgmap v6550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:29:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10935.290005 secs 2024-07-26T14:29:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10935.290005 secs 2024-07-26T14:29:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10935.290005 secs 2024-07-26T14:29:56.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:55 smithi134 ceph-mon[31966]: pgmap v6551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:29:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:55 smithi189 ceph-mon[36790]: pgmap v6551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:29:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:55 smithi193 ceph-mon[37473]: pgmap v6551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:29:56.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:29:56.720+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:29:56.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:29:56.720+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:29:56.722 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:29:56.721+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:29:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10954.714186 secs 2024-07-26T14:29:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10954.714186 secs 2024-07-26T14:29:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10954.714186 secs 2024-07-26T14:29:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:57 smithi134 ceph-mon[31966]: pgmap v6552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:57 smithi189 ceph-mon[36790]: pgmap v6552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:29:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:57 smithi193 ceph-mon[37473]: pgmap v6552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10940.290115 secs 2024-07-26T14:30:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:29:59 smithi134 ceph-mon[31966]: pgmap v6553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10940.290115 secs 2024-07-26T14:30:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:29:59 smithi189 ceph-mon[36790]: pgmap v6553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10940.290115 secs 2024-07-26T14:30:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:29:59 smithi193 ceph-mon[37473]: pgmap v6553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:30:01.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:30:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:30:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:01 smithi189 ceph-mon[36790]: pgmap v6554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10959.714335 secs 2024-07-26T14:30:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:01 smithi193 ceph-mon[37473]: pgmap v6554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10959.714335 secs 2024-07-26T14:30:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:01 smithi134 ceph-mon[31966]: pgmap v6554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10959.714335 secs 2024-07-26T14:30:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:03 smithi189 ceph-mon[36790]: pgmap v6555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:03 smithi193 ceph-mon[37473]: pgmap v6555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:03 smithi134 ceph-mon[31966]: pgmap v6555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10945.290181 secs 2024-07-26T14:30:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10945.290181 secs 2024-07-26T14:30:05.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10945.290181 secs 2024-07-26T14:30:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:05 smithi193 ceph-mon[37473]: pgmap v6556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:30:06.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:05 smithi134 ceph-mon[31966]: pgmap v6556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:30:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:05 smithi189 ceph-mon[36790]: pgmap v6556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:30:07.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:30:07.134+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:30:07.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:30:07.134+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:07.136 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:30:07.134+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:30:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10964.714467 secs 2024-07-26T14:30:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10964.714467 secs 2024-07-26T14:30:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10964.714467 secs 2024-07-26T14:30:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:07 smithi193 ceph-mon[37473]: pgmap v6557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:08.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:07 smithi134 ceph-mon[31966]: pgmap v6557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:07 smithi189 ceph-mon[36790]: pgmap v6557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10950.290313 secs 2024-07-26T14:30:09.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10950.290313 secs 2024-07-26T14:30:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10950.290313 secs 2024-07-26T14:30:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:10 smithi193 ceph-mon[37473]: pgmap v6558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:10.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:10 smithi134 ceph-mon[31966]: pgmap v6558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:10 smithi189 ceph-mon[36790]: pgmap v6558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:11.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:30:11.731+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:30:11.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:30:11.731+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:11.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:30:11.731+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:30:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:12 smithi193 ceph-mon[37473]: pgmap v6559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10969.714551 secs 2024-07-26T14:30:12.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:12 smithi134 ceph-mon[31966]: pgmap v6559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10969.714551 secs 2024-07-26T14:30:12.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:12 smithi189 ceph-mon[36790]: pgmap v6559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10969.714551 secs 2024-07-26T14:30:13.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:30:13.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:30:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:30:14.356 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:14 smithi193 ceph-mon[37473]: pgmap v6560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:14.356 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10955.290406 secs 2024-07-26T14:30:14.357 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:30:14.357 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:30:14.357 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:30:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:14 smithi134 ceph-mon[31966]: pgmap v6560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10955.290406 secs 2024-07-26T14:30:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:30:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:30:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:30:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:14 smithi189 ceph-mon[36790]: pgmap v6560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10955.290406 secs 2024-07-26T14:30:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:30:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:30:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:30:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:15 smithi134 ceph-mon[31966]: pgmap v6561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:30:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:15 smithi134 ceph-mon[31966]: pgmap v6562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:30:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:15 smithi189 ceph-mon[36790]: pgmap v6561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:30:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:15 smithi189 ceph-mon[36790]: pgmap v6562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:30:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:15 smithi193 ceph-mon[37473]: pgmap v6561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:30:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:15 smithi193 ceph-mon[37473]: pgmap v6562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:30:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:17 smithi134 ceph-mon[31966]: pgmap v6563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:30:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10974.714640 secs 2024-07-26T14:30:17.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:17 smithi189 ceph-mon[36790]: pgmap v6563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:30:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10974.714640 secs 2024-07-26T14:30:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:17 smithi193 ceph-mon[37473]: pgmap v6563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:30:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10974.714640 secs 2024-07-26T14:30:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:19 smithi134 ceph-mon[31966]: pgmap v6564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:30:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10960.290532 secs 2024-07-26T14:30:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:19 smithi189 ceph-mon[36790]: pgmap v6564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:30:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10960.290532 secs 2024-07-26T14:30:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:19 smithi193 ceph-mon[37473]: pgmap v6564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:30:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10960.290532 secs 2024-07-26T14:30:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:21 smithi134 ceph-mon[31966]: pgmap v6565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:30:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:21 smithi189 ceph-mon[36790]: pgmap v6565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:30:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:21 smithi193 ceph-mon[37473]: pgmap v6565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:30:22.151 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:30:22.150+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:30:22.151 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:30:22.150+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:22.151 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:30:22.150+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:30:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10979.714768 secs 2024-07-26T14:30:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10979.714768 secs 2024-07-26T14:30:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10979.714768 secs 2024-07-26T14:30:23.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:23 smithi134 ceph-mon[31966]: pgmap v6566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:30:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:23 smithi189 ceph-mon[36790]: pgmap v6566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:30:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:23 smithi193 ceph-mon[37473]: pgmap v6566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:30:24.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:30:24.238+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:30:24.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:30:24.239+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 87 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:30:24.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:30:24.241+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:30:24.242 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:30:24.241+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:30:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10965.290627 secs 2024-07-26T14:30:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10965.290627 secs 2024-07-26T14:30:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10965.290627 secs 2024-07-26T14:30:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:25 smithi134 ceph-mon[31966]: pgmap v6567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:30:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:25 smithi189 ceph-mon[36790]: pgmap v6567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:30:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:25 smithi193 ceph-mon[37473]: pgmap v6567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:30:26.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:30:26.738+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:30:26.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:30:26.738+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:26.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:30:26.738+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:30:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:27 smithi134 ceph-mon[31966]: pgmap v6568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10984.714862 secs 2024-07-26T14:30:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:27 smithi189 ceph-mon[36790]: pgmap v6568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10984.714862 secs 2024-07-26T14:30:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:27 smithi193 ceph-mon[37473]: pgmap v6568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10984.714862 secs 2024-07-26T14:30:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:29 smithi134 ceph-mon[31966]: pgmap v6569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10970.290733 secs 2024-07-26T14:30:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:29 smithi189 ceph-mon[36790]: pgmap v6569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10970.290733 secs 2024-07-26T14:30:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:29 smithi193 ceph-mon[37473]: pgmap v6569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10970.290733 secs 2024-07-26T14:30:31.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:31 smithi134 ceph-mon[31966]: pgmap v6570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:30:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:31 smithi189 ceph-mon[36790]: pgmap v6570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:30:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:31 smithi193 ceph-mon[37473]: pgmap v6570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:30:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10989.714981 secs 2024-07-26T14:30:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10989.714981 secs 2024-07-26T14:30:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10989.714981 secs 2024-07-26T14:30:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:33 smithi134 ceph-mon[31966]: pgmap v6571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:33 smithi189 ceph-mon[36790]: pgmap v6571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:33 smithi193 ceph-mon[37473]: pgmap v6571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:34.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10975.290878 secs 2024-07-26T14:30:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10975.290878 secs 2024-07-26T14:30:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10975.290878 secs 2024-07-26T14:30:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:30:34.766+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:30:34.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:30:34.766+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:30:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:30:34.767+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:30:34.768 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:30:34.767+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:30:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:35 smithi134 ceph-mon[31966]: pgmap v6572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:35.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:35 smithi189 ceph-mon[36790]: pgmap v6572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:35 smithi193 ceph-mon[37473]: pgmap v6572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:30:37.165+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:30:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:30:37.166+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:30:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:30:37.166+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:37 smithi134 ceph-mon[31966]: pgmap v6573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10994.715074 secs 2024-07-26T14:30:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:37 smithi189 ceph-mon[36790]: pgmap v6573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10994.715074 secs 2024-07-26T14:30:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:37 smithi193 ceph-mon[37473]: pgmap v6573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10994.715074 secs 2024-07-26T14:30:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:39 smithi134 ceph-mon[31966]: pgmap v6574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10980.291003 secs 2024-07-26T14:30:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:39 smithi189 ceph-mon[36790]: pgmap v6574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10980.291003 secs 2024-07-26T14:30:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:39 smithi193 ceph-mon[37473]: pgmap v6574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10980.291003 secs 2024-07-26T14:30:41.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:41 smithi134 ceph-mon[31966]: pgmap v6575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:30:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:41 smithi189 ceph-mon[36790]: pgmap v6575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:30:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:41 smithi193 ceph-mon[37473]: pgmap v6575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:30:41.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:30:41.753+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:30:41.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:30:41.753+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:30:41.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:30:41.753+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 10999.715204 secs 2024-07-26T14:30:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 10999.715204 secs 2024-07-26T14:30:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 10999.715204 secs 2024-07-26T14:30:43.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:43 smithi134 ceph-mon[31966]: pgmap v6576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:43 smithi189 ceph-mon[36790]: pgmap v6576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:43 smithi193 ceph-mon[37473]: pgmap v6576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10985.291124 secs 2024-07-26T14:30:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10985.291124 secs 2024-07-26T14:30:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10985.291124 secs 2024-07-26T14:30:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:45 smithi189 ceph-mon[36790]: pgmap v6577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:45 smithi193 ceph-mon[37473]: pgmap v6577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:45 smithi134 ceph-mon[31966]: pgmap v6577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:47 smithi189 ceph-mon[36790]: pgmap v6578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11004.715340 secs 2024-07-26T14:30:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:47 smithi193 ceph-mon[37473]: pgmap v6578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11004.715340 secs 2024-07-26T14:30:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:47 smithi134 ceph-mon[31966]: pgmap v6578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11004.715340 secs 2024-07-26T14:30:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:48 smithi193 ceph-mon[37473]: pgmap v6579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:48 smithi134 ceph-mon[31966]: pgmap v6579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:48 smithi189 ceph-mon[36790]: pgmap v6579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10990.291195 secs 2024-07-26T14:30:49.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10990.291195 secs 2024-07-26T14:30:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10990.291195 secs 2024-07-26T14:30:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:50 smithi193 ceph-mon[37473]: pgmap v6580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:30:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:50 smithi134 ceph-mon[31966]: pgmap v6580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:30:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:50 smithi189 ceph-mon[36790]: pgmap v6580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:30:51.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11009.715432 secs 2024-07-26T14:30:51.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11009.715432 secs 2024-07-26T14:30:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11009.715432 secs 2024-07-26T14:30:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:30:52.181+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:30:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:30:52.181+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:52.182 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:30:52.181+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:30:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:52 smithi134 ceph-mon[31966]: pgmap v6581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:52 smithi189 ceph-mon[36790]: pgmap v6581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:52 smithi193 ceph-mon[37473]: pgmap v6581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 10995.291315 secs 2024-07-26T14:30:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:54 smithi134 ceph-mon[31966]: pgmap v6582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 10995.291315 secs 2024-07-26T14:30:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:54 smithi189 ceph-mon[36790]: pgmap v6582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 10995.291315 secs 2024-07-26T14:30:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:54 smithi193 ceph-mon[37473]: pgmap v6582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:56.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:30:56.754+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:30:56.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:30:56.754+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:30:56.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:30:56.754+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:30:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:56 smithi134 ceph-mon[31966]: pgmap v6583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11014.715551 secs 2024-07-26T14:30:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:56 smithi189 ceph-mon[36790]: pgmap v6583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11014.715551 secs 2024-07-26T14:30:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:56 smithi193 ceph-mon[37473]: pgmap v6583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:30:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11014.715551 secs 2024-07-26T14:30:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:58 smithi134 ceph-mon[31966]: pgmap v6584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:58 smithi189 ceph-mon[36790]: pgmap v6584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:30:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:58 smithi193 ceph-mon[37473]: pgmap v6584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:30:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11000.291434 secs 2024-07-26T14:31:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:30:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11000.291434 secs 2024-07-26T14:31:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:30:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11000.291434 secs 2024-07-26T14:31:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:00 smithi134 ceph-mon[31966]: pgmap v6585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:31:01.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:00 smithi193 ceph-mon[37473]: pgmap v6585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:31:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:00 smithi189 ceph-mon[36790]: pgmap v6585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:31:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11019.715640 secs 2024-07-26T14:31:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11019.715640 secs 2024-07-26T14:31:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11019.715640 secs 2024-07-26T14:31:03.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:02 smithi134 ceph-mon[31966]: pgmap v6586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:02 smithi189 ceph-mon[36790]: pgmap v6586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:02 smithi193 ceph-mon[37473]: pgmap v6586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11005.291556 secs 2024-07-26T14:31:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:04 smithi134 ceph-mon[31966]: pgmap v6587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11005.291556 secs 2024-07-26T14:31:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:04 smithi189 ceph-mon[36790]: pgmap v6587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11005.291556 secs 2024-07-26T14:31:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:04 smithi193 ceph-mon[37473]: pgmap v6587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:06 smithi134 ceph-mon[31966]: pgmap v6588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11024.715782 secs 2024-07-26T14:31:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:06 smithi189 ceph-mon[36790]: pgmap v6588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11024.715782 secs 2024-07-26T14:31:07.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:31:07.197+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:31:07.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:31:07.197+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:31:07.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:31:07.197+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:06 smithi193 ceph-mon[37473]: pgmap v6588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11024.715782 secs 2024-07-26T14:31:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:08 smithi134 ceph-mon[31966]: pgmap v6589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:09.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:08 smithi189 ceph-mon[36790]: pgmap v6589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:08 smithi193 ceph-mon[37473]: pgmap v6589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11010.291674 secs 2024-07-26T14:31:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11010.291674 secs 2024-07-26T14:31:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11010.291674 secs 2024-07-26T14:31:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:10 smithi189 ceph-mon[36790]: pgmap v6590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:31:11.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:10 smithi193 ceph-mon[37473]: pgmap v6590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:31:11.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:10 smithi134 ceph-mon[31966]: pgmap v6590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:31:11.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:31:11.770+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:31:11.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:31:11.770+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:11.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:31:11.770+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:31:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11029.715921 secs 2024-07-26T14:31:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11029.715921 secs 2024-07-26T14:31:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11029.715921 secs 2024-07-26T14:31:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:12 smithi193 ceph-mon[37473]: pgmap v6591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:13.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:12 smithi134 ceph-mon[31966]: pgmap v6591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:12 smithi189 ceph-mon[36790]: pgmap v6591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11015.291793 secs 2024-07-26T14:31:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:14 smithi193 ceph-mon[37473]: pgmap v6592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:31:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11015.291793 secs 2024-07-26T14:31:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:14 smithi134 ceph-mon[31966]: pgmap v6592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:31:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11015.291793 secs 2024-07-26T14:31:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:14 smithi189 ceph-mon[36790]: pgmap v6592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:31:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:31:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:31:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:15 smithi193 ceph-mon[37473]: pgmap v6593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:31:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:15 smithi193 ceph-mon[37473]: pgmap v6594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:31:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:31:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:31:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:31:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:15 smithi134 ceph-mon[31966]: pgmap v6593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:31:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:15 smithi134 ceph-mon[31966]: pgmap v6594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:31:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:31:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:31:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:31:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:15 smithi189 ceph-mon[36790]: pgmap v6593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:31:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:15 smithi189 ceph-mon[36790]: pgmap v6594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:31:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:31:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11034.716057 secs 2024-07-26T14:31:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11034.716057 secs 2024-07-26T14:31:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11034.716057 secs 2024-07-26T14:31:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:17 smithi193 ceph-mon[37473]: pgmap v6595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:31:18.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:17 smithi134 ceph-mon[31966]: pgmap v6595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:31:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:17 smithi189 ceph-mon[36790]: pgmap v6595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:31:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11020.291911 secs 2024-07-26T14:31:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11020.291911 secs 2024-07-26T14:31:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11020.291911 secs 2024-07-26T14:31:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:19 smithi193 ceph-mon[37473]: pgmap v6596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:31:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:19 smithi134 ceph-mon[31966]: pgmap v6596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:31:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:19 smithi189 ceph-mon[36790]: pgmap v6596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:31:22.210 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:31:22.209+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:31:22.210 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:31:22.209+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:31:22.210 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:31:22.209+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:22 smithi193 ceph-mon[37473]: pgmap v6597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:31:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11039.716182 secs 2024-07-26T14:31:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:22 smithi134 ceph-mon[31966]: pgmap v6597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:31:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11039.716182 secs 2024-07-26T14:31:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:22 smithi189 ceph-mon[36790]: pgmap v6597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:31:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11039.716182 secs 2024-07-26T14:31:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:24 smithi193 ceph-mon[37473]: pgmap v6598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:31:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11025.292036 secs 2024-07-26T14:31:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:24 smithi134 ceph-mon[31966]: pgmap v6598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:31:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11025.292036 secs 2024-07-26T14:31:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:24 smithi189 ceph-mon[36790]: pgmap v6598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:31:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11025.292036 secs 2024-07-26T14:31:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:26 smithi193 ceph-mon[37473]: pgmap v6599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:31:26.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:26 smithi134 ceph-mon[31966]: pgmap v6599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:31:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:26 smithi189 ceph-mon[36790]: pgmap v6599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:31:26.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:31:26.784+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:31:26.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:31:26.784+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:31:26.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:31:26.784+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11044.716344 secs 2024-07-26T14:31:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11044.716344 secs 2024-07-26T14:31:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11044.716344 secs 2024-07-26T14:31:28.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:28 smithi134 ceph-mon[31966]: pgmap v6600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:28 smithi189 ceph-mon[36790]: pgmap v6600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:28 smithi193 ceph-mon[37473]: pgmap v6600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11030.292132 secs 2024-07-26T14:31:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11030.292132 secs 2024-07-26T14:31:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11030.292132 secs 2024-07-26T14:31:30.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:30 smithi189 ceph-mon[36790]: pgmap v6601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:30.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:30 smithi134 ceph-mon[31966]: pgmap v6601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:30 smithi193 ceph-mon[37473]: pgmap v6601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:32.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:32 smithi193 ceph-mon[37473]: pgmap v6602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:32.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11049.716455 secs 2024-07-26T14:31:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:32 smithi134 ceph-mon[31966]: pgmap v6602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11049.716455 secs 2024-07-26T14:31:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:32 smithi189 ceph-mon[36790]: pgmap v6602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11049.716455 secs 2024-07-26T14:31:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:34 smithi134 ceph-mon[31966]: pgmap v6603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11035.292200 secs 2024-07-26T14:31:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:34 smithi189 ceph-mon[36790]: pgmap v6603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11035.292200 secs 2024-07-26T14:31:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:34 smithi193 ceph-mon[37473]: pgmap v6603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11035.292200 secs 2024-07-26T14:31:36.328 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:36 smithi134 ceph-mon[31966]: pgmap v6604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:36 smithi189 ceph-mon[36790]: pgmap v6604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:36 smithi193 ceph-mon[37473]: pgmap v6604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:37.213 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:31:37.212+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:31:37.213 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:31:37.212+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:31:37.213 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:31:37.212+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11054.716574 secs 2024-07-26T14:31:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11054.716574 secs 2024-07-26T14:31:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11054.716574 secs 2024-07-26T14:31:38.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:38 smithi134 ceph-mon[31966]: pgmap v6605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:38 smithi189 ceph-mon[36790]: pgmap v6605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:38 smithi193 ceph-mon[37473]: pgmap v6605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11040.292263 secs 2024-07-26T14:31:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11040.292263 secs 2024-07-26T14:31:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11040.292263 secs 2024-07-26T14:31:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:40 smithi134 ceph-mon[31966]: pgmap v6606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:40 smithi189 ceph-mon[36790]: pgmap v6606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:40 smithi193 ceph-mon[37473]: pgmap v6606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:41.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:31:41.799+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:31:41.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:31:41.799+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:31:41.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:31:41.799+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:42 smithi134 ceph-mon[31966]: pgmap v6607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11059.716688 secs 2024-07-26T14:31:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:42 smithi189 ceph-mon[36790]: pgmap v6607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11059.716688 secs 2024-07-26T14:31:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:42 smithi193 ceph-mon[37473]: pgmap v6607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11059.716688 secs 2024-07-26T14:31:44.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:44 smithi134 ceph-mon[31966]: pgmap v6608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11045.292402 secs 2024-07-26T14:31:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:44 smithi189 ceph-mon[36790]: pgmap v6608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11045.292402 secs 2024-07-26T14:31:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:44 smithi193 ceph-mon[37473]: pgmap v6608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11045.292402 secs 2024-07-26T14:31:46.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:46 smithi134 ceph-mon[31966]: pgmap v6609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:46 smithi189 ceph-mon[36790]: pgmap v6609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:46 smithi193 ceph-mon[37473]: pgmap v6609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11064.716766 secs 2024-07-26T14:31:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11064.716766 secs 2024-07-26T14:31:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11064.716766 secs 2024-07-26T14:31:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:48 smithi134 ceph-mon[31966]: pgmap v6610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:48 smithi189 ceph-mon[36790]: pgmap v6610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:48 smithi193 ceph-mon[37473]: pgmap v6610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11050.292522 secs 2024-07-26T14:31:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11050.292522 secs 2024-07-26T14:31:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11050.292522 secs 2024-07-26T14:31:50.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:50 smithi134 ceph-mon[31966]: pgmap v6611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:50 smithi189 ceph-mon[36790]: pgmap v6611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:50 smithi193 ceph-mon[37473]: pgmap v6611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:31:52.228 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:31:52.227+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:31:52.228 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:31:52.227+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:52.229 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:31:52.227+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:31:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:52 smithi134 ceph-mon[31966]: pgmap v6612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11069.716890 secs 2024-07-26T14:31:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:52 smithi189 ceph-mon[36790]: pgmap v6612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11069.716890 secs 2024-07-26T14:31:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:52 smithi193 ceph-mon[37473]: pgmap v6612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11069.716890 secs 2024-07-26T14:31:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:54 smithi134 ceph-mon[31966]: pgmap v6613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11055.292646 secs 2024-07-26T14:31:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:54 smithi189 ceph-mon[36790]: pgmap v6613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11055.292646 secs 2024-07-26T14:31:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:54 smithi193 ceph-mon[37473]: pgmap v6613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:31:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11055.292646 secs 2024-07-26T14:31:56.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:56 smithi134 ceph-mon[31966]: pgmap v6614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:56 smithi189 ceph-mon[36790]: pgmap v6614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:56 smithi193 ceph-mon[37473]: pgmap v6614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:31:56.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:31:56.800+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:31:56.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:31:56.800+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:31:56.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:31:56.800+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:31:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11074.717015 secs 2024-07-26T14:31:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11074.717015 secs 2024-07-26T14:31:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11074.717015 secs 2024-07-26T14:31:58.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:58 smithi134 ceph-mon[31966]: pgmap v6615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:58 smithi189 ceph-mon[36790]: pgmap v6615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:58 smithi193 ceph-mon[37473]: pgmap v6615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:31:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:31:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11060.292767 secs 2024-07-26T14:31:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:31:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11060.292767 secs 2024-07-26T14:31:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:31:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11060.292767 secs 2024-07-26T14:32:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:00 smithi189 ceph-mon[36790]: pgmap v6616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:00 smithi193 ceph-mon[37473]: pgmap v6616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:00 smithi134 ceph-mon[31966]: pgmap v6616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:02 smithi189 ceph-mon[36790]: pgmap v6617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11079.717146 secs 2024-07-26T14:32:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:02 smithi193 ceph-mon[37473]: pgmap v6617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11079.717146 secs 2024-07-26T14:32:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:02 smithi134 ceph-mon[31966]: pgmap v6617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11079.717146 secs 2024-07-26T14:32:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:04 smithi189 ceph-mon[36790]: pgmap v6618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11065.292860 secs 2024-07-26T14:32:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:04 smithi193 ceph-mon[37473]: pgmap v6618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11065.292860 secs 2024-07-26T14:32:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:04 smithi134 ceph-mon[31966]: pgmap v6618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11065.292860 secs 2024-07-26T14:32:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:06 smithi189 ceph-mon[36790]: pgmap v6619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:06 smithi193 ceph-mon[37473]: pgmap v6619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:06 smithi134 ceph-mon[31966]: pgmap v6619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:32:07.243+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:32:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:32:07.243+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:32:07.244 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:32:07.243+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11084.717261 secs 2024-07-26T14:32:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11084.717261 secs 2024-07-26T14:32:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11084.717261 secs 2024-07-26T14:32:08.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:08 smithi134 ceph-mon[31966]: pgmap v6620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:08 smithi193 ceph-mon[37473]: pgmap v6620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:08 smithi189 ceph-mon[36790]: pgmap v6620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11070.292988 secs 2024-07-26T14:32:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:09 smithi193 ceph-mon[37473]: pgmap v6621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11070.292988 secs 2024-07-26T14:32:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:09 smithi134 ceph-mon[31966]: pgmap v6621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11070.292988 secs 2024-07-26T14:32:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:09 smithi189 ceph-mon[36790]: pgmap v6621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:11.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:32:11.816+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:32:11.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:32:11.816+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:32:11.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:32:11.816+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:12.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:12 smithi134 ceph-mon[31966]: pgmap v6622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11089.717390 secs 2024-07-26T14:32:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:12 smithi189 ceph-mon[36790]: pgmap v6622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11089.717390 secs 2024-07-26T14:32:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:12 smithi193 ceph-mon[37473]: pgmap v6622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11089.717390 secs 2024-07-26T14:32:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:14 smithi134 ceph-mon[31966]: pgmap v6623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11075.293106 secs 2024-07-26T14:32:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:14 smithi189 ceph-mon[36790]: pgmap v6623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11075.293106 secs 2024-07-26T14:32:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:14 smithi193 ceph-mon[37473]: pgmap v6623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11075.293106 secs 2024-07-26T14:32:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:15 smithi134 ceph-mon[31966]: pgmap v6624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:32:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:15 smithi189 ceph-mon[36790]: pgmap v6624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:32:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:15 smithi193 ceph-mon[37473]: pgmap v6624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:32:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:32:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:32:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:16 smithi134 ceph-mon[31966]: pgmap v6625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:32:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:16 smithi134 ceph-mon[31966]: pgmap v6626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:32:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:32:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:32:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:32:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:16 smithi189 ceph-mon[36790]: pgmap v6625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:32:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:16 smithi189 ceph-mon[36790]: pgmap v6626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:32:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:32:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:32:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:32:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:16 smithi193 ceph-mon[37473]: pgmap v6625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:32:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:16 smithi193 ceph-mon[37473]: pgmap v6626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:32:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:32:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11094.717513 secs 2024-07-26T14:32:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11094.717513 secs 2024-07-26T14:32:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11094.717513 secs 2024-07-26T14:32:18.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:18 smithi134 ceph-mon[31966]: pgmap v6627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:32:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:18 smithi189 ceph-mon[36790]: pgmap v6627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:32:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:18 smithi193 ceph-mon[37473]: pgmap v6627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:32:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11080.293175 secs 2024-07-26T14:32:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11080.293175 secs 2024-07-26T14:32:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11080.293175 secs 2024-07-26T14:32:20.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:20 smithi134 ceph-mon[31966]: pgmap v6628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:32:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:20 smithi189 ceph-mon[36790]: pgmap v6628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:32:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:20 smithi193 ceph-mon[37473]: pgmap v6628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:32:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11099.717660 secs 2024-07-26T14:32:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11099.717660 secs 2024-07-26T14:32:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11099.717660 secs 2024-07-26T14:32:22.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:32:22.259+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:32:22.260 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:32:22.259+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:22.260 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:32:22.259+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:32:22.920 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:22 smithi134 ceph-mon[31966]: pgmap v6629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:32:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:22 smithi189 ceph-mon[36790]: pgmap v6629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:32:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:22 smithi193 ceph-mon[37473]: pgmap v6629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:32:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11085.293303 secs 2024-07-26T14:32:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11085.293303 secs 2024-07-26T14:32:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11085.293303 secs 2024-07-26T14:32:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:25 smithi134 ceph-mon[31966]: pgmap v6630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:32:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:25 smithi189 ceph-mon[36790]: pgmap v6630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:32:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:25 smithi193 ceph-mon[37473]: pgmap v6630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:32:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:26 smithi193 ceph-mon[37473]: pgmap v6631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:32:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:26 smithi134 ceph-mon[31966]: pgmap v6631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:32:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:26 smithi189 ceph-mon[36790]: pgmap v6631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:32:26.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:32:26.825+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:32:26.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:32:26.825+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:26.827 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:32:26.825+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:32:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11104.717804 secs 2024-07-26T14:32:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11104.717804 secs 2024-07-26T14:32:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11104.717804 secs 2024-07-26T14:32:28.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:28 smithi134 ceph-mon[31966]: pgmap v6632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:28 smithi189 ceph-mon[36790]: pgmap v6632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:28 smithi193 ceph-mon[37473]: pgmap v6632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11090.293424 secs 2024-07-26T14:32:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11090.293424 secs 2024-07-26T14:32:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11090.293424 secs 2024-07-26T14:32:30.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:30 smithi189 ceph-mon[36790]: pgmap v6633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:30 smithi134 ceph-mon[31966]: pgmap v6633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:30 smithi193 ceph-mon[37473]: pgmap v6633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11109.717933 secs 2024-07-26T14:32:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11109.717933 secs 2024-07-26T14:32:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11109.717933 secs 2024-07-26T14:32:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:32 smithi134 ceph-mon[31966]: pgmap v6634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:32.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:32 smithi193 ceph-mon[37473]: pgmap v6634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:32 smithi189 ceph-mon[36790]: pgmap v6634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:34.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11095.293502 secs 2024-07-26T14:32:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11095.293502 secs 2024-07-26T14:32:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11095.293502 secs 2024-07-26T14:32:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:35 smithi134 ceph-mon[31966]: pgmap v6635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:35 smithi189 ceph-mon[36790]: pgmap v6635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:35 smithi193 ceph-mon[37473]: pgmap v6635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:36 smithi193 ceph-mon[37473]: pgmap v6636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:36 smithi134 ceph-mon[31966]: pgmap v6636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:36 smithi189 ceph-mon[36790]: pgmap v6636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:37.262 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:32:37.261+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:32:37.262 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:32:37.261+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:32:37.262 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:32:37.261+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11114.718055 secs 2024-07-26T14:32:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11114.718055 secs 2024-07-26T14:32:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11114.718055 secs 2024-07-26T14:32:38.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:38 smithi134 ceph-mon[31966]: pgmap v6637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:38 smithi189 ceph-mon[36790]: pgmap v6637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:38 smithi193 ceph-mon[37473]: pgmap v6637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11100.293642 secs 2024-07-26T14:32:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11100.293642 secs 2024-07-26T14:32:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11100.293642 secs 2024-07-26T14:32:40.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:40 smithi134 ceph-mon[31966]: pgmap v6638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:40 smithi189 ceph-mon[36790]: pgmap v6638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:40 smithi193 ceph-mon[37473]: pgmap v6638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:41.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:32:41.832+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:32:41.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:32:41.832+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:41.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:32:41.832+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:32:41.834 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11119.718186 secs 2024-07-26T14:32:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11119.718186 secs 2024-07-26T14:32:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11119.718186 secs 2024-07-26T14:32:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:42 smithi134 ceph-mon[31966]: pgmap v6639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:42 smithi189 ceph-mon[36790]: pgmap v6639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:42 smithi193 ceph-mon[37473]: pgmap v6639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11105.293765 secs 2024-07-26T14:32:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11105.293765 secs 2024-07-26T14:32:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11105.293765 secs 2024-07-26T14:32:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:45 smithi134 ceph-mon[31966]: pgmap v6640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:45 smithi189 ceph-mon[36790]: pgmap v6640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:45 smithi193 ceph-mon[37473]: pgmap v6640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:46 smithi193 ceph-mon[37473]: pgmap v6641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:46 smithi134 ceph-mon[31966]: pgmap v6641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:46 smithi189 ceph-mon[36790]: pgmap v6641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11124.718321 secs 2024-07-26T14:32:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11124.718321 secs 2024-07-26T14:32:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11124.718321 secs 2024-07-26T14:32:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:48 smithi134 ceph-mon[31966]: pgmap v6642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:48 smithi189 ceph-mon[36790]: pgmap v6642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:48 smithi193 ceph-mon[37473]: pgmap v6642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11110.293868 secs 2024-07-26T14:32:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11110.293868 secs 2024-07-26T14:32:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11110.293868 secs 2024-07-26T14:32:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:50 smithi134 ceph-mon[31966]: pgmap v6643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:50 smithi189 ceph-mon[36790]: pgmap v6643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:50 smithi193 ceph-mon[37473]: pgmap v6643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:51.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11129.718451 secs 2024-07-26T14:32:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11129.718451 secs 2024-07-26T14:32:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11129.718451 secs 2024-07-26T14:32:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:32:52.275+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:32:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:32:52.275+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:52.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:32:52.275+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:32:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:52 smithi134 ceph-mon[31966]: pgmap v6644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:52 smithi189 ceph-mon[36790]: pgmap v6644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:52 smithi193 ceph-mon[37473]: pgmap v6644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:32:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11115.294013 secs 2024-07-26T14:32:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11115.294013 secs 2024-07-26T14:32:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11115.294013 secs 2024-07-26T14:32:55.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:55 smithi134 ceph-mon[31966]: pgmap v6645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:55 smithi189 ceph-mon[36790]: pgmap v6645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:55 smithi193 ceph-mon[37473]: pgmap v6645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:32:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:56 smithi193 ceph-mon[37473]: pgmap v6646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:56 smithi134 ceph-mon[31966]: pgmap v6646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:56 smithi189 ceph-mon[36790]: pgmap v6646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:32:56.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:32:56.843+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:32:56.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:32:56.843+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:32:56.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:32:56.843+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:32:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11134.718586 secs 2024-07-26T14:32:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11134.718586 secs 2024-07-26T14:32:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11134.718586 secs 2024-07-26T14:32:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:58 smithi134 ceph-mon[31966]: pgmap v6647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:58 smithi189 ceph-mon[36790]: pgmap v6647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:58 smithi193 ceph-mon[37473]: pgmap v6647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:32:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:32:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11120.294080 secs 2024-07-26T14:32:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:32:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11120.294080 secs 2024-07-26T14:32:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:32:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11120.294080 secs 2024-07-26T14:33:00.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:00 smithi189 ceph-mon[36790]: pgmap v6648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:00 smithi134 ceph-mon[31966]: pgmap v6648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:00 smithi193 ceph-mon[37473]: pgmap v6648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:01.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11139.718705 secs 2024-07-26T14:33:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11139.718705 secs 2024-07-26T14:33:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11139.718705 secs 2024-07-26T14:33:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:02 smithi134 ceph-mon[31966]: pgmap v6649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:02.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:02 smithi193 ceph-mon[37473]: pgmap v6649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:02 smithi189 ceph-mon[36790]: pgmap v6649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11125.294148 secs 2024-07-26T14:33:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11125.294148 secs 2024-07-26T14:33:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11125.294148 secs 2024-07-26T14:33:05.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:05 smithi134 ceph-mon[31966]: pgmap v6650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:05 smithi189 ceph-mon[36790]: pgmap v6650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:05 smithi193 ceph-mon[37473]: pgmap v6650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:06 smithi134 ceph-mon[31966]: pgmap v6651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:06.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:06 smithi189 ceph-mon[36790]: pgmap v6651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:06 smithi193 ceph-mon[37473]: pgmap v6651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:33:07.291+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:33:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:33:07.291+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:33:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:33:07.291+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11144.718784 secs 2024-07-26T14:33:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11144.718784 secs 2024-07-26T14:33:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11144.718784 secs 2024-07-26T14:33:08.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:08 smithi134 ceph-mon[31966]: pgmap v6652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:33:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:08 smithi189 ceph-mon[36790]: pgmap v6652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:33:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:08 smithi193 ceph-mon[37473]: pgmap v6652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:33:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11130.294233 secs 2024-07-26T14:33:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11130.294233 secs 2024-07-26T14:33:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11130.294233 secs 2024-07-26T14:33:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:10 smithi134 ceph-mon[31966]: pgmap v6653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:10 smithi189 ceph-mon[36790]: pgmap v6653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:10 smithi193 ceph-mon[37473]: pgmap v6653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:11.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:33:11.852+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:33:11.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:33:11.852+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:33:11.853 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:33:11.852+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:11.854 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11149.718909 secs 2024-07-26T14:33:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11149.718909 secs 2024-07-26T14:33:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11149.718909 secs 2024-07-26T14:33:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:12 smithi134 ceph-mon[31966]: pgmap v6654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:12.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:12 smithi189 ceph-mon[36790]: pgmap v6654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:12 smithi193 ceph-mon[37473]: pgmap v6654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:33:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11135.294298 secs 2024-07-26T14:33:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11135.294298 secs 2024-07-26T14:33:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11135.294298 secs 2024-07-26T14:33:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:15 smithi134 ceph-mon[31966]: pgmap v6655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:15 smithi189 ceph-mon[36790]: pgmap v6655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:15 smithi193 ceph-mon[37473]: pgmap v6655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:33:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:16 smithi134 ceph-mon[31966]: pgmap v6656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:16 smithi189 ceph-mon[36790]: pgmap v6656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:16 smithi193 ceph-mon[37473]: pgmap v6656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11154.719060 secs 2024-07-26T14:33:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:33:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11154.719060 secs 2024-07-26T14:33:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:33:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11154.719060 secs 2024-07-26T14:33:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:33:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:33:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:33:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:18 smithi134 ceph-mon[31966]: pgmap v6657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:33:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:18 smithi134 ceph-mon[31966]: pgmap v6658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:33:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:33:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:33:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:33:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:18 smithi189 ceph-mon[36790]: pgmap v6657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:33:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:18 smithi189 ceph-mon[36790]: pgmap v6658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:33:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:33:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:33:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:33:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:18 smithi193 ceph-mon[37473]: pgmap v6657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:33:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:18 smithi193 ceph-mon[37473]: pgmap v6658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:33:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:33:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11140.294419 secs 2024-07-26T14:33:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11140.294419 secs 2024-07-26T14:33:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11140.294419 secs 2024-07-26T14:33:20.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:20 smithi134 ceph-mon[31966]: pgmap v6659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:33:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:20 smithi189 ceph-mon[36790]: pgmap v6659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:33:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:20 smithi193 ceph-mon[37473]: pgmap v6659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:33:22.302 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:33:22.301+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:33:22.302 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:33:22.301+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:33:22.303 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:33:22.301+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11159.719195 secs 2024-07-26T14:33:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:22 smithi189 ceph-mon[36790]: pgmap v6660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:33:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11159.719195 secs 2024-07-26T14:33:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:22 smithi193 ceph-mon[37473]: pgmap v6660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:33:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11159.719195 secs 2024-07-26T14:33:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:22 smithi134 ceph-mon[31966]: pgmap v6660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:33:24.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:33:24.268+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:33:24.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:33:24.268+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:33:24.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:33:24.269+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 88 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:33:24.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:33:24.269+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:33:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:24 smithi134 ceph-mon[31966]: pgmap v6661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11145.294556 secs 2024-07-26T14:33:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:24 smithi189 ceph-mon[36790]: pgmap v6661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11145.294556 secs 2024-07-26T14:33:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:24 smithi193 ceph-mon[37473]: pgmap v6661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11145.294556 secs 2024-07-26T14:33:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:26 smithi189 ceph-mon[36790]: pgmap v6662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:26 smithi193 ceph-mon[37473]: pgmap v6662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:26.865 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:33:26.864+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:33:26.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:33:26.864+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:33:26.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:33:26.864+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:26.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:26 smithi134 ceph-mon[31966]: pgmap v6662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:33:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11164.719329 secs 2024-07-26T14:33:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11164.719329 secs 2024-07-26T14:33:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11164.719329 secs 2024-07-26T14:33:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:28 smithi193 ceph-mon[37473]: pgmap v6663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:33:28.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:28 smithi134 ceph-mon[31966]: pgmap v6663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:33:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:28 smithi189 ceph-mon[36790]: pgmap v6663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:33:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11150.294674 secs 2024-07-26T14:33:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11150.294674 secs 2024-07-26T14:33:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11150.294674 secs 2024-07-26T14:33:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:30 smithi193 ceph-mon[37473]: pgmap v6664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:33:30.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:30 smithi189 ceph-mon[36790]: pgmap v6664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:33:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:30 smithi134 ceph-mon[31966]: pgmap v6664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:33:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11169.719458 secs 2024-07-26T14:33:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:32 smithi193 ceph-mon[37473]: pgmap v6665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11169.719458 secs 2024-07-26T14:33:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:32 smithi134 ceph-mon[31966]: pgmap v6665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11169.719458 secs 2024-07-26T14:33:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:32 smithi189 ceph-mon[36790]: pgmap v6665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:34 smithi193 ceph-mon[37473]: pgmap v6666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:33:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11155.294814 secs 2024-07-26T14:33:34.800 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:33:34.799+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:33:34.800 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:33:34.799+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:33:34.801 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:33:34.800+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:33:34.801 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:33:34.800+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:33:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:34 smithi134 ceph-mon[31966]: pgmap v6666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:33:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11155.294814 secs 2024-07-26T14:33:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:34 smithi189 ceph-mon[36790]: pgmap v6666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:33:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11155.294814 secs 2024-07-26T14:33:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:36 smithi193 ceph-mon[37473]: pgmap v6667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:36 smithi134 ceph-mon[31966]: pgmap v6667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:36 smithi189 ceph-mon[36790]: pgmap v6667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:37.313 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:33:37.312+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:33:37.313 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:33:37.312+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:33:37.314 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:33:37.312+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11174.719583 secs 2024-07-26T14:33:37.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11174.719583 secs 2024-07-26T14:33:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11174.719583 secs 2024-07-26T14:33:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:38 smithi193 ceph-mon[37473]: pgmap v6668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:33:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:38 smithi134 ceph-mon[31966]: pgmap v6668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:33:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:38 smithi189 ceph-mon[36790]: pgmap v6668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:33:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11160.294885 secs 2024-07-26T14:33:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11160.294885 secs 2024-07-26T14:33:39.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11160.294885 secs 2024-07-26T14:33:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:40 smithi193 ceph-mon[37473]: pgmap v6669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:33:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:40 smithi134 ceph-mon[31966]: pgmap v6669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:33:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:40 smithi189 ceph-mon[36790]: pgmap v6669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:33:41.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:33:41.867+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:33:41.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:33:41.867+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:41.869 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:33:41.867+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:33:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11179.719705 secs 2024-07-26T14:33:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:42 smithi193 ceph-mon[37473]: pgmap v6670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11179.719705 secs 2024-07-26T14:33:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:42 smithi134 ceph-mon[31966]: pgmap v6670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11179.719705 secs 2024-07-26T14:33:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:42 smithi189 ceph-mon[36790]: pgmap v6670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:44 smithi134 ceph-mon[31966]: pgmap v6671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:33:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11165.295004 secs 2024-07-26T14:33:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:44 smithi189 ceph-mon[36790]: pgmap v6671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:33:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11165.295004 secs 2024-07-26T14:33:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:44 smithi193 ceph-mon[37473]: pgmap v6671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:33:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11165.295004 secs 2024-07-26T14:33:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:46 smithi134 ceph-mon[31966]: pgmap v6672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11184.719845 secs 2024-07-26T14:33:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:46 smithi189 ceph-mon[36790]: pgmap v6672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11184.719845 secs 2024-07-26T14:33:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:46 smithi193 ceph-mon[37473]: pgmap v6672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11184.719845 secs 2024-07-26T14:33:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:48 smithi134 ceph-mon[31966]: pgmap v6673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:33:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:48 smithi189 ceph-mon[36790]: pgmap v6673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:33:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:48 smithi193 ceph-mon[37473]: pgmap v6673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:33:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11170.295129 secs 2024-07-26T14:33:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11170.295129 secs 2024-07-26T14:33:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11170.295129 secs 2024-07-26T14:33:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:50 smithi134 ceph-mon[31966]: pgmap v6674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:33:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:50 smithi189 ceph-mon[36790]: pgmap v6674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:33:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:50 smithi193 ceph-mon[37473]: pgmap v6674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:33:51.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11189.719974 secs 2024-07-26T14:33:51.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11189.719974 secs 2024-07-26T14:33:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11189.719974 secs 2024-07-26T14:33:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:33:52.320+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:33:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:33:52.320+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:33:52.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:33:52.320+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:52 smithi134 ceph-mon[31966]: pgmap v6675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:52.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:52 smithi189 ceph-mon[36790]: pgmap v6675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:52 smithi193 ceph-mon[37473]: pgmap v6675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:54 smithi134 ceph-mon[31966]: pgmap v6676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:33:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11175.295249 secs 2024-07-26T14:33:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:54 smithi189 ceph-mon[36790]: pgmap v6676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:33:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11175.295249 secs 2024-07-26T14:33:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:54 smithi193 ceph-mon[37473]: pgmap v6676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:33:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11175.295249 secs 2024-07-26T14:33:56.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:33:56.883+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:33:56.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:33:56.883+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:33:56.884 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:33:56.883+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:33:56.884 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:56 smithi134 ceph-mon[31966]: pgmap v6677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:56.884 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11194.720099 secs 2024-07-26T14:33:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:56 smithi189 ceph-mon[36790]: pgmap v6677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11194.720099 secs 2024-07-26T14:33:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:56 smithi193 ceph-mon[37473]: pgmap v6677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:33:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11194.720099 secs 2024-07-26T14:33:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:58 smithi134 ceph-mon[31966]: pgmap v6678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:33:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:58 smithi189 ceph-mon[36790]: pgmap v6678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:33:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:58 smithi193 ceph-mon[37473]: pgmap v6678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:33:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:33:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11180.295332 secs 2024-07-26T14:33:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:33:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11180.295332 secs 2024-07-26T14:34:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:33:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11180.295332 secs 2024-07-26T14:34:00.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:00 smithi134 ceph-mon[31966]: pgmap v6679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:34:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:00 smithi189 ceph-mon[36790]: pgmap v6679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:34:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:00 smithi193 ceph-mon[37473]: pgmap v6679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:34:01.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11199.720220 secs 2024-07-26T14:34:01.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11199.720220 secs 2024-07-26T14:34:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11199.720220 secs 2024-07-26T14:34:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:02 smithi134 ceph-mon[31966]: pgmap v6680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:02.924 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:02 smithi193 ceph-mon[37473]: pgmap v6680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:02 smithi189 ceph-mon[36790]: pgmap v6680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:04 smithi134 ceph-mon[31966]: pgmap v6681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:34:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11185.295452 secs 2024-07-26T14:34:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:04 smithi189 ceph-mon[36790]: pgmap v6681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:34:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11185.295452 secs 2024-07-26T14:34:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:04 smithi193 ceph-mon[37473]: pgmap v6681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:34:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11185.295452 secs 2024-07-26T14:34:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:06 smithi134 ceph-mon[31966]: pgmap v6682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11204.720335 secs 2024-07-26T14:34:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:06 smithi189 ceph-mon[36790]: pgmap v6682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11204.720335 secs 2024-07-26T14:34:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:06 smithi193 ceph-mon[37473]: pgmap v6682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11204.720335 secs 2024-07-26T14:34:07.336 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:34:07.335+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:34:07.336 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:34:07.335+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:34:07.336 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:34:07.335+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:08 smithi134 ceph-mon[31966]: pgmap v6683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:34:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:08 smithi189 ceph-mon[36790]: pgmap v6683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:34:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:08 smithi193 ceph-mon[37473]: pgmap v6683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:34:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11190.295574 secs 2024-07-26T14:34:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11190.295574 secs 2024-07-26T14:34:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11190.295574 secs 2024-07-26T14:34:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:10 smithi134 ceph-mon[31966]: pgmap v6684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:34:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:10 smithi189 ceph-mon[36790]: pgmap v6684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:34:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:10 smithi193 ceph-mon[37473]: pgmap v6684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:34:11.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:34:11.897+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:34:11.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:34:11.897+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:11.900 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:34:11.897+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:34:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11209.720378 secs 2024-07-26T14:34:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11209.720378 secs 2024-07-26T14:34:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11209.720378 secs 2024-07-26T14:34:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:12 smithi134 ceph-mon[31966]: pgmap v6685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:12 smithi189 ceph-mon[36790]: pgmap v6685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:12 smithi193 ceph-mon[37473]: pgmap v6685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:14 smithi134 ceph-mon[31966]: pgmap v6686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:34:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11195.295687 secs 2024-07-26T14:34:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:14 smithi189 ceph-mon[36790]: pgmap v6686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:34:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11195.295687 secs 2024-07-26T14:34:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:14 smithi193 ceph-mon[37473]: pgmap v6686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:34:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11195.295687 secs 2024-07-26T14:34:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:16 smithi134 ceph-mon[31966]: pgmap v6687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11214.720500 secs 2024-07-26T14:34:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:16 smithi189 ceph-mon[36790]: pgmap v6687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11214.720500 secs 2024-07-26T14:34:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:16 smithi193 ceph-mon[37473]: pgmap v6687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11214.720500 secs 2024-07-26T14:34:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:34:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:34:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:34:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:18 smithi189 ceph-mon[36790]: pgmap v6688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:34:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:34:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:34:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:34:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:18 smithi193 ceph-mon[37473]: pgmap v6688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:34:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:34:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:34:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:34:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:18 smithi134 ceph-mon[31966]: pgmap v6688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:34:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:34:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:34:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:34:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11200.295781 secs 2024-07-26T14:34:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:19 smithi189 ceph-mon[36790]: pgmap v6689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:34:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:19 smithi189 ceph-mon[36790]: pgmap v6690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:34:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11200.295781 secs 2024-07-26T14:34:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:19 smithi193 ceph-mon[37473]: pgmap v6689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:34:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:19 smithi193 ceph-mon[37473]: pgmap v6690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:34:20.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11200.295781 secs 2024-07-26T14:34:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:19 smithi134 ceph-mon[31966]: pgmap v6689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:34:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:19 smithi134 ceph-mon[31966]: pgmap v6690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:34:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:21 smithi193 ceph-mon[37473]: pgmap v6691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:34:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11219.720613 secs 2024-07-26T14:34:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:21 smithi134 ceph-mon[31966]: pgmap v6691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:34:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11219.720613 secs 2024-07-26T14:34:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:21 smithi189 ceph-mon[36790]: pgmap v6691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:34:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11219.720613 secs 2024-07-26T14:34:22.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:34:22.350+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:34:22.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:34:22.350+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:34:22.352 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:34:22.351+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:23 smithi193 ceph-mon[37473]: pgmap v6692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:34:24.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:23 smithi134 ceph-mon[31966]: pgmap v6692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:34:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:23 smithi189 ceph-mon[36790]: pgmap v6692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:34:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11205.295902 secs 2024-07-26T14:34:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11205.295902 secs 2024-07-26T14:34:25.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11205.295902 secs 2024-07-26T14:34:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:25 smithi193 ceph-mon[37473]: pgmap v6693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:34:26.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:25 smithi134 ceph-mon[31966]: pgmap v6693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:34:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:25 smithi189 ceph-mon[36790]: pgmap v6693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:34:26.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:34:26.913+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:34:26.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:34:26.913+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:26.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:34:26.913+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:34:27.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11224.720732 secs 2024-07-26T14:34:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11224.720732 secs 2024-07-26T14:34:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11224.720732 secs 2024-07-26T14:34:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:27 smithi193 ceph-mon[37473]: pgmap v6694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:28.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:27 smithi134 ceph-mon[31966]: pgmap v6694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:27 smithi189 ceph-mon[36790]: pgmap v6694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11210.296016 secs 2024-07-26T14:34:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:29 smithi193 ceph-mon[37473]: pgmap v6695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:34:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11210.296016 secs 2024-07-26T14:34:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:29 smithi134 ceph-mon[31966]: pgmap v6695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:34:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11210.296016 secs 2024-07-26T14:34:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:29 smithi189 ceph-mon[36790]: pgmap v6695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:34:31.995 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:31 smithi189 ceph-mon[36790]: pgmap v6696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:31.995 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11229.720849 secs 2024-07-26T14:34:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:31 smithi193 ceph-mon[37473]: pgmap v6696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11229.720849 secs 2024-07-26T14:34:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:31 smithi134 ceph-mon[31966]: pgmap v6696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11229.720849 secs 2024-07-26T14:34:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:33 smithi193 ceph-mon[37473]: pgmap v6697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:34:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:33 smithi134 ceph-mon[31966]: pgmap v6697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:34:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:33 smithi189 ceph-mon[36790]: pgmap v6697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:34:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11215.296079 secs 2024-07-26T14:34:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11215.296079 secs 2024-07-26T14:34:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11215.296079 secs 2024-07-26T14:34:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:35 smithi193 ceph-mon[37473]: pgmap v6698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:34:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:35 smithi134 ceph-mon[31966]: pgmap v6698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:34:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:35 smithi189 ceph-mon[36790]: pgmap v6698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:34:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11234.720943 secs 2024-07-26T14:34:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11234.720943 secs 2024-07-26T14:34:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11234.720943 secs 2024-07-26T14:34:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:34:37.366+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:34:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:34:37.366+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:34:37.367 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:34:37.366+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:38.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:37 smithi134 ceph-mon[31966]: pgmap v6699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:37 smithi189 ceph-mon[36790]: pgmap v6699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:37 smithi193 ceph-mon[37473]: pgmap v6699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11220.296153 secs 2024-07-26T14:34:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:39 smithi134 ceph-mon[31966]: pgmap v6700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:34:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11220.296153 secs 2024-07-26T14:34:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:39 smithi189 ceph-mon[36790]: pgmap v6700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:34:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11220.296153 secs 2024-07-26T14:34:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:39 smithi193 ceph-mon[37473]: pgmap v6700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:34:41.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:34:41.929+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:34:41.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:34:41.929+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:34:41.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:34:41.929+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:41 smithi134 ceph-mon[31966]: pgmap v6701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11239.721062 secs 2024-07-26T14:34:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:41 smithi189 ceph-mon[36790]: pgmap v6701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11239.721062 secs 2024-07-26T14:34:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:41 smithi193 ceph-mon[37473]: pgmap v6701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11239.721062 secs 2024-07-26T14:34:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:43 smithi134 ceph-mon[31966]: pgmap v6702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:34:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:43 smithi189 ceph-mon[36790]: pgmap v6702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:34:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:43 smithi193 ceph-mon[37473]: pgmap v6702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:34:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11225.296289 secs 2024-07-26T14:34:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11225.296289 secs 2024-07-26T14:34:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11225.296289 secs 2024-07-26T14:34:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:45 smithi134 ceph-mon[31966]: pgmap v6703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:34:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:45 smithi189 ceph-mon[36790]: pgmap v6703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:34:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:45 smithi193 ceph-mon[37473]: pgmap v6703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:34:47.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11244.721175 secs 2024-07-26T14:34:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11244.721175 secs 2024-07-26T14:34:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11244.721175 secs 2024-07-26T14:34:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:47 smithi134 ceph-mon[31966]: pgmap v6704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:34:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:47 smithi189 ceph-mon[36790]: pgmap v6704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:34:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:47 smithi193 ceph-mon[37473]: pgmap v6704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:34:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11230.296435 secs 2024-07-26T14:34:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:49 smithi134 ceph-mon[31966]: pgmap v6705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:34:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11230.296435 secs 2024-07-26T14:34:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:49 smithi189 ceph-mon[36790]: pgmap v6705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:34:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11230.296435 secs 2024-07-26T14:34:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:49 smithi193 ceph-mon[37473]: pgmap v6705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:34:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:51 smithi134 ceph-mon[31966]: pgmap v6706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:34:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11249.721333 secs 2024-07-26T14:34:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:51 smithi189 ceph-mon[36790]: pgmap v6706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:34:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11249.721333 secs 2024-07-26T14:34:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:51 smithi193 ceph-mon[37473]: pgmap v6706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:34:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11249.721333 secs 2024-07-26T14:34:52.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:34:52.382+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:34:52.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:34:52.382+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:34:52.383 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:34:52.382+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:53 smithi134 ceph-mon[31966]: pgmap v6707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:34:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:53 smithi189 ceph-mon[36790]: pgmap v6707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:34:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:53 smithi193 ceph-mon[37473]: pgmap v6707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:34:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11235.296553 secs 2024-07-26T14:34:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11235.296553 secs 2024-07-26T14:34:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11235.296553 secs 2024-07-26T14:34:56.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:55 smithi134 ceph-mon[31966]: pgmap v6708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:34:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:55 smithi189 ceph-mon[36790]: pgmap v6708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:34:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:55 smithi193 ceph-mon[37473]: pgmap v6708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:34:56.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:34:56.944+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:34:56.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:34:56.944+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:34:56.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:34:56.944+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:34:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11254.721448 secs 2024-07-26T14:34:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11254.721448 secs 2024-07-26T14:34:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11254.721448 secs 2024-07-26T14:34:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:57 smithi134 ceph-mon[31966]: pgmap v6709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:57 smithi189 ceph-mon[36790]: pgmap v6709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:34:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:57 smithi193 ceph-mon[37473]: pgmap v6709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11240.296625 secs 2024-07-26T14:35:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:34:59 smithi134 ceph-mon[31966]: pgmap v6710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11240.296625 secs 2024-07-26T14:35:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:34:59 smithi189 ceph-mon[36790]: pgmap v6710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11240.296625 secs 2024-07-26T14:35:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:34:59 smithi193 ceph-mon[37473]: pgmap v6710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:01 smithi134 ceph-mon[31966]: pgmap v6711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11259.721570 secs 2024-07-26T14:35:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:01 smithi189 ceph-mon[36790]: pgmap v6711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11259.721570 secs 2024-07-26T14:35:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:01 smithi193 ceph-mon[37473]: pgmap v6711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11259.721570 secs 2024-07-26T14:35:04.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:03 smithi134 ceph-mon[31966]: pgmap v6712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:03 smithi189 ceph-mon[36790]: pgmap v6712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:03 smithi193 ceph-mon[37473]: pgmap v6712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11245.296775 secs 2024-07-26T14:35:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11245.296775 secs 2024-07-26T14:35:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11245.296775 secs 2024-07-26T14:35:06.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:05 smithi134 ceph-mon[31966]: pgmap v6713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:35:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:05 smithi189 ceph-mon[36790]: pgmap v6713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:35:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:05 smithi193 ceph-mon[37473]: pgmap v6713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:35:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11264.721686 secs 2024-07-26T14:35:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11264.721686 secs 2024-07-26T14:35:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11264.721686 secs 2024-07-26T14:35:07.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:35:07.393+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:35:07.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:35:07.393+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:07.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:35:07.393+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:35:08.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:07 smithi134 ceph-mon[31966]: pgmap v6714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:07 smithi189 ceph-mon[36790]: pgmap v6714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:07 smithi193 ceph-mon[37473]: pgmap v6714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11250.296873 secs 2024-07-26T14:35:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:09 smithi134 ceph-mon[31966]: pgmap v6715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11250.296873 secs 2024-07-26T14:35:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:09 smithi189 ceph-mon[36790]: pgmap v6715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11250.296873 secs 2024-07-26T14:35:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:09 smithi193 ceph-mon[37473]: pgmap v6715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:11.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:35:11.960+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:35:11.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:35:11.960+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:35:11.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:35:11.960+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:11 smithi189 ceph-mon[36790]: pgmap v6716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11269.721811 secs 2024-07-26T14:35:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:11 smithi193 ceph-mon[37473]: pgmap v6716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11269.721811 secs 2024-07-26T14:35:12.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:11 smithi134 ceph-mon[31966]: pgmap v6716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11269.721811 secs 2024-07-26T14:35:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:13 smithi189 ceph-mon[36790]: pgmap v6717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:14.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:13 smithi134 ceph-mon[31966]: pgmap v6717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:13 smithi193 ceph-mon[37473]: pgmap v6717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11255.296991 secs 2024-07-26T14:35:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11255.296991 secs 2024-07-26T14:35:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11255.296991 secs 2024-07-26T14:35:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:15 smithi193 ceph-mon[37473]: pgmap v6718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:35:16.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:15 smithi134 ceph-mon[31966]: pgmap v6718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:35:16.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:15 smithi189 ceph-mon[36790]: pgmap v6718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:35:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11274.721929 secs 2024-07-26T14:35:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11274.721929 secs 2024-07-26T14:35:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11274.721929 secs 2024-07-26T14:35:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:17 smithi193 ceph-mon[37473]: pgmap v6719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:18.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:17 smithi134 ceph-mon[31966]: pgmap v6719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:17 smithi189 ceph-mon[36790]: pgmap v6719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:35:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:35:19.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:35:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11260.297111 secs 2024-07-26T14:35:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:19 smithi193 ceph-mon[37473]: pgmap v6720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:35:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:35:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:35:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11260.297111 secs 2024-07-26T14:35:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:19 smithi134 ceph-mon[31966]: pgmap v6720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:35:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:35:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:35:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11260.297111 secs 2024-07-26T14:35:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:19 smithi189 ceph-mon[36790]: pgmap v6720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:35:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:35:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:35:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:35:21.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:20 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:35:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:20 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:35:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:20 smithi193 ceph-mon[37473]: pgmap v6721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:35:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:20 smithi193 ceph-mon[37473]: pgmap v6722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:35:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:20 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:35:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:20 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:35:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:20 smithi134 ceph-mon[31966]: pgmap v6721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:35:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:20 smithi134 ceph-mon[31966]: pgmap v6722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:35:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:20 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:35:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:20 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:35:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:20 smithi189 ceph-mon[36790]: pgmap v6721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:35:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:20 smithi189 ceph-mon[36790]: pgmap v6722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:35:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11279.722071 secs 2024-07-26T14:35:22.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:35:22.409+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:35:22.410 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:35:22.409+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:22.410 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:35:22.409+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:35:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11279.722071 secs 2024-07-26T14:35:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11279.722071 secs 2024-07-26T14:35:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:23 smithi193 ceph-mon[37473]: pgmap v6723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:35:23.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:23 smithi134 ceph-mon[31966]: pgmap v6723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:35:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:23 smithi189 ceph-mon[36790]: pgmap v6723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:35:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11265.297234 secs 2024-07-26T14:35:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11265.297234 secs 2024-07-26T14:35:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11265.297234 secs 2024-07-26T14:35:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:25 smithi134 ceph-mon[31966]: pgmap v6724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:35:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:25 smithi189 ceph-mon[36790]: pgmap v6724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:35:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:25 smithi193 ceph-mon[37473]: pgmap v6724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:35:26.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:35:26.976+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:35:26.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:35:26.976+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:35:26.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:35:26.976+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:27 smithi134 ceph-mon[31966]: pgmap v6725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:35:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11284.722181 secs 2024-07-26T14:35:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:27 smithi189 ceph-mon[36790]: pgmap v6725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:35:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11284.722181 secs 2024-07-26T14:35:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:27 smithi193 ceph-mon[37473]: pgmap v6725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:35:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11284.722181 secs 2024-07-26T14:35:29.288 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:29 smithi134 ceph-mon[31966]: pgmap v6726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:35:29.288 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11270.297353 secs 2024-07-26T14:35:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:29 smithi189 ceph-mon[36790]: pgmap v6726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:35:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11270.297353 secs 2024-07-26T14:35:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:29 smithi193 ceph-mon[37473]: pgmap v6726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:35:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11270.297353 secs 2024-07-26T14:35:31.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:31 smithi134 ceph-mon[31966]: pgmap v6727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:35:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:31 smithi189 ceph-mon[36790]: pgmap v6727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:35:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:31 smithi193 ceph-mon[37473]: pgmap v6727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:35:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11289.722315 secs 2024-07-26T14:35:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11289.722315 secs 2024-07-26T14:35:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11289.722315 secs 2024-07-26T14:35:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:33 smithi134 ceph-mon[31966]: pgmap v6728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:33 smithi189 ceph-mon[36790]: pgmap v6728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:33 smithi193 ceph-mon[37473]: pgmap v6728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:34.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11275.297478 secs 2024-07-26T14:35:34.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11275.297478 secs 2024-07-26T14:35:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11275.297478 secs 2024-07-26T14:35:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:35 smithi134 ceph-mon[31966]: pgmap v6729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:35 smithi189 ceph-mon[36790]: pgmap v6729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:35 smithi193 ceph-mon[37473]: pgmap v6729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:37.416 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:35:37.415+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:35:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:35:37.415+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:35:37.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:35:37.415+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:37 smithi134 ceph-mon[31966]: pgmap v6730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11294.722354 secs 2024-07-26T14:35:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:37 smithi189 ceph-mon[36790]: pgmap v6730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11294.722354 secs 2024-07-26T14:35:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:37 smithi193 ceph-mon[37473]: pgmap v6730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11294.722354 secs 2024-07-26T14:35:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:39 smithi134 ceph-mon[31966]: pgmap v6731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11280.297600 secs 2024-07-26T14:35:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:39 smithi189 ceph-mon[36790]: pgmap v6731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11280.297600 secs 2024-07-26T14:35:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:39 smithi193 ceph-mon[37473]: pgmap v6731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11280.297600 secs 2024-07-26T14:35:41.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:41 smithi134 ceph-mon[31966]: pgmap v6732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:35:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:41 smithi189 ceph-mon[36790]: pgmap v6732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:35:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:41 smithi193 ceph-mon[37473]: pgmap v6732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:35:41.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:35:41.991+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:35:41.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:35:41.991+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:35:41.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:35:41.991+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11299.722478 secs 2024-07-26T14:35:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11299.722478 secs 2024-07-26T14:35:42.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11299.722478 secs 2024-07-26T14:35:43.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:43 smithi134 ceph-mon[31966]: pgmap v6733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:43 smithi189 ceph-mon[36790]: pgmap v6733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:43 smithi193 ceph-mon[37473]: pgmap v6733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:44.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11285.297724 secs 2024-07-26T14:35:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11285.297724 secs 2024-07-26T14:35:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11285.297724 secs 2024-07-26T14:35:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:45 smithi134 ceph-mon[31966]: pgmap v6734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:45 smithi189 ceph-mon[36790]: pgmap v6734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:45 smithi193 ceph-mon[37473]: pgmap v6734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:47 smithi134 ceph-mon[31966]: pgmap v6735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11304.722604 secs 2024-07-26T14:35:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:47 smithi189 ceph-mon[36790]: pgmap v6735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11304.722604 secs 2024-07-26T14:35:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:47 smithi193 ceph-mon[37473]: pgmap v6735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11304.722604 secs 2024-07-26T14:35:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:49 smithi134 ceph-mon[31966]: pgmap v6736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11290.297844 secs 2024-07-26T14:35:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:49 smithi189 ceph-mon[36790]: pgmap v6736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11290.297844 secs 2024-07-26T14:35:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:49 smithi193 ceph-mon[37473]: pgmap v6736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11290.297844 secs 2024-07-26T14:35:51.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:51 smithi134 ceph-mon[31966]: pgmap v6737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:35:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:51 smithi189 ceph-mon[36790]: pgmap v6737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:35:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:51 smithi193 ceph-mon[37473]: pgmap v6737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:35:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11309.722718 secs 2024-07-26T14:35:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:35:52.431+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:35:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:35:52.431+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:35:52.432 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:35:52.431+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11309.722718 secs 2024-07-26T14:35:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11309.722718 secs 2024-07-26T14:35:53.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:53 smithi134 ceph-mon[31966]: pgmap v6738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:53 smithi189 ceph-mon[36790]: pgmap v6738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:53 smithi193 ceph-mon[37473]: pgmap v6738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11295.297967 secs 2024-07-26T14:35:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11295.297967 secs 2024-07-26T14:35:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11295.297967 secs 2024-07-26T14:35:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:55 smithi134 ceph-mon[31966]: pgmap v6739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:55 smithi189 ceph-mon[36790]: pgmap v6739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:55 smithi193 ceph-mon[37473]: pgmap v6739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:57.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:35:57.000+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:35:57.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:35:57.000+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:35:57.002 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:35:57.000+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:35:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:57 smithi134 ceph-mon[31966]: pgmap v6740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11314.722862 secs 2024-07-26T14:35:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:57 smithi189 ceph-mon[36790]: pgmap v6740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11314.722862 secs 2024-07-26T14:35:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:57 smithi193 ceph-mon[37473]: pgmap v6740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:35:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11314.722862 secs 2024-07-26T14:35:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:59 smithi189 ceph-mon[36790]: pgmap v6741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:35:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11300.298092 secs 2024-07-26T14:35:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:59 smithi193 ceph-mon[37473]: pgmap v6741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:35:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11300.298092 secs 2024-07-26T14:35:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:59 smithi134 ceph-mon[31966]: pgmap v6741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:35:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:35:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11300.298092 secs 2024-07-26T14:36:01.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:01 smithi134 ceph-mon[31966]: pgmap v6742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:36:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:01 smithi189 ceph-mon[36790]: pgmap v6742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:36:01.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:01 smithi193 ceph-mon[37473]: pgmap v6742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:36:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11319.722979 secs 2024-07-26T14:36:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11319.722979 secs 2024-07-26T14:36:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11319.722979 secs 2024-07-26T14:36:03.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:03 smithi193 ceph-mon[37473]: pgmap v6743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:03.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:03 smithi134 ceph-mon[31966]: pgmap v6743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:03 smithi189 ceph-mon[36790]: pgmap v6743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11305.298205 secs 2024-07-26T14:36:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11305.298205 secs 2024-07-26T14:36:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11305.298205 secs 2024-07-26T14:36:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:05 smithi193 ceph-mon[37473]: pgmap v6744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:05 smithi134 ceph-mon[31966]: pgmap v6744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:05 smithi189 ceph-mon[36790]: pgmap v6744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:07.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:36:07.432+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:36:07.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:36:07.432+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:36:07.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:36:07.432+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:07 smithi193 ceph-mon[37473]: pgmap v6745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11324.723072 secs 2024-07-26T14:36:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:07 smithi134 ceph-mon[31966]: pgmap v6745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11324.723072 secs 2024-07-26T14:36:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:07 smithi189 ceph-mon[36790]: pgmap v6745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11324.723072 secs 2024-07-26T14:36:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:09 smithi193 ceph-mon[37473]: pgmap v6746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11310.298325 secs 2024-07-26T14:36:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:09 smithi134 ceph-mon[31966]: pgmap v6746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11310.298325 secs 2024-07-26T14:36:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:09 smithi189 ceph-mon[36790]: pgmap v6746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11310.298325 secs 2024-07-26T14:36:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:11 smithi193 ceph-mon[37473]: pgmap v6747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:36:11.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:11 smithi134 ceph-mon[31966]: pgmap v6747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:36:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:11 smithi189 ceph-mon[36790]: pgmap v6747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:36:12.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:36:12.008+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:36:12.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:36:12.008+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:36:12.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:36:12.008+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:12.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11329.723174 secs 2024-07-26T14:36:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11329.723174 secs 2024-07-26T14:36:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11329.723174 secs 2024-07-26T14:36:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:13 smithi193 ceph-mon[37473]: pgmap v6748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:13.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:13 smithi134 ceph-mon[31966]: pgmap v6748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:13 smithi189 ceph-mon[36790]: pgmap v6748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11315.298428 secs 2024-07-26T14:36:14.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11315.298428 secs 2024-07-26T14:36:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11315.298428 secs 2024-07-26T14:36:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:15 smithi193 ceph-mon[37473]: pgmap v6749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:15 smithi134 ceph-mon[31966]: pgmap v6749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:15 smithi189 ceph-mon[36790]: pgmap v6749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:16 smithi134 ceph-mon[31966]: pgmap v6750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:16 smithi189 ceph-mon[36790]: pgmap v6750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:16 smithi193 ceph-mon[37473]: pgmap v6750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11334.723298 secs 2024-07-26T14:36:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11334.723298 secs 2024-07-26T14:36:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11334.723298 secs 2024-07-26T14:36:18.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:18 smithi134 ceph-mon[31966]: pgmap v6751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:18 smithi189 ceph-mon[36790]: pgmap v6751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:18 smithi193 ceph-mon[37473]: pgmap v6751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11320.298507 secs 2024-07-26T14:36:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11320.298507 secs 2024-07-26T14:36:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11320.298507 secs 2024-07-26T14:36:20.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:20 smithi134 ceph-mon[31966]: pgmap v6752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:36:20.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:36:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:20 smithi189 ceph-mon[36790]: pgmap v6752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:36:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:36:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:20 smithi193 ceph-mon[37473]: pgmap v6752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:36:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:36:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:36:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:36:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:36:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:36:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:36:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:36:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: pgmap v6753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:36:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: pgmap v6754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:21.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:36:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:36:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:36:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:36:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:36:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:36:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:36:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: pgmap v6753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:36:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: pgmap v6754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:36:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:36:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:36:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:36:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:36:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:36:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:36:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:36:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: pgmap v6753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:36:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: pgmap v6754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:36:22.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:36:22.443+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:36:22.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:36:22.443+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:36:22.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:36:22.443+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11339.723370 secs 2024-07-26T14:36:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11339.723370 secs 2024-07-26T14:36:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11339.723370 secs 2024-07-26T14:36:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:23 smithi134 ceph-mon[31966]: pgmap v6755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:23 smithi189 ceph-mon[36790]: pgmap v6755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:23 smithi193 ceph-mon[37473]: pgmap v6755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:24.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:36:24.300+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:36:24.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:36:24.300+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:36:24.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:36:24.300+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 89 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:36:24.301 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:36:24.300+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:36:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11325.298605 secs 2024-07-26T14:36:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11325.298605 secs 2024-07-26T14:36:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11325.298605 secs 2024-07-26T14:36:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:25 smithi134 ceph-mon[31966]: pgmap v6756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:25 smithi189 ceph-mon[36790]: pgmap v6756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:25 smithi193 ceph-mon[37473]: pgmap v6756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11344.723492 secs 2024-07-26T14:36:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11344.723492 secs 2024-07-26T14:36:27.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:36:27.008+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:36:27.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:36:27.008+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:36:27.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:36:27.008+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11344.723492 secs 2024-07-26T14:36:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:27 smithi134 ceph-mon[31966]: pgmap v6757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:36:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:27 smithi189 ceph-mon[36790]: pgmap v6757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:36:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:27 smithi193 ceph-mon[37473]: pgmap v6757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:36:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11330.298721 secs 2024-07-26T14:36:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:30 smithi193 ceph-mon[37473]: pgmap v6758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11330.298721 secs 2024-07-26T14:36:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:30 smithi134 ceph-mon[31966]: pgmap v6758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11330.298721 secs 2024-07-26T14:36:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:30 smithi189 ceph-mon[36790]: pgmap v6758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:36:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:32 smithi193 ceph-mon[37473]: pgmap v6759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:36:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11349.723580 secs 2024-07-26T14:36:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:32 smithi134 ceph-mon[31966]: pgmap v6759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:36:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11349.723580 secs 2024-07-26T14:36:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:32 smithi189 ceph-mon[36790]: pgmap v6759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:36:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11349.723580 secs 2024-07-26T14:36:34.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:34 smithi134 ceph-mon[31966]: pgmap v6760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11335.298860 secs 2024-07-26T14:36:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:34 smithi189 ceph-mon[36790]: pgmap v6760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11335.298860 secs 2024-07-26T14:36:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:34 smithi193 ceph-mon[37473]: pgmap v6760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11335.298860 secs 2024-07-26T14:36:34.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:36:34.831+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:36:34.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:36:34.831+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:36:34.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:36:34.831+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:36:34.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:36:34.831+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:36:36.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:36 smithi134 ceph-mon[31966]: pgmap v6761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:36 smithi189 ceph-mon[36790]: pgmap v6761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:36 smithi193 ceph-mon[37473]: pgmap v6761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11354.723695 secs 2024-07-26T14:36:37.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11354.723695 secs 2024-07-26T14:36:37.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:36:37.449+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:36:37.450 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:36:37.449+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:36:37.450 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:36:37.449+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11354.723695 secs 2024-07-26T14:36:38.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:38 smithi134 ceph-mon[31966]: pgmap v6762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:38 smithi189 ceph-mon[36790]: pgmap v6762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:38 smithi193 ceph-mon[37473]: pgmap v6762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11340.298979 secs 2024-07-26T14:36:39.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11340.298979 secs 2024-07-26T14:36:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11340.298979 secs 2024-07-26T14:36:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:40 smithi134 ceph-mon[31966]: pgmap v6763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:40 smithi189 ceph-mon[36790]: pgmap v6763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:40 smithi193 ceph-mon[37473]: pgmap v6763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:42.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:36:42.024+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:36:42.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:36:42.024+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:36:42.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:36:42.024+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:42 smithi134 ceph-mon[31966]: pgmap v6764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11359.723795 secs 2024-07-26T14:36:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:42 smithi189 ceph-mon[36790]: pgmap v6764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11359.723795 secs 2024-07-26T14:36:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:42 smithi193 ceph-mon[37473]: pgmap v6764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11359.723795 secs 2024-07-26T14:36:44.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:44 smithi134 ceph-mon[31966]: pgmap v6765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11345.299106 secs 2024-07-26T14:36:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:44 smithi189 ceph-mon[36790]: pgmap v6765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11345.299106 secs 2024-07-26T14:36:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:44 smithi193 ceph-mon[37473]: pgmap v6765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11345.299106 secs 2024-07-26T14:36:46.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:46 smithi134 ceph-mon[31966]: pgmap v6766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:46 smithi189 ceph-mon[36790]: pgmap v6766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:46 smithi193 ceph-mon[37473]: pgmap v6766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11364.723944 secs 2024-07-26T14:36:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11364.723944 secs 2024-07-26T14:36:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11364.723944 secs 2024-07-26T14:36:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:48 smithi134 ceph-mon[31966]: pgmap v6767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:48 smithi189 ceph-mon[36790]: pgmap v6767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:48 smithi193 ceph-mon[37473]: pgmap v6767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11350.299208 secs 2024-07-26T14:36:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11350.299208 secs 2024-07-26T14:36:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11350.299208 secs 2024-07-26T14:36:50.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:50 smithi134 ceph-mon[31966]: pgmap v6768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:50 smithi189 ceph-mon[36790]: pgmap v6768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:50 smithi193 ceph-mon[37473]: pgmap v6768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:52 smithi134 ceph-mon[31966]: pgmap v6769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11369.724084 secs 2024-07-26T14:36:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:52 smithi189 ceph-mon[36790]: pgmap v6769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11369.724084 secs 2024-07-26T14:36:52.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:36:52.465+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:36:52.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:36:52.465+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:52.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:36:52.465+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:36:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:52 smithi193 ceph-mon[37473]: pgmap v6769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11369.724084 secs 2024-07-26T14:36:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:54 smithi134 ceph-mon[31966]: pgmap v6770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11355.299327 secs 2024-07-26T14:36:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:54 smithi189 ceph-mon[36790]: pgmap v6770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11355.299327 secs 2024-07-26T14:36:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:54 smithi193 ceph-mon[37473]: pgmap v6770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:36:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11355.299327 secs 2024-07-26T14:36:56.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:56 smithi134 ceph-mon[31966]: pgmap v6771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:56 smithi189 ceph-mon[36790]: pgmap v6771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:56 smithi193 ceph-mon[37473]: pgmap v6771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:36:57.040 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:36:57.039+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:36:57.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:36:57.039+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:36:57.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:36:57.039+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:36:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11374.724220 secs 2024-07-26T14:36:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11374.724220 secs 2024-07-26T14:36:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11374.724220 secs 2024-07-26T14:36:58.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:58 smithi134 ceph-mon[31966]: pgmap v6772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:58 smithi189 ceph-mon[36790]: pgmap v6772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:58 smithi193 ceph-mon[37473]: pgmap v6772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:36:59.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:36:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11360.299447 secs 2024-07-26T14:36:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:36:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11360.299447 secs 2024-07-26T14:36:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:36:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11360.299447 secs 2024-07-26T14:37:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:00 smithi134 ceph-mon[31966]: pgmap v6773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:00 smithi189 ceph-mon[36790]: pgmap v6773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:00 smithi193 ceph-mon[37473]: pgmap v6773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:02 smithi134 ceph-mon[31966]: pgmap v6774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11379.724364 secs 2024-07-26T14:37:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:02 smithi189 ceph-mon[36790]: pgmap v6774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11379.724364 secs 2024-07-26T14:37:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:02 smithi193 ceph-mon[37473]: pgmap v6774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11379.724364 secs 2024-07-26T14:37:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:04 smithi189 ceph-mon[36790]: pgmap v6775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:37:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11365.299565 secs 2024-07-26T14:37:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:04 smithi193 ceph-mon[37473]: pgmap v6775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:37:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11365.299565 secs 2024-07-26T14:37:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:04 smithi134 ceph-mon[31966]: pgmap v6775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:37:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11365.299565 secs 2024-07-26T14:37:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:06 smithi189 ceph-mon[36790]: pgmap v6776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:37:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:06 smithi193 ceph-mon[37473]: pgmap v6776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:37:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:06 smithi134 ceph-mon[31966]: pgmap v6776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:37:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11384.724485 secs 2024-07-26T14:37:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:37:07.472+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:37:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:37:07.472+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:07.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:37:07.472+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:37:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11384.724485 secs 2024-07-26T14:37:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11384.724485 secs 2024-07-26T14:37:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:08 smithi193 ceph-mon[37473]: pgmap v6777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:08 smithi134 ceph-mon[31966]: pgmap v6777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:08 smithi189 ceph-mon[36790]: pgmap v6777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:09.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:37:09.307+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:37:09.309 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:37:09.307+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:37:09.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:37:09.308+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 90 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T14:37:09.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:37:09.308+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T14:37:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11370.299684 secs 2024-07-26T14:37:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11370.299684 secs 2024-07-26T14:37:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11370.299684 secs 2024-07-26T14:37:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:10 smithi193 ceph-mon[37473]: pgmap v6778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:37:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:10 smithi134 ceph-mon[31966]: pgmap v6778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:37:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:10 smithi189 ceph-mon[36790]: pgmap v6778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:37:12.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:37:12.054+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:37:12.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:37:12.055+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:37:12.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:37:12.055+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:12.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:12 smithi134 ceph-mon[31966]: pgmap v6779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:37:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11389.724621 secs 2024-07-26T14:37:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:12 smithi193 ceph-mon[37473]: pgmap v6779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:37:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11389.724621 secs 2024-07-26T14:37:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:12 smithi189 ceph-mon[36790]: pgmap v6779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:37:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11389.724621 secs 2024-07-26T14:37:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:14 smithi193 ceph-mon[37473]: pgmap v6780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:37:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11375.299801 secs 2024-07-26T14:37:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:14 smithi134 ceph-mon[31966]: pgmap v6780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:37:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11375.299801 secs 2024-07-26T14:37:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:14 smithi189 ceph-mon[36790]: pgmap v6780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:37:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11375.299801 secs 2024-07-26T14:37:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:16 smithi193 ceph-mon[37473]: pgmap v6781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:16 smithi134 ceph-mon[31966]: pgmap v6781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:16 smithi189 ceph-mon[36790]: pgmap v6781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11394.724736 secs 2024-07-26T14:37:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11394.724736 secs 2024-07-26T14:37:17.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11394.724736 secs 2024-07-26T14:37:18.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:18 smithi193 ceph-mon[37473]: pgmap v6782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:18.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:18 smithi134 ceph-mon[31966]: pgmap v6782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:18.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:18 smithi189 ceph-mon[36790]: pgmap v6782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11380.299928 secs 2024-07-26T14:37:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11380.299928 secs 2024-07-26T14:37:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11380.299928 secs 2024-07-26T14:37:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:37:19.839+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:37:19.840 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:37:19.839+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:37:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:37:19.840+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:37:19.841 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:37:19.840+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:37:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:20 smithi193 ceph-mon[37473]: pgmap v6783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:20.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:20 smithi134 ceph-mon[31966]: pgmap v6783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:20 smithi189 ceph-mon[36790]: pgmap v6783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:37:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:37:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:37:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:37:22.480+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:37:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:37:22.480+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:22.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:37:22.480+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:37:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:22 smithi193 ceph-mon[37473]: pgmap v6784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11399.724834 secs 2024-07-26T14:37:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:37:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:37:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:37:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:22 smithi134 ceph-mon[31966]: pgmap v6784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11399.724834 secs 2024-07-26T14:37:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:37:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:37:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:37:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:22 smithi189 ceph-mon[36790]: pgmap v6784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11399.724834 secs 2024-07-26T14:37:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:37:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:37:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:37:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:23 smithi134 ceph-mon[31966]: pgmap v6785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:37:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:23 smithi134 ceph-mon[31966]: pgmap v6786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:37:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:23 smithi189 ceph-mon[36790]: pgmap v6785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:37:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:23 smithi189 ceph-mon[36790]: pgmap v6786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:37:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:23 smithi193 ceph-mon[37473]: pgmap v6785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:37:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:23 smithi193 ceph-mon[37473]: pgmap v6786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:37:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11385.300035 secs 2024-07-26T14:37:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:24 smithi134 ceph-mon[31966]: pgmap v6787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:37:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11385.300035 secs 2024-07-26T14:37:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:24 smithi189 ceph-mon[36790]: pgmap v6787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:37:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11385.300035 secs 2024-07-26T14:37:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:24 smithi193 ceph-mon[37473]: pgmap v6787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:37:27.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:37:27.070+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:37:27.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:37:27.070+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:37:27.072 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:37:27.070+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11404.724945 secs 2024-07-26T14:37:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:27 smithi134 ceph-mon[31966]: pgmap v6788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:37:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11404.724945 secs 2024-07-26T14:37:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:27 smithi193 ceph-mon[37473]: pgmap v6788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:37:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11404.724945 secs 2024-07-26T14:37:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:27 smithi189 ceph-mon[36790]: pgmap v6788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:37:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:29 smithi193 ceph-mon[37473]: pgmap v6789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:37:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11390.300150 secs 2024-07-26T14:37:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:29 smithi134 ceph-mon[31966]: pgmap v6789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:37:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11390.300150 secs 2024-07-26T14:37:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:29 smithi189 ceph-mon[36790]: pgmap v6789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:37:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11390.300150 secs 2024-07-26T14:37:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:31 smithi193 ceph-mon[37473]: pgmap v6790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:37:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:31 smithi134 ceph-mon[31966]: pgmap v6790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:37:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:31 smithi189 ceph-mon[36790]: pgmap v6790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:37:32.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11409.725069 secs 2024-07-26T14:37:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11409.725069 secs 2024-07-26T14:37:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11409.725069 secs 2024-07-26T14:37:33.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:33 smithi134 ceph-mon[31966]: pgmap v6791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:37:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:33 smithi189 ceph-mon[36790]: pgmap v6791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:37:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:33 smithi193 ceph-mon[37473]: pgmap v6791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:37:34.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11395.300267 secs 2024-07-26T14:37:34.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:34 smithi134 ceph-mon[31966]: pgmap v6792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:34.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11395.300267 secs 2024-07-26T14:37:34.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:34 smithi193 ceph-mon[37473]: pgmap v6792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11395.300267 secs 2024-07-26T14:37:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:34 smithi189 ceph-mon[36790]: pgmap v6792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:37.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:37:37.488+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:37:37.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:37:37.488+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:37:37.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:37:37.488+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11414.725191 secs 2024-07-26T14:37:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:37 smithi193 ceph-mon[37473]: pgmap v6793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11414.725191 secs 2024-07-26T14:37:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:37 smithi134 ceph-mon[31966]: pgmap v6793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11414.725191 secs 2024-07-26T14:37:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:37 smithi189 ceph-mon[36790]: pgmap v6793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:39 smithi193 ceph-mon[37473]: pgmap v6794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11400.300389 secs 2024-07-26T14:37:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:39 smithi134 ceph-mon[31966]: pgmap v6794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11400.300389 secs 2024-07-26T14:37:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:39 smithi189 ceph-mon[36790]: pgmap v6794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11400.300389 secs 2024-07-26T14:37:40.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:40 smithi134 ceph-mon[31966]: pgmap v6795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:40 smithi189 ceph-mon[36790]: pgmap v6795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:40 smithi193 ceph-mon[37473]: pgmap v6795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:41.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11419.725332 secs 2024-07-26T14:37:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11419.725332 secs 2024-07-26T14:37:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11419.725332 secs 2024-07-26T14:37:42.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:37:42.086+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:37:42.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:37:42.086+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:37:42.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:37:42.086+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:42 smithi134 ceph-mon[31966]: pgmap v6796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:42 smithi189 ceph-mon[36790]: pgmap v6796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:42 smithi193 ceph-mon[37473]: pgmap v6796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:44.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11405.300511 secs 2024-07-26T14:37:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11405.300511 secs 2024-07-26T14:37:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11405.300511 secs 2024-07-26T14:37:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:45 smithi134 ceph-mon[31966]: pgmap v6797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:45 smithi189 ceph-mon[36790]: pgmap v6797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:45 smithi193 ceph-mon[37473]: pgmap v6797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11424.725452 secs 2024-07-26T14:37:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:47 smithi134 ceph-mon[31966]: pgmap v6798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11424.725452 secs 2024-07-26T14:37:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:47 smithi189 ceph-mon[36790]: pgmap v6798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11424.725452 secs 2024-07-26T14:37:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:47 smithi193 ceph-mon[37473]: pgmap v6798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:48 smithi134 ceph-mon[31966]: pgmap v6799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:48 smithi189 ceph-mon[36790]: pgmap v6799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:48 smithi193 ceph-mon[37473]: pgmap v6799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11410.300634 secs 2024-07-26T14:37:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11410.300634 secs 2024-07-26T14:37:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11410.300634 secs 2024-07-26T14:37:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:50 smithi134 ceph-mon[31966]: pgmap v6800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:50 smithi189 ceph-mon[36790]: pgmap v6800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:50 smithi193 ceph-mon[37473]: pgmap v6800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:51.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11429.725597 secs 2024-07-26T14:37:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11429.725597 secs 2024-07-26T14:37:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11429.725597 secs 2024-07-26T14:37:52.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:37:52.497+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:37:52.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:37:52.497+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:37:52.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:37:52.497+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:52 smithi134 ceph-mon[31966]: pgmap v6801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:52 smithi189 ceph-mon[36790]: pgmap v6801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:52 smithi193 ceph-mon[37473]: pgmap v6801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:37:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11415.300765 secs 2024-07-26T14:37:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11415.300765 secs 2024-07-26T14:37:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11415.300765 secs 2024-07-26T14:37:55.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:55 smithi134 ceph-mon[31966]: pgmap v6802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:55 smithi189 ceph-mon[36790]: pgmap v6802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:55 smithi193 ceph-mon[37473]: pgmap v6802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:37:57.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:37:57.098+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:37:57.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:37:57.098+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:37:57.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:37:57.098+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:37:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11434.725687 secs 2024-07-26T14:37:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:57 smithi134 ceph-mon[31966]: pgmap v6803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11434.725687 secs 2024-07-26T14:37:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:57 smithi189 ceph-mon[36790]: pgmap v6803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11434.725687 secs 2024-07-26T14:37:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:57 smithi193 ceph-mon[37473]: pgmap v6803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:37:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:58 smithi134 ceph-mon[31966]: pgmap v6804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:58 smithi189 ceph-mon[36790]: pgmap v6804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:58 smithi193 ceph-mon[37473]: pgmap v6804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:37:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:37:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11420.300859 secs 2024-07-26T14:37:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:37:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11420.300859 secs 2024-07-26T14:37:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:37:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11420.300859 secs 2024-07-26T14:38:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:00 smithi134 ceph-mon[31966]: pgmap v6805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:00 smithi189 ceph-mon[36790]: pgmap v6805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:00 smithi193 ceph-mon[37473]: pgmap v6805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:01.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11439.725811 secs 2024-07-26T14:38:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11439.725811 secs 2024-07-26T14:38:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11439.725811 secs 2024-07-26T14:38:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:02 smithi134 ceph-mon[31966]: pgmap v6806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:02 smithi189 ceph-mon[36790]: pgmap v6806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:02 smithi193 ceph-mon[37473]: pgmap v6806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11425.300979 secs 2024-07-26T14:38:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11425.300979 secs 2024-07-26T14:38:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11425.300979 secs 2024-07-26T14:38:05.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:05 smithi134 ceph-mon[31966]: pgmap v6807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:05 smithi189 ceph-mon[36790]: pgmap v6807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:05 smithi193 ceph-mon[37473]: pgmap v6807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:06 smithi134 ceph-mon[31966]: pgmap v6808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:06 smithi189 ceph-mon[36790]: pgmap v6808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:06 smithi193 ceph-mon[37473]: pgmap v6808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:38:07.513+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:38:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:38:07.513+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:38:07.513+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:38:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11444.725952 secs 2024-07-26T14:38:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11444.725952 secs 2024-07-26T14:38:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11444.725952 secs 2024-07-26T14:38:08.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:08 smithi134 ceph-mon[31966]: pgmap v6809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:38:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:08 smithi189 ceph-mon[36790]: pgmap v6809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:38:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:08 smithi193 ceph-mon[37473]: pgmap v6809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:38:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11430.301122 secs 2024-07-26T14:38:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11430.301122 secs 2024-07-26T14:38:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11430.301122 secs 2024-07-26T14:38:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:10 smithi134 ceph-mon[31966]: pgmap v6810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:10 smithi189 ceph-mon[36790]: pgmap v6810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:10 smithi193 ceph-mon[37473]: pgmap v6810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11449.726075 secs 2024-07-26T14:38:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11449.726075 secs 2024-07-26T14:38:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11449.726075 secs 2024-07-26T14:38:12.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:38:12.114+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:38:12.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:38:12.114+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:38:12.115 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:38:12.114+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:12 smithi134 ceph-mon[31966]: pgmap v6811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:12 smithi189 ceph-mon[36790]: pgmap v6811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:12 smithi193 ceph-mon[37473]: pgmap v6811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11435.301257 secs 2024-07-26T14:38:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11435.301257 secs 2024-07-26T14:38:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11435.301257 secs 2024-07-26T14:38:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:15 smithi134 ceph-mon[31966]: pgmap v6812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:15 smithi189 ceph-mon[36790]: pgmap v6812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:15 smithi193 ceph-mon[37473]: pgmap v6812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:16 smithi134 ceph-mon[31966]: pgmap v6813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:16 smithi189 ceph-mon[36790]: pgmap v6813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:16 smithi193 ceph-mon[37473]: pgmap v6813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11454.726207 secs 2024-07-26T14:38:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11454.726207 secs 2024-07-26T14:38:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11454.726207 secs 2024-07-26T14:38:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:18 smithi134 ceph-mon[31966]: pgmap v6814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:38:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:18 smithi189 ceph-mon[36790]: pgmap v6814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:38:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:18 smithi193 ceph-mon[37473]: pgmap v6814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:38:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11440.301398 secs 2024-07-26T14:38:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11440.301398 secs 2024-07-26T14:38:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11440.301398 secs 2024-07-26T14:38:20.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:20 smithi134 ceph-mon[31966]: pgmap v6815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:20 smithi189 ceph-mon[36790]: pgmap v6815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:20 smithi193 ceph-mon[37473]: pgmap v6815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11459.726347 secs 2024-07-26T14:38:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11459.726347 secs 2024-07-26T14:38:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11459.726347 secs 2024-07-26T14:38:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:38:22.528+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:38:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:38:22.528+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:38:22.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:38:22.528+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:22 smithi134 ceph-mon[31966]: pgmap v6816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:38:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:38:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:22 smithi189 ceph-mon[36790]: pgmap v6816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:38:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:38:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:22 smithi193 ceph-mon[37473]: pgmap v6816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:38:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:38:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:38:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:38:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:38:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:38:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:38:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:38:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11445.301525 secs 2024-07-26T14:38:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:38:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:38:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:25 smithi134 ceph-mon[31966]: pgmap v6817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:38:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11445.301525 secs 2024-07-26T14:38:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:38:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:38:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:25 smithi189 ceph-mon[36790]: pgmap v6817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:38:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11445.301525 secs 2024-07-26T14:38:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:38:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:38:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:25 smithi193 ceph-mon[37473]: pgmap v6817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:38:27.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:38:27.120+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:38:27.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:38:27.120+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:38:27.122 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:38:27.120+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11464.726426 secs 2024-07-26T14:38:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:27 smithi134 ceph-mon[31966]: pgmap v6818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:38:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11464.726426 secs 2024-07-26T14:38:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:27 smithi189 ceph-mon[36790]: pgmap v6818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:38:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11464.726426 secs 2024-07-26T14:38:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:27 smithi193 ceph-mon[37473]: pgmap v6818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:38:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:29 smithi134 ceph-mon[31966]: pgmap v6819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T14:38:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11450.301649 secs 2024-07-26T14:38:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:29 smithi189 ceph-mon[36790]: pgmap v6819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T14:38:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11450.301649 secs 2024-07-26T14:38:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:29 smithi193 ceph-mon[37473]: pgmap v6819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T14:38:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11450.301649 secs 2024-07-26T14:38:31.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:31 smithi134 ceph-mon[31966]: pgmap v6820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:38:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:31 smithi189 ceph-mon[36790]: pgmap v6820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:38:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:31 smithi193 ceph-mon[37473]: pgmap v6820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T14:38:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11469.726550 secs 2024-07-26T14:38:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11469.726550 secs 2024-07-26T14:38:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11469.726550 secs 2024-07-26T14:38:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:33 smithi134 ceph-mon[31966]: pgmap v6821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:38:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:33 smithi189 ceph-mon[36790]: pgmap v6821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:38:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:33 smithi193 ceph-mon[37473]: pgmap v6821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T14:38:34.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11455.301765 secs 2024-07-26T14:38:34.424 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11455.301765 secs 2024-07-26T14:38:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11455.301765 secs 2024-07-26T14:38:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:35 smithi134 ceph-mon[31966]: pgmap v6822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:35 smithi189 ceph-mon[36790]: pgmap v6822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:35 smithi193 ceph-mon[37473]: pgmap v6822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11474.726659 secs 2024-07-26T14:38:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:37 smithi189 ceph-mon[36790]: pgmap v6823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11474.726659 secs 2024-07-26T14:38:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:37 smithi193 ceph-mon[37473]: pgmap v6823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:37.542 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:38:37.541+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:38:37.542 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:38:37.541+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:38:37.542 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:38:37.541+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11474.726659 secs 2024-07-26T14:38:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:37 smithi134 ceph-mon[31966]: pgmap v6823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:39 smithi189 ceph-mon[36790]: pgmap v6824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:38:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11460.301886 secs 2024-07-26T14:38:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:39 smithi193 ceph-mon[37473]: pgmap v6824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:38:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11460.301886 secs 2024-07-26T14:38:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:39 smithi134 ceph-mon[31966]: pgmap v6824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:38:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11460.301886 secs 2024-07-26T14:38:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:41 smithi189 ceph-mon[36790]: pgmap v6825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:41 smithi193 ceph-mon[37473]: pgmap v6825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:41 smithi134 ceph-mon[31966]: pgmap v6825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:42.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:38:42.124+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:38:42.125 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:38:42.124+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:42.126 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:38:42.124+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:38:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11479.726776 secs 2024-07-26T14:38:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11479.726776 secs 2024-07-26T14:38:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11479.726776 secs 2024-07-26T14:38:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:43 smithi193 ceph-mon[37473]: pgmap v6826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:43.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:43 smithi134 ceph-mon[31966]: pgmap v6826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:43 smithi189 ceph-mon[36790]: pgmap v6826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:38:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11465.302004 secs 2024-07-26T14:38:44.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11465.302004 secs 2024-07-26T14:38:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11465.302004 secs 2024-07-26T14:38:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:45 smithi193 ceph-mon[37473]: pgmap v6827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:45 smithi134 ceph-mon[31966]: pgmap v6827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:45 smithi189 ceph-mon[36790]: pgmap v6827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:38:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11484.726896 secs 2024-07-26T14:38:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:47 smithi193 ceph-mon[37473]: pgmap v6828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11484.726896 secs 2024-07-26T14:38:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:47 smithi134 ceph-mon[31966]: pgmap v6828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11484.726896 secs 2024-07-26T14:38:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:47 smithi189 ceph-mon[36790]: pgmap v6828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:38:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:49 smithi193 ceph-mon[37473]: pgmap v6829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:38:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11470.302129 secs 2024-07-26T14:38:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:49 smithi134 ceph-mon[31966]: pgmap v6829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:38:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11470.302129 secs 2024-07-26T14:38:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:49 smithi189 ceph-mon[36790]: pgmap v6829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:38:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11470.302129 secs 2024-07-26T14:38:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:51 smithi193 ceph-mon[37473]: pgmap v6830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:38:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:51 smithi134 ceph-mon[31966]: pgmap v6830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:38:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:51 smithi189 ceph-mon[36790]: pgmap v6830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:38:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11489.727016 secs 2024-07-26T14:38:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11489.727016 secs 2024-07-26T14:38:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:38:52.545+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:38:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:38:52.545+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:38:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:38:52.545+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11489.727016 secs 2024-07-26T14:38:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:53 smithi193 ceph-mon[37473]: pgmap v6831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T14:38:53.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:53 smithi134 ceph-mon[31966]: pgmap v6831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T14:38:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:53 smithi189 ceph-mon[36790]: pgmap v6831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T14:38:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11475.302247 secs 2024-07-26T14:38:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11475.302247 secs 2024-07-26T14:38:54.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11475.302247 secs 2024-07-26T14:38:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:55 smithi193 ceph-mon[37473]: pgmap v6832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:38:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:55 smithi134 ceph-mon[31966]: pgmap v6832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:38:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:55 smithi189 ceph-mon[36790]: pgmap v6832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:38:57.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:38:57.134+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:38:57.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:38:57.134+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:38:57.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:38:57.134+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:38:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11494.727144 secs 2024-07-26T14:38:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:57 smithi134 ceph-mon[31966]: pgmap v6833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:38:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11494.727144 secs 2024-07-26T14:38:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:57 smithi193 ceph-mon[37473]: pgmap v6833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:38:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11494.727144 secs 2024-07-26T14:38:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:57 smithi189 ceph-mon[36790]: pgmap v6833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:38:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:59 smithi193 ceph-mon[37473]: pgmap v6834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:38:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:38:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11480.302373 secs 2024-07-26T14:38:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:59 smithi134 ceph-mon[31966]: pgmap v6834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:38:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:38:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11480.302373 secs 2024-07-26T14:38:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:59 smithi189 ceph-mon[36790]: pgmap v6834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:38:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:38:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11480.302373 secs 2024-07-26T14:39:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:01 smithi134 ceph-mon[31966]: pgmap v6835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:01 smithi189 ceph-mon[36790]: pgmap v6835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:01 smithi193 ceph-mon[37473]: pgmap v6835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:02.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11499.727263 secs 2024-07-26T14:39:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11499.727263 secs 2024-07-26T14:39:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11499.727263 secs 2024-07-26T14:39:03.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:03 smithi134 ceph-mon[31966]: pgmap v6836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:03 smithi189 ceph-mon[36790]: pgmap v6836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:03 smithi193 ceph-mon[37473]: pgmap v6836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11485.302471 secs 2024-07-26T14:39:04.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11485.302471 secs 2024-07-26T14:39:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11485.302471 secs 2024-07-26T14:39:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:05 smithi134 ceph-mon[31966]: pgmap v6837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:05 smithi189 ceph-mon[36790]: pgmap v6837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:05 smithi193 ceph-mon[37473]: pgmap v6837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:06 smithi134 ceph-mon[31966]: pgmap v6838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:06 smithi189 ceph-mon[36790]: pgmap v6838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:06 smithi193 ceph-mon[37473]: pgmap v6838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:39:07.551+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:39:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:39:07.551+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:07.552 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:39:07.551+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:39:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11504.727369 secs 2024-07-26T14:39:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11504.727369 secs 2024-07-26T14:39:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11504.727369 secs 2024-07-26T14:39:08.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:08 smithi134 ceph-mon[31966]: pgmap v6839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:08 smithi189 ceph-mon[36790]: pgmap v6839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:08 smithi193 ceph-mon[37473]: pgmap v6839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11490.302610 secs 2024-07-26T14:39:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11490.302610 secs 2024-07-26T14:39:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11490.302610 secs 2024-07-26T14:39:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:10 smithi134 ceph-mon[31966]: pgmap v6840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:10 smithi189 ceph-mon[36790]: pgmap v6840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:10 smithi193 ceph-mon[37473]: pgmap v6840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11509.727455 secs 2024-07-26T14:39:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11509.727455 secs 2024-07-26T14:39:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11509.727455 secs 2024-07-26T14:39:12.142 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:39:12.141+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:39:12.142 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:39:12.141+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:39:12.142 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:39:12.141+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:12 smithi134 ceph-mon[31966]: pgmap v6841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:12 smithi189 ceph-mon[36790]: pgmap v6841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:12 smithi193 ceph-mon[37473]: pgmap v6841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11495.302737 secs 2024-07-26T14:39:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11495.302737 secs 2024-07-26T14:39:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11495.302737 secs 2024-07-26T14:39:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:15 smithi134 ceph-mon[31966]: pgmap v6842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:15 smithi189 ceph-mon[36790]: pgmap v6842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:15 smithi193 ceph-mon[37473]: pgmap v6842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11514.727549 secs 2024-07-26T14:39:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:17 smithi134 ceph-mon[31966]: pgmap v6843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11514.727549 secs 2024-07-26T14:39:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:17 smithi189 ceph-mon[36790]: pgmap v6843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11514.727549 secs 2024-07-26T14:39:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:17 smithi193 ceph-mon[37473]: pgmap v6843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:19 smithi134 ceph-mon[31966]: pgmap v6844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11500.302854 secs 2024-07-26T14:39:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:19 smithi189 ceph-mon[36790]: pgmap v6844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11500.302854 secs 2024-07-26T14:39:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:19 smithi193 ceph-mon[37473]: pgmap v6844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11500.302854 secs 2024-07-26T14:39:21.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:21 smithi134 ceph-mon[31966]: pgmap v6845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:21 smithi189 ceph-mon[36790]: pgmap v6845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:21 smithi193 ceph-mon[37473]: pgmap v6845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11519.727641 secs 2024-07-26T14:39:22.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11519.727641 secs 2024-07-26T14:39:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11519.727641 secs 2024-07-26T14:39:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:39:22.567+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:39:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:39:22.567+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:22.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:39:22.567+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:39:23.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:23 smithi134 ceph-mon[31966]: pgmap v6846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:23 smithi189 ceph-mon[36790]: pgmap v6846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:23 smithi193 ceph-mon[37473]: pgmap v6846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:24.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:39:24.333+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:39:24.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:39:24.333+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:39:24.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:39:24.334+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 91 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T14:39:24.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:39:24.334+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T14:39:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11505.303018 secs 2024-07-26T14:39:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11505.303018 secs 2024-07-26T14:39:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11505.303018 secs 2024-07-26T14:39:25.409 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:25 smithi134 ceph-mon[31966]: pgmap v6847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:25.409 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:39:25.409 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:25.409 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:25.409 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:25.409 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:25 smithi189 ceph-mon[36790]: pgmap v6847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:39:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:25.517 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:25 smithi193 ceph-mon[37473]: pgmap v6847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:39:25.517 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:39:25.517 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:25.517 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:25.517 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:25.517 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:39:27.156+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:39:27.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:39:27.156+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:27.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:39:27.156+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:39:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:27 smithi134 ceph-mon[31966]: 1 slow requests, 1 included below; oldest blocked for > 11524.727744 secs 2024-07-26T14:39:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:27 smithi134 ceph-mon[31966]: slow request 11524.727744 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T14:39:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:27 smithi134 ceph-mon[31966]: pgmap v6848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:27 smithi193 ceph-mon[37473]: 1 slow requests, 1 included below; oldest blocked for > 11524.727744 secs 2024-07-26T14:39:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:27 smithi193 ceph-mon[37473]: slow request 11524.727744 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T14:39:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:27 smithi193 ceph-mon[37473]: pgmap v6848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:27 smithi189 ceph-mon[36790]: 1 slow requests, 1 included below; oldest blocked for > 11524.727744 secs 2024-07-26T14:39:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:27 smithi189 ceph-mon[36790]: slow request 11524.727744 seconds old, received at 2024-07-26T11:27:21.335704+0000: client_request(client.25409:57 rmdir #0x10000000201/src 2024-07-26T11:27:21.334508+0000 caller_uid=0, caller_gid=0{0,}) currently acquired locks 2024-07-26T14:39:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:27 smithi189 ceph-mon[36790]: pgmap v6848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:28 smithi193 ceph-mon[37473]: pgmap v6849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:28 smithi134 ceph-mon[31966]: pgmap v6849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:28 smithi189 ceph-mon[36790]: pgmap v6849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11510.303082 secs 2024-07-26T14:39:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:39:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:39:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11510.303082 secs 2024-07-26T14:39:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:39:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:39:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11510.303082 secs 2024-07-26T14:39:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:39:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:39:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:39:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:30 smithi193 ceph-mon[37473]: pgmap v6850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T14:39:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:30 smithi193 ceph-mon[37473]: pgmap v6851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:39:31.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:30 smithi134 ceph-mon[31966]: pgmap v6850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T14:39:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:30 smithi134 ceph-mon[31966]: pgmap v6851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:39:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:30 smithi189 ceph-mon[36790]: pgmap v6850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-07-26T14:39:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:30 smithi189 ceph-mon[36790]: pgmap v6851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:39:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11529.727844 secs 2024-07-26T14:39:32.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11529.727844 secs 2024-07-26T14:39:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11529.727844 secs 2024-07-26T14:39:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:32 smithi193 ceph-mon[37473]: pgmap v6852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:32 smithi134 ceph-mon[31966]: pgmap v6852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:32 smithi189 ceph-mon[36790]: pgmap v6852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:39:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:39:34.863+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:39:34.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:39:34.863+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:39:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:39:34.864+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:39:34.865 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:39:34.864+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:39:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11515.303217 secs 2024-07-26T14:39:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:34 smithi193 ceph-mon[37473]: pgmap v6853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:39:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11515.303217 secs 2024-07-26T14:39:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:34 smithi134 ceph-mon[31966]: pgmap v6853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:39:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11515.303217 secs 2024-07-26T14:39:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:34 smithi189 ceph-mon[36790]: pgmap v6853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:39:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:36 smithi193 ceph-mon[37473]: pgmap v6854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:39:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11534.727964 secs 2024-07-26T14:39:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:36 smithi134 ceph-mon[31966]: pgmap v6854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:39:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11534.727964 secs 2024-07-26T14:39:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:36 smithi189 ceph-mon[36790]: pgmap v6854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:39:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11534.727964 secs 2024-07-26T14:39:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:39:37.576+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:39:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:39:37.576+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:39:37.576+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:39:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:38 smithi193 ceph-mon[37473]: pgmap v6855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:39:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:38 smithi134 ceph-mon[31966]: pgmap v6855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:39:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:39 smithi189 ceph-mon[36790]: pgmap v6855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:39:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:40 smithi193 ceph-mon[37473]: 2 slow requests, 1 included below; oldest blocked for > 11520.303303 secs 2024-07-26T14:39:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:40 smithi193 ceph-mon[37473]: slow request 11520.303302 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T14:39:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:40 smithi134 ceph-mon[31966]: 2 slow requests, 1 included below; oldest blocked for > 11520.303303 secs 2024-07-26T14:39:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:40 smithi134 ceph-mon[31966]: slow request 11520.303302 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T14:39:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:40 smithi189 ceph-mon[36790]: 2 slow requests, 1 included below; oldest blocked for > 11520.303303 secs 2024-07-26T14:39:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:40 smithi189 ceph-mon[36790]: slow request 11520.303302 seconds old, received at 2024-07-26T11:27:38.211068+0000: peer_request:client.25409:57 currently dispatched 2024-07-26T14:39:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:41 smithi193 ceph-mon[37473]: pgmap v6856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:39:41.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:41 smithi134 ceph-mon[31966]: pgmap v6856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:39:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:41 smithi189 ceph-mon[36790]: pgmap v6856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:39:42.163 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:39:42.162+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:39:42.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:39:42.162+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:39:42.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:39:42.162+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11539.728086 secs 2024-07-26T14:39:42.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11539.728086 secs 2024-07-26T14:39:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11539.728086 secs 2024-07-26T14:39:43.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:43 smithi134 ceph-mon[31966]: pgmap v6857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:43 smithi189 ceph-mon[36790]: pgmap v6857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:43 smithi193 ceph-mon[37473]: pgmap v6857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11525.303368 secs 2024-07-26T14:39:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:45 smithi134 ceph-mon[31966]: pgmap v6858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11525.303368 secs 2024-07-26T14:39:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:45 smithi189 ceph-mon[36790]: pgmap v6858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:45.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11525.303368 secs 2024-07-26T14:39:45.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:45 smithi193 ceph-mon[37473]: pgmap v6858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:47 smithi193 ceph-mon[37473]: pgmap v6859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11544.728200 secs 2024-07-26T14:39:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:47 smithi134 ceph-mon[31966]: pgmap v6859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11544.728200 secs 2024-07-26T14:39:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:47 smithi189 ceph-mon[36790]: pgmap v6859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11544.728200 secs 2024-07-26T14:39:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:49 smithi134 ceph-mon[31966]: pgmap v6860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11530.303502 secs 2024-07-26T14:39:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:49 smithi189 ceph-mon[36790]: pgmap v6860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11530.303502 secs 2024-07-26T14:39:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:49 smithi193 ceph-mon[37473]: pgmap v6860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11530.303502 secs 2024-07-26T14:39:51.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:51 smithi134 ceph-mon[31966]: pgmap v6861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:39:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:51 smithi189 ceph-mon[36790]: pgmap v6861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:39:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:51 smithi193 ceph-mon[37473]: pgmap v6861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:39:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11549.728360 secs 2024-07-26T14:39:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11549.728360 secs 2024-07-26T14:39:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11549.728360 secs 2024-07-26T14:39:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:39:52.592+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:39:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:39:52.592+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:39:52.592+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:39:53.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:53 smithi134 ceph-mon[31966]: pgmap v6862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:53 smithi189 ceph-mon[36790]: pgmap v6862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:53 smithi193 ceph-mon[37473]: pgmap v6862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11535.303636 secs 2024-07-26T14:39:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11535.303636 secs 2024-07-26T14:39:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11535.303636 secs 2024-07-26T14:39:55.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:55 smithi134 ceph-mon[31966]: pgmap v6863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:55 smithi189 ceph-mon[36790]: pgmap v6863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:55 smithi193 ceph-mon[37473]: pgmap v6863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:57.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:39:57.178+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:39:57.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:39:57.178+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:39:57.179 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:39:57.178+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:39:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:57 smithi134 ceph-mon[31966]: pgmap v6864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11554.728454 secs 2024-07-26T14:39:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:57 smithi189 ceph-mon[36790]: pgmap v6864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11554.728454 secs 2024-07-26T14:39:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:57 smithi193 ceph-mon[37473]: pgmap v6864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:39:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11554.728454 secs 2024-07-26T14:39:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:59 smithi134 ceph-mon[31966]: pgmap v6865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:39:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11540.303788 secs 2024-07-26T14:39:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:59 smithi189 ceph-mon[36790]: pgmap v6865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:39:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11540.303788 secs 2024-07-26T14:39:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:59 smithi193 ceph-mon[37473]: pgmap v6865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:39:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:39:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11540.303788 secs 2024-07-26T14:40:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:40:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:40:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:40:01.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:01 smithi134 ceph-mon[31966]: pgmap v6866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:01 smithi189 ceph-mon[36790]: pgmap v6866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:01.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:01 smithi193 ceph-mon[37473]: pgmap v6866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11559.728574 secs 2024-07-26T14:40:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11559.728574 secs 2024-07-26T14:40:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11559.728574 secs 2024-07-26T14:40:03.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:03 smithi134 ceph-mon[31966]: pgmap v6867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:03 smithi189 ceph-mon[36790]: pgmap v6867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:03.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:03 smithi193 ceph-mon[37473]: pgmap v6867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:04 smithi134 ceph-mon[31966]: 2 slow requests, 1 included below; oldest blocked for > 11545.303917 secs 2024-07-26T14:40:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:04 smithi134 ceph-mon[31966]: slow request 11522.558633 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T14:40:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:04 smithi189 ceph-mon[36790]: 2 slow requests, 1 included below; oldest blocked for > 11545.303917 secs 2024-07-26T14:40:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:04 smithi189 ceph-mon[36790]: slow request 11522.558633 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T14:40:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:04 smithi193 ceph-mon[37473]: 2 slow requests, 1 included below; oldest blocked for > 11545.303917 secs 2024-07-26T14:40:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:04 smithi193 ceph-mon[37473]: slow request 11522.558633 seconds old, received at 2024-07-26T11:28:00.956350+0000: client_request(client.26981:17 mksnap #0x1//scheduled-2024-07-26-11_28_00_UTC 2024-07-26T11:28:00.956223+0000 caller_uid=0, caller_gid=0{}) currently failed to xlock, waiting 2024-07-26T14:40:05.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:05 smithi134 ceph-mon[31966]: pgmap v6868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:05.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:05 smithi193 ceph-mon[37473]: pgmap v6868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:05 smithi189 ceph-mon[36790]: pgmap v6868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:07 smithi134 ceph-mon[31966]: pgmap v6869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11564.728652 secs 2024-07-26T14:40:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:07 smithi189 ceph-mon[36790]: pgmap v6869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11564.728652 secs 2024-07-26T14:40:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:07 smithi193 ceph-mon[37473]: pgmap v6869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11564.728652 secs 2024-07-26T14:40:07.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:40:07.607+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:40:07.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:40:07.607+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:07.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:40:07.607+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:40:09.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:09 smithi134 ceph-mon[31966]: pgmap v6870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11550.304050 secs 2024-07-26T14:40:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:09 smithi189 ceph-mon[36790]: pgmap v6870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11550.304050 secs 2024-07-26T14:40:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:09 smithi193 ceph-mon[37473]: pgmap v6870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11550.304050 secs 2024-07-26T14:40:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:11 smithi134 ceph-mon[31966]: pgmap v6871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:11 smithi189 ceph-mon[36790]: pgmap v6871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:11 smithi193 ceph-mon[37473]: pgmap v6871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:12.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:40:12.186+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:40:12.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:40:12.186+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:40:12.188 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:40:12.186+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11569.728771 secs 2024-07-26T14:40:12.527 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11569.728771 secs 2024-07-26T14:40:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11569.728771 secs 2024-07-26T14:40:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:13 smithi189 ceph-mon[36790]: pgmap v6872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:13 smithi193 ceph-mon[37473]: pgmap v6872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:13.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:13 smithi134 ceph-mon[31966]: pgmap v6872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11555.304117 secs 2024-07-26T14:40:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11555.304117 secs 2024-07-26T14:40:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11555.304117 secs 2024-07-26T14:40:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:15 smithi189 ceph-mon[36790]: pgmap v6873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:15 smithi193 ceph-mon[37473]: pgmap v6873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:15.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:15 smithi134 ceph-mon[31966]: pgmap v6873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:17 smithi193 ceph-mon[37473]: pgmap v6874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11574.728888 secs 2024-07-26T14:40:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:17 smithi134 ceph-mon[31966]: pgmap v6874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11574.728888 secs 2024-07-26T14:40:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:17 smithi189 ceph-mon[36790]: pgmap v6874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11574.728888 secs 2024-07-26T14:40:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:19 smithi193 ceph-mon[37473]: pgmap v6875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11560.304246 secs 2024-07-26T14:40:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:19 smithi134 ceph-mon[31966]: pgmap v6875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11560.304246 secs 2024-07-26T14:40:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:19 smithi189 ceph-mon[36790]: pgmap v6875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11560.304246 secs 2024-07-26T14:40:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:21 smithi193 ceph-mon[37473]: pgmap v6876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:21 smithi134 ceph-mon[31966]: pgmap v6876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:21 smithi189 ceph-mon[36790]: pgmap v6876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11579.729038 secs 2024-07-26T14:40:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:40:22.623+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:40:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:40:22.623+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:22.624 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:40:22.623+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:40:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11579.729038 secs 2024-07-26T14:40:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11579.729038 secs 2024-07-26T14:40:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:23 smithi193 ceph-mon[37473]: pgmap v6877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:23 smithi134 ceph-mon[31966]: pgmap v6877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:23 smithi189 ceph-mon[36790]: pgmap v6877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11565.304377 secs 2024-07-26T14:40:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11565.304377 secs 2024-07-26T14:40:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11565.304377 secs 2024-07-26T14:40:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:25 smithi193 ceph-mon[37473]: pgmap v6878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:25 smithi134 ceph-mon[31966]: pgmap v6878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:25 smithi189 ceph-mon[36790]: pgmap v6878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:27.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:40:27.202+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:40:27.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:40:27.202+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:40:27.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:40:27.202+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:27 smithi193 ceph-mon[37473]: pgmap v6879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11584.729189 secs 2024-07-26T14:40:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:27 smithi134 ceph-mon[31966]: pgmap v6879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11584.729189 secs 2024-07-26T14:40:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:27 smithi189 ceph-mon[36790]: pgmap v6879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11584.729189 secs 2024-07-26T14:40:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:29 smithi193 ceph-mon[37473]: pgmap v6880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11570.304529 secs 2024-07-26T14:40:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:29 smithi134 ceph-mon[31966]: pgmap v6880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11570.304529 secs 2024-07-26T14:40:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:29 smithi189 ceph-mon[36790]: pgmap v6880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11570.304529 secs 2024-07-26T14:40:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:40:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:40:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:40:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:31 smithi134 ceph-mon[31966]: pgmap v6881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:40:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:40:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:40:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:31 smithi189 ceph-mon[36790]: pgmap v6881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:40:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:40:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:40:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:31 smithi193 ceph-mon[37473]: pgmap v6881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:40:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:40:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:40:32.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:32 smithi189 ceph-mon[36790]: pgmap v6882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:40:32.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:32 smithi189 ceph-mon[36790]: pgmap v6883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:40:32.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11589.729330 secs 2024-07-26T14:40:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:32 smithi134 ceph-mon[31966]: pgmap v6882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:40:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:32 smithi134 ceph-mon[31966]: pgmap v6883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:40:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11589.729330 secs 2024-07-26T14:40:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:32 smithi193 ceph-mon[37473]: pgmap v6882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:40:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:32 smithi193 ceph-mon[37473]: pgmap v6883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:40:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11589.729330 secs 2024-07-26T14:40:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:34 smithi134 ceph-mon[31966]: pgmap v6884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:40:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11575.304658 secs 2024-07-26T14:40:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:34 smithi189 ceph-mon[36790]: pgmap v6884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:40:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11575.304658 secs 2024-07-26T14:40:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:34 smithi193 ceph-mon[37473]: pgmap v6884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:40:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11575.304658 secs 2024-07-26T14:40:36.575 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:36 smithi193 ceph-mon[37473]: pgmap v6885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:40:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:36 smithi134 ceph-mon[31966]: pgmap v6885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:40:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:36 smithi189 ceph-mon[36790]: pgmap v6885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:40:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:40:37.638+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:40:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:40:37.638+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:37.640 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:40:37.638+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:40:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11594.729441 secs 2024-07-26T14:40:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11594.729441 secs 2024-07-26T14:40:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11594.729441 secs 2024-07-26T14:40:38.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:38 smithi134 ceph-mon[31966]: pgmap v6886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:40:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:38 smithi189 ceph-mon[36790]: pgmap v6886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:40:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:38 smithi193 ceph-mon[37473]: pgmap v6886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:40:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11580.304780 secs 2024-07-26T14:40:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11580.304780 secs 2024-07-26T14:40:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11580.304780 secs 2024-07-26T14:40:40.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:40 smithi134 ceph-mon[31966]: pgmap v6887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:40:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:40 smithi189 ceph-mon[36790]: pgmap v6887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:40:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:40 smithi193 ceph-mon[37473]: pgmap v6887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:40:41.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:41 smithi134 ceph-mon[31966]: pgmap v6888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:40:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:41 smithi189 ceph-mon[36790]: pgmap v6888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:40:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:41 smithi193 ceph-mon[37473]: pgmap v6888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:40:42.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:40:42.217+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:40:42.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:40:42.217+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:40:42.219 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:40:42.217+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11599.729520 secs 2024-07-26T14:40:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11599.729520 secs 2024-07-26T14:40:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11599.729520 secs 2024-07-26T14:40:43.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:43 smithi134 ceph-mon[31966]: pgmap v6889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:43 smithi189 ceph-mon[36790]: pgmap v6889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:43 smithi193 ceph-mon[37473]: pgmap v6889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11585.304925 secs 2024-07-26T14:40:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11585.304925 secs 2024-07-26T14:40:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11585.304925 secs 2024-07-26T14:40:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:45 smithi134 ceph-mon[31966]: pgmap v6890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:45 smithi189 ceph-mon[36790]: pgmap v6890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:45 smithi193 ceph-mon[37473]: pgmap v6890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11604.729642 secs 2024-07-26T14:40:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11604.729642 secs 2024-07-26T14:40:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11604.729642 secs 2024-07-26T14:40:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:47 smithi134 ceph-mon[31966]: pgmap v6891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:47 smithi189 ceph-mon[36790]: pgmap v6891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:47 smithi193 ceph-mon[37473]: pgmap v6891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11590.305050 secs 2024-07-26T14:40:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:49 smithi134 ceph-mon[31966]: pgmap v6892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11590.305050 secs 2024-07-26T14:40:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:49 smithi189 ceph-mon[36790]: pgmap v6892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11590.305050 secs 2024-07-26T14:40:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:49 smithi193 ceph-mon[37473]: pgmap v6892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:51 smithi134 ceph-mon[31966]: pgmap v6893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11609.729763 secs 2024-07-26T14:40:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:51 smithi189 ceph-mon[36790]: pgmap v6893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11609.729763 secs 2024-07-26T14:40:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:51 smithi193 ceph-mon[37473]: pgmap v6893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11609.729763 secs 2024-07-26T14:40:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:40:52.654+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:40:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:40:52.654+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:40:52.655 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:40:52.654+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:53 smithi134 ceph-mon[31966]: pgmap v6894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:53 smithi189 ceph-mon[36790]: pgmap v6894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:53 smithi193 ceph-mon[37473]: pgmap v6894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:40:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11595.305182 secs 2024-07-26T14:40:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11595.305182 secs 2024-07-26T14:40:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11595.305182 secs 2024-07-26T14:40:56.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:55 smithi134 ceph-mon[31966]: pgmap v6895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:55 smithi189 ceph-mon[36790]: pgmap v6895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:55 smithi193 ceph-mon[37473]: pgmap v6895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:40:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11614.729881 secs 2024-07-26T14:40:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11614.729881 secs 2024-07-26T14:40:57.230 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:40:57.229+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:40:57.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:40:57.229+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:40:57.231 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:40:57.229+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:40:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11614.729881 secs 2024-07-26T14:40:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:57 smithi134 ceph-mon[31966]: pgmap v6896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:57 smithi189 ceph-mon[36790]: pgmap v6896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:40:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:57 smithi193 ceph-mon[37473]: pgmap v6896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11600.305266 secs 2024-07-26T14:41:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:40:59 smithi134 ceph-mon[31966]: pgmap v6897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11600.305266 secs 2024-07-26T14:41:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:40:59 smithi189 ceph-mon[36790]: pgmap v6897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11600.305266 secs 2024-07-26T14:41:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:40:59 smithi193 ceph-mon[37473]: pgmap v6897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:01 smithi134 ceph-mon[31966]: pgmap v6898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11619.730003 secs 2024-07-26T14:41:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:01 smithi189 ceph-mon[36790]: pgmap v6898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11619.730003 secs 2024-07-26T14:41:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:01 smithi193 ceph-mon[37473]: pgmap v6898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11619.730003 secs 2024-07-26T14:41:04.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:03 smithi134 ceph-mon[31966]: pgmap v6899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:03 smithi189 ceph-mon[36790]: pgmap v6899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:03 smithi193 ceph-mon[37473]: pgmap v6899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11605.305352 secs 2024-07-26T14:41:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11605.305352 secs 2024-07-26T14:41:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11605.305352 secs 2024-07-26T14:41:06.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:05 smithi134 ceph-mon[31966]: pgmap v6900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:41:06.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:05 smithi193 ceph-mon[37473]: pgmap v6900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:41:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:05 smithi189 ceph-mon[36790]: pgmap v6900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:41:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11624.730093 secs 2024-07-26T14:41:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11624.730093 secs 2024-07-26T14:41:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11624.730093 secs 2024-07-26T14:41:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:41:07.670+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:41:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:41:07.670+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:41:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:41:07.670+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:08.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:07 smithi134 ceph-mon[31966]: pgmap v6901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:07 smithi189 ceph-mon[36790]: pgmap v6901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:07 smithi193 ceph-mon[37473]: pgmap v6901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:10.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11610.305497 secs 2024-07-26T14:41:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:09 smithi134 ceph-mon[31966]: pgmap v6902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11610.305497 secs 2024-07-26T14:41:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:09 smithi189 ceph-mon[36790]: pgmap v6902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11610.305497 secs 2024-07-26T14:41:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:09 smithi193 ceph-mon[37473]: pgmap v6902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:11 smithi134 ceph-mon[31966]: pgmap v6903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11629.730220 secs 2024-07-26T14:41:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:11 smithi189 ceph-mon[36790]: pgmap v6903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11629.730220 secs 2024-07-26T14:41:12.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:41:12.245+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:41:12.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:41:12.245+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:41:12.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:41:12.245+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:11 smithi193 ceph-mon[37473]: pgmap v6903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11629.730220 secs 2024-07-26T14:41:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:13 smithi189 ceph-mon[36790]: pgmap v6904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:13 smithi193 ceph-mon[37473]: pgmap v6904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:13 smithi134 ceph-mon[31966]: pgmap v6904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11615.305625 secs 2024-07-26T14:41:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11615.305625 secs 2024-07-26T14:41:15.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11615.305625 secs 2024-07-26T14:41:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:15 smithi189 ceph-mon[36790]: pgmap v6905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:41:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:15 smithi193 ceph-mon[37473]: pgmap v6905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:41:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:15 smithi134 ceph-mon[31966]: pgmap v6905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:41:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11634.730342 secs 2024-07-26T14:41:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11634.730342 secs 2024-07-26T14:41:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11634.730342 secs 2024-07-26T14:41:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:17 smithi193 ceph-mon[37473]: pgmap v6906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:18.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:17 smithi134 ceph-mon[31966]: pgmap v6906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:17 smithi189 ceph-mon[36790]: pgmap v6906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11620.305770 secs 2024-07-26T14:41:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:19 smithi193 ceph-mon[37473]: pgmap v6907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11620.305770 secs 2024-07-26T14:41:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:19 smithi134 ceph-mon[31966]: pgmap v6907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11620.305770 secs 2024-07-26T14:41:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:19 smithi189 ceph-mon[36790]: pgmap v6907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:21 smithi193 ceph-mon[37473]: pgmap v6908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11639.730466 secs 2024-07-26T14:41:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:21 smithi134 ceph-mon[31966]: pgmap v6908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11639.730466 secs 2024-07-26T14:41:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:21 smithi189 ceph-mon[36790]: pgmap v6908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11639.730466 secs 2024-07-26T14:41:22.686 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:41:22.685+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:41:22.687 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:41:22.685+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:22.687 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:41:22.685+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:41:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:23 smithi193 ceph-mon[37473]: pgmap v6909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:23 smithi134 ceph-mon[31966]: pgmap v6909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:23 smithi189 ceph-mon[36790]: pgmap v6909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11625.305916 secs 2024-07-26T14:41:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11625.305916 secs 2024-07-26T14:41:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11625.305916 secs 2024-07-26T14:41:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:25 smithi193 ceph-mon[37473]: pgmap v6910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:41:26.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:25 smithi134 ceph-mon[31966]: pgmap v6910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:41:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:25 smithi189 ceph-mon[36790]: pgmap v6910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:41:27.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:41:27.260+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:41:27.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:41:27.260+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:41:27.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:41:27.260+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:27.262 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11644.730594 secs 2024-07-26T14:41:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11644.730594 secs 2024-07-26T14:41:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11644.730594 secs 2024-07-26T14:41:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:27 smithi193 ceph-mon[37473]: pgmap v6911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:28.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:27 smithi134 ceph-mon[31966]: pgmap v6911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:27 smithi189 ceph-mon[36790]: pgmap v6911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11630.306049 secs 2024-07-26T14:41:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:30 smithi193 ceph-mon[37473]: pgmap v6912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:30.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11630.306049 secs 2024-07-26T14:41:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:30 smithi134 ceph-mon[31966]: pgmap v6912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11630.306049 secs 2024-07-26T14:41:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:30 smithi189 ceph-mon[36790]: pgmap v6912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:41:31.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:41:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:41:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:32 smithi193 ceph-mon[37473]: pgmap v6913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11649.730686 secs 2024-07-26T14:41:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:41:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:41:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:41:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:32 smithi134 ceph-mon[31966]: pgmap v6913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11649.730686 secs 2024-07-26T14:41:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:41:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:41:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:41:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:32 smithi189 ceph-mon[36790]: pgmap v6913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11649.730686 secs 2024-07-26T14:41:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:41:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:41:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:41:33.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:33 smithi189 ceph-mon[36790]: pgmap v6914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:41:33.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:33 smithi189 ceph-mon[36790]: pgmap v6915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:41:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:33 smithi134 ceph-mon[31966]: pgmap v6914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:41:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:33 smithi134 ceph-mon[31966]: pgmap v6915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:41:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:33 smithi193 ceph-mon[37473]: pgmap v6914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:41:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:33 smithi193 ceph-mon[37473]: pgmap v6915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:41:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11635.306205 secs 2024-07-26T14:41:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:35 smithi134 ceph-mon[31966]: pgmap v6916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:41:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11635.306205 secs 2024-07-26T14:41:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:35 smithi189 ceph-mon[36790]: pgmap v6916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:41:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11635.306205 secs 2024-07-26T14:41:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:35 smithi193 ceph-mon[37473]: pgmap v6916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:41:37.325 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:37 smithi193 ceph-mon[37473]: pgmap v6917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:41:37.325 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11654.730806 secs 2024-07-26T14:41:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:37 smithi134 ceph-mon[31966]: pgmap v6917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:41:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11654.730806 secs 2024-07-26T14:41:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:37 smithi189 ceph-mon[36790]: pgmap v6917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:41:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11654.730806 secs 2024-07-26T14:41:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:41:37.701+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:41:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:41:37.701+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:37.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:41:37.701+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:41:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:39 smithi134 ceph-mon[31966]: pgmap v6918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:41:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:39 smithi189 ceph-mon[36790]: pgmap v6918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:41:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:39 smithi193 ceph-mon[37473]: pgmap v6918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:41:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11640.306350 secs 2024-07-26T14:41:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11640.306350 secs 2024-07-26T14:41:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11640.306350 secs 2024-07-26T14:41:41.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:41 smithi134 ceph-mon[31966]: pgmap v6919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:41:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:41 smithi189 ceph-mon[36790]: pgmap v6919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:41:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:41 smithi193 ceph-mon[37473]: pgmap v6919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:41:42.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:41:42.264+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:41:42.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:41:42.264+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:41:42.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:41:42.264+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11659.730959 secs 2024-07-26T14:41:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11659.730959 secs 2024-07-26T14:41:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11659.730959 secs 2024-07-26T14:41:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:43 smithi134 ceph-mon[31966]: pgmap v6920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:41:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:43 smithi189 ceph-mon[36790]: pgmap v6920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:41:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:43 smithi193 ceph-mon[37473]: pgmap v6920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:41:44.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11645.306503 secs 2024-07-26T14:41:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11645.306503 secs 2024-07-26T14:41:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11645.306503 secs 2024-07-26T14:41:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:45 smithi134 ceph-mon[31966]: pgmap v6921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:41:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:45 smithi189 ceph-mon[36790]: pgmap v6921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:41:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:45 smithi193 ceph-mon[37473]: pgmap v6921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:41:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:47 smithi134 ceph-mon[31966]: pgmap v6922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11664.731078 secs 2024-07-26T14:41:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:47 smithi189 ceph-mon[36790]: pgmap v6922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11664.731078 secs 2024-07-26T14:41:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:47 smithi193 ceph-mon[37473]: pgmap v6922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11664.731078 secs 2024-07-26T14:41:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:49 smithi134 ceph-mon[31966]: pgmap v6923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11650.306607 secs 2024-07-26T14:41:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:49 smithi189 ceph-mon[36790]: pgmap v6923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11650.306607 secs 2024-07-26T14:41:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:49 smithi193 ceph-mon[37473]: pgmap v6923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11650.306607 secs 2024-07-26T14:41:51.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:51 smithi134 ceph-mon[31966]: pgmap v6924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:51 smithi189 ceph-mon[36790]: pgmap v6924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:51 smithi193 ceph-mon[37473]: pgmap v6924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11669.731200 secs 2024-07-26T14:41:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11669.731200 secs 2024-07-26T14:41:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11669.731200 secs 2024-07-26T14:41:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:41:52.712+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:41:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:41:52.713+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:41:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:41:52.713+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:53.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:53 smithi134 ceph-mon[31966]: pgmap v6925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:41:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:53 smithi189 ceph-mon[36790]: pgmap v6925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:41:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:53 smithi193 ceph-mon[37473]: pgmap v6925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:41:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11655.306730 secs 2024-07-26T14:41:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11655.306730 secs 2024-07-26T14:41:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11655.306730 secs 2024-07-26T14:41:55.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:55 smithi134 ceph-mon[31966]: pgmap v6926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:55 smithi189 ceph-mon[36790]: pgmap v6926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:55 smithi193 ceph-mon[37473]: pgmap v6926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:57.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:41:57.265+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:41:57.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:41:57.265+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:41:57.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:41:57.265+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:41:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:57 smithi134 ceph-mon[31966]: pgmap v6927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11674.731318 secs 2024-07-26T14:41:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:57 smithi189 ceph-mon[36790]: pgmap v6927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11674.731318 secs 2024-07-26T14:41:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:57 smithi193 ceph-mon[37473]: pgmap v6927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:41:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11674.731318 secs 2024-07-26T14:41:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:59 smithi134 ceph-mon[31966]: pgmap v6928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:41:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11660.306843 secs 2024-07-26T14:41:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:59 smithi189 ceph-mon[36790]: pgmap v6928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:41:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11660.306843 secs 2024-07-26T14:41:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:59 smithi193 ceph-mon[37473]: pgmap v6928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:41:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:41:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11660.306843 secs 2024-07-26T14:42:01.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:01 smithi134 ceph-mon[31966]: pgmap v6929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:42:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:01 smithi189 ceph-mon[36790]: pgmap v6929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:42:01.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:01 smithi193 ceph-mon[37473]: pgmap v6929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:42:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11679.731348 secs 2024-07-26T14:42:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11679.731348 secs 2024-07-26T14:42:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11679.731348 secs 2024-07-26T14:42:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:03 smithi189 ceph-mon[36790]: pgmap v6930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:42:03.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:03 smithi193 ceph-mon[37473]: pgmap v6930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:42:03.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:03 smithi134 ceph-mon[31966]: pgmap v6930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:42:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11665.307018 secs 2024-07-26T14:42:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11665.307018 secs 2024-07-26T14:42:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11665.307018 secs 2024-07-26T14:42:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:05 smithi189 ceph-mon[36790]: pgmap v6931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:05 smithi193 ceph-mon[37473]: pgmap v6931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:05 smithi134 ceph-mon[31966]: pgmap v6931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:07 smithi193 ceph-mon[37473]: pgmap v6932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11684.731468 secs 2024-07-26T14:42:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:07 smithi134 ceph-mon[31966]: pgmap v6932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11684.731468 secs 2024-07-26T14:42:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:07 smithi189 ceph-mon[36790]: pgmap v6932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11684.731468 secs 2024-07-26T14:42:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:07.728+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:42:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:07.728+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:42:07.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:07.728+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:09 smithi193 ceph-mon[37473]: pgmap v6933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11670.307086 secs 2024-07-26T14:42:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:09 smithi134 ceph-mon[31966]: pgmap v6933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11670.307086 secs 2024-07-26T14:42:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:09 smithi189 ceph-mon[36790]: pgmap v6933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11670.307086 secs 2024-07-26T14:42:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:11 smithi193 ceph-mon[37473]: pgmap v6934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:11.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:11 smithi134 ceph-mon[31966]: pgmap v6934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:11 smithi189 ceph-mon[36790]: pgmap v6934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:12.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:12.280+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:42:12.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:12.280+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:12.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:12.280+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:42:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11689.731588 secs 2024-07-26T14:42:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11689.731588 secs 2024-07-26T14:42:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11689.731588 secs 2024-07-26T14:42:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:13 smithi193 ceph-mon[37473]: pgmap v6935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:13.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:13 smithi134 ceph-mon[31966]: pgmap v6935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:13 smithi189 ceph-mon[36790]: pgmap v6935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11675.307226 secs 2024-07-26T14:42:14.572 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11675.307226 secs 2024-07-26T14:42:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11675.307226 secs 2024-07-26T14:42:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:15 smithi193 ceph-mon[37473]: pgmap v6936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:15 smithi134 ceph-mon[31966]: pgmap v6936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:15 smithi189 ceph-mon[36790]: pgmap v6936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:17 smithi193 ceph-mon[37473]: pgmap v6937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11694.731708 secs 2024-07-26T14:42:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:17 smithi134 ceph-mon[31966]: pgmap v6937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11694.731708 secs 2024-07-26T14:42:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:17 smithi189 ceph-mon[36790]: pgmap v6937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11694.731708 secs 2024-07-26T14:42:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:19 smithi193 ceph-mon[37473]: pgmap v6938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11680.307344 secs 2024-07-26T14:42:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:19 smithi134 ceph-mon[31966]: pgmap v6938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11680.307344 secs 2024-07-26T14:42:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:19 smithi189 ceph-mon[36790]: pgmap v6938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11680.307344 secs 2024-07-26T14:42:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:21 smithi193 ceph-mon[37473]: pgmap v6939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:21 smithi134 ceph-mon[31966]: pgmap v6939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:21 smithi189 ceph-mon[36790]: pgmap v6939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:22.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:22.325+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1740 msgr2=0x7f9b300c4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T14:42:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:22.325+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1740 msgr2=0x7f9b300c4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T14:42:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:22.325+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1740 0x7f9b300c4130 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009770 tx=0x7f9b3805d040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:42:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:22.325+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1740 0x7f9b300c4130 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009770 tx=0x7f9b3805d040 comp rx=0 tx=0).stop 2024-07-26T14:42:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:22.325+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1740 msgr2=0x7f9b300c4130 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T14:42:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:22.325+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1740 msgr2=0x7f9b300c4130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:42:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:22.325+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1740 0x7f9b300c4130 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:42:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:22.325+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:42:22.327 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:22.326+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:42:22.328 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:22.327+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b3805d040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:42:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11699.731791 secs 2024-07-26T14:42:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11699.731791 secs 2024-07-26T14:42:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11699.731791 secs 2024-07-26T14:42:22.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:22.744+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:42:22.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:22.744+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:42:22.745 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:22.744+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:23 smithi193 ceph-mon[37473]: pgmap v6940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:23 smithi134 ceph-mon[31966]: pgmap v6940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:23 smithi189 ceph-mon[36790]: pgmap v6940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:24.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:24.363+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:42:24.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:24.363+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:42:24.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:24.364+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 92 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T14:42:24.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:24.364+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T14:42:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11685.307461 secs 2024-07-26T14:42:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11685.307461 secs 2024-07-26T14:42:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11685.307461 secs 2024-07-26T14:42:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:25 smithi134 ceph-mon[31966]: pgmap v6941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:25 smithi189 ceph-mon[36790]: pgmap v6941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:25 smithi193 ceph-mon[37473]: pgmap v6941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:27.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:27.285+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:42:27.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:27.285+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:42:27.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:27.285+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:27 smithi134 ceph-mon[31966]: pgmap v6942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11704.731870 secs 2024-07-26T14:42:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:27 smithi189 ceph-mon[36790]: pgmap v6942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11704.731870 secs 2024-07-26T14:42:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:27 smithi193 ceph-mon[37473]: pgmap v6942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11704.731870 secs 2024-07-26T14:42:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:29 smithi134 ceph-mon[31966]: pgmap v6943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11690.307584 secs 2024-07-26T14:42:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:29 smithi189 ceph-mon[36790]: pgmap v6943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11690.307584 secs 2024-07-26T14:42:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:29 smithi193 ceph-mon[37473]: pgmap v6943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11690.307584 secs 2024-07-26T14:42:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:31 smithi134 ceph-mon[31966]: pgmap v6944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:31 smithi189 ceph-mon[36790]: pgmap v6944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:31 smithi193 ceph-mon[37473]: pgmap v6944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11709.732024 secs 2024-07-26T14:42:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:42:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11709.732024 secs 2024-07-26T14:42:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:42:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11709.732024 secs 2024-07-26T14:42:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:42:32.880 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:32.879+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 msgr2=0x7f21ac0c4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T14:42:32.880 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:32.879+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 msgr2=0x7f21ac0c4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T14:42:32.880 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:32.879+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f21b80d3f20 tx=0x7f21b80aa000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:42:32.880 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:32.879+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f21b80d3f20 tx=0x7f21b80aa000 comp rx=0 tx=0).stop 2024-07-26T14:42:32.880 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:32.879+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 msgr2=0x7f21ac0c4210 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T14:42:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:32.879+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 msgr2=0x7f21ac0c4210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:42:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:32.879+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:42:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:32.879+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:42:32.881 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:32.880+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:42:32.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:32.881+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f2198003490 tx=0x7f21b8070000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:42:33.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:33 smithi189 ceph-mon[36790]: pgmap v6945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:33.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:42:33.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:42:33.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:42:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:33 smithi134 ceph-mon[31966]: pgmap v6945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:42:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:42:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:42:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:33 smithi193 ceph-mon[37473]: pgmap v6945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:42:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:42:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:42:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:34 smithi134 ceph-mon[31966]: pgmap v6946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:42:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:34 smithi134 ceph-mon[31966]: pgmap v6947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11695.307706 secs 2024-07-26T14:42:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:34 smithi189 ceph-mon[36790]: pgmap v6946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:42:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:34 smithi189 ceph-mon[36790]: pgmap v6947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11695.307706 secs 2024-07-26T14:42:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:34 smithi193 ceph-mon[37473]: pgmap v6946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:42:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:34 smithi193 ceph-mon[37473]: pgmap v6947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11695.307706 secs 2024-07-26T14:42:34.895 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:34.895+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:42:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:34.895+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:42:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:34.896+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:42:34.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:34.896+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:42:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:36 smithi134 ceph-mon[31966]: pgmap v6948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:36 smithi189 ceph-mon[36790]: pgmap v6948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:36 smithi193 ceph-mon[37473]: pgmap v6948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11714.732135 secs 2024-07-26T14:42:37.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11714.732135 secs 2024-07-26T14:42:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11714.732135 secs 2024-07-26T14:42:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:37.754+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:42:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:37.754+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:37.754+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:42:38.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:38 smithi134 ceph-mon[31966]: pgmap v6949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:42:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:38 smithi189 ceph-mon[36790]: pgmap v6949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:42:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:38 smithi193 ceph-mon[37473]: pgmap v6949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:42:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11700.307829 secs 2024-07-26T14:42:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:39 smithi134 ceph-mon[31966]: pgmap v6950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11700.307829 secs 2024-07-26T14:42:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:39 smithi189 ceph-mon[36790]: pgmap v6950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11700.307829 secs 2024-07-26T14:42:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:39 smithi193 ceph-mon[37473]: pgmap v6950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:42.289 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:42.288+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:42:42.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:42.288+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:42:42.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:42.288+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11719.732255 secs 2024-07-26T14:42:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:42 smithi193 ceph-mon[37473]: pgmap v6951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11719.732255 secs 2024-07-26T14:42:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:42 smithi134 ceph-mon[31966]: pgmap v6951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11719.732255 secs 2024-07-26T14:42:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:42 smithi189 ceph-mon[36790]: pgmap v6951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:42:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:44 smithi193 ceph-mon[37473]: pgmap v6952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:42:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11705.307993 secs 2024-07-26T14:42:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:44 smithi134 ceph-mon[31966]: pgmap v6952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:42:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11705.307993 secs 2024-07-26T14:42:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:44 smithi189 ceph-mon[36790]: pgmap v6952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:42:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11705.307993 secs 2024-07-26T14:42:46.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:46 smithi134 ceph-mon[31966]: pgmap v6953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:46 smithi193 ceph-mon[37473]: pgmap v6953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:46 smithi189 ceph-mon[36790]: pgmap v6953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11724.732349 secs 2024-07-26T14:42:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11724.732349 secs 2024-07-26T14:42:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11724.732349 secs 2024-07-26T14:42:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:48 smithi193 ceph-mon[37473]: pgmap v6954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:48 smithi134 ceph-mon[31966]: pgmap v6954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:48 smithi189 ceph-mon[36790]: pgmap v6954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11710.308108 secs 2024-07-26T14:42:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11710.308108 secs 2024-07-26T14:42:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11710.308108 secs 2024-07-26T14:42:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:50 smithi193 ceph-mon[37473]: pgmap v6955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:50 smithi134 ceph-mon[31966]: pgmap v6955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:50 smithi189 ceph-mon[36790]: pgmap v6955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:42:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11729.732472 secs 2024-07-26T14:42:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:52 smithi193 ceph-mon[37473]: pgmap v6956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11729.732472 secs 2024-07-26T14:42:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:52 smithi134 ceph-mon[31966]: pgmap v6956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11729.732472 secs 2024-07-26T14:42:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:52 smithi189 ceph-mon[36790]: pgmap v6956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:52.765+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:42:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:52.765+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:52.766 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:42:52.765+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:42:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:54 smithi193 ceph-mon[37473]: pgmap v6957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11715.308229 secs 2024-07-26T14:42:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:54 smithi134 ceph-mon[31966]: pgmap v6957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11715.308229 secs 2024-07-26T14:42:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:54 smithi189 ceph-mon[36790]: pgmap v6957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:42:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11715.308229 secs 2024-07-26T14:42:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:56 smithi193 ceph-mon[37473]: pgmap v6958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:56 smithi134 ceph-mon[31966]: pgmap v6958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:56 smithi189 ceph-mon[36790]: pgmap v6958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:42:57.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:57.304+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:42:57.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:57.304+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:42:57.305 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:42:57.304+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:42:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11734.732592 secs 2024-07-26T14:42:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11734.732592 secs 2024-07-26T14:42:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11734.732592 secs 2024-07-26T14:42:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:58 smithi193 ceph-mon[37473]: pgmap v6959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:58 smithi134 ceph-mon[31966]: pgmap v6959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:58 smithi189 ceph-mon[36790]: pgmap v6959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:42:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:42:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11720.308343 secs 2024-07-26T14:42:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:42:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11720.308343 secs 2024-07-26T14:42:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:42:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11720.308343 secs 2024-07-26T14:43:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:00 smithi134 ceph-mon[31966]: pgmap v6960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:00 smithi189 ceph-mon[36790]: pgmap v6960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:00 smithi193 ceph-mon[37473]: pgmap v6960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11739.732713 secs 2024-07-26T14:43:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:02 smithi134 ceph-mon[31966]: pgmap v6961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11739.732713 secs 2024-07-26T14:43:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:02 smithi189 ceph-mon[36790]: pgmap v6961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11739.732713 secs 2024-07-26T14:43:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:02 smithi193 ceph-mon[37473]: pgmap v6961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:04 smithi134 ceph-mon[31966]: pgmap v6962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11725.308454 secs 2024-07-26T14:43:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:04 smithi189 ceph-mon[36790]: pgmap v6962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11725.308454 secs 2024-07-26T14:43:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:04 smithi193 ceph-mon[37473]: pgmap v6962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11725.308454 secs 2024-07-26T14:43:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:06 smithi134 ceph-mon[31966]: pgmap v6963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:06 smithi189 ceph-mon[36790]: pgmap v6963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:06 smithi193 ceph-mon[37473]: pgmap v6963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11744.732826 secs 2024-07-26T14:43:07.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11744.732826 secs 2024-07-26T14:43:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11744.732826 secs 2024-07-26T14:43:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:43:07.780+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:43:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:43:07.780+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:07.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:43:07.780+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:43:08.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:08 smithi134 ceph-mon[31966]: pgmap v6964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:08 smithi189 ceph-mon[36790]: pgmap v6964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:08 smithi193 ceph-mon[37473]: pgmap v6964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11730.308553 secs 2024-07-26T14:43:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11730.308553 secs 2024-07-26T14:43:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11730.308553 secs 2024-07-26T14:43:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:10 smithi134 ceph-mon[31966]: pgmap v6965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:10 smithi189 ceph-mon[36790]: pgmap v6965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:10 smithi193 ceph-mon[37473]: pgmap v6965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:12.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:43:12.305+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:43:12.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:43:12.305+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:12.307 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:43:12.305+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:43:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11749.732893 secs 2024-07-26T14:43:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:12 smithi134 ceph-mon[31966]: pgmap v6966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11749.732893 secs 2024-07-26T14:43:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:12 smithi189 ceph-mon[36790]: pgmap v6966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11749.732893 secs 2024-07-26T14:43:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:12 smithi193 ceph-mon[37473]: pgmap v6966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:14 smithi134 ceph-mon[31966]: pgmap v6967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11735.308680 secs 2024-07-26T14:43:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:14 smithi189 ceph-mon[36790]: pgmap v6967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11735.308680 secs 2024-07-26T14:43:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:14 smithi193 ceph-mon[37473]: pgmap v6967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11735.308680 secs 2024-07-26T14:43:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:16 smithi134 ceph-mon[31966]: pgmap v6968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:16 smithi189 ceph-mon[36790]: pgmap v6968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:16 smithi193 ceph-mon[37473]: pgmap v6968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11754.733034 secs 2024-07-26T14:43:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11754.733034 secs 2024-07-26T14:43:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11754.733034 secs 2024-07-26T14:43:18.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:18 smithi134 ceph-mon[31966]: pgmap v6969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:18 smithi189 ceph-mon[36790]: pgmap v6969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:18 smithi193 ceph-mon[37473]: pgmap v6969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11740.308760 secs 2024-07-26T14:43:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:19 smithi134 ceph-mon[31966]: pgmap v6970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11740.308760 secs 2024-07-26T14:43:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:19 smithi189 ceph-mon[36790]: pgmap v6970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11740.308760 secs 2024-07-26T14:43:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:19 smithi193 ceph-mon[37473]: pgmap v6970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11759.733155 secs 2024-07-26T14:43:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:22 smithi193 ceph-mon[37473]: pgmap v6971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11759.733155 secs 2024-07-26T14:43:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:22 smithi134 ceph-mon[31966]: pgmap v6971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11759.733155 secs 2024-07-26T14:43:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:22 smithi189 ceph-mon[36790]: pgmap v6971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:43:22.795+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:43:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:43:22.796+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:43:22.797 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:43:22.796+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:24 smithi193 ceph-mon[37473]: pgmap v6972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11745.308883 secs 2024-07-26T14:43:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:24 smithi134 ceph-mon[31966]: pgmap v6972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11745.308883 secs 2024-07-26T14:43:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:24 smithi189 ceph-mon[36790]: pgmap v6972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11745.308883 secs 2024-07-26T14:43:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:26 smithi193 ceph-mon[37473]: pgmap v6973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:26 smithi134 ceph-mon[31966]: pgmap v6973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:26 smithi189 ceph-mon[36790]: pgmap v6973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:27.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:43:27.309+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:43:27.310 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:43:27.309+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:43:27.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:43:27.309+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11764.733274 secs 2024-07-26T14:43:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11764.733274 secs 2024-07-26T14:43:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11764.733274 secs 2024-07-26T14:43:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:28 smithi193 ceph-mon[37473]: pgmap v6974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:28.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:28 smithi134 ceph-mon[31966]: pgmap v6974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:28 smithi189 ceph-mon[36790]: pgmap v6974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11750.309008 secs 2024-07-26T14:43:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11750.309008 secs 2024-07-26T14:43:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11750.309008 secs 2024-07-26T14:43:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:30 smithi193 ceph-mon[37473]: pgmap v6975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:30 smithi134 ceph-mon[31966]: pgmap v6975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:30 smithi189 ceph-mon[36790]: pgmap v6975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11769.733402 secs 2024-07-26T14:43:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:32 smithi134 ceph-mon[31966]: pgmap v6976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11769.733402 secs 2024-07-26T14:43:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:32 smithi193 ceph-mon[37473]: pgmap v6976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11769.733402 secs 2024-07-26T14:43:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:32 smithi189 ceph-mon[36790]: pgmap v6976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:34 smithi134 ceph-mon[31966]: pgmap v6977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:43:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11755.309085 secs 2024-07-26T14:43:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:34 smithi189 ceph-mon[36790]: pgmap v6977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:43:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11755.309085 secs 2024-07-26T14:43:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:34 smithi193 ceph-mon[37473]: pgmap v6977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:43:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11755.309085 secs 2024-07-26T14:43:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:43:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:43:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:35 smithi134 ceph-mon[31966]: pgmap v6978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:43:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:35 smithi134 ceph-mon[31966]: pgmap v6979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:43:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:43:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:43:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:43:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:35 smithi189 ceph-mon[36790]: pgmap v6978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:43:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:35 smithi189 ceph-mon[36790]: pgmap v6979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:43:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:43:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:43:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:43:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:35 smithi193 ceph-mon[37473]: pgmap v6978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:43:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:35 smithi193 ceph-mon[37473]: pgmap v6979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:43:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:43:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:36 smithi134 ceph-mon[31966]: pgmap v6980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:43:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:36 smithi189 ceph-mon[36790]: pgmap v6980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:43:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:36 smithi193 ceph-mon[37473]: pgmap v6980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:43:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11774.733469 secs 2024-07-26T14:43:37.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11774.733469 secs 2024-07-26T14:43:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11774.733469 secs 2024-07-26T14:43:37.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:43:37.804+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:43:37.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:43:37.804+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:37.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:43:37.804+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:43:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:38 smithi134 ceph-mon[31966]: pgmap v6981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:43:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:38 smithi189 ceph-mon[36790]: pgmap v6981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:43:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:38 smithi193 ceph-mon[37473]: pgmap v6981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:43:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11760.309193 secs 2024-07-26T14:43:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11760.309193 secs 2024-07-26T14:43:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11760.309193 secs 2024-07-26T14:43:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:40 smithi189 ceph-mon[36790]: pgmap v6982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:43:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:40 smithi193 ceph-mon[37473]: pgmap v6982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:43:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:40 smithi134 ceph-mon[31966]: pgmap v6982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:43:41.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11779.733626 secs 2024-07-26T14:43:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11779.733626 secs 2024-07-26T14:43:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11779.733626 secs 2024-07-26T14:43:42.325 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:43:42.324+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:43:42.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:43:42.324+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:43:42.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:43:42.324+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:42 smithi189 ceph-mon[36790]: pgmap v6983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:43:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:42 smithi193 ceph-mon[37473]: pgmap v6983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:43:43.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:42 smithi134 ceph-mon[31966]: pgmap v6983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:43:44.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11765.309305 secs 2024-07-26T14:43:44.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11765.309305 secs 2024-07-26T14:43:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11765.309305 secs 2024-07-26T14:43:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:45 smithi134 ceph-mon[31966]: pgmap v6984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:43:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:45 smithi189 ceph-mon[36790]: pgmap v6984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:43:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:45 smithi193 ceph-mon[37473]: pgmap v6984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:43:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:46 smithi134 ceph-mon[31966]: pgmap v6985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:46 smithi189 ceph-mon[36790]: pgmap v6985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:46 smithi193 ceph-mon[37473]: pgmap v6985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11784.733804 secs 2024-07-26T14:43:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11784.733804 secs 2024-07-26T14:43:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11784.733804 secs 2024-07-26T14:43:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:48 smithi134 ceph-mon[31966]: pgmap v6986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:48 smithi189 ceph-mon[36790]: pgmap v6986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:48 smithi193 ceph-mon[37473]: pgmap v6986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11770.309436 secs 2024-07-26T14:43:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11770.309436 secs 2024-07-26T14:43:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11770.309436 secs 2024-07-26T14:43:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:50 smithi189 ceph-mon[36790]: pgmap v6987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:50 smithi193 ceph-mon[37473]: pgmap v6987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:50 smithi134 ceph-mon[31966]: pgmap v6987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:51.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11789.733919 secs 2024-07-26T14:43:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11789.733919 secs 2024-07-26T14:43:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11789.733919 secs 2024-07-26T14:43:52.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:43:52.819+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:43:52.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:43:52.820+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:43:52.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:43:52.820+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:52 smithi189 ceph-mon[36790]: pgmap v6988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:52 smithi193 ceph-mon[37473]: pgmap v6988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:53.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:52 smithi134 ceph-mon[31966]: pgmap v6988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:43:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11775.309577 secs 2024-07-26T14:43:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11775.309577 secs 2024-07-26T14:43:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11775.309577 secs 2024-07-26T14:43:55.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:55 smithi134 ceph-mon[31966]: pgmap v6989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:55 smithi189 ceph-mon[36790]: pgmap v6989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:55 smithi193 ceph-mon[37473]: pgmap v6989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:43:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:56 smithi134 ceph-mon[31966]: pgmap v6990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:56 smithi189 ceph-mon[36790]: pgmap v6990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:56 smithi193 ceph-mon[37473]: pgmap v6990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:43:57.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:43:57.336+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:43:57.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:43:57.336+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:43:57.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:43:57.336+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:43:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11794.733987 secs 2024-07-26T14:43:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11794.733987 secs 2024-07-26T14:43:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11794.733987 secs 2024-07-26T14:43:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:58 smithi134 ceph-mon[31966]: pgmap v6991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:58.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:58 smithi189 ceph-mon[36790]: pgmap v6991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:58 smithi193 ceph-mon[37473]: pgmap v6991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:43:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:43:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11780.309735 secs 2024-07-26T14:43:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:43:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11780.309735 secs 2024-07-26T14:43:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:43:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11780.309735 secs 2024-07-26T14:44:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:00 smithi189 ceph-mon[36790]: pgmap v6992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:00 smithi193 ceph-mon[37473]: pgmap v6992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:00.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:00 smithi134 ceph-mon[31966]: pgmap v6992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11799.734119 secs 2024-07-26T14:44:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11799.734119 secs 2024-07-26T14:44:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11799.734119 secs 2024-07-26T14:44:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:02 smithi189 ceph-mon[36790]: pgmap v6993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:44:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:02 smithi193 ceph-mon[37473]: pgmap v6993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:44:03.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:02 smithi134 ceph-mon[31966]: pgmap v6993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:44:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11785.309880 secs 2024-07-26T14:44:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11785.309880 secs 2024-07-26T14:44:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11785.309880 secs 2024-07-26T14:44:05.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:05 smithi134 ceph-mon[31966]: pgmap v6994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:05 smithi189 ceph-mon[36790]: pgmap v6994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:05 smithi193 ceph-mon[37473]: pgmap v6994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:06 smithi134 ceph-mon[31966]: pgmap v6995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:06 smithi189 ceph-mon[36790]: pgmap v6995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:06 smithi193 ceph-mon[37473]: pgmap v6995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11804.734268 secs 2024-07-26T14:44:07.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11804.734268 secs 2024-07-26T14:44:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11804.734268 secs 2024-07-26T14:44:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:44:07.829+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:44:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:44:07.829+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:44:07.829+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:44:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:08 smithi189 ceph-mon[36790]: pgmap v6996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:08 smithi193 ceph-mon[37473]: pgmap v6996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:08 smithi134 ceph-mon[31966]: pgmap v6996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11790.309994 secs 2024-07-26T14:44:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11790.309994 secs 2024-07-26T14:44:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11790.309994 secs 2024-07-26T14:44:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:10 smithi189 ceph-mon[36790]: pgmap v6997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:10 smithi193 ceph-mon[37473]: pgmap v6997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:10.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:10 smithi134 ceph-mon[31966]: pgmap v6997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11809.734397 secs 2024-07-26T14:44:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11809.734397 secs 2024-07-26T14:44:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11809.734397 secs 2024-07-26T14:44:12.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:44:12.337+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:44:12.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:44:12.337+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:44:12.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:44:12.337+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:12 smithi189 ceph-mon[36790]: pgmap v6998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:12 smithi193 ceph-mon[37473]: pgmap v6998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:12 smithi134 ceph-mon[31966]: pgmap v6998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11795.310151 secs 2024-07-26T14:44:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11795.310151 secs 2024-07-26T14:44:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11795.310151 secs 2024-07-26T14:44:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:15 smithi134 ceph-mon[31966]: pgmap v6999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:15.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:15 smithi189 ceph-mon[36790]: pgmap v6999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:15 smithi193 ceph-mon[37473]: pgmap v6999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:16 smithi134 ceph-mon[31966]: pgmap v7000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:16.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:16 smithi189 ceph-mon[36790]: pgmap v7000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:16 smithi193 ceph-mon[37473]: pgmap v7000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11814.734518 secs 2024-07-26T14:44:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11814.734518 secs 2024-07-26T14:44:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11814.734518 secs 2024-07-26T14:44:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:18 smithi189 ceph-mon[36790]: pgmap v7001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:18 smithi193 ceph-mon[37473]: pgmap v7001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:18 smithi134 ceph-mon[31966]: pgmap v7001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11800.310266 secs 2024-07-26T14:44:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11800.310266 secs 2024-07-26T14:44:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11800.310266 secs 2024-07-26T14:44:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:20 smithi189 ceph-mon[36790]: pgmap v7002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:20 smithi193 ceph-mon[37473]: pgmap v7002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:20 smithi134 ceph-mon[31966]: pgmap v7002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11819.734642 secs 2024-07-26T14:44:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11819.734642 secs 2024-07-26T14:44:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11819.734642 secs 2024-07-26T14:44:22.836 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:44:22.835+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:44:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:44:22.835+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:22.837 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:44:22.835+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:44:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:22 smithi189 ceph-mon[36790]: pgmap v7003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:44:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:22 smithi193 ceph-mon[37473]: pgmap v7003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:44:23.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:22 smithi134 ceph-mon[31966]: pgmap v7003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:44:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11805.310390 secs 2024-07-26T14:44:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11805.310390 secs 2024-07-26T14:44:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11805.310390 secs 2024-07-26T14:44:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:25 smithi134 ceph-mon[31966]: pgmap v7004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:25 smithi189 ceph-mon[36790]: pgmap v7004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:25 smithi193 ceph-mon[37473]: pgmap v7004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:26 smithi134 ceph-mon[31966]: pgmap v7005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:26 smithi189 ceph-mon[36790]: pgmap v7005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:26 smithi193 ceph-mon[37473]: pgmap v7005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:27.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:44:27.340+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:44:27.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:44:27.340+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:27.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:44:27.340+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:44:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11824.734767 secs 2024-07-26T14:44:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11824.734767 secs 2024-07-26T14:44:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11824.734767 secs 2024-07-26T14:44:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:28 smithi189 ceph-mon[36790]: pgmap v7006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:28 smithi193 ceph-mon[37473]: pgmap v7006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:28.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:28 smithi134 ceph-mon[31966]: pgmap v7006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11810.310516 secs 2024-07-26T14:44:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11810.310516 secs 2024-07-26T14:44:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11810.310516 secs 2024-07-26T14:44:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:30 smithi193 ceph-mon[37473]: pgmap v7007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:30 smithi134 ceph-mon[31966]: pgmap v7007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:30 smithi189 ceph-mon[36790]: pgmap v7007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11829.734909 secs 2024-07-26T14:44:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11829.734909 secs 2024-07-26T14:44:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11829.734909 secs 2024-07-26T14:44:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:32 smithi189 ceph-mon[36790]: pgmap v7008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:32 smithi193 ceph-mon[37473]: pgmap v7008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:32 smithi134 ceph-mon[31966]: pgmap v7008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:34.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11815.310669 secs 2024-07-26T14:44:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11815.310669 secs 2024-07-26T14:44:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11815.310669 secs 2024-07-26T14:44:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:35 smithi134 ceph-mon[31966]: pgmap v7009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:35 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:44:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:35 smithi189 ceph-mon[36790]: pgmap v7009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:35 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:44:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:35 smithi193 ceph-mon[37473]: pgmap v7009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:35 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:44:36.561 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:36 smithi134 ceph-mon[31966]: pgmap v7010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:36 smithi189 ceph-mon[36790]: pgmap v7010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:36 smithi193 ceph-mon[37473]: pgmap v7010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11834.735015 secs 2024-07-26T14:44:37.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11834.735015 secs 2024-07-26T14:44:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11834.735015 secs 2024-07-26T14:44:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:44:37.850+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:44:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:44:37.850+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:37.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:44:37.850+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:44:38.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:38 smithi134 ceph-mon[31966]: pgmap v7011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:44:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:38 smithi189 ceph-mon[36790]: pgmap v7011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:44:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:38 smithi193 ceph-mon[37473]: pgmap v7011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:44:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11820.310789 secs 2024-07-26T14:44:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11820.310789 secs 2024-07-26T14:44:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11820.310789 secs 2024-07-26T14:44:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:41 smithi193 ceph-mon[37473]: pgmap v7012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:44:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:44:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:44:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:44:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:44:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:41 smithi193 ceph-mon[37473]: pgmap v7013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:41 smithi193 ceph-mon[37473]: pgmap v7014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:44:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:44:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11839.735159 secs 2024-07-26T14:44:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:41 smithi134 ceph-mon[31966]: pgmap v7012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:44:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:44:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:44:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:44:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:44:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:41 smithi134 ceph-mon[31966]: pgmap v7013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:41 smithi134 ceph-mon[31966]: pgmap v7014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:44:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:44:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11839.735159 secs 2024-07-26T14:44:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:41 smithi189 ceph-mon[36790]: pgmap v7012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:44:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:44:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:44:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:44:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:44:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:41 smithi189 ceph-mon[36790]: pgmap v7013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:44:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:41 smithi189 ceph-mon[36790]: pgmap v7014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T14:44:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:44:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11839.735159 secs 2024-07-26T14:44:42.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:44:42.356+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:44:42.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:44:42.356+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:44:42.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:44:42.356+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:43 smithi134 ceph-mon[31966]: pgmap v7015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:44:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:43 smithi189 ceph-mon[36790]: pgmap v7015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:44:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:43 smithi193 ceph-mon[37473]: pgmap v7015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-07-26T14:44:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11825.310883 secs 2024-07-26T14:44:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11825.310883 secs 2024-07-26T14:44:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11825.310883 secs 2024-07-26T14:44:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:45 smithi134 ceph-mon[31966]: pgmap v7016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:45 smithi189 ceph-mon[36790]: pgmap v7016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:45 smithi193 ceph-mon[37473]: pgmap v7016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:44:47.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11844.735298 secs 2024-07-26T14:44:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11844.735298 secs 2024-07-26T14:44:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11844.735298 secs 2024-07-26T14:44:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:47 smithi134 ceph-mon[31966]: pgmap v7017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T14:44:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:47 smithi189 ceph-mon[36790]: pgmap v7017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T14:44:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:47 smithi193 ceph-mon[37473]: pgmap v7017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T14:44:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11830.311006 secs 2024-07-26T14:44:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:49 smithi134 ceph-mon[31966]: pgmap v7018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:44:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11830.311006 secs 2024-07-26T14:44:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:49 smithi189 ceph-mon[36790]: pgmap v7018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:44:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11830.311006 secs 2024-07-26T14:44:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:49 smithi193 ceph-mon[37473]: pgmap v7018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T14:44:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:51 smithi134 ceph-mon[31966]: pgmap v7019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:44:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11849.735423 secs 2024-07-26T14:44:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:51 smithi189 ceph-mon[36790]: pgmap v7019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:44:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11849.735423 secs 2024-07-26T14:44:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:51 smithi193 ceph-mon[37473]: pgmap v7019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:44:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11849.735423 secs 2024-07-26T14:44:52.867 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:44:52.865+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:44:52.867 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:44:52.866+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:44:52.867 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:44:52.866+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:53 smithi134 ceph-mon[31966]: pgmap v7020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:53 smithi189 ceph-mon[36790]: pgmap v7020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:53 smithi193 ceph-mon[37473]: pgmap v7020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:44:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11835.311148 secs 2024-07-26T14:44:55.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11835.311148 secs 2024-07-26T14:44:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11835.311148 secs 2024-07-26T14:44:56.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:55 smithi134 ceph-mon[31966]: pgmap v7021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:44:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:55 smithi189 ceph-mon[36790]: pgmap v7021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:44:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:55 smithi193 ceph-mon[37473]: pgmap v7021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:44:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11854.735490 secs 2024-07-26T14:44:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11854.735490 secs 2024-07-26T14:44:57.374 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11854.735490 secs 2024-07-26T14:44:57.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:44:57.371+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:44:57.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:44:57.371+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:44:57.375 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:44:57.371+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:44:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:57 smithi189 ceph-mon[36790]: pgmap v7022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:57 smithi193 ceph-mon[37473]: pgmap v7022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:44:58.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:57 smithi134 ceph-mon[31966]: pgmap v7022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11840.311262 secs 2024-07-26T14:45:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:44:59 smithi189 ceph-mon[36790]: pgmap v7023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:00.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11840.311262 secs 2024-07-26T14:45:00.277 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:44:59 smithi193 ceph-mon[37473]: pgmap v7023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11840.311262 secs 2024-07-26T14:45:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:44:59 smithi134 ceph-mon[31966]: pgmap v7023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:01 smithi189 ceph-mon[36790]: pgmap v7024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11859.735572 secs 2024-07-26T14:45:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:01 smithi193 ceph-mon[37473]: pgmap v7024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11859.735572 secs 2024-07-26T14:45:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:01 smithi134 ceph-mon[31966]: pgmap v7024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11859.735572 secs 2024-07-26T14:45:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:03 smithi193 ceph-mon[37473]: pgmap v7025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:03 smithi134 ceph-mon[31966]: pgmap v7025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:03 smithi189 ceph-mon[36790]: pgmap v7025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11845.311382 secs 2024-07-26T14:45:05.289 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11845.311382 secs 2024-07-26T14:45:05.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11845.311382 secs 2024-07-26T14:45:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:05 smithi193 ceph-mon[37473]: pgmap v7026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:45:06.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:05 smithi134 ceph-mon[31966]: pgmap v7026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:45:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:05 smithi189 ceph-mon[36790]: pgmap v7026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:45:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11864.735691 secs 2024-07-26T14:45:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11864.735691 secs 2024-07-26T14:45:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11864.735691 secs 2024-07-26T14:45:07.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:07.881+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:45:07.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:07.881+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:07.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:07.881+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:45:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:07 smithi193 ceph-mon[37473]: pgmap v7027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:08.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:07 smithi134 ceph-mon[31966]: pgmap v7027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:07 smithi189 ceph-mon[36790]: pgmap v7027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11850.311512 secs 2024-07-26T14:45:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:09 smithi193 ceph-mon[37473]: pgmap v7028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:10.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11850.311512 secs 2024-07-26T14:45:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:09 smithi134 ceph-mon[31966]: pgmap v7028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11850.311512 secs 2024-07-26T14:45:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:09 smithi189 ceph-mon[36790]: pgmap v7028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:11 smithi193 ceph-mon[37473]: pgmap v7029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11869.735819 secs 2024-07-26T14:45:12.385 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:12.384+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:45:12.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:12.384+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:12.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:12.384+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:45:12.386 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:11 smithi134 ceph-mon[31966]: pgmap v7029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:12.386 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11869.735819 secs 2024-07-26T14:45:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:11 smithi189 ceph-mon[36790]: pgmap v7029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11869.735819 secs 2024-07-26T14:45:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:13 smithi193 ceph-mon[37473]: pgmap v7030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:13 smithi134 ceph-mon[31966]: pgmap v7030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:13 smithi189 ceph-mon[36790]: pgmap v7030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11855.311640 secs 2024-07-26T14:45:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11855.311640 secs 2024-07-26T14:45:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11855.311640 secs 2024-07-26T14:45:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:16 smithi193 ceph-mon[37473]: pgmap v7031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:45:16.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:16 smithi134 ceph-mon[31966]: pgmap v7031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:45:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:16 smithi189 ceph-mon[36790]: pgmap v7031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:45:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11874.735944 secs 2024-07-26T14:45:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11874.735944 secs 2024-07-26T14:45:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11874.735944 secs 2024-07-26T14:45:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:18 smithi193 ceph-mon[37473]: pgmap v7032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:18.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:18 smithi134 ceph-mon[31966]: pgmap v7032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:18 smithi189 ceph-mon[36790]: pgmap v7032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11860.311783 secs 2024-07-26T14:45:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:20 smithi134 ceph-mon[31966]: pgmap v7033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11860.311783 secs 2024-07-26T14:45:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:20 smithi189 ceph-mon[36790]: pgmap v7033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11860.311783 secs 2024-07-26T14:45:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:20 smithi193 ceph-mon[37473]: pgmap v7033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:22 smithi134 ceph-mon[31966]: pgmap v7034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11879.736093 secs 2024-07-26T14:45:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:22 smithi189 ceph-mon[36790]: pgmap v7034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11879.736093 secs 2024-07-26T14:45:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:22 smithi193 ceph-mon[37473]: pgmap v7034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11879.736093 secs 2024-07-26T14:45:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:22.897+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:45:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:22.897+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:45:22.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:22.897+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:24.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:24.395+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T14:45:24.396 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:24.395+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T14:45:24.396 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:24 smithi134 ceph-mon[31966]: pgmap v7035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:24.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:24.395+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T14:45:24.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:24.395+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T14:45:24.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:24.395+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 93 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T14:45:24.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:24.395+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T14:45:24.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:24.396+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T14:45:24.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:24.396+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T14:45:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:24 smithi189 ceph-mon[36790]: pgmap v7035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:24 smithi193 ceph-mon[37473]: pgmap v7035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11865.311914 secs 2024-07-26T14:45:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11865.311914 secs 2024-07-26T14:45:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11865.311914 secs 2024-07-26T14:45:26.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:26 smithi134 ceph-mon[31966]: pgmap v7036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:45:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:26 smithi189 ceph-mon[36790]: pgmap v7036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:45:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:26 smithi193 ceph-mon[37473]: pgmap v7036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:45:27.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:27.395+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:45:27.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:27.395+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:45:27.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:27.395+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:27.397 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11884.736216 secs 2024-07-26T14:45:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11884.736216 secs 2024-07-26T14:45:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11884.736216 secs 2024-07-26T14:45:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:28 smithi134 ceph-mon[31966]: pgmap v7037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:28 smithi189 ceph-mon[36790]: pgmap v7037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:28 smithi193 ceph-mon[37473]: pgmap v7037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:30.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11870.312037 secs 2024-07-26T14:45:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:30 smithi134 ceph-mon[31966]: pgmap v7038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11870.312037 secs 2024-07-26T14:45:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:30 smithi189 ceph-mon[36790]: pgmap v7038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11870.312037 secs 2024-07-26T14:45:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:30 smithi193 ceph-mon[37473]: pgmap v7038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:32 smithi134 ceph-mon[31966]: pgmap v7039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11889.736324 secs 2024-07-26T14:45:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:32 smithi189 ceph-mon[36790]: pgmap v7039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11889.736324 secs 2024-07-26T14:45:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:32 smithi193 ceph-mon[37473]: pgmap v7039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11889.736324 secs 2024-07-26T14:45:34.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:34 smithi134 ceph-mon[31966]: pgmap v7040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:34 smithi189 ceph-mon[36790]: pgmap v7040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:34 smithi193 ceph-mon[37473]: pgmap v7040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:34.928 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:34.926+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:45:34.928 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:34.926+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:45:34.929 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:34.927+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T14:45:34.929 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:34.927+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:45:34.930 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:34.927+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 94 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T14:45:34.930 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:34.927+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:45:34.930 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:34.928+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:45:34.930 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:34.928+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:45:35.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11875.312142 secs 2024-07-26T14:45:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11875.312142 secs 2024-07-26T14:45:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11875.312142 secs 2024-07-26T14:45:36.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:36 smithi134 ceph-mon[31966]: pgmap v7041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:45:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:36 smithi189 ceph-mon[36790]: pgmap v7041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:45:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:36 smithi193 ceph-mon[37473]: pgmap v7041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:45:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11894.736444 secs 2024-07-26T14:45:37.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11894.736444 secs 2024-07-26T14:45:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11894.736444 secs 2024-07-26T14:45:37.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:37.906+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:45:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:37.906+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:37.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:37.906+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:45:38.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:38 smithi134 ceph-mon[31966]: pgmap v7042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:38 smithi189 ceph-mon[36790]: pgmap v7042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:38 smithi193 ceph-mon[37473]: pgmap v7042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11880.312291 secs 2024-07-26T14:45:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11880.312291 secs 2024-07-26T14:45:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11880.312291 secs 2024-07-26T14:45:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:40 smithi134 ceph-mon[31966]: pgmap v7043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:40 smithi189 ceph-mon[36790]: pgmap v7043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:40 smithi193 ceph-mon[37473]: pgmap v7043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:41.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:45:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:45:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:45:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:42 smithi189 ceph-mon[36790]: pgmap v7044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11899.736515 secs 2024-07-26T14:45:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:42.219 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:42 smithi193 ceph-mon[37473]: pgmap v7044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:42.219 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11899.736515 secs 2024-07-26T14:45:42.219 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:42.220 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:42.220 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:42.220 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:42.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:42.410+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:45:42.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:42.411+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:45:42.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:42.411+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:42.413 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:42 smithi134 ceph-mon[31966]: pgmap v7044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:45:42.413 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11899.736515 secs 2024-07-26T14:45:42.413 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:42.413 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:42.413 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:42.413 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:45:43.306 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:43 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:45:43.306 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:43 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:45:43.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:43 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:45:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:43 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:45:43.470 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:43 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:45:43.470 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:43 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:45:44.299 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:44 smithi189 ceph-mon[36790]: pgmap v7045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:44.299 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11885.312377 secs 2024-07-26T14:45:44.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:44 smithi134 ceph-mon[31966]: pgmap v7045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:44.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11885.312377 secs 2024-07-26T14:45:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:44 smithi193 ceph-mon[37473]: pgmap v7045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:45:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11885.312377 secs 2024-07-26T14:45:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:45:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:45:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:45 smithi134 ceph-mon[31966]: pgmap v7046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:45:46.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:45:46.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:45:46.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:45:46.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:45:46.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:45 smithi134 ceph-mon[31966]: pgmap v7047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T14:45:46.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:45 smithi134 ceph-mon[31966]: pgmap v7048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T14:45:46.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:45:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:45:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:45:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:45 smithi189 ceph-mon[36790]: pgmap v7046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:45:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:45:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:45:46.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:45:46.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:45:46.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:45 smithi189 ceph-mon[36790]: pgmap v7047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T14:45:46.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:45 smithi189 ceph-mon[36790]: pgmap v7048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T14:45:46.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:45:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:45:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:45:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:45 smithi193 ceph-mon[37473]: pgmap v7046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:45:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:45:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:45:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:45:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:45:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:45 smithi193 ceph-mon[37473]: pgmap v7047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T14:45:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:45 smithi193 ceph-mon[37473]: pgmap v7048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T14:45:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:45:47.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11904.736659 secs 2024-07-26T14:45:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11904.736659 secs 2024-07-26T14:45:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11904.736659 secs 2024-07-26T14:45:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:47 smithi134 ceph-mon[31966]: pgmap v7049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T14:45:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:47 smithi189 ceph-mon[36790]: pgmap v7049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T14:45:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:47 smithi193 ceph-mon[37473]: pgmap v7049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T14:45:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11890.312502 secs 2024-07-26T14:45:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11890.312502 secs 2024-07-26T14:45:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11890.312502 secs 2024-07-26T14:45:50.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:50 smithi134 ceph-mon[31966]: pgmap v7050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:45:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:50 smithi189 ceph-mon[36790]: pgmap v7050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:45:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:50 smithi193 ceph-mon[37473]: pgmap v7050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:45:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11909.736807 secs 2024-07-26T14:45:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:52 smithi134 ceph-mon[31966]: pgmap v7051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T14:45:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11909.736807 secs 2024-07-26T14:45:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:52 smithi189 ceph-mon[36790]: pgmap v7051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T14:45:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11909.736807 secs 2024-07-26T14:45:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:52 smithi193 ceph-mon[37473]: pgmap v7051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T14:45:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:52.921+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:45:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:52.921+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:52.922 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:45:52.921+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:45:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:54 smithi134 ceph-mon[31966]: pgmap v7052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T14:45:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11895.312631 secs 2024-07-26T14:45:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:54 smithi189 ceph-mon[36790]: pgmap v7052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T14:45:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11895.312631 secs 2024-07-26T14:45:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:54 smithi193 ceph-mon[37473]: pgmap v7052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T14:45:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11895.312631 secs 2024-07-26T14:45:56.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:56 smithi134 ceph-mon[31966]: pgmap v7053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:45:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:56 smithi189 ceph-mon[36790]: pgmap v7053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:45:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:56 smithi193 ceph-mon[37473]: pgmap v7053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:45:57.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:57.424+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:45:57.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:57.424+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:45:57.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:45:57.424+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:45:57.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11914.736909 secs 2024-07-26T14:45:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11914.736909 secs 2024-07-26T14:45:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11914.736909 secs 2024-07-26T14:45:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:58 smithi189 ceph-mon[36790]: pgmap v7054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:58 smithi193 ceph-mon[37473]: pgmap v7054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:58.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:58 smithi134 ceph-mon[31966]: pgmap v7054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:45:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:45:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11900.312802 secs 2024-07-26T14:45:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:45:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11900.312802 secs 2024-07-26T14:45:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:45:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11900.312802 secs 2024-07-26T14:46:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:00 smithi193 ceph-mon[37473]: pgmap v7055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:00 smithi134 ceph-mon[31966]: pgmap v7055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:00 smithi189 ceph-mon[36790]: pgmap v7055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11919.737029 secs 2024-07-26T14:46:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:02 smithi193 ceph-mon[37473]: pgmap v7056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11919.737029 secs 2024-07-26T14:46:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:02 smithi134 ceph-mon[31966]: pgmap v7056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11919.737029 secs 2024-07-26T14:46:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:02 smithi189 ceph-mon[36790]: pgmap v7056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:04 smithi193 ceph-mon[37473]: pgmap v7057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11905.312881 secs 2024-07-26T14:46:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:04 smithi134 ceph-mon[31966]: pgmap v7057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11905.312881 secs 2024-07-26T14:46:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:04 smithi189 ceph-mon[36790]: pgmap v7057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11905.312881 secs 2024-07-26T14:46:06.495 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:06 smithi189 ceph-mon[36790]: pgmap v7058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:06 smithi193 ceph-mon[37473]: pgmap v7058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:06 smithi134 ceph-mon[31966]: pgmap v7058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11924.737091 secs 2024-07-26T14:46:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11924.737091 secs 2024-07-26T14:46:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11924.737091 secs 2024-07-26T14:46:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:46:07.933+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:46:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:46:07.933+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:46:07.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:46:07.933+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:08 smithi193 ceph-mon[37473]: pgmap v7059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:08.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:08 smithi134 ceph-mon[31966]: pgmap v7059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:08 smithi189 ceph-mon[36790]: pgmap v7059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11910.312957 secs 2024-07-26T14:46:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11910.312957 secs 2024-07-26T14:46:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11910.312957 secs 2024-07-26T14:46:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:10 smithi193 ceph-mon[37473]: pgmap v7060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:10 smithi134 ceph-mon[31966]: pgmap v7060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:10 smithi189 ceph-mon[36790]: pgmap v7060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:12.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:46:12.430+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:46:12.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:46:12.430+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:12.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:46:12.430+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:46:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11929.737169 secs 2024-07-26T14:46:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:12 smithi193 ceph-mon[37473]: pgmap v7061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11929.737169 secs 2024-07-26T14:46:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:12 smithi134 ceph-mon[31966]: pgmap v7061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11929.737169 secs 2024-07-26T14:46:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:12 smithi189 ceph-mon[36790]: pgmap v7061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:14 smithi193 ceph-mon[37473]: pgmap v7062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11915.313069 secs 2024-07-26T14:46:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:14 smithi134 ceph-mon[31966]: pgmap v7062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11915.313069 secs 2024-07-26T14:46:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:14 smithi189 ceph-mon[36790]: pgmap v7062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11915.313069 secs 2024-07-26T14:46:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:16 smithi193 ceph-mon[37473]: pgmap v7063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:16 smithi134 ceph-mon[31966]: pgmap v7063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:16.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:16 smithi189 ceph-mon[36790]: pgmap v7063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11934.737261 secs 2024-07-26T14:46:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11934.737261 secs 2024-07-26T14:46:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11934.737261 secs 2024-07-26T14:46:18.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:18 smithi193 ceph-mon[37473]: pgmap v7064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:18.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:18 smithi134 ceph-mon[31966]: pgmap v7064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:18 smithi189 ceph-mon[36790]: pgmap v7064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11920.313135 secs 2024-07-26T14:46:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11920.313135 secs 2024-07-26T14:46:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11920.313135 secs 2024-07-26T14:46:20.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:20 smithi134 ceph-mon[31966]: pgmap v7065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:20 smithi189 ceph-mon[36790]: pgmap v7065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:20 smithi193 ceph-mon[37473]: pgmap v7065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11939.737401 secs 2024-07-26T14:46:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:22 smithi134 ceph-mon[31966]: pgmap v7066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11939.737401 secs 2024-07-26T14:46:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:22 smithi189 ceph-mon[36790]: pgmap v7066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11939.737401 secs 2024-07-26T14:46:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:22 smithi193 ceph-mon[37473]: pgmap v7066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:22.949 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:46:22.948+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:46:22.950 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:46:22.948+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:22.950 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:46:22.948+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:46:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:24 smithi134 ceph-mon[31966]: pgmap v7067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11925.313258 secs 2024-07-26T14:46:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:24 smithi189 ceph-mon[36790]: pgmap v7067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11925.313258 secs 2024-07-26T14:46:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:24 smithi193 ceph-mon[37473]: pgmap v7067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11925.313258 secs 2024-07-26T14:46:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:26 smithi134 ceph-mon[31966]: pgmap v7068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:26 smithi189 ceph-mon[36790]: pgmap v7068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:26 smithi193 ceph-mon[37473]: pgmap v7068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:27.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:46:27.445+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:46:27.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:46:27.445+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:46:27.446 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:46:27.445+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11944.737515 secs 2024-07-26T14:46:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11944.737515 secs 2024-07-26T14:46:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11944.737515 secs 2024-07-26T14:46:28.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:28 smithi134 ceph-mon[31966]: pgmap v7069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:28 smithi189 ceph-mon[36790]: pgmap v7069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:28 smithi193 ceph-mon[37473]: pgmap v7069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11930.313345 secs 2024-07-26T14:46:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11930.313345 secs 2024-07-26T14:46:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11930.313345 secs 2024-07-26T14:46:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:30 smithi134 ceph-mon[31966]: pgmap v7070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:30 smithi189 ceph-mon[36790]: pgmap v7070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:30 smithi193 ceph-mon[37473]: pgmap v7070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11949.737637 secs 2024-07-26T14:46:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:32 smithi134 ceph-mon[31966]: pgmap v7071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11949.737637 secs 2024-07-26T14:46:32.742 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:32 smithi189 ceph-mon[36790]: pgmap v7071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11949.737637 secs 2024-07-26T14:46:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:32 smithi193 ceph-mon[37473]: pgmap v7071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:34 smithi134 ceph-mon[31966]: pgmap v7072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11935.313434 secs 2024-07-26T14:46:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:34 smithi189 ceph-mon[36790]: pgmap v7072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11935.313434 secs 2024-07-26T14:46:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:34 smithi193 ceph-mon[37473]: pgmap v7072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11935.313434 secs 2024-07-26T14:46:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:36 smithi134 ceph-mon[31966]: pgmap v7073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:36 smithi189 ceph-mon[36790]: pgmap v7073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:36 smithi193 ceph-mon[37473]: pgmap v7073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11954.737749 secs 2024-07-26T14:46:37.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11954.737749 secs 2024-07-26T14:46:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11954.737749 secs 2024-07-26T14:46:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:46:37.957+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:46:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:46:37.957+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:37.958 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:46:37.957+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:46:38.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:38 smithi134 ceph-mon[31966]: pgmap v7074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:38 smithi189 ceph-mon[36790]: pgmap v7074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:38 smithi193 ceph-mon[37473]: pgmap v7074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:46:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11940.313563 secs 2024-07-26T14:46:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11940.313563 secs 2024-07-26T14:46:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11940.313563 secs 2024-07-26T14:46:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:40 smithi134 ceph-mon[31966]: pgmap v7075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:40 smithi189 ceph-mon[36790]: pgmap v7075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:40 smithi193 ceph-mon[37473]: pgmap v7075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:46:42.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:46:42.449+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:46:42.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:46:42.449+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:46:42.450 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:46:42.449+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11959.737845 secs 2024-07-26T14:46:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:42 smithi134 ceph-mon[31966]: pgmap v7076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11959.737845 secs 2024-07-26T14:46:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:42 smithi189 ceph-mon[36790]: pgmap v7076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11959.737845 secs 2024-07-26T14:46:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:42 smithi193 ceph-mon[37473]: pgmap v7076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:43 smithi189 ceph-mon[36790]: pgmap v7077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:43 smithi193 ceph-mon[37473]: pgmap v7077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:43.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:43 smithi134 ceph-mon[31966]: pgmap v7077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11945.313707 secs 2024-07-26T14:46:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11945.313707 secs 2024-07-26T14:46:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11945.313707 secs 2024-07-26T14:46:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:45 smithi193 ceph-mon[37473]: pgmap v7078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:46:45.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:45 smithi134 ceph-mon[31966]: pgmap v7078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:45.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:46:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:45 smithi189 ceph-mon[36790]: pgmap v7078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:46:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:46:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:46:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:46:46.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:46:46.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:46:46.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:46:46.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:46:46.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: pgmap v7079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:46:46.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: pgmap v7080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:46:46.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:46:46.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.825 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.825 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.825 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.825 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:46:46.825 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:46:46.825 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:46:46.825 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:46:46.825 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:46:46.825 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:46:46.825 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: pgmap v7079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:46:46.826 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: pgmap v7080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:46:46.826 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:46:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:46:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:46:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:46:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:46:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:46:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:46:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:46:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: pgmap v7079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:46:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: pgmap v7080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:46:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:46:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11964.737918 secs 2024-07-26T14:46:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11964.737918 secs 2024-07-26T14:46:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11964.737918 secs 2024-07-26T14:46:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:48 smithi193 ceph-mon[37473]: pgmap v7081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:46:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:48 smithi134 ceph-mon[31966]: pgmap v7081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:46:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:48 smithi189 ceph-mon[36790]: pgmap v7081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:46:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11950.313812 secs 2024-07-26T14:46:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11950.313812 secs 2024-07-26T14:46:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11950.313812 secs 2024-07-26T14:46:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:50 smithi193 ceph-mon[37473]: pgmap v7082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:46:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:50 smithi134 ceph-mon[31966]: pgmap v7082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:46:50.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:50 smithi189 ceph-mon[36790]: pgmap v7082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:46:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11969.738073 secs 2024-07-26T14:46:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11969.738073 secs 2024-07-26T14:46:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11969.738073 secs 2024-07-26T14:46:52.974 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:46:52.973+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:46:52.974 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:46:52.973+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:46:52.974 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:46:52.973+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:52 smithi193 ceph-mon[37473]: pgmap v7083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:46:53.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:52 smithi134 ceph-mon[31966]: pgmap v7083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:46:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:52 smithi189 ceph-mon[36790]: pgmap v7083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:46:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11955.313961 secs 2024-07-26T14:46:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11955.313961 secs 2024-07-26T14:46:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11955.313961 secs 2024-07-26T14:46:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:55 smithi189 ceph-mon[36790]: pgmap v7084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:46:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:55 smithi193 ceph-mon[37473]: pgmap v7084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:46:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:55 smithi134 ceph-mon[31966]: pgmap v7084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:46:57.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:46:57.457+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:46:57.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:46:57.457+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:46:57.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:46:57.457+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:46:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11974.738190 secs 2024-07-26T14:46:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:57 smithi134 ceph-mon[31966]: pgmap v7085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:46:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11974.738190 secs 2024-07-26T14:46:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:57 smithi189 ceph-mon[36790]: pgmap v7085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:46:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11974.738190 secs 2024-07-26T14:46:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:57 smithi193 ceph-mon[37473]: pgmap v7085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:46:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:59 smithi134 ceph-mon[31966]: pgmap v7086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:46:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11960.314045 secs 2024-07-26T14:46:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:59 smithi189 ceph-mon[36790]: pgmap v7086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:46:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11960.314045 secs 2024-07-26T14:46:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:59 smithi193 ceph-mon[37473]: pgmap v7086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:46:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:46:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11960.314045 secs 2024-07-26T14:47:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:01 smithi134 ceph-mon[31966]: pgmap v7087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:01 smithi189 ceph-mon[36790]: pgmap v7087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:01 smithi193 ceph-mon[37473]: pgmap v7087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11979.738320 secs 2024-07-26T14:47:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11979.738320 secs 2024-07-26T14:47:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11979.738320 secs 2024-07-26T14:47:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:03 smithi134 ceph-mon[31966]: pgmap v7088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:03 smithi189 ceph-mon[36790]: pgmap v7088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:03 smithi193 ceph-mon[37473]: pgmap v7088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11965.314169 secs 2024-07-26T14:47:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11965.314169 secs 2024-07-26T14:47:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11965.314169 secs 2024-07-26T14:47:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:05 smithi134 ceph-mon[31966]: pgmap v7089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:05 smithi189 ceph-mon[36790]: pgmap v7089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:05 smithi193 ceph-mon[37473]: pgmap v7089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:06 smithi134 ceph-mon[31966]: pgmap v7090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:06 smithi189 ceph-mon[36790]: pgmap v7090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:06 smithi193 ceph-mon[37473]: pgmap v7090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:07.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11984.738417 secs 2024-07-26T14:47:07.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11984.738417 secs 2024-07-26T14:47:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11984.738417 secs 2024-07-26T14:47:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:47:07.984+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:47:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:47:07.984+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:07.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:47:07.984+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:47:08.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:08 smithi189 ceph-mon[36790]: pgmap v7091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:08 smithi193 ceph-mon[37473]: pgmap v7091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:08 smithi134 ceph-mon[31966]: pgmap v7091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11970.314292 secs 2024-07-26T14:47:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11970.314292 secs 2024-07-26T14:47:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11970.314292 secs 2024-07-26T14:47:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:10 smithi134 ceph-mon[31966]: pgmap v7092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:10 smithi193 ceph-mon[37473]: pgmap v7092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:10 smithi189 ceph-mon[36790]: pgmap v7092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11989.738533 secs 2024-07-26T14:47:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11989.738533 secs 2024-07-26T14:47:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11989.738533 secs 2024-07-26T14:47:12.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:47:12.469+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:47:12.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:47:12.469+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:12.471 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:47:12.469+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:47:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:12 smithi134 ceph-mon[31966]: pgmap v7093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:12 smithi193 ceph-mon[37473]: pgmap v7093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:12 smithi189 ceph-mon[36790]: pgmap v7093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11975.314372 secs 2024-07-26T14:47:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11975.314372 secs 2024-07-26T14:47:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11975.314372 secs 2024-07-26T14:47:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:15 smithi189 ceph-mon[36790]: pgmap v7094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:15 smithi193 ceph-mon[37473]: pgmap v7094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:15 smithi134 ceph-mon[31966]: pgmap v7094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11994.738654 secs 2024-07-26T14:47:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:17 smithi134 ceph-mon[31966]: pgmap v7095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11994.738654 secs 2024-07-26T14:47:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:17 smithi189 ceph-mon[36790]: pgmap v7095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11994.738654 secs 2024-07-26T14:47:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:17 smithi193 ceph-mon[37473]: pgmap v7095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:19 smithi134 ceph-mon[31966]: pgmap v7096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11980.314492 secs 2024-07-26T14:47:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:19 smithi189 ceph-mon[36790]: pgmap v7096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11980.314492 secs 2024-07-26T14:47:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:19 smithi193 ceph-mon[37473]: pgmap v7096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11980.314492 secs 2024-07-26T14:47:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:21 smithi134 ceph-mon[31966]: pgmap v7097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:21.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:21 smithi189 ceph-mon[36790]: pgmap v7097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:21 smithi193 ceph-mon[37473]: pgmap v7097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 11999.738806 secs 2024-07-26T14:47:22.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 11999.738806 secs 2024-07-26T14:47:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 11999.738806 secs 2024-07-26T14:47:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:47:23.000+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:47:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:47:23.000+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:47:23.001 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:47:23.000+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:23 smithi134 ceph-mon[31966]: pgmap v7098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:23 smithi189 ceph-mon[36790]: pgmap v7098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:23 smithi193 ceph-mon[37473]: pgmap v7098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11985.314613 secs 2024-07-26T14:47:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:24 smithi134 ceph-mon[31966]: pgmap v7099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11985.314613 secs 2024-07-26T14:47:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:24 smithi189 ceph-mon[36790]: pgmap v7099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11985.314613 secs 2024-07-26T14:47:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:24 smithi193 ceph-mon[37473]: pgmap v7099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:27.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:47:27.485+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:47:27.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:47:27.485+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:47:27.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:47:27.485+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12004.738927 secs 2024-07-26T14:47:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:27 smithi134 ceph-mon[31966]: pgmap v7100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12004.738927 secs 2024-07-26T14:47:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:27 smithi189 ceph-mon[36790]: pgmap v7100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12004.738927 secs 2024-07-26T14:47:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:27 smithi193 ceph-mon[37473]: pgmap v7100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:29 smithi134 ceph-mon[31966]: pgmap v7101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11990.314735 secs 2024-07-26T14:47:29.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:29 smithi189 ceph-mon[36790]: pgmap v7101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11990.314735 secs 2024-07-26T14:47:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:29 smithi193 ceph-mon[37473]: pgmap v7101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11990.314735 secs 2024-07-26T14:47:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:31 smithi134 ceph-mon[31966]: pgmap v7102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:31 smithi189 ceph-mon[36790]: pgmap v7102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:31 smithi193 ceph-mon[37473]: pgmap v7102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12009.739047 secs 2024-07-26T14:47:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12009.739047 secs 2024-07-26T14:47:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12009.739047 secs 2024-07-26T14:47:33.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:33 smithi134 ceph-mon[31966]: pgmap v7103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:33 smithi189 ceph-mon[36790]: pgmap v7103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:33 smithi193 ceph-mon[37473]: pgmap v7103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 11995.314855 secs 2024-07-26T14:47:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 11995.314855 secs 2024-07-26T14:47:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 11995.314855 secs 2024-07-26T14:47:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:35 smithi134 ceph-mon[31966]: pgmap v7104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:35 smithi189 ceph-mon[36790]: pgmap v7104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:35 smithi193 ceph-mon[37473]: pgmap v7104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:36 smithi189 ceph-mon[36790]: pgmap v7105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:36 smithi193 ceph-mon[37473]: pgmap v7105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:36 smithi134 ceph-mon[31966]: pgmap v7105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:37.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12014.739177 secs 2024-07-26T14:47:37.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12014.739177 secs 2024-07-26T14:47:37.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12014.739177 secs 2024-07-26T14:47:38.014 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:47:38.013+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:47:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:47:38.013+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:47:38.015 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:47:38.013+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:38 smithi189 ceph-mon[36790]: pgmap v7106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:38 smithi193 ceph-mon[37473]: pgmap v7106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:38 smithi134 ceph-mon[31966]: pgmap v7106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:47:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12000.315009 secs 2024-07-26T14:47:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12000.315009 secs 2024-07-26T14:47:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12000.315009 secs 2024-07-26T14:47:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:40 smithi193 ceph-mon[37473]: pgmap v7107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:40 smithi134 ceph-mon[31966]: pgmap v7107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:40 smithi189 ceph-mon[36790]: pgmap v7107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12019.739311 secs 2024-07-26T14:47:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12019.739311 secs 2024-07-26T14:47:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12019.739311 secs 2024-07-26T14:47:42.497 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:47:42.496+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:47:42.497 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:47:42.496+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:42.498 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:47:42.496+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:47:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:42 smithi134 ceph-mon[31966]: pgmap v7108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:42 smithi193 ceph-mon[37473]: pgmap v7108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:42 smithi189 ceph-mon[36790]: pgmap v7108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:47:44.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12005.315134 secs 2024-07-26T14:47:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12005.315134 secs 2024-07-26T14:47:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12005.315134 secs 2024-07-26T14:47:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:45 smithi189 ceph-mon[36790]: pgmap v7109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:45 smithi193 ceph-mon[37473]: pgmap v7109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:45 smithi134 ceph-mon[31966]: pgmap v7109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:47:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12024.739423 secs 2024-07-26T14:47:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:47 smithi134 ceph-mon[31966]: pgmap v7110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:47:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12024.739423 secs 2024-07-26T14:47:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:47 smithi189 ceph-mon[36790]: pgmap v7110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:47:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12024.739423 secs 2024-07-26T14:47:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:47 smithi193 ceph-mon[37473]: pgmap v7110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:47:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:47:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:47:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:47:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:47:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:47:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:47:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:47:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:47:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:47:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:47:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:49 smithi134 ceph-mon[31966]: pgmap v7111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:47:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:49 smithi134 ceph-mon[31966]: pgmap v7112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12010.315226 secs 2024-07-26T14:47:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:49 smithi189 ceph-mon[36790]: pgmap v7111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:47:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:49 smithi189 ceph-mon[36790]: pgmap v7112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12010.315226 secs 2024-07-26T14:47:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:49 smithi193 ceph-mon[37473]: pgmap v7111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:47:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:49 smithi193 ceph-mon[37473]: pgmap v7112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12010.315226 secs 2024-07-26T14:47:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:50 smithi134 ceph-mon[31966]: pgmap v7113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:47:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:50 smithi189 ceph-mon[36790]: pgmap v7113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:47:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:50 smithi193 ceph-mon[37473]: pgmap v7113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:47:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12029.739546 secs 2024-07-26T14:47:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:52 smithi134 ceph-mon[31966]: pgmap v7114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:47:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12029.739546 secs 2024-07-26T14:47:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:52 smithi189 ceph-mon[36790]: pgmap v7114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:47:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12029.739546 secs 2024-07-26T14:47:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:52 smithi193 ceph-mon[37473]: pgmap v7114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:47:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:47:53.029+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:47:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:47:53.029+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:53.030 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:47:53.029+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:47:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:54 smithi134 ceph-mon[31966]: pgmap v7115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12015.315342 secs 2024-07-26T14:47:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:54 smithi189 ceph-mon[36790]: pgmap v7115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12015.315342 secs 2024-07-26T14:47:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:54 smithi193 ceph-mon[37473]: pgmap v7115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12015.315342 secs 2024-07-26T14:47:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:56 smithi134 ceph-mon[31966]: pgmap v7116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:56 smithi189 ceph-mon[36790]: pgmap v7116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:56 smithi193 ceph-mon[37473]: pgmap v7116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:47:57.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:47:57.512+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:47:57.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:47:57.512+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:47:57.513 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:47:57.512+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:47:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12034.739640 secs 2024-07-26T14:47:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12034.739640 secs 2024-07-26T14:47:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12034.739640 secs 2024-07-26T14:47:58.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:58 smithi134 ceph-mon[31966]: pgmap v7117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:47:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:58 smithi189 ceph-mon[36790]: pgmap v7117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:47:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:58 smithi193 ceph-mon[37473]: pgmap v7117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:47:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:47:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12020.315466 secs 2024-07-26T14:47:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:47:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12020.315466 secs 2024-07-26T14:48:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:47:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12020.315466 secs 2024-07-26T14:48:00.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:00 smithi134 ceph-mon[31966]: pgmap v7118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:48:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:00 smithi189 ceph-mon[36790]: pgmap v7118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:48:01.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:00 smithi193 ceph-mon[37473]: pgmap v7118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:48:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12039.739772 secs 2024-07-26T14:48:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:02 smithi134 ceph-mon[31966]: pgmap v7119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12039.739772 secs 2024-07-26T14:48:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:02 smithi189 ceph-mon[36790]: pgmap v7119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12039.739772 secs 2024-07-26T14:48:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:02 smithi193 ceph-mon[37473]: pgmap v7119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:04 smithi134 ceph-mon[31966]: pgmap v7120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12025.315592 secs 2024-07-26T14:48:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:04 smithi189 ceph-mon[36790]: pgmap v7120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12025.315592 secs 2024-07-26T14:48:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:04 smithi193 ceph-mon[37473]: pgmap v7120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12025.315592 secs 2024-07-26T14:48:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:06 smithi134 ceph-mon[31966]: pgmap v7121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:06.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:06 smithi189 ceph-mon[36790]: pgmap v7121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:06 smithi193 ceph-mon[37473]: pgmap v7121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12044.739895 secs 2024-07-26T14:48:07.924 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12044.739895 secs 2024-07-26T14:48:07.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12044.739895 secs 2024-07-26T14:48:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:48:08.035+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:48:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:48:08.035+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:48:08.035+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:48:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:08 smithi134 ceph-mon[31966]: pgmap v7122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:08 smithi189 ceph-mon[36790]: pgmap v7122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:08 smithi193 ceph-mon[37473]: pgmap v7122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12030.315734 secs 2024-07-26T14:48:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12030.315734 secs 2024-07-26T14:48:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12030.315734 secs 2024-07-26T14:48:10.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:10 smithi134 ceph-mon[31966]: pgmap v7123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:48:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:10 smithi189 ceph-mon[36790]: pgmap v7123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:48:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:10 smithi193 ceph-mon[37473]: pgmap v7123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:48:12.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:48:12.528+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:48:12.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:48:12.528+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:12.529 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:48:12.528+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:48:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12049.740008 secs 2024-07-26T14:48:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:12 smithi134 ceph-mon[31966]: pgmap v7124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12049.740008 secs 2024-07-26T14:48:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:12 smithi189 ceph-mon[36790]: pgmap v7124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12049.740008 secs 2024-07-26T14:48:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:12 smithi193 ceph-mon[37473]: pgmap v7124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:14 smithi134 ceph-mon[31966]: pgmap v7125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:48:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12035.315858 secs 2024-07-26T14:48:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:14 smithi189 ceph-mon[36790]: pgmap v7125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:48:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12035.315858 secs 2024-07-26T14:48:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:14 smithi193 ceph-mon[37473]: pgmap v7125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:48:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12035.315858 secs 2024-07-26T14:48:17.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:16 smithi134 ceph-mon[31966]: pgmap v7126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:48:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12054.740151 secs 2024-07-26T14:48:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:16 smithi189 ceph-mon[36790]: pgmap v7126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:48:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12054.740151 secs 2024-07-26T14:48:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:16 smithi193 ceph-mon[37473]: pgmap v7126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:48:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12054.740151 secs 2024-07-26T14:48:19.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:18 smithi134 ceph-mon[31966]: pgmap v7127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:48:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:18 smithi189 ceph-mon[36790]: pgmap v7127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:48:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:18 smithi193 ceph-mon[37473]: pgmap v7127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:48:20.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12040.315987 secs 2024-07-26T14:48:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12040.315987 secs 2024-07-26T14:48:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12040.315987 secs 2024-07-26T14:48:21.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:20 smithi134 ceph-mon[31966]: pgmap v7128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:48:21.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:20 smithi189 ceph-mon[36790]: pgmap v7128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:48:21.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:20 smithi193 ceph-mon[37473]: pgmap v7128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:48:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12059.740268 secs 2024-07-26T14:48:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12059.740268 secs 2024-07-26T14:48:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12059.740268 secs 2024-07-26T14:48:23.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:48:23.041+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:48:23.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:48:23.042+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:48:23.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:48:23.042+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:23.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:22 smithi134 ceph-mon[31966]: pgmap v7129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:48:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:22 smithi189 ceph-mon[36790]: pgmap v7129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:48:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:22 smithi193 ceph-mon[37473]: pgmap v7129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:48:24.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:48:24.425+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T14:48:24.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:48:24.425+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T14:48:24.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:48:24.426+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 95 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T14:48:24.427 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:48:24.426+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T14:48:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:24 smithi134 ceph-mon[31966]: pgmap v7130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:48:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12045.316138 secs 2024-07-26T14:48:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:24 smithi189 ceph-mon[36790]: pgmap v7130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:48:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12045.316138 secs 2024-07-26T14:48:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:24 smithi193 ceph-mon[37473]: pgmap v7130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:48:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12045.316138 secs 2024-07-26T14:48:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:26 smithi134 ceph-mon[31966]: pgmap v7131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12064.740367 secs 2024-07-26T14:48:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:26 smithi189 ceph-mon[36790]: pgmap v7131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12064.740367 secs 2024-07-26T14:48:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:26 smithi193 ceph-mon[37473]: pgmap v7131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12064.740367 secs 2024-07-26T14:48:27.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:48:27.543+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:48:27.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:48:27.543+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:48:27.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:48:27.543+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:29.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:28 smithi134 ceph-mon[31966]: pgmap v7132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:28 smithi189 ceph-mon[36790]: pgmap v7132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:28 smithi193 ceph-mon[37473]: pgmap v7132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12050.316275 secs 2024-07-26T14:48:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12050.316275 secs 2024-07-26T14:48:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12050.316275 secs 2024-07-26T14:48:31.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:30 smithi134 ceph-mon[31966]: pgmap v7133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:48:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:30 smithi189 ceph-mon[36790]: pgmap v7133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:48:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:30 smithi193 ceph-mon[37473]: pgmap v7133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:48:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12069.740490 secs 2024-07-26T14:48:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12069.740490 secs 2024-07-26T14:48:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12069.740490 secs 2024-07-26T14:48:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:32 smithi134 ceph-mon[31966]: pgmap v7134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:32 smithi189 ceph-mon[36790]: pgmap v7134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:32 smithi193 ceph-mon[37473]: pgmap v7134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:34.959 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:48:34.959+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:48:34.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:48:34.959+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:48:34.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:48:34.959+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:48:34.960 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:48:34.959+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:48:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:34 smithi134 ceph-mon[31966]: pgmap v7135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12055.316351 secs 2024-07-26T14:48:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:34 smithi189 ceph-mon[36790]: pgmap v7135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12055.316351 secs 2024-07-26T14:48:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:34 smithi193 ceph-mon[37473]: pgmap v7135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12055.316351 secs 2024-07-26T14:48:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:36 smithi134 ceph-mon[31966]: pgmap v7136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12074.740606 secs 2024-07-26T14:48:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:36 smithi189 ceph-mon[36790]: pgmap v7136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12074.740606 secs 2024-07-26T14:48:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:36 smithi193 ceph-mon[37473]: pgmap v7136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12074.740606 secs 2024-07-26T14:48:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:48:38.057+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:48:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:48:38.057+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:38.058 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:48:38.057+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:48:39.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:38 smithi134 ceph-mon[31966]: pgmap v7137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:39.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:38 smithi189 ceph-mon[36790]: pgmap v7137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:38 smithi193 ceph-mon[37473]: pgmap v7137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12060.316424 secs 2024-07-26T14:48:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12060.316424 secs 2024-07-26T14:48:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12060.316424 secs 2024-07-26T14:48:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:40 smithi189 ceph-mon[36790]: pgmap v7138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:48:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:40 smithi193 ceph-mon[37473]: pgmap v7138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:48:41.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:40 smithi134 ceph-mon[31966]: pgmap v7138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:48:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12079.740743 secs 2024-07-26T14:48:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12079.740743 secs 2024-07-26T14:48:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12079.740743 secs 2024-07-26T14:48:42.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:48:42.544+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:48:42.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:48:42.544+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:42.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:48:42.544+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:48:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:42 smithi189 ceph-mon[36790]: pgmap v7139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:42 smithi193 ceph-mon[37473]: pgmap v7139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:43.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:42 smithi134 ceph-mon[31966]: pgmap v7139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:44 smithi193 ceph-mon[37473]: pgmap v7140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12065.316561 secs 2024-07-26T14:48:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:44 smithi134 ceph-mon[31966]: pgmap v7140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12065.316561 secs 2024-07-26T14:48:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:44 smithi189 ceph-mon[36790]: pgmap v7140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:48:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12065.316561 secs 2024-07-26T14:48:47.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:46 smithi134 ceph-mon[31966]: pgmap v7141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:47.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12084.740866 secs 2024-07-26T14:48:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:46 smithi193 ceph-mon[37473]: pgmap v7141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12084.740866 secs 2024-07-26T14:48:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:46 smithi189 ceph-mon[36790]: pgmap v7141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:48:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12084.740866 secs 2024-07-26T14:48:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:48:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:48:48.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:48:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:49 smithi193 ceph-mon[37473]: pgmap v7142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:48:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:48:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:48:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12070.316676 secs 2024-07-26T14:48:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:49 smithi134 ceph-mon[31966]: pgmap v7142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:48:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:48:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:48:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12070.316676 secs 2024-07-26T14:48:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:49 smithi189 ceph-mon[36790]: pgmap v7142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:48:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:48:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:48:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12070.316676 secs 2024-07-26T14:48:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:48:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:48:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:50 smithi134 ceph-mon[31966]: pgmap v7143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:48:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:50 smithi134 ceph-mon[31966]: pgmap v7144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:48:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:48:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:48:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:48:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:50 smithi189 ceph-mon[36790]: pgmap v7143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:48:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:50 smithi189 ceph-mon[36790]: pgmap v7144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:48:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:48:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:48:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:48:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:50 smithi193 ceph-mon[37473]: pgmap v7143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:48:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:50 smithi193 ceph-mon[37473]: pgmap v7144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:48:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:48:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12089.740993 secs 2024-07-26T14:48:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:52 smithi134 ceph-mon[31966]: pgmap v7145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:48:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12089.740993 secs 2024-07-26T14:48:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:52 smithi189 ceph-mon[36790]: pgmap v7145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:48:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12089.740993 secs 2024-07-26T14:48:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:52 smithi193 ceph-mon[37473]: pgmap v7145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:48:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:48:53.072+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:48:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:48:53.073+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:48:53.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:48:53.073+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:54 smithi134 ceph-mon[31966]: pgmap v7146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:48:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12075.316802 secs 2024-07-26T14:48:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:54 smithi189 ceph-mon[36790]: pgmap v7146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:48:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12075.316802 secs 2024-07-26T14:48:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:54 smithi193 ceph-mon[37473]: pgmap v7146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:48:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12075.316802 secs 2024-07-26T14:48:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:56 smithi134 ceph-mon[31966]: pgmap v7147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:48:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:56 smithi189 ceph-mon[36790]: pgmap v7147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:48:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:56 smithi193 ceph-mon[37473]: pgmap v7147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T14:48:57.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:48:57.560+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:48:57.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:48:57.560+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:48:57.561 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:48:57.560+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:48:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12094.741148 secs 2024-07-26T14:48:57.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12094.741148 secs 2024-07-26T14:48:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12094.741148 secs 2024-07-26T14:48:58.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:58 smithi134 ceph-mon[31966]: pgmap v7148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T14:48:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:58 smithi189 ceph-mon[36790]: pgmap v7148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T14:48:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:58 smithi193 ceph-mon[37473]: pgmap v7148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T14:48:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:48:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12080.316924 secs 2024-07-26T14:48:59.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:48:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12080.316924 secs 2024-07-26T14:48:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:48:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12080.316924 secs 2024-07-26T14:49:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:00 smithi134 ceph-mon[31966]: pgmap v7149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:49:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:00 smithi189 ceph-mon[36790]: pgmap v7149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:49:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:00 smithi193 ceph-mon[37473]: pgmap v7149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:49:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12099.741248 secs 2024-07-26T14:49:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:02 smithi134 ceph-mon[31966]: pgmap v7150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12099.741248 secs 2024-07-26T14:49:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:02 smithi189 ceph-mon[36790]: pgmap v7150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12099.741248 secs 2024-07-26T14:49:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:02 smithi193 ceph-mon[37473]: pgmap v7150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:03 smithi193 ceph-mon[37473]: pgmap v7151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:03.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:03 smithi134 ceph-mon[31966]: pgmap v7151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:03 smithi189 ceph-mon[36790]: pgmap v7151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12085.316990 secs 2024-07-26T14:49:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12085.316990 secs 2024-07-26T14:49:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12085.316990 secs 2024-07-26T14:49:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:05 smithi193 ceph-mon[37473]: pgmap v7152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:05.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:05 smithi134 ceph-mon[31966]: pgmap v7152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:05 smithi189 ceph-mon[36790]: pgmap v7152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:07.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12104.741379 secs 2024-07-26T14:49:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12104.741379 secs 2024-07-26T14:49:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12104.741379 secs 2024-07-26T14:49:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:07 smithi193 ceph-mon[37473]: pgmap v7153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:49:08.088+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:49:08.090 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:49:08.088+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:49:08.090 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:49:08.088+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:08.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:07 smithi134 ceph-mon[31966]: pgmap v7153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:07 smithi189 ceph-mon[36790]: pgmap v7153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12090.317109 secs 2024-07-26T14:49:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12090.317109 secs 2024-07-26T14:49:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12090.317109 secs 2024-07-26T14:49:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:10 smithi193 ceph-mon[37473]: pgmap v7154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:10 smithi134 ceph-mon[31966]: pgmap v7154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:10 smithi189 ceph-mon[36790]: pgmap v7154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:12.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:49:12.575+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:49:12.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:49:12.575+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:12.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:49:12.575+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:49:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12109.741480 secs 2024-07-26T14:49:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:12 smithi134 ceph-mon[31966]: pgmap v7155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12109.741480 secs 2024-07-26T14:49:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:12 smithi189 ceph-mon[36790]: pgmap v7155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12109.741480 secs 2024-07-26T14:49:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:12 smithi193 ceph-mon[37473]: pgmap v7155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:14 smithi134 ceph-mon[31966]: pgmap v7156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12095.317173 secs 2024-07-26T14:49:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:14 smithi189 ceph-mon[36790]: pgmap v7156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12095.317173 secs 2024-07-26T14:49:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:14 smithi193 ceph-mon[37473]: pgmap v7156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12095.317173 secs 2024-07-26T14:49:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:16 smithi134 ceph-mon[31966]: pgmap v7157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:16.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:16 smithi189 ceph-mon[36790]: pgmap v7157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:16 smithi193 ceph-mon[37473]: pgmap v7157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12114.741545 secs 2024-07-26T14:49:17.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12114.741545 secs 2024-07-26T14:49:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12114.741545 secs 2024-07-26T14:49:18.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:18 smithi134 ceph-mon[31966]: pgmap v7158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:18 smithi189 ceph-mon[36790]: pgmap v7158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:18 smithi193 ceph-mon[37473]: pgmap v7158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12100.317295 secs 2024-07-26T14:49:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12100.317295 secs 2024-07-26T14:49:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12100.317295 secs 2024-07-26T14:49:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:20 smithi189 ceph-mon[36790]: pgmap v7159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:20 smithi193 ceph-mon[37473]: pgmap v7159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:20 smithi134 ceph-mon[31966]: pgmap v7159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:21 smithi189 ceph-mon[36790]: pgmap v7160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:21 smithi193 ceph-mon[37473]: pgmap v7160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:21 smithi134 ceph-mon[31966]: pgmap v7160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12119.741657 secs 2024-07-26T14:49:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12119.741657 secs 2024-07-26T14:49:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12119.741657 secs 2024-07-26T14:49:23.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:49:23.103+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:49:23.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:49:23.103+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:23.104 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:49:23.103+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:49:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:23 smithi193 ceph-mon[37473]: pgmap v7161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:23.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:23 smithi134 ceph-mon[31966]: pgmap v7161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:23.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:23 smithi189 ceph-mon[36790]: pgmap v7161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12105.317379 secs 2024-07-26T14:49:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12105.317379 secs 2024-07-26T14:49:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12105.317379 secs 2024-07-26T14:49:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:25 smithi193 ceph-mon[37473]: pgmap v7162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:25.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:25 smithi134 ceph-mon[31966]: pgmap v7162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:25.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:25 smithi189 ceph-mon[36790]: pgmap v7162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12124.741769 secs 2024-07-26T14:49:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12124.741769 secs 2024-07-26T14:49:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12124.741769 secs 2024-07-26T14:49:27.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:49:27.591+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:49:27.592 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:49:27.591+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:49:27.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:49:27.591+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:27 smithi134 ceph-mon[31966]: pgmap v7163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:27 smithi193 ceph-mon[37473]: pgmap v7163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:27 smithi189 ceph-mon[36790]: pgmap v7163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12110.317529 secs 2024-07-26T14:49:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12110.317529 secs 2024-07-26T14:49:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12110.317529 secs 2024-07-26T14:49:30.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:30 smithi134 ceph-mon[31966]: pgmap v7164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:30 smithi193 ceph-mon[37473]: pgmap v7164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:30 smithi189 ceph-mon[36790]: pgmap v7164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12129.741860 secs 2024-07-26T14:49:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:32 smithi134 ceph-mon[31966]: pgmap v7165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12129.741860 secs 2024-07-26T14:49:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:32 smithi189 ceph-mon[36790]: pgmap v7165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12129.741860 secs 2024-07-26T14:49:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:32 smithi193 ceph-mon[37473]: pgmap v7165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:34 smithi134 ceph-mon[31966]: pgmap v7166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12115.317647 secs 2024-07-26T14:49:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:34 smithi189 ceph-mon[36790]: pgmap v7166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12115.317647 secs 2024-07-26T14:49:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:34 smithi193 ceph-mon[37473]: pgmap v7166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12115.317647 secs 2024-07-26T14:49:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:36 smithi134 ceph-mon[31966]: pgmap v7167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:36 smithi189 ceph-mon[36790]: pgmap v7167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:36 smithi193 ceph-mon[37473]: pgmap v7167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12134.741988 secs 2024-07-26T14:49:37.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12134.741988 secs 2024-07-26T14:49:37.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12134.741988 secs 2024-07-26T14:49:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:49:38.112+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:49:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:49:38.112+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:49:38.112+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:49:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:38 smithi189 ceph-mon[36790]: pgmap v7168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:38 smithi193 ceph-mon[37473]: pgmap v7168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:38 smithi134 ceph-mon[31966]: pgmap v7168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12120.317773 secs 2024-07-26T14:49:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12120.317773 secs 2024-07-26T14:49:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12120.317773 secs 2024-07-26T14:49:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:40 smithi189 ceph-mon[36790]: pgmap v7169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:40 smithi193 ceph-mon[37473]: pgmap v7169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:40 smithi134 ceph-mon[31966]: pgmap v7169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:41 smithi193 ceph-mon[37473]: pgmap v7170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:41.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:41 smithi134 ceph-mon[31966]: pgmap v7170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:41 smithi189 ceph-mon[36790]: pgmap v7170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:42.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:49:42.599+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:49:42.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:49:42.599+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:49:42.600 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:49:42.599+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12139.742111 secs 2024-07-26T14:49:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12139.742111 secs 2024-07-26T14:49:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12139.742111 secs 2024-07-26T14:49:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:43 smithi193 ceph-mon[37473]: pgmap v7171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:43.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:43 smithi134 ceph-mon[31966]: pgmap v7171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:43 smithi189 ceph-mon[36790]: pgmap v7171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12125.317899 secs 2024-07-26T14:49:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12125.317899 secs 2024-07-26T14:49:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12125.317899 secs 2024-07-26T14:49:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:45 smithi134 ceph-mon[31966]: pgmap v7172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:45 smithi193 ceph-mon[37473]: pgmap v7172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:45 smithi189 ceph-mon[36790]: pgmap v7172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12144.742256 secs 2024-07-26T14:49:47.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12144.742256 secs 2024-07-26T14:49:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12144.742256 secs 2024-07-26T14:49:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:47 smithi193 ceph-mon[37473]: pgmap v7173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:47 smithi134 ceph-mon[31966]: pgmap v7173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:47 smithi189 ceph-mon[36790]: pgmap v7173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:49:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12130.318035 secs 2024-07-26T14:49:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12130.318035 secs 2024-07-26T14:49:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12130.318035 secs 2024-07-26T14:49:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:50 smithi189 ceph-mon[36790]: pgmap v7174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:49:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:50 smithi193 ceph-mon[37473]: pgmap v7174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:49:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:50.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:50 smithi134 ceph-mon[31966]: pgmap v7174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:49:50.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:49:50.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:50.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:50.645 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:50.645 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12149.742362 secs 2024-07-26T14:49:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:52 smithi134 ceph-mon[31966]: pgmap v7175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12149.742362 secs 2024-07-26T14:49:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:52 smithi189 ceph-mon[36790]: pgmap v7175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12149.742362 secs 2024-07-26T14:49:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:52 smithi193 ceph-mon[37473]: pgmap v7175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:49:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:53.128 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:49:53.127+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:49:53.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:49:53.127+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:53.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:49:53.127+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:49:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:54 smithi134 ceph-mon[31966]: pgmap v7176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12135.318165 secs 2024-07-26T14:49:54.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:54 smithi189 ceph-mon[36790]: pgmap v7176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12135.318165 secs 2024-07-26T14:49:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:54 smithi193 ceph-mon[37473]: pgmap v7176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:49:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12135.318165 secs 2024-07-26T14:49:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:49:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:49:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:49:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:49:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:49:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:49:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:49:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:56 smithi134 ceph-mon[31966]: pgmap v7177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:49:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:56 smithi134 ceph-mon[31966]: pgmap v7178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:49:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:56 smithi189 ceph-mon[36790]: pgmap v7177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:49:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:56 smithi189 ceph-mon[36790]: pgmap v7178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:49:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:56 smithi193 ceph-mon[37473]: pgmap v7177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:49:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:56 smithi193 ceph-mon[37473]: pgmap v7178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:49:57.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:49:57.614+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:49:57.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:49:57.614+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:49:57.616 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:49:57.614+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:49:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12154.742476 secs 2024-07-26T14:49:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12154.742476 secs 2024-07-26T14:49:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12154.742476 secs 2024-07-26T14:49:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:58 smithi134 ceph-mon[31966]: pgmap v7179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:49:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:58 smithi189 ceph-mon[36790]: pgmap v7179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:49:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:58 smithi193 ceph-mon[37473]: pgmap v7179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:49:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:49:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12140.318293 secs 2024-07-26T14:49:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:49:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12140.318293 secs 2024-07-26T14:49:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:49:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12140.318293 secs 2024-07-26T14:50:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:00 smithi189 ceph-mon[36790]: pgmap v7180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:50:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:50:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:00 smithi193 ceph-mon[37473]: pgmap v7180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:50:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:50:00.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:00 smithi134 ceph-mon[31966]: pgmap v7180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:50:00.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T14:50:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:01 smithi134 ceph-mon[31966]: pgmap v7181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:50:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:01 smithi193 ceph-mon[37473]: pgmap v7181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:50:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:01 smithi189 ceph-mon[36790]: pgmap v7181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:50:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12159.742605 secs 2024-07-26T14:50:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12159.742605 secs 2024-07-26T14:50:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12159.742605 secs 2024-07-26T14:50:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:03 smithi193 ceph-mon[37473]: pgmap v7182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:50:03.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:03 smithi134 ceph-mon[31966]: pgmap v7182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:50:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:03 smithi189 ceph-mon[36790]: pgmap v7182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:50:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12145.318379 secs 2024-07-26T14:50:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12145.318379 secs 2024-07-26T14:50:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12145.318379 secs 2024-07-26T14:50:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:05 smithi193 ceph-mon[37473]: pgmap v7183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:50:05.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:05 smithi134 ceph-mon[31966]: pgmap v7183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:50:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:05 smithi189 ceph-mon[36790]: pgmap v7183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:50:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12164.742731 secs 2024-07-26T14:50:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:07 smithi134 ceph-mon[31966]: pgmap v7184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12164.742731 secs 2024-07-26T14:50:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:07 smithi189 ceph-mon[36790]: pgmap v7184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12164.742731 secs 2024-07-26T14:50:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:07 smithi193 ceph-mon[37473]: pgmap v7184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:50:08.143+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:50:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:50:08.143+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:08.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:50:08.143+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:50:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:09 smithi134 ceph-mon[31966]: pgmap v7185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12150.318509 secs 2024-07-26T14:50:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:09 smithi189 ceph-mon[36790]: pgmap v7185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12150.318509 secs 2024-07-26T14:50:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:09 smithi193 ceph-mon[37473]: pgmap v7185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12150.318509 secs 2024-07-26T14:50:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:11 smithi134 ceph-mon[31966]: pgmap v7186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:11 smithi189 ceph-mon[36790]: pgmap v7186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:11 smithi193 ceph-mon[37473]: pgmap v7186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:12.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:50:12.630+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:50:12.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:50:12.630+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:50:12.632 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:50:12.630+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12169.742850 secs 2024-07-26T14:50:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12169.742850 secs 2024-07-26T14:50:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12169.742850 secs 2024-07-26T14:50:13.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:13 smithi134 ceph-mon[31966]: pgmap v7187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:13 smithi189 ceph-mon[36790]: pgmap v7187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:13 smithi193 ceph-mon[37473]: pgmap v7187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12155.318667 secs 2024-07-26T14:50:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12155.318667 secs 2024-07-26T14:50:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12155.318667 secs 2024-07-26T14:50:15.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:15 smithi134 ceph-mon[31966]: pgmap v7188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:15 smithi189 ceph-mon[36790]: pgmap v7188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:15 smithi193 ceph-mon[37473]: pgmap v7188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:50:17.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12174.742966 secs 2024-07-26T14:50:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:17 smithi134 ceph-mon[31966]: pgmap v7189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12174.742966 secs 2024-07-26T14:50:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:17 smithi189 ceph-mon[36790]: pgmap v7189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12174.742966 secs 2024-07-26T14:50:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:17 smithi193 ceph-mon[37473]: pgmap v7189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:19 smithi134 ceph-mon[31966]: pgmap v7190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12160.318779 secs 2024-07-26T14:50:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:19 smithi189 ceph-mon[36790]: pgmap v7190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12160.318779 secs 2024-07-26T14:50:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:19 smithi193 ceph-mon[37473]: pgmap v7190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12160.318779 secs 2024-07-26T14:50:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:21 smithi134 ceph-mon[31966]: pgmap v7191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:21 smithi189 ceph-mon[36790]: pgmap v7191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:21 smithi193 ceph-mon[37473]: pgmap v7191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12179.743092 secs 2024-07-26T14:50:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12179.743092 secs 2024-07-26T14:50:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12179.743092 secs 2024-07-26T14:50:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:50:23.156+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:50:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:50:23.156+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:50:23.156+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:50:23.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:23 smithi134 ceph-mon[31966]: pgmap v7192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:23.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:23 smithi189 ceph-mon[36790]: pgmap v7192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:23 smithi193 ceph-mon[37473]: pgmap v7192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12165.318899 secs 2024-07-26T14:50:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12165.318899 secs 2024-07-26T14:50:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12165.318899 secs 2024-07-26T14:50:25.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:25 smithi134 ceph-mon[31966]: pgmap v7193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:50:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:25 smithi189 ceph-mon[36790]: pgmap v7193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:50:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:25 smithi193 ceph-mon[37473]: pgmap v7193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:50:27.636 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:50:27.635+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:50:27.636 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:50:27.635+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:50:27.636 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:50:27.635+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12184.743214 secs 2024-07-26T14:50:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:27 smithi134 ceph-mon[31966]: pgmap v7194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12184.743214 secs 2024-07-26T14:50:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:27 smithi189 ceph-mon[36790]: pgmap v7194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12184.743214 secs 2024-07-26T14:50:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:27 smithi193 ceph-mon[37473]: pgmap v7194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:29 smithi134 ceph-mon[31966]: pgmap v7195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12170.319032 secs 2024-07-26T14:50:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:29 smithi189 ceph-mon[36790]: pgmap v7195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12170.319032 secs 2024-07-26T14:50:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:29 smithi193 ceph-mon[37473]: pgmap v7195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12170.319032 secs 2024-07-26T14:50:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:31 smithi134 ceph-mon[31966]: pgmap v7196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:31 smithi189 ceph-mon[36790]: pgmap v7196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:31 smithi193 ceph-mon[37473]: pgmap v7196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12189.743325 secs 2024-07-26T14:50:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12189.743325 secs 2024-07-26T14:50:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12189.743325 secs 2024-07-26T14:50:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:33 smithi189 ceph-mon[36790]: pgmap v7197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:33 smithi193 ceph-mon[37473]: pgmap v7197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:33 smithi134 ceph-mon[31966]: pgmap v7197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12175.319146 secs 2024-07-26T14:50:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12175.319146 secs 2024-07-26T14:50:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12175.319146 secs 2024-07-26T14:50:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:35 smithi189 ceph-mon[36790]: pgmap v7198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:50:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:35 smithi193 ceph-mon[37473]: pgmap v7198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:50:36.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:35 smithi134 ceph-mon[31966]: pgmap v7198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:50:37.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12194.743353 secs 2024-07-26T14:50:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12194.743353 secs 2024-07-26T14:50:37.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12194.743353 secs 2024-07-26T14:50:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:37 smithi193 ceph-mon[37473]: pgmap v7199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:38.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:37 smithi189 ceph-mon[36790]: pgmap v7199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:50:38.170+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:50:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:50:38.170+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:50:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:50:38.170+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:37 smithi134 ceph-mon[31966]: pgmap v7199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:39 smithi193 ceph-mon[37473]: pgmap v7200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12180.319265 secs 2024-07-26T14:50:40.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:39 smithi134 ceph-mon[31966]: pgmap v7200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:40.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12180.319265 secs 2024-07-26T14:50:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:39 smithi189 ceph-mon[36790]: pgmap v7200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12180.319265 secs 2024-07-26T14:50:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:41 smithi193 ceph-mon[37473]: pgmap v7201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12199.743472 secs 2024-07-26T14:50:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:41 smithi134 ceph-mon[31966]: pgmap v7201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12199.743472 secs 2024-07-26T14:50:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:41 smithi189 ceph-mon[36790]: pgmap v7201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12199.743472 secs 2024-07-26T14:50:42.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:50:42.650+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:50:42.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:50:42.650+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:50:42.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:50:42.650+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:43 smithi193 ceph-mon[37473]: pgmap v7202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:44.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:43 smithi134 ceph-mon[31966]: pgmap v7202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:43 smithi189 ceph-mon[36790]: pgmap v7202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12185.319331 secs 2024-07-26T14:50:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12185.319331 secs 2024-07-26T14:50:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12185.319331 secs 2024-07-26T14:50:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:45 smithi193 ceph-mon[37473]: pgmap v7203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:50:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:45 smithi134 ceph-mon[31966]: pgmap v7203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:50:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:45 smithi189 ceph-mon[36790]: pgmap v7203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:50:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12204.743587 secs 2024-07-26T14:50:47.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12204.743587 secs 2024-07-26T14:50:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12204.743587 secs 2024-07-26T14:50:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:47 smithi193 ceph-mon[37473]: pgmap v7204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:47 smithi134 ceph-mon[31966]: pgmap v7204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:47 smithi189 ceph-mon[36790]: pgmap v7204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12190.319445 secs 2024-07-26T14:50:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:49 smithi193 ceph-mon[37473]: pgmap v7205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:50.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12190.319445 secs 2024-07-26T14:50:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:49 smithi134 ceph-mon[31966]: pgmap v7205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12190.319445 secs 2024-07-26T14:50:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:49 smithi189 ceph-mon[36790]: pgmap v7205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:51 smithi193 ceph-mon[37473]: pgmap v7206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12209.743650 secs 2024-07-26T14:50:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:51 smithi134 ceph-mon[31966]: pgmap v7206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12209.743650 secs 2024-07-26T14:50:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:51 smithi189 ceph-mon[36790]: pgmap v7206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:50:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12209.743650 secs 2024-07-26T14:50:53.187 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:50:53.185+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:50:53.187 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:50:53.186+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:50:53.187 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:50:53.186+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:53 smithi193 ceph-mon[37473]: pgmap v7207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:53 smithi134 ceph-mon[31966]: pgmap v7207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:53 smithi189 ceph-mon[36790]: pgmap v7207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:50:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12195.319573 secs 2024-07-26T14:50:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:50:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12195.319573 secs 2024-07-26T14:50:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:50:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12195.319573 secs 2024-07-26T14:50:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:50:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:55 smithi134 ceph-mon[31966]: pgmap v7208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:50:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:50:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:50:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:50:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:55 smithi189 ceph-mon[36790]: pgmap v7208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:50:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:50:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:50:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:50:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:55 smithi193 ceph-mon[37473]: pgmap v7208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:50:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:50:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:50:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:50:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:56 smithi134 ceph-mon[31966]: pgmap v7209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:50:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:56 smithi134 ceph-mon[31966]: pgmap v7210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:50:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12214.743760 secs 2024-07-26T14:50:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:56 smithi189 ceph-mon[36790]: pgmap v7209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:50:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:56 smithi189 ceph-mon[36790]: pgmap v7210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:50:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12214.743760 secs 2024-07-26T14:50:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:56 smithi193 ceph-mon[37473]: pgmap v7209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:50:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:56 smithi193 ceph-mon[37473]: pgmap v7210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:50:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12214.743760 secs 2024-07-26T14:50:57.665 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:50:57.664+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:50:57.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:50:57.664+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:50:57.666 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:50:57.664+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:50:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:58 smithi134 ceph-mon[31966]: pgmap v7211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:50:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:58 smithi189 ceph-mon[36790]: pgmap v7211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:50:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:58 smithi193 ceph-mon[37473]: pgmap v7211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:51:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:50:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12200.319716 secs 2024-07-26T14:51:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:50:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12200.319716 secs 2024-07-26T14:51:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:50:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12200.319716 secs 2024-07-26T14:51:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:00 smithi134 ceph-mon[31966]: pgmap v7212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:51:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:00 smithi189 ceph-mon[36790]: pgmap v7212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:51:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:00 smithi193 ceph-mon[37473]: pgmap v7212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:51:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12219.743876 secs 2024-07-26T14:51:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12219.743876 secs 2024-07-26T14:51:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12219.743876 secs 2024-07-26T14:51:03.176 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:02 smithi134 ceph-mon[31966]: pgmap v7213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:51:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:02 smithi189 ceph-mon[36790]: pgmap v7213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:51:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:02 smithi193 ceph-mon[37473]: pgmap v7213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:51:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12205.319834 secs 2024-07-26T14:51:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:04 smithi134 ceph-mon[31966]: pgmap v7214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:51:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12205.319834 secs 2024-07-26T14:51:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:04 smithi189 ceph-mon[36790]: pgmap v7214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:51:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12205.319834 secs 2024-07-26T14:51:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:04 smithi193 ceph-mon[37473]: pgmap v7214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:51:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:06 smithi134 ceph-mon[31966]: pgmap v7215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:51:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12224.743995 secs 2024-07-26T14:51:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:06 smithi189 ceph-mon[36790]: pgmap v7215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:51:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12224.743995 secs 2024-07-26T14:51:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:06 smithi193 ceph-mon[37473]: pgmap v7215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:51:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12224.743995 secs 2024-07-26T14:51:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:51:08.192+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:51:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:51:08.192+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:08.193 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:51:08.192+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:51:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:08 smithi134 ceph-mon[31966]: pgmap v7216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:08 smithi189 ceph-mon[36790]: pgmap v7216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:09.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:08 smithi193 ceph-mon[37473]: pgmap v7216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:10.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12210.319961 secs 2024-07-26T14:51:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12210.319961 secs 2024-07-26T14:51:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12210.319961 secs 2024-07-26T14:51:11.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:10 smithi134 ceph-mon[31966]: pgmap v7217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:51:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:10 smithi189 ceph-mon[36790]: pgmap v7217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:51:11.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:10 smithi193 ceph-mon[37473]: pgmap v7217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:51:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12229.744142 secs 2024-07-26T14:51:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12229.744142 secs 2024-07-26T14:51:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12229.744142 secs 2024-07-26T14:51:12.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:51:12.669+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:51:12.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:51:12.669+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:51:12.670 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:51:12.669+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:12 smithi134 ceph-mon[31966]: pgmap v7218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:12 smithi189 ceph-mon[36790]: pgmap v7218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:12 smithi193 ceph-mon[37473]: pgmap v7218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12215.319959 secs 2024-07-26T14:51:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:14 smithi134 ceph-mon[31966]: pgmap v7219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12215.319959 secs 2024-07-26T14:51:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:14 smithi189 ceph-mon[36790]: pgmap v7219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12215.319959 secs 2024-07-26T14:51:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:14 smithi193 ceph-mon[37473]: pgmap v7219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:17.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:16 smithi134 ceph-mon[31966]: pgmap v7220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12234.744292 secs 2024-07-26T14:51:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:16 smithi189 ceph-mon[36790]: pgmap v7220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12234.744292 secs 2024-07-26T14:51:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:16 smithi193 ceph-mon[37473]: pgmap v7220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12234.744292 secs 2024-07-26T14:51:19.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:18 smithi134 ceph-mon[31966]: pgmap v7221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:18 smithi189 ceph-mon[36790]: pgmap v7221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:18 smithi193 ceph-mon[37473]: pgmap v7221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12220.320079 secs 2024-07-26T14:51:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12220.320079 secs 2024-07-26T14:51:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12220.320079 secs 2024-07-26T14:51:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:20 smithi189 ceph-mon[36790]: pgmap v7222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:51:21.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:20 smithi193 ceph-mon[37473]: pgmap v7222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:51:21.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:20 smithi134 ceph-mon[31966]: pgmap v7222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:51:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12239.744432 secs 2024-07-26T14:51:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12239.744432 secs 2024-07-26T14:51:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12239.744432 secs 2024-07-26T14:51:23.208 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:51:23.207+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:51:23.209 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:51:23.207+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:23.209 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:51:23.207+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:51:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:22 smithi193 ceph-mon[37473]: pgmap v7223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:23.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:22 smithi134 ceph-mon[31966]: pgmap v7223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:22 smithi189 ceph-mon[36790]: pgmap v7223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:24.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:51:24.457+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T14:51:24.458 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:51:24.457+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T14:51:24.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:51:24.458+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 96 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T14:51:24.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:51:24.458+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T14:51:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12225.320146 secs 2024-07-26T14:51:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:24 smithi193 ceph-mon[37473]: pgmap v7224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12225.320146 secs 2024-07-26T14:51:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:24 smithi134 ceph-mon[31966]: pgmap v7224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12225.320146 secs 2024-07-26T14:51:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:24 smithi189 ceph-mon[36790]: pgmap v7224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:26 smithi193 ceph-mon[37473]: pgmap v7225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12244.744545 secs 2024-07-26T14:51:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:26 smithi134 ceph-mon[31966]: pgmap v7225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12244.744545 secs 2024-07-26T14:51:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:26 smithi189 ceph-mon[36790]: pgmap v7225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12244.744545 secs 2024-07-26T14:51:27.685 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:51:27.684+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:51:27.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:51:27.684+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:51:27.686 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:51:27.684+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:28 smithi193 ceph-mon[37473]: pgmap v7226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:28 smithi134 ceph-mon[31966]: pgmap v7226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:28 smithi189 ceph-mon[36790]: pgmap v7226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12230.320211 secs 2024-07-26T14:51:30.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12230.320211 secs 2024-07-26T14:51:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12230.320211 secs 2024-07-26T14:51:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:30 smithi193 ceph-mon[37473]: pgmap v7227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:51:31.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:30 smithi134 ceph-mon[31966]: pgmap v7227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:51:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:30 smithi189 ceph-mon[36790]: pgmap v7227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:51:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12249.744660 secs 2024-07-26T14:51:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12249.744660 secs 2024-07-26T14:51:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12249.744660 secs 2024-07-26T14:51:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:32 smithi193 ceph-mon[37473]: pgmap v7228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:32 smithi134 ceph-mon[31966]: pgmap v7228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:32 smithi189 ceph-mon[36790]: pgmap v7228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:51:34.990+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:51:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:51:34.990+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:51:34.991 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:51:34.990+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:51:34.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:51:34.991+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:51:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12235.320330 secs 2024-07-26T14:51:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:34 smithi193 ceph-mon[37473]: pgmap v7229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12235.320330 secs 2024-07-26T14:51:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:34 smithi134 ceph-mon[31966]: pgmap v7229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12235.320330 secs 2024-07-26T14:51:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:34 smithi189 ceph-mon[36790]: pgmap v7229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:37 smithi193 ceph-mon[37473]: pgmap v7230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12254.744774 secs 2024-07-26T14:51:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:37 smithi134 ceph-mon[31966]: pgmap v7230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12254.744774 secs 2024-07-26T14:51:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:37 smithi189 ceph-mon[36790]: pgmap v7230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12254.744774 secs 2024-07-26T14:51:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:51:38.212+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:51:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:51:38.212+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:51:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:51:38.212+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:39 smithi193 ceph-mon[37473]: pgmap v7231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:39 smithi134 ceph-mon[31966]: pgmap v7231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:39 smithi189 ceph-mon[36790]: pgmap v7231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12240.320453 secs 2024-07-26T14:51:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12240.320453 secs 2024-07-26T14:51:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12240.320453 secs 2024-07-26T14:51:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:41 smithi193 ceph-mon[37473]: pgmap v7232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:51:41.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:41 smithi134 ceph-mon[31966]: pgmap v7232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:51:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:41 smithi189 ceph-mon[36790]: pgmap v7232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:51:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12259.744905 secs 2024-07-26T14:51:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12259.744905 secs 2024-07-26T14:51:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12259.744905 secs 2024-07-26T14:51:42.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:51:42.689+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:51:42.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:51:42.689+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:51:42.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:51:42.689+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:43.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:43 smithi134 ceph-mon[31966]: pgmap v7233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:43.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:43 smithi189 ceph-mon[36790]: pgmap v7233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:43 smithi193 ceph-mon[37473]: pgmap v7233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12245.320594 secs 2024-07-26T14:51:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:45 smithi134 ceph-mon[31966]: pgmap v7234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12245.320594 secs 2024-07-26T14:51:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:45 smithi189 ceph-mon[36790]: pgmap v7234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12245.320594 secs 2024-07-26T14:51:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:45 smithi193 ceph-mon[37473]: pgmap v7234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:47 smithi134 ceph-mon[31966]: pgmap v7235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12264.745029 secs 2024-07-26T14:51:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:47 smithi189 ceph-mon[36790]: pgmap v7235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12264.745029 secs 2024-07-26T14:51:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:47 smithi193 ceph-mon[37473]: pgmap v7235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12264.745029 secs 2024-07-26T14:51:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:49 smithi134 ceph-mon[31966]: pgmap v7236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:49 smithi189 ceph-mon[36790]: pgmap v7236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:49 smithi193 ceph-mon[37473]: pgmap v7236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:50.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12250.320712 secs 2024-07-26T14:51:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12250.320712 secs 2024-07-26T14:51:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12250.320712 secs 2024-07-26T14:51:51.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:51 smithi134 ceph-mon[31966]: pgmap v7237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:51:51.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:51 smithi189 ceph-mon[36790]: pgmap v7237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:51:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:51 smithi193 ceph-mon[37473]: pgmap v7237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:51:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12269.745168 secs 2024-07-26T14:51:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12269.745168 secs 2024-07-26T14:51:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12269.745168 secs 2024-07-26T14:51:53.226 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:51:53.225+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:51:53.227 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:51:53.225+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:51:53.227 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:51:53.225+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:53.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:53 smithi134 ceph-mon[31966]: pgmap v7238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:53 smithi189 ceph-mon[36790]: pgmap v7238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:53.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:53 smithi193 ceph-mon[37473]: pgmap v7238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:55.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12255.320830 secs 2024-07-26T14:51:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:55 smithi134 ceph-mon[31966]: pgmap v7239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12255.320830 secs 2024-07-26T14:51:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:55 smithi189 ceph-mon[36790]: pgmap v7239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12255.320830 secs 2024-07-26T14:51:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:55 smithi193 ceph-mon[37473]: pgmap v7239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:51:56.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:51:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:51:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:51:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:57 smithi134 ceph-mon[31966]: pgmap v7240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12274.745241 secs 2024-07-26T14:51:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:51:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:51:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:51:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:57 smithi189 ceph-mon[36790]: pgmap v7240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12274.745241 secs 2024-07-26T14:51:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:51:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:51:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:51:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:57 smithi193 ceph-mon[37473]: pgmap v7240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:51:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12274.745241 secs 2024-07-26T14:51:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:51:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:51:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:51:57.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:51:57.694+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:51:57.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:51:57.694+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:51:57.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:51:57.694+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:51:58.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:58 smithi134 ceph-mon[31966]: pgmap v7241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:51:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:51:58 smithi134 ceph-mon[31966]: pgmap v7242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:51:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:58 smithi189 ceph-mon[36790]: pgmap v7241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:51:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:51:58 smithi189 ceph-mon[36790]: pgmap v7242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:51:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:58 smithi193 ceph-mon[37473]: pgmap v7241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:51:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:51:58 smithi193 ceph-mon[37473]: pgmap v7242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:52:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12260.320985 secs 2024-07-26T14:52:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:00 smithi134 ceph-mon[31966]: pgmap v7243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12260.320985 secs 2024-07-26T14:52:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:00 smithi189 ceph-mon[36790]: pgmap v7243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12260.320985 secs 2024-07-26T14:52:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:00 smithi193 ceph-mon[37473]: pgmap v7243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:02 smithi134 ceph-mon[31966]: pgmap v7244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12279.745357 secs 2024-07-26T14:52:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:02 smithi189 ceph-mon[36790]: pgmap v7244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12279.745357 secs 2024-07-26T14:52:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:02 smithi193 ceph-mon[37473]: pgmap v7244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12279.745357 secs 2024-07-26T14:52:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:04 smithi134 ceph-mon[31966]: pgmap v7245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:04 smithi189 ceph-mon[36790]: pgmap v7245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:04 smithi193 ceph-mon[37473]: pgmap v7245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:05.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12265.321050 secs 2024-07-26T14:52:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12265.321050 secs 2024-07-26T14:52:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12265.321050 secs 2024-07-26T14:52:06.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:06 smithi134 ceph-mon[31966]: pgmap v7246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:52:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:06 smithi189 ceph-mon[36790]: pgmap v7246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:52:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:06 smithi193 ceph-mon[37473]: pgmap v7246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:52:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12284.745431 secs 2024-07-26T14:52:07.424 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12284.745431 secs 2024-07-26T14:52:07.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12284.745431 secs 2024-07-26T14:52:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:52:08.232+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:52:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:52:08.232+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:08.233 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:52:08.232+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:52:08.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:08 smithi134 ceph-mon[31966]: pgmap v7247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:52:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:08 smithi189 ceph-mon[36790]: pgmap v7247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:52:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:08 smithi193 ceph-mon[37473]: pgmap v7247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:52:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12270.321133 secs 2024-07-26T14:52:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12270.321133 secs 2024-07-26T14:52:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12270.321133 secs 2024-07-26T14:52:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:10 smithi189 ceph-mon[36790]: pgmap v7248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:10 smithi193 ceph-mon[37473]: pgmap v7248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:10 smithi134 ceph-mon[31966]: pgmap v7248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:12 smithi189 ceph-mon[36790]: pgmap v7249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12289.745547 secs 2024-07-26T14:52:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:12 smithi193 ceph-mon[37473]: pgmap v7249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12289.745547 secs 2024-07-26T14:52:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:12 smithi134 ceph-mon[31966]: pgmap v7249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12289.745547 secs 2024-07-26T14:52:12.711 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:52:12.710+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:52:12.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:52:12.710+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:12.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:52:12.710+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:52:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:14 smithi193 ceph-mon[37473]: pgmap v7250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:52:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12275.321227 secs 2024-07-26T14:52:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:14 smithi134 ceph-mon[31966]: pgmap v7250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:52:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12275.321227 secs 2024-07-26T14:52:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:14 smithi189 ceph-mon[36790]: pgmap v7250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:52:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12275.321227 secs 2024-07-26T14:52:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:16 smithi193 ceph-mon[37473]: pgmap v7251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:52:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:16 smithi134 ceph-mon[31966]: pgmap v7251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:52:16.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:16 smithi189 ceph-mon[36790]: pgmap v7251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:52:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12294.745656 secs 2024-07-26T14:52:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12294.745656 secs 2024-07-26T14:52:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12294.745656 secs 2024-07-26T14:52:18.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:18 smithi193 ceph-mon[37473]: pgmap v7252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:18.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:18 smithi134 ceph-mon[31966]: pgmap v7252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:18 smithi189 ceph-mon[36790]: pgmap v7252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12280.321306 secs 2024-07-26T14:52:19.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12280.321306 secs 2024-07-26T14:52:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12280.321306 secs 2024-07-26T14:52:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:20 smithi193 ceph-mon[37473]: pgmap v7253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:52:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:20 smithi134 ceph-mon[31966]: pgmap v7253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:52:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:20 smithi189 ceph-mon[36790]: pgmap v7253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:52:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:22 smithi193 ceph-mon[37473]: pgmap v7254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:52:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12299.745748 secs 2024-07-26T14:52:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:22 smithi134 ceph-mon[31966]: pgmap v7254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:52:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12299.745748 secs 2024-07-26T14:52:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:22 smithi189 ceph-mon[36790]: pgmap v7254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:52:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12299.745748 secs 2024-07-26T14:52:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:52:23.234+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:52:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:52:23.234+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:23.235 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:52:23.234+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:52:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:24 smithi193 ceph-mon[37473]: pgmap v7255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:52:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12285.321419 secs 2024-07-26T14:52:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:24 smithi134 ceph-mon[31966]: pgmap v7255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:52:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12285.321419 secs 2024-07-26T14:52:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:24 smithi189 ceph-mon[36790]: pgmap v7255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:52:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12285.321419 secs 2024-07-26T14:52:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:26 smithi193 ceph-mon[37473]: pgmap v7256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:52:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:26 smithi134 ceph-mon[31966]: pgmap v7256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:52:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:26 smithi189 ceph-mon[36790]: pgmap v7256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:52:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12304.745886 secs 2024-07-26T14:52:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12304.745886 secs 2024-07-26T14:52:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12304.745886 secs 2024-07-26T14:52:27.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:52:27.726+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:52:27.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:52:27.726+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:27.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:52:27.726+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:52:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:28 smithi193 ceph-mon[37473]: pgmap v7257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:52:28.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:28 smithi134 ceph-mon[31966]: pgmap v7257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:52:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:28 smithi189 ceph-mon[36790]: pgmap v7257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:52:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12290.321512 secs 2024-07-26T14:52:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12290.321512 secs 2024-07-26T14:52:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12290.321512 secs 2024-07-26T14:52:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:30 smithi134 ceph-mon[31966]: pgmap v7258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:30 smithi189 ceph-mon[36790]: pgmap v7258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:30 smithi193 ceph-mon[37473]: pgmap v7258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:32 smithi134 ceph-mon[31966]: pgmap v7259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:52:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12309.746003 secs 2024-07-26T14:52:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:32 smithi189 ceph-mon[36790]: pgmap v7259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:52:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12309.746003 secs 2024-07-26T14:52:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:32 smithi193 ceph-mon[37473]: pgmap v7259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:52:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12309.746003 secs 2024-07-26T14:52:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:34 smithi134 ceph-mon[31966]: pgmap v7260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:52:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12295.321637 secs 2024-07-26T14:52:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:34 smithi189 ceph-mon[36790]: pgmap v7260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:52:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12295.321637 secs 2024-07-26T14:52:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:34 smithi193 ceph-mon[37473]: pgmap v7260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:52:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12295.321637 secs 2024-07-26T14:52:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:36 smithi134 ceph-mon[31966]: pgmap v7261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:52:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:36 smithi189 ceph-mon[36790]: pgmap v7261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:52:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:36 smithi193 ceph-mon[37473]: pgmap v7261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:52:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12314.746119 secs 2024-07-26T14:52:37.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12314.746119 secs 2024-07-26T14:52:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12314.746119 secs 2024-07-26T14:52:38.250 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:52:38.250+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:52:38.251 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:52:38.251+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:52:38.251 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:52:38.251+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:38.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:38 smithi134 ceph-mon[31966]: pgmap v7262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:38 smithi189 ceph-mon[36790]: pgmap v7262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:38 smithi193 ceph-mon[37473]: pgmap v7262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:39.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12300.321776 secs 2024-07-26T14:52:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12300.321776 secs 2024-07-26T14:52:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12300.321776 secs 2024-07-26T14:52:40.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:40 smithi134 ceph-mon[31966]: pgmap v7263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:40 smithi189 ceph-mon[36790]: pgmap v7263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:40 smithi193 ceph-mon[37473]: pgmap v7263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:42 smithi134 ceph-mon[31966]: pgmap v7264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12319.746240 secs 2024-07-26T14:52:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:42 smithi189 ceph-mon[36790]: pgmap v7264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12319.746240 secs 2024-07-26T14:52:42.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:52:42.741+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:52:42.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:52:42.741+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:42.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:52:42.741+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:52:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:42 smithi193 ceph-mon[37473]: pgmap v7264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12319.746240 secs 2024-07-26T14:52:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:44 smithi134 ceph-mon[31966]: pgmap v7265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:52:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12305.321843 secs 2024-07-26T14:52:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:44 smithi189 ceph-mon[36790]: pgmap v7265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:52:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12305.321843 secs 2024-07-26T14:52:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:44 smithi193 ceph-mon[37473]: pgmap v7265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:52:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12305.321843 secs 2024-07-26T14:52:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:46 smithi134 ceph-mon[31966]: pgmap v7266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:52:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:46 smithi189 ceph-mon[36790]: pgmap v7266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:52:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:46 smithi193 ceph-mon[37473]: pgmap v7266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:52:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12324.746378 secs 2024-07-26T14:52:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12324.746378 secs 2024-07-26T14:52:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12324.746378 secs 2024-07-26T14:52:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:48 smithi134 ceph-mon[31966]: pgmap v7267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:48 smithi189 ceph-mon[36790]: pgmap v7267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:48 smithi193 ceph-mon[37473]: pgmap v7267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12310.321980 secs 2024-07-26T14:52:49.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12310.321980 secs 2024-07-26T14:52:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12310.321980 secs 2024-07-26T14:52:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:50 smithi134 ceph-mon[31966]: pgmap v7268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:50 smithi189 ceph-mon[36790]: pgmap v7268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:50 smithi193 ceph-mon[37473]: pgmap v7268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:52:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:52 smithi134 ceph-mon[31966]: pgmap v7269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12329.746441 secs 2024-07-26T14:52:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:52 smithi189 ceph-mon[36790]: pgmap v7269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12329.746441 secs 2024-07-26T14:52:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:52 smithi193 ceph-mon[37473]: pgmap v7269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:52:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12329.746441 secs 2024-07-26T14:52:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:52:53.254+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:52:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:52:53.254+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:53.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:52:53.254+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:52:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:54 smithi134 ceph-mon[31966]: pgmap v7270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:52:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12315.322074 secs 2024-07-26T14:52:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:54 smithi189 ceph-mon[36790]: pgmap v7270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:52:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12315.322074 secs 2024-07-26T14:52:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:54 smithi193 ceph-mon[37473]: pgmap v7270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:52:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12315.322074 secs 2024-07-26T14:52:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:56 smithi134 ceph-mon[31966]: pgmap v7271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:52:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:56 smithi189 ceph-mon[36790]: pgmap v7271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:52:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:56 smithi193 ceph-mon[37473]: pgmap v7271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:52:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12334.746519 secs 2024-07-26T14:52:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:52:57.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12334.746519 secs 2024-07-26T14:52:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:52:57.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:52:57.756+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:52:57.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:52:57.756+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:52:57.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:52:57.756+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:52:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12334.746519 secs 2024-07-26T14:52:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:52:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:58 smithi134 ceph-mon[31966]: pgmap v7272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:52:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:52:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:52:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:58 smithi189 ceph-mon[36790]: pgmap v7272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:52:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:52:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:52:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:58 smithi193 ceph-mon[37473]: pgmap v7272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:52:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:52:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:52:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:52:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:59 smithi134 ceph-mon[31966]: pgmap v7273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:52:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:59 smithi134 ceph-mon[31966]: pgmap v7274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:52:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12320.322223 secs 2024-07-26T14:52:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:59 smithi189 ceph-mon[36790]: pgmap v7273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:52:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:59 smithi189 ceph-mon[36790]: pgmap v7274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:52:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12320.322223 secs 2024-07-26T14:52:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:59 smithi193 ceph-mon[37473]: pgmap v7273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:52:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:59 smithi193 ceph-mon[37473]: pgmap v7274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:52:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:52:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12320.322223 secs 2024-07-26T14:53:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:01 smithi189 ceph-mon[36790]: pgmap v7275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:53:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:01 smithi193 ceph-mon[37473]: pgmap v7275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:53:01.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:01 smithi134 ceph-mon[31966]: pgmap v7275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:53:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12339.746644 secs 2024-07-26T14:53:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12339.746644 secs 2024-07-26T14:53:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12339.746644 secs 2024-07-26T14:53:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:03 smithi189 ceph-mon[36790]: pgmap v7276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:53:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:03 smithi193 ceph-mon[37473]: pgmap v7276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:53:03.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:03 smithi134 ceph-mon[31966]: pgmap v7276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T14:53:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12325.322329 secs 2024-07-26T14:53:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12325.322329 secs 2024-07-26T14:53:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12325.322329 secs 2024-07-26T14:53:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:05 smithi193 ceph-mon[37473]: pgmap v7277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:53:05.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:05 smithi134 ceph-mon[31966]: pgmap v7277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:53:05.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:05 smithi189 ceph-mon[36790]: pgmap v7277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:53:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12344.746769 secs 2024-07-26T14:53:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:07 smithi193 ceph-mon[37473]: pgmap v7278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:53:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12344.746769 secs 2024-07-26T14:53:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:07 smithi134 ceph-mon[31966]: pgmap v7278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:53:07.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12344.746769 secs 2024-07-26T14:53:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:07 smithi189 ceph-mon[36790]: pgmap v7278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:53:08.261 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:53:08.261+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:53:08.261 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:53:08.261+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:53:08.261 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:53:08.261+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:09.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:09 smithi189 ceph-mon[36790]: pgmap v7279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:53:09.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12330.322395 secs 2024-07-26T14:53:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:09 smithi193 ceph-mon[37473]: pgmap v7279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:53:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12330.322395 secs 2024-07-26T14:53:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:09 smithi134 ceph-mon[31966]: pgmap v7279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:53:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12330.322395 secs 2024-07-26T14:53:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:10 smithi193 ceph-mon[37473]: pgmap v7280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:10.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:10 smithi134 ceph-mon[31966]: pgmap v7280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:10 smithi189 ceph-mon[36790]: pgmap v7280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12349.746888 secs 2024-07-26T14:53:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12349.746888 secs 2024-07-26T14:53:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12349.746888 secs 2024-07-26T14:53:12.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:53:12.772+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:53:12.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:53:12.772+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:12.774 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:53:12.772+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:53:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:12 smithi193 ceph-mon[37473]: pgmap v7281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:12 smithi134 ceph-mon[31966]: pgmap v7281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:12 smithi189 ceph-mon[36790]: pgmap v7281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12335.322476 secs 2024-07-26T14:53:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:15 smithi134 ceph-mon[31966]: pgmap v7282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12335.322476 secs 2024-07-26T14:53:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:15 smithi189 ceph-mon[36790]: pgmap v7282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12335.322476 secs 2024-07-26T14:53:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:15 smithi193 ceph-mon[37473]: pgmap v7282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12354.747006 secs 2024-07-26T14:53:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:17 smithi134 ceph-mon[31966]: pgmap v7283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12354.747006 secs 2024-07-26T14:53:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:17 smithi189 ceph-mon[36790]: pgmap v7283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12354.747006 secs 2024-07-26T14:53:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:17 smithi193 ceph-mon[37473]: pgmap v7283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:19 smithi134 ceph-mon[31966]: pgmap v7284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:53:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12340.322610 secs 2024-07-26T14:53:19.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:19 smithi189 ceph-mon[36790]: pgmap v7284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:53:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12340.322610 secs 2024-07-26T14:53:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:19 smithi193 ceph-mon[37473]: pgmap v7284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:53:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12340.322610 secs 2024-07-26T14:53:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:21 smithi189 ceph-mon[36790]: pgmap v7285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:21 smithi193 ceph-mon[37473]: pgmap v7285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:21 smithi134 ceph-mon[31966]: pgmap v7285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12359.747144 secs 2024-07-26T14:53:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12359.747144 secs 2024-07-26T14:53:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12359.747144 secs 2024-07-26T14:53:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:53:23.276+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:53:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:53:23.276+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:23.277 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:53:23.276+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:53:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:23 smithi189 ceph-mon[36790]: pgmap v7286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:53:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:23 smithi193 ceph-mon[37473]: pgmap v7286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:53:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:23 smithi134 ceph-mon[31966]: pgmap v7286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:53:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12345.322679 secs 2024-07-26T14:53:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12345.322679 secs 2024-07-26T14:53:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12345.322679 secs 2024-07-26T14:53:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:25 smithi193 ceph-mon[37473]: pgmap v7287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:25 smithi134 ceph-mon[31966]: pgmap v7287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:25 smithi189 ceph-mon[36790]: pgmap v7287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12364.747276 secs 2024-07-26T14:53:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:27 smithi193 ceph-mon[37473]: pgmap v7288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12364.747276 secs 2024-07-26T14:53:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:27 smithi134 ceph-mon[31966]: pgmap v7288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12364.747276 secs 2024-07-26T14:53:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:27 smithi189 ceph-mon[36790]: pgmap v7288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:27.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:53:27.785+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:53:27.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:53:27.785+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:53:27.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:53:27.785+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:29 smithi193 ceph-mon[37473]: pgmap v7289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:53:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12350.322808 secs 2024-07-26T14:53:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:29 smithi134 ceph-mon[31966]: pgmap v7289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:53:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12350.322808 secs 2024-07-26T14:53:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:29 smithi189 ceph-mon[36790]: pgmap v7289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:53:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12350.322808 secs 2024-07-26T14:53:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:31 smithi193 ceph-mon[37473]: pgmap v7290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:31 smithi134 ceph-mon[31966]: pgmap v7290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:31 smithi189 ceph-mon[36790]: pgmap v7290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12369.747406 secs 2024-07-26T14:53:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12369.747406 secs 2024-07-26T14:53:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12369.747406 secs 2024-07-26T14:53:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:33 smithi193 ceph-mon[37473]: pgmap v7291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:33.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:33 smithi134 ceph-mon[31966]: pgmap v7291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:33 smithi189 ceph-mon[36790]: pgmap v7291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12355.322963 secs 2024-07-26T14:53:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12355.322963 secs 2024-07-26T14:53:34.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12355.322963 secs 2024-07-26T14:53:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:35 smithi193 ceph-mon[37473]: pgmap v7292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:35 smithi134 ceph-mon[31966]: pgmap v7292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:35 smithi189 ceph-mon[36790]: pgmap v7292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12374.747496 secs 2024-07-26T14:53:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:37 smithi193 ceph-mon[37473]: pgmap v7293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12374.747496 secs 2024-07-26T14:53:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:37 smithi134 ceph-mon[31966]: pgmap v7293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12374.747496 secs 2024-07-26T14:53:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:37 smithi189 ceph-mon[36790]: pgmap v7293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:38.292 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:53:38.291+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:53:38.293 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:53:38.291+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:38.293 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:53:38.291+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:53:39.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:53:39.482+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T14:53:39.483 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:53:39.482+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T14:53:39.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:53:39.483+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 97 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T14:53:39.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:53:39.483+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T14:53:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:39 smithi193 ceph-mon[37473]: pgmap v7294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:53:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12360.323088 secs 2024-07-26T14:53:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:39 smithi134 ceph-mon[31966]: pgmap v7294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:53:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12360.323088 secs 2024-07-26T14:53:39.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:39 smithi189 ceph-mon[36790]: pgmap v7294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:53:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12360.323088 secs 2024-07-26T14:53:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:40 smithi193 ceph-mon[37473]: pgmap v7295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:41.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:40 smithi134 ceph-mon[31966]: pgmap v7295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:40 smithi189 ceph-mon[36790]: pgmap v7295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12379.747616 secs 2024-07-26T14:53:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12379.747616 secs 2024-07-26T14:53:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12379.747616 secs 2024-07-26T14:53:42.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:53:42.786+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:53:42.787 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:53:42.786+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:42.788 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:53:42.786+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:53:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:43 smithi134 ceph-mon[31966]: pgmap v7296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:43 smithi193 ceph-mon[37473]: pgmap v7296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:43 smithi189 ceph-mon[36790]: pgmap v7296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12365.323155 secs 2024-07-26T14:53:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12365.323155 secs 2024-07-26T14:53:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12365.323155 secs 2024-07-26T14:53:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:45 smithi193 ceph-mon[37473]: pgmap v7297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:45 smithi134 ceph-mon[31966]: pgmap v7297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:45 smithi189 ceph-mon[36790]: pgmap v7297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12384.747733 secs 2024-07-26T14:53:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:47 smithi193 ceph-mon[37473]: pgmap v7298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12384.747733 secs 2024-07-26T14:53:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:47 smithi134 ceph-mon[31966]: pgmap v7298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12384.747733 secs 2024-07-26T14:53:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:47 smithi189 ceph-mon[36790]: pgmap v7298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:49 smithi193 ceph-mon[37473]: pgmap v7299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:53:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12370.323238 secs 2024-07-26T14:53:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:49 smithi134 ceph-mon[31966]: pgmap v7299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:53:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12370.323238 secs 2024-07-26T14:53:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:49 smithi189 ceph-mon[36790]: pgmap v7299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:53:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12370.323238 secs 2024-07-26T14:53:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:53:50.014+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:53:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:53:50.014+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:53:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:53:50.015+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:53:50.015 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:53:50.015+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:53:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:51 smithi193 ceph-mon[37473]: pgmap v7300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:51 smithi134 ceph-mon[31966]: pgmap v7300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:51 smithi189 ceph-mon[36790]: pgmap v7300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12389.747806 secs 2024-07-26T14:53:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12389.747806 secs 2024-07-26T14:53:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12389.747806 secs 2024-07-26T14:53:53.303 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:53:53.302+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:53:53.303 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:53:53.302+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:53.303 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:53:53.302+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:53:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:53 smithi193 ceph-mon[37473]: pgmap v7301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:53.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:53 smithi134 ceph-mon[31966]: pgmap v7301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:53 smithi189 ceph-mon[36790]: pgmap v7301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:53:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12375.323396 secs 2024-07-26T14:53:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12375.323396 secs 2024-07-26T14:53:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12375.323396 secs 2024-07-26T14:53:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:55 smithi193 ceph-mon[37473]: pgmap v7302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:55 smithi134 ceph-mon[31966]: pgmap v7302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:55 smithi189 ceph-mon[36790]: pgmap v7302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:53:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12394.747904 secs 2024-07-26T14:53:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:57 smithi134 ceph-mon[31966]: pgmap v7303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12394.747904 secs 2024-07-26T14:53:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:57 smithi189 ceph-mon[36790]: pgmap v7303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12394.747904 secs 2024-07-26T14:53:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:57 smithi193 ceph-mon[37473]: pgmap v7303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:53:57.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:53:57.801+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:53:57.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:53:57.802+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:53:57.803 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:53:57.802+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:53:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:59 smithi134 ceph-mon[31966]: pgmap v7304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:53:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:53:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:53:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12380.323487 secs 2024-07-26T14:53:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:59 smithi189 ceph-mon[36790]: pgmap v7304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:53:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:53:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:53:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12380.323487 secs 2024-07-26T14:53:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:59 smithi193 ceph-mon[37473]: pgmap v7304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:53:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:53:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:53:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12380.323487 secs 2024-07-26T14:54:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:54:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:54:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:00 smithi134 ceph-mon[31966]: pgmap v7305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:54:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:00 smithi134 ceph-mon[31966]: pgmap v7306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:54:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:54:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:54:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:54:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:00 smithi189 ceph-mon[36790]: pgmap v7305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:54:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:00 smithi189 ceph-mon[36790]: pgmap v7306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:54:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:54:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:54:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:54:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:00 smithi193 ceph-mon[37473]: pgmap v7305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:54:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:00 smithi193 ceph-mon[37473]: pgmap v7306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T14:54:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:54:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12399.748030 secs 2024-07-26T14:54:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:02 smithi134 ceph-mon[31966]: pgmap v7307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:54:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12399.748030 secs 2024-07-26T14:54:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:02 smithi189 ceph-mon[36790]: pgmap v7307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:54:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12399.748030 secs 2024-07-26T14:54:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:02 smithi193 ceph-mon[37473]: pgmap v7307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T14:54:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:04 smithi134 ceph-mon[31966]: pgmap v7308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:54:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12385.323613 secs 2024-07-26T14:54:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:04 smithi189 ceph-mon[36790]: pgmap v7308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:54:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12385.323613 secs 2024-07-26T14:54:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:04 smithi193 ceph-mon[37473]: pgmap v7308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:54:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12385.323613 secs 2024-07-26T14:54:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:06 smithi134 ceph-mon[31966]: pgmap v7309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:54:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:06 smithi189 ceph-mon[36790]: pgmap v7309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:54:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:06 smithi193 ceph-mon[37473]: pgmap v7309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:54:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12404.748145 secs 2024-07-26T14:54:07.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12404.748145 secs 2024-07-26T14:54:07.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12404.748145 secs 2024-07-26T14:54:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:54:08.307+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:54:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:54:08.307+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:54:08.307+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:54:08.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:08 smithi134 ceph-mon[31966]: pgmap v7310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:54:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:08 smithi189 ceph-mon[36790]: pgmap v7310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:54:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:08 smithi193 ceph-mon[37473]: pgmap v7310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:54:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12390.323736 secs 2024-07-26T14:54:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12390.323736 secs 2024-07-26T14:54:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12390.323736 secs 2024-07-26T14:54:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:10 smithi134 ceph-mon[31966]: pgmap v7311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:54:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:10 smithi189 ceph-mon[36790]: pgmap v7311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:54:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:10 smithi193 ceph-mon[37473]: pgmap v7311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:54:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12409.748293 secs 2024-07-26T14:54:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:12 smithi134 ceph-mon[31966]: pgmap v7312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12409.748293 secs 2024-07-26T14:54:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:12 smithi189 ceph-mon[36790]: pgmap v7312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12409.748293 secs 2024-07-26T14:54:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:12 smithi193 ceph-mon[37473]: pgmap v7312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:12.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:54:12.804+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:54:12.805 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:54:12.804+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:54:12.806 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:54:12.804+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:14 smithi134 ceph-mon[31966]: pgmap v7313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:54:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12395.323885 secs 2024-07-26T14:54:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:14 smithi189 ceph-mon[36790]: pgmap v7313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:54:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12395.323885 secs 2024-07-26T14:54:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:14 smithi193 ceph-mon[37473]: pgmap v7313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:54:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12395.323885 secs 2024-07-26T14:54:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:16 smithi134 ceph-mon[31966]: pgmap v7314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:16 smithi189 ceph-mon[36790]: pgmap v7314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:16 smithi193 ceph-mon[37473]: pgmap v7314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12414.748353 secs 2024-07-26T14:54:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12414.748353 secs 2024-07-26T14:54:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12414.748353 secs 2024-07-26T14:54:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:18 smithi189 ceph-mon[36790]: pgmap v7315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:54:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:18 smithi193 ceph-mon[37473]: pgmap v7315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:54:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:18 smithi134 ceph-mon[31966]: pgmap v7315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:54:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12400.324012 secs 2024-07-26T14:54:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12400.324012 secs 2024-07-26T14:54:19.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12400.324012 secs 2024-07-26T14:54:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:20 smithi189 ceph-mon[36790]: pgmap v7316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:54:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:20 smithi193 ceph-mon[37473]: pgmap v7316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:54:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:20 smithi134 ceph-mon[31966]: pgmap v7316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:54:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12419.748483 secs 2024-07-26T14:54:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:22 smithi193 ceph-mon[37473]: pgmap v7317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12419.748483 secs 2024-07-26T14:54:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:22 smithi134 ceph-mon[31966]: pgmap v7317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12419.748483 secs 2024-07-26T14:54:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:22 smithi189 ceph-mon[36790]: pgmap v7317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:23.324 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:54:23.323+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:54:23.324 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:54:23.323+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:54:23.324 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:54:23.323+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:24.491 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:54:24.490+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T14:54:24.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:54:24.490+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T14:54:24.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:54:24.491+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T14:54:24.492 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:54:24.491+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 98 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T14:54:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:24 smithi193 ceph-mon[37473]: pgmap v7318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:54:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12405.324140 secs 2024-07-26T14:54:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:24 smithi134 ceph-mon[31966]: pgmap v7318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:54:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12405.324140 secs 2024-07-26T14:54:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:24 smithi189 ceph-mon[36790]: pgmap v7318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:54:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12405.324140 secs 2024-07-26T14:54:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:26 smithi193 ceph-mon[37473]: pgmap v7319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:26 smithi134 ceph-mon[31966]: pgmap v7319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:26 smithi189 ceph-mon[36790]: pgmap v7319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12424.748605 secs 2024-07-26T14:54:27.821 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:54:27.820+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:54:27.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:54:27.820+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:54:27.822 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:54:27.820+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:27.822 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12424.748605 secs 2024-07-26T14:54:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12424.748605 secs 2024-07-26T14:54:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:28 smithi193 ceph-mon[37473]: pgmap v7320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:54:28.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:28 smithi134 ceph-mon[31966]: pgmap v7320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:54:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:28 smithi189 ceph-mon[36790]: pgmap v7320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:54:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12410.324297 secs 2024-07-26T14:54:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12410.324297 secs 2024-07-26T14:54:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12410.324297 secs 2024-07-26T14:54:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:30 smithi193 ceph-mon[37473]: pgmap v7321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:54:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:30 smithi134 ceph-mon[31966]: pgmap v7321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:54:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:30 smithi189 ceph-mon[36790]: pgmap v7321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:54:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12429.748722 secs 2024-07-26T14:54:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:32 smithi193 ceph-mon[37473]: pgmap v7322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12429.748722 secs 2024-07-26T14:54:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:32 smithi134 ceph-mon[31966]: pgmap v7322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12429.748722 secs 2024-07-26T14:54:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:32 smithi189 ceph-mon[36790]: pgmap v7322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:33 smithi193 ceph-mon[37473]: pgmap v7323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:54:33.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:33 smithi134 ceph-mon[31966]: pgmap v7323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:54:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:33 smithi189 ceph-mon[36790]: pgmap v7323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:54:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12415.324402 secs 2024-07-26T14:54:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12415.324402 secs 2024-07-26T14:54:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12415.324402 secs 2024-07-26T14:54:35.023 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:54:35.022+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:54:35.023 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:54:35.022+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:54:35.023 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:54:35.023+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:54:35.023 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:54:35.023+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:54:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:35 smithi193 ceph-mon[37473]: pgmap v7324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:35.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:35 smithi134 ceph-mon[31966]: pgmap v7324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:35 smithi189 ceph-mon[36790]: pgmap v7324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12434.748843 secs 2024-07-26T14:54:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12434.748843 secs 2024-07-26T14:54:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12434.748843 secs 2024-07-26T14:54:38.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:37 smithi134 ceph-mon[31966]: pgmap v7325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:54:38.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:37 smithi189 ceph-mon[36790]: pgmap v7325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:54:38.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:37 smithi193 ceph-mon[37473]: pgmap v7325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:54:38.339 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:54:38.339+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:54:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:54:38.339+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:38.340 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:54:38.339+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:54:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12420.324485 secs 2024-07-26T14:54:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12420.324485 secs 2024-07-26T14:54:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12420.324485 secs 2024-07-26T14:54:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:40 smithi193 ceph-mon[37473]: pgmap v7326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:54:40.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:40 smithi134 ceph-mon[31966]: pgmap v7326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:54:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:40 smithi189 ceph-mon[36790]: pgmap v7326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:54:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12439.748963 secs 2024-07-26T14:54:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:42 smithi134 ceph-mon[31966]: pgmap v7327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12439.748963 secs 2024-07-26T14:54:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:42 smithi189 ceph-mon[36790]: pgmap v7327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12439.748963 secs 2024-07-26T14:54:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:42 smithi193 ceph-mon[37473]: pgmap v7327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:42.837 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:54:42.836+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:54:42.837 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:54:42.836+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:42.837 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:54:42.836+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:54:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:44 smithi134 ceph-mon[31966]: pgmap v7328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:54:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12425.324605 secs 2024-07-26T14:54:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:44 smithi189 ceph-mon[36790]: pgmap v7328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:54:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12425.324605 secs 2024-07-26T14:54:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:44 smithi193 ceph-mon[37473]: pgmap v7328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:54:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12425.324605 secs 2024-07-26T14:54:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:46 smithi134 ceph-mon[31966]: pgmap v7329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:46 smithi189 ceph-mon[36790]: pgmap v7329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:46 smithi193 ceph-mon[37473]: pgmap v7329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12444.749085 secs 2024-07-26T14:54:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12444.749085 secs 2024-07-26T14:54:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12444.749085 secs 2024-07-26T14:54:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:48 smithi134 ceph-mon[31966]: pgmap v7330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:54:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:48 smithi189 ceph-mon[36790]: pgmap v7330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:54:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:48 smithi193 ceph-mon[37473]: pgmap v7330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:54:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12430.324762 secs 2024-07-26T14:54:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12430.324762 secs 2024-07-26T14:54:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12430.324762 secs 2024-07-26T14:54:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:50 smithi134 ceph-mon[31966]: pgmap v7331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:54:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:50 smithi189 ceph-mon[36790]: pgmap v7331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:54:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:50 smithi193 ceph-mon[37473]: pgmap v7331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:54:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12449.749151 secs 2024-07-26T14:54:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:52 smithi189 ceph-mon[36790]: pgmap v7332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12449.749151 secs 2024-07-26T14:54:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:52 smithi193 ceph-mon[37473]: pgmap v7332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12449.749151 secs 2024-07-26T14:54:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:52 smithi134 ceph-mon[31966]: pgmap v7332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:54:53.354+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:54:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:54:53.354+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:53.355 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:54:53.354+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:54:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:54 smithi189 ceph-mon[36790]: pgmap v7333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:54:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12435.324851 secs 2024-07-26T14:54:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:54 smithi193 ceph-mon[37473]: pgmap v7333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:54:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12435.324851 secs 2024-07-26T14:54:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:54 smithi134 ceph-mon[31966]: pgmap v7333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:54:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12435.324851 secs 2024-07-26T14:54:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:56 smithi189 ceph-mon[36790]: pgmap v7334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:56 smithi193 ceph-mon[37473]: pgmap v7334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:56.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:56 smithi134 ceph-mon[31966]: pgmap v7334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:54:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12454.749271 secs 2024-07-26T14:54:57.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:54:57.837+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:54:57.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:54:57.837+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:54:57.839 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:54:57.837+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:54:57.839 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12454.749271 secs 2024-07-26T14:54:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12454.749271 secs 2024-07-26T14:54:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:58 smithi193 ceph-mon[37473]: pgmap v7335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:54:58.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:58 smithi134 ceph-mon[31966]: pgmap v7335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:54:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:58 smithi189 ceph-mon[36790]: pgmap v7335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:54:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:54:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12440.324997 secs 2024-07-26T14:54:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:54:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12440.324997 secs 2024-07-26T14:54:59.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:54:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12440.324997 secs 2024-07-26T14:55:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:00 smithi193 ceph-mon[37473]: pgmap v7336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:55:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:55:00.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:00 smithi134 ceph-mon[31966]: pgmap v7336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:55:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:55:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:00 smithi189 ceph-mon[36790]: pgmap v7336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:55:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:55:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:55:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:55:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:55:01.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:55:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:55:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:55:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:55:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:55:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:55:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:02 smithi193 ceph-mon[37473]: pgmap v7337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:55:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12459.749413 secs 2024-07-26T14:55:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:02 smithi134 ceph-mon[31966]: pgmap v7337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:55:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12459.749413 secs 2024-07-26T14:55:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:02 smithi189 ceph-mon[36790]: pgmap v7337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:55:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12459.749413 secs 2024-07-26T14:55:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:04 smithi193 ceph-mon[37473]: pgmap v7338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:55:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12445.325111 secs 2024-07-26T14:55:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:04 smithi134 ceph-mon[31966]: pgmap v7338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:55:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12445.325111 secs 2024-07-26T14:55:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:04 smithi189 ceph-mon[36790]: pgmap v7338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T14:55:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12445.325111 secs 2024-07-26T14:55:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:06 smithi193 ceph-mon[37473]: pgmap v7339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:55:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:06 smithi134 ceph-mon[31966]: pgmap v7339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:55:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:06 smithi189 ceph-mon[36790]: pgmap v7339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:55:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12464.749542 secs 2024-07-26T14:55:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12464.749542 secs 2024-07-26T14:55:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12464.749542 secs 2024-07-26T14:55:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:55:08.369+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:55:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:55:08.369+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:08.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:55:08.369+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:55:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:08 smithi193 ceph-mon[37473]: pgmap v7340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:55:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:08 smithi134 ceph-mon[31966]: pgmap v7340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:55:08.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:08 smithi189 ceph-mon[36790]: pgmap v7340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:55:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:09 smithi193 ceph-mon[37473]: pgmap v7341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:55:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12450.325242 secs 2024-07-26T14:55:09.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:09 smithi189 ceph-mon[36790]: pgmap v7341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:55:09.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12450.325242 secs 2024-07-26T14:55:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:09 smithi134 ceph-mon[31966]: pgmap v7341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:55:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12450.325242 secs 2024-07-26T14:55:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:11 smithi193 ceph-mon[37473]: pgmap v7342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:55:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:11 smithi134 ceph-mon[31966]: pgmap v7342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:55:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:11 smithi189 ceph-mon[36790]: pgmap v7342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T14:55:12.854 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:55:12.853+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:55:12.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:55:12.853+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:55:12.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:55:12.853+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:12.855 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12469.749658 secs 2024-07-26T14:55:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12469.749658 secs 2024-07-26T14:55:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12469.749658 secs 2024-07-26T14:55:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:13 smithi134 ceph-mon[31966]: pgmap v7343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:13 smithi189 ceph-mon[36790]: pgmap v7343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:13 smithi193 ceph-mon[37473]: pgmap v7343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12455.325318 secs 2024-07-26T14:55:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12455.325318 secs 2024-07-26T14:55:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12455.325318 secs 2024-07-26T14:55:15.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:15 smithi134 ceph-mon[31966]: pgmap v7344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:55:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:15 smithi189 ceph-mon[36790]: pgmap v7344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:55:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:15 smithi193 ceph-mon[37473]: pgmap v7344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:55:17.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12474.749802 secs 2024-07-26T14:55:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:17 smithi134 ceph-mon[31966]: pgmap v7345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12474.749802 secs 2024-07-26T14:55:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:17 smithi189 ceph-mon[36790]: pgmap v7345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12474.749802 secs 2024-07-26T14:55:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:17 smithi193 ceph-mon[37473]: pgmap v7345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:19 smithi134 ceph-mon[31966]: pgmap v7346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12460.325445 secs 2024-07-26T14:55:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:19 smithi189 ceph-mon[36790]: pgmap v7346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12460.325445 secs 2024-07-26T14:55:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:19 smithi193 ceph-mon[37473]: pgmap v7346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12460.325445 secs 2024-07-26T14:55:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:21 smithi134 ceph-mon[31966]: pgmap v7347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:21 smithi189 ceph-mon[36790]: pgmap v7347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:21 smithi193 ceph-mon[37473]: pgmap v7347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12479.749869 secs 2024-07-26T14:55:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12479.749869 secs 2024-07-26T14:55:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12479.749869 secs 2024-07-26T14:55:23.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:55:23.385+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:55:23.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:55:23.385+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:55:23.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:55:23.385+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:23.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:23 smithi134 ceph-mon[31966]: pgmap v7348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:23 smithi189 ceph-mon[36790]: pgmap v7348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:23 smithi193 ceph-mon[37473]: pgmap v7348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12465.325578 secs 2024-07-26T14:55:24.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12465.325578 secs 2024-07-26T14:55:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12465.325578 secs 2024-07-26T14:55:25.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:25 smithi134 ceph-mon[31966]: pgmap v7349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:55:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:25 smithi189 ceph-mon[36790]: pgmap v7349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:55:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:25 smithi193 ceph-mon[37473]: pgmap v7349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:55:27.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:55:27.869+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:55:27.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:55:27.869+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:27.870 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:55:27.869+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:55:27.870 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12484.749983 secs 2024-07-26T14:55:27.870 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:27 smithi134 ceph-mon[31966]: pgmap v7350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12484.749983 secs 2024-07-26T14:55:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:27 smithi189 ceph-mon[36790]: pgmap v7350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12484.749983 secs 2024-07-26T14:55:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:27 smithi193 ceph-mon[37473]: pgmap v7350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:29 smithi134 ceph-mon[31966]: pgmap v7351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12470.325701 secs 2024-07-26T14:55:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:29 smithi189 ceph-mon[36790]: pgmap v7351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12470.325701 secs 2024-07-26T14:55:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:29 smithi193 ceph-mon[37473]: pgmap v7351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12470.325701 secs 2024-07-26T14:55:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:31 smithi134 ceph-mon[31966]: pgmap v7352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:31.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:31 smithi189 ceph-mon[36790]: pgmap v7352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:31 smithi193 ceph-mon[37473]: pgmap v7352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12489.750123 secs 2024-07-26T14:55:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12489.750123 secs 2024-07-26T14:55:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12489.750123 secs 2024-07-26T14:55:33.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:33 smithi134 ceph-mon[31966]: pgmap v7353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:33 smithi189 ceph-mon[36790]: pgmap v7353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:33 smithi193 ceph-mon[37473]: pgmap v7353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12475.325858 secs 2024-07-26T14:55:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12475.325858 secs 2024-07-26T14:55:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12475.325858 secs 2024-07-26T14:55:35.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:35 smithi134 ceph-mon[31966]: pgmap v7354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:55:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:35 smithi189 ceph-mon[36790]: pgmap v7354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:55:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:35 smithi193 ceph-mon[37473]: pgmap v7354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:55:37.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12494.750196 secs 2024-07-26T14:55:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:37 smithi134 ceph-mon[31966]: pgmap v7355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12494.750196 secs 2024-07-26T14:55:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:37 smithi189 ceph-mon[36790]: pgmap v7355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12494.750196 secs 2024-07-26T14:55:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:37 smithi193 ceph-mon[37473]: pgmap v7355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:55:38.400+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:55:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:55:38.400+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:38.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:55:38.400+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:55:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:39 smithi134 ceph-mon[31966]: pgmap v7356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12480.325965 secs 2024-07-26T14:55:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:39 smithi189 ceph-mon[36790]: pgmap v7356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12480.325965 secs 2024-07-26T14:55:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:39 smithi193 ceph-mon[37473]: pgmap v7356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12480.325965 secs 2024-07-26T14:55:41.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:41 smithi134 ceph-mon[31966]: pgmap v7357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:41 smithi189 ceph-mon[36790]: pgmap v7357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:41 smithi193 ceph-mon[37473]: pgmap v7357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:42.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:55:42.881+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:55:42.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:55:42.881+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:55:42.883 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:55:42.881+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12499.750273 secs 2024-07-26T14:55:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12499.750273 secs 2024-07-26T14:55:43.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12499.750273 secs 2024-07-26T14:55:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:43 smithi189 ceph-mon[36790]: pgmap v7358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:43 smithi193 ceph-mon[37473]: pgmap v7358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:44.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:43 smithi134 ceph-mon[31966]: pgmap v7358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12485.326117 secs 2024-07-26T14:55:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12485.326117 secs 2024-07-26T14:55:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12485.326117 secs 2024-07-26T14:55:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:45 smithi193 ceph-mon[37473]: pgmap v7359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:55:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:45 smithi134 ceph-mon[31966]: pgmap v7359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:55:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:45 smithi189 ceph-mon[36790]: pgmap v7359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:55:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12504.750392 secs 2024-07-26T14:55:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:47 smithi193 ceph-mon[37473]: pgmap v7360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12504.750392 secs 2024-07-26T14:55:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:47 smithi134 ceph-mon[31966]: pgmap v7360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12504.750392 secs 2024-07-26T14:55:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:47 smithi189 ceph-mon[36790]: pgmap v7360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:49.974 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:49 smithi134 ceph-mon[31966]: pgmap v7361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:49.974 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12490.326237 secs 2024-07-26T14:55:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:49 smithi193 ceph-mon[37473]: pgmap v7361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12490.326237 secs 2024-07-26T14:55:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:49 smithi189 ceph-mon[36790]: pgmap v7361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12490.326237 secs 2024-07-26T14:55:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:51 smithi193 ceph-mon[37473]: pgmap v7362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:51 smithi134 ceph-mon[31966]: pgmap v7362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:51 smithi189 ceph-mon[36790]: pgmap v7362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12509.750476 secs 2024-07-26T14:55:53.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12509.750476 secs 2024-07-26T14:55:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12509.750476 secs 2024-07-26T14:55:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:55:53.411+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:55:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:55:53.411+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:55:53.412 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:55:53.411+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:53 smithi193 ceph-mon[37473]: pgmap v7363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:53 smithi134 ceph-mon[31966]: pgmap v7363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:53 smithi189 ceph-mon[36790]: pgmap v7363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:55:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12495.326364 secs 2024-07-26T14:55:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12495.326364 secs 2024-07-26T14:55:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12495.326364 secs 2024-07-26T14:55:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:55 smithi193 ceph-mon[37473]: pgmap v7364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:55:56.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:55 smithi134 ceph-mon[31966]: pgmap v7364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:55:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:55 smithi189 ceph-mon[36790]: pgmap v7364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:55:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12514.750594 secs 2024-07-26T14:55:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12514.750594 secs 2024-07-26T14:55:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12514.750594 secs 2024-07-26T14:55:57.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:55:57.897+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:55:57.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:55:57.897+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:55:57.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:55:57.897+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:55:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:57 smithi193 ceph-mon[37473]: pgmap v7365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:57 smithi134 ceph-mon[31966]: pgmap v7365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:55:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:57 smithi189 ceph-mon[36790]: pgmap v7365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:59 smithi193 ceph-mon[37473]: pgmap v7366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:55:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12500.326490 secs 2024-07-26T14:56:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:59 smithi134 ceph-mon[31966]: pgmap v7366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:55:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12500.326490 secs 2024-07-26T14:56:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:59 smithi189 ceph-mon[36790]: pgmap v7366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:55:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12500.326490 secs 2024-07-26T14:56:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:56:01.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:56:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:56:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:01 smithi134 ceph-mon[31966]: pgmap v7367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12519.750665 secs 2024-07-26T14:56:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:56:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:56:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:56:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:01 smithi189 ceph-mon[36790]: pgmap v7367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12519.750665 secs 2024-07-26T14:56:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:56:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:56:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:56:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:01 smithi193 ceph-mon[37473]: pgmap v7367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12519.750665 secs 2024-07-26T14:56:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:56:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:56:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:56:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:56:03.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:02 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:56:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:02 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:56:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:02 smithi134 ceph-mon[31966]: pgmap v7368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:56:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:02 smithi134 ceph-mon[31966]: pgmap v7369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:56:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:02 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:56:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:02 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:56:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:02 smithi189 ceph-mon[36790]: pgmap v7368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:56:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:02 smithi189 ceph-mon[36790]: pgmap v7369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:56:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:02 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T14:56:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:02 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T14:56:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:02 smithi193 ceph-mon[37473]: pgmap v7368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:56:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:02 smithi193 ceph-mon[37473]: pgmap v7369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T14:56:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12505.326640 secs 2024-07-26T14:56:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:04 smithi134 ceph-mon[31966]: pgmap v7370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12505.326640 secs 2024-07-26T14:56:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:04 smithi189 ceph-mon[36790]: pgmap v7370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12505.326640 secs 2024-07-26T14:56:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:04 smithi193 ceph-mon[37473]: pgmap v7370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:07.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:06 smithi193 ceph-mon[37473]: pgmap v7371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:07.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12524.750790 secs 2024-07-26T14:56:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:06 smithi134 ceph-mon[31966]: pgmap v7371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12524.750790 secs 2024-07-26T14:56:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:06 smithi189 ceph-mon[36790]: pgmap v7371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12524.750790 secs 2024-07-26T14:56:08.422 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:56:08.421+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:56:08.422 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:56:08.421+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:08.422 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:56:08.421+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:56:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:08 smithi134 ceph-mon[31966]: pgmap v7372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:08 smithi189 ceph-mon[36790]: pgmap v7372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:08 smithi193 ceph-mon[37473]: pgmap v7372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T14:56:10.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12510.326800 secs 2024-07-26T14:56:10.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12510.326800 secs 2024-07-26T14:56:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12510.326800 secs 2024-07-26T14:56:11.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:10 smithi134 ceph-mon[31966]: pgmap v7373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:56:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:10 smithi189 ceph-mon[36790]: pgmap v7373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:56:11.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:10 smithi193 ceph-mon[37473]: pgmap v7373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T14:56:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12529.750937 secs 2024-07-26T14:56:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12529.750937 secs 2024-07-26T14:56:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12529.750937 secs 2024-07-26T14:56:12.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:56:12.897+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:56:12.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:56:12.897+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:12.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:56:12.897+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:56:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:12 smithi134 ceph-mon[31966]: pgmap v7374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:56:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:12 smithi189 ceph-mon[36790]: pgmap v7374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:56:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:12 smithi193 ceph-mon[37473]: pgmap v7374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:56:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12515.326926 secs 2024-07-26T14:56:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:14 smithi134 ceph-mon[31966]: pgmap v7375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12515.326926 secs 2024-07-26T14:56:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:14 smithi189 ceph-mon[36790]: pgmap v7375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12515.326926 secs 2024-07-26T14:56:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:14 smithi193 ceph-mon[37473]: pgmap v7375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:17 smithi134 ceph-mon[31966]: pgmap v7376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12534.751073 secs 2024-07-26T14:56:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:17 smithi189 ceph-mon[36790]: pgmap v7376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12534.751073 secs 2024-07-26T14:56:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:17 smithi193 ceph-mon[37473]: pgmap v7376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12534.751073 secs 2024-07-26T14:56:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:19 smithi134 ceph-mon[31966]: pgmap v7377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:19 smithi189 ceph-mon[36790]: pgmap v7377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:19 smithi193 ceph-mon[37473]: pgmap v7377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12520.327055 secs 2024-07-26T14:56:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12520.327055 secs 2024-07-26T14:56:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12520.327055 secs 2024-07-26T14:56:21.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:21 smithi134 ceph-mon[31966]: pgmap v7378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:56:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:21 smithi189 ceph-mon[36790]: pgmap v7378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:56:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:21 smithi193 ceph-mon[37473]: pgmap v7378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:56:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12539.751191 secs 2024-07-26T14:56:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12539.751191 secs 2024-07-26T14:56:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12539.751191 secs 2024-07-26T14:56:23.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:23 smithi134 ceph-mon[31966]: pgmap v7379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:23.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:56:23.432+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:56:23.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:56:23.432+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:23.433 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:56:23.432+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:56:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:23 smithi189 ceph-mon[36790]: pgmap v7379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:23 smithi193 ceph-mon[37473]: pgmap v7379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12525.327177 secs 2024-07-26T14:56:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:25 smithi134 ceph-mon[31966]: pgmap v7380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12525.327177 secs 2024-07-26T14:56:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:25 smithi189 ceph-mon[36790]: pgmap v7380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12525.327177 secs 2024-07-26T14:56:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:25 smithi193 ceph-mon[37473]: pgmap v7380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:27 smithi134 ceph-mon[31966]: pgmap v7381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12544.751340 secs 2024-07-26T14:56:27.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:27 smithi189 ceph-mon[36790]: pgmap v7381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12544.751340 secs 2024-07-26T14:56:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:27 smithi193 ceph-mon[37473]: pgmap v7381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12544.751340 secs 2024-07-26T14:56:27.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:56:27.913+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:56:27.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:56:27.913+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:27.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:56:27.913+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:56:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:29 smithi134 ceph-mon[31966]: pgmap v7382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:29.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:29 smithi189 ceph-mon[36790]: pgmap v7382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:29 smithi193 ceph-mon[37473]: pgmap v7382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:30.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12530.327294 secs 2024-07-26T14:56:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12530.327294 secs 2024-07-26T14:56:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12530.327294 secs 2024-07-26T14:56:31.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:31 smithi134 ceph-mon[31966]: pgmap v7383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:56:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:31 smithi189 ceph-mon[36790]: pgmap v7383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:56:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:31 smithi193 ceph-mon[37473]: pgmap v7383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:56:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12549.751466 secs 2024-07-26T14:56:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12549.751466 secs 2024-07-26T14:56:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12549.751466 secs 2024-07-26T14:56:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:33 smithi134 ceph-mon[31966]: pgmap v7384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:33 smithi189 ceph-mon[36790]: pgmap v7384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:33 smithi193 ceph-mon[37473]: pgmap v7384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12535.327422 secs 2024-07-26T14:56:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:35 smithi134 ceph-mon[31966]: pgmap v7385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12535.327422 secs 2024-07-26T14:56:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:35 smithi189 ceph-mon[36790]: pgmap v7385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12535.327422 secs 2024-07-26T14:56:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:35 smithi193 ceph-mon[37473]: pgmap v7385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:37 smithi134 ceph-mon[31966]: pgmap v7386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12554.751577 secs 2024-07-26T14:56:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:37 smithi189 ceph-mon[36790]: pgmap v7386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12554.751577 secs 2024-07-26T14:56:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:37 smithi193 ceph-mon[37473]: pgmap v7386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12554.751577 secs 2024-07-26T14:56:38.448 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:56:38.448+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:56:38.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:56:38.448+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:56:38.449 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:56:38.448+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:39 smithi134 ceph-mon[31966]: pgmap v7387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:39 smithi189 ceph-mon[36790]: pgmap v7387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:39 smithi193 ceph-mon[37473]: pgmap v7387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12540.327543 secs 2024-07-26T14:56:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12540.327543 secs 2024-07-26T14:56:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12540.327543 secs 2024-07-26T14:56:41.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:41 smithi134 ceph-mon[31966]: pgmap v7388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:56:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:41 smithi189 ceph-mon[36790]: pgmap v7388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:56:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:41 smithi193 ceph-mon[37473]: pgmap v7388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:56:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12559.751691 secs 2024-07-26T14:56:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12559.751691 secs 2024-07-26T14:56:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12559.751691 secs 2024-07-26T14:56:42.921 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:56:42.920+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:56:42.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:56:42.920+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:42.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:56:42.920+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:56:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:43 smithi189 ceph-mon[36790]: pgmap v7389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:43 smithi193 ceph-mon[37473]: pgmap v7389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:43.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:43 smithi134 ceph-mon[31966]: pgmap v7389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12545.327695 secs 2024-07-26T14:56:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:45 smithi189 ceph-mon[36790]: pgmap v7390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12545.327695 secs 2024-07-26T14:56:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:45 smithi193 ceph-mon[37473]: pgmap v7390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12545.327695 secs 2024-07-26T14:56:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:45 smithi134 ceph-mon[31966]: pgmap v7390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:47 smithi189 ceph-mon[36790]: pgmap v7391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12564.751808 secs 2024-07-26T14:56:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:47 smithi193 ceph-mon[37473]: pgmap v7391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12564.751808 secs 2024-07-26T14:56:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:47 smithi134 ceph-mon[31966]: pgmap v7391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12564.751808 secs 2024-07-26T14:56:49.475 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:49 smithi134 ceph-mon[31966]: pgmap v7392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:49 smithi193 ceph-mon[37473]: pgmap v7392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:49 smithi189 ceph-mon[36790]: pgmap v7392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12550.327829 secs 2024-07-26T14:56:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12550.327829 secs 2024-07-26T14:56:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12550.327829 secs 2024-07-26T14:56:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:51 smithi193 ceph-mon[37473]: pgmap v7393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:56:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:51 smithi134 ceph-mon[31966]: pgmap v7393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:56:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:51 smithi189 ceph-mon[36790]: pgmap v7393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:56:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12569.751934 secs 2024-07-26T14:56:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12569.751934 secs 2024-07-26T14:56:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12569.751934 secs 2024-07-26T14:56:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:56:53.457+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:56:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:56:53.457+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:53.457 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:56:53.457+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:56:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:53 smithi193 ceph-mon[37473]: pgmap v7394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:53.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:53 smithi134 ceph-mon[31966]: pgmap v7394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:53.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:53 smithi189 ceph-mon[36790]: pgmap v7394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12555.328001 secs 2024-07-26T14:56:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12555.328001 secs 2024-07-26T14:56:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12555.328001 secs 2024-07-26T14:56:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:55 smithi193 ceph-mon[37473]: pgmap v7395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:55 smithi134 ceph-mon[31966]: pgmap v7395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:55 smithi189 ceph-mon[36790]: pgmap v7395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:56:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:57 smithi193 ceph-mon[37473]: pgmap v7396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12574.752075 secs 2024-07-26T14:56:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:57 smithi134 ceph-mon[31966]: pgmap v7396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12574.752075 secs 2024-07-26T14:56:57.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:57 smithi189 ceph-mon[36790]: pgmap v7396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:56:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12574.752075 secs 2024-07-26T14:56:57.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:56:57.935+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:56:57.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:56:57.935+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:56:57.937 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:56:57.935+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:56:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:59 smithi193 ceph-mon[37473]: pgmap v7397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:56:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:56:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12560.328100 secs 2024-07-26T14:56:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:59 smithi134 ceph-mon[31966]: pgmap v7397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:56:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:56:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12560.328100 secs 2024-07-26T14:56:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:59 smithi189 ceph-mon[36790]: pgmap v7397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T14:56:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:56:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12560.328100 secs 2024-07-26T14:57:01.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:01 smithi193 ceph-mon[37473]: pgmap v7398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:57:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:01 smithi134 ceph-mon[31966]: pgmap v7398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:57:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:01 smithi189 ceph-mon[36790]: pgmap v7398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T14:57:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12579.752214 secs 2024-07-26T14:57:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:57:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12579.752214 secs 2024-07-26T14:57:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:57:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12579.752214 secs 2024-07-26T14:57:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:57:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:03 smithi134 ceph-mon[31966]: pgmap v7399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:57:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:03 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:57:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:03 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:57:03.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:03 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:57:03.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:03 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:57:03.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:57:03.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:57:03.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:57:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:03 smithi189 ceph-mon[36790]: pgmap v7399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:57:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:03 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:57:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:03 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:57:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:03 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:57:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:03 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:57:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:57:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:57:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:57:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:03 smithi193 ceph-mon[37473]: pgmap v7399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:57:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T14:57:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:03 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T14:57:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:03 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T14:57:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:03 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T14:57:03.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:03 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T14:57:03.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:57:03.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:57:03.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:57:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:04 smithi134 ceph-mon[31966]: pgmap v7400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T14:57:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:04 smithi134 ceph-mon[31966]: pgmap v7401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:57:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12565.328172 secs 2024-07-26T14:57:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:04 smithi189 ceph-mon[36790]: pgmap v7400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T14:57:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:04 smithi189 ceph-mon[36790]: pgmap v7401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:57:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12565.328172 secs 2024-07-26T14:57:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:04 smithi193 ceph-mon[37473]: pgmap v7400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T14:57:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:04 smithi193 ceph-mon[37473]: pgmap v7401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T14:57:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12565.328172 secs 2024-07-26T14:57:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:06 smithi134 ceph-mon[31966]: pgmap v7402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:57:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:06 smithi189 ceph-mon[36790]: pgmap v7402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:57:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:06 smithi193 ceph-mon[37473]: pgmap v7402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T14:57:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12584.752354 secs 2024-07-26T14:57:07.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12584.752354 secs 2024-07-26T14:57:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12584.752354 secs 2024-07-26T14:57:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:08.464+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:57:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:08.464+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:08.465 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:08.464+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:57:08.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:08 smithi134 ceph-mon[31966]: pgmap v7403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:57:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:08 smithi189 ceph-mon[36790]: pgmap v7403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:57:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:08 smithi193 ceph-mon[37473]: pgmap v7403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:57:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12570.328288 secs 2024-07-26T14:57:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12570.328288 secs 2024-07-26T14:57:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12570.328288 secs 2024-07-26T14:57:10.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:10 smithi189 ceph-mon[36790]: pgmap v7404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:57:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:10 smithi134 ceph-mon[31966]: pgmap v7404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:57:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:10 smithi193 ceph-mon[37473]: pgmap v7404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:57:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:12 smithi134 ceph-mon[31966]: pgmap v7405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:57:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12589.752468 secs 2024-07-26T14:57:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:12 smithi189 ceph-mon[36790]: pgmap v7405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:57:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12589.752468 secs 2024-07-26T14:57:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:12 smithi193 ceph-mon[37473]: pgmap v7405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T14:57:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12589.752468 secs 2024-07-26T14:57:12.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:12.950+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:57:12.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:12.950+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:12.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:12.951+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:57:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:14 smithi134 ceph-mon[31966]: pgmap v7406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:57:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12575.328382 secs 2024-07-26T14:57:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:14 smithi189 ceph-mon[36790]: pgmap v7406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:57:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12575.328382 secs 2024-07-26T14:57:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:14 smithi193 ceph-mon[37473]: pgmap v7406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:57:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12575.328382 secs 2024-07-26T14:57:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:16 smithi134 ceph-mon[31966]: pgmap v7407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:16 smithi189 ceph-mon[36790]: pgmap v7407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:16 smithi193 ceph-mon[37473]: pgmap v7407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12594.752559 secs 2024-07-26T14:57:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12594.752559 secs 2024-07-26T14:57:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12594.752559 secs 2024-07-26T14:57:18.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:18 smithi134 ceph-mon[31966]: pgmap v7408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:18 smithi189 ceph-mon[36790]: pgmap v7408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:18 smithi193 ceph-mon[37473]: pgmap v7408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12580.328505 secs 2024-07-26T14:57:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12580.328505 secs 2024-07-26T14:57:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12580.328505 secs 2024-07-26T14:57:20.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:20 smithi134 ceph-mon[31966]: pgmap v7409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:20 smithi189 ceph-mon[36790]: pgmap v7409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:20 smithi193 ceph-mon[37473]: pgmap v7409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:22.328+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T14:57:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:22.328+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T14:57:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:22.328+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b3805d040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:57:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:22.328+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b3805d040 comp rx=0 tx=0).stop 2024-07-26T14:57:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:22.328+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T14:57:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:22.328+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:57:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:22.328+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:57:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:22.328+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 0x7f9b300c4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:57:22.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:22.329+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 0x7f9b300c4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:57:22.332 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:22.331+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 0x7f9b300c4130 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009990 tx=0x7f9b38008070 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:57:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:22 smithi134 ceph-mon[31966]: pgmap v7410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12599.752680 secs 2024-07-26T14:57:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:22 smithi189 ceph-mon[36790]: pgmap v7410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12599.752680 secs 2024-07-26T14:57:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:22 smithi193 ceph-mon[37473]: pgmap v7410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12599.752680 secs 2024-07-26T14:57:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:23.473+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:57:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:23.473+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:23.474 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:23.473+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:57:24.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:24.521+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T14:57:24.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:24.521+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T14:57:24.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:24.521+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 99 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T14:57:24.522 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:24.521+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T14:57:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:24 smithi134 ceph-mon[31966]: pgmap v7411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12585.328627 secs 2024-07-26T14:57:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:24 smithi189 ceph-mon[36790]: pgmap v7411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12585.328627 secs 2024-07-26T14:57:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:24 smithi193 ceph-mon[37473]: pgmap v7411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12585.328627 secs 2024-07-26T14:57:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:26 smithi134 ceph-mon[31966]: pgmap v7412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:26 smithi189 ceph-mon[36790]: pgmap v7412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:26 smithi193 ceph-mon[37473]: pgmap v7412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12604.752792 secs 2024-07-26T14:57:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12604.752792 secs 2024-07-26T14:57:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12604.752792 secs 2024-07-26T14:57:27.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:27.957+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:57:27.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:27.957+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:27.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:27.957+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:57:28.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:28 smithi134 ceph-mon[31966]: pgmap v7413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:28 smithi189 ceph-mon[36790]: pgmap v7413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:28 smithi193 ceph-mon[37473]: pgmap v7413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12590.328748 secs 2024-07-26T14:57:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12590.328748 secs 2024-07-26T14:57:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12590.328748 secs 2024-07-26T14:57:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:30 smithi134 ceph-mon[31966]: pgmap v7414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:30.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:30 smithi189 ceph-mon[36790]: pgmap v7414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:30 smithi193 ceph-mon[37473]: pgmap v7414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:32 smithi134 ceph-mon[31966]: pgmap v7415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12609.752890 secs 2024-07-26T14:57:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:32 smithi189 ceph-mon[36790]: pgmap v7415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12609.752890 secs 2024-07-26T14:57:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:32 smithi193 ceph-mon[37473]: pgmap v7415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12609.752890 secs 2024-07-26T14:57:32.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:32.882+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 msgr2=0x7f21ac0c14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T14:57:32.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:32.882+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 msgr2=0x7f21ac0c14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T14:57:32.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:32.882+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f2198003490 tx=0x7f21b8070000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T14:57:32.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:32.882+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f2198003490 tx=0x7f21b8070000 comp rx=0 tx=0).stop 2024-07-26T14:57:32.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:32.882+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 msgr2=0x7f21ac0c14c0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T14:57:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:32.882+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 msgr2=0x7f21ac0c14c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T14:57:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:32.882+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c10b0 0x7f21ac0c14c0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T14:57:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:32.882+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T14:57:32.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:32.883+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T14:57:32.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:32.884+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f21b8031850 tx=0x7f21b806c000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T14:57:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:34 smithi189 ceph-mon[36790]: pgmap v7416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12595.328870 secs 2024-07-26T14:57:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:34 smithi193 ceph-mon[37473]: pgmap v7416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12595.328870 secs 2024-07-26T14:57:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:34 smithi134 ceph-mon[31966]: pgmap v7416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12595.328870 secs 2024-07-26T14:57:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:35.053+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:57:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:35.053+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:57:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:35.054+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T14:57:35.055 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:35.054+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T14:57:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:36 smithi189 ceph-mon[36790]: pgmap v7417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:36 smithi193 ceph-mon[37473]: pgmap v7417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:36 smithi134 ceph-mon[31966]: pgmap v7417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:57:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12614.753007 secs 2024-07-26T14:57:37.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12614.753007 secs 2024-07-26T14:57:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12614.753007 secs 2024-07-26T14:57:38.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:38.480+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:57:38.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:38.480+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:38.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:38.480+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:57:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:38 smithi193 ceph-mon[37473]: pgmap v7418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:38 smithi134 ceph-mon[31966]: pgmap v7418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:38 smithi189 ceph-mon[36790]: pgmap v7418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12600.329006 secs 2024-07-26T14:57:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12600.329006 secs 2024-07-26T14:57:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12600.329006 secs 2024-07-26T14:57:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:40 smithi193 ceph-mon[37473]: pgmap v7419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:40.846 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:40 smithi189 ceph-mon[36790]: pgmap v7419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:40 smithi134 ceph-mon[31966]: pgmap v7419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:42 smithi193 ceph-mon[37473]: pgmap v7420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12619.753125 secs 2024-07-26T14:57:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:42 smithi134 ceph-mon[31966]: pgmap v7420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12619.753125 secs 2024-07-26T14:57:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:42 smithi189 ceph-mon[36790]: pgmap v7420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12619.753125 secs 2024-07-26T14:57:42.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:42.972+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:57:42.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:42.972+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:42.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:42.972+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:57:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:44 smithi193 ceph-mon[37473]: pgmap v7421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12605.329131 secs 2024-07-26T14:57:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:44 smithi134 ceph-mon[31966]: pgmap v7421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12605.329131 secs 2024-07-26T14:57:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:44 smithi189 ceph-mon[36790]: pgmap v7421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12605.329131 secs 2024-07-26T14:57:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:46 smithi193 ceph-mon[37473]: pgmap v7422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:46 smithi134 ceph-mon[31966]: pgmap v7422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:46 smithi189 ceph-mon[36790]: pgmap v7422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12624.753241 secs 2024-07-26T14:57:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12624.753241 secs 2024-07-26T14:57:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12624.753241 secs 2024-07-26T14:57:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:48 smithi193 ceph-mon[37473]: pgmap v7423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:48 smithi134 ceph-mon[31966]: pgmap v7423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:48 smithi189 ceph-mon[36790]: pgmap v7423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12610.329265 secs 2024-07-26T14:57:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12610.329265 secs 2024-07-26T14:57:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12610.329265 secs 2024-07-26T14:57:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:50 smithi193 ceph-mon[37473]: pgmap v7424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:50 smithi134 ceph-mon[31966]: pgmap v7424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:50 smithi189 ceph-mon[36790]: pgmap v7424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:52 smithi193 ceph-mon[37473]: pgmap v7425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12629.753358 secs 2024-07-26T14:57:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:52 smithi134 ceph-mon[31966]: pgmap v7425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12629.753358 secs 2024-07-26T14:57:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:52 smithi189 ceph-mon[36790]: pgmap v7425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12629.753358 secs 2024-07-26T14:57:53.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:53.495+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:57:53.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:53.496+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:53.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:57:53.496+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:57:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:54 smithi193 ceph-mon[37473]: pgmap v7426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12615.329391 secs 2024-07-26T14:57:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:54 smithi134 ceph-mon[31966]: pgmap v7426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12615.329391 secs 2024-07-26T14:57:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:54 smithi189 ceph-mon[36790]: pgmap v7426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:57:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12615.329391 secs 2024-07-26T14:57:56.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:56 smithi134 ceph-mon[31966]: pgmap v7427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:56 smithi189 ceph-mon[36790]: pgmap v7427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:56 smithi193 ceph-mon[37473]: pgmap v7427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:57:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12634.753475 secs 2024-07-26T14:57:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12634.753475 secs 2024-07-26T14:57:57.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:57.988+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:57:57.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:57.988+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:57:57.990 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:57:57.988+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:57:58.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12634.753475 secs 2024-07-26T14:57:58.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:58 smithi134 ceph-mon[31966]: pgmap v7428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:58 smithi189 ceph-mon[36790]: pgmap v7428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:58 smithi193 ceph-mon[37473]: pgmap v7428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:57:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12620.329456 secs 2024-07-26T14:57:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:57:59 smithi134 ceph-mon[31966]: pgmap v7429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:57:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12620.329456 secs 2024-07-26T14:57:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:57:59 smithi189 ceph-mon[36790]: pgmap v7429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12620.329456 secs 2024-07-26T14:58:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:57:59 smithi193 ceph-mon[37473]: pgmap v7429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:02 smithi134 ceph-mon[31966]: pgmap v7430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12639.753610 secs 2024-07-26T14:58:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:02 smithi189 ceph-mon[36790]: pgmap v7430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12639.753610 secs 2024-07-26T14:58:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:02 smithi193 ceph-mon[37473]: pgmap v7430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12639.753610 secs 2024-07-26T14:58:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:04 smithi134 ceph-mon[31966]: pgmap v7431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:58:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:58:04.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:04 smithi189 ceph-mon[36790]: pgmap v7431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:58:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:58:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:04 smithi193 ceph-mon[37473]: pgmap v7431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T14:58:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:58:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12625.329548 secs 2024-07-26T14:58:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:58:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:58:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:05 smithi134 ceph-mon[31966]: pgmap v7432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T14:58:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:05 smithi134 ceph-mon[31966]: pgmap v7433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:58:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:58:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12625.329548 secs 2024-07-26T14:58:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:58:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:58:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:05 smithi189 ceph-mon[36790]: pgmap v7432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T14:58:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:05 smithi189 ceph-mon[36790]: pgmap v7433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:58:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:58:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12625.329548 secs 2024-07-26T14:58:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:58:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:58:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:05 smithi193 ceph-mon[37473]: pgmap v7432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T14:58:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:05 smithi193 ceph-mon[37473]: pgmap v7433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:58:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:58:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12644.753721 secs 2024-07-26T14:58:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:07 smithi193 ceph-mon[37473]: pgmap v7434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:58:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12644.753721 secs 2024-07-26T14:58:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:07 smithi134 ceph-mon[31966]: pgmap v7434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:58:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12644.753721 secs 2024-07-26T14:58:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:07 smithi189 ceph-mon[36790]: pgmap v7434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T14:58:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:58:08.511+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:58:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:58:08.511+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:08.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:58:08.511+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:58:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:09 smithi193 ceph-mon[37473]: pgmap v7435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:58:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12630.329634 secs 2024-07-26T14:58:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:09 smithi134 ceph-mon[31966]: pgmap v7435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:58:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12630.329634 secs 2024-07-26T14:58:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:09 smithi189 ceph-mon[36790]: pgmap v7435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:58:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12630.329634 secs 2024-07-26T14:58:11.495 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:11 smithi189 ceph-mon[36790]: pgmap v7436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:58:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:11 smithi193 ceph-mon[37473]: pgmap v7436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:58:11.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:11 smithi134 ceph-mon[31966]: pgmap v7436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:58:12.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12649.753829 secs 2024-07-26T14:58:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12649.753829 secs 2024-07-26T14:58:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12649.753829 secs 2024-07-26T14:58:12.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:58:12.991+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:58:12.992 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:58:12.991+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:12.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:58:12.991+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:58:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:13 smithi193 ceph-mon[37473]: pgmap v7437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:58:13.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:13 smithi134 ceph-mon[31966]: pgmap v7437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:58:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:13 smithi189 ceph-mon[36790]: pgmap v7437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T14:58:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12635.329767 secs 2024-07-26T14:58:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12635.329767 secs 2024-07-26T14:58:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12635.329767 secs 2024-07-26T14:58:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:15 smithi193 ceph-mon[37473]: pgmap v7438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:58:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:15 smithi134 ceph-mon[31966]: pgmap v7438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:58:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:15 smithi189 ceph-mon[36790]: pgmap v7438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:58:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12654.753970 secs 2024-07-26T14:58:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:17 smithi193 ceph-mon[37473]: pgmap v7439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12654.753970 secs 2024-07-26T14:58:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:17 smithi134 ceph-mon[31966]: pgmap v7439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12654.753970 secs 2024-07-26T14:58:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:17 smithi189 ceph-mon[36790]: pgmap v7439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:19 smithi193 ceph-mon[37473]: pgmap v7440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12640.329903 secs 2024-07-26T14:58:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:19 smithi134 ceph-mon[31966]: pgmap v7440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12640.329903 secs 2024-07-26T14:58:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:19 smithi189 ceph-mon[36790]: pgmap v7440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12640.329903 secs 2024-07-26T14:58:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:21 smithi134 ceph-mon[31966]: pgmap v7441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:21 smithi189 ceph-mon[36790]: pgmap v7441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:21 smithi193 ceph-mon[37473]: pgmap v7441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12659.754062 secs 2024-07-26T14:58:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12659.754062 secs 2024-07-26T14:58:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12659.754062 secs 2024-07-26T14:58:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:58:23.527+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:58:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:58:23.527+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:58:23.528 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:58:23.527+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:23 smithi134 ceph-mon[31966]: pgmap v7442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:23 smithi189 ceph-mon[36790]: pgmap v7442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:23 smithi193 ceph-mon[37473]: pgmap v7442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12645.330025 secs 2024-07-26T14:58:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12645.330025 secs 2024-07-26T14:58:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12645.330025 secs 2024-07-26T14:58:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:25 smithi134 ceph-mon[31966]: pgmap v7443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:25 smithi189 ceph-mon[36790]: pgmap v7443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:25 smithi193 ceph-mon[37473]: pgmap v7443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12664.754135 secs 2024-07-26T14:58:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:27 smithi134 ceph-mon[31966]: pgmap v7444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12664.754135 secs 2024-07-26T14:58:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:27 smithi189 ceph-mon[36790]: pgmap v7444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12664.754135 secs 2024-07-26T14:58:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:27 smithi193 ceph-mon[37473]: pgmap v7444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:28.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:58:28.006+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:58:28.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:58:28.006+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:28.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:58:28.006+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:58:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:29 smithi134 ceph-mon[31966]: pgmap v7445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12650.330136 secs 2024-07-26T14:58:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:29 smithi189 ceph-mon[36790]: pgmap v7445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12650.330136 secs 2024-07-26T14:58:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:29 smithi193 ceph-mon[37473]: pgmap v7445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:29.793 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12650.330136 secs 2024-07-26T14:58:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:31 smithi134 ceph-mon[31966]: pgmap v7446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:31.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:31 smithi189 ceph-mon[36790]: pgmap v7446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:31 smithi193 ceph-mon[37473]: pgmap v7446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12669.754256 secs 2024-07-26T14:58:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12669.754256 secs 2024-07-26T14:58:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12669.754256 secs 2024-07-26T14:58:33.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:33 smithi134 ceph-mon[31966]: pgmap v7447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:33 smithi189 ceph-mon[36790]: pgmap v7447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:33 smithi193 ceph-mon[37473]: pgmap v7447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12655.330243 secs 2024-07-26T14:58:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12655.330243 secs 2024-07-26T14:58:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12655.330243 secs 2024-07-26T14:58:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:35 smithi134 ceph-mon[31966]: pgmap v7448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:35 smithi189 ceph-mon[36790]: pgmap v7448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:35 smithi193 ceph-mon[37473]: pgmap v7448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12674.754377 secs 2024-07-26T14:58:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:37 smithi134 ceph-mon[31966]: pgmap v7449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12674.754377 secs 2024-07-26T14:58:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:37 smithi189 ceph-mon[36790]: pgmap v7449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12674.754377 secs 2024-07-26T14:58:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:37 smithi193 ceph-mon[37473]: pgmap v7449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:38.543 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:58:38.542+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:58:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:58:38.542+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:58:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:58:38.542+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:39 smithi134 ceph-mon[31966]: pgmap v7450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12660.330339 secs 2024-07-26T14:58:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:39 smithi189 ceph-mon[36790]: pgmap v7450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12660.330339 secs 2024-07-26T14:58:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:39 smithi193 ceph-mon[37473]: pgmap v7450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12660.330339 secs 2024-07-26T14:58:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:41 smithi134 ceph-mon[31966]: pgmap v7451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:41.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:41 smithi189 ceph-mon[36790]: pgmap v7451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:41 smithi193 ceph-mon[37473]: pgmap v7451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12679.754486 secs 2024-07-26T14:58:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12679.754486 secs 2024-07-26T14:58:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12679.754486 secs 2024-07-26T14:58:43.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:58:43.022+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:58:43.023 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:58:43.022+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:43.024 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:58:43.022+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:58:43.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:43 smithi134 ceph-mon[31966]: pgmap v7452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:43 smithi189 ceph-mon[36790]: pgmap v7452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:43 smithi193 ceph-mon[37473]: pgmap v7452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12665.330427 secs 2024-07-26T14:58:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12665.330427 secs 2024-07-26T14:58:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12665.330427 secs 2024-07-26T14:58:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:45 smithi134 ceph-mon[31966]: pgmap v7453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:45 smithi189 ceph-mon[36790]: pgmap v7453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:45 smithi193 ceph-mon[37473]: pgmap v7453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12684.754602 secs 2024-07-26T14:58:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:47 smithi134 ceph-mon[31966]: pgmap v7454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12684.754602 secs 2024-07-26T14:58:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:47 smithi189 ceph-mon[36790]: pgmap v7454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12684.754602 secs 2024-07-26T14:58:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:47 smithi193 ceph-mon[37473]: pgmap v7454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:49 smithi134 ceph-mon[31966]: pgmap v7455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12670.330548 secs 2024-07-26T14:58:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:49 smithi189 ceph-mon[36790]: pgmap v7455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12670.330548 secs 2024-07-26T14:58:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:49 smithi193 ceph-mon[37473]: pgmap v7455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:58:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12670.330548 secs 2024-07-26T14:58:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:51 smithi134 ceph-mon[31966]: pgmap v7456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:51 smithi189 ceph-mon[36790]: pgmap v7456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:51 smithi193 ceph-mon[37473]: pgmap v7456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12689.754716 secs 2024-07-26T14:58:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12689.754716 secs 2024-07-26T14:58:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12689.754716 secs 2024-07-26T14:58:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:58:53.558+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:58:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:58:53.558+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:53.559 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:58:53.558+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:58:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:53 smithi189 ceph-mon[36790]: pgmap v7457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:53 smithi193 ceph-mon[37473]: pgmap v7457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:53.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:53 smithi134 ceph-mon[31966]: pgmap v7457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:58:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12675.330677 secs 2024-07-26T14:58:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12675.330677 secs 2024-07-26T14:58:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12675.330677 secs 2024-07-26T14:58:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:55 smithi193 ceph-mon[37473]: pgmap v7458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:55.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:55 smithi134 ceph-mon[31966]: pgmap v7458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:55 smithi189 ceph-mon[36790]: pgmap v7458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:58:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12694.754852 secs 2024-07-26T14:58:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:57 smithi193 ceph-mon[37473]: pgmap v7459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12694.754852 secs 2024-07-26T14:58:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:57 smithi134 ceph-mon[31966]: pgmap v7459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:57.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12694.754852 secs 2024-07-26T14:58:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:57 smithi189 ceph-mon[36790]: pgmap v7459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:58:58.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:58:58.037+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:58:58.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:58:58.037+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:58:58.039 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:58:58.037+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:58:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:59 smithi193 ceph-mon[37473]: pgmap v7460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:58:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:58:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12680.330826 secs 2024-07-26T14:58:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:59 smithi134 ceph-mon[31966]: pgmap v7460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:58:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:58:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12680.330826 secs 2024-07-26T14:58:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:59 smithi189 ceph-mon[36790]: pgmap v7460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T14:58:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:58:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12680.330826 secs 2024-07-26T14:59:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:01 smithi193 ceph-mon[37473]: pgmap v7461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:59:01.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:01 smithi134 ceph-mon[31966]: pgmap v7461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:59:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:01 smithi189 ceph-mon[36790]: pgmap v7461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T14:59:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12699.754973 secs 2024-07-26T14:59:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12699.754973 secs 2024-07-26T14:59:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12699.754973 secs 2024-07-26T14:59:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:03 smithi193 ceph-mon[37473]: pgmap v7462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:59:03.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:03 smithi134 ceph-mon[31966]: pgmap v7462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:59:03.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:03 smithi189 ceph-mon[36790]: pgmap v7462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T14:59:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12685.330984 secs 2024-07-26T14:59:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:59:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12685.330984 secs 2024-07-26T14:59:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:59:04.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12685.330984 secs 2024-07-26T14:59:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T14:59:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:05 smithi193 ceph-mon[37473]: pgmap v7463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:59:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:59:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:59:05.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:05 smithi134 ceph-mon[31966]: pgmap v7463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:59:05.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:59:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:59:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:05 smithi189 ceph-mon[36790]: pgmap v7463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T14:59:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:59:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:59:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:59:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:59:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:59:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:59:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:59:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:59:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T14:59:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T14:59:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T14:59:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:07 smithi193 ceph-mon[37473]: pgmap v7464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:07 smithi193 ceph-mon[37473]: pgmap v7465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:59:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12704.755078 secs 2024-07-26T14:59:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:07 smithi134 ceph-mon[31966]: pgmap v7464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:07 smithi134 ceph-mon[31966]: pgmap v7465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:59:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12704.755078 secs 2024-07-26T14:59:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:07 smithi189 ceph-mon[36790]: pgmap v7464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:07 smithi189 ceph-mon[36790]: pgmap v7465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:59:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12704.755078 secs 2024-07-26T14:59:08.574 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:59:08.573+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:59:08.575 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:59:08.573+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:08.575 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:59:08.573+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:59:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:09 smithi134 ceph-mon[31966]: pgmap v7466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:59:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12690.331079 secs 2024-07-26T14:59:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:09 smithi189 ceph-mon[36790]: pgmap v7466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:59:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12690.331079 secs 2024-07-26T14:59:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:09 smithi193 ceph-mon[37473]: pgmap v7466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:59:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12690.331079 secs 2024-07-26T14:59:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:11 smithi134 ceph-mon[31966]: pgmap v7467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:59:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:11 smithi189 ceph-mon[36790]: pgmap v7467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:59:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:11 smithi193 ceph-mon[37473]: pgmap v7467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T14:59:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12709.755188 secs 2024-07-26T14:59:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12709.755188 secs 2024-07-26T14:59:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12709.755188 secs 2024-07-26T14:59:13.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:59:13.053+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:59:13.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:59:13.053+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:59:13.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:59:13.053+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:13.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:13 smithi134 ceph-mon[31966]: pgmap v7468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:59:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:13 smithi189 ceph-mon[36790]: pgmap v7468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:59:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:13 smithi193 ceph-mon[37473]: pgmap v7468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:59:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12695.331172 secs 2024-07-26T14:59:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:14 smithi134 ceph-mon[31966]: pgmap v7469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:59:14.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12695.331172 secs 2024-07-26T14:59:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:14 smithi189 ceph-mon[36790]: pgmap v7469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:59:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12695.331172 secs 2024-07-26T14:59:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:14 smithi193 ceph-mon[37473]: pgmap v7469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T14:59:17.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:17 smithi134 ceph-mon[31966]: pgmap v7470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:59:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12714.755324 secs 2024-07-26T14:59:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:17 smithi189 ceph-mon[36790]: pgmap v7470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:59:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12714.755324 secs 2024-07-26T14:59:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:17 smithi193 ceph-mon[37473]: pgmap v7470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T14:59:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12714.755324 secs 2024-07-26T14:59:19.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:19 smithi134 ceph-mon[31966]: pgmap v7471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:19 smithi189 ceph-mon[36790]: pgmap v7471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:19 smithi193 ceph-mon[37473]: pgmap v7471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:20.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12700.331236 secs 2024-07-26T14:59:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12700.331236 secs 2024-07-26T14:59:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12700.331236 secs 2024-07-26T14:59:21.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:21 smithi134 ceph-mon[31966]: pgmap v7472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:21 smithi189 ceph-mon[36790]: pgmap v7472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:21 smithi193 ceph-mon[37473]: pgmap v7472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12719.755382 secs 2024-07-26T14:59:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12719.755382 secs 2024-07-26T14:59:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12719.755382 secs 2024-07-26T14:59:23.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:23 smithi134 ceph-mon[31966]: pgmap v7473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:23 smithi189 ceph-mon[36790]: pgmap v7473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:23 smithi193 ceph-mon[37473]: pgmap v7473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:23.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:59:23.585+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:59:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:59:23.586+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:59:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:59:23.586+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12705.331308 secs 2024-07-26T14:59:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:25 smithi134 ceph-mon[31966]: pgmap v7474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12705.331308 secs 2024-07-26T14:59:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:25 smithi189 ceph-mon[36790]: pgmap v7474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12705.331308 secs 2024-07-26T14:59:25.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:25 smithi193 ceph-mon[37473]: pgmap v7474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:27 smithi134 ceph-mon[31966]: pgmap v7475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12724.755462 secs 2024-07-26T14:59:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:27 smithi189 ceph-mon[36790]: pgmap v7475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12724.755462 secs 2024-07-26T14:59:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:27 smithi193 ceph-mon[37473]: pgmap v7475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12724.755462 secs 2024-07-26T14:59:28.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:59:28.068+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:59:28.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:59:28.068+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:28.070 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:59:28.068+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:59:29.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:29 smithi134 ceph-mon[31966]: pgmap v7476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:29 smithi189 ceph-mon[36790]: pgmap v7476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:29 smithi193 ceph-mon[37473]: pgmap v7476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:30.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12710.331407 secs 2024-07-26T14:59:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12710.331407 secs 2024-07-26T14:59:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12710.331407 secs 2024-07-26T14:59:31.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:31 smithi134 ceph-mon[31966]: pgmap v7477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:31 smithi189 ceph-mon[36790]: pgmap v7477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:31 smithi193 ceph-mon[37473]: pgmap v7477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12729.755528 secs 2024-07-26T14:59:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12729.755528 secs 2024-07-26T14:59:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12729.755528 secs 2024-07-26T14:59:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:33 smithi134 ceph-mon[31966]: pgmap v7478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:33.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:33 smithi189 ceph-mon[36790]: pgmap v7478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:33 smithi193 ceph-mon[37473]: pgmap v7478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12715.331555 secs 2024-07-26T14:59:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:35 smithi189 ceph-mon[36790]: pgmap v7479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12715.331555 secs 2024-07-26T14:59:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:35 smithi193 ceph-mon[37473]: pgmap v7479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12715.331555 secs 2024-07-26T14:59:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:35 smithi134 ceph-mon[31966]: pgmap v7479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:37 smithi189 ceph-mon[36790]: pgmap v7480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12734.755649 secs 2024-07-26T14:59:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:37 smithi193 ceph-mon[37473]: pgmap v7480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12734.755649 secs 2024-07-26T14:59:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:37 smithi134 ceph-mon[31966]: pgmap v7480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12734.755649 secs 2024-07-26T14:59:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:59:38.596+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:59:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:59:38.596+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:38.597 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:59:38.596+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:59:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:39 smithi189 ceph-mon[36790]: pgmap v7481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:39 smithi193 ceph-mon[37473]: pgmap v7481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:39 smithi134 ceph-mon[31966]: pgmap v7481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12720.331702 secs 2024-07-26T14:59:40.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12720.331702 secs 2024-07-26T14:59:40.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12720.331702 secs 2024-07-26T14:59:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:41 smithi193 ceph-mon[37473]: pgmap v7482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:59:41.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:41 smithi189 ceph-mon[36790]: pgmap v7482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:59:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:41 smithi134 ceph-mon[31966]: pgmap v7482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:59:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12739.755728 secs 2024-07-26T14:59:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12739.755728 secs 2024-07-26T14:59:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12739.755728 secs 2024-07-26T14:59:43.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:59:43.084+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:59:43.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:59:43.084+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:43.085 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:59:43.084+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:59:43.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:43 smithi134 ceph-mon[31966]: pgmap v7483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:43 smithi193 ceph-mon[37473]: pgmap v7483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:43 smithi189 ceph-mon[36790]: pgmap v7483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12725.331851 secs 2024-07-26T14:59:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:45 smithi193 ceph-mon[37473]: pgmap v7484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12725.331851 secs 2024-07-26T14:59:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:45 smithi134 ceph-mon[31966]: pgmap v7484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:45.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12725.331851 secs 2024-07-26T14:59:45.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:45 smithi189 ceph-mon[36790]: pgmap v7484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:47 smithi193 ceph-mon[37473]: pgmap v7485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12744.755840 secs 2024-07-26T14:59:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:47 smithi134 ceph-mon[31966]: pgmap v7485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12744.755840 secs 2024-07-26T14:59:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:47 smithi189 ceph-mon[36790]: pgmap v7485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12744.755840 secs 2024-07-26T14:59:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:49 smithi193 ceph-mon[37473]: pgmap v7486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12730.331977 secs 2024-07-26T14:59:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:49 smithi134 ceph-mon[31966]: pgmap v7486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12730.331977 secs 2024-07-26T14:59:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:49 smithi189 ceph-mon[36790]: pgmap v7486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12730.331977 secs 2024-07-26T14:59:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:51 smithi193 ceph-mon[37473]: pgmap v7487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:59:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:51 smithi134 ceph-mon[31966]: pgmap v7487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:59:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:51 smithi189 ceph-mon[36790]: pgmap v7487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T14:59:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12749.755967 secs 2024-07-26T14:59:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12749.755967 secs 2024-07-26T14:59:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12749.755967 secs 2024-07-26T14:59:53.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:53 smithi134 ceph-mon[31966]: pgmap v7488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:53 smithi193 ceph-mon[37473]: pgmap v7488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:53.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:59:53.609+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:59:53.611 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:59:53.609+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:59:53.611 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T14:59:53.609+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:53 smithi189 ceph-mon[36790]: pgmap v7488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T14:59:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12735.332046 secs 2024-07-26T14:59:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12735.332046 secs 2024-07-26T14:59:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12735.332046 secs 2024-07-26T14:59:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:55 smithi193 ceph-mon[37473]: pgmap v7489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:55 smithi134 ceph-mon[31966]: pgmap v7489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:55 smithi189 ceph-mon[36790]: pgmap v7489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T14:59:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:57 smithi193 ceph-mon[37473]: pgmap v7490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12754.756077 secs 2024-07-26T14:59:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:57 smithi134 ceph-mon[31966]: pgmap v7490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12754.756077 secs 2024-07-26T14:59:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:57 smithi189 ceph-mon[36790]: pgmap v7490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T14:59:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12754.756077 secs 2024-07-26T14:59:58.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:59:58.099+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T14:59:58.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:59:58.099+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T14:59:58.100 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T14:59:58.099+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T14:59:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:59 smithi193 ceph-mon[37473]: pgmap v7491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 14:59:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12740.332185 secs 2024-07-26T14:59:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:59 smithi134 ceph-mon[31966]: pgmap v7491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 14:59:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12740.332185 secs 2024-07-26T14:59:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:59 smithi189 ceph-mon[36790]: pgmap v7491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T14:59:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 14:59:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12740.332185 secs 2024-07-26T15:00:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:00:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:00:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:00:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:01 smithi134 ceph-mon[31966]: pgmap v7492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:01.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:01 smithi189 ceph-mon[36790]: pgmap v7492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:01 smithi193 ceph-mon[37473]: pgmap v7492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12759.756214 secs 2024-07-26T15:00:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12759.756214 secs 2024-07-26T15:00:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12759.756214 secs 2024-07-26T15:00:03.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:03 smithi134 ceph-mon[31966]: pgmap v7493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:03 smithi189 ceph-mon[36790]: pgmap v7493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:03 smithi193 ceph-mon[37473]: pgmap v7493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12745.332298 secs 2024-07-26T15:00:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12745.332298 secs 2024-07-26T15:00:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12745.332298 secs 2024-07-26T15:00:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:05 smithi134 ceph-mon[31966]: pgmap v7494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:05 smithi189 ceph-mon[36790]: pgmap v7494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:05 smithi193 ceph-mon[37473]: pgmap v7494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:06.573 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:00:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:00:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:00:07.522 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:07 smithi189 ceph-mon[36790]: pgmap v7495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:07.522 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12764.756353 secs 2024-07-26T15:00:07.522 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:07.523 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:07.523 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:07.523 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:07 smithi134 ceph-mon[31966]: pgmap v7495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12764.756353 secs 2024-07-26T15:00:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:07 smithi193 ceph-mon[37473]: pgmap v7495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12764.756353 secs 2024-07-26T15:00:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:07.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:07.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:07.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:08.626 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:00:08.625+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:00:08.626 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:00:08.625+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:00:08.626 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:00:08.625+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:09.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:09 smithi134 ceph-mon[31966]: pgmap v7496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:00:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:09 smithi189 ceph-mon[36790]: pgmap v7496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:00:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:09 smithi193 ceph-mon[37473]: pgmap v7496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:00:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:10.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12750.332380 secs 2024-07-26T15:00:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12750.332380 secs 2024-07-26T15:00:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12750.332380 secs 2024-07-26T15:00:11.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:11 smithi189 ceph-mon[36790]: pgmap v7497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:11.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:11.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:11 smithi193 ceph-mon[37473]: pgmap v7497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:11.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:11 smithi134 ceph-mon[31966]: pgmap v7497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12769.756447 secs 2024-07-26T15:00:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:00:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:00:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:12 smithi193 ceph-mon[37473]: pgmap v7498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:00:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:12 smithi193 ceph-mon[37473]: pgmap v7499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:00:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12769.756447 secs 2024-07-26T15:00:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:00:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:00:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:12 smithi134 ceph-mon[31966]: pgmap v7498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:00:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:12 smithi134 ceph-mon[31966]: pgmap v7499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:00:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12769.756447 secs 2024-07-26T15:00:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:00:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:00:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:12 smithi189 ceph-mon[36790]: pgmap v7498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:00:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:12 smithi189 ceph-mon[36790]: pgmap v7499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:00:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:00:13.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:00:13.105+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:00:13.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:00:13.105+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:13.107 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:00:13.105+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:00:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:14 smithi134 ceph-mon[31966]: pgmap v7500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:00:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12755.332511 secs 2024-07-26T15:00:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:14 smithi189 ceph-mon[36790]: pgmap v7500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:00:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12755.332511 secs 2024-07-26T15:00:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:14 smithi193 ceph-mon[37473]: pgmap v7500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:00:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12755.332511 secs 2024-07-26T15:00:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:16 smithi189 ceph-mon[36790]: pgmap v7501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:00:16.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:16 smithi193 ceph-mon[37473]: pgmap v7501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:00:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:16 smithi134 ceph-mon[31966]: pgmap v7501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:00:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12774.756568 secs 2024-07-26T15:00:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12774.756568 secs 2024-07-26T15:00:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12774.756568 secs 2024-07-26T15:00:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:18 smithi189 ceph-mon[36790]: pgmap v7502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:00:18.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:18 smithi193 ceph-mon[37473]: pgmap v7502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:00:18.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:18 smithi134 ceph-mon[31966]: pgmap v7502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:00:19.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12760.332590 secs 2024-07-26T15:00:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12760.332590 secs 2024-07-26T15:00:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12760.332590 secs 2024-07-26T15:00:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:20 smithi134 ceph-mon[31966]: pgmap v7503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:00:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:20 smithi189 ceph-mon[36790]: pgmap v7503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:00:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:20 smithi193 ceph-mon[37473]: pgmap v7503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:00:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12779.756696 secs 2024-07-26T15:00:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:22 smithi134 ceph-mon[31966]: pgmap v7504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:00:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12779.756696 secs 2024-07-26T15:00:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:22 smithi189 ceph-mon[36790]: pgmap v7504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:00:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12779.756696 secs 2024-07-26T15:00:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:22 smithi193 ceph-mon[37473]: pgmap v7504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:00:23.636 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:00:23.635+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:00:23.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:00:23.635+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:00:23.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:00:23.635+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:24.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:00:24.551+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:00:24.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:00:24.551+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:00:24.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:00:24.552+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:00:24.553 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:00:24.552+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 100 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:00:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:24 smithi134 ceph-mon[31966]: pgmap v7505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:00:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12765.332720 secs 2024-07-26T15:00:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:24 smithi189 ceph-mon[36790]: pgmap v7505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:00:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12765.332720 secs 2024-07-26T15:00:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:24 smithi193 ceph-mon[37473]: pgmap v7505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:00:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12765.332720 secs 2024-07-26T15:00:26.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:26 smithi134 ceph-mon[31966]: pgmap v7506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:26 smithi189 ceph-mon[36790]: pgmap v7506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:26 smithi193 ceph-mon[37473]: pgmap v7506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12784.756817 secs 2024-07-26T15:00:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12784.756817 secs 2024-07-26T15:00:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12784.756817 secs 2024-07-26T15:00:28.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:00:28.113+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:00:28.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:00:28.113+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:28.114 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:00:28.113+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:00:28.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:28 smithi134 ceph-mon[31966]: pgmap v7507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:28 smithi189 ceph-mon[36790]: pgmap v7507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:28 smithi193 ceph-mon[37473]: pgmap v7507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12770.332836 secs 2024-07-26T15:00:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12770.332836 secs 2024-07-26T15:00:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12770.332836 secs 2024-07-26T15:00:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:30 smithi134 ceph-mon[31966]: pgmap v7508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:30 smithi189 ceph-mon[36790]: pgmap v7508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:30 smithi193 ceph-mon[37473]: pgmap v7508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12789.756954 secs 2024-07-26T15:00:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:32 smithi134 ceph-mon[31966]: pgmap v7509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12789.756954 secs 2024-07-26T15:00:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:32 smithi189 ceph-mon[36790]: pgmap v7509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12789.756954 secs 2024-07-26T15:00:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:32 smithi193 ceph-mon[37473]: pgmap v7509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:34 smithi134 ceph-mon[31966]: pgmap v7510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:00:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12775.332930 secs 2024-07-26T15:00:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:34 smithi189 ceph-mon[36790]: pgmap v7510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:00:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12775.332930 secs 2024-07-26T15:00:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:34 smithi193 ceph-mon[37473]: pgmap v7510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:00:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12775.332930 secs 2024-07-26T15:00:35.085 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:00:35.085+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:00:35.086 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:00:35.085+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:00:35.086 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:00:35.086+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:00:35.086 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:00:35.086+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:00:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:36 smithi134 ceph-mon[31966]: pgmap v7511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:36 smithi189 ceph-mon[36790]: pgmap v7511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:36 smithi193 ceph-mon[37473]: pgmap v7511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12794.757089 secs 2024-07-26T15:00:37.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12794.757089 secs 2024-07-26T15:00:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12794.757089 secs 2024-07-26T15:00:38.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:00:38.651+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:00:38.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:00:38.651+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:38.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:00:38.651+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:00:38.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:38 smithi134 ceph-mon[31966]: pgmap v7512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:38 smithi189 ceph-mon[36790]: pgmap v7512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:38 smithi193 ceph-mon[37473]: pgmap v7512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:00:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12780.333058 secs 2024-07-26T15:00:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12780.333058 secs 2024-07-26T15:00:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12780.333058 secs 2024-07-26T15:00:40.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:40 smithi134 ceph-mon[31966]: pgmap v7513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:40 smithi189 ceph-mon[36790]: pgmap v7513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:40 smithi193 ceph-mon[37473]: pgmap v7513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12799.757237 secs 2024-07-26T15:00:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:42 smithi134 ceph-mon[31966]: pgmap v7514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12799.757237 secs 2024-07-26T15:00:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:42 smithi189 ceph-mon[36790]: pgmap v7514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12799.757237 secs 2024-07-26T15:00:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:42 smithi193 ceph-mon[37473]: pgmap v7514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:43.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:00:43.127+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:00:43.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:00:43.128+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:43.129 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:00:43.128+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:00:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:44 smithi134 ceph-mon[31966]: pgmap v7515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:00:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12785.333133 secs 2024-07-26T15:00:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:44 smithi189 ceph-mon[36790]: pgmap v7515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:00:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12785.333133 secs 2024-07-26T15:00:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:44 smithi193 ceph-mon[37473]: pgmap v7515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:00:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12785.333133 secs 2024-07-26T15:00:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:46 smithi134 ceph-mon[31966]: pgmap v7516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:46.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:46 smithi189 ceph-mon[36790]: pgmap v7516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:46 smithi193 ceph-mon[37473]: pgmap v7516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12804.757380 secs 2024-07-26T15:00:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12804.757380 secs 2024-07-26T15:00:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12804.757380 secs 2024-07-26T15:00:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:48 smithi189 ceph-mon[36790]: pgmap v7517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:00:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:48 smithi193 ceph-mon[37473]: pgmap v7517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:00:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:48 smithi134 ceph-mon[31966]: pgmap v7517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:00:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12790.333246 secs 2024-07-26T15:00:49.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12790.333246 secs 2024-07-26T15:00:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12790.333246 secs 2024-07-26T15:00:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:50 smithi193 ceph-mon[37473]: pgmap v7518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:50 smithi134 ceph-mon[31966]: pgmap v7518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:50 smithi189 ceph-mon[36790]: pgmap v7518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:00:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12809.757499 secs 2024-07-26T15:00:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:52 smithi193 ceph-mon[37473]: pgmap v7519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12809.757499 secs 2024-07-26T15:00:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:52 smithi134 ceph-mon[31966]: pgmap v7519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12809.757499 secs 2024-07-26T15:00:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:52 smithi189 ceph-mon[36790]: pgmap v7519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:53.657 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:00:53.656+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:00:53.657 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:00:53.656+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:00:53.657 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:00:53.656+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:54 smithi193 ceph-mon[37473]: pgmap v7520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:00:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12795.333331 secs 2024-07-26T15:00:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:54 smithi134 ceph-mon[31966]: pgmap v7520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:00:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12795.333331 secs 2024-07-26T15:00:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:54 smithi189 ceph-mon[36790]: pgmap v7520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:00:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12795.333331 secs 2024-07-26T15:00:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:56 smithi193 ceph-mon[37473]: pgmap v7521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:56.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:56 smithi134 ceph-mon[31966]: pgmap v7521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:56.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:56 smithi189 ceph-mon[36790]: pgmap v7521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:00:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12814.757621 secs 2024-07-26T15:00:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12814.757621 secs 2024-07-26T15:00:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12814.757621 secs 2024-07-26T15:00:58.144 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:00:58.143+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:00:58.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:00:58.143+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:00:58.145 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:00:58.143+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:00:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:58 smithi193 ceph-mon[37473]: pgmap v7522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:00:58.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:58 smithi134 ceph-mon[31966]: pgmap v7522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:00:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:58 smithi189 ceph-mon[36790]: pgmap v7522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:00:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:00:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12800.333469 secs 2024-07-26T15:00:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:00:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12800.333469 secs 2024-07-26T15:00:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:00:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12800.333469 secs 2024-07-26T15:01:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:00 smithi193 ceph-mon[37473]: pgmap v7523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:01:00.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:00 smithi134 ceph-mon[31966]: pgmap v7523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:01:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:00 smithi189 ceph-mon[36790]: pgmap v7523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:01:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12819.757738 secs 2024-07-26T15:01:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:02 smithi193 ceph-mon[37473]: pgmap v7524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12819.757738 secs 2024-07-26T15:01:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:02 smithi134 ceph-mon[31966]: pgmap v7524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12819.757738 secs 2024-07-26T15:01:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:02 smithi189 ceph-mon[36790]: pgmap v7524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:04 smithi193 ceph-mon[37473]: pgmap v7525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12805.333579 secs 2024-07-26T15:01:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:04 smithi134 ceph-mon[31966]: pgmap v7525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12805.333579 secs 2024-07-26T15:01:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:04 smithi189 ceph-mon[36790]: pgmap v7525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12805.333579 secs 2024-07-26T15:01:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:06 smithi193 ceph-mon[37473]: pgmap v7526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:06 smithi134 ceph-mon[31966]: pgmap v7526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:06 smithi189 ceph-mon[36790]: pgmap v7526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12824.757849 secs 2024-07-26T15:01:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12824.757849 secs 2024-07-26T15:01:07.952 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12824.757849 secs 2024-07-26T15:01:08.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:01:08.672+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:01:08.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:01:08.672+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:01:08.673 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:01:08.672+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:08.877 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:08 smithi193 ceph-mon[37473]: pgmap v7527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:01:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:08 smithi134 ceph-mon[31966]: pgmap v7527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:01:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:08 smithi189 ceph-mon[36790]: pgmap v7527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:01:09.896 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12810.333667 secs 2024-07-26T15:01:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12810.333667 secs 2024-07-26T15:01:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12810.333667 secs 2024-07-26T15:01:10.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:10 smithi134 ceph-mon[31966]: pgmap v7528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:01:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:10 smithi189 ceph-mon[36790]: pgmap v7528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:01:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:10 smithi193 ceph-mon[37473]: pgmap v7528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:01:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:01:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:01:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:01:12.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12829.757975 secs 2024-07-26T15:01:12.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:12 smithi189 ceph-mon[36790]: pgmap v7529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:12.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:01:12.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:01:12.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:01:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12829.757975 secs 2024-07-26T15:01:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:12 smithi134 ceph-mon[31966]: pgmap v7529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:01:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:01:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:01:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12829.757975 secs 2024-07-26T15:01:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:12 smithi193 ceph-mon[37473]: pgmap v7529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:01:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:01:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:01:13.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:01:13.152+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:01:13.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:01:13.152+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:13.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:01:13.152+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:01:13.866 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:13 smithi189 ceph-mon[36790]: pgmap v7530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 8 op/s 2024-07-26T15:01:13.866 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:13 smithi189 ceph-mon[36790]: pgmap v7531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:01:13.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:13 smithi134 ceph-mon[31966]: pgmap v7530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 8 op/s 2024-07-26T15:01:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:13 smithi134 ceph-mon[31966]: pgmap v7531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:01:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:13 smithi193 ceph-mon[37473]: pgmap v7530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 8 op/s 2024-07-26T15:01:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:13 smithi193 ceph-mon[37473]: pgmap v7531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:01:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12815.333794 secs 2024-07-26T15:01:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12815.333794 secs 2024-07-26T15:01:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12815.333794 secs 2024-07-26T15:01:15.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:15 smithi134 ceph-mon[31966]: pgmap v7532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:01:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:15 smithi189 ceph-mon[36790]: pgmap v7532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:01:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:15 smithi193 ceph-mon[37473]: pgmap v7532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:01:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:16 smithi134 ceph-mon[31966]: pgmap v7533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:16 smithi189 ceph-mon[36790]: pgmap v7533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:17.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:16 smithi193 ceph-mon[37473]: pgmap v7533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:17.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12834.758035 secs 2024-07-26T15:01:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12834.758035 secs 2024-07-26T15:01:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12834.758035 secs 2024-07-26T15:01:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:18 smithi193 ceph-mon[37473]: pgmap v7534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:01:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:18 smithi134 ceph-mon[31966]: pgmap v7534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:01:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:18 smithi189 ceph-mon[36790]: pgmap v7534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:01:19.858 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12820.333920 secs 2024-07-26T15:01:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12820.333920 secs 2024-07-26T15:01:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12820.333920 secs 2024-07-26T15:01:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:20 smithi134 ceph-mon[31966]: pgmap v7535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:01:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:20 smithi189 ceph-mon[36790]: pgmap v7535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:01:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:20 smithi193 ceph-mon[37473]: pgmap v7535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:01:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12839.758159 secs 2024-07-26T15:01:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12839.758159 secs 2024-07-26T15:01:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12839.758159 secs 2024-07-26T15:01:23.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:22 smithi134 ceph-mon[31966]: pgmap v7536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:01:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:22 smithi189 ceph-mon[36790]: pgmap v7536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:01:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:22 smithi193 ceph-mon[37473]: pgmap v7536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:01:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:01:23.688+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:01:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:01:23.688+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:01:23.689 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:01:23.688+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:24.558 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12825.334045 secs 2024-07-26T15:01:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12825.334045 secs 2024-07-26T15:01:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12825.334045 secs 2024-07-26T15:01:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:25 smithi189 ceph-mon[36790]: pgmap v7537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:25.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:25 smithi134 ceph-mon[31966]: pgmap v7537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:25 smithi193 ceph-mon[37473]: pgmap v7537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12844.758277 secs 2024-07-26T15:01:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:27 smithi134 ceph-mon[31966]: pgmap v7538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12844.758277 secs 2024-07-26T15:01:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:27 smithi189 ceph-mon[36790]: pgmap v7538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12844.758277 secs 2024-07-26T15:01:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:27 smithi193 ceph-mon[37473]: pgmap v7538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:28.170 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:01:28.167+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:01:28.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:01:28.168+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:28.171 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:01:28.168+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:01:28.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:28 smithi134 ceph-mon[31966]: pgmap v7539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:28 smithi189 ceph-mon[36790]: pgmap v7539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:28 smithi193 ceph-mon[37473]: pgmap v7539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12830.334165 secs 2024-07-26T15:01:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12830.334165 secs 2024-07-26T15:01:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12830.334165 secs 2024-07-26T15:01:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:30 smithi134 ceph-mon[31966]: pgmap v7540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:30 smithi189 ceph-mon[36790]: pgmap v7540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:30 smithi193 ceph-mon[37473]: pgmap v7540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12849.758395 secs 2024-07-26T15:01:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12849.758395 secs 2024-07-26T15:01:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12849.758395 secs 2024-07-26T15:01:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:32 smithi134 ceph-mon[31966]: pgmap v7541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T15:01:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:32 smithi189 ceph-mon[36790]: pgmap v7541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T15:01:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:32 smithi193 ceph-mon[37473]: pgmap v7541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T15:01:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12835.334288 secs 2024-07-26T15:01:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12835.334288 secs 2024-07-26T15:01:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12835.334288 secs 2024-07-26T15:01:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:35 smithi134 ceph-mon[31966]: pgmap v7542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:35 smithi189 ceph-mon[36790]: pgmap v7542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:35 smithi193 ceph-mon[37473]: pgmap v7542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:36 smithi134 ceph-mon[31966]: pgmap v7543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:36 smithi189 ceph-mon[36790]: pgmap v7543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:36 smithi193 ceph-mon[37473]: pgmap v7543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:37.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12854.758529 secs 2024-07-26T15:01:37.924 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12854.758529 secs 2024-07-26T15:01:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12854.758529 secs 2024-07-26T15:01:38.704 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:01:38.704+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:01:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:01:38.704+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:01:38.705 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:01:38.704+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:38 smithi134 ceph-mon[31966]: pgmap v7544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:38 smithi189 ceph-mon[36790]: pgmap v7544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:38 smithi193 ceph-mon[37473]: pgmap v7544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12840.334433 secs 2024-07-26T15:01:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12840.334433 secs 2024-07-26T15:01:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12840.334433 secs 2024-07-26T15:01:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:40 smithi134 ceph-mon[31966]: pgmap v7545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:40 smithi189 ceph-mon[36790]: pgmap v7545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:40 smithi193 ceph-mon[37473]: pgmap v7545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12859.758639 secs 2024-07-26T15:01:42.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12859.758639 secs 2024-07-26T15:01:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12859.758639 secs 2024-07-26T15:01:43.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:42 smithi134 ceph-mon[31966]: pgmap v7546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T15:01:43.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:01:43.183+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:01:43.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:01:43.183+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:43.185 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:01:43.183+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:01:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:42 smithi189 ceph-mon[36790]: pgmap v7546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T15:01:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:42 smithi193 ceph-mon[37473]: pgmap v7546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T15:01:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12845.334535 secs 2024-07-26T15:01:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12845.334535 secs 2024-07-26T15:01:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12845.334535 secs 2024-07-26T15:01:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:45 smithi134 ceph-mon[31966]: pgmap v7547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:45 smithi189 ceph-mon[36790]: pgmap v7547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:45 smithi193 ceph-mon[37473]: pgmap v7547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:46 smithi134 ceph-mon[31966]: pgmap v7548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:46 smithi189 ceph-mon[36790]: pgmap v7548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:46 smithi193 ceph-mon[37473]: pgmap v7548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12864.758753 secs 2024-07-26T15:01:47.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12864.758753 secs 2024-07-26T15:01:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12864.758753 secs 2024-07-26T15:01:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:48 smithi134 ceph-mon[31966]: pgmap v7549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:48 smithi189 ceph-mon[36790]: pgmap v7549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:48 smithi193 ceph-mon[37473]: pgmap v7549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12850.334657 secs 2024-07-26T15:01:49.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12850.334657 secs 2024-07-26T15:01:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12850.334657 secs 2024-07-26T15:01:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:50 smithi134 ceph-mon[31966]: pgmap v7550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:50 smithi189 ceph-mon[36790]: pgmap v7550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:50 smithi193 ceph-mon[37473]: pgmap v7550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12869.758871 secs 2024-07-26T15:01:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12869.758871 secs 2024-07-26T15:01:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12869.758871 secs 2024-07-26T15:01:53.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:52 smithi134 ceph-mon[31966]: pgmap v7551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:52 smithi189 ceph-mon[36790]: pgmap v7551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:52 smithi193 ceph-mon[37473]: pgmap v7551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:53.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:01:53.717+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:01:53.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:01:53.717+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:01:53.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:01:53.717+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:54.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:01:54.568+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:01:54.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:01:54.568+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:01:54.569 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12855.334788 secs 2024-07-26T15:01:54.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:01:54.569+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T15:01:54.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:01:54.569+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 101 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T15:01:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12855.334788 secs 2024-07-26T15:01:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12855.334788 secs 2024-07-26T15:01:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:55 smithi134 ceph-mon[31966]: pgmap v7552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:55 smithi189 ceph-mon[36790]: pgmap v7552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:55 smithi193 ceph-mon[37473]: pgmap v7552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:56.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:56 smithi134 ceph-mon[31966]: pgmap v7553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:56 smithi189 ceph-mon[36790]: pgmap v7553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:56 smithi193 ceph-mon[37473]: pgmap v7553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:01:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12874.758987 secs 2024-07-26T15:01:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12874.758987 secs 2024-07-26T15:01:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12874.758987 secs 2024-07-26T15:01:58.193 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:01:58.192+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:01:58.194 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:01:58.192+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:01:58.194 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:01:58.192+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:01:58.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:58 smithi134 ceph-mon[31966]: pgmap v7554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:58.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:58 smithi189 ceph-mon[36790]: pgmap v7554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:58 smithi193 ceph-mon[37473]: pgmap v7554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:01:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:01:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12860.334918 secs 2024-07-26T15:01:59.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:01:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12860.334918 secs 2024-07-26T15:02:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:01:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12860.334918 secs 2024-07-26T15:02:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:00 smithi189 ceph-mon[36790]: pgmap v7555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:00 smithi193 ceph-mon[37473]: pgmap v7555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:00 smithi134 ceph-mon[31966]: pgmap v7555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12879.759120 secs 2024-07-26T15:02:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12879.759120 secs 2024-07-26T15:02:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12879.759120 secs 2024-07-26T15:02:03.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:02 smithi134 ceph-mon[31966]: pgmap v7556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:02 smithi189 ceph-mon[36790]: pgmap v7556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:02 smithi193 ceph-mon[37473]: pgmap v7556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12865.335042 secs 2024-07-26T15:02:04.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12865.335042 secs 2024-07-26T15:02:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12865.335042 secs 2024-07-26T15:02:05.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:02:05.101+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:02:05.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:02:05.101+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:02:05.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:02:05.101+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:02:05.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:02:05.101+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:02:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:05 smithi134 ceph-mon[31966]: pgmap v7557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:05 smithi189 ceph-mon[36790]: pgmap v7557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:05 smithi193 ceph-mon[37473]: pgmap v7557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:06 smithi134 ceph-mon[31966]: pgmap v7558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:06 smithi189 ceph-mon[36790]: pgmap v7558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:06 smithi193 ceph-mon[37473]: pgmap v7558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12884.759236 secs 2024-07-26T15:02:07.924 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12884.759236 secs 2024-07-26T15:02:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12884.759236 secs 2024-07-26T15:02:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:02:08.720+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:02:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:02:08.720+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:02:08.720+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:02:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:08 smithi134 ceph-mon[31966]: pgmap v7559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:08 smithi189 ceph-mon[36790]: pgmap v7559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:08 smithi193 ceph-mon[37473]: pgmap v7559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12870.335163 secs 2024-07-26T15:02:09.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12870.335163 secs 2024-07-26T15:02:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12870.335163 secs 2024-07-26T15:02:10.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:10 smithi134 ceph-mon[31966]: pgmap v7560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:10 smithi189 ceph-mon[36790]: pgmap v7560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:10 smithi193 ceph-mon[37473]: pgmap v7560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12889.759363 secs 2024-07-26T15:02:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12889.759363 secs 2024-07-26T15:02:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12889.759363 secs 2024-07-26T15:02:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:12 smithi134 ceph-mon[31966]: pgmap v7561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:02:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:12 smithi189 ceph-mon[36790]: pgmap v7561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:02:13.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:02:13.207+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:02:13.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:02:13.207+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:13.208 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:02:13.207+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:02:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:12 smithi193 ceph-mon[37473]: pgmap v7561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:02:14.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:02:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:02:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:02:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:02:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:02:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:02:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:02:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:02:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:02:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12875.335285 secs 2024-07-26T15:02:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:14 smithi134 ceph-mon[31966]: pgmap v7562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:02:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:14 smithi134 ceph-mon[31966]: pgmap v7563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12875.335285 secs 2024-07-26T15:02:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:14 smithi189 ceph-mon[36790]: pgmap v7562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:02:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:14 smithi189 ceph-mon[36790]: pgmap v7563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12875.335285 secs 2024-07-26T15:02:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:14 smithi193 ceph-mon[37473]: pgmap v7562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:02:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:14 smithi193 ceph-mon[37473]: pgmap v7563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:17.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:17 smithi193 ceph-mon[37473]: pgmap v7564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12894.759472 secs 2024-07-26T15:02:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:17 smithi134 ceph-mon[31966]: pgmap v7564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12894.759472 secs 2024-07-26T15:02:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:17 smithi189 ceph-mon[36790]: pgmap v7564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12894.759472 secs 2024-07-26T15:02:19.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:19 smithi193 ceph-mon[37473]: pgmap v7565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:19 smithi134 ceph-mon[31966]: pgmap v7565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:19 smithi189 ceph-mon[36790]: pgmap v7565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:02:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12880.335433 secs 2024-07-26T15:02:20.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12880.335433 secs 2024-07-26T15:02:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12880.335433 secs 2024-07-26T15:02:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:21 smithi193 ceph-mon[37473]: pgmap v7566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:02:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:21 smithi134 ceph-mon[31966]: pgmap v7566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:02:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:21 smithi189 ceph-mon[36790]: pgmap v7566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:02:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12899.759556 secs 2024-07-26T15:02:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12899.759556 secs 2024-07-26T15:02:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12899.759556 secs 2024-07-26T15:02:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:23 smithi193 ceph-mon[37473]: pgmap v7567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:02:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:23 smithi134 ceph-mon[31966]: pgmap v7567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:02:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:23 smithi189 ceph-mon[36790]: pgmap v7567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:02:23.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:02:23.728+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:02:23.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:02:23.728+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:02:23.729 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:02:23.728+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:24.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12885.335523 secs 2024-07-26T15:02:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12885.335523 secs 2024-07-26T15:02:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12885.335523 secs 2024-07-26T15:02:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:25 smithi134 ceph-mon[31966]: pgmap v7568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:02:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:25 smithi189 ceph-mon[36790]: pgmap v7568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:02:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:25 smithi193 ceph-mon[37473]: pgmap v7568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:02:27.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:27 smithi134 ceph-mon[31966]: pgmap v7569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12904.759673 secs 2024-07-26T15:02:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:27 smithi189 ceph-mon[36790]: pgmap v7569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12904.759673 secs 2024-07-26T15:02:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:27 smithi193 ceph-mon[37473]: pgmap v7569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12904.759673 secs 2024-07-26T15:02:28.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:02:28.222+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:02:28.224 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:02:28.222+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:02:28.225 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:02:28.222+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:29 smithi134 ceph-mon[31966]: pgmap v7570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12890.335622 secs 2024-07-26T15:02:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:29 smithi189 ceph-mon[36790]: pgmap v7570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12890.335622 secs 2024-07-26T15:02:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:29 smithi193 ceph-mon[37473]: pgmap v7570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12890.335622 secs 2024-07-26T15:02:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:31 smithi134 ceph-mon[31966]: pgmap v7571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:31 smithi189 ceph-mon[36790]: pgmap v7571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:31 smithi193 ceph-mon[37473]: pgmap v7571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:32.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12909.759774 secs 2024-07-26T15:02:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12909.759774 secs 2024-07-26T15:02:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12909.759774 secs 2024-07-26T15:02:33.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:33 smithi134 ceph-mon[31966]: pgmap v7572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:33 smithi189 ceph-mon[36790]: pgmap v7572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:33 smithi193 ceph-mon[37473]: pgmap v7572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12895.335751 secs 2024-07-26T15:02:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12895.335751 secs 2024-07-26T15:02:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12895.335751 secs 2024-07-26T15:02:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:35 smithi134 ceph-mon[31966]: pgmap v7573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:35 smithi189 ceph-mon[36790]: pgmap v7573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:35 smithi193 ceph-mon[37473]: pgmap v7573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:37 smithi134 ceph-mon[31966]: pgmap v7574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12914.759895 secs 2024-07-26T15:02:37.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:37 smithi189 ceph-mon[36790]: pgmap v7574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12914.759895 secs 2024-07-26T15:02:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:37 smithi193 ceph-mon[37473]: pgmap v7574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12914.759895 secs 2024-07-26T15:02:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:02:38.741+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:02:38.742 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:02:38.741+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:38.743 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:02:38.741+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:02:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:39 smithi134 ceph-mon[31966]: pgmap v7575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12900.335865 secs 2024-07-26T15:02:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:39 smithi189 ceph-mon[36790]: pgmap v7575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12900.335865 secs 2024-07-26T15:02:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:39 smithi193 ceph-mon[37473]: pgmap v7575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12900.335865 secs 2024-07-26T15:02:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:41 smithi134 ceph-mon[31966]: pgmap v7576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:41 smithi189 ceph-mon[36790]: pgmap v7576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:41 smithi193 ceph-mon[37473]: pgmap v7576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12919.759962 secs 2024-07-26T15:02:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12919.759962 secs 2024-07-26T15:02:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12919.759962 secs 2024-07-26T15:02:43.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:02:43.238+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:02:43.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:02:43.238+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:02:43.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:02:43.238+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:43.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:43 smithi134 ceph-mon[31966]: pgmap v7577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:43 smithi189 ceph-mon[36790]: pgmap v7577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:43 smithi193 ceph-mon[37473]: pgmap v7577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12905.336013 secs 2024-07-26T15:02:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12905.336013 secs 2024-07-26T15:02:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12905.336013 secs 2024-07-26T15:02:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:45 smithi134 ceph-mon[31966]: pgmap v7578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:45 smithi189 ceph-mon[36790]: pgmap v7578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:45 smithi193 ceph-mon[37473]: pgmap v7578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:47.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:47 smithi134 ceph-mon[31966]: pgmap v7579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12924.760086 secs 2024-07-26T15:02:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:47 smithi189 ceph-mon[36790]: pgmap v7579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12924.760086 secs 2024-07-26T15:02:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:47 smithi193 ceph-mon[37473]: pgmap v7579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12924.760086 secs 2024-07-26T15:02:49.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:49 smithi134 ceph-mon[31966]: pgmap v7580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12910.336112 secs 2024-07-26T15:02:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:49 smithi189 ceph-mon[36790]: pgmap v7580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12910.336112 secs 2024-07-26T15:02:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:49 smithi193 ceph-mon[37473]: pgmap v7580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12910.336112 secs 2024-07-26T15:02:51.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:51 smithi134 ceph-mon[31966]: pgmap v7581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:51 smithi189 ceph-mon[36790]: pgmap v7581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:51.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:51 smithi193 ceph-mon[37473]: pgmap v7581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:02:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12929.760239 secs 2024-07-26T15:02:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12929.760239 secs 2024-07-26T15:02:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12929.760239 secs 2024-07-26T15:02:53.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:53 smithi134 ceph-mon[31966]: pgmap v7582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:53 smithi189 ceph-mon[36790]: pgmap v7582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:02:53.757+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:02:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:02:53.757+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:02:53.758 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:02:53.757+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:53 smithi193 ceph-mon[37473]: pgmap v7582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12915.336235 secs 2024-07-26T15:02:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12915.336235 secs 2024-07-26T15:02:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12915.336235 secs 2024-07-26T15:02:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:55 smithi189 ceph-mon[36790]: pgmap v7583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:55 smithi193 ceph-mon[37473]: pgmap v7583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:55.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:55 smithi134 ceph-mon[31966]: pgmap v7583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:57 smithi189 ceph-mon[36790]: pgmap v7584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12934.760339 secs 2024-07-26T15:02:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:57 smithi193 ceph-mon[37473]: pgmap v7584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12934.760339 secs 2024-07-26T15:02:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:57 smithi134 ceph-mon[31966]: pgmap v7584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:02:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12934.760339 secs 2024-07-26T15:02:58.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:02:58.253+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:02:58.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:02:58.253+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:02:58.255 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:02:58.253+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:02:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:59 smithi189 ceph-mon[36790]: pgmap v7585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:02:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12920.336360 secs 2024-07-26T15:02:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:59 smithi193 ceph-mon[37473]: pgmap v7585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:02:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12920.336360 secs 2024-07-26T15:02:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:59 smithi134 ceph-mon[31966]: pgmap v7585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:02:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:02:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12920.336360 secs 2024-07-26T15:03:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:01 smithi189 ceph-mon[36790]: pgmap v7586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:01 smithi193 ceph-mon[37473]: pgmap v7586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:01.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:01 smithi134 ceph-mon[31966]: pgmap v7586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12939.760455 secs 2024-07-26T15:03:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12939.760455 secs 2024-07-26T15:03:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12939.760455 secs 2024-07-26T15:03:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:03 smithi193 ceph-mon[37473]: pgmap v7587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:03.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:03 smithi134 ceph-mon[31966]: pgmap v7587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:03 smithi189 ceph-mon[36790]: pgmap v7587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12925.336490 secs 2024-07-26T15:03:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12925.336490 secs 2024-07-26T15:03:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12925.336490 secs 2024-07-26T15:03:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:05 smithi193 ceph-mon[37473]: pgmap v7588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:05.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:05 smithi134 ceph-mon[31966]: pgmap v7588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:05 smithi189 ceph-mon[36790]: pgmap v7588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:07 smithi193 ceph-mon[37473]: pgmap v7589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12944.760566 secs 2024-07-26T15:03:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:07 smithi134 ceph-mon[31966]: pgmap v7589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12944.760566 secs 2024-07-26T15:03:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:07 smithi189 ceph-mon[36790]: pgmap v7589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12944.760566 secs 2024-07-26T15:03:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:03:08.765+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:03:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:03:08.765+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:03:08.766 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:03:08.765+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:09 smithi193 ceph-mon[37473]: pgmap v7590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12930.336633 secs 2024-07-26T15:03:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:09 smithi134 ceph-mon[31966]: pgmap v7590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12930.336633 secs 2024-07-26T15:03:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:09 smithi189 ceph-mon[36790]: pgmap v7590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12930.336633 secs 2024-07-26T15:03:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:11 smithi193 ceph-mon[37473]: pgmap v7591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:11 smithi134 ceph-mon[31966]: pgmap v7591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:11 smithi189 ceph-mon[36790]: pgmap v7591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12949.760672 secs 2024-07-26T15:03:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12949.760672 secs 2024-07-26T15:03:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12949.760672 secs 2024-07-26T15:03:13.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:03:13.269+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:03:13.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:03:13.269+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:13.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:03:13.269+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:03:13.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:13 smithi134 ceph-mon[31966]: pgmap v7592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:13.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:13 smithi193 ceph-mon[37473]: pgmap v7592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:13.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:13 smithi189 ceph-mon[36790]: pgmap v7592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12935.336763 secs 2024-07-26T15:03:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:03:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12935.336763 secs 2024-07-26T15:03:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:03:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12935.336763 secs 2024-07-26T15:03:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:03:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:15 smithi193 ceph-mon[37473]: pgmap v7593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:03:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:03:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:03:15.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:15 smithi134 ceph-mon[31966]: pgmap v7593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:03:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:03:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:03:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:15 smithi189 ceph-mon[36790]: pgmap v7593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:03:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:03:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:03:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:16 smithi134 ceph-mon[31966]: pgmap v7594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:03:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:16 smithi134 ceph-mon[31966]: pgmap v7595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:03:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:16 smithi189 ceph-mon[36790]: pgmap v7594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:03:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:16 smithi189 ceph-mon[36790]: pgmap v7595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:03:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:16 smithi193 ceph-mon[37473]: pgmap v7594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:03:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:16 smithi193 ceph-mon[37473]: pgmap v7595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:03:17.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12954.760774 secs 2024-07-26T15:03:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12954.760774 secs 2024-07-26T15:03:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12954.760774 secs 2024-07-26T15:03:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:18 smithi134 ceph-mon[31966]: pgmap v7596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:03:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:18 smithi189 ceph-mon[36790]: pgmap v7596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:03:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:18 smithi193 ceph-mon[37473]: pgmap v7596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:03:19.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12940.336827 secs 2024-07-26T15:03:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12940.336827 secs 2024-07-26T15:03:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12940.336827 secs 2024-07-26T15:03:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:20 smithi134 ceph-mon[31966]: pgmap v7597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:03:20.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:20 smithi189 ceph-mon[36790]: pgmap v7597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:03:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:20 smithi193 ceph-mon[37473]: pgmap v7597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:03:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:22 smithi134 ceph-mon[31966]: pgmap v7598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:03:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12959.760912 secs 2024-07-26T15:03:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:22 smithi189 ceph-mon[36790]: pgmap v7598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:03:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12959.760912 secs 2024-07-26T15:03:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:22 smithi193 ceph-mon[37473]: pgmap v7598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:03:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12959.760912 secs 2024-07-26T15:03:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:03:23.780+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:03:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:03:23.780+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:23.782 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:03:23.780+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:03:24.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:03:24.583+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:03:24.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:03:24.583+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:03:24.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:03:24.584+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 102 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:03:24.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:03:24.584+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:03:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:24 smithi134 ceph-mon[31966]: pgmap v7599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:03:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12945.336998 secs 2024-07-26T15:03:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:24 smithi189 ceph-mon[36790]: pgmap v7599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:03:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12945.336998 secs 2024-07-26T15:03:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:24 smithi193 ceph-mon[37473]: pgmap v7599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:03:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12945.336998 secs 2024-07-26T15:03:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:26 smithi134 ceph-mon[31966]: pgmap v7600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:03:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:26 smithi189 ceph-mon[36790]: pgmap v7600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:03:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:26 smithi193 ceph-mon[37473]: pgmap v7600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:03:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12964.761028 secs 2024-07-26T15:03:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12964.761028 secs 2024-07-26T15:03:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12964.761028 secs 2024-07-26T15:03:28.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:03:28.280+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:03:28.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:03:28.280+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:28.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:03:28.280+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:03:28.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:28 smithi134 ceph-mon[31966]: pgmap v7601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:28 smithi189 ceph-mon[36790]: pgmap v7601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:28 smithi193 ceph-mon[37473]: pgmap v7601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12950.337090 secs 2024-07-26T15:03:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12950.337090 secs 2024-07-26T15:03:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12950.337090 secs 2024-07-26T15:03:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:30 smithi134 ceph-mon[31966]: pgmap v7602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:30 smithi189 ceph-mon[36790]: pgmap v7602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:30 smithi193 ceph-mon[37473]: pgmap v7602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:31 smithi134 ceph-mon[31966]: pgmap v7603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:31 smithi189 ceph-mon[36790]: pgmap v7603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:31 smithi193 ceph-mon[37473]: pgmap v7603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12969.761147 secs 2024-07-26T15:03:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12969.761147 secs 2024-07-26T15:03:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12969.761147 secs 2024-07-26T15:03:33.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:33 smithi134 ceph-mon[31966]: pgmap v7604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:33 smithi189 ceph-mon[36790]: pgmap v7604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:33 smithi193 ceph-mon[37473]: pgmap v7604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12955.337201 secs 2024-07-26T15:03:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12955.337201 secs 2024-07-26T15:03:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12955.337201 secs 2024-07-26T15:03:35.117 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:03:35.116+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:03:35.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:03:35.116+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:03:35.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:03:35.117+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:03:35.118 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:03:35.117+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:03:35.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:35 smithi134 ceph-mon[31966]: pgmap v7605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:35 smithi189 ceph-mon[36790]: pgmap v7605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:35 smithi193 ceph-mon[37473]: pgmap v7605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12974.761286 secs 2024-07-26T15:03:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12974.761286 secs 2024-07-26T15:03:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12974.761286 secs 2024-07-26T15:03:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:37 smithi134 ceph-mon[31966]: pgmap v7606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:38.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:37 smithi193 ceph-mon[37473]: pgmap v7606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:37 smithi189 ceph-mon[36790]: pgmap v7606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:03:38.791+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:03:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:03:38.792+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:03:38.793 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:03:38.792+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12960.337325 secs 2024-07-26T15:03:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:40 smithi193 ceph-mon[37473]: pgmap v7607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12960.337325 secs 2024-07-26T15:03:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:40 smithi134 ceph-mon[31966]: pgmap v7607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12960.337325 secs 2024-07-26T15:03:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:40 smithi189 ceph-mon[36790]: pgmap v7607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:42 smithi134 ceph-mon[31966]: pgmap v7608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12979.761357 secs 2024-07-26T15:03:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:42 smithi189 ceph-mon[36790]: pgmap v7608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12979.761357 secs 2024-07-26T15:03:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:42 smithi193 ceph-mon[37473]: pgmap v7608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12979.761357 secs 2024-07-26T15:03:43.295 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:03:43.294+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:03:43.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:03:43.294+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:03:43.296 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:03:43.294+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:44.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:44 smithi134 ceph-mon[31966]: pgmap v7609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:44 smithi189 ceph-mon[36790]: pgmap v7609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:44 smithi193 ceph-mon[37473]: pgmap v7609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12965.337419 secs 2024-07-26T15:03:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12965.337419 secs 2024-07-26T15:03:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12965.337419 secs 2024-07-26T15:03:46.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:46 smithi134 ceph-mon[31966]: pgmap v7610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:46 smithi189 ceph-mon[36790]: pgmap v7610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:46 smithi193 ceph-mon[37473]: pgmap v7610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12984.761471 secs 2024-07-26T15:03:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12984.761471 secs 2024-07-26T15:03:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12984.761471 secs 2024-07-26T15:03:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:48 smithi134 ceph-mon[31966]: pgmap v7611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:48 smithi189 ceph-mon[36790]: pgmap v7611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:48 smithi193 ceph-mon[37473]: pgmap v7611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:50.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12970.337542 secs 2024-07-26T15:03:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:50 smithi134 ceph-mon[31966]: pgmap v7612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12970.337542 secs 2024-07-26T15:03:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:50 smithi189 ceph-mon[36790]: pgmap v7612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12970.337542 secs 2024-07-26T15:03:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:50 smithi193 ceph-mon[37473]: pgmap v7612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:03:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:52 smithi134 ceph-mon[31966]: pgmap v7613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12989.761566 secs 2024-07-26T15:03:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:52 smithi189 ceph-mon[36790]: pgmap v7613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12989.761566 secs 2024-07-26T15:03:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:52 smithi193 ceph-mon[37473]: pgmap v7613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:03:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12989.761566 secs 2024-07-26T15:03:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:03:53.805+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:03:53.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:03:53.805+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:53.807 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:03:53.805+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:03:54.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:54 smithi134 ceph-mon[31966]: pgmap v7614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:54 smithi189 ceph-mon[36790]: pgmap v7614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:54 smithi193 ceph-mon[37473]: pgmap v7614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:03:55.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12975.337658 secs 2024-07-26T15:03:55.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12975.337658 secs 2024-07-26T15:03:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12975.337658 secs 2024-07-26T15:03:56.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:56 smithi134 ceph-mon[31966]: pgmap v7615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:56 smithi189 ceph-mon[36790]: pgmap v7615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:56 smithi193 ceph-mon[37473]: pgmap v7615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:03:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12994.761635 secs 2024-07-26T15:03:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12994.761635 secs 2024-07-26T15:03:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12994.761635 secs 2024-07-26T15:03:58.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:03:58.303+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:03:58.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:03:58.303+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:03:58.306 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:03:58.303+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:03:58.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:03:58 smithi134 ceph-mon[31966]: pgmap v7616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:03:58 smithi189 ceph-mon[36790]: pgmap v7616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:03:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:03:58 smithi193 ceph-mon[37473]: pgmap v7616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12980.337778 secs 2024-07-26T15:04:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:00 smithi134 ceph-mon[31966]: pgmap v7617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12980.337778 secs 2024-07-26T15:04:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:00 smithi189 ceph-mon[36790]: pgmap v7617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12980.337778 secs 2024-07-26T15:04:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:00 smithi193 ceph-mon[37473]: pgmap v7617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:02 smithi134 ceph-mon[31966]: pgmap v7618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 12999.761754 secs 2024-07-26T15:04:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:02 smithi189 ceph-mon[36790]: pgmap v7618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 12999.761754 secs 2024-07-26T15:04:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:02 smithi193 ceph-mon[37473]: pgmap v7618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 12999.761754 secs 2024-07-26T15:04:04.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:04 smithi134 ceph-mon[31966]: pgmap v7619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:04:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:04 smithi189 ceph-mon[36790]: pgmap v7619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:04:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:04 smithi193 ceph-mon[37473]: pgmap v7619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:04:05.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12985.337906 secs 2024-07-26T15:04:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12985.337906 secs 2024-07-26T15:04:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12985.337906 secs 2024-07-26T15:04:06.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:06 smithi134 ceph-mon[31966]: pgmap v7620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:04:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:06 smithi189 ceph-mon[36790]: pgmap v7620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:04:06.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:06 smithi193 ceph-mon[37473]: pgmap v7620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:04:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13004.761869 secs 2024-07-26T15:04:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13004.761869 secs 2024-07-26T15:04:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13004.761869 secs 2024-07-26T15:04:08.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:08 smithi134 ceph-mon[31966]: pgmap v7621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:08.424 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:08 smithi193 ceph-mon[37473]: pgmap v7621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:08.442 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:08 smithi189 ceph-mon[36790]: pgmap v7621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:04:08.821+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:04:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:04:08.821+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:08.822 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:04:08.821+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:04:10.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12990.338044 secs 2024-07-26T15:04:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:10 smithi134 ceph-mon[31966]: pgmap v7622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12990.338044 secs 2024-07-26T15:04:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:10 smithi189 ceph-mon[36790]: pgmap v7622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12990.338044 secs 2024-07-26T15:04:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:10 smithi193 ceph-mon[37473]: pgmap v7622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:12.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:12 smithi134 ceph-mon[31966]: pgmap v7623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13009.762009 secs 2024-07-26T15:04:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:12 smithi189 ceph-mon[36790]: pgmap v7623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13009.762009 secs 2024-07-26T15:04:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:12 smithi193 ceph-mon[37473]: pgmap v7623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13009.762009 secs 2024-07-26T15:04:13.319 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:04:13.318+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:04:13.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:04:13.318+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:13.320 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:04:13.318+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:04:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:14 smithi134 ceph-mon[31966]: pgmap v7624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:04:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:14 smithi189 ceph-mon[36790]: pgmap v7624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:04:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:14 smithi193 ceph-mon[37473]: pgmap v7624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:04:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 12995.338125 secs 2024-07-26T15:04:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:04:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 12995.338125 secs 2024-07-26T15:04:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:04:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 12995.338125 secs 2024-07-26T15:04:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:04:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:16 smithi134 ceph-mon[31966]: pgmap v7625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:04:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:04:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:04:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:04:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:16 smithi189 ceph-mon[36790]: pgmap v7625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:04:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:04:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:04:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:04:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:16 smithi193 ceph-mon[37473]: pgmap v7625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:04:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:04:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:04:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:04:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13014.762154 secs 2024-07-26T15:04:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:17 smithi134 ceph-mon[31966]: pgmap v7626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:04:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:17 smithi134 ceph-mon[31966]: pgmap v7627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:04:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13014.762154 secs 2024-07-26T15:04:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:17 smithi189 ceph-mon[36790]: pgmap v7626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:04:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:17 smithi189 ceph-mon[36790]: pgmap v7627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:04:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13014.762154 secs 2024-07-26T15:04:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:17 smithi193 ceph-mon[37473]: pgmap v7626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:04:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:17 smithi193 ceph-mon[37473]: pgmap v7627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:04:19.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:19 smithi134 ceph-mon[31966]: pgmap v7628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:04:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13000.338254 secs 2024-07-26T15:04:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:19 smithi189 ceph-mon[36790]: pgmap v7628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:04:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13000.338254 secs 2024-07-26T15:04:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:19 smithi193 ceph-mon[37473]: pgmap v7628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:04:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13000.338254 secs 2024-07-26T15:04:21.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:21 smithi134 ceph-mon[31966]: pgmap v7629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:04:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:21 smithi189 ceph-mon[36790]: pgmap v7629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:04:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:21 smithi193 ceph-mon[37473]: pgmap v7629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:04:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13019.762241 secs 2024-07-26T15:04:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13019.762241 secs 2024-07-26T15:04:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13019.762241 secs 2024-07-26T15:04:23.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:23 smithi134 ceph-mon[31966]: pgmap v7630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:04:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:23 smithi189 ceph-mon[36790]: pgmap v7630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:04:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:23 smithi193 ceph-mon[37473]: pgmap v7630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:04:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:04:23.836+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:04:23.837 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:04:23.836+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:23.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:04:23.836+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:04:24.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13005.338342 secs 2024-07-26T15:04:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13005.338342 secs 2024-07-26T15:04:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13005.338342 secs 2024-07-26T15:04:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:25 smithi134 ceph-mon[31966]: pgmap v7631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:04:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:25 smithi189 ceph-mon[36790]: pgmap v7631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:04:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:25 smithi193 ceph-mon[37473]: pgmap v7631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:04:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13024.762372 secs 2024-07-26T15:04:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:27 smithi134 ceph-mon[31966]: pgmap v7632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:04:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13024.762372 secs 2024-07-26T15:04:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:27 smithi189 ceph-mon[36790]: pgmap v7632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:04:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13024.762372 secs 2024-07-26T15:04:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:27 smithi193 ceph-mon[37473]: pgmap v7632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:04:28.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:04:28.319+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:04:28.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:04:28.319+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:04:28.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:04:28.319+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:29 smithi134 ceph-mon[31966]: pgmap v7633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:04:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13010.338458 secs 2024-07-26T15:04:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:29 smithi189 ceph-mon[36790]: pgmap v7633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:04:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13010.338458 secs 2024-07-26T15:04:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:29 smithi193 ceph-mon[37473]: pgmap v7633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:04:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13010.338458 secs 2024-07-26T15:04:31.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:31 smithi134 ceph-mon[31966]: pgmap v7634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:31.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:31 smithi189 ceph-mon[36790]: pgmap v7634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:31 smithi193 ceph-mon[37473]: pgmap v7634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13029.762468 secs 2024-07-26T15:04:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13029.762468 secs 2024-07-26T15:04:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13029.762468 secs 2024-07-26T15:04:33.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:33 smithi134 ceph-mon[31966]: pgmap v7635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:33 smithi189 ceph-mon[36790]: pgmap v7635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:33 smithi193 ceph-mon[37473]: pgmap v7635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13015.338603 secs 2024-07-26T15:04:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13015.338603 secs 2024-07-26T15:04:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13015.338603 secs 2024-07-26T15:04:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:35 smithi134 ceph-mon[31966]: pgmap v7636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:35 smithi189 ceph-mon[36790]: pgmap v7636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:35 smithi193 ceph-mon[37473]: pgmap v7636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:04:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13034.762588 secs 2024-07-26T15:04:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:37 smithi134 ceph-mon[31966]: pgmap v7637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13034.762588 secs 2024-07-26T15:04:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:37 smithi189 ceph-mon[36790]: pgmap v7637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13034.762588 secs 2024-07-26T15:04:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:37 smithi193 ceph-mon[37473]: pgmap v7637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:04:38.852+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:04:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:04:38.852+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:04:38.852+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:04:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:39 smithi134 ceph-mon[31966]: pgmap v7638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:04:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13020.338719 secs 2024-07-26T15:04:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:39 smithi189 ceph-mon[36790]: pgmap v7638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:04:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13020.338719 secs 2024-07-26T15:04:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:39 smithi193 ceph-mon[37473]: pgmap v7638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:04:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13020.338719 secs 2024-07-26T15:04:41.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:41 smithi134 ceph-mon[31966]: pgmap v7639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:41 smithi189 ceph-mon[36790]: pgmap v7639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:41 smithi193 ceph-mon[37473]: pgmap v7639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13039.762651 secs 2024-07-26T15:04:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13039.762651 secs 2024-07-26T15:04:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13039.762651 secs 2024-07-26T15:04:43.322 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:04:43.321+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:04:43.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:04:43.321+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:43.323 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:04:43.321+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:04:43.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:43 smithi134 ceph-mon[31966]: pgmap v7640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:43 smithi189 ceph-mon[36790]: pgmap v7640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:43 smithi193 ceph-mon[37473]: pgmap v7640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:04:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13025.338883 secs 2024-07-26T15:04:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13025.338883 secs 2024-07-26T15:04:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13025.338883 secs 2024-07-26T15:04:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:45 smithi189 ceph-mon[36790]: pgmap v7641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:04:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:45 smithi193 ceph-mon[37473]: pgmap v7641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:04:45.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:45 smithi134 ceph-mon[31966]: pgmap v7641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:04:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13044.762775 secs 2024-07-26T15:04:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:47 smithi189 ceph-mon[36790]: pgmap v7642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:04:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13044.762775 secs 2024-07-26T15:04:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:47 smithi193 ceph-mon[37473]: pgmap v7642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:04:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13044.762775 secs 2024-07-26T15:04:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:47 smithi134 ceph-mon[31966]: pgmap v7642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:04:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:49 smithi189 ceph-mon[36790]: pgmap v7643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:04:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13030.338998 secs 2024-07-26T15:04:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:49 smithi193 ceph-mon[37473]: pgmap v7643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:04:49.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13030.338998 secs 2024-07-26T15:04:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:49 smithi134 ceph-mon[31966]: pgmap v7643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:04:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13030.338998 secs 2024-07-26T15:04:51.725 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:51 smithi134 ceph-mon[31966]: pgmap v7644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:04:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:51 smithi193 ceph-mon[37473]: pgmap v7644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:04:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:51 smithi189 ceph-mon[36790]: pgmap v7644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:04:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13049.762932 secs 2024-07-26T15:04:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13049.762932 secs 2024-07-26T15:04:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13049.762932 secs 2024-07-26T15:04:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:53 smithi193 ceph-mon[37473]: pgmap v7645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:04:53.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:04:53.867+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:04:53.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:04:53.867+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:04:53.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:04:53.867+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:53.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:53 smithi134 ceph-mon[31966]: pgmap v7645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:04:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:53 smithi189 ceph-mon[36790]: pgmap v7645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:04:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13035.339121 secs 2024-07-26T15:04:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13035.339121 secs 2024-07-26T15:04:54.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13035.339121 secs 2024-07-26T15:04:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:55 smithi193 ceph-mon[37473]: pgmap v7646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:04:55.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:55 smithi134 ceph-mon[31966]: pgmap v7646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:04:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:55 smithi189 ceph-mon[36790]: pgmap v7646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:04:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13054.763008 secs 2024-07-26T15:04:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:57 smithi193 ceph-mon[37473]: pgmap v7647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13054.763008 secs 2024-07-26T15:04:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:57 smithi134 ceph-mon[31966]: pgmap v7647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13054.763008 secs 2024-07-26T15:04:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:57 smithi189 ceph-mon[36790]: pgmap v7647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:04:58.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:04:58.336+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:04:58.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:04:58.336+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:04:58.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:04:58.336+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:04:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:59 smithi193 ceph-mon[37473]: pgmap v7648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:04:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:04:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13040.339255 secs 2024-07-26T15:04:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:59 smithi134 ceph-mon[31966]: pgmap v7648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:04:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:04:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13040.339255 secs 2024-07-26T15:04:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:59 smithi189 ceph-mon[36790]: pgmap v7648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:04:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:04:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13040.339255 secs 2024-07-26T15:05:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:01 smithi193 ceph-mon[37473]: pgmap v7649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:01.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:01 smithi134 ceph-mon[31966]: pgmap v7649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:01 smithi189 ceph-mon[36790]: pgmap v7649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13059.763146 secs 2024-07-26T15:05:02.857 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13059.763146 secs 2024-07-26T15:05:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13059.763146 secs 2024-07-26T15:05:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:03 smithi193 ceph-mon[37473]: pgmap v7650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:05:03.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:03 smithi134 ceph-mon[31966]: pgmap v7650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:05:03.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:03 smithi189 ceph-mon[36790]: pgmap v7650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:05:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13045.339395 secs 2024-07-26T15:05:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13045.339395 secs 2024-07-26T15:05:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13045.339395 secs 2024-07-26T15:05:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:05 smithi193 ceph-mon[37473]: pgmap v7651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:05.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:05 smithi134 ceph-mon[31966]: pgmap v7651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:05 smithi189 ceph-mon[36790]: pgmap v7651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13064.763268 secs 2024-07-26T15:05:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:07 smithi134 ceph-mon[31966]: pgmap v7652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13064.763268 secs 2024-07-26T15:05:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:07 smithi189 ceph-mon[36790]: pgmap v7652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13064.763268 secs 2024-07-26T15:05:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:07 smithi193 ceph-mon[37473]: pgmap v7652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:08.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:05:08.877+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:05:08.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:05:08.877+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:08.878 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:05:08.877+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:05:09.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:09 smithi193 ceph-mon[37473]: pgmap v7653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:05:09.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13050.339516 secs 2024-07-26T15:05:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:09 smithi134 ceph-mon[31966]: pgmap v7653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:05:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13050.339516 secs 2024-07-26T15:05:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:09 smithi189 ceph-mon[36790]: pgmap v7653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:05:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13050.339516 secs 2024-07-26T15:05:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:11 smithi134 ceph-mon[31966]: pgmap v7654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:11 smithi189 ceph-mon[36790]: pgmap v7654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:11 smithi193 ceph-mon[37473]: pgmap v7654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13069.763389 secs 2024-07-26T15:05:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13069.763389 secs 2024-07-26T15:05:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13069.763389 secs 2024-07-26T15:05:13.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:05:13.343+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:05:13.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:05:13.343+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:13.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:05:13.343+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:05:13.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:13 smithi134 ceph-mon[31966]: pgmap v7655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:13 smithi189 ceph-mon[36790]: pgmap v7655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:13 smithi193 ceph-mon[37473]: pgmap v7655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:05:14.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13055.339627 secs 2024-07-26T15:05:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13055.339627 secs 2024-07-26T15:05:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13055.339627 secs 2024-07-26T15:05:15.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:15 smithi134 ceph-mon[31966]: pgmap v7656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:15 smithi189 ceph-mon[36790]: pgmap v7656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:15 smithi193 ceph-mon[37473]: pgmap v7656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:05:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:05:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:05:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:05:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13074.763504 secs 2024-07-26T15:05:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:17 smithi134 ceph-mon[31966]: pgmap v7657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:05:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:05:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:05:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13074.763504 secs 2024-07-26T15:05:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:17 smithi189 ceph-mon[36790]: pgmap v7657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:05:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:05:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:05:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13074.763504 secs 2024-07-26T15:05:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:17 smithi193 ceph-mon[37473]: pgmap v7657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:05:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:05:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:05:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:18 smithi134 ceph-mon[31966]: pgmap v7658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:05:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:18 smithi134 ceph-mon[31966]: pgmap v7659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:05:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:18 smithi189 ceph-mon[36790]: pgmap v7658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:05:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:18 smithi189 ceph-mon[36790]: pgmap v7659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:05:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:18 smithi193 ceph-mon[37473]: pgmap v7658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:05:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:18 smithi193 ceph-mon[37473]: pgmap v7659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:05:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13060.339742 secs 2024-07-26T15:05:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13060.339742 secs 2024-07-26T15:05:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13060.339742 secs 2024-07-26T15:05:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:20 smithi134 ceph-mon[31966]: pgmap v7660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:05:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:20 smithi189 ceph-mon[36790]: pgmap v7660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:05:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:20 smithi193 ceph-mon[37473]: pgmap v7660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:05:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:21 smithi189 ceph-mon[36790]: pgmap v7661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:05:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:21 smithi193 ceph-mon[37473]: pgmap v7661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:05:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:21 smithi134 ceph-mon[31966]: pgmap v7661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:05:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13079.763619 secs 2024-07-26T15:05:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13079.763619 secs 2024-07-26T15:05:23.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13079.763619 secs 2024-07-26T15:05:23.881 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:05:23.880+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:05:23.881 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:05:23.880+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:23.881 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:05:23.880+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:05:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:23 smithi189 ceph-mon[36790]: pgmap v7662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:05:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:23 smithi193 ceph-mon[37473]: pgmap v7662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:05:24.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:23 smithi134 ceph-mon[31966]: pgmap v7662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:05:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13065.339863 secs 2024-07-26T15:05:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13065.339863 secs 2024-07-26T15:05:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13065.339863 secs 2024-07-26T15:05:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:25 smithi193 ceph-mon[37473]: pgmap v7663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:05:26.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:25 smithi134 ceph-mon[31966]: pgmap v7663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:05:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:25 smithi189 ceph-mon[36790]: pgmap v7663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:05:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13084.763697 secs 2024-07-26T15:05:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13084.763697 secs 2024-07-26T15:05:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13084.763697 secs 2024-07-26T15:05:28.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:27 smithi134 ceph-mon[31966]: pgmap v7664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:05:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:27 smithi189 ceph-mon[36790]: pgmap v7664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:05:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:27 smithi193 ceph-mon[37473]: pgmap v7664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:05:28.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:05:28.359+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:05:28.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:05:28.360+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:05:28.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:05:28.360+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:29.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13070.339984 secs 2024-07-26T15:05:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13070.339984 secs 2024-07-26T15:05:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13070.339984 secs 2024-07-26T15:05:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:30 smithi134 ceph-mon[31966]: pgmap v7665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:05:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:30 smithi189 ceph-mon[36790]: pgmap v7665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:05:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:30 smithi193 ceph-mon[37473]: pgmap v7665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:05:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13089.763837 secs 2024-07-26T15:05:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:32 smithi193 ceph-mon[37473]: pgmap v7666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:05:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13089.763837 secs 2024-07-26T15:05:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:32 smithi134 ceph-mon[31966]: pgmap v7666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:05:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13089.763837 secs 2024-07-26T15:05:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:32 smithi189 ceph-mon[36790]: pgmap v7666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:05:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:34 smithi193 ceph-mon[37473]: pgmap v7667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:05:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13075.340122 secs 2024-07-26T15:05:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:34 smithi134 ceph-mon[31966]: pgmap v7667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:05:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13075.340122 secs 2024-07-26T15:05:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:34 smithi189 ceph-mon[36790]: pgmap v7667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:05:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13075.340122 secs 2024-07-26T15:05:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:36 smithi193 ceph-mon[37473]: pgmap v7668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:05:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:36 smithi134 ceph-mon[31966]: pgmap v7668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:05:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:36 smithi189 ceph-mon[36790]: pgmap v7668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:05:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13094.763961 secs 2024-07-26T15:05:37.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13094.763961 secs 2024-07-26T15:05:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13094.763961 secs 2024-07-26T15:05:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:38 smithi193 ceph-mon[37473]: pgmap v7669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:05:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:05:38.894+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:05:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:05:38.894+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:05:38.895 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:05:38.894+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:38 smithi134 ceph-mon[31966]: pgmap v7669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:05:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:38 smithi189 ceph-mon[36790]: pgmap v7669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:05:39.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13080.340246 secs 2024-07-26T15:05:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13080.340246 secs 2024-07-26T15:05:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13080.340246 secs 2024-07-26T15:05:40.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:40 smithi193 ceph-mon[37473]: pgmap v7670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:05:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:40 smithi134 ceph-mon[31966]: pgmap v7670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:05:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:40 smithi189 ceph-mon[36790]: pgmap v7670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:05:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13099.764077 secs 2024-07-26T15:05:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:42 smithi193 ceph-mon[37473]: pgmap v7671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13099.764077 secs 2024-07-26T15:05:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:42 smithi134 ceph-mon[31966]: pgmap v7671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13099.764077 secs 2024-07-26T15:05:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:42 smithi189 ceph-mon[36790]: pgmap v7671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:43.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:05:43.374+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:05:43.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:05:43.374+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:43.376 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:05:43.374+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:05:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:44 smithi193 ceph-mon[37473]: pgmap v7672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:05:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13085.340368 secs 2024-07-26T15:05:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:44 smithi134 ceph-mon[31966]: pgmap v7672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:05:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13085.340368 secs 2024-07-26T15:05:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:44 smithi189 ceph-mon[36790]: pgmap v7672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:05:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13085.340368 secs 2024-07-26T15:05:46.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:46 smithi193 ceph-mon[37473]: pgmap v7673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:46 smithi134 ceph-mon[31966]: pgmap v7673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:46 smithi189 ceph-mon[36790]: pgmap v7673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13104.764216 secs 2024-07-26T15:05:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13104.764216 secs 2024-07-26T15:05:47.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13104.764216 secs 2024-07-26T15:05:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:48 smithi193 ceph-mon[37473]: pgmap v7674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:05:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:48 smithi134 ceph-mon[31966]: pgmap v7674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:05:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:48 smithi189 ceph-mon[36790]: pgmap v7674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:05:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13090.340496 secs 2024-07-26T15:05:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13090.340496 secs 2024-07-26T15:05:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13090.340496 secs 2024-07-26T15:05:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:50 smithi134 ceph-mon[31966]: pgmap v7675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:05:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:50 smithi189 ceph-mon[36790]: pgmap v7675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:05:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:50 smithi193 ceph-mon[37473]: pgmap v7675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:05:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13109.764369 secs 2024-07-26T15:05:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:52 smithi134 ceph-mon[31966]: pgmap v7676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13109.764369 secs 2024-07-26T15:05:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:52 smithi189 ceph-mon[36790]: pgmap v7676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13109.764369 secs 2024-07-26T15:05:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:52 smithi193 ceph-mon[37473]: pgmap v7676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:53.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:05:53.902+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:05:53.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:05:53.902+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:53.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:05:53.902+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:05:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:54 smithi134 ceph-mon[31966]: pgmap v7677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:05:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13095.340618 secs 2024-07-26T15:05:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:54 smithi189 ceph-mon[36790]: pgmap v7677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:05:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13095.340618 secs 2024-07-26T15:05:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:54 smithi193 ceph-mon[37473]: pgmap v7677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:05:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13095.340618 secs 2024-07-26T15:05:56.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:56 smithi134 ceph-mon[31966]: pgmap v7678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:56 smithi189 ceph-mon[36790]: pgmap v7678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:56 smithi193 ceph-mon[37473]: pgmap v7678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:05:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13114.764486 secs 2024-07-26T15:05:57.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13114.764486 secs 2024-07-26T15:05:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13114.764486 secs 2024-07-26T15:05:58.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:05:58.378+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:05:58.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:05:58.378+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:05:58.380 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:05:58.378+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:05:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:58 smithi134 ceph-mon[31966]: pgmap v7679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:05:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:58 smithi189 ceph-mon[36790]: pgmap v7679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:05:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:58 smithi193 ceph-mon[37473]: pgmap v7679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:05:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:05:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13100.340739 secs 2024-07-26T15:05:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:05:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13100.340739 secs 2024-07-26T15:06:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:05:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13100.340739 secs 2024-07-26T15:06:00.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:00 smithi134 ceph-mon[31966]: pgmap v7680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:06:00.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:00 smithi189 ceph-mon[36790]: pgmap v7680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:06:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:00 smithi193 ceph-mon[37473]: pgmap v7680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:06:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13119.764602 secs 2024-07-26T15:06:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:02 smithi134 ceph-mon[31966]: pgmap v7681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13119.764602 secs 2024-07-26T15:06:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:02 smithi189 ceph-mon[36790]: pgmap v7681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13119.764602 secs 2024-07-26T15:06:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:02 smithi193 ceph-mon[37473]: pgmap v7681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:04 smithi134 ceph-mon[31966]: pgmap v7682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13105.340876 secs 2024-07-26T15:06:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:04 smithi189 ceph-mon[36790]: pgmap v7682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13105.340876 secs 2024-07-26T15:06:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:04 smithi193 ceph-mon[37473]: pgmap v7682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13105.340876 secs 2024-07-26T15:06:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:06 smithi134 ceph-mon[31966]: pgmap v7683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:06.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:06 smithi189 ceph-mon[36790]: pgmap v7683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:06 smithi193 ceph-mon[37473]: pgmap v7683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13124.764683 secs 2024-07-26T15:06:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13124.764683 secs 2024-07-26T15:06:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13124.764683 secs 2024-07-26T15:06:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:06:08.918+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:06:08.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:06:08.918+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:08.919 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:06:08.918+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:06:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:08 smithi134 ceph-mon[31966]: pgmap v7684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:06:08.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:08 smithi189 ceph-mon[36790]: pgmap v7684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:06:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:08 smithi193 ceph-mon[37473]: pgmap v7684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:06:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13110.341043 secs 2024-07-26T15:06:09.924 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13110.341043 secs 2024-07-26T15:06:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13110.341043 secs 2024-07-26T15:06:10.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:10 smithi134 ceph-mon[31966]: pgmap v7685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:06:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:10 smithi189 ceph-mon[36790]: pgmap v7685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:06:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:10 smithi193 ceph-mon[37473]: pgmap v7685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:06:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13129.764804 secs 2024-07-26T15:06:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:12 smithi134 ceph-mon[31966]: pgmap v7686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13129.764804 secs 2024-07-26T15:06:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:12 smithi189 ceph-mon[36790]: pgmap v7686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13129.764804 secs 2024-07-26T15:06:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:12 smithi193 ceph-mon[37473]: pgmap v7686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:13.386 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:06:13.385+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:06:13.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:06:13.385+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:06:13.387 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:06:13.385+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:14 smithi134 ceph-mon[31966]: pgmap v7687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13115.341161 secs 2024-07-26T15:06:14.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:14 smithi189 ceph-mon[36790]: pgmap v7687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13115.341161 secs 2024-07-26T15:06:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:14 smithi193 ceph-mon[37473]: pgmap v7687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13115.341161 secs 2024-07-26T15:06:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:15 smithi189 ceph-mon[36790]: pgmap v7688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:15 smithi193 ceph-mon[37473]: pgmap v7688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:16.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:15 smithi134 ceph-mon[31966]: pgmap v7688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13134.764923 secs 2024-07-26T15:06:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13134.764923 secs 2024-07-26T15:06:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13134.764923 secs 2024-07-26T15:06:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:17 smithi134 ceph-mon[31966]: pgmap v7689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:06:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:06:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:17 smithi189 ceph-mon[36790]: pgmap v7689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:06:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:06:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:17 smithi193 ceph-mon[37473]: pgmap v7689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:06:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:06:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:18 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:06:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:18 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:06:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:06:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:06:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:18 smithi134 ceph-mon[31966]: pgmap v7690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:06:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:18 smithi134 ceph-mon[31966]: pgmap v7691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:06:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:18 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:06:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:18 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:06:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:06:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:06:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:18 smithi189 ceph-mon[36790]: pgmap v7690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:06:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:18 smithi189 ceph-mon[36790]: pgmap v7691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:06:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:06:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:18 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:06:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:18 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:06:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:06:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:06:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:18 smithi193 ceph-mon[37473]: pgmap v7690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:06:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:18 smithi193 ceph-mon[37473]: pgmap v7691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:06:20.168 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13120.341222 secs 2024-07-26T15:06:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13120.341222 secs 2024-07-26T15:06:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13120.341222 secs 2024-07-26T15:06:21.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:20 smithi134 ceph-mon[31966]: pgmap v7692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:21.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:20 smithi189 ceph-mon[36790]: pgmap v7692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:21.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:20 smithi193 ceph-mon[37473]: pgmap v7692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13139.765044 secs 2024-07-26T15:06:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13139.765044 secs 2024-07-26T15:06:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13139.765044 secs 2024-07-26T15:06:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:22 smithi189 ceph-mon[36790]: pgmap v7693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:22 smithi193 ceph-mon[37473]: pgmap v7693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:23.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:22 smithi134 ceph-mon[31966]: pgmap v7693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:06:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:06:23.933+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:06:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:06:23.933+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:06:23.934 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:06:23.933+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:24.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:06:24.616+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:06:24.617 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:06:24.616+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:06:24.617 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13125.341305 secs 2024-07-26T15:06:24.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:06:24.616+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T15:06:24.618 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:06:24.616+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 103 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T15:06:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13125.341305 secs 2024-07-26T15:06:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13125.341305 secs 2024-07-26T15:06:25.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:25 smithi134 ceph-mon[31966]: pgmap v7694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:06:25.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:25 smithi189 ceph-mon[36790]: pgmap v7694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:06:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:25 smithi193 ceph-mon[37473]: pgmap v7694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:06:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:26 smithi189 ceph-mon[36790]: pgmap v7695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:26 smithi193 ceph-mon[37473]: pgmap v7695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:26 smithi134 ceph-mon[31966]: pgmap v7695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13144.765156 secs 2024-07-26T15:06:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13144.765156 secs 2024-07-26T15:06:28.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13144.765156 secs 2024-07-26T15:06:28.390 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:06:28.389+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:06:28.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:06:28.389+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:28.391 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:06:28.389+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:06:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:28 smithi193 ceph-mon[37473]: pgmap v7696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:06:29.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:28 smithi134 ceph-mon[31966]: pgmap v7696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:06:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:28 smithi189 ceph-mon[36790]: pgmap v7696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:06:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13130.341419 secs 2024-07-26T15:06:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13130.341419 secs 2024-07-26T15:06:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13130.341419 secs 2024-07-26T15:06:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:30 smithi193 ceph-mon[37473]: pgmap v7697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:31.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:30 smithi134 ceph-mon[31966]: pgmap v7697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:30 smithi189 ceph-mon[36790]: pgmap v7697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13149.765292 secs 2024-07-26T15:06:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13149.765292 secs 2024-07-26T15:06:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13149.765292 secs 2024-07-26T15:06:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:32 smithi134 ceph-mon[31966]: pgmap v7698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:32 smithi189 ceph-mon[36790]: pgmap v7698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:32 smithi193 ceph-mon[37473]: pgmap v7698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:34.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13135.341538 secs 2024-07-26T15:06:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13135.341538 secs 2024-07-26T15:06:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13135.341538 secs 2024-07-26T15:06:35.149 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:06:35.148+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:06:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:06:35.148+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:06:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:06:35.149+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:06:35.150 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:06:35.149+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:06:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:35 smithi134 ceph-mon[31966]: pgmap v7699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:06:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:35 smithi189 ceph-mon[36790]: pgmap v7699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:06:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:35 smithi193 ceph-mon[37473]: pgmap v7699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:06:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:36 smithi189 ceph-mon[36790]: pgmap v7700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:36 smithi193 ceph-mon[37473]: pgmap v7700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:36 smithi134 ceph-mon[31966]: pgmap v7700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13154.765417 secs 2024-07-26T15:06:38.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13154.765417 secs 2024-07-26T15:06:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13154.765417 secs 2024-07-26T15:06:38.949 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:06:38.949+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:06:38.950 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:06:38.949+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:38.950 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:06:38.949+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:06:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:38 smithi193 ceph-mon[37473]: pgmap v7701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:39.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:38 smithi134 ceph-mon[31966]: pgmap v7701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:38 smithi189 ceph-mon[36790]: pgmap v7701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13140.341662 secs 2024-07-26T15:06:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13140.341662 secs 2024-07-26T15:06:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13140.341662 secs 2024-07-26T15:06:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:40 smithi193 ceph-mon[37473]: pgmap v7702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:41.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:40 smithi134 ceph-mon[31966]: pgmap v7702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:40 smithi189 ceph-mon[36790]: pgmap v7702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13159.765498 secs 2024-07-26T15:06:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13159.765498 secs 2024-07-26T15:06:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13159.765498 secs 2024-07-26T15:06:43.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:42 smithi134 ceph-mon[31966]: pgmap v7703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:42 smithi189 ceph-mon[36790]: pgmap v7703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:42 smithi193 ceph-mon[37473]: pgmap v7703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:43.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:06:43.405+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:06:43.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:06:43.405+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:06:43.406 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:06:43.405+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13145.341779 secs 2024-07-26T15:06:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13145.341779 secs 2024-07-26T15:06:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13145.341779 secs 2024-07-26T15:06:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:45 smithi134 ceph-mon[31966]: pgmap v7704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:06:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:45 smithi189 ceph-mon[36790]: pgmap v7704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:06:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:45 smithi193 ceph-mon[37473]: pgmap v7704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:06:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:46 smithi193 ceph-mon[37473]: pgmap v7705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:47.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:46 smithi134 ceph-mon[31966]: pgmap v7705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:46 smithi189 ceph-mon[36790]: pgmap v7705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13164.765620 secs 2024-07-26T15:06:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13164.765620 secs 2024-07-26T15:06:48.510 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13164.765620 secs 2024-07-26T15:06:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:48 smithi193 ceph-mon[37473]: pgmap v7706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:49.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:48 smithi134 ceph-mon[31966]: pgmap v7706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:48 smithi189 ceph-mon[36790]: pgmap v7706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13150.341895 secs 2024-07-26T15:06:50.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13150.341895 secs 2024-07-26T15:06:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13150.341895 secs 2024-07-26T15:06:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:50 smithi193 ceph-mon[37473]: pgmap v7707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:51.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:50 smithi134 ceph-mon[31966]: pgmap v7707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:50 smithi189 ceph-mon[36790]: pgmap v7707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13169.765720 secs 2024-07-26T15:06:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13169.765720 secs 2024-07-26T15:06:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13169.765720 secs 2024-07-26T15:06:53.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:52 smithi134 ceph-mon[31966]: pgmap v7708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:52 smithi189 ceph-mon[36790]: pgmap v7708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:52 smithi193 ceph-mon[37473]: pgmap v7708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:06:53.961+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:06:53.961 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:06:53.961+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:53.962 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:06:53.961+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:06:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13155.341995 secs 2024-07-26T15:06:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13155.341995 secs 2024-07-26T15:06:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13155.341995 secs 2024-07-26T15:06:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:55 smithi134 ceph-mon[31966]: pgmap v7709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:06:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:55 smithi189 ceph-mon[36790]: pgmap v7709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:06:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:55 smithi193 ceph-mon[37473]: pgmap v7709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:06:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:56 smithi193 ceph-mon[37473]: pgmap v7710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:56 smithi134 ceph-mon[31966]: pgmap v7710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:56 smithi189 ceph-mon[36790]: pgmap v7710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:06:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13174.765795 secs 2024-07-26T15:06:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13174.765795 secs 2024-07-26T15:06:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13174.765795 secs 2024-07-26T15:06:58.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:06:58.408+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:06:58.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:06:58.408+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:06:58.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:06:58.408+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:06:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:58 smithi193 ceph-mon[37473]: pgmap v7711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:58 smithi134 ceph-mon[31966]: pgmap v7711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:06:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:58 smithi189 ceph-mon[36790]: pgmap v7711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:06:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13160.342090 secs 2024-07-26T15:07:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:06:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13160.342090 secs 2024-07-26T15:07:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:06:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13160.342090 secs 2024-07-26T15:07:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:00 smithi193 ceph-mon[37473]: pgmap v7712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:00 smithi134 ceph-mon[31966]: pgmap v7712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:00 smithi189 ceph-mon[36790]: pgmap v7712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13179.765887 secs 2024-07-26T15:07:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13179.765887 secs 2024-07-26T15:07:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13179.765887 secs 2024-07-26T15:07:03.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:02 smithi134 ceph-mon[31966]: pgmap v7713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:02 smithi189 ceph-mon[36790]: pgmap v7713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:02 smithi193 ceph-mon[37473]: pgmap v7713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13165.342241 secs 2024-07-26T15:07:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13165.342241 secs 2024-07-26T15:07:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13165.342241 secs 2024-07-26T15:07:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:05 smithi134 ceph-mon[31966]: pgmap v7714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:05 smithi189 ceph-mon[36790]: pgmap v7714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:05 smithi193 ceph-mon[37473]: pgmap v7714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:06 smithi134 ceph-mon[31966]: pgmap v7715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:06 smithi193 ceph-mon[37473]: pgmap v7715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:06 smithi189 ceph-mon[36790]: pgmap v7715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13184.766009 secs 2024-07-26T15:07:08.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13184.766009 secs 2024-07-26T15:07:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13184.766009 secs 2024-07-26T15:07:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:07:08.975+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:07:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:07:08.976+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:07:08.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:07:08.976+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:08 smithi193 ceph-mon[37473]: pgmap v7716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:08 smithi134 ceph-mon[31966]: pgmap v7716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:08 smithi189 ceph-mon[36790]: pgmap v7716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13170.342365 secs 2024-07-26T15:07:10.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13170.342365 secs 2024-07-26T15:07:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13170.342365 secs 2024-07-26T15:07:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:10 smithi193 ceph-mon[37473]: pgmap v7717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:11.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:10 smithi134 ceph-mon[31966]: pgmap v7717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:10 smithi189 ceph-mon[36790]: pgmap v7717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13189.766090 secs 2024-07-26T15:07:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13189.766090 secs 2024-07-26T15:07:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13189.766090 secs 2024-07-26T15:07:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:12 smithi134 ceph-mon[31966]: pgmap v7718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:12 smithi189 ceph-mon[36790]: pgmap v7718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:12 smithi193 ceph-mon[37473]: pgmap v7718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:13.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:07:13.419+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:07:13.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:07:13.419+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:13.421 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:07:13.419+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:07:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13175.342481 secs 2024-07-26T15:07:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13175.342481 secs 2024-07-26T15:07:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13175.342481 secs 2024-07-26T15:07:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:15 smithi134 ceph-mon[31966]: pgmap v7719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:15 smithi189 ceph-mon[36790]: pgmap v7719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:15 smithi193 ceph-mon[37473]: pgmap v7719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:16 smithi193 ceph-mon[37473]: pgmap v7720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:17.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:16 smithi134 ceph-mon[31966]: pgmap v7720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:16 smithi189 ceph-mon[36790]: pgmap v7720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13194.766211 secs 2024-07-26T15:07:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13194.766211 secs 2024-07-26T15:07:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13194.766211 secs 2024-07-26T15:07:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:18 smithi193 ceph-mon[37473]: pgmap v7721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:19.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:18 smithi134 ceph-mon[31966]: pgmap v7721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:19.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:18 smithi189 ceph-mon[36790]: pgmap v7721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13180.342631 secs 2024-07-26T15:07:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:07:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:19 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:07:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:19 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:07:20.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:19 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:07:20.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:19 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:07:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13180.342631 secs 2024-07-26T15:07:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:07:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:19 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:07:20.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:19 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:07:20.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:19 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:07:20.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:19 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:07:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13180.342631 secs 2024-07-26T15:07:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:07:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:07:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:19 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:07:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:19 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:07:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:19 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:07:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:19 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:07:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:07:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:07:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:20 smithi193 ceph-mon[37473]: pgmap v7722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:07:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:20 smithi193 ceph-mon[37473]: pgmap v7723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:07:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:07:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:07:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:07:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:20 smithi134 ceph-mon[31966]: pgmap v7722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:07:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:20 smithi134 ceph-mon[31966]: pgmap v7723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:07:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:07:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:07:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:07:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:20 smithi189 ceph-mon[36790]: pgmap v7722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:07:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:20 smithi189 ceph-mon[36790]: pgmap v7723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:07:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:07:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13199.766355 secs 2024-07-26T15:07:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13199.766355 secs 2024-07-26T15:07:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13199.766355 secs 2024-07-26T15:07:23.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:22 smithi134 ceph-mon[31966]: pgmap v7724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:07:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:22 smithi189 ceph-mon[36790]: pgmap v7724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:07:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:22 smithi193 ceph-mon[37473]: pgmap v7724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:07:23.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:07:23.991+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:07:23.992 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:07:23.991+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:23.993 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:07:23.991+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:07:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13185.342758 secs 2024-07-26T15:07:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:24 smithi134 ceph-mon[31966]: pgmap v7725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13185.342758 secs 2024-07-26T15:07:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:24 smithi189 ceph-mon[36790]: pgmap v7725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13185.342758 secs 2024-07-26T15:07:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:24 smithi193 ceph-mon[37473]: pgmap v7725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:26 smithi134 ceph-mon[31966]: pgmap v7726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13204.766485 secs 2024-07-26T15:07:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:26 smithi189 ceph-mon[36790]: pgmap v7726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13204.766485 secs 2024-07-26T15:07:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:26 smithi193 ceph-mon[37473]: pgmap v7726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13204.766485 secs 2024-07-26T15:07:28.435 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:07:28.434+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:07:28.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:07:28.434+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:28.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:07:28.434+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:07:29.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:28 smithi134 ceph-mon[31966]: pgmap v7727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:28 smithi189 ceph-mon[36790]: pgmap v7727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:28 smithi193 ceph-mon[37473]: pgmap v7727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:07:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13190.342881 secs 2024-07-26T15:07:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13190.342881 secs 2024-07-26T15:07:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13190.342881 secs 2024-07-26T15:07:31.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:30 smithi134 ceph-mon[31966]: pgmap v7728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:07:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:30 smithi189 ceph-mon[36790]: pgmap v7728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:07:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:30 smithi193 ceph-mon[37473]: pgmap v7728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:07:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13209.766616 secs 2024-07-26T15:07:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13209.766616 secs 2024-07-26T15:07:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13209.766616 secs 2024-07-26T15:07:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:32 smithi189 ceph-mon[36790]: pgmap v7729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:32 smithi193 ceph-mon[37473]: pgmap v7729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:32 smithi134 ceph-mon[31966]: pgmap v7729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13195.343000 secs 2024-07-26T15:07:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:34 smithi189 ceph-mon[36790]: pgmap v7730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13195.343000 secs 2024-07-26T15:07:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:34 smithi193 ceph-mon[37473]: pgmap v7730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13195.343000 secs 2024-07-26T15:07:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:34 smithi134 ceph-mon[31966]: pgmap v7730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:36 smithi193 ceph-mon[37473]: pgmap v7731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13214.766732 secs 2024-07-26T15:07:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:36 smithi134 ceph-mon[31966]: pgmap v7731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:37.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13214.766732 secs 2024-07-26T15:07:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:36 smithi189 ceph-mon[36790]: pgmap v7731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13214.766732 secs 2024-07-26T15:07:39.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:07:39.007+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:07:39.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:07:39.007+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:39.008 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:07:39.007+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:07:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:38 smithi193 ceph-mon[37473]: pgmap v7732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:38 smithi134 ceph-mon[31966]: pgmap v7732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:38 smithi189 ceph-mon[36790]: pgmap v7732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13200.343096 secs 2024-07-26T15:07:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13200.343096 secs 2024-07-26T15:07:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13200.343096 secs 2024-07-26T15:07:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:40 smithi193 ceph-mon[37473]: pgmap v7733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:41.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:40 smithi134 ceph-mon[31966]: pgmap v7733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:40 smithi189 ceph-mon[36790]: pgmap v7733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13219.766879 secs 2024-07-26T15:07:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13219.766879 secs 2024-07-26T15:07:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13219.766879 secs 2024-07-26T15:07:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:42 smithi193 ceph-mon[37473]: pgmap v7734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:43.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:42 smithi134 ceph-mon[31966]: pgmap v7734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:42 smithi189 ceph-mon[36790]: pgmap v7734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:43.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:07:43.449+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:07:43.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:07:43.450+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:07:43.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:07:43.450+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:45.245 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13205.343182 secs 2024-07-26T15:07:45.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:44 smithi189 ceph-mon[36790]: pgmap v7735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13205.343182 secs 2024-07-26T15:07:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:44 smithi193 ceph-mon[37473]: pgmap v7735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13205.343182 secs 2024-07-26T15:07:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:44 smithi134 ceph-mon[31966]: pgmap v7735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:46 smithi193 ceph-mon[37473]: pgmap v7736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13224.766941 secs 2024-07-26T15:07:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:46 smithi134 ceph-mon[31966]: pgmap v7736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13224.766941 secs 2024-07-26T15:07:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:46 smithi189 ceph-mon[36790]: pgmap v7736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13224.766941 secs 2024-07-26T15:07:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:48 smithi193 ceph-mon[37473]: pgmap v7737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:48 smithi134 ceph-mon[31966]: pgmap v7737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:48 smithi189 ceph-mon[36790]: pgmap v7737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13210.343301 secs 2024-07-26T15:07:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13210.343301 secs 2024-07-26T15:07:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13210.343301 secs 2024-07-26T15:07:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:51 smithi193 ceph-mon[37473]: pgmap v7738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:51.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:51 smithi134 ceph-mon[31966]: pgmap v7738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:51 smithi189 ceph-mon[36790]: pgmap v7738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:07:52.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13229.767089 secs 2024-07-26T15:07:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13229.767089 secs 2024-07-26T15:07:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13229.767089 secs 2024-07-26T15:07:53.352 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:53 smithi134 ceph-mon[31966]: pgmap v7739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:53 smithi189 ceph-mon[36790]: pgmap v7739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:53 smithi193 ceph-mon[37473]: pgmap v7739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:54.011 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:07:54.009+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:07:54.011 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:07:54.009+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:07:54.011 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:07:54.009+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13215.343420 secs 2024-07-26T15:07:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:55 smithi134 ceph-mon[31966]: pgmap v7740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13215.343420 secs 2024-07-26T15:07:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:55 smithi189 ceph-mon[36790]: pgmap v7740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13215.343420 secs 2024-07-26T15:07:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:55 smithi193 ceph-mon[37473]: pgmap v7740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:57 smithi134 ceph-mon[31966]: pgmap v7741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13234.767230 secs 2024-07-26T15:07:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:57 smithi189 ceph-mon[36790]: pgmap v7741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13234.767230 secs 2024-07-26T15:07:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:57 smithi193 ceph-mon[37473]: pgmap v7741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:07:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13234.767230 secs 2024-07-26T15:07:58.465 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:07:58.464+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:07:58.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:07:58.464+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:07:58.466 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:07:58.464+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:07:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:07:59 smithi134 ceph-mon[31966]: pgmap v7742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:07:59 smithi189 ceph-mon[36790]: pgmap v7742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:07:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:07:59 smithi193 ceph-mon[37473]: pgmap v7742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13220.343541 secs 2024-07-26T15:08:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13220.343541 secs 2024-07-26T15:08:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13220.343541 secs 2024-07-26T15:08:01.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:01 smithi134 ceph-mon[31966]: pgmap v7743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:01 smithi189 ceph-mon[36790]: pgmap v7743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:01.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:01 smithi193 ceph-mon[37473]: pgmap v7743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13239.767361 secs 2024-07-26T15:08:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13239.767361 secs 2024-07-26T15:08:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13239.767361 secs 2024-07-26T15:08:03.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:03 smithi134 ceph-mon[31966]: pgmap v7744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:03 smithi189 ceph-mon[36790]: pgmap v7744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:03.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:03 smithi193 ceph-mon[37473]: pgmap v7744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:05.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13225.343684 secs 2024-07-26T15:08:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:05 smithi134 ceph-mon[31966]: pgmap v7745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13225.343684 secs 2024-07-26T15:08:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:05 smithi189 ceph-mon[36790]: pgmap v7745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13225.343684 secs 2024-07-26T15:08:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:05 smithi193 ceph-mon[37473]: pgmap v7745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:07 smithi134 ceph-mon[31966]: pgmap v7746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13244.767505 secs 2024-07-26T15:08:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:07 smithi189 ceph-mon[36790]: pgmap v7746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13244.767505 secs 2024-07-26T15:08:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:07 smithi193 ceph-mon[37473]: pgmap v7746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13244.767505 secs 2024-07-26T15:08:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:08:09.010+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:08:09.011 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:08:09.010+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:09.012 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:08:09.010+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:08:09.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:09 smithi134 ceph-mon[31966]: pgmap v7747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:09 smithi189 ceph-mon[36790]: pgmap v7747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:09 smithi193 ceph-mon[37473]: pgmap v7747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:10.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13230.343803 secs 2024-07-26T15:08:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13230.343803 secs 2024-07-26T15:08:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13230.343803 secs 2024-07-26T15:08:11.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:11 smithi134 ceph-mon[31966]: pgmap v7748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:11 smithi189 ceph-mon[36790]: pgmap v7748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:11 smithi193 ceph-mon[37473]: pgmap v7748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:12.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13249.767596 secs 2024-07-26T15:08:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13249.767596 secs 2024-07-26T15:08:12.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13249.767596 secs 2024-07-26T15:08:13.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:13 smithi134 ceph-mon[31966]: pgmap v7749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:13 smithi189 ceph-mon[36790]: pgmap v7749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:13.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:08:13.480+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:08:13.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:08:13.480+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:08:13.481 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:08:13.480+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:13 smithi193 ceph-mon[37473]: pgmap v7749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13235.343925 secs 2024-07-26T15:08:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:15 smithi134 ceph-mon[31966]: pgmap v7750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13235.343925 secs 2024-07-26T15:08:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:15 smithi189 ceph-mon[36790]: pgmap v7750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13235.343925 secs 2024-07-26T15:08:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:15 smithi193 ceph-mon[37473]: pgmap v7750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:17 smithi134 ceph-mon[31966]: pgmap v7751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13254.767664 secs 2024-07-26T15:08:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:17 smithi189 ceph-mon[36790]: pgmap v7751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13254.767664 secs 2024-07-26T15:08:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:17 smithi193 ceph-mon[37473]: pgmap v7751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13254.767664 secs 2024-07-26T15:08:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:19 smithi134 ceph-mon[31966]: pgmap v7752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13240.344017 secs 2024-07-26T15:08:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:19 smithi189 ceph-mon[36790]: pgmap v7752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13240.344017 secs 2024-07-26T15:08:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:19 smithi193 ceph-mon[37473]: pgmap v7752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13240.344017 secs 2024-07-26T15:08:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:08:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:08:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:08:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:21 smithi189 ceph-mon[36790]: pgmap v7753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:08:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:08:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:08:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:21 smithi193 ceph-mon[37473]: pgmap v7753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:08:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:08:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:08:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:21 smithi134 ceph-mon[31966]: pgmap v7753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:08:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:08:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:08:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:08:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:22 smithi189 ceph-mon[36790]: pgmap v7754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:08:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:22 smithi189 ceph-mon[36790]: pgmap v7755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:08:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13259.767778 secs 2024-07-26T15:08:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:22 smithi193 ceph-mon[37473]: pgmap v7754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:08:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:22 smithi193 ceph-mon[37473]: pgmap v7755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:08:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13259.767778 secs 2024-07-26T15:08:22.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:22 smithi134 ceph-mon[31966]: pgmap v7754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:08:22.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:22 smithi134 ceph-mon[31966]: pgmap v7755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:08:22.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13259.767778 secs 2024-07-26T15:08:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:08:24.025+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:08:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:08:24.026+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:08:24.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:08:24.026+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:24 smithi193 ceph-mon[37473]: pgmap v7756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:08:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13245.344147 secs 2024-07-26T15:08:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:24 smithi134 ceph-mon[31966]: pgmap v7756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:08:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13245.344147 secs 2024-07-26T15:08:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:24 smithi189 ceph-mon[36790]: pgmap v7756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:08:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13245.344147 secs 2024-07-26T15:08:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:26 smithi193 ceph-mon[37473]: pgmap v7757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:08:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:26 smithi134 ceph-mon[31966]: pgmap v7757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:08:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:26 smithi189 ceph-mon[36790]: pgmap v7757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:08:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13264.767843 secs 2024-07-26T15:08:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13264.767843 secs 2024-07-26T15:08:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13264.767843 secs 2024-07-26T15:08:28.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:08:28.493+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:08:28.494 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:08:28.493+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:08:28.495 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:08:28.493+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:28 smithi193 ceph-mon[37473]: pgmap v7758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:08:28.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:28 smithi134 ceph-mon[31966]: pgmap v7758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:08:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:28 smithi189 ceph-mon[36790]: pgmap v7758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:08:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13250.344234 secs 2024-07-26T15:08:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13250.344234 secs 2024-07-26T15:08:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13250.344234 secs 2024-07-26T15:08:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:30 smithi193 ceph-mon[37473]: pgmap v7759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:08:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:30 smithi134 ceph-mon[31966]: pgmap v7759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:08:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:30 smithi189 ceph-mon[36790]: pgmap v7759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:08:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:32 smithi193 ceph-mon[37473]: pgmap v7760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:08:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13269.767964 secs 2024-07-26T15:08:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:32 smithi134 ceph-mon[31966]: pgmap v7760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:08:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13269.767964 secs 2024-07-26T15:08:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:32 smithi189 ceph-mon[36790]: pgmap v7760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:08:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13269.767964 secs 2024-07-26T15:08:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:34 smithi193 ceph-mon[37473]: pgmap v7761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:08:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13255.344356 secs 2024-07-26T15:08:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:34 smithi134 ceph-mon[31966]: pgmap v7761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:08:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13255.344356 secs 2024-07-26T15:08:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:34 smithi189 ceph-mon[36790]: pgmap v7761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:08:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13255.344356 secs 2024-07-26T15:08:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:36 smithi193 ceph-mon[37473]: pgmap v7762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:08:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:36 smithi134 ceph-mon[31966]: pgmap v7762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:08:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:36 smithi189 ceph-mon[36790]: pgmap v7762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:08:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13274.768101 secs 2024-07-26T15:08:37.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13274.768101 secs 2024-07-26T15:08:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13274.768101 secs 2024-07-26T15:08:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:38 smithi134 ceph-mon[31966]: pgmap v7763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T15:08:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:38 smithi189 ceph-mon[36790]: pgmap v7763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T15:08:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:38 smithi193 ceph-mon[37473]: pgmap v7763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T15:08:39.037 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:08:39.036+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:08:39.037 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:08:39.036+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:08:39.037 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:08:39.036+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13260.344455 secs 2024-07-26T15:08:39.924 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13260.344455 secs 2024-07-26T15:08:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13260.344455 secs 2024-07-26T15:08:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:40 smithi134 ceph-mon[31966]: pgmap v7764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:08:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:40 smithi189 ceph-mon[36790]: pgmap v7764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:08:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:40 smithi193 ceph-mon[37473]: pgmap v7764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:08:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:42 smithi134 ceph-mon[31966]: pgmap v7765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13279.768230 secs 2024-07-26T15:08:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:42 smithi189 ceph-mon[36790]: pgmap v7765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13279.768230 secs 2024-07-26T15:08:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:42 smithi193 ceph-mon[37473]: pgmap v7765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13279.768230 secs 2024-07-26T15:08:43.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:08:43.508+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:08:43.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:08:43.508+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:43.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:08:43.508+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:08:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:44 smithi134 ceph-mon[31966]: pgmap v7766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13265.344596 secs 2024-07-26T15:08:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:44 smithi189 ceph-mon[36790]: pgmap v7766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13265.344596 secs 2024-07-26T15:08:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:44 smithi193 ceph-mon[37473]: pgmap v7766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13265.344596 secs 2024-07-26T15:08:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:46 smithi134 ceph-mon[31966]: pgmap v7767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:46 smithi189 ceph-mon[36790]: pgmap v7767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:46 smithi193 ceph-mon[37473]: pgmap v7767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13284.768348 secs 2024-07-26T15:08:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13284.768348 secs 2024-07-26T15:08:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13284.768348 secs 2024-07-26T15:08:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:48 smithi134 ceph-mon[31966]: pgmap v7768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:08:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:48 smithi189 ceph-mon[36790]: pgmap v7768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:08:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:48 smithi193 ceph-mon[37473]: pgmap v7768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:08:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13270.344736 secs 2024-07-26T15:08:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13270.344736 secs 2024-07-26T15:08:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13270.344736 secs 2024-07-26T15:08:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:50 smithi134 ceph-mon[31966]: pgmap v7769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:08:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:50 smithi189 ceph-mon[36790]: pgmap v7769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:08:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:50 smithi193 ceph-mon[37473]: pgmap v7769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:08:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:52 smithi134 ceph-mon[31966]: pgmap v7770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13289.768479 secs 2024-07-26T15:08:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:52 smithi189 ceph-mon[36790]: pgmap v7770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13289.768479 secs 2024-07-26T15:08:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:52 smithi193 ceph-mon[37473]: pgmap v7770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13289.768479 secs 2024-07-26T15:08:54.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:08:54.052+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:08:54.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:08:54.052+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:08:54.053 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:08:54.052+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:54 smithi134 ceph-mon[31966]: pgmap v7771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13275.344848 secs 2024-07-26T15:08:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:54 smithi189 ceph-mon[36790]: pgmap v7771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13275.344848 secs 2024-07-26T15:08:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:54 smithi193 ceph-mon[37473]: pgmap v7771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:08:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13275.344848 secs 2024-07-26T15:08:56.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:56 smithi134 ceph-mon[31966]: pgmap v7772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:56 smithi189 ceph-mon[36790]: pgmap v7772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:56 smithi193 ceph-mon[37473]: pgmap v7772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:08:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13294.768602 secs 2024-07-26T15:08:57.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13294.768602 secs 2024-07-26T15:08:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13294.768602 secs 2024-07-26T15:08:58.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:08:58.519+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:08:58.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:08:58.519+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:08:58.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:08:58.519+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:08:58.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:58 smithi134 ceph-mon[31966]: pgmap v7773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:08:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:58 smithi189 ceph-mon[36790]: pgmap v7773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:08:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:58 smithi193 ceph-mon[37473]: pgmap v7773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:08:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:08:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13280.344970 secs 2024-07-26T15:08:59.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:08:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13280.344970 secs 2024-07-26T15:09:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:08:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13280.344970 secs 2024-07-26T15:09:00.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:00 smithi134 ceph-mon[31966]: pgmap v7774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:09:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:00 smithi189 ceph-mon[36790]: pgmap v7774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:09:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:00 smithi193 ceph-mon[37473]: pgmap v7774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:09:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:02 smithi134 ceph-mon[31966]: pgmap v7775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13299.768735 secs 2024-07-26T15:09:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:02 smithi189 ceph-mon[36790]: pgmap v7775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13299.768735 secs 2024-07-26T15:09:03.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:02 smithi193 ceph-mon[37473]: pgmap v7775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:03.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13299.768735 secs 2024-07-26T15:09:04.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:04 smithi134 ceph-mon[31966]: pgmap v7776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13285.345034 secs 2024-07-26T15:09:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:04 smithi189 ceph-mon[36790]: pgmap v7776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13285.345034 secs 2024-07-26T15:09:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:04 smithi193 ceph-mon[37473]: pgmap v7776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13285.345034 secs 2024-07-26T15:09:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:06 smithi134 ceph-mon[31966]: pgmap v7777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:06 smithi189 ceph-mon[36790]: pgmap v7777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:06 smithi193 ceph-mon[37473]: pgmap v7777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13304.768866 secs 2024-07-26T15:09:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13304.768866 secs 2024-07-26T15:09:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13304.768866 secs 2024-07-26T15:09:08.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:08 smithi134 ceph-mon[31966]: pgmap v7778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:08 smithi189 ceph-mon[36790]: pgmap v7778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:08 smithi193 ceph-mon[37473]: pgmap v7778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:09.068 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:09:09.067+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:09:09.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:09:09.067+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:09:09.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:09:09.067+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13290.345157 secs 2024-07-26T15:09:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13290.345157 secs 2024-07-26T15:09:10.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13290.345157 secs 2024-07-26T15:09:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:10 smithi189 ceph-mon[36790]: pgmap v7779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:09:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:10 smithi193 ceph-mon[37473]: pgmap v7779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:09:11.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:10 smithi134 ceph-mon[31966]: pgmap v7779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:09:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:11 smithi189 ceph-mon[36790]: pgmap v7780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:11 smithi193 ceph-mon[37473]: pgmap v7780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:12.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:11 smithi134 ceph-mon[31966]: pgmap v7780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13309.768995 secs 2024-07-26T15:09:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13309.768995 secs 2024-07-26T15:09:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13309.768995 secs 2024-07-26T15:09:13.535 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:09:13.534+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:09:13.535 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:09:13.534+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:13.536 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:09:13.534+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:09:13.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:13 smithi134 ceph-mon[31966]: pgmap v7781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:13 smithi193 ceph-mon[37473]: pgmap v7781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:13 smithi189 ceph-mon[36790]: pgmap v7781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13295.345272 secs 2024-07-26T15:09:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13295.345272 secs 2024-07-26T15:09:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13295.345272 secs 2024-07-26T15:09:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:15 smithi193 ceph-mon[37473]: pgmap v7782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:16.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:15 smithi189 ceph-mon[36790]: pgmap v7782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:16.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:15 smithi134 ceph-mon[31966]: pgmap v7782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:17.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13314.769130 secs 2024-07-26T15:09:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13314.769130 secs 2024-07-26T15:09:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13314.769130 secs 2024-07-26T15:09:18.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:17 smithi134 ceph-mon[31966]: pgmap v7783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:17 smithi189 ceph-mon[36790]: pgmap v7783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:17 smithi193 ceph-mon[37473]: pgmap v7783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:09:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13300.345363 secs 2024-07-26T15:09:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:20 smithi134 ceph-mon[31966]: pgmap v7784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:09:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13300.345363 secs 2024-07-26T15:09:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:20 smithi189 ceph-mon[36790]: pgmap v7784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:09:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13300.345363 secs 2024-07-26T15:09:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:20 smithi193 ceph-mon[37473]: pgmap v7784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:09:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:22 smithi134 ceph-mon[31966]: pgmap v7785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:22.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13319.769234 secs 2024-07-26T15:09:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:09:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:09:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:09:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:22 smithi189 ceph-mon[36790]: pgmap v7785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13319.769234 secs 2024-07-26T15:09:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:09:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:09:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:09:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:22 smithi193 ceph-mon[37473]: pgmap v7785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:22.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13319.769234 secs 2024-07-26T15:09:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:09:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:09:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:09:23.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:09:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:09:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:09:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:09:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:09:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:09:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:09:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:09:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:09:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:09:24.083+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:09:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:09:24.083+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:09:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:09:24.083+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:24.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:24 smithi134 ceph-mon[31966]: pgmap v7786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T15:09:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:24 smithi134 ceph-mon[31966]: pgmap v7787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:09:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:24 smithi189 ceph-mon[36790]: pgmap v7786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T15:09:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:24 smithi189 ceph-mon[36790]: pgmap v7787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:09:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:24 smithi193 ceph-mon[37473]: pgmap v7786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T15:09:24.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:24 smithi193 ceph-mon[37473]: pgmap v7787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:09:24.647 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:09:24.646+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:09:24.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:09:24.646+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:09:24.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:09:24.647+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 104 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:09:24.648 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:09:24.647+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:09:25.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13305.345490 secs 2024-07-26T15:09:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13305.345490 secs 2024-07-26T15:09:25.527 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13305.345490 secs 2024-07-26T15:09:26.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:26 smithi134 ceph-mon[31966]: pgmap v7788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:09:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:26 smithi189 ceph-mon[36790]: pgmap v7788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:09:26.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:26 smithi193 ceph-mon[37473]: pgmap v7788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:09:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13324.769387 secs 2024-07-26T15:09:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13324.769387 secs 2024-07-26T15:09:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13324.769387 secs 2024-07-26T15:09:28.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:28 smithi134 ceph-mon[31966]: pgmap v7789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:09:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:28 smithi189 ceph-mon[36790]: pgmap v7789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:09:28.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:28 smithi193 ceph-mon[37473]: pgmap v7789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:09:28.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:09:28.549+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:09:28.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:09:28.549+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:28.551 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:09:28.549+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:09:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13310.345588 secs 2024-07-26T15:09:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:30 smithi134 ceph-mon[31966]: pgmap v7790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:09:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13310.345588 secs 2024-07-26T15:09:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:30 smithi189 ceph-mon[36790]: pgmap v7790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:09:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13310.345588 secs 2024-07-26T15:09:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:30 smithi193 ceph-mon[37473]: pgmap v7790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:09:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:32 smithi134 ceph-mon[31966]: pgmap v7791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:09:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13329.769521 secs 2024-07-26T15:09:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:32 smithi189 ceph-mon[36790]: pgmap v7791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:09:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13329.769521 secs 2024-07-26T15:09:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:32 smithi193 ceph-mon[37473]: pgmap v7791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:09:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13329.769521 secs 2024-07-26T15:09:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:34 smithi134 ceph-mon[31966]: pgmap v7792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:09:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:34 smithi189 ceph-mon[36790]: pgmap v7792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:09:34.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:34 smithi193 ceph-mon[37473]: pgmap v7792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:09:35.181 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:09:35.180+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:09:35.181 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:09:35.180+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:09:35.187 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:09:35.186+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:09:35.187 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:09:35.186+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:09:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13315.345707 secs 2024-07-26T15:09:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13315.345707 secs 2024-07-26T15:09:35.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13315.345707 secs 2024-07-26T15:09:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:36 smithi189 ceph-mon[36790]: pgmap v7793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:36.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:36 smithi193 ceph-mon[37473]: pgmap v7793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:36 smithi134 ceph-mon[31966]: pgmap v7793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:37.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13334.769586 secs 2024-07-26T15:09:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13334.769586 secs 2024-07-26T15:09:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13334.769586 secs 2024-07-26T15:09:38.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:38 smithi193 ceph-mon[37473]: pgmap v7794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:38.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:38 smithi134 ceph-mon[31966]: pgmap v7794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:38 smithi189 ceph-mon[36790]: pgmap v7794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:39.099 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:09:39.099+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:09:39.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:09:39.099+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:39.100 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:09:39.099+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:09:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13320.345847 secs 2024-07-26T15:09:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:40 smithi193 ceph-mon[37473]: pgmap v7795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:40.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13320.345847 secs 2024-07-26T15:09:40.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:40 smithi134 ceph-mon[31966]: pgmap v7795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13320.345847 secs 2024-07-26T15:09:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:40 smithi189 ceph-mon[36790]: pgmap v7795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:42 smithi193 ceph-mon[37473]: pgmap v7796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13339.769706 secs 2024-07-26T15:09:42.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:42 smithi134 ceph-mon[31966]: pgmap v7796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13339.769706 secs 2024-07-26T15:09:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:42 smithi189 ceph-mon[36790]: pgmap v7796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13339.769706 secs 2024-07-26T15:09:43.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:09:43.557+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:09:43.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:09:43.557+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:43.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:09:43.557+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:09:44.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:44 smithi193 ceph-mon[37473]: pgmap v7797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:44 smithi134 ceph-mon[31966]: pgmap v7797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:44 smithi189 ceph-mon[36790]: pgmap v7797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13325.346001 secs 2024-07-26T15:09:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13325.346001 secs 2024-07-26T15:09:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13325.346001 secs 2024-07-26T15:09:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:46 smithi193 ceph-mon[37473]: pgmap v7798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:46.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:46 smithi189 ceph-mon[36790]: pgmap v7798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:46.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:46 smithi134 ceph-mon[31966]: pgmap v7798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13344.769821 secs 2024-07-26T15:09:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13344.769821 secs 2024-07-26T15:09:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13344.769821 secs 2024-07-26T15:09:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:48 smithi193 ceph-mon[37473]: pgmap v7799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:48.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:48 smithi134 ceph-mon[31966]: pgmap v7799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:48 smithi189 ceph-mon[36790]: pgmap v7799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13330.346123 secs 2024-07-26T15:09:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:50 smithi193 ceph-mon[37473]: pgmap v7800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:50.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13330.346123 secs 2024-07-26T15:09:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:50 smithi134 ceph-mon[31966]: pgmap v7800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13330.346123 secs 2024-07-26T15:09:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:50 smithi189 ceph-mon[36790]: pgmap v7800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:52 smithi193 ceph-mon[37473]: pgmap v7801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13349.769939 secs 2024-07-26T15:09:52.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:52 smithi134 ceph-mon[31966]: pgmap v7801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13349.769939 secs 2024-07-26T15:09:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:52 smithi189 ceph-mon[36790]: pgmap v7801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13349.769939 secs 2024-07-26T15:09:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:09:54.111+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:09:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:09:54.111+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:09:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:09:54.111+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:54.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:54 smithi193 ceph-mon[37473]: pgmap v7802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:54 smithi134 ceph-mon[31966]: pgmap v7802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:54 smithi189 ceph-mon[36790]: pgmap v7802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:09:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13335.346224 secs 2024-07-26T15:09:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13335.346224 secs 2024-07-26T15:09:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13335.346224 secs 2024-07-26T15:09:56.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:56 smithi193 ceph-mon[37473]: pgmap v7803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:56 smithi134 ceph-mon[31966]: pgmap v7803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:56 smithi189 ceph-mon[36790]: pgmap v7803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:09:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13354.770058 secs 2024-07-26T15:09:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13354.770058 secs 2024-07-26T15:09:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13354.770058 secs 2024-07-26T15:09:58.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:09:58.558+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:09:58.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:09:58.558+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:09:58.560 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:09:58.558+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:09:58.560 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:09:58 smithi134 ceph-mon[31966]: pgmap v7804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:09:58 smithi189 ceph-mon[36790]: pgmap v7804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:09:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:09:58 smithi193 ceph-mon[37473]: pgmap v7804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13340.346366 secs 2024-07-26T15:10:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:00 smithi134 ceph-mon[31966]: pgmap v7805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:10:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13340.346366 secs 2024-07-26T15:10:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:00 smithi189 ceph-mon[36790]: pgmap v7805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:10:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13340.346366 secs 2024-07-26T15:10:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:00 smithi193 ceph-mon[37473]: pgmap v7805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:10:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:02 smithi134 ceph-mon[31966]: pgmap v7806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13359.770200 secs 2024-07-26T15:10:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:02 smithi189 ceph-mon[36790]: pgmap v7806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13359.770200 secs 2024-07-26T15:10:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:02 smithi193 ceph-mon[37473]: pgmap v7806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13359.770200 secs 2024-07-26T15:10:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:04 smithi134 ceph-mon[31966]: pgmap v7807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:04 smithi189 ceph-mon[36790]: pgmap v7807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:04 smithi193 ceph-mon[37473]: pgmap v7807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13345.346472 secs 2024-07-26T15:10:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13345.346472 secs 2024-07-26T15:10:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13345.346472 secs 2024-07-26T15:10:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:06 smithi134 ceph-mon[31966]: pgmap v7808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:06 smithi189 ceph-mon[36790]: pgmap v7808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:06 smithi193 ceph-mon[37473]: pgmap v7808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:07.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13364.770332 secs 2024-07-26T15:10:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13364.770332 secs 2024-07-26T15:10:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13364.770332 secs 2024-07-26T15:10:08.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:08 smithi134 ceph-mon[31966]: pgmap v7809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:08 smithi189 ceph-mon[36790]: pgmap v7809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:08 smithi193 ceph-mon[37473]: pgmap v7809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:10:09.115+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:10:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:10:09.115+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:09.116 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:10:09.115+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:10:09.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:10:09.654+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:10:09.656 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:10:09.654+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:10:09.656 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13350.346616 secs 2024-07-26T15:10:09.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:10:09.655+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T15:10:09.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:10:09.655+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 105 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T15:10:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13350.346616 secs 2024-07-26T15:10:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13350.346616 secs 2024-07-26T15:10:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:10 smithi134 ceph-mon[31966]: pgmap v7810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:10.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:10 smithi193 ceph-mon[37473]: pgmap v7810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:10 smithi189 ceph-mon[36790]: pgmap v7810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:12 smithi193 ceph-mon[37473]: pgmap v7811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13369.770430 secs 2024-07-26T15:10:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:12 smithi134 ceph-mon[31966]: pgmap v7811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13369.770430 secs 2024-07-26T15:10:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:12 smithi189 ceph-mon[36790]: pgmap v7811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13369.770430 secs 2024-07-26T15:10:13.563 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:10:13.562+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:10:13.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:10:13.562+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:13.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:10:13.562+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:10:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:14 smithi193 ceph-mon[37473]: pgmap v7812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13355.346754 secs 2024-07-26T15:10:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:14 smithi134 ceph-mon[31966]: pgmap v7812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13355.346754 secs 2024-07-26T15:10:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:14 smithi189 ceph-mon[36790]: pgmap v7812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13355.346754 secs 2024-07-26T15:10:16.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:16 smithi189 ceph-mon[36790]: pgmap v7813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:16 smithi193 ceph-mon[37473]: pgmap v7813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:16 smithi134 ceph-mon[31966]: pgmap v7813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13374.770549 secs 2024-07-26T15:10:17.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13374.770549 secs 2024-07-26T15:10:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13374.770549 secs 2024-07-26T15:10:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:18 smithi193 ceph-mon[37473]: pgmap v7814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:18 smithi134 ceph-mon[31966]: pgmap v7814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:18 smithi189 ceph-mon[36790]: pgmap v7814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13360.346909 secs 2024-07-26T15:10:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13360.346909 secs 2024-07-26T15:10:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13360.346909 secs 2024-07-26T15:10:20.189 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:10:20.189+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:10:20.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:10:20.189+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:10:20.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:10:20.189+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:10:20.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:10:20.189+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:10:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:20 smithi193 ceph-mon[37473]: pgmap v7815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:20 smithi134 ceph-mon[31966]: pgmap v7815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:20 smithi189 ceph-mon[36790]: pgmap v7815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:22 smithi193 ceph-mon[37473]: pgmap v7816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13379.770653 secs 2024-07-26T15:10:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:22 smithi134 ceph-mon[31966]: pgmap v7816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13379.770653 secs 2024-07-26T15:10:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:22 smithi189 ceph-mon[36790]: pgmap v7816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13379.770653 secs 2024-07-26T15:10:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:10:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:10:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:23.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:10:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:24.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:10:24.128+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:10:24.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:10:24.128+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:24.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:10:24.128+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:10:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:24 smithi193 ceph-mon[37473]: pgmap v7817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13365.346964 secs 2024-07-26T15:10:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:24 smithi134 ceph-mon[31966]: pgmap v7817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13365.346964 secs 2024-07-26T15:10:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:24 smithi189 ceph-mon[36790]: pgmap v7817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13365.346964 secs 2024-07-26T15:10:25.958 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:25 smithi189 ceph-mon[36790]: pgmap v7818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:25.959 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:25.959 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:25.959 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:25.959 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:25.959 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:25 smithi193 ceph-mon[37473]: pgmap v7818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:25 smithi134 ceph-mon[31966]: pgmap v7818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.424 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13384.770768 secs 2024-07-26T15:10:27.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13384.770768 secs 2024-07-26T15:10:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13384.770768 secs 2024-07-26T15:10:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:27 smithi193 ceph-mon[37473]: pgmap v7819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:28.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:27 smithi134 ceph-mon[31966]: pgmap v7819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:27 smithi189 ceph-mon[36790]: pgmap v7819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:28.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:10:28.577+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:10:28.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:10:28.577+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:10:28.579 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:10:28.577+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13370.347094 secs 2024-07-26T15:10:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:10:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:10:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:29 smithi134 ceph-mon[31966]: pgmap v7820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T15:10:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:29 smithi134 ceph-mon[31966]: pgmap v7821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:10:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13370.347094 secs 2024-07-26T15:10:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:10:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:10:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:29 smithi189 ceph-mon[36790]: pgmap v7820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T15:10:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:29 smithi189 ceph-mon[36790]: pgmap v7821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:10:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13370.347094 secs 2024-07-26T15:10:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:10:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:10:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:29 smithi193 ceph-mon[37473]: pgmap v7820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T15:10:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:29 smithi193 ceph-mon[37473]: pgmap v7821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:10:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:10:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:31 smithi134 ceph-mon[31966]: pgmap v7822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:10:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:31 smithi189 ceph-mon[36790]: pgmap v7822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:10:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:31 smithi193 ceph-mon[37473]: pgmap v7822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:10:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13389.770885 secs 2024-07-26T15:10:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13389.770885 secs 2024-07-26T15:10:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13389.770885 secs 2024-07-26T15:10:33.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:33 smithi134 ceph-mon[31966]: pgmap v7823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:10:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:33 smithi189 ceph-mon[36790]: pgmap v7823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:10:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:33 smithi193 ceph-mon[37473]: pgmap v7823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:10:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13375.347194 secs 2024-07-26T15:10:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13375.347194 secs 2024-07-26T15:10:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13375.347194 secs 2024-07-26T15:10:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:35 smithi189 ceph-mon[36790]: pgmap v7824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:10:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:35 smithi193 ceph-mon[37473]: pgmap v7824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:10:36.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:35 smithi134 ceph-mon[31966]: pgmap v7824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:10:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13394.771007 secs 2024-07-26T15:10:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:37 smithi189 ceph-mon[36790]: pgmap v7825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:10:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13394.771007 secs 2024-07-26T15:10:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:37 smithi193 ceph-mon[37473]: pgmap v7825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:10:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13394.771007 secs 2024-07-26T15:10:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:37 smithi134 ceph-mon[31966]: pgmap v7825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:10:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:38 smithi193 ceph-mon[37473]: pgmap v7826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:10:39.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:10:39.142+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:10:39.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:10:39.142+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:39.143 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:10:39.142+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:10:39.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:38 smithi134 ceph-mon[31966]: pgmap v7826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:10:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:38 smithi189 ceph-mon[36790]: pgmap v7826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:10:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13380.347314 secs 2024-07-26T15:10:40.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13380.347314 secs 2024-07-26T15:10:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13380.347314 secs 2024-07-26T15:10:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:40 smithi193 ceph-mon[37473]: pgmap v7827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:41.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:40 smithi134 ceph-mon[31966]: pgmap v7827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:40 smithi189 ceph-mon[36790]: pgmap v7827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13399.771128 secs 2024-07-26T15:10:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13399.771128 secs 2024-07-26T15:10:42.203 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13399.771128 secs 2024-07-26T15:10:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:42 smithi189 ceph-mon[36790]: pgmap v7828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:42 smithi193 ceph-mon[37473]: pgmap v7828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:43.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:42 smithi134 ceph-mon[31966]: pgmap v7828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:43.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:10:43.593+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:10:43.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:10:43.593+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:10:43.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:10:43.593+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:44.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13385.347462 secs 2024-07-26T15:10:44.692 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13385.347462 secs 2024-07-26T15:10:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13385.347462 secs 2024-07-26T15:10:45.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:45 smithi134 ceph-mon[31966]: pgmap v7829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:45 smithi189 ceph-mon[36790]: pgmap v7829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:45 smithi193 ceph-mon[37473]: pgmap v7829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13404.771242 secs 2024-07-26T15:10:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:47 smithi134 ceph-mon[31966]: pgmap v7830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13404.771242 secs 2024-07-26T15:10:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:47 smithi189 ceph-mon[36790]: pgmap v7830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13404.771242 secs 2024-07-26T15:10:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:47 smithi193 ceph-mon[37473]: pgmap v7830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13390.347599 secs 2024-07-26T15:10:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:49 smithi134 ceph-mon[31966]: pgmap v7831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13390.347599 secs 2024-07-26T15:10:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:49 smithi189 ceph-mon[36790]: pgmap v7831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13390.347599 secs 2024-07-26T15:10:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:49 smithi193 ceph-mon[37473]: pgmap v7831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:51 smithi189 ceph-mon[36790]: pgmap v7832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:51 smithi193 ceph-mon[37473]: pgmap v7832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:51 smithi134 ceph-mon[31966]: pgmap v7832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13409.771390 secs 2024-07-26T15:10:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13409.771390 secs 2024-07-26T15:10:53.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13409.771390 secs 2024-07-26T15:10:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:53 smithi189 ceph-mon[36790]: pgmap v7833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:53 smithi193 ceph-mon[37473]: pgmap v7833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:10:54.143+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:10:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:10:54.143+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:10:54.143+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:10:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:53 smithi134 ceph-mon[31966]: pgmap v7833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13395.347737 secs 2024-07-26T15:10:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13395.347737 secs 2024-07-26T15:10:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13395.347737 secs 2024-07-26T15:10:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:55 smithi193 ceph-mon[37473]: pgmap v7834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:56.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:55 smithi134 ceph-mon[31966]: pgmap v7834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:55 smithi189 ceph-mon[36790]: pgmap v7834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:10:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:56 smithi193 ceph-mon[37473]: pgmap v7835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:56 smithi134 ceph-mon[31966]: pgmap v7835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:56 smithi189 ceph-mon[36790]: pgmap v7835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:10:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13414.771462 secs 2024-07-26T15:10:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13414.771462 secs 2024-07-26T15:10:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13414.771462 secs 2024-07-26T15:10:58.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:10:58.609+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:10:58.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:10:58.609+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:10:58.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:10:58.609+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:10:58.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:58 smithi134 ceph-mon[31966]: pgmap v7836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:58 smithi189 ceph-mon[36790]: pgmap v7836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:10:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:58 smithi193 ceph-mon[37473]: pgmap v7836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:10:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13400.347863 secs 2024-07-26T15:11:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:10:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13400.347863 secs 2024-07-26T15:11:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:10:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13400.347863 secs 2024-07-26T15:11:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:00 smithi193 ceph-mon[37473]: pgmap v7837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:00 smithi134 ceph-mon[31966]: pgmap v7837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:00 smithi189 ceph-mon[36790]: pgmap v7837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13419.771565 secs 2024-07-26T15:11:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13419.771565 secs 2024-07-26T15:11:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13419.771565 secs 2024-07-26T15:11:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:02 smithi189 ceph-mon[36790]: pgmap v7838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:02 smithi193 ceph-mon[37473]: pgmap v7838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:03.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:02 smithi134 ceph-mon[31966]: pgmap v7838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13405.347995 secs 2024-07-26T15:11:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13405.347995 secs 2024-07-26T15:11:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13405.347995 secs 2024-07-26T15:11:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:05 smithi134 ceph-mon[31966]: pgmap v7839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:11:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:05 smithi189 ceph-mon[36790]: pgmap v7839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:11:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:05 smithi193 ceph-mon[37473]: pgmap v7839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:11:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13424.771683 secs 2024-07-26T15:11:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:07 smithi189 ceph-mon[36790]: pgmap v7840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13424.771683 secs 2024-07-26T15:11:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:07 smithi193 ceph-mon[37473]: pgmap v7840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:08.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13424.771683 secs 2024-07-26T15:11:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:07 smithi134 ceph-mon[31966]: pgmap v7840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:09.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:11:09.159+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:11:09.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:11:09.159+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:11:09.160 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:11:09.159+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13410.348119 secs 2024-07-26T15:11:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:09 smithi189 ceph-mon[36790]: pgmap v7841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13410.348119 secs 2024-07-26T15:11:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:09 smithi193 ceph-mon[37473]: pgmap v7841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:10.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13410.348119 secs 2024-07-26T15:11:10.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:09 smithi134 ceph-mon[31966]: pgmap v7841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:11 smithi189 ceph-mon[36790]: pgmap v7842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:11 smithi193 ceph-mon[37473]: pgmap v7842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:11 smithi134 ceph-mon[31966]: pgmap v7842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13429.771769 secs 2024-07-26T15:11:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13429.771769 secs 2024-07-26T15:11:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13429.771769 secs 2024-07-26T15:11:13.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:11:13.617+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:11:13.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:11:13.617+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:11:13.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:11:13.617+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:13.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:13 smithi134 ceph-mon[31966]: pgmap v7843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:13 smithi193 ceph-mon[37473]: pgmap v7843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:13 smithi189 ceph-mon[36790]: pgmap v7843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13415.348250 secs 2024-07-26T15:11:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:14 smithi193 ceph-mon[37473]: pgmap v7844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:11:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13415.348250 secs 2024-07-26T15:11:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:14 smithi134 ceph-mon[31966]: pgmap v7844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:11:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13415.348250 secs 2024-07-26T15:11:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:14 smithi189 ceph-mon[36790]: pgmap v7844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:11:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13434.771900 secs 2024-07-26T15:11:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:17 smithi189 ceph-mon[36790]: pgmap v7845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13434.771900 secs 2024-07-26T15:11:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:17 smithi193 ceph-mon[37473]: pgmap v7845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13434.771900 secs 2024-07-26T15:11:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:17 smithi134 ceph-mon[31966]: pgmap v7845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13420.348383 secs 2024-07-26T15:11:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:19 smithi189 ceph-mon[36790]: pgmap v7846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13420.348383 secs 2024-07-26T15:11:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:19 smithi193 ceph-mon[37473]: pgmap v7846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:20.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13420.348383 secs 2024-07-26T15:11:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:19 smithi134 ceph-mon[31966]: pgmap v7846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:21.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:21 smithi193 ceph-mon[37473]: pgmap v7847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:21 smithi134 ceph-mon[31966]: pgmap v7847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:21 smithi189 ceph-mon[36790]: pgmap v7847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13439.772018 secs 2024-07-26T15:11:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13439.772018 secs 2024-07-26T15:11:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13439.772018 secs 2024-07-26T15:11:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:23 smithi193 ceph-mon[37473]: pgmap v7848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:23.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:23 smithi134 ceph-mon[31966]: pgmap v7848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:23 smithi189 ceph-mon[36790]: pgmap v7848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:11:24.174+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:11:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:11:24.174+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:11:24.175 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:11:24.174+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13425.348517 secs 2024-07-26T15:11:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13425.348517 secs 2024-07-26T15:11:24.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13425.348517 secs 2024-07-26T15:11:25.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:25 smithi134 ceph-mon[31966]: pgmap v7849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:11:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:25 smithi189 ceph-mon[36790]: pgmap v7849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:11:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:25 smithi193 ceph-mon[37473]: pgmap v7849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:11:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13444.772155 secs 2024-07-26T15:11:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:27 smithi189 ceph-mon[36790]: pgmap v7850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13444.772155 secs 2024-07-26T15:11:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:27 smithi193 ceph-mon[37473]: pgmap v7850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:28.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13444.772155 secs 2024-07-26T15:11:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:27 smithi134 ceph-mon[31966]: pgmap v7850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:28.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:11:28.633+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:11:28.635 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:11:28.633+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:28.635 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:11:28.633+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:11:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13430.348592 secs 2024-07-26T15:11:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:29 smithi189 ceph-mon[36790]: pgmap v7851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:11:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13430.348592 secs 2024-07-26T15:11:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:29 smithi193 ceph-mon[37473]: pgmap v7851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:11:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13430.348592 secs 2024-07-26T15:11:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:29 smithi134 ceph-mon[31966]: pgmap v7851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:11:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:11:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:11:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:11:31.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:11:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:11:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:11:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:11:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:11:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:11:31.976 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:31 smithi134 ceph-mon[31966]: pgmap v7852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:11:31.977 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:31 smithi134 ceph-mon[31966]: pgmap v7853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:11:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:31 smithi193 ceph-mon[37473]: pgmap v7852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:11:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:31 smithi193 ceph-mon[37473]: pgmap v7853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:11:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:31 smithi189 ceph-mon[36790]: pgmap v7852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:11:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:31 smithi189 ceph-mon[36790]: pgmap v7853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:11:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13449.772294 secs 2024-07-26T15:11:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:32 smithi193 ceph-mon[37473]: pgmap v7854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:11:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13449.772294 secs 2024-07-26T15:11:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:32 smithi134 ceph-mon[31966]: pgmap v7854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:11:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13449.772294 secs 2024-07-26T15:11:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:32 smithi189 ceph-mon[36790]: pgmap v7854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:11:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13435.348726 secs 2024-07-26T15:11:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:34 smithi134 ceph-mon[31966]: pgmap v7855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:11:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13435.348726 secs 2024-07-26T15:11:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:34 smithi189 ceph-mon[36790]: pgmap v7855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:11:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13435.348726 secs 2024-07-26T15:11:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:34 smithi193 ceph-mon[37473]: pgmap v7855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:11:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:36 smithi134 ceph-mon[31966]: pgmap v7856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:11:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13454.772429 secs 2024-07-26T15:11:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:36 smithi189 ceph-mon[36790]: pgmap v7856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:11:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13454.772429 secs 2024-07-26T15:11:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:36 smithi193 ceph-mon[37473]: pgmap v7856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:11:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13454.772429 secs 2024-07-26T15:11:39.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:38 smithi134 ceph-mon[31966]: pgmap v7857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:11:39.190 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:11:39.190+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:11:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:11:39.190+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:39.191 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:11:39.190+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:11:39.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:38 smithi189 ceph-mon[36790]: pgmap v7857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:11:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:38 smithi193 ceph-mon[37473]: pgmap v7857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:11:40.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13440.348843 secs 2024-07-26T15:11:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13440.348843 secs 2024-07-26T15:11:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13440.348843 secs 2024-07-26T15:11:41.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:40 smithi134 ceph-mon[31966]: pgmap v7858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:11:41.174 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:40 smithi193 ceph-mon[37473]: pgmap v7858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:11:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:40 smithi189 ceph-mon[36790]: pgmap v7858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:11:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13459.772514 secs 2024-07-26T15:11:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13459.772514 secs 2024-07-26T15:11:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13459.772514 secs 2024-07-26T15:11:43.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:42 smithi134 ceph-mon[31966]: pgmap v7859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:42 smithi189 ceph-mon[36790]: pgmap v7859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:42 smithi193 ceph-mon[37473]: pgmap v7859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:43.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:11:43.641+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:11:43.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:11:43.641+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:11:43.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:11:43.641+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13445.348966 secs 2024-07-26T15:11:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:44 smithi189 ceph-mon[36790]: pgmap v7860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13445.348966 secs 2024-07-26T15:11:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:44 smithi193 ceph-mon[37473]: pgmap v7860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13445.348966 secs 2024-07-26T15:11:45.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:44 smithi134 ceph-mon[31966]: pgmap v7860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:46 smithi189 ceph-mon[36790]: pgmap v7861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13464.772627 secs 2024-07-26T15:11:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:46 smithi193 ceph-mon[37473]: pgmap v7861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13464.772627 secs 2024-07-26T15:11:47.336 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:46 smithi134 ceph-mon[31966]: pgmap v7861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:47.337 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13464.772627 secs 2024-07-26T15:11:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:48 smithi189 ceph-mon[36790]: pgmap v7862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:48 smithi193 ceph-mon[37473]: pgmap v7862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:48 smithi134 ceph-mon[31966]: pgmap v7862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13450.349075 secs 2024-07-26T15:11:50.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13450.349075 secs 2024-07-26T15:11:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13450.349075 secs 2024-07-26T15:11:51.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:50 smithi193 ceph-mon[37473]: pgmap v7863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:11:51.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:50 smithi134 ceph-mon[31966]: pgmap v7863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:11:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:50 smithi189 ceph-mon[36790]: pgmap v7863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:11:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13469.772694 secs 2024-07-26T15:11:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13469.772694 secs 2024-07-26T15:11:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13469.772694 secs 2024-07-26T15:11:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:52 smithi193 ceph-mon[37473]: pgmap v7864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:53.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:52 smithi134 ceph-mon[31966]: pgmap v7864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:52 smithi189 ceph-mon[36790]: pgmap v7864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:11:54.204+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:11:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:11:54.205+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:11:54.206 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:11:54.205+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13455.349157 secs 2024-07-26T15:11:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:54 smithi193 ceph-mon[37473]: pgmap v7865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:55.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13455.349157 secs 2024-07-26T15:11:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:54 smithi134 ceph-mon[31966]: pgmap v7865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13455.349157 secs 2024-07-26T15:11:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:54 smithi189 ceph-mon[36790]: pgmap v7865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:56 smithi193 ceph-mon[37473]: pgmap v7866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13474.772774 secs 2024-07-26T15:11:57.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:56 smithi134 ceph-mon[31966]: pgmap v7866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13474.772774 secs 2024-07-26T15:11:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:56 smithi189 ceph-mon[36790]: pgmap v7866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:11:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13474.772774 secs 2024-07-26T15:11:58.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:11:58.656+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:11:58.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:11:58.656+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:11:58.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:11:58.656+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:11:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:58 smithi193 ceph-mon[37473]: pgmap v7867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:59.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:58 smithi134 ceph-mon[31966]: pgmap v7867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:11:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:58 smithi189 ceph-mon[36790]: pgmap v7867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:12:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:11:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13460.349278 secs 2024-07-26T15:12:00.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:11:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13460.349278 secs 2024-07-26T15:12:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:11:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13460.349278 secs 2024-07-26T15:12:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:00 smithi193 ceph-mon[37473]: pgmap v7868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:12:01.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:00 smithi134 ceph-mon[31966]: pgmap v7868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:12:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:00 smithi189 ceph-mon[36790]: pgmap v7868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:12:02.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13479.772910 secs 2024-07-26T15:12:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13479.772910 secs 2024-07-26T15:12:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13479.772910 secs 2024-07-26T15:12:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:03 smithi193 ceph-mon[37473]: pgmap v7869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:03.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:03 smithi134 ceph-mon[31966]: pgmap v7869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:03 smithi189 ceph-mon[36790]: pgmap v7869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13465.349390 secs 2024-07-26T15:12:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:05 smithi193 ceph-mon[37473]: pgmap v7870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:12:05.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13465.349390 secs 2024-07-26T15:12:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:05 smithi134 ceph-mon[31966]: pgmap v7870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:12:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13465.349390 secs 2024-07-26T15:12:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:05 smithi189 ceph-mon[36790]: pgmap v7870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:12:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:07 smithi193 ceph-mon[37473]: pgmap v7871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:12:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13484.773028 secs 2024-07-26T15:12:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:07 smithi134 ceph-mon[31966]: pgmap v7871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:12:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13484.773028 secs 2024-07-26T15:12:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:07 smithi189 ceph-mon[36790]: pgmap v7871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:12:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13484.773028 secs 2024-07-26T15:12:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:09.220+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:12:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:09.220+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:12:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:09.220+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:09.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:09 smithi134 ceph-mon[31966]: pgmap v7872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:12:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:09 smithi189 ceph-mon[36790]: pgmap v7872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:12:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:09 smithi193 ceph-mon[37473]: pgmap v7872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:12:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13470.349511 secs 2024-07-26T15:12:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13470.349511 secs 2024-07-26T15:12:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13470.349511 secs 2024-07-26T15:12:11.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:11 smithi134 ceph-mon[31966]: pgmap v7873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:12:11.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:11 smithi193 ceph-mon[37473]: pgmap v7873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:12:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:11 smithi189 ceph-mon[36790]: pgmap v7873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:12:12.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13489.773178 secs 2024-07-26T15:12:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13489.773178 secs 2024-07-26T15:12:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13489.773178 secs 2024-07-26T15:12:13.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:13 smithi134 ceph-mon[31966]: pgmap v7874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:12:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:13 smithi189 ceph-mon[36790]: pgmap v7874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:12:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:13 smithi193 ceph-mon[37473]: pgmap v7874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:12:13.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:13.671+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:12:13.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:13.671+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:12:13.673 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:13.671+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13475.349625 secs 2024-07-26T15:12:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:15 smithi134 ceph-mon[31966]: pgmap v7875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:12:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13475.349625 secs 2024-07-26T15:12:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:15 smithi189 ceph-mon[36790]: pgmap v7875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:12:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13475.349625 secs 2024-07-26T15:12:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:15 smithi193 ceph-mon[37473]: pgmap v7875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:12:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:17 smithi193 ceph-mon[37473]: pgmap v7876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13494.773294 secs 2024-07-26T15:12:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:17 smithi134 ceph-mon[31966]: pgmap v7876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13494.773294 secs 2024-07-26T15:12:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:17 smithi189 ceph-mon[36790]: pgmap v7876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13494.773294 secs 2024-07-26T15:12:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:19 smithi134 ceph-mon[31966]: pgmap v7877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:12:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13480.349752 secs 2024-07-26T15:12:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:19 smithi189 ceph-mon[36790]: pgmap v7877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:12:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13480.349752 secs 2024-07-26T15:12:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:19 smithi193 ceph-mon[37473]: pgmap v7877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:12:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13480.349752 secs 2024-07-26T15:12:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:21 smithi134 ceph-mon[31966]: pgmap v7878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:12:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:21 smithi189 ceph-mon[36790]: pgmap v7878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:12:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:21 smithi193 ceph-mon[37473]: pgmap v7878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:12:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:22.332+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 msgr2=0x7f9b300c4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T15:12:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:22.332+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 msgr2=0x7f9b300c4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T15:12:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:22.332+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 0x7f9b300c4130 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009990 tx=0x7f9b38008070 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:12:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:22.332+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 0x7f9b300c4130 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009990 tx=0x7f9b38008070 comp rx=0 tx=0).stop 2024-07-26T15:12:22.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:22.332+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 msgr2=0x7f9b300c4130 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T15:12:22.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:22.332+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 msgr2=0x7f9b300c4130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:12:22.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:22.332+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 0x7f9b300c4130 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:12:22.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:22.332+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:12:22.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:22.333+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:12:22.336 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:22.335+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b38008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:12:22.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13499.773397 secs 2024-07-26T15:12:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13499.773397 secs 2024-07-26T15:12:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13499.773397 secs 2024-07-26T15:12:23.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:23 smithi134 ceph-mon[31966]: pgmap v7879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:23 smithi189 ceph-mon[36790]: pgmap v7879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:23 smithi193 ceph-mon[37473]: pgmap v7879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:24.236 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:24.236+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:12:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:24.236+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:12:24.237 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:24.236+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:24.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:24.677+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:12:24.678 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:24.677+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:12:24.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:24.677+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T15:12:24.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:24.678+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 106 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T15:12:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13485.349870 secs 2024-07-26T15:12:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13485.349870 secs 2024-07-26T15:12:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13485.349870 secs 2024-07-26T15:12:25.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:25 smithi134 ceph-mon[31966]: pgmap v7880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:12:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:25 smithi189 ceph-mon[36790]: pgmap v7880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:12:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:25 smithi193 ceph-mon[37473]: pgmap v7880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:12:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:27 smithi134 ceph-mon[31966]: pgmap v7881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:12:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13504.773492 secs 2024-07-26T15:12:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:27 smithi189 ceph-mon[36790]: pgmap v7881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:12:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13504.773492 secs 2024-07-26T15:12:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:27 smithi193 ceph-mon[37473]: pgmap v7881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:12:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13504.773492 secs 2024-07-26T15:12:28.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:28.678+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:12:28.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:28.678+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:12:28.679 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:28.678+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:29 smithi134 ceph-mon[31966]: pgmap v7882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:12:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13490.350026 secs 2024-07-26T15:12:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:29 smithi189 ceph-mon[36790]: pgmap v7882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:12:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13490.350026 secs 2024-07-26T15:12:30.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:29 smithi193 ceph-mon[37473]: pgmap v7882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:12:30.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13490.350026 secs 2024-07-26T15:12:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:12:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:12:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:12:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:31 smithi134 ceph-mon[31966]: pgmap v7883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:12:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:12:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:12:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:12:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:31 smithi189 ceph-mon[36790]: pgmap v7883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:12:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:12:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:12:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:12:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:31 smithi193 ceph-mon[37473]: pgmap v7883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:12:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:12:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:12:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:12:32.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:32.886+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 msgr2=0x7f21ac0c4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T15:12:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:32.886+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 msgr2=0x7f21ac0c4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T15:12:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:32.886+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f21b8031850 tx=0x7f21b806c000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:12:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:32.886+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f21b8031850 tx=0x7f21b806c000 comp rx=0 tx=0).stop 2024-07-26T15:12:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:32.886+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 msgr2=0x7f21ac0c4210 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T15:12:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:32.886+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 msgr2=0x7f21ac0c4210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:12:32.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:32.886+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:12:32.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:32.886+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:12:32.892 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:32.886+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:12:32.892 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:32.888+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f2198005820 tx=0x7f21b80aa000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:12:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:32 smithi134 ceph-mon[31966]: pgmap v7884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:12:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:32 smithi134 ceph-mon[31966]: pgmap v7885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:12:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13509.773574 secs 2024-07-26T15:12:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:32 smithi189 ceph-mon[36790]: pgmap v7884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:12:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:32 smithi189 ceph-mon[36790]: pgmap v7885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:12:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13509.773574 secs 2024-07-26T15:12:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:32 smithi193 ceph-mon[37473]: pgmap v7884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T15:12:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:32 smithi193 ceph-mon[37473]: pgmap v7885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:12:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13509.773574 secs 2024-07-26T15:12:34.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:34 smithi134 ceph-mon[31966]: pgmap v7886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:12:34.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13495.350201 secs 2024-07-26T15:12:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:34 smithi189 ceph-mon[36790]: pgmap v7886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:12:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13495.350201 secs 2024-07-26T15:12:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:34 smithi193 ceph-mon[37473]: pgmap v7886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:12:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13495.350201 secs 2024-07-26T15:12:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:35.213+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:12:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:35.213+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:12:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:35.214+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:12:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:35.214+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:12:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:36 smithi134 ceph-mon[31966]: pgmap v7887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:12:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:36 smithi189 ceph-mon[36790]: pgmap v7887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:12:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:36 smithi193 ceph-mon[37473]: pgmap v7887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:12:37.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13514.773685 secs 2024-07-26T15:12:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13514.773685 secs 2024-07-26T15:12:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13514.773685 secs 2024-07-26T15:12:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:38 smithi134 ceph-mon[31966]: pgmap v7888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:12:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:38 smithi189 ceph-mon[36790]: pgmap v7888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:12:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:38 smithi193 ceph-mon[37473]: pgmap v7888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:12:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:39.248+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:12:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:39.248+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:12:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:39.248+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13500.350333 secs 2024-07-26T15:12:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13500.350333 secs 2024-07-26T15:12:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13500.350333 secs 2024-07-26T15:12:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:40 smithi134 ceph-mon[31966]: pgmap v7889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:12:40.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:40 smithi189 ceph-mon[36790]: pgmap v7889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:12:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:40 smithi193 ceph-mon[37473]: pgmap v7889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:12:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:42 smithi134 ceph-mon[31966]: pgmap v7890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:12:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13519.773812 secs 2024-07-26T15:12:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:42 smithi189 ceph-mon[36790]: pgmap v7890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:12:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13519.773812 secs 2024-07-26T15:12:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:42 smithi193 ceph-mon[37473]: pgmap v7890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:12:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13519.773812 secs 2024-07-26T15:12:43.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:43.693+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:12:43.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:43.693+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:43.695 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:43.693+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:12:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:44 smithi134 ceph-mon[31966]: pgmap v7891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:12:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13505.350453 secs 2024-07-26T15:12:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:44 smithi189 ceph-mon[36790]: pgmap v7891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:12:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13505.350453 secs 2024-07-26T15:12:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:44 smithi193 ceph-mon[37473]: pgmap v7891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:12:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13505.350453 secs 2024-07-26T15:12:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:46 smithi134 ceph-mon[31966]: pgmap v7892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:46.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:46 smithi189 ceph-mon[36790]: pgmap v7892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:46 smithi193 ceph-mon[37473]: pgmap v7892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13524.773928 secs 2024-07-26T15:12:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13524.773928 secs 2024-07-26T15:12:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13524.773928 secs 2024-07-26T15:12:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:48 smithi189 ceph-mon[36790]: pgmap v7893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:12:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:48 smithi193 ceph-mon[37473]: pgmap v7893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:12:49.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:48 smithi134 ceph-mon[31966]: pgmap v7893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:12:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13510.350538 secs 2024-07-26T15:12:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13510.350538 secs 2024-07-26T15:12:50.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13510.350538 secs 2024-07-26T15:12:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:50 smithi189 ceph-mon[36790]: pgmap v7894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:12:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:50 smithi193 ceph-mon[37473]: pgmap v7894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:12:51.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:50 smithi134 ceph-mon[31966]: pgmap v7894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:12:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:52 smithi193 ceph-mon[37473]: pgmap v7895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13529.774018 secs 2024-07-26T15:12:53.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:52 smithi134 ceph-mon[31966]: pgmap v7895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13529.774018 secs 2024-07-26T15:12:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:52 smithi189 ceph-mon[36790]: pgmap v7895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13529.774018 secs 2024-07-26T15:12:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:54.264+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:12:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:54.264+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:12:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:12:54.264+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:54.974 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:54 smithi134 ceph-mon[31966]: pgmap v7896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:12:54.974 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13515.350660 secs 2024-07-26T15:12:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:54 smithi193 ceph-mon[37473]: pgmap v7896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:12:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13515.350660 secs 2024-07-26T15:12:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:54 smithi189 ceph-mon[36790]: pgmap v7896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:12:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13515.350660 secs 2024-07-26T15:12:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:56 smithi193 ceph-mon[37473]: pgmap v7897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:56 smithi134 ceph-mon[31966]: pgmap v7897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:56 smithi189 ceph-mon[36790]: pgmap v7897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:12:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13534.774140 secs 2024-07-26T15:12:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13534.774140 secs 2024-07-26T15:12:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13534.774140 secs 2024-07-26T15:12:58.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:58.694+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:12:58.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:58.694+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:12:58.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:12:58.694+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:12:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:58 smithi193 ceph-mon[37473]: pgmap v7898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:12:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:58 smithi134 ceph-mon[31966]: pgmap v7898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:12:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:58 smithi189 ceph-mon[36790]: pgmap v7898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:13:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:12:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13520.350805 secs 2024-07-26T15:13:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:12:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13520.350805 secs 2024-07-26T15:13:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:12:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13520.350805 secs 2024-07-26T15:13:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:00 smithi193 ceph-mon[37473]: pgmap v7899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:00 smithi134 ceph-mon[31966]: pgmap v7899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:00 smithi189 ceph-mon[36790]: pgmap v7899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:01 smithi193 ceph-mon[37473]: pgmap v7900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:02.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:01 smithi134 ceph-mon[31966]: pgmap v7900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:01 smithi189 ceph-mon[36790]: pgmap v7900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13539.774266 secs 2024-07-26T15:13:03.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13539.774266 secs 2024-07-26T15:13:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13539.774266 secs 2024-07-26T15:13:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:03 smithi193 ceph-mon[37473]: pgmap v7901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:04.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:03 smithi134 ceph-mon[31966]: pgmap v7901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:03 smithi189 ceph-mon[36790]: pgmap v7901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13525.350978 secs 2024-07-26T15:13:05.160 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13525.350978 secs 2024-07-26T15:13:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13525.350978 secs 2024-07-26T15:13:06.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:05 smithi134 ceph-mon[31966]: pgmap v7902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:05 smithi189 ceph-mon[36790]: pgmap v7902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:06.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:05 smithi193 ceph-mon[37473]: pgmap v7902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13544.774384 secs 2024-07-26T15:13:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13544.774384 secs 2024-07-26T15:13:07.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13544.774384 secs 2024-07-26T15:13:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:07 smithi193 ceph-mon[37473]: pgmap v7903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:13:08.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:07 smithi134 ceph-mon[31966]: pgmap v7903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:13:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:07 smithi189 ceph-mon[36790]: pgmap v7903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:13:09.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:13:09.279+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:13:09.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:13:09.279+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:13:09.281 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:13:09.279+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:10.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13530.351100 secs 2024-07-26T15:13:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:10 smithi134 ceph-mon[31966]: pgmap v7904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13530.351100 secs 2024-07-26T15:13:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:10 smithi189 ceph-mon[36790]: pgmap v7904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13530.351100 secs 2024-07-26T15:13:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:10 smithi193 ceph-mon[37473]: pgmap v7904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:12.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:12 smithi134 ceph-mon[31966]: pgmap v7905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13549.774522 secs 2024-07-26T15:13:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:12 smithi189 ceph-mon[36790]: pgmap v7905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13549.774522 secs 2024-07-26T15:13:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:12 smithi193 ceph-mon[37473]: pgmap v7905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13549.774522 secs 2024-07-26T15:13:13.696 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:13:13.695+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:13:13.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:13:13.695+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:13.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:13:13.695+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:13:14.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:14 smithi134 ceph-mon[31966]: pgmap v7906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:14 smithi189 ceph-mon[36790]: pgmap v7906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:14.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:14 smithi193 ceph-mon[37473]: pgmap v7906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13535.351220 secs 2024-07-26T15:13:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13535.351220 secs 2024-07-26T15:13:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13535.351220 secs 2024-07-26T15:13:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:16 smithi134 ceph-mon[31966]: pgmap v7907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:16 smithi189 ceph-mon[36790]: pgmap v7907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:16 smithi193 ceph-mon[37473]: pgmap v7907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13554.774639 secs 2024-07-26T15:13:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13554.774639 secs 2024-07-26T15:13:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13554.774639 secs 2024-07-26T15:13:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:18 smithi134 ceph-mon[31966]: pgmap v7908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:13:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:18 smithi189 ceph-mon[36790]: pgmap v7908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:13:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:18 smithi193 ceph-mon[37473]: pgmap v7908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:13:19.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13540.351289 secs 2024-07-26T15:13:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13540.351289 secs 2024-07-26T15:13:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13540.351289 secs 2024-07-26T15:13:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:20 smithi134 ceph-mon[31966]: pgmap v7909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:20 smithi189 ceph-mon[36790]: pgmap v7909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:20 smithi193 ceph-mon[37473]: pgmap v7909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:22 smithi134 ceph-mon[31966]: pgmap v7910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13559.774763 secs 2024-07-26T15:13:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:22 smithi189 ceph-mon[36790]: pgmap v7910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13559.774763 secs 2024-07-26T15:13:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:22 smithi193 ceph-mon[37473]: pgmap v7910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13559.774763 secs 2024-07-26T15:13:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:13:24.295+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:13:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:13:24.295+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:24.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:13:24.295+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:13:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:24 smithi134 ceph-mon[31966]: pgmap v7911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13545.351389 secs 2024-07-26T15:13:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:24 smithi189 ceph-mon[36790]: pgmap v7911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13545.351389 secs 2024-07-26T15:13:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:24 smithi193 ceph-mon[37473]: pgmap v7911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13545.351389 secs 2024-07-26T15:13:26.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:26 smithi134 ceph-mon[31966]: pgmap v7912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:26 smithi189 ceph-mon[36790]: pgmap v7912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:26 smithi193 ceph-mon[37473]: pgmap v7912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13564.774903 secs 2024-07-26T15:13:27.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13564.774903 secs 2024-07-26T15:13:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13564.774903 secs 2024-07-26T15:13:28.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:13:28.710+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:13:28.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:13:28.710+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:28.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:13:28.710+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:13:28.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:28 smithi134 ceph-mon[31966]: pgmap v7913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:13:28.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:28 smithi189 ceph-mon[36790]: pgmap v7913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:13:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:28 smithi193 ceph-mon[37473]: pgmap v7913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:13:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13550.351521 secs 2024-07-26T15:13:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13550.351521 secs 2024-07-26T15:13:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13550.351521 secs 2024-07-26T15:13:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:30 smithi134 ceph-mon[31966]: pgmap v7914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:30 smithi189 ceph-mon[36790]: pgmap v7914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:31.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:30 smithi193 ceph-mon[37473]: pgmap v7914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:13:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:13:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:13:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:32 smithi134 ceph-mon[31966]: pgmap v7915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13569.775030 secs 2024-07-26T15:13:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:13:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:13:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:13:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:32 smithi189 ceph-mon[36790]: pgmap v7915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13569.775030 secs 2024-07-26T15:13:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:13:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:13:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:13:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:32 smithi193 ceph-mon[37473]: pgmap v7915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13569.775030 secs 2024-07-26T15:13:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:13:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:13:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:13:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:33 smithi134 ceph-mon[31966]: pgmap v7916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:13:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:33 smithi134 ceph-mon[31966]: pgmap v7917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:13:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:33 smithi189 ceph-mon[36790]: pgmap v7916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:13:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:33 smithi189 ceph-mon[36790]: pgmap v7917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:13:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:33 smithi193 ceph-mon[37473]: pgmap v7916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:13:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:33 smithi193 ceph-mon[37473]: pgmap v7917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:13:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13555.351646 secs 2024-07-26T15:13:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13555.351646 secs 2024-07-26T15:13:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13555.351646 secs 2024-07-26T15:13:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:35 smithi134 ceph-mon[31966]: pgmap v7918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:13:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:35 smithi189 ceph-mon[36790]: pgmap v7918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:13:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:35 smithi193 ceph-mon[37473]: pgmap v7918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:13:37.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13574.775105 secs 2024-07-26T15:13:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:37 smithi134 ceph-mon[31966]: pgmap v7919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:13:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13574.775105 secs 2024-07-26T15:13:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:37 smithi189 ceph-mon[36790]: pgmap v7919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:13:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13574.775105 secs 2024-07-26T15:13:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:37 smithi193 ceph-mon[37473]: pgmap v7919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:13:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:13:39.310+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:13:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:13:39.310+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:13:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:13:39.310+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:39 smithi134 ceph-mon[31966]: pgmap v7920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:13:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13560.351778 secs 2024-07-26T15:13:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:39 smithi189 ceph-mon[36790]: pgmap v7920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:13:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13560.351778 secs 2024-07-26T15:13:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:39 smithi193 ceph-mon[37473]: pgmap v7920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:13:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13560.351778 secs 2024-07-26T15:13:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:41 smithi189 ceph-mon[36790]: pgmap v7921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:13:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:41 smithi193 ceph-mon[37473]: pgmap v7921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:13:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:41 smithi134 ceph-mon[31966]: pgmap v7921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:13:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13579.775221 secs 2024-07-26T15:13:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13579.775221 secs 2024-07-26T15:13:43.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13579.775221 secs 2024-07-26T15:13:43.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:13:43.726+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:13:43.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:13:43.726+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:43.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:13:43.726+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:13:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:43 smithi189 ceph-mon[36790]: pgmap v7922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:13:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:43 smithi193 ceph-mon[37473]: pgmap v7922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:13:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:43 smithi134 ceph-mon[31966]: pgmap v7922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:13:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13565.351913 secs 2024-07-26T15:13:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13565.351913 secs 2024-07-26T15:13:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13565.351913 secs 2024-07-26T15:13:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:45 smithi193 ceph-mon[37473]: pgmap v7923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:45 smithi134 ceph-mon[31966]: pgmap v7923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:45 smithi189 ceph-mon[36790]: pgmap v7923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13584.775344 secs 2024-07-26T15:13:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:47 smithi193 ceph-mon[37473]: pgmap v7924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13584.775344 secs 2024-07-26T15:13:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:47 smithi134 ceph-mon[31966]: pgmap v7924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13584.775344 secs 2024-07-26T15:13:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:47 smithi189 ceph-mon[36790]: pgmap v7924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:49.995 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:49 smithi189 ceph-mon[36790]: pgmap v7925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:49.995 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13570.352032 secs 2024-07-26T15:13:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:49 smithi193 ceph-mon[37473]: pgmap v7925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13570.352032 secs 2024-07-26T15:13:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:49 smithi134 ceph-mon[31966]: pgmap v7925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13570.352032 secs 2024-07-26T15:13:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:51 smithi193 ceph-mon[37473]: pgmap v7926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:51 smithi134 ceph-mon[31966]: pgmap v7926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:51 smithi189 ceph-mon[36790]: pgmap v7926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:53.028 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13589.775460 secs 2024-07-26T15:13:53.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13589.775460 secs 2024-07-26T15:13:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13589.775460 secs 2024-07-26T15:13:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:53 smithi193 ceph-mon[37473]: pgmap v7927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:54.075 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:53 smithi134 ceph-mon[31966]: pgmap v7927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:53 smithi189 ceph-mon[36790]: pgmap v7927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:13:54.327 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:13:54.326+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:13:54.327 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:13:54.326+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:13:54.327 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:13:54.326+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13575.352118 secs 2024-07-26T15:13:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:54 smithi193 ceph-mon[37473]: pgmap v7928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13575.352118 secs 2024-07-26T15:13:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:54 smithi134 ceph-mon[31966]: pgmap v7928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13575.352118 secs 2024-07-26T15:13:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:54 smithi189 ceph-mon[36790]: pgmap v7928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:13:57.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13594.775580 secs 2024-07-26T15:13:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:57 smithi134 ceph-mon[31966]: pgmap v7929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13594.775580 secs 2024-07-26T15:13:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:57 smithi189 ceph-mon[36790]: pgmap v7929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13594.775580 secs 2024-07-26T15:13:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:57 smithi193 ceph-mon[37473]: pgmap v7929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:13:58.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:13:58.728+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:13:58.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:13:58.728+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:13:58.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:13:58.728+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:13:59.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:13:59 smithi134 ceph-mon[31966]: pgmap v7930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:13:59 smithi189 ceph-mon[36790]: pgmap v7930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:13:59.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:13:59 smithi193 ceph-mon[37473]: pgmap v7930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:14:00.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13580.352199 secs 2024-07-26T15:14:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13580.352199 secs 2024-07-26T15:14:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13580.352199 secs 2024-07-26T15:14:01.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:01 smithi134 ceph-mon[31966]: pgmap v7931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:01 smithi189 ceph-mon[36790]: pgmap v7931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:01.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:01 smithi193 ceph-mon[37473]: pgmap v7931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13599.775697 secs 2024-07-26T15:14:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13599.775697 secs 2024-07-26T15:14:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13599.775697 secs 2024-07-26T15:14:03.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:03 smithi134 ceph-mon[31966]: pgmap v7932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:03 smithi189 ceph-mon[36790]: pgmap v7932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:03.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:03 smithi193 ceph-mon[37473]: pgmap v7932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13585.352263 secs 2024-07-26T15:14:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13585.352263 secs 2024-07-26T15:14:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13585.352263 secs 2024-07-26T15:14:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:05 smithi134 ceph-mon[31966]: pgmap v7933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:05 smithi189 ceph-mon[36790]: pgmap v7933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:05 smithi193 ceph-mon[37473]: pgmap v7933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13604.775816 secs 2024-07-26T15:14:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:07 smithi134 ceph-mon[31966]: pgmap v7934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13604.775816 secs 2024-07-26T15:14:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:07 smithi189 ceph-mon[36790]: pgmap v7934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13604.775816 secs 2024-07-26T15:14:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:07 smithi193 ceph-mon[37473]: pgmap v7934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:14:09.341+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:14:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:14:09.341+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:14:09.343 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:14:09.341+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:09 smithi134 ceph-mon[31966]: pgmap v7935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:14:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13590.352363 secs 2024-07-26T15:14:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:09 smithi189 ceph-mon[36790]: pgmap v7935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:14:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13590.352363 secs 2024-07-26T15:14:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:09 smithi193 ceph-mon[37473]: pgmap v7935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:14:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13590.352363 secs 2024-07-26T15:14:11.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:11 smithi134 ceph-mon[31966]: pgmap v7936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:11 smithi189 ceph-mon[36790]: pgmap v7936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:11 smithi193 ceph-mon[37473]: pgmap v7936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13609.775959 secs 2024-07-26T15:14:12.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13609.775959 secs 2024-07-26T15:14:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13609.775959 secs 2024-07-26T15:14:13.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:13 smithi134 ceph-mon[31966]: pgmap v7937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:13 smithi189 ceph-mon[36790]: pgmap v7937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:13.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:14:13.743+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:14:13.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:14:13.743+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:13.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:14:13.743+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:14:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:13 smithi193 ceph-mon[37473]: pgmap v7937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:14.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13595.352511 secs 2024-07-26T15:14:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13595.352511 secs 2024-07-26T15:14:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13595.352511 secs 2024-07-26T15:14:15.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:15 smithi134 ceph-mon[31966]: pgmap v7938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:15 smithi189 ceph-mon[36790]: pgmap v7938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:15 smithi193 ceph-mon[37473]: pgmap v7938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13614.776088 secs 2024-07-26T15:14:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:17 smithi134 ceph-mon[31966]: pgmap v7939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13614.776088 secs 2024-07-26T15:14:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:17 smithi189 ceph-mon[36790]: pgmap v7939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13614.776088 secs 2024-07-26T15:14:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:17 smithi193 ceph-mon[37473]: pgmap v7939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:19 smithi189 ceph-mon[36790]: pgmap v7940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:14:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13600.352653 secs 2024-07-26T15:14:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:19 smithi193 ceph-mon[37473]: pgmap v7940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:14:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13600.352653 secs 2024-07-26T15:14:19.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:19 smithi134 ceph-mon[31966]: pgmap v7940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:14:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13600.352653 secs 2024-07-26T15:14:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:21 smithi189 ceph-mon[36790]: pgmap v7941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:21 smithi193 ceph-mon[37473]: pgmap v7941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:21 smithi134 ceph-mon[31966]: pgmap v7941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13619.776181 secs 2024-07-26T15:14:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13619.776181 secs 2024-07-26T15:14:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13619.776181 secs 2024-07-26T15:14:23.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:23 smithi193 ceph-mon[37473]: pgmap v7942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:23.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:23 smithi134 ceph-mon[31966]: pgmap v7942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:23 smithi189 ceph-mon[36790]: pgmap v7942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:24.345 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:14:24.344+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:14:24.345 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:14:24.344+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:24.345 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:14:24.344+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:14:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13605.352794 secs 2024-07-26T15:14:24.827 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13605.352794 secs 2024-07-26T15:14:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13605.352794 secs 2024-07-26T15:14:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:25 smithi193 ceph-mon[37473]: pgmap v7943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:25.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:25 smithi134 ceph-mon[31966]: pgmap v7943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:25 smithi189 ceph-mon[36790]: pgmap v7943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:14:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13624.776331 secs 2024-07-26T15:14:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:27 smithi193 ceph-mon[37473]: pgmap v7944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13624.776331 secs 2024-07-26T15:14:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:27 smithi134 ceph-mon[31966]: pgmap v7944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13624.776331 secs 2024-07-26T15:14:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:27 smithi189 ceph-mon[36790]: pgmap v7944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:28.757 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:14:28.756+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:14:28.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:14:28.756+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:28.758 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:14:28.756+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:14:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:29 smithi193 ceph-mon[37473]: pgmap v7945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:14:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13610.352921 secs 2024-07-26T15:14:29.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:29 smithi134 ceph-mon[31966]: pgmap v7945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:14:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13610.352921 secs 2024-07-26T15:14:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:29 smithi189 ceph-mon[36790]: pgmap v7945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:14:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13610.352921 secs 2024-07-26T15:14:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:31 smithi193 ceph-mon[37473]: pgmap v7946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:31 smithi134 ceph-mon[31966]: pgmap v7946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:31 smithi189 ceph-mon[36790]: pgmap v7946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13629.776468 secs 2024-07-26T15:14:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:32 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:14:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13629.776468 secs 2024-07-26T15:14:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:32 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:14:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13629.776468 secs 2024-07-26T15:14:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:32 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:14:33.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:33 smithi193 ceph-mon[37473]: pgmap v7947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:14:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:14:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:14:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:33 smithi134 ceph-mon[31966]: pgmap v7947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:14:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:14:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:14:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:33 smithi189 ceph-mon[36790]: pgmap v7947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:14:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:14:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:14:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:34 smithi193 ceph-mon[37473]: pgmap v7948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:14:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:34 smithi193 ceph-mon[37473]: pgmap v7949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:14:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13615.353041 secs 2024-07-26T15:14:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:34 smithi134 ceph-mon[31966]: pgmap v7948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:14:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:34 smithi134 ceph-mon[31966]: pgmap v7949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:14:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13615.353041 secs 2024-07-26T15:14:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:34 smithi189 ceph-mon[36790]: pgmap v7948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:14:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:34 smithi189 ceph-mon[36790]: pgmap v7949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:14:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13615.353041 secs 2024-07-26T15:14:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:36 smithi193 ceph-mon[37473]: pgmap v7950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:14:36.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:36 smithi134 ceph-mon[31966]: pgmap v7950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:14:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:36 smithi189 ceph-mon[36790]: pgmap v7950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:14:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13634.776622 secs 2024-07-26T15:14:37.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13634.776622 secs 2024-07-26T15:14:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13634.776622 secs 2024-07-26T15:14:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:38 smithi134 ceph-mon[31966]: pgmap v7951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2024-07-26T15:14:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:38 smithi193 ceph-mon[37473]: pgmap v7951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2024-07-26T15:14:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:38 smithi189 ceph-mon[36790]: pgmap v7951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2024-07-26T15:14:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:14:39.357+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:14:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:14:39.357+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:14:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:14:39.357+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:39.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13620.353111 secs 2024-07-26T15:14:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13620.353111 secs 2024-07-26T15:14:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13620.353111 secs 2024-07-26T15:14:40.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:40 smithi134 ceph-mon[31966]: pgmap v7952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:14:40.942 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:40 smithi189 ceph-mon[36790]: pgmap v7952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:14:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:40 smithi193 ceph-mon[37473]: pgmap v7952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:14:42.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13639.776757 secs 2024-07-26T15:14:42.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:42 smithi193 ceph-mon[37473]: pgmap v7953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:14:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13639.776757 secs 2024-07-26T15:14:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:42 smithi134 ceph-mon[31966]: pgmap v7953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:14:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13639.776757 secs 2024-07-26T15:14:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:42 smithi189 ceph-mon[36790]: pgmap v7953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:14:43.761 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:14:43.760+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:14:43.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:14:43.760+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:43.762 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:14:43.760+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:14:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:44 smithi134 ceph-mon[31966]: pgmap v7954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:14:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13625.353233 secs 2024-07-26T15:14:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:44 smithi189 ceph-mon[36790]: pgmap v7954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:14:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13625.353233 secs 2024-07-26T15:14:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:44 smithi193 ceph-mon[37473]: pgmap v7954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:14:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13625.353233 secs 2024-07-26T15:14:46.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:46 smithi134 ceph-mon[31966]: pgmap v7955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:46 smithi189 ceph-mon[36790]: pgmap v7955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:46 smithi193 ceph-mon[37473]: pgmap v7955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13644.776877 secs 2024-07-26T15:14:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13644.776877 secs 2024-07-26T15:14:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13644.776877 secs 2024-07-26T15:14:48.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:48 smithi134 ceph-mon[31966]: pgmap v7956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:48 smithi189 ceph-mon[36790]: pgmap v7956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:49.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:48 smithi193 ceph-mon[37473]: pgmap v7956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:49.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13630.353297 secs 2024-07-26T15:14:49.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13630.353297 secs 2024-07-26T15:14:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13630.353297 secs 2024-07-26T15:14:50.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:50 smithi134 ceph-mon[31966]: pgmap v7957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:14:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:50 smithi189 ceph-mon[36790]: pgmap v7957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:14:51.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:50 smithi193 ceph-mon[37473]: pgmap v7957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:14:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13649.777008 secs 2024-07-26T15:14:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:52 smithi134 ceph-mon[31966]: pgmap v7958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13649.777008 secs 2024-07-26T15:14:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:52 smithi189 ceph-mon[36790]: pgmap v7958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13649.777008 secs 2024-07-26T15:14:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:52 smithi193 ceph-mon[37473]: pgmap v7958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:14:54.370+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:14:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:14:54.370+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:14:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:14:54.370+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:54 smithi134 ceph-mon[31966]: pgmap v7959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:14:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13635.353423 secs 2024-07-26T15:14:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:54 smithi189 ceph-mon[36790]: pgmap v7959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:14:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13635.353423 secs 2024-07-26T15:14:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:54 smithi193 ceph-mon[37473]: pgmap v7959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:14:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13635.353423 secs 2024-07-26T15:14:56.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:56 smithi134 ceph-mon[31966]: pgmap v7960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:56 smithi189 ceph-mon[36790]: pgmap v7960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:57.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:56 smithi193 ceph-mon[37473]: pgmap v7960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:14:57.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13654.777081 secs 2024-07-26T15:14:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13654.777081 secs 2024-07-26T15:14:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13654.777081 secs 2024-07-26T15:14:58.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:14:58.775+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:14:58.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:14:58.776+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:14:58.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:14:58.776+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:14:58.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:58 smithi134 ceph-mon[31966]: pgmap v7961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:58 smithi189 ceph-mon[36790]: pgmap v7961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:59.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:58 smithi193 ceph-mon[37473]: pgmap v7961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:14:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:14:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13640.353544 secs 2024-07-26T15:14:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:14:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13640.353544 secs 2024-07-26T15:15:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:14:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13640.353544 secs 2024-07-26T15:15:00.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:00 smithi134 ceph-mon[31966]: pgmap v7962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:15:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:00 smithi189 ceph-mon[36790]: pgmap v7962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:15:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:00 smithi193 ceph-mon[37473]: pgmap v7962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:15:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13659.777241 secs 2024-07-26T15:15:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:02 smithi134 ceph-mon[31966]: pgmap v7963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13659.777241 secs 2024-07-26T15:15:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:02 smithi189 ceph-mon[36790]: pgmap v7963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13659.777241 secs 2024-07-26T15:15:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:02 smithi193 ceph-mon[37473]: pgmap v7963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:04 smithi134 ceph-mon[31966]: pgmap v7964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13645.353682 secs 2024-07-26T15:15:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:04 smithi189 ceph-mon[36790]: pgmap v7964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13645.353682 secs 2024-07-26T15:15:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:04 smithi193 ceph-mon[37473]: pgmap v7964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13645.353682 secs 2024-07-26T15:15:06.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:06 smithi134 ceph-mon[31966]: pgmap v7965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:06 smithi189 ceph-mon[36790]: pgmap v7965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:06 smithi193 ceph-mon[37473]: pgmap v7965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13664.777366 secs 2024-07-26T15:15:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13664.777366 secs 2024-07-26T15:15:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13664.777366 secs 2024-07-26T15:15:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:08 smithi189 ceph-mon[36790]: pgmap v7966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:15:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:08 smithi193 ceph-mon[37473]: pgmap v7966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:15:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:08 smithi134 ceph-mon[31966]: pgmap v7966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:15:09.373 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:15:09.372+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:15:09.373 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:15:09.372+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:15:09.373 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:15:09.372+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13650.353824 secs 2024-07-26T15:15:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13650.353824 secs 2024-07-26T15:15:10.088 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13650.353824 secs 2024-07-26T15:15:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:10 smithi189 ceph-mon[36790]: pgmap v7967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:15:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:10 smithi193 ceph-mon[37473]: pgmap v7967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:15:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:10 smithi134 ceph-mon[31966]: pgmap v7967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:15:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13669.777494 secs 2024-07-26T15:15:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:12 smithi189 ceph-mon[36790]: pgmap v7968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13669.777494 secs 2024-07-26T15:15:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:12 smithi193 ceph-mon[37473]: pgmap v7968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13669.777494 secs 2024-07-26T15:15:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:12 smithi134 ceph-mon[31966]: pgmap v7968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:13.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:15:13.791+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:15:13.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:15:13.791+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:15:13.793 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:15:13.791+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:14 smithi193 ceph-mon[37473]: pgmap v7969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13655.353917 secs 2024-07-26T15:15:15.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:14 smithi134 ceph-mon[31966]: pgmap v7969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13655.353917 secs 2024-07-26T15:15:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:14 smithi189 ceph-mon[36790]: pgmap v7969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13655.353917 secs 2024-07-26T15:15:17.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:16 smithi193 ceph-mon[37473]: pgmap v7970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:17.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:16 smithi134 ceph-mon[31966]: pgmap v7970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:16 smithi189 ceph-mon[36790]: pgmap v7970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13674.777617 secs 2024-07-26T15:15:18.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13674.777617 secs 2024-07-26T15:15:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13674.777617 secs 2024-07-26T15:15:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:18 smithi193 ceph-mon[37473]: pgmap v7971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:15:19.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:18 smithi189 ceph-mon[36790]: pgmap v7971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:15:19.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:18 smithi134 ceph-mon[31966]: pgmap v7971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:15:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13660.354059 secs 2024-07-26T15:15:20.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13660.354059 secs 2024-07-26T15:15:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13660.354059 secs 2024-07-26T15:15:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:20 smithi193 ceph-mon[37473]: pgmap v7972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:15:21.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:20 smithi134 ceph-mon[31966]: pgmap v7972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:15:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:20 smithi189 ceph-mon[36790]: pgmap v7972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:15:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13679.777742 secs 2024-07-26T15:15:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:22 smithi193 ceph-mon[37473]: pgmap v7973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:23.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13679.777742 secs 2024-07-26T15:15:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:22 smithi134 ceph-mon[31966]: pgmap v7973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13679.777742 secs 2024-07-26T15:15:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:22 smithi189 ceph-mon[36790]: pgmap v7973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:24.389 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:15:24.388+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:15:24.389 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:15:24.388+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:24.389 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:15:24.388+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:15:24.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:15:24.708+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:15:24.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:15:24.708+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:15:24.709 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:15:24.708+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 107 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T15:15:24.710 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:15:24.708+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T15:15:24.975 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:24 smithi134 ceph-mon[31966]: pgmap v7974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:24.975 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13665.354139 secs 2024-07-26T15:15:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:24 smithi193 ceph-mon[37473]: pgmap v7974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13665.354139 secs 2024-07-26T15:15:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:24 smithi189 ceph-mon[36790]: pgmap v7974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13665.354139 secs 2024-07-26T15:15:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:26 smithi193 ceph-mon[37473]: pgmap v7975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:26 smithi134 ceph-mon[31966]: pgmap v7975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:26 smithi189 ceph-mon[36790]: pgmap v7975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13684.777865 secs 2024-07-26T15:15:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13684.777865 secs 2024-07-26T15:15:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13684.777865 secs 2024-07-26T15:15:28.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:15:28.807+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:15:28.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:15:28.807+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:15:28.808 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:15:28.807+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:28 smithi193 ceph-mon[37473]: pgmap v7976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:15:29.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:28 smithi134 ceph-mon[31966]: pgmap v7976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:15:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:28 smithi189 ceph-mon[36790]: pgmap v7976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:15:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13670.354225 secs 2024-07-26T15:15:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:29 smithi134 ceph-mon[31966]: pgmap v7977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:15:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13670.354225 secs 2024-07-26T15:15:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:29 smithi189 ceph-mon[36790]: pgmap v7977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:15:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13670.354225 secs 2024-07-26T15:15:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:29 smithi193 ceph-mon[37473]: pgmap v7977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:15:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13689.777990 secs 2024-07-26T15:15:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:32 smithi193 ceph-mon[37473]: pgmap v7978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:32.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13689.777990 secs 2024-07-26T15:15:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:32 smithi134 ceph-mon[31966]: pgmap v7978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13689.777990 secs 2024-07-26T15:15:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:32 smithi189 ceph-mon[36790]: pgmap v7978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:34 smithi134 ceph-mon[31966]: pgmap v7979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13675.354344 secs 2024-07-26T15:15:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:34 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:15:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:34 smithi189 ceph-mon[36790]: pgmap v7979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13675.354344 secs 2024-07-26T15:15:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:34 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:15:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:34 smithi193 ceph-mon[37473]: pgmap v7979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13675.354344 secs 2024-07-26T15:15:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:34 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:15:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:15:35.244+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:15:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:15:35.244+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:15:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:15:35.245+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:15:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:15:35.245+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:15:36.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:36 smithi134 ceph-mon[31966]: pgmap v7980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:36 smithi189 ceph-mon[36790]: pgmap v7980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:36 smithi193 ceph-mon[37473]: pgmap v7980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:37.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13694.778063 secs 2024-07-26T15:15:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13694.778063 secs 2024-07-26T15:15:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13694.778063 secs 2024-07-26T15:15:38.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:38 smithi134 ceph-mon[31966]: pgmap v7981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:15:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:38 smithi189 ceph-mon[36790]: pgmap v7981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:15:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:38 smithi193 ceph-mon[37473]: pgmap v7981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:15:39.403 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:15:39.399+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:15:39.403 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:15:39.402+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:39.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:15:39.402+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:15:39.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13680.354464 secs 2024-07-26T15:15:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13680.354464 secs 2024-07-26T15:15:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13680.354464 secs 2024-07-26T15:15:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:40 smithi134 ceph-mon[31966]: pgmap v7982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:15:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:15:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:15:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:15:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:15:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:40 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:15:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:40 smithi189 ceph-mon[36790]: pgmap v7982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:15:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:15:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:15:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:15:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:15:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:40 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:15:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:40 smithi193 ceph-mon[37473]: pgmap v7982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:15:41.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:15:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:15:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:15:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:15:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:40 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:15:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:41 smithi134 ceph-mon[31966]: pgmap v7983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:15:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:41 smithi134 ceph-mon[31966]: pgmap v7984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:15:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:41 smithi189 ceph-mon[36790]: pgmap v7983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:15:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:41 smithi189 ceph-mon[36790]: pgmap v7984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:15:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:41 smithi193 ceph-mon[37473]: pgmap v7983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:15:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:41 smithi193 ceph-mon[37473]: pgmap v7984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:15:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13699.778204 secs 2024-07-26T15:15:42.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13699.778204 secs 2024-07-26T15:15:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13699.778204 secs 2024-07-26T15:15:43.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:15:43.811+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:15:43.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:15:43.811+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:43.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:15:43.811+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:15:43.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:43 smithi134 ceph-mon[31966]: pgmap v7985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T15:15:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:43 smithi189 ceph-mon[36790]: pgmap v7985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T15:15:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:43 smithi193 ceph-mon[37473]: pgmap v7985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T15:15:44.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13685.354559 secs 2024-07-26T15:15:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13685.354559 secs 2024-07-26T15:15:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13685.354559 secs 2024-07-26T15:15:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:45 smithi134 ceph-mon[31966]: pgmap v7986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:15:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:45 smithi189 ceph-mon[36790]: pgmap v7986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:15:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:45 smithi193 ceph-mon[37473]: pgmap v7986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:15:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:47 smithi134 ceph-mon[31966]: pgmap v7987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:15:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13704.778353 secs 2024-07-26T15:15:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:47 smithi189 ceph-mon[36790]: pgmap v7987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:15:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13704.778353 secs 2024-07-26T15:15:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:47 smithi193 ceph-mon[37473]: pgmap v7987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:15:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13704.778353 secs 2024-07-26T15:15:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:49 smithi134 ceph-mon[31966]: pgmap v7988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:15:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13690.354674 secs 2024-07-26T15:15:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:49 smithi189 ceph-mon[36790]: pgmap v7988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:15:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13690.354674 secs 2024-07-26T15:15:50.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:49 smithi193 ceph-mon[37473]: pgmap v7988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:15:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13690.354674 secs 2024-07-26T15:15:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:51 smithi134 ceph-mon[31966]: pgmap v7989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:15:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:51 smithi189 ceph-mon[36790]: pgmap v7989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:15:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:51 smithi193 ceph-mon[37473]: pgmap v7989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:15:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13709.778476 secs 2024-07-26T15:15:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13709.778476 secs 2024-07-26T15:15:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13709.778476 secs 2024-07-26T15:15:53.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:53 smithi134 ceph-mon[31966]: pgmap v7990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:53 smithi189 ceph-mon[36790]: pgmap v7990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:54.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:53 smithi193 ceph-mon[37473]: pgmap v7990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:15:54.404+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:15:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:15:54.404+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:15:54.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:15:54.404+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13695.354805 secs 2024-07-26T15:15:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13695.354805 secs 2024-07-26T15:15:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13695.354805 secs 2024-07-26T15:15:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:55 smithi134 ceph-mon[31966]: pgmap v7991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:55 smithi189 ceph-mon[36790]: pgmap v7991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:55 smithi193 ceph-mon[37473]: pgmap v7991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:57 smithi134 ceph-mon[31966]: pgmap v7992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13714.778589 secs 2024-07-26T15:15:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:57 smithi189 ceph-mon[36790]: pgmap v7992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13714.778589 secs 2024-07-26T15:15:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:57 smithi193 ceph-mon[37473]: pgmap v7992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:15:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13714.778589 secs 2024-07-26T15:15:58.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:15:58.827+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:15:58.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:15:58.827+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:15:58.828 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:15:58.827+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:15:59.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:59 smithi134 ceph-mon[31966]: pgmap v7993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:15:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13700.354978 secs 2024-07-26T15:15:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:59 smithi189 ceph-mon[36790]: pgmap v7993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:15:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:15:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13700.354978 secs 2024-07-26T15:16:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:59 smithi193 ceph-mon[37473]: pgmap v7993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:15:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13700.354978 secs 2024-07-26T15:16:01.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:01 smithi134 ceph-mon[31966]: pgmap v7994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:16:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:01 smithi189 ceph-mon[36790]: pgmap v7994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:16:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:01 smithi193 ceph-mon[37473]: pgmap v7994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:16:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13719.778656 secs 2024-07-26T15:16:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13719.778656 secs 2024-07-26T15:16:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13719.778656 secs 2024-07-26T15:16:03.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:03 smithi134 ceph-mon[31966]: pgmap v7995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:03 smithi189 ceph-mon[36790]: pgmap v7995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:03 smithi193 ceph-mon[37473]: pgmap v7995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13705.355042 secs 2024-07-26T15:16:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13705.355042 secs 2024-07-26T15:16:05.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13705.355042 secs 2024-07-26T15:16:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:05 smithi189 ceph-mon[36790]: pgmap v7996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:06.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:05 smithi193 ceph-mon[37473]: pgmap v7996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:06.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:05 smithi134 ceph-mon[31966]: pgmap v7996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:07 smithi134 ceph-mon[31966]: pgmap v7997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13724.778804 secs 2024-07-26T15:16:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:07 smithi189 ceph-mon[36790]: pgmap v7997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13724.778804 secs 2024-07-26T15:16:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:07 smithi193 ceph-mon[37473]: pgmap v7997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13724.778804 secs 2024-07-26T15:16:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:16:09.415+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:16:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:16:09.415+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:16:09.417 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:16:09.415+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:09 smithi193 ceph-mon[37473]: pgmap v7998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13710.355167 secs 2024-07-26T15:16:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:09 smithi134 ceph-mon[31966]: pgmap v7998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13710.355167 secs 2024-07-26T15:16:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:09 smithi189 ceph-mon[36790]: pgmap v7998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13710.355167 secs 2024-07-26T15:16:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:11 smithi193 ceph-mon[37473]: pgmap v7999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:16:12.040 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:11 smithi134 ceph-mon[31966]: pgmap v7999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:16:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:11 smithi189 ceph-mon[36790]: pgmap v7999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:16:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13729.778964 secs 2024-07-26T15:16:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13729.778964 secs 2024-07-26T15:16:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13729.778964 secs 2024-07-26T15:16:13.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:16:13.833+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:16:13.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:16:13.833+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:13.835 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:16:13.833+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:16:14.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:13 smithi193 ceph-mon[37473]: pgmap v8000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:14.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:13 smithi134 ceph-mon[31966]: pgmap v8000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:13 smithi189 ceph-mon[36790]: pgmap v8000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13715.355287 secs 2024-07-26T15:16:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13715.355287 secs 2024-07-26T15:16:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13715.355287 secs 2024-07-26T15:16:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:15 smithi193 ceph-mon[37473]: pgmap v8001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:16.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:15 smithi134 ceph-mon[31966]: pgmap v8001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:15 smithi189 ceph-mon[36790]: pgmap v8001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:17 smithi193 ceph-mon[37473]: pgmap v8002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:18.066 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13734.779090 secs 2024-07-26T15:16:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:17 smithi134 ceph-mon[31966]: pgmap v8002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13734.779090 secs 2024-07-26T15:16:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:17 smithi189 ceph-mon[36790]: pgmap v8002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13734.779090 secs 2024-07-26T15:16:19.995 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:19 smithi189 ceph-mon[36790]: pgmap v8003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:19.995 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13720.355372 secs 2024-07-26T15:16:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:19 smithi193 ceph-mon[37473]: pgmap v8003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13720.355372 secs 2024-07-26T15:16:20.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:19 smithi134 ceph-mon[31966]: pgmap v8003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:20.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13720.355372 secs 2024-07-26T15:16:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:21 smithi193 ceph-mon[37473]: pgmap v8004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:16:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:21 smithi134 ceph-mon[31966]: pgmap v8004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:16:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:21 smithi189 ceph-mon[36790]: pgmap v8004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:16:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13739.779235 secs 2024-07-26T15:16:23.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13739.779235 secs 2024-07-26T15:16:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13739.779235 secs 2024-07-26T15:16:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:23 smithi193 ceph-mon[37473]: pgmap v8005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:24.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:23 smithi134 ceph-mon[31966]: pgmap v8005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:23 smithi189 ceph-mon[36790]: pgmap v8005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:16:24.420+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:16:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:16:24.420+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:24.421 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:16:24.420+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:16:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13725.355484 secs 2024-07-26T15:16:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:24 smithi134 ceph-mon[31966]: pgmap v8006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13725.355484 secs 2024-07-26T15:16:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:24 smithi189 ceph-mon[36790]: pgmap v8006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13725.355484 secs 2024-07-26T15:16:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:24 smithi193 ceph-mon[37473]: pgmap v8006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:26 smithi134 ceph-mon[31966]: pgmap v8007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13744.779379 secs 2024-07-26T15:16:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:26 smithi189 ceph-mon[36790]: pgmap v8007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13744.779379 secs 2024-07-26T15:16:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:26 smithi193 ceph-mon[37473]: pgmap v8007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13744.779379 secs 2024-07-26T15:16:28.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:16:28.848+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:16:28.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:16:28.849+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:16:28.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:16:28.849+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:29.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:28 smithi134 ceph-mon[31966]: pgmap v8008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:28 smithi189 ceph-mon[36790]: pgmap v8008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:28 smithi193 ceph-mon[37473]: pgmap v8008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13730.355633 secs 2024-07-26T15:16:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13730.355633 secs 2024-07-26T15:16:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13730.355633 secs 2024-07-26T15:16:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:30 smithi193 ceph-mon[37473]: pgmap v8009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:16:31.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:30 smithi134 ceph-mon[31966]: pgmap v8009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:16:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:30 smithi189 ceph-mon[36790]: pgmap v8009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:16:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13749.779453 secs 2024-07-26T15:16:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13749.779453 secs 2024-07-26T15:16:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13749.779453 secs 2024-07-26T15:16:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:32 smithi193 ceph-mon[37473]: pgmap v8010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:33.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:32 smithi134 ceph-mon[31966]: pgmap v8010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:33.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:32 smithi189 ceph-mon[36790]: pgmap v8010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13735.355752 secs 2024-07-26T15:16:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:34 smithi193 ceph-mon[37473]: pgmap v8011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:35.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13735.355752 secs 2024-07-26T15:16:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:34 smithi134 ceph-mon[31966]: pgmap v8011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13735.355752 secs 2024-07-26T15:16:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:34 smithi189 ceph-mon[36790]: pgmap v8011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:36 smithi193 ceph-mon[37473]: pgmap v8012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:37.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13754.779599 secs 2024-07-26T15:16:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:36 smithi134 ceph-mon[31966]: pgmap v8012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13754.779599 secs 2024-07-26T15:16:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:36 smithi189 ceph-mon[36790]: pgmap v8012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13754.779599 secs 2024-07-26T15:16:39.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:38 smithi193 ceph-mon[37473]: pgmap v8013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:39.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:38 smithi134 ceph-mon[31966]: pgmap v8013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:39.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:16:39.436+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:16:39.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:16:39.436+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:16:39.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:16:39.436+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:38 smithi189 ceph-mon[36790]: pgmap v8013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13740.355891 secs 2024-07-26T15:16:40.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13740.355891 secs 2024-07-26T15:16:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13740.355891 secs 2024-07-26T15:16:41.272 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:41 smithi189 ceph-mon[36790]: pgmap v8014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:16:41.272 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:16:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:41 smithi193 ceph-mon[37473]: pgmap v8014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:16:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:16:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:41 smithi134 ceph-mon[31966]: pgmap v8014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:16:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:16:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:42 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:16:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13759.779666 secs 2024-07-26T15:16:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:42 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:16:42.281 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:42.281 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:42.281 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:42.281 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:42.281 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:42 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:16:42.282 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13759.779666 secs 2024-07-26T15:16:42.282 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:42 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:16:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:16:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:42 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:16:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13759.779666 secs 2024-07-26T15:16:42.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:42 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:16:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:43 smithi193 ceph-mon[37473]: pgmap v8015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:43.275 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:43 smithi189 ceph-mon[36790]: pgmap v8015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:43.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:43 smithi134 ceph-mon[31966]: pgmap v8015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:43.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:16:43.856+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:16:43.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:16:43.856+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:16:43.859 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:16:43.856+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13745.355987 secs 2024-07-26T15:16:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:16:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:44 smithi134 ceph-mon[31966]: pgmap v8016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:16:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:16:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:16:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:16:45.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:16:45.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:44 smithi134 ceph-mon[31966]: pgmap v8017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:16:45.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:44 smithi134 ceph-mon[31966]: pgmap v8018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:16:45.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:16:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13745.355987 secs 2024-07-26T15:16:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:16:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:44 smithi189 ceph-mon[36790]: pgmap v8016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:16:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:16:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:16:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:16:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:16:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:44 smithi189 ceph-mon[36790]: pgmap v8017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:16:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:44 smithi189 ceph-mon[36790]: pgmap v8018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:16:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:16:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13745.355987 secs 2024-07-26T15:16:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:16:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:44 smithi193 ceph-mon[37473]: pgmap v8016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:16:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:16:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:16:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:16:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:16:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:44 smithi193 ceph-mon[37473]: pgmap v8017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:16:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:44 smithi193 ceph-mon[37473]: pgmap v8018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:16:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:16:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:47 smithi134 ceph-mon[31966]: pgmap v8019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T15:16:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13764.779753 secs 2024-07-26T15:16:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:47 smithi189 ceph-mon[36790]: pgmap v8019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T15:16:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13764.779753 secs 2024-07-26T15:16:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:47 smithi193 ceph-mon[37473]: pgmap v8019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T15:16:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13764.779753 secs 2024-07-26T15:16:49.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:49 smithi134 ceph-mon[31966]: pgmap v8020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:16:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:49 smithi189 ceph-mon[36790]: pgmap v8020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:16:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:49 smithi193 ceph-mon[37473]: pgmap v8020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:16:50.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13750.356101 secs 2024-07-26T15:16:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13750.356101 secs 2024-07-26T15:16:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13750.356101 secs 2024-07-26T15:16:51.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:51 smithi134 ceph-mon[31966]: pgmap v8021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:16:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:51 smithi189 ceph-mon[36790]: pgmap v8021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:16:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:51 smithi193 ceph-mon[37473]: pgmap v8021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:16:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13769.779815 secs 2024-07-26T15:16:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13769.779815 secs 2024-07-26T15:16:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13769.779815 secs 2024-07-26T15:16:53.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:53 smithi134 ceph-mon[31966]: pgmap v8022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T15:16:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:53 smithi189 ceph-mon[36790]: pgmap v8022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T15:16:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:53 smithi193 ceph-mon[37473]: pgmap v8022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T15:16:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:16:54.436+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:16:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:16:54.436+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:16:54.436+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:16:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13755.356196 secs 2024-07-26T15:16:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:55 smithi134 ceph-mon[31966]: pgmap v8023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:16:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13755.356196 secs 2024-07-26T15:16:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:55 smithi189 ceph-mon[36790]: pgmap v8023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:16:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13755.356196 secs 2024-07-26T15:16:55.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:55 smithi193 ceph-mon[37473]: pgmap v8023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:16:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:57 smithi134 ceph-mon[31966]: pgmap v8024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13774.779955 secs 2024-07-26T15:16:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:57 smithi189 ceph-mon[36790]: pgmap v8024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13774.779955 secs 2024-07-26T15:16:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:57 smithi193 ceph-mon[37473]: pgmap v8024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:16:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13774.779955 secs 2024-07-26T15:16:58.873 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:16:58.871+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:16:58.873 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:16:58.872+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:16:58.873 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:16:58.872+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:16:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:16:59 smithi134 ceph-mon[31966]: pgmap v8025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:16:59 smithi189 ceph-mon[36790]: pgmap v8025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:16:59.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:16:59 smithi193 ceph-mon[37473]: pgmap v8025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13760.356322 secs 2024-07-26T15:17:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13760.356322 secs 2024-07-26T15:17:00.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13760.356322 secs 2024-07-26T15:17:01.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:01 smithi134 ceph-mon[31966]: pgmap v8026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:01 smithi189 ceph-mon[36790]: pgmap v8026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:01.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:01 smithi193 ceph-mon[37473]: pgmap v8026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13779.780082 secs 2024-07-26T15:17:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13779.780082 secs 2024-07-26T15:17:02.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13779.780082 secs 2024-07-26T15:17:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:03 smithi189 ceph-mon[36790]: pgmap v8027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:03.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:03 smithi193 ceph-mon[37473]: pgmap v8027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:03.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:03 smithi134 ceph-mon[31966]: pgmap v8027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13765.356459 secs 2024-07-26T15:17:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:05 smithi189 ceph-mon[36790]: pgmap v8028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13765.356459 secs 2024-07-26T15:17:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:05 smithi193 ceph-mon[37473]: pgmap v8028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13765.356459 secs 2024-07-26T15:17:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:05 smithi134 ceph-mon[31966]: pgmap v8028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:07 smithi189 ceph-mon[36790]: pgmap v8029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13784.780222 secs 2024-07-26T15:17:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:07 smithi193 ceph-mon[37473]: pgmap v8029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13784.780222 secs 2024-07-26T15:17:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:07 smithi134 ceph-mon[31966]: pgmap v8029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13784.780222 secs 2024-07-26T15:17:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:17:09.452+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:17:09.453 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:17:09.452+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:09.454 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:17:09.452+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:17:09.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:09 smithi193 ceph-mon[37473]: pgmap v8030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:09 smithi134 ceph-mon[31966]: pgmap v8030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:09 smithi189 ceph-mon[36790]: pgmap v8030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13770.356600 secs 2024-07-26T15:17:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13770.356600 secs 2024-07-26T15:17:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13770.356600 secs 2024-07-26T15:17:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:11 smithi193 ceph-mon[37473]: pgmap v8031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:11.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:11 smithi134 ceph-mon[31966]: pgmap v8031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:11 smithi189 ceph-mon[36790]: pgmap v8031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13789.780349 secs 2024-07-26T15:17:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13789.780349 secs 2024-07-26T15:17:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13789.780349 secs 2024-07-26T15:17:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:13 smithi193 ceph-mon[37473]: pgmap v8032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:13.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:13 smithi134 ceph-mon[31966]: pgmap v8032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:13 smithi189 ceph-mon[36790]: pgmap v8032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:13.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:17:13.887+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:17:13.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:17:13.887+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:13.889 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:17:13.887+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:17:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13775.356659 secs 2024-07-26T15:17:15.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:15 smithi193 ceph-mon[37473]: pgmap v8033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13775.356659 secs 2024-07-26T15:17:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:15 smithi134 ceph-mon[31966]: pgmap v8033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13775.356659 secs 2024-07-26T15:17:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:15 smithi189 ceph-mon[36790]: pgmap v8033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:17 smithi189 ceph-mon[36790]: pgmap v8034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13794.780441 secs 2024-07-26T15:17:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:17 smithi193 ceph-mon[37473]: pgmap v8034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13794.780441 secs 2024-07-26T15:17:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:17 smithi134 ceph-mon[31966]: pgmap v8034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13794.780441 secs 2024-07-26T15:17:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:19 smithi189 ceph-mon[36790]: pgmap v8035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13780.356781 secs 2024-07-26T15:17:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:19 smithi193 ceph-mon[37473]: pgmap v8035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13780.356781 secs 2024-07-26T15:17:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:19 smithi134 ceph-mon[31966]: pgmap v8035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13780.356781 secs 2024-07-26T15:17:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:21 smithi189 ceph-mon[36790]: pgmap v8036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:21 smithi193 ceph-mon[37473]: pgmap v8036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:21 smithi134 ceph-mon[31966]: pgmap v8036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13799.780568 secs 2024-07-26T15:17:23.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13799.780568 secs 2024-07-26T15:17:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13799.780568 secs 2024-07-26T15:17:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:23 smithi193 ceph-mon[37473]: pgmap v8037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:24.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:23 smithi134 ceph-mon[31966]: pgmap v8037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:23 smithi189 ceph-mon[36790]: pgmap v8037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:24.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:17:24.463+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:17:24.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:17:24.463+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:17:24.464 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:17:24.463+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13785.356896 secs 2024-07-26T15:17:25.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13785.356896 secs 2024-07-26T15:17:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13785.356896 secs 2024-07-26T15:17:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:25 smithi193 ceph-mon[37473]: pgmap v8038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:26.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:25 smithi134 ceph-mon[31966]: pgmap v8038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:25 smithi189 ceph-mon[36790]: pgmap v8038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:27 smithi193 ceph-mon[37473]: pgmap v8039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13804.780693 secs 2024-07-26T15:17:28.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:27 smithi134 ceph-mon[31966]: pgmap v8039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13804.780693 secs 2024-07-26T15:17:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:27 smithi189 ceph-mon[36790]: pgmap v8039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13804.780693 secs 2024-07-26T15:17:28.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:17:28.903+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:17:28.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:17:28.903+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:28.904 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:17:28.903+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:17:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:29 smithi193 ceph-mon[37473]: pgmap v8040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13790.356990 secs 2024-07-26T15:17:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:29 smithi134 ceph-mon[31966]: pgmap v8040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13790.356990 secs 2024-07-26T15:17:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:29 smithi189 ceph-mon[36790]: pgmap v8040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13790.356990 secs 2024-07-26T15:17:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:31 smithi193 ceph-mon[37473]: pgmap v8041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:31 smithi134 ceph-mon[31966]: pgmap v8041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:31 smithi189 ceph-mon[36790]: pgmap v8041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:33.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13809.780843 secs 2024-07-26T15:17:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13809.780843 secs 2024-07-26T15:17:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13809.780843 secs 2024-07-26T15:17:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:33 smithi193 ceph-mon[37473]: pgmap v8042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:34.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:33 smithi134 ceph-mon[31966]: pgmap v8042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:33 smithi189 ceph-mon[36790]: pgmap v8042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13795.357115 secs 2024-07-26T15:17:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13795.357115 secs 2024-07-26T15:17:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13795.357115 secs 2024-07-26T15:17:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:35 smithi193 ceph-mon[37473]: pgmap v8043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:36.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:35 smithi134 ceph-mon[31966]: pgmap v8043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:35 smithi189 ceph-mon[36790]: pgmap v8043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:37 smithi193 ceph-mon[37473]: pgmap v8044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13814.780962 secs 2024-07-26T15:17:38.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:37 smithi134 ceph-mon[31966]: pgmap v8044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13814.780962 secs 2024-07-26T15:17:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:37 smithi189 ceph-mon[36790]: pgmap v8044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13814.780962 secs 2024-07-26T15:17:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:17:39.471+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:17:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:17:39.472+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:17:39.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:17:39.472+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:40.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:39 smithi134 ceph-mon[31966]: pgmap v8045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13800.357235 secs 2024-07-26T15:17:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:39 smithi189 ceph-mon[36790]: pgmap v8045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13800.357235 secs 2024-07-26T15:17:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:39 smithi193 ceph-mon[37473]: pgmap v8045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13800.357235 secs 2024-07-26T15:17:42.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:41 smithi193 ceph-mon[37473]: pgmap v8046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:41 smithi134 ceph-mon[31966]: pgmap v8046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:41 smithi189 ceph-mon[36790]: pgmap v8046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:17:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13819.781094 secs 2024-07-26T15:17:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:42 smithi134 ceph-mon[31966]: pgmap v8047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13819.781094 secs 2024-07-26T15:17:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:42 smithi189 ceph-mon[36790]: pgmap v8047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13819.781094 secs 2024-07-26T15:17:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:42 smithi193 ceph-mon[37473]: pgmap v8047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:17:43.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:17:43.918+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:17:43.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:17:43.918+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:43.920 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:17:43.918+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:17:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13805.357318 secs 2024-07-26T15:17:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:45 smithi134 ceph-mon[31966]: pgmap v8048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:17:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:45 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:17:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:45 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:17:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:45 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:17:45.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:45 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:17:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13805.357318 secs 2024-07-26T15:17:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:45 smithi189 ceph-mon[36790]: pgmap v8048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:17:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:45 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:17:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:45 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:17:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:45 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:17:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:45 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:17:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13805.357318 secs 2024-07-26T15:17:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:45 smithi193 ceph-mon[37473]: pgmap v8048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:17:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:17:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:17:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:45 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:17:45.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:45 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:17:45.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:45 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:17:45.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:45 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:17:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:17:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:17:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:46 smithi134 ceph-mon[31966]: pgmap v8049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:17:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:46 smithi134 ceph-mon[31966]: pgmap v8050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:17:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:17:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:17:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:17:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:46 smithi189 ceph-mon[36790]: pgmap v8049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:17:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:46 smithi189 ceph-mon[36790]: pgmap v8050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:17:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:17:46.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:17:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:17:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:46 smithi193 ceph-mon[37473]: pgmap v8049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:17:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:46 smithi193 ceph-mon[37473]: pgmap v8050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:17:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:17:47.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13824.781195 secs 2024-07-26T15:17:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13824.781195 secs 2024-07-26T15:17:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13824.781195 secs 2024-07-26T15:17:48.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:48 smithi134 ceph-mon[31966]: pgmap v8051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:17:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:48 smithi189 ceph-mon[36790]: pgmap v8051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:17:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:48 smithi193 ceph-mon[37473]: pgmap v8051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:17:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13810.357434 secs 2024-07-26T15:17:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:50 smithi134 ceph-mon[31966]: pgmap v8052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:17:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13810.357434 secs 2024-07-26T15:17:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:50 smithi189 ceph-mon[36790]: pgmap v8052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:17:50.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13810.357434 secs 2024-07-26T15:17:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:50 smithi193 ceph-mon[37473]: pgmap v8052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:17:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13829.781344 secs 2024-07-26T15:17:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:52 smithi134 ceph-mon[31966]: pgmap v8053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:17:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13829.781344 secs 2024-07-26T15:17:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:52 smithi189 ceph-mon[36790]: pgmap v8053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:17:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13829.781344 secs 2024-07-26T15:17:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:52 smithi193 ceph-mon[37473]: pgmap v8053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:17:54.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:17:54.480+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:17:54.481 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:17:54.480+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:17:54.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:17:54.480+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:54.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:54 smithi134 ceph-mon[31966]: pgmap v8054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:17:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:54 smithi189 ceph-mon[36790]: pgmap v8054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:17:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:54 smithi193 ceph-mon[37473]: pgmap v8054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:17:55.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13815.357573 secs 2024-07-26T15:17:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13815.357573 secs 2024-07-26T15:17:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13815.357573 secs 2024-07-26T15:17:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:56 smithi134 ceph-mon[31966]: pgmap v8055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:17:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:56 smithi189 ceph-mon[36790]: pgmap v8055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:17:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:56 smithi193 ceph-mon[37473]: pgmap v8055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:17:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13834.781462 secs 2024-07-26T15:17:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13834.781462 secs 2024-07-26T15:17:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13834.781462 secs 2024-07-26T15:17:58.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:17:58 smithi134 ceph-mon[31966]: pgmap v8056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:17:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:17:58 smithi189 ceph-mon[36790]: pgmap v8056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:17:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:17:58 smithi193 ceph-mon[37473]: pgmap v8056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:17:58.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:17:58.934+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:17:58.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:17:58.934+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:17:58.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:17:58.934+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13820.357695 secs 2024-07-26T15:18:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:00 smithi134 ceph-mon[31966]: pgmap v8057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13820.357695 secs 2024-07-26T15:18:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:00 smithi189 ceph-mon[36790]: pgmap v8057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:00.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13820.357695 secs 2024-07-26T15:18:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:00 smithi193 ceph-mon[37473]: pgmap v8057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13839.781618 secs 2024-07-26T15:18:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:02 smithi134 ceph-mon[31966]: pgmap v8058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13839.781618 secs 2024-07-26T15:18:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:02 smithi189 ceph-mon[36790]: pgmap v8058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13839.781618 secs 2024-07-26T15:18:02.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:02 smithi193 ceph-mon[37473]: pgmap v8058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:04.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:04 smithi134 ceph-mon[31966]: pgmap v8059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:18:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:04 smithi189 ceph-mon[36790]: pgmap v8059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:18:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:04 smithi193 ceph-mon[37473]: pgmap v8059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:18:05.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13825.357813 secs 2024-07-26T15:18:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13825.357813 secs 2024-07-26T15:18:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13825.357813 secs 2024-07-26T15:18:06.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:06 smithi134 ceph-mon[31966]: pgmap v8060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:06 smithi189 ceph-mon[36790]: pgmap v8060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:06.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:06 smithi193 ceph-mon[37473]: pgmap v8060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:07.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13844.781759 secs 2024-07-26T15:18:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13844.781759 secs 2024-07-26T15:18:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13844.781759 secs 2024-07-26T15:18:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:08 smithi134 ceph-mon[31966]: pgmap v8061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:08 smithi189 ceph-mon[36790]: pgmap v8061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:08.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:08 smithi193 ceph-mon[37473]: pgmap v8061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:09.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:09.495+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:09.495+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:09.497 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:09.495+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:09.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13830.357908 secs 2024-07-26T15:18:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13830.357908 secs 2024-07-26T15:18:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13830.357908 secs 2024-07-26T15:18:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:10 smithi134 ceph-mon[31966]: pgmap v8062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:10 smithi189 ceph-mon[36790]: pgmap v8062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:10 smithi193 ceph-mon[37473]: pgmap v8062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13849.781899 secs 2024-07-26T15:18:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:12 smithi134 ceph-mon[31966]: pgmap v8063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13849.781899 secs 2024-07-26T15:18:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:12 smithi189 ceph-mon[36790]: pgmap v8063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13849.781899 secs 2024-07-26T15:18:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:12 smithi193 ceph-mon[37473]: pgmap v8063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:13.950+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:13.950+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:13.950+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:14 smithi189 ceph-mon[36790]: pgmap v8064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:18:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13835.358000 secs 2024-07-26T15:18:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:14 smithi193 ceph-mon[37473]: pgmap v8064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:18:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13835.358000 secs 2024-07-26T15:18:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:14 smithi134 ceph-mon[31966]: pgmap v8064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:18:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13835.358000 secs 2024-07-26T15:18:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:16 smithi189 ceph-mon[36790]: pgmap v8065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:16.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:16 smithi193 ceph-mon[37473]: pgmap v8065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:16.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:16 smithi134 ceph-mon[31966]: pgmap v8065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13854.782014 secs 2024-07-26T15:18:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13854.782014 secs 2024-07-26T15:18:17.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13854.782014 secs 2024-07-26T15:18:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:18 smithi189 ceph-mon[36790]: pgmap v8066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:18 smithi193 ceph-mon[37473]: pgmap v8066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:18 smithi134 ceph-mon[31966]: pgmap v8066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:18:19.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13840.358118 secs 2024-07-26T15:18:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13840.358118 secs 2024-07-26T15:18:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13840.358118 secs 2024-07-26T15:18:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:20 smithi193 ceph-mon[37473]: pgmap v8067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:20.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:20 smithi189 ceph-mon[36790]: pgmap v8067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:20.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:20 smithi134 ceph-mon[31966]: pgmap v8067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13859.782145 secs 2024-07-26T15:18:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:22 smithi193 ceph-mon[37473]: pgmap v8068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13859.782145 secs 2024-07-26T15:18:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:22 smithi134 ceph-mon[31966]: pgmap v8068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13859.782145 secs 2024-07-26T15:18:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:22 smithi189 ceph-mon[36790]: pgmap v8068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:24.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:24.511+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:24.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:24.511+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:24.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:24.511+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:24.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:24.738+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:18:24.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:24.738+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:18:24.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:24 smithi134 ceph-mon[31966]: pgmap v8069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:18:24.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13845.358232 secs 2024-07-26T15:18:24.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:24.739+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T15:18:24.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:24.739+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T15:18:24.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:24.739+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 108 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T15:18:24.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:24.739+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T15:18:24.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:24.740+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T15:18:24.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:24.740+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T15:18:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:24 smithi193 ceph-mon[37473]: pgmap v8069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:18:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13845.358232 secs 2024-07-26T15:18:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:24 smithi189 ceph-mon[36790]: pgmap v8069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:18:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13845.358232 secs 2024-07-26T15:18:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:26 smithi193 ceph-mon[37473]: pgmap v8070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:26 smithi134 ceph-mon[31966]: pgmap v8070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:26 smithi189 ceph-mon[36790]: pgmap v8070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13864.782248 secs 2024-07-26T15:18:28.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13864.782248 secs 2024-07-26T15:18:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13864.782248 secs 2024-07-26T15:18:28.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:28.965+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:28.966 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:28.965+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:28.967 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:28.965+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:28 smithi193 ceph-mon[37473]: pgmap v8071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:18:29.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:28 smithi134 ceph-mon[31966]: pgmap v8071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:18:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:28 smithi189 ceph-mon[36790]: pgmap v8071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:18:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13850.358316 secs 2024-07-26T15:18:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13850.358316 secs 2024-07-26T15:18:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13850.358316 secs 2024-07-26T15:18:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:30 smithi193 ceph-mon[37473]: pgmap v8072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:18:31.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:30 smithi134 ceph-mon[31966]: pgmap v8072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:18:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:30 smithi189 ceph-mon[36790]: pgmap v8072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:18:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13869.782351 secs 2024-07-26T15:18:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:32 smithi193 ceph-mon[37473]: pgmap v8073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:18:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13869.782351 secs 2024-07-26T15:18:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:32 smithi134 ceph-mon[31966]: pgmap v8073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:18:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13869.782351 secs 2024-07-26T15:18:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:32 smithi189 ceph-mon[36790]: pgmap v8073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:18:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:34 smithi193 ceph-mon[37473]: pgmap v8074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:18:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13855.358435 secs 2024-07-26T15:18:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:34 smithi134 ceph-mon[31966]: pgmap v8074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:18:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13855.358435 secs 2024-07-26T15:18:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:34 smithi189 ceph-mon[36790]: pgmap v8074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:18:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13855.358435 secs 2024-07-26T15:18:35.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:35.276+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:18:35.277 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:35.276+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:18:35.277 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:35.276+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:18:35.277 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:35.276+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 109 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:18:35.278 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:35.277+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:18:35.278 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:35.277+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:18:35.278 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:35.277+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:18:35.278 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:35.277+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:18:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:36 smithi193 ceph-mon[37473]: pgmap v8075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:18:37.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:36 smithi134 ceph-mon[31966]: pgmap v8075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:18:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:36 smithi189 ceph-mon[36790]: pgmap v8075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:18:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13874.782482 secs 2024-07-26T15:18:38.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13874.782482 secs 2024-07-26T15:18:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13874.782482 secs 2024-07-26T15:18:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:38 smithi193 ceph-mon[37473]: pgmap v8076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:18:39.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:38 smithi134 ceph-mon[31966]: pgmap v8076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:18:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:38 smithi189 ceph-mon[36790]: pgmap v8076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:18:39.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:39.512+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:39.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:39.512+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:39.514 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:39.512+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:40.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13860.358499 secs 2024-07-26T15:18:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13860.358499 secs 2024-07-26T15:18:40.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13860.358499 secs 2024-07-26T15:18:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:40 smithi134 ceph-mon[31966]: pgmap v8077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:40 smithi189 ceph-mon[36790]: pgmap v8077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:41.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:40 smithi193 ceph-mon[37473]: pgmap v8077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:18:43.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13879.782613 secs 2024-07-26T15:18:43.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:42 smithi193 ceph-mon[37473]: pgmap v8078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:43.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13879.782613 secs 2024-07-26T15:18:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:42 smithi134 ceph-mon[31966]: pgmap v8078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13879.782613 secs 2024-07-26T15:18:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:42 smithi189 ceph-mon[36790]: pgmap v8078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:43.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:43.981+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:43.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:43.981+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:43.982 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:43.981+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:44 smithi134 ceph-mon[31966]: pgmap v8079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:18:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13865.358621 secs 2024-07-26T15:18:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:44 smithi189 ceph-mon[36790]: pgmap v8079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:18:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13865.358621 secs 2024-07-26T15:18:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:44 smithi193 ceph-mon[37473]: pgmap v8079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:18:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13865.358621 secs 2024-07-26T15:18:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:45 smithi134 ceph-mon[31966]: pgmap v8080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:18:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:45 smithi189 ceph-mon[36790]: pgmap v8080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:18:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:45 smithi193 ceph-mon[37473]: pgmap v8080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:18:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:18:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:18:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:18:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:46 smithi134 ceph-mon[31966]: pgmap v8081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:18:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:46 smithi134 ceph-mon[31966]: pgmap v8082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:18:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:18:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:18:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:18:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:46 smithi189 ceph-mon[36790]: pgmap v8081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:18:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:46 smithi189 ceph-mon[36790]: pgmap v8082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:18:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:18:47.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:18:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:18:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:46 smithi193 ceph-mon[37473]: pgmap v8081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:18:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:46 smithi193 ceph-mon[37473]: pgmap v8082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:18:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:18:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13884.782730 secs 2024-07-26T15:18:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13884.782730 secs 2024-07-26T15:18:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13884.782730 secs 2024-07-26T15:18:49.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:48 smithi134 ceph-mon[31966]: pgmap v8083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:18:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:48 smithi189 ceph-mon[36790]: pgmap v8083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:18:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:48 smithi193 ceph-mon[37473]: pgmap v8083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:18:50.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13870.358685 secs 2024-07-26T15:18:50.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13870.358685 secs 2024-07-26T15:18:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13870.358685 secs 2024-07-26T15:18:51.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:50 smithi134 ceph-mon[31966]: pgmap v8084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:18:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:50 smithi189 ceph-mon[36790]: pgmap v8084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:18:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:50 smithi193 ceph-mon[37473]: pgmap v8084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:18:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13889.782855 secs 2024-07-26T15:18:52.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13889.782855 secs 2024-07-26T15:18:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13889.782855 secs 2024-07-26T15:18:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:53 smithi193 ceph-mon[37473]: pgmap v8085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T15:18:53.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:53 smithi134 ceph-mon[31966]: pgmap v8085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T15:18:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:53 smithi189 ceph-mon[36790]: pgmap v8085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T15:18:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:54.522+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:54.522+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:54.523 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:18:54.522+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13875.358782 secs 2024-07-26T15:18:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13875.358782 secs 2024-07-26T15:18:54.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13875.358782 secs 2024-07-26T15:18:55.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:55 smithi193 ceph-mon[37473]: pgmap v8086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:18:55.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:55 smithi134 ceph-mon[31966]: pgmap v8086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:18:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:55 smithi189 ceph-mon[36790]: pgmap v8086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:18:57.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13894.782997 secs 2024-07-26T15:18:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:57 smithi193 ceph-mon[37473]: pgmap v8087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:18:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13894.782997 secs 2024-07-26T15:18:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:57 smithi134 ceph-mon[31966]: pgmap v8087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:18:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13894.782997 secs 2024-07-26T15:18:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:57 smithi189 ceph-mon[36790]: pgmap v8087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:18:58.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:58.987+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:58.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:58.987+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:18:58.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:18:58.987+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:18:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:59 smithi134 ceph-mon[31966]: pgmap v8088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:18:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:18:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13880.358905 secs 2024-07-26T15:18:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:59 smithi189 ceph-mon[36790]: pgmap v8088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:18:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:18:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13880.358905 secs 2024-07-26T15:19:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:59 smithi193 ceph-mon[37473]: pgmap v8088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:19:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:18:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13880.358905 secs 2024-07-26T15:19:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:01 smithi134 ceph-mon[31966]: pgmap v8089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:01 smithi189 ceph-mon[36790]: pgmap v8089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:02.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:01 smithi193 ceph-mon[37473]: pgmap v8089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:02.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13899.783119 secs 2024-07-26T15:19:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13899.783119 secs 2024-07-26T15:19:03.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13899.783119 secs 2024-07-26T15:19:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:03 smithi134 ceph-mon[31966]: pgmap v8090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:03 smithi189 ceph-mon[36790]: pgmap v8090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:04.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:03 smithi193 ceph-mon[37473]: pgmap v8090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13885.359020 secs 2024-07-26T15:19:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13885.359020 secs 2024-07-26T15:19:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13885.359020 secs 2024-07-26T15:19:05.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:05 smithi134 ceph-mon[31966]: pgmap v8091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:19:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:05 smithi189 ceph-mon[36790]: pgmap v8091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:19:06.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:05 smithi193 ceph-mon[37473]: pgmap v8091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:19:07.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13904.783249 secs 2024-07-26T15:19:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:07 smithi134 ceph-mon[31966]: pgmap v8092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13904.783249 secs 2024-07-26T15:19:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:07 smithi189 ceph-mon[36790]: pgmap v8092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13904.783249 secs 2024-07-26T15:19:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:07 smithi193 ceph-mon[37473]: pgmap v8092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:19:09.536+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:19:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:19:09.536+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:09.538 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:19:09.536+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:19:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:09 smithi134 ceph-mon[31966]: pgmap v8093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:19:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13890.359109 secs 2024-07-26T15:19:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:09 smithi189 ceph-mon[36790]: pgmap v8093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:19:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13890.359109 secs 2024-07-26T15:19:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:09 smithi193 ceph-mon[37473]: pgmap v8093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:19:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13890.359109 secs 2024-07-26T15:19:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:11 smithi134 ceph-mon[31966]: pgmap v8094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:11 smithi189 ceph-mon[36790]: pgmap v8094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:11 smithi193 ceph-mon[37473]: pgmap v8094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:12.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13909.783396 secs 2024-07-26T15:19:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13909.783396 secs 2024-07-26T15:19:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13909.783396 secs 2024-07-26T15:19:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:13 smithi134 ceph-mon[31966]: pgmap v8095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:13.924 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:13 smithi193 ceph-mon[37473]: pgmap v8095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:13 smithi189 ceph-mon[36790]: pgmap v8095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:14.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:19:14.002+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:19:14.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:19:14.002+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:14.004 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:19:14.002+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:19:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13895.359222 secs 2024-07-26T15:19:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13895.359222 secs 2024-07-26T15:19:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13895.359222 secs 2024-07-26T15:19:15.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:15 smithi134 ceph-mon[31966]: pgmap v8096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:19:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:15 smithi189 ceph-mon[36790]: pgmap v8096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:19:16.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:15 smithi193 ceph-mon[37473]: pgmap v8096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:19:17.896 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13914.783460 secs 2024-07-26T15:19:17.897 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:17 smithi134 ceph-mon[31966]: pgmap v8097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13914.783460 secs 2024-07-26T15:19:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:17 smithi189 ceph-mon[36790]: pgmap v8097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13914.783460 secs 2024-07-26T15:19:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:17 smithi193 ceph-mon[37473]: pgmap v8097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:19 smithi134 ceph-mon[31966]: pgmap v8098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:19:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13900.359301 secs 2024-07-26T15:19:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:19 smithi189 ceph-mon[36790]: pgmap v8098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:19:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13900.359301 secs 2024-07-26T15:19:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:19 smithi193 ceph-mon[37473]: pgmap v8098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:19:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13900.359301 secs 2024-07-26T15:19:21.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:21 smithi134 ceph-mon[31966]: pgmap v8099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:21 smithi189 ceph-mon[36790]: pgmap v8099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:21 smithi193 ceph-mon[37473]: pgmap v8099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:22.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13919.783544 secs 2024-07-26T15:19:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13919.783544 secs 2024-07-26T15:19:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13919.783544 secs 2024-07-26T15:19:23.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:23 smithi134 ceph-mon[31966]: pgmap v8100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:23 smithi189 ceph-mon[36790]: pgmap v8100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:23 smithi193 ceph-mon[37473]: pgmap v8100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:24.547 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:19:24.546+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:19:24.546+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:19:24.548 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:19:24.546+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13905.359418 secs 2024-07-26T15:19:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13905.359418 secs 2024-07-26T15:19:25.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13905.359418 secs 2024-07-26T15:19:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:25 smithi134 ceph-mon[31966]: pgmap v8101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:19:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:25 smithi189 ceph-mon[36790]: pgmap v8101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:19:26.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:25 smithi193 ceph-mon[37473]: pgmap v8101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:19:27.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13924.783607 secs 2024-07-26T15:19:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:27 smithi134 ceph-mon[31966]: pgmap v8102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13924.783607 secs 2024-07-26T15:19:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:27 smithi189 ceph-mon[36790]: pgmap v8102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13924.783607 secs 2024-07-26T15:19:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:27 smithi193 ceph-mon[37473]: pgmap v8102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:29.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:19:29.008+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:19:29.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:19:29.008+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:19:29.010 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:19:29.008+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:29 smithi134 ceph-mon[31966]: pgmap v8103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:19:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13910.359542 secs 2024-07-26T15:19:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:29 smithi189 ceph-mon[36790]: pgmap v8103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:19:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13910.359542 secs 2024-07-26T15:19:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:29 smithi193 ceph-mon[37473]: pgmap v8103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:19:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13910.359542 secs 2024-07-26T15:19:31.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:31 smithi134 ceph-mon[31966]: pgmap v8104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:31 smithi189 ceph-mon[36790]: pgmap v8104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:31 smithi193 ceph-mon[37473]: pgmap v8104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13929.783735 secs 2024-07-26T15:19:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13929.783735 secs 2024-07-26T15:19:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13929.783735 secs 2024-07-26T15:19:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:33 smithi189 ceph-mon[36790]: pgmap v8105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:33 smithi193 ceph-mon[37473]: pgmap v8105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:33 smithi134 ceph-mon[31966]: pgmap v8105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13915.359666 secs 2024-07-26T15:19:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13915.359666 secs 2024-07-26T15:19:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13915.359666 secs 2024-07-26T15:19:36.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:35 smithi193 ceph-mon[37473]: pgmap v8106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:19:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:35 smithi134 ceph-mon[31966]: pgmap v8106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:19:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:35 smithi189 ceph-mon[36790]: pgmap v8106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:19:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13934.783847 secs 2024-07-26T15:19:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:37 smithi193 ceph-mon[37473]: pgmap v8107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:38.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13934.783847 secs 2024-07-26T15:19:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:37 smithi134 ceph-mon[31966]: pgmap v8107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13934.783847 secs 2024-07-26T15:19:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:37 smithi189 ceph-mon[36790]: pgmap v8107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:19:39.562+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:19:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:19:39.562+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:19:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:19:39.562+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:39 smithi193 ceph-mon[37473]: pgmap v8108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:19:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13920.359784 secs 2024-07-26T15:19:40.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:39 smithi134 ceph-mon[31966]: pgmap v8108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:19:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13920.359784 secs 2024-07-26T15:19:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:39 smithi189 ceph-mon[36790]: pgmap v8108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:19:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13920.359784 secs 2024-07-26T15:19:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:41 smithi193 ceph-mon[37473]: pgmap v8109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:41 smithi134 ceph-mon[31966]: pgmap v8109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:41 smithi189 ceph-mon[36790]: pgmap v8109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:43.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13939.783972 secs 2024-07-26T15:19:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13939.783972 secs 2024-07-26T15:19:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13939.783972 secs 2024-07-26T15:19:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:43 smithi193 ceph-mon[37473]: pgmap v8110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:44.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:19:44.023+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:19:44.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:19:44.024+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:19:44.025 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:19:44.024+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:44.025 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:43 smithi134 ceph-mon[31966]: pgmap v8110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:43 smithi189 ceph-mon[36790]: pgmap v8110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:19:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13925.359904 secs 2024-07-26T15:19:45.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13925.359904 secs 2024-07-26T15:19:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13925.359904 secs 2024-07-26T15:19:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:45 smithi193 ceph-mon[37473]: pgmap v8111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:19:46.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:45 smithi134 ceph-mon[31966]: pgmap v8111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:19:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:45 smithi189 ceph-mon[36790]: pgmap v8111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:19:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:19:47.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:19:47.192 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:19:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13944.784094 secs 2024-07-26T15:19:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:47 smithi193 ceph-mon[37473]: pgmap v8112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:19:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:19:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13944.784094 secs 2024-07-26T15:19:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:47 smithi134 ceph-mon[31966]: pgmap v8112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:19:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:19:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13944.784094 secs 2024-07-26T15:19:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:47 smithi189 ceph-mon[36790]: pgmap v8112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:19:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:19:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:19:49.128 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:19:49.129 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:19:49.129 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:19:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:19:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:19:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:19:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:19:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:19:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:19:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:49 smithi134 ceph-mon[31966]: pgmap v8113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T15:19:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:49 smithi134 ceph-mon[31966]: pgmap v8114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13930.360018 secs 2024-07-26T15:19:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:49 smithi189 ceph-mon[36790]: pgmap v8113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T15:19:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:49 smithi189 ceph-mon[36790]: pgmap v8114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13930.360018 secs 2024-07-26T15:19:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:49 smithi193 ceph-mon[37473]: pgmap v8113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T15:19:50.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:49 smithi193 ceph-mon[37473]: pgmap v8114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13930.360018 secs 2024-07-26T15:19:52.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:51 smithi134 ceph-mon[31966]: pgmap v8115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:51 smithi189 ceph-mon[36790]: pgmap v8115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:51 smithi193 ceph-mon[37473]: pgmap v8115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:53.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13949.784224 secs 2024-07-26T15:19:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13949.784224 secs 2024-07-26T15:19:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13949.784224 secs 2024-07-26T15:19:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:53 smithi134 ceph-mon[31966]: pgmap v8116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:19:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:53 smithi189 ceph-mon[36790]: pgmap v8116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:19:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:53 smithi193 ceph-mon[37473]: pgmap v8116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:19:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:19:54.577+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:19:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:19:54.577+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:19:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:19:54.577+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:55.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13935.360098 secs 2024-07-26T15:19:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13935.360098 secs 2024-07-26T15:19:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13935.360098 secs 2024-07-26T15:19:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:55 smithi134 ceph-mon[31966]: pgmap v8117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:55 smithi189 ceph-mon[36790]: pgmap v8117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:55 smithi193 ceph-mon[37473]: pgmap v8117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:57.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:56 smithi134 ceph-mon[31966]: pgmap v8118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:56 smithi189 ceph-mon[36790]: pgmap v8118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:57.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:56 smithi193 ceph-mon[37473]: pgmap v8118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:19:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13954.784374 secs 2024-07-26T15:19:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13954.784374 secs 2024-07-26T15:19:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13954.784374 secs 2024-07-26T15:19:59.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:19:59.039+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:19:59.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:19:59.039+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:19:59.041 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:19:59.039+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:19:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:58 smithi134 ceph-mon[31966]: pgmap v8119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:19:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:58 smithi189 ceph-mon[36790]: pgmap v8119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:19:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:58 smithi193 ceph-mon[37473]: pgmap v8119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:20:00.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:19:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13940.360180 secs 2024-07-26T15:20:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:19:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13940.360180 secs 2024-07-26T15:20:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:19:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13940.360180 secs 2024-07-26T15:20:01.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:20:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:00 smithi134 ceph-mon[31966]: pgmap v8120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:20:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:00 smithi189 ceph-mon[36790]: pgmap v8120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:20:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:00 smithi193 ceph-mon[37473]: pgmap v8120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13959.784503 secs 2024-07-26T15:20:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13959.784503 secs 2024-07-26T15:20:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13959.784503 secs 2024-07-26T15:20:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:03 smithi193 ceph-mon[37473]: pgmap v8121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:03 smithi134 ceph-mon[31966]: pgmap v8121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:03 smithi189 ceph-mon[36790]: pgmap v8121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13945.360248 secs 2024-07-26T15:20:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:05 smithi134 ceph-mon[31966]: pgmap v8122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13945.360248 secs 2024-07-26T15:20:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:05 smithi189 ceph-mon[36790]: pgmap v8122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:05.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13945.360248 secs 2024-07-26T15:20:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:05 smithi193 ceph-mon[37473]: pgmap v8122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13964.784601 secs 2024-07-26T15:20:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:07 smithi134 ceph-mon[31966]: pgmap v8123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13964.784601 secs 2024-07-26T15:20:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:07 smithi189 ceph-mon[36790]: pgmap v8123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13964.784601 secs 2024-07-26T15:20:07.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:07 smithi193 ceph-mon[37473]: pgmap v8123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:20:09.592+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:20:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:20:09.592+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:20:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:20:09.592+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:09 smithi134 ceph-mon[31966]: pgmap v8124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:20:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:09 smithi189 ceph-mon[36790]: pgmap v8124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:20:09.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:09 smithi193 ceph-mon[37473]: pgmap v8124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:20:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13950.360331 secs 2024-07-26T15:20:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13950.360331 secs 2024-07-26T15:20:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13950.360331 secs 2024-07-26T15:20:11.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:11 smithi134 ceph-mon[31966]: pgmap v8125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:20:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:11 smithi189 ceph-mon[36790]: pgmap v8125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:20:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:11 smithi193 ceph-mon[37473]: pgmap v8125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:20:12.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13969.784725 secs 2024-07-26T15:20:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13969.784725 secs 2024-07-26T15:20:12.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13969.784725 secs 2024-07-26T15:20:13.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:13 smithi193 ceph-mon[37473]: pgmap v8126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:20:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:13 smithi189 ceph-mon[36790]: pgmap v8126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:20:13.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:13 smithi134 ceph-mon[31966]: pgmap v8126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:20:14.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:20:14.054+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:20:14.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:20:14.054+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:20:14.056 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:20:14.054+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13955.360469 secs 2024-07-26T15:20:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13955.360469 secs 2024-07-26T15:20:14.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13955.360469 secs 2024-07-26T15:20:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:15 smithi189 ceph-mon[36790]: pgmap v8127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:20:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:15 smithi193 ceph-mon[37473]: pgmap v8127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:20:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:15 smithi134 ceph-mon[31966]: pgmap v8127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:20:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13974.784862 secs 2024-07-26T15:20:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:17 smithi193 ceph-mon[37473]: pgmap v8128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:20:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13974.784862 secs 2024-07-26T15:20:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:17 smithi134 ceph-mon[31966]: pgmap v8128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:20:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13974.784862 secs 2024-07-26T15:20:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:17 smithi189 ceph-mon[36790]: pgmap v8128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:20:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:19 smithi193 ceph-mon[37473]: pgmap v8129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:20:20.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13960.360611 secs 2024-07-26T15:20:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:19 smithi134 ceph-mon[31966]: pgmap v8129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:20:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13960.360611 secs 2024-07-26T15:20:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:19 smithi189 ceph-mon[36790]: pgmap v8129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:20:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13960.360611 secs 2024-07-26T15:20:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:21 smithi193 ceph-mon[37473]: pgmap v8130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:22.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:21 smithi134 ceph-mon[31966]: pgmap v8130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:21 smithi189 ceph-mon[36790]: pgmap v8130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:23.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13979.784961 secs 2024-07-26T15:20:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13979.784961 secs 2024-07-26T15:20:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13979.784961 secs 2024-07-26T15:20:24.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:23 smithi134 ceph-mon[31966]: pgmap v8131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:23 smithi189 ceph-mon[36790]: pgmap v8131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:23 smithi193 ceph-mon[37473]: pgmap v8131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:24.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:20:24.595+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:20:24.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:20:24.595+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:20:24.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:20:24.595+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:25.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13965.360738 secs 2024-07-26T15:20:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13965.360738 secs 2024-07-26T15:20:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13965.360738 secs 2024-07-26T15:20:26.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:25 smithi134 ceph-mon[31966]: pgmap v8132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:25 smithi189 ceph-mon[36790]: pgmap v8132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:25 smithi193 ceph-mon[37473]: pgmap v8132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13984.785086 secs 2024-07-26T15:20:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:27 smithi134 ceph-mon[31966]: pgmap v8133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13984.785086 secs 2024-07-26T15:20:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:27 smithi189 ceph-mon[36790]: pgmap v8133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13984.785086 secs 2024-07-26T15:20:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:27 smithi193 ceph-mon[37473]: pgmap v8133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:29.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:20:29.062+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:20:29.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:20:29.062+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:20:29.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:20:29.062+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:29 smithi134 ceph-mon[31966]: pgmap v8134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:20:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13970.360834 secs 2024-07-26T15:20:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:29 smithi189 ceph-mon[36790]: pgmap v8134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:20:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13970.360834 secs 2024-07-26T15:20:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:29 smithi193 ceph-mon[37473]: pgmap v8134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:20:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13970.360834 secs 2024-07-26T15:20:32.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:31 smithi134 ceph-mon[31966]: pgmap v8135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:31 smithi189 ceph-mon[36790]: pgmap v8135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:32.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:31 smithi193 ceph-mon[37473]: pgmap v8135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13989.785220 secs 2024-07-26T15:20:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:32 smithi134 ceph-mon[31966]: pgmap v8136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13989.785220 secs 2024-07-26T15:20:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:32 smithi189 ceph-mon[36790]: pgmap v8136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13989.785220 secs 2024-07-26T15:20:33.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:32 smithi193 ceph-mon[37473]: pgmap v8136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13975.360984 secs 2024-07-26T15:20:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:35 smithi134 ceph-mon[31966]: pgmap v8137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13975.360984 secs 2024-07-26T15:20:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:35 smithi189 ceph-mon[36790]: pgmap v8137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13975.360984 secs 2024-07-26T15:20:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:35 smithi193 ceph-mon[37473]: pgmap v8137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13994.785340 secs 2024-07-26T15:20:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:37 smithi134 ceph-mon[31966]: pgmap v8138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13994.785340 secs 2024-07-26T15:20:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:37 smithi189 ceph-mon[36790]: pgmap v8138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13994.785340 secs 2024-07-26T15:20:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:37 smithi193 ceph-mon[37473]: pgmap v8138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:39.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:20:39.608+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:20:39.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:20:39.608+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:39.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:20:39.608+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:20:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:39 smithi134 ceph-mon[31966]: pgmap v8139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:20:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13980.361105 secs 2024-07-26T15:20:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:39 smithi189 ceph-mon[36790]: pgmap v8139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:20:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13980.361105 secs 2024-07-26T15:20:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:39 smithi193 ceph-mon[37473]: pgmap v8139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:20:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13980.361105 secs 2024-07-26T15:20:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:41 smithi134 ceph-mon[31966]: pgmap v8140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:41 smithi189 ceph-mon[36790]: pgmap v8140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:41.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:41 smithi193 ceph-mon[37473]: pgmap v8140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 13999.785459 secs 2024-07-26T15:20:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 13999.785459 secs 2024-07-26T15:20:42.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 13999.785459 secs 2024-07-26T15:20:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:43 smithi189 ceph-mon[36790]: pgmap v8141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:43.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:43 smithi193 ceph-mon[37473]: pgmap v8141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:43.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:43 smithi134 ceph-mon[31966]: pgmap v8141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:44.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:20:44.077+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:20:44.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:20:44.077+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:20:44.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:20:44.077+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13985.361234 secs 2024-07-26T15:20:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13985.361234 secs 2024-07-26T15:20:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13985.361234 secs 2024-07-26T15:20:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:45 smithi193 ceph-mon[37473]: pgmap v8142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:45 smithi134 ceph-mon[31966]: pgmap v8142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:45 smithi189 ceph-mon[36790]: pgmap v8142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:20:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14004.785590 secs 2024-07-26T15:20:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:47 smithi193 ceph-mon[37473]: pgmap v8143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14004.785590 secs 2024-07-26T15:20:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:47 smithi134 ceph-mon[31966]: pgmap v8143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14004.785590 secs 2024-07-26T15:20:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:47 smithi189 ceph-mon[36790]: pgmap v8143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:49 smithi193 ceph-mon[37473]: pgmap v8144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:20:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:20:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13990.361309 secs 2024-07-26T15:20:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:49.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:49.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:49.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:49.805 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:49 smithi189 ceph-mon[36790]: pgmap v8144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:20:49.808 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:20:49.809 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13990.361309 secs 2024-07-26T15:20:49.809 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:49.810 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:49.811 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:49.811 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:49.822 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:49 smithi134 ceph-mon[31966]: pgmap v8144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:20:49.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:20:49.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13990.361309 secs 2024-07-26T15:20:49.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:49.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:49.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:49.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:51 smithi189 ceph-mon[36790]: pgmap v8145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:51 smithi193 ceph-mon[37473]: pgmap v8145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:51 smithi134 ceph-mon[31966]: pgmap v8145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:20:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14009.785686 secs 2024-07-26T15:20:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14009.785686 secs 2024-07-26T15:20:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14009.785686 secs 2024-07-26T15:20:52.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:53.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:53 smithi193 ceph-mon[37473]: pgmap v8146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:53.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:53 smithi134 ceph-mon[31966]: pgmap v8146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:53 smithi189 ceph-mon[36790]: pgmap v8146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:20:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:20:54.609+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:20:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:20:54.609+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:20:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:20:54.609+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:20:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:20:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:54 smithi193 ceph-mon[37473]: pgmap v8147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:20:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:54 smithi193 ceph-mon[37473]: pgmap v8148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:20:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 13995.361378 secs 2024-07-26T15:20:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:20:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:20:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:54 smithi134 ceph-mon[31966]: pgmap v8147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:20:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:54 smithi134 ceph-mon[31966]: pgmap v8148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:20:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 13995.361378 secs 2024-07-26T15:20:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:20:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:20:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:54 smithi189 ceph-mon[36790]: pgmap v8147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:20:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:54 smithi189 ceph-mon[36790]: pgmap v8148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:20:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:20:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 13995.361378 secs 2024-07-26T15:20:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:56 smithi193 ceph-mon[37473]: pgmap v8149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:20:56.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:56 smithi134 ceph-mon[31966]: pgmap v8149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:20:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:56 smithi189 ceph-mon[36790]: pgmap v8149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:20:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14014.785809 secs 2024-07-26T15:20:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14014.785809 secs 2024-07-26T15:20:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14014.785809 secs 2024-07-26T15:20:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:58 smithi134 ceph-mon[31966]: pgmap v8150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2024-07-26T15:20:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:58 smithi189 ceph-mon[36790]: pgmap v8150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2024-07-26T15:20:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:58 smithi193 ceph-mon[37473]: pgmap v8150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2024-07-26T15:20:59.083 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:20:59.081+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:20:59.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:20:59.081+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:20:59.084 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:20:59.081+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:20:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:20:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14000.361505 secs 2024-07-26T15:20:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:20:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14000.361505 secs 2024-07-26T15:21:00.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:20:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14000.361505 secs 2024-07-26T15:21:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:00 smithi134 ceph-mon[31966]: pgmap v8151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:21:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:00 smithi189 ceph-mon[36790]: pgmap v8151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:21:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:00 smithi193 ceph-mon[37473]: pgmap v8151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:21:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14019.785969 secs 2024-07-26T15:21:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:02 smithi134 ceph-mon[31966]: pgmap v8152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:21:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14019.785969 secs 2024-07-26T15:21:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:02 smithi189 ceph-mon[36790]: pgmap v8152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:21:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14019.785969 secs 2024-07-26T15:21:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:02 smithi193 ceph-mon[37473]: pgmap v8152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:21:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:04 smithi134 ceph-mon[31966]: pgmap v8153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:21:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14005.361651 secs 2024-07-26T15:21:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:04 smithi189 ceph-mon[36790]: pgmap v8153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:21:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14005.361651 secs 2024-07-26T15:21:05.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:04 smithi193 ceph-mon[37473]: pgmap v8153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:21:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14005.361651 secs 2024-07-26T15:21:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:06 smithi134 ceph-mon[31966]: pgmap v8154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:06 smithi189 ceph-mon[36790]: pgmap v8154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:07.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:06 smithi193 ceph-mon[37473]: pgmap v8154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14024.786092 secs 2024-07-26T15:21:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14024.786092 secs 2024-07-26T15:21:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14024.786092 secs 2024-07-26T15:21:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:08 smithi134 ceph-mon[31966]: pgmap v8155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:08 smithi189 ceph-mon[36790]: pgmap v8155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:08 smithi193 ceph-mon[37473]: pgmap v8155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:21:09.619+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:21:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:21:09.619+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:21:09.621 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:21:09.619+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:09.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14010.361703 secs 2024-07-26T15:21:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14010.361703 secs 2024-07-26T15:21:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14010.361703 secs 2024-07-26T15:21:10.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:10 smithi134 ceph-mon[31966]: pgmap v8156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:21:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:10 smithi189 ceph-mon[36790]: pgmap v8156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:21:11.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:10 smithi193 ceph-mon[37473]: pgmap v8156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:21:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14029.786226 secs 2024-07-26T15:21:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:12 smithi134 ceph-mon[31966]: pgmap v8157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14029.786226 secs 2024-07-26T15:21:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:12 smithi189 ceph-mon[36790]: pgmap v8157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:13.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14029.786226 secs 2024-07-26T15:21:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:12 smithi193 ceph-mon[37473]: pgmap v8157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:14.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:21:14.085+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:21:14.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:21:14.086+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:21:14.087 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:21:14.086+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:14 smithi134 ceph-mon[31966]: pgmap v8158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:21:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14015.361825 secs 2024-07-26T15:21:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:14 smithi189 ceph-mon[36790]: pgmap v8158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:21:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14015.361825 secs 2024-07-26T15:21:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:14 smithi193 ceph-mon[37473]: pgmap v8158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:21:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14015.361825 secs 2024-07-26T15:21:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:16 smithi134 ceph-mon[31966]: pgmap v8159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:16 smithi189 ceph-mon[36790]: pgmap v8159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:16 smithi193 ceph-mon[37473]: pgmap v8159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14034.786357 secs 2024-07-26T15:21:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14034.786357 secs 2024-07-26T15:21:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14034.786357 secs 2024-07-26T15:21:18.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:18 smithi134 ceph-mon[31966]: pgmap v8160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:18 smithi189 ceph-mon[36790]: pgmap v8160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:19.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:18 smithi193 ceph-mon[37473]: pgmap v8160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14020.361967 secs 2024-07-26T15:21:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14020.361967 secs 2024-07-26T15:21:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14020.361967 secs 2024-07-26T15:21:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:20 smithi134 ceph-mon[31966]: pgmap v8161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:21:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:20 smithi189 ceph-mon[36790]: pgmap v8161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:21:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:20 smithi193 ceph-mon[37473]: pgmap v8161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:21:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14039.786490 secs 2024-07-26T15:21:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:22 smithi134 ceph-mon[31966]: pgmap v8162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14039.786490 secs 2024-07-26T15:21:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:22 smithi189 ceph-mon[36790]: pgmap v8162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14039.786490 secs 2024-07-26T15:21:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:22 smithi193 ceph-mon[37473]: pgmap v8162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:21:24.620+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:21:24.621 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:21:24.620+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:21:24.622 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:21:24.620+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:24.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:21:24.769+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:21:24.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:21:24.769+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:21:24.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:21:24.770+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 110 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T15:21:24.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:21:24.770+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T15:21:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:24 smithi134 ceph-mon[31966]: pgmap v8163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:21:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14025.362041 secs 2024-07-26T15:21:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:24 smithi189 ceph-mon[36790]: pgmap v8163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:21:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14025.362041 secs 2024-07-26T15:21:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:24 smithi193 ceph-mon[37473]: pgmap v8163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:21:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14025.362041 secs 2024-07-26T15:21:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:26 smithi189 ceph-mon[36790]: pgmap v8164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:26 smithi193 ceph-mon[37473]: pgmap v8164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:26 smithi134 ceph-mon[31966]: pgmap v8164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14044.786635 secs 2024-07-26T15:21:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14044.786635 secs 2024-07-26T15:21:28.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14044.786635 secs 2024-07-26T15:21:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:28 smithi189 ceph-mon[36790]: pgmap v8165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:29.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:28 smithi193 ceph-mon[37473]: pgmap v8165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:29.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:21:29.101+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:21:29.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:21:29.101+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:21:29.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:21:29.101+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:29.103 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:28 smithi134 ceph-mon[31966]: pgmap v8165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14030.362177 secs 2024-07-26T15:21:30.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14030.362177 secs 2024-07-26T15:21:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14030.362177 secs 2024-07-26T15:21:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:30 smithi193 ceph-mon[37473]: pgmap v8166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:21:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:30 smithi134 ceph-mon[31966]: pgmap v8166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:21:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:30 smithi189 ceph-mon[36790]: pgmap v8166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:21:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14049.786782 secs 2024-07-26T15:21:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:32 smithi193 ceph-mon[37473]: pgmap v8167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:33.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14049.786782 secs 2024-07-26T15:21:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:32 smithi134 ceph-mon[31966]: pgmap v8167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14049.786782 secs 2024-07-26T15:21:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:32 smithi189 ceph-mon[36790]: pgmap v8167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:34 smithi193 ceph-mon[37473]: pgmap v8168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:21:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14035.362308 secs 2024-07-26T15:21:35.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:34 smithi134 ceph-mon[31966]: pgmap v8168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:21:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14035.362308 secs 2024-07-26T15:21:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:34 smithi189 ceph-mon[36790]: pgmap v8168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:21:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14035.362308 secs 2024-07-26T15:21:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:21:35.308+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:21:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:21:35.308+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:21:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:21:35.308+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:21:35.309 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:21:35.308+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:21:37.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:36 smithi193 ceph-mon[37473]: pgmap v8169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:36 smithi134 ceph-mon[31966]: pgmap v8169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:36 smithi189 ceph-mon[36790]: pgmap v8169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:38.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14054.786927 secs 2024-07-26T15:21:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14054.786927 secs 2024-07-26T15:21:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14054.786927 secs 2024-07-26T15:21:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:38 smithi193 ceph-mon[37473]: pgmap v8170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:39.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:38 smithi134 ceph-mon[31966]: pgmap v8170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:38 smithi189 ceph-mon[36790]: pgmap v8170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:39.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:21:39.636+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:21:39.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:21:39.636+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:21:39.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:21:39.636+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14040.362426 secs 2024-07-26T15:21:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14040.362426 secs 2024-07-26T15:21:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14040.362426 secs 2024-07-26T15:21:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:40 smithi193 ceph-mon[37473]: pgmap v8171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:21:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:40 smithi134 ceph-mon[31966]: pgmap v8171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:21:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:40 smithi189 ceph-mon[36790]: pgmap v8171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:21:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14059.787047 secs 2024-07-26T15:21:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:42 smithi193 ceph-mon[37473]: pgmap v8172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14059.787047 secs 2024-07-26T15:21:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:42 smithi134 ceph-mon[31966]: pgmap v8172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14059.787047 secs 2024-07-26T15:21:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:42 smithi189 ceph-mon[36790]: pgmap v8172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:44.118 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:21:44.117+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:21:44.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:21:44.117+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:21:44.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:21:44.117+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:44 smithi193 ceph-mon[37473]: pgmap v8173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:21:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14045.362556 secs 2024-07-26T15:21:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:44 smithi134 ceph-mon[31966]: pgmap v8173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:21:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14045.362556 secs 2024-07-26T15:21:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:44 smithi189 ceph-mon[36790]: pgmap v8173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:21:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14045.362556 secs 2024-07-26T15:21:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:46 smithi193 ceph-mon[37473]: pgmap v8174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:46 smithi134 ceph-mon[31966]: pgmap v8174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:46 smithi189 ceph-mon[36790]: pgmap v8174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:48.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14064.787171 secs 2024-07-26T15:21:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14064.787171 secs 2024-07-26T15:21:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14064.787171 secs 2024-07-26T15:21:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:48 smithi134 ceph-mon[31966]: pgmap v8175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:48 smithi189 ceph-mon[36790]: pgmap v8175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:49.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:48 smithi193 ceph-mon[37473]: pgmap v8175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:21:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14050.362683 secs 2024-07-26T15:21:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14050.362683 secs 2024-07-26T15:21:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14050.362683 secs 2024-07-26T15:21:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:50 smithi134 ceph-mon[31966]: pgmap v8176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:21:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:50 smithi189 ceph-mon[36790]: pgmap v8176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:21:51.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:50 smithi193 ceph-mon[37473]: pgmap v8176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T15:21:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14069.787296 secs 2024-07-26T15:21:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:52 smithi134 ceph-mon[31966]: pgmap v8177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14069.787296 secs 2024-07-26T15:21:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:52 smithi189 ceph-mon[36790]: pgmap v8177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14069.787296 secs 2024-07-26T15:21:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:52 smithi193 ceph-mon[37473]: pgmap v8177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:21:54.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:21:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:21:54.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:21:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:21:54.651+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:21:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:21:54.651+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:21:54.652 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:21:54.651+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:21:55.081 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:54 smithi134 ceph-mon[31966]: pgmap v8178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:21:55.081 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14055.362812 secs 2024-07-26T15:21:55.081 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:21:55.081 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:21:55.081 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:21:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:54 smithi189 ceph-mon[36790]: pgmap v8178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:21:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14055.362812 secs 2024-07-26T15:21:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:21:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:21:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:21:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:54 smithi193 ceph-mon[37473]: pgmap v8178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:21:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14055.362812 secs 2024-07-26T15:21:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:21:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:21:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:21:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:55 smithi134 ceph-mon[31966]: pgmap v8179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:21:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:55 smithi134 ceph-mon[31966]: pgmap v8180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:21:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:55 smithi189 ceph-mon[36790]: pgmap v8179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:21:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:55 smithi189 ceph-mon[36790]: pgmap v8180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:21:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:55 smithi193 ceph-mon[37473]: pgmap v8179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:21:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:55 smithi193 ceph-mon[37473]: pgmap v8180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:21:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14074.787354 secs 2024-07-26T15:21:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:57 smithi134 ceph-mon[31966]: pgmap v8181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:21:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14074.787354 secs 2024-07-26T15:21:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:57 smithi189 ceph-mon[36790]: pgmap v8181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:21:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14074.787354 secs 2024-07-26T15:21:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:57 smithi193 ceph-mon[37473]: pgmap v8181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:21:59.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:21:59.132+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:21:59.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:21:59.132+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:21:59.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:21:59.132+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14060.362929 secs 2024-07-26T15:22:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:21:59 smithi134 ceph-mon[31966]: pgmap v8182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14060.362929 secs 2024-07-26T15:22:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:21:59 smithi189 ceph-mon[36790]: pgmap v8182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14060.362929 secs 2024-07-26T15:22:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:21:59 smithi193 ceph-mon[37473]: pgmap v8182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:02.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:01 smithi134 ceph-mon[31966]: pgmap v8183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:01 smithi189 ceph-mon[36790]: pgmap v8183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:01 smithi193 ceph-mon[37473]: pgmap v8183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14079.787478 secs 2024-07-26T15:22:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:02 smithi134 ceph-mon[31966]: pgmap v8184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:22:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14079.787478 secs 2024-07-26T15:22:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:02 smithi189 ceph-mon[36790]: pgmap v8184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:22:03.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14079.787478 secs 2024-07-26T15:22:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:02 smithi193 ceph-mon[37473]: pgmap v8184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:22:04.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14065.363062 secs 2024-07-26T15:22:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14065.363062 secs 2024-07-26T15:22:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14065.363062 secs 2024-07-26T15:22:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:05 smithi193 ceph-mon[37473]: pgmap v8185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:22:05.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:05 smithi134 ceph-mon[31966]: pgmap v8185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:22:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:05 smithi189 ceph-mon[36790]: pgmap v8185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:22:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14084.787539 secs 2024-07-26T15:22:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:07 smithi193 ceph-mon[37473]: pgmap v8186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14084.787539 secs 2024-07-26T15:22:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:07 smithi134 ceph-mon[31966]: pgmap v8186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14084.787539 secs 2024-07-26T15:22:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:07 smithi189 ceph-mon[36790]: pgmap v8186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:22:09.666+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:22:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:22:09.666+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:09.668 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:22:09.666+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:22:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14070.363165 secs 2024-07-26T15:22:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:09 smithi193 ceph-mon[37473]: pgmap v8187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14070.363165 secs 2024-07-26T15:22:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:09 smithi134 ceph-mon[31966]: pgmap v8187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14070.363165 secs 2024-07-26T15:22:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:09 smithi189 ceph-mon[36790]: pgmap v8187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:11 smithi193 ceph-mon[37473]: pgmap v8188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:11 smithi134 ceph-mon[31966]: pgmap v8188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:11 smithi189 ceph-mon[36790]: pgmap v8188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:13.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14089.787674 secs 2024-07-26T15:22:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14089.787674 secs 2024-07-26T15:22:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14089.787674 secs 2024-07-26T15:22:14.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:22:14.139+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:22:14.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:22:14.139+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:14.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:22:14.139+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:22:14.142 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:13 smithi134 ceph-mon[31966]: pgmap v8189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:14.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:13 smithi193 ceph-mon[37473]: pgmap v8189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:13 smithi189 ceph-mon[36790]: pgmap v8189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14075.363290 secs 2024-07-26T15:22:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14075.363290 secs 2024-07-26T15:22:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14075.363290 secs 2024-07-26T15:22:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:15 smithi134 ceph-mon[31966]: pgmap v8190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:22:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:15 smithi189 ceph-mon[36790]: pgmap v8190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:22:16.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:15 smithi193 ceph-mon[37473]: pgmap v8190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:22:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14094.787769 secs 2024-07-26T15:22:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:17 smithi134 ceph-mon[31966]: pgmap v8191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14094.787769 secs 2024-07-26T15:22:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:17 smithi189 ceph-mon[36790]: pgmap v8191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:18.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14094.787769 secs 2024-07-26T15:22:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:17 smithi193 ceph-mon[37473]: pgmap v8191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14080.363414 secs 2024-07-26T15:22:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:19 smithi134 ceph-mon[31966]: pgmap v8192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14080.363414 secs 2024-07-26T15:22:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:19 smithi189 ceph-mon[36790]: pgmap v8192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14080.363414 secs 2024-07-26T15:22:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:19 smithi193 ceph-mon[37473]: pgmap v8192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:21 smithi134 ceph-mon[31966]: pgmap v8193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:21 smithi189 ceph-mon[36790]: pgmap v8193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:21 smithi193 ceph-mon[37473]: pgmap v8193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:23.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14099.787900 secs 2024-07-26T15:22:23.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14099.787900 secs 2024-07-26T15:22:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14099.787900 secs 2024-07-26T15:22:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:23 smithi134 ceph-mon[31966]: pgmap v8194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:23 smithi189 ceph-mon[36790]: pgmap v8194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:24.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:23 smithi193 ceph-mon[37473]: pgmap v8194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:24.671 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:22:24.670+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:22:24.671 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:22:24.670+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:24.671 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:22:24.670+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:22:25.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14085.363498 secs 2024-07-26T15:22:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14085.363498 secs 2024-07-26T15:22:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14085.363498 secs 2024-07-26T15:22:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:25 smithi134 ceph-mon[31966]: pgmap v8195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:22:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:25 smithi189 ceph-mon[36790]: pgmap v8195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:22:26.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:25 smithi193 ceph-mon[37473]: pgmap v8195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:22:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:26 smithi134 ceph-mon[31966]: pgmap v8196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:26 smithi189 ceph-mon[36790]: pgmap v8196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:27.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:26 smithi193 ceph-mon[37473]: pgmap v8196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14104.788048 secs 2024-07-26T15:22:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14104.788048 secs 2024-07-26T15:22:28.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14104.788048 secs 2024-07-26T15:22:29.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:22:29.152+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:22:29.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:22:29.152+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:29.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:22:29.152+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:22:29.154 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:28 smithi134 ceph-mon[31966]: pgmap v8197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:28 smithi189 ceph-mon[36790]: pgmap v8197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:29.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:28 smithi193 ceph-mon[37473]: pgmap v8197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14090.363617 secs 2024-07-26T15:22:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14090.363617 secs 2024-07-26T15:22:30.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14090.363617 secs 2024-07-26T15:22:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:30 smithi134 ceph-mon[31966]: pgmap v8198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:30 smithi189 ceph-mon[36790]: pgmap v8198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:31.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:30 smithi193 ceph-mon[37473]: pgmap v8198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:32.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14109.788176 secs 2024-07-26T15:22:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14109.788176 secs 2024-07-26T15:22:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14109.788176 secs 2024-07-26T15:22:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:33 smithi134 ceph-mon[31966]: pgmap v8199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:22:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:33 smithi189 ceph-mon[36790]: pgmap v8199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:22:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:33 smithi193 ceph-mon[37473]: pgmap v8199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:22:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14095.363739 secs 2024-07-26T15:22:34.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14095.363739 secs 2024-07-26T15:22:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14095.363739 secs 2024-07-26T15:22:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:35 smithi193 ceph-mon[37473]: pgmap v8200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:22:35.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:35 smithi134 ceph-mon[31966]: pgmap v8200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:22:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:35 smithi189 ceph-mon[36790]: pgmap v8200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:22:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14114.788325 secs 2024-07-26T15:22:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:37 smithi193 ceph-mon[37473]: pgmap v8201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:22:38.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14114.788325 secs 2024-07-26T15:22:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:37 smithi134 ceph-mon[31966]: pgmap v8201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:22:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14114.788325 secs 2024-07-26T15:22:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:37 smithi189 ceph-mon[36790]: pgmap v8201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:22:39.686 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:22:39.685+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:22:39.687 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:22:39.685+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:39.687 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:22:39.685+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:22:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14100.363862 secs 2024-07-26T15:22:40.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:39 smithi134 ceph-mon[31966]: pgmap v8202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:22:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14100.363862 secs 2024-07-26T15:22:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:39 smithi189 ceph-mon[36790]: pgmap v8202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:22:40.278 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14100.363862 secs 2024-07-26T15:22:40.278 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:39 smithi193 ceph-mon[37473]: pgmap v8202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:22:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:41 smithi134 ceph-mon[31966]: pgmap v8203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:22:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:41 smithi189 ceph-mon[36790]: pgmap v8203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:22:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:41 smithi193 ceph-mon[37473]: pgmap v8203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:22:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14119.788429 secs 2024-07-26T15:22:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14119.788429 secs 2024-07-26T15:22:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14119.788429 secs 2024-07-26T15:22:44.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:22:44.167+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:22:44.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:22:44.167+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:44.169 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:22:44.167+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:22:44.170 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:43 smithi134 ceph-mon[31966]: pgmap v8204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:22:44.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:43 smithi193 ceph-mon[37473]: pgmap v8204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:22:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:43 smithi189 ceph-mon[36790]: pgmap v8204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:22:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14105.364017 secs 2024-07-26T15:22:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14105.364017 secs 2024-07-26T15:22:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14105.364017 secs 2024-07-26T15:22:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:45 smithi134 ceph-mon[31966]: pgmap v8205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:22:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:45 smithi189 ceph-mon[36790]: pgmap v8205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:22:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:45 smithi193 ceph-mon[37473]: pgmap v8205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:22:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14124.788550 secs 2024-07-26T15:22:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:47 smithi134 ceph-mon[31966]: pgmap v8206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14124.788550 secs 2024-07-26T15:22:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:47 smithi189 ceph-mon[36790]: pgmap v8206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:48.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14124.788550 secs 2024-07-26T15:22:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:47 smithi193 ceph-mon[37473]: pgmap v8206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14110.364133 secs 2024-07-26T15:22:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:49 smithi134 ceph-mon[31966]: pgmap v8207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14110.364133 secs 2024-07-26T15:22:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:49 smithi189 ceph-mon[36790]: pgmap v8207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14110.364133 secs 2024-07-26T15:22:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:49 smithi193 ceph-mon[37473]: pgmap v8207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:51 smithi134 ceph-mon[31966]: pgmap v8208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:51 smithi189 ceph-mon[36790]: pgmap v8208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:51 smithi193 ceph-mon[37473]: pgmap v8208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:22:53.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14129.788636 secs 2024-07-26T15:22:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14129.788636 secs 2024-07-26T15:22:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14129.788636 secs 2024-07-26T15:22:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:53 smithi134 ceph-mon[31966]: pgmap v8209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:53 smithi189 ceph-mon[36790]: pgmap v8209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:53 smithi193 ceph-mon[37473]: pgmap v8209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:22:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:22:54.701+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:22:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:22:54.701+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:22:54.702 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:22:54.701+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:55.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14115.364258 secs 2024-07-26T15:22:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14115.364258 secs 2024-07-26T15:22:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14115.364258 secs 2024-07-26T15:22:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:55 smithi134 ceph-mon[31966]: pgmap v8210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:22:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:22:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:55 smithi189 ceph-mon[36790]: pgmap v8210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:22:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:22:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:55 smithi193 ceph-mon[37473]: pgmap v8210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:22:56.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:22:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:22:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:22:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:56 smithi134 ceph-mon[31966]: pgmap v8211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:22:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:56 smithi134 ceph-mon[31966]: pgmap v8212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:22:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:22:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:22:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:56 smithi189 ceph-mon[36790]: pgmap v8211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:22:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:56 smithi189 ceph-mon[36790]: pgmap v8212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:22:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:22:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:22:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:56 smithi193 ceph-mon[37473]: pgmap v8211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:22:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:56 smithi193 ceph-mon[37473]: pgmap v8212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:22:58.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14134.788703 secs 2024-07-26T15:22:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14134.788703 secs 2024-07-26T15:22:58.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14134.788703 secs 2024-07-26T15:22:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:58 smithi134 ceph-mon[31966]: pgmap v8213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:59.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:22:59.182+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:22:59.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:22:59.182+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:22:59.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:22:59.182+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:22:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:58 smithi189 ceph-mon[36790]: pgmap v8213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:22:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:58 smithi193 ceph-mon[37473]: pgmap v8213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:23:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:22:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14120.364378 secs 2024-07-26T15:23:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:22:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14120.364378 secs 2024-07-26T15:23:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:22:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14120.364378 secs 2024-07-26T15:23:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:00 smithi134 ceph-mon[31966]: pgmap v8214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:23:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:00 smithi189 ceph-mon[36790]: pgmap v8214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:23:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:00 smithi193 ceph-mon[37473]: pgmap v8214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:23:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14139.788830 secs 2024-07-26T15:23:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:02 smithi193 ceph-mon[37473]: pgmap v8215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:23:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14139.788830 secs 2024-07-26T15:23:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:02 smithi134 ceph-mon[31966]: pgmap v8215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:23:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14139.788830 secs 2024-07-26T15:23:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:02 smithi189 ceph-mon[36790]: pgmap v8215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:23:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14125.364444 secs 2024-07-26T15:23:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:04 smithi193 ceph-mon[37473]: pgmap v8216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:23:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14125.364444 secs 2024-07-26T15:23:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:04 smithi134 ceph-mon[31966]: pgmap v8216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:23:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14125.364444 secs 2024-07-26T15:23:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:04 smithi189 ceph-mon[36790]: pgmap v8216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:23:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:07 smithi134 ceph-mon[31966]: pgmap v8217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:23:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14144.788895 secs 2024-07-26T15:23:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:07 smithi189 ceph-mon[36790]: pgmap v8217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:23:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14144.788895 secs 2024-07-26T15:23:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:07 smithi193 ceph-mon[37473]: pgmap v8217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:23:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14144.788895 secs 2024-07-26T15:23:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:23:09.717+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:23:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:23:09.717+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:09.718 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:23:09.717+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:23:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:09 smithi134 ceph-mon[31966]: pgmap v8218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:23:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14130.364542 secs 2024-07-26T15:23:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:09 smithi189 ceph-mon[36790]: pgmap v8218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:23:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14130.364542 secs 2024-07-26T15:23:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:09 smithi193 ceph-mon[37473]: pgmap v8218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:23:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14130.364542 secs 2024-07-26T15:23:11.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:10 smithi134 ceph-mon[31966]: pgmap v8219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:10 smithi189 ceph-mon[36790]: pgmap v8219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:10 smithi193 ceph-mon[37473]: pgmap v8219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14149.789048 secs 2024-07-26T15:23:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:12 smithi193 ceph-mon[37473]: pgmap v8220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14149.789048 secs 2024-07-26T15:23:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:12 smithi134 ceph-mon[31966]: pgmap v8220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14149.789048 secs 2024-07-26T15:23:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:12 smithi189 ceph-mon[36790]: pgmap v8220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:14.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:23:14.187+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:23:14.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:23:14.187+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:23:14.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:23:14.187+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14135.364661 secs 2024-07-26T15:23:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:14 smithi193 ceph-mon[37473]: pgmap v8221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:15.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14135.364661 secs 2024-07-26T15:23:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:14 smithi134 ceph-mon[31966]: pgmap v8221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14135.364661 secs 2024-07-26T15:23:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:14 smithi189 ceph-mon[36790]: pgmap v8221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:16 smithi193 ceph-mon[37473]: pgmap v8222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14154.789163 secs 2024-07-26T15:23:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:16 smithi134 ceph-mon[31966]: pgmap v8222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14154.789163 secs 2024-07-26T15:23:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:16 smithi189 ceph-mon[36790]: pgmap v8222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14154.789163 secs 2024-07-26T15:23:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:18 smithi193 ceph-mon[37473]: pgmap v8223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:23:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:18 smithi134 ceph-mon[31966]: pgmap v8223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:23:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:18 smithi189 ceph-mon[36790]: pgmap v8223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:23:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14140.364781 secs 2024-07-26T15:23:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14140.364781 secs 2024-07-26T15:23:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14140.364781 secs 2024-07-26T15:23:21.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:21 smithi193 ceph-mon[37473]: pgmap v8224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:21 smithi134 ceph-mon[31966]: pgmap v8224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:21 smithi189 ceph-mon[36790]: pgmap v8224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14159.789284 secs 2024-07-26T15:23:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14159.789284 secs 2024-07-26T15:23:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14159.789284 secs 2024-07-26T15:23:23.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:23 smithi189 ceph-mon[36790]: pgmap v8225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:23.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:23 smithi134 ceph-mon[31966]: pgmap v8225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:23 smithi193 ceph-mon[37473]: pgmap v8225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:23:24.731+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:23:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:23:24.731+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:24.732 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:23:24.731+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:23:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14145.364895 secs 2024-07-26T15:23:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:25 smithi134 ceph-mon[31966]: pgmap v8226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14145.364895 secs 2024-07-26T15:23:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:25 smithi189 ceph-mon[36790]: pgmap v8226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14145.364895 secs 2024-07-26T15:23:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:25 smithi193 ceph-mon[37473]: pgmap v8226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:27 smithi134 ceph-mon[31966]: pgmap v8227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14164.789406 secs 2024-07-26T15:23:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:27 smithi189 ceph-mon[36790]: pgmap v8227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14164.789406 secs 2024-07-26T15:23:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:27 smithi193 ceph-mon[37473]: pgmap v8227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14164.789406 secs 2024-07-26T15:23:29.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:23:29.189+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:23:29.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:23:29.189+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:23:29.191 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:23:29.189+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:29 smithi134 ceph-mon[31966]: pgmap v8228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:23:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:29 smithi189 ceph-mon[36790]: pgmap v8228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:23:29.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:29 smithi193 ceph-mon[37473]: pgmap v8228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:23:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14150.365012 secs 2024-07-26T15:23:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14150.365012 secs 2024-07-26T15:23:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14150.365012 secs 2024-07-26T15:23:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:31 smithi134 ceph-mon[31966]: pgmap v8229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:31 smithi189 ceph-mon[36790]: pgmap v8229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:31.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:31 smithi193 ceph-mon[37473]: pgmap v8229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14169.789507 secs 2024-07-26T15:23:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14169.789507 secs 2024-07-26T15:23:32.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14169.789507 secs 2024-07-26T15:23:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:33 smithi134 ceph-mon[31966]: pgmap v8230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:33 smithi189 ceph-mon[36790]: pgmap v8230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:33.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:33 smithi193 ceph-mon[37473]: pgmap v8230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14155.365137 secs 2024-07-26T15:23:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:35 smithi134 ceph-mon[31966]: pgmap v8231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14155.365137 secs 2024-07-26T15:23:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:35 smithi189 ceph-mon[36790]: pgmap v8231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14155.365137 secs 2024-07-26T15:23:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:35 smithi193 ceph-mon[37473]: pgmap v8231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:37 smithi134 ceph-mon[31966]: pgmap v8232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14174.789627 secs 2024-07-26T15:23:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:37 smithi189 ceph-mon[36790]: pgmap v8232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14174.789627 secs 2024-07-26T15:23:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:37 smithi193 ceph-mon[37473]: pgmap v8232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14174.789627 secs 2024-07-26T15:23:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:39 smithi134 ceph-mon[31966]: pgmap v8233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:23:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:39 smithi189 ceph-mon[36790]: pgmap v8233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:23:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:39 smithi193 ceph-mon[37473]: pgmap v8233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:23:39.737 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:23:39.736+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:23:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:23:39.736+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:39.738 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:23:39.736+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:23:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14160.365219 secs 2024-07-26T15:23:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14160.365219 secs 2024-07-26T15:23:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14160.365219 secs 2024-07-26T15:23:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:41 smithi134 ceph-mon[31966]: pgmap v8234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:41 smithi189 ceph-mon[36790]: pgmap v8234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:41 smithi193 ceph-mon[37473]: pgmap v8234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:42.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14179.789750 secs 2024-07-26T15:23:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14179.789750 secs 2024-07-26T15:23:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14179.789750 secs 2024-07-26T15:23:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:43 smithi134 ceph-mon[31966]: pgmap v8235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:43.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:43 smithi189 ceph-mon[36790]: pgmap v8235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:43 smithi193 ceph-mon[37473]: pgmap v8235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:44.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:23:44.203+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:23:44.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:23:44.203+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:23:44.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:23:44.203+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14165.365302 secs 2024-07-26T15:23:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:45 smithi134 ceph-mon[31966]: pgmap v8236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14165.365302 secs 2024-07-26T15:23:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:45 smithi189 ceph-mon[36790]: pgmap v8236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:45.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14165.365302 secs 2024-07-26T15:23:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:45 smithi193 ceph-mon[37473]: pgmap v8236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:47 smithi134 ceph-mon[31966]: pgmap v8237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14184.789829 secs 2024-07-26T15:23:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:47 smithi189 ceph-mon[36790]: pgmap v8237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14184.789829 secs 2024-07-26T15:23:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:47 smithi193 ceph-mon[37473]: pgmap v8237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14184.789829 secs 2024-07-26T15:23:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:49 smithi134 ceph-mon[31966]: pgmap v8238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:23:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:49 smithi189 ceph-mon[36790]: pgmap v8238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:23:49.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:49 smithi193 ceph-mon[37473]: pgmap v8238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:23:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14170.365440 secs 2024-07-26T15:23:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14170.365440 secs 2024-07-26T15:23:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14170.365440 secs 2024-07-26T15:23:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:51 smithi134 ceph-mon[31966]: pgmap v8239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:51 smithi189 ceph-mon[36790]: pgmap v8239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:51.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:51 smithi193 ceph-mon[37473]: pgmap v8239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:23:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14189.789946 secs 2024-07-26T15:23:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14189.789946 secs 2024-07-26T15:23:52.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14189.789946 secs 2024-07-26T15:23:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:53 smithi134 ceph-mon[31966]: pgmap v8240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:53 smithi189 ceph-mon[36790]: pgmap v8240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:53.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:53 smithi193 ceph-mon[37473]: pgmap v8240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:23:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:23:54.752+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:23:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:23:54.752+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:23:54.753 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:23:54.752+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14175.365527 secs 2024-07-26T15:23:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:55 smithi189 ceph-mon[36790]: pgmap v8241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14175.365527 secs 2024-07-26T15:23:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:55 smithi193 ceph-mon[37473]: pgmap v8241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:55.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14175.365527 secs 2024-07-26T15:23:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:55 smithi134 ceph-mon[31966]: pgmap v8241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:23:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:23:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:23:56.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:23:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:57 smithi193 ceph-mon[37473]: pgmap v8242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14194.790025 secs 2024-07-26T15:23:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:23:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:23:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:23:57.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:57 smithi134 ceph-mon[31966]: pgmap v8242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:57.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14194.790025 secs 2024-07-26T15:23:57.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:23:57.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:23:57.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:23:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:57 smithi189 ceph-mon[36790]: pgmap v8242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:23:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14194.790025 secs 2024-07-26T15:23:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:23:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:23:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:23:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:58 smithi193 ceph-mon[37473]: pgmap v8243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:23:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:23:58 smithi193 ceph-mon[37473]: pgmap v8244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:23:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:58 smithi134 ceph-mon[31966]: pgmap v8243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:23:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:23:58 smithi134 ceph-mon[31966]: pgmap v8244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:23:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:58 smithi189 ceph-mon[36790]: pgmap v8243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:23:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:23:58 smithi189 ceph-mon[36790]: pgmap v8244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:23:59.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:23:59.211+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:23:59.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:23:59.211+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:23:59.213 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:23:59.211+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14180.365643 secs 2024-07-26T15:24:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:00 smithi193 ceph-mon[37473]: pgmap v8245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:24:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14180.365643 secs 2024-07-26T15:24:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:00 smithi134 ceph-mon[31966]: pgmap v8245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:24:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14180.365643 secs 2024-07-26T15:24:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:00 smithi189 ceph-mon[36790]: pgmap v8245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:24:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14199.790151 secs 2024-07-26T15:24:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:02 smithi193 ceph-mon[37473]: pgmap v8246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:24:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14199.790151 secs 2024-07-26T15:24:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:02 smithi134 ceph-mon[31966]: pgmap v8246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:24:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14199.790151 secs 2024-07-26T15:24:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:02 smithi189 ceph-mon[36790]: pgmap v8246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:24:04.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:04 smithi193 ceph-mon[37473]: pgmap v8247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:24:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:04 smithi134 ceph-mon[31966]: pgmap v8247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:24:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:04 smithi189 ceph-mon[36790]: pgmap v8247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:24:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14185.365787 secs 2024-07-26T15:24:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14185.365787 secs 2024-07-26T15:24:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14185.365787 secs 2024-07-26T15:24:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:06 smithi193 ceph-mon[37473]: pgmap v8248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:24:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:06 smithi134 ceph-mon[31966]: pgmap v8248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:24:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:06 smithi189 ceph-mon[36790]: pgmap v8248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:24:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14204.790289 secs 2024-07-26T15:24:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14204.790289 secs 2024-07-26T15:24:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14204.790289 secs 2024-07-26T15:24:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:08 smithi193 ceph-mon[37473]: pgmap v8249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:24:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:08 smithi134 ceph-mon[31966]: pgmap v8249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:24:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:08 smithi189 ceph-mon[36790]: pgmap v8249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:24:09.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:24:09.752+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:09.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:24:09.752+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:09.754 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:24:09.752+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14190.365917 secs 2024-07-26T15:24:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:10 smithi193 ceph-mon[37473]: pgmap v8250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14190.365917 secs 2024-07-26T15:24:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:10 smithi134 ceph-mon[31966]: pgmap v8250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14190.365917 secs 2024-07-26T15:24:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:10 smithi189 ceph-mon[36790]: pgmap v8250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14209.790347 secs 2024-07-26T15:24:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:12 smithi193 ceph-mon[37473]: pgmap v8251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14209.790347 secs 2024-07-26T15:24:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:12 smithi134 ceph-mon[31966]: pgmap v8251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14209.790347 secs 2024-07-26T15:24:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:12 smithi189 ceph-mon[36790]: pgmap v8251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:14.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:24:14.218+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:14.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:24:14.218+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:14.220 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:24:14.218+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:14.475 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:14 smithi134 ceph-mon[31966]: pgmap v8252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:24:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:14 smithi193 ceph-mon[37473]: pgmap v8252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:24:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:14 smithi189 ceph-mon[36790]: pgmap v8252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:24:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14195.366034 secs 2024-07-26T15:24:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14195.366034 secs 2024-07-26T15:24:15.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14195.366034 secs 2024-07-26T15:24:16.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:16 smithi134 ceph-mon[31966]: pgmap v8253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:16 smithi189 ceph-mon[36790]: pgmap v8253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:16 smithi193 ceph-mon[37473]: pgmap v8253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:17.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14214.790441 secs 2024-07-26T15:24:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14214.790441 secs 2024-07-26T15:24:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14214.790441 secs 2024-07-26T15:24:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:18 smithi134 ceph-mon[31966]: pgmap v8254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:18 smithi189 ceph-mon[36790]: pgmap v8254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:18.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:18 smithi193 ceph-mon[37473]: pgmap v8254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14200.366164 secs 2024-07-26T15:24:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:20 smithi134 ceph-mon[31966]: pgmap v8255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14200.366164 secs 2024-07-26T15:24:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:20 smithi189 ceph-mon[36790]: pgmap v8255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14200.366164 secs 2024-07-26T15:24:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:20 smithi193 ceph-mon[37473]: pgmap v8255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14219.790508 secs 2024-07-26T15:24:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:22 smithi134 ceph-mon[31966]: pgmap v8256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14219.790508 secs 2024-07-26T15:24:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:22 smithi189 ceph-mon[36790]: pgmap v8256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:22.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14219.790508 secs 2024-07-26T15:24:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:22 smithi193 ceph-mon[37473]: pgmap v8256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:24 smithi134 ceph-mon[31966]: pgmap v8257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:24:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:24 smithi189 ceph-mon[36790]: pgmap v8257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:24:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:24:24.768+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:24.769 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:24:24.768+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:24.770 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:24:24.768+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:24.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:24 smithi193 ceph-mon[37473]: pgmap v8257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:24:24.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:24:24.801+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:24:24.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:24:24.801+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:24:24.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:24:24.801+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T15:24:24.802 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:24:24.801+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 111 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T15:24:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14205.366275 secs 2024-07-26T15:24:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14205.366275 secs 2024-07-26T15:24:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14205.366275 secs 2024-07-26T15:24:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:26 smithi134 ceph-mon[31966]: pgmap v8258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:26 smithi189 ceph-mon[36790]: pgmap v8258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:26.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:26 smithi193 ceph-mon[37473]: pgmap v8258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14224.790601 secs 2024-07-26T15:24:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14224.790601 secs 2024-07-26T15:24:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14224.790601 secs 2024-07-26T15:24:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:28 smithi134 ceph-mon[31966]: pgmap v8259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:28 smithi189 ceph-mon[36790]: pgmap v8259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:28.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:28 smithi193 ceph-mon[37473]: pgmap v8259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:29.235 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:24:29.234+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:29.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:24:29.234+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:29.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:24:29.234+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14210.366409 secs 2024-07-26T15:24:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:30 smithi134 ceph-mon[31966]: pgmap v8260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14210.366409 secs 2024-07-26T15:24:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:30 smithi189 ceph-mon[36790]: pgmap v8260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:30.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14210.366409 secs 2024-07-26T15:24:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:30 smithi193 ceph-mon[37473]: pgmap v8260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14229.790724 secs 2024-07-26T15:24:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:32 smithi134 ceph-mon[31966]: pgmap v8261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14229.790724 secs 2024-07-26T15:24:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:32 smithi189 ceph-mon[36790]: pgmap v8261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14229.790724 secs 2024-07-26T15:24:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:32 smithi193 ceph-mon[37473]: pgmap v8261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:34 smithi134 ceph-mon[31966]: pgmap v8262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:24:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:34 smithi189 ceph-mon[36790]: pgmap v8262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:24:34.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:34 smithi193 ceph-mon[37473]: pgmap v8262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:24:35.340 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:24:35.340+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:24:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:24:35.340+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:24:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:24:35.340+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:24:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:24:35.340+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:24:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14215.366540 secs 2024-07-26T15:24:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14215.366540 secs 2024-07-26T15:24:35.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14215.366540 secs 2024-07-26T15:24:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:36 smithi134 ceph-mon[31966]: pgmap v8263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:36 smithi189 ceph-mon[36790]: pgmap v8263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:36 smithi193 ceph-mon[37473]: pgmap v8263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14234.790843 secs 2024-07-26T15:24:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14234.790843 secs 2024-07-26T15:24:37.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14234.790843 secs 2024-07-26T15:24:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:38 smithi134 ceph-mon[31966]: pgmap v8264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:38 smithi189 ceph-mon[36790]: pgmap v8264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:38 smithi193 ceph-mon[37473]: pgmap v8264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:24:39.784+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:24:39.784+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:24:39.784+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14220.366662 secs 2024-07-26T15:24:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:40 smithi134 ceph-mon[31966]: pgmap v8265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14220.366662 secs 2024-07-26T15:24:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:40 smithi189 ceph-mon[36790]: pgmap v8265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14220.366662 secs 2024-07-26T15:24:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:40 smithi193 ceph-mon[37473]: pgmap v8265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14239.790960 secs 2024-07-26T15:24:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:42 smithi134 ceph-mon[31966]: pgmap v8266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14239.790960 secs 2024-07-26T15:24:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:42 smithi189 ceph-mon[36790]: pgmap v8266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14239.790960 secs 2024-07-26T15:24:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:42 smithi193 ceph-mon[37473]: pgmap v8266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:44.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:24:44.249+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:44.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:24:44.249+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:44.251 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:24:44.249+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:44 smithi134 ceph-mon[31966]: pgmap v8267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:24:44.729 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:44 smithi189 ceph-mon[36790]: pgmap v8267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:24:44.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:44 smithi193 ceph-mon[37473]: pgmap v8267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:24:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14225.366770 secs 2024-07-26T15:24:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14225.366770 secs 2024-07-26T15:24:45.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14225.366770 secs 2024-07-26T15:24:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:46 smithi189 ceph-mon[36790]: pgmap v8268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:46 smithi193 ceph-mon[37473]: pgmap v8268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:46 smithi134 ceph-mon[31966]: pgmap v8268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14244.791074 secs 2024-07-26T15:24:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14244.791074 secs 2024-07-26T15:24:47.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14244.791074 secs 2024-07-26T15:24:48.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:48 smithi193 ceph-mon[37473]: pgmap v8269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:48 smithi134 ceph-mon[31966]: pgmap v8269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:48 smithi189 ceph-mon[36790]: pgmap v8269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14230.366904 secs 2024-07-26T15:24:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14230.366904 secs 2024-07-26T15:24:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14230.366904 secs 2024-07-26T15:24:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:50 smithi193 ceph-mon[37473]: pgmap v8270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:50 smithi134 ceph-mon[31966]: pgmap v8270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:50 smithi189 ceph-mon[36790]: pgmap v8270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:24:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14249.791188 secs 2024-07-26T15:24:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:52 smithi193 ceph-mon[37473]: pgmap v8271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14249.791188 secs 2024-07-26T15:24:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:52 smithi134 ceph-mon[31966]: pgmap v8271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14249.791188 secs 2024-07-26T15:24:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:52 smithi189 ceph-mon[36790]: pgmap v8271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:54.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:54 smithi189 ceph-mon[36790]: pgmap v8272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:24:54.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14235.367030 secs 2024-07-26T15:24:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:54 smithi193 ceph-mon[37473]: pgmap v8272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:24:54.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14235.367030 secs 2024-07-26T15:24:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:24:54.799+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:54.800 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:24:54.799+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:54.801 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:24:54.799+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:54 smithi134 ceph-mon[31966]: pgmap v8272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:24:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14235.367030 secs 2024-07-26T15:24:56.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:56 smithi193 ceph-mon[37473]: pgmap v8273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:56 smithi134 ceph-mon[31966]: pgmap v8273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:56 smithi189 ceph-mon[36790]: pgmap v8273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:24:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14254.791286 secs 2024-07-26T15:24:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:24:57.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14254.791286 secs 2024-07-26T15:24:57.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:24:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14254.791286 secs 2024-07-26T15:24:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:24:58.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:58 smithi193 ceph-mon[37473]: pgmap v8274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:24:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:24:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:24:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:58 smithi134 ceph-mon[31966]: pgmap v8274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:24:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:24:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:24:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:58 smithi189 ceph-mon[36790]: pgmap v8274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:24:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:24:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:24:58.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:24:59.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:24:59.264+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:59.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:24:59.264+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:24:59.266 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:24:59.264+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:24:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:59 smithi134 ceph-mon[31966]: pgmap v8275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:24:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:59 smithi134 ceph-mon[31966]: pgmap v8276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:24:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:24:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14240.367120 secs 2024-07-26T15:24:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:59 smithi193 ceph-mon[37473]: pgmap v8275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:24:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:59 smithi193 ceph-mon[37473]: pgmap v8276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:24:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:24:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14240.367120 secs 2024-07-26T15:24:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:59 smithi189 ceph-mon[36790]: pgmap v8275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:24:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:59 smithi189 ceph-mon[36790]: pgmap v8276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:24:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:24:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14240.367120 secs 2024-07-26T15:25:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:01 smithi193 ceph-mon[37473]: pgmap v8277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:01.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:01 smithi134 ceph-mon[31966]: pgmap v8277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:01 smithi189 ceph-mon[36790]: pgmap v8277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14259.791397 secs 2024-07-26T15:25:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14259.791397 secs 2024-07-26T15:25:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14259.791397 secs 2024-07-26T15:25:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:03 smithi193 ceph-mon[37473]: pgmap v8278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:25:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:03 smithi134 ceph-mon[31966]: pgmap v8278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:25:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:03 smithi189 ceph-mon[36790]: pgmap v8278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:25:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14245.367265 secs 2024-07-26T15:25:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14245.367265 secs 2024-07-26T15:25:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14245.367265 secs 2024-07-26T15:25:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:05 smithi134 ceph-mon[31966]: pgmap v8279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:05 smithi189 ceph-mon[36790]: pgmap v8279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:06.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:05 smithi193 ceph-mon[37473]: pgmap v8279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14264.791536 secs 2024-07-26T15:25:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:07 smithi134 ceph-mon[31966]: pgmap v8280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:25:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14264.791536 secs 2024-07-26T15:25:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:07 smithi189 ceph-mon[36790]: pgmap v8280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:25:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14264.791536 secs 2024-07-26T15:25:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:07 smithi193 ceph-mon[37473]: pgmap v8280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:25:09.812 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:25:09.811+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:25:09.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:25:09.811+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:25:09.813 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:25:09.811+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:09 smithi134 ceph-mon[31966]: pgmap v8281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:25:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14250.367377 secs 2024-07-26T15:25:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:09 smithi189 ceph-mon[36790]: pgmap v8281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:25:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14250.367377 secs 2024-07-26T15:25:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:09 smithi193 ceph-mon[37473]: pgmap v8281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:25:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14250.367377 secs 2024-07-26T15:25:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:11 smithi134 ceph-mon[31966]: pgmap v8282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:11 smithi189 ceph-mon[36790]: pgmap v8282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:12.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:11 smithi193 ceph-mon[37473]: pgmap v8282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14269.791674 secs 2024-07-26T15:25:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14269.791674 secs 2024-07-26T15:25:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14269.791674 secs 2024-07-26T15:25:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:13 smithi134 ceph-mon[31966]: pgmap v8283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:13 smithi189 ceph-mon[36790]: pgmap v8283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:13 smithi193 ceph-mon[37473]: pgmap v8283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:14.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:25:14.273+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:25:14.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:25:14.273+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:14.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:25:14.273+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:25:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14255.367514 secs 2024-07-26T15:25:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14255.367514 secs 2024-07-26T15:25:15.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14255.367514 secs 2024-07-26T15:25:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:15 smithi134 ceph-mon[31966]: pgmap v8284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:15.926 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:15 smithi193 ceph-mon[37473]: pgmap v8284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:15 smithi189 ceph-mon[36790]: pgmap v8284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14274.791811 secs 2024-07-26T15:25:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:17 smithi134 ceph-mon[31966]: pgmap v8285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14274.791811 secs 2024-07-26T15:25:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:17 smithi189 ceph-mon[36790]: pgmap v8285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:18.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14274.791811 secs 2024-07-26T15:25:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:17 smithi193 ceph-mon[37473]: pgmap v8285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:19 smithi134 ceph-mon[31966]: pgmap v8286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:25:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14260.367650 secs 2024-07-26T15:25:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:19 smithi189 ceph-mon[36790]: pgmap v8286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:25:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14260.367650 secs 2024-07-26T15:25:20.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:19 smithi193 ceph-mon[37473]: pgmap v8286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:25:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14260.367650 secs 2024-07-26T15:25:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:21 smithi134 ceph-mon[31966]: pgmap v8287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:21 smithi189 ceph-mon[36790]: pgmap v8287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:22.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:21 smithi193 ceph-mon[37473]: pgmap v8287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14279.791932 secs 2024-07-26T15:25:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14279.791932 secs 2024-07-26T15:25:23.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14279.791932 secs 2024-07-26T15:25:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:23 smithi134 ceph-mon[31966]: pgmap v8288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:23 smithi189 ceph-mon[36790]: pgmap v8288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:24.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:23 smithi193 ceph-mon[37473]: pgmap v8288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:25:24.826+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:25:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:25:24.827+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:25:24.828 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:25:24.827+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:24.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14265.367749 secs 2024-07-26T15:25:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14265.367749 secs 2024-07-26T15:25:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14265.367749 secs 2024-07-26T15:25:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:25 smithi134 ceph-mon[31966]: pgmap v8289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:25 smithi189 ceph-mon[36790]: pgmap v8289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:25 smithi193 ceph-mon[37473]: pgmap v8289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14284.792064 secs 2024-07-26T15:25:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:27 smithi134 ceph-mon[31966]: pgmap v8290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14284.792064 secs 2024-07-26T15:25:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:27 smithi189 ceph-mon[36790]: pgmap v8290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14284.792064 secs 2024-07-26T15:25:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:27 smithi193 ceph-mon[37473]: pgmap v8290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:29.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:25:29.288+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:25:29.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:25:29.289+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:25:29.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:25:29.289+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:29 smithi134 ceph-mon[31966]: pgmap v8291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:25:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14270.367871 secs 2024-07-26T15:25:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:29 smithi189 ceph-mon[36790]: pgmap v8291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:25:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14270.367871 secs 2024-07-26T15:25:30.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:29 smithi193 ceph-mon[37473]: pgmap v8291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:25:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14270.367871 secs 2024-07-26T15:25:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:31 smithi134 ceph-mon[31966]: pgmap v8292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:31 smithi189 ceph-mon[36790]: pgmap v8292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:32.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:31 smithi193 ceph-mon[37473]: pgmap v8292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14289.792188 secs 2024-07-26T15:25:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14289.792188 secs 2024-07-26T15:25:33.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14289.792188 secs 2024-07-26T15:25:33.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:33 smithi134 ceph-mon[31966]: pgmap v8293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:33 smithi189 ceph-mon[36790]: pgmap v8293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:34.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:33 smithi193 ceph-mon[37473]: pgmap v8293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14275.367988 secs 2024-07-26T15:25:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14275.367988 secs 2024-07-26T15:25:35.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14275.367988 secs 2024-07-26T15:25:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:35 smithi134 ceph-mon[31966]: pgmap v8294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:35 smithi189 ceph-mon[36790]: pgmap v8294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:35 smithi193 ceph-mon[37473]: pgmap v8294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14294.792309 secs 2024-07-26T15:25:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:37 smithi189 ceph-mon[36790]: pgmap v8295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14294.792309 secs 2024-07-26T15:25:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:37 smithi193 ceph-mon[37473]: pgmap v8295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14294.792309 secs 2024-07-26T15:25:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:37 smithi134 ceph-mon[31966]: pgmap v8295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:25:39.832+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:25:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:25:39.832+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:25:39.834 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:25:39.832+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:39 smithi189 ceph-mon[36790]: pgmap v8296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:25:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14280.368122 secs 2024-07-26T15:25:40.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:39 smithi193 ceph-mon[37473]: pgmap v8296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:25:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14280.368122 secs 2024-07-26T15:25:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:39 smithi134 ceph-mon[31966]: pgmap v8296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:25:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14280.368122 secs 2024-07-26T15:25:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:41 smithi189 ceph-mon[36790]: pgmap v8297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:42.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:41 smithi193 ceph-mon[37473]: pgmap v8297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:42.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:41 smithi134 ceph-mon[31966]: pgmap v8297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14299.792419 secs 2024-07-26T15:25:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14299.792419 secs 2024-07-26T15:25:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14299.792419 secs 2024-07-26T15:25:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:43 smithi193 ceph-mon[37473]: pgmap v8298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:43 smithi134 ceph-mon[31966]: pgmap v8298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:43 smithi189 ceph-mon[36790]: pgmap v8298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:44.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:25:44.295+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:25:44.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:25:44.295+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:25:44.297 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:25:44.295+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:45.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14285.368214 secs 2024-07-26T15:25:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14285.368214 secs 2024-07-26T15:25:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14285.368214 secs 2024-07-26T15:25:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:45 smithi193 ceph-mon[37473]: pgmap v8299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:45 smithi134 ceph-mon[31966]: pgmap v8299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:45 smithi189 ceph-mon[36790]: pgmap v8299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14304.792514 secs 2024-07-26T15:25:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:47 smithi193 ceph-mon[37473]: pgmap v8300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14304.792514 secs 2024-07-26T15:25:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:47 smithi134 ceph-mon[31966]: pgmap v8300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14304.792514 secs 2024-07-26T15:25:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:47 smithi189 ceph-mon[36790]: pgmap v8300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:49 smithi193 ceph-mon[37473]: pgmap v8301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:25:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14290.368290 secs 2024-07-26T15:25:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:49 smithi134 ceph-mon[31966]: pgmap v8301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:25:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14290.368290 secs 2024-07-26T15:25:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:49 smithi189 ceph-mon[36790]: pgmap v8301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:25:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14290.368290 secs 2024-07-26T15:25:52.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:51 smithi193 ceph-mon[37473]: pgmap v8302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:51 smithi134 ceph-mon[31966]: pgmap v8302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:51 smithi189 ceph-mon[36790]: pgmap v8302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14309.792591 secs 2024-07-26T15:25:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14309.792591 secs 2024-07-26T15:25:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14309.792591 secs 2024-07-26T15:25:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:53 smithi193 ceph-mon[37473]: pgmap v8303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:54.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:53 smithi189 ceph-mon[36790]: pgmap v8303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:53 smithi134 ceph-mon[31966]: pgmap v8303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:25:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:25:54.837+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:25:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:25:54.837+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:25:54.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:25:54.837+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:25:55.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14295.368408 secs 2024-07-26T15:25:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14295.368408 secs 2024-07-26T15:25:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14295.368408 secs 2024-07-26T15:25:56.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:55 smithi193 ceph-mon[37473]: pgmap v8304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:55 smithi134 ceph-mon[31966]: pgmap v8304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:55 smithi189 ceph-mon[36790]: pgmap v8304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:25:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14314.792728 secs 2024-07-26T15:25:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:57 smithi193 ceph-mon[37473]: pgmap v8305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14314.792728 secs 2024-07-26T15:25:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:57 smithi134 ceph-mon[31966]: pgmap v8305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14314.792728 secs 2024-07-26T15:25:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:57 smithi189 ceph-mon[36790]: pgmap v8305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:25:59.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:25:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:25:59.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:25:59.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:25:59.310+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:25:59.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:25:59.310+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:25:59.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:25:59.311+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:59 smithi134 ceph-mon[31966]: pgmap v8306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14300.368535 secs 2024-07-26T15:26:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:26:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:26:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:25:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:26:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:59 smithi189 ceph-mon[36790]: pgmap v8306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14300.368535 secs 2024-07-26T15:26:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:26:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:26:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:25:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:26:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:59 smithi193 ceph-mon[37473]: pgmap v8306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14300.368535 secs 2024-07-26T15:26:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:26:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:26:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:25:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:26:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:00 smithi134 ceph-mon[31966]: pgmap v8307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:26:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:00 smithi134 ceph-mon[31966]: pgmap v8308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:26:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:00 smithi189 ceph-mon[36790]: pgmap v8307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:26:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:00 smithi189 ceph-mon[36790]: pgmap v8308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:26:01.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:00 smithi193 ceph-mon[37473]: pgmap v8307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:26:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:00 smithi193 ceph-mon[37473]: pgmap v8308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:26:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14319.792845 secs 2024-07-26T15:26:03.444 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:02 smithi134 ceph-mon[31966]: pgmap v8309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:03.446 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14319.792845 secs 2024-07-26T15:26:03.446 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:02 smithi193 ceph-mon[37473]: pgmap v8309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:03.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14319.792845 secs 2024-07-26T15:26:03.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:02 smithi189 ceph-mon[36790]: pgmap v8309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:04 smithi134 ceph-mon[31966]: pgmap v8310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:26:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14305.368664 secs 2024-07-26T15:26:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:04 smithi189 ceph-mon[36790]: pgmap v8310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:26:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14305.368664 secs 2024-07-26T15:26:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:04 smithi193 ceph-mon[37473]: pgmap v8310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:26:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14305.368664 secs 2024-07-26T15:26:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:06 smithi134 ceph-mon[31966]: pgmap v8311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:26:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:06 smithi189 ceph-mon[36790]: pgmap v8311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:26:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:06 smithi193 ceph-mon[37473]: pgmap v8311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:26:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14324.792966 secs 2024-07-26T15:26:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14324.792966 secs 2024-07-26T15:26:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14324.792966 secs 2024-07-26T15:26:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:08 smithi134 ceph-mon[31966]: pgmap v8312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:26:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:08 smithi189 ceph-mon[36790]: pgmap v8312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:26:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:08 smithi193 ceph-mon[37473]: pgmap v8312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:26:09.853 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:26:09.852+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:26:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:26:09.852+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:26:09.854 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:26:09.852+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14310.368790 secs 2024-07-26T15:26:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14310.368790 secs 2024-07-26T15:26:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14310.368790 secs 2024-07-26T15:26:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:10 smithi134 ceph-mon[31966]: pgmap v8313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:26:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:10 smithi189 ceph-mon[36790]: pgmap v8313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:26:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:10 smithi193 ceph-mon[37473]: pgmap v8313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:26:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14329.793085 secs 2024-07-26T15:26:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:12 smithi134 ceph-mon[31966]: pgmap v8314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14329.793085 secs 2024-07-26T15:26:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:12 smithi189 ceph-mon[36790]: pgmap v8314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14329.793085 secs 2024-07-26T15:26:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:12 smithi193 ceph-mon[37473]: pgmap v8314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:14.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:26:14.312+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:26:14.314 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:26:14.312+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:26:14.315 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:26:14.313+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14315.368957 secs 2024-07-26T15:26:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:14 smithi134 ceph-mon[31966]: pgmap v8315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14315.368957 secs 2024-07-26T15:26:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:14 smithi189 ceph-mon[36790]: pgmap v8315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14315.368957 secs 2024-07-26T15:26:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:14 smithi193 ceph-mon[37473]: pgmap v8315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:16 smithi134 ceph-mon[31966]: pgmap v8316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:16 smithi189 ceph-mon[36790]: pgmap v8316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:16 smithi193 ceph-mon[37473]: pgmap v8316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14334.793203 secs 2024-07-26T15:26:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14334.793203 secs 2024-07-26T15:26:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14334.793203 secs 2024-07-26T15:26:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:18 smithi134 ceph-mon[31966]: pgmap v8317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:18 smithi189 ceph-mon[36790]: pgmap v8317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:18 smithi193 ceph-mon[37473]: pgmap v8317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14320.369053 secs 2024-07-26T15:26:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14320.369053 secs 2024-07-26T15:26:20.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14320.369053 secs 2024-07-26T15:26:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:20 smithi134 ceph-mon[31966]: pgmap v8318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:26:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:20 smithi189 ceph-mon[36790]: pgmap v8318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:26:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:20 smithi193 ceph-mon[37473]: pgmap v8318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:26:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14339.793294 secs 2024-07-26T15:26:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:22 smithi134 ceph-mon[31966]: pgmap v8319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14339.793294 secs 2024-07-26T15:26:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:22 smithi189 ceph-mon[36790]: pgmap v8319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14339.793294 secs 2024-07-26T15:26:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:22 smithi193 ceph-mon[37473]: pgmap v8319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:23 smithi193 ceph-mon[37473]: pgmap v8320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:23 smithi134 ceph-mon[31966]: pgmap v8320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:23 smithi189 ceph-mon[36790]: pgmap v8320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:26:24.868+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:26:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:26:24.868+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:26:24.869 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:26:24.868+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:25.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14325.369166 secs 2024-07-26T15:26:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14325.369166 secs 2024-07-26T15:26:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14325.369166 secs 2024-07-26T15:26:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:25 smithi193 ceph-mon[37473]: pgmap v8321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:25 smithi134 ceph-mon[31966]: pgmap v8321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:25 smithi189 ceph-mon[36790]: pgmap v8321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:27.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14344.793411 secs 2024-07-26T15:26:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14344.793411 secs 2024-07-26T15:26:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14344.793411 secs 2024-07-26T15:26:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:28 smithi134 ceph-mon[31966]: pgmap v8322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:28 smithi189 ceph-mon[36790]: pgmap v8322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:28 smithi193 ceph-mon[37473]: pgmap v8322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:29.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:26:29.328+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:26:29.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:26:29.328+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:26:29.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:26:29.328+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14330.369289 secs 2024-07-26T15:26:29.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14330.369289 secs 2024-07-26T15:26:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14330.369289 secs 2024-07-26T15:26:30.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:30 smithi134 ceph-mon[31966]: pgmap v8323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:26:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:30 smithi189 ceph-mon[36790]: pgmap v8323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:26:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:30 smithi193 ceph-mon[37473]: pgmap v8323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:26:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14349.793532 secs 2024-07-26T15:26:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:32 smithi134 ceph-mon[31966]: pgmap v8324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14349.793532 secs 2024-07-26T15:26:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:32 smithi189 ceph-mon[36790]: pgmap v8324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14349.793532 secs 2024-07-26T15:26:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:32 smithi193 ceph-mon[37473]: pgmap v8324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14335.369372 secs 2024-07-26T15:26:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:34 smithi134 ceph-mon[31966]: pgmap v8325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14335.369372 secs 2024-07-26T15:26:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:34 smithi189 ceph-mon[36790]: pgmap v8325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14335.369372 secs 2024-07-26T15:26:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:34 smithi193 ceph-mon[37473]: pgmap v8325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:36 smithi134 ceph-mon[31966]: pgmap v8326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:36 smithi189 ceph-mon[36790]: pgmap v8326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:36 smithi193 ceph-mon[37473]: pgmap v8326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14354.793627 secs 2024-07-26T15:26:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14354.793627 secs 2024-07-26T15:26:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14354.793627 secs 2024-07-26T15:26:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:38 smithi189 ceph-mon[36790]: pgmap v8327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:39.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:38 smithi193 ceph-mon[37473]: pgmap v8327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:38 smithi134 ceph-mon[31966]: pgmap v8327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:39.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:26:39.823+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:26:39.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:26:39.823+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:26:39.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:26:39.824+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 112 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T15:26:39.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:26:39.824+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T15:26:39.881 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:26:39.880+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:26:39.881 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:26:39.880+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:26:39.881 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:26:39.880+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14340.369491 secs 2024-07-26T15:26:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14340.369491 secs 2024-07-26T15:26:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14340.369491 secs 2024-07-26T15:26:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:40 smithi193 ceph-mon[37473]: pgmap v8328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:26:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:40 smithi134 ceph-mon[31966]: pgmap v8328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:26:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:40 smithi189 ceph-mon[36790]: pgmap v8328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:26:42.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:41 smithi193 ceph-mon[37473]: pgmap v8329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:41 smithi134 ceph-mon[31966]: pgmap v8329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:41 smithi189 ceph-mon[36790]: pgmap v8329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:43.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14359.793728 secs 2024-07-26T15:26:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14359.793728 secs 2024-07-26T15:26:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14359.793728 secs 2024-07-26T15:26:44.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:43 smithi193 ceph-mon[37473]: pgmap v8330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:44.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:26:44.344+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:26:44.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:26:44.344+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:26:44.346 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:26:44.344+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:44.346 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:43 smithi134 ceph-mon[31966]: pgmap v8330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:43 smithi189 ceph-mon[36790]: pgmap v8330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:45.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14345.369621 secs 2024-07-26T15:26:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14345.369621 secs 2024-07-26T15:26:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14345.369621 secs 2024-07-26T15:26:46.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:45 smithi193 ceph-mon[37473]: pgmap v8331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:45 smithi134 ceph-mon[31966]: pgmap v8331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:45 smithi189 ceph-mon[36790]: pgmap v8331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14364.793848 secs 2024-07-26T15:26:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14364.793848 secs 2024-07-26T15:26:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14364.793848 secs 2024-07-26T15:26:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:48 smithi134 ceph-mon[31966]: pgmap v8332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:48 smithi189 ceph-mon[36790]: pgmap v8332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:48.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:48 smithi193 ceph-mon[37473]: pgmap v8332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:26:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14350.369737 secs 2024-07-26T15:26:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14350.369737 secs 2024-07-26T15:26:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14350.369737 secs 2024-07-26T15:26:50.364 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:26:50.363+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:26:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:26:50.363+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:26:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:26:50.364+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:26:50.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:26:50.364+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:26:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:50 smithi134 ceph-mon[31966]: pgmap v8333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:26:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:50 smithi189 ceph-mon[36790]: pgmap v8333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:26:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:50 smithi193 ceph-mon[37473]: pgmap v8333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:26:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14369.793930 secs 2024-07-26T15:26:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:52 smithi134 ceph-mon[31966]: pgmap v8334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14369.793930 secs 2024-07-26T15:26:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:52 smithi189 ceph-mon[36790]: pgmap v8334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:53.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14369.793930 secs 2024-07-26T15:26:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:52 smithi193 ceph-mon[37473]: pgmap v8334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:26:54.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:26:54.888+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:26:54.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:26:54.888+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:26:54.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:26:54.888+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:26:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14355.369896 secs 2024-07-26T15:26:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:54 smithi134 ceph-mon[31966]: pgmap v8335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:26:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14355.369896 secs 2024-07-26T15:26:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:54 smithi189 ceph-mon[36790]: pgmap v8335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:26:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14355.369896 secs 2024-07-26T15:26:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:54 smithi193 ceph-mon[37473]: pgmap v8335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:26:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:56 smithi189 ceph-mon[36790]: pgmap v8336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:26:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:56 smithi193 ceph-mon[37473]: pgmap v8336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:26:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:56 smithi134 ceph-mon[31966]: pgmap v8336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:26:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14374.794074 secs 2024-07-26T15:26:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14374.794074 secs 2024-07-26T15:26:58.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14374.794074 secs 2024-07-26T15:26:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:58 smithi189 ceph-mon[36790]: pgmap v8337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:26:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:58 smithi193 ceph-mon[37473]: pgmap v8337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:26:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:58 smithi134 ceph-mon[31966]: pgmap v8337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:26:59.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:26:59.350+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:26:59.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:26:59.350+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:26:59.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:26:59.351+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:26:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14360.370018 secs 2024-07-26T15:27:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:26:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14360.370018 secs 2024-07-26T15:27:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:26:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14360.370018 secs 2024-07-26T15:27:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:00 smithi193 ceph-mon[37473]: pgmap v8338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:27:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:27:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:00 smithi134 ceph-mon[31966]: pgmap v8338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:27:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:27:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:00 smithi189 ceph-mon[36790]: pgmap v8338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:27:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:27:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:01 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:27:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:01 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:27:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:27:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:27:02.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:01 smithi193 ceph-mon[37473]: pgmap v8339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T15:27:02.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:01 smithi193 ceph-mon[37473]: pgmap v8340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:27:02.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:27:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:01 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:27:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:01 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:27:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:27:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:27:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:01 smithi134 ceph-mon[31966]: pgmap v8339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T15:27:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:01 smithi134 ceph-mon[31966]: pgmap v8340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:27:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:27:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:27:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:01 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:27:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:01 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:27:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:27:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:27:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:01 smithi189 ceph-mon[36790]: pgmap v8339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T15:27:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:01 smithi189 ceph-mon[36790]: pgmap v8340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:27:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:27:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14379.794171 secs 2024-07-26T15:27:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14379.794171 secs 2024-07-26T15:27:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14379.794171 secs 2024-07-26T15:27:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:03 smithi193 ceph-mon[37473]: pgmap v8341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:27:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:03 smithi134 ceph-mon[31966]: pgmap v8341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:27:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:03 smithi189 ceph-mon[36790]: pgmap v8341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:27:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14365.370147 secs 2024-07-26T15:27:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14365.370147 secs 2024-07-26T15:27:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14365.370147 secs 2024-07-26T15:27:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:05 smithi193 ceph-mon[37473]: pgmap v8342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:27:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:05 smithi134 ceph-mon[31966]: pgmap v8342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:27:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:05 smithi189 ceph-mon[36790]: pgmap v8342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:27:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14384.794297 secs 2024-07-26T15:27:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:07 smithi134 ceph-mon[31966]: pgmap v8343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:27:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14384.794297 secs 2024-07-26T15:27:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:07 smithi189 ceph-mon[36790]: pgmap v8343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:27:08.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14384.794297 secs 2024-07-26T15:27:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:07 smithi193 ceph-mon[37473]: pgmap v8343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:27:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:09.903+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:27:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:09.904+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:27:09.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:09.904+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14370.370277 secs 2024-07-26T15:27:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:09 smithi134 ceph-mon[31966]: pgmap v8344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:27:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14370.370277 secs 2024-07-26T15:27:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:09 smithi189 ceph-mon[36790]: pgmap v8344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:27:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14370.370277 secs 2024-07-26T15:27:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:09 smithi193 ceph-mon[37473]: pgmap v8344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:27:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:11 smithi134 ceph-mon[31966]: pgmap v8345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:27:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:11 smithi189 ceph-mon[36790]: pgmap v8345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:27:12.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:11 smithi193 ceph-mon[37473]: pgmap v8345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:27:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14389.794420 secs 2024-07-26T15:27:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14389.794420 secs 2024-07-26T15:27:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14389.794420 secs 2024-07-26T15:27:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:13 smithi134 ceph-mon[31966]: pgmap v8346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:13 smithi189 ceph-mon[36790]: pgmap v8346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:14.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:13 smithi193 ceph-mon[37473]: pgmap v8346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:14.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:14.354+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:27:14.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:14.354+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:27:14.356 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:14.354+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14375.370398 secs 2024-07-26T15:27:15.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14375.370398 secs 2024-07-26T15:27:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14375.370398 secs 2024-07-26T15:27:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:15 smithi134 ceph-mon[31966]: pgmap v8347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:15 smithi189 ceph-mon[36790]: pgmap v8347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:15 smithi193 ceph-mon[37473]: pgmap v8347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14394.794532 secs 2024-07-26T15:27:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:17 smithi134 ceph-mon[31966]: pgmap v8348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14394.794532 secs 2024-07-26T15:27:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:17 smithi189 ceph-mon[36790]: pgmap v8348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14394.794532 secs 2024-07-26T15:27:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:17 smithi193 ceph-mon[37473]: pgmap v8348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:18 smithi193 ceph-mon[37473]: pgmap v8349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:18 smithi134 ceph-mon[31966]: pgmap v8349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:18 smithi189 ceph-mon[36790]: pgmap v8349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14380.370560 secs 2024-07-26T15:27:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14380.370560 secs 2024-07-26T15:27:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14380.370560 secs 2024-07-26T15:27:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:20 smithi193 ceph-mon[37473]: pgmap v8350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:20 smithi134 ceph-mon[31966]: pgmap v8350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:20 smithi189 ceph-mon[36790]: pgmap v8350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:22.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14399.794674 secs 2024-07-26T15:27:22.337 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:22.336+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T15:27:22.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:22.336+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T15:27:22.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:22.336+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b38008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:27:22.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:22.336+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b38008040 comp rx=0 tx=0).stop 2024-07-26T15:27:22.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:22.336+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T15:27:22.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:22.336+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 msgr2=0x7f9b300c0ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:27:22.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:22.336+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c0be0 0x7f9b300c0ff0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:27:22.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:22.336+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 0x7f9b300c4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:27:22.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:22.337+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 0x7f9b300c4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:27:22.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:22.338+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 0x7f9b300c4130 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9b300b6950 tx=0x7f9b38039050 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:27:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14399.794674 secs 2024-07-26T15:27:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14399.794674 secs 2024-07-26T15:27:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:23 smithi134 ceph-mon[31966]: pgmap v8351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:23 smithi189 ceph-mon[36790]: pgmap v8351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:23 smithi193 ceph-mon[37473]: pgmap v8351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14385.370720 secs 2024-07-26T15:27:24.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:24.832+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:27:24.833 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:24.832+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:27:24.834 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14385.370720 secs 2024-07-26T15:27:24.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:24.832+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T15:27:24.834 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:24.832+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 113 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T15:27:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:24.910+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:27:24.911 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:24.910+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:27:24.912 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:24.910+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14385.370720 secs 2024-07-26T15:27:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:25 smithi134 ceph-mon[31966]: pgmap v8352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:25 smithi189 ceph-mon[36790]: pgmap v8352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:25 smithi193 ceph-mon[37473]: pgmap v8352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14404.794806 secs 2024-07-26T15:27:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:27 smithi134 ceph-mon[31966]: pgmap v8353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14404.794806 secs 2024-07-26T15:27:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:27 smithi189 ceph-mon[36790]: pgmap v8353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14404.794806 secs 2024-07-26T15:27:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:27 smithi193 ceph-mon[37473]: pgmap v8353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:29.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:29.362+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:27:29.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:29.362+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:29.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:29.362+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:27:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14390.370852 secs 2024-07-26T15:27:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:29 smithi134 ceph-mon[31966]: pgmap v8354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14390.370852 secs 2024-07-26T15:27:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:29 smithi189 ceph-mon[36790]: pgmap v8354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14390.370852 secs 2024-07-26T15:27:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:29 smithi193 ceph-mon[37473]: pgmap v8354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:31 smithi134 ceph-mon[31966]: pgmap v8355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:31 smithi189 ceph-mon[36790]: pgmap v8355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:31 smithi193 ceph-mon[37473]: pgmap v8355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:32.888+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 msgr2=0x7f21ac0c10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T15:27:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:32.888+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 msgr2=0x7f21ac0c10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T15:27:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:32.888+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f2198005820 tx=0x7f21b80aa000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:27:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:32.888+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f2198005820 tx=0x7f21b80aa000 comp rx=0 tx=0).stop 2024-07-26T15:27:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:32.888+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 msgr2=0x7f21ac0c10b0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T15:27:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:32.888+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 msgr2=0x7f21ac0c10b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:27:32.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:32.888+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:27:32.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:32.888+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:27:32.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:32.889+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:27:32.892 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:32.891+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2198003490 tx=0x7f21b80c3000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:27:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14409.794952 secs 2024-07-26T15:27:33.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14409.794952 secs 2024-07-26T15:27:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14409.794952 secs 2024-07-26T15:27:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:33 smithi134 ceph-mon[31966]: pgmap v8356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:33 smithi189 ceph-mon[36790]: pgmap v8356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:33 smithi193 ceph-mon[37473]: pgmap v8356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14395.371018 secs 2024-07-26T15:27:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14395.371018 secs 2024-07-26T15:27:35.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14395.371018 secs 2024-07-26T15:27:35.372 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:35.371+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:27:35.372 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:35.371+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:27:35.373 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:35.372+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:27:35.373 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:35.372+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:27:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:35 smithi134 ceph-mon[31966]: pgmap v8357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:35 smithi189 ceph-mon[36790]: pgmap v8357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:35 smithi193 ceph-mon[37473]: pgmap v8357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:36 smithi189 ceph-mon[36790]: pgmap v8358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:36 smithi193 ceph-mon[37473]: pgmap v8358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:36 smithi134 ceph-mon[31966]: pgmap v8358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14414.795088 secs 2024-07-26T15:27:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14414.795088 secs 2024-07-26T15:27:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14414.795088 secs 2024-07-26T15:27:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:38 smithi193 ceph-mon[37473]: pgmap v8359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:38 smithi134 ceph-mon[31966]: pgmap v8359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:38 smithi189 ceph-mon[36790]: pgmap v8359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:39.919+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:27:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:39.919+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:27:39.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:39.919+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14400.371137 secs 2024-07-26T15:27:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14400.371137 secs 2024-07-26T15:27:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14400.371137 secs 2024-07-26T15:27:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:40 smithi193 ceph-mon[37473]: pgmap v8360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:40 smithi134 ceph-mon[31966]: pgmap v8360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:40 smithi189 ceph-mon[36790]: pgmap v8360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14419.795206 secs 2024-07-26T15:27:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14419.795206 secs 2024-07-26T15:27:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14419.795206 secs 2024-07-26T15:27:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:43 smithi134 ceph-mon[31966]: pgmap v8361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:43 smithi189 ceph-mon[36790]: pgmap v8361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:43 smithi193 ceph-mon[37473]: pgmap v8361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:44.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:44.376+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:27:44.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:44.377+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:27:44.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:44.377+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14405.371266 secs 2024-07-26T15:27:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14405.371266 secs 2024-07-26T15:27:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14405.371266 secs 2024-07-26T15:27:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:45 smithi134 ceph-mon[31966]: pgmap v8362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:45 smithi189 ceph-mon[36790]: pgmap v8362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:46.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:45 smithi193 ceph-mon[37473]: pgmap v8362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14424.795334 secs 2024-07-26T15:27:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:47 smithi134 ceph-mon[31966]: pgmap v8363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14424.795334 secs 2024-07-26T15:27:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:47 smithi189 ceph-mon[36790]: pgmap v8363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14424.795334 secs 2024-07-26T15:27:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:47 smithi193 ceph-mon[37473]: pgmap v8363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14410.371415 secs 2024-07-26T15:27:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:49 smithi134 ceph-mon[31966]: pgmap v8364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14410.371415 secs 2024-07-26T15:27:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:49 smithi189 ceph-mon[36790]: pgmap v8364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14410.371415 secs 2024-07-26T15:27:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:49 smithi193 ceph-mon[37473]: pgmap v8364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:51 smithi134 ceph-mon[31966]: pgmap v8365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:51 smithi189 ceph-mon[36790]: pgmap v8365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:52.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:51 smithi193 ceph-mon[37473]: pgmap v8365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14429.795402 secs 2024-07-26T15:27:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14429.795402 secs 2024-07-26T15:27:53.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14429.795402 secs 2024-07-26T15:27:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:53 smithi134 ceph-mon[31966]: pgmap v8366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:53 smithi189 ceph-mon[36790]: pgmap v8366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:53 smithi193 ceph-mon[37473]: pgmap v8366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:27:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:54.934+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:27:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:54.934+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:27:54.934+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:27:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14415.371541 secs 2024-07-26T15:27:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:54 smithi189 ceph-mon[36790]: pgmap v8367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14415.371541 secs 2024-07-26T15:27:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:54 smithi193 ceph-mon[37473]: pgmap v8367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14415.371541 secs 2024-07-26T15:27:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:54 smithi134 ceph-mon[31966]: pgmap v8367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:27:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14434.795514 secs 2024-07-26T15:27:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:57 smithi134 ceph-mon[31966]: pgmap v8368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14434.795514 secs 2024-07-26T15:27:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:57 smithi189 ceph-mon[36790]: pgmap v8368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14434.795514 secs 2024-07-26T15:27:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:57 smithi193 ceph-mon[37473]: pgmap v8368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:27:59.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:59.392+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:27:59.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:59.392+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:27:59.394 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:27:59.392+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14420.371694 secs 2024-07-26T15:28:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:27:59 smithi134 ceph-mon[31966]: pgmap v8369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14420.371694 secs 2024-07-26T15:28:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:27:59 smithi189 ceph-mon[36790]: pgmap v8369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14420.371694 secs 2024-07-26T15:28:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:27:59 smithi193 ceph-mon[37473]: pgmap v8369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:01 smithi134 ceph-mon[31966]: pgmap v8370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:28:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:01 smithi189 ceph-mon[36790]: pgmap v8370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:28:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:01 smithi193 ceph-mon[37473]: pgmap v8370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:28:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:02.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:28:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14439.795588 secs 2024-07-26T15:28:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:02 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:28:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:02 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:28:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:02 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:28:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:02 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:28:03.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:28:03.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:28:03.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:28:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14439.795588 secs 2024-07-26T15:28:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:02 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:28:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:02 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:28:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:02 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:28:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:02 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:28:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:28:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:28:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:28:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14439.795588 secs 2024-07-26T15:28:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:02 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:28:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:02 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:28:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:02 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:28:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:02 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:28:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:28:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:28:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:28:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:03 smithi189 ceph-mon[36790]: pgmap v8371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:28:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:03 smithi189 ceph-mon[36790]: pgmap v8372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:28:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:03 smithi193 ceph-mon[37473]: pgmap v8371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:28:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:03 smithi193 ceph-mon[37473]: pgmap v8372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:28:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:03 smithi134 ceph-mon[31966]: pgmap v8371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:28:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:03 smithi134 ceph-mon[31966]: pgmap v8372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:28:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14425.371845 secs 2024-07-26T15:28:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:04 smithi189 ceph-mon[36790]: pgmap v8373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14425.371845 secs 2024-07-26T15:28:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:04 smithi193 ceph-mon[37473]: pgmap v8373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14425.371845 secs 2024-07-26T15:28:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:04 smithi134 ceph-mon[31966]: pgmap v8373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:07 smithi134 ceph-mon[31966]: pgmap v8374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:07.501 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14444.795700 secs 2024-07-26T15:28:07.504 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:07 smithi189 ceph-mon[36790]: pgmap v8374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:07.504 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14444.795700 secs 2024-07-26T15:28:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:07 smithi193 ceph-mon[37473]: pgmap v8374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14444.795700 secs 2024-07-26T15:28:09.352 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:09 smithi134 ceph-mon[31966]: pgmap v8375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:09 smithi189 ceph-mon[36790]: pgmap v8375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:09 smithi193 ceph-mon[37473]: pgmap v8375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:09.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:28:09.935+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:09.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:28:09.935+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:09.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:28:09.935+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14430.371972 secs 2024-07-26T15:28:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14430.371972 secs 2024-07-26T15:28:10.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14430.371972 secs 2024-07-26T15:28:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:11 smithi134 ceph-mon[31966]: pgmap v8376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:11 smithi189 ceph-mon[36790]: pgmap v8376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:11 smithi193 ceph-mon[37473]: pgmap v8376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:28:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14449.795824 secs 2024-07-26T15:28:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14449.795824 secs 2024-07-26T15:28:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14449.795824 secs 2024-07-26T15:28:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:13 smithi134 ceph-mon[31966]: pgmap v8377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:28:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:13 smithi189 ceph-mon[36790]: pgmap v8377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:28:13.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:13 smithi193 ceph-mon[37473]: pgmap v8377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:28:14.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:28:14.394+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:14.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:28:14.394+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:14.395 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:28:14.394+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14435.372068 secs 2024-07-26T15:28:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:15 smithi134 ceph-mon[31966]: pgmap v8378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14435.372068 secs 2024-07-26T15:28:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:15 smithi189 ceph-mon[36790]: pgmap v8378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14435.372068 secs 2024-07-26T15:28:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:15 smithi193 ceph-mon[37473]: pgmap v8378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:16 smithi193 ceph-mon[37473]: pgmap v8379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:16 smithi134 ceph-mon[31966]: pgmap v8379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:16 smithi189 ceph-mon[36790]: pgmap v8379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:18 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14454.795973 secs 2024-07-26T15:28:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:18 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14454.795973 secs 2024-07-26T15:28:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:18 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14454.795973 secs 2024-07-26T15:28:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:19 smithi193 ceph-mon[37473]: pgmap v8380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:19 smithi134 ceph-mon[31966]: pgmap v8380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:19 smithi189 ceph-mon[36790]: pgmap v8380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14440.372210 secs 2024-07-26T15:28:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14440.372210 secs 2024-07-26T15:28:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14440.372210 secs 2024-07-26T15:28:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:21 smithi134 ceph-mon[31966]: pgmap v8381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:21 smithi189 ceph-mon[36790]: pgmap v8381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:21.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:21 smithi193 ceph-mon[37473]: pgmap v8381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14459.796092 secs 2024-07-26T15:28:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14459.796092 secs 2024-07-26T15:28:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14459.796092 secs 2024-07-26T15:28:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:23 smithi134 ceph-mon[31966]: pgmap v8382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:23 smithi189 ceph-mon[36790]: pgmap v8382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:23 smithi193 ceph-mon[37473]: pgmap v8382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:24.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:28:24.950+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:24.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:28:24.951+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:24.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:28:24.951+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14445.372341 secs 2024-07-26T15:28:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:25 smithi134 ceph-mon[31966]: pgmap v8383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14445.372341 secs 2024-07-26T15:28:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:25 smithi189 ceph-mon[36790]: pgmap v8383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14445.372341 secs 2024-07-26T15:28:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:25 smithi193 ceph-mon[37473]: pgmap v8383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:27 smithi134 ceph-mon[31966]: pgmap v8384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14464.796189 secs 2024-07-26T15:28:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:27 smithi189 ceph-mon[36790]: pgmap v8384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14464.796189 secs 2024-07-26T15:28:27.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:27 smithi193 ceph-mon[37473]: pgmap v8384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14464.796189 secs 2024-07-26T15:28:29.410 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:28:29.408+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:29.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:28:29.408+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:29.411 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:28:29.408+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:29.411 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:29 smithi134 ceph-mon[31966]: pgmap v8385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:29 smithi189 ceph-mon[36790]: pgmap v8385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:29 smithi193 ceph-mon[37473]: pgmap v8385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14450.372422 secs 2024-07-26T15:28:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14450.372422 secs 2024-07-26T15:28:30.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14450.372422 secs 2024-07-26T15:28:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:31 smithi134 ceph-mon[31966]: pgmap v8386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:31 smithi189 ceph-mon[36790]: pgmap v8386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:31 smithi193 ceph-mon[37473]: pgmap v8386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14469.796328 secs 2024-07-26T15:28:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14469.796328 secs 2024-07-26T15:28:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14469.796328 secs 2024-07-26T15:28:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:33 smithi134 ceph-mon[31966]: pgmap v8387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:33.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:33 smithi189 ceph-mon[36790]: pgmap v8387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:33 smithi193 ceph-mon[37473]: pgmap v8387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14455.372558 secs 2024-07-26T15:28:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:35 smithi134 ceph-mon[31966]: pgmap v8388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14455.372558 secs 2024-07-26T15:28:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:35 smithi189 ceph-mon[36790]: pgmap v8388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:35.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14455.372558 secs 2024-07-26T15:28:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:35 smithi193 ceph-mon[37473]: pgmap v8388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:37 smithi134 ceph-mon[31966]: pgmap v8389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14474.796447 secs 2024-07-26T15:28:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:37 smithi189 ceph-mon[36790]: pgmap v8389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14474.796447 secs 2024-07-26T15:28:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:37 smithi193 ceph-mon[37473]: pgmap v8389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14474.796447 secs 2024-07-26T15:28:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:39 smithi134 ceph-mon[31966]: pgmap v8390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:39 smithi189 ceph-mon[36790]: pgmap v8390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:39.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:39 smithi193 ceph-mon[37473]: pgmap v8390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:39.954 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:28:39.953+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:39.955 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:28:39.953+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:39.955 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:28:39.953+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14460.372707 secs 2024-07-26T15:28:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14460.372707 secs 2024-07-26T15:28:40.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14460.372707 secs 2024-07-26T15:28:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:41 smithi134 ceph-mon[31966]: pgmap v8391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:41 smithi189 ceph-mon[36790]: pgmap v8391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:41 smithi193 ceph-mon[37473]: pgmap v8391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14479.796564 secs 2024-07-26T15:28:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14479.796564 secs 2024-07-26T15:28:42.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14479.796564 secs 2024-07-26T15:28:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:43 smithi134 ceph-mon[31966]: pgmap v8392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:43 smithi189 ceph-mon[36790]: pgmap v8392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:43.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:43 smithi193 ceph-mon[37473]: pgmap v8392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:44.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:28:44.424+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:44.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:28:44.424+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:44.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:28:44.424+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14465.372770 secs 2024-07-26T15:28:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:45 smithi134 ceph-mon[31966]: pgmap v8393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14465.372770 secs 2024-07-26T15:28:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:45 smithi189 ceph-mon[36790]: pgmap v8393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14465.372770 secs 2024-07-26T15:28:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:45 smithi193 ceph-mon[37473]: pgmap v8393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:47 smithi134 ceph-mon[31966]: pgmap v8394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14484.796653 secs 2024-07-26T15:28:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:47 smithi189 ceph-mon[36790]: pgmap v8394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14484.796653 secs 2024-07-26T15:28:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:47 smithi193 ceph-mon[37473]: pgmap v8394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14484.796653 secs 2024-07-26T15:28:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:49 smithi134 ceph-mon[31966]: pgmap v8395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:49 smithi189 ceph-mon[36790]: pgmap v8395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:49 smithi193 ceph-mon[37473]: pgmap v8395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14470.372894 secs 2024-07-26T15:28:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14470.372894 secs 2024-07-26T15:28:50.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14470.372894 secs 2024-07-26T15:28:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:51 smithi134 ceph-mon[31966]: pgmap v8396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:51 smithi189 ceph-mon[36790]: pgmap v8396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:51 smithi193 ceph-mon[37473]: pgmap v8396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14489.796754 secs 2024-07-26T15:28:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14489.796754 secs 2024-07-26T15:28:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14489.796754 secs 2024-07-26T15:28:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:53 smithi189 ceph-mon[36790]: pgmap v8397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:53 smithi193 ceph-mon[37473]: pgmap v8397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:53 smithi134 ceph-mon[31966]: pgmap v8397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:28:54.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:28:54.969+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:54.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:28:54.969+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:54.970 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:28:54.969+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14475.373029 secs 2024-07-26T15:28:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:55 smithi189 ceph-mon[36790]: pgmap v8398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14475.373029 secs 2024-07-26T15:28:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:55 smithi193 ceph-mon[37473]: pgmap v8398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14475.373029 secs 2024-07-26T15:28:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:55 smithi134 ceph-mon[31966]: pgmap v8398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:28:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:57 smithi134 ceph-mon[31966]: pgmap v8399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:57 smithi189 ceph-mon[36790]: pgmap v8399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:57.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:57 smithi193 ceph-mon[37473]: pgmap v8399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:28:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:58 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14494.796874 secs 2024-07-26T15:28:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:58 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14494.796874 secs 2024-07-26T15:28:58.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:58 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14494.796874 secs 2024-07-26T15:28:59.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:28:59 smithi134 ceph-mon[31966]: pgmap v8400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:59.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:28:59.439+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:59.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:28:59.440+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:28:59.441 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:28:59.440+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:28:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:28:59 smithi189 ceph-mon[36790]: pgmap v8400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:28:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:28:59 smithi193 ceph-mon[37473]: pgmap v8400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14480.373124 secs 2024-07-26T15:29:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14480.373124 secs 2024-07-26T15:29:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14480.373124 secs 2024-07-26T15:29:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:01 smithi134 ceph-mon[31966]: pgmap v8401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:01 smithi189 ceph-mon[36790]: pgmap v8401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:01 smithi193 ceph-mon[37473]: pgmap v8401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:02.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14499.797010 secs 2024-07-26T15:29:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14499.797010 secs 2024-07-26T15:29:02.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14499.797010 secs 2024-07-26T15:29:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:03 smithi134 ceph-mon[31966]: pgmap v8402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:29:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:03 smithi189 ceph-mon[36790]: pgmap v8402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:29:03.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:03 smithi193 ceph-mon[37473]: pgmap v8402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:29:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:29:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:29:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:29:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:04 smithi134 ceph-mon[31966]: pgmap v8403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:29:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:04 smithi134 ceph-mon[31966]: pgmap v8404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:29:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:29:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:29:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:29:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:04 smithi189 ceph-mon[36790]: pgmap v8403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:29:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:04 smithi189 ceph-mon[36790]: pgmap v8404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:29:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:29:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:29:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:29:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:04 smithi193 ceph-mon[37473]: pgmap v8403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:29:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:04 smithi193 ceph-mon[37473]: pgmap v8404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:29:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:29:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14485.373246 secs 2024-07-26T15:29:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14485.373246 secs 2024-07-26T15:29:05.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14485.373246 secs 2024-07-26T15:29:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:06 smithi134 ceph-mon[31966]: pgmap v8405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:29:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:06 smithi189 ceph-mon[36790]: pgmap v8405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:29:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:06 smithi193 ceph-mon[37473]: pgmap v8405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:29:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14504.797102 secs 2024-07-26T15:29:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14504.797102 secs 2024-07-26T15:29:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14504.797102 secs 2024-07-26T15:29:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:08 smithi134 ceph-mon[31966]: pgmap v8406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T15:29:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:08 smithi189 ceph-mon[36790]: pgmap v8406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T15:29:08.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:08 smithi193 ceph-mon[37473]: pgmap v8406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T15:29:09.982 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:29:09.980+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:29:09.982 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:29:09.980+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:29:09.982 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:29:09.980+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14490.373371 secs 2024-07-26T15:29:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:10 smithi134 ceph-mon[31966]: pgmap v8407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:29:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14490.373371 secs 2024-07-26T15:29:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:10 smithi189 ceph-mon[36790]: pgmap v8407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:29:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14490.373371 secs 2024-07-26T15:29:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:10 smithi193 ceph-mon[37473]: pgmap v8407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:29:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14509.797229 secs 2024-07-26T15:29:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:12 smithi134 ceph-mon[31966]: pgmap v8408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:29:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14509.797229 secs 2024-07-26T15:29:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:12 smithi189 ceph-mon[36790]: pgmap v8408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:29:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14509.797229 secs 2024-07-26T15:29:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:12 smithi193 ceph-mon[37473]: pgmap v8408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T15:29:14.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:29:14.455+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:29:14.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:29:14.455+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:29:14.457 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:29:14.455+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:14 smithi134 ceph-mon[31966]: pgmap v8409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:29:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:14 smithi189 ceph-mon[36790]: pgmap v8409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:29:14.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:14 smithi193 ceph-mon[37473]: pgmap v8409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:29:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14495.373499 secs 2024-07-26T15:29:15.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14495.373499 secs 2024-07-26T15:29:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14495.373499 secs 2024-07-26T15:29:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:16 smithi134 ceph-mon[31966]: pgmap v8410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:29:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:16 smithi189 ceph-mon[36790]: pgmap v8410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:29:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:16 smithi193 ceph-mon[37473]: pgmap v8410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:29:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14514.797347 secs 2024-07-26T15:29:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14514.797347 secs 2024-07-26T15:29:17.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14514.797347 secs 2024-07-26T15:29:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:18 smithi134 ceph-mon[31966]: pgmap v8411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:29:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:18 smithi189 ceph-mon[36790]: pgmap v8411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:29:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:18 smithi193 ceph-mon[37473]: pgmap v8411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:29:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14500.373641 secs 2024-07-26T15:29:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:20 smithi134 ceph-mon[31966]: pgmap v8412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:29:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14500.373641 secs 2024-07-26T15:29:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:20 smithi189 ceph-mon[36790]: pgmap v8412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:29:20.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14500.373641 secs 2024-07-26T15:29:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:20 smithi193 ceph-mon[37473]: pgmap v8412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:29:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14519.797444 secs 2024-07-26T15:29:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:22 smithi134 ceph-mon[31966]: pgmap v8413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14519.797444 secs 2024-07-26T15:29:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:22 smithi189 ceph-mon[36790]: pgmap v8413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14519.797444 secs 2024-07-26T15:29:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:22 smithi193 ceph-mon[37473]: pgmap v8413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:24 smithi134 ceph-mon[31966]: pgmap v8414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:24 smithi189 ceph-mon[36790]: pgmap v8414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:24 smithi193 ceph-mon[37473]: pgmap v8414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:29:24.981+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:29:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:29:24.981+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:24.983 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:29:24.981+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:29:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14505.373765 secs 2024-07-26T15:29:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14505.373765 secs 2024-07-26T15:29:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14505.373765 secs 2024-07-26T15:29:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:26 smithi134 ceph-mon[31966]: pgmap v8415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:26 smithi189 ceph-mon[36790]: pgmap v8415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:26 smithi193 ceph-mon[37473]: pgmap v8415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14524.797541 secs 2024-07-26T15:29:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14524.797541 secs 2024-07-26T15:29:27.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14524.797541 secs 2024-07-26T15:29:28.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:28 smithi134 ceph-mon[31966]: pgmap v8416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:29:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:28 smithi189 ceph-mon[36790]: pgmap v8416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:29:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:28 smithi193 ceph-mon[37473]: pgmap v8416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:29:29.472 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:29:29.471+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:29:29.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:29:29.471+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:29:29.473 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:29:29.471+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14510.373893 secs 2024-07-26T15:29:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:30 smithi134 ceph-mon[31966]: pgmap v8417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:29:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14510.373893 secs 2024-07-26T15:29:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:30 smithi189 ceph-mon[36790]: pgmap v8417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:29:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14510.373893 secs 2024-07-26T15:29:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:30 smithi193 ceph-mon[37473]: pgmap v8417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:29:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14529.797611 secs 2024-07-26T15:29:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:32 smithi134 ceph-mon[31966]: pgmap v8418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14529.797611 secs 2024-07-26T15:29:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:32 smithi189 ceph-mon[36790]: pgmap v8418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:32.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14529.797611 secs 2024-07-26T15:29:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:32 smithi193 ceph-mon[37473]: pgmap v8418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:34 smithi134 ceph-mon[31966]: pgmap v8419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:34 smithi189 ceph-mon[36790]: pgmap v8419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:34 smithi193 ceph-mon[37473]: pgmap v8419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14515.374016 secs 2024-07-26T15:29:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14515.374016 secs 2024-07-26T15:29:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14515.374016 secs 2024-07-26T15:29:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:36 smithi189 ceph-mon[36790]: pgmap v8420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:36.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:36 smithi193 ceph-mon[37473]: pgmap v8420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:36 smithi134 ceph-mon[31966]: pgmap v8420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14534.797694 secs 2024-07-26T15:29:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14534.797694 secs 2024-07-26T15:29:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14534.797694 secs 2024-07-26T15:29:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:38 smithi189 ceph-mon[36790]: pgmap v8421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:29:38.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:38 smithi193 ceph-mon[37473]: pgmap v8421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:29:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:38 smithi134 ceph-mon[31966]: pgmap v8421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:29:39.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:29:39.985+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:29:39.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:29:39.985+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:29:39.986 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:29:39.985+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14520.374116 secs 2024-07-26T15:29:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:40 smithi193 ceph-mon[37473]: pgmap v8422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:29:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14520.374116 secs 2024-07-26T15:29:40.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:40 smithi134 ceph-mon[31966]: pgmap v8422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:29:40.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14520.374116 secs 2024-07-26T15:29:40.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:40 smithi189 ceph-mon[36790]: pgmap v8422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:29:42.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14539.797780 secs 2024-07-26T15:29:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:42 smithi193 ceph-mon[37473]: pgmap v8423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14539.797780 secs 2024-07-26T15:29:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:42 smithi134 ceph-mon[31966]: pgmap v8423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:42.947 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14539.797780 secs 2024-07-26T15:29:42.947 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:42 smithi189 ceph-mon[36790]: pgmap v8423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:44.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:29:44.482+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:29:44.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:29:44.482+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:44.484 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:29:44.482+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:29:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:44 smithi193 ceph-mon[37473]: pgmap v8424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:44 smithi134 ceph-mon[31966]: pgmap v8424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:44 smithi189 ceph-mon[36790]: pgmap v8424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:45.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14525.374238 secs 2024-07-26T15:29:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14525.374238 secs 2024-07-26T15:29:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14525.374238 secs 2024-07-26T15:29:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:46 smithi193 ceph-mon[37473]: pgmap v8425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:46 smithi134 ceph-mon[31966]: pgmap v8425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:46 smithi189 ceph-mon[36790]: pgmap v8425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:47.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14544.797870 secs 2024-07-26T15:29:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14544.797870 secs 2024-07-26T15:29:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14544.797870 secs 2024-07-26T15:29:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:48 smithi193 ceph-mon[37473]: pgmap v8426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:29:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:48 smithi134 ceph-mon[31966]: pgmap v8426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:29:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:48 smithi189 ceph-mon[36790]: pgmap v8426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:29:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14530.374395 secs 2024-07-26T15:29:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:50 smithi193 ceph-mon[37473]: pgmap v8427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:29:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14530.374395 secs 2024-07-26T15:29:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:50 smithi134 ceph-mon[31966]: pgmap v8427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:29:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14530.374395 secs 2024-07-26T15:29:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:50 smithi189 ceph-mon[36790]: pgmap v8427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:29:52.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14549.798014 secs 2024-07-26T15:29:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:52 smithi193 ceph-mon[37473]: pgmap v8428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14549.798014 secs 2024-07-26T15:29:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:52 smithi134 ceph-mon[31966]: pgmap v8428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14549.798014 secs 2024-07-26T15:29:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:52 smithi189 ceph-mon[36790]: pgmap v8428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:54 smithi193 ceph-mon[37473]: pgmap v8429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14535.374524 secs 2024-07-26T15:29:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:54 smithi134 ceph-mon[31966]: pgmap v8429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14535.374524 secs 2024-07-26T15:29:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:54 smithi189 ceph-mon[36790]: pgmap v8429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:29:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14535.374524 secs 2024-07-26T15:29:54.996 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:29:54.995+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:29:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:29:54.995+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:54.997 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:29:54.995+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:29:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:56 smithi193 ceph-mon[37473]: pgmap v8430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:56 smithi134 ceph-mon[31966]: pgmap v8430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:56 smithi189 ceph-mon[36790]: pgmap v8430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:29:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14554.798134 secs 2024-07-26T15:29:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14554.798134 secs 2024-07-26T15:29:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14554.798134 secs 2024-07-26T15:29:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:58 smithi193 ceph-mon[37473]: pgmap v8431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:29:58.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:58 smithi134 ceph-mon[31966]: pgmap v8431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:29:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:58 smithi189 ceph-mon[36790]: pgmap v8431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:29:59.499 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:29:59.498+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:29:59.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:29:59.498+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:29:59.500 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:29:59.498+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:29:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:29:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14540.374666 secs 2024-07-26T15:29:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:29:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14540.374666 secs 2024-07-26T15:30:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:29:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14540.374666 secs 2024-07-26T15:30:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:00 smithi134 ceph-mon[31966]: pgmap v8432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:30:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:30:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:00 smithi189 ceph-mon[36790]: pgmap v8432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:30:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:30:01.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:00 smithi193 ceph-mon[37473]: pgmap v8432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:30:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:30:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14559.798269 secs 2024-07-26T15:30:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:02 smithi134 ceph-mon[31966]: pgmap v8433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14559.798269 secs 2024-07-26T15:30:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:02 smithi189 ceph-mon[36790]: pgmap v8433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14559.798269 secs 2024-07-26T15:30:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:02 smithi193 ceph-mon[37473]: pgmap v8433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:03.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:30:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:30:04.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:30:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:04 smithi134 ceph-mon[31966]: pgmap v8434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14545.374777 secs 2024-07-26T15:30:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:30:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:30:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:04 smithi189 ceph-mon[36790]: pgmap v8434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14545.374777 secs 2024-07-26T15:30:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:30:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:30:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:04 smithi193 ceph-mon[37473]: pgmap v8434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14545.374777 secs 2024-07-26T15:30:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:30:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:30:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:30:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:30:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:30:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:30:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:30:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:30:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:30:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:30:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:30:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:06 smithi134 ceph-mon[31966]: pgmap v8435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T15:30:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:06 smithi134 ceph-mon[31966]: pgmap v8436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:30:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:06 smithi189 ceph-mon[36790]: pgmap v8435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T15:30:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:06 smithi189 ceph-mon[36790]: pgmap v8436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:30:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:06 smithi193 ceph-mon[37473]: pgmap v8435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T15:30:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:06 smithi193 ceph-mon[37473]: pgmap v8436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:30:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14564.798421 secs 2024-07-26T15:30:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14564.798421 secs 2024-07-26T15:30:08.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14564.798421 secs 2024-07-26T15:30:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:08 smithi134 ceph-mon[31966]: pgmap v8437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:30:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:08 smithi189 ceph-mon[36790]: pgmap v8437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:30:09.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:08 smithi193 ceph-mon[37473]: pgmap v8437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:30:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14550.374923 secs 2024-07-26T15:30:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14550.374923 secs 2024-07-26T15:30:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:30:10.011+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:30:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:30:10.011+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:30:10.011+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:30:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14550.374923 secs 2024-07-26T15:30:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:10 smithi134 ceph-mon[31966]: pgmap v8438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:30:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:10 smithi189 ceph-mon[36790]: pgmap v8438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:30:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:10 smithi193 ceph-mon[37473]: pgmap v8438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:30:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:12 smithi134 ceph-mon[31966]: pgmap v8439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:30:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14569.798485 secs 2024-07-26T15:30:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:12 smithi189 ceph-mon[36790]: pgmap v8439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:30:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14569.798485 secs 2024-07-26T15:30:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:12 smithi193 ceph-mon[37473]: pgmap v8439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:30:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14569.798485 secs 2024-07-26T15:30:14.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:30:14.504+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:30:14.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:30:14.504+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:14.507 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:30:14.504+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:30:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:14 smithi193 ceph-mon[37473]: pgmap v8440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:30:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14555.375048 secs 2024-07-26T15:30:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:14 smithi134 ceph-mon[31966]: pgmap v8440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:30:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14555.375048 secs 2024-07-26T15:30:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:14 smithi189 ceph-mon[36790]: pgmap v8440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:30:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14555.375048 secs 2024-07-26T15:30:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:16 smithi134 ceph-mon[31966]: pgmap v8441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:30:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:16 smithi189 ceph-mon[36790]: pgmap v8441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:30:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:16 smithi193 ceph-mon[37473]: pgmap v8441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:30:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14574.798609 secs 2024-07-26T15:30:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14574.798609 secs 2024-07-26T15:30:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14574.798609 secs 2024-07-26T15:30:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:18 smithi134 ceph-mon[31966]: pgmap v8442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:18 smithi189 ceph-mon[36790]: pgmap v8442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:18 smithi193 ceph-mon[37473]: pgmap v8442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14560.375150 secs 2024-07-26T15:30:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14560.375150 secs 2024-07-26T15:30:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14560.375150 secs 2024-07-26T15:30:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:20 smithi134 ceph-mon[31966]: pgmap v8443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:20 smithi189 ceph-mon[36790]: pgmap v8443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:21.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:20 smithi193 ceph-mon[37473]: pgmap v8443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:22 smithi134 ceph-mon[31966]: pgmap v8444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14579.798693 secs 2024-07-26T15:30:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:22 smithi189 ceph-mon[36790]: pgmap v8444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14579.798693 secs 2024-07-26T15:30:23.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:22 smithi193 ceph-mon[37473]: pgmap v8444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14579.798693 secs 2024-07-26T15:30:24.863 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:30:24.862+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:30:24.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:30:24.862+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:30:24.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:30:24.862+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T15:30:24.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:30:24.863+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 114 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T15:30:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:24 smithi134 ceph-mon[31966]: pgmap v8445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14565.375267 secs 2024-07-26T15:30:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:24 smithi189 ceph-mon[36790]: pgmap v8445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14565.375267 secs 2024-07-26T15:30:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:24 smithi193 ceph-mon[37473]: pgmap v8445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14565.375267 secs 2024-07-26T15:30:25.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:30:25.026+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:30:25.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:30:25.026+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:25.027 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:30:25.026+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:30:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:26 smithi134 ceph-mon[31966]: pgmap v8446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:26 smithi189 ceph-mon[36790]: pgmap v8446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:27.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:26 smithi193 ceph-mon[37473]: pgmap v8446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14584.798816 secs 2024-07-26T15:30:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14584.798816 secs 2024-07-26T15:30:28.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14584.798816 secs 2024-07-26T15:30:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:28 smithi134 ceph-mon[31966]: pgmap v8447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:28 smithi189 ceph-mon[36790]: pgmap v8447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:28 smithi193 ceph-mon[37473]: pgmap v8447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:29.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:30:29.506+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:30:29.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:30:29.506+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:29.508 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:30:29.506+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:30:29.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14570.375403 secs 2024-07-26T15:30:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14570.375403 secs 2024-07-26T15:30:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14570.375403 secs 2024-07-26T15:30:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:30 smithi189 ceph-mon[36790]: pgmap v8448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:30 smithi193 ceph-mon[37473]: pgmap v8448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:30 smithi134 ceph-mon[31966]: pgmap v8448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:32 smithi189 ceph-mon[36790]: pgmap v8449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14589.798937 secs 2024-07-26T15:30:33.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:32 smithi193 ceph-mon[37473]: pgmap v8449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:33.027 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14589.798937 secs 2024-07-26T15:30:33.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:32 smithi134 ceph-mon[31966]: pgmap v8449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:33.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14589.798937 secs 2024-07-26T15:30:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:34 smithi193 ceph-mon[37473]: pgmap v8450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14575.375530 secs 2024-07-26T15:30:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:34 smithi134 ceph-mon[31966]: pgmap v8450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14575.375530 secs 2024-07-26T15:30:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:34 smithi189 ceph-mon[36790]: pgmap v8450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14575.375530 secs 2024-07-26T15:30:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:30:35.403+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:30:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:30:35.403+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:30:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:30:35.404+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:30:35.405 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:30:35.404+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:30:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:36 smithi193 ceph-mon[37473]: pgmap v8451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:36 smithi134 ceph-mon[31966]: pgmap v8451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:36 smithi189 ceph-mon[36790]: pgmap v8451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14594.799039 secs 2024-07-26T15:30:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14594.799039 secs 2024-07-26T15:30:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14594.799039 secs 2024-07-26T15:30:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:38 smithi193 ceph-mon[37473]: pgmap v8452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:38 smithi134 ceph-mon[31966]: pgmap v8452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:38 smithi189 ceph-mon[36790]: pgmap v8452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14580.375680 secs 2024-07-26T15:30:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:30:40.040+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:30:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:30:40.040+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:30:40.040+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:30:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14580.375680 secs 2024-07-26T15:30:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14580.375680 secs 2024-07-26T15:30:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:40 smithi193 ceph-mon[37473]: pgmap v8453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:40 smithi134 ceph-mon[31966]: pgmap v8453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:40 smithi189 ceph-mon[36790]: pgmap v8453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:42 smithi193 ceph-mon[37473]: pgmap v8454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14599.799182 secs 2024-07-26T15:30:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:42 smithi134 ceph-mon[31966]: pgmap v8454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14599.799182 secs 2024-07-26T15:30:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:42 smithi189 ceph-mon[36790]: pgmap v8454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14599.799182 secs 2024-07-26T15:30:44.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:30:44.515+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:30:44.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:30:44.515+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:30:44.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:30:44.515+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:44 smithi134 ceph-mon[31966]: pgmap v8455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14585.375802 secs 2024-07-26T15:30:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:44 smithi193 ceph-mon[37473]: pgmap v8455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14585.375802 secs 2024-07-26T15:30:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:44 smithi189 ceph-mon[36790]: pgmap v8455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14585.375802 secs 2024-07-26T15:30:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:46 smithi193 ceph-mon[37473]: pgmap v8456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:46 smithi134 ceph-mon[31966]: pgmap v8456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:46 smithi189 ceph-mon[36790]: pgmap v8456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14604.799320 secs 2024-07-26T15:30:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14604.799320 secs 2024-07-26T15:30:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14604.799320 secs 2024-07-26T15:30:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:48 smithi193 ceph-mon[37473]: pgmap v8457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:48 smithi134 ceph-mon[31966]: pgmap v8457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:48 smithi189 ceph-mon[36790]: pgmap v8457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14590.375917 secs 2024-07-26T15:30:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14590.375917 secs 2024-07-26T15:30:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14590.375917 secs 2024-07-26T15:30:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:50 smithi134 ceph-mon[31966]: pgmap v8458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:50 smithi189 ceph-mon[36790]: pgmap v8458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:50 smithi193 ceph-mon[37473]: pgmap v8458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:30:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:52 smithi134 ceph-mon[31966]: pgmap v8459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14609.799421 secs 2024-07-26T15:30:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:52 smithi189 ceph-mon[36790]: pgmap v8459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14609.799421 secs 2024-07-26T15:30:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:52 smithi193 ceph-mon[37473]: pgmap v8459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14609.799421 secs 2024-07-26T15:30:55.043 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:30:55.042+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:30:55.044 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:30:55.042+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:55.044 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:30:55.042+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:30:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:54 smithi134 ceph-mon[31966]: pgmap v8460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14595.376034 secs 2024-07-26T15:30:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:54 smithi189 ceph-mon[36790]: pgmap v8460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14595.376034 secs 2024-07-26T15:30:55.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:54 smithi193 ceph-mon[37473]: pgmap v8460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:30:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14595.376034 secs 2024-07-26T15:30:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:56 smithi134 ceph-mon[31966]: pgmap v8461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:56 smithi189 ceph-mon[36790]: pgmap v8461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:56 smithi193 ceph-mon[37473]: pgmap v8461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:30:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14614.799562 secs 2024-07-26T15:30:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14614.799562 secs 2024-07-26T15:30:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14614.799562 secs 2024-07-26T15:30:59.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:58 smithi134 ceph-mon[31966]: pgmap v8462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:58 smithi189 ceph-mon[36790]: pgmap v8462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:58 smithi193 ceph-mon[37473]: pgmap v8462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:30:59.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:30:59.531+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:30:59.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:30:59.531+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:30:59.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:30:59.531+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:30:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14600.376154 secs 2024-07-26T15:31:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:30:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14600.376154 secs 2024-07-26T15:31:00.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:30:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14600.376154 secs 2024-07-26T15:31:01.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:00 smithi134 ceph-mon[31966]: pgmap v8463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:00 smithi189 ceph-mon[36790]: pgmap v8463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:00 smithi193 ceph-mon[37473]: pgmap v8463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:02 smithi134 ceph-mon[31966]: pgmap v8464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14619.799627 secs 2024-07-26T15:31:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:02 smithi189 ceph-mon[36790]: pgmap v8464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14619.799627 secs 2024-07-26T15:31:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:02 smithi193 ceph-mon[37473]: pgmap v8464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14619.799627 secs 2024-07-26T15:31:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:04 smithi134 ceph-mon[31966]: pgmap v8465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14605.376278 secs 2024-07-26T15:31:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:04 smithi189 ceph-mon[36790]: pgmap v8465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14605.376278 secs 2024-07-26T15:31:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:04 smithi193 ceph-mon[37473]: pgmap v8465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14605.376278 secs 2024-07-26T15:31:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:31:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:06.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:06.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:06.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:05 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:31:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:05 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:31:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:05 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:07.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:06 smithi134 ceph-mon[31966]: pgmap v8466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:06 smithi189 ceph-mon[36790]: pgmap v8466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:06 smithi193 ceph-mon[37473]: pgmap v8466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14624.799777 secs 2024-07-26T15:31:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14624.799777 secs 2024-07-26T15:31:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14624.799777 secs 2024-07-26T15:31:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:09.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:08 smithi134 ceph-mon[31966]: pgmap v8467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:08 smithi189 ceph-mon[36790]: pgmap v8467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:08 smithi193 ceph-mon[37473]: pgmap v8467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:10.059 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:31:10.057+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:10.059 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:31:10.057+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:10.059 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:31:10.057+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14610.376359 secs 2024-07-26T15:31:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:09 smithi193 ceph-mon[37473]: pgmap v8468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14610.376359 secs 2024-07-26T15:31:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:09 smithi134 ceph-mon[31966]: pgmap v8468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14610.376359 secs 2024-07-26T15:31:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:09 smithi189 ceph-mon[36790]: pgmap v8468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:31:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:31:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:11 smithi193 ceph-mon[37473]: pgmap v8469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:11 smithi193 ceph-mon[37473]: pgmap v8470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:31:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:31:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:31:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:11 smithi134 ceph-mon[31966]: pgmap v8469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:11 smithi134 ceph-mon[31966]: pgmap v8470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:31:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:31:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:31:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:11 smithi189 ceph-mon[36790]: pgmap v8469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:11 smithi189 ceph-mon[36790]: pgmap v8470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:31:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:31:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14629.799895 secs 2024-07-26T15:31:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14629.799895 secs 2024-07-26T15:31:12.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14629.799895 secs 2024-07-26T15:31:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:13 smithi134 ceph-mon[31966]: pgmap v8471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T15:31:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:13 smithi189 ceph-mon[36790]: pgmap v8471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T15:31:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:13 smithi193 ceph-mon[37473]: pgmap v8471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T15:31:14.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:31:14.535+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:14.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:31:14.535+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:14.537 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:31:14.535+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14615.376479 secs 2024-07-26T15:31:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:15 smithi134 ceph-mon[31966]: pgmap v8472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:31:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14615.376479 secs 2024-07-26T15:31:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:15 smithi189 ceph-mon[36790]: pgmap v8472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:31:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14615.376479 secs 2024-07-26T15:31:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:15 smithi193 ceph-mon[37473]: pgmap v8472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:31:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:17 smithi134 ceph-mon[31966]: pgmap v8473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T15:31:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14634.800056 secs 2024-07-26T15:31:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:17 smithi189 ceph-mon[36790]: pgmap v8473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T15:31:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14634.800056 secs 2024-07-26T15:31:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:17 smithi193 ceph-mon[37473]: pgmap v8473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T15:31:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14634.800056 secs 2024-07-26T15:31:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:19 smithi134 ceph-mon[31966]: pgmap v8474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:31:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:19 smithi189 ceph-mon[36790]: pgmap v8474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:31:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:19 smithi193 ceph-mon[37473]: pgmap v8474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:31:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14620.376621 secs 2024-07-26T15:31:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14620.376621 secs 2024-07-26T15:31:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14620.376621 secs 2024-07-26T15:31:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:21 smithi134 ceph-mon[31966]: pgmap v8475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:31:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:21 smithi189 ceph-mon[36790]: pgmap v8475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:31:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:21 smithi193 ceph-mon[37473]: pgmap v8475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:31:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14639.800122 secs 2024-07-26T15:31:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14639.800122 secs 2024-07-26T15:31:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14639.800122 secs 2024-07-26T15:31:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:23 smithi134 ceph-mon[31966]: pgmap v8476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:23 smithi189 ceph-mon[36790]: pgmap v8476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:23.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:23 smithi193 ceph-mon[37473]: pgmap v8476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:31:25.072+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:31:25.073+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:31:25.073+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14625.376738 secs 2024-07-26T15:31:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:25 smithi134 ceph-mon[31966]: pgmap v8477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14625.376738 secs 2024-07-26T15:31:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:25 smithi189 ceph-mon[36790]: pgmap v8477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14625.376738 secs 2024-07-26T15:31:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:25 smithi193 ceph-mon[37473]: pgmap v8477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:27 smithi134 ceph-mon[31966]: pgmap v8478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14644.800259 secs 2024-07-26T15:31:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:27 smithi189 ceph-mon[36790]: pgmap v8478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14644.800259 secs 2024-07-26T15:31:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:27 smithi193 ceph-mon[37473]: pgmap v8478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14644.800259 secs 2024-07-26T15:31:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:29 smithi134 ceph-mon[31966]: pgmap v8479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:29 smithi189 ceph-mon[36790]: pgmap v8479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:29 smithi193 ceph-mon[37473]: pgmap v8479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:31:29.551+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:31:29.551+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:31:29.551+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14630.376876 secs 2024-07-26T15:31:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14630.376876 secs 2024-07-26T15:31:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14630.376876 secs 2024-07-26T15:31:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:31 smithi189 ceph-mon[36790]: pgmap v8480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:31 smithi193 ceph-mon[37473]: pgmap v8480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:31 smithi134 ceph-mon[31966]: pgmap v8480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14649.800380 secs 2024-07-26T15:31:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14649.800380 secs 2024-07-26T15:31:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14649.800380 secs 2024-07-26T15:31:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:33 smithi193 ceph-mon[37473]: pgmap v8481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:33 smithi134 ceph-mon[31966]: pgmap v8481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:33 smithi189 ceph-mon[36790]: pgmap v8481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14635.376967 secs 2024-07-26T15:31:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:35 smithi193 ceph-mon[37473]: pgmap v8482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14635.376967 secs 2024-07-26T15:31:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:35 smithi134 ceph-mon[31966]: pgmap v8482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14635.376967 secs 2024-07-26T15:31:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:35 smithi189 ceph-mon[36790]: pgmap v8482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:37 smithi193 ceph-mon[37473]: pgmap v8483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14654.800472 secs 2024-07-26T15:31:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:37 smithi134 ceph-mon[31966]: pgmap v8483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14654.800472 secs 2024-07-26T15:31:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:37 smithi189 ceph-mon[36790]: pgmap v8483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14654.800472 secs 2024-07-26T15:31:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:39 smithi193 ceph-mon[37473]: pgmap v8484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:39 smithi134 ceph-mon[31966]: pgmap v8484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:39 smithi189 ceph-mon[36790]: pgmap v8484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:31:40.078+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:31:40.078+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:31:40.078+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14640.377050 secs 2024-07-26T15:31:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14640.377050 secs 2024-07-26T15:31:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14640.377050 secs 2024-07-26T15:31:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:41 smithi193 ceph-mon[37473]: pgmap v8485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:41 smithi134 ceph-mon[31966]: pgmap v8485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:41 smithi189 ceph-mon[36790]: pgmap v8485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14659.800588 secs 2024-07-26T15:31:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14659.800588 secs 2024-07-26T15:31:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14659.800588 secs 2024-07-26T15:31:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:43 smithi193 ceph-mon[37473]: pgmap v8486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:43 smithi134 ceph-mon[31966]: pgmap v8486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:43 smithi189 ceph-mon[36790]: pgmap v8486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:44.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:31:44.552+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:44.554 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:31:44.552+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:44.555 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:31:44.552+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14645.377117 secs 2024-07-26T15:31:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:45 smithi193 ceph-mon[37473]: pgmap v8487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14645.377117 secs 2024-07-26T15:31:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:45 smithi134 ceph-mon[31966]: pgmap v8487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14645.377117 secs 2024-07-26T15:31:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:45 smithi189 ceph-mon[36790]: pgmap v8487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:47 smithi193 ceph-mon[37473]: pgmap v8488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14664.800673 secs 2024-07-26T15:31:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:47 smithi134 ceph-mon[31966]: pgmap v8488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14664.800673 secs 2024-07-26T15:31:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:47 smithi189 ceph-mon[36790]: pgmap v8488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14664.800673 secs 2024-07-26T15:31:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:49 smithi193 ceph-mon[37473]: pgmap v8489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:49 smithi134 ceph-mon[31966]: pgmap v8489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:49 smithi189 ceph-mon[36790]: pgmap v8489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14650.377197 secs 2024-07-26T15:31:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14650.377197 secs 2024-07-26T15:31:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14650.377197 secs 2024-07-26T15:31:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:51 smithi134 ceph-mon[31966]: pgmap v8490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:51 smithi189 ceph-mon[36790]: pgmap v8490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:51 smithi193 ceph-mon[37473]: pgmap v8490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14669.800792 secs 2024-07-26T15:31:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14669.800792 secs 2024-07-26T15:31:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14669.800792 secs 2024-07-26T15:31:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:53 smithi134 ceph-mon[31966]: pgmap v8491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:53 smithi189 ceph-mon[36790]: pgmap v8491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:53 smithi193 ceph-mon[37473]: pgmap v8491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:31:55.095 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:31:55.094+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:31:55.094+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:55.096 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:31:55.094+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14655.377291 secs 2024-07-26T15:31:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:55 smithi134 ceph-mon[31966]: pgmap v8492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14655.377291 secs 2024-07-26T15:31:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:55 smithi189 ceph-mon[36790]: pgmap v8492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14655.377291 secs 2024-07-26T15:31:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:55 smithi193 ceph-mon[37473]: pgmap v8492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:31:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:57 smithi134 ceph-mon[31966]: pgmap v8493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14674.800908 secs 2024-07-26T15:31:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:57 smithi189 ceph-mon[36790]: pgmap v8493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14674.800908 secs 2024-07-26T15:31:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:57 smithi193 ceph-mon[37473]: pgmap v8493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:31:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14674.800908 secs 2024-07-26T15:31:59.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:31:59 smithi134 ceph-mon[31966]: pgmap v8494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:59.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:31:59.567+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:59.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:31:59.568+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:31:59.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:31:59.568+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:31:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:31:59 smithi189 ceph-mon[36790]: pgmap v8494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:31:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:31:59 smithi193 ceph-mon[37473]: pgmap v8494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14660.377413 secs 2024-07-26T15:32:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14660.377413 secs 2024-07-26T15:32:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14660.377413 secs 2024-07-26T15:32:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:01 smithi134 ceph-mon[31966]: pgmap v8495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:01 smithi189 ceph-mon[36790]: pgmap v8495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:01 smithi193 ceph-mon[37473]: pgmap v8495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14679.801007 secs 2024-07-26T15:32:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14679.801007 secs 2024-07-26T15:32:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14679.801007 secs 2024-07-26T15:32:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:03 smithi134 ceph-mon[31966]: pgmap v8496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:03 smithi189 ceph-mon[36790]: pgmap v8496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:03 smithi193 ceph-mon[37473]: pgmap v8496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14665.377549 secs 2024-07-26T15:32:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:05 smithi134 ceph-mon[31966]: pgmap v8497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14665.377549 secs 2024-07-26T15:32:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:05 smithi189 ceph-mon[36790]: pgmap v8497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14665.377549 secs 2024-07-26T15:32:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:05 smithi193 ceph-mon[37473]: pgmap v8497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:07 smithi134 ceph-mon[31966]: pgmap v8498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14684.801128 secs 2024-07-26T15:32:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:07 smithi189 ceph-mon[36790]: pgmap v8498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14684.801128 secs 2024-07-26T15:32:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:07 smithi193 ceph-mon[37473]: pgmap v8498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14684.801128 secs 2024-07-26T15:32:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:09 smithi134 ceph-mon[31966]: pgmap v8499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:09 smithi189 ceph-mon[36790]: pgmap v8499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:09 smithi193 ceph-mon[37473]: pgmap v8499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:10.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:32:10.096+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:32:10.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:32:10.096+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:32:10.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:32:10.096+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:10.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14670.377687 secs 2024-07-26T15:32:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:32:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14670.377687 secs 2024-07-26T15:32:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:32:10.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14670.377687 secs 2024-07-26T15:32:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:32:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:11 smithi134 ceph-mon[31966]: pgmap v8500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:32:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:32:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:32:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:11 smithi189 ceph-mon[36790]: pgmap v8500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:32:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:32:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:32:11.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:11 smithi193 ceph-mon[37473]: pgmap v8500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:32:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:32:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:32:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14689.801189 secs 2024-07-26T15:32:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:12 smithi134 ceph-mon[31966]: pgmap v8501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:32:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:12 smithi134 ceph-mon[31966]: pgmap v8502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:32:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14689.801189 secs 2024-07-26T15:32:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:12 smithi189 ceph-mon[36790]: pgmap v8501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:32:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:12 smithi189 ceph-mon[36790]: pgmap v8502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:32:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14689.801189 secs 2024-07-26T15:32:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:12 smithi193 ceph-mon[37473]: pgmap v8501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:32:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:12 smithi193 ceph-mon[37473]: pgmap v8502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:32:14.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:32:14.576+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:32:14.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:32:14.576+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:14.578 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:32:14.576+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:32:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:14 smithi134 ceph-mon[31966]: pgmap v8503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:14 smithi189 ceph-mon[36790]: pgmap v8503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:14 smithi193 ceph-mon[37473]: pgmap v8503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14675.377806 secs 2024-07-26T15:32:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14675.377806 secs 2024-07-26T15:32:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14675.377806 secs 2024-07-26T15:32:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:16 smithi134 ceph-mon[31966]: pgmap v8504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:16 smithi189 ceph-mon[36790]: pgmap v8504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:16 smithi193 ceph-mon[37473]: pgmap v8504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14694.801330 secs 2024-07-26T15:32:17.674 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14694.801330 secs 2024-07-26T15:32:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14694.801330 secs 2024-07-26T15:32:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:18 smithi134 ceph-mon[31966]: pgmap v8505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T15:32:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:18 smithi189 ceph-mon[36790]: pgmap v8505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T15:32:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:18 smithi193 ceph-mon[37473]: pgmap v8505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T15:32:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14680.377959 secs 2024-07-26T15:32:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:20 smithi134 ceph-mon[31966]: pgmap v8506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14680.377959 secs 2024-07-26T15:32:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:20 smithi189 ceph-mon[36790]: pgmap v8506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14680.377959 secs 2024-07-26T15:32:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:20 smithi193 ceph-mon[37473]: pgmap v8506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:32:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14699.801446 secs 2024-07-26T15:32:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:22 smithi134 ceph-mon[31966]: pgmap v8507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:32:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14699.801446 secs 2024-07-26T15:32:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:22 smithi189 ceph-mon[36790]: pgmap v8507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:32:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14699.801446 secs 2024-07-26T15:32:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:22 smithi193 ceph-mon[37473]: pgmap v8507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:32:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:24 smithi189 ceph-mon[36790]: pgmap v8508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:24 smithi193 ceph-mon[37473]: pgmap v8508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:24 smithi134 ceph-mon[31966]: pgmap v8508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:25.113 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:32:25.112+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:32:25.113 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:32:25.112+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:32:25.113 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:32:25.112+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14685.377963 secs 2024-07-26T15:32:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14685.377963 secs 2024-07-26T15:32:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14685.377963 secs 2024-07-26T15:32:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:26 smithi189 ceph-mon[36790]: pgmap v8509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:26 smithi193 ceph-mon[37473]: pgmap v8509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:26 smithi134 ceph-mon[31966]: pgmap v8509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14704.801560 secs 2024-07-26T15:32:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14704.801560 secs 2024-07-26T15:32:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14704.801560 secs 2024-07-26T15:32:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:28 smithi193 ceph-mon[37473]: pgmap v8510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:28 smithi134 ceph-mon[31966]: pgmap v8510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:28 smithi189 ceph-mon[36790]: pgmap v8510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:29.593 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:32:29.592+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:32:29.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:32:29.592+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:32:29.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:32:29.592+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14690.378020 secs 2024-07-26T15:32:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:30 smithi193 ceph-mon[37473]: pgmap v8511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14690.378020 secs 2024-07-26T15:32:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:30 smithi134 ceph-mon[31966]: pgmap v8511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14690.378020 secs 2024-07-26T15:32:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:30 smithi189 ceph-mon[36790]: pgmap v8511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:32:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14709.801689 secs 2024-07-26T15:32:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:32 smithi193 ceph-mon[37473]: pgmap v8512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14709.801689 secs 2024-07-26T15:32:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:32 smithi134 ceph-mon[31966]: pgmap v8512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14709.801689 secs 2024-07-26T15:32:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:32 smithi189 ceph-mon[36790]: pgmap v8512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:34 smithi193 ceph-mon[37473]: pgmap v8513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:34 smithi134 ceph-mon[31966]: pgmap v8513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:34 smithi189 ceph-mon[36790]: pgmap v8513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14695.378123 secs 2024-07-26T15:32:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14695.378123 secs 2024-07-26T15:32:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14695.378123 secs 2024-07-26T15:32:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:36 smithi193 ceph-mon[37473]: pgmap v8514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:36 smithi134 ceph-mon[31966]: pgmap v8514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:36 smithi189 ceph-mon[36790]: pgmap v8514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14714.801804 secs 2024-07-26T15:32:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14714.801804 secs 2024-07-26T15:32:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14714.801804 secs 2024-07-26T15:32:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:38 smithi193 ceph-mon[37473]: pgmap v8515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:38 smithi134 ceph-mon[31966]: pgmap v8515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:38 smithi189 ceph-mon[36790]: pgmap v8515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:40.128 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:32:40.127+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:32:40.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:32:40.127+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:32:40.129 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:32:40.127+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14700.378242 secs 2024-07-26T15:32:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:40 smithi193 ceph-mon[37473]: pgmap v8516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:32:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14700.378242 secs 2024-07-26T15:32:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:40 smithi134 ceph-mon[31966]: pgmap v8516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:32:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14700.378242 secs 2024-07-26T15:32:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:40 smithi189 ceph-mon[36790]: pgmap v8516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:32:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14719.801897 secs 2024-07-26T15:32:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:42 smithi193 ceph-mon[37473]: pgmap v8517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:32:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14719.801897 secs 2024-07-26T15:32:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:42 smithi134 ceph-mon[31966]: pgmap v8517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:32:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14719.801897 secs 2024-07-26T15:32:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:42 smithi189 ceph-mon[36790]: pgmap v8517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:32:44.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:32:44.607+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:32:44.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:32:44.607+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:44.609 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:32:44.607+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:32:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:44 smithi193 ceph-mon[37473]: pgmap v8518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:32:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14705.378359 secs 2024-07-26T15:32:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:44 smithi134 ceph-mon[31966]: pgmap v8518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:32:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14705.378359 secs 2024-07-26T15:32:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:44 smithi189 ceph-mon[36790]: pgmap v8518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:32:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14705.378359 secs 2024-07-26T15:32:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:46 smithi134 ceph-mon[31966]: pgmap v8519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:32:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:46 smithi189 ceph-mon[36790]: pgmap v8519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:32:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:46 smithi193 ceph-mon[37473]: pgmap v8519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:32:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14724.801990 secs 2024-07-26T15:32:47.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14724.801990 secs 2024-07-26T15:32:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14724.801990 secs 2024-07-26T15:32:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:48 smithi134 ceph-mon[31966]: pgmap v8520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:32:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:48 smithi189 ceph-mon[36790]: pgmap v8520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:32:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:48 smithi193 ceph-mon[37473]: pgmap v8520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:32:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14710.378479 secs 2024-07-26T15:32:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14710.378479 secs 2024-07-26T15:32:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14710.378479 secs 2024-07-26T15:32:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:50 smithi134 ceph-mon[31966]: pgmap v8521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:32:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:50 smithi189 ceph-mon[36790]: pgmap v8521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:32:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:50 smithi193 ceph-mon[37473]: pgmap v8521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:32:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14729.802111 secs 2024-07-26T15:32:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:52 smithi134 ceph-mon[31966]: pgmap v8522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14729.802111 secs 2024-07-26T15:32:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:52 smithi189 ceph-mon[36790]: pgmap v8522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14729.802111 secs 2024-07-26T15:32:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:52 smithi193 ceph-mon[37473]: pgmap v8522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:54 smithi134 ceph-mon[31966]: pgmap v8523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14715.378591 secs 2024-07-26T15:32:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:54 smithi189 ceph-mon[36790]: pgmap v8523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14715.378591 secs 2024-07-26T15:32:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:54 smithi193 ceph-mon[37473]: pgmap v8523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:32:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14715.378591 secs 2024-07-26T15:32:55.144 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:32:55.143+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:32:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:32:55.143+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:32:55.145 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:32:55.143+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:56 smithi134 ceph-mon[31966]: pgmap v8524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:56 smithi189 ceph-mon[36790]: pgmap v8524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:56 smithi193 ceph-mon[37473]: pgmap v8524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:32:57.844 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14734.802246 secs 2024-07-26T15:32:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14734.802246 secs 2024-07-26T15:32:58.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14734.802246 secs 2024-07-26T15:32:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:58 smithi134 ceph-mon[31966]: pgmap v8525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:58 smithi189 ceph-mon[36790]: pgmap v8525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:58 smithi193 ceph-mon[37473]: pgmap v8525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:32:59.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:32:59.617+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:32:59.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:32:59.617+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:32:59.619 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:32:59.617+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:32:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:32:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14720.378691 secs 2024-07-26T15:32:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:32:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14720.378691 secs 2024-07-26T15:33:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:32:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14720.378691 secs 2024-07-26T15:33:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:00 smithi134 ceph-mon[31966]: pgmap v8526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:00 smithi189 ceph-mon[36790]: pgmap v8526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:00 smithi193 ceph-mon[37473]: pgmap v8526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14739.802366 secs 2024-07-26T15:33:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:02 smithi134 ceph-mon[31966]: pgmap v8527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14739.802366 secs 2024-07-26T15:33:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:02 smithi189 ceph-mon[36790]: pgmap v8527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14739.802366 secs 2024-07-26T15:33:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:02 smithi193 ceph-mon[37473]: pgmap v8527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:04 smithi134 ceph-mon[31966]: pgmap v8528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14725.378779 secs 2024-07-26T15:33:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:04 smithi189 ceph-mon[36790]: pgmap v8528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14725.378779 secs 2024-07-26T15:33:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:04 smithi193 ceph-mon[37473]: pgmap v8528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14725.378779 secs 2024-07-26T15:33:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:06 smithi134 ceph-mon[31966]: pgmap v8529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:06 smithi189 ceph-mon[36790]: pgmap v8529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:06 smithi193 ceph-mon[37473]: pgmap v8529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14744.802481 secs 2024-07-26T15:33:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14744.802481 secs 2024-07-26T15:33:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14744.802481 secs 2024-07-26T15:33:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:08 smithi134 ceph-mon[31966]: pgmap v8530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:33:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:08 smithi189 ceph-mon[36790]: pgmap v8530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:33:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:08 smithi193 ceph-mon[37473]: pgmap v8530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:33:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14730.378897 secs 2024-07-26T15:33:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14730.378897 secs 2024-07-26T15:33:10.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14730.378897 secs 2024-07-26T15:33:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:33:10.145+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:33:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:33:10.145+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:33:10.147 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:33:10.145+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:10 smithi134 ceph-mon[31966]: pgmap v8531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:10 smithi189 ceph-mon[36790]: pgmap v8531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:10 smithi193 ceph-mon[37473]: pgmap v8531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:33:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:33:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:33:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:33:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14749.802604 secs 2024-07-26T15:33:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:12 smithi134 ceph-mon[31966]: pgmap v8532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:33:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:33:12.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:33:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14749.802604 secs 2024-07-26T15:33:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:12 smithi189 ceph-mon[36790]: pgmap v8532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:33:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:33:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:33:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14749.802604 secs 2024-07-26T15:33:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:12 smithi193 ceph-mon[37473]: pgmap v8532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:33:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:33:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:33:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:13 smithi134 ceph-mon[31966]: pgmap v8533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:33:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:13 smithi189 ceph-mon[36790]: pgmap v8533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:33:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:13 smithi193 ceph-mon[37473]: pgmap v8533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:33:14.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:33:14.632+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:33:14.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:33:14.632+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:14.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:33:14.632+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:33:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14735.379023 secs 2024-07-26T15:33:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14735.379023 secs 2024-07-26T15:33:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14735.379023 secs 2024-07-26T15:33:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:15 smithi189 ceph-mon[36790]: pgmap v8534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:33:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:15 smithi193 ceph-mon[37473]: pgmap v8534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:33:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:15 smithi134 ceph-mon[31966]: pgmap v8534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:33:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14754.802718 secs 2024-07-26T15:33:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:17 smithi189 ceph-mon[36790]: pgmap v8535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:33:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14754.802718 secs 2024-07-26T15:33:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:17 smithi193 ceph-mon[37473]: pgmap v8535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:33:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14754.802718 secs 2024-07-26T15:33:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:17 smithi134 ceph-mon[31966]: pgmap v8535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:33:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:19 smithi193 ceph-mon[37473]: pgmap v8536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:33:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14740.379145 secs 2024-07-26T15:33:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:19 smithi134 ceph-mon[31966]: pgmap v8536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:33:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14740.379145 secs 2024-07-26T15:33:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:19 smithi189 ceph-mon[36790]: pgmap v8536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:33:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14740.379145 secs 2024-07-26T15:33:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:21 smithi193 ceph-mon[37473]: pgmap v8537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:33:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:21 smithi134 ceph-mon[31966]: pgmap v8537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:33:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:21 smithi189 ceph-mon[36790]: pgmap v8537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:33:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14759.802868 secs 2024-07-26T15:33:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14759.802868 secs 2024-07-26T15:33:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14759.802868 secs 2024-07-26T15:33:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:23 smithi193 ceph-mon[37473]: pgmap v8538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:33:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:23 smithi134 ceph-mon[31966]: pgmap v8538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:33:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:23 smithi189 ceph-mon[36790]: pgmap v8538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:33:24.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:33:24.892+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:33:24.894 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:33:24.892+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:33:24.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:33:24.893+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 115 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:33:24.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:33:24.893+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:33:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14745.379263 secs 2024-07-26T15:33:25.148 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:33:25.147+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:33:25.149 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:33:25.147+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:33:25.149 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:33:25.147+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14745.379263 secs 2024-07-26T15:33:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14745.379263 secs 2024-07-26T15:33:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:25 smithi193 ceph-mon[37473]: pgmap v8539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:33:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:25 smithi134 ceph-mon[31966]: pgmap v8539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:33:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:25 smithi189 ceph-mon[36790]: pgmap v8539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:33:27.995 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14764.802986 secs 2024-07-26T15:33:27.996 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:27 smithi189 ceph-mon[36790]: pgmap v8540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14764.802986 secs 2024-07-26T15:33:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:27 smithi193 ceph-mon[37473]: pgmap v8540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14764.802986 secs 2024-07-26T15:33:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:27 smithi134 ceph-mon[31966]: pgmap v8540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:29.637 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:33:29.635+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:33:29.637 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:33:29.635+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:29.637 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:33:29.635+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:33:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:29 smithi134 ceph-mon[31966]: pgmap v8541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14750.379387 secs 2024-07-26T15:33:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:29 smithi193 ceph-mon[37473]: pgmap v8541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14750.379387 secs 2024-07-26T15:33:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:29 smithi189 ceph-mon[36790]: pgmap v8541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14750.379387 secs 2024-07-26T15:33:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:31 smithi193 ceph-mon[37473]: pgmap v8542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:31 smithi134 ceph-mon[31966]: pgmap v8542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:31 smithi189 ceph-mon[36790]: pgmap v8542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14769.803105 secs 2024-07-26T15:33:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14769.803105 secs 2024-07-26T15:33:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14769.803105 secs 2024-07-26T15:33:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:33 smithi193 ceph-mon[37473]: pgmap v8543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:33:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:33 smithi134 ceph-mon[31966]: pgmap v8543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:33:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:33 smithi189 ceph-mon[36790]: pgmap v8543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:33:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14755.379483 secs 2024-07-26T15:33:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14755.379483 secs 2024-07-26T15:33:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14755.379483 secs 2024-07-26T15:33:35.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:33:35.434+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:33:35.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:33:35.434+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:33:35.436 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:33:35.435+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:33:35.437 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:33:35.435+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:33:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:35 smithi134 ceph-mon[31966]: pgmap v8544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:33:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:35 smithi189 ceph-mon[36790]: pgmap v8544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:33:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:35 smithi193 ceph-mon[37473]: pgmap v8544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:33:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14774.803239 secs 2024-07-26T15:33:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:37 smithi134 ceph-mon[31966]: pgmap v8545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14774.803239 secs 2024-07-26T15:33:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:37 smithi189 ceph-mon[36790]: pgmap v8545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14774.803239 secs 2024-07-26T15:33:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:37 smithi193 ceph-mon[37473]: pgmap v8545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:40.164 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:33:40.162+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:33:40.164 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:33:40.162+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:33:40.164 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:33:40.162+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:39 smithi134 ceph-mon[31966]: pgmap v8546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14760.379582 secs 2024-07-26T15:33:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:39 smithi189 ceph-mon[36790]: pgmap v8546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14760.379582 secs 2024-07-26T15:33:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:39 smithi193 ceph-mon[37473]: pgmap v8546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14760.379582 secs 2024-07-26T15:33:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:41 smithi134 ceph-mon[31966]: pgmap v8547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:41 smithi189 ceph-mon[36790]: pgmap v8547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:41 smithi193 ceph-mon[37473]: pgmap v8547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14779.803388 secs 2024-07-26T15:33:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14779.803388 secs 2024-07-26T15:33:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14779.803388 secs 2024-07-26T15:33:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:43 smithi134 ceph-mon[31966]: pgmap v8548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:33:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:43 smithi189 ceph-mon[36790]: pgmap v8548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:33:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:43 smithi193 ceph-mon[37473]: pgmap v8548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:33:44.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:33:44.650+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:33:44.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:33:44.651+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:33:44.653 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:33:44.650+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14765.379701 secs 2024-07-26T15:33:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14765.379701 secs 2024-07-26T15:33:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14765.379701 secs 2024-07-26T15:33:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:45 smithi134 ceph-mon[31966]: pgmap v8549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:33:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:45 smithi189 ceph-mon[36790]: pgmap v8549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:33:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:45 smithi193 ceph-mon[37473]: pgmap v8549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:33:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14784.803463 secs 2024-07-26T15:33:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:47 smithi134 ceph-mon[31966]: pgmap v8550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14784.803463 secs 2024-07-26T15:33:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:47 smithi189 ceph-mon[36790]: pgmap v8550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14784.803463 secs 2024-07-26T15:33:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:47 smithi193 ceph-mon[37473]: pgmap v8550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:49 smithi134 ceph-mon[31966]: pgmap v8551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14770.379827 secs 2024-07-26T15:33:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:49 smithi189 ceph-mon[36790]: pgmap v8551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14770.379827 secs 2024-07-26T15:33:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:49 smithi193 ceph-mon[37473]: pgmap v8551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14770.379827 secs 2024-07-26T15:33:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:51 smithi134 ceph-mon[31966]: pgmap v8552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:51 smithi189 ceph-mon[36790]: pgmap v8552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:51 smithi193 ceph-mon[37473]: pgmap v8552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14789.803529 secs 2024-07-26T15:33:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14789.803529 secs 2024-07-26T15:33:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14789.803529 secs 2024-07-26T15:33:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:53 smithi134 ceph-mon[31966]: pgmap v8553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:33:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:53 smithi189 ceph-mon[36790]: pgmap v8553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:33:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:53 smithi193 ceph-mon[37473]: pgmap v8553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:33:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14775.379958 secs 2024-07-26T15:33:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:33:55.171+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:33:55.172 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:33:55.171+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:33:55.173 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:33:55.171+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14775.379958 secs 2024-07-26T15:33:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14775.379958 secs 2024-07-26T15:33:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:55 smithi134 ceph-mon[31966]: pgmap v8554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:33:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:55 smithi189 ceph-mon[36790]: pgmap v8554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:33:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:55 smithi193 ceph-mon[37473]: pgmap v8554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:33:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14794.803650 secs 2024-07-26T15:33:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:57 smithi134 ceph-mon[31966]: pgmap v8555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14794.803650 secs 2024-07-26T15:33:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:57 smithi189 ceph-mon[36790]: pgmap v8555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14794.803650 secs 2024-07-26T15:33:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:57 smithi193 ceph-mon[37473]: pgmap v8555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:33:59.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:33:59.659+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:33:59.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:33:59.659+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:33:59.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:33:59.659+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:33:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:59 smithi134 ceph-mon[31966]: pgmap v8556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:33:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:33:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14780.380085 secs 2024-07-26T15:34:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:59 smithi189 ceph-mon[36790]: pgmap v8556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:33:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14780.380085 secs 2024-07-26T15:34:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:59 smithi193 ceph-mon[37473]: pgmap v8556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:33:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14780.380085 secs 2024-07-26T15:34:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:01 smithi134 ceph-mon[31966]: pgmap v8557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:01 smithi189 ceph-mon[36790]: pgmap v8557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:01 smithi193 ceph-mon[37473]: pgmap v8557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14799.803750 secs 2024-07-26T15:34:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14799.803750 secs 2024-07-26T15:34:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14799.803750 secs 2024-07-26T15:34:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:03 smithi134 ceph-mon[31966]: pgmap v8558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:34:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:03 smithi189 ceph-mon[36790]: pgmap v8558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:34:04.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:03 smithi193 ceph-mon[37473]: pgmap v8558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:34:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14785.380201 secs 2024-07-26T15:34:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14785.380201 secs 2024-07-26T15:34:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14785.380201 secs 2024-07-26T15:34:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:05 smithi134 ceph-mon[31966]: pgmap v8559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:34:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:05 smithi189 ceph-mon[36790]: pgmap v8559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:34:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:05 smithi193 ceph-mon[37473]: pgmap v8559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:34:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14804.803869 secs 2024-07-26T15:34:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:07 smithi189 ceph-mon[36790]: pgmap v8560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14804.803869 secs 2024-07-26T15:34:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:07 smithi193 ceph-mon[37473]: pgmap v8560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14804.803869 secs 2024-07-26T15:34:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:07 smithi134 ceph-mon[31966]: pgmap v8560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:10.187 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:34:10.186+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:34:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:34:10.186+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:10.188 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:34:10.186+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:34:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:09 smithi189 ceph-mon[36790]: pgmap v8561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14790.380346 secs 2024-07-26T15:34:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:09 smithi193 ceph-mon[37473]: pgmap v8561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14790.380346 secs 2024-07-26T15:34:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:09 smithi134 ceph-mon[31966]: pgmap v8561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14790.380346 secs 2024-07-26T15:34:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:11 smithi189 ceph-mon[36790]: pgmap v8562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:11 smithi193 ceph-mon[37473]: pgmap v8562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:11 smithi134 ceph-mon[31966]: pgmap v8562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14809.804004 secs 2024-07-26T15:34:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:34:13.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14809.804004 secs 2024-07-26T15:34:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:34:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14809.804004 secs 2024-07-26T15:34:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:34:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:13 smithi193 ceph-mon[37473]: pgmap v8563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:34:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:34:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:34:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:34:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:13 smithi134 ceph-mon[31966]: pgmap v8563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:34:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:34:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:34:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:34:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:13 smithi189 ceph-mon[36790]: pgmap v8563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:34:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:34:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:34:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:34:14.676 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:34:14.675+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:34:14.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:34:14.675+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:34:14.677 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:34:14.675+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14795.380488 secs 2024-07-26T15:34:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:14 smithi193 ceph-mon[37473]: pgmap v8564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:34:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:14 smithi193 ceph-mon[37473]: pgmap v8565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:34:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14795.380488 secs 2024-07-26T15:34:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:14 smithi134 ceph-mon[31966]: pgmap v8564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:34:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:14 smithi134 ceph-mon[31966]: pgmap v8565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:34:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14795.380488 secs 2024-07-26T15:34:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:14 smithi189 ceph-mon[36790]: pgmap v8564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:34:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:14 smithi189 ceph-mon[36790]: pgmap v8565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:34:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:16 smithi193 ceph-mon[37473]: pgmap v8566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:34:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:16 smithi134 ceph-mon[31966]: pgmap v8566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:34:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:16 smithi189 ceph-mon[36790]: pgmap v8566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:34:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14814.804125 secs 2024-07-26T15:34:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14814.804125 secs 2024-07-26T15:34:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14814.804125 secs 2024-07-26T15:34:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:18 smithi193 ceph-mon[37473]: pgmap v8567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:34:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:18 smithi134 ceph-mon[31966]: pgmap v8567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:34:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:18 smithi189 ceph-mon[36790]: pgmap v8567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:34:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14800.380602 secs 2024-07-26T15:34:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14800.380602 secs 2024-07-26T15:34:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14800.380602 secs 2024-07-26T15:34:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:20 smithi193 ceph-mon[37473]: pgmap v8568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:34:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:20 smithi134 ceph-mon[31966]: pgmap v8568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:34:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:20 smithi189 ceph-mon[36790]: pgmap v8568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:34:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:22 smithi193 ceph-mon[37473]: pgmap v8569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:22 smithi134 ceph-mon[31966]: pgmap v8569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:22 smithi189 ceph-mon[36790]: pgmap v8569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:23 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14819.804242 secs 2024-07-26T15:34:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:23 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14819.804242 secs 2024-07-26T15:34:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:23 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14819.804242 secs 2024-07-26T15:34:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:24 smithi193 ceph-mon[37473]: pgmap v8570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:34:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:24 smithi134 ceph-mon[31966]: pgmap v8570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:34:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:24 smithi189 ceph-mon[36790]: pgmap v8570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:34:25.203 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:34:25.202+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:34:25.203 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:34:25.202+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:25.203 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:34:25.202+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:34:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14805.380723 secs 2024-07-26T15:34:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14805.380723 secs 2024-07-26T15:34:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14805.380723 secs 2024-07-26T15:34:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:26 smithi134 ceph-mon[31966]: pgmap v8571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:26 smithi189 ceph-mon[36790]: pgmap v8571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:26 smithi193 ceph-mon[37473]: pgmap v8571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14824.804386 secs 2024-07-26T15:34:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14824.804386 secs 2024-07-26T15:34:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14824.804386 secs 2024-07-26T15:34:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:28 smithi134 ceph-mon[31966]: pgmap v8572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:28 smithi189 ceph-mon[36790]: pgmap v8572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:28 smithi193 ceph-mon[37473]: pgmap v8572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:29.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:34:29.690+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:34:29.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:34:29.690+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:29.692 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:34:29.690+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:34:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14810.380846 secs 2024-07-26T15:34:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14810.380846 secs 2024-07-26T15:34:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14810.380846 secs 2024-07-26T15:34:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:30 smithi134 ceph-mon[31966]: pgmap v8573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:34:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:30 smithi189 ceph-mon[36790]: pgmap v8573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:34:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:30 smithi193 ceph-mon[37473]: pgmap v8573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:34:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14829.804515 secs 2024-07-26T15:34:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:32 smithi189 ceph-mon[36790]: pgmap v8574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14829.804515 secs 2024-07-26T15:34:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:32 smithi193 ceph-mon[37473]: pgmap v8574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14829.804515 secs 2024-07-26T15:34:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:32 smithi134 ceph-mon[31966]: pgmap v8574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14815.380981 secs 2024-07-26T15:34:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:34 smithi189 ceph-mon[36790]: pgmap v8575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14815.380981 secs 2024-07-26T15:34:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:34 smithi193 ceph-mon[37473]: pgmap v8575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14815.380981 secs 2024-07-26T15:34:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:34 smithi134 ceph-mon[31966]: pgmap v8575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:36 smithi193 ceph-mon[37473]: pgmap v8576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:36 smithi134 ceph-mon[31966]: pgmap v8576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:36 smithi189 ceph-mon[36790]: pgmap v8576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14834.804623 secs 2024-07-26T15:34:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14834.804623 secs 2024-07-26T15:34:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14834.804623 secs 2024-07-26T15:34:38.985 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:38 smithi134 ceph-mon[31966]: pgmap v8577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:38 smithi193 ceph-mon[37473]: pgmap v8577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:38 smithi189 ceph-mon[36790]: pgmap v8577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14820.381073 secs 2024-07-26T15:34:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14820.381073 secs 2024-07-26T15:34:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14820.381073 secs 2024-07-26T15:34:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:34:40.215+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:34:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:34:40.215+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:34:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:34:40.215+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:40 smithi193 ceph-mon[37473]: pgmap v8578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:34:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:40 smithi134 ceph-mon[31966]: pgmap v8578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:34:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:40 smithi189 ceph-mon[36790]: pgmap v8578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:34:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14839.804745 secs 2024-07-26T15:34:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:42 smithi189 ceph-mon[36790]: pgmap v8579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14839.804745 secs 2024-07-26T15:34:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:42 smithi193 ceph-mon[37473]: pgmap v8579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14839.804745 secs 2024-07-26T15:34:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:42 smithi134 ceph-mon[31966]: pgmap v8579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:44.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:34:44.695+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:34:44.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:34:44.695+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:34:44.697 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:34:44.695+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14825.381159 secs 2024-07-26T15:34:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:44 smithi193 ceph-mon[37473]: pgmap v8580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14825.381159 secs 2024-07-26T15:34:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:44 smithi134 ceph-mon[31966]: pgmap v8580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14825.381159 secs 2024-07-26T15:34:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:44 smithi189 ceph-mon[36790]: pgmap v8580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:46 smithi193 ceph-mon[37473]: pgmap v8581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:46 smithi134 ceph-mon[31966]: pgmap v8581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:46 smithi189 ceph-mon[36790]: pgmap v8581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14844.804860 secs 2024-07-26T15:34:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14844.804860 secs 2024-07-26T15:34:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14844.804860 secs 2024-07-26T15:34:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:48 smithi193 ceph-mon[37473]: pgmap v8582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:48 smithi134 ceph-mon[31966]: pgmap v8582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:48 smithi189 ceph-mon[36790]: pgmap v8582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14830.381277 secs 2024-07-26T15:34:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14830.381277 secs 2024-07-26T15:34:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14830.381277 secs 2024-07-26T15:34:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:50 smithi193 ceph-mon[37473]: pgmap v8583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:34:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:50 smithi134 ceph-mon[31966]: pgmap v8583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:34:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:50 smithi189 ceph-mon[36790]: pgmap v8583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:34:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14849.804973 secs 2024-07-26T15:34:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:52 smithi193 ceph-mon[37473]: pgmap v8584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14849.804973 secs 2024-07-26T15:34:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:52 smithi134 ceph-mon[31966]: pgmap v8584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14849.804973 secs 2024-07-26T15:34:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:52 smithi189 ceph-mon[36790]: pgmap v8584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:54.910 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:34:54.909+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:34:54.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:34:54.909+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:34:54.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:34:54.909+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 116 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T15:34:54.911 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:34:54.910+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T15:34:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14835.381423 secs 2024-07-26T15:34:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:54 smithi193 ceph-mon[37473]: pgmap v8585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14835.381423 secs 2024-07-26T15:34:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:54 smithi134 ceph-mon[31966]: pgmap v8585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14835.381423 secs 2024-07-26T15:34:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:54 smithi189 ceph-mon[36790]: pgmap v8585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:34:55.217+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:34:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:34:55.217+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:34:55.218 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:34:55.217+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:34:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:56 smithi193 ceph-mon[37473]: pgmap v8586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:56 smithi134 ceph-mon[31966]: pgmap v8586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:56 smithi189 ceph-mon[36790]: pgmap v8586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:34:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14854.805031 secs 2024-07-26T15:34:58.094 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14854.805031 secs 2024-07-26T15:34:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14854.805031 secs 2024-07-26T15:34:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:58 smithi193 ceph-mon[37473]: pgmap v8587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:59.075 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:58 smithi134 ceph-mon[31966]: pgmap v8587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:58 smithi189 ceph-mon[36790]: pgmap v8587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:34:59.712 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:34:59.711+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:34:59.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:34:59.711+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:34:59.713 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:34:59.711+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:34:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14840.381539 secs 2024-07-26T15:35:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:34:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14840.381539 secs 2024-07-26T15:35:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:34:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14840.381539 secs 2024-07-26T15:35:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:00 smithi193 ceph-mon[37473]: pgmap v8588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:35:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:00 smithi134 ceph-mon[31966]: pgmap v8588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:35:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:00 smithi189 ceph-mon[36790]: pgmap v8588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:35:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14859.805153 secs 2024-07-26T15:35:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:02 smithi134 ceph-mon[31966]: pgmap v8589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14859.805153 secs 2024-07-26T15:35:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:02 smithi189 ceph-mon[36790]: pgmap v8589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14859.805153 secs 2024-07-26T15:35:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:02 smithi193 ceph-mon[37473]: pgmap v8589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14845.381676 secs 2024-07-26T15:35:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:04 smithi134 ceph-mon[31966]: pgmap v8590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14845.381676 secs 2024-07-26T15:35:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:04 smithi189 ceph-mon[36790]: pgmap v8590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:05.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14845.381676 secs 2024-07-26T15:35:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:04 smithi193 ceph-mon[37473]: pgmap v8590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:35:05.450+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:35:05.451 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:35:05.450+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:35:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:35:05.451+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:35:05.452 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:35:05.451+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:35:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:06 smithi134 ceph-mon[31966]: pgmap v8591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:06 smithi189 ceph-mon[36790]: pgmap v8591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:07.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:06 smithi193 ceph-mon[37473]: pgmap v8591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14864.805274 secs 2024-07-26T15:35:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14864.805274 secs 2024-07-26T15:35:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14864.805274 secs 2024-07-26T15:35:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:08 smithi134 ceph-mon[31966]: pgmap v8592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:08 smithi189 ceph-mon[36790]: pgmap v8592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:08 smithi193 ceph-mon[37473]: pgmap v8592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14850.381791 secs 2024-07-26T15:35:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14850.381791 secs 2024-07-26T15:35:10.233 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:35:10.232+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:35:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:35:10.232+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:35:10.234 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:35:10.232+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:10.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14850.381791 secs 2024-07-26T15:35:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:10 smithi134 ceph-mon[31966]: pgmap v8593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:35:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:10 smithi189 ceph-mon[36790]: pgmap v8593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:35:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:10 smithi193 ceph-mon[37473]: pgmap v8593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:35:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14869.805402 secs 2024-07-26T15:35:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:12 smithi134 ceph-mon[31966]: pgmap v8594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14869.805402 secs 2024-07-26T15:35:13.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:12 smithi189 ceph-mon[36790]: pgmap v8594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:13.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14869.805402 secs 2024-07-26T15:35:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:12 smithi193 ceph-mon[37473]: pgmap v8594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:14.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:35:14.721+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:35:14.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:35:14.721+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:35:14.724 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:35:14.721+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14855.381887 secs 2024-07-26T15:35:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:14 smithi134 ceph-mon[31966]: pgmap v8595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:35:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14855.381887 secs 2024-07-26T15:35:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:14 smithi189 ceph-mon[36790]: pgmap v8595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:35:15.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14855.381887 secs 2024-07-26T15:35:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:14 smithi193 ceph-mon[37473]: pgmap v8595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:35:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:35:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:35:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:15 smithi134 ceph-mon[31966]: pgmap v8596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:35:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:15 smithi134 ceph-mon[31966]: pgmap v8597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:35:16.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:35:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:35:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:35:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:15 smithi189 ceph-mon[36790]: pgmap v8596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:35:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:15 smithi189 ceph-mon[36790]: pgmap v8597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:35:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:35:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:35:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:35:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:15 smithi193 ceph-mon[37473]: pgmap v8596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:35:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:15 smithi193 ceph-mon[37473]: pgmap v8597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:35:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:35:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14874.805551 secs 2024-07-26T15:35:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:17 smithi193 ceph-mon[37473]: pgmap v8598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:35:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14874.805551 secs 2024-07-26T15:35:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:17 smithi134 ceph-mon[31966]: pgmap v8598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:35:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14874.805551 secs 2024-07-26T15:35:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:17 smithi189 ceph-mon[36790]: pgmap v8598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:35:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14860.382018 secs 2024-07-26T15:35:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:19 smithi134 ceph-mon[31966]: pgmap v8599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:35:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14860.382018 secs 2024-07-26T15:35:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:19 smithi189 ceph-mon[36790]: pgmap v8599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:35:20.227 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14860.382018 secs 2024-07-26T15:35:20.227 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:19 smithi193 ceph-mon[37473]: pgmap v8599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:35:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:21 smithi134 ceph-mon[31966]: pgmap v8600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:35:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:21 smithi189 ceph-mon[36790]: pgmap v8600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:35:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:21 smithi193 ceph-mon[37473]: pgmap v8600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:35:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14879.805653 secs 2024-07-26T15:35:23.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14879.805653 secs 2024-07-26T15:35:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14879.805653 secs 2024-07-26T15:35:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:23 smithi189 ceph-mon[36790]: pgmap v8601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:35:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:23 smithi193 ceph-mon[37473]: pgmap v8601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:35:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:23 smithi134 ceph-mon[31966]: pgmap v8601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:35:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14865.382134 secs 2024-07-26T15:35:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:35:25.248+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:35:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:35:25.248+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:35:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:35:25.248+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14865.382134 secs 2024-07-26T15:35:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14865.382134 secs 2024-07-26T15:35:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:25 smithi189 ceph-mon[36790]: pgmap v8602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:35:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:25 smithi193 ceph-mon[37473]: pgmap v8602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:35:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:25 smithi134 ceph-mon[31966]: pgmap v8602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:35:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14884.805743 secs 2024-07-26T15:35:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:27 smithi193 ceph-mon[37473]: pgmap v8603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14884.805743 secs 2024-07-26T15:35:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:27 smithi134 ceph-mon[31966]: pgmap v8603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14884.805743 secs 2024-07-26T15:35:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:27 smithi189 ceph-mon[36790]: pgmap v8603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:29.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:35:29.723+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:35:29.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:35:29.723+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:29.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:35:29.723+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:35:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14870.382219 secs 2024-07-26T15:35:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:29 smithi134 ceph-mon[31966]: pgmap v8604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:30.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14870.382219 secs 2024-07-26T15:35:30.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:29 smithi189 ceph-mon[36790]: pgmap v8604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14870.382219 secs 2024-07-26T15:35:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:29 smithi193 ceph-mon[37473]: pgmap v8604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:31 smithi193 ceph-mon[37473]: pgmap v8605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:31 smithi134 ceph-mon[31966]: pgmap v8605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:31 smithi189 ceph-mon[36790]: pgmap v8605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14889.805843 secs 2024-07-26T15:35:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14889.805843 secs 2024-07-26T15:35:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14889.805843 secs 2024-07-26T15:35:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:33 smithi193 ceph-mon[37473]: pgmap v8606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:33 smithi134 ceph-mon[31966]: pgmap v8606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:33 smithi189 ceph-mon[36790]: pgmap v8606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14875.382341 secs 2024-07-26T15:35:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14875.382341 secs 2024-07-26T15:35:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14875.382341 secs 2024-07-26T15:35:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:35 smithi193 ceph-mon[37473]: pgmap v8607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:35:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:35 smithi134 ceph-mon[31966]: pgmap v8607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:35:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:35 smithi189 ceph-mon[36790]: pgmap v8607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:35:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14894.805915 secs 2024-07-26T15:35:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:37 smithi193 ceph-mon[37473]: pgmap v8608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14894.805915 secs 2024-07-26T15:35:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:37 smithi134 ceph-mon[31966]: pgmap v8608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14894.805915 secs 2024-07-26T15:35:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:37 smithi189 ceph-mon[36790]: pgmap v8608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:35:40.263+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:35:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:35:40.263+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:35:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:35:40.263+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14880.382467 secs 2024-07-26T15:35:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:39 smithi193 ceph-mon[37473]: pgmap v8609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14880.382467 secs 2024-07-26T15:35:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:39 smithi134 ceph-mon[31966]: pgmap v8609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14880.382467 secs 2024-07-26T15:35:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:39 smithi189 ceph-mon[36790]: pgmap v8609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:42 smithi193 ceph-mon[37473]: pgmap v8610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:42 smithi134 ceph-mon[31966]: pgmap v8610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:42 smithi189 ceph-mon[36790]: pgmap v8610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:43 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14899.806062 secs 2024-07-26T15:35:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:43 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14899.806062 secs 2024-07-26T15:35:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:43 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14899.806062 secs 2024-07-26T15:35:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:44 smithi193 ceph-mon[37473]: pgmap v8611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:44 smithi134 ceph-mon[31966]: pgmap v8611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:44 smithi189 ceph-mon[36790]: pgmap v8611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:35:44.740 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:35:44.738+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:35:44.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:35:44.739+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:35:44.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:35:44.739+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14885.382565 secs 2024-07-26T15:35:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:45 smithi134 ceph-mon[31966]: pgmap v8612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:35:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14885.382565 secs 2024-07-26T15:35:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:45 smithi189 ceph-mon[36790]: pgmap v8612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:35:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14885.382565 secs 2024-07-26T15:35:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:45 smithi193 ceph-mon[37473]: pgmap v8612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:35:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14904.806180 secs 2024-07-26T15:35:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:47 smithi189 ceph-mon[36790]: pgmap v8613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14904.806180 secs 2024-07-26T15:35:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:47 smithi193 ceph-mon[37473]: pgmap v8613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14904.806180 secs 2024-07-26T15:35:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:47 smithi134 ceph-mon[31966]: pgmap v8613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:35:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14890.382640 secs 2024-07-26T15:35:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:49 smithi189 ceph-mon[36790]: pgmap v8614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:35:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14890.382640 secs 2024-07-26T15:35:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:49 smithi193 ceph-mon[37473]: pgmap v8614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:35:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14890.382640 secs 2024-07-26T15:35:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:49 smithi134 ceph-mon[31966]: pgmap v8614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:35:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:51 smithi189 ceph-mon[36790]: pgmap v8615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:35:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:51 smithi193 ceph-mon[37473]: pgmap v8615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:35:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:51 smithi134 ceph-mon[31966]: pgmap v8615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:35:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14909.806350 secs 2024-07-26T15:35:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14909.806350 secs 2024-07-26T15:35:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14909.806350 secs 2024-07-26T15:35:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:53 smithi193 ceph-mon[37473]: pgmap v8616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:35:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:53 smithi134 ceph-mon[31966]: pgmap v8616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:35:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:53 smithi189 ceph-mon[36790]: pgmap v8616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:35:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14895.382757 secs 2024-07-26T15:35:55.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:35:55.279+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:35:55.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:35:55.279+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:35:55.280 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:35:55.279+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:35:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14895.382757 secs 2024-07-26T15:35:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14895.382757 secs 2024-07-26T15:35:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:55 smithi193 ceph-mon[37473]: pgmap v8617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:35:56.297 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:55 smithi134 ceph-mon[31966]: pgmap v8617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:35:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:55 smithi189 ceph-mon[36790]: pgmap v8617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:35:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14914.806473 secs 2024-07-26T15:35:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:57 smithi193 ceph-mon[37473]: pgmap v8618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:35:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14914.806473 secs 2024-07-26T15:35:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:57 smithi134 ceph-mon[31966]: pgmap v8618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:35:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14914.806473 secs 2024-07-26T15:35:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:57 smithi189 ceph-mon[36790]: pgmap v8618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:35:59.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:35:59.752+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:35:59.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:35:59.752+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:35:59.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:35:59.752+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14900.382875 secs 2024-07-26T15:36:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:35:59 smithi134 ceph-mon[31966]: pgmap v8619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:36:00.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14900.382875 secs 2024-07-26T15:36:00.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:35:59 smithi189 ceph-mon[36790]: pgmap v8619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:36:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14900.382875 secs 2024-07-26T15:36:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:35:59 smithi193 ceph-mon[37473]: pgmap v8619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:36:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:01 smithi193 ceph-mon[37473]: pgmap v8620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:01 smithi134 ceph-mon[31966]: pgmap v8620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:01 smithi189 ceph-mon[36790]: pgmap v8620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14919.806576 secs 2024-07-26T15:36:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14919.806576 secs 2024-07-26T15:36:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14919.806576 secs 2024-07-26T15:36:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:03 smithi193 ceph-mon[37473]: pgmap v8621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:03 smithi134 ceph-mon[31966]: pgmap v8621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:03 smithi189 ceph-mon[36790]: pgmap v8621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14905.383007 secs 2024-07-26T15:36:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14905.383007 secs 2024-07-26T15:36:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14905.383007 secs 2024-07-26T15:36:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:06 smithi193 ceph-mon[37473]: pgmap v8622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:36:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:06 smithi134 ceph-mon[31966]: pgmap v8622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:36:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:06 smithi189 ceph-mon[36790]: pgmap v8622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:36:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:08 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14924.806716 secs 2024-07-26T15:36:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:08 smithi193 ceph-mon[37473]: pgmap v8623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:08 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14924.806716 secs 2024-07-26T15:36:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:08 smithi134 ceph-mon[31966]: pgmap v8623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:08 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14924.806716 secs 2024-07-26T15:36:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:08 smithi189 ceph-mon[36790]: pgmap v8623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14910.383116 secs 2024-07-26T15:36:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:10 smithi193 ceph-mon[37473]: pgmap v8624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:36:10.294+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:36:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:36:10.294+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:36:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:36:10.294+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14910.383116 secs 2024-07-26T15:36:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:10 smithi134 ceph-mon[31966]: pgmap v8624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14910.383116 secs 2024-07-26T15:36:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:10 smithi189 ceph-mon[36790]: pgmap v8624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:11 smithi134 ceph-mon[31966]: pgmap v8625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:11 smithi189 ceph-mon[36790]: pgmap v8625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:11.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:11 smithi193 ceph-mon[37473]: pgmap v8625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14929.806826 secs 2024-07-26T15:36:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14929.806826 secs 2024-07-26T15:36:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14929.806826 secs 2024-07-26T15:36:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:13 smithi134 ceph-mon[31966]: pgmap v8626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:13 smithi189 ceph-mon[36790]: pgmap v8626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:13 smithi193 ceph-mon[37473]: pgmap v8626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:14.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:36:14.762+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:36:14.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:36:14.762+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:36:14.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:36:14.762+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14915.383233 secs 2024-07-26T15:36:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14915.383233 secs 2024-07-26T15:36:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14915.383233 secs 2024-07-26T15:36:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:15 smithi134 ceph-mon[31966]: pgmap v8627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:36:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:36:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:15 smithi189 ceph-mon[36790]: pgmap v8627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:36:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:36:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:15 smithi193 ceph-mon[37473]: pgmap v8627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:36:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:36:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:36:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:36:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:36:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:36:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:36:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:36:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:36:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:36:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:36:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:17 smithi134 ceph-mon[31966]: pgmap v8628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:36:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:17 smithi134 ceph-mon[31966]: pgmap v8629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:36:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14934.806882 secs 2024-07-26T15:36:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:17 smithi189 ceph-mon[36790]: pgmap v8628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:36:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:17 smithi189 ceph-mon[36790]: pgmap v8629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:36:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14934.806882 secs 2024-07-26T15:36:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:17 smithi193 ceph-mon[37473]: pgmap v8628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:36:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:17 smithi193 ceph-mon[37473]: pgmap v8629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:36:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14934.806882 secs 2024-07-26T15:36:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:19 smithi134 ceph-mon[31966]: pgmap v8630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:36:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14920.383387 secs 2024-07-26T15:36:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:19 smithi189 ceph-mon[36790]: pgmap v8630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:36:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14920.383387 secs 2024-07-26T15:36:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:19 smithi193 ceph-mon[37473]: pgmap v8630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:36:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14920.383387 secs 2024-07-26T15:36:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:21 smithi134 ceph-mon[31966]: pgmap v8631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:36:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:21 smithi189 ceph-mon[36790]: pgmap v8631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:36:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:21 smithi193 ceph-mon[37473]: pgmap v8631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:36:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14939.807019 secs 2024-07-26T15:36:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14939.807019 secs 2024-07-26T15:36:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14939.807019 secs 2024-07-26T15:36:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:23 smithi134 ceph-mon[31966]: pgmap v8632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:36:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:23 smithi189 ceph-mon[36790]: pgmap v8632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:36:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:23 smithi193 ceph-mon[37473]: pgmap v8632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:36:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14925.383498 secs 2024-07-26T15:36:24.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:36:24.924+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:36:24.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:36:24.924+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:36:24.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:36:24.925+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 117 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T15:36:24.926 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:36:24.925+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T15:36:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14925.383498 secs 2024-07-26T15:36:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14925.383498 secs 2024-07-26T15:36:25.312 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:36:25.310+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:36:25.312 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:36:25.310+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:36:25.312 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:36:25.310+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:25 smithi134 ceph-mon[31966]: pgmap v8633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:36:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:25 smithi189 ceph-mon[36790]: pgmap v8633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:36:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:25 smithi193 ceph-mon[37473]: pgmap v8633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:36:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:27 smithi189 ceph-mon[36790]: pgmap v8634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:36:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14944.807139 secs 2024-07-26T15:36:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:27 smithi193 ceph-mon[37473]: pgmap v8634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:36:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14944.807139 secs 2024-07-26T15:36:28.040 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:27 smithi134 ceph-mon[31966]: pgmap v8634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:36:28.041 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14944.807139 secs 2024-07-26T15:36:29.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:36:29.768+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:36:29.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:36:29.768+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:29.770 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:36:29.768+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:36:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:29 smithi189 ceph-mon[36790]: pgmap v8635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14930.383639 secs 2024-07-26T15:36:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:29 smithi193 ceph-mon[37473]: pgmap v8635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14930.383639 secs 2024-07-26T15:36:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:29 smithi134 ceph-mon[31966]: pgmap v8635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14930.383639 secs 2024-07-26T15:36:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:31 smithi189 ceph-mon[36790]: pgmap v8636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:31 smithi193 ceph-mon[37473]: pgmap v8636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:31 smithi134 ceph-mon[31966]: pgmap v8636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14949.807205 secs 2024-07-26T15:36:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14949.807205 secs 2024-07-26T15:36:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14949.807205 secs 2024-07-26T15:36:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:33 smithi193 ceph-mon[37473]: pgmap v8637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:33 smithi134 ceph-mon[31966]: pgmap v8637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:33 smithi189 ceph-mon[36790]: pgmap v8637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14935.383780 secs 2024-07-26T15:36:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14935.383780 secs 2024-07-26T15:36:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14935.383780 secs 2024-07-26T15:36:35.467 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:36:35.466+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:36:35.467 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:36:35.466+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:36:35.468 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:36:35.466+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:36:35.468 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:36:35.466+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:36:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:35 smithi193 ceph-mon[37473]: pgmap v8638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:35 smithi134 ceph-mon[31966]: pgmap v8638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:35 smithi189 ceph-mon[36790]: pgmap v8638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:37 smithi193 ceph-mon[37473]: pgmap v8639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14954.807350 secs 2024-07-26T15:36:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:37 smithi134 ceph-mon[31966]: pgmap v8639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14954.807350 secs 2024-07-26T15:36:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:37 smithi189 ceph-mon[36790]: pgmap v8639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14954.807350 secs 2024-07-26T15:36:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:39 smithi193 ceph-mon[37473]: pgmap v8640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14940.383897 secs 2024-07-26T15:36:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:39 smithi134 ceph-mon[31966]: pgmap v8640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14940.383897 secs 2024-07-26T15:36:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:39 smithi189 ceph-mon[36790]: pgmap v8640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14940.383897 secs 2024-07-26T15:36:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:36:40.325+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:36:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:36:40.325+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:36:40.327 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:36:40.325+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:41 smithi193 ceph-mon[37473]: pgmap v8641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:41 smithi134 ceph-mon[31966]: pgmap v8641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:41 smithi189 ceph-mon[36790]: pgmap v8641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14959.807398 secs 2024-07-26T15:36:43.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14959.807398 secs 2024-07-26T15:36:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14959.807398 secs 2024-07-26T15:36:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:43 smithi193 ceph-mon[37473]: pgmap v8642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:43 smithi134 ceph-mon[31966]: pgmap v8642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:43 smithi189 ceph-mon[36790]: pgmap v8642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:44.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:36:44.784+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:36:44.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:36:44.784+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:36:44.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:36:44.784+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14945.384011 secs 2024-07-26T15:36:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14945.384011 secs 2024-07-26T15:36:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14945.384011 secs 2024-07-26T15:36:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:45 smithi193 ceph-mon[37473]: pgmap v8643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:45 smithi134 ceph-mon[31966]: pgmap v8643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:45 smithi189 ceph-mon[36790]: pgmap v8643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:47 smithi193 ceph-mon[37473]: pgmap v8644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14964.807519 secs 2024-07-26T15:36:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:47 smithi134 ceph-mon[31966]: pgmap v8644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14964.807519 secs 2024-07-26T15:36:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:47 smithi189 ceph-mon[36790]: pgmap v8644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14964.807519 secs 2024-07-26T15:36:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:49 smithi193 ceph-mon[37473]: pgmap v8645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14950.384096 secs 2024-07-26T15:36:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:49 smithi134 ceph-mon[31966]: pgmap v8645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14950.384096 secs 2024-07-26T15:36:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:49 smithi189 ceph-mon[36790]: pgmap v8645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:36:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14950.384096 secs 2024-07-26T15:36:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:51 smithi134 ceph-mon[31966]: pgmap v8646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:51 smithi189 ceph-mon[36790]: pgmap v8646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:51 smithi193 ceph-mon[37473]: pgmap v8646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14969.807638 secs 2024-07-26T15:36:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14969.807638 secs 2024-07-26T15:36:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14969.807638 secs 2024-07-26T15:36:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:53 smithi134 ceph-mon[31966]: pgmap v8647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:53 smithi189 ceph-mon[36790]: pgmap v8647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:53 smithi193 ceph-mon[37473]: pgmap v8647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:36:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14955.384217 secs 2024-07-26T15:36:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14955.384217 secs 2024-07-26T15:36:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14955.384217 secs 2024-07-26T15:36:55.337 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:36:55.335+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:36:55.337 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:36:55.336+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:36:55.337 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:36:55.336+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:36:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:55 smithi134 ceph-mon[31966]: pgmap v8648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:55 smithi189 ceph-mon[36790]: pgmap v8648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:55 smithi193 ceph-mon[37473]: pgmap v8648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:36:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:57 smithi134 ceph-mon[31966]: pgmap v8649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14974.807709 secs 2024-07-26T15:36:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:57 smithi189 ceph-mon[36790]: pgmap v8649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14974.807709 secs 2024-07-26T15:36:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:57 smithi193 ceph-mon[37473]: pgmap v8649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:36:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14974.807709 secs 2024-07-26T15:36:59.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:36:59.793+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:36:59.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:36:59.793+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:36:59.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:36:59.793+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:59 smithi134 ceph-mon[31966]: pgmap v8650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:37:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:36:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14960.384338 secs 2024-07-26T15:37:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:59 smithi189 ceph-mon[36790]: pgmap v8650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:37:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:36:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14960.384338 secs 2024-07-26T15:37:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:59 smithi193 ceph-mon[37473]: pgmap v8650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:37:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:36:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14960.384338 secs 2024-07-26T15:37:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:01 smithi134 ceph-mon[31966]: pgmap v8651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:01 smithi189 ceph-mon[36790]: pgmap v8651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:01 smithi193 ceph-mon[37473]: pgmap v8651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14979.807857 secs 2024-07-26T15:37:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14979.807857 secs 2024-07-26T15:37:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14979.807857 secs 2024-07-26T15:37:04.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:03 smithi134 ceph-mon[31966]: pgmap v8652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:03 smithi189 ceph-mon[36790]: pgmap v8652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:03 smithi193 ceph-mon[37473]: pgmap v8652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14965.384456 secs 2024-07-26T15:37:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14965.384456 secs 2024-07-26T15:37:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14965.384456 secs 2024-07-26T15:37:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:05 smithi134 ceph-mon[31966]: pgmap v8653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:05 smithi189 ceph-mon[36790]: pgmap v8653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:05 smithi193 ceph-mon[37473]: pgmap v8653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:07 smithi134 ceph-mon[31966]: pgmap v8654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14984.808000 secs 2024-07-26T15:37:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:07 smithi189 ceph-mon[36790]: pgmap v8654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14984.808000 secs 2024-07-26T15:37:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:07 smithi193 ceph-mon[37473]: pgmap v8654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14984.808000 secs 2024-07-26T15:37:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:09 smithi134 ceph-mon[31966]: pgmap v8655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:37:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14970.384580 secs 2024-07-26T15:37:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:09 smithi189 ceph-mon[36790]: pgmap v8655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:37:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14970.384580 secs 2024-07-26T15:37:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:09 smithi193 ceph-mon[37473]: pgmap v8655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:37:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14970.384580 secs 2024-07-26T15:37:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:37:10.351+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:37:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:37:10.351+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:10.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:37:10.351+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:37:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:11 smithi134 ceph-mon[31966]: pgmap v8656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:37:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:11 smithi189 ceph-mon[36790]: pgmap v8656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:37:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:11 smithi193 ceph-mon[37473]: pgmap v8656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:37:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14989.808121 secs 2024-07-26T15:37:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14989.808121 secs 2024-07-26T15:37:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14989.808121 secs 2024-07-26T15:37:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:13 smithi134 ceph-mon[31966]: pgmap v8657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T15:37:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:13 smithi189 ceph-mon[36790]: pgmap v8657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T15:37:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:13 smithi193 ceph-mon[37473]: pgmap v8657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T15:37:14.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:37:14.809+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:37:14.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:37:14.809+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:37:14.810 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:37:14.809+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14975.384714 secs 2024-07-26T15:37:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14975.384714 secs 2024-07-26T15:37:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14975.384714 secs 2024-07-26T15:37:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:15 smithi134 ceph-mon[31966]: pgmap v8658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:15 smithi189 ceph-mon[36790]: pgmap v8658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:15 smithi193 ceph-mon[37473]: pgmap v8658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:37:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:37:17.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:37:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:17 smithi189 ceph-mon[36790]: pgmap v8659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:37:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14994.808271 secs 2024-07-26T15:37:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:37:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:37:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:37:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:17 smithi193 ceph-mon[37473]: pgmap v8659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:37:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14994.808271 secs 2024-07-26T15:37:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:18.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:18.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:37:18.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:37:18.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:37:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:17 smithi134 ceph-mon[31966]: pgmap v8659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:37:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14994.808271 secs 2024-07-26T15:37:18.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:18.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:18.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:18.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:37:18.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:37:18.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:37:18.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:37:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:18 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:37:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:18 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:37:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:18 smithi189 ceph-mon[36790]: pgmap v8660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:37:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:18 smithi189 ceph-mon[36790]: pgmap v8661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:37:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:18 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:37:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:18 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:37:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:18 smithi193 ceph-mon[37473]: pgmap v8660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:37:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:18 smithi193 ceph-mon[37473]: pgmap v8661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:37:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:18 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:37:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:18 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:37:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:18 smithi134 ceph-mon[31966]: pgmap v8660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:37:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:18 smithi134 ceph-mon[31966]: pgmap v8661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:37:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14980.384831 secs 2024-07-26T15:37:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14980.384831 secs 2024-07-26T15:37:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14980.384831 secs 2024-07-26T15:37:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:20 smithi193 ceph-mon[37473]: pgmap v8662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:37:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:20 smithi134 ceph-mon[31966]: pgmap v8662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:37:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:20 smithi189 ceph-mon[36790]: pgmap v8662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:37:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 14999.808395 secs 2024-07-26T15:37:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:22 smithi193 ceph-mon[37473]: pgmap v8663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:37:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 14999.808395 secs 2024-07-26T15:37:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:22 smithi134 ceph-mon[31966]: pgmap v8663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:37:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 14999.808395 secs 2024-07-26T15:37:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:22 smithi189 ceph-mon[36790]: pgmap v8663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:37:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:24 smithi193 ceph-mon[37473]: pgmap v8664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14985.384928 secs 2024-07-26T15:37:25.368 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:37:25.367+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:37:25.368 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:37:25.367+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:37:25.368 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:37:25.367+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:24 smithi134 ceph-mon[31966]: pgmap v8664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14985.384928 secs 2024-07-26T15:37:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:24 smithi189 ceph-mon[36790]: pgmap v8664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14985.384928 secs 2024-07-26T15:37:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:26 smithi193 ceph-mon[37473]: pgmap v8665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:26 smithi134 ceph-mon[31966]: pgmap v8665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:26 smithi189 ceph-mon[36790]: pgmap v8665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:37:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15004.808493 secs 2024-07-26T15:37:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15004.808493 secs 2024-07-26T15:37:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15004.808493 secs 2024-07-26T15:37:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:28 smithi193 ceph-mon[37473]: pgmap v8666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:37:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:28 smithi134 ceph-mon[31966]: pgmap v8666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:37:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:28 smithi189 ceph-mon[36790]: pgmap v8666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:37:29.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:37:29.824+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:37:29.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:37:29.824+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:37:29.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:37:29.824+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14990.385046 secs 2024-07-26T15:37:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14990.385046 secs 2024-07-26T15:37:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14990.385046 secs 2024-07-26T15:37:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:30 smithi193 ceph-mon[37473]: pgmap v8667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:31.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:30 smithi189 ceph-mon[36790]: pgmap v8667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:30 smithi134 ceph-mon[31966]: pgmap v8667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15009.808577 secs 2024-07-26T15:37:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:32 smithi193 ceph-mon[37473]: pgmap v8668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15009.808577 secs 2024-07-26T15:37:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:32 smithi134 ceph-mon[31966]: pgmap v8668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15009.808577 secs 2024-07-26T15:37:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:32 smithi189 ceph-mon[36790]: pgmap v8668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:35 smithi193 ceph-mon[37473]: pgmap v8669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:37:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 14995.385186 secs 2024-07-26T15:37:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:35 smithi134 ceph-mon[31966]: pgmap v8669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:37:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 14995.385186 secs 2024-07-26T15:37:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:35 smithi189 ceph-mon[36790]: pgmap v8669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:37:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 14995.385186 secs 2024-07-26T15:37:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:37 smithi193 ceph-mon[37473]: pgmap v8670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:37 smithi134 ceph-mon[31966]: pgmap v8670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:37 smithi189 ceph-mon[36790]: pgmap v8670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:38 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15014.808697 secs 2024-07-26T15:37:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:38 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15014.808697 secs 2024-07-26T15:37:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:38 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15014.808697 secs 2024-07-26T15:37:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:39 smithi193 ceph-mon[37473]: pgmap v8671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:39 smithi134 ceph-mon[31966]: pgmap v8671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:39 smithi189 ceph-mon[36790]: pgmap v8671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:37:40.380+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:37:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:37:40.380+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:37:40.382 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:37:40.380+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15000.385332 secs 2024-07-26T15:37:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15000.385332 secs 2024-07-26T15:37:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15000.385332 secs 2024-07-26T15:37:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:41 smithi134 ceph-mon[31966]: pgmap v8672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:41 smithi189 ceph-mon[36790]: pgmap v8672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:41 smithi193 ceph-mon[37473]: pgmap v8672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:42 smithi134 ceph-mon[31966]: pgmap v8673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:42 smithi189 ceph-mon[36790]: pgmap v8673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:42 smithi193 ceph-mon[37473]: pgmap v8673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:43 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15019.808781 secs 2024-07-26T15:37:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:43 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15019.808781 secs 2024-07-26T15:37:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:43 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15019.808781 secs 2024-07-26T15:37:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:44 smithi134 ceph-mon[31966]: pgmap v8674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:37:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:44 smithi189 ceph-mon[36790]: pgmap v8674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:37:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:44 smithi193 ceph-mon[37473]: pgmap v8674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:37:44.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:37:44.839+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:37:44.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:37:44.839+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:37:44.841 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:37:44.839+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15005.385418 secs 2024-07-26T15:37:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15005.385418 secs 2024-07-26T15:37:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15005.385418 secs 2024-07-26T15:37:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:46 smithi134 ceph-mon[31966]: pgmap v8675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:46 smithi189 ceph-mon[36790]: pgmap v8675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:46 smithi193 ceph-mon[37473]: pgmap v8675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15024.808903 secs 2024-07-26T15:37:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15024.808903 secs 2024-07-26T15:37:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15024.808903 secs 2024-07-26T15:37:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:48 smithi189 ceph-mon[36790]: pgmap v8676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:48 smithi193 ceph-mon[37473]: pgmap v8676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:48 smithi134 ceph-mon[31966]: pgmap v8676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15010.385499 secs 2024-07-26T15:37:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:50 smithi134 ceph-mon[31966]: pgmap v8677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15010.385499 secs 2024-07-26T15:37:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:50 smithi189 ceph-mon[36790]: pgmap v8677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15010.385499 secs 2024-07-26T15:37:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:50 smithi193 ceph-mon[37473]: pgmap v8677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:37:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15029.809034 secs 2024-07-26T15:37:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:52 smithi134 ceph-mon[31966]: pgmap v8678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15029.809034 secs 2024-07-26T15:37:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:52 smithi189 ceph-mon[36790]: pgmap v8678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15029.809034 secs 2024-07-26T15:37:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:52 smithi193 ceph-mon[37473]: pgmap v8678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:54 smithi134 ceph-mon[31966]: pgmap v8679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:37:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:54 smithi189 ceph-mon[36790]: pgmap v8679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:37:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:54 smithi193 ceph-mon[37473]: pgmap v8679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:37:55.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:37:55.383+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:37:55.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:37:55.383+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:37:55.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:37:55.383+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:37:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15015.385627 secs 2024-07-26T15:37:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15015.385627 secs 2024-07-26T15:37:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15015.385627 secs 2024-07-26T15:37:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:56 smithi134 ceph-mon[31966]: pgmap v8680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:56 smithi189 ceph-mon[36790]: pgmap v8680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:56 smithi193 ceph-mon[37473]: pgmap v8680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:37:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15034.809151 secs 2024-07-26T15:37:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15034.809151 secs 2024-07-26T15:37:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15034.809151 secs 2024-07-26T15:37:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:37:58 smithi134 ceph-mon[31966]: pgmap v8681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:37:58 smithi189 ceph-mon[36790]: pgmap v8681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:37:58 smithi193 ceph-mon[37473]: pgmap v8681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:37:59.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:37:59.853+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:37:59.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:37:59.853+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:37:59.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:37:59.853+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15020.385753 secs 2024-07-26T15:38:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:00 smithi134 ceph-mon[31966]: pgmap v8682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15020.385753 secs 2024-07-26T15:38:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:00 smithi189 ceph-mon[36790]: pgmap v8682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15020.385753 secs 2024-07-26T15:38:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:00 smithi193 ceph-mon[37473]: pgmap v8682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15039.809278 secs 2024-07-26T15:38:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:02 smithi134 ceph-mon[31966]: pgmap v8683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15039.809278 secs 2024-07-26T15:38:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:02 smithi189 ceph-mon[36790]: pgmap v8683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15039.809278 secs 2024-07-26T15:38:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:02 smithi193 ceph-mon[37473]: pgmap v8683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:04 smithi134 ceph-mon[31966]: pgmap v8684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:04 smithi189 ceph-mon[36790]: pgmap v8684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:04 smithi193 ceph-mon[37473]: pgmap v8684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15025.385901 secs 2024-07-26T15:38:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15025.385901 secs 2024-07-26T15:38:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15025.385901 secs 2024-07-26T15:38:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:06 smithi189 ceph-mon[36790]: pgmap v8685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:06 smithi193 ceph-mon[37473]: pgmap v8685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:06 smithi134 ceph-mon[31966]: pgmap v8685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15044.809395 secs 2024-07-26T15:38:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15044.809395 secs 2024-07-26T15:38:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15044.809395 secs 2024-07-26T15:38:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:08 smithi193 ceph-mon[37473]: pgmap v8686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:38:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:08 smithi134 ceph-mon[31966]: pgmap v8686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:38:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:08 smithi189 ceph-mon[36790]: pgmap v8686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:38:10.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:38:10.398+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:38:10.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:38:10.398+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:10.400 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:38:10.398+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:38:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15030.386026 secs 2024-07-26T15:38:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:10 smithi193 ceph-mon[37473]: pgmap v8687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15030.386026 secs 2024-07-26T15:38:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:10 smithi134 ceph-mon[31966]: pgmap v8687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15030.386026 secs 2024-07-26T15:38:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:10 smithi189 ceph-mon[36790]: pgmap v8687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:38:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15049.809478 secs 2024-07-26T15:38:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:12 smithi193 ceph-mon[37473]: pgmap v8688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15049.809478 secs 2024-07-26T15:38:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:12 smithi134 ceph-mon[31966]: pgmap v8688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15049.809478 secs 2024-07-26T15:38:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:12 smithi189 ceph-mon[36790]: pgmap v8688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:14 smithi193 ceph-mon[37473]: pgmap v8689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:14.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:38:14.866+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:38:14.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:38:14.866+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:14.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:38:14.866+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:38:14.869 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:14 smithi134 ceph-mon[31966]: pgmap v8689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:14 smithi189 ceph-mon[36790]: pgmap v8689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15035.386148 secs 2024-07-26T15:38:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15035.386148 secs 2024-07-26T15:38:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15035.386148 secs 2024-07-26T15:38:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:16 smithi193 ceph-mon[37473]: pgmap v8690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:16 smithi134 ceph-mon[31966]: pgmap v8690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:16 smithi189 ceph-mon[36790]: pgmap v8690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15054.809608 secs 2024-07-26T15:38:17.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15054.809608 secs 2024-07-26T15:38:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15054.809608 secs 2024-07-26T15:38:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:18 smithi193 ceph-mon[37473]: pgmap v8691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:38:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:38:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:18 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:38:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:18 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:38:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:18 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:38:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:18 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:38:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:38:18.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:38:18.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:18 smithi134 ceph-mon[31966]: pgmap v8691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:38:18.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:38:18.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:18 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:38:18.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:18 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:38:18.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:18 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:38:18.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:18 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:38:18.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:38:18.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:38:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:18 smithi189 ceph-mon[36790]: pgmap v8691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:38:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:38:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:38:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:18 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:38:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:18 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:38:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:18 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:38:18.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:18 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:38:18.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:38:18.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:38:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:19 smithi193 ceph-mon[37473]: pgmap v8692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:38:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:19 smithi193 ceph-mon[37473]: pgmap v8693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:38:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:38:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:19 smithi134 ceph-mon[31966]: pgmap v8692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:38:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:19 smithi134 ceph-mon[31966]: pgmap v8693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:38:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:38:19.947 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:19 smithi189 ceph-mon[36790]: pgmap v8692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:38:19.947 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:19 smithi189 ceph-mon[36790]: pgmap v8693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:38:19.947 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:38:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15040.386243 secs 2024-07-26T15:38:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15040.386243 secs 2024-07-26T15:38:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15040.386243 secs 2024-07-26T15:38:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:21 smithi193 ceph-mon[37473]: pgmap v8694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:38:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:21 smithi134 ceph-mon[31966]: pgmap v8694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:38:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:21 smithi189 ceph-mon[36790]: pgmap v8694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:38:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15059.809674 secs 2024-07-26T15:38:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15059.809674 secs 2024-07-26T15:38:23.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15059.809674 secs 2024-07-26T15:38:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:23 smithi134 ceph-mon[31966]: pgmap v8695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:23 smithi189 ceph-mon[36790]: pgmap v8695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:23 smithi193 ceph-mon[37473]: pgmap v8695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:38:25.413+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:38:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:38:25.413+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:38:25.413+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:38:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15045.386339 secs 2024-07-26T15:38:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:25 smithi134 ceph-mon[31966]: pgmap v8696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:38:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15045.386339 secs 2024-07-26T15:38:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:25 smithi189 ceph-mon[36790]: pgmap v8696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:38:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15045.386339 secs 2024-07-26T15:38:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:25 smithi193 ceph-mon[37473]: pgmap v8696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:38:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15064.809763 secs 2024-07-26T15:38:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:27 smithi134 ceph-mon[31966]: pgmap v8697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:38:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15064.809763 secs 2024-07-26T15:38:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:27 smithi189 ceph-mon[36790]: pgmap v8697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:38:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15064.809763 secs 2024-07-26T15:38:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:27 smithi193 ceph-mon[37473]: pgmap v8697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:38:29.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:38:29.880+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:38:29.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:38:29.880+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:38:29.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:38:29.880+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:29.883 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:29 smithi134 ceph-mon[31966]: pgmap v8698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:38:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:29 smithi189 ceph-mon[36790]: pgmap v8698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:38:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:29 smithi193 ceph-mon[37473]: pgmap v8698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:38:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15050.386459 secs 2024-07-26T15:38:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15050.386459 secs 2024-07-26T15:38:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15050.386459 secs 2024-07-26T15:38:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:31 smithi134 ceph-mon[31966]: pgmap v8699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:31 smithi189 ceph-mon[36790]: pgmap v8699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:31 smithi193 ceph-mon[37473]: pgmap v8699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:32.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15069.809827 secs 2024-07-26T15:38:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15069.809827 secs 2024-07-26T15:38:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15069.809827 secs 2024-07-26T15:38:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:33 smithi134 ceph-mon[31966]: pgmap v8700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:33 smithi189 ceph-mon[36790]: pgmap v8700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:33 smithi193 ceph-mon[37473]: pgmap v8700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15055.386586 secs 2024-07-26T15:38:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15055.386586 secs 2024-07-26T15:38:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15055.386586 secs 2024-07-26T15:38:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:35 smithi134 ceph-mon[31966]: pgmap v8701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:38:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:35 smithi189 ceph-mon[36790]: pgmap v8701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:38:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:35 smithi193 ceph-mon[37473]: pgmap v8701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:38:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15074.809963 secs 2024-07-26T15:38:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:37 smithi134 ceph-mon[31966]: pgmap v8702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15074.809963 secs 2024-07-26T15:38:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:37 smithi189 ceph-mon[36790]: pgmap v8702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15074.809963 secs 2024-07-26T15:38:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:37 smithi193 ceph-mon[37473]: pgmap v8702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:39 smithi134 ceph-mon[31966]: pgmap v8703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15060.386710 secs 2024-07-26T15:38:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:39 smithi189 ceph-mon[36790]: pgmap v8703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15060.386710 secs 2024-07-26T15:38:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:39 smithi193 ceph-mon[37473]: pgmap v8703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15060.386710 secs 2024-07-26T15:38:40.428 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:38:40.427+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:38:40.428 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:38:40.427+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:40.428 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:38:40.427+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:38:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:41 smithi134 ceph-mon[31966]: pgmap v8704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:41 smithi189 ceph-mon[36790]: pgmap v8704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:41 smithi193 ceph-mon[37473]: pgmap v8704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15079.810081 secs 2024-07-26T15:38:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15079.810081 secs 2024-07-26T15:38:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15079.810081 secs 2024-07-26T15:38:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:43 smithi134 ceph-mon[31966]: pgmap v8705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:43 smithi189 ceph-mon[36790]: pgmap v8705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:43 smithi193 ceph-mon[37473]: pgmap v8705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:44.897 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:38:44.896+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:38:44.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:38:44.896+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:38:44.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:38:44.896+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:44.898 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15065.386832 secs 2024-07-26T15:38:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15065.386832 secs 2024-07-26T15:38:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15065.386832 secs 2024-07-26T15:38:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:45 smithi134 ceph-mon[31966]: pgmap v8706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:45 smithi189 ceph-mon[36790]: pgmap v8706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:45 smithi193 ceph-mon[37473]: pgmap v8706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15084.810199 secs 2024-07-26T15:38:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:47 smithi134 ceph-mon[31966]: pgmap v8707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15084.810199 secs 2024-07-26T15:38:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:47 smithi189 ceph-mon[36790]: pgmap v8707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15084.810199 secs 2024-07-26T15:38:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:47 smithi193 ceph-mon[37473]: pgmap v8707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:49 smithi134 ceph-mon[31966]: pgmap v8708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15070.386934 secs 2024-07-26T15:38:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:49 smithi189 ceph-mon[36790]: pgmap v8708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15070.386934 secs 2024-07-26T15:38:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:49 smithi193 ceph-mon[37473]: pgmap v8708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15070.386934 secs 2024-07-26T15:38:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:51 smithi134 ceph-mon[31966]: pgmap v8709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:51 smithi189 ceph-mon[36790]: pgmap v8709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:51 smithi193 ceph-mon[37473]: pgmap v8709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15089.810336 secs 2024-07-26T15:38:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15089.810336 secs 2024-07-26T15:38:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15089.810336 secs 2024-07-26T15:38:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:53 smithi189 ceph-mon[36790]: pgmap v8710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:53 smithi193 ceph-mon[37473]: pgmap v8710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:53 smithi134 ceph-mon[31966]: pgmap v8710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:38:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15075.387035 secs 2024-07-26T15:38:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15075.387035 secs 2024-07-26T15:38:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15075.387035 secs 2024-07-26T15:38:55.443 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:38:55.442+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:38:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:38:55.442+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:38:55.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:38:55.442+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:38:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:55 smithi193 ceph-mon[37473]: pgmap v8711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:55 smithi134 ceph-mon[31966]: pgmap v8711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:55 smithi189 ceph-mon[36790]: pgmap v8711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:38:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15094.810459 secs 2024-07-26T15:38:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:57 smithi193 ceph-mon[37473]: pgmap v8712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15094.810459 secs 2024-07-26T15:38:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:57 smithi134 ceph-mon[31966]: pgmap v8712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15094.810459 secs 2024-07-26T15:38:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:57 smithi189 ceph-mon[36790]: pgmap v8712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:38:59.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:38:59.911+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:38:59.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:38:59.911+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:38:59.913 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:38:59.911+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:59 smithi193 ceph-mon[37473]: pgmap v8713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:38:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15080.387167 secs 2024-07-26T15:39:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:59 smithi134 ceph-mon[31966]: pgmap v8713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:38:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15080.387167 secs 2024-07-26T15:39:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:59 smithi189 ceph-mon[36790]: pgmap v8713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:38:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15080.387167 secs 2024-07-26T15:39:01.975 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:01 smithi134 ceph-mon[31966]: pgmap v8714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:01 smithi193 ceph-mon[37473]: pgmap v8714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:01 smithi189 ceph-mon[36790]: pgmap v8714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15099.810612 secs 2024-07-26T15:39:03.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15099.810612 secs 2024-07-26T15:39:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15099.810612 secs 2024-07-26T15:39:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:03 smithi193 ceph-mon[37473]: pgmap v8715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:03 smithi134 ceph-mon[31966]: pgmap v8715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:03 smithi189 ceph-mon[36790]: pgmap v8715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:05.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15085.387287 secs 2024-07-26T15:39:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15085.387287 secs 2024-07-26T15:39:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15085.387287 secs 2024-07-26T15:39:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:05 smithi193 ceph-mon[37473]: pgmap v8716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:05 smithi134 ceph-mon[31966]: pgmap v8716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:05 smithi189 ceph-mon[36790]: pgmap v8716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15104.810726 secs 2024-07-26T15:39:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:07 smithi193 ceph-mon[37473]: pgmap v8717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15104.810726 secs 2024-07-26T15:39:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:07 smithi134 ceph-mon[31966]: pgmap v8717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15104.810726 secs 2024-07-26T15:39:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:07 smithi189 ceph-mon[36790]: pgmap v8717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:09 smithi193 ceph-mon[37473]: pgmap v8718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15090.387426 secs 2024-07-26T15:39:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:09 smithi134 ceph-mon[31966]: pgmap v8718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15090.387426 secs 2024-07-26T15:39:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:09 smithi189 ceph-mon[36790]: pgmap v8718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15090.387426 secs 2024-07-26T15:39:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:39:10.457+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:39:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:39:10.457+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:10.459 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:39:10.457+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:39:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:11 smithi193 ceph-mon[37473]: pgmap v8719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:11 smithi134 ceph-mon[31966]: pgmap v8719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:11 smithi189 ceph-mon[36790]: pgmap v8719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15109.810853 secs 2024-07-26T15:39:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15109.810853 secs 2024-07-26T15:39:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15109.810853 secs 2024-07-26T15:39:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:13 smithi193 ceph-mon[37473]: pgmap v8720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:13 smithi134 ceph-mon[31966]: pgmap v8720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:13 smithi189 ceph-mon[36790]: pgmap v8720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:14.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:39:14.927+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:39:14.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:39:14.927+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:39:14.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:39:14.927+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15095.387559 secs 2024-07-26T15:39:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15095.387559 secs 2024-07-26T15:39:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15095.387559 secs 2024-07-26T15:39:16.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:15 smithi134 ceph-mon[31966]: pgmap v8721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:15 smithi189 ceph-mon[36790]: pgmap v8721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:15 smithi193 ceph-mon[37473]: pgmap v8721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15114.810996 secs 2024-07-26T15:39:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:17 smithi134 ceph-mon[31966]: pgmap v8722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15114.810996 secs 2024-07-26T15:39:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:17 smithi189 ceph-mon[36790]: pgmap v8722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15114.810996 secs 2024-07-26T15:39:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:17 smithi193 ceph-mon[37473]: pgmap v8722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:39:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:39:19.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:39:20.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:19 smithi134 ceph-mon[31966]: pgmap v8723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:20.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15100.387685 secs 2024-07-26T15:39:20.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:39:20.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:39:20.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:39:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:19 smithi189 ceph-mon[36790]: pgmap v8723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15100.387685 secs 2024-07-26T15:39:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:39:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:39:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:39:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:19 smithi193 ceph-mon[37473]: pgmap v8723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15100.387685 secs 2024-07-26T15:39:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:39:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:39:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:39:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:20 smithi134 ceph-mon[31966]: pgmap v8724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:39:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:20 smithi134 ceph-mon[31966]: pgmap v8725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:39:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:20 smithi189 ceph-mon[36790]: pgmap v8724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:39:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:20 smithi189 ceph-mon[36790]: pgmap v8725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:39:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:20 smithi193 ceph-mon[37473]: pgmap v8724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:39:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:20 smithi193 ceph-mon[37473]: pgmap v8725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:39:23.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15119.811141 secs 2024-07-26T15:39:23.076 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:22 smithi193 ceph-mon[37473]: pgmap v8726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:39:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15119.811141 secs 2024-07-26T15:39:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:22 smithi134 ceph-mon[31966]: pgmap v8726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:39:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15119.811141 secs 2024-07-26T15:39:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:22 smithi189 ceph-mon[36790]: pgmap v8726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:39:24.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:39:24.956+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:39:24.957 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:39:24.956+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:39:24.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:39:24.956+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:39:24.958 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:39:24.956+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 118 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:39:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15105.387835 secs 2024-07-26T15:39:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:24 smithi134 ceph-mon[31966]: pgmap v8727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:39:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15105.387835 secs 2024-07-26T15:39:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:24 smithi189 ceph-mon[36790]: pgmap v8727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:39:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15105.387835 secs 2024-07-26T15:39:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:24 smithi193 ceph-mon[37473]: pgmap v8727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:39:25.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:39:25.472+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:39:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:39:25.472+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:25.474 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:39:25.472+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:39:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:26 smithi134 ceph-mon[31966]: pgmap v8728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:39:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:26 smithi189 ceph-mon[36790]: pgmap v8728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:39:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:26 smithi193 ceph-mon[37473]: pgmap v8728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:39:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15124.811274 secs 2024-07-26T15:39:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15124.811274 secs 2024-07-26T15:39:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15124.811274 secs 2024-07-26T15:39:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:28 smithi134 ceph-mon[31966]: pgmap v8729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:39:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:28 smithi189 ceph-mon[36790]: pgmap v8729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:39:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:28 smithi193 ceph-mon[37473]: pgmap v8729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:39:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:39:29.943+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:39:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:39:29.943+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:29.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:39:29.943+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:39:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15110.387933 secs 2024-07-26T15:39:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15110.387933 secs 2024-07-26T15:39:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15110.387933 secs 2024-07-26T15:39:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:30 smithi134 ceph-mon[31966]: pgmap v8730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:39:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:30 smithi189 ceph-mon[36790]: pgmap v8730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:39:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:30 smithi193 ceph-mon[37473]: pgmap v8730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:39:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15129.811388 secs 2024-07-26T15:39:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:32 smithi134 ceph-mon[31966]: pgmap v8731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15129.811388 secs 2024-07-26T15:39:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:32 smithi189 ceph-mon[36790]: pgmap v8731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15129.811388 secs 2024-07-26T15:39:33.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:32 smithi193 ceph-mon[37473]: pgmap v8731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15115.388054 secs 2024-07-26T15:39:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:34 smithi134 ceph-mon[31966]: pgmap v8732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15115.388054 secs 2024-07-26T15:39:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:34 smithi189 ceph-mon[36790]: pgmap v8732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15115.388054 secs 2024-07-26T15:39:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:34 smithi193 ceph-mon[37473]: pgmap v8732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:35.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:39:35.497+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:39:35.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:39:35.497+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:39:35.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:39:35.497+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:39:35.499 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:39:35.497+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:39:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:36 smithi134 ceph-mon[31966]: pgmap v8733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:36 smithi189 ceph-mon[36790]: pgmap v8733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:36 smithi193 ceph-mon[37473]: pgmap v8733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15134.811500 secs 2024-07-26T15:39:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15134.811500 secs 2024-07-26T15:39:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15134.811500 secs 2024-07-26T15:39:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:38 smithi134 ceph-mon[31966]: pgmap v8734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:38 smithi189 ceph-mon[36790]: pgmap v8734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:38 smithi193 ceph-mon[37473]: pgmap v8734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15120.388183 secs 2024-07-26T15:39:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15120.388183 secs 2024-07-26T15:39:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15120.388183 secs 2024-07-26T15:39:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:39:40.480+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:39:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:39:40.480+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:39:40.482 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:39:40.480+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:40 smithi134 ceph-mon[31966]: pgmap v8735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:40 smithi189 ceph-mon[36790]: pgmap v8735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:40 smithi193 ceph-mon[37473]: pgmap v8735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15139.811589 secs 2024-07-26T15:39:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:42 smithi134 ceph-mon[31966]: pgmap v8736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15139.811589 secs 2024-07-26T15:39:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:42 smithi189 ceph-mon[36790]: pgmap v8736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15139.811589 secs 2024-07-26T15:39:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:42 smithi193 ceph-mon[37473]: pgmap v8736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:44.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:39:44.947+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:39:44.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:39:44.947+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:44.949 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:39:44.947+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:39:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15125.388322 secs 2024-07-26T15:39:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:44 smithi134 ceph-mon[31966]: pgmap v8737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15125.388322 secs 2024-07-26T15:39:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:44 smithi189 ceph-mon[36790]: pgmap v8737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15125.388322 secs 2024-07-26T15:39:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:44 smithi193 ceph-mon[37473]: pgmap v8737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:46 smithi134 ceph-mon[31966]: pgmap v8738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:46 smithi189 ceph-mon[36790]: pgmap v8738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:46 smithi193 ceph-mon[37473]: pgmap v8738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15144.811655 secs 2024-07-26T15:39:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15144.811655 secs 2024-07-26T15:39:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15144.811655 secs 2024-07-26T15:39:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:48 smithi189 ceph-mon[36790]: pgmap v8739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:48 smithi193 ceph-mon[37473]: pgmap v8739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:48 smithi134 ceph-mon[31966]: pgmap v8739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15130.388402 secs 2024-07-26T15:39:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15130.388402 secs 2024-07-26T15:39:50.281 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15130.388402 secs 2024-07-26T15:39:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:50 smithi193 ceph-mon[37473]: pgmap v8740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:50 smithi134 ceph-mon[31966]: pgmap v8740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:50 smithi189 ceph-mon[36790]: pgmap v8740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:39:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15149.811749 secs 2024-07-26T15:39:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:52 smithi193 ceph-mon[37473]: pgmap v8741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15149.811749 secs 2024-07-26T15:39:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:52 smithi134 ceph-mon[31966]: pgmap v8741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15149.811749 secs 2024-07-26T15:39:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:52 smithi189 ceph-mon[36790]: pgmap v8741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15135.388504 secs 2024-07-26T15:39:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:54 smithi193 ceph-mon[37473]: pgmap v8742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15135.388504 secs 2024-07-26T15:39:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:54 smithi134 ceph-mon[31966]: pgmap v8742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15135.388504 secs 2024-07-26T15:39:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:54 smithi189 ceph-mon[36790]: pgmap v8742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:55.487 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:39:55.486+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:39:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:39:55.486+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:39:55.488 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:39:55.486+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:39:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:56 smithi193 ceph-mon[37473]: pgmap v8743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:56 smithi134 ceph-mon[31966]: pgmap v8743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:56 smithi189 ceph-mon[36790]: pgmap v8743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:39:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15154.811860 secs 2024-07-26T15:39:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15154.811860 secs 2024-07-26T15:39:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15154.811860 secs 2024-07-26T15:39:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:58 smithi193 ceph-mon[37473]: pgmap v8744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:58 smithi134 ceph-mon[31966]: pgmap v8744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:58 smithi189 ceph-mon[36790]: pgmap v8744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:39:59.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:39:59.958+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:39:59.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:39:59.958+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:39:59.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:39:59.958+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:39:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15140.388625 secs 2024-07-26T15:40:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:39:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15140.388625 secs 2024-07-26T15:40:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:39:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15140.388625 secs 2024-07-26T15:40:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:00 smithi193 ceph-mon[37473]: pgmap v8745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:40:01.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:00 smithi134 ceph-mon[31966]: pgmap v8745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:01.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:40:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:00 smithi189 ceph-mon[36790]: pgmap v8745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:40:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15159.812003 secs 2024-07-26T15:40:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:02 smithi193 ceph-mon[37473]: pgmap v8746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15159.812003 secs 2024-07-26T15:40:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:02 smithi134 ceph-mon[31966]: pgmap v8746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15159.812003 secs 2024-07-26T15:40:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:02 smithi189 ceph-mon[36790]: pgmap v8746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15145.388760 secs 2024-07-26T15:40:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:05 smithi193 ceph-mon[37473]: pgmap v8747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15145.388760 secs 2024-07-26T15:40:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:05 smithi134 ceph-mon[31966]: pgmap v8747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15145.388760 secs 2024-07-26T15:40:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:05 smithi189 ceph-mon[36790]: pgmap v8747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:07 smithi193 ceph-mon[37473]: pgmap v8748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:07 smithi134 ceph-mon[31966]: pgmap v8748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:07 smithi189 ceph-mon[36790]: pgmap v8748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:08 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15164.812143 secs 2024-07-26T15:40:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:08 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15164.812143 secs 2024-07-26T15:40:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:08 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15164.812143 secs 2024-07-26T15:40:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:09 smithi134 ceph-mon[31966]: pgmap v8749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:09 smithi189 ceph-mon[36790]: pgmap v8749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:09 smithi193 ceph-mon[37473]: pgmap v8749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15150.388877 secs 2024-07-26T15:40:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15150.388877 secs 2024-07-26T15:40:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:40:10.492+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:40:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:40:10.492+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:10.493 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:40:10.492+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:40:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15150.388877 secs 2024-07-26T15:40:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:11 smithi134 ceph-mon[31966]: pgmap v8750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:11 smithi189 ceph-mon[36790]: pgmap v8750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:11 smithi193 ceph-mon[37473]: pgmap v8750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:13 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15169.812273 secs 2024-07-26T15:40:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:13 smithi134 ceph-mon[31966]: pgmap v8751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:13 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15169.812273 secs 2024-07-26T15:40:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:13 smithi189 ceph-mon[36790]: pgmap v8751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:13 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15169.812273 secs 2024-07-26T15:40:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:13 smithi193 ceph-mon[37473]: pgmap v8751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:14.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:40:14.960+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:40:14.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:40:14.960+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:40:14.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:40:14.960+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15155.388993 secs 2024-07-26T15:40:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:15 smithi134 ceph-mon[31966]: pgmap v8752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:40:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15155.388993 secs 2024-07-26T15:40:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:15 smithi189 ceph-mon[36790]: pgmap v8752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:40:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15155.388993 secs 2024-07-26T15:40:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:15 smithi193 ceph-mon[37473]: pgmap v8752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T15:40:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:17 smithi134 ceph-mon[31966]: pgmap v8753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:40:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:17 smithi189 ceph-mon[36790]: pgmap v8753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:40:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:17 smithi193 ceph-mon[37473]: pgmap v8753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:40:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:18 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15174.812396 secs 2024-07-26T15:40:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:18 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15174.812396 secs 2024-07-26T15:40:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:18 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15174.812396 secs 2024-07-26T15:40:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:19 smithi134 ceph-mon[31966]: pgmap v8754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:40:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:19 smithi189 ceph-mon[36790]: pgmap v8754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:40:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:19 smithi193 ceph-mon[37473]: pgmap v8754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:40:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15160.389123 secs 2024-07-26T15:40:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:20 smithi134 ceph-mon[31966]: pgmap v8755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:40:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:40:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15160.389123 secs 2024-07-26T15:40:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:20 smithi189 ceph-mon[36790]: pgmap v8755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:40:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:40:20.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15160.389123 secs 2024-07-26T15:40:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:20 smithi193 ceph-mon[37473]: pgmap v8755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:40:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:40:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:40:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:40:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:40:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:40:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:40:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:40:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15179.812488 secs 2024-07-26T15:40:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:40:22.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:40:22.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:22 smithi134 ceph-mon[31966]: pgmap v8756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:22.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:22 smithi134 ceph-mon[31966]: pgmap v8757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:40:22.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:40:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15179.812488 secs 2024-07-26T15:40:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:40:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:40:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:22 smithi189 ceph-mon[36790]: pgmap v8756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:22 smithi189 ceph-mon[36790]: pgmap v8757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:40:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:40:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15179.812488 secs 2024-07-26T15:40:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:40:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:40:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:22 smithi193 ceph-mon[37473]: pgmap v8756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:22 smithi193 ceph-mon[37473]: pgmap v8757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:40:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:40:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:24 smithi134 ceph-mon[31966]: pgmap v8758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:40:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15165.389236 secs 2024-07-26T15:40:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:24 smithi189 ceph-mon[36790]: pgmap v8758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:40:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15165.389236 secs 2024-07-26T15:40:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:24 smithi193 ceph-mon[37473]: pgmap v8758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:40:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15165.389236 secs 2024-07-26T15:40:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:40:25.507+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:40:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:40:25.507+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:40:25.507+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:40:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:26 smithi134 ceph-mon[31966]: pgmap v8759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:40:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:26 smithi189 ceph-mon[36790]: pgmap v8759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:40:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:26 smithi193 ceph-mon[37473]: pgmap v8759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:40:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15184.812610 secs 2024-07-26T15:40:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15184.812610 secs 2024-07-26T15:40:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15184.812610 secs 2024-07-26T15:40:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:28 smithi134 ceph-mon[31966]: pgmap v8760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:40:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:28 smithi189 ceph-mon[36790]: pgmap v8760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:40:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:28 smithi193 ceph-mon[37473]: pgmap v8760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:40:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15170.389317 secs 2024-07-26T15:40:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15170.389317 secs 2024-07-26T15:40:29.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:40:29.976+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:40:29.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:40:29.976+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:40:29.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:40:29.976+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15170.389317 secs 2024-07-26T15:40:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:30 smithi134 ceph-mon[31966]: pgmap v8761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:40:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:30 smithi189 ceph-mon[36790]: pgmap v8761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:40:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:30 smithi193 ceph-mon[37473]: pgmap v8761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:40:32.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15189.812736 secs 2024-07-26T15:40:32.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:32 smithi134 ceph-mon[31966]: pgmap v8762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:40:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15189.812736 secs 2024-07-26T15:40:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:32 smithi189 ceph-mon[36790]: pgmap v8762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:40:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15189.812736 secs 2024-07-26T15:40:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:32 smithi193 ceph-mon[37473]: pgmap v8762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:40:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:34 smithi134 ceph-mon[31966]: pgmap v8763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15175.389442 secs 2024-07-26T15:40:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:34 smithi189 ceph-mon[36790]: pgmap v8763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15175.389442 secs 2024-07-26T15:40:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:34 smithi193 ceph-mon[37473]: pgmap v8763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15175.389442 secs 2024-07-26T15:40:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:36 smithi134 ceph-mon[31966]: pgmap v8764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:36 smithi189 ceph-mon[36790]: pgmap v8764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:36 smithi193 ceph-mon[37473]: pgmap v8764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15194.812880 secs 2024-07-26T15:40:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15194.812880 secs 2024-07-26T15:40:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15194.812880 secs 2024-07-26T15:40:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:38 smithi134 ceph-mon[31966]: pgmap v8765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:38 smithi189 ceph-mon[36790]: pgmap v8765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:38 smithi193 ceph-mon[37473]: pgmap v8765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15180.389563 secs 2024-07-26T15:40:40.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15180.389563 secs 2024-07-26T15:40:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15180.389563 secs 2024-07-26T15:40:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:40:40.514+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:40:40.515 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:40:40.514+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:40.516 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:40:40.514+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:40:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:40 smithi189 ceph-mon[36790]: pgmap v8766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:40 smithi193 ceph-mon[37473]: pgmap v8766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:40 smithi134 ceph-mon[31966]: pgmap v8766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15199.813023 secs 2024-07-26T15:40:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:42 smithi189 ceph-mon[36790]: pgmap v8767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15199.813023 secs 2024-07-26T15:40:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:42 smithi193 ceph-mon[37473]: pgmap v8767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15199.813023 secs 2024-07-26T15:40:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:42 smithi134 ceph-mon[31966]: pgmap v8767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:44 smithi189 ceph-mon[36790]: pgmap v8768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15185.389660 secs 2024-07-26T15:40:44.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:40:44.991+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:40:44.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:40:44.991+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:44.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:40:44.991+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:40:44.994 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:44 smithi134 ceph-mon[31966]: pgmap v8768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:44.994 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15185.389660 secs 2024-07-26T15:40:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:44 smithi193 ceph-mon[37473]: pgmap v8768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15185.389660 secs 2024-07-26T15:40:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:46 smithi193 ceph-mon[37473]: pgmap v8769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:46 smithi134 ceph-mon[31966]: pgmap v8769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:46 smithi189 ceph-mon[36790]: pgmap v8769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15204.813149 secs 2024-07-26T15:40:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15204.813149 secs 2024-07-26T15:40:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15204.813149 secs 2024-07-26T15:40:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:48 smithi193 ceph-mon[37473]: pgmap v8770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:48 smithi134 ceph-mon[31966]: pgmap v8770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:48 smithi189 ceph-mon[36790]: pgmap v8770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15190.389784 secs 2024-07-26T15:40:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15190.389784 secs 2024-07-26T15:40:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15190.389784 secs 2024-07-26T15:40:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:50 smithi193 ceph-mon[37473]: pgmap v8771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:50 smithi134 ceph-mon[31966]: pgmap v8771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:50 smithi189 ceph-mon[36790]: pgmap v8771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:40:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15209.813276 secs 2024-07-26T15:40:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:52 smithi193 ceph-mon[37473]: pgmap v8772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15209.813276 secs 2024-07-26T15:40:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:52 smithi134 ceph-mon[31966]: pgmap v8772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15209.813276 secs 2024-07-26T15:40:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:52 smithi189 ceph-mon[36790]: pgmap v8772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:54 smithi193 ceph-mon[37473]: pgmap v8773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15195.389915 secs 2024-07-26T15:40:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:54 smithi134 ceph-mon[31966]: pgmap v8773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15195.389915 secs 2024-07-26T15:40:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:54 smithi189 ceph-mon[36790]: pgmap v8773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15195.389915 secs 2024-07-26T15:40:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:40:55.530+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:40:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:40:55.530+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:40:55.531 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:40:55.530+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:40:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:56 smithi193 ceph-mon[37473]: pgmap v8774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:56 smithi134 ceph-mon[31966]: pgmap v8774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:56 smithi189 ceph-mon[36790]: pgmap v8774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:40:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15214.813406 secs 2024-07-26T15:40:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15214.813406 secs 2024-07-26T15:40:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15214.813406 secs 2024-07-26T15:40:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:58 smithi193 ceph-mon[37473]: pgmap v8775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:58 smithi134 ceph-mon[31966]: pgmap v8775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:40:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:58 smithi189 ceph-mon[36790]: pgmap v8775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:00.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:41:00.006+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:41:00.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:41:00.006+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:00.008 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:41:00.006+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:41:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:40:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15200.390034 secs 2024-07-26T15:41:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:40:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15200.390034 secs 2024-07-26T15:41:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:40:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15200.390034 secs 2024-07-26T15:41:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:00 smithi193 ceph-mon[37473]: pgmap v8776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:41:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:00 smithi134 ceph-mon[31966]: pgmap v8776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:41:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:00 smithi189 ceph-mon[36790]: pgmap v8776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:41:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15219.813533 secs 2024-07-26T15:41:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:02 smithi134 ceph-mon[31966]: pgmap v8777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15219.813533 secs 2024-07-26T15:41:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:02 smithi189 ceph-mon[36790]: pgmap v8777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15219.813533 secs 2024-07-26T15:41:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:02 smithi193 ceph-mon[37473]: pgmap v8777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:04 smithi134 ceph-mon[31966]: pgmap v8778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15205.390153 secs 2024-07-26T15:41:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:04 smithi189 ceph-mon[36790]: pgmap v8778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15205.390153 secs 2024-07-26T15:41:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:04 smithi193 ceph-mon[37473]: pgmap v8778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15205.390153 secs 2024-07-26T15:41:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:06 smithi134 ceph-mon[31966]: pgmap v8779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:06 smithi189 ceph-mon[36790]: pgmap v8779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:06 smithi193 ceph-mon[37473]: pgmap v8779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:08.105 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15224.813666 secs 2024-07-26T15:41:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15224.813666 secs 2024-07-26T15:41:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15224.813666 secs 2024-07-26T15:41:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:08 smithi134 ceph-mon[31966]: pgmap v8780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:08 smithi189 ceph-mon[36790]: pgmap v8780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:08 smithi193 ceph-mon[37473]: pgmap v8780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15210.390247 secs 2024-07-26T15:41:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15210.390247 secs 2024-07-26T15:41:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15210.390247 secs 2024-07-26T15:41:10.546 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:41:10.545+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:41:10.547 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:41:10.545+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:41:10.547 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:41:10.545+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:10 smithi134 ceph-mon[31966]: pgmap v8781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:41:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:10 smithi189 ceph-mon[36790]: pgmap v8781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:41:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:10 smithi193 ceph-mon[37473]: pgmap v8781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:41:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15229.813816 secs 2024-07-26T15:41:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:12 smithi134 ceph-mon[31966]: pgmap v8782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15229.813816 secs 2024-07-26T15:41:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:12 smithi189 ceph-mon[36790]: pgmap v8782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15229.813816 secs 2024-07-26T15:41:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:12 smithi193 ceph-mon[37473]: pgmap v8782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:15.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:41:15.018+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:41:15.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:41:15.018+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:41:15.020 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:41:15.018+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15215.390376 secs 2024-07-26T15:41:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:14 smithi134 ceph-mon[31966]: pgmap v8783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15215.390376 secs 2024-07-26T15:41:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:14 smithi189 ceph-mon[36790]: pgmap v8783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15215.390376 secs 2024-07-26T15:41:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:14 smithi193 ceph-mon[37473]: pgmap v8783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:16 smithi134 ceph-mon[31966]: pgmap v8784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:16 smithi189 ceph-mon[36790]: pgmap v8784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:16 smithi193 ceph-mon[37473]: pgmap v8784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15234.813941 secs 2024-07-26T15:41:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15234.813941 secs 2024-07-26T15:41:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15234.813941 secs 2024-07-26T15:41:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:18 smithi134 ceph-mon[31966]: pgmap v8785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:18 smithi189 ceph-mon[36790]: pgmap v8785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:18 smithi193 ceph-mon[37473]: pgmap v8785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15220.390502 secs 2024-07-26T15:41:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15220.390502 secs 2024-07-26T15:41:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15220.390502 secs 2024-07-26T15:41:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:20 smithi134 ceph-mon[31966]: pgmap v8786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:41:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:20 smithi189 ceph-mon[36790]: pgmap v8786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:41:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:20 smithi193 ceph-mon[37473]: pgmap v8786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:41:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:41:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:41:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:41:23.018 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15239.814065 secs 2024-07-26T15:41:23.018 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:22 smithi189 ceph-mon[36790]: pgmap v8787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:23.018 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:23.018 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:23.018 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:23.018 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:23.054 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15239.814065 secs 2024-07-26T15:41:23.054 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:22 smithi193 ceph-mon[37473]: pgmap v8787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:23.055 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:23.055 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:23.055 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:23.055 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15239.814065 secs 2024-07-26T15:41:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:22 smithi134 ceph-mon[31966]: pgmap v8787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:23.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:23.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15225.390600 secs 2024-07-26T15:41:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:24 smithi189 ceph-mon[36790]: pgmap v8788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:24.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15225.390600 secs 2024-07-26T15:41:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:24 smithi193 ceph-mon[37473]: pgmap v8788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15225.390600 secs 2024-07-26T15:41:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:24 smithi134 ceph-mon[31966]: pgmap v8788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:25.561 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:41:25.560+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:41:25.562 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:41:25.560+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:41:25.562 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:41:25.560+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:26 smithi134 ceph-mon[31966]: pgmap v8789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:26 smithi189 ceph-mon[36790]: pgmap v8789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:26 smithi193 ceph-mon[37473]: pgmap v8789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:28 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15244.814196 secs 2024-07-26T15:41:28.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:28.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:28.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:41:28.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:41:28.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:28 smithi134 ceph-mon[31966]: pgmap v8790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:41:28.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:28 smithi134 ceph-mon[31966]: pgmap v8791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T15:41:28.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:28 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15244.814196 secs 2024-07-26T15:41:28.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:28.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:28.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:41:28.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:41:28.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:28 smithi189 ceph-mon[36790]: pgmap v8790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:41:28.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:28 smithi189 ceph-mon[36790]: pgmap v8791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T15:41:28.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:28 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15244.814196 secs 2024-07-26T15:41:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:41:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:41:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:28 smithi193 ceph-mon[37473]: pgmap v8790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:41:28.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:28 smithi193 ceph-mon[37473]: pgmap v8791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T15:41:28.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:41:30.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:41:30.033+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:41:30.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:41:30.033+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:41:30.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:41:30.033+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15230.390747 secs 2024-07-26T15:41:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:30 smithi134 ceph-mon[31966]: pgmap v8792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:41:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15230.390747 secs 2024-07-26T15:41:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:30 smithi193 ceph-mon[37473]: pgmap v8792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:41:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15230.390747 secs 2024-07-26T15:41:30.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:30 smithi189 ceph-mon[36790]: pgmap v8792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:41:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15249.814336 secs 2024-07-26T15:41:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:32 smithi193 ceph-mon[37473]: pgmap v8793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:41:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15249.814336 secs 2024-07-26T15:41:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:32 smithi134 ceph-mon[31966]: pgmap v8793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:41:32.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15249.814336 secs 2024-07-26T15:41:32.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:32 smithi189 ceph-mon[36790]: pgmap v8793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:41:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:34 smithi193 ceph-mon[37473]: pgmap v8794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:41:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:34 smithi134 ceph-mon[31966]: pgmap v8794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:41:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:34 smithi189 ceph-mon[36790]: pgmap v8794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:41:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15235.390809 secs 2024-07-26T15:41:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15235.390809 secs 2024-07-26T15:41:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15235.390809 secs 2024-07-26T15:41:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:36 smithi193 ceph-mon[37473]: pgmap v8795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:41:36.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:36 smithi189 ceph-mon[36790]: pgmap v8795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:41:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:36 smithi134 ceph-mon[31966]: pgmap v8795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:41:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15254.814400 secs 2024-07-26T15:41:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15254.814400 secs 2024-07-26T15:41:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15254.814400 secs 2024-07-26T15:41:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:38 smithi193 ceph-mon[37473]: pgmap v8796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:41:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:38 smithi134 ceph-mon[31966]: pgmap v8796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:41:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:38 smithi189 ceph-mon[36790]: pgmap v8796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:41:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15240.390874 secs 2024-07-26T15:41:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:40 smithi193 ceph-mon[37473]: pgmap v8797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:41:40.572+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:41:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:41:40.572+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:41:40.573 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:41:40.572+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15240.390874 secs 2024-07-26T15:41:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:40 smithi134 ceph-mon[31966]: pgmap v8797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15240.390874 secs 2024-07-26T15:41:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:40 smithi189 ceph-mon[36790]: pgmap v8797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15259.814520 secs 2024-07-26T15:41:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:42 smithi193 ceph-mon[37473]: pgmap v8798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15259.814520 secs 2024-07-26T15:41:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:42 smithi134 ceph-mon[31966]: pgmap v8798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15259.814520 secs 2024-07-26T15:41:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:42 smithi189 ceph-mon[36790]: pgmap v8798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:44 smithi193 ceph-mon[37473]: pgmap v8799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:44 smithi134 ceph-mon[31966]: pgmap v8799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:44 smithi189 ceph-mon[36790]: pgmap v8799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:45.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:41:45.041+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:41:45.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:41:45.041+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:41:45.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:41:45.041+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15245.391001 secs 2024-07-26T15:41:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15245.391001 secs 2024-07-26T15:41:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15245.391001 secs 2024-07-26T15:41:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:46 smithi134 ceph-mon[31966]: pgmap v8800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:46 smithi189 ceph-mon[36790]: pgmap v8800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:46 smithi193 ceph-mon[37473]: pgmap v8800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15264.814640 secs 2024-07-26T15:41:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15264.814640 secs 2024-07-26T15:41:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15264.814640 secs 2024-07-26T15:41:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:48 smithi134 ceph-mon[31966]: pgmap v8801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:41:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:48 smithi189 ceph-mon[36790]: pgmap v8801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:41:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:48 smithi193 ceph-mon[37473]: pgmap v8801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:41:50.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15250.391065 secs 2024-07-26T15:41:50.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:50 smithi134 ceph-mon[31966]: pgmap v8802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15250.391065 secs 2024-07-26T15:41:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:50 smithi189 ceph-mon[36790]: pgmap v8802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15250.391065 secs 2024-07-26T15:41:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:50 smithi193 ceph-mon[37473]: pgmap v8802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:41:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15269.814739 secs 2024-07-26T15:41:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:52 smithi134 ceph-mon[31966]: pgmap v8803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15269.814739 secs 2024-07-26T15:41:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:52 smithi189 ceph-mon[36790]: pgmap v8803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15269.814739 secs 2024-07-26T15:41:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:52 smithi193 ceph-mon[37473]: pgmap v8803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:54 smithi134 ceph-mon[31966]: pgmap v8804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:54 smithi189 ceph-mon[36790]: pgmap v8804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:54 smithi193 ceph-mon[37473]: pgmap v8804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:41:55.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15255.391184 secs 2024-07-26T15:41:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:41:55.582+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:41:55.585 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:41:55.582+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:41:55.585 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:41:55.582+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:41:55.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15255.391184 secs 2024-07-26T15:41:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15255.391184 secs 2024-07-26T15:41:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:56 smithi134 ceph-mon[31966]: pgmap v8805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:56 smithi189 ceph-mon[36790]: pgmap v8805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:56 smithi193 ceph-mon[37473]: pgmap v8805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:41:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15274.814859 secs 2024-07-26T15:41:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15274.814859 secs 2024-07-26T15:41:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15274.814859 secs 2024-07-26T15:41:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:41:58 smithi134 ceph-mon[31966]: pgmap v8806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:41:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:41:58 smithi189 ceph-mon[36790]: pgmap v8806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:41:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:41:58 smithi193 ceph-mon[37473]: pgmap v8806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:00.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:00.052+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:42:00.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:00.052+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:42:00.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:00.052+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:00 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15260.391272 secs 2024-07-26T15:42:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:00 smithi134 ceph-mon[31966]: pgmap v8807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:00 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15260.391272 secs 2024-07-26T15:42:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:00 smithi189 ceph-mon[36790]: pgmap v8807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:00 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15260.391272 secs 2024-07-26T15:42:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:00 smithi193 ceph-mon[37473]: pgmap v8807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15279.814956 secs 2024-07-26T15:42:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:02 smithi134 ceph-mon[31966]: pgmap v8808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15279.814956 secs 2024-07-26T15:42:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:02 smithi189 ceph-mon[36790]: pgmap v8808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15279.814956 secs 2024-07-26T15:42:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:02 smithi193 ceph-mon[37473]: pgmap v8808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:04 smithi134 ceph-mon[31966]: pgmap v8809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:42:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:04 smithi189 ceph-mon[36790]: pgmap v8809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:42:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:04 smithi193 ceph-mon[37473]: pgmap v8809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:42:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:05 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15265.391409 secs 2024-07-26T15:42:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:05 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15265.391409 secs 2024-07-26T15:42:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:05 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15265.391409 secs 2024-07-26T15:42:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:06 smithi134 ceph-mon[31966]: pgmap v8810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:06 smithi189 ceph-mon[36790]: pgmap v8810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:06 smithi193 ceph-mon[37473]: pgmap v8810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15284.815052 secs 2024-07-26T15:42:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15284.815052 secs 2024-07-26T15:42:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15284.815052 secs 2024-07-26T15:42:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:08 smithi134 ceph-mon[31966]: pgmap v8811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:08 smithi189 ceph-mon[36790]: pgmap v8811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:08 smithi193 ceph-mon[37473]: pgmap v8811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:10.598+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:42:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:10.598+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:42:10.599 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:10.598+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:10 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15270.391482 secs 2024-07-26T15:42:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:10 smithi134 ceph-mon[31966]: pgmap v8812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:10 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15270.391482 secs 2024-07-26T15:42:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:10 smithi189 ceph-mon[36790]: pgmap v8812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:10 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15270.391482 secs 2024-07-26T15:42:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:10 smithi193 ceph-mon[37473]: pgmap v8812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15289.815168 secs 2024-07-26T15:42:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:12 smithi134 ceph-mon[31966]: pgmap v8813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15289.815168 secs 2024-07-26T15:42:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:12 smithi189 ceph-mon[36790]: pgmap v8813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15289.815168 secs 2024-07-26T15:42:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:12 smithi193 ceph-mon[37473]: pgmap v8813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:14 smithi134 ceph-mon[31966]: pgmap v8814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:42:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:14 smithi189 ceph-mon[36790]: pgmap v8814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:42:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:14 smithi193 ceph-mon[37473]: pgmap v8814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:42:15.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:15.068+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:42:15.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:15.068+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:15.069 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:15.068+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:42:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:15 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15275.391647 secs 2024-07-26T15:42:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:15 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15275.391647 secs 2024-07-26T15:42:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:15 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15275.391647 secs 2024-07-26T15:42:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:16 smithi189 ceph-mon[36790]: pgmap v8815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:16 smithi193 ceph-mon[37473]: pgmap v8815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:16 smithi134 ceph-mon[31966]: pgmap v8815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15294.815287 secs 2024-07-26T15:42:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15294.815287 secs 2024-07-26T15:42:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15294.815287 secs 2024-07-26T15:42:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:18 smithi189 ceph-mon[36790]: pgmap v8816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:18 smithi193 ceph-mon[37473]: pgmap v8816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:18 smithi134 ceph-mon[31966]: pgmap v8816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15280.391723 secs 2024-07-26T15:42:20.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:20 smithi193 ceph-mon[37473]: pgmap v8817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15280.391723 secs 2024-07-26T15:42:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:20 smithi134 ceph-mon[31966]: pgmap v8817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15280.391723 secs 2024-07-26T15:42:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:20 smithi189 ceph-mon[36790]: pgmap v8817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:22.340 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:22.339+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 msgr2=0x7f9b300c4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T15:42:22.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:22.339+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 msgr2=0x7f9b300c4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T15:42:22.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:22.339+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 0x7f9b300c4130 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9b300b6950 tx=0x7f9b38039050 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:42:22.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:22.339+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 0x7f9b300c4130 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9b300b6950 tx=0x7f9b38039050 comp rx=0 tx=0).stop 2024-07-26T15:42:22.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:22.339+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 msgr2=0x7f9b300c4130 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T15:42:22.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:22.339+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 msgr2=0x7f9b300c4130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:42:22.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:22.339+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c17e0 0x7f9b300c4130 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:42:22.341 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:22.339+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 0x7f9b300c0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:42:22.342 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:22.339+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 0x7f9b300c0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:42:22.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:22.341+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 0x7f9b300c0be0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b38044040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:42:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15299.815386 secs 2024-07-26T15:42:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:22 smithi134 ceph-mon[31966]: pgmap v8818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15299.815386 secs 2024-07-26T15:42:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:22 smithi193 ceph-mon[37473]: pgmap v8818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15299.815386 secs 2024-07-26T15:42:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:22 smithi189 ceph-mon[36790]: pgmap v8818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:24 smithi193 ceph-mon[37473]: pgmap v8819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:42:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:24 smithi134 ceph-mon[31966]: pgmap v8819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:42:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:24 smithi189 ceph-mon[36790]: pgmap v8819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:42:24.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:24.986+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:42:24.988 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:24.986+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 119 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:42:24.993 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:24.991+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:42:24.994 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:24.992+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:42:25.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:25.601+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:42:25.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:25.601+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:25.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:25.601+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:42:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15285.391870 secs 2024-07-26T15:42:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15285.391870 secs 2024-07-26T15:42:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15285.391870 secs 2024-07-26T15:42:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:26 smithi193 ceph-mon[37473]: pgmap v8820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:42:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:26 smithi134 ceph-mon[31966]: pgmap v8820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:42:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:26 smithi189 ceph-mon[36790]: pgmap v8820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:42:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15304.815505 secs 2024-07-26T15:42:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:42:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15304.815505 secs 2024-07-26T15:42:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:42:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15304.815505 secs 2024-07-26T15:42:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:42:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:28 smithi193 ceph-mon[37473]: pgmap v8821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:42:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:42:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:42:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:42:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:28 smithi134 ceph-mon[31966]: pgmap v8821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:42:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:42:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:42:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:42:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:28 smithi189 ceph-mon[36790]: pgmap v8821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:42:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:42:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:42:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:42:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:29 smithi193 ceph-mon[37473]: pgmap v8822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:42:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:29 smithi193 ceph-mon[37473]: pgmap v8823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:42:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:29 smithi134 ceph-mon[31966]: pgmap v8822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:42:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:29 smithi134 ceph-mon[31966]: pgmap v8823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:42:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:29 smithi189 ceph-mon[36790]: pgmap v8822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:42:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:29 smithi189 ceph-mon[36790]: pgmap v8823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:42:30.078 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:30.076+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:42:30.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:30.076+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:30.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:30.076+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:42:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15290.392001 secs 2024-07-26T15:42:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15290.392001 secs 2024-07-26T15:42:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15290.392001 secs 2024-07-26T15:42:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:31 smithi193 ceph-mon[37473]: pgmap v8824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:42:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:31 smithi134 ceph-mon[31966]: pgmap v8824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:42:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:31 smithi189 ceph-mon[36790]: pgmap v8824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:42:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15309.815603 secs 2024-07-26T15:42:32.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15309.815603 secs 2024-07-26T15:42:32.894 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:32.892+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 msgr2=0x7f21ac0c4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000898) for more than 900000000 us, fault. 2024-07-26T15:42:32.894 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:32.892+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2198003490 tx=0x7f21b80c3000 comp rx=0 tx=0).stop 2024-07-26T15:42:32.894 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:32.892+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 msgr2=0x7f21ac0c4210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:42:32.894 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:32.892+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:42:32.894 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:32.892+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:42:32.895 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:32.893+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:42:32.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:32.895+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f21b8005f70 tx=0x7f21b80ba000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:42:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15309.815603 secs 2024-07-26T15:42:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:33 smithi193 ceph-mon[37473]: pgmap v8825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T15:42:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:33 smithi134 ceph-mon[31966]: pgmap v8825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T15:42:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:33 smithi189 ceph-mon[36790]: pgmap v8825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T15:42:35.530 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:35.528+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:42:35.530 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:35.528+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:42:35.538 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:35.537+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:42:35.539 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:35.537+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:42:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:35 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15295.392117 secs 2024-07-26T15:42:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:35 smithi134 ceph-mon[31966]: pgmap v8826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:42:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:35 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15295.392117 secs 2024-07-26T15:42:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:35 smithi189 ceph-mon[36790]: pgmap v8826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:42:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:35 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15295.392117 secs 2024-07-26T15:42:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:35 smithi193 ceph-mon[37473]: pgmap v8826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:42:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15314.815730 secs 2024-07-26T15:42:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:37 smithi134 ceph-mon[31966]: pgmap v8827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:42:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15314.815730 secs 2024-07-26T15:42:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:37 smithi189 ceph-mon[36790]: pgmap v8827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:42:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15314.815730 secs 2024-07-26T15:42:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:37 smithi193 ceph-mon[37473]: pgmap v8827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:42:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:39 smithi134 ceph-mon[31966]: pgmap v8828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:42:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:39 smithi189 ceph-mon[36790]: pgmap v8828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:42:40.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:39 smithi193 ceph-mon[37473]: pgmap v8828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:42:40.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:40.616+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:42:40.618 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:40.616+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:42:40.618 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:40.616+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15300.392204 secs 2024-07-26T15:42:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15300.392204 secs 2024-07-26T15:42:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15300.392204 secs 2024-07-26T15:42:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:41 smithi134 ceph-mon[31966]: pgmap v8829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:41 smithi189 ceph-mon[36790]: pgmap v8829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:41 smithi193 ceph-mon[37473]: pgmap v8829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15319.815825 secs 2024-07-26T15:42:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15319.815825 secs 2024-07-26T15:42:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15319.815825 secs 2024-07-26T15:42:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:43 smithi134 ceph-mon[31966]: pgmap v8830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:42:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:43 smithi189 ceph-mon[36790]: pgmap v8830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:42:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:43 smithi193 ceph-mon[37473]: pgmap v8830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:42:45.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:45.086+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:42:45.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:45.086+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:42:45.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:42:45.086+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15305.392264 secs 2024-07-26T15:42:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:45 smithi134 ceph-mon[31966]: pgmap v8831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15305.392264 secs 2024-07-26T15:42:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:45 smithi189 ceph-mon[36790]: pgmap v8831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15305.392264 secs 2024-07-26T15:42:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:45 smithi193 ceph-mon[37473]: pgmap v8831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15324.815905 secs 2024-07-26T15:42:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:47 smithi134 ceph-mon[31966]: pgmap v8832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15324.815905 secs 2024-07-26T15:42:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:47 smithi189 ceph-mon[36790]: pgmap v8832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:48.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15324.815905 secs 2024-07-26T15:42:48.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:47 smithi193 ceph-mon[37473]: pgmap v8832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:49 smithi134 ceph-mon[31966]: pgmap v8833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:42:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:49 smithi189 ceph-mon[36790]: pgmap v8833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:42:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:49 smithi193 ceph-mon[37473]: pgmap v8833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:42:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15310.392378 secs 2024-07-26T15:42:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15310.392378 secs 2024-07-26T15:42:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15310.392378 secs 2024-07-26T15:42:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:51 smithi134 ceph-mon[31966]: pgmap v8834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15329.816043 secs 2024-07-26T15:42:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:51 smithi189 ceph-mon[36790]: pgmap v8834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15329.816043 secs 2024-07-26T15:42:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:51 smithi193 ceph-mon[37473]: pgmap v8834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15329.816043 secs 2024-07-26T15:42:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:53 smithi134 ceph-mon[31966]: pgmap v8835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:53 smithi189 ceph-mon[36790]: pgmap v8835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:53 smithi193 ceph-mon[37473]: pgmap v8835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:42:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15315.392519 secs 2024-07-26T15:42:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15315.392519 secs 2024-07-26T15:42:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15315.392519 secs 2024-07-26T15:42:55.626 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:55.625+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:42:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:55.625+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:42:55.627 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:42:55.625+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:42:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:55 smithi134 ceph-mon[31966]: pgmap v8836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:55.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:55 smithi193 ceph-mon[37473]: pgmap v8836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:55 smithi189 ceph-mon[36790]: pgmap v8836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:42:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15334.816160 secs 2024-07-26T15:42:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15334.816160 secs 2024-07-26T15:42:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15334.816160 secs 2024-07-26T15:42:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:57 smithi134 ceph-mon[31966]: pgmap v8837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:57 smithi189 ceph-mon[36790]: pgmap v8837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:57 smithi193 ceph-mon[37473]: pgmap v8837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:42:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:59 smithi134 ceph-mon[31966]: pgmap v8838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:42:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:42:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15320.392674 secs 2024-07-26T15:42:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:59 smithi189 ceph-mon[36790]: pgmap v8838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:42:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:42:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15320.392674 secs 2024-07-26T15:43:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:59 smithi193 ceph-mon[37473]: pgmap v8838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:42:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15320.392674 secs 2024-07-26T15:43:00.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:43:00.101+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:43:00.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:43:00.101+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:00.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:43:00.101+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:43:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:01 smithi134 ceph-mon[31966]: pgmap v8839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15339.816280 secs 2024-07-26T15:43:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:01 smithi189 ceph-mon[36790]: pgmap v8839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15339.816280 secs 2024-07-26T15:43:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:01 smithi193 ceph-mon[37473]: pgmap v8839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15339.816280 secs 2024-07-26T15:43:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:03 smithi134 ceph-mon[31966]: pgmap v8840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:03 smithi189 ceph-mon[36790]: pgmap v8840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:03 smithi193 ceph-mon[37473]: pgmap v8840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15325.392808 secs 2024-07-26T15:43:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15325.392808 secs 2024-07-26T15:43:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15325.392808 secs 2024-07-26T15:43:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:05 smithi189 ceph-mon[36790]: pgmap v8841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:05 smithi193 ceph-mon[37473]: pgmap v8841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:05 smithi134 ceph-mon[31966]: pgmap v8841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15344.816400 secs 2024-07-26T15:43:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15344.816400 secs 2024-07-26T15:43:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15344.816400 secs 2024-07-26T15:43:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:07 smithi189 ceph-mon[36790]: pgmap v8842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:07 smithi193 ceph-mon[37473]: pgmap v8842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:07 smithi134 ceph-mon[31966]: pgmap v8842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:09.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:43:09.994+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:43:09.996 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:09 smithi134 ceph-mon[31966]: pgmap v8843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:09.996 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15330.392921 secs 2024-07-26T15:43:09.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:43:09.995+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 120 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:43:10.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:43:09.999+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:43:10.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:43:10.000+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:43:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:09 smithi193 ceph-mon[37473]: pgmap v8843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15330.392921 secs 2024-07-26T15:43:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:09 smithi189 ceph-mon[36790]: pgmap v8843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15330.392921 secs 2024-07-26T15:43:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:43:10.640+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:43:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:43:10.640+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:10.642 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:43:10.640+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:43:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:11 smithi193 ceph-mon[37473]: pgmap v8844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15349.816536 secs 2024-07-26T15:43:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:11 smithi134 ceph-mon[31966]: pgmap v8844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15349.816536 secs 2024-07-26T15:43:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:11 smithi189 ceph-mon[36790]: pgmap v8844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15349.816536 secs 2024-07-26T15:43:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:13 smithi193 ceph-mon[37473]: pgmap v8845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:13 smithi134 ceph-mon[31966]: pgmap v8845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:13 smithi189 ceph-mon[36790]: pgmap v8845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15335.393012 secs 2024-07-26T15:43:15.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:43:15.117+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:43:15.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:43:15.117+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:15.119 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:43:15.117+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:43:15.120 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15335.393012 secs 2024-07-26T15:43:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15335.393012 secs 2024-07-26T15:43:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:15 smithi193 ceph-mon[37473]: pgmap v8846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:15 smithi134 ceph-mon[31966]: pgmap v8846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:15 smithi189 ceph-mon[36790]: pgmap v8846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:17.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15354.816657 secs 2024-07-26T15:43:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15354.816657 secs 2024-07-26T15:43:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15354.816657 secs 2024-07-26T15:43:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:17 smithi193 ceph-mon[37473]: pgmap v8847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:17 smithi134 ceph-mon[31966]: pgmap v8847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:17 smithi189 ceph-mon[36790]: pgmap v8847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:19 smithi193 ceph-mon[37473]: pgmap v8848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15340.393075 secs 2024-07-26T15:43:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:19 smithi134 ceph-mon[31966]: pgmap v8848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15340.393075 secs 2024-07-26T15:43:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:19 smithi189 ceph-mon[36790]: pgmap v8848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15340.393075 secs 2024-07-26T15:43:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:43:20.537+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:43:20.538 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:43:20.537+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:43:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:43:20.538+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:43:20.539 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:43:20.538+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:43:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:21 smithi193 ceph-mon[37473]: pgmap v8849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15359.816778 secs 2024-07-26T15:43:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:21 smithi134 ceph-mon[31966]: pgmap v8849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15359.816778 secs 2024-07-26T15:43:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:21 smithi189 ceph-mon[36790]: pgmap v8849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15359.816778 secs 2024-07-26T15:43:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:23 smithi193 ceph-mon[37473]: pgmap v8850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:23 smithi134 ceph-mon[31966]: pgmap v8850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:23 smithi189 ceph-mon[36790]: pgmap v8850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:43:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15345.393139 secs 2024-07-26T15:43:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15345.393139 secs 2024-07-26T15:43:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15345.393139 secs 2024-07-26T15:43:25.645 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:43:25.643+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:43:25.645 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:43:25.643+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:25.645 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:43:25.643+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:43:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:25 smithi193 ceph-mon[37473]: pgmap v8851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:25 smithi134 ceph-mon[31966]: pgmap v8851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:25 smithi189 ceph-mon[36790]: pgmap v8851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:43:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15364.816909 secs 2024-07-26T15:43:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15364.816909 secs 2024-07-26T15:43:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15364.816909 secs 2024-07-26T15:43:28.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:27 smithi134 ceph-mon[31966]: pgmap v8852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:27 smithi189 ceph-mon[36790]: pgmap v8852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:27 smithi193 ceph-mon[37473]: pgmap v8852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:43:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15350.393210 secs 2024-07-26T15:43:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:43:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15350.393210 secs 2024-07-26T15:43:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:43:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15350.393210 secs 2024-07-26T15:43:30.134 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:43:30.132+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:43:30.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:43:30.132+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:30.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:43:30.132+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:43:30.135 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:29 smithi134 ceph-mon[31966]: pgmap v8853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:30.135 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:43:30.135 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:43:30.135 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:43:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:29 smithi189 ceph-mon[36790]: pgmap v8853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:30.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:43:30.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:43:30.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:43:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:29 smithi193 ceph-mon[37473]: pgmap v8853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:43:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:43:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:43:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:30 smithi134 ceph-mon[31966]: pgmap v8854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:43:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:30 smithi134 ceph-mon[31966]: pgmap v8855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:43:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:30 smithi189 ceph-mon[36790]: pgmap v8854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:43:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:30 smithi189 ceph-mon[36790]: pgmap v8855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:43:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:30 smithi193 ceph-mon[37473]: pgmap v8854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:43:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:30 smithi193 ceph-mon[37473]: pgmap v8855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:43:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15369.817036 secs 2024-07-26T15:43:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15369.817036 secs 2024-07-26T15:43:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15369.817036 secs 2024-07-26T15:43:33.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:32 smithi134 ceph-mon[31966]: pgmap v8856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:43:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:32 smithi189 ceph-mon[36790]: pgmap v8856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:43:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:32 smithi193 ceph-mon[37473]: pgmap v8856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:43:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15355.393290 secs 2024-07-26T15:43:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15355.393290 secs 2024-07-26T15:43:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15355.393290 secs 2024-07-26T15:43:35.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:34 smithi134 ceph-mon[31966]: pgmap v8857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:43:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:34 smithi189 ceph-mon[36790]: pgmap v8857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:43:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:34 smithi193 ceph-mon[37473]: pgmap v8857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:43:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:36 smithi134 ceph-mon[31966]: pgmap v8858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:43:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15374.817161 secs 2024-07-26T15:43:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:36 smithi189 ceph-mon[36790]: pgmap v8858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:43:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15374.817161 secs 2024-07-26T15:43:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:36 smithi193 ceph-mon[37473]: pgmap v8858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:43:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15374.817161 secs 2024-07-26T15:43:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:38 smithi134 ceph-mon[31966]: pgmap v8859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:43:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15360.393385 secs 2024-07-26T15:43:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:38 smithi189 ceph-mon[36790]: pgmap v8859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:43:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15360.393385 secs 2024-07-26T15:43:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:38 smithi193 ceph-mon[37473]: pgmap v8859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:43:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15360.393385 secs 2024-07-26T15:43:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:43:40.659+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:43:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:43:40.659+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:43:40.659+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:43:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:40 smithi134 ceph-mon[31966]: pgmap v8860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:43:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:40 smithi189 ceph-mon[36790]: pgmap v8860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:43:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:40 smithi193 ceph-mon[37473]: pgmap v8860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:43:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15379.817291 secs 2024-07-26T15:43:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15379.817291 secs 2024-07-26T15:43:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15379.817291 secs 2024-07-26T15:43:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:42 smithi134 ceph-mon[31966]: pgmap v8861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:42 smithi189 ceph-mon[36790]: pgmap v8861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:42 smithi193 ceph-mon[37473]: pgmap v8861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15365.393500 secs 2024-07-26T15:43:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15365.393500 secs 2024-07-26T15:43:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15365.393500 secs 2024-07-26T15:43:45.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:43:45.147+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:43:45.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:43:45.147+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:43:45.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:43:45.147+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:45.150 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:44 smithi134 ceph-mon[31966]: pgmap v8862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:44 smithi189 ceph-mon[36790]: pgmap v8862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:44 smithi193 ceph-mon[37473]: pgmap v8862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:46 smithi134 ceph-mon[31966]: pgmap v8863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15384.817411 secs 2024-07-26T15:43:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:46 smithi189 ceph-mon[36790]: pgmap v8863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15384.817411 secs 2024-07-26T15:43:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:46 smithi193 ceph-mon[37473]: pgmap v8863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15384.817411 secs 2024-07-26T15:43:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:48 smithi134 ceph-mon[31966]: pgmap v8864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15370.393567 secs 2024-07-26T15:43:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:48 smithi189 ceph-mon[36790]: pgmap v8864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15370.393567 secs 2024-07-26T15:43:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:48 smithi193 ceph-mon[37473]: pgmap v8864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15370.393567 secs 2024-07-26T15:43:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:50 smithi134 ceph-mon[31966]: pgmap v8865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:43:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:50 smithi189 ceph-mon[36790]: pgmap v8865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:43:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:50 smithi193 ceph-mon[37473]: pgmap v8865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:43:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15389.817531 secs 2024-07-26T15:43:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15389.817531 secs 2024-07-26T15:43:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15389.817531 secs 2024-07-26T15:43:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:52 smithi134 ceph-mon[31966]: pgmap v8866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:52 smithi189 ceph-mon[36790]: pgmap v8866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:52 smithi193 ceph-mon[37473]: pgmap v8866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15375.393673 secs 2024-07-26T15:43:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15375.393673 secs 2024-07-26T15:43:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15375.393673 secs 2024-07-26T15:43:55.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:54 smithi193 ceph-mon[37473]: pgmap v8867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:54 smithi189 ceph-mon[36790]: pgmap v8867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:54 smithi134 ceph-mon[31966]: pgmap v8867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:43:55.674+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:43:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:43:55.674+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:43:55.676 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:43:55.674+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:43:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:56 smithi189 ceph-mon[36790]: pgmap v8868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15394.817661 secs 2024-07-26T15:43:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:56 smithi193 ceph-mon[37473]: pgmap v8868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15394.817661 secs 2024-07-26T15:43:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:56 smithi134 ceph-mon[31966]: pgmap v8868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:43:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15394.817661 secs 2024-07-26T15:43:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:58 smithi189 ceph-mon[36790]: pgmap v8869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:43:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15380.393808 secs 2024-07-26T15:43:59.226 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:58 smithi134 ceph-mon[31966]: pgmap v8869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:59.226 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:43:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15380.393808 secs 2024-07-26T15:43:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:58 smithi193 ceph-mon[37473]: pgmap v8869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:43:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:43:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15380.393808 secs 2024-07-26T15:44:00.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:44:00.149+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:44:00.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:44:00.149+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:44:00.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:44:00.149+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:00 smithi189 ceph-mon[36790]: pgmap v8870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:00 smithi193 ceph-mon[37473]: pgmap v8870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:00 smithi134 ceph-mon[31966]: pgmap v8870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15399.817794 secs 2024-07-26T15:44:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15399.817794 secs 2024-07-26T15:44:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15399.817794 secs 2024-07-26T15:44:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:02 smithi193 ceph-mon[37473]: pgmap v8871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:03.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:02 smithi134 ceph-mon[31966]: pgmap v8871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:02 smithi189 ceph-mon[36790]: pgmap v8871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15385.393926 secs 2024-07-26T15:44:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15385.393926 secs 2024-07-26T15:44:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15385.393926 secs 2024-07-26T15:44:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:04 smithi193 ceph-mon[37473]: pgmap v8872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:04 smithi134 ceph-mon[31966]: pgmap v8872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:04 smithi189 ceph-mon[36790]: pgmap v8872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:06 smithi193 ceph-mon[37473]: pgmap v8873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15404.817893 secs 2024-07-26T15:44:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:06 smithi134 ceph-mon[31966]: pgmap v8873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15404.817893 secs 2024-07-26T15:44:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:06 smithi189 ceph-mon[36790]: pgmap v8873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15404.817893 secs 2024-07-26T15:44:09.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:08 smithi189 ceph-mon[36790]: pgmap v8874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:09.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15390.394043 secs 2024-07-26T15:44:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:08 smithi193 ceph-mon[37473]: pgmap v8874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15390.394043 secs 2024-07-26T15:44:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:08 smithi134 ceph-mon[31966]: pgmap v8874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15390.394043 secs 2024-07-26T15:44:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:44:10.689+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:44:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:44:10.689+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:10.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:44:10.690+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:44:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:10 smithi193 ceph-mon[37473]: pgmap v8875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:10 smithi134 ceph-mon[31966]: pgmap v8875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:10 smithi189 ceph-mon[36790]: pgmap v8875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15409.818039 secs 2024-07-26T15:44:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15409.818039 secs 2024-07-26T15:44:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15409.818039 secs 2024-07-26T15:44:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:13 smithi193 ceph-mon[37473]: pgmap v8876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:13 smithi134 ceph-mon[31966]: pgmap v8876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:13 smithi189 ceph-mon[36790]: pgmap v8876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15395.394171 secs 2024-07-26T15:44:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15395.394171 secs 2024-07-26T15:44:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15395.394171 secs 2024-07-26T15:44:15.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:44:15.165+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:44:15.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:44:15.165+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:15.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:44:15.165+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:44:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:15 smithi193 ceph-mon[37473]: pgmap v8877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:15 smithi134 ceph-mon[31966]: pgmap v8877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:15 smithi189 ceph-mon[36790]: pgmap v8877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:17 smithi193 ceph-mon[37473]: pgmap v8878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15414.818154 secs 2024-07-26T15:44:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:17 smithi134 ceph-mon[31966]: pgmap v8878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15414.818154 secs 2024-07-26T15:44:17.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:17 smithi189 ceph-mon[36790]: pgmap v8878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:17.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15414.818154 secs 2024-07-26T15:44:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:19 smithi134 ceph-mon[31966]: pgmap v8879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:19.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15400.394268 secs 2024-07-26T15:44:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:19 smithi189 ceph-mon[36790]: pgmap v8879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15400.394268 secs 2024-07-26T15:44:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:19 smithi193 ceph-mon[37473]: pgmap v8879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15400.394268 secs 2024-07-26T15:44:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:21 smithi134 ceph-mon[31966]: pgmap v8880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:21 smithi189 ceph-mon[36790]: pgmap v8880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:21.527 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:21 smithi193 ceph-mon[37473]: pgmap v8880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15419.818255 secs 2024-07-26T15:44:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15419.818255 secs 2024-07-26T15:44:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15419.818255 secs 2024-07-26T15:44:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:23 smithi134 ceph-mon[31966]: pgmap v8881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:23 smithi189 ceph-mon[36790]: pgmap v8881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:23 smithi193 ceph-mon[37473]: pgmap v8881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15405.394415 secs 2024-07-26T15:44:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15405.394415 secs 2024-07-26T15:44:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15405.394415 secs 2024-07-26T15:44:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:25 smithi134 ceph-mon[31966]: pgmap v8882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:25.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:25 smithi193 ceph-mon[37473]: pgmap v8882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:25 smithi189 ceph-mon[36790]: pgmap v8882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:44:25.695+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:44:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:44:25.695+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:25.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:44:25.695+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:44:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:27 smithi134 ceph-mon[31966]: pgmap v8883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15424.818356 secs 2024-07-26T15:44:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:27 smithi189 ceph-mon[36790]: pgmap v8883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15424.818356 secs 2024-07-26T15:44:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:27 smithi193 ceph-mon[37473]: pgmap v8883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15424.818356 secs 2024-07-26T15:44:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:29 smithi134 ceph-mon[31966]: pgmap v8884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15410.394535 secs 2024-07-26T15:44:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:29 smithi189 ceph-mon[36790]: pgmap v8884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15410.394535 secs 2024-07-26T15:44:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:29 smithi193 ceph-mon[37473]: pgmap v8884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15410.394535 secs 2024-07-26T15:44:30.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:44:30.180+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:44:30.182 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:44:30.180+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:44:30.183 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:44:30.180+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:30.348 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:44:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:44:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:44:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:31 smithi134 ceph-mon[31966]: pgmap v8885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:44:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:44:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:44:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:31 smithi189 ceph-mon[36790]: pgmap v8885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:44:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:44:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:44:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:31 smithi193 ceph-mon[37473]: pgmap v8885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:44:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:44:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:44:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:32 smithi134 ceph-mon[31966]: pgmap v8886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:44:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:32 smithi134 ceph-mon[31966]: pgmap v8887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15429.818468 secs 2024-07-26T15:44:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:32 smithi189 ceph-mon[36790]: pgmap v8886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:44:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:32 smithi189 ceph-mon[36790]: pgmap v8887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15429.818468 secs 2024-07-26T15:44:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:32 smithi193 ceph-mon[37473]: pgmap v8886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:44:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:32 smithi193 ceph-mon[37473]: pgmap v8887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15429.818468 secs 2024-07-26T15:44:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:34 smithi134 ceph-mon[31966]: pgmap v8888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15415.394666 secs 2024-07-26T15:44:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:34 smithi189 ceph-mon[36790]: pgmap v8888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15415.394666 secs 2024-07-26T15:44:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:34 smithi193 ceph-mon[37473]: pgmap v8888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15415.394666 secs 2024-07-26T15:44:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:36 smithi134 ceph-mon[31966]: pgmap v8889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:44:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:36 smithi189 ceph-mon[36790]: pgmap v8889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:44:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:36 smithi193 ceph-mon[37473]: pgmap v8889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:44:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15434.818565 secs 2024-07-26T15:44:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15434.818565 secs 2024-07-26T15:44:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15434.818565 secs 2024-07-26T15:44:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:38 smithi134 ceph-mon[31966]: pgmap v8890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:44:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:38 smithi189 ceph-mon[36790]: pgmap v8890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:44:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:38 smithi193 ceph-mon[37473]: pgmap v8890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:44:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15420.394797 secs 2024-07-26T15:44:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15420.394797 secs 2024-07-26T15:44:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15420.394797 secs 2024-07-26T15:44:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:40 smithi134 ceph-mon[31966]: pgmap v8891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:40 smithi189 ceph-mon[36790]: pgmap v8891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:40 smithi193 ceph-mon[37473]: pgmap v8891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:44:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:44:40.710+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:44:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:44:40.710+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:40.712 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:44:40.710+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:44:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:41 smithi134 ceph-mon[31966]: pgmap v8892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:44:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:41 smithi189 ceph-mon[36790]: pgmap v8892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:44:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:41 smithi193 ceph-mon[37473]: pgmap v8892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:44:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15439.818689 secs 2024-07-26T15:44:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15439.818689 secs 2024-07-26T15:44:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15439.818689 secs 2024-07-26T15:44:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:43 smithi134 ceph-mon[31966]: pgmap v8893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:43 smithi189 ceph-mon[36790]: pgmap v8893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:43 smithi193 ceph-mon[37473]: pgmap v8893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15425.394906 secs 2024-07-26T15:44:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15425.394906 secs 2024-07-26T15:44:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15425.394906 secs 2024-07-26T15:44:45.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:44:45.188+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:44:45.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:44:45.188+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:45.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:44:45.188+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:44:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:45 smithi189 ceph-mon[36790]: pgmap v8894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:45 smithi193 ceph-mon[37473]: pgmap v8894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:45 smithi134 ceph-mon[31966]: pgmap v8894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15444.818806 secs 2024-07-26T15:44:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15444.818806 secs 2024-07-26T15:44:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15444.818806 secs 2024-07-26T15:44:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:47 smithi189 ceph-mon[36790]: pgmap v8895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:47 smithi193 ceph-mon[37473]: pgmap v8895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:47 smithi134 ceph-mon[31966]: pgmap v8895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15430.395029 secs 2024-07-26T15:44:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15430.395029 secs 2024-07-26T15:44:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15430.395029 secs 2024-07-26T15:44:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:49 smithi189 ceph-mon[36790]: pgmap v8896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:49 smithi193 ceph-mon[37473]: pgmap v8896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:49 smithi134 ceph-mon[31966]: pgmap v8896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:51 smithi134 ceph-mon[31966]: pgmap v8897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15449.818908 secs 2024-07-26T15:44:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:51 smithi189 ceph-mon[36790]: pgmap v8897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15449.818908 secs 2024-07-26T15:44:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:51 smithi193 ceph-mon[37473]: pgmap v8897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15449.818908 secs 2024-07-26T15:44:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:53 smithi134 ceph-mon[31966]: pgmap v8898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15435.395150 secs 2024-07-26T15:44:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:53 smithi189 ceph-mon[36790]: pgmap v8898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15435.395150 secs 2024-07-26T15:44:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:53 smithi193 ceph-mon[37473]: pgmap v8898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:44:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15435.395150 secs 2024-07-26T15:44:55.727 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:44:55.726+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:44:55.728 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:44:55.726+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:44:55.728 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:44:55.726+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:44:56.075 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:55 smithi193 ceph-mon[37473]: pgmap v8899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:55 smithi134 ceph-mon[31966]: pgmap v8899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:56.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:55 smithi189 ceph-mon[36790]: pgmap v8899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:44:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15454.819045 secs 2024-07-26T15:44:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15454.819045 secs 2024-07-26T15:44:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15454.819045 secs 2024-07-26T15:44:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:57 smithi134 ceph-mon[31966]: pgmap v8900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:58.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:57 smithi189 ceph-mon[36790]: pgmap v8900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:57 smithi193 ceph-mon[37473]: pgmap v8900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:44:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15440.395270 secs 2024-07-26T15:44:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15440.395270 secs 2024-07-26T15:44:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15440.395270 secs 2024-07-26T15:45:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:44:59 smithi134 ceph-mon[31966]: pgmap v8901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:44:59 smithi189 ceph-mon[36790]: pgmap v8901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:00.200 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:45:00.199+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:45:00.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:45:00.199+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:45:00.201 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:45:00.199+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:44:59 smithi193 ceph-mon[37473]: pgmap v8901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:02.089 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:01 smithi134 ceph-mon[31966]: pgmap v8902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:02.089 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15459.819166 secs 2024-07-26T15:45:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:01 smithi189 ceph-mon[36790]: pgmap v8902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15459.819166 secs 2024-07-26T15:45:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:01 smithi193 ceph-mon[37473]: pgmap v8902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15459.819166 secs 2024-07-26T15:45:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:03 smithi134 ceph-mon[31966]: pgmap v8903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15445.395389 secs 2024-07-26T15:45:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:03 smithi189 ceph-mon[36790]: pgmap v8903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15445.395389 secs 2024-07-26T15:45:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:03 smithi193 ceph-mon[37473]: pgmap v8903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15445.395389 secs 2024-07-26T15:45:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:05 smithi134 ceph-mon[31966]: pgmap v8904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:45:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:05 smithi189 ceph-mon[36790]: pgmap v8904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:45:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:05 smithi193 ceph-mon[37473]: pgmap v8904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:45:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15464.819285 secs 2024-07-26T15:45:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15464.819285 secs 2024-07-26T15:45:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15464.819285 secs 2024-07-26T15:45:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:07 smithi134 ceph-mon[31966]: pgmap v8905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:07 smithi189 ceph-mon[36790]: pgmap v8905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:07 smithi193 ceph-mon[37473]: pgmap v8905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15450.395508 secs 2024-07-26T15:45:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15450.395508 secs 2024-07-26T15:45:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15450.395508 secs 2024-07-26T15:45:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:09 smithi134 ceph-mon[31966]: pgmap v8906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:09 smithi189 ceph-mon[36790]: pgmap v8906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:09 smithi193 ceph-mon[37473]: pgmap v8906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:45:10.742+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:45:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:45:10.742+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:10.743 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:45:10.742+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:45:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:11 smithi134 ceph-mon[31966]: pgmap v8907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15469.819411 secs 2024-07-26T15:45:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:11 smithi189 ceph-mon[36790]: pgmap v8907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15469.819411 secs 2024-07-26T15:45:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:11 smithi193 ceph-mon[37473]: pgmap v8907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15469.819411 secs 2024-07-26T15:45:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:13 smithi134 ceph-mon[31966]: pgmap v8908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15455.395628 secs 2024-07-26T15:45:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:13 smithi189 ceph-mon[36790]: pgmap v8908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15455.395628 secs 2024-07-26T15:45:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:13 smithi193 ceph-mon[37473]: pgmap v8908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15455.395628 secs 2024-07-26T15:45:15.216 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:45:15.215+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:45:15.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:45:15.215+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:15.217 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:45:15.215+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:45:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:15 smithi134 ceph-mon[31966]: pgmap v8909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:45:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:15 smithi189 ceph-mon[36790]: pgmap v8909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:45:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:15 smithi193 ceph-mon[37473]: pgmap v8909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:45:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15474.819528 secs 2024-07-26T15:45:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15474.819528 secs 2024-07-26T15:45:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15474.819528 secs 2024-07-26T15:45:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:17 smithi134 ceph-mon[31966]: pgmap v8910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:17 smithi189 ceph-mon[36790]: pgmap v8910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:17 smithi193 ceph-mon[37473]: pgmap v8910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15460.395761 secs 2024-07-26T15:45:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15460.395761 secs 2024-07-26T15:45:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15460.395761 secs 2024-07-26T15:45:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:19 smithi189 ceph-mon[36790]: pgmap v8911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:19 smithi193 ceph-mon[37473]: pgmap v8911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:19 smithi134 ceph-mon[31966]: pgmap v8911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:21 smithi189 ceph-mon[36790]: pgmap v8912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15479.819611 secs 2024-07-26T15:45:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:21 smithi193 ceph-mon[37473]: pgmap v8912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15479.819611 secs 2024-07-26T15:45:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:21 smithi134 ceph-mon[31966]: pgmap v8912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15479.819611 secs 2024-07-26T15:45:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:23 smithi189 ceph-mon[36790]: pgmap v8913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:24.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15465.395876 secs 2024-07-26T15:45:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:23 smithi193 ceph-mon[37473]: pgmap v8913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15465.395876 secs 2024-07-26T15:45:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:23 smithi134 ceph-mon[31966]: pgmap v8913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15465.395876 secs 2024-07-26T15:45:25.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:45:25.016+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:45:25.018 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:45:25.017+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 121 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:45:25.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:45:25.020+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:45:25.022 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:45:25.021+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:45:25.759 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:45:25.757+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:45:25.759 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:45:25.757+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:25.759 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:45:25.757+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:45:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:25 smithi189 ceph-mon[36790]: pgmap v8914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:45:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:25 smithi193 ceph-mon[37473]: pgmap v8914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:45:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:25 smithi134 ceph-mon[31966]: pgmap v8914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:45:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15484.819673 secs 2024-07-26T15:45:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15484.819673 secs 2024-07-26T15:45:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15484.819673 secs 2024-07-26T15:45:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:27 smithi193 ceph-mon[37473]: pgmap v8915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:27 smithi134 ceph-mon[31966]: pgmap v8915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:27 smithi189 ceph-mon[36790]: pgmap v8915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15470.395999 secs 2024-07-26T15:45:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15470.395999 secs 2024-07-26T15:45:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15470.395999 secs 2024-07-26T15:45:30.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:45:30.226+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:45:30.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:45:30.226+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:45:30.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:45:30.226+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:30.229 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:29 smithi134 ceph-mon[31966]: pgmap v8916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:29 smithi193 ceph-mon[37473]: pgmap v8916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:29 smithi189 ceph-mon[36790]: pgmap v8916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:45:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:45:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:45:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:31 smithi193 ceph-mon[37473]: pgmap v8917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15489.819737 secs 2024-07-26T15:45:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:45:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:45:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:31 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:45:32.297 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:31 smithi134 ceph-mon[31966]: pgmap v8917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:32.297 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15489.819737 secs 2024-07-26T15:45:32.297 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:45:32.297 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:45:32.297 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:31 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:45:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:31 smithi189 ceph-mon[36790]: pgmap v8917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15489.819737 secs 2024-07-26T15:45:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:45:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:45:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:31 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:45:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:32 smithi193 ceph-mon[37473]: pgmap v8918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:45:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:32 smithi134 ceph-mon[31966]: pgmap v8918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:45:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:32 smithi189 ceph-mon[36790]: pgmap v8918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:45:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15475.396116 secs 2024-07-26T15:45:34.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15475.396116 secs 2024-07-26T15:45:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15475.396116 secs 2024-07-26T15:45:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:35 smithi193 ceph-mon[37473]: pgmap v8919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:45:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:35 smithi134 ceph-mon[31966]: pgmap v8919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:45:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:35 smithi189 ceph-mon[36790]: pgmap v8919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:45:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:45:35.561+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:45:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:45:35.561+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:45:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:45:35.562+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:45:35.563 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:45:35.562+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:45:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:37 smithi193 ceph-mon[37473]: pgmap v8920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:45:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15494.819808 secs 2024-07-26T15:45:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:37 smithi134 ceph-mon[31966]: pgmap v8920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:45:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15494.819808 secs 2024-07-26T15:45:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:37 smithi189 ceph-mon[36790]: pgmap v8920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:45:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15494.819808 secs 2024-07-26T15:45:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:39 smithi193 ceph-mon[37473]: pgmap v8921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T15:45:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15480.396189 secs 2024-07-26T15:45:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:39 smithi134 ceph-mon[31966]: pgmap v8921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T15:45:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15480.396189 secs 2024-07-26T15:45:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:39 smithi189 ceph-mon[36790]: pgmap v8921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T15:45:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15480.396189 secs 2024-07-26T15:45:40.774 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:45:40.773+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:45:40.774 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:45:40.773+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:45:40.774 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:45:40.773+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:41 smithi134 ceph-mon[31966]: pgmap v8922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:45:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:41 smithi189 ceph-mon[36790]: pgmap v8922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:45:41.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:41 smithi193 ceph-mon[37473]: pgmap v8922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:45:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15499.819907 secs 2024-07-26T15:45:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15499.819907 secs 2024-07-26T15:45:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15499.819907 secs 2024-07-26T15:45:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:43 smithi134 ceph-mon[31966]: pgmap v8923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:45:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:43 smithi189 ceph-mon[36790]: pgmap v8923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:45:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:43 smithi193 ceph-mon[37473]: pgmap v8923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:45:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15485.396312 secs 2024-07-26T15:45:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15485.396312 secs 2024-07-26T15:45:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15485.396312 secs 2024-07-26T15:45:45.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:45:45.236+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:45:45.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:45:45.236+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:45.239 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:45:45.236+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:45:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:45 smithi134 ceph-mon[31966]: pgmap v8924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:45 smithi189 ceph-mon[36790]: pgmap v8924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:45 smithi193 ceph-mon[37473]: pgmap v8924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:47 smithi134 ceph-mon[31966]: pgmap v8925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15504.819979 secs 2024-07-26T15:45:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:47 smithi189 ceph-mon[36790]: pgmap v8925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15504.819979 secs 2024-07-26T15:45:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:47 smithi193 ceph-mon[37473]: pgmap v8925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15504.819979 secs 2024-07-26T15:45:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:49 smithi134 ceph-mon[31966]: pgmap v8926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15490.396454 secs 2024-07-26T15:45:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:49 smithi189 ceph-mon[36790]: pgmap v8926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15490.396454 secs 2024-07-26T15:45:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:49 smithi193 ceph-mon[37473]: pgmap v8926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15490.396454 secs 2024-07-26T15:45:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:51 smithi134 ceph-mon[31966]: pgmap v8927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:45:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:51 smithi189 ceph-mon[36790]: pgmap v8927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:45:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:51 smithi193 ceph-mon[37473]: pgmap v8927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:45:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15509.820067 secs 2024-07-26T15:45:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15509.820067 secs 2024-07-26T15:45:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15509.820067 secs 2024-07-26T15:45:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:53 smithi134 ceph-mon[31966]: pgmap v8928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:53 smithi189 ceph-mon[36790]: pgmap v8928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:53 smithi193 ceph-mon[37473]: pgmap v8928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15495.396575 secs 2024-07-26T15:45:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15495.396575 secs 2024-07-26T15:45:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15495.396575 secs 2024-07-26T15:45:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:55 smithi134 ceph-mon[31966]: pgmap v8929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:55 smithi189 ceph-mon[36790]: pgmap v8929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:55 smithi193 ceph-mon[37473]: pgmap v8929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:55.790 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:45:55.789+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:45:55.790 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:45:55.789+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:45:55.790 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:45:55.789+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:45:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:57 smithi134 ceph-mon[31966]: pgmap v8930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15514.820161 secs 2024-07-26T15:45:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:57 smithi189 ceph-mon[36790]: pgmap v8930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15514.820161 secs 2024-07-26T15:45:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:57 smithi193 ceph-mon[37473]: pgmap v8930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:45:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15514.820161 secs 2024-07-26T15:45:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:59 smithi134 ceph-mon[31966]: pgmap v8931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:45:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15500.396674 secs 2024-07-26T15:45:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:59 smithi189 ceph-mon[36790]: pgmap v8931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:45:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15500.396674 secs 2024-07-26T15:45:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:59 smithi193 ceph-mon[37473]: pgmap v8931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:45:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:45:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15500.396674 secs 2024-07-26T15:46:00.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:46:00.244+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:46:00.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:46:00.244+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:46:00.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:46:00.244+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:01 smithi134 ceph-mon[31966]: pgmap v8932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:46:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:01 smithi189 ceph-mon[36790]: pgmap v8932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:46:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:01 smithi193 ceph-mon[37473]: pgmap v8932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:46:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15519.820294 secs 2024-07-26T15:46:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15519.820294 secs 2024-07-26T15:46:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15519.820294 secs 2024-07-26T15:46:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:03 smithi134 ceph-mon[31966]: pgmap v8933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:03 smithi189 ceph-mon[36790]: pgmap v8933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:03 smithi193 ceph-mon[37473]: pgmap v8933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:04.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15505.396829 secs 2024-07-26T15:46:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15505.396829 secs 2024-07-26T15:46:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15505.396829 secs 2024-07-26T15:46:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:05 smithi134 ceph-mon[31966]: pgmap v8934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:05 smithi189 ceph-mon[36790]: pgmap v8934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:05 smithi193 ceph-mon[37473]: pgmap v8934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:06 smithi134 ceph-mon[31966]: pgmap v8935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:06 smithi189 ceph-mon[36790]: pgmap v8935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:06 smithi193 ceph-mon[37473]: pgmap v8935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15524.820353 secs 2024-07-26T15:46:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15524.820353 secs 2024-07-26T15:46:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15524.820353 secs 2024-07-26T15:46:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:08 smithi189 ceph-mon[36790]: pgmap v8936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:08 smithi193 ceph-mon[37473]: pgmap v8936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:08 smithi134 ceph-mon[31966]: pgmap v8936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15510.396996 secs 2024-07-26T15:46:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15510.396996 secs 2024-07-26T15:46:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15510.396996 secs 2024-07-26T15:46:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:10 smithi189 ceph-mon[36790]: pgmap v8937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:46:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:10 smithi193 ceph-mon[37473]: pgmap v8937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:46:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:10 smithi134 ceph-mon[31966]: pgmap v8937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:46:10.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:46:10.803+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:46:10.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:46:10.804+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:46:10.805 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:46:10.804+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15529.820487 secs 2024-07-26T15:46:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15529.820487 secs 2024-07-26T15:46:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15529.820487 secs 2024-07-26T15:46:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:12 smithi193 ceph-mon[37473]: pgmap v8938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:12 smithi134 ceph-mon[31966]: pgmap v8938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:12 smithi189 ceph-mon[36790]: pgmap v8938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15515.397129 secs 2024-07-26T15:46:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15515.397129 secs 2024-07-26T15:46:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15515.397129 secs 2024-07-26T15:46:15.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:14 smithi193 ceph-mon[37473]: pgmap v8939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:14 smithi134 ceph-mon[31966]: pgmap v8939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:14 smithi189 ceph-mon[36790]: pgmap v8939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:15.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:46:15.244+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:46:15.246 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:46:15.244+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:46:15.247 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:46:15.244+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:16 smithi193 ceph-mon[37473]: pgmap v8940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15534.820544 secs 2024-07-26T15:46:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:16 smithi134 ceph-mon[31966]: pgmap v8940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15534.820544 secs 2024-07-26T15:46:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:16 smithi189 ceph-mon[36790]: pgmap v8940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15534.820544 secs 2024-07-26T15:46:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:18 smithi134 ceph-mon[31966]: pgmap v8941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15520.397194 secs 2024-07-26T15:46:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:18 smithi193 ceph-mon[37473]: pgmap v8941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15520.397194 secs 2024-07-26T15:46:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:18 smithi189 ceph-mon[36790]: pgmap v8941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15520.397194 secs 2024-07-26T15:46:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:20 smithi193 ceph-mon[37473]: pgmap v8942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:46:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:20 smithi134 ceph-mon[31966]: pgmap v8942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:46:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:20 smithi189 ceph-mon[36790]: pgmap v8942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:46:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15539.820714 secs 2024-07-26T15:46:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15539.820714 secs 2024-07-26T15:46:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15539.820714 secs 2024-07-26T15:46:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:23 smithi193 ceph-mon[37473]: pgmap v8943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:23 smithi134 ceph-mon[31966]: pgmap v8943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:23 smithi189 ceph-mon[36790]: pgmap v8943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15525.397292 secs 2024-07-26T15:46:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15525.397292 secs 2024-07-26T15:46:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15525.397292 secs 2024-07-26T15:46:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:25 smithi134 ceph-mon[31966]: pgmap v8944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:25 smithi189 ceph-mon[36790]: pgmap v8944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:25 smithi193 ceph-mon[37473]: pgmap v8944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:46:25.819+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:46:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:46:25.819+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:25.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:46:25.819+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:46:27.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:27 smithi193 ceph-mon[37473]: pgmap v8945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:27.326 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15544.820771 secs 2024-07-26T15:46:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:27 smithi134 ceph-mon[31966]: pgmap v8945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15544.820771 secs 2024-07-26T15:46:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:27 smithi189 ceph-mon[36790]: pgmap v8945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15544.820771 secs 2024-07-26T15:46:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:29 smithi134 ceph-mon[31966]: pgmap v8946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15530.397422 secs 2024-07-26T15:46:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:29 smithi189 ceph-mon[36790]: pgmap v8946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15530.397422 secs 2024-07-26T15:46:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:29 smithi193 ceph-mon[37473]: pgmap v8946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15530.397422 secs 2024-07-26T15:46:30.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:46:30.260+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:46:30.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:46:30.260+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:30.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:46:30.260+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:46:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:31 smithi134 ceph-mon[31966]: pgmap v8947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:46:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:31 smithi189 ceph-mon[36790]: pgmap v8947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:46:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:31 smithi193 ceph-mon[37473]: pgmap v8947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:46:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15549.820912 secs 2024-07-26T15:46:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15549.820912 secs 2024-07-26T15:46:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15549.820912 secs 2024-07-26T15:46:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:33 smithi134 ceph-mon[31966]: pgmap v8948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:33 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:46:33.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:33 smithi189 ceph-mon[36790]: pgmap v8948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:33.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:33 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:46:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:33 smithi193 ceph-mon[37473]: pgmap v8948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:33 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:46:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15535.397548 secs 2024-07-26T15:46:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15535.397548 secs 2024-07-26T15:46:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15535.397548 secs 2024-07-26T15:46:35.098 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:35 smithi134 ceph-mon[31966]: pgmap v8949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:35 smithi189 ceph-mon[36790]: pgmap v8949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:35 smithi193 ceph-mon[37473]: pgmap v8949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:37.389 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:37 smithi134 ceph-mon[31966]: pgmap v8950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:37.389 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15554.820959 secs 2024-07-26T15:46:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:37 smithi189 ceph-mon[36790]: pgmap v8950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15554.820959 secs 2024-07-26T15:46:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:37 smithi193 ceph-mon[37473]: pgmap v8950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15554.820959 secs 2024-07-26T15:46:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:39 smithi134 ceph-mon[31966]: pgmap v8951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:46:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:46:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:46:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:46:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:46:39.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15540.397666 secs 2024-07-26T15:46:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:39 smithi189 ceph-mon[36790]: pgmap v8951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:46:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:46:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:46:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:46:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:46:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15540.397666 secs 2024-07-26T15:46:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:39 smithi193 ceph-mon[37473]: pgmap v8951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:46:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:46:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:46:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:46:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:46:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15540.397666 secs 2024-07-26T15:46:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:40 smithi134 ceph-mon[31966]: pgmap v8952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:46:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:40 smithi134 ceph-mon[31966]: pgmap v8953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:46:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:40 smithi189 ceph-mon[36790]: pgmap v8952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:46:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:40 smithi189 ceph-mon[36790]: pgmap v8953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:46:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:40 smithi193 ceph-mon[37473]: pgmap v8952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:46:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:40 smithi193 ceph-mon[37473]: pgmap v8953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:46:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:46:40.830+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:46:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:46:40.830+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:40.832 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:46:40.830+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:46:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:42 smithi134 ceph-mon[31966]: pgmap v8954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:46:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15559.821089 secs 2024-07-26T15:46:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:42 smithi189 ceph-mon[36790]: pgmap v8954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:46:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15559.821089 secs 2024-07-26T15:46:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:42 smithi193 ceph-mon[37473]: pgmap v8954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s 2024-07-26T15:46:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15559.821089 secs 2024-07-26T15:46:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:44 smithi134 ceph-mon[31966]: pgmap v8955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T15:46:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15545.397789 secs 2024-07-26T15:46:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:44 smithi189 ceph-mon[36790]: pgmap v8955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T15:46:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15545.397789 secs 2024-07-26T15:46:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:44 smithi193 ceph-mon[37473]: pgmap v8955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-07-26T15:46:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15545.397789 secs 2024-07-26T15:46:45.278 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:46:45.275+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:46:45.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:46:45.275+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:45.279 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:46:45.275+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:46:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:46 smithi134 ceph-mon[31966]: pgmap v8956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:46:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:46 smithi189 ceph-mon[36790]: pgmap v8956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:46:46.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:46 smithi193 ceph-mon[37473]: pgmap v8956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:46:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15564.821209 secs 2024-07-26T15:46:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15564.821209 secs 2024-07-26T15:46:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15564.821209 secs 2024-07-26T15:46:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:48 smithi134 ceph-mon[31966]: pgmap v8957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T15:46:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:48 smithi189 ceph-mon[36790]: pgmap v8957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T15:46:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:48 smithi193 ceph-mon[37473]: pgmap v8957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-07-26T15:46:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15550.397887 secs 2024-07-26T15:46:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15550.397887 secs 2024-07-26T15:46:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15550.397887 secs 2024-07-26T15:46:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:50 smithi134 ceph-mon[31966]: pgmap v8958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:46:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:50 smithi189 ceph-mon[36790]: pgmap v8958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:46:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:50 smithi193 ceph-mon[37473]: pgmap v8958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:46:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:52 smithi134 ceph-mon[31966]: pgmap v8959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15569.821346 secs 2024-07-26T15:46:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:52 smithi189 ceph-mon[36790]: pgmap v8959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15569.821346 secs 2024-07-26T15:46:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:52 smithi193 ceph-mon[37473]: pgmap v8959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15569.821346 secs 2024-07-26T15:46:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:54 smithi134 ceph-mon[31966]: pgmap v8960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15555.397986 secs 2024-07-26T15:46:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:54 smithi189 ceph-mon[36790]: pgmap v8960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15555.397986 secs 2024-07-26T15:46:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:54 smithi193 ceph-mon[37473]: pgmap v8960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:46:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15555.397986 secs 2024-07-26T15:46:55.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:55 smithi193 ceph-mon[37473]: pgmap v8961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:55 smithi134 ceph-mon[31966]: pgmap v8961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:55 smithi189 ceph-mon[36790]: pgmap v8961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:46:55.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:46:55.846+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:46:55.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:46:55.846+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:46:55.848 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:46:55.846+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:46:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15574.821462 secs 2024-07-26T15:46:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15574.821462 secs 2024-07-26T15:46:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15574.821462 secs 2024-07-26T15:46:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:57 smithi193 ceph-mon[37473]: pgmap v8962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:57 smithi134 ceph-mon[31966]: pgmap v8962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:57 smithi189 ceph-mon[36790]: pgmap v8962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:46:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:59 smithi134 ceph-mon[31966]: pgmap v8963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:46:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15560.398116 secs 2024-07-26T15:46:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:59 smithi189 ceph-mon[36790]: pgmap v8963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:46:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15560.398116 secs 2024-07-26T15:46:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:59 smithi193 ceph-mon[37473]: pgmap v8963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:46:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:46:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15560.398116 secs 2024-07-26T15:47:00.281 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:47:00.279+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:47:00.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:47:00.279+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:47:00.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:47:00.280+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:01 smithi134 ceph-mon[31966]: pgmap v8964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15579.821577 secs 2024-07-26T15:47:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:01 smithi189 ceph-mon[36790]: pgmap v8964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15579.821577 secs 2024-07-26T15:47:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:01 smithi193 ceph-mon[37473]: pgmap v8964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15579.821577 secs 2024-07-26T15:47:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:03 smithi134 ceph-mon[31966]: pgmap v8965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:03 smithi189 ceph-mon[36790]: pgmap v8965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:03 smithi193 ceph-mon[37473]: pgmap v8965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:04.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15565.398246 secs 2024-07-26T15:47:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15565.398246 secs 2024-07-26T15:47:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15565.398246 secs 2024-07-26T15:47:05.595 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:05 smithi134 ceph-mon[31966]: pgmap v8966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:05 smithi189 ceph-mon[36790]: pgmap v8966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:05 smithi193 ceph-mon[37473]: pgmap v8966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15584.821671 secs 2024-07-26T15:47:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15584.821671 secs 2024-07-26T15:47:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15584.821671 secs 2024-07-26T15:47:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:07 smithi134 ceph-mon[31966]: pgmap v8967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:07 smithi189 ceph-mon[36790]: pgmap v8967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:07 smithi193 ceph-mon[37473]: pgmap v8967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:09 smithi134 ceph-mon[31966]: pgmap v8968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:47:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15570.398390 secs 2024-07-26T15:47:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:09 smithi189 ceph-mon[36790]: pgmap v8968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:47:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15570.398390 secs 2024-07-26T15:47:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:09 smithi193 ceph-mon[37473]: pgmap v8968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:47:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15570.398390 secs 2024-07-26T15:47:10.863 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:47:10.862+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:47:10.863 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:47:10.862+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:47:10.864 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:47:10.862+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:11 smithi134 ceph-mon[31966]: pgmap v8969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15589.821755 secs 2024-07-26T15:47:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:11 smithi189 ceph-mon[36790]: pgmap v8969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:11.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15589.821755 secs 2024-07-26T15:47:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:11 smithi193 ceph-mon[37473]: pgmap v8969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15589.821755 secs 2024-07-26T15:47:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:13 smithi134 ceph-mon[31966]: pgmap v8970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:13 smithi189 ceph-mon[36790]: pgmap v8970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:13 smithi193 ceph-mon[37473]: pgmap v8970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15575.398516 secs 2024-07-26T15:47:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15575.398516 secs 2024-07-26T15:47:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15575.398516 secs 2024-07-26T15:47:15.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:47:15.288+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:47:15.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:47:15.288+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:47:15.290 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:47:15.288+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:15 smithi134 ceph-mon[31966]: pgmap v8971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:15 smithi189 ceph-mon[36790]: pgmap v8971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:15 smithi193 ceph-mon[37473]: pgmap v8971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:16.632 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15594.821907 secs 2024-07-26T15:47:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15594.821907 secs 2024-07-26T15:47:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15594.821907 secs 2024-07-26T15:47:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:17 smithi134 ceph-mon[31966]: pgmap v8972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:17 smithi189 ceph-mon[36790]: pgmap v8972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:17 smithi193 ceph-mon[37473]: pgmap v8972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:19 smithi134 ceph-mon[31966]: pgmap v8973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:47:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15580.398643 secs 2024-07-26T15:47:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:19 smithi189 ceph-mon[36790]: pgmap v8973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:47:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15580.398643 secs 2024-07-26T15:47:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:19 smithi193 ceph-mon[37473]: pgmap v8973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:47:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15580.398643 secs 2024-07-26T15:47:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:21 smithi134 ceph-mon[31966]: pgmap v8974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15599.822022 secs 2024-07-26T15:47:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:21 smithi189 ceph-mon[36790]: pgmap v8974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15599.822022 secs 2024-07-26T15:47:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:21 smithi193 ceph-mon[37473]: pgmap v8974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15599.822022 secs 2024-07-26T15:47:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:23 smithi134 ceph-mon[31966]: pgmap v8975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:23 smithi189 ceph-mon[36790]: pgmap v8975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:23 smithi193 ceph-mon[37473]: pgmap v8975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:47:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15585.398725 secs 2024-07-26T15:47:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15585.398725 secs 2024-07-26T15:47:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15585.398725 secs 2024-07-26T15:47:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:25 smithi134 ceph-mon[31966]: pgmap v8976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:25 smithi189 ceph-mon[36790]: pgmap v8976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:25.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:25 smithi193 ceph-mon[37473]: pgmap v8976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:47:25.879 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:47:25.878+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:47:25.879 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:47:25.878+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:47:25.879 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:47:25.878+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15604.822121 secs 2024-07-26T15:47:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15604.822121 secs 2024-07-26T15:47:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15604.822121 secs 2024-07-26T15:47:27.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:27 smithi193 ceph-mon[37473]: pgmap v8977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:27 smithi189 ceph-mon[36790]: pgmap v8977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:27 smithi134 ceph-mon[31966]: pgmap v8977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:29 smithi189 ceph-mon[36790]: pgmap v8978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:47:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15590.398857 secs 2024-07-26T15:47:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:29 smithi193 ceph-mon[37473]: pgmap v8978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:47:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15590.398857 secs 2024-07-26T15:47:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:29 smithi134 ceph-mon[31966]: pgmap v8978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:47:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15590.398857 secs 2024-07-26T15:47:30.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:47:30.292+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:47:30.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:47:30.292+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:47:30.294 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:47:30.292+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:31 smithi189 ceph-mon[36790]: pgmap v8979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15609.822247 secs 2024-07-26T15:47:31.773 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:31 smithi193 ceph-mon[37473]: pgmap v8979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15609.822247 secs 2024-07-26T15:47:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:31 smithi134 ceph-mon[31966]: pgmap v8979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:47:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15609.822247 secs 2024-07-26T15:47:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:33 smithi189 ceph-mon[36790]: pgmap v8980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:47:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:33 smithi193 ceph-mon[37473]: pgmap v8980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:47:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:33 smithi134 ceph-mon[31966]: pgmap v8980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:47:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15595.399020 secs 2024-07-26T15:47:34.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15595.399020 secs 2024-07-26T15:47:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15595.399020 secs 2024-07-26T15:47:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:35 smithi193 ceph-mon[37473]: pgmap v8981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:47:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:35 smithi134 ceph-mon[31966]: pgmap v8981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:47:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:35 smithi189 ceph-mon[36790]: pgmap v8981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:47:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15614.822381 secs 2024-07-26T15:47:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15614.822381 secs 2024-07-26T15:47:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15614.822381 secs 2024-07-26T15:47:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:37 smithi193 ceph-mon[37473]: pgmap v8982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:47:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:37 smithi134 ceph-mon[31966]: pgmap v8982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:47:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:37 smithi189 ceph-mon[36790]: pgmap v8982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:47:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:47:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:47:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:47:39.623 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:39 smithi193 ceph-mon[37473]: pgmap v8983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:47:39.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15600.399094 secs 2024-07-26T15:47:39.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:39.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:39.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:39.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:39 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:39.665 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:39 smithi189 ceph-mon[36790]: pgmap v8983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:47:39.666 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15600.399094 secs 2024-07-26T15:47:39.666 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:39.666 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:39.666 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:39.666 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:39 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:39 smithi134 ceph-mon[31966]: pgmap v8983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:47:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15600.399094 secs 2024-07-26T15:47:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:39.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:39.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:39.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:39 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:47:40.688 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:40 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:47:40.688 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:40 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:47:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:40 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:47:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:40 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:47:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:47:40.886+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:47:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:47:40.886+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:47:40.886+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:47:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:40 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:47:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:40 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:47:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:41 smithi193 ceph-mon[37473]: pgmap v8984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:47:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15619.822463 secs 2024-07-26T15:47:41.860 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:41 smithi189 ceph-mon[36790]: pgmap v8984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:47:41.860 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15619.822463 secs 2024-07-26T15:47:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:41 smithi134 ceph-mon[31966]: pgmap v8984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:47:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15619.822463 secs 2024-07-26T15:47:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:47:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:47:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:43 smithi193 ceph-mon[37473]: pgmap v8985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T15:47:43.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:47:43.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:47:43.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:47:43.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:47:43.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:43 smithi193 ceph-mon[37473]: pgmap v8986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:47:43.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:43 smithi193 ceph-mon[37473]: pgmap v8987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T15:47:43.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:47:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:47:43.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:47:43.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:43 smithi134 ceph-mon[31966]: pgmap v8985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T15:47:43.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:47:43.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:47:43.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:47:43.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:47:43.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:43 smithi134 ceph-mon[31966]: pgmap v8986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:47:43.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:43 smithi134 ceph-mon[31966]: pgmap v8987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T15:47:43.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:47:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:47:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:47:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:43 smithi189 ceph-mon[36790]: pgmap v8985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T15:47:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:47:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:47:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:47:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:47:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:43 smithi189 ceph-mon[36790]: pgmap v8986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T15:47:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:43 smithi189 ceph-mon[36790]: pgmap v8987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T15:47:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:47:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15605.399185 secs 2024-07-26T15:47:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15605.399185 secs 2024-07-26T15:47:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15605.399185 secs 2024-07-26T15:47:45.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:47:45.308+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:47:45.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:47:45.308+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:47:45.311 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:47:45.308+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:45 smithi193 ceph-mon[37473]: pgmap v8988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T15:47:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:45 smithi134 ceph-mon[31966]: pgmap v8988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T15:47:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:45 smithi189 ceph-mon[36790]: pgmap v8988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T15:47:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15624.822581 secs 2024-07-26T15:47:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15624.822581 secs 2024-07-26T15:47:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15624.822581 secs 2024-07-26T15:47:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:47 smithi134 ceph-mon[31966]: pgmap v8989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T15:47:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:47 smithi189 ceph-mon[36790]: pgmap v8989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T15:47:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:47 smithi193 ceph-mon[37473]: pgmap v8989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-07-26T15:47:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:49 smithi134 ceph-mon[31966]: pgmap v8990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:47:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15610.399311 secs 2024-07-26T15:47:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:49 smithi189 ceph-mon[36790]: pgmap v8990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:47:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15610.399311 secs 2024-07-26T15:47:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:49 smithi193 ceph-mon[37473]: pgmap v8990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:47:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15610.399311 secs 2024-07-26T15:47:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:51 smithi134 ceph-mon[31966]: pgmap v8991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T15:47:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15629.822666 secs 2024-07-26T15:47:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:51 smithi189 ceph-mon[36790]: pgmap v8991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T15:47:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15629.822666 secs 2024-07-26T15:47:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:51 smithi193 ceph-mon[37473]: pgmap v8991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-07-26T15:47:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15629.822666 secs 2024-07-26T15:47:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:53 smithi134 ceph-mon[31966]: pgmap v8992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:47:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:53 smithi189 ceph-mon[36790]: pgmap v8992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:47:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:53 smithi193 ceph-mon[37473]: pgmap v8992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T15:47:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15615.399398 secs 2024-07-26T15:47:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15615.399398 secs 2024-07-26T15:47:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15615.399398 secs 2024-07-26T15:47:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:47:55.888+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:47:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:47:55.888+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:47:55.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:47:55.888+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:47:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:55 smithi134 ceph-mon[31966]: pgmap v8993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:47:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:55 smithi189 ceph-mon[36790]: pgmap v8993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:47:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:55 smithi193 ceph-mon[37473]: pgmap v8993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:47:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15634.822733 secs 2024-07-26T15:47:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15634.822733 secs 2024-07-26T15:47:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15634.822733 secs 2024-07-26T15:47:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:57 smithi134 ceph-mon[31966]: pgmap v8994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:47:57.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:57 smithi193 ceph-mon[37473]: pgmap v8994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:47:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:57 smithi189 ceph-mon[36790]: pgmap v8994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:47:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:59 smithi134 ceph-mon[31966]: pgmap v8995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:47:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:47:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15620.399483 secs 2024-07-26T15:47:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:59 smithi189 ceph-mon[36790]: pgmap v8995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:47:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:47:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15620.399483 secs 2024-07-26T15:48:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:59 smithi193 ceph-mon[37473]: pgmap v8995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:48:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:47:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15620.399483 secs 2024-07-26T15:48:00.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:48:00.310+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:48:00.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:48:00.310+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:48:00.312 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:48:00.310+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:01 smithi134 ceph-mon[31966]: pgmap v8996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15639.822874 secs 2024-07-26T15:48:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:01 smithi189 ceph-mon[36790]: pgmap v8996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15639.822874 secs 2024-07-26T15:48:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:01 smithi193 ceph-mon[37473]: pgmap v8996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15639.822874 secs 2024-07-26T15:48:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:03 smithi134 ceph-mon[31966]: pgmap v8997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:03 smithi189 ceph-mon[36790]: pgmap v8997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:03 smithi193 ceph-mon[37473]: pgmap v8997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15625.399582 secs 2024-07-26T15:48:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15625.399582 secs 2024-07-26T15:48:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15625.399582 secs 2024-07-26T15:48:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:05 smithi134 ceph-mon[31966]: pgmap v8998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:05 smithi189 ceph-mon[36790]: pgmap v8998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:05 smithi193 ceph-mon[37473]: pgmap v8998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15644.823012 secs 2024-07-26T15:48:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15644.823012 secs 2024-07-26T15:48:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15644.823012 secs 2024-07-26T15:48:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:07 smithi134 ceph-mon[31966]: pgmap v8999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:07 smithi189 ceph-mon[36790]: pgmap v8999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:07 smithi193 ceph-mon[37473]: pgmap v8999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:09 smithi134 ceph-mon[31966]: pgmap v9000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:48:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15630.399705 secs 2024-07-26T15:48:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:09 smithi189 ceph-mon[36790]: pgmap v9000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:48:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15630.399705 secs 2024-07-26T15:48:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:09 smithi193 ceph-mon[37473]: pgmap v9000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:48:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15630.399705 secs 2024-07-26T15:48:10.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:48:10.903+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:48:10.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:48:10.904+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:48:10.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:48:10.904+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:11 smithi134 ceph-mon[31966]: pgmap v9001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15649.823129 secs 2024-07-26T15:48:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:11 smithi189 ceph-mon[36790]: pgmap v9001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15649.823129 secs 2024-07-26T15:48:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:11 smithi193 ceph-mon[37473]: pgmap v9001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15649.823129 secs 2024-07-26T15:48:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:13 smithi134 ceph-mon[31966]: pgmap v9002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:13 smithi189 ceph-mon[36790]: pgmap v9002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:13 smithi193 ceph-mon[37473]: pgmap v9002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15635.399842 secs 2024-07-26T15:48:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15635.399842 secs 2024-07-26T15:48:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15635.399842 secs 2024-07-26T15:48:15.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:48:15.318+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:48:15.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:48:15.319+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:48:15.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:48:15.319+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:15 smithi134 ceph-mon[31966]: pgmap v9003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:15 smithi189 ceph-mon[36790]: pgmap v9003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:15 smithi193 ceph-mon[37473]: pgmap v9003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15654.823242 secs 2024-07-26T15:48:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15654.823242 secs 2024-07-26T15:48:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15654.823242 secs 2024-07-26T15:48:17.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:17 smithi189 ceph-mon[36790]: pgmap v9004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:17 smithi193 ceph-mon[37473]: pgmap v9004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:17 smithi134 ceph-mon[31966]: pgmap v9004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15640.400001 secs 2024-07-26T15:48:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15640.400001 secs 2024-07-26T15:48:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15640.400001 secs 2024-07-26T15:48:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:19 smithi193 ceph-mon[37473]: pgmap v9005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:48:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:19 smithi134 ceph-mon[31966]: pgmap v9005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:48:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:19 smithi189 ceph-mon[36790]: pgmap v9005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:48:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:21 smithi193 ceph-mon[37473]: pgmap v9006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15659.823316 secs 2024-07-26T15:48:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:21 smithi134 ceph-mon[31966]: pgmap v9006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15659.823316 secs 2024-07-26T15:48:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:21 smithi189 ceph-mon[36790]: pgmap v9006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15659.823316 secs 2024-07-26T15:48:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:23 smithi193 ceph-mon[37473]: pgmap v9007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15645.400145 secs 2024-07-26T15:48:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:23 smithi134 ceph-mon[31966]: pgmap v9007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15645.400145 secs 2024-07-26T15:48:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:23 smithi189 ceph-mon[36790]: pgmap v9007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15645.400145 secs 2024-07-26T15:48:25.048 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:48:25.046+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:48:25.049 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:48:25.047+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 122 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:48:25.050 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:48:25.049+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:48:25.051 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:48:25.049+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:48:25.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:48:25.919+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:48:25.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:48:25.919+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:25.921 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:48:25.919+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:48:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:25 smithi193 ceph-mon[37473]: pgmap v9008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:25 smithi134 ceph-mon[31966]: pgmap v9008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:25 smithi189 ceph-mon[36790]: pgmap v9008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15664.823384 secs 2024-07-26T15:48:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15664.823384 secs 2024-07-26T15:48:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15664.823384 secs 2024-07-26T15:48:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:27 smithi193 ceph-mon[37473]: pgmap v9009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:27 smithi134 ceph-mon[31966]: pgmap v9009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:27 smithi189 ceph-mon[36790]: pgmap v9009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15650.400220 secs 2024-07-26T15:48:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15650.400220 secs 2024-07-26T15:48:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15650.400220 secs 2024-07-26T15:48:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:29 smithi193 ceph-mon[37473]: pgmap v9010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:48:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:29 smithi134 ceph-mon[31966]: pgmap v9010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:48:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:29 smithi189 ceph-mon[36790]: pgmap v9010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:48:30.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:48:30.331+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:48:30.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:48:30.331+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:30.334 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:48:30.331+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:48:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:31 smithi193 ceph-mon[37473]: pgmap v9011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15669.823453 secs 2024-07-26T15:48:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:31 smithi134 ceph-mon[31966]: pgmap v9011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15669.823453 secs 2024-07-26T15:48:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:31 smithi189 ceph-mon[36790]: pgmap v9011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15669.823453 secs 2024-07-26T15:48:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:33 smithi193 ceph-mon[37473]: pgmap v9012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15655.400346 secs 2024-07-26T15:48:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:33 smithi134 ceph-mon[31966]: pgmap v9012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15655.400346 secs 2024-07-26T15:48:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:33 smithi189 ceph-mon[36790]: pgmap v9012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15655.400346 secs 2024-07-26T15:48:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:48:35.593+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:48:35.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:48:35.593+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:48:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:48:35.593+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:48:35.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:48:35.593+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:48:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:35 smithi193 ceph-mon[37473]: pgmap v9013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:35 smithi134 ceph-mon[31966]: pgmap v9013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:35 smithi189 ceph-mon[36790]: pgmap v9013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:48:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15674.823520 secs 2024-07-26T15:48:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15674.823520 secs 2024-07-26T15:48:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15674.823520 secs 2024-07-26T15:48:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:37 smithi193 ceph-mon[37473]: pgmap v9014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:37 smithi134 ceph-mon[31966]: pgmap v9014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:37 smithi189 ceph-mon[36790]: pgmap v9014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:39.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15660.400471 secs 2024-07-26T15:48:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15660.400471 secs 2024-07-26T15:48:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15660.400471 secs 2024-07-26T15:48:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:39 smithi134 ceph-mon[31966]: pgmap v9015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:48:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:39 smithi189 ceph-mon[36790]: pgmap v9015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:48:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:39 smithi193 ceph-mon[37473]: pgmap v9015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:48:40.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:48:40.935+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:48:40.936 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:48:40.935+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:40.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:48:40.935+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:48:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:41 smithi134 ceph-mon[31966]: pgmap v9016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15679.823641 secs 2024-07-26T15:48:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:41 smithi189 ceph-mon[36790]: pgmap v9016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15679.823641 secs 2024-07-26T15:48:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:41 smithi193 ceph-mon[37473]: pgmap v9016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15679.823641 secs 2024-07-26T15:48:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:48:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:48:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:48:44.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: pgmap v9017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:44.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:48:44.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:48:44.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:48:44.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:48:44.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:48:44.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:48:44.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:48:44.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15665.400598 secs 2024-07-26T15:48:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: pgmap v9017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:48:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:48:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:48:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:48:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:48:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:48:44.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:48:44.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15665.400598 secs 2024-07-26T15:48:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: pgmap v9017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:48:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:48:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:48:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:48:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:48:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:48:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:48:44.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:48:44.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:48:44.276 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15665.400598 secs 2024-07-26T15:48:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:44 smithi134 ceph-mon[31966]: pgmap v9018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:48:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:44 smithi134 ceph-mon[31966]: pgmap v9019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:48:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:44 smithi189 ceph-mon[36790]: pgmap v9018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:48:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:44 smithi189 ceph-mon[36790]: pgmap v9019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:48:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:44 smithi193 ceph-mon[37473]: pgmap v9018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:48:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:44 smithi193 ceph-mon[37473]: pgmap v9019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:48:45.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:48:45.333+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:48:45.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:48:45.333+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:48:45.335 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:48:45.333+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:46 smithi134 ceph-mon[31966]: pgmap v9020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:48:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15684.823761 secs 2024-07-26T15:48:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:46 smithi189 ceph-mon[36790]: pgmap v9020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:48:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15684.823761 secs 2024-07-26T15:48:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:46 smithi193 ceph-mon[37473]: pgmap v9020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:48:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15684.823761 secs 2024-07-26T15:48:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:48 smithi134 ceph-mon[31966]: pgmap v9021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15670.400684 secs 2024-07-26T15:48:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:48 smithi189 ceph-mon[36790]: pgmap v9021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15670.400684 secs 2024-07-26T15:48:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:48 smithi193 ceph-mon[37473]: pgmap v9021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15670.400684 secs 2024-07-26T15:48:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:50 smithi134 ceph-mon[31966]: pgmap v9022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:48:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:50 smithi189 ceph-mon[36790]: pgmap v9022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:48:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:50 smithi193 ceph-mon[37473]: pgmap v9022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:48:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15689.823878 secs 2024-07-26T15:48:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15689.823878 secs 2024-07-26T15:48:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15689.823878 secs 2024-07-26T15:48:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:52 smithi134 ceph-mon[31966]: pgmap v9023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:48:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:52 smithi189 ceph-mon[36790]: pgmap v9023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:48:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:52 smithi193 ceph-mon[37473]: pgmap v9023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:48:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15675.400828 secs 2024-07-26T15:48:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15675.400828 secs 2024-07-26T15:48:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15675.400828 secs 2024-07-26T15:48:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:54 smithi134 ceph-mon[31966]: pgmap v9024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:48:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:54 smithi189 ceph-mon[36790]: pgmap v9024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:48:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:54 smithi193 ceph-mon[37473]: pgmap v9024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:48:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:48:55.950+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:48:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:48:55.950+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:48:55.952 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:48:55.950+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:48:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:56 smithi134 ceph-mon[31966]: pgmap v9025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15694.823980 secs 2024-07-26T15:48:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:56 smithi189 ceph-mon[36790]: pgmap v9025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15694.823980 secs 2024-07-26T15:48:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:56 smithi193 ceph-mon[37473]: pgmap v9025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:48:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15694.823980 secs 2024-07-26T15:48:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:58 smithi134 ceph-mon[31966]: pgmap v9026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:48:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:48:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15680.400925 secs 2024-07-26T15:48:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:58 smithi189 ceph-mon[36790]: pgmap v9026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:48:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:48:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15680.400925 secs 2024-07-26T15:48:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:58 smithi193 ceph-mon[37473]: pgmap v9026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:48:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:48:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15680.400925 secs 2024-07-26T15:49:00.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:49:00.343+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:49:00.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:49:00.343+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:49:00.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:49:00.343+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:00 smithi134 ceph-mon[31966]: pgmap v9027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:49:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:00 smithi189 ceph-mon[36790]: pgmap v9027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:49:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:00 smithi193 ceph-mon[37473]: pgmap v9027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T15:49:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15699.824101 secs 2024-07-26T15:49:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15699.824101 secs 2024-07-26T15:49:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15699.824101 secs 2024-07-26T15:49:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:02 smithi134 ceph-mon[31966]: pgmap v9028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:49:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:02 smithi189 ceph-mon[36790]: pgmap v9028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:49:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:02 smithi193 ceph-mon[37473]: pgmap v9028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:49:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15685.401053 secs 2024-07-26T15:49:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15685.401053 secs 2024-07-26T15:49:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15685.401053 secs 2024-07-26T15:49:05.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:04 smithi134 ceph-mon[31966]: pgmap v9029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:49:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:04 smithi189 ceph-mon[36790]: pgmap v9029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:49:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:04 smithi193 ceph-mon[37473]: pgmap v9029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:49:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:06 smithi189 ceph-mon[36790]: pgmap v9030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:49:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15704.824224 secs 2024-07-26T15:49:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:06 smithi193 ceph-mon[37473]: pgmap v9030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:49:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15704.824224 secs 2024-07-26T15:49:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:06 smithi134 ceph-mon[31966]: pgmap v9030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:49:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15704.824224 secs 2024-07-26T15:49:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:08 smithi189 ceph-mon[36790]: pgmap v9031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:49:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15690.401140 secs 2024-07-26T15:49:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:08 smithi193 ceph-mon[37473]: pgmap v9031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:49:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15690.401140 secs 2024-07-26T15:49:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:08 smithi134 ceph-mon[31966]: pgmap v9031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:49:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15690.401140 secs 2024-07-26T15:49:10.964 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:49:10.962+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:49:10.964 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:49:10.962+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:10.965 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:49:10.962+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:49:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:10 smithi134 ceph-mon[31966]: pgmap v9032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:49:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:10 smithi189 ceph-mon[36790]: pgmap v9032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:49:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:10 smithi193 ceph-mon[37473]: pgmap v9032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:49:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15709.824370 secs 2024-07-26T15:49:12.276 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15709.824370 secs 2024-07-26T15:49:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15709.824370 secs 2024-07-26T15:49:13.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:12 smithi189 ceph-mon[36790]: pgmap v9033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:12 smithi193 ceph-mon[37473]: pgmap v9033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:12 smithi134 ceph-mon[31966]: pgmap v9033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15695.401277 secs 2024-07-26T15:49:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15695.401277 secs 2024-07-26T15:49:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15695.401277 secs 2024-07-26T15:49:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:14 smithi193 ceph-mon[37473]: pgmap v9034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:49:15.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:49:15.349+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:49:15.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:49:15.349+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:15.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:49:15.349+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:49:15.352 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:14 smithi134 ceph-mon[31966]: pgmap v9034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:49:15.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:14 smithi189 ceph-mon[36790]: pgmap v9034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:49:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:16 smithi193 ceph-mon[37473]: pgmap v9035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15714.824436 secs 2024-07-26T15:49:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:16 smithi134 ceph-mon[31966]: pgmap v9035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15714.824436 secs 2024-07-26T15:49:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:16 smithi189 ceph-mon[36790]: pgmap v9035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15714.824436 secs 2024-07-26T15:49:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:18 smithi193 ceph-mon[37473]: pgmap v9036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:49:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15700.401401 secs 2024-07-26T15:49:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:18 smithi134 ceph-mon[31966]: pgmap v9036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:49:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15700.401401 secs 2024-07-26T15:49:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:18 smithi189 ceph-mon[36790]: pgmap v9036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:49:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15700.401401 secs 2024-07-26T15:49:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:20 smithi193 ceph-mon[37473]: pgmap v9037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:49:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:20 smithi134 ceph-mon[31966]: pgmap v9037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:49:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:20 smithi189 ceph-mon[36790]: pgmap v9037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:49:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15719.824526 secs 2024-07-26T15:49:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15719.824526 secs 2024-07-26T15:49:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15719.824526 secs 2024-07-26T15:49:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:22 smithi193 ceph-mon[37473]: pgmap v9038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:23.275 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:22 smithi189 ceph-mon[36790]: pgmap v9038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:22 smithi134 ceph-mon[31966]: pgmap v9038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15705.401526 secs 2024-07-26T15:49:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15705.401526 secs 2024-07-26T15:49:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15705.401526 secs 2024-07-26T15:49:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:25 smithi193 ceph-mon[37473]: pgmap v9039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:49:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:25 smithi134 ceph-mon[31966]: pgmap v9039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:49:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:25 smithi189 ceph-mon[36790]: pgmap v9039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:49:25.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:49:25.978+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:49:25.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:49:25.978+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:25.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:49:25.978+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:49:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:27 smithi193 ceph-mon[37473]: pgmap v9040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15724.824652 secs 2024-07-26T15:49:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:27 smithi134 ceph-mon[31966]: pgmap v9040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15724.824652 secs 2024-07-26T15:49:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:27 smithi189 ceph-mon[36790]: pgmap v9040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15724.824652 secs 2024-07-26T15:49:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:29 smithi193 ceph-mon[37473]: pgmap v9041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:49:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15710.401646 secs 2024-07-26T15:49:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:29 smithi134 ceph-mon[31966]: pgmap v9041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:49:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15710.401646 secs 2024-07-26T15:49:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:29 smithi189 ceph-mon[36790]: pgmap v9041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:49:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15710.401646 secs 2024-07-26T15:49:30.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:49:30.359+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:49:30.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:49:30.359+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:49:30.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:49:30.359+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:31 smithi134 ceph-mon[31966]: pgmap v9042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:49:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:31 smithi189 ceph-mon[36790]: pgmap v9042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:49:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:31 smithi193 ceph-mon[37473]: pgmap v9042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:49:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15729.824778 secs 2024-07-26T15:49:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15729.824778 secs 2024-07-26T15:49:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15729.824778 secs 2024-07-26T15:49:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:33 smithi134 ceph-mon[31966]: pgmap v9043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:33 smithi189 ceph-mon[36790]: pgmap v9043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:33 smithi193 ceph-mon[37473]: pgmap v9043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15715.401766 secs 2024-07-26T15:49:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15715.401766 secs 2024-07-26T15:49:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15715.401766 secs 2024-07-26T15:49:35.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:35 smithi134 ceph-mon[31966]: pgmap v9044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:49:35.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:35 smithi189 ceph-mon[36790]: pgmap v9044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:49:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:35 smithi193 ceph-mon[37473]: pgmap v9044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:49:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:37 smithi134 ceph-mon[31966]: pgmap v9045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:37.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15734.824918 secs 2024-07-26T15:49:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:37 smithi189 ceph-mon[36790]: pgmap v9045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15734.824918 secs 2024-07-26T15:49:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:37 smithi193 ceph-mon[37473]: pgmap v9045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15734.824918 secs 2024-07-26T15:49:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:39 smithi134 ceph-mon[31966]: pgmap v9046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:49:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15720.401898 secs 2024-07-26T15:49:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:39 smithi189 ceph-mon[36790]: pgmap v9046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:49:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15720.401898 secs 2024-07-26T15:49:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:39 smithi193 ceph-mon[37473]: pgmap v9046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:49:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15720.401898 secs 2024-07-26T15:49:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:49:40.983+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:49:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:49:40.983+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:40.985 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:49:40.983+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:49:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:41 smithi134 ceph-mon[31966]: pgmap v9047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:49:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:41 smithi189 ceph-mon[36790]: pgmap v9047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:49:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:41 smithi193 ceph-mon[37473]: pgmap v9047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:49:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15739.825016 secs 2024-07-26T15:49:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15739.825016 secs 2024-07-26T15:49:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15739.825016 secs 2024-07-26T15:49:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:43 smithi134 ceph-mon[31966]: pgmap v9048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:43.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:43 smithi189 ceph-mon[36790]: pgmap v9048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:43 smithi193 ceph-mon[37473]: pgmap v9048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15725.402029 secs 2024-07-26T15:49:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:49:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15725.402029 secs 2024-07-26T15:49:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:49:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15725.402029 secs 2024-07-26T15:49:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:49:45.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:49:45.374+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:49:45.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:49:45.375+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:45.377 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:49:45.375+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:49:45.377 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:45 smithi134 ceph-mon[31966]: pgmap v9049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:49:45.377 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:49:45.377 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:49:45.377 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:49:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:45 smithi189 ceph-mon[36790]: pgmap v9049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:49:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:49:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:49:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:49:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:45 smithi193 ceph-mon[37473]: pgmap v9049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:49:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:49:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:49:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:49:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:46 smithi134 ceph-mon[31966]: pgmap v9050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:49:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:46 smithi134 ceph-mon[31966]: pgmap v9051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:49:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:46 smithi189 ceph-mon[36790]: pgmap v9050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:49:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:46 smithi189 ceph-mon[36790]: pgmap v9051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:49:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:46 smithi193 ceph-mon[37473]: pgmap v9050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:49:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:46 smithi193 ceph-mon[37473]: pgmap v9051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T15:49:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15744.825129 secs 2024-07-26T15:49:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15744.825129 secs 2024-07-26T15:49:47.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15744.825129 secs 2024-07-26T15:49:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:48 smithi134 ceph-mon[31966]: pgmap v9052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:49:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:48 smithi189 ceph-mon[36790]: pgmap v9052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:49:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:48 smithi193 ceph-mon[37473]: pgmap v9052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:49:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15730.402174 secs 2024-07-26T15:49:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15730.402174 secs 2024-07-26T15:49:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15730.402174 secs 2024-07-26T15:49:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:50 smithi134 ceph-mon[31966]: pgmap v9053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:49:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:50 smithi189 ceph-mon[36790]: pgmap v9053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:49:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:50 smithi193 ceph-mon[37473]: pgmap v9053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:49:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:52 smithi134 ceph-mon[31966]: pgmap v9054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:49:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15749.825252 secs 2024-07-26T15:49:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:52 smithi189 ceph-mon[36790]: pgmap v9054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:49:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15749.825252 secs 2024-07-26T15:49:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:52 smithi193 ceph-mon[37473]: pgmap v9054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:49:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15749.825252 secs 2024-07-26T15:49:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:54 smithi134 ceph-mon[31966]: pgmap v9055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:49:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15735.402301 secs 2024-07-26T15:49:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:54 smithi189 ceph-mon[36790]: pgmap v9055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:49:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15735.402301 secs 2024-07-26T15:49:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:54 smithi193 ceph-mon[37473]: pgmap v9055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:49:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15735.402301 secs 2024-07-26T15:49:55.999 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:49:55.998+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:49:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:49:55.998+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:49:56.000 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:49:55.998+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:49:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:56 smithi134 ceph-mon[31966]: pgmap v9056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:49:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:56 smithi189 ceph-mon[36790]: pgmap v9056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:49:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:56 smithi193 ceph-mon[37473]: pgmap v9056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:49:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15754.825360 secs 2024-07-26T15:49:57.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15754.825360 secs 2024-07-26T15:49:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15754.825360 secs 2024-07-26T15:49:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:58 smithi134 ceph-mon[31966]: pgmap v9057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:58 smithi189 ceph-mon[36790]: pgmap v9057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:58 smithi193 ceph-mon[37473]: pgmap v9057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:49:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:49:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15740.402449 secs 2024-07-26T15:49:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:49:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15740.402449 secs 2024-07-26T15:49:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:49:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15740.402449 secs 2024-07-26T15:50:00.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:50:00.390+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:50:00.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:50:00.390+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:50:00.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:50:00.390+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:00 smithi189 ceph-mon[36790]: pgmap v9058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:50:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:00 smithi193 ceph-mon[37473]: pgmap v9058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:50:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:00 smithi134 ceph-mon[31966]: pgmap v9058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T15:50:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:01 smithi193 ceph-mon[37473]: pgmap v9059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:01 smithi134 ceph-mon[31966]: pgmap v9059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:01 smithi189 ceph-mon[36790]: pgmap v9059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15759.825487 secs 2024-07-26T15:50:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15759.825487 secs 2024-07-26T15:50:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15759.825487 secs 2024-07-26T15:50:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:03 smithi193 ceph-mon[37473]: pgmap v9060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:03 smithi134 ceph-mon[31966]: pgmap v9060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:03 smithi189 ceph-mon[36790]: pgmap v9060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15745.402582 secs 2024-07-26T15:50:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15745.402582 secs 2024-07-26T15:50:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15745.402582 secs 2024-07-26T15:50:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:05 smithi193 ceph-mon[37473]: pgmap v9061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:50:05.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:05 smithi134 ceph-mon[31966]: pgmap v9061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:50:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:05 smithi189 ceph-mon[36790]: pgmap v9061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:50:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15764.825606 secs 2024-07-26T15:50:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15764.825606 secs 2024-07-26T15:50:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15764.825606 secs 2024-07-26T15:50:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:07 smithi193 ceph-mon[37473]: pgmap v9062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:07 smithi134 ceph-mon[31966]: pgmap v9062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:07 smithi189 ceph-mon[36790]: pgmap v9062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15750.402706 secs 2024-07-26T15:50:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15750.402706 secs 2024-07-26T15:50:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15750.402706 secs 2024-07-26T15:50:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:09 smithi193 ceph-mon[37473]: pgmap v9063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:09 smithi134 ceph-mon[31966]: pgmap v9063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:09 smithi189 ceph-mon[36790]: pgmap v9063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:50:10.999+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:50:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:50:10.999+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:11.001 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:50:10.999+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:50:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:11 smithi134 ceph-mon[31966]: pgmap v9064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15769.825746 secs 2024-07-26T15:50:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:11 smithi189 ceph-mon[36790]: pgmap v9064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15769.825746 secs 2024-07-26T15:50:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:11 smithi193 ceph-mon[37473]: pgmap v9064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15769.825746 secs 2024-07-26T15:50:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:13 smithi134 ceph-mon[31966]: pgmap v9065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15755.402830 secs 2024-07-26T15:50:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:13 smithi189 ceph-mon[36790]: pgmap v9065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15755.402830 secs 2024-07-26T15:50:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:13 smithi193 ceph-mon[37473]: pgmap v9065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15755.402830 secs 2024-07-26T15:50:15.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:50:15.406+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:50:15.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:50:15.406+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:50:15.409 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:50:15.406+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:15 smithi134 ceph-mon[31966]: pgmap v9066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:50:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:15 smithi189 ceph-mon[36790]: pgmap v9066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:50:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:15 smithi193 ceph-mon[37473]: pgmap v9066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:50:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15774.825865 secs 2024-07-26T15:50:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15774.825865 secs 2024-07-26T15:50:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15774.825865 secs 2024-07-26T15:50:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:17 smithi134 ceph-mon[31966]: pgmap v9067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:17 smithi189 ceph-mon[36790]: pgmap v9067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:17 smithi193 ceph-mon[37473]: pgmap v9067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15760.402988 secs 2024-07-26T15:50:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15760.402988 secs 2024-07-26T15:50:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15760.402988 secs 2024-07-26T15:50:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:19 smithi134 ceph-mon[31966]: pgmap v9068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:19 smithi189 ceph-mon[36790]: pgmap v9068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:19 smithi193 ceph-mon[37473]: pgmap v9068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:21 smithi134 ceph-mon[31966]: pgmap v9069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15779.825996 secs 2024-07-26T15:50:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:21 smithi189 ceph-mon[36790]: pgmap v9069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15779.825996 secs 2024-07-26T15:50:22.273 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:21 smithi193 ceph-mon[37473]: pgmap v9069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15779.825996 secs 2024-07-26T15:50:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:23 smithi134 ceph-mon[31966]: pgmap v9070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15765.403073 secs 2024-07-26T15:50:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:23 smithi189 ceph-mon[36790]: pgmap v9070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:24.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15765.403073 secs 2024-07-26T15:50:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:23 smithi193 ceph-mon[37473]: pgmap v9070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15765.403073 secs 2024-07-26T15:50:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:50:26.014+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:50:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:50:26.015+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:50:26.016 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:50:26.015+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:25 smithi134 ceph-mon[31966]: pgmap v9071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:50:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:25 smithi189 ceph-mon[36790]: pgmap v9071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:50:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:25 smithi193 ceph-mon[37473]: pgmap v9071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:50:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15784.826124 secs 2024-07-26T15:50:27.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15784.826124 secs 2024-07-26T15:50:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15784.826124 secs 2024-07-26T15:50:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:27 smithi134 ceph-mon[31966]: pgmap v9072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:27 smithi189 ceph-mon[36790]: pgmap v9072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:27 smithi193 ceph-mon[37473]: pgmap v9072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15770.403201 secs 2024-07-26T15:50:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15770.403201 secs 2024-07-26T15:50:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15770.403201 secs 2024-07-26T15:50:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:29 smithi134 ceph-mon[31966]: pgmap v9073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:29 smithi189 ceph-mon[36790]: pgmap v9073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:29 smithi193 ceph-mon[37473]: pgmap v9073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:30.422 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:50:30.421+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:50:30.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:50:30.421+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:50:30.423 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:50:30.421+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:31 smithi134 ceph-mon[31966]: pgmap v9074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15789.826190 secs 2024-07-26T15:50:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:31 smithi189 ceph-mon[36790]: pgmap v9074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15789.826190 secs 2024-07-26T15:50:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:31 smithi193 ceph-mon[37473]: pgmap v9074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15789.826190 secs 2024-07-26T15:50:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:33 smithi134 ceph-mon[31966]: pgmap v9075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15775.403340 secs 2024-07-26T15:50:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:33 smithi189 ceph-mon[36790]: pgmap v9075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15775.403340 secs 2024-07-26T15:50:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:33 smithi193 ceph-mon[37473]: pgmap v9075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15775.403340 secs 2024-07-26T15:50:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:35 smithi189 ceph-mon[36790]: pgmap v9076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:50:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:35 smithi193 ceph-mon[37473]: pgmap v9076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:50:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:35 smithi134 ceph-mon[31966]: pgmap v9076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:50:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15794.826348 secs 2024-07-26T15:50:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15794.826348 secs 2024-07-26T15:50:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15794.826348 secs 2024-07-26T15:50:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:37 smithi189 ceph-mon[36790]: pgmap v9077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:37 smithi193 ceph-mon[37473]: pgmap v9077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:37 smithi134 ceph-mon[31966]: pgmap v9077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15780.403414 secs 2024-07-26T15:50:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15780.403414 secs 2024-07-26T15:50:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15780.403414 secs 2024-07-26T15:50:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:39 smithi193 ceph-mon[37473]: pgmap v9078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:39 smithi134 ceph-mon[31966]: pgmap v9078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:39 smithi189 ceph-mon[36790]: pgmap v9078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:50:41.028+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:50:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:50:41.029+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:50:41.030 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:50:41.029+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:42.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:41 smithi189 ceph-mon[36790]: pgmap v9079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:42.246 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15799.826487 secs 2024-07-26T15:50:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:41 smithi193 ceph-mon[37473]: pgmap v9079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15799.826487 secs 2024-07-26T15:50:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:41 smithi134 ceph-mon[31966]: pgmap v9079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:50:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15799.826487 secs 2024-07-26T15:50:44.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:43 smithi134 ceph-mon[31966]: pgmap v9080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:44.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15785.403560 secs 2024-07-26T15:50:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:43 smithi193 ceph-mon[37473]: pgmap v9080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15785.403560 secs 2024-07-26T15:50:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:43 smithi189 ceph-mon[36790]: pgmap v9080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:50:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15785.403560 secs 2024-07-26T15:50:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:50:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:50:45.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:50:45.436+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:50:45.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:50:45.436+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:50:45.438 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:50:45.436+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:50:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:45 smithi193 ceph-mon[37473]: pgmap v9081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:50:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:50:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:50:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:45 smithi134 ceph-mon[31966]: pgmap v9081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:50:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:50:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:50:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:45 smithi189 ceph-mon[36790]: pgmap v9081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:50:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:50:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:50:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15804.826632 secs 2024-07-26T15:50:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:50:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:50:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:50:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15804.826632 secs 2024-07-26T15:50:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:50:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:50:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:50:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15804.826632 secs 2024-07-26T15:50:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:50:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:50:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:50:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:48 smithi193 ceph-mon[37473]: pgmap v9082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T15:50:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:48 smithi193 ceph-mon[37473]: pgmap v9083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:50:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:48 smithi134 ceph-mon[31966]: pgmap v9082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T15:50:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:48 smithi134 ceph-mon[31966]: pgmap v9083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:50:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:48 smithi189 ceph-mon[36790]: pgmap v9082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T15:50:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:48 smithi189 ceph-mon[36790]: pgmap v9083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:50:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15790.403694 secs 2024-07-26T15:50:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15790.403694 secs 2024-07-26T15:50:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15790.403694 secs 2024-07-26T15:50:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:50 smithi134 ceph-mon[31966]: pgmap v9084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:50:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:50 smithi189 ceph-mon[36790]: pgmap v9084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:50:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:50 smithi193 ceph-mon[37473]: pgmap v9084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:50:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:52 smithi134 ceph-mon[31966]: pgmap v9085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:50:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15809.826766 secs 2024-07-26T15:50:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:52 smithi189 ceph-mon[36790]: pgmap v9085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:50:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15809.826766 secs 2024-07-26T15:50:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:52 smithi193 ceph-mon[37473]: pgmap v9085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:50:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15809.826766 secs 2024-07-26T15:50:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:54 smithi134 ceph-mon[31966]: pgmap v9086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T15:50:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15795.403830 secs 2024-07-26T15:50:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:54 smithi189 ceph-mon[36790]: pgmap v9086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T15:50:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15795.403830 secs 2024-07-26T15:50:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:54 smithi193 ceph-mon[37473]: pgmap v9086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T15:50:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15795.403830 secs 2024-07-26T15:50:56.038 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:50:56.037+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:50:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:50:56.037+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:50:56.039 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:50:56.037+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:50:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:56 smithi134 ceph-mon[31966]: pgmap v9087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:50:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:56 smithi189 ceph-mon[36790]: pgmap v9087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:50:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:56 smithi193 ceph-mon[37473]: pgmap v9087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T15:50:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15814.826913 secs 2024-07-26T15:50:57.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15814.826913 secs 2024-07-26T15:50:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15814.826913 secs 2024-07-26T15:50:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:58 smithi134 ceph-mon[31966]: pgmap v9088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:50:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:58 smithi189 ceph-mon[36790]: pgmap v9088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:50:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:58 smithi193 ceph-mon[37473]: pgmap v9088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:50:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:50:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15800.403981 secs 2024-07-26T15:50:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:50:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15800.403981 secs 2024-07-26T15:50:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:50:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15800.403981 secs 2024-07-26T15:51:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:00 smithi134 ceph-mon[31966]: pgmap v9089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:00 smithi189 ceph-mon[36790]: pgmap v9089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:00.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:00.452+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:51:00.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:00.452+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:51:00.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:00.452+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:00 smithi193 ceph-mon[37473]: pgmap v9089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:02 smithi134 ceph-mon[31966]: pgmap v9090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15819.826989 secs 2024-07-26T15:51:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:02 smithi189 ceph-mon[36790]: pgmap v9090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15819.826989 secs 2024-07-26T15:51:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:02 smithi193 ceph-mon[37473]: pgmap v9090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15819.826989 secs 2024-07-26T15:51:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:04 smithi134 ceph-mon[31966]: pgmap v9091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:51:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15805.404104 secs 2024-07-26T15:51:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:04 smithi189 ceph-mon[36790]: pgmap v9091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:51:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15805.404104 secs 2024-07-26T15:51:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:04 smithi193 ceph-mon[37473]: pgmap v9091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:51:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15805.404104 secs 2024-07-26T15:51:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:06 smithi134 ceph-mon[31966]: pgmap v9092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:51:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:06 smithi189 ceph-mon[36790]: pgmap v9092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:51:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:06 smithi193 ceph-mon[37473]: pgmap v9092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:51:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15824.827109 secs 2024-07-26T15:51:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15824.827109 secs 2024-07-26T15:51:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15824.827109 secs 2024-07-26T15:51:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:08 smithi134 ceph-mon[31966]: pgmap v9093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:08 smithi189 ceph-mon[36790]: pgmap v9093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:08 smithi193 ceph-mon[37473]: pgmap v9093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15810.404190 secs 2024-07-26T15:51:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15810.404190 secs 2024-07-26T15:51:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15810.404190 secs 2024-07-26T15:51:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:10 smithi134 ceph-mon[31966]: pgmap v9094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:10 smithi189 ceph-mon[36790]: pgmap v9094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:10 smithi193 ceph-mon[37473]: pgmap v9094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:11.054 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:11.052+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:51:11.054 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:11.052+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:51:11.054 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:11.052+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:12 smithi134 ceph-mon[31966]: pgmap v9095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15829.827197 secs 2024-07-26T15:51:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:12 smithi189 ceph-mon[36790]: pgmap v9095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15829.827197 secs 2024-07-26T15:51:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:12 smithi193 ceph-mon[37473]: pgmap v9095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15829.827197 secs 2024-07-26T15:51:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:14 smithi134 ceph-mon[31966]: pgmap v9096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:51:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15815.404320 secs 2024-07-26T15:51:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:14 smithi189 ceph-mon[36790]: pgmap v9096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:51:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15815.404320 secs 2024-07-26T15:51:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:14 smithi193 ceph-mon[37473]: pgmap v9096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:51:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15815.404320 secs 2024-07-26T15:51:15.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:15.459+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:51:15.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:15.459+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:51:15.461 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:15.459+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:16 smithi134 ceph-mon[31966]: pgmap v9097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:51:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:16 smithi189 ceph-mon[36790]: pgmap v9097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:51:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:16 smithi193 ceph-mon[37473]: pgmap v9097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:51:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15834.827363 secs 2024-07-26T15:51:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15834.827363 secs 2024-07-26T15:51:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15834.827363 secs 2024-07-26T15:51:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:18 smithi134 ceph-mon[31966]: pgmap v9098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:18 smithi189 ceph-mon[36790]: pgmap v9098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:18 smithi193 ceph-mon[37473]: pgmap v9098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15820.404455 secs 2024-07-26T15:51:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15820.404455 secs 2024-07-26T15:51:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15820.404455 secs 2024-07-26T15:51:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:20 smithi189 ceph-mon[36790]: pgmap v9099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:20 smithi193 ceph-mon[37473]: pgmap v9099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:20 smithi134 ceph-mon[31966]: pgmap v9099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:22 smithi189 ceph-mon[36790]: pgmap v9100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15839.827461 secs 2024-07-26T15:51:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:22 smithi193 ceph-mon[37473]: pgmap v9100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15839.827461 secs 2024-07-26T15:51:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:22 smithi134 ceph-mon[31966]: pgmap v9100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15839.827461 secs 2024-07-26T15:51:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:24 smithi189 ceph-mon[36790]: pgmap v9101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:51:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15825.404588 secs 2024-07-26T15:51:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:24 smithi193 ceph-mon[37473]: pgmap v9101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:51:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15825.404588 secs 2024-07-26T15:51:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:24 smithi134 ceph-mon[31966]: pgmap v9101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:51:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15825.404588 secs 2024-07-26T15:51:25.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:25.077+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T15:51:25.079 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:25.077+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T15:51:25.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:25.078+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T15:51:25.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:25.078+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 123 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T15:51:25.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:25.078+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T15:51:25.080 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:25.078+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T15:51:25.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:25.079+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:51:25.081 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:25.079+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:51:26.069 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:26.068+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:51:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:26.068+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:26.070 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:26.068+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:51:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:26 smithi189 ceph-mon[36790]: pgmap v9102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:51:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:26 smithi193 ceph-mon[37473]: pgmap v9102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:51:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:26 smithi134 ceph-mon[31966]: pgmap v9102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:51:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15844.827533 secs 2024-07-26T15:51:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15844.827533 secs 2024-07-26T15:51:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15844.827533 secs 2024-07-26T15:51:28.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:28 smithi193 ceph-mon[37473]: pgmap v9103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:28 smithi134 ceph-mon[31966]: pgmap v9103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:28 smithi189 ceph-mon[36790]: pgmap v9103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15830.404717 secs 2024-07-26T15:51:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15830.404717 secs 2024-07-26T15:51:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15830.404717 secs 2024-07-26T15:51:30.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:30.468+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:51:30.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:30.468+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:51:30.470 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:30.468+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:30.470 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:30 smithi134 ceph-mon[31966]: pgmap v9104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:30 smithi193 ceph-mon[37473]: pgmap v9104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:30 smithi189 ceph-mon[36790]: pgmap v9104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:31 smithi193 ceph-mon[37473]: pgmap v9105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:31 smithi134 ceph-mon[31966]: pgmap v9105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:31 smithi189 ceph-mon[36790]: pgmap v9105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15849.827616 secs 2024-07-26T15:51:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15849.827616 secs 2024-07-26T15:51:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15849.827616 secs 2024-07-26T15:51:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:33 smithi193 ceph-mon[37473]: pgmap v9106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:51:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:33 smithi134 ceph-mon[31966]: pgmap v9106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:51:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:33 smithi189 ceph-mon[36790]: pgmap v9106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:51:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15835.404839 secs 2024-07-26T15:51:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15835.404839 secs 2024-07-26T15:51:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15835.404839 secs 2024-07-26T15:51:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:35 smithi193 ceph-mon[37473]: pgmap v9107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:51:35.626 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:35.625+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:51:35.626 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:35.625+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:51:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:35.625+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T15:51:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:35.625+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 124 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T15:51:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:35.626+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:51:35.627 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:35.626+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:51:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:35.626+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:51:35.628 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:35.626+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:51:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:35 smithi134 ceph-mon[31966]: pgmap v9107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:51:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:35 smithi189 ceph-mon[36790]: pgmap v9107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:51:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15854.827692 secs 2024-07-26T15:51:36.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15854.827692 secs 2024-07-26T15:51:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15854.827692 secs 2024-07-26T15:51:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:37 smithi134 ceph-mon[31966]: pgmap v9108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:37 smithi189 ceph-mon[36790]: pgmap v9108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:37 smithi193 ceph-mon[37473]: pgmap v9108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:39 smithi134 ceph-mon[31966]: pgmap v9109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15840.404987 secs 2024-07-26T15:51:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:39 smithi189 ceph-mon[36790]: pgmap v9109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15840.404987 secs 2024-07-26T15:51:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:39 smithi193 ceph-mon[37473]: pgmap v9109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15840.404987 secs 2024-07-26T15:51:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:41.081+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:51:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:41.081+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:51:41.082 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:41.081+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:41 smithi134 ceph-mon[31966]: pgmap v9110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15859.827806 secs 2024-07-26T15:51:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:41 smithi189 ceph-mon[36790]: pgmap v9110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15859.827806 secs 2024-07-26T15:51:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:41 smithi193 ceph-mon[37473]: pgmap v9110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15859.827806 secs 2024-07-26T15:51:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:43 smithi134 ceph-mon[31966]: pgmap v9111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:51:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:43 smithi189 ceph-mon[36790]: pgmap v9111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:51:44.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:43 smithi193 ceph-mon[37473]: pgmap v9111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:51:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15845.405039 secs 2024-07-26T15:51:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15845.405039 secs 2024-07-26T15:51:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15845.405039 secs 2024-07-26T15:51:45.485 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:45.483+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:51:45.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:45.483+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:45.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:51:45.483+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:51:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:45 smithi134 ceph-mon[31966]: pgmap v9112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:51:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:45 smithi189 ceph-mon[36790]: pgmap v9112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:51:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:45 smithi193 ceph-mon[37473]: pgmap v9112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:51:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15864.827945 secs 2024-07-26T15:51:47.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15864.827945 secs 2024-07-26T15:51:47.083 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15864.827945 secs 2024-07-26T15:51:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:47 smithi189 ceph-mon[36790]: pgmap v9113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:51:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:47 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:47 smithi193 ceph-mon[37473]: pgmap v9113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:51:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:48.023 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:47 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:48.053 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:47 smithi134 ceph-mon[31966]: pgmap v9113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:48.053 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:51:48.053 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:48.053 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:48.053 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:48.053 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:47 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:49 smithi134 ceph-mon[31966]: pgmap v9114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15850.405110 secs 2024-07-26T15:51:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:49 smithi189 ceph-mon[36790]: pgmap v9114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15850.405110 secs 2024-07-26T15:51:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:49 smithi193 ceph-mon[37473]: pgmap v9114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:51:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15850.405110 secs 2024-07-26T15:51:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:50.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:50.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:51 smithi134 ceph-mon[31966]: pgmap v9115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:51.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:51.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15869.828080 secs 2024-07-26T15:51:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:51 smithi189 ceph-mon[36790]: pgmap v9115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15869.828080 secs 2024-07-26T15:51:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:51 smithi193 ceph-mon[37473]: pgmap v9115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:51:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15869.828080 secs 2024-07-26T15:51:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:51:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:51:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:52 smithi189 ceph-mon[36790]: pgmap v9116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:51:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:52 smithi189 ceph-mon[36790]: pgmap v9117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:51:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:51:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:51:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:52 smithi193 ceph-mon[37473]: pgmap v9116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:51:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:52 smithi193 ceph-mon[37473]: pgmap v9117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:51:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:51:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:51:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:52 smithi134 ceph-mon[31966]: pgmap v9116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:51:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:52 smithi134 ceph-mon[31966]: pgmap v9117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:51:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:51:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15855.405203 secs 2024-07-26T15:51:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15855.405203 secs 2024-07-26T15:51:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15855.405203 secs 2024-07-26T15:51:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:54 smithi193 ceph-mon[37473]: pgmap v9118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:51:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:54 smithi134 ceph-mon[31966]: pgmap v9118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:51:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:54 smithi189 ceph-mon[36790]: pgmap v9118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:51:56.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:56.096+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:51:56.097 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:56.096+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:51:56.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:51:56.096+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:51:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:56 smithi193 ceph-mon[37473]: pgmap v9119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:51:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15874.828197 secs 2024-07-26T15:51:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:56 smithi134 ceph-mon[31966]: pgmap v9119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:51:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15874.828197 secs 2024-07-26T15:51:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:56 smithi189 ceph-mon[36790]: pgmap v9119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:51:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15874.828197 secs 2024-07-26T15:51:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:58 smithi193 ceph-mon[37473]: pgmap v9120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:51:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:51:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15860.405322 secs 2024-07-26T15:51:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:58 smithi134 ceph-mon[31966]: pgmap v9120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:51:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:51:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15860.405322 secs 2024-07-26T15:51:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:58 smithi189 ceph-mon[36790]: pgmap v9120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:51:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:51:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15860.405322 secs 2024-07-26T15:52:00.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:52:00.499+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:52:00.501 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:52:00.499+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:00.502 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:52:00.499+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:52:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:00 smithi193 ceph-mon[37473]: pgmap v9121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:52:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:00 smithi134 ceph-mon[31966]: pgmap v9121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:52:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:00 smithi189 ceph-mon[36790]: pgmap v9121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:52:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15879.828289 secs 2024-07-26T15:52:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15879.828289 secs 2024-07-26T15:52:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15879.828289 secs 2024-07-26T15:52:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:02 smithi134 ceph-mon[31966]: pgmap v9122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:52:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:02 smithi189 ceph-mon[36790]: pgmap v9122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:52:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:02 smithi193 ceph-mon[37473]: pgmap v9122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:52:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15865.405444 secs 2024-07-26T15:52:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15865.405444 secs 2024-07-26T15:52:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15865.405444 secs 2024-07-26T15:52:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:04 smithi134 ceph-mon[31966]: pgmap v9123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:04 smithi189 ceph-mon[36790]: pgmap v9123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:04 smithi193 ceph-mon[37473]: pgmap v9123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:06 smithi134 ceph-mon[31966]: pgmap v9124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:07.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15884.828409 secs 2024-07-26T15:52:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:06 smithi189 ceph-mon[36790]: pgmap v9124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15884.828409 secs 2024-07-26T15:52:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:06 smithi193 ceph-mon[37473]: pgmap v9124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15884.828409 secs 2024-07-26T15:52:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:08 smithi134 ceph-mon[31966]: pgmap v9125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15870.405587 secs 2024-07-26T15:52:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:08 smithi189 ceph-mon[36790]: pgmap v9125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15870.405587 secs 2024-07-26T15:52:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:08 smithi193 ceph-mon[37473]: pgmap v9125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15870.405587 secs 2024-07-26T15:52:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:52:11.104+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:52:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:52:11.104+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:11.106 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:52:11.104+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:52:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:10 smithi134 ceph-mon[31966]: pgmap v9126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:52:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:10 smithi189 ceph-mon[36790]: pgmap v9126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:52:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:10 smithi193 ceph-mon[37473]: pgmap v9126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:52:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15889.828490 secs 2024-07-26T15:52:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15889.828490 secs 2024-07-26T15:52:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15889.828490 secs 2024-07-26T15:52:13.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:12 smithi189 ceph-mon[36790]: pgmap v9127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:12 smithi134 ceph-mon[31966]: pgmap v9127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:12 smithi193 ceph-mon[37473]: pgmap v9127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15875.405715 secs 2024-07-26T15:52:14.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15875.405715 secs 2024-07-26T15:52:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15875.405715 secs 2024-07-26T15:52:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:14 smithi134 ceph-mon[31966]: pgmap v9128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:14 smithi189 ceph-mon[36790]: pgmap v9128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:14 smithi193 ceph-mon[37473]: pgmap v9128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:15.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:52:15.515+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:52:15.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:52:15.515+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:52:15.517 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:52:15.515+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:16 smithi134 ceph-mon[31966]: pgmap v9129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15894.828573 secs 2024-07-26T15:52:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:16 smithi189 ceph-mon[36790]: pgmap v9129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15894.828573 secs 2024-07-26T15:52:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:16 smithi193 ceph-mon[37473]: pgmap v9129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15894.828573 secs 2024-07-26T15:52:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:18 smithi134 ceph-mon[31966]: pgmap v9130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15880.405828 secs 2024-07-26T15:52:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:18 smithi189 ceph-mon[36790]: pgmap v9130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15880.405828 secs 2024-07-26T15:52:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:18 smithi193 ceph-mon[37473]: pgmap v9130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15880.405828 secs 2024-07-26T15:52:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:20 smithi134 ceph-mon[31966]: pgmap v9131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:52:21.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:20 smithi189 ceph-mon[36790]: pgmap v9131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:52:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:20 smithi193 ceph-mon[37473]: pgmap v9131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:52:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15899.828693 secs 2024-07-26T15:52:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15899.828693 secs 2024-07-26T15:52:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15899.828693 secs 2024-07-26T15:52:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:22 smithi134 ceph-mon[31966]: pgmap v9132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:22 smithi189 ceph-mon[36790]: pgmap v9132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:22 smithi193 ceph-mon[37473]: pgmap v9132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15885.405935 secs 2024-07-26T15:52:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15885.405935 secs 2024-07-26T15:52:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15885.405935 secs 2024-07-26T15:52:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:24 smithi134 ceph-mon[31966]: pgmap v9133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:24 smithi189 ceph-mon[36790]: pgmap v9133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:24 smithi193 ceph-mon[37473]: pgmap v9133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:52:26.119+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:52:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:52:26.120+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:26.121 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:52:26.120+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:52:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:26 smithi134 ceph-mon[31966]: pgmap v9134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15904.828815 secs 2024-07-26T15:52:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:26 smithi189 ceph-mon[36790]: pgmap v9134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15904.828815 secs 2024-07-26T15:52:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:26 smithi193 ceph-mon[37473]: pgmap v9134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15904.828815 secs 2024-07-26T15:52:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:28 smithi134 ceph-mon[31966]: pgmap v9135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15890.406038 secs 2024-07-26T15:52:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:28 smithi189 ceph-mon[36790]: pgmap v9135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15890.406038 secs 2024-07-26T15:52:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:28 smithi193 ceph-mon[37473]: pgmap v9135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15890.406038 secs 2024-07-26T15:52:30.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:52:30.530+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:52:30.532 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:52:30.530+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:30.533 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:52:30.530+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:52:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:30 smithi134 ceph-mon[31966]: pgmap v9136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:52:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:30 smithi189 ceph-mon[36790]: pgmap v9136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:52:31.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:30 smithi193 ceph-mon[37473]: pgmap v9136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:52:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15909.828934 secs 2024-07-26T15:52:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15909.828934 secs 2024-07-26T15:52:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15909.828934 secs 2024-07-26T15:52:33.169 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:32 smithi134 ceph-mon[31966]: pgmap v9137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:32 smithi189 ceph-mon[36790]: pgmap v9137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:32 smithi193 ceph-mon[37473]: pgmap v9137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15895.406123 secs 2024-07-26T15:52:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15895.406123 secs 2024-07-26T15:52:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15895.406123 secs 2024-07-26T15:52:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:34 smithi189 ceph-mon[36790]: pgmap v9138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:34 smithi193 ceph-mon[37473]: pgmap v9138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:34 smithi134 ceph-mon[31966]: pgmap v9138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:36 smithi189 ceph-mon[36790]: pgmap v9139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15914.829033 secs 2024-07-26T15:52:37.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:36 smithi134 ceph-mon[31966]: pgmap v9139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:37.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15914.829033 secs 2024-07-26T15:52:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:36 smithi193 ceph-mon[37473]: pgmap v9139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15914.829033 secs 2024-07-26T15:52:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:38 smithi189 ceph-mon[36790]: pgmap v9140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15900.406240 secs 2024-07-26T15:52:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:38 smithi193 ceph-mon[37473]: pgmap v9140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15900.406240 secs 2024-07-26T15:52:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:38 smithi134 ceph-mon[31966]: pgmap v9140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15900.406240 secs 2024-07-26T15:52:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:52:41.135+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:52:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:52:41.135+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:41.137 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:52:41.135+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:52:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:40 smithi193 ceph-mon[37473]: pgmap v9141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:52:41.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:40 smithi134 ceph-mon[31966]: pgmap v9141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:52:41.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:40 smithi189 ceph-mon[36790]: pgmap v9141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:52:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15919.829148 secs 2024-07-26T15:52:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15919.829148 secs 2024-07-26T15:52:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15919.829148 secs 2024-07-26T15:52:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:42 smithi193 ceph-mon[37473]: pgmap v9142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:43.344 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:42 smithi189 ceph-mon[36790]: pgmap v9142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:43.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:42 smithi134 ceph-mon[31966]: pgmap v9142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15905.406357 secs 2024-07-26T15:52:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15905.406357 secs 2024-07-26T15:52:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15905.406357 secs 2024-07-26T15:52:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:44 smithi193 ceph-mon[37473]: pgmap v9143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:44 smithi134 ceph-mon[31966]: pgmap v9143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:45.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:44 smithi189 ceph-mon[36790]: pgmap v9143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:45.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:52:45.546+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:52:45.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:52:45.546+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:45.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:52:45.546+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:52:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:46 smithi193 ceph-mon[37473]: pgmap v9144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15924.829250 secs 2024-07-26T15:52:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:46 smithi134 ceph-mon[31966]: pgmap v9144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15924.829250 secs 2024-07-26T15:52:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:46 smithi189 ceph-mon[36790]: pgmap v9144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15924.829250 secs 2024-07-26T15:52:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:48 smithi193 ceph-mon[37473]: pgmap v9145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15910.406490 secs 2024-07-26T15:52:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:48 smithi134 ceph-mon[31966]: pgmap v9145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15910.406490 secs 2024-07-26T15:52:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:48 smithi189 ceph-mon[36790]: pgmap v9145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:52:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15910.406490 secs 2024-07-26T15:52:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:50 smithi193 ceph-mon[37473]: pgmap v9146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:52:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:50 smithi134 ceph-mon[31966]: pgmap v9146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:52:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:50 smithi189 ceph-mon[36790]: pgmap v9146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:52:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15929.829375 secs 2024-07-26T15:52:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:52:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15929.829375 secs 2024-07-26T15:52:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:52:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15929.829375 secs 2024-07-26T15:52:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:52:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:53 smithi193 ceph-mon[37473]: pgmap v9147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:52:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:52:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:52:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:53 smithi134 ceph-mon[31966]: pgmap v9147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:52:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:52:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:52:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:53 smithi189 ceph-mon[36790]: pgmap v9147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:52:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:52:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:52:53.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:52:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:54 smithi193 ceph-mon[37473]: pgmap v9148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T15:52:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:54 smithi193 ceph-mon[37473]: pgmap v9149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:52:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15915.406630 secs 2024-07-26T15:52:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:54 smithi134 ceph-mon[31966]: pgmap v9148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T15:52:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:54 smithi134 ceph-mon[31966]: pgmap v9149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:52:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15915.406630 secs 2024-07-26T15:52:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:54 smithi189 ceph-mon[36790]: pgmap v9148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T15:52:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:54 smithi189 ceph-mon[36790]: pgmap v9149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:52:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15915.406630 secs 2024-07-26T15:52:56.152 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:52:56.151+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:52:56.153 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:52:56.151+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:52:56.153 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:52:56.151+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:52:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:56 smithi193 ceph-mon[37473]: pgmap v9150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:52:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:56 smithi134 ceph-mon[31966]: pgmap v9150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:52:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:56 smithi189 ceph-mon[36790]: pgmap v9150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:52:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15934.829500 secs 2024-07-26T15:52:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15934.829500 secs 2024-07-26T15:52:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15934.829500 secs 2024-07-26T15:52:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:58 smithi134 ceph-mon[31966]: pgmap v9151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T15:52:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:58 smithi189 ceph-mon[36790]: pgmap v9151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T15:52:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:58 smithi193 ceph-mon[37473]: pgmap v9151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T15:52:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:52:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15920.406775 secs 2024-07-26T15:52:59.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:52:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15920.406775 secs 2024-07-26T15:52:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:52:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15920.406775 secs 2024-07-26T15:53:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:00 smithi134 ceph-mon[31966]: pgmap v9152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:00.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:00 smithi189 ceph-mon[36790]: pgmap v9152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:00 smithi193 ceph-mon[37473]: pgmap v9152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:00.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:53:00.547+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:53:00.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:53:00.547+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:53:00.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:53:00.547+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:53:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:02 smithi134 ceph-mon[31966]: pgmap v9153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15939.829617 secs 2024-07-26T15:53:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:02 smithi189 ceph-mon[36790]: pgmap v9153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15939.829617 secs 2024-07-26T15:53:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:02 smithi193 ceph-mon[37473]: pgmap v9153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15939.829617 secs 2024-07-26T15:53:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:04 smithi134 ceph-mon[31966]: pgmap v9154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:53:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15925.406843 secs 2024-07-26T15:53:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:04 smithi189 ceph-mon[36790]: pgmap v9154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:53:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15925.406843 secs 2024-07-26T15:53:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:04 smithi193 ceph-mon[37473]: pgmap v9154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:53:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15925.406843 secs 2024-07-26T15:53:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:06 smithi134 ceph-mon[31966]: pgmap v9155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:53:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:06 smithi189 ceph-mon[36790]: pgmap v9155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:53:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:06 smithi193 ceph-mon[37473]: pgmap v9155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:53:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15944.829730 secs 2024-07-26T15:53:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15944.829730 secs 2024-07-26T15:53:07.523 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15944.829730 secs 2024-07-26T15:53:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:08 smithi134 ceph-mon[31966]: pgmap v9156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:08 smithi189 ceph-mon[36790]: pgmap v9156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:08 smithi193 ceph-mon[37473]: pgmap v9156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15930.406949 secs 2024-07-26T15:53:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15930.406949 secs 2024-07-26T15:53:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15930.406949 secs 2024-07-26T15:53:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:10 smithi134 ceph-mon[31966]: pgmap v9157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:53:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:10 smithi189 ceph-mon[36790]: pgmap v9157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:53:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:10 smithi193 ceph-mon[37473]: pgmap v9157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:53:11.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:53:11.166+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:53:11.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:53:11.166+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:53:11.168 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:53:11.166+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:53:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:12 smithi134 ceph-mon[31966]: pgmap v9158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:53:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15949.829853 secs 2024-07-26T15:53:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:12 smithi189 ceph-mon[36790]: pgmap v9158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:53:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15949.829853 secs 2024-07-26T15:53:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:12 smithi193 ceph-mon[37473]: pgmap v9158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:53:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15949.829853 secs 2024-07-26T15:53:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:14 smithi134 ceph-mon[31966]: pgmap v9159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:53:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15935.407052 secs 2024-07-26T15:53:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:14 smithi189 ceph-mon[36790]: pgmap v9159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:53:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15935.407052 secs 2024-07-26T15:53:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:14 smithi193 ceph-mon[37473]: pgmap v9159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:53:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15935.407052 secs 2024-07-26T15:53:15.564 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:53:15.562+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:53:15.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:53:15.562+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:53:15.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:53:15.562+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:53:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:16 smithi134 ceph-mon[31966]: pgmap v9160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:53:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:16 smithi189 ceph-mon[36790]: pgmap v9160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:53:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:16 smithi193 ceph-mon[37473]: pgmap v9160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:53:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15954.829983 secs 2024-07-26T15:53:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15954.829983 secs 2024-07-26T15:53:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15954.829983 secs 2024-07-26T15:53:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:18 smithi134 ceph-mon[31966]: pgmap v9161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:18 smithi189 ceph-mon[36790]: pgmap v9161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:18 smithi193 ceph-mon[37473]: pgmap v9161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15940.407147 secs 2024-07-26T15:53:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15940.407147 secs 2024-07-26T15:53:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15940.407147 secs 2024-07-26T15:53:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:20 smithi134 ceph-mon[31966]: pgmap v9162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:20 smithi189 ceph-mon[36790]: pgmap v9162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:20 smithi193 ceph-mon[37473]: pgmap v9162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:22 smithi134 ceph-mon[31966]: pgmap v9163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15959.830103 secs 2024-07-26T15:53:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:22 smithi189 ceph-mon[36790]: pgmap v9163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15959.830103 secs 2024-07-26T15:53:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:22 smithi193 ceph-mon[37473]: pgmap v9163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15959.830103 secs 2024-07-26T15:53:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:24 smithi134 ceph-mon[31966]: pgmap v9164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:53:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15945.407272 secs 2024-07-26T15:53:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:24 smithi189 ceph-mon[36790]: pgmap v9164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:53:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15945.407272 secs 2024-07-26T15:53:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:24 smithi193 ceph-mon[37473]: pgmap v9164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:53:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15945.407272 secs 2024-07-26T15:53:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:53:26.182+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:53:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:53:26.182+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:53:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:53:26.182+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:53:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:26 smithi134 ceph-mon[31966]: pgmap v9165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:53:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:26 smithi189 ceph-mon[36790]: pgmap v9165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:53:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:26 smithi193 ceph-mon[37473]: pgmap v9165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T15:53:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15964.830180 secs 2024-07-26T15:53:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15964.830180 secs 2024-07-26T15:53:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15964.830180 secs 2024-07-26T15:53:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:28 smithi189 ceph-mon[36790]: pgmap v9166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:28 smithi193 ceph-mon[37473]: pgmap v9166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:28 smithi134 ceph-mon[31966]: pgmap v9166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15950.407396 secs 2024-07-26T15:53:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15950.407396 secs 2024-07-26T15:53:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15950.407396 secs 2024-07-26T15:53:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:30 smithi189 ceph-mon[36790]: pgmap v9167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:53:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:30 smithi193 ceph-mon[37473]: pgmap v9167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:53:30.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:53:30.578+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:53:30.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:53:30.578+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:53:30.580 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:53:30.578+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:53:30.580 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:30 smithi134 ceph-mon[31966]: pgmap v9167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:53:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:32 smithi189 ceph-mon[36790]: pgmap v9168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:53:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15969.830270 secs 2024-07-26T15:53:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:32 smithi193 ceph-mon[37473]: pgmap v9168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:53:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15969.830270 secs 2024-07-26T15:53:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:32 smithi134 ceph-mon[31966]: pgmap v9168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:53:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15969.830270 secs 2024-07-26T15:53:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:34 smithi193 ceph-mon[37473]: pgmap v9169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:53:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15955.407507 secs 2024-07-26T15:53:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:34 smithi134 ceph-mon[31966]: pgmap v9169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:53:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15955.407507 secs 2024-07-26T15:53:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:34 smithi189 ceph-mon[36790]: pgmap v9169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:53:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15955.407507 secs 2024-07-26T15:53:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:36 smithi193 ceph-mon[37473]: pgmap v9170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:53:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:36 smithi134 ceph-mon[31966]: pgmap v9170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:53:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:36 smithi189 ceph-mon[36790]: pgmap v9170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:53:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15974.830425 secs 2024-07-26T15:53:37.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15974.830425 secs 2024-07-26T15:53:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15974.830425 secs 2024-07-26T15:53:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:38 smithi193 ceph-mon[37473]: pgmap v9171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:38 smithi134 ceph-mon[31966]: pgmap v9171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:38 smithi189 ceph-mon[36790]: pgmap v9171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15960.407637 secs 2024-07-26T15:53:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15960.407637 secs 2024-07-26T15:53:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15960.407637 secs 2024-07-26T15:53:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:40 smithi193 ceph-mon[37473]: pgmap v9172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:40 smithi134 ceph-mon[31966]: pgmap v9172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:40 smithi189 ceph-mon[36790]: pgmap v9172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:41.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:53:41.197+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:53:41.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:53:41.197+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:53:41.199 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:53:41.197+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:53:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:41 smithi134 ceph-mon[31966]: pgmap v9173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15979.830570 secs 2024-07-26T15:53:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:41 smithi189 ceph-mon[36790]: pgmap v9173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15979.830570 secs 2024-07-26T15:53:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:41 smithi193 ceph-mon[37473]: pgmap v9173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15979.830570 secs 2024-07-26T15:53:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:43 smithi193 ceph-mon[37473]: pgmap v9174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:53:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15965.407758 secs 2024-07-26T15:53:44.397 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:43 smithi189 ceph-mon[36790]: pgmap v9174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:53:44.397 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15965.407758 secs 2024-07-26T15:53:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:43 smithi134 ceph-mon[31966]: pgmap v9174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:53:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15965.407758 secs 2024-07-26T15:53:45.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:53:45.593+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:53:45.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:53:45.593+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:53:45.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:53:45.593+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:53:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:45 smithi193 ceph-mon[37473]: pgmap v9175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:53:46.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:45 smithi134 ceph-mon[31966]: pgmap v9175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:53:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:45 smithi189 ceph-mon[36790]: pgmap v9175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:53:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15984.830687 secs 2024-07-26T15:53:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15984.830687 secs 2024-07-26T15:53:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15984.830687 secs 2024-07-26T15:53:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:48 smithi193 ceph-mon[37473]: pgmap v9176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:48 smithi134 ceph-mon[31966]: pgmap v9176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:48 smithi189 ceph-mon[36790]: pgmap v9176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:53:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15970.407900 secs 2024-07-26T15:53:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15970.407900 secs 2024-07-26T15:53:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15970.407900 secs 2024-07-26T15:53:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:50 smithi193 ceph-mon[37473]: pgmap v9177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:50 smithi134 ceph-mon[31966]: pgmap v9177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:50 smithi189 ceph-mon[36790]: pgmap v9177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:53:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:52 smithi193 ceph-mon[37473]: pgmap v9178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15989.830812 secs 2024-07-26T15:53:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:52 smithi134 ceph-mon[31966]: pgmap v9178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15989.830812 secs 2024-07-26T15:53:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:52 smithi189 ceph-mon[36790]: pgmap v9178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:53:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15989.830812 secs 2024-07-26T15:53:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:54 smithi134 ceph-mon[31966]: pgmap v9179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:53:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:53:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15975.408038 secs 2024-07-26T15:53:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:54 smithi189 ceph-mon[36790]: pgmap v9179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:53:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:53:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15975.408038 secs 2024-07-26T15:53:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:54 smithi193 ceph-mon[37473]: pgmap v9179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:53:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:53:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15975.408038 secs 2024-07-26T15:53:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:53:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:53:55.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:55 smithi134 ceph-mon[31966]: pgmap v9180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:53:55.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:55 smithi134 ceph-mon[31966]: pgmap v9181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:55.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:53:55.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:53:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:53:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:55 smithi189 ceph-mon[36790]: pgmap v9180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:53:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:55 smithi189 ceph-mon[36790]: pgmap v9181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:53:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:53:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:53:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:55 smithi193 ceph-mon[37473]: pgmap v9180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:53:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:55 smithi193 ceph-mon[37473]: pgmap v9181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:53:56.215 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:53:56.213+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:53:56.215 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:53:56.213+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:53:56.215 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:53:56.213+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:53:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:57 smithi134 ceph-mon[31966]: pgmap v9182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:53:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15994.830933 secs 2024-07-26T15:53:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:57 smithi189 ceph-mon[36790]: pgmap v9182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:53:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15994.830933 secs 2024-07-26T15:53:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:57 smithi193 ceph-mon[37473]: pgmap v9182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:53:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15994.830933 secs 2024-07-26T15:53:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:59 smithi134 ceph-mon[31966]: pgmap v9183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:53:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15980.408168 secs 2024-07-26T15:53:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:59 smithi189 ceph-mon[36790]: pgmap v9183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:53:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15980.408168 secs 2024-07-26T15:53:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:59 smithi193 ceph-mon[37473]: pgmap v9183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:53:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:53:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15980.408168 secs 2024-07-26T15:54:00.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:54:00.608+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:54:00.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:54:00.608+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:54:00.611 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:54:00.608+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:54:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:01 smithi189 ceph-mon[36790]: pgmap v9184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:54:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:01 smithi193 ceph-mon[37473]: pgmap v9184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:54:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:01 smithi134 ceph-mon[31966]: pgmap v9184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:54:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 15999.831065 secs 2024-07-26T15:54:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 15999.831065 secs 2024-07-26T15:54:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 15999.831065 secs 2024-07-26T15:54:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:03 smithi189 ceph-mon[36790]: pgmap v9185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:54:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:03 smithi193 ceph-mon[37473]: pgmap v9185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:54:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:03 smithi134 ceph-mon[31966]: pgmap v9185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:54:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15985.408292 secs 2024-07-26T15:54:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15985.408292 secs 2024-07-26T15:54:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15985.408292 secs 2024-07-26T15:54:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:05 smithi193 ceph-mon[37473]: pgmap v9186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:54:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:05 smithi134 ceph-mon[31966]: pgmap v9186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:54:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:05 smithi189 ceph-mon[36790]: pgmap v9186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:54:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:07 smithi193 ceph-mon[37473]: pgmap v9187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16004.831143 secs 2024-07-26T15:54:07.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:07 smithi134 ceph-mon[31966]: pgmap v9187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:07.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16004.831143 secs 2024-07-26T15:54:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:07 smithi189 ceph-mon[36790]: pgmap v9187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16004.831143 secs 2024-07-26T15:54:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:09 smithi193 ceph-mon[37473]: pgmap v9188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:54:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15990.408423 secs 2024-07-26T15:54:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:09 smithi134 ceph-mon[31966]: pgmap v9188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:54:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15990.408423 secs 2024-07-26T15:54:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:09 smithi189 ceph-mon[36790]: pgmap v9188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:54:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15990.408423 secs 2024-07-26T15:54:11.217 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:54:11.215+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:54:11.217 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:54:11.215+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:54:11.217 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:54:11.215+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:54:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:11 smithi193 ceph-mon[37473]: pgmap v9189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:11 smithi134 ceph-mon[31966]: pgmap v9189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:11 smithi189 ceph-mon[36790]: pgmap v9189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16009.831261 secs 2024-07-26T15:54:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16009.831261 secs 2024-07-26T15:54:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16009.831261 secs 2024-07-26T15:54:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:13 smithi193 ceph-mon[37473]: pgmap v9190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:13 smithi134 ceph-mon[31966]: pgmap v9190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:13 smithi189 ceph-mon[36790]: pgmap v9190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 15995.408565 secs 2024-07-26T15:54:14.594 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 15995.408565 secs 2024-07-26T15:54:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 15995.408565 secs 2024-07-26T15:54:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:15 smithi193 ceph-mon[37473]: pgmap v9191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:15.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:54:15.624+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:54:15.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:54:15.624+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:54:15.626 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:54:15.624+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:54:15.627 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:15 smithi134 ceph-mon[31966]: pgmap v9191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:15 smithi189 ceph-mon[36790]: pgmap v9191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:16 smithi134 ceph-mon[31966]: pgmap v9192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16014.831380 secs 2024-07-26T15:54:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:16 smithi189 ceph-mon[36790]: pgmap v9192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16014.831380 secs 2024-07-26T15:54:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:16 smithi193 ceph-mon[37473]: pgmap v9192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16014.831380 secs 2024-07-26T15:54:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:19 smithi134 ceph-mon[31966]: pgmap v9193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:54:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16000.408707 secs 2024-07-26T15:54:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:19 smithi189 ceph-mon[36790]: pgmap v9193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:54:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16000.408707 secs 2024-07-26T15:54:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:19 smithi193 ceph-mon[37473]: pgmap v9193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:54:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16000.408707 secs 2024-07-26T15:54:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:21 smithi134 ceph-mon[31966]: pgmap v9194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:21 smithi189 ceph-mon[36790]: pgmap v9194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:21 smithi193 ceph-mon[37473]: pgmap v9194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16019.831505 secs 2024-07-26T15:54:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16019.831505 secs 2024-07-26T15:54:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16019.831505 secs 2024-07-26T15:54:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:23 smithi189 ceph-mon[36790]: pgmap v9195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:23 smithi193 ceph-mon[37473]: pgmap v9195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:23 smithi134 ceph-mon[31966]: pgmap v9195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16005.408833 secs 2024-07-26T15:54:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16005.408833 secs 2024-07-26T15:54:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16005.408833 secs 2024-07-26T15:54:25.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:54:25.108+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:54:25.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:54:25.108+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:54:25.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:54:25.108+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 125 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:54:25.110 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:54:25.108+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:54:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:25 smithi134 ceph-mon[31966]: pgmap v9196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:25 smithi193 ceph-mon[37473]: pgmap v9196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:25 smithi189 ceph-mon[36790]: pgmap v9196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:54:26.228+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:54:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:54:26.228+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:54:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:54:26.228+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:54:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:27 smithi193 ceph-mon[37473]: pgmap v9197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16024.831626 secs 2024-07-26T15:54:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:27 smithi134 ceph-mon[31966]: pgmap v9197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16024.831626 secs 2024-07-26T15:54:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:27 smithi189 ceph-mon[36790]: pgmap v9197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16024.831626 secs 2024-07-26T15:54:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:29 smithi193 ceph-mon[37473]: pgmap v9198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:54:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16010.408972 secs 2024-07-26T15:54:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:29 smithi134 ceph-mon[31966]: pgmap v9198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:54:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16010.408972 secs 2024-07-26T15:54:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:29 smithi189 ceph-mon[36790]: pgmap v9198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:54:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16010.408972 secs 2024-07-26T15:54:30.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:54:30.640+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:54:30.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:54:30.640+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:54:30.641 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:54:30.640+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:54:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:31 smithi193 ceph-mon[37473]: pgmap v9199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:31 smithi134 ceph-mon[31966]: pgmap v9199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:31 smithi189 ceph-mon[36790]: pgmap v9199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16029.831750 secs 2024-07-26T15:54:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16029.831750 secs 2024-07-26T15:54:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16029.831750 secs 2024-07-26T15:54:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:33 smithi193 ceph-mon[37473]: pgmap v9200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:33 smithi134 ceph-mon[31966]: pgmap v9200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:33 smithi189 ceph-mon[36790]: pgmap v9200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16015.409112 secs 2024-07-26T15:54:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16015.409112 secs 2024-07-26T15:54:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16015.409112 secs 2024-07-26T15:54:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:35 smithi193 ceph-mon[37473]: pgmap v9201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:54:35.657+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:54:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:54:35.657+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:54:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:54:35.658+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:54:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:54:35.658+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:54:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:35 smithi134 ceph-mon[31966]: pgmap v9201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:35 smithi189 ceph-mon[36790]: pgmap v9201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:37 smithi193 ceph-mon[37473]: pgmap v9202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16034.831870 secs 2024-07-26T15:54:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:37 smithi134 ceph-mon[31966]: pgmap v9202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16034.831870 secs 2024-07-26T15:54:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:37 smithi189 ceph-mon[36790]: pgmap v9202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16034.831870 secs 2024-07-26T15:54:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:39 smithi193 ceph-mon[37473]: pgmap v9203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:54:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16020.409197 secs 2024-07-26T15:54:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:39 smithi134 ceph-mon[31966]: pgmap v9203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:54:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16020.409197 secs 2024-07-26T15:54:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:39 smithi189 ceph-mon[36790]: pgmap v9203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:54:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16020.409197 secs 2024-07-26T15:54:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:40 smithi193 ceph-mon[37473]: pgmap v9204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:40 smithi134 ceph-mon[31966]: pgmap v9204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:40 smithi189 ceph-mon[36790]: pgmap v9204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:54:41.243+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:54:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:54:41.243+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:54:41.245 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:54:41.243+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:54:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16039.831991 secs 2024-07-26T15:54:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16039.831991 secs 2024-07-26T15:54:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16039.831991 secs 2024-07-26T15:54:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:42 smithi134 ceph-mon[31966]: pgmap v9205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:42 smithi189 ceph-mon[36790]: pgmap v9205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:42 smithi193 ceph-mon[37473]: pgmap v9205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:54:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16025.409326 secs 2024-07-26T15:54:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16025.409326 secs 2024-07-26T15:54:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16025.409326 secs 2024-07-26T15:54:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:44 smithi193 ceph-mon[37473]: pgmap v9206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:44 smithi134 ceph-mon[31966]: pgmap v9206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:44 smithi189 ceph-mon[36790]: pgmap v9206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:45.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:54:45.655+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:54:45.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:54:45.655+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:54:45.657 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:54:45.655+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:54:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:47 smithi189 ceph-mon[36790]: pgmap v9207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16044.832129 secs 2024-07-26T15:54:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:47 smithi193 ceph-mon[37473]: pgmap v9207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16044.832129 secs 2024-07-26T15:54:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:47 smithi134 ceph-mon[31966]: pgmap v9207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16044.832129 secs 2024-07-26T15:54:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:49 smithi189 ceph-mon[36790]: pgmap v9208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:54:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16030.409431 secs 2024-07-26T15:54:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:49 smithi193 ceph-mon[37473]: pgmap v9208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:54:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16030.409431 secs 2024-07-26T15:54:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:49 smithi134 ceph-mon[31966]: pgmap v9208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:54:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16030.409431 secs 2024-07-26T15:54:51.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:51 smithi189 ceph-mon[36790]: pgmap v9209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:51 smithi193 ceph-mon[37473]: pgmap v9209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:51 smithi134 ceph-mon[31966]: pgmap v9209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:54:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16049.832246 secs 2024-07-26T15:54:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16049.832246 secs 2024-07-26T15:54:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16049.832246 secs 2024-07-26T15:54:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:53 smithi193 ceph-mon[37473]: pgmap v9210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:54:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:53 smithi134 ceph-mon[31966]: pgmap v9210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:54:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:53 smithi189 ceph-mon[36790]: pgmap v9210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:54:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16035.409557 secs 2024-07-26T15:54:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16035.409557 secs 2024-07-26T15:54:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16035.409557 secs 2024-07-26T15:54:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:55 smithi193 ceph-mon[37473]: pgmap v9211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:54:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:55 smithi134 ceph-mon[31966]: pgmap v9211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:54:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:55 smithi189 ceph-mon[36790]: pgmap v9211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:54:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:54:56.261 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:54:56.259+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:54:56.261 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:54:56.259+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:54:56.261 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:54:56.259+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:54:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:54:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:54:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:54:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:54:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:54:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:54:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:54:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:54:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:54:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:57 smithi193 ceph-mon[37473]: pgmap v9212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:54:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:57 smithi193 ceph-mon[37473]: pgmap v9213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16054.832365 secs 2024-07-26T15:54:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:57 smithi134 ceph-mon[31966]: pgmap v9212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:54:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:57 smithi134 ceph-mon[31966]: pgmap v9213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16054.832365 secs 2024-07-26T15:54:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:57 smithi189 ceph-mon[36790]: pgmap v9212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:54:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:57 smithi189 ceph-mon[36790]: pgmap v9213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:54:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16054.832365 secs 2024-07-26T15:54:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:59 smithi193 ceph-mon[37473]: pgmap v9214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:54:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:54:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16040.409688 secs 2024-07-26T15:54:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:59 smithi134 ceph-mon[31966]: pgmap v9214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:54:59.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:54:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16040.409688 secs 2024-07-26T15:54:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:59 smithi189 ceph-mon[36790]: pgmap v9214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:54:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:54:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16040.409688 secs 2024-07-26T15:55:00.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:55:00.670+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:55:00.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:55:00.670+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:55:00.672 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:55:00.670+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:55:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:01 smithi193 ceph-mon[37473]: pgmap v9215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:55:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:01 smithi134 ceph-mon[31966]: pgmap v9215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:55:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:01 smithi189 ceph-mon[36790]: pgmap v9215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:55:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16059.832499 secs 2024-07-26T15:55:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16059.832499 secs 2024-07-26T15:55:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16059.832499 secs 2024-07-26T15:55:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:03 smithi193 ceph-mon[37473]: pgmap v9216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:55:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:03 smithi134 ceph-mon[31966]: pgmap v9216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:55:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:03 smithi189 ceph-mon[36790]: pgmap v9216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:55:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:04 smithi134 ceph-mon[31966]: pgmap v9217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:55:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16045.409848 secs 2024-07-26T15:55:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:04 smithi189 ceph-mon[36790]: pgmap v9217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:55:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16045.409848 secs 2024-07-26T15:55:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:04 smithi193 ceph-mon[37473]: pgmap v9217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T15:55:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16045.409848 secs 2024-07-26T15:55:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:06 smithi134 ceph-mon[31966]: pgmap v9218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:55:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16064.832622 secs 2024-07-26T15:55:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:06 smithi189 ceph-mon[36790]: pgmap v9218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:55:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16064.832622 secs 2024-07-26T15:55:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:06 smithi193 ceph-mon[37473]: pgmap v9218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:55:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16064.832622 secs 2024-07-26T15:55:08.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:08 smithi134 ceph-mon[31966]: pgmap v9219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:55:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:08 smithi189 ceph-mon[36790]: pgmap v9219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:55:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:08 smithi193 ceph-mon[37473]: pgmap v9219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:55:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16050.410007 secs 2024-07-26T15:55:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16050.410007 secs 2024-07-26T15:55:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16050.410007 secs 2024-07-26T15:55:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:10 smithi134 ceph-mon[31966]: pgmap v9220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:55:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:10 smithi189 ceph-mon[36790]: pgmap v9220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:55:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:10 smithi193 ceph-mon[37473]: pgmap v9220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:55:11.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:55:11.275+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:55:11.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:55:11.275+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:55:11.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:55:11.275+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:55:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16069.832708 secs 2024-07-26T15:55:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16069.832708 secs 2024-07-26T15:55:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16069.832708 secs 2024-07-26T15:55:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:12 smithi134 ceph-mon[31966]: pgmap v9221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:12 smithi189 ceph-mon[36790]: pgmap v9221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:12 smithi193 ceph-mon[37473]: pgmap v9221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:14 smithi189 ceph-mon[36790]: pgmap v9222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:55:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16055.410095 secs 2024-07-26T15:55:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:14 smithi193 ceph-mon[37473]: pgmap v9222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:55:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16055.410095 secs 2024-07-26T15:55:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:14 smithi134 ceph-mon[31966]: pgmap v9222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:55:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16055.410095 secs 2024-07-26T15:55:15.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:55:15.679+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:55:15.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:55:15.679+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:55:15.681 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:55:15.679+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:55:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:16 smithi189 ceph-mon[36790]: pgmap v9223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16074.832823 secs 2024-07-26T15:55:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:16 smithi193 ceph-mon[37473]: pgmap v9223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16074.832823 secs 2024-07-26T15:55:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:16 smithi134 ceph-mon[31966]: pgmap v9223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16074.832823 secs 2024-07-26T15:55:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:18 smithi189 ceph-mon[36790]: pgmap v9224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:55:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:18 smithi193 ceph-mon[37473]: pgmap v9224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:55:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:18 smithi134 ceph-mon[31966]: pgmap v9224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:55:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16060.410208 secs 2024-07-26T15:55:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16060.410208 secs 2024-07-26T15:55:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16060.410208 secs 2024-07-26T15:55:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:20 smithi193 ceph-mon[37473]: pgmap v9225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:55:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:20 smithi134 ceph-mon[31966]: pgmap v9225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:55:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:20 smithi189 ceph-mon[36790]: pgmap v9225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:55:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16079.832963 secs 2024-07-26T15:55:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16079.832963 secs 2024-07-26T15:55:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16079.832963 secs 2024-07-26T15:55:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:22 smithi193 ceph-mon[37473]: pgmap v9226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:22 smithi134 ceph-mon[31966]: pgmap v9226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:22 smithi189 ceph-mon[36790]: pgmap v9226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:24 smithi193 ceph-mon[37473]: pgmap v9227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:55:24.796 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16065.410344 secs 2024-07-26T15:55:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:24 smithi134 ceph-mon[31966]: pgmap v9227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:55:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16065.410344 secs 2024-07-26T15:55:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:24 smithi189 ceph-mon[36790]: pgmap v9227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:55:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16065.410344 secs 2024-07-26T15:55:26.290 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:55:26.288+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:55:26.290 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:55:26.288+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:55:26.290 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:55:26.288+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:55:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:26 smithi193 ceph-mon[37473]: pgmap v9228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16084.833077 secs 2024-07-26T15:55:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:26 smithi134 ceph-mon[31966]: pgmap v9228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16084.833077 secs 2024-07-26T15:55:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:26 smithi189 ceph-mon[36790]: pgmap v9228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16084.833077 secs 2024-07-26T15:55:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:28 smithi193 ceph-mon[37473]: pgmap v9229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:55:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:28 smithi134 ceph-mon[31966]: pgmap v9229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:55:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:28 smithi189 ceph-mon[36790]: pgmap v9229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:55:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16070.410504 secs 2024-07-26T15:55:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16070.410504 secs 2024-07-26T15:55:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16070.410504 secs 2024-07-26T15:55:30.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:55:30.688+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:55:30.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:55:30.688+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:55:30.690 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:55:30.688+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:55:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:30 smithi193 ceph-mon[37473]: pgmap v9230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:55:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:30 smithi134 ceph-mon[31966]: pgmap v9230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:55:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:30 smithi189 ceph-mon[36790]: pgmap v9230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:55:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16089.833200 secs 2024-07-26T15:55:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16089.833200 secs 2024-07-26T15:55:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16089.833200 secs 2024-07-26T15:55:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:32 smithi193 ceph-mon[37473]: pgmap v9231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:32 smithi134 ceph-mon[31966]: pgmap v9231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:32 smithi189 ceph-mon[36790]: pgmap v9231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:34 smithi193 ceph-mon[37473]: pgmap v9232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:55:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16075.410631 secs 2024-07-26T15:55:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:34 smithi134 ceph-mon[31966]: pgmap v9232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:55:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16075.410631 secs 2024-07-26T15:55:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:34 smithi189 ceph-mon[36790]: pgmap v9232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:55:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16075.410631 secs 2024-07-26T15:55:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:36 smithi193 ceph-mon[37473]: pgmap v9233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16094.833364 secs 2024-07-26T15:55:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:36 smithi134 ceph-mon[31966]: pgmap v9233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16094.833364 secs 2024-07-26T15:55:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:36 smithi189 ceph-mon[36790]: pgmap v9233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16094.833364 secs 2024-07-26T15:55:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:38 smithi134 ceph-mon[31966]: pgmap v9234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:55:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:38 smithi189 ceph-mon[36790]: pgmap v9234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:55:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:38 smithi193 ceph-mon[37473]: pgmap v9234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T15:55:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16080.410779 secs 2024-07-26T15:55:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16080.410779 secs 2024-07-26T15:55:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16080.410779 secs 2024-07-26T15:55:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:40 smithi134 ceph-mon[31966]: pgmap v9235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:55:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:40 smithi189 ceph-mon[36790]: pgmap v9235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:55:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:40 smithi193 ceph-mon[37473]: pgmap v9235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T15:55:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:55:41.304+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:55:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:55:41.304+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:55:41.305 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:55:41.304+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:55:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16099.833479 secs 2024-07-26T15:55:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16099.833479 secs 2024-07-26T15:55:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16099.833479 secs 2024-07-26T15:55:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:42 smithi134 ceph-mon[31966]: pgmap v9236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:42 smithi189 ceph-mon[36790]: pgmap v9236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:42 smithi193 ceph-mon[37473]: pgmap v9236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:44 smithi134 ceph-mon[31966]: pgmap v9237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:55:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16085.410923 secs 2024-07-26T15:55:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:44 smithi189 ceph-mon[36790]: pgmap v9237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:55:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16085.410923 secs 2024-07-26T15:55:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:44 smithi193 ceph-mon[37473]: pgmap v9237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:55:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16085.410923 secs 2024-07-26T15:55:45.705 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:55:45.704+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:55:45.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:55:45.704+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:55:45.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:55:45.704+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:55:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:46 smithi134 ceph-mon[31966]: pgmap v9238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:55:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16104.833613 secs 2024-07-26T15:55:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:46 smithi189 ceph-mon[36790]: pgmap v9238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:55:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16104.833613 secs 2024-07-26T15:55:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:46 smithi193 ceph-mon[37473]: pgmap v9238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:55:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16104.833613 secs 2024-07-26T15:55:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:48 smithi134 ceph-mon[31966]: pgmap v9239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:55:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:48 smithi189 ceph-mon[36790]: pgmap v9239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:55:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:48 smithi193 ceph-mon[37473]: pgmap v9239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:55:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16090.411029 secs 2024-07-26T15:55:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16090.411029 secs 2024-07-26T15:55:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16090.411029 secs 2024-07-26T15:55:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:50 smithi134 ceph-mon[31966]: pgmap v9240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:55:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:50 smithi189 ceph-mon[36790]: pgmap v9240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:55:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:50 smithi193 ceph-mon[37473]: pgmap v9240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T15:55:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16109.833744 secs 2024-07-26T15:55:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16109.833744 secs 2024-07-26T15:55:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16109.833744 secs 2024-07-26T15:55:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:52 smithi134 ceph-mon[31966]: pgmap v9241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:55:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:52 smithi189 ceph-mon[36790]: pgmap v9241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:55:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:52 smithi193 ceph-mon[37473]: pgmap v9241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:55:54.889 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:54 smithi134 ceph-mon[31966]: pgmap v9242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:55:54.889 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16095.411159 secs 2024-07-26T15:55:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:54 smithi189 ceph-mon[36790]: pgmap v9242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:55:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16095.411159 secs 2024-07-26T15:55:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:54 smithi193 ceph-mon[37473]: pgmap v9242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T15:55:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16095.411159 secs 2024-07-26T15:55:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:55:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:55:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:55:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:55:56.319+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:55:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:55:56.319+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:55:56.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:55:56.319+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:55:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:56 smithi134 ceph-mon[31966]: pgmap v9243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16114.833842 secs 2024-07-26T15:55:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:55:56.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:55:56.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:55:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:56 smithi189 ceph-mon[36790]: pgmap v9243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16114.833842 secs 2024-07-26T15:55:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:55:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:55:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:55:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:56 smithi193 ceph-mon[37473]: pgmap v9243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16114.833842 secs 2024-07-26T15:55:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:55:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:55:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:55:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:57 smithi134 ceph-mon[31966]: pgmap v9244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:55:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:57 smithi134 ceph-mon[31966]: pgmap v9245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:57 smithi189 ceph-mon[36790]: pgmap v9244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:55:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:57 smithi189 ceph-mon[36790]: pgmap v9245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:57 smithi193 ceph-mon[37473]: pgmap v9244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:55:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:57 smithi193 ceph-mon[37473]: pgmap v9245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:55:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:59 smithi134 ceph-mon[31966]: pgmap v9246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:55:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:55:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16100.411237 secs 2024-07-26T15:55:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:59 smithi189 ceph-mon[36790]: pgmap v9246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:55:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:55:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16100.411237 secs 2024-07-26T15:56:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:59 smithi193 ceph-mon[37473]: pgmap v9246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:56:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:55:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16100.411237 secs 2024-07-26T15:56:00.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:56:00.705+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:56:00.707 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:56:00.705+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:56:00.708 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:56:00.705+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:56:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:01 smithi134 ceph-mon[31966]: pgmap v9247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:56:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16119.833976 secs 2024-07-26T15:56:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:01 smithi189 ceph-mon[36790]: pgmap v9247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:56:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16119.833976 secs 2024-07-26T15:56:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:01 smithi193 ceph-mon[37473]: pgmap v9247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:56:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16119.833976 secs 2024-07-26T15:56:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:03 smithi134 ceph-mon[31966]: pgmap v9248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:56:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:03 smithi189 ceph-mon[36790]: pgmap v9248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:56:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:03 smithi193 ceph-mon[37473]: pgmap v9248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:56:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16105.411310 secs 2024-07-26T15:56:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16105.411310 secs 2024-07-26T15:56:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16105.411310 secs 2024-07-26T15:56:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:05 smithi134 ceph-mon[31966]: pgmap v9249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:56:05.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:05 smithi189 ceph-mon[36790]: pgmap v9249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:56:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:05 smithi193 ceph-mon[37473]: pgmap v9249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:56:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16124.834108 secs 2024-07-26T15:56:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16124.834108 secs 2024-07-26T15:56:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16124.834108 secs 2024-07-26T15:56:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:07 smithi134 ceph-mon[31966]: pgmap v9250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:56:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:07 smithi189 ceph-mon[36790]: pgmap v9250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:56:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:07 smithi193 ceph-mon[37473]: pgmap v9250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:56:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:09 smithi189 ceph-mon[36790]: pgmap v9251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16110.411392 secs 2024-07-26T15:56:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:09 smithi193 ceph-mon[37473]: pgmap v9251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16110.411392 secs 2024-07-26T15:56:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:09 smithi134 ceph-mon[31966]: pgmap v9251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16110.411392 secs 2024-07-26T15:56:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:56:11.323+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:56:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:56:11.323+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:56:11.325 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:56:11.323+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:56:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:11 smithi189 ceph-mon[36790]: pgmap v9252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16129.834235 secs 2024-07-26T15:56:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:11 smithi193 ceph-mon[37473]: pgmap v9252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16129.834235 secs 2024-07-26T15:56:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:11 smithi134 ceph-mon[31966]: pgmap v9252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16129.834235 secs 2024-07-26T15:56:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:13 smithi189 ceph-mon[36790]: pgmap v9253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:13 smithi193 ceph-mon[37473]: pgmap v9253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:13 smithi134 ceph-mon[31966]: pgmap v9253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16115.411509 secs 2024-07-26T15:56:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16115.411509 secs 2024-07-26T15:56:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16115.411509 secs 2024-07-26T15:56:15.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:56:15.721+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:56:15.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:56:15.721+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:56:15.723 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:56:15.721+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:56:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:15 smithi193 ceph-mon[37473]: pgmap v9254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:56:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:15 smithi134 ceph-mon[31966]: pgmap v9254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:56:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:15 smithi189 ceph-mon[36790]: pgmap v9254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:56:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16134.834297 secs 2024-07-26T15:56:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16134.834297 secs 2024-07-26T15:56:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16134.834297 secs 2024-07-26T15:56:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:17 smithi134 ceph-mon[31966]: pgmap v9255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:18.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:17 smithi189 ceph-mon[36790]: pgmap v9255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:17 smithi193 ceph-mon[37473]: pgmap v9255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16120.411664 secs 2024-07-26T15:56:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16120.411664 secs 2024-07-26T15:56:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16120.411664 secs 2024-07-26T15:56:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:19 smithi134 ceph-mon[31966]: pgmap v9256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:19 smithi189 ceph-mon[36790]: pgmap v9256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:19 smithi193 ceph-mon[37473]: pgmap v9256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:21 smithi134 ceph-mon[31966]: pgmap v9257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16139.834417 secs 2024-07-26T15:56:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:21 smithi189 ceph-mon[36790]: pgmap v9257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:22.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16139.834417 secs 2024-07-26T15:56:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:21 smithi193 ceph-mon[37473]: pgmap v9257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16139.834417 secs 2024-07-26T15:56:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:23 smithi134 ceph-mon[31966]: pgmap v9258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16125.411813 secs 2024-07-26T15:56:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:23 smithi189 ceph-mon[36790]: pgmap v9258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16125.411813 secs 2024-07-26T15:56:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:23 smithi193 ceph-mon[37473]: pgmap v9258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16125.411813 secs 2024-07-26T15:56:26.121 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:25 smithi134 ceph-mon[31966]: pgmap v9259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:56:26.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:25 smithi189 ceph-mon[36790]: pgmap v9259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:56:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:25 smithi193 ceph-mon[37473]: pgmap v9259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:56:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:56:26.327+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:56:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:56:26.327+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:56:26.329 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:56:26.327+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:56:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16144.834481 secs 2024-07-26T15:56:27.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16144.834481 secs 2024-07-26T15:56:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16144.834481 secs 2024-07-26T15:56:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:27 smithi134 ceph-mon[31966]: pgmap v9260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:27 smithi189 ceph-mon[36790]: pgmap v9260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:27 smithi193 ceph-mon[37473]: pgmap v9260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16130.411995 secs 2024-07-26T15:56:29.175 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16130.411995 secs 2024-07-26T15:56:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16130.411995 secs 2024-07-26T15:56:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:29 smithi134 ceph-mon[31966]: pgmap v9261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:29 smithi189 ceph-mon[36790]: pgmap v9261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:29 smithi193 ceph-mon[37473]: pgmap v9261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:30.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:56:30.736+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:56:30.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:56:30.736+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:56:30.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:56:30.736+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:56:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:31 smithi134 ceph-mon[31966]: pgmap v9262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16149.834549 secs 2024-07-26T15:56:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:31 smithi189 ceph-mon[36790]: pgmap v9262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16149.834549 secs 2024-07-26T15:56:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:31 smithi193 ceph-mon[37473]: pgmap v9262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16149.834549 secs 2024-07-26T15:56:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:33 smithi134 ceph-mon[31966]: pgmap v9263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16135.412129 secs 2024-07-26T15:56:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:33 smithi189 ceph-mon[36790]: pgmap v9263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16135.412129 secs 2024-07-26T15:56:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:33 smithi193 ceph-mon[37473]: pgmap v9263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16135.412129 secs 2024-07-26T15:56:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:35 smithi134 ceph-mon[31966]: pgmap v9264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:56:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:35 smithi189 ceph-mon[36790]: pgmap v9264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:56:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:35 smithi193 ceph-mon[37473]: pgmap v9264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:56:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16154.834663 secs 2024-07-26T15:56:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16154.834663 secs 2024-07-26T15:56:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16154.834663 secs 2024-07-26T15:56:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:37 smithi134 ceph-mon[31966]: pgmap v9265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:38.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:37 smithi189 ceph-mon[36790]: pgmap v9265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:37 smithi193 ceph-mon[37473]: pgmap v9265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16140.412235 secs 2024-07-26T15:56:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16140.412235 secs 2024-07-26T15:56:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16140.412235 secs 2024-07-26T15:56:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:39 smithi134 ceph-mon[31966]: pgmap v9266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:39 smithi189 ceph-mon[36790]: pgmap v9266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:39 smithi193 ceph-mon[37473]: pgmap v9266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:56:41.342+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:56:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:56:41.343+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:56:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:56:41.343+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:56:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:41 smithi134 ceph-mon[31966]: pgmap v9267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16159.834784 secs 2024-07-26T15:56:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:41 smithi189 ceph-mon[36790]: pgmap v9267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16159.834784 secs 2024-07-26T15:56:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:41 smithi193 ceph-mon[37473]: pgmap v9267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16159.834784 secs 2024-07-26T15:56:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:43 smithi134 ceph-mon[31966]: pgmap v9268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16145.412354 secs 2024-07-26T15:56:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:43 smithi189 ceph-mon[36790]: pgmap v9268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16145.412354 secs 2024-07-26T15:56:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:43 smithi193 ceph-mon[37473]: pgmap v9268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16145.412354 secs 2024-07-26T15:56:45.744 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:56:45.742+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:56:45.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:56:45.742+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:56:45.745 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:56:45.743+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:56:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:45 smithi134 ceph-mon[31966]: pgmap v9269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:56:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:45 smithi189 ceph-mon[36790]: pgmap v9269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:56:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:45 smithi193 ceph-mon[37473]: pgmap v9269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:56:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16164.834942 secs 2024-07-26T15:56:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16164.834942 secs 2024-07-26T15:56:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16164.834942 secs 2024-07-26T15:56:48.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:47 smithi189 ceph-mon[36790]: pgmap v9270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:47 smithi193 ceph-mon[37473]: pgmap v9270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:47 smithi134 ceph-mon[31966]: pgmap v9270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16150.412477 secs 2024-07-26T15:56:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16150.412477 secs 2024-07-26T15:56:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16150.412477 secs 2024-07-26T15:56:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:49 smithi193 ceph-mon[37473]: pgmap v9271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:49 smithi134 ceph-mon[31966]: pgmap v9271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:49 smithi189 ceph-mon[36790]: pgmap v9271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:51 smithi193 ceph-mon[37473]: pgmap v9272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16169.835065 secs 2024-07-26T15:56:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:51 smithi134 ceph-mon[31966]: pgmap v9272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16169.835065 secs 2024-07-26T15:56:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:51 smithi189 ceph-mon[36790]: pgmap v9272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16169.835065 secs 2024-07-26T15:56:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:53 smithi193 ceph-mon[37473]: pgmap v9273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16155.412614 secs 2024-07-26T15:56:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:53 smithi134 ceph-mon[31966]: pgmap v9273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16155.412614 secs 2024-07-26T15:56:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:53 smithi189 ceph-mon[36790]: pgmap v9273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:56:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16155.412614 secs 2024-07-26T15:56:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:55 smithi193 ceph-mon[37473]: pgmap v9274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:56:56.359 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:56:56.358+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:56:56.358+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:56:56.360 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:56:56.358+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:56:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:55 smithi134 ceph-mon[31966]: pgmap v9274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:56:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:55 smithi189 ceph-mon[36790]: pgmap v9274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:56:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16174.835185 secs 2024-07-26T15:56:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:56:57.353 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16174.835185 secs 2024-07-26T15:56:57.353 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:56:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16174.835185 secs 2024-07-26T15:56:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:56:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:57 smithi193 ceph-mon[37473]: pgmap v9275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:56:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:56:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:56:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:57 smithi134 ceph-mon[31966]: pgmap v9275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:58.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:56:58.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:56:58.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:56:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:57 smithi189 ceph-mon[36790]: pgmap v9275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:56:58.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:56:58.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:56:58.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:56:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:58 smithi193 ceph-mon[37473]: pgmap v9276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T15:56:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:58 smithi193 ceph-mon[37473]: pgmap v9277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:56:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:56:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16160.412731 secs 2024-07-26T15:56:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:58 smithi134 ceph-mon[31966]: pgmap v9276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T15:56:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:58 smithi134 ceph-mon[31966]: pgmap v9277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:56:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:56:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16160.412731 secs 2024-07-26T15:56:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:58 smithi189 ceph-mon[36790]: pgmap v9276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T15:56:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:58 smithi189 ceph-mon[36790]: pgmap v9277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:56:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:56:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16160.412731 secs 2024-07-26T15:57:00.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:00.758+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:57:00.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:00.758+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:57:00.760 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:00.758+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:57:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:00 smithi134 ceph-mon[31966]: pgmap v9278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:57:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:00 smithi193 ceph-mon[37473]: pgmap v9278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:57:01.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:00 smithi189 ceph-mon[36790]: pgmap v9278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:57:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16179.835254 secs 2024-07-26T15:57:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16179.835254 secs 2024-07-26T15:57:02.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16179.835254 secs 2024-07-26T15:57:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:03 smithi193 ceph-mon[37473]: pgmap v9279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:57:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:03 smithi134 ceph-mon[31966]: pgmap v9279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:57:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:03 smithi189 ceph-mon[36790]: pgmap v9279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T15:57:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16165.412877 secs 2024-07-26T15:57:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16165.412877 secs 2024-07-26T15:57:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16165.412877 secs 2024-07-26T15:57:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:05 smithi193 ceph-mon[37473]: pgmap v9280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:57:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:05 smithi134 ceph-mon[31966]: pgmap v9280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:57:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:05 smithi189 ceph-mon[36790]: pgmap v9280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:57:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:07 smithi134 ceph-mon[31966]: pgmap v9281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:57:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16184.835409 secs 2024-07-26T15:57:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:07 smithi189 ceph-mon[36790]: pgmap v9281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:57:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16184.835409 secs 2024-07-26T15:57:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:07 smithi193 ceph-mon[37473]: pgmap v9281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:57:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16184.835409 secs 2024-07-26T15:57:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:09 smithi134 ceph-mon[31966]: pgmap v9282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:57:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16170.413011 secs 2024-07-26T15:57:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:09 smithi189 ceph-mon[36790]: pgmap v9282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:57:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16170.413011 secs 2024-07-26T15:57:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:09 smithi193 ceph-mon[37473]: pgmap v9282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:57:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16170.413011 secs 2024-07-26T15:57:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:11.374+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:57:11.375 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:11.374+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:57:11.376 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:11.374+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:57:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:11 smithi134 ceph-mon[31966]: pgmap v9283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:57:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:11 smithi189 ceph-mon[36790]: pgmap v9283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:57:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:11 smithi193 ceph-mon[37473]: pgmap v9283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:57:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16189.835553 secs 2024-07-26T15:57:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16189.835553 secs 2024-07-26T15:57:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16189.835553 secs 2024-07-26T15:57:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:13 smithi134 ceph-mon[31966]: pgmap v9284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:13 smithi189 ceph-mon[36790]: pgmap v9284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:13 smithi193 ceph-mon[37473]: pgmap v9284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16175.413132 secs 2024-07-26T15:57:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16175.413132 secs 2024-07-26T15:57:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16175.413132 secs 2024-07-26T15:57:15.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:15 smithi189 ceph-mon[36790]: pgmap v9285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:15 smithi134 ceph-mon[31966]: pgmap v9285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:15 smithi193 ceph-mon[37473]: pgmap v9285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:15.764 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:15.763+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:57:15.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:15.763+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:57:15.765 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:15.763+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:57:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:17 smithi134 ceph-mon[31966]: pgmap v9286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16194.835697 secs 2024-07-26T15:57:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:17 smithi189 ceph-mon[36790]: pgmap v9286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16194.835697 secs 2024-07-26T15:57:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:17 smithi193 ceph-mon[37473]: pgmap v9286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16194.835697 secs 2024-07-26T15:57:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:19 smithi134 ceph-mon[31966]: pgmap v9287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16180.413248 secs 2024-07-26T15:57:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:19 smithi189 ceph-mon[36790]: pgmap v9287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16180.413248 secs 2024-07-26T15:57:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:19 smithi193 ceph-mon[37473]: pgmap v9287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16180.413248 secs 2024-07-26T15:57:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:21 smithi134 ceph-mon[31966]: pgmap v9288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:57:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:21 smithi189 ceph-mon[36790]: pgmap v9288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:57:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:21 smithi193 ceph-mon[37473]: pgmap v9288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:57:22.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:22.342+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 msgr2=0x7f9b300c0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000893) for more than 900000000 us, fault. 2024-07-26T15:57:22.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:22.342+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 0x7f9b300c0be0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b38044040 comp rx=0 tx=0).stop 2024-07-26T15:57:22.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:22.342+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 msgr2=0x7f9b300c0be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:57:22.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:22.342+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 0x7f9b300c0be0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:57:22.343 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:22.342+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 0x7f9b300c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:57:22.344 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:22.342+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 0x7f9b300c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:57:22.345 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:22.345+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 0x7f9b300c15d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800e640 tx=0x7f9b38008040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:57:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16199.835867 secs 2024-07-26T15:57:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16199.835867 secs 2024-07-26T15:57:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16199.835867 secs 2024-07-26T15:57:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:23 smithi134 ceph-mon[31966]: pgmap v9289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:23 smithi189 ceph-mon[36790]: pgmap v9289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:23 smithi193 ceph-mon[37473]: pgmap v9289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16185.413330 secs 2024-07-26T15:57:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16185.413330 secs 2024-07-26T15:57:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16185.413330 secs 2024-07-26T15:57:25.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:25.138+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:57:25.140 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:25.138+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:57:25.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:25.139+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T15:57:25.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:25.139+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 126 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T15:57:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:25 smithi134 ceph-mon[31966]: pgmap v9290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:25 smithi189 ceph-mon[36790]: pgmap v9290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:25 smithi193 ceph-mon[37473]: pgmap v9290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:26.389+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:57:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:26.389+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:57:26.391 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:26.389+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:57:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:27 smithi134 ceph-mon[31966]: pgmap v9291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16204.835990 secs 2024-07-26T15:57:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:27 smithi189 ceph-mon[36790]: pgmap v9291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16204.835990 secs 2024-07-26T15:57:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:27 smithi193 ceph-mon[37473]: pgmap v9291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16204.835990 secs 2024-07-26T15:57:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:29 smithi134 ceph-mon[31966]: pgmap v9292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16190.413443 secs 2024-07-26T15:57:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:29 smithi189 ceph-mon[36790]: pgmap v9292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16190.413443 secs 2024-07-26T15:57:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:29 smithi193 ceph-mon[37473]: pgmap v9292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16190.413443 secs 2024-07-26T15:57:30.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:30.771+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:57:30.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:30.771+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:57:30.773 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:30.771+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:57:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:31 smithi134 ceph-mon[31966]: pgmap v9293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:57:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:31 smithi189 ceph-mon[36790]: pgmap v9293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:57:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:31 smithi193 ceph-mon[37473]: pgmap v9293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:57:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16209.836120 secs 2024-07-26T15:57:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16209.836120 secs 2024-07-26T15:57:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16209.836120 secs 2024-07-26T15:57:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:32.895+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 msgr2=0x7f21ac0c10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T15:57:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:32.895+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 msgr2=0x7f21ac0c10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T15:57:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:32.895+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f21b8005f70 tx=0x7f21b80ba000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T15:57:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:32.895+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f21b8005f70 tx=0x7f21b80ba000 comp rx=0 tx=0).stop 2024-07-26T15:57:32.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:32.895+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 msgr2=0x7f21ac0c10b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T15:57:32.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:32.895+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T15:57:32.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:32.895+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T15:57:32.898 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:32.896+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T15:57:32.899 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:32.898+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f21980057a0 tx=0x7f21b8045000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T15:57:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:33 smithi134 ceph-mon[31966]: pgmap v9294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:33.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:33 smithi189 ceph-mon[36790]: pgmap v9294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:33 smithi193 ceph-mon[37473]: pgmap v9294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16195.413584 secs 2024-07-26T15:57:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16195.413584 secs 2024-07-26T15:57:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16195.413584 secs 2024-07-26T15:57:35.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:35 smithi189 ceph-mon[36790]: pgmap v9295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:35 smithi193 ceph-mon[37473]: pgmap v9295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:35 smithi134 ceph-mon[31966]: pgmap v9295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:35.690+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:57:35.691 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:35.690+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:57:35.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:35.690+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:57:35.692 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:35.690+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:57:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:37 smithi189 ceph-mon[36790]: pgmap v9296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16214.836246 secs 2024-07-26T15:57:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:37 smithi193 ceph-mon[37473]: pgmap v9296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16214.836246 secs 2024-07-26T15:57:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:37 smithi134 ceph-mon[31966]: pgmap v9296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16214.836246 secs 2024-07-26T15:57:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:39 smithi189 ceph-mon[36790]: pgmap v9297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16200.413710 secs 2024-07-26T15:57:39.475 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:39 smithi134 ceph-mon[31966]: pgmap v9297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:39.475 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16200.413710 secs 2024-07-26T15:57:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:39 smithi193 ceph-mon[37473]: pgmap v9297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16200.413710 secs 2024-07-26T15:57:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:41.404+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:57:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:41.404+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:57:41.406 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:41.404+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:57:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:41 smithi193 ceph-mon[37473]: pgmap v9298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:57:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:41 smithi134 ceph-mon[31966]: pgmap v9298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:57:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:41 smithi189 ceph-mon[36790]: pgmap v9298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:57:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16219.836369 secs 2024-07-26T15:57:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16219.836369 secs 2024-07-26T15:57:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16219.836369 secs 2024-07-26T15:57:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:43 smithi193 ceph-mon[37473]: pgmap v9299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:43.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:43 smithi134 ceph-mon[31966]: pgmap v9299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:43 smithi189 ceph-mon[36790]: pgmap v9299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16205.413818 secs 2024-07-26T15:57:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16205.413818 secs 2024-07-26T15:57:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16205.413818 secs 2024-07-26T15:57:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:45 smithi193 ceph-mon[37473]: pgmap v9300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:45.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:45 smithi189 ceph-mon[36790]: pgmap v9300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:45 smithi134 ceph-mon[31966]: pgmap v9300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:45.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:45.775+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:57:45.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:45.775+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:57:45.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:57:45.775+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:57:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:47 smithi193 ceph-mon[37473]: pgmap v9301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16224.836497 secs 2024-07-26T15:57:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:47 smithi134 ceph-mon[31966]: pgmap v9301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16224.836497 secs 2024-07-26T15:57:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:47 smithi189 ceph-mon[36790]: pgmap v9301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16224.836497 secs 2024-07-26T15:57:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:49 smithi193 ceph-mon[37473]: pgmap v9302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16210.413967 secs 2024-07-26T15:57:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:49 smithi134 ceph-mon[31966]: pgmap v9302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16210.413967 secs 2024-07-26T15:57:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:49 smithi189 ceph-mon[36790]: pgmap v9302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16210.413967 secs 2024-07-26T15:57:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:51 smithi193 ceph-mon[37473]: pgmap v9303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:57:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:51 smithi134 ceph-mon[31966]: pgmap v9303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:57:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:51 smithi189 ceph-mon[36790]: pgmap v9303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:57:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16229.836579 secs 2024-07-26T15:57:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16229.836579 secs 2024-07-26T15:57:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16229.836579 secs 2024-07-26T15:57:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:53 smithi193 ceph-mon[37473]: pgmap v9304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:53 smithi134 ceph-mon[31966]: pgmap v9304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:53 smithi189 ceph-mon[36790]: pgmap v9304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16215.414099 secs 2024-07-26T15:57:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16215.414099 secs 2024-07-26T15:57:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16215.414099 secs 2024-07-26T15:57:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:55 smithi193 ceph-mon[37473]: pgmap v9305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:55 smithi134 ceph-mon[31966]: pgmap v9305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:55 smithi189 ceph-mon[36790]: pgmap v9305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:56.406+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:57:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:56.406+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:57:56.408 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:57:56.406+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:57:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:57 smithi193 ceph-mon[37473]: pgmap v9306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16234.836669 secs 2024-07-26T15:57:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:57 smithi134 ceph-mon[31966]: pgmap v9306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16234.836669 secs 2024-07-26T15:57:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:57 smithi189 ceph-mon[36790]: pgmap v9306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:57:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16234.836669 secs 2024-07-26T15:57:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:57:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:57:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:57:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:59 smithi134 ceph-mon[31966]: pgmap v9307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:59.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16220.414222 secs 2024-07-26T15:57:59.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:59.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:59.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:59.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:59.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:57:59.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:57:59.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:57:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:57:59.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:59 smithi193 ceph-mon[37473]: pgmap v9307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:59.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16220.414222 secs 2024-07-26T15:57:59.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:59.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:59.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:59.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:59.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:57:59.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:57:59.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:57:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:57:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:59 smithi189 ceph-mon[36790]: pgmap v9307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:57:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16220.414222 secs 2024-07-26T15:57:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:57:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:57:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:57:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:57:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:58:00.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:00 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:58:00.575 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:00 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:58:00.575 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:00 smithi134 ceph-mon[31966]: pgmap v9308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:58:00.575 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:00 smithi134 ceph-mon[31966]: pgmap v9309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:00 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:58:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:00 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:58:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:00 smithi189 ceph-mon[36790]: pgmap v9308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:58:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:00 smithi189 ceph-mon[36790]: pgmap v9309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:00 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T15:58:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:00 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T15:58:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:00 smithi193 ceph-mon[37473]: pgmap v9308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T15:58:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:00 smithi193 ceph-mon[37473]: pgmap v9309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:00.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:58:00.780+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:58:00.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:58:00.780+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:58:00.782 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:58:00.780+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:58:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:01 smithi134 ceph-mon[31966]: pgmap v9310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16239.836795 secs 2024-07-26T15:58:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:01 smithi189 ceph-mon[36790]: pgmap v9310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16239.836795 secs 2024-07-26T15:58:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:01 smithi193 ceph-mon[37473]: pgmap v9310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16239.836795 secs 2024-07-26T15:58:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:03 smithi193 ceph-mon[37473]: pgmap v9311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16225.414304 secs 2024-07-26T15:58:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:03 smithi134 ceph-mon[31966]: pgmap v9311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16225.414304 secs 2024-07-26T15:58:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:03 smithi189 ceph-mon[36790]: pgmap v9311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:58:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16225.414304 secs 2024-07-26T15:58:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:05 smithi193 ceph-mon[37473]: pgmap v9312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:58:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:05 smithi134 ceph-mon[31966]: pgmap v9312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:58:06.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:05 smithi189 ceph-mon[36790]: pgmap v9312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T15:58:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16244.836864 secs 2024-07-26T15:58:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16244.836864 secs 2024-07-26T15:58:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16244.836864 secs 2024-07-26T15:58:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:07 smithi193 ceph-mon[37473]: pgmap v9313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:58:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:07 smithi134 ceph-mon[31966]: pgmap v9313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:58:08.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:07 smithi189 ceph-mon[36790]: pgmap v9313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:58:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16230.414379 secs 2024-07-26T15:58:09.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16230.414379 secs 2024-07-26T15:58:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16230.414379 secs 2024-07-26T15:58:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:09 smithi193 ceph-mon[37473]: pgmap v9314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:58:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:09 smithi134 ceph-mon[31966]: pgmap v9314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:58:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:09 smithi189 ceph-mon[36790]: pgmap v9314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:58:11.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:58:11.416+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:58:11.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:58:11.416+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:58:11.418 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:58:11.416+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:58:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:12 smithi193 ceph-mon[37473]: pgmap v9315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16249.836988 secs 2024-07-26T15:58:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:12 smithi134 ceph-mon[31966]: pgmap v9315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16249.836988 secs 2024-07-26T15:58:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:12 smithi189 ceph-mon[36790]: pgmap v9315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16249.836988 secs 2024-07-26T15:58:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:14 smithi193 ceph-mon[37473]: pgmap v9316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16235.414491 secs 2024-07-26T15:58:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:14 smithi134 ceph-mon[31966]: pgmap v9316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16235.414491 secs 2024-07-26T15:58:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:14 smithi189 ceph-mon[36790]: pgmap v9316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16235.414491 secs 2024-07-26T15:58:15.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:58:15.795+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:58:15.799 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:58:15.796+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:58:15.800 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:58:15.796+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:58:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:16 smithi134 ceph-mon[31966]: pgmap v9317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:16 smithi193 ceph-mon[37473]: pgmap v9317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:16.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:16 smithi189 ceph-mon[36790]: pgmap v9317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16254.837111 secs 2024-07-26T15:58:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16254.837111 secs 2024-07-26T15:58:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16254.837111 secs 2024-07-26T15:58:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:18 smithi134 ceph-mon[31966]: pgmap v9318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:18.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:18 smithi189 ceph-mon[36790]: pgmap v9318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:18 smithi193 ceph-mon[37473]: pgmap v9318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16240.414609 secs 2024-07-26T15:58:19.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16240.414609 secs 2024-07-26T15:58:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16240.414609 secs 2024-07-26T15:58:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:20 smithi134 ceph-mon[31966]: pgmap v9319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:20 smithi189 ceph-mon[36790]: pgmap v9319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:20 smithi193 ceph-mon[37473]: pgmap v9319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:22 smithi134 ceph-mon[31966]: pgmap v9320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16259.837254 secs 2024-07-26T15:58:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:22 smithi189 ceph-mon[36790]: pgmap v9320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16259.837254 secs 2024-07-26T15:58:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:22 smithi193 ceph-mon[37473]: pgmap v9320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16259.837254 secs 2024-07-26T15:58:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:24 smithi134 ceph-mon[31966]: pgmap v9321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16245.414732 secs 2024-07-26T15:58:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:24 smithi189 ceph-mon[36790]: pgmap v9321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16245.414732 secs 2024-07-26T15:58:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:24 smithi193 ceph-mon[37473]: pgmap v9321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16245.414732 secs 2024-07-26T15:58:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:26 smithi134 ceph-mon[31966]: pgmap v9322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:58:26.424+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:58:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:58:26.424+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:58:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:58:26.424+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:58:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:26 smithi189 ceph-mon[36790]: pgmap v9322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:26 smithi193 ceph-mon[37473]: pgmap v9322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16264.837373 secs 2024-07-26T15:58:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16264.837373 secs 2024-07-26T15:58:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16264.837373 secs 2024-07-26T15:58:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:28 smithi134 ceph-mon[31966]: pgmap v9323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:28 smithi189 ceph-mon[36790]: pgmap v9323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:28 smithi193 ceph-mon[37473]: pgmap v9323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16250.414869 secs 2024-07-26T15:58:29.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16250.414869 secs 2024-07-26T15:58:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16250.414869 secs 2024-07-26T15:58:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:30 smithi134 ceph-mon[31966]: pgmap v9324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:30 smithi189 ceph-mon[36790]: pgmap v9324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:30 smithi193 ceph-mon[37473]: pgmap v9324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:30.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:58:30.811+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:58:30.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:58:30.811+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:58:30.813 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:58:30.811+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:58:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:32 smithi134 ceph-mon[31966]: pgmap v9325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16269.837496 secs 2024-07-26T15:58:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:32 smithi189 ceph-mon[36790]: pgmap v9325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16269.837496 secs 2024-07-26T15:58:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:32 smithi193 ceph-mon[37473]: pgmap v9325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16269.837496 secs 2024-07-26T15:58:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:34 smithi134 ceph-mon[31966]: pgmap v9326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16255.414987 secs 2024-07-26T15:58:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:34 smithi189 ceph-mon[36790]: pgmap v9326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16255.414987 secs 2024-07-26T15:58:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:34 smithi193 ceph-mon[37473]: pgmap v9326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16255.414987 secs 2024-07-26T15:58:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:36 smithi134 ceph-mon[31966]: pgmap v9327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:36 smithi189 ceph-mon[36790]: pgmap v9327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:36 smithi193 ceph-mon[37473]: pgmap v9327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16274.837623 secs 2024-07-26T15:58:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16274.837623 secs 2024-07-26T15:58:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16274.837623 secs 2024-07-26T15:58:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:38 smithi134 ceph-mon[31966]: pgmap v9328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:38 smithi189 ceph-mon[36790]: pgmap v9328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:38 smithi193 ceph-mon[37473]: pgmap v9328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16260.415142 secs 2024-07-26T15:58:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16260.415142 secs 2024-07-26T15:58:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16260.415142 secs 2024-07-26T15:58:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:40 smithi134 ceph-mon[31966]: pgmap v9329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:40 smithi189 ceph-mon[36790]: pgmap v9329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:40 smithi193 ceph-mon[37473]: pgmap v9329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:58:41.440+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:58:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:58:41.440+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:58:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:58:41.440+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:58:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:42 smithi134 ceph-mon[31966]: pgmap v9330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16279.837745 secs 2024-07-26T15:58:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:42 smithi189 ceph-mon[36790]: pgmap v9330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16279.837745 secs 2024-07-26T15:58:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:42 smithi193 ceph-mon[37473]: pgmap v9330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16279.837745 secs 2024-07-26T15:58:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:44 smithi134 ceph-mon[31966]: pgmap v9331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16265.415265 secs 2024-07-26T15:58:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:44 smithi189 ceph-mon[36790]: pgmap v9331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16265.415265 secs 2024-07-26T15:58:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:44 smithi193 ceph-mon[37473]: pgmap v9331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16265.415265 secs 2024-07-26T15:58:45.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:58:45.823+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:58:45.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:58:45.823+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:58:45.825 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:58:45.823+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:58:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:46 smithi134 ceph-mon[31966]: pgmap v9332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:46 smithi189 ceph-mon[36790]: pgmap v9332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:46 smithi193 ceph-mon[37473]: pgmap v9332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16284.837869 secs 2024-07-26T15:58:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16284.837869 secs 2024-07-26T15:58:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16284.837869 secs 2024-07-26T15:58:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:48 smithi134 ceph-mon[31966]: pgmap v9333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:48 smithi189 ceph-mon[36790]: pgmap v9333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:48 smithi193 ceph-mon[37473]: pgmap v9333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16270.415386 secs 2024-07-26T15:58:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16270.415386 secs 2024-07-26T15:58:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16270.415386 secs 2024-07-26T15:58:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:50 smithi189 ceph-mon[36790]: pgmap v9334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:50 smithi193 ceph-mon[37473]: pgmap v9334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:50 smithi134 ceph-mon[31966]: pgmap v9334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:58:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:52 smithi189 ceph-mon[36790]: pgmap v9335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16289.837986 secs 2024-07-26T15:58:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:52 smithi193 ceph-mon[37473]: pgmap v9335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16289.837986 secs 2024-07-26T15:58:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:52 smithi134 ceph-mon[31966]: pgmap v9335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:58:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16289.837986 secs 2024-07-26T15:58:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:54 smithi189 ceph-mon[36790]: pgmap v9336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16275.415514 secs 2024-07-26T15:58:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:54 smithi193 ceph-mon[37473]: pgmap v9336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16275.415514 secs 2024-07-26T15:58:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:54 smithi134 ceph-mon[31966]: pgmap v9336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:58:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16275.415514 secs 2024-07-26T15:58:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:58:56.455+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:58:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:58:56.455+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:58:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:58:56.455+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:58:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:56 smithi193 ceph-mon[37473]: pgmap v9337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:56 smithi134 ceph-mon[31966]: pgmap v9337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:56 smithi189 ceph-mon[36790]: pgmap v9337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T15:58:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16294.838086 secs 2024-07-26T15:58:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16294.838086 secs 2024-07-26T15:58:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16294.838086 secs 2024-07-26T15:58:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:58 smithi193 ceph-mon[37473]: pgmap v9338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:58 smithi134 ceph-mon[31966]: pgmap v9338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:58 smithi189 ceph-mon[36790]: pgmap v9338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:58:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16280.415651 secs 2024-07-26T15:58:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:58:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:58:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16280.415651 secs 2024-07-26T15:58:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:58:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:58:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16280.415651 secs 2024-07-26T15:58:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:58:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T15:59:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:00 smithi193 ceph-mon[37473]: pgmap v9339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:59:00.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:59:00.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:59:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:00 smithi134 ceph-mon[31966]: pgmap v9339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:59:00.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:59:00.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:59:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:00 smithi189 ceph-mon[36790]: pgmap v9339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T15:59:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T15:59:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T15:59:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T15:59:00.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:59:00.838+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:59:00.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:59:00.838+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:59:00.840 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:59:00.838+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:59:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:01 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:59:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:01 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:59:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:01 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:59:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:01 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:59:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:01 smithi193 ceph-mon[37473]: pgmap v9340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:59:01.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:01 smithi193 ceph-mon[37473]: pgmap v9341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:01 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:59:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:01 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:59:01.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:01 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:59:01.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:01 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:59:01.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:01 smithi134 ceph-mon[31966]: pgmap v9340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:59:01.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:01 smithi134 ceph-mon[31966]: pgmap v9341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:01 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T15:59:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:01 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T15:59:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:01 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T15:59:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:01 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T15:59:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:01 smithi189 ceph-mon[36790]: pgmap v9340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T15:59:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:01 smithi189 ceph-mon[36790]: pgmap v9341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16299.838209 secs 2024-07-26T15:59:02.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16299.838209 secs 2024-07-26T15:59:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16299.838209 secs 2024-07-26T15:59:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:03 smithi193 ceph-mon[37473]: pgmap v9342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:59:03.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:03 smithi134 ceph-mon[31966]: pgmap v9342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:59:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:03 smithi189 ceph-mon[36790]: pgmap v9342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T15:59:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16285.415781 secs 2024-07-26T15:59:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16285.415781 secs 2024-07-26T15:59:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16285.415781 secs 2024-07-26T15:59:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:05 smithi193 ceph-mon[37473]: pgmap v9343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:05 smithi134 ceph-mon[31966]: pgmap v9343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:05 smithi189 ceph-mon[36790]: pgmap v9343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:07 smithi193 ceph-mon[37473]: pgmap v9344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:59:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16304.838355 secs 2024-07-26T15:59:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:07 smithi134 ceph-mon[31966]: pgmap v9344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:59:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16304.838355 secs 2024-07-26T15:59:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:07 smithi189 ceph-mon[36790]: pgmap v9344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T15:59:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16304.838355 secs 2024-07-26T15:59:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:09 smithi134 ceph-mon[31966]: pgmap v9345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:09.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16290.415926 secs 2024-07-26T15:59:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:09 smithi189 ceph-mon[36790]: pgmap v9345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16290.415926 secs 2024-07-26T15:59:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:09 smithi193 ceph-mon[37473]: pgmap v9345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T15:59:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16290.415926 secs 2024-07-26T15:59:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:59:11.464+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:59:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:59:11.464+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:59:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:59:11.464+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:59:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:11 smithi134 ceph-mon[31966]: pgmap v9346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:59:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:11 smithi189 ceph-mon[36790]: pgmap v9346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:59:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:11 smithi193 ceph-mon[37473]: pgmap v9346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T15:59:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16309.838479 secs 2024-07-26T15:59:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16309.838479 secs 2024-07-26T15:59:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16309.838479 secs 2024-07-26T15:59:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:13 smithi134 ceph-mon[31966]: pgmap v9347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:13 smithi189 ceph-mon[36790]: pgmap v9347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:13 smithi193 ceph-mon[37473]: pgmap v9347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16295.416071 secs 2024-07-26T15:59:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16295.416071 secs 2024-07-26T15:59:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16295.416071 secs 2024-07-26T15:59:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:15 smithi134 ceph-mon[31966]: pgmap v9348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:15 smithi189 ceph-mon[36790]: pgmap v9348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:15 smithi193 ceph-mon[37473]: pgmap v9348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:15.855 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:59:15.853+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:59:15.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:59:15.854+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:59:15.856 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:59:15.854+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:59:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:16 smithi134 ceph-mon[31966]: pgmap v9349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16314.838575 secs 2024-07-26T15:59:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:16 smithi189 ceph-mon[36790]: pgmap v9349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16314.838575 secs 2024-07-26T15:59:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:16 smithi193 ceph-mon[37473]: pgmap v9349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16314.838575 secs 2024-07-26T15:59:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:19 smithi189 ceph-mon[36790]: pgmap v9350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:59:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16300.416155 secs 2024-07-26T15:59:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:19 smithi193 ceph-mon[37473]: pgmap v9350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:59:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16300.416155 secs 2024-07-26T15:59:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:19 smithi134 ceph-mon[31966]: pgmap v9350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:59:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16300.416155 secs 2024-07-26T15:59:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:21 smithi193 ceph-mon[37473]: pgmap v9351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:21.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:21 smithi134 ceph-mon[31966]: pgmap v9351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:21 smithi189 ceph-mon[36790]: pgmap v9351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16319.838661 secs 2024-07-26T15:59:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16319.838661 secs 2024-07-26T15:59:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16319.838661 secs 2024-07-26T15:59:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:23 smithi193 ceph-mon[37473]: pgmap v9352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:23 smithi134 ceph-mon[31966]: pgmap v9352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:23 smithi189 ceph-mon[36790]: pgmap v9352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16305.416220 secs 2024-07-26T15:59:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16305.416220 secs 2024-07-26T15:59:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16305.416220 secs 2024-07-26T15:59:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:25 smithi193 ceph-mon[37473]: pgmap v9353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:25 smithi134 ceph-mon[31966]: pgmap v9353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:25 smithi189 ceph-mon[36790]: pgmap v9353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:26.477 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:59:26.475+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:59:26.477 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:59:26.476+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:59:26.478 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:59:26.476+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:59:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:27 smithi193 ceph-mon[37473]: pgmap v9354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16324.838759 secs 2024-07-26T15:59:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:27 smithi134 ceph-mon[31966]: pgmap v9354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16324.838759 secs 2024-07-26T15:59:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:27 smithi189 ceph-mon[36790]: pgmap v9354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16324.838759 secs 2024-07-26T15:59:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:29 smithi193 ceph-mon[37473]: pgmap v9355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:59:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16310.416341 secs 2024-07-26T15:59:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:29 smithi134 ceph-mon[31966]: pgmap v9355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:59:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16310.416341 secs 2024-07-26T15:59:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:29 smithi189 ceph-mon[36790]: pgmap v9355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:59:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16310.416341 secs 2024-07-26T15:59:30.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:59:30.867+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:59:30.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:59:30.867+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:59:30.868 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:59:30.867+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:59:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:31 smithi193 ceph-mon[37473]: pgmap v9356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:31 smithi134 ceph-mon[31966]: pgmap v9356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:31 smithi189 ceph-mon[36790]: pgmap v9356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16329.838887 secs 2024-07-26T15:59:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16329.838887 secs 2024-07-26T15:59:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16329.838887 secs 2024-07-26T15:59:33.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:33 smithi193 ceph-mon[37473]: pgmap v9357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:33 smithi134 ceph-mon[31966]: pgmap v9357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:33 smithi189 ceph-mon[36790]: pgmap v9357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16315.416485 secs 2024-07-26T15:59:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16315.416485 secs 2024-07-26T15:59:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16315.416485 secs 2024-07-26T15:59:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:35 smithi193 ceph-mon[37473]: pgmap v9358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:35 smithi134 ceph-mon[31966]: pgmap v9358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:35 smithi189 ceph-mon[36790]: pgmap v9358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:37 smithi134 ceph-mon[31966]: pgmap v9359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16334.838971 secs 2024-07-26T15:59:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:37 smithi189 ceph-mon[36790]: pgmap v9359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16334.838971 secs 2024-07-26T15:59:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:37 smithi193 ceph-mon[37473]: pgmap v9359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16334.838971 secs 2024-07-26T15:59:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:39 smithi134 ceph-mon[31966]: pgmap v9360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:59:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16320.416613 secs 2024-07-26T15:59:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:39 smithi189 ceph-mon[36790]: pgmap v9360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:59:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16320.416613 secs 2024-07-26T15:59:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:39 smithi193 ceph-mon[37473]: pgmap v9360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:59:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16320.416613 secs 2024-07-26T15:59:40.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:59:40.161+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T15:59:40.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:59:40.161+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T15:59:40.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:59:40.162+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 127 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T15:59:40.164 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:59:40.162+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T15:59:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:59:41.483+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:59:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:59:41.483+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:59:41.485 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:59:41.483+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:59:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:41 smithi134 ceph-mon[31966]: pgmap v9361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:59:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:41 smithi189 ceph-mon[36790]: pgmap v9361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:59:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:41 smithi193 ceph-mon[37473]: pgmap v9361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:59:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16339.839127 secs 2024-07-26T15:59:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16339.839127 secs 2024-07-26T15:59:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16339.839127 secs 2024-07-26T15:59:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:43 smithi134 ceph-mon[31966]: pgmap v9362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:59:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:43 smithi189 ceph-mon[36790]: pgmap v9362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:59:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:43 smithi193 ceph-mon[37473]: pgmap v9362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T15:59:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16325.416753 secs 2024-07-26T15:59:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16325.416753 secs 2024-07-26T15:59:44.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16325.416753 secs 2024-07-26T15:59:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:45 smithi134 ceph-mon[31966]: pgmap v9363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:59:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:45 smithi189 ceph-mon[36790]: pgmap v9363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:59:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:45 smithi193 ceph-mon[37473]: pgmap v9363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T15:59:45.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:59:45.877+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:59:45.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:59:45.877+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:59:45.879 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T15:59:45.877+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:59:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:47 smithi134 ceph-mon[31966]: pgmap v9364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:59:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16344.839243 secs 2024-07-26T15:59:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:47 smithi189 ceph-mon[36790]: pgmap v9364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:59:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16344.839243 secs 2024-07-26T15:59:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:47 smithi193 ceph-mon[37473]: pgmap v9364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T15:59:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16344.839243 secs 2024-07-26T15:59:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:48 smithi134 ceph-mon[31966]: pgmap v9365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:59:48.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:48 smithi189 ceph-mon[36790]: pgmap v9365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:59:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:48 smithi193 ceph-mon[37473]: pgmap v9365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T15:59:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16330.416878 secs 2024-07-26T15:59:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16330.416878 secs 2024-07-26T15:59:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16330.416878 secs 2024-07-26T15:59:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:50 smithi134 ceph-mon[31966]: pgmap v9366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:50 smithi189 ceph-mon[36790]: pgmap v9366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:59:50.714+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:59:50.715 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:59:50.714+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:59:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:59:50.714+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T15:59:50.716 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:59:50.714+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T15:59:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:50 smithi193 ceph-mon[37473]: pgmap v9366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16349.839369 secs 2024-07-26T15:59:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16349.839369 secs 2024-07-26T15:59:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16349.839369 secs 2024-07-26T15:59:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:52 smithi134 ceph-mon[31966]: pgmap v9367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:52 smithi189 ceph-mon[36790]: pgmap v9367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:52 smithi193 ceph-mon[37473]: pgmap v9367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T15:59:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16335.417002 secs 2024-07-26T15:59:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16335.417002 secs 2024-07-26T15:59:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16335.417002 secs 2024-07-26T15:59:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:54 smithi134 ceph-mon[31966]: pgmap v9368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:54 smithi189 ceph-mon[36790]: pgmap v9368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:54 smithi193 ceph-mon[37473]: pgmap v9368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T15:59:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:59:56.499+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T15:59:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:59:56.499+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T15:59:56.500 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T15:59:56.499+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T15:59:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:57 smithi193 ceph-mon[37473]: pgmap v9369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16354.839493 secs 2024-07-26T15:59:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:57 smithi134 ceph-mon[31966]: pgmap v9369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16354.839493 secs 2024-07-26T15:59:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:57 smithi189 ceph-mon[36790]: pgmap v9369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T15:59:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16354.839493 secs 2024-07-26T15:59:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:59 smithi193 ceph-mon[37473]: pgmap v9370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:59:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 15:59:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16340.417089 secs 2024-07-26T15:59:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:59 smithi134 ceph-mon[31966]: pgmap v9370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:59:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 15:59:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16340.417089 secs 2024-07-26T15:59:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:59 smithi189 ceph-mon[36790]: pgmap v9370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T15:59:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 15:59:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16340.417089 secs 2024-07-26T16:00:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T16:00:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T16:00:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T16:00:00.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:00:00.892+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:00:00.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:00:00.893+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:00:00.895 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:00:00.893+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:00:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:01 smithi193 ceph-mon[37473]: pgmap v9371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:00:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:01 smithi134 ceph-mon[31966]: pgmap v9371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:00:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:01 smithi189 ceph-mon[36790]: pgmap v9371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:00:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16359.839632 secs 2024-07-26T16:00:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:00:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:00:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:00:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16359.839632 secs 2024-07-26T16:00:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:00:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:00:02.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:00:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16359.839632 secs 2024-07-26T16:00:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:00:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:00:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:00:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:03 smithi193 ceph-mon[37473]: pgmap v9372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:00:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:03 smithi193 ceph-mon[37473]: pgmap v9373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:00:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:03 smithi134 ceph-mon[31966]: pgmap v9372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:00:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:03 smithi134 ceph-mon[31966]: pgmap v9373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:00:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:03 smithi189 ceph-mon[36790]: pgmap v9372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:00:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:03 smithi189 ceph-mon[36790]: pgmap v9373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:00:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16345.417171 secs 2024-07-26T16:00:04.573 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16345.417171 secs 2024-07-26T16:00:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16345.417171 secs 2024-07-26T16:00:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:05 smithi193 ceph-mon[37473]: pgmap v9374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:00:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:05 smithi134 ceph-mon[31966]: pgmap v9374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:00:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:05 smithi189 ceph-mon[36790]: pgmap v9374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:00:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:07 smithi193 ceph-mon[37473]: pgmap v9375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:00:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16364.839768 secs 2024-07-26T16:00:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:07 smithi134 ceph-mon[31966]: pgmap v9375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:00:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16364.839768 secs 2024-07-26T16:00:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:07 smithi189 ceph-mon[36790]: pgmap v9375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:00:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16364.839768 secs 2024-07-26T16:00:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:09 smithi134 ceph-mon[31966]: pgmap v9376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:00:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16350.417294 secs 2024-07-26T16:00:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:09 smithi189 ceph-mon[36790]: pgmap v9376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:00:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16350.417294 secs 2024-07-26T16:00:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:09 smithi193 ceph-mon[37473]: pgmap v9376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:00:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16350.417294 secs 2024-07-26T16:00:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:00:11.510+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:00:11.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:00:11.510+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:00:11.512 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:00:11.510+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:00:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:11 smithi134 ceph-mon[31966]: pgmap v9377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:00:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:11 smithi189 ceph-mon[36790]: pgmap v9377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:00:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:11 smithi193 ceph-mon[37473]: pgmap v9377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:00:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16369.839893 secs 2024-07-26T16:00:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16369.839893 secs 2024-07-26T16:00:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16369.839893 secs 2024-07-26T16:00:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:13 smithi134 ceph-mon[31966]: pgmap v9378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:00:13.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:13 smithi189 ceph-mon[36790]: pgmap v9378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:00:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:13 smithi193 ceph-mon[37473]: pgmap v9378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:00:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16355.417397 secs 2024-07-26T16:00:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16355.417397 secs 2024-07-26T16:00:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16355.417397 secs 2024-07-26T16:00:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:15 smithi134 ceph-mon[31966]: pgmap v9379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:15 smithi189 ceph-mon[36790]: pgmap v9379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:15 smithi193 ceph-mon[37473]: pgmap v9379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:15.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:00:15.905+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:00:15.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:00:15.905+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:00:15.907 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:00:15.905+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:00:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:17 smithi134 ceph-mon[31966]: pgmap v9380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16374.839983 secs 2024-07-26T16:00:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:17 smithi189 ceph-mon[36790]: pgmap v9380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16374.839983 secs 2024-07-26T16:00:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:17 smithi193 ceph-mon[37473]: pgmap v9380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16374.839983 secs 2024-07-26T16:00:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:18 smithi134 ceph-mon[31966]: pgmap v9381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:00:18.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:18 smithi189 ceph-mon[36790]: pgmap v9381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:00:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:18 smithi193 ceph-mon[37473]: pgmap v9381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:00:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16360.417519 secs 2024-07-26T16:00:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16360.417519 secs 2024-07-26T16:00:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16360.417519 secs 2024-07-26T16:00:20.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:20 smithi134 ceph-mon[31966]: pgmap v9382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:00:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:20 smithi189 ceph-mon[36790]: pgmap v9382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:00:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:20 smithi193 ceph-mon[37473]: pgmap v9382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:00:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16379.840102 secs 2024-07-26T16:00:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16379.840102 secs 2024-07-26T16:00:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16379.840102 secs 2024-07-26T16:00:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:22 smithi134 ceph-mon[31966]: pgmap v9383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:00:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:22 smithi189 ceph-mon[36790]: pgmap v9383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:00:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:22 smithi193 ceph-mon[37473]: pgmap v9383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:00:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:24 smithi134 ceph-mon[31966]: pgmap v9384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:00:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16365.417648 secs 2024-07-26T16:00:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:24 smithi189 ceph-mon[36790]: pgmap v9384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:00:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16365.417648 secs 2024-07-26T16:00:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:24 smithi193 ceph-mon[37473]: pgmap v9384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:00:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16365.417648 secs 2024-07-26T16:00:25.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:00:25.170+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T16:00:25.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:00:25.170+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T16:00:25.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:00:25.170+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 128 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T16:00:25.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:00:25.171+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T16:00:26.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:00:26.525+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:00:26.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:00:26.525+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:00:26.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:00:26.525+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:00:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:26 smithi189 ceph-mon[36790]: pgmap v9385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:00:26.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16384.840207 secs 2024-07-26T16:00:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:26 smithi193 ceph-mon[37473]: pgmap v9385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:00:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16384.840207 secs 2024-07-26T16:00:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:26 smithi134 ceph-mon[31966]: pgmap v9385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:00:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16384.840207 secs 2024-07-26T16:00:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:28 smithi189 ceph-mon[36790]: pgmap v9386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:00:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:28 smithi193 ceph-mon[37473]: pgmap v9386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:00:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:28 smithi134 ceph-mon[31966]: pgmap v9386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:00:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16370.417779 secs 2024-07-26T16:00:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16370.417779 secs 2024-07-26T16:00:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16370.417779 secs 2024-07-26T16:00:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:30 smithi193 ceph-mon[37473]: pgmap v9387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:00:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:30 smithi134 ceph-mon[31966]: pgmap v9387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:00:30.922 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:00:30.920+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:00:30.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:00:30.920+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:00:30.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:00:30.921+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:00:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:30 smithi189 ceph-mon[36790]: pgmap v9387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:00:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16389.840354 secs 2024-07-26T16:00:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16389.840354 secs 2024-07-26T16:00:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16389.840354 secs 2024-07-26T16:00:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:32 smithi193 ceph-mon[37473]: pgmap v9388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:32 smithi134 ceph-mon[31966]: pgmap v9388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:32 smithi189 ceph-mon[36790]: pgmap v9388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:34 smithi193 ceph-mon[37473]: pgmap v9389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16375.417925 secs 2024-07-26T16:00:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:34 smithi134 ceph-mon[31966]: pgmap v9389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16375.417925 secs 2024-07-26T16:00:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:34 smithi189 ceph-mon[36790]: pgmap v9389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16375.417925 secs 2024-07-26T16:00:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:00:35.722+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:00:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:00:35.722+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:00:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:00:35.723+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:00:35.724 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:00:35.723+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:00:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:36 smithi193 ceph-mon[37473]: pgmap v9390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16394.840423 secs 2024-07-26T16:00:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:36 smithi134 ceph-mon[31966]: pgmap v9390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16394.840423 secs 2024-07-26T16:00:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:36 smithi189 ceph-mon[36790]: pgmap v9390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16394.840423 secs 2024-07-26T16:00:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:38 smithi193 ceph-mon[37473]: pgmap v9391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:00:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:38 smithi134 ceph-mon[31966]: pgmap v9391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:00:38.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:38 smithi189 ceph-mon[36790]: pgmap v9391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:00:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16380.418009 secs 2024-07-26T16:00:39.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16380.418009 secs 2024-07-26T16:00:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16380.418009 secs 2024-07-26T16:00:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:40 smithi193 ceph-mon[37473]: pgmap v9392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:00:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:40 smithi134 ceph-mon[31966]: pgmap v9392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:00:40.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:40 smithi189 ceph-mon[36790]: pgmap v9392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:00:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:00:41.534+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:00:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:00:41.534+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:00:41.536 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:00:41.534+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:00:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16399.840544 secs 2024-07-26T16:00:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16399.840544 secs 2024-07-26T16:00:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16399.840544 secs 2024-07-26T16:00:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:42 smithi193 ceph-mon[37473]: pgmap v9393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:42 smithi134 ceph-mon[31966]: pgmap v9393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:42 smithi189 ceph-mon[36790]: pgmap v9393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:44 smithi193 ceph-mon[37473]: pgmap v9394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16385.418187 secs 2024-07-26T16:00:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:44 smithi134 ceph-mon[31966]: pgmap v9394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16385.418187 secs 2024-07-26T16:00:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:44 smithi189 ceph-mon[36790]: pgmap v9394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16385.418187 secs 2024-07-26T16:00:45.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:00:45.936+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:00:45.938 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:00:45.936+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:00:45.939 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:00:45.936+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:00:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:46 smithi134 ceph-mon[31966]: pgmap v9395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16404.840625 secs 2024-07-26T16:00:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:46 smithi189 ceph-mon[36790]: pgmap v9395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16404.840625 secs 2024-07-26T16:00:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:46 smithi193 ceph-mon[37473]: pgmap v9395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16404.840625 secs 2024-07-26T16:00:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:48 smithi134 ceph-mon[31966]: pgmap v9396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:00:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:48 smithi189 ceph-mon[36790]: pgmap v9396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:00:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:48 smithi193 ceph-mon[37473]: pgmap v9396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:00:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16390.418272 secs 2024-07-26T16:00:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16390.418272 secs 2024-07-26T16:00:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16390.418272 secs 2024-07-26T16:00:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:50 smithi134 ceph-mon[31966]: pgmap v9397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:00:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:50 smithi189 ceph-mon[36790]: pgmap v9397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:00:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:50 smithi193 ceph-mon[37473]: pgmap v9397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:00:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16409.840761 secs 2024-07-26T16:00:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16409.840761 secs 2024-07-26T16:00:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16409.840761 secs 2024-07-26T16:00:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:52 smithi134 ceph-mon[31966]: pgmap v9398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:52 smithi189 ceph-mon[36790]: pgmap v9398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:52 smithi193 ceph-mon[37473]: pgmap v9398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:54 smithi134 ceph-mon[31966]: pgmap v9399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:54.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16395.418360 secs 2024-07-26T16:00:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:54 smithi189 ceph-mon[36790]: pgmap v9399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16395.418360 secs 2024-07-26T16:00:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:54 smithi193 ceph-mon[37473]: pgmap v9399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:00:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16395.418360 secs 2024-07-26T16:00:56.538 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:00:56.537+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:00:56.539 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:00:56.537+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:00:56.539 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:00:56.537+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:00:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:56 smithi134 ceph-mon[31966]: pgmap v9400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16414.840843 secs 2024-07-26T16:00:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:56 smithi189 ceph-mon[36790]: pgmap v9400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16414.840843 secs 2024-07-26T16:00:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:56 smithi193 ceph-mon[37473]: pgmap v9400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:00:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16414.840843 secs 2024-07-26T16:00:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:58 smithi134 ceph-mon[31966]: pgmap v9401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:00:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:58 smithi189 ceph-mon[36790]: pgmap v9401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:00:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:58 smithi193 ceph-mon[37473]: pgmap v9401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:00:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:00:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16400.418452 secs 2024-07-26T16:00:59.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:00:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16400.418452 secs 2024-07-26T16:00:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:00:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16400.418452 secs 2024-07-26T16:01:00.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:00 smithi134 ceph-mon[31966]: pgmap v9402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:01:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:00 smithi189 ceph-mon[36790]: pgmap v9402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:01:00.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:01:00.952+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:01:00.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:01:00.952+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:01:00.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:01:00.952+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:01:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:00 smithi193 ceph-mon[37473]: pgmap v9402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:01:01.859 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16419.840923 secs 2024-07-26T16:01:01.859 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:01:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16419.840923 secs 2024-07-26T16:01:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:01:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16419.840923 secs 2024-07-26T16:01:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:01:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:03 smithi134 ceph-mon[31966]: pgmap v9403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:03.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:01:03.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:03 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:01:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:03 smithi189 ceph-mon[36790]: pgmap v9403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:01:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:03 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:01:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:03 smithi193 ceph-mon[37473]: pgmap v9403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:01:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:03 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:01:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:01:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:01:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:04 smithi134 ceph-mon[31966]: pgmap v9404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T16:01:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:04 smithi134 ceph-mon[31966]: pgmap v9405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:01:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:01:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16405.418604 secs 2024-07-26T16:01:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:01:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:01:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:04 smithi189 ceph-mon[36790]: pgmap v9404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T16:01:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:04 smithi189 ceph-mon[36790]: pgmap v9405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:01:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:01:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16405.418604 secs 2024-07-26T16:01:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:01:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:01:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:04 smithi193 ceph-mon[37473]: pgmap v9404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T16:01:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:04 smithi193 ceph-mon[37473]: pgmap v9405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:01:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:01:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16405.418604 secs 2024-07-26T16:01:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:06 smithi193 ceph-mon[37473]: pgmap v9406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:01:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:06 smithi134 ceph-mon[31966]: pgmap v9406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:01:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:06 smithi189 ceph-mon[36790]: pgmap v9406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:01:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16424.841004 secs 2024-07-26T16:01:07.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16424.841004 secs 2024-07-26T16:01:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16424.841004 secs 2024-07-26T16:01:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:08 smithi193 ceph-mon[37473]: pgmap v9407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:01:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:08 smithi134 ceph-mon[31966]: pgmap v9407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:01:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:08 smithi189 ceph-mon[36790]: pgmap v9407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:01:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16410.418750 secs 2024-07-26T16:01:09.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16410.418750 secs 2024-07-26T16:01:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16410.418750 secs 2024-07-26T16:01:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:10 smithi193 ceph-mon[37473]: pgmap v9408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:01:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:10 smithi134 ceph-mon[31966]: pgmap v9408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:01:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:10 smithi189 ceph-mon[36790]: pgmap v9408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:01:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:01:11.552+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:01:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:01:11.552+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:01:11.555 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:01:11.552+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:01:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:12 smithi193 ceph-mon[37473]: pgmap v9409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:01:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16429.841132 secs 2024-07-26T16:01:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:12 smithi134 ceph-mon[31966]: pgmap v9409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:01:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16429.841132 secs 2024-07-26T16:01:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:12 smithi189 ceph-mon[36790]: pgmap v9409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:01:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16429.841132 secs 2024-07-26T16:01:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:14 smithi193 ceph-mon[37473]: pgmap v9410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:01:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16415.418892 secs 2024-07-26T16:01:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:14 smithi134 ceph-mon[31966]: pgmap v9410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:01:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16415.418892 secs 2024-07-26T16:01:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:14 smithi189 ceph-mon[36790]: pgmap v9410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:01:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16415.418892 secs 2024-07-26T16:01:15.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:01:15.967+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:01:15.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:01:15.967+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:01:15.969 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:01:15.967+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:01:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:16 smithi193 ceph-mon[37473]: pgmap v9411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:16 smithi134 ceph-mon[31966]: pgmap v9411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:16 smithi189 ceph-mon[36790]: pgmap v9411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16434.841260 secs 2024-07-26T16:01:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16434.841260 secs 2024-07-26T16:01:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16434.841260 secs 2024-07-26T16:01:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:18 smithi193 ceph-mon[37473]: pgmap v9412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:18.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:18 smithi189 ceph-mon[36790]: pgmap v9412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:18 smithi134 ceph-mon[31966]: pgmap v9412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16420.419028 secs 2024-07-26T16:01:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16420.419028 secs 2024-07-26T16:01:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16420.419028 secs 2024-07-26T16:01:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:20 smithi134 ceph-mon[31966]: pgmap v9413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:20 smithi189 ceph-mon[36790]: pgmap v9413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:20 smithi193 ceph-mon[37473]: pgmap v9413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:22 smithi134 ceph-mon[31966]: pgmap v9414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:22.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16439.841402 secs 2024-07-26T16:01:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:22 smithi189 ceph-mon[36790]: pgmap v9414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16439.841402 secs 2024-07-26T16:01:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:22 smithi193 ceph-mon[37473]: pgmap v9414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16439.841402 secs 2024-07-26T16:01:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:24 smithi134 ceph-mon[31966]: pgmap v9415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16425.419157 secs 2024-07-26T16:01:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:24 smithi189 ceph-mon[36790]: pgmap v9415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16425.419157 secs 2024-07-26T16:01:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:24 smithi193 ceph-mon[37473]: pgmap v9415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16425.419157 secs 2024-07-26T16:01:26.566 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:01:26.565+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:01:26.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:01:26.565+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:01:26.567 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:01:26.565+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:01:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:26 smithi134 ceph-mon[31966]: pgmap v9416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:26 smithi189 ceph-mon[36790]: pgmap v9416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:26 smithi193 ceph-mon[37473]: pgmap v9416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16444.841523 secs 2024-07-26T16:01:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16444.841523 secs 2024-07-26T16:01:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16444.841523 secs 2024-07-26T16:01:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:28 smithi134 ceph-mon[31966]: pgmap v9417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:28.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:28 smithi189 ceph-mon[36790]: pgmap v9417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:28 smithi193 ceph-mon[37473]: pgmap v9417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16430.419294 secs 2024-07-26T16:01:29.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16430.419294 secs 2024-07-26T16:01:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16430.419294 secs 2024-07-26T16:01:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:30 smithi134 ceph-mon[31966]: pgmap v9418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:30 smithi189 ceph-mon[36790]: pgmap v9418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:30 smithi193 ceph-mon[37473]: pgmap v9418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:30.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:01:30.983+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:01:30.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:01:30.983+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:01:30.985 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:01:30.983+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:01:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16449.841640 secs 2024-07-26T16:01:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16449.841640 secs 2024-07-26T16:01:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16449.841640 secs 2024-07-26T16:01:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:32 smithi134 ceph-mon[31966]: pgmap v9419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:32 smithi189 ceph-mon[36790]: pgmap v9419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:32 smithi193 ceph-mon[37473]: pgmap v9419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:33 smithi134 ceph-mon[31966]: pgmap v9420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:33 smithi189 ceph-mon[36790]: pgmap v9420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:33 smithi193 ceph-mon[37473]: pgmap v9420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16435.419383 secs 2024-07-26T16:01:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16435.419383 secs 2024-07-26T16:01:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16435.419383 secs 2024-07-26T16:01:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:35 smithi134 ceph-mon[31966]: pgmap v9421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:35 smithi189 ceph-mon[36790]: pgmap v9421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:35 smithi193 ceph-mon[37473]: pgmap v9421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16454.841786 secs 2024-07-26T16:01:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16454.841786 secs 2024-07-26T16:01:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16454.841786 secs 2024-07-26T16:01:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:37 smithi134 ceph-mon[31966]: pgmap v9422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:37 smithi189 ceph-mon[36790]: pgmap v9422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:37 smithi193 ceph-mon[37473]: pgmap v9422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16440.419487 secs 2024-07-26T16:01:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16440.419487 secs 2024-07-26T16:01:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16440.419487 secs 2024-07-26T16:01:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:39 smithi134 ceph-mon[31966]: pgmap v9423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:39 smithi189 ceph-mon[36790]: pgmap v9423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:39 smithi193 ceph-mon[37473]: pgmap v9423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:01:41.571+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:01:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:01:41.571+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:01:41.573 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:01:41.571+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:01:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:42 smithi193 ceph-mon[37473]: pgmap v9424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16459.841924 secs 2024-07-26T16:01:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:42 smithi134 ceph-mon[31966]: pgmap v9424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16459.841924 secs 2024-07-26T16:01:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:42 smithi189 ceph-mon[36790]: pgmap v9424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16459.841924 secs 2024-07-26T16:01:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:44 smithi193 ceph-mon[37473]: pgmap v9425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16445.419615 secs 2024-07-26T16:01:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:44 smithi134 ceph-mon[31966]: pgmap v9425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16445.419615 secs 2024-07-26T16:01:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:44 smithi189 ceph-mon[36790]: pgmap v9425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16445.419615 secs 2024-07-26T16:01:46.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:01:45.998+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:01:46.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:01:45.998+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:01:46.001 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:01:45.998+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:01:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:46 smithi134 ceph-mon[31966]: pgmap v9426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:46 smithi193 ceph-mon[37473]: pgmap v9426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:46 smithi189 ceph-mon[36790]: pgmap v9426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16464.842041 secs 2024-07-26T16:01:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16464.842041 secs 2024-07-26T16:01:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16464.842041 secs 2024-07-26T16:01:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:48 smithi193 ceph-mon[37473]: pgmap v9427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:48.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:48 smithi189 ceph-mon[36790]: pgmap v9427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:48 smithi134 ceph-mon[31966]: pgmap v9427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16450.419751 secs 2024-07-26T16:01:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16450.419751 secs 2024-07-26T16:01:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16450.419751 secs 2024-07-26T16:01:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:50 smithi193 ceph-mon[37473]: pgmap v9428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:50 smithi134 ceph-mon[31966]: pgmap v9428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:50.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:50 smithi189 ceph-mon[36790]: pgmap v9428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:01:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:52 smithi134 ceph-mon[31966]: pgmap v9429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16469.842146 secs 2024-07-26T16:01:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:52 smithi189 ceph-mon[36790]: pgmap v9429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16469.842146 secs 2024-07-26T16:01:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:52 smithi193 ceph-mon[37473]: pgmap v9429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16469.842146 secs 2024-07-26T16:01:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:54 smithi134 ceph-mon[31966]: pgmap v9430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16455.419900 secs 2024-07-26T16:01:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:54 smithi189 ceph-mon[36790]: pgmap v9430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16455.419900 secs 2024-07-26T16:01:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:54 smithi193 ceph-mon[37473]: pgmap v9430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:01:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16455.419900 secs 2024-07-26T16:01:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:01:56.587+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:01:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:01:56.587+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:01:56.589 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:01:56.587+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:01:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:56 smithi134 ceph-mon[31966]: pgmap v9431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:56 smithi189 ceph-mon[36790]: pgmap v9431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:56 smithi193 ceph-mon[37473]: pgmap v9431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:01:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16474.842246 secs 2024-07-26T16:01:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16474.842246 secs 2024-07-26T16:01:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16474.842246 secs 2024-07-26T16:01:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:58 smithi134 ceph-mon[31966]: pgmap v9432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:58 smithi189 ceph-mon[36790]: pgmap v9432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:58 smithi193 ceph-mon[37473]: pgmap v9432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:01:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:01:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16460.420051 secs 2024-07-26T16:01:59.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:01:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16460.420051 secs 2024-07-26T16:01:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:01:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16460.420051 secs 2024-07-26T16:02:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:00 smithi134 ceph-mon[31966]: pgmap v9433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:00 smithi189 ceph-mon[36790]: pgmap v9433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:00 smithi193 ceph-mon[37473]: pgmap v9433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:01.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:02:01.010+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:02:01.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:02:01.010+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:02:01.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:02:01.010+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:02:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16479.842360 secs 2024-07-26T16:02:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:02 smithi134 ceph-mon[31966]: pgmap v9434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16479.842360 secs 2024-07-26T16:02:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:02 smithi189 ceph-mon[36790]: pgmap v9434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16479.842360 secs 2024-07-26T16:02:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:02 smithi193 ceph-mon[37473]: pgmap v9434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:04.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:04 smithi189 ceph-mon[36790]: pgmap v9435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:04.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:02:04.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16465.420115 secs 2024-07-26T16:02:04.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:04.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:04.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:04.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:04.623 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:04 smithi134 ceph-mon[31966]: pgmap v9435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:04.623 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:02:04.623 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16465.420115 secs 2024-07-26T16:02:04.623 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:04.624 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:04.624 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:04.624 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:04.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:04 smithi193 ceph-mon[37473]: pgmap v9435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:04.677 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:02:04.677 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16465.420115 secs 2024-07-26T16:02:04.677 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:04.677 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:04.677 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:04.677 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:06 smithi134 ceph-mon[31966]: pgmap v9436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:06.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:06.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:06.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:06.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:06 smithi189 ceph-mon[36790]: pgmap v9436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:06 smithi193 ceph-mon[37473]: pgmap v9436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:06.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:06.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:06.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16484.842425 secs 2024-07-26T16:02:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16484.842425 secs 2024-07-26T16:02:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16484.842425 secs 2024-07-26T16:02:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:08 smithi134 ceph-mon[31966]: pgmap v9437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:02:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:02:08.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:08 smithi189 ceph-mon[36790]: pgmap v9437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:02:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:02:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:08 smithi193 ceph-mon[37473]: pgmap v9437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:02:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:02:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:02:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:09 smithi134 ceph-mon[31966]: pgmap v9438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:09 smithi134 ceph-mon[31966]: pgmap v9439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:02:09.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16470.420223 secs 2024-07-26T16:02:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:09 smithi189 ceph-mon[36790]: pgmap v9438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:09 smithi189 ceph-mon[36790]: pgmap v9439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:02:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16470.420223 secs 2024-07-26T16:02:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:09 smithi193 ceph-mon[37473]: pgmap v9438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:09 smithi193 ceph-mon[37473]: pgmap v9439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:02:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16470.420223 secs 2024-07-26T16:02:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:11 smithi134 ceph-mon[31966]: pgmap v9440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:02:11.594 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:02:11.592+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:02:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:02:11.592+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:02:11.595 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:02:11.592+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:02:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:11 smithi189 ceph-mon[36790]: pgmap v9440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:02:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:11 smithi193 ceph-mon[37473]: pgmap v9440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:02:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16489.842492 secs 2024-07-26T16:02:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16489.842492 secs 2024-07-26T16:02:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16489.842492 secs 2024-07-26T16:02:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:13 smithi134 ceph-mon[31966]: pgmap v9441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T16:02:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:13 smithi189 ceph-mon[36790]: pgmap v9441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T16:02:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:13 smithi193 ceph-mon[37473]: pgmap v9441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T16:02:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16475.420354 secs 2024-07-26T16:02:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16475.420354 secs 2024-07-26T16:02:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16475.420354 secs 2024-07-26T16:02:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:15 smithi134 ceph-mon[31966]: pgmap v9442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T16:02:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:15 smithi189 ceph-mon[36790]: pgmap v9442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T16:02:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:15 smithi193 ceph-mon[37473]: pgmap v9442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T16:02:16.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:02:16.026+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:02:16.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:02:16.026+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:02:16.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:02:16.026+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:02:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:17 smithi134 ceph-mon[31966]: pgmap v9443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T16:02:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16494.842576 secs 2024-07-26T16:02:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:17 smithi189 ceph-mon[36790]: pgmap v9443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T16:02:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16494.842576 secs 2024-07-26T16:02:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:17 smithi193 ceph-mon[37473]: pgmap v9443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T16:02:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16494.842576 secs 2024-07-26T16:02:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:19 smithi134 ceph-mon[31966]: pgmap v9444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:02:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16480.420489 secs 2024-07-26T16:02:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:19 smithi189 ceph-mon[36790]: pgmap v9444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:02:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16480.420489 secs 2024-07-26T16:02:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:19 smithi193 ceph-mon[37473]: pgmap v9444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:02:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16480.420489 secs 2024-07-26T16:02:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:20 smithi134 ceph-mon[31966]: pgmap v9445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:20 smithi189 ceph-mon[36790]: pgmap v9445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:20 smithi193 ceph-mon[37473]: pgmap v9445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16499.842640 secs 2024-07-26T16:02:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16499.842640 secs 2024-07-26T16:02:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16499.842640 secs 2024-07-26T16:02:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:22 smithi134 ceph-mon[31966]: pgmap v9446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:02:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:22 smithi189 ceph-mon[36790]: pgmap v9446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:02:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:22 smithi193 ceph-mon[37473]: pgmap v9446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:02:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16485.420623 secs 2024-07-26T16:02:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16485.420623 secs 2024-07-26T16:02:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16485.420623 secs 2024-07-26T16:02:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:24 smithi134 ceph-mon[31966]: pgmap v9447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:24 smithi189 ceph-mon[36790]: pgmap v9447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:24 smithi193 ceph-mon[37473]: pgmap v9447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:02:26.608+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:02:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:02:26.608+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:02:26.610 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:02:26.608+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:02:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:27 smithi189 ceph-mon[36790]: pgmap v9448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16504.842733 secs 2024-07-26T16:02:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:27 smithi193 ceph-mon[37473]: pgmap v9448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16504.842733 secs 2024-07-26T16:02:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:27 smithi134 ceph-mon[31966]: pgmap v9448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16504.842733 secs 2024-07-26T16:02:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:29 smithi193 ceph-mon[37473]: pgmap v9449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:29.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16490.420746 secs 2024-07-26T16:02:29.676 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:29 smithi134 ceph-mon[31966]: pgmap v9449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:29.676 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16490.420746 secs 2024-07-26T16:02:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:29 smithi189 ceph-mon[36790]: pgmap v9449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16490.420746 secs 2024-07-26T16:02:31.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:02:31.041+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:02:31.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:02:31.041+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:02:31.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:02:31.041+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:02:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:31 smithi134 ceph-mon[31966]: pgmap v9450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:31 smithi193 ceph-mon[37473]: pgmap v9450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:31 smithi189 ceph-mon[36790]: pgmap v9450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16509.842801 secs 2024-07-26T16:02:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16509.842801 secs 2024-07-26T16:02:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16509.842801 secs 2024-07-26T16:02:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:33 smithi193 ceph-mon[37473]: pgmap v9451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:02:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:33 smithi134 ceph-mon[31966]: pgmap v9451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:02:33.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:33 smithi189 ceph-mon[36790]: pgmap v9451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:02:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16495.420869 secs 2024-07-26T16:02:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16495.420869 secs 2024-07-26T16:02:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16495.420869 secs 2024-07-26T16:02:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:35 smithi193 ceph-mon[37473]: pgmap v9452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:35 smithi134 ceph-mon[31966]: pgmap v9452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:35 smithi189 ceph-mon[36790]: pgmap v9452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:37 smithi193 ceph-mon[37473]: pgmap v9453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16514.842940 secs 2024-07-26T16:02:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:37 smithi134 ceph-mon[31966]: pgmap v9453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16514.842940 secs 2024-07-26T16:02:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:37 smithi189 ceph-mon[36790]: pgmap v9453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16514.842940 secs 2024-07-26T16:02:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:39 smithi193 ceph-mon[37473]: pgmap v9454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16500.420997 secs 2024-07-26T16:02:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:39 smithi134 ceph-mon[31966]: pgmap v9454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16500.420997 secs 2024-07-26T16:02:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:39 smithi189 ceph-mon[36790]: pgmap v9454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16500.420997 secs 2024-07-26T16:02:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:41 smithi193 ceph-mon[37473]: pgmap v9455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:41.625 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:02:41.624+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:02:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:02:41.624+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:02:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:02:41.624+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:02:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:41 smithi134 ceph-mon[31966]: pgmap v9455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:41 smithi189 ceph-mon[36790]: pgmap v9455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16519.843090 secs 2024-07-26T16:02:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16519.843090 secs 2024-07-26T16:02:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16519.843090 secs 2024-07-26T16:02:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:43 smithi193 ceph-mon[37473]: pgmap v9456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:43 smithi134 ceph-mon[31966]: pgmap v9456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:43 smithi189 ceph-mon[36790]: pgmap v9456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16505.421159 secs 2024-07-26T16:02:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16505.421159 secs 2024-07-26T16:02:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16505.421159 secs 2024-07-26T16:02:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:45 smithi134 ceph-mon[31966]: pgmap v9457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:45 smithi189 ceph-mon[36790]: pgmap v9457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:45 smithi193 ceph-mon[37473]: pgmap v9457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:46.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:02:46.057+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:02:46.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:02:46.057+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:02:46.059 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:02:46.057+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:02:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:47 smithi134 ceph-mon[31966]: pgmap v9458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16524.843235 secs 2024-07-26T16:02:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:47 smithi189 ceph-mon[36790]: pgmap v9458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16524.843235 secs 2024-07-26T16:02:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:47 smithi193 ceph-mon[37473]: pgmap v9458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16524.843235 secs 2024-07-26T16:02:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:49 smithi134 ceph-mon[31966]: pgmap v9459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16510.421289 secs 2024-07-26T16:02:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:49 smithi189 ceph-mon[36790]: pgmap v9459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16510.421289 secs 2024-07-26T16:02:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:49 smithi193 ceph-mon[37473]: pgmap v9459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16510.421289 secs 2024-07-26T16:02:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:51 smithi134 ceph-mon[31966]: pgmap v9460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:51 smithi189 ceph-mon[36790]: pgmap v9460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:51 smithi193 ceph-mon[37473]: pgmap v9460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16529.843352 secs 2024-07-26T16:02:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16529.843352 secs 2024-07-26T16:02:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16529.843352 secs 2024-07-26T16:02:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:53 smithi134 ceph-mon[31966]: pgmap v9461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:53 smithi189 ceph-mon[36790]: pgmap v9461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:53 smithi193 ceph-mon[37473]: pgmap v9461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:02:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16515.421427 secs 2024-07-26T16:02:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16515.421427 secs 2024-07-26T16:02:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16515.421427 secs 2024-07-26T16:02:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:55 smithi134 ceph-mon[31966]: pgmap v9462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:55 smithi189 ceph-mon[36790]: pgmap v9462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:55 smithi193 ceph-mon[37473]: pgmap v9462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:02:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:02:56.629+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:02:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:02:56.629+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:02:56.631 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:02:56.629+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:02:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:57 smithi134 ceph-mon[31966]: pgmap v9463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16534.843481 secs 2024-07-26T16:02:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:57 smithi189 ceph-mon[36790]: pgmap v9463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16534.843481 secs 2024-07-26T16:02:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:57 smithi193 ceph-mon[37473]: pgmap v9463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:02:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16534.843481 secs 2024-07-26T16:02:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:59 smithi134 ceph-mon[31966]: pgmap v9464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:59.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:02:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16520.421561 secs 2024-07-26T16:02:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:59 smithi189 ceph-mon[36790]: pgmap v9464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:02:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16520.421561 secs 2024-07-26T16:02:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:59 smithi193 ceph-mon[37473]: pgmap v9464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:02:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:02:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16520.421561 secs 2024-07-26T16:03:01.074 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:03:01.072+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:03:01.074 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:03:01.072+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:03:01.075 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:03:01.072+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:03:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:01 smithi134 ceph-mon[31966]: pgmap v9465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:01 smithi189 ceph-mon[36790]: pgmap v9465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:01 smithi193 ceph-mon[37473]: pgmap v9465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16539.843564 secs 2024-07-26T16:03:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16539.843564 secs 2024-07-26T16:03:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16539.843564 secs 2024-07-26T16:03:03.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:03 smithi134 ceph-mon[31966]: pgmap v9466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:03:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:03 smithi189 ceph-mon[36790]: pgmap v9466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:03:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:03 smithi193 ceph-mon[37473]: pgmap v9466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:03:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16525.421694 secs 2024-07-26T16:03:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16525.421694 secs 2024-07-26T16:03:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16525.421694 secs 2024-07-26T16:03:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:05 smithi134 ceph-mon[31966]: pgmap v9467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:05 smithi189 ceph-mon[36790]: pgmap v9467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:05 smithi193 ceph-mon[37473]: pgmap v9467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:03:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:06 smithi134 ceph-mon[31966]: pgmap v9468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16544.843631 secs 2024-07-26T16:03:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:06 smithi189 ceph-mon[36790]: pgmap v9468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16544.843631 secs 2024-07-26T16:03:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:06 smithi193 ceph-mon[37473]: pgmap v9468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16544.843631 secs 2024-07-26T16:03:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:09 smithi193 ceph-mon[37473]: pgmap v9469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:03:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16530.421793 secs 2024-07-26T16:03:09.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:09 smithi134 ceph-mon[31966]: pgmap v9469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:09.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:03:09.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16530.421793 secs 2024-07-26T16:03:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:09 smithi189 ceph-mon[36790]: pgmap v9469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:03:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16530.421793 secs 2024-07-26T16:03:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:03:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:03:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:10 smithi193 ceph-mon[37473]: pgmap v9470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:03:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:10 smithi193 ceph-mon[37473]: pgmap v9471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:03:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:03:10.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:03:10.575 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:03:10.575 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:10 smithi134 ceph-mon[31966]: pgmap v9470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:03:10.575 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:10 smithi134 ceph-mon[31966]: pgmap v9471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:03:10.575 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:03:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:03:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:03:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:10 smithi189 ceph-mon[36790]: pgmap v9470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:03:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:10 smithi189 ceph-mon[36790]: pgmap v9471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:03:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:03:11.646 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:03:11.644+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:03:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:03:11.645+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:03:11.647 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:03:11.645+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:03:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16549.843760 secs 2024-07-26T16:03:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:11 smithi134 ceph-mon[31966]: pgmap v9472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:03:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16549.843760 secs 2024-07-26T16:03:11.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:11 smithi189 ceph-mon[36790]: pgmap v9472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:03:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16549.843760 secs 2024-07-26T16:03:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:11 smithi193 ceph-mon[37473]: pgmap v9472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:03:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:13 smithi134 ceph-mon[31966]: pgmap v9473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:03:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:13 smithi189 ceph-mon[36790]: pgmap v9473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:03:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:13 smithi193 ceph-mon[37473]: pgmap v9473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:03:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16535.421921 secs 2024-07-26T16:03:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16535.421921 secs 2024-07-26T16:03:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16535.421921 secs 2024-07-26T16:03:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:15 smithi134 ceph-mon[31966]: pgmap v9474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:03:15.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:15 smithi189 ceph-mon[36790]: pgmap v9474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:03:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:15 smithi193 ceph-mon[37473]: pgmap v9474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:03:16.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:03:16.088+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:03:16.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:03:16.088+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:03:16.090 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:03:16.088+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:03:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16554.843885 secs 2024-07-26T16:03:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16554.843885 secs 2024-07-26T16:03:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16554.843885 secs 2024-07-26T16:03:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:17 smithi189 ceph-mon[36790]: pgmap v9475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:03:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:17 smithi193 ceph-mon[37473]: pgmap v9475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:03:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:17 smithi134 ceph-mon[31966]: pgmap v9475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:03:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:18 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16540.422053 secs 2024-07-26T16:03:19.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:18 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16540.422053 secs 2024-07-26T16:03:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:18 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16540.422053 secs 2024-07-26T16:03:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:19 smithi134 ceph-mon[31966]: pgmap v9476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:03:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:19 smithi189 ceph-mon[36790]: pgmap v9476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:03:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:19 smithi193 ceph-mon[37473]: pgmap v9476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:03:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16559.844038 secs 2024-07-26T16:03:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:21 smithi134 ceph-mon[31966]: pgmap v9477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16559.844038 secs 2024-07-26T16:03:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:21 smithi189 ceph-mon[36790]: pgmap v9477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16559.844038 secs 2024-07-26T16:03:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:21 smithi193 ceph-mon[37473]: pgmap v9477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:23 smithi134 ceph-mon[31966]: pgmap v9478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:23 smithi189 ceph-mon[36790]: pgmap v9478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:23 smithi193 ceph-mon[37473]: pgmap v9478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16545.422175 secs 2024-07-26T16:03:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16545.422175 secs 2024-07-26T16:03:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16545.422175 secs 2024-07-26T16:03:25.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:03:25.202+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T16:03:25.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:03:25.202+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T16:03:25.204 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:03:25.203+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T16:03:25.205 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:03:25.203+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 129 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T16:03:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:25 smithi134 ceph-mon[31966]: pgmap v9479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:25.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:25 smithi189 ceph-mon[36790]: pgmap v9479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:25 smithi193 ceph-mon[37473]: pgmap v9479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:03:26.660+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:03:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:03:26.660+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:03:26.662 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:03:26.660+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:03:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16564.844162 secs 2024-07-26T16:03:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16564.844162 secs 2024-07-26T16:03:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16564.844162 secs 2024-07-26T16:03:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:27 smithi189 ceph-mon[36790]: pgmap v9480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:03:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:27 smithi193 ceph-mon[37473]: pgmap v9480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:03:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:27 smithi134 ceph-mon[31966]: pgmap v9480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:03:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16550.422299 secs 2024-07-26T16:03:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16550.422299 secs 2024-07-26T16:03:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16550.422299 secs 2024-07-26T16:03:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:29 smithi134 ceph-mon[31966]: pgmap v9481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:03:30.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:29 smithi189 ceph-mon[36790]: pgmap v9481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:03:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:29 smithi193 ceph-mon[37473]: pgmap v9481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:03:31.105 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:03:31.103+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:03:31.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:03:31.103+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:03:31.106 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:03:31.103+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:03:31.675 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16569.844282 secs 2024-07-26T16:03:31.675 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:31 smithi134 ceph-mon[31966]: pgmap v9482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16569.844282 secs 2024-07-26T16:03:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:31 smithi189 ceph-mon[36790]: pgmap v9482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16569.844282 secs 2024-07-26T16:03:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:31 smithi193 ceph-mon[37473]: pgmap v9482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:33 smithi134 ceph-mon[31966]: pgmap v9483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:33 smithi189 ceph-mon[36790]: pgmap v9483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:33 smithi193 ceph-mon[37473]: pgmap v9483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16555.422401 secs 2024-07-26T16:03:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16555.422401 secs 2024-07-26T16:03:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16555.422401 secs 2024-07-26T16:03:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:35 smithi189 ceph-mon[36790]: pgmap v9484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:03:35.755+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:03:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:03:35.755+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:03:35.756 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:03:35.755+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:03:35.757 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:03:35.755+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:03:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:35 smithi193 ceph-mon[37473]: pgmap v9484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:35 smithi134 ceph-mon[31966]: pgmap v9484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16574.844379 secs 2024-07-26T16:03:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16574.844379 secs 2024-07-26T16:03:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16574.844379 secs 2024-07-26T16:03:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:37 smithi189 ceph-mon[36790]: pgmap v9485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:03:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:37 smithi193 ceph-mon[37473]: pgmap v9485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:03:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:37 smithi134 ceph-mon[31966]: pgmap v9485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:03:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16560.422528 secs 2024-07-26T16:03:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16560.422528 secs 2024-07-26T16:03:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16560.422528 secs 2024-07-26T16:03:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:39 smithi134 ceph-mon[31966]: pgmap v9486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:03:40.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:39 smithi189 ceph-mon[36790]: pgmap v9486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:03:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:39 smithi193 ceph-mon[37473]: pgmap v9486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:03:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16579.844493 secs 2024-07-26T16:03:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:41 smithi134 ceph-mon[31966]: pgmap v9487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:03:41.676+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:03:41.677 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:03:41.676+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:03:41.678 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:03:41.676+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:03:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16579.844493 secs 2024-07-26T16:03:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:41 smithi189 ceph-mon[36790]: pgmap v9487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16579.844493 secs 2024-07-26T16:03:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:41 smithi193 ceph-mon[37473]: pgmap v9487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:43 smithi134 ceph-mon[31966]: pgmap v9488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:43 smithi189 ceph-mon[36790]: pgmap v9488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:43 smithi193 ceph-mon[37473]: pgmap v9488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16565.422661 secs 2024-07-26T16:03:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16565.422661 secs 2024-07-26T16:03:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16565.422661 secs 2024-07-26T16:03:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:45 smithi189 ceph-mon[36790]: pgmap v9489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:45 smithi193 ceph-mon[37473]: pgmap v9489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:45 smithi134 ceph-mon[31966]: pgmap v9489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:46.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:03:46.119+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:03:46.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:03:46.119+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:03:46.121 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:03:46.119+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:03:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16584.844609 secs 2024-07-26T16:03:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16584.844609 secs 2024-07-26T16:03:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16584.844609 secs 2024-07-26T16:03:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:48 smithi193 ceph-mon[37473]: pgmap v9490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:03:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:48 smithi134 ceph-mon[31966]: pgmap v9490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:03:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:48 smithi189 ceph-mon[36790]: pgmap v9490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:03:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16570.422806 secs 2024-07-26T16:03:49.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16570.422806 secs 2024-07-26T16:03:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16570.422806 secs 2024-07-26T16:03:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:50 smithi134 ceph-mon[31966]: pgmap v9491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:03:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:50 smithi189 ceph-mon[36790]: pgmap v9491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:03:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:50 smithi193 ceph-mon[37473]: pgmap v9491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:03:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16589.844729 secs 2024-07-26T16:03:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:51 smithi134 ceph-mon[31966]: pgmap v9492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16589.844729 secs 2024-07-26T16:03:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:51 smithi189 ceph-mon[36790]: pgmap v9492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16589.844729 secs 2024-07-26T16:03:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:51 smithi193 ceph-mon[37473]: pgmap v9492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:53 smithi134 ceph-mon[31966]: pgmap v9493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:53 smithi189 ceph-mon[36790]: pgmap v9493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:53 smithi193 ceph-mon[37473]: pgmap v9493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:03:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16575.422969 secs 2024-07-26T16:03:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16575.422969 secs 2024-07-26T16:03:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16575.422969 secs 2024-07-26T16:03:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:55 smithi189 ceph-mon[36790]: pgmap v9494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:55 smithi193 ceph-mon[37473]: pgmap v9494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:55 smithi134 ceph-mon[31966]: pgmap v9494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:03:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:03:56.679+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:03:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:03:56.679+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:03:56.681 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:03:56.679+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:03:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16594.844840 secs 2024-07-26T16:03:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16594.844840 secs 2024-07-26T16:03:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16594.844840 secs 2024-07-26T16:03:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:57 smithi193 ceph-mon[37473]: pgmap v9495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:03:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:57 smithi134 ceph-mon[31966]: pgmap v9495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:03:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:57 smithi189 ceph-mon[36790]: pgmap v9495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:03:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16580.423096 secs 2024-07-26T16:03:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16580.423096 secs 2024-07-26T16:03:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16580.423096 secs 2024-07-26T16:04:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:03:59 smithi134 ceph-mon[31966]: pgmap v9496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:04:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:03:59 smithi189 ceph-mon[36790]: pgmap v9496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:04:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:03:59 smithi193 ceph-mon[37473]: pgmap v9496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:04:01.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:04:01.126+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:04:01.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:04:01.126+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:04:01.128 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:04:01.126+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:04:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16599.844916 secs 2024-07-26T16:04:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:01 smithi134 ceph-mon[31966]: pgmap v9497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16599.844916 secs 2024-07-26T16:04:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:01 smithi189 ceph-mon[36790]: pgmap v9497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16599.844916 secs 2024-07-26T16:04:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:01 smithi193 ceph-mon[37473]: pgmap v9497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:03 smithi189 ceph-mon[36790]: pgmap v9498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:03 smithi193 ceph-mon[37473]: pgmap v9498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:03 smithi134 ceph-mon[31966]: pgmap v9498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16585.423212 secs 2024-07-26T16:04:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16585.423212 secs 2024-07-26T16:04:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16585.423212 secs 2024-07-26T16:04:05.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:05 smithi189 ceph-mon[36790]: pgmap v9499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:05 smithi193 ceph-mon[37473]: pgmap v9499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:05 smithi134 ceph-mon[31966]: pgmap v9499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16604.845037 secs 2024-07-26T16:04:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16604.845037 secs 2024-07-26T16:04:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16604.845037 secs 2024-07-26T16:04:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:07 smithi193 ceph-mon[37473]: pgmap v9500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:04:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:07 smithi134 ceph-mon[31966]: pgmap v9500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:04:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:07 smithi189 ceph-mon[36790]: pgmap v9500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:04:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16590.423355 secs 2024-07-26T16:04:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16590.423355 secs 2024-07-26T16:04:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16590.423355 secs 2024-07-26T16:04:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:09 smithi134 ceph-mon[31966]: pgmap v9501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:04:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:04:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:09 smithi189 ceph-mon[36790]: pgmap v9501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:04:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:04:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:09 smithi193 ceph-mon[37473]: pgmap v9501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:04:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:04:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:04:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:04:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:04:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:04:11.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:04:11.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:04:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:04:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:04:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:04:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:04:11.695+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:04:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:04:11.695+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:04:11.697 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:04:11.695+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:04:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:11 smithi134 ceph-mon[31966]: pgmap v9502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:04:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:11 smithi134 ceph-mon[31966]: pgmap v9503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:04:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16609.845181 secs 2024-07-26T16:04:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:11 smithi189 ceph-mon[36790]: pgmap v9502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:04:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:11 smithi189 ceph-mon[36790]: pgmap v9503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:04:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16609.845181 secs 2024-07-26T16:04:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:11 smithi193 ceph-mon[37473]: pgmap v9502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:04:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:11 smithi193 ceph-mon[37473]: pgmap v9503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:04:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16609.845181 secs 2024-07-26T16:04:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:13 smithi134 ceph-mon[31966]: pgmap v9504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:04:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16595.423470 secs 2024-07-26T16:04:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:13 smithi189 ceph-mon[36790]: pgmap v9504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:04:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16595.423470 secs 2024-07-26T16:04:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:13 smithi193 ceph-mon[37473]: pgmap v9504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:04:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16595.423470 secs 2024-07-26T16:04:16.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:04:16.139+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:04:16.141 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:04:16.139+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:04:16.142 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:04:16.139+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:04:16.142 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:15 smithi134 ceph-mon[31966]: pgmap v9505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:04:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:15 smithi189 ceph-mon[36790]: pgmap v9505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:04:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:15 smithi193 ceph-mon[37473]: pgmap v9505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:04:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16614.845338 secs 2024-07-26T16:04:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16614.845338 secs 2024-07-26T16:04:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16614.845338 secs 2024-07-26T16:04:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:18 smithi193 ceph-mon[37473]: pgmap v9506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:18 smithi134 ceph-mon[31966]: pgmap v9506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:18 smithi189 ceph-mon[36790]: pgmap v9506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:19.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16600.423594 secs 2024-07-26T16:04:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16600.423594 secs 2024-07-26T16:04:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16600.423594 secs 2024-07-26T16:04:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:20 smithi134 ceph-mon[31966]: pgmap v9507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:04:20.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:20 smithi189 ceph-mon[36790]: pgmap v9507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:04:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:20 smithi193 ceph-mon[37473]: pgmap v9507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:04:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:22 smithi134 ceph-mon[31966]: pgmap v9508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:04:22.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16619.845484 secs 2024-07-26T16:04:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:22 smithi189 ceph-mon[36790]: pgmap v9508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:04:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16619.845484 secs 2024-07-26T16:04:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:22 smithi193 ceph-mon[37473]: pgmap v9508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:04:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16619.845484 secs 2024-07-26T16:04:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:24 smithi134 ceph-mon[31966]: pgmap v9509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16605.423710 secs 2024-07-26T16:04:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:24 smithi189 ceph-mon[36790]: pgmap v9509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16605.423710 secs 2024-07-26T16:04:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:24 smithi193 ceph-mon[37473]: pgmap v9509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16605.423710 secs 2024-07-26T16:04:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:26 smithi134 ceph-mon[31966]: pgmap v9510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:26 smithi189 ceph-mon[36790]: pgmap v9510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:26 smithi193 ceph-mon[37473]: pgmap v9510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:04:26.711+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:04:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:04:26.711+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:04:26.712 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:04:26.711+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:04:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16624.845609 secs 2024-07-26T16:04:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16624.845609 secs 2024-07-26T16:04:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16624.845609 secs 2024-07-26T16:04:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:28 smithi134 ceph-mon[31966]: pgmap v9511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:28.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:28 smithi189 ceph-mon[36790]: pgmap v9511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:28 smithi193 ceph-mon[37473]: pgmap v9511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16610.423831 secs 2024-07-26T16:04:29.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16610.423831 secs 2024-07-26T16:04:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16610.423831 secs 2024-07-26T16:04:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:30 smithi134 ceph-mon[31966]: pgmap v9512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:30 smithi189 ceph-mon[36790]: pgmap v9512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:30 smithi193 ceph-mon[37473]: pgmap v9512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:31.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:04:31.147+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:04:31.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:04:31.147+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:04:31.149 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:04:31.147+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:04:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:32 smithi134 ceph-mon[31966]: pgmap v9513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16629.845732 secs 2024-07-26T16:04:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:32 smithi189 ceph-mon[36790]: pgmap v9513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16629.845732 secs 2024-07-26T16:04:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:32 smithi193 ceph-mon[37473]: pgmap v9513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16629.845732 secs 2024-07-26T16:04:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:34 smithi134 ceph-mon[31966]: pgmap v9514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16615.423971 secs 2024-07-26T16:04:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:34 smithi189 ceph-mon[36790]: pgmap v9514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16615.423971 secs 2024-07-26T16:04:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:34 smithi193 ceph-mon[37473]: pgmap v9514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16615.423971 secs 2024-07-26T16:04:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:36 smithi134 ceph-mon[31966]: pgmap v9515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:36 smithi189 ceph-mon[36790]: pgmap v9515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:36 smithi193 ceph-mon[37473]: pgmap v9515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16634.845859 secs 2024-07-26T16:04:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16634.845859 secs 2024-07-26T16:04:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16634.845859 secs 2024-07-26T16:04:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:38 smithi134 ceph-mon[31966]: pgmap v9516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:38.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:38 smithi189 ceph-mon[36790]: pgmap v9516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:38 smithi193 ceph-mon[37473]: pgmap v9516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16620.424111 secs 2024-07-26T16:04:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16620.424111 secs 2024-07-26T16:04:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16620.424111 secs 2024-07-26T16:04:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:40 smithi134 ceph-mon[31966]: pgmap v9517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:40 smithi189 ceph-mon[36790]: pgmap v9517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:40 smithi193 ceph-mon[37473]: pgmap v9517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:04:41.718+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:04:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:04:41.718+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:04:41.720 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:04:41.718+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:04:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:42 smithi134 ceph-mon[31966]: pgmap v9518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16639.846018 secs 2024-07-26T16:04:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:42 smithi189 ceph-mon[36790]: pgmap v9518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16639.846018 secs 2024-07-26T16:04:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:42 smithi193 ceph-mon[37473]: pgmap v9518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16639.846018 secs 2024-07-26T16:04:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:44 smithi134 ceph-mon[31966]: pgmap v9519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16625.424236 secs 2024-07-26T16:04:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:44 smithi189 ceph-mon[36790]: pgmap v9519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16625.424236 secs 2024-07-26T16:04:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:44 smithi193 ceph-mon[37473]: pgmap v9519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16625.424236 secs 2024-07-26T16:04:46.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:04:46.151+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:04:46.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:04:46.151+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:04:46.154 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:04:46.151+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:04:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:46 smithi134 ceph-mon[31966]: pgmap v9520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:46 smithi189 ceph-mon[36790]: pgmap v9520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:46 smithi193 ceph-mon[37473]: pgmap v9520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16644.846121 secs 2024-07-26T16:04:47.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16644.846121 secs 2024-07-26T16:04:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16644.846121 secs 2024-07-26T16:04:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:48 smithi134 ceph-mon[31966]: pgmap v9521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:48 smithi189 ceph-mon[36790]: pgmap v9521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:48 smithi193 ceph-mon[37473]: pgmap v9521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16630.424344 secs 2024-07-26T16:04:49.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16630.424344 secs 2024-07-26T16:04:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16630.424344 secs 2024-07-26T16:04:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:50 smithi134 ceph-mon[31966]: pgmap v9522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:50.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:50 smithi189 ceph-mon[36790]: pgmap v9522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:50.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:50 smithi193 ceph-mon[37473]: pgmap v9522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:52 smithi189 ceph-mon[36790]: pgmap v9523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16649.846252 secs 2024-07-26T16:04:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:52 smithi193 ceph-mon[37473]: pgmap v9523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16649.846252 secs 2024-07-26T16:04:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:52 smithi134 ceph-mon[31966]: pgmap v9523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16649.846252 secs 2024-07-26T16:04:54.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:54 smithi189 ceph-mon[36790]: pgmap v9524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16635.424460 secs 2024-07-26T16:04:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:54 smithi193 ceph-mon[37473]: pgmap v9524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16635.424460 secs 2024-07-26T16:04:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:54 smithi134 ceph-mon[31966]: pgmap v9524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:04:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16635.424460 secs 2024-07-26T16:04:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:56 smithi189 ceph-mon[36790]: pgmap v9525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:56 smithi193 ceph-mon[37473]: pgmap v9525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:56 smithi134 ceph-mon[31966]: pgmap v9525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:04:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:04:56.734+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:04:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:04:56.734+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:04:56.736 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:04:56.734+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:04:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16654.846370 secs 2024-07-26T16:04:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16654.846370 secs 2024-07-26T16:04:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16654.846370 secs 2024-07-26T16:04:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:58 smithi193 ceph-mon[37473]: pgmap v9526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:58 smithi134 ceph-mon[31966]: pgmap v9526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:58 smithi189 ceph-mon[36790]: pgmap v9526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:04:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:04:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16640.424584 secs 2024-07-26T16:04:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:04:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16640.424584 secs 2024-07-26T16:04:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:04:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16640.424584 secs 2024-07-26T16:05:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:00 smithi134 ceph-mon[31966]: pgmap v9527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:00 smithi193 ceph-mon[37473]: pgmap v9527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:00 smithi189 ceph-mon[36790]: pgmap v9527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:01.157 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:05:01.155+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:05:01.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:05:01.155+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:05:01.158 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:05:01.155+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:05:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:02 smithi193 ceph-mon[37473]: pgmap v9528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16659.846493 secs 2024-07-26T16:05:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:02 smithi134 ceph-mon[31966]: pgmap v9528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16659.846493 secs 2024-07-26T16:05:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:02 smithi189 ceph-mon[36790]: pgmap v9528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16659.846493 secs 2024-07-26T16:05:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:04 smithi193 ceph-mon[37473]: pgmap v9529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16645.424712 secs 2024-07-26T16:05:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:04 smithi134 ceph-mon[31966]: pgmap v9529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16645.424712 secs 2024-07-26T16:05:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:04 smithi189 ceph-mon[36790]: pgmap v9529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16645.424712 secs 2024-07-26T16:05:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:06 smithi193 ceph-mon[37473]: pgmap v9530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:05:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:06 smithi134 ceph-mon[31966]: pgmap v9530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:05:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:06 smithi189 ceph-mon[36790]: pgmap v9530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:05:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16664.846577 secs 2024-07-26T16:05:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16664.846577 secs 2024-07-26T16:05:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16664.846577 secs 2024-07-26T16:05:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:08 smithi193 ceph-mon[37473]: pgmap v9531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:08 smithi134 ceph-mon[31966]: pgmap v9531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:08 smithi189 ceph-mon[36790]: pgmap v9531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16650.424839 secs 2024-07-26T16:05:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16650.424839 secs 2024-07-26T16:05:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16650.424839 secs 2024-07-26T16:05:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:10 smithi193 ceph-mon[37473]: pgmap v9532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:10.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:10 smithi134 ceph-mon[31966]: pgmap v9532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:10 smithi189 ceph-mon[36790]: pgmap v9532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:05:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:05:11.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:05:11.750 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:05:11.750+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:05:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:05:11.750+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:05:11.751 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:05:11.750+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:05:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:12 smithi134 ceph-mon[31966]: pgmap v9533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16669.846645 secs 2024-07-26T16:05:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:05:12.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:05:12.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:05:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:12 smithi189 ceph-mon[36790]: pgmap v9533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16669.846645 secs 2024-07-26T16:05:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:05:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:05:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:05:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:12 smithi193 ceph-mon[37473]: pgmap v9533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16669.846645 secs 2024-07-26T16:05:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:05:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:05:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:05:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:13 smithi134 ceph-mon[31966]: pgmap v9534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:05:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:13 smithi134 ceph-mon[31966]: pgmap v9535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:05:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:13 smithi189 ceph-mon[36790]: pgmap v9534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:05:13.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:13 smithi189 ceph-mon[36790]: pgmap v9535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:05:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:13 smithi193 ceph-mon[37473]: pgmap v9534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:05:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:13 smithi193 ceph-mon[37473]: pgmap v9535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:05:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16655.424987 secs 2024-07-26T16:05:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16655.424987 secs 2024-07-26T16:05:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16655.424987 secs 2024-07-26T16:05:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:15 smithi134 ceph-mon[31966]: pgmap v9536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:05:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:15 smithi189 ceph-mon[36790]: pgmap v9536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:05:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:15 smithi193 ceph-mon[37473]: pgmap v9536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:05:16.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:05:16.170+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:05:16.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:05:16.170+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:05:16.172 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:05:16.170+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:05:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:17 smithi134 ceph-mon[31966]: pgmap v9537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:05:17.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16674.846770 secs 2024-07-26T16:05:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:17 smithi189 ceph-mon[36790]: pgmap v9537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:05:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16674.846770 secs 2024-07-26T16:05:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:17 smithi193 ceph-mon[37473]: pgmap v9537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:05:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16674.846770 secs 2024-07-26T16:05:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:19 smithi134 ceph-mon[31966]: pgmap v9538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:05:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16660.425052 secs 2024-07-26T16:05:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:19 smithi189 ceph-mon[36790]: pgmap v9538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:05:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16660.425052 secs 2024-07-26T16:05:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:19 smithi193 ceph-mon[37473]: pgmap v9538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:05:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16660.425052 secs 2024-07-26T16:05:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:21 smithi134 ceph-mon[31966]: pgmap v9539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:05:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:21 smithi189 ceph-mon[36790]: pgmap v9539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:05:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:21 smithi193 ceph-mon[37473]: pgmap v9539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:05:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16679.846910 secs 2024-07-26T16:05:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16679.846910 secs 2024-07-26T16:05:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16679.846910 secs 2024-07-26T16:05:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:23 smithi134 ceph-mon[31966]: pgmap v9540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:05:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:23 smithi189 ceph-mon[36790]: pgmap v9540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:05:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:23 smithi193 ceph-mon[37473]: pgmap v9540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:05:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16665.425169 secs 2024-07-26T16:05:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16665.425169 secs 2024-07-26T16:05:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16665.425169 secs 2024-07-26T16:05:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:25 smithi134 ceph-mon[31966]: pgmap v9541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:25.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:25 smithi189 ceph-mon[36790]: pgmap v9541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:25 smithi193 ceph-mon[37473]: pgmap v9541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:05:26.756+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:05:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:05:26.757+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:05:26.759 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:05:26.757+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:05:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:27 smithi134 ceph-mon[31966]: pgmap v9542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16684.847030 secs 2024-07-26T16:05:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:27 smithi189 ceph-mon[36790]: pgmap v9542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16684.847030 secs 2024-07-26T16:05:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:27 smithi193 ceph-mon[37473]: pgmap v9542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16684.847030 secs 2024-07-26T16:05:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:29 smithi134 ceph-mon[31966]: pgmap v9543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16670.425305 secs 2024-07-26T16:05:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:29 smithi189 ceph-mon[36790]: pgmap v9543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16670.425305 secs 2024-07-26T16:05:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:29 smithi193 ceph-mon[37473]: pgmap v9543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16670.425305 secs 2024-07-26T16:05:31.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:05:31.174+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:05:31.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:05:31.174+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:05:31.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:05:31.174+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:05:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:31 smithi134 ceph-mon[31966]: pgmap v9544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:05:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:31 smithi189 ceph-mon[36790]: pgmap v9544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:05:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:31 smithi193 ceph-mon[37473]: pgmap v9544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:05:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16689.847164 secs 2024-07-26T16:05:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16689.847164 secs 2024-07-26T16:05:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16689.847164 secs 2024-07-26T16:05:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:33 smithi134 ceph-mon[31966]: pgmap v9545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:33 smithi189 ceph-mon[36790]: pgmap v9545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:33 smithi193 ceph-mon[37473]: pgmap v9545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16675.425467 secs 2024-07-26T16:05:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16675.425467 secs 2024-07-26T16:05:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16675.425467 secs 2024-07-26T16:05:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:35 smithi134 ceph-mon[31966]: pgmap v9546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:35 smithi189 ceph-mon[36790]: pgmap v9546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:35 smithi193 ceph-mon[37473]: pgmap v9546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:36 smithi134 ceph-mon[31966]: pgmap v9547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16694.847229 secs 2024-07-26T16:05:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:36 smithi189 ceph-mon[36790]: pgmap v9547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16694.847229 secs 2024-07-26T16:05:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:36 smithi193 ceph-mon[37473]: pgmap v9547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16694.847229 secs 2024-07-26T16:05:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:38 smithi134 ceph-mon[31966]: pgmap v9548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:38 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16680.425534 secs 2024-07-26T16:05:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:38 smithi189 ceph-mon[36790]: pgmap v9548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:38 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16680.425534 secs 2024-07-26T16:05:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:38 smithi193 ceph-mon[37473]: pgmap v9548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:38 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16680.425534 secs 2024-07-26T16:05:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:40 smithi134 ceph-mon[31966]: pgmap v9549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:05:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:40 smithi189 ceph-mon[36790]: pgmap v9549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:05:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:40 smithi193 ceph-mon[37473]: pgmap v9549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:05:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:05:41.765+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:05:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:05:41.765+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:05:41.767 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:05:41.765+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:05:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16699.847363 secs 2024-07-26T16:05:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16699.847363 secs 2024-07-26T16:05:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16699.847363 secs 2024-07-26T16:05:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:42 smithi134 ceph-mon[31966]: pgmap v9550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:42 smithi189 ceph-mon[36790]: pgmap v9550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:42 smithi193 ceph-mon[37473]: pgmap v9550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16685.425664 secs 2024-07-26T16:05:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16685.425664 secs 2024-07-26T16:05:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16685.425664 secs 2024-07-26T16:05:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:44 smithi134 ceph-mon[31966]: pgmap v9551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:44 smithi189 ceph-mon[36790]: pgmap v9551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:44 smithi193 ceph-mon[37473]: pgmap v9551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:46.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:05:46.190+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:05:46.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:05:46.190+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:05:46.192 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:05:46.190+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:05:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:46 smithi134 ceph-mon[31966]: pgmap v9552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16704.847451 secs 2024-07-26T16:05:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:46 smithi189 ceph-mon[36790]: pgmap v9552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16704.847451 secs 2024-07-26T16:05:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:46 smithi193 ceph-mon[37473]: pgmap v9552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16704.847451 secs 2024-07-26T16:05:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:48 smithi134 ceph-mon[31966]: pgmap v9553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16690.425764 secs 2024-07-26T16:05:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:48 smithi189 ceph-mon[36790]: pgmap v9553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16690.425764 secs 2024-07-26T16:05:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:48 smithi193 ceph-mon[37473]: pgmap v9553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16690.425764 secs 2024-07-26T16:05:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:50 smithi134 ceph-mon[31966]: pgmap v9554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:05:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:50 smithi189 ceph-mon[36790]: pgmap v9554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:05:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:50 smithi193 ceph-mon[37473]: pgmap v9554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:05:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16709.847555 secs 2024-07-26T16:05:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16709.847555 secs 2024-07-26T16:05:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16709.847555 secs 2024-07-26T16:05:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:52 smithi134 ceph-mon[31966]: pgmap v9555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:52 smithi189 ceph-mon[36790]: pgmap v9555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:52 smithi193 ceph-mon[37473]: pgmap v9555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16695.425917 secs 2024-07-26T16:05:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16695.425917 secs 2024-07-26T16:05:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16695.425917 secs 2024-07-26T16:05:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:54 smithi134 ceph-mon[31966]: pgmap v9556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:54 smithi189 ceph-mon[36790]: pgmap v9556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:54 smithi193 ceph-mon[37473]: pgmap v9556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:05:56.779+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:05:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:05:56.779+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:05:56.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:05:56.779+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:05:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:56 smithi134 ceph-mon[31966]: pgmap v9557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16714.847678 secs 2024-07-26T16:05:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:56 smithi189 ceph-mon[36790]: pgmap v9557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16714.847678 secs 2024-07-26T16:05:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:56 smithi193 ceph-mon[37473]: pgmap v9557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:05:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16714.847678 secs 2024-07-26T16:05:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:58 smithi189 ceph-mon[36790]: pgmap v9558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:05:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16700.426044 secs 2024-07-26T16:05:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:58 smithi193 ceph-mon[37473]: pgmap v9558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:05:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16700.426044 secs 2024-07-26T16:05:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:58 smithi134 ceph-mon[31966]: pgmap v9558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:05:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:05:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16700.426044 secs 2024-07-26T16:06:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:00 smithi189 ceph-mon[36790]: pgmap v9559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:06:01.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:06:01.205+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:06:01.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:06:01.205+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:06:01.207 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:06:01.205+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:06:01.208 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:00 smithi134 ceph-mon[31966]: pgmap v9559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:06:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:00 smithi193 ceph-mon[37473]: pgmap v9559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:06:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16719.847779 secs 2024-07-26T16:06:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16719.847779 secs 2024-07-26T16:06:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16719.847779 secs 2024-07-26T16:06:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:02 smithi193 ceph-mon[37473]: pgmap v9560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:02 smithi134 ceph-mon[31966]: pgmap v9560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:03.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:02 smithi189 ceph-mon[36790]: pgmap v9560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16705.426174 secs 2024-07-26T16:06:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16705.426174 secs 2024-07-26T16:06:04.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16705.426174 secs 2024-07-26T16:06:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:04 smithi193 ceph-mon[37473]: pgmap v9561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:06:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:04 smithi134 ceph-mon[31966]: pgmap v9561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:06:05.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:04 smithi189 ceph-mon[36790]: pgmap v9561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:06:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:06 smithi193 ceph-mon[37473]: pgmap v9562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:06:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16724.847847 secs 2024-07-26T16:06:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:06 smithi134 ceph-mon[31966]: pgmap v9562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:06:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16724.847847 secs 2024-07-26T16:06:07.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:06 smithi189 ceph-mon[36790]: pgmap v9562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:06:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16724.847847 secs 2024-07-26T16:06:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:08 smithi193 ceph-mon[37473]: pgmap v9563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:06:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16710.426307 secs 2024-07-26T16:06:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:08 smithi134 ceph-mon[31966]: pgmap v9563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:06:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16710.426307 secs 2024-07-26T16:06:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:08 smithi189 ceph-mon[36790]: pgmap v9563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:06:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16710.426307 secs 2024-07-26T16:06:11.227 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:10 smithi134 ceph-mon[31966]: pgmap v9564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:06:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:10 smithi193 ceph-mon[37473]: pgmap v9564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:06:11.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:10 smithi189 ceph-mon[36790]: pgmap v9564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-07-26T16:06:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:06:11.794+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:06:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:06:11.794+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:06:11.796 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:06:11.794+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:06:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16729.847978 secs 2024-07-26T16:06:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:06:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16729.847978 secs 2024-07-26T16:06:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:06:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16729.847978 secs 2024-07-26T16:06:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:06:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:12 smithi193 ceph-mon[37473]: pgmap v9565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:06:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:06:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:06:13.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:06:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:12 smithi134 ceph-mon[31966]: pgmap v9565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:06:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:06:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:06:13.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:06:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:12 smithi189 ceph-mon[36790]: pgmap v9565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:06:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:06:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:06:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:06:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:14 smithi193 ceph-mon[37473]: pgmap v9566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T16:06:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:14 smithi193 ceph-mon[37473]: pgmap v9567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:06:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16715.426416 secs 2024-07-26T16:06:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:14 smithi134 ceph-mon[31966]: pgmap v9566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T16:06:14.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:14 smithi134 ceph-mon[31966]: pgmap v9567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:06:14.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16715.426416 secs 2024-07-26T16:06:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:14 smithi189 ceph-mon[36790]: pgmap v9566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-07-26T16:06:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:14 smithi189 ceph-mon[36790]: pgmap v9567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:06:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16715.426416 secs 2024-07-26T16:06:16.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:06:16.220+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:06:16.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:06:16.221+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:06:16.223 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:06:16.221+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:06:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:16 smithi193 ceph-mon[37473]: pgmap v9568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:06:16.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:16 smithi134 ceph-mon[31966]: pgmap v9568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:06:16.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:16 smithi189 ceph-mon[36790]: pgmap v9568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:06:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16734.848126 secs 2024-07-26T16:06:17.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16734.848126 secs 2024-07-26T16:06:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16734.848126 secs 2024-07-26T16:06:18.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:18 smithi134 ceph-mon[31966]: pgmap v9569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:06:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:18 smithi189 ceph-mon[36790]: pgmap v9569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:06:18.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:18 smithi193 ceph-mon[37473]: pgmap v9569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:06:19.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16720.426541 secs 2024-07-26T16:06:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16720.426541 secs 2024-07-26T16:06:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16720.426541 secs 2024-07-26T16:06:20.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:20 smithi189 ceph-mon[36790]: pgmap v9570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:06:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:20 smithi134 ceph-mon[31966]: pgmap v9570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:06:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:20 smithi193 ceph-mon[37473]: pgmap v9570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:06:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:22 smithi134 ceph-mon[31966]: pgmap v9571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:06:22.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16739.848255 secs 2024-07-26T16:06:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:22 smithi189 ceph-mon[36790]: pgmap v9571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:06:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16739.848255 secs 2024-07-26T16:06:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:22 smithi193 ceph-mon[37473]: pgmap v9571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:06:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16739.848255 secs 2024-07-26T16:06:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:24 smithi134 ceph-mon[31966]: pgmap v9572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:06:24.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16725.426672 secs 2024-07-26T16:06:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:24 smithi189 ceph-mon[36790]: pgmap v9572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:06:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16725.426672 secs 2024-07-26T16:06:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:24 smithi193 ceph-mon[37473]: pgmap v9572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:06:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16725.426672 secs 2024-07-26T16:06:25.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:06:25.234+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T16:06:25.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:06:25.234+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T16:06:25.236 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:06:25.235+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 130 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T16:06:25.237 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:06:25.235+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T16:06:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:26 smithi134 ceph-mon[31966]: pgmap v9573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:26.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:26 smithi189 ceph-mon[36790]: pgmap v9573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:26 smithi193 ceph-mon[37473]: pgmap v9573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:06:26.800+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:06:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:06:26.800+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:06:26.802 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:06:26.800+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:06:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16744.848400 secs 2024-07-26T16:06:27.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16744.848400 secs 2024-07-26T16:06:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16744.848400 secs 2024-07-26T16:06:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:28 smithi134 ceph-mon[31966]: pgmap v9574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:28.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:28 smithi189 ceph-mon[36790]: pgmap v9574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:28 smithi193 ceph-mon[37473]: pgmap v9574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16730.426834 secs 2024-07-26T16:06:29.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16730.426834 secs 2024-07-26T16:06:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16730.426834 secs 2024-07-26T16:06:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:30 smithi134 ceph-mon[31966]: pgmap v9575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:30 smithi189 ceph-mon[36790]: pgmap v9575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:30 smithi193 ceph-mon[37473]: pgmap v9575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:31.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:06:31.236+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:06:31.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:06:31.236+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:06:31.238 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:06:31.236+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:06:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:32 smithi134 ceph-mon[31966]: pgmap v9576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16749.848539 secs 2024-07-26T16:06:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:32 smithi189 ceph-mon[36790]: pgmap v9576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16749.848539 secs 2024-07-26T16:06:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:32 smithi193 ceph-mon[37473]: pgmap v9576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16749.848539 secs 2024-07-26T16:06:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:34 smithi134 ceph-mon[31966]: pgmap v9577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16735.426987 secs 2024-07-26T16:06:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:34 smithi189 ceph-mon[36790]: pgmap v9577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16735.426987 secs 2024-07-26T16:06:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:34 smithi193 ceph-mon[37473]: pgmap v9577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16735.426987 secs 2024-07-26T16:06:35.787 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:06:35.786+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:06:35.944 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:06:35.786+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:06:35.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:06:35.786+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:06:35.945 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:06:35.786+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:06:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:36 smithi134 ceph-mon[31966]: pgmap v9578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:36.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:36 smithi189 ceph-mon[36790]: pgmap v9578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:36 smithi193 ceph-mon[37473]: pgmap v9578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16754.848672 secs 2024-07-26T16:06:37.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16754.848672 secs 2024-07-26T16:06:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16754.848672 secs 2024-07-26T16:06:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:38 smithi134 ceph-mon[31966]: pgmap v9579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:06:38.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:38 smithi189 ceph-mon[36790]: pgmap v9579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:06:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:38 smithi193 ceph-mon[37473]: pgmap v9579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:06:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16740.427105 secs 2024-07-26T16:06:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16740.427105 secs 2024-07-26T16:06:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16740.427105 secs 2024-07-26T16:06:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:40 smithi134 ceph-mon[31966]: pgmap v9580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:40 smithi189 ceph-mon[36790]: pgmap v9580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:40 smithi193 ceph-mon[37473]: pgmap v9580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:41.817 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:06:41.815+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:06:41.818 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:06:41.816+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:06:41.818 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:06:41.816+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:06:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:42 smithi134 ceph-mon[31966]: pgmap v9581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16759.848817 secs 2024-07-26T16:06:42.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:42 smithi189 ceph-mon[36790]: pgmap v9581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16759.848817 secs 2024-07-26T16:06:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:42 smithi193 ceph-mon[37473]: pgmap v9581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16759.848817 secs 2024-07-26T16:06:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:44 smithi134 ceph-mon[31966]: pgmap v9582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16745.427231 secs 2024-07-26T16:06:44.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:44 smithi189 ceph-mon[36790]: pgmap v9582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16745.427231 secs 2024-07-26T16:06:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:44 smithi193 ceph-mon[37473]: pgmap v9582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16745.427231 secs 2024-07-26T16:06:46.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:06:46.252+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:06:46.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:06:46.252+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:06:46.254 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:06:46.252+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:06:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:46 smithi134 ceph-mon[31966]: pgmap v9583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:06:46.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:46 smithi189 ceph-mon[36790]: pgmap v9583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:06:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:46 smithi193 ceph-mon[37473]: pgmap v9583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:06:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16764.848920 secs 2024-07-26T16:06:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16764.848920 secs 2024-07-26T16:06:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16764.848920 secs 2024-07-26T16:06:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:48 smithi134 ceph-mon[31966]: pgmap v9584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:06:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:48 smithi189 ceph-mon[36790]: pgmap v9584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:06:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:48 smithi193 ceph-mon[37473]: pgmap v9584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:06:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16750.427363 secs 2024-07-26T16:06:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16750.427363 secs 2024-07-26T16:06:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16750.427363 secs 2024-07-26T16:06:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:50 smithi189 ceph-mon[36790]: pgmap v9585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:50 smithi193 ceph-mon[37473]: pgmap v9585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:50 smithi134 ceph-mon[31966]: pgmap v9585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:52 smithi189 ceph-mon[36790]: pgmap v9586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16769.849006 secs 2024-07-26T16:06:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:52 smithi193 ceph-mon[37473]: pgmap v9586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16769.849006 secs 2024-07-26T16:06:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:52 smithi134 ceph-mon[31966]: pgmap v9586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16769.849006 secs 2024-07-26T16:06:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:54 smithi193 ceph-mon[37473]: pgmap v9587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16755.427496 secs 2024-07-26T16:06:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:54 smithi134 ceph-mon[31966]: pgmap v9587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16755.427496 secs 2024-07-26T16:06:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:54 smithi189 ceph-mon[36790]: pgmap v9587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:06:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16755.427496 secs 2024-07-26T16:06:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:56 smithi193 ceph-mon[37473]: pgmap v9588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:56 smithi134 ceph-mon[31966]: pgmap v9588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:56 smithi189 ceph-mon[36790]: pgmap v9588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:06:56.819+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:06:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:06:56.819+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:06:56.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:06:56.819+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:06:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16774.849127 secs 2024-07-26T16:06:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16774.849127 secs 2024-07-26T16:06:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16774.849127 secs 2024-07-26T16:06:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:58 smithi193 ceph-mon[37473]: pgmap v9589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:58 smithi134 ceph-mon[31966]: pgmap v9589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:58 smithi189 ceph-mon[36790]: pgmap v9589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:06:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:06:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16760.427594 secs 2024-07-26T16:06:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:06:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16760.427594 secs 2024-07-26T16:06:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:06:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16760.427594 secs 2024-07-26T16:07:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:00 smithi193 ceph-mon[37473]: pgmap v9590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:00 smithi134 ceph-mon[31966]: pgmap v9590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:00 smithi189 ceph-mon[36790]: pgmap v9590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:01.269 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:07:01.267+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:07:01.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:07:01.267+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:07:01.270 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:07:01.267+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:07:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:02 smithi193 ceph-mon[37473]: pgmap v9591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16779.849248 secs 2024-07-26T16:07:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:02 smithi134 ceph-mon[31966]: pgmap v9591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16779.849248 secs 2024-07-26T16:07:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:02 smithi189 ceph-mon[36790]: pgmap v9591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16779.849248 secs 2024-07-26T16:07:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:04 smithi193 ceph-mon[37473]: pgmap v9592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16765.427722 secs 2024-07-26T16:07:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:04 smithi134 ceph-mon[31966]: pgmap v9592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16765.427722 secs 2024-07-26T16:07:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:04 smithi189 ceph-mon[36790]: pgmap v9592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16765.427722 secs 2024-07-26T16:07:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:06 smithi193 ceph-mon[37473]: pgmap v9593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:07:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:06 smithi134 ceph-mon[31966]: pgmap v9593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:07:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:06 smithi189 ceph-mon[36790]: pgmap v9593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:07:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16784.849369 secs 2024-07-26T16:07:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16784.849369 secs 2024-07-26T16:07:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16784.849369 secs 2024-07-26T16:07:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:08 smithi134 ceph-mon[31966]: pgmap v9594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:07:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:08 smithi189 ceph-mon[36790]: pgmap v9594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:07:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:08 smithi193 ceph-mon[37473]: pgmap v9594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:07:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16770.427866 secs 2024-07-26T16:07:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16770.427866 secs 2024-07-26T16:07:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16770.427866 secs 2024-07-26T16:07:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:10 smithi134 ceph-mon[31966]: pgmap v9595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:10.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:10 smithi189 ceph-mon[36790]: pgmap v9595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:10 smithi193 ceph-mon[37473]: pgmap v9595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:11.836 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:07:11.835+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:07:11.837 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:07:11.835+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:07:11.837 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:07:11.835+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:07:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:12 smithi134 ceph-mon[31966]: pgmap v9596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16789.849498 secs 2024-07-26T16:07:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:12 smithi189 ceph-mon[36790]: pgmap v9596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16789.849498 secs 2024-07-26T16:07:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:12 smithi193 ceph-mon[37473]: pgmap v9596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16789.849498 secs 2024-07-26T16:07:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:07:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:07:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:07:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:14 smithi134 ceph-mon[31966]: pgmap v9597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16775.428031 secs 2024-07-26T16:07:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:07:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:07:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:07:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:14 smithi189 ceph-mon[36790]: pgmap v9597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16775.428031 secs 2024-07-26T16:07:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:07:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:07:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:07:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:14 smithi193 ceph-mon[37473]: pgmap v9597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16775.428031 secs 2024-07-26T16:07:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:07:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:07:14.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:07:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:15 smithi134 ceph-mon[31966]: pgmap v9598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:07:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:15 smithi134 ceph-mon[31966]: pgmap v9599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:15 smithi189 ceph-mon[36790]: pgmap v9598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:07:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:15 smithi189 ceph-mon[36790]: pgmap v9599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:15 smithi193 ceph-mon[37473]: pgmap v9598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:07:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:15 smithi193 ceph-mon[37473]: pgmap v9599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:16.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:07:16.280+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:07:16.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:07:16.280+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:07:16.282 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:07:16.280+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:07:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:17 smithi134 ceph-mon[31966]: pgmap v9600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16794.849613 secs 2024-07-26T16:07:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:17 smithi189 ceph-mon[36790]: pgmap v9600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16794.849613 secs 2024-07-26T16:07:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:17 smithi193 ceph-mon[37473]: pgmap v9600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16794.849613 secs 2024-07-26T16:07:19.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:19 smithi134 ceph-mon[31966]: pgmap v9601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:19.675 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16780.428166 secs 2024-07-26T16:07:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:19 smithi189 ceph-mon[36790]: pgmap v9601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16780.428166 secs 2024-07-26T16:07:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:19 smithi193 ceph-mon[37473]: pgmap v9601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16780.428166 secs 2024-07-26T16:07:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:21 smithi134 ceph-mon[31966]: pgmap v9602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:21 smithi189 ceph-mon[36790]: pgmap v9602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:21 smithi193 ceph-mon[37473]: pgmap v9602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:07:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16799.849736 secs 2024-07-26T16:07:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16799.849736 secs 2024-07-26T16:07:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16799.849736 secs 2024-07-26T16:07:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:23 smithi134 ceph-mon[31966]: pgmap v9603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T16:07:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:23 smithi189 ceph-mon[36790]: pgmap v9603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T16:07:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:23 smithi193 ceph-mon[37473]: pgmap v9603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T16:07:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16785.428268 secs 2024-07-26T16:07:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16785.428268 secs 2024-07-26T16:07:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16785.428268 secs 2024-07-26T16:07:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:25 smithi134 ceph-mon[31966]: pgmap v9604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:07:25.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:25 smithi189 ceph-mon[36790]: pgmap v9604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:07:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:25 smithi193 ceph-mon[37473]: pgmap v9604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:07:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:07:26.850+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:07:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:07:26.850+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:07:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:07:26.850+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:07:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:27 smithi134 ceph-mon[31966]: pgmap v9605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16804.849850 secs 2024-07-26T16:07:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:27 smithi189 ceph-mon[36790]: pgmap v9605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16804.849850 secs 2024-07-26T16:07:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:27 smithi193 ceph-mon[37473]: pgmap v9605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16804.849850 secs 2024-07-26T16:07:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:29 smithi134 ceph-mon[31966]: pgmap v9606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16790.428336 secs 2024-07-26T16:07:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:29 smithi189 ceph-mon[36790]: pgmap v9606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16790.428336 secs 2024-07-26T16:07:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:29 smithi193 ceph-mon[37473]: pgmap v9606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16790.428336 secs 2024-07-26T16:07:31.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:07:31.289+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:07:31.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:07:31.289+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:07:31.291 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:07:31.289+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:07:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:31 smithi134 ceph-mon[31966]: pgmap v9607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16809.849950 secs 2024-07-26T16:07:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:31 smithi189 ceph-mon[36790]: pgmap v9607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16809.849950 secs 2024-07-26T16:07:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:31 smithi193 ceph-mon[37473]: pgmap v9607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16809.849950 secs 2024-07-26T16:07:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:33 smithi134 ceph-mon[31966]: pgmap v9608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:33 smithi189 ceph-mon[36790]: pgmap v9608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:33 smithi193 ceph-mon[37473]: pgmap v9608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16795.428418 secs 2024-07-26T16:07:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16795.428418 secs 2024-07-26T16:07:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16795.428418 secs 2024-07-26T16:07:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:35 smithi134 ceph-mon[31966]: pgmap v9609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:35.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:35 smithi189 ceph-mon[36790]: pgmap v9609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:35 smithi193 ceph-mon[37473]: pgmap v9609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16814.850049 secs 2024-07-26T16:07:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16814.850049 secs 2024-07-26T16:07:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16814.850049 secs 2024-07-26T16:07:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:37 smithi134 ceph-mon[31966]: pgmap v9610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:37 smithi189 ceph-mon[36790]: pgmap v9610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:37 smithi193 ceph-mon[37473]: pgmap v9610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:38 smithi189 ceph-mon[36790]: pgmap v9611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:38 smithi193 ceph-mon[37473]: pgmap v9611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:38 smithi134 ceph-mon[31966]: pgmap v9611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16800.428539 secs 2024-07-26T16:07:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16800.428539 secs 2024-07-26T16:07:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16800.428539 secs 2024-07-26T16:07:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:40 smithi193 ceph-mon[37473]: pgmap v9612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:40 smithi134 ceph-mon[31966]: pgmap v9612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:40 smithi189 ceph-mon[36790]: pgmap v9612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16819.850196 secs 2024-07-26T16:07:41.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:07:41.866+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:07:41.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:07:41.866+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:07:41.868 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:07:41.866+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:07:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16819.850196 secs 2024-07-26T16:07:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16819.850196 secs 2024-07-26T16:07:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:42 smithi193 ceph-mon[37473]: pgmap v9613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:42.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:42 smithi134 ceph-mon[31966]: pgmap v9613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:42 smithi189 ceph-mon[36790]: pgmap v9613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16805.428649 secs 2024-07-26T16:07:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16805.428649 secs 2024-07-26T16:07:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16805.428649 secs 2024-07-26T16:07:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:44 smithi134 ceph-mon[31966]: pgmap v9614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:44 smithi189 ceph-mon[36790]: pgmap v9614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:44 smithi193 ceph-mon[37473]: pgmap v9614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:46.303 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:07:46.301+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:07:46.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:07:46.301+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:07:46.304 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:07:46.301+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:07:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:47 smithi193 ceph-mon[37473]: pgmap v9615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16824.850327 secs 2024-07-26T16:07:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:47 smithi134 ceph-mon[31966]: pgmap v9615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16824.850327 secs 2024-07-26T16:07:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:47 smithi189 ceph-mon[36790]: pgmap v9615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16824.850327 secs 2024-07-26T16:07:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:49 smithi193 ceph-mon[37473]: pgmap v9616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16810.428772 secs 2024-07-26T16:07:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:49 smithi134 ceph-mon[31966]: pgmap v9616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16810.428772 secs 2024-07-26T16:07:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:49 smithi189 ceph-mon[36790]: pgmap v9616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16810.428772 secs 2024-07-26T16:07:51.496 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:51 smithi189 ceph-mon[36790]: pgmap v9617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:51 smithi193 ceph-mon[37473]: pgmap v9617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:51 smithi134 ceph-mon[31966]: pgmap v9617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16829.850484 secs 2024-07-26T16:07:52.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16829.850484 secs 2024-07-26T16:07:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16829.850484 secs 2024-07-26T16:07:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:53 smithi193 ceph-mon[37473]: pgmap v9618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:53 smithi134 ceph-mon[31966]: pgmap v9618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:53 smithi189 ceph-mon[36790]: pgmap v9618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:07:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16815.428916 secs 2024-07-26T16:07:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16815.428916 secs 2024-07-26T16:07:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16815.428916 secs 2024-07-26T16:07:55.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:07:55.250+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T16:07:55.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:07:55.250+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T16:07:55.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:07:55.250+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T16:07:55.252 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:07:55.251+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 131 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T16:07:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:55 smithi193 ceph-mon[37473]: pgmap v9619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:55 smithi134 ceph-mon[31966]: pgmap v9619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:55 smithi189 ceph-mon[36790]: pgmap v9619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:07:56.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:07:56.881+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:07:56.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:07:56.881+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:07:56.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:07:56.881+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:07:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:57 smithi134 ceph-mon[31966]: pgmap v9620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16834.850597 secs 2024-07-26T16:07:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:57 smithi189 ceph-mon[36790]: pgmap v9620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16834.850597 secs 2024-07-26T16:07:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:57 smithi193 ceph-mon[37473]: pgmap v9620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:07:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16834.850597 secs 2024-07-26T16:07:59.576 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:59 smithi193 ceph-mon[37473]: pgmap v9621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:59.577 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:07:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16820.429040 secs 2024-07-26T16:07:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:59 smithi134 ceph-mon[31966]: pgmap v9621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:07:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16820.429040 secs 2024-07-26T16:07:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:59 smithi189 ceph-mon[36790]: pgmap v9621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:07:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:07:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16820.429040 secs 2024-07-26T16:08:01.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:08:01.315+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:08:01.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:08:01.315+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:08:01.317 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:08:01.315+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:08:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:01 smithi134 ceph-mon[31966]: pgmap v9622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:01 smithi189 ceph-mon[36790]: pgmap v9622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:01 smithi193 ceph-mon[37473]: pgmap v9622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16839.850699 secs 2024-07-26T16:08:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16839.850699 secs 2024-07-26T16:08:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16839.850699 secs 2024-07-26T16:08:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:03 smithi134 ceph-mon[31966]: pgmap v9623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:08:03.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:03 smithi189 ceph-mon[36790]: pgmap v9623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:08:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:03 smithi193 ceph-mon[37473]: pgmap v9623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:08:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16825.429170 secs 2024-07-26T16:08:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16825.429170 secs 2024-07-26T16:08:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16825.429170 secs 2024-07-26T16:08:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:05 smithi134 ceph-mon[31966]: pgmap v9624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:08:05.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:05 smithi189 ceph-mon[36790]: pgmap v9624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:08:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:05 smithi193 ceph-mon[37473]: pgmap v9624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:08:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:08:05.801+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:08:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:08:05.801+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:08:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:08:05.802+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:08:05.803 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:08:05.802+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:08:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:07 smithi134 ceph-mon[31966]: pgmap v9625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16844.850817 secs 2024-07-26T16:08:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:07 smithi189 ceph-mon[36790]: pgmap v9625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16844.850817 secs 2024-07-26T16:08:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:07 smithi193 ceph-mon[37473]: pgmap v9625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16844.850817 secs 2024-07-26T16:08:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:09 smithi134 ceph-mon[31966]: pgmap v9626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:08:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16830.429290 secs 2024-07-26T16:08:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:09 smithi189 ceph-mon[36790]: pgmap v9626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:08:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16830.429290 secs 2024-07-26T16:08:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:09 smithi193 ceph-mon[37473]: pgmap v9626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:08:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16830.429290 secs 2024-07-26T16:08:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:11 smithi134 ceph-mon[31966]: pgmap v9627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:08:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:11 smithi189 ceph-mon[36790]: pgmap v9627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:08:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:11 smithi193 ceph-mon[37473]: pgmap v9627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:08:11.893 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:08:11.891+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:08:11.893 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:08:11.891+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:08:11.893 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:08:11.891+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:08:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16849.850932 secs 2024-07-26T16:08:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16849.850932 secs 2024-07-26T16:08:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16849.850932 secs 2024-07-26T16:08:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:13 smithi134 ceph-mon[31966]: pgmap v9628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:08:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:13 smithi189 ceph-mon[36790]: pgmap v9628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:08:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:13 smithi193 ceph-mon[37473]: pgmap v9628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:08:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16835.429424 secs 2024-07-26T16:08:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16835.429424 secs 2024-07-26T16:08:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16835.429424 secs 2024-07-26T16:08:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:15 smithi134 ceph-mon[31966]: pgmap v9629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:08:15.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:08:15.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:15.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:15.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:15.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:15.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:08:15.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:08:15.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:08:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:15 smithi189 ceph-mon[36790]: pgmap v9629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:08:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:08:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:08:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:08:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:08:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:15 smithi193 ceph-mon[37473]: pgmap v9629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:08:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:08:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:08:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:08:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:08:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:08:16.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:08:16.319+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:08:16.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:08:16.319+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:08:16.321 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:08:16.319+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:08:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:16 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T16:08:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:16 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T16:08:16.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:16 smithi134 ceph-mon[31966]: pgmap v9630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T16:08:16.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:16 smithi134 ceph-mon[31966]: pgmap v9631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:08:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:16 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T16:08:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:16 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T16:08:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:16 smithi189 ceph-mon[36790]: pgmap v9630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T16:08:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:16 smithi189 ceph-mon[36790]: pgmap v9631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:08:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:16 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T16:08:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:16 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T16:08:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:16 smithi193 ceph-mon[37473]: pgmap v9630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 8 op/s 2024-07-26T16:08:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:16 smithi193 ceph-mon[37473]: pgmap v9631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:08:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16854.851057 secs 2024-07-26T16:08:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16854.851057 secs 2024-07-26T16:08:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16854.851057 secs 2024-07-26T16:08:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:18 smithi134 ceph-mon[31966]: pgmap v9632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:08:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:18 smithi189 ceph-mon[36790]: pgmap v9632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:08:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:18 smithi193 ceph-mon[37473]: pgmap v9632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:08:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16840.429554 secs 2024-07-26T16:08:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16840.429554 secs 2024-07-26T16:08:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16840.429554 secs 2024-07-26T16:08:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:20 smithi134 ceph-mon[31966]: pgmap v9633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:08:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:20 smithi189 ceph-mon[36790]: pgmap v9633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:08:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:20 smithi193 ceph-mon[37473]: pgmap v9633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:08:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16859.851202 secs 2024-07-26T16:08:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16859.851202 secs 2024-07-26T16:08:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16859.851202 secs 2024-07-26T16:08:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:22 smithi134 ceph-mon[31966]: pgmap v9634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:08:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:22 smithi189 ceph-mon[36790]: pgmap v9634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:08:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:22 smithi193 ceph-mon[37473]: pgmap v9634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:08:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:24 smithi134 ceph-mon[31966]: pgmap v9635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:08:24.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16845.429703 secs 2024-07-26T16:08:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:24 smithi189 ceph-mon[36790]: pgmap v9635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:08:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16845.429703 secs 2024-07-26T16:08:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:24 smithi193 ceph-mon[37473]: pgmap v9635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:08:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16845.429703 secs 2024-07-26T16:08:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:26 smithi134 ceph-mon[31966]: pgmap v9636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:08:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16864.851342 secs 2024-07-26T16:08:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:26 smithi189 ceph-mon[36790]: pgmap v9636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:08:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16864.851342 secs 2024-07-26T16:08:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:26 smithi193 ceph-mon[37473]: pgmap v9636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:08:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16864.851342 secs 2024-07-26T16:08:26.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:08:26.903+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:08:26.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:08:26.903+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:08:26.905 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:08:26.903+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:08:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:27 smithi189 ceph-mon[36790]: pgmap v9637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:08:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:27 smithi193 ceph-mon[37473]: pgmap v9637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:08:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:27 smithi134 ceph-mon[31966]: pgmap v9637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:08:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:28 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16850.429833 secs 2024-07-26T16:08:29.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:28 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16850.429833 secs 2024-07-26T16:08:29.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:28 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16850.429833 secs 2024-07-26T16:08:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:29 smithi134 ceph-mon[31966]: pgmap v9638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:08:30.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:29 smithi189 ceph-mon[36790]: pgmap v9638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:08:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:29 smithi193 ceph-mon[37473]: pgmap v9638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:08:31.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:08:31.324+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:08:31.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:08:31.324+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:08:31.326 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:08:31.324+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:08:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16869.851464 secs 2024-07-26T16:08:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:32 smithi134 ceph-mon[31966]: pgmap v9639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16869.851464 secs 2024-07-26T16:08:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:32 smithi189 ceph-mon[36790]: pgmap v9639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16869.851464 secs 2024-07-26T16:08:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:32 smithi193 ceph-mon[37473]: pgmap v9639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:34 smithi134 ceph-mon[31966]: pgmap v9640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:08:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16855.429990 secs 2024-07-26T16:08:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:34 smithi189 ceph-mon[36790]: pgmap v9640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:08:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16855.429990 secs 2024-07-26T16:08:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:34 smithi193 ceph-mon[37473]: pgmap v9640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:08:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16855.429990 secs 2024-07-26T16:08:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:36 smithi134 ceph-mon[31966]: pgmap v9641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:36 smithi189 ceph-mon[36790]: pgmap v9641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:36 smithi193 ceph-mon[37473]: pgmap v9641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16874.851551 secs 2024-07-26T16:08:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16874.851551 secs 2024-07-26T16:08:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16874.851551 secs 2024-07-26T16:08:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:38 smithi134 ceph-mon[31966]: pgmap v9642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:08:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:38 smithi189 ceph-mon[36790]: pgmap v9642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:08:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:38 smithi193 ceph-mon[37473]: pgmap v9642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:08:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16860.430145 secs 2024-07-26T16:08:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16860.430145 secs 2024-07-26T16:08:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16860.430145 secs 2024-07-26T16:08:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:40 smithi134 ceph-mon[31966]: pgmap v9643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:08:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:40 smithi189 ceph-mon[36790]: pgmap v9643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:08:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:40 smithi193 ceph-mon[37473]: pgmap v9643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:08:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16879.851675 secs 2024-07-26T16:08:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16879.851675 secs 2024-07-26T16:08:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16879.851675 secs 2024-07-26T16:08:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:08:41.918+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:08:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:08:41.918+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:08:41.919 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:08:41.918+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:08:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:42 smithi134 ceph-mon[31966]: pgmap v9644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:42 smithi189 ceph-mon[36790]: pgmap v9644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:42 smithi193 ceph-mon[37473]: pgmap v9644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:44 smithi134 ceph-mon[31966]: pgmap v9645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:08:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16865.430303 secs 2024-07-26T16:08:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:44 smithi189 ceph-mon[36790]: pgmap v9645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:08:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16865.430303 secs 2024-07-26T16:08:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:44 smithi193 ceph-mon[37473]: pgmap v9645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:08:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16865.430303 secs 2024-07-26T16:08:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:45 smithi193 ceph-mon[37473]: pgmap v9646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:45 smithi134 ceph-mon[31966]: pgmap v9646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:45 smithi189 ceph-mon[36790]: pgmap v9646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:46.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:08:46.336+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:08:46.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:08:46.336+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:08:46.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:08:46.336+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:08:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16884.851771 secs 2024-07-26T16:08:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16884.851771 secs 2024-07-26T16:08:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16884.851771 secs 2024-07-26T16:08:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:47 smithi193 ceph-mon[37473]: pgmap v9647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:08:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:47 smithi134 ceph-mon[31966]: pgmap v9647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:08:47.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:47 smithi189 ceph-mon[36790]: pgmap v9647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:08:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16870.430411 secs 2024-07-26T16:08:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16870.430411 secs 2024-07-26T16:08:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16870.430411 secs 2024-07-26T16:08:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:49 smithi134 ceph-mon[31966]: pgmap v9648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:08:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:49 smithi189 ceph-mon[36790]: pgmap v9648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:08:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:49 smithi193 ceph-mon[37473]: pgmap v9648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:08:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16889.851894 secs 2024-07-26T16:08:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:52 smithi134 ceph-mon[31966]: pgmap v9649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16889.851894 secs 2024-07-26T16:08:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:52 smithi189 ceph-mon[36790]: pgmap v9649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16889.851894 secs 2024-07-26T16:08:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:52 smithi193 ceph-mon[37473]: pgmap v9649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:54 smithi134 ceph-mon[31966]: pgmap v9650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:08:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16875.430497 secs 2024-07-26T16:08:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:54 smithi189 ceph-mon[36790]: pgmap v9650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:08:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16875.430497 secs 2024-07-26T16:08:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:54 smithi193 ceph-mon[37473]: pgmap v9650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:08:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16875.430497 secs 2024-07-26T16:08:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:56 smithi134 ceph-mon[31966]: pgmap v9651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:56 smithi189 ceph-mon[36790]: pgmap v9651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:56 smithi193 ceph-mon[37473]: pgmap v9651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:08:56.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:08:56.918+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:08:56.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:08:56.918+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:08:56.920 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:08:56.918+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:08:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16894.852025 secs 2024-07-26T16:08:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16894.852025 secs 2024-07-26T16:08:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16894.852025 secs 2024-07-26T16:08:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:58 smithi134 ceph-mon[31966]: pgmap v9652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:08:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:58 smithi189 ceph-mon[36790]: pgmap v9652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:08:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:58 smithi193 ceph-mon[37473]: pgmap v9652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:08:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:08:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16880.430613 secs 2024-07-26T16:08:59.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:08:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16880.430613 secs 2024-07-26T16:08:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:08:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16880.430613 secs 2024-07-26T16:09:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:00 smithi134 ceph-mon[31966]: pgmap v9653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:09:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:00 smithi189 ceph-mon[36790]: pgmap v9653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:09:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:00 smithi193 ceph-mon[37473]: pgmap v9653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:09:01.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:09:01.344+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:09:01.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:09:01.344+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:09:01.347 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:09:01.344+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:09:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16899.852164 secs 2024-07-26T16:09:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16899.852164 secs 2024-07-26T16:09:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16899.852164 secs 2024-07-26T16:09:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:02 smithi189 ceph-mon[36790]: pgmap v9654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:02 smithi193 ceph-mon[37473]: pgmap v9654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:02 smithi134 ceph-mon[31966]: pgmap v9654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:03 smithi193 ceph-mon[37473]: pgmap v9655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:03 smithi134 ceph-mon[31966]: pgmap v9655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:03 smithi189 ceph-mon[36790]: pgmap v9655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16885.430732 secs 2024-07-26T16:09:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16885.430732 secs 2024-07-26T16:09:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16885.430732 secs 2024-07-26T16:09:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:05 smithi193 ceph-mon[37473]: pgmap v9656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:05 smithi134 ceph-mon[31966]: pgmap v9656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:05.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:05 smithi189 ceph-mon[36790]: pgmap v9656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16904.852332 secs 2024-07-26T16:09:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16904.852332 secs 2024-07-26T16:09:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16904.852332 secs 2024-07-26T16:09:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:07 smithi193 ceph-mon[37473]: pgmap v9657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:09:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:07 smithi134 ceph-mon[31966]: pgmap v9657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:09:07.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:07 smithi189 ceph-mon[36790]: pgmap v9657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:09:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16890.430871 secs 2024-07-26T16:09:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16890.430871 secs 2024-07-26T16:09:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16890.430871 secs 2024-07-26T16:09:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:09 smithi134 ceph-mon[31966]: pgmap v9658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:09:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:09 smithi189 ceph-mon[36790]: pgmap v9658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:09:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:09 smithi193 ceph-mon[37473]: pgmap v9658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:09:11.922 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:09:11.920+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:09:11.922 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:09:11.920+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:09:11.922 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:09:11.920+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:09:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16909.852490 secs 2024-07-26T16:09:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:12 smithi134 ceph-mon[31966]: pgmap v9659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16909.852490 secs 2024-07-26T16:09:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:12 smithi189 ceph-mon[36790]: pgmap v9659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16909.852490 secs 2024-07-26T16:09:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:12 smithi193 ceph-mon[37473]: pgmap v9659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:14 smithi134 ceph-mon[31966]: pgmap v9660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16895.430966 secs 2024-07-26T16:09:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:14 smithi189 ceph-mon[36790]: pgmap v9660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16895.430966 secs 2024-07-26T16:09:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:14 smithi193 ceph-mon[37473]: pgmap v9660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16895.430966 secs 2024-07-26T16:09:16.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:09:16.349+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:09:16.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:09:16.349+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:09:16.352 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:09:16.349+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:09:16.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:16 smithi134 ceph-mon[31966]: pgmap v9661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:16.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:09:16.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:16 smithi189 ceph-mon[36790]: pgmap v9661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:09:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:16 smithi193 ceph-mon[37473]: pgmap v9661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:09:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:16.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:09:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:17 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T16:09:17.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:17 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T16:09:17.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:17 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T16:09:17.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:17 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T16:09:17.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16914.852594 secs 2024-07-26T16:09:17.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:09:17.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:09:17.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:09:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:17 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T16:09:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:17 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T16:09:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:17 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T16:09:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:17 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T16:09:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16914.852594 secs 2024-07-26T16:09:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:09:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:09:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:09:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:17 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T16:09:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:17 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T16:09:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:17 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T16:09:17.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:17 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T16:09:17.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16914.852594 secs 2024-07-26T16:09:17.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:09:17.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:09:17.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:09:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:18 smithi134 ceph-mon[31966]: pgmap v9662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:09:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:18 smithi134 ceph-mon[31966]: pgmap v9663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:18.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:18 smithi189 ceph-mon[36790]: pgmap v9662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:09:18.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:18 smithi189 ceph-mon[36790]: pgmap v9663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:18 smithi193 ceph-mon[37473]: pgmap v9662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:09:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:18 smithi193 ceph-mon[37473]: pgmap v9663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16900.431071 secs 2024-07-26T16:09:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16900.431071 secs 2024-07-26T16:09:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16900.431071 secs 2024-07-26T16:09:20.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:20 smithi189 ceph-mon[36790]: pgmap v9664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:09:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:20 smithi193 ceph-mon[37473]: pgmap v9664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:09:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:20 smithi134 ceph-mon[31966]: pgmap v9664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:09:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16919.852737 secs 2024-07-26T16:09:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16919.852737 secs 2024-07-26T16:09:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16919.852737 secs 2024-07-26T16:09:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:22 smithi189 ceph-mon[36790]: pgmap v9665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:09:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:22 smithi193 ceph-mon[37473]: pgmap v9665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:09:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:22 smithi134 ceph-mon[31966]: pgmap v9665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:09:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:23 smithi193 ceph-mon[37473]: pgmap v9666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:23 smithi134 ceph-mon[31966]: pgmap v9666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:23 smithi189 ceph-mon[36790]: pgmap v9666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16905.431192 secs 2024-07-26T16:09:24.873 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16905.431192 secs 2024-07-26T16:09:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16905.431192 secs 2024-07-26T16:09:25.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:09:25.266+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T16:09:25.267 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:09:25.266+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T16:09:25.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:09:25.266+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 132 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T16:09:25.268 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:09:25.266+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T16:09:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:25 smithi134 ceph-mon[31966]: pgmap v9667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:09:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:25 smithi193 ceph-mon[37473]: pgmap v9667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:09:25.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:25 smithi189 ceph-mon[36790]: pgmap v9667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:09:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16924.852883 secs 2024-07-26T16:09:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16924.852883 secs 2024-07-26T16:09:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:09:26.936+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:09:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:09:26.936+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:09:26.937 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:09:26.936+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:09:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16924.852883 secs 2024-07-26T16:09:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:28 smithi134 ceph-mon[31966]: pgmap v9668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:09:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:28 smithi189 ceph-mon[36790]: pgmap v9668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:09:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:28 smithi193 ceph-mon[37473]: pgmap v9668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:09:29.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16910.431288 secs 2024-07-26T16:09:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16910.431288 secs 2024-07-26T16:09:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16910.431288 secs 2024-07-26T16:09:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:30 smithi189 ceph-mon[36790]: pgmap v9669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:30 smithi193 ceph-mon[37473]: pgmap v9669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:30 smithi134 ceph-mon[31966]: pgmap v9669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:31.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:09:31.352+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:09:31.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:09:31.352+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:09:31.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:09:31.352+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:09:32.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:32 smithi189 ceph-mon[36790]: pgmap v9670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16929.852952 secs 2024-07-26T16:09:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:32 smithi193 ceph-mon[37473]: pgmap v9670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16929.852952 secs 2024-07-26T16:09:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:32 smithi134 ceph-mon[31966]: pgmap v9670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16929.852952 secs 2024-07-26T16:09:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:34 smithi193 ceph-mon[37473]: pgmap v9671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:09:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16915.431412 secs 2024-07-26T16:09:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:34 smithi134 ceph-mon[31966]: pgmap v9671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:09:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16915.431412 secs 2024-07-26T16:09:34.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:34 smithi189 ceph-mon[36790]: pgmap v9671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:09:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16915.431412 secs 2024-07-26T16:09:35.819 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:09:35.817+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:09:35.884 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:09:35.817+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:09:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:09:35.818+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:09:35.885 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:09:35.818+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:09:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:36 smithi193 ceph-mon[37473]: pgmap v9672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:09:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:36 smithi134 ceph-mon[31966]: pgmap v9672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:09:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:36 smithi189 ceph-mon[36790]: pgmap v9672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:09:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16934.853072 secs 2024-07-26T16:09:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16934.853072 secs 2024-07-26T16:09:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16934.853072 secs 2024-07-26T16:09:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:38 smithi193 ceph-mon[37473]: pgmap v9673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:38 smithi134 ceph-mon[31966]: pgmap v9673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:38 smithi189 ceph-mon[36790]: pgmap v9673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16920.431536 secs 2024-07-26T16:09:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16920.431536 secs 2024-07-26T16:09:39.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16920.431536 secs 2024-07-26T16:09:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:40 smithi193 ceph-mon[37473]: pgmap v9674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:40 smithi134 ceph-mon[31966]: pgmap v9674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:40 smithi189 ceph-mon[36790]: pgmap v9674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:09:41.951+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:09:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:09:41.951+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:09:41.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:09:41.951+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:09:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:42 smithi193 ceph-mon[37473]: pgmap v9675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16939.853199 secs 2024-07-26T16:09:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:42 smithi134 ceph-mon[31966]: pgmap v9675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16939.853199 secs 2024-07-26T16:09:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:42 smithi189 ceph-mon[36790]: pgmap v9675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16939.853199 secs 2024-07-26T16:09:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:44 smithi193 ceph-mon[37473]: pgmap v9676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:09:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16925.431659 secs 2024-07-26T16:09:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:44 smithi134 ceph-mon[31966]: pgmap v9676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:09:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16925.431659 secs 2024-07-26T16:09:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:44 smithi189 ceph-mon[36790]: pgmap v9676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:09:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16925.431659 secs 2024-07-26T16:09:46.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:09:46.360+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:09:46.362 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:09:46.360+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:09:46.363 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:09:46.360+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:09:46.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:46 smithi193 ceph-mon[37473]: pgmap v9677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:09:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:46 smithi134 ceph-mon[31966]: pgmap v9677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:09:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:46 smithi189 ceph-mon[36790]: pgmap v9677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:09:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16944.853347 secs 2024-07-26T16:09:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16944.853347 secs 2024-07-26T16:09:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16944.853347 secs 2024-07-26T16:09:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:48 smithi193 ceph-mon[37473]: pgmap v9678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:48 smithi134 ceph-mon[31966]: pgmap v9678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:48 smithi189 ceph-mon[36790]: pgmap v9678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16930.431777 secs 2024-07-26T16:09:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16930.431777 secs 2024-07-26T16:09:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16930.431777 secs 2024-07-26T16:09:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:50 smithi134 ceph-mon[31966]: pgmap v9679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:50 smithi189 ceph-mon[36790]: pgmap v9679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:50 smithi193 ceph-mon[37473]: pgmap v9679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:09:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:52 smithi134 ceph-mon[31966]: pgmap v9680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16949.853505 secs 2024-07-26T16:09:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:52 smithi189 ceph-mon[36790]: pgmap v9680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16949.853505 secs 2024-07-26T16:09:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:52 smithi193 ceph-mon[37473]: pgmap v9680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16949.853505 secs 2024-07-26T16:09:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:54 smithi134 ceph-mon[31966]: pgmap v9681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:09:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16935.431927 secs 2024-07-26T16:09:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:54 smithi189 ceph-mon[36790]: pgmap v9681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:09:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16935.431927 secs 2024-07-26T16:09:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:54 smithi193 ceph-mon[37473]: pgmap v9681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:09:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16935.431927 secs 2024-07-26T16:09:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:56 smithi134 ceph-mon[31966]: pgmap v9682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:09:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:56 smithi189 ceph-mon[36790]: pgmap v9682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:09:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:56 smithi193 ceph-mon[37473]: pgmap v9682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:09:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:09:56.967+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:09:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:09:56.967+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:09:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:09:56.967+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:09:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16954.853637 secs 2024-07-26T16:09:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16954.853637 secs 2024-07-26T16:09:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16954.853637 secs 2024-07-26T16:09:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:58 smithi134 ceph-mon[31966]: pgmap v9683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:58.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:58 smithi189 ceph-mon[36790]: pgmap v9683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:58 smithi193 ceph-mon[37473]: pgmap v9683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:09:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:09:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16940.432047 secs 2024-07-26T16:09:59.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:09:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16940.432047 secs 2024-07-26T16:09:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:09:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16940.432047 secs 2024-07-26T16:10:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:00 smithi134 ceph-mon[31966]: pgmap v9684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T16:10:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:00 smithi189 ceph-mon[36790]: pgmap v9684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T16:10:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:00 smithi193 ceph-mon[37473]: pgmap v9684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T16:10:01.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:10:01.375+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:10:01.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:10:01.376+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:10:01.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:10:01.376+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:10:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:02 smithi134 ceph-mon[31966]: pgmap v9685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16959.853721 secs 2024-07-26T16:10:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:02 smithi189 ceph-mon[36790]: pgmap v9685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16959.853721 secs 2024-07-26T16:10:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:02 smithi193 ceph-mon[37473]: pgmap v9685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16959.853721 secs 2024-07-26T16:10:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:04 smithi134 ceph-mon[31966]: pgmap v9686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:10:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16945.432167 secs 2024-07-26T16:10:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:04 smithi189 ceph-mon[36790]: pgmap v9686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:10:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16945.432167 secs 2024-07-26T16:10:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:04 smithi193 ceph-mon[37473]: pgmap v9686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:10:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16945.432167 secs 2024-07-26T16:10:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:06 smithi134 ceph-mon[31966]: pgmap v9687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:10:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:06 smithi189 ceph-mon[36790]: pgmap v9687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:10:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:06 smithi193 ceph-mon[37473]: pgmap v9687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:10:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16964.853863 secs 2024-07-26T16:10:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16964.853863 secs 2024-07-26T16:10:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16964.853863 secs 2024-07-26T16:10:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:08 smithi134 ceph-mon[31966]: pgmap v9688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:08 smithi189 ceph-mon[36790]: pgmap v9688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:08 smithi193 ceph-mon[37473]: pgmap v9688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16950.432286 secs 2024-07-26T16:10:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16950.432286 secs 2024-07-26T16:10:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16950.432286 secs 2024-07-26T16:10:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:10 smithi134 ceph-mon[31966]: pgmap v9689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:10 smithi189 ceph-mon[36790]: pgmap v9689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:10 smithi193 ceph-mon[37473]: pgmap v9689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:11.980 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:10:11.978+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:10:11.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:10:11.979+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:10:11.981 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:10:11.979+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:10:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:12 smithi134 ceph-mon[31966]: pgmap v9690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:12.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16969.853987 secs 2024-07-26T16:10:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:12 smithi189 ceph-mon[36790]: pgmap v9690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16969.853987 secs 2024-07-26T16:10:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:12 smithi193 ceph-mon[37473]: pgmap v9690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16969.853987 secs 2024-07-26T16:10:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:14 smithi134 ceph-mon[31966]: pgmap v9691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16955.432372 secs 2024-07-26T16:10:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:14 smithi189 ceph-mon[36790]: pgmap v9691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16955.432372 secs 2024-07-26T16:10:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:14 smithi193 ceph-mon[37473]: pgmap v9691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16955.432372 secs 2024-07-26T16:10:16.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:10:16.379+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:10:16.381 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:10:16.379+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:10:16.382 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:10:16.379+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:10:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:16 smithi134 ceph-mon[31966]: pgmap v9692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:10:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:16 smithi189 ceph-mon[36790]: pgmap v9692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:10:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:16 smithi193 ceph-mon[37473]: pgmap v9692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:10:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16974.854111 secs 2024-07-26T16:10:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:10:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16974.854111 secs 2024-07-26T16:10:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:10:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16974.854111 secs 2024-07-26T16:10:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:10:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:18 smithi134 ceph-mon[31966]: pgmap v9693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:10:18.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:10:18.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:10:18.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:18 smithi189 ceph-mon[36790]: pgmap v9693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:18.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:10:18.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:10:18.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:10:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:18 smithi193 ceph-mon[37473]: pgmap v9693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:10:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:10:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:10:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:19 smithi134 ceph-mon[31966]: pgmap v9694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T16:10:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:19 smithi134 ceph-mon[31966]: pgmap v9695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:10:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16960.432476 secs 2024-07-26T16:10:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:19 smithi189 ceph-mon[36790]: pgmap v9694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T16:10:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:19 smithi189 ceph-mon[36790]: pgmap v9695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:10:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16960.432476 secs 2024-07-26T16:10:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:19 smithi193 ceph-mon[37473]: pgmap v9694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T16:10:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:19 smithi193 ceph-mon[37473]: pgmap v9695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:10:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16960.432476 secs 2024-07-26T16:10:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:21 smithi189 ceph-mon[36790]: pgmap v9696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:10:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16979.854235 secs 2024-07-26T16:10:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:21 smithi193 ceph-mon[37473]: pgmap v9696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:10:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16979.854235 secs 2024-07-26T16:10:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:21 smithi134 ceph-mon[31966]: pgmap v9696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:10:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16979.854235 secs 2024-07-26T16:10:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:23 smithi189 ceph-mon[36790]: pgmap v9697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:10:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:23 smithi193 ceph-mon[37473]: pgmap v9697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:10:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:23 smithi134 ceph-mon[31966]: pgmap v9697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:10:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16965.432608 secs 2024-07-26T16:10:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16965.432608 secs 2024-07-26T16:10:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16965.432608 secs 2024-07-26T16:10:25.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:25 smithi189 ceph-mon[36790]: pgmap v9698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:10:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:25 smithi193 ceph-mon[37473]: pgmap v9698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:10:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:25 smithi134 ceph-mon[31966]: pgmap v9698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:10:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:26 smithi193 ceph-mon[37473]: pgmap v9699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:10:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16984.854397 secs 2024-07-26T16:10:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:26 smithi134 ceph-mon[31966]: pgmap v9699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:10:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16984.854397 secs 2024-07-26T16:10:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:26 smithi189 ceph-mon[36790]: pgmap v9699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:10:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16984.854397 secs 2024-07-26T16:10:26.988 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:10:26.986+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:10:26.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:10:26.986+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:10:26.989 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:10:26.986+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:10:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:28 smithi193 ceph-mon[37473]: pgmap v9700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:10:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:28 smithi134 ceph-mon[31966]: pgmap v9700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:10:29.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:28 smithi189 ceph-mon[36790]: pgmap v9700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:10:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16970.432736 secs 2024-07-26T16:10:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16970.432736 secs 2024-07-26T16:10:30.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16970.432736 secs 2024-07-26T16:10:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:30 smithi193 ceph-mon[37473]: pgmap v9701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:10:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:30 smithi134 ceph-mon[31966]: pgmap v9701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:10:31.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:30 smithi189 ceph-mon[36790]: pgmap v9701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:10:31.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:10:31.395+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:10:31.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:10:31.395+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:10:31.397 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:10:31.395+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:10:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16989.854498 secs 2024-07-26T16:10:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16989.854498 secs 2024-07-26T16:10:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16989.854498 secs 2024-07-26T16:10:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:32 smithi193 ceph-mon[37473]: pgmap v9702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:33.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:32 smithi134 ceph-mon[31966]: pgmap v9702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:33.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:32 smithi189 ceph-mon[36790]: pgmap v9702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16975.432877 secs 2024-07-26T16:10:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:34 smithi193 ceph-mon[37473]: pgmap v9703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16975.432877 secs 2024-07-26T16:10:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:34 smithi134 ceph-mon[31966]: pgmap v9703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:35.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16975.432877 secs 2024-07-26T16:10:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:34 smithi189 ceph-mon[36790]: pgmap v9703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:36 smithi193 ceph-mon[37473]: pgmap v9704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16994.854613 secs 2024-07-26T16:10:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:36 smithi134 ceph-mon[31966]: pgmap v9704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16994.854613 secs 2024-07-26T16:10:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:36 smithi189 ceph-mon[36790]: pgmap v9704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16994.854613 secs 2024-07-26T16:10:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:38 smithi193 ceph-mon[37473]: pgmap v9705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:38 smithi134 ceph-mon[31966]: pgmap v9705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:38 smithi189 ceph-mon[36790]: pgmap v9705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16980.433002 secs 2024-07-26T16:10:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16980.433002 secs 2024-07-26T16:10:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16980.433002 secs 2024-07-26T16:10:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:40 smithi134 ceph-mon[31966]: pgmap v9706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:10:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:40 smithi189 ceph-mon[36790]: pgmap v9706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:10:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:40 smithi193 ceph-mon[37473]: pgmap v9706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:10:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:10:42.000+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:10:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:10:42.000+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:10:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:10:42.000+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:10:42.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 16999.854743 secs 2024-07-26T16:10:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 16999.854743 secs 2024-07-26T16:10:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 16999.854743 secs 2024-07-26T16:10:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:42 smithi134 ceph-mon[31966]: pgmap v9707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:42 smithi189 ceph-mon[36790]: pgmap v9707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:42 smithi193 ceph-mon[37473]: pgmap v9707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16985.433085 secs 2024-07-26T16:10:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:44 smithi134 ceph-mon[31966]: pgmap v9708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16985.433085 secs 2024-07-26T16:10:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:44 smithi189 ceph-mon[36790]: pgmap v9708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16985.433085 secs 2024-07-26T16:10:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:44 smithi193 ceph-mon[37473]: pgmap v9708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:46.412 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:10:46.410+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:10:46.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:10:46.410+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:10:46.413 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:10:46.410+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:10:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:46 smithi134 ceph-mon[31966]: pgmap v9709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17004.854869 secs 2024-07-26T16:10:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:46 smithi189 ceph-mon[36790]: pgmap v9709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17004.854869 secs 2024-07-26T16:10:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:46 smithi193 ceph-mon[37473]: pgmap v9709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17004.854869 secs 2024-07-26T16:10:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:48 smithi134 ceph-mon[31966]: pgmap v9710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:48 smithi189 ceph-mon[36790]: pgmap v9710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:48 smithi193 ceph-mon[37473]: pgmap v9710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16990.433150 secs 2024-07-26T16:10:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16990.433150 secs 2024-07-26T16:10:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16990.433150 secs 2024-07-26T16:10:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:50 smithi134 ceph-mon[31966]: pgmap v9711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:10:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:50 smithi189 ceph-mon[36790]: pgmap v9711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:10:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:50 smithi193 ceph-mon[37473]: pgmap v9711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:10:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17009.854997 secs 2024-07-26T16:10:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17009.854997 secs 2024-07-26T16:10:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17009.854997 secs 2024-07-26T16:10:53.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:52 smithi189 ceph-mon[36790]: pgmap v9712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:52 smithi134 ceph-mon[31966]: pgmap v9712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:52 smithi193 ceph-mon[37473]: pgmap v9712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 16995.433269 secs 2024-07-26T16:10:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:54 smithi134 ceph-mon[31966]: pgmap v9713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 16995.433269 secs 2024-07-26T16:10:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:54 smithi189 ceph-mon[36790]: pgmap v9713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 16995.433269 secs 2024-07-26T16:10:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:54 smithi193 ceph-mon[37473]: pgmap v9713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:10:57.003+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:10:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:10:57.003+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:10:57.005 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:10:57.003+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:10:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:56 smithi134 ceph-mon[31966]: pgmap v9714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17014.855119 secs 2024-07-26T16:10:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:56 smithi189 ceph-mon[36790]: pgmap v9714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17014.855119 secs 2024-07-26T16:10:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:56 smithi193 ceph-mon[37473]: pgmap v9714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:10:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17014.855119 secs 2024-07-26T16:10:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:58 smithi134 ceph-mon[31966]: pgmap v9715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:10:58 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17000.433388 secs 2024-07-26T16:10:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:58 smithi189 ceph-mon[36790]: pgmap v9715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:10:58 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17000.433388 secs 2024-07-26T16:10:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:58 smithi193 ceph-mon[37473]: pgmap v9715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:10:59.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:10:58 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17000.433388 secs 2024-07-26T16:11:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:00 smithi134 ceph-mon[31966]: pgmap v9716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:11:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:00 smithi189 ceph-mon[36790]: pgmap v9716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:11:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:00 smithi193 ceph-mon[37473]: pgmap v9716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:11:01.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:11:01.425+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:11:01.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:11:01.426+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:11:01.428 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:11:01.426+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:11:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17019.855264 secs 2024-07-26T16:11:02.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17019.855264 secs 2024-07-26T16:11:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17019.855264 secs 2024-07-26T16:11:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:02 smithi134 ceph-mon[31966]: pgmap v9717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:02 smithi189 ceph-mon[36790]: pgmap v9717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:02 smithi193 ceph-mon[37473]: pgmap v9717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:04.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17005.433499 secs 2024-07-26T16:11:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17005.433499 secs 2024-07-26T16:11:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17005.433499 secs 2024-07-26T16:11:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:04 smithi134 ceph-mon[31966]: pgmap v9718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:04 smithi189 ceph-mon[36790]: pgmap v9718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:04 smithi193 ceph-mon[37473]: pgmap v9718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:06 smithi134 ceph-mon[31966]: pgmap v9719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17024.855419 secs 2024-07-26T16:11:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:06 smithi189 ceph-mon[36790]: pgmap v9719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17024.855419 secs 2024-07-26T16:11:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:06 smithi193 ceph-mon[37473]: pgmap v9719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17024.855419 secs 2024-07-26T16:11:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:08 smithi134 ceph-mon[31966]: pgmap v9720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17010.433647 secs 2024-07-26T16:11:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:08 smithi189 ceph-mon[36790]: pgmap v9720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17010.433647 secs 2024-07-26T16:11:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:08 smithi193 ceph-mon[37473]: pgmap v9720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17010.433647 secs 2024-07-26T16:11:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:10 smithi134 ceph-mon[31966]: pgmap v9721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:11:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:10 smithi189 ceph-mon[36790]: pgmap v9721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:11:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:10 smithi193 ceph-mon[37473]: pgmap v9721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:11:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:11:12.019+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:11:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:11:12.019+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:11:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:11:12.019+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:11:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17029.855559 secs 2024-07-26T16:11:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17029.855559 secs 2024-07-26T16:11:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17029.855559 secs 2024-07-26T16:11:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:12 smithi134 ceph-mon[31966]: pgmap v9722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:12 smithi189 ceph-mon[36790]: pgmap v9722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:12 smithi193 ceph-mon[37473]: pgmap v9722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17015.433766 secs 2024-07-26T16:11:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17015.433766 secs 2024-07-26T16:11:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17015.433766 secs 2024-07-26T16:11:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:14 smithi189 ceph-mon[36790]: pgmap v9723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:14 smithi193 ceph-mon[37473]: pgmap v9723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:14 smithi134 ceph-mon[31966]: pgmap v9723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:16.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:11:16.441+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:11:16.443 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:11:16.441+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:11:16.444 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:11:16.441+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:11:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:16 smithi189 ceph-mon[36790]: pgmap v9724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17034.855660 secs 2024-07-26T16:11:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:16 smithi193 ceph-mon[37473]: pgmap v9724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17034.855660 secs 2024-07-26T16:11:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:16 smithi134 ceph-mon[31966]: pgmap v9724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17034.855660 secs 2024-07-26T16:11:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:11:18.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:11:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:11:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:19 smithi189 ceph-mon[36790]: pgmap v9725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:19.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:11:19.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:11:19.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17020.433849 secs 2024-07-26T16:11:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:19 smithi193 ceph-mon[37473]: pgmap v9725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:11:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:11:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17020.433849 secs 2024-07-26T16:11:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:19 smithi134 ceph-mon[31966]: pgmap v9725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:11:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:11:19.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17020.433849 secs 2024-07-26T16:11:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:11:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:11:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:11:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:11:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:11:20.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:11:20.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:11:20.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:11:20.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:11:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:21 smithi193 ceph-mon[37473]: pgmap v9726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T16:11:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:21 smithi193 ceph-mon[37473]: pgmap v9727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:11:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17039.855754 secs 2024-07-26T16:11:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:21 smithi134 ceph-mon[31966]: pgmap v9726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T16:11:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:21 smithi134 ceph-mon[31966]: pgmap v9727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:11:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17039.855754 secs 2024-07-26T16:11:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:21 smithi189 ceph-mon[36790]: pgmap v9726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-07-26T16:11:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:21 smithi189 ceph-mon[36790]: pgmap v9727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:11:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17039.855754 secs 2024-07-26T16:11:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:22 smithi193 ceph-mon[37473]: pgmap v9728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:11:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:22 smithi134 ceph-mon[31966]: pgmap v9728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:11:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:22 smithi189 ceph-mon[36790]: pgmap v9728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:11:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:24 smithi193 ceph-mon[37473]: pgmap v9729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:11:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17025.433980 secs 2024-07-26T16:11:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:24 smithi134 ceph-mon[31966]: pgmap v9729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:11:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17025.433980 secs 2024-07-26T16:11:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:24 smithi189 ceph-mon[36790]: pgmap v9729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:11:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17025.433980 secs 2024-07-26T16:11:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:26 smithi193 ceph-mon[37473]: pgmap v9730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:11:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17044.855871 secs 2024-07-26T16:11:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:26 smithi134 ceph-mon[31966]: pgmap v9730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:11:26.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17044.855871 secs 2024-07-26T16:11:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:26 smithi189 ceph-mon[36790]: pgmap v9730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:11:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17044.855871 secs 2024-07-26T16:11:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:11:27.031+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:11:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:11:27.031+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:11:27.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:11:27.031+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:11:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:28 smithi134 ceph-mon[31966]: pgmap v9731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:11:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:28 smithi189 ceph-mon[36790]: pgmap v9731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:11:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:28 smithi193 ceph-mon[37473]: pgmap v9731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:11:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17030.434103 secs 2024-07-26T16:11:29.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17030.434103 secs 2024-07-26T16:11:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17030.434103 secs 2024-07-26T16:11:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:30 smithi134 ceph-mon[31966]: pgmap v9732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:11:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:30 smithi189 ceph-mon[36790]: pgmap v9732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:11:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:30 smithi193 ceph-mon[37473]: pgmap v9732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:11:31.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:11:31.445+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:11:31.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:11:31.445+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:11:31.448 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:11:31.445+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:11:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17049.856010 secs 2024-07-26T16:11:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17049.856010 secs 2024-07-26T16:11:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17049.856010 secs 2024-07-26T16:11:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:32 smithi134 ceph-mon[31966]: pgmap v9733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:32 smithi189 ceph-mon[36790]: pgmap v9733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:32 smithi193 ceph-mon[37473]: pgmap v9733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:34 smithi134 ceph-mon[31966]: pgmap v9734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17035.434218 secs 2024-07-26T16:11:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:34 smithi189 ceph-mon[36790]: pgmap v9734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17035.434218 secs 2024-07-26T16:11:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:34 smithi193 ceph-mon[37473]: pgmap v9734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17035.434218 secs 2024-07-26T16:11:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:36 smithi134 ceph-mon[31966]: pgmap v9735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17054.856087 secs 2024-07-26T16:11:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:36 smithi189 ceph-mon[36790]: pgmap v9735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17054.856087 secs 2024-07-26T16:11:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:36 smithi193 ceph-mon[37473]: pgmap v9735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17054.856087 secs 2024-07-26T16:11:38.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:38 smithi134 ceph-mon[31966]: pgmap v9736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:11:38.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:38 smithi189 ceph-mon[36790]: pgmap v9736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:11:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:38 smithi193 ceph-mon[37473]: pgmap v9736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:11:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17040.434345 secs 2024-07-26T16:11:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17040.434345 secs 2024-07-26T16:11:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17040.434345 secs 2024-07-26T16:11:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:40 smithi134 ceph-mon[31966]: pgmap v9737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:11:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:40 smithi189 ceph-mon[36790]: pgmap v9737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:11:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:40 smithi193 ceph-mon[37473]: pgmap v9737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:11:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17059.856205 secs 2024-07-26T16:11:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17059.856205 secs 2024-07-26T16:11:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17059.856205 secs 2024-07-26T16:11:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:11:42.046+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:11:42.048 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:11:42.047+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:11:42.049 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:11:42.047+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:11:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:42 smithi134 ceph-mon[31966]: pgmap v9738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:42 smithi189 ceph-mon[36790]: pgmap v9738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:42 smithi193 ceph-mon[37473]: pgmap v9738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:44 smithi134 ceph-mon[31966]: pgmap v9739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:44.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17045.434472 secs 2024-07-26T16:11:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:44 smithi189 ceph-mon[36790]: pgmap v9739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17045.434472 secs 2024-07-26T16:11:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:44 smithi193 ceph-mon[37473]: pgmap v9739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17045.434472 secs 2024-07-26T16:11:46.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:11:46.460+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:11:46.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:11:46.460+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:11:46.463 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:11:46.460+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:11:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:46 smithi134 ceph-mon[31966]: pgmap v9740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:46.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17064.856322 secs 2024-07-26T16:11:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:46 smithi189 ceph-mon[36790]: pgmap v9740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17064.856322 secs 2024-07-26T16:11:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:46 smithi193 ceph-mon[37473]: pgmap v9740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17064.856322 secs 2024-07-26T16:11:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:48 smithi134 ceph-mon[31966]: pgmap v9741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:11:48.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:48 smithi189 ceph-mon[36790]: pgmap v9741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:11:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:48 smithi193 ceph-mon[37473]: pgmap v9741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:11:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17050.434603 secs 2024-07-26T16:11:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17050.434603 secs 2024-07-26T16:11:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17050.434603 secs 2024-07-26T16:11:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:50 smithi134 ceph-mon[31966]: pgmap v9742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:11:50.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:50 smithi189 ceph-mon[36790]: pgmap v9742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:11:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:50 smithi193 ceph-mon[37473]: pgmap v9742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:11:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17069.856438 secs 2024-07-26T16:11:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17069.856438 secs 2024-07-26T16:11:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17069.856438 secs 2024-07-26T16:11:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:52 smithi134 ceph-mon[31966]: pgmap v9743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:52 smithi189 ceph-mon[36790]: pgmap v9743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:52 smithi193 ceph-mon[37473]: pgmap v9743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:54 smithi134 ceph-mon[31966]: pgmap v9744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17055.434725 secs 2024-07-26T16:11:54.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:54 smithi189 ceph-mon[36790]: pgmap v9744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17055.434725 secs 2024-07-26T16:11:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:54 smithi193 ceph-mon[37473]: pgmap v9744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:11:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17055.434725 secs 2024-07-26T16:11:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:56 smithi134 ceph-mon[31966]: pgmap v9745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17074.856555 secs 2024-07-26T16:11:56.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:56 smithi189 ceph-mon[36790]: pgmap v9745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17074.856555 secs 2024-07-26T16:11:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:56 smithi193 ceph-mon[37473]: pgmap v9745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:11:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17074.856555 secs 2024-07-26T16:11:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:11:57.062+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:11:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:11:57.062+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:11:57.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:11:57.062+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:11:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:58 smithi134 ceph-mon[31966]: pgmap v9746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:11:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:58 smithi189 ceph-mon[36790]: pgmap v9746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:11:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:58 smithi193 ceph-mon[37473]: pgmap v9746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:11:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:11:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17060.434876 secs 2024-07-26T16:12:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:11:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17060.434876 secs 2024-07-26T16:12:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:11:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17060.434876 secs 2024-07-26T16:12:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:00 smithi189 ceph-mon[36790]: pgmap v9747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:12:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:00 smithi193 ceph-mon[37473]: pgmap v9747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:12:01.033 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:00 smithi134 ceph-mon[31966]: pgmap v9747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:12:01.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:01.475+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:12:01.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:01.475+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:12:01.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:01.475+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:12:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17079.856658 secs 2024-07-26T16:12:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17079.856658 secs 2024-07-26T16:12:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17079.856658 secs 2024-07-26T16:12:02.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:02 smithi189 ceph-mon[36790]: pgmap v9748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:02 smithi193 ceph-mon[37473]: pgmap v9748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:02 smithi134 ceph-mon[31966]: pgmap v9748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:04 smithi193 ceph-mon[37473]: pgmap v9749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17065.434998 secs 2024-07-26T16:12:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:04 smithi134 ceph-mon[31966]: pgmap v9749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17065.434998 secs 2024-07-26T16:12:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:04 smithi189 ceph-mon[36790]: pgmap v9749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17065.434998 secs 2024-07-26T16:12:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:06 smithi193 ceph-mon[37473]: pgmap v9750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17084.856742 secs 2024-07-26T16:12:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:06 smithi134 ceph-mon[31966]: pgmap v9750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17084.856742 secs 2024-07-26T16:12:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:06 smithi189 ceph-mon[36790]: pgmap v9750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17084.856742 secs 2024-07-26T16:12:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:08 smithi193 ceph-mon[37473]: pgmap v9751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:12:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:08 smithi134 ceph-mon[31966]: pgmap v9751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:12:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:08 smithi189 ceph-mon[36790]: pgmap v9751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:12:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17070.435141 secs 2024-07-26T16:12:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17070.435141 secs 2024-07-26T16:12:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17070.435141 secs 2024-07-26T16:12:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:10 smithi193 ceph-mon[37473]: pgmap v9752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:12:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:10 smithi134 ceph-mon[31966]: pgmap v9752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:12:11.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:10 smithi189 ceph-mon[36790]: pgmap v9752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:12:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17089.856823 secs 2024-07-26T16:12:12.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17089.856823 secs 2024-07-26T16:12:12.076 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:12.075+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:12:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:12.075+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:12:12.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:12.075+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:12:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17089.856823 secs 2024-07-26T16:12:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:12 smithi193 ceph-mon[37473]: pgmap v9753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:12 smithi134 ceph-mon[31966]: pgmap v9753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:12 smithi189 ceph-mon[36790]: pgmap v9753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:14 smithi193 ceph-mon[37473]: pgmap v9754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17075.435258 secs 2024-07-26T16:12:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:14 smithi134 ceph-mon[31966]: pgmap v9754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17075.435258 secs 2024-07-26T16:12:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:14 smithi189 ceph-mon[36790]: pgmap v9754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17075.435258 secs 2024-07-26T16:12:16.478 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:16.476+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:12:16.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:16.476+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:12:16.479 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:16.476+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:12:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:16 smithi193 ceph-mon[37473]: pgmap v9755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17094.856946 secs 2024-07-26T16:12:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:16 smithi134 ceph-mon[31966]: pgmap v9755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17094.856946 secs 2024-07-26T16:12:17.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:16 smithi189 ceph-mon[36790]: pgmap v9755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17094.856946 secs 2024-07-26T16:12:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:19 smithi193 ceph-mon[37473]: pgmap v9756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17080.435379 secs 2024-07-26T16:12:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:19 smithi134 ceph-mon[31966]: pgmap v9756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17080.435379 secs 2024-07-26T16:12:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:19 smithi189 ceph-mon[36790]: pgmap v9756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17080.435379 secs 2024-07-26T16:12:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:12:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:20 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:12:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:20.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:20.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:20 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:12:20.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:20.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:20.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:20.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:20 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:21 smithi193 ceph-mon[37473]: pgmap v9757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:12:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:21 smithi134 ceph-mon[31966]: pgmap v9757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:12:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:21 smithi189 ceph-mon[36790]: pgmap v9757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:12:22.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:22.346+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 msgr2=0x7f9b300c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T16:12:22.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:22.346+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 msgr2=0x7f9b300c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T16:12:22.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:22.346+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 0x7f9b300c15d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800e640 tx=0x7f9b38008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:12:22.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:22.346+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 0x7f9b300c15d0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f9b3800e640 tx=0x7f9b38008040 comp rx=0 tx=0).stop 2024-07-26T16:12:22.349 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:22.346+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 msgr2=0x7f9b300c15d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T16:12:22.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:22.346+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 msgr2=0x7f9b300c15d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T16:12:22.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:22.346+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 0x7f9b300c15d0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T16:12:22.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:22.346+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 0x7f9b300c0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T16:12:22.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:22.347+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 0x7f9b300c0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:12:22.351 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:22.349+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 0x7f9b300c0be0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b38061040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T16:12:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17099.857036 secs 2024-07-26T16:12:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:22 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17099.857036 secs 2024-07-26T16:12:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:22 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17099.857036 secs 2024-07-26T16:12:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:22 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:23 smithi193 ceph-mon[37473]: pgmap v9758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:23 smithi134 ceph-mon[31966]: pgmap v9758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:23 smithi189 ceph-mon[36790]: pgmap v9758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:24 smithi134 ceph-mon[31966]: pgmap v9759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17085.435529 secs 2024-07-26T16:12:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:24 smithi189 ceph-mon[36790]: pgmap v9759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17085.435529 secs 2024-07-26T16:12:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:24 smithi193 ceph-mon[37473]: pgmap v9759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17085.435529 secs 2024-07-26T16:12:25.298 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:25.296+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T16:12:25.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:25.297+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T16:12:25.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:25.297+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 133 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T16:12:25.299 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:25.298+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T16:12:26.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:26.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:26.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:12:26.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:12:26.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:26 smithi134 ceph-mon[31966]: pgmap v9760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:12:26.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:26 smithi134 ceph-mon[31966]: pgmap v9761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:12:26.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:12:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:12:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:26 smithi189 ceph-mon[36790]: pgmap v9760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:12:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:26 smithi189 ceph-mon[36790]: pgmap v9761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:12:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:12:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:12:26.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:26 smithi193 ceph-mon[37473]: pgmap v9760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:12:26.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:26 smithi193 ceph-mon[37473]: pgmap v9761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:12:26.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:12:27.086 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:27.084+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:12:27.086 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:27.084+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:12:27.086 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:27.084+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:12:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17104.857193 secs 2024-07-26T16:12:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17104.857193 secs 2024-07-26T16:12:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17104.857193 secs 2024-07-26T16:12:28.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:28 smithi189 ceph-mon[36790]: pgmap v9762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T16:12:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:28 smithi193 ceph-mon[37473]: pgmap v9762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T16:12:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:28 smithi134 ceph-mon[31966]: pgmap v9762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T16:12:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17090.435659 secs 2024-07-26T16:12:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17090.435659 secs 2024-07-26T16:12:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17090.435659 secs 2024-07-26T16:12:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:30 smithi193 ceph-mon[37473]: pgmap v9763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:12:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:30 smithi134 ceph-mon[31966]: pgmap v9763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:12:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:30 smithi189 ceph-mon[36790]: pgmap v9763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:12:31.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:31.484+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:12:31.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:31.485+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:12:31.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:31.485+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:12:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17109.857275 secs 2024-07-26T16:12:32.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:32 smithi193 ceph-mon[37473]: pgmap v9764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:12:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17109.857275 secs 2024-07-26T16:12:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:32 smithi134 ceph-mon[31966]: pgmap v9764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:12:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17109.857275 secs 2024-07-26T16:12:32.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:32 smithi189 ceph-mon[36790]: pgmap v9764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:12:32.900 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:32.899+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 msgr2=0x7f21ac0c4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000923) for more than 900000000 us, fault. 2024-07-26T16:12:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:32.899+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f21980057a0 tx=0x7f21b8045000 comp rx=0 tx=0).stop 2024-07-26T16:12:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:32.899+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 msgr2=0x7f21ac0c4210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T16:12:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:32.899+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T16:12:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:32.899+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T16:12:32.901 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:32.900+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:12:32.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:32.901+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f21b8005f70 tx=0x7f21b80c3000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T16:12:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:34 smithi193 ceph-mon[37473]: pgmap v9765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:12:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17095.435779 secs 2024-07-26T16:12:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:34 smithi134 ceph-mon[31966]: pgmap v9765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:12:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17095.435779 secs 2024-07-26T16:12:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:34 smithi189 ceph-mon[36790]: pgmap v9765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:12:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17095.435779 secs 2024-07-26T16:12:35.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:35.849+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:12:35.851 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:35.849+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:12:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:35.850+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:12:35.852 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:35.850+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:12:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:36 smithi193 ceph-mon[37473]: pgmap v9766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:12:36.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:36 smithi134 ceph-mon[31966]: pgmap v9766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:12:36.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:36 smithi189 ceph-mon[36790]: pgmap v9766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:12:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17114.857406 secs 2024-07-26T16:12:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17114.857406 secs 2024-07-26T16:12:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17114.857406 secs 2024-07-26T16:12:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:38 smithi193 ceph-mon[37473]: pgmap v9767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:38 smithi134 ceph-mon[31966]: pgmap v9767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:38 smithi189 ceph-mon[36790]: pgmap v9767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17100.435906 secs 2024-07-26T16:12:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17100.435906 secs 2024-07-26T16:12:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17100.435906 secs 2024-07-26T16:12:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:40 smithi193 ceph-mon[37473]: pgmap v9768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:40 smithi134 ceph-mon[31966]: pgmap v9768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:40.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:40 smithi189 ceph-mon[36790]: pgmap v9768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:42.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:42.100+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:12:42.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:42.100+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:12:42.102 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:42.100+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:12:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17119.857562 secs 2024-07-26T16:12:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:42 smithi193 ceph-mon[37473]: pgmap v9769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17119.857562 secs 2024-07-26T16:12:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:42 smithi134 ceph-mon[31966]: pgmap v9769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17119.857562 secs 2024-07-26T16:12:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:42 smithi189 ceph-mon[36790]: pgmap v9769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:44 smithi193 ceph-mon[37473]: pgmap v9770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17105.436033 secs 2024-07-26T16:12:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:44 smithi134 ceph-mon[31966]: pgmap v9770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17105.436033 secs 2024-07-26T16:12:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:44 smithi189 ceph-mon[36790]: pgmap v9770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17105.436033 secs 2024-07-26T16:12:46.495 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:46.492+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:12:46.495 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:46.492+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:12:46.495 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:12:46.492+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:12:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:46 smithi193 ceph-mon[37473]: pgmap v9771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:46 smithi134 ceph-mon[31966]: pgmap v9771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:46.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:46 smithi189 ceph-mon[36790]: pgmap v9771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17124.857679 secs 2024-07-26T16:12:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17124.857679 secs 2024-07-26T16:12:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17124.857679 secs 2024-07-26T16:12:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:48 smithi134 ceph-mon[31966]: pgmap v9772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:48 smithi189 ceph-mon[36790]: pgmap v9772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:48 smithi193 ceph-mon[37473]: pgmap v9772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17110.436179 secs 2024-07-26T16:12:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17110.436179 secs 2024-07-26T16:12:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17110.436179 secs 2024-07-26T16:12:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:50 smithi134 ceph-mon[31966]: pgmap v9773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:50 smithi189 ceph-mon[36790]: pgmap v9773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:50 smithi193 ceph-mon[37473]: pgmap v9773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:12:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17129.857798 secs 2024-07-26T16:12:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:52 smithi134 ceph-mon[31966]: pgmap v9774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17129.857798 secs 2024-07-26T16:12:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:52 smithi189 ceph-mon[36790]: pgmap v9774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17129.857798 secs 2024-07-26T16:12:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:52 smithi193 ceph-mon[37473]: pgmap v9774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:54.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:54 smithi134 ceph-mon[31966]: pgmap v9775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:54.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17115.436303 secs 2024-07-26T16:12:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:54 smithi189 ceph-mon[36790]: pgmap v9775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17115.436303 secs 2024-07-26T16:12:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:54 smithi193 ceph-mon[37473]: pgmap v9775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:12:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17115.436303 secs 2024-07-26T16:12:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:56 smithi134 ceph-mon[31966]: pgmap v9776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:56 smithi189 ceph-mon[36790]: pgmap v9776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:56 smithi193 ceph-mon[37473]: pgmap v9776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:12:57.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:57.101+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:12:57.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:57.101+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:12:57.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:12:57.101+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:12:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17134.857876 secs 2024-07-26T16:12:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17134.857876 secs 2024-07-26T16:12:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17134.857876 secs 2024-07-26T16:12:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:58 smithi134 ceph-mon[31966]: pgmap v9777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:58 smithi189 ceph-mon[36790]: pgmap v9777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:58 smithi193 ceph-mon[37473]: pgmap v9777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:12:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:12:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17120.436435 secs 2024-07-26T16:12:59.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:12:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17120.436435 secs 2024-07-26T16:12:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:12:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17120.436435 secs 2024-07-26T16:13:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:00 smithi134 ceph-mon[31966]: pgmap v9778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:00 smithi189 ceph-mon[36790]: pgmap v9778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:00 smithi193 ceph-mon[37473]: pgmap v9778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:01.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:13:01.508+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:13:01.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:13:01.508+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:13:01.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:13:01.508+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:13:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17139.857993 secs 2024-07-26T16:13:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:02 smithi134 ceph-mon[31966]: pgmap v9779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17139.857993 secs 2024-07-26T16:13:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:02 smithi189 ceph-mon[36790]: pgmap v9779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17139.857993 secs 2024-07-26T16:13:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:02 smithi193 ceph-mon[37473]: pgmap v9779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:04 smithi134 ceph-mon[31966]: pgmap v9780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:13:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17125.436543 secs 2024-07-26T16:13:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:04 smithi189 ceph-mon[36790]: pgmap v9780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:13:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17125.436543 secs 2024-07-26T16:13:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:04 smithi193 ceph-mon[37473]: pgmap v9780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:13:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17125.436543 secs 2024-07-26T16:13:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:06 smithi134 ceph-mon[31966]: pgmap v9781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:13:06.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:06 smithi189 ceph-mon[36790]: pgmap v9781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:13:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:06 smithi193 ceph-mon[37473]: pgmap v9781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:13:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17144.858132 secs 2024-07-26T16:13:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17144.858132 secs 2024-07-26T16:13:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17144.858132 secs 2024-07-26T16:13:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:08 smithi134 ceph-mon[31966]: pgmap v9782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:13:08.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:08 smithi189 ceph-mon[36790]: pgmap v9782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:13:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:08 smithi193 ceph-mon[37473]: pgmap v9782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:13:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17130.436679 secs 2024-07-26T16:13:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17130.436679 secs 2024-07-26T16:13:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17130.436679 secs 2024-07-26T16:13:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:10 smithi134 ceph-mon[31966]: pgmap v9783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:13:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:10 smithi189 ceph-mon[36790]: pgmap v9783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:13:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:10 smithi193 ceph-mon[37473]: pgmap v9783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:13:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:13:12.116+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:13:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:13:12.117+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:13:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:13:12.117+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:13:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17149.858227 secs 2024-07-26T16:13:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:12 smithi134 ceph-mon[31966]: pgmap v9784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:13:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17149.858227 secs 2024-07-26T16:13:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:12 smithi189 ceph-mon[36790]: pgmap v9784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:13:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17149.858227 secs 2024-07-26T16:13:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:12 smithi193 ceph-mon[37473]: pgmap v9784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:13:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:14 smithi134 ceph-mon[31966]: pgmap v9785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:13:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17135.436834 secs 2024-07-26T16:13:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:14 smithi189 ceph-mon[36790]: pgmap v9785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:13:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17135.436834 secs 2024-07-26T16:13:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:14 smithi193 ceph-mon[37473]: pgmap v9785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:13:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17135.436834 secs 2024-07-26T16:13:16.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:13:16.523+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:13:16.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:13:16.523+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:13:16.526 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:13:16.523+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:13:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:16 smithi134 ceph-mon[31966]: pgmap v9786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:16 smithi189 ceph-mon[36790]: pgmap v9786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:16 smithi193 ceph-mon[37473]: pgmap v9786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17154.858345 secs 2024-07-26T16:13:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17154.858345 secs 2024-07-26T16:13:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17154.858345 secs 2024-07-26T16:13:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:18 smithi134 ceph-mon[31966]: pgmap v9787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:13:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:18 smithi189 ceph-mon[36790]: pgmap v9787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:13:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:18 smithi193 ceph-mon[37473]: pgmap v9787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:13:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17140.436994 secs 2024-07-26T16:13:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17140.436994 secs 2024-07-26T16:13:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17140.436994 secs 2024-07-26T16:13:20.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:20 smithi189 ceph-mon[36790]: pgmap v9788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:20 smithi193 ceph-mon[37473]: pgmap v9788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:20 smithi134 ceph-mon[31966]: pgmap v9788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:13:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17159.858463 secs 2024-07-26T16:13:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17159.858463 secs 2024-07-26T16:13:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17159.858463 secs 2024-07-26T16:13:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:22 smithi193 ceph-mon[37473]: pgmap v9789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:22 smithi134 ceph-mon[31966]: pgmap v9789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:22 smithi189 ceph-mon[36790]: pgmap v9789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:24.746 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:24 smithi189 ceph-mon[36790]: pgmap v9790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:13:24.746 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17145.437086 secs 2024-07-26T16:13:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:24 smithi193 ceph-mon[37473]: pgmap v9790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:13:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17145.437086 secs 2024-07-26T16:13:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:24 smithi134 ceph-mon[31966]: pgmap v9790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:13:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17145.437086 secs 2024-07-26T16:13:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:13:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:13:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:13:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:26 smithi193 ceph-mon[37473]: pgmap v9791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17164.858573 secs 2024-07-26T16:13:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:13:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:13:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:13:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:26 smithi134 ceph-mon[31966]: pgmap v9791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:26.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17164.858573 secs 2024-07-26T16:13:26.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:13:26.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:13:26.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:13:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:26 smithi189 ceph-mon[36790]: pgmap v9791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17164.858573 secs 2024-07-26T16:13:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:13:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:13:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:13:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:13:27.132+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:13:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:13:27.132+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:13:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:13:27.132+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:13:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:27 smithi193 ceph-mon[37473]: pgmap v9792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:13:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:27 smithi193 ceph-mon[37473]: pgmap v9793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:13:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:27 smithi134 ceph-mon[31966]: pgmap v9792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:13:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:27 smithi134 ceph-mon[31966]: pgmap v9793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:13:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:27 smithi189 ceph-mon[36790]: pgmap v9792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:13:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:27 smithi189 ceph-mon[36790]: pgmap v9793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:13:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:28 smithi193 ceph-mon[37473]: pgmap v9794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:13:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:28 smithi134 ceph-mon[31966]: pgmap v9794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:13:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:28 smithi189 ceph-mon[36790]: pgmap v9794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:13:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17150.437144 secs 2024-07-26T16:13:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17150.437144 secs 2024-07-26T16:13:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17150.437144 secs 2024-07-26T16:13:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:30 smithi193 ceph-mon[37473]: pgmap v9795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:13:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:30 smithi134 ceph-mon[31966]: pgmap v9795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:13:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:30 smithi189 ceph-mon[36790]: pgmap v9795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:13:31.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:13:31.529+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:13:31.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:13:31.529+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:13:31.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:13:31.529+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:13:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17169.858651 secs 2024-07-26T16:13:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17169.858651 secs 2024-07-26T16:13:31.947 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17169.858651 secs 2024-07-26T16:13:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:32 smithi134 ceph-mon[31966]: pgmap v9796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:32 smithi189 ceph-mon[36790]: pgmap v9796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:32 smithi193 ceph-mon[37473]: pgmap v9796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:33 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17155.437288 secs 2024-07-26T16:13:34.217 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:33 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17155.437288 secs 2024-07-26T16:13:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:33 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17155.437288 secs 2024-07-26T16:13:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:34 smithi189 ceph-mon[36790]: pgmap v9797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:13:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:34 smithi193 ceph-mon[37473]: pgmap v9797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:13:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:34 smithi134 ceph-mon[31966]: pgmap v9797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:13:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17174.858761 secs 2024-07-26T16:13:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:36 smithi193 ceph-mon[37473]: pgmap v9798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:13:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17174.858761 secs 2024-07-26T16:13:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:36 smithi134 ceph-mon[31966]: pgmap v9798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:13:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17174.858761 secs 2024-07-26T16:13:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:36 smithi189 ceph-mon[36790]: pgmap v9798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:13:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:39 smithi193 ceph-mon[37473]: pgmap v9799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:13:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17160.437419 secs 2024-07-26T16:13:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:39 smithi134 ceph-mon[31966]: pgmap v9799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:13:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17160.437419 secs 2024-07-26T16:13:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:39 smithi189 ceph-mon[36790]: pgmap v9799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:13:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17160.437419 secs 2024-07-26T16:13:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:40 smithi134 ceph-mon[31966]: pgmap v9800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:40 smithi189 ceph-mon[36790]: pgmap v9800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:40 smithi193 ceph-mon[37473]: pgmap v9800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17179.858850 secs 2024-07-26T16:13:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17179.858850 secs 2024-07-26T16:13:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17179.858850 secs 2024-07-26T16:13:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:13:42.141+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:13:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:13:42.141+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:13:42.142 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:13:42.141+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:13:42.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:42 smithi134 ceph-mon[31966]: pgmap v9801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:13:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:42 smithi189 ceph-mon[36790]: pgmap v9801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:13:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:42 smithi193 ceph-mon[37473]: pgmap v9801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:13:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:43 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17165.437522 secs 2024-07-26T16:13:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:43 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17165.437522 secs 2024-07-26T16:13:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:43 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17165.437522 secs 2024-07-26T16:13:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:44 smithi189 ceph-mon[36790]: pgmap v9802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:13:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:44 smithi193 ceph-mon[37473]: pgmap v9802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:13:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:44 smithi134 ceph-mon[31966]: pgmap v9802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:13:46.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:13:46.536+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:13:46.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:13:46.536+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:13:46.538 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:13:46.536+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:13:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17184.858973 secs 2024-07-26T16:13:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:46 smithi193 ceph-mon[37473]: pgmap v9803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17184.858973 secs 2024-07-26T16:13:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:46 smithi134 ceph-mon[31966]: pgmap v9803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17184.858973 secs 2024-07-26T16:13:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:46 smithi189 ceph-mon[36790]: pgmap v9803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:49 smithi193 ceph-mon[37473]: pgmap v9804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:13:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17170.437692 secs 2024-07-26T16:13:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:49 smithi134 ceph-mon[31966]: pgmap v9804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:13:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17170.437692 secs 2024-07-26T16:13:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:49 smithi189 ceph-mon[36790]: pgmap v9804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:13:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17170.437692 secs 2024-07-26T16:13:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:50 smithi193 ceph-mon[37473]: pgmap v9805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:50 smithi134 ceph-mon[31966]: pgmap v9805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:50.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:50 smithi189 ceph-mon[36790]: pgmap v9805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17189.859102 secs 2024-07-26T16:13:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17189.859102 secs 2024-07-26T16:13:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17189.859102 secs 2024-07-26T16:13:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:52 smithi193 ceph-mon[37473]: pgmap v9806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:13:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:52 smithi134 ceph-mon[31966]: pgmap v9806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:13:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:52 smithi189 ceph-mon[36790]: pgmap v9806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:13:54.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:53 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17175.437824 secs 2024-07-26T16:13:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:53 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17175.437824 secs 2024-07-26T16:13:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:53 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17175.437824 secs 2024-07-26T16:13:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:54 smithi189 ceph-mon[36790]: pgmap v9807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:13:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:54 smithi193 ceph-mon[37473]: pgmap v9807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:13:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:54 smithi134 ceph-mon[31966]: pgmap v9807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:13:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17194.859221 secs 2024-07-26T16:13:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:56 smithi193 ceph-mon[37473]: pgmap v9808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17194.859221 secs 2024-07-26T16:13:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:56 smithi134 ceph-mon[31966]: pgmap v9808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17194.859221 secs 2024-07-26T16:13:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:56 smithi189 ceph-mon[36790]: pgmap v9808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:13:57.158 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:13:57.156+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:13:57.158 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:13:57.156+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:13:57.158 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:13:57.156+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:13:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:58 smithi134 ceph-mon[31966]: pgmap v9809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:13:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:58 smithi189 ceph-mon[36790]: pgmap v9809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:13:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:58 smithi193 ceph-mon[37473]: pgmap v9809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:13:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:13:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17180.437988 secs 2024-07-26T16:13:59.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:13:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17180.437988 secs 2024-07-26T16:13:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:13:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17180.437988 secs 2024-07-26T16:14:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:00 smithi134 ceph-mon[31966]: pgmap v9810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:00.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:00 smithi189 ceph-mon[36790]: pgmap v9810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:00 smithi193 ceph-mon[37473]: pgmap v9810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:01.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:14:01.542+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:14:01.544 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:14:01.542+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:14:01.545 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:14:01.542+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:14:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17199.859379 secs 2024-07-26T16:14:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17199.859379 secs 2024-07-26T16:14:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17199.859379 secs 2024-07-26T16:14:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:02 smithi134 ceph-mon[31966]: pgmap v9811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:14:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:02 smithi189 ceph-mon[36790]: pgmap v9811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:14:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:02 smithi193 ceph-mon[37473]: pgmap v9811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:14:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17185.438131 secs 2024-07-26T16:14:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17185.438131 secs 2024-07-26T16:14:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17185.438131 secs 2024-07-26T16:14:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:04 smithi189 ceph-mon[36790]: pgmap v9812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:14:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:04 smithi193 ceph-mon[37473]: pgmap v9812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:14:05.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:04 smithi134 ceph-mon[31966]: pgmap v9812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:14:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17204.859489 secs 2024-07-26T16:14:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:06 smithi193 ceph-mon[37473]: pgmap v9813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17204.859489 secs 2024-07-26T16:14:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:06 smithi134 ceph-mon[31966]: pgmap v9813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17204.859489 secs 2024-07-26T16:14:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:06 smithi189 ceph-mon[36790]: pgmap v9813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:08 smithi134 ceph-mon[31966]: pgmap v9814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:08 smithi189 ceph-mon[36790]: pgmap v9814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:08 smithi193 ceph-mon[37473]: pgmap v9814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17190.438238 secs 2024-07-26T16:14:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17190.438238 secs 2024-07-26T16:14:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17190.438238 secs 2024-07-26T16:14:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:10 smithi134 ceph-mon[31966]: pgmap v9815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:10 smithi189 ceph-mon[36790]: pgmap v9815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:10 smithi193 ceph-mon[37473]: pgmap v9815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17209.859585 secs 2024-07-26T16:14:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17209.859585 secs 2024-07-26T16:14:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17209.859585 secs 2024-07-26T16:14:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:14:12.168+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:14:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:14:12.168+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:14:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:14:12.168+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:14:12.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:12 smithi134 ceph-mon[31966]: pgmap v9816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:14:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:12 smithi189 ceph-mon[36790]: pgmap v9816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:14:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:12 smithi193 ceph-mon[37473]: pgmap v9816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:14:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:13 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17195.438376 secs 2024-07-26T16:14:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:13 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17195.438376 secs 2024-07-26T16:14:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:13 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17195.438376 secs 2024-07-26T16:14:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:14 smithi189 ceph-mon[36790]: pgmap v9817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:14:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:14 smithi193 ceph-mon[37473]: pgmap v9817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:14:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:14 smithi134 ceph-mon[31966]: pgmap v9817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:14:16.547 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:14:16.545+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:14:16.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:14:16.545+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:14:16.548 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:14:16.545+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:14:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17214.859705 secs 2024-07-26T16:14:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:16 smithi193 ceph-mon[37473]: pgmap v9818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17214.859705 secs 2024-07-26T16:14:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:16 smithi134 ceph-mon[31966]: pgmap v9818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17214.859705 secs 2024-07-26T16:14:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:16 smithi189 ceph-mon[36790]: pgmap v9818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:18 smithi134 ceph-mon[31966]: pgmap v9819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:18 smithi189 ceph-mon[36790]: pgmap v9819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:18 smithi193 ceph-mon[37473]: pgmap v9819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17200.438492 secs 2024-07-26T16:14:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17200.438492 secs 2024-07-26T16:14:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17200.438492 secs 2024-07-26T16:14:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:20 smithi134 ceph-mon[31966]: pgmap v9820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:20.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:20 smithi189 ceph-mon[36790]: pgmap v9820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:20 smithi193 ceph-mon[37473]: pgmap v9820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17219.859810 secs 2024-07-26T16:14:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17219.859810 secs 2024-07-26T16:14:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17219.859810 secs 2024-07-26T16:14:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:22 smithi134 ceph-mon[31966]: pgmap v9821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:14:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:22 smithi189 ceph-mon[36790]: pgmap v9821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:14:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:22 smithi193 ceph-mon[37473]: pgmap v9821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:14:24.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:23 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17205.438639 secs 2024-07-26T16:14:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:23 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17205.438639 secs 2024-07-26T16:14:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:23 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17205.438639 secs 2024-07-26T16:14:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:24 smithi189 ceph-mon[36790]: pgmap v9822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:14:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:24 smithi193 ceph-mon[37473]: pgmap v9822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:14:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:24 smithi134 ceph-mon[31966]: pgmap v9822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:14:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17224.859937 secs 2024-07-26T16:14:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:26 smithi189 ceph-mon[36790]: pgmap v9823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:14:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17224.859937 secs 2024-07-26T16:14:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:26 smithi193 ceph-mon[37473]: pgmap v9823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:14:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17224.859937 secs 2024-07-26T16:14:26.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:26 smithi134 ceph-mon[31966]: pgmap v9823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:14:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:14:27.183+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:14:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:14:27.183+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:14:27.185 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:14:27.183+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:14:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:14:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:14:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:14:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:14:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:14:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:28 smithi193 ceph-mon[37473]: pgmap v9824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T16:14:28.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:28 smithi193 ceph-mon[37473]: pgmap v9825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:14:28.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:14:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:14:28.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:14:28.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:28 smithi134 ceph-mon[31966]: pgmap v9824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T16:14:28.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:28 smithi134 ceph-mon[31966]: pgmap v9825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:14:28.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:14:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:14:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:14:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:28 smithi189 ceph-mon[36790]: pgmap v9824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T16:14:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:28 smithi189 ceph-mon[36790]: pgmap v9825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:14:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:14:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17210.438720 secs 2024-07-26T16:14:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17210.438720 secs 2024-07-26T16:14:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17210.438720 secs 2024-07-26T16:14:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:30 smithi134 ceph-mon[31966]: pgmap v9826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:14:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:30 smithi189 ceph-mon[36790]: pgmap v9826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:14:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:30 smithi193 ceph-mon[37473]: pgmap v9826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:14:31.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:14:31.560+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:14:31.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:14:31.560+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:14:31.562 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:14:31.560+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:14:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17229.860085 secs 2024-07-26T16:14:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17229.860085 secs 2024-07-26T16:14:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17229.860085 secs 2024-07-26T16:14:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:32 smithi134 ceph-mon[31966]: pgmap v9827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:14:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:32 smithi189 ceph-mon[36790]: pgmap v9827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:14:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:32 smithi193 ceph-mon[37473]: pgmap v9827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:14:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:34 smithi134 ceph-mon[31966]: pgmap v9828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:14:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17215.438864 secs 2024-07-26T16:14:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:34 smithi189 ceph-mon[36790]: pgmap v9828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:14:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17215.438864 secs 2024-07-26T16:14:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:34 smithi193 ceph-mon[37473]: pgmap v9828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:14:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17215.438864 secs 2024-07-26T16:14:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:36 smithi134 ceph-mon[31966]: pgmap v9829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:14:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17234.860227 secs 2024-07-26T16:14:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:36 smithi189 ceph-mon[36790]: pgmap v9829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:14:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17234.860227 secs 2024-07-26T16:14:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:36 smithi193 ceph-mon[37473]: pgmap v9829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:14:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17234.860227 secs 2024-07-26T16:14:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:38 smithi134 ceph-mon[31966]: pgmap v9830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:14:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:38 smithi189 ceph-mon[36790]: pgmap v9830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:14:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:38 smithi193 ceph-mon[37473]: pgmap v9830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:14:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17220.438983 secs 2024-07-26T16:14:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17220.438983 secs 2024-07-26T16:14:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17220.438983 secs 2024-07-26T16:14:40.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:40 smithi189 ceph-mon[36790]: pgmap v9831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:14:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:40 smithi193 ceph-mon[37473]: pgmap v9831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:14:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:40 smithi134 ceph-mon[31966]: pgmap v9831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:14:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17239.860391 secs 2024-07-26T16:14:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17239.860391 secs 2024-07-26T16:14:42.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17239.860391 secs 2024-07-26T16:14:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:14:42.198+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:14:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:14:42.198+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:14:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:14:42.198+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:14:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:42 smithi193 ceph-mon[37473]: pgmap v9832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:42 smithi134 ceph-mon[31966]: pgmap v9832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:43.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:42 smithi189 ceph-mon[36790]: pgmap v9832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:44 smithi193 ceph-mon[37473]: pgmap v9833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17225.439119 secs 2024-07-26T16:14:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:44 smithi134 ceph-mon[31966]: pgmap v9833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17225.439119 secs 2024-07-26T16:14:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:44 smithi189 ceph-mon[36790]: pgmap v9833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17225.439119 secs 2024-07-26T16:14:46.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:14:46.563+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:14:46.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:14:46.563+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:14:46.565 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:14:46.563+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:14:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:46 smithi134 ceph-mon[31966]: pgmap v9834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17244.860514 secs 2024-07-26T16:14:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:46 smithi193 ceph-mon[37473]: pgmap v9834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17244.860514 secs 2024-07-26T16:14:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:46 smithi189 ceph-mon[36790]: pgmap v9834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17244.860514 secs 2024-07-26T16:14:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:48 smithi193 ceph-mon[37473]: pgmap v9835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:48 smithi134 ceph-mon[31966]: pgmap v9835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:48 smithi189 ceph-mon[36790]: pgmap v9835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17230.439186 secs 2024-07-26T16:14:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17230.439186 secs 2024-07-26T16:14:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17230.439186 secs 2024-07-26T16:14:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:50 smithi193 ceph-mon[37473]: pgmap v9836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:14:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:50 smithi134 ceph-mon[31966]: pgmap v9836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:14:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:50 smithi189 ceph-mon[36790]: pgmap v9836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:14:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17249.860653 secs 2024-07-26T16:14:52.041 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17249.860653 secs 2024-07-26T16:14:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17249.860653 secs 2024-07-26T16:14:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:52 smithi193 ceph-mon[37473]: pgmap v9837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:52 smithi134 ceph-mon[31966]: pgmap v9837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:53.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:52 smithi189 ceph-mon[36790]: pgmap v9837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:54 smithi193 ceph-mon[37473]: pgmap v9838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17235.439292 secs 2024-07-26T16:14:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:54 smithi134 ceph-mon[31966]: pgmap v9838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17235.439292 secs 2024-07-26T16:14:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:54 smithi189 ceph-mon[36790]: pgmap v9838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17235.439292 secs 2024-07-26T16:14:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:56 smithi193 ceph-mon[37473]: pgmap v9839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17254.860780 secs 2024-07-26T16:14:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:56 smithi134 ceph-mon[31966]: pgmap v9839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17254.860780 secs 2024-07-26T16:14:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:56 smithi189 ceph-mon[36790]: pgmap v9839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:14:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17254.860780 secs 2024-07-26T16:14:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:14:57.211+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:14:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:14:57.211+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:14:57.213 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:14:57.211+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:14:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:58 smithi193 ceph-mon[37473]: pgmap v9840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:58 smithi134 ceph-mon[31966]: pgmap v9840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:14:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:58 smithi189 ceph-mon[36790]: pgmap v9840: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:14:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17240.439448 secs 2024-07-26T16:15:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:14:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17240.439448 secs 2024-07-26T16:15:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:14:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17240.439448 secs 2024-07-26T16:15:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:00 smithi134 ceph-mon[31966]: pgmap v9841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:15:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:00 smithi189 ceph-mon[36790]: pgmap v9841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:15:01.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:00 smithi193 ceph-mon[37473]: pgmap v9841: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:15:01.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:15:01.579+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:15:01.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:15:01.579+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:15:01.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:15:01.579+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:15:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17259.860929 secs 2024-07-26T16:15:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17259.860929 secs 2024-07-26T16:15:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17259.860929 secs 2024-07-26T16:15:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:02 smithi134 ceph-mon[31966]: pgmap v9842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:02 smithi189 ceph-mon[36790]: pgmap v9842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:03.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:02 smithi193 ceph-mon[37473]: pgmap v9842: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:04 smithi134 ceph-mon[31966]: pgmap v9843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17245.439569 secs 2024-07-26T16:15:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:04 smithi189 ceph-mon[36790]: pgmap v9843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17245.439569 secs 2024-07-26T16:15:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:04 smithi193 ceph-mon[37473]: pgmap v9843: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17245.439569 secs 2024-07-26T16:15:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:06 smithi134 ceph-mon[31966]: pgmap v9844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17264.861051 secs 2024-07-26T16:15:07.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:06 smithi189 ceph-mon[36790]: pgmap v9844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17264.861051 secs 2024-07-26T16:15:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:06 smithi193 ceph-mon[37473]: pgmap v9844: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17264.861051 secs 2024-07-26T16:15:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:08 smithi134 ceph-mon[31966]: pgmap v9845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:09.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:08 smithi189 ceph-mon[36790]: pgmap v9845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:08 smithi193 ceph-mon[37473]: pgmap v9845: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17250.439684 secs 2024-07-26T16:15:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17250.439684 secs 2024-07-26T16:15:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17250.439684 secs 2024-07-26T16:15:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:10 smithi134 ceph-mon[31966]: pgmap v9846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:15:11.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:10 smithi189 ceph-mon[36790]: pgmap v9846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:15:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:10 smithi193 ceph-mon[37473]: pgmap v9846: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:15:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17269.861202 secs 2024-07-26T16:15:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17269.861202 secs 2024-07-26T16:15:12.218 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:15:12.216+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:15:12.218 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:15:12.216+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:15:12.218 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:15:12.216+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:15:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17269.861202 secs 2024-07-26T16:15:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:13 smithi134 ceph-mon[31966]: pgmap v9847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:13.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:13 smithi189 ceph-mon[36790]: pgmap v9847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:13 smithi193 ceph-mon[37473]: pgmap v9847: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17255.439831 secs 2024-07-26T16:15:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17255.439831 secs 2024-07-26T16:15:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17255.439831 secs 2024-07-26T16:15:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:15 smithi134 ceph-mon[31966]: pgmap v9848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:15 smithi189 ceph-mon[36790]: pgmap v9848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:15 smithi193 ceph-mon[37473]: pgmap v9848: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:16.596 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:15:16.594+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:15:16.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:15:16.594+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:15:16.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:15:16.594+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:15:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:17 smithi134 ceph-mon[31966]: pgmap v9849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17274.861379 secs 2024-07-26T16:15:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:17 smithi189 ceph-mon[36790]: pgmap v9849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17274.861379 secs 2024-07-26T16:15:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:17 smithi193 ceph-mon[37473]: pgmap v9849: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17274.861379 secs 2024-07-26T16:15:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:19 smithi134 ceph-mon[31966]: pgmap v9850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17260.439985 secs 2024-07-26T16:15:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:19 smithi189 ceph-mon[36790]: pgmap v9850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17260.439985 secs 2024-07-26T16:15:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:19 smithi193 ceph-mon[37473]: pgmap v9850: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17260.439985 secs 2024-07-26T16:15:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:21 smithi134 ceph-mon[31966]: pgmap v9851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:15:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:21 smithi189 ceph-mon[36790]: pgmap v9851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:15:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:21 smithi193 ceph-mon[37473]: pgmap v9851: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:15:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17279.861507 secs 2024-07-26T16:15:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17279.861507 secs 2024-07-26T16:15:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17279.861507 secs 2024-07-26T16:15:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:23 smithi134 ceph-mon[31966]: pgmap v9852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:23 smithi189 ceph-mon[36790]: pgmap v9852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:23 smithi193 ceph-mon[37473]: pgmap v9852: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17265.440135 secs 2024-07-26T16:15:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17265.440135 secs 2024-07-26T16:15:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17265.440135 secs 2024-07-26T16:15:25.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:15:25.328+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T16:15:25.330 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:15:25.328+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T16:15:25.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:15:25.329+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T16:15:25.331 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:15:25.329+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 134 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T16:15:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:25 smithi134 ceph-mon[31966]: pgmap v9853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:25.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:25 smithi189 ceph-mon[36790]: pgmap v9853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:25 smithi193 ceph-mon[37473]: pgmap v9853: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:27.224 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:15:27.222+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:15:27.224 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:15:27.222+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:15:27.224 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:15:27.222+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:15:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:27 smithi134 ceph-mon[31966]: pgmap v9854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17284.861634 secs 2024-07-26T16:15:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:27 smithi189 ceph-mon[36790]: pgmap v9854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17284.861634 secs 2024-07-26T16:15:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:27 smithi193 ceph-mon[37473]: pgmap v9854: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17284.861634 secs 2024-07-26T16:15:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:15:28.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:15:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:15:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:29 smithi134 ceph-mon[31966]: pgmap v9855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:29.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17270.440261 secs 2024-07-26T16:15:29.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:15:29.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:15:29.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:15:29.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:29 smithi193 ceph-mon[37473]: pgmap v9855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:29.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17270.440261 secs 2024-07-26T16:15:29.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:15:29.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:15:29.676 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:15:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:29 smithi189 ceph-mon[36790]: pgmap v9855: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17270.440261 secs 2024-07-26T16:15:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:15:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:15:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:15:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:30 smithi134 ceph-mon[31966]: pgmap v9856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:15:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:30 smithi134 ceph-mon[31966]: pgmap v9857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:15:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:30 smithi189 ceph-mon[36790]: pgmap v9856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:15:30.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:30 smithi189 ceph-mon[36790]: pgmap v9857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:15:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:30 smithi193 ceph-mon[37473]: pgmap v9856: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:15:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:30 smithi193 ceph-mon[37473]: pgmap v9857: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:15:31.599 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:15:31.597+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:15:31.599 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:15:31.597+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:15:31.599 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:15:31.597+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:15:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:32 smithi134 ceph-mon[31966]: pgmap v9858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:15:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17289.861744 secs 2024-07-26T16:15:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:32 smithi189 ceph-mon[36790]: pgmap v9858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:15:32.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17289.861744 secs 2024-07-26T16:15:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:32 smithi193 ceph-mon[37473]: pgmap v9858: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:15:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17289.861744 secs 2024-07-26T16:15:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:34 smithi189 ceph-mon[36790]: pgmap v9859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:15:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17275.440355 secs 2024-07-26T16:15:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:34 smithi193 ceph-mon[37473]: pgmap v9859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:15:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17275.440355 secs 2024-07-26T16:15:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:34 smithi134 ceph-mon[31966]: pgmap v9859: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:15:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17275.440355 secs 2024-07-26T16:15:35.882 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:15:35.881+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:15:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:15:35.881+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:15:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:15:35.881+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:15:35.883 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:15:35.882+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:15:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:36 smithi189 ceph-mon[36790]: pgmap v9860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:15:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:36 smithi193 ceph-mon[37473]: pgmap v9860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:15:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:36 smithi134 ceph-mon[31966]: pgmap v9860: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:15:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17294.861868 secs 2024-07-26T16:15:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17294.861868 secs 2024-07-26T16:15:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17294.861868 secs 2024-07-26T16:15:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:38 smithi193 ceph-mon[37473]: pgmap v9861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:15:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:38 smithi134 ceph-mon[31966]: pgmap v9861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:15:38.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:38 smithi189 ceph-mon[36790]: pgmap v9861: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:15:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17280.440451 secs 2024-07-26T16:15:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17280.440451 secs 2024-07-26T16:15:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17280.440451 secs 2024-07-26T16:15:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:40 smithi193 ceph-mon[37473]: pgmap v9862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:15:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:40 smithi134 ceph-mon[31966]: pgmap v9862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:15:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:40 smithi189 ceph-mon[36790]: pgmap v9862: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:15:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17299.861990 secs 2024-07-26T16:15:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17299.861990 secs 2024-07-26T16:15:41.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17299.861990 secs 2024-07-26T16:15:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:15:42.238+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:15:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:15:42.238+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:15:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:15:42.238+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:15:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:42 smithi193 ceph-mon[37473]: pgmap v9863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:42.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:42 smithi134 ceph-mon[31966]: pgmap v9863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:42 smithi189 ceph-mon[36790]: pgmap v9863: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:44 smithi193 ceph-mon[37473]: pgmap v9864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17285.440600 secs 2024-07-26T16:15:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:44 smithi134 ceph-mon[31966]: pgmap v9864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17285.440600 secs 2024-07-26T16:15:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:44 smithi189 ceph-mon[36790]: pgmap v9864: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17285.440600 secs 2024-07-26T16:15:46.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:15:46.612+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:15:46.614 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:15:46.612+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:15:46.615 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:15:46.612+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:15:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:46 smithi193 ceph-mon[37473]: pgmap v9865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:15:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17304.862133 secs 2024-07-26T16:15:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:46 smithi134 ceph-mon[31966]: pgmap v9865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:15:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17304.862133 secs 2024-07-26T16:15:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:46 smithi189 ceph-mon[36790]: pgmap v9865: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:15:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17304.862133 secs 2024-07-26T16:15:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:48 smithi193 ceph-mon[37473]: pgmap v9866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:48 smithi134 ceph-mon[31966]: pgmap v9866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:48.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:48 smithi189 ceph-mon[36790]: pgmap v9866: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17290.440730 secs 2024-07-26T16:15:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17290.440730 secs 2024-07-26T16:15:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17290.440730 secs 2024-07-26T16:15:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:50 smithi193 ceph-mon[37473]: pgmap v9867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:50 smithi134 ceph-mon[31966]: pgmap v9867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:50 smithi189 ceph-mon[36790]: pgmap v9867: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17309.862261 secs 2024-07-26T16:15:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17309.862261 secs 2024-07-26T16:15:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17309.862261 secs 2024-07-26T16:15:52.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:52 smithi193 ceph-mon[37473]: pgmap v9868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:52 smithi134 ceph-mon[31966]: pgmap v9868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:52 smithi189 ceph-mon[36790]: pgmap v9868: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:53 smithi134 ceph-mon[31966]: pgmap v9869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:53.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:53 smithi189 ceph-mon[36790]: pgmap v9869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:53 smithi193 ceph-mon[37473]: pgmap v9869: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:15:54.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17295.440889 secs 2024-07-26T16:15:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17295.440889 secs 2024-07-26T16:15:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17295.440889 secs 2024-07-26T16:15:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:55 smithi134 ceph-mon[31966]: pgmap v9870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:15:55.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:55 smithi189 ceph-mon[36790]: pgmap v9870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:15:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:55 smithi193 ceph-mon[37473]: pgmap v9870: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:15:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17314.862388 secs 2024-07-26T16:15:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17314.862388 secs 2024-07-26T16:15:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17314.862388 secs 2024-07-26T16:15:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:15:57.253+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:15:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:15:57.253+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:15:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:15:57.253+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:15:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:57 smithi134 ceph-mon[31966]: pgmap v9871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:57 smithi189 ceph-mon[36790]: pgmap v9871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:15:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:57 smithi193 ceph-mon[37473]: pgmap v9871: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17300.441015 secs 2024-07-26T16:16:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:15:59 smithi134 ceph-mon[31966]: pgmap v9872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17300.441015 secs 2024-07-26T16:16:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:15:59 smithi189 ceph-mon[36790]: pgmap v9872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17300.441015 secs 2024-07-26T16:16:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:15:59 smithi193 ceph-mon[37473]: pgmap v9872: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:01.624 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:16:01.622+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:16:01.624 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:16:01.622+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:16:01.624 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:16:01.622+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:16:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:01 smithi134 ceph-mon[31966]: pgmap v9873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17319.862491 secs 2024-07-26T16:16:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:01 smithi189 ceph-mon[36790]: pgmap v9873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17319.862491 secs 2024-07-26T16:16:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:01 smithi193 ceph-mon[37473]: pgmap v9873: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17319.862491 secs 2024-07-26T16:16:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:03 smithi134 ceph-mon[31966]: pgmap v9874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:03 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17305.441150 secs 2024-07-26T16:16:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:03 smithi189 ceph-mon[36790]: pgmap v9874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:03 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17305.441150 secs 2024-07-26T16:16:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:03 smithi193 ceph-mon[37473]: pgmap v9874: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:03 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17305.441150 secs 2024-07-26T16:16:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:05 smithi134 ceph-mon[31966]: pgmap v9875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:16:06.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:05 smithi189 ceph-mon[36790]: pgmap v9875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:16:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:05 smithi193 ceph-mon[37473]: pgmap v9875: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:16:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17324.862616 secs 2024-07-26T16:16:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17324.862616 secs 2024-07-26T16:16:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17324.862616 secs 2024-07-26T16:16:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:07 smithi189 ceph-mon[36790]: pgmap v9876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:07 smithi193 ceph-mon[37473]: pgmap v9876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:07 smithi134 ceph-mon[31966]: pgmap v9876: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:08 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17310.441232 secs 2024-07-26T16:16:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:08 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17310.441232 secs 2024-07-26T16:16:09.354 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:08 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17310.441232 secs 2024-07-26T16:16:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:09 smithi189 ceph-mon[36790]: pgmap v9877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:09 smithi193 ceph-mon[37473]: pgmap v9877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:10.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:16:10.336+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T16:16:10.338 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:16:10.336+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T16:16:10.339 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:09 smithi134 ceph-mon[31966]: pgmap v9877: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:10.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:16:10.337+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T16:16:10.339 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:16:10.337+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 135 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T16:16:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:16:12.257+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:16:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:16:12.257+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:16:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:16:12.257+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:16:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:11 smithi193 ceph-mon[37473]: pgmap v9878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:12.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17329.862770 secs 2024-07-26T16:16:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:11 smithi134 ceph-mon[31966]: pgmap v9878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17329.862770 secs 2024-07-26T16:16:12.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:11 smithi189 ceph-mon[36790]: pgmap v9878: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17329.862770 secs 2024-07-26T16:16:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:14 smithi134 ceph-mon[31966]: pgmap v9879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17315.441363 secs 2024-07-26T16:16:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:14 smithi189 ceph-mon[36790]: pgmap v9879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17315.441363 secs 2024-07-26T16:16:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:14 smithi193 ceph-mon[37473]: pgmap v9879: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17315.441363 secs 2024-07-26T16:16:16.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:16:16.638+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:16:16.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:16:16.638+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:16:16.640 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:16:16.638+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:16:16.640 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:16 smithi134 ceph-mon[31966]: pgmap v9880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:16:16.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:16 smithi189 ceph-mon[36790]: pgmap v9880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:16:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:16 smithi193 ceph-mon[37473]: pgmap v9880: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:16:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17334.862904 secs 2024-07-26T16:16:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17334.862904 secs 2024-07-26T16:16:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17334.862904 secs 2024-07-26T16:16:18.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:18 smithi134 ceph-mon[31966]: pgmap v9881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:18.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:18 smithi189 ceph-mon[36790]: pgmap v9881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:18 smithi193 ceph-mon[37473]: pgmap v9881: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17320.441503 secs 2024-07-26T16:16:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17320.441503 secs 2024-07-26T16:16:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17320.441503 secs 2024-07-26T16:16:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:20 smithi134 ceph-mon[31966]: pgmap v9882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:20.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:20 smithi189 ceph-mon[36790]: pgmap v9882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:20 smithi193 ceph-mon[37473]: pgmap v9882: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:20.890 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:16:20.889+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:16:20.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:16:20.889+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:16:20.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:16:20.890+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:16:20.891 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:16:20.890+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:16:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:22 smithi134 ceph-mon[31966]: pgmap v9883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:22.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17339.862972 secs 2024-07-26T16:16:22.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:22 smithi189 ceph-mon[36790]: pgmap v9883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17339.862972 secs 2024-07-26T16:16:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:22 smithi193 ceph-mon[37473]: pgmap v9883: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17339.862972 secs 2024-07-26T16:16:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:24 smithi134 ceph-mon[31966]: pgmap v9884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17325.441641 secs 2024-07-26T16:16:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:24 smithi189 ceph-mon[36790]: pgmap v9884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17325.441641 secs 2024-07-26T16:16:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:24 smithi193 ceph-mon[37473]: pgmap v9884: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17325.441641 secs 2024-07-26T16:16:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:26 smithi134 ceph-mon[31966]: pgmap v9885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:16:26.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:26 smithi189 ceph-mon[36790]: pgmap v9885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:16:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:26 smithi193 ceph-mon[37473]: pgmap v9885: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:16:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:16:27.259+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:16:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:16:27.259+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:16:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:16:27.259+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:16:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17344.863097 secs 2024-07-26T16:16:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17344.863097 secs 2024-07-26T16:16:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17344.863097 secs 2024-07-26T16:16:28.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:28 smithi189 ceph-mon[36790]: pgmap v9886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:28 smithi193 ceph-mon[37473]: pgmap v9886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:28 smithi134 ceph-mon[31966]: pgmap v9886: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17330.441770 secs 2024-07-26T16:16:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:29 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:16:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17330.441770 secs 2024-07-26T16:16:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:29 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:16:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17330.441770 secs 2024-07-26T16:16:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:29 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:16:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:30 smithi193 ceph-mon[37473]: pgmap v9887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:16:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:16:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:16:30.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:16:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:30 smithi134 ceph-mon[31966]: pgmap v9887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:16:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:16:30.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:16:30.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:16:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:30 smithi189 ceph-mon[36790]: pgmap v9887: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:16:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:16:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:16:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:16:31.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:16:31.647+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:16:31.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:16:31.647+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:16:31.649 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:16:31.647+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:16:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:31 smithi193 ceph-mon[37473]: pgmap v9888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:16:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:31 smithi193 ceph-mon[37473]: pgmap v9889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:16:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:31 smithi134 ceph-mon[31966]: pgmap v9888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:16:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:31 smithi134 ceph-mon[31966]: pgmap v9889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:16:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:31 smithi189 ceph-mon[36790]: pgmap v9888: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:16:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:31 smithi189 ceph-mon[36790]: pgmap v9889: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:16:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17349.863254 secs 2024-07-26T16:16:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17349.863254 secs 2024-07-26T16:16:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17349.863254 secs 2024-07-26T16:16:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:33 smithi193 ceph-mon[37473]: pgmap v9890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.0 KiB/s rd, 5 op/s 2024-07-26T16:16:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:33 smithi134 ceph-mon[31966]: pgmap v9890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.0 KiB/s rd, 5 op/s 2024-07-26T16:16:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:33 smithi189 ceph-mon[36790]: pgmap v9890: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.0 KiB/s rd, 5 op/s 2024-07-26T16:16:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17335.441901 secs 2024-07-26T16:16:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17335.441901 secs 2024-07-26T16:16:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17335.441901 secs 2024-07-26T16:16:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:35 smithi193 ceph-mon[37473]: pgmap v9891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:16:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:35 smithi134 ceph-mon[31966]: pgmap v9891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:16:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:35 smithi189 ceph-mon[36790]: pgmap v9891: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:16:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17354.863384 secs 2024-07-26T16:16:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17354.863384 secs 2024-07-26T16:16:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17354.863384 secs 2024-07-26T16:16:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:37 smithi193 ceph-mon[37473]: pgmap v9892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:16:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:37 smithi134 ceph-mon[31966]: pgmap v9892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:16:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:37 smithi189 ceph-mon[36790]: pgmap v9892: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:16:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:39 smithi193 ceph-mon[37473]: pgmap v9893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:16:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17340.442028 secs 2024-07-26T16:16:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:39 smithi134 ceph-mon[31966]: pgmap v9893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:16:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17340.442028 secs 2024-07-26T16:16:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:39 smithi189 ceph-mon[36790]: pgmap v9893: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:16:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17340.442028 secs 2024-07-26T16:16:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:41 smithi193 ceph-mon[37473]: pgmap v9894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:16:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17359.863525 secs 2024-07-26T16:16:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:41 smithi134 ceph-mon[31966]: pgmap v9894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:16:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17359.863525 secs 2024-07-26T16:16:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:41 smithi189 ceph-mon[36790]: pgmap v9894: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:16:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17359.863525 secs 2024-07-26T16:16:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:16:42.274+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:16:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:16:42.274+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:16:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:16:42.274+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:16:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:43 smithi193 ceph-mon[37473]: pgmap v9895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:43 smithi134 ceph-mon[31966]: pgmap v9895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:43 smithi189 ceph-mon[36790]: pgmap v9895: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17345.442159 secs 2024-07-26T16:16:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:44 smithi134 ceph-mon[31966]: pgmap v9896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17345.442159 secs 2024-07-26T16:16:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:44 smithi189 ceph-mon[36790]: pgmap v9896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17345.442159 secs 2024-07-26T16:16:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:44 smithi193 ceph-mon[37473]: pgmap v9896: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:46.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:16:46.656+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:16:46.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:16:46.656+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:16:46.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:16:46.656+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:16:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:47 smithi134 ceph-mon[31966]: pgmap v9897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:47.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17364.863654 secs 2024-07-26T16:16:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:47 smithi189 ceph-mon[36790]: pgmap v9897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17364.863654 secs 2024-07-26T16:16:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:47 smithi193 ceph-mon[37473]: pgmap v9897: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17364.863654 secs 2024-07-26T16:16:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:49 smithi134 ceph-mon[31966]: pgmap v9898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:49.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17350.442292 secs 2024-07-26T16:16:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:49 smithi189 ceph-mon[36790]: pgmap v9898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17350.442292 secs 2024-07-26T16:16:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:49 smithi193 ceph-mon[37473]: pgmap v9898: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17350.442292 secs 2024-07-26T16:16:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:51 smithi134 ceph-mon[31966]: pgmap v9899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:51.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:51 smithi189 ceph-mon[36790]: pgmap v9899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:51 smithi193 ceph-mon[37473]: pgmap v9899: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17369.863800 secs 2024-07-26T16:16:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17369.863800 secs 2024-07-26T16:16:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17369.863800 secs 2024-07-26T16:16:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:53 smithi134 ceph-mon[31966]: pgmap v9900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:53.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:53 smithi189 ceph-mon[36790]: pgmap v9900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:53 smithi193 ceph-mon[37473]: pgmap v9900: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17355.442416 secs 2024-07-26T16:16:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17355.442416 secs 2024-07-26T16:16:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17355.442416 secs 2024-07-26T16:16:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:55 smithi189 ceph-mon[36790]: pgmap v9901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:55 smithi193 ceph-mon[37473]: pgmap v9901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:55 smithi134 ceph-mon[31966]: pgmap v9901: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:57.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:16:57.290+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:16:57.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:16:57.290+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:16:57.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:16:57.290+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:16:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:57 smithi134 ceph-mon[31966]: pgmap v9902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17374.863884 secs 2024-07-26T16:16:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:57 smithi189 ceph-mon[36790]: pgmap v9902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17374.863884 secs 2024-07-26T16:16:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:57 smithi193 ceph-mon[37473]: pgmap v9902: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:16:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17374.863884 secs 2024-07-26T16:16:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:59 smithi189 ceph-mon[36790]: pgmap v9903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:16:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17360.442531 secs 2024-07-26T16:16:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:59 smithi193 ceph-mon[37473]: pgmap v9903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:16:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17360.442531 secs 2024-07-26T16:16:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:59 smithi134 ceph-mon[31966]: pgmap v9903: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:16:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:16:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17360.442531 secs 2024-07-26T16:17:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:01 smithi189 ceph-mon[36790]: pgmap v9904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:01 smithi193 ceph-mon[37473]: pgmap v9904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:01.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:17:01.658+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:17:01.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:17:01.658+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:01.661 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:17:01.658+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:17:01.661 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:01 smithi134 ceph-mon[31966]: pgmap v9904: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17379.864029 secs 2024-07-26T16:17:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17379.864029 secs 2024-07-26T16:17:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17379.864029 secs 2024-07-26T16:17:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:03 smithi193 ceph-mon[37473]: pgmap v9905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:03 smithi134 ceph-mon[31966]: pgmap v9905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:03 smithi189 ceph-mon[36790]: pgmap v9905: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17365.442623 secs 2024-07-26T16:17:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17365.442623 secs 2024-07-26T16:17:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17365.442623 secs 2024-07-26T16:17:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:05 smithi193 ceph-mon[37473]: pgmap v9906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:05 smithi134 ceph-mon[31966]: pgmap v9906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:05.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:05 smithi189 ceph-mon[36790]: pgmap v9906: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:07 smithi193 ceph-mon[37473]: pgmap v9907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17384.864153 secs 2024-07-26T16:17:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:07 smithi134 ceph-mon[31966]: pgmap v9907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17384.864153 secs 2024-07-26T16:17:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:07 smithi189 ceph-mon[36790]: pgmap v9907: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17384.864153 secs 2024-07-26T16:17:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:09 smithi193 ceph-mon[37473]: pgmap v9908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17370.442730 secs 2024-07-26T16:17:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:09 smithi134 ceph-mon[31966]: pgmap v9908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17370.442730 secs 2024-07-26T16:17:09.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:09 smithi189 ceph-mon[36790]: pgmap v9908: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17370.442730 secs 2024-07-26T16:17:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:11 smithi193 ceph-mon[37473]: pgmap v9909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:11 smithi134 ceph-mon[31966]: pgmap v9909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:11 smithi189 ceph-mon[36790]: pgmap v9909: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:17:12.299+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:17:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:17:12.299+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:17:12.299+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:17:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17389.864280 secs 2024-07-26T16:17:12.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17389.864280 secs 2024-07-26T16:17:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17389.864280 secs 2024-07-26T16:17:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:13 smithi193 ceph-mon[37473]: pgmap v9910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:13 smithi134 ceph-mon[31966]: pgmap v9910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:13.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:13 smithi189 ceph-mon[36790]: pgmap v9910: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17375.442891 secs 2024-07-26T16:17:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17375.442891 secs 2024-07-26T16:17:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17375.442891 secs 2024-07-26T16:17:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:15 smithi193 ceph-mon[37473]: pgmap v9911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:15 smithi134 ceph-mon[31966]: pgmap v9911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:15 smithi189 ceph-mon[36790]: pgmap v9911: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:16.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:17:16.667+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:17:16.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:17:16.667+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:17:16.669 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:17:16.667+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:17 smithi193 ceph-mon[37473]: pgmap v9912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17394.864383 secs 2024-07-26T16:17:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:17 smithi134 ceph-mon[31966]: pgmap v9912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17394.864383 secs 2024-07-26T16:17:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:17 smithi189 ceph-mon[36790]: pgmap v9912: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17394.864383 secs 2024-07-26T16:17:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:19 smithi134 ceph-mon[31966]: pgmap v9913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17380.442982 secs 2024-07-26T16:17:19.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:19 smithi189 ceph-mon[36790]: pgmap v9913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17380.442982 secs 2024-07-26T16:17:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:19 smithi193 ceph-mon[37473]: pgmap v9913: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17380.442982 secs 2024-07-26T16:17:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:21 smithi134 ceph-mon[31966]: pgmap v9914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:21.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:21 smithi189 ceph-mon[36790]: pgmap v9914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:21 smithi193 ceph-mon[37473]: pgmap v9914: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17399.864513 secs 2024-07-26T16:17:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17399.864513 secs 2024-07-26T16:17:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17399.864513 secs 2024-07-26T16:17:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:23 smithi134 ceph-mon[31966]: pgmap v9915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:23 smithi189 ceph-mon[36790]: pgmap v9915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:23 smithi193 ceph-mon[37473]: pgmap v9915: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:24.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17385.443067 secs 2024-07-26T16:17:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17385.443067 secs 2024-07-26T16:17:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17385.443067 secs 2024-07-26T16:17:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:25 smithi134 ceph-mon[31966]: pgmap v9916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:25.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:25 smithi189 ceph-mon[36790]: pgmap v9916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:25 smithi193 ceph-mon[37473]: pgmap v9916: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:27.314 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:17:27.312+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:17:27.314 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:17:27.312+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:27.314 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:17:27.312+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:17:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:27 smithi134 ceph-mon[31966]: pgmap v9917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17404.864642 secs 2024-07-26T16:17:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:27 smithi189 ceph-mon[36790]: pgmap v9917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17404.864642 secs 2024-07-26T16:17:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:27 smithi193 ceph-mon[37473]: pgmap v9917: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17404.864642 secs 2024-07-26T16:17:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:29 smithi134 ceph-mon[31966]: pgmap v9918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17390.443149 secs 2024-07-26T16:17:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:29 smithi189 ceph-mon[36790]: pgmap v9918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17390.443149 secs 2024-07-26T16:17:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:29 smithi193 ceph-mon[37473]: pgmap v9918: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17390.443149 secs 2024-07-26T16:17:30.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:30 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:17:30.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:30 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:17:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:30 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:17:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:31 smithi134 ceph-mon[31966]: pgmap v9919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:31.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:31 smithi193 ceph-mon[37473]: pgmap v9919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:31.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:17:31.682+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:17:31.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:17:31.682+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:31.684 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:17:31.682+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:17:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:31 smithi189 ceph-mon[36790]: pgmap v9919: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:32.582 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17409.864762 secs 2024-07-26T16:17:32.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17409.864762 secs 2024-07-26T16:17:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17409.864762 secs 2024-07-26T16:17:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:33 smithi134 ceph-mon[31966]: pgmap v9920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:33.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:33 smithi189 ceph-mon[36790]: pgmap v9920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:33 smithi193 ceph-mon[37473]: pgmap v9920: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17395.443278 secs 2024-07-26T16:17:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17395.443278 secs 2024-07-26T16:17:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17395.443278 secs 2024-07-26T16:17:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:35 smithi134 ceph-mon[31966]: pgmap v9921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:35.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:35 smithi189 ceph-mon[36790]: pgmap v9921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:35 smithi193 ceph-mon[37473]: pgmap v9921: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:37.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:37 smithi134 ceph-mon[31966]: pgmap v9922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:37.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17414.864896 secs 2024-07-26T16:17:37.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:17:37.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:17:37.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:17:37.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:17:37.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:17:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:37 smithi189 ceph-mon[36790]: pgmap v9922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:37.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17414.864896 secs 2024-07-26T16:17:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:17:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:17:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:17:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:17:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:17:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:37 smithi193 ceph-mon[37473]: pgmap v9922: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17414.864896 secs 2024-07-26T16:17:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:17:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:17:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:17:37.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:17:37.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:17:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:38 smithi134 ceph-mon[31966]: pgmap v9923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:17:38.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:38 smithi134 ceph-mon[31966]: pgmap v9924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T16:17:38.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:38 smithi189 ceph-mon[36790]: pgmap v9923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:17:38.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:38 smithi189 ceph-mon[36790]: pgmap v9924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T16:17:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:38 smithi193 ceph-mon[37473]: pgmap v9923: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:17:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:38 smithi193 ceph-mon[37473]: pgmap v9924: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T16:17:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17400.443371 secs 2024-07-26T16:17:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17400.443371 secs 2024-07-26T16:17:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17400.443371 secs 2024-07-26T16:17:40.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:40 smithi134 ceph-mon[31966]: pgmap v9925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:17:40.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:40 smithi189 ceph-mon[36790]: pgmap v9925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:17:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:40 smithi193 ceph-mon[37473]: pgmap v9925: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:17:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:17:42.319+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:17:42.321 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:17:42.320+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:17:42.322 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:17:42.320+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:42 smithi134 ceph-mon[31966]: pgmap v9926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T16:17:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17419.865028 secs 2024-07-26T16:17:42.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:42 smithi189 ceph-mon[36790]: pgmap v9926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T16:17:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17419.865028 secs 2024-07-26T16:17:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:42 smithi193 ceph-mon[37473]: pgmap v9926: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T16:17:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17419.865028 secs 2024-07-26T16:17:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:44 smithi134 ceph-mon[31966]: pgmap v9927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T16:17:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17405.443500 secs 2024-07-26T16:17:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:44 smithi189 ceph-mon[36790]: pgmap v9927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T16:17:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17405.443500 secs 2024-07-26T16:17:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:44 smithi193 ceph-mon[37473]: pgmap v9927: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T16:17:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17405.443500 secs 2024-07-26T16:17:46.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:46 smithi134 ceph-mon[31966]: pgmap v9928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:17:46.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:17:46.688+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:17:46.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:17:46.688+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:17:46.691 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:17:46.688+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:46.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:46 smithi189 ceph-mon[36790]: pgmap v9928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:17:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:46 smithi193 ceph-mon[37473]: pgmap v9928: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:17:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17424.865168 secs 2024-07-26T16:17:47.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17424.865168 secs 2024-07-26T16:17:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17424.865168 secs 2024-07-26T16:17:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:48 smithi134 ceph-mon[31966]: pgmap v9929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:17:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:48 smithi189 ceph-mon[36790]: pgmap v9929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:17:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:48 smithi193 ceph-mon[37473]: pgmap v9929: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:17:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17410.443618 secs 2024-07-26T16:17:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17410.443618 secs 2024-07-26T16:17:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17410.443618 secs 2024-07-26T16:17:50.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:50 smithi189 ceph-mon[36790]: pgmap v9930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:50 smithi193 ceph-mon[37473]: pgmap v9930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:50 smithi134 ceph-mon[31966]: pgmap v9930: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:52 smithi189 ceph-mon[36790]: pgmap v9931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17429.865245 secs 2024-07-26T16:17:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:52 smithi193 ceph-mon[37473]: pgmap v9931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17429.865245 secs 2024-07-26T16:17:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:52 smithi134 ceph-mon[31966]: pgmap v9931: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17429.865245 secs 2024-07-26T16:17:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:54 smithi189 ceph-mon[36790]: pgmap v9932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17415.443746 secs 2024-07-26T16:17:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:54 smithi193 ceph-mon[37473]: pgmap v9932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17415.443746 secs 2024-07-26T16:17:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:54 smithi134 ceph-mon[31966]: pgmap v9932: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:17:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17415.443746 secs 2024-07-26T16:17:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:56 smithi193 ceph-mon[37473]: pgmap v9933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:17:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:56 smithi134 ceph-mon[31966]: pgmap v9933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:17:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:56 smithi189 ceph-mon[36790]: pgmap v9933: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:17:57.330 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:17:57.328+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:17:57.330 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:17:57.328+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:17:57.330 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:17:57.328+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:17:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17434.865388 secs 2024-07-26T16:17:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17434.865388 secs 2024-07-26T16:17:57.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17434.865388 secs 2024-07-26T16:17:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:58 smithi193 ceph-mon[37473]: pgmap v9934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:58 smithi134 ceph-mon[31966]: pgmap v9934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:58.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:58 smithi189 ceph-mon[36790]: pgmap v9934: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:17:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:17:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17420.443878 secs 2024-07-26T16:17:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:17:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17420.443878 secs 2024-07-26T16:17:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:17:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17420.443878 secs 2024-07-26T16:18:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:00 smithi193 ceph-mon[37473]: pgmap v9935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:00 smithi134 ceph-mon[31966]: pgmap v9935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:00 smithi189 ceph-mon[36790]: pgmap v9935: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:01.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:18:01.704+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:18:01.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:18:01.704+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:18:01.706 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:18:01.704+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:18:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17439.865474 secs 2024-07-26T16:18:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17439.865474 secs 2024-07-26T16:18:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17439.865474 secs 2024-07-26T16:18:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:02 smithi193 ceph-mon[37473]: pgmap v9936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:02 smithi134 ceph-mon[31966]: pgmap v9936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:02 smithi189 ceph-mon[36790]: pgmap v9936: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:04 smithi193 ceph-mon[37473]: pgmap v9937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17425.444007 secs 2024-07-26T16:18:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:04 smithi134 ceph-mon[31966]: pgmap v9937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17425.444007 secs 2024-07-26T16:18:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:04 smithi189 ceph-mon[36790]: pgmap v9937: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17425.444007 secs 2024-07-26T16:18:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:06 smithi193 ceph-mon[37473]: pgmap v9938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:18:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17444.865597 secs 2024-07-26T16:18:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:06 smithi134 ceph-mon[31966]: pgmap v9938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:18:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17444.865597 secs 2024-07-26T16:18:06.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:06 smithi189 ceph-mon[36790]: pgmap v9938: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:18:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17444.865597 secs 2024-07-26T16:18:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:08 smithi193 ceph-mon[37473]: pgmap v9939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:08 smithi134 ceph-mon[31966]: pgmap v9939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:08 smithi189 ceph-mon[36790]: pgmap v9939: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17430.444110 secs 2024-07-26T16:18:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17430.444110 secs 2024-07-26T16:18:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17430.444110 secs 2024-07-26T16:18:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:10 smithi134 ceph-mon[31966]: pgmap v9940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:10 smithi189 ceph-mon[36790]: pgmap v9940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:10 smithi193 ceph-mon[37473]: pgmap v9940: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17449.865679 secs 2024-07-26T16:18:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17449.865679 secs 2024-07-26T16:18:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17449.865679 secs 2024-07-26T16:18:12.345 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:18:12.343+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:18:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:18:12.344+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:18:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:18:12.344+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:18:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:12 smithi134 ceph-mon[31966]: pgmap v9941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:12 smithi189 ceph-mon[36790]: pgmap v9941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:12 smithi193 ceph-mon[37473]: pgmap v9941: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:13 smithi134 ceph-mon[31966]: pgmap v9942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:13.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:13 smithi189 ceph-mon[36790]: pgmap v9942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:13 smithi193 ceph-mon[37473]: pgmap v9942: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17435.444244 secs 2024-07-26T16:18:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17435.444244 secs 2024-07-26T16:18:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17435.444244 secs 2024-07-26T16:18:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:15 smithi134 ceph-mon[31966]: pgmap v9943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:18:15.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:15 smithi189 ceph-mon[36790]: pgmap v9943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:18:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:15 smithi193 ceph-mon[37473]: pgmap v9943: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:18:16.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:18:16.712+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:18:16.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:18:16.712+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:18:16.714 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:18:16.712+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:18:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17454.865808 secs 2024-07-26T16:18:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17454.865808 secs 2024-07-26T16:18:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17454.865808 secs 2024-07-26T16:18:17.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:17 smithi134 ceph-mon[31966]: pgmap v9944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:17 smithi189 ceph-mon[36790]: pgmap v9944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:17 smithi193 ceph-mon[37473]: pgmap v9944: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:19 smithi189 ceph-mon[36790]: pgmap v9945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17440.444364 secs 2024-07-26T16:18:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:19 smithi193 ceph-mon[37473]: pgmap v9945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17440.444364 secs 2024-07-26T16:18:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:19 smithi134 ceph-mon[31966]: pgmap v9945: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17440.444364 secs 2024-07-26T16:18:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:21 smithi189 ceph-mon[36790]: pgmap v9946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17459.865929 secs 2024-07-26T16:18:22.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:21 smithi193 ceph-mon[37473]: pgmap v9946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:22.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17459.865929 secs 2024-07-26T16:18:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:21 smithi134 ceph-mon[31966]: pgmap v9946: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17459.865929 secs 2024-07-26T16:18:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:23 smithi193 ceph-mon[37473]: pgmap v9947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:23 smithi134 ceph-mon[31966]: pgmap v9947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:24.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:23 smithi189 ceph-mon[36790]: pgmap v9947: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17445.444460 secs 2024-07-26T16:18:25.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17445.444460 secs 2024-07-26T16:18:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17445.444460 secs 2024-07-26T16:18:25.360 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:18:25.359+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T16:18:25.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:18:25.359+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T16:18:25.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:18:25.359+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x7f9b4c10f330 2024-07-26T16:18:25.361 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:18:25.359+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 136 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x55c65e5ee670 2024-07-26T16:18:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:25 smithi134 ceph-mon[31966]: pgmap v9948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:18:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:25 smithi193 ceph-mon[37473]: pgmap v9948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:18:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:25 smithi189 ceph-mon[36790]: pgmap v9948: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:18:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17464.866053 secs 2024-07-26T16:18:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17464.866053 secs 2024-07-26T16:18:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17464.866053 secs 2024-07-26T16:18:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:18:27.359+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:18:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:18:27.359+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:18:27.361 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:18:27.359+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:18:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:27 smithi193 ceph-mon[37473]: pgmap v9949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:27 smithi134 ceph-mon[31966]: pgmap v9949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:27 smithi189 ceph-mon[36790]: pgmap v9949: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:29 smithi193 ceph-mon[37473]: pgmap v9950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17450.444584 secs 2024-07-26T16:18:30.153 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:29 smithi134 ceph-mon[31966]: pgmap v9950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:30.153 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17450.444584 secs 2024-07-26T16:18:30.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:29 smithi189 ceph-mon[36790]: pgmap v9950: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:30.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17450.444584 secs 2024-07-26T16:18:31.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:18:31.728+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:18:31.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:18:31.728+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:18:31.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:18:31.728+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:18:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:31 smithi193 ceph-mon[37473]: pgmap v9951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17469.866174 secs 2024-07-26T16:18:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:31 smithi134 ceph-mon[31966]: pgmap v9951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:32.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17469.866174 secs 2024-07-26T16:18:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:31 smithi189 ceph-mon[36790]: pgmap v9951: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17469.866174 secs 2024-07-26T16:18:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:33 smithi193 ceph-mon[37473]: pgmap v9952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:33 smithi134 ceph-mon[31966]: pgmap v9952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:33 smithi189 ceph-mon[36790]: pgmap v9952: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17455.444703 secs 2024-07-26T16:18:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17455.444703 secs 2024-07-26T16:18:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17455.444703 secs 2024-07-26T16:18:35.914 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:18:35.913+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:18:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:18:35.913+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:18:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:18:35.913+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:18:35.915 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:18:35.913+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:18:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:35 smithi193 ceph-mon[37473]: pgmap v9953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:18:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:35 smithi134 ceph-mon[31966]: pgmap v9953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:18:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:35 smithi189 ceph-mon[36790]: pgmap v9953: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:18:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17474.866295 secs 2024-07-26T16:18:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17474.866295 secs 2024-07-26T16:18:37.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17474.866295 secs 2024-07-26T16:18:38.016 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:37 smithi189 ceph-mon[36790]: pgmap v9954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:38.017 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:37 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:18:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:37 smithi193 ceph-mon[37473]: pgmap v9954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:37 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:18:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:37 smithi134 ceph-mon[31966]: pgmap v9954: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:37 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:18:39.018 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:39.018 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:39.018 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:39.018 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:38 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:39.018 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:38 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T16:18:39.018 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:38 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T16:18:39.135 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:39.135 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:39.135 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:39.135 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:38 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:39.135 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:38 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T16:18:39.135 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:38 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T16:18:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:38 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:18:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:38 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T16:18:39.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:38 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T16:18:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:39 smithi193 ceph-mon[37473]: pgmap v9955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:40.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17460.444799 secs 2024-07-26T16:18:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:39 smithi134 ceph-mon[31966]: pgmap v9955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17460.444799 secs 2024-07-26T16:18:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:39 smithi189 ceph-mon[36790]: pgmap v9955: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17460.444799 secs 2024-07-26T16:18:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:41 smithi189 ceph-mon[36790]: pgmap v9956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:18:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:18:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:18:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:18:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:18:41.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:18:41.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:41 smithi189 ceph-mon[36790]: pgmap v9957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:18:41.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:41 smithi189 ceph-mon[36790]: pgmap v9958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T16:18:41.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:18:41.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17479.866384 secs 2024-07-26T16:18:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:41 smithi193 ceph-mon[37473]: pgmap v9956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:18:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:18:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:18:42.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:18:42.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:18:42.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:18:42.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:41 smithi193 ceph-mon[37473]: pgmap v9957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:18:42.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:41 smithi193 ceph-mon[37473]: pgmap v9958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T16:18:42.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:18:42.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17479.866384 secs 2024-07-26T16:18:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:41 smithi134 ceph-mon[31966]: pgmap v9956: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:18:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:18:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:18:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:18:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:18:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:18:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:41 smithi134 ceph-mon[31966]: pgmap v9957: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-07-26T16:18:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:41 smithi134 ceph-mon[31966]: pgmap v9958: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T16:18:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:18:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17479.866384 secs 2024-07-26T16:18:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:18:42.375+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:18:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:18:42.375+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:18:42.377 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:18:42.375+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:18:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:44 smithi134 ceph-mon[31966]: pgmap v9959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T16:18:44.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17465.444928 secs 2024-07-26T16:18:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:44 smithi189 ceph-mon[36790]: pgmap v9959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T16:18:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17465.444928 secs 2024-07-26T16:18:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:44 smithi193 ceph-mon[37473]: pgmap v9959: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-07-26T16:18:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17465.444928 secs 2024-07-26T16:18:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:46 smithi134 ceph-mon[31966]: pgmap v9960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:18:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:46 smithi189 ceph-mon[36790]: pgmap v9960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:18:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:46 smithi193 ceph-mon[37473]: pgmap v9960: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:18:46.746 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:18:46.743+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:18:46.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:18:46.743+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:18:46.747 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:18:46.743+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:18:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17484.866473 secs 2024-07-26T16:18:47.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17484.866473 secs 2024-07-26T16:18:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17484.866473 secs 2024-07-26T16:18:48.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:48 smithi134 ceph-mon[31966]: pgmap v9961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T16:18:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:48 smithi189 ceph-mon[36790]: pgmap v9961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T16:18:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:48 smithi193 ceph-mon[37473]: pgmap v9961: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-07-26T16:18:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17470.445022 secs 2024-07-26T16:18:49.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17470.445022 secs 2024-07-26T16:18:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17470.445022 secs 2024-07-26T16:18:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:50 smithi134 ceph-mon[31966]: pgmap v9962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:18:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:50 smithi189 ceph-mon[36790]: pgmap v9962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:18:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:50 smithi193 ceph-mon[37473]: pgmap v9962: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-07-26T16:18:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:52 smithi134 ceph-mon[31966]: pgmap v9963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:18:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17489.866601 secs 2024-07-26T16:18:52.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:52 smithi189 ceph-mon[36790]: pgmap v9963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:18:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17489.866601 secs 2024-07-26T16:18:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:52 smithi193 ceph-mon[37473]: pgmap v9963: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:18:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17489.866601 secs 2024-07-26T16:18:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:54 smithi134 ceph-mon[31966]: pgmap v9964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17475.445171 secs 2024-07-26T16:18:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:54 smithi189 ceph-mon[36790]: pgmap v9964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17475.445171 secs 2024-07-26T16:18:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:54 smithi193 ceph-mon[37473]: pgmap v9964: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:18:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17475.445171 secs 2024-07-26T16:18:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:56 smithi134 ceph-mon[31966]: pgmap v9965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:56 smithi189 ceph-mon[36790]: pgmap v9965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:56 smithi193 ceph-mon[37473]: pgmap v9965: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:18:57.376+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:18:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:18:57.376+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:18:57.378 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:18:57.376+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:18:57.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17494.866734 secs 2024-07-26T16:18:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17494.866734 secs 2024-07-26T16:18:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17494.866734 secs 2024-07-26T16:18:58.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:58 smithi189 ceph-mon[36790]: pgmap v9966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:58 smithi193 ceph-mon[37473]: pgmap v9966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:58 smithi134 ceph-mon[31966]: pgmap v9966: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:18:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:18:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17480.445294 secs 2024-07-26T16:18:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:18:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17480.445294 secs 2024-07-26T16:18:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:18:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17480.445294 secs 2024-07-26T16:19:00.526 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:00 smithi193 ceph-mon[37473]: pgmap v9967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:00 smithi134 ceph-mon[31966]: pgmap v9967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:00 smithi189 ceph-mon[36790]: pgmap v9967: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:01.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:19:01.752+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:19:01.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:19:01.752+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:19:01.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:19:01.752+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:19:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:02 smithi193 ceph-mon[37473]: pgmap v9968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17499.866803 secs 2024-07-26T16:19:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:02 smithi134 ceph-mon[31966]: pgmap v9968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17499.866803 secs 2024-07-26T16:19:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:02 smithi189 ceph-mon[36790]: pgmap v9968: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17499.866803 secs 2024-07-26T16:19:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:04 smithi193 ceph-mon[37473]: pgmap v9969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17485.445425 secs 2024-07-26T16:19:04.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:04 smithi134 ceph-mon[31966]: pgmap v9969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:04.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17485.445425 secs 2024-07-26T16:19:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:04 smithi189 ceph-mon[36790]: pgmap v9969: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17485.445425 secs 2024-07-26T16:19:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:06 smithi193 ceph-mon[37473]: pgmap v9970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:06 smithi134 ceph-mon[31966]: pgmap v9970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:06 smithi189 ceph-mon[36790]: pgmap v9970: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17504.866917 secs 2024-07-26T16:19:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17504.866917 secs 2024-07-26T16:19:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17504.866917 secs 2024-07-26T16:19:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:08 smithi193 ceph-mon[37473]: pgmap v9971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:08 smithi134 ceph-mon[31966]: pgmap v9971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:08 smithi189 ceph-mon[36790]: pgmap v9971: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17490.445548 secs 2024-07-26T16:19:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17490.445548 secs 2024-07-26T16:19:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17490.445548 secs 2024-07-26T16:19:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:10 smithi193 ceph-mon[37473]: pgmap v9972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:10.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:10 smithi134 ceph-mon[31966]: pgmap v9972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:10 smithi189 ceph-mon[36790]: pgmap v9972: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:12.393 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:19:12.392+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:19:12.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:19:12.392+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:19:12.394 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:19:12.392+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:19:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:12 smithi193 ceph-mon[37473]: pgmap v9973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17509.867046 secs 2024-07-26T16:19:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:12 smithi134 ceph-mon[31966]: pgmap v9973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:12.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17509.867046 secs 2024-07-26T16:19:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:12 smithi189 ceph-mon[36790]: pgmap v9973: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17509.867046 secs 2024-07-26T16:19:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:14 smithi193 ceph-mon[37473]: pgmap v9974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17495.445687 secs 2024-07-26T16:19:14.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:14 smithi134 ceph-mon[31966]: pgmap v9974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:14.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17495.445687 secs 2024-07-26T16:19:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:14 smithi189 ceph-mon[36790]: pgmap v9974: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17495.445687 secs 2024-07-26T16:19:16.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:16 smithi193 ceph-mon[37473]: pgmap v9975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:16.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:16 smithi134 ceph-mon[31966]: pgmap v9975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:16.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:16 smithi189 ceph-mon[36790]: pgmap v9975: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:16.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:19:16.761+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:19:16.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:19:16.761+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:19:16.763 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:19:16.761+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:19:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17514.867148 secs 2024-07-26T16:19:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17514.867148 secs 2024-07-26T16:19:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17514.867148 secs 2024-07-26T16:19:18.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:18 smithi189 ceph-mon[36790]: pgmap v9976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:18 smithi193 ceph-mon[37473]: pgmap v9976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:18 smithi134 ceph-mon[31966]: pgmap v9976: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17500.445813 secs 2024-07-26T16:19:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17500.445813 secs 2024-07-26T16:19:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17500.445813 secs 2024-07-26T16:19:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:20 smithi193 ceph-mon[37473]: pgmap v9977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:20 smithi134 ceph-mon[31966]: pgmap v9977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:20.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:20 smithi189 ceph-mon[36790]: pgmap v9977: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:22 smithi193 ceph-mon[37473]: pgmap v9978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17519.867277 secs 2024-07-26T16:19:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:22 smithi134 ceph-mon[31966]: pgmap v9978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17519.867277 secs 2024-07-26T16:19:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:22 smithi189 ceph-mon[36790]: pgmap v9978: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17519.867277 secs 2024-07-26T16:19:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:24 smithi193 ceph-mon[37473]: pgmap v9979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17505.445965 secs 2024-07-26T16:19:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:24 smithi134 ceph-mon[31966]: pgmap v9979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17505.445965 secs 2024-07-26T16:19:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:24 smithi189 ceph-mon[36790]: pgmap v9979: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17505.445965 secs 2024-07-26T16:19:26.746 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:26 smithi189 ceph-mon[36790]: pgmap v9980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:26 smithi193 ceph-mon[37473]: pgmap v9980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:26 smithi134 ceph-mon[31966]: pgmap v9980: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:27.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:19:27.407+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:19:27.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:19:27.407+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:19:27.409 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:19:27.407+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:19:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17524.867397 secs 2024-07-26T16:19:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17524.867397 secs 2024-07-26T16:19:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17524.867397 secs 2024-07-26T16:19:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:28 smithi193 ceph-mon[37473]: pgmap v9981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:28 smithi134 ceph-mon[31966]: pgmap v9981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:28 smithi189 ceph-mon[36790]: pgmap v9981: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17510.446070 secs 2024-07-26T16:19:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17510.446070 secs 2024-07-26T16:19:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17510.446070 secs 2024-07-26T16:19:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:30 smithi193 ceph-mon[37473]: pgmap v9982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:30 smithi134 ceph-mon[31966]: pgmap v9982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:30 smithi189 ceph-mon[36790]: pgmap v9982: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17529.867482 secs 2024-07-26T16:19:31.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:19:31.775+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:19:31.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:19:31.775+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:19:31.777 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:19:31.775+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:19:31.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17529.867482 secs 2024-07-26T16:19:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17529.867482 secs 2024-07-26T16:19:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:32 smithi193 ceph-mon[37473]: pgmap v9983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:32 smithi134 ceph-mon[31966]: pgmap v9983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:32 smithi189 ceph-mon[36790]: pgmap v9983: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:34.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:34 smithi193 ceph-mon[37473]: pgmap v9984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:34.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17515.446193 secs 2024-07-26T16:19:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:34 smithi134 ceph-mon[31966]: pgmap v9984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17515.446193 secs 2024-07-26T16:19:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:34 smithi189 ceph-mon[36790]: pgmap v9984: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17515.446193 secs 2024-07-26T16:19:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:36 smithi134 ceph-mon[31966]: pgmap v9985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17534.867598 secs 2024-07-26T16:19:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:36 smithi189 ceph-mon[36790]: pgmap v9985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17534.867598 secs 2024-07-26T16:19:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:36 smithi193 ceph-mon[37473]: pgmap v9985: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17534.867598 secs 2024-07-26T16:19:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:38 smithi134 ceph-mon[31966]: pgmap v9986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:38.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:38 smithi189 ceph-mon[36790]: pgmap v9986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:38 smithi193 ceph-mon[37473]: pgmap v9986: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17520.446323 secs 2024-07-26T16:19:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17520.446323 secs 2024-07-26T16:19:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17520.446323 secs 2024-07-26T16:19:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:40 smithi134 ceph-mon[31966]: pgmap v9987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:40 smithi189 ceph-mon[36790]: pgmap v9987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:40 smithi193 ceph-mon[37473]: pgmap v9987: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17539.867724 secs 2024-07-26T16:19:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:41 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:19:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17539.867724 secs 2024-07-26T16:19:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:41 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:19:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17539.867724 secs 2024-07-26T16:19:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:41 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:19:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:19:42.423+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:19:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:19:42.423+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:19:42.425 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:19:42.423+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:19:42.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:42 smithi134 ceph-mon[31966]: pgmap v9988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:42.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:42.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:42.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:42.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:42.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:42.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:42.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:42.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:42.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:19:42.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:19:42.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:19:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:42 smithi189 ceph-mon[36790]: pgmap v9988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:19:42.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:19:42.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:19:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:42 smithi193 ceph-mon[37473]: pgmap v9988: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:43.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:43.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:43.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:43.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:43.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:43.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:19:43.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:19:43.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:19:43.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:19:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:43 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T16:19:43.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:43 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T16:19:43.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:43 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T16:19:43.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:43 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T16:19:43.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:43 smithi134 ceph-mon[31966]: pgmap v9989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T16:19:43.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:43 smithi134 ceph-mon[31966]: pgmap v9990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T16:19:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:43 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T16:19:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:43 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T16:19:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:43 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T16:19:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:43 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T16:19:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:43 smithi189 ceph-mon[36790]: pgmap v9989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T16:19:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:43 smithi189 ceph-mon[36790]: pgmap v9990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T16:19:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:43 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T16:19:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:43 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T16:19:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:43 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T16:19:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:43 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T16:19:44.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:43 smithi193 ceph-mon[37473]: pgmap v9989: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T16:19:44.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:43 smithi193 ceph-mon[37473]: pgmap v9990: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-07-26T16:19:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17525.446460 secs 2024-07-26T16:19:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:44 smithi134 ceph-mon[31966]: pgmap v9991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:19:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17525.446460 secs 2024-07-26T16:19:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:44 smithi189 ceph-mon[36790]: pgmap v9991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:19:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17525.446460 secs 2024-07-26T16:19:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:44 smithi193 ceph-mon[37473]: pgmap v9991: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:19:46.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:19:46.783+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:19:46.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:19:46.783+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:19:46.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:19:46.783+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:19:47.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17544.867808 secs 2024-07-26T16:19:47.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:47 smithi134 ceph-mon[31966]: pgmap v9992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:19:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17544.867808 secs 2024-07-26T16:19:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:47 smithi189 ceph-mon[36790]: pgmap v9992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:19:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17544.867808 secs 2024-07-26T16:19:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:47 smithi193 ceph-mon[37473]: pgmap v9992: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:19:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:49 smithi134 ceph-mon[31966]: pgmap v9993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:19:49.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17530.446550 secs 2024-07-26T16:19:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:49 smithi189 ceph-mon[36790]: pgmap v9993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:19:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17530.446550 secs 2024-07-26T16:19:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:49 smithi193 ceph-mon[37473]: pgmap v9993: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:19:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17530.446550 secs 2024-07-26T16:19:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:51 smithi134 ceph-mon[31966]: pgmap v9994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:19:51.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:51 smithi189 ceph-mon[36790]: pgmap v9994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:19:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:51 smithi193 ceph-mon[37473]: pgmap v9994: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:19:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17549.867889 secs 2024-07-26T16:19:52.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17549.867889 secs 2024-07-26T16:19:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17549.867889 secs 2024-07-26T16:19:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:53 smithi134 ceph-mon[31966]: pgmap v9995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:19:53.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:53 smithi189 ceph-mon[36790]: pgmap v9995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:19:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:53 smithi193 ceph-mon[37473]: pgmap v9995: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:19:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17535.446664 secs 2024-07-26T16:19:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17535.446664 secs 2024-07-26T16:19:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17535.446664 secs 2024-07-26T16:19:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:55 smithi134 ceph-mon[31966]: pgmap v9996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:19:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:55 smithi189 ceph-mon[36790]: pgmap v9996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:19:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:55 smithi193 ceph-mon[37473]: pgmap v9996: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:19:57.440 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:19:57.439+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:19:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:19:57.439+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:19:57.441 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:19:57.439+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:19:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17554.867988 secs 2024-07-26T16:19:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:57 smithi134 ceph-mon[31966]: pgmap v9997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17554.867988 secs 2024-07-26T16:19:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:57 smithi189 ceph-mon[36790]: pgmap v9997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17554.867988 secs 2024-07-26T16:19:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:57 smithi193 ceph-mon[37473]: pgmap v9997: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:19:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:59 smithi134 ceph-mon[31966]: pgmap v9998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:19:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17540.446797 secs 2024-07-26T16:19:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:59 smithi189 ceph-mon[36790]: pgmap v9998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:19:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17540.446797 secs 2024-07-26T16:19:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:59 smithi193 ceph-mon[37473]: pgmap v9998: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:19:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:19:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17540.446797 secs 2024-07-26T16:20:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T16:20:00.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T16:20:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T16:20:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:01 smithi134 ceph-mon[31966]: pgmap v9999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:01 smithi189 ceph-mon[36790]: pgmap v9999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:01 smithi193 ceph-mon[37473]: pgmap v9999: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:01.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:20:01.792+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:20:01.794 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:20:01.792+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:20:01.795 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:20:01.792+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:20:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17559.868088 secs 2024-07-26T16:20:02.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17559.868088 secs 2024-07-26T16:20:02.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17559.868088 secs 2024-07-26T16:20:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:03 smithi134 ceph-mon[31966]: pgmap v10000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:03 smithi189 ceph-mon[36790]: pgmap v10000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:03 smithi193 ceph-mon[37473]: pgmap v10000: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17545.446901 secs 2024-07-26T16:20:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17545.446901 secs 2024-07-26T16:20:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17545.446901 secs 2024-07-26T16:20:05.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:05 smithi189 ceph-mon[36790]: pgmap v10001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:05 smithi193 ceph-mon[37473]: pgmap v10001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:05 smithi134 ceph-mon[31966]: pgmap v10001: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17564.868214 secs 2024-07-26T16:20:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:07 smithi193 ceph-mon[37473]: pgmap v10002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17564.868214 secs 2024-07-26T16:20:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:07 smithi134 ceph-mon[31966]: pgmap v10002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17564.868214 secs 2024-07-26T16:20:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:07 smithi189 ceph-mon[36790]: pgmap v10002: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:09 smithi193 ceph-mon[37473]: pgmap v10003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:20:09.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17550.447038 secs 2024-07-26T16:20:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:09 smithi134 ceph-mon[31966]: pgmap v10003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:20:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17550.447038 secs 2024-07-26T16:20:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:09 smithi189 ceph-mon[36790]: pgmap v10003: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:20:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17550.447038 secs 2024-07-26T16:20:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:11 smithi193 ceph-mon[37473]: pgmap v10004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:11 smithi134 ceph-mon[31966]: pgmap v10004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:11 smithi189 ceph-mon[36790]: pgmap v10004: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:20:12.440+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:20:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:20:12.440+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:20:12.442 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:20:12.440+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:20:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17569.868333 secs 2024-07-26T16:20:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17569.868333 secs 2024-07-26T16:20:12.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17569.868333 secs 2024-07-26T16:20:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:13 smithi193 ceph-mon[37473]: pgmap v10005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:13 smithi134 ceph-mon[31966]: pgmap v10005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:13 smithi189 ceph-mon[36790]: pgmap v10005: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17555.447167 secs 2024-07-26T16:20:14.824 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17555.447167 secs 2024-07-26T16:20:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17555.447167 secs 2024-07-26T16:20:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:15 smithi193 ceph-mon[37473]: pgmap v10006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:15 smithi134 ceph-mon[31966]: pgmap v10006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:15.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:15 smithi189 ceph-mon[36790]: pgmap v10006: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:16.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:20:16.807+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:20:16.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:20:16.808+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:20:16.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:20:16.808+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:20:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17574.868453 secs 2024-07-26T16:20:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:17 smithi193 ceph-mon[37473]: pgmap v10007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17574.868453 secs 2024-07-26T16:20:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:17 smithi134 ceph-mon[31966]: pgmap v10007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17574.868453 secs 2024-07-26T16:20:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:17 smithi189 ceph-mon[36790]: pgmap v10007: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:19 smithi193 ceph-mon[37473]: pgmap v10008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:20:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17560.447265 secs 2024-07-26T16:20:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:19 smithi134 ceph-mon[31966]: pgmap v10008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:20:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17560.447265 secs 2024-07-26T16:20:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:19 smithi189 ceph-mon[36790]: pgmap v10008: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:20:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17560.447265 secs 2024-07-26T16:20:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:21 smithi193 ceph-mon[37473]: pgmap v10009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17579.868569 secs 2024-07-26T16:20:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:21 smithi134 ceph-mon[31966]: pgmap v10009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17579.868569 secs 2024-07-26T16:20:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:21 smithi189 ceph-mon[36790]: pgmap v10009: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17579.868569 secs 2024-07-26T16:20:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:23 smithi193 ceph-mon[37473]: pgmap v10010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:23 smithi134 ceph-mon[31966]: pgmap v10010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:23 smithi189 ceph-mon[36790]: pgmap v10010: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17565.447412 secs 2024-07-26T16:20:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17565.447412 secs 2024-07-26T16:20:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17565.447412 secs 2024-07-26T16:20:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:25 smithi134 ceph-mon[31966]: pgmap v10011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:25.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:25 smithi189 ceph-mon[36790]: pgmap v10011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:25 smithi193 ceph-mon[37473]: pgmap v10011: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17584.868685 secs 2024-07-26T16:20:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17584.868685 secs 2024-07-26T16:20:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17584.868685 secs 2024-07-26T16:20:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:20:27.442+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:20:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:20:27.442+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:20:27.444 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:20:27.442+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:20:27.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:27 smithi189 ceph-mon[36790]: pgmap v10012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:27 smithi134 ceph-mon[31966]: pgmap v10012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:27 smithi193 ceph-mon[37473]: pgmap v10012: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:29 smithi134 ceph-mon[31966]: pgmap v10013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:20:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17570.447542 secs 2024-07-26T16:20:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:29 smithi189 ceph-mon[36790]: pgmap v10013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:20:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17570.447542 secs 2024-07-26T16:20:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:29 smithi193 ceph-mon[37473]: pgmap v10013: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:20:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17570.447542 secs 2024-07-26T16:20:31.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:20:31.823+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:20:31.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:20:31.823+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:20:31.826 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:20:31.823+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:20:31.826 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:31 smithi134 ceph-mon[31966]: pgmap v10014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:31.826 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17589.868826 secs 2024-07-26T16:20:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:31 smithi189 ceph-mon[36790]: pgmap v10014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17589.868826 secs 2024-07-26T16:20:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:31 smithi193 ceph-mon[37473]: pgmap v10014: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17589.868826 secs 2024-07-26T16:20:33.827 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:33 smithi193 ceph-mon[37473]: pgmap v10015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:33 smithi134 ceph-mon[31966]: pgmap v10015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:33.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:33 smithi189 ceph-mon[36790]: pgmap v10015: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17575.447676 secs 2024-07-26T16:20:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17575.447676 secs 2024-07-26T16:20:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17575.447676 secs 2024-07-26T16:20:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:35 smithi134 ceph-mon[31966]: pgmap v10016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:35.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:35 smithi189 ceph-mon[36790]: pgmap v10016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:35 smithi193 ceph-mon[37473]: pgmap v10016: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:20:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17594.868952 secs 2024-07-26T16:20:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17594.868952 secs 2024-07-26T16:20:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17594.868952 secs 2024-07-26T16:20:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:37 smithi134 ceph-mon[31966]: pgmap v10017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:37 smithi189 ceph-mon[36790]: pgmap v10017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:37 smithi193 ceph-mon[37473]: pgmap v10017: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:38 smithi134 ceph-mon[31966]: pgmap v10018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:20:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:38 smithi189 ceph-mon[36790]: pgmap v10018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:20:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:38 smithi193 ceph-mon[37473]: pgmap v10018: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:20:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17580.447789 secs 2024-07-26T16:20:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17580.447789 secs 2024-07-26T16:20:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17580.447789 secs 2024-07-26T16:20:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:40 smithi134 ceph-mon[31966]: pgmap v10019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:40 smithi189 ceph-mon[36790]: pgmap v10019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:40 smithi193 ceph-mon[37473]: pgmap v10019: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17599.869071 secs 2024-07-26T16:20:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17599.869071 secs 2024-07-26T16:20:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17599.869071 secs 2024-07-26T16:20:42.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:20:42.458+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:20:42.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:20:42.458+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:20:42.460 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:20:42.458+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:20:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:42 smithi134 ceph-mon[31966]: pgmap v10020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:42 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:20:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:42 smithi189 ceph-mon[36790]: pgmap v10020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:42 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:20:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:42 smithi193 ceph-mon[37473]: pgmap v10020: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:20:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:42 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:20:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:20:43.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:20:43.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:43 smithi134 ceph-mon[31966]: pgmap v10021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:20:43.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:43 smithi134 ceph-mon[31966]: pgmap v10022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:20:43.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:43 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:20:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:20:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:20:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:43 smithi189 ceph-mon[36790]: pgmap v10021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:20:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:43 smithi189 ceph-mon[36790]: pgmap v10022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:20:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:43 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:20:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:20:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:20:44.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:43 smithi193 ceph-mon[37473]: pgmap v10021: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:20:44.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:43 smithi193 ceph-mon[37473]: pgmap v10022: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:20:44.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:43 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:20:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17585.447927 secs 2024-07-26T16:20:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17585.447927 secs 2024-07-26T16:20:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17585.447927 secs 2024-07-26T16:20:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:45 smithi134 ceph-mon[31966]: pgmap v10023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:20:45.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:45 smithi189 ceph-mon[36790]: pgmap v10023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:20:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:45 smithi193 ceph-mon[37473]: pgmap v10023: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:20:46.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:20:46.839+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:20:46.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:20:46.839+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:20:46.842 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:20:46.839+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:20:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17604.869193 secs 2024-07-26T16:20:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17604.869193 secs 2024-07-26T16:20:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17604.869193 secs 2024-07-26T16:20:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:47 smithi134 ceph-mon[31966]: pgmap v10024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:20:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:47 smithi189 ceph-mon[36790]: pgmap v10024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:20:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:47 smithi193 ceph-mon[37473]: pgmap v10024: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:20:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17590.448044 secs 2024-07-26T16:20:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17590.448044 secs 2024-07-26T16:20:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17590.448044 secs 2024-07-26T16:20:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:50 smithi193 ceph-mon[37473]: pgmap v10025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:20:50.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:50 smithi134 ceph-mon[31966]: pgmap v10025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:20:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:50 smithi189 ceph-mon[36790]: pgmap v10025: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:20:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17609.869323 secs 2024-07-26T16:20:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:52 smithi193 ceph-mon[37473]: pgmap v10026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:20:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17609.869323 secs 2024-07-26T16:20:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:52 smithi134 ceph-mon[31966]: pgmap v10026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:20:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17609.869323 secs 2024-07-26T16:20:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:52 smithi189 ceph-mon[36790]: pgmap v10026: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:20:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:54 smithi193 ceph-mon[37473]: pgmap v10027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:20:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17595.448201 secs 2024-07-26T16:20:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:54 smithi134 ceph-mon[31966]: pgmap v10027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:20:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17595.448201 secs 2024-07-26T16:20:54.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:54 smithi189 ceph-mon[36790]: pgmap v10027: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:20:54.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17595.448201 secs 2024-07-26T16:20:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:56 smithi193 ceph-mon[37473]: pgmap v10028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17614.869394 secs 2024-07-26T16:20:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:56 smithi134 ceph-mon[31966]: pgmap v10028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17614.869394 secs 2024-07-26T16:20:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:56 smithi189 ceph-mon[36790]: pgmap v10028: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:20:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17614.869394 secs 2024-07-26T16:20:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:20:57.471+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:20:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:20:57.471+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:20:57.473 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:20:57.471+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:20:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:58 smithi193 ceph-mon[37473]: pgmap v10029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:20:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:58 smithi134 ceph-mon[31966]: pgmap v10029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:20:58.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:58 smithi189 ceph-mon[36790]: pgmap v10029: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:20:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:20:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17600.448283 secs 2024-07-26T16:20:59.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:20:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17600.448283 secs 2024-07-26T16:21:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:20:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17600.448283 secs 2024-07-26T16:21:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:00 smithi134 ceph-mon[31966]: pgmap v10030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:00 smithi189 ceph-mon[36790]: pgmap v10030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:00 smithi193 ceph-mon[37473]: pgmap v10030: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:01.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:21:01.854+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:21:01.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:21:01.854+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:21:01.857 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:21:01.854+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:21:01.857 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17619.869518 secs 2024-07-26T16:21:01.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17619.869518 secs 2024-07-26T16:21:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17619.869518 secs 2024-07-26T16:21:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:02 smithi134 ceph-mon[31966]: pgmap v10031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:02.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:02 smithi189 ceph-mon[36790]: pgmap v10031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:02 smithi193 ceph-mon[37473]: pgmap v10031: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:04.827 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:04 smithi193 ceph-mon[37473]: pgmap v10032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:21:04.827 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17605.448383 secs 2024-07-26T16:21:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:04 smithi134 ceph-mon[31966]: pgmap v10032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:21:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17605.448383 secs 2024-07-26T16:21:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:04 smithi189 ceph-mon[36790]: pgmap v10032: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:21:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17605.448383 secs 2024-07-26T16:21:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:06 smithi134 ceph-mon[31966]: pgmap v10033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:06.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17624.869633 secs 2024-07-26T16:21:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:06 smithi189 ceph-mon[36790]: pgmap v10033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17624.869633 secs 2024-07-26T16:21:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:06 smithi193 ceph-mon[37473]: pgmap v10033: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17624.869633 secs 2024-07-26T16:21:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:08 smithi134 ceph-mon[31966]: pgmap v10034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:21:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:08 smithi189 ceph-mon[36790]: pgmap v10034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:21:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:08 smithi193 ceph-mon[37473]: pgmap v10034: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:21:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17610.448481 secs 2024-07-26T16:21:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17610.448481 secs 2024-07-26T16:21:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17610.448481 secs 2024-07-26T16:21:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:10 smithi134 ceph-mon[31966]: pgmap v10035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:10 smithi189 ceph-mon[36790]: pgmap v10035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:10 smithi193 ceph-mon[37473]: pgmap v10035: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17629.869701 secs 2024-07-26T16:21:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:11 smithi134 ceph-mon[31966]: pgmap v10036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17629.869701 secs 2024-07-26T16:21:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:11 smithi189 ceph-mon[36790]: pgmap v10036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17629.869701 secs 2024-07-26T16:21:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:11 smithi193 ceph-mon[37473]: pgmap v10036: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:21:12.487+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:21:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:21:12.487+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:21:12.489 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:21:12.487+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:21:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:14 smithi193 ceph-mon[37473]: pgmap v10037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:21:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17615.448599 secs 2024-07-26T16:21:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:14 smithi134 ceph-mon[31966]: pgmap v10037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:21:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17615.448599 secs 2024-07-26T16:21:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:14 smithi189 ceph-mon[36790]: pgmap v10037: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:21:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17615.448599 secs 2024-07-26T16:21:16.866 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:21:16.864+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:21:16.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:21:16.864+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:21:16.867 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:21:16.864+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:21:16.867 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:16 smithi134 ceph-mon[31966]: pgmap v10038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:16.867 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17634.869832 secs 2024-07-26T16:21:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:16 smithi189 ceph-mon[36790]: pgmap v10038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17634.869832 secs 2024-07-26T16:21:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:16 smithi193 ceph-mon[37473]: pgmap v10038: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17634.869832 secs 2024-07-26T16:21:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:18 smithi134 ceph-mon[31966]: pgmap v10039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:18 smithi189 ceph-mon[36790]: pgmap v10039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:18 smithi193 ceph-mon[37473]: pgmap v10039: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17620.448694 secs 2024-07-26T16:21:19.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17620.448694 secs 2024-07-26T16:21:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17620.448694 secs 2024-07-26T16:21:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:20 smithi134 ceph-mon[31966]: pgmap v10040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:20.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:20 smithi189 ceph-mon[36790]: pgmap v10040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:20 smithi193 ceph-mon[37473]: pgmap v10040: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17639.869957 secs 2024-07-26T16:21:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17639.869957 secs 2024-07-26T16:21:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17639.869957 secs 2024-07-26T16:21:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:22 smithi134 ceph-mon[31966]: pgmap v10041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:22 smithi189 ceph-mon[36790]: pgmap v10041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:22 smithi193 ceph-mon[37473]: pgmap v10041: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:24 smithi134 ceph-mon[31966]: pgmap v10042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:21:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17625.448819 secs 2024-07-26T16:21:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:24 smithi189 ceph-mon[36790]: pgmap v10042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:21:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17625.448819 secs 2024-07-26T16:21:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:24 smithi193 ceph-mon[37473]: pgmap v10042: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:21:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17625.448819 secs 2024-07-26T16:21:25.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:21:25.390+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T16:21:25.392 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:21:25.390+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T16:21:25.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:21:25.391+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T16:21:25.393 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:21:25.391+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 137 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T16:21:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:26 smithi134 ceph-mon[31966]: pgmap v10043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:26.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17644.870079 secs 2024-07-26T16:21:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:26 smithi189 ceph-mon[36790]: pgmap v10043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17644.870079 secs 2024-07-26T16:21:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:26 smithi193 ceph-mon[37473]: pgmap v10043: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17644.870079 secs 2024-07-26T16:21:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:21:27.496+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:21:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:21:27.496+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:21:27.498 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:21:27.496+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:21:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:28 smithi134 ceph-mon[31966]: pgmap v10044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:28 smithi189 ceph-mon[36790]: pgmap v10044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:28 smithi193 ceph-mon[37473]: pgmap v10044: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17630.448953 secs 2024-07-26T16:21:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17630.448953 secs 2024-07-26T16:21:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17630.448953 secs 2024-07-26T16:21:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:30 smithi134 ceph-mon[31966]: pgmap v10045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:21:30.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:30 smithi189 ceph-mon[36790]: pgmap v10045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:21:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:30 smithi193 ceph-mon[37473]: pgmap v10045: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:21:31.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:21:31.880+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:21:31.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:21:31.880+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:21:31.882 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:21:31.880+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:21:31.882 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17649.870220 secs 2024-07-26T16:21:31.883 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:31 smithi134 ceph-mon[31966]: pgmap v10046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17649.870220 secs 2024-07-26T16:21:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:31 smithi189 ceph-mon[36790]: pgmap v10046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17649.870220 secs 2024-07-26T16:21:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:31 smithi193 ceph-mon[37473]: pgmap v10046: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:34 smithi134 ceph-mon[31966]: pgmap v10047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:21:34.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17635.449021 secs 2024-07-26T16:21:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:34 smithi189 ceph-mon[36790]: pgmap v10047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:21:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17635.449021 secs 2024-07-26T16:21:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:34 smithi193 ceph-mon[37473]: pgmap v10047: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:21:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17635.449021 secs 2024-07-26T16:21:35.946 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:21:35.944+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:21:35.950 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:21:35.944+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:21:35.951 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:21:35.945+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:21:35.951 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:21:35.945+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:21:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:36 smithi134 ceph-mon[31966]: pgmap v10048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17654.870365 secs 2024-07-26T16:21:36.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:36 smithi189 ceph-mon[36790]: pgmap v10048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17654.870365 secs 2024-07-26T16:21:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:36 smithi193 ceph-mon[37473]: pgmap v10048: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17654.870365 secs 2024-07-26T16:21:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:38 smithi134 ceph-mon[31966]: pgmap v10049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:38.960 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:38 smithi189 ceph-mon[36790]: pgmap v10049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:39.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:38 smithi193 ceph-mon[37473]: pgmap v10049: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:21:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17640.449160 secs 2024-07-26T16:21:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17640.449160 secs 2024-07-26T16:21:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17640.449160 secs 2024-07-26T16:21:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:40 smithi134 ceph-mon[31966]: pgmap v10050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:21:40.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:40 smithi189 ceph-mon[36790]: pgmap v10050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:21:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:40 smithi193 ceph-mon[37473]: pgmap v10050: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:21:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17659.870499 secs 2024-07-26T16:21:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17659.870499 secs 2024-07-26T16:21:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17659.870499 secs 2024-07-26T16:21:42.513 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:21:42.511+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:21:42.514 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:21:42.512+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:21:42.514 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:21:42.512+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:21:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:42 smithi134 ceph-mon[31966]: pgmap v10051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:42 smithi189 ceph-mon[36790]: pgmap v10051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:42 smithi193 ceph-mon[37473]: pgmap v10051: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:21:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:44 smithi134 ceph-mon[31966]: pgmap v10052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:21:44.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17645.449260 secs 2024-07-26T16:21:44.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:21:44.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:21:44.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:44 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:21:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:44 smithi189 ceph-mon[36790]: pgmap v10052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:21:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17645.449260 secs 2024-07-26T16:21:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:21:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:21:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:44 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:21:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:44 smithi193 ceph-mon[37473]: pgmap v10052: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:21:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17645.449260 secs 2024-07-26T16:21:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:21:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:21:45.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:44 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:21:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:21:45.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:21:45.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:21:45.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:21:45.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:21:45.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:21:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:21:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:21:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:21:46.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:21:46.895+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:21:46.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:21:46.895+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:21:46.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:21:46.895+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:21:46.898 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:46 smithi134 ceph-mon[31966]: pgmap v10053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:21:46.898 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:46 smithi134 ceph-mon[31966]: pgmap v10054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:46.898 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17664.870628 secs 2024-07-26T16:21:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:46 smithi189 ceph-mon[36790]: pgmap v10053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:21:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:46 smithi189 ceph-mon[36790]: pgmap v10054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17664.870628 secs 2024-07-26T16:21:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:46 smithi193 ceph-mon[37473]: pgmap v10053: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:21:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:46 smithi193 ceph-mon[37473]: pgmap v10054: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17664.870628 secs 2024-07-26T16:21:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:47 smithi134 ceph-mon[31966]: pgmap v10055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T16:21:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:47 smithi189 ceph-mon[36790]: pgmap v10055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T16:21:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:47 smithi193 ceph-mon[37473]: pgmap v10055: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-07-26T16:21:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:48 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17650.449350 secs 2024-07-26T16:21:49.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:48 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17650.449350 secs 2024-07-26T16:21:49.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:48 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17650.449350 secs 2024-07-26T16:21:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:50 smithi193 ceph-mon[37473]: pgmap v10056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:50 smithi134 ceph-mon[31966]: pgmap v10056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:50 smithi189 ceph-mon[36790]: pgmap v10056: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17669.870764 secs 2024-07-26T16:21:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:52 smithi134 ceph-mon[31966]: pgmap v10057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17669.870764 secs 2024-07-26T16:21:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:52 smithi189 ceph-mon[36790]: pgmap v10057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17669.870764 secs 2024-07-26T16:21:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:52 smithi193 ceph-mon[37473]: pgmap v10057: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:54 smithi134 ceph-mon[31966]: pgmap v10058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17655.449496 secs 2024-07-26T16:21:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:54 smithi189 ceph-mon[36790]: pgmap v10058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17655.449496 secs 2024-07-26T16:21:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:54 smithi193 ceph-mon[37473]: pgmap v10058: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:21:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17655.449496 secs 2024-07-26T16:21:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:56 smithi134 ceph-mon[31966]: pgmap v10059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:21:56.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:56 smithi189 ceph-mon[36790]: pgmap v10059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:21:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:56 smithi193 ceph-mon[37473]: pgmap v10059: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:21:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:21:57.527+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:21:57.529 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:21:57.527+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:21:57.530 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:21:57.527+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:21:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17674.870895 secs 2024-07-26T16:21:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17674.870895 secs 2024-07-26T16:21:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17674.870895 secs 2024-07-26T16:21:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:58 smithi134 ceph-mon[31966]: pgmap v10060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:58 smithi189 ceph-mon[36790]: pgmap v10060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:58 smithi193 ceph-mon[37473]: pgmap v10060: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:21:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:21:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17660.449620 secs 2024-07-26T16:21:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:21:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17660.449620 secs 2024-07-26T16:21:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:21:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17660.449620 secs 2024-07-26T16:22:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:00 smithi134 ceph-mon[31966]: pgmap v10061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:00 smithi189 ceph-mon[36790]: pgmap v10061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:00 smithi193 ceph-mon[37473]: pgmap v10061: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:01.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:22:01.906+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:22:01.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:22:01.906+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:01.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:22:01.906+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:22:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17679.871004 secs 2024-07-26T16:22:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:02 smithi134 ceph-mon[31966]: pgmap v10062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17679.871004 secs 2024-07-26T16:22:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:02 smithi189 ceph-mon[36790]: pgmap v10062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17679.871004 secs 2024-07-26T16:22:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:02 smithi193 ceph-mon[37473]: pgmap v10062: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:04 smithi134 ceph-mon[31966]: pgmap v10063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:22:04.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17665.449738 secs 2024-07-26T16:22:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:04 smithi189 ceph-mon[36790]: pgmap v10063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:22:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17665.449738 secs 2024-07-26T16:22:04.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:04 smithi193 ceph-mon[37473]: pgmap v10063: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:22:04.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17665.449738 secs 2024-07-26T16:22:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:06 smithi189 ceph-mon[36790]: pgmap v10064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:22:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:06 smithi193 ceph-mon[37473]: pgmap v10064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:22:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:06 smithi134 ceph-mon[31966]: pgmap v10064: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:22:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17684.871133 secs 2024-07-26T16:22:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17684.871133 secs 2024-07-26T16:22:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17684.871133 secs 2024-07-26T16:22:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:08 smithi189 ceph-mon[36790]: pgmap v10065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:22:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:08 smithi193 ceph-mon[37473]: pgmap v10065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:22:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:08 smithi134 ceph-mon[31966]: pgmap v10065: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:22:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17670.449836 secs 2024-07-26T16:22:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17670.449836 secs 2024-07-26T16:22:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17670.449836 secs 2024-07-26T16:22:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:10 smithi193 ceph-mon[37473]: pgmap v10066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:22:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:10 smithi134 ceph-mon[31966]: pgmap v10066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:22:10.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:10 smithi189 ceph-mon[36790]: pgmap v10066: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:22:12.544 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:22:12.543+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:22:12.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:22:12.543+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:22:12.545 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:22:12.543+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17689.871259 secs 2024-07-26T16:22:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:12 smithi193 ceph-mon[37473]: pgmap v10067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:22:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17689.871259 secs 2024-07-26T16:22:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:12 smithi134 ceph-mon[31966]: pgmap v10067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:22:12.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17689.871259 secs 2024-07-26T16:22:12.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:12 smithi189 ceph-mon[36790]: pgmap v10067: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:22:14.725 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:14 smithi134 ceph-mon[31966]: pgmap v10068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:22:14.725 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17675.449980 secs 2024-07-26T16:22:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:14 smithi193 ceph-mon[37473]: pgmap v10068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:22:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17675.449980 secs 2024-07-26T16:22:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:14 smithi189 ceph-mon[36790]: pgmap v10068: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:22:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17675.449980 secs 2024-07-26T16:22:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:16 smithi193 ceph-mon[37473]: pgmap v10069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:16 smithi134 ceph-mon[31966]: pgmap v10069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:16.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:22:16.921+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:22:16.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:22:16.921+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:22:16.923 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:22:16.921+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:16 smithi189 ceph-mon[36790]: pgmap v10069: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17694.871395 secs 2024-07-26T16:22:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17694.871395 secs 2024-07-26T16:22:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17694.871395 secs 2024-07-26T16:22:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:18 smithi193 ceph-mon[37473]: pgmap v10070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:18 smithi134 ceph-mon[31966]: pgmap v10070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:18 smithi189 ceph-mon[36790]: pgmap v10070: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17680.450046 secs 2024-07-26T16:22:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17680.450046 secs 2024-07-26T16:22:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17680.450046 secs 2024-07-26T16:22:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:20 smithi193 ceph-mon[37473]: pgmap v10071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:20 smithi134 ceph-mon[31966]: pgmap v10071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:20.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:20 smithi189 ceph-mon[36790]: pgmap v10071: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17699.871517 secs 2024-07-26T16:22:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:22 smithi193 ceph-mon[37473]: pgmap v10072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17699.871517 secs 2024-07-26T16:22:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:22 smithi134 ceph-mon[31966]: pgmap v10072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17699.871517 secs 2024-07-26T16:22:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:22 smithi189 ceph-mon[36790]: pgmap v10072: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:24 smithi193 ceph-mon[37473]: pgmap v10073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:22:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17685.450193 secs 2024-07-26T16:22:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:24 smithi134 ceph-mon[31966]: pgmap v10073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:22:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17685.450193 secs 2024-07-26T16:22:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:24 smithi189 ceph-mon[36790]: pgmap v10073: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:22:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17685.450193 secs 2024-07-26T16:22:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:26 smithi193 ceph-mon[37473]: pgmap v10074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:26 smithi134 ceph-mon[31966]: pgmap v10074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:26 smithi189 ceph-mon[36790]: pgmap v10074: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:22:27.558+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:22:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:22:27.558+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:22:27.560 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:22:27.558+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17704.871617 secs 2024-07-26T16:22:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17704.871617 secs 2024-07-26T16:22:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17704.871617 secs 2024-07-26T16:22:28.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:28 smithi189 ceph-mon[36790]: pgmap v10075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:28 smithi134 ceph-mon[31966]: pgmap v10075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:28 smithi193 ceph-mon[37473]: pgmap v10075: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17690.450318 secs 2024-07-26T16:22:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17690.450318 secs 2024-07-26T16:22:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17690.450318 secs 2024-07-26T16:22:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:30 smithi134 ceph-mon[31966]: pgmap v10076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:30 smithi189 ceph-mon[36790]: pgmap v10076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:30 smithi193 ceph-mon[37473]: pgmap v10076: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17709.871700 secs 2024-07-26T16:22:31.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:22:31.922+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:22:31.924 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:22:31.922+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:31.925 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:22:31.922+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:22:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17709.871700 secs 2024-07-26T16:22:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17709.871700 secs 2024-07-26T16:22:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:32 smithi134 ceph-mon[31966]: pgmap v10077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:32 smithi189 ceph-mon[36790]: pgmap v10077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:32 smithi193 ceph-mon[37473]: pgmap v10077: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:34.827 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:34 smithi193 ceph-mon[37473]: pgmap v10078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:22:34.827 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17695.450443 secs 2024-07-26T16:22:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:34 smithi134 ceph-mon[31966]: pgmap v10078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:22:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17695.450443 secs 2024-07-26T16:22:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:34 smithi189 ceph-mon[36790]: pgmap v10078: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:22:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17695.450443 secs 2024-07-26T16:22:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:36 smithi134 ceph-mon[31966]: pgmap v10079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17714.871838 secs 2024-07-26T16:22:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:36 smithi189 ceph-mon[36790]: pgmap v10079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17714.871838 secs 2024-07-26T16:22:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:36 smithi193 ceph-mon[37473]: pgmap v10079: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17714.871838 secs 2024-07-26T16:22:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:38 smithi134 ceph-mon[31966]: pgmap v10080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:38 smithi189 ceph-mon[36790]: pgmap v10080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:38 smithi193 ceph-mon[37473]: pgmap v10080: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17700.450562 secs 2024-07-26T16:22:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17700.450562 secs 2024-07-26T16:22:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17700.450562 secs 2024-07-26T16:22:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:40 smithi134 ceph-mon[31966]: pgmap v10081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:40.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:40 smithi189 ceph-mon[36790]: pgmap v10081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:40 smithi193 ceph-mon[37473]: pgmap v10081: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17719.871958 secs 2024-07-26T16:22:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17719.871958 secs 2024-07-26T16:22:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17719.871958 secs 2024-07-26T16:22:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:22:42.566+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:22:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:22:42.566+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:22:42.568 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:22:42.566+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:42 smithi134 ceph-mon[31966]: pgmap v10082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:42 smithi189 ceph-mon[36790]: pgmap v10082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:42 smithi193 ceph-mon[37473]: pgmap v10082: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:43 smithi134 ceph-mon[31966]: pgmap v10083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:22:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:43 smithi189 ceph-mon[36790]: pgmap v10083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:22:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:43 smithi193 ceph-mon[37473]: pgmap v10083: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:22:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17705.450691 secs 2024-07-26T16:22:44.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17705.450691 secs 2024-07-26T16:22:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17705.450691 secs 2024-07-26T16:22:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:45 smithi134 ceph-mon[31966]: pgmap v10084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:45.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:45 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:22:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:45 smithi189 ceph-mon[36790]: pgmap v10084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:45.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:45 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:22:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:45 smithi193 ceph-mon[37473]: pgmap v10084: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:22:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:45 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:22:46.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:22:46.928+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:22:46.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:22:46.928+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:46.931 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:22:46.928+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:22:47.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:47.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:47.174 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:47.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:46 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:47.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17724.872066 secs 2024-07-26T16:22:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:46 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17724.872066 secs 2024-07-26T16:22:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:46 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:47.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17724.872066 secs 2024-07-26T16:22:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:48 smithi134 ceph-mon[31966]: pgmap v10085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:48.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:48.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:48.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:48.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:48 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:48 smithi189 ceph-mon[36790]: pgmap v10085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:48.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:48 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:48 smithi193 ceph-mon[37473]: pgmap v10085: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:22:48.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:48.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:48.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:48.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:48 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17710.450830 secs 2024-07-26T16:22:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:49 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17710.450830 secs 2024-07-26T16:22:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:49 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17710.450830 secs 2024-07-26T16:22:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:49 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:50.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:50 smithi134 ceph-mon[31966]: pgmap v10086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:50.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:22:50.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:22:50.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:50 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:50.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:50 smithi189 ceph-mon[36790]: pgmap v10086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:50.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:22:50.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:22:50.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:50 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:50 smithi193 ceph-mon[37473]: pgmap v10086: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:22:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:22:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:22:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:50 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:22:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:51 smithi134 ceph-mon[31966]: pgmap v10087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:22:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:51 smithi134 ceph-mon[31966]: pgmap v10088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:22:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:51 smithi189 ceph-mon[36790]: pgmap v10087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:22:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:51 smithi189 ceph-mon[36790]: pgmap v10088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:22:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:51 smithi193 ceph-mon[37473]: pgmap v10087: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:22:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:51 smithi193 ceph-mon[37473]: pgmap v10088: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:22:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17729.872138 secs 2024-07-26T16:22:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17729.872138 secs 2024-07-26T16:22:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17729.872138 secs 2024-07-26T16:22:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:53 smithi134 ceph-mon[31966]: pgmap v10089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T16:22:53.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:53 smithi189 ceph-mon[36790]: pgmap v10089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T16:22:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:53 smithi193 ceph-mon[37473]: pgmap v10089: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-07-26T16:22:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17715.450973 secs 2024-07-26T16:22:54.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17715.450973 secs 2024-07-26T16:22:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17715.450973 secs 2024-07-26T16:22:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:55 smithi134 ceph-mon[31966]: pgmap v10090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T16:22:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:55 smithi189 ceph-mon[36790]: pgmap v10090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T16:22:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:55 smithi193 ceph-mon[37473]: pgmap v10090: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T16:22:57.571 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:22:57.568+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:22:57.571 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:22:57.569+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:22:57.571 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:22:57.569+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:22:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17734.872278 secs 2024-07-26T16:22:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:57 smithi134 ceph-mon[31966]: pgmap v10091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T16:22:57.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17734.872278 secs 2024-07-26T16:22:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:57 smithi189 ceph-mon[36790]: pgmap v10091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T16:22:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17734.872278 secs 2024-07-26T16:22:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:57 smithi193 ceph-mon[37473]: pgmap v10091: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 8 op/s 2024-07-26T16:22:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:59 smithi134 ceph-mon[31966]: pgmap v10092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T16:22:59.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:22:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17720.451070 secs 2024-07-26T16:22:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:59 smithi189 ceph-mon[36790]: pgmap v10092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T16:22:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:22:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17720.451070 secs 2024-07-26T16:22:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:59 smithi193 ceph-mon[37473]: pgmap v10092: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-07-26T16:22:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:22:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17720.451070 secs 2024-07-26T16:23:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:01 smithi189 ceph-mon[36790]: pgmap v10093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:23:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:01 smithi193 ceph-mon[37473]: pgmap v10093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:23:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:01 smithi134 ceph-mon[31966]: pgmap v10093: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:23:01.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:23:01.943+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:23:01.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:23:01.943+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:23:01.947 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:23:01.943+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:23:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17739.872355 secs 2024-07-26T16:23:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17739.872355 secs 2024-07-26T16:23:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17739.872355 secs 2024-07-26T16:23:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:03 smithi189 ceph-mon[36790]: pgmap v10094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:03 smithi193 ceph-mon[37473]: pgmap v10094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:03 smithi134 ceph-mon[31966]: pgmap v10094: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17725.451190 secs 2024-07-26T16:23:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17725.451190 secs 2024-07-26T16:23:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17725.451190 secs 2024-07-26T16:23:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:05 smithi193 ceph-mon[37473]: pgmap v10095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:05 smithi134 ceph-mon[31966]: pgmap v10095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:05.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:05 smithi189 ceph-mon[36790]: pgmap v10095: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17744.872443 secs 2024-07-26T16:23:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:07 smithi134 ceph-mon[31966]: pgmap v10096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17744.872443 secs 2024-07-26T16:23:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:07 smithi193 ceph-mon[37473]: pgmap v10096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17744.872443 secs 2024-07-26T16:23:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:07 smithi189 ceph-mon[36790]: pgmap v10096: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:09 smithi193 ceph-mon[37473]: pgmap v10097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:23:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17730.451270 secs 2024-07-26T16:23:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:09 smithi134 ceph-mon[31966]: pgmap v10097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:23:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17730.451270 secs 2024-07-26T16:23:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:09 smithi189 ceph-mon[36790]: pgmap v10097: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:23:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17730.451270 secs 2024-07-26T16:23:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:11 smithi193 ceph-mon[37473]: pgmap v10098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:11 smithi134 ceph-mon[31966]: pgmap v10098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:11.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:11 smithi189 ceph-mon[36790]: pgmap v10098: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:12.573 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:23:12.571+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:23:12.573 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:23:12.571+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:23:12.573 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:23:12.571+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:23:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17749.872511 secs 2024-07-26T16:23:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17749.872511 secs 2024-07-26T16:23:12.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17749.872511 secs 2024-07-26T16:23:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:13 smithi193 ceph-mon[37473]: pgmap v10099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:13 smithi134 ceph-mon[31966]: pgmap v10099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:13 smithi189 ceph-mon[36790]: pgmap v10099: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17735.451336 secs 2024-07-26T16:23:14.826 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17735.451336 secs 2024-07-26T16:23:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17735.451336 secs 2024-07-26T16:23:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:15 smithi193 ceph-mon[37473]: pgmap v10100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:15 smithi134 ceph-mon[31966]: pgmap v10100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:15 smithi189 ceph-mon[36790]: pgmap v10100: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:16.960 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:23:16.958+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:23:16.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:23:16.958+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:23:16.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:23:16.958+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:23:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17754.872626 secs 2024-07-26T16:23:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:17 smithi193 ceph-mon[37473]: pgmap v10101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17754.872626 secs 2024-07-26T16:23:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:17 smithi134 ceph-mon[31966]: pgmap v10101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17754.872626 secs 2024-07-26T16:23:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:17 smithi189 ceph-mon[36790]: pgmap v10101: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:19 smithi134 ceph-mon[31966]: pgmap v10102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:23:19.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17740.451466 secs 2024-07-26T16:23:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:19 smithi189 ceph-mon[36790]: pgmap v10102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:23:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17740.451466 secs 2024-07-26T16:23:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:19 smithi193 ceph-mon[37473]: pgmap v10102: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:23:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17740.451466 secs 2024-07-26T16:23:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:21 smithi134 ceph-mon[31966]: pgmap v10103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17759.872764 secs 2024-07-26T16:23:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:21 smithi189 ceph-mon[36790]: pgmap v10103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17759.872764 secs 2024-07-26T16:23:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:21 smithi193 ceph-mon[37473]: pgmap v10103: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17759.872764 secs 2024-07-26T16:23:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:23 smithi134 ceph-mon[31966]: pgmap v10104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:23 smithi189 ceph-mon[36790]: pgmap v10104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:23 smithi193 ceph-mon[37473]: pgmap v10104: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17745.451590 secs 2024-07-26T16:23:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17745.451590 secs 2024-07-26T16:23:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17745.451590 secs 2024-07-26T16:23:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:25 smithi134 ceph-mon[31966]: pgmap v10105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:25.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:25 smithi189 ceph-mon[36790]: pgmap v10105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:25 smithi193 ceph-mon[37473]: pgmap v10105: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17764.872863 secs 2024-07-26T16:23:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17764.872863 secs 2024-07-26T16:23:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17764.872863 secs 2024-07-26T16:23:27.589 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:23:27.587+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:23:27.589 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:23:27.587+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:23:27.589 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:23:27.587+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:23:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:27 smithi134 ceph-mon[31966]: pgmap v10106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:27 smithi189 ceph-mon[36790]: pgmap v10106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:27 smithi193 ceph-mon[37473]: pgmap v10106: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:29 smithi134 ceph-mon[31966]: pgmap v10107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:23:29.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17750.451742 secs 2024-07-26T16:23:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:29 smithi189 ceph-mon[36790]: pgmap v10107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:23:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17750.451742 secs 2024-07-26T16:23:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:29 smithi193 ceph-mon[37473]: pgmap v10107: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:23:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17750.451742 secs 2024-07-26T16:23:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:31 smithi134 ceph-mon[31966]: pgmap v10108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17769.872983 secs 2024-07-26T16:23:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:31 smithi189 ceph-mon[36790]: pgmap v10108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17769.872983 secs 2024-07-26T16:23:31.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:23:31.973+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:23:31.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:23:31.973+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:23:31.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:23:31.973+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:23:32.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:31 smithi193 ceph-mon[37473]: pgmap v10108: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:32.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17769.872983 secs 2024-07-26T16:23:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:33 smithi134 ceph-mon[31966]: pgmap v10109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:33.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:33 smithi189 ceph-mon[36790]: pgmap v10109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:33 smithi193 ceph-mon[37473]: pgmap v10109: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17755.451837 secs 2024-07-26T16:23:34.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17755.451837 secs 2024-07-26T16:23:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17755.451837 secs 2024-07-26T16:23:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:35 smithi134 ceph-mon[31966]: pgmap v10110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:35.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:35 smithi189 ceph-mon[36790]: pgmap v10110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:35 smithi193 ceph-mon[37473]: pgmap v10110: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17774.873102 secs 2024-07-26T16:23:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17774.873102 secs 2024-07-26T16:23:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17774.873102 secs 2024-07-26T16:23:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:37 smithi134 ceph-mon[31966]: pgmap v10111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:37.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:37 smithi189 ceph-mon[36790]: pgmap v10111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:37 smithi193 ceph-mon[37473]: pgmap v10111: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:38 smithi134 ceph-mon[31966]: pgmap v10112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:23:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:38 smithi189 ceph-mon[36790]: pgmap v10112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:23:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:38 smithi193 ceph-mon[37473]: pgmap v10112: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:23:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17760.451982 secs 2024-07-26T16:23:39.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17760.451982 secs 2024-07-26T16:23:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17760.451982 secs 2024-07-26T16:23:40.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:40 smithi134 ceph-mon[31966]: pgmap v10113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:40.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:40 smithi189 ceph-mon[36790]: pgmap v10113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:40 smithi193 ceph-mon[37473]: pgmap v10113: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17779.873243 secs 2024-07-26T16:23:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17779.873243 secs 2024-07-26T16:23:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17779.873243 secs 2024-07-26T16:23:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:23:42.603+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:23:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:23:42.603+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:23:42.604 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:23:42.603+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:23:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:42 smithi134 ceph-mon[31966]: pgmap v10114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:42 smithi189 ceph-mon[36790]: pgmap v10114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:42 smithi193 ceph-mon[37473]: pgmap v10114: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:23:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17765.452080 secs 2024-07-26T16:23:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17765.452080 secs 2024-07-26T16:23:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17765.452080 secs 2024-07-26T16:23:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:45 smithi134 ceph-mon[31966]: pgmap v10115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:45 smithi189 ceph-mon[36790]: pgmap v10115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:45 smithi193 ceph-mon[37473]: pgmap v10115: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:23:46.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:23:46.988+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:23:46.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:23:46.988+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:23:46.991 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:23:46.988+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:23:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17784.873359 secs 2024-07-26T16:23:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:47 smithi134 ceph-mon[31966]: pgmap v10116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17784.873359 secs 2024-07-26T16:23:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:47 smithi189 ceph-mon[36790]: pgmap v10116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17784.873359 secs 2024-07-26T16:23:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:47 smithi193 ceph-mon[37473]: pgmap v10116: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:23:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:49 smithi134 ceph-mon[31966]: pgmap v10117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:23:49.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17770.452197 secs 2024-07-26T16:23:49.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:49 smithi189 ceph-mon[36790]: pgmap v10117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:23:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17770.452197 secs 2024-07-26T16:23:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:49 smithi193 ceph-mon[37473]: pgmap v10117: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:23:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17770.452197 secs 2024-07-26T16:23:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:51 smithi134 ceph-mon[31966]: pgmap v10118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:23:51.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:51 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:23:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:51 smithi189 ceph-mon[36790]: pgmap v10118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:23:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:51 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:23:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:51 smithi193 ceph-mon[37473]: pgmap v10118: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:23:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:51 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:23:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17789.873446 secs 2024-07-26T16:23:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:23:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:23:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:52 smithi134 ceph-mon[31966]: pgmap v10119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T16:23:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:52 smithi134 ceph-mon[31966]: pgmap v10120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:23:52.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:23:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17789.873446 secs 2024-07-26T16:23:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:23:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:23:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:52 smithi189 ceph-mon[36790]: pgmap v10119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T16:23:52.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:52 smithi189 ceph-mon[36790]: pgmap v10120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:23:52.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:23:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17789.873446 secs 2024-07-26T16:23:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:23:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:23:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:52 smithi193 ceph-mon[37473]: pgmap v10119: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-07-26T16:23:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:52 smithi193 ceph-mon[37473]: pgmap v10120: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-07-26T16:23:52.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:23:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:54 smithi193 ceph-mon[37473]: pgmap v10121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:23:54.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17775.452294 secs 2024-07-26T16:23:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:54 smithi134 ceph-mon[31966]: pgmap v10121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:23:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17775.452294 secs 2024-07-26T16:23:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:54 smithi189 ceph-mon[36790]: pgmap v10121: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:23:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17775.452294 secs 2024-07-26T16:23:56.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:56 smithi134 ceph-mon[31966]: pgmap v10122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:23:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:56 smithi189 ceph-mon[36790]: pgmap v10122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:23:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:56 smithi193 ceph-mon[37473]: pgmap v10122: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:23:57.608 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:23:57.606+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:23:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:23:57.606+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:23:57.609 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:23:57.606+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:23:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17794.873559 secs 2024-07-26T16:23:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17794.873559 secs 2024-07-26T16:23:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17794.873559 secs 2024-07-26T16:23:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:58 smithi134 ceph-mon[31966]: pgmap v10123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:23:58.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:58 smithi189 ceph-mon[36790]: pgmap v10123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:23:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:58 smithi193 ceph-mon[37473]: pgmap v10123: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:23:59.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:23:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17780.452377 secs 2024-07-26T16:23:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:23:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17780.452377 secs 2024-07-26T16:24:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:23:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17780.452377 secs 2024-07-26T16:24:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:00 smithi134 ceph-mon[31966]: pgmap v10124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:24:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:00 smithi189 ceph-mon[36790]: pgmap v10124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:24:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:00 smithi193 ceph-mon[37473]: pgmap v10124: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:24:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17799.873685 secs 2024-07-26T16:24:01.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17799.873685 secs 2024-07-26T16:24:02.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:02.004+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:24:02.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:02.004+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:24:02.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:02.004+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:24:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17799.873685 secs 2024-07-26T16:24:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:02 smithi134 ceph-mon[31966]: pgmap v10125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:24:02.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:02 smithi189 ceph-mon[36790]: pgmap v10125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:24:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:02 smithi193 ceph-mon[37473]: pgmap v10125: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:24:04.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:04 smithi134 ceph-mon[31966]: pgmap v10126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:24:04.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17785.452498 secs 2024-07-26T16:24:04.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:04 smithi189 ceph-mon[36790]: pgmap v10126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:24:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17785.452498 secs 2024-07-26T16:24:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:04 smithi193 ceph-mon[37473]: pgmap v10126: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:24:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17785.452498 secs 2024-07-26T16:24:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:06 smithi134 ceph-mon[31966]: pgmap v10127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:06.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17804.873801 secs 2024-07-26T16:24:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:06 smithi189 ceph-mon[36790]: pgmap v10127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17804.873801 secs 2024-07-26T16:24:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:06 smithi193 ceph-mon[37473]: pgmap v10127: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17804.873801 secs 2024-07-26T16:24:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:08 smithi134 ceph-mon[31966]: pgmap v10128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:24:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:08 smithi189 ceph-mon[36790]: pgmap v10128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:24:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:08 smithi193 ceph-mon[37473]: pgmap v10128: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:24:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17790.452571 secs 2024-07-26T16:24:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17790.452571 secs 2024-07-26T16:24:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17790.452571 secs 2024-07-26T16:24:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:10 smithi134 ceph-mon[31966]: pgmap v10129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:10 smithi189 ceph-mon[36790]: pgmap v10129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:10 smithi193 ceph-mon[37473]: pgmap v10129: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17809.873920 secs 2024-07-26T16:24:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17809.873920 secs 2024-07-26T16:24:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17809.873920 secs 2024-07-26T16:24:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:12.622+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:24:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:12.622+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:24:12.624 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:12.622+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:24:12.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:12 smithi134 ceph-mon[31966]: pgmap v10130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:12 smithi189 ceph-mon[36790]: pgmap v10130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:12 smithi193 ceph-mon[37473]: pgmap v10130: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:13 smithi134 ceph-mon[31966]: pgmap v10131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:24:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:13 smithi189 ceph-mon[36790]: pgmap v10131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:24:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:13 smithi193 ceph-mon[37473]: pgmap v10131: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:24:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17795.452693 secs 2024-07-26T16:24:14.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17795.452693 secs 2024-07-26T16:24:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17795.452693 secs 2024-07-26T16:24:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:15 smithi134 ceph-mon[31966]: pgmap v10132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:15 smithi189 ceph-mon[36790]: pgmap v10132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:15 smithi193 ceph-mon[37473]: pgmap v10132: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17814.874043 secs 2024-07-26T16:24:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17814.874043 secs 2024-07-26T16:24:17.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:17.019+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:24:17.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:17.019+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:24:17.021 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:17.019+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:24:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17814.874043 secs 2024-07-26T16:24:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:18 smithi134 ceph-mon[31966]: pgmap v10133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:24:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:18 smithi189 ceph-mon[36790]: pgmap v10133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:24:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:18 smithi193 ceph-mon[37473]: pgmap v10133: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:24:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17800.452831 secs 2024-07-26T16:24:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17800.452831 secs 2024-07-26T16:24:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17800.452831 secs 2024-07-26T16:24:20.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:20 smithi134 ceph-mon[31966]: pgmap v10134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:20.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:20 smithi189 ceph-mon[36790]: pgmap v10134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:20 smithi193 ceph-mon[37473]: pgmap v10134: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17819.874162 secs 2024-07-26T16:24:21.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17819.874162 secs 2024-07-26T16:24:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17819.874162 secs 2024-07-26T16:24:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:22 smithi134 ceph-mon[31966]: pgmap v10135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:22 smithi189 ceph-mon[36790]: pgmap v10135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:22 smithi193 ceph-mon[37473]: pgmap v10135: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:24 smithi134 ceph-mon[31966]: pgmap v10136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:24:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17805.452989 secs 2024-07-26T16:24:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:24 smithi189 ceph-mon[36790]: pgmap v10136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:24:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17805.452989 secs 2024-07-26T16:24:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:24 smithi193 ceph-mon[37473]: pgmap v10136: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:24:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17805.452989 secs 2024-07-26T16:24:25.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:25.422+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T16:24:25.424 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:25.422+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T16:24:25.425 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:25.423+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f9b240036c0 con 0x7f9b4c10f330 2024-07-26T16:24:25.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:25.423+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 138 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f9b20006430 con 0x55c65e5ee670 2024-07-26T16:24:25.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:25.423+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T16:24:25.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:25.423+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T16:24:25.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:25.424+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T16:24:25.426 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:25.424+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T16:24:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:25 smithi134 ceph-mon[31966]: pgmap v10137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:25.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:25 smithi189 ceph-mon[36790]: pgmap v10137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:25 smithi193 ceph-mon[37473]: pgmap v10137: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:26.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17824.874289 secs 2024-07-26T16:24:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17824.874289 secs 2024-07-26T16:24:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17824.874289 secs 2024-07-26T16:24:27.639 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:27.637+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:24:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:27.637+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:24:27.640 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:27.637+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:24:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:28 smithi134 ceph-mon[31966]: pgmap v10138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:24:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:28 smithi189 ceph-mon[36790]: pgmap v10138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:24:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:28 smithi193 ceph-mon[37473]: pgmap v10138: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:24:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17810.453115 secs 2024-07-26T16:24:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17810.453115 secs 2024-07-26T16:24:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17810.453115 secs 2024-07-26T16:24:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:30 smithi134 ceph-mon[31966]: pgmap v10139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:24:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:30 smithi189 ceph-mon[36790]: pgmap v10139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:24:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:30 smithi193 ceph-mon[37473]: pgmap v10139: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:24:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17829.874413 secs 2024-07-26T16:24:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17829.874413 secs 2024-07-26T16:24:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17829.874413 secs 2024-07-26T16:24:32.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:32.035+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:24:32.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:32.035+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:24:32.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:32.035+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:24:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:32 smithi134 ceph-mon[31966]: pgmap v10140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:24:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:32 smithi189 ceph-mon[36790]: pgmap v10140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:24:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:32 smithi193 ceph-mon[37473]: pgmap v10140: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:24:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:33 smithi134 ceph-mon[31966]: pgmap v10141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:24:33.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:33 smithi189 ceph-mon[36790]: pgmap v10141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:24:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:33 smithi193 ceph-mon[37473]: pgmap v10141: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:24:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17815.453241 secs 2024-07-26T16:24:34.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17815.453241 secs 2024-07-26T16:24:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17815.453241 secs 2024-07-26T16:24:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:35 smithi134 ceph-mon[31966]: pgmap v10142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:24:35.942 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:35 smithi193 ceph-mon[37473]: pgmap v10142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:24:35.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:35 smithi189 ceph-mon[36790]: pgmap v10142: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:24:35.977 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:35.976+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:24:35.978 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:35.976+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:24:35.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:35.976+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 612+0+0 (secure 0 0 0) 0x7f2194006610 con 0x55a08e2d94d0 2024-07-26T16:24:35.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:35.976+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:24:35.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:35.976+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 139 ==== auth_reply(proto 2 0 (0) Success) ==== 783+0+0 (secure 0 0 0) 0x7f2198003690 con 0x7f21c8105550 2024-07-26T16:24:35.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:35.976+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:24:35.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:35.977+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:24:35.979 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:35.977+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:24:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17834.874471 secs 2024-07-26T16:24:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17834.874471 secs 2024-07-26T16:24:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17834.874471 secs 2024-07-26T16:24:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:37 smithi134 ceph-mon[31966]: pgmap v10143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:24:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:37 smithi189 ceph-mon[36790]: pgmap v10143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:24:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:37 smithi193 ceph-mon[37473]: pgmap v10143: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:24:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17820.453360 secs 2024-07-26T16:24:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17820.453360 secs 2024-07-26T16:24:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17820.453360 secs 2024-07-26T16:24:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:40 smithi134 ceph-mon[31966]: pgmap v10144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:24:40.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:40 smithi189 ceph-mon[36790]: pgmap v10144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:24:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:40 smithi193 ceph-mon[37473]: pgmap v10144: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:24:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17839.874565 secs 2024-07-26T16:24:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17839.874565 secs 2024-07-26T16:24:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17839.874565 secs 2024-07-26T16:24:42.654 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:42.652+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:24:42.655 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:42.652+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:24:42.655 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:42.652+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:24:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:42 smithi134 ceph-mon[31966]: pgmap v10145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:42 smithi189 ceph-mon[36790]: pgmap v10145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:42 smithi193 ceph-mon[37473]: pgmap v10145: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:44 smithi134 ceph-mon[31966]: pgmap v10146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:24:44.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17825.453485 secs 2024-07-26T16:24:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:44 smithi189 ceph-mon[36790]: pgmap v10146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:24:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17825.453485 secs 2024-07-26T16:24:45.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:44 smithi193 ceph-mon[37473]: pgmap v10146: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:24:45.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17825.453485 secs 2024-07-26T16:24:46.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:46 smithi134 ceph-mon[31966]: pgmap v10147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:46.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17844.874667 secs 2024-07-26T16:24:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:46 smithi189 ceph-mon[36790]: pgmap v10147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17844.874667 secs 2024-07-26T16:24:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:46 smithi193 ceph-mon[37473]: pgmap v10147: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17844.874667 secs 2024-07-26T16:24:47.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:47.040+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:24:47.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:47.040+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:24:47.042 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:24:47.040+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:24:48.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:48 smithi134 ceph-mon[31966]: pgmap v10148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:24:49.106 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:48 smithi193 ceph-mon[37473]: pgmap v10148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:24:49.107 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:48 smithi189 ceph-mon[36790]: pgmap v10148: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:24:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17830.453622 secs 2024-07-26T16:24:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17830.453622 secs 2024-07-26T16:24:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17830.453622 secs 2024-07-26T16:24:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:50 smithi134 ceph-mon[31966]: pgmap v10149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:24:50.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:50 smithi189 ceph-mon[36790]: pgmap v10149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:24:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:50 smithi193 ceph-mon[37473]: pgmap v10149: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:24:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17849.874792 secs 2024-07-26T16:24:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17849.874792 secs 2024-07-26T16:24:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17849.874792 secs 2024-07-26T16:24:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:52 smithi134 ceph-mon[31966]: pgmap v10150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:52.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:52 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:24:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:52 smithi189 ceph-mon[36790]: pgmap v10150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:52 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:24:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:52 smithi193 ceph-mon[37473]: pgmap v10150: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:24:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:52 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:24:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:24:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:24:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:53 smithi134 ceph-mon[31966]: pgmap v10151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T16:24:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:53 smithi134 ceph-mon[31966]: pgmap v10152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:24:53.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:24:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:24:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:24:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:53 smithi189 ceph-mon[36790]: pgmap v10151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T16:24:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:53 smithi189 ceph-mon[36790]: pgmap v10152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:24:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:24:54.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:24:54.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:24:54.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:53 smithi193 ceph-mon[37473]: pgmap v10151: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T16:24:54.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:53 smithi193 ceph-mon[37473]: pgmap v10152: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:24:54.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:24:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17835.453778 secs 2024-07-26T16:24:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17835.453778 secs 2024-07-26T16:24:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17835.453778 secs 2024-07-26T16:24:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:55 smithi134 ceph-mon[31966]: pgmap v10153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:24:55.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:55 smithi189 ceph-mon[36790]: pgmap v10153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:24:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:55 smithi193 ceph-mon[37473]: pgmap v10153: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:24:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17854.874918 secs 2024-07-26T16:24:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17854.874918 secs 2024-07-26T16:24:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17854.874918 secs 2024-07-26T16:24:57.663 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:57.661+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:24:57.663 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:57.661+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:24:57.663 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:24:57.661+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:24:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:57 smithi189 ceph-mon[36790]: pgmap v10154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:24:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:57 smithi193 ceph-mon[37473]: pgmap v10154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:24:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:57 smithi134 ceph-mon[31966]: pgmap v10154: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:25:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17840.453910 secs 2024-07-26T16:25:00.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:24:59 smithi193 ceph-mon[37473]: pgmap v10155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:25:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17840.453910 secs 2024-07-26T16:25:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:24:59 smithi134 ceph-mon[31966]: pgmap v10155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:25:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17840.453910 secs 2024-07-26T16:25:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:24:59 smithi189 ceph-mon[36790]: pgmap v10155: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:25:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:01 smithi193 ceph-mon[37473]: pgmap v10156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:25:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17859.875039 secs 2024-07-26T16:25:02.057 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:25:02.055+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:25:02.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:25:02.055+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:02.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:25:02.055+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:25:02.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:01 smithi134 ceph-mon[31966]: pgmap v10156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:25:02.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17859.875039 secs 2024-07-26T16:25:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:01 smithi189 ceph-mon[36790]: pgmap v10156: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:25:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17859.875039 secs 2024-07-26T16:25:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:03 smithi193 ceph-mon[37473]: pgmap v10157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:25:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:03 smithi134 ceph-mon[31966]: pgmap v10157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:25:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:03 smithi189 ceph-mon[36790]: pgmap v10157: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:25:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17845.454057 secs 2024-07-26T16:25:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17845.454057 secs 2024-07-26T16:25:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17845.454057 secs 2024-07-26T16:25:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:05 smithi134 ceph-mon[31966]: pgmap v10158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:05 smithi189 ceph-mon[36790]: pgmap v10158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:05 smithi193 ceph-mon[37473]: pgmap v10158: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17864.875184 secs 2024-07-26T16:25:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17864.875184 secs 2024-07-26T16:25:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17864.875184 secs 2024-07-26T16:25:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:07 smithi134 ceph-mon[31966]: pgmap v10159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:07 smithi189 ceph-mon[36790]: pgmap v10159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:07 smithi193 ceph-mon[37473]: pgmap v10159: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17850.454183 secs 2024-07-26T16:25:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:09 smithi134 ceph-mon[31966]: pgmap v10160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17850.454183 secs 2024-07-26T16:25:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:09 smithi189 ceph-mon[36790]: pgmap v10160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17850.454183 secs 2024-07-26T16:25:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:09 smithi193 ceph-mon[37473]: pgmap v10160: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:11 smithi134 ceph-mon[31966]: pgmap v10161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17869.875298 secs 2024-07-26T16:25:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:11 smithi189 ceph-mon[36790]: pgmap v10161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17869.875298 secs 2024-07-26T16:25:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:11 smithi193 ceph-mon[37473]: pgmap v10161: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:12.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17869.875298 secs 2024-07-26T16:25:12.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:25:12.676+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:25:12.679 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:25:12.677+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:25:12.680 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:25:12.677+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:13 smithi134 ceph-mon[31966]: pgmap v10162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:13 smithi189 ceph-mon[36790]: pgmap v10162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:13 smithi193 ceph-mon[37473]: pgmap v10162: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:15.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17855.454270 secs 2024-07-26T16:25:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17855.454270 secs 2024-07-26T16:25:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17855.454270 secs 2024-07-26T16:25:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:15 smithi134 ceph-mon[31966]: pgmap v10163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:15 smithi189 ceph-mon[36790]: pgmap v10163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:15 smithi193 ceph-mon[37473]: pgmap v10163: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:17.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:25:17.070+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:25:17.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:25:17.071+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:25:17.073 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:25:17.071+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17874.875411 secs 2024-07-26T16:25:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17874.875411 secs 2024-07-26T16:25:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17874.875411 secs 2024-07-26T16:25:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:17 smithi134 ceph-mon[31966]: pgmap v10164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:17 smithi189 ceph-mon[36790]: pgmap v10164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:17 smithi193 ceph-mon[37473]: pgmap v10164: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17860.454401 secs 2024-07-26T16:25:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:19 smithi134 ceph-mon[31966]: pgmap v10165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17860.454401 secs 2024-07-26T16:25:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:19 smithi189 ceph-mon[36790]: pgmap v10165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17860.454401 secs 2024-07-26T16:25:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:19 smithi193 ceph-mon[37473]: pgmap v10165: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:21 smithi134 ceph-mon[31966]: pgmap v10166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17879.875517 secs 2024-07-26T16:25:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:21 smithi189 ceph-mon[36790]: pgmap v10166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17879.875517 secs 2024-07-26T16:25:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:21 smithi193 ceph-mon[37473]: pgmap v10166: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17879.875517 secs 2024-07-26T16:25:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:23 smithi134 ceph-mon[31966]: pgmap v10167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:24.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:23 smithi189 ceph-mon[36790]: pgmap v10167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:23 smithi193 ceph-mon[37473]: pgmap v10167: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17865.454501 secs 2024-07-26T16:25:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17865.454501 secs 2024-07-26T16:25:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17865.454501 secs 2024-07-26T16:25:26.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:25 smithi134 ceph-mon[31966]: pgmap v10168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:25 smithi189 ceph-mon[36790]: pgmap v10168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:25 smithi193 ceph-mon[37473]: pgmap v10168: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:27.175 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17884.875584 secs 2024-07-26T16:25:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17884.875584 secs 2024-07-26T16:25:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17884.875584 secs 2024-07-26T16:25:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:25:27.692+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:25:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:25:27.692+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:27.694 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:25:27.692+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:25:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:27 smithi134 ceph-mon[31966]: pgmap v10169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:27 smithi189 ceph-mon[36790]: pgmap v10169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:27 smithi193 ceph-mon[37473]: pgmap v10169: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:30.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17870.454587 secs 2024-07-26T16:25:30.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:29 smithi134 ceph-mon[31966]: pgmap v10170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:30.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17870.454587 secs 2024-07-26T16:25:30.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:29 smithi189 ceph-mon[36790]: pgmap v10170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17870.454587 secs 2024-07-26T16:25:30.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:29 smithi193 ceph-mon[37473]: pgmap v10170: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:32.088 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:25:32.086+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:25:32.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:25:32.086+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:32.089 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:25:32.086+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:25:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:31 smithi134 ceph-mon[31966]: pgmap v10171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17889.875709 secs 2024-07-26T16:25:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:31 smithi189 ceph-mon[36790]: pgmap v10171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17889.875709 secs 2024-07-26T16:25:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:31 smithi193 ceph-mon[37473]: pgmap v10171: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17889.875709 secs 2024-07-26T16:25:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:33 smithi134 ceph-mon[31966]: pgmap v10172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:33 smithi189 ceph-mon[36790]: pgmap v10172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:33 smithi193 ceph-mon[37473]: pgmap v10172: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:35.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17875.454700 secs 2024-07-26T16:25:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17875.454700 secs 2024-07-26T16:25:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17875.454700 secs 2024-07-26T16:25:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:35 smithi134 ceph-mon[31966]: pgmap v10173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:35 smithi189 ceph-mon[36790]: pgmap v10173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:35 smithi193 ceph-mon[37473]: pgmap v10173: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17894.875775 secs 2024-07-26T16:25:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17894.875775 secs 2024-07-26T16:25:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17894.875775 secs 2024-07-26T16:25:38.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:37 smithi189 ceph-mon[36790]: pgmap v10174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:37 smithi193 ceph-mon[37473]: pgmap v10174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:37 smithi134 ceph-mon[31966]: pgmap v10174: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17880.454815 secs 2024-07-26T16:25:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:39 smithi189 ceph-mon[36790]: pgmap v10175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17880.454815 secs 2024-07-26T16:25:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:39 smithi193 ceph-mon[37473]: pgmap v10175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17880.454815 secs 2024-07-26T16:25:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:39 smithi134 ceph-mon[31966]: pgmap v10175: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:41 smithi193 ceph-mon[37473]: pgmap v10176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17899.875882 secs 2024-07-26T16:25:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:41 smithi134 ceph-mon[31966]: pgmap v10176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17899.875882 secs 2024-07-26T16:25:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:41 smithi189 ceph-mon[36790]: pgmap v10176: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17899.875882 secs 2024-07-26T16:25:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:25:42.708+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:25:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:25:42.708+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:25:42.710 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:25:42.708+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:43 smithi193 ceph-mon[37473]: pgmap v10177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:43 smithi134 ceph-mon[31966]: pgmap v10177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:43 smithi189 ceph-mon[36790]: pgmap v10177: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17885.454934 secs 2024-07-26T16:25:45.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17885.454934 secs 2024-07-26T16:25:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17885.454934 secs 2024-07-26T16:25:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:45 smithi134 ceph-mon[31966]: pgmap v10178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:45 smithi193 ceph-mon[37473]: pgmap v10178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:45 smithi189 ceph-mon[36790]: pgmap v10178: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:25:47.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:25:47.102+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:25:47.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:25:47.102+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:25:47.104 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:25:47.102+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17904.876007 secs 2024-07-26T16:25:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17904.876007 secs 2024-07-26T16:25:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17904.876007 secs 2024-07-26T16:25:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:47 smithi193 ceph-mon[37473]: pgmap v10179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:47 smithi134 ceph-mon[31966]: pgmap v10179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:47 smithi189 ceph-mon[36790]: pgmap v10179: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17890.455051 secs 2024-07-26T16:25:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17890.455051 secs 2024-07-26T16:25:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17890.455051 secs 2024-07-26T16:25:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:50 smithi134 ceph-mon[31966]: pgmap v10180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:50 smithi189 ceph-mon[36790]: pgmap v10180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:50 smithi193 ceph-mon[37473]: pgmap v10180: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:52 smithi134 ceph-mon[31966]: pgmap v10181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17909.876148 secs 2024-07-26T16:25:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:52 smithi189 ceph-mon[36790]: pgmap v10181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17909.876148 secs 2024-07-26T16:25:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:52 smithi193 ceph-mon[37473]: pgmap v10181: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:25:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17909.876148 secs 2024-07-26T16:25:53.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:53 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:25:53.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:53 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:25:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:53 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:25:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:54 smithi134 ceph-mon[31966]: pgmap v10182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17895.455130 secs 2024-07-26T16:25:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:25:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:25:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:25:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:54 smithi189 ceph-mon[36790]: pgmap v10182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17895.455130 secs 2024-07-26T16:25:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:25:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:25:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:25:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:54 smithi193 ceph-mon[37473]: pgmap v10182: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:25:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17895.455130 secs 2024-07-26T16:25:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:25:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:25:54.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:25:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:55 smithi134 ceph-mon[31966]: pgmap v10183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:25:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:55 smithi134 ceph-mon[31966]: pgmap v10184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:25:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:55 smithi189 ceph-mon[36790]: pgmap v10183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:25:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:55 smithi189 ceph-mon[36790]: pgmap v10184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:25:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:55 smithi193 ceph-mon[37473]: pgmap v10183: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:25:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:55 smithi193 ceph-mon[37473]: pgmap v10184: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:25:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:57 smithi134 ceph-mon[31966]: pgmap v10185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:25:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17914.876280 secs 2024-07-26T16:25:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:57 smithi189 ceph-mon[36790]: pgmap v10185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:25:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17914.876280 secs 2024-07-26T16:25:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:57 smithi193 ceph-mon[37473]: pgmap v10185: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:25:57.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17914.876280 secs 2024-07-26T16:25:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:25:57.723+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:25:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:25:57.723+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:25:57.725 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:25:57.723+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:25:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:59 smithi134 ceph-mon[31966]: pgmap v10186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:25:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:25:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17900.455243 secs 2024-07-26T16:25:59.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:59 smithi189 ceph-mon[36790]: pgmap v10186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:25:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:25:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17900.455243 secs 2024-07-26T16:25:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:59 smithi193 ceph-mon[37473]: pgmap v10186: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:25:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:25:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17900.455243 secs 2024-07-26T16:26:01.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:01 smithi134 ceph-mon[31966]: pgmap v10187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:26:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:01 smithi189 ceph-mon[36790]: pgmap v10187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:26:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:01 smithi193 ceph-mon[37473]: pgmap v10187: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:26:02.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:26:02.117+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:26:02.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:26:02.117+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:26:02.120 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:26:02.117+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:26:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17919.876397 secs 2024-07-26T16:26:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17919.876397 secs 2024-07-26T16:26:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17919.876397 secs 2024-07-26T16:26:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:03 smithi134 ceph-mon[31966]: pgmap v10188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:26:03.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:03 smithi189 ceph-mon[36790]: pgmap v10188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:26:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:03 smithi193 ceph-mon[37473]: pgmap v10188: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:26:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17905.455341 secs 2024-07-26T16:26:04.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17905.455341 secs 2024-07-26T16:26:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17905.455341 secs 2024-07-26T16:26:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:05 smithi134 ceph-mon[31966]: pgmap v10189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:26:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:05 smithi189 ceph-mon[36790]: pgmap v10189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:26:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:05 smithi193 ceph-mon[37473]: pgmap v10189: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:26:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:07 smithi134 ceph-mon[31966]: pgmap v10190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:07.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17924.876494 secs 2024-07-26T16:26:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:07 smithi189 ceph-mon[36790]: pgmap v10190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17924.876494 secs 2024-07-26T16:26:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:07 smithi193 ceph-mon[37473]: pgmap v10190: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:07.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17924.876494 secs 2024-07-26T16:26:09.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:09 smithi134 ceph-mon[31966]: pgmap v10191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:26:09.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17910.455460 secs 2024-07-26T16:26:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:09 smithi189 ceph-mon[36790]: pgmap v10191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:26:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17910.455460 secs 2024-07-26T16:26:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:09 smithi193 ceph-mon[37473]: pgmap v10191: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:26:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17910.455460 secs 2024-07-26T16:26:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:11 smithi134 ceph-mon[31966]: pgmap v10192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:26:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:11 smithi189 ceph-mon[36790]: pgmap v10192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:26:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:11 smithi193 ceph-mon[37473]: pgmap v10192: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:26:12.426 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17929.876620 secs 2024-07-26T16:26:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17929.876620 secs 2024-07-26T16:26:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17929.876620 secs 2024-07-26T16:26:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:26:12.732+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:26:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:26:12.732+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:26:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:26:12.732+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:26:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:13 smithi134 ceph-mon[31966]: pgmap v10193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:13 smithi189 ceph-mon[36790]: pgmap v10193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:13 smithi193 ceph-mon[37473]: pgmap v10193: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17915.455582 secs 2024-07-26T16:26:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17915.455582 secs 2024-07-26T16:26:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17915.455582 secs 2024-07-26T16:26:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:15 smithi134 ceph-mon[31966]: pgmap v10194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:15 smithi189 ceph-mon[36790]: pgmap v10194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:15 smithi193 ceph-mon[37473]: pgmap v10194: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:17.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:26:17.133+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:26:17.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:26:17.133+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:26:17.135 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:26:17.133+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:26:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:17 smithi134 ceph-mon[31966]: pgmap v10195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:17.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17934.876741 secs 2024-07-26T16:26:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:17 smithi189 ceph-mon[36790]: pgmap v10195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17934.876741 secs 2024-07-26T16:26:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:17 smithi193 ceph-mon[37473]: pgmap v10195: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17934.876741 secs 2024-07-26T16:26:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:19 smithi134 ceph-mon[31966]: pgmap v10196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:26:19.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17920.455702 secs 2024-07-26T16:26:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:19 smithi189 ceph-mon[36790]: pgmap v10196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:26:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17920.455702 secs 2024-07-26T16:26:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:19 smithi193 ceph-mon[37473]: pgmap v10196: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:26:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17920.455702 secs 2024-07-26T16:26:21.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:21 smithi189 ceph-mon[36790]: pgmap v10197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:26:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:21 smithi193 ceph-mon[37473]: pgmap v10197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:26:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:21 smithi134 ceph-mon[31966]: pgmap v10197: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:26:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17939.876862 secs 2024-07-26T16:26:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17939.876862 secs 2024-07-26T16:26:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17939.876862 secs 2024-07-26T16:26:23.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:23 smithi189 ceph-mon[36790]: pgmap v10198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:23 smithi193 ceph-mon[37473]: pgmap v10198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:23 smithi134 ceph-mon[31966]: pgmap v10198: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17925.455849 secs 2024-07-26T16:26:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17925.455849 secs 2024-07-26T16:26:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17925.455849 secs 2024-07-26T16:26:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:25 smithi193 ceph-mon[37473]: pgmap v10199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:25 smithi134 ceph-mon[31966]: pgmap v10199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:25.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:25 smithi189 ceph-mon[36790]: pgmap v10199: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:27 smithi193 ceph-mon[37473]: pgmap v10200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17944.876955 secs 2024-07-26T16:26:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:27 smithi134 ceph-mon[31966]: pgmap v10200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17944.876955 secs 2024-07-26T16:26:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:27 smithi189 ceph-mon[36790]: pgmap v10200: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17944.876955 secs 2024-07-26T16:26:27.750 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:26:27.748+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:26:27.750 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:26:27.748+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:26:27.750 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:26:27.748+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:26:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:29 smithi193 ceph-mon[37473]: pgmap v10201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:26:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17930.455983 secs 2024-07-26T16:26:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:29 smithi134 ceph-mon[31966]: pgmap v10201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:26:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17930.455983 secs 2024-07-26T16:26:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:29 smithi189 ceph-mon[36790]: pgmap v10201: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:26:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17930.455983 secs 2024-07-26T16:26:31.496 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:31 smithi189 ceph-mon[36790]: pgmap v10202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:26:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:31 smithi193 ceph-mon[37473]: pgmap v10202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:26:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:31 smithi134 ceph-mon[31966]: pgmap v10202: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:26:32.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:26:32.145+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:26:32.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:26:32.145+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:26:32.146 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:26:32.145+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:26:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17949.877034 secs 2024-07-26T16:26:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17949.877034 secs 2024-07-26T16:26:32.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17949.877034 secs 2024-07-26T16:26:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:33 smithi193 ceph-mon[37473]: pgmap v10203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:33 smithi134 ceph-mon[31966]: pgmap v10203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:33.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:33 smithi189 ceph-mon[36790]: pgmap v10203: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17935.456051 secs 2024-07-26T16:26:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17935.456051 secs 2024-07-26T16:26:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17935.456051 secs 2024-07-26T16:26:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:35 smithi193 ceph-mon[37473]: pgmap v10204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:35 smithi134 ceph-mon[31966]: pgmap v10204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:35.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:35 smithi189 ceph-mon[36790]: pgmap v10204: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:37 smithi193 ceph-mon[37473]: pgmap v10205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:37.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17954.877156 secs 2024-07-26T16:26:37.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:37 smithi134 ceph-mon[31966]: pgmap v10205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:37.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17954.877156 secs 2024-07-26T16:26:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:37 smithi189 ceph-mon[36790]: pgmap v10205: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17954.877156 secs 2024-07-26T16:26:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:39 smithi193 ceph-mon[37473]: pgmap v10206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:26:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17940.456117 secs 2024-07-26T16:26:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:39 smithi134 ceph-mon[31966]: pgmap v10206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:26:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17940.456117 secs 2024-07-26T16:26:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:39 smithi189 ceph-mon[36790]: pgmap v10206: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:26:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17940.456117 secs 2024-07-26T16:26:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:41 smithi193 ceph-mon[37473]: pgmap v10207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:26:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:41 smithi134 ceph-mon[31966]: pgmap v10207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:26:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:41 smithi189 ceph-mon[36790]: pgmap v10207: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:26:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17959.877256 secs 2024-07-26T16:26:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17959.877256 secs 2024-07-26T16:26:42.765 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:26:42.763+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:26:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:26:42.763+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:26:42.766 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:26:42.763+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:26:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17959.877256 secs 2024-07-26T16:26:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:43 smithi134 ceph-mon[31966]: pgmap v10208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:43 smithi189 ceph-mon[36790]: pgmap v10208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:43 smithi193 ceph-mon[37473]: pgmap v10208: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17945.456230 secs 2024-07-26T16:26:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17945.456230 secs 2024-07-26T16:26:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17945.456230 secs 2024-07-26T16:26:45.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:45 smithi134 ceph-mon[31966]: pgmap v10209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:45.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:45 smithi189 ceph-mon[36790]: pgmap v10209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:45 smithi193 ceph-mon[37473]: pgmap v10209: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:47.151 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:26:47.149+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:26:47.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:26:47.149+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:26:47.152 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:26:47.149+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:26:47.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:47 smithi134 ceph-mon[31966]: pgmap v10210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:47.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17964.877375 secs 2024-07-26T16:26:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:47 smithi189 ceph-mon[36790]: pgmap v10210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17964.877375 secs 2024-07-26T16:26:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:47 smithi193 ceph-mon[37473]: pgmap v10210: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:26:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17964.877375 secs 2024-07-26T16:26:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:49 smithi134 ceph-mon[31966]: pgmap v10211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:26:49.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17950.456347 secs 2024-07-26T16:26:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:49 smithi189 ceph-mon[36790]: pgmap v10211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:26:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17950.456347 secs 2024-07-26T16:26:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:49 smithi193 ceph-mon[37473]: pgmap v10211: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:26:49.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17950.456347 secs 2024-07-26T16:26:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:51 smithi134 ceph-mon[31966]: pgmap v10212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:26:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:51 smithi189 ceph-mon[36790]: pgmap v10212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:26:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:51 smithi193 ceph-mon[37473]: pgmap v10212: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:26:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17969.877507 secs 2024-07-26T16:26:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17969.877507 secs 2024-07-26T16:26:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17969.877507 secs 2024-07-26T16:26:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:53 smithi134 ceph-mon[31966]: pgmap v10213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:53.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:53 smithi189 ceph-mon[36790]: pgmap v10213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:53 smithi193 ceph-mon[37473]: pgmap v10213: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:26:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17955.456467 secs 2024-07-26T16:26:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:54 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:26:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17955.456467 secs 2024-07-26T16:26:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:54 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:26:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17955.456467 secs 2024-07-26T16:26:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:54 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:26:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:55 smithi134 ceph-mon[31966]: pgmap v10214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:26:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:26:55.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:26:55.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:55 smithi189 ceph-mon[36790]: pgmap v10214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:26:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:26:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:26:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:55 smithi193 ceph-mon[37473]: pgmap v10214: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:26:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:26:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:26:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:26:56.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:56 smithi134 ceph-mon[31966]: pgmap v10215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:26:56.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:56 smithi134 ceph-mon[31966]: pgmap v10216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:26:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:56 smithi189 ceph-mon[36790]: pgmap v10215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:26:56.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:56 smithi189 ceph-mon[36790]: pgmap v10216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:26:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:56 smithi193 ceph-mon[37473]: pgmap v10215: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:26:56.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:56 smithi193 ceph-mon[37473]: pgmap v10216: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:26:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17974.877630 secs 2024-07-26T16:26:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17974.877630 secs 2024-07-26T16:26:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17974.877630 secs 2024-07-26T16:26:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:26:57.779+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:26:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:26:57.779+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:26:57.781 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:26:57.779+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:26:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:58 smithi134 ceph-mon[31966]: pgmap v10217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T16:26:58.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:58 smithi189 ceph-mon[36790]: pgmap v10217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T16:26:58.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:58 smithi193 ceph-mon[37473]: pgmap v10217: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T16:26:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:26:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17960.456533 secs 2024-07-26T16:26:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:26:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17960.456533 secs 2024-07-26T16:26:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:26:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17960.456533 secs 2024-07-26T16:27:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:00 smithi134 ceph-mon[31966]: pgmap v10218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:27:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:00 smithi189 ceph-mon[36790]: pgmap v10218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:27:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:00 smithi193 ceph-mon[37473]: pgmap v10218: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:27:02.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:02.165+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:27:02.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:02.165+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:27:02.167 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:02.165+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:02 smithi134 ceph-mon[31966]: pgmap v10219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:27:02.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17979.877753 secs 2024-07-26T16:27:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:02 smithi189 ceph-mon[36790]: pgmap v10219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:27:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17979.877753 secs 2024-07-26T16:27:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:02 smithi193 ceph-mon[37473]: pgmap v10219: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:27:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17979.877753 secs 2024-07-26T16:27:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:04 smithi134 ceph-mon[31966]: pgmap v10220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:27:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17965.456659 secs 2024-07-26T16:27:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:04 smithi189 ceph-mon[36790]: pgmap v10220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:27:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17965.456659 secs 2024-07-26T16:27:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:04 smithi193 ceph-mon[37473]: pgmap v10220: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:27:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17965.456659 secs 2024-07-26T16:27:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:06 smithi134 ceph-mon[31966]: pgmap v10221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:27:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:06 smithi189 ceph-mon[36790]: pgmap v10221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:27:06.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:06 smithi193 ceph-mon[37473]: pgmap v10221: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:27:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17984.877875 secs 2024-07-26T16:27:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17984.877875 secs 2024-07-26T16:27:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17984.877875 secs 2024-07-26T16:27:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:08 smithi134 ceph-mon[31966]: pgmap v10222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:08 smithi189 ceph-mon[36790]: pgmap v10222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:08 smithi193 ceph-mon[37473]: pgmap v10222: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17970.456801 secs 2024-07-26T16:27:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17970.456801 secs 2024-07-26T16:27:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17970.456801 secs 2024-07-26T16:27:10.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:10 smithi189 ceph-mon[36790]: pgmap v10223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:10 smithi193 ceph-mon[37473]: pgmap v10223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:10 smithi134 ceph-mon[31966]: pgmap v10223: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:12 smithi189 ceph-mon[36790]: pgmap v10224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17989.878017 secs 2024-07-26T16:27:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:12 smithi193 ceph-mon[37473]: pgmap v10224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17989.878017 secs 2024-07-26T16:27:12.796 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:12.795+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:27:12.797 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:12.795+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:12.797 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:12.795+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:27:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:12 smithi134 ceph-mon[31966]: pgmap v10224: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:12.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17989.878017 secs 2024-07-26T16:27:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:14 smithi189 ceph-mon[36790]: pgmap v10225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:27:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17975.456921 secs 2024-07-26T16:27:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:14 smithi193 ceph-mon[37473]: pgmap v10225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:27:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17975.456921 secs 2024-07-26T16:27:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:14 smithi134 ceph-mon[31966]: pgmap v10225: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:27:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17975.456921 secs 2024-07-26T16:27:16.725 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:16 smithi134 ceph-mon[31966]: pgmap v10226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:16 smithi193 ceph-mon[37473]: pgmap v10226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:16 smithi189 ceph-mon[36790]: pgmap v10226: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:17.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:17.170+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:27:17.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:17.170+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:27:17.173 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:17.171+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17994.878137 secs 2024-07-26T16:27:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17994.878137 secs 2024-07-26T16:27:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17994.878137 secs 2024-07-26T16:27:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:18 smithi193 ceph-mon[37473]: pgmap v10227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:18 smithi134 ceph-mon[31966]: pgmap v10227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:18 smithi189 ceph-mon[36790]: pgmap v10227: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17980.457019 secs 2024-07-26T16:27:19.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17980.457019 secs 2024-07-26T16:27:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17980.457019 secs 2024-07-26T16:27:20.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:20 smithi193 ceph-mon[37473]: pgmap v10228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:20 smithi134 ceph-mon[31966]: pgmap v10228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:20.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:20 smithi189 ceph-mon[36790]: pgmap v10228: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:22.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:22.350+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 msgr2=0x7f9b300c0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T16:27:22.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:22.350+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 msgr2=0x7f9b300c0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T16:27:22.353 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:22.350+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 0x7f9b300c0be0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b38061040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:27:22.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:22.350+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 0x7f9b300c0be0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b38061040 comp rx=0 tx=0).stop 2024-07-26T16:27:22.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:22.350+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 msgr2=0x7f9b300c0be0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T16:27:22.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:22.350+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 msgr2=0x7f9b300c0be0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T16:27:22.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:22.350+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1bf0 0x7f9b300c0be0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T16:27:22.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:22.350+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 0x7f9b300c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T16:27:22.354 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:22.351+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 0x7f9b300c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:27:22.355 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:22.353+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 0x7f9b300c15d0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f9b3806d000 tx=0x7f9b38043000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T16:27:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:22 smithi134 ceph-mon[31966]: pgmap v10229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 17999.878257 secs 2024-07-26T16:27:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:22 smithi193 ceph-mon[37473]: pgmap v10229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 17999.878257 secs 2024-07-26T16:27:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:22 smithi189 ceph-mon[36790]: pgmap v10229: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 17999.878257 secs 2024-07-26T16:27:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:24 smithi193 ceph-mon[37473]: pgmap v10230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:27:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17985.457155 secs 2024-07-26T16:27:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:24 smithi134 ceph-mon[31966]: pgmap v10230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:27:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17985.457155 secs 2024-07-26T16:27:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:24 smithi189 ceph-mon[36790]: pgmap v10230: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:27:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17985.457155 secs 2024-07-26T16:27:25.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:25.451+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T16:27:25.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:25.451+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T16:27:25.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:25.452+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 140 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T16:27:25.454 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:25.452+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T16:27:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:26 smithi193 ceph-mon[37473]: pgmap v10231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:26 smithi134 ceph-mon[31966]: pgmap v10231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:26 smithi189 ceph-mon[36790]: pgmap v10231: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18004.878398 secs 2024-07-26T16:27:27.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:27.804+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:27:27.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:27.804+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:27.806 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:27.804+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:27:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18004.878398 secs 2024-07-26T16:27:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18004.878398 secs 2024-07-26T16:27:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:28 smithi193 ceph-mon[37473]: pgmap v10232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:28 smithi134 ceph-mon[31966]: pgmap v10232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:28 smithi189 ceph-mon[36790]: pgmap v10232: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17990.457258 secs 2024-07-26T16:27:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17990.457258 secs 2024-07-26T16:27:29.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17990.457258 secs 2024-07-26T16:27:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:30 smithi193 ceph-mon[37473]: pgmap v10233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:30 smithi134 ceph-mon[31966]: pgmap v10233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:30 smithi189 ceph-mon[36790]: pgmap v10233: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:32.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:32.181+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:27:32.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:32.181+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:32.184 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:32.181+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:27:32.906 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:32.902+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 msgr2=0x7f21ac0c10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000927) for more than 900000000 us, fault. 2024-07-26T16:27:32.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:32.902+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f21b8005f70 tx=0x7f21b80c3000 comp rx=0 tx=0).stop 2024-07-26T16:27:32.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:32.902+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 msgr2=0x7f21ac0c10b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T16:27:32.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:32.902+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0850 0x7f21ac0c10b0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T16:27:32.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:32.902+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T16:27:32.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:32.903+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:27:32.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:32.904+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f21980038d0 tx=0x7f21b8063000 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T16:27:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:32 smithi134 ceph-mon[31966]: pgmap v10234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:32.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18009.878541 secs 2024-07-26T16:27:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:32 smithi189 ceph-mon[36790]: pgmap v10234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18009.878541 secs 2024-07-26T16:27:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:32 smithi193 ceph-mon[37473]: pgmap v10234: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18009.878541 secs 2024-07-26T16:27:34.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:34 smithi193 ceph-mon[37473]: pgmap v10235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:27:34.826 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 17995.457359 secs 2024-07-26T16:27:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:34 smithi134 ceph-mon[31966]: pgmap v10235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:27:34.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 17995.457359 secs 2024-07-26T16:27:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:34 smithi189 ceph-mon[36790]: pgmap v10235: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:27:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 17995.457359 secs 2024-07-26T16:27:36.009 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:36.007+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:27:36.009 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:36.007+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:27:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:36.008+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:27:36.010 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:36.008+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:27:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:36 smithi134 ceph-mon[31966]: pgmap v10236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:36 smithi189 ceph-mon[36790]: pgmap v10236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:36 smithi193 ceph-mon[37473]: pgmap v10236: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18014.878669 secs 2024-07-26T16:27:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18014.878669 secs 2024-07-26T16:27:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18014.878669 secs 2024-07-26T16:27:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:38 smithi134 ceph-mon[31966]: pgmap v10237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:38.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:38 smithi189 ceph-mon[36790]: pgmap v10237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:38 smithi193 ceph-mon[37473]: pgmap v10237: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18000.457485 secs 2024-07-26T16:27:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18000.457485 secs 2024-07-26T16:27:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18000.457485 secs 2024-07-26T16:27:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:40 smithi134 ceph-mon[31966]: pgmap v10238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:40.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:40 smithi189 ceph-mon[36790]: pgmap v10238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:40 smithi193 ceph-mon[37473]: pgmap v10238: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18019.878794 secs 2024-07-26T16:27:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18019.878794 secs 2024-07-26T16:27:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18019.878794 secs 2024-07-26T16:27:42.821 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:42.820+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:27:42.822 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:42.820+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:42.822 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:42.820+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:27:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:42 smithi134 ceph-mon[31966]: pgmap v10239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:42 smithi189 ceph-mon[36790]: pgmap v10239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:42 smithi193 ceph-mon[37473]: pgmap v10239: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:44 smithi134 ceph-mon[31966]: pgmap v10240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:27:44.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18005.457630 secs 2024-07-26T16:27:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:44 smithi189 ceph-mon[36790]: pgmap v10240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:27:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18005.457630 secs 2024-07-26T16:27:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:44 smithi193 ceph-mon[37473]: pgmap v10240: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:27:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18005.457630 secs 2024-07-26T16:27:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:46 smithi134 ceph-mon[31966]: pgmap v10241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:46.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18024.878879 secs 2024-07-26T16:27:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:46 smithi189 ceph-mon[36790]: pgmap v10241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18024.878879 secs 2024-07-26T16:27:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:46 smithi193 ceph-mon[37473]: pgmap v10241: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18024.878879 secs 2024-07-26T16:27:47.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:47.197+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:27:47.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:47.197+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:47.199 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:27:47.197+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:27:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:48 smithi134 ceph-mon[31966]: pgmap v10242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:48.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:48 smithi189 ceph-mon[36790]: pgmap v10242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:48 smithi193 ceph-mon[37473]: pgmap v10242: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:27:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18010.457751 secs 2024-07-26T16:27:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18010.457751 secs 2024-07-26T16:27:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18010.457751 secs 2024-07-26T16:27:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:50 smithi134 ceph-mon[31966]: pgmap v10243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:50.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:50 smithi189 ceph-mon[36790]: pgmap v10243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:50 smithi193 ceph-mon[37473]: pgmap v10243: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:27:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18029.879011 secs 2024-07-26T16:27:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18029.879011 secs 2024-07-26T16:27:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18029.879011 secs 2024-07-26T16:27:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:52 smithi134 ceph-mon[31966]: pgmap v10244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:52 smithi189 ceph-mon[36790]: pgmap v10244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:52 smithi193 ceph-mon[37473]: pgmap v10244: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:54 smithi134 ceph-mon[31966]: pgmap v10245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:27:54.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18015.457879 secs 2024-07-26T16:27:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:54 smithi189 ceph-mon[36790]: pgmap v10245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:27:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18015.457879 secs 2024-07-26T16:27:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:54 smithi193 ceph-mon[37473]: pgmap v10245: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:27:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18015.457879 secs 2024-07-26T16:27:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:55 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:27:55.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:55 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:27:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:55 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:27:56.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:56 smithi134 ceph-mon[31966]: pgmap v10246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:56.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18034.879103 secs 2024-07-26T16:27:56.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:27:56.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:27:56.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:27:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:56 smithi189 ceph-mon[36790]: pgmap v10246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18034.879103 secs 2024-07-26T16:27:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:27:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:27:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:27:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:56 smithi193 ceph-mon[37473]: pgmap v10246: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:27:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18034.879103 secs 2024-07-26T16:27:57.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:27:57.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:27:57.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:27:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:57.821+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:27:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:57.821+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:27:57.822 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:27:57.821+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:27:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:57 smithi134 ceph-mon[31966]: pgmap v10247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:27:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:57 smithi134 ceph-mon[31966]: pgmap v10248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:27:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:57 smithi189 ceph-mon[36790]: pgmap v10247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:27:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:57 smithi189 ceph-mon[36790]: pgmap v10248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:27:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:57 smithi193 ceph-mon[37473]: pgmap v10247: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:27:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:57 smithi193 ceph-mon[37473]: pgmap v10248: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:27:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:58 smithi193 ceph-mon[37473]: pgmap v10249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:27:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:58 smithi134 ceph-mon[31966]: pgmap v10249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:27:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:58 smithi189 ceph-mon[36790]: pgmap v10249: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:28:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:27:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18020.457959 secs 2024-07-26T16:28:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:27:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18020.457959 secs 2024-07-26T16:28:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:27:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18020.457959 secs 2024-07-26T16:28:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:00 smithi193 ceph-mon[37473]: pgmap v10250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:28:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:00 smithi134 ceph-mon[31966]: pgmap v10250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:28:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:00 smithi189 ceph-mon[36790]: pgmap v10250: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:28:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18039.879240 secs 2024-07-26T16:28:02.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18039.879240 secs 2024-07-26T16:28:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18039.879240 secs 2024-07-26T16:28:02.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:28:02.200+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:28:02.202 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:28:02.200+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:28:02.203 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:28:02.200+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:28:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:02 smithi193 ceph-mon[37473]: pgmap v10251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T16:28:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:02 smithi134 ceph-mon[31966]: pgmap v10251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T16:28:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:02 smithi189 ceph-mon[36790]: pgmap v10251: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T16:28:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18025.458063 secs 2024-07-26T16:28:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18025.458063 secs 2024-07-26T16:28:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18025.458063 secs 2024-07-26T16:28:05.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:05 smithi134 ceph-mon[31966]: pgmap v10252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:28:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:05 smithi189 ceph-mon[36790]: pgmap v10252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:28:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:05 smithi193 ceph-mon[37473]: pgmap v10252: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:28:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18044.879336 secs 2024-07-26T16:28:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:07 smithi134 ceph-mon[31966]: pgmap v10253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:28:07.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18044.879336 secs 2024-07-26T16:28:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:07 smithi189 ceph-mon[36790]: pgmap v10253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:28:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18044.879336 secs 2024-07-26T16:28:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:07 smithi193 ceph-mon[37473]: pgmap v10253: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:28:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:09 smithi134 ceph-mon[31966]: pgmap v10254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:09.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18030.458132 secs 2024-07-26T16:28:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:09 smithi189 ceph-mon[36790]: pgmap v10254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18030.458132 secs 2024-07-26T16:28:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:09 smithi193 ceph-mon[37473]: pgmap v10254: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18030.458132 secs 2024-07-26T16:28:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:11 smithi134 ceph-mon[31966]: pgmap v10255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:11.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:11 smithi189 ceph-mon[36790]: pgmap v10255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:11 smithi193 ceph-mon[37473]: pgmap v10255: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18049.879458 secs 2024-07-26T16:28:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18049.879458 secs 2024-07-26T16:28:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18049.879458 secs 2024-07-26T16:28:12.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:28:12.836+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:28:12.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:28:12.836+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:28:12.838 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:28:12.836+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:28:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:13 smithi134 ceph-mon[31966]: pgmap v10256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:13.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:13 smithi189 ceph-mon[36790]: pgmap v10256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:13 smithi193 ceph-mon[37473]: pgmap v10256: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18035.458267 secs 2024-07-26T16:28:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18035.458267 secs 2024-07-26T16:28:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18035.458267 secs 2024-07-26T16:28:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:15 smithi134 ceph-mon[31966]: pgmap v10257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:15 smithi189 ceph-mon[36790]: pgmap v10257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:15 smithi193 ceph-mon[37473]: pgmap v10257: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:17.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:28:17.215+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:28:17.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:28:17.216+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:28:17.218 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:28:17.216+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:28:17.475 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18054.879580 secs 2024-07-26T16:28:17.475 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:17 smithi134 ceph-mon[31966]: pgmap v10258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18054.879580 secs 2024-07-26T16:28:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:17 smithi189 ceph-mon[36790]: pgmap v10258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18054.879580 secs 2024-07-26T16:28:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:17 smithi193 ceph-mon[37473]: pgmap v10258: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:18 smithi193 ceph-mon[37473]: pgmap v10259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:18 smithi134 ceph-mon[31966]: pgmap v10259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:18 smithi189 ceph-mon[36790]: pgmap v10259: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18040.458397 secs 2024-07-26T16:28:20.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18040.458397 secs 2024-07-26T16:28:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18040.458397 secs 2024-07-26T16:28:21.001 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:20 smithi134 ceph-mon[31966]: pgmap v10260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:20 smithi193 ceph-mon[37473]: pgmap v10260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:20 smithi189 ceph-mon[36790]: pgmap v10260: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18059.879678 secs 2024-07-26T16:28:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18059.879678 secs 2024-07-26T16:28:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18059.879678 secs 2024-07-26T16:28:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:22 smithi193 ceph-mon[37473]: pgmap v10261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:22 smithi134 ceph-mon[31966]: pgmap v10261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:23.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:22 smithi189 ceph-mon[36790]: pgmap v10261: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18045.458543 secs 2024-07-26T16:28:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18045.458543 secs 2024-07-26T16:28:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18045.458543 secs 2024-07-26T16:28:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:25 smithi134 ceph-mon[31966]: pgmap v10262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:25.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:25 smithi189 ceph-mon[36790]: pgmap v10262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:25 smithi193 ceph-mon[37473]: pgmap v10262: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18064.879812 secs 2024-07-26T16:28:27.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:27 smithi134 ceph-mon[31966]: pgmap v10263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18064.879812 secs 2024-07-26T16:28:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:27 smithi189 ceph-mon[36790]: pgmap v10263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18064.879812 secs 2024-07-26T16:28:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:27 smithi193 ceph-mon[37473]: pgmap v10263: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:27.842 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:28:27.840+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:28:27.842 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:28:27.840+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:28:27.842 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:28:27.840+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:28:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:29 smithi134 ceph-mon[31966]: pgmap v10264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:29.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18050.458677 secs 2024-07-26T16:28:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:29 smithi189 ceph-mon[36790]: pgmap v10264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18050.458677 secs 2024-07-26T16:28:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:29 smithi193 ceph-mon[37473]: pgmap v10264: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18050.458677 secs 2024-07-26T16:28:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:31 smithi134 ceph-mon[31966]: pgmap v10265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:31 smithi189 ceph-mon[36790]: pgmap v10265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:31 smithi193 ceph-mon[37473]: pgmap v10265: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:32.228 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:28:32.226+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:28:32.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:28:32.226+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:28:32.229 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:28:32.226+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:28:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18069.879930 secs 2024-07-26T16:28:32.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18069.879930 secs 2024-07-26T16:28:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18069.879930 secs 2024-07-26T16:28:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:33 smithi134 ceph-mon[31966]: pgmap v10266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:33.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:33 smithi189 ceph-mon[36790]: pgmap v10266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:33 smithi193 ceph-mon[37473]: pgmap v10266: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18055.458825 secs 2024-07-26T16:28:34.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18055.458825 secs 2024-07-26T16:28:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18055.458825 secs 2024-07-26T16:28:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:35 smithi134 ceph-mon[31966]: pgmap v10267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:35.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:35 smithi189 ceph-mon[36790]: pgmap v10267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:35 smithi193 ceph-mon[37473]: pgmap v10267: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18074.880070 secs 2024-07-26T16:28:37.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:37 smithi134 ceph-mon[31966]: pgmap v10268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18074.880070 secs 2024-07-26T16:28:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:37 smithi189 ceph-mon[36790]: pgmap v10268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18074.880070 secs 2024-07-26T16:28:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:37 smithi193 ceph-mon[37473]: pgmap v10268: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:39 smithi134 ceph-mon[31966]: pgmap v10269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:39.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18060.458991 secs 2024-07-26T16:28:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:39 smithi189 ceph-mon[36790]: pgmap v10269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18060.458991 secs 2024-07-26T16:28:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:39 smithi193 ceph-mon[37473]: pgmap v10269: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18060.458991 secs 2024-07-26T16:28:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:41 smithi189 ceph-mon[36790]: pgmap v10270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:41 smithi193 ceph-mon[37473]: pgmap v10270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:41 smithi134 ceph-mon[31966]: pgmap v10270: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18079.880210 secs 2024-07-26T16:28:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18079.880210 secs 2024-07-26T16:28:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:28:42.855+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:28:42.857 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:28:42.855+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:28:42.858 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:28:42.855+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:28:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18079.880210 secs 2024-07-26T16:28:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:43 smithi189 ceph-mon[36790]: pgmap v10271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:43 smithi193 ceph-mon[37473]: pgmap v10271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:43 smithi134 ceph-mon[31966]: pgmap v10271: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18065.459100 secs 2024-07-26T16:28:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18065.459100 secs 2024-07-26T16:28:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18065.459100 secs 2024-07-26T16:28:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:45 smithi193 ceph-mon[37473]: pgmap v10272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:45 smithi134 ceph-mon[31966]: pgmap v10272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:45 smithi189 ceph-mon[36790]: pgmap v10272: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:47.240 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:28:47.238+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:28:47.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:28:47.238+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:28:47.241 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:28:47.238+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:28:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18084.880333 secs 2024-07-26T16:28:47.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:47 smithi134 ceph-mon[31966]: pgmap v10273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18084.880333 secs 2024-07-26T16:28:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:47 smithi193 ceph-mon[37473]: pgmap v10273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18084.880333 secs 2024-07-26T16:28:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:47 smithi189 ceph-mon[36790]: pgmap v10273: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:49 smithi193 ceph-mon[37473]: pgmap v10274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:49.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18070.459166 secs 2024-07-26T16:28:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:49 smithi134 ceph-mon[31966]: pgmap v10274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18070.459166 secs 2024-07-26T16:28:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:49 smithi189 ceph-mon[36790]: pgmap v10274: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:28:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18070.459166 secs 2024-07-26T16:28:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:51 smithi193 ceph-mon[37473]: pgmap v10275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:51 smithi134 ceph-mon[31966]: pgmap v10275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:51 smithi189 ceph-mon[36790]: pgmap v10275: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18089.880453 secs 2024-07-26T16:28:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18089.880453 secs 2024-07-26T16:28:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18089.880453 secs 2024-07-26T16:28:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:53 smithi193 ceph-mon[37473]: pgmap v10276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:53 smithi134 ceph-mon[31966]: pgmap v10276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:53 smithi189 ceph-mon[36790]: pgmap v10276: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:28:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18075.459248 secs 2024-07-26T16:28:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18075.459248 secs 2024-07-26T16:28:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18075.459248 secs 2024-07-26T16:28:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:55 smithi193 ceph-mon[37473]: pgmap v10277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:55 smithi134 ceph-mon[31966]: pgmap v10277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:55.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:55 smithi189 ceph-mon[36790]: pgmap v10277: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:28:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:56 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:28:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:56 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:28:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:56 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:28:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:28:57.869+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:28:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:28:57.870+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:28:57.872 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:28:57.870+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:28:57.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18094.880529 secs 2024-07-26T16:28:57.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:57 smithi134 ceph-mon[31966]: pgmap v10278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:57.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:57.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:57.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:57.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:57.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:28:57.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:28:57.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:28:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18094.880529 secs 2024-07-26T16:28:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:57 smithi189 ceph-mon[36790]: pgmap v10278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:28:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:28:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:28:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18094.880529 secs 2024-07-26T16:28:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:57 smithi193 ceph-mon[37473]: pgmap v10278: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:28:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:58.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:58.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:58.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:28:58.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:28:58.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:28:58.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:28:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:58 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T16:28:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:58 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T16:28:58.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:58 smithi134 ceph-mon[31966]: pgmap v10279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 8 op/s 2024-07-26T16:28:58.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:58 smithi134 ceph-mon[31966]: pgmap v10280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:28:58.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:58 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T16:28:58.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:58 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T16:28:58.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:58 smithi189 ceph-mon[36790]: pgmap v10279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 8 op/s 2024-07-26T16:28:58.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:58 smithi189 ceph-mon[36790]: pgmap v10280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:28:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:58 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T16:28:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:58 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T16:28:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:58 smithi193 ceph-mon[37473]: pgmap v10279: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 8 op/s 2024-07-26T16:28:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:58 smithi193 ceph-mon[37473]: pgmap v10280: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:28:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:28:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18080.459363 secs 2024-07-26T16:28:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:28:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18080.459363 secs 2024-07-26T16:29:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:28:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18080.459363 secs 2024-07-26T16:29:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:00 smithi134 ceph-mon[31966]: pgmap v10281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:29:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:00 smithi189 ceph-mon[36790]: pgmap v10281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:29:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:00 smithi193 ceph-mon[37473]: pgmap v10281: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:29:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18099.880631 secs 2024-07-26T16:29:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18099.880631 secs 2024-07-26T16:29:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18099.880631 secs 2024-07-26T16:29:02.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:29:02.254+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:29:02.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:29:02.254+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:29:02.256 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:29:02.254+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:02 smithi134 ceph-mon[31966]: pgmap v10282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:29:02.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:02 smithi189 ceph-mon[36790]: pgmap v10282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:29:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:02 smithi193 ceph-mon[37473]: pgmap v10282: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:29:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:04 smithi134 ceph-mon[31966]: pgmap v10283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:29:04.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18085.459462 secs 2024-07-26T16:29:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:04 smithi189 ceph-mon[36790]: pgmap v10283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:29:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18085.459462 secs 2024-07-26T16:29:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:04 smithi193 ceph-mon[37473]: pgmap v10283: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:29:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18085.459462 secs 2024-07-26T16:29:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:06 smithi134 ceph-mon[31966]: pgmap v10284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:29:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18104.880777 secs 2024-07-26T16:29:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:06 smithi189 ceph-mon[36790]: pgmap v10284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:29:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18104.880777 secs 2024-07-26T16:29:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:06 smithi193 ceph-mon[37473]: pgmap v10284: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:29:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18104.880777 secs 2024-07-26T16:29:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:08 smithi134 ceph-mon[31966]: pgmap v10285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:29:08.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:08 smithi189 ceph-mon[36790]: pgmap v10285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:29:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:08 smithi193 ceph-mon[37473]: pgmap v10285: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:29:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18090.459588 secs 2024-07-26T16:29:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18090.459588 secs 2024-07-26T16:29:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18090.459588 secs 2024-07-26T16:29:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:10 smithi134 ceph-mon[31966]: pgmap v10286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:10 smithi189 ceph-mon[36790]: pgmap v10286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:10 smithi193 ceph-mon[37473]: pgmap v10286: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18109.880899 secs 2024-07-26T16:29:11.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18109.880899 secs 2024-07-26T16:29:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18109.880899 secs 2024-07-26T16:29:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:29:12.885+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:29:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:29:12.885+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:12.887 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:29:12.885+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:29:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:12 smithi134 ceph-mon[31966]: pgmap v10287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:12 smithi189 ceph-mon[36790]: pgmap v10287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:12 smithi193 ceph-mon[37473]: pgmap v10287: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:14 smithi134 ceph-mon[31966]: pgmap v10288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18095.459706 secs 2024-07-26T16:29:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:14 smithi189 ceph-mon[36790]: pgmap v10288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18095.459706 secs 2024-07-26T16:29:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:14 smithi193 ceph-mon[37473]: pgmap v10288: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18095.459706 secs 2024-07-26T16:29:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:16 smithi134 ceph-mon[31966]: pgmap v10289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18114.881018 secs 2024-07-26T16:29:16.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:16 smithi189 ceph-mon[36790]: pgmap v10289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18114.881018 secs 2024-07-26T16:29:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:16 smithi193 ceph-mon[37473]: pgmap v10289: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18114.881018 secs 2024-07-26T16:29:17.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:29:17.259+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:29:17.261 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:29:17.259+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:29:17.262 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:29:17.259+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:18 smithi134 ceph-mon[31966]: pgmap v10290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:29:18.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:18 smithi189 ceph-mon[36790]: pgmap v10290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:29:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:18 smithi193 ceph-mon[37473]: pgmap v10290: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:29:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18100.459786 secs 2024-07-26T16:29:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18100.459786 secs 2024-07-26T16:29:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18100.459786 secs 2024-07-26T16:29:20.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:20 smithi189 ceph-mon[36790]: pgmap v10291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:20 smithi193 ceph-mon[37473]: pgmap v10291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:20 smithi134 ceph-mon[31966]: pgmap v10291: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18119.881137 secs 2024-07-26T16:29:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18119.881137 secs 2024-07-26T16:29:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18119.881137 secs 2024-07-26T16:29:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:22 smithi189 ceph-mon[36790]: pgmap v10292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:22 smithi193 ceph-mon[37473]: pgmap v10292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:23.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:22 smithi134 ceph-mon[31966]: pgmap v10292: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:23 smithi193 ceph-mon[37473]: pgmap v10293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:23 smithi134 ceph-mon[31966]: pgmap v10293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:24.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:23 smithi189 ceph-mon[36790]: pgmap v10293: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18105.459897 secs 2024-07-26T16:29:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18105.459897 secs 2024-07-26T16:29:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18105.459897 secs 2024-07-26T16:29:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:25 smithi193 ceph-mon[37473]: pgmap v10294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:25 smithi134 ceph-mon[31966]: pgmap v10294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:25 smithi189 ceph-mon[36790]: pgmap v10294: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18124.881255 secs 2024-07-26T16:29:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18124.881255 secs 2024-07-26T16:29:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18124.881255 secs 2024-07-26T16:29:27.896 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:29:27.894+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:29:27.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:29:27.894+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:27.897 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:29:27.894+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:29:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:27 smithi193 ceph-mon[37473]: pgmap v10295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:29:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:27 smithi134 ceph-mon[31966]: pgmap v10295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:29:28.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:27 smithi189 ceph-mon[36790]: pgmap v10295: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:29:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18110.459985 secs 2024-07-26T16:29:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18110.459985 secs 2024-07-26T16:29:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18110.459985 secs 2024-07-26T16:29:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:30 smithi134 ceph-mon[31966]: pgmap v10296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:30 smithi189 ceph-mon[36790]: pgmap v10296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:30 smithi193 ceph-mon[37473]: pgmap v10296: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:32.271 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:29:32.269+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:29:32.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:29:32.269+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:29:32.272 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:29:32.269+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18129.881374 secs 2024-07-26T16:29:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:32 smithi134 ceph-mon[31966]: pgmap v10297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18129.881374 secs 2024-07-26T16:29:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:32 smithi189 ceph-mon[36790]: pgmap v10297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18129.881374 secs 2024-07-26T16:29:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:32 smithi193 ceph-mon[37473]: pgmap v10297: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:34.827 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:34 smithi193 ceph-mon[37473]: pgmap v10298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:34.827 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18115.460059 secs 2024-07-26T16:29:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:34 smithi134 ceph-mon[31966]: pgmap v10298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:34.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18115.460059 secs 2024-07-26T16:29:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:34 smithi189 ceph-mon[36790]: pgmap v10298: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18115.460059 secs 2024-07-26T16:29:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:36 smithi134 ceph-mon[31966]: pgmap v10299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:36 smithi189 ceph-mon[36790]: pgmap v10299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:36 smithi193 ceph-mon[37473]: pgmap v10299: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18134.881492 secs 2024-07-26T16:29:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18134.881492 secs 2024-07-26T16:29:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18134.881492 secs 2024-07-26T16:29:38.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:38 smithi134 ceph-mon[31966]: pgmap v10300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:29:38.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:38 smithi189 ceph-mon[36790]: pgmap v10300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:29:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:38 smithi193 ceph-mon[37473]: pgmap v10300: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:29:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18120.460175 secs 2024-07-26T16:29:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18120.460175 secs 2024-07-26T16:29:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18120.460175 secs 2024-07-26T16:29:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:40 smithi134 ceph-mon[31966]: pgmap v10301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:29:40.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:40 smithi189 ceph-mon[36790]: pgmap v10301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:29:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:40 smithi193 ceph-mon[37473]: pgmap v10301: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:29:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18139.881600 secs 2024-07-26T16:29:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18139.881600 secs 2024-07-26T16:29:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18139.881600 secs 2024-07-26T16:29:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:29:42.901+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:29:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:29:42.901+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:29:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:29:42.901+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:42 smithi134 ceph-mon[31966]: pgmap v10302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:29:42.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:42 smithi189 ceph-mon[36790]: pgmap v10302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:29:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:42 smithi193 ceph-mon[37473]: pgmap v10302: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:29:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:44 smithi134 ceph-mon[31966]: pgmap v10303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:29:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18125.460308 secs 2024-07-26T16:29:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:44 smithi189 ceph-mon[36790]: pgmap v10303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:29:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18125.460308 secs 2024-07-26T16:29:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:44 smithi193 ceph-mon[37473]: pgmap v10303: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:29:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18125.460308 secs 2024-07-26T16:29:46.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:46 smithi134 ceph-mon[31966]: pgmap v10304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:29:46.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18144.881701 secs 2024-07-26T16:29:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:46 smithi189 ceph-mon[36790]: pgmap v10304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:29:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18144.881701 secs 2024-07-26T16:29:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:46 smithi193 ceph-mon[37473]: pgmap v10304: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:29:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18144.881701 secs 2024-07-26T16:29:47.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:29:47.284+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:29:47.286 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:29:47.284+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:29:47.287 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:29:47.284+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:48 smithi134 ceph-mon[31966]: pgmap v10305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:29:48.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:48 smithi189 ceph-mon[36790]: pgmap v10305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:29:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:48 smithi193 ceph-mon[37473]: pgmap v10305: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:29:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18130.460431 secs 2024-07-26T16:29:49.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18130.460431 secs 2024-07-26T16:29:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18130.460431 secs 2024-07-26T16:29:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:50 smithi134 ceph-mon[31966]: pgmap v10306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:50.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:50 smithi189 ceph-mon[36790]: pgmap v10306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:50 smithi193 ceph-mon[37473]: pgmap v10306: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18149.881800 secs 2024-07-26T16:29:51.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18149.881800 secs 2024-07-26T16:29:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18149.881800 secs 2024-07-26T16:29:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:52 smithi134 ceph-mon[31966]: pgmap v10307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:52 smithi189 ceph-mon[36790]: pgmap v10307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:52 smithi193 ceph-mon[37473]: pgmap v10307: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:54.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:54 smithi134 ceph-mon[31966]: pgmap v10308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:54.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18135.460522 secs 2024-07-26T16:29:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:54 smithi189 ceph-mon[36790]: pgmap v10308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18135.460522 secs 2024-07-26T16:29:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:54 smithi193 ceph-mon[37473]: pgmap v10308: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:29:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18135.460522 secs 2024-07-26T16:29:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:56 smithi189 ceph-mon[36790]: pgmap v10309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18154.881920 secs 2024-07-26T16:29:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:56 smithi193 ceph-mon[37473]: pgmap v10309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18154.881920 secs 2024-07-26T16:29:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:56 smithi134 ceph-mon[31966]: pgmap v10309: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:29:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18154.881920 secs 2024-07-26T16:29:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:29:57.916+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:29:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:29:57.916+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:29:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:29:57.916+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:29:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:57 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:29:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:57 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:29:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:57 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:29:58.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:58 smithi189 ceph-mon[36790]: pgmap v10310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:29:58.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:58.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:58.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:58.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:58.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:58.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:58.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:58.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:58.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:58 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:29:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:58 smithi193 ceph-mon[37473]: pgmap v10310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:29:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:59.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:59.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:59.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:59.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:59.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:59.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:59.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:58 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:29:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:58 smithi134 ceph-mon[31966]: pgmap v10310: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 8 op/s 2024-07-26T16:29:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:29:59.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:58 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:30:00.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:59 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T16:30:00.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:59 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T16:30:00.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:59 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T16:30:00.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:59 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T16:30:00.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18140.460674 secs 2024-07-26T16:30:00.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:30:00.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:59 smithi193 ceph-mon[37473]: pgmap v10311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:30:00.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:59 smithi193 ceph-mon[37473]: pgmap v10312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:30:00.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:29:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:30:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:59 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T16:30:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:59 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T16:30:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:59 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T16:30:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:59 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T16:30:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18140.460674 secs 2024-07-26T16:30:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:30:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:59 smithi134 ceph-mon[31966]: pgmap v10311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:30:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:59 smithi134 ceph-mon[31966]: pgmap v10312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:30:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:29:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:30:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:59 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T16:30:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:59 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T16:30:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:59 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T16:30:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:59 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T16:30:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18140.460674 secs 2024-07-26T16:30:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:30:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:59 smithi189 ceph-mon[36790]: pgmap v10311: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:30:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:59 smithi189 ceph-mon[36790]: pgmap v10312: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:30:00.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:29:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:30:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T16:30:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T16:30:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T16:30:02.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:01 smithi193 ceph-mon[37473]: pgmap v10313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:30:02.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18159.882044 secs 2024-07-26T16:30:02.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:01 smithi189 ceph-mon[36790]: pgmap v10313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:30:02.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18159.882044 secs 2024-07-26T16:30:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:01 smithi134 ceph-mon[31966]: pgmap v10313: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:30:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18159.882044 secs 2024-07-26T16:30:02.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:30:02.300+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:30:02.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:30:02.300+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:30:02.302 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:30:02.300+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:30:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:03 smithi193 ceph-mon[37473]: pgmap v10314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:30:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:03 smithi134 ceph-mon[31966]: pgmap v10314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:30:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:03 smithi189 ceph-mon[36790]: pgmap v10314: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:30:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18145.460764 secs 2024-07-26T16:30:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18145.460764 secs 2024-07-26T16:30:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18145.460764 secs 2024-07-26T16:30:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:05 smithi134 ceph-mon[31966]: pgmap v10315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:30:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:05 smithi189 ceph-mon[36790]: pgmap v10315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:30:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:05 smithi193 ceph-mon[37473]: pgmap v10315: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:30:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18164.882121 secs 2024-07-26T16:30:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18164.882121 secs 2024-07-26T16:30:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18164.882121 secs 2024-07-26T16:30:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:07 smithi134 ceph-mon[31966]: pgmap v10316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 6 op/s 2024-07-26T16:30:08.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:07 smithi189 ceph-mon[36790]: pgmap v10316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 6 op/s 2024-07-26T16:30:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:07 smithi193 ceph-mon[37473]: pgmap v10316: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 6 op/s 2024-07-26T16:30:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18150.460903 secs 2024-07-26T16:30:10.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:09 smithi134 ceph-mon[31966]: pgmap v10317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:30:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18150.460903 secs 2024-07-26T16:30:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:09 smithi189 ceph-mon[36790]: pgmap v10317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:30:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18150.460903 secs 2024-07-26T16:30:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:09 smithi193 ceph-mon[37473]: pgmap v10317: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:30:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:11 smithi134 ceph-mon[31966]: pgmap v10318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:12.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18169.882244 secs 2024-07-26T16:30:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:11 smithi189 ceph-mon[36790]: pgmap v10318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18169.882244 secs 2024-07-26T16:30:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:11 smithi193 ceph-mon[37473]: pgmap v10318: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18169.882244 secs 2024-07-26T16:30:12.924 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:30:12.922+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:30:12.925 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:30:12.922+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:30:12.925 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:30:12.922+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:30:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:13 smithi134 ceph-mon[31966]: pgmap v10319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:13 smithi189 ceph-mon[36790]: pgmap v10319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:13 smithi193 ceph-mon[37473]: pgmap v10319: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18155.461024 secs 2024-07-26T16:30:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18155.461024 secs 2024-07-26T16:30:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18155.461024 secs 2024-07-26T16:30:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:15 smithi134 ceph-mon[31966]: pgmap v10320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:30:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:15 smithi189 ceph-mon[36790]: pgmap v10320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:30:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:15 smithi193 ceph-mon[37473]: pgmap v10320: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:30:17.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18174.882362 secs 2024-07-26T16:30:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18174.882362 secs 2024-07-26T16:30:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18174.882362 secs 2024-07-26T16:30:17.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:30:17.315+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:30:17.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:30:17.315+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:30:17.318 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:30:17.315+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:30:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:17 smithi134 ceph-mon[31966]: pgmap v10321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:17 smithi189 ceph-mon[36790]: pgmap v10321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:17 smithi193 ceph-mon[37473]: pgmap v10321: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18160.461102 secs 2024-07-26T16:30:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:19 smithi134 ceph-mon[31966]: pgmap v10322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:30:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18160.461102 secs 2024-07-26T16:30:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:19 smithi189 ceph-mon[36790]: pgmap v10322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:30:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18160.461102 secs 2024-07-26T16:30:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:19 smithi193 ceph-mon[37473]: pgmap v10322: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:30:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:21 smithi134 ceph-mon[31966]: pgmap v10323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:30:22.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18179.882484 secs 2024-07-26T16:30:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:21 smithi189 ceph-mon[36790]: pgmap v10323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:30:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18179.882484 secs 2024-07-26T16:30:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:21 smithi193 ceph-mon[37473]: pgmap v10323: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:30:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18179.882484 secs 2024-07-26T16:30:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:23 smithi134 ceph-mon[31966]: pgmap v10324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:30:24.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:23 smithi189 ceph-mon[36790]: pgmap v10324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:30:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:23 smithi193 ceph-mon[37473]: pgmap v10324: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:30:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18165.461170 secs 2024-07-26T16:30:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18165.461170 secs 2024-07-26T16:30:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18165.461170 secs 2024-07-26T16:30:25.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:30:25.483+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T16:30:25.486 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:30:25.483+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T16:30:25.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:30:25.484+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 141 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T16:30:25.487 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:30:25.484+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T16:30:26.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:25 smithi134 ceph-mon[31966]: pgmap v10325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:30:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:25 smithi189 ceph-mon[36790]: pgmap v10325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:30:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:25 smithi193 ceph-mon[37473]: pgmap v10325: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:30:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18184.882607 secs 2024-07-26T16:30:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18184.882607 secs 2024-07-26T16:30:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18184.882607 secs 2024-07-26T16:30:27.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:30:27.936+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:30:27.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:30:27.936+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:30:27.938 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:30:27.936+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:30:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:27 smithi134 ceph-mon[31966]: pgmap v10326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:30:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:27 smithi189 ceph-mon[36790]: pgmap v10326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:30:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:27 smithi193 ceph-mon[37473]: pgmap v10326: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:30:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18170.461250 secs 2024-07-26T16:30:30.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:29 smithi134 ceph-mon[31966]: pgmap v10327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:30:30.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18170.461250 secs 2024-07-26T16:30:30.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:29 smithi189 ceph-mon[36790]: pgmap v10327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:30:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18170.461250 secs 2024-07-26T16:30:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:29 smithi193 ceph-mon[37473]: pgmap v10327: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:30:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:31 smithi134 ceph-mon[31966]: pgmap v10328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:32.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18189.882727 secs 2024-07-26T16:30:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:31 smithi189 ceph-mon[36790]: pgmap v10328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18189.882727 secs 2024-07-26T16:30:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:31 smithi193 ceph-mon[37473]: pgmap v10328: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18189.882727 secs 2024-07-26T16:30:32.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:30:32.331+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:30:32.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:30:32.331+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:30:32.333 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:30:32.331+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:30:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:33 smithi134 ceph-mon[31966]: pgmap v10329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:33 smithi189 ceph-mon[36790]: pgmap v10329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:33 smithi193 ceph-mon[37473]: pgmap v10329: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18175.461376 secs 2024-07-26T16:30:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18175.461376 secs 2024-07-26T16:30:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18175.461376 secs 2024-07-26T16:30:36.040 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:30:36.038+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:30:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:30:36.038+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:30:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:30:36.038+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:30:36.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:30:36.038+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:30:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:35 smithi134 ceph-mon[31966]: pgmap v10330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:30:36.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:35 smithi189 ceph-mon[36790]: pgmap v10330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:30:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:35 smithi193 ceph-mon[37473]: pgmap v10330: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:30:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18194.882825 secs 2024-07-26T16:30:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18194.882825 secs 2024-07-26T16:30:37.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18194.882825 secs 2024-07-26T16:30:38.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:37 smithi189 ceph-mon[36790]: pgmap v10331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:37 smithi193 ceph-mon[37473]: pgmap v10331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:38.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:37 smithi134 ceph-mon[31966]: pgmap v10331: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18180.461513 secs 2024-07-26T16:30:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:39 smithi189 ceph-mon[36790]: pgmap v10332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18180.461513 secs 2024-07-26T16:30:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:39 smithi193 ceph-mon[37473]: pgmap v10332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18180.461513 secs 2024-07-26T16:30:40.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:39 smithi134 ceph-mon[31966]: pgmap v10332: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:41 smithi193 ceph-mon[37473]: pgmap v10333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:42.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18199.882889 secs 2024-07-26T16:30:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:41 smithi134 ceph-mon[31966]: pgmap v10333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18199.882889 secs 2024-07-26T16:30:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:41 smithi189 ceph-mon[36790]: pgmap v10333: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18199.882889 secs 2024-07-26T16:30:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:30:42.951+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:30:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:30:42.951+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:30:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:30:42.951+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:30:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:43 smithi193 ceph-mon[37473]: pgmap v10334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:43 smithi134 ceph-mon[31966]: pgmap v10334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:43 smithi189 ceph-mon[36790]: pgmap v10334: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18185.461581 secs 2024-07-26T16:30:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18185.461581 secs 2024-07-26T16:30:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18185.461581 secs 2024-07-26T16:30:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:45 smithi193 ceph-mon[37473]: pgmap v10335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:30:46.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:45 smithi134 ceph-mon[31966]: pgmap v10335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:30:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:45 smithi189 ceph-mon[36790]: pgmap v10335: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:30:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18204.882975 secs 2024-07-26T16:30:47.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18204.882975 secs 2024-07-26T16:30:47.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:30:47.346+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:30:47.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:30:47.346+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:30:47.350 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:30:47.346+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:30:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18204.882975 secs 2024-07-26T16:30:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:47 smithi193 ceph-mon[37473]: pgmap v10336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:47 smithi134 ceph-mon[31966]: pgmap v10336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:47 smithi189 ceph-mon[36790]: pgmap v10336: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18190.461718 secs 2024-07-26T16:30:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:49 smithi193 ceph-mon[37473]: pgmap v10337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18190.461718 secs 2024-07-26T16:30:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:49 smithi134 ceph-mon[31966]: pgmap v10337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18190.461718 secs 2024-07-26T16:30:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:49 smithi189 ceph-mon[36790]: pgmap v10337: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:51 smithi193 ceph-mon[37473]: pgmap v10338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18209.883097 secs 2024-07-26T16:30:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:51 smithi134 ceph-mon[31966]: pgmap v10338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18209.883097 secs 2024-07-26T16:30:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:51 smithi189 ceph-mon[36790]: pgmap v10338: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18209.883097 secs 2024-07-26T16:30:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:54 smithi193 ceph-mon[37473]: pgmap v10339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:53 smithi134 ceph-mon[31966]: pgmap v10339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:54 smithi189 ceph-mon[36790]: pgmap v10339: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:30:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18195.461858 secs 2024-07-26T16:30:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18195.461858 secs 2024-07-26T16:30:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18195.461858 secs 2024-07-26T16:30:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:56 smithi193 ceph-mon[37473]: pgmap v10340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:30:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:56 smithi134 ceph-mon[31966]: pgmap v10340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:30:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:56 smithi189 ceph-mon[36790]: pgmap v10340: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:30:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18214.883224 secs 2024-07-26T16:30:57.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18214.883224 secs 2024-07-26T16:30:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18214.883224 secs 2024-07-26T16:30:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:30:57.960+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:30:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:30:57.960+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:30:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:30:57.960+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:30:58.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:58 smithi134 ceph-mon[31966]: pgmap v10341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:58.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:58 smithi189 ceph-mon[36790]: pgmap v10341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:58 smithi193 ceph-mon[37473]: pgmap v10341: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:30:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18200.461985 secs 2024-07-26T16:30:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:30:59 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:30:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18200.461985 secs 2024-07-26T16:30:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:30:59 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:30:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18200.461985 secs 2024-07-26T16:30:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:30:59 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:31:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:00 smithi134 ceph-mon[31966]: pgmap v10342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:00.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:31:00.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:31:00.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:31:00.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:00 smithi189 ceph-mon[36790]: pgmap v10342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:00.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:31:00.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:31:00.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:31:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:00 smithi193 ceph-mon[37473]: pgmap v10342: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:31:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:31:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:31:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:01 smithi134 ceph-mon[31966]: pgmap v10343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:31:01.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:01 smithi134 ceph-mon[31966]: pgmap v10344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:31:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:01 smithi189 ceph-mon[36790]: pgmap v10343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:31:01.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:01 smithi189 ceph-mon[36790]: pgmap v10344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:31:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:01 smithi193 ceph-mon[37473]: pgmap v10343: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:31:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:01 smithi193 ceph-mon[37473]: pgmap v10344: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:31:02.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:31:02.362+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:31:02.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:31:02.362+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:31:02.365 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:31:02.362+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:31:02.366 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18219.883348 secs 2024-07-26T16:31:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18219.883348 secs 2024-07-26T16:31:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18219.883348 secs 2024-07-26T16:31:03.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:03 smithi189 ceph-mon[36790]: pgmap v10345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:31:03.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:03 smithi134 ceph-mon[31966]: pgmap v10345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:31:03.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:03 smithi193 ceph-mon[37473]: pgmap v10345: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:31:04.362 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18205.462052 secs 2024-07-26T16:31:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18205.462052 secs 2024-07-26T16:31:04.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18205.462052 secs 2024-07-26T16:31:05.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:05 smithi134 ceph-mon[31966]: pgmap v10346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:31:05.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:05 smithi189 ceph-mon[36790]: pgmap v10346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:31:05.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:05 smithi193 ceph-mon[37473]: pgmap v10346: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:31:07.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:07 smithi134 ceph-mon[31966]: pgmap v10347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:31:07.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18224.883439 secs 2024-07-26T16:31:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:07 smithi189 ceph-mon[36790]: pgmap v10347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:31:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18224.883439 secs 2024-07-26T16:31:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:07 smithi193 ceph-mon[37473]: pgmap v10347: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:31:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18224.883439 secs 2024-07-26T16:31:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:09 smithi134 ceph-mon[31966]: pgmap v10348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:31:09.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18210.462171 secs 2024-07-26T16:31:09.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:09 smithi189 ceph-mon[36790]: pgmap v10348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:31:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18210.462171 secs 2024-07-26T16:31:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:09 smithi193 ceph-mon[37473]: pgmap v10348: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:31:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18210.462171 secs 2024-07-26T16:31:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:11 smithi134 ceph-mon[31966]: pgmap v10349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:31:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:11 smithi189 ceph-mon[36790]: pgmap v10349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:31:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:11 smithi193 ceph-mon[37473]: pgmap v10349: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:31:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18229.883572 secs 2024-07-26T16:31:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18229.883572 secs 2024-07-26T16:31:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18229.883572 secs 2024-07-26T16:31:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:31:12.976+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:31:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:31:12.976+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:31:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:31:12.976+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:31:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:13 smithi134 ceph-mon[31966]: pgmap v10350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:13 smithi189 ceph-mon[36790]: pgmap v10350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:13 smithi193 ceph-mon[37473]: pgmap v10350: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18215.462272 secs 2024-07-26T16:31:14.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18215.462272 secs 2024-07-26T16:31:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18215.462272 secs 2024-07-26T16:31:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:15 smithi134 ceph-mon[31966]: pgmap v10351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:15 smithi189 ceph-mon[36790]: pgmap v10351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:15 smithi193 ceph-mon[37473]: pgmap v10351: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:17.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:31:17.376+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:31:17.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:31:17.376+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:31:17.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:31:17.376+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:31:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:17 smithi134 ceph-mon[31966]: pgmap v10352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18234.883653 secs 2024-07-26T16:31:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:17 smithi189 ceph-mon[36790]: pgmap v10352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18234.883653 secs 2024-07-26T16:31:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:17 smithi193 ceph-mon[37473]: pgmap v10352: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18234.883653 secs 2024-07-26T16:31:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:19 smithi134 ceph-mon[31966]: pgmap v10353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:19.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18220.462354 secs 2024-07-26T16:31:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:19 smithi189 ceph-mon[36790]: pgmap v10353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18220.462354 secs 2024-07-26T16:31:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:19 smithi193 ceph-mon[37473]: pgmap v10353: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18220.462354 secs 2024-07-26T16:31:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:21 smithi134 ceph-mon[31966]: pgmap v10354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:21.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:21 smithi189 ceph-mon[36790]: pgmap v10354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:21 smithi193 ceph-mon[37473]: pgmap v10354: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18239.883791 secs 2024-07-26T16:31:22.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18239.883791 secs 2024-07-26T16:31:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18239.883791 secs 2024-07-26T16:31:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:23 smithi134 ceph-mon[31966]: pgmap v10355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:23.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:23 smithi189 ceph-mon[36790]: pgmap v10355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:23 smithi193 ceph-mon[37473]: pgmap v10355: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18225.462496 secs 2024-07-26T16:31:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18225.462496 secs 2024-07-26T16:31:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18225.462496 secs 2024-07-26T16:31:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:25 smithi189 ceph-mon[36790]: pgmap v10356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:25 smithi193 ceph-mon[37473]: pgmap v10356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:25 smithi134 ceph-mon[31966]: pgmap v10356: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:27 smithi189 ceph-mon[36790]: pgmap v10357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18244.883899 secs 2024-07-26T16:31:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:27 smithi193 ceph-mon[37473]: pgmap v10357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18244.883899 secs 2024-07-26T16:31:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:27 smithi134 ceph-mon[31966]: pgmap v10357: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18244.883899 secs 2024-07-26T16:31:27.993 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:31:27.991+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:31:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:31:27.991+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:31:27.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:31:27.991+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:31:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:29 smithi189 ceph-mon[36790]: pgmap v10358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18230.462616 secs 2024-07-26T16:31:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:29 smithi193 ceph-mon[37473]: pgmap v10358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18230.462616 secs 2024-07-26T16:31:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:29 smithi134 ceph-mon[31966]: pgmap v10358: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18230.462616 secs 2024-07-26T16:31:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:31 smithi193 ceph-mon[37473]: pgmap v10359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:31 smithi134 ceph-mon[31966]: pgmap v10359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:31 smithi189 ceph-mon[36790]: pgmap v10359: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:32.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:31:32.382+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:31:32.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:31:32.382+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:31:32.384 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:31:32.382+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:31:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18249.884020 secs 2024-07-26T16:31:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18249.884020 secs 2024-07-26T16:31:32.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18249.884020 secs 2024-07-26T16:31:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:33 smithi193 ceph-mon[37473]: pgmap v10360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:33.595 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:33 smithi189 ceph-mon[36790]: pgmap v10360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:33 smithi134 ceph-mon[31966]: pgmap v10360: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:34.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18235.462755 secs 2024-07-26T16:31:34.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18235.462755 secs 2024-07-26T16:31:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18235.462755 secs 2024-07-26T16:31:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:35 smithi193 ceph-mon[37473]: pgmap v10361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:35 smithi134 ceph-mon[31966]: pgmap v10361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:35.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:35 smithi189 ceph-mon[36790]: pgmap v10361: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:37 smithi193 ceph-mon[37473]: pgmap v10362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18254.884142 secs 2024-07-26T16:31:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:37 smithi134 ceph-mon[31966]: pgmap v10362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18254.884142 secs 2024-07-26T16:31:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:37 smithi189 ceph-mon[36790]: pgmap v10362: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18254.884142 secs 2024-07-26T16:31:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:39 smithi193 ceph-mon[37473]: pgmap v10363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:39.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18240.462872 secs 2024-07-26T16:31:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:39 smithi134 ceph-mon[31966]: pgmap v10363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18240.462872 secs 2024-07-26T16:31:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:39 smithi189 ceph-mon[36790]: pgmap v10363: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18240.462872 secs 2024-07-26T16:31:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:41 smithi193 ceph-mon[37473]: pgmap v10364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:41 smithi134 ceph-mon[31966]: pgmap v10364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:41 smithi189 ceph-mon[36790]: pgmap v10364: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18259.884220 secs 2024-07-26T16:31:42.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18259.884220 secs 2024-07-26T16:31:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18259.884220 secs 2024-07-26T16:31:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:31:42.992+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:31:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:31:42.992+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:31:42.994 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:31:42.992+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:31:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:43 smithi193 ceph-mon[37473]: pgmap v10365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:43 smithi134 ceph-mon[31966]: pgmap v10365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:43 smithi189 ceph-mon[36790]: pgmap v10365: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18245.463015 secs 2024-07-26T16:31:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18245.463015 secs 2024-07-26T16:31:44.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18245.463015 secs 2024-07-26T16:31:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:45 smithi193 ceph-mon[37473]: pgmap v10366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:45 smithi134 ceph-mon[31966]: pgmap v10366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:45.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:45 smithi189 ceph-mon[36790]: pgmap v10366: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:47.399 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:31:47.397+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:31:47.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:31:47.397+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:31:47.400 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:31:47.397+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:31:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:47 smithi134 ceph-mon[31966]: pgmap v10367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18264.884317 secs 2024-07-26T16:31:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:47 smithi189 ceph-mon[36790]: pgmap v10367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18264.884317 secs 2024-07-26T16:31:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:47 smithi193 ceph-mon[37473]: pgmap v10367: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18264.884317 secs 2024-07-26T16:31:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:49 smithi134 ceph-mon[31966]: pgmap v10368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:49.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18250.463128 secs 2024-07-26T16:31:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:49 smithi189 ceph-mon[36790]: pgmap v10368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18250.463128 secs 2024-07-26T16:31:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:49 smithi193 ceph-mon[37473]: pgmap v10368: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:49.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18250.463128 secs 2024-07-26T16:31:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:51 smithi134 ceph-mon[31966]: pgmap v10369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:51 smithi189 ceph-mon[36790]: pgmap v10369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:51 smithi193 ceph-mon[37473]: pgmap v10369: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:31:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18269.884426 secs 2024-07-26T16:31:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18269.884426 secs 2024-07-26T16:31:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18269.884426 secs 2024-07-26T16:31:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:53 smithi134 ceph-mon[31966]: pgmap v10370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:31:53.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:53 smithi189 ceph-mon[36790]: pgmap v10370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:31:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:53 smithi193 ceph-mon[37473]: pgmap v10370: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:31:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18255.463248 secs 2024-07-26T16:31:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18255.463248 secs 2024-07-26T16:31:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18255.463248 secs 2024-07-26T16:31:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:55 smithi134 ceph-mon[31966]: pgmap v10371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:55 smithi189 ceph-mon[36790]: pgmap v10371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:55 smithi193 ceph-mon[37473]: pgmap v10371: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:57 smithi134 ceph-mon[31966]: pgmap v10372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18274.884565 secs 2024-07-26T16:31:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:57 smithi189 ceph-mon[36790]: pgmap v10372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18274.884565 secs 2024-07-26T16:31:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:57 smithi193 ceph-mon[37473]: pgmap v10372: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:31:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18274.884565 secs 2024-07-26T16:31:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:31:58.000+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:31:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:31:58.000+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:31:58.002 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:31:58.000+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:31:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:59 smithi134 ceph-mon[31966]: pgmap v10373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:59.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:31:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18260.463359 secs 2024-07-26T16:31:59.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:59 smithi189 ceph-mon[36790]: pgmap v10373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:31:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18260.463359 secs 2024-07-26T16:31:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:59 smithi193 ceph-mon[37473]: pgmap v10373: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:31:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:31:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18260.463359 secs 2024-07-26T16:32:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:00 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:32:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:00 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:32:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:00 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:32:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:01 smithi134 ceph-mon[31966]: pgmap v10374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:32:01.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:01 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:32:01.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:01 smithi189 ceph-mon[36790]: pgmap v10374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:32:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:01 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:32:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:01 smithi193 ceph-mon[37473]: pgmap v10374: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:32:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:01 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:32:02.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:32:02.400+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:32:02.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:32:02.400+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:32:02.402 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:32:02.400+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:32:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18279.884681 secs 2024-07-26T16:32:02.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:32:02.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:32:02.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:32:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18279.884681 secs 2024-07-26T16:32:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:32:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:32:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:32:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18279.884681 secs 2024-07-26T16:32:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:32:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:32:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:32:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:03 smithi134 ceph-mon[31966]: pgmap v10375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:32:03.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:03 smithi134 ceph-mon[31966]: pgmap v10376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:32:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:03 smithi189 ceph-mon[36790]: pgmap v10375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:32:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:03 smithi189 ceph-mon[36790]: pgmap v10376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:32:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:03 smithi193 ceph-mon[37473]: pgmap v10375: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-07-26T16:32:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:03 smithi193 ceph-mon[37473]: pgmap v10376: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:32:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18265.463494 secs 2024-07-26T16:32:04.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18265.463494 secs 2024-07-26T16:32:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18265.463494 secs 2024-07-26T16:32:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:05 smithi134 ceph-mon[31966]: pgmap v10377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:32:05.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:05 smithi189 ceph-mon[36790]: pgmap v10377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:32:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:05 smithi193 ceph-mon[37473]: pgmap v10377: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:32:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:07 smithi134 ceph-mon[31966]: pgmap v10378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:32:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18284.884820 secs 2024-07-26T16:32:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:07 smithi189 ceph-mon[36790]: pgmap v10378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:32:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18284.884820 secs 2024-07-26T16:32:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:07 smithi193 ceph-mon[37473]: pgmap v10378: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:32:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18284.884820 secs 2024-07-26T16:32:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:09 smithi134 ceph-mon[31966]: pgmap v10379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:32:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18270.463612 secs 2024-07-26T16:32:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:09 smithi189 ceph-mon[36790]: pgmap v10379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:32:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18270.463612 secs 2024-07-26T16:32:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:09 smithi193 ceph-mon[37473]: pgmap v10379: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:32:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18270.463612 secs 2024-07-26T16:32:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:11 smithi134 ceph-mon[31966]: pgmap v10380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:32:11.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:11 smithi189 ceph-mon[36790]: pgmap v10380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:32:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:11 smithi193 ceph-mon[37473]: pgmap v10380: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:32:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18289.884964 secs 2024-07-26T16:32:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18289.884964 secs 2024-07-26T16:32:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18289.884964 secs 2024-07-26T16:32:13.017 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:32:13.016+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:32:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:32:13.016+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:32:13.018 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:32:13.016+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:32:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:13 smithi134 ceph-mon[31966]: pgmap v10381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:32:13.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:13 smithi189 ceph-mon[36790]: pgmap v10381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:32:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:13 smithi193 ceph-mon[37473]: pgmap v10381: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:32:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18275.463712 secs 2024-07-26T16:32:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18275.463712 secs 2024-07-26T16:32:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18275.463712 secs 2024-07-26T16:32:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:15 smithi134 ceph-mon[31966]: pgmap v10382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:15 smithi189 ceph-mon[36790]: pgmap v10382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:15 smithi193 ceph-mon[37473]: pgmap v10382: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:17.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:32:17.415+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:32:17.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:32:17.415+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:32:17.418 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:32:17.415+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:32:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:17 smithi134 ceph-mon[31966]: pgmap v10383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18294.885057 secs 2024-07-26T16:32:17.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:17 smithi189 ceph-mon[36790]: pgmap v10383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18294.885057 secs 2024-07-26T16:32:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:17 smithi193 ceph-mon[37473]: pgmap v10383: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18294.885057 secs 2024-07-26T16:32:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:19 smithi189 ceph-mon[36790]: pgmap v10384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18280.463784 secs 2024-07-26T16:32:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:19 smithi193 ceph-mon[37473]: pgmap v10384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18280.463784 secs 2024-07-26T16:32:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:19 smithi134 ceph-mon[31966]: pgmap v10384: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:19.925 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18280.463784 secs 2024-07-26T16:32:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:21 smithi189 ceph-mon[36790]: pgmap v10385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:21 smithi193 ceph-mon[37473]: pgmap v10385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:21 smithi134 ceph-mon[31966]: pgmap v10385: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18299.885196 secs 2024-07-26T16:32:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18299.885196 secs 2024-07-26T16:32:22.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18299.885196 secs 2024-07-26T16:32:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:23 smithi193 ceph-mon[37473]: pgmap v10386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:23 smithi134 ceph-mon[31966]: pgmap v10386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:23.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:23 smithi189 ceph-mon[36790]: pgmap v10386: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18285.463897 secs 2024-07-26T16:32:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18285.463897 secs 2024-07-26T16:32:24.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18285.463897 secs 2024-07-26T16:32:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:25 smithi193 ceph-mon[37473]: pgmap v10387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:25 smithi134 ceph-mon[31966]: pgmap v10387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:25.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:25 smithi189 ceph-mon[36790]: pgmap v10387: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:27 smithi193 ceph-mon[37473]: pgmap v10388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:27.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18304.885334 secs 2024-07-26T16:32:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:27 smithi134 ceph-mon[31966]: pgmap v10388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:27.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18304.885334 secs 2024-07-26T16:32:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:27 smithi189 ceph-mon[36790]: pgmap v10388: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18304.885334 secs 2024-07-26T16:32:28.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:32:28.031+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:32:28.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:32:28.031+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:32:28.033 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:32:28.031+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:32:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:29 smithi134 ceph-mon[31966]: pgmap v10389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:29.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18290.464010 secs 2024-07-26T16:32:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:29 smithi193 ceph-mon[37473]: pgmap v10389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18290.464010 secs 2024-07-26T16:32:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:29 smithi189 ceph-mon[36790]: pgmap v10389: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18290.464010 secs 2024-07-26T16:32:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:31 smithi193 ceph-mon[37473]: pgmap v10390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:31 smithi134 ceph-mon[31966]: pgmap v10390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:31.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:31 smithi189 ceph-mon[36790]: pgmap v10390: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:32.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:32:32.431+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:32:32.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:32:32.431+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:32:32.433 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:32:32.431+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:32:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18309.885454 secs 2024-07-26T16:32:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18309.885454 secs 2024-07-26T16:32:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18309.885454 secs 2024-07-26T16:32:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:33 smithi193 ceph-mon[37473]: pgmap v10391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:33.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:33 smithi189 ceph-mon[36790]: pgmap v10391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:33 smithi134 ceph-mon[31966]: pgmap v10391: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18295.464082 secs 2024-07-26T16:32:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18295.464082 secs 2024-07-26T16:32:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18295.464082 secs 2024-07-26T16:32:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:35 smithi193 ceph-mon[37473]: pgmap v10392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:35 smithi134 ceph-mon[31966]: pgmap v10392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:35.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:35 smithi189 ceph-mon[36790]: pgmap v10392: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:37 smithi193 ceph-mon[37473]: pgmap v10393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18314.885581 secs 2024-07-26T16:32:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:37 smithi134 ceph-mon[31966]: pgmap v10393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:37.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18314.885581 secs 2024-07-26T16:32:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:37 smithi189 ceph-mon[36790]: pgmap v10393: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18314.885581 secs 2024-07-26T16:32:39.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:39 smithi134 ceph-mon[31966]: pgmap v10394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:39.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18300.464219 secs 2024-07-26T16:32:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:39 smithi189 ceph-mon[36790]: pgmap v10394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18300.464219 secs 2024-07-26T16:32:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:39 smithi193 ceph-mon[37473]: pgmap v10394: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18300.464219 secs 2024-07-26T16:32:40.510 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:32:40.508+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T16:32:40.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:32:40.508+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T16:32:40.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:32:40.509+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 142 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T16:32:40.511 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:32:40.509+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T16:32:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:41 smithi134 ceph-mon[31966]: pgmap v10395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:41 smithi189 ceph-mon[36790]: pgmap v10395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:41 smithi193 ceph-mon[37473]: pgmap v10395: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18319.885700 secs 2024-07-26T16:32:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18319.885700 secs 2024-07-26T16:32:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18319.885700 secs 2024-07-26T16:32:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:32:43.038+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:32:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:32:43.038+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:32:43.040 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:32:43.038+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:32:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:43 smithi134 ceph-mon[31966]: pgmap v10396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:43 smithi189 ceph-mon[36790]: pgmap v10396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:43 smithi193 ceph-mon[37473]: pgmap v10396: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18305.464302 secs 2024-07-26T16:32:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18305.464302 secs 2024-07-26T16:32:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18305.464302 secs 2024-07-26T16:32:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:45 smithi134 ceph-mon[31966]: pgmap v10397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:45 smithi189 ceph-mon[36790]: pgmap v10397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:45 smithi193 ceph-mon[37473]: pgmap v10397: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:47.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:32:47.434+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:32:47.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:32:47.434+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:32:47.436 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:32:47.434+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:32:47.729 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:47 smithi134 ceph-mon[31966]: pgmap v10398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:47.729 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18324.885819 secs 2024-07-26T16:32:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:47 smithi189 ceph-mon[36790]: pgmap v10398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18324.885819 secs 2024-07-26T16:32:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:47 smithi193 ceph-mon[37473]: pgmap v10398: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18324.885819 secs 2024-07-26T16:32:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:49 smithi134 ceph-mon[31966]: pgmap v10399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18310.464384 secs 2024-07-26T16:32:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:49 smithi189 ceph-mon[36790]: pgmap v10399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18310.464384 secs 2024-07-26T16:32:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:49 smithi193 ceph-mon[37473]: pgmap v10399: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18310.464384 secs 2024-07-26T16:32:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:32:51.062+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:32:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:32:51.062+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:32:51.064 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:32:51.063+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:32:51.065 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:32:51.063+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:32:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:51 smithi134 ceph-mon[31966]: pgmap v10400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:51 smithi189 ceph-mon[36790]: pgmap v10400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:51 smithi193 ceph-mon[37473]: pgmap v10400: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:32:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18329.885941 secs 2024-07-26T16:32:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18329.885941 secs 2024-07-26T16:32:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18329.885941 secs 2024-07-26T16:32:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:53 smithi134 ceph-mon[31966]: pgmap v10401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:53.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:53 smithi189 ceph-mon[36790]: pgmap v10401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:53 smithi193 ceph-mon[37473]: pgmap v10401: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18315.464461 secs 2024-07-26T16:32:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18315.464461 secs 2024-07-26T16:32:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18315.464461 secs 2024-07-26T16:32:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:55 smithi134 ceph-mon[31966]: pgmap v10402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:55.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:55 smithi189 ceph-mon[36790]: pgmap v10402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:55 smithi193 ceph-mon[37473]: pgmap v10402: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:57.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:57 smithi134 ceph-mon[31966]: pgmap v10403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:57.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18334.886064 secs 2024-07-26T16:32:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:57 smithi189 ceph-mon[36790]: pgmap v10403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18334.886064 secs 2024-07-26T16:32:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:57 smithi193 ceph-mon[37473]: pgmap v10403: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:32:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18334.886064 secs 2024-07-26T16:32:58.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:32:58.040+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:32:58.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:32:58.040+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:32:58.041 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:32:58.040+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:32:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:59 smithi134 ceph-mon[31966]: pgmap v10404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:59.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:32:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18320.464600 secs 2024-07-26T16:32:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:59 smithi189 ceph-mon[36790]: pgmap v10404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:32:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:32:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18320.464600 secs 2024-07-26T16:33:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:59 smithi193 ceph-mon[37473]: pgmap v10404: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:32:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18320.464600 secs 2024-07-26T16:33:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:01 smithi134 ceph-mon[31966]: pgmap v10405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:01.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18339.886190 secs 2024-07-26T16:33:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:01 smithi189 ceph-mon[36790]: pgmap v10405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:01.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18339.886190 secs 2024-07-26T16:33:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:01 smithi193 ceph-mon[37473]: pgmap v10405: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18339.886190 secs 2024-07-26T16:33:02.451 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:33:02.449+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:33:02.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:33:02.449+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:33:02.452 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:33:02.449+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:33:02.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:33:02.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:02.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:02.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:02.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:02 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:02.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:33:02.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:02.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:02.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:02.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:02 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:33:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:03.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:02 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:03 smithi134 ceph-mon[31966]: pgmap v10406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:03 smithi189 ceph-mon[36790]: pgmap v10406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:03 smithi193 ceph-mon[37473]: pgmap v10406: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18325.464667 secs 2024-07-26T16:33:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:04 smithi134 ceph-mon[31966]: pgmap v10407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:04 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18325.464667 secs 2024-07-26T16:33:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:04 smithi189 ceph-mon[36790]: pgmap v10407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:04 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18325.464667 secs 2024-07-26T16:33:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:04 smithi193 ceph-mon[37473]: pgmap v10407: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:05.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:05.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:04 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:06 smithi134 ceph-mon[31966]: pgmap v10408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:07.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:07.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:06 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:07.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18344.886329 secs 2024-07-26T16:33:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:06 smithi189 ceph-mon[36790]: pgmap v10408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:06 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18344.886329 secs 2024-07-26T16:33:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:06 smithi193 ceph-mon[37473]: pgmap v10408: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:07.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:07.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:06 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:07.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18344.886329 secs 2024-07-26T16:33:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:33:08.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:33:08.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:07 smithi134 ceph-mon[31966]: pgmap v10409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:33:08.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:07 smithi134 ceph-mon[31966]: pgmap v10410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:33:08.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:08.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:33:08.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:33:08.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:07 smithi189 ceph-mon[36790]: pgmap v10409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:33:08.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:07 smithi189 ceph-mon[36790]: pgmap v10410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:33:08.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:33:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:33:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:07 smithi193 ceph-mon[37473]: pgmap v10409: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:33:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:07 smithi193 ceph-mon[37473]: pgmap v10410: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:33:08.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:33:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18330.464805 secs 2024-07-26T16:33:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:09 smithi193 ceph-mon[37473]: pgmap v10411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18330.464805 secs 2024-07-26T16:33:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:09 smithi134 ceph-mon[31966]: pgmap v10411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18330.464805 secs 2024-07-26T16:33:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:09 smithi189 ceph-mon[36790]: pgmap v10411: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:11 smithi193 ceph-mon[37473]: pgmap v10412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18349.886461 secs 2024-07-26T16:33:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:11 smithi134 ceph-mon[31966]: pgmap v10412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18349.886461 secs 2024-07-26T16:33:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:11 smithi189 ceph-mon[36790]: pgmap v10412: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18349.886461 secs 2024-07-26T16:33:13.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:33:13.054+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:33:13.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:33:13.054+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:33:13.056 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:33:13.054+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:33:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:13 smithi193 ceph-mon[37473]: pgmap v10413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:13 smithi134 ceph-mon[31966]: pgmap v10413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:13 smithi189 ceph-mon[36790]: pgmap v10413: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:33:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18335.464966 secs 2024-07-26T16:33:15.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18335.464966 secs 2024-07-26T16:33:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18335.464966 secs 2024-07-26T16:33:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:15 smithi134 ceph-mon[31966]: pgmap v10414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:33:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:15 smithi193 ceph-mon[37473]: pgmap v10414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:33:16.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:15 smithi189 ceph-mon[36790]: pgmap v10414: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:33:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18354.886544 secs 2024-07-26T16:33:17.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18354.886544 secs 2024-07-26T16:33:17.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18354.886544 secs 2024-07-26T16:33:17.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:33:17.452+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:33:17.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:33:17.452+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:33:17.455 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:33:17.452+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:33:18.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:17 smithi134 ceph-mon[31966]: pgmap v10415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:33:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:17 smithi193 ceph-mon[37473]: pgmap v10415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:33:18.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:17 smithi189 ceph-mon[36790]: pgmap v10415: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:33:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18340.465079 secs 2024-07-26T16:33:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:20 smithi193 ceph-mon[37473]: pgmap v10416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18340.465079 secs 2024-07-26T16:33:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:20 smithi134 ceph-mon[31966]: pgmap v10416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:20.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18340.465079 secs 2024-07-26T16:33:20.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:20 smithi189 ceph-mon[36790]: pgmap v10416: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:22 smithi193 ceph-mon[37473]: pgmap v10417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:22.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18359.886701 secs 2024-07-26T16:33:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:22 smithi134 ceph-mon[31966]: pgmap v10417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:22.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18359.886701 secs 2024-07-26T16:33:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:22 smithi189 ceph-mon[36790]: pgmap v10417: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18359.886701 secs 2024-07-26T16:33:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:24 smithi193 ceph-mon[37473]: pgmap v10418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:24 smithi134 ceph-mon[31966]: pgmap v10418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:24.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:24 smithi189 ceph-mon[36790]: pgmap v10418: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18345.465189 secs 2024-07-26T16:33:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18345.465189 secs 2024-07-26T16:33:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18345.465189 secs 2024-07-26T16:33:25.519 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:33:25.516+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T16:33:25.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:33:25.516+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T16:33:25.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:33:25.517+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 143 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T16:33:25.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:33:25.517+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T16:33:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:26 smithi134 ceph-mon[31966]: pgmap v10419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:26.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:26 smithi189 ceph-mon[36790]: pgmap v10419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:26.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:26 smithi193 ceph-mon[37473]: pgmap v10419: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18364.886830 secs 2024-07-26T16:33:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18364.886830 secs 2024-07-26T16:33:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18364.886830 secs 2024-07-26T16:33:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:33:28.069+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:33:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:33:28.069+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:33:28.071 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:33:28.069+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:33:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:28 smithi134 ceph-mon[31966]: pgmap v10420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:28.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:28 smithi189 ceph-mon[36790]: pgmap v10420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:28.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:28 smithi193 ceph-mon[37473]: pgmap v10420: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18350.465304 secs 2024-07-26T16:33:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:30 smithi134 ceph-mon[31966]: pgmap v10421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:30.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18350.465304 secs 2024-07-26T16:33:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:30 smithi189 ceph-mon[36790]: pgmap v10421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18350.465304 secs 2024-07-26T16:33:30.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:30 smithi193 ceph-mon[37473]: pgmap v10421: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:32 smithi134 ceph-mon[31966]: pgmap v10422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:32.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18369.886960 secs 2024-07-26T16:33:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:32 smithi189 ceph-mon[36790]: pgmap v10422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18369.886960 secs 2024-07-26T16:33:32.459 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:33:32.456+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:33:32.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:33:32.456+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:33:32.460 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:33:32.456+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:33:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:32 smithi193 ceph-mon[37473]: pgmap v10422: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:32.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18369.886960 secs 2024-07-26T16:33:34.327 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:34 smithi193 ceph-mon[37473]: pgmap v10423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:34.327 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18355.465414 secs 2024-07-26T16:33:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:34 smithi134 ceph-mon[31966]: pgmap v10423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:34.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18355.465414 secs 2024-07-26T16:33:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:34 smithi189 ceph-mon[36790]: pgmap v10423: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18355.465414 secs 2024-07-26T16:33:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:33:36.070+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:33:36.072 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:33:36.070+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:33:36.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:33:36.071+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:33:36.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:33:36.071+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:33:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:36 smithi134 ceph-mon[31966]: pgmap v10424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:36.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:36 smithi189 ceph-mon[36790]: pgmap v10424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:36.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:36 smithi193 ceph-mon[37473]: pgmap v10424: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18374.887070 secs 2024-07-26T16:33:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18374.887070 secs 2024-07-26T16:33:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18374.887070 secs 2024-07-26T16:33:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:38 smithi134 ceph-mon[31966]: pgmap v10425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:38.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:38 smithi189 ceph-mon[36790]: pgmap v10425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:38.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:38 smithi193 ceph-mon[37473]: pgmap v10425: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18360.465532 secs 2024-07-26T16:33:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18360.465532 secs 2024-07-26T16:33:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18360.465532 secs 2024-07-26T16:33:40.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:40 smithi134 ceph-mon[31966]: pgmap v10426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:40.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:40 smithi189 ceph-mon[36790]: pgmap v10426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:40.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:40 smithi193 ceph-mon[37473]: pgmap v10426: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:42 smithi134 ceph-mon[31966]: pgmap v10427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:42.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18379.887177 secs 2024-07-26T16:33:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:42 smithi189 ceph-mon[36790]: pgmap v10427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18379.887177 secs 2024-07-26T16:33:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:42 smithi193 ceph-mon[37473]: pgmap v10427: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18379.887177 secs 2024-07-26T16:33:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:33:43.071+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:33:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:33:43.071+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:33:43.073 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:33:43.071+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:33:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:44 smithi134 ceph-mon[31966]: pgmap v10428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18365.465653 secs 2024-07-26T16:33:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:44 smithi189 ceph-mon[36790]: pgmap v10428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18365.465653 secs 2024-07-26T16:33:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:44 smithi193 ceph-mon[37473]: pgmap v10428: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18365.465653 secs 2024-07-26T16:33:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:46 smithi134 ceph-mon[31966]: pgmap v10429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:46 smithi189 ceph-mon[36790]: pgmap v10429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:46 smithi193 ceph-mon[37473]: pgmap v10429: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18384.887298 secs 2024-07-26T16:33:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18384.887298 secs 2024-07-26T16:33:47.474 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:33:47.472+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:33:47.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:33:47.472+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:33:47.475 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:33:47.472+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:33:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18384.887298 secs 2024-07-26T16:33:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:48 smithi134 ceph-mon[31966]: pgmap v10430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:48.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:48 smithi189 ceph-mon[36790]: pgmap v10430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:48 smithi193 ceph-mon[37473]: pgmap v10430: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:49.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18370.465767 secs 2024-07-26T16:33:49.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18370.465767 secs 2024-07-26T16:33:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18370.465767 secs 2024-07-26T16:33:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:50 smithi134 ceph-mon[31966]: pgmap v10431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:50 smithi189 ceph-mon[36790]: pgmap v10431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:50 smithi193 ceph-mon[37473]: pgmap v10431: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:52 smithi134 ceph-mon[31966]: pgmap v10432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18389.887418 secs 2024-07-26T16:33:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:52 smithi189 ceph-mon[36790]: pgmap v10432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18389.887418 secs 2024-07-26T16:33:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:52 smithi193 ceph-mon[37473]: pgmap v10432: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18389.887418 secs 2024-07-26T16:33:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:54 smithi134 ceph-mon[31966]: pgmap v10433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:54.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18375.465888 secs 2024-07-26T16:33:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:54 smithi189 ceph-mon[36790]: pgmap v10433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18375.465888 secs 2024-07-26T16:33:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:54 smithi193 ceph-mon[37473]: pgmap v10433: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:33:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18375.465888 secs 2024-07-26T16:33:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:56 smithi134 ceph-mon[31966]: pgmap v10434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:56.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:56 smithi189 ceph-mon[36790]: pgmap v10434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:56 smithi193 ceph-mon[37473]: pgmap v10434: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:33:57.425 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18394.887515 secs 2024-07-26T16:33:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18394.887515 secs 2024-07-26T16:33:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18394.887515 secs 2024-07-26T16:33:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:33:58.075+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:33:58.077 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:33:58.075+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:33:58.078 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:33:58.075+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:33:58.446 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:58 smithi189 ceph-mon[36790]: pgmap v10435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:58 smithi193 ceph-mon[37473]: pgmap v10435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:58.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:58 smithi134 ceph-mon[31966]: pgmap v10435: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:33:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:33:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18380.466011 secs 2024-07-26T16:33:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:33:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18380.466011 secs 2024-07-26T16:33:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:33:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18380.466011 secs 2024-07-26T16:34:00.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:00 smithi189 ceph-mon[36790]: pgmap v10436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:00.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:00 smithi193 ceph-mon[37473]: pgmap v10436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:00.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:00 smithi134 ceph-mon[31966]: pgmap v10436: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:02 smithi189 ceph-mon[36790]: pgmap v10437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18399.887645 secs 2024-07-26T16:34:02.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:34:02.487+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:34:02.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:34:02.487+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:34:02.490 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:34:02.487+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:34:02.490 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:02 smithi134 ceph-mon[31966]: pgmap v10437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:02.490 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18399.887645 secs 2024-07-26T16:34:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:02 smithi193 ceph-mon[37473]: pgmap v10437: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18399.887645 secs 2024-07-26T16:34:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:04 smithi193 ceph-mon[37473]: pgmap v10438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:04.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18385.466106 secs 2024-07-26T16:34:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:04 smithi134 ceph-mon[31966]: pgmap v10438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:04.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18385.466106 secs 2024-07-26T16:34:04.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:04 smithi189 ceph-mon[36790]: pgmap v10438: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18385.466106 secs 2024-07-26T16:34:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:06 smithi193 ceph-mon[37473]: pgmap v10439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:06.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:06 smithi134 ceph-mon[31966]: pgmap v10439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:06.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:06 smithi189 ceph-mon[36790]: pgmap v10439: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:34:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18404.887765 secs 2024-07-26T16:34:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:07 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:34:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18404.887765 secs 2024-07-26T16:34:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:07 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:34:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18404.887765 secs 2024-07-26T16:34:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:07 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:34:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:08 smithi193 ceph-mon[37473]: pgmap v10440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:34:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:34:08.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:34:08.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:08 smithi134 ceph-mon[31966]: pgmap v10440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:08.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:34:08.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:34:08.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:34:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:08 smithi189 ceph-mon[36790]: pgmap v10440: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:34:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:34:08.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:34:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:09 smithi193 ceph-mon[37473]: pgmap v10441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T16:34:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:09 smithi193 ceph-mon[37473]: pgmap v10442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18390.466225 secs 2024-07-26T16:34:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:09 smithi134 ceph-mon[31966]: pgmap v10441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T16:34:09.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:09 smithi134 ceph-mon[31966]: pgmap v10442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:09.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18390.466225 secs 2024-07-26T16:34:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:09 smithi189 ceph-mon[36790]: pgmap v10441: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T16:34:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:09 smithi189 ceph-mon[36790]: pgmap v10442: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18390.466225 secs 2024-07-26T16:34:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:11 smithi193 ceph-mon[37473]: pgmap v10443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:11 smithi134 ceph-mon[31966]: pgmap v10443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:11.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:11 smithi189 ceph-mon[36790]: pgmap v10443: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18409.887905 secs 2024-07-26T16:34:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18409.887905 secs 2024-07-26T16:34:12.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18409.887905 secs 2024-07-26T16:34:13.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:34:13.091+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:34:13.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:34:13.091+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:34:13.093 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:34:13.091+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:34:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:13 smithi193 ceph-mon[37473]: pgmap v10444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:34:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:13 smithi134 ceph-mon[31966]: pgmap v10444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:34:13.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:13 smithi189 ceph-mon[36790]: pgmap v10444: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:34:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18395.466323 secs 2024-07-26T16:34:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18395.466323 secs 2024-07-26T16:34:14.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18395.466323 secs 2024-07-26T16:34:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:15 smithi193 ceph-mon[37473]: pgmap v10445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:15 smithi134 ceph-mon[31966]: pgmap v10445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:15 smithi189 ceph-mon[36790]: pgmap v10445: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:17.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:34:17.503+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:34:17.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:34:17.503+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:34:17.506 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:34:17.503+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:34:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:17 smithi193 ceph-mon[37473]: pgmap v10446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18414.888022 secs 2024-07-26T16:34:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:17 smithi134 ceph-mon[31966]: pgmap v10446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18414.888022 secs 2024-07-26T16:34:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:17 smithi189 ceph-mon[36790]: pgmap v10446: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18414.888022 secs 2024-07-26T16:34:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:19 smithi193 ceph-mon[37473]: pgmap v10447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:34:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18400.466445 secs 2024-07-26T16:34:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:19 smithi134 ceph-mon[31966]: pgmap v10447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:34:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18400.466445 secs 2024-07-26T16:34:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:19 smithi189 ceph-mon[36790]: pgmap v10447: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:34:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18400.466445 secs 2024-07-26T16:34:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:21 smithi193 ceph-mon[37473]: pgmap v10448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:21 smithi134 ceph-mon[31966]: pgmap v10448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:21 smithi189 ceph-mon[36790]: pgmap v10448: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18419.888101 secs 2024-07-26T16:34:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18419.888101 secs 2024-07-26T16:34:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18419.888101 secs 2024-07-26T16:34:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:23 smithi134 ceph-mon[31966]: pgmap v10449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:34:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:23 smithi189 ceph-mon[36790]: pgmap v10449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:34:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:23 smithi193 ceph-mon[37473]: pgmap v10449: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:34:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18405.466585 secs 2024-07-26T16:34:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18405.466585 secs 2024-07-26T16:34:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18405.466585 secs 2024-07-26T16:34:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:25 smithi134 ceph-mon[31966]: pgmap v10450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:25.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:25 smithi189 ceph-mon[36790]: pgmap v10450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:25 smithi193 ceph-mon[37473]: pgmap v10450: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:27 smithi134 ceph-mon[31966]: pgmap v10451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:27.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18424.888212 secs 2024-07-26T16:34:27.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:27 smithi189 ceph-mon[36790]: pgmap v10451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18424.888212 secs 2024-07-26T16:34:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:27 smithi193 ceph-mon[37473]: pgmap v10451: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18424.888212 secs 2024-07-26T16:34:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:34:28.096+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:34:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:34:28.096+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:34:28.098 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:34:28.096+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:34:29.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:29 smithi134 ceph-mon[31966]: pgmap v10452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:29.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18410.466702 secs 2024-07-26T16:34:29.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:29 smithi189 ceph-mon[36790]: pgmap v10452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18410.466702 secs 2024-07-26T16:34:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:29 smithi193 ceph-mon[37473]: pgmap v10452: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18410.466702 secs 2024-07-26T16:34:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:31 smithi134 ceph-mon[31966]: pgmap v10453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:31.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:31 smithi189 ceph-mon[36790]: pgmap v10453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:31 smithi193 ceph-mon[37473]: pgmap v10453: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:32.520 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:34:32.518+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:34:32.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:34:32.518+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:34:32.521 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:34:32.518+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:34:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18429.888339 secs 2024-07-26T16:34:32.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18429.888339 secs 2024-07-26T16:34:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18429.888339 secs 2024-07-26T16:34:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:33 smithi134 ceph-mon[31966]: pgmap v10454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:34:33.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:33 smithi189 ceph-mon[36790]: pgmap v10454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:34:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:33 smithi193 ceph-mon[37473]: pgmap v10454: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:34:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18415.466836 secs 2024-07-26T16:34:34.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18415.466836 secs 2024-07-26T16:34:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18415.466836 secs 2024-07-26T16:34:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:35 smithi134 ceph-mon[31966]: pgmap v10455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:35.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:35 smithi189 ceph-mon[36790]: pgmap v10455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:35 smithi193 ceph-mon[37473]: pgmap v10455: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:34:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:37 smithi134 ceph-mon[31966]: pgmap v10456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:37.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18434.888463 secs 2024-07-26T16:34:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:37 smithi189 ceph-mon[36790]: pgmap v10456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18434.888463 secs 2024-07-26T16:34:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:37 smithi193 ceph-mon[37473]: pgmap v10456: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18434.888463 secs 2024-07-26T16:34:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:39 smithi134 ceph-mon[31966]: pgmap v10457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:39.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18420.466954 secs 2024-07-26T16:34:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:39 smithi189 ceph-mon[36790]: pgmap v10457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18420.466954 secs 2024-07-26T16:34:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:39 smithi193 ceph-mon[37473]: pgmap v10457: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18420.466954 secs 2024-07-26T16:34:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:41 smithi134 ceph-mon[31966]: pgmap v10458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:41.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:41 smithi189 ceph-mon[36790]: pgmap v10458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:41 smithi193 ceph-mon[37473]: pgmap v10458: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18439.888578 secs 2024-07-26T16:34:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18439.888578 secs 2024-07-26T16:34:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18439.888578 secs 2024-07-26T16:34:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:34:43.101+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:34:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:34:43.101+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:34:43.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:34:43.101+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:34:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:43 smithi134 ceph-mon[31966]: pgmap v10459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:34:43.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:43 smithi189 ceph-mon[36790]: pgmap v10459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:34:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:43 smithi193 ceph-mon[37473]: pgmap v10459: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:34:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18425.467018 secs 2024-07-26T16:34:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18425.467018 secs 2024-07-26T16:34:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18425.467018 secs 2024-07-26T16:34:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:45 smithi134 ceph-mon[31966]: pgmap v10460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:45.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:45 smithi189 ceph-mon[36790]: pgmap v10460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:45 smithi193 ceph-mon[37473]: pgmap v10460: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:47.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:34:47.529+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:34:47.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:34:47.529+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:34:47.531 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:34:47.529+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:34:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:47 smithi134 ceph-mon[31966]: pgmap v10461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:34:47.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18444.888671 secs 2024-07-26T16:34:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:47 smithi189 ceph-mon[36790]: pgmap v10461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:34:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18444.888671 secs 2024-07-26T16:34:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:47 smithi193 ceph-mon[37473]: pgmap v10461: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:34:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18444.888671 secs 2024-07-26T16:34:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:49 smithi134 ceph-mon[31966]: pgmap v10462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:34:49.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18430.467100 secs 2024-07-26T16:34:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:49 smithi189 ceph-mon[36790]: pgmap v10462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:34:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18430.467100 secs 2024-07-26T16:34:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:49 smithi193 ceph-mon[37473]: pgmap v10462: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:34:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18430.467100 secs 2024-07-26T16:34:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:51 smithi134 ceph-mon[31966]: pgmap v10463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:34:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:51 smithi189 ceph-mon[36790]: pgmap v10463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:34:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:51 smithi193 ceph-mon[37473]: pgmap v10463: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-07-26T16:34:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18449.888792 secs 2024-07-26T16:34:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18449.888792 secs 2024-07-26T16:34:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18449.888792 secs 2024-07-26T16:34:53.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:53 smithi189 ceph-mon[36790]: pgmap v10464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:34:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:53 smithi193 ceph-mon[37473]: pgmap v10464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:34:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:53 smithi134 ceph-mon[31966]: pgmap v10464: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:34:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18435.467226 secs 2024-07-26T16:34:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18435.467226 secs 2024-07-26T16:34:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18435.467226 secs 2024-07-26T16:34:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:55 smithi193 ceph-mon[37473]: pgmap v10465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:55 smithi134 ceph-mon[31966]: pgmap v10465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:55.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:55 smithi189 ceph-mon[36790]: pgmap v10465: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:34:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:57 smithi193 ceph-mon[37473]: pgmap v10466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18454.888904 secs 2024-07-26T16:34:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:57 smithi134 ceph-mon[31966]: pgmap v10466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18454.888904 secs 2024-07-26T16:34:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:57 smithi189 ceph-mon[36790]: pgmap v10466: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:34:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18454.888904 secs 2024-07-26T16:34:58.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:34:58.117+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:34:58.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:34:58.117+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:34:58.119 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:34:58.117+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:34:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:59 smithi193 ceph-mon[37473]: pgmap v10467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:34:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18440.467352 secs 2024-07-26T16:34:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:59 smithi134 ceph-mon[31966]: pgmap v10467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:34:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18440.467352 secs 2024-07-26T16:34:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:59 smithi189 ceph-mon[36790]: pgmap v10467: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:34:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:34:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18440.467352 secs 2024-07-26T16:35:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:01 smithi193 ceph-mon[37473]: pgmap v10468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:01 smithi134 ceph-mon[31966]: pgmap v10468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:01 smithi189 ceph-mon[36790]: pgmap v10468: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:02.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:35:02.540+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:35:02.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:35:02.540+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:35:02.542 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:35:02.540+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:35:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18459.889024 secs 2024-07-26T16:35:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18459.889024 secs 2024-07-26T16:35:02.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18459.889024 secs 2024-07-26T16:35:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:03 smithi193 ceph-mon[37473]: pgmap v10469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:03 smithi134 ceph-mon[31966]: pgmap v10469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:03.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:03 smithi189 ceph-mon[36790]: pgmap v10469: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18445.467435 secs 2024-07-26T16:35:04.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18445.467435 secs 2024-07-26T16:35:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18445.467435 secs 2024-07-26T16:35:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:05 smithi193 ceph-mon[37473]: pgmap v10470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:05 smithi134 ceph-mon[31966]: pgmap v10470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:05.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:05 smithi189 ceph-mon[36790]: pgmap v10470: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:07 smithi193 ceph-mon[37473]: pgmap v10471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:07.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18464.889166 secs 2024-07-26T16:35:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:07 smithi134 ceph-mon[31966]: pgmap v10471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18464.889166 secs 2024-07-26T16:35:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:07 smithi189 ceph-mon[36790]: pgmap v10471: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18464.889166 secs 2024-07-26T16:35:08.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:08 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:35:08.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:08 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:35:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:08 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:35:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:09 smithi193 ceph-mon[37473]: pgmap v10472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:09.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18450.467527 secs 2024-07-26T16:35:09.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:35:09.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:35:09.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:09 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:35:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:09 smithi134 ceph-mon[31966]: pgmap v10472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18450.467527 secs 2024-07-26T16:35:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:35:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:35:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:09 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:35:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:09 smithi189 ceph-mon[36790]: pgmap v10472: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18450.467527 secs 2024-07-26T16:35:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:35:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:35:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:09 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:35:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:10 smithi193 ceph-mon[37473]: pgmap v10473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:35:10.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:10 smithi193 ceph-mon[37473]: pgmap v10474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:35:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:10 smithi134 ceph-mon[31966]: pgmap v10473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:35:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:10 smithi134 ceph-mon[31966]: pgmap v10474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:35:10.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:10 smithi189 ceph-mon[36790]: pgmap v10473: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:35:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:10 smithi189 ceph-mon[36790]: pgmap v10474: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:35:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18469.889241 secs 2024-07-26T16:35:12.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:12 smithi134 ceph-mon[31966]: pgmap v10475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:35:12.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18469.889241 secs 2024-07-26T16:35:12.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:12 smithi189 ceph-mon[36790]: pgmap v10475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:35:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18469.889241 secs 2024-07-26T16:35:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:12 smithi193 ceph-mon[37473]: pgmap v10475: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:35:13.130 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:35:13.128+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:35:13.130 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:35:13.128+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:35:13.130 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:35:13.128+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:35:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:14 smithi134 ceph-mon[31966]: pgmap v10476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:35:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18455.467645 secs 2024-07-26T16:35:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:14 smithi189 ceph-mon[36790]: pgmap v10476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:35:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18455.467645 secs 2024-07-26T16:35:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:14 smithi193 ceph-mon[37473]: pgmap v10476: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:35:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18455.467645 secs 2024-07-26T16:35:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:16 smithi134 ceph-mon[31966]: pgmap v10477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:35:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:16 smithi189 ceph-mon[36790]: pgmap v10477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:35:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:16 smithi193 ceph-mon[37473]: pgmap v10477: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:35:17.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:35:17.555+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:35:17.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:35:17.555+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:35:17.558 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:35:17.555+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:35:17.823 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18474.889368 secs 2024-07-26T16:35:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18474.889368 secs 2024-07-26T16:35:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18474.889368 secs 2024-07-26T16:35:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:18 smithi134 ceph-mon[31966]: pgmap v10478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:35:18.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:18 smithi189 ceph-mon[36790]: pgmap v10478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:35:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:18 smithi193 ceph-mon[37473]: pgmap v10478: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:35:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18460.467765 secs 2024-07-26T16:35:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18460.467765 secs 2024-07-26T16:35:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18460.467765 secs 2024-07-26T16:35:20.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:20 smithi134 ceph-mon[31966]: pgmap v10479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:35:20.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:20 smithi189 ceph-mon[36790]: pgmap v10479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:35:21.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:20 smithi193 ceph-mon[37473]: pgmap v10479: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:35:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18479.889492 secs 2024-07-26T16:35:22.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:22 smithi134 ceph-mon[31966]: pgmap v10480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18479.889492 secs 2024-07-26T16:35:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:22 smithi189 ceph-mon[36790]: pgmap v10480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18479.889492 secs 2024-07-26T16:35:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:22 smithi193 ceph-mon[37473]: pgmap v10480: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:24 smithi134 ceph-mon[31966]: pgmap v10481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18465.467832 secs 2024-07-26T16:35:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:24 smithi189 ceph-mon[36790]: pgmap v10481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18465.467832 secs 2024-07-26T16:35:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:24 smithi193 ceph-mon[37473]: pgmap v10481: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18465.467832 secs 2024-07-26T16:35:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:26 smithi134 ceph-mon[31966]: pgmap v10482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:26.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:26 smithi189 ceph-mon[36790]: pgmap v10482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:26 smithi193 ceph-mon[37473]: pgmap v10482: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18484.889608 secs 2024-07-26T16:35:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18484.889608 secs 2024-07-26T16:35:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18484.889608 secs 2024-07-26T16:35:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:35:28.144+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:35:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:35:28.144+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:35:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:35:28.144+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:35:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:28 smithi134 ceph-mon[31966]: pgmap v10483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:28 smithi189 ceph-mon[36790]: pgmap v10483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:28 smithi193 ceph-mon[37473]: pgmap v10483: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:29.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18470.467978 secs 2024-07-26T16:35:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18470.467978 secs 2024-07-26T16:35:30.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18470.467978 secs 2024-07-26T16:35:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:30 smithi134 ceph-mon[31966]: pgmap v10484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:30 smithi189 ceph-mon[36790]: pgmap v10484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:30 smithi193 ceph-mon[37473]: pgmap v10484: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:32.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:35:32.566+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:35:32.568 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:35:32.566+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:35:32.569 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:35:32.566+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:35:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18489.889697 secs 2024-07-26T16:35:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:32 smithi134 ceph-mon[31966]: pgmap v10485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18489.889697 secs 2024-07-26T16:35:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:32 smithi189 ceph-mon[36790]: pgmap v10485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18489.889697 secs 2024-07-26T16:35:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:32 smithi193 ceph-mon[37473]: pgmap v10485: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:34 smithi134 ceph-mon[31966]: pgmap v10486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:34.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18475.468103 secs 2024-07-26T16:35:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:34 smithi189 ceph-mon[36790]: pgmap v10486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18475.468103 secs 2024-07-26T16:35:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:34 smithi193 ceph-mon[37473]: pgmap v10486: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18475.468103 secs 2024-07-26T16:35:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:36 smithi134 ceph-mon[31966]: pgmap v10487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:36 smithi189 ceph-mon[36790]: pgmap v10487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:36 smithi193 ceph-mon[37473]: pgmap v10487: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18494.889819 secs 2024-07-26T16:35:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18494.889819 secs 2024-07-26T16:35:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18494.889819 secs 2024-07-26T16:35:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:38 smithi134 ceph-mon[31966]: pgmap v10488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:38.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:38 smithi189 ceph-mon[36790]: pgmap v10488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:38 smithi193 ceph-mon[37473]: pgmap v10488: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18480.468255 secs 2024-07-26T16:35:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18480.468255 secs 2024-07-26T16:35:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18480.468255 secs 2024-07-26T16:35:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:40 smithi134 ceph-mon[31966]: pgmap v10489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:40.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:40 smithi189 ceph-mon[36790]: pgmap v10489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:40 smithi193 ceph-mon[37473]: pgmap v10489: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18499.889960 secs 2024-07-26T16:35:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18499.889960 secs 2024-07-26T16:35:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18499.889960 secs 2024-07-26T16:35:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:42 smithi189 ceph-mon[36790]: pgmap v10490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:42 smithi193 ceph-mon[37473]: pgmap v10490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:35:43.148+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:35:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:35:43.148+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:35:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:35:43.148+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:35:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:42 smithi134 ceph-mon[31966]: pgmap v10490: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:44 smithi189 ceph-mon[36790]: pgmap v10491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18485.468324 secs 2024-07-26T16:35:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:44 smithi193 ceph-mon[37473]: pgmap v10491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:45.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18485.468324 secs 2024-07-26T16:35:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:44 smithi134 ceph-mon[31966]: pgmap v10491: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18485.468324 secs 2024-07-26T16:35:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:46 smithi189 ceph-mon[36790]: pgmap v10492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18504.890091 secs 2024-07-26T16:35:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:46 smithi193 ceph-mon[37473]: pgmap v10492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18504.890091 secs 2024-07-26T16:35:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:46 smithi134 ceph-mon[31966]: pgmap v10492: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18504.890091 secs 2024-07-26T16:35:47.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:35:47.582+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:35:47.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:35:47.582+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:35:47.584 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:35:47.582+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:35:48.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:48 smithi189 ceph-mon[36790]: pgmap v10493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:35:49.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:48 smithi193 ceph-mon[37473]: pgmap v10493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:35:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:48 smithi134 ceph-mon[31966]: pgmap v10493: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-07-26T16:35:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18490.468474 secs 2024-07-26T16:35:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18490.468474 secs 2024-07-26T16:35:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18490.468474 secs 2024-07-26T16:35:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:50 smithi193 ceph-mon[37473]: pgmap v10494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:50 smithi134 ceph-mon[31966]: pgmap v10494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:50 smithi189 ceph-mon[36790]: pgmap v10494: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:35:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18509.890251 secs 2024-07-26T16:35:52.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18509.890251 secs 2024-07-26T16:35:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18509.890251 secs 2024-07-26T16:35:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:52 smithi193 ceph-mon[37473]: pgmap v10495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:52 smithi134 ceph-mon[31966]: pgmap v10495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:52 smithi189 ceph-mon[36790]: pgmap v10495: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:54 smithi193 ceph-mon[37473]: pgmap v10496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18495.468593 secs 2024-07-26T16:35:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:54 smithi134 ceph-mon[31966]: pgmap v10496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18495.468593 secs 2024-07-26T16:35:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:54 smithi189 ceph-mon[36790]: pgmap v10496: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:35:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18495.468593 secs 2024-07-26T16:35:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:56 smithi193 ceph-mon[37473]: pgmap v10497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18514.890351 secs 2024-07-26T16:35:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:56 smithi134 ceph-mon[31966]: pgmap v10497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18514.890351 secs 2024-07-26T16:35:57.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:56 smithi189 ceph-mon[36790]: pgmap v10497: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:35:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18514.890351 secs 2024-07-26T16:35:58.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:35:58.161+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:35:58.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:35:58.161+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:35:58.163 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:35:58.161+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:35:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:58 smithi193 ceph-mon[37473]: pgmap v10498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:58 smithi134 ceph-mon[31966]: pgmap v10498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:35:59.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:58 smithi189 ceph-mon[36790]: pgmap v10498: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:35:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18500.468713 secs 2024-07-26T16:36:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:35:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18500.468713 secs 2024-07-26T16:36:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:35:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18500.468713 secs 2024-07-26T16:36:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:00 smithi193 ceph-mon[37473]: pgmap v10499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:00 smithi134 ceph-mon[31966]: pgmap v10499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:01.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:00 smithi189 ceph-mon[36790]: pgmap v10499: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18519.890474 secs 2024-07-26T16:36:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18519.890474 secs 2024-07-26T16:36:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18519.890474 secs 2024-07-26T16:36:02.594 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:36:02.592+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:36:02.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:36:02.592+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:36:02.595 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:36:02.592+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:36:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:02 smithi134 ceph-mon[31966]: pgmap v10500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:02 smithi193 ceph-mon[37473]: pgmap v10500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:02 smithi189 ceph-mon[36790]: pgmap v10500: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:03 smithi193 ceph-mon[37473]: pgmap v10501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:36:04.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:03 smithi189 ceph-mon[36790]: pgmap v10501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:36:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:03 smithi134 ceph-mon[31966]: pgmap v10501: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:36:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18505.468803 secs 2024-07-26T16:36:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18505.468803 secs 2024-07-26T16:36:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18505.468803 secs 2024-07-26T16:36:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:05 smithi134 ceph-mon[31966]: pgmap v10502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:05 smithi189 ceph-mon[36790]: pgmap v10502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:05 smithi193 ceph-mon[37473]: pgmap v10502: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18524.890566 secs 2024-07-26T16:36:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18524.890566 secs 2024-07-26T16:36:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18524.890566 secs 2024-07-26T16:36:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:07 smithi134 ceph-mon[31966]: pgmap v10503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:08.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:07 smithi189 ceph-mon[36790]: pgmap v10503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:07 smithi193 ceph-mon[37473]: pgmap v10503: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18510.468919 secs 2024-07-26T16:36:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18510.468919 secs 2024-07-26T16:36:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18510.468919 secs 2024-07-26T16:36:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:10 smithi134 ceph-mon[31966]: pgmap v10504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:36:10.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:10 smithi189 ceph-mon[36790]: pgmap v10504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:36:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:10 smithi193 ceph-mon[37473]: pgmap v10504: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:36:11.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:36:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:36:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:11 smithi134 ceph-mon[31966]: pgmap v10505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:36:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:11 smithi134 ceph-mon[31966]: pgmap v10506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:36:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:36:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:36:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:36:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:11 smithi189 ceph-mon[36790]: pgmap v10505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:36:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:11 smithi189 ceph-mon[36790]: pgmap v10506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:36:11.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:36:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:36:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:36:11.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:11 smithi193 ceph-mon[37473]: pgmap v10505: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:36:11.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:11 smithi193 ceph-mon[37473]: pgmap v10506: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:36:11.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:36:12.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18529.890663 secs 2024-07-26T16:36:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18529.890663 secs 2024-07-26T16:36:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18529.890663 secs 2024-07-26T16:36:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:36:13.176+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:36:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:36:13.176+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:36:13.178 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:36:13.176+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:36:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:13 smithi134 ceph-mon[31966]: pgmap v10507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T16:36:13.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:13 smithi189 ceph-mon[36790]: pgmap v10507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T16:36:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:13 smithi193 ceph-mon[37473]: pgmap v10507: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T16:36:14.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18515.469038 secs 2024-07-26T16:36:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18515.469038 secs 2024-07-26T16:36:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18515.469038 secs 2024-07-26T16:36:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:15 smithi189 ceph-mon[36790]: pgmap v10508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:36:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:15 smithi193 ceph-mon[37473]: pgmap v10508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:36:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:15 smithi134 ceph-mon[31966]: pgmap v10508: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:36:17.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:36:17.608+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:36:17.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:36:17.608+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:36:17.610 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:36:17.608+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:36:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18534.890789 secs 2024-07-26T16:36:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:17 smithi134 ceph-mon[31966]: pgmap v10509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:36:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18534.890789 secs 2024-07-26T16:36:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:17 smithi189 ceph-mon[36790]: pgmap v10509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:36:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18534.890789 secs 2024-07-26T16:36:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:17 smithi193 ceph-mon[37473]: pgmap v10509: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:36:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:18 smithi189 ceph-mon[36790]: pgmap v10510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:36:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:18 smithi193 ceph-mon[37473]: pgmap v10510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:36:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:18 smithi134 ceph-mon[31966]: pgmap v10510: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:36:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18520.469155 secs 2024-07-26T16:36:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18520.469155 secs 2024-07-26T16:36:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18520.469155 secs 2024-07-26T16:36:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:20 smithi193 ceph-mon[37473]: pgmap v10511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:36:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:20 smithi134 ceph-mon[31966]: pgmap v10511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:36:21.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:20 smithi189 ceph-mon[36790]: pgmap v10511: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:36:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18539.890915 secs 2024-07-26T16:36:22.281 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18539.890915 secs 2024-07-26T16:36:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18539.890915 secs 2024-07-26T16:36:23.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:22 smithi193 ceph-mon[37473]: pgmap v10512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T16:36:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:22 smithi134 ceph-mon[31966]: pgmap v10512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T16:36:23.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:22 smithi189 ceph-mon[36790]: pgmap v10512: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T16:36:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18525.469282 secs 2024-07-26T16:36:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18525.469282 secs 2024-07-26T16:36:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18525.469282 secs 2024-07-26T16:36:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:25 smithi134 ceph-mon[31966]: pgmap v10513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:25.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:25 smithi189 ceph-mon[36790]: pgmap v10513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:25 smithi193 ceph-mon[37473]: pgmap v10513: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:36:25.549 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:36:25.548+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T16:36:25.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:36:25.548+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T16:36:25.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:36:25.548+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T16:36:25.550 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:36:25.548+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 144 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T16:36:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18544.891016 secs 2024-07-26T16:36:27.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:27 smithi134 ceph-mon[31966]: pgmap v10514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18544.891016 secs 2024-07-26T16:36:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:27 smithi189 ceph-mon[36790]: pgmap v10514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18544.891016 secs 2024-07-26T16:36:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:27 smithi193 ceph-mon[37473]: pgmap v10514: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:28.193 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:36:28.192+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:36:28.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:36:28.192+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:36:28.194 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:36:28.192+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:36:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:29 smithi134 ceph-mon[31966]: pgmap v10515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:36:29.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18530.469372 secs 2024-07-26T16:36:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:29 smithi189 ceph-mon[36790]: pgmap v10515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:36:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18530.469372 secs 2024-07-26T16:36:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:29 smithi193 ceph-mon[37473]: pgmap v10515: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:36:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18530.469372 secs 2024-07-26T16:36:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:31 smithi134 ceph-mon[31966]: pgmap v10516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:31 smithi189 ceph-mon[36790]: pgmap v10516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:31 smithi193 ceph-mon[37473]: pgmap v10516: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:32.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:36:32.619+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:36:32.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:36:32.619+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:36:32.621 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:36:32.619+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:36:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18549.891144 secs 2024-07-26T16:36:32.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18549.891144 secs 2024-07-26T16:36:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18549.891144 secs 2024-07-26T16:36:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:33 smithi134 ceph-mon[31966]: pgmap v10517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T16:36:33.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:33 smithi189 ceph-mon[36790]: pgmap v10517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T16:36:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:33 smithi193 ceph-mon[37473]: pgmap v10517: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T16:36:34.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18535.469511 secs 2024-07-26T16:36:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18535.469511 secs 2024-07-26T16:36:34.926 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18535.469511 secs 2024-07-26T16:36:35.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:35 smithi134 ceph-mon[31966]: pgmap v10518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:35.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:35 smithi189 ceph-mon[36790]: pgmap v10518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:35 smithi193 ceph-mon[37473]: pgmap v10518: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:36:36.101+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:36:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:36:36.101+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:36:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:36:36.102+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:36:36.103 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:36:36.102+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:36:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18554.891265 secs 2024-07-26T16:36:37.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:37 smithi134 ceph-mon[31966]: pgmap v10519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18554.891265 secs 2024-07-26T16:36:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:37 smithi189 ceph-mon[36790]: pgmap v10519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18554.891265 secs 2024-07-26T16:36:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:37 smithi193 ceph-mon[37473]: pgmap v10519: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:39 smithi134 ceph-mon[31966]: pgmap v10520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:36:39.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18540.469605 secs 2024-07-26T16:36:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:39 smithi189 ceph-mon[36790]: pgmap v10520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:36:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18540.469605 secs 2024-07-26T16:36:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:39 smithi193 ceph-mon[37473]: pgmap v10520: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:36:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18540.469605 secs 2024-07-26T16:36:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:41 smithi134 ceph-mon[31966]: pgmap v10521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:41 smithi189 ceph-mon[36790]: pgmap v10521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:41 smithi193 ceph-mon[37473]: pgmap v10521: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18559.891393 secs 2024-07-26T16:36:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18559.891393 secs 2024-07-26T16:36:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18559.891393 secs 2024-07-26T16:36:43.209 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:36:43.207+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:36:43.209 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:36:43.207+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:36:43.209 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:36:43.207+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:36:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:43 smithi134 ceph-mon[31966]: pgmap v10522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T16:36:43.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:43 smithi189 ceph-mon[36790]: pgmap v10522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T16:36:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:43 smithi193 ceph-mon[37473]: pgmap v10522: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-07-26T16:36:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18545.469731 secs 2024-07-26T16:36:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18545.469731 secs 2024-07-26T16:36:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18545.469731 secs 2024-07-26T16:36:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:45 smithi134 ceph-mon[31966]: pgmap v10523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:45.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:45 smithi189 ceph-mon[36790]: pgmap v10523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:45 smithi193 ceph-mon[37473]: pgmap v10523: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:47.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:36:47.627+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:36:47.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:36:47.627+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:36:47.630 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:36:47.627+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:36:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18564.891453 secs 2024-07-26T16:36:47.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:47 smithi134 ceph-mon[31966]: pgmap v10524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18564.891453 secs 2024-07-26T16:36:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:47 smithi189 ceph-mon[36790]: pgmap v10524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18564.891453 secs 2024-07-26T16:36:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:47 smithi193 ceph-mon[37473]: pgmap v10524: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:49 smithi134 ceph-mon[31966]: pgmap v10525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:36:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18550.469867 secs 2024-07-26T16:36:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:49 smithi189 ceph-mon[36790]: pgmap v10525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:36:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18550.469867 secs 2024-07-26T16:36:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:49 smithi193 ceph-mon[37473]: pgmap v10525: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:36:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18550.469867 secs 2024-07-26T16:36:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:51 smithi134 ceph-mon[31966]: pgmap v10526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:51.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18569.891543 secs 2024-07-26T16:36:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:51 smithi189 ceph-mon[36790]: pgmap v10526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18569.891543 secs 2024-07-26T16:36:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:51 smithi193 ceph-mon[37473]: pgmap v10526: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:36:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18569.891543 secs 2024-07-26T16:36:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:53 smithi134 ceph-mon[31966]: pgmap v10527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T16:36:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:53 smithi189 ceph-mon[36790]: pgmap v10527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T16:36:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:53 smithi193 ceph-mon[37473]: pgmap v10527: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T16:36:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18555.470021 secs 2024-07-26T16:36:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18555.470021 secs 2024-07-26T16:36:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18555.470021 secs 2024-07-26T16:36:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:55 smithi134 ceph-mon[31966]: pgmap v10528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:55.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:55 smithi189 ceph-mon[36790]: pgmap v10528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:55 smithi193 ceph-mon[37473]: pgmap v10528: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:36:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18574.891628 secs 2024-07-26T16:36:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18574.891628 secs 2024-07-26T16:36:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18574.891628 secs 2024-07-26T16:36:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:57 smithi189 ceph-mon[36790]: pgmap v10529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:36:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:57 smithi193 ceph-mon[37473]: pgmap v10529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:36:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:57 smithi134 ceph-mon[31966]: pgmap v10529: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:36:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:36:58.212+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:36:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:36:58.212+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:36:58.214 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:36:58.212+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:36:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:59 smithi189 ceph-mon[36790]: pgmap v10530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:36:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:36:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18560.470132 secs 2024-07-26T16:37:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:59 smithi193 ceph-mon[37473]: pgmap v10530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:37:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:36:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18560.470132 secs 2024-07-26T16:37:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:59 smithi134 ceph-mon[31966]: pgmap v10530: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:37:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:36:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18560.470132 secs 2024-07-26T16:37:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:01 smithi193 ceph-mon[37473]: pgmap v10531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:37:02.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18579.891743 secs 2024-07-26T16:37:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:01 smithi134 ceph-mon[31966]: pgmap v10531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:37:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18579.891743 secs 2024-07-26T16:37:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:01 smithi189 ceph-mon[36790]: pgmap v10531: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:37:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18579.891743 secs 2024-07-26T16:37:02.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:37:02.632+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:37:02.634 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:37:02.632+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:37:02.635 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:37:02.632+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:37:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:03 smithi193 ceph-mon[37473]: pgmap v10532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T16:37:04.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:03 smithi134 ceph-mon[31966]: pgmap v10532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T16:37:04.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:03 smithi189 ceph-mon[36790]: pgmap v10532: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 7 op/s 2024-07-26T16:37:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18565.470249 secs 2024-07-26T16:37:05.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18565.470249 secs 2024-07-26T16:37:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18565.470249 secs 2024-07-26T16:37:06.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:05 smithi193 ceph-mon[37473]: pgmap v10533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:05 smithi134 ceph-mon[31966]: pgmap v10533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:06.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:05 smithi189 ceph-mon[36790]: pgmap v10533: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:37:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18584.891884 secs 2024-07-26T16:37:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18584.891884 secs 2024-07-26T16:37:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18584.891884 secs 2024-07-26T16:37:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:07 smithi193 ceph-mon[37473]: pgmap v10534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:07 smithi134 ceph-mon[31966]: pgmap v10534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:08.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:07 smithi189 ceph-mon[36790]: pgmap v10534: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:09 smithi193 ceph-mon[37473]: pgmap v10535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18570.470312 secs 2024-07-26T16:37:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:09 smithi134 ceph-mon[31966]: pgmap v10535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18570.470312 secs 2024-07-26T16:37:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:09 smithi189 ceph-mon[36790]: pgmap v10535: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18570.470312 secs 2024-07-26T16:37:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:10 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:37:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:10 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:37:11.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:10 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:37:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:11 smithi193 ceph-mon[37473]: pgmap v10536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18589.892008 secs 2024-07-26T16:37:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:37:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:37:12.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:11 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:37:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:11 smithi134 ceph-mon[31966]: pgmap v10536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:12.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18589.892008 secs 2024-07-26T16:37:12.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:37:12.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:37:12.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:11 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:37:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:11 smithi189 ceph-mon[36790]: pgmap v10536: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18589.892008 secs 2024-07-26T16:37:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:37:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:37:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:11 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:37:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:12 smithi193 ceph-mon[37473]: pgmap v10537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:37:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:12 smithi193 ceph-mon[37473]: pgmap v10538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-26T16:37:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:12 smithi134 ceph-mon[31966]: pgmap v10537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:37:13.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:12 smithi134 ceph-mon[31966]: pgmap v10538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-26T16:37:13.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:12 smithi189 ceph-mon[36790]: pgmap v10537: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:37:13.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:12 smithi189 ceph-mon[36790]: pgmap v10538: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2024-07-26T16:37:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:37:13.216+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:37:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:37:13.216+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:37:13.218 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:37:13.216+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:37:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:14 smithi193 ceph-mon[37473]: pgmap v10539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:37:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18575.470430 secs 2024-07-26T16:37:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:14 smithi134 ceph-mon[31966]: pgmap v10539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:37:15.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18575.470430 secs 2024-07-26T16:37:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:14 smithi189 ceph-mon[36790]: pgmap v10539: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:37:15.195 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18575.470430 secs 2024-07-26T16:37:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:16 smithi134 ceph-mon[31966]: pgmap v10540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:37:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18594.892133 secs 2024-07-26T16:37:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:16 smithi189 ceph-mon[36790]: pgmap v10540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:37:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18594.892133 secs 2024-07-26T16:37:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:16 smithi193 ceph-mon[37473]: pgmap v10540: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:37:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18594.892133 secs 2024-07-26T16:37:17.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:37:17.640+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:37:17.642 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:37:17.640+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:37:17.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:37:17.640+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:37:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:18 smithi134 ceph-mon[31966]: pgmap v10541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:37:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:18 smithi189 ceph-mon[36790]: pgmap v10541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:37:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:18 smithi193 ceph-mon[37473]: pgmap v10541: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:37:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18580.470548 secs 2024-07-26T16:37:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18580.470548 secs 2024-07-26T16:37:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18580.470548 secs 2024-07-26T16:37:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:20 smithi134 ceph-mon[31966]: pgmap v10542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:37:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:20 smithi189 ceph-mon[36790]: pgmap v10542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:37:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:20 smithi193 ceph-mon[37473]: pgmap v10542: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:37:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18599.892263 secs 2024-07-26T16:37:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:21 smithi134 ceph-mon[31966]: pgmap v10543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T16:37:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18599.892263 secs 2024-07-26T16:37:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:21 smithi189 ceph-mon[36790]: pgmap v10543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T16:37:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18599.892263 secs 2024-07-26T16:37:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:21 smithi193 ceph-mon[37473]: pgmap v10543: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-07-26T16:37:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:24 smithi134 ceph-mon[31966]: pgmap v10544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:37:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18585.470669 secs 2024-07-26T16:37:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:24 smithi189 ceph-mon[36790]: pgmap v10544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:37:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18585.470669 secs 2024-07-26T16:37:25.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:24 smithi193 ceph-mon[37473]: pgmap v10544: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:37:25.026 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18585.470669 secs 2024-07-26T16:37:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:26 smithi134 ceph-mon[31966]: pgmap v10545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:37:26.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18604.892391 secs 2024-07-26T16:37:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:26 smithi189 ceph-mon[36790]: pgmap v10545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:37:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18604.892391 secs 2024-07-26T16:37:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:26 smithi193 ceph-mon[37473]: pgmap v10545: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:37:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18604.892391 secs 2024-07-26T16:37:28.234 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:37:28.231+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:37:28.234 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:37:28.231+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:37:28.234 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:37:28.231+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:37:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:28 smithi189 ceph-mon[36790]: pgmap v10546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:37:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:28 smithi193 ceph-mon[37473]: pgmap v10546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:37:29.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:28 smithi134 ceph-mon[31966]: pgmap v10546: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:37:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18590.470781 secs 2024-07-26T16:37:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18590.470781 secs 2024-07-26T16:37:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18590.470781 secs 2024-07-26T16:37:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:30 smithi189 ceph-mon[36790]: pgmap v10547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:37:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:30 smithi193 ceph-mon[37473]: pgmap v10547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:37:31.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:30 smithi134 ceph-mon[31966]: pgmap v10547: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:37:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18609.892470 secs 2024-07-26T16:37:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18609.892470 secs 2024-07-26T16:37:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18609.892470 secs 2024-07-26T16:37:32.651 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:37:32.649+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:37:32.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:37:32.649+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:37:32.652 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:37:32.649+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:37:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:32 smithi134 ceph-mon[31966]: pgmap v10548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:32 smithi193 ceph-mon[37473]: pgmap v10548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:33.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:32 smithi189 ceph-mon[36790]: pgmap v10548: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:34 smithi193 ceph-mon[37473]: pgmap v10549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:35.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18595.470899 secs 2024-07-26T16:37:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:34 smithi134 ceph-mon[31966]: pgmap v10549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18595.470899 secs 2024-07-26T16:37:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:34 smithi189 ceph-mon[36790]: pgmap v10549: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18595.470899 secs 2024-07-26T16:37:36.996 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:36 smithi189 ceph-mon[36790]: pgmap v10550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:36.996 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18614.892597 secs 2024-07-26T16:37:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:36 smithi193 ceph-mon[37473]: pgmap v10550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18614.892597 secs 2024-07-26T16:37:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:36 smithi134 ceph-mon[31966]: pgmap v10550: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18614.892597 secs 2024-07-26T16:37:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:38 smithi193 ceph-mon[37473]: pgmap v10551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:38 smithi134 ceph-mon[31966]: pgmap v10551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:39.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:38 smithi189 ceph-mon[36790]: pgmap v10551: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18600.471019 secs 2024-07-26T16:37:40.106 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18600.471019 secs 2024-07-26T16:37:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18600.471019 secs 2024-07-26T16:37:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:40 smithi193 ceph-mon[37473]: pgmap v10552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:37:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:40 smithi134 ceph-mon[31966]: pgmap v10552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:37:41.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:40 smithi189 ceph-mon[36790]: pgmap v10552: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:37:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18619.892735 secs 2024-07-26T16:37:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18619.892735 secs 2024-07-26T16:37:42.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18619.892735 secs 2024-07-26T16:37:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:42 smithi193 ceph-mon[37473]: pgmap v10553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:42 smithi134 ceph-mon[31966]: pgmap v10553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:42 smithi189 ceph-mon[36790]: pgmap v10553: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:37:43.232+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:37:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:37:43.232+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:37:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:37:43.232+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:37:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:44 smithi193 ceph-mon[37473]: pgmap v10554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18605.471172 secs 2024-07-26T16:37:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:44 smithi134 ceph-mon[31966]: pgmap v10554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18605.471172 secs 2024-07-26T16:37:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:44 smithi189 ceph-mon[36790]: pgmap v10554: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18605.471172 secs 2024-07-26T16:37:47.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:46 smithi193 ceph-mon[37473]: pgmap v10555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:47.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18624.892865 secs 2024-07-26T16:37:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:46 smithi134 ceph-mon[31966]: pgmap v10555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:47.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18624.892865 secs 2024-07-26T16:37:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:46 smithi189 ceph-mon[36790]: pgmap v10555: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18624.892865 secs 2024-07-26T16:37:47.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:37:47.656+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:37:47.658 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:37:47.656+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:37:47.659 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:37:47.656+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:37:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:48 smithi193 ceph-mon[37473]: pgmap v10556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:48 smithi134 ceph-mon[31966]: pgmap v10556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:48 smithi189 ceph-mon[36790]: pgmap v10556: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18610.471286 secs 2024-07-26T16:37:50.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18610.471286 secs 2024-07-26T16:37:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18610.471286 secs 2024-07-26T16:37:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:50 smithi134 ceph-mon[31966]: pgmap v10557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:37:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:50 smithi189 ceph-mon[36790]: pgmap v10557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:37:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:50 smithi193 ceph-mon[37473]: pgmap v10557: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:37:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18629.892988 secs 2024-07-26T16:37:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18629.892988 secs 2024-07-26T16:37:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18629.892988 secs 2024-07-26T16:37:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:52 smithi134 ceph-mon[31966]: pgmap v10558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:52 smithi189 ceph-mon[36790]: pgmap v10558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:52 smithi193 ceph-mon[37473]: pgmap v10558: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:53 smithi134 ceph-mon[31966]: pgmap v10559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:53 smithi189 ceph-mon[36790]: pgmap v10559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:53 smithi193 ceph-mon[37473]: pgmap v10559: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18615.471383 secs 2024-07-26T16:37:55.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18615.471383 secs 2024-07-26T16:37:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18615.471383 secs 2024-07-26T16:37:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:55 smithi134 ceph-mon[31966]: pgmap v10560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:56.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:55 smithi189 ceph-mon[36790]: pgmap v10560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:55 smithi193 ceph-mon[37473]: pgmap v10560: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:37:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18634.893105 secs 2024-07-26T16:37:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18634.893105 secs 2024-07-26T16:37:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18634.893105 secs 2024-07-26T16:37:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:57 smithi134 ceph-mon[31966]: pgmap v10561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:57 smithi189 ceph-mon[36790]: pgmap v10561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:58.249 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:37:58.248+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:37:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:37:58.249+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:37:58.250 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:37:58.249+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:37:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:57 smithi193 ceph-mon[37473]: pgmap v10561: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:37:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:37:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18620.471517 secs 2024-07-26T16:37:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:37:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18620.471517 secs 2024-07-26T16:37:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:37:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18620.471517 secs 2024-07-26T16:38:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:00 smithi134 ceph-mon[31966]: pgmap v10562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:00.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:00 smithi189 ceph-mon[36790]: pgmap v10562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:00 smithi193 ceph-mon[37473]: pgmap v10562: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18639.893234 secs 2024-07-26T16:38:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18639.893234 secs 2024-07-26T16:38:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18639.893234 secs 2024-07-26T16:38:02.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:38:02.669+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:38:02.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:38:02.669+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:38:02.671 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:38:02.669+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:38:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:02 smithi134 ceph-mon[31966]: pgmap v10563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:02 smithi193 ceph-mon[37473]: pgmap v10563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:03.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:02 smithi189 ceph-mon[36790]: pgmap v10563: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:04 smithi193 ceph-mon[37473]: pgmap v10564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18625.471660 secs 2024-07-26T16:38:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:04 smithi134 ceph-mon[31966]: pgmap v10564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:05.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18625.471660 secs 2024-07-26T16:38:05.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:04 smithi189 ceph-mon[36790]: pgmap v10564: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18625.471660 secs 2024-07-26T16:38:06.998 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:06 smithi189 ceph-mon[36790]: pgmap v10565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:06.998 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18644.893326 secs 2024-07-26T16:38:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:06 smithi193 ceph-mon[37473]: pgmap v10565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18644.893326 secs 2024-07-26T16:38:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:06 smithi134 ceph-mon[31966]: pgmap v10565: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:07.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18644.893326 secs 2024-07-26T16:38:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:08 smithi193 ceph-mon[37473]: pgmap v10566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:08 smithi134 ceph-mon[31966]: pgmap v10566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:08 smithi189 ceph-mon[36790]: pgmap v10566: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18630.471808 secs 2024-07-26T16:38:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18630.471808 secs 2024-07-26T16:38:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18630.471808 secs 2024-07-26T16:38:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:10 smithi193 ceph-mon[37473]: pgmap v10567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:11.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:10 smithi134 ceph-mon[31966]: pgmap v10567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:11.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:10 smithi189 ceph-mon[36790]: pgmap v10567: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18649.893448 secs 2024-07-26T16:38:12.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18649.893448 secs 2024-07-26T16:38:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18649.893448 secs 2024-07-26T16:38:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:12 smithi193 ceph-mon[37473]: pgmap v10568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:12 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:38:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:12 smithi134 ceph-mon[31966]: pgmap v10568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:13.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:12 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:38:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:12 smithi189 ceph-mon[36790]: pgmap v10568: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:13.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:12 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:38:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:38:13.257+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:38:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:38:13.257+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:38:13.259 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:38:13.257+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:38:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:38:14.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:38:14.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:38:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:38:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:38:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:38:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:38:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:38:14.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:38:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:14 smithi134 ceph-mon[31966]: pgmap v10569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T16:38:15.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:14 smithi134 ceph-mon[31966]: pgmap v10570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:38:15.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18635.471918 secs 2024-07-26T16:38:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:14 smithi189 ceph-mon[36790]: pgmap v10569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T16:38:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:14 smithi189 ceph-mon[36790]: pgmap v10570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:38:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18635.471918 secs 2024-07-26T16:38:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:14 smithi193 ceph-mon[37473]: pgmap v10569: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-07-26T16:38:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:14 smithi193 ceph-mon[37473]: pgmap v10570: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:38:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18635.471918 secs 2024-07-26T16:38:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:16 smithi134 ceph-mon[31966]: pgmap v10571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:38:17.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18654.893567 secs 2024-07-26T16:38:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:16 smithi189 ceph-mon[36790]: pgmap v10571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:38:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18654.893567 secs 2024-07-26T16:38:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:16 smithi193 ceph-mon[37473]: pgmap v10571: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 7 op/s 2024-07-26T16:38:17.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18654.893567 secs 2024-07-26T16:38:17.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:38:17.684+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:38:17.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:38:17.684+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:38:17.687 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:38:17.684+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:38:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:18 smithi134 ceph-mon[31966]: pgmap v10572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:38:19.200 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:18 smithi189 ceph-mon[36790]: pgmap v10572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:38:19.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:18 smithi193 ceph-mon[37473]: pgmap v10572: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-07-26T16:38:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18640.472019 secs 2024-07-26T16:38:20.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18640.472019 secs 2024-07-26T16:38:20.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18640.472019 secs 2024-07-26T16:38:21.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:20 smithi134 ceph-mon[31966]: pgmap v10573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:38:21.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:20 smithi189 ceph-mon[36790]: pgmap v10573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:38:21.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:20 smithi193 ceph-mon[37473]: pgmap v10573: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:38:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:21 smithi134 ceph-mon[31966]: pgmap v10574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:38:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18659.893693 secs 2024-07-26T16:38:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:21 smithi189 ceph-mon[36790]: pgmap v10574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:38:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18659.893693 secs 2024-07-26T16:38:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:21 smithi193 ceph-mon[37473]: pgmap v10574: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 8 op/s 2024-07-26T16:38:22.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18659.893693 secs 2024-07-26T16:38:24.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:23 smithi189 ceph-mon[36790]: pgmap v10575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:38:24.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:23 smithi193 ceph-mon[37473]: pgmap v10575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:38:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:23 smithi134 ceph-mon[31966]: pgmap v10575: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:38:25.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18645.472088 secs 2024-07-26T16:38:25.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18645.472088 secs 2024-07-26T16:38:25.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18645.472088 secs 2024-07-26T16:38:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:25 smithi189 ceph-mon[36790]: pgmap v10576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:26.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:25 smithi193 ceph-mon[37473]: pgmap v10576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:26.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:25 smithi134 ceph-mon[31966]: pgmap v10576: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:27.209 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18664.893816 secs 2024-07-26T16:38:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18664.893816 secs 2024-07-26T16:38:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18664.893816 secs 2024-07-26T16:38:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:38:28.262+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:38:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:38:28.262+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:38:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:38:28.262+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:38:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:27 smithi193 ceph-mon[37473]: pgmap v10577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:28.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:27 smithi134 ceph-mon[31966]: pgmap v10577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:28.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:27 smithi189 ceph-mon[36790]: pgmap v10577: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18650.472181 secs 2024-07-26T16:38:30.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:29 smithi193 ceph-mon[37473]: pgmap v10578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18650.472181 secs 2024-07-26T16:38:30.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:29 smithi134 ceph-mon[31966]: pgmap v10578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18650.472181 secs 2024-07-26T16:38:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:29 smithi189 ceph-mon[36790]: pgmap v10578: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:31 smithi193 ceph-mon[37473]: pgmap v10579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18669.893933 secs 2024-07-26T16:38:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:31 smithi134 ceph-mon[31966]: pgmap v10579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:32.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18669.893933 secs 2024-07-26T16:38:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:31 smithi189 ceph-mon[36790]: pgmap v10579: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18669.893933 secs 2024-07-26T16:38:32.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:38:32.700+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:38:32.702 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:38:32.700+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:38:32.703 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:38:32.700+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:38:34.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:33 smithi193 ceph-mon[37473]: pgmap v10580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:33 smithi134 ceph-mon[31966]: pgmap v10580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:34.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:33 smithi189 ceph-mon[36790]: pgmap v10580: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18655.472296 secs 2024-07-26T16:38:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18655.472296 secs 2024-07-26T16:38:35.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18655.472296 secs 2024-07-26T16:38:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:35 smithi193 ceph-mon[37473]: pgmap v10581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:36.345 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:35 smithi189 ceph-mon[36790]: pgmap v10581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:36.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:35 smithi134 ceph-mon[31966]: pgmap v10581: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18674.894023 secs 2024-07-26T16:38:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18674.894023 secs 2024-07-26T16:38:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18674.894023 secs 2024-07-26T16:38:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:38 smithi193 ceph-mon[37473]: pgmap v10582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:38.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:38 smithi134 ceph-mon[31966]: pgmap v10582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:38.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:38 smithi189 ceph-mon[36790]: pgmap v10582: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:40 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18660.472386 secs 2024-07-26T16:38:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:40 smithi193 ceph-mon[37473]: pgmap v10583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:40.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:40 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18660.472386 secs 2024-07-26T16:38:40.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:40 smithi134 ceph-mon[31966]: pgmap v10583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:40.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:40 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18660.472386 secs 2024-07-26T16:38:40.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:40 smithi189 ceph-mon[36790]: pgmap v10583: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:42 smithi193 ceph-mon[37473]: pgmap v10584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18679.894138 secs 2024-07-26T16:38:42.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:42 smithi134 ceph-mon[31966]: pgmap v10584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:42.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18679.894138 secs 2024-07-26T16:38:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:42 smithi189 ceph-mon[36790]: pgmap v10584: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18679.894138 secs 2024-07-26T16:38:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:38:43.277+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:38:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:38:43.277+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:38:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:38:43.277+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:38:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:44 smithi193 ceph-mon[37473]: pgmap v10585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:44.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:44 smithi134 ceph-mon[31966]: pgmap v10585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:44.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:44 smithi189 ceph-mon[36790]: pgmap v10585: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:45.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:45 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18665.472539 secs 2024-07-26T16:38:45.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:45 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18665.472539 secs 2024-07-26T16:38:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:45 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18665.472539 secs 2024-07-26T16:38:46.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:46 smithi134 ceph-mon[31966]: pgmap v10586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:46.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:46 smithi189 ceph-mon[36790]: pgmap v10586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:46.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:46 smithi193 ceph-mon[37473]: pgmap v10586: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:47.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18684.894250 secs 2024-07-26T16:38:47.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18684.894250 secs 2024-07-26T16:38:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18684.894250 secs 2024-07-26T16:38:47.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:38:47.715+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:38:47.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:38:47.715+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:38:47.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:38:47.715+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:38:48.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:48 smithi134 ceph-mon[31966]: pgmap v10587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:48.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:48 smithi189 ceph-mon[36790]: pgmap v10587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:48.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:48 smithi193 ceph-mon[37473]: pgmap v10587: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:50.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:50 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18670.472620 secs 2024-07-26T16:38:50.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:50 smithi134 ceph-mon[31966]: pgmap v10588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:50 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18670.472620 secs 2024-07-26T16:38:50.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:50 smithi189 ceph-mon[36790]: pgmap v10588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:50 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18670.472620 secs 2024-07-26T16:38:50.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:50 smithi193 ceph-mon[37473]: pgmap v10588: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:52.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:52 smithi134 ceph-mon[31966]: pgmap v10589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:52.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18689.894374 secs 2024-07-26T16:38:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:52 smithi189 ceph-mon[36790]: pgmap v10589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:52.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18689.894374 secs 2024-07-26T16:38:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:52 smithi193 ceph-mon[37473]: pgmap v10589: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18689.894374 secs 2024-07-26T16:38:54.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:54 smithi134 ceph-mon[31966]: pgmap v10590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:54 smithi189 ceph-mon[36790]: pgmap v10590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:54 smithi193 ceph-mon[37473]: pgmap v10590: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:38:55.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:55 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18675.472688 secs 2024-07-26T16:38:55.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:55 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18675.472688 secs 2024-07-26T16:38:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:55 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18675.472688 secs 2024-07-26T16:38:56.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:56 smithi134 ceph-mon[31966]: pgmap v10591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:56.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:56 smithi189 ceph-mon[36790]: pgmap v10591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:56.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:56 smithi193 ceph-mon[37473]: pgmap v10591: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:38:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18694.894499 secs 2024-07-26T16:38:57.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18694.894499 secs 2024-07-26T16:38:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18694.894499 secs 2024-07-26T16:38:58.294 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:38:58.292+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:38:58.295 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:38:58.293+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:38:58.295 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:38:58.293+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:38:58.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:58 smithi134 ceph-mon[31966]: pgmap v10592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:58.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:58 smithi189 ceph-mon[36790]: pgmap v10592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:58.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:58 smithi193 ceph-mon[37473]: pgmap v10592: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:38:59.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:38:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18680.472792 secs 2024-07-26T16:38:59.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:38:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18680.472792 secs 2024-07-26T16:38:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:38:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18680.472792 secs 2024-07-26T16:39:00.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:00 smithi134 ceph-mon[31966]: pgmap v10593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:00.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:00 smithi189 ceph-mon[36790]: pgmap v10593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:00.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:00 smithi193 ceph-mon[37473]: pgmap v10593: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:02.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:02 smithi134 ceph-mon[31966]: pgmap v10594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:02.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18699.894627 secs 2024-07-26T16:39:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:02 smithi189 ceph-mon[36790]: pgmap v10594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:02.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18699.894627 secs 2024-07-26T16:39:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:02 smithi193 ceph-mon[37473]: pgmap v10594: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:02.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18699.894627 secs 2024-07-26T16:39:02.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:39:02.731+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:39:02.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:39:02.731+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:39:02.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:39:02.731+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:39:04.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:04 smithi134 ceph-mon[31966]: pgmap v10595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:04.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18685.472910 secs 2024-07-26T16:39:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:04 smithi189 ceph-mon[36790]: pgmap v10595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:04.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18685.472910 secs 2024-07-26T16:39:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:04 smithi193 ceph-mon[37473]: pgmap v10595: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:04.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18685.472910 secs 2024-07-26T16:39:06.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:06 smithi134 ceph-mon[31966]: pgmap v10596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:39:06.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:06 smithi189 ceph-mon[36790]: pgmap v10596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:39:06.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:06 smithi193 ceph-mon[37473]: pgmap v10596: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:39:07.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18704.894713 secs 2024-07-26T16:39:07.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18704.894713 secs 2024-07-26T16:39:07.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18704.894713 secs 2024-07-26T16:39:08.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:08 smithi134 ceph-mon[31966]: pgmap v10597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:08.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:08 smithi189 ceph-mon[36790]: pgmap v10597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:08.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:08 smithi193 ceph-mon[37473]: pgmap v10597: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:09.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18690.472992 secs 2024-07-26T16:39:09.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18690.472992 secs 2024-07-26T16:39:09.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18690.472992 secs 2024-07-26T16:39:10.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:10 smithi134 ceph-mon[31966]: pgmap v10598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:10.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:10 smithi189 ceph-mon[36790]: pgmap v10598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:10.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:10 smithi193 ceph-mon[37473]: pgmap v10598: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:12.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:12 smithi134 ceph-mon[31966]: pgmap v10599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:12.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18709.894853 secs 2024-07-26T16:39:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:12 smithi189 ceph-mon[36790]: pgmap v10599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:12.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18709.894853 secs 2024-07-26T16:39:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:12 smithi193 ceph-mon[37473]: pgmap v10599: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:12.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18709.894853 secs 2024-07-26T16:39:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:39:13.308+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:39:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:39:13.308+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:39:13.310 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:39:13.308+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:39:13.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:13 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:39:13.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:13 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:39:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:13 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:39:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:14 smithi189 ceph-mon[36790]: pgmap v10600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18695.473109 secs 2024-07-26T16:39:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:14.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:14.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:14.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:39:14.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:39:14.445 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:39:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:14 smithi193 ceph-mon[37473]: pgmap v10600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18695.473109 secs 2024-07-26T16:39:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:14.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:14.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:14.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:14.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:39:14.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:39:14.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:39:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:14 smithi134 ceph-mon[31966]: pgmap v10600: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18695.473109 secs 2024-07-26T16:39:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:39:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:39:14.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:39:14.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:39:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:15 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T16:39:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:15 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T16:39:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:15 smithi189 ceph-mon[36790]: pgmap v10601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:39:15.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:15 smithi189 ceph-mon[36790]: pgmap v10602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:39:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:15 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T16:39:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:15 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T16:39:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:15 smithi193 ceph-mon[37473]: pgmap v10601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:39:15.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:15 smithi193 ceph-mon[37473]: pgmap v10602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:39:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:15 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi134 to 176.4M 2024-07-26T16:39:15.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:15 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi134 to 185050521: error parsing value: Value '185050521' is below minimum 939524096 2024-07-26T16:39:15.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:15 smithi134 ceph-mon[31966]: pgmap v10601: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:39:15.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:15 smithi134 ceph-mon[31966]: pgmap v10602: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:39:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:17 smithi193 ceph-mon[37473]: pgmap v10603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:39:17.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18714.895006 secs 2024-07-26T16:39:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:17 smithi134 ceph-mon[31966]: pgmap v10603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:39:17.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18714.895006 secs 2024-07-26T16:39:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:17 smithi189 ceph-mon[36790]: pgmap v10603: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:39:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18714.895006 secs 2024-07-26T16:39:17.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:39:17.747+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:39:17.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:39:17.747+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:39:17.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:39:17.747+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:39:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:19 smithi193 ceph-mon[37473]: pgmap v10604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:39:19.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18700.473206 secs 2024-07-26T16:39:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:19 smithi134 ceph-mon[31966]: pgmap v10604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:39:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18700.473206 secs 2024-07-26T16:39:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:19 smithi189 ceph-mon[36790]: pgmap v10604: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:39:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18700.473206 secs 2024-07-26T16:39:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:21 smithi193 ceph-mon[37473]: pgmap v10605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:39:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:21 smithi134 ceph-mon[31966]: pgmap v10605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:39:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:21 smithi189 ceph-mon[36790]: pgmap v10605: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:39:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18719.895131 secs 2024-07-26T16:39:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18719.895131 secs 2024-07-26T16:39:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18719.895131 secs 2024-07-26T16:39:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:23 smithi193 ceph-mon[37473]: pgmap v10606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:39:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:23 smithi134 ceph-mon[31966]: pgmap v10606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:39:23.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:23 smithi189 ceph-mon[36790]: pgmap v10606: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2024-07-26T16:39:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18705.473283 secs 2024-07-26T16:39:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18705.473283 secs 2024-07-26T16:39:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18705.473283 secs 2024-07-26T16:39:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:25 smithi193 ceph-mon[37473]: pgmap v10607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:39:25.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:39:25.579+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T16:39:25.581 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:39:25.579+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T16:39:25.582 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:25 smithi134 ceph-mon[31966]: pgmap v10607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:39:25.585 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:39:25.580+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T16:39:25.586 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:39:25.580+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 145 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T16:39:25.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:25 smithi189 ceph-mon[36790]: pgmap v10607: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:39:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:27 smithi193 ceph-mon[37473]: pgmap v10608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18724.895261 secs 2024-07-26T16:39:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:27 smithi134 ceph-mon[31966]: pgmap v10608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18724.895261 secs 2024-07-26T16:39:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:27 smithi189 ceph-mon[36790]: pgmap v10608: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18724.895261 secs 2024-07-26T16:39:28.325 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:39:28.323+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:39:28.326 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:39:28.323+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:39:28.326 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:39:28.323+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:39:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:29 smithi134 ceph-mon[31966]: pgmap v10609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18710.473403 secs 2024-07-26T16:39:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:29 smithi193 ceph-mon[37473]: pgmap v10609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18710.473403 secs 2024-07-26T16:39:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:29 smithi189 ceph-mon[36790]: pgmap v10609: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18710.473403 secs 2024-07-26T16:39:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:31 smithi193 ceph-mon[37473]: pgmap v10610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:31.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:31 smithi134 ceph-mon[31966]: pgmap v10610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:31.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:31 smithi189 ceph-mon[36790]: pgmap v10610: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:32.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18729.895371 secs 2024-07-26T16:39:32.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18729.895371 secs 2024-07-26T16:39:32.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:39:32.753+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:39:32.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:39:32.753+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:39:32.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:39:32.753+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:39:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18729.895371 secs 2024-07-26T16:39:33.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:33 smithi134 ceph-mon[31966]: pgmap v10611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:39:33.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:33 smithi189 ceph-mon[36790]: pgmap v10611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:39:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:33 smithi193 ceph-mon[37473]: pgmap v10611: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:39:34.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18715.473532 secs 2024-07-26T16:39:34.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18715.473532 secs 2024-07-26T16:39:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18715.473532 secs 2024-07-26T16:39:35.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:35 smithi134 ceph-mon[31966]: pgmap v10612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:35.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:35 smithi189 ceph-mon[36790]: pgmap v10612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:35.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:35 smithi193 ceph-mon[37473]: pgmap v10612: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:36.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:39:36.132+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:39:36.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:39:36.132+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:39:36.134 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:39:36.133+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:39:36.135 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:39:36.133+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:39:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:37 smithi134 ceph-mon[31966]: pgmap v10613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:37.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18734.895489 secs 2024-07-26T16:39:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:37 smithi189 ceph-mon[36790]: pgmap v10613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:37.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18734.895489 secs 2024-07-26T16:39:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:37 smithi193 ceph-mon[37473]: pgmap v10613: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18734.895489 secs 2024-07-26T16:39:39.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:39 smithi134 ceph-mon[31966]: pgmap v10614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:39.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18720.473655 secs 2024-07-26T16:39:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:39 smithi189 ceph-mon[36790]: pgmap v10614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:39.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18720.473655 secs 2024-07-26T16:39:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:39 smithi193 ceph-mon[37473]: pgmap v10614: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18720.473655 secs 2024-07-26T16:39:41.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:41 smithi134 ceph-mon[31966]: pgmap v10615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:41.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:41 smithi189 ceph-mon[36790]: pgmap v10615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:41.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:41 smithi193 ceph-mon[37473]: pgmap v10615: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18739.895604 secs 2024-07-26T16:39:42.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18739.895604 secs 2024-07-26T16:39:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18739.895604 secs 2024-07-26T16:39:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:39:43.339+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:39:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:39:43.339+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:39:43.341 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:39:43.339+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:39:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:43 smithi134 ceph-mon[31966]: pgmap v10616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:39:43.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:43 smithi189 ceph-mon[36790]: pgmap v10616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:39:43.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:43 smithi193 ceph-mon[37473]: pgmap v10616: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:39:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18725.473775 secs 2024-07-26T16:39:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18725.473775 secs 2024-07-26T16:39:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18725.473775 secs 2024-07-26T16:39:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:45 smithi134 ceph-mon[31966]: pgmap v10617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:45.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:45 smithi189 ceph-mon[36790]: pgmap v10617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:45.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:45 smithi193 ceph-mon[37473]: pgmap v10617: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:47 smithi134 ceph-mon[31966]: pgmap v10618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:47.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18744.895721 secs 2024-07-26T16:39:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:47 smithi189 ceph-mon[36790]: pgmap v10618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18744.895721 secs 2024-07-26T16:39:47.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:39:47.769+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:39:47.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:39:47.769+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:39:47.771 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:39:47.769+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:39:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:47 smithi193 ceph-mon[37473]: pgmap v10618: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18744.895721 secs 2024-07-26T16:39:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:49 smithi134 ceph-mon[31966]: pgmap v10619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18730.473901 secs 2024-07-26T16:39:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:49 smithi189 ceph-mon[36790]: pgmap v10619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18730.473901 secs 2024-07-26T16:39:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:49 smithi193 ceph-mon[37473]: pgmap v10619: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18730.473901 secs 2024-07-26T16:39:51.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:51 smithi134 ceph-mon[31966]: pgmap v10620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:51 smithi189 ceph-mon[36790]: pgmap v10620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:51.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:51 smithi193 ceph-mon[37473]: pgmap v10620: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18749.895842 secs 2024-07-26T16:39:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18749.895842 secs 2024-07-26T16:39:52.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18749.895842 secs 2024-07-26T16:39:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:53 smithi134 ceph-mon[31966]: pgmap v10621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:39:53.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:53 smithi189 ceph-mon[36790]: pgmap v10621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:39:53.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:53 smithi193 ceph-mon[37473]: pgmap v10621: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:39:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18735.474038 secs 2024-07-26T16:39:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18735.474038 secs 2024-07-26T16:39:54.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18735.474038 secs 2024-07-26T16:39:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:55 smithi134 ceph-mon[31966]: pgmap v10622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:55 smithi189 ceph-mon[36790]: pgmap v10622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:55.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:55 smithi193 ceph-mon[37473]: pgmap v10622: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:39:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:57 smithi134 ceph-mon[31966]: pgmap v10623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18754.895964 secs 2024-07-26T16:39:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:57 smithi189 ceph-mon[36790]: pgmap v10623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18754.895964 secs 2024-07-26T16:39:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:57 smithi193 ceph-mon[37473]: pgmap v10623: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:39:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18754.895964 secs 2024-07-26T16:39:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:39:58.346+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:39:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:39:58.346+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:39:58.348 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:39:58.346+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:39:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:59 smithi134 ceph-mon[31966]: pgmap v10624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:59.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:39:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18740.474121 secs 2024-07-26T16:39:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:59 smithi189 ceph-mon[36790]: pgmap v10624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:39:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18740.474121 secs 2024-07-26T16:39:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:59 smithi193 ceph-mon[37473]: pgmap v10624: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:39:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:39:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18740.474121 secs 2024-07-26T16:40:00.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:00 smithi134 ceph-mon[31966]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T16:40:00.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:00 smithi189 ceph-mon[36790]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T16:40:00.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:00 smithi193 ceph-mon[37473]: overall HEALTH_WARN 4 failed cephadm daemon(s); 2 MDSs report slow requests 2024-07-26T16:40:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:01 smithi134 ceph-mon[31966]: pgmap v10625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:01 smithi189 ceph-mon[36790]: pgmap v10625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:01 smithi193 ceph-mon[37473]: pgmap v10625: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18759.896080 secs 2024-07-26T16:40:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18759.896080 secs 2024-07-26T16:40:02.785 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:40:02.783+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:40:02.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:40:02.783+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:40:02.786 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:40:02.783+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:40:02.786 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18759.896080 secs 2024-07-26T16:40:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:03 smithi189 ceph-mon[36790]: pgmap v10626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:40:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:03 smithi193 ceph-mon[37473]: pgmap v10626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:40:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:03 smithi134 ceph-mon[31966]: pgmap v10626: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:40:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18745.474242 secs 2024-07-26T16:40:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18745.474242 secs 2024-07-26T16:40:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18745.474242 secs 2024-07-26T16:40:05.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:05 smithi189 ceph-mon[36790]: pgmap v10627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:05 smithi193 ceph-mon[37473]: pgmap v10627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:05 smithi134 ceph-mon[31966]: pgmap v10627: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:07 smithi193 ceph-mon[37473]: pgmap v10628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18764.896209 secs 2024-07-26T16:40:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:07 smithi134 ceph-mon[31966]: pgmap v10628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:07.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18764.896209 secs 2024-07-26T16:40:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:07 smithi189 ceph-mon[36790]: pgmap v10628: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18764.896209 secs 2024-07-26T16:40:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:09 smithi193 ceph-mon[37473]: pgmap v10629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:09.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18750.474312 secs 2024-07-26T16:40:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:09 smithi134 ceph-mon[31966]: pgmap v10629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18750.474312 secs 2024-07-26T16:40:09.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:09 smithi189 ceph-mon[36790]: pgmap v10629: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18750.474312 secs 2024-07-26T16:40:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:11 smithi193 ceph-mon[37473]: pgmap v10630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:11 smithi134 ceph-mon[31966]: pgmap v10630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:11 smithi189 ceph-mon[36790]: pgmap v10630: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18769.896360 secs 2024-07-26T16:40:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18769.896360 secs 2024-07-26T16:40:12.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18769.896360 secs 2024-07-26T16:40:13.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:40:13.351+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:40:13.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:40:13.351+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:40:13.353 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:40:13.351+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:40:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:13 smithi193 ceph-mon[37473]: pgmap v10631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:40:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:13 smithi134 ceph-mon[31966]: pgmap v10631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:40:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:13 smithi189 ceph-mon[36790]: pgmap v10631: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:40:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18755.474470 secs 2024-07-26T16:40:14.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:14 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:40:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18755.474470 secs 2024-07-26T16:40:14.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:14 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:40:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18755.474470 secs 2024-07-26T16:40:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:14 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:40:15.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:15 smithi134 ceph-mon[31966]: pgmap v10632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:15.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:15 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T16:40:15.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:15 smithi134 ceph-mon[31966]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T16:40:15.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:15 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T16:40:15.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:15 smithi134 ceph-mon[31966]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T16:40:15.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:40:15.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:40:15.674 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:40:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:15 smithi193 ceph-mon[37473]: pgmap v10632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:15 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T16:40:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:15 smithi193 ceph-mon[37473]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T16:40:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:15 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T16:40:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:15 smithi193 ceph-mon[37473]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T16:40:15.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:40:15.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:40:15.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:40:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:15 smithi189 ceph-mon[36790]: pgmap v10632: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:40:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-07-26T16:40:15.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:15 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi193 to 176.2M 2024-07-26T16:40:15.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:15 smithi189 ceph-mon[36790]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-07-26T16:40:15.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:15 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi193 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-07-26T16:40:15.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:15 smithi189 ceph-mon[36790]: Unable to set osd_memory_target on smithi189 to 184786739: error parsing value: Value '184786739' is below minimum 939524096 2024-07-26T16:40:15.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:40:15.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:40:15.945 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:40:16.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:16 smithi193 ceph-mon[37473]: pgmap v10633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:40:16.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:16 smithi193 ceph-mon[37473]: pgmap v10634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:40:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:16 smithi134 ceph-mon[31966]: pgmap v10633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:40:16.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:16 smithi134 ceph-mon[31966]: pgmap v10634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:40:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:16 smithi189 ceph-mon[36790]: pgmap v10633: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:40:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:16 smithi189 ceph-mon[36790]: pgmap v10634: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:40:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18774.896444 secs 2024-07-26T16:40:17.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:40:17.798+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:40:17.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:40:17.799+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:40:17.801 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:40:17.799+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:40:17.801 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18774.896444 secs 2024-07-26T16:40:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18774.896444 secs 2024-07-26T16:40:18.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:18 smithi193 ceph-mon[37473]: pgmap v10635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:40:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:18 smithi134 ceph-mon[31966]: pgmap v10635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:40:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:18 smithi189 ceph-mon[36790]: pgmap v10635: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:40:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18760.474638 secs 2024-07-26T16:40:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18760.474638 secs 2024-07-26T16:40:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18760.474638 secs 2024-07-26T16:40:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:20 smithi134 ceph-mon[31966]: pgmap v10636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:40:20.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:20 smithi189 ceph-mon[36790]: pgmap v10636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:40:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:20 smithi193 ceph-mon[37473]: pgmap v10636: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:40:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18779.896563 secs 2024-07-26T16:40:22.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:22 smithi134 ceph-mon[31966]: pgmap v10637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:40:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18779.896563 secs 2024-07-26T16:40:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:22 smithi189 ceph-mon[36790]: pgmap v10637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:40:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18779.896563 secs 2024-07-26T16:40:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:22 smithi193 ceph-mon[37473]: pgmap v10637: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 8 op/s 2024-07-26T16:40:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:24 smithi134 ceph-mon[31966]: pgmap v10638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:40:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18765.474778 secs 2024-07-26T16:40:24.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:24 smithi189 ceph-mon[36790]: pgmap v10638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:40:24.946 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18765.474778 secs 2024-07-26T16:40:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:24 smithi193 ceph-mon[37473]: pgmap v10638: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:40:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18765.474778 secs 2024-07-26T16:40:26.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:26 smithi134 ceph-mon[31966]: pgmap v10639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:40:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:26 smithi189 ceph-mon[36790]: pgmap v10639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:40:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:26 smithi193 ceph-mon[37473]: pgmap v10639: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:40:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18784.896689 secs 2024-07-26T16:40:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18784.896689 secs 2024-07-26T16:40:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18784.896689 secs 2024-07-26T16:40:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:40:28.363+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:40:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:40:28.363+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:40:28.365 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:40:28.363+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:40:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:28 smithi134 ceph-mon[31966]: pgmap v10640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:40:28.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:28 smithi189 ceph-mon[36790]: pgmap v10640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:40:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:28 smithi193 ceph-mon[37473]: pgmap v10640: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:40:29.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18770.474923 secs 2024-07-26T16:40:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18770.474923 secs 2024-07-26T16:40:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18770.474923 secs 2024-07-26T16:40:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:30 smithi134 ceph-mon[31966]: pgmap v10641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:40:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:30 smithi189 ceph-mon[36790]: pgmap v10641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:40:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:30 smithi193 ceph-mon[37473]: pgmap v10641: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:40:32.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:40:32.814+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:40:32.816 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:40:32.814+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:40:32.817 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:40:32.814+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:40:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18789.896816 secs 2024-07-26T16:40:32.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:32 smithi134 ceph-mon[31966]: pgmap v10642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18789.896816 secs 2024-07-26T16:40:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:32 smithi189 ceph-mon[36790]: pgmap v10642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18789.896816 secs 2024-07-26T16:40:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:32 smithi193 ceph-mon[37473]: pgmap v10642: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:34.844 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:34 smithi193 ceph-mon[37473]: pgmap v10643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:34.844 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18775.475027 secs 2024-07-26T16:40:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:34 smithi134 ceph-mon[31966]: pgmap v10643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18775.475027 secs 2024-07-26T16:40:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:34 smithi189 ceph-mon[36790]: pgmap v10643: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:34.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18775.475027 secs 2024-07-26T16:40:36.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:36 smithi134 ceph-mon[31966]: pgmap v10644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:36.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:36 smithi189 ceph-mon[36790]: pgmap v10644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:37.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:36 smithi193 ceph-mon[37473]: pgmap v10644: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18794.896930 secs 2024-07-26T16:40:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18794.896930 secs 2024-07-26T16:40:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18794.896930 secs 2024-07-26T16:40:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:38 smithi134 ceph-mon[31966]: pgmap v10645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:40:38.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:38 smithi189 ceph-mon[36790]: pgmap v10645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:40:39.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:38 smithi193 ceph-mon[37473]: pgmap v10645: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:40:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18780.475153 secs 2024-07-26T16:40:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18780.475153 secs 2024-07-26T16:40:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18780.475153 secs 2024-07-26T16:40:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:40 smithi134 ceph-mon[31966]: pgmap v10646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:40:40.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:40 smithi189 ceph-mon[36790]: pgmap v10646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:40:41.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:40 smithi193 ceph-mon[37473]: pgmap v10646: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:40:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18799.897080 secs 2024-07-26T16:40:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:42 smithi134 ceph-mon[31966]: pgmap v10647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18799.897080 secs 2024-07-26T16:40:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:42 smithi189 ceph-mon[36790]: pgmap v10647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18799.897080 secs 2024-07-26T16:40:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:42 smithi193 ceph-mon[37473]: pgmap v10647: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:40:43.368+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:40:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:40:43.368+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:40:43.370 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:40:43.368+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:40:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:44 smithi134 ceph-mon[31966]: pgmap v10648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18785.475288 secs 2024-07-26T16:40:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:44 smithi189 ceph-mon[36790]: pgmap v10648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18785.475288 secs 2024-07-26T16:40:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:44 smithi193 ceph-mon[37473]: pgmap v10648: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18785.475288 secs 2024-07-26T16:40:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:46 smithi134 ceph-mon[31966]: pgmap v10649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:46.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:46 smithi189 ceph-mon[36790]: pgmap v10649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:47.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:46 smithi193 ceph-mon[37473]: pgmap v10649: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:47.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:40:47.830+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:40:47.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:40:47.830+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:40:47.832 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:40:47.830+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:40:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18804.897175 secs 2024-07-26T16:40:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18804.897175 secs 2024-07-26T16:40:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18804.897175 secs 2024-07-26T16:40:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:48 smithi134 ceph-mon[31966]: pgmap v10650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:40:48.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:48 smithi189 ceph-mon[36790]: pgmap v10650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:40:49.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:48 smithi193 ceph-mon[37473]: pgmap v10650: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:40:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18790.475367 secs 2024-07-26T16:40:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18790.475367 secs 2024-07-26T16:40:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18790.475367 secs 2024-07-26T16:40:50.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:50 smithi134 ceph-mon[31966]: pgmap v10651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:40:50.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:50 smithi189 ceph-mon[36790]: pgmap v10651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:40:51.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:50 smithi193 ceph-mon[37473]: pgmap v10651: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:40:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18809.897295 secs 2024-07-26T16:40:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:52 smithi189 ceph-mon[36790]: pgmap v10652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18809.897295 secs 2024-07-26T16:40:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:52 smithi193 ceph-mon[37473]: pgmap v10652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18809.897295 secs 2024-07-26T16:40:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:52 smithi134 ceph-mon[31966]: pgmap v10652: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:54 smithi189 ceph-mon[36790]: pgmap v10653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18795.475442 secs 2024-07-26T16:40:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:54 smithi193 ceph-mon[37473]: pgmap v10653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18795.475442 secs 2024-07-26T16:40:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:54 smithi134 ceph-mon[31966]: pgmap v10653: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:40:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18795.475442 secs 2024-07-26T16:40:55.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:40:55.595+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T16:40:55.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:40:55.595+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T16:40:55.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:40:55.595+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 146 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T16:40:55.597 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:40:55.596+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T16:40:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:56 smithi189 ceph-mon[36790]: pgmap v10654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:56 smithi193 ceph-mon[37473]: pgmap v10654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:56 smithi134 ceph-mon[31966]: pgmap v10654: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:40:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18814.897378 secs 2024-07-26T16:40:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18814.897378 secs 2024-07-26T16:40:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18814.897378 secs 2024-07-26T16:40:58.385 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:40:58.383+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:40:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:40:58.384+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:40:58.386 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:40:58.384+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:40:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:58 smithi193 ceph-mon[37473]: pgmap v10655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:40:59.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:58 smithi134 ceph-mon[31966]: pgmap v10655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:40:59.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:58 smithi189 ceph-mon[36790]: pgmap v10655: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:41:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:40:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18800.475563 secs 2024-07-26T16:41:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:40:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18800.475563 secs 2024-07-26T16:41:00.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:40:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18800.475563 secs 2024-07-26T16:41:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:00 smithi193 ceph-mon[37473]: pgmap v10656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:41:01.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:00 smithi134 ceph-mon[31966]: pgmap v10656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:41:01.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:00 smithi189 ceph-mon[36790]: pgmap v10656: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:41:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18819.897500 secs 2024-07-26T16:41:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18819.897500 secs 2024-07-26T16:41:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18819.897500 secs 2024-07-26T16:41:02.844 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:41:02.842+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:41:02.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:41:02.842+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:41:02.845 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:41:02.842+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:41:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:02 smithi193 ceph-mon[37473]: pgmap v10657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:03.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:02 smithi134 ceph-mon[31966]: pgmap v10657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:03.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:02 smithi189 ceph-mon[36790]: pgmap v10657: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:04 smithi193 ceph-mon[37473]: pgmap v10658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:05.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18805.475685 secs 2024-07-26T16:41:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:04 smithi134 ceph-mon[31966]: pgmap v10658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18805.475685 secs 2024-07-26T16:41:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:04 smithi189 ceph-mon[36790]: pgmap v10658: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18805.475685 secs 2024-07-26T16:41:06.150 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:41:06.148+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:41:06.150 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:41:06.148+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:41:06.150 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:41:06.149+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:41:06.150 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:41:06.149+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:41:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:06 smithi193 ceph-mon[37473]: pgmap v10659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18824.897618 secs 2024-07-26T16:41:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:06 smithi134 ceph-mon[31966]: pgmap v10659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:07.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18824.897618 secs 2024-07-26T16:41:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:06 smithi189 ceph-mon[36790]: pgmap v10659: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18824.897618 secs 2024-07-26T16:41:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:08 smithi193 ceph-mon[37473]: pgmap v10660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:41:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:08 smithi134 ceph-mon[31966]: pgmap v10660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:41:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:08 smithi189 ceph-mon[36790]: pgmap v10660: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:41:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18810.475806 secs 2024-07-26T16:41:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18810.475806 secs 2024-07-26T16:41:10.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18810.475806 secs 2024-07-26T16:41:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:10 smithi193 ceph-mon[37473]: pgmap v10661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:41:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:10 smithi134 ceph-mon[31966]: pgmap v10661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:41:11.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:10 smithi189 ceph-mon[36790]: pgmap v10661: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:41:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18829.897709 secs 2024-07-26T16:41:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18829.897709 secs 2024-07-26T16:41:12.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18829.897709 secs 2024-07-26T16:41:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:12 smithi134 ceph-mon[31966]: pgmap v10662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:13.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:12 smithi189 ceph-mon[36790]: pgmap v10662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:12 smithi193 ceph-mon[37473]: pgmap v10662: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:13.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:41:13.399+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:41:13.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:41:13.399+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:41:13.401 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:41:13.399+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:41:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:14 smithi134 ceph-mon[31966]: pgmap v10663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18815.475992 secs 2024-07-26T16:41:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:14 smithi189 ceph-mon[36790]: pgmap v10663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:15.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18815.475992 secs 2024-07-26T16:41:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:14 smithi193 ceph-mon[37473]: pgmap v10663: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:15.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18815.475992 secs 2024-07-26T16:41:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:15 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:41:16.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:15 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:41:16.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:15 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:41:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:16 smithi134 ceph-mon[31966]: pgmap v10664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:17.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18834.897836 secs 2024-07-26T16:41:17.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:41:17.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:41:17.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:41:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:16 smithi189 ceph-mon[36790]: pgmap v10664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18834.897836 secs 2024-07-26T16:41:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:41:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:41:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:41:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:16 smithi193 ceph-mon[37473]: pgmap v10664: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:17.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18834.897836 secs 2024-07-26T16:41:17.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:41:17.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:41:17.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:41:17.850 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:41:17.848+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:41:17.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:41:17.848+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:41:17.851 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:41:17.848+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:41:18.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:17 smithi134 ceph-mon[31966]: pgmap v10665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:41:18.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:17 smithi134 ceph-mon[31966]: pgmap v10666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:17 smithi189 ceph-mon[36790]: pgmap v10665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:41:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:17 smithi189 ceph-mon[36790]: pgmap v10666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:17 smithi193 ceph-mon[37473]: pgmap v10665: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:41:18.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:17 smithi193 ceph-mon[37473]: pgmap v10666: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:19.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:19 smithi134 ceph-mon[31966]: pgmap v10667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:41:19.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:19 smithi189 ceph-mon[36790]: pgmap v10667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:41:19.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:19 smithi193 ceph-mon[37473]: pgmap v10667: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:41:20.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:20 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18820.476139 secs 2024-07-26T16:41:20.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:20 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18820.476139 secs 2024-07-26T16:41:20.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:20 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18820.476139 secs 2024-07-26T16:41:21.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:21 smithi134 ceph-mon[31966]: pgmap v10668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:41:21.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:21 smithi189 ceph-mon[36790]: pgmap v10668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:41:21.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:21 smithi193 ceph-mon[37473]: pgmap v10668: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:41:22.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18839.897976 secs 2024-07-26T16:41:22.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18839.897976 secs 2024-07-26T16:41:22.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18839.897976 secs 2024-07-26T16:41:23.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:23 smithi134 ceph-mon[31966]: pgmap v10669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:23.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:23 smithi189 ceph-mon[36790]: pgmap v10669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:23.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:23 smithi193 ceph-mon[37473]: pgmap v10669: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:24.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18825.476271 secs 2024-07-26T16:41:24.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18825.476271 secs 2024-07-26T16:41:24.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18825.476271 secs 2024-07-26T16:41:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:25 smithi189 ceph-mon[36790]: pgmap v10670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:41:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:25 smithi193 ceph-mon[37473]: pgmap v10670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:41:25.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:25 smithi134 ceph-mon[31966]: pgmap v10670: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:41:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18844.898079 secs 2024-07-26T16:41:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:27 smithi193 ceph-mon[37473]: pgmap v10671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:41:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18844.898079 secs 2024-07-26T16:41:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:27 smithi134 ceph-mon[31966]: pgmap v10671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:41:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18844.898079 secs 2024-07-26T16:41:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:27 smithi189 ceph-mon[36790]: pgmap v10671: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:41:28.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:41:28.402+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:41:28.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:41:28.402+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:41:28.404 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:41:28.402+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:41:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:29 smithi193 ceph-mon[37473]: pgmap v10672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18830.476387 secs 2024-07-26T16:41:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:29 smithi134 ceph-mon[31966]: pgmap v10672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18830.476387 secs 2024-07-26T16:41:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:29 smithi189 ceph-mon[36790]: pgmap v10672: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18830.476387 secs 2024-07-26T16:41:31.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:31 smithi193 ceph-mon[37473]: pgmap v10673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:31 smithi134 ceph-mon[31966]: pgmap v10673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:31.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:31 smithi189 ceph-mon[36790]: pgmap v10673: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18849.898230 secs 2024-07-26T16:41:32.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:41:32.861+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:41:32.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:41:32.861+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:41:32.864 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:41:32.861+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:41:32.864 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18849.898230 secs 2024-07-26T16:41:32.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18849.898230 secs 2024-07-26T16:41:33.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:33 smithi193 ceph-mon[37473]: pgmap v10674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:41:33.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:33 smithi134 ceph-mon[31966]: pgmap v10674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:41:33.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:33 smithi189 ceph-mon[36790]: pgmap v10674: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:41:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18835.476524 secs 2024-07-26T16:41:34.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18835.476524 secs 2024-07-26T16:41:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18835.476524 secs 2024-07-26T16:41:35.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:35 smithi134 ceph-mon[31966]: pgmap v10675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:41:35.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:35 smithi189 ceph-mon[36790]: pgmap v10675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:41:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:35 smithi193 ceph-mon[37473]: pgmap v10675: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:41:37.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18854.898356 secs 2024-07-26T16:41:37.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:37 smithi134 ceph-mon[31966]: pgmap v10676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18854.898356 secs 2024-07-26T16:41:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:37 smithi189 ceph-mon[36790]: pgmap v10676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18854.898356 secs 2024-07-26T16:41:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:37 smithi193 ceph-mon[37473]: pgmap v10676: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:39 smithi134 ceph-mon[31966]: pgmap v10677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:39.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18840.476654 secs 2024-07-26T16:41:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:39 smithi189 ceph-mon[36790]: pgmap v10677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:39.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18840.476654 secs 2024-07-26T16:41:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:39 smithi193 ceph-mon[37473]: pgmap v10677: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18840.476654 secs 2024-07-26T16:41:41.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:41 smithi134 ceph-mon[31966]: pgmap v10678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:41.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:41 smithi189 ceph-mon[36790]: pgmap v10678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:41 smithi193 ceph-mon[37473]: pgmap v10678: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18859.898516 secs 2024-07-26T16:41:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18859.898516 secs 2024-07-26T16:41:43.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18859.898516 secs 2024-07-26T16:41:43.419 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:41:43.417+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:41:43.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:41:43.417+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:41:43.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:41:43.417+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:41:43.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:43 smithi134 ceph-mon[31966]: pgmap v10679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:41:43.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:43 smithi189 ceph-mon[36790]: pgmap v10679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:41:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:43 smithi193 ceph-mon[37473]: pgmap v10679: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:41:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18845.476811 secs 2024-07-26T16:41:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18845.476811 secs 2024-07-26T16:41:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18845.476811 secs 2024-07-26T16:41:45.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:45 smithi134 ceph-mon[31966]: pgmap v10680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:41:45.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:45 smithi189 ceph-mon[36790]: pgmap v10680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:41:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:45 smithi193 ceph-mon[37473]: pgmap v10680: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:41:47.872 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:41:47.870+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:41:47.873 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:41:47.870+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:41:47.873 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:41:47.870+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:41:47.873 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18864.898647 secs 2024-07-26T16:41:47.873 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:47 smithi134 ceph-mon[31966]: pgmap v10681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18864.898647 secs 2024-07-26T16:41:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:47 smithi189 ceph-mon[36790]: pgmap v10681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18864.898647 secs 2024-07-26T16:41:48.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:47 smithi193 ceph-mon[37473]: pgmap v10681: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:49 smithi134 ceph-mon[31966]: pgmap v10682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:49.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18850.476926 secs 2024-07-26T16:41:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:49 smithi189 ceph-mon[36790]: pgmap v10682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18850.476926 secs 2024-07-26T16:41:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:49 smithi193 ceph-mon[37473]: pgmap v10682: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:50.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18850.476926 secs 2024-07-26T16:41:51.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:51 smithi134 ceph-mon[31966]: pgmap v10683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:51.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:51 smithi189 ceph-mon[36790]: pgmap v10683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:52.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:51 smithi193 ceph-mon[37473]: pgmap v10683: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18869.898767 secs 2024-07-26T16:41:52.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18869.898767 secs 2024-07-26T16:41:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18869.898767 secs 2024-07-26T16:41:53.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:53 smithi134 ceph-mon[31966]: pgmap v10684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:41:53.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:53 smithi189 ceph-mon[36790]: pgmap v10684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:41:54.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:53 smithi193 ceph-mon[37473]: pgmap v10684: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:41:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18855.477051 secs 2024-07-26T16:41:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18855.477051 secs 2024-07-26T16:41:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18855.477051 secs 2024-07-26T16:41:55.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:55 smithi134 ceph-mon[31966]: pgmap v10685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:41:55.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:55 smithi189 ceph-mon[36790]: pgmap v10685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:41:56.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:55 smithi193 ceph-mon[37473]: pgmap v10685: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:41:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18874.898893 secs 2024-07-26T16:41:57.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:57 smithi134 ceph-mon[31966]: pgmap v10686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18874.898893 secs 2024-07-26T16:41:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:57 smithi189 ceph-mon[36790]: pgmap v10686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18874.898893 secs 2024-07-26T16:41:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:57 smithi193 ceph-mon[37473]: pgmap v10686: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:41:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:41:58.418+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:41:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:41:58.418+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:41:58.420 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:41:58.418+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:41:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:59 smithi134 ceph-mon[31966]: pgmap v10687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:59.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:41:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18860.477172 secs 2024-07-26T16:41:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:59 smithi189 ceph-mon[36790]: pgmap v10687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:41:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:41:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18860.477172 secs 2024-07-26T16:42:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:59 smithi193 ceph-mon[37473]: pgmap v10687: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:42:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:41:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18860.477172 secs 2024-07-26T16:42:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:01 smithi134 ceph-mon[31966]: pgmap v10688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:01.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:01 smithi189 ceph-mon[36790]: pgmap v10688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:02.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:01 smithi193 ceph-mon[37473]: pgmap v10688: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:02.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:02.885+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:42:02.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:02.886+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:42:02.888 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:02.886+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:42:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18879.898986 secs 2024-07-26T16:42:02.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18879.898986 secs 2024-07-26T16:42:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18879.898986 secs 2024-07-26T16:42:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:03 smithi134 ceph-mon[31966]: pgmap v10689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:42:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:03 smithi189 ceph-mon[36790]: pgmap v10689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:42:04.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:03 smithi193 ceph-mon[37473]: pgmap v10689: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:42:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18865.477302 secs 2024-07-26T16:42:04.925 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18865.477302 secs 2024-07-26T16:42:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18865.477302 secs 2024-07-26T16:42:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:05 smithi134 ceph-mon[31966]: pgmap v10690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:42:05.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:05 smithi189 ceph-mon[36790]: pgmap v10690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:42:06.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:05 smithi193 ceph-mon[37473]: pgmap v10690: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:42:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18884.899112 secs 2024-07-26T16:42:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:07 smithi189 ceph-mon[36790]: pgmap v10691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18884.899112 secs 2024-07-26T16:42:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:07 smithi193 ceph-mon[37473]: pgmap v10691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18884.899112 secs 2024-07-26T16:42:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:07 smithi134 ceph-mon[31966]: pgmap v10691: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:09 smithi189 ceph-mon[36790]: pgmap v10692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:42:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18870.477426 secs 2024-07-26T16:42:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:09 smithi193 ceph-mon[37473]: pgmap v10692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:42:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18870.477426 secs 2024-07-26T16:42:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:09 smithi134 ceph-mon[31966]: pgmap v10692: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-07-26T16:42:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18870.477426 secs 2024-07-26T16:42:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:11 smithi189 ceph-mon[36790]: pgmap v10693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:42:12.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:11 smithi193 ceph-mon[37473]: pgmap v10693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:42:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:11 smithi134 ceph-mon[31966]: pgmap v10693: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:42:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18889.899234 secs 2024-07-26T16:42:13.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18889.899234 secs 2024-07-26T16:42:13.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18889.899234 secs 2024-07-26T16:42:13.427 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:13.426+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:42:13.427 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:13.426+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:42:13.427 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:13.426+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:42:14.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:13 smithi193 ceph-mon[37473]: pgmap v10694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:42:14.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:13 smithi134 ceph-mon[31966]: pgmap v10694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:42:14.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:13 smithi189 ceph-mon[36790]: pgmap v10694: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:42:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18875.477523 secs 2024-07-26T16:42:15.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18875.477523 secs 2024-07-26T16:42:15.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18875.477523 secs 2024-07-26T16:42:16.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:15 smithi193 ceph-mon[37473]: pgmap v10695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:42:16.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:15 smithi134 ceph-mon[31966]: pgmap v10695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:42:16.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:15 smithi189 ceph-mon[36790]: pgmap v10695: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:42:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:16 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18894.899385 secs 2024-07-26T16:42:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:16 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:42:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:16 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18894.899385 secs 2024-07-26T16:42:17.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:16 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:42:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:16 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18894.899385 secs 2024-07-26T16:42:17.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:16 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:42:17.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:17.901+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:42:17.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:17.901+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:42:17.903 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:17.901+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:42:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:17 smithi193 ceph-mon[37473]: pgmap v10696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:42:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:42:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:17 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:42:18.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:17 smithi134 ceph-mon[31966]: pgmap v10696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:42:18.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:42:18.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:17 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:42:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:17 smithi189 ceph-mon[36790]: pgmap v10696: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:42:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:42:18.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:17 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:42:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:42:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:42:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:18 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:42:19.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:42:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:42:19.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:18 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:42:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:42:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:42:19.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:18 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:42:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:19 smithi193 ceph-mon[37473]: pgmap v10697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:42:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:19 smithi193 ceph-mon[37473]: pgmap v10698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:42:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18880.477657 secs 2024-07-26T16:42:20.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:19 smithi134 ceph-mon[31966]: pgmap v10697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:42:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:19 smithi134 ceph-mon[31966]: pgmap v10698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:42:20.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18880.477657 secs 2024-07-26T16:42:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:19 smithi189 ceph-mon[36790]: pgmap v10697: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:42:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:19 smithi189 ceph-mon[36790]: pgmap v10698: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:42:20.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18880.477657 secs 2024-07-26T16:42:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:21 smithi193 ceph-mon[37473]: pgmap v10699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:42:22.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:21 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18899.899515 secs 2024-07-26T16:42:22.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:21 smithi134 ceph-mon[31966]: pgmap v10699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:42:22.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:21 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18899.899515 secs 2024-07-26T16:42:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:21 smithi189 ceph-mon[36790]: pgmap v10699: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:42:22.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:21 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18899.899515 secs 2024-07-26T16:42:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:22.354+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 msgr2=0x7f9b300c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-07-26T16:42:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:22.355+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 msgr2=0x7f9b300c15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T16:42:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:22.355+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 0x7f9b300c15d0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f9b3806d000 tx=0x7f9b38043000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:42:22.357 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:22.355+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 0x7f9b300c15d0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f9b3806d000 tx=0x7f9b38043000 comp rx=0 tx=0).stop 2024-07-26T16:42:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:22.355+0000 7f9b49e00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 msgr2=0x7f9b300c15d0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T16:42:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:22.355+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 msgr2=0x7f9b300c15d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T16:42:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:22.355+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c11c0 0x7f9b300c15d0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T16:42:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:22.355+0000 7f9b42a00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1b10 0x7f9b300c0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T16:42:22.358 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:22.356+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1b10 0x7f9b300c0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:42:22.359 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:22.357+0000 7f9b49e00640 1 --2- 172.21.15.134:0/4006618154 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f9b300c1b10 0x7f9b300c0be0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f9b380096b0 tx=0x7f9b3805d040 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T16:42:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:23 smithi193 ceph-mon[37473]: pgmap v10700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:42:24.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:23 smithi134 ceph-mon[31966]: pgmap v10700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:42:24.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:23 smithi189 ceph-mon[36790]: pgmap v10700: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:42:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18885.477754 secs 2024-07-26T16:42:25.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18885.477754 secs 2024-07-26T16:42:25.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18885.477754 secs 2024-07-26T16:42:25.612 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:25.610+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T16:42:25.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:25.610+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T16:42:25.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:25.611+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T16:42:25.613 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:25.611+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 147 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T16:42:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:25 smithi134 ceph-mon[31966]: pgmap v10701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:42:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:25 smithi193 ceph-mon[37473]: pgmap v10701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:42:26.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:25 smithi189 ceph-mon[36790]: pgmap v10701: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:42:27.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:26 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18904.899604 secs 2024-07-26T16:42:27.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:26 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18904.899604 secs 2024-07-26T16:42:27.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:26 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18904.899604 secs 2024-07-26T16:42:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:27 smithi134 ceph-mon[31966]: pgmap v10702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:42:28.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:27 smithi189 ceph-mon[36790]: pgmap v10702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:42:28.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:27 smithi193 ceph-mon[37473]: pgmap v10702: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:42:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:28.441+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:42:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:28.441+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:42:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:28.441+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:42:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:29 smithi134 ceph-mon[31966]: pgmap v10703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:42:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18890.477859 secs 2024-07-26T16:42:30.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:29 smithi189 ceph-mon[36790]: pgmap v10703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:42:30.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18890.477859 secs 2024-07-26T16:42:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:29 smithi193 ceph-mon[37473]: pgmap v10703: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:42:30.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18890.477859 secs 2024-07-26T16:42:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:31 smithi134 ceph-mon[31966]: pgmap v10704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18909.899752 secs 2024-07-26T16:42:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:31 smithi189 ceph-mon[36790]: pgmap v10704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18909.899752 secs 2024-07-26T16:42:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:31 smithi193 ceph-mon[37473]: pgmap v10704: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18909.899752 secs 2024-07-26T16:42:32.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:32.905+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 msgr2=0x7f21ac0c4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-07-26T16:42:32.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:32.905+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 msgr2=0x7f21ac0c4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-07-26T16:42:32.907 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:32.905+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f21980038d0 tx=0x7f21b8063000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-07-26T16:42:32.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:32.905+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f21980038d0 tx=0x7f21b8063000 comp rx=0 tx=0).stop 2024-07-26T16:42:32.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:32.905+0000 7f21c7400640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 msgr2=0x7f21ac0c4210 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.134:6800/2671962751, fault. 2024-07-26T16:42:32.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:32.905+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 msgr2=0x7f21ac0c4210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-07-26T16:42:32.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:32.905+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0440 0x7f21ac0c4210 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-07-26T16:42:32.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:32.905+0000 7f21c4c00640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0020 0x7f21ac0c10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-07-26T16:42:32.908 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:32.906+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0020 0x7f21ac0c10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-07-26T16:42:32.909 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:32.907+0000 7f21c7400640 1 --2- 172.21.15.134:0/939165295 >> [v2:172.21.15.134:6800/2671962751,v1:172.21.15.134:6801/2671962751] conn(0x7f21ac0c0020 0x7f21ac0c10b0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f21980067c0 tx=0x7f21b806a070 comp rx=0 tx=0).ready entity=mgr.14211 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-07-26T16:42:32.914 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:32.912+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:42:32.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:32.912+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:42:32.915 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:32.912+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:42:34.077 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:33 smithi193 ceph-mon[37473]: pgmap v10705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:42:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:33 smithi134 ceph-mon[31966]: pgmap v10705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:42:34.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:33 smithi189 ceph-mon[36790]: pgmap v10705: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:42:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18895.477988 secs 2024-07-26T16:42:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18895.477988 secs 2024-07-26T16:42:35.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18895.477988 secs 2024-07-26T16:42:36.074 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:35 smithi134 ceph-mon[31966]: pgmap v10706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:42:36.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:36.164+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:42:36.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:36.164+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:42:36.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:36.165+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:42:36.166 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:36.165+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:42:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:35 smithi189 ceph-mon[36790]: pgmap v10706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:42:36.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:35 smithi193 ceph-mon[37473]: pgmap v10706: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:42:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18914.899823 secs 2024-07-26T16:42:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18914.899823 secs 2024-07-26T16:42:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18914.899823 secs 2024-07-26T16:42:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:37 smithi134 ceph-mon[31966]: pgmap v10707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:38.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:37 smithi189 ceph-mon[36790]: pgmap v10707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:38.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:37 smithi193 ceph-mon[37473]: pgmap v10707: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:39 smithi134 ceph-mon[31966]: pgmap v10708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:42:40.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18900.478070 secs 2024-07-26T16:42:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:39 smithi189 ceph-mon[36790]: pgmap v10708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:42:40.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18900.478070 secs 2024-07-26T16:42:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:39 smithi193 ceph-mon[37473]: pgmap v10708: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:42:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18900.478070 secs 2024-07-26T16:42:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:41 smithi134 ceph-mon[31966]: pgmap v10709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18919.899974 secs 2024-07-26T16:42:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:41 smithi189 ceph-mon[36790]: pgmap v10709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18919.899974 secs 2024-07-26T16:42:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:41 smithi193 ceph-mon[37473]: pgmap v10709: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18919.899974 secs 2024-07-26T16:42:43.458 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:43.457+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:42:43.458 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:43.457+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:42:43.459 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:43.457+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:42:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:43 smithi134 ceph-mon[31966]: pgmap v10710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:42:44.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:43 smithi189 ceph-mon[36790]: pgmap v10710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:42:44.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:43 smithi193 ceph-mon[37473]: pgmap v10710: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:42:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18905.478190 secs 2024-07-26T16:42:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18905.478190 secs 2024-07-26T16:42:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18905.478190 secs 2024-07-26T16:42:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:45 smithi134 ceph-mon[31966]: pgmap v10711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:42:46.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:45 smithi189 ceph-mon[36790]: pgmap v10711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:42:46.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:45 smithi193 ceph-mon[37473]: pgmap v10711: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:42:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18924.900118 secs 2024-07-26T16:42:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18924.900118 secs 2024-07-26T16:42:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18924.900118 secs 2024-07-26T16:42:47.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:47.928+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:42:47.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:47.928+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:42:47.930 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:42:47.928+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:42:48.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:47 smithi134 ceph-mon[31966]: pgmap v10712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:47 smithi189 ceph-mon[36790]: pgmap v10712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:47 smithi193 ceph-mon[37473]: pgmap v10712: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:48 smithi134 ceph-mon[31966]: pgmap v10713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:42:49.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:48 smithi189 ceph-mon[36790]: pgmap v10713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:42:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:48 smithi193 ceph-mon[37473]: pgmap v10713: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:42:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18910.478298 secs 2024-07-26T16:42:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18910.478298 secs 2024-07-26T16:42:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18910.478298 secs 2024-07-26T16:42:51.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:50 smithi134 ceph-mon[31966]: pgmap v10714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:51.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:50 smithi189 ceph-mon[36790]: pgmap v10714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:51.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:50 smithi193 ceph-mon[37473]: pgmap v10714: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18929.900247 secs 2024-07-26T16:42:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18929.900247 secs 2024-07-26T16:42:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18929.900247 secs 2024-07-26T16:42:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:52 smithi134 ceph-mon[31966]: pgmap v10715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:42:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:52 smithi189 ceph-mon[36790]: pgmap v10715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:42:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:52 smithi193 ceph-mon[37473]: pgmap v10715: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:42:54.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18915.478394 secs 2024-07-26T16:42:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18915.478394 secs 2024-07-26T16:42:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18915.478394 secs 2024-07-26T16:42:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:55 smithi193 ceph-mon[37473]: pgmap v10716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:42:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:55 smithi134 ceph-mon[31966]: pgmap v10716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:42:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:55 smithi189 ceph-mon[36790]: pgmap v10716: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:42:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18934.900367 secs 2024-07-26T16:42:57.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:57 smithi134 ceph-mon[31966]: pgmap v10717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18934.900367 secs 2024-07-26T16:42:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:57 smithi189 ceph-mon[36790]: pgmap v10717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:57.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18934.900367 secs 2024-07-26T16:42:57.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:57 smithi193 ceph-mon[37473]: pgmap v10717: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:42:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:58.472+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:42:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:58.472+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:42:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:42:58.472+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:42:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:59 smithi189 ceph-mon[36790]: pgmap v10718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:42:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:42:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18920.478514 secs 2024-07-26T16:42:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:59 smithi193 ceph-mon[37473]: pgmap v10718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:42:59.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:42:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18920.478514 secs 2024-07-26T16:42:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:59 smithi134 ceph-mon[31966]: pgmap v10718: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:42:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:42:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18920.478514 secs 2024-07-26T16:43:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:01 smithi189 ceph-mon[36790]: pgmap v10719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:01.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:01 smithi193 ceph-mon[37473]: pgmap v10719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:01.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:01 smithi134 ceph-mon[31966]: pgmap v10719: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18939.900494 secs 2024-07-26T16:43:02.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18939.900494 secs 2024-07-26T16:43:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18939.900494 secs 2024-07-26T16:43:02.945 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:43:02.943+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:43:02.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:43:02.943+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:43:02.946 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:43:02.943+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:43:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:03 smithi193 ceph-mon[37473]: pgmap v10720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:43:03.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:03 smithi134 ceph-mon[31966]: pgmap v10720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:43:03.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:03 smithi189 ceph-mon[36790]: pgmap v10720: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:43:04.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18925.478641 secs 2024-07-26T16:43:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18925.478641 secs 2024-07-26T16:43:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18925.478641 secs 2024-07-26T16:43:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:05 smithi193 ceph-mon[37473]: pgmap v10721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:43:05.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:05 smithi134 ceph-mon[31966]: pgmap v10721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:43:05.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:05 smithi189 ceph-mon[36790]: pgmap v10721: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:43:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18944.900560 secs 2024-07-26T16:43:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:07 smithi193 ceph-mon[37473]: pgmap v10722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18944.900560 secs 2024-07-26T16:43:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:07 smithi134 ceph-mon[31966]: pgmap v10722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18944.900560 secs 2024-07-26T16:43:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:07 smithi189 ceph-mon[36790]: pgmap v10722: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:09.746 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:09 smithi189 ceph-mon[36790]: pgmap v10723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:09.746 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18930.478795 secs 2024-07-26T16:43:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:09 smithi193 ceph-mon[37473]: pgmap v10723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18930.478795 secs 2024-07-26T16:43:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:09 smithi134 ceph-mon[31966]: pgmap v10723: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:09.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18930.478795 secs 2024-07-26T16:43:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:11 smithi193 ceph-mon[37473]: pgmap v10724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:11.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:11 smithi134 ceph-mon[31966]: pgmap v10724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:11.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:11 smithi189 ceph-mon[36790]: pgmap v10724: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18949.900680 secs 2024-07-26T16:43:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18949.900680 secs 2024-07-26T16:43:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18949.900680 secs 2024-07-26T16:43:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:43:13.487+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:43:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:43:13.488+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:43:13.490 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:43:13.488+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:43:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:13 smithi193 ceph-mon[37473]: pgmap v10725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:43:13.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:13 smithi134 ceph-mon[31966]: pgmap v10725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:43:13.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:13 smithi189 ceph-mon[36790]: pgmap v10725: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:43:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18935.478915 secs 2024-07-26T16:43:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18935.478915 secs 2024-07-26T16:43:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18935.478915 secs 2024-07-26T16:43:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:15 smithi193 ceph-mon[37473]: pgmap v10726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:43:15.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:15 smithi134 ceph-mon[31966]: pgmap v10726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:43:15.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:15 smithi189 ceph-mon[36790]: pgmap v10726: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:43:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18954.900797 secs 2024-07-26T16:43:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:17 smithi193 ceph-mon[37473]: pgmap v10727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18954.900797 secs 2024-07-26T16:43:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:17 smithi134 ceph-mon[31966]: pgmap v10727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18954.900797 secs 2024-07-26T16:43:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:17 smithi189 ceph-mon[36790]: pgmap v10727: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:17.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:43:17.952+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:43:17.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:43:17.952+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:43:17.954 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:43:17.952+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:43:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:19 smithi193 ceph-mon[37473]: pgmap v10728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:43:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:43:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18940.478992 secs 2024-07-26T16:43:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:19.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:19 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:19.811 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:19 smithi189 ceph-mon[36790]: pgmap v10728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:43:19.811 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:43:19.811 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18940.478992 secs 2024-07-26T16:43:19.811 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:19.811 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:19.811 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:19.811 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:19 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:19.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:19 smithi134 ceph-mon[31966]: pgmap v10728: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.9 KiB/s rd, 8 op/s 2024-07-26T16:43:19.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:43:19.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18940.478992 secs 2024-07-26T16:43:19.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:19.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:19.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:19.867 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:19 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:21 smithi134 ceph-mon[31966]: pgmap v10729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:43:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:21 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:21 smithi189 ceph-mon[36790]: pgmap v10729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:43:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:21 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:21 smithi193 ceph-mon[37473]: pgmap v10729: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:43:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:21.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:21 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18959.900919 secs 2024-07-26T16:43:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18959.900919 secs 2024-07-26T16:43:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18959.900919 secs 2024-07-26T16:43:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:23 smithi193 ceph-mon[37473]: pgmap v10730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:43:23.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:23.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:23 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:23 smithi134 ceph-mon[31966]: pgmap v10730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:43:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:23 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:23 smithi189 ceph-mon[36790]: pgmap v10730: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:43:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:23 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:25.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:25 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18945.479113 secs 2024-07-26T16:43:25.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:25.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:25.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:43:25.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:43:25.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:25 smithi134 ceph-mon[31966]: pgmap v10731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:43:25.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:25 smithi134 ceph-mon[31966]: pgmap v10732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:43:25.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:25 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18945.479113 secs 2024-07-26T16:43:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:43:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:43:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:25 smithi189 ceph-mon[36790]: pgmap v10731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:43:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:25 smithi189 ceph-mon[36790]: pgmap v10732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:43:25.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:25 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18945.479113 secs 2024-07-26T16:43:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:43:25.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:43:25.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:25 smithi193 ceph-mon[37473]: pgmap v10731: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:43:25.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:25 smithi193 ceph-mon[37473]: pgmap v10732: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:43:25.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:43:27.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:27 smithi134 ceph-mon[31966]: pgmap v10733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:43:27.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18964.901009 secs 2024-07-26T16:43:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:27 smithi189 ceph-mon[36790]: pgmap v10733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:43:27.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18964.901009 secs 2024-07-26T16:43:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:27 smithi193 ceph-mon[37473]: pgmap v10733: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:43:27.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18964.901009 secs 2024-07-26T16:43:28.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:43:28.494+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:43:28.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:43:28.494+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:43:28.496 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:43:28.494+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:43:29.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:29 smithi134 ceph-mon[31966]: pgmap v10734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:43:29.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:29 smithi189 ceph-mon[36790]: pgmap v10734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:43:29.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:29 smithi193 ceph-mon[37473]: pgmap v10734: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s 2024-07-26T16:43:30.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:30 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18950.479200 secs 2024-07-26T16:43:30.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:30 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18950.479200 secs 2024-07-26T16:43:30.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:30 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18950.479200 secs 2024-07-26T16:43:31.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:31 smithi134 ceph-mon[31966]: pgmap v10735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:43:31.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:31 smithi189 ceph-mon[36790]: pgmap v10735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:43:31.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:31 smithi193 ceph-mon[37473]: pgmap v10735: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 7 op/s 2024-07-26T16:43:32.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18969.901131 secs 2024-07-26T16:43:32.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18969.901131 secs 2024-07-26T16:43:32.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18969.901131 secs 2024-07-26T16:43:32.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:43:32.967+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:43:32.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:43:32.967+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:43:32.971 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:43:32.967+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:43:33.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:33 smithi134 ceph-mon[31966]: pgmap v10736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:43:33.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:33 smithi189 ceph-mon[36790]: pgmap v10736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:43:33.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:33 smithi193 ceph-mon[37473]: pgmap v10736: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 7 op/s 2024-07-26T16:43:34.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18955.479355 secs 2024-07-26T16:43:34.425 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18955.479355 secs 2024-07-26T16:43:34.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18955.479355 secs 2024-07-26T16:43:35.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:35 smithi134 ceph-mon[31966]: pgmap v10737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:43:35.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:35 smithi189 ceph-mon[36790]: pgmap v10737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:43:35.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:35 smithi193 ceph-mon[37473]: pgmap v10737: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:43:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:37 smithi134 ceph-mon[31966]: pgmap v10738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:37.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18974.901281 secs 2024-07-26T16:43:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:37 smithi189 ceph-mon[36790]: pgmap v10738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:37.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18974.901281 secs 2024-07-26T16:43:37.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:37 smithi193 ceph-mon[37473]: pgmap v10738: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:37.525 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18974.901281 secs 2024-07-26T16:43:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:39 smithi134 ceph-mon[31966]: pgmap v10739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:39.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18960.479435 secs 2024-07-26T16:43:39.443 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:39 smithi189 ceph-mon[36790]: pgmap v10739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:39.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18960.479435 secs 2024-07-26T16:43:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:39 smithi193 ceph-mon[37473]: pgmap v10739: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:39.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18960.479435 secs 2024-07-26T16:43:41.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:41 smithi189 ceph-mon[36790]: pgmap v10740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:43:41.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:41 smithi193 ceph-mon[37473]: pgmap v10740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:43:41.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:41 smithi134 ceph-mon[31966]: pgmap v10740: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:43:42.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18979.901356 secs 2024-07-26T16:43:42.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18979.901356 secs 2024-07-26T16:43:42.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18979.901356 secs 2024-07-26T16:43:43.444 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:43 smithi189 ceph-mon[36790]: pgmap v10741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:43:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:43:43.509+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:43:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:43:43.509+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:43:43.511 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:43:43.509+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:43:43.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:43 smithi193 ceph-mon[37473]: pgmap v10741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:43:43.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:43 smithi134 ceph-mon[31966]: pgmap v10741: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:43:44.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18965.479573 secs 2024-07-26T16:43:44.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18965.479573 secs 2024-07-26T16:43:44.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18965.479573 secs 2024-07-26T16:43:45.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:45 smithi193 ceph-mon[37473]: pgmap v10742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:45.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:45 smithi134 ceph-mon[31966]: pgmap v10742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:45.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:45 smithi189 ceph-mon[36790]: pgmap v10742: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:47 smithi193 ceph-mon[37473]: pgmap v10743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:47.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18984.901423 secs 2024-07-26T16:43:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:47 smithi134 ceph-mon[31966]: pgmap v10743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:47.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18984.901423 secs 2024-07-26T16:43:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:47 smithi189 ceph-mon[36790]: pgmap v10743: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:47.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18984.901423 secs 2024-07-26T16:43:47.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:43:47.971+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:43:47.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:43:47.971+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:43:47.975 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:43:47.971+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:43:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:49 smithi193 ceph-mon[37473]: pgmap v10744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:49.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18970.479699 secs 2024-07-26T16:43:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:49 smithi134 ceph-mon[31966]: pgmap v10744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:49.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18970.479699 secs 2024-07-26T16:43:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:49 smithi189 ceph-mon[36790]: pgmap v10744: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:49.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18970.479699 secs 2024-07-26T16:43:51.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:51 smithi134 ceph-mon[31966]: pgmap v10745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:43:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:51 smithi193 ceph-mon[37473]: pgmap v10745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:43:51.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:51 smithi189 ceph-mon[36790]: pgmap v10745: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:43:52.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18989.901496 secs 2024-07-26T16:43:52.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18989.901496 secs 2024-07-26T16:43:52.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18989.901496 secs 2024-07-26T16:43:53.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:53 smithi193 ceph-mon[37473]: pgmap v10746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:43:53.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:53 smithi134 ceph-mon[31966]: pgmap v10746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:43:53.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:53 smithi189 ceph-mon[36790]: pgmap v10746: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:43:54.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18975.479819 secs 2024-07-26T16:43:54.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18975.479819 secs 2024-07-26T16:43:54.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18975.479819 secs 2024-07-26T16:43:55.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:55 smithi193 ceph-mon[37473]: pgmap v10747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:55.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:55 smithi134 ceph-mon[31966]: pgmap v10747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:55.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:55 smithi189 ceph-mon[36790]: pgmap v10747: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:57 smithi193 ceph-mon[37473]: pgmap v10748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:57.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18994.901604 secs 2024-07-26T16:43:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:57 smithi134 ceph-mon[31966]: pgmap v10748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:57.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18994.901604 secs 2024-07-26T16:43:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:57 smithi189 ceph-mon[36790]: pgmap v10748: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:43:57.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18994.901604 secs 2024-07-26T16:43:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:43:58.525+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:43:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:43:58.525+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:43:58.527 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:43:58.525+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:43:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:59 smithi193 ceph-mon[37473]: pgmap v10749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:59.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:43:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18980.479956 secs 2024-07-26T16:43:59.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:59 smithi134 ceph-mon[31966]: pgmap v10749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:59.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:43:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18980.479956 secs 2024-07-26T16:43:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:59 smithi189 ceph-mon[36790]: pgmap v10749: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:43:59.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:43:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18980.479956 secs 2024-07-26T16:44:01.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:01 smithi193 ceph-mon[37473]: pgmap v10750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:01.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:01 smithi134 ceph-mon[31966]: pgmap v10750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:01.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:01 smithi189 ceph-mon[36790]: pgmap v10750: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:02.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 18999.901720 secs 2024-07-26T16:44:02.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 18999.901720 secs 2024-07-26T16:44:02.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 18999.901720 secs 2024-07-26T16:44:02.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:44:02.987+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:44:02.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:44:02.987+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:44:02.989 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:44:02.987+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:44:03.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:03 smithi134 ceph-mon[31966]: pgmap v10751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:03.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:03 smithi189 ceph-mon[36790]: pgmap v10751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:03.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:03 smithi193 ceph-mon[37473]: pgmap v10751: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:04.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18985.480051 secs 2024-07-26T16:44:04.675 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18985.480051 secs 2024-07-26T16:44:04.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18985.480051 secs 2024-07-26T16:44:05.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:05 smithi134 ceph-mon[31966]: pgmap v10752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:05.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:05 smithi189 ceph-mon[36790]: pgmap v10752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:05.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:05 smithi193 ceph-mon[37473]: pgmap v10752: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:07.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:07 smithi134 ceph-mon[31966]: pgmap v10753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:07.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19004.901840 secs 2024-07-26T16:44:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:07 smithi189 ceph-mon[36790]: pgmap v10753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:07.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19004.901840 secs 2024-07-26T16:44:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:07 smithi193 ceph-mon[37473]: pgmap v10753: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:07.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19004.901840 secs 2024-07-26T16:44:09.475 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:09 smithi134 ceph-mon[31966]: pgmap v10754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:09.476 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18990.480177 secs 2024-07-26T16:44:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:09 smithi189 ceph-mon[36790]: pgmap v10754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:09.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18990.480177 secs 2024-07-26T16:44:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:09 smithi193 ceph-mon[37473]: pgmap v10754: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:09.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18990.480177 secs 2024-07-26T16:44:11.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:11 smithi134 ceph-mon[31966]: pgmap v10755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:11.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:11 smithi189 ceph-mon[36790]: pgmap v10755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:11.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:11 smithi193 ceph-mon[37473]: pgmap v10755: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:12.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19009.901980 secs 2024-07-26T16:44:12.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19009.901980 secs 2024-07-26T16:44:12.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19009.901980 secs 2024-07-26T16:44:13.528 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:44:13.526+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:44:13.528 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:44:13.526+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:44:13.528 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:44:13.526+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:44:13.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:13 smithi134 ceph-mon[31966]: pgmap v10756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:13.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:13 smithi189 ceph-mon[36790]: pgmap v10756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:13.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:13 smithi193 ceph-mon[37473]: pgmap v10756: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 18995.480288 secs 2024-07-26T16:44:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 18995.480288 secs 2024-07-26T16:44:14.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 18995.480288 secs 2024-07-26T16:44:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:15 smithi134 ceph-mon[31966]: pgmap v10757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:15 smithi189 ceph-mon[36790]: pgmap v10757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:15.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:15 smithi193 ceph-mon[37473]: pgmap v10757: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:17.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:17 smithi134 ceph-mon[31966]: pgmap v10758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:17.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19014.902077 secs 2024-07-26T16:44:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:17 smithi189 ceph-mon[36790]: pgmap v10758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:17.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19014.902077 secs 2024-07-26T16:44:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:17 smithi193 ceph-mon[37473]: pgmap v10758: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:17.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19014.902077 secs 2024-07-26T16:44:17.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:44:17.993+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:44:17.995 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:44:17.993+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:44:17.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:44:17.993+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:44:19.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:19 smithi134 ceph-mon[31966]: pgmap v10759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:19.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19000.480430 secs 2024-07-26T16:44:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:19 smithi189 ceph-mon[36790]: pgmap v10759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:19.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19000.480430 secs 2024-07-26T16:44:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:19 smithi193 ceph-mon[37473]: pgmap v10759: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:19.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19000.480430 secs 2024-07-26T16:44:21.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:21 smithi134 ceph-mon[31966]: pgmap v10760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:21.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:21 smithi189 ceph-mon[36790]: pgmap v10760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:21.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:21 smithi193 ceph-mon[37473]: pgmap v10760: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:22.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19019.902201 secs 2024-07-26T16:44:22.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19019.902201 secs 2024-07-26T16:44:22.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19019.902201 secs 2024-07-26T16:44:23.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:23 smithi134 ceph-mon[31966]: pgmap v10761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:23.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:23 smithi189 ceph-mon[36790]: pgmap v10761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:23.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:23 smithi193 ceph-mon[37473]: pgmap v10761: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19005.480568 secs 2024-07-26T16:44:24.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:24 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:44:24.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19005.480568 secs 2024-07-26T16:44:24.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:24 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:44:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19005.480568 secs 2024-07-26T16:44:24.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:24 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:44:25.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:25 smithi134 ceph-mon[31966]: pgmap v10762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:25.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:44:25.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:44:25.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:44:25.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:25 smithi189 ceph-mon[36790]: pgmap v10762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:25.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:44:25.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:44:25.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:44:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:25 smithi193 ceph-mon[37473]: pgmap v10762: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:44:25.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:44:25.775 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:44:26.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:26 smithi134 ceph-mon[31966]: pgmap v10763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:44:26.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:26 smithi134 ceph-mon[31966]: pgmap v10764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:44:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:26 smithi189 ceph-mon[36790]: pgmap v10763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:44:26.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:26 smithi189 ceph-mon[36790]: pgmap v10764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:44:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:26 smithi193 ceph-mon[37473]: pgmap v10763: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:44:26.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:26 smithi193 ceph-mon[37473]: pgmap v10764: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:44:27.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19024.902339 secs 2024-07-26T16:44:27.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19024.902339 secs 2024-07-26T16:44:27.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19024.902339 secs 2024-07-26T16:44:28.537 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:44:28.535+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:44:28.537 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:44:28.535+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:44:28.537 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:44:28.535+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:44:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:28 smithi134 ceph-mon[31966]: pgmap v10765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T16:44:28.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:28 smithi189 ceph-mon[36790]: pgmap v10765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T16:44:28.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:28 smithi193 ceph-mon[37473]: pgmap v10765: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-07-26T16:44:29.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19010.480710 secs 2024-07-26T16:44:29.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19010.480710 secs 2024-07-26T16:44:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19010.480710 secs 2024-07-26T16:44:30.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:30 smithi189 ceph-mon[36790]: pgmap v10766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:44:30.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:30 smithi193 ceph-mon[37473]: pgmap v10766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:44:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:30 smithi134 ceph-mon[31966]: pgmap v10766: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:44:32.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19029.902466 secs 2024-07-26T16:44:32.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:32 smithi189 ceph-mon[36790]: pgmap v10767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:44:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19029.902466 secs 2024-07-26T16:44:32.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:32 smithi193 ceph-mon[37473]: pgmap v10767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:44:32.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19029.902466 secs 2024-07-26T16:44:32.938 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:32 smithi134 ceph-mon[31966]: pgmap v10767: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:44:33.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:44:32.997+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:44:33.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:44:32.997+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:44:33.000 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:44:32.997+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:44:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:34 smithi189 ceph-mon[36790]: pgmap v10768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:44:34.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19015.480843 secs 2024-07-26T16:44:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:34 smithi193 ceph-mon[37473]: pgmap v10768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:44:34.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19015.480843 secs 2024-07-26T16:44:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:34 smithi134 ceph-mon[31966]: pgmap v10768: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:44:34.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19015.480843 secs 2024-07-26T16:44:36.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:36 smithi189 ceph-mon[36790]: pgmap v10769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:44:36.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:36 smithi193 ceph-mon[37473]: pgmap v10769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:44:36.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:36 smithi134 ceph-mon[31966]: pgmap v10769: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:44:37.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19034.902587 secs 2024-07-26T16:44:37.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19034.902587 secs 2024-07-26T16:44:37.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19034.902587 secs 2024-07-26T16:44:38.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:38 smithi193 ceph-mon[37473]: pgmap v10770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:38.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:38 smithi134 ceph-mon[31966]: pgmap v10770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:38.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:38 smithi189 ceph-mon[36790]: pgmap v10770: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-07-26T16:44:39.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19020.480980 secs 2024-07-26T16:44:39.845 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19020.480980 secs 2024-07-26T16:44:39.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19020.480980 secs 2024-07-26T16:44:40.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:40 smithi193 ceph-mon[37473]: pgmap v10771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:40.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:40 smithi134 ceph-mon[31966]: pgmap v10771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:40.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:40 smithi189 ceph-mon[36790]: pgmap v10771: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:42 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19039.902718 secs 2024-07-26T16:44:42.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:42 smithi193 ceph-mon[37473]: pgmap v10772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:42.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:42 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19039.902718 secs 2024-07-26T16:44:42.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:42 smithi134 ceph-mon[31966]: pgmap v10772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:42 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19039.902718 secs 2024-07-26T16:44:42.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:42 smithi189 ceph-mon[36790]: pgmap v10772: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:44:43.544+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:44:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:44:43.544+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:44:43.546 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:44:43.544+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:44:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:44 smithi193 ceph-mon[37473]: pgmap v10773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:44.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19025.481101 secs 2024-07-26T16:44:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:44 smithi134 ceph-mon[31966]: pgmap v10773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:44.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19025.481101 secs 2024-07-26T16:44:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:44 smithi189 ceph-mon[36790]: pgmap v10773: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:44.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19025.481101 secs 2024-07-26T16:44:46.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:46 smithi193 ceph-mon[37473]: pgmap v10774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:46.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:46 smithi134 ceph-mon[31966]: pgmap v10774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:46.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:46 smithi189 ceph-mon[36790]: pgmap v10774: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:47.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:47 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19044.902841 secs 2024-07-26T16:44:47.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:47 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19044.902841 secs 2024-07-26T16:44:47.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:47 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19044.902841 secs 2024-07-26T16:44:48.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:44:48.010+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:44:48.012 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:44:48.010+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:44:48.013 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:44:48.010+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:44:48.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:48 smithi193 ceph-mon[37473]: pgmap v10775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:44:48.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:48 smithi134 ceph-mon[31966]: pgmap v10775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:44:48.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:48 smithi189 ceph-mon[36790]: pgmap v10775: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:44:49.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19030.481220 secs 2024-07-26T16:44:49.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19030.481220 secs 2024-07-26T16:44:49.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19030.481220 secs 2024-07-26T16:44:50.774 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:50 smithi193 ceph-mon[37473]: pgmap v10776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:50.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:50 smithi134 ceph-mon[31966]: pgmap v10776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:50.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:50 smithi189 ceph-mon[36790]: pgmap v10776: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-07-26T16:44:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:52 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19049.902959 secs 2024-07-26T16:44:52.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:52 smithi134 ceph-mon[31966]: pgmap v10777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:52 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19049.902959 secs 2024-07-26T16:44:52.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:52 smithi189 ceph-mon[36790]: pgmap v10777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:52 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19049.902959 secs 2024-07-26T16:44:53.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:52 smithi193 ceph-mon[37473]: pgmap v10777: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:54.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:54 smithi134 ceph-mon[31966]: pgmap v10778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:54.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19035.481343 secs 2024-07-26T16:44:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:54 smithi189 ceph-mon[36790]: pgmap v10778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:54.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19035.481343 secs 2024-07-26T16:44:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:54 smithi193 ceph-mon[37473]: pgmap v10778: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:44:55.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19035.481343 secs 2024-07-26T16:44:56.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:56 smithi134 ceph-mon[31966]: pgmap v10779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:56.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:56 smithi189 ceph-mon[36790]: pgmap v10779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:57.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:56 smithi193 ceph-mon[37473]: pgmap v10779: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:44:57.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:57 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19054.903089 secs 2024-07-26T16:44:57.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:57 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19054.903089 secs 2024-07-26T16:44:58.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:57 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19054.903089 secs 2024-07-26T16:44:58.557 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:44:58.555+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:44:58.557 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:44:58.555+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:44:58.558 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:44:58.555+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:44:58.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:58 smithi134 ceph-mon[31966]: pgmap v10780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:44:58.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:58 smithi189 ceph-mon[36790]: pgmap v10780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:44:59.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:58 smithi193 ceph-mon[37473]: pgmap v10780: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:44:59.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:44:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19040.481482 secs 2024-07-26T16:44:59.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:44:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19040.481482 secs 2024-07-26T16:45:00.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:44:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19040.481482 secs 2024-07-26T16:45:00.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:00 smithi134 ceph-mon[31966]: pgmap v10781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:45:00.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:00 smithi189 ceph-mon[36790]: pgmap v10781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:45:01.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:00 smithi193 ceph-mon[37473]: pgmap v10781: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:45:02.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:02 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19059.903167 secs 2024-07-26T16:45:02.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:02 smithi134 ceph-mon[31966]: pgmap v10782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:02.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:02 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19059.903167 secs 2024-07-26T16:45:02.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:02 smithi189 ceph-mon[36790]: pgmap v10782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:02 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19059.903167 secs 2024-07-26T16:45:03.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:02 smithi193 ceph-mon[37473]: pgmap v10782: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:03.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:45:03.026+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:45:03.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:45:03.026+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:45:03.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:45:03.026+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:45:04.827 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:04 smithi193 ceph-mon[37473]: pgmap v10783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:45:04.827 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19045.481631 secs 2024-07-26T16:45:04.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:04 smithi134 ceph-mon[31966]: pgmap v10783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:45:04.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19045.481631 secs 2024-07-26T16:45:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:04 smithi189 ceph-mon[36790]: pgmap v10783: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:45:04.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19045.481631 secs 2024-07-26T16:45:06.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:06 smithi134 ceph-mon[31966]: pgmap v10784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:06.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:06 smithi189 ceph-mon[36790]: pgmap v10784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:07.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:06 smithi193 ceph-mon[37473]: pgmap v10784: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:07.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:07 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19064.903337 secs 2024-07-26T16:45:07.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:07 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19064.903337 secs 2024-07-26T16:45:08.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:07 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19064.903337 secs 2024-07-26T16:45:08.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:08 smithi134 ceph-mon[31966]: pgmap v10785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:45:08.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:08 smithi189 ceph-mon[36790]: pgmap v10785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:45:09.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:08 smithi193 ceph-mon[37473]: pgmap v10785: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:45:09.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19050.481784 secs 2024-07-26T16:45:09.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19050.481784 secs 2024-07-26T16:45:10.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19050.481784 secs 2024-07-26T16:45:10.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:10 smithi134 ceph-mon[31966]: pgmap v10786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:45:10.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:10 smithi189 ceph-mon[36790]: pgmap v10786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:45:11.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:10 smithi193 ceph-mon[37473]: pgmap v10786: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:45:12.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:12 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19069.903450 secs 2024-07-26T16:45:12.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:12 smithi134 ceph-mon[31966]: pgmap v10787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:45:12.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:12 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19069.903450 secs 2024-07-26T16:45:12.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:12 smithi189 ceph-mon[36790]: pgmap v10787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:45:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:12 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19069.903450 secs 2024-07-26T16:45:13.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:12 smithi193 ceph-mon[37473]: pgmap v10787: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:45:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:45:13.560+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:45:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:45:13.560+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:45:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:45:13.560+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:45:14.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:14 smithi134 ceph-mon[31966]: pgmap v10788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:45:14.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19055.481903 secs 2024-07-26T16:45:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:14 smithi189 ceph-mon[36790]: pgmap v10788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:45:14.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19055.481903 secs 2024-07-26T16:45:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:14 smithi193 ceph-mon[37473]: pgmap v10788: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-07-26T16:45:15.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19055.481903 secs 2024-07-26T16:45:16.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:16 smithi134 ceph-mon[31966]: pgmap v10789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:45:16.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:16 smithi189 ceph-mon[36790]: pgmap v10789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:45:17.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:16 smithi193 ceph-mon[37473]: pgmap v10789: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 7 op/s 2024-07-26T16:45:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19074.903524 secs 2024-07-26T16:45:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19074.903524 secs 2024-07-26T16:45:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19074.903524 secs 2024-07-26T16:45:18.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:45:18.032+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:45:18.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:45:18.032+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:45:18.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:45:18.032+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:45:18.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:18 smithi134 ceph-mon[31966]: pgmap v10790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:45:18.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:18 smithi189 ceph-mon[36790]: pgmap v10790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:45:19.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:18 smithi193 ceph-mon[37473]: pgmap v10790: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:45:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19060.482055 secs 2024-07-26T16:45:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19060.482055 secs 2024-07-26T16:45:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19060.482055 secs 2024-07-26T16:45:20.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:20 smithi134 ceph-mon[31966]: pgmap v10791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:45:20.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:20 smithi189 ceph-mon[36790]: pgmap v10791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:45:21.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:20 smithi193 ceph-mon[37473]: pgmap v10791: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-07-26T16:45:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19079.903642 secs 2024-07-26T16:45:22.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:22 smithi134 ceph-mon[31966]: pgmap v10792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19079.903642 secs 2024-07-26T16:45:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:22 smithi189 ceph-mon[36790]: pgmap v10792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19079.903642 secs 2024-07-26T16:45:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:22 smithi193 ceph-mon[37473]: pgmap v10792: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:24 smithi134 ceph-mon[31966]: pgmap v10793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:45:24.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19065.482204 secs 2024-07-26T16:45:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:24 smithi189 ceph-mon[36790]: pgmap v10793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:45:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19065.482204 secs 2024-07-26T16:45:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:24 smithi193 ceph-mon[37473]: pgmap v10793: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:45:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19065.482204 secs 2024-07-26T16:45:25.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:45:25.641+0000 7f9b41600640 1 -- 172.21.15.134:0/4006618154 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b24006560 con 0x7f9b4c10f330 2024-07-26T16:45:25.643 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:45:25.641+0000 7f9b2aa00640 1 -- 172.21.15.134:0/1872420438 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f9b20003690 con 0x55c65e5ee670 2024-07-26T16:45:25.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:45:25.642+0000 7f9b42a00640 1 -- 172.21.15.134:0/4006618154 <== mon.2 v2:172.21.15.189:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f9b20003690 con 0x7f9b4c10f330 2024-07-26T16:45:25.644 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:45:25.642+0000 7f9b2be00640 1 -- 172.21.15.134:0/1872420438 <== mon.1 v2:172.21.15.193:3300/0 148 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f9b24006560 con 0x55c65e5ee670 2024-07-26T16:45:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:25 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:45:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:25 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:45:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:25 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:45:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:26 smithi189 ceph-mon[36790]: pgmap v10794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:45:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:45:26.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:26 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:45:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:26 smithi193 ceph-mon[37473]: pgmap v10794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:45:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:45:27.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:26 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:45:27.049 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:26 smithi134 ceph-mon[31966]: pgmap v10794: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:27.049 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:45:27.049 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:45:27.049 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:26 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:45:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19084.903763 secs 2024-07-26T16:45:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:27 smithi189 ceph-mon[36790]: pgmap v10795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:45:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19084.903763 secs 2024-07-26T16:45:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:27 smithi193 ceph-mon[37473]: pgmap v10795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:45:28.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19084.903763 secs 2024-07-26T16:45:28.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:27 smithi134 ceph-mon[31966]: pgmap v10795: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:45:28.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:45:28.576+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:45:28.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:45:28.576+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:45:28.578 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:45:28.576+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:45:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:29 smithi189 ceph-mon[36790]: pgmap v10796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:45:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19070.482293 secs 2024-07-26T16:45:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:29 smithi193 ceph-mon[37473]: pgmap v10796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:45:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19070.482293 secs 2024-07-26T16:45:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:29 smithi134 ceph-mon[31966]: pgmap v10796: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:45:30.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19070.482293 secs 2024-07-26T16:45:32.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:31 smithi193 ceph-mon[37473]: pgmap v10797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:45:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:31 smithi134 ceph-mon[31966]: pgmap v10797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:45:32.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:31 smithi189 ceph-mon[36790]: pgmap v10797: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s 2024-07-26T16:45:33.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:32 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19089.903894 secs 2024-07-26T16:45:33.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:45:33.040+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:45:33.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:45:33.040+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:45:33.043 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:45:33.040+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:45:33.043 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:32 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19089.903894 secs 2024-07-26T16:45:33.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:32 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19089.903894 secs 2024-07-26T16:45:34.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:33 smithi193 ceph-mon[37473]: pgmap v10798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:45:34.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:33 smithi134 ceph-mon[31966]: pgmap v10798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:45:34.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:33 smithi189 ceph-mon[36790]: pgmap v10798: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-07-26T16:45:35.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19075.482448 secs 2024-07-26T16:45:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19075.482448 secs 2024-07-26T16:45:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19075.482448 secs 2024-07-26T16:45:36.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:35 smithi193 ceph-mon[37473]: pgmap v10799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:45:36.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:35 smithi134 ceph-mon[31966]: pgmap v10799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:45:36.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:35 smithi189 ceph-mon[36790]: pgmap v10799: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:45:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:45:36.195+0000 7f21bf400640 1 -- 172.21.15.134:0/939165295 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:45:36.197 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:45:36.195+0000 7f21abe00640 1 -- 172.21.15.134:0/3456275332 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:45:36.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:45:36.196+0000 7f21c4c00640 1 -- 172.21.15.134:0/939165295 <== mon.2 v2:172.21.15.189:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f21980065a0 con 0x7f21c8105550 2024-07-26T16:45:36.198 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:45:36.196+0000 7f21b5600640 1 -- 172.21.15.134:0/3456275332 <== mon.2 v2:172.21.15.189:3300/0 149 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f2194006440 con 0x55a08e2d94d0 2024-07-26T16:45:38.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:37 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19094.904010 secs 2024-07-26T16:45:38.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:37 smithi193 ceph-mon[37473]: pgmap v10800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:45:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:37 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19094.904010 secs 2024-07-26T16:45:38.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:37 smithi134 ceph-mon[31966]: pgmap v10800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:45:38.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:37 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19094.904010 secs 2024-07-26T16:45:38.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:37 smithi189 ceph-mon[36790]: pgmap v10800: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-07-26T16:45:39.996 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:39 smithi189 ceph-mon[36790]: pgmap v10801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:45:39.996 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19080.482590 secs 2024-07-26T16:45:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:39 smithi193 ceph-mon[37473]: pgmap v10801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:45:40.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19080.482590 secs 2024-07-26T16:45:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:39 smithi134 ceph-mon[31966]: pgmap v10801: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:45:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19080.482590 secs 2024-07-26T16:45:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:41 smithi193 ceph-mon[37473]: pgmap v10802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:42.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19099.904148 secs 2024-07-26T16:45:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:41 smithi134 ceph-mon[31966]: pgmap v10802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:42.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19099.904148 secs 2024-07-26T16:45:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:41 smithi189 ceph-mon[36790]: pgmap v10802: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19099.904148 secs 2024-07-26T16:45:43.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:45:43.584+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:45:43.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:45:43.584+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:45:43.586 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:45:43.584+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:45:44.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:43 smithi193 ceph-mon[37473]: pgmap v10803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:45:44.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:43 smithi134 ceph-mon[31966]: pgmap v10803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:45:44.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:43 smithi189 ceph-mon[36790]: pgmap v10803: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:45:45.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19085.482734 secs 2024-07-26T16:45:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19085.482734 secs 2024-07-26T16:45:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19085.482734 secs 2024-07-26T16:45:46.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:45 smithi193 ceph-mon[37473]: pgmap v10804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:45:46.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:45 smithi134 ceph-mon[31966]: pgmap v10804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:45:46.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:45 smithi189 ceph-mon[36790]: pgmap v10804: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:45:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19104.904268 secs 2024-07-26T16:45:47.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19104.904268 secs 2024-07-26T16:45:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19104.904268 secs 2024-07-26T16:45:48.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:45:48.056+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:45:48.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:45:48.056+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:45:48.058 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:45:48.056+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:45:48.059 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:47 smithi134 ceph-mon[31966]: pgmap v10805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:48.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:47 smithi189 ceph-mon[36790]: pgmap v10805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:48.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:47 smithi193 ceph-mon[37473]: pgmap v10805: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:49 smithi134 ceph-mon[31966]: pgmap v10806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:45:50.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19090.482862 secs 2024-07-26T16:45:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:49 smithi189 ceph-mon[36790]: pgmap v10806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:45:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19090.482862 secs 2024-07-26T16:45:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:49 smithi193 ceph-mon[37473]: pgmap v10806: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:45:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19090.482862 secs 2024-07-26T16:45:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:51 smithi134 ceph-mon[31966]: pgmap v10807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:52.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19109.904393 secs 2024-07-26T16:45:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:51 smithi189 ceph-mon[36790]: pgmap v10807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:52.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19109.904393 secs 2024-07-26T16:45:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:51 smithi193 ceph-mon[37473]: pgmap v10807: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19109.904393 secs 2024-07-26T16:45:54.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:53 smithi134 ceph-mon[31966]: pgmap v10808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:45:54.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:53 smithi189 ceph-mon[36790]: pgmap v10808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:45:54.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:53 smithi193 ceph-mon[37473]: pgmap v10808: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:45:55.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:54 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19095.482990 secs 2024-07-26T16:45:55.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:54 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19095.482990 secs 2024-07-26T16:45:55.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:54 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19095.482990 secs 2024-07-26T16:45:56.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:55 smithi134 ceph-mon[31966]: pgmap v10809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:45:56.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:55 smithi189 ceph-mon[36790]: pgmap v10809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:45:56.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:55 smithi193 ceph-mon[37473]: pgmap v10809: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:45:57.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:56 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19114.904478 secs 2024-07-26T16:45:57.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:56 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19114.904478 secs 2024-07-26T16:45:57.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:56 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19114.904478 secs 2024-07-26T16:45:58.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:57 smithi134 ceph-mon[31966]: pgmap v10810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:58.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:57 smithi189 ceph-mon[36790]: pgmap v10810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:58.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:57 smithi193 ceph-mon[37473]: pgmap v10810: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:45:58.601 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:45:58.599+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:45:58.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:45:58.600+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:45:58.602 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:45:58.600+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:46:00.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:59 smithi134 ceph-mon[31966]: pgmap v10811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:46:00.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:45:59 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19100.483119 secs 2024-07-26T16:46:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:59 smithi189 ceph-mon[36790]: pgmap v10811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:46:00.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:45:59 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19100.483119 secs 2024-07-26T16:46:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:59 smithi193 ceph-mon[37473]: pgmap v10811: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:46:00.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:45:59 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19100.483119 secs 2024-07-26T16:46:02.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:01 smithi134 ceph-mon[31966]: pgmap v10812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:02.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:01 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19119.904546 secs 2024-07-26T16:46:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:01 smithi189 ceph-mon[36790]: pgmap v10812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:02.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:01 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19119.904546 secs 2024-07-26T16:46:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:01 smithi193 ceph-mon[37473]: pgmap v10812: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:02.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:01 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19119.904546 secs 2024-07-26T16:46:03.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:46:03.059+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:46:03.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:46:03.059+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:46:03.061 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:46:03.059+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:46:04.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:03 smithi134 ceph-mon[31966]: pgmap v10813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:46:04.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:03 smithi189 ceph-mon[36790]: pgmap v10813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:46:04.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:03 smithi193 ceph-mon[37473]: pgmap v10813: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:46:05.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:04 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19105.483240 secs 2024-07-26T16:46:05.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:04 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19105.483240 secs 2024-07-26T16:46:05.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:04 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19105.483240 secs 2024-07-26T16:46:06.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:05 smithi134 ceph-mon[31966]: pgmap v10814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:46:06.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:05 smithi189 ceph-mon[36790]: pgmap v10814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:46:06.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:05 smithi193 ceph-mon[37473]: pgmap v10814: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:46:07.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:06 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19124.904664 secs 2024-07-26T16:46:07.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:06 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19124.904664 secs 2024-07-26T16:46:07.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:06 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19124.904664 secs 2024-07-26T16:46:08.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:07 smithi134 ceph-mon[31966]: pgmap v10815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:08.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:07 smithi189 ceph-mon[36790]: pgmap v10815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:08.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:07 smithi193 ceph-mon[37473]: pgmap v10815: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:09.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:08 smithi134 ceph-mon[31966]: pgmap v10816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:46:09.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:08 smithi189 ceph-mon[36790]: pgmap v10816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:46:09.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:08 smithi193 ceph-mon[37473]: pgmap v10816: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:46:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:09 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19110.483304 secs 2024-07-26T16:46:10.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:09 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19110.483304 secs 2024-07-26T16:46:10.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:09 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19110.483304 secs 2024-07-26T16:46:11.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:10 smithi134 ceph-mon[31966]: pgmap v10817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:11.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:10 smithi189 ceph-mon[36790]: pgmap v10817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:11.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:10 smithi193 ceph-mon[37473]: pgmap v10817: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:12.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:11 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19129.904810 secs 2024-07-26T16:46:12.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:11 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19129.904810 secs 2024-07-26T16:46:12.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:11 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19129.904810 secs 2024-07-26T16:46:13.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:12 smithi189 ceph-mon[36790]: pgmap v10818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:46:13.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:12 smithi193 ceph-mon[37473]: pgmap v10818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:46:13.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:12 smithi134 ceph-mon[31966]: pgmap v10818: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:46:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:46:13.615+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:46:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:46:13.615+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:46:13.617 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:46:13.615+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:46:14.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:14 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19115.483475 secs 2024-07-26T16:46:14.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:14 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19115.483475 secs 2024-07-26T16:46:14.693 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:14 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19115.483475 secs 2024-07-26T16:46:15.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:15 smithi193 ceph-mon[37473]: pgmap v10819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:46:15.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:15 smithi134 ceph-mon[31966]: pgmap v10819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:46:15.694 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:15 smithi189 ceph-mon[36790]: pgmap v10819: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:46:17.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:17 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19134.904931 secs 2024-07-26T16:46:17.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:17 smithi134 ceph-mon[31966]: pgmap v10820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:17 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19134.904931 secs 2024-07-26T16:46:17.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:17 smithi189 ceph-mon[36790]: pgmap v10820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:17 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19134.904931 secs 2024-07-26T16:46:18.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:17 smithi193 ceph-mon[37473]: pgmap v10820: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:18.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:46:18.074+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:46:18.076 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:46:18.074+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:46:18.077 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:46:18.074+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:46:19.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:19 smithi134 ceph-mon[31966]: pgmap v10821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:46:19.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:19 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19120.483610 secs 2024-07-26T16:46:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:19 smithi189 ceph-mon[36790]: pgmap v10821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:46:19.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:19 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19120.483610 secs 2024-07-26T16:46:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:19 smithi193 ceph-mon[37473]: pgmap v10821: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:46:20.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:19 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19120.483610 secs 2024-07-26T16:46:21.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:21 smithi134 ceph-mon[31966]: pgmap v10822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:21.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:21 smithi189 ceph-mon[36790]: pgmap v10822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:22.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:21 smithi193 ceph-mon[37473]: pgmap v10822: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:22.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:22 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19139.905051 secs 2024-07-26T16:46:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:22 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19139.905051 secs 2024-07-26T16:46:23.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:22 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19139.905051 secs 2024-07-26T16:46:23.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:23 smithi134 ceph-mon[31966]: pgmap v10823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:46:23.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:23 smithi189 ceph-mon[36790]: pgmap v10823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:46:24.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:23 smithi193 ceph-mon[37473]: pgmap v10823: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-07-26T16:46:24.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:24 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19125.483753 secs 2024-07-26T16:46:24.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:24 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19125.483753 secs 2024-07-26T16:46:25.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:24 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19125.483753 secs 2024-07-26T16:46:25.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:25 smithi134 ceph-mon[31966]: pgmap v10824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:46:25.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:25 smithi189 ceph-mon[36790]: pgmap v10824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:46:26.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:25 smithi193 ceph-mon[37473]: pgmap v10824: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2024-07-26T16:46:27.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:27 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19144.905170 secs 2024-07-26T16:46:27.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:27 smithi134 ceph-mon[31966]: pgmap v10825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:27.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:27 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:46:27.943 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:27 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19144.905170 secs 2024-07-26T16:46:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:27 smithi189 ceph-mon[36790]: pgmap v10825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:27.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:27 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:46:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:27 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19144.905170 secs 2024-07-26T16:46:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:27 smithi193 ceph-mon[37473]: pgmap v10825: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:28.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:27 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-07-26T16:46:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:46:28.631+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:46:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:46:28.631+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:46:28.633 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:46:28.631+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:46:28.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:46:28.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:46:28.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:28 smithi134 ceph-mon[31966]: pgmap v10826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T16:46:28.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:28 smithi134 ceph-mon[31966]: pgmap v10827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:28.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:28 smithi134 ceph-mon[31966]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:46:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:46:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:46:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:28 smithi189 ceph-mon[36790]: pgmap v10826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T16:46:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:28 smithi189 ceph-mon[36790]: pgmap v10827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:28.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:28 smithi189 ceph-mon[36790]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:46:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-07-26T16:46:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-07-26T16:46:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:28 smithi193 ceph-mon[37473]: pgmap v10826: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-07-26T16:46:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:28 smithi193 ceph-mon[37473]: pgmap v10827: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:29.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:28 smithi193 ceph-mon[37473]: from='mgr.14211 172.21.15.134:0/323789710' entity='mgr.x' 2024-07-26T16:46:29.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:29 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19130.483825 secs 2024-07-26T16:46:29.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:29 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19130.483825 secs 2024-07-26T16:46:30.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:29 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19130.483825 secs 2024-07-26T16:46:30.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:30 smithi134 ceph-mon[31966]: pgmap v10828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:46:30.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:30 smithi189 ceph-mon[36790]: pgmap v10828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:46:31.024 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:30 smithi193 ceph-mon[37473]: pgmap v10828: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 7 op/s 2024-07-26T16:46:32.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:31 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19149.905258 secs 2024-07-26T16:46:32.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:31 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19149.905258 secs 2024-07-26T16:46:32.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:31 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19149.905258 secs 2024-07-26T16:46:33.092 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:46:33.090+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:46:33.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:46:33.090+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:46:33.093 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:46:33.090+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:46:33.093 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:32 smithi134 ceph-mon[31966]: pgmap v10829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:46:33.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:32 smithi189 ceph-mon[36790]: pgmap v10829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:46:33.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:32 smithi193 ceph-mon[37473]: pgmap v10829: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:46:35.077 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:34 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19135.483979 secs 2024-07-26T16:46:35.077 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:34 smithi193 ceph-mon[37473]: pgmap v10830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:35.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:34 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19135.483979 secs 2024-07-26T16:46:35.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:34 smithi134 ceph-mon[31966]: pgmap v10830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:34 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19135.483979 secs 2024-07-26T16:46:35.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:34 smithi189 ceph-mon[36790]: pgmap v10830: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:37.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:36 smithi134 ceph-mon[31966]: pgmap v10831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:37.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:36 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19154.905376 secs 2024-07-26T16:46:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:36 smithi189 ceph-mon[36790]: pgmap v10831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:37.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:36 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19154.905376 secs 2024-07-26T16:46:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:36 smithi193 ceph-mon[37473]: pgmap v10831: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-07-26T16:46:37.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:36 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19154.905376 secs 2024-07-26T16:46:39.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:38 smithi134 ceph-mon[31966]: pgmap v10832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:46:39.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:38 smithi189 ceph-mon[36790]: pgmap v10832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:46:39.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:38 smithi193 ceph-mon[37473]: pgmap v10832: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 4.0 KiB/s rd, 7 op/s 2024-07-26T16:46:40.095 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:39 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19140.484103 secs 2024-07-26T16:46:40.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:39 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19140.484103 secs 2024-07-26T16:46:40.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:39 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19140.484103 secs 2024-07-26T16:46:41.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:40 smithi134 ceph-mon[31966]: pgmap v10833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:46:41.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:40 smithi189 ceph-mon[36790]: pgmap v10833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:46:41.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:40 smithi193 ceph-mon[37473]: pgmap v10833: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:46:42.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:41 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19159.905517 secs 2024-07-26T16:46:42.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:41 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19159.905517 secs 2024-07-26T16:46:42.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:41 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19159.905517 secs 2024-07-26T16:46:43.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:42 smithi134 ceph-mon[31966]: pgmap v10834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:43.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:42 smithi189 ceph-mon[36790]: pgmap v10834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:43.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:42 smithi193 ceph-mon[37473]: pgmap v10834: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:43.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:46:43.635+0000 7f21c7e00640 1 -- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 msgr2=0x55a08e6bba80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:46:43.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:46:43.635+0000 7f21c7e00640 1 --2- 172.21.15.134:0/3456275332 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55a08e2b8d80 0x55a08e6bba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:46:43.637 INFO:teuthology.task.background_exec.ubuntu@smithi134.front.sepia.ceph.com.smithi134.stderr:2024-07-26T16:46:43.635+0000 7f21b5600640 1 client.27041 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:46:45.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:44 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19145.484233 secs 2024-07-26T16:46:45.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:44 smithi134 ceph-mon[31966]: pgmap v10835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:46:45.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:44 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19145.484233 secs 2024-07-26T16:46:45.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:44 smithi189 ceph-mon[36790]: pgmap v10835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:46:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:44 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19145.484233 secs 2024-07-26T16:46:45.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:44 smithi193 ceph-mon[37473]: pgmap v10835: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:46:47.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:46 smithi134 ceph-mon[31966]: pgmap v10836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:47.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:46 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19164.905641 secs 2024-07-26T16:46:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:46 smithi189 ceph-mon[36790]: pgmap v10836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:47.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:46 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19164.905641 secs 2024-07-26T16:46:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:46 smithi193 ceph-mon[37473]: pgmap v10836: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:47.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:46 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19164.905641 secs 2024-07-26T16:46:48.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:46:48.105+0000 7f9b49e00640 1 -- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 msgr2=0x55c65e9d1dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.134:6840/4067549550 2024-07-26T16:46:48.108 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:46:48.105+0000 7f9b49e00640 1 --2- 172.21.15.134:0/1872420438 >> [v2:172.21.15.134:6840/4067549550,v1:172.21.15.134:6841/4067549550] conn(0x55c65e5cdcd0 0x55c65e9d1dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-07-26T16:46:48.109 INFO:teuthology.orchestra.run.smithi134.stderr:2024-07-26T16:46:48.105+0000 7f9b2be00640 1 client.27043 ms_handle_refused on v2:172.21.15.134:6840/4067549550 2024-07-26T16:46:49.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:48 smithi134 ceph-mon[31966]: pgmap v10837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:46:49.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:48 smithi189 ceph-mon[36790]: pgmap v10837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:46:49.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:48 smithi193 ceph-mon[37473]: pgmap v10837: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-07-26T16:46:50.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:49 smithi134 ceph-mon[31966]: 2 slow requests, 0 included below; oldest blocked for > 19150.484362 secs 2024-07-26T16:46:50.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:49 smithi189 ceph-mon[36790]: 2 slow requests, 0 included below; oldest blocked for > 19150.484362 secs 2024-07-26T16:46:50.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:49 smithi193 ceph-mon[37473]: 2 slow requests, 0 included below; oldest blocked for > 19150.484362 secs 2024-07-26T16:46:51.173 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:50 smithi134 ceph-mon[31966]: pgmap v10838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:46:51.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:50 smithi189 ceph-mon[36790]: pgmap v10838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:46:51.275 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:50 smithi193 ceph-mon[37473]: pgmap v10838: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-07-26T16:46:52.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:51 smithi134 ceph-mon[31966]: 1 slow requests, 0 included below; oldest blocked for > 19169.905783 secs 2024-07-26T16:46:52.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:51 smithi189 ceph-mon[36790]: 1 slow requests, 0 included below; oldest blocked for > 19169.905783 secs 2024-07-26T16:46:52.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:51 smithi193 ceph-mon[37473]: 1 slow requests, 0 included below; oldest blocked for > 19169.905783 secs 2024-07-26T16:46:53.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Jul 26 16:46:52 smithi134 ceph-mon[31966]: pgmap v10839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:53.194 INFO:journalctl@ceph.mon.b.smithi189.stdout:Jul 26 16:46:52 smithi189 ceph-mon[36790]: pgmap v10839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:53.274 INFO:journalctl@ceph.mon.c.smithi193.stdout:Jul 26 16:46:52 smithi193 ceph-mon[37473]: pgmap v10839: 129 pgs: 129 active+clean; 51 GiB data, 112 GiB used, 960 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-07-26T16:46:54.266 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-07-26T16:46:54.267 DEBUG:teuthology.task.console_log:Killing console logger for smithi134 2024-07-26T16:46:54.268 DEBUG:teuthology.task.console_log:Killing console logger for smithi189 2024-07-26T16:46:54.268 DEBUG:teuthology.task.console_log:Killing console logger for smithi193 2024-07-26T16:46:54.268 DEBUG:teuthology.exit:Finished running handlers